2021-02-10T10:51:53.303 INFO:root:teuthology version: 1.1.0-14fad317 2021-02-10T10:51:53.303 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_khiremat@teuthology --archive /home/teuthworker/archive/khiremat-2021-02-10_10:13:45-fs-wip-khiremat-mgr-authid-bp-7-nautilus-distro-basic-gibba/5872718 --description fs/basic_functional/{begin clusters/1-mds-4-client-coloc conf/{client mds mon osd} mount/fuse objectstore/bluestore-bitmap overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} supported-random-distros$/{rhel_7} tasks/volumes} --name khiremat-2021-02-10_10:13:45-fs-wip-khiremat-mgr-authid-bp-7-nautilus-distro-basic-gibba --verbose -- /home/teuthworker/archive/khiremat-2021-02-10_10:13:45-fs-wip-khiremat-mgr-authid-bp-7-nautilus-distro-basic-gibba/5872718/orig.config.yaml 2021-02-10T10:51:53.346 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2021-02-10T10:51:53.412 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/khiremat-2021-02-10_10:13:45-fs-wip-khiremat-mgr-authid-bp-7-nautilus-distro-basic-gibba/5872718 branch: wip-khiremat-mgr-authid-bp-7-nautilus description: fs/basic_functional/{begin clusters/1-mds-4-client-coloc conf/{client mds mon osd} mount/fuse objectstore/bluestore-bitmap overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} supported-random-distros$/{rhel_7} tasks/volumes} email: khiremat@redhat.com first_in_suite: false job_id: '5872718' kernel: kdb: true sha1: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: gibba name: khiremat-2021-02-10_10:13:45-fs-wip-khiremat-mgr-authid-bp-7-nautilus-distro-basic-gibba nuke-on-error: true openstack: - volumes: count: 4 size: 30 - machine: disk: 200 os_type: rhel os_version: '7.9' overrides: admin_socket: branch: wip-khiremat-mgr-authid-bp-7-nautilus ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 pid file: '' mds: debug mds: 20 debug ms: 1 mds bal frag: true mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 mgr: debug client: 10 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug filestore: 20 debug journal: 20 debug ms: 20 debug osd: 25 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(TOO_FEW_PGS\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - OSD full dropping all updates - OSD near full - pausewr flag - failsafe engaged, dropping updates - failsafe disengaged, no longer dropping - is full \(reached quota - POOL_FULL - POOL_BACKFILLFULL sha1: 962fac13b1a781b8c7d25c1eed50d93813b8f5f0 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: sha1: 962fac13b1a781b8c7d25c1eed50d93813b8f5f0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-khiremat-mgr-authid-bp-7-nautilus sha1: 962fac13b1a781b8c7d25c1eed50d93813b8f5f0 owner: scheduled_khiremat@teuthology priority: 50 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - client.1 - - mon.b - mon.c - mgr.x - mds.b - osd.4 - osd.5 - osd.6 - osd.7 - client.2 - client.3 sha1: 962fac13b1a781b8c7d25c1eed50d93813b8f5f0 sleep_before_teardown: 0 suite: fs suite_branch: wip-khiremat-mgr-authid-bp-7-nautilus suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_962fac13b1a781b8c7d25c1eed50d93813b8f5f0/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 962fac13b1a781b8c7d25c1eed50d93813b8f5f0 targets: gibba002.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2bmtFOE1KHoyhHloefVeAs5eOjJHc91Zkg84KoprB2NN75v56Zrnct2fAo7z4L3T2CG9e97gcvCwjA9SbXGMyASI8Qoinc2p9FRje3mos8DriJf+4yWh6E9sZUlAyVkF5xSKS8Z9eS/FceXbqWb42s+KOXCt3AVNZFrkBlu8jyeUFz5BCaEl+6CHQqxzJCWa/KpKGgqFCQf1N0DV9eAXBCBOwo5jpVcgvq8oEq28fZmx6b8Yr77QeFDj0fcn32I57e0aQTE13jmc2lAZbSUWM/HPyzVefFxIWfg/YXaA5MWE1+gEUXpc5k3cSmyq6ZNXdpI3EaqGHUuJKxNaDR2vT gibba013.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBkalw17+zsGdqNpcFqATXsFUNbxrsotr+ltN6gwKELl9WUVYSNUgi/nlVJOdtCaJzywCFRbScdjK67BCw40xZ6ReZuBXgznlPP2VAyakp3rjQYVNZdQEuMhLOVaQiradixCkYypPJV3d+RwTL5yj29g/05PRqFiSDcsH1vqr4KwKzqd14lZa9XxMzUB3oKk0XlhF4N2+WWwc1waQ5Oy8pAifhoLcOaHC+eRwvTwzBbMlmJFMoDOq/ocQd/MgfcMR4wp3QwRbGz4eIJ753fuHtAhemPdv83QPpudbH+1/Q3sH5jL1D/KezMkgRG9CWngAHRGr7Ea0zHPspcxlIpV4N tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell rpm: - python3-cephfs extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - ceph: null - ceph-fuse: null - cephfs_test_runner: fail_on_skip: false modules: - tasks.cephfs.test_volumes teuthology_branch: master timestamp: 2021-02-10_10:13:45 tube: gibba user: khiremat verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.gibba.29737 2021-02-10T10:51:53.412 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_962fac13b1a781b8c7d25c1eed50d93813b8f5f0/qa; will attempt to use it 2021-02-10T10:51:53.420 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_962fac13b1a781b8c7d25c1eed50d93813b8f5f0/qa/tasks 2021-02-10T10:51:53.420 INFO:teuthology.run_tasks:Running task internal.check_packages... 2021-02-10T10:51:53.422 INFO:teuthology.task.internal:Checking packages... 2021-02-10T10:51:53.478 INFO:teuthology.task.internal:Checking packages for os_type 'rhel', flavor 'basic' and ceph hash '962fac13b1a781b8c7d25c1eed50d93813b8f5f0' 2021-02-10T10:51:53.478 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2021-02-10T10:51:53.478 INFO:teuthology.packaging:ref: None 2021-02-10T10:51:53.478 INFO:teuthology.packaging:tag: None 2021-02-10T10:51:53.479 INFO:teuthology.packaging:branch: wip-khiremat-mgr-authid-bp-7-nautilus 2021-02-10T10:51:53.479 INFO:teuthology.packaging:sha1: 962fac13b1a781b8c7d25c1eed50d93813b8f5f0 2021-02-10T10:51:53.479 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F7%2Fx86_64&ref=wip-khiremat-mgr-authid-bp-7-nautilus 2021-02-10T10:51:53.652 INFO:teuthology.task.internal:Found packages for ceph version 14.2.16-280.g962fac13 2021-02-10T10:51:53.652 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2021-02-10T10:51:53.663 INFO:teuthology.task.internal:no buildpackages task found 2021-02-10T10:51:53.663 INFO:teuthology.run_tasks:Running task internal.save_config... 2021-02-10T10:51:53.676 INFO:teuthology.task.internal:Saving configuration 2021-02-10T10:51:53.692 INFO:teuthology.run_tasks:Running task internal.check_lock... 2021-02-10T10:51:53.702 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-02-10T10:51:53.720 DEBUG:teuthology.task.internal.check_lock:machine status is {'is_vm': False, 'locked': True, 'locked_since': '2021-02-10 10:46:43.448891', 'description': '/home/teuthworker/archive/khiremat-2021-02-10_10:13:45-fs-wip-khiremat-mgr-authid-bp-7-nautilus-distro-basic-gibba/5872718', 'locked_by': 'scheduled_khiremat@teuthology', 'up': True, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJWDlVoR3dmWMhBhwhNwB/+TwXGwd6SgkbigjBP0SzZvR3rIeW4s7fnWccz6M8kbrSjeqWu+yrgqr0oYXJd0oq8=', 'os_version': '7.9', 'machine_type': 'gibba', 'vm_host': None, 'os_type': 'rhel', 'arch': 'x86_64', 'mac_address': None, 'name': 'gibba002.front.sepia.ceph.com'} 2021-02-10T10:51:53.739 DEBUG:teuthology.task.internal.check_lock:machine status is {'is_vm': False, 'locked': True, 'locked_since': '2021-02-10 10:46:43.448873', 'description': '/home/teuthworker/archive/khiremat-2021-02-10_10:13:45-fs-wip-khiremat-mgr-authid-bp-7-nautilus-distro-basic-gibba/5872718', 'locked_by': 'scheduled_khiremat@teuthology', 'up': True, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCho6hXyinymxuKBt1zNoDgZUDW9w6JYyYkrOM03caWhlht351oRxSEehLQw5YmnpxtHrtmZgOjryNCnYoXikjI=', 'os_version': '7.9', 'machine_type': 'gibba', 'vm_host': None, 'os_type': 'rhel', 'arch': 'x86_64', 'mac_address': None, 'name': 'gibba013.front.sepia.ceph.com'} 2021-02-10T10:51:53.739 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2021-02-10T10:51:53.750 INFO:teuthology.task.internal:roles: ubuntu@gibba002.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.1'] 2021-02-10T10:51:53.750 INFO:teuthology.task.internal:roles: ubuntu@gibba013.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.2', 'client.3'] 2021-02-10T10:51:53.750 INFO:teuthology.run_tasks:Running task console_log... 2021-02-10T10:51:53.818 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fa9b9831620>, signals=[15]) 2021-02-10T10:51:53.819 INFO:teuthology.run_tasks:Running task internal.connect... 2021-02-10T10:51:53.829 INFO:teuthology.task.internal:Opening connections... 2021-02-10T10:51:53.829 DEBUG:teuthology.task.internal:connecting to ubuntu@gibba002.front.sepia.ceph.com 2021-02-10T10:51:53.831 DEBUG:teuthology.orchestra.connection:{'hostname': 'gibba002.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-02-10T10:51:53.896 DEBUG:teuthology.task.internal:connecting to ubuntu@gibba013.front.sepia.ceph.com 2021-02-10T10:51:53.898 DEBUG:teuthology.orchestra.connection:{'hostname': 'gibba013.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-02-10T10:51:53.963 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2021-02-10T10:51:53.973 DEBUG:teuthology.orchestra.run.gibba002:> uname -m 2021-02-10T10:51:54.351 INFO:teuthology.orchestra.run.gibba002.stdout:x86_64 2021-02-10T10:51:54.352 DEBUG:teuthology.orchestra.run.gibba002:> cat /etc/os-release 2021-02-10T10:51:54.685 INFO:teuthology.orchestra.run.gibba002.stdout:NAME="Red Hat Enterprise Linux Server" 2021-02-10T10:51:54.685 INFO:teuthology.orchestra.run.gibba002.stdout:VERSION="7.9 (Maipo)" 2021-02-10T10:51:54.685 INFO:teuthology.orchestra.run.gibba002.stdout:ID="rhel" 2021-02-10T10:51:54.685 INFO:teuthology.orchestra.run.gibba002.stdout:ID_LIKE="fedora" 2021-02-10T10:51:54.685 INFO:teuthology.orchestra.run.gibba002.stdout:VARIANT="Server" 2021-02-10T10:51:54.685 INFO:teuthology.orchestra.run.gibba002.stdout:VARIANT_ID="server" 2021-02-10T10:51:54.685 INFO:teuthology.orchestra.run.gibba002.stdout:VERSION_ID="7.9" 2021-02-10T10:51:54.685 INFO:teuthology.orchestra.run.gibba002.stdout:PRETTY_NAME="Employee SKU" 2021-02-10T10:51:54.685 INFO:teuthology.orchestra.run.gibba002.stdout:ANSI_COLOR="0;31" 2021-02-10T10:51:54.685 INFO:teuthology.orchestra.run.gibba002.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:7.9:GA:server" 2021-02-10T10:51:54.685 INFO:teuthology.orchestra.run.gibba002.stdout:HOME_URL="https://www.redhat.com/" 2021-02-10T10:51:54.686 INFO:teuthology.orchestra.run.gibba002.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2021-02-10T10:51:54.686 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T10:51:54.686 INFO:teuthology.orchestra.run.gibba002.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 7" 2021-02-10T10:51:54.686 INFO:teuthology.orchestra.run.gibba002.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=7.9 2021-02-10T10:51:54.686 INFO:teuthology.orchestra.run.gibba002.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2021-02-10T10:51:54.686 INFO:teuthology.orchestra.run.gibba002.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="7.9" 2021-02-10T10:51:54.687 INFO:teuthology.lock.ops:Updating gibba002.front.sepia.ceph.com on lock server 2021-02-10T10:51:54.702 DEBUG:teuthology.orchestra.run.gibba013:> uname -m 2021-02-10T10:51:55.052 INFO:teuthology.orchestra.run.gibba013.stdout:x86_64 2021-02-10T10:51:55.053 DEBUG:teuthology.orchestra.run.gibba013:> cat /etc/os-release 2021-02-10T10:51:55.389 INFO:teuthology.orchestra.run.gibba013.stdout:NAME="Red Hat Enterprise Linux Server" 2021-02-10T10:51:55.389 INFO:teuthology.orchestra.run.gibba013.stdout:VERSION="7.9 (Maipo)" 2021-02-10T10:51:55.389 INFO:teuthology.orchestra.run.gibba013.stdout:ID="rhel" 2021-02-10T10:51:55.389 INFO:teuthology.orchestra.run.gibba013.stdout:ID_LIKE="fedora" 2021-02-10T10:51:55.389 INFO:teuthology.orchestra.run.gibba013.stdout:VARIANT="Server" 2021-02-10T10:51:55.390 INFO:teuthology.orchestra.run.gibba013.stdout:VARIANT_ID="server" 2021-02-10T10:51:55.390 INFO:teuthology.orchestra.run.gibba013.stdout:VERSION_ID="7.9" 2021-02-10T10:51:55.390 INFO:teuthology.orchestra.run.gibba013.stdout:PRETTY_NAME="Employee SKU" 2021-02-10T10:51:55.390 INFO:teuthology.orchestra.run.gibba013.stdout:ANSI_COLOR="0;31" 2021-02-10T10:51:55.390 INFO:teuthology.orchestra.run.gibba013.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:7.9:GA:server" 2021-02-10T10:51:55.390 INFO:teuthology.orchestra.run.gibba013.stdout:HOME_URL="https://www.redhat.com/" 2021-02-10T10:51:55.390 INFO:teuthology.orchestra.run.gibba013.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2021-02-10T10:51:55.390 INFO:teuthology.orchestra.run.gibba013.stdout: 2021-02-10T10:51:55.390 INFO:teuthology.orchestra.run.gibba013.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 7" 2021-02-10T10:51:55.390 INFO:teuthology.orchestra.run.gibba013.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=7.9 2021-02-10T10:51:55.390 INFO:teuthology.orchestra.run.gibba013.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2021-02-10T10:51:55.390 INFO:teuthology.orchestra.run.gibba013.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="7.9" 2021-02-10T10:51:55.391 INFO:teuthology.lock.ops:Updating gibba013.front.sepia.ceph.com on lock server 2021-02-10T10:51:55.424 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2021-02-10T10:51:55.438 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2021-02-10T10:51:55.487 INFO:teuthology.task.internal:Checking for old test directory... 2021-02-10T10:51:55.487 DEBUG:teuthology.orchestra.run.gibba002:> test '!' -e /home/ubuntu/cephtest 2021-02-10T10:51:55.500 DEBUG:teuthology.orchestra.run.gibba013:> test '!' -e /home/ubuntu/cephtest 2021-02-10T10:51:55.835 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2021-02-10T10:51:55.846 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2021-02-10T10:51:55.846 DEBUG:teuthology.orchestra.run.gibba002:> test -z $(ls -A /var/lib/ceph) 2021-02-10T10:51:55.850 DEBUG:teuthology.orchestra.run.gibba013:> test -z $(ls -A /var/lib/ceph) 2021-02-10T10:51:56.211 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2021-02-10T10:51:56.262 INFO:teuthology.run_tasks:Running task kernel... 2021-02-10T10:51:56.282 DEBUG:teuthology.task.kernel:normalized config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mgr.y': {'kdb': True, 'sha1': 'distro'}, 'mds.a': {'kdb': True, 'sha1': 'distro'}, 'osd.0': {'kdb': True, 'sha1': 'distro'}, 'osd.1': {'kdb': True, 'sha1': 'distro'}, 'osd.2': {'kdb': True, 'sha1': 'distro'}, 'osd.3': {'kdb': True, 'sha1': 'distro'}, 'client.0': {'kdb': True, 'sha1': 'distro'}, 'client.1': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}, 'mon.c': {'kdb': True, 'sha1': 'distro'}, 'mgr.x': {'kdb': True, 'sha1': 'distro'}, 'mds.b': {'kdb': True, 'sha1': 'distro'}, 'osd.4': {'kdb': True, 'sha1': 'distro'}, 'osd.5': {'kdb': True, 'sha1': 'distro'}, 'osd.6': {'kdb': True, 'sha1': 'distro'}, 'osd.7': {'kdb': True, 'sha1': 'distro'}, 'client.2': {'kdb': True, 'sha1': 'distro'}, 'client.3': {'kdb': True, 'sha1': 'distro'}} 2021-02-10T10:51:56.283 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2021-02-10T10:51:56.283 DEBUG:teuthology.orchestra.run.gibba002:> uname -r 2021-02-10T10:51:56.617 INFO:teuthology.orchestra.run.gibba002.stdout:3.10.0-1160.6.1.el7.x86_64 2021-02-10T10:51:56.617 INFO:teuthology.task.kernel:Running kernel on gibba002: 3.10.0-1160.6.1.el7.x86_64 2021-02-10T10:51:56.617 DEBUG:teuthology.orchestra.run.gibba002:> sudo yum install -y kernel 2021-02-10T10:51:57.886 INFO:teuthology.orchestra.run.gibba002.stdout:Loaded plugins: fastestmirror, langpacks, priorities, product-id, search- 2021-02-10T10:51:57.887 INFO:teuthology.orchestra.run.gibba002.stdout: : disabled-repos, subscription-manager 2021-02-10T10:51:58.020 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T10:51:58.020 INFO:teuthology.orchestra.run.gibba002.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2021-02-10T10:51:58.020 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T10:51:58.045 INFO:teuthology.orchestra.run.gibba002.stdout:Determining fastest mirrors 2021-02-10T10:51:58.218 INFO:teuthology.orchestra.run.gibba002.stdout: * epel: download-cc-rdu01.fedoraproject.org 2021-02-10T10:51:59.310 INFO:teuthology.orchestra.run.gibba002.stdout:Nothing to do 2021-02-10T10:51:59.323 DEBUG:teuthology.orchestra.run.gibba002:> echo no | sudo yum reinstall kernel || true 2021-02-10T10:51:59.780 INFO:teuthology.orchestra.run.gibba002.stdout:Loaded plugins: fastestmirror, langpacks, priorities, product-id, search- 2021-02-10T10:51:59.780 INFO:teuthology.orchestra.run.gibba002.stdout: : disabled-repos, subscription-manager 2021-02-10T10:51:59.841 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T10:51:59.842 INFO:teuthology.orchestra.run.gibba002.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2021-02-10T10:51:59.842 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T10:51:59.899 INFO:teuthology.orchestra.run.gibba002.stderr:Skipping the running kernel: kernel-3.10.0-1160.6.1.el7.x86_64 2021-02-10T10:51:59.917 INFO:teuthology.orchestra.run.gibba002.stdout:Loading mirror speeds from cached hostfile 2021-02-10T10:51:59.917 INFO:teuthology.orchestra.run.gibba002.stdout: * epel: download-cc-rdu01.fedoraproject.org 2021-02-10T10:51:59.920 INFO:teuthology.orchestra.run.gibba002.stdout:Installed package kernel-3.10.0-1160.el7.x86_64 (from anaconda) not available. 2021-02-10T10:51:59.920 INFO:teuthology.orchestra.run.gibba002.stderr:Error: Nothing to do 2021-02-10T10:51:59.933 DEBUG:teuthology.orchestra.run.gibba002:> rpm -q kernel --last | head -n 1 2021-02-10T10:52:00.381 INFO:teuthology.orchestra.run.gibba002.stdout:kernel-3.10.0-1160.6.1.el7.x86_64 Wed 11 Nov 2020 03:14:13 PM UTC 2021-02-10T10:52:00.381 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 3.10.0-1160.6.1.el7.x86_64 2021-02-10T10:52:00.382 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2021-02-10T10:52:00.382 DEBUG:teuthology.orchestra.run.gibba013:> uname -r 2021-02-10T10:52:00.717 INFO:teuthology.orchestra.run.gibba013.stdout:3.10.0-1160.6.1.el7.x86_64 2021-02-10T10:52:00.717 INFO:teuthology.task.kernel:Running kernel on gibba013: 3.10.0-1160.6.1.el7.x86_64 2021-02-10T10:52:00.717 DEBUG:teuthology.orchestra.run.gibba013:> sudo yum install -y kernel 2021-02-10T10:52:01.979 INFO:teuthology.orchestra.run.gibba013.stdout:Loaded plugins: fastestmirror, langpacks, priorities, product-id, search- 2021-02-10T10:52:01.979 INFO:teuthology.orchestra.run.gibba013.stdout: : disabled-repos, subscription-manager 2021-02-10T10:52:02.116 INFO:teuthology.orchestra.run.gibba013.stdout: 2021-02-10T10:52:02.116 INFO:teuthology.orchestra.run.gibba013.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2021-02-10T10:52:02.116 INFO:teuthology.orchestra.run.gibba013.stdout: 2021-02-10T10:52:02.143 INFO:teuthology.orchestra.run.gibba013.stdout:Determining fastest mirrors 2021-02-10T10:52:02.235 INFO:teuthology.orchestra.run.gibba013.stdout: * epel: download-cc-rdu01.fedoraproject.org 2021-02-10T10:52:03.251 INFO:teuthology.orchestra.run.gibba013.stdout:Nothing to do 2021-02-10T10:52:03.263 DEBUG:teuthology.orchestra.run.gibba013:> echo no | sudo yum reinstall kernel || true 2021-02-10T10:52:03.721 INFO:teuthology.orchestra.run.gibba013.stdout:Loaded plugins: fastestmirror, langpacks, priorities, product-id, search- 2021-02-10T10:52:03.721 INFO:teuthology.orchestra.run.gibba013.stdout: : disabled-repos, subscription-manager 2021-02-10T10:52:03.777 INFO:teuthology.orchestra.run.gibba013.stdout: 2021-02-10T10:52:03.778 INFO:teuthology.orchestra.run.gibba013.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2021-02-10T10:52:03.778 INFO:teuthology.orchestra.run.gibba013.stdout: 2021-02-10T10:52:03.840 INFO:teuthology.orchestra.run.gibba013.stderr:Skipping the running kernel: kernel-3.10.0-1160.6.1.el7.x86_64 2021-02-10T10:52:03.859 INFO:teuthology.orchestra.run.gibba013.stdout:Loading mirror speeds from cached hostfile 2021-02-10T10:52:03.859 INFO:teuthology.orchestra.run.gibba013.stdout: * epel: download-cc-rdu01.fedoraproject.org 2021-02-10T10:52:03.862 INFO:teuthology.orchestra.run.gibba013.stdout:Installed package kernel-3.10.0-1160.el7.x86_64 (from anaconda) not available. 2021-02-10T10:52:03.863 INFO:teuthology.orchestra.run.gibba013.stderr:Error: Nothing to do 2021-02-10T10:52:03.874 DEBUG:teuthology.orchestra.run.gibba013:> rpm -q kernel --last | head -n 1 2021-02-10T10:52:04.320 INFO:teuthology.orchestra.run.gibba013.stdout:kernel-3.10.0-1160.6.1.el7.x86_64 Wed 11 Nov 2020 03:14:13 PM UTC 2021-02-10T10:52:04.321 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 3.10.0-1160.6.1.el7.x86_64 2021-02-10T10:52:04.321 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2021-02-10T10:52:04.321 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2021-02-10T10:52:04.321 DEBUG:teuthology.orchestra.run.gibba002:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2021-02-10T10:52:04.678 INFO:teuthology.orchestra.run.gibba002.stdout:ttyS1 2021-02-10T10:52:04.689 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2021-02-10T10:52:04.690 DEBUG:teuthology.orchestra.run.gibba013:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2021-02-10T10:52:05.040 INFO:teuthology.orchestra.run.gibba013.stdout:ttyS1 2021-02-10T10:52:05.051 INFO:teuthology.run_tasks:Running task internal.base... 2021-02-10T10:52:05.062 INFO:teuthology.task.internal:Creating test directory... 2021-02-10T10:52:05.062 DEBUG:teuthology.orchestra.run.gibba002:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2021-02-10T10:52:05.065 DEBUG:teuthology.orchestra.run.gibba013:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2021-02-10T10:52:05.398 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2021-02-10T10:52:05.409 INFO:teuthology.run_tasks:Running task internal.archive... 2021-02-10T10:52:05.421 INFO:teuthology.task.internal:Creating archive directory... 2021-02-10T10:52:05.421 DEBUG:teuthology.orchestra.run.gibba002:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2021-02-10T10:52:05.424 DEBUG:teuthology.orchestra.run.gibba013:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2021-02-10T10:52:05.774 INFO:teuthology.run_tasks:Running task internal.coredump... 2021-02-10T10:52:05.790 INFO:teuthology.task.internal:Enabling coredump saving... 2021-02-10T10:52:05.791 DEBUG:teuthology.orchestra.run.gibba002:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2021-02-10T10:52:05.795 DEBUG:teuthology.orchestra.run.gibba013:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2021-02-10T10:52:06.135 INFO:teuthology.orchestra.run.gibba002.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-02-10T10:52:06.138 INFO:teuthology.orchestra.run.gibba013.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-02-10T10:52:06.145 INFO:teuthology.orchestra.run.gibba002.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-02-10T10:52:06.148 INFO:teuthology.orchestra.run.gibba013.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-02-10T10:52:06.149 INFO:teuthology.run_tasks:Running task internal.sudo... 2021-02-10T10:52:06.156 INFO:teuthology.task.internal:Configuring sudo... 2021-02-10T10:52:06.156 DEBUG:teuthology.orchestra.run.gibba002:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2021-02-10T10:52:06.159 DEBUG:teuthology.orchestra.run.gibba013:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2021-02-10T10:52:06.506 INFO:teuthology.run_tasks:Running task internal.syslog... 2021-02-10T10:52:06.516 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2021-02-10T10:52:06.516 DEBUG:teuthology.orchestra.run.gibba002:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2021-02-10T10:52:06.519 DEBUG:teuthology.orchestra.run.gibba013:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2021-02-10T10:52:06.857 DEBUG:teuthology.orchestra.run.gibba002:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2021-02-10T10:52:07.211 DEBUG:teuthology.orchestra.run.gibba002:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2021-02-10T10:52:07.566 DEBUG:teuthology.orchestra.run.gibba002:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2021-02-10T10:52:07.919 DEBUG:teuthology.orchestra.run.gibba002:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2021-02-10T10:52:08.262 DEBUG:teuthology.orchestra.run.gibba002:> set -ex 2021-02-10T10:52:08.262 DEBUG:teuthology.orchestra.run.gibba002:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2021-02-10T10:52:08.605 DEBUG:teuthology.orchestra.run.gibba013:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2021-02-10T10:52:08.970 DEBUG:teuthology.orchestra.run.gibba013:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2021-02-10T10:52:09.329 DEBUG:teuthology.orchestra.run.gibba013:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2021-02-10T10:52:09.688 DEBUG:teuthology.orchestra.run.gibba013:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2021-02-10T10:52:10.033 DEBUG:teuthology.orchestra.run.gibba013:> set -ex 2021-02-10T10:52:10.033 DEBUG:teuthology.orchestra.run.gibba013:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2021-02-10T10:52:10.383 DEBUG:teuthology.orchestra.run.gibba002:> sudo service rsyslog restart 2021-02-10T10:52:10.391 DEBUG:teuthology.orchestra.run.gibba013:> sudo service rsyslog restart 2021-02-10T10:52:10.769 INFO:teuthology.orchestra.run.gibba002.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2021-02-10T10:52:10.782 INFO:teuthology.orchestra.run.gibba013.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2021-02-10T10:52:10.798 INFO:teuthology.run_tasks:Running task internal.timer... 2021-02-10T10:52:10.807 INFO:teuthology.task.internal:Starting timer... 2021-02-10T10:52:10.807 INFO:teuthology.run_tasks:Running task pcp... 2021-02-10T10:52:10.826 INFO:teuthology.run_tasks:Running task selinux... 2021-02-10T10:52:10.843 DEBUG:teuthology.orchestra.run.gibba002:> sudo service auditd rotate 2021-02-10T10:52:11.210 INFO:teuthology.orchestra.run.gibba002.stdout:Rotating logs: [ OK ] 2021-02-10T10:52:11.211 DEBUG:teuthology.orchestra.run.gibba013:> sudo service auditd rotate 2021-02-10T10:52:11.576 INFO:teuthology.orchestra.run.gibba013.stdout:Rotating logs: [ OK ] 2021-02-10T10:52:11.577 DEBUG:teuthology.task.selinux:Getting current SELinux state 2021-02-10T10:52:11.578 DEBUG:teuthology.orchestra.run.gibba002:> /usr/sbin/getenforce 2021-02-10T10:52:11.919 INFO:teuthology.orchestra.run.gibba002.stdout:Permissive 2021-02-10T10:52:11.919 DEBUG:teuthology.orchestra.run.gibba013:> /usr/sbin/getenforce 2021-02-10T10:52:12.264 INFO:teuthology.orchestra.run.gibba013.stdout:Permissive 2021-02-10T10:52:12.264 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@gibba002.front.sepia.ceph.com': 'permissive', 'ubuntu@gibba013.front.sepia.ceph.com': 'permissive'} 2021-02-10T10:52:12.265 DEBUG:teuthology.orchestra.run.gibba002:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\)' 2021-02-10T10:52:12.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T10:52:12.609 DEBUG:teuthology.orchestra.run.gibba013:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\)' 2021-02-10T10:52:12.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T10:52:12.954 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2021-02-10T10:52:12.955 DEBUG:teuthology.orchestra.run.gibba002:> sudo /usr/sbin/setenforce permissive 2021-02-10T10:52:13.298 DEBUG:teuthology.orchestra.run.gibba013:> sudo /usr/sbin/setenforce permissive 2021-02-10T10:52:13.641 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2021-02-10T10:52:13.656 DEBUG:teuthology.repo_utils:Setting repo remote to git://git.ceph.com/git/ceph-cm-ansible.git 2021-02-10T10:52:13.669 INFO:teuthology.repo_utils:Fetching master from origin 2021-02-10T10:52:13.861 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master to origin/master 2021-02-10T10:52:14.046 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}] 2021-02-10T10:52:14.062 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit gibba002.front.sepia.ceph.com,gibba013.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/cephlab.yml 2021-02-10T10:52:14.859 INFO:teuthology.task.ansible.out:Using /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/ansible.cfg as config file 2021-02-10T10:52:16.927 INFO:teuthology.task.ansible.out:[WARNING]: file /home/teuthworker/src/git.ceph.com_git_ceph-cm- ansible_master/roles/testnode/tasks/setup-opensuse.yml is empty and had no tasks to include 2021-02-10T10:52:19.806 INFO:teuthology.task.ansible.out: PLAY [all] ********************************************************************* 2021-02-10T10:52:19.845 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:52:19 +0000 (0:00:00.146) 0:00:00.146 **** 2021-02-10T10:52:19.863 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:52:19 +0000 (0:00:00.018) 0:00:00.165 **** 2021-02-10T10:52:23.566 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [gibba002.front.sepia.ceph.com] 2021-02-10T10:52:23.637 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:52:23 +0000 (0:00:03.773) 0:00:03.938 **** 2021-02-10T10:52:23.679 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] 2021-02-10T10:52:23.680 INFO:teuthology.task.ansible.out: 2021-02-10T10:52:23.761 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:52:23 +0000 (0:00:00.124) 0:00:04.063 **** 2021-02-10T10:52:24.247 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the sudo group.] ******************************** 2021-02-10T10:52:24.248 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} ok: [gibba013.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} 2021-02-10T10:52:24.280 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:52:24 +0000 (0:00:00.519) 0:00:04.582 **** 2021-02-10T10:52:24.312 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:52:24 +0000 (0:00:00.030) 0:00:04.613 **** 2021-02-10T10:52:24.976 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the ansible user.] ****************************** 2021-02-10T10:52:24.978 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 1100, "groups": "sudo", "home": "/home/cm", "move_home": false, "name": "cm", "shell": "/bin/bash", "state": "present", "uid": 1100} ok: [gibba002.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 1100, "groups": "sudo", "home": "/home/cm", "move_home": false, "name": "cm", "shell": "/bin/bash", "state": "present", "uid": 1100} 2021-02-10T10:52:25.012 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:52:25 +0000 (0:00:00.700) 0:00:05.314 **** 2021-02-10T10:52:25.043 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:52:25 +0000 (0:00:00.029) 0:00:05.344 **** 2021-02-10T10:52:25.048 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Delete the ansible users password.] ******************** skipping: [gibba002.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:52:25.088 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:52:25 +0000 (0:00:00.045) 0:00:05.389 **** 2021-02-10T10:52:25.100 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:52:25.149 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:52:25 +0000 (0:00:00.061) 0:00:05.451 **** 2021-02-10T10:52:25.675 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Ensure includedir is present in sudoers.] ************** 2021-02-10T10:52:25.677 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-02-10T10:52:25.678 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-02-10T10:52:25.718 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:52:25 +0000 (0:00:00.567) 0:00:06.019 **** 2021-02-10T10:52:25.749 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:52:25 +0000 (0:00:00.030) 0:00:06.049 **** 2021-02-10T10:52:28.056 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the cephlab_sudo sudoers.d file.] *************** 2021-02-10T10:52:28.058 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => {"changed": false, "checksum": "6c8e8b7ecc895eb2094cafe75c34b58f183442a7", "dest": "/etc/sudoers.d/cephlab_sudo", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/cephlab_sudo", "secontext": "system_u:object_r:etc_runtime_t:s0", "size": 182, "state": "file", "uid": 0} ok: [gibba013.front.sepia.ceph.com] => {"changed": false, "checksum": "6c8e8b7ecc895eb2094cafe75c34b58f183442a7", "dest": "/etc/sudoers.d/cephlab_sudo", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/cephlab_sudo", "secontext": "system_u:object_r:etc_runtime_t:s0", "size": 182, "state": "file", "uid": 0} 2021-02-10T10:52:28.091 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:52:28 +0000 (0:00:02.343) 0:00:08.392 **** 2021-02-10T10:52:28.122 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:52:28 +0000 (0:00:00.031) 0:00:08.424 **** 2021-02-10T10:52:28.724 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Add authorized keys for the ansible user.] ************* 2021-02-10T10:52:28.726 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => {"changed": false, "comment": null, "exclusive": true, "follow": false, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4 2021-02-10T10:52:28.727 INFO:teuthology.task.ansible.out:r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} ok: [gibba013.front.sepia.ceph.com] => {"changed": false, "comment": null, "exclusive": true, "follow": false, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.c 2021-02-10T10:52:28.729 INFO:teuthology.task.ansible.out:om\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@ak 2021-02-10T10:52:28.730 INFO:teuthology.task.ansible.out:lap", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2021-02-10T10:52:28.856 INFO:teuthology.task.ansible.out: PLAY [teuthology] ************************************************************** skipping: no hosts matched 2021-02-10T10:52:28.864 INFO:teuthology.task.ansible.out: PLAY [testnodes] *************************************************************** 2021-02-10T10:52:29.012 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:52:29 +0000 (0:00:00.889) 0:00:09.314 **** 2021-02-10T10:52:29.045 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:52:29 +0000 (0:00:00.032) 0:00:09.346 **** 2021-02-10T10:52:30.088 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [gibba002.front.sepia.ceph.com] 2021-02-10T10:52:30.131 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] 2021-02-10T10:52:30.176 INFO:teuthology.task.ansible.out:[WARNING]: Using run_once with the free strategy is not currently supported. This task will still be executed for every host in the inventory list. 2021-02-10T10:52:30.177 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:52:30 +0000 (0:00:01.132) 0:00:10.478 **** 2021-02-10T10:52:30.251 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:52:30 +0000 (0:00:00.074) 0:00:10.553 **** 2021-02-10T10:52:30.634 INFO:teuthology.task.ansible.out: TASK [users : Check keys_repo HEAD sha1] *************************************** 2021-02-10T10:52:30.635 INFO:teuthology.task.ansible.out:changed: [gibba013.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "git ls-remote https://github.com/ceph/keys HEAD | awk '{ print $1 }'", "delta": "0:00:00.154114", "end": "2021-02-10 10:52:30.614150", "rc": 0, "start": "2021-02-10 10:52:30.460036", "stderr": "", "stderr_lines": [], "stdout": "df8946f08e870e2d2565b79099289ca48bd825f3", "stdout_lines": ["df8946f08e870e2d2565b79099289ca48bd825f3"]} 2021-02-10T10:52:30.683 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:52:30 +0000 (0:00:00.431) 0:00:10.984 **** 2021-02-10T10:52:30.689 INFO:teuthology.task.ansible.out:changed: [gibba002.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "git ls-remote https://github.com/ceph/keys HEAD | awk '{ print $1 }'", "delta": "0:00:00.164878", "end": "2021-02-10 10:52:30.624645", "rc": 0, "start": "2021-02-10 10:52:30.459767", "stderr": "", "stderr_lines": [], "stdout": "df8946f08e870e2d2565b79099289ca48bd825f3", "stdout_lines": ["df8946f08e870e2d2565b79099289ca48bd825f3"]} 2021-02-10T10:52:30.724 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:52:30 +0000 (0:00:00.041) 0:00:11.026 **** 2021-02-10T10:52:31.126 INFO:teuthology.task.ansible.out: TASK [users : Check host's /keys-repo-sha1 sentinel file] ********************** 2021-02-10T10:52:31.127 INFO:teuthology.task.ansible.out:changed: [gibba013.front.sepia.ceph.com] => {"changed": true, "cmd": ["cat", "/keys-repo-sha1"], "delta": "0:00:00.004493", "end": "2021-02-10 10:52:31.108209", "failed_when_result": false, "rc": 0, "start": "2021-02-10 10:52:31.103716", "stderr": "", "stderr_lines": [], "stdout": "46fd2a03b54c2ce7520f33746fa0586027b558c3", "stdout_lines": ["46fd2a03b54c2ce7520f33746fa0586027b558c3"]} 2021-02-10T10:52:31.163 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:52:31 +0000 (0:00:00.438) 0:00:11.464 **** 2021-02-10T10:52:31.169 INFO:teuthology.task.ansible.out:changed: [gibba002.front.sepia.ceph.com] => {"changed": true, "cmd": ["cat", "/keys-repo-sha1"], "delta": "0:00:00.004382", "end": "2021-02-10 10:52:31.149359", "failed_when_result": false, "rc": 0, "start": "2021-02-10 10:52:31.144977", "stderr": "", "stderr_lines": [], "stdout": "46fd2a03b54c2ce7520f33746fa0586027b558c3", "stdout_lines": ["46fd2a03b54c2ce7520f33746fa0586027b558c3"]} 2021-02-10T10:52:31.205 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:52:31 +0000 (0:00:00.041) 0:00:11.506 **** 2021-02-10T10:52:31.211 INFO:teuthology.task.ansible.out: TASK [users : Determine if we can skip users and pubkeys updates] ************** 2021-02-10T10:52:31.212 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:52:31.246 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:52:31 +0000 (0:00:00.041) 0:00:11.548 **** 2021-02-10T10:52:31.252 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:52:31.287 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:52:31 +0000 (0:00:00.040) 0:00:11.588 **** 2021-02-10T10:52:31.292 INFO:teuthology.task.ansible.out: TASK [users : Merge extra_admin_users into managed_admin_users] **************** 2021-02-10T10:52:31.294 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:52:31.326 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:52:31 +0000 (0:00:00.039) 0:00:11.628 **** 2021-02-10T10:52:31.333 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:52:31.366 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:52:31 +0000 (0:00:00.040) 0:00:11.668 **** 2021-02-10T10:52:31.372 INFO:teuthology.task.ansible.out: TASK [users : Remove managed_admin_users from managed_users] ******************* 2021-02-10T10:52:31.373 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:52:31.408 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:52:31 +0000 (0:00:00.041) 0:00:11.710 **** 2021-02-10T10:52:31.414 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:52:31.448 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:52:31 +0000 (0:00:00.040) 0:00:11.750 **** 2021-02-10T10:52:31.453 INFO:teuthology.task.ansible.out: TASK [users : Filter the managed_users list] *********************************** 2021-02-10T10:52:31.455 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:52:31.488 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:52:31 +0000 (0:00:00.039) 0:00:11.790 **** 2021-02-10T10:52:31.495 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:52:31.529 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:52:31 +0000 (0:00:00.041) 0:00:11.831 **** 2021-02-10T10:52:31.535 INFO:teuthology.task.ansible.out: TASK [users : Filter the managed_admin_users list] ***************************** 2021-02-10T10:52:31.537 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:52:31.570 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:52:31 +0000 (0:00:00.040) 0:00:11.872 **** 2021-02-10T10:52:31.576 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:52:31.611 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:52:31 +0000 (0:00:00.040) 0:00:11.913 **** 2021-02-10T10:52:32.023 INFO:teuthology.task.ansible.out: TASK [users : Set UID_MIN to 1001] ********************************************* 2021-02-10T10:52:32.025 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-02-10T10:52:32.060 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:52:32 +0000 (0:00:00.448) 0:00:12.361 **** 2021-02-10T10:52:32.067 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-02-10T10:52:32.101 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:52:32 +0000 (0:00:00.041) 0:00:12.403 **** 2021-02-10T10:52:32.543 INFO:teuthology.task.ansible.out: TASK [users : Create all admin users with sudo access.] ************************ 2021-02-10T10:52:32.545 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1101, "groups": "sudo", "home": "/home/andrewschoen", "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "move_home": false, "name": "andrewschoen", "shell": "/bin/bash", "state": "present", "uid": 1101} 2021-02-10T10:52:32.597 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1101, "groups": "sudo", "home": "/home/andrewschoen", "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "move_home": false, "name": "andrewschoen", "shell": "/bin/bash", "state": "present", "uid": 1101} 2021-02-10T10:52:32.982 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1102, "groups": "sudo", "home": "/home/zack", "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "move_home": false, "name": "zack", "shell": "/bin/bash", "state": "present", "uid": 1102} 2021-02-10T10:52:33.024 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1102, "groups": "sudo", "home": "/home/zack", "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "move_home": false, "name": "zack", "shell": "/bin/bash", "state": "present", "uid": 1102} 2021-02-10T10:52:33.026 INFO:teuthology.task.ansible.out: 2021-02-10T10:52:33.424 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1103, "groups": "sudo", "home": "/home/kdreyer", "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "move_home": false, "name": "kdreyer", "shell": "/bin/bash", "state": "present", "uid": 1103} 2021-02-10T10:52:33.466 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1103, "groups": "sudo", "home": "/home/kdreyer", "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "move_home": false, "name": "kdreyer", "shell": "/bin/bash", "state": "present", "uid": 1103} 2021-02-10T10:52:33.862 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1104, "groups": "sudo", "home": "/home/dmick", "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "move_home": false, "name": "dmick", "shell": "/bin/bash", "state": "present", "uid": 1104} 2021-02-10T10:52:33.906 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1104, "groups": "sudo", "home": "/home/dmick", "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "move_home": false, "name": "dmick", "shell": "/bin/bash", "state": "present", "uid": 1104} 2021-02-10T10:52:34.292 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1105, "groups": "sudo", "home": "/home/sage", "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "move_home": false, "name": "sage", "shell": "/bin/bash", "state": "present", "uid": 1105} 2021-02-10T10:52:34.345 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1105, "groups": "sudo", "home": "/home/sage", "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "move_home": false, "name": "sage", "shell": "/bin/bash", "state": "present", "uid": 1105} 2021-02-10T10:52:34.741 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1106, "groups": "sudo", "home": "/home/dgalloway", "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "move_home": false, "name": "dgalloway", "shell": "/bin/bash", "state": "present", "uid": 1106} 2021-02-10T10:52:34.783 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1106, "groups": "sudo", "home": "/home/dgalloway", "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "move_home": false, "name": "dgalloway", "shell": "/bin/bash", "state": "present", "uid": 1106} 2021-02-10T10:52:35.169 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1107, "groups": "sudo", "home": "/home/akraitma", "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "move_home": false, "name": "akraitma", "shell": "/bin/bash", "state": "present", "uid": 1107} 2021-02-10T10:52:35.223 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1107, "groups": "sudo", "home": "/home/akraitma", "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "move_home": false, "name": "akraitma", "shell": "/bin/bash", "state": "present", "uid": 1107} 2021-02-10T10:52:35.618 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1108, "groups": "sudo", "home": "/home/jcaratzas", "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "move_home": false, "name": "jcaratzas", "shell": "/bin/bash", "state": "present", "uid": 1108} 2021-02-10T10:52:35.661 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1108, "groups": "sudo", "home": "/home/jcaratzas", "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "move_home": false, "name": "jcaratzas", "shell": "/bin/bash", "state": "present", "uid": 1108} 2021-02-10T10:52:36.057 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1109, "groups": "sudo", "home": "/home/trociny", "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "move_home": false, "name": "trociny", "shell": "/bin/bash", "state": "present", "uid": 1109} 2021-02-10T10:52:36.100 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1109, "groups": "sudo", "home": "/home/trociny", "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "move_home": false, "name": "trociny", "shell": "/bin/bash", "state": "present", "uid": 1109} 2021-02-10T10:52:36.497 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1110, "groups": "sudo", "home": "/home/smithfarm", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "move_home": false, "name": "smithfarm", "shell": "/bin/bash", "state": "present", "uid": 1110} 2021-02-10T10:52:36.540 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1110, "groups": "sudo", "home": "/home/smithfarm", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "move_home": false, "name": "smithfarm", "shell": "/bin/bash", "state": "present", "uid": 1110} 2021-02-10T10:52:36.940 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1111, "groups": "sudo", "home": "/home/mbenjamin", "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "move_home": false, "name": "mbenjamin", "shell": "/bin/bash", "state": "present", "uid": 1111} 2021-02-10T10:52:36.994 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1111, "groups": "sudo", "home": "/home/mbenjamin", "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "move_home": false, "name": "mbenjamin", "shell": "/bin/bash", "state": "present", "uid": 1111} 2021-02-10T10:52:37.374 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1112, "groups": "sudo", "home": "/home/aemerson", "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "move_home": false, "name": "aemerson", "shell": "/bin/bash", "state": "present", "uid": 1112} 2021-02-10T10:52:37.432 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1112, "groups": "sudo", "home": "/home/aemerson", "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "move_home": false, "name": "aemerson", "shell": "/bin/bash", "state": "present", "uid": 1112} 2021-02-10T10:52:37.826 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1113, "groups": "sudo", "home": "/home/amaredia", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "move_home": false, "name": "amaredia", "shell": "/bin/bash", "state": "present", "uid": 1113} 2021-02-10T10:52:37.879 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1113, "groups": "sudo", "home": "/home/amaredia", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "move_home": false, "name": "amaredia", "shell": "/bin/bash", "state": "present", "uid": 1113} 2021-02-10T10:52:38.266 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1114, "groups": "sudo", "home": "/home/tserlin", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "move_home": false, "name": "tserlin", "shell": "/bin/bash", "state": "present", "uid": 1114} 2021-02-10T10:52:38.309 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1114, "groups": "sudo", "home": "/home/tserlin", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "move_home": false, "name": "tserlin", "shell": "/bin/bash", "state": "present", "uid": 1114} 2021-02-10T10:52:38.697 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1115, "groups": "sudo", "home": "/home/dis", "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "move_home": false, "name": "dis", "shell": "/bin/bash", "state": "present", "uid": 1115} 2021-02-10T10:52:38.751 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1115, "groups": "sudo", "home": "/home/dis", "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "move_home": false, "name": "dis", "shell": "/bin/bash", "state": "present", "uid": 1115} 2021-02-10T10:52:39.139 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1116, "groups": "sudo", "home": "/home/gregf", "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "move_home": false, "name": "gregf", "shell": "/bin/bash", "state": "present", "uid": 1116} 2021-02-10T10:52:39.192 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1116, "groups": "sudo", "home": "/home/gregf", "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "move_home": false, "name": "gregf", "shell": "/bin/bash", "state": "present", "uid": 1116} 2021-02-10T10:52:39.585 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1117, "groups": "sudo", "home": "/home/joshd", "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "move_home": false, "name": "joshd", "shell": "/bin/bash", "state": "present", "uid": 1117} 2021-02-10T10:52:39.628 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1117, "groups": "sudo", "home": "/home/joshd", "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "move_home": false, "name": "joshd", "shell": "/bin/bash", "state": "present", "uid": 1117} 2021-02-10T10:52:40.026 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1118, "groups": "sudo", "home": "/home/davidz", "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "move_home": false, "name": "davidz", "shell": "/bin/bash", "state": "present", "uid": 1118} 2021-02-10T10:52:40.069 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1118, "groups": "sudo", "home": "/home/davidz", "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "move_home": false, "name": "davidz", "shell": "/bin/bash", "state": "present", "uid": 1118} 2021-02-10T10:52:40.467 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1119, "groups": "sudo", "home": "/home/cmeno", "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "move_home": false, "name": "cmeno", "shell": "/bin/bash", "state": "present", "uid": 1119} 2021-02-10T10:52:40.520 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1119, "groups": "sudo", "home": "/home/cmeno", "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "move_home": false, "name": "cmeno", "shell": "/bin/bash", "state": "present", "uid": 1119} 2021-02-10T10:52:40.907 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1120, "groups": "sudo", "home": "/home/ivancich", "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "move_home": false, "name": "ivancich", "shell": "/bin/bash", "state": "present", "uid": 1120} 2021-02-10T10:52:40.961 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1120, "groups": "sudo", "home": "/home/ivancich", "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "move_home": false, "name": "ivancich", "shell": "/bin/bash", "state": "present", "uid": 1120} 2021-02-10T10:52:41.346 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1121, "groups": "sudo", "home": "/home/wusui", "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "move_home": false, "name": "wusui", "shell": "/bin/bash", "state": "present", "uid": 1121} 2021-02-10T10:52:41.399 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1121, "groups": "sudo", "home": "/home/wusui", "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "move_home": false, "name": "wusui", "shell": "/bin/bash", "state": "present", "uid": 1121} 2021-02-10T10:52:41.796 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1123, "groups": "sudo", "home": "/home/yuriw", "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "move_home": false, "name": "yuriw", "shell": "/bin/bash", "state": "present", "uid": 1123} 2021-02-10T10:52:41.850 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1123, "groups": "sudo", "home": "/home/yuriw", "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "move_home": false, "name": "yuriw", "shell": "/bin/bash", "state": "present", "uid": 1123} 2021-02-10T10:52:42.242 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1124, "groups": "sudo", "home": "/home/tamil", "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "move_home": false, "name": "tamil", "shell": "/bin/bash", "state": "present", "uid": 1124} 2021-02-10T10:52:42.297 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1124, "groups": "sudo", "home": "/home/tamil", "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "move_home": false, "name": "tamil", "shell": "/bin/bash", "state": "present", "uid": 1124} 2021-02-10T10:52:42.685 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1125, "groups": "sudo", "home": "/home/jowilkin", "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "move_home": false, "name": "jowilkin", "shell": "/bin/bash", "state": "present", "uid": 1125} 2021-02-10T10:52:42.742 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1125, "groups": "sudo", "home": "/home/jowilkin", "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "move_home": false, "name": "jowilkin", "shell": "/bin/bash", "state": "present", "uid": 1125} 2021-02-10T10:52:43.147 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1126, "groups": "sudo", "home": "/home/bhubbard", "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "move_home": false, "name": "bhubbard", "shell": "/bin/bash", "state": "present", "uid": 1126} 2021-02-10T10:52:43.190 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1126, "groups": "sudo", "home": "/home/bhubbard", "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "move_home": false, "name": "bhubbard", "shell": "/bin/bash", "state": "present", "uid": 1126} 2021-02-10T10:52:43.591 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1127, "groups": "sudo", "home": "/home/yehudasa", "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "move_home": false, "name": "yehudasa", "shell": "/bin/bash", "state": "present", "uid": 1127} 2021-02-10T10:52:43.645 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1127, "groups": "sudo", "home": "/home/yehudasa", "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "move_home": false, "name": "yehudasa", "shell": "/bin/bash", "state": "present", "uid": 1127} 2021-02-10T10:52:44.035 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1128, "groups": "sudo", "home": "/home/dang", "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "move_home": false, "name": "dang", "shell": "/bin/bash", "state": "present", "uid": 1128} 2021-02-10T10:52:44.082 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1128, "groups": "sudo", "home": "/home/dang", "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "move_home": false, "name": "dang", "shell": "/bin/bash", "state": "present", "uid": 1128} 2021-02-10T10:52:44.491 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1129, "groups": "sudo", "home": "/home/branto", "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "move_home": false, "name": "branto", "shell": "/bin/bash", "state": "present", "uid": 1129} 2021-02-10T10:52:44.523 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1129, "groups": "sudo", "home": "/home/branto", "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "move_home": false, "name": "branto", "shell": "/bin/bash", "state": "present", "uid": 1129} 2021-02-10T10:52:44.924 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1130, "groups": "sudo", "home": "/home/xiaoxichen", "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "move_home": false, "name": "xiaoxichen", "shell": "/bin/bash", "state": "present", "uid": 1130} 2021-02-10T10:52:44.967 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1130, "groups": "sudo", "home": "/home/xiaoxichen", "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "move_home": false, "name": "xiaoxichen", "shell": "/bin/bash", "state": "present", "uid": 1130} 2021-02-10T10:52:45.366 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1131, "groups": "sudo", "home": "/home/ffilz", "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "move_home": false, "name": "ffilz", "shell": "/bin/bash", "state": "present", "uid": 1131} 2021-02-10T10:52:45.409 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1131, "groups": "sudo", "home": "/home/ffilz", "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "move_home": false, "name": "ffilz", "shell": "/bin/bash", "state": "present", "uid": 1131} 2021-02-10T10:52:45.807 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1132, "groups": "sudo", "home": "/home/joao", "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "move_home": false, "name": "joao", "shell": "/bin/bash", "state": "present", "uid": 1132} 2021-02-10T10:52:45.850 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1132, "groups": "sudo", "home": "/home/joao", "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "move_home": false, "name": "joao", "shell": "/bin/bash", "state": "present", "uid": 1132} 2021-02-10T10:52:46.249 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1133, "groups": "sudo", "home": "/home/nhm", "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "move_home": false, "name": "nhm", "shell": "/bin/bash", "state": "present", "uid": 1133} 2021-02-10T10:52:46.282 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1133, "groups": "sudo", "home": "/home/nhm", "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "move_home": false, "name": "nhm", "shell": "/bin/bash", "state": "present", "uid": 1133} 2021-02-10T10:52:46.283 INFO:teuthology.task.ansible.out: 2021-02-10T10:52:46.695 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1134, "groups": "sudo", "home": "/home/nwatkins", "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "move_home": false, "name": "nwatkins", "shell": "/bin/bash", "state": "present", "uid": 1134} 2021-02-10T10:52:46.728 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1134, "groups": "sudo", "home": "/home/nwatkins", "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "move_home": false, "name": "nwatkins", "shell": "/bin/bash", "state": "present", "uid": 1134} 2021-02-10T10:52:47.128 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1135, "groups": "sudo", "home": "/home/mkidd", "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "move_home": false, "name": "mkidd", "shell": "/bin/bash", "state": "present", "uid": 1135} 2021-02-10T10:52:47.160 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1135, "groups": "sudo", "home": "/home/mkidd", "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "move_home": false, "name": "mkidd", "shell": "/bin/bash", "state": "present", "uid": 1135} 2021-02-10T10:52:47.573 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1136, "groups": "sudo", "home": "/home/jlopez", "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "move_home": false, "name": "jlopez", "shell": "/bin/bash", "state": "present", "uid": 1136} 2021-02-10T10:52:47.607 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1136, "groups": "sudo", "home": "/home/jlopez", "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "move_home": false, "name": "jlopez", "shell": "/bin/bash", "state": "present", "uid": 1136} 2021-02-10T10:52:48.020 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1137, "groups": "sudo", "home": "/home/haomaiwang", "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "move_home": false, "name": "haomaiwang", "shell": "/bin/bash", "state": "present", "uid": 1137} 2021-02-10T10:52:48.043 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1137, "groups": "sudo", "home": "/home/haomaiwang", "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "move_home": false, "name": "haomaiwang", "shell": "/bin/bash", "state": "present", "uid": 1137} 2021-02-10T10:52:48.457 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1138, "groups": "sudo", "home": "/home/jdillaman", "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "move_home": false, "name": "jdillaman", "shell": "/bin/bash", "state": "present", "uid": 1138} 2021-02-10T10:52:48.490 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1138, "groups": "sudo", "home": "/home/jdillaman", "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "move_home": false, "name": "jdillaman", "shell": "/bin/bash", "state": "present", "uid": 1138} 2021-02-10T10:52:48.901 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1139, "groups": "sudo", "home": "/home/kchai", "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "move_home": false, "name": "kchai", "shell": "/bin/bash", "state": "present", "uid": 1139} 2021-02-10T10:52:48.934 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1139, "groups": "sudo", "home": "/home/kchai", "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "move_home": false, "name": "kchai", "shell": "/bin/bash", "state": "present", "uid": 1139} 2021-02-10T10:52:49.348 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1140} 2021-02-10T10:52:49.371 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1140} 2021-02-10T10:52:49.794 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1141, "groups": "sudo", "home": "/home/dfuller", "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "move_home": false, "name": "dfuller", "shell": "/bin/bash", "state": "present", "uid": 1141} 2021-02-10T10:52:49.828 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1141, "groups": "sudo", "home": "/home/dfuller", "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "move_home": false, "name": "dfuller", "shell": "/bin/bash", "state": "present", "uid": 1141} 2021-02-10T10:52:50.238 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1142, "groups": "sudo", "home": "/home/owasserm", "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "move_home": false, "name": "owasserm", "shell": "/bin/bash", "state": "present", "uid": 1142} 2021-02-10T10:52:50.270 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1142, "groups": "sudo", "home": "/home/owasserm", "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "move_home": false, "name": "owasserm", "shell": "/bin/bash", "state": "present", "uid": 1142} 2021-02-10T10:52:50.682 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1143, "groups": "sudo", "home": "/home/vasu", "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "move_home": false, "name": "vasu", "shell": "/bin/bash", "state": "present", "uid": 1143} 2021-02-10T10:52:50.714 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1143, "groups": "sudo", "home": "/home/vasu", "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "move_home": false, "name": "vasu", "shell": "/bin/bash", "state": "present", "uid": 1143} 2021-02-10T10:52:51.123 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1144, "groups": "sudo", "home": "/home/smohan", "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "move_home": false, "name": "smohan", "shell": "/bin/bash", "state": "present", "uid": 1144} 2021-02-10T10:52:51.145 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1144, "groups": "sudo", "home": "/home/smohan", "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "move_home": false, "name": "smohan", "shell": "/bin/bash", "state": "present", "uid": 1144} 2021-02-10T10:52:51.556 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1145, "groups": "sudo", "home": "/home/abhi", "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "move_home": false, "name": "abhi", "shell": "/bin/bash", "state": "present", "uid": 1145} 2021-02-10T10:52:51.589 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1145, "groups": "sudo", "home": "/home/abhi", "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "move_home": false, "name": "abhi", "shell": "/bin/bash", "state": "present", "uid": 1145} 2021-02-10T10:52:51.997 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1146, "groups": "sudo", "home": "/home/cbodley", "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "move_home": false, "name": "cbodley", "shell": "/bin/bash", "state": "present", "uid": 1146} 2021-02-10T10:52:52.029 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1146, "groups": "sudo", "home": "/home/cbodley", "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "move_home": false, "name": "cbodley", "shell": "/bin/bash", "state": "present", "uid": 1146} 2021-02-10T10:52:52.436 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1147, "groups": "sudo", "home": "/home/mwatts", "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "move_home": false, "name": "mwatts", "shell": "/bin/bash", "state": "present", "uid": 1147} 2021-02-10T10:52:52.459 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1147, "groups": "sudo", "home": "/home/mwatts", "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "move_home": false, "name": "mwatts", "shell": "/bin/bash", "state": "present", "uid": 1147} 2021-02-10T10:52:52.875 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1148} 2021-02-10T10:52:52.897 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1148} 2021-02-10T10:52:53.314 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1149, "groups": "sudo", "home": "/home/pdonnell", "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "move_home": false, "name": "pdonnell", "shell": "/bin/bash", "state": "present", "uid": 1149} 2021-02-10T10:52:53.336 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1149, "groups": "sudo", "home": "/home/pdonnell", "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "move_home": false, "name": "pdonnell", "shell": "/bin/bash", "state": "present", "uid": 1149} 2021-02-10T10:52:53.764 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1150, "groups": "sudo", "home": "/home/jlayton", "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "move_home": false, "name": "jlayton", "shell": "/bin/bash", "state": "present", "uid": 1150} 2021-02-10T10:52:53.776 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1150, "groups": "sudo", "home": "/home/jlayton", "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "move_home": false, "name": "jlayton", "shell": "/bin/bash", "state": "present", "uid": 1150} 2021-02-10T10:52:54.192 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1151, "groups": "sudo", "home": "/home/rzarzynski", "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "move_home": false, "name": "rzarzynski", "shell": "/bin/bash", "state": "present", "uid": 1151} 2021-02-10T10:52:54.214 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1151, "groups": "sudo", "home": "/home/rzarzynski", "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "move_home": false, "name": "rzarzynski", "shell": "/bin/bash", "state": "present", "uid": 1151} 2021-02-10T10:52:54.637 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1152, "groups": "sudo", "home": "/home/rdias", "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "move_home": false, "name": "rdias", "shell": "/bin/bash", "state": "present", "uid": 1152} 2021-02-10T10:52:54.660 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1152, "groups": "sudo", "home": "/home/rdias", "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "move_home": false, "name": "rdias", "shell": "/bin/bash", "state": "present", "uid": 1152} 2021-02-10T10:52:55.082 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1153, "groups": "sudo", "home": "/home/vshankar", "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "move_home": false, "name": "vshankar", "shell": "/bin/bash", "state": "present", "uid": 1153} 2021-02-10T10:52:55.104 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1153, "groups": "sudo", "home": "/home/vshankar", "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "move_home": false, "name": "vshankar", "shell": "/bin/bash", "state": "present", "uid": 1153} 2021-02-10T10:52:55.527 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1154, "groups": "sudo", "home": "/home/akupczyk", "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "move_home": false, "name": "akupczyk", "shell": "/bin/bash", "state": "present", "uid": 1154} 2021-02-10T10:52:55.550 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1154, "groups": "sudo", "home": "/home/akupczyk", "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "move_home": false, "name": "akupczyk", "shell": "/bin/bash", "state": "present", "uid": 1154} 2021-02-10T10:52:55.961 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1155, "groups": "sudo", "home": "/home/nojha", "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "move_home": false, "name": "nojha", "shell": "/bin/bash", "state": "present", "uid": 1155} 2021-02-10T10:52:55.994 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1155, "groups": "sudo", "home": "/home/nojha", "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "move_home": false, "name": "nojha", "shell": "/bin/bash", "state": "present", "uid": 1155} 2021-02-10T10:52:56.403 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1156, "groups": "sudo", "home": "/home/ifed01", "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "move_home": false, "name": "ifed01", "shell": "/bin/bash", "state": "present", "uid": 1156} 2021-02-10T10:52:56.472 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1156, "groups": "sudo", "home": "/home/ifed01", "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "move_home": false, "name": "ifed01", "shell": "/bin/bash", "state": "present", "uid": 1156} 2021-02-10T10:52:56.859 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1157, "groups": "sudo", "home": "/home/myoungwon", "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "move_home": false, "name": "myoungwon", "shell": "/bin/bash", "state": "present", "uid": 1157} 2021-02-10T10:52:56.891 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1157, "groups": "sudo", "home": "/home/myoungwon", "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "move_home": false, "name": "myoungwon", "shell": "/bin/bash", "state": "present", "uid": 1157} 2021-02-10T10:52:57.290 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1158, "groups": "sudo", "home": "/home/gabrioux", "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "move_home": false, "name": "gabrioux", "shell": "/bin/bash", "state": "present", "uid": 1158} 2021-02-10T10:52:57.344 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1158, "groups": "sudo", "home": "/home/gabrioux", "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "move_home": false, "name": "gabrioux", "shell": "/bin/bash", "state": "present", "uid": 1158} 2021-02-10T10:52:57.756 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1159, "groups": "sudo", "home": "/home/leseb", "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "move_home": false, "name": "leseb", "shell": "/bin/bash", "state": "present", "uid": 1159} 2021-02-10T10:52:57.787 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1159, "groups": "sudo", "home": "/home/leseb", "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "move_home": false, "name": "leseb", "shell": "/bin/bash", "state": "present", "uid": 1159} 2021-02-10T10:52:58.187 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1160, "groups": "sudo", "home": "/home/hchen", "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "move_home": false, "name": "hchen", "shell": "/bin/bash", "state": "present", "uid": 1160} 2021-02-10T10:52:58.231 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1160, "groups": "sudo", "home": "/home/hchen", "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "move_home": false, "name": "hchen", "shell": "/bin/bash", "state": "present", "uid": 1160} 2021-02-10T10:52:58.633 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1161, "groups": "sudo", "home": "/home/jcollin", "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "move_home": false, "name": "jcollin", "shell": "/bin/bash", "state": "present", "uid": 1161} 2021-02-10T10:52:58.665 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1161, "groups": "sudo", "home": "/home/jcollin", "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "move_home": false, "name": "jcollin", "shell": "/bin/bash", "state": "present", "uid": 1161} 2021-02-10T10:52:59.066 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1162, "groups": "sudo", "home": "/home/xxg", "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "move_home": false, "name": "xxg", "shell": "/bin/bash", "state": "present", "uid": 1162} 2021-02-10T10:52:59.109 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1162, "groups": "sudo", "home": "/home/xxg", "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "move_home": false, "name": "xxg", "shell": "/bin/bash", "state": "present", "uid": 1162} 2021-02-10T10:52:59.507 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1163, "groups": "sudo", "home": "/home/pcuzner", "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "move_home": false, "name": "pcuzner", "shell": "/bin/bash", "state": "present", "uid": 1163} 2021-02-10T10:52:59.549 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1163, "groups": "sudo", "home": "/home/pcuzner", "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "move_home": false, "name": "pcuzner", "shell": "/bin/bash", "state": "present", "uid": 1163} 2021-02-10T10:52:59.938 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1164, "groups": "sudo", "home": "/home/mkogan", "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "move_home": false, "name": "mkogan", "shell": "/bin/bash", "state": "present", "uid": 1164} 2021-02-10T10:52:59.984 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1164, "groups": "sudo", "home": "/home/mkogan", "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "move_home": false, "name": "mkogan", "shell": "/bin/bash", "state": "present", "uid": 1164} 2021-02-10T10:53:00.395 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1140} 2021-02-10T10:53:00.428 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1140} 2021-02-10T10:53:00.831 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1165, "groups": "sudo", "home": "/home/henrix", "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "move_home": false, "name": "henrix", "shell": "/bin/bash", "state": "present", "uid": 1165} 2021-02-10T10:53:00.874 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1165, "groups": "sudo", "home": "/home/henrix", "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "move_home": false, "name": "henrix", "shell": "/bin/bash", "state": "present", "uid": 1165} 2021-02-10T10:53:01.275 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1166, "groups": "sudo", "home": "/home/pbs1108", "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "move_home": false, "name": "pbs1108", "shell": "/bin/bash", "state": "present", "uid": 1166} 2021-02-10T10:53:01.319 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1166, "groups": "sudo", "home": "/home/pbs1108", "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "move_home": false, "name": "pbs1108", "shell": "/bin/bash", "state": "present", "uid": 1166} 2021-02-10T10:53:01.737 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1167, "groups": "sudo", "home": "/home/clacroix", "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "move_home": false, "name": "clacroix", "shell": "/bin/bash", "state": "present", "uid": 1167} 2021-02-10T10:53:01.770 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1167, "groups": "sudo", "home": "/home/clacroix", "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "move_home": false, "name": "clacroix", "shell": "/bin/bash", "state": "present", "uid": 1167} 2021-02-10T10:53:02.188 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1168, "groups": "sudo", "home": "/home/epuertat", "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "move_home": false, "name": "epuertat", "shell": "/bin/bash", "state": "present", "uid": 1168} 2021-02-10T10:53:02.220 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1168, "groups": "sudo", "home": "/home/epuertat", "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "move_home": false, "name": "epuertat", "shell": "/bin/bash", "state": "present", "uid": 1168} 2021-02-10T10:53:02.656 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1169, "groups": "sudo", "home": "/home/tdehler", "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "move_home": false, "name": "tdehler", "shell": "/bin/bash", "state": "present", "uid": 1169} 2021-02-10T10:53:02.705 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1169, "groups": "sudo", "home": "/home/tdehler", "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "move_home": false, "name": "tdehler", "shell": "/bin/bash", "state": "present", "uid": 1169} 2021-02-10T10:53:03.108 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1170, "groups": "sudo", "home": "/home/laura", "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "move_home": false, "name": "laura", "shell": "/bin/bash", "state": "present", "uid": 1170} 2021-02-10T10:53:03.152 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1170, "groups": "sudo", "home": "/home/laura", "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "move_home": false, "name": "laura", "shell": "/bin/bash", "state": "present", "uid": 1170} 2021-02-10T10:53:03.554 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1171, "groups": "sudo", "home": "/home/yaarit", "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "move_home": false, "name": "yaarit", "shell": "/bin/bash", "state": "present", "uid": 1171} 2021-02-10T10:53:03.587 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1171, "groups": "sudo", "home": "/home/yaarit", "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "move_home": false, "name": "yaarit", "shell": "/bin/bash", "state": "present", "uid": 1171} 2021-02-10T10:53:04.000 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1172, "groups": "sudo", "home": "/home/rpavani1998", "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "move_home": false, "name": "rpavani1998", "shell": "/bin/bash", "state": "present", "uid": 1172} 2021-02-10T10:53:04.032 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1172, "groups": "sudo", "home": "/home/rpavani1998", "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "move_home": false, "name": "rpavani1998", "shell": "/bin/bash", "state": "present", "uid": 1172} 2021-02-10T10:53:04.458 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1173, "groups": "sudo", "home": "/home/rishabh", "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "move_home": false, "name": "rishabh", "shell": "/bin/bash", "state": "present", "uid": 1173} 2021-02-10T10:53:04.469 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1173, "groups": "sudo", "home": "/home/rishabh", "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "move_home": false, "name": "rishabh", "shell": "/bin/bash", "state": "present", "uid": 1173} 2021-02-10T10:53:04.911 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1174, "groups": "sudo", "home": "/home/skrah", "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "move_home": false, "name": "skrah", "shell": "/bin/bash", "state": "present", "uid": 1174} 2021-02-10T10:53:04.922 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1174, "groups": "sudo", "home": "/home/skrah", "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "move_home": false, "name": "skrah", "shell": "/bin/bash", "state": "present", "uid": 1174} 2021-02-10T10:53:05.348 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1175, "groups": "sudo", "home": "/home/smanjara", "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "move_home": false, "name": "smanjara", "shell": "/bin/bash", "state": "present", "uid": 1175} 2021-02-10T10:53:05.360 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1175, "groups": "sudo", "home": "/home/smanjara", "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "move_home": false, "name": "smanjara", "shell": "/bin/bash", "state": "present", "uid": 1175} 2021-02-10T10:53:05.791 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1176, "groups": "sudo", "home": "/home/bengland", "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "move_home": false, "name": "bengland", "shell": "/bin/bash", "state": "present", "uid": 1176} 2021-02-10T10:53:05.804 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1176, "groups": "sudo", "home": "/home/bengland", "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "move_home": false, "name": "bengland", "shell": "/bin/bash", "state": "present", "uid": 1176} 2021-02-10T10:53:06.236 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1177, "groups": "sudo", "home": "/home/pnawracay", "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "move_home": false, "name": "pnawracay", "shell": "/bin/bash", "state": "present", "uid": 1177} 2021-02-10T10:53:06.237 INFO:teuthology.task.ansible.out: 2021-02-10T10:53:06.238 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1177, "groups": "sudo", "home": "/home/pnawracay", "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "move_home": false, "name": "pnawracay", "shell": "/bin/bash", "state": "present", "uid": 1177} 2021-02-10T10:53:06.681 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1178, "groups": "sudo", "home": "/home/alfonsomthd", "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "move_home": false, "name": "alfonsomthd", "shell": "/bin/bash", "state": "present", "uid": 1178} 2021-02-10T10:53:06.682 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1178, "groups": "sudo", "home": "/home/alfonsomthd", "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "move_home": false, "name": "alfonsomthd", "shell": "/bin/bash", "state": "present", "uid": 1178} 2021-02-10T10:53:07.122 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1179, "groups": "sudo", "home": "/home/oliveiradan", "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "move_home": false, "name": "oliveiradan", "shell": "/bin/bash", "state": "present", "uid": 1179} 2021-02-10T10:53:07.123 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1179, "groups": "sudo", "home": "/home/oliveiradan", "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "move_home": false, "name": "oliveiradan", "shell": "/bin/bash", "state": "present", "uid": 1179} 2021-02-10T10:53:07.561 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1180, "groups": "sudo", "home": "/home/swagner", "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "move_home": false, "name": "swagner", "shell": "/bin/bash", "state": "present", "uid": 1180} 2021-02-10T10:53:07.562 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1180, "groups": "sudo", "home": "/home/swagner", "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "move_home": false, "name": "swagner", "shell": "/bin/bash", "state": "present", "uid": 1180} 2021-02-10T10:53:08.000 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1181, "groups": "sudo", "home": "/home/yuvalif", "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "move_home": false, "name": "yuvalif", "shell": "/bin/bash", "state": "present", "uid": 1181} 2021-02-10T10:53:08.001 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1181, "groups": "sudo", "home": "/home/yuvalif", "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "move_home": false, "name": "yuvalif", "shell": "/bin/bash", "state": "present", "uid": 1181} 2021-02-10T10:53:08.444 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1182, "groups": "sudo", "home": "/home/kkeithle", "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "move_home": false, "name": "kkeithle", "shell": "/bin/bash", "state": "present", "uid": 1182} 2021-02-10T10:53:08.445 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1182, "groups": "sudo", "home": "/home/kkeithle", "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "move_home": false, "name": "kkeithle", "shell": "/bin/bash", "state": "present", "uid": 1182} 2021-02-10T10:53:08.877 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1183, "groups": "sudo", "home": "/home/emmericp", "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "move_home": false, "name": "emmericp", "shell": "/bin/bash", "state": "present", "uid": 1183} 2021-02-10T10:53:08.889 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1183, "groups": "sudo", "home": "/home/emmericp", "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "move_home": false, "name": "emmericp", "shell": "/bin/bash", "state": "present", "uid": 1183} 2021-02-10T10:53:09.323 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1184, "groups": "sudo", "home": "/home/mchangir", "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "move_home": false, "name": "mchangir", "shell": "/bin/bash", "state": "present", "uid": 1184} 2021-02-10T10:53:09.324 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1184, "groups": "sudo", "home": "/home/mchangir", "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "move_home": false, "name": "mchangir", "shell": "/bin/bash", "state": "present", "uid": 1184} 2021-02-10T10:53:09.763 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1185, "groups": "sudo", "home": "/home/sidharthanup", "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "move_home": false, "name": "sidharthanup", "shell": "/bin/bash", "state": "present", "uid": 1185} 2021-02-10T10:53:09.764 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1185, "groups": "sudo", "home": "/home/sidharthanup", "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "move_home": false, "name": "sidharthanup", "shell": "/bin/bash", "state": "present", "uid": 1185} 2021-02-10T10:53:10.201 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1186, "groups": "sudo", "home": "/home/varsha", "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "move_home": false, "name": "varsha", "shell": "/bin/bash", "state": "present", "uid": 1186} 2021-02-10T10:53:10.202 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1186, "groups": "sudo", "home": "/home/varsha", "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "move_home": false, "name": "varsha", "shell": "/bin/bash", "state": "present", "uid": 1186} 2021-02-10T10:53:10.640 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1187, "groups": "sudo", "home": "/home/sjust", "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "move_home": false, "name": "sjust", "shell": "/bin/bash", "state": "present", "uid": 1187} 2021-02-10T10:53:10.641 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1187, "groups": "sudo", "home": "/home/sjust", "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "move_home": false, "name": "sjust", "shell": "/bin/bash", "state": "present", "uid": 1187} 2021-02-10T10:53:11.086 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1188, "groups": "sudo", "home": "/home/ideepika", "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "move_home": false, "name": "ideepika", "shell": "/bin/bash", "state": "present", "uid": 1188} 2021-02-10T10:53:11.087 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1188, "groups": "sudo", "home": "/home/ideepika", "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "move_home": false, "name": "ideepika", "shell": "/bin/bash", "state": "present", "uid": 1188} 2021-02-10T10:53:11.529 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1189, "groups": "sudo", "home": "/home/gsalomon", "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "move_home": false, "name": "gsalomon", "shell": "/bin/bash", "state": "present", "uid": 1189} 2021-02-10T10:53:11.540 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1189, "groups": "sudo", "home": "/home/gsalomon", "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "move_home": false, "name": "gsalomon", "shell": "/bin/bash", "state": "present", "uid": 1189} 2021-02-10T10:53:11.974 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1190, "groups": "sudo", "home": "/home/soumyakoduri", "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, "move_home": false, "name": "soumyakoduri", "shell": "/bin/bash", "state": "present", "uid": 1190} ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1190, "groups": "sudo", "home": "/home/soumyakoduri", "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, "move_home": false, "name": "soumyakoduri", "shell": "/bin/bash", "state": "present", "uid": 1190} 2021-02-10T10:53:12.416 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1191, "groups": "sudo", "home": "/home/kyr", "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "move_home": false, "name": "kyr", "shell": "/bin/bash", "state": "present", "uid": 1191} 2021-02-10T10:53:12.417 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1191, "groups": "sudo", "home": "/home/kyr", "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "move_home": false, "name": "kyr", "shell": "/bin/bash", "state": "present", "uid": 1191} 2021-02-10T10:53:12.864 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1192, "groups": "sudo", "home": "/home/sseshasa", "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "move_home": false, "name": "sseshasa", "shell": "/bin/bash", "state": "present", "uid": 1192} 2021-02-10T10:53:12.865 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1192, "groups": "sudo", "home": "/home/sseshasa", "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "move_home": false, "name": "sseshasa", "shell": "/bin/bash", "state": "present", "uid": 1192} 2021-02-10T10:53:13.304 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1193, "groups": "sudo", "home": "/home/rfriedma", "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "move_home": false, "name": "rfriedma", "shell": "/bin/bash", "state": "present", "uid": 1193} 2021-02-10T10:53:13.305 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1193, "groups": "sudo", "home": "/home/rfriedma", "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "move_home": false, "name": "rfriedma", "shell": "/bin/bash", "state": "present", "uid": 1193} 2021-02-10T10:53:13.742 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1194, "groups": "sudo", "home": "/home/lmb", "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "move_home": false, "name": "lmb", "shell": "/bin/bash", "state": "present", "uid": 1194} 2021-02-10T10:53:13.743 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1194, "groups": "sudo", "home": "/home/lmb", "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "move_home": false, "name": "lmb", "shell": "/bin/bash", "state": "present", "uid": 1194} 2021-02-10T10:53:14.185 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1195, "groups": "sudo", "home": "/home/ksirivad", "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "move_home": false, "name": "ksirivad", "shell": "/bin/bash", "state": "present", "uid": 1195} 2021-02-10T10:53:14.186 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1195, "groups": "sudo", "home": "/home/ksirivad", "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "move_home": false, "name": "ksirivad", "shell": "/bin/bash", "state": "present", "uid": 1195} 2021-02-10T10:53:14.617 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1196, "groups": "sudo", "home": "/home/ofriedma", "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "move_home": false, "name": "ofriedma", "shell": "/bin/bash", "state": "present", "uid": 1196} 2021-02-10T10:53:14.628 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1196, "groups": "sudo", "home": "/home/ofriedma", "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "move_home": false, "name": "ofriedma", "shell": "/bin/bash", "state": "present", "uid": 1196} 2021-02-10T10:53:15.056 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1197, "groups": "sudo", "home": "/home/h4iresis", "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "move_home": false, "name": "h4iresis", "shell": "/bin/bash", "state": "present", "uid": 1197} 2021-02-10T10:53:15.067 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1197, "groups": "sudo", "home": "/home/h4iresis", "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "move_home": false, "name": "h4iresis", "shell": "/bin/bash", "state": "present", "uid": 1197} 2021-02-10T10:53:15.497 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1198, "groups": "sudo", "home": "/home/shyukri", "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "move_home": false, "name": "shyukri", "shell": "/bin/bash", "state": "present", "uid": 1198} 2021-02-10T10:53:15.508 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1198, "groups": "sudo", "home": "/home/shyukri", "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "move_home": false, "name": "shyukri", "shell": "/bin/bash", "state": "present", "uid": 1198} 2021-02-10T10:53:15.940 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1199, "groups": "sudo", "home": "/home/scarvalhojr", "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "move_home": false, "name": "scarvalhojr", "shell": "/bin/bash", "state": "present", "uid": 1199} ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1199, "groups": "sudo", "home": "/home/scarvalhojr", "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "move_home": false, "name": "scarvalhojr", "shell": "/bin/bash", "state": "present", "uid": 1199} 2021-02-10T10:53:16.371 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1200, "groups": "sudo", "home": "/home/rakesh", "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "move_home": false, "name": "rakesh", "shell": "/bin/bash", "state": "present", "uid": 1200} 2021-02-10T10:53:16.372 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1200, "groups": "sudo", "home": "/home/rakesh", "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "move_home": false, "name": "rakesh", "shell": "/bin/bash", "state": "present", "uid": 1200} 2021-02-10T10:53:16.812 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1201, "groups": "sudo", "home": "/home/ryg-", "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "move_home": false, "name": "ryg-", "shell": "/bin/bash", "state": "present", "uid": 1201} 2021-02-10T10:53:16.813 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1201, "groups": "sudo", "home": "/home/ryg-", "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "move_home": false, "name": "ryg-", "shell": "/bin/bash", "state": "present", "uid": 1201} 2021-02-10T10:53:17.253 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1148} 2021-02-10T10:53:17.254 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1148} 2021-02-10T10:53:17.693 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1202, "groups": "sudo", "home": "/home/zdover", "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "move_home": false, "name": "zdover", "shell": "/bin/bash", "state": "present", "uid": 1202} 2021-02-10T10:53:17.694 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1202, "groups": "sudo", "home": "/home/zdover", "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "move_home": false, "name": "zdover", "shell": "/bin/bash", "state": "present", "uid": 1202} 2021-02-10T10:53:18.126 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1203, "groups": "sudo", "home": "/home/rraja", "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "move_home": false, "name": "rraja", "shell": "/bin/bash", "state": "present", "uid": 1203} 2021-02-10T10:53:18.137 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1203, "groups": "sudo", "home": "/home/rraja", "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "move_home": false, "name": "rraja", "shell": "/bin/bash", "state": "present", "uid": 1203} 2021-02-10T10:53:18.578 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1204, "groups": "sudo", "home": "/home/toabctl", "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "move_home": false, "name": "toabctl", "shell": "/bin/bash", "state": "present", "uid": 1204} 2021-02-10T10:53:18.579 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1204, "groups": "sudo", "home": "/home/toabctl", "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "move_home": false, "name": "toabctl", "shell": "/bin/bash", "state": "present", "uid": 1204} 2021-02-10T10:53:19.010 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1205, "groups": "sudo", "home": "/home/mgfritch", "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "move_home": false, "name": "mgfritch", "shell": "/bin/bash", "state": "present", "uid": 1205} 2021-02-10T10:53:19.011 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1205, "groups": "sudo", "home": "/home/mgfritch", "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "move_home": false, "name": "mgfritch", "shell": "/bin/bash", "state": "present", "uid": 1205} 2021-02-10T10:53:19.453 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1206, "groups": "sudo", "home": "/home/xiubli", "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "move_home": false, "name": "xiubli", "shell": "/bin/bash", "state": "present", "uid": 1206} 2021-02-10T10:53:19.453 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1206, "groups": "sudo", "home": "/home/xiubli", "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "move_home": false, "name": "xiubli", "shell": "/bin/bash", "state": "present", "uid": 1206} 2021-02-10T10:53:19.896 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1207, "groups": "sudo", "home": "/home/jafaj", "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "move_home": false, "name": "jafaj", "shell": "/bin/bash", "state": "present", "uid": 1207} 2021-02-10T10:53:19.897 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1207, "groups": "sudo", "home": "/home/jafaj", "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "move_home": false, "name": "jafaj", "shell": "/bin/bash", "state": "present", "uid": 1207} 2021-02-10T10:53:20.335 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1208, "groups": "sudo", "home": "/home/jschmid", "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "move_home": false, "name": "jschmid", "shell": "/bin/bash", "state": "present", "uid": 1208} 2021-02-10T10:53:20.335 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1208, "groups": "sudo", "home": "/home/jschmid", "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "move_home": false, "name": "jschmid", "shell": "/bin/bash", "state": "present", "uid": 1208} 2021-02-10T10:53:20.774 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1209, "groups": "sudo", "home": "/home/kachau", "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "move_home": false, "name": "kachau", "shell": "/bin/bash", "state": "present", "uid": 1209} 2021-02-10T10:53:20.775 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1209, "groups": "sudo", "home": "/home/kachau", "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "move_home": false, "name": "kachau", "shell": "/bin/bash", "state": "present", "uid": 1209} 2021-02-10T10:53:21.206 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1210, "groups": "sudo", "home": "/home/khiremat", "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "move_home": false, "name": "khiremat", "shell": "/bin/bash", "state": "present", "uid": 1210} 2021-02-10T10:53:21.218 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1210, "groups": "sudo", "home": "/home/khiremat", "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "move_home": false, "name": "khiremat", "shell": "/bin/bash", "state": "present", "uid": 1210} 2021-02-10T10:53:21.648 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1211, "groups": "sudo", "home": "/home/kapandya", "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "move_home": false, "name": "kapandya", "shell": "/bin/bash", "state": "present", "uid": 1211} 2021-02-10T10:53:21.650 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1211, "groups": "sudo", "home": "/home/kapandya", "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "move_home": false, "name": "kapandya", "shell": "/bin/bash", "state": "present", "uid": 1211} 2021-02-10T10:53:22.089 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1212, "groups": "sudo", "home": "/home/prasad", "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "move_home": false, "name": "prasad", "shell": "/bin/bash", "state": "present", "uid": 1212} 2021-02-10T10:53:22.101 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1212, "groups": "sudo", "home": "/home/prasad", "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "move_home": false, "name": "prasad", "shell": "/bin/bash", "state": "present", "uid": 1212} 2021-02-10T10:53:22.530 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1213, "groups": "sudo", "home": "/home/vikhyat", "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "move_home": false, "name": "vikhyat", "shell": "/bin/bash", "state": "present", "uid": 1213} 2021-02-10T10:53:22.531 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1213, "groups": "sudo", "home": "/home/vikhyat", "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "move_home": false, "name": "vikhyat", "shell": "/bin/bash", "state": "present", "uid": 1213} 2021-02-10T10:53:22.973 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1214, "groups": "sudo", "home": "/home/shraddhaag", "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "move_home": false, "name": "shraddhaag", "shell": "/bin/bash", "state": "present", "uid": 1214} 2021-02-10T10:53:22.974 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1214, "groups": "sudo", "home": "/home/shraddhaag", "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "move_home": false, "name": "shraddhaag", "shell": "/bin/bash", "state": "present", "uid": 1214} 2021-02-10T10:53:23.410 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1215, "groups": "sudo", "home": "/home/gkyratsas", "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "move_home": false, "name": "gkyratsas", "shell": "/bin/bash", "state": "present", "uid": 1215} 2021-02-10T10:53:23.411 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1215, "groups": "sudo", "home": "/home/gkyratsas", "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "move_home": false, "name": "gkyratsas", "shell": "/bin/bash", "state": "present", "uid": 1215} 2021-02-10T10:53:23.853 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1216, "groups": "sudo", "home": "/home/prateek", "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "move_home": false, "name": "prateek", "shell": "/bin/bash", "state": "present", "uid": 1216} 2021-02-10T10:53:23.854 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1216, "groups": "sudo", "home": "/home/prateek", "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "move_home": false, "name": "prateek", "shell": "/bin/bash", "state": "present", "uid": 1216} 2021-02-10T10:53:24.292 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1217, "groups": "sudo", "home": "/home/jelopez", "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "move_home": false, "name": "jelopez", "shell": "/bin/bash", "state": "present", "uid": 1217} 2021-02-10T10:53:24.303 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1217, "groups": "sudo", "home": "/home/jelopez", "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "move_home": false, "name": "jelopez", "shell": "/bin/bash", "state": "present", "uid": 1217} 2021-02-10T10:53:24.734 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1218, "groups": "sudo", "home": "/home/dcain", "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "move_home": false, "name": "dcain", "shell": "/bin/bash", "state": "present", "uid": 1218} 2021-02-10T10:53:24.745 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1218, "groups": "sudo", "home": "/home/dcain", "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "move_home": false, "name": "dcain", "shell": "/bin/bash", "state": "present", "uid": 1218} 2021-02-10T10:53:25.175 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1219, "groups": "sudo", "home": "/home/sunny", "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "move_home": false, "name": "sunny", "shell": "/bin/bash", "state": "present", "uid": 1219} 2021-02-10T10:53:25.187 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1219, "groups": "sudo", "home": "/home/sunny", "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "move_home": false, "name": "sunny", "shell": "/bin/bash", "state": "present", "uid": 1219} 2021-02-10T10:53:25.633 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1220, "groups": "sudo", "home": "/home/xuxuehan", "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "move_home": false, "name": "xuxuehan", "shell": "/bin/bash", "state": "present", "uid": 1220} 2021-02-10T10:53:25.634 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1220, "groups": "sudo", "home": "/home/xuxuehan", "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "move_home": false, "name": "xuxuehan", "shell": "/bin/bash", "state": "present", "uid": 1220} 2021-02-10T10:53:26.066 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1221, "groups": "sudo", "home": "/home/aclewett", "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "move_home": false, "name": "aclewett", "shell": "/bin/bash", "state": "present", "uid": 1221} 2021-02-10T10:53:26.078 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1221, "groups": "sudo", "home": "/home/aclewett", "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "move_home": false, "name": "aclewett", "shell": "/bin/bash", "state": "present", "uid": 1221} 2021-02-10T10:53:26.507 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1222, "groups": "sudo", "home": "/home/cblum", "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "move_home": false, "name": "cblum", "shell": "/bin/bash", "state": "present", "uid": 1222} 2021-02-10T10:53:26.519 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1222, "groups": "sudo", "home": "/home/cblum", "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "move_home": false, "name": "cblum", "shell": "/bin/bash", "state": "present", "uid": 1222} 2021-02-10T10:53:26.950 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1223, "groups": "sudo", "home": "/home/mlatimer", "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "move_home": false, "name": "mlatimer", "shell": "/bin/bash", "state": "present", "uid": 1223} 2021-02-10T10:53:26.962 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1223, "groups": "sudo", "home": "/home/mlatimer", "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "move_home": false, "name": "mlatimer", "shell": "/bin/bash", "state": "present", "uid": 1223} 2021-02-10T10:53:27.391 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1224, "groups": "sudo", "home": "/home/shyamsundarr", "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "move_home": false, "name": "shyamsundarr", "shell": "/bin/bash", "state": "present", "uid": 1224} 2021-02-10T10:53:27.498 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1224, "groups": "sudo", "home": "/home/shyamsundarr", "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "move_home": false, "name": "shyamsundarr", "shell": "/bin/bash", "state": "present", "uid": 1224} 2021-02-10T10:53:27.833 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1225, "groups": "sudo", "home": "/home/shuzhenyi", "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "move_home": false, "name": "shuzhenyi", "shell": "/bin/bash", "state": "present", "uid": 1225} 2021-02-10T10:53:27.866 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1225, "groups": "sudo", "home": "/home/shuzhenyi", "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "move_home": false, "name": "shuzhenyi", "shell": "/bin/bash", "state": "present", "uid": 1225} 2021-02-10T10:53:28.276 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1226, "groups": "sudo", "home": "/home/mdipalma", "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "move_home": false, "name": "mdipalma", "shell": "/bin/bash", "state": "present", "uid": 1226} 2021-02-10T10:53:28.319 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1226, "groups": "sudo", "home": "/home/mdipalma", "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "move_home": false, "name": "mdipalma", "shell": "/bin/bash", "state": "present", "uid": 1226} 2021-02-10T10:53:28.719 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1227, "groups": "sudo", "home": "/home/kchang", "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "move_home": false, "name": "kchang", "shell": "/bin/bash", "state": "present", "uid": 1227} 2021-02-10T10:53:28.762 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1227, "groups": "sudo", "home": "/home/kchang", "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "move_home": false, "name": "kchang", "shell": "/bin/bash", "state": "present", "uid": 1227} 2021-02-10T10:53:29.168 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1228, "groups": "sudo", "home": "/home/vereddy", "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "move_home": false, "name": "vereddy", "shell": "/bin/bash", "state": "present", "uid": 1228} 2021-02-10T10:53:29.190 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1228, "groups": "sudo", "home": "/home/vereddy", "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "move_home": false, "name": "vereddy", "shell": "/bin/bash", "state": "present", "uid": 1228} 2021-02-10T10:53:29.608 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1229, "groups": "sudo", "home": "/home/benhanokh", "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "move_home": false, "name": "benhanokh", "shell": "/bin/bash", "state": "present", "uid": 1229} 2021-02-10T10:53:29.632 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1229, "groups": "sudo", "home": "/home/benhanokh", "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "move_home": false, "name": "benhanokh", "shell": "/bin/bash", "state": "present", "uid": 1229} 2021-02-10T10:53:30.045 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1230, "groups": "sudo", "home": "/home/jsalomon", "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "move_home": false, "name": "jsalomon", "shell": "/bin/bash", "state": "present", "uid": 1230} 2021-02-10T10:53:30.078 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1230, "groups": "sudo", "home": "/home/jsalomon", "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "move_home": false, "name": "jsalomon", "shell": "/bin/bash", "state": "present", "uid": 1230} 2021-02-10T10:53:30.487 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1231, "groups": "sudo", "home": "/home/jolmomar", "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "move_home": false, "name": "jolmomar", "shell": "/bin/bash", "state": "present", "uid": 1231} 2021-02-10T10:53:30.519 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1231, "groups": "sudo", "home": "/home/jolmomar", "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "move_home": false, "name": "jolmomar", "shell": "/bin/bash", "state": "present", "uid": 1231} 2021-02-10T10:53:30.926 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1232, "groups": "sudo", "home": "/home/hmunjulu", "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "move_home": false, "name": "hmunjulu", "shell": "/bin/bash", "state": "present", "uid": 1232} 2021-02-10T10:53:30.958 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1232, "groups": "sudo", "home": "/home/hmunjulu", "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "move_home": false, "name": "hmunjulu", "shell": "/bin/bash", "state": "present", "uid": 1232} 2021-02-10T10:53:31.366 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1233, "groups": "sudo", "home": "/home/julpark", "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "move_home": false, "name": "julpark", "shell": "/bin/bash", "state": "present", "uid": 1233} 2021-02-10T10:53:31.409 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1233, "groups": "sudo", "home": "/home/julpark", "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "move_home": false, "name": "julpark", "shell": "/bin/bash", "state": "present", "uid": 1233} 2021-02-10T10:53:31.806 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1235, "groups": "sudo", "home": "/home/pdhange", "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "move_home": false, "name": "pdhange", "shell": "/bin/bash", "state": "present", "uid": 1235} 2021-02-10T10:53:31.859 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1235, "groups": "sudo", "home": "/home/pdhange", "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "move_home": false, "name": "pdhange", "shell": "/bin/bash", "state": "present", "uid": 1235} 2021-02-10T10:53:32.663 INFO:teuthology.task.ansible.out:changed: [gibba002.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10103, "groups": "sudo", "home": "/home/dpivonka", "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, "name": "dpivonka", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10102} 2021-02-10T10:53:32.724 INFO:teuthology.task.ansible.out:changed: [gibba013.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10103, "groups": "sudo", "home": "/home/dpivonka", "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, "name": "dpivonka", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10102} 2021-02-10T10:53:33.485 INFO:teuthology.task.ansible.out:changed: [gibba002.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10104, "groups": "sudo", "home": "/home/adking", "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "name": "adking", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10103} 2021-02-10T10:53:33.550 INFO:teuthology.task.ansible.out:changed: [gibba013.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10104, "groups": "sudo", "home": "/home/adking", "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "name": "adking", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10103} 2021-02-10T10:53:34.317 INFO:teuthology.task.ansible.out:changed: [gibba002.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10105, "groups": "sudo", "home": "/home/amathuri", "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267"}, "name": "amathuri", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10104} 2021-02-10T10:53:34.329 INFO:teuthology.task.ansible.out:changed: [gibba013.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10105, "groups": "sudo", "home": "/home/amathuri", "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267"}, "name": "amathuri", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10104} 2021-02-10T10:53:35.096 INFO:teuthology.task.ansible.out:changed: [gibba013.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10106, "groups": "sudo", "home": "/home/nizamial09", "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "name": "nizamial09", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10105} 2021-02-10T10:53:35.212 INFO:teuthology.task.ansible.out:changed: [gibba002.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10106, "groups": "sudo", "home": "/home/nizamial09", "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "name": "nizamial09", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10105} 2021-02-10T10:53:35.319 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:53:35 +0000 (0:01:03.217) 0:01:15.621 **** 2021-02-10T10:53:35.353 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:53:35 +0000 (0:00:00.034) 0:01:15.655 **** 2021-02-10T10:53:35.361 INFO:teuthology.task.ansible.out: TASK [users : Create all users without sudo access.] *************************** 2021-02-10T10:53:35.401 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:53:35 +0000 (0:00:00.047) 0:01:15.703 **** 2021-02-10T10:53:35.443 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:53:35 +0000 (0:00:00.041) 0:01:15.744 **** 2021-02-10T10:53:35.450 INFO:teuthology.task.ansible.out: TASK [users : Merge managed_users and managed_admin users] ********************* 2021-02-10T10:53:35.452 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => {"ansible_facts": {"pubkey_users": [{"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, {"name": "jcaratzas", "ovpn": "jcaratza@fedora r 2021-02-10T10:53:35.454 INFO:teuthology.task.ansible.out:T6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c 2021-02-10T10:53:35.455 INFO:teuthology.task.ansible.out:63f3ea91e7dacd78"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, {"name": "ivancich 2021-02-10T10:53:35.456 INFO:teuthology.task.ansible.out:", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiR 2021-02-10T10:53:35.458 INFO:teuthology.task.ansible.out:enVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, {"name": "jdillaman", "ovpn": "jdi 2021-02-10T10:53:35.458 INFO:teuthology.task.ansible.out:llaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, {"name": "vumrao", "ovpn": "vumrao@vumr 2021-02-10T10:53:35.459 INFO:teuthology.task.ansible.out:ao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69m 2021-02-10T10:53:35.460 INFO:teuthology.task.ansible.out:h95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d84 2021-02-10T10:53:35.462 INFO:teuthology.task.ansible.out:1a16dda2504cca6d95df831a652a30d2e5655ed"}, {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, {"name": "oliveiradan", "ovpn": "oliveiradan 2021-02-10T10:53:35.463 INFO:teuthology.task.ansible.out:@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOen 2021-02-10T10:53:35.465 INFO:teuthology.task.ansible.out:VV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9g 2021-02-10T10:53:35.466 INFO:teuthology.task.ansible.out:I46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, {"name": " 2021-02-10T10:53:35.467 INFO:teuthology.task.ansible.out:dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, {"name": "acl 2021-02-10T10:53:35.468 INFO:teuthology.task.ansible.out:ewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, {"name": "dpivonka", "ovpn": "dpivonka@th 2021-02-10T10:53:35.470 INFO:teuthology.task.ansible.out:inkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267"}, {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}]}, "changed": false} 2021-02-10T10:53:35.512 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:53:35 +0000 (0:00:00.069) 0:01:15.814 **** 2021-02-10T10:53:35.523 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => {"ansible_facts": {"pubkey_users": [{"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, {"name": "jcaratzas", "ovpn": "jcaratza@fedora r 2021-02-10T10:53:35.525 INFO:teuthology.task.ansible.out:T6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c 2021-02-10T10:53:35.526 INFO:teuthology.task.ansible.out:63f3ea91e7dacd78"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, {"name": "ivancich 2021-02-10T10:53:35.528 INFO:teuthology.task.ansible.out:", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiR 2021-02-10T10:53:35.529 INFO:teuthology.task.ansible.out:enVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0 2021-02-10T10:53:35.529 INFO:teuthology.task.ansible.out:a7ab1bce36aeb"}, {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, {"name": "vumrao", "ovpn": "vumrao@vumr 2021-02-10T10:53:35.531 INFO:teuthology.task.ansible.out:ao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69m 2021-02-10T10:53:35.532 INFO:teuthology.task.ansible.out:h95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d84 2021-02-10T10:53:35.534 INFO:teuthology.task.ansible.out:1a16dda2504cca6d95df831a652a30d2e5655ed"}, {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, {"name": "oliveiradan", "ovpn": "oliveiradan 2021-02-10T10:53:35.535 INFO:teuthology.task.ansible.out:@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOen 2021-02-10T10:53:35.536 INFO:teuthology.task.ansible.out:VV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9g 2021-02-10T10:53:35.538 INFO:teuthology.task.ansible.out:I46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, {"name": " 2021-02-10T10:53:35.539 INFO:teuthology.task.ansible.out:dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NIN 2021-02-10T10:53:35.541 INFO:teuthology.task.ansible.out:j0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa33958 2021-02-10T10:53:35.541 INFO:teuthology.task.ansible.out:62e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267"}, {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}]}, "changed": false} 2021-02-10T10:53:35.584 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:53:35 +0000 (0:00:00.072) 0:01:15.886 **** 2021-02-10T10:53:36.403 INFO:teuthology.task.ansible.out: TASK [users : Clone the keys repo] ********************************************* ok: [gibba013.front.sepia.ceph.com -> localhost] => {"after": "df8946f08e870e2d2565b79099289ca48bd825f3", "attempts": 1, "before": "df8946f08e870e2d2565b79099289ca48bd825f3", "changed": false, "remote_url_changed": false} 2021-02-10T10:53:36.441 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:53:36 +0000 (0:00:00.856) 0:01:16.743 **** 2021-02-10T10:53:36.448 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com -> localhost] => {"after": "df8946f08e870e2d2565b79099289ca48bd825f3", "attempts": 1, "before": "df8946f08e870e2d2565b79099289ca48bd825f3", "changed": false, "remote_url_changed": false} 2021-02-10T10:53:36.483 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:53:36 +0000 (0:00:00.042) 0:01:16.785 **** 2021-02-10T10:53:36.941 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-02-10T10:53:36.943 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDdkebJGXKQWBG/7wZG334P+EAdC+5VNMrjLjXhaZZbt3V9vKDQCodcU3zSOjMa/GDGg2WHQdO/vioCc6O/m61KTbgLDaaPSzHH16j/e1SaU8BIgKKU42AUBJzpPeaZ2wCC1i12Hq8ULwjEW29IDsnMortzlCSQmaTKc3DN2nCXZWQXA1rSiQ2AhIq5WNJY9S36lsg5tkex4xUajyNW4vPubLtkSv+WrzNK7ZJ8zfzP6eJEJ+Uj7CLdanBCgwoLGSPfGA5c/gyUT7mHibOUwvFB5sFVsozMPp2PXzUUIv70adJpOskLYfKY7TUSuaB4AUOPu6NVauzVIl3Tsn+uvHqvtn8KC3BPpyH7+JiFQWEt+aO+/ynprfoqCjktuf07SXo0ewoVYT2S++JEBa/N2hKRkzIoSPJ+Rvoazc8GZIjXpQRyhMdTV/LCSeyUTJ7lDQVrQVhUIHKnMm+hZmf4tLQKZvNAThO984CpY93ST5Ww07fjRLMXNZGuaz+zHKNexhWSPKr+ggChnZCOLl/F6zbDnBZoCaHR6hH4+QLz0K/FAgxN2Nn00RiOFkmIlzEioxMcwC/peTl3WzRfqPINdTftiTzcFi/c2QmlHhnxsueIwNhHOn4frkGkFmexdDrXRStERzSBJ13X4kWduPmQsn+bx5/7xwqC6M4bXpkYFOefOw== aschoen@redhat.com", "key_options": null, "keyfile": "/home/andrewschoen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "andrewschoen", "validate_certs": true} 2021-02-10T10:53:36.974 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDdkebJGXKQWBG/7wZG334P+EAdC+5VNMrjLjXhaZZbt3V9vKDQCodcU3zSOjMa/GDGg2WHQdO/vioCc6O/m61KTbgLDaaPSzHH16j/e1SaU8BIgKKU42AUBJzpPeaZ2wCC1i12Hq8ULwjEW29IDsnMortzlCSQmaTKc3DN2nCXZWQXA1rSiQ2AhIq5WNJY9S36lsg5tkex4xUajyNW4vPubLtkSv+WrzNK7ZJ8zfzP6eJEJ+Uj7CLdanBCgwoLGSPfGA5c/gyUT7mHibOUwvFB5sFVsozMPp2PXzUUIv70adJpOskLYfKY7TUSuaB4AUOPu6NVauzVIl3Tsn+uvHqvtn8KC3BPpyH7+JiFQWEt+aO+/ynprfoqCjktuf07SXo0ewoVYT2S++JEBa/N2hKRkzIoSPJ+Rvoazc8GZIjXpQRyhMdTV/LCSeyUTJ7lDQVrQVhUIHKnMm+hZmf4tLQKZvNAThO984CpY93ST5Ww07fjRLMXNZGuaz+zHKNexhWSPKr+ggChnZCOLl/F6zbDnBZoCaHR6hH4+QLz0K/FAgxN2Nn00RiOFkmIlzEioxMcwC/peTl3WzRfqPINdTftiTzcFi/c2QmlHhnxsueIwNhHOn4frkGkFmexdDrXRStERzSBJ13X4kWduPmQsn+bx5/7xwqC6M4bXpkYFOefOw== aschoen@redhat.com", "key_options": null, "keyfile": "/home/andrewschoen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "andrewschoen", "validate_certs": true} 2021-02-10T10:53:37.390 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local", "key_options": null, "keyfile": "/home/zack/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zack", "validate_certs": true} 2021-02-10T10:53:37.432 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local", "key_options": null, "keyfile": "/home/zack/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zack", "validate_certs": true} 2021-02-10T10:53:37.844 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4eHmz10szeHNS3dNejKokW85ksB+iR4HGOFsmQM11Ni68Nm5aqEKvkOZU8TpY92vpCQL0A68GlrXB845cACdyk6HUJYyNNNMC43l1FYWOwjMqQBSdj8W3VQDTA6eiG60mt5fgI8fyR38rKzIA1MnTBkSSjuh5kQVJ9bdEp3GuY5oc8vxDNBlGJ6LYnyEWt/pqL2J+mpjqnOjsC+EbE2exhP9O+mvzpQiyo/+dEN1COwX3//pNRXGfOSeOczHNsJE8Eu+j/n/BlW57++sJyFMkzS7bUxMSGM6quvjQZ7RT1c5JM6vLEiQyzQxoRgzY93h1yKlOstBi0NamtpqHQZGP kdreyer@mbp.ktdreyer.com", "key_options": null, "keyfile": "/home/kdreyer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kdreyer", "validate_certs": true} 2021-02-10T10:53:37.877 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4eHmz10szeHNS3dNejKokW85ksB+iR4HGOFsmQM11Ni68Nm5aqEKvkOZU8TpY92vpCQL0A68GlrXB845cACdyk6HUJYyNNNMC43l1FYWOwjMqQBSdj8W3VQDTA6eiG60mt5fgI8fyR38rKzIA1MnTBkSSjuh5kQVJ9bdEp3GuY5oc8vxDNBlGJ6LYnyEWt/pqL2J+mpjqnOjsC+EbE2exhP9O+mvzpQiyo/+dEN1COwX3//pNRXGfOSeOczHNsJE8Eu+j/n/BlW57++sJyFMkzS7bUxMSGM6quvjQZ7RT1c5JM6vLEiQyzQxoRgzY93h1yKlOstBi0NamtpqHQZGP kdreyer@mbp.ktdreyer.com", "key_options": null, "keyfile": "/home/kdreyer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kdreyer", "validate_certs": true} 2021-02-10T10:53:38.304 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/onMoOCUhkp+UiJAq+ye4P70sfeV+uMhgWKLfLBh/1SYq1xxb9bSmjBnETO/9KvukRhzlReBQ7y+BKRhm037P7LuTUY4r0Y8JB6BXkjjCY1yXZJC6kvB7xFsvyiDum5rrdeWMQ/Rhh7YJpze5KY1nHLkyU9X7Ewak0kLEyduJEKekbsOom6l0wZ2ZYrBEKFWJXUDk5icUuWWg3mhKaKnLNWVg1lUw8jzYTJYfjR9GUgzZWXtzkAaI3ugpLOlQ+KEls03/WAFcWA9F9ExqzIsJpvEks27LeFBGVE3PjVNKMqqBZQlZt 2021-02-10T10:53:38.305 INFO:teuthology.task.ansible.out:IEz1iguWJaWMi5lDIfXkvVOFabtIpX/BZRf dmick@dmick-MacBookAir\nssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOoIm6sOytKwqNf9FSQn0Olu72ZRDjAYeNFQVWTn2PIB ed25519-hplaptop", "key_options": null, "keyfile": "/home/dmick/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dmick", "validate_certs": true} 2021-02-10T10:53:38.325 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/onMoOCUhkp+UiJAq+ye4P70sfeV+uMhgWKLfLBh/1SYq1xxb9bSmjBnETO/9KvukRhzlReBQ7y+BKRhm037P7LuTUY4r0Y8JB6BXkjjCY1yXZJC6kvB7xFsvyiDum5rrdeWMQ/Rhh7YJpze5KY1nHLkyU9X7Ewak0kLEyduJEKekbsOom6l0wZ2ZYrBEKFWJXUDk5icUuWWg3mhKaKnLNWVg1lUw8jzYTJYfjR9GUgzZWXtzkAaI3ugpLOlQ+KEls03/WAFcWA9F9ExqzIsJpvEks27LeFBGVE3PjVNKMqqBZQlZt 2021-02-10T10:53:38.327 INFO:teuthology.task.ansible.out:IEz1iguWJaWMi5lDIfXkvVOFabtIpX/BZRf dmick@dmick-MacBookAir\nssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOoIm6sOytKwqNf9FSQn0Olu72ZRDjAYeNFQVWTn2PIB ed25519-hplaptop", "key_options": null, "keyfile": "/home/dmick/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dmick", "validate_certs": true} 2021-02-10T10:53:38.752 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJ4z0gLwjexgyixdTuiwaxpPnq7w4FOKxHsQKKAHTlKRu11ynB+9APbtqaAYss8yDqHaHyewPdE+9JMYhZX8rUgcKv2H7jBCQ/Enfl0UyY643VQnuG8lr2Ipk6ELuL5NuuTb7Npo4zKyq/0649kPD46k/X/qLuKzhN49k50S3XJq5WwHi6CJvR8DzPOkjied0o1BHpfcCCIMorUcyfVxqmeu2m+uSE15ncQsd8BIvH4/YNoncUGYxqVhGB+FNUlWdhV0SPRW7V/gmyvFW5qdYX2QhNZAQHChdEH1E9AI7WiCFkyLRx39qYgXgqj8vOOnlwHgy/oJnpnu/Mlvj1uETX sage@montreal\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCVPfoCpv2/9uWCwEo4clWt3i1O8ZPpB2FcQbBLxY3iLQWZJDe74rNWPBj0cJVCytTHxYV0e5HxYyChOG60ca5odHlPNB+/Z4WLU9LVuHiCYsElivJlwvD+l9kzI+UMGJb3HCHqx2yIRGXpEkSMFR1+w5Oz9sInPD7JJmO8Hb9KkoeqZtkJL0aic/M5bDWBut6SSfYlmh8qnQYbkCDiXg/KtOcy1ZJXQ9ZLjsCbpSzeBNbdUOxEQ1+5vUe00vHpl0vOKtRrzgTne2ct1IxbSjdL16ZgHi5BQAREXc9UUG3z7NPOmnwkuKkkpyLmwcOusot5tYHrxks+5i6EoGEJrvJ3 sage@maetl", "key_options": null, "keyfile": "/home/sage/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sage", "validate_certs": true} 2021-02-10T10:53:38.763 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJ4z0gLwjexgyixdTuiwaxpPnq7w4FOKxHsQKKAHTlKRu11ynB+9APbtqaAYss8yDqHaHyewPdE+9JMYhZX8rUgcKv2H7jBCQ/Enfl0UyY643VQnuG8lr2Ipk6ELuL5NuuTb7Npo4zKyq/0649kPD46k/X/qLuKzhN49k50S3XJq5WwHi6CJvR8DzPOkjied0o1BHpfcCCIMorUcyfVxqmeu2m+uSE15ncQsd8BIvH4/YNoncUGYxqVhGB+FNUlWdhV0SPRW7V/gmyvFW5qdYX2QhNZAQHChdEH1E9AI7WiCFkyLRx39qYgXgqj8vOOnlwHgy/oJnpnu/Mlvj1uETX sage@montreal\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCVPfoCpv2/9uWCwEo4clWt3i1O8ZPpB2FcQbBLxY3iLQWZJDe74rNWPBj0cJVCytTHxYV0e5HxYyChOG60ca5odHlPNB+/Z4WLU9LVuHiCYsElivJlwvD+l9kzI+UMGJb3HCHqx2yIRGXpEkSMFR1+w5Oz9sInPD7JJmO8Hb9KkoeqZtkJL0aic/M5bDWBut6SSfYlmh8qnQYbkCDiXg/KtOcy1ZJXQ9ZLjsCbpSzeBNbdUOxEQ1+5vUe00vHpl0vOKtRrzgTne2ct1IxbSjdL16ZgHi5BQAREXc9UUG3z7NPOmnwkuKkkpyLmwcOusot5tYHrxks+5i6EoGEJrvJ3 sage@maetl", "key_options": null, "keyfile": "/home/sage/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sage", "validate_certs": true} 2021-02-10T10:53:39.190 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBAN3MlkYQNsm1uB6uJQ7hWJ1Ro9Gusrhi/b/KshPMFh51VWBQn71LdCCjdrsE6FWs5FYE/hr7MITJe+si7aZCmuf8U9tc+oiYh3spXtRhJVIM80FOVRXUmt2XR8I+oNbpcFDayYu956Volafksk3oYjsO356swj7+Q+AUQVF4bDJRAAAAFQCkB2YZKItSDL7XfAl6ZmJ21vqUMwAAAIEAsZspNIUwssVJGO4r3M3BECWklWLEFviIjGVKdPWulN10eRCczZrH7vMOs8coHKDQqYdS2RejlojT03FjFAqeBiTeAlcmgHoev+vqTg2Yhqx3gROzNc6NmG887owCOU4j7LJQP6SX8I8EpSXCMNOZliv2ZokjX2DuRsuIDxQdc84AAACBAMAp++fcqmGS7x7ic2x4rhyiYmdqlMIl8lrM3GkVR6qGuIVBvOGl38ieJKcTBLeRgyC04uIMfk8P9lLvZw3UF97qH3V0imKfq8aYnBGxCv+IjtdOrR5k4hrUW+P0qipvU+4dgHlpqwTKB4MtD9uDxvhENWvqcCnOS5rI6dddaZT6 dgallowa@dgallowa.csb\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o 2021-02-10T10:53:39.191 INFO:teuthology.task.ansible.out:8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50", "key_options": null, "keyfile": "/home/dgalloway/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dgalloway", "validate_certs": true} 2021-02-10T10:53:39.201 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBAN3MlkYQNsm1uB6uJQ7hWJ1Ro9Gusrhi/b/KshPMFh51VWBQn71LdCCjdrsE6FWs5FYE/hr7MITJe+si7aZCmuf8U9tc+oiYh3spXtRhJVIM80FOVRXUmt2XR8I+oNbpcFDayYu956Volafksk3oYjsO356swj7+Q+AUQVF4bDJRAAAAFQCkB2YZKItSDL7XfAl6ZmJ21vqUMwAAAIEAsZspNIUwssVJGO4r3M3BECWklWLEFviIjGVKdPWulN10eRCczZrH7vMOs8coHKDQqYdS2RejlojT03FjFAqeBiTeAlcmgHoev+vqTg2Yhqx3gROzNc6NmG887owCOU4j7LJQP6SX8I8EpSXCMNOZliv2ZokjX2DuRsuIDxQdc84AAACBAMAp++fcqmGS7x7ic2x4rhyiYmdqlMIl8lrM3GkVR6qGuIVBvOGl38ieJKcTBLeRgyC04uIMfk8P9lLvZw3UF97qH3V0imKfq8aYnBGxCv+IjtdOrR5k4hrUW+P0qipvU+4dgHlpqwTKB4MtD9uDxvhENWvqcCnOS5rI6dddaZT6 dgallowa@dgallowa.csb\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o 2021-02-10T10:53:39.203 INFO:teuthology.task.ansible.out:8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50", "key_options": null, "keyfile": "/home/dgalloway/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dgalloway", "validate_certs": true} 2021-02-10T10:53:39.651 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/akraitma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akraitma", "validate_certs": true} 2021-02-10T10:53:39.652 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/akraitma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akraitma", "validate_certs": true} 2021-02-10T10:53:40.103 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCgaz5AsLB+hPwnHjlf6vktlUQ8iWsKA+hViok+j+uCm0U/iho7O7Qd8E/3tp9yBx10Wkvuz1mmf6v2AO7Qa8+SE1QQPmA+4xLp4xXCqlUGFjnzm/7uQcQZtlWdfBHzQ9jVyPAgUpnGVUtW8PhW0txe8MEtecOO2C3Snx0yuTOx6O11jNvT938QjOeRQWw3X2YFUtMIOiwEhKtqstBIGXlMdaQKfvdlbA9MjyYy6VfK1x8lVcu8umUf/z56YmMkC2iO117/obhdu1XsU9SmjM51wHMOUCXSVqH8E3BWu0yohy4QmZDkdFihyYa+hoS9dOqGC91dFaQc46xrdmV4wqg1 jcaratza@MacBook-Pro\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDAV78mqEy/VkB2BthIE/m7iE5sbTjTRS0KuH7AB0KPvZBsDIw5cL8jKyxiaPC85+aOkBa4LrP0TUFBzN87Snt+PLMz2QHBOvlSTfUuRkHVo3vWJFzVuUSzzddkYds3hAxUriAHKv6CRSy1yFBxPQQHTSATUHKWqihMElbUCZrrrfH8A7J9vwixzCSLg4KbINjfa77rxefcZwugUqiYiVUWUSOb023f+QEaDm7OP+/uzz3SjF1Y0LMwa75yctttIfkRBKjY7uFCuZdKc3l2A/fdPxbFwlZmIox3LLngdTzHMVXGbVHqq9Ixo8FSAXiI1Buy2LTMoXMJwQYSvzsv7xNwujKVlUjoE9cOOIdzCC5AziwSiJB4+tNh73c5tRUzMMC98HLOhPJriSPvOt1pMEdWDX3/evcyxvRgJUKgYnmQG7n8v8O/CLyG0sAJDmGC3U9S/3fUDUjZIKWSsc85Q3xRaJwZJIS2VZwH0zG1Gmynih8PPq7VeDx7tKuNm4uTYT8= bigjust@desktop-jsho3l9.lan", "key_options": null, "keyfile": "/home/jcaratzas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcaratzas", "validate_certs": true} 2021-02-10T10:53:40.114 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCgaz5AsLB+hPwnHjlf6vktlUQ8iWsKA+hViok+j+uCm0U/iho7O7Qd8E/3tp9yBx10Wkvuz1mmf6v2AO7Qa8+SE1QQPmA+4xLp4xXCqlUGFjnzm/7uQcQZtlWdfBHzQ9jVyPAgUpnGVUtW8PhW0txe8MEtecOO2C3Snx0yuTOx6O11jNvT938QjOeRQWw3X2YFUtMIOiwEhKtqstBIGXlMdaQKfvdlbA9MjyYy6VfK1x8lVcu8umUf/z56YmMkC2iO117/obhdu1XsU9SmjM51wHMOUCXSVqH8E3BWu0yohy4QmZDkdFihyYa+hoS9dOqGC91dFaQc46xrdmV4wqg1 jcaratza@MacBook-Pro\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDAV78mqEy/VkB2BthIE/m7iE5sbTjTRS0KuH7AB0KPvZBsDIw5cL8jKyxiaPC85+aOkBa4LrP0TUFBzN87Snt+PLMz2QHBOvlSTfUuRkHVo3vWJFzVuUSzzddkYds3hAxUriAHKv6CRSy1yFBxPQQHTSATUHKWqihMElbUCZrrrfH8A7J9vwixzCSLg4KbINjfa77rxefcZwugUqiYiVUWUSOb023f+QEaDm7OP+/uzz3SjF1Y0LMwa75yctttIfkRBKjY7uFCuZdKc3l2A/fdPxbFwlZmIox3LLngdTzHMVXGbVHqq9Ixo8FSAXiI1Buy2LTMoXMJwQYSvzsv7xNwujKVlUjoE9cOOIdzCC5AziwSiJB4+tNh73c5tRUzMMC98HLOhPJriSPvOt1pMEdWDX3/evcyxvRgJUKgYnmQG7n8v8O/CLyG0sAJDmGC3U9S/3fUDUjZIKWSsc85Q3xRaJwZJIS2VZwH0zG1Gmynih8PPq7VeDx7tKuNm4uTYT8= bigjust@desktop-jsho3l9.lan", "key_options": null, "keyfile": "/home/jcaratzas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcaratzas", "validate_certs": true} 2021-02-10T10:53:40.552 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDSc2rWtLtZM0CBWwaxhSfd0T++ZTX9c8kQrYT0rsQlJ6Hxj4c3+ypwh4iivdtFpyz4clxWkb/wimi48PvjNyZdudB9BeQUBeFfLiro3LnFEbKhM1RwRI89s34mRwmC0LLUZwcdX80zc6/fKuPzvBzR1PiY9tWRyIpYErQAHa2jEOE2MpFDHnlbXdfXFh5Tw3OmC1qBlQvgcrivA9lVaVlf+hfwmOQwXFIG3PCJLNlXDshCSUGAfYKBRvQaKS8sOf9rQaflPrwqJ8hDgVVQkpp69yVsGtahbBmjs6QS6C8EkGUFQmvrIcrQ8+5mRlRAmK2DMTP3cAcsYcMOJ+9KwP+H mgolub@zhuzha", "key_options": null, "keyfile": "/home/trociny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "trociny", "validate_certs": true} 2021-02-10T10:53:40.563 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDSc2rWtLtZM0CBWwaxhSfd0T++ZTX9c8kQrYT0rsQlJ6Hxj4c3+ypwh4iivdtFpyz4clxWkb/wimi48PvjNyZdudB9BeQUBeFfLiro3LnFEbKhM1RwRI89s34mRwmC0LLUZwcdX80zc6/fKuPzvBzR1PiY9tWRyIpYErQAHa2jEOE2MpFDHnlbXdfXFh5Tw3OmC1qBlQvgcrivA9lVaVlf+hfwmOQwXFIG3PCJLNlXDshCSUGAfYKBRvQaKS8sOf9rQaflPrwqJ8hDgVVQkpp69yVsGtahbBmjs6QS6C8EkGUFQmvrIcrQ8+5mRlRAmK2DMTP3cAcsYcMOJ+9KwP+H mgolub@zhuzha", "key_options": null, "keyfile": "/home/trociny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "trociny", "validate_certs": true} 2021-02-10T10:53:40.564 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:53:40.574 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:53:41.010 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANarZ+C62DPffRIhaCuCtXLChrfqwXgWqGHOLLkmPTtZ3/EE8ZiN+LY1ZcmP939AxXi8Oe+9LepwAdsoQKZctlC7XToyhjIk+rVSPyn3zl3BTC2AxfCKq2vqj65a1sc35Whrw7OaUa2fr5iHSazP6HuG01DhJ9l8HJsLKmiwWChZAAAAFQCNmunc7BNLWrqyrYwINTfscowkbQAAAIEAyYERcunZYJAZ9Eno18kM+ylq0jcQmnPqyIvssWXTttUZnaGQvbVxABeyr6ZKuUyZQfyetofquOGBKPBuzV/7posl5ShDuAcxbS1zhrmk9glTA1yFLvznDdWdLOWG06tjOc78G2AmldKHCdPsmgh2D7WyRr4QY4rcCvwKfZ6+eEYAAACBAJZrOlOlygfPZ4ZB0+EbdYACz1+L/iYNNAOBp51TRutNceLVrNJNOVdmCMLuF0A1go4kPSD/05aJOkOyW86gYLA5DXaW3961+kPInOFAe8OTyQYE7221ThYHRnCfVFjdknuCDomj69wk4SsR+ZPXvSXs1jSJWKhLfUSe9n4bmw7w anet2@secant.private.linuxbox.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5CMht/crEocrHqGjf0sxE9AhNcW3XBxwlniJwaBJLhekLQEQJoxAy2xUUmPYb2/e7rdmTBxzQipUaAGWVk1FHANac4cbgXzNT+9cnF7EIRkPa/x9jaKNiU0ozMhu/GOoeDXMrHIDgVGWPF3Ag/r17ORpYnuFTjsqoCWyh+mqskV+lmAp04rv2tvqouHtEs6gLebKNtB8c6twhrBKF62Ie+rSr3r7hQaqiZitHbg/isbU5RMBl6H48wHAVSK2/uTfMorb 2021-02-10T10:53:41.012 INFO:teuthology.task.ansible.out:nuEYRG8HvlnPRvQHscJYKiZSt/G7XqZ3x/7c4czEF+OpdQbAgg46vh4fNVMxZk9KHwEkOE06ROuwgAejJ mbenjamin@duel", "key_options": null, "keyfile": "/home/mbenjamin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mbenjamin", "validate_certs": true} ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANarZ+C62DPffRIhaCuCtXLChrfqwXgWqGHOLLkmPTtZ3/EE8ZiN+LY1ZcmP939AxXi8Oe+9LepwAdsoQKZctlC7XToyhjIk+rVSPyn3zl3BTC2AxfCKq2vqj65a1sc35Whrw7OaUa2fr5iHSazP6HuG01DhJ9l8HJsLKmiwWChZAAAAFQCNmunc7BNLWrqyrYwINTfscowkbQAAAIEAyYERcunZYJAZ9Eno18kM+ylq0jcQmnPqyIvssWXTttUZnaGQvbVxABeyr6ZKuUyZQfyetofquOGBKPBuzV/7posl5ShDuAcxbS1zhrmk9glTA1yFLvznDdWdLOWG06tjOc78G2AmldKHCdPsmgh2D7WyRr4QY4rcCvwKfZ6+eEYAAACBAJZrOlOlygfPZ4ZB0+EbdYACz1+L/iYNNAOBp51TRutNceLVrNJNOVdmCMLuF0A1go4kPSD/05aJOkOyW86gYLA5DXaW3961+kPInOFAe8OTyQYE7221ThYHRnCfVFjdknuCDomj69wk4SsR+ZPXvSXs1jSJWKhLfUSe9n4bmw7w anet2@secant.private.linuxbox.com\nssh-rsa AAAAB3NzaC1yc2EAAAAD 2021-02-10T10:53:41.013 INFO:teuthology.task.ansible.out:AQABAAABAQC5CMht/crEocrHqGjf0sxE9AhNcW3XBxwlniJwaBJLhekLQEQJoxAy2xUUmPYb2/e7rdmTBxzQipUaAGWVk1FHANac4cbgXzNT+9cnF7EIRkPa/x9jaKNiU0ozMhu/GOoeDXMrHIDgVGWPF3Ag/r17ORpYnuFTjsqoCWyh+mqskV+lmAp04rv2tvqouHtEs6gLebKNtB8c6twhrBKF62Ie+rSr3r7hQaqiZitHbg/isbU5RMBl6H48wHAVSK2/uTfMorbnuEYRG8HvlnPRvQHscJYKiZSt/G7XqZ3x/7c4czEF+OpdQbAgg46vh4fNVMxZk9KHwEkOE06ROuwgAejJ mbenjamin@duel", "key_options": null, "keyfile": "/home/mbenjamin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mbenjamin", "validate_certs": true} 2021-02-10T10:53:41.458 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvG6+jM8aulc5CID/yE6AXPXczxq8tMgY+2sdo/ihWpgDZuFwGL4hwj8nyh5cTqZ2LEa4kRkUUIoH9avLCFN5pDBVvcj1/O06ZGnkq/hV6791UiPF5S5RGmz3MGxv0RDfjA0OLFav7u1s+ZJ6z2W1Tpy/W+g/fMYqbPc6rV+pm3Uo16+BWz/diZH8smqcty4chnnCy6gqOrcAxCZLpeyjX6Ydn3fnTLTAmhoQ9eUz7LjP4ZnFD5zL6jrSBlJ5WNTLevUzvS8eL7RuP8D3Qfn8ukIzPXG2vs594FhSc/HnvubKricvKMm0P0ki4Z0vM9VruQn7Tbeqdwpi1IfgyiAMX aemerson@work-yubi\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAv8/FJn7hz2WieoQFUGXS0bhwSwL1LCI9kkjWYNkWb8IJPrBetlkr5TsvHEKDPZGF5Sv8LsYA6njPnbOGyAHpYlnv/Gz4Hwq3hLgmXCs6JGWBQv8NS+pjwnG6Nh6JfFI0ANeg652Rk9Qt9uLYeKVKmHXXrVZAD4NMvJGfEvneQzuwwpylUmV80WAdaQw/H0EP4NScZhNhknwAs/+KPFZH+RFoUj/Cb3poIFjwyviIgyal5hjAEsoczCW3hAla/9IyC2VDS027WPiPFmBFaYK5lYYLStBQdRx+6amH76XScLFy0FEPLL5qiN0ClAptvh1oIURgipdXZwaxpf06AXG8qw== aemerson@electric", "key_options": null, "keyfile": "/home/aemerson/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aemers 2021-02-10T10:53:41.459 INFO:teuthology.task.ansible.out:on", "validate_certs": true} ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvG6+jM8aulc5CID/yE6AXPXczxq8tMgY+2sdo/ihWpgDZuFwGL4hwj8nyh5cTqZ2LEa4kRkUUIoH9avLCFN5pDBVvcj1/O06ZGnkq/hV6791UiPF5S5RGmz3MGxv0RDfjA0OLFav7u1s+ZJ6z2W1Tpy/W+g/fMYqbPc6rV+pm3Uo16+BWz/diZH8smqcty4chnnCy6gqOrcAxCZLpeyjX6Ydn3fnTLTAmhoQ9eUz7LjP4ZnFD5zL6jrSBlJ5WNTLevUzvS8eL7RuP8D3Qfn8ukIzPXG2vs594FhSc/HnvubKricvKMm0P0ki4Z0vM9VruQn7Tbeqdwpi1IfgyiAMX aemerson@work-yubi\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAv8/FJn7hz2WieoQFUGXS0bhwSwL1LCI9kkjWYNkWb8IJPrBetlkr5TsvHEKDPZGF5Sv8LsYA6njPnbOGyAHpYlnv/Gz4Hwq3hLgmXCs6JGWBQv8NS+pjwnG6Nh6JfFI0ANeg652Rk9Qt9uLYeKVKmHXXrVZAD4NMvJGfEvneQzuwwpylUmV80WAdaQw/H0EP4NScZhNhknwAs/+KPFZH+RFoUj/Cb3poIFjwyviIgyal5hjAEsoczCW3hAla/9IyC2VDS027WPiPFmBFaYK5lYYLStBQdRx+6amH76XScLFy0FEPLL5qiN0ClAptvh1oIURgipdXZwaxpf06AXG8qw== aemerson@electric", "key_options": null, "keyfile": "/home/aemerson/.ssh/authorized_keys", "manage_dir": true, "path": null, "stat 2021-02-10T10:53:41.461 INFO:teuthology.task.ansible.out:e": "present", "user": "aemerson", "validate_certs": true} skipping: [gibba013.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:53:41.470 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:53:41.471 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:53:41.482 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:53:41.915 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHk8c8wUTTYolOOmMlniNrsjfRyF9pCCnaEQ2DXR73P0hbFMme5Md5cF3PTtPIOHSNkWk/qg7s6gZD8LGyFmloO42RBgLsW1qM8RWTDbPy5EQdrjxL9byVzBBCDoQPIS2M+0XnhYa6CBj38Sw8XPlS41CebAJEOQeyLG2I6CAI0rTij6gsqDfALbIezd/eWvmbksKYM1Lb4Gvj+k/OL/xDZzIZHF/P+mK+trY45gsUHxKZamkeHyAT4f3kiNJA+UwzTWGhafy6UE7yGqhZjt9xOZSQGuwroL7EtAzntxayPRNxQPfGDV+fYYf0QBbTtHyJeikPUuw7aR1Z3KwuSmWD dis@zambezi", "key_options": null, "keyfile": "/home/dis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dis", "validate_certs": true} 2021-02-10T10:53:41.916 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHk8c8wUTTYolOOmMlniNrsjfRyF9pCCnaEQ2DXR73P0hbFMme5Md5cF3PTtPIOHSNkWk/qg7s6gZD8LGyFmloO42RBgLsW1qM8RWTDbPy5EQdrjxL9byVzBBCDoQPIS2M+0XnhYa6CBj38Sw8XPlS41CebAJEOQeyLG2I6CAI0rTij6gsqDfALbIezd/eWvmbksKYM1Lb4Gvj+k/OL/xDZzIZHF/P+mK+trY45gsUHxKZamkeHyAT4f3kiNJA+UwzTWGhafy6UE7yGqhZjt9xOZSQGuwroL7EtAzntxayPRNxQPfGDV+fYYf0QBbTtHyJeikPUuw7aR1Z3KwuSmWD dis@zambezi", "key_options": null, "keyfile": "/home/dis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dis", "validate_certs": true} 2021-02-10T10:53:42.365 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANaS5gZUnIp5UL/PM3xOrQiNVxfona7sD1XEXGYvNUyoojPWWAvdQRwdA/Ay8JsAjKeWUt0IwfWaPfbEoXU1TKAI+yyv5WzQUZAKmBFuYM0CUQnKrk72Dp6n8P2tnYen0ect22xEiW2+Y0JZ3wfV3IB3BQPoVCC9FQ+P3LHhYii/AAAAFQCgSceiZFeNItwJ2rJeQgVVTQKljwAAAIEA0Gd2VNCbbQ48vAF+m/i/CeQHJ9Dom9IOBN3v8XnSlOhtNzY1pJ0NRzrFQURCCeVQJaywr0et5FCXIDVqTuR4qfCu5XXU1KHpn/AvHstOd93Y0XYMwSXfNszOtpH7w9SEm7uBfKnLQNF843/MHZL4ZIdfxk73YjfMjrYFcvJwvM0AAACBAI9Wi2wG9Rrdbrgk8tqbnKMxxJCYt/Vakn5mQSc/WHqMUn14+Kb7oBtfTsfPFNK952FzU09ptNQvNWTPNMniI8l8lQi2B+Z28XUlFLL76uxJXWko99xFpf/d1kFBL0M7GRGmvvrMfeTCI+WIqh2jBVqbl4o8kjNx4p0VWyoiWeFJ gregf@pudgy\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAwSe7szyqMTqKK3D/JsXIa1PtPf1p8Njk0sMDSsJPA0gBGysmSgstke9fglOgeNIbNOBNDOY3vltuNCgUiQgVWOYhUaVpj+2vtSo 2021-02-10T10:53:42.366 INFO:teuthology.task.ansible.out:AWpXMQbNaWPGKhbxvJWMCLVckLENyoUZUBKLmTvRyKDjjq8CJ1TnfA2Xfo3a796EIOtD6essU3lAVIfIUILGD+HqptiHSB0spUJUVRxxlO2xw9rLYij1eUPRHrr9xWvDamr5JtKVtCLOL3M9QlltHieKlbJSCZq/EVxW9NAGA6IiqOhw2E1k59oGZUq4Qg+YLgYkB80Ms7nb1xpjQ8txvY+73xL/8i7Nqmr7ekKJfyTpRe03nCQ== gfarnum@mozart.thompsonian.net\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzqJQfdAKaggPDgSpk8P6QjLeyQYOz183fVSKnqAa5EeaCJozS1dn8z+iphswT3YL6wqIMqVHMfkSUcEeTBwZmvydaghOMJEIQSP1PCzH0V2XprELrEcBC3QTvaElPjjEYgBVUvEw8sL/RIQ7mgt9vxD0NcCDspsz35fmU222KzFxD41PZ5XzHQ3Lumwdax7mcpqrw8Aa5/tgcEqAIu54HYRf1Qmk6+ueMAvpwkX3eiSEtkVvuV1Kxpc3/0mOfJ7cn+eyPX3qUbo4oY76hIYseoUIwEaVE/pNW+pjwDSy4gSOcuxYDEGKzCwZPZeQtRQWzRt3YNafxi5xsByobGLlZw== gfarnum@GF-Macbook.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1ik3LbyKsPiGwdh+vGrKSYLvhMg7OkqMwDNYU5LqWBSotogxyGQJUuD80ZavxmmdXC4qScddHtWOJjtyX1CTSDUe7INpSgGPGnmSExVFpcYRnmxHAB+9M/c9VorUkQ8IOKLmRO/BdYma3Rl9NzhHI73yd4zJlx1K4NuQnhnj0Z4yXO+k44RkBWDdMicBB5lBP0hETaH2Iq1u5Dai3u6N5N9bSPXv2PGKvvU395K0Wts2CxMhOBc7LAixysNzNWLrnG1ghS4jPVG2TIHOElP5lIXUv1Wlkool5rTUWgnVY/7loWkDus2JryDM7hLGFXqgT7HHcpI4gSITqRhWxzsUF gregf@fedoragreg.localdomain", "key_options": null, "keyfile": "/home/gregf/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gregf", "validate_certs": true} ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gregf", "ovpn": "gregf@ 2021-02-10T10:53:42.368 INFO:teuthology.task.ansible.out:kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANaS5gZUnIp5UL/PM3xOrQiNVxfona7sD1XEXGYvNUyoojPWWAvdQRwdA/Ay8JsAjKeWUt0IwfWaPfbEoXU1TKAI+yyv5WzQUZAKmBFuYM0CUQnKrk72Dp6n8P2tnYen0ect22xEiW2+Y0JZ3wfV3IB3BQPoVCC9FQ+P3LHhYii/AAAAFQCgSceiZFeNItwJ2rJeQgVVTQKljwAAAIEA0Gd2VNCbbQ48vAF+m/i/CeQHJ9Dom9IOBN3v8XnSlOhtNzY1pJ0NRzrFQURCCeVQJaywr0et5FCXIDVqTuR4qfCu5XXU1KHpn/AvHstOd93Y0XYMwSXfNszOtpH7w9SEm7uBfKnLQNF843/MHZL4ZIdfxk73YjfMjrYFcvJwvM0AAACBAI9Wi2wG9Rrdbrgk8tqbnKMxxJCYt/Vakn5mQSc/WHqMUn14+Kb7oBtfTsfPFNK952FzU09ptNQvNWTPNMniI8l8lQi2B+Z28XUlFLL76uxJXWko99xFpf/d1kFBL0M7GRGmvvrMfeTCI+WIqh2jBVqbl4o8kjNx4p0VWyoiWeFJ gregf@pudgy\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAwSe7szyqMTqKK3D/JsXIa1PtPf1p8Njk0sMDSsJPA0gBGysmSgstke9fglOgeNIbNOBNDOY3vltuNCgUiQgVWOYhUaVpj+2vtSoAWpXMQbNaWPGKhbxvJWMCLVckLENyoUZUBKLmTvRyKDjjq8CJ1TnfA2Xfo3a796EIOtD6essU3lAVIfIUILGD+HqptiHSB0spUJUVRxxlO2xw9rLYij1eUPRHrr9xWvDamr5JtKVtCLOL3M9QlltHieKlbJSCZq/EVxW9NAGA6IiqOhw2E1k59oGZUq4Qg+YLgYkB80Ms7nb1xpjQ8txvY+73xL/8i7Nqmr7ekKJfyTpRe03nCQ== gfarnum@mozart.thompsonian.net\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzqJQfdAKaggPDgSpk8P6QjLeyQYOz183fVSKnqAa5EeaCJozS1dn8z+iphswT3YL6wqIMqVHMfkSUcEeTBwZmvydaghOMJEIQSP1PCzH0V2XprELrEcBC3QTvaElPjjEYgBVUvEw8sL/RIQ7mgt9vxD0NcCDspsz35fmU222KzFxD41PZ5XzHQ3Lumwdax7mcpqrw8Aa5/tgcEqAIu54HYRf1Qmk6+ueMAvpwkX3eiSEtkVvuV1Kxpc3/0mOfJ7cn+eyPX3qUbo4oY76hIYseoUIwEaVE/pNW+pjwDSy4gSOcuxYDEGKzCwZPZeQtRQWzRt3YNafxi5xsByobGLlZw== gfarnum@GF-Macbook.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1ik3LbyKsPiGwdh 2021-02-10T10:53:42.369 INFO:teuthology.task.ansible.out:+vGrKSYLvhMg7OkqMwDNYU5LqWBSotogxyGQJUuD80ZavxmmdXC4qScddHtWOJjtyX1CTSDUe7INpSgGPGnmSExVFpcYRnmxHAB+9M/c9VorUkQ8IOKLmRO/BdYma3Rl9NzhHI73yd4zJlx1K4NuQnhnj0Z4yXO+k44RkBWDdMicBB5lBP0hETaH2Iq1u5Dai3u6N5N9bSPXv2PGKvvU395K0Wts2CxMhOBc7LAixysNzNWLrnG1ghS4jPVG2TIHOElP5lIXUv1Wlkool5rTUWgnVY/7loWkDus2JryDM7hLGFXqgT7HHcpI4gSITqRhWxzsUF gregf@fedoragreg.localdomain", "key_options": null, "keyfile": "/home/gregf/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gregf", "validate_certs": true} 2021-02-10T10:53:42.807 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDgS5EfJyOxBkl/TtJikfb38rm6muQo7MwYufkLSBw9W2jFzt/ZMbVXyVsdZM4I6hWfZm1/q60UcER4yoA1xwYGOTmbFXrLUqqKxKTnCdyejuQ3miB7aiuisWO4vjb2+ppBwL9Mx775DPeQJehFyFK0qgh33tZO/9LWWYy7NOZdA9L2ei1PgMWa0VBeijcDR5GQSXzh0Yi+h/Ezu8jevu+NkW2IUJF6zPGjd9M9lAy7KG2dQjqqCX4XWmUS+MHJS6f9bsC5i39GWy29FvX/qFEArOMzuYRIFWE8iza95KSBxptb+Hw8PFBjQ1kTh6nC5ZV2X3ETcoj8fToTcgW3dTRF jdurgin@new-angeles", "key_options": null, "keyfile": "/home/joshd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joshd", "validate_certs": true} 2021-02-10T10:53:42.808 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDgS5EfJyOxBkl/TtJikfb38rm6muQo7MwYufkLSBw9W2jFzt/ZMbVXyVsdZM4I6hWfZm1/q60UcER4yoA1xwYGOTmbFXrLUqqKxKTnCdyejuQ3miB7aiuisWO4vjb2+ppBwL9Mx775DPeQJehFyFK0qgh33tZO/9LWWYy7NOZdA9L2ei1PgMWa0VBeijcDR5GQSXzh0Yi+h/Ezu8jevu+NkW2IUJF6zPGjd9M9lAy7KG2dQjqqCX4XWmUS+MHJS6f9bsC5i39GWy29FvX/qFEArOMzuYRIFWE8iza95KSBxptb+Hw8PFBjQ1kTh6nC5ZV2X3ETcoj8fToTcgW3dTRF jdurgin@new-angeles", "key_options": null, "keyfile": "/home/joshd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joshd", "validate_certs": true} 2021-02-10T10:53:43.245 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCyOWnAdvzh5zR2ts9e8g0wG4wZ0uNr0YJUCncAOTJB0L8xkQU76jWzX3BRNU5Wk8VtLvpJAQ61P7UXyzgGWSO9FHZjD3RDrc2V9uh/lqRKmx1COtDvcart/JwzwHlsWdrYdcXvEaAiFrZJ6x1jJAjIL6tvMO7bqBz4YNruKpGWsQ6p5P/M5iIEPMj1oKanLoULXqReC6eYFTw3LNAXjberPY5GxYwaKS0HuzySRvSblShyA85jawTSlEbsOSnMqqGu5vGJiC415UFQQ5qKBtzrgAjlUgMbaqGTnTmBYn7L751gIWRetgWqwuHgfIe2ZGM3mmh0DX8E03DUbEt4UxyR dzafman@TrustyTahr\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDDtSwGzH7V5jzppggL0+nYlBSM1HqGjkInwxutra5iEyFoBuV73MNSBk3HrvIVGN13+PCdHf4Oy8ywbhC/4yKbxS1NG76d/GU8ijZFuBlEwxwHZpcThpSTuYrNx7FJVp8Yjl45DtD5s8FMI/0YZZhLlIqT6QCTxlTIPXDb6zefEWvAVziE0bmhRpe4B6AOLOGTB9fivQr5YzxhfaSS5Vu4U1x5gmosouSJ+aMFrQWLqa/eoCy8t9PnxdMjlHjHCZ7Qv1JfpaAPw3qOK30GrVeXR/hPN3aFodTuIJpzdynjk5HO6/VKW5+mWBO7pmQqn9LNQEjADXbLh7PG//nOONbH dzafman@ubuntu16", "key_options": null, "keyfile": "/home/davidz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "davidz", "validate_certs": true} 2021-02-10T10:53:43.257 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCyOWnAdvzh5zR2ts9e8g0wG4wZ0uNr0YJUCncAOTJB0L8xkQU76jWzX3BRNU5Wk8VtLvpJAQ61P7UXyzgGWSO9FHZjD3RDrc2V9uh/lqRKmx1COtDvcart/JwzwHlsWdrYdcXvEaAiFrZJ6x1jJAjIL6tvMO7bqBz4YNruKpGWsQ6p5P/M5iIEPMj1oKanLoULXqReC6eYFTw3LNAXjberPY5GxYwaKS0HuzySRvSblShyA85jawTSlEbsOSnMqqGu5vGJiC415UFQQ5qKBtzrgAjlUgMbaqGTnTmBYn7L751gIWRetgWqwuHgfIe2ZGM3mmh0DX8E03DUbEt4UxyR dzafman@TrustyTahr\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDDtSwGzH7V5jzppggL0+nYlBSM1HqGjkInwxutra5iEyFoBuV73MNSBk3HrvIVGN13+PCdHf4Oy8ywbhC/4yKbxS1NG76d/GU8ijZFuBlEwxwHZpcThpSTuYrNx7FJVp8Yjl45DtD5s8FMI/0YZZhLlIqT6QCTxlTIPXDb6zefEWvAVziE0bmhRpe4B6AOLOGTB9fivQr5YzxhfaSS5Vu4U1x5gmosouSJ+aMFrQWLqa/eoCy8t9PnxdMjlHjHCZ7Qv1JfpaAPw3qOK30GrVeXR/hPN3aFodTuIJpzdynjk5HO6/VKW5+mWBO7pmQqn9LNQEjADXbLh7PG//nOONbH dzafman@ubuntu16", "key_options": null, "keyfile": "/home/davidz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "davidz", "validate_certs": true} 2021-02-10T10:53:43.694 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "key": "ecdsa-sha2-nistp521 AAAAE2VjZHNhLXNoYTItbmlzdHA1MjEAAAAIbmlzdHA1MjEAAACFBAHT7CHaceRLB5mJv9SYKog1wBXsavzH3G3WCBEvVyK76JIslDudZm2ycJv66r+VlGaD8sFI5Mb/KKFMKcKesLRX+wHqZg9MlkoJgm7WGoRQ5I0M1Vt7TL3uJkVCaGZHtO7BXqzTx53C1AaJ1HN+DG2MGy5mm6tHwL901UVqG4EzjBnjxA== christinameno@boxtop", "key_options": null, "keyfile": "/home/cmeno/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cmeno", "validate_certs": true} 2021-02-10T10:53:43.705 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "key": "ecdsa-sha2-nistp521 AAAAE2VjZHNhLXNoYTItbmlzdHA1MjEAAAAIbmlzdHA1MjEAAACFBAHT7CHaceRLB5mJv9SYKog1wBXsavzH3G3WCBEvVyK76JIslDudZm2ycJv66r+VlGaD8sFI5Mb/KKFMKcKesLRX+wHqZg9MlkoJgm7WGoRQ5I0M1Vt7TL3uJkVCaGZHtO7BXqzTx53C1AaJ1HN+DG2MGy5mm6tHwL901UVqG4EzjBnjxA== christinameno@boxtop", "key_options": null, "keyfile": "/home/cmeno/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cmeno", "validate_certs": true} 2021-02-10T10:53:44.146 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDkIZyWnJXYv/z3EVeKHatjFUAdaomIMqzvFLrA9VskeM0Ctv8a83Wk3BbYvTFQWALjZjS2HVYzi5p6kTQ4qxXlcO7HxRw0j4otNauilvgIc4Lu5N4pUtYf0piEo+nrXz+WhELfmJ8t1Yx00uGEl9pOF04SOuXtpwwrCFAO6WMCcjDuYhoI2VBKO3W/OWcIuQ8ZY3j4iDdofFOUGyyoYVyOi/HILKjluYB+ues5+wBEkDuoi3MafCnIj8M0O5fU8/JoOTksOu779UN6blPiNdEgnwD00jA+RWYY6b12ZVkiF4rUVGCiiKKUwOOWkRhkA5OEcBgZ8cGWFgGfZM1GJA// ivancich@sleepy", "key_options": null, "keyfile": "/home/ivancich/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ivancich", "validate_certs": true} 2021-02-10T10:53:44.157 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDkIZyWnJXYv/z3EVeKHatjFUAdaomIMqzvFLrA9VskeM0Ctv8a83Wk3BbYvTFQWALjZjS2HVYzi5p6kTQ4qxXlcO7HxRw0j4otNauilvgIc4Lu5N4pUtYf0piEo+nrXz+WhELfmJ8t1Yx00uGEl9pOF04SOuXtpwwrCFAO6WMCcjDuYhoI2VBKO3W/OWcIuQ8ZY3j4iDdofFOUGyyoYVyOi/HILKjluYB+ues5+wBEkDuoi3MafCnIj8M0O5fU8/JoOTksOu779UN6blPiNdEgnwD00jA+RWYY6b12ZVkiF4rUVGCiiKKUwOOWkRhkA5OEcBgZ8cGWFgGfZM1GJA// ivancich@sleepy", "key_options": null, "keyfile": "/home/ivancich/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ivancich", "validate_certs": true} 2021-02-10T10:53:44.586 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsTZI+b02boZv8jVLHQoFlloD2WF/3xT+ZfZxzwddL7Q6eqo7j8i21LAbLAMWXYMyhiClrloDeHfCZ42gG5bqSThF8LoCw+GspF6SahTI0lmptGu69/OeHqqCpW1DnsWB3wWqfi90XbQJrpg3cdRaqOCmkGVJrqfFWzff8Zg9IuojIH/aO9drCTRGa1pHHCyEdW9ZNeYFwrMjfP+wJi/mC/ABAoXyDy0NWT0uIA+36abNg2/ptok90rdan0r72hjFEz/1hpupKWri+VXAepiyDMuSgJm9uJCXDFaNTn3jocdX+sbVtltThQ2OBxe1T60vRLbXx3lvhxxZ1fUkcP/FF wusui@ubuntu\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD2UzyfBk1nZ8SLCtW9UjorMeOo4u9AM/YL8ZI74GdprR+odERZR17/QfwT5vmLVNa51+CQt1HcGodlQYfR/8n/OUl/gJyVagBuU2zgmLPO3tOor5RBfms5MZ2VZ9Vm3+WrCO/vrAq/5HVyGbQsKiQHXkIRSL2NIgfKRTOVTNmuTSXtZNDgdWGIBmcOFQv+Im9CRopoI8ZtJLQj7xfxtGtUas7FbWTcf0o92+xvWuGWq5WQgmZ8MUIfvBLYLmicuVfsWjZ1YsC1PNhuhkwrW7MhaD71Ej3ll243tG1QD7eAmo+psIp8phlU5s/5FeA5y5OzDZ2T963oHGhq018TYO0b wusui@teuthology", "key_options": null, "keyfile": "/home/wusui/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "wusui", "validate_certs": true} 2021-02-10T10:53:44.609 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsTZI+b02boZv8jVLHQoFlloD2WF/3xT+ZfZxzwddL7Q6eqo7j8i21LAbLAMWXYMyhiClrloDeHfCZ42gG5bqSThF8LoCw+GspF6SahTI0lmptGu69/OeHqqCpW1DnsWB3wWqfi90XbQJrpg3cdRaqOCmkGVJrqfFWzff8Zg9IuojIH/aO9drCTRGa1pHHCyEdW9ZNeYFwrMjfP+wJi/mC/ABAoXyDy0NWT0uIA+36abNg2/ptok90rdan0r72hjFEz/1hpupKWri+VXAepiyDMuSgJm9uJCXDFaNTn3jocdX+sbVtltThQ2OBxe1T60vRLbXx3lvhxxZ1fUkcP/FF wusui@ubuntu\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD2UzyfBk1nZ8SLCtW9UjorMeOo4u9AM/YL8ZI74GdprR+odERZR17/QfwT5vmLVNa51+CQt1HcGodlQYfR/8n/OUl/gJyVagBuU2zgmLPO3tOor5RBfms5MZ2VZ9Vm3+WrCO/vrAq/5HVyGbQsKiQHXkIRSL2NIgfKRTOVTNmuTSXtZNDgdWGIBmcOFQv+Im9CRopoI8ZtJLQj7xfxtGtUas7FbWTcf0o92+xvWuGWq5WQgmZ8MUIfvBLYLmicuVfsWjZ1YsC1PNhuhkwrW7MhaD71Ej3ll243tG1QD7eAmo+psIp8phlU5s/5FeA5y5OzDZ2T963oHGhq018TYO0b wusui@teuthology", "key_options": null, "keyfile": "/home/wusui/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "wusui", "validate_certs": true} 2021-02-10T10:53:45.049 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCw3MhMNDYjkiQADoC1mg9ga0Yj174FHIFt4RQIC8qK+6cB0N3CM5R4JwP8pR4tYkVFWmnYRScYK5VVdC873Ac2BzYxmse97zLJTbUnYWTdY+OFi1FUozsy3PwtiNHSd48Z/PBvkinQIey2QlMKaW3/yz9Of/Yfm4prxg9lBsd6/tOOFj3AsqIQ1OVhMfgeVEkdCL7AAbEYyMxePZ3UFrDwj05HDLgmFzx1xsNJFOuBwLfU/QNHoXJQrcVy0rZq9fB5rg74iz1JZs0uDWocQQK4Nd3Rv+wRGBuew2fmkrS2bQWXzMoSL+/i7pbf74LKcbWAE0wXEImqQeREHxq4ZZ5J yuriweinstein@Yuris-MacBook-Pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJUmq7btpBMjjODHXp1888xoBuX+rhiZg3ufuo69RgDxUMymJn3CPJ/5NGXoZ3SGAWKk0w0Dx6K/VO4y92mAV+cO81y8wZR4J3fPV3IbWRjN0X3IFoAvJi5n9v5hpG7z3RZCGmQnOZ99m1i9dge/3S8NpuPLIvYkSo9z00d+mNP6D8rGnqOmEBfEhUQXLeH7rbPwaMWPgB/hrpiPCyLR/+sGC5bq9PAsKnocqZ12cnvCtYxpnA7et6nL15ftEpUrKYAmRDxotuoUcsllP66GSszjbrgN6OBNDSqpGt/+RSJJwzZEGQ073gJuMveKy9tvD/PyEttjvVKL2KsBuXGSuL ubuntu@yw\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmXWgRRes8bzekTH2fTcxdAAquFn1SjPhws1Z9C 2021-02-10T10:53:45.051 INFO:teuthology.task.ansible.out:kxNaqKe25XnzPehqrhg9etouUwvORQHdDJdTlmhTSYtn588Omq+1bvniab0KzAYgGZsWdPma/3E9qF7ChXIMik9Umip0drzYTaexCuC/Nu46brTxlnL0vaxnnVjDwmHC0CEJs4qN4e2wSlsA0uBDRWjxWFMdTUgSzII5xasHypUYZ0y3MNdbfY+SVF69DEDH8EOQMGYm8JQXNJFloQvmvLUFqT/63FsdpzRVTph4A8hvI1f8i6ULb7fPQN5ZS62wdXd/KvrBXrCvhOi7R63RM60mkMRXm1XajTGmCKiikfiqgTr yuriw@Yuris-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC41kRr6B/RC32DrQKRByalH832H6BpCQF9XdTvgYi4+LJt+kAuIGLDF4xm4uApTmWRrLspWm0Kksc0ger03uNx5qJqMEUnJvjxJnSDC0/XyxWapVxIviFkvv0BjbhVNOOaMU55ywrZ9hHgMdaFwuMuCOaB7f9Tm6IUXm2k82OXueeIG23Kziq+IQBysWOjvNUy/LLpl7l4+xOkhNUupUiQCqHvFH98VUdYWkHeyamafGVZmYbFLrOXgmviKo0u8LPxcT7psKLk0yFG7fG0DSak6DDdKJqb+G/NqMSPxRSAI4FbXQ6UOZb32Ozr1Sql9yfOXEYWZuatn7W5F8DjSv3p yuriw@yuriw-RH", "key_options": null, "keyfile": "/home/yuriw/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuriw", "validate_certs": true} ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, 2021-02-10T10:53:45.052 INFO:teuthology.task.ansible.out:"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCw3MhMNDYjkiQADoC1mg9ga0Yj174FHIFt4RQIC8qK+6cB0N3CM5R4JwP8pR4tYkVFWmnYRScYK5VVdC873Ac2BzYxmse97zLJTbUnYWTdY+OFi1FUozsy3PwtiNHSd48Z/PBvkinQIey2QlMKaW3/yz9Of/Yfm4prxg9lBsd6/tOOFj3AsqIQ1OVhMfgeVEkdCL7AAbEYyMxePZ3UFrDwj05HDLgmFzx1xsNJFOuBwLfU/QNHoXJQrcVy0rZq9fB5rg74iz1JZs0uDWocQQK4Nd3Rv+wRGBuew2fmkrS2bQWXzMoSL+/i7pbf74LKcbWAE0wXEImqQeREHxq4ZZ5J yuriweinstein@Yuris-MacBook-Pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJUmq7btpBMjjODHXp1888xoBuX+rhiZg3ufuo69RgDxUMymJn3CPJ/5NGXoZ3SGAWKk0w0Dx6K/VO4y92mAV+cO81y8wZR4J3fPV3IbWRjN0X3IFoAvJi5n9v5hpG7z3RZCGmQnOZ99m1i9dge/3S8NpuPLIvYkSo9z00d+mNP6D8rGnqOmEBfEhUQXLeH7rbPwaMWPgB/hrpiPCyLR/+sGC5bq9PAsKnocqZ12cnvCtYxpnA7et6nL15ftEpUrKYAmRDxotuoUcsllP66GSszjbrgN6OBNDSqpGt/+RSJJwzZEGQ073gJuMveKy9tvD/PyEttjvVKL2KsBuXGSuL ubuntu@yw\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmXWgRRes8bzekTH2fTcxdAAquFn1SjPhws1Z9CkxNaqKe25XnzPehqrhg9etouUwvORQHdDJdTlmhTSYtn588Omq+1bvniab0KzAYgGZsWdPma/3E9qF7ChXIMik9Umip0drzYTaexCuC/Nu46brTxlnL0vaxnnVjDwmHC0CEJs4qN4e2wSlsA0uBDRWjxWFMdTUgSzII5xasHypUYZ0y3MNdbfY+SVF69DEDH8EOQMGYm8JQXNJFloQvmvLUFqT/63FsdpzRVTph4A8hvI1f8i6ULb7fPQN5ZS62wdXd/KvrBXrCvhOi7R63RM60mkMRXm1XajTGmCKiikfiqgTr yuriw@Yuris-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC41kRr6B/RC32DrQKRByalH832H6BpCQF9XdTvgYi4+LJt+kAuIGLDF4xm4uApTmWRrLspWm0Kksc0ger03uNx5qJqMEUnJvjxJnSDC0/XyxWapVxIviFkvv0BjbhVNOOaMU55ywrZ9hHgMdaFwuMuCOaB7f9Tm6IUXm2k82OXueeIG23Kziq+IQBysWOjvNUy/LLpl7l4+xOkhNUupUiQCqHvFH98VUdYWkHeyamafGVZmYbFLrOXgmviKo0u8LPxcT7psKLk0yFG7fG0DSak6DDdKJqb+G/NqMSPxRSAI4FbXQ6UOZb32Ozr1Sql9yfOXEYWZuatn7W5F8DjSv3p yuriw@yuriw-RH", "key_options": null, "keyfile": "/home/yuriw/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuriw", "validate_certs": true} 2021-02-10T10:53:45.493 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxrS30JmTZbnBd73s9PGZYxCkWH3u4En9Q0bYJan45W+j+y5vMzaNUVsp8We1tphqPQ9B0ygi80OxbHWu5gyv4g+M/JFtrnxv3QgBCYWeBWfh652Vbdlk1thl6btFnHUhVMSn5tkdnFxyVDJHIROoVXkHRaCNAUUy7tB60IhPAMomNcCGu9R8+opONFLXKTIfkYQ1cGbysr90DMs8ASEkvxdv40hIWhK4ufvL7CFFtgXuoC0c1WedBFOkr5LDegzL13xN97K9o5J8CAU0MUKk6H8KNKbQuNq1zANwymMFY599GIlxwai8iluJgfOObDRdCDmKM2f9RAEUXQM7oYoH7 tamil@tamil-VirtualBox\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvwiWVhNLeOLg72cpVIwbUYnmFh+Cqf/mE7Rc/Xci91on3GPrq++dC8t2GhTB5Zyq46EE27SmUkiZpHRnz7xwiJKM/tTgNEkffXDK3j0pA2E22iqV9FpfSA9Zdi1Puq7CPT7/rix4rF3SPvMtg/R1fkyornLFdzhJiMExCKJXGwf+QsKk4NVnr+nO3EEkKOoP1wio1bceZ53rCt9Tyu8NawAB2nL2fvtBPQDAriQ8bLX69XLbOfhFI6GJo815/j6CQxNmvA0l5ABJc8+A71jZmndzumFRgqBuhiCM5ohsGR0JGahUQjFEA4quNTx8EWTGRsqhAYszzpqJzuN+Xb6tJ tamil@macbookair\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEPzEyDGI7poyXVPG70vgQpvlU/lTt6Vvv5CoHWJoSrHklDdJEqekhCB87BX4DTrqbb5xoM7yfxle/z9UzgVwSUWqsPdZHA4EbrNFRKFxi/gc2YrygVSjumTyzf9WJJ5vCZvTQexOj7rjWVnJkjXHKMXZXIAQN/bJf6J21t+4A3XLrl30/4XXyq664NONUNjBHxHtlbS2LM6Th/hK4B23oj2BugX2j/50vWgd9H36bPr8+JGskOi6MH2ndZv5pYtLznFzQmsbixjaN8kCuyQdlRCOndPEFoHFhDla2MeB+h3XQLDXW7J1TR2URFSfcNq5ndDS4i0b+t75Fjhb7AtLT tamil@ubuntu-delldesktop\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6tJ9y3solDCzFdXcczO8GJEwMqMYXWU5kuMz0rr/US6F72j0spSNmwGemMOZE9s7KtHgior/Ufw8/vMRkRhoBYB 2021-02-10T10:53:45.495 INFO:teuthology.task.ansible.out:0C5KT9utGNqrtDdvH98h43XnKZ3K/yaHv/T+u2EYWNgHtOXMtgIVFnGwSD7RzF6TbwSfurprTV6ZUyyOxRcmHnkmwpbNwiwIoXB03k2PITdYCC3lEBEY0OC+vAW/E3+bFbV3RSEA0Cm31jmMyZyPeYiXiEnPXUIClsoDj34NwJTIgctwIsqM4eYUBg3pHliWYKEWDs55HZYoj3W/6Jw7MASzyAYFSmP8rQu8iS7tO1iEBRGOXFNxHirGQ1WqUn tmuthamizhan@MacBook-Air.local", "key_options": null, "keyfile": "/home/tamil/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tamil", "validate_certs": true} 2021-02-10T10:53:45.505 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxrS30JmTZbnBd73s9PGZYxCkWH3u4En9Q0bYJan45W+j+y5vMzaNUVsp8We1tphqPQ9B0ygi80OxbHWu5gyv4g+M/JFtrnxv3QgBCYWeBWfh652Vbdlk1thl6btFnHUhVMSn5tkdnFxyVDJHIROoVXkHRaCNAUUy7tB60IhPAMomNcCGu9R8+opONFLXKTIfkYQ1cGbysr90DMs8ASEkvxdv40hIWhK4ufvL7CFFtgXuoC0c1WedBFOkr5LDegzL13xN97K9o5J8CAU0MUKk6H8KNKbQuNq1zANwymMFY599GIlxwai8iluJgfOObDRdCDmKM2f9RAEUXQM7oYoH7 tamil@tamil-VirtualBox\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvwiWVhNLeOLg72cpVIwbUYnmFh+Cqf/mE7Rc/Xci91on3GPrq++dC8t2GhTB5Zyq46EE27SmUkiZpHRnz7xwiJKM/tTgNEkffXDK3j0pA2E22iqV9FpfSA9Zdi1Puq7CPT7/rix4rF3SPvMtg/R1fkyornLFdzhJiMExCKJXGwf+QsKk4NVnr+nO3EEkKOoP1wio1bceZ53rCt9Tyu8NawAB2nL2fvtBPQDAriQ8bLX69XLbOfhFI6GJo815/j6CQxNmvA0l5ABJc8+A71jZmndzumFRgqBuhiCM5ohsGR0JGahUQjFEA4quNTx8EWTGRsqhAYszzpqJzuN+Xb6tJ tamil@macbookair\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEPzEyDGI7poyXVPG70vgQpvlU/lTt6Vvv5CoHWJoSrHklDdJEqekhCB87BX4DTrqbb5xoM7yfxle/z9UzgVwSUWqsPdZHA4EbrNFRKFxi/gc2YrygVSjumTyzf9WJJ5vCZvTQexOj7rjWVnJkjXHKMXZXIAQN/bJf6J21t+4A3XLrl30/4XXyq664NONUNjBHxHtlbS2LM6Th/hK4B23oj2BugX2j/50vWgd9H36bPr8+JGskOi6MH2ndZv5pYtLznFzQmsbixjaN8kCuyQdlRCOndPEFoHFhDla2MeB+h3XQLDXW7J1TR2URFSfcNq5ndDS4i0b+t75Fjhb7AtLT tamil@ubuntu-delldesktop\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6tJ9y3solDCzFdXcczO8GJEwMqMYXWU5kuMz0rr/US6F72j0spSNmwGemMOZE9s7KtHgior/Ufw8/vMRkRhoBYB 2021-02-10T10:53:45.506 INFO:teuthology.task.ansible.out:0C5KT9utGNqrtDdvH98h43XnKZ3K/yaHv/T+u2EYWNgHtOXMtgIVFnGwSD7RzF6TbwSfurprTV6ZUyyOxRcmHnkmwpbNwiwIoXB03k2PITdYCC3lEBEY0OC+vAW/E3+bFbV3RSEA0Cm31jmMyZyPeYiXiEnPXUIClsoDj34NwJTIgctwIsqM4eYUBg3pHliWYKEWDs55HZYoj3W/6Jw7MASzyAYFSmP8rQu8iS7tO1iEBRGOXFNxHirGQ1WqUn tmuthamizhan@MacBook-Air.local", "key_options": null, "keyfile": "/home/tamil/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tamil", "validate_certs": true} 2021-02-10T10:53:45.961 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDYGVqKZgLx2LJ49x0kGo/Z5r8MVefSUCYhhrK/DK8ht+YK1AqLXErjAedgpVz0WuwirRjh5YkiXGMYiOdkMUYWGk5nwCHEKnnCpne7Xh1OOqge/QGHlUigWVH8VIHBsZ+wTW25mqOrgDSGB8+Qk7Dr2o1sLGUHcx7vITDWOrer4KjF2+H5TQH/7irUVORu6fzeGKtGyOrTwQ+ZT1+fuhfznDULFGyvKGdcXxxNOg+21Fssk2nCNTlgRCGjHSrfelfuMT7QEt+9puWd7MWXYQg7xx8jkJGMpEQRFxJTxnfUlUXf3x2lc9Q/SvKfpVMtc0N+bk2F9+euyvRvjtn4LJNx jowilkin@jowilkin.wilkins", "key_options": null, "keyfile": "/home/jowilkin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jowilkin", "validate_certs": true} 2021-02-10T10:53:45.962 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDYGVqKZgLx2LJ49x0kGo/Z5r8MVefSUCYhhrK/DK8ht+YK1AqLXErjAedgpVz0WuwirRjh5YkiXGMYiOdkMUYWGk5nwCHEKnnCpne7Xh1OOqge/QGHlUigWVH8VIHBsZ+wTW25mqOrgDSGB8+Qk7Dr2o1sLGUHcx7vITDWOrer4KjF2+H5TQH/7irUVORu6fzeGKtGyOrTwQ+ZT1+fuhfznDULFGyvKGdcXxxNOg+21Fssk2nCNTlgRCGjHSrfelfuMT7QEt+9puWd7MWXYQg7xx8jkJGMpEQRFxJTxnfUlUXf3x2lc9Q/SvKfpVMtc0N+bk2F9+euyvRvjtn4LJNx jowilkin@jowilkin.wilkins", "key_options": null, "keyfile": "/home/jowilkin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jowilkin", "validate_certs": true} 2021-02-10T10:53:46.393 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDP/yVttBEs7mpVfJL+reXb/BfB7DsLrA5ArVqEJjySi3NOz3e/wE8KRWbHD1UdraAU3P8b96GdAQxGOs4n0muOMejmaXWZCP3HkRL+rcd+03ZI1lb6ga9XjBARZfb2h+kg5jTJp/Ll6mwAYbSXvQ4MuDQzwGRDX5Q5Xme7OZ4Wb+vBmmkihHydySyP9YXbWWsBw4jvZCFSKMGsUVy/HIt1coZx/PyCG/K2Kl0nrE5ux2QqfdTIpzYo4c3y6uMIHDvt29lRmk3Gnh2//U5eN8mmRRKJlkvE8DKy3VYsl0JsP9pTWUZEYgV0EX251WTO0akQ2bKuszqd9Ai04RWnXUu5 brad@dhcp-1-165.bne.redhat.com", "key_options": null, "keyfile": "/home/bhubbard/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bhubbard", "validate_certs": true} 2021-02-10T10:53:46.404 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDP/yVttBEs7mpVfJL+reXb/BfB7DsLrA5ArVqEJjySi3NOz3e/wE8KRWbHD1UdraAU3P8b96GdAQxGOs4n0muOMejmaXWZCP3HkRL+rcd+03ZI1lb6ga9XjBARZfb2h+kg5jTJp/Ll6mwAYbSXvQ4MuDQzwGRDX5Q5Xme7OZ4Wb+vBmmkihHydySyP9YXbWWsBw4jvZCFSKMGsUVy/HIt1coZx/PyCG/K2Kl0nrE5ux2QqfdTIpzYo4c3y6uMIHDvt29lRmk3Gnh2//U5eN8mmRRKJlkvE8DKy3VYsl0JsP9pTWUZEYgV0EX251WTO0akQ2bKuszqd9Ai04RWnXUu5 brad@dhcp-1-165.bne.redhat.com", "key_options": null, "keyfile": "/home/bhubbard/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bhubbard", "validate_certs": true} 2021-02-10T10:53:46.845 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAyRDKb+InDsnW4E/S12Pr6rK3wDfUG3j+iw/NsCWwN5xpJEA1/NhVlRzY8/jejc6ddmnNKWbm/UcYVrbo7yClRTGu45g0Qm9lmfvgDgi1k8mwUCy+edbQ/R8FB9cIHHCnUuWzRmE2XGaeDiSVQgiYqq3mOPY/hwe3G7jwFalYPsbmY1jNqpzODmKFMRd1bOwdHEOMTfh0owHWqhhXakU8em30lYraQVYH24FplH2NUBArxmTuctyfweHxx7yH3ZFcXFUpUMSLuVDWb1A7fMiL6eW6wiwh+dD+q5iFNvuioZITP2EjIsb9nS+eM9bOJZuZ+5pln2EsHee27LjunbwgWw== yehudasa@fatty\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDF3PxJYgI3tS7SG+RnUvxMxr0/XwBAio0oiP4AC+o8rUYYyPd/RU27TR1kphbuJhQWVtaZ6kFSBQeNQUxv6rb8hmMzvlP1FDEoGkjaOHOuYEc2DFkkEAfaR2AlNKn3pUQgLE00TLHctCgd57Cn2Ppt8xEi/vTuo0gX3xjWOL4N9QDElCXKNdxdyyJ/RNb/Mid+YxgKKjf8ppmhYsQ+Qqu4YwPzLkfK1Qb3YtR6Hbjb4jxC7C1Z13h0vUgHe7qSmUCYNo9eC+lEgNmmnLb2sXJaUUJM8MEex6v+4muWM9hV0cq/Wplrx8fhMWMa978sxH8yVkE3Svqc/K4ZWN59l3xT yehudasa@yehudasa-XPS-L401X\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDvplQk8xkSXF03uSJh3H4zyZHRMjvgUJBgGN27Kb7gAlIQv306wPKIprOwp+jDhEcPnEWBMRJMUJuBi0vEiqmDHt8mg8ZYcs+Gi7MbUgBokILdJuSG89KAgq7gBCuIhYbSg35pfvXvIZswiXvySqMrJNB495GYj/FrcWEBfQMuedMSkIfNzqPV9yR5YRh8fnSs3fjCzV96vzu7O4AzgwICbM6XmqVLXPax/32Hq0dz7sAawlQFCJYPHzVzjwmih46T 2021-02-10T10:53:46.847 INFO:teuthology.task.ansible.out:RJnXrxewsRhk9GwKfI4SWaFIRdOdadGFSLgdZ+2iGE7hTZ3sRgkLOj+4zKa+xTbtjjXfw5up7J3hAmrK15Tz yehuda@yehuda-940X3G\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfNxvAw6cb33jg36vw8kJu48/BttmFJ4BFeanYqwRgtWZrKpAAGHEgCiNeyIttMrI3t7Spcdyp6dy6n0P3WUFVgHWEYtGqXwXqDC9UXISkUJVVDZoEynpGC92RghhzH53+7jX1Ytpr30r/xb1ktryTPCoEJRqyrg2j1b167i6PV5tuJCA+77gW5LWBmMXJCprjnfnJ92Gp8ZewBB6gL1x+3RFkMxGsibL0Qw1CTKfyAHNYc/jos6YiL0b42Ha+NmKwvzZ1GgIleRMleisSOyKPUlPdR7UnmtEXa9bl3Gc/hc7Zt+v4iUypr2WL0X4Yl7rJNH/AAd/Z6jPkAX9nHYzh yehudasa@yehudasa-desktop", "key_options": null, "keyfile": "/home/yehudasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yehudasa", "validate_certs": true} 2021-02-10T10:53:46.857 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAyRDKb+InDsnW4E/S12Pr6rK3wDfUG3j+iw/NsCWwN5xpJEA1/NhVlRzY8/jejc6ddmnNKWbm/UcYVrbo7yClRTGu45g0Qm9lmfvgDgi1k8mwUCy+edbQ/R8FB9cIHHCnUuWzRmE2XGaeDiSVQgiYqq3mOPY/hwe3G7jwFalYPsbmY1jNqpzODmKFMRd1bOwdHEOMTfh0owHWqhhXakU8em30lYraQVYH24FplH2NUBArxmTuctyfweHxx7yH3ZFcXFUpUMSLuVDWb1A7fMiL6eW6wiwh+dD+q5iFNvuioZITP2EjIsb9nS+eM9bOJZuZ+5pln2EsHee27LjunbwgWw== yehudasa@fatty\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDF3PxJYgI3tS7SG+RnUvxMxr0/XwBAio0oiP4AC+o8rUYYyPd/RU27TR1kphbuJhQWVtaZ6kFSBQeNQUxv6rb8hmMzvlP1FDEoGkjaOHOuYEc2DFkkEAfaR2AlNKn3pUQgLE00TLHctCgd57Cn2Ppt8xEi/vTuo0gX3xjWOL4N9QDElCXKNdxdyyJ/RNb/Mid+YxgKKjf8ppmhYsQ+Qqu4YwPzLkfK1Qb3YtR6Hbjb4jxC7C1Z13h0vUgHe7qSmUCYNo9eC+lEgNmmnLb2sXJaUUJM8MEex6v+4muWM9hV0cq/Wplrx8fhMWMa978sxH8yVkE3Svqc/K4ZWN59l3xT yehudasa@yehudasa-XPS-L401X\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDvplQk8xkSXF03uSJh3H4zyZHRMjvgUJBgGN27Kb7gAlIQv306wPKIprOwp+jDhEcPnEWBMRJMUJuBi0vEiqmDHt8mg8ZYcs+Gi7MbUgBokILdJuSG89KAgq7gBCuIhYbSg35pfvXvIZswiXvySqMrJNB495GYj/FrcWEBfQMuedMSkIfNzqPV9yR5YRh8fnSs3fjCzV96vzu7O4AzgwICbM6XmqVLXPax/32Hq0dz7sAawlQFCJYPHzVzjwmih46T 2021-02-10T10:53:46.858 INFO:teuthology.task.ansible.out:RJnXrxewsRhk9GwKfI4SWaFIRdOdadGFSLgdZ+2iGE7hTZ3sRgkLOj+4zKa+xTbtjjXfw5up7J3hAmrK15Tz yehuda@yehuda-940X3G\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfNxvAw6cb33jg36vw8kJu48/BttmFJ4BFeanYqwRgtWZrKpAAGHEgCiNeyIttMrI3t7Spcdyp6dy6n0P3WUFVgHWEYtGqXwXqDC9UXISkUJVVDZoEynpGC92RghhzH53+7jX1Ytpr30r/xb1ktryTPCoEJRqyrg2j1b167i6PV5tuJCA+77gW5LWBmMXJCprjnfnJ92Gp8ZewBB6gL1x+3RFkMxGsibL0Qw1CTKfyAHNYc/jos6YiL0b42Ha+NmKwvzZ1GgIleRMleisSOyKPUlPdR7UnmtEXa9bl3Gc/hc7Zt+v4iUypr2WL0X4Yl7rJNH/AAd/Z6jPkAX9nHYzh yehudasa@yehudasa-desktop", "key_options": null, "keyfile": "/home/yehudasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yehudasa", "validate_certs": true} 2021-02-10T10:53:47.286 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDx37xbP0jYqopygG66LbSMRPoqpGUPzQZ3G8eqJCk2I dang@revan.aa.redhat.com-2015-07-17", "key_options": null, "keyfile": "/home/dang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dang", "validate_certs": true} 2021-02-10T10:53:47.297 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDx37xbP0jYqopygG66LbSMRPoqpGUPzQZ3G8eqJCk2I dang@revan.aa.redhat.com-2015-07-17", "key_options": null, "keyfile": "/home/dang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dang", "validate_certs": true} 2021-02-10T10:53:47.733 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2YyDoeKdaCuJif79AxJAorqpUj33WHbHu/eIWpHju+iIF3f94K0qyEXgs4AwZin9pzfyd8N1aKNT8ZFPss6L0Xk68aO/6tdInq+1lTMMUC5v/LBqsIV2btPTFHrQWA0R10v05aZFvFaSqhHNsZaj98vQV7P+jiSs3yPWqCIWUTQ1VXnCeQnB4R5tob0gpMuonhkKq7vrJPhf6ge9Av+yjL7+8oP5LJqFJ5npDxkwZ0u3L0QQ9ftAT/e7Y7ESpwpgaI3Trl0EwAXFmiLjXEUA+FBbE7ko9kLqXqRjc1C0BdRG9sO0QKBGDBEx1W3w8rCPKgaDlVyqsBT1yvIjqqXDOQ== borix@dhcp-31-190.brq.redhat.com", "key_options": null, "keyfile": "/home/branto/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "branto", "validate_certs": true} 2021-02-10T10:53:47.744 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2YyDoeKdaCuJif79AxJAorqpUj33WHbHu/eIWpHju+iIF3f94K0qyEXgs4AwZin9pzfyd8N1aKNT8ZFPss6L0Xk68aO/6tdInq+1lTMMUC5v/LBqsIV2btPTFHrQWA0R10v05aZFvFaSqhHNsZaj98vQV7P+jiSs3yPWqCIWUTQ1VXnCeQnB4R5tob0gpMuonhkKq7vrJPhf6ge9Av+yjL7+8oP5LJqFJ5npDxkwZ0u3L0QQ9ftAT/e7Y7ESpwpgaI3Trl0EwAXFmiLjXEUA+FBbE7ko9kLqXqRjc1C0BdRG9sO0QKBGDBEx1W3w8rCPKgaDlVyqsBT1yvIjqqXDOQ== borix@dhcp-31-190.brq.redhat.com", "key_options": null, "keyfile": "/home/branto/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "branto", "validate_certs": true} 2021-02-10T10:53:48.183 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDO00mus2XPX/EfUZ/YsKI84TnWj5TbZP7Fs01EkoOTPgNoYysVdf0PwhAooz1qPrv+HUsUMpQT6uNYfCHyqx9rScR+XUlyQuFe7Sjp/dn3y+b1PKa6G4HX7f1eejR3ZaLZXA4R/8oucAXqQS5pDr3Y7prwkrp9cyk8uawBXoDcAlA1SMIPZPrtO1+iZw20NDhixsG7NE5jMFfw3q8xTckZEqBbteybci7X1v4lEYXCVlbSbUhyZ2Af8HEj09JBDOL2bI3yLvj9a5Oj/HtKubyApzZpA7rgeoii2UZQGuSRI1VJktp9YC0zGk5S843xYW92utbALKyQ6DdB2F5gc0DX xiaoxchen@LM-SHC-16501223", "key_options": null, "keyfile": "/home/xiaoxichen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiaoxichen", "validate_certs": true} 2021-02-10T10:53:48.184 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDO00mus2XPX/EfUZ/YsKI84TnWj5TbZP7Fs01EkoOTPgNoYysVdf0PwhAooz1qPrv+HUsUMpQT6uNYfCHyqx9rScR+XUlyQuFe7Sjp/dn3y+b1PKa6G4HX7f1eejR3ZaLZXA4R/8oucAXqQS5pDr3Y7prwkrp9cyk8uawBXoDcAlA1SMIPZPrtO1+iZw20NDhixsG7NE5jMFfw3q8xTckZEqBbteybci7X1v4lEYXCVlbSbUhyZ2Af8HEj09JBDOL2bI3yLvj9a5Oj/HtKubyApzZpA7rgeoii2UZQGuSRI1VJktp9YC0zGk5S843xYW92utbALKyQ6DdB2F5gc0DX xiaoxchen@LM-SHC-16501223", "key_options": null, "keyfile": "/home/xiaoxichen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiaoxichen", "validate_certs": true} 2021-02-10T10:53:48.624 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCz7e2G3Ziaus2yMviKh9s65e+yaqy1+ezs5IC6US90/8WumNAWqC4Yr3ckvCc120r3VwpJWcvaAWCF+4IZeO43lSNLadbCnJONed5YklcUuIJvjC1eDbZTMLJ7h0ijgNXgAFmFq0AV2NHuE2L2VKXk2hsIqDVGjZLUpi7W+ic07xnScEoXXDSk5mhJ6bKmFsUYCC91g2roYs9Ui5fGd+ey28SFOADI7osDLpLkiTF2OzGel93o5qobNZwKR0AxGsBIta17mvf3I3307bGnvugDXpNB0FaS1QZhIzMO4G4LZL9zahCvv4wm5GmuO4F0+Ac7dwFu3KZKtHQe3g6WHuAv ffilz@redhat.com", "key_options": null, "keyfile": "/home/ffilz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ffilz", "validate_certs": true} 2021-02-10T10:53:48.636 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCz7e2G3Ziaus2yMviKh9s65e+yaqy1+ezs5IC6US90/8WumNAWqC4Yr3ckvCc120r3VwpJWcvaAWCF+4IZeO43lSNLadbCnJONed5YklcUuIJvjC1eDbZTMLJ7h0ijgNXgAFmFq0AV2NHuE2L2VKXk2hsIqDVGjZLUpi7W+ic07xnScEoXXDSk5mhJ6bKmFsUYCC91g2roYs9Ui5fGd+ey28SFOADI7osDLpLkiTF2OzGel93o5qobNZwKR0AxGsBIta17mvf3I3307bGnvugDXpNB0FaS1QZhIzMO4G4LZL9zahCvv4wm5GmuO4F0+Ac7dwFu3KZKtHQe3g6WHuAv ffilz@redhat.com", "key_options": null, "keyfile": "/home/ffilz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ffilz", "validate_certs": true} 2021-02-10T10:53:49.066 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpGZK2cWUtE85Bt7fl+kSeiC0lM/EN+H2X3ePGq3LiD4g5R5n6nJsLaJaIZXdJ96G0JSnj07G8Xu1CkVbGDHuVg2O9tnxLhz6tzDwFYNFrQIpALlQwOq8OiwGDDyFkMXQ3JqCCWvyD0HLysOJQIBIlb/o92ZCSAGOyNX1JjrhXF+mlBk1AUW11UeZOeGZQ34MKHapH7T/DuexCz5gfZI/jY9Rg3hr4jb1zGttvAF2eeLJCjqD/Vi4mowYB/Kn/PJhNixeA2sli2i8r6yO/bFmA/PWnfSl+EqD4mUJWb5ATeJ3hyW8qVLOogY6XVu4Sd1VgL7l1RYzXuxJV3Rb1Yw2T joao@sepia\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDV0jbKpbmXW1bhRIOBIdVV1L73qjls7s5pxiW2EMPvZLYzxLBE92e3Vf8bSMWlm1Lyb7a+u5BR85+2c+h2F59sgU1Saek5ebPpFU5qBn8AzYH18iMVSPciByOzwS0NXwWnksckWxUkQgP9bZP/1ORQ7jWZEOHZ5mEQ0oMkP0bfG1cpFrNTxOhDmwbCiQ8dp6KeZaMg4hErALr5Eh/EESxKIkuwQzk9bDbtjPRDu0F2dE0NCjs0k0BXmDA80GksxuOacRxE95K6/er11qst/VqA7o6BfhxJMTXWMdj8NhzIUNjZ8LqLTG6P/XGR24sA2d2WBhlEKRMZNqeEjzmi1REr joao@tagide", "key_options": null, "keyfile": "/home/joao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joao", "validate_certs": true} 2021-02-10T10:53:49.077 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpGZK2cWUtE85Bt7fl+kSeiC0lM/EN+H2X3ePGq3LiD4g5R5n6nJsLaJaIZXdJ96G0JSnj07G8Xu1CkVbGDHuVg2O9tnxLhz6tzDwFYNFrQIpALlQwOq8OiwGDDyFkMXQ3JqCCWvyD0HLysOJQIBIlb/o92ZCSAGOyNX1JjrhXF+mlBk1AUW11UeZOeGZQ34MKHapH7T/DuexCz5gfZI/jY9Rg3hr4jb1zGttvAF2eeLJCjqD/Vi4mowYB/Kn/PJhNixeA2sli2i8r6yO/bFmA/PWnfSl+EqD4mUJWb5ATeJ3hyW8qVLOogY6XVu4Sd1VgL7l1RYzXuxJV3Rb1Yw2T joao@sepia\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDV0jbKpbmXW1bhRIOBIdVV1L73qjls7s5pxiW2EMPvZLYzxLBE92e3Vf8bSMWlm1Lyb7a+u5BR85+2c+h2F59sgU1Saek5ebPpFU5qBn8AzYH18iMVSPciByOzwS0NXwWnksckWxUkQgP9bZP/1ORQ7jWZEOHZ5mEQ0oMkP0bfG1cpFrNTxOhDmwbCiQ8dp6KeZaMg4hErALr5Eh/EESxKIkuwQzk9bDbtjPRDu0F2dE0NCjs0k0BXmDA80GksxuOacRxE95K6/er11qst/VqA7o6BfhxJMTXWMdj8NhzIUNjZ8LqLTG6P/XGR24sA2d2WBhlEKRMZNqeEjzmi1REr joao@tagide", "key_options": null, "keyfile": "/home/joao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joao", "validate_certs": true} 2021-02-10T10:53:49.507 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHhYb/mGgHbEntAN2iPFTV1k8MYDs+jtsJDPE9MWtUZXksdeG8cEfXDBGhiBsprjELsW4+czJBqqESugFMevEysTVKIo5lc95bf1nApxfiRyMnAxb5wQ2OeLY62x26TWJhPu6HMajV+gb8CKjxDobf1Ix4ELbYlL77SOH1VVDtVZ6HiwtYngXAPMONCdI99cuwJlBpv5RjDoLwljOoxEXIh+1Fsvth+iymp0rIPuJx91DzYYgWZFfCLPHRgDy0H7YRdQWUzhrxTyzuT27vsNh5L8HgdowzOu9f2HwGzlTlxli94IJLMBY2359xsqjSswZzAv+BiPDAZMUwJpYvRvSt nhm@clusterfaq.org\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1JAKymiRDyVar5P75cMycxz9u8VsLuNtf/0t1na4ofa4YmanKswPQQz1Am82pvyPqXlBbdIGVneyMrkEoFAk4nTtfmCl7WdtaxnH87Dr9BOLEp+lIPHy7jpS/dcVmKRPB1qCYZ5GNAZ9g+DrXdaeOJdUOpIK49BP1IK8Uw5eBWAx/ae6/12yz+d/6e9kBvO7OUamMSne6Ghe2pRoI8jFOdWMbQy3ofb5wSHZGPNDn0YTGbNWHlfEAcrFmGQSTwohca4SqCJ1NgKnYu4qPVmb3Gbpm+MpUPs8Gy++rQ8wVZ5xWZW/yReIbJCqqVY4vrQzVywQXRucIrB1VyvShuNW3 nhm@metropolis\nssh-dss AAAAB3NzaC1kc3MAAACBAM7BfQAinQmLlkTgt7IIg3nQyyaJRiWA4agfu/HncrcAmepGqPCwy451g1eLMD3AC4QptHnV80Y47YQTyX1VQixIQ6M9zHG6Bo7IXzyqFHFpqnDZN25DaMM4na5bf1CwC9RtR7VWubojatPylOpflkChUBuAGWnzwkz2cxl4Dc47AAAAFQD35l/Lg4eTqZ83eTgJ5lihzHS/awAAAIB/ZdMSE6M0zEvdNPsWY28bvQ0QXN7w1ncVXGachA1JOEFGYvsbHeoywLpSYoiPLV57/wZT2dUzG70u80IQGildZ1WgQNn7C9r2Z3ZUgKmRGyC+PpYEa6nv+U4CHdMViBRXxNrP0bj2CfQvU 2021-02-10T10:53:49.509 INFO:teuthology.task.ansible.out:xUxrHWtOu/da2W23cpSs6Wz8gtz2QAAAIEAvf4vx/32Sy+kbvqOCYz0wu0ItlECN0qOk3gAMvphYMI2sW2ez2+Tsua4eoPRXT10zFxiRTkHQd5FSIvNWXV2iqed6rpwTu5K8vJz2duMIyI+xWLHo4XudtZdNxggkf+RW9JvPOMLMp4sCzBXxpreHBsb1CynndKeYVrgMLybZ28= mark@Marks-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfDOM2v4h6Dk4pMWPQxxHI5rhFtLjkme/4hAp6ki/kJ1XOSzBQdQmD2UPDKvfO9lKO7v12G2Wt+re+0giAcSkfej4+xqV53u7d2pTbMI5U3zohZt8AgESx22DAxj1wBZYuyGO2TRI7DRQMxC47E8EJgcmZBBnMKk1Rz8k71pTmg+CI2vjWx/2x2bup9Mqe4uMA8ubjg2qRlWEqPKIKcHZyrAPdIlpy2FSZ9SPa0mJpdl6aYcFNgonD8BO6A0u2RPO90KmEWrRWu69vRvbbTDiqE0CkJJlAlu3jeFVD8US9vymBHSI0BDFUbukuyG1tAujytqGBRLzxQDU9JOYMP45v nhm@mocha", "key_options": null, "keyfile": "/home/nhm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nhm", "validate_certs": true} 2021-02-10T10:53:49.519 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHhYb/mGgHbEntAN2iPFTV1k8MYDs+jtsJDPE9MWtUZXksdeG8cEfXDBGhiBsprjELsW4+czJBqqESugFMevEysTVKIo5lc95bf1nApxfiRyMnAxb5wQ2OeLY62x26TWJhPu6HMajV+gb8CKjxDobf1Ix4ELbYlL77SOH1VVDtVZ6HiwtYngXAPMONCdI99cuwJlBpv5RjDoLwljOoxEXIh+1Fsvth+iymp0rIPuJx91DzYYgWZFfCLPHRgDy0H7YRdQWUzhrxTyzuT27vsNh5L8HgdowzOu9f2HwGzlTlxli94IJLMBY2359xsqjSswZzAv+BiPDAZMUwJpYvRvSt nhm@clusterfaq.org\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1JAKymiRDyVar5P75cMycxz9u8VsLuNtf/0t1na4ofa4YmanKswPQQz1Am82pvyPqXlBbdIGVneyMrkEoFAk4nTtfmCl7WdtaxnH87Dr9BOLEp+lIPHy7jpS/dcVmKRPB1qCYZ5GNAZ9g+DrXdaeOJdUOpIK49BP1IK8Uw5eBWAx/ae6/12yz+d/6e9kBvO7OUamMSne6Ghe2pRoI8jFOdWMbQy3ofb5wSHZGPNDn0YTGbNWHlfEAcrFmGQSTwohca4SqCJ1NgKnYu4qPVmb3Gbpm+MpUPs8Gy++rQ8wVZ5xWZW/yReIbJCqqVY4vrQzVywQXRucIrB1VyvShuNW3 nhm@metropolis\nssh-dss AAAAB3NzaC1kc3MAAACBAM7BfQAinQmLlkTgt7IIg3nQyyaJRiWA4agfu/HncrcAmepGqPCwy451g1eLMD3AC4QptHnV80Y47YQTyX1VQixIQ6M9zHG6Bo7IXzyqFHFpqnDZN25DaMM4na5bf1CwC9RtR7VWubojatPylOpflkChUBuAGWnzwkz2cxl4Dc47AAAAFQD35l/Lg4eTqZ83eTgJ5lihzHS/awAAAIB/ZdMSE6M0zEvdNPsWY28bvQ0QXN7w1ncVXGachA1JOEFGYvsbHeoywLpSYoiPLV57/wZT2dUzG70u80IQGildZ1WgQNn7C9r2Z3ZUgKmRGyC+PpYEa6nv+U4CHdMViBRXxNrP0bj2CfQvU 2021-02-10T10:53:49.520 INFO:teuthology.task.ansible.out:xUxrHWtOu/da2W23cpSs6Wz8gtz2QAAAIEAvf4vx/32Sy+kbvqOCYz0wu0ItlECN0qOk3gAMvphYMI2sW2ez2+Tsua4eoPRXT10zFxiRTkHQd5FSIvNWXV2iqed6rpwTu5K8vJz2duMIyI+xWLHo4XudtZdNxggkf+RW9JvPOMLMp4sCzBXxpreHBsb1CynndKeYVrgMLybZ28= mark@Marks-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfDOM2v4h6Dk4pMWPQxxHI5rhFtLjkme/4hAp6ki/kJ1XOSzBQdQmD2UPDKvfO9lKO7v12G2Wt+re+0giAcSkfej4+xqV53u7d2pTbMI5U3zohZt8AgESx22DAxj1wBZYuyGO2TRI7DRQMxC47E8EJgcmZBBnMKk1Rz8k71pTmg+CI2vjWx/2x2bup9Mqe4uMA8ubjg2qRlWEqPKIKcHZyrAPdIlpy2FSZ9SPa0mJpdl6aYcFNgonD8BO6A0u2RPO90KmEWrRWu69vRvbbTDiqE0CkJJlAlu3jeFVD8US9vymBHSI0BDFUbukuyG1tAujytqGBRLzxQDU9JOYMP45v nhm@mocha", "key_options": null, "keyfile": "/home/nhm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nhm", "validate_certs": true} 2021-02-10T10:53:49.960 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaetIvR7YgpCq142fBQtOYsJrOMW+Wxu1beNXPJ8qzT5MxD05sdKntVE0DPveUydaeI8/vHeNjXDKN8mt1eO7Lf/rDKHOW/Y/36bUfQAODru9sdNCEJfSPgq1y6+X1ISljvLBXOOl+nQXlx3n5OjThNQbmdthgCT2Fk4hEoK7gn4+ol6foTkTdUXIBam8LuPU6f1KIG6ae1oAVFWLeavAbvmxUTmVl04+hKJWSPnvZxWupaPuiHvsImaQ7GwBjKhzL1wSMly0+pa2zbVYZKkLBA7Fx6fzP7jcEieOOPM1YU0jR5JF+WVzeRFJ1V9n75yOw6S4ef/X/zwWx42cdfx3N nwatkins@martini\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDRivdSX6cbPrdMrjz1oYgn0hb8VQo+QU7wD+AcCQi/CH06ny+2QjSr7bfdpgQ81Wgwqt9+e3KcLD5qdt3tneC8sgSAR5QUQrWKhPejrqdV+3990k8k0KoKIOZ9MGFc1FBv5IpuT/2d535iHJb0ZEg8mFKqKDw71sBEoQT4WElatf5DeVC507hP7GnrhZgtTPcrVddOLkmmqTQtLE1zvo9Y/eel8O2JMImMd6mdVBIcFQ1alU1wV4jLdMpagoPelvVTVeNkGRnOgvvQac5RXtGSMP1R5PoGz+1FgRhDCtdTXRfOdtrlJPtUjYiGPpIoKIUymlvQGjUrrK3WiQIIkIfDtFFu48EN4drjgQpCEQ2D0NMKEmk74mjx3CJDOF1Nj/2Fa1kjECiz4EM7sR8dDG5cW5Y8XsDoomgugM+rsOH9kWFFClHUJmdlNGWLhJnu9UQHLYZjQw3zT0F+FXUsUxKJUVTUdxQx5HBO9QFsSe13NPDnX64+SQ2VVStt8oQAvFO+6260zbux0RVaJM5dTNwnIHD/uI3PeR1z2b+iKhyj/9VP2uBRpGydLw/etQu/lS2K30S+kZIDAp0sqwXH89bdq3AJBipcryrnW1Ov9K5Mfl1UlV95/MivrnOFISWN1F9XR1kuIIYdCkI7hq/FHZ9gd84AjTiERfxRv8VCIHBVEQ== nwatkins@redhat.com", "key_options": null, "keyfile": "/home/nwatkins/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nwatkins", "validate_certs": true} 2021-02-10T10:53:49.961 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaetIvR7YgpCq142fBQtOYsJrOMW+Wxu1beNXPJ8qzT5MxD05sdKntVE0DPveUydaeI8/vHeNjXDKN8mt1eO7Lf/rDKHOW/Y/36bUfQAODru9sdNCEJfSPgq1y6+X1ISljvLBXOOl+nQXlx3n5OjThNQbmdthgCT2Fk4hEoK7gn4+ol6foTkTdUXIBam8LuPU6f1KIG6ae1oAVFWLeavAbvmxUTmVl04+hKJWSPnvZxWupaPuiHvsImaQ7GwBjKhzL1wSMly0+pa2zbVYZKkLBA7Fx6fzP7jcEieOOPM1YU0jR5JF+WVzeRFJ1V9n75yOw6S4ef/X/zwWx42cdfx3N nwatkins@martini\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDRivdSX6cbPrdMrjz1oYgn0hb8VQo+QU7wD+AcCQi/CH06ny+2QjSr7bfdpgQ81Wgwqt9+e3KcLD5qdt3tneC8sgSAR5QUQrWKhPejrqdV+3990k8k0KoKIOZ9MGFc1FBv5IpuT/2d535iHJb0ZEg8mFKqKDw71sBEoQT4WElatf5DeVC507hP7GnrhZgtTPcrVddOLkmmqTQtLE1zvo9Y/eel8O2JMImMd6mdVBIcFQ1alU1wV4jLdMpagoPelvVTVeNkGRnOgvvQac5RXtGSMP1R5PoGz+1FgRhDCtdTXRfOdtrlJPtUjYiGPpIoKIUymlvQGjUrrK3WiQIIkIfDtFFu48EN4drjgQpCEQ2D0NMKEmk74mjx3CJDOF1Nj/2Fa1kjECiz4EM7sR8dDG5cW5Y8XsDoomgugM+rsOH9kWFFClHUJmdlNGWLhJnu9UQHLYZjQw3zT0F+FXUsUxKJUVTUdxQx5HBO9QFsSe13NPDnX64+SQ2VVStt8oQAvFO+6260zbux0RVaJM5dTNwnIHD/uI3PeR1z2b+iKhyj/9VP2uBRpGydLw/etQu/lS2K30S+kZIDAp0sqwXH89bdq3AJBipcryrnW1Ov9K5Mfl1UlV95/MivrnOFISWN1F9XR1kuIIYdCkI7hq/FHZ9gd84AjTiERfxRv8VCIHBVEQ== nwatkins@redhat.com", "key_options": null, "keyfile": "/home/nwatkins/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nwatkins", "validate_certs": true} 2021-02-10T10:53:50.397 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJD+2BHStkaMJ8RStIIz+yDJCGNT1wbwUnmuO71vNlFxzIyHGvCq3ktx2xTwZz/kj6ZayXYWIRH//BhK2/iK/v+u2WQMBURfMHbf0lDMmNfbHCDBjYvEYgFo4Jwvzq4WHus0AHh9TdVt+TpcxyGPhVbUJh8NZut+w4cRIKu2xGoZZGDeLgv9ijXvDPrCdBxtpvQnF99C0bCTdPM7MK1FSCQEaRmNUgzEj0H7R+gmWi0T4HUD8ZY7y0vWbYPqT3Z07My1u2BfRBYpXnv1655/b777TkSlzSasORhSTT0AkW25u90Rsj2JWr39+49/Ia9uOyFtZIirugEv2v7v/l0grN linuxkidd@zenbook", "key_options": null, "keyfile": "/home/mkidd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkidd", "validate_certs": true} 2021-02-10T10:53:50.398 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJD+2BHStkaMJ8RStIIz+yDJCGNT1wbwUnmuO71vNlFxzIyHGvCq3ktx2xTwZz/kj6ZayXYWIRH//BhK2/iK/v+u2WQMBURfMHbf0lDMmNfbHCDBjYvEYgFo4Jwvzq4WHus0AHh9TdVt+TpcxyGPhVbUJh8NZut+w4cRIKu2xGoZZGDeLgv9ijXvDPrCdBxtpvQnF99C0bCTdPM7MK1FSCQEaRmNUgzEj0H7R+gmWi0T4HUD8ZY7y0vWbYPqT3Z07My1u2BfRBYpXnv1655/b777TkSlzSasORhSTT0AkW25u90Rsj2JWr39+49/Ia9uOyFtZIirugEv2v7v/l0grN linuxkidd@zenbook", "key_options": null, "keyfile": "/home/mkidd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkidd", "validate_certs": true} 2021-02-10T10:53:50.839 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrRvNeBa+/GXxkSDmaim7U1wFpdBvgcK9/eP1I4fPS5xAWyxXI0s0ryX4Jwb7auhZdcqPjgUPQvIVVRridKJPz7l6jUlbIqdnZu4VHuaGG/DfIVxsSmw5aI5qI9Fg/s16JPwOUuzcLHsSSh8tFxY9L1yvs3ztQnKHW6nN5+zhPbW8UjmTNUksXXAeCM/OZn+IOOzzvfh0vVe/mghIW2JpOjHkZv4qwaWYcHg6+wWE4m8l84VIegDBiONknZXcQaRj82nHtfdnb2FSiV4+gKP18i90ai0iTKKXUkMJNFKp1T5MuF7UuJLA7l10FmgQJsZBS2Loa5oeqbHYeztNAfUxF JCLO@oh-mbp-jcl", "key_options": null, "keyfile": "/home/jlopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlopez", "validate_certs": true} 2021-02-10T10:53:50.840 INFO:teuthology.task.ansible.out: 2021-02-10T10:53:50.853 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrRvNeBa+/GXxkSDmaim7U1wFpdBvgcK9/eP1I4fPS5xAWyxXI0s0ryX4Jwb7auhZdcqPjgUPQvIVVRridKJPz7l6jUlbIqdnZu4VHuaGG/DfIVxsSmw5aI5qI9Fg/s16JPwOUuzcLHsSSh8tFxY9L1yvs3ztQnKHW6nN5+zhPbW8UjmTNUksXXAeCM/OZn+IOOzzvfh0vVe/mghIW2JpOjHkZv4qwaWYcHg6+wWE4m8l84VIegDBiONknZXcQaRj82nHtfdnb2FSiV4+gKP18i90ai0iTKKXUkMJNFKp1T5MuF7UuJLA7l10FmgQJsZBS2Loa5oeqbHYeztNAfUxF JCLO@oh-mbp-jcl", "key_options": null, "keyfile": "/home/jlopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlopez", "validate_certs": true} 2021-02-10T10:53:51.283 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDFkUz8Wr2RPiY8+xf2hL4qMT+jA4mo1gmpIF8q7uVR03vX5NhzQVxVnXiyLCFv6susSMxpBafa7aLutqigXhDGJWUbsJII5yTnFYVer0QmudEsLmoDyvI1HCNk1IVY6aIESE50N3Ndx+agWasIU8TGKqJdDMPyiUtfrvsrGWi/91ZLX6aJSrFqBJtTpxDoNe4XL6QRPNv0wb8GF0az5graXkzVe3GkaQS9lyw8m2ougDvW1SLvoeE+lhUUGhJGkkCZKv1f+QNLhUHyQ3XkXqh4MrP3pPUKIuQ7nKiWDjCP6Db+ZcejjTRC19D2gLjhrw3Z2J7nn/loijlFzV5dp+BL haomaiwang@gmail.com", "key_options": null, "keyfile": "/home/haomaiwang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "haomaiwang", "validate_certs": true} 2021-02-10T10:53:51.283 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDFkUz8Wr2RPiY8+xf2hL4qMT+jA4mo1gmpIF8q7uVR03vX5NhzQVxVnXiyLCFv6susSMxpBafa7aLutqigXhDGJWUbsJII5yTnFYVer0QmudEsLmoDyvI1HCNk1IVY6aIESE50N3Ndx+agWasIU8TGKqJdDMPyiUtfrvsrGWi/91ZLX6aJSrFqBJtTpxDoNe4XL6QRPNv0wb8GF0az5graXkzVe3GkaQS9lyw8m2ougDvW1SLvoeE+lhUUGhJGkkCZKv1f+QNLhUHyQ3XkXqh4MrP3pPUKIuQ7nKiWDjCP6Db+ZcejjTRC19D2gLjhrw3Z2J7nn/loijlFzV5dp+BL haomaiwang@gmail.com", "key_options": null, "keyfile": "/home/haomaiwang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "haomaiwang", "validate_certs": true} 2021-02-10T10:53:51.732 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA296in+8Sr9slF8hrzAV3dNz0q94Q4FlFaMYw08II5PQm6HOpabGOGOwOud/PPeh3lMUHhs45N5WnbgdfQdzz9lpDoNBcqxeACdMicdECMHtYWf2GuFUdTAnPBFVl8wFW5JcziQYJePUcAUJRXGGfdMLMppAi8VEuhMw/z3jz89UN+zf51Y9eT3rgAVSCR+Zn47kite4OB2S1aVMEcZBE0fPnh60i+rpN26s80fh3wG2N9JrPBEvst6LbL0aEixrlRUtoFHZuWNCIjntVMANa+idpuWSxN2dzfcoxpGLF1TFzqx0moznp0PnDs7ZAFFS1/EkeSCgBflyhPV6csG3VNw== jdillaman@ceph1.example.com", "key_options": null, "keyfile": "/home/jdillaman/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jdillaman", "validate_certs": true} 2021-02-10T10:53:51.732 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA296in+8Sr9slF8hrzAV3dNz0q94Q4FlFaMYw08II5PQm6HOpabGOGOwOud/PPeh3lMUHhs45N5WnbgdfQdzz9lpDoNBcqxeACdMicdECMHtYWf2GuFUdTAnPBFVl8wFW5JcziQYJePUcAUJRXGGfdMLMppAi8VEuhMw/z3jz89UN+zf51Y9eT3rgAVSCR+Zn47kite4OB2S1aVMEcZBE0fPnh60i+rpN26s80fh3wG2N9JrPBEvst6LbL0aEixrlRUtoFHZuWNCIjntVMANa+idpuWSxN2dzfcoxpGLF1TFzqx0moznp0PnDs7ZAFFS1/EkeSCgBflyhPV6csG3VNw== jdillaman@ceph1.example.com", "key_options": null, "keyfile": "/home/jdillaman/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jdillaman", "validate_certs": true} 2021-02-10T10:53:52.170 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAuTvkaUtEaLbLcYA9Zw3MoHI5ZPudKf6CAbAZdhp4RMVujimR61TT6LdRF20g6F7pH7GrI/1WP2bu7a3sZyCZ1nCJCOhrTTS+2wwy7oGo/0yY4l+Bw6i7LGmJgqpO5dDxwADMlaHUwJ7Rf1Q+Pd7NmggkAVTmaN6qUGffz9hZsj7ibD/Z8/GFtQsCEMloTLtnz4NB6Gxe54UXEKz9pvjpXBhs0xWsJtDKaPFrbKRKswuH45XPuwJhGr249ZYtwQqcgXSWhOUcPTwf5Y6fj27aL2dx566DxQzqzNbcTcO96MguIy8Ufnc0kIOQQw9oHJZtg82zwM6+2mWr9J/K94Gv9w== kefu@save.node2", "key_options": null, "keyfile": "/home/kchai/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchai", "validate_certs": true} 2021-02-10T10:53:52.170 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAuTvkaUtEaLbLcYA9Zw3MoHI5ZPudKf6CAbAZdhp4RMVujimR61TT6LdRF20g6F7pH7GrI/1WP2bu7a3sZyCZ1nCJCOhrTTS+2wwy7oGo/0yY4l+Bw6i7LGmJgqpO5dDxwADMlaHUwJ7Rf1Q+Pd7NmggkAVTmaN6qUGffz9hZsj7ibD/Z8/GFtQsCEMloTLtnz4NB6Gxe54UXEKz9pvjpXBhs0xWsJtDKaPFrbKRKswuH45XPuwJhGr249ZYtwQqcgXSWhOUcPTwf5Y6fj27aL2dx566DxQzqzNbcTcO96MguIy8Ufnc0kIOQQw9oHJZtg82zwM6+2mWr9J/K94Gv9w== kefu@save.node2", "key_options": null, "keyfile": "/home/kchai/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchai", "validate_certs": true} 2021-02-10T10:53:52.608 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2021-02-10T10:53:52.609 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2021-02-10T10:53:53.057 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/r/Xf58szqJ3wLdTE+mRK1Z11f/+VABmx6804NO77ncdC5ZvD3Kx6WP/oje01iwWEVXNAgEOaXp6j+U10aPvBMaS30qf/IhUcrXwO1YcGzZ0pV6UZMxcTuJw6w1VXy/YtgW5BlIfSm78zabztES9BqCgjYYM2rml3L1teDF4WU1QiI97iW0R57yc0t/gqXGpxqSFQ+iNYAl0Q1MGUgYltLvPrhm9WI2WhR2/GivdVMEM5lViNlegxneAZRb9YZMCG2kmSP+9Zd2NR367OrlgzmgeY/9u+KGXQiyfaO330s+iksxVwwE3fDl+KnqXuHSJNccJV7iXzZKej2QAire4X dfuller@Douglass-MacBook-Pro.local", "key_options": null, "keyfile": "/home/dfuller/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dfuller", "validate_certs": true} 2021-02-10T10:53:53.058 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/r/Xf58szqJ3wLdTE+mRK1Z11f/+VABmx6804NO77ncdC5ZvD3Kx6WP/oje01iwWEVXNAgEOaXp6j+U10aPvBMaS30qf/IhUcrXwO1YcGzZ0pV6UZMxcTuJw6w1VXy/YtgW5BlIfSm78zabztES9BqCgjYYM2rml3L1teDF4WU1QiI97iW0R57yc0t/gqXGpxqSFQ+iNYAl0Q1MGUgYltLvPrhm9WI2WhR2/GivdVMEM5lViNlegxneAZRb9YZMCG2kmSP+9Zd2NR367OrlgzmgeY/9u+KGXQiyfaO330s+iksxVwwE3fDl+KnqXuHSJNccJV7iXzZKej2QAire4X dfuller@Douglass-MacBook-Pro.local", "key_options": null, "keyfile": "/home/dfuller/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dfuller", "validate_certs": true} 2021-02-10T10:53:53.494 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIKepJjIovBTCkdVyvAdptljhw9bYcwtIqV/ojdbpeuJtxDo9UyrEkNoLeVpZgOcrbgrkOE7AEixo3qB9PbD8z5NRUleV7wipSh3QwrDmSylSVReJKOwYdxUKkzesZYBGIGYzAbTKqcVzPUWCeW314+xmv90XkO/+1vgbC5uoXfbfnaiDHQimBom8QV5Egr+EhhcuCqMEqaVSvjMr1T0e6IL9gUens6Mys6czI37go6QvmgEzkBbDJJ64PsiOOFHVNk4DJVJ1c7TRtsnpIHM1YVFjsH1AnAL6UufbgNSFf5dziUB/Bz815P1/jsOGpHkUKE37DbHU5xtLGl3MmimMt4ndcG6EggBGEYBwfEWPRAsZs3qngssrNH5W0CzbysLaoovzq5VqS8SdLMu5XrRLBEB0uWWzLq2bobYymk3t/oby5zrHJV78xAuLmldqjvlPZSdkgwWNv9YBLkteBVr2HhjtuashJ6amNPbA4kCGhXMpeRhfvfqs31CBEKbTV+A8ODWbWAoQxLCcqS24x0yMi73N0UzuqDTaJMcE8ODJvOxcLNaV688QDbrC/UehpX/Bp/DM7fLHj4YhkRpZxkkN5w2kb/bBbWya4DrffhHKqn9CH1z5ynj0237PhTwCXjtRn0JzpeRgN8OraFgR2H3uPc9XNqlkFxxOBK0lRngbxzw== owasserm@redhat.com", "key_options": null, "keyfile": "/home/owasserm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "owasserm", "validate_certs": true} 2021-02-10T10:53:53.495 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIKepJjIovBTCkdVyvAdptljhw9bYcwtIqV/ojdbpeuJtxDo9UyrEkNoLeVpZgOcrbgrkOE7AEixo3qB9PbD8z5NRUleV7wipSh3QwrDmSylSVReJKOwYdxUKkzesZYBGIGYzAbTKqcVzPUWCeW314+xmv90XkO/+1vgbC5uoXfbfnaiDHQimBom8QV5Egr+EhhcuCqMEqaVSvjMr1T0e6IL9gUens6Mys6czI37go6QvmgEzkBbDJJ64PsiOOFHVNk4DJVJ1c7TRtsnpIHM1YVFjsH1AnAL6UufbgNSFf5dziUB/Bz815P1/jsOGpHkUKE37DbHU5xtLGl3MmimMt4ndcG6EggBGEYBwfEWPRAsZs3qngssrNH5W0CzbysLaoovzq5VqS8SdLMu5XrRLBEB0uWWzLq2bobYymk3t/oby5zrHJV78xAuLmldqjvlPZSdkgwWNv9YBLkteBVr2HhjtuashJ6amNPbA4kCGhXMpeRhfvfqs31CBEKbTV+A8ODWbWAoQxLCcqS24x0yMi73N0UzuqDTaJMcE8ODJvOxcLNaV688QDbrC/UehpX/Bp/DM7fLHj4YhkRpZxkkN5w2kb/bBbWya4DrffhHKqn9CH1z5ynj0237PhTwCXjtRn0JzpeRgN8OraFgR2H3uPc9XNqlkFxxOBK0lRngbxzw== owasserm@redhat.com", "key_options": null, "keyfile": "/home/owasserm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "owasserm", "validate_certs": true} 2021-02-10T10:53:53.937 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDVFy1sjGvvHwS7Q6E61ykH4dUGfxW9+a7Xo3JMpTBT33/E8bTrYF6ZHSdH+uxScisV9Vll222NJCLXyx2s9EOcv4L88vWDTZGCaUb3iA5dweLm7EYi1NXT/p+jWDA7MAF1FQGSAQafTHpNRxLcrVPgsLDy35weV4xj7ShEU+w7EBDb75jlJXqy0luJLKNBrLzz/wDV9Oe6y3APwQBhtO7vGrVzrWEF/LWLe7U9ad3DQjvLM7+H+QVpNy+nt7us73K8vHDk1KrPlCCjrcN+fSGPD/ILlxrDrDeqmmBCtPMa7PSSjCyAV/EW0fsmArcHrHXKeKWTKXG9sDL0KBqos1kB vasu@vasuSrv\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQConeaJ2kvUOMy+jeKvXZTw655L3KK0ZG0HONIySFrRNkGa/M/hYf8oMNVh9SEax7vLNme0kwk7d71EhgtMGNsvHtQOkshWxM9P0XPk4HgB/OL+2LQ/7M3SwD4vuXPbqvrQS63YMoxcHBue5Ol5hXmM+qSoZajpUrfB2woqbp7aKsCpoK4Cgj4XfLHfh0n239ae38DU6GoAcL0GzMsxMdQhWzBwOO+ZHfowohc1pfY0+xGIMW1OlMN1eO7kuK2gGS/YRVOnLw9uu/2NErIjBzCihNN0PYWur2JE+KmU1x6VMaky49cKmAlc5WUqZ+FpV3gvnb3cLjBClnCkdjIcLeRR vasukulkarni@ovpn-116-227.phx2.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRFyOO7qitrq6Hk2NSl28h4laRTxA2zezy1wwjTqs6UL+oZjNRYQjeWD3dDNPZrUmc9JyI3j1EN11cCsKAp8YXz3ItB46M0BPeJ4TXCpg36vxs7/twSDMq/EoXA5sRBbUJFf+bthcNemwG1WsJoxRVzMezbWBTHvz5i4Hbh4sL3PBbac3FhT/3b5Q4tKdO6bXGXmNLAoALyVZZMfoSz9CWzvrO+g8Houb9iJiaXHp293y+ND9Wicl9wHPckO1VcocZ+ldByujgX1cz6m+pxmf6AKY1O 2021-02-10T10:53:53.938 INFO:teuthology.task.ansible.out:MJj5PjmpN0HrgpV2QLVA+ba9qBiDwi1g8sRHXkOETMZ2jq/2a/mmTtvrsF vasu@teuthology", "key_options": null, "keyfile": "/home/vasu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vasu", "validate_certs": true} 2021-02-10T10:53:53.948 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDVFy1sjGvvHwS7Q6E61ykH4dUGfxW9+a7Xo3JMpTBT33/E8bTrYF6ZHSdH+uxScisV9Vll222NJCLXyx2s9EOcv4L88vWDTZGCaUb3iA5dweLm7EYi1NXT/p+jWDA7MAF1FQGSAQafTHpNRxLcrVPgsLDy35weV4xj7ShEU+w7EBDb75jlJXqy0luJLKNBrLzz/wDV9Oe6y3APwQBhtO7vGrVzrWEF/LWLe7U9ad3DQjvLM7+H+QVpNy+nt7us73K8vHDk1KrPlCCjrcN+fSGPD/ILlxrDrDeqmmBCtPMa7PSSjCyAV/EW0fsmArcHrHXKeKWTKXG9sDL0KBqos1kB vasu@vasuSrv\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQConeaJ2kvUOMy+jeKvXZTw655L3KK0ZG0HONIySFrRNkGa/M/hYf8oMNVh9SEax7vLNme0kwk7d71EhgtMGNsvHtQOkshWxM9P0XPk4HgB/OL+2LQ/7M3SwD4vuXPbqvrQS63YMoxcHBue5Ol5hXmM+qSoZajpUrfB2woqbp7aKsCpoK4Cgj4XfLHfh0n239ae38DU6GoAcL0GzMsxMdQhWzBwOO+ZHfowohc1pfY0+xGIMW1OlMN1eO7kuK2gGS/YRVOnLw9uu/2NErIjBzCihNN0PYWur2JE+KmU1x6VMaky49cKmAlc5WUqZ+FpV3gvnb3cLjBClnCkdjIcLeRR vasukulkarni@ovpn-116-227.phx2.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRFyOO7qitrq6Hk2NSl28h4laRTxA2zezy1wwjTqs6UL+oZjNRYQjeWD3dDNPZrUmc9JyI3j1EN11cCsKAp8YXz3ItB46M0BPeJ4TXCpg36vxs7/twSDMq/EoXA5sRBbUJFf+bthcNemwG1WsJoxRVzMezbWBTHvz5i4Hbh4sL3PBbac3FhT/3b5Q4tKdO6bXGXmNLAoALyVZZMfoSz9CWzvrO+g8Houb9iJiaXHp293y+ND9Wicl9wHPckO1VcocZ+ldByujgX1cz6m+pxmf6AKY1O 2021-02-10T10:53:53.949 INFO:teuthology.task.ansible.out:MJj5PjmpN0HrgpV2QLVA+ba9qBiDwi1g8sRHXkOETMZ2jq/2a/mmTtvrsF vasu@teuthology", "key_options": null, "keyfile": "/home/vasu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vasu", "validate_certs": true} 2021-02-10T10:53:54.386 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDlOImwCV0AGu5L29ABaefV9Vqjv43WYJYFuLTxGQQdeQGcways6iNKGIX/nzf3UWkwgg25OVSpzB7zf6FjBx+HcQU+RJ7FLzeaz0s8ZxBXqWwtOYOvP9cXmAXA07hD5UAfxZKC1twBzL82bKJ2ZSmzfQmBjvpUOqRs7XwsQIClOW/ycY+jLgLRLFFDxzADsgnn6UEGlHeMETYocUbe89P3wbY5fHhqZmn6e5ncuAQMsiLRa9d+QhSdpD07Y/P65EzqRF4UOX+zpOc6gPlgHFir9pvf0bTbbBG/qKN9+KGqzZMRJGKenpHM6EMfLkf0QmrT4s4zd4ZnUQPq3HN3pul3 redhat@unused\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9n4ege98BNmvgUxogdcpptMILFfEisAQaiLUvSKpTsoy2gF7C6xSJXgDaziT5dW4ad2El4xmKQ/e6/e14Vx5i9pGbJz4NvL36Xmb3ent2PtY18i6d3CbnmGI8J7YH4YS9QvR7FC/KuvwLR6AdJ+uQShGhqG8kWJMjbqrtaxu8l1hq8ymBXw3O1Ht+xrZ/r3dEkAIVF3bMCSUnZpEbSsNoeiydEQl0ZCw+meXYmSw15vYVtKpYscZxqADqhSJJxIEHZA4E88dq6aYdV2AZ/s4cJK+tBl/ckBnclI3VGHSGn6oaHoO/fCHtKiENplDuKoLGQPO6xJ2Q/apHSGD7McY3 redhat@network\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDn/XgNfvqytLHgfBUdQIhJiWBlj+JrXqGZYvAZZJIWs32Ugm1/lCQTssGZrEj3+Qv81jLfkP6VQ8gXSWJkzUK4csxRxkZKXqcYLDE1csdmAFDOfWUFJwhMsCiiCyBiW9oapioLpvd40nw8k9qM4QuzaUhVonoGXMDDL1/dCgkbIjiQxxTJPHDEJFjtzul+VuhlhHqE6S1syoJG+UKo3Q1tOBCs8rwaPXEfCXFsyv+ZmcEv1+mpM8QRjdr1D1hfUfwb4CHtaa2AE6B5wL+zoY6+jsrTwZSwCfeIdgEZ8e3V2iXP5F7G6eXotu5r7mGYJClHZFLSfbNPN5J+EbRORl9/ wbhack3r@dhcp35-24.lab.eng.blr.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHYkp1nuRSYSAl6me1xuoTU30DW2YpAR6EH1c2adUC9tNtTZbMnab11aitQLAtZb9SKsRgnoB5KpH+zJ0VNKCuJU227r9vZwcVUruSGwA6X7LOrSkK0gFqNRZkVoKhC9oGHSU0zegYBDxx8VLxSV3CE56b+KEosEo6iJe7TkXy3wXHK6am0uhQPEdSHtdUdD683CGIffdF1RFmAFnzTPn96lC9rLe1mIGO+jVQx4zIo3i33RFQLy++4DtQw6Lrgp26Xr85HmQekjgyiya0VkR9Ypy7fgF2ucBGegnOgZDGCCY4G2RuqiZyKz 2021-02-10T10:53:54.387 INFO:teuthology.task.ansible.out:2TZaOecDwrLkbYM7EGp8up5uLFRZsV shmohan@shmohan.remote.csb", "key_options": null, "keyfile": "/home/smohan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smohan", "validate_certs": true} ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDlOImwCV0AGu5L29ABaefV9Vqjv43WYJYFuLTxGQQdeQGcways6iNKGIX/nzf3UWkwgg25OVSpzB7zf6FjBx+HcQU+RJ7FLzeaz0s8ZxBXqWwtOYOvP9cXmAXA07hD5UAfxZKC1twBzL82bKJ2ZSmzfQmBjvpUOqRs7XwsQIClOW/ycY+jLgLRLFFDxzADsgnn6UEGlHeMETYocUbe89P3wbY5fHhqZmn6e5ncuAQMsiLRa9d+QhSdpD07Y/P65EzqRF4UOX+zpOc6gPlgHFir9pvf0bTbbBG/qKN9+KGqzZMRJGKenpHM6EMfLkf0QmrT4s4zd4ZnUQPq3HN3pul3 redhat@unused\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9n4ege98BNmvgUxogdcpptMILFfEisAQaiLUvSKpTsoy2gF7C6xSJXgDaziT5dW4ad2El4xmKQ/e6/e14Vx5i9pGbJz4NvL36Xmb3ent2PtY18i6d3CbnmGI8J7YH4YS9QvR7FC/KuvwLR6AdJ+uQShGhqG8kWJMjbqrtaxu8l1hq8ymBXw3O1Ht+xrZ/r3dEkAIVF3bMCSUnZpEbSsNoeiydEQl0ZCw+meXYmSw15vYVtKpYscZxqADqhSJJxIEHZA4E88dq6aYdV2AZ/s4cJK+tBl/ckBnclI3VGHSGn6oaHoO/fCHtKiENplDuKoLGQPO6xJ2Q/apHSGD7McY3 redhat@network\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDn/XgNfvqytLHgfBUdQIhJiWBlj+JrXqGZYvAZZJIWs32Ugm1/lCQTssGZrEj3+Qv81jLfkP6VQ8gXSWJkzUK4csxRxkZKXqcYLDE1csdmAFDOfWUFJwhMsCiiCyBiW9oapioLpvd40nw8k9qM4QuzaUhVonoGXMDDL1/dCgkbIjiQxxTJPHDEJFjtzul+VuhlhHqE6S1syoJG+UKo3Q1tOBCs8rwaPXEfCXFsyv+ZmcEv1+mpM8QRjdr1D1hfUfwb4CHtaa2AE6B5wL+zoY6+jsrTwZSwCfeIdgEZ8e3V2iXP5F7G6eXotu5r7mGYJClHZFLSfbNPN5J+EbRORl9/ wbhack3r@dhcp35-24.lab.eng.blr.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHYkp1nuRSYSAl6me1xuoTU30DW2YpAR6EH1c2adUC9tNtTZbMnab11aitQLAtZb9SKsRgnoB5KpH+zJ0VNKCu 2021-02-10T10:53:54.388 INFO:teuthology.task.ansible.out:JU227r9vZwcVUruSGwA6X7LOrSkK0gFqNRZkVoKhC9oGHSU0zegYBDxx8VLxSV3CE56b+KEosEo6iJe7TkXy3wXHK6am0uhQPEdSHtdUdD683CGIffdF1RFmAFnzTPn96lC9rLe1mIGO+jVQx4zIo3i33RFQLy++4DtQw6Lrgp26Xr85HmQekjgyiya0VkR9Ypy7fgF2ucBGegnOgZDGCCY4G2RuqiZyKz2TZaOecDwrLkbYM7EGp8up5uLFRZsV shmohan@shmohan.remote.csb", "key_options": null, "keyfile": "/home/smohan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smohan", "validate_certs": true} 2021-02-10T10:53:54.828 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDZePxJmYerihzA0aQbMt7K7bctPEDcwPHiO8EVuJowp6zwhSmtDnYuFybKJsee4l8+rDKR6T5Ugdl5pLjSzQ+Ba00OSH7+qWC6shvVDYju5mIooJnc+kwx8RgapbGtDdl/yYskBb1KD5g/3+i/2aM+Igk+7nQR7Cny5+WnUEvHd7PHBbUXQSuu+mlUlnXCVMM8g8qlc1I3WQL5eQoIqGjSWRgZUIsn3P5oryYBzbuG9jqLwbaohJvZRJqkimQDe7zCFk6bB3z39pyOWUN+sldRvyL8FPyEmqog4IwpjcYyCuCnPhnb6fNq9yB65RUIde82Kkzx3LZrYOXTrdHgD/lr abhi@trusty", "key_options": null, "keyfile": "/home/abhi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "abhi", "validate_certs": true} 2021-02-10T10:53:54.829 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDZePxJmYerihzA0aQbMt7K7bctPEDcwPHiO8EVuJowp6zwhSmtDnYuFybKJsee4l8+rDKR6T5Ugdl5pLjSzQ+Ba00OSH7+qWC6shvVDYju5mIooJnc+kwx8RgapbGtDdl/yYskBb1KD5g/3+i/2aM+Igk+7nQR7Cny5+WnUEvHd7PHBbUXQSuu+mlUlnXCVMM8g8qlc1I3WQL5eQoIqGjSWRgZUIsn3P5oryYBzbuG9jqLwbaohJvZRJqkimQDe7zCFk6bB3z39pyOWUN+sldRvyL8FPyEmqog4IwpjcYyCuCnPhnb6fNq9yB65RUIde82Kkzx3LZrYOXTrdHgD/lr abhi@trusty", "key_options": null, "keyfile": "/home/abhi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "abhi", "validate_certs": true} 2021-02-10T10:53:55.266 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDcsBe6CzgWh64iZAH8zbU6ra4yEiUBSsiUn0AHzyQurvjeRpL2Mut6B1WEi0kJG7BAM6b13JrhnD9BW9yIMIUIPpVTa2o4zaXP90+bvjGrgDeJbxv3T9/TGh13k+WWyampxm2lo90/+Paql/HObQyyIv7Wan0DOdIe2/Ue4ooWH0tu3kj+3unJTuw82oEDEn+9tGs0oleG2c8XN0gQyVl1DzR2Gqf9PCzXK7hInC3niY2huoSAZzVFEgsniKEWqTnVWYi49X2SsJhUyHCj+mqrTG3YnW30nha3EXOWIbQGblXVoTIOc+gNreqdGByOMICKSnQmPSG4DAl3c1Y4h4A71T3Q043IUv7v7S9tAEKq2h6oJ9I/a5z0u4kaL/+B7Cw7l4bitTdcZ0rcqSvOkKUNNgxujbucMSSc4Tsh5L0762xaziD9nQZl5bteb3VtsMAueqhu7L3sJy69+HGg7oF68t5TguErXfI2UNVGOn3eRj+zN/zSwLUXMoROlpM5yHXSzHvpsbgZiHBImM2CeG4qaPtqzFw7VH6NWJv4ypb0htOMWsbXmLshNTca1KXrrBcVozSJC8wKgEOxu4spk26G8x1IVTsi2ypyYX+irtAqZrrpIPdQ0Ss9NjKPhFNXAxJgLHj6TlH8UNh1gRfN/QTyzpFSe4EH6uOlnWuxcuM8ZQ== cbodley@redhat.com", "key_options": null, "keyfile": "/home/cbodley/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cbodley", "validate_certs": true} 2021-02-10T10:53:55.267 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDcsBe6CzgWh64iZAH8zbU6ra4yEiUBSsiUn0AHzyQurvjeRpL2Mut6B1WEi0kJG7BAM6b13JrhnD9BW9yIMIUIPpVTa2o4zaXP90+bvjGrgDeJbxv3T9/TGh13k+WWyampxm2lo90/+Paql/HObQyyIv7Wan0DOdIe2/Ue4ooWH0tu3kj+3unJTuw82oEDEn+9tGs0oleG2c8XN0gQyVl1DzR2Gqf9PCzXK7hInC3niY2huoSAZzVFEgsniKEWqTnVWYi49X2SsJhUyHCj+mqrTG3YnW30nha3EXOWIbQGblXVoTIOc+gNreqdGByOMICKSnQmPSG4DAl3c1Y4h4A71T3Q043IUv7v7S9tAEKq2h6oJ9I/a5z0u4kaL/+B7Cw7l4bitTdcZ0rcqSvOkKUNNgxujbucMSSc4Tsh5L0762xaziD9nQZl5bteb3VtsMAueqhu7L3sJy69+HGg7oF68t5TguErXfI2UNVGOn3eRj+zN/zSwLUXMoROlpM5yHXSzHvpsbgZiHBImM2CeG4qaPtqzFw7VH6NWJv4ypb0htOMWsbXmLshNTca1KXrrBcVozSJC8wKgEOxu4spk26G8x1IVTsi2ypyYX+irtAqZrrpIPdQ0Ss9NjKPhFNXAxJgLHj6TlH8UNh1gRfN/QTyzpFSe4EH6uOlnWuxcuM8ZQ== cbodley@redhat.com", "key_options": null, "keyfile": "/home/cbodley/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cbodley", "validate_certs": true} 2021-02-10T10:53:55.716 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrtHyRuOSEXAM6mwL4qoM+NM4hpIKhaRS+/Y6RDoFKszkXFw/jC9ZYh7q39rQpWQBKHVe4RyPN9yyGWydFwaR/BjZi14ippmYOqVG4f7b+k39NFbMJRcSxYC+NvkqxK7+zpsnIWC+RdJ+blZi+zCbaaAYcMDSvBZW19FsJSLrO7hh4R1cuzgV+Xrmwdc1/VH+HqPMu3tHGv7N0SfsklWI8Inr0aiL83QqqgKPcVxljSWZGYvvFyTQOrkNQeaLURJNER9WIIac9o57WOINw8kR+ojebcOUBsrxZH9j+gMjsJ752Rph03wdMUpFcnS6yfvI4Ial7x/sGms4FE+wL/vVF mdw@pika", "key_options": null, "keyfile": "/home/mwatts/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mwatts", "validate_certs": true} 2021-02-10T10:53:55.717 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrtHyRuOSEXAM6mwL4qoM+NM4hpIKhaRS+/Y6RDoFKszkXFw/jC9ZYh7q39rQpWQBKHVe4RyPN9yyGWydFwaR/BjZi14ippmYOqVG4f7b+k39NFbMJRcSxYC+NvkqxK7+zpsnIWC+RdJ+blZi+zCbaaAYcMDSvBZW19FsJSLrO7hh4R1cuzgV+Xrmwdc1/VH+HqPMu3tHGv7N0SfsklWI8Inr0aiL83QqqgKPcVxljSWZGYvvFyTQOrkNQeaLURJNER9WIIac9o57WOINw8kR+ojebcOUBsrxZH9j+gMjsJ752Rph03wdMUpFcnS6yfvI4Ial7x/sGms4FE+wL/vVF mdw@pika", "key_options": null, "keyfile": "/home/mwatts/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mwatts", "validate_certs": true} 2021-02-10T10:53:56.167 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com", "key_options": nul 2021-02-10T10:53:56.169 INFO:teuthology.task.ansible.out:l, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/ 2021-02-10T10:53:56.170 INFO:teuthology.task.ansible.out:HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com", "key_options": null, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2021-02-10T10:53:56.619 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQD3vdP9QHyNFUNPOdplezEbdYsQfcC91ncATI6Fl/xHifmUbcvjIRqZHkCY6FNWUpMeg9S3+wlHJtzjVqJJ/MlQML3EtwnzQkeWv8bjtuZ07OqULYIv0zEx63XUfSnrdFleJ8+KDSpn7krkYE44q85B88IL+ZBeq5eCXFJ1TtmsDcvsN8WqZNWMdUGqgH3Bb4D8+YKrEfGxoievyxZ1A1DnBHan2l8foTKYggRrnuSbTb59cFpxY3R7Arc5Cn9kDJT1nDVBA5SSGCfRMiNyBOUzQVYe/UjKTaCpngglXC3wYnjoGvGcpksHLQCxyuJPB6djC1tdF+3Jehdf8y8+2TBzWXiiPq2XwS58sER5GcLLLQv26pPL8hgu6CdI0sHxgcgQBgpjoLwUMHmv13CnN/AAcZY4RRDK2X5zd6wFSEhuuCoUqh/m7K219IxI6fVkLaApC851D/ua/pj9OMjiOaJ+D13cPrgfqxva4eK8FQZvqQm7aUK/TT216QNYmn7hkzNjMTkjv/L5yYQ5Ym92AHy44wHzLPrtDB5mk94vHOjG+KlAt9ofuLncMSAXWe+fOqxuIQ6VfbL2LUv0EGYhHe4XOVj98q5utsVQxQbXAZzNQaq9FWXs2A4xcVP7aWj0Fooi17LN/W0ng4Iijir5hyLr7k7M1G6resgiY7p83biNow== pdonnell@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQCpgLVt7Zphmf6nqD+9BC0N91jdmaY6ZpvrYWlzbhxXITQ6juIEtegnlZ+8ohdeEfHFBiyu1xOSQSM1X1ugvpPQE94UqQg7JVb0tuoUCNJHYnIuv8T4YgyP0/vEV48k2G0BBWlP+IJ5f1iDvZuLHWLR5JSgGqh5ELaaHr7Cbyb6JlImldb7SDIoPHMCZ+n6dLO8l5Zy7qWelw6aeA770xmPucBr6t/BVns2g77XgJz6zFd99yEaDy27i8jfzL7WqR9i93yVE9EsL7tuOjg6G9KUzxBF0bg9iVsJXypvcBuG0JwQ+SDKqYw/cN4iLy5qBa6Siqrfpc/LKU21VNmJ0e5eXNgKKbMBpBb8GVmkoLaSRcXw7LclWopjNDA+meUhDAwU7b97N7cqN8BYhBKQyiShXFHMKLuNo2fSuTRt/icfb9VisKNGBdvwgV0oOlrqbXMzZKmj9amb9b+3JDBTGB7ioCsj5x/Evf9srXRrQ9kikIr1aBbVXIWir/lasNWVydcmUYsNi8L7W0vXArLwp+GgOXJFeZUiD3z9LG4CouRxMSwoj3lgWeL2ZR8feH4D/SP/+QmBdU6+aG97au9qreeXi5X8QTR3LND0DwwEKczX+3s+Fa950Oieom/gXC57PVKVh90KVoAGw3HzJTlZrauOtLkYbTUAX6woOMhAtJ5oWuDDT4UrHQj37jRrrl2dETLwKwUJN2+2L07vnnOZ4ABcGWLsU6d3bXeUBlw3pdsYqPebDKUEpjL2kUOw+zmfjEf 2021-02-10T10:53:56.620 INFO:teuthology.task.ansible.out:dVx9BfrvExvDX+3MkxccCJ8FY7od1s5fGEFHsXnIiMnLO/HZb/JN+Aoe+/ExadIcYwJnWcCJyly2/532kGFUlzOliCwPxgJpjWCKQAv6LjWmxoPXiA4JI0bp6y7lMEjmV2Bp3mwphlsW/r8Ie5hg1UPh/ttHRp/VZgcV8e6nNmP/qc6k9b2DWTNFqtWqm1hXLOwq08HzUEeH9mTqvenyAbv2RFMOS18Y2K+4BKUzHwZujrz9qEh/kqtpFScEIXkOJ+HZz3mI6XfjWFIPS487yRGfUvW3+nSWhvHc8vJY21bFRKdjD84qCVlZ2BfOCGUZSKftVItEiqXMTRs3RZd5epfCVZbJuA0hG16jUseeHqqZqDtlbLkzkxtS6v3iGRnPh/Ke/94FHHCzvYY8cmMYAVaM6p/G3edWftbaEtN70Fo7VKxPrpPzuSRZ7Vxmnb+Zad95DxxAC/vtMy/vkBVIP6dyn4HGxqa1oECFMe4oXdhnxZQoPzvWxb8upcXeAeOMMbcIlMq5a+dCBPGFNKhNhrb+9uAEDHwjE7rZlMJVSvt9v1S4hll6jiYIap+7n4reSKrS45aO9U0xiKXJJDohh pdonnell@icewind\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQC8MdUJzIvTCn0fM+hv1HAO/Lwjy/C/V7stQRWGIxyrRHbniyX8iuaU3j7YNFglSQPJq+YRM9UnGauM9/0J9tHKLsMp+zRQYrtxU/ThiTUN8TRJY11mGUol4yLsqkTtm6oq5sl6o+LfuA4hzaYc3XLrFHRJuCUgbtPG71Q0C/y3U+itrDK4wXUxBa4YzZzcIMg+c88VXwoouPbiL0gW5PCXzMaNtKlOqBUKlE935MLn70wIEfN/EwAAbss0ga1moqeqmi9Nlvm20DNIVFmkAGtYoq6CHXtL/uehzai5Sueop8Vmd8+1U8gDTo6JpiagdD8ZNf0zAbRx587KAT5ZMnahJpO276j7RNa85ZFkNkEnBLqBHRkCkTCYq1GehYyRaR69uuKyczFiIoTfAQgPh7VlnpT9525b/4nFsBDOkWxxWAIJR61YiKemPV/n/e9SwhvVsMDb2B/RBriwR7Mi2SZ/lBHGB+z0OA+EE2rGZy4LFVOfUczqIdpltlNotXan4M4F361i+bX19KE+f2RLySn3+GiFv0CdLwslS/y4xysz+FWxw6B0Y/EkpiDU3CdWM61+THGGm/DVhfL/Pvca7k8PonHFYw/x5w1nBcADpkM9JF+m5eOnp+weTEzl58sS+3GnoAuL8YFJwMW1RmE50xM0o5E3SDc8XftUvaSZz8mhII3uwDDukbELfmkLbnIeipQKyjO+GEx0JBp4Itz82qGuao0Wi32U9wHzhSWnX4GQ0pKZnq9frSlyCNwHu/iMiKF8sA2bX3dvplFJ9nKWPefcU8bdl/upr5hYrd2GttjUSkWZa+UuF264yx3ztcN7jotRhEmmdEcHVzKQ26/NyRI4/9DQpDKY7zw4O2wFkXWg5L9kodXwWzgy4cyOr4dMD9LBVLOpHc0CAji5/+lfuYQbu1mwVdfOzCXDqByjt7zCbeN7KuUh403sJ8PTQVp642xA9i4AkPv5g+fqi5C0++1DbZP0lmLfKtTswHhoN+kAqZu2N4Mu6mNVNKyZgW5h1pZzNSGWYcZttyYmMCHhPketReL0LPJ+VUsqQCVohuta3aaSe44InA28GZsXzMj5B0el/84bnE7cN4b+rfGD25NT73JEt3RgPa9W9jqT1hwd/nf53BDKpp7DlHjR77WSWT4h4F7bhbM1RsGSqAfZjCUJT6Rq+IuM2XcrLDY8zIeArkqaZLXMzfGj6KJ7cQes05P5LYfY4osXiq68Gs/7gzF31PQRRGOYAoqMN7752sn0ZMcpnUj3mJutQcncRUN4+naTMjU36WWD7it0YuQBnb2IJ4iKjYLInw8pr2zKgWtgr1TSCM4xKp/v1/ZMmbBmS+dT 2021-02-10T10:53:56.622 INFO:teuthology.task.ansible.out:zNC7Mpkt8vdCNjzZ5KZ5 pdonnell@aglarond", "key_options": null, "keyfile": "/home/pdonnell/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdonnell", "validate_certs": true} ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQD3vdP9QHyNFUNPOdplezEbdYsQfcC91ncATI6Fl/xHifmUbcvjIRqZHkCY6FNWUpMeg9S3+wlHJtzjVqJJ/MlQML3EtwnzQkeWv8bjtuZ07OqULYIv0zEx63XUfSnrdFleJ8+KDSpn7krkYE44q85B88IL+ZBeq5eCXFJ1TtmsDcvsN8WqZNWMdUGqgH3Bb4D8+YKrEfGxoievyxZ1A1DnBHan2l8foTKYggRrnuSbTb59cFpxY3R7Arc5Cn9kDJT1nDVBA5SSGCfRMiNyBOUzQVYe/UjKTaCpngglXC3wYnjoGvGcpksHLQCxyuJPB6djC1tdF+3Jehdf8y8+2TBzWXiiPq2XwS58sER5GcLLLQv26pPL8hgu6CdI0sHxgcgQBgpjoLwUMHmv13CnN/AAcZY4RRDK2X5zd6wFSEhuuCoUqh/m7K219IxI6fVkLaApC851D/ua/pj9OMjiOaJ+D13cPrgfqxva4eK8FQZvqQm7aUK/TT216QNYmn7hkzNjMTkjv/L5yYQ5Ym92AHy44wHzLPrtDB5mk94vHOjG+KlAt9ofuLncMSAXWe+fOqxuIQ6VfbL2LUv0EGYhHe4XOVj98q5utsVQxQbXAZzNQaq9FWXs2A4xcVP7aWj0Fooi17LN/W0ng4Iijir5hyLr7k7M1G6resgiY7p83biNow== pdonnell@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQCpgLVt7Zphmf6nqD+9BC0N91jdmaY6ZpvrYWlzbhxXITQ6juIEtegnlZ+8ohdeEfHFBiyu1xOSQSM1X1ugvpPQE94UqQg7JVb0tuoUCNJHYnIuv8T4YgyP0/vEV48k2G0BBWlP+IJ5f1iDvZuLHWLR5JSgGqh5ELaaHr7Cbyb6JlImldb7SDIoPHMCZ+n6dLO8l5Zy7qWelw6aeA770xmPucBr6t/BVns2g77XgJz6zFd99yEaDy27i8jfzL7WqR9i93yVE9EsL7tuOjg6G9KUzxBF0bg9iVsJXypvcBuG0JwQ+SDKqYw/cN4iLy5qBa6Siqrfpc/LKU21VNmJ0e5eXNgKKbMBpBb8GVmkoLaSRcXw7LclWopjNDA+meUhDAwU7b97N7cqN8BYhBKQyiShXFHMKLuNo2fSuTRt/icfb9VisKNGBdvwgV0oOlrqbXMzZKmj9amb9b+3JDBTGB7ioCsj5x/Evf9srXRrQ9kikIr1aBbVXIWir/lasNWVydcmUYsNi8L7W0vXArLwp+GgOXJFeZUiD3z9LG4CouRxMSwo 2021-02-10T10:53:56.623 INFO:teuthology.task.ansible.out:j3lgWeL2ZR8feH4D/SP/+QmBdU6+aG97au9qreeXi5X8QTR3LND0DwwEKczX+3s+Fa950Oieom/gXC57PVKVh90KVoAGw3HzJTlZrauOtLkYbTUAX6woOMhAtJ5oWuDDT4UrHQj37jRrrl2dETLwKwUJN2+2L07vnnOZ4ABcGWLsU6d3bXeUBlw3pdsYqPebDKUEpjL2kUOw+zmfjEfdVx9BfrvExvDX+3MkxccCJ8FY7od1s5fGEFHsXnIiMnLO/HZb/JN+Aoe+/ExadIcYwJnWcCJyly2/532kGFUlzOliCwPxgJpjWCKQAv6LjWmxoPXiA4JI0bp6y7lMEjmV2Bp3mwphlsW/r8Ie5hg1UPh/ttHRp/VZgcV8e6nNmP/qc6k9b2DWTNFqtWqm1hXLOwq08HzUEeH9mTqvenyAbv2RFMOS18Y2K+4BKUzHwZujrz9qEh/kqtpFScEIXkOJ+HZz3mI6XfjWFIPS487yRGfUvW3+nSWhvHc8vJY21bFRKdjD84qCVlZ2BfOCGUZSKftVItEiqXMTRs3RZd5epfCVZbJuA0hG16jUseeHqqZqDtlbLkzkxtS6v3iGRnPh/Ke/94FHHCzvYY8cmMYAVaM6p/G3edWftbaEtN70Fo7VKxPrpPzuSRZ7Vxmnb+Zad95DxxAC/vtMy/vkBVIP6dyn4HGxqa1oECFMe4oXdhnxZQoPzvWxb8upcXeAeOMMbcIlMq5a+dCBPGFNKhNhrb+9uAEDHwjE7rZlMJVSvt9v1S4hll6jiYIap+7n4reSKrS45aO9U0xiKXJJDohh pdonnell@icewind\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQC8MdUJzIvTCn0fM+hv1HAO/Lwjy/C/V7stQRWGIxyrRHbniyX8iuaU3j7YNFglSQPJq+YRM9UnGauM9/0J9tHKLsMp+zRQYrtxU/ThiTUN8TRJY11mGUol4yLsqkTtm6oq5sl6o+LfuA4hzaYc3XLrFHRJuCUgbtPG71Q0C/y3U+itrDK4wXUxBa4YzZzcIMg+c88VXwoouPbiL0gW5PCXzMaNtKlOqBUKlE935MLn70wIEfN/EwAAbss0ga1moqeqmi9Nlvm20DNIVFmkAGtYoq6CHXtL/uehzai5Sueop8Vmd8+1U8gDTo6JpiagdD8ZNf0zAbRx587KAT5ZMnahJpO276j7RNa85ZFkNkEnBLqBHRkCkTCYq1GehYyRaR69uuKyczFiIoTfAQgPh7VlnpT9525b/4nFsBDOkWxxWAIJR61YiKemPV/n/e9SwhvVsMDb2B/RBriwR7Mi2SZ/lBHGB+z0OA+EE2rGZy4LFVOfUczqIdpltlNotXan4M4F361i+bX19KE+f2RLySn3+GiFv0CdLwslS/y4xysz+FWxw6B0Y/EkpiDU3CdWM61+THGGm/DVhfL/Pvca7k8PonHFYw/x5w1nBcADpkM9JF+m5eOnp+weTEzl58sS+3GnoAuL8YFJwMW1RmE50xM0o5E3SDc8XftUvaSZz8mhII3uwDDukbELfmkLbnIeipQKyjO+GEx0JBp4Itz82qGuao0Wi32U9wHzhSWnX4GQ0pKZnq9frSlyCNwHu/iMiKF8sA2bX3dvplFJ9nKWPefcU8bdl/upr5hYrd2GttjUSkWZa+UuF264yx3ztcN7jotRhEmmdEcHVzKQ26/NyRI4/9DQpDKY7zw4O2wFkXWg5L9kodXwWzgy4cyOr4dMD9LBVLOpHc0CAji5/+lfuYQbu1mwVdfOzCXDqByjt7zCbeN7KuUh403sJ8PTQVp642xA9i4AkPv5g+fqi5C0++1DbZP0lmLfKtTswHhoN+kAqZu2N4Mu6mNVNKyZgW5h1pZzNSGWYcZttyYmMCHhPketReL0LPJ+VUsqQCVohuta3aaSe44InA28GZsXzMj5B0el/84bnE7cN4b+rfGD25NT73JEt3RgPa9W9jqT1hwd/nf53 2021-02-10T10:53:56.624 INFO:teuthology.task.ansible.out:BDKpp7DlHjR77WSWT4h4F7bhbM1RsGSqAfZjCUJT6Rq+IuM2XcrLDY8zIeArkqaZLXMzfGj6KJ7cQes05P5LYfY4osXiq68Gs/7gzF31PQRRGOYAoqMN7752sn0ZMcpnUj3mJutQcncRUN4+naTMjU36WWD7it0YuQBnb2IJ4iKjYLInw8pr2zKgWtgr1TSCM4xKp/v1/ZMmbBmS+dTzNC7Mpkt8vdCNjzZ5KZ5 pdonnell@aglarond", "key_options": null, "keyfile": "/home/pdonnell/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdonnell", "validate_certs": true} 2021-02-10T10:53:57.059 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDKzhyWb3PmU88m8BbgpEuBzYM6+x28z6aZwHSesiyfXMfF4Y7IZjtKaTxH0ph/O9GQCPhRCVhRd8TlgKWJGhts1E+VkQfHfT18CgmrPWzj3gifc2FJhL97vB1Pd3jksGnOPAikgYPHP46SVMTtyvXCkOUuo7E29WrkdpjawcqM1rfpw63OnzHf7Ar7Sqh/othB7ajyVQ92t+XWSEqj7K+NRfTaq5L5RBWbwarM885o8xdI5mimOZ0QjzKOhFDORrI+HEVlkutyB8E20nZgo7TUy+kYeLVhEqb3A6Pi2QbA9GFLT69kA/H9nebhj7Cuim9yrx4OKeZuIncG0UxSQF3QmWflAZHBs8L/jniTz4DJ5id5jR29tvGK2LeddCEMB1L2cHwWHlG0aX5jaNEJe+nG6H5Tcg3+M1RPM8twBkRQmS3UzAyX/MTH1kktyFUHn8CdBXe0WTuoFSvXLB6BCHWzDYZOt0tIV/xMtBwhy+XGYA4oZSsDU5m7VlSVLBGL8DbRX69Z8AGey2y3FoFu/roY6NmDhTxUedJ2ZfoKSwdZD6jjatdO1621nonBnJxrDgaXCWzT7HtnWcc3GwxZoBrqkvgpsx88f4lNi22OpYVPV6A4hECe04u6OJnHSWVk3ft3p5PaLNF8aEXbiIgzZKAwX3LxpvEDFH/t1sdihhpmjw== jlayton@synchrony.poochiereds.net", "key_options": null, "keyfile": "/home/jlayton/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlayton", "validate_certs": true} 2021-02-10T10:53:57.060 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDKzhyWb3PmU88m8BbgpEuBzYM6+x28z6aZwHSesiyfXMfF4Y7IZjtKaTxH0ph/O9GQCPhRCVhRd8TlgKWJGhts1E+VkQfHfT18CgmrPWzj3gifc2FJhL97vB1Pd3jksGnOPAikgYPHP46SVMTtyvXCkOUuo7E29WrkdpjawcqM1rfpw63OnzHf7Ar7Sqh/othB7ajyVQ92t+XWSEqj7K+NRfTaq5L5RBWbwarM885o8xdI5mimOZ0QjzKOhFDORrI+HEVlkutyB8E20nZgo7TUy+kYeLVhEqb3A6Pi2QbA9GFLT69kA/H9nebhj7Cuim9yrx4OKeZuIncG0UxSQF3QmWflAZHBs8L/jniTz4DJ5id5jR29tvGK2LeddCEMB1L2cHwWHlG0aX5jaNEJe+nG6H5Tcg3+M1RPM8twBkRQmS3UzAyX/MTH1kktyFUHn8CdBXe0WTuoFSvXLB6BCHWzDYZOt0tIV/xMtBwhy+XGYA4oZSsDU5m7VlSVLBGL8DbRX69Z8AGey2y3FoFu/roY6NmDhTxUedJ2ZfoKSwdZD6jjatdO1621nonBnJxrDgaXCWzT7HtnWcc3GwxZoBrqkvgpsx88f4lNi22OpYVPV6A4hECe04u6OJnHSWVk3ft3p5PaLNF8aEXbiIgzZKAwX3LxpvEDFH/t1sdihhpmjw== jlayton@synchrony.poochiereds.net", "key_options": null, "keyfile": "/home/jlayton/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlayton", "validate_certs": true} 2021-02-10T10:53:57.508 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCijPApnekNl/dsY+fleKYstEzNkKflBEcCoKtjItUCDJCVLVFOsDBrV94EGvDWpr13aj9sSwpUzRbeu/vOW8ZT853HH6Dqg2+eoYwG6vDRprZb93nw6l4vtU59/M0yKkyi9CPp2+YUo31cCFzYWpq8bv2j1xBruDSUCUdn9nJrHiLFqA4QEwSpw+uVCjSKzQL+LjlKmE8Plfd+8mNCQeY/OFsjxAztwN+4M50SZCnOVCIwEXoYRNd82JOXMOaqZJazSWTO8Tml6+Nu0hyEcvHkHxWRDSM+6LmXNnjlVUPyDN2h1/VAYATpOZ2VhmRarDoftCp9Wzi7eQdL/CGiMyuD rzarzyns@redhat.com", "key_options": null, "keyfile": "/home/rzarzynski/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rzarzynski", "validate_certs": true} 2021-02-10T10:53:57.508 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCijPApnekNl/dsY+fleKYstEzNkKflBEcCoKtjItUCDJCVLVFOsDBrV94EGvDWpr13aj9sSwpUzRbeu/vOW8ZT853HH6Dqg2+eoYwG6vDRprZb93nw6l4vtU59/M0yKkyi9CPp2+YUo31cCFzYWpq8bv2j1xBruDSUCUdn9nJrHiLFqA4QEwSpw+uVCjSKzQL+LjlKmE8Plfd+8mNCQeY/OFsjxAztwN+4M50SZCnOVCIwEXoYRNd82JOXMOaqZJazSWTO8Tml6+Nu0hyEcvHkHxWRDSM+6LmXNnjlVUPyDN2h1/VAYATpOZ2VhmRarDoftCp9Wzi7eQdL/CGiMyuD rzarzyns@redhat.com", "key_options": null, "keyfile": "/home/rzarzynski/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rzarzynski", "validate_certs": true} 2021-02-10T10:53:57.959 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdGthG4qx90VQ8mJ7TI/IFFzptkSqBHYZsFIVu06IQgHbnFBvjao6heGt4UlkWJkegPCDWpKVAe7wFuXDpPlj/VssRHB5LBnYHKGfkNAzMNiMjpEZsTdy+9w39CCujipz0AlOSfrUdANBZ11AnG9VgQSPEukX5487juob55lFM6J3x+v4opBF3rSlux3+thbr8V0w4oCHMD9/aHvVjFoBnxo4oirR4jc/jgC2fQCMiAJhFKJGhjkKFiNHxcB3npBuRnpy+59BcScOAzSXdJ7DGnwxQLjymnhqsAcWinjjJhOjaq6NaoeKv5I7zQLaVGaQdIyuuAK1Dnb7trpfQU7mR rdias@rdias-suse-laptop", "key_options": null, "keyfile": "/home/rdias/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rdias", "validate_certs": true} 2021-02-10T10:53:57.970 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdGthG4qx90VQ8mJ7TI/IFFzptkSqBHYZsFIVu06IQgHbnFBvjao6heGt4UlkWJkegPCDWpKVAe7wFuXDpPlj/VssRHB5LBnYHKGfkNAzMNiMjpEZsTdy+9w39CCujipz0AlOSfrUdANBZ11AnG9VgQSPEukX5487juob55lFM6J3x+v4opBF3rSlux3+thbr8V0w4oCHMD9/aHvVjFoBnxo4oirR4jc/jgC2fQCMiAJhFKJGhjkKFiNHxcB3npBuRnpy+59BcScOAzSXdJ7DGnwxQLjymnhqsAcWinjjJhOjaq6NaoeKv5I7zQLaVGaQdIyuuAK1Dnb7trpfQU7mR rdias@rdias-suse-laptop", "key_options": null, "keyfile": "/home/rdias/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rdias", "validate_certs": true} 2021-02-10T10:53:58.408 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEEjb7yrcY6I6XbMU0hCps1NqBuYmA6HDmMcS/l3i5wJRBitIXYVlv6pefUG9gdBYR5NneccLfVFURu+DC63MJa2ELJvVXsQ6YUOX39bJUT9KlZs8mH+nlTD7tlPmP17Ae3k8sT0H42Q+/5+b5YvdAU+03aPhzntmy38SyvLKrvowRk63wHgcE9p23LY9tY3+HSJu86cb/IdePk1GLjnrMeT9aM+5j2eETrmBjMArP+/2UIsT5eb8MBkQzJ299SCC2gfV/qIVkorFl9xdejctAwNwdT0SPTMxCHxaiK3wZ0e/5HHTbSgR5YlxOlKYq93XBVYIaFKPCePk1At1a3Pjh cephalopod@octopus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDcrTZoKO5e5xs2PB2UuAruvGlJESTHNAtqipbC8yJtRg8WGVDLt//rGu36QNSbpbJYMZgqgKsXBYolJB1ckqLYLv5j8RAoVwbW4QqKaV95y7x0bbXs/YAPnRBhEO1nfpT8GHQusUZvTbz38FDaQwstsYoncmsyZwBrqus9ycFavpY4wfpKoWtzeiK5Wjq7ervThOs98ZJKXJXOOxeI7mDakE3Jv7q2tBNu/rU1kzxIl65kWURzgxox6EmIZXQK3FzKt0jjdKtApVYSlx4ODLPI4iyxOv+MbRe3Nnj6FDhfz8/FeP/U5wWxdYx2WslzZCOI1BJ0VDLu3xEpLlu4vPzp vshankar@h3ckers-pride", "key_options": null, "keyfile": "/home/vshankar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vshankar", "validate_certs": true} 2021-02-10T10:53:58.408 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEEjb7yrcY6I6XbMU0hCps1NqBuYmA6HDmMcS/l3i5wJRBitIXYVlv6pefUG9gdBYR5NneccLfVFURu+DC63MJa2ELJvVXsQ6YUOX39bJUT9KlZs8mH+nlTD7tlPmP17Ae3k8sT0H42Q+/5+b5YvdAU+03aPhzntmy38SyvLKrvowRk63wHgcE9p23LY9tY3+HSJu86cb/IdePk1GLjnrMeT9aM+5j2eETrmBjMArP+/2UIsT5eb8MBkQzJ299SCC2gfV/qIVkorFl9xdejctAwNwdT0SPTMxCHxaiK3wZ0e/5HHTbSgR5YlxOlKYq93XBVYIaFKPCePk1At1a3Pjh cephalopod@octopus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDcrTZoKO5e5xs2PB2UuAruvGlJESTHNAtqipbC8yJtRg8WGVDLt//rGu36QNSbpbJYMZgqgKsXBYolJB1ckqLYLv5j8RAoVwbW4QqKaV95y7x0bbXs/YAPnRBhEO1nfpT8GHQusUZvTbz38FDaQwstsYoncmsyZwBrqus9ycFavpY4wfpKoWtzeiK5Wjq7ervThOs98ZJKXJXOOxeI7mDakE3Jv7q2tBNu/rU1kzxIl65kWURzgxox6EmIZXQK3FzKt0jjdKtApVYSlx4ODLPI4iyxOv+MbRe3Nnj6FDhfz8/FeP/U5wWxdYx2WslzZCOI1BJ0VDLu3xEpLlu4vPzp vshankar@h3ckers-pride", "key_options": null, "keyfile": "/home/vshankar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vshankar", "validate_certs": true} 2021-02-10T10:53:58.857 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+CtnvyVRSkIvD15O/q7/vQmD4oE8q7Mo6oT3kqTXaEaQ7iDayC2biUC0PC4OztPlNJxPUcKK5DfWw4F79OKeheqUbZRhsCC3Ge2JGoXHmsf0kBYdioMGiAxm+f8M/v1KESSSxvEnU7o+oi6VtFchh4Dl/WZ54rqZc/oQrlGNTHulFAuyoIAlKkCe3N0GdHN54PB+26QbLXvOuqpWmjVFbcjgRC62mkZl3LuCBhviXS4Rl5/ZE0QIHubCwV5XVrsqvscLS64bi06W8RN2vL56Vn3N9TGV4d2dlbE1PaBBESoTQ7Q0iUN32OAZLo7nF/1fiPY3Xc8r35juya5k+KR9z adam@TP50", "key_options": null, "keyfile": "/home/akupczyk/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akupczyk", "validate_certs": true} 2021-02-10T10:53:58.858 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+CtnvyVRSkIvD15O/q7/vQmD4oE8q7Mo6oT3kqTXaEaQ7iDayC2biUC0PC4OztPlNJxPUcKK5DfWw4F79OKeheqUbZRhsCC3Ge2JGoXHmsf0kBYdioMGiAxm+f8M/v1KESSSxvEnU7o+oi6VtFchh4Dl/WZ54rqZc/oQrlGNTHulFAuyoIAlKkCe3N0GdHN54PB+26QbLXvOuqpWmjVFbcjgRC62mkZl3LuCBhviXS4Rl5/ZE0QIHubCwV5XVrsqvscLS64bi06W8RN2vL56Vn3N9TGV4d2dlbE1PaBBESoTQ7Q0iUN32OAZLo7nF/1fiPY3Xc8r35juya5k+KR9z adam@TP50", "key_options": null, "keyfile": "/home/akupczyk/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akupczyk", "validate_certs": true} 2021-02-10T10:53:59.297 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDeUJcLeyfqIk3ErPoFqh9kOnvUPsk/2hfaYNC2wZLMOSTIs3zXM8xjki8eN6vle0dmmeeAdwF+e2hJ3tCFKsEUQRvMc7ugIBA7L8/DboShYu7ItyQ3QRvy9pi6jnA/tlHA1LHalvyhMw0yZ4kvPbrfNMhZzOKzvL6jX166rqLA5AuiIfdYMBpE5rYq11GVjm0uQKw4UJpBMooNJeovEfQlNBTSV+IJSTiGBpBD7SgsuW3USMeEF3fEgQOcnvYdv5Fbp/crloyLWYc0HTidkLxpjP7vqY7E4BuBJug0zIB7voR3MYq6rAeuWjByPvq6coDlUjnw/QlrvarCbjjSaHdaRXLj/Dj+QqJFviESBr64ZynhHL21Bl6rexW7AwYyPa73Erbus3dN0aVQQ/ApcDTGHT7KRlDQ2YYg05YM4swpVVnrIdV58FzLWkll+ypCK/+kdtk5eRSe/KzZA3s7z4d4WLgolFWs8Fky8Rx1xDGewvMJgFUWVUbcGC13RocbNSxXG80cbIx8L/sclX4lDfGouu2sUXhP19vq1rJ7gZa6XTPohXjPbFS+En2cdlGdKXhppdvjT4QWKNzgQoF7qgWHMqJnMFJ6beT/2zP34KoM3t0ulee04z88VSn8Th8aqvLlGbdrt5LrE90escSaIAR1vNKZ1hCFwGieSKoRtYJTCw== nojha@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6NWn9EzQJoeXkUbaNclYcR9Gga7HFhhXMGxdo/9ypYwlEzQTA6FnEreG3s1EmlQ4hofuc0kMhrjh0YeJZMrETWMoA0SmoX+6iR26UWVtOTk7i1v2TqkTY7llmQneHqQgmZEBXK0jF1HVP5kHY5dhagtcpfsGY5dc0eGUjfqAyh/xrCxciOE/JQ03tDcKIc5eOQIqv46JqnUS90Zr/Cov2qGKKHTa/VO1a2viOmDC/uHi5xXvII1Kh9rHp1oZ2Wm3fiGgT8xZaTJFFhWpdq3AdKfkEaV93yQcCdaUsGBHwDHcPOhxpBZpqZvcCR3I/tigDav4QViB1PRN7JnTIWCIznfrQK3r2Frl6BVFDvydwgRVS43ctEKPOJj8ELMUWX+fpjsKrA9Kn7nUU35ti7qIWKg1Cnz3lzVA29TsEg10E+nT4SVKZtRSWw/2AX0bsfxfSY30yjw5PP+bjnVh/bOOkqR2UkPBj4eeOKMD8nV1an79xP0eH+t36INaJhq6qwxE= nojha@nojha-mac", "key_options": null, "keyfile": "/home/ 2021-02-10T10:53:59.299 INFO:teuthology.task.ansible.out:nojha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nojha", "validate_certs": true} ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDeUJcLeyfqIk3ErPoFqh9kOnvUPsk/2hfaYNC2wZLMOSTIs3zXM8xjki8eN6vle0dmmeeAdwF+e2hJ3tCFKsEUQRvMc7ugIBA7L8/DboShYu7ItyQ3QRvy9pi6jnA/tlHA1LHalvyhMw0yZ4kvPbrfNMhZzOKzvL6jX166rqLA5AuiIfdYMBpE5rYq11GVjm0uQKw4UJpBMooNJeovEfQlNBTSV+IJSTiGBpBD7SgsuW3USMeEF3fEgQOcnvYdv5Fbp/crloyLWYc0HTidkLxpjP7vqY7E4BuBJug0zIB7voR3MYq6rAeuWjByPvq6coDlUjnw/QlrvarCbjjSaHdaRXLj/Dj+QqJFviESBr64ZynhHL21Bl6rexW7AwYyPa73Erbus3dN0aVQQ/ApcDTGHT7KRlDQ2YYg05YM4swpVVnrIdV58FzLWkll+ypCK/+kdtk5eRSe/KzZA3s7z4d4WLgolFWs8Fky8Rx1xDGewvMJgFUWVUbcGC13RocbNSxXG80cbIx8L/sclX4lDfGouu2sUXhP19vq1rJ7gZa6XTPohXjPbFS+En2cdlGdKXhppdvjT4QWKNzgQoF7qgWHMqJnMFJ6beT/2zP34KoM3t0ulee04z88VSn8Th8aqvLlGbdrt5LrE90escSaIAR1vNKZ1hCFwGieSKoRtYJTCw== nojha@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6NWn9EzQJoeXkUbaNclYcR9Gga7HFhhXMGxdo/9ypYwlEzQTA6FnEreG3s1EmlQ4hofuc0kMhrjh0YeJZMrETWMoA0SmoX+6iR26UWVtOTk7i1v2TqkTY7llmQneHqQgmZEBXK0jF1HVP5kHY5dhagtcpfsGY5dc0eGUjfqAyh/xrCxciOE/JQ03tDcKIc5eOQIqv46JqnUS90Zr/Cov2qGKKHTa/VO1a2viOmDC/uHi5xXvII1Kh9rHp1oZ2Wm3fiGgT8xZaTJFFhWpdq3AdKfkEaV93yQcCdaUsGBHwDHcPOhxpBZpqZvcCR3I/tigDav4QViB1PRN7JnTIWCIznfrQK3r2Frl6BVFDvydwgRVS43ctEKPOJj8ELMUWX+fpjsKrA9Kn7nUU35ti7qIWKg1Cnz3lzVA29TsEg10E+nT4SVKZtRSWw/2AX0bsf 2021-02-10T10:53:59.300 INFO:teuthology.task.ansible.out:xfSY30yjw5PP+bjnVh/bOOkqR2UkPBj4eeOKMD8nV1an79xP0eH+t36INaJhq6qwxE= nojha@nojha-mac", "key_options": null, "keyfile": "/home/nojha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nojha", "validate_certs": true} 2021-02-10T10:53:59.743 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+c7zV5QmZb0WsgA2UZogoDNaoHcngRVLA8+BYk/U10ZlvCckJSfIXWMlBnqguZgdm/jBwsw5mGVBgHxUSqFNybwIlU00kyPZU6VbyVP+ZNVUf0XIE1NlemldgPBxxcOzSHkHhWrUqA3IKiCllcQVDQzutfUnq+jDT6NkVn4/cyNzNlargt7FR2L8gvskh62jLTBbhEuHMBp3jLCvaHJOk708gXm9qZ5iLOGV0pu7xdg58KE0p8ILClQBTbvoqPgZE9LA3qaOk1NR1n22kuW8JRHaKfD2Hn0Yu/MfKct783Bxb3mxWBKXxmM4otdmsR/R14NH/hIPce9wSkOOnQAC9 ifed@37526", "key_options": null, "keyfile": "/home/ifed01/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ifed01", "validate_certs": true} 2021-02-10T10:53:59.744 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+c7zV5QmZb0WsgA2UZogoDNaoHcngRVLA8+BYk/U10ZlvCckJSfIXWMlBnqguZgdm/jBwsw5mGVBgHxUSqFNybwIlU00kyPZU6VbyVP+ZNVUf0XIE1NlemldgPBxxcOzSHkHhWrUqA3IKiCllcQVDQzutfUnq+jDT6NkVn4/cyNzNlargt7FR2L8gvskh62jLTBbhEuHMBp3jLCvaHJOk708gXm9qZ5iLOGV0pu7xdg58KE0p8ILClQBTbvoqPgZE9LA3qaOk1NR1n22kuW8JRHaKfD2Hn0Yu/MfKct783Bxb3mxWBKXxmM4otdmsR/R14NH/hIPce9wSkOOnQAC9 ifed@37526", "key_options": null, "keyfile": "/home/ifed01/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ifed01", "validate_certs": true} 2021-02-10T10:54:00.187 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCnOq6OHTRJuv7n0GZygBue/AE2q47pFM3KyrvmZxPRNmwtXn2r3ngHQQyc6DLZdFIv9Yn0VWu1KB/ryCdN3Xy7HAxBfHdKjtQ4vvfsWUbQF2JjhEjrsl+usgpqK1Tz6yuL4uAtFgLTN9o6gcH2bsQ15w+rEHCiYVzaGp7Cj+y+/VPBRIEHETSUZxwqiDH57uFTHxJrBIn6yhCorbn5QOjfmHUGmkxl3Geph++LbuQeha5F/DHaylMulsYZZBDEhIG8G2coDmVz8Cp1gw0kBWv63JHKnXaJboGXty2Zj36O/wdZKTE+7cV6JhxfJczhZBTb2cImIhspGwdHfnFgBoXz myoungwon@fg2server", "key_options": null, "keyfile": "/home/myoungwon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "myoungwon", "validate_certs": true} 2021-02-10T10:54:00.198 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCnOq6OHTRJuv7n0GZygBue/AE2q47pFM3KyrvmZxPRNmwtXn2r3ngHQQyc6DLZdFIv9Yn0VWu1KB/ryCdN3Xy7HAxBfHdKjtQ4vvfsWUbQF2JjhEjrsl+usgpqK1Tz6yuL4uAtFgLTN9o6gcH2bsQ15w+rEHCiYVzaGp7Cj+y+/VPBRIEHETSUZxwqiDH57uFTHxJrBIn6yhCorbn5QOjfmHUGmkxl3Geph++LbuQeha5F/DHaylMulsYZZBDEhIG8G2coDmVz8Cp1gw0kBWv63JHKnXaJboGXty2Zj36O/wdZKTE+7cV6JhxfJczhZBTb2cImIhspGwdHfnFgBoXz myoungwon@fg2server", "key_options": null, "keyfile": "/home/myoungwon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "myoungwon", "validate_certs": true} 2021-02-10T10:54:00.628 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQDZPRW12u7LvaiynB87ad5h/JFtYIVjkKIOym9Y5CAbvatHhZ9fB1WAnYIn2HqiSSPUXT8sFy8w0TmM+aH0/ZGcfXoe0CdTWLtUbXKNlPI5h7TO9rHXESP2iRBpnKs3vK8WnMY0/D5BeESrPfwVScucmzaDrjUqjdZKR/aEeqVvDpQsdaCaoqNg23t4nhQ54BEvK8O+14nQ26Rtmm40QHO1ze+0UK+7nUMMFwxvd1EKjVp21EgYAFEbyQ5Kbbl+fmS/11AGy/yTai0c4qd7ohBzZvx8+sY+c1novG9Q7LhtyxuuYwzYIrGBCkKH5+iC7nt2rGBP+mnTfIgdbJUiXX3cdrMESwOSZy6x6zgbXXME+yfM0myhomKmUkpeZAQInX2KYrnEuPD9kteyK9tZcG6gJSaiUdvlab49Alx5lM2hc62M4s4WKaIgroCoApd8FtNAlTQsdYuycntgNrbPy0LXQ5hw8K8hkvc6RbxISdj/RXWguQSVbJWdS9Pf5NwD3Tq7dsOghJWhx4vBkhW3IECVrPQWVyAmgQ9oj1PoL7/2LZ6h9P8F0KC1qmhm+pWdEPLugOlU6lzQX7RJcukYlsINThM1RGwbPE4h9XcMyHAnODiDF63TDfQ01oKbVXj9XpOGY5eP0EQxKJYjCfWW0Ot6nK2ZCt8OYvoRkWMGnLv11Nub3NNOgh5TB/4nDvEU1cIb/Ikb4BRzVIEbast/pqUJmXe+GQ+ho7mp8y8a+lIxu54BR+VN6isjwODCVt1DYohgixEoul7p5pXvTM+JUY2GD/ddTGwPvVAN66sWnT9bjZZunQwTHqasFCIHNyPhnEnYIIqCo9O1I/D8zRQG3SYNxT6ED9GPARrE/H0TvdOmKaT3e0F2GRJRSToNxol9dpJovUziojOhXyBUjlTqnMY2RVilX90MnzgqR3zGO0aSruTGyNT2Faymqxn6T/50mSdJ8EyWH23BaaXUUW/z3Cjov+yCQjLY48SOWwlxKXw3hGz8xTAxauBYI01ojsVLCpCUGFyKlWS5iFwiLFtr1KPYqeCoS6CoWjiOiFgGe+w3lPPfp2h4rc6auIx2DZ1ByYhWLeHnpkV7sG+q5ckBIRAYmiKyXSGdV4zbXITMWDm+3zu4oj5VPRCfjMlout8OZ4TQm5PUdKZZhSE1jULG9tPDFL7nWwM3z0HET+d9kQYcEO6QB+ma9gZHp1qZJbXvCIC5fQ9y9El++A6N3dBr+ED10Ye88PLoWPeumyAmc24BDdI0skGk89Q0IzoLqagK6HGhPAso8si9wr84CEi4ncpizVLDU2cFKCpef4Sa03HoMbwfTgERtyptKFRbMT+SpuZsnrhF635h9uqFnszowUXV guits@Elisheba", "key_options": null, "keyfile": "/home/gabrioux/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "g 2021-02-10T10:54:00.629 INFO:teuthology.task.ansible.out:abrioux", "validate_certs": true} 2021-02-10T10:54:00.640 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQDZPRW12u7LvaiynB87ad5h/JFtYIVjkKIOym9Y5CAbvatHhZ9fB1WAnYIn2HqiSSPUXT8sFy8w0TmM+aH0/ZGcfXoe0CdTWLtUbXKNlPI5h7TO9rHXESP2iRBpnKs3vK8WnMY0/D5BeESrPfwVScucmzaDrjUqjdZKR/aEeqVvDpQsdaCaoqNg23t4nhQ54BEvK8O+14nQ26Rtmm40QHO1ze+0UK+7nUMMFwxvd1EKjVp21EgYAFEbyQ5Kbbl+fmS/11AGy/yTai0c4qd7ohBzZvx8+sY+c1novG9Q7LhtyxuuYwzYIrGBCkKH5+iC7nt2rGBP+mnTfIgdbJUiXX3cdrMESwOSZy6x6zgbXXME+yfM0myhomKmUkpeZAQInX2KYrnEuPD9kteyK9tZcG6gJSaiUdvlab49Alx5lM2hc62M4s4WKaIgroCoApd8FtNAlTQsdYuycntgNrbPy0LXQ5hw8K8hkvc6RbxISdj/RXWguQSVbJWdS9Pf5NwD3Tq7dsOghJWhx4vBkhW3IECVrPQWVyAmgQ9oj1PoL7/2LZ6h9P8F0KC1qmhm+pWdEPLugOlU6lzQX7RJcukYlsINThM1RGwbPE4h9XcMyHAnODiDF63TDfQ01oKbVXj9XpOGY5eP0EQxKJYjCfWW0Ot6nK2ZCt8OYvoRkWMGnLv11Nub3NNOgh5TB/4nDvEU1cIb/Ikb4BRzVIEbast/pqUJmXe+GQ+ho7mp8y8a+lIxu54BR+VN6isjwODCVt1DYohgixEoul7p5pXvTM+JUY2GD/ddTGwPvVAN66sWnT9bjZZunQwTHqasFCIHNyPhnEnYIIqCo9O1I/D8zRQG3SYNxT6ED9GPARrE/H0TvdOmKaT3e0F2GRJRSToNxol9dpJovUziojOhXyBUjlTqnMY2RVilX90MnzgqR3zGO0aSruTGyNT2Faymqxn6T/50mSdJ8EyWH23BaaXUUW/z3Cjov+yCQjLY48SOWwlxKXw3hGz8xTAxauBYI01ojsVLCpCUGFyKlWS5iFwiLFtr1KPYqeCoS6CoWjiOiFgGe+w3lPPfp2h4rc6auIx2DZ1ByYhWLeHnpkV7sG+q5ckBIRAYmiKyXSGdV4zbXITMWDm+3zu4oj5VPRCfjMlout8OZ4TQm5PUdKZZhSE1jULG9tPDFL7nWwM3z0HET+d9kQYcEO6QB+ma9gZHp1qZJbXvCIC5fQ9y9El++A6N3dBr+ED10Ye88PLoWPeumyAmc24BDdI0skGk89Q0IzoLqagK6HGhPAso8si9wr84CEi4ncpizVLDU2cFKCpef4Sa03HoMbwfTgERtyptKFRbMT+SpuZsnrhF635h9uqFnszowUXV guits@Elisheba", "key_options": null, "keyfile": "/home/gabrioux/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "g 2021-02-10T10:54:00.641 INFO:teuthology.task.ansible.out:abrioux", "validate_certs": true} 2021-02-10T10:54:01.086 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpDFqu0Zg94mCjrA0FTB0vMiSBml1pqYCDKHhKBrfaza8u3bhtsmls9i+6U/vIwCEVk4GGPNhWduTJK0h9R5MS2eCf3GJcE0XHqRp/8msUCDPOVSZwC7hoYHZKunUIUaIUajOnR+L2slA/a9JaqataFH7zdrKbPdSKBzf5CXYrc0fK+Fz5OOsaI9KITbM0nESVpNCDzRqnA/aarF9MlN79vpTtvwQ+i0z2Km3cxqwV70GPMuFVo4CGf8pCGpbMrIg9OtoSC1svQ2vZzLkH2WrbDqTfYlOEYJJXaiBKcGhGnLZXfLIGSgvHjpv6xHo63yaMihocyg2MwTS8TZ/VtAaH seb@redhat.com", "key_options": null, "keyfile": "/home/leseb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "leseb", "validate_certs": true} 2021-02-10T10:54:01.087 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpDFqu0Zg94mCjrA0FTB0vMiSBml1pqYCDKHhKBrfaza8u3bhtsmls9i+6U/vIwCEVk4GGPNhWduTJK0h9R5MS2eCf3GJcE0XHqRp/8msUCDPOVSZwC7hoYHZKunUIUaIUajOnR+L2slA/a9JaqataFH7zdrKbPdSKBzf5CXYrc0fK+Fz5OOsaI9KITbM0nESVpNCDzRqnA/aarF9MlN79vpTtvwQ+i0z2Km3cxqwV70GPMuFVo4CGf8pCGpbMrIg9OtoSC1svQ2vZzLkH2WrbDqTfYlOEYJJXaiBKcGhGnLZXfLIGSgvHjpv6xHo63yaMihocyg2MwTS8TZ/VtAaH seb@redhat.com", "key_options": null, "keyfile": "/home/leseb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "leseb", "validate_certs": true} 2021-02-10T10:54:01.534 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzjJ6nlPn/OXP3knQ6orVId2QLswql9gRetKYlicKDBrdFg0MES03EIuAonu0aUR4NEReR7P1iB8eNnezrHyAu5mxIJlZp0rz7ljcBcTLL+11IJU9I9OjBFwzXGqJ1KK4B4Qfw35oc2O6F5V0JP2nHhHgyny37udJLImhxHy3RNmDwc86JhEBCVR3ouYiZuxgxxbA9BXc6+Lkdu9hEMMCDlaEXwNmMyASVU9Wf3dKnMTSCdar2PANquWbpNQZ5cgoTm39xgust5CEyYJ53aOdrti0DYK7qyzxUXre6ckZGrH4oGv2XrDaxpACIiHXt+slvow51gsZnBRHeayMAxQzGQ== hchen@hchen.csb", "key_options": null, "keyfile": "/home/hchen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hchen", "validate_certs": true} 2021-02-10T10:54:01.543 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzjJ6nlPn/OXP3knQ6orVId2QLswql9gRetKYlicKDBrdFg0MES03EIuAonu0aUR4NEReR7P1iB8eNnezrHyAu5mxIJlZp0rz7ljcBcTLL+11IJU9I9OjBFwzXGqJ1KK4B4Qfw35oc2O6F5V0JP2nHhHgyny37udJLImhxHy3RNmDwc86JhEBCVR3ouYiZuxgxxbA9BXc6+Lkdu9hEMMCDlaEXwNmMyASVU9Wf3dKnMTSCdar2PANquWbpNQZ5cgoTm39xgust5CEyYJ53aOdrti0DYK7qyzxUXre6ckZGrH4oGv2XrDaxpACIiHXt+slvow51gsZnBRHeayMAxQzGQ== hchen@hchen.csb", "key_options": null, "keyfile": "/home/hchen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hchen", "validate_certs": true} 2021-02-10T10:54:01.973 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDCqZKc+/7iPLhapA/7SYmLxwyKEM95D6eFfsDBNKbEUey/B1OlqYSo+yKB/LdFDyUcd7shMDnhFFg47lNMANrsZdtE4iMPVAa1YhcrTgNbV3HSp9cthKixvU/gkt9Fnb6PEckRzr19wEwxtoNg3Fh/R6YCiSjPwqHmArtC1G2hzk560EJpozx6xylMR5IafatNDoZVw2YQ35fZvHbhHD6meQELducHJ1bKEEHxVIRSTcmXDpBz/vAvBoQOEPuH6PmC/oSwL9HVVIsazf65F4QfK98d+8NCK9TemgXyGDk3Sjz6xdm76yBhFzDQEAzO12U1YtS7Q7zGVsjmSDav2dd7 jcollin@earth\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC89dHSsgjgtIjJ3e/pcMthliQlX2vtdXL7pk62nD64230Fm1PfOVCnzn4LqDhBbGgJT+BrlwOPfzStpLgnUT3l67S834SJzhvSTTpQp2pznnQg1rZOiXjvNhbQwVYV4rphXWqay8weWvFIb5QfXE66mYA1etZtcOb1SMY2ofAjUiA8bAQS5aHpzcMVs4N1A9x5GJx/7/Bj1h8zPpSXikDYasJc3++UMGc45C/AAf725YwijK7xMnZNXv8uTBoGzO/Ko24R3VdvtfMVn4Hi4/FwuRd5bRYcJXR/nUdtA3IinqvlnTpFV2LbfgVTHiUMGhvewRTB+8S8ij4nnAlko3H3bhqZZTk/y7d4K/IerY0mozVIc2x8rMLfqhF9OKv1QFB9fskRZcF/ENHITnmLqCJPg9rgmyv1gCQ5Tbds1OI+IYQIowdiwtaViiTSIy7/CedbHqMz6aDnPsJFMs24lcUXe2d8uZ26RqMinMFGXg0zF7tifuVmxj7zG8DT7MFORDm5f/Hhkng05R5XZesStTBxpEU0MGuzWhENt3RuJdstqmY+wkDVW3XwjUvKG/gVYuJYfnBAsRZIZezw2wfvodhc8EmeHELvY1YtB3DGTBXKE1lsKGN7yV5+nl4UtR2IGzBn9GovWVi5ooB8UlSAeUmZix/eAA5teL8ldFrDht7+AQ== jcollin@teuthology\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeApVHu5er+R9h52RLSVYVhXFQSn/nMN38uWoDYPw2bqYBLNEAAMcPl+rnqHubELpXTGXjJYL4n2mjic+2rOLZpDgPLmOOm4PwgYstJjE1KLegKqVzXIa9HfASKMXaHSgwtC6P5vDQkm2MNl8vXuTAITF7VYdDS33Kkk 2021-02-10T10:54:01.975 INFO:teuthology.task.ansible.out:e6ViJRZI7szYjbP7z9A1fy26THrvnuBT1Q18yy9vgk/sKo8vjLQfHSLvO90CJzouNEKqmkVpmJEqCPbE+s6Qr4+d2KY7L6Q6GAXxrqXK1jSDrrPsJguE6zZ1oWy/7XFMOr9ssduFWocnjQjPtQj3M+V+7FYAsf5uqQnzeMQTjO6AXYxTI7 jcollin@stratocaster\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDk/DYRp5E8Dla/4XHCF504X20W0u7CijZSCFWT1/5082NgnYeR90Prth3p+4xsw3YC9H5pEAS3XwcbdUb/NtWvZVIAIcMd7NOOEClSIj2Agk7NJxZ/4SfQRLrK3H2Qli/MVnrTxnEa9oeS4zqzv9dKRwFJDUFCoH2CpoumFVAp02uyzk7D/+XqFrXzMJUiSHXfl3YM2rt/vNmCtHtVp04UcG0Kyl2TXY37frRlSOkMMu0zbIC551gOyae6NGYhkYHa7u2SnngXcKM78b8MDvMv3QtuUN5P4WpH5qFRdlKNKsQGrzcQpBDY8AKQ+rX0K1ENhBwer49bihYNMKvjYmvTdLTNWgPCu7dt8JSgJQ8JMMhve0pjvFdpcKDLp0fvXDvJvZdLJM1ID/66kywJspHjShuqQNXRMfd1R9C4F20c+R9LY6eTGwrpLG1DNEez1VBHmOu5hQl03T2PfaThfLfMHRTw8JpGXGLMQarCCTm1PC6Z2Iw7KzorxP1eofGA5q8XoCbwGD01uQRZrwxPq0SHzTAkW2C2M8oEz+k4Mrx7+vGYzSS14hqxrzk91wuYhZCAKuGpqwm1ouJ4wxvWFewjbxLgucMfGSp09zJ2XSYRmtm7SAUifFob6Z9czHGTQerHCOTPuYqzlap2gCCVYtEvQyNB+4IqK/9FtEXTr28x0Q== jcollin@stratocaster", "key_options": null, "keyfile": "/home/jcollin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcollin", "validate_certs": true} 2021-02-10T10:54:01.984 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDCqZKc+/7iPLhapA/7SYmLxwyKEM95D6eFfsDBNKbEUey/B1OlqYSo+yKB/LdFDyUcd7shMDnhFFg47lNMANrsZdtE4iMPVAa1YhcrTgNbV3HSp9cthKixvU/gkt9Fnb6PEckRzr19wEwxtoNg3Fh/R6YCiSjPwqHmArtC1G2hzk560EJpozx6xylMR5IafatNDoZVw2YQ35fZvHbhHD6meQELducHJ1bKEEHxVIRSTcmXDpBz/vAvBoQOEPuH6PmC/oSwL9HVVIsazf65F4QfK98d+8NCK9TemgXyGDk3Sjz6xdm76yBhFzDQEAzO12U1YtS7Q7zGVsjmSDav2dd7 jcollin@earth\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC89dHSsgjgtIjJ3e/pcMthliQlX2vtdXL7pk62nD64230Fm1PfOVCnzn4LqDhBbGgJT+BrlwOPfzStpLgnUT3l67S834SJzhvSTTpQp2pznnQg1rZOiXjvNhbQwVYV4rphXWqay8weWvFIb5QfXE66mYA1etZtcOb1SMY2ofAjUiA8bAQS5aHpzcMVs4N1A9x5GJx/7/Bj1h8zPpSXikDYasJc3++UMGc45C/AAf725YwijK7xMnZNXv8uTBoGzO/Ko24R3VdvtfMVn4Hi4/FwuRd5bRYcJXR/nUdtA3IinqvlnTpFV2LbfgVTHiUMGhvewRTB+8S8ij4nnAlko3H3bhqZZTk/y7d4K/IerY0mozVIc2x8rMLfqhF9OKv1QFB9fskRZcF/ENHITnmLqCJPg9rgmyv1gCQ5Tbds1OI+IYQIowdiwtaViiTSIy7/CedbHqMz6aDnPsJFMs24lcUXe2d8uZ26RqMinMFGXg0zF7tifuVmxj7zG8DT7MFORDm5f/Hhkng05R5XZesStTBxpEU0MGuzWhENt3RuJdstqmY+wkDVW3XwjUvKG/gVYuJYfnBAsRZIZezw2wfvodhc8EmeHELvY1YtB3DGTBXKE1lsKGN7yV5+nl4UtR2IGzBn9GovWVi5ooB8UlSAeUmZix/eAA5teL8ldFrDht7+AQ== jcollin@teuthology\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeApVHu5er+R9h52RLSVYVhXFQSn/nMN38uWoDYPw2bqYBLNEAAMcPl+rnqHubELpXTGXjJYL4n2mjic+2rOLZpDgPLmOOm4PwgYstJjE1KLegKqVzXIa9HfASKMXaHSgwtC6P5vDQkm2MNl8vXuTAITF7VYdDS33Kkk 2021-02-10T10:54:01.986 INFO:teuthology.task.ansible.out:e6ViJRZI7szYjbP7z9A1fy26THrvnuBT1Q18yy9vgk/sKo8vjLQfHSLvO90CJzouNEKqmkVpmJEqCPbE+s6Qr4+d2KY7L6Q6GAXxrqXK1jSDrrPsJguE6zZ1oWy/7XFMOr9ssduFWocnjQjPtQj3M+V+7FYAsf5uqQnzeMQTjO6AXYxTI7 jcollin@stratocaster\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDk/DYRp5E8Dla/4XHCF504X20W0u7CijZSCFWT1/5082NgnYeR90Prth3p+4xsw3YC9H5pEAS3XwcbdUb/NtWvZVIAIcMd7NOOEClSIj2Agk7NJxZ/4SfQRLrK3H2Qli/MVnrTxnEa9oeS4zqzv9dKRwFJDUFCoH2CpoumFVAp02uyzk7D/+XqFrXzMJUiSHXfl3YM2rt/vNmCtHtVp04UcG0Kyl2TXY37frRlSOkMMu0zbIC551gOyae6NGYhkYHa7u2SnngXcKM78b8MDvMv3QtuUN5P4WpH5qFRdlKNKsQGrzcQpBDY8AKQ+rX0K1ENhBwer49bihYNMKvjYmvTdLTNWgPCu7dt8JSgJQ8JMMhve0pjvFdpcKDLp0fvXDvJvZdLJM1ID/66kywJspHjShuqQNXRMfd1R9C4F20c+R9LY6eTGwrpLG1DNEez1VBHmOu5hQl03T2PfaThfLfMHRTw8JpGXGLMQarCCTm1PC6Z2Iw7KzorxP1eofGA5q8XoCbwGD01uQRZrwxPq0SHzTAkW2C2M8oEz+k4Mrx7+vGYzSS14hqxrzk91wuYhZCAKuGpqwm1ouJ4wxvWFewjbxLgucMfGSp09zJ2XSYRmtm7SAUifFob6Z9czHGTQerHCOTPuYqzlap2gCCVYtEvQyNB+4IqK/9FtEXTr28x0Q== jcollin@stratocaster", "key_options": null, "keyfile": "/home/jcollin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcollin", "validate_certs": true} 2021-02-10T10:54:02.412 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdxi9xdXNBYctMHH1cuuyNcPC4I4xVAeD3HZsKH+QwIILiKOTHu/6HZg7Yo1WsuwYSA0MDc+5LCSGu7RsmOEVpcSQ4gtNoFVwsThDyI6yhsm9WQqqXwSPs7ZE+urqd/BDfQ/7oMqJm5bViQMP0vmBExHuIFagj+8f6nspkE2Z7lBbaw8bEbAJZFx74oReOAELsL+Pbtf9Iaqrap4Z3WysmhfP6Rm/q7mncBBmo/fl/bejhThr8fghTPJb81M+XLhMk/rAdVrGdKV9iSt6lLiWdvf0lpTZO4LdIfJSBXr1Xv+lHRtaGXID2ahH7zl/W6dM4baz+hU7ZrbP+vdKg/SVH xxg@xxg", "key_options": null, "keyfile": "/home/xxg/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xxg", "validate_certs": true} 2021-02-10T10:54:02.435 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdxi9xdXNBYctMHH1cuuyNcPC4I4xVAeD3HZsKH+QwIILiKOTHu/6HZg7Yo1WsuwYSA0MDc+5LCSGu7RsmOEVpcSQ4gtNoFVwsThDyI6yhsm9WQqqXwSPs7ZE+urqd/BDfQ/7oMqJm5bViQMP0vmBExHuIFagj+8f6nspkE2Z7lBbaw8bEbAJZFx74oReOAELsL+Pbtf9Iaqrap4Z3WysmhfP6Rm/q7mncBBmo/fl/bejhThr8fghTPJb81M+XLhMk/rAdVrGdKV9iSt6lLiWdvf0lpTZO4LdIfJSBXr1Xv+lHRtaGXID2ahH7zl/W6dM4baz+hU7ZrbP+vdKg/SVH xxg@xxg", "key_options": null, "keyfile": "/home/xxg/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xxg", "validate_certs": true} 2021-02-10T10:54:02.876 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsA63hT3ZAwQ9hsvfY+o2Gy15M1GRnRw1ZBSXEiCs9OJku7T9+LCvvH85Y/7V+mOGcsCZKqVTd5Vy3sHAvAF3M7CYJ+5IIDJKCYslC0wkKcX4jj26aMToAZPms8uvs527BNgKOcvwgIfPjs4WJG06fey5AhsuuL30etbAvbBudxRAKdZSTOsMveneKtXQu3Dx7y0rjfI/RBAhCAvisnMqYFUdajoJIl5gVXl6+8mauUOP5O0mvyvJH2jQveqZXTdsIPyp5mq1nA5psqvivWGz2bEJ+owI0GjfuYbyAi6i+BL4Bsd3GeoAs1G3Dicfr228VIUsT+R3xLBQUj3xIsVAZ pcuzner@redhat.com", "key_options": null, "keyfile": "/home/pcuzner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pcuzner", "validate_certs": true} 2021-02-10T10:54:02.877 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsA63hT3ZAwQ9hsvfY+o2Gy15M1GRnRw1ZBSXEiCs9OJku7T9+LCvvH85Y/7V+mOGcsCZKqVTd5Vy3sHAvAF3M7CYJ+5IIDJKCYslC0wkKcX4jj26aMToAZPms8uvs527BNgKOcvwgIfPjs4WJG06fey5AhsuuL30etbAvbBudxRAKdZSTOsMveneKtXQu3Dx7y0rjfI/RBAhCAvisnMqYFUdajoJIl5gVXl6+8mauUOP5O0mvyvJH2jQveqZXTdsIPyp5mq1nA5psqvivWGz2bEJ+owI0GjfuYbyAi6i+BL4Bsd3GeoAs1G3Dicfr228VIUsT+R3xLBQUj3xIsVAZ pcuzner@redhat.com", "key_options": null, "keyfile": "/home/pcuzner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pcuzner", "validate_certs": true} 2021-02-10T10:54:03.327 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaRX+aBLjb/h12ctmvoMvZro6NrrYDuxROF6QMVpEVucA7jHhwCC/j9dJAq0hOVBO8R3S0bvKQ6bmnCb/4y2J6tq18xKUFCrChNii5OQ+AgM5v1353oZ9SrHN8p9ajvS8w313o2OpXhpaPv3khVDjKZsDmeH8YL/Nx3WQKqIJC1mI4jrI0sZGGJLifEy3pPkO0W1KmguN9AwAWK6+kckAjMa3ElkfK/xuAqBYgNKQUxP9oMgTQRcgpWF8Nzb0npN3v+HOUIRrth1INSchkDKtWob2T+0Zqxwm+Vpi0iFWwqjcN/OVdaDGf34//A3M/tsiQ1g9SqhdhZJCfu5XLmM63 mkogan@mkP50.redhat.com", "key_options": null, "keyfile": "/home/mkogan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkogan", "validate_certs": true} 2021-02-10T10:54:03.328 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaRX+aBLjb/h12ctmvoMvZro6NrrYDuxROF6QMVpEVucA7jHhwCC/j9dJAq0hOVBO8R3S0bvKQ6bmnCb/4y2J6tq18xKUFCrChNii5OQ+AgM5v1353oZ9SrHN8p9ajvS8w313o2OpXhpaPv3khVDjKZsDmeH8YL/Nx3WQKqIJC1mI4jrI0sZGGJLifEy3pPkO0W1KmguN9AwAWK6+kckAjMa3ElkfK/xuAqBYgNKQUxP9oMgTQRcgpWF8Nzb0npN3v+HOUIRrth1INSchkDKtWob2T+0Zqxwm+Vpi0iFWwqjcN/OVdaDGf34//A3M/tsiQ1g9SqhdhZJCfu5XLmM63 mkogan@mkP50.redhat.com", "key_options": null, "keyfile": "/home/mkogan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkogan", "validate_certs": true} 2021-02-10T10:54:03.771 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2021-02-10T10:54:03.772 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2021-02-10T10:54:04.263 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDZDKRAePFqjugc2ljw4QmM6r7B2dPI39OBB+2HKONtDZI4q+uwgZ1yqnhrbJaU9ux0W5wNzJyp2VfOPhw/9G9jo+efJPg5IySnTonhi1zeCXNpBMY70dJZVCyGEDdWKehvFM9xFBOxBC/h7gOyKqEc28BKoy4cHfULzm4er8PU30MyFBngu8NaoKqyQ3GYAfWOAXF5lUdalbJwl3a+HBb8PjUhbGuznAZF1Ji4JgnNuEZdlNVOKwhEkAFZ1JwXvgLfrEJVQs5b2V9Sp2Vm5ZngsLg7b3RzV5I4B3Lj7QTTraV1zLJGiS5iRdjY7D4W5Uxi2k1X7vaOjySJWWadHyeicNNBhLSfHl6JJ3aMuA8oAAmtgo/8IkJ96GFtgM9j31Wg8wkeTyQrwSeqCuv1uDFr8rp+OvMClhHnDgUklJLWjjYPCDVYR5KV+iOiHL58WJg/b4h6NsoP91wO8P8EcSaiMdq7P77w5nJxDI0KKb//dC8aHtWnRRRlrIS+PNPqKJSNTaSwOTieXvzplhzKFzu2dr79qs+/67iaNE6HBvIX5jVHWCFV3wllyjsC8WYPh4kEVtg2SY00SDOOrMrqANV8DDtHFwwxQTEjz1z5ESuwy9UcUSQfP1/zsAp2OQzPutKs6UipHyBK3pLm6MjaeeaEWu8fmetCqALxYV3h17O+Yw== miguel@hermes.olymp", "key_options": null, "keyfile": "/home/henrix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "henrix", "validate_certs": true} 2021-02-10T10:54:04.266 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDZDKRAePFqjugc2ljw4QmM6r7B2dPI39OBB+2HKONtDZI4q+uwgZ1yqnhrbJaU9ux0W5wNzJyp2VfOPhw/9G9jo+efJPg5IySnTonhi1zeCXNpBMY70dJZVCyGEDdWKehvFM9xFBOxBC/h7gOyKqEc28BKoy4cHfULzm4er8PU30MyFBngu8NaoKqyQ3GYAfWOAXF5lUdalbJwl3a+HBb8PjUhbGuznAZF1Ji4JgnNuEZdlNVOKwhEkAFZ1JwXvgLfrEJVQs5b2V9Sp2Vm5ZngsLg7b3RzV5I4B3Lj7QTTraV1zLJGiS5iRdjY7D4W5Uxi2k1X7vaOjySJWWadHyeicNNBhLSfHl6JJ3aMuA8oAAmtgo/8IkJ96GFtgM9j31Wg8wkeTyQrwSeqCuv1uDFr8rp+OvMClhHnDgUklJLWjjYPCDVYR5KV+iOiHL58WJg/b4h6NsoP91wO8P8EcSaiMdq7P77w5nJxDI0KKb//dC8aHtWnRRRlrIS+PNPqKJSNTaSwOTieXvzplhzKFzu2dr79qs+/67iaNE6HBvIX5jVHWCFV3wllyjsC8WYPh4kEVtg2SY00SDOOrMrqANV8DDtHFwwxQTEjz1z5ESuwy9UcUSQfP1/zsAp2OQzPutKs6UipHyBK3pLm6MjaeeaEWu8fmetCqALxYV3h17O+Yw== miguel@hermes.olymp", "key_options": null, "keyfile": "/home/henrix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "henrix", "validate_certs": true} 2021-02-10T10:54:04.736 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNwmgnIsDDJmsmYzkTAEo3iLj7Jd5Ni2R5vD1Z0NRZ39zJMtbicZ0ec3nAMzFyHrmPnKNFltS1mhehbLbtKNKq1fSo2SMHs11z+N8Nl94F4AY8IhXZ8U9GAorEJFVIxHUNEDqwl55Ov6MfP60UI7rNgy2rUt8aOBmAJVhLEFmJJawJgyPhTrUZAT20OoILphrt81o5bTNfmx++Zs5Dc9VtgIh53hexwKgTvG6fxfKdra9m+J2GXpniOsZjPJq/A0je+t7H3UlxCSu0dRy2Un6l5ZAcoeg74SP50dFgGaxkTQqI6wle6NjtPIdssgrdYTgTcOg9M4jWqAr/5drTUSGh root@centos7-bspark", "key_options": null, "keyfile": "/home/pbs1108/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pbs1108", "validate_certs": true} 2021-02-10T10:54:04.736 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNwmgnIsDDJmsmYzkTAEo3iLj7Jd5Ni2R5vD1Z0NRZ39zJMtbicZ0ec3nAMzFyHrmPnKNFltS1mhehbLbtKNKq1fSo2SMHs11z+N8Nl94F4AY8IhXZ8U9GAorEJFVIxHUNEDqwl55Ov6MfP60UI7rNgy2rUt8aOBmAJVhLEFmJJawJgyPhTrUZAT20OoILphrt81o5bTNfmx++Zs5Dc9VtgIh53hexwKgTvG6fxfKdra9m+J2GXpniOsZjPJq/A0je+t7H3UlxCSu0dRy2Un6l5ZAcoeg74SP50dFgGaxkTQqI6wle6NjtPIdssgrdYTgTcOg9M4jWqAr/5drTUSGh root@centos7-bspark", "key_options": null, "keyfile": "/home/pbs1108/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pbs1108", "validate_certs": true} 2021-02-10T10:54:05.185 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDnvVnbK2SNfL8hTmrA09L41RudKW/F7Ty0vvlHjrFjDkdWXVgrafFWGKQZ8dSR6FJ9JLU0X59xiEo4z31ini/Dt2XozFMIr7RdeB3xpiifuUUG5Lgsasd2ezniUoOEukL1a9KkCIxNgmKJuvtlHCt+ZTtyL513yjRICByZbnmBSSpUiett50X10vtvaNBu4LikUJKLc8gP0PHLbaGsU3uOUd655lkFWjWhgExlU8bHWiAZq18Ss63jDlSKFVSk903GKSNcHXMHNPYaXlrYw8NFzejoy/XjxEfmF/Lv3MeAUhHQWx25sh5aDv48mzo8FaIJsL1MrD0QxF1gllobWAfp clacroix@redhat.com", "key_options": null, "keyfile": "/home/clacroix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "clacroix", "validate_certs": true} 2021-02-10T10:54:05.196 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDnvVnbK2SNfL8hTmrA09L41RudKW/F7Ty0vvlHjrFjDkdWXVgrafFWGKQZ8dSR6FJ9JLU0X59xiEo4z31ini/Dt2XozFMIr7RdeB3xpiifuUUG5Lgsasd2ezniUoOEukL1a9KkCIxNgmKJuvtlHCt+ZTtyL513yjRICByZbnmBSSpUiett50X10vtvaNBu4LikUJKLc8gP0PHLbaGsU3uOUd655lkFWjWhgExlU8bHWiAZq18Ss63jDlSKFVSk903GKSNcHXMHNPYaXlrYw8NFzejoy/XjxEfmF/Lv3MeAUhHQWx25sh5aDv48mzo8FaIJsL1MrD0QxF1gllobWAfp clacroix@redhat.com", "key_options": null, "keyfile": "/home/clacroix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "clacroix", "validate_certs": true} 2021-02-10T10:54:05.629 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAQEAyiGmj+nOAD+il+FvXvZSpUOJQp6hKh6UPaAIQvQZkdeNRWN7wcDXMUlLj+TBFWa1OO+9xqFlnbgVMDR4qy5orUBYlMapUb1rutjbk64XeY58b6uguHwSoMd81QpCCIN4KOk3cldV8UiyX382baoZ9Je2hsOnnBGMsRtHuVHZ90Bj1rIITkwajq//KRs7kjb/qhjk0xkipDIcaSwKjzPlmmnYn69noODwDexpvLYlCF+LowSZEdrOUvVIigSFG7cLTDaU2tLhxSgbvWu8f0pwvjsyW7gOgM+VeRZ6uHC79F1qvgdgQSxflddIgrQpH0rzf6RNxQjR3SBkZ4V1BESa0w== epuertat@epuertat", "key_options": null, "keyfile": "/home/epuertat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "epuertat", "validate_certs": true} 2021-02-10T10:54:05.641 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAQEAyiGmj+nOAD+il+FvXvZSpUOJQp6hKh6UPaAIQvQZkdeNRWN7wcDXMUlLj+TBFWa1OO+9xqFlnbgVMDR4qy5orUBYlMapUb1rutjbk64XeY58b6uguHwSoMd81QpCCIN4KOk3cldV8UiyX382baoZ9Je2hsOnnBGMsRtHuVHZ90Bj1rIITkwajq//KRs7kjb/qhjk0xkipDIcaSwKjzPlmmnYn69noODwDexpvLYlCF+LowSZEdrOUvVIigSFG7cLTDaU2tLhxSgbvWu8f0pwvjsyW7gOgM+VeRZ6uHC79F1qvgdgQSxflddIgrQpH0rzf6RNxQjR3SBkZ4V1BESa0w== epuertat@epuertat", "key_options": null, "keyfile": "/home/epuertat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "epuertat", "validate_certs": true} 2021-02-10T10:54:06.093 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDRZyWiDO5JqWHJdUUBr8YReiKXxnL87juFAfkcmw2E9KmANVH2sn3K+OgUCQfBs1Wdg6AbcL2GyMm9alodUruzOm127/bH9OcGmu+8SNsMAYrAF9EzsKJa8Ba/I05rl5a87p1jbYG9Rpo7u3OvLx0LjgqydRYcmjwo/niifYNDTaN0K3v8s1mN4TorUZW/i3XBMBIBjlIiNGs/I7IZreR+1N4V8Nwrp2EWxDv7AwG3nvokL2RTHcsV0KgmAnciv8nXXjIf+uqOPuGA+F2qW/wz92bG8R0O4CjH+QZt4k6UoUIXFRW+LdOVAqmUrb4W0OJ8vsVeukW46SvPnYUTkZnF sonea@think", "key_options": null, "keyfile": "/home/tdehler/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tdehler", "validate_certs": true} 2021-02-10T10:54:06.094 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDRZyWiDO5JqWHJdUUBr8YReiKXxnL87juFAfkcmw2E9KmANVH2sn3K+OgUCQfBs1Wdg6AbcL2GyMm9alodUruzOm127/bH9OcGmu+8SNsMAYrAF9EzsKJa8Ba/I05rl5a87p1jbYG9Rpo7u3OvLx0LjgqydRYcmjwo/niifYNDTaN0K3v8s1mN4TorUZW/i3XBMBIBjlIiNGs/I7IZreR+1N4V8Nwrp2EWxDv7AwG3nvokL2RTHcsV0KgmAnciv8nXXjIf+uqOPuGA+F2qW/wz92bG8R0O4CjH+QZt4k6UoUIXFRW+LdOVAqmUrb4W0OJ8vsVeukW46SvPnYUTkZnF sonea@think", "key_options": null, "keyfile": "/home/tdehler/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tdehler", "validate_certs": true} 2021-02-10T10:54:06.533 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEBD7CxJJfbLW4jtuLnTwkxVbhCfh/JL3Qj8oeLEkny1tE7QCeXn1bP5bScWSgLH34Wh4TAuH+8/g/koFwcCy1YuoGBdBV4KV7O9MTnFngdqNY6LKzCipm27tnWZEa10hrF8dRFTmhracd6kmmVGHgvGQPVAVONmix+2IZ83wdj8XJSQHeE3PHFLO39tU+ez/T76QaiBLDVo4ctHVt+N9bBbHV6Cr8fyiUUU6YEKMlcedvxGpnMK8oeOSNuDeh+DldcM9KYzNGEOM34jGRIvJp57dxQDQCtPPW5hvBXg7yMyrv6TcJzDd2LFD6AC/Qw9yrEQPXXyBCDkAlCqDR0+rH", "key_options": null, "keyfile": "/home/laura/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "laura", "validate_certs": true} 2021-02-10T10:54:06.534 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEBD7CxJJfbLW4jtuLnTwkxVbhCfh/JL3Qj8oeLEkny1tE7QCeXn1bP5bScWSgLH34Wh4TAuH+8/g/koFwcCy1YuoGBdBV4KV7O9MTnFngdqNY6LKzCipm27tnWZEa10hrF8dRFTmhracd6kmmVGHgvGQPVAVONmix+2IZ83wdj8XJSQHeE3PHFLO39tU+ez/T76QaiBLDVo4ctHVt+N9bBbHV6Cr8fyiUUU6YEKMlcedvxGpnMK8oeOSNuDeh+DldcM9KYzNGEOM34jGRIvJp57dxQDQCtPPW5hvBXg7yMyrv6TcJzDd2LFD6AC/Qw9yrEQPXXyBCDkAlCqDR0+rH", "key_options": null, "keyfile": "/home/laura/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "laura", "validate_certs": true} 2021-02-10T10:54:06.975 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2k8xsrsLxEqJx3rV6ZZwLH0OjYCDB2uIYzcAyrKEOKlVWO1ICKxkw+gZ9tSiPFNLfq13io9BIrAKJwlrdLCJLmlgCQwpwUQqzmRlRZEm+WmD0dl8LPeQSidyJkPkJgYBEshFPjDHIj10Ei1pJS5svI2l7G1wMWxlt2kHL8Og/4zdFTRctxwqZcvzLmRQcs8lSSpqTkhfeKfmrpR+je8XFbCJJ9m/nJbgyCiZKaGN8nf36pCQDEva4rvaSrK0a6YxL0RWN8W3lqtBPKIehxffr3lYDEkD9EUW5vf6AUKC0ioh4jkPo9WCjaoFsOauMOMVQPHehEAMufEO4Kym6nfrf yaarit@localhost.localdomain", "key_options": null, "keyfile": "/home/yaarit/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yaarit", "validate_certs": true} 2021-02-10T10:54:06.976 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2k8xsrsLxEqJx3rV6ZZwLH0OjYCDB2uIYzcAyrKEOKlVWO1ICKxkw+gZ9tSiPFNLfq13io9BIrAKJwlrdLCJLmlgCQwpwUQqzmRlRZEm+WmD0dl8LPeQSidyJkPkJgYBEshFPjDHIj10Ei1pJS5svI2l7G1wMWxlt2kHL8Og/4zdFTRctxwqZcvzLmRQcs8lSSpqTkhfeKfmrpR+je8XFbCJJ9m/nJbgyCiZKaGN8nf36pCQDEva4rvaSrK0a6YxL0RWN8W3lqtBPKIehxffr3lYDEkD9EUW5vf6AUKC0ioh4jkPo9WCjaoFsOauMOMVQPHehEAMufEO4Kym6nfrf yaarit@localhost.localdomain", "key_options": null, "keyfile": "/home/yaarit/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yaarit", "validate_certs": true} 2021-02-10T10:54:07.414 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDUznqo3VDmcKRL64P7sbjuxXCW/0ljFTc5zZoZ9UvtZvU2jrRQ2X9GyxcrcQnAWlF5YZHqeTDU5o8ixn5fq1MSmaPRBNDKKVQOfR/vFQV73LiKx4BFsC1/EdaPie/XqltaLDTtfqhxADJlz1H3a3ZlkeVHMROTqo/TidL1YUYlRH4qHxWWBuFuVS+EYfTegSZdjwp+DuJA0iN6X99+DfH2DhiHENjEmlbKN13+H+EPzm0K3ya2w8D6ANodB3AxMn2tGs8mLtG/P4kCjoDnoUVFzKTyUpTtgOjis47uSY4dL8mTbIK9kZAiFxnT5sMClu6KRStDjRIrxbTBQJ8wvlat admin1@rajula", "key_options": null, "keyfile": "/home/rpavani1998/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rpavani1998", "validate_certs": true} 2021-02-10T10:54:07.414 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDUznqo3VDmcKRL64P7sbjuxXCW/0ljFTc5zZoZ9UvtZvU2jrRQ2X9GyxcrcQnAWlF5YZHqeTDU5o8ixn5fq1MSmaPRBNDKKVQOfR/vFQV73LiKx4BFsC1/EdaPie/XqltaLDTtfqhxADJlz1H3a3ZlkeVHMROTqo/TidL1YUYlRH4qHxWWBuFuVS+EYfTegSZdjwp+DuJA0iN6X99+DfH2DhiHENjEmlbKN13+H+EPzm0K3ya2w8D6ANodB3AxMn2tGs8mLtG/P4kCjoDnoUVFzKTyUpTtgOjis47uSY4dL8mTbIK9kZAiFxnT5sMClu6KRStDjRIrxbTBQJ8wvlat admin1@rajula", "key_options": null, "keyfile": "/home/rpavani1998/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rpavani1998", "validate_certs": true} 2021-02-10T10:54:07.855 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4NYxUHx8HMgbVIHadruN1kAJS5be0aZA9rvGtYDfwD5siKNfXFRyLG3thkjxtEi7DPYMxgbpFOXW9EQKh5sQ0ohsADZOi8FSIAfQlOQ+7HAEp7DjBtjxoqB31jJ1e293YtO+nqAhcOMyGfDx+71w5sS2EFqy7PscA9gD9aV2iMxIR/JD1LxXabiqj12jU5AOxr7ZQ1+gM9aroQuDpNxY4U8jBTxx5OtIYRkeI/aN2gQj1mXGMKGo0ItFn4lsXGLxGxMpa9K2UFw2dWQVTw9VNm79nixrCX+lC2nyaJPsRYZipXF1ID2NRk7oJ79dJ/uRBBXpunihlEC3aF+nOgEPH ridave@redhat.com", "key_options": null, "keyfile": "/home/rishabh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rishabh", "validate_certs": true} 2021-02-10T10:54:07.857 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4NYxUHx8HMgbVIHadruN1kAJS5be0aZA9rvGtYDfwD5siKNfXFRyLG3thkjxtEi7DPYMxgbpFOXW9EQKh5sQ0ohsADZOi8FSIAfQlOQ+7HAEp7DjBtjxoqB31jJ1e293YtO+nqAhcOMyGfDx+71w5sS2EFqy7PscA9gD9aV2iMxIR/JD1LxXabiqj12jU5AOxr7ZQ1+gM9aroQuDpNxY4U8jBTxx5OtIYRkeI/aN2gQj1mXGMKGo0ItFn4lsXGLxGxMpa9K2UFw2dWQVTw9VNm79nixrCX+lC2nyaJPsRYZipXF1ID2NRk7oJ79dJ/uRBBXpunihlEC3aF+nOgEPH ridave@redhat.com", "key_options": null, "keyfile": "/home/rishabh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rishabh", "validate_certs": true} 2021-02-10T10:54:08.297 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCeAhoTum3/VmRbxSyw5gwgcPF1EWFK5mEPX3PZyo3xPsUAJazALfge3aEuCrZBbsjOCoj3GfFhr/ziEF+OWvhBoPK5hQzP0V8tE9cWJwvHmD72vGE36o1hFO0ONE4DEuu5KRDqQJUgeyQWXsP26uHTwIcvMpy8egZOfyshqF+lQVXzsAex7Ninmidsf/sRNLhzT1kLrKT+ZMSbZ7Rqh46TMlmLD1LzJZNcb6PGVwlfQZmV6T7m0COu/skeSxMcy8hqMvHFNvvfFSqkfMw0KBGLDxMQr4L+e7sXwQDnEZHdzoh0iS3xDp/Onr1FSmPyEjz51e1sMHBsqklHWidSZVLt/+giB4o6Wdgzc0um/QxW2Ne2qjtDlViOgRE9Fzcw/cRu/Xsn8qtXTxYyxXswjkA8qX7WR22OYgjYChw9siUsKkuD2yAbPMVl+cZWDeBhQnPMbDMVaghS4SAVt5GK16jij4+r3IE9vGHBmGwJYJ/OrtcxBrUrDopwsGLcU+WHdv0FXuiw+YbfnzpthC8rVj4oeJ2IUOQ5JB0ub7NmbR4QWjc33JfD+AeCB75x6io07V2riRjMD9gLC5nwytJNG7Yytf6xI2i/CprW7HhsDs2ykOVcDSBMnL1oGW6FeHaDRW39r6BXSt1jjZQ2E6pDEBTDO41tPEL9EVvv9TK87GTylQ==", "key_options": null, "keyfile": "/home/skrah/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "skrah", "validate_certs": true} 2021-02-10T10:54:08.298 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCeAhoTum3/VmRbxSyw5gwgcPF1EWFK5mEPX3PZyo3xPsUAJazALfge3aEuCrZBbsjOCoj3GfFhr/ziEF+OWvhBoPK5hQzP0V8tE9cWJwvHmD72vGE36o1hFO0ONE4DEuu5KRDqQJUgeyQWXsP26uHTwIcvMpy8egZOfyshqF+lQVXzsAex7Ninmidsf/sRNLhzT1kLrKT+ZMSbZ7Rqh46TMlmLD1LzJZNcb6PGVwlfQZmV6T7m0COu/skeSxMcy8hqMvHFNvvfFSqkfMw0KBGLDxMQr4L+e7sXwQDnEZHdzoh0iS3xDp/Onr1FSmPyEjz51e1sMHBsqklHWidSZVLt/+giB4o6Wdgzc0um/QxW2Ne2qjtDlViOgRE9Fzcw/cRu/Xsn8qtXTxYyxXswjkA8qX7WR22OYgjYChw9siUsKkuD2yAbPMVl+cZWDeBhQnPMbDMVaghS4SAVt5GK16jij4+r3IE9vGHBmGwJYJ/OrtcxBrUrDopwsGLcU+WHdv0FXuiw+YbfnzpthC8rVj4oeJ2IUOQ5JB0ub7NmbR4QWjc33JfD+AeCB75x6io07V2riRjMD9gLC5nwytJNG7Yytf6xI2i/CprW7HhsDs2ykOVcDSBMnL1oGW6FeHaDRW39r6BXSt1jjZQ2E6pDEBTDO41tPEL9EVvv9TK87GTylQ==", "key_options": null, "keyfile": "/home/skrah/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "skrah", "validate_certs": true} 2021-02-10T10:54:08.750 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDklgWHrL9TMaMxIyN9MxVR6nR3xh9zfj0iR63/yOzaslX0WoVBk7CYTsRlH2GFVctvixVwrWbQgMaKe49utal/3FA6sMl8S2DTrtgC1a/exUTvU0U/+ZhyUlkyjQfRgtYOUwodNupL5NsC8VZqZshgmgXC34uye3bplx3VwvpUt9r6QxJN3/9Dbi6HxudhDCLfmvROhwQYHaFF+GglcbyjAt3uvsuwXVKBd1/vMQwbcUo9/zQlrnmWBRuqTNkXGrZauWigoi/hyuWYQ3MUbRc3LVpJncMv7BEtcj8fCVvPB09SFee3nSeTtFkm01GvRcYd3HBz/YL4Rz92wXl9VIxqs8aaJBzy+MD0NO496Rfqtz6YuZ1m7AxDIccDW+OpcPgYwaUKFXNTuVLWZXhkS0p4swefcKWRlQFF3BLOebN/Kr9jq1MFa+5pX7Gj4WF5uCIX0/fx40KVZNhfl3qywBy/d5MKwXO7PCHMiGcu2FIXdZ334MxdgiIVoX54PpRP4hs= smanjara@localhost.localdomain", "key_options": null, "keyfile": "/home/smanjara/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smanjara", "validate_certs": true} 2021-02-10T10:54:08.750 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDklgWHrL9TMaMxIyN9MxVR6nR3xh9zfj0iR63/yOzaslX0WoVBk7CYTsRlH2GFVctvixVwrWbQgMaKe49utal/3FA6sMl8S2DTrtgC1a/exUTvU0U/+ZhyUlkyjQfRgtYOUwodNupL5NsC8VZqZshgmgXC34uye3bplx3VwvpUt9r6QxJN3/9Dbi6HxudhDCLfmvROhwQYHaFF+GglcbyjAt3uvsuwXVKBd1/vMQwbcUo9/zQlrnmWBRuqTNkXGrZauWigoi/hyuWYQ3MUbRc3LVpJncMv7BEtcj8fCVvPB09SFee3nSeTtFkm01GvRcYd3HBz/YL4Rz92wXl9VIxqs8aaJBzy+MD0NO496Rfqtz6YuZ1m7AxDIccDW+OpcPgYwaUKFXNTuVLWZXhkS0p4swefcKWRlQFF3BLOebN/Kr9jq1MFa+5pX7Gj4WF5uCIX0/fx40KVZNhfl3qywBy/d5MKwXO7PCHMiGcu2FIXdZ334MxdgiIVoX54PpRP4hs= smanjara@localhost.localdomain", "key_options": null, "keyfile": "/home/smanjara/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smanjara", "validate_certs": true} 2021-02-10T10:54:09.188 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDpp2NWr5T6s+jzHiz/FfCKzN7ZBnUSzyE7ieuntzjeGotyupAAjVBoSpCJgNDybm+WkE4dShv5mttPJ2/q1UZCLRj3nN+y3NG333aLZuyugR0ht5qtvGVegg58PwQ8CMobR7ca7bl5UdEy0UIrM3d/qpDOuPkuE4MIJcrtZGICE8NAVnmHNOB9Sm4ovAYkewTuMAv40j2Dl4F9dD6bg+OJfBOJ9+Zj+MpowYlAILtEB9Dc20zB8GGUEtr9yfOVroblTYhvTxGYRaE7aVBe83DNwkWGbDc/qUNR6bpA04w/P+LclYVsVKRoHOs1EXHSwNMFromA8RsKS+rajwu5uRg5 ben@dhcp-185-182.bos.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC93wGgyyQmTxa7f+BZcYPUapTH2N1mnV1ikl9nzJzHhgrtJ6pv00LhtH1HqGhvoqnqZjBD5XRAKyjMRcMBT5MYcXXep5RqZ9sJ6Xz8rHkBjfl/OOQUeZ/ZyG/Ksr1jpGraN2xtrjQQzbqN1YvirCqfLk2exPvVeEk4CGnNybwKwCWob1uKV46ASrYeQK8pfHxrW7pTo+xdErHOvb7uumZm89/YisUJGicO1qSHBeuw1MxjhnD+fIcxHKWY5NNJ4qScW6a5AABk1JJsoHkh9IGCVaf/zJv8D3gil5SiM8LIBKaNa1fn6oObPl/Ky/wkiIo2gmUZawvAVA8iJnX9eUPl ben@bene-laptop", "key_options": null, "keyfile": "/home/bengland/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bengland", "validate_certs": true} 2021-02-10T10:54:09.199 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDpp2NWr5T6s+jzHiz/FfCKzN7ZBnUSzyE7ieuntzjeGotyupAAjVBoSpCJgNDybm+WkE4dShv5mttPJ2/q1UZCLRj3nN+y3NG333aLZuyugR0ht5qtvGVegg58PwQ8CMobR7ca7bl5UdEy0UIrM3d/qpDOuPkuE4MIJcrtZGICE8NAVnmHNOB9Sm4ovAYkewTuMAv40j2Dl4F9dD6bg+OJfBOJ9+Zj+MpowYlAILtEB9Dc20zB8GGUEtr9yfOVroblTYhvTxGYRaE7aVBe83DNwkWGbDc/qUNR6bpA04w/P+LclYVsVKRoHOs1EXHSwNMFromA8RsKS+rajwu5uRg5 ben@dhcp-185-182.bos.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC93wGgyyQmTxa7f+BZcYPUapTH2N1mnV1ikl9nzJzHhgrtJ6pv00LhtH1HqGhvoqnqZjBD5XRAKyjMRcMBT5MYcXXep5RqZ9sJ6Xz8rHkBjfl/OOQUeZ/ZyG/Ksr1jpGraN2xtrjQQzbqN1YvirCqfLk2exPvVeEk4CGnNybwKwCWob1uKV46ASrYeQK8pfHxrW7pTo+xdErHOvb7uumZm89/YisUJGicO1qSHBeuw1MxjhnD+fIcxHKWY5NNJ4qScW6a5AABk1JJsoHkh9IGCVaf/zJv8D3gil5SiM8LIBKaNa1fn6oObPl/Ky/wkiIo2gmUZawvAVA8iJnX9eUPl ben@bene-laptop", "key_options": null, "keyfile": "/home/bengland/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bengland", "validate_certs": true} 2021-02-10T10:54:09.649 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC3YhI4am/T6s4rJRvbC/jAvrNcDhE/zaJiVDAjEuA1W/5Xt9ObX52eAnp+tOOtelXrX8ylYF/z5wGNQLjYrWepXhuqaEGxeVayCHzts6LMpmvyZ6sYnAw9HFO9rzrTawm70DuvWqY2Zt4KA6fM+lfXk3Qdpi5RCPDKk1jpXwSCyUtUKSWTDRtYtvvJTcO3H/xCJxQq/F9sU8yqgQcqFPfMNrV4a8gNj9KACqjt4IEIfuxKlKhlMA4yu1VTmGcNuhQS+geJzsEGuxZUQb0Io+q4o0b+40BQtazKgOdTcQbgNtqh3wZpjJsvB4AM+GNPjZY3mCsd+WPGg6dqfU08Voo2b7BfSwyCn14fQHnj1nSwRIea2juFY6AMfVJjstM5iIz7lOH2c3ydWWhzvSmMeoQ47zBXlg8JytWmDrXiyd21BSIpTL0Tn8P95wHYka8U9KRV0ZY7Jr2w/ThPdy2C+Cdir+P+OEL1sKHSnmEyQEeONa9P5yH1Rd1kM/FNESAIRiMLGspzXQeWoJhHswVaKoSBW/7f4b9VENxda8qAQi6ob8A71cAjbpRrkUmd5EBCSFMNs7lscus0/deUVvlpsrMjDAJa8S9gqhGHNU0YvF1WBIz3paLRAHDm9DWrHc5lreizJzX5ul4P/Z4ITjpO+9ZxKOcyi8wnJ8Og+QUYLpqOIw==", "key_options": null, "keyfile": "/home/pnawracay/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pnawracay", "validate_certs": true} 2021-02-10T10:54:09.649 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC3YhI4am/T6s4rJRvbC/jAvrNcDhE/zaJiVDAjEuA1W/5Xt9ObX52eAnp+tOOtelXrX8ylYF/z5wGNQLjYrWepXhuqaEGxeVayCHzts6LMpmvyZ6sYnAw9HFO9rzrTawm70DuvWqY2Zt4KA6fM+lfXk3Qdpi5RCPDKk1jpXwSCyUtUKSWTDRtYtvvJTcO3H/xCJxQq/F9sU8yqgQcqFPfMNrV4a8gNj9KACqjt4IEIfuxKlKhlMA4yu1VTmGcNuhQS+geJzsEGuxZUQb0Io+q4o0b+40BQtazKgOdTcQbgNtqh3wZpjJsvB4AM+GNPjZY3mCsd+WPGg6dqfU08Voo2b7BfSwyCn14fQHnj1nSwRIea2juFY6AMfVJjstM5iIz7lOH2c3ydWWhzvSmMeoQ47zBXlg8JytWmDrXiyd21BSIpTL0Tn8P95wHYka8U9KRV0ZY7Jr2w/ThPdy2C+Cdir+P+OEL1sKHSnmEyQEeONa9P5yH1Rd1kM/FNESAIRiMLGspzXQeWoJhHswVaKoSBW/7f4b9VENxda8qAQi6ob8A71cAjbpRrkUmd5EBCSFMNs7lscus0/deUVvlpsrMjDAJa8S9gqhGHNU0YvF1WBIz3paLRAHDm9DWrHc5lreizJzX5ul4P/Z4ITjpO+9ZxKOcyi8wnJ8Og+QUYLpqOIw==", "key_options": null, "keyfile": "/home/pnawracay/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pnawracay", "validate_certs": true} 2021-02-10T10:54:10.090 INFO:teuthology.task.ansible.out:changed: [gibba002.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDQu/CuhCIFVGCLbA0cjn+qqcwX12cRcFA1sHyJtkj6Ltfmge8FnfgAsex/LaPJwbtbBiVl9xRcazHNfBaDw5DxMRSJF+e7WTydZC2Jm4qIj4D3Lr+imGPrJTK+kjN0QXRXknGsNn36oRF/I41cGqYRGwawFGSPZBHuav+ehDk1doTrbDlll4zMUWm2iXTcBIcJo4NR1LoE2c9/MPfbH7jKm5ke7aS0ets+EANEplkeClfCVAashtIF+Bz6a9qk4F2n0baLT/s3IQkXCwwO2fxxY03x8gWiB1n4E7sousEB+J1Skl/z0bwmaiY0C+raPoV/FyZxQuh0ocvdU7fnQbKuNoGE/Usfm9Y3g8eH81b/k4fMJL2lqXHkPft6zUV9bJsp3v1iGEkIf2+ER/QLczteNz6kw/uTPQctMEDXM17GEnBXISVrQcPSW3i93eFwWacs8iT3G4bjEQvpxSBQrhDdCCoxy8bVWqkVeDWw5u2q/3WL9Ov1NrY+G4/DYb2qONNQds0vTHCGvemU5GNjtaeCbojkZ7Z3KOeAY8080kKrsO7fve+ZMACCIFD+se8kF7NN4ZJASSqq9lJpBWvtbc0O8jCcVGNuN7exXjrcbiN1L/9HkARV+KEJU3jbXObAhAnfAaeRWBwmTfZodCu+fDd/Jor6l36tjD4kC4/CPrDtyQ== alfonsomthd", "key_options": null, "keyfile": "/home/alfonsomthd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "alfonsomthd", "validate_certs": true} 2021-02-10T10:54:10.091 INFO:teuthology.task.ansible.out:changed: [gibba013.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDQu/CuhCIFVGCLbA0cjn+qqcwX12cRcFA1sHyJtkj6Ltfmge8FnfgAsex/LaPJwbtbBiVl9xRcazHNfBaDw5DxMRSJF+e7WTydZC2Jm4qIj4D3Lr+imGPrJTK+kjN0QXRXknGsNn36oRF/I41cGqYRGwawFGSPZBHuav+ehDk1doTrbDlll4zMUWm2iXTcBIcJo4NR1LoE2c9/MPfbH7jKm5ke7aS0ets+EANEplkeClfCVAashtIF+Bz6a9qk4F2n0baLT/s3IQkXCwwO2fxxY03x8gWiB1n4E7sousEB+J1Skl/z0bwmaiY0C+raPoV/FyZxQuh0ocvdU7fnQbKuNoGE/Usfm9Y3g8eH81b/k4fMJL2lqXHkPft6zUV9bJsp3v1iGEkIf2+ER/QLczteNz6kw/uTPQctMEDXM17GEnBXISVrQcPSW3i93eFwWacs8iT3G4bjEQvpxSBQrhDdCCoxy8bVWqkVeDWw5u2q/3WL9Ov1NrY+G4/DYb2qONNQds0vTHCGvemU5GNjtaeCbojkZ7Z3KOeAY8080kKrsO7fve+ZMACCIFD+se8kF7NN4ZJASSqq9lJpBWvtbc0O8jCcVGNuN7exXjrcbiN1L/9HkARV+KEJU3jbXObAhAnfAaeRWBwmTfZodCu+fDd/Jor6l36tjD4kC4/CPrDtyQ== alfonsomthd", "key_options": null, "keyfile": "/home/alfonsomthd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "alfonsomthd", "validate_certs": true} 2021-02-10T10:54:10.533 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCoL9+aMxLtUG3aRIuT4kbjc1CPuQ+wbS30ocuJ2YIExlKMSxVr+LYqbjcz0T4f8gAKG539dZQgD9BAsHM8yfOVEvei0KsQHn87pzgYGR3DtaMx5IEG5xNyARPu+VuEoe8qlL3cAPwTiaU78datD5Cz9GEClhL1Fhn/Z8o7BMbs5xlcUIQ/J+s9/sKMcqmmBJy5IHQIEUHCl4SFwsev5g5ADlnLNNxjpalExuBCEeoSma9F3b2YEJHvpA5JhCIMZpVfXbgh2UhUsyfAQN6KgOIQ/XlKBMQDtizf2vZ4b8hW4ELcvKSADfTsyyqEs/Ql8jEJu32GFKf2mYPbgTS6E7FZfQz77X+5OBhojFAstB8LSSrH80FrG1gCbVZLnW0jXcSehKnU0oNmUWOKxxaT1b00fo/iZ4P8O1uJRY17/uaVCPNKdGTZzIlzyC16wsFFBxBl0d7HfojQY6qfww7AQJhvDkV2sP0/MIJzwriWOzZEmfr8Osqery5vFiYcSvMbdMZDfMEYd3z7Va2RiTIeyXuEYp5hzf8eyJRP/f2FEw8/BBrm5JCxeSWUx7qwgPHrJoWAVxCIg/M3YS/vfVSFc8Hh88sgc2upq0OqYr0uq6EFzUZi4aoOkOVYn9u/uc/p1AR0KbiQOkERMUDWTyADxrHaL0nQmGy+XyiJQjhq5ckZ+w== doliveira@suse.com", "key_options": null, "keyfile": "/home/oliveiradan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "oliveiradan", "validate_certs": true} 2021-02-10T10:54:10.544 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCoL9+aMxLtUG3aRIuT4kbjc1CPuQ+wbS30ocuJ2YIExlKMSxVr+LYqbjcz0T4f8gAKG539dZQgD9BAsHM8yfOVEvei0KsQHn87pzgYGR3DtaMx5IEG5xNyARPu+VuEoe8qlL3cAPwTiaU78datD5Cz9GEClhL1Fhn/Z8o7BMbs5xlcUIQ/J+s9/sKMcqmmBJy5IHQIEUHCl4SFwsev5g5ADlnLNNxjpalExuBCEeoSma9F3b2YEJHvpA5JhCIMZpVfXbgh2UhUsyfAQN6KgOIQ/XlKBMQDtizf2vZ4b8hW4ELcvKSADfTsyyqEs/Ql8jEJu32GFKf2mYPbgTS6E7FZfQz77X+5OBhojFAstB8LSSrH80FrG1gCbVZLnW0jXcSehKnU0oNmUWOKxxaT1b00fo/iZ4P8O1uJRY17/uaVCPNKdGTZzIlzyC16wsFFBxBl0d7HfojQY6qfww7AQJhvDkV2sP0/MIJzwriWOzZEmfr8Osqery5vFiYcSvMbdMZDfMEYd3z7Va2RiTIeyXuEYp5hzf8eyJRP/f2FEw8/BBrm5JCxeSWUx7qwgPHrJoWAVxCIg/M3YS/vfVSFc8Hh88sgc2upq0OqYr0uq6EFzUZi4aoOkOVYn9u/uc/p1AR0KbiQOkERMUDWTyADxrHaL0nQmGy+XyiJQjhq5ckZ+w== doliveira@suse.com", "key_options": null, "keyfile": "/home/oliveiradan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "oliveiradan", "validate_certs": true} 2021-02-10T10:54:10.989 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDh9bzZJulXGES+l9Xh6Aq15RZ8uQCGuDNhlNQtDblE1ISKJ/DqGYXp6wUW54+oVNA7eZiXz+fi3mq5pPEtZZOfd3ixEDzDJ4E3cVXoDZCqeWEmea6KvybeY10YxvB56TEI8U2KKAd56PRl20klXpLCjzXNqG7n0aXFcpCMbXDu369VX6lOk24K/7++7Wc5SttcvVu19sT2kqzsB/S1Y5cxiE6RM6wVtqBoksp/kRCIA16ruNwx3GUabDfbEoUGXlkkUP7+TZgAbHtBsYy6mCQhwi0S2+WG+HhHDUPjHhV+MdN9ffibCtOEGo52itVLVky09VeBocuA6H22JDGXPBgjcgf2NsZIqcKqGHhUkXmH92fhRSFOBLKHstrBq8jWRP/mNrgj8cQksDsakQYQbDg5dyabp+M0/iL2Q3YVq7erZI8aZMA7ZF3WgoQNYZg5E7oejM8URIlFP3x1ne2ClRC9a74phSCxeU/NVamGN3G3dImzEXGOSNyRggHJ4jGrIGc7tLPCzmI5OkomcB5OxReqf0r1TNXuUAqw8M4EoWt+0xoAmH5zVlUHf+psUCJIEV/4pbgtoJiSNq+LVY4jyDEFbvTAL7MqyXorMV7Tqlj+/3d7RXfhW9lR/SHb1Z3jcHvz4ZzYMzKWeJEjz+Y0NwIdFDhcPmUOYtEDmuRRrgYiSQ== sebstian.wagner@it-novum.com", "key_options": null, "keyfile": "/home/swagner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "swagner", "validate_certs": true} 2021-02-10T10:54:10.990 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDh9bzZJulXGES+l9Xh6Aq15RZ8uQCGuDNhlNQtDblE1ISKJ/DqGYXp6wUW54+oVNA7eZiXz+fi3mq5pPEtZZOfd3ixEDzDJ4E3cVXoDZCqeWEmea6KvybeY10YxvB56TEI8U2KKAd56PRl20klXpLCjzXNqG7n0aXFcpCMbXDu369VX6lOk24K/7++7Wc5SttcvVu19sT2kqzsB/S1Y5cxiE6RM6wVtqBoksp/kRCIA16ruNwx3GUabDfbEoUGXlkkUP7+TZgAbHtBsYy6mCQhwi0S2+WG+HhHDUPjHhV+MdN9ffibCtOEGo52itVLVky09VeBocuA6H22JDGXPBgjcgf2NsZIqcKqGHhUkXmH92fhRSFOBLKHstrBq8jWRP/mNrgj8cQksDsakQYQbDg5dyabp+M0/iL2Q3YVq7erZI8aZMA7ZF3WgoQNYZg5E7oejM8URIlFP3x1ne2ClRC9a74phSCxeU/NVamGN3G3dImzEXGOSNyRggHJ4jGrIGc7tLPCzmI5OkomcB5OxReqf0r1TNXuUAqw8M4EoWt+0xoAmH5zVlUHf+psUCJIEV/4pbgtoJiSNq+LVY4jyDEFbvTAL7MqyXorMV7Tqlj+/3d7RXfhW9lR/SHb1Z3jcHvz4ZzYMzKWeJEjz+Y0NwIdFDhcPmUOYtEDmuRRrgYiSQ== sebstian.wagner@it-novum.com", "key_options": null, "keyfile": "/home/swagner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "swagner", "validate_certs": true} 2021-02-10T10:54:11.436 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDzYzqIDimPjMOUOUz1A122eV52fl3mgGgNBRYRlOWl/lOxM5oOat5FPkJlnXTVyQFl0MlnYgIv9fTnjHVTM0wFmFUItlUl151ZmK5XExiGpGVsqG3UWNyKJxZpk9Uh2a0T/5CKzzwXnv2fJMNxzfqgopYDrgq4TFEZcdvgWJ39dcRTXST/rGJQ8wOq9z7UNzKU5VOGxvgi6f/nGevGIHrSaTcY3Uu8BW8kXUrf9vfl+ZSxVjuNHXF/VmrJH5Ne8mVu7DzPMrl8ZZjxsfJKT/eQqF9kjqht5586OSPKIwg/LEOyYj7Wyyf3QbDAIp5bCygQfmBBl+kIxphJYq3EwUZT ylifshit@dhcp-4-102.tlv.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFgwQYD9amVSYq6JzjHyGmKBJATQCM9xsf6QnF3yb0Lvry2nzY9F61evVXQ95TmF/AqasRx5p2WansIvegG5ag5mytvRwFXtHzPs4o/J+XjA9o4+A5l6j9cOFLIDD/TwrwlUjgMa2PhqUztSpI1op906391RLrJAyflfDghJxeVl6dJab6TEsslV0W+yaCS3BBxMXXrCbxCB5Ms3KCRKDz0DqbhFc+Nw05L1O1pC98HBFRIp1SaXlMcT3++CRQ3wugpOMkDMNLbycl+vdMBFY3UZ28rRowS3Ta7V5EARZVVH7KxU5cL+8Hr/8DE1/ITOmrluaEZ1KwYgoxZbZ3w7Ptw0FK6aiCZ49YtqKT1WakoVfO8m7pCdxkqUOA91loMQtxvCwZ/Vfoxaut16gklmPz8N11xvo1H5XB8/ZrvK5yKdl3JDl+wwtfE/fwo4E/fte+rPRisoWBRsmsx8EGxs75sXnb6n3VjZ65vYEKsNMLg+5EAaevWLLeqF0a42UWGJrHdb4VKX/OAplHrOk7kA/VwArow8f9n+iaRp2hgDU9OhQiFVRhgqwahlmCFFXkcP4/4WlxMCuZrsaNMZgMjVhfsp7cskKOezSfZZey2MB2fREMS1wg0j40CdYRn4jTNEBr/LVszsoM7t7AwXUpnqXeYIe+gePx1PJb5XSzSPZ9Pw== yuvalif@yahoo.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDDp0xz0ml2XuoQe70TxVeTo3Ib5xMinDrYNJ5eq5XYNa8ZLGcEByeDK4myzixTloAo31Rb/yDyGqv6kQ6nFChihO2aab3Ni/bsMOaJat1scrznaIYXGvLvw5KYx/qBDUxb3qW0vO0N/bB3dUPS1l 2021-02-10T10:54:11.437 INFO:teuthology.task.ansible.out:XTJaVf+KuljUk7v38ov2gsPWlofRYZNkPGQfUKXFEXc39OVj+Ex5LkehH/Z1LwjRYJa7jAdSm8hE5m8m6WborrzAA4/iFuLbhtM/Bqm4boSqa7koBVux422QX/HblSQ2KAeT1GFOS7xdcO6ndyp6mIpCZWunTjZwSLyK20NvIV/AzdAro4GZWi0D34PPtbSG344ZT1dSLeDRhYVkDjS6uutrmw9C+Ns1bo8s1ydhxL2XEsU14SWRWlbyrMF9PXyJOsaNiBNWGwc9lTly0ZlmPnN3l0AC3RIKOstkEYmI+TMj21r1tltz48e+6dH1an51wwnhNRYiODd3hbS67tmIg05BuSU5+whJ93wzXw2rC4FLE= ylifshit@localhost.localdomain", "key_options": null, "keyfile": "/home/yuvalif/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuvalif", "validate_certs": true} 2021-02-10T10:54:11.447 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDzYzqIDimPjMOUOUz1A122eV52fl3mgGgNBRYRlOWl/lOxM5oOat5FPkJlnXTVyQFl0MlnYgIv9fTnjHVTM0wFmFUItlUl151ZmK5XExiGpGVsqG3UWNyKJxZpk9Uh2a0T/5CKzzwXnv2fJMNxzfqgopYDrgq4TFEZcdvgWJ39dcRTXST/rGJQ8wOq9z7UNzKU5VOGxvgi6f/nGevGIHrSaTcY3Uu8BW8kXUrf9vfl+ZSxVjuNHXF/VmrJH5Ne8mVu7DzPMrl8ZZjxsfJKT/eQqF9kjqht5586OSPKIwg/LEOyYj7Wyyf3QbDAIp5bCygQfmBBl+kIxphJYq3EwUZT ylifshit@dhcp-4-102.tlv.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFgwQYD9amVSYq6JzjHyGmKBJATQCM9xsf6QnF3yb0Lvry2nzY9F61evVXQ95TmF/AqasRx5p2WansIvegG5ag5mytvRwFXtHzPs4o/J+XjA9o4+A5l6j9cOFLIDD/TwrwlUjgMa2PhqUztSpI1op906391RLrJAyflfDghJxeVl6dJab6TEsslV0W+yaCS3BBxMXXrCbxCB5Ms3KCRKDz0DqbhFc+Nw05L1O1pC98HBFRIp1SaXlMcT3++CRQ3wugpOMkDMNLbycl+vdMBFY3UZ28rRowS3Ta7V5EARZVVH7KxU5cL+8Hr/8DE1/ITOmrluaEZ1KwYgoxZbZ3w7Ptw0FK6aiCZ49YtqKT1WakoVfO8m7pCdxkqUOA91loMQtxvCwZ/Vfoxaut16gklmPz8N11xvo1H5XB8/ZrvK5yKdl3JDl+wwtfE/fwo4E/fte+rPRisoWBRsmsx8EGxs75sXnb6n3VjZ65vYEKsNMLg+5EAaevWLLeqF0a42UWGJrHdb4VKX/OAplHrOk7kA/VwArow8f9n+iaRp2hgDU9OhQiFVRhgqwahlmCFFXkcP4/4WlxMCuZrsaNMZgMjVhfsp7cskKOezSfZZey2MB2fREMS1wg0j40CdYRn4jTNEBr/LVszsoM7t7AwXUpnqXeYIe+gePx1PJb5XSzSPZ9Pw== yuvalif@yahoo.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDDp0xz0ml2XuoQe70TxVeTo3Ib5xMinDrYNJ5eq5XYNa8ZLGcEByeDK4myzixTloAo31Rb/yDyGqv6kQ6nFChihO2aab3Ni/bsMOaJat1scrznaIYXGvLvw5KYx/qBDUxb3qW0vO0N/bB3dUPS1l 2021-02-10T10:54:11.449 INFO:teuthology.task.ansible.out:XTJaVf+KuljUk7v38ov2gsPWlofRYZNkPGQfUKXFEXc39OVj+Ex5LkehH/Z1LwjRYJa7jAdSm8hE5m8m6WborrzAA4/iFuLbhtM/Bqm4boSqa7koBVux422QX/HblSQ2KAeT1GFOS7xdcO6ndyp6mIpCZWunTjZwSLyK20NvIV/AzdAro4GZWi0D34PPtbSG344ZT1dSLeDRhYVkDjS6uutrmw9C+Ns1bo8s1ydhxL2XEsU14SWRWlbyrMF9PXyJOsaNiBNWGwc9lTly0ZlmPnN3l0AC3RIKOstkEYmI+TMj21r1tltz48e+6dH1an51wwnhNRYiODd3hbS67tmIg05BuSU5+whJ93wzXw2rC4FLE= ylifshit@localhost.localdomain", "key_options": null, "keyfile": "/home/yuvalif/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuvalif", "validate_certs": true} 2021-02-10T10:54:11.885 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXa9uTf3q+7tGnHzSxv0WbgpRKrBIYjAZ5x3V74vQUSD1aAbmyvOqgPnts7XttLYImjjRqiisyC03L3KgR4b3SOeyL3KFxrk3KkHAJ/AJH+Y/UafyDhHNl3gRNg0B9rjsa7h2dSlcrFKnAX0pDY1HapuNPrzYWrXsEbo7C6wteFh7PyPmTkHu0NzIIPC4FcLl8SGaww/HeBh8v91IymgFSgXG0jNC063FEExWdti54yNLGh3gpW0FytsiRN9eEOKUOgcveT1tejz9ORv8evO4Y9SCQ/lsyOKQKNcG5C7J8nsbwSjqQ2cf6i2aYFZ0Oq1UXuhJ8APJCoiKgLJy7AiYf kkeithle@dhcp-100-19-89.bos.redhat.com", "key_options": null, "keyfile": "/home/kkeithle/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kkeithle", "validate_certs": true} 2021-02-10T10:54:11.897 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXa9uTf3q+7tGnHzSxv0WbgpRKrBIYjAZ5x3V74vQUSD1aAbmyvOqgPnts7XttLYImjjRqiisyC03L3KgR4b3SOeyL3KFxrk3KkHAJ/AJH+Y/UafyDhHNl3gRNg0B9rjsa7h2dSlcrFKnAX0pDY1HapuNPrzYWrXsEbo7C6wteFh7PyPmTkHu0NzIIPC4FcLl8SGaww/HeBh8v91IymgFSgXG0jNC063FEExWdti54yNLGh3gpW0FytsiRN9eEOKUOgcveT1tejz9ORv8evO4Y9SCQ/lsyOKQKNcG5C7J8nsbwSjqQ2cf6i2aYFZ0Oq1UXuhJ8APJCoiKgLJy7AiYf kkeithle@dhcp-100-19-89.bos.redhat.com", "key_options": null, "keyfile": "/home/kkeithle/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kkeithle", "validate_certs": true} 2021-02-10T10:54:12.351 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBYa5uEOlFv1s9K9D1SbcaApLwX3LBwASSnRXDHI7MtU paul.emmerich@croit.io", "key_options": null, "keyfile": "/home/emmericp/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "emmericp", "validate_certs": true} 2021-02-10T10:54:12.353 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBYa5uEOlFv1s9K9D1SbcaApLwX3LBwASSnRXDHI7MtU paul.emmerich@croit.io", "key_options": null, "keyfile": "/home/emmericp/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "emmericp", "validate_certs": true} 2021-02-10T10:54:12.796 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCj28V9sMybkAzhkiaNiDDr8TfNVyr5DNdbOqszdGEVLy1Rxh/UYnxUclsR9n8lt9VnCft3M4NDTnVwlxOHE8+0oolJE9HP+DQomsgj3abP04EGJyOA0X8E7eIFQNLRbMFCD0yX/SNNlYPsS5fPTFhIa/gbBPmDBE07kULts66Tvc0Yo/2yIfwV1DV34bcWWVGi01XybIHBl9qbvlfBKAUrFEcv3ZdpHNhuZw6IkZY8xoYxo8iF+toLYeGX+gFxqCZUUOO+NBVmb0hVL1LI6bU66IrMOQDkFurqpvoraBTJqwZgXvMPceFUkcVIH4Nm7cK1X2WyMtnSw2XBpLMT8C4V mchangir@indraprastha", "key_options": null, "keyfile": "/home/mchangir/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mchangir", "validate_certs": true} 2021-02-10T10:54:12.808 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCj28V9sMybkAzhkiaNiDDr8TfNVyr5DNdbOqszdGEVLy1Rxh/UYnxUclsR9n8lt9VnCft3M4NDTnVwlxOHE8+0oolJE9HP+DQomsgj3abP04EGJyOA0X8E7eIFQNLRbMFCD0yX/SNNlYPsS5fPTFhIa/gbBPmDBE07kULts66Tvc0Yo/2yIfwV1DV34bcWWVGi01XybIHBl9qbvlfBKAUrFEcv3ZdpHNhuZw6IkZY8xoYxo8iF+toLYeGX+gFxqCZUUOO+NBVmb0hVL1LI6bU66IrMOQDkFurqpvoraBTJqwZgXvMPceFUkcVIH4Nm7cK1X2WyMtnSw2XBpLMT8C4V mchangir@indraprastha", "key_options": null, "keyfile": "/home/mchangir/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mchangir", "validate_certs": true} 2021-02-10T10:54:13.249 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC1OVCo0wMITAp8PNUqsTRPFz0SuBL0r1RzKFJUoxQm7gsP9qttZUQRlfiuZ7YRzkuGLOEBngPuIcV995o0lZ0+LVHA8jxWEJamb6sM9YgnonbGKDHfl4lT8LJd4YtjcWnMGvUUdPiPgIkfrpxXtx+RzJ7eCTmgRjAJziZrRh3AqXuKty+s9MDnmK8EQ2smpLdkMtZG5W3U2PiVaGgq2FIIOAoqr58xGqFgZ4tIpvWFpNxVsDZXIg2NdC+ouuKQvuJRWq9eERRHwq26CR+kDFNZeYSyhYKxNBqauLCsAoXNFCJzrdX5Oi9Ox4lJTsEtYX+SMziVipQS54F7MnRAUccukQ86IbXz8C5Ok2EzNo6smwRZ6UPKRvebWjIvfbeR/iwPRnrWcNgjGcaTR6yhlRJh8pBkW84nyY/bJa4WXsoNLgutdsYkjl18KKhbBldjdf0ovfL7FVgqYfAo/wpQYJJeMebP5QAczxvZIegjHRrAvbs/nbQTmuAkes3WL1hhBjSTwyIbWCzAuBJk8MNTmeZkVBvub7XoIyIySppfoJC/1a+ZsxGkzAlgrIxXPYrrBzTtXd/9YRiIgCYx3UtfdbPkE6hT6GblH6ST6QESfWyyvfGxwlOmu+F+icRyVVJbgM9T/PD2W6I04u2/1mcfoHwaQGdwvbbBkbjWzm+zwNSisQ== sanupkri@redhat.com", "key_options": null, "keyfile": "/home/sidharthanup/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sidharthanup", "validate_certs": true} 2021-02-10T10:54:13.249 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC1OVCo0wMITAp8PNUqsTRPFz0SuBL0r1RzKFJUoxQm7gsP9qttZUQRlfiuZ7YRzkuGLOEBngPuIcV995o0lZ0+LVHA8jxWEJamb6sM9YgnonbGKDHfl4lT8LJd4YtjcWnMGvUUdPiPgIkfrpxXtx+RzJ7eCTmgRjAJziZrRh3AqXuKty+s9MDnmK8EQ2smpLdkMtZG5W3U2PiVaGgq2FIIOAoqr58xGqFgZ4tIpvWFpNxVsDZXIg2NdC+ouuKQvuJRWq9eERRHwq26CR+kDFNZeYSyhYKxNBqauLCsAoXNFCJzrdX5Oi9Ox4lJTsEtYX+SMziVipQS54F7MnRAUccukQ86IbXz8C5Ok2EzNo6smwRZ6UPKRvebWjIvfbeR/iwPRnrWcNgjGcaTR6yhlRJh8pBkW84nyY/bJa4WXsoNLgutdsYkjl18KKhbBldjdf0ovfL7FVgqYfAo/wpQYJJeMebP5QAczxvZIegjHRrAvbs/nbQTmuAkes3WL1hhBjSTwyIbWCzAuBJk8MNTmeZkVBvub7XoIyIySppfoJC/1a+ZsxGkzAlgrIxXPYrrBzTtXd/9YRiIgCYx3UtfdbPkE6hT6GblH6ST6QESfWyyvfGxwlOmu+F+icRyVVJbgM9T/PD2W6I04u2/1mcfoHwaQGdwvbbBkbjWzm+zwNSisQ== sanupkri@redhat.com", "key_options": null, "keyfile": "/home/sidharthanup/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sidharthanup", "validate_certs": true} 2021-02-10T10:54:13.691 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCkeYXWjU7wVdODC32U40QRFgFH7ZEuEsOeWWaKx3Y5EBuAcwkTHiF9Bq/2mrDu+c9W46S8L38Ogc9GZ0DcCXtN/Z4z8iKOIIi5U1GzvwdbKuiAPCd8bdYP0vo0MOIRgylJo5hcnB5BCwCXRN1hmci1ES5UbSeQg0rydMiyYXHZ64AT47+qVUpn/mVP6PVw00pPqt16l9YxhbPFG74Y/WnY4FyrqXGQJfv0tcP7+UR7GQdFiI4QQV3ZUSRHRLWGcmoiiOmGimLhtzEmP4Jzy4gF4wg9q8jyX54NjfxDRZs+YuaoqqWKz/HSz5ic7W2RGKHrRby2N9wNu19nmC1NzIn3 varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCW58OUR3pniaYgvr9xAExeNm4eMxgaJfFbEfEYPztTWv/xxqqihBYCeFltfgWbu7ERTy+hVNo1PJWyyczjl5Ivc3FdRmhykosygs7X74PPh02c7Em4MsSTr7SSPNN20rkJbWqiYMMOBbqbiZ7JXiSUNVilLHgZqZBEQc5Rf5I1PyBLHHJ2ZruPKnDPHCRLdwJ+MdXy2haKmGrQLhtzIKaDzh8VkfojL454k0aHCIh34Z0oBdP9757mNuYJUVEkKZF5t2oML84lWGMgwZxUWN2yO9ALTti4tNH1fXA6NipS9ztE+3hEJe1fWXEoUQDX9JNu4qEHTeCz0secQMkyZVDZSQk8yDUQxQ76yiett37sQB1WQR/3fclcahmqQ4dUS7M3Onyz6k6zia896PBK6G48TBUbFa1rw5Hx7WfNZluDMkv4YxkQvfz7XVzZx2/lhs45Kjh1UR9n7LaW/NID98vokcvASw1N6Bk/6pOd/45iPyEJemwovnQLs6ecwlMIikM= varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDcQ8xMjwVf+KiFIQARcbFT+wg4NHysBRrFxk3Bx5X2k2hGKcEYR2tY/ApHZYfJ6me3LUvJEXg2c/6xa6ibVTUp/90l4y0cF6RmF5uB45J0xuBYoFJd8xJkVrhDIlZix4aFfb7TrmJEqjleBrkTwam7Q9aE3lss16nVOxU4hiViYiCANJ45wobK6tz7y5btfzrHE/iYpwniDvzLwK7EpkEK336VVd6f9wV0vG+ErEFxnNVxheXLLm05LuvdVQvFqJcTFrc7/GZ54yKg5IgwVwXqSAyOXIhgduUqk+oN1ciA7jnSqdiOKQPS/VmYkbhw2YP85l8wGVreIYUmOG19QEyVLB82oB9KS1LezM2KCH61XW22+85dHf4PvuL9DULhGt6wBCvpO10OHxw0JvHXbEgBPhiyd9j2yJmcQozbRPth/sMUvLKNXOGabZhhkPBYL7xZICHc+UwxrXs0h/j9jOjLyHzBAWMomF2lzgbvtzjcbxJ1a5oclJ6IW8yHJllkOh0= varsha@localhost.localdo 2021-02-10T10:54:13.693 INFO:teuthology.task.ansible.out:main", "key_options": null, "keyfile": "/home/varsha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "varsha", "validate_certs": true} ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCkeYXWjU7wVdODC32U40QRFgFH7ZEuEsOeWWaKx3Y5EBuAcwkTHiF9Bq/2mrDu+c9W46S8L38Ogc9GZ0DcCXtN/Z4z8iKOIIi5U1GzvwdbKuiAPCd8bdYP0vo0MOIRgylJo5hcnB5BCwCXRN1hmci1ES5UbSeQg0rydMiyYXHZ64AT47+qVUpn/mVP6PVw00pPqt16l9YxhbPFG74Y/WnY4FyrqXGQJfv0tcP7+UR7GQdFiI4QQV3ZUSRHRLWGcmoiiOmGimLhtzEmP4Jzy4gF4wg9q8jyX54NjfxDRZs+YuaoqqWKz/HSz5ic7W2RGKHrRby2N9wNu19nmC1NzIn3 varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCW58OUR3pniaYgvr9xAExeNm4eMxgaJfFbEfEYPztTWv/xxqqihBYCeFltfgWbu7ERTy+hVNo1PJWyyczjl5Ivc3FdRmhykosygs7X74PPh02c7Em4MsSTr7SSPNN20rkJbWqiYMMOBbqbiZ7JXiSUNVilLHgZqZBEQc5Rf5I1PyBLHHJ2ZruPKnDPHCRLdwJ+MdXy2haKmGrQLhtzIKaDzh8VkfojL454k0aHCIh34Z0oBdP9757mNuYJUVEkKZF5t2oML84lWGMgwZxUWN2yO9ALTti4tNH1fXA6NipS9ztE+3hEJe1fWXEoUQDX9JNu4qEHTeCz0secQMkyZVDZSQk8yDUQxQ76yiett37sQB1WQR/3fclcahmqQ4dUS7M3Onyz6k6zia896PBK6G48TBUbFa1rw5Hx7WfNZluDMkv4YxkQvfz7XVzZx2/lhs45Kjh1UR9n7LaW/NID98vokcvASw1N6Bk/6pOd/45iPyEJemwovnQLs6ecwlMIikM= varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDcQ8xMjwVf+KiFIQARcbFT+wg4NHysBRrFxk3Bx5X2k2hGKcEYR2tY/ApHZYfJ6me3LUvJEXg2c/6xa6ibVTUp/90l4y0cF6RmF5uB45J0xuBYoFJd8xJkVrhDIlZix4aFfb7TrmJEqjleBrkTwam7Q9aE3lss16nVOxU4hiViYiCANJ45wobK6tz7y5btfzrHE/iYpwniDvzLwK7EpkEK336VVd6f9wV0vG+ErEFxnNVxheXLLm05LuvdVQvFqJcTFrc7/GZ54yKg5IgwVwXqSAyOXIhgduUqk+oN1ciA7jnSqdiOKQPS/VmYkbhw2YP85l8wGVreIYUmOG19QEyVLB82oB9KS1LezM2KCH61XW22 2021-02-10T10:54:13.694 INFO:teuthology.task.ansible.out:+85dHf4PvuL9DULhGt6wBCvpO10OHxw0JvHXbEgBPhiyd9j2yJmcQozbRPth/sMUvLKNXOGabZhhkPBYL7xZICHc+UwxrXs0h/j9jOjLyHzBAWMomF2lzgbvtzjcbxJ1a5oclJ6IW8yHJllkOh0= varsha@localhost.localdomain", "key_options": null, "keyfile": "/home/varsha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "varsha", "validate_certs": true} 2021-02-10T10:54:14.136 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6RDgqwA0GrvE6KqRaGCgYnFhnxdi9ehjIWvfaMa1oq0CqYxUAee2bNbp/X2Bd18Vqk8iH9fpE0+qrLgAH/8BpD+MtgGikCFMORvazeUtOCF/OCDO0AMg1jvl8aTdvrQU8ZTDPXa65Avpkc9vwugcoplnwrW7lKx++XqCWsSlfHgEdyMV3wrt/ZDFwI2AS2bZBpWM6ViG1KSjXjgTi/T3hSZPY+buYvWjzoEjQO0946iI3iiKKCGohEpCUnG02oJXcS0vvgl8dWE71451lbainEKJ4hbDhfDmy57s6VXfVUANsczwiPWiSC/ZmMlDaCCBeKHO0Gz0xXaDXd9j4SA7P sam@Pondermatic\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCYylAUS12hF6XndDQ3CkrUddKmXNSd8pOlHwO+pWaDTJPKxwtPL6sM3aOtaAPLrGkv1zD2k3iZaxmDwfPYGaXOyb0KS5ZCddtOyNXj5JLrjxh2niqFPP8SIkdHLOP+0Hv1/tB0Jz75RV+tNIT1SJTm7OZ8wZNy0qYycITSooJ14iEgyZ7rOa3EEzbtIVpHS4+YqIuIJdV2oV5egCrJyRv4sPUrS8ERXD4QZSOlHE+W4Tf+WVrEfsHOdDLRlAXJUzhuqgWtVouiYcW5ex294z28eA+12lKaPQpE0cft/NJ6FLs41yqh4ZN588bl8Ada+yTXEaHWioQEEJMlca2jIgrP sjust@rex004.front.sepia.ceph.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC08RrDQ2M7Zjcpx7RL9e+tjv48kBgDNyTenX1QdxW22JA2JR3D3ThA/ej9HtiPWcIVz16oZ4tzrCEVwaCMfbsmXq2 2021-02-10T10:54:14.138 INFO:teuthology.task.ansible.out:z08ZXYqwiBuoXJVshU7lIKVqAtadT4F6Q9eQvxEo4zWdBib1W0/cqNLlYzU/DuwlUqYnWBDLGxc4fD5Kxrz366oWUDB/ULCpl77TMqfTKlStCgGXot0zLqiscqzq1W7XsuA7A5vQw/JTq7MvIME+SkUuGWX0WKwTMIRZKNWBGmlf51bNru3GmmAnKBUFsJv9H4G0FPqGY/aH02rr5dAgTTmfKt+XbWRfMTMpjJqVRGX03+1mQNoYQ3kNDz0wl sjust@office\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA5UMPySf/h9r2/OryRXrO9Qq9k5J9Dn9nvJYyaYCE82H/qkBJGboAJgyIXfbsGYO5uPM25d3fGRH4magzAQm2hGWNbbt1YWn9HaIyKCCXo9JX600/dv3OZbHyGGmgXPa4Anl16j1c2vxRRt2+ArjlyyhP7cAcRkilUGdagK9u6ALrwnz9hhXYV+BmXrxMm7zZCmdWHcWz+YxoA65es4+w+aLvVZWmUCZLF8uZrcpPSw+oK3SfZE8JPRc5vWulXbWEEfIoHb9AszPIpSyzbfPWXelmc3XlG0KD66t7w+AnNlQPt3JVePRCfFHPYeTj9B0k8sxPyUqTQIl5pu6GppRMOQ== sam@Deepthought", "key_options": null, "keyfile": "/home/sjust/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sjust", "validate_certs": true} ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6RDgqwA0GrvE6KqRaGCgYnFhnxdi9ehjIWvfaMa1oq0CqYxUAee2bNbp/X2Bd18Vqk8iH9f 2021-02-10T10:54:14.139 INFO:teuthology.task.ansible.out:pE0+qrLgAH/8BpD+MtgGikCFMORvazeUtOCF/OCDO0AMg1jvl8aTdvrQU8ZTDPXa65Avpkc9vwugcoplnwrW7lKx++XqCWsSlfHgEdyMV3wrt/ZDFwI2AS2bZBpWM6ViG1KSjXjgTi/T3hSZPY+buYvWjzoEjQO0946iI3iiKKCGohEpCUnG02oJXcS0vvgl8dWE71451lbainEKJ4hbDhfDmy57s6VXfVUANsczwiPWiSC/ZmMlDaCCBeKHO0Gz0xXaDXd9j4SA7P sam@Pondermatic\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCYylAUS12hF6XndDQ3CkrUddKmXNSd8pOlHwO+pWaDTJPKxwtPL6sM3aOtaAPLrGkv1zD2k3iZaxmDwfPYGaXOyb0KS5ZCddtOyNXj5JLrjxh2niqFPP8SIkdHLOP+0Hv1/tB0Jz75RV+tNIT1SJTm7OZ8wZNy0qYycITSooJ14iEgyZ7rOa3EEzbtIVpHS4+YqIuIJdV2oV5egCrJyRv4sPUrS8ERXD4QZSOlHE+W4Tf+WVrEfsHOdDLRlAXJUzhuqgWtVouiYcW5ex294z28eA+12lKaPQpE0cft/NJ6FLs41yqh4ZN588bl8Ada+yTXEaHWioQEEJMlca2jIgrP sjust@rex004.front.sepia.ceph.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC08RrDQ2M7Zjcpx7RL9e+tjv48kBgDNyTenX1QdxW22JA2JR3D3ThA/ej9HtiPWcIVz16oZ4tzrCEVwaCMfbsmXq2z08ZXYqwiBuoXJVshU7lIKVqAtadT4F6Q9eQvxEo4zWdBib1W0/cqNLlYzU/DuwlUqYnWBDLGxc4fD5Kxrz366oWUDB/ULCpl77TMqfTKlStCgGXot0zLqiscqzq1W7XsuA7A5vQw/JTq7MvIME+SkUuGWX0WKwTMIRZKNWBGmlf51bNru3GmmAnKBUFsJv9H4G0FPqGY/aH02rr5dAgTTmfKt+XbWRfMTMpjJqVRGX03+1mQNoYQ3kNDz0wl sjust@office\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA5UMPySf/h9r2/OryRXrO9Qq9k5J9Dn9nvJYyaYCE82H/qkBJGboAJgyIXfbsGYO5uPM25d3fGRH4magzAQm2hGWNbbt1YWn9HaIyKCCXo9JX600/dv3OZbHyGGmgXPa4Anl16j1c2vxRRt2+ArjlyyhP7cAcRkilUGdagK9u6ALrwnz9hhXYV+BmXrxMm7zZCmdWHcWz+YxoA65es4+w+aLvVZWmUCZLF8uZrcpPSw+oK3SfZE8JPRc5vWulXbWEEfIoHb9AszPIpSyzbfPWXelmc3XlG0KD66t7w+AnNlQPt3JVePRCfFHPYeTj9B0k8sxPyUqTQIl5pu6GppRMOQ== sam@Deepthought", "key_options": null, "keyfile": "/home/sjust/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sjust", "validate_certs": true} 2021-02-10T10:54:14.577 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDj6DxTZNodVnwJZrnSZ/jv9Cw4uYLb8MGpccmOvtwl93sVBvqq597AYnBqYiV4VBbKU3GVFbT41zkplia1iCfXoddttCOyWmEKviBXRnAsd5f80dxL0pfzrBJvCaH4NAfYgVw9JFWsvT981b+nB6vum4Vz8Dw0PM5GPnVUjsZNFkpmDPUBL4yHPmMCGP2WtUurJVD3Gm9t72XlG6Dl/vB9Qw/i4IY1SoCCbuhhNgE7IN8JXeGmVJbRwDeUlP+Q8Ti7iedV7YB/W1opt6eT5EPzX8NcxTc0c7rrDgsUWjFQYRSup0GLK9ofdHs0E2RbPPO5OSc5Z2+B1IEV7fZeyJwAyHwUgwx7sSGy7aROdvinaj/hOBglcgISyQ2CY7LTV8NGWJquo7yRCJXhZVIHfgqfBcrgWdNbr+vNBXnWe5PCtLhV8ryHPSJ/7Ru+7voQxgvwux5SdZwxXTbWDKni1f6/8bNrhRx6bmLK9xuYAgt2FhlIivI7L5kQF4etcBiaf68= deepika@x1cabon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdqbYcbJYtgCrAbLo7PcS82caXxcT8cpX0RkULaFXfaiUDs8Foy6Yf/XoEHIdpgw8kX6qBdydYXxti6HFhYDNsL5KleKu/yA+bVKlvri39CRNkNxFHAUL2CRBT5Hefj1AJQBm/BKVEN4/eb05OlJJ808Qb/aNycot/YwgpNvUIaO2yTMA0meq82wNdZmUcmyLB9QXz4QInTcoAsxq70KxX95YC938rTWpbg1hM8gzLkJJt/OD/Z7TMYv/Sc5KGOFqo930AmObQ5KPmffkWGYbLT65TxlCULnvIr63lWHdGTj5mYjib/v9ncu24az9GYRWYcicV0B/def+KVrrIp3Mz ideepika@teuthology", "key_options": null, "keyfile": "/home/ideepika/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ideepika", "validate_certs": true} 2021-02-10T10:54:14.577 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDj6DxTZNodVnwJZrnSZ/jv9Cw4uYLb8MGpccmOvtwl93sVBvqq597AYnBqYiV4VBbKU3GVFbT41zkplia1iCfXoddttCOyWmEKviBXRnAsd5f80dxL0pfzrBJvCaH4NAfYgVw9JFWsvT981b+nB6vum4Vz8Dw0PM5GPnVUjsZNFkpmDPUBL4yHPmMCGP2WtUurJVD3Gm9t72XlG6Dl/vB9Qw/i4IY1SoCCbuhhNgE7IN8JXeGmVJbRwDeUlP+Q8Ti7iedV7YB/W1opt6eT5EPzX8NcxTc0c7rrDgsUWjFQYRSup0GLK9ofdHs0E2RbPPO5OSc5Z2+B1IEV7fZeyJwAyHwUgwx7sSGy7aROdvinaj/hOBglcgISyQ2CY7LTV8NGWJquo7yRCJXhZVIHfgqfBcrgWdNbr+vNBXnWe5PCtLhV8ryHPSJ/7Ru+7voQxgvwux5SdZwxXTbWDKni1f6/8bNrhRx6bmLK9xuYAgt2FhlIivI7L5kQF4etcBiaf68= deepika@x1cabon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdqbYcbJYtgCrAbLo7PcS82caXxcT8cpX0RkULaFXfaiUDs8Foy6Yf/XoEHIdpgw8kX6qBdydYXxti6HFhYDNsL5KleKu/yA+bVKlvri39CRNkNxFHAUL2CRBT5Hefj1AJQBm/BKVEN4/eb05OlJJ808Qb/aNycot/YwgpNvUIaO2yTMA0meq82wNdZmUcmyLB9QXz4QInTcoAsxq70KxX95YC938rTWpbg1hM8gzLkJJt/OD/Z7TMYv/Sc5KGOFqo930AmObQ5KPmffkWGYbLT65TxlCULnvIr63lWHdGTj5mYjib/v9ncu24az9GYRWYcicV0B/def+KVrrIp3Mz ideepika@teuthology", "key_options": null, "keyfile": "/home/ideepika/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ideepika", "validate_certs": true} 2021-02-10T10:54:15.019 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDC0h+TpaUkm2V6Hbf7aPCTys/nsm51CmY9s+bdwEXAiSAWpq/7Xi2LlN0mZgsrAYBHbr5L+Pz8x0YdigvWysUK6U9XRSPM28cbWmh8F25re9HEk8ARESkMZKTv/hKFs1v7kTHbT9hNNynbQGQOqEaazjr0shcli+Byvop27riX3QDDSsd5xfCjjyDz9fiA9YESlF0kbXyFUKPsoaBhRv9FUhtH6DxlygxtxTsKyKgeoiKw2sCfeUMckcJKnGCMtkXW4U8h7NL/fPZFyBaMDRi/TtoBKaVGLaWqF7RuukfjoMCwhuIDmuTG3WhanH4HJrfNenPZkpZUBhzWwZcleao8SD3mXXfZ9gcZb8QB2ymEtTqrAEyDoNWO4q8ljNP3WJXIgF0h0Sjmx/CV/i0ArEYlBMwBD7DFVnPJqCqSyypuQdDGsDNQZxaRCiYg7nIslFOpDr+0PLp78m+CHyYbSFLTvVVJoJ1k0YhMnDPHM29nt7fMGx6Qrigwnuk0yKg1bfgJ/t0TOqEw0bqGKIvHnyoP/n0ObazEsiPl5pRr2ehs/Rqf35Iwhc1i02Hhz/3Sqwc1eAgqm4/OVghxqUJGpRocua5Gf/vO32wwoyGxh2sl7y6K2pNDA3glhKYbZeIh1fjnaTjikgleqIQkCBxbTeAE7n5+T6TcFtnII6RSxYv3tQ== gal.salomon@gmail.com", "key_options": null, "keyfile": "/home/gsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gsalomon", "validate_certs": true} 2021-02-10T10:54:15.020 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDC0h+TpaUkm2V6Hbf7aPCTys/nsm51CmY9s+bdwEXAiSAWpq/7Xi2LlN0mZgsrAYBHbr5L+Pz8x0YdigvWysUK6U9XRSPM28cbWmh8F25re9HEk8ARESkMZKTv/hKFs1v7kTHbT9hNNynbQGQOqEaazjr0shcli+Byvop27riX3QDDSsd5xfCjjyDz9fiA9YESlF0kbXyFUKPsoaBhRv9FUhtH6DxlygxtxTsKyKgeoiKw2sCfeUMckcJKnGCMtkXW4U8h7NL/fPZFyBaMDRi/TtoBKaVGLaWqF7RuukfjoMCwhuIDmuTG3WhanH4HJrfNenPZkpZUBhzWwZcleao8SD3mXXfZ9gcZb8QB2ymEtTqrAEyDoNWO4q8ljNP3WJXIgF0h0Sjmx/CV/i0ArEYlBMwBD7DFVnPJqCqSyypuQdDGsDNQZxaRCiYg7nIslFOpDr+0PLp78m+CHyYbSFLTvVVJoJ1k0YhMnDPHM29nt7fMGx6Qrigwnuk0yKg1bfgJ/t0TOqEw0bqGKIvHnyoP/n0ObazEsiPl5pRr2ehs/Rqf35Iwhc1i02Hhz/3Sqwc1eAgqm4/OVghxqUJGpRocua5Gf/vO32wwoyGxh2sl7y6K2pNDA3glhKYbZeIh1fjnaTjikgleqIQkCBxbTeAE7n5+T6TcFtnII6RSxYv3tQ== gal.salomon@gmail.com", "key_options": null, "keyfile": "/home/gsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gsalomon", "validate_certs": true} 2021-02-10T10:54:15.463 INFO:teuthology.task.ansible.out:changed: [gibba013.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCrGYBGPDQqn+2jnxKhDurSrAJvTT8dZMYGVCt6NbjpFhDFYtyvrF8Jv5NKBQynG18LwscN7a57L0Oq0e4jOEPRz//CDcAfP+L6p45L4BdUzTysN4+rQ7waMnIiYp7VjSswhGH/BnYIVKwmDqzM2fo1w/oLIT6mgwUYqAdceeswAkXBt7jgagL8ox3E4oHBqER7f60AHdTviuW953t5luVQiMjyF2fOwZuo2ycufIzBee0CwS+xyRuJQwdgD3HL4Qs2vtBCX+H8n3wC6A9JrMA5tzCDr5VZKBJZURIniwvaXfJodjO4SvBL5jAfXoMqriv8ZWHbap2fk82YE8/uPqNvSISjixJEBgQinklZ+iAOYm5tfPbhPM/8ErpN/yJ0EoGRn+nj6X6nAjXAIk3AENopeqtM7mLZ26RFIuEV5ld7eqUvAfiy6Hk2LH9Q1Z4CXaxD0gDGzUxgdAaaAgLAR4btWi2n7Tb7ZNDdOfctg2ya9ftzZBOHvCaVXSpJCry/GMU= skoduri@localhost.localdomain", "key_options": null, "keyfile": "/home/soumyakoduri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "soumyakoduri", "validate_certs": true} 2021-02-10T10:54:15.464 INFO:teuthology.task.ansible.out:changed: [gibba002.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCrGYBGPDQqn+2jnxKhDurSrAJvTT8dZMYGVCt6NbjpFhDFYtyvrF8Jv5NKBQynG18LwscN7a57L0Oq0e4jOEPRz//CDcAfP+L6p45L4BdUzTysN4+rQ7waMnIiYp7VjSswhGH/BnYIVKwmDqzM2fo1w/oLIT6mgwUYqAdceeswAkXBt7jgagL8ox3E4oHBqER7f60AHdTviuW953t5luVQiMjyF2fOwZuo2ycufIzBee0CwS+xyRuJQwdgD3HL4Qs2vtBCX+H8n3wC6A9JrMA5tzCDr5VZKBJZURIniwvaXfJodjO4SvBL5jAfXoMqriv8ZWHbap2fk82YE8/uPqNvSISjixJEBgQinklZ+iAOYm5tfPbhPM/8ErpN/yJ0EoGRn+nj6X6nAjXAIk3AENopeqtM7mLZ26RFIuEV5ld7eqUvAfiy6Hk2LH9Q1Z4CXaxD0gDGzUxgdAaaAgLAR4btWi2n7Tb7ZNDdOfctg2ya9ftzZBOHvCaVXSpJCry/GMU= skoduri@localhost.localdomain", "key_options": null, "keyfile": "/home/soumyakoduri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "soumyakoduri", "validate_certs": true} 2021-02-10T10:54:15.901 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDPeuSTD7kv8yQX6dW+7Vpq1mqrcr8JGubPYa1DpGTV/gE50d5L42UspUN2FXmICDMiV1h4Q9WQAFma3168f1I5I94qFkV+/56szCMoILClm7IW/EV4t8fTxoyOpEa9l4qSU3qgJpDbY04CkiyVJFuQj24Zq3gD/70jYgGpy8RfA0BihSR6F+A0WZLh/fESjqfor3/pFWhUCRnPEqrZsXyuB8lRgDTyGp4s1cHyUqmFMuCPDRMtok0rSVpaRZj2Zj8lboah19UI/L89ptQgEdqsOKzlkxiT+1GdyIY+jpPCM0a3yZz75l3Qiu0k6q7qUV8T0hm1GZiBguvST1KMcwnt kyrylo.shatskyy@suse.com", "key_options": null, "keyfile": "/home/kyr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kyr", "validate_certs": true} 2021-02-10T10:54:15.913 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDPeuSTD7kv8yQX6dW+7Vpq1mqrcr8JGubPYa1DpGTV/gE50d5L42UspUN2FXmICDMiV1h4Q9WQAFma3168f1I5I94qFkV+/56szCMoILClm7IW/EV4t8fTxoyOpEa9l4qSU3qgJpDbY04CkiyVJFuQj24Zq3gD/70jYgGpy8RfA0BihSR6F+A0WZLh/fESjqfor3/pFWhUCRnPEqrZsXyuB8lRgDTyGp4s1cHyUqmFMuCPDRMtok0rSVpaRZj2Zj8lboah19UI/L89ptQgEdqsOKzlkxiT+1GdyIY+jpPCM0a3yZz75l3Qiu0k6q7qUV8T0hm1GZiBguvST1KMcwnt kyrylo.shatskyy@suse.com", "key_options": null, "keyfile": "/home/kyr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kyr", "validate_certs": true} 2021-02-10T10:54:16.345 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeERrQ0NVXhuTYzI08rlKRbHZVFXSv1YC7sGq6aCWRbTCv7Ad/t/3OJz3vfcd2TjBMvABfrYCFgTFWIaJBaeA3BXUUU9y9TwnzuxKv4C46WJYoalVGyLEpWpcGbJ5v+leA6Je/+lunenplb7TBKGE/YTgggEXyX7KnunYlv+Jdm33TmAZTcHrLk5QR/6FdRGKmdlLna0dXZc3v04+AI8W+p9ghAv1+e8ENxJDTF8p0MawBiKWcSj2mysOhQ2Cj91qZ0qTnLLeu0CPf4rBO+qjYIshiCuTLhYt2UazlzME4nCCsVqK7Nz9olA/lgvJLRaPxgQQ7iCMmIZilS6B4Y3oL sseshasa@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCo0zniUEXuQ4JJuuRtfnEK8WHqr97AmXEniGPY5U23MTxhMuP8CSkGWkUHdmXw0H1tDKbYgk97M6TspJropNGKljla2H1/3e2T1BOI1L2WfcsOh6VPYeyOj2ssMJeEDdQgd0J3g+4oj2QS+guPlT24UhO0CBhRZ6waVlJMgagDYyDLCLGI4ASoBYx0AOaoyjnEQtRd/gyejzv7jntkHc9YNBPsJ37X6hTYNA/s2ORJE7QMXd6xI60mTBuCdD3JcPYoUSvyCUa7rIrDc70lAcW5XtxewvMBogTZJLoFG/d6Ugij49h4AfwrtoUCtLyJOWJX6PTfomXely8Ym6KpN/3981+m70dhcHPDJ66QCWMd1LAgftVXTIsebA+Bxfg4Vsbrv7j1lJXzmMfsOl+1wVPe+Y9h28GyeIlalW3laURZGV7dhexGZ8YNxPuzxtAlWtfw+kddbjw0j4ODtgVXrj4SDxqmTTYYuMBkrQ3H33sOgPLx8DyA6e4pckb6lOOEvb41Yi5tyP7BIJlYN2PpNVDkSQd/t5Q0di4Ct+3vqLRCkPuJiJAUyaBGT4HM7yQc25+Y0B3sXNON7BjeaBXB7OwKZMp9w1jwbJt4JL/aGOw3HVljVyVgu+pIuQ2ISd9VBcmAJK5EGdWe5Q57h5fJkD+TVrXgeNTaufdclh5vWOH1qQ== sseshasa@teuthology", "key_options": null, "keyfile": "/home/sseshasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sseshasa", "validate_certs": true} 2021-02-10T10:54:16.357 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeERrQ0NVXhuTYzI08rlKRbHZVFXSv1YC7sGq6aCWRbTCv7Ad/t/3OJz3vfcd2TjBMvABfrYCFgTFWIaJBaeA3BXUUU9y9TwnzuxKv4C46WJYoalVGyLEpWpcGbJ5v+leA6Je/+lunenplb7TBKGE/YTgggEXyX7KnunYlv+Jdm33TmAZTcHrLk5QR/6FdRGKmdlLna0dXZc3v04+AI8W+p9ghAv1+e8ENxJDTF8p0MawBiKWcSj2mysOhQ2Cj91qZ0qTnLLeu0CPf4rBO+qjYIshiCuTLhYt2UazlzME4nCCsVqK7Nz9olA/lgvJLRaPxgQQ7iCMmIZilS6B4Y3oL sseshasa@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCo0zniUEXuQ4JJuuRtfnEK8WHqr97AmXEniGPY5U23MTxhMuP8CSkGWkUHdmXw0H1tDKbYgk97M6TspJropNGKljla2H1/3e2T1BOI1L2WfcsOh6VPYeyOj2ssMJeEDdQgd0J3g+4oj2QS+guPlT24UhO0CBhRZ6waVlJMgagDYyDLCLGI4ASoBYx0AOaoyjnEQtRd/gyejzv7jntkHc9YNBPsJ37X6hTYNA/s2ORJE7QMXd6xI60mTBuCdD3JcPYoUSvyCUa7rIrDc70lAcW5XtxewvMBogTZJLoFG/d6Ugij49h4AfwrtoUCtLyJOWJX6PTfomXely8Ym6KpN/3981+m70dhcHPDJ66QCWMd1LAgftVXTIsebA+Bxfg4Vsbrv7j1lJXzmMfsOl+1wVPe+Y9h28GyeIlalW3laURZGV7dhexGZ8YNxPuzxtAlWtfw+kddbjw0j4ODtgVXrj4SDxqmTTYYuMBkrQ3H33sOgPLx8DyA6e4pckb6lOOEvb41Yi5tyP7BIJlYN2PpNVDkSQd/t5Q0di4Ct+3vqLRCkPuJiJAUyaBGT4HM7yQc25+Y0B3sXNON7BjeaBXB7OwKZMp9w1jwbJt4JL/aGOw3HVljVyVgu+pIuQ2ISd9VBcmAJK5EGdWe5Q57h5fJkD+TVrXgeNTaufdclh5vWOH1qQ== sseshasa@teuthology", "key_options": null, "keyfile": "/home/sseshasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sseshasa", "validate_certs": true} 2021-02-10T10:54:16.788 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDIC6uqbIg9XaTlDZd7KCX7FXSzfGFMEWXhx42ktz3ehOPLviA/yNpVd+LmYHSgzd5Szd3mMxleisiE+JdSq/q5YyPAF6eD6+wfynkpMv5aWNHGI0yVfKbCwdOX8bY6yqhyVwMDo14uoFvdDcM4Lj+vz8F75VJHUU5ihJNOW24EllobTLzR4XI6AcM+mjbXRnBLLVoBSuDaU2D0kWqMn8rDCJKdUJnoiy9oxIok8eMLfGkGsL55oOx9BflMSz/0jM9X5GLQahYiECbXjvCIarCkUFE8txNoXM9Djmg8QqhAQDhY0hzJTauFJJaD8Ax1uFQlXONed7tv7a7Tb3dH+/jb rfriedma@dhcp-4-239.tlv.redhat.com", "key_options": null, "keyfile": "/home/rfriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rfriedma", "validate_certs": true} 2021-02-10T10:54:16.800 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDIC6uqbIg9XaTlDZd7KCX7FXSzfGFMEWXhx42ktz3ehOPLviA/yNpVd+LmYHSgzd5Szd3mMxleisiE+JdSq/q5YyPAF6eD6+wfynkpMv5aWNHGI0yVfKbCwdOX8bY6yqhyVwMDo14uoFvdDcM4Lj+vz8F75VJHUU5ihJNOW24EllobTLzR4XI6AcM+mjbXRnBLLVoBSuDaU2D0kWqMn8rDCJKdUJnoiy9oxIok8eMLfGkGsL55oOx9BflMSz/0jM9X5GLQahYiECbXjvCIarCkUFE8txNoXM9Djmg8QqhAQDhY0hzJTauFJJaD8Ax1uFQlXONed7tv7a7Tb3dH+/jb rfriedma@dhcp-4-239.tlv.redhat.com", "key_options": null, "keyfile": "/home/rfriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rfriedma", "validate_certs": true} 2021-02-10T10:54:17.239 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIXHA8BwV4OnqlaHHyYofsp5X+wanose+aneJVx2Or05PcWA9c7xFEOerkvEn9LvivSRd9G9o/S8SvnGH91tG2bbOAidVZ69FGul9OKa/BCJYUP0vVBX09/Td3n7+R2lVlvu5XiyC5SUzCjaQEZLOpjq23rqaqdCDXE6wFnxfpkmcsccbG/xUiOEBeBGI92lntdx99Sn6CxWWA27lMMaYkCRvX6ziRZEm6O44N2nMm+qYm0jmMyGv1iyQkJxtt0YaLPp+X9t2trUEzEjgBCXqqzEzBjB7T4rbzhAuvU3NRHiAoWXSnrY5AQOTTvC/wpaZV/gF2NDNThjMcpcEdNstleOctXtN4XTTJQewbG+t7I9knR00TPpsoUvUtpbcoFOluGgWTDTFa6GCmDPppx23nZgulOniZUVTI2CbScSpc8sfp1JE7Q6vLECY9lJ+GpVCbDAbIEE9WHXSkJ/6L9yf+ZoHbpqCkEFDcWLfz3B5NFoJ5gne8B8uzd6ptKVtJrP8pMvoOyydyvQlmkgh8tNDsOnp6kaUNG0NHmtfGlCTkbM7oKsKbf87Cv1HgYuZl/HP7go5sOX02LfBfbvWrawZzOaZ3dewEq74w5hVMKEHrmJPstIpcHSxZpq33f2j4Z/uKW9tJat6rgSWK1I+T3tCd++76apxbOYl5TDOaoxCyjQ== lars@marowsky-bree.de", "key_options": null, "keyfile": "/home/lmb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "lmb", "validate_certs": true} 2021-02-10T10:54:17.240 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIXHA8BwV4OnqlaHHyYofsp5X+wanose+aneJVx2Or05PcWA9c7xFEOerkvEn9LvivSRd9G9o/S8SvnGH91tG2bbOAidVZ69FGul9OKa/BCJYUP0vVBX09/Td3n7+R2lVlvu5XiyC5SUzCjaQEZLOpjq23rqaqdCDXE6wFnxfpkmcsccbG/xUiOEBeBGI92lntdx99Sn6CxWWA27lMMaYkCRvX6ziRZEm6O44N2nMm+qYm0jmMyGv1iyQkJxtt0YaLPp+X9t2trUEzEjgBCXqqzEzBjB7T4rbzhAuvU3NRHiAoWXSnrY5AQOTTvC/wpaZV/gF2NDNThjMcpcEdNstleOctXtN4XTTJQewbG+t7I9knR00TPpsoUvUtpbcoFOluGgWTDTFa6GCmDPppx23nZgulOniZUVTI2CbScSpc8sfp1JE7Q6vLECY9lJ+GpVCbDAbIEE9WHXSkJ/6L9yf+ZoHbpqCkEFDcWLfz3B5NFoJ5gne8B8uzd6ptKVtJrP8pMvoOyydyvQlmkgh8tNDsOnp6kaUNG0NHmtfGlCTkbM7oKsKbf87Cv1HgYuZl/HP7go5sOX02LfBfbvWrawZzOaZ3dewEq74w5hVMKEHrmJPstIpcHSxZpq33f2j4Z/uKW9tJat6rgSWK1I+T3tCd++76apxbOYl5TDOaoxCyjQ== lars@marowsky-bree.de", "key_options": null, "keyfile": "/home/lmb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "lmb", "validate_certs": true} 2021-02-10T10:54:17.719 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMDUcToIhs8tiIozgPudiYyD2upEX+VvsiJRti7zrA/1HPIDARYt2+xyWCASm5x5vdohRM7wnBLUlbBVnvr6ZUQvfShNDsS1Uu7hEfYF38MtW+dVwvGlm14J4m1gbjzqVlZvO6bg6rfGo7Qer0bKUlxtqbisQ/kCg/lVWR04CNxdRtZxEpaQ21y4C5jAlrpFTzxQSuiCLEyY3EAKdZUaZElFEcXlna0ysCP8inlzhF402PkSdl9DkkPpg3IR2dUEGqsblWSdLVk19EFavjtGAAnsvaBWWm2rCLNC5RmhGQaS/Z52Aes1nBuT4cI6qLjbEMqAzLwxiOR7l4IPMpK5EcuRK+FaqQG0wMAjxPc95CmxDzi58ZRJ5ERV8vgps73F1AJXWU9Lb1P77BTsLXqiwko2mvPf6e0LwCxQuxYJWc+xvO7Ji011B+nm5eLCCC/LcDtTPjkimTh7nkvUNi/yTKCXi9KwLISAHHd335KnfrL28rU3GahMXs4NjOxuN8eKM= ksirivad@ksirivad\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDS3HkgHb6z+gi4tN5rBL/N+WLJgN8RtUokt0YnRjpor2epRbmqgmEFRbRFqjLK9LiTuLxmkwol1/G56AJms9ixue33M6jVs73nc259LDxn2qj/NCtxeU7xdjSDuhKyItgM0yuiFjWOKSYTghNtwoR26PX7SQVbycD4NN1ZuwUbdMQA31VrLvQR7d0lYKlis2auH2Xb6WifotoEyszZz1KYRwzJvOHtHku/jAyNmTD/8kAsuCjEgUPSyyehAfdWn59tY2gM7aFyj/dtRoAghekv5gkov9zx+zjN6D2LSN7e+DRti05iJMifA3nfmdj7f8aHzXDtsDL0XNkmmT3g9N2OQ8WL4ESOFASXPPj9JGUROealgnaQ+Y64evDClhbt2L7FLbgj5sgKb+XYhpGnAiC+v3t1Rq3HZrstOYf2qSYguIo+zW9gHwzJDqH5YNCpSF4ZDCfkJ94ngl5ptQmZ/FcrH2hBs2cMoyK9EyDpcI9hBeACTzu19kaB6DrOC+UTvBgYL2sNbCvi+caIqhENJICZ21rI4u4EZqu7PVb+Y3CLgSjTEgclGYBZoxlrT/X2QHrIfZF7lLGeeh0bureedCrBg4b61SL7ZU3Me9x1iWOZMSJgVvg+qPSrzZiHbA46k8i0af5Q+geL1BuuGy76zbYd/KR0KiNZnXjdlHyRz76rpQ== kamoltatsirivadhna@Ksirivad-MacBook-Pro.loca 2021-02-10T10:54:17.721 INFO:teuthology.task.ansible.out:l", "key_options": null, "keyfile": "/home/ksirivad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ksirivad", "validate_certs": true} ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMDUcToIhs8tiIozgPudiYyD2upEX+VvsiJRti7zrA/1HPIDARYt2+xyWCASm5x5vdohRM7wnBLUlbBVnvr6ZUQvfShNDsS1Uu7hEfYF38MtW+dVwvGlm14J4m1gbjzqVlZvO6bg6rfGo7Qer0bKUlxtqbisQ/kCg/lVWR04CNxdRtZxEpaQ21y4C5jAlrpFTzxQSuiCLEyY3EAKdZUaZElFEcXlna0ysCP8inlzhF402PkSdl9DkkPpg3IR2dUEGqsblWSdLVk19EFavjtGAAnsvaBWWm2rCLNC5RmhGQaS/Z52Aes1nBuT4cI6qLjbEMqAzLwxiOR7l4IPMpK5EcuRK+FaqQG0wMAjxPc95CmxDzi58ZRJ5ERV8vgps73F1AJXWU9Lb1P77BTsLXqiwko2mvPf6e0LwCxQuxYJWc+xvO7Ji011B+nm5eLCCC/LcDtTPjkimTh7nkvUNi/yTKCXi9KwLISAHHd335KnfrL28rU3GahMXs4NjOxuN8eKM= ksirivad@ksirivad\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDS3HkgHb6z+gi4tN5rBL/N+WLJgN8RtUokt0YnRjpor2epRbmqgmEFRbRFqjLK9LiTuLxmkwol1/G56AJms9ixue33M6jVs73nc259LDxn2qj/NCtxeU7xdjSDuhKyItgM0yuiFjWOKSYTghNtwoR26PX7SQVbycD4NN1ZuwUbdMQA31VrLvQR7d0lYKlis2auH2Xb6WifotoEyszZz1KYRwzJvOHtHku/jAyNmTD/8kAsuCjEgUPSyyehAfdWn59tY2gM7aFyj/dtRoAghekv5gkov9zx+zjN6D2LSN7e+DRti05iJMifA3nfmdj7f8aHzXDtsDL0XNkmmT3g9N2OQ8WL4ESOFASXPPj9JGUROealgnaQ+Y64evDClhbt2L7FLbgj5sgKb+XYhpGnAiC+v3t1Rq3HZrstOYf2qSYguIo+zW9gHwzJDqH5YNCpSF4ZDCfkJ94ngl5ptQmZ/FcrH2hBs2cMoyK9EyDpcI9hBeACTzu19kaB6DrOC+UTvBgYL2sNbCvi+caIqhENJICZ21rI4u4EZqu7PVb+Y3CLgSj 2021-02-10T10:54:17.722 INFO:teuthology.task.ansible.out:TEgclGYBZoxlrT/X2QHrIfZF7lLGeeh0bureedCrBg4b61SL7ZU3Me9x1iWOZMSJgVvg+qPSrzZiHbA46k8i0af5Q+geL1BuuGy76zbYd/KR0KiNZnXjdlHyRz76rpQ== kamoltatsirivadhna@Ksirivad-MacBook-Pro.local", "key_options": null, "keyfile": "/home/ksirivad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ksirivad", "validate_certs": true} 2021-02-10T10:54:18.176 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0POsA+0NyxecLo3Y821G6W3+jJPKFMuLSIl4FSLi25vuyuM5+mFGmCNWv8gg93eQhG9Y3SOygqmkaxzxDtBS/V6VNC2f8hZsVw9B+HbnfggvCNQSgjJ5rhvf6IHfx/aHq4IHF+stqdcoWhwR/68wIhGNqtP7Nhqnzk2aN1kG9bU2iVk4CH8XGIwqwET3aWNuCMcHi2AhrvDBpZu7bzo/6ARUJDqEfTEC/XIkzv8/mEkJ5igcycARVy6dChuCwfZc8rhfwQpekwx1Y9/WBFdNQlYkN4fVe8yIqT5VsQ3AcRv21GJk8L7mwB4hF9dVAi4p1cSyxatR3kDifsFc8VQk9 toor@localhost.localdomain", "key_options": null, "keyfile": "/home/ofriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ofriedma", "validate_certs": true} 2021-02-10T10:54:18.187 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0POsA+0NyxecLo3Y821G6W3+jJPKFMuLSIl4FSLi25vuyuM5+mFGmCNWv8gg93eQhG9Y3SOygqmkaxzxDtBS/V6VNC2f8hZsVw9B+HbnfggvCNQSgjJ5rhvf6IHfx/aHq4IHF+stqdcoWhwR/68wIhGNqtP7Nhqnzk2aN1kG9bU2iVk4CH8XGIwqwET3aWNuCMcHi2AhrvDBpZu7bzo/6ARUJDqEfTEC/XIkzv8/mEkJ5igcycARVy6dChuCwfZc8rhfwQpekwx1Y9/WBFdNQlYkN4fVe8yIqT5VsQ3AcRv21GJk8L7mwB4hF9dVAi4p1cSyxatR3kDifsFc8VQk9 toor@localhost.localdomain", "key_options": null, "keyfile": "/home/ofriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ofriedma", "validate_certs": true} 2021-02-10T10:54:18.621 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDc/8sWrMq+bqKBPmDPplUgdMoY2SQL+57jRSh27PAoYW9rNVR/UgK7EA9oxFLfoi9wJ9+VKU4opeoohOXLWQXXatLvfXjGE3fRYIonQfwJqKrasqQJTGWzRZ9t1/2Cb0VHkoAvipCJzWmgB46X6q73XW5Td1H3b+AINd6E4Rsm1QjBOhk0DMjCdfwrVUZIfGdsEo+5u/AwioHwH+NDJvXTWT8ZigeZBNHJSY/EQn1+15ut0QB2rvXkJasvvDlVbEsPQtM4C3VX1ronmAnKMvCp1O4EtAXaoKSOx7XEN++UAvVn7czDCWk/UFgpzZOqTcQJwg82XtS4sZP23CpNe0G9 h4iresis@fedora", "key_options": null, "keyfile": "/home/h4iresis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "h4iresis", "validate_certs": true} 2021-02-10T10:54:18.643 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDc/8sWrMq+bqKBPmDPplUgdMoY2SQL+57jRSh27PAoYW9rNVR/UgK7EA9oxFLfoi9wJ9+VKU4opeoohOXLWQXXatLvfXjGE3fRYIonQfwJqKrasqQJTGWzRZ9t1/2Cb0VHkoAvipCJzWmgB46X6q73XW5Td1H3b+AINd6E4Rsm1QjBOhk0DMjCdfwrVUZIfGdsEo+5u/AwioHwH+NDJvXTWT8ZigeZBNHJSY/EQn1+15ut0QB2rvXkJasvvDlVbEsPQtM4C3VX1ronmAnKMvCp1O4EtAXaoKSOx7XEN++UAvVn7czDCWk/UFgpzZOqTcQJwg82XtS4sZP23CpNe0G9 h4iresis@fedora", "key_options": null, "keyfile": "/home/h4iresis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "h4iresis", "validate_certs": true} 2021-02-10T10:54:19.065 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCqhWVTosH12864VGY5S3/k0yKc59GbV0m2WxD28gAf+uvJ1/y8gekD2wDzBv0DWqEekYLV7L68gpapDXyGosQeZ6n5OO/NZFZmW+Xq1P7P1QHJil6kVpwk927lzX440pwNf3+PRvTbLzlbuVFWJRpiykv+juJyIFjzXLJRF67wB2qgZVm5cRs/GOi4fnvTaou/Ae51mvsO8M3JL5cphApFKD/AO/X+3pHBhLjvaW33cYsp0Z4Rce1nm8wxSmvuJRF+HAfjBlknhDl5ZaJGA96RYA+g9WbAXimFprg4xeiRp7liPExQwtVgackJweoC4MahC1glmjC3eOh9dvrrxLnALOLMOkyCSK/tOF/iq2PJaLViObanPI8mnNkMLbdJyzUBf8hRGdSLcw98qxaf0sSZ+vz4VQS6e9CwX7ikkNNsTeYXjnJ+1yU3mfCW1CXJDr7kBDVG2OTLefCpMY8WBAYDiEaK0RW83cBcFIKfecKYd0qStHRaOsvmyWdFYgBh2vM1UWjro5WXlb+7aJPrUp8ZruapkUXUwhjHA7FwGA4ysE0tsOVVFm0Dw8tvyVVkVNm/9EObG28dWctioZ2+ZEzTWgMJU26Q+wOJ/U4qZyIvpMRUgPZc5nM9J8kcmvNWtb6DV9k/6p8h28kiOxPkxLtPP+3TlBCHjq//mhEnJZ/d8w== shshyukriev@suse.com", "key_options": null, "keyfile": "/home/shyukri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyukri", "validate_certs": true} 2021-02-10T10:54:19.097 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCqhWVTosH12864VGY5S3/k0yKc59GbV0m2WxD28gAf+uvJ1/y8gekD2wDzBv0DWqEekYLV7L68gpapDXyGosQeZ6n5OO/NZFZmW+Xq1P7P1QHJil6kVpwk927lzX440pwNf3+PRvTbLzlbuVFWJRpiykv+juJyIFjzXLJRF67wB2qgZVm5cRs/GOi4fnvTaou/Ae51mvsO8M3JL5cphApFKD/AO/X+3pHBhLjvaW33cYsp0Z4Rce1nm8wxSmvuJRF+HAfjBlknhDl5ZaJGA96RYA+g9WbAXimFprg4xeiRp7liPExQwtVgackJweoC4MahC1glmjC3eOh9dvrrxLnALOLMOkyCSK/tOF/iq2PJaLViObanPI8mnNkMLbdJyzUBf8hRGdSLcw98qxaf0sSZ+vz4VQS6e9CwX7ikkNNsTeYXjnJ+1yU3mfCW1CXJDr7kBDVG2OTLefCpMY8WBAYDiEaK0RW83cBcFIKfecKYd0qStHRaOsvmyWdFYgBh2vM1UWjro5WXlb+7aJPrUp8ZruapkUXUwhjHA7FwGA4ysE0tsOVVFm0Dw8tvyVVkVNm/9EObG28dWctioZ2+ZEzTWgMJU26Q+wOJ/U4qZyIvpMRUgPZc5nM9J8kcmvNWtb6DV9k/6p8h28kiOxPkxLtPP+3TlBCHjq//mhEnJZ/d8w== shshyukriev@suse.com", "key_options": null, "keyfile": "/home/shyukri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyukri", "validate_certs": true} 2021-02-10T10:54:19.507 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxDJtmmcnrbA4AUsmBBZq/M4xNdIbOT+cQQxhUKeCu8F9mht+RznwyaaJbVIlD+UNv9LMhRyzI5oktSalokNltlrtPQMoP52PXH7KFFGnVnexCHAk4nGvEhuZCniO5T+ENm2wFR47lFEQK7lyJJLp5PFiSM7vM1MK6UkMvRmCTRildjaYLyf3EEvBXifbsGmR0tBIXtml6KITCAaT6l0E94055oEGN2G5NhZIvfLcojt5liEkXEHQNdfkUDXM0Go43JZzgsSWDXC/7RYLrAJwz5mq4qCa9YZL/3Spjso37+1oLq7lOQ0TDJUAQXmnW0So/NkzECx4MolVcylhTzmgj sergio@milestones", "key_options": null, "keyfile": "/home/scarvalhojr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "scarvalhojr", "validate_certs": true} 2021-02-10T10:54:19.561 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxDJtmmcnrbA4AUsmBBZq/M4xNdIbOT+cQQxhUKeCu8F9mht+RznwyaaJbVIlD+UNv9LMhRyzI5oktSalokNltlrtPQMoP52PXH7KFFGnVnexCHAk4nGvEhuZCniO5T+ENm2wFR47lFEQK7lyJJLp5PFiSM7vM1MK6UkMvRmCTRildjaYLyf3EEvBXifbsGmR0tBIXtml6KITCAaT6l0E94055oEGN2G5NhZIvfLcojt5liEkXEHQNdfkUDXM0Go43JZzgsSWDXC/7RYLrAJwz5mq4qCa9YZL/3Spjso37+1oLq7lOQ0TDJUAQXmnW0So/NkzECx4MolVcylhTzmgj sergio@milestones", "key_options": null, "keyfile": "/home/scarvalhojr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "scarvalhojr", "validate_certs": true} 2021-02-10T10:54:19.958 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEOigBGMeElb3uHXX36HxLtPJMG+iG/6D7NDFCVR6AyYsmimBQb6SlSIR6SaFULb6HT/IFVk0Jf/z0HoipAP7Xu+wvP/J6LFoMdguxylh1ov8raqxQJrGLl9n64EI+qWINwEM0MKMUnDQ3tmMKBl78LKQVQTYBx1+9cXsT6DE1yhgmV79XIvOOb+v53agSi/kLJF2i9K04kpi1OejXg95Ub2eLMY69Jm8PnLy1Jq8DhC8pz7nrfzbKtOC7HbytTAO1XTlb90lVhqC/1yTMAUc/rpJwoSEHc2epGeZWdMsJhqhX2EqMwcV53M3qcFSGreGNJ/Ncd4/JHxz7q0IU411t rakesh@localhost.localdomain", "key_options": null, "keyfile": "/home/rakesh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rakesh", "validate_certs": true} 2021-02-10T10:54:20.001 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEOigBGMeElb3uHXX36HxLtPJMG+iG/6D7NDFCVR6AyYsmimBQb6SlSIR6SaFULb6HT/IFVk0Jf/z0HoipAP7Xu+wvP/J6LFoMdguxylh1ov8raqxQJrGLl9n64EI+qWINwEM0MKMUnDQ3tmMKBl78LKQVQTYBx1+9cXsT6DE1yhgmV79XIvOOb+v53agSi/kLJF2i9K04kpi1OejXg95Ub2eLMY69Jm8PnLy1Jq8DhC8pz7nrfzbKtOC7HbytTAO1XTlb90lVhqC/1yTMAUc/rpJwoSEHc2epGeZWdMsJhqhX2EqMwcV53M3qcFSGreGNJ/Ncd4/JHxz7q0IU411t rakesh@localhost.localdomain", "key_options": null, "keyfile": "/home/rakesh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rakesh", "validate_certs": true} 2021-02-10T10:54:20.400 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvRwrsN62CReT0/kOeFEXGA3THhzJ+hwZVpUzo7VMRZeSo6LgNlVGdrQchMHnlwBvuwaVNJlBAkQr+5J5BB3ikrmbcnLHfPutXBBBB6uDfSwwbqjzUY7elTbz9ul6AkegR9K65O44ouD9+WC76D+Tp0maqnP1w0O3h1GVxDHe34LpEwBvZJ+/BSYmBpwRZqeXFRADORPzdtW08PSiRErGHb8j+j5i+br37JfOOh+A/9dHwQw9alBVL2DijwvrPMa7dV5yX00dlMSTL2LLo9IdI4aFPmie8oic0eanFbQDq210ilOR65IGeshKHEKOq0KFFz0nSK9OdO03OkSEN/0d5 ryg@tapir", "key_options": null, "keyfile": "/home/ryg-/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ryg-", "validate_certs": true} 2021-02-10T10:54:20.433 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvRwrsN62CReT0/kOeFEXGA3THhzJ+hwZVpUzo7VMRZeSo6LgNlVGdrQchMHnlwBvuwaVNJlBAkQr+5J5BB3ikrmbcnLHfPutXBBBB6uDfSwwbqjzUY7elTbz9ul6AkegR9K65O44ouD9+WC76D+Tp0maqnP1w0O3h1GVxDHe34LpEwBvZJ+/BSYmBpwRZqeXFRADORPzdtW08PSiRErGHb8j+j5i+br37JfOOh+A/9dHwQw9alBVL2DijwvrPMa7dV5yX00dlMSTL2LLo9IdI4aFPmie8oic0eanFbQDq210ilOR65IGeshKHEKOq0KFFz0nSK9OdO03OkSEN/0d5 ryg@tapir", "key_options": null, "keyfile": "/home/ryg-/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ryg-", "validate_certs": true} 2021-02-10T10:54:20.840 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com", "key_options": null, "keyfile": "/home/prsrivas/.ssh/authorized_keys", " 2021-02-10T10:54:20.842 INFO:teuthology.task.ansible.out:manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2021-02-10T10:54:20.884 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com", "key_options": null, "keyfile": "/home/prsrivas/.ssh/authorized_keys", " 2021-02-10T10:54:20.885 INFO:teuthology.task.ansible.out:manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2021-02-10T10:54:21.281 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDOJg56HsyNa5vHHb52G6O5vtv9I2tdlNMjFlG3SVCCu/bVZwVeGzp1ekuNGU10owmnIvhlSldMjLn2wwTwiDjbvYJVVCPvzK63vCUcMevRsvr9EyU16JdAgrmrGMyuYYEb+wh8XWI9Dc72o84fMaBYFSSer1S9nUBL+KVLVhIe7culgADimrDgDVh2t7u2JHBI0m0nKFGJZNcIyYf+GKOM+n7SaQXIYIytvSteU5BvUIKPsDd4z5lVlWiKO3kG6jF2RnKFV6KWBWI9m8YGItdGu25mXELXN9MOlApZF3AoGTmCE2Web7uDVGvO6rz6b+dc7RXTOlbWIhtMoRqQf6mz zac.dover@gmail.com", "key_options": null, "keyfile": "/home/zdover/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zdover", "validate_certs": true} 2021-02-10T10:54:21.334 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDOJg56HsyNa5vHHb52G6O5vtv9I2tdlNMjFlG3SVCCu/bVZwVeGzp1ekuNGU10owmnIvhlSldMjLn2wwTwiDjbvYJVVCPvzK63vCUcMevRsvr9EyU16JdAgrmrGMyuYYEb+wh8XWI9Dc72o84fMaBYFSSer1S9nUBL+KVLVhIe7culgADimrDgDVh2t7u2JHBI0m0nKFGJZNcIyYf+GKOM+n7SaQXIYIytvSteU5BvUIKPsDd4z5lVlWiKO3kG6jF2RnKFV6KWBWI9m8YGItdGu25mXELXN9MOlApZF3AoGTmCE2Web7uDVGvO6rz6b+dc7RXTOlbWIhtMoRqQf6mz zac.dover@gmail.com", "key_options": null, "keyfile": "/home/zdover/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zdover", "validate_certs": true} 2021-02-10T10:54:21.735 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC4zwWf+kMEDgaSOrX7AlD+UCOgf4/fRiorQVJqC8GfMNynfiFVZaThngwIrwtQROfXHbzspi+AcRvDLewY25mXyuxVgOfSbuSIEqiUNvzcVq9IItI1J58xeDBpIktAsoHHS6JENV56fAT6krzRAQLnSa1k2b0pyKuPBzFSeNgiaQMhjrCLAaCi9yLsdKpnWgXhyfrdqvN2r/Ctts/zlTPqtRYdvdIL/p/l+vg7Q8S3OhuAGtimaEq3RE70xadaPbGhsLISMTrbfYXbxw84KKZpNYwO4zmQ2ztcvC9D7A/8ov3+A5Jq42G+pq7ORuix2NoTWtADMiwvNe1HyzS/7cX/BuXLFssRY1bzRvp+x2ExZ7b+bFx+TE/RduRh0eyDNLg7SyL83aeTr2Usa+mPoCAMKed9XtYbRnUu7ox4wmfeAgx6jM9uG5uQfauJ1XpDddrVBG8KYBvSd0BM2cyRdcAXqwxg9AVPZcdE7sbBk44CGVhw+ziZ3e3cLBTO4y44KRQre+1Bhe/Ze8OTQWLFugl/XagFlJE/L6fGUIPuFojmQvlw5dBZwtUURnxsu8yHm3iIsj4y8nR1zHMj+zuMk6rF3ZW0epPAECAdq4gG8X2wv3HTq0vLBkKpi1KMAmntEtcVwF0fzMt5XYqRd4jceFgonAAipJNDknHgi6yaUzBYmQ== rraja@redhat.com", "key_options": null, "keyfile": "/home/rraja/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rraja", "validate_certs": true} 2021-02-10T10:54:21.768 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC4zwWf+kMEDgaSOrX7AlD+UCOgf4/fRiorQVJqC8GfMNynfiFVZaThngwIrwtQROfXHbzspi+AcRvDLewY25mXyuxVgOfSbuSIEqiUNvzcVq9IItI1J58xeDBpIktAsoHHS6JENV56fAT6krzRAQLnSa1k2b0pyKuPBzFSeNgiaQMhjrCLAaCi9yLsdKpnWgXhyfrdqvN2r/Ctts/zlTPqtRYdvdIL/p/l+vg7Q8S3OhuAGtimaEq3RE70xadaPbGhsLISMTrbfYXbxw84KKZpNYwO4zmQ2ztcvC9D7A/8ov3+A5Jq42G+pq7ORuix2NoTWtADMiwvNe1HyzS/7cX/BuXLFssRY1bzRvp+x2ExZ7b+bFx+TE/RduRh0eyDNLg7SyL83aeTr2Usa+mPoCAMKed9XtYbRnUu7ox4wmfeAgx6jM9uG5uQfauJ1XpDddrVBG8KYBvSd0BM2cyRdcAXqwxg9AVPZcdE7sbBk44CGVhw+ziZ3e3cLBTO4y44KRQre+1Bhe/Ze8OTQWLFugl/XagFlJE/L6fGUIPuFojmQvlw5dBZwtUURnxsu8yHm3iIsj4y8nR1zHMj+zuMk6rF3ZW0epPAECAdq4gG8X2wv3HTq0vLBkKpi1KMAmntEtcVwF0fzMt5XYqRd4jceFgonAAipJNDknHgi6yaUzBYmQ== rraja@redhat.com", "key_options": null, "keyfile": "/home/rraja/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rraja", "validate_certs": true} 2021-02-10T10:54:22.172 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2mEyxfkW1u84u5BocLCAXESj2qObWkxUd20cmPolhJYlIWLOafa69Dw5A9CjFBp0U6Jx5d0JWcwmAc5ZeffQ7t7vzmtHCZnpKkyZhN9bu6vuCh4N+I643SI/K7tdbB2E8ZE6uI7ORMAan5+t0bNSVolUVq6X1w7dIA5El54K10WNdiwgRKW6e3anNbJ3rBLGEk7gawZqDDOqTvVQmOBGWek3BjSReDvg3rkmTQLxJkiKdxnx4/s7gzH1xTpsS/OludIuSxF0BK064ZcT1t09qcjnT4Os1inV2GfYswt+UhpgqrHqXjEs0Z3y4Oy77QBKL1rDf93CP4a4+eJjqXe+v tbechtold@suse.com", "key_options": null, "keyfile": "/home/toabctl/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "toabctl", "validate_certs": true} 2021-02-10T10:54:22.225 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2mEyxfkW1u84u5BocLCAXESj2qObWkxUd20cmPolhJYlIWLOafa69Dw5A9CjFBp0U6Jx5d0JWcwmAc5ZeffQ7t7vzmtHCZnpKkyZhN9bu6vuCh4N+I643SI/K7tdbB2E8ZE6uI7ORMAan5+t0bNSVolUVq6X1w7dIA5El54K10WNdiwgRKW6e3anNbJ3rBLGEk7gawZqDDOqTvVQmOBGWek3BjSReDvg3rkmTQLxJkiKdxnx4/s7gzH1xTpsS/OludIuSxF0BK064ZcT1t09qcjnT4Os1inV2GfYswt+UhpgqrHqXjEs0Z3y4Oy77QBKL1rDf93CP4a4+eJjqXe+v tbechtold@suse.com", "key_options": null, "keyfile": "/home/toabctl/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "toabctl", "validate_certs": true} 2021-02-10T10:54:22.613 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCitWIN7jm5KEmB6jqcA+ufKRLodxkJ1GIdaaueCdeXdMHgWNH9FUDCUJk5dcA11MxQO15DRH/cdhb3QHPGiMv4XrBeK6HHYwGttzkcUEz1EDUhpowOrnFETbPa7K39ObmRcgdvoGXOSrNZ4mvptCTz9Xb0MF+Gcg5iJ4g+qZzIVa2/ueecYY8ZorbaH4m/NWUub6Rf01Q04HfxTvKrUFswrbCoE/Aigz/Oh88RiS854XoTvV7rE1tKbyKkaungwMAaQSNnZkq4hi8nbp6pzTxJLV7N/ZYMhVpdxSbWcghFg6cy7aeoCzZMN97WcDhpdfbhRT0dbqCkMmHV6t0usDDnOss2v68UCzkxuA1Hj4uy8rRHjXBtYqI6xsrwCjvmgP05B14TpmFdG+3GdRHx2+QlhnGj7nuPf1KoWW/f3tD85DXoEmVl6RJ4KTrMeEVlOn3kdMLVH0y2hIqPTYMP5cMVNi0EdfeoBdd9kxMKMvoSVBAgmLzGVv17o1lNvVS6V1kyt7gME2PfNR2Np+U1cJVJfUJRTm28mHXBVoDXG9s0zv8+LpQ/gVPS+vsPAfxWWYGW8ZmEV3ZorDOZycSPybjHkPFzDAdUhPG0Db0o+F9ug+22F7pvifsdDZ42zqGGLbyzfIQE3AtyI7LR7Sc5tpN9rKTeSBYJ0yNcDKEj5FUHDw== mfritch@suse.com", "key_options": null, "keyfile": "/home/mgfritch/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mgfritch", "validate_certs": true} 2021-02-10T10:54:22.667 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCitWIN7jm5KEmB6jqcA+ufKRLodxkJ1GIdaaueCdeXdMHgWNH9FUDCUJk5dcA11MxQO15DRH/cdhb3QHPGiMv4XrBeK6HHYwGttzkcUEz1EDUhpowOrnFETbPa7K39ObmRcgdvoGXOSrNZ4mvptCTz9Xb0MF+Gcg5iJ4g+qZzIVa2/ueecYY8ZorbaH4m/NWUub6Rf01Q04HfxTvKrUFswrbCoE/Aigz/Oh88RiS854XoTvV7rE1tKbyKkaungwMAaQSNnZkq4hi8nbp6pzTxJLV7N/ZYMhVpdxSbWcghFg6cy7aeoCzZMN97WcDhpdfbhRT0dbqCkMmHV6t0usDDnOss2v68UCzkxuA1Hj4uy8rRHjXBtYqI6xsrwCjvmgP05B14TpmFdG+3GdRHx2+QlhnGj7nuPf1KoWW/f3tD85DXoEmVl6RJ4KTrMeEVlOn3kdMLVH0y2hIqPTYMP5cMVNi0EdfeoBdd9kxMKMvoSVBAgmLzGVv17o1lNvVS6V1kyt7gME2PfNR2Np+U1cJVJfUJRTm28mHXBVoDXG9s0zv8+LpQ/gVPS+vsPAfxWWYGW8ZmEV3ZorDOZycSPybjHkPFzDAdUhPG0Db0o+F9ug+22F7pvifsdDZ42zqGGLbyzfIQE3AtyI7LR7Sc5tpN9rKTeSBYJ0yNcDKEj5FUHDw== mfritch@suse.com", "key_options": null, "keyfile": "/home/mgfritch/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mgfritch", "validate_certs": true} 2021-02-10T10:54:23.065 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5bCXQ/B16AKl2oevC93R0U+3OrTyd3XEypouJy4LjIzXLX/Go1C1ViERwDAeHmqiUTqJUL+KZ0kMo8U7zlhNBk9PucbMZCqdqx5hkiCAo4y65lEXmuPGimRU2mK9TpcDTLWoyrD/EEhwzT6gmw3ytVTxi7qAb0tQUCQB+pW4+Zr5u6HZzgu9kLGxp7zsGtMA84Zqm5ql72mGuS0bUN1ajm8LWBetFuzOOW6YGfnKt29cLV2h1itVewA9DKM5G70B91eK4LjoNddCkqVA8L+v6LMQNU2pejUzPap98hmFUpQMNxlWQ0+B7UjfbkHWe+mIsSkdr3FfCfqWvsL0RiHcZ root@lxbceph0", "key_options": null, "keyfile": "/home/xiubli/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiubli", "validate_certs": true} 2021-02-10T10:54:23.108 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5bCXQ/B16AKl2oevC93R0U+3OrTyd3XEypouJy4LjIzXLX/Go1C1ViERwDAeHmqiUTqJUL+KZ0kMo8U7zlhNBk9PucbMZCqdqx5hkiCAo4y65lEXmuPGimRU2mK9TpcDTLWoyrD/EEhwzT6gmw3ytVTxi7qAb0tQUCQB+pW4+Zr5u6HZzgu9kLGxp7zsGtMA84Zqm5ql72mGuS0bUN1ajm8LWBetFuzOOW6YGfnKt29cLV2h1itVewA9DKM5G70B91eK4LjoNddCkqVA8L+v6LMQNU2pejUzPap98hmFUpQMNxlWQ0+B7UjfbkHWe+mIsSkdr3FfCfqWvsL0RiHcZ root@lxbceph0", "key_options": null, "keyfile": "/home/xiubli/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiubli", "validate_certs": true} 2021-02-10T10:54:23.504 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDT+6B5Fr99p8xe9Zo5x8ULcdeA6ne727iVJp/xaguGwGHj6AhuK+R/CjznnHk0yA+6kiblNzjvbnsuzO8T9c8Ayv5KU07MxP1y1pymdYBpEh3NTXmhB6QOTCNTQbT14PL6MVgzK17SqarFCYXoEGuyK9jaBdN9QM76vhEaXc/2Osg+d8boJTkOk7lOPHbzeOCbQU3d+jX0VFOgAQS5OFz3Ug/X7nB0D0LN8bt5FlCZiGO95Iwls/SyPDbVXc9ot8oh4ME7kd7sElokBdYdXPauibRxWKjSDn+hV6kZfSzqwnMFaZ1TC+Av6zNsBuZucNSp3/Ws3T5i9i+21eyS4+qVE8IiX2aVKYuC9afb2lPLbdG2Iy1CqiaeGPkZfiGT9P8iQPOkqae45859UzFAd/XzKjBcmus3lSArKG1EyZl03HQ5P2J60pErCXq8W9mPjxAze2ReLRXJVEgTXnmdhCptcg6PZ4d/k4BEddbdnoX6/BEgFxLEJ64HpTeTVcDXHpipCpxFSRvgyPH/wQGKAvt8AfKBhmeZs7LuyGOhJiOon4DDaShHpE9wTnJ5qlnyg44+u6NCwelGPMAkHKEO4OgcjSsyXm+HYUpxG2s4poloIJzvU6oa3dvJsMZFhcb8CsW75sBYE+YngS/1fg1WU88ZOdaa1yF4F87pVZycpZqY1w== jan@jfsuselaptop", "key_options": null, "keyfile": "/home/jafaj/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jafaj", "validate_certs": true} 2021-02-10T10:54:23.548 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDT+6B5Fr99p8xe9Zo5x8ULcdeA6ne727iVJp/xaguGwGHj6AhuK+R/CjznnHk0yA+6kiblNzjvbnsuzO8T9c8Ayv5KU07MxP1y1pymdYBpEh3NTXmhB6QOTCNTQbT14PL6MVgzK17SqarFCYXoEGuyK9jaBdN9QM76vhEaXc/2Osg+d8boJTkOk7lOPHbzeOCbQU3d+jX0VFOgAQS5OFz3Ug/X7nB0D0LN8bt5FlCZiGO95Iwls/SyPDbVXc9ot8oh4ME7kd7sElokBdYdXPauibRxWKjSDn+hV6kZfSzqwnMFaZ1TC+Av6zNsBuZucNSp3/Ws3T5i9i+21eyS4+qVE8IiX2aVKYuC9afb2lPLbdG2Iy1CqiaeGPkZfiGT9P8iQPOkqae45859UzFAd/XzKjBcmus3lSArKG1EyZl03HQ5P2J60pErCXq8W9mPjxAze2ReLRXJVEgTXnmdhCptcg6PZ4d/k4BEddbdnoX6/BEgFxLEJ64HpTeTVcDXHpipCpxFSRvgyPH/wQGKAvt8AfKBhmeZs7LuyGOhJiOon4DDaShHpE9wTnJ5qlnyg44+u6NCwelGPMAkHKEO4OgcjSsyXm+HYUpxG2s4poloIJzvU6oa3dvJsMZFhcb8CsW75sBYE+YngS/1fg1WU88ZOdaa1yF4F87pVZycpZqY1w== jan@jfsuselaptop", "key_options": null, "keyfile": "/home/jafaj/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jafaj", "validate_certs": true} 2021-02-10T10:54:23.944 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo8wE/b2/EDwV018ENfcENVfXSF7vnXAwBSzNVScgxdqr2DVKTWDzhoI0m5CXTqI2HLnJVFDTgN3t5/fFcmwmSAP8ic8wU1JznSPZXN7qEQrcEsU/cwHeuzyutibg1ItTWWtoXYV8SFnVom8NF3UwXF8I+Vj6VF1EStWCcClgPjLbiOm8vIgaouXtbUIFv25GpadXyUtwiHEq1NKUPLgka8Usm8ibyS9hY9KmNWlD9AfmnwAImc8Urn2dErcsfAltxoGUHCI7ZBHGpzg+6dpynNR+giJMrjwWzREaCHmiJX3g6jmHuuY54eSOs6cxqi3E4jNHVWXPe/tEF3l0klEvl jschmid@suse.com", "key_options": null, "keyfile": "/home/jschmid/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jschmid", "validate_certs": true} 2021-02-10T10:54:23.986 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo8wE/b2/EDwV018ENfcENVfXSF7vnXAwBSzNVScgxdqr2DVKTWDzhoI0m5CXTqI2HLnJVFDTgN3t5/fFcmwmSAP8ic8wU1JznSPZXN7qEQrcEsU/cwHeuzyutibg1ItTWWtoXYV8SFnVom8NF3UwXF8I+Vj6VF1EStWCcClgPjLbiOm8vIgaouXtbUIFv25GpadXyUtwiHEq1NKUPLgka8Usm8ibyS9hY9KmNWlD9AfmnwAImc8Urn2dErcsfAltxoGUHCI7ZBHGpzg+6dpynNR+giJMrjwWzREaCHmiJX3g6jmHuuY54eSOs6cxqi3E4jNHVWXPe/tEF3l0klEvl jschmid@suse.com", "key_options": null, "keyfile": "/home/jschmid/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jschmid", "validate_certs": true} 2021-02-10T10:54:24.385 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDnQfkkxYWLhMdoQ4RsZxWTXBpbzsssgeRfvV5q+n0mW60dxQ+9rjDXUTsRkmh45B5EhNWlKwyIUcQ5HpgJk6XQvYU6MPboWjriCTngmoUbwNIjOjyazkBbxfbwUiRDb+x1EyVm4Kt0rgj3nhqqxf3krqO2SlKgsVaY9b7+NTfshXb9T73/RRLtvwUOFkjp2B+w0SQs7Ka1GenmHSY/6xP25Mb8q9QW/2nuf1mV9rTd5l2ZHREzgG7gQFhjgUq3w6zCCfcCGwrPb4omqd9xiYLDw8l1vfQ7UPbuOqK1rFM67r7JQDG3yyJjCdWgRUI/ysbbXj0fyT+QOPZGK618V8yZPVlq5loQzYI39zAWZU9L9X9bkQCsJfjU5Yo9f/zOaecnWfUxC84NKpbG7k6XACdUIB9JGGtL6RPtgBU+k6Ts7EemIUCyMYm/GX+PYU3GnmmdqW9yBeyZoldwDCWLq05vjLXj+R3J2UrMl5x+fhP+KEFd1ZgNc7PV+CF35/ux8UE= kachau@localhost.localdomain", "key_options": null, "keyfile": "/home/kachau/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kachau", "validate_certs": true} 2021-02-10T10:54:24.438 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDnQfkkxYWLhMdoQ4RsZxWTXBpbzsssgeRfvV5q+n0mW60dxQ+9rjDXUTsRkmh45B5EhNWlKwyIUcQ5HpgJk6XQvYU6MPboWjriCTngmoUbwNIjOjyazkBbxfbwUiRDb+x1EyVm4Kt0rgj3nhqqxf3krqO2SlKgsVaY9b7+NTfshXb9T73/RRLtvwUOFkjp2B+w0SQs7Ka1GenmHSY/6xP25Mb8q9QW/2nuf1mV9rTd5l2ZHREzgG7gQFhjgUq3w6zCCfcCGwrPb4omqd9xiYLDw8l1vfQ7UPbuOqK1rFM67r7JQDG3yyJjCdWgRUI/ysbbXj0fyT+QOPZGK618V8yZPVlq5loQzYI39zAWZU9L9X9bkQCsJfjU5Yo9f/zOaecnWfUxC84NKpbG7k6XACdUIB9JGGtL6RPtgBU+k6Ts7EemIUCyMYm/GX+PYU3GnmmdqW9yBeyZoldwDCWLq05vjLXj+R3J2UrMl5x+fhP+KEFd1ZgNc7PV+CF35/ux8UE= kachau@localhost.localdomain", "key_options": null, "keyfile": "/home/kachau/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kachau", "validate_certs": true} 2021-02-10T10:54:24.848 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQujcXnq9qOCmTWs44GwKOw48BxZGBmo9AeFk3KPRx33Cz5hS7UHPYvJy03KdoMySgxJtuxsOSvjJSPxpXesTtArO5giW8RLyuVsu9q5j/GhPKyABttuGQyexiRokrHQFLVifzgqwsUARwHarWH16Oa1n6fZPFNqSH56c872zS4Pwqgkzx99NKRWh/B+fIk8VmDzzP1qvQAnUXDeTeTOzapAL+8fSNTp3QnhmZbvCHYwUxfSqJyzq1wBL+517WrvzvX7yCY9B0wzm72sIO4daV6UDyAL1B72QpB04vEZZ/skwS/0jmokfBn43HHCMPy/Mxywfi54alFJzIu4pti30V kravishankar@localhost.localdomain", "key_options": null, "keyfile": "/home/khiremat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "khiremat", "validate_certs": true} 2021-02-10T10:54:24.890 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQujcXnq9qOCmTWs44GwKOw48BxZGBmo9AeFk3KPRx33Cz5hS7UHPYvJy03KdoMySgxJtuxsOSvjJSPxpXesTtArO5giW8RLyuVsu9q5j/GhPKyABttuGQyexiRokrHQFLVifzgqwsUARwHarWH16Oa1n6fZPFNqSH56c872zS4Pwqgkzx99NKRWh/B+fIk8VmDzzP1qvQAnUXDeTeTOzapAL+8fSNTp3QnhmZbvCHYwUxfSqJyzq1wBL+517WrvzvX7yCY9B0wzm72sIO4daV6UDyAL1B72QpB04vEZZ/skwS/0jmokfBn43HHCMPy/Mxywfi54alFJzIu4pti30V kravishankar@localhost.localdomain", "key_options": null, "keyfile": "/home/khiremat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "khiremat", "validate_certs": true} 2021-02-10T10:54:25.308 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCoqsDnRqAeNPi5xiIfFl2i2zqcwLTvcAso3R9+QAfSChw4+wcQzmMV8uQz1jt8UReefg2QL6bwvcQPj5Zw7ADqUjfxm1K0ezgH6RrtHgRZ1SVseTvJUvzmkDZ2gr2ToZAJYsaV7gBLJLZ7TOiQ6jzrtitiKqJZdpZfzadjShecY1TdMo8bSS7XfbRcwri40u82TRM825B4L//QuhMI7akPqa3iJbGEAQPhkGLI3kz19ryVG1UwvAkkt7TQ68zz+piXSmB2CMRti51shT4NuAOq6Qe9y2Obicf88ItkYxvySMQ5jWi2bo7Zhwuq9jUH8lK6I6PHrYc8LOsR9gE/hqkXxTQrkOXEDtHN1mr/KKLrXO0GeBdgh8Gj4gcwgd1RUCE+T9q09WO9j7c0piWK9Fs1+q1FjH06/AfmP4qob1TlJMHhtPyOp+lk2XvABP+sqW4vyrgmXu5vJU+Hgq3X9KE7bKh4WkCem24Np4uA+irhpvZmIoHyINsYW46lGfwJvM8= kapandya@localhost.localdomain", "key_options": null, "keyfile": "/home/kapandya/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kapandya", "validate_certs": true} 2021-02-10T10:54:25.411 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCoqsDnRqAeNPi5xiIfFl2i2zqcwLTvcAso3R9+QAfSChw4+wcQzmMV8uQz1jt8UReefg2QL6bwvcQPj5Zw7ADqUjfxm1K0ezgH6RrtHgRZ1SVseTvJUvzmkDZ2gr2ToZAJYsaV7gBLJLZ7TOiQ6jzrtitiKqJZdpZfzadjShecY1TdMo8bSS7XfbRcwri40u82TRM825B4L//QuhMI7akPqa3iJbGEAQPhkGLI3kz19ryVG1UwvAkkt7TQ68zz+piXSmB2CMRti51shT4NuAOq6Qe9y2Obicf88ItkYxvySMQ5jWi2bo7Zhwuq9jUH8lK6I6PHrYc8LOsR9gE/hqkXxTQrkOXEDtHN1mr/KKLrXO0GeBdgh8Gj4gcwgd1RUCE+T9q09WO9j7c0piWK9Fs1+q1FjH06/AfmP4qob1TlJMHhtPyOp+lk2XvABP+sqW4vyrgmXu5vJU+Hgq3X9KE7bKh4WkCem24Np4uA+irhpvZmIoHyINsYW46lGfwJvM8= kapandya@localhost.localdomain", "key_options": null, "keyfile": "/home/kapandya/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kapandya", "validate_certs": true} 2021-02-10T10:54:25.760 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNEEz49M5KdLcp0cbKrE9iihrH5PEhsPgnAxac+aQwvnTstOOlUOyR+TvHiRa7sgXHuFX/UXo+k3Qmhs5U/sMZWZmmN3Jb3fKT78lSBbGnByclfPoGVTTLOHxwo+RFE79Sp3e30oshJ3DwqYgiX6JjM8B2Y7d5Tv0Ss5uEZswvmCEjdTLDSsoxrsOjVxBpuArOfCxes8PnKDdq21mWldpP66iwFel2GgOGSc85iVHKcOE2OpVMazMzXoMSMFoOpCULKjFfXBuDIqkXwJmtLJaiCgcVmTR56KXh7JpZ7vAeJ9ttv9qgOckqKdI3jEjeDT+TrdLlRWghBE7FRkm0vmun prasad.krishnan@macprasad", "key_options": null, "keyfile": "/home/prasad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prasad", "validate_certs": true} 2021-02-10T10:54:25.823 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNEEz49M5KdLcp0cbKrE9iihrH5PEhsPgnAxac+aQwvnTstOOlUOyR+TvHiRa7sgXHuFX/UXo+k3Qmhs5U/sMZWZmmN3Jb3fKT78lSBbGnByclfPoGVTTLOHxwo+RFE79Sp3e30oshJ3DwqYgiX6JjM8B2Y7d5Tv0Ss5uEZswvmCEjdTLDSsoxrsOjVxBpuArOfCxes8PnKDdq21mWldpP66iwFel2GgOGSc85iVHKcOE2OpVMazMzXoMSMFoOpCULKjFfXBuDIqkXwJmtLJaiCgcVmTR56KXh7JpZ7vAeJ9ttv9qgOckqKdI3jEjeDT+TrdLlRWghBE7FRkm0vmun prasad.krishnan@macprasad", "key_options": null, "keyfile": "/home/prasad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prasad", "validate_certs": true} 2021-02-10T10:54:26.203 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCu9CW99ySPXSKQpLbefu1A0KqbUQIzvUOJizIu4azEgdlOqUnOMR/GC6jgOdiVL5mNb/U5bg7yQuEHgPAtmPRotvbI6/FSwIAR9+GgiuKSOJCa4s7QzWbHwv9g7wS2SLURw8UERSzlLlT8KQyGj+GkBhToWp4PTXC7te61c8XJNGF6giLmIAky4SH8uPI1zeLwwviJQomjFWZZ1RTbcRJxvGPjPtuUKDlF7QgXTLE9ZgsNopusC6PfRd7N8Udkl2OYPAzA+Jun0IRnQyjRXG3c2sYXQR8YYGICsxvnm4l81vKdxRL34K7LqBUHQXUcoCjJy7+/2/1dca/AMHgi4sZwgeYQ/U9jZg5jbm7o4MpDUFsS8W6OO/ZPiuYTswGBNRXhPmbGFnfa9dBXBS0p+buHF8PcTSLH0Dp2nhHkK1ehGmBFVPw6nPj3wXA9BKuNc+g7AiNUyRKyCWroFJ0VwUQGbI6ZMccO2cI7QpYmIqPCl3am/vUSNq4j0m/iURR7+t0= vikhyat@redhat", "key_options": null, "keyfile": "/home/vikhyat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vikhyat", "validate_certs": true} 2021-02-10T10:54:26.267 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCu9CW99ySPXSKQpLbefu1A0KqbUQIzvUOJizIu4azEgdlOqUnOMR/GC6jgOdiVL5mNb/U5bg7yQuEHgPAtmPRotvbI6/FSwIAR9+GgiuKSOJCa4s7QzWbHwv9g7wS2SLURw8UERSzlLlT8KQyGj+GkBhToWp4PTXC7te61c8XJNGF6giLmIAky4SH8uPI1zeLwwviJQomjFWZZ1RTbcRJxvGPjPtuUKDlF7QgXTLE9ZgsNopusC6PfRd7N8Udkl2OYPAzA+Jun0IRnQyjRXG3c2sYXQR8YYGICsxvnm4l81vKdxRL34K7LqBUHQXUcoCjJy7+/2/1dca/AMHgi4sZwgeYQ/U9jZg5jbm7o4MpDUFsS8W6OO/ZPiuYTswGBNRXhPmbGFnfa9dBXBS0p+buHF8PcTSLH0Dp2nhHkK1ehGmBFVPw6nPj3wXA9BKuNc+g7AiNUyRKyCWroFJ0VwUQGbI6ZMccO2cI7QpYmIqPCl3am/vUSNq4j0m/iURR7+t0= vikhyat@redhat", "key_options": null, "keyfile": "/home/vikhyat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vikhyat", "validate_certs": true} 2021-02-10T10:54:26.645 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCipD0d1tI/Cff3AeOZTQ9eb6pZ/NOCAOw7MZG/JgpCs8+T3WqiUEYefsb2PuSa85vV2GC7kq0M9ux7+U+vZZf05geYiU2S9eTucLEu2+YE+RaAvIK+ZB134/33JSPGIVubXgGWZw/ZFCEYXmYk3kyEg1YDIk/BEaoG7PsDiSB4j0W7/WiV2bEEdSlTXt5C1/kH0qn4H9wQMiBSpdMgdaRWw60HjObDd6LASeUVgabbyclypravZdpYBnits4ayoN8VG1aZe4Qhm1SzP/CfxAAYfxo9NbqSBWJmn3iSN8UKHhCFrRCTaYic3z3usr8784VsA7z/LxK0fs6xjhoxRTa3 shraddha.agrawal000@gmail.com", "key_options": null, "keyfile": "/home/shraddhaag/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shraddhaag", "validate_certs": true} 2021-02-10T10:54:26.713 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCipD0d1tI/Cff3AeOZTQ9eb6pZ/NOCAOw7MZG/JgpCs8+T3WqiUEYefsb2PuSa85vV2GC7kq0M9ux7+U+vZZf05geYiU2S9eTucLEu2+YE+RaAvIK+ZB134/33JSPGIVubXgGWZw/ZFCEYXmYk3kyEg1YDIk/BEaoG7PsDiSB4j0W7/WiV2bEEdSlTXt5C1/kH0qn4H9wQMiBSpdMgdaRWw60HjObDd6LASeUVgabbyclypravZdpYBnits4ayoN8VG1aZe4Qhm1SzP/CfxAAYfxo9NbqSBWJmn3iSN8UKHhCFrRCTaYic3z3usr8784VsA7z/LxK0fs6xjhoxRTa3 shraddha.agrawal000@gmail.com", "key_options": null, "keyfile": "/home/shraddhaag/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shraddhaag", "validate_certs": true} 2021-02-10T10:54:27.089 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpE5V9JG1WhkHSYqjqaijBb1MLyjcKQlmj5WQ8w5832n6x9J4yDO8wI4qmiX3XeiXPlmW6XjAq5keyukrQe/yVI6NDXzgRxaHDmNj1AR73gyUVqRXpJprrqqsnLHawIRovrBD/HNhTv/2G8xG3BHakac1+qj7tZe4Jjx0aG1HbRhhJ5Q98EgvTkDAhktnlvF9OXXC1WI7QirHx0pRHa40SYyFLfOQDbQSCzb1TAAY5NGn3SwK3s2mWeoEsAJpCLrkNfSTbR3Z0mylAJUn+PayqhJegjwKmLaONjx8ZHXN970JHgojBBGOD30J/NnJQ7N6eRl+ekjpHmLAYGbe2oCwD gkyratsas@suse.com", "key_options": null, "keyfile": "/home/gkyratsas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gkyratsas", "validate_certs": true} 2021-02-10T10:54:27.165 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpE5V9JG1WhkHSYqjqaijBb1MLyjcKQlmj5WQ8w5832n6x9J4yDO8wI4qmiX3XeiXPlmW6XjAq5keyukrQe/yVI6NDXzgRxaHDmNj1AR73gyUVqRXpJprrqqsnLHawIRovrBD/HNhTv/2G8xG3BHakac1+qj7tZe4Jjx0aG1HbRhhJ5Q98EgvTkDAhktnlvF9OXXC1WI7QirHx0pRHa40SYyFLfOQDbQSCzb1TAAY5NGn3SwK3s2mWeoEsAJpCLrkNfSTbR3Z0mylAJUn+PayqhJegjwKmLaONjx8ZHXN970JHgojBBGOD30J/NnJQ7N6eRl+ekjpHmLAYGbe2oCwD gkyratsas@suse.com", "key_options": null, "keyfile": "/home/gkyratsas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gkyratsas", "validate_certs": true} 2021-02-10T10:54:27.534 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC9sZe9p48s6RSpluK1R97X9tsN3tWiidPbdyXvyMUXifYQb3nth9wMWUakrVBp+QGEkqALLIf3so3MfgbRvYexzNfHvRT7Lxn8D/isiCpWa4aLM/0alprV+RI38dR0mI3IVb3aIEYmgp2IY4yzybWTHCmS1VUrUJJ0pOfXOssq0xgo3qeaSQBHYRKasqR8GWAD5MWpG6/kg6gxI7o0aHVVTXy5ADB7NS1LEZr06xxNm4exeYvl9TzXrbGZWf2gC8ZpXhmFHI80A+TKEZRU6wmcju7G69nyEEF8/ebcBKmZB4eTsDN7qR7lc9dxKxiiZcJXc/MWy4bSu69NhOeuq2xx7bdexS8OHKdgYH9gsJolu5zgN20IHzaRgswO3XIe3nh2Ud8wntwBCwmJ9bf5YByITGjpOhntkvWAP3AXkQ1B4HXCdlUNW+4pMexDe2D05LHOwxcS8RVIH6ELu8rWe/QBA+DFcyZz4xE55pbfVt3FCEFrAxUn5qkyzszwPMjtK4XAnn9kmnw2B0k6XLZOZd+n7W8ege6wih+gwfDhYMZX0qzi3HdAK7nk5vEOFibCztLaogzfc6fdFRx8OIWJQXqx5I1641FU3UE6sVxku4IMDTq2GVyUWKdSN6y8gBiDPVe3xu4qM4xghSnaAmfB45IrcNvibBjikNva2FHt8lcutQ== prateek@prateek-Dell", "key_options": null, "keyfile": "/home/prateek/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prateek", "validate_certs": true} 2021-02-10T10:54:27.598 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC9sZe9p48s6RSpluK1R97X9tsN3tWiidPbdyXvyMUXifYQb3nth9wMWUakrVBp+QGEkqALLIf3so3MfgbRvYexzNfHvRT7Lxn8D/isiCpWa4aLM/0alprV+RI38dR0mI3IVb3aIEYmgp2IY4yzybWTHCmS1VUrUJJ0pOfXOssq0xgo3qeaSQBHYRKasqR8GWAD5MWpG6/kg6gxI7o0aHVVTXy5ADB7NS1LEZr06xxNm4exeYvl9TzXrbGZWf2gC8ZpXhmFHI80A+TKEZRU6wmcju7G69nyEEF8/ebcBKmZB4eTsDN7qR7lc9dxKxiiZcJXc/MWy4bSu69NhOeuq2xx7bdexS8OHKdgYH9gsJolu5zgN20IHzaRgswO3XIe3nh2Ud8wntwBCwmJ9bf5YByITGjpOhntkvWAP3AXkQ1B4HXCdlUNW+4pMexDe2D05LHOwxcS8RVIH6ELu8rWe/QBA+DFcyZz4xE55pbfVt3FCEFrAxUn5qkyzszwPMjtK4XAnn9kmnw2B0k6XLZOZd+n7W8ege6wih+gwfDhYMZX0qzi3HdAK7nk5vEOFibCztLaogzfc6fdFRx8OIWJQXqx5I1641FU3UE6sVxku4IMDTq2GVyUWKdSN6y8gBiDPVe3xu4qM4xghSnaAmfB45IrcNvibBjikNva2FHt8lcutQ== prateek@prateek-Dell", "key_options": null, "keyfile": "/home/prateek/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prateek", "validate_certs": true} 2021-02-10T10:54:27.985 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFmmTp8+YDXU27xnSPxhWwouslpkvp1eULVp63taoXceDOnH8EMWv0wiEjN/P+Fcf/UB+iU8bELyHubZiqnOUr9f2pGX4jkJ6Yty1p5BUK4my7Si5J0/JawEDVQm4HpFWMKUVKYumvz3UFuBoINHVTqr1cLIDAt7mdeKfJqFB45AqxDUIPP4d8iznNCw+QO2CMxk0o/wMNHbygZZrHAiDngXQxwhe6oR17RfolhREy0JjHgojRjxgagpG7O8WW3OPI5Outnz5Pvraas5q7w+b3xbB9vdTObW/Nii5VOpTdIsFzBvp46hLoRSIcB8pRLUrxPJBPOY7G0zbopDRLB2zLNGKIH2zxEHJOeY928uUbWT22b9mGu/L95FFY7hmeRAicCUeixMTgXgczeQlHDn5eqQzLI3PNP7JX/0dauCD27GRKsDUnpIDj9RqgCQDVueIQQMC6QMa9dYQmps0r9Ljp8R84WjYFRP9oOI+POOzbgmRO0JCVnPqz9MKyGVby9/WFBHcvG6xmwACez00oqe62z+rhcavGXViX5CxGMV6vUTXugBa4ZkXlzs+djzYWfZ33ZKVeRIwj2Pu1PRTGzB8e2BaKqJMN0FVtzpV4hom4Y/9vIPyDpjRjNuKzrkYW1M8PxBxn+EJUPMLhr3jFMCo6HSxHoFXy8BzZoFkZWqap3w== JCL@IT-MINI-JCL.local", "key_options": null, "keyfile": "/home/jelopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jelopez", "validate_certs": true} 2021-02-10T10:54:28.052 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFmmTp8+YDXU27xnSPxhWwouslpkvp1eULVp63taoXceDOnH8EMWv0wiEjN/P+Fcf/UB+iU8bELyHubZiqnOUr9f2pGX4jkJ6Yty1p5BUK4my7Si5J0/JawEDVQm4HpFWMKUVKYumvz3UFuBoINHVTqr1cLIDAt7mdeKfJqFB45AqxDUIPP4d8iznNCw+QO2CMxk0o/wMNHbygZZrHAiDngXQxwhe6oR17RfolhREy0JjHgojRjxgagpG7O8WW3OPI5Outnz5Pvraas5q7w+b3xbB9vdTObW/Nii5VOpTdIsFzBvp46hLoRSIcB8pRLUrxPJBPOY7G0zbopDRLB2zLNGKIH2zxEHJOeY928uUbWT22b9mGu/L95FFY7hmeRAicCUeixMTgXgczeQlHDn5eqQzLI3PNP7JX/0dauCD27GRKsDUnpIDj9RqgCQDVueIQQMC6QMa9dYQmps0r9Ljp8R84WjYFRP9oOI+POOzbgmRO0JCVnPqz9MKyGVby9/WFBHcvG6xmwACez00oqe62z+rhcavGXViX5CxGMV6vUTXugBa4ZkXlzs+djzYWfZ33ZKVeRIwj2Pu1PRTGzB8e2BaKqJMN0FVtzpV4hom4Y/9vIPyDpjRjNuKzrkYW1M8PxBxn+EJUPMLhr3jFMCo6HSxHoFXy8BzZoFkZWqap3w== JCL@IT-MINI-JCL.local", "key_options": null, "keyfile": "/home/jelopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jelopez", "validate_certs": true} 2021-02-10T10:54:28.420 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdbJDYY43Zu0EKAGgrSKEobyXIxdhT0AeTT3yBoNY+YdYDScimJsQzK4hKNJIn1xC3klr0dSnsqeJru+SA4GMhbdcsmVyudhutCw75R/vFoRHOwaeVS2SMiBAtDbAYGQ3dh6LapkmXeLURcycNbebUV4HM9XsYqnuL8J2XyY4tRKgHrAd6OQw4jsXDOxwTRlHM822N/2kgv8iEwiXAJoUoymbr8mnfSxuuaH0oiKBwuaK1wuaDrlJyXkXNslPq89/1f8hJfjixY2dQdW7Xk5m3t5ptQxWENpw79OMRJ4cDW4G7LrtFONom5KQ+9TBZJIhNgJobkqI4qwinrWF8qMW5 dcain@fullsteam", "key_options": null, "keyfile": "/home/dcain/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dcain", "validate_certs": true} 2021-02-10T10:54:28.494 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdbJDYY43Zu0EKAGgrSKEobyXIxdhT0AeTT3yBoNY+YdYDScimJsQzK4hKNJIn1xC3klr0dSnsqeJru+SA4GMhbdcsmVyudhutCw75R/vFoRHOwaeVS2SMiBAtDbAYGQ3dh6LapkmXeLURcycNbebUV4HM9XsYqnuL8J2XyY4tRKgHrAd6OQw4jsXDOxwTRlHM822N/2kgv8iEwiXAJoUoymbr8mnfSxuuaH0oiKBwuaK1wuaDrlJyXkXNslPq89/1f8hJfjixY2dQdW7Xk5m3t5ptQxWENpw79OMRJ4cDW4G7LrtFONom5KQ+9TBZJIhNgJobkqI4qwinrWF8qMW5 dcain@fullsteam", "key_options": null, "keyfile": "/home/dcain/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dcain", "validate_certs": true} 2021-02-10T10:54:28.870 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDs2+6XwRPdiDHT+63Yab00VBulg96/OLEORtNUvm7TDRMpcx4OD2oOnbsE4rG3dOdXNgT17H15c6MWFX2a2uC2h1j4hJTa+dG567HrYXRHG60Kwg5Sviiuq/ETPoH/r3OSvHs0HJ4Y9qYPQ5j0k7gG1XgIY3QHDWNRsXiyRXP9FEfoaOKABDTUoLPjqRX+4ExJd1Nz6DLxqIlre8wfqF5L1l12vec8JU3HB5NSRa87HxKFtLLIAtw7B9H/HggqX5wWVEnXq2qN4o6OOePhaO5cjWAZxOtT+AJTyHkeyaWLwIGOHBVZS5BKti6jLc+PxQxAflYlu9X3xh/2FlA+9H+ABZwMHE3ruUfTVbviYBbtccMCvkzcvFgnvaUCHjd7Iv/T8/JHNfBgFgigcX0Vei0+PZQ5oC5b7YpKT241NA8DCGZNgr1KuL9LazkvbkbpDcaHUOrkWsmSW1RRMC6ffYkbAI9QiBJFpImDAWP1u+1yCGhzPKBfGNt91GcLCD6ML90= sunny@devBox", "key_options": null, "keyfile": "/home/sunny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sunny", "validate_certs": true} 2021-02-10T10:54:28.933 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDs2+6XwRPdiDHT+63Yab00VBulg96/OLEORtNUvm7TDRMpcx4OD2oOnbsE4rG3dOdXNgT17H15c6MWFX2a2uC2h1j4hJTa+dG567HrYXRHG60Kwg5Sviiuq/ETPoH/r3OSvHs0HJ4Y9qYPQ5j0k7gG1XgIY3QHDWNRsXiyRXP9FEfoaOKABDTUoLPjqRX+4ExJd1Nz6DLxqIlre8wfqF5L1l12vec8JU3HB5NSRa87HxKFtLLIAtw7B9H/HggqX5wWVEnXq2qN4o6OOePhaO5cjWAZxOtT+AJTyHkeyaWLwIGOHBVZS5BKti6jLc+PxQxAflYlu9X3xh/2FlA+9H+ABZwMHE3ruUfTVbviYBbtccMCvkzcvFgnvaUCHjd7Iv/T8/JHNfBgFgigcX0Vei0+PZQ5oC5b7YpKT241NA8DCGZNgr1KuL9LazkvbkbpDcaHUOrkWsmSW1RRMC6ffYkbAI9QiBJFpImDAWP1u+1yCGhzPKBfGNt91GcLCD6ML90= sunny@devBox", "key_options": null, "keyfile": "/home/sunny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sunny", "validate_certs": true} 2021-02-10T10:54:29.319 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC4pIcNiZKVIfT9GxyTe/B8G2GyHKxrAle3iExEzLH5bBCKtBrottX1y66KBzQSsrAb0z9h+A+uxlswlY6VM/gNos9VRB6x4NRB4ADY1Bku+JsKs5aXwRB1zz2ueZhrFN9hJwMl1heLICx85ksHBdm0q+JPdUAV+/LBdr0H/IjWveV34WL3Awf7QikzyKhzUciMB2gN9PJXTW0bl8PP56y60ys++L3nMSRJ+15uPcWl8MS3D2vABII0Tu+3PsxzrTTNodElyQ6G/fwyFfWhdp7fkidjR7mDDx8NyWfCl2rKejXv9LnVC0wb3tFoQqvwdz6shN4E51EIEGXHcLUeRpBhL2pFOqIZ0qfnmLIQVZ88RKMTgo42CgtmS5UzxCLOyKOh9GxF7DDEajTinJFJJVbNJ5xF494H2geSnmP1o1AecGsGbCZ4ndhS9unSfIZLAyv1J3wZlAlMsvxXgY7yrh1srUiASVyKK8jba1IT+v3TXR8kQUCNsLKKbvuX/l0IXME= xuxuehan@bogon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBd4eSizc2zAkWDy3mFZUhELO2wXDgkVxDNM/Nw2zT1li3L+6YOjFfdT4NOxpkg5laLd54bdKb5GVRA6iKqAO+aqo53xHHJGC8OY5eJaUPjtpypgIWO4Qzbos2ZzsC+nX5jvi8O9nM+JkabSW5iFv3gGPTqHixFYkxK2NIBa/UtXFOhmb1wZ6CG5Q+WObCosBUEMWokC8ntCADfhi6qDsO2fol6mYUhYEE/0nhQ5uLRrVIM5QM5dhrQVEeAs4KcYpmPDw6q6AKgx/f5zObh5OSJFTVsgx2cqzdQVJVDDx2OD2QVRtZ9fgvI2EGmKSwe1TUQOdqa5bmsVIp2Jfi8MC1 xuxuehan@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwIokXSvPYP3TRlSw0SBgmMFW5jVltGf4c6CqjIW4KKCgsL4c4Vs0FztRyDGc9p0krZUoGJCZHErVToXm6/lSq8JlfrlLX4X4yh37fUifWqc2UTJGeH8kLaP3WEt9vT8eQ9h8t2GqlMlGQU8KzVAdC+c5IwrbGRsS6S7D09/y0rklkhdJ6nuj8cssN/2TzLF9YIvVcFZHdyznk84eB7TgpPKD88MOpTnEgWFlD4vAWEuHpkuxgj1KI9LscgSFcW9/RxLeRxcV0GypZ8jPmczoCnin/ZOstCTpbcmjLd 2021-02-10T10:54:29.320 INFO:teuthology.task.ansible.out:+LW1kgGXXQSkf09iHeJ/cyCw6ooZOkynVwwmI9Uz2ZaMedCUBpG0bKclIerJ9jn+CoTs3U3Orhd9+6MVtCHX9VNJbVa4q0KCS1sp5S3HDrYkgUyawUxdYorrLZcqfRHWWo1luiridmuQdr8lQ6dvNpz4rhsCrAG6S1eeaC/s+o0Rzpv1YxU64JzquZ6WNR+W3U6Bg3iYkSoDCK9byf0rsptyL8= luqixi@luqixideMacBook-Air.local", "key_options": null, "keyfile": "/home/xuxuehan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xuxuehan", "validate_certs": true} 2021-02-10T10:54:29.381 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC4pIcNiZKVIfT9GxyTe/B8G2GyHKxrAle3iExEzLH5bBCKtBrottX1y66KBzQSsrAb0z9h+A+uxlswlY6VM/gNos9VRB6x4NRB4ADY1Bku+JsKs5aXwRB1zz2ueZhrFN9hJwMl1heLICx85ksHBdm0q+JPdUAV+/LBdr0H/IjWveV34WL3Awf7QikzyKhzUciMB2gN9PJXTW0bl8PP56y60ys++L3nMSRJ+15uPcWl8MS3D2vABII0Tu+3PsxzrTTNodElyQ6G/fwyFfWhdp7fkidjR7mDDx8NyWfCl2rKejXv9LnVC0wb3tFoQqvwdz6shN4E51EIEGXHcLUeRpBhL2pFOqIZ0qfnmLIQVZ88RKMTgo42CgtmS5UzxCLOyKOh9GxF7DDEajTinJFJJVbNJ5xF494H2geSnmP1o1AecGsGbCZ4ndhS9unSfIZLAyv1J3wZlAlMsvxXgY7yrh1srUiASVyKK8jba1IT+v3TXR8kQUCNsLKKbvuX/l0IXME= xuxuehan@bogon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBd4eSizc2zAkWDy3mFZUhELO2wXDgkVxDNM/Nw2zT1li3L+6YOjFfdT4NOxpkg5laLd54bdKb5GVRA6iKqAO+aqo53xHHJGC8OY5eJaUPjtpypgIWO4Qzbos2ZzsC+nX5jvi8O9nM+JkabSW5iFv3gGPTqHixFYkxK2NIBa/UtXFOhmb1wZ6CG5Q+WObCosBUEMWokC8ntCADfhi6qDsO2fol6mYUhYEE/0nhQ5uLRrVIM5QM5dhrQVEeAs4KcYpmPDw6q6AKgx/f5zObh5OSJFTVsgx2cqzdQVJVDDx2OD2QVRtZ9fgvI2EGmKSwe1TUQOdqa5bmsVIp2Jfi8MC1 xuxuehan@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwIokXSvPYP3TRlSw0SBgmMFW5jVltGf4c6CqjIW4KKCgsL4c4Vs0FztRyDGc9p0krZUoGJCZHErVToXm6/lSq8JlfrlLX4X4yh37fUifWqc2UTJGeH8kLaP3WEt9vT8eQ9h8t2GqlMlGQU8KzVAdC+c5IwrbGRsS6S7D09/y0rklkhdJ6nuj8cssN/2TzLF9YIvVcFZHdyznk84eB7TgpPKD88MOpTnEgWFlD4vAWEuHpkuxgj1KI9LscgSFcW9/RxLeRxcV0GypZ8jPmczoCnin/ZOstCTpbcmjLd 2021-02-10T10:54:29.383 INFO:teuthology.task.ansible.out:+LW1kgGXXQSkf09iHeJ/cyCw6ooZOkynVwwmI9Uz2ZaMedCUBpG0bKclIerJ9jn+CoTs3U3Orhd9+6MVtCHX9VNJbVa4q0KCS1sp5S3HDrYkgUyawUxdYorrLZcqfRHWWo1luiridmuQdr8lQ6dvNpz4rhsCrAG6S1eeaC/s+o0Rzpv1YxU64JzquZ6WNR+W3U6Bg3iYkSoDCK9byf0rsptyL8= luqixi@luqixideMacBook-Air.local", "key_options": null, "keyfile": "/home/xuxuehan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xuxuehan", "validate_certs": true} 2021-02-10T10:54:29.773 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDMOuxxTZyxz/TEVA+ODgtLrNcSsUhKE64BZg/Ub96LLffQXQc9SYT/0YI4wxu2vSsky05PnVSON7jqGxunCt0+AdRqxXhW4ttmRIOz8rq8SiMnnLayV34nIF1G0aYTkgY4ej4+VmSVIHU3bb5mp/Q3D0hxQwDPdgOJWNyW9oYerHbnx6lVgchvsZHB9VO09GNHcAsyUz4p+xBf9iR7eS74p4dZmUbth/oBBqpOa4NgNr69iyZ9a/pNDLxaVQoIpDFrevk5UU5nUHQq2frFNxF/Y2Ja1EBXCavvPF/m5v3J3xNEu3SI/GbLFCPKCKeBDpXJbe1QqTBKJjMl9qGmKEJ3 aclewett@mac", "key_options": null, "keyfile": "/home/aclewett/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aclewett", "validate_certs": true} 2021-02-10T10:54:29.827 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDMOuxxTZyxz/TEVA+ODgtLrNcSsUhKE64BZg/Ub96LLffQXQc9SYT/0YI4wxu2vSsky05PnVSON7jqGxunCt0+AdRqxXhW4ttmRIOz8rq8SiMnnLayV34nIF1G0aYTkgY4ej4+VmSVIHU3bb5mp/Q3D0hxQwDPdgOJWNyW9oYerHbnx6lVgchvsZHB9VO09GNHcAsyUz4p+xBf9iR7eS74p4dZmUbth/oBBqpOa4NgNr69iyZ9a/pNDLxaVQoIpDFrevk5UU5nUHQq2frFNxF/Y2Ja1EBXCavvPF/m5v3J3xNEu3SI/GbLFCPKCKeBDpXJbe1QqTBKJjMl9qGmKEJ3 aclewett@mac", "key_options": null, "keyfile": "/home/aclewett/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aclewett", "validate_certs": true} 2021-02-10T10:54:30.215 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdTLDprr3DXtsgiqrB/miglJwSkY7BAv+OW9sC0ZHQf8Y4dFoUtr7q9fd6W7ehKL6aMjya/8eQSMPRoEIZFD8f2KPCuE74Wgwgv2cpAkFmLjcPtjixqLo3LGFCyEJWGLQxjlthdzx59mpCWgBsEJQmlKP36JIYjILIMtXtsETIUGFa8QPIn9sMdIgutPPf3yB/6DHcR7C6yIAuvbJactEsiCt4C8+uJinvCii3PzYOMAEd+mictPGk8nOCQFZrI+ZmwTz4UbJylxFqnX8lhhoPIUEfNAXlb3lkZWH1LaFOxarbmupuPYcg2rT0+kFtyNr3/mAx0FO01jADYooqcW1Z cardno:000604936806", "key_options": null, "keyfile": "/home/cblum/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cblum", "validate_certs": true} 2021-02-10T10:54:30.268 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdTLDprr3DXtsgiqrB/miglJwSkY7BAv+OW9sC0ZHQf8Y4dFoUtr7q9fd6W7ehKL6aMjya/8eQSMPRoEIZFD8f2KPCuE74Wgwgv2cpAkFmLjcPtjixqLo3LGFCyEJWGLQxjlthdzx59mpCWgBsEJQmlKP36JIYjILIMtXtsETIUGFa8QPIn9sMdIgutPPf3yB/6DHcR7C6yIAuvbJactEsiCt4C8+uJinvCii3PzYOMAEd+mictPGk8nOCQFZrI+ZmwTz4UbJylxFqnX8lhhoPIUEfNAXlb3lkZWH1LaFOxarbmupuPYcg2rT0+kFtyNr3/mAx0FO01jADYooqcW1Z cardno:000604936806", "key_options": null, "keyfile": "/home/cblum/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cblum", "validate_certs": true} 2021-02-10T10:54:30.676 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDPUCIVfNQOOG+uAeKWo7tLAdgGO/oQu4exTE3XLfyTVdd8u6pbQCmIDFEnbGgMtlYuWJlCJtxJIzEZJ5HIwf+po3y7kb7HxEeWnEb9DsqJB4nCRE8fHnNLgF3bh9gvE7sOZdNXEKqtsnGcaaS81oXQ6QgkFGQzlOBOqxj7jIvad7AtNWD3WbJi/uTVDsWcukU51tn/aEW+hHMD7Y5QJkLH7O0PrCeEZCUCWoiMP7sNwACBym0DsqX+POkGJOaGx9EeNhGVA4B6o3xYXJPqmFbgMWMfG3/Y3SiheZjxzySMQ7FKW4Q817NaWqbjtLZQsSZw7JNqNFkcdTzVZwwCOJ1AAecajpniKeQLnyiWQ3Z/EkSGGwtDj67wo7uqT85tpOyLfYhQm6Zp+xSh+I6s0YJ0xO/huLKzeT5D1r5vVMk2VdcLD3CCxXUwZgaMUxYAC2Wq6mpIkxVZFQLbGuA/dWkQbFsrpbf7ES+kh+uzp4R6dQ29OjXbVcSfF2vyIuA+VUKoyj370YOZQhs50yuAJ4BSHlEgoJdRzi3XUX48h45XHiZcCY3X8LTX6a+0KAIgqeAWdFhRgQtAxoRYjhOxC3k/boBc8sEQ3o4pkxnhtE5Y/tnM2CPkPMX9HWYirNf9GyZTYcR/5rDf4ETQ18G68WLaNK758OuUPjSkwttcyLd4dw== mlatimer@suse.com", "key_options": null, "keyfile": "/home/mlatimer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mlatimer", "validate_certs": true} 2021-02-10T10:54:30.719 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDPUCIVfNQOOG+uAeKWo7tLAdgGO/oQu4exTE3XLfyTVdd8u6pbQCmIDFEnbGgMtlYuWJlCJtxJIzEZJ5HIwf+po3y7kb7HxEeWnEb9DsqJB4nCRE8fHnNLgF3bh9gvE7sOZdNXEKqtsnGcaaS81oXQ6QgkFGQzlOBOqxj7jIvad7AtNWD3WbJi/uTVDsWcukU51tn/aEW+hHMD7Y5QJkLH7O0PrCeEZCUCWoiMP7sNwACBym0DsqX+POkGJOaGx9EeNhGVA4B6o3xYXJPqmFbgMWMfG3/Y3SiheZjxzySMQ7FKW4Q817NaWqbjtLZQsSZw7JNqNFkcdTzVZwwCOJ1AAecajpniKeQLnyiWQ3Z/EkSGGwtDj67wo7uqT85tpOyLfYhQm6Zp+xSh+I6s0YJ0xO/huLKzeT5D1r5vVMk2VdcLD3CCxXUwZgaMUxYAC2Wq6mpIkxVZFQLbGuA/dWkQbFsrpbf7ES+kh+uzp4R6dQ29OjXbVcSfF2vyIuA+VUKoyj370YOZQhs50yuAJ4BSHlEgoJdRzi3XUX48h45XHiZcCY3X8LTX6a+0KAIgqeAWdFhRgQtAxoRYjhOxC3k/boBc8sEQ3o4pkxnhtE5Y/tnM2CPkPMX9HWYirNf9GyZTYcR/5rDf4ETQ18G68WLaNK758OuUPjSkwttcyLd4dw== mlatimer@suse.com", "key_options": null, "keyfile": "/home/mlatimer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mlatimer", "validate_certs": true} 2021-02-10T10:54:31.144 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCYh28a9R3a7jpRfPqVAL9EcD+S6KkZzKTfCmRn4s8Fg39kUZsgnWUZQ3jUxKSeBA00te7WMR8CsIEzulm7f+/egNuPcvzNI/SaaCDtVMaxqlEuW85ll8uBQi3s4SLmOu9oS+5oid4cqTrGTcG6uhl29oWS/53SxoGh+T+y6GesO2cYRuTpcUHxGMvIgZe/bKtVGdP1y7rQd62e8emb9UupMBxaHpkYj9AaMuLjovHbBgecWbe4PlzPWNzl+eITVk2d6CxxIem4/bw9VHkRjQZlhL2nbwRgtczLYhlaZAd4ePEtsqA+7nTDjTDhXXHVIf1pDAsJbZEdgQCSGH3FzxtrQ2hZJrH/NIKdHk/OtI06e7POSGj7PcHzTvMjQSFnTpJXinxwzn0lvFSlmABN/00lIE0p2ZXfPs6dgWJ17JnTq6dMUoTiGhfKphPHyfSXJCzfa2VB+lQQ5eAaa9cRXy9pLmWGHXVGz2hsBMtyYmTjR/wr8kdTzeK+Jk4opBtGgcxDustg59b9x/jvgoc3HCdtnkqLTug2KmutB6kb+4KnVO07GsSp0RmNtDmCRcXs9YN/eia9ICtIGDWPc3YDRppsw1Vktza/+ZW5vGzIKILZTo4SNGcg19zQw4Qxd9eLkXfB66vE2CQvA90CGo7mcye+UWjjXLr7i0cDEszAloQ9aQ== srangana@redhat.com", "key_options": null, "keyfile": "/home/shyamsundarr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyamsundarr", "validate_certs": true} 2021-02-10T10:54:31.207 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCYh28a9R3a7jpRfPqVAL9EcD+S6KkZzKTfCmRn4s8Fg39kUZsgnWUZQ3jUxKSeBA00te7WMR8CsIEzulm7f+/egNuPcvzNI/SaaCDtVMaxqlEuW85ll8uBQi3s4SLmOu9oS+5oid4cqTrGTcG6uhl29oWS/53SxoGh+T+y6GesO2cYRuTpcUHxGMvIgZe/bKtVGdP1y7rQd62e8emb9UupMBxaHpkYj9AaMuLjovHbBgecWbe4PlzPWNzl+eITVk2d6CxxIem4/bw9VHkRjQZlhL2nbwRgtczLYhlaZAd4ePEtsqA+7nTDjTDhXXHVIf1pDAsJbZEdgQCSGH3FzxtrQ2hZJrH/NIKdHk/OtI06e7POSGj7PcHzTvMjQSFnTpJXinxwzn0lvFSlmABN/00lIE0p2ZXfPs6dgWJ17JnTq6dMUoTiGhfKphPHyfSXJCzfa2VB+lQQ5eAaa9cRXy9pLmWGHXVGz2hsBMtyYmTjR/wr8kdTzeK+Jk4opBtGgcxDustg59b9x/jvgoc3HCdtnkqLTug2KmutB6kb+4KnVO07GsSp0RmNtDmCRcXs9YN/eia9ICtIGDWPc3YDRppsw1Vktza/+ZW5vGzIKILZTo4SNGcg19zQw4Qxd9eLkXfB66vE2CQvA90CGo7mcye+UWjjXLr7i0cDEszAloQ9aQ== srangana@redhat.com", "key_options": null, "keyfile": "/home/shyamsundarr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyamsundarr", "validate_certs": true} 2021-02-10T10:54:31.597 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRZyOnj6D9tCw5o4+ujpos5Uj1O0XVvea798DsXPRh8bEE1pLkARQJLv4iW6LsG3x/CXruahmwT+6C/VHppIsbgtpadMWdCTPYrsjctPK0/N9nr4kQEL6XeK3/GHFnwqYcc+LySpNNfbKrgN4qWOwdZBhQjSV7K7FJd4ZYSm4wp6TVbvKKp7o+JdeM2zhstfC9AcCtBv/TpxixFFq/+y8/Z0PwqN1YyHduDOr1dr1DWn4ZzrkhiTYfPOltx0jweD6zTJoTAJDqae7kBA+3XH3dUGX3n7Tchsa3bnIKVz9sLQvH/vPro61I+Xig6MQh77JsylNvpwb1LLOEzDf7dxl bigwhite@wrjj37deMacBook-Pro-2.local", "key_options": null, "keyfile": "/home/shuzhenyi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shuzhenyi", "validate_certs": true} 2021-02-10T10:54:31.598 INFO:teuthology.task.ansible.out: 2021-02-10T10:54:31.644 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRZyOnj6D9tCw5o4+ujpos5Uj1O0XVvea798DsXPRh8bEE1pLkARQJLv4iW6LsG3x/CXruahmwT+6C/VHppIsbgtpadMWdCTPYrsjctPK0/N9nr4kQEL6XeK3/GHFnwqYcc+LySpNNfbKrgN4qWOwdZBhQjSV7K7FJd4ZYSm4wp6TVbvKKp7o+JdeM2zhstfC9AcCtBv/TpxixFFq/+y8/Z0PwqN1YyHduDOr1dr1DWn4ZzrkhiTYfPOltx0jweD6zTJoTAJDqae7kBA+3XH3dUGX3n7Tchsa3bnIKVz9sLQvH/vPro61I+Xig6MQh77JsylNvpwb1LLOEzDf7dxl bigwhite@wrjj37deMacBook-Pro-2.local", "key_options": null, "keyfile": "/home/shuzhenyi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shuzhenyi", "validate_certs": true} 2021-02-10T10:54:32.037 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDEcI3X9mNxtHRSvi2gVFavu21kRCWDd1erXW3d/1zWnoUvRxe9ft2xA1c3Bmo067GqhFudPcXoYJvz3atAGR9k3n1UTtTJe95XRZ1Zmn/boi6mn7I2oRVFpW/KjxaiM3iN2SHKknMtcs6qRTioapYCU7u924oc0izqPBvkrC73cfVF4emH1ERWEcQBTxWub6N6Mn3xaDmJ5xnGlNvDY70CC9osoa2VRNJM7GU9eUQClHirPbSooDvvVKfTGD7YfMH76j6PdDR/a/cgNeiMevtMbeFGhu1SOEHb1ihCNoGq2vQFPPWwwTCzMI05XuUEc4iaxw/kbGBMpWbmIvswIMQrRwgwLZk67GGpf/dEeutI6WseeNKNuHnTzzx43AW9G2coGq0RKE7OuA5Cqj0CO7Kcf5UA4AqIIroLETVBrrEV6qzpkD74QBZXPl7gbSZIyxLpvtQoMfSSzN+Mj184LmQoURCqIT+yM5gddF5/S5I+rpw81vBj5M9ZskyzVQAo0H8= michelledipalma@mdipalma-mac", "key_options": null, "keyfile": "/home/mdipalma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mdipalma", "validate_certs": true} 2021-02-10T10:54:32.093 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDEcI3X9mNxtHRSvi2gVFavu21kRCWDd1erXW3d/1zWnoUvRxe9ft2xA1c3Bmo067GqhFudPcXoYJvz3atAGR9k3n1UTtTJe95XRZ1Zmn/boi6mn7I2oRVFpW/KjxaiM3iN2SHKknMtcs6qRTioapYCU7u924oc0izqPBvkrC73cfVF4emH1ERWEcQBTxWub6N6Mn3xaDmJ5xnGlNvDY70CC9osoa2VRNJM7GU9eUQClHirPbSooDvvVKfTGD7YfMH76j6PdDR/a/cgNeiMevtMbeFGhu1SOEHb1ihCNoGq2vQFPPWwwTCzMI05XuUEc4iaxw/kbGBMpWbmIvswIMQrRwgwLZk67GGpf/dEeutI6WseeNKNuHnTzzx43AW9G2coGq0RKE7OuA5Cqj0CO7Kcf5UA4AqIIroLETVBrrEV6qzpkD74QBZXPl7gbSZIyxLpvtQoMfSSzN+Mj184LmQoURCqIT+yM5gddF5/S5I+rpw81vBj5M9ZskyzVQAo0H8= michelledipalma@mdipalma-mac", "key_options": null, "keyfile": "/home/mdipalma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mdipalma", "validate_certs": true} 2021-02-10T10:54:32.491 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+tqgRGdlbb7NMefZVPCtbzVZBapxmmPcKaUtqT7E/9AKRoHuroZQ7KJQfO15POpIfEja7BugBt5F95aW2Bd9/xhulzC4/JjiZYc6ST5Na03BrcudI3YcktoGR8Mm7ZVzSojvm0SKtLLHM5Su17JpCC7BitMdao1mETlUOklEXc6pgbr9XHOayo1zeuos8DB90iA8TQ2soR/POD1Rix4GKniCHXz10+iXQFYPsif1HsQ3rh8Vi1M0cTq98bMSmfsKODil8KIxf+lvLCL4GAgDYEyxnstiTFh5GAcIMOOo2dl1fDb/Oa981CxWU+xxJD+M7AEJVeyAqqAnShvZU6y1L kiefer@kiefer-uchi", "key_options": null, "keyfile": "/home/kchang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchang", "validate_certs": true} 2021-02-10T10:54:32.545 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+tqgRGdlbb7NMefZVPCtbzVZBapxmmPcKaUtqT7E/9AKRoHuroZQ7KJQfO15POpIfEja7BugBt5F95aW2Bd9/xhulzC4/JjiZYc6ST5Na03BrcudI3YcktoGR8Mm7ZVzSojvm0SKtLLHM5Su17JpCC7BitMdao1mETlUOklEXc6pgbr9XHOayo1zeuos8DB90iA8TQ2soR/POD1Rix4GKniCHXz10+iXQFYPsif1HsQ3rh8Vi1M0cTq98bMSmfsKODil8KIxf+lvLCL4GAgDYEyxnstiTFh5GAcIMOOo2dl1fDb/Oa981CxWU+xxJD+M7AEJVeyAqqAnShvZU6y1L kiefer@kiefer-uchi", "key_options": null, "keyfile": "/home/kchang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchang", "validate_certs": true} 2021-02-10T10:54:32.935 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDjj3bbbLvd4IHeFrSJyM9N6uKXA7ElQrzfcisKRGjUPON8IsAWkXQeAnHoBsIQv7tRGudXOrK+tbfv7crLI0/CxL+o0cCVW8bxuTEtdz07ZFsND9cppCc7Yq+wHsg7nQ/n0QfxnlYc0T9pATXUoT+aW4c3NQoDbsupLSkNs1C+G/MEBzz6iH6tgrv43OAOUccfQbC+IYx4RIKL5MteEjVrZanNELIG+jsYe+CZLdsONk1nR8+xKQw19Sm/jkKdl1K9NXoVPEBQmFXrYl6G+7tCdWeU3Vq24lM1X5p1mVspcG6FyyctcuP/8+zhL2L208DrXlOVPcqHR/4FEf3Wmk89 vereddy@vereddy.remote.csb", "key_options": null, "keyfile": "/home/vereddy/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vereddy", "validate_certs": true} 2021-02-10T10:54:32.989 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDjj3bbbLvd4IHeFrSJyM9N6uKXA7ElQrzfcisKRGjUPON8IsAWkXQeAnHoBsIQv7tRGudXOrK+tbfv7crLI0/CxL+o0cCVW8bxuTEtdz07ZFsND9cppCc7Yq+wHsg7nQ/n0QfxnlYc0T9pATXUoT+aW4c3NQoDbsupLSkNs1C+G/MEBzz6iH6tgrv43OAOUccfQbC+IYx4RIKL5MteEjVrZanNELIG+jsYe+CZLdsONk1nR8+xKQw19Sm/jkKdl1K9NXoVPEBQmFXrYl6G+7tCdWeU3Vq24lM1X5p1mVspcG6FyyctcuP/8+zhL2L208DrXlOVPcqHR/4FEf3Wmk89 vereddy@vereddy.remote.csb", "key_options": null, "keyfile": "/home/vereddy/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vereddy", "validate_certs": true} 2021-02-10T10:54:33.376 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+EkUbnaeC7Dqtq2N4/dX9v+WGlrgRS2ygHhlbZy4KHbd6+Q9T5NAWtqnFwqcbFpYYUBwMvz8zLo6HYN6gNskRN0qTiFKPQdtnydF0Tx1KHrdGGcX3G2lxyGH5P8nrqReXAPPvRiVc2/Mj7APCTmY2JftkSS1qYcTleeueivVyho6z6uD1AyC7OXANhtn7kGeIY/NvVKBhDr+3TuPmpPwPCQMcJUyZWCHYyp47v2vzNBy2Etp7WjGwOnfZ5ULiJGjH6PCXx0YfpmTUOPVkNbV6Om+0YJ3fcLlmQ/2nsdsWfOKvoAr+1uzO81h7ZBX9LVEitj8YU3WeSCV2vqS2VvOpq5NyHXWGq+ZkOCfhy4Z9MvMG+6hqp4p4z1JMpKsZ191kP7eAQua41GkTSQPlEfF+um1UV6ZjNzhB68utOXyGDoDcxct9oOdnhhjwu5W4/0t9iBHuMn02iHRYVrUWOH2FXVysR0p5x/L2xdvNPiTgFAZ2eN+EFWmdan4e9mAAw3M= gbenhano@localhost.localdomain", "key_options": null, "keyfile": "/home/benhanokh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "benhanokh", "validate_certs": true} 2021-02-10T10:54:33.428 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+EkUbnaeC7Dqtq2N4/dX9v+WGlrgRS2ygHhlbZy4KHbd6+Q9T5NAWtqnFwqcbFpYYUBwMvz8zLo6HYN6gNskRN0qTiFKPQdtnydF0Tx1KHrdGGcX3G2lxyGH5P8nrqReXAPPvRiVc2/Mj7APCTmY2JftkSS1qYcTleeueivVyho6z6uD1AyC7OXANhtn7kGeIY/NvVKBhDr+3TuPmpPwPCQMcJUyZWCHYyp47v2vzNBy2Etp7WjGwOnfZ5ULiJGjH6PCXx0YfpmTUOPVkNbV6Om+0YJ3fcLlmQ/2nsdsWfOKvoAr+1uzO81h7ZBX9LVEitj8YU3WeSCV2vqS2VvOpq5NyHXWGq+ZkOCfhy4Z9MvMG+6hqp4p4z1JMpKsZ191kP7eAQua41GkTSQPlEfF+um1UV6ZjNzhB68utOXyGDoDcxct9oOdnhhjwu5W4/0t9iBHuMn02iHRYVrUWOH2FXVysR0p5x/L2xdvNPiTgFAZ2eN+EFWmdan4e9mAAw3M= gbenhano@localhost.localdomain", "key_options": null, "keyfile": "/home/benhanokh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "benhanokh", "validate_certs": true} 2021-02-10T10:54:33.817 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBYtdHKBXQcXQFkEW8fSwlFK/xZM16VcFY288ABI/DPfejTStHhxiRvmlsWzMy0Ar5WadZ8FU1ptjA3MKBEY1Do1r1Ul369k3zHCvxXyg9QGGMNPYfHmnNeoF4GEgoJpuounPs0mkLmrPTR+cR+/XuaZZt38nsad8M/zJetnolvqCWpcV87D7+Rt8F32J6kGDknC4ikEe8I0n7F+goOmJhlgkqegbKEeJ/EMnOItwVZHFwaC+sxOq4IlYp2f1aZMKGAtr5+ICwYH1SMCcvnp3qWL8O//t2tvguZgQxzIs6Nf5GaxwdQkh4CvjbWiCkIbGFJ57YICS6k++KMXMbEnwh jsalomon@Josh-laptop.tlv.redhat.com", "key_options": null, "keyfile": "/home/jsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jsalomon", "validate_certs": true} 2021-02-10T10:54:33.870 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBYtdHKBXQcXQFkEW8fSwlFK/xZM16VcFY288ABI/DPfejTStHhxiRvmlsWzMy0Ar5WadZ8FU1ptjA3MKBEY1Do1r1Ul369k3zHCvxXyg9QGGMNPYfHmnNeoF4GEgoJpuounPs0mkLmrPTR+cR+/XuaZZt38nsad8M/zJetnolvqCWpcV87D7+Rt8F32J6kGDknC4ikEe8I0n7F+goOmJhlgkqegbKEeJ/EMnOItwVZHFwaC+sxOq4IlYp2f1aZMKGAtr5+ICwYH1SMCcvnp3qWL8O//t2tvguZgQxzIs6Nf5GaxwdQkh4CvjbWiCkIbGFJ57YICS6k++KMXMbEnwh jsalomon@Josh-laptop.tlv.redhat.com", "key_options": null, "keyfile": "/home/jsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jsalomon", "validate_certs": true} 2021-02-10T10:54:34.259 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9ltUNitnn0FNwMeiGkBnozpBN5ziVR2MreaqkfqTQY2TgI24559xuILyy+TnAKumhemZ8i8nIQ876Iso85OzIXbUbRM45hpL/bMQX7pWyMIRNeFKrCDwf18OT0DPc8aeDOAFQ6YoN0j4awUENjdZuuAW80cfcjoQO3m7ORF8wBF0IvMWwPDIQQrz+Li31ECgXg6VC7K3e+BNHlfNQ/2/ahS0+V4J6JYFsylZL0r2tydhDMd6rR8QA9c9BgeBz7+KWobGtiZVe7KReTzJ0O38PxNd8br3aS50j5BiN63gch8eFxPAROaUMnuiavu6OTDy4/aeL3Zc5WpUy3VyyumWf jolmomar@localhost.localdomain", "key_options": null, "keyfile": "/home/jolmomar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jolmomar", "validate_certs": true} 2021-02-10T10:54:34.314 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9ltUNitnn0FNwMeiGkBnozpBN5ziVR2MreaqkfqTQY2TgI24559xuILyy+TnAKumhemZ8i8nIQ876Iso85OzIXbUbRM45hpL/bMQX7pWyMIRNeFKrCDwf18OT0DPc8aeDOAFQ6YoN0j4awUENjdZuuAW80cfcjoQO3m7ORF8wBF0IvMWwPDIQQrz+Li31ECgXg6VC7K3e+BNHlfNQ/2/ahS0+V4J6JYFsylZL0r2tydhDMd6rR8QA9c9BgeBz7+KWobGtiZVe7KReTzJ0O38PxNd8br3aS50j5BiN63gch8eFxPAROaUMnuiavu6OTDy4/aeL3Zc5WpUy3VyyumWf jolmomar@localhost.localdomain", "key_options": null, "keyfile": "/home/jolmomar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jolmomar", "validate_certs": true} 2021-02-10T10:54:34.711 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQKiUAlB8dxudAY60M7jnIJvs9yb2pNaT+BVv1Vk1kMs0XjPjcw9cyT3DtQ1xGLF367phRi/cfQbXOYbHG+uyBzQ4EEnBFvziktWRx4mfeWYxKjkjbfkVfpKPsxztqCbwm6+NYzMxGK2cExN7jUHnghHAjfA0IUpVVg9peBD3kUveLwS+UxuJAnuj+G4480LS6GDyLrhFEXU7uOIIlW4S9oLnU6vJL6GAzwyLE4ijTA8Ek03SJ01Ey/gpU9AqnQNp9u12l/7sJely6sI9pAuIzmS8Ht324hexxSWN5sFjXgLvLfFK83ziDBdrFBr0WXnWsFdRpHs2CwSq1JTkByqbN hmunjulu@hmunjulu.remote.csb", "key_options": null, "keyfile": "/home/hmunjulu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hmunjulu", "validate_certs": true} 2021-02-10T10:54:34.753 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQKiUAlB8dxudAY60M7jnIJvs9yb2pNaT+BVv1Vk1kMs0XjPjcw9cyT3DtQ1xGLF367phRi/cfQbXOYbHG+uyBzQ4EEnBFvziktWRx4mfeWYxKjkjbfkVfpKPsxztqCbwm6+NYzMxGK2cExN7jUHnghHAjfA0IUpVVg9peBD3kUveLwS+UxuJAnuj+G4480LS6GDyLrhFEXU7uOIIlW4S9oLnU6vJL6GAzwyLE4ijTA8Ek03SJ01Ey/gpU9AqnQNp9u12l/7sJely6sI9pAuIzmS8Ht324hexxSWN5sFjXgLvLfFK83ziDBdrFBr0WXnWsFdRpHs2CwSq1JTkByqbN hmunjulu@hmunjulu.remote.csb", "key_options": null, "keyfile": "/home/hmunjulu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hmunjulu", "validate_certs": true} 2021-02-10T10:54:35.148 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoefxOfeArToGucfWG7y1tyLF2I1J/IkBSlmxxEJyHyDUB6ElXjYIJReoiPfmB0xLEmINpmt8QA4XN/8UbR1r9LFOexlZZbX1LSbihi11LXCvfIJWqkHVwIyKHRTu/duMrntBFVENJ0LVf0GSWVTtllXaDA+fMkp6/ra/hlRHQbx19UXy/9+Pogj8Xl+SntnMGiHlX8FFq2A2tUqhdQSf/1GW1HfUIa5WtwdEqIpKpAgMM7sOW9i3QAmLAMo8HiKrArSiy7jrRUODCqsTJsfifxEbmnOS0ZtmDh0eeWL/Dyqbz99Rg4WbFNeK8oyZYr7D6bEAhd2az2A+wycPx4Nq7 julpark@julpark.remote.csb", "key_options": null, "keyfile": "/home/julpark/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "julpark", "validate_certs": true} 2021-02-10T10:54:35.202 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoefxOfeArToGucfWG7y1tyLF2I1J/IkBSlmxxEJyHyDUB6ElXjYIJReoiPfmB0xLEmINpmt8QA4XN/8UbR1r9LFOexlZZbX1LSbihi11LXCvfIJWqkHVwIyKHRTu/duMrntBFVENJ0LVf0GSWVTtllXaDA+fMkp6/ra/hlRHQbx19UXy/9+Pogj8Xl+SntnMGiHlX8FFq2A2tUqhdQSf/1GW1HfUIa5WtwdEqIpKpAgMM7sOW9i3QAmLAMo8HiKrArSiy7jrRUODCqsTJsfifxEbmnOS0ZtmDh0eeWL/Dyqbz99Rg4WbFNeK8oyZYr7D6bEAhd2az2A+wycPx4Nq7 julpark@julpark.remote.csb", "key_options": null, "keyfile": "/home/julpark/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "julpark", "validate_certs": true} 2021-02-10T10:54:35.598 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQD03otezqmbR7v9dq62mWH5Kkg692jdmDZk6Ve4vux0A0YhxBuXDCyBWLE1Z4lLNxQahvK0pGsgokp+Uk+Q6aU+ZnF/E2jLp6YWerFXIHY3M7byObJx9+QuVBfNI0bZBkV3+vSjhSCGOYU+TrTHsKZqhsTcejZjmL3gFLmWfU6dTh0sX05qz97cdJl3CMPhCIR3KjtkiPygj417/N7wet/Efv1t79TvyS7dtK1oz+zRLEpKi/fZjvpXb8xec7as72Y0XexsWMDyZJ/N8PijTQ5L6EfLMV7ae/bWGzNAKsoC1dHVY7q/Q4pzktC6zU0Rc9sDRMtrgFg+q+04BfveSLfxTb9tqf7SbSvh7IAG1eKiEuDT/uy3T6Uiw+CiZ0eIqYzZscO/aDjUArGbS8eyaycybdYCBK9Jz38i0mYKkXg855yoVuIExr6MXieYwyVoVW8gbx1bAslaafrzra2dkAFByOJ0sbosPFRgR2+boUSxD6IaE+ALgno4T/0CI2RMiH0= prashant@cephdev", "key_options": null, "keyfile": "/home/pdhange/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdhange", "validate_certs": true} 2021-02-10T10:54:35.641 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQD03otezqmbR7v9dq62mWH5Kkg692jdmDZk6Ve4vux0A0YhxBuXDCyBWLE1Z4lLNxQahvK0pGsgokp+Uk+Q6aU+ZnF/E2jLp6YWerFXIHY3M7byObJx9+QuVBfNI0bZBkV3+vSjhSCGOYU+TrTHsKZqhsTcejZjmL3gFLmWfU6dTh0sX05qz97cdJl3CMPhCIR3KjtkiPygj417/N7wet/Efv1t79TvyS7dtK1oz+zRLEpKi/fZjvpXb8xec7as72Y0XexsWMDyZJ/N8PijTQ5L6EfLMV7ae/bWGzNAKsoC1dHVY7q/Q4pzktC6zU0Rc9sDRMtrgFg+q+04BfveSLfxTb9tqf7SbSvh7IAG1eKiEuDT/uy3T6Uiw+CiZ0eIqYzZscO/aDjUArGbS8eyaycybdYCBK9Jz38i0mYKkXg855yoVuIExr6MXieYwyVoVW8gbx1bAslaafrzra2dkAFByOJ0sbosPFRgR2+boUSxD6IaE+ALgno4T/0CI2RMiH0= prashant@cephdev", "key_options": null, "keyfile": "/home/pdhange/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdhange", "validate_certs": true} 2021-02-10T10:54:36.040 INFO:teuthology.task.ansible.out:changed: [gibba002.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDg7xW5+Mk7pG5MjLxOc7GTvDTCPBJj2L3MEQZIapmmo6t8uPODwDewXncSzrKDH6zFVE4TTTD9fJdu+BhyT+jCFgXrEOMMabj7vJsg0wO0SWqr+4NECFPhD9y4ewfYiewBeXisclA5Bto91v93AtYpZBCdmPYSi0BEcPVc6S3iBUD0PzIrL62I/kBAtm0f/bScRiT7XvQJXvkbEObW2yS2XhUv2qgq1VlFvO1jl3B85KMTHVb8+LrKpQWOKfRfxJ74c51y6tnEjZLFO6Xn0JevqOhd8gPQIXueFZOM0g0pNp1mbtGaaBlEIibbrzo2p9kzsdvJeHMq1XuMIi1/Qyj9 dpivonka@dhcp-41-144.bos.redhat.com", "key_options": null, "keyfile": "/home/dpivonka/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dpivonka", "validate_certs": true} 2021-02-10T10:54:36.082 INFO:teuthology.task.ansible.out:changed: [gibba013.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDg7xW5+Mk7pG5MjLxOc7GTvDTCPBJj2L3MEQZIapmmo6t8uPODwDewXncSzrKDH6zFVE4TTTD9fJdu+BhyT+jCFgXrEOMMabj7vJsg0wO0SWqr+4NECFPhD9y4ewfYiewBeXisclA5Bto91v93AtYpZBCdmPYSi0BEcPVc6S3iBUD0PzIrL62I/kBAtm0f/bScRiT7XvQJXvkbEObW2yS2XhUv2qgq1VlFvO1jl3B85KMTHVb8+LrKpQWOKfRfxJ74c51y6tnEjZLFO6Xn0JevqOhd8gPQIXueFZOM0g0pNp1mbtGaaBlEIibbrzo2p9kzsdvJeHMq1XuMIi1/Qyj9 dpivonka@dhcp-41-144.bos.redhat.com", "key_options": null, "keyfile": "/home/dpivonka/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dpivonka", "validate_certs": true} 2021-02-10T10:54:36.486 INFO:teuthology.task.ansible.out:changed: [gibba002.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC0m+8wna4RnrOcuCUj971EBe4cFiTXVkEWkIsUX3yeCUwMaS/ZjaoOjcdYX7cYpCp1ilo4FSO5BsochU0RiMbQEwLGcE0zB2q1ac8+yYKCuWWyknig2t4bOReu5OrXvl6mj/9R+p7clFlYhwIoNG5ZZ/2eTbo3nH7gWgaOKaPbTtvEGg/5R/qjQKg8OTC8oix6kLsaNTisQJ78Zd7b4ukg9bO0jlq/2UhQOBM072sU1/kJpWTkcA0xxm3nsC4rWPwapgz7UwBmMdyx116GZgWdfQecTLKO7b8pyJw21npk2ntUmeMi7WPZGX6VI3IpaAhBjW3KiATAb8JbAQh8F9bmejkw6x7zrP3hYKQOg3A+U8DKRRo/Ad+f0zz9gjC42DUvMEOr09RBdxZcEAPY7H/YRWEqbA2zPX3Gfth4b7/letzM3m9ToUvVfSt1CdjaZD+gIia9pCgDMb+j1j2w9reG4F7Us+W/1f5pYCyFyrm4XO3AdoYM9/FOAOR3vffTIQU= adking@localhost.localdomain", "key_options": null, "keyfile": "/home/adking/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "adking", "validate_certs": true} 2021-02-10T10:54:36.538 INFO:teuthology.task.ansible.out:changed: [gibba013.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC0m+8wna4RnrOcuCUj971EBe4cFiTXVkEWkIsUX3yeCUwMaS/ZjaoOjcdYX7cYpCp1ilo4FSO5BsochU0RiMbQEwLGcE0zB2q1ac8+yYKCuWWyknig2t4bOReu5OrXvl6mj/9R+p7clFlYhwIoNG5ZZ/2eTbo3nH7gWgaOKaPbTtvEGg/5R/qjQKg8OTC8oix6kLsaNTisQJ78Zd7b4ukg9bO0jlq/2UhQOBM072sU1/kJpWTkcA0xxm3nsC4rWPwapgz7UwBmMdyx116GZgWdfQecTLKO7b8pyJw21npk2ntUmeMi7WPZGX6VI3IpaAhBjW3KiATAb8JbAQh8F9bmejkw6x7zrP3hYKQOg3A+U8DKRRo/Ad+f0zz9gjC42DUvMEOr09RBdxZcEAPY7H/YRWEqbA2zPX3Gfth4b7/letzM3m9ToUvVfSt1CdjaZD+gIia9pCgDMb+j1j2w9reG4F7Us+W/1f5pYCyFyrm4XO3AdoYM9/FOAOR3vffTIQU= adking@localhost.localdomain", "key_options": null, "keyfile": "/home/adking/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "adking", "validate_certs": true} 2021-02-10T10:54:36.924 INFO:teuthology.task.ansible.out:changed: [gibba002.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCs6q2b5NcUsNXl+w1Mrc7idNvGdwWkL0XRX5zTw22UNpQyqPkSrRTFKi4NxoY8ucv8JoLnaJL7hYSRHKTh7SqH78uPLZX0FLtoHvpYLgUgJzzJ5AOW5uekIhCMCm+3egmFnEBwFcNxtGnU2es02ByXNEXRxt4/LQ0f8K6lMqxVRSKYcltvWzlA7WFFdTnYMunbCYOdpw88RXm2pYjGIHRBnie7+pFiFSS6K/mkzQcfLiJmahgLSaYslI0ho1vQd8xfNLmZ700C5F4VTUlT/mJK/dI2LF3OfaPTsyqKNzOchrjLFx5b6dhZ4WZyqu7VKsHNm+xHZwT8VP3LPGSVVBdmEsiCsmV4g4QTI5ssZLi7siZDpaFp72nVAIO1i6kBGuiF3or+J4LmIh8ZRKzTc0DRDOw7S+IkWu1fv2UhYg83z6wrz4HeczI+bnolFFWW6HgGV3MFaUeRTRvrbxw2DPoEGQ+cjP0wsjyEhxlScwdh8W8IbVpNl+7mBe45YGUH2hg83Lst5QQxD4PHelriu40moWRHFKCnKfNesb4L1uqu29o8f+/PgpLdrWE/P5zIKe2ReWJyScYAI7jTMlFw+xgf+0TKryk44NMy90l3ei0umlx4qsFbbEv+lhzAALrUJ/da7fsQw7eIVykmcscBeYr99/YSYk4MGdk4jVCiHsQkdw== amathuri@redhat.com", "key_options": null, "keyfile": "/home/amathuri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amathuri", "validate_certs": true} 2021-02-10T10:54:36.987 INFO:teuthology.task.ansible.out:changed: [gibba013.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCs6q2b5NcUsNXl+w1Mrc7idNvGdwWkL0XRX5zTw22UNpQyqPkSrRTFKi4NxoY8ucv8JoLnaJL7hYSRHKTh7SqH78uPLZX0FLtoHvpYLgUgJzzJ5AOW5uekIhCMCm+3egmFnEBwFcNxtGnU2es02ByXNEXRxt4/LQ0f8K6lMqxVRSKYcltvWzlA7WFFdTnYMunbCYOdpw88RXm2pYjGIHRBnie7+pFiFSS6K/mkzQcfLiJmahgLSaYslI0ho1vQd8xfNLmZ700C5F4VTUlT/mJK/dI2LF3OfaPTsyqKNzOchrjLFx5b6dhZ4WZyqu7VKsHNm+xHZwT8VP3LPGSVVBdmEsiCsmV4g4QTI5ssZLi7siZDpaFp72nVAIO1i6kBGuiF3or+J4LmIh8ZRKzTc0DRDOw7S+IkWu1fv2UhYg83z6wrz4HeczI+bnolFFWW6HgGV3MFaUeRTRvrbxw2DPoEGQ+cjP0wsjyEhxlScwdh8W8IbVpNl+7mBe45YGUH2hg83Lst5QQxD4PHelriu40moWRHFKCnKfNesb4L1uqu29o8f+/PgpLdrWE/P5zIKe2ReWJyScYAI7jTMlFw+xgf+0TKryk44NMy90l3ei0umlx4qsFbbEv+lhzAALrUJ/da7fsQw7eIVykmcscBeYr99/YSYk4MGdk4jVCiHsQkdw== amathuri@redhat.com", "key_options": null, "keyfile": "/home/amathuri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amathuri", "validate_certs": true} 2021-02-10T10:54:37.384 INFO:teuthology.task.ansible.out:changed: [gibba002.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDr6H7hmtGCUrO3GppgMHwbmA4XT/Vipz/uLG/N8h6ZxtaLQKpCefRpdKcFlOdDwxkTmT5q9vmXQ8nl95lTGq+eeMxl4ho104ZD9e9AIYmgYqFy7/qNP8gkqduIPlypLL9m5rElTcGd7zqfMqgljMvAmnXLtZNJlIuFtszVU5WPI1QCwfQ3t5aHF6j66bpVoN9uWItVGB1LApQz9TtViWsQ9Hl8X4YauJToO+TkuSX+H/OGH7ZC6CPOzNZb3b3vF/jLMnTzXl86PJs1se4DjKkRGLVCVbr2WhLwQeZ8TEn3Hg1Gn5X8zNM5cTo6U8SnXBAe9ZYbuyfvghrNLdlZpmN+8l0/l5zC3z+N+sEzklXwHw4MQLF+KSloVFd9NN/kyenufTZAyp8VxllZ9TgdC+CPORIWyzmu1elOtYo7LNuE6CIh/Zm9B54SO6hVHhVoOCqe/dIyjeHJA0WKIZblK1XZEvWuu0E2T3jhxfe1GICyTukeB8WoBQH9DZ1yzMPt59E= nia@redhat.com", "key_options": null, "keyfile": "/home/nizamial09/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nizamial09", "validate_certs": true} 2021-02-10T10:54:37.467 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:54:37 +0000 (0:01:00.984) 0:02:17.769 **** 2021-02-10T10:54:37.475 INFO:teuthology.task.ansible.out:changed: [gibba013.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDr6H7hmtGCUrO3GppgMHwbmA4XT/Vipz/uLG/N8h6ZxtaLQKpCefRpdKcFlOdDwxkTmT5q9vmXQ8nl95lTGq+eeMxl4ho104ZD9e9AIYmgYqFy7/qNP8gkqduIPlypLL9m5rElTcGd7zqfMqgljMvAmnXLtZNJlIuFtszVU5WPI1QCwfQ3t5aHF6j66bpVoN9uWItVGB1LApQz9TtViWsQ9Hl8X4YauJToO+TkuSX+H/OGH7ZC6CPOzNZb3b3vF/jLMnTzXl86PJs1se4DjKkRGLVCVbr2WhLwQeZ8TEn3Hg1Gn5X8zNM5cTo6U8SnXBAe9ZYbuyfvghrNLdlZpmN+8l0/l5zC3z+N+sEzklXwHw4MQLF+KSloVFd9NN/kyenufTZAyp8VxllZ9TgdC+CPORIWyzmu1elOtYo7LNuE6CIh/Zm9B54SO6hVHhVoOCqe/dIyjeHJA0WKIZblK1XZEvWuu0E2T3jhxfe1GICyTukeB8WoBQH9DZ1yzMPt59E= nia@redhat.com", "key_options": null, "keyfile": "/home/nizamial09/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nizamial09", "validate_certs": true} 2021-02-10T10:54:37.521 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-02-10T10:54:37.523 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:37.558 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:54:37 +0000 (0:00:00.090) 0:02:17.860 **** 2021-02-10T10:54:37.565 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:37.565 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:37.567 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "skip_reason": "Conditional result was False"} skipping: [gibba002.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd00 2021-02-10T10:54:37.568 INFO:teuthology.task.ansible.out:50353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "skip_reason": "Conditional result was False"} skipping: [gibba002.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:37.581 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:37.581 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:37.583 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:37.604 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:37.615 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:37.615 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:37.626 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:37.627 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:37.638 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:37.649 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:37.649 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:37.660 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:38.037 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "key_options": null, "keyfile": "/home/smithfarm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smithfarm", "validate_certs": true} 2021-02-10T10:54:38.049 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:38.049 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:38.112 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "key_options": null, "keyfile": "/home/smithfarm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smithfarm", "validate_certs": true} 2021-02-10T10:54:38.123 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:38.135 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:38.499 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "key_options": null, "keyfile": "/home/amaredia/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amaredia", "validate_certs": true} 2021-02-10T10:54:38.573 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "key_options": null, "keyfile": "/home/amaredia/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amaredia", "validate_certs": true} 2021-02-10T10:54:38.938 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "key_options": null, "keyfile": "/home/tserlin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tserlin", "validate_certs": true} 2021-02-10T10:54:38.950 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:38.961 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:38.971 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:38.982 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:38.983 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:38.994 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.005 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.006 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.017 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.018 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "key_options": null, "keyfile": "/home/tserlin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tserlin", "validate_certs": true} 2021-02-10T10:54:39.030 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.030 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.032 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.042 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.042 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.044 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "skip_reason": "Conditional result was False"} skipping: [gibba002.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.055 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.055 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.066 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.066 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.068 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "skip_reason": "Conditional result was False"} skipping: [gibba002.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.078 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.079 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.089 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.090 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.091 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.102 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.104 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "skip_reason": "Conditional result was False"} skipping: [gibba002.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.114 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.115 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "skip_reason": "Conditional result was False"} skipping: [gibba013.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.115 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.126 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.127 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.138 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.138 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.139 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "skip_reason": "Conditional result was False"} skipping: [gibba002.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.150 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.150 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.152 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "skip_reason": "Conditional result was False"} skipping: [gibba002.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.163 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.163 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.175 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.175 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.177 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "skip_reason": "Conditional result was False"} skipping: [gibba002.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.189 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.190 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.191 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.192 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.203 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.204 INFO:teuthology.task.ansible.out: skipping: [gibba002.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "skip_reason": "Conditional result was False"} skipping: [gibba013.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.216 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.217 INFO:teuthology.task.ansible.out: skipping: [gibba013.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.218 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.229 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.230 INFO:teuthology.task.ansible.out: skipping: [gibba002.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "skip_reason": "Conditional result was False"} skipping: [gibba013.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.232 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.233 INFO:teuthology.task.ansible.out: 2021-02-10T10:54:39.243 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.244 INFO:teuthology.task.ansible.out: skipping: [gibba002.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "skip_reason": "Conditional result was False"} skipping: [gibba013.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.246 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.256 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.258 INFO:teuthology.task.ansible.out: skipping: [gibba002.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.259 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.270 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.271 INFO:teuthology.task.ansible.out: skipping: [gibba013.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.282 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.283 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.285 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "skip_reason": "Conditional result was False"} skipping: [gibba013.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.295 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.296 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.296 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.307 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.308 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.308 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.310 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.320 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.320 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.321 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.332 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.332 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.334 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.344 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.344 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.346 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "skip_reason": "Conditional result was False"} skipping: [gibba002.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.356 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.357 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.358 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.369 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.369 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.370 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.380 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.381 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.382 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.393 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.394 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "skip_reason": "Conditional result was False"} skipping: [gibba002.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.404 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.405 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.406 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "skip_reason": "Conditional result was False"} skipping: [gibba002.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.417 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.419 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.429 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.430 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "skip_reason": "Conditional result was False"} skipping: [gibba013.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "skip_reason": "Conditional result was False"} skipping: [gibba002.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.442 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.442 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.444 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.454 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.455 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.456 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.466 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.467 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.468 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.478 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.479 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.480 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.490 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.491 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.492 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.503 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.505 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "skip_reason": "Conditional result was False"} skipping: [gibba002.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.514 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.527 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.528 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.529 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "skip_reason": "Conditional result was False"} skipping: [gibba013.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.542 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.543 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.544 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, "skip_reason": "Conditional result was False"} skipping: [gibba013.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.555 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.555 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.557 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "skip_reason": "Conditional result was False"} skipping: [gibba013.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "skip_reason": "Conditional result was False"} skipping: [gibba002.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.568 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.569 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.570 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.580 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.582 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "skip_reason": "Conditional result was False"} skipping: [gibba002.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.592 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.592 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.603 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.604 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.605 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "skip_reason": "Conditional result was False"} skipping: [gibba002.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.616 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.616 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.618 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.628 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.628 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.629 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.640 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.640 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.642 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.652 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.653 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.654 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "skip_reason": "Conditional result was False"} skipping: [gibba013.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.665 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.665 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.667 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.677 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.677 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.679 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.689 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.689 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.691 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.702 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.702 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.704 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "skip_reason": "Conditional result was False"} skipping: [gibba002.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.714 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.715 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.716 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.728 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.728 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.730 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.740 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.741 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.752 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.754 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "skip_reason": "Conditional result was False"} skipping: [gibba013.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "skip_reason": "Conditional result was False"} skipping: [gibba002.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.764 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.766 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "skip_reason": "Conditional result was False"} skipping: [gibba013.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.777 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.788 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.788 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.789 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "skip_reason": "Conditional result was False"} skipping: [gibba002.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.801 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.801 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.802 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "skip_reason": "Conditional result was False"} skipping: [gibba002.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.813 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.814 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.815 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "skip_reason": "Conditional result was False"} skipping: [gibba002.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.826 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.826 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.828 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.839 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.840 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.841 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "skip_reason": "Conditional result was False"} skipping: [gibba013.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.853 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.853 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.855 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.865 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.867 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.867 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.878 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.879 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "skip_reason": "Conditional result was False"} skipping: [gibba002.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.880 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.890 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.891 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.902 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.903 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "skip_reason": "Conditional result was False"} skipping: [gibba002.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "skip_reason": "Conditional result was False"} skipping: [gibba013.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.914 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.915 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.926 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.927 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.944 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.945 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.978 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:54:39 +0000 (0:00:02.420) 0:02:20.280 **** 2021-02-10T10:54:39.985 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.986 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.987 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "skip_reason": "Conditional result was False"} skipping: [gibba013.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:39.989 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "skip_reason": "Conditional result was False"} skipping: [gibba013.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:40.000 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:40.018 INFO:teuthology.task.ansible.out: TASK [users : Filter the revoked_users list] *********************************** 2021-02-10T10:54:40.020 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:40.050 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:54:40 +0000 (0:00:00.071) 0:02:20.351 **** 2021-02-10T10:54:40.082 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:54:40 +0000 (0:00:00.031) 0:02:20.383 **** 2021-02-10T10:54:40.111 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:54:40.146 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:54:40 +0000 (0:00:00.064) 0:02:20.448 **** 2021-02-10T10:54:40.527 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2021-02-10T10:54:40.528 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item=zcerza) => {"ansible_loop_var": "item", "changed": false, "item": "zcerza", "name": "zcerza", "state": "absent"} 2021-02-10T10:54:40.621 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item=zcerza) => {"ansible_loop_var": "item", "changed": false, "item": "zcerza", "name": "zcerza", "state": "absent"} 2021-02-10T10:54:40.955 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item=aschoen) => {"ansible_loop_var": "item", "changed": false, "item": "aschoen", "name": "aschoen", "state": "absent"} 2021-02-10T10:54:41.061 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item=aschoen) => {"ansible_loop_var": "item", "changed": false, "item": "aschoen", "name": "aschoen", "state": "absent"} 2021-02-10T10:54:41.395 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item=andrew) => {"ansible_loop_var": "item", "changed": false, "item": "andrew", "name": "andrew", "state": "absent"} 2021-02-10T10:54:41.490 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item=andrew) => {"ansible_loop_var": "item", "changed": false, "item": "andrew", "name": "andrew", "state": "absent"} 2021-02-10T10:54:41.834 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item=sweil) => {"ansible_loop_var": "item", "changed": false, "item": "sweil", "name": "sweil", "state": "absent"} 2021-02-10T10:54:41.928 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item=sweil) => {"ansible_loop_var": "item", "changed": false, "item": "sweil", "name": "sweil", "state": "absent"} 2021-02-10T10:54:42.274 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item=brad) => {"ansible_loop_var": "item", "changed": false, "item": "brad", "name": "brad", "state": "absent"} 2021-02-10T10:54:42.369 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item=brad) => {"ansible_loop_var": "item", "changed": false, "item": "brad", "name": "brad", "state": "absent"} 2021-02-10T10:54:42.716 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item=kefu) => {"ansible_loop_var": "item", "changed": false, "item": "kefu", "name": "kefu", "state": "absent"} 2021-02-10T10:54:42.801 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item=kefu) => {"ansible_loop_var": "item", "changed": false, "item": "kefu", "name": "kefu", "state": "absent"} 2021-02-10T10:54:43.145 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item=shylesh) => {"ansible_loop_var": "item", "changed": false, "item": "shylesh", "name": "shylesh", "state": "absent"} 2021-02-10T10:54:43.239 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item=shylesh) => {"ansible_loop_var": "item", "changed": false, "item": "shylesh", "name": "shylesh", "state": "absent"} 2021-02-10T10:54:43.585 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item=gmeno) => {"ansible_loop_var": "item", "changed": false, "item": "gmeno", "name": "gmeno", "state": "absent"} 2021-02-10T10:54:43.670 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item=gmeno) => {"ansible_loop_var": "item", "changed": false, "item": "gmeno", "name": "gmeno", "state": "absent"} 2021-02-10T10:54:44.017 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item=alfredodeza) => {"ansible_loop_var": "item", "changed": false, "item": "alfredodeza", "name": "alfredodeza", "state": "absent"} 2021-02-10T10:54:44.112 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item=alfredodeza) => {"ansible_loop_var": "item", "changed": false, "item": "alfredodeza", "name": "alfredodeza", "state": "absent"} 2021-02-10T10:54:44.461 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item=trhoden) => {"ansible_loop_var": "item", "changed": false, "item": "trhoden", "name": "trhoden", "state": "absent"} 2021-02-10T10:54:44.546 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item=trhoden) => {"ansible_loop_var": "item", "changed": false, "item": "trhoden", "name": "trhoden", "state": "absent"} 2021-02-10T10:54:44.902 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item=nishtha) => {"ansible_loop_var": "item", "changed": false, "item": "nishtha", "name": "nishtha", "state": "absent"} 2021-02-10T10:54:44.986 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item=nishtha) => {"ansible_loop_var": "item", "changed": false, "item": "nishtha", "name": "nishtha", "state": "absent"} 2021-02-10T10:54:45.341 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item=yguang) => {"ansible_loop_var": "item", "changed": false, "item": "yguang", "name": "yguang", "state": "absent"} 2021-02-10T10:54:45.426 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item=yguang) => {"ansible_loop_var": "item", "changed": false, "item": "yguang", "name": "yguang", "state": "absent"} 2021-02-10T10:54:45.772 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item=sdieffen) => {"ansible_loop_var": "item", "changed": false, "item": "sdieffen", "name": "sdieffen", "state": "absent"} 2021-02-10T10:54:45.856 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item=sdieffen) => {"ansible_loop_var": "item", "changed": false, "item": "sdieffen", "name": "sdieffen", "state": "absent"} 2021-02-10T10:54:46.211 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item=brian) => {"ansible_loop_var": "item", "changed": false, "item": "brian", "name": "brian", "state": "absent"} 2021-02-10T10:54:46.297 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item=brian) => {"ansible_loop_var": "item", "changed": false, "item": "brian", "name": "brian", "state": "absent"} 2021-02-10T10:54:46.643 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item=pmcgarry) => {"ansible_loop_var": "item", "changed": false, "item": "pmcgarry", "name": "pmcgarry", "state": "absent"} 2021-02-10T10:54:46.738 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item=pmcgarry) => {"ansible_loop_var": "item", "changed": false, "item": "pmcgarry", "name": "pmcgarry", "state": "absent"} 2021-02-10T10:54:47.087 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item=karnan) => {"ansible_loop_var": "item", "changed": false, "item": "karnan", "name": "karnan", "state": "absent"} 2021-02-10T10:54:47.171 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item=karnan) => {"ansible_loop_var": "item", "changed": false, "item": "karnan", "name": "karnan", "state": "absent"} 2021-02-10T10:54:47.518 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item=ryneli) => {"ansible_loop_var": "item", "changed": false, "item": "ryneli", "name": "ryneli", "state": "absent"} 2021-02-10T10:54:47.609 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item=ryneli) => {"ansible_loop_var": "item", "changed": false, "item": "ryneli", "name": "ryneli", "state": "absent"} 2021-02-10T10:54:47.978 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item=dlambrig) => {"ansible_loop_var": "item", "changed": false, "item": "dlambrig", "name": "dlambrig", "state": "absent"} 2021-02-10T10:54:48.064 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item=dlambrig) => {"ansible_loop_var": "item", "changed": false, "item": "dlambrig", "name": "dlambrig", "state": "absent"} 2021-02-10T10:54:48.423 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item=icolle) => {"ansible_loop_var": "item", "changed": false, "item": "icolle", "name": "icolle", "state": "absent"} 2021-02-10T10:54:48.498 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item=icolle) => {"ansible_loop_var": "item", "changed": false, "item": "icolle", "name": "icolle", "state": "absent"} 2021-02-10T10:54:48.856 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item=soumya) => {"ansible_loop_var": "item", "changed": false, "item": "soumya", "name": "soumya", "state": "absent"} 2021-02-10T10:54:48.931 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item=soumya) => {"ansible_loop_var": "item", "changed": false, "item": "soumya", "name": "soumya", "state": "absent"} 2021-02-10T10:54:49.286 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item=jspray) => {"ansible_loop_var": "item", "changed": false, "item": "jspray", "name": "jspray", "state": "absent"} 2021-02-10T10:54:49.371 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item=jspray) => {"ansible_loop_var": "item", "changed": false, "item": "jspray", "name": "jspray", "state": "absent"} 2021-02-10T10:54:49.735 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item=loic) => {"ansible_loop_var": "item", "changed": false, "item": "loic", "name": "loic", "state": "absent"} 2021-02-10T10:54:49.809 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item=loic) => {"ansible_loop_var": "item", "changed": false, "item": "loic", "name": "loic", "state": "absent"} 2021-02-10T10:54:50.162 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item=erwan) => {"ansible_loop_var": "item", "changed": false, "item": "erwan", "name": "erwan", "state": "absent"} 2021-02-10T10:54:50.248 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item=erwan) => {"ansible_loop_var": "item", "changed": false, "item": "erwan", "name": "erwan", "state": "absent"} 2021-02-10T10:54:50.606 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item=jj) => {"ansible_loop_var": "item", "changed": false, "item": "jj", "name": "jj", "state": "absent"} 2021-02-10T10:54:50.679 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item=jj) => {"ansible_loop_var": "item", "changed": false, "item": "jj", "name": "jj", "state": "absent"} 2021-02-10T10:54:51.045 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item=amarangone) => {"ansible_loop_var": "item", "changed": false, "item": "amarangone", "name": "amarangone", "state": "absent"} 2021-02-10T10:54:51.121 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item=amarangone) => {"ansible_loop_var": "item", "changed": false, "item": "amarangone", "name": "amarangone", "state": "absent"} 2021-02-10T10:54:51.490 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item=oprypin) => {"ansible_loop_var": "item", "changed": false, "item": "oprypin", "name": "oprypin", "state": "absent"} 2021-02-10T10:54:51.564 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item=oprypin) => {"ansible_loop_var": "item", "changed": false, "item": "oprypin", "name": "oprypin", "state": "absent"} 2021-02-10T10:54:51.930 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item=adamyanova) => {"ansible_loop_var": "item", "changed": false, "item": "adamyanova", "name": "adamyanova", "state": "absent"} 2021-02-10T10:54:51.993 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item=adamyanova) => {"ansible_loop_var": "item", "changed": false, "item": "adamyanova", "name": "adamyanova", "state": "absent"} 2021-02-10T10:54:52.372 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item=sbillah) => {"ansible_loop_var": "item", "changed": false, "item": "sbillah", "name": "sbillah", "state": "absent"} 2021-02-10T10:54:52.451 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item=sbillah) => {"ansible_loop_var": "item", "changed": false, "item": "sbillah", "name": "sbillah", "state": "absent"} 2021-02-10T10:54:52.829 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item=onyb) => {"ansible_loop_var": "item", "changed": false, "item": "onyb", "name": "onyb", "state": "absent"} 2021-02-10T10:54:52.915 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item=onyb) => {"ansible_loop_var": "item", "changed": false, "item": "onyb", "name": "onyb", "state": "absent"} 2021-02-10T10:54:53.273 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item=jwilliamson) => {"ansible_loop_var": "item", "changed": false, "item": "jwilliamson", "name": "jwilliamson", "state": "absent"} 2021-02-10T10:54:53.357 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item=jwilliamson) => {"ansible_loop_var": "item", "changed": false, "item": "jwilliamson", "name": "jwilliamson", "state": "absent"} 2021-02-10T10:54:53.734 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item=kmroz) => {"ansible_loop_var": "item", "changed": false, "item": "kmroz", "name": "kmroz", "state": "absent"} 2021-02-10T10:54:53.807 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item=kmroz) => {"ansible_loop_var": "item", "changed": false, "item": "kmroz", "name": "kmroz", "state": "absent"} 2021-02-10T10:54:54.175 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item=shehbazj) => {"ansible_loop_var": "item", "changed": false, "item": "shehbazj", "name": "shehbazj", "state": "absent"} 2021-02-10T10:54:54.251 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item=shehbazj) => {"ansible_loop_var": "item", "changed": false, "item": "shehbazj", "name": "shehbazj", "state": "absent"} 2021-02-10T10:54:54.616 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item=abhishekvrshny) => {"ansible_loop_var": "item", "changed": false, "item": "abhishekvrshny", "name": "abhishekvrshny", "state": "absent"} 2021-02-10T10:54:54.691 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item=abhishekvrshny) => {"ansible_loop_var": "item", "changed": false, "item": "abhishekvrshny", "name": "abhishekvrshny", "state": "absent"} 2021-02-10T10:54:55.047 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item=asheplyakov) => {"ansible_loop_var": "item", "changed": false, "item": "asheplyakov", "name": "asheplyakov", "state": "absent"} 2021-02-10T10:54:55.133 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item=asheplyakov) => {"ansible_loop_var": "item", "changed": false, "item": "asheplyakov", "name": "asheplyakov", "state": "absent"} 2021-02-10T10:54:55.490 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item=liupan) => {"ansible_loop_var": "item", "changed": false, "item": "liupan", "name": "liupan", "state": "absent"} 2021-02-10T10:54:55.574 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item=liupan) => {"ansible_loop_var": "item", "changed": false, "item": "liupan", "name": "liupan", "state": "absent"} 2021-02-10T10:54:55.921 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item=adeza) => {"ansible_loop_var": "item", "changed": false, "item": "adeza", "name": "adeza", "state": "absent"} 2021-02-10T10:54:56.006 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item=adeza) => {"ansible_loop_var": "item", "changed": false, "item": "adeza", "name": "adeza", "state": "absent"} 2021-02-10T10:54:56.364 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item=pranith) => {"ansible_loop_var": "item", "changed": false, "item": "pranith", "name": "pranith", "state": "absent"} 2021-02-10T10:54:56.448 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item=pranith) => {"ansible_loop_var": "item", "changed": false, "item": "pranith", "name": "pranith", "state": "absent"} 2021-02-10T10:54:57.117 INFO:teuthology.task.ansible.out:changed: [gibba002.front.sepia.ceph.com] => (item=dorinda) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "dorinda", "name": "dorinda", "remove": false, "state": "absent"} 2021-02-10T10:54:57.181 INFO:teuthology.task.ansible.out:changed: [gibba013.front.sepia.ceph.com] => (item=dorinda) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "dorinda", "name": "dorinda", "remove": false, "state": "absent"} 2021-02-10T10:54:57.911 INFO:teuthology.task.ansible.out:changed: [gibba002.front.sepia.ceph.com] => (item=zyan) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "zyan", "name": "zyan", "remove": false, "state": "absent"} 2021-02-10T10:54:57.953 INFO:teuthology.task.ansible.out:changed: [gibba013.front.sepia.ceph.com] => (item=zyan) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "zyan", "name": "zyan", "remove": false, "state": "absent"} 2021-02-10T10:54:58.345 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item=nlevine) => {"ansible_loop_var": "item", "changed": false, "item": "nlevine", "name": "nlevine", "state": "absent"} 2021-02-10T10:54:58.399 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item=nlevine) => {"ansible_loop_var": "item", "changed": false, "item": "nlevine", "name": "nlevine", "state": "absent"} 2021-02-10T10:54:58.788 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item=tbrekke) => {"ansible_loop_var": "item", "changed": false, "item": "tbrekke", "name": "tbrekke", "state": "absent"} 2021-02-10T10:54:58.831 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item=tbrekke) => {"ansible_loop_var": "item", "changed": false, "item": "tbrekke", "name": "tbrekke", "state": "absent"} 2021-02-10T10:54:59.222 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item=taco) => {"ansible_loop_var": "item", "changed": false, "item": "taco", "name": "taco", "state": "absent"} 2021-02-10T10:54:59.276 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item=taco) => {"ansible_loop_var": "item", "changed": false, "item": "taco", "name": "taco", "state": "absent"} 2021-02-10T10:54:59.664 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item=louis) => {"ansible_loop_var": "item", "changed": false, "item": "louis", "name": "louis", "state": "absent"} 2021-02-10T10:54:59.709 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item=louis) => {"ansible_loop_var": "item", "changed": false, "item": "louis", "name": "louis", "state": "absent"} 2021-02-10T10:55:00.099 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item=amarango) => {"ansible_loop_var": "item", "changed": false, "item": "amarango", "name": "amarango", "state": "absent"} 2021-02-10T10:55:00.152 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item=amarango) => {"ansible_loop_var": "item", "changed": false, "item": "amarango", "name": "amarango", "state": "absent"} 2021-02-10T10:55:00.543 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item=oobe) => {"ansible_loop_var": "item", "changed": false, "item": "oobe", "name": "oobe", "state": "absent"} 2021-02-10T10:55:00.587 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item=oobe) => {"ansible_loop_var": "item", "changed": false, "item": "oobe", "name": "oobe", "state": "absent"} 2021-02-10T10:55:00.989 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item=rturk) => {"ansible_loop_var": "item", "changed": false, "item": "rturk", "name": "rturk", "state": "absent"} 2021-02-10T10:55:01.022 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item=rturk) => {"ansible_loop_var": "item", "changed": false, "item": "rturk", "name": "rturk", "state": "absent"} 2021-02-10T10:55:01.438 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item=fche) => {"ansible_loop_var": "item", "changed": false, "item": "fche", "name": "fche", "state": "absent"} 2021-02-10T10:55:01.472 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item=fche) => {"ansible_loop_var": "item", "changed": false, "item": "fche", "name": "fche", "state": "absent"} 2021-02-10T10:55:01.894 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item=jbainbri) => {"ansible_loop_var": "item", "changed": false, "item": "jbainbri", "name": "jbainbri", "state": "absent"} 2021-02-10T10:55:01.916 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item=jbainbri) => {"ansible_loop_var": "item", "changed": false, "item": "jbainbri", "name": "jbainbri", "state": "absent"} 2021-02-10T10:55:02.334 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item=kdhananj) => {"ansible_loop_var": "item", "changed": false, "item": "kdhananj", "name": "kdhananj", "state": "absent"} 2021-02-10T10:55:02.403 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:55:02 +0000 (0:00:22.257) 0:02:42.705 **** 2021-02-10T10:55:02.414 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item=kdhananj) => {"ansible_loop_var": "item", "changed": false, "item": "kdhananj", "name": "kdhananj", "state": "absent"} 2021-02-10T10:55:02.472 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:55:02 +0000 (0:00:00.069) 0:02:42.774 **** 2021-02-10T10:55:05.396 INFO:teuthology.task.ansible.out: TASK [users : Write /keys-repo-sha1 sentinel file] ***************************** 2021-02-10T10:55:05.396 INFO:teuthology.task.ansible.out:changed: [gibba013.front.sepia.ceph.com] => {"changed": true, "checksum": "5ebc9fcf8859a9c716ab10089a7bec7e44d29368", "dest": "/keys-repo-sha1", "gid": 0, "group": "root", "md5sum": "ee06ed012f51c2e3764171d4b9180349", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:etc_runtime_t:s0", "size": 40, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1612954502.5163991-142002955341403/source", "state": "file", "uid": 0} 2021-02-10T10:55:05.397 INFO:teuthology.task.ansible.out:changed: [gibba002.front.sepia.ceph.com] => {"changed": true, "checksum": "5ebc9fcf8859a9c716ab10089a7bec7e44d29368", "dest": "/keys-repo-sha1", "gid": 0, "group": "root", "md5sum": "ee06ed012f51c2e3764171d4b9180349", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:etc_runtime_t:s0", "size": 40, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1612954502.4441657-194266861923565/source", "state": "file", "uid": 0} 2021-02-10T10:55:05.451 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:55:05 +0000 (0:00:02.978) 0:02:45.752 **** 2021-02-10T10:55:05.489 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:55:05 +0000 (0:00:00.038) 0:02:45.791 **** 2021-02-10T10:55:05.496 INFO:teuthology.task.ansible.out: TASK [common : Log the OS name, version and release] *************************** 2021-02-10T10:55:05.497 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => { "msg": "Host gibba002.front.sepia.ceph.com is running RedHat 7.9 (Maipo)" } 2021-02-10T10:55:05.546 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:55:05 +0000 (0:00:00.056) 0:02:45.847 **** 2021-02-10T10:55:05.555 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => { "msg": "Host gibba013.front.sepia.ceph.com is running RedHat 7.9 (Maipo)" } 2021-02-10T10:55:05.599 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:55:05 +0000 (0:00:00.052) 0:02:45.900 **** 2021-02-10T10:55:05.608 INFO:teuthology.task.ansible.out: TASK [common : Including major version specific variables.] ******************** 2021-02-10T10:55:05.609 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/redhat_7.yml) => {"ansible_facts": {"rhsm_repos": ["rhel-7-server-rpms", "rhel-7-server-optional-rpms", "rhel-7-server-extras-rpms", "rhel-ha-for-rhel-7-server-rpms"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/redhat_7.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/redhat_7.yml"} 2021-02-10T10:55:05.647 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:55:05 +0000 (0:00:00.048) 0:02:45.949 **** 2021-02-10T10:55:05.655 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/redhat_7.yml) => {"ansible_facts": {"rhsm_repos": ["rhel-7-server-rpms", "rhel-7-server-optional-rpms", "rhel-7-server-extras-rpms", "rhel-ha-for-rhel-7-server-rpms"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/redhat_7.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/redhat_7.yml"} 2021-02-10T10:55:05.691 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:55:05 +0000 (0:00:00.043) 0:02:45.993 **** 2021-02-10T10:55:05.698 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 6)] *********************** 2021-02-10T10:55:05.699 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:55:05.736 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:55:05 +0000 (0:00:00.045) 0:02:46.038 **** 2021-02-10T10:55:05.744 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:55:05.779 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:55:05 +0000 (0:00:00.042) 0:02:46.081 **** 2021-02-10T10:55:06.268 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 7)] *********************** 2021-02-10T10:55:06.270 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => {"changed": false, "cmd": "timedatectl | grep -E \"Time ?zone\" | sed -e \"s/.*: \\(.*\\) (.*/\\1/\"", "delta": "0:00:00.077080", "end": "2021-02-10 10:55:06.246491", "rc": 0, "start": "2021-02-10 10:55:06.169411", "stderr": "", "stderr_lines": [], "stdout": "Etc/UTC", "stdout_lines": ["Etc/UTC"]} 2021-02-10T10:55:06.304 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:55:06 +0000 (0:00:00.524) 0:02:46.606 **** 2021-02-10T10:55:06.311 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => {"changed": false, "cmd": "timedatectl | grep -E \"Time ?zone\" | sed -e \"s/.*: \\(.*\\) (.*/\\1/\"", "delta": "0:00:00.063757", "end": "2021-02-10 10:55:06.276135", "rc": 0, "start": "2021-02-10 10:55:06.212378", "stderr": "", "stderr_lines": [], "stdout": "Etc/UTC", "stdout_lines": ["Etc/UTC"]} 2021-02-10T10:55:06.352 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:55:06 +0000 (0:00:00.047) 0:02:46.653 **** 2021-02-10T10:55:06.358 INFO:teuthology.task.ansible.out: TASK [common : Reboot RHEL7 to workaround systemd bug] ************************* 2021-02-10T10:55:06.360 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:55:06.396 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:55:06 +0000 (0:00:00.044) 0:02:46.698 **** 2021-02-10T10:55:06.405 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:55:06.440 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:55:06 +0000 (0:00:00.043) 0:02:46.741 **** 2021-02-10T10:55:06.446 INFO:teuthology.task.ansible.out: TASK [common : Wait for reboot in case of systemd workaround] ****************** 2021-02-10T10:55:06.447 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:55:06.481 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:55:06 +0000 (0:00:00.041) 0:02:46.782 **** 2021-02-10T10:55:06.489 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:55:06.523 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:55:06 +0000 (0:00:00.042) 0:02:46.825 **** 2021-02-10T10:55:06.529 INFO:teuthology.task.ansible.out: TASK [common : Set /etc/localtime (RHEL/CentOS 6)] ***************************** 2021-02-10T10:55:06.530 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:55:06.564 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:55:06 +0000 (0:00:00.040) 0:02:46.865 **** 2021-02-10T10:55:06.570 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:55:06.604 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:55:06 +0000 (0:00:00.040) 0:02:46.906 **** 2021-02-10T10:55:06.611 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone (RHEL/CentOS >= 7)] **************************** 2021-02-10T10:55:06.612 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:55:06.645 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:55:06 +0000 (0:00:00.041) 0:02:46.947 **** 2021-02-10T10:55:06.652 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:55:06.687 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:55:06 +0000 (0:00:00.041) 0:02:46.989 **** 2021-02-10T10:55:06.693 INFO:teuthology.task.ansible.out: TASK [common : Install yum on Fedora 22 and later] ***************************** 2021-02-10T10:55:06.694 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:55:06.728 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:55:06 +0000 (0:00:00.040) 0:02:47.029 **** 2021-02-10T10:55:06.734 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:55:06.770 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:55:06 +0000 (0:00:00.042) 0:02:47.072 **** 2021-02-10T10:55:06.787 INFO:teuthology.task.ansible.out: TASK [common : Set entitlements_path] ****************************************** 2021-02-10T10:55:06.788 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => {"ansible_facts": {"entitlements_path": "/etc/ansible/secrets/entitlements.yml"}, "changed": false} 2021-02-10T10:55:06.822 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:55:06 +0000 (0:00:00.051) 0:02:47.123 **** 2021-02-10T10:55:06.829 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => {"ansible_facts": {"entitlements_path": "/etc/ansible/secrets/entitlements.yml"}, "changed": false} 2021-02-10T10:55:06.863 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:55:06 +0000 (0:00:00.041) 0:02:47.165 **** 2021-02-10T10:55:06.880 INFO:teuthology.task.ansible.out: TASK [common : Include Red Hat encrypted variables.] *************************** 2021-02-10T10:55:06.882 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} ok: [gibba002.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-02-10T10:55:06.925 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:55:06 +0000 (0:00:00.061) 0:02:47.226 **** 2021-02-10T10:55:06.931 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-02-10T10:55:06.932 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-02-10T10:55:06.966 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:55:06 +0000 (0:00:00.041) 0:02:47.268 **** 2021-02-10T10:55:06.973 INFO:teuthology.task.ansible.out: TASK [common : Set have_entitlements] ****************************************** 2021-02-10T10:55:06.974 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => {"ansible_facts": {"have_entitlements": true}, "changed": false} 2021-02-10T10:55:07.017 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:55:07 +0000 (0:00:00.051) 0:02:47.319 **** 2021-02-10T10:55:07.025 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => {"ansible_facts": {"have_entitlements": true}, "changed": false} 2021-02-10T10:55:07.066 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:55:07 +0000 (0:00:00.048) 0:02:47.368 **** 2021-02-10T10:55:08.723 INFO:teuthology.task.ansible.out: TASK [common : Install CA Cert from Satellite Server] ************************** 2021-02-10T10:55:08.725 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => {"changed": false, "msg": "", "rc": 0, "results": ["katello-ca-consumer-satellite.front.sepia.ceph.com-1.0-3.noarch providing /root/.ansible/tmp/ansible-moduletmp-1612954507.95-R_NyI8/katello-ca-consumer-latest.noarchvejJYR.rpm is already installed"]} 2021-02-10T10:55:08.769 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:55:08 +0000 (0:00:01.702) 0:02:49.070 **** 2021-02-10T10:55:08.779 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => {"changed": false, "msg": "", "rc": 0, "results": ["katello-ca-consumer-satellite.front.sepia.ceph.com-1.0-3.noarch providing /root/.ansible/tmp/ansible-moduletmp-1612954507.95-JvVJG2/katello-ca-consumer-latest.noarchlKaiO8.rpm is already installed"]} 2021-02-10T10:55:08.820 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:55:08 +0000 (0:00:00.051) 0:02:49.122 **** 2021-02-10T10:55:09.631 INFO:teuthology.task.ansible.out: TASK [common : Determine if node is registered with subscription-manager.] ***** 2021-02-10T10:55:09.632 INFO:teuthology.task.ansible.out:fatal: [gibba002.front.sepia.ceph.com]: FAILED! => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} ...ignoring 2021-02-10T10:55:09.665 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:55:09 +0000 (0:00:00.844) 0:02:49.967 **** 2021-02-10T10:55:09.686 INFO:teuthology.task.ansible.out:fatal: [gibba013.front.sepia.ceph.com]: FAILED! => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} ...ignoring 2021-02-10T10:55:09.721 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:55:09 +0000 (0:00:00.055) 0:02:50.022 **** 2021-02-10T10:55:09.728 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we're already registered] **************** 2021-02-10T10:55:09.729 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => {"ansible_facts": {"rhsm_registered": false}, "changed": false} 2021-02-10T10:55:09.765 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:55:09 +0000 (0:00:00.044) 0:02:50.067 **** 2021-02-10T10:55:09.775 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => {"ansible_facts": {"rhsm_registered": false}, "changed": false} 2021-02-10T10:55:09.825 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:55:09 +0000 (0:00:00.059) 0:02:50.126 **** 2021-02-10T10:55:14.579 INFO:teuthology.task.ansible.out: TASK [common : Register with subscription-manager.] **************************** 2021-02-10T10:55:14.581 INFO:teuthology.task.ansible.out:changed: [gibba002.front.sepia.ceph.com] => {"attempts": 1, "censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} 2021-02-10T10:55:14.635 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:55:14 +0000 (0:00:04.810) 0:02:54.936 **** 2021-02-10T10:55:14.684 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we just registered] ********************** 2021-02-10T10:55:14.686 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => {"ansible_facts": {"rhsm_registered": true}, "changed": false} 2021-02-10T10:55:14.717 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:55:14 +0000 (0:00:00.082) 0:02:55.019 **** 2021-02-10T10:55:15.426 INFO:teuthology.task.ansible.out: TASK [common : List CDN releases available to system] ************************** 2021-02-10T10:55:15.427 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => {"changed": false, "cmd": "subscription-manager release --list | grep -E '[0-9]'", "delta": "0:00:00.229739", "end": "2021-02-10 10:55:15.401663", "failed_when_result": false, "rc": 0, "start": "2021-02-10 10:55:15.171924", "stderr": "", "stderr_lines": [], "stdout": "7.4\n7.5\n7.6\n7.7\n7.8\n7.9\n7Server\n8\n8.0\n8.1\n8.2\n8.3", "stdout_lines": ["7.4", "7.5", "7.6", "7.7", "7.8", "7.9", "7Server", "8", "8.0", "8.1", "8.2", "8.3"]} 2021-02-10T10:55:15.463 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:55:15 +0000 (0:00:00.745) 0:02:55.765 **** 2021-02-10T10:55:15.480 INFO:teuthology.task.ansible.out: TASK [common : Register with subscription-manager.] **************************** 2021-02-10T10:55:15.482 INFO:teuthology.task.ansible.out:changed: [gibba013.front.sepia.ceph.com] => {"attempts": 1, "censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} 2021-02-10T10:55:15.520 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:55:15 +0000 (0:00:00.057) 0:02:55.822 **** 2021-02-10T10:55:15.531 INFO:teuthology.task.ansible.out: TASK [common : Unregister Beta/Alpha/RC system with subscription-manager] ****** skipping: [gibba002.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:55:15.582 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:55:15 +0000 (0:00:00.061) 0:02:55.884 **** 2021-02-10T10:55:15.590 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we just registered] ********************** 2021-02-10T10:55:15.592 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => {"ansible_facts": {"rhsm_registered": true}, "changed": false} 2021-02-10T10:55:15.636 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:55:15 +0000 (0:00:00.053) 0:02:55.937 **** 2021-02-10T10:55:15.646 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered to false if Beta/Alpha/RC release] ********** 2021-02-10T10:55:15.647 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:55:15.707 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:55:15 +0000 (0:00:00.071) 0:02:56.009 **** 2021-02-10T10:55:15.769 INFO:teuthology.task.ansible.out: TASK [common : Configure internal beta repos in /etc/yum.repos.d/] ************* skipping: [gibba002.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-02-10T10:55:15.830 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:55:15 +0000 (0:00:00.122) 0:02:56.131 **** 2021-02-10T10:55:15.831 INFO:teuthology.task.ansible.out: 2021-02-10T10:55:15.877 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2021-02-10T10:55:15.879 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:55:15.916 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:55:15 +0000 (0:00:00.086) 0:02:56.218 **** 2021-02-10T10:55:16.385 INFO:teuthology.task.ansible.out: TASK [common : List CDN releases available to system] ************************** 2021-02-10T10:55:16.387 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => {"changed": false, "cmd": "subscription-manager release --list | grep -E '[0-9]'", "delta": "0:00:00.279214", "end": "2021-02-10 10:55:16.368424", "failed_when_result": false, "rc": 0, "start": "2021-02-10 10:55:16.089210", "stderr": "", "stderr_lines": [], "stdout": "7.4\n7.5\n7.6\n7.7\n7.8\n7.9\n7Server\n8\n8.0\n8.1\n8.2\n8.3", "stdout_lines": ["7.4", "7.5", "7.6", "7.7", "7.8", "7.9", "7Server", "8", "8.0", "8.1", "8.2", "8.3"]} 2021-02-10T10:55:16.427 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:55:16 +0000 (0:00:00.507) 0:02:56.725 **** 2021-02-10T10:55:16.462 INFO:teuthology.task.ansible.out: TASK [common : Unregister Beta/Alpha/RC system with subscription-manager] ****** 2021-02-10T10:55:16.464 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:55:16.506 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:55:16 +0000 (0:00:00.082) 0:02:56.808 **** 2021-02-10T10:55:16.540 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered to false if Beta/Alpha/RC release] ********** skipping: [gibba013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:55:16.592 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:55:16 +0000 (0:00:00.085) 0:02:56.893 **** 2021-02-10T10:55:16.623 INFO:teuthology.task.ansible.out: TASK [common : Configure internal beta repos in /etc/yum.repos.d/] ************* skipping: [gibba013.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-02-10T10:55:16.660 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:55:16 +0000 (0:00:00.068) 0:02:56.962 **** 2021-02-10T10:55:16.699 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ skipping: [gibba013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:55:16.737 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:55:16 +0000 (0:00:00.076) 0:02:57.038 **** 2021-02-10T10:55:17.746 INFO:teuthology.task.ansible.out: TASK [common : Get list of enabled RHSM repos] ********************************* 2021-02-10T10:55:17.748 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => {"changed": false, "cmd": "subscription-manager repos --list | grep -B4 'Enabled:.*1' | grep 'Repo ID:' | sed -e 's/Repo ID:\\s*\\(.*\\)/\\1/' | sort", "delta": "0:00:01.350392", "end": "2021-02-10 10:55:17.728167", "rc": 0, "start": "2021-02-10 10:55:16.377775", "stderr": "", "stderr_lines": [], "stdout": "rhel-7-server-rpms", "stdout_lines": ["rhel-7-server-rpms"]} 2021-02-10T10:55:17.782 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:55:17 +0000 (0:00:01.045) 0:02:58.083 **** 2021-02-10T10:55:17.819 INFO:teuthology.task.ansible.out: TASK [common : Store list of enabled repos] ************************************ 2021-02-10T10:55:17.821 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => {"ansible_facts": {"repo_list": ["rhel-7-server-rpms"]}, "changed": false} 2021-02-10T10:55:17.853 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:55:17 +0000 (0:00:00.071) 0:02:58.155 **** 2021-02-10T10:55:17.893 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos false if entitlements are missing or if we unregistered] *** 2021-02-10T10:55:17.895 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:55:17.932 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:55:17 +0000 (0:00:00.078) 0:02:58.233 **** 2021-02-10T10:55:17.970 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if rhsm_repos differs from repo_list] **** 2021-02-10T10:55:17.971 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => {"ansible_facts": {"replace_repos": true}, "changed": false} 2021-02-10T10:55:18.006 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:55:18 +0000 (0:00:00.074) 0:02:58.308 **** 2021-02-10T10:55:18.055 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if newly-subscribed] ********************* 2021-02-10T10:55:18.056 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => {"ansible_facts": {"replace_repos": true}, "changed": false} 2021-02-10T10:55:18.090 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:55:18 +0000 (0:00:00.084) 0:02:58.392 **** 2021-02-10T10:55:18.478 INFO:teuthology.task.ansible.out: TASK [common : Get list of enabled RHSM repos] ********************************* 2021-02-10T10:55:18.479 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => {"changed": false, "cmd": "subscription-manager repos --list | grep -B4 'Enabled:.*1' | grep 'Repo ID:' | sed -e 's/Repo ID:\\s*\\(.*\\)/\\1/' | sort", "delta": "0:00:01.284394", "end": "2021-02-10 10:55:18.456804", "rc": 0, "start": "2021-02-10 10:55:17.172410", "stderr": "", "stderr_lines": [], "stdout": "rhel-7-server-rpms", "stdout_lines": ["rhel-7-server-rpms"]} 2021-02-10T10:55:18.514 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:55:18 +0000 (0:00:00.423) 0:02:58.815 **** 2021-02-10T10:55:18.551 INFO:teuthology.task.ansible.out: TASK [common : Store list of enabled repos] ************************************ ok: [gibba013.front.sepia.ceph.com] => {"ansible_facts": {"repo_list": ["rhel-7-server-rpms"]}, "changed": false} 2021-02-10T10:55:18.584 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:55:18 +0000 (0:00:00.069) 0:02:58.885 **** 2021-02-10T10:55:18.611 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos false if entitlements are missing or if we unregistered] *** 2021-02-10T10:55:18.612 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:55:18.656 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:55:18 +0000 (0:00:00.072) 0:02:58.957 **** 2021-02-10T10:55:18.696 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if rhsm_repos differs from repo_list] **** 2021-02-10T10:55:18.697 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => {"ansible_facts": {"replace_repos": true}, "changed": false} 2021-02-10T10:55:18.729 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:55:18 +0000 (0:00:00.072) 0:02:59.030 **** 2021-02-10T10:55:18.779 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if newly-subscribed] ********************* 2021-02-10T10:55:18.781 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => {"ansible_facts": {"replace_repos": true}, "changed": false} 2021-02-10T10:55:18.811 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:55:18 +0000 (0:00:00.082) 0:02:59.113 **** 2021-02-10T10:55:20.211 INFO:teuthology.task.ansible.out: TASK [common : Disable all rhsm repos] ***************************************** 2021-02-10T10:55:20.212 INFO:teuthology.task.ansible.out:changed: [gibba002.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} 2021-02-10T10:55:20.251 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:55:20 +0000 (0:00:01.439) 0:03:00.552 **** 2021-02-10T10:55:21.111 INFO:teuthology.task.ansible.out:changed: [gibba013.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} 2021-02-10T10:55:21.155 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:55:21 +0000 (0:00:00.904) 0:03:01.457 **** 2021-02-10T10:55:22.842 INFO:teuthology.task.ansible.out: TASK [common : Enable necessary rhsm repos] ************************************ 2021-02-10T10:55:22.843 INFO:teuthology.task.ansible.out:changed: [gibba002.front.sepia.ceph.com] => {"changed": true, "cmd": ["subscription-manager", "repos", "--enable=rhel-7-server-rpms", "--enable=rhel-7-server-optional-rpms", "--enable=rhel-7-server-extras-rpms", "--enable=rhel-ha-for-rhel-7-server-rpms"], "delta": "0:00:02.135318", "end": "2021-02-10 10:55:22.821118", "rc": 0, "start": "2021-02-10 10:55:20.685800", "stderr": "", "stderr_lines": [], "stdout": "Repository 'rhel-7-server-rpms' is enabled for this system.\nRepository 'rhel-7-server-optional-rpms' is enabled for this system.\nRepository 'rhel-ha-for-rhel-7-server-rpms' is enabled for this system.\nRepository 'rhel-7-server-extras-rpms' is enabled for this system.", "stdout_lines": ["Repository 'rhel-7-server-rpms' is enabled for this system.", "Repository 'rhel-7-server-optional-rpms' is enabled for this system.", "Repository 'rhel-ha-for-rhel-7-server-rpms' is enabled for this system.", "Repository 'rhel-7-server-extras-rpms' is enabled for this system."]} 2021-02-10T10:55:22.884 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:55:22 +0000 (0:00:01.728) 0:03:03.186 **** 2021-02-10T10:55:23.359 INFO:teuthology.task.ansible.out: TASK [common : Remove old apt-mirror repository definition.] ******************* 2021-02-10T10:55:23.361 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => {"changed": false, "path": "/etc/yum.repos.d/cd.repo", "state": "absent"} 2021-02-10T10:55:23.395 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:55:23 +0000 (0:00:00.511) 0:03:03.697 **** 2021-02-10T10:55:23.645 INFO:teuthology.task.ansible.out: TASK [common : Enable necessary rhsm repos] ************************************ 2021-02-10T10:55:23.646 INFO:teuthology.task.ansible.out:changed: [gibba013.front.sepia.ceph.com] => {"changed": true, "cmd": ["subscription-manager", "repos", "--enable=rhel-7-server-rpms", "--enable=rhel-7-server-optional-rpms", "--enable=rhel-7-server-extras-rpms", "--enable=rhel-ha-for-rhel-7-server-rpms"], "delta": "0:00:02.028015", "end": "2021-02-10 10:55:23.632748", "rc": 0, "start": "2021-02-10 10:55:21.604733", "stderr": "", "stderr_lines": [], "stdout": "Repository 'rhel-7-server-rpms' is enabled for this system.\nRepository 'rhel-7-server-optional-rpms' is enabled for this system.\nRepository 'rhel-ha-for-rhel-7-server-rpms' is enabled for this system.\nRepository 'rhel-7-server-extras-rpms' is enabled for this system.", "stdout_lines": ["Repository 'rhel-7-server-rpms' is enabled for this system.", "Repository 'rhel-7-server-optional-rpms' is enabled for this system.", "Repository 'rhel-ha-for-rhel-7-server-rpms' is enabled for this system.", "Repository 'rhel-7-server-extras-rpms' is enabled for this system."]} 2021-02-10T10:55:23.681 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:55:23 +0000 (0:00:00.285) 0:03:03.982 **** 2021-02-10T10:55:23.855 INFO:teuthology.task.ansible.out: TASK [common : Increase the yum timeout.] ************************************** 2021-02-10T10:55:23.856 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-02-10T10:55:23.899 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:55:23 +0000 (0:00:00.218) 0:03:04.201 **** 2021-02-10T10:55:24.149 INFO:teuthology.task.ansible.out: TASK [common : Remove old apt-mirror repository definition.] ******************* 2021-02-10T10:55:24.151 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => {"changed": false, "path": "/etc/yum.repos.d/cd.repo", "state": "absent"} 2021-02-10T10:55:24.184 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:55:24 +0000 (0:00:00.285) 0:03:04.486 **** 2021-02-10T10:55:24.646 INFO:teuthology.task.ansible.out: TASK [common : Increase the yum timeout.] ************************************** 2021-02-10T10:55:24.648 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-02-10T10:55:24.690 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:55:24 +0000 (0:00:00.506) 0:03:04.992 **** 2021-02-10T10:55:25.841 INFO:teuthology.task.ansible.out: TASK [common : Configure epel repos in /etc/yum.repos.d/] ********************** 2021-02-10T10:55:25.842 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'key': 'epel', 'value': {'name': 'Extra Packages for Enterprise Linux', 'mirrorlist': 'file:///etc/yum.repos.d/epel-mirrorlist', 'failovermethod': 'priority', 'enabled': '1', 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "86c0d58bde7ad4cfaafe592d2fefd92c920c42ca", "dest": "/etc/yum.repos.d/epel.repo", "gid": 0, "group": "root", "item": {"key": "epel", "value": {"enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 236, "state": "file", "uid": 0} 2021-02-10T10:55:26.637 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'key': 'epel', 'value': {'name': 'Extra Packages for Enterprise Linux', 'mirrorlist': 'file:///etc/yum.repos.d/epel-mirrorlist', 'failovermethod': 'priority', 'enabled': '1', 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "86c0d58bde7ad4cfaafe592d2fefd92c920c42ca", "dest": "/etc/yum.repos.d/epel.repo", "gid": 0, "group": "root", "item": {"key": "epel", "value": {"enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 236, "state": "file", "uid": 0} 2021-02-10T10:55:27.749 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'key': 'epel-testing', 'value': {'name': 'Extra Packages for Enterprise Linux - Testing', 'mirrorlist': 'file:///etc/yum.repos.d/epel-testing-mirrorlist', 'failovermethod': 'priority', 'enabled': 0, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "a1bad5509ed35b895c2061841236b1cfe10619ae", "dest": "/etc/yum.repos.d/epel-testing.repo", "gid": 0, "group": "root", "item": {"key": "epel-testing", "value": {"enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-testing.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 262, "state": "file", "uid": 0} 2021-02-10T10:55:27.785 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:55:27 +0000 (0:00:03.094) 0:03:08.086 **** 2021-02-10T10:55:28.544 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'key': 'epel-testing', 'value': {'name': 'Extra Packages for Enterprise Linux - Testing', 'mirrorlist': 'file:///etc/yum.repos.d/epel-testing-mirrorlist', 'failovermethod': 'priority', 'enabled': 0, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "a1bad5509ed35b895c2061841236b1cfe10619ae", "dest": "/etc/yum.repos.d/epel-testing.repo", "gid": 0, "group": "root", "item": {"key": "epel-testing", "value": {"enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-testing.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 262, "state": "file", "uid": 0} 2021-02-10T10:55:28.584 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:55:28 +0000 (0:00:00.799) 0:03:08.885 **** 2021-02-10T10:55:29.720 INFO:teuthology.task.ansible.out: TASK [common : Configure local epel mirrorlists] ******************************* 2021-02-10T10:55:29.721 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item=epel-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "73c387451a4fa299c9cb53e25b182ae50e8b8563", "dest": "/etc/yum.repos.d/epel-mirrorlist", "gid": 0, "group": "root", "item": "epel-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 418, "state": "file", "uid": 0} 2021-02-10T10:55:30.521 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item=epel-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "73c387451a4fa299c9cb53e25b182ae50e8b8563", "dest": "/etc/yum.repos.d/epel-mirrorlist", "gid": 0, "group": "root", "item": "epel-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 418, "state": "file", "uid": 0} 2021-02-10T10:55:31.645 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item=epel-testing-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d25e11bf71dfd21a39ace708c03408897a4f263", "dest": "/etc/yum.repos.d/epel-testing-mirrorlist", "gid": 0, "group": "root", "item": "epel-testing-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-testing-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 474, "state": "file", "uid": 0} 2021-02-10T10:55:31.682 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:55:31 +0000 (0:00:03.097) 0:03:11.983 **** 2021-02-10T10:55:31.720 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2021-02-10T10:55:31.722 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:55:31.759 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:55:31 +0000 (0:00:00.077) 0:03:12.061 **** 2021-02-10T10:55:31.788 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2021-02-10T10:55:31.789 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:55:31.823 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:55:31 +0000 (0:00:00.063) 0:03:12.125 **** 2021-02-10T10:55:31.850 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone in /etc/timezone] ****************************** 2021-02-10T10:55:31.852 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:55:31.885 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:55:31 +0000 (0:00:00.061) 0:03:12.186 **** 2021-02-10T10:55:31.912 INFO:teuthology.task.ansible.out: TASK [common : Inform the OS of the new timezone] ****************************** 2021-02-10T10:55:31.914 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:55:31.947 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:55:31 +0000 (0:00:00.062) 0:03:12.248 **** 2021-02-10T10:55:31.996 INFO:teuthology.task.ansible.out: TASK [common : Mask sleep units] *********************************************** 2021-02-10T10:55:31.997 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => (item=sleep.target) => {"ansible_loop_var": "item", "changed": false, "item": "sleep.target", "skip_reason": "Conditional result was False"} 2021-02-10T10:55:32.007 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => (item=suspend.target) => {"ansible_loop_var": "item", "changed": false, "item": "suspend.target", "skip_reason": "Conditional result was False"} 2021-02-10T10:55:32.008 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => (item=hibernate.target) => {"ansible_loop_var": "item", "changed": false, "item": "hibernate.target", "skip_reason": "Conditional result was False"} 2021-02-10T10:55:32.019 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => (item=hybrid-sleep.target) => {"ansible_loop_var": "item", "changed": false, "item": "hybrid-sleep.target", "skip_reason": "Conditional result was False"} 2021-02-10T10:55:32.054 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:55:32 +0000 (0:00:00.106) 0:03:12.355 **** 2021-02-10T10:55:32.081 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** skipping: [gibba002.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:55:32.115 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:55:32 +0000 (0:00:00.060) 0:03:12.416 **** 2021-02-10T10:55:32.152 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone] *********************************************** 2021-02-10T10:55:32.154 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:55:32.184 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:55:32 +0000 (0:00:00.069) 0:03:12.486 **** 2021-02-10T10:55:32.212 INFO:teuthology.task.ansible.out: TASK [common : Add base OpenSUSE Leap repo] ************************************ 2021-02-10T10:55:32.213 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:55:32.245 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:55:32 +0000 (0:00:00.060) 0:03:12.547 **** 2021-02-10T10:55:32.272 INFO:teuthology.task.ansible.out: TASK [common : Add updates OpenSUSE Leap repo] ********************************* 2021-02-10T10:55:32.273 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:55:32.318 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:55:32 +0000 (0:00:00.072) 0:03:12.619 **** 2021-02-10T10:55:32.347 INFO:teuthology.task.ansible.out: TASK [common : Refresh repos] ************************************************** 2021-02-10T10:55:32.348 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:55:32.391 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:55:32 +0000 (0:00:00.072) 0:03:12.692 **** 2021-02-10T10:55:32.442 INFO:teuthology.task.ansible.out: TASK [common : Configure local epel mirrorlists] ******************************* 2021-02-10T10:55:32.443 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item=epel-testing-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d25e11bf71dfd21a39ace708c03408897a4f263", "dest": "/etc/yum.repos.d/epel-testing-mirrorlist", "gid": 0, "group": "root", "item": "epel-testing-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-testing-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 474, "state": "file", "uid": 0} 2021-02-10T10:55:32.479 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:55:32 +0000 (0:00:00.088) 0:03:12.780 **** 2021-02-10T10:55:32.521 INFO:teuthology.task.ansible.out: 2021-02-10T10:55:32.523 INFO:teuthology.task.ansible.out:TASK [common : Clean yum cache] ************************************************ skipping: [gibba013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:55:32.557 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:55:32 +0000 (0:00:00.078) 0:03:12.859 **** 2021-02-10T10:55:32.586 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2021-02-10T10:55:32.587 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:55:32.633 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:55:32 +0000 (0:00:00.075) 0:03:12.935 **** 2021-02-10T10:55:32.663 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone in /etc/timezone] ****************************** 2021-02-10T10:55:32.664 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:55:32.702 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:55:32 +0000 (0:00:00.068) 0:03:13.003 **** 2021-02-10T10:55:32.730 INFO:teuthology.task.ansible.out: TASK [common : Inform the OS of the new timezone] ****************************** 2021-02-10T10:55:32.733 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:55:32.765 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:55:32 +0000 (0:00:00.063) 0:03:13.067 **** 2021-02-10T10:55:32.793 INFO:teuthology.task.ansible.out: TASK [common : Mask sleep units] *********************************************** 2021-02-10T10:55:32.794 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => (item=sleep.target) => {"ansible_loop_var": "item", "changed": false, "item": "sleep.target", "skip_reason": "Conditional result was False"} 2021-02-10T10:55:32.804 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => (item=suspend.target) => {"ansible_loop_var": "item", "changed": false, "item": "suspend.target", "skip_reason": "Conditional result was False"} 2021-02-10T10:55:32.805 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => (item=hibernate.target) => {"ansible_loop_var": "item", "changed": false, "item": "hibernate.target", "skip_reason": "Conditional result was False"} 2021-02-10T10:55:32.806 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => (item=hybrid-sleep.target) => {"ansible_loop_var": "item", "changed": false, "item": "hybrid-sleep.target", "skip_reason": "Conditional result was False"} 2021-02-10T10:55:32.855 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:55:32 +0000 (0:00:00.089) 0:03:13.157 **** 2021-02-10T10:55:32.885 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2021-02-10T10:55:32.887 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:55:32.922 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:55:32 +0000 (0:00:00.066) 0:03:13.224 **** 2021-02-10T10:55:32.949 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone] *********************************************** 2021-02-10T10:55:32.950 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:55:32.983 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:55:32 +0000 (0:00:00.061) 0:03:13.285 **** 2021-02-10T10:55:33.010 INFO:teuthology.task.ansible.out: TASK [common : Add base OpenSUSE Leap repo] ************************************ 2021-02-10T10:55:33.012 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:55:33.043 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:55:33 +0000 (0:00:00.059) 0:03:13.345 **** 2021-02-10T10:55:33.081 INFO:teuthology.task.ansible.out: TASK [common : Add updates OpenSUSE Leap repo] ********************************* skipping: [gibba013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:55:33.118 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:55:33 +0000 (0:00:00.074) 0:03:13.420 **** 2021-02-10T10:55:33.144 INFO:teuthology.task.ansible.out: TASK [common : Refresh repos] ************************************************** skipping: [gibba013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:55:33.181 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:55:33 +0000 (0:00:00.062) 0:03:13.483 **** 2021-02-10T10:55:34.516 INFO:teuthology.task.ansible.out: TASK [common : Set the hardware clock] ***************************************** 2021-02-10T10:55:34.517 INFO:teuthology.task.ansible.out:changed: [gibba013.front.sepia.ceph.com] => {"changed": true, "cmd": ["hwclock", "--systohc"], "delta": "0:00:00.896485", "end": "2021-02-10 10:55:34.500648", "rc": 0, "start": "2021-02-10 10:55:33.604163", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} changed: [gibba002.front.sepia.ceph.com] => {"changed": true, "cmd": ["hwclock", "--systohc"], "delta": "0:00:01.690947", "end": "2021-02-10 10:55:34.500608", "rc": 0, "start": "2021-02-10 10:55:32.809661", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-10T10:55:34.558 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:55:34 +0000 (0:00:01.377) 0:03:14.860 **** 2021-02-10T10:55:34.596 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:55:34 +0000 (0:00:00.037) 0:03:14.897 **** 2021-02-10T10:55:35.391 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (RedHat)] ***************************** 2021-02-10T10:55:35.393 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => {"changed": false, "msg": "", "rc": 0, "results": ["krb5-workstation-1.15.1-50.el7.x86_64 providing krb5-workstation is already installed"]} 2021-02-10T10:55:35.427 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:55:35 +0000 (0:00:00.830) 0:03:15.728 **** 2021-02-10T10:55:35.434 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => {"changed": false, "msg": "", "rc": 0, "results": ["krb5-workstation-1.15.1-50.el7.x86_64 providing krb5-workstation is already installed"]} 2021-02-10T10:55:35.470 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:55:35 +0000 (0:00:00.043) 0:03:15.772 **** 2021-02-10T10:55:35.476 INFO:teuthology.task.ansible.out: TASK [common : Clean apt cache] ************************************************ 2021-02-10T10:55:35.477 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:55:35.510 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:55:35 +0000 (0:00:00.040) 0:03:15.812 **** 2021-02-10T10:55:35.517 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:55:35.551 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:55:35 +0000 (0:00:00.041) 0:03:15.853 **** 2021-02-10T10:55:35.558 INFO:teuthology.task.ansible.out: TASK [common : Update apt cache] *********************************************** 2021-02-10T10:55:35.559 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:55:35.593 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:55:35 +0000 (0:00:00.041) 0:03:15.895 **** 2021-02-10T10:55:35.600 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:55:35.633 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:55:35 +0000 (0:00:00.040) 0:03:15.935 **** 2021-02-10T10:55:35.640 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (Debian)] ***************************** 2021-02-10T10:55:35.642 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:55:35.680 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:55:35 +0000 (0:00:00.046) 0:03:15.981 **** 2021-02-10T10:55:35.691 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:55:35.729 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:55:35 +0000 (0:00:00.049) 0:03:16.031 **** 2021-02-10T10:55:35.866 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (OpenSUSE Leap)] ********************** skipping: [gibba002.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:55:35.903 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:55:35 +0000 (0:00:00.173) 0:03:16.204 **** 2021-02-10T10:55:35.911 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:55:35.946 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:55:35 +0000 (0:00:00.043) 0:03:16.248 **** 2021-02-10T10:55:37.886 INFO:teuthology.task.ansible.out: TASK [common : Add krb5 config file] ******************************************* 2021-02-10T10:55:37.887 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => {"changed": false, "checksum": "b3c1a33d19a4e448c29374429629dff57657b576", "dest": "/etc/krb5.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/krb5.conf", "secontext": "system_u:object_r:krb5_conf_t:s0", "size": 131, "state": "file", "uid": 0} 2021-02-10T10:55:37.921 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:55:37 +0000 (0:00:01.974) 0:03:18.222 **** 2021-02-10T10:55:37.939 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => {"changed": false, "checksum": "b3c1a33d19a4e448c29374429629dff57657b576", "dest": "/etc/krb5.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/krb5.conf", "secontext": "system_u:object_r:krb5_conf_t:s0", "size": 131, "state": "file", "uid": 0} 2021-02-10T10:55:37.973 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:55:37 +0000 (0:00:00.052) 0:03:18.275 **** 2021-02-10T10:56:05.512 INFO:teuthology.task.ansible.out: TASK [common : Make sure smartmontools is installed] *************************** 2021-02-10T10:56:05.513 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => {"changed": false, "changes": {"installed": [], "updated": []}, "msg": "", "rc": 0, "results": ["All packages providing smartmontools are up to date", ""]} ok: [gibba013.front.sepia.ceph.com] => {"changed": false, "changes": {"installed": [], "updated": []}, "msg": "", "rc": 0, "results": ["All packages providing smartmontools are up to date", ""]} 2021-02-10T10:56:05.558 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:56:05 +0000 (0:00:27.584) 0:03:45.859 **** 2021-02-10T10:56:05.596 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:56:05 +0000 (0:00:00.037) 0:03:45.897 **** 2021-02-10T10:56:07.542 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2021-02-10T10:56:07.543 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item=megacli) => {"ansible_loop_var": "item", "changed": false, "checksum": "cb3ef2aa6a3e3002b234f166e739a39b1f3a1bde", "dest": "/usr/sbin/megacli", "gid": 0, "group": "root", "item": "megacli", "mode": "0755", "owner": "root", "path": "/usr/sbin/megacli", "secontext": "system_u:object_r:bin_t:s0", "size": 2716736, "state": "file", "uid": 0} 2021-02-10T10:56:07.544 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item=megacli) => {"ansible_loop_var": "item", "changed": false, "checksum": "cb3ef2aa6a3e3002b234f166e739a39b1f3a1bde", "dest": "/usr/sbin/megacli", "gid": 0, "group": "root", "item": "megacli", "mode": "0755", "owner": "root", "path": "/usr/sbin/megacli", "secontext": "system_u:object_r:bin_t:s0", "size": 2716736, "state": "file", "uid": 0} 2021-02-10T10:56:09.451 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item=cli64) => {"ansible_loop_var": "item", "changed": false, "checksum": "a3b8773b14e8594ffbd9fc156c09436eb0184bc7", "dest": "/usr/sbin/cli64", "gid": 0, "group": "root", "item": "cli64", "mode": "0755", "owner": "root", "path": "/usr/sbin/cli64", "secontext": "system_u:object_r:bin_t:s0", "size": 1952038, "state": "file", "uid": 0} 2021-02-10T10:56:09.451 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item=cli64) => {"ansible_loop_var": "item", "changed": false, "checksum": "a3b8773b14e8594ffbd9fc156c09436eb0184bc7", "dest": "/usr/sbin/cli64", "gid": 0, "group": "root", "item": "cli64", "mode": "0755", "owner": "root", "path": "/usr/sbin/cli64", "secontext": "system_u:object_r:bin_t:s0", "size": 1952038, "state": "file", "uid": 0} 2021-02-10T10:56:11.337 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item=nvme) => {"ansible_loop_var": "item", "changed": false, "checksum": "98735dcacd6fec55dd92b67281b478603b7955b1", "dest": "/usr/sbin/nvme", "gid": 0, "group": "root", "item": "nvme", "mode": "0755", "owner": "root", "path": "/usr/sbin/nvme", "secontext": "system_u:object_r:bin_t:s0", "size": 580432, "state": "file", "uid": 0} 2021-02-10T10:56:11.340 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item=nvme) => {"ansible_loop_var": "item", "changed": false, "checksum": "98735dcacd6fec55dd92b67281b478603b7955b1", "dest": "/usr/sbin/nvme", "gid": 0, "group": "root", "item": "nvme", "mode": "0755", "owner": "root", "path": "/usr/sbin/nvme", "secontext": "system_u:object_r:bin_t:s0", "size": 580432, "state": "file", "uid": 0} 2021-02-10T10:56:11.376 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:56:11 +0000 (0:00:05.781) 0:03:51.678 **** 2021-02-10T10:56:11.410 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:56:11 +0000 (0:00:00.033) 0:03:51.711 **** 2021-02-10T10:56:11.822 INFO:teuthology.task.ansible.out: TASK [common : Create /usr/libexec.] ******************************************* 2021-02-10T10:56:11.824 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/libexec", "secontext": "system_u:object_r:bin_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-02-10T10:56:11.858 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:56:11 +0000 (0:00:00.448) 0:03:52.160 **** 2021-02-10T10:56:11.865 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/libexec", "secontext": "system_u:object_r:bin_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-02-10T10:56:11.900 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:56:11 +0000 (0:00:00.042) 0:03:52.202 **** 2021-02-10T10:56:14.525 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2021-02-10T10:56:14.527 INFO:teuthology.task.ansible.out:changed: [gibba013.front.sepia.ceph.com] => (item=smart.sh) => {"ansible_loop_var": "item", "changed": true, "checksum": "00d0404b545463be4a65f475e0870e07a48f35b6", "dest": "/usr/libexec/smart.sh", "gid": 0, "group": "root", "item": "smart.sh", "md5sum": "14041bf2666d8f8d277567958c7e9f69", "mode": "0755", "owner": "root", "secontext": "system_u:object_r:bin_t:s0", "size": 13355, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1612954571.926457-17376708006425/source", "state": "file", "uid": 0} 2021-02-10T10:56:14.548 INFO:teuthology.task.ansible.out:changed: [gibba002.front.sepia.ceph.com] => (item=smart.sh) => {"ansible_loop_var": "item", "changed": true, "checksum": "00d0404b545463be4a65f475e0870e07a48f35b6", "dest": "/usr/libexec/smart.sh", "gid": 0, "group": "root", "item": "smart.sh", "md5sum": "14041bf2666d8f8d277567958c7e9f69", "mode": "0755", "owner": "root", "secontext": "system_u:object_r:bin_t:s0", "size": 13355, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1612954571.8842132-3834613643261/source", "state": "file", "uid": 0} 2021-02-10T10:56:16.395 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item=raid.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d9a89805d8a9e4c7281934e887da31f7ac464fb", "dest": "/usr/libexec/raid.pl", "gid": 0, "group": "root", "item": "raid.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/raid.pl", "secontext": "system_u:object_r:bin_t:s0", "size": 8602, "state": "file", "uid": 0} 2021-02-10T10:56:16.427 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item=raid.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d9a89805d8a9e4c7281934e887da31f7ac464fb", "dest": "/usr/libexec/raid.pl", "gid": 0, "group": "root", "item": "raid.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/raid.pl", "secontext": "system_u:object_r:bin_t:s0", "size": 8602, "state": "file", "uid": 0} 2021-02-10T10:56:18.278 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item=diskusage.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9c74425885a83f23bc37817de46979ca5173ccb7", "dest": "/usr/libexec/diskusage.pl", "gid": 0, "group": "root", "item": "diskusage.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/diskusage.pl", "secontext": "system_u:object_r:bin_t:s0", "size": 3605, "state": "file", "uid": 0} 2021-02-10T10:56:18.318 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:56:18 +0000 (0:00:06.417) 0:03:58.619 **** 2021-02-10T10:56:18.324 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item=diskusage.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9c74425885a83f23bc37817de46979ca5173ccb7", "dest": "/usr/libexec/diskusage.pl", "gid": 0, "group": "root", "item": "diskusage.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/diskusage.pl", "secontext": "system_u:object_r:bin_t:s0", "size": 3605, "state": "file", "uid": 0} 2021-02-10T10:56:18.374 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:56:18 +0000 (0:00:00.056) 0:03:58.675 **** 2021-02-10T10:56:18.380 INFO:teuthology.task.ansible.out: TASK [common : Include apt_system vars] **************************************** 2021-02-10T10:56:18.382 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:56:18.415 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:56:18 +0000 (0:00:00.041) 0:03:58.716 **** 2021-02-10T10:56:18.421 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:56:18.457 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:56:18 +0000 (0:00:00.041) 0:03:58.758 **** 2021-02-10T10:56:18.464 INFO:teuthology.task.ansible.out: TASK [common : Include yum_systems vars] *************************************** 2021-02-10T10:56:18.465 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => {"ansible_facts": {"nagios_plugins_directory": "/usr/lib64/nagios/plugins", "nrpe_group": "nrpe", "nrpe_packages": ["nagios-common", "nrpe", "nagios-plugins", "nagios-plugins-load"], "nrpe_service_name": "nrpe", "nrpe_user": "nrpe"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/yum_systems.yml"], "changed": false} 2021-02-10T10:56:18.506 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:56:18 +0000 (0:00:00.048) 0:03:58.807 **** 2021-02-10T10:56:18.517 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => {"ansible_facts": {"nagios_plugins_directory": "/usr/lib64/nagios/plugins", "nrpe_group": "nrpe", "nrpe_packages": ["nagios-common", "nrpe", "nagios-plugins", "nagios-plugins-load"], "nrpe_service_name": "nrpe", "nrpe_user": "nrpe"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/yum_systems.yml"], "changed": false} 2021-02-10T10:56:18.570 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:56:18 +0000 (0:00:00.064) 0:03:58.872 **** 2021-02-10T10:56:18.994 INFO:teuthology.task.ansible.out: TASK [common : Check for epel] ************************************************* 2021-02-10T10:56:18.996 INFO:teuthology.task.ansible.out:changed: [gibba013.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} 2021-02-10T10:56:19.041 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:56:19 +0000 (0:00:00.471) 0:03:59.343 **** 2021-02-10T10:56:19.051 INFO:teuthology.task.ansible.out:changed: [gibba002.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} 2021-02-10T10:56:19.095 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:56:19 +0000 (0:00:00.052) 0:03:59.396 **** 2021-02-10T10:56:19.096 INFO:teuthology.task.ansible.out: 2021-02-10T10:56:19.104 INFO:teuthology.task.ansible.out: 2021-02-10T10:56:19.106 INFO:teuthology.task.ansible.out:TASK [common : Install nrpe without epel] ************************************** skipping: [gibba013.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-02-10T10:56:19.106 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-02-10T10:56:19.107 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-02-10T10:56:19.120 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-02-10T10:56:19.153 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:56:19 +0000 (0:00:00.058) 0:03:59.455 **** 2021-02-10T10:56:19.161 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-02-10T10:56:19.161 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-02-10T10:56:19.163 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-02-10T10:56:19.195 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-02-10T10:56:19.232 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:56:19 +0000 (0:00:00.079) 0:03:59.534 **** 2021-02-10T10:56:24.210 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (RHEL/CentOS)] ************ 2021-02-10T10:56:24.212 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => {"changed": false, "changes": {"installed": [], "updated": []}, "msg": "", "rc": 0, "results": ["All packages providing nagios-common are up to date", "All packages providing nrpe are up to date", "All packages providing nagios-plugins are up to date", "All packages providing nagios-plugins-load are up to date", ""]} 2021-02-10T10:56:24.252 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:56:24 +0000 (0:00:05.019) 0:04:04.553 **** 2021-02-10T10:56:24.279 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (Ubuntu/Fedora)] ********** 2021-02-10T10:56:24.280 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:56:24.325 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:56:24 +0000 (0:00:00.073) 0:04:04.627 **** 2021-02-10T10:56:25.302 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (RHEL/CentOS)] ************ 2021-02-10T10:56:25.303 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => {"changed": false, "changes": {"installed": [], "updated": []}, "msg": "", "rc": 0, "results": ["All packages providing nagios-common are up to date", "All packages providing nrpe are up to date", "All packages providing nagios-plugins are up to date", "All packages providing nagios-plugins-load are up to date", ""]} 2021-02-10T10:56:25.342 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:56:25 +0000 (0:00:01.016) 0:04:05.643 **** 2021-02-10T10:56:25.370 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (Ubuntu/Fedora)] ********** 2021-02-10T10:56:25.371 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:56:25.406 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:56:25 +0000 (0:00:00.064) 0:04:05.707 **** 2021-02-10T10:56:26.273 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios sudoers.d for raid utilities.] ******************** 2021-02-10T10:56:26.274 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => {"changed": false, "checksum": "b423b985ccbb69b65a711d372645a5f704b30e8b", "dest": "/etc/sudoers.d/90-nagios", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/90-nagios", "secontext": "system_u:object_r:etc_t:s0", "size": 178, "state": "file", "uid": 0} 2021-02-10T10:56:26.310 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:56:26 +0000 (0:00:00.903) 0:04:06.611 **** 2021-02-10T10:56:27.330 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => {"changed": false, "checksum": "b423b985ccbb69b65a711d372645a5f704b30e8b", "dest": "/etc/sudoers.d/90-nagios", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/90-nagios", "secontext": "system_u:object_r:etc_t:s0", "size": 178, "state": "file", "uid": 0} 2021-02-10T10:56:27.366 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:56:27 +0000 (0:00:01.055) 0:04:07.667 **** 2021-02-10T10:56:28.242 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios check_mem script] ********************************* 2021-02-10T10:56:28.243 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => {"changed": false, "checksum": "39df629b10ded370443e2e4c84d690332c95104d", "dest": "/usr/lib64/nagios/plugins/check_mem.sh", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib64/nagios/plugins/check_mem.sh", "secontext": "system_u:object_r:nagios_unconfined_plugin_exec_t:s0", "size": 1964, "state": "file", "uid": 0} 2021-02-10T10:56:28.286 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:56:28 +0000 (0:00:00.920) 0:04:08.588 **** 2021-02-10T10:56:28.325 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ 2021-02-10T10:56:28.326 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => (item=DAEMON_OPTS) => {"ansible_loop_var": "item", "changed": false, "item": "DAEMON_OPTS", "skip_reason": "Conditional result was False"} 2021-02-10T10:56:28.336 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => (item=NRPE_OPTS) => {"ansible_loop_var": "item", "changed": false, "item": "NRPE_OPTS", "skip_reason": "Conditional result was False"} 2021-02-10T10:56:28.372 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:56:28 +0000 (0:00:00.086) 0:04:08.674 **** 2021-02-10T10:56:28.840 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (RHEL/CentOS)] ******************* 2021-02-10T10:56:28.842 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-02-10T10:56:28.875 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:56:28 +0000 (0:00:00.503) 0:04:09.177 **** 2021-02-10T10:56:29.290 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios check_mem script] ********************************* 2021-02-10T10:56:29.292 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => {"changed": false, "checksum": "39df629b10ded370443e2e4c84d690332c95104d", "dest": "/usr/lib64/nagios/plugins/check_mem.sh", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib64/nagios/plugins/check_mem.sh", "secontext": "system_u:object_r:nagios_unconfined_plugin_exec_t:s0", "size": 1964, "state": "file", "uid": 0} 2021-02-10T10:56:29.328 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:56:29 +0000 (0:00:00.452) 0:04:09.629 **** 2021-02-10T10:56:29.367 INFO:teuthology.task.ansible.out: TASK [common : Check firewalld status] ***************************************** fatal: [gibba002.front.sepia.ceph.com]: FAILED! => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} ...ignoring TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ skipping: [gibba013.front.sepia.ceph.com] => (item=DAEMON_OPTS) => {"ansible_loop_var": "item", "changed": false, "item": "DAEMON_OPTS", "skip_reason": "Conditional result was False"} 2021-02-10T10:56:29.402 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:56:29 +0000 (0:00:00.074) 0:04:09.704 **** 2021-02-10T10:56:29.412 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => (item=NRPE_OPTS) => {"ansible_loop_var": "item", "changed": false, "item": "NRPE_OPTS", "skip_reason": "Conditional result was False"} 2021-02-10T10:56:29.449 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:56:29 +0000 (0:00:00.047) 0:04:09.751 **** 2021-02-10T10:56:29.455 INFO:teuthology.task.ansible.out: TASK [common : Open nrpe port if firewalld enabled] **************************** 2021-02-10T10:56:29.457 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:56:29.494 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:56:29 +0000 (0:00:00.044) 0:04:09.795 **** 2021-02-10T10:56:29.916 INFO:teuthology.task.ansible.out: 2021-02-10T10:56:29.916 INFO:teuthology.task.ansible.out:TASK [common : Configure nagios nrpe settings (RHEL/CentOS)] ******************* 2021-02-10T10:56:29.917 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-02-10T10:56:29.953 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:56:29 +0000 (0:00:00.458) 0:04:10.254 **** 2021-02-10T10:56:30.439 INFO:teuthology.task.ansible.out: TASK [common : Check firewalld status] ***************************************** fatal: [gibba013.front.sepia.ceph.com]: FAILED! => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} ...ignoring 2021-02-10T10:56:30.474 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:56:30 +0000 (0:00:00.521) 0:04:10.776 **** 2021-02-10T10:56:30.502 INFO:teuthology.task.ansible.out: TASK [common : Open nrpe port if firewalld enabled] **************************** skipping: [gibba013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:56:30.537 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:56:30 +0000 (0:00:00.062) 0:04:10.838 **** 2021-02-10T10:56:31.431 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios nrpe config.] ************************************* 2021-02-10T10:56:31.433 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => {"changed": false, "checksum": "aa9150d4582f919d7a8504c01c8d86f05e3eeb5d", "dest": "/etc/nagios/nrpe.cfg", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/nagios/nrpe.cfg", "secontext": "system_u:object_r:nrpe_etc_t:s0", "size": 1184, "state": "file", "uid": 0} 2021-02-10T10:56:31.469 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:56:31 +0000 (0:00:00.932) 0:04:11.771 **** 2021-02-10T10:56:32.464 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => {"changed": false, "checksum": "aa9150d4582f919d7a8504c01c8d86f05e3eeb5d", "dest": "/etc/nagios/nrpe.cfg", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/nagios/nrpe.cfg", "secontext": "system_u:object_r:nrpe_etc_t:s0", "size": 1184, "state": "file", "uid": 0} 2021-02-10T10:56:32.500 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:56:32 +0000 (0:00:01.030) 0:04:12.802 **** 2021-02-10T10:56:32.510 INFO:teuthology.task.ansible.out: TASK [common : Make sure nagios nrpe service is running.] ********************** 2021-02-10T10:56:32.511 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "nrpe", "state": "started", "status": {"ActiveEnterTimestamp": "Wed 2021-02-10 10:51:12 UTC", "ActiveEnterTimestampMonotonic": "19864667", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "local-fs.target nss-lookup.target var-run.mount systemd-journald.socket system.slice network-online.target basic.target time-sync.target network.target", "AllowIsolate": "no", "AmbientCapabilities": "0", "AssertResult": "yes", "AssertTimestamp": "Wed 2021-02-10 10:51:12 UTC", "AssertTimestampMonotonic": "19851755", "Before": "xdm.service shutdown.target multi-user.target getty@tty1.service", "BlockIOAccounting": "no", "BlockIOWeight": "18446744073709551615", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "18446744073709551615", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "18446744073709551615", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Wed 2021-02-10 10:51:12 UTC", "ConditionTimestampMonotonic": "19851752", "Conflicts": "nrpe.socket shutdown.target", "ControlGroup": "/system.slice/nrpe.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Nagios Remote Program Executor", "DevicePolicy": "auto", "Documentation": "http://www.nagios.org/documentation", "EnvironmentFile": "/etc/sysconfig/nrpe (ignore_errors=no)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1191", "ExecMainStartTimestamp": "Wed 2021-02-10 10:51:12 UTC", "ExecMainStartTimestampMonotonic": "19864060", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c /e 2021-02-10T10:56:32.513 INFO:teuthology.task.ansible.out:tc/nagios/nrpe.cfg -f $NRPE_SSL_OPT ; ignore_errors=no ; start_time=[Wed 2021-02-10 10:51:12 UTC] ; stop_time=[n/a] ; pid=1191 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/bin/rm ; argv[]=/bin/rm -f /var/run/nrpe/nrpe.pid ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/nrpe.service", "Group": "nrpe", "GuessMainPID": "yes", "IOScheduling": "0", "Id": "nrpe.service", "IgnoreOnIsolate": "no", "IgnoreOnSnapshot": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Wed 2021-02-10 10:51:12 UTC", "InactiveExitTimestampMonotonic": "19864667", "JobTimeoutAction": "none", "JobTimeoutUSec": "0", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "18446744073709551615", "LimitCORE": "18446744073709551615", "LimitCPU": "18446744073709551615", "LimitDATA": "18446744073709551615", "LimitFSIZE": "18446744073709551615", "LimitLOCKS": "18446744073709551615", "LimitMEMLOCK": "65536", "LimitMSGQUEUE": "819200", "LimitNICE": "0", "LimitNOFILE": "4096", "LimitNPROC": "127560", "LimitRSS": "18446744073709551615", "LimitRTPRIO": "0", "LimitRTTIME": "18446744073709551615", "LimitSIGPENDING": "127560", "LimitSTACK": "18446744073709551615", "LoadState": "loaded", "MainPID": "1191", "MemoryAccounting": "no", "MemoryCurrent": "18446744073709551615", "MemoryLimit": "18446744073709551615", "MountFlags": "0", "Names": "nrpe.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "no", "ProtectHome": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "Requires": "basic.target system.slice", "Restart": "no", "RestartUSec": "100ms", "Re 2021-02-10T10:56:32.514 INFO:teuthology.task.ansible.out:sult": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitInterval": "10000000", "StartupBlockIOWeight": "18446744073709551615", "StartupCPUShares": "18446744073709551615", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "no", "TasksCurrent": "18446744073709551615", "TasksMax": "18446744073709551615", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "simple", "UMask": "0022", "UnitFilePreset": "disabled", "UnitFileState": "enabled", "User": "nrpe", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Wed 2021-02-10 10:51:12 UTC", "WatchdogTimestampMonotonic": "19864635", "WatchdogUSec": "0"}} 2021-02-10T10:56:32.547 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:56:32 +0000 (0:00:00.047) 0:04:12.849 **** 2021-02-10T10:56:32.994 INFO:teuthology.task.ansible.out: TASK [common : Get SELinux status] ********************************************* 2021-02-10T10:56:32.996 INFO:teuthology.task.ansible.out:changed: [gibba002.front.sepia.ceph.com] => {"changed": true, "cmd": ["getenforce"], "delta": "0:00:00.001342", "end": "2021-02-10 10:56:32.973070", "rc": 0, "start": "2021-02-10 10:56:32.971728", "stderr": "", "stderr_lines": [], "stdout": "Permissive", "stdout_lines": ["Permissive"]} 2021-02-10T10:56:33.032 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:56:33 +0000 (0:00:00.484) 0:04:13.334 **** 2021-02-10T10:56:33.040 INFO:teuthology.task.ansible.out: TASK [common : Make sure nagios nrpe service is running.] ********************** 2021-02-10T10:56:33.041 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "nrpe", "state": "started", "status": {"ActiveEnterTimestamp": "Wed 2021-02-10 10:51:13 UTC", "ActiveEnterTimestampMonotonic": "19039866", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "basic.target nss-lookup.target var-run.mount system.slice network-online.target local-fs.target time-sync.target systemd-journald.socket network.target", "AllowIsolate": "no", "AmbientCapabilities": "0", "AssertResult": "yes", "AssertTimestamp": "Wed 2021-02-10 10:51:13 UTC", "AssertTimestampMonotonic": "19013153", "Before": "multi-user.target shutdown.target xdm.service getty@tty1.service", "BlockIOAccounting": "no", "BlockIOWeight": "18446744073709551615", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "18446744073709551615", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "18446744073709551615", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Wed 2021-02-10 10:51:13 UTC", "ConditionTimestampMonotonic": "19013151", "Conflicts": "shutdown.target nrpe.socket", "ControlGroup": "/system.slice/nrpe.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Nagios Remote Program Executor", "DevicePolicy": "auto", "Documentation": "http://www.nagios.org/documentation", "EnvironmentFile": "/etc/sysconfig/nrpe (ignore_errors=no)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1191", "ExecMainStartTimestamp": "Wed 2021-02-10 10:51:13 UTC", "ExecMainStartTimestampMonotonic": "19039719", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c /e 2021-02-10T10:56:33.042 INFO:teuthology.task.ansible.out:tc/nagios/nrpe.cfg -f $NRPE_SSL_OPT ; ignore_errors=no ; start_time=[Wed 2021-02-10 10:51:13 UTC] ; stop_time=[n/a] ; pid=1191 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/bin/rm ; argv[]=/bin/rm -f /var/run/nrpe/nrpe.pid ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/nrpe.service", "Group": "nrpe", "GuessMainPID": "yes", "IOScheduling": "0", "Id": "nrpe.service", "IgnoreOnIsolate": "no", "IgnoreOnSnapshot": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Wed 2021-02-10 10:51:13 UTC", "InactiveExitTimestampMonotonic": "19039866", "JobTimeoutAction": "none", "JobTimeoutUSec": "0", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "18446744073709551615", "LimitCORE": "18446744073709551615", "LimitCPU": "18446744073709551615", "LimitDATA": "18446744073709551615", "LimitFSIZE": "18446744073709551615", "LimitLOCKS": "18446744073709551615", "LimitMEMLOCK": "65536", "LimitMSGQUEUE": "819200", "LimitNICE": "0", "LimitNOFILE": "4096", "LimitNPROC": "127560", "LimitRSS": "18446744073709551615", "LimitRTPRIO": "0", "LimitRTTIME": "18446744073709551615", "LimitSIGPENDING": "127560", "LimitSTACK": "18446744073709551615", "LoadState": "loaded", "MainPID": "1191", "MemoryAccounting": "no", "MemoryCurrent": "18446744073709551615", "MemoryLimit": "18446744073709551615", "MountFlags": "0", "Names": "nrpe.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "no", "ProtectHome": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "Requires": "basic.target system.slice", "Restart": "no", "RestartUSec": "100ms", "Re 2021-02-10T10:56:33.044 INFO:teuthology.task.ansible.out:sult": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitInterval": "10000000", "StartupBlockIOWeight": "18446744073709551615", "StartupCPUShares": "18446744073709551615", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "no", "TasksCurrent": "18446744073709551615", "TasksMax": "18446744073709551615", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "simple", "UMask": "0022", "UnitFilePreset": "disabled", "UnitFileState": "enabled", "User": "nrpe", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Wed 2021-02-10 10:51:13 UTC", "WatchdogTimestampMonotonic": "19039806", "WatchdogUSec": "0"}} 2021-02-10T10:56:33.085 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:56:33 +0000 (0:00:00.052) 0:04:13.386 **** 2021-02-10T10:56:33.532 INFO:teuthology.task.ansible.out: TASK [common : Get SELinux status] ********************************************* 2021-02-10T10:56:33.534 INFO:teuthology.task.ansible.out:changed: [gibba013.front.sepia.ceph.com] => {"changed": true, "cmd": ["getenforce"], "delta": "0:00:00.001373", "end": "2021-02-10 10:56:33.520850", "rc": 0, "start": "2021-02-10 10:56:33.519477", "stderr": "", "stderr_lines": [], "stdout": "Permissive", "stdout_lines": ["Permissive"]} 2021-02-10T10:56:33.565 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:56:33 +0000 (0:00:00.480) 0:04:13.867 **** 2021-02-10T10:56:33.866 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Install SELinux dependencies] **************************** 2021-02-10T10:56:33.867 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => {"changed": false, "msg": "", "rc": 0, "results": ["libsemanage-python-2.5-14.el7.x86_64 providing libsemanage-python is already installed", "policycoreutils-python-2.5-34.el7.x86_64 providing policycoreutils-python is already installed"]} 2021-02-10T10:56:33.908 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:56:33 +0000 (0:00:00.342) 0:04:14.209 **** 2021-02-10T10:56:34.376 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => {"changed": false, "msg": "", "rc": 0, "results": ["libsemanage-python-2.5-14.el7.x86_64 providing libsemanage-python is already installed", "policycoreutils-python-2.5-34.el7.x86_64 providing policycoreutils-python is already installed"]} 2021-02-10T10:56:34.412 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:56:34 +0000 (0:00:00.503) 0:04:14.713 **** 2021-02-10T10:56:38.507 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Ensure SELinux policy is up to date] ********************* 2021-02-10T10:56:38.508 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => {"changed": false, "changes": {"installed": [], "updated": []}, "msg": "", "rc": 0, "results": ["All packages providing selinux-policy-targeted are up to date", ""]} 2021-02-10T10:56:38.558 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:56:38 +0000 (0:00:04.145) 0:04:18.859 **** 2021-02-10T10:56:38.908 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => {"changed": false, "changes": {"installed": [], "updated": []}, "msg": "", "rc": 0, "results": ["All packages providing selinux-policy-targeted are up to date", ""]} 2021-02-10T10:56:38.943 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:56:38 +0000 (0:00:00.385) 0:04:19.244 **** 2021-02-10T10:56:39.572 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Set SELinux boolean nagios_run_sudo true] **************** 2021-02-10T10:56:39.574 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => {"changed": false, "name": "nagios_run_sudo", "persistent": true, "state": true} 2021-02-10T10:56:39.607 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:56:39 +0000 (0:00:00.663) 0:04:19.908 **** 2021-02-10T10:56:39.791 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => {"changed": false, "name": "nagios_run_sudo", "persistent": true, "state": true} 2021-02-10T10:56:39.826 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:56:39 +0000 (0:00:00.218) 0:04:20.127 **** 2021-02-10T10:56:45.681 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove SELinux policy package] *************************** 2021-02-10T10:56:45.683 INFO:teuthology.task.ansible.out:changed: [gibba002.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-r", "nrpe"], "delta": "0:00:05.619638", "end": "2021-02-10 10:56:45.661658", "failed_when_result": false, "rc": 0, "start": "2021-02-10 10:56:40.042020", "stderr": "", "stderr_lines": [], "stdout": "libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority).", "stdout_lines": ["libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority)."]} 2021-02-10T10:56:45.717 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:56:45 +0000 (0:00:05.891) 0:04:26.019 **** 2021-02-10T10:56:45.831 INFO:teuthology.task.ansible.out:changed: [gibba013.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-r", "nrpe"], "delta": "0:00:05.559058", "end": "2021-02-10 10:56:45.814131", "failed_when_result": false, "rc": 0, "start": "2021-02-10 10:56:40.255073", "stderr": "", "stderr_lines": [], "stdout": "libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority).", "stdout_lines": ["libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority)."]} 2021-02-10T10:56:45.833 INFO:teuthology.task.ansible.out: 2021-02-10T10:56:45.869 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:56:45 +0000 (0:00:00.151) 0:04:26.171 **** 2021-02-10T10:56:47.666 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Copy SELinux type enforcement file] ********************** 2021-02-10T10:56:47.667 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => {"changed": false, "checksum": "99d9b2ea41fb2d6613898129718064833d2de428", "dest": "/tmp/nrpe.te", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/tmp/nrpe.te", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 934, "state": "file", "uid": 0} 2021-02-10T10:56:47.703 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:56:47 +0000 (0:00:01.833) 0:04:28.004 **** 2021-02-10T10:56:47.806 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => {"changed": false, "checksum": "99d9b2ea41fb2d6613898129718064833d2de428", "dest": "/tmp/nrpe.te", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/tmp/nrpe.te", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 934, "state": "file", "uid": 0} 2021-02-10T10:56:47.848 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:56:47 +0000 (0:00:00.145) 0:04:28.150 **** 2021-02-10T10:56:48.171 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Compile SELinux module file] ***************************** 2021-02-10T10:56:48.173 INFO:teuthology.task.ansible.out:changed: [gibba002.front.sepia.ceph.com] => {"changed": true, "cmd": ["checkmodule", "-M", "-m", "-o", "/tmp/nrpe.mod", "/tmp/nrpe.te"], "delta": "0:00:00.015522", "end": "2021-02-10 10:56:48.155823", "rc": 0, "start": "2021-02-10 10:56:48.140301", "stderr": "", "stderr_lines": [], "stdout": "checkmodule: loading policy configuration from /tmp/nrpe.te\ncheckmodule: policy configuration loaded\ncheckmodule: writing binary representation (version 19) to /tmp/nrpe.mod", "stdout_lines": ["checkmodule: loading policy configuration from /tmp/nrpe.te", "checkmodule: policy configuration loaded", "checkmodule: writing binary representation (version 19) to /tmp/nrpe.mod"]} 2021-02-10T10:56:48.206 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:56:48 +0000 (0:00:00.357) 0:04:28.507 **** 2021-02-10T10:56:48.327 INFO:teuthology.task.ansible.out:changed: [gibba013.front.sepia.ceph.com] => {"changed": true, "cmd": ["checkmodule", "-M", "-m", "-o", "/tmp/nrpe.mod", "/tmp/nrpe.te"], "delta": "0:00:00.012493", "end": "2021-02-10 10:56:48.306967", "rc": 0, "start": "2021-02-10 10:56:48.294474", "stderr": "", "stderr_lines": [], "stdout": "checkmodule: loading policy configuration from /tmp/nrpe.te\ncheckmodule: policy configuration loaded\ncheckmodule: writing binary representation (version 19) to /tmp/nrpe.mod", "stdout_lines": ["checkmodule: loading policy configuration from /tmp/nrpe.te", "checkmodule: policy configuration loaded", "checkmodule: writing binary representation (version 19) to /tmp/nrpe.mod"]} 2021-02-10T10:56:48.362 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:56:48 +0000 (0:00:00.156) 0:04:28.664 **** 2021-02-10T10:56:48.662 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Build SELinux policy package] **************************** 2021-02-10T10:56:48.663 INFO:teuthology.task.ansible.out:changed: [gibba002.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule_package", "-o", "/tmp/nrpe.pp", "-m", "/tmp/nrpe.mod"], "delta": "0:00:00.013191", "end": "2021-02-10 10:56:48.649500", "rc": 0, "start": "2021-02-10 10:56:48.636309", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-10T10:56:48.703 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:56:48 +0000 (0:00:00.341) 0:04:29.005 **** 2021-02-10T10:56:48.827 INFO:teuthology.task.ansible.out:changed: [gibba013.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule_package", "-o", "/tmp/nrpe.pp", "-m", "/tmp/nrpe.mod"], "delta": "0:00:00.013658", "end": "2021-02-10 10:56:48.805921", "rc": 0, "start": "2021-02-10 10:56:48.792263", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-10T10:56:48.863 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:56:48 +0000 (0:00:00.159) 0:04:29.164 **** 2021-02-10T10:56:54.721 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Load SELinux policy package] ***************************** 2021-02-10T10:56:54.722 INFO:teuthology.task.ansible.out:changed: [gibba002.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-i", "/tmp/nrpe.pp"], "delta": "0:00:05.559607", "end": "2021-02-10 10:56:54.696835", "rc": 0, "start": "2021-02-10 10:56:49.137228", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-10T10:56:54.766 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:56:54 +0000 (0:00:05.903) 0:04:35.068 **** 2021-02-10T10:56:54.941 INFO:teuthology.task.ansible.out:changed: [gibba013.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-i", "/tmp/nrpe.pp"], "delta": "0:00:05.623937", "end": "2021-02-10 10:56:54.923239", "rc": 0, "start": "2021-02-10 10:56:49.299302", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-10T10:56:54.976 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:56:54 +0000 (0:00:00.209) 0:04:35.278 **** 2021-02-10T10:56:55.256 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove temporary files] ********************************** 2021-02-10T10:56:55.257 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => {"changed": false, "path": "/tmp/nrpe.*", "state": "absent"} 2021-02-10T10:56:55.292 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:56:55 +0000 (0:00:00.316) 0:04:35.594 **** 2021-02-10T10:56:55.320 INFO:teuthology.task.ansible.out: TASK [testnode : Include package type specific vars.] ************************** 2021-02-10T10:56:55.321 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => {"ansible_facts": {"ceph_dependency_packages_to_remove": ["boost-random", "boost-program-options", "leveldb", "xmlstarlet", "python-jinja2", "python-ceph", "python-flask", "python-requests", "boost-random", "python-urllib3", "python-babel", "hdparm", "python-markupsafe", "python-werkzeug", "python-itsdangerous"], "ceph_packages_to_remove": ["ceph", "ceph-base", "ceph-selinux", "ceph-common", "ceph-debuginfo", "ceph-release", "libcephfs1", "ceph-radosgw", "python-ceph", "python-rados", "python-rbd", "python-cephfs", "librbd1", "librados2", "mod_fastcgi"], "ntp_service_name": "ntpd", "packages_to_remove": ["device-mapper-multipath"], "pip_packages_to_install": ["remoto>=0.0.35"], "ssh_service_name": "sshd"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/yum_systems.yml"], "changed": false} 2021-02-10T10:56:55.356 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:56:55 +0000 (0:00:00.063) 0:04:35.657 **** 2021-02-10T10:56:55.396 INFO:teuthology.task.ansible.out: TASK [testnode : Including distro specific variables.] ************************* 2021-02-10T10:56:55.398 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2021-02-10T10:56:55.431 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:56:55 +0000 (0:00:00.075) 0:04:35.733 **** 2021-02-10T10:56:55.449 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove temporary files] ********************************** 2021-02-10T10:56:55.451 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => {"changed": false, "path": "/tmp/nrpe.*", "state": "absent"} 2021-02-10T10:56:55.487 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:56:55 +0000 (0:00:00.056) 0:04:35.789 **** 2021-02-10T10:56:55.488 INFO:teuthology.task.ansible.out: 2021-02-10T10:56:55.495 INFO:teuthology.task.ansible.out: 2021-02-10T10:56:55.496 INFO:teuthology.task.ansible.out:TASK [testnode : Including major version specific variables.] ****************** ok: [gibba002.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/redhat_7.yml) => {"ansible_facts": {"common_yum_repos": {"lab-extras": {"baseurl": "http://{{ mirror_host }}/lab-extras/rhel7/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}, "rhel-7-fcgi-ceph": {"baseurl": "http://{{ gitbuilder_host }}/mod_fastcgi-rpm-rhel7-x86_64-basic/ref/master/", "enabled": 1, "gpgcheck": 0, "name": "RHEL 7 Local fastcgi Repo"}}, "epel_packages": ["cryptopp-devel", "cryptopp", "dbench", "fcgi", "fuse-sshfs", "perl-JSON-XS", "leveldb", "xmlstarlet"], "nfs_service": "nfs-server", "packages": ["@core", "@base", "yum-plugin-priorities", "yum-plugin-fastestmirror", "redhat-lsb", "sysstat", "gdb", "git-all", "python-configobj", "gcc-c++", "libedit", "openssl098e", "boost-thread", "xfsprogs", "xfsprogs-devel", "gdisk", "parted", "libgcrypt", "fuse", "fuse-libs", "lvm2", "openssl", "libuuid", "btrfs-progs", "attr", "valgrind", "python-nose", "mpich", "ant", "lsof", "iozone", "libtool", "automake", "gettext", "libuuid-devel", "libacl-devel", "bc", "xfsdump", "blktrace", "numpy", "python-matplotlib", "qemu-kvm", "usbredir", "genisoimage", "httpd", "httpd-devel", "httpd-tools", "mod_ssl", "mod_fastcgi-2.4.7-1.ceph.el7", "libev-devel", "perl-XML-Twig", "java-1.6.0-openjdk-devel", "junit4", "nfs-utils", "ncurses-devel", "python-devel", "python-virtualenv", "perl-CPAN", "podman", "podman-docker", "python3"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/redhat_7.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/redhat_7.yml"} 2021-02-10T10:56:55.534 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:56:55 +0000 (0:00:00.047) 0:04:35.836 **** 2021-02-10T10:56:55.542 INFO:teuthology.task.ansible.out: TASK [testnode : Include package type specific vars.] ************************** 2021-02-10T10:56:55.543 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => {"ansible_facts": {"ceph_dependency_packages_to_remove": ["boost-random", "boost-program-options", "leveldb", "xmlstarlet", "python-jinja2", "python-ceph", "python-flask", "python-requests", "boost-random", "python-urllib3", "python-babel", "hdparm", "python-markupsafe", "python-werkzeug", "python-itsdangerous"], "ceph_packages_to_remove": ["ceph", "ceph-base", "ceph-selinux", "ceph-common", "ceph-debuginfo", "ceph-release", "libcephfs1", "ceph-radosgw", "python-ceph", "python-rados", "python-rbd", "python-cephfs", "librbd1", "librados2", "mod_fastcgi"], "ntp_service_name": "ntpd", "packages_to_remove": ["device-mapper-multipath"], "pip_packages_to_install": ["remoto>=0.0.35"], "ssh_service_name": "sshd"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/yum_systems.yml"], "changed": false} 2021-02-10T10:56:55.583 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:56:55 +0000 (0:00:00.048) 0:04:35.884 **** 2021-02-10T10:56:55.591 INFO:teuthology.task.ansible.out: TASK [testnode : Including version specific variables.] ************************ 2021-02-10T10:56:55.591 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2021-02-10T10:56:55.628 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:56:55 +0000 (0:00:00.045) 0:04:35.929 **** 2021-02-10T10:56:55.634 INFO:teuthology.task.ansible.out: TASK [testnode : Including distro specific variables.] ************************* 2021-02-10T10:56:55.635 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2021-02-10T10:56:55.672 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:56:55 +0000 (0:00:00.044) 0:04:35.974 **** 2021-02-10T10:56:55.721 INFO:teuthology.task.ansible.out: TASK [testnode : Including major version specific variables.] ****************** 2021-02-10T10:56:55.722 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/redhat_7.yml) => {"ansible_facts": {"common_yum_repos": {"lab-extras": {"baseurl": "http://{{ mirror_host }}/lab-extras/rhel7/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}, "rhel-7-fcgi-ceph": {"baseurl": "http://{{ gitbuilder_host }}/mod_fastcgi-rpm-rhel7-x86_64-basic/ref/master/", "enabled": 1, "gpgcheck": 0, "name": "RHEL 7 Local fastcgi Repo"}}, "epel_packages": ["cryptopp-devel", "cryptopp", "dbench", "fcgi", "fuse-sshfs", "perl-JSON-XS", "leveldb", "xmlstarlet"], "nfs_service": "nfs-server", "packages": ["@core", "@base", "yum-plugin-priorities", "yum-plugin-fastestmirror", "redhat-lsb", "sysstat", "gdb", "git-all", "python-configobj", "gcc-c++", "libedit", "openssl098e", "boost-thread", "xfsprogs", "xfsprogs-devel", "gdisk", "parted", "libgcrypt", "fuse", "fuse-libs", "lvm2", "openssl", "libuuid", "btrfs-progs", "attr", "valgrind", "python-nose", "mpich", "ant", "lsof", "iozone", "libtool", "automake", "gettext", "libuuid-devel", "libacl-devel", "bc", "xfsdump", "blktrace", "numpy", "python-matplotlib", "qemu-kvm", "usbredir", "genisoimage", "httpd", "httpd-devel", "httpd-tools", "mod_ssl", "mod_fastcgi-2.4.7-1.ceph.el7", "libev-devel", "perl-XML-Twig", "java-1.6.0-openjdk-devel", "junit4", "nfs-utils", "ncurses-devel", "python-devel", "python-virtualenv", "perl-CPAN", "podman", "podman-docker", "python3"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/redhat_7.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/redhat_7.yml"} 2021-02-10T10:56:55.756 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:56:55 +0000 (0:00:00.083) 0:04:36.057 **** 2021-02-10T10:56:55.803 INFO:teuthology.task.ansible.out: TASK [testnode : Including version specific variables.] ************************ 2021-02-10T10:56:55.805 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2021-02-10T10:56:55.837 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:56:55 +0000 (0:00:00.081) 0:04:36.139 **** 2021-02-10T10:56:56.073 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the sudo group exists.] ******************************** 2021-02-10T10:56:56.074 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} 2021-02-10T10:56:56.114 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:56:56 +0000 (0:00:00.277) 0:04:36.416 **** 2021-02-10T10:56:56.280 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} 2021-02-10T10:56:56.316 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:56:56 +0000 (0:00:00.201) 0:04:36.618 **** 2021-02-10T10:56:56.561 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the teuthology_user group exists.] ********************* ok: [gibba002.front.sepia.ceph.com] => {"changed": false, "gid": 1236, "name": "ubuntu", "state": "present", "system": false} 2021-02-10T10:56:56.594 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:56:56 +0000 (0:00:00.278) 0:04:36.896 **** 2021-02-10T10:56:56.622 INFO:teuthology.task.ansible.out: TASK [testnode : Create the teuthology user.] ********************************** 2021-02-10T10:56:56.623 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:56:56.656 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:56:56 +0000 (0:00:00.061) 0:04:36.958 **** 2021-02-10T10:56:56.683 INFO:teuthology.task.ansible.out: TASK [testnode : Delete the teuthology users password.] ************************ 2021-02-10T10:56:56.685 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:56:56.726 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:56:56 +0000 (0:00:00.069) 0:04:37.027 **** 2021-02-10T10:56:56.764 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the teuthology_user group exists.] ********************* 2021-02-10T10:56:56.766 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => {"changed": false, "gid": 1236, "name": "ubuntu", "state": "present", "system": false} 2021-02-10T10:56:56.798 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:56:56 +0000 (0:00:00.072) 0:04:37.100 **** 2021-02-10T10:56:56.825 INFO:teuthology.task.ansible.out: TASK [testnode : Create the teuthology user.] ********************************** skipping: [gibba013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:56:56.859 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:56:56 +0000 (0:00:00.060) 0:04:37.160 **** 2021-02-10T10:56:56.891 INFO:teuthology.task.ansible.out: TASK [testnode : Delete the teuthology users password.] ************************ 2021-02-10T10:56:56.892 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:56:56.925 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:56:56 +0000 (0:00:00.066) 0:04:37.226 **** 2021-02-10T10:56:57.197 INFO:teuthology.task.ansible.out: TASK [testnode : Add a user for xfstests to test user quotas.] ***************** 2021-02-10T10:56:57.199 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 10101, "home": "/home/fsgqa", "move_home": false, "name": "fsgqa", "shell": "/bin/bash", "state": "present", "uid": 10101} 2021-02-10T10:56:57.235 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:56:57 +0000 (0:00:00.309) 0:04:37.536 **** 2021-02-10T10:56:57.403 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 10101, "home": "/home/fsgqa", "move_home": false, "name": "fsgqa", "shell": "/bin/bash", "state": "present", "uid": 10101} 2021-02-10T10:56:57.444 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:56:57 +0000 (0:00:00.209) 0:04:37.746 **** 2021-02-10T10:56:59.242 INFO:teuthology.task.ansible.out: 2021-02-10T10:56:59.243 INFO:teuthology.task.ansible.out:TASK [testnode : Set a high max open files limit for the teuthology user.] ***** ok: [gibba002.front.sepia.ceph.com] => {"changed": false, "checksum": "dc140f4bb26594b8d5e0753bab8d93c0cced7772", "dest": "/etc/security/limits.d/ubuntu.conf", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/security/limits.d/ubuntu.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 112, "state": "file", "uid": 0} 2021-02-10T10:56:59.281 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:56:59 +0000 (0:00:01.836) 0:04:39.582 **** 2021-02-10T10:56:59.311 INFO:teuthology.task.ansible.out: TASK [testnode : Set hostname var] ********************************************* 2021-02-10T10:56:59.313 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => {"ansible_facts": {"hostname": "gibba002"}, "changed": false} 2021-02-10T10:56:59.346 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:56:59 +0000 (0:00:00.065) 0:04:39.648 **** 2021-02-10T10:56:59.438 INFO:teuthology.task.ansible.out: TASK [testnode : Set a high max open files limit for the teuthology user.] ***** 2021-02-10T10:56:59.439 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => {"changed": false, "checksum": "dc140f4bb26594b8d5e0753bab8d93c0cced7772", "dest": "/etc/security/limits.d/ubuntu.conf", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/security/limits.d/ubuntu.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 112, "state": "file", "uid": 0} 2021-02-10T10:56:59.473 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:56:59 +0000 (0:00:00.126) 0:04:39.774 **** 2021-02-10T10:56:59.501 INFO:teuthology.task.ansible.out: TASK [testnode : Set hostname var] ********************************************* 2021-02-10T10:56:59.503 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => {"ansible_facts": {"hostname": "gibba013"}, "changed": false} 2021-02-10T10:56:59.534 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:56:59 +0000 (0:00:00.061) 0:04:39.836 **** 2021-02-10T10:57:00.453 INFO:teuthology.task.ansible.out: TASK [testnode : Set the system's hostname] ************************************ 2021-02-10T10:57:00.454 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => {"ansible_facts": {"ansible_domain": "", "ansible_fqdn": "gibba013", "ansible_hostname": "gibba013", "ansible_nodename": "gibba013"}, "changed": false, "name": "gibba013"} ok: [gibba002.front.sepia.ceph.com] => {"ansible_facts": {"ansible_domain": "", "ansible_fqdn": "gibba002", "ansible_hostname": "gibba002", "ansible_nodename": "gibba002"}, "changed": false, "name": "gibba002"} 2021-02-10T10:57:00.487 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:57:00 +0000 (0:00:00.952) 0:04:40.788 **** 2021-02-10T10:57:00.518 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:57:00 +0000 (0:00:00.031) 0:04:40.820 **** 2021-02-10T10:57:02.436 INFO:teuthology.task.ansible.out: TASK [testnode : Upload distro major version specific sshd_config] ************* ok: [gibba002.front.sepia.ceph.com] => {"changed": false, "checksum": "a5e550b2aa74f6355e926d70d73a50ad9ab4b4be", "dest": "/etc/ssh/sshd_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/sshd_config", "secontext": "system_u:object_r:etc_t:s0", "size": 1043, "state": "file", "uid": 0} 2021-02-10T10:57:02.471 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:57:02 +0000 (0:00:01.953) 0:04:42.773 **** 2021-02-10T10:57:02.481 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => {"changed": false, "checksum": "a5e550b2aa74f6355e926d70d73a50ad9ab4b4be", "dest": "/etc/ssh/sshd_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/sshd_config", "secontext": "system_u:object_r:etc_t:s0", "size": 1043, "state": "file", "uid": 0} 2021-02-10T10:57:02.518 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:57:02 +0000 (0:00:00.046) 0:04:42.820 **** 2021-02-10T10:57:04.411 INFO:teuthology.task.ansible.out: TASK [testnode : Upload ssh_config] ******************************************** 2021-02-10T10:57:04.412 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => {"changed": false, "checksum": "58d78b02605c8368f1913de12e145f4ad930fafc", "dest": "/etc/ssh/ssh_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/ssh_config", "secontext": "system_u:object_r:etc_t:s0", "size": 570, "state": "file", "uid": 0} 2021-02-10T10:57:04.445 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:57:04 +0000 (0:00:01.927) 0:04:44.747 **** 2021-02-10T10:57:04.474 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => {"changed": false, "checksum": "58d78b02605c8368f1913de12e145f4ad930fafc", "dest": "/etc/ssh/ssh_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/ssh_config", "secontext": "system_u:object_r:etc_t:s0", "size": 570, "state": "file", "uid": 0} 2021-02-10T10:57:04.510 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:57:04 +0000 (0:00:00.064) 0:04:44.811 **** 2021-02-10T10:57:05.211 INFO:teuthology.task.ansible.out: TASK [testnode : Add ssh pubkeys] ********************************************** 2021-02-10T10:57:05.212 INFO:teuthology.task.ansible.out:changed: [gibba002.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "comment": null, "exclusive": false, "follow": false, "key": "https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub", "key_options": null, "keyfile": "/home/ubuntu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ubuntu", "validate_certs": true} 2021-02-10T10:57:05.252 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:57:05 +0000 (0:00:00.742) 0:04:45.554 **** 2021-02-10T10:57:05.270 INFO:teuthology.task.ansible.out:changed: [gibba013.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "comment": null, "exclusive": false, "follow": false, "key": "https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub", "key_options": null, "keyfile": "/home/ubuntu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ubuntu", "validate_certs": true} 2021-02-10T10:57:05.310 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:57:05 +0000 (0:00:00.058) 0:04:45.612 **** 2021-02-10T10:57:07.198 INFO:teuthology.task.ansible.out: TASK [testnode : Create remote.conf] ******************************************* 2021-02-10T10:57:07.200 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => {"changed": false, "checksum": "68d0b8ee2f88a77895f257aa69f32a42e47f455b", "dest": "/etc/security/limits.d/remote.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.d/remote.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 116, "state": "file", "uid": 0} 2021-02-10T10:57:07.237 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:57:07 +0000 (0:00:01.927) 0:04:47.539 **** 2021-02-10T10:57:07.267 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => {"changed": false, "checksum": "68d0b8ee2f88a77895f257aa69f32a42e47f455b", "dest": "/etc/security/limits.d/remote.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.d/remote.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 116, "state": "file", "uid": 0} 2021-02-10T10:57:07.302 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:57:07 +0000 (0:00:00.064) 0:04:47.604 **** 2021-02-10T10:57:07.701 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2021-02-10T10:57:07.702 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => {"changed": false, "dest": "/etc/fuse.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 0, "state": "file", "uid": 0} 2021-02-10T10:57:07.746 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:57:07 +0000 (0:00:00.444) 0:04:48.048 **** 2021-02-10T10:57:07.766 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => {"changed": false, "dest": "/etc/fuse.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 0, "state": "file", "uid": 0} 2021-02-10T10:57:07.820 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:57:07 +0000 (0:00:00.073) 0:04:48.122 **** 2021-02-10T10:57:08.207 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2021-02-10T10:57:08.208 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => {"changed": false, "gid": 10102, "name": "kvm", "state": "present", "system": false} 2021-02-10T10:57:08.245 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:57:08 +0000 (0:00:00.425) 0:04:48.547 **** 2021-02-10T10:57:08.273 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => {"changed": false, "gid": 10102, "name": "kvm", "state": "present", "system": false} 2021-02-10T10:57:08.306 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:57:08 +0000 (0:00:00.061) 0:04:48.608 **** 2021-02-10T10:57:08.705 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2021-02-10T10:57:08.705 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => {"append": true, "changed": false, "comment": "", "group": 1236, "groups": "kvm,disk", "home": "/home/ubuntu", "move_home": false, "name": "ubuntu", "shell": "/bin/bash", "state": "present", "uid": 1000} 2021-02-10T10:57:08.742 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:57:08 +0000 (0:00:00.435) 0:04:49.043 **** 2021-02-10T10:57:08.781 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => {"append": true, "changed": false, "comment": "", "group": 1236, "groups": "kvm,disk", "home": "/home/ubuntu", "move_home": false, "name": "ubuntu", "shell": "/bin/bash", "state": "present", "uid": 1000} 2021-02-10T10:57:08.782 INFO:teuthology.task.ansible.out: 2021-02-10T10:57:08.827 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:57:08 +0000 (0:00:00.085) 0:04:49.129 **** 2021-02-10T10:57:10.689 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2021-02-10T10:57:10.690 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => {"changed": false, "checksum": "be3223c8b0917bde02657a5f9ddd95277cc3f812", "dest": "/etc/sudoers", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers", "secontext": "system_u:object_r:etc_t:s0", "size": 1808, "state": "file", "uid": 0} 2021-02-10T10:57:10.726 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:57:10 +0000 (0:00:01.898) 0:04:51.027 **** 2021-02-10T10:57:10.765 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => {"changed": false, "checksum": "be3223c8b0917bde02657a5f9ddd95277cc3f812", "dest": "/etc/sudoers", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers", "secontext": "system_u:object_r:etc_t:s0", "size": 1808, "state": "file", "uid": 0} 2021-02-10T10:57:10.801 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:57:10 +0000 (0:00:00.074) 0:04:51.102 **** 2021-02-10T10:57:12.664 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2021-02-10T10:57:12.666 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => {"changed": false, "checksum": "12a2e17405b9e00ea2061e25929905d08e5fe19f", "dest": "/etc/security/limits.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 854, "state": "file", "uid": 0} 2021-02-10T10:57:12.707 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:57:12 +0000 (0:00:01.905) 0:04:53.008 **** 2021-02-10T10:57:12.746 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => {"changed": false, "checksum": "12a2e17405b9e00ea2061e25929905d08e5fe19f", "dest": "/etc/security/limits.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 854, "state": "file", "uid": 0} 2021-02-10T10:57:12.781 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:57:12 +0000 (0:00:00.074) 0:04:53.082 **** 2021-02-10T10:57:14.745 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2021-02-10T10:57:14.896 INFO:teuthology.task.ansible.out:[WARNING]: Consider using the yum, dnf or zypper module rather than running 'rpm'. If you need to use command because yum, dnf or zypper is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message. changed: [gibba002.front.sepia.ceph.com] => {"changed": true, "cmd": ["rpm", "--rebuilddb"], "delta": "0:00:01.582306", "end": "2021-02-10 10:57:14.724411", "rc": 0, "start": "2021-02-10 10:57:13.142105", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} Wednesday 10 February 2021 10:57:14 +0000 (0:00:02.002) 0:04:55.085 **** changed: [gibba013.front.sepia.ceph.com] => {"changed": true, "cmd": ["rpm", "--rebuilddb"], "delta": "0:00:01.526109", "end": "2021-02-10 10:57:14.739307", "rc": 0, "start": "2021-02-10 10:57:13.213198", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} Wednesday 10 February 2021 10:57:14 +0000 (0:00:00.051) 0:04:55.137 **** TASK [testnode : Configure local mirrorlists] ********************************** skipping: [gibba002.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Wednesday 10 February 2021 10:57:14 +0000 (0:00:00.045) 0:04:55.183 **** skipping: [gibba013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:57:14.922 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:57:14 +0000 (0:00:00.041) 0:04:55.224 **** 2021-02-10T10:57:16.817 INFO:teuthology.task.ansible.out: TASK [testnode : Configure common additional repos in /etc/yum.repos.d/] ******* 2021-02-10T10:57:16.819 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'key': 'rhel-7-fcgi-ceph', 'value': {'name': 'RHEL 7 Local fastcgi Repo', 'baseurl': 'http://gitbuilder.ceph.com/mod_fastcgi-rpm-rhel7-x86_64-basic/ref/master/', 'enabled': 1, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "e6aba9c9ab05e27255ed430f065502af284a869b", "dest": "/etc/yum.repos.d/rhel-7-fcgi-ceph.repo", "gid": 0, "group": "root", "item": {"key": "rhel-7-fcgi-ceph", "value": {"baseurl": "http://gitbuilder.ceph.com/mod_fastcgi-rpm-rhel7-x86_64-basic/ref/master/", "enabled": 1, "gpgcheck": 0, "name": "RHEL 7 Local fastcgi Repo"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/rhel-7-fcgi-ceph.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 245, "state": "file", "uid": 0} 2021-02-10T10:57:16.851 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'key': 'rhel-7-fcgi-ceph', 'value': {'name': 'RHEL 7 Local fastcgi Repo', 'baseurl': 'http://gitbuilder.ceph.com/mod_fastcgi-rpm-rhel7-x86_64-basic/ref/master/', 'enabled': 1, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "e6aba9c9ab05e27255ed430f065502af284a869b", "dest": "/etc/yum.repos.d/rhel-7-fcgi-ceph.repo", "gid": 0, "group": "root", "item": {"key": "rhel-7-fcgi-ceph", "value": {"baseurl": "http://gitbuilder.ceph.com/mod_fastcgi-rpm-rhel7-x86_64-basic/ref/master/", "enabled": 1, "gpgcheck": 0, "name": "RHEL 7 Local fastcgi Repo"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/rhel-7-fcgi-ceph.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 245, "state": "file", "uid": 0} 2021-02-10T10:57:18.742 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'key': 'lab-extras', 'value': {'name': 'lab-extras', 'baseurl': 'http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/', 'enabled': 1, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "e3b44e00721d55ceb436a2b7c0ec58cd5bc39dc9", "dest": "/etc/yum.repos.d/lab-extras.repo", "gid": 0, "group": "root", "item": {"key": "lab-extras", "value": {"baseurl": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/lab-extras.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 207, "state": "file", "uid": 0} 2021-02-10T10:57:18.780 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:57:18 +0000 (0:00:03.857) 0:04:59.082 **** 2021-02-10T10:57:18.788 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'key': 'lab-extras', 'value': {'name': 'lab-extras', 'baseurl': 'http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/', 'enabled': 1, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "e3b44e00721d55ceb436a2b7c0ec58cd5bc39dc9", "dest": "/etc/yum.repos.d/lab-extras.repo", "gid": 0, "group": "root", "item": {"key": "lab-extras", "value": {"baseurl": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/lab-extras.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 207, "state": "file", "uid": 0} 2021-02-10T10:57:18.827 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:57:18 +0000 (0:00:00.046) 0:04:59.129 **** 2021-02-10T10:57:18.836 INFO:teuthology.task.ansible.out: TASK [testnode : Configure version specific repos in /etc/yum.repos.d/] ******** 2021-02-10T10:57:18.887 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:57:18 +0000 (0:00:00.059) 0:04:59.189 **** 2021-02-10T10:57:18.942 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:57:18 +0000 (0:00:00.054) 0:04:59.244 **** 2021-02-10T10:57:18.949 INFO:teuthology.task.ansible.out: TASK [testnode : Enable copr repos] ******************************************** 2021-02-10T10:57:18.987 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:57:18 +0000 (0:00:00.044) 0:04:59.288 **** 2021-02-10T10:57:19.030 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:57:19 +0000 (0:00:00.043) 0:04:59.332 **** 2021-02-10T10:57:19.038 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG] ************************************************** 2021-02-10T10:57:19.039 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:57:19.072 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:57:19 +0000 (0:00:00.041) 0:04:59.374 **** 2021-02-10T10:57:19.080 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:57:19.115 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:57:19 +0000 (0:00:00.043) 0:04:59.417 **** 2021-02-10T10:57:19.122 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG keys] ********************************************* 2021-02-10T10:57:19.124 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} skipping: [gibba002.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2021-02-10T10:57:19.161 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:57:19 +0000 (0:00:00.045) 0:04:59.462 **** 2021-02-10T10:57:19.167 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} 2021-02-10T10:57:19.168 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2021-02-10T10:57:19.202 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:57:19 +0000 (0:00:00.041) 0:04:59.504 **** 2021-02-10T10:57:19.214 INFO:teuthology.task.ansible.out: TASK [testnode : Clean yum cache] ********************************************** 2021-02-10T10:57:19.216 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:57:19.252 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:57:19 +0000 (0:00:00.049) 0:04:59.553 **** 2021-02-10T10:57:19.261 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:57:19.309 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:57:19 +0000 (0:00:00.057) 0:04:59.610 **** 2021-02-10T10:57:19.316 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-02-10T10:57:19.317 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:57:19.351 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:57:19 +0000 (0:00:00.042) 0:04:59.653 **** 2021-02-10T10:57:19.358 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:57:19.403 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:57:19 +0000 (0:00:00.051) 0:04:59.704 **** 2021-02-10T10:57:20.216 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure yum_utils is present.] ********************************* 2021-02-10T10:57:20.217 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => {"changed": false, "msg": "", "rc": 0, "results": ["yum-utils-1.1.31-54.el7_8.noarch providing yum-utils is already installed"]} 2021-02-10T10:57:20.251 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:57:20 +0000 (0:00:00.848) 0:05:00.552 **** 2021-02-10T10:57:20.289 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => {"changed": false, "msg": "", "rc": 0, "results": ["yum-utils-1.1.31-54.el7_8.noarch providing yum-utils is already installed"]} 2021-02-10T10:57:20.290 INFO:teuthology.task.ansible.out: 2021-02-10T10:57:20.324 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:57:20 +0000 (0:00:00.073) 0:05:00.626 **** 2021-02-10T10:57:21.397 INFO:teuthology.task.ansible.out: TASK [testnode : Removing saved yum transactions] ****************************** 2021-02-10T10:57:21.398 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => {"changed": false, "cmd": ["yum-complete-transaction", "--cleanup-only"], "delta": "0:00:00.682073", "end": "2021-02-10 10:57:21.371518", "rc": 0, "start": "2021-02-10 10:57:20.689445", "stderr": "", "stderr_lines": [], "stdout": "Loaded plugins: fastestmirror, langpacks, priorities, product-id, subscription-\n : manager\nLoading mirror speeds from cached hostfile\n * epel: download-cc-rdu01.fedoraproject.org\nNo unfinished transactions left.", "stdout_lines": ["Loaded plugins: fastestmirror, langpacks, priorities, product-id, subscription-", " : manager", "Loading mirror speeds from cached hostfile", " * epel: download-cc-rdu01.fedoraproject.org", "No unfinished transactions left."]} 2021-02-10T10:57:21.437 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:57:21 +0000 (0:00:01.112) 0:05:01.738 **** 2021-02-10T10:57:21.455 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => {"changed": false, "cmd": ["yum-complete-transaction", "--cleanup-only"], "delta": "0:00:00.661487", "end": "2021-02-10 10:57:21.425715", "rc": 0, "start": "2021-02-10 10:57:20.764228", "stderr": "", "stderr_lines": [], "stdout": "Loaded plugins: fastestmirror, langpacks, priorities, product-id, subscription-\n : manager\nLoading mirror speeds from cached hostfile\n * epel: download-cc-rdu01.fedoraproject.org\nNo unfinished transactions left.", "stdout_lines": ["Loaded plugins: fastestmirror, langpacks, priorities, product-id, subscription-", " : manager", "Loading mirror speeds from cached hostfile", " * epel: download-cc-rdu01.fedoraproject.org", "No unfinished transactions left."]} 2021-02-10T10:57:21.489 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:57:21 +0000 (0:00:00.052) 0:05:01.791 **** 2021-02-10T10:57:21.907 INFO:teuthology.task.ansible.out: TASK [testnode : Check if ceph-debuginfo is installed] ************************* 2021-02-10T10:57:21.908 INFO:teuthology.task.ansible.out:fatal: [gibba002.front.sepia.ceph.com]: FAILED! => {"changed": false, "cmd": ["rpm", "-q", "ceph-debuginfo"], "delta": "0:00:00.018197", "end": "2021-02-10 10:57:21.885899", "msg": "non-zero return code", "rc": 1, "start": "2021-02-10 10:57:21.867702", "stderr": "", "stderr_lines": [], "stdout": "package ceph-debuginfo is not installed", "stdout_lines": ["package ceph-debuginfo is not installed"]} ...ignoring 2021-02-10T10:57:21.943 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:57:21 +0000 (0:00:00.454) 0:05:02.245 **** 2021-02-10T10:57:21.963 INFO:teuthology.task.ansible.out:fatal: [gibba013.front.sepia.ceph.com]: FAILED! => {"changed": false, "cmd": ["rpm", "-q", "ceph-debuginfo"], "delta": "0:00:00.018046", "end": "2021-02-10 10:57:21.941074", "msg": "non-zero return code", "rc": 1, "start": "2021-02-10 10:57:21.923028", "stderr": "", "stderr_lines": [], "stdout": "package ceph-debuginfo is not installed", "stdout_lines": ["package ceph-debuginfo is not installed"]} ...ignoring 2021-02-10T10:57:21.997 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:57:21 +0000 (0:00:00.053) 0:05:02.298 **** 2021-02-10T10:57:22.003 INFO:teuthology.task.ansible.out: TASK [testnode : Work around https://bugzilla.redhat.com/show_bug.cgi?id=1234967] *** 2021-02-10T10:57:22.003 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:57:22.037 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:57:22 +0000 (0:00:00.040) 0:05:02.339 **** 2021-02-10T10:57:22.043 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:57:22.078 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:57:22 +0000 (0:00:00.040) 0:05:02.380 **** 2021-02-10T10:57:31.571 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ changed: [gibba002.front.sepia.ceph.com] => {"changed": true, "changes": {"removed": ["librbd1", "librados2", "mod_fastcgi"]}, "msg": "", "rc": 0, "results": ["ceph is not installed", "ceph-base is not installed", "ceph-selinux is not installed", "ceph-common is not installed", "ceph-debuginfo is not installed", "ceph-release is not installed", "libcephfs1 is not installed", "ceph-radosgw is not installed", "python-ceph is not installed", "python-rados is not installed", "python-rbd is not installed", "python-cephfs is not installed", "Loaded plugins: fastestmirror, langpacks, priorities, product-id, search-\n : disabled-repos, subscription-manager\nResolving Dependencies\n--> Running transaction check\n---> Package librados2.x86_64 1:10.2.5-4.el7 will be erased\n--> Processing Dependency: librados.so.2()(64bit) for package: 10:qemu-img-1.5.3-175.el7_9.1.x86_64\n--> Processing Dependency: librados.so.2()(64bit) for package: 10:qemu-kvm-common-1.5.3-175.el7_9.1.x86_64\n--> Processing Dependency: librados.so.2()(64bit) for package: 10:qemu-kvm-1.5.3-175.el7_9.1.x86_64\n---> Package librbd1.x86_64 1:10.2.5-4.el7 will be erased\n---> Package mod_fastcgi.x86_64 0:2.4.7-1.ceph.el7 will be erased\n--> Running transaction check\n---> Package qemu-img.x86_64 10:1.5.3-175.el7_9.1 will be erased\n---> Package qemu-kvm.x86_64 10:1.5.3-175.el7_9.1 will be erased\n---> Package qemu-kvm-common.x86_64 10:1.5.3-175.el7_9.1 will be erased\n--> Finished Dependency Resolution\n\nDependencies Resolved\n\n================================================================================\n Package Arch Version Repository Size\n================================================================================\nRemoving:\n librados2 x86_64 1:10.2.5-4.el7 @rhel-7-server-rpms 5.5 M\n librbd1 x86_64 1:10.2.5-4.el7 @rhel 2021-02-10T10:57:31.786 INFO:teuthology.task.ansible.out:-7-server-rpms 7.4 M\n mod_fastcgi x86_64 2.4.7-1.ceph.el7 @rhel-7-fcgi-ceph 213 k\nRemoving for dependencies:\n qemu-img x86_64 10:1.5.3-175.el7_9.1 @rhel-7-server-rpms 2.3 M\n qemu-kvm x86_64 10:1.5.3-175.el7_9.1 @rhel-7-server-rpms 5.7 M\n qemu-kvm-common x86_64 10:1.5.3-175.el7_9.1 @rhel-7-server-rpms 755 k\n\nTransaction Summary\n================================================================================\nRemove 3 Packages (+3 Dependent packages)\n\nInstalled size: 22 M\nDownloading packages:\nRunning transaction check\nRunning transaction test\nTransaction test succeeded\nRunning transaction\n Erasing : 10:qemu-kvm-1.5.3-175.el7_9.1.x86_64 1/6 \n Erasing : 10:qemu-img-1.5.3-175.el7_9.1.x86_64 2/6 \n Erasing : 10:qemu-kvm-common-1.5.3-175.el7_9.1.x86_64 3/6 \n Erasing : 1:librbd1-10.2.5-4.el7.x86_64 4/6 \n Erasing : 1:librados2-10.2.5-4.el7.x86_64 5/6 \n Erasing : mod_fastcgi-2.4.7-1.ceph.el7.x86_64 6/6 \nwarning: file /var/run/mod_fastcgi: remove failed: No such file or directory\nLoaded plugins: fastestmirror, langpacks, priorities, product-id, subscription-\n : manager\n Verifying : 1:librbd1-10.2.5-4.el7.x86_64 1/6 \n Verifying : 10:qemu-kvm-1.5.3-175.el7_9.1.x86_64 2/6 \n Verifying : mod_fastcgi-2.4.7-1.ceph.el7.x86_64 3/6 \n Verifying : 10:qemu-kvm-common-1.5.3-175.el7_9.1.x86_64 4/6 \n Verifying : 1:librados2-10.2.5-4.el7.x86_64 5/6 \n Verifying : 10:qemu-img-1.5.3-175.el7_9.1.x86_64 6/6 \n\nRemoved:\n librados2.x86_64 1:10.2.5-4.el7 librbd1.x86_64 1:10.2.5-4.el7 \n mod_fastcgi.x86_64 0:2.4.7-1.ceph.el7 \n\nDependency Remove 2021-02-10T10:57:31.787 INFO:teuthology.task.ansible.out:d:\n qemu-img.x86_64 10:1.5.3-175.el7_9.1 \n qemu-kvm.x86_64 10:1.5.3-175.el7_9.1 \n qemu-kvm-common.x86_64 10:1.5.3-175.el7_9.1 \n\nComplete!\n"]} Wednesday 10 February 2021 10:57:31 +0000 (0:00:09.526) 0:05:11.906 **** 2021-02-10T10:57:32.000 INFO:teuthology.task.ansible.out:changed: [gibba013.front.sepia.ceph.com] => {"changed": true, "changes": {"removed": ["librbd1", "librados2", "mod_fastcgi"]}, "msg": "", "rc": 0, "results": ["ceph is not installed", "ceph-base is not installed", "ceph-selinux is not installed", "ceph-common is not installed", "ceph-debuginfo is not installed", "ceph-release is not installed", "libcephfs1 is not installed", "ceph-radosgw is not installed", "python-ceph is not installed", "python-rados is not installed", "python-rbd is not installed", "python-cephfs is not installed", "Loaded plugins: fastestmirror, langpacks, priorities, product-id, search-\n : disabled-repos, subscription-manager\nResolving Dependencies\n--> Running transaction check\n---> Package librados2.x86_64 1:10.2.5-4.el7 will be erased\n--> Processing Dependency: librados.so.2()(64bit) for package: 10:qemu-img-1.5.3-175.el7_9.1.x86_64\n--> Processing Dependency: librados.so.2()(64bit) for package: 10:qemu-kvm-common-1.5.3-175.el7_9.1.x86_64\n--> Processing Dependency: librados.so.2()(64bit) for package: 10:qemu-kvm-1.5.3-175.el7_9.1.x86_64\n---> Package librbd1.x86_64 1:10.2.5-4.el7 will be erased\n---> Package mod_fastcgi.x86_64 0:2.4.7-1.ceph.el7 will be erased\n--> Running transaction check\n---> Package qemu-img.x86_64 10:1.5.3-175.el7_9.1 will be erased\n---> Package qemu-kvm.x86_64 10:1.5.3-175.el7_9.1 will be erased\n---> Package qemu-kvm-common.x86_64 10:1.5.3-175.el7_9.1 will be erased\n--> Finished Dependency Resolution\n\nDependencies Resolved\n\n================================================================================\n Package Arch Version Repository Size\n================================================================================\nRemoving:\n librados2 x86_64 1:10.2.5-4.el7 @rhel-7-server-rpms 5.5 M\n librbd1 x86_64 1:10.2.5-4.el7 @rhel-7-server-rpms 7.4 M\n mod_fastcgi x86_64 2.4.7-1.ceph.el7 @rhel-7- 2021-02-10T10:57:32.001 INFO:teuthology.task.ansible.out:fcgi-ceph 213 k\nRemoving for dependencies:\n qemu-img x86_64 10:1.5.3-175.el7_9.1 @rhel-7-server-rpms 2.3 M\n qemu-kvm x86_64 10:1.5.3-175.el7_9.1 @rhel-7-server-rpms 5.7 M\n qemu-kvm-common x86_64 10:1.5.3-175.el7_9.1 @rhel-7-server-rpms 755 k\n\nTransaction Summary\n================================================================================\nRemove 3 Packages (+3 Dependent packages)\n\nInstalled size: 22 M\nDownloading packages:\nRunning transaction check\nRunning transaction test\nTransaction test succeeded\nRunning transaction\n Erasing : 10:qemu-kvm-1.5.3-175.el7_9.1.x86_64 1/6 \n Erasing : 10:qemu-img-1.5.3-175.el7_9.1.x86_64 2/6 \n Erasing : 10:qemu-kvm-common-1.5.3-175.el7_9.1.x86_64 3/6 \n Erasing : 1:librbd1-10.2.5-4.el7.x86_64 4/6 \n Erasing : 1:librados2-10.2.5-4.el7.x86_64 5/6 \n Erasing : mod_fastcgi-2.4.7-1.ceph.el7.x86_64 6/6 \nwarning: file /var/run/mod_fastcgi: remove failed: No such file or directory\nLoaded plugins: fastestmirror, langpacks, priorities, product-id, subscription-\n : manager\n Verifying : 1:librbd1-10.2.5-4.el7.x86_64 1/6 \n Verifying : 10:qemu-kvm-1.5.3-175.el7_9.1.x86_64 2/6 \n Verifying : mod_fastcgi-2.4.7-1.ceph.el7.x86_64 3/6 \n Verifying : 10:qemu-kvm-common-1.5.3-175.el7_9.1.x86_64 4/6 \n Verifying : 1:librados2-10.2.5-4.el7.x86_64 5/6 \n Verifying : 10:qemu-img-1.5.3-175.el7_9.1.x86_64 6/6 \n\nRemoved:\n librados2.x86_64 1:10.2.5-4.el7 librbd1.x86_64 1:10.2.5-4.el7 \n mod_fastcgi.x86_64 0:2.4.7-1.ceph.el7 \n\nDependency Removed:\n qemu-img.x86_64 10:1.5.3-175.el7_9.1 2021-02-10T10:57:32.003 INFO:teuthology.task.ansible.out:\n qemu-kvm.x86_64 10:1.5.3-175.el7_9.1 \n qemu-kvm-common.x86_64 10:1.5.3-175.el7_9.1 \n\nComplete!\n"]} 2021-02-10T10:57:32.043 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:57:32 +0000 (0:00:00.438) 0:05:12.344 **** 2021-02-10T10:57:36.324 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2021-02-10T10:57:36.325 INFO:teuthology.task.ansible.out:changed: [gibba002.front.sepia.ceph.com] => {"changed": true, "changes": {"removed": ["boost-random", "leveldb", "xmlstarlet", "boost-random"]}, "msg": "", "rc": 0, "results": ["boost-program-options is not installed", "python-jinja2 is not installed", "python-ceph is not installed", "python-flask is not installed", "python-requests is not installed", "python-urllib3 is not installed", "python-babel is not installed", "hdparm is not installed", "python-markupsafe is not installed", "python-werkzeug is not installed", "python-itsdangerous is not installed", "Loaded plugins: fastestmirror, langpacks, priorities, product-id, search-\n : disabled-repos, subscription-manager\nResolving Dependencies\n--> Running transaction check\n---> Package boost-random.x86_64 0:1.53.0-28.el7 will be erased\n---> Package leveldb.x86_64 0:1.12.0-11.el7 will be erased\n---> Package xmlstarlet.x86_64 0:1.6.1-1.el7 will be erased\n--> Finished Dependency Resolution\n\nDependencies Resolved\n\n================================================================================\n Package Arch Version Repository Size\n================================================================================\nRemoving:\n boost-random x86_64 1.53.0-28.el7 @rhel-7-server-rpms 24 k\n leveldb x86_64 1.12.0-11.el7 @epel 439 k\n xmlstarlet x86_64 1.6.1-1.el7 @epel 192 k\n\nTransaction Summary\n================================================================================\nRemove 3 Packages\n\nInstalled size: 655 k\nDownloading packages:\nRunning transaction check\nRunning transaction test\nTransaction test succeeded\nRunning transaction\n Erasing : xmlstarlet-1.6.1-1.el7.x86_64 1/3 \n Erasing : boost-random-1.53.0-28.el7.x86_64 2/3 \n Erasing : leveldb-1.12.0-11.el7.x86_64 2021-02-10T10:57:36.326 INFO:teuthology.task.ansible.out: 3/3 \nLoaded plugins: fastestmirror, langpacks, priorities, product-id, subscription-\n : manager\n Verifying : leveldb-1.12.0-11.el7.x86_64 1/3 \n Verifying : boost-random-1.53.0-28.el7.x86_64 2/3 \n Verifying : xmlstarlet-1.6.1-1.el7.x86_64 3/3 \n\nRemoved:\n boost-random.x86_64 0:1.53.0-28.el7 leveldb.x86_64 0:1.12.0-11.el7 \n xmlstarlet.x86_64 0:1.6.1-1.el7 \n\nComplete!\n"]} 2021-02-10T10:57:36.358 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:57:36 +0000 (0:00:04.315) 0:05:16.660 **** 2021-02-10T10:57:36.648 INFO:teuthology.task.ansible.out:changed: [gibba013.front.sepia.ceph.com] => {"changed": true, "changes": {"removed": ["boost-random", "leveldb", "xmlstarlet", "boost-random"]}, "msg": "", "rc": 0, "results": ["boost-program-options is not installed", "python-jinja2 is not installed", "python-ceph is not installed", "python-flask is not installed", "python-requests is not installed", "python-urllib3 is not installed", "python-babel is not installed", "hdparm is not installed", "python-markupsafe is not installed", "python-werkzeug is not installed", "python-itsdangerous is not installed", "Loaded plugins: fastestmirror, langpacks, priorities, product-id, search-\n : disabled-repos, subscription-manager\nResolving Dependencies\n--> Running transaction check\n---> Package boost-random.x86_64 0:1.53.0-28.el7 will be erased\n---> Package leveldb.x86_64 0:1.12.0-11.el7 will be erased\n---> Package xmlstarlet.x86_64 0:1.6.1-1.el7 will be erased\n--> Finished Dependency Resolution\n\nDependencies Resolved\n\n================================================================================\n Package Arch Version Repository Size\n================================================================================\nRemoving:\n boost-random x86_64 1.53.0-28.el7 @rhel-7-server-rpms 24 k\n leveldb x86_64 1.12.0-11.el7 @epel 439 k\n xmlstarlet x86_64 1.6.1-1.el7 @epel 192 k\n\nTransaction Summary\n================================================================================\nRemove 3 Packages\n\nInstalled size: 655 k\nDownloading packages:\nRunning transaction check\nRunning transaction test\nTransaction test succeeded\nRunning transaction\n Erasing : xmlstarlet-1.6.1-1.el7.x86_64 1/3 \n Erasing : boost-random-1.53.0-28.el7.x86_64 2/3 \n Erasing : leveldb-1.12.0-11.el7.x86_64 2021-02-10T10:57:36.649 INFO:teuthology.task.ansible.out: 3/3 \nLoaded plugins: fastestmirror, langpacks, priorities, product-id, subscription-\n : manager\n Verifying : leveldb-1.12.0-11.el7.x86_64 1/3 \n Verifying : boost-random-1.53.0-28.el7.x86_64 2/3 \n Verifying : xmlstarlet-1.6.1-1.el7.x86_64 3/3 \n\nRemoved:\n boost-random.x86_64 0:1.53.0-28.el7 leveldb.x86_64 0:1.12.0-11.el7 \n xmlstarlet.x86_64 0:1.6.1-1.el7 \n\nComplete!\n"]} 2021-02-10T10:57:36.683 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:57:36 +0000 (0:00:00.324) 0:05:16.985 **** 2021-02-10T10:57:52.702 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-02-10T10:57:52.703 INFO:teuthology.task.ansible.out:changed: [gibba002.front.sepia.ceph.com] => {"changed": true, "changes": {"installed": ["qemu-kvm", "mod_fastcgi-2.4.7-1.ceph.el7"]}, "msg": "", "rc": 0, "results": ["yum-plugin-priorities-1.1.31-54.el7_8.noarch providing yum-plugin-priorities is already installed", "yum-plugin-fastestmirror-1.1.31-54.el7_8.noarch providing yum-plugin-fastestmirror is already installed", "redhat-lsb-4.1-27.el7.x86_64 providing redhat-lsb is already installed", "sysstat-10.1.5-19.el7.x86_64 providing sysstat is already installed", "gdb-7.6.1-120.el7.x86_64 providing gdb is already installed", "git-all-1.8.3.1-23.el7_8.noarch providing git-all is already installed", "python-configobj-4.7.2-7.el7.noarch providing python-configobj is already installed", "gcc-c++-4.8.5-44.el7.x86_64 providing gcc-c++ is already installed", "libedit-3.0-12.20121213cvs.el7.x86_64 providing libedit is already installed", "openssl098e-0.9.8e-29.el7_2.3.x86_64 providing openssl098e is already installed", "boost-thread-1.53.0-28.el7.x86_64 providing boost-thread is already installed", "xfsprogs-4.5.0-22.el7.x86_64 providing xfsprogs is already installed", "xfsprogs-devel-4.5.0-22.el7.x86_64 providing xfsprogs-devel is already installed", "gdisk-0.8.10-3.el7.x86_64 providing gdisk is already installed", "parted-3.1-32.el7.x86_64 providing parted is already installed", "libgcrypt-1.5.3-14.el7.x86_64 providing libgcrypt is already installed", "fuse-2.9.2-11.el7.x86_64 providing fuse is already installed", "fuse-libs-2.9.2-11.el7.x86_64 providing fuse-libs is already installed", "7:lvm2-2.02.187-6.el7.x86_64 providing lvm2 is already installed", "1:openssl-1.0.2k-19.el7.x86_64 providing openssl is already installed", "libuuid-2.23.2-65.el7.x86_64 providing libuuid is already installed", "btrfs-progs-4.9.1-1.el7.x86_64 providing btrfs-progs is already installed", "attr-2.4.46-13.el7.x86_64 providing attr is already installed", "1:valgrind-3.15.0-11.el7.x86_64 providing valgrind is already installed", "python-nose-1 2021-02-10T10:57:52.705 INFO:teuthology.task.ansible.out:.3.7-1.el7.noarch providing python-nose is already installed", "mpich-3.0-3.0.4-10.el7.x86_64 providing mpich is already installed", "ant-1.9.4-2.el7.noarch providing ant is already installed", "lsof-4.87-6.el7.x86_64 providing lsof is already installed", "iozone-3.424-2_ceph.el7.x86_64 providing iozone is already installed", "libtool-2.4.2-22.el7_3.x86_64 providing libtool is already installed", "automake-1.13.4-3.el7.noarch providing automake is already installed", "gettext-0.19.8.1-3.el7.x86_64 providing gettext is already installed", "libuuid-devel-2.23.2-65.el7.x86_64 providing libuuid-devel is already installed", "libacl-devel-2.2.51-15.el7.x86_64 providing libacl-devel is already installed", "bc-1.06.95-13.el7.x86_64 providing bc is already installed", "xfsdump-3.1.7-1.el7.x86_64 providing xfsdump is already installed", "blktrace-1.0.5-9.el7.x86_64 providing blktrace is already installed", "1:numpy-1.7.1-13.el7.x86_64 providing numpy is already installed", "python-matplotlib-1.2.0-16.el7.x86_64 providing python-matplotlib is already installed", "usbredir-0.7.1-3.el7.x86_64 providing usbredir is already installed", "genisoimage-1.1.11-25.el7.x86_64 providing genisoimage is already installed", "httpd-2.4.6-97.el7_9.x86_64 providing httpd is already installed", "httpd-devel-2.4.6-97.el7_9.x86_64 providing httpd-devel is already installed", "httpd-tools-2.4.6-97.el7_9.x86_64 providing httpd-tools is already installed", "1:mod_ssl-2.4.6-97.el7_9.x86_64 providing mod_ssl is already installed", "libev-devel-4.15-7.el7.x86_64 providing libev-devel is already installed", "perl-XML-Twig-3.44-2.el7.noarch providing perl-XML-Twig is already installed", "1:java-1.6.0-openjdk-devel-1.6.0.41-1.13.13.1.el7_3.x86_64 providing java-1.6.0-openjdk-devel is already installed", "junit-4.11-8.el7.noarch providing junit4 is already installed", "1:nfs-utils-1.3.0-0.68.el7.x86_64 providing nfs-utils is already installed", "ncurses-devel-5.9-14.20130511.el7_4.x86_64 providing ncurses-d 2021-02-10T10:57:52.706 INFO:teuthology.task.ansible.out:evel is already installed", "python-devel-2.7.5-90.el7.x86_64 providing python-devel is already installed", "python-virtualenv-15.1.0-4.el7_8.noarch providing python-virtualenv is already installed", "perl-CPAN-1.9800-297.el7.noarch providing perl-CPAN is already installed", "podman-1.6.4-26.el7_9.x86_64 providing podman is already installed", "podman-docker-1.6.4-26.el7_9.noarch providing podman-docker is already installed", "python3-3.6.8-18.el7.x86_64 providing python3 is already installed", "Loaded plugins: fastestmirror, langpacks, priorities, product-id, search-\n : disabled-repos, subscription-manager\nLoading mirror speeds from cached hostfile\n * epel: download-cc-rdu01.fedoraproject.org\nResolving Dependencies\n--> Running transaction check\n---> Package mod_fastcgi.x86_64 0:2.4.7-1.ceph.el7 will be installed\n---> Package qemu-kvm.x86_64 10:1.5.3-175.el7_9.3 will be installed\n--> Processing Dependency: qemu-kvm-common = 10:1.5.3-175.el7_9.3 for package: 10:qemu-kvm-1.5.3-175.el7_9.3.x86_64\n--> Processing Dependency: qemu-img = 10:1.5.3-175.el7_9.3 for package: 10:qemu-kvm-1.5.3-175.el7_9.3.x86_64\n--> Processing Dependency: librbd.so.1()(64bit) for package: 10:qemu-kvm-1.5.3-175.el7_9.3.x86_64\n--> Processing Dependency: librados.so.2()(64bit) for package: 10:qemu-kvm-1.5.3-175.el7_9.3.x86_64\n--> Running transaction check\n---> Package librados2.x86_64 1:10.2.5-4.el7 will be installed\n--> Processing Dependency: libboost_random-mt.so.1.53.0()(64bit) for package: 1:librados2-10.2.5-4.el7.x86_64\n---> Package librbd1.x86_64 1:10.2.5-4.el7 will be installed\n---> Package qemu-img.x86_64 10:1.5.3-175.el7_9.3 will be installed\n---> Package qemu-kvm-common.x86_64 10:1.5.3-175.el7_9.3 will be installed\n--> Running transaction check\n---> Package boost-random.x86_64 0:1.53.0-28.el7 will be installed\n--> Finished Dependency Resolution\n\nDependencies Resolved\n\n================================================================================\n P 2021-02-10T10:57:52.707 INFO:teuthology.task.ansible.out:ackage Arch Version Repository Size\n================================================================================\nInstalling:\n mod_fastcgi x86_64 2.4.7-1.ceph.el7 rhel-7-fcgi-ceph 70 k\n qemu-kvm x86_64 10:1.5.3-175.el7_9.3 rhel-7-server-rpms 1.9 M\nInstalling for dependencies:\n boost-random x86_64 1.53.0-28.el7 rhel-7-server-rpms 39 k\n librados2 x86_64 1:10.2.5-4.el7 rhel-7-server-rpms 1.8 M\n librbd1 x86_64 1:10.2.5-4.el7 rhel-7-server-rpms 2.4 M\n qemu-img x86_64 10:1.5.3-175.el7_9.3 rhel-7-server-rpms 704 k\n qemu-kvm-common x86_64 10:1.5.3-175.el7_9.3 rhel-7-server-rpms 440 k\n\nTransaction Summary\n================================================================================\nInstall 2 Packages (+5 Dependent packages)\n\nTotal download size: 7.3 M\nInstalled size: 22 M\nDownloading packages:\n--------------------------------------------------------------------------------\nTotal 28 MB/s | 7.3 MB 00:00 \nRunning transaction check\nRunning transaction test\nTransaction test succeeded\nRunning transaction\n Installing : boost-random-1.53.0-28.el7.x86_64 1/7 \n Installing : 1:librados2-10.2.5-4.el7.x86_64 2/7 \n Installing : 1:librbd1-10.2.5-4.el7.x86_64 3/7 \n Installing : 10:qemu-img-1.5.3-175.el7_9.3.x86_64 4/7 \n Installing : 10:qemu-kvm-common-1.5.3-175.el7_9.3.x86_64 5/7 \n Installing : 10:qemu-kvm-1.5.3-175.el7_9.3.x86_64 6/7 \n Installing : mod_fastcgi-2.4.7-1.ceph.el7.x86_64 7/7 \nLoaded plugins: fastestmirror, langpacks, priorities, product-id, subscription-\n : manager\n Verifying : 1:librbd1-10.2.5-4.el7.x86_64 2021-02-10T10:57:52.709 INFO:teuthology.task.ansible.out: 1/7 \n Verifying : mod_fastcgi-2.4.7-1.ceph.el7.x86_64 2/7 \n Verifying : 10:qemu-img-1.5.3-175.el7_9.3.x86_64 3/7 \n Ve 2021-02-10T10:57:52.709 INFO:teuthology.task.ansible.out:rifying : boost-random-1.53.0-28.el7.x86_64 4/7 \n Verifying : 1:librados2-10.2.5-4.el7.x86_64 5/7 \n Verifying : 10:qemu-kvm-1.5.3-175.el7_9.3.x86_64 6/7 \n Verifying : 10:qemu-kvm-common-1.5.3-175.el7_9.3.x86_64 7/7 \n\nInstalled:\n mod_fastcgi.x86_64 0:2.4.7-1.ceph.el7 qemu-kvm.x86_64 10:1.5.3-175.el7_9.3 \n\nDependency Installed:\n boost-random.x86_64 0:1.53.0-28.el7 \n librados2.x86_64 1:10.2.5-4.el7 \n librbd1.x86_64 1:10.2.5-4.el7 \n qemu-img.x86_64 10:1.5.3-175.el7_9.3 \n qemu-kvm-common.x86_64 10:1.5.3-175.el7_9.3 \n\nComplete!\n"]} 2021-02-10T10:57:52.756 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:57:52 +0000 (0:00:16.072) 0:05:33.058 **** 2021-02-10T10:57:52.879 INFO:teuthology.task.ansible.out:changed: [gibba013.front.sepia.ceph.com] => {"changed": true, "changes": {"installed": ["qemu-kvm", "mod_fastcgi-2.4.7-1.ceph.el7"]}, "msg": "", "rc": 0, "results": ["yum-plugin-priorities-1.1.31-54.el7_8.noarch providing yum-plugin-priorities is already installed", "yum-plugin-fastestmirror-1.1.31-54.el7_8.noarch providing yum-plugin-fastestmirror is already installed", "redhat-lsb-4.1-27.el7.x86_64 providing redhat-lsb is already installed", "sysstat-10.1.5-19.el7.x86_64 providing sysstat is already installed", "gdb-7.6.1-120.el7.x86_64 providing gdb is already installed", "git-all-1.8.3.1-23.el7_8.noarch providing git-all is already installed", "python-configobj-4.7.2-7.el7.noarch providing python-configobj is already installed", "gcc-c++-4.8.5-44.el7.x86_64 providing gcc-c++ is already installed", "libedit-3.0-12.20121213cvs.el7.x86_64 providing libedit is already installed", "openssl098e-0.9.8e-29.el7_2.3.x86_64 providing openssl098e is already installed", "boost-thread-1.53.0-28.el7.x86_64 providing boost-thread is already installed", "xfsprogs-4.5.0-22.el7.x86_64 providing xfsprogs is already installed", "xfsprogs-devel-4.5.0-22.el7.x86_64 providing xfsprogs-devel is already installed", "gdisk-0.8.10-3.el7.x86_64 providing gdisk is already installed", "parted-3.1-32.el7.x86_64 providing parted is already installed", "libgcrypt-1.5.3-14.el7.x86_64 providing libgcrypt is already installed", "fuse-2.9.2-11.el7.x86_64 providing fuse is already installed", "fuse-libs-2.9.2-11.el7.x86_64 providing fuse-libs is already installed", "7:lvm2-2.02.187-6.el7.x86_64 providing lvm2 is already installed", "1:openssl-1.0.2k-19.el7.x86_64 providing openssl is already installed", "libuuid-2.23.2-65.el7.x86_64 providing libuuid is already installed", "btrfs-progs-4.9.1-1.el7.x86_64 providing btrfs-progs is already installed", "attr-2.4.46-13.el7.x86_64 providing attr is already installed", "1:valgrind-3.15.0-11.el7.x86_64 providing valgrind is already installed", "python-nose-1 2021-02-10T10:57:52.881 INFO:teuthology.task.ansible.out:.3.7-1.el7.noarch providing python-nose is already installed", "mpich-3.0-3.0.4-10.el7.x86_64 providing mpich is already installed", "ant-1.9.4-2.el7.noarch providing ant is already installed", "lsof-4.87-6.el7.x86_64 providing lsof is already installed", "iozone-3.424-2_ceph.el7.x86_64 providing iozone is already installed", "libtool-2.4.2-22.el7_3.x86_64 providing libtool is already installed", "automake-1.13.4-3.el7.noarch providing automake is already installed", "gettext-0.19.8.1-3.el7.x86_64 providing gettext is already installed", "libuuid-devel-2.23.2-65.el7.x86_64 providing libuuid-devel is already installed", "libacl-devel-2.2.51-15.el7.x86_64 providing libacl-devel is already installed", "bc-1.06.95-13.el7.x86_64 providing bc is already installed", "xfsdump-3.1.7-1.el7.x86_64 providing xfsdump is already installed", "blktrace-1.0.5-9.el7.x86_64 providing blktrace is already installed", "1:numpy-1.7.1-13.el7.x86_64 providing numpy is already installed", "python-matplotlib-1.2.0-16.el7.x86_64 providing python-matplotlib is already installed", "usbredir-0.7.1-3.el7.x86_64 providing usbredir is already installed", "genisoimage-1.1.11-25.el7.x86_64 providing genisoimage is already installed", "httpd-2.4.6-97.el7_9.x86_64 providing httpd is already installed", "httpd-devel-2.4.6-97.el7_9.x86_64 providing httpd-devel is already installed", "httpd-tools-2.4.6-97.el7_9.x86_64 providing httpd-tools is already installed", "1:mod_ssl-2.4.6-97.el7_9.x86_64 providing mod_ssl is already installed", "libev-devel-4.15-7.el7.x86_64 providing libev-devel is already installed", "perl-XML-Twig-3.44-2.el7.noarch providing perl-XML-Twig is already installed", "1:java-1.6.0-openjdk-devel-1.6.0.41-1.13.13.1.el7_3.x86_64 providing java-1.6.0-openjdk-devel is already installed", "junit-4.11-8.el7.noarch providing junit4 is already installed", "1:nfs-utils-1.3.0-0.68.el7.x86_64 providing nfs-utils is already installed", "ncurses-devel-5.9-14.20130511.el7_4.x86_64 providing ncurses-d 2021-02-10T10:57:52.882 INFO:teuthology.task.ansible.out:evel is already installed", "python-devel-2.7.5-90.el7.x86_64 providing python-devel is already installed", "python-virtualenv-15.1.0-4.el7_8.noarch providing python-virtualenv is already installed", "perl-CPAN-1.9800-297.el7.noarch providing perl-CPAN is already installed", "podman-1.6.4-26.el7_9.x86_64 providing podman is already installed", "podman-docker-1.6.4-26.el7_9.noarch providing podman-docker is already installed", "python3-3.6.8-18.el7.x86_64 providing python3 is already installed", "Loaded plugins: fastestmirror, langpacks, priorities, product-id, search-\n : disabled-repos, subscription-manager\nLoading mirror speeds from cached hostfile\n * epel: download-cc-rdu01.fedoraproject.org\nResolving Dependencies\n--> Running transaction check\n---> Package mod_fastcgi.x86_64 0:2.4.7-1.ceph.el7 will be installed\n---> Package qemu-kvm.x86_64 10:1.5.3-175.el7_9.3 will be installed\n--> Processing Dependency: qemu-kvm-common = 10:1.5.3-175.el7_9.3 for package: 10:qemu-kvm-1.5.3-175.el7_9.3.x86_64\n--> Processing Dependency: qemu-img = 10:1.5.3-175.el7_9.3 for package: 10:qemu-kvm-1.5.3-175.el7_9.3.x86_64\n--> Processing Dependency: librbd.so.1()(64bit) for package: 10:qemu-kvm-1.5.3-175.el7_9.3.x86_64\n--> Processing Dependency: librados.so.2()(64bit) for package: 10:qemu-kvm-1.5.3-175.el7_9.3.x86_64\n--> Running transaction check\n---> Package librados2.x86_64 1:10.2.5-4.el7 will be installed\n--> Processing Dependency: libboost_random-mt.so.1.53.0()(64bit) for package: 1:librados2-10.2.5-4.el7.x86_64\n---> Package librbd1.x86_64 1:10.2.5-4.el7 will be installed\n---> Package qemu-img.x86_64 10:1.5.3-175.el7_9.3 will be installed\n---> Package qemu-kvm-common.x86_64 10:1.5.3-175.el7_9.3 will be installed\n--> Running transaction check\n---> Package boost-random.x86_64 0:1.53.0-28.el7 will be installed\n--> Finished Dependency Resolution\n\nDependencies Resolved\n\n================================================================================\n P 2021-02-10T10:57:52.884 INFO:teuthology.task.ansible.out:ackage Arch Version Repository Size\n================================================================================\nInstalling:\n mod_fastcgi x86_64 2.4.7-1.ceph.el7 rhel-7-fcgi-ceph 70 k\n qemu-kvm x86_64 10:1.5.3-175.el7_9.3 rhel-7-server-rpms 1.9 M\nInstalling for dependencies:\n boost-random x86_64 1.53.0-28.el7 rhel-7-server-rpms 39 k\n librados2 x86_64 1:10.2.5-4.el7 rhel-7-server-rpms 1.8 M\n librbd1 x86_64 1:10.2.5-4.el7 rhel-7-server-rpms 2.4 M\n qemu-img x86_64 10:1.5.3-175.el7_9.3 rhel-7-server-rpms 704 k\n qemu-kvm-common x86_64 10:1.5.3-175.el7_9.3 rhel-7-server-rpms 440 k\n\nTransaction Summary\n================================================================================\nInstall 2 Packages (+5 Dependent packages)\n\nTotal download size: 7.3 M\nInstalled size: 22 M\nDownloading packages:\n--------------------------------------------------------------------------------\nTotal 31 MB/s | 7.3 MB 00:00 \nRunning transaction check\nRunning transaction test\nTransaction test succeeded\nRunning transaction\n Installing : boost-random-1.53.0-28.el7.x86_64 1/7 \n Installing : 1:librados2-10.2.5-4.el7.x86_64 2/7 \n Installing : 1:librbd1-10.2.5-4.el7.x86_64 3/7 \n Installing : 10:qemu-img-1.5.3-175.el7_9.3.x86_64 4/7 \n Installing : 10:qemu-kvm-common-1.5.3-175.el7_9.3.x86_64 5/7 \n Installing : 10:qemu-kvm-1.5.3-175.el7_9.3.x86_64 6/7 \n Installing : mod_fastcgi-2.4.7-1.ceph.el7.x86_64 7/7 \nLoaded plugins: fastestmirror, langpacks, priorities, product-id, subscription-\n : manager\n Verifying : 1:librbd1-10.2.5-4.el7.x86_64 2021-02-10T10:57:52.885 INFO:teuthology.task.ansible.out: 1/7 \n Verifying : mod_fastcgi-2.4.7-1.ceph.el7.x86_64 2/7 \n Verifying : 10:qemu-img-1.5.3-175.el7_9.3.x86_64 3/7 \n Verifying : boost-random-1.53.0-28.el7.x86_64 4/7 \n Verifying : 1:librados2-10.2.5-4.el7.x86_64 5/7 \n Verifying : 10:qemu-kvm-1.5.3-175.el7_9.3.x86_64 6/7 \n Verifying : 10:qemu-kvm-common-1.5.3-175.el7_9.3.x86_64 7/7 \n\nInstalled:\n mod_fastcgi.x86_64 0:2.4.7-1.ceph.el7 qemu-kvm.x86_64 10:1.5.3-175.el7_9.3 \n\nDependency Installed:\n boost-random.x86_64 0:1.53.0-28.el7 \n librados2.x86_64 1:10.2.5-4.el7 \n librbd1.x86_64 1:10.2.5-4.el7 \n qemu-img.x86_64 10:1.5.3-175.el7_9.3 \n qemu-kvm-common.x86_64 10:1.5.3-175.el7_9.3 \n\nComplete!\n"]} 2021-02-10T10:57:52.918 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:57:52 +0000 (0:00:00.161) 0:05:33.219 **** 2021-02-10T10:57:57.224 INFO:teuthology.task.ansible.out: TASK [testnode : Install epel packages] **************************************** 2021-02-10T10:57:57.225 INFO:teuthology.task.ansible.out:changed: [gibba002.front.sepia.ceph.com] => {"changed": true, "changes": {"installed": ["leveldb", "xmlstarlet"]}, "msg": "", "rc": 0, "results": ["cryptopp-devel-5.6.2-10.el7.x86_64 providing cryptopp-devel is already installed", "cryptopp-5.6.2-10.el7.x86_64 providing cryptopp is already installed", "dbench-4.0-10.el7.x86_64 providing dbench is already installed", "fcgi-2.4.0-25.el7.x86_64 providing fcgi is already installed", "fuse-sshfs-2.10-1.el7.x86_64 providing fuse-sshfs is already installed", "1:perl-JSON-XS-3.01-2.el7.x86_64 providing perl-JSON-XS is already installed", "Loaded plugins: fastestmirror, langpacks, priorities, product-id, search-\n : disabled-repos, subscription-manager\nLoading mirror speeds from cached hostfile\n * epel: download-cc-rdu01.fedoraproject.org\nResolving Dependencies\n--> Running transaction check\n---> Package leveldb.x86_64 0:1.12.0-11.el7 will be installed\n---> Package xmlstarlet.x86_64 0:1.6.1-1.el7 will be installed\n--> Finished Dependency Resolution\n\nDependencies Resolved\n\n================================================================================\n Package Arch Version Repository Size\n================================================================================\nInstalling:\n leveldb x86_64 1.12.0-11.el7 epel 161 k\n xmlstarlet x86_64 1.6.1-1.el7 epel 62 k\n\nTransaction Summary\n================================================================================\nInstall 2 Packages\n\nTotal download size: 223 k\nInstalled size: 631 k\nDownloading packages:\n--------------------------------------------------------------------------------\nTotal 3.9 MB/s | 223 kB 00:00 \nRunning transaction check\nRunning transaction test\nTransaction test succeeded\nRunning transaction\n Installing : xmlstarlet-1.6.1-1.el7.x86_64 2021-02-10T10:57:57.227 INFO:teuthology.task.ansible.out: 1/2 \n Installing : leveldb-1.12.0-11.el7.x86_64 2/2 \nLoaded plugins: fastestmirror, langpacks, priorities, product-id, subscription-\n : manager\n Verifying : leveldb-1.12.0-11.el7.x86_64 1/2 \n Verifying : xmlstarlet-1.6.1-1.el7.x86_64 2/2 \n\nInstalled:\n leveldb.x86_64 0:1.12.0-11.el7 xmlstarlet.x86_64 0:1.6.1-1.el7 \n\nComplete!\n"]} 2021-02-10T10:57:57.258 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:57:57 +0000 (0:00:04.340) 0:05:37.560 **** 2021-02-10T10:57:57.453 INFO:teuthology.task.ansible.out:changed: [gibba013.front.sepia.ceph.com] => {"changed": true, "changes": {"installed": ["leveldb", "xmlstarlet"]}, "msg": "", "rc": 0, "results": ["cryptopp-devel-5.6.2-10.el7.x86_64 providing cryptopp-devel is already installed", "cryptopp-5.6.2-10.el7.x86_64 providing cryptopp is already installed", "dbench-4.0-10.el7.x86_64 providing dbench is already installed", "fcgi-2.4.0-25.el7.x86_64 providing fcgi is already installed", "fuse-sshfs-2.10-1.el7.x86_64 providing fuse-sshfs is already installed", "1:perl-JSON-XS-3.01-2.el7.x86_64 providing perl-JSON-XS is already installed", "Loaded plugins: fastestmirror, langpacks, priorities, product-id, search-\n : disabled-repos, subscription-manager\nLoading mirror speeds from cached hostfile\n * epel: download-cc-rdu01.fedoraproject.org\nResolving Dependencies\n--> Running transaction check\n---> Package leveldb.x86_64 0:1.12.0-11.el7 will be installed\n---> Package xmlstarlet.x86_64 0:1.6.1-1.el7 will be installed\n--> Finished Dependency Resolution\n\nDependencies Resolved\n\n================================================================================\n Package Arch Version Repository Size\n================================================================================\nInstalling:\n leveldb x86_64 1.12.0-11.el7 epel 161 k\n xmlstarlet x86_64 1.6.1-1.el7 epel 62 k\n\nTransaction Summary\n================================================================================\nInstall 2 Packages\n\nTotal download size: 223 k\nInstalled size: 631 k\nDownloading packages:\n--------------------------------------------------------------------------------\nTotal 5.6 MB/s | 223 kB 00:00 \nRunning transaction check\nRunning transaction test\nTransaction test succeeded\nRunning transaction\n Installing : xmlstarlet-1.6.1-1.el7.x86_64 2021-02-10T10:57:57.455 INFO:teuthology.task.ansible.out: 1/2 \n Installing : leveldb-1.12.0-11.el7.x86_64 2/2 \nLoaded plugins: fastestmirror, langpacks, priorities, product-id, subscription-\n : manager\n Verifying : leveldb-1.12.0-11.el7.x86_64 1/2 \n Verifying : xmlstarlet-1.6.1-1.el7.x86_64 2/2 \n\nInstalled:\n leveldb.x86_64 0:1.12.0-11.el7 xmlstarlet.x86_64 0:1.6.1-1.el7 \n\nComplete!\n"]} 2021-02-10T10:57:57.488 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:57:57 +0000 (0:00:00.230) 0:05:37.790 **** 2021-02-10T10:57:58.111 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2021-02-10T10:57:58.113 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => {"changed": false, "msg": "", "rc": 0, "results": ["device-mapper-multipath is not installed"]} 2021-02-10T10:57:58.147 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:57:58 +0000 (0:00:00.658) 0:05:38.449 **** 2021-02-10T10:57:58.174 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-02-10T10:57:58.176 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:57:58.211 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:57:58 +0000 (0:00:00.064) 0:05:38.513 **** 2021-02-10T10:57:58.366 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2021-02-10T10:57:58.368 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => {"changed": false, "msg": "", "rc": 0, "results": ["device-mapper-multipath is not installed"]} 2021-02-10T10:57:58.402 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:57:58 +0000 (0:00:00.190) 0:05:38.704 **** 2021-02-10T10:57:58.429 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-02-10T10:57:58.430 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:57:58.462 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:57:58 +0000 (0:00:00.060) 0:05:38.764 **** 2021-02-10T10:57:58.719 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable firewalld] *********************************** 2021-02-10T10:57:58.719 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => {"changed": false, "enabled": false, "name": "firewalld", "state": "stopped", "status": {"ActiveEnterTimestampMonotonic": "0", "ActiveExitTimestampMonotonic": "0", "ActiveState": "inactive", "After": "polkit.service system.slice basic.target dbus.service", "AllowIsolate": "no", "AmbientCapabilities": "0", "AssertResult": "no", "AssertTimestampMonotonic": "0", "Before": "network-pre.target shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "18446744073709551615", "BusName": "org.fedoraproject.FirewallD1", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "18446744073709551615", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "18446744073709551615", "CollectMode": "inactive", "ConditionResult": "no", "ConditionTimestampMonotonic": "0", "Conflicts": "ipset.service iptables.service ebtables.service ip6tables.service shutdown.target", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "firewalld - dynamic firewall daemon", "DevicePolicy": "auto", "Documentation": "man:firewalld(1)", "EnvironmentFile": "/etc/sysconfig/firewalld (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "0", "ExecMainStartTimestampMonotonic": "0", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/firewalld ; argv[]=/usr/sbin/firewalld --nofork --nopid $FIREWALLD_ARGS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/firewalld.service", "GuessMainPID": "yes", "IOScheduling": "0", "Id": "firewalld.service", "Ign 2021-02-10T10:57:58.721 INFO:teuthology.task.ansible.out:oreOnIsolate": "no", "IgnoreOnSnapshot": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestampMonotonic": "0", "JobTimeoutAction": "none", "JobTimeoutUSec": "0", "KillMode": "mixed", "KillSignal": "15", "LimitAS": "18446744073709551615", "LimitCORE": "18446744073709551615", "LimitCPU": "18446744073709551615", "LimitDATA": "18446744073709551615", "LimitFSIZE": "18446744073709551615", "LimitLOCKS": "18446744073709551615", "LimitMEMLOCK": "65536", "LimitMSGQUEUE": "819200", "LimitNICE": "0", "LimitNOFILE": "4096", "LimitNPROC": "127560", "LimitRSS": "18446744073709551615", "LimitRTPRIO": "0", "LimitRTTIME": "18446744073709551615", "LimitSIGPENDING": "127560", "LimitSTACK": "18446744073709551615", "LoadState": "loaded", "MainPID": "0", "MemoryAccounting": "no", "MemoryCurrent": "18446744073709551615", "MemoryLimit": "18446744073709551615", "MountFlags": "0", "Names": "firewalld.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "no", "ProtectHome": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "Requires": "system.slice basic.target", "Restart": "no", "RestartUSec": "100ms", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "null", "StandardInput": "null", "StandardOutput": "null", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitInterval": "10000000", "StartupBlockIOWeight": "18446744073709551615", "StartupCPUShares": "18446744073709551615", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "dead", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", 2021-02-10T10:57:58.722 INFO:teuthology.task.ansible.out:"TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "no", "TasksCurrent": "18446744073709551615", "TasksMax": "18446744073709551615", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "dbus", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "disabled", "Wants": "network-pre.target", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} 2021-02-10T10:57:58.755 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:57:58 +0000 (0:00:00.292) 0:05:39.056 **** 2021-02-10T10:57:58.961 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => {"changed": false, "enabled": false, "name": "firewalld", "state": "stopped", "status": {"ActiveEnterTimestampMonotonic": "0", "ActiveExitTimestampMonotonic": "0", "ActiveState": "inactive", "After": "system.slice polkit.service basic.target dbus.service", "AllowIsolate": "no", "AmbientCapabilities": "0", "AssertResult": "no", "AssertTimestampMonotonic": "0", "Before": "network-pre.target shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "18446744073709551615", "BusName": "org.fedoraproject.FirewallD1", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "18446744073709551615", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "18446744073709551615", "CollectMode": "inactive", "ConditionResult": "no", "ConditionTimestampMonotonic": "0", "Conflicts": "iptables.service ip6tables.service ipset.service shutdown.target ebtables.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "firewalld - dynamic firewall daemon", "DevicePolicy": "auto", "Documentation": "man:firewalld(1)", "EnvironmentFile": "/etc/sysconfig/firewalld (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "0", "ExecMainStartTimestampMonotonic": "0", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/firewalld ; argv[]=/usr/sbin/firewalld --nofork --nopid $FIREWALLD_ARGS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/firewalld.service", "GuessMainPID": "yes", "IOScheduling": "0", "Id": "firewalld.service", "Ign 2021-02-10T10:57:58.963 INFO:teuthology.task.ansible.out:oreOnIsolate": "no", "IgnoreOnSnapshot": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestampMonotonic": "0", "JobTimeoutAction": "none", "JobTimeoutUSec": "0", "KillMode": "mixed", "KillSignal": "15", "LimitAS": "18446744073709551615", "LimitCORE": "18446744073709551615", "LimitCPU": "18446744073709551615", "LimitDATA": "18446744073709551615", "LimitFSIZE": "18446744073709551615", "LimitLOCKS": "18446744073709551615", "LimitMEMLOCK": "65536", "LimitMSGQUEUE": "819200", "LimitNICE": "0", "LimitNOFILE": "4096", "LimitNPROC": "127560", "LimitRSS": "18446744073709551615", "LimitRTPRIO": "0", "LimitRTTIME": "18446744073709551615", "LimitSIGPENDING": "127560", "LimitSTACK": "18446744073709551615", "LoadState": "loaded", "MainPID": "0", "MemoryAccounting": "no", "MemoryCurrent": "18446744073709551615", "MemoryLimit": "18446744073709551615", "MountFlags": "0", "Names": "firewalld.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "no", "ProtectHome": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "Requires": "basic.target system.slice", "Restart": "no", "RestartUSec": "100ms", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "null", "StandardInput": "null", "StandardOutput": "null", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitInterval": "10000000", "StartupBlockIOWeight": "18446744073709551615", "StartupCPUShares": "18446744073709551615", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "dead", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", 2021-02-10T10:57:58.964 INFO:teuthology.task.ansible.out:"TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "no", "TasksCurrent": "18446744073709551615", "TasksMax": "18446744073709551615", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "dbus", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "disabled", "Wants": "network-pre.target", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} 2021-02-10T10:57:58.997 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:57:58 +0000 (0:00:00.241) 0:05:39.298 **** 2021-02-10T10:57:59.242 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable iptables] ************************************ 2021-02-10T10:57:59.246 INFO:teuthology.task.ansible.out:fatal: [gibba002.front.sepia.ceph.com]: FAILED! => {"changed": false, "msg": "Could not find the requested service iptables: host"} ...ignoring 2021-02-10T10:57:59.276 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:57:59 +0000 (0:00:00.279) 0:05:39.578 **** 2021-02-10T10:57:59.494 INFO:teuthology.task.ansible.out:fatal: [gibba013.front.sepia.ceph.com]: FAILED! => {"changed": false, "msg": "Could not find the requested service iptables: host"} ...ignoring 2021-02-10T10:57:59.529 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:57:59 +0000 (0:00:00.253) 0:05:39.831 **** 2021-02-10T10:58:00.302 INFO:teuthology.task.ansible.out: TASK [testnode : Enable SELinux] *********************************************** 2021-02-10T10:58:00.303 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => {"changed": false, "configfile": "/etc/selinux/config", "msg": "", "policy": "targeted", "reboot_required": false, "state": "permissive"} ok: [gibba013.front.sepia.ceph.com] => {"changed": false, "configfile": "/etc/selinux/config", "msg": "", "policy": "targeted", "reboot_required": false, "state": "permissive"} 2021-02-10T10:58:00.336 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:00 +0000 (0:00:00.806) 0:05:40.637 **** 2021-02-10T10:58:00.374 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:00 +0000 (0:00:00.038) 0:05:40.675 **** 2021-02-10T10:58:00.380 INFO:teuthology.task.ansible.out: TASK [testnode : Install abrt] ************************************************* 2021-02-10T10:58:00.381 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:00.414 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:00 +0000 (0:00:00.040) 0:05:40.716 **** 2021-02-10T10:58:00.416 INFO:teuthology.task.ansible.out: 2021-02-10T10:58:00.423 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:00.458 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:00 +0000 (0:00:00.043) 0:05:40.759 **** 2021-02-10T10:58:00.463 INFO:teuthology.task.ansible.out: TASK [testnode : Enable abrt-auto-reporting] *********************************** 2021-02-10T10:58:00.465 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:00.500 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:00 +0000 (0:00:00.042) 0:05:40.802 **** 2021-02-10T10:58:00.508 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:00.542 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:00 +0000 (0:00:00.041) 0:05:40.844 **** 2021-02-10T10:58:00.548 INFO:teuthology.task.ansible.out: TASK [testnode : Set OpenGPGCheck in abrt-action-save-package-data.conf] ******* 2021-02-10T10:58:00.548 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:00.582 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:00 +0000 (0:00:00.040) 0:05:40.884 **** 2021-02-10T10:58:00.588 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:00.622 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:00 +0000 (0:00:00.040) 0:05:40.924 **** 2021-02-10T10:58:00.628 INFO:teuthology.task.ansible.out: TASK [testnode : Set ProcessUnpackaged in abrt-action-save-package-data.conf] *** 2021-02-10T10:58:00.630 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:00.663 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:00 +0000 (0:00:00.040) 0:05:40.964 **** 2021-02-10T10:58:00.669 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:00.703 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:00 +0000 (0:00:00.040) 0:05:41.005 **** 2021-02-10T10:58:00.709 INFO:teuthology.task.ansible.out: TASK [testnode : Restart abrtd] ************************************************ 2021-02-10T10:58:00.710 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:00.744 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:00 +0000 (0:00:00.040) 0:05:41.045 **** 2021-02-10T10:58:00.751 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:00.785 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:00 +0000 (0:00:00.041) 0:05:41.086 **** 2021-02-10T10:58:00.791 INFO:teuthology.task.ansible.out: TASK [testnode : Check for custom repos] *************************************** 2021-02-10T10:58:00.791 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:00.825 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:00 +0000 (0:00:00.039) 0:05:41.126 **** 2021-02-10T10:58:00.832 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:00.868 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:00 +0000 (0:00:00.042) 0:05:41.169 **** 2021-02-10T10:58:00.874 INFO:teuthology.task.ansible.out: TASK [testnode : Remove custom repos] ****************************************** 2021-02-10T10:58:00.908 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:00 +0000 (0:00:00.040) 0:05:41.209 **** 2021-02-10T10:58:00.950 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:00 +0000 (0:00:00.042) 0:05:41.252 **** 2021-02-10T10:58:00.952 INFO:teuthology.task.ansible.out: 2021-02-10T10:58:00.957 INFO:teuthology.task.ansible.out: TASK [testnode : Set apt preferences] ****************************************** 2021-02-10T10:58:00.959 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:00.991 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:00 +0000 (0:00:00.041) 0:05:41.293 **** 2021-02-10T10:58:00.998 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:01.031 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:01 +0000 (0:00:00.039) 0:05:41.333 **** 2021-02-10T10:58:01.037 INFO:teuthology.task.ansible.out: TASK [testnode : Add sources list] ********************************************* 2021-02-10T10:58:01.038 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:01.071 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:01 +0000 (0:00:00.039) 0:05:41.373 **** 2021-02-10T10:58:01.078 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:01.112 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:01 +0000 (0:00:00.041) 0:05:41.414 **** 2021-02-10T10:58:01.118 INFO:teuthology.task.ansible.out: TASK [testnode : Install apt keys] ********************************************* 2021-02-10T10:58:01.119 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2021-02-10T10:58:01.120 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} 2021-02-10T10:58:01.157 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:01 +0000 (0:00:00.045) 0:05:41.459 **** 2021-02-10T10:58:01.163 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2021-02-10T10:58:01.164 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} 2021-02-10T10:58:01.198 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:01 +0000 (0:00:00.040) 0:05:41.499 **** 2021-02-10T10:58:01.205 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-apt] ******************************************* 2021-02-10T10:58:01.205 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:01.240 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:01 +0000 (0:00:00.042) 0:05:41.542 **** 2021-02-10T10:58:01.247 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:01.280 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:01 +0000 (0:00:00.040) 0:05:41.582 **** 2021-02-10T10:58:01.285 INFO:teuthology.task.ansible.out: TASK [testnode : Add local apt repos.] ***************************************** 2021-02-10T10:58:01.318 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:01 +0000 (0:00:00.037) 0:05:41.620 **** 2021-02-10T10:58:01.358 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:01 +0000 (0:00:00.039) 0:05:41.659 **** 2021-02-10T10:58:01.365 INFO:teuthology.task.ansible.out: TASK [testnode : Blast the apt cache.] ***************************************** 2021-02-10T10:58:01.366 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:01.398 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:01 +0000 (0:00:00.040) 0:05:41.700 **** 2021-02-10T10:58:01.404 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:01.438 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:01 +0000 (0:00:00.039) 0:05:41.739 **** 2021-02-10T10:58:01.443 INFO:teuthology.task.ansible.out: TASK [testnode : Update apt cache.] ******************************************** 2021-02-10T10:58:01.445 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:01.478 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:01 +0000 (0:00:00.040) 0:05:41.779 **** 2021-02-10T10:58:01.483 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:01.517 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:01 +0000 (0:00:00.038) 0:05:41.818 **** 2021-02-10T10:58:01.521 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure packages are not present.] ***************************** 2021-02-10T10:58:01.523 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:01.558 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:01 +0000 (0:00:00.041) 0:05:41.860 **** 2021-02-10T10:58:01.565 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:01.600 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:01 +0000 (0:00:00.042) 0:05:41.902 **** 2021-02-10T10:58:01.607 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-02-10T10:58:01.608 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:01.643 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:01 +0000 (0:00:00.042) 0:05:41.945 **** 2021-02-10T10:58:01.651 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:01.688 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:01 +0000 (0:00:00.044) 0:05:41.989 **** 2021-02-10T10:58:01.694 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade non aarch64 packages] ********************************* 2021-02-10T10:58:01.696 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:01.727 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:01 +0000 (0:00:00.039) 0:05:42.028 **** 2021-02-10T10:58:01.737 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:01.771 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:01 +0000 (0:00:00.044) 0:05:42.073 **** 2021-02-10T10:58:01.777 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-02-10T10:58:01.779 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:01.812 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:01 +0000 (0:00:00.040) 0:05:42.113 **** 2021-02-10T10:58:01.819 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:01.854 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:01 +0000 (0:00:00.042) 0:05:42.156 **** 2021-02-10T10:58:01.860 INFO:teuthology.task.ansible.out: TASK [testnode : Install non aarch64 packages] ********************************* 2021-02-10T10:58:01.861 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:01.898 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:01 +0000 (0:00:00.043) 0:05:42.200 **** 2021-02-10T10:58:01.905 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:01.942 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:01 +0000 (0:00:00.043) 0:05:42.243 **** 2021-02-10T10:58:01.957 INFO:teuthology.task.ansible.out: TASK [testnode : Use perl to add user_xattr to the root mount options in fstab.] *** 2021-02-10T10:58:01.958 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:01.991 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:01 +0000 (0:00:00.049) 0:05:42.292 **** 2021-02-10T10:58:01.999 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:02.037 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:02 +0000 (0:00:00.045) 0:05:42.338 **** 2021-02-10T10:58:02.044 INFO:teuthology.task.ansible.out: TASK [testnode : Enable xattr for this boot.] ********************************** 2021-02-10T10:58:02.044 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:02.081 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:02 +0000 (0:00:00.044) 0:05:42.383 **** 2021-02-10T10:58:02.089 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:02.131 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:02 +0000 (0:00:00.049) 0:05:42.432 **** 2021-02-10T10:58:02.136 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure fuse group exists.] ************************************ 2021-02-10T10:58:02.136 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:02.172 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:02 +0000 (0:00:00.041) 0:05:42.474 **** 2021-02-10T10:58:02.178 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:02.212 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:02 +0000 (0:00:00.039) 0:05:42.513 **** 2021-02-10T10:58:02.216 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/fuse.conf.] *************************************** 2021-02-10T10:58:02.217 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:02.250 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:02 +0000 (0:00:00.038) 0:05:42.552 **** 2021-02-10T10:58:02.256 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:02.297 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:02 +0000 (0:00:00.046) 0:05:42.599 **** 2021-02-10T10:58:02.303 INFO:teuthology.task.ansible.out: TASK [testnode : Add teuthology user to groups fuse, kvm and disk.] ************ 2021-02-10T10:58:02.305 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:02.338 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:02 +0000 (0:00:00.040) 0:05:42.639 **** 2021-02-10T10:58:02.345 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:02.379 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:02 +0000 (0:00:00.040) 0:05:42.680 **** 2021-02-10T10:58:02.385 INFO:teuthology.task.ansible.out: TASK [testnode : Set up static IP in /etc/hosts.] ****************************** 2021-02-10T10:58:02.386 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:02.419 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:02 +0000 (0:00:00.039) 0:05:42.720 **** 2021-02-10T10:58:02.425 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:02.460 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:02 +0000 (0:00:00.041) 0:05:42.761 **** 2021-02-10T10:58:02.465 INFO:teuthology.task.ansible.out: TASK [testnode : Stop apache2] ************************************************* 2021-02-10T10:58:02.467 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:02.500 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:02 +0000 (0:00:00.039) 0:05:42.801 **** 2021-02-10T10:58:02.505 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:02.539 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:02 +0000 (0:00:00.039) 0:05:42.841 **** 2021-02-10T10:58:02.546 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2021-02-10T10:58:02.547 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:02.580 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:02 +0000 (0:00:00.040) 0:05:42.882 **** 2021-02-10T10:58:02.587 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:02.620 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:02 +0000 (0:00:00.040) 0:05:42.922 **** 2021-02-10T10:58:02.626 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2021-02-10T10:58:02.626 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:02.660 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:02 +0000 (0:00:00.039) 0:05:42.962 **** 2021-02-10T10:58:02.667 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:02.701 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:02 +0000 (0:00:00.040) 0:05:43.002 **** 2021-02-10T10:58:02.707 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2021-02-10T10:58:02.709 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:02.747 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:02 +0000 (0:00:00.046) 0:05:43.049 **** 2021-02-10T10:58:02.754 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:02.792 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:02 +0000 (0:00:00.044) 0:05:43.093 **** 2021-02-10T10:58:02.798 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2021-02-10T10:58:02.799 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:02.832 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:02 +0000 (0:00:00.040) 0:05:43.134 **** 2021-02-10T10:58:02.839 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:02.873 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:02 +0000 (0:00:00.040) 0:05:43.175 **** 2021-02-10T10:58:02.880 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2021-02-10T10:58:02.882 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:02.915 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:02 +0000 (0:00:00.041) 0:05:43.216 **** 2021-02-10T10:58:02.921 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:02.956 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:02 +0000 (0:00:00.040) 0:05:43.257 **** 2021-02-10T10:58:02.961 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2021-02-10T10:58:02.962 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:02.996 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:02 +0000 (0:00:00.039) 0:05:43.297 **** 2021-02-10T10:58:03.005 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:03.038 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:03 +0000 (0:00:00.042) 0:05:43.340 **** 2021-02-10T10:58:03.044 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2021-02-10T10:58:03.046 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:03.079 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:03 +0000 (0:00:00.040) 0:05:43.381 **** 2021-02-10T10:58:03.085 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:03.120 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:03 +0000 (0:00:00.041) 0:05:43.422 **** 2021-02-10T10:58:03.127 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2021-02-10T10:58:03.129 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:03.162 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:03 +0000 (0:00:00.041) 0:05:43.464 **** 2021-02-10T10:58:03.169 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:03.203 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:03 +0000 (0:00:00.041) 0:05:43.505 **** 2021-02-10T10:58:03.209 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-02-10T10:58:03.209 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:03.244 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:03 +0000 (0:00:00.040) 0:05:43.545 **** 2021-02-10T10:58:03.250 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:03.285 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:03 +0000 (0:00:00.041) 0:05:43.587 **** 2021-02-10T10:58:03.292 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** skipping: [gibba002.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:03.327 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:03 +0000 (0:00:00.041) 0:05:43.628 **** 2021-02-10T10:58:03.335 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:03.371 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:03 +0000 (0:00:00.044) 0:05:43.673 **** 2021-02-10T10:58:03.377 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-02-10T10:58:03.379 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:03.413 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:03 +0000 (0:00:00.041) 0:05:43.714 **** 2021-02-10T10:58:03.420 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:03.454 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:03 +0000 (0:00:00.041) 0:05:43.756 **** 2021-02-10T10:58:03.460 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-02-10T10:58:03.460 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:03.495 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:03 +0000 (0:00:00.040) 0:05:43.796 **** 2021-02-10T10:58:03.502 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:03.536 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:03 +0000 (0:00:00.041) 0:05:43.838 **** 2021-02-10T10:58:03.542 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2021-02-10T10:58:03.544 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:03.577 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:03 +0000 (0:00:00.040) 0:05:43.878 **** 2021-02-10T10:58:03.583 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:03.622 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:03 +0000 (0:00:00.045) 0:05:43.924 **** 2021-02-10T10:58:03.628 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-02-10T10:58:03.629 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:03.666 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:03 +0000 (0:00:00.044) 0:05:43.968 **** 2021-02-10T10:58:03.675 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:03.714 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:03 +0000 (0:00:00.047) 0:05:44.015 **** 2021-02-10T10:58:03.721 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-02-10T10:58:03.722 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:03.761 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:03 +0000 (0:00:00.047) 0:05:44.063 **** 2021-02-10T10:58:03.769 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:03.806 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:03 +0000 (0:00:00.044) 0:05:44.107 **** 2021-02-10T10:58:03.812 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-02-10T10:58:03.813 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:03.848 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:03 +0000 (0:00:00.042) 0:05:44.150 **** 2021-02-10T10:58:03.855 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:03.894 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:03 +0000 (0:00:00.045) 0:05:44.196 **** 2021-02-10T10:58:05.833 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2021-02-10T10:58:05.834 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => {"changed": false, "checksum": "0b949a4622a315acfc90571fddbd013b4dc0f5f9", "dest": "/etc/exports", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/exports", "secontext": "system_u:object_r:exports_t:s0", "size": 578, "state": "file", "uid": 0} 2021-02-10T10:58:05.878 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:05 +0000 (0:00:01.983) 0:05:46.179 **** 2021-02-10T10:58:05.888 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => {"changed": false, "checksum": "0b949a4622a315acfc90571fddbd013b4dc0f5f9", "dest": "/etc/exports", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/exports", "secontext": "system_u:object_r:exports_t:s0", "size": 578, "state": "file", "uid": 0} 2021-02-10T10:58:05.934 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:05 +0000 (0:00:00.056) 0:05:46.236 **** 2021-02-10T10:58:06.392 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2021-02-10T10:58:06.393 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "nfs-server", "status": {"ActiveEnterTimestamp": "Wed 2021-02-10 10:51:13 UTC", "ActiveEnterTimestampMonotonic": "20194941", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "gssproxy.service proc-fs-nfsd.mount nfs-idmapd.service nfs-mountd.service rpc-gssd.service local-fs.target system.slice rpc-statd.service nfs-config.service systemd-journald.socket rpcbind.socket tmp.mount network-online.target -.mount", "AllowIsolate": "no", "AmbientCapabilities": "0", "AssertResult": "yes", "AssertTimestamp": "Wed 2021-02-10 10:51:12 UTC", "AssertTimestampMonotonic": "20006743", "Before": "rpc-statd-notify.service", "BlockIOAccounting": "no", "BlockIOWeight": "18446744073709551615", "BoundBy": "nfs-idmapd.service nfs-mountd.service", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "18446744073709551615", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "18446744073709551615", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Wed 2021-02-10 10:51:12 UTC", "ConditionTimestampMonotonic": "20006743", "ControlGroup": "/system.slice/nfs-server.service", "ControlPID": "0", "DefaultDependencies": "no", "Delegate": "no", "Description": "NFS server and services", "DevicePolicy": "auto", "DropInPaths": "/run/systemd/generator/nfs-server.service.d/order-with-mounts.conf", "EnvironmentFile": "/run/sysconfig/nfs-utils (ignore_errors=yes)", "ExecMainCode": "1", "ExecMainExitTimestamp": "Wed 2021-02-10 10:51:13 UTC", "ExecMainExitTimestampMonotonic": "20183188", "ExecMainPID": "1238", "ExecMainStartTimestamp": "Wed 2021-02-10 10:51:12 UTC", "ExecMainStartTimestampMonotonic": "20014701", "ExecMainStatus": "0", "ExecReload": "{ path=/usr/sbin/exportfs ; argv[]=/usr/sbin/exportfs -r ; ignore_errors=yes 2021-02-10T10:58:06.394 INFO:teuthology.task.ansible.out: ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/rpc.nfsd ; argv[]=/usr/sbin/rpc.nfsd $RPCNFSDARGS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStartPost": "{ path=/bin/sh ; argv[]=/bin/sh -c if systemctl -q is-active gssproxy; then systemctl reload gssproxy ; fi ; ignore_errors=yes ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStartPre": "{ path=/usr/sbin/exportfs ; argv[]=/usr/sbin/exportfs -r ; ignore_errors=yes ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStop": "{ path=/usr/sbin/rpc.nfsd ; argv[]=/usr/sbin/rpc.nfsd 0 ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/usr/sbin/exportfs ; argv[]=/usr/sbin/exportfs -f ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/nfs-server.service", "GuessMainPID": "yes", "IOScheduling": "0", "Id": "nfs-server.service", "IgnoreOnIsolate": "no", "IgnoreOnSnapshot": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Wed 2021-02-10 10:51:12 UTC", "InactiveExitTimestampMonotonic": "20007629", "JobTimeoutAction": "none", "JobTimeoutUSec": "0", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "18446744073709551615", "LimitCORE": "18446744073709551615", "LimitCPU": "18446744073709551615", "LimitDATA": "18446744073709551615", "LimitFSIZE": "18446744073709551615", "LimitLOCKS": "18446744073709551615", "LimitMEMLOCK": "65536", "LimitMSGQUEUE": "819200", "LimitNICE": "0", "LimitNOFILE": "4096", "LimitNPROC": "127560", "LimitRSS": "18446744073709551615", "LimitRTPRIO": "0", "LimitRTTIME": "18446744073709551615", "LimitSIGPENDING": "127560", "LimitSTACK": "18446744073709551615", "LoadState": "lo 2021-02-10T10:58:06.395 INFO:teuthology.task.ansible.out:aded", "MainPID": "0", "MemoryAccounting": "no", "MemoryCurrent": "18446744073709551615", "MemoryLimit": "18446744073709551615", "MountFlags": "0", "Names": "nfs-server.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "no", "ProtectHome": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "yes", "Requires": "proc-fs-nfsd.mount nfs-mountd.service network.target system.slice -.mount", "RequiresMountsFor": "/tmp", "Restart": "no", "RestartUSec": "100ms", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitInterval": "10000000", "StartupBlockIOWeight": "18446744073709551615", "StartupCPUShares": "18446744073709551615", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "exited", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "no", "TasksCurrent": "18446744073709551615", "TasksMax": "18446744073709551615", "TimeoutStartUSec": "0", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "oneshot", "UMask": "0022", "UnitFilePreset": "disabled", "UnitFileState": "enabled", "WantedBy": "multi-user.target", "Wants": "auth-rpcgss-module.service rpcbind.socket network-online.target nfs-idmapd.service rpc-statd-notify.service rpc-statd.service nfs-config.service", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} 2021-02-10T10:58:06.426 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:06 +0000 (0:00:00.491) 0:05:46.728 **** 2021-02-10T10:58:06.446 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "nfs-server", "status": {"ActiveEnterTimestamp": "Wed 2021-02-10 10:51:14 UTC", "ActiveEnterTimestampMonotonic": "19390162", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "local-fs.target tmp.mount nfs-idmapd.service system.slice rpc-statd.service proc-fs-nfsd.mount nfs-config.service rpcbind.socket nfs-mountd.service -.mount network-online.target rpc-gssd.service systemd-journald.socket gssproxy.service", "AllowIsolate": "no", "AmbientCapabilities": "0", "AssertResult": "yes", "AssertTimestamp": "Wed 2021-02-10 10:51:13 UTC", "AssertTimestampMonotonic": "19157865", "Before": "rpc-statd-notify.service", "BlockIOAccounting": "no", "BlockIOWeight": "18446744073709551615", "BoundBy": "nfs-mountd.service nfs-idmapd.service", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "18446744073709551615", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "18446744073709551615", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Wed 2021-02-10 10:51:13 UTC", "ConditionTimestampMonotonic": "19157865", "ControlGroup": "/system.slice/nfs-server.service", "ControlPID": "0", "DefaultDependencies": "no", "Delegate": "no", "Description": "NFS server and services", "DevicePolicy": "auto", "DropInPaths": "/run/systemd/generator/nfs-server.service.d/order-with-mounts.conf", "EnvironmentFile": "/run/sysconfig/nfs-utils (ignore_errors=yes)", "ExecMainCode": "1", "ExecMainExitTimestamp": "Wed 2021-02-10 10:51:14 UTC", "ExecMainExitTimestampMonotonic": "19353457", "ExecMainPID": "1250", "ExecMainStartTimestamp": "Wed 2021-02-10 10:51:14 UTC", "ExecMainStartTimestampMonotonic": "19224040", "ExecMainStatus": "0", "ExecReload": "{ path=/usr/sbin/exportfs ; argv[]=/usr/sbin/exportfs -r ; ignore_errors=yes 2021-02-10T10:58:06.447 INFO:teuthology.task.ansible.out: ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/rpc.nfsd ; argv[]=/usr/sbin/rpc.nfsd $RPCNFSDARGS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStartPost": "{ path=/bin/sh ; argv[]=/bin/sh -c if systemctl -q is-active gssproxy; then systemctl reload gssproxy ; fi ; ignore_errors=yes ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStartPre": "{ path=/usr/sbin/exportfs ; argv[]=/usr/sbin/exportfs -r ; ignore_errors=yes ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStop": "{ path=/usr/sbin/rpc.nfsd ; argv[]=/usr/sbin/rpc.nfsd 0 ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/usr/sbin/exportfs ; argv[]=/usr/sbin/exportfs -f ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/nfs-server.service", "GuessMainPID": "yes", "IOScheduling": "0", "Id": "nfs-server.service", "IgnoreOnIsolate": "no", "IgnoreOnSnapshot": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Wed 2021-02-10 10:51:14 UTC", "InactiveExitTimestampMonotonic": "19158264", "JobTimeoutAction": "none", "JobTimeoutUSec": "0", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "18446744073709551615", "LimitCORE": "18446744073709551615", "LimitCPU": "18446744073709551615", "LimitDATA": "18446744073709551615", "LimitFSIZE": "18446744073709551615", "LimitLOCKS": "18446744073709551615", "LimitMEMLOCK": "65536", "LimitMSGQUEUE": "819200", "LimitNICE": "0", "LimitNOFILE": "4096", "LimitNPROC": "127560", "LimitRSS": "18446744073709551615", "LimitRTPRIO": "0", "LimitRTTIME": "18446744073709551615", "LimitSIGPENDING": "127560", "LimitSTACK": "18446744073709551615", "LoadState": "lo 2021-02-10T10:58:06.449 INFO:teuthology.task.ansible.out:aded", "MainPID": "0", "MemoryAccounting": "no", "MemoryCurrent": "18446744073709551615", "MemoryLimit": "18446744073709551615", "MountFlags": "0", "Names": "nfs-server.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "no", "ProtectHome": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "yes", "Requires": "system.slice network.target proc-fs-nfsd.mount nfs-mountd.service -.mount", "RequiresMountsFor": "/tmp", "Restart": "no", "RestartUSec": "100ms", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitInterval": "10000000", "StartupBlockIOWeight": "18446744073709551615", "StartupCPUShares": "18446744073709551615", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "exited", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "no", "TasksCurrent": "18446744073709551615", "TasksMax": "18446744073709551615", "TimeoutStartUSec": "0", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "oneshot", "UMask": "0022", "UnitFilePreset": "disabled", "UnitFileState": "enabled", "WantedBy": "multi-user.target", "Wants": "rpc-statd-notify.service nfs-config.service network-online.target rpcbind.socket nfs-idmapd.service rpc-statd.service auth-rpcgss-module.service", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} 2021-02-10T10:58:06.481 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:06 +0000 (0:00:00.055) 0:05:46.783 **** 2021-02-10T10:58:06.483 INFO:teuthology.task.ansible.out: 2021-02-10T10:58:06.488 INFO:teuthology.task.ansible.out: 2021-02-10T10:58:06.490 INFO:teuthology.task.ansible.out:TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* skipping: [gibba002.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:06.524 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:06 +0000 (0:00:00.042) 0:05:46.826 **** 2021-02-10T10:58:06.532 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:06.565 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:06 +0000 (0:00:00.041) 0:05:46.867 **** 2021-02-10T10:58:06.572 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-02-10T10:58:06.572 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:06.609 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:06 +0000 (0:00:00.043) 0:05:46.910 **** 2021-02-10T10:58:06.616 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:06.652 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:06 +0000 (0:00:00.043) 0:05:46.954 **** 2021-02-10T10:58:06.660 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-02-10T10:58:06.661 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:06.696 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:06 +0000 (0:00:00.044) 0:05:46.998 **** 2021-02-10T10:58:06.706 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:06.743 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:06 +0000 (0:00:00.047) 0:05:47.045 **** 2021-02-10T10:58:06.751 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-02-10T10:58:06.751 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:06.795 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:06 +0000 (0:00:00.051) 0:05:47.097 **** 2021-02-10T10:58:06.803 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:06.839 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:06 +0000 (0:00:00.043) 0:05:47.141 **** 2021-02-10T10:58:06.845 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-02-10T10:58:06.846 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:06.880 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:06 +0000 (0:00:00.040) 0:05:47.182 **** 2021-02-10T10:58:06.887 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:06.922 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:06 +0000 (0:00:00.042) 0:05:47.224 **** 2021-02-10T10:58:06.930 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-02-10T10:58:06.931 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:06.965 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:06 +0000 (0:00:00.043) 0:05:47.267 **** 2021-02-10T10:58:06.973 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:07.007 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:07 +0000 (0:00:00.041) 0:05:47.309 **** 2021-02-10T10:58:07.014 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-02-10T10:58:07.015 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:07.049 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:07 +0000 (0:00:00.042) 0:05:47.351 **** 2021-02-10T10:58:07.056 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:07.090 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:07 +0000 (0:00:00.040) 0:05:47.392 **** 2021-02-10T10:58:07.096 INFO:teuthology.task.ansible.out: TASK [testnode : Set grub config.] ********************************************* 2021-02-10T10:58:07.098 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:07.143 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:07 +0000 (0:00:00.052) 0:05:47.444 **** 2021-02-10T10:58:07.152 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:07.190 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:07 +0000 (0:00:00.047) 0:05:47.492 **** 2021-02-10T10:58:07.196 INFO:teuthology.task.ansible.out: TASK [testnode : Remove /etc/ceph] ********************************************* 2021-02-10T10:58:07.198 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:07.234 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:07 +0000 (0:00:00.043) 0:05:47.535 **** 2021-02-10T10:58:07.240 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:07.279 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:07 +0000 (0:00:00.045) 0:05:47.580 **** 2021-02-10T10:58:07.285 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2021-02-10T10:58:07.285 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:07.320 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:07 +0000 (0:00:00.041) 0:05:47.622 **** 2021-02-10T10:58:07.329 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:07.362 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:07 +0000 (0:00:00.041) 0:05:47.664 **** 2021-02-10T10:58:07.368 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2021-02-10T10:58:07.370 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:07.404 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:07 +0000 (0:00:00.041) 0:05:47.706 **** 2021-02-10T10:58:07.411 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:07.446 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:07 +0000 (0:00:00.041) 0:05:47.747 **** 2021-02-10T10:58:07.452 INFO:teuthology.task.ansible.out: TASK [testnode : Upload weekly kernel-clean crontab.] ************************** 2021-02-10T10:58:07.452 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:07.487 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:07 +0000 (0:00:00.041) 0:05:47.789 **** 2021-02-10T10:58:07.495 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:07.530 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:07 +0000 (0:00:00.043) 0:05:47.832 **** 2021-02-10T10:58:07.538 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/grub.d/02_force_timeout.] ************************* 2021-02-10T10:58:07.539 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:07.573 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:07 +0000 (0:00:00.042) 0:05:47.874 **** 2021-02-10T10:58:07.580 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:07.615 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:07 +0000 (0:00:00.042) 0:05:47.917 **** 2021-02-10T10:58:07.623 INFO:teuthology.task.ansible.out: TASK [testnode : Enable kernel modules to load at boot time.] ****************** 2021-02-10T10:58:07.625 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:07.659 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:07 +0000 (0:00:00.043) 0:05:47.961 **** 2021-02-10T10:58:07.666 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:07.702 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:07 +0000 (0:00:00.042) 0:05:48.003 **** 2021-02-10T10:58:07.708 INFO:teuthology.task.ansible.out: TASK [testnode : Enabling auto-fsck fix to prevent boot hangup.] *************** 2021-02-10T10:58:07.709 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:07.751 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:07 +0000 (0:00:00.049) 0:05:48.053 **** 2021-02-10T10:58:07.759 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:07.795 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:07 +0000 (0:00:00.043) 0:05:48.096 **** 2021-02-10T10:58:07.801 INFO:teuthology.task.ansible.out: TASK [testnode : Work around broken wget on wheezy.] *************************** 2021-02-10T10:58:07.802 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:07.837 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:07 +0000 (0:00:00.042) 0:05:48.138 **** 2021-02-10T10:58:07.847 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:07.882 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:07 +0000 (0:00:00.044) 0:05:48.183 **** 2021-02-10T10:58:07.887 INFO:teuthology.task.ansible.out: TASK [testnode : Stop collectl] ************************************************ 2021-02-10T10:58:07.888 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:07.921 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:07 +0000 (0:00:00.039) 0:05:48.223 **** 2021-02-10T10:58:07.927 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:07.960 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:07 +0000 (0:00:00.038) 0:05:48.262 **** 2021-02-10T10:58:07.965 INFO:teuthology.task.ansible.out: TASK [testnode : Add PATH to the teuthology_user .bashrc.] ********************* 2021-02-10T10:58:07.966 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:07.999 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:07 +0000 (0:00:00.039) 0:05:48.301 **** 2021-02-10T10:58:08.005 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:08.038 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:08 +0000 (0:00:00.039) 0:05:48.340 **** 2021-02-10T10:58:08.045 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if we need to edit /etc/profile.] **************** 2021-02-10T10:58:08.045 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:08.091 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:08 +0000 (0:00:00.052) 0:05:48.393 **** 2021-02-10T10:58:08.098 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:08.138 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:08 +0000 (0:00:00.046) 0:05:48.440 **** 2021-02-10T10:58:08.145 INFO:teuthology.task.ansible.out: TASK [testnode : Update /etc/profile if needed.] ******************************* skipping: [gibba002.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:08.179 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:08 +0000 (0:00:00.040) 0:05:48.480 **** 2021-02-10T10:58:08.185 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:08.219 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:08 +0000 (0:00:00.039) 0:05:48.520 **** 2021-02-10T10:58:08.225 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-02-10T10:58:08.227 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => {"ansible_facts": {"nvme_card_present": false}, "changed": false} 2021-02-10T10:58:08.261 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:08 +0000 (0:00:00.042) 0:05:48.562 **** 2021-02-10T10:58:08.268 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => {"ansible_facts": {"nvme_card_present": false}, "changed": false} 2021-02-10T10:58:08.303 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:08 +0000 (0:00:00.042) 0:05:48.604 **** 2021-02-10T10:58:08.310 INFO:teuthology.task.ansible.out: TASK [testnode : Check for NVMe drive] ***************************************** 2021-02-10T10:58:08.310 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => (item=sda) => {"ansible_loop_var": "item", "changed": false, "item": "sda", "skip_reason": "Conditional result was False"} 2021-02-10T10:58:08.323 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item=nvme0n1) => {"ansible_facts": {"nvme_card_present": true}, "ansible_loop_var": "item", "changed": false, "item": "nvme0n1"} 2021-02-10T10:58:08.323 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => (item=dm-4) => {"ansible_loop_var": "item", "changed": false, "item": "dm-4", "skip_reason": "Conditional result was False"} 2021-02-10T10:58:08.324 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => (item=dm-2) => {"ansible_loop_var": "item", "changed": false, "item": "dm-2", "skip_reason": "Conditional result was False"} 2021-02-10T10:58:08.334 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => (item=dm-3) => {"ansible_loop_var": "item", "changed": false, "item": "dm-3", "skip_reason": "Conditional result was False"} 2021-02-10T10:58:08.345 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => (item=dm-0) => {"ansible_loop_var": "item", "changed": false, "item": "dm-0", "skip_reason": "Conditional result was False"} 2021-02-10T10:58:08.347 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => (item=dm-1) => {"ansible_loop_var": "item", "changed": false, "item": "dm-1", "skip_reason": "Conditional result was False"} 2021-02-10T10:58:08.348 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => (item=sda) => {"ansible_loop_var": "item", "changed": false, "item": "sda", "skip_reason": "Conditional result was False"} 2021-02-10T10:58:08.385 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:08 +0000 (0:00:00.082) 0:05:48.687 **** 2021-02-10T10:58:08.393 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item=nvme0n1) => {"ansible_facts": {"nvme_card_present": true}, "ansible_loop_var": "item", "changed": false, "item": "nvme0n1"} 2021-02-10T10:58:08.393 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => (item=dm-4) => {"ansible_loop_var": "item", "changed": false, "item": "dm-4", "skip_reason": "Conditional result was False"} 2021-02-10T10:58:08.394 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => (item=dm-2) => {"ansible_loop_var": "item", "changed": false, "item": "dm-2", "skip_reason": "Conditional result was False"} skipping: [gibba013.front.sepia.ceph.com] => (item=dm-3) => {"ansible_loop_var": "item", "changed": false, "item": "dm-3", "skip_reason": "Conditional result was False"} 2021-02-10T10:58:08.406 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => (item=dm-0) => {"ansible_loop_var": "item", "changed": false, "item": "dm-0", "skip_reason": "Conditional result was False"} 2021-02-10T10:58:08.406 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => (item=dm-1) => {"ansible_loop_var": "item", "changed": false, "item": "dm-1", "skip_reason": "Conditional result was False"} 2021-02-10T10:58:08.446 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:08 +0000 (0:00:00.060) 0:05:48.747 **** 2021-02-10T10:58:08.608 INFO:teuthology.task.ansible.out: TASK [testnode : Check for teuthology-lock command] **************************** 2021-02-10T10:58:08.610 INFO:teuthology.task.ansible.out:changed: [gibba002.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "which teuthology-lock", "delta": "0:00:00.003732", "end": "2021-02-10 10:58:08.587023", "rc": 0, "start": "2021-02-10 10:58:08.583291", "stderr": "", "stderr_lines": [], "stdout": "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/virtualenv/bin/teuthology-lock", "stdout_lines": ["/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/virtualenv/bin/teuthology-lock"]} 2021-02-10T10:58:08.656 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:08 +0000 (0:00:00.210) 0:05:48.957 **** 2021-02-10T10:58:08.666 INFO:teuthology.task.ansible.out:changed: [gibba013.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "which teuthology-lock", "delta": "0:00:00.003253", "end": "2021-02-10 10:58:08.619204", "rc": 0, "start": "2021-02-10 10:58:08.615951", "stderr": "", "stderr_lines": [], "stdout": "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/virtualenv/bin/teuthology-lock", "stdout_lines": ["/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/virtualenv/bin/teuthology-lock"]} 2021-02-10T10:58:08.711 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:08 +0000 (0:00:00.055) 0:05:49.013 **** 2021-02-10T10:58:08.718 INFO:teuthology.task.ansible.out: TASK [testnode : Mark system down if NVMe card missing] ************************ 2021-02-10T10:58:08.719 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:08.757 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:08 +0000 (0:00:00.045) 0:05:49.058 **** 2021-02-10T10:58:08.762 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:08.802 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:08 +0000 (0:00:00.044) 0:05:49.103 **** 2021-02-10T10:58:08.808 INFO:teuthology.task.ansible.out: TASK [testnode : Update description in paddles lock DB if NVMe card missing] *** 2021-02-10T10:58:08.809 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:08.845 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:08 +0000 (0:00:00.043) 0:05:49.147 **** 2021-02-10T10:58:08.852 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:08.886 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:08 +0000 (0:00:00.040) 0:05:49.187 **** 2021-02-10T10:58:08.892 INFO:teuthology.task.ansible.out: TASK [testnode : Fail rest of playbook due to missing NVMe card] *************** 2021-02-10T10:58:08.894 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:08.930 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:08 +0000 (0:00:00.043) 0:05:49.231 **** 2021-02-10T10:58:08.935 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:08.970 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:08 +0000 (0:00:00.039) 0:05:49.271 **** 2021-02-10T10:58:08.978 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure apt dependencies are installed] ********************* 2021-02-10T10:58:08.979 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:09.013 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:09 +0000 (0:00:00.043) 0:05:49.314 **** 2021-02-10T10:58:09.021 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:09.060 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:09 +0000 (0:00:00.047) 0:05:49.362 **** 2021-02-10T10:58:09.847 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure rpm dependencies are installed] ********************* 2021-02-10T10:58:09.848 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => {"changed": false, "msg": "", "rc": 0, "results": ["gdisk-0.8.10-3.el7.x86_64 providing gdisk is already installed", "7:device-mapper-1.02.170-6.el7.x86_64 providing device-mapper is already installed"]} 2021-02-10T10:58:09.881 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:09 +0000 (0:00:00.821) 0:05:50.183 **** 2021-02-10T10:58:09.888 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => {"changed": false, "msg": "", "rc": 0, "results": ["gdisk-0.8.10-3.el7.x86_64 providing gdisk is already installed", "7:device-mapper-1.02.170-6.el7.x86_64 providing device-mapper is already installed"]} 2021-02-10T10:58:09.924 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:09 +0000 (0:00:00.041) 0:05:50.225 **** 2021-02-10T10:58:09.930 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2021-02-10T10:58:09.930 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item={'block_used': 13742194, 'uuid': 'aeb0602f-c6fc-4ca1-8920-0340080ffa5a', 'size_total': 984369086464, 'block_total': 240324484, 'mount': '/', 'block_available': 226582290, 'size_available': 928081059840, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,seclabel,relatime,data=ordered', 'device': '/dev/sda1', 'inode_used': 90165, 'block_size': 4096, 'inode_available': 60964811}) => {"ansible_facts": {"root_disk": "/dev/sda"}, "ansible_loop_var": "item", "changed": false, "item": {"block_available": 226582290, "block_size": 4096, "block_total": 240324484, "block_used": 13742194, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60964811, "inode_total": 61054976, "inode_used": 90165, "mount": "/", "options": "rw,seclabel,relatime,data=ordered", "size_available": 928081059840, "size_total": 984369086464, "uuid": "aeb0602f-c6fc-4ca1-8920-0340080ffa5a"}} 2021-02-10T10:58:09.967 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:09 +0000 (0:00:00.043) 0:05:50.269 **** 2021-02-10T10:58:09.975 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item={'block_used': 13742185, 'uuid': 'aeb0602f-c6fc-4ca1-8920-0340080ffa5a', 'size_total': 984369086464, 'block_total': 240324484, 'mount': '/', 'block_available': 226582299, 'size_available': 928081096704, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,seclabel,relatime,data=ordered', 'device': '/dev/sda1', 'inode_used': 90165, 'block_size': 4096, 'inode_available': 60964811}) => {"ansible_facts": {"root_disk": "/dev/sda"}, "ansible_loop_var": "item", "changed": false, "item": {"block_available": 226582299, "block_size": 4096, "block_total": 240324484, "block_used": 13742185, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60964811, "inode_total": 61054976, "inode_used": 90165, "mount": "/", "options": "rw,seclabel,relatime,data=ordered", "size_available": 928081096704, "size_total": 984369086464, "uuid": "aeb0602f-c6fc-4ca1-8920-0340080ffa5a"}} 2021-02-10T10:58:10.012 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:10 +0000 (0:00:00.044) 0:05:50.313 **** 2021-02-10T10:58:10.425 INFO:teuthology.task.ansible.out: TASK [testnode : Compile list of non-root partitions] ************************** 2021-02-10T10:58:10.426 INFO:teuthology.task.ansible.out:changed: [gibba002.front.sepia.ceph.com] => {"changed": true, "cmd": "lsblk --list --noheadings | grep part | grep -v sda | awk '{ print $1 }'", "delta": "0:00:00.004462", "end": "2021-02-10 10:58:10.407343", "rc": 0, "start": "2021-02-10 10:58:10.402881", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-10T10:58:10.459 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:10 +0000 (0:00:00.447) 0:05:50.761 **** 2021-02-10T10:58:10.466 INFO:teuthology.task.ansible.out:changed: [gibba013.front.sepia.ceph.com] => {"changed": true, "cmd": "lsblk --list --noheadings | grep part | grep -v sda | awk '{ print $1 }'", "delta": "0:00:00.004449", "end": "2021-02-10 10:58:10.445277", "rc": 0, "start": "2021-02-10 10:58:10.440828", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-10T10:58:10.500 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:10 +0000 (0:00:00.041) 0:05:50.802 **** 2021-02-10T10:58:10.506 INFO:teuthology.task.ansible.out: TASK [testnode : Unmount any non-root mountpoints] ***************************** 2021-02-10T10:58:10.507 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => (item={'block_used': 13742194, 'uuid': 'aeb0602f-c6fc-4ca1-8920-0340080ffa5a', 'size_total': 984369086464, 'block_total': 240324484, 'mount': '/', 'block_available': 226582290, 'size_available': 928081059840, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,seclabel,relatime,data=ordered', 'device': '/dev/sda1', 'inode_used': 90165, 'block_size': 4096, 'inode_available': 60964811}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226582290, "block_size": 4096, "block_total": 240324484, "block_used": 13742194, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60964811, "inode_total": 61054976, "inode_used": 90165, "mount": "/", "options": "rw,seclabel,relatime,data=ordered", "size_available": 928081059840, "size_total": 984369086464, "uuid": "aeb0602f-c6fc-4ca1-8920-0340080ffa5a"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:10.553 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:10 +0000 (0:00:00.052) 0:05:50.855 **** 2021-02-10T10:58:10.561 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => (item={'block_used': 13742185, 'uuid': 'aeb0602f-c6fc-4ca1-8920-0340080ffa5a', 'size_total': 984369086464, 'block_total': 240324484, 'mount': '/', 'block_available': 226582299, 'size_available': 928081096704, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,seclabel,relatime,data=ordered', 'device': '/dev/sda1', 'inode_used': 90165, 'block_size': 4096, 'inode_available': 60964811}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226582299, "block_size": 4096, "block_total": 240324484, "block_used": 13742185, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60964811, "inode_total": 61054976, "inode_used": 90165, "mount": "/", "options": "rw,seclabel,relatime,data=ordered", "size_available": 928081096704, "size_total": 984369086464, "uuid": "aeb0602f-c6fc-4ca1-8920-0340080ffa5a"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:10.596 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:10 +0000 (0:00:00.043) 0:05:50.898 **** 2021-02-10T10:58:10.603 INFO:teuthology.task.ansible.out: TASK [testnode : Wipe filesystems on non-root partitions] ********************** 2021-02-10T10:58:10.638 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:10 +0000 (0:00:00.041) 0:05:50.939 **** 2021-02-10T10:58:10.682 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:10 +0000 (0:00:00.043) 0:05:50.983 **** 2021-02-10T10:58:10.690 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2021-02-10T10:58:10.691 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'scheduler_mode': 'deadline', 'rotational': '1', 'vendor': 'ATA', 'sectors': '1953525168', 'links': {'masters': [], 'labels': [], 'ids': ['ata-TOSHIBA_MG04ACA100N_Y9IXK25PF6XF', 'wwn-0x50000399ac701b81'], 'uuids': []}, 'partitions': {'sda1': {'sectorsize': 512, 'uuid': 'aeb0602f-c6fc-4ca1-8920-0340080ffa5a', 'links': {'masters': [], 'labels': [], 'ids': ['ata-TOSHIBA_MG04ACA100N_Y9IXK25PF6XF-part1', 'wwn-0x50000399ac701b81-part1'], 'uuids': ['aeb0602f-c6fc-4ca1-8920-0340080ffa5a']}, 'sectors': '1953522688', 'start': '2048', 'holders': [], 'size': '931.51 GB'}}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'SATA controller: Intel Corporation Cannon Lake PCH SATA AHCI Controller (rev 10)', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'holders': [], 'wwn': '0x50000399ac701b81', 'model': 'TOSHIBA MG04ACA1', 'serial': 'Y9IXK25PF6XF', 'size': '931.51 GB'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation Cannon Lake PCH SATA AHCI Controller (rev 10)", "links": {"ids": ["ata-TOSHIBA_MG04ACA100N_Y9IXK25PF6XF", "wwn-0x50000399ac701b81"], "labels": [], "masters": [], "uuids": []}, "model": "TOSHIBA MG04ACA1", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-TOSHIBA_MG04ACA100N_Y9IXK25PF6XF-part1", "wwn-0x50000399ac701b81-part1"], "labels": [], "masters": [], "uuids": ["aeb0602f-c6fc-4ca1-8920-0340080ffa5a"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "aeb0602f-c6fc-4ca1-8920-0340080ffa5a"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Y9IXK25PF6XF", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x50000399ac701b81"}}, "skip_reason": "Conditiona 2021-02-10T10:58:10.692 INFO:teuthology.task.ansible.out:l result was False"} 2021-02-10T10:58:10.725 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'scheduler_mode': 'deadline', 'rotational': '1', 'vendor': 'ATA', 'sectors': '1953525168', 'links': {'masters': [], 'labels': [], 'ids': ['ata-TOSHIBA_MG04ACA100N_Y9I6K2L3F6XF', 'wwn-0x50000399ab9813f5'], 'uuids': []}, 'partitions': {'sda1': {'sectorsize': 512, 'uuid': 'aeb0602f-c6fc-4ca1-8920-0340080ffa5a', 'links': {'masters': [], 'labels': [], 'ids': ['ata-TOSHIBA_MG04ACA100N_Y9I6K2L3F6XF-part1', 'wwn-0x50000399ab9813f5-part1'], 'uuids': ['aeb0602f-c6fc-4ca1-8920-0340080ffa5a']}, 'sectors': '1953522688', 'start': '2048', 'holders': [], 'size': '931.51 GB'}}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'SATA controller: Intel Corporation Cannon Lake PCH SATA AHCI Controller (rev 10)', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'holders': [], 'wwn': '0x50000399ab9813f5', 'model': 'TOSHIBA MG04ACA1', 'serial': 'Y9I6K2L3F6XF', 'size': '931.51 GB'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation Cannon Lake PCH SATA AHCI Controller (rev 10)", "links": {"ids": ["ata-TOSHIBA_MG04ACA100N_Y9I6K2L3F6XF", "wwn-0x50000399ab9813f5"], "labels": [], "masters": [], "uuids": []}, "model": "TOSHIBA MG04ACA1", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-TOSHIBA_MG04ACA100N_Y9I6K2L3F6XF-part1", "wwn-0x50000399ab9813f5-part1"], "labels": [], "masters": [], "uuids": ["aeb0602f-c6fc-4ca1-8920-0340080ffa5a"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "aeb0602f-c6fc-4ca1-8920-0340080ffa5a"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Y9I6K2L3F6XF", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x50000399ab9813f5"}}, "skip_reason": "Conditiona 2021-02-10T10:58:10.726 INFO:teuthology.task.ansible.out:l result was False"} 2021-02-10T10:58:12.286 INFO:teuthology.task.ansible.out:changed: [gibba002.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'scheduler_mode': 'none', 'rotational': '0', 'vendor': None, 'sectors': '732585168', 'links': {'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4'], 'labels': [], 'ids': ['lvm-pv-uuid-b7bhRS-DW1k-CHIq-p1mY-yvuH-bfjY-RApqeu', 'nvme-INTEL_SSDPEL1K375GA_PHKM9200008B375A', 'nvme-nvme.8086-50484b4d393230303030384233373541-494e54454c2053534450454c314b3337354741-00000001'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'Non-Volatile memory controller: Intel Corporation NVMe Datacenter SSD [Optane]', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'holders': ['vg_nvme-lv_1', 'vg_nvme-lv_2', 'vg_nvme-lv_3', 'vg_nvme-lv_4', 'vg_nvme-lv_5'], 'partitions': {}, 'model': 'INTEL SSDPEL1K375GA', 'serial': 'PHKM9200008B375A', 'size': '349.32 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/nvme0n1 || sgdisk --zap-all /dev/nvme0n1", "delta": "0:00:01.157925", "end": "2021-02-10 10:58:12.258181", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_1", "vg_nvme-lv_2", "vg_nvme-lv_3", "vg_nvme-lv_4", "vg_nvme-lv_5"], "host": "Non-Volatile memory controller: Intel Corporation NVMe Datacenter SSD [Optane]", "links": {"ids": ["lvm-pv-uuid-b7bhRS-DW1k-CHIq-p1mY-yvuH-bfjY-RApqeu", "nvme-INTEL_SSDPEL1K375GA_PHKM9200008B375A", "nvme-nvme.8086-50484b4d393230303030384233373541-494e54454c2053534450454c314b3337354741-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEL1K375GA", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "732585168", "sectorsize": "512", "serial": "PHKM9200008B375A", "size": "349.32 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-10 10:58:11.100256", "stderr": "", "stderr_lines": [], "stdout": "Creating new 2021-02-10T10:58:12.288 INFO:teuthology.task.ansible.out:GPT entries.\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-02-10T10:58:12.548 INFO:teuthology.task.ansible.out:changed: [gibba013.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'scheduler_mode': 'none', 'rotational': '0', 'vendor': None, 'sectors': '732585168', 'links': {'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4'], 'labels': [], 'ids': ['lvm-pv-uuid-rgba5W-42U8-zOPj-kx01-4mXg-mQ2o-e6DsyD', 'nvme-INTEL_SSDPEL1K375GA_PHKM9200001J375A', 'nvme-nvme.8086-50484b4d393230303030314a33373541-494e54454c2053534450454c314b3337354741-00000001'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'Non-Volatile memory controller: Intel Corporation NVMe Datacenter SSD [Optane]', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'holders': ['vg_nvme-lv_1', 'vg_nvme-lv_2', 'vg_nvme-lv_3', 'vg_nvme-lv_4', 'vg_nvme-lv_5'], 'partitions': {}, 'model': 'INTEL SSDPEL1K375GA', 'serial': 'PHKM9200001J375A', 'size': '349.32 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/nvme0n1 || sgdisk --zap-all /dev/nvme0n1", "delta": "0:00:01.385738", "end": "2021-02-10 10:58:12.523500", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_1", "vg_nvme-lv_2", "vg_nvme-lv_3", "vg_nvme-lv_4", "vg_nvme-lv_5"], "host": "Non-Volatile memory controller: Intel Corporation NVMe Datacenter SSD [Optane]", "links": {"ids": ["lvm-pv-uuid-rgba5W-42U8-zOPj-kx01-4mXg-mQ2o-e6DsyD", "nvme-INTEL_SSDPEL1K375GA_PHKM9200001J375A", "nvme-nvme.8086-50484b4d393230303030314a33373541-494e54454c2053534450454c314b3337354741-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEL1K375GA", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "732585168", "sectorsize": "512", "serial": "PHKM9200001J375A", "size": "349.32 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-10 10:58:11.137762", "stderr": "", "stderr_lines": [], "stdout": "Creating new 2021-02-10T10:58:12.550 INFO:teuthology.task.ansible.out:GPT entries.\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-02-10T10:58:13.774 INFO:teuthology.task.ansible.out:changed: [gibba002.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '27262976', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-2WBfdgWiAT4ia77vCTuZlAyUc0RReyY2p1LLl3TMqI0KB787Xi5C3Rhph2AYP6Zf'], 'uuids': ['bc85eabf-bb8e-4f55-8e20-67100b24c5d1']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'holders': [], 'partitions': {}, 'model': None, 'serial': 'PHKM9200008B375A', 'size': '13.00 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-4 || sgdisk --zap-all /dev/dm-4", "delta": "0:00:01.051948", "end": "2021-02-10 10:58:13.753793", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-2WBfdgWiAT4ia77vCTuZlAyUc0RReyY2p1LLl3TMqI0KB787Xi5C3Rhph2AYP6Zf"], "labels": [], "masters": [], "uuids": ["bc85eabf-bb8e-4f55-8e20-67100b24c5d1"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "27262976", "sectorsize": "512", "serial": "PHKM9200008B375A", "size": "13.00 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-10 10:58:12.701845", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot.\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-02-10T10:58:14.036 INFO:teuthology.task.ansible.out:changed: [gibba013.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '27262976', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-NlMncYOnyQBeUf2HgazZrUXsftKpT1HQbrUzxM6GvUh0mlnkR2nJw5fGmaxr7E0y'], 'uuids': ['ce7b600d-9ec8-46b0-8d1e-09dbffa6baa4']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'holders': [], 'partitions': {}, 'model': None, 'serial': 'PHKM9200001J375A', 'size': '13.00 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-4 || sgdisk --zap-all /dev/dm-4", "delta": "0:00:01.055787", "end": "2021-02-10 10:58:14.015385", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-NlMncYOnyQBeUf2HgazZrUXsftKpT1HQbrUzxM6GvUh0mlnkR2nJw5fGmaxr7E0y"], "labels": [], "masters": [], "uuids": ["ce7b600d-9ec8-46b0-8d1e-09dbffa6baa4"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "27262976", "sectorsize": "512", "serial": "PHKM9200001J375A", "size": "13.00 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-10 10:58:12.959598", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot.\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-02-10T10:58:15.271 INFO:teuthology.task.ansible.out:changed: [gibba002.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '176160768', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-2WBfdgWiAT4ia77vCTuZlAyUc0RReyY2yH30b9FW6CBHypeI3sphHA2Z6WC9dG0E'], 'uuids': ['3e33465b-c876-4092-81e8-56264c761761']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'holders': [], 'partitions': {}, 'model': None, 'serial': 'PHKM9200008B375A', 'size': '84.00 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-2 || sgdisk --zap-all /dev/dm-2", "delta": "0:00:01.058581", "end": "2021-02-10 10:58:15.245153", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-2WBfdgWiAT4ia77vCTuZlAyUc0RReyY2yH30b9FW6CBHypeI3sphHA2Z6WC9dG0E"], "labels": [], "masters": [], "uuids": ["3e33465b-c876-4092-81e8-56264c761761"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "176160768", "sectorsize": "512", "serial": "PHKM9200008B375A", "size": "84.00 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-10 10:58:14.186572", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot.\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-02-10T10:58:15.526 INFO:teuthology.task.ansible.out:changed: [gibba013.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '176160768', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-NlMncYOnyQBeUf2HgazZrUXsftKpT1HQBFHqUZ75wgu2aEZ4V7d3bubJmUAdYpu2'], 'uuids': ['d4d10677-b2c2-496d-9b3c-80fa13262607']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'holders': [], 'partitions': {}, 'model': None, 'serial': 'PHKM9200001J375A', 'size': '84.00 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-2 || sgdisk --zap-all /dev/dm-2", "delta": "0:00:01.045598", "end": "2021-02-10 10:58:15.505636", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-NlMncYOnyQBeUf2HgazZrUXsftKpT1HQBFHqUZ75wgu2aEZ4V7d3bubJmUAdYpu2"], "labels": [], "masters": [], "uuids": ["d4d10677-b2c2-496d-9b3c-80fa13262607"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "176160768", "sectorsize": "512", "serial": "PHKM9200001J375A", "size": "84.00 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-10 10:58:14.460038", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot.\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-02-10T10:58:16.759 INFO:teuthology.task.ansible.out:changed: [gibba002.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '176160768', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-2WBfdgWiAT4ia77vCTuZlAyUc0RReyY2GD1wAebfVT9qGlK1qZRC8RjNegIfYWlc'], 'uuids': ['99ebacdc-b777-499b-a52f-ca8708cf22ef']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'holders': [], 'partitions': {}, 'model': None, 'serial': 'PHKM9200008B375A', 'size': '84.00 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-3 || sgdisk --zap-all /dev/dm-3", "delta": "0:00:01.059005", "end": "2021-02-10 10:58:16.738066", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-2WBfdgWiAT4ia77vCTuZlAyUc0RReyY2GD1wAebfVT9qGlK1qZRC8RjNegIfYWlc"], "labels": [], "masters": [], "uuids": ["99ebacdc-b777-499b-a52f-ca8708cf22ef"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "176160768", "sectorsize": "512", "serial": "PHKM9200008B375A", "size": "84.00 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-10 10:58:15.679061", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot.\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-02-10T10:58:17.014 INFO:teuthology.task.ansible.out:changed: [gibba013.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '176160768', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-NlMncYOnyQBeUf2HgazZrUXsftKpT1HQZ7ONiRgkbCkAfySu701xFDekTi8yKBWL'], 'uuids': ['0e337204-461d-4966-8a63-8a5a5f070b69']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'holders': [], 'partitions': {}, 'model': None, 'serial': 'PHKM9200001J375A', 'size': '84.00 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-3 || sgdisk --zap-all /dev/dm-3", "delta": "0:00:01.049557", "end": "2021-02-10 10:58:16.990623", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-NlMncYOnyQBeUf2HgazZrUXsftKpT1HQZ7ONiRgkbCkAfySu701xFDekTi8yKBWL"], "labels": [], "masters": [], "uuids": ["0e337204-461d-4966-8a63-8a5a5f070b69"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "176160768", "sectorsize": "512", "serial": "PHKM9200001J375A", "size": "84.00 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-10 10:58:15.941066", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot.\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-02-10T10:58:18.268 INFO:teuthology.task.ansible.out:changed: [gibba002.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '176160768', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-2WBfdgWiAT4ia77vCTuZlAyUc0RReyY2f1MVjyWcRwdmfhiojuNZzH4Fjfh0Rg18'], 'uuids': ['0dac7cc3-6ceb-4955-a8a2-9859b7b9d4d4']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'holders': [], 'partitions': {}, 'model': None, 'serial': 'PHKM9200008B375A', 'size': '84.00 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-0 || sgdisk --zap-all /dev/dm-0", "delta": "0:00:01.065309", "end": "2021-02-10 10:58:18.238215", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-2WBfdgWiAT4ia77vCTuZlAyUc0RReyY2f1MVjyWcRwdmfhiojuNZzH4Fjfh0Rg18"], "labels": [], "masters": [], "uuids": ["0dac7cc3-6ceb-4955-a8a2-9859b7b9d4d4"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "176160768", "sectorsize": "512", "serial": "PHKM9200008B375A", "size": "84.00 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-10 10:58:17.172906", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot.\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-02-10T10:58:18.508 INFO:teuthology.task.ansible.out:changed: [gibba013.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '176160768', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-NlMncYOnyQBeUf2HgazZrUXsftKpT1HQnt2HTQXVCgXehT4pQ8KHPZoWZz93tpIO'], 'uuids': ['0de34736-6834-4254-b402-9e60b03bcd41']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'holders': [], 'partitions': {}, 'model': None, 'serial': 'PHKM9200001J375A', 'size': '84.00 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-0 || sgdisk --zap-all /dev/dm-0", "delta": "0:00:01.053211", "end": "2021-02-10 10:58:18.490958", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-NlMncYOnyQBeUf2HgazZrUXsftKpT1HQnt2HTQXVCgXehT4pQ8KHPZoWZz93tpIO"], "labels": [], "masters": [], "uuids": ["0de34736-6834-4254-b402-9e60b03bcd41"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "176160768", "sectorsize": "512", "serial": "PHKM9200001J375A", "size": "84.00 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-10 10:58:17.437747", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot.\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-02-10T10:58:19.757 INFO:teuthology.task.ansible.out:changed: [gibba002.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '176160768', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-2WBfdgWiAT4ia77vCTuZlAyUc0RReyY2Clr8lcequIZ7066HyrMtNoBIamtJcMy0'], 'uuids': ['51a2e578-c647-45ed-9660-72ee057ae6e4']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'holders': [], 'partitions': {}, 'model': None, 'serial': 'PHKM9200008B375A', 'size': '84.00 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1", "delta": "0:00:01.061008", "end": "2021-02-10 10:58:19.738466", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-2WBfdgWiAT4ia77vCTuZlAyUc0RReyY2Clr8lcequIZ7066HyrMtNoBIamtJcMy0"], "labels": [], "masters": [], "uuids": ["51a2e578-c647-45ed-9660-72ee057ae6e4"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "176160768", "sectorsize": "512", "serial": "PHKM9200008B375A", "size": "84.00 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-10 10:58:18.677458", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot.\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-02-10T10:58:19.803 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:19 +0000 (0:00:09.121) 0:06:00.104 **** 2021-02-10T10:58:19.852 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2021-02-10T10:58:19.853 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'scheduler_mode': 'deadline', 'rotational': '1', 'vendor': 'ATA', 'sectors': '1953525168', 'links': {'masters': [], 'labels': [], 'ids': ['ata-TOSHIBA_MG04ACA100N_Y9IXK25PF6XF', 'wwn-0x50000399ac701b81'], 'uuids': []}, 'partitions': {'sda1': {'sectorsize': 512, 'uuid': 'aeb0602f-c6fc-4ca1-8920-0340080ffa5a', 'links': {'masters': [], 'labels': [], 'ids': ['ata-TOSHIBA_MG04ACA100N_Y9IXK25PF6XF-part1', 'wwn-0x50000399ac701b81-part1'], 'uuids': ['aeb0602f-c6fc-4ca1-8920-0340080ffa5a']}, 'sectors': '1953522688', 'start': '2048', 'holders': [], 'size': '931.51 GB'}}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'SATA controller: Intel Corporation Cannon Lake PCH SATA AHCI Controller (rev 10)', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'holders': [], 'wwn': '0x50000399ac701b81', 'model': 'TOSHIBA MG04ACA1', 'serial': 'Y9IXK25PF6XF', 'size': '931.51 GB'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation Cannon Lake PCH SATA AHCI Controller (rev 10)", "links": {"ids": ["ata-TOSHIBA_MG04ACA100N_Y9IXK25PF6XF", "wwn-0x50000399ac701b81"], "labels": [], "masters": [], "uuids": []}, "model": "TOSHIBA MG04ACA1", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-TOSHIBA_MG04ACA100N_Y9IXK25PF6XF-part1", "wwn-0x50000399ac701b81-part1"], "labels": [], "masters": [], "uuids": ["aeb0602f-c6fc-4ca1-8920-0340080ffa5a"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "aeb0602f-c6fc-4ca1-8920-0340080ffa5a"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Y9IXK25PF6XF", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x50000399ac701b81"}}, "skip_reason": "Conditiona 2021-02-10T10:58:19.854 INFO:teuthology.task.ansible.out:l result was False"} 2021-02-10T10:58:20.000 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2021-02-10T10:58:20.001 INFO:teuthology.task.ansible.out:changed: [gibba013.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '176160768', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-NlMncYOnyQBeUf2HgazZrUXsftKpT1HQNz0CDkdIeZtBPJRzLBW7LuQsxzsouWnN'], 'uuids': ['e68e8bad-acb3-43a9-be98-ee4712cf44df']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'holders': [], 'partitions': {}, 'model': None, 'serial': 'PHKM9200001J375A', 'size': '84.00 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1", "delta": "0:00:01.052836", "end": "2021-02-10 10:58:19.974257", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-NlMncYOnyQBeUf2HgazZrUXsftKpT1HQNz0CDkdIeZtBPJRzLBW7LuQsxzsouWnN"], "labels": [], "masters": [], "uuids": ["e68e8bad-acb3-43a9-be98-ee4712cf44df"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "176160768", "sectorsize": "512", "serial": "PHKM9200001J375A", "size": "84.00 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-10 10:58:18.921421", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot.\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-02-10T10:58:20.045 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:20 +0000 (0:00:00.242) 0:06:00.347 **** 2021-02-10T10:58:20.084 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2021-02-10T10:58:20.085 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'scheduler_mode': 'deadline', 'rotational': '1', 'vendor': 'ATA', 'sectors': '1953525168', 'links': {'masters': [], 'labels': [], 'ids': ['ata-TOSHIBA_MG04ACA100N_Y9I6K2L3F6XF', 'wwn-0x50000399ab9813f5'], 'uuids': []}, 'partitions': {'sda1': {'sectorsize': 512, 'uuid': 'aeb0602f-c6fc-4ca1-8920-0340080ffa5a', 'links': {'masters': [], 'labels': [], 'ids': ['ata-TOSHIBA_MG04ACA100N_Y9I6K2L3F6XF-part1', 'wwn-0x50000399ab9813f5-part1'], 'uuids': ['aeb0602f-c6fc-4ca1-8920-0340080ffa5a']}, 'sectors': '1953522688', 'start': '2048', 'holders': [], 'size': '931.51 GB'}}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'SATA controller: Intel Corporation Cannon Lake PCH SATA AHCI Controller (rev 10)', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'holders': [], 'wwn': '0x50000399ab9813f5', 'model': 'TOSHIBA MG04ACA1', 'serial': 'Y9I6K2L3F6XF', 'size': '931.51 GB'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation Cannon Lake PCH SATA AHCI Controller (rev 10)", "links": {"ids": ["ata-TOSHIBA_MG04ACA100N_Y9I6K2L3F6XF", "wwn-0x50000399ab9813f5"], "labels": [], "masters": [], "uuids": []}, "model": "TOSHIBA MG04ACA1", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-TOSHIBA_MG04ACA100N_Y9I6K2L3F6XF-part1", "wwn-0x50000399ab9813f5-part1"], "labels": [], "masters": [], "uuids": ["aeb0602f-c6fc-4ca1-8920-0340080ffa5a"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "aeb0602f-c6fc-4ca1-8920-0340080ffa5a"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Y9I6K2L3F6XF", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x50000399ab9813f5"}}, "skip_reason": "Conditiona 2021-02-10T10:58:20.086 INFO:teuthology.task.ansible.out:l result was False"} 2021-02-10T10:58:20.316 INFO:teuthology.task.ansible.out:changed: [gibba002.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'scheduler_mode': 'none', 'rotational': '0', 'vendor': None, 'sectors': '732585168', 'links': {'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4'], 'labels': [], 'ids': ['lvm-pv-uuid-b7bhRS-DW1k-CHIq-p1mY-yvuH-bfjY-RApqeu', 'nvme-INTEL_SSDPEL1K375GA_PHKM9200008B375A', 'nvme-nvme.8086-50484b4d393230303030384233373541-494e54454c2053534450454c314b3337354741-00000001'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'Non-Volatile memory controller: Intel Corporation NVMe Datacenter SSD [Optane]', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'holders': ['vg_nvme-lv_1', 'vg_nvme-lv_2', 'vg_nvme-lv_3', 'vg_nvme-lv_4', 'vg_nvme-lv_5'], 'partitions': {}, 'model': 'INTEL SSDPEL1K375GA', 'serial': 'PHKM9200008B375A', 'size': '349.32 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=110", "delta": "0:00:00.026680", "end": "2021-02-10 10:58:20.289803", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_1", "vg_nvme-lv_2", "vg_nvme-lv_3", "vg_nvme-lv_4", "vg_nvme-lv_5"], "host": "Non-Volatile memory controller: Intel Corporation NVMe Datacenter SSD [Optane]", "links": {"ids": ["lvm-pv-uuid-b7bhRS-DW1k-CHIq-p1mY-yvuH-bfjY-RApqeu", "nvme-INTEL_SSDPEL1K375GA_PHKM9200008B375A", "nvme-nvme.8086-50484b4d393230303030384233373541-494e54454c2053534450454c314b3337354741-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEL1K375GA", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "732585168", "sectorsize": "512", "serial": "PHKM9200008B375A", "size": "349.32 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-10 10:58:20.263123", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB) 2021-02-10T10:58:20.318 INFO:teuthology.task.ansible.out:copied, 0.0244516 s, 4.7 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB) copied, 0.0244516 s, 4.7 GB/s"], "stdout": "", "stdout_lines": []} 2021-02-10T10:58:20.548 INFO:teuthology.task.ansible.out:changed: [gibba013.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'scheduler_mode': 'none', 'rotational': '0', 'vendor': None, 'sectors': '732585168', 'links': {'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4'], 'labels': [], 'ids': ['lvm-pv-uuid-rgba5W-42U8-zOPj-kx01-4mXg-mQ2o-e6DsyD', 'nvme-INTEL_SSDPEL1K375GA_PHKM9200001J375A', 'nvme-nvme.8086-50484b4d393230303030314a33373541-494e54454c2053534450454c314b3337354741-00000001'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'Non-Volatile memory controller: Intel Corporation NVMe Datacenter SSD [Optane]', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'holders': ['vg_nvme-lv_1', 'vg_nvme-lv_2', 'vg_nvme-lv_3', 'vg_nvme-lv_4', 'vg_nvme-lv_5'], 'partitions': {}, 'model': 'INTEL SSDPEL1K375GA', 'serial': 'PHKM9200001J375A', 'size': '349.32 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=110", "delta": "0:00:00.026005", "end": "2021-02-10 10:58:20.527340", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_1", "vg_nvme-lv_2", "vg_nvme-lv_3", "vg_nvme-lv_4", "vg_nvme-lv_5"], "host": "Non-Volatile memory controller: Intel Corporation NVMe Datacenter SSD [Optane]", "links": {"ids": ["lvm-pv-uuid-rgba5W-42U8-zOPj-kx01-4mXg-mQ2o-e6DsyD", "nvme-INTEL_SSDPEL1K375GA_PHKM9200001J375A", "nvme-nvme.8086-50484b4d393230303030314a33373541-494e54454c2053534450454c314b3337354741-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEL1K375GA", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "732585168", "sectorsize": "512", "serial": "PHKM9200001J375A", "size": "349.32 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-10 10:58:20.501335", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB) 2021-02-10T10:58:20.549 INFO:teuthology.task.ansible.out:copied, 0.0237758 s, 4.9 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB) copied, 0.0237758 s, 4.9 GB/s"], "stdout": "", "stdout_lines": []} 2021-02-10T10:58:20.829 INFO:teuthology.task.ansible.out:changed: [gibba002.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '27262976', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-2WBfdgWiAT4ia77vCTuZlAyUc0RReyY2p1LLl3TMqI0KB787Xi5C3Rhph2AYP6Zf'], 'uuids': ['bc85eabf-bb8e-4f55-8e20-67100b24c5d1']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'holders': [], 'partitions': {}, 'model': None, 'serial': 'PHKM9200008B375A', 'size': '13.00 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-4 bs=1M count=110", "delta": "0:00:00.080501", "end": "2021-02-10 10:58:20.806836", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-2WBfdgWiAT4ia77vCTuZlAyUc0RReyY2p1LLl3TMqI0KB787Xi5C3Rhph2AYP6Zf"], "labels": [], "masters": [], "uuids": ["bc85eabf-bb8e-4f55-8e20-67100b24c5d1"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "27262976", "sectorsize": "512", "serial": "PHKM9200008B375A", "size": "13.00 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-10 10:58:20.726335", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB) copied, 0.0784464 s, 1.5 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB) copied, 0.0784464 s, 1.5 GB/s"], "stdout": "", "stdout_lines": []} 2021-02-10T10:58:21.060 INFO:teuthology.task.ansible.out:changed: [gibba013.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '27262976', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-NlMncYOnyQBeUf2HgazZrUXsftKpT1HQbrUzxM6GvUh0mlnkR2nJw5fGmaxr7E0y'], 'uuids': ['ce7b600d-9ec8-46b0-8d1e-09dbffa6baa4']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'holders': [], 'partitions': {}, 'model': None, 'serial': 'PHKM9200001J375A', 'size': '13.00 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-4 bs=1M count=110", "delta": "0:00:00.079944", "end": "2021-02-10 10:58:21.041747", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-NlMncYOnyQBeUf2HgazZrUXsftKpT1HQbrUzxM6GvUh0mlnkR2nJw5fGmaxr7E0y"], "labels": [], "masters": [], "uuids": ["ce7b600d-9ec8-46b0-8d1e-09dbffa6baa4"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "27262976", "sectorsize": "512", "serial": "PHKM9200001J375A", "size": "13.00 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-10 10:58:20.961803", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB) copied, 0.0778637 s, 1.5 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB) copied, 0.0778637 s, 1.5 GB/s"], "stdout": "", "stdout_lines": []} 2021-02-10T10:58:21.345 INFO:teuthology.task.ansible.out:changed: [gibba002.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '176160768', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-2WBfdgWiAT4ia77vCTuZlAyUc0RReyY2yH30b9FW6CBHypeI3sphHA2Z6WC9dG0E'], 'uuids': ['3e33465b-c876-4092-81e8-56264c761761']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'holders': [], 'partitions': {}, 'model': None, 'serial': 'PHKM9200008B375A', 'size': '84.00 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-2 bs=1M count=110", "delta": "0:00:00.081015", "end": "2021-02-10 10:58:21.321231", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-2WBfdgWiAT4ia77vCTuZlAyUc0RReyY2yH30b9FW6CBHypeI3sphHA2Z6WC9dG0E"], "labels": [], "masters": [], "uuids": ["3e33465b-c876-4092-81e8-56264c761761"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "176160768", "sectorsize": "512", "serial": "PHKM9200008B375A", "size": "84.00 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-10 10:58:21.240216", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB) copied, 0.0789963 s, 1.5 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB) copied, 0.0789963 s, 1.5 GB/s"], "stdout": "", "stdout_lines": []} 2021-02-10T10:58:21.575 INFO:teuthology.task.ansible.out:changed: [gibba013.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '176160768', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-NlMncYOnyQBeUf2HgazZrUXsftKpT1HQBFHqUZ75wgu2aEZ4V7d3bubJmUAdYpu2'], 'uuids': ['d4d10677-b2c2-496d-9b3c-80fa13262607']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'holders': [], 'partitions': {}, 'model': None, 'serial': 'PHKM9200001J375A', 'size': '84.00 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-2 bs=1M count=110", "delta": "0:00:00.081364", "end": "2021-02-10 10:58:21.553720", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-NlMncYOnyQBeUf2HgazZrUXsftKpT1HQBFHqUZ75wgu2aEZ4V7d3bubJmUAdYpu2"], "labels": [], "masters": [], "uuids": ["d4d10677-b2c2-496d-9b3c-80fa13262607"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "176160768", "sectorsize": "512", "serial": "PHKM9200001J375A", "size": "84.00 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-10 10:58:21.472356", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB) copied, 0.0791959 s, 1.5 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB) copied, 0.0791959 s, 1.5 GB/s"], "stdout": "", "stdout_lines": []} 2021-02-10T10:58:21.856 INFO:teuthology.task.ansible.out:changed: [gibba002.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '176160768', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-2WBfdgWiAT4ia77vCTuZlAyUc0RReyY2GD1wAebfVT9qGlK1qZRC8RjNegIfYWlc'], 'uuids': ['99ebacdc-b777-499b-a52f-ca8708cf22ef']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'holders': [], 'partitions': {}, 'model': None, 'serial': 'PHKM9200008B375A', 'size': '84.00 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-3 bs=1M count=110", "delta": "0:00:00.080138", "end": "2021-02-10 10:58:21.832002", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-2WBfdgWiAT4ia77vCTuZlAyUc0RReyY2GD1wAebfVT9qGlK1qZRC8RjNegIfYWlc"], "labels": [], "masters": [], "uuids": ["99ebacdc-b777-499b-a52f-ca8708cf22ef"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "176160768", "sectorsize": "512", "serial": "PHKM9200008B375A", "size": "84.00 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-10 10:58:21.751864", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB) copied, 0.0780103 s, 1.5 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB) copied, 0.0780103 s, 1.5 GB/s"], "stdout": "", "stdout_lines": []} 2021-02-10T10:58:22.086 INFO:teuthology.task.ansible.out:changed: [gibba013.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '176160768', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-NlMncYOnyQBeUf2HgazZrUXsftKpT1HQZ7ONiRgkbCkAfySu701xFDekTi8yKBWL'], 'uuids': ['0e337204-461d-4966-8a63-8a5a5f070b69']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'holders': [], 'partitions': {}, 'model': None, 'serial': 'PHKM9200001J375A', 'size': '84.00 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-3 bs=1M count=110", "delta": "0:00:00.080451", "end": "2021-02-10 10:58:22.063871", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-NlMncYOnyQBeUf2HgazZrUXsftKpT1HQZ7ONiRgkbCkAfySu701xFDekTi8yKBWL"], "labels": [], "masters": [], "uuids": ["0e337204-461d-4966-8a63-8a5a5f070b69"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "176160768", "sectorsize": "512", "serial": "PHKM9200001J375A", "size": "84.00 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-10 10:58:21.983420", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB) copied, 0.0783072 s, 1.5 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB) copied, 0.0783072 s, 1.5 GB/s"], "stdout": "", "stdout_lines": []} 2021-02-10T10:58:22.368 INFO:teuthology.task.ansible.out:changed: [gibba002.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '176160768', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-2WBfdgWiAT4ia77vCTuZlAyUc0RReyY2f1MVjyWcRwdmfhiojuNZzH4Fjfh0Rg18'], 'uuids': ['0dac7cc3-6ceb-4955-a8a2-9859b7b9d4d4']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'holders': [], 'partitions': {}, 'model': None, 'serial': 'PHKM9200008B375A', 'size': '84.00 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-0 bs=1M count=110", "delta": "0:00:00.079579", "end": "2021-02-10 10:58:22.344336", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-2WBfdgWiAT4ia77vCTuZlAyUc0RReyY2f1MVjyWcRwdmfhiojuNZzH4Fjfh0Rg18"], "labels": [], "masters": [], "uuids": ["0dac7cc3-6ceb-4955-a8a2-9859b7b9d4d4"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "176160768", "sectorsize": "512", "serial": "PHKM9200008B375A", "size": "84.00 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-10 10:58:22.264757", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB) copied, 0.0774901 s, 1.5 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB) copied, 0.0774901 s, 1.5 GB/s"], "stdout": "", "stdout_lines": []} 2021-02-10T10:58:22.599 INFO:teuthology.task.ansible.out:changed: [gibba013.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '176160768', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-NlMncYOnyQBeUf2HgazZrUXsftKpT1HQnt2HTQXVCgXehT4pQ8KHPZoWZz93tpIO'], 'uuids': ['0de34736-6834-4254-b402-9e60b03bcd41']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'holders': [], 'partitions': {}, 'model': None, 'serial': 'PHKM9200001J375A', 'size': '84.00 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-0 bs=1M count=110", "delta": "0:00:00.080589", "end": "2021-02-10 10:58:22.576757", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-NlMncYOnyQBeUf2HgazZrUXsftKpT1HQnt2HTQXVCgXehT4pQ8KHPZoWZz93tpIO"], "labels": [], "masters": [], "uuids": ["0de34736-6834-4254-b402-9e60b03bcd41"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "176160768", "sectorsize": "512", "serial": "PHKM9200001J375A", "size": "84.00 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-10 10:58:22.496168", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB) copied, 0.0785969 s, 1.5 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB) copied, 0.0785969 s, 1.5 GB/s"], "stdout": "", "stdout_lines": []} 2021-02-10T10:58:22.872 INFO:teuthology.task.ansible.out:changed: [gibba002.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '176160768', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-2WBfdgWiAT4ia77vCTuZlAyUc0RReyY2Clr8lcequIZ7066HyrMtNoBIamtJcMy0'], 'uuids': ['51a2e578-c647-45ed-9660-72ee057ae6e4']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'holders': [], 'partitions': {}, 'model': None, 'serial': 'PHKM9200008B375A', 'size': '84.00 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-1 bs=1M count=110", "delta": "0:00:00.080802", "end": "2021-02-10 10:58:22.854729", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-2WBfdgWiAT4ia77vCTuZlAyUc0RReyY2Clr8lcequIZ7066HyrMtNoBIamtJcMy0"], "labels": [], "masters": [], "uuids": ["51a2e578-c647-45ed-9660-72ee057ae6e4"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "176160768", "sectorsize": "512", "serial": "PHKM9200008B375A", "size": "84.00 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-10 10:58:22.773927", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB) copied, 0.0787875 s, 1.5 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB) copied, 0.0787875 s, 1.5 GB/s"], "stdout": "", "stdout_lines": []} 2021-02-10T10:58:22.926 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:22 +0000 (0:00:02.880) 0:06:03.227 **** 2021-02-10T10:58:23.113 INFO:teuthology.task.ansible.out:changed: [gibba013.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '176160768', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-NlMncYOnyQBeUf2HgazZrUXsftKpT1HQNz0CDkdIeZtBPJRzLBW7LuQsxzsouWnN'], 'uuids': ['e68e8bad-acb3-43a9-be98-ee4712cf44df']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'holders': [], 'partitions': {}, 'model': None, 'serial': 'PHKM9200001J375A', 'size': '84.00 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-1 bs=1M count=110", "delta": "0:00:00.080710", "end": "2021-02-10 10:58:23.094163", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-NlMncYOnyQBeUf2HgazZrUXsftKpT1HQNz0CDkdIeZtBPJRzLBW7LuQsxzsouWnN"], "labels": [], "masters": [], "uuids": ["e68e8bad-acb3-43a9-be98-ee4712cf44df"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "176160768", "sectorsize": "512", "serial": "PHKM9200001J375A", "size": "84.00 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-10 10:58:23.013453", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB) copied, 0.078576 s, 1.5 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB) copied, 0.078576 s, 1.5 GB/s"], "stdout": "", "stdout_lines": []} 2021-02-10T10:58:23.165 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:23 +0000 (0:00:00.238) 0:06:03.466 **** 2021-02-10T10:58:23.473 INFO:teuthology.task.ansible.out: TASK [testnode : Remove all LVM data] ****************************************** 2021-02-10T10:58:23.474 INFO:teuthology.task.ansible.out:changed: [gibba002.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "cmd": "dmsetup remove_all --force", "delta": "0:00:00.097159", "end": "2021-02-10 10:58:23.448833", "rc": 0, "start": "2021-02-10 10:58:23.351674", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-10T10:58:23.511 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:23 +0000 (0:00:00.345) 0:06:03.812 **** 2021-02-10T10:58:23.699 INFO:teuthology.task.ansible.out:changed: [gibba013.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "cmd": "dmsetup remove_all --force", "delta": "0:00:00.081904", "end": "2021-02-10 10:58:23.672514", "rc": 0, "start": "2021-02-10 10:58:23.590610", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-10T10:58:23.740 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:23 +0000 (0:00:00.229) 0:06:04.042 **** 2021-02-10T10:58:23.996 INFO:teuthology.task.ansible.out: TASK [testnode : Check for physical volumes] *********************************** 2021-02-10T10:58:23.998 INFO:teuthology.task.ansible.out:changed: [gibba002.front.sepia.ceph.com] => {"changed": true, "cmd": "pvdisplay | grep 'PV Name' | awk '{ print $3 }'", "delta": "0:00:00.032902", "end": "2021-02-10 10:58:23.974728", "rc": 0, "start": "2021-02-10 10:58:23.941826", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-10T10:58:24.032 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:24 +0000 (0:00:00.291) 0:06:04.334 **** 2021-02-10T10:58:24.060 INFO:teuthology.task.ansible.out: TASK [testnode : Remove physical volumes] ************************************** 2021-02-10T10:58:24.096 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:24 +0000 (0:00:00.063) 0:06:04.398 **** 2021-02-10T10:58:24.123 INFO:teuthology.task.ansible.out: TASK [testnode : Write a new partition table to {{ item.value.device }}] ******* 2021-02-10T10:58:24.124 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:24.160 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:24 +0000 (0:00:00.063) 0:06:04.461 **** 2021-02-10T10:58:24.187 INFO:teuthology.task.ansible.out: TASK [testnode : Write partition entries to {{ item.0.device }}] *************** skipping: [gibba002.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:24.224 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:24 +0000 (0:00:00.063) 0:06:04.525 **** 2021-02-10T10:58:24.233 INFO:teuthology.task.ansible.out: TASK [testnode : Check for physical volumes] *********************************** 2021-02-10T10:58:24.234 INFO:teuthology.task.ansible.out:changed: [gibba013.front.sepia.ceph.com] => {"changed": true, "cmd": "pvdisplay | grep 'PV Name' | awk '{ print $3 }'", "delta": "0:00:00.039419", "end": "2021-02-10 10:58:24.207326", "rc": 0, "start": "2021-02-10 10:58:24.167907", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-10T10:58:24.270 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:24 +0000 (0:00:00.045) 0:06:04.571 **** 2021-02-10T10:58:24.276 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2021-02-10T10:58:24.277 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:24.320 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:24 +0000 (0:00:00.049) 0:06:04.621 **** 2021-02-10T10:58:24.327 INFO:teuthology.task.ansible.out: TASK [testnode : Remove physical volumes] ************************************** 2021-02-10T10:58:24.365 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:24 +0000 (0:00:00.045) 0:06:04.666 **** 2021-02-10T10:58:24.371 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs for {{ item.0.device }}] ****************** 2021-02-10T10:58:24.372 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:24.407 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:24 +0000 (0:00:00.041) 0:06:04.708 **** 2021-02-10T10:58:24.413 INFO:teuthology.task.ansible.out: TASK [testnode : Write a new partition table to {{ item.value.device }}] ******* 2021-02-10T10:58:24.415 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:24.455 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:24 +0000 (0:00:00.048) 0:06:04.756 **** 2021-02-10T10:58:24.492 INFO:teuthology.task.ansible.out: TASK [testnode : Write partition entries to {{ item.0.device }}] *************** 2021-02-10T10:58:24.494 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:24.526 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:24 +0000 (0:00:00.071) 0:06:04.828 **** 2021-02-10T10:58:24.553 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2021-02-10T10:58:24.555 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:24.589 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:24 +0000 (0:00:00.063) 0:06:04.891 **** 2021-02-10T10:58:24.627 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs for {{ item.0.device }}] ****************** 2021-02-10T10:58:24.628 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:24.660 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:24 +0000 (0:00:00.070) 0:06:04.961 **** 2021-02-10T10:58:24.865 INFO:teuthology.task.ansible.out: TASK [testnode : Edit lvm.conf to support LVM on kRBD.] ************************ 2021-02-10T10:58:24.867 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-02-10T10:58:24.901 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:24 +0000 (0:00:00.241) 0:06:05.203 **** 2021-02-10T10:58:24.962 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2021-02-10T10:58:25.191 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => (item={'block_used': 13742194, 'uuid': 'aeb0602f-c6fc-4ca1-8920-0340080ffa5a', 'size_total': 984369086464, 'block_total': 240324484, 'mount': '/', 'block_available': 226582290, 'size_available': 928081059840, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,seclabel,relatime,data=ordered', 'device': '/dev/sda1', 'inode_used': 90165, 'block_size': 4096, 'inode_available': 60964811}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226582290, "block_size": 4096, "block_total": 240324484, "block_used": 13742194, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60964811, "inode_total": 61054976, "inode_used": 90165, "mount": "/", "options": "rw,seclabel,relatime,data=ordered", "size_available": 928081059840, "size_total": 984369086464, "uuid": "aeb0602f-c6fc-4ca1-8920-0340080ffa5a"}, "skip_reason": "Conditional result was False"} Wednesday 10 February 2021 10:58:24 +0000 (0:00:00.094) 0:06:05.297 **** TASK [testnode : Combine list of non-root disks] ******************************* skipping: [gibba002.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Wednesday 10 February 2021 10:58:25 +0000 (0:00:00.079) 0:06:05.377 **** TASK [testnode : set_fact] ***************************************************** skipping: [gibba002.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Wednesday 10 February 2021 10:58:25 +0000 (0:00:00.061) 0:06:05.438 **** TASK [testnode : Edit lvm.conf to support LVM on kRBD.] ************************ ok: [gibba013.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-02-10T10:58:25.199 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:25 +0000 (0:00:00.061) 0:06:05.500 **** 2021-02-10T10:58:25.208 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-02-10T10:58:25.210 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:25.244 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:25 +0000 (0:00:00.045) 0:06:05.546 **** 2021-02-10T10:58:25.261 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2021-02-10T10:58:25.263 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => (item={'block_used': 13742185, 'uuid': 'aeb0602f-c6fc-4ca1-8920-0340080ffa5a', 'size_total': 984369086464, 'block_total': 240324484, 'mount': '/', 'block_available': 226582299, 'size_available': 928081096704, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,seclabel,relatime,data=ordered', 'device': '/dev/sda1', 'inode_used': 90165, 'block_size': 4096, 'inode_available': 60964811}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226582299, "block_size": 4096, "block_total": 240324484, "block_used": 13742185, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60964811, "inode_total": 61054976, "inode_used": 90165, "mount": "/", "options": "rw,seclabel,relatime,data=ordered", "size_available": 928081096704, "size_total": 984369086464, "uuid": "aeb0602f-c6fc-4ca1-8920-0340080ffa5a"}, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:25.300 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:25 +0000 (0:00:00.055) 0:06:05.601 **** 2021-02-10T10:58:25.307 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume_groups dict] ************************************ skipping: [gibba002.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:25.343 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:25 +0000 (0:00:00.042) 0:06:05.644 **** 2021-02-10T10:58:25.350 INFO:teuthology.task.ansible.out: TASK [testnode : Combine list of non-root disks] ******************************* 2021-02-10T10:58:25.351 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:25.387 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:25 +0000 (0:00:00.044) 0:06:05.688 **** 2021-02-10T10:58:25.393 INFO:teuthology.task.ansible.out: TASK [testnode : Determine desired logical volume percentage size] ************* skipping: [gibba002.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:25.428 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:25 +0000 (0:00:00.041) 0:06:05.729 **** 2021-02-10T10:58:25.434 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-02-10T10:58:25.436 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:25.470 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:25 +0000 (0:00:00.041) 0:06:05.771 **** 2021-02-10T10:58:25.476 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical_volumes dict] ********************************** 2021-02-10T10:58:25.478 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:25.511 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:25 +0000 (0:00:00.041) 0:06:05.813 **** 2021-02-10T10:58:25.518 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-02-10T10:58:25.519 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:25.553 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:25 +0000 (0:00:00.042) 0:06:05.855 **** 2021-02-10T10:58:25.581 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume_groups dict] ************************************ skipping: [gibba013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:25.614 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:25 +0000 (0:00:00.060) 0:06:05.916 **** 2021-02-10T10:58:25.642 INFO:teuthology.task.ansible.out: TASK [testnode : Determine desired logical volume percentage size] ************* 2021-02-10T10:58:25.644 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:25.679 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:25 +0000 (0:00:00.064) 0:06:05.980 **** 2021-02-10T10:58:25.706 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical_volumes dict] ********************************** 2021-02-10T10:58:25.707 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:25.742 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:25 +0000 (0:00:00.063) 0:06:06.043 **** 2021-02-10T10:58:26.359 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume group(s)] *************************************** 2021-02-10T10:58:26.360 INFO:teuthology.task.ansible.out:changed: [gibba002.front.sepia.ceph.com] => (item={'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "vg_nvme", "value": {"pvs": "/dev/nvme0n1"}}} 2021-02-10T10:58:26.396 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:26 +0000 (0:00:00.654) 0:06:06.698 **** 2021-02-10T10:58:26.403 INFO:teuthology.task.ansible.out:changed: [gibba013.front.sepia.ceph.com] => (item={'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "vg_nvme", "value": {"pvs": "/dev/nvme0n1"}}} 2021-02-10T10:58:26.440 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:26 +0000 (0:00:00.043) 0:06:06.742 **** 2021-02-10T10:58:27.361 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* changed: [gibba013.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '84G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "84G", "vg": "vg_nvme"}}, "msg": ""} 2021-02-10T10:58:27.394 INFO:teuthology.task.ansible.out:changed: [gibba002.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '84G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "84G", "vg": "vg_nvme"}}, "msg": ""} 2021-02-10T10:58:27.917 INFO:teuthology.task.ansible.out:changed: [gibba013.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '84G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "84G", "vg": "vg_nvme"}}, "msg": ""} 2021-02-10T10:58:27.948 INFO:teuthology.task.ansible.out:changed: [gibba002.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '84G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "84G", "vg": "vg_nvme"}}, "msg": ""} 2021-02-10T10:58:28.478 INFO:teuthology.task.ansible.out:changed: [gibba002.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '84G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "84G", "vg": "vg_nvme"}}, "msg": ""} 2021-02-10T10:58:28.479 INFO:teuthology.task.ansible.out: 2021-02-10T10:58:28.512 INFO:teuthology.task.ansible.out:changed: [gibba013.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '84G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "84G", "vg": "vg_nvme"}}, "msg": ""} 2021-02-10T10:58:29.037 INFO:teuthology.task.ansible.out:changed: [gibba002.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '84G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "84G", "vg": "vg_nvme"}}, "msg": ""} 2021-02-10T10:58:29.048 INFO:teuthology.task.ansible.out:changed: [gibba013.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '84G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "84G", "vg": "vg_nvme"}}, "msg": ""} 2021-02-10T10:58:29.550 INFO:teuthology.task.ansible.out:changed: [gibba002.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '13G'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_5", "value": {"size": "13G", "vg": "vg_nvme"}}, "msg": ""} 2021-02-10T10:58:29.590 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:29 +0000 (0:00:03.149) 0:06:09.891 **** 2021-02-10T10:58:29.597 INFO:teuthology.task.ansible.out:changed: [gibba013.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '13G'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_5", "value": {"size": "13G", "vg": "vg_nvme"}}, "msg": ""} 2021-02-10T10:58:29.636 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:29 +0000 (0:00:00.046) 0:06:09.938 **** 2021-02-10T10:58:30.043 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2021-02-10T10:58:30.044 INFO:teuthology.task.ansible.out:changed: [gibba002.front.sepia.ceph.com] => {"changed": true, "path": "/scratch_devs", "state": "absent"} 2021-02-10T10:58:30.078 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:30 +0000 (0:00:00.442) 0:06:10.380 **** 2021-02-10T10:58:30.096 INFO:teuthology.task.ansible.out:changed: [gibba013.front.sepia.ceph.com] => {"changed": true, "path": "/scratch_devs", "state": "absent"} 2021-02-10T10:58:30.131 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:30 +0000 (0:00:00.052) 0:06:10.432 **** 2021-02-10T10:58:30.544 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2021-02-10T10:58:30.545 INFO:teuthology.task.ansible.out:changed: [gibba002.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '84G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "84G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-02-10T10:58:30.597 INFO:teuthology.task.ansible.out:changed: [gibba013.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '84G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "84G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-02-10T10:58:30.974 INFO:teuthology.task.ansible.out:changed: [gibba002.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '84G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "84G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-02-10T10:58:31.038 INFO:teuthology.task.ansible.out:changed: [gibba013.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '84G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "84G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-02-10T10:58:31.414 INFO:teuthology.task.ansible.out:changed: [gibba002.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '84G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "84G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-02-10T10:58:31.467 INFO:teuthology.task.ansible.out:changed: [gibba013.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '84G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "84G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-02-10T10:58:31.842 INFO:teuthology.task.ansible.out:changed: [gibba002.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '84G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "84G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-02-10T10:58:31.854 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '13G'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "lv_5", "value": {"size": "13G", "vg": "vg_nvme"}}, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:31.893 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:31 +0000 (0:00:01.761) 0:06:12.194 **** 2021-02-10T10:58:31.901 INFO:teuthology.task.ansible.out:changed: [gibba013.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '84G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "84G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-02-10T10:58:31.913 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '13G'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "lv_5", "value": {"size": "13G", "vg": "vg_nvme"}}, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:31.948 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:31 +0000 (0:00:00.055) 0:06:12.250 **** 2021-02-10T10:58:31.954 INFO:teuthology.task.ansible.out: TASK [testnode : Create filesystems] ******************************************* 2021-02-10T10:58:31.955 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:31.992 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:31 +0000 (0:00:00.043) 0:06:12.294 **** 2021-02-10T10:58:31.998 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:32.032 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:32 +0000 (0:00:00.039) 0:06:12.333 **** 2021-02-10T10:58:32.036 INFO:teuthology.task.ansible.out: TASK [testnode : Mount filesystems] ******************************************** 2021-02-10T10:58:32.038 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:32.070 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:32 +0000 (0:00:00.038) 0:06:12.372 **** 2021-02-10T10:58:32.077 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:32.113 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:32 +0000 (0:00:00.042) 0:06:12.414 **** 2021-02-10T10:58:32.514 INFO:teuthology.task.ansible.out: TASK [testnode : Create /var/lib/ceph] ***************************************** 2021-02-10T10:58:32.516 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/var/lib/ceph", "secontext": "unconfined_u:object_r:var_lib_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-02-10T10:58:32.548 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:32 +0000 (0:00:00.435) 0:06:12.850 **** 2021-02-10T10:58:32.566 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/var/lib/ceph", "secontext": "unconfined_u:object_r:var_lib_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-02-10T10:58:32.607 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:32 +0000 (0:00:00.058) 0:06:12.908 **** 2021-02-10T10:58:32.613 INFO:teuthology.task.ansible.out: TASK [testnode : Set xfs_opts on newer OSes] *********************************** 2021-02-10T10:58:32.614 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:32.652 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:32 +0000 (0:00:00.045) 0:06:12.953 **** 2021-02-10T10:58:32.658 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:32.693 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:32 +0000 (0:00:00.041) 0:06:12.995 **** 2021-02-10T10:58:33.417 INFO:teuthology.task.ansible.out: TASK [testnode : Create xfs filesystem on /dev/vg_nvme/lv_5] ******************* 2021-02-10T10:58:33.419 INFO:teuthology.task.ansible.out:changed: [gibba002.front.sepia.ceph.com] => {"changed": true} 2021-02-10T10:58:33.453 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:33 +0000 (0:00:00.759) 0:06:13.755 **** 2021-02-10T10:58:33.460 INFO:teuthology.task.ansible.out:changed: [gibba013.front.sepia.ceph.com] => {"changed": true} 2021-02-10T10:58:33.495 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:33 +0000 (0:00:00.041) 0:06:13.797 **** 2021-02-10T10:58:34.247 INFO:teuthology.task.ansible.out: TASK [testnode : Mount /dev/vg_nvme/lv_5 to /var/lib/ceph] ********************* 2021-02-10T10:58:34.248 INFO:teuthology.task.ansible.out:changed: [gibba013.front.sepia.ceph.com] => {"changed": true, "dump": "0", "fstab": "/etc/fstab", "fstype": "xfs", "name": "/var/lib/ceph", "opts": "defaults,nofail", "passno": "0", "src": "/dev/vg_nvme/lv_5"} 2021-02-10T10:58:34.280 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:34 +0000 (0:00:00.784) 0:06:14.581 **** 2021-02-10T10:58:34.308 INFO:teuthology.task.ansible.out: TASK [testnode : Install cachefilesd] ****************************************** skipping: [gibba013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:34.341 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:34 +0000 (0:00:00.061) 0:06:14.642 **** 2021-02-10T10:58:34.367 INFO:teuthology.task.ansible.out: TASK [testnode : Install cachefilesd conf file] ******************************** 2021-02-10T10:58:34.369 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:34.416 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:34 +0000 (0:00:00.075) 0:06:14.718 **** 2021-02-10T10:58:34.445 INFO:teuthology.task.ansible.out: TASK [testnode : Restart cachefilesd] ****************************************** 2021-02-10T10:58:34.446 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:34.479 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:34 +0000 (0:00:00.062) 0:06:14.780 **** 2021-02-10T10:58:34.506 INFO:teuthology.task.ansible.out: TASK [testnode : Restart cachefilesd] ****************************************** 2021-02-10T10:58:34.508 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:34.540 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:34 +0000 (0:00:00.060) 0:06:14.841 **** 2021-02-10T10:58:34.567 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the teuthology user.] *********** 2021-02-10T10:58:34.568 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:34.603 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:34 +0000 (0:00:00.062) 0:06:14.904 **** 2021-02-10T10:58:34.630 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the root user.] ***************** 2021-02-10T10:58:34.631 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:34.666 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:34 +0000 (0:00:00.062) 0:06:14.967 **** 2021-02-10T10:58:34.694 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for the teuthology user.] ****************** 2021-02-10T10:58:34.695 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:34.728 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:34 +0000 (0:00:00.061) 0:06:15.029 **** 2021-02-10T10:58:34.766 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for root.] ********************************* 2021-02-10T10:58:34.767 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:34.801 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:34 +0000 (0:00:00.072) 0:06:15.102 **** 2021-02-10T10:58:34.828 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure perl-doc is installed on apt systems.] ***************** 2021-02-10T10:58:34.829 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:34.861 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:34 +0000 (0:00:00.059) 0:06:15.162 **** 2021-02-10T10:58:34.888 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if Amazon::S3 is installed.] ********************* 2021-02-10T10:58:34.889 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:34.922 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:34 +0000 (0:00:00.061) 0:06:15.224 **** 2021-02-10T10:58:34.949 INFO:teuthology.task.ansible.out: TASK [testnode : Install Amazon::S3.] ****************************************** 2021-02-10T10:58:34.950 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:34.982 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:34 +0000 (0:00:00.060) 0:06:15.284 **** 2021-02-10T10:58:35.250 INFO:teuthology.task.ansible.out: TASK [testnode : Mount /dev/vg_nvme/lv_5 to /var/lib/ceph] ********************* 2021-02-10T10:58:35.252 INFO:teuthology.task.ansible.out:changed: [gibba002.front.sepia.ceph.com] => {"changed": true, "dump": "0", "fstab": "/etc/fstab", "fstype": "xfs", "name": "/var/lib/ceph", "opts": "defaults,nofail", "passno": "0", "src": "/dev/vg_nvme/lv_5"} 2021-02-10T10:58:35.287 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:35 +0000 (0:00:00.304) 0:06:15.588 **** 2021-02-10T10:58:35.315 INFO:teuthology.task.ansible.out: TASK [testnode : Install cachefilesd] ****************************************** 2021-02-10T10:58:35.317 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:35.348 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:35 +0000 (0:00:00.061) 0:06:15.650 **** 2021-02-10T10:58:35.375 INFO:teuthology.task.ansible.out: TASK [testnode : Install cachefilesd conf file] ******************************** 2021-02-10T10:58:35.377 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:35.410 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:35 +0000 (0:00:00.061) 0:06:15.711 **** 2021-02-10T10:58:35.437 INFO:teuthology.task.ansible.out: TASK [testnode : Restart cachefilesd] ****************************************** 2021-02-10T10:58:35.438 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:35.470 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:35 +0000 (0:00:00.060) 0:06:15.772 **** 2021-02-10T10:58:35.498 INFO:teuthology.task.ansible.out: TASK [testnode : Restart cachefilesd] ****************************************** 2021-02-10T10:58:35.500 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:35.535 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:35 +0000 (0:00:00.064) 0:06:15.837 **** 2021-02-10T10:58:35.563 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the teuthology user.] *********** 2021-02-10T10:58:35.565 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:35.597 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:35 +0000 (0:00:00.062) 0:06:15.899 **** 2021-02-10T10:58:35.626 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the root user.] ***************** 2021-02-10T10:58:35.627 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:35.663 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:35 +0000 (0:00:00.065) 0:06:15.964 **** 2021-02-10T10:58:35.693 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for the teuthology user.] ****************** 2021-02-10T10:58:35.694 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:35.729 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:35 +0000 (0:00:00.066) 0:06:16.030 **** 2021-02-10T10:58:35.758 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for root.] ********************************* 2021-02-10T10:58:35.759 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:35.801 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:35 +0000 (0:00:00.072) 0:06:16.103 **** 2021-02-10T10:58:35.810 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on rpm based systems.] ******************** 2021-02-10T10:58:35.811 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => {"changed": false, "msg": "", "rc": 0, "results": ["ntp-4.2.6p5-29.el7_8.2.x86_64 providing ntp is already installed"]} 2021-02-10T10:58:35.847 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:35 +0000 (0:00:00.045) 0:06:16.149 **** 2021-02-10T10:58:35.853 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure perl-doc is installed on apt systems.] ***************** 2021-02-10T10:58:35.853 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:35.893 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:35 +0000 (0:00:00.045) 0:06:16.194 **** 2021-02-10T10:58:35.900 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on apt based systems.] ******************** 2021-02-10T10:58:35.900 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:35.933 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:35 +0000 (0:00:00.040) 0:06:16.235 **** 2021-02-10T10:58:35.940 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if Amazon::S3 is installed.] ********************* 2021-02-10T10:58:35.940 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:35.980 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:35 +0000 (0:00:00.046) 0:06:16.282 **** 2021-02-10T10:58:35.987 INFO:teuthology.task.ansible.out: TASK [testnode : Remove racy /etc/network/if-up.d/ntpdate on xenial] *********** 2021-02-10T10:58:35.988 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:36.026 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:36 +0000 (0:00:00.045) 0:06:16.328 **** 2021-02-10T10:58:36.032 INFO:teuthology.task.ansible.out: TASK [testnode : Install Amazon::S3.] ****************************************** 2021-02-10T10:58:36.034 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:36.083 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:36 +0000 (0:00:00.057) 0:06:16.385 **** 2021-02-10T10:58:36.901 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on rpm based systems.] ******************** 2021-02-10T10:58:36.903 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => {"changed": false, "msg": "", "rc": 0, "results": ["ntp-4.2.6p5-29.el7_8.2.x86_64 providing ntp is already installed"]} 2021-02-10T10:58:36.936 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:36 +0000 (0:00:00.853) 0:06:17.238 **** 2021-02-10T10:58:36.964 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on apt based systems.] ******************** 2021-02-10T10:58:36.966 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:36.998 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:36 +0000 (0:00:00.061) 0:06:17.299 **** 2021-02-10T10:58:37.025 INFO:teuthology.task.ansible.out: TASK [testnode : Remove racy /etc/network/if-up.d/ntpdate on xenial] *********** 2021-02-10T10:58:37.027 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:37.059 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:37 +0000 (0:00:00.061) 0:06:17.361 **** 2021-02-10T10:58:37.987 INFO:teuthology.task.ansible.out: TASK [testnode : Create the ntp.conf file.] ************************************ 2021-02-10T10:58:37.989 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => {"changed": false, "checksum": "645fd5116197cd9da73b328bc11bf633a40d80e1", "dest": "/etc/ntp.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/ntp.conf", "secontext": "system_u:object_r:net_conf_t:s0", "size": 2767, "state": "file", "uid": 0} 2021-02-10T10:58:38.022 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:38 +0000 (0:00:00.962) 0:06:18.323 **** 2021-02-10T10:58:38.052 INFO:teuthology.task.ansible.out: TASK [testnode : Create the chrony.conf file] ********************************** 2021-02-10T10:58:38.053 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:38.086 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:38 +0000 (0:00:00.063) 0:06:18.387 **** 2021-02-10T10:58:38.644 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ntpd is running.] *********************************** 2021-02-10T10:58:38.645 INFO:teuthology.task.ansible.out:changed: [gibba013.front.sepia.ceph.com] => {"changed": true, "enabled": true, "name": "ntpd", "state": "started", "status": {"ActiveEnterTimestampMonotonic": "0", "ActiveExitTimestampMonotonic": "0", "ActiveState": "inactive", "After": "system.slice sntp.service syslog.target basic.target -.mount tmp.mount systemd-journald.socket ntpdate.service", "AllowIsolate": "no", "AmbientCapabilities": "0", "AssertResult": "no", "AssertTimestampMonotonic": "0", "Before": "chronyd.service shutdown.target multi-user.target", "BlockIOAccounting": "no", "BlockIOWeight": "18446744073709551615", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "18446744073709551615", "CanIsolate": "no", "CanReload": "no", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "18446744073709551615", "CollectMode": "inactive", "ConditionResult": "no", "ConditionTimestampMonotonic": "0", "ConflictedBy": "chronyd.service", "Conflicts": "shutdown.target", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Network Time Service", "DevicePolicy": "auto", "EnvironmentFile": "/etc/sysconfig/ntpd (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "0", "ExecMainStartTimestampMonotonic": "0", "ExecMainStatus": "0", "ExecStart": "{ path=/usr/sbin/ntpd ; argv[]=/usr/sbin/ntpd -u ntp:ntp $OPTIONS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/ntpd.service", "GuessMainPID": "yes", "IOScheduling": "0", "Id": "ntpd.service", "IgnoreOnIsolate": "no", "IgnoreOnSnapshot": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestampMonotonic": "0", "JobTimeoutAction": "none", "JobTimeoutUSec": "0", "KillMode": "control-group", "KillSignal": "15", "Limi 2021-02-10T10:58:38.646 INFO:teuthology.task.ansible.out:tAS": "18446744073709551615", "LimitCORE": "18446744073709551615", "LimitCPU": "18446744073709551615", "LimitDATA": "18446744073709551615", "LimitFSIZE": "18446744073709551615", "LimitLOCKS": "18446744073709551615", "LimitMEMLOCK": "65536", "LimitMSGQUEUE": "819200", "LimitNICE": "0", "LimitNOFILE": "4096", "LimitNPROC": "127560", "LimitRSS": "18446744073709551615", "LimitRTPRIO": "0", "LimitRTTIME": "18446744073709551615", "LimitSIGPENDING": "127560", "LimitSTACK": "18446744073709551615", "LoadState": "loaded", "MainPID": "0", "MemoryAccounting": "no", "MemoryCurrent": "18446744073709551615", "MemoryLimit": "18446744073709551615", "MountFlags": "0", "Names": "ntpd.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "ProtectHome": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "Requires": "-.mount basic.target system.slice", "RequiresMountsFor": "/var/tmp", "Restart": "no", "RestartUSec": "100ms", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitInterval": "10000000", "StartupBlockIOWeight": "18446744073709551615", "StartupCPUShares": "18446744073709551615", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "dead", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "no", "TasksCurrent": "18446744073709551615", "TasksMax": "18446744073709551615", "TimeoutStartUSec": "1min 30s", "Timeou 2021-02-10T10:58:38.647 INFO:teuthology.task.ansible.out:tStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UMask": "0022", "UnitFilePreset": "disabled", "UnitFileState": "enabled", "WantedBy": "multi-user.target", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} 2021-02-10T10:58:38.804 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:38 +0000 (0:00:00.718) 0:06:19.106 **** 2021-02-10T10:58:38.832 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-02-10T10:58:38.833 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => {"ansible_facts": {"pip_executable": "pip", "pip_version": "python-pip"}, "changed": false} 2021-02-10T10:58:38.864 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:38 +0000 (0:00:00.059) 0:06:19.166 **** 2021-02-10T10:58:38.892 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-02-10T10:58:38.893 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:38.927 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:38 +0000 (0:00:00.061) 0:06:19.227 **** 2021-02-10T10:58:38.994 INFO:teuthology.task.ansible.out: TASK [testnode : Create the ntp.conf file.] ************************************ 2021-02-10T10:58:38.995 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => {"changed": false, "checksum": "645fd5116197cd9da73b328bc11bf633a40d80e1", "dest": "/etc/ntp.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/ntp.conf", "secontext": "system_u:object_r:net_conf_t:s0", "size": 2767, "state": "file", "uid": 0} 2021-02-10T10:58:39.029 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:39 +0000 (0:00:00.103) 0:06:19.331 **** 2021-02-10T10:58:39.056 INFO:teuthology.task.ansible.out: TASK [testnode : Create the chrony.conf file] ********************************** 2021-02-10T10:58:39.058 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:39.092 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:39 +0000 (0:00:00.063) 0:06:19.394 **** 2021-02-10T10:58:40.055 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on rpm based systems.] ********************* 2021-02-10T10:58:40.056 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => {"changed": false, "msg": "", "rc": 0, "results": ["python2-pip-8.1.2-14.el7.noarch providing python-pip is already installed"]} 2021-02-10T10:58:40.089 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:40 +0000 (0:00:00.996) 0:06:20.391 **** 2021-02-10T10:58:40.117 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on apt based systems.] ********************* 2021-02-10T10:58:40.118 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:40.151 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:40 +0000 (0:00:00.061) 0:06:20.453 **** 2021-02-10T10:58:40.179 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on zypper based systems.] ****************** 2021-02-10T10:58:40.180 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:40.222 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:40 +0000 (0:00:00.071) 0:06:20.524 **** 2021-02-10T10:58:40.628 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ntpd is running.] *********************************** 2021-02-10T10:58:40.629 INFO:teuthology.task.ansible.out:changed: [gibba002.front.sepia.ceph.com] => {"changed": true, "enabled": true, "name": "ntpd", "state": "started", "status": {"ActiveEnterTimestampMonotonic": "0", "ActiveExitTimestampMonotonic": "0", "ActiveState": "inactive", "After": "systemd-journald.socket system.slice tmp.mount -.mount ntpdate.service syslog.target basic.target sntp.service", "AllowIsolate": "no", "AmbientCapabilities": "0", "AssertResult": "no", "AssertTimestampMonotonic": "0", "Before": "multi-user.target chronyd.service shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "18446744073709551615", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "18446744073709551615", "CanIsolate": "no", "CanReload": "no", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "18446744073709551615", "CollectMode": "inactive", "ConditionResult": "no", "ConditionTimestampMonotonic": "0", "ConflictedBy": "chronyd.service", "Conflicts": "shutdown.target", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Network Time Service", "DevicePolicy": "auto", "EnvironmentFile": "/etc/sysconfig/ntpd (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "0", "ExecMainStartTimestampMonotonic": "0", "ExecMainStatus": "0", "ExecStart": "{ path=/usr/sbin/ntpd ; argv[]=/usr/sbin/ntpd -u ntp:ntp $OPTIONS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/ntpd.service", "GuessMainPID": "yes", "IOScheduling": "0", "Id": "ntpd.service", "IgnoreOnIsolate": "no", "IgnoreOnSnapshot": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestampMonotonic": "0", "JobTimeoutAction": "none", "JobTimeoutUSec": "0", "KillMode": "control-group", "KillSignal": "15", "Limi 2021-02-10T10:58:40.630 INFO:teuthology.task.ansible.out:tAS": "18446744073709551615", "LimitCORE": "18446744073709551615", "LimitCPU": "18446744073709551615", "LimitDATA": "18446744073709551615", "LimitFSIZE": "18446744073709551615", "LimitLOCKS": "18446744073709551615", "LimitMEMLOCK": "65536", "LimitMSGQUEUE": "819200", "LimitNICE": "0", "LimitNOFILE": "4096", "LimitNPROC": "127560", "LimitRSS": "18446744073709551615", "LimitRTPRIO": "0", "LimitRTTIME": "18446744073709551615", "LimitSIGPENDING": "127560", "LimitSTACK": "18446744073709551615", "LoadState": "loaded", "MainPID": "0", "MemoryAccounting": "no", "MemoryCurrent": "18446744073709551615", "MemoryLimit": "18446744073709551615", "MountFlags": "0", "Names": "ntpd.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "ProtectHome": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "Requires": "-.mount system.slice basic.target", "RequiresMountsFor": "/var/tmp", "Restart": "no", "RestartUSec": "100ms", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitInterval": "10000000", "StartupBlockIOWeight": "18446744073709551615", "StartupCPUShares": "18446744073709551615", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "dead", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "no", "TasksCurrent": "18446744073709551615", "TasksMax": "18446744073709551615", "TimeoutStartUSec": "1min 30s", "Timeou 2021-02-10T10:58:40.631 INFO:teuthology.task.ansible.out:tStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UMask": "0022", "UnitFilePreset": "disabled", "UnitFileState": "enabled", "WantedBy": "multi-user.target", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} 2021-02-10T10:58:40.671 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:40 +0000 (0:00:00.448) 0:06:20.972 **** 2021-02-10T10:58:40.691 INFO:teuthology.task.ansible.out: TASK [testnode : Create the .pip directory for the teuthology user.] *********** 2021-02-10T10:58:40.692 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => {"changed": false, "gid": 1236, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 4096, "state": "directory", "uid": 1000} 2021-02-10T10:58:40.726 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:40 +0000 (0:00:00.054) 0:06:21.027 **** 2021-02-10T10:58:40.732 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-02-10T10:58:40.734 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => {"ansible_facts": {"pip_executable": "pip", "pip_version": "python-pip"}, "changed": false} 2021-02-10T10:58:40.768 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:40 +0000 (0:00:00.042) 0:06:21.069 **** 2021-02-10T10:58:40.795 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-02-10T10:58:40.797 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:40.828 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:40 +0000 (0:00:00.060) 0:06:21.130 **** 2021-02-10T10:58:41.961 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on rpm based systems.] ********************* 2021-02-10T10:58:41.962 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => {"changed": false, "msg": "", "rc": 0, "results": ["python2-pip-8.1.2-14.el7.noarch providing python-pip is already installed"]} 2021-02-10T10:58:41.994 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:41 +0000 (0:00:01.165) 0:06:22.295 **** 2021-02-10T10:58:42.020 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on apt based systems.] ********************* 2021-02-10T10:58:42.022 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:42.057 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:42 +0000 (0:00:00.062) 0:06:22.358 **** 2021-02-10T10:58:42.084 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on zypper based systems.] ****************** 2021-02-10T10:58:42.085 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:42.118 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:42 +0000 (0:00:00.061) 0:06:22.420 **** 2021-02-10T10:58:42.571 INFO:teuthology.task.ansible.out: TASK [testnode : Create the .pip directory for the teuthology user.] *********** 2021-02-10T10:58:42.572 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => {"changed": false, "gid": 1236, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 4096, "state": "directory", "uid": 1000} 2021-02-10T10:58:42.603 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:42 +0000 (0:00:00.484) 0:06:22.904 **** 2021-02-10T10:58:42.641 INFO:teuthology.task.ansible.out: TASK [testnode : Create pip.conf and configure it to use our mirror] *********** 2021-02-10T10:58:42.642 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => {"changed": false, "checksum": "959d37ab30545da205b389a6b7561b225a478a06", "dest": "/home/ubuntu/.pip/pip.conf", "gid": 1236, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip/pip.conf", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 73, "state": "file", "uid": 1000} 2021-02-10T10:58:42.675 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:42 +0000 (0:00:00.072) 0:06:22.977 **** 2021-02-10T10:58:44.001 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages via pip] ************************************* 2021-02-10T10:58:44.003 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => {"changed": false, "cmd": ["/bin/pip", "install", "remoto>=0.0.35"], "name": ["remoto>=0.0.35"], "requirements": null, "state": "present", "stderr": "You are using pip version 8.1.2, however version 21.0.1 is available.\nYou should consider upgrading via the 'pip install --upgrade pip' command.\n", "stderr_lines": ["You are using pip version 8.1.2, however version 21.0.1 is available.", "You should consider upgrading via the 'pip install --upgrade pip' command."], "stdout": "Requirement already satisfied (use --upgrade to upgrade): remoto>=0.0.35 in /usr/lib/python2.7/site-packages\nRequirement already satisfied (use --upgrade to upgrade): execnet in /usr/lib/python2.7/site-packages (from remoto>=0.0.35)\nRequirement already satisfied (use --upgrade to upgrade): apipkg>=1.4 in /usr/lib/python2.7/site-packages (from execnet->remoto>=0.0.35)\n", "stdout_lines": ["Requirement already satisfied (use --upgrade to upgrade): remoto>=0.0.35 in /usr/lib/python2.7/site-packages", "Requirement already satisfied (use --upgrade to upgrade): execnet in /usr/lib/python2.7/site-packages (from remoto>=0.0.35)", "Requirement already satisfied (use --upgrade to upgrade): apipkg>=1.4 in /usr/lib/python2.7/site-packages (from execnet->remoto>=0.0.35)"], "version": null, "virtualenv": null} 2021-02-10T10:58:44.034 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:44 +0000 (0:00:01.358) 0:06:24.335 **** 2021-02-10T10:58:44.060 INFO:teuthology.task.ansible.out: TASK [testnode : Purge resolvconf] ********************************************* 2021-02-10T10:58:44.062 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:44.094 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:44 +0000 (0:00:00.060) 0:06:24.396 **** 2021-02-10T10:58:44.123 INFO:teuthology.task.ansible.out: TASK [testnode : Set interface] ************************************************ 2021-02-10T10:58:44.125 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => {"ansible_facts": {"interface": "enp1s0f0"}, "changed": false} 2021-02-10T10:58:44.155 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:44 +0000 (0:00:00.060) 0:06:24.457 **** 2021-02-10T10:58:44.534 INFO:teuthology.task.ansible.out: TASK [testnode : Create pip.conf and configure it to use our mirror] *********** 2021-02-10T10:58:44.535 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => {"changed": false, "checksum": "959d37ab30545da205b389a6b7561b225a478a06", "dest": "/home/ubuntu/.pip/pip.conf", "gid": 1236, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip/pip.conf", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 73, "state": "file", "uid": 1000} 2021-02-10T10:58:44.569 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:44 +0000 (0:00:00.413) 0:06:24.871 **** 2021-02-10T10:58:44.596 INFO:teuthology.task.ansible.out: TASK [testnode : Check for presence of /etc/network/interfaces] **************** 2021-02-10T10:58:44.598 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => {"changed": false, "stat": {"exists": false}} 2021-02-10T10:58:44.631 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:44 +0000 (0:00:00.061) 0:06:24.932 **** 2021-02-10T10:58:44.658 INFO:teuthology.task.ansible.out: TASK [testnode : Rewrite /etc/network/interfaces to use dhcp] ****************** 2021-02-10T10:58:44.659 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:44.691 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:44 +0000 (0:00:00.060) 0:06:24.992 **** 2021-02-10T10:58:44.717 INFO:teuthology.task.ansible.out: TASK [testnode : Set bounce_interface if we just enabled dhcp] ***************** 2021-02-10T10:58:44.719 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => {"ansible_facts": {"bounce_interface": false}, "changed": false} 2021-02-10T10:58:44.751 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:44 +0000 (0:00:00.059) 0:06:25.052 **** 2021-02-10T10:58:44.777 INFO:teuthology.task.ansible.out: TASK [testnode : ifdown and ifup] ********************************************** 2021-02-10T10:58:44.779 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:44.811 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:44 +0000 (0:00:00.060) 0:06:25.113 **** 2021-02-10T10:58:45.256 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure lab_domain is in search domains in /etc/resolv.conf] *** 2021-02-10T10:58:45.257 INFO:teuthology.task.ansible.out:changed: [gibba013.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line replaced"} 2021-02-10T10:58:45.290 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:45 +0000 (0:00:00.478) 0:06:25.592 **** 2021-02-10T10:58:45.709 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages via pip] ************************************* 2021-02-10T10:58:45.710 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => {"changed": false, "cmd": ["/bin/pip", "install", "remoto>=0.0.35"], "name": ["remoto>=0.0.35"], "requirements": null, "state": "present", "stderr": "You are using pip version 8.1.2, however version 21.0.1 is available.\nYou should consider upgrading via the 'pip install --upgrade pip' command.\n", "stderr_lines": ["You are using pip version 8.1.2, however version 21.0.1 is available.", "You should consider upgrading via the 'pip install --upgrade pip' command."], "stdout": "Requirement already satisfied (use --upgrade to upgrade): remoto>=0.0.35 in /usr/lib/python2.7/site-packages\nRequirement already satisfied (use --upgrade to upgrade): execnet in /usr/lib/python2.7/site-packages (from remoto>=0.0.35)\nRequirement already satisfied (use --upgrade to upgrade): apipkg>=1.4 in /usr/lib/python2.7/site-packages (from execnet->remoto>=0.0.35)\n", "stdout_lines": ["Requirement already satisfied (use --upgrade to upgrade): remoto>=0.0.35 in /usr/lib/python2.7/site-packages", "Requirement already satisfied (use --upgrade to upgrade): execnet in /usr/lib/python2.7/site-packages (from remoto>=0.0.35)", "Requirement already satisfied (use --upgrade to upgrade): apipkg>=1.4 in /usr/lib/python2.7/site-packages (from execnet->remoto>=0.0.35)"], "version": null, "virtualenv": null} 2021-02-10T10:58:45.744 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:45 +0000 (0:00:00.453) 0:06:26.045 **** 2021-02-10T10:58:45.751 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure domain is set in /etc/resolv.conf] ********************* 2021-02-10T10:58:45.751 INFO:teuthology.task.ansible.out:changed: [gibba013.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line added"} 2021-02-10T10:58:45.785 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:45 +0000 (0:00:00.041) 0:06:26.087 **** 2021-02-10T10:58:45.792 INFO:teuthology.task.ansible.out: TASK [testnode : Purge resolvconf] ********************************************* 2021-02-10T10:58:45.793 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:45.825 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:45 +0000 (0:00:00.039) 0:06:26.127 **** 2021-02-10T10:58:45.831 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ethtool is installed] ******************************* 2021-02-10T10:58:45.833 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:45.866 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:45 +0000 (0:00:00.040) 0:06:26.168 **** 2021-02-10T10:58:45.872 INFO:teuthology.task.ansible.out: TASK [testnode : Set interface] ************************************************ 2021-02-10T10:58:45.873 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => {"ansible_facts": {"interface": "enp1s0f0"}, "changed": false} 2021-02-10T10:58:45.908 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:45 +0000 (0:00:00.041) 0:06:26.210 **** 2021-02-10T10:58:45.915 INFO:teuthology.task.ansible.out: TASK [testnode : grep ethtool for secondary NIC MAC address] ******************* 2021-02-10T10:58:45.916 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => (item=eno1) => {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False"} 2021-02-10T10:58:45.918 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => (item=eno2) => {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False"} 2021-02-10T10:58:45.919 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => (item=lo) => {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False"} 2021-02-10T10:58:45.920 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => (item=enp1s0f1) => {"ansible_loop_var": "item", "changed": false, "item": "enp1s0f1", "skip_reason": "Conditional result was False"} skipping: [gibba013.front.sepia.ceph.com] => (item=enp1s0f0) => {"ansible_loop_var": "item", "changed": false, "item": "enp1s0f0", "skip_reason": "Conditional result was False"} 2021-02-10T10:58:45.956 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:45 +0000 (0:00:00.048) 0:06:26.258 **** 2021-02-10T10:58:45.995 INFO:teuthology.task.ansible.out: TASK [testnode : Define net_to_configure var] ********************************** 2021-02-10T10:58:45.996 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [gibba013.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno2', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:46.006 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'lo', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:46.007 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp1s0f1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp1s0f1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:46.018 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp1s0f0', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp1s0f0", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:46.052 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:46 +0000 (0:00:00.095) 0:06:26.354 **** 2021-02-10T10:58:46.079 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-02-10T10:58:46.081 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:46.118 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:46 +0000 (0:00:00.065) 0:06:26.420 **** 2021-02-10T10:58:46.148 INFO:teuthology.task.ansible.out: TASK [testnode : Check if {{ nic_to_configure }} is 10Gb] ********************** 2021-02-10T10:58:46.150 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:46.182 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:46 +0000 (0:00:00.063) 0:06:26.484 **** 2021-02-10T10:58:46.209 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU to 9000 if 10Gb] ************************************** 2021-02-10T10:58:46.210 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:46.245 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:46 +0000 (0:00:00.063) 0:06:26.547 **** 2021-02-10T10:58:46.272 INFO:teuthology.task.ansible.out: TASK [testnode : Write Ubuntu network config for {{ nic_to_configure }}] ******* 2021-02-10T10:58:46.274 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:46.305 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:46 +0000 (0:00:00.060) 0:06:26.607 **** 2021-02-10T10:58:46.333 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU on Ubuntu] ******************************************** 2021-02-10T10:58:46.334 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:46.366 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:46 +0000 (0:00:00.060) 0:06:26.668 **** 2021-02-10T10:58:46.372 INFO:teuthology.task.ansible.out: TASK [testnode : Check for presence of /etc/network/interfaces] **************** 2021-02-10T10:58:46.372 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => {"changed": false, "stat": {"exists": false}} 2021-02-10T10:58:46.408 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:46 +0000 (0:00:00.041) 0:06:26.709 **** 2021-02-10T10:58:46.414 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2021-02-10T10:58:46.415 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:46.454 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:46 +0000 (0:00:00.045) 0:06:26.755 **** 2021-02-10T10:58:46.460 INFO:teuthology.task.ansible.out: TASK [testnode : Rewrite /etc/network/interfaces to use dhcp] ****************** 2021-02-10T10:58:46.460 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:46.495 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:46 +0000 (0:00:00.041) 0:06:26.797 **** 2021-02-10T10:58:46.503 INFO:teuthology.task.ansible.out: TASK [testnode : Write RHEL/CentOS network config for {{ nic_to_configure }}] *** 2021-02-10T10:58:46.696 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Wednesday 10 February 2021 10:58:46 +0000 (0:00:00.047) 0:06:26.845 **** TASK [testnode : Set bounce_interface if we just enabled dhcp] ***************** ok: [gibba002.front.sepia.ceph.com] => {"ansible_facts": {"bounce_interface": false}, "changed": false} Wednesday 10 February 2021 10:58:46 +0000 (0:00:00.041) 0:06:26.886 **** TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** skipping: [gibba013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Wednesday 10 February 2021 10:58:46 +0000 (0:00:00.041) 0:06:26.927 **** TASK [testnode : ifdown and ifup] ********************************************** skipping: [gibba002.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Wednesday 10 February 2021 10:58:46 +0000 (0:00:00.040) 0:06:26.968 **** 2021-02-10T10:58:47.092 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2021-02-10T10:58:47.092 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item=/var/run/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/run/ -name \"*ceph*\"", "delta": "0:00:00.004520", "end": "2021-02-10 10:58:47.060228", "failed_when_result": false, "item": "/var/run/", "rc": 0, "start": "2021-02-10 10:58:47.055708", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-10T10:58:47.115 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure lab_domain is in search domains in /etc/resolv.conf] *** 2021-02-10T10:58:47.116 INFO:teuthology.task.ansible.out:changed: [gibba002.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line replaced"} 2021-02-10T10:58:47.149 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:47 +0000 (0:00:00.482) 0:06:27.450 **** 2021-02-10T10:58:47.520 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2021-02-10T10:58:47.521 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item=/etc/systemd/system/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/systemd/system/ -name \"*ceph*\"", "delta": "0:00:00.002234", "end": "2021-02-10 10:58:47.492810", "failed_when_result": false, "item": "/etc/systemd/system/", "rc": 0, "start": "2021-02-10 10:58:47.490576", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-10T10:58:47.595 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure domain is set in /etc/resolv.conf] ********************* 2021-02-10T10:58:47.596 INFO:teuthology.task.ansible.out:changed: [gibba002.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line added"} 2021-02-10T10:58:47.628 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:47 +0000 (0:00:00.479) 0:06:27.930 **** 2021-02-10T10:58:47.655 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ethtool is installed] ******************************* 2021-02-10T10:58:47.657 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:47.688 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:47 +0000 (0:00:00.059) 0:06:27.990 **** 2021-02-10T10:58:47.727 INFO:teuthology.task.ansible.out: TASK [testnode : grep ethtool for secondary NIC MAC address] ******************* 2021-02-10T10:58:47.728 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => (item=eno1) => {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False"} 2021-02-10T10:58:47.738 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => (item=eno2) => {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False"} 2021-02-10T10:58:47.739 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => (item=lo) => {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False"} 2021-02-10T10:58:47.749 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => (item=enp1s0f1) => {"ansible_loop_var": "item", "changed": false, "item": "enp1s0f1", "skip_reason": "Conditional result was False"} 2021-02-10T10:58:47.751 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => (item=enp1s0f0) => {"ansible_loop_var": "item", "changed": false, "item": "enp1s0f0", "skip_reason": "Conditional result was False"} 2021-02-10T10:58:47.790 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:47 +0000 (0:00:00.101) 0:06:28.091 **** 2021-02-10T10:58:47.828 INFO:teuthology.task.ansible.out: TASK [testnode : Define net_to_configure var] ********************************** 2021-02-10T10:58:47.829 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [gibba002.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno2', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:47.840 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'lo', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:47.841 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp1s0f1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp1s0f1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:47.854 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp1s0f0', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp1s0f0", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:47.889 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:47 +0000 (0:00:00.099) 0:06:28.190 **** 2021-02-10T10:58:47.916 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-02-10T10:58:47.918 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:47.951 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:47 +0000 (0:00:00.062) 0:06:28.253 **** 2021-02-10T10:58:47.979 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2021-02-10T10:58:47.981 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item=/etc/ceph) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/ceph -name \"*ceph*\"", "delta": "0:00:00.002156", "end": "2021-02-10 10:58:47.953424", "failed_when_result": false, "item": "/etc/ceph", "msg": "non-zero return code", "rc": 1, "start": "2021-02-10 10:58:47.951268", "stderr": "find: ‘/etc/ceph’: No such file or directory", "stderr_lines": ["find: ‘/etc/ceph’: No such file or directory"], "stdout": "", "stdout_lines": []} TASK [testnode : Check if {{ nic_to_configure }} is 10Gb] ********************** skipping: [gibba002.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:48.014 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:48 +0000 (0:00:00.062) 0:06:28.315 **** 2021-02-10T10:58:48.041 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU to 9000 if 10Gb] ************************************** 2021-02-10T10:58:48.042 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:48.079 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:48 +0000 (0:00:00.065) 0:06:28.380 **** 2021-02-10T10:58:48.107 INFO:teuthology.task.ansible.out: TASK [testnode : Write Ubuntu network config for {{ nic_to_configure }}] ******* 2021-02-10T10:58:48.108 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:48.140 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:48 +0000 (0:00:00.061) 0:06:28.442 **** 2021-02-10T10:58:48.167 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU on Ubuntu] ******************************************** 2021-02-10T10:58:48.169 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:48.203 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:48 +0000 (0:00:00.062) 0:06:28.504 **** 2021-02-10T10:58:48.236 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** skipping: [gibba002.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:48.272 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:48 +0000 (0:00:00.067) 0:06:28.572 **** 2021-02-10T10:58:48.297 INFO:teuthology.task.ansible.out: TASK [testnode : Write RHEL/CentOS network config for {{ nic_to_configure }}] *** skipping: [gibba002.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:48.332 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:48 +0000 (0:00:00.061) 0:06:28.634 **** 2021-02-10T10:58:48.360 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2021-02-10T10:58:48.361 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:48.394 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:48 +0000 (0:00:00.062) 0:06:28.696 **** 2021-02-10T10:58:48.474 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** ok: [gibba013.front.sepia.ceph.com] => (item=/var/log/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/log/ -name \"*ceph*\"", "delta": "0:00:00.027818", "end": "2021-02-10 10:58:48.434201", "failed_when_result": false, "item": "/var/log/", "rc": 0, "start": "2021-02-10 10:58:48.406383", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-10T10:58:48.510 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:48 +0000 (0:00:00.115) 0:06:28.812 **** 2021-02-10T10:58:48.537 INFO:teuthology.task.ansible.out: TASK [testnode : Delete any leftover Ceph artifacts from previous jobs] ******** 2021-02-10T10:58:48.572 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:48 +0000 (0:00:00.062) 0:06:28.874 **** 2021-02-10T10:58:49.033 INFO:teuthology.task.ansible.out: TASK [testnode : Touch /ceph-qa-ready] ***************************************** 2021-02-10T10:58:49.035 INFO:teuthology.task.ansible.out:changed: [gibba013.front.sepia.ceph.com] => {"changed": true, "dest": "/ceph-qa-ready", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_runtime_t:s0", "size": 0, "state": "file", "uid": 0} 2021-02-10T10:58:49.889 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2021-02-10T10:58:49.890 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item=/var/run/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/run/ -name \"*ceph*\"", "delta": "0:00:01.005997", "end": "2021-02-10 10:58:49.854595", "failed_when_result": false, "item": "/var/run/", "rc": 0, "start": "2021-02-10 10:58:48.848598", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-10T10:58:50.319 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item=/etc/systemd/system/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/systemd/system/ -name \"*ceph*\"", "delta": "0:00:00.002286", "end": "2021-02-10 10:58:50.288791", "failed_when_result": false, "item": "/etc/systemd/system/", "rc": 0, "start": "2021-02-10 10:58:50.286505", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-10T10:58:50.744 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item=/etc/ceph) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/ceph -name \"*ceph*\"", "delta": "0:00:00.002088", "end": "2021-02-10 10:58:50.718221", "failed_when_result": false, "item": "/etc/ceph", "msg": "non-zero return code", "rc": 1, "start": "2021-02-10 10:58:50.716133", "stderr": "find: ‘/etc/ceph’: No such file or directory", "stderr_lines": ["find: ‘/etc/ceph’: No such file or directory"], "stdout": "", "stdout_lines": []} 2021-02-10T10:58:51.189 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item=/var/log/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/log/ -name \"*ceph*\"", "delta": "0:00:00.025716", "end": "2021-02-10 10:58:51.167521", "failed_when_result": false, "item": "/var/log/", "rc": 0, "start": "2021-02-10 10:58:51.141805", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-10T10:58:51.222 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:51 +0000 (0:00:02.649) 0:06:31.524 **** 2021-02-10T10:58:51.249 INFO:teuthology.task.ansible.out: TASK [testnode : Delete any leftover Ceph artifacts from previous jobs] ******** 2021-02-10T10:58:51.281 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:51 +0000 (0:00:00.058) 0:06:31.583 **** 2021-02-10T10:58:51.725 INFO:teuthology.task.ansible.out: TASK [testnode : Touch /ceph-qa-ready] ***************************************** 2021-02-10T10:58:51.726 INFO:teuthology.task.ansible.out:changed: [gibba002.front.sepia.ceph.com] => {"changed": true, "dest": "/ceph-qa-ready", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_runtime_t:s0", "size": 0, "state": "file", "uid": 0} 2021-02-10T10:58:51.829 INFO:teuthology.task.ansible.out: PLAY [testnodes,senta,vossi,jenkins_slaves] ************************************ 2021-02-10T10:58:51.862 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:51 +0000 (0:00:00.580) 0:06:32.163 **** 2021-02-10T10:58:51.894 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:51 +0000 (0:00:00.032) 0:06:32.196 **** 2021-02-10T10:58:52.958 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [gibba002.front.sepia.ceph.com] 2021-02-10T10:58:53.034 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:53 +0000 (0:00:01.139) 0:06:33.335 **** 2021-02-10T10:58:53.061 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-02-10T10:58:53.063 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:53.095 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:53 +0000 (0:00:00.061) 0:06:33.397 **** 2021-02-10T10:58:53.123 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-02-10T10:58:53.125 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => {"ansible_facts": {"package_manager": "yum"}, "changed": false} 2021-02-10T10:58:53.158 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:53 +0000 (0:00:00.062) 0:06:33.459 **** 2021-02-10T10:58:53.206 INFO:teuthology.task.ansible.out: TASK [container-host : Including distro specific variables] ******************** 2021-02-10T10:58:53.208 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/yum_systems.yml) => {"ansible_facts": {"container_packages": ["podman", "podman-docker"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/yum_systems.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/yum_systems.yml"} 2021-02-10T10:58:53.242 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:53 +0000 (0:00:00.084) 0:06:33.544 **** 2021-02-10T10:58:54.004 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [gibba013.front.sepia.ceph.com] 2021-02-10T10:58:54.078 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:54 +0000 (0:00:00.836) 0:06:34.380 **** 2021-02-10T10:58:54.105 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-02-10T10:58:54.106 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:58:54.140 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:54 +0000 (0:00:00.061) 0:06:34.441 **** 2021-02-10T10:58:54.179 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-02-10T10:58:54.181 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => {"ansible_facts": {"package_manager": "yum"}, "changed": false} 2021-02-10T10:58:54.214 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:54 +0000 (0:00:00.074) 0:06:34.516 **** 2021-02-10T10:58:54.251 INFO:teuthology.task.ansible.out: TASK [container-host : Including distro specific variables] ******************** 2021-02-10T10:58:54.253 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/yum_systems.yml) => {"ansible_facts": {"container_packages": ["podman", "podman-docker"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/yum_systems.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/yum_systems.yml"} 2021-02-10T10:58:54.294 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:58:54 +0000 (0:00:00.080) 0:06:34.596 **** 2021-02-10T10:59:03.796 INFO:teuthology.task.ansible.out: TASK [container-host : Install container packages] ***************************** 2021-02-10T10:59:03.797 INFO:teuthology.task.ansible.out:changed: [gibba002.front.sepia.ceph.com] => {"changed": true, "changes": {"installed": [], "updated": [["podman", "1.6.4-27.el7_9.x86_64 from rhel-7-server-extras-rpms"], ["podman-docker", "1.6.4-27.el7_9.noarch from rhel-7-server-extras-rpms"]]}, "msg": "", "rc": 0, "results": ["Loaded plugins: fastestmirror, langpacks, priorities, product-id, search-\n : disabled-repos, subscription-manager\nLoading mirror speeds from cached hostfile\n * epel: download-cc-rdu01.fedoraproject.org\nResolving Dependencies\n--> Running transaction check\n---> Package podman.x86_64 0:1.6.4-26.el7_9 will be updated\n---> Package podman.x86_64 0:1.6.4-27.el7_9 will be an update\n---> Package podman-docker.noarch 0:1.6.4-26.el7_9 will be updated\n---> Package podman-docker.noarch 0:1.6.4-27.el7_9 will be an update\n--> Finished Dependency Resolution\n\nDependencies Resolved\n\n================================================================================\n Package Arch Version Repository Size\n================================================================================\nUpdating:\n podman x86_64 1.6.4-27.el7_9 rhel-7-server-extras-rpms 13 M\n podman-docker noarch 1.6.4-27.el7_9 rhel-7-server-extras-rpms 31 k\n\nTransaction Summary\n================================================================================\nUpgrade 2 Packages\n\nTotal download size: 13 M\nDownloading packages:\nDelta RPMs disabled because /usr/bin/applydeltarpm not installed.\n--------------------------------------------------------------------------------\nTotal 80 MB/s | 13 MB 00:00 \nRunning transaction check\nRunning transaction test\nTransaction test succeeded\nRunning transaction\n Updating : podman-1.6.4-27.el7_9.x86_64 1/4 \n Updating : podman-docker-1.6.4-27.el7_9.noarch 2/4 \n Cleanup : podman-docke 2021-02-10T10:59:03.799 INFO:teuthology.task.ansible.out:r-1.6.4-26.el7_9.noarch 3/4 \n Cleanup : podman-1.6.4-26.el7_9.x86_64 4/4 \nLoaded plugins: fastestmirror, langpacks, priorities, product-id, subscription-\n : manager\n Verifying : podman-docker-1.6.4-27.el7_9.noarch 1/4 \n Verifying : podman-1.6.4-27.el7_9.x86_64 2/4 \n Verifying : podman-docker-1.6.4-26.el7_9.noarch 3/4 \n Verifying : podman-1.6.4-26.el7_9.x86_64 4/4 \n\nUpdated:\n podman.x86_64 0:1.6.4-27.el7_9 podman-docker.noarch 0:1.6.4-27.el7_9 \n\nComplete!\n"]} 2021-02-10T10:59:03.852 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:59:03 +0000 (0:00:09.557) 0:06:44.154 **** 2021-02-10T10:59:03.889 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-02-10T10:59:03.891 INFO:teuthology.task.ansible.out:ok: [gibba002.front.sepia.ceph.com] => {"ansible_facts": {"container_service_conf": "/etc/containers/registries.conf"}, "changed": false} 2021-02-10T10:59:03.924 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:59:03 +0000 (0:00:00.071) 0:06:44.226 **** 2021-02-10T10:59:03.962 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-02-10T10:59:03.964 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:59:03.996 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:59:03 +0000 (0:00:00.071) 0:06:44.298 **** 2021-02-10T10:59:04.493 INFO:teuthology.task.ansible.out: TASK [container-host : Create container_mirror_cert_paths] ********************* 2021-02-10T10:59:04.494 INFO:teuthology.task.ansible.out:changed: [gibba002.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "secontext": "unconfined_u:object_r:cert_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-02-10T10:59:04.786 INFO:teuthology.task.ansible.out: TASK [container-host : Install container packages] ***************************** 2021-02-10T10:59:04.787 INFO:teuthology.task.ansible.out:changed: [gibba013.front.sepia.ceph.com] => {"changed": true, "changes": {"installed": [], "updated": [["podman", "1.6.4-27.el7_9.x86_64 from rhel-7-server-extras-rpms"], ["podman-docker", "1.6.4-27.el7_9.noarch from rhel-7-server-extras-rpms"]]}, "msg": "", "rc": 0, "results": ["Loaded plugins: fastestmirror, langpacks, priorities, product-id, search-\n : disabled-repos, subscription-manager\nLoading mirror speeds from cached hostfile\n * epel: download-cc-rdu01.fedoraproject.org\nResolving Dependencies\n--> Running transaction check\n---> Package podman.x86_64 0:1.6.4-26.el7_9 will be updated\n---> Package podman.x86_64 0:1.6.4-27.el7_9 will be an update\n---> Package podman-docker.noarch 0:1.6.4-26.el7_9 will be updated\n---> Package podman-docker.noarch 0:1.6.4-27.el7_9 will be an update\n--> Finished Dependency Resolution\n\nDependencies Resolved\n\n================================================================================\n Package Arch Version Repository Size\n================================================================================\nUpdating:\n podman x86_64 1.6.4-27.el7_9 rhel-7-server-extras-rpms 13 M\n podman-docker noarch 1.6.4-27.el7_9 rhel-7-server-extras-rpms 31 k\n\nTransaction Summary\n================================================================================\nUpgrade 2 Packages\n\nTotal download size: 13 M\nDownloading packages:\nDelta RPMs disabled because /usr/bin/applydeltarpm not installed.\n--------------------------------------------------------------------------------\nTotal 73 MB/s | 13 MB 00:00 \nRunning transaction check\nRunning transaction test\nTransaction test succeeded\nRunning transaction\n Updating : podman-1.6.4-27.el7_9.x86_64 1/4 \n Updating : podman-docker-1.6.4-27.el7_9.noarch 2/4 \n Cleanup : podman-docke 2021-02-10T10:59:04.789 INFO:teuthology.task.ansible.out:r-1.6.4-26.el7_9.noarch 3/4 \n Cleanup : podman-1.6.4-26.el7_9.x86_64 4/4 \nLoaded plugins: fastestmirror, langpacks, priorities, product-id, subscription-\n : manager\n Verifying : podman-docker-1.6.4-27.el7_9.noarch 1/4 \n Verifying : podman-1.6.4-27.el7_9.x86_64 2/4 \n Verifying : podman-docker-1.6.4-26.el7_9.noarch 3/4 \n Verifying : podman-1.6.4-26.el7_9.x86_64 4/4 \n\nUpdated:\n podman.x86_64 0:1.6.4-27.el7_9 podman-docker.noarch 0:1.6.4-27.el7_9 \n\nComplete!\n"]} 2021-02-10T10:59:04.822 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:59:04 +0000 (0:00:00.826) 0:06:45.124 **** 2021-02-10T10:59:04.850 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-02-10T10:59:04.851 INFO:teuthology.task.ansible.out:ok: [gibba013.front.sepia.ceph.com] => {"ansible_facts": {"container_service_conf": "/etc/containers/registries.conf"}, "changed": false} 2021-02-10T10:59:04.885 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:59:04 +0000 (0:00:00.062) 0:06:45.186 **** 2021-02-10T10:59:04.911 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-02-10T10:59:04.913 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:59:04.947 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:59:04 +0000 (0:00:00.062) 0:06:45.248 **** 2021-02-10T10:59:04.953 INFO:teuthology.task.ansible.out: TASK [container-host : Create container_mirror_cert_paths] ********************* 2021-02-10T10:59:04.954 INFO:teuthology.task.ansible.out:changed: [gibba002.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-02-10T10:59:04.993 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:59:04 +0000 (0:00:00.045) 0:06:45.294 **** 2021-02-10T10:59:05.416 INFO:teuthology.task.ansible.out:changed: [gibba013.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "secontext": "unconfined_u:object_r:cert_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-02-10T10:59:05.852 INFO:teuthology.task.ansible.out:changed: [gibba013.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-02-10T10:59:05.853 INFO:teuthology.task.ansible.out: 2021-02-10T10:59:05.890 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:59:05 +0000 (0:00:00.897) 0:06:46.192 **** 2021-02-10T10:59:07.618 INFO:teuthology.task.ansible.out: TASK [container-host : Copy docker-mirror.front.sepia.ceph.com:5000 self-signed cert] *** 2021-02-10T10:59:07.619 INFO:teuthology.task.ansible.out:changed: [gibba002.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "checksum": "bcdfbb1de91ef82e0e4ed8af725c9ad52593eb79", "dest": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "md5sum": "a3fa0991425692932fcf703c1a9a483e", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:cert_t:s0", "size": 2334, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1612954745.0329661-127815165155282/source", "state": "file", "uid": 0} 2021-02-10T10:59:08.504 INFO:teuthology.task.ansible.out:changed: [gibba013.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "checksum": "bcdfbb1de91ef82e0e4ed8af725c9ad52593eb79", "dest": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "md5sum": "a3fa0991425692932fcf703c1a9a483e", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:cert_t:s0", "size": 2334, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1612954745.9312165-150311216645084/source", "state": "file", "uid": 0} 2021-02-10T10:59:10.186 INFO:teuthology.task.ansible.out:changed: [gibba002.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "checksum": "bcdfbb1de91ef82e0e4ed8af725c9ad52593eb79", "dest": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "md5sum": "a3fa0991425692932fcf703c1a9a483e", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:etc_t:s0", "size": 2334, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1612954747.6244395-265733318229027/source", "state": "file", "uid": 0} 2021-02-10T10:59:10.225 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:59:10 +0000 (0:00:04.335) 0:06:50.527 **** 2021-02-10T10:59:11.074 INFO:teuthology.task.ansible.out:changed: [gibba013.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "checksum": "bcdfbb1de91ef82e0e4ed8af725c9ad52593eb79", "dest": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "md5sum": "a3fa0991425692932fcf703c1a9a483e", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:etc_t:s0", "size": 2334, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1612954748.512856-22094045946017/source", "state": "file", "uid": 0} 2021-02-10T10:59:11.111 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:59:11 +0000 (0:00:00.885) 0:06:51.413 **** 2021-02-10T10:59:12.764 INFO:teuthology.task.ansible.out: TASK [container-host : Install registries-conf-ctl] **************************** 2021-02-10T10:59:12.765 INFO:teuthology.task.ansible.out:changed: [gibba002.front.sepia.ceph.com] => {"changed": true, "cmd": ["/bin/pip2", "install", "-U", "git+https://github.com/sebastian-philipp/registries-conf-ctl"], "name": ["git+https://github.com/sebastian-philipp/registries-conf-ctl"], "requirements": null, "state": "latest", "stderr": "You are using pip version 8.1.2, however version 21.0.1 is available.\nYou should consider upgrading via the 'pip install --upgrade pip' command.\n", "stderr_lines": ["You are using pip version 8.1.2, however version 21.0.1 is available.", "You should consider upgrading via the 'pip install --upgrade pip' command."], "stdout": "Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl\n Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-HbgT8r-build\nCollecting toml (from registries-conf-ctl==1.0)\n Downloading https://files.pythonhosted.org/packages/44/6f/7120676b6d73228c96e17f1f794d8ab046fc910d781c8d151120c3f1569e/toml-0.10.2-py2.py3-none-any.whl\nCollecting docopt (from registries-conf-ctl==1.0)\n Downloading https://files.pythonhosted.org/packages/a2/55/8f8cab2afd404cf578136ef2cc5dfb50baa1761b68c9da1fb1e4eed343c9/docopt-0.6.2.tar.gz\nCollecting typing (from registries-conf-ctl==1.0)\n Downloading https://files.pythonhosted.org/packages/3b/c0/e44213fcb799eac02881e2485724ba5b0914600bc9df6ed922e364fdc059/typing-3.7.4.3-py2-none-any.whl\nInstalling collected packages: toml, docopt, typing, registries-conf-ctl\n Running setup.py install for docopt: started\n Running setup.py install for docopt: finished with status 'done'\n Running setup.py install for registries-conf-ctl: started\n Running setup.py install for registries-conf-ctl: finished with status 'done'\nSuccessfully installed docopt-0.6.2 registries-conf-ctl-1.0 toml-0.10.2 typing-3.7.4.3\n", "stdout_lines": ["Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl", " Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-HbgT8r-build", "Co 2021-02-10T10:59:12.767 INFO:teuthology.task.ansible.out:llecting toml (from registries-conf-ctl==1.0)", " Downloading https://files.pythonhosted.org/packages/44/6f/7120676b6d73228c96e17f1f794d8ab046fc910d781c8d151120c3f1569e/toml-0.10.2-py2.py3-none-any.whl", "Collecting docopt (from registries-conf-ctl==1.0)", " Downloading https://files.pythonhosted.org/packages/a2/55/8f8cab2afd404cf578136ef2cc5dfb50baa1761b68c9da1fb1e4eed343c9/docopt-0.6.2.tar.gz", "Collecting typing (from registries-conf-ctl==1.0)", " Downloading https://files.pythonhosted.org/packages/3b/c0/e44213fcb799eac02881e2485724ba5b0914600bc9df6ed922e364fdc059/typing-3.7.4.3-py2-none-any.whl", "Installing collected packages: toml, docopt, typing, registries-conf-ctl", " Running setup.py install for docopt: started", " Running setup.py install for docopt: finished with status 'done'", " Running setup.py install for registries-conf-ctl: started", " Running setup.py install for registries-conf-ctl: finished with status 'done'", "Successfully installed docopt-0.6.2 registries-conf-ctl-1.0 toml-0.10.2 typing-3.7.4.3"], "version": null, "virtualenv": null} 2021-02-10T10:59:12.821 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:59:12 +0000 (0:00:01.709) 0:06:53.123 **** 2021-02-10T10:59:12.859 INFO:teuthology.task.ansible.out: TASK [container-host : Check for docker's daemon.json] ************************* 2021-02-10T10:59:12.861 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:59:12.895 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:59:12 +0000 (0:00:00.074) 0:06:53.197 **** 2021-02-10T10:59:12.922 INFO:teuthology.task.ansible.out: TASK [container-host : Create /etc/containers/registries.conf if necessary] **** 2021-02-10T10:59:12.923 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:59:12.960 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:59:12 +0000 (0:00:00.065) 0:06:53.262 **** 2021-02-10T10:59:13.486 INFO:teuthology.task.ansible.out: TASK [container-host : Add local docker.io registry mirror] ******************** 2021-02-10T10:59:13.487 INFO:teuthology.task.ansible.out:changed: [gibba002.front.sepia.ceph.com] => {"changed": true, "cmd": ["registries-conf-ctl", "add-mirror", "docker.io", "docker-mirror.front.sepia.ceph.com:5000"], "delta": "0:00:00.067912", "end": "2021-02-10 10:59:13.464029", "rc": 0, "start": "2021-02-10 10:59:13.396117", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-10T10:59:13.521 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:59:13 +0000 (0:00:00.560) 0:06:53.823 **** 2021-02-10T10:59:13.548 INFO:teuthology.task.ansible.out: TASK [container-host : Restart docker service] ********************************* 2021-02-10T10:59:13.550 INFO:teuthology.task.ansible.out:skipping: [gibba002.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:59:13.733 INFO:teuthology.task.ansible.out: TASK [container-host : Install registries-conf-ctl] **************************** 2021-02-10T10:59:13.735 INFO:teuthology.task.ansible.out:changed: [gibba013.front.sepia.ceph.com] => {"changed": true, "cmd": ["/bin/pip2", "install", "-U", "git+https://github.com/sebastian-philipp/registries-conf-ctl"], "name": ["git+https://github.com/sebastian-philipp/registries-conf-ctl"], "requirements": null, "state": "latest", "stderr": "You are using pip version 8.1.2, however version 21.0.1 is available.\nYou should consider upgrading via the 'pip install --upgrade pip' command.\n", "stderr_lines": ["You are using pip version 8.1.2, however version 21.0.1 is available.", "You should consider upgrading via the 'pip install --upgrade pip' command."], "stdout": "Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl\n Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-KPGsNu-build\nCollecting toml (from registries-conf-ctl==1.0)\n Downloading https://files.pythonhosted.org/packages/44/6f/7120676b6d73228c96e17f1f794d8ab046fc910d781c8d151120c3f1569e/toml-0.10.2-py2.py3-none-any.whl\nCollecting docopt (from registries-conf-ctl==1.0)\n Downloading https://files.pythonhosted.org/packages/a2/55/8f8cab2afd404cf578136ef2cc5dfb50baa1761b68c9da1fb1e4eed343c9/docopt-0.6.2.tar.gz\nCollecting typing (from registries-conf-ctl==1.0)\n Downloading https://files.pythonhosted.org/packages/3b/c0/e44213fcb799eac02881e2485724ba5b0914600bc9df6ed922e364fdc059/typing-3.7.4.3-py2-none-any.whl\nInstalling collected packages: toml, docopt, typing, registries-conf-ctl\n Running setup.py install for docopt: started\n Running setup.py install for docopt: finished with status 'done'\n Running setup.py install for registries-conf-ctl: started\n Running setup.py install for registries-conf-ctl: finished with status 'done'\nSuccessfully installed docopt-0.6.2 registries-conf-ctl-1.0 toml-0.10.2 typing-3.7.4.3\n", "stdout_lines": ["Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl", " Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-KPGsNu-build", "Co 2021-02-10T10:59:13.736 INFO:teuthology.task.ansible.out:llecting toml (from registries-conf-ctl==1.0)", " Downloading https://files.pythonhosted.org/packages/44/6f/7120676b6d73228c96e17f1f794d8ab046fc910d781c8d151120c3f1569e/toml-0.10.2-py2.py3-none-any.whl", "Collecting docopt (from registries-conf-ctl==1.0)", " Downloading https://files.pythonhosted.org/packages/a2/55/8f8cab2afd404cf578136ef2cc5dfb50baa1761b68c9da1fb1e4eed343c9/docopt-0.6.2.tar.gz", "Collecting typing (from registries-conf-ctl==1.0)", " Downloading https://files.pythonhosted.org/packages/3b/c0/e44213fcb799eac02881e2485724ba5b0914600bc9df6ed922e364fdc059/typing-3.7.4.3-py2-none-any.whl", "Installing collected packages: toml, docopt, typing, registries-conf-ctl", " Running setup.py install for docopt: started", " Running setup.py install for docopt: finished with status 'done'", " Running setup.py install for registries-conf-ctl: started", " Running setup.py install for registries-conf-ctl: finished with status 'done'", "Successfully installed docopt-0.6.2 registries-conf-ctl-1.0 toml-0.10.2 typing-3.7.4.3"], "version": null, "virtualenv": null} 2021-02-10T10:59:13.767 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:59:13 +0000 (0:00:00.245) 0:06:54.069 **** 2021-02-10T10:59:13.808 INFO:teuthology.task.ansible.out: TASK [container-host : Check for docker's daemon.json] ************************* 2021-02-10T10:59:13.811 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:59:13.847 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:59:13 +0000 (0:00:00.078) 0:06:54.148 **** 2021-02-10T10:59:13.886 INFO:teuthology.task.ansible.out: TASK [container-host : Create /etc/containers/registries.conf if necessary] **** 2021-02-10T10:59:13.887 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:59:13.926 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:59:13 +0000 (0:00:00.079) 0:06:54.227 **** 2021-02-10T10:59:14.479 INFO:teuthology.task.ansible.out: TASK [container-host : Add local docker.io registry mirror] ******************** 2021-02-10T10:59:14.480 INFO:teuthology.task.ansible.out:changed: [gibba013.front.sepia.ceph.com] => {"changed": true, "cmd": ["registries-conf-ctl", "add-mirror", "docker.io", "docker-mirror.front.sepia.ceph.com:5000"], "delta": "0:00:00.072840", "end": "2021-02-10 10:59:14.461691", "rc": 0, "start": "2021-02-10 10:59:14.388851", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-10T10:59:14.526 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 10:59:14 +0000 (0:00:00.599) 0:06:54.827 **** 2021-02-10T10:59:14.563 INFO:teuthology.task.ansible.out: TASK [container-host : Restart docker service] ********************************* 2021-02-10T10:59:14.565 INFO:teuthology.task.ansible.out:skipping: [gibba013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T10:59:14.681 INFO:teuthology.task.ansible.out: PLAY [cobbler] ***************************************************************** skipping: no hosts matched 2021-02-10T10:59:14.686 INFO:teuthology.task.ansible.out: PLAY [paddles] ***************************************************************** skipping: no hosts matched 2021-02-10T10:59:14.690 INFO:teuthology.task.ansible.out: PLAY [pulpito] ***************************************************************** skipping: no hosts matched 2021-02-10T10:59:14.691 INFO:teuthology.task.ansible.out: PLAY RECAP ********************************************************************* gibba002.front.sepia.ceph.com : ok=141 changed=45 unreachable=0 failed=0 skipped=169 rescued=0 ignored=4 gibba013.front.sepia.ceph.com : ok=141 changed=45 unreachable=0 failed=0 skipped=169 rescued=0 ignored=4 Wednesday 10 February 2021 10:59:14 +0000 (0:00:00.164) 0:06:54.992 **** =============================================================================== users : Create all admin users with sudo access. ----------------------- 63.22s users : Update authorized_keys using the keys repo --------------------- 60.98s common : Make sure smartmontools is installed -------------------------- 27.58s users : Remove revoked users ------------------------------------------- 22.26s testnode : Install packages -------------------------------------------- 16.07s container-host : Install container packages ----------------------------- 9.56s testnode : Ensure ceph packages are not present. ------------------------ 9.53s testnode : Zap all non-root disks --------------------------------------- 9.12s common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/. --- 6.42s common : nrpe - Load SELinux policy package ----------------------------- 5.90s common : nrpe - Remove SELinux policy package --------------------------- 5.89s common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/. --- 5.78s common : Install nrpe package and dependencies (RHEL/CentOS) ------------ 5.02s common : Register with subscription-manager. ---------------------------- 4.81s testnode : Install epel packages ---------------------------------------- 4.34s container-host : Copy docker-mirror.front.sepia.ceph.com:5000 self-signed cert --- 4.34s testnode : Ensure ceph dependency packages are not present. ------------- 4.32s common : nrpe - Ensure SELinux policy is up to date --------------------- 4.15s testnode 2021-02-10T10:59:14.692 INFO:teuthology.task.ansible.out: : Configure common additional repos in /etc/yum.repos.d/ ------- 3.86s Gathering Facts --------------------------------------------------------- 3.77s 2021-02-10T10:59:15.037 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@gibba002.front.sepia.ceph.com'), Remote(name='ubuntu@gibba013.front.sepia.ceph.com')] 2021-02-10T10:59:15.038 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-02-10T10:59:15.039 DEBUG:teuthology.orchestra.connection:{'hostname': 'gibba002.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-02-10T10:59:15.157 DEBUG:teuthology.orchestra.run.gibba002:> true 2021-02-10T10:59:15.648 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-02-10T10:59:15.649 DEBUG:teuthology.orchestra.connection:{'hostname': 'gibba013.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-02-10T10:59:15.714 DEBUG:teuthology.orchestra.run.gibba013:> true 2021-02-10T10:59:16.205 INFO:teuthology.run_tasks:Running task clock... 2021-02-10T10:59:16.219 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2021-02-10T10:59:16.219 INFO:teuthology.orchestra.run:Running command with timeout 360 2021-02-10T10:59:16.220 DEBUG:teuthology.orchestra.run.gibba002:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2021-02-10T10:59:16.570 INFO:teuthology.orchestra.run.gibba002.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2021-02-10T10:59:23.606 INFO:teuthology.orchestra.run.gibba002.stdout:ntpd: time slew +0.000631s 2021-02-10T10:59:23.617 INFO:teuthology.orchestra.run.gibba002.stderr:Failed to start ntp.service: Unit not found. 2021-02-10T10:59:23.657 INFO:teuthology.orchestra.run.gibba002.stdout: remote refid st t when poll reach delay offset jitter 2021-02-10T10:59:23.657 INFO:teuthology.orchestra.run.gibba002.stdout:============================================================================== 2021-02-10T10:59:23.657 INFO:teuthology.orchestra.run.gibba002.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2021-02-10T10:59:23.657 INFO:teuthology.orchestra.run.gibba002.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2021-02-10T10:59:23.657 INFO:teuthology.orchestra.run.gibba002.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2021-02-10T10:59:23.657 INFO:teuthology.orchestra.run.gibba002.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2021-02-10T10:59:23.657 INFO:teuthology.orchestra.run:Running command with timeout 360 2021-02-10T10:59:23.658 DEBUG:teuthology.orchestra.run.gibba013:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2021-02-10T10:59:24.010 INFO:teuthology.orchestra.run.gibba013.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2021-02-10T10:59:31.047 INFO:teuthology.orchestra.run.gibba013.stdout:ntpd: time slew +0.000405s 2021-02-10T10:59:31.059 INFO:teuthology.orchestra.run.gibba013.stderr:Failed to start ntp.service: Unit not found. 2021-02-10T10:59:31.098 INFO:teuthology.orchestra.run.gibba013.stdout: remote refid st t when poll reach delay offset jitter 2021-02-10T10:59:31.099 INFO:teuthology.orchestra.run.gibba013.stdout:============================================================================== 2021-02-10T10:59:31.099 INFO:teuthology.orchestra.run.gibba013.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2021-02-10T10:59:31.099 INFO:teuthology.orchestra.run.gibba013.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2021-02-10T10:59:31.099 INFO:teuthology.orchestra.run.gibba013.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2021-02-10T10:59:31.099 INFO:teuthology.orchestra.run.gibba013.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2021-02-10T10:59:31.099 INFO:teuthology.run_tasks:Running task install... 2021-02-10T10:59:31.110 DEBUG:teuthology.task.install:project ceph 2021-02-10T10:59:31.110 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'sha1': '962fac13b1a781b8c7d25c1eed50d93813b8f5f0'}} 2021-02-10T10:59:31.110 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell'], 'rpm': ['python3-cephfs']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel']}, 'sha1': '962fac13b1a781b8c7d25c1eed50d93813b8f5f0'} 2021-02-10T10:59:31.111 INFO:teuthology.task.install:Using flavor: basic 2021-02-10T10:59:31.120 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python-ceph', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-cloud', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-ssh', 'ceph-fuse', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python-ceph', 'rbd-fuse']} 2021-02-10T10:59:31.120 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell'], 'rpm': ['python3-cephfs']} 2021-02-10T10:59:31.120 DEBUG:teuthology.orchestra.run.gibba002:> sudo lsb_release -is 2021-02-10T10:59:31.480 INFO:teuthology.orchestra.run.gibba002.stdout:RedHatEnterpriseServer 2021-02-10T10:59:31.481 DEBUG:teuthology.misc:System to be installed: RedHatEnterpriseServer 2021-02-10T10:59:31.481 DEBUG:teuthology.orchestra.run.gibba013:> sudo lsb_release -is 2021-02-10T10:59:31.482 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell'], 'rpm': ['python3-cephfs']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel']}, 'extras': None, 'flavor': 'basic', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '962fac13b1a781b8c7d25c1eed50d93813b8f5f0', 'tag': None, 'wait_for_package': False} 2021-02-10T10:59:31.482 DEBUG:teuthology.orchestra.run.gibba002:> sudo lsb_release -is 2021-02-10T10:59:31.835 INFO:teuthology.orchestra.run.gibba002.stdout:RedHatEnterpriseServer 2021-02-10T10:59:31.836 DEBUG:teuthology.misc:System to be installed: RedHatEnterpriseServer 2021-02-10T10:59:31.836 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F7%2Fx86_64&sha1=962fac13b1a781b8c7d25c1eed50d93813b8f5f0 2021-02-10T10:59:31.843 INFO:teuthology.orchestra.run.gibba013.stdout:RedHatEnterpriseServer 2021-02-10T10:59:31.843 DEBUG:teuthology.misc:System to be installed: RedHatEnterpriseServer 2021-02-10T10:59:31.843 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell'], 'rpm': ['python3-cephfs']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel']}, 'extras': None, 'flavor': 'basic', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '962fac13b1a781b8c7d25c1eed50d93813b8f5f0', 'tag': None, 'wait_for_package': False} 2021-02-10T10:59:31.843 DEBUG:teuthology.orchestra.run.gibba013:> sudo lsb_release -is 2021-02-10T10:59:32.009 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-khiremat-mgr-authid-bp-7-nautilus/962fac13b1a781b8c7d25c1eed50d93813b8f5f0/centos/7/flavors/default/ 2021-02-10T10:59:32.010 INFO:teuthology.task.install.rpm:Package version is 14.2.16-280.g962fac13 2021-02-10T10:59:32.164 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-khiremat-mgr-authid-bp-7-nautilus/962fac13b1a781b8c7d25c1eed50d93813b8f5f0/centos/7/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-khiremat-mgr-authid-bp-7-nautilus/962fac13b1a781b8c7d25c1eed50d93813b8f5f0/centos/7/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-khiremat-mgr-authid-bp-7-nautilus/962fac13b1a781b8c7d25c1eed50d93813b8f5f0/centos/7/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2021-02-10T10:59:32.164 DEBUG:teuthology.orchestra.run.gibba002:> set -ex 2021-02-10T10:59:32.164 DEBUG:teuthology.orchestra.run.gibba002:> sudo dd of=/etc/yum.repos.d/ceph.repo 2021-02-10T10:59:32.195 INFO:teuthology.orchestra.run.gibba013.stdout:RedHatEnterpriseServer 2021-02-10T10:59:32.196 DEBUG:teuthology.misc:System to be installed: RedHatEnterpriseServer 2021-02-10T10:59:32.196 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F7%2Fx86_64&sha1=962fac13b1a781b8c7d25c1eed50d93813b8f5f0 2021-02-10T10:59:32.359 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-khiremat-mgr-authid-bp-7-nautilus/962fac13b1a781b8c7d25c1eed50d93813b8f5f0/centos/7/flavors/default/ 2021-02-10T10:59:32.359 INFO:teuthology.task.install.rpm:Package version is 14.2.16-280.g962fac13 2021-02-10T10:59:32.485 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-khiremat-mgr-authid-bp-7-nautilus/962fac13b1a781b8c7d25c1eed50d93813b8f5f0/centos/7/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-khiremat-mgr-authid-bp-7-nautilus/962fac13b1a781b8c7d25c1eed50d93813b8f5f0/centos/7/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-khiremat-mgr-authid-bp-7-nautilus/962fac13b1a781b8c7d25c1eed50d93813b8f5f0/centos/7/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2021-02-10T10:59:32.486 DEBUG:teuthology.orchestra.run.gibba013:> set -ex 2021-02-10T10:59:32.486 DEBUG:teuthology.orchestra.run.gibba013:> sudo dd of=/etc/yum.repos.d/ceph.repo 2021-02-10T10:59:32.506 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-cloud, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-ssh, ceph-fuse, libcephfs2, libcephfs-devel, librados2, librbd1, python-ceph, rbd-fuse, python3-cephfs, bison, flex, elfutils-libelf-devel, openssl-devel, bison, flex, elfutils-libelf-devel, openssl-devel, bison, flex, elfutils-libelf-devel, openssl-devel on remote rpm x86_64 2021-02-10T10:59:32.507 DEBUG:teuthology.orchestra.run.gibba002:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/962fac13b1a781b8c7d25c1eed50d93813b8f5f0/;g' /etc/yum.repos.d/ceph.repo ; fi 2021-02-10T10:59:32.830 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-cloud, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-ssh, ceph-fuse, libcephfs2, libcephfs-devel, librados2, librbd1, python-ceph, rbd-fuse, python3-cephfs, bison, flex, elfutils-libelf-devel, openssl-devel, bison, flex, elfutils-libelf-devel, openssl-devel, bison, flex, elfutils-libelf-devel, openssl-devel, bison, flex, elfutils-libelf-devel, openssl-devel on remote rpm x86_64 2021-02-10T10:59:32.831 DEBUG:teuthology.orchestra.run.gibba013:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/962fac13b1a781b8c7d25c1eed50d93813b8f5f0/;g' /etc/yum.repos.d/ceph.repo ; fi 2021-02-10T10:59:32.850 DEBUG:teuthology.orchestra.run.gibba002:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2021-02-10T10:59:33.174 DEBUG:teuthology.orchestra.run.gibba013:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2021-02-10T10:59:33.201 DEBUG:teuthology.orchestra.run.gibba002:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2021-02-10T10:59:33.527 DEBUG:teuthology.orchestra.run.gibba013:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2021-02-10T10:59:33.543 INFO:teuthology.orchestra.run.gibba002.stdout:check_obsoletes = 1 2021-02-10T10:59:33.545 DEBUG:teuthology.orchestra.run.gibba002:> sudo yum clean all 2021-02-10T10:59:33.870 INFO:teuthology.orchestra.run.gibba013.stdout:check_obsoletes = 1 2021-02-10T10:59:33.871 DEBUG:teuthology.orchestra.run.gibba013:> sudo yum clean all 2021-02-10T10:59:33.994 INFO:teuthology.orchestra.run.gibba002.stdout:Loaded plugins: fastestmirror, langpacks, priorities, product-id, search- 2021-02-10T10:59:33.994 INFO:teuthology.orchestra.run.gibba002.stdout: : disabled-repos, subscription-manager 2021-02-10T10:59:34.255 INFO:teuthology.orchestra.run.gibba002.stdout:Cleaning repos: ceph ceph-noarch ceph-source epel lab-extras rhel-7-fcgi-ceph 2021-02-10T10:59:34.256 INFO:teuthology.orchestra.run.gibba002.stdout: : rhel-7-server-extras-rpms rhel-7-server-optional-rpms 2021-02-10T10:59:34.256 INFO:teuthology.orchestra.run.gibba002.stdout: : rhel-7-server-rpms rhel-ha-for-rhel-7-server-rpms 2021-02-10T10:59:34.319 INFO:teuthology.orchestra.run.gibba013.stdout:Loaded plugins: fastestmirror, langpacks, priorities, product-id, search- 2021-02-10T10:59:34.319 INFO:teuthology.orchestra.run.gibba013.stdout: : disabled-repos, subscription-manager 2021-02-10T10:59:34.412 INFO:teuthology.orchestra.run.gibba002.stdout:Cleaning up list of fastest mirrors 2021-02-10T10:59:34.425 DEBUG:teuthology.orchestra.run.gibba002:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-cloud ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-ssh ceph-fuse libcephfs2 libcephfs-devel librados2 librbd1 python-ceph rbd-fuse python3-cephfs bison flex elfutils-libelf-devel openssl-devel bison flex elfutils-libelf-devel openssl-devel bison flex elfutils-libelf-devel openssl-devel bison flex elfutils-libelf-devel openssl-devel 2021-02-10T10:59:34.591 INFO:teuthology.orchestra.run.gibba013.stdout:Cleaning repos: ceph ceph-noarch ceph-source epel lab-extras rhel-7-fcgi-ceph 2021-02-10T10:59:34.592 INFO:teuthology.orchestra.run.gibba013.stdout: : rhel-7-server-extras-rpms rhel-7-server-optional-rpms 2021-02-10T10:59:34.592 INFO:teuthology.orchestra.run.gibba013.stdout: : rhel-7-server-rpms rhel-ha-for-rhel-7-server-rpms 2021-02-10T10:59:34.738 INFO:teuthology.orchestra.run.gibba013.stdout:Cleaning up list of fastest mirrors 2021-02-10T10:59:34.825 DEBUG:teuthology.orchestra.run.gibba013:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-cloud ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-ssh ceph-fuse libcephfs2 libcephfs-devel librados2 librbd1 python-ceph rbd-fuse python3-cephfs bison flex elfutils-libelf-devel openssl-devel bison flex elfutils-libelf-devel openssl-devel bison flex elfutils-libelf-devel openssl-devel bison flex elfutils-libelf-devel openssl-devel 2021-02-10T10:59:34.872 INFO:teuthology.orchestra.run.gibba002.stdout:Loaded plugins: fastestmirror, langpacks, priorities, product-id, search- 2021-02-10T10:59:34.872 INFO:teuthology.orchestra.run.gibba002.stdout: : disabled-repos, subscription-manager 2021-02-10T10:59:35.179 INFO:teuthology.orchestra.run.gibba002.stdout:Determining fastest mirrors 2021-02-10T10:59:35.252 INFO:teuthology.orchestra.run.gibba002.stdout: * epel: download-cc-rdu01.fedoraproject.org 2021-02-10T10:59:35.278 INFO:teuthology.orchestra.run.gibba013.stdout:Loaded plugins: fastestmirror, langpacks, priorities, product-id, search- 2021-02-10T10:59:35.278 INFO:teuthology.orchestra.run.gibba013.stdout: : disabled-repos, subscription-manager 2021-02-10T10:59:35.571 INFO:teuthology.orchestra.run.gibba013.stdout:Determining fastest mirrors 2021-02-10T10:59:35.643 INFO:teuthology.orchestra.run.gibba013.stdout: * epel: download-cc-rdu01.fedoraproject.org 2021-02-10T11:00:00.955 INFO:teuthology.orchestra.run.gibba002.stdout:29 packages excluded due to repository priority protections 2021-02-10T11:00:06.412 INFO:teuthology.orchestra.run.gibba013.stdout:29 packages excluded due to repository priority protections 2021-02-10T11:00:17.853 INFO:teuthology.orchestra.run.gibba002.stdout:Resolving Dependencies 2021-02-10T11:00:17.853 INFO:teuthology.orchestra.run.gibba002.stdout:--> Running transaction check 2021-02-10T11:00:17.853 INFO:teuthology.orchestra.run.gibba002.stdout:---> Package bison.x86_64 0:3.0.4-2.el7 will be installed 2021-02-10T11:00:17.858 INFO:teuthology.orchestra.run.gibba002.stdout:---> Package ceph.x86_64 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:00:17.860 INFO:teuthology.orchestra.run.gibba002.stdout:--> Processing Dependency: ceph-mon = 2:14.2.16-280.g962fac13.el7 for package: 2:ceph-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:00:17.861 INFO:teuthology.orchestra.run.gibba002.stdout:--> Processing Dependency: ceph-osd = 2:14.2.16-280.g962fac13.el7 for package: 2:ceph-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:00:17.862 INFO:teuthology.orchestra.run.gibba002.stdout:--> Processing Dependency: ceph-mds = 2:14.2.16-280.g962fac13.el7 for package: 2:ceph-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:00:17.863 INFO:teuthology.orchestra.run.gibba002.stdout:---> Package ceph-fuse.x86_64 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:00:17.870 INFO:teuthology.orchestra.run.gibba002.stdout:---> Package ceph-mgr.x86_64 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:00:17.872 INFO:teuthology.orchestra.run.gibba002.stdout:--> Processing Dependency: ceph-base = 2:14.2.16-280.g962fac13.el7 for package: 2:ceph-mgr-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:00:17.873 INFO:teuthology.orchestra.run.gibba002.stdout:--> Processing Dependency: python-requests for package: 2:ceph-mgr-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:00:17.876 INFO:teuthology.orchestra.run.gibba002.stdout:--> Processing Dependency: python-pecan for package: 2:ceph-mgr-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:00:17.877 INFO:teuthology.orchestra.run.gibba002.stdout:--> Processing Dependency: python-cherrypy for package: 2:ceph-mgr-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:00:17.877 INFO:teuthology.orchestra.run.gibba002.stdout:--> Processing Dependency: python-werkzeug for package: 2:ceph-mgr-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:00:17.878 INFO:teuthology.orchestra.run.gibba002.stdout:--> Processing Dependency: python-bcrypt for package: 2:ceph-mgr-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:00:17.879 INFO:teuthology.orchestra.run.gibba002.stdout:---> Package ceph-mgr-dashboard.noarch 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:00:17.879 INFO:teuthology.orchestra.run.gibba002.stdout:--> Processing Dependency: ceph-grafana-dashboards = 2:14.2.16-280.g962fac13.el7 for package: 2:ceph-mgr-dashboard-14.2.16-280.g962fac13.el7.noarch 2021-02-10T11:00:17.880 INFO:teuthology.orchestra.run.gibba002.stdout:--> Processing Dependency: python-routes for package: 2:ceph-mgr-dashboard-14.2.16-280.g962fac13.el7.noarch 2021-02-10T11:00:17.881 INFO:teuthology.orchestra.run.gibba002.stdout:---> Package ceph-mgr-diskprediction-cloud.noarch 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:00:17.881 INFO:teuthology.orchestra.run.gibba002.stdout:---> Package ceph-mgr-diskprediction-local.noarch 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:00:17.881 INFO:teuthology.orchestra.run.gibba002.stdout:--> Processing Dependency: scipy for package: 2:ceph-mgr-diskprediction-local-14.2.16-280.g962fac13.el7.noarch 2021-02-10T11:00:17.883 INFO:teuthology.orchestra.run.gibba002.stdout:---> Package ceph-mgr-rook.noarch 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:00:17.883 INFO:teuthology.orchestra.run.gibba002.stdout:--> Processing Dependency: python-kubernetes for package: 2:ceph-mgr-rook-14.2.16-280.g962fac13.el7.noarch 2021-02-10T11:00:17.884 INFO:teuthology.orchestra.run.gibba002.stdout:---> Package ceph-mgr-ssh.noarch 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:00:17.884 INFO:teuthology.orchestra.run.gibba002.stdout:--> Processing Dependency: python-remoto for package: 2:ceph-mgr-ssh-14.2.16-280.g962fac13.el7.noarch 2021-02-10T11:00:17.884 INFO:teuthology.orchestra.run.gibba002.stdout:---> Package ceph-radosgw.x86_64 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:00:17.888 INFO:teuthology.orchestra.run.gibba002.stdout:--> Processing Dependency: ceph-selinux = 2:14.2.16-280.g962fac13.el7 for package: 2:ceph-radosgw-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:00:17.889 INFO:teuthology.orchestra.run.gibba002.stdout:--> Processing Dependency: librgw2 = 2:14.2.16-280.g962fac13.el7 for package: 2:ceph-radosgw-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:00:17.890 INFO:teuthology.orchestra.run.gibba002.stdout:--> Processing Dependency: librabbitmq.so.4()(64bit) for package: 2:ceph-radosgw-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:00:17.891 INFO:teuthology.orchestra.run.gibba002.stdout:--> Processing Dependency: liboath.so.0()(64bit) for package: 2:ceph-radosgw-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:00:17.892 INFO:teuthology.orchestra.run.gibba002.stdout:---> Package ceph-test.x86_64 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:00:17.895 INFO:teuthology.orchestra.run.gibba002.stdout:--> Processing Dependency: ceph-common = 2:14.2.16-280.g962fac13.el7 for package: 2:ceph-test-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:00:17.896 INFO:teuthology.orchestra.run.gibba002.stdout:--> Processing Dependency: jq for package: 2:ceph-test-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:00:17.897 INFO:teuthology.orchestra.run.gibba002.stdout:--> Processing Dependency: socat for package: 2:ceph-test-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:00:17.898 INFO:teuthology.orchestra.run.gibba002.stdout:--> Processing Dependency: libradosstriper.so.1()(64bit) for package: 2:ceph-test-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:00:17.899 INFO:teuthology.orchestra.run.gibba002.stdout:---> Package elfutils-libelf-devel.x86_64 0:0.176-5.el7 will be installed 2021-02-10T11:00:17.900 INFO:teuthology.orchestra.run.gibba002.stdout:--> Processing Dependency: pkgconfig(zlib) for package: elfutils-libelf-devel-0.176-5.el7.x86_64 2021-02-10T11:00:17.904 INFO:teuthology.orchestra.run.gibba002.stdout:---> Package flex.x86_64 0:2.5.37-6.el7 will be installed 2021-02-10T11:00:17.905 INFO:teuthology.orchestra.run.gibba002.stdout:---> Package libcephfs-devel.x86_64 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:00:17.905 INFO:teuthology.orchestra.run.gibba002.stdout:--> Processing Dependency: librados-devel = 2:14.2.16-280.g962fac13.el7 for package: 2:libcephfs-devel-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:00:17.906 INFO:teuthology.orchestra.run.gibba002.stdout:---> Package libcephfs2.x86_64 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:00:17.907 INFO:teuthology.orchestra.run.gibba002.stdout:---> Package librados2.x86_64 1:10.2.5-4.el7 will be updated 2021-02-10T11:00:17.907 INFO:teuthology.orchestra.run.gibba002.stdout:---> Package librados2.x86_64 2:14.2.16-280.g962fac13.el7 will be an update 2021-02-10T11:00:17.909 INFO:teuthology.orchestra.run.gibba002.stdout:--> Processing Dependency: liblttng-ust.so.0()(64bit) for package: 2:librados2-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:00:17.910 INFO:teuthology.orchestra.run.gibba002.stdout:---> Package librbd1.x86_64 1:10.2.5-4.el7 will be updated 2021-02-10T11:00:17.910 INFO:teuthology.orchestra.run.gibba002.stdout:---> Package librbd1.x86_64 2:14.2.16-280.g962fac13.el7 will be an update 2021-02-10T11:00:17.910 INFO:teuthology.orchestra.run.gibba002.stdout:---> Package openssl-devel.x86_64 1:1.0.2k-21.el7_9 will be installed 2021-02-10T11:00:17.911 INFO:teuthology.orchestra.run.gibba002.stdout:--> Processing Dependency: openssl-libs(x86-64) = 1:1.0.2k-21.el7_9 for package: 1:openssl-devel-1.0.2k-21.el7_9.x86_64 2021-02-10T11:00:17.929 INFO:teuthology.orchestra.run.gibba002.stdout:--> Processing Dependency: krb5-devel(x86-64) for package: 1:openssl-devel-1.0.2k-21.el7_9.x86_64 2021-02-10T11:00:17.935 INFO:teuthology.orchestra.run.gibba002.stdout:---> Package python-ceph-compat.x86_64 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:00:17.935 INFO:teuthology.orchestra.run.gibba002.stdout:--> Processing Dependency: python-rados = 2:14.2.16-280.g962fac13.el7 for package: 2:python-ceph-compat-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:00:17.936 INFO:teuthology.orchestra.run.gibba002.stdout:--> Processing Dependency: python-cephfs = 2:14.2.16-280.g962fac13.el7 for package: 2:python-ceph-compat-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:00:17.936 INFO:teuthology.orchestra.run.gibba002.stdout:--> Processing Dependency: python-rgw = 2:14.2.16-280.g962fac13.el7 for package: 2:python-ceph-compat-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:00:17.937 INFO:teuthology.orchestra.run.gibba002.stdout:--> Processing Dependency: python-rbd = 2:14.2.16-280.g962fac13.el7 for package: 2:python-ceph-compat-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:00:17.938 INFO:teuthology.orchestra.run.gibba002.stdout:---> Package python3-cephfs.x86_64 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:00:17.939 INFO:teuthology.orchestra.run.gibba002.stdout:--> Processing Dependency: python3-ceph-argparse = 2:14.2.16-280.g962fac13.el7 for package: 2:python3-cephfs-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:00:17.940 INFO:teuthology.orchestra.run.gibba002.stdout:--> Processing Dependency: python3-rados = 2:14.2.16-280.g962fac13.el7 for package: 2:python3-cephfs-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:00:17.941 INFO:teuthology.orchestra.run.gibba002.stdout:---> Package rbd-fuse.x86_64 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:00:17.941 INFO:teuthology.orchestra.run.gibba002.stdout:--> Running transaction check 2021-02-10T11:00:17.942 INFO:teuthology.orchestra.run.gibba002.stdout:---> Package ceph-base.x86_64 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:00:17.946 INFO:teuthology.orchestra.run.gibba002.stdout:---> Package ceph-common.x86_64 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:00:17.948 INFO:teuthology.orchestra.run.gibba002.stdout:--> Processing Dependency: python-ceph-argparse = 2:14.2.16-280.g962fac13.el7 for package: 2:ceph-common-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:00:17.949 INFO:teuthology.orchestra.run.gibba002.stdout:--> Processing Dependency: python-prettytable for package: 2:ceph-common-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:00:17.949 INFO:teuthology.orchestra.run.gibba002.stdout:--> Processing Dependency: librdkafka.so.1()(64bit) for package: 2:ceph-common-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:00:17.951 INFO:teuthology.orchestra.run.gibba002.stdout:--> Processing Dependency: libbabeltrace-ctf.so.1()(64bit) for package: 2:ceph-common-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:00:17.952 INFO:teuthology.orchestra.run.gibba002.stdout:--> Processing Dependency: libbabeltrace.so.1()(64bit) for package: 2:ceph-common-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:00:17.952 INFO:teuthology.orchestra.run.gibba002.stdout:---> Package ceph-grafana-dashboards.noarch 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:00:17.952 INFO:teuthology.orchestra.run.gibba002.stdout:---> Package ceph-mds.x86_64 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:00:17.952 INFO:teuthology.orchestra.run.gibba002.stdout:---> Package ceph-mon.x86_64 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:00:17.953 INFO:teuthology.orchestra.run.gibba002.stdout:---> Package ceph-osd.x86_64 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:00:17.955 INFO:teuthology.orchestra.run.gibba002.stdout:---> Package ceph-selinux.x86_64 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:00:17.957 INFO:teuthology.orchestra.run.gibba002.stdout:---> Package jq.x86_64 0:1.6-2.el7 will be installed 2021-02-10T11:00:17.957 INFO:teuthology.orchestra.run.gibba002.stdout:--> Processing Dependency: libonig.so.5()(64bit) for package: jq-1.6-2.el7.x86_64 2021-02-10T11:00:17.958 INFO:teuthology.orchestra.run.gibba002.stdout:---> Package krb5-devel.x86_64 0:1.15.1-50.el7 will be installed 2021-02-10T11:00:17.960 INFO:teuthology.orchestra.run.gibba002.stdout:--> Processing Dependency: libverto-devel for package: krb5-devel-1.15.1-50.el7.x86_64 2021-02-10T11:00:17.962 INFO:teuthology.orchestra.run.gibba002.stdout:--> Processing Dependency: libselinux-devel for package: krb5-devel-1.15.1-50.el7.x86_64 2021-02-10T11:00:17.967 INFO:teuthology.orchestra.run.gibba002.stdout:--> Processing Dependency: libcom_err-devel for package: krb5-devel-1.15.1-50.el7.x86_64 2021-02-10T11:00:17.977 INFO:teuthology.orchestra.run.gibba002.stdout:--> Processing Dependency: keyutils-libs-devel for package: krb5-devel-1.15.1-50.el7.x86_64 2021-02-10T11:00:17.979 INFO:teuthology.orchestra.run.gibba002.stdout:---> Package liboath.x86_64 0:2.6.2-1.el7 will be installed 2021-02-10T11:00:17.979 INFO:teuthology.orchestra.run.gibba002.stdout:---> Package librabbitmq.x86_64 0:0.8.0-3.el7 will be installed 2021-02-10T11:00:17.979 INFO:teuthology.orchestra.run.gibba002.stdout:---> Package librados-devel.x86_64 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:00:17.980 INFO:teuthology.orchestra.run.gibba002.stdout:---> Package libradosstriper1.x86_64 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:00:17.980 INFO:teuthology.orchestra.run.gibba002.stdout:---> Package librgw2.x86_64 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:00:17.981 INFO:teuthology.orchestra.run.gibba002.stdout:---> Package lttng-ust.x86_64 0:2.4.1-4.el7 will be installed 2021-02-10T11:00:17.982 INFO:teuthology.orchestra.run.gibba002.stdout:--> Processing Dependency: liburcu-bp.so.1()(64bit) for package: lttng-ust-2.4.1-4.el7.x86_64 2021-02-10T11:00:17.982 INFO:teuthology.orchestra.run.gibba002.stdout:--> Processing Dependency: liburcu-cds.so.1()(64bit) for package: lttng-ust-2.4.1-4.el7.x86_64 2021-02-10T11:00:17.983 INFO:teuthology.orchestra.run.gibba002.stdout:---> Package openssl-libs.x86_64 1:1.0.2k-19.el7 will be updated 2021-02-10T11:00:17.984 INFO:teuthology.orchestra.run.gibba002.stdout:--> Processing Dependency: openssl-libs(x86-64) = 1:1.0.2k-19.el7 for package: 1:openssl-1.0.2k-19.el7.x86_64 2021-02-10T11:00:17.984 INFO:teuthology.orchestra.run.gibba002.stdout:---> Package openssl-libs.x86_64 1:1.0.2k-21.el7_9 will be an update 2021-02-10T11:00:17.985 INFO:teuthology.orchestra.run.gibba002.stdout:---> Package python-cephfs.x86_64 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:00:17.985 INFO:teuthology.orchestra.run.gibba002.stdout:---> Package python-cherrypy.noarch 0:3.2.2-4.el7 will be installed 2021-02-10T11:00:17.994 INFO:teuthology.orchestra.run.gibba002.stdout:---> Package python-pecan.noarch 0:0.4.5-2.el7 will be installed 2021-02-10T11:00:17.994 INFO:teuthology.orchestra.run.gibba002.stdout:--> Processing Dependency: python-mako >= 0.4.0 for package: python-pecan-0.4.5-2.el7.noarch 2021-02-10T11:00:17.995 INFO:teuthology.orchestra.run.gibba002.stdout:--> Processing Dependency: python-simplegeneric >= 0.8 for package: python-pecan-0.4.5-2.el7.noarch 2021-02-10T11:00:17.995 INFO:teuthology.orchestra.run.gibba002.stdout:--> Processing Dependency: python-webob >= 1.2 for package: python-pecan-0.4.5-2.el7.noarch 2021-02-10T11:00:17.997 INFO:teuthology.orchestra.run.gibba002.stdout:--> Processing Dependency: python-webtest >= 1.3.1 for package: python-pecan-0.4.5-2.el7.noarch 2021-02-10T11:00:17.997 INFO:teuthology.orchestra.run.gibba002.stdout:--> Processing Dependency: python-singledispatch for package: python-pecan-0.4.5-2.el7.noarch 2021-02-10T11:00:17.998 INFO:teuthology.orchestra.run.gibba002.stdout:---> Package python-rados.x86_64 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:00:17.998 INFO:teuthology.orchestra.run.gibba002.stdout:---> Package python-rbd.x86_64 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:00:17.999 INFO:teuthology.orchestra.run.gibba002.stdout:---> Package python-requests.noarch 0:2.6.0-10.el7 will be installed 2021-02-10T11:00:17.999 INFO:teuthology.orchestra.run.gibba002.stdout:--> Processing Dependency: python-urllib3 >= 1.10.2-1 for package: python-requests-2.6.0-10.el7.noarch 2021-02-10T11:00:18.000 INFO:teuthology.orchestra.run.gibba002.stdout:---> Package python-rgw.x86_64 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:00:18.001 INFO:teuthology.orchestra.run.gibba002.stdout:---> Package python-routes.noarch 0:1.13-2.el7 will be installed 2021-02-10T11:00:18.001 INFO:teuthology.orchestra.run.gibba002.stdout:--> Processing Dependency: python-repoze-lru for package: python-routes-1.13-2.el7.noarch 2021-02-10T11:00:18.002 INFO:teuthology.orchestra.run.gibba002.stdout:---> Package python-werkzeug.noarch 0:0.9.1-2.el7 will be installed 2021-02-10T11:00:18.002 INFO:teuthology.orchestra.run.gibba002.stdout:---> Package python2-bcrypt.x86_64 0:3.1.6-2.el7 will be installed 2021-02-10T11:00:18.002 INFO:teuthology.orchestra.run.gibba002.stdout:--> Processing Dependency: python2-six for package: python2-bcrypt-3.1.6-2.el7.x86_64 2021-02-10T11:00:18.002 INFO:teuthology.orchestra.run.gibba002.stdout:---> Package python2-kubernetes.noarch 1:11.0.0-6.el7 will be installed 2021-02-10T11:00:18.003 INFO:teuthology.orchestra.run.gibba002.stdout:--> Processing Dependency: python-certifi for package: 1:python2-kubernetes-11.0.0-6.el7.noarch 2021-02-10T11:00:18.004 INFO:teuthology.orchestra.run.gibba002.stdout:--> Processing Dependency: python-google-auth for package: 1:python2-kubernetes-11.0.0-6.el7.noarch 2021-02-10T11:00:18.004 INFO:teuthology.orchestra.run.gibba002.stdout:--> Processing Dependency: python-requests-oauthlib for package: 1:python2-kubernetes-11.0.0-6.el7.noarch 2021-02-10T11:00:18.005 INFO:teuthology.orchestra.run.gibba002.stdout:--> Processing Dependency: python-websocket-client for package: 1:python2-kubernetes-11.0.0-6.el7.noarch 2021-02-10T11:00:18.008 INFO:teuthology.orchestra.run.gibba002.stdout:---> Package python2-remoto.noarch 0:1.0.0-1.el7 will be installed 2021-02-10T11:00:18.008 INFO:teuthology.orchestra.run.gibba002.stdout:--> Processing Dependency: python2-execnet >= 1.2.0 for package: python2-remoto-1.0.0-1.el7.noarch 2021-02-10T11:00:18.009 INFO:teuthology.orchestra.run.gibba002.stdout:---> Package python3-ceph-argparse.x86_64 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:00:18.009 INFO:teuthology.orchestra.run.gibba002.stdout:---> Package python3-rados.x86_64 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:00:18.009 INFO:teuthology.orchestra.run.gibba002.stdout:---> Package scipy.x86_64 0:0.12.1-6.el7 will be installed 2021-02-10T11:00:18.011 INFO:teuthology.orchestra.run.gibba002.stdout:--> Processing Dependency: f2py for package: scipy-0.12.1-6.el7.x86_64 2021-02-10T11:00:18.013 INFO:teuthology.orchestra.run.gibba002.stdout:--> Processing Dependency: libumfpack.so.5()(64bit) for package: scipy-0.12.1-6.el7.x86_64 2021-02-10T11:00:18.014 INFO:teuthology.orchestra.run.gibba002.stdout:--> Processing Dependency: libamd.so.2()(64bit) for package: scipy-0.12.1-6.el7.x86_64 2021-02-10T11:00:18.014 INFO:teuthology.orchestra.run.gibba002.stdout:---> Package socat.x86_64 0:1.7.3.2-2.el7 will be installed 2021-02-10T11:00:18.015 INFO:teuthology.orchestra.run.gibba002.stdout:---> Package zlib-devel.x86_64 0:1.2.7-19.el7_9 will be installed 2021-02-10T11:00:18.016 INFO:teuthology.orchestra.run.gibba002.stdout:--> Processing Dependency: zlib = 1.2.7-19.el7_9 for package: zlib-devel-1.2.7-19.el7_9.x86_64 2021-02-10T11:00:18.018 INFO:teuthology.orchestra.run.gibba002.stdout:--> Running transaction check 2021-02-10T11:00:18.018 INFO:teuthology.orchestra.run.gibba002.stdout:---> Package keyutils-libs-devel.x86_64 0:1.5.8-3.el7 will be installed 2021-02-10T11:00:18.018 INFO:teuthology.orchestra.run.gibba002.stdout:---> Package libbabeltrace.x86_64 0:1.2.4-3.el7 will be installed 2021-02-10T11:00:18.019 INFO:teuthology.orchestra.run.gibba002.stdout:---> Package libcom_err-devel.x86_64 0:1.42.9-19.el7 will be installed 2021-02-10T11:00:18.019 INFO:teuthology.orchestra.run.gibba002.stdout:---> Package librdkafka.x86_64 0:0.11.5-1.el7 will be installed 2021-02-10T11:00:18.020 INFO:teuthology.orchestra.run.gibba002.stdout:---> Package libselinux-devel.x86_64 0:2.5-15.el7 will be installed 2021-02-10T11:00:18.020 INFO:teuthology.orchestra.run.gibba002.stdout:--> Processing Dependency: libsepol-devel(x86-64) >= 2.5-10 for package: libselinux-devel-2.5-15.el7.x86_64 2021-02-10T11:00:18.022 INFO:teuthology.orchestra.run.gibba002.stdout:--> Processing Dependency: pkgconfig(libsepol) for package: libselinux-devel-2.5-15.el7.x86_64 2021-02-10T11:00:18.022 INFO:teuthology.orchestra.run.gibba002.stdout:--> Processing Dependency: pkgconfig(libpcre) for package: libselinux-devel-2.5-15.el7.x86_64 2021-02-10T11:00:18.027 INFO:teuthology.orchestra.run.gibba002.stdout:---> Package libverto-devel.x86_64 0:0.2.5-4.el7 will be installed 2021-02-10T11:00:18.027 INFO:teuthology.orchestra.run.gibba002.stdout:---> Package numpy-f2py.x86_64 1:1.7.1-13.el7 will be installed 2021-02-10T11:00:18.027 INFO:teuthology.orchestra.run.gibba002.stdout:---> Package oniguruma.x86_64 0:6.8.2-1.el7 will be installed 2021-02-10T11:00:18.027 INFO:teuthology.orchestra.run.gibba002.stdout:---> Package openssl.x86_64 1:1.0.2k-19.el7 will be updated 2021-02-10T11:00:18.028 INFO:teuthology.orchestra.run.gibba002.stdout:---> Package openssl.x86_64 1:1.0.2k-21.el7_9 will be an update 2021-02-10T11:00:18.029 INFO:teuthology.orchestra.run.gibba002.stdout:---> Package python-ceph-argparse.x86_64 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:00:18.029 INFO:teuthology.orchestra.run.gibba002.stdout:---> Package python-execnet.noarch 0:1.7.1-3.el7 will be installed 2021-02-10T11:00:18.030 INFO:teuthology.orchestra.run.gibba002.stdout:--> Processing Dependency: python-apipkg for package: python-execnet-1.7.1-3.el7.noarch 2021-02-10T11:00:18.030 INFO:teuthology.orchestra.run.gibba002.stdout:---> Package python-mako.noarch 0:0.8.1-2.el7 will be installed 2021-02-10T11:00:18.030 INFO:teuthology.orchestra.run.gibba002.stdout:--> Processing Dependency: python-markupsafe for package: python-mako-0.8.1-2.el7.noarch 2021-02-10T11:00:18.031 INFO:teuthology.orchestra.run.gibba002.stdout:--> Processing Dependency: python-beaker for package: python-mako-0.8.1-2.el7.noarch 2021-02-10T11:00:18.032 INFO:teuthology.orchestra.run.gibba002.stdout:---> Package python-prettytable.noarch 0:0.7.2-3.el7 will be installed 2021-02-10T11:00:18.032 INFO:teuthology.orchestra.run.gibba002.stdout:---> Package python-repoze-lru.noarch 0:0.4-3.el7 will be installed 2021-02-10T11:00:18.032 INFO:teuthology.orchestra.run.gibba002.stdout:---> Package python-simplegeneric.noarch 0:0.8-7.el7 will be installed 2021-02-10T11:00:18.032 INFO:teuthology.orchestra.run.gibba002.stdout:---> Package python-singledispatch.noarch 0:3.4.0.2-2.el7 will be installed 2021-02-10T11:00:18.032 INFO:teuthology.orchestra.run.gibba002.stdout:---> Package python-urllib3.noarch 0:1.10.2-7.el7 will be installed 2021-02-10T11:00:18.032 INFO:teuthology.orchestra.run.gibba002.stdout:---> Package python-webob.noarch 0:1.2.3-7.el7 will be installed 2021-02-10T11:00:18.033 INFO:teuthology.orchestra.run.gibba002.stdout:---> Package python-websocket-client.noarch 0:0.56.0-3.git3c25814.el7 will be installed 2021-02-10T11:00:18.033 INFO:teuthology.orchestra.run.gibba002.stdout:---> Package python-webtest.noarch 0:1.3.4-6.el7 will be installed 2021-02-10T11:00:18.033 INFO:teuthology.orchestra.run.gibba002.stdout:---> Package python2-certifi.noarch 0:2018.10.15-5.el7 will be installed 2021-02-10T11:00:18.034 INFO:teuthology.orchestra.run.gibba002.stdout:---> Package python2-google-auth.noarch 1:1.1.1-5.el7 will be installed 2021-02-10T11:00:18.034 INFO:teuthology.orchestra.run.gibba002.stdout:--> Processing Dependency: python-cachetools for package: 1:python2-google-auth-1.1.1-5.el7.noarch 2021-02-10T11:00:18.034 INFO:teuthology.orchestra.run.gibba002.stdout:--> Processing Dependency: python2-pyasn1-modules for package: 1:python2-google-auth-1.1.1-5.el7.noarch 2021-02-10T11:00:18.034 INFO:teuthology.orchestra.run.gibba002.stdout:--> Processing Dependency: python2-rsa for package: 1:python2-google-auth-1.1.1-5.el7.noarch 2021-02-10T11:00:18.035 INFO:teuthology.orchestra.run.gibba002.stdout:---> Package python2-requests-oauthlib.noarch 0:0.8.0-5.el7 will be installed 2021-02-10T11:00:18.035 INFO:teuthology.orchestra.run.gibba002.stdout:--> Processing Dependency: python2-oauthlib for package: python2-requests-oauthlib-0.8.0-5.el7.noarch 2021-02-10T11:00:18.036 INFO:teuthology.orchestra.run.gibba002.stdout:---> Package python2-six.noarch 0:1.9.0-0.el7 will be installed 2021-02-10T11:00:18.036 INFO:teuthology.orchestra.run.gibba002.stdout:---> Package suitesparse.x86_64 0:4.0.2-10.el7 will be installed 2021-02-10T11:00:18.037 INFO:teuthology.orchestra.run.gibba002.stdout:--> Processing Dependency: libtbbmalloc.so.2()(64bit) for package: suitesparse-4.0.2-10.el7.x86_64 2021-02-10T11:00:18.038 INFO:teuthology.orchestra.run.gibba002.stdout:--> Processing Dependency: libtbb.so.2()(64bit) for package: suitesparse-4.0.2-10.el7.x86_64 2021-02-10T11:00:18.038 INFO:teuthology.orchestra.run.gibba002.stdout:---> Package userspace-rcu.x86_64 0:0.7.16-1.el7 will be installed 2021-02-10T11:00:18.039 INFO:teuthology.orchestra.run.gibba002.stdout:---> Package zlib.x86_64 0:1.2.7-18.el7 will be updated 2021-02-10T11:00:18.040 INFO:teuthology.orchestra.run.gibba002.stdout:---> Package zlib.x86_64 0:1.2.7-19.el7_9 will be an update 2021-02-10T11:00:18.040 INFO:teuthology.orchestra.run.gibba002.stdout:--> Running transaction check 2021-02-10T11:00:18.040 INFO:teuthology.orchestra.run.gibba002.stdout:---> Package libsepol-devel.x86_64 0:2.5-10.el7 will be installed 2021-02-10T11:00:18.041 INFO:teuthology.orchestra.run.gibba002.stdout:---> Package pcre-devel.x86_64 0:8.32-17.el7 will be installed 2021-02-10T11:00:18.041 INFO:teuthology.orchestra.run.gibba002.stdout:---> Package python-beaker.noarch 0:1.5.4-10.el7 will be installed 2021-02-10T11:00:18.041 INFO:teuthology.orchestra.run.gibba002.stdout:--> Processing Dependency: python-paste for package: python-beaker-1.5.4-10.el7.noarch 2021-02-10T11:00:18.042 INFO:teuthology.orchestra.run.gibba002.stdout:---> Package python-cachetools.noarch 0:1.0.3-1.el7 will be installed 2021-02-10T11:00:18.042 INFO:teuthology.orchestra.run.gibba002.stdout:---> Package python-markupsafe.x86_64 0:0.11-10.el7 will be installed 2021-02-10T11:00:18.042 INFO:teuthology.orchestra.run.gibba002.stdout:---> Package python2-apipkg.noarch 0:1.5-5.el7 will be installed 2021-02-10T11:00:18.042 INFO:teuthology.orchestra.run.gibba002.stdout:---> Package python2-oauthlib.noarch 0:2.0.1-8.el7 will be installed 2021-02-10T11:00:18.043 INFO:teuthology.orchestra.run.gibba002.stdout:--> Processing Dependency: python-jwcrypto for package: python2-oauthlib-2.0.1-8.el7.noarch 2021-02-10T11:00:18.044 INFO:teuthology.orchestra.run.gibba002.stdout:---> Package python2-pyasn1-modules.noarch 0:0.1.9-7.el7 will be installed 2021-02-10T11:00:18.044 INFO:teuthology.orchestra.run.gibba002.stdout:---> Package python2-rsa.noarch 0:3.4.2-2.el7 will be installed 2021-02-10T11:00:18.044 INFO:teuthology.orchestra.run.gibba002.stdout:---> Package tbb.x86_64 0:4.1-9.20130314.el7 will be installed 2021-02-10T11:00:18.044 INFO:teuthology.orchestra.run.gibba002.stdout:--> Running transaction check 2021-02-10T11:00:18.045 INFO:teuthology.orchestra.run.gibba002.stdout:---> Package python-jwcrypto.noarch 0:0.4.2-1.el7 will be installed 2021-02-10T11:00:18.045 INFO:teuthology.orchestra.run.gibba002.stdout:---> Package python-paste.noarch 0:1.7.5.1-9.20111221hg1498.el7 will be installed 2021-02-10T11:00:18.045 INFO:teuthology.orchestra.run.gibba002.stdout:--> Processing Dependency: python-tempita for package: python-paste-1.7.5.1-9.20111221hg1498.el7.noarch 2021-02-10T11:00:18.045 INFO:teuthology.orchestra.run.gibba002.stdout:--> Running transaction check 2021-02-10T11:00:18.045 INFO:teuthology.orchestra.run.gibba002.stdout:---> Package python-tempita.noarch 0:0.5.1-6.el7 will be installed 2021-02-10T11:00:18.642 INFO:teuthology.orchestra.run.gibba002.stdout:--> Finished Dependency Resolution 2021-02-10T11:00:18.697 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:00:18.697 INFO:teuthology.orchestra.run.gibba002.stdout:Dependencies Resolved 2021-02-10T11:00:18.706 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:00:18.706 INFO:teuthology.orchestra.run.gibba002.stdout:================================================================================ 2021-02-10T11:00:18.706 INFO:teuthology.orchestra.run.gibba002.stdout: Package Arch Version Repository Size 2021-02-10T11:00:18.706 INFO:teuthology.orchestra.run.gibba002.stdout:================================================================================ 2021-02-10T11:00:18.706 INFO:teuthology.orchestra.run.gibba002.stdout:Installing: 2021-02-10T11:00:18.706 INFO:teuthology.orchestra.run.gibba002.stdout: bison x86_64 3.0.4-2.el7 rhel-7-server-rpms 674 k 2021-02-10T11:00:18.706 INFO:teuthology.orchestra.run.gibba002.stdout: ceph x86_64 2:14.2.16-280.g962fac13.el7 ceph 2.0 k 2021-02-10T11:00:18.707 INFO:teuthology.orchestra.run.gibba002.stdout: ceph-fuse x86_64 2:14.2.16-280.g962fac13.el7 ceph 571 k 2021-02-10T11:00:18.707 INFO:teuthology.orchestra.run.gibba002.stdout: ceph-mgr x86_64 2:14.2.16-280.g962fac13.el7 ceph 1.8 M 2021-02-10T11:00:18.707 INFO:teuthology.orchestra.run.gibba002.stdout: ceph-mgr-dashboard noarch 2:14.2.16-280.g962fac13.el7 ceph-noarch 4.0 M 2021-02-10T11:00:18.707 INFO:teuthology.orchestra.run.gibba002.stdout: ceph-mgr-diskprediction-cloud 2021-02-10T11:00:18.707 INFO:teuthology.orchestra.run.gibba002.stdout: noarch 2:14.2.16-280.g962fac13.el7 ceph-noarch 83 k 2021-02-10T11:00:18.707 INFO:teuthology.orchestra.run.gibba002.stdout: ceph-mgr-diskprediction-local 2021-02-10T11:00:18.707 INFO:teuthology.orchestra.run.gibba002.stdout: noarch 2:14.2.16-280.g962fac13.el7 ceph-noarch 1.1 M 2021-02-10T11:00:18.707 INFO:teuthology.orchestra.run.gibba002.stdout: ceph-mgr-rook noarch 2:14.2.16-280.g962fac13.el7 ceph-noarch 21 k 2021-02-10T11:00:18.707 INFO:teuthology.orchestra.run.gibba002.stdout: ceph-mgr-ssh noarch 2:14.2.16-280.g962fac13.el7 ceph-noarch 21 k 2021-02-10T11:00:18.707 INFO:teuthology.orchestra.run.gibba002.stdout: ceph-radosgw x86_64 2:14.2.16-280.g962fac13.el7 ceph 5.8 M 2021-02-10T11:00:18.707 INFO:teuthology.orchestra.run.gibba002.stdout: ceph-test x86_64 2:14.2.16-280.g962fac13.el7 ceph 34 M 2021-02-10T11:00:18.707 INFO:teuthology.orchestra.run.gibba002.stdout: elfutils-libelf-devel 2021-02-10T11:00:18.708 INFO:teuthology.orchestra.run.gibba002.stdout: x86_64 0.176-5.el7 rhel-7-server-rpms 40 k 2021-02-10T11:00:18.708 INFO:teuthology.orchestra.run.gibba002.stdout: flex x86_64 2.5.37-6.el7 rhel-7-server-rpms 293 k 2021-02-10T11:00:18.708 INFO:teuthology.orchestra.run.gibba002.stdout: libcephfs-devel x86_64 2:14.2.16-280.g962fac13.el7 ceph 17 k 2021-02-10T11:00:18.708 INFO:teuthology.orchestra.run.gibba002.stdout: libcephfs2 x86_64 2:14.2.16-280.g962fac13.el7 ceph 500 k 2021-02-10T11:00:18.708 INFO:teuthology.orchestra.run.gibba002.stdout: openssl-devel x86_64 1:1.0.2k-21.el7_9 rhel-7-server-rpms 1.5 M 2021-02-10T11:00:18.708 INFO:teuthology.orchestra.run.gibba002.stdout: python-ceph-compat x86_64 2:14.2.16-280.g962fac13.el7 ceph 2.2 k 2021-02-10T11:00:18.708 INFO:teuthology.orchestra.run.gibba002.stdout: python3-cephfs x86_64 2:14.2.16-280.g962fac13.el7 ceph 151 k 2021-02-10T11:00:18.708 INFO:teuthology.orchestra.run.gibba002.stdout: rbd-fuse x86_64 2:14.2.16-280.g962fac13.el7 ceph 69 k 2021-02-10T11:00:18.708 INFO:teuthology.orchestra.run.gibba002.stdout:Updating: 2021-02-10T11:00:18.708 INFO:teuthology.orchestra.run.gibba002.stdout: librados2 x86_64 2:14.2.16-280.g962fac13.el7 ceph 3.4 M 2021-02-10T11:00:18.708 INFO:teuthology.orchestra.run.gibba002.stdout: librbd1 x86_64 2:14.2.16-280.g962fac13.el7 ceph 1.7 M 2021-02-10T11:00:18.708 INFO:teuthology.orchestra.run.gibba002.stdout:Installing for dependencies: 2021-02-10T11:00:18.708 INFO:teuthology.orchestra.run.gibba002.stdout: ceph-base x86_64 2:14.2.16-280.g962fac13.el7 ceph 5.4 M 2021-02-10T11:00:18.709 INFO:teuthology.orchestra.run.gibba002.stdout: ceph-common x86_64 2:14.2.16-280.g962fac13.el7 ceph 19 M 2021-02-10T11:00:18.709 INFO:teuthology.orchestra.run.gibba002.stdout: ceph-grafana-dashboards 2021-02-10T11:00:18.709 INFO:teuthology.orchestra.run.gibba002.stdout: noarch 2:14.2.16-280.g962fac13.el7 ceph-noarch 19 k 2021-02-10T11:00:18.709 INFO:teuthology.orchestra.run.gibba002.stdout: ceph-mds x86_64 2:14.2.16-280.g962fac13.el7 ceph 1.9 M 2021-02-10T11:00:18.709 INFO:teuthology.orchestra.run.gibba002.stdout: ceph-mon x86_64 2:14.2.16-280.g962fac13.el7 ceph 4.3 M 2021-02-10T11:00:18.709 INFO:teuthology.orchestra.run.gibba002.stdout: ceph-osd x86_64 2:14.2.16-280.g962fac13.el7 ceph 16 M 2021-02-10T11:00:18.709 INFO:teuthology.orchestra.run.gibba002.stdout: ceph-selinux x86_64 2:14.2.16-280.g962fac13.el7 ceph 20 k 2021-02-10T11:00:18.709 INFO:teuthology.orchestra.run.gibba002.stdout: jq x86_64 1.6-2.el7 epel 167 k 2021-02-10T11:00:18.709 INFO:teuthology.orchestra.run.gibba002.stdout: keyutils-libs-devel 2021-02-10T11:00:18.709 INFO:teuthology.orchestra.run.gibba002.stdout: x86_64 1.5.8-3.el7 rhel-7-server-rpms 37 k 2021-02-10T11:00:18.709 INFO:teuthology.orchestra.run.gibba002.stdout: krb5-devel x86_64 1.15.1-50.el7 rhel-7-server-rpms 273 k 2021-02-10T11:00:18.709 INFO:teuthology.orchestra.run.gibba002.stdout: libbabeltrace x86_64 1.2.4-3.el7 epel 147 k 2021-02-10T11:00:18.709 INFO:teuthology.orchestra.run.gibba002.stdout: libcom_err-devel x86_64 1.42.9-19.el7 rhel-7-server-rpms 32 k 2021-02-10T11:00:18.710 INFO:teuthology.orchestra.run.gibba002.stdout: liboath x86_64 2.6.2-1.el7 epel 51 k 2021-02-10T11:00:18.710 INFO:teuthology.orchestra.run.gibba002.stdout: librabbitmq x86_64 0.8.0-3.el7 rhel-7-server-rpms 37 k 2021-02-10T11:00:18.710 INFO:teuthology.orchestra.run.gibba002.stdout: librados-devel x86_64 2:14.2.16-280.g962fac13.el7 ceph 86 k 2021-02-10T11:00:18.710 INFO:teuthology.orchestra.run.gibba002.stdout: libradosstriper1 x86_64 2:14.2.16-280.g962fac13.el7 ceph 364 k 2021-02-10T11:00:18.710 INFO:teuthology.orchestra.run.gibba002.stdout: librdkafka x86_64 0.11.5-1.el7 epel 368 k 2021-02-10T11:00:18.710 INFO:teuthology.orchestra.run.gibba002.stdout: librgw2 x86_64 2:14.2.16-280.g962fac13.el7 ceph 5.1 M 2021-02-10T11:00:18.710 INFO:teuthology.orchestra.run.gibba002.stdout: libselinux-devel x86_64 2.5-15.el7 rhel-7-server-rpms 187 k 2021-02-10T11:00:18.710 INFO:teuthology.orchestra.run.gibba002.stdout: libsepol-devel x86_64 2.5-10.el7 rhel-7-server-rpms 77 k 2021-02-10T11:00:18.710 INFO:teuthology.orchestra.run.gibba002.stdout: libverto-devel x86_64 0.2.5-4.el7 rhel-7-server-rpms 12 k 2021-02-10T11:00:18.710 INFO:teuthology.orchestra.run.gibba002.stdout: lttng-ust x86_64 2.4.1-4.el7 epel 176 k 2021-02-10T11:00:18.710 INFO:teuthology.orchestra.run.gibba002.stdout: numpy-f2py x86_64 1:1.7.1-13.el7 rhel-7-server-rpms 206 k 2021-02-10T11:00:18.710 INFO:teuthology.orchestra.run.gibba002.stdout: oniguruma x86_64 6.8.2-1.el7 epel 181 k 2021-02-10T11:00:18.711 INFO:teuthology.orchestra.run.gibba002.stdout: pcre-devel x86_64 8.32-17.el7 rhel-7-server-rpms 480 k 2021-02-10T11:00:18.711 INFO:teuthology.orchestra.run.gibba002.stdout: python-beaker noarch 1.5.4-10.el7 rhel-7-server-rpms 80 k 2021-02-10T11:00:18.711 INFO:teuthology.orchestra.run.gibba002.stdout: python-cachetools noarch 1.0.3-1.el7 epel 22 k 2021-02-10T11:00:18.711 INFO:teuthology.orchestra.run.gibba002.stdout: python-ceph-argparse 2021-02-10T11:00:18.711 INFO:teuthology.orchestra.run.gibba002.stdout: x86_64 2:14.2.16-280.g962fac13.el7 ceph 35 k 2021-02-10T11:00:18.711 INFO:teuthology.orchestra.run.gibba002.stdout: python-cephfs x86_64 2:14.2.16-280.g962fac13.el7 ceph 113 k 2021-02-10T11:00:18.711 INFO:teuthology.orchestra.run.gibba002.stdout: python-cherrypy noarch 3.2.2-4.el7 rhel-7-server-rpms 422 k 2021-02-10T11:00:18.711 INFO:teuthology.orchestra.run.gibba002.stdout: python-execnet noarch 1.7.1-3.el7 epel 260 k 2021-02-10T11:00:18.711 INFO:teuthology.orchestra.run.gibba002.stdout: python-jwcrypto noarch 0.4.2-1.el7 rhel-7-server-rpms 57 k 2021-02-10T11:00:18.711 INFO:teuthology.orchestra.run.gibba002.stdout: python-mako noarch 0.8.1-2.el7 rhel-7-server-rpms 307 k 2021-02-10T11:00:18.711 INFO:teuthology.orchestra.run.gibba002.stdout: python-markupsafe x86_64 0.11-10.el7 rhel-7-server-rpms 25 k 2021-02-10T11:00:18.711 INFO:teuthology.orchestra.run.gibba002.stdout: python-paste noarch 1.7.5.1-9.20111221hg1498.el7 2021-02-10T11:00:18.711 INFO:teuthology.orchestra.run.gibba002.stdout: rhel-7-server-rpms 866 k 2021-02-10T11:00:18.712 INFO:teuthology.orchestra.run.gibba002.stdout: python-pecan noarch 0.4.5-2.el7 epel 255 k 2021-02-10T11:00:18.712 INFO:teuthology.orchestra.run.gibba002.stdout: python-prettytable noarch 0.7.2-3.el7 rhel-7-server-rpms 37 k 2021-02-10T11:00:18.712 INFO:teuthology.orchestra.run.gibba002.stdout: python-rados x86_64 2:14.2.16-280.g962fac13.el7 ceph 199 k 2021-02-10T11:00:18.712 INFO:teuthology.orchestra.run.gibba002.stdout: python-rbd x86_64 2:14.2.16-280.g962fac13.el7 ceph 194 k 2021-02-10T11:00:18.712 INFO:teuthology.orchestra.run.gibba002.stdout: python-repoze-lru noarch 0.4-3.el7 epel 13 k 2021-02-10T11:00:18.712 INFO:teuthology.orchestra.run.gibba002.stdout: python-requests noarch 2.6.0-10.el7 rhel-7-server-rpms 95 k 2021-02-10T11:00:18.712 INFO:teuthology.orchestra.run.gibba002.stdout: python-rgw x86_64 2:14.2.16-280.g962fac13.el7 ceph 77 k 2021-02-10T11:00:18.712 INFO:teuthology.orchestra.run.gibba002.stdout: python-routes noarch 1.13-2.el7 epel 640 k 2021-02-10T11:00:18.712 INFO:teuthology.orchestra.run.gibba002.stdout: python-simplegeneric 2021-02-10T11:00:18.712 INFO:teuthology.orchestra.run.gibba002.stdout: noarch 0.8-7.el7 epel 12 k 2021-02-10T11:00:18.712 INFO:teuthology.orchestra.run.gibba002.stdout: python-singledispatch 2021-02-10T11:00:18.712 INFO:teuthology.orchestra.run.gibba002.stdout: noarch 3.4.0.2-2.el7 epel 18 k 2021-02-10T11:00:18.712 INFO:teuthology.orchestra.run.gibba002.stdout: python-tempita noarch 0.5.1-6.el7 rhel-7-server-rpms 33 k 2021-02-10T11:00:18.713 INFO:teuthology.orchestra.run.gibba002.stdout: python-urllib3 noarch 1.10.2-7.el7 rhel-7-server-rpms 103 k 2021-02-10T11:00:18.713 INFO:teuthology.orchestra.run.gibba002.stdout: python-webob noarch 1.2.3-7.el7 rhel-7-server-optional-rpms 2021-02-10T11:00:18.713 INFO:teuthology.orchestra.run.gibba002.stdout: 202 k 2021-02-10T11:00:18.713 INFO:teuthology.orchestra.run.gibba002.stdout: python-websocket-client 2021-02-10T11:00:18.713 INFO:teuthology.orchestra.run.gibba002.stdout: noarch 0.56.0-3.git3c25814.el7 rhel-7-server-extras-rpms 2021-02-10T11:00:18.713 INFO:teuthology.orchestra.run.gibba002.stdout: 66 k 2021-02-10T11:00:18.713 INFO:teuthology.orchestra.run.gibba002.stdout: python-webtest noarch 1.3.4-6.el7 rhel-7-server-optional-rpms 2021-02-10T11:00:18.713 INFO:teuthology.orchestra.run.gibba002.stdout: 102 k 2021-02-10T11:00:18.713 INFO:teuthology.orchestra.run.gibba002.stdout: python-werkzeug noarch 0.9.1-2.el7 rhel-7-server-extras-rpms 2021-02-10T11:00:18.713 INFO:teuthology.orchestra.run.gibba002.stdout: 562 k 2021-02-10T11:00:18.713 INFO:teuthology.orchestra.run.gibba002.stdout: python2-apipkg noarch 1.5-5.el7 epel 17 k 2021-02-10T11:00:18.713 INFO:teuthology.orchestra.run.gibba002.stdout: python2-bcrypt x86_64 3.1.6-2.el7 epel 39 k 2021-02-10T11:00:18.713 INFO:teuthology.orchestra.run.gibba002.stdout: python2-certifi noarch 2018.10.15-5.el7 epel 12 k 2021-02-10T11:00:18.714 INFO:teuthology.orchestra.run.gibba002.stdout: python2-google-auth 2021-02-10T11:00:18.714 INFO:teuthology.orchestra.run.gibba002.stdout: noarch 1:1.1.1-5.el7 epel 76 k 2021-02-10T11:00:18.714 INFO:teuthology.orchestra.run.gibba002.stdout: python2-kubernetes noarch 1:11.0.0-6.el7 epel 1.1 M 2021-02-10T11:00:18.714 INFO:teuthology.orchestra.run.gibba002.stdout: python2-oauthlib noarch 2.0.1-8.el7 rhel-7-server-rpms 146 k 2021-02-10T11:00:18.714 INFO:teuthology.orchestra.run.gibba002.stdout: python2-pyasn1-modules 2021-02-10T11:00:18.714 INFO:teuthology.orchestra.run.gibba002.stdout: noarch 0.1.9-7.el7 rhel-7-server-optional-rpms 2021-02-10T11:00:18.714 INFO:teuthology.orchestra.run.gibba002.stdout: 59 k 2021-02-10T11:00:18.714 INFO:teuthology.orchestra.run.gibba002.stdout: python2-remoto noarch 1.0.0-1.el7 epel 42 k 2021-02-10T11:00:18.714 INFO:teuthology.orchestra.run.gibba002.stdout: python2-requests-oauthlib 2021-02-10T11:00:18.714 INFO:teuthology.orchestra.run.gibba002.stdout: noarch 0.8.0-5.el7 rhel-7-server-rpms 35 k 2021-02-10T11:00:18.714 INFO:teuthology.orchestra.run.gibba002.stdout: python2-rsa noarch 3.4.2-2.el7 epel 68 k 2021-02-10T11:00:18.714 INFO:teuthology.orchestra.run.gibba002.stdout: python2-six noarch 1.9.0-0.el7 epel 2.9 k 2021-02-10T11:00:18.714 INFO:teuthology.orchestra.run.gibba002.stdout: python3-ceph-argparse 2021-02-10T11:00:18.715 INFO:teuthology.orchestra.run.gibba002.stdout: x86_64 2:14.2.16-280.g962fac13.el7 ceph 35 k 2021-02-10T11:00:18.715 INFO:teuthology.orchestra.run.gibba002.stdout: python3-rados x86_64 2:14.2.16-280.g962fac13.el7 ceph 317 k 2021-02-10T11:00:18.715 INFO:teuthology.orchestra.run.gibba002.stdout: scipy x86_64 0.12.1-6.el7 rhel-7-server-rpms 9.3 M 2021-02-10T11:00:18.715 INFO:teuthology.orchestra.run.gibba002.stdout: socat x86_64 1.7.3.2-2.el7 rhel-7-server-rpms 290 k 2021-02-10T11:00:18.715 INFO:teuthology.orchestra.run.gibba002.stdout: suitesparse x86_64 4.0.2-10.el7 rhel-7-server-rpms 928 k 2021-02-10T11:00:18.715 INFO:teuthology.orchestra.run.gibba002.stdout: tbb x86_64 4.1-9.20130314.el7 rhel-7-server-rpms 124 k 2021-02-10T11:00:18.715 INFO:teuthology.orchestra.run.gibba002.stdout: userspace-rcu x86_64 0.7.16-1.el7 epel 73 k 2021-02-10T11:00:18.715 INFO:teuthology.orchestra.run.gibba002.stdout: zlib-devel x86_64 1.2.7-19.el7_9 rhel-7-server-rpms 50 k 2021-02-10T11:00:18.715 INFO:teuthology.orchestra.run.gibba002.stdout:Updating for dependencies: 2021-02-10T11:00:18.715 INFO:teuthology.orchestra.run.gibba002.stdout: openssl x86_64 1:1.0.2k-21.el7_9 rhel-7-server-rpms 493 k 2021-02-10T11:00:18.716 INFO:teuthology.orchestra.run.gibba002.stdout: openssl-libs x86_64 1:1.0.2k-21.el7_9 rhel-7-server-rpms 1.2 M 2021-02-10T11:00:18.716 INFO:teuthology.orchestra.run.gibba002.stdout: zlib x86_64 1.2.7-19.el7_9 rhel-7-server-rpms 90 k 2021-02-10T11:00:18.716 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:00:18.716 INFO:teuthology.orchestra.run.gibba002.stdout:Transaction Summary 2021-02-10T11:00:18.716 INFO:teuthology.orchestra.run.gibba002.stdout:================================================================================ 2021-02-10T11:00:18.716 INFO:teuthology.orchestra.run.gibba002.stdout:Install 19 Packages (+70 Dependent packages) 2021-02-10T11:00:18.716 INFO:teuthology.orchestra.run.gibba002.stdout:Upgrade 2 Packages (+ 3 Dependent packages) 2021-02-10T11:00:18.716 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:00:18.717 INFO:teuthology.orchestra.run.gibba002.stdout:Total download size: 130 M 2021-02-10T11:00:18.717 INFO:teuthology.orchestra.run.gibba002.stdout:Downloading packages: 2021-02-10T11:00:18.717 INFO:teuthology.orchestra.run.gibba002.stdout:Delta RPMs disabled because /usr/bin/applydeltarpm not installed. 2021-02-10T11:00:23.373 INFO:teuthology.orchestra.run.gibba013.stdout:Resolving Dependencies 2021-02-10T11:00:23.374 INFO:teuthology.orchestra.run.gibba013.stdout:--> Running transaction check 2021-02-10T11:00:23.374 INFO:teuthology.orchestra.run.gibba013.stdout:---> Package bison.x86_64 0:3.0.4-2.el7 will be installed 2021-02-10T11:00:23.379 INFO:teuthology.orchestra.run.gibba013.stdout:---> Package ceph.x86_64 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:00:23.381 INFO:teuthology.orchestra.run.gibba013.stdout:--> Processing Dependency: ceph-mon = 2:14.2.16-280.g962fac13.el7 for package: 2:ceph-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:00:23.382 INFO:teuthology.orchestra.run.gibba013.stdout:--> Processing Dependency: ceph-osd = 2:14.2.16-280.g962fac13.el7 for package: 2:ceph-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:00:23.383 INFO:teuthology.orchestra.run.gibba013.stdout:--> Processing Dependency: ceph-mds = 2:14.2.16-280.g962fac13.el7 for package: 2:ceph-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:00:23.384 INFO:teuthology.orchestra.run.gibba013.stdout:---> Package ceph-fuse.x86_64 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:00:23.391 INFO:teuthology.orchestra.run.gibba013.stdout:---> Package ceph-mgr.x86_64 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:00:23.393 INFO:teuthology.orchestra.run.gibba013.stdout:--> Processing Dependency: ceph-base = 2:14.2.16-280.g962fac13.el7 for package: 2:ceph-mgr-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:00:23.394 INFO:teuthology.orchestra.run.gibba013.stdout:--> Processing Dependency: python-requests for package: 2:ceph-mgr-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:00:23.397 INFO:teuthology.orchestra.run.gibba013.stdout:--> Processing Dependency: python-pecan for package: 2:ceph-mgr-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:00:23.398 INFO:teuthology.orchestra.run.gibba013.stdout:--> Processing Dependency: python-cherrypy for package: 2:ceph-mgr-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:00:23.398 INFO:teuthology.orchestra.run.gibba013.stdout:--> Processing Dependency: python-werkzeug for package: 2:ceph-mgr-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:00:23.399 INFO:teuthology.orchestra.run.gibba013.stdout:--> Processing Dependency: python-bcrypt for package: 2:ceph-mgr-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:00:23.400 INFO:teuthology.orchestra.run.gibba013.stdout:---> Package ceph-mgr-dashboard.noarch 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:00:23.400 INFO:teuthology.orchestra.run.gibba013.stdout:--> Processing Dependency: ceph-grafana-dashboards = 2:14.2.16-280.g962fac13.el7 for package: 2:ceph-mgr-dashboard-14.2.16-280.g962fac13.el7.noarch 2021-02-10T11:00:23.401 INFO:teuthology.orchestra.run.gibba013.stdout:--> Processing Dependency: python-routes for package: 2:ceph-mgr-dashboard-14.2.16-280.g962fac13.el7.noarch 2021-02-10T11:00:23.402 INFO:teuthology.orchestra.run.gibba013.stdout:---> Package ceph-mgr-diskprediction-cloud.noarch 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:00:23.402 INFO:teuthology.orchestra.run.gibba013.stdout:---> Package ceph-mgr-diskprediction-local.noarch 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:00:23.402 INFO:teuthology.orchestra.run.gibba013.stdout:--> Processing Dependency: scipy for package: 2:ceph-mgr-diskprediction-local-14.2.16-280.g962fac13.el7.noarch 2021-02-10T11:00:23.404 INFO:teuthology.orchestra.run.gibba013.stdout:---> Package ceph-mgr-rook.noarch 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:00:23.404 INFO:teuthology.orchestra.run.gibba013.stdout:--> Processing Dependency: python-kubernetes for package: 2:ceph-mgr-rook-14.2.16-280.g962fac13.el7.noarch 2021-02-10T11:00:23.405 INFO:teuthology.orchestra.run.gibba013.stdout:---> Package ceph-mgr-ssh.noarch 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:00:23.405 INFO:teuthology.orchestra.run.gibba013.stdout:--> Processing Dependency: python-remoto for package: 2:ceph-mgr-ssh-14.2.16-280.g962fac13.el7.noarch 2021-02-10T11:00:23.406 INFO:teuthology.orchestra.run.gibba013.stdout:---> Package ceph-radosgw.x86_64 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:00:23.409 INFO:teuthology.orchestra.run.gibba013.stdout:--> Processing Dependency: ceph-selinux = 2:14.2.16-280.g962fac13.el7 for package: 2:ceph-radosgw-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:00:23.410 INFO:teuthology.orchestra.run.gibba013.stdout:--> Processing Dependency: librgw2 = 2:14.2.16-280.g962fac13.el7 for package: 2:ceph-radosgw-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:00:23.411 INFO:teuthology.orchestra.run.gibba013.stdout:--> Processing Dependency: librabbitmq.so.4()(64bit) for package: 2:ceph-radosgw-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:00:23.412 INFO:teuthology.orchestra.run.gibba013.stdout:--> Processing Dependency: liboath.so.0()(64bit) for package: 2:ceph-radosgw-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:00:23.413 INFO:teuthology.orchestra.run.gibba013.stdout:---> Package ceph-test.x86_64 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:00:23.416 INFO:teuthology.orchestra.run.gibba013.stdout:--> Processing Dependency: ceph-common = 2:14.2.16-280.g962fac13.el7 for package: 2:ceph-test-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:00:23.418 INFO:teuthology.orchestra.run.gibba013.stdout:--> Processing Dependency: jq for package: 2:ceph-test-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:00:23.418 INFO:teuthology.orchestra.run.gibba013.stdout:--> Processing Dependency: socat for package: 2:ceph-test-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:00:23.420 INFO:teuthology.orchestra.run.gibba013.stdout:--> Processing Dependency: libradosstriper.so.1()(64bit) for package: 2:ceph-test-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:00:23.421 INFO:teuthology.orchestra.run.gibba013.stdout:---> Package elfutils-libelf-devel.x86_64 0:0.176-5.el7 will be installed 2021-02-10T11:00:23.421 INFO:teuthology.orchestra.run.gibba013.stdout:--> Processing Dependency: pkgconfig(zlib) for package: elfutils-libelf-devel-0.176-5.el7.x86_64 2021-02-10T11:00:23.426 INFO:teuthology.orchestra.run.gibba013.stdout:---> Package flex.x86_64 0:2.5.37-6.el7 will be installed 2021-02-10T11:00:23.426 INFO:teuthology.orchestra.run.gibba013.stdout:---> Package libcephfs-devel.x86_64 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:00:23.426 INFO:teuthology.orchestra.run.gibba013.stdout:--> Processing Dependency: librados-devel = 2:14.2.16-280.g962fac13.el7 for package: 2:libcephfs-devel-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:00:23.427 INFO:teuthology.orchestra.run.gibba013.stdout:---> Package libcephfs2.x86_64 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:00:23.428 INFO:teuthology.orchestra.run.gibba013.stdout:---> Package librados2.x86_64 1:10.2.5-4.el7 will be updated 2021-02-10T11:00:23.428 INFO:teuthology.orchestra.run.gibba013.stdout:---> Package librados2.x86_64 2:14.2.16-280.g962fac13.el7 will be an update 2021-02-10T11:00:23.430 INFO:teuthology.orchestra.run.gibba013.stdout:--> Processing Dependency: liblttng-ust.so.0()(64bit) for package: 2:librados2-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:00:23.430 INFO:teuthology.orchestra.run.gibba013.stdout:---> Package librbd1.x86_64 1:10.2.5-4.el7 will be updated 2021-02-10T11:00:23.431 INFO:teuthology.orchestra.run.gibba013.stdout:---> Package librbd1.x86_64 2:14.2.16-280.g962fac13.el7 will be an update 2021-02-10T11:00:23.431 INFO:teuthology.orchestra.run.gibba013.stdout:---> Package openssl-devel.x86_64 1:1.0.2k-21.el7_9 will be installed 2021-02-10T11:00:23.432 INFO:teuthology.orchestra.run.gibba013.stdout:--> Processing Dependency: openssl-libs(x86-64) = 1:1.0.2k-21.el7_9 for package: 1:openssl-devel-1.0.2k-21.el7_9.x86_64 2021-02-10T11:00:23.436 INFO:teuthology.orchestra.run.gibba013.stdout:--> Processing Dependency: krb5-devel(x86-64) for package: 1:openssl-devel-1.0.2k-21.el7_9.x86_64 2021-02-10T11:00:23.441 INFO:teuthology.orchestra.run.gibba013.stdout:---> Package python-ceph-compat.x86_64 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:00:23.442 INFO:teuthology.orchestra.run.gibba013.stdout:--> Processing Dependency: python-rados = 2:14.2.16-280.g962fac13.el7 for package: 2:python-ceph-compat-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:00:23.442 INFO:teuthology.orchestra.run.gibba013.stdout:--> Processing Dependency: python-cephfs = 2:14.2.16-280.g962fac13.el7 for package: 2:python-ceph-compat-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:00:23.443 INFO:teuthology.orchestra.run.gibba013.stdout:--> Processing Dependency: python-rgw = 2:14.2.16-280.g962fac13.el7 for package: 2:python-ceph-compat-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:00:23.444 INFO:teuthology.orchestra.run.gibba013.stdout:--> Processing Dependency: python-rbd = 2:14.2.16-280.g962fac13.el7 for package: 2:python-ceph-compat-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:00:23.444 INFO:teuthology.orchestra.run.gibba013.stdout:---> Package python3-cephfs.x86_64 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:00:23.446 INFO:teuthology.orchestra.run.gibba013.stdout:--> Processing Dependency: python3-ceph-argparse = 2:14.2.16-280.g962fac13.el7 for package: 2:python3-cephfs-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:00:23.446 INFO:teuthology.orchestra.run.gibba013.stdout:--> Processing Dependency: python3-rados = 2:14.2.16-280.g962fac13.el7 for package: 2:python3-cephfs-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:00:23.447 INFO:teuthology.orchestra.run.gibba013.stdout:---> Package rbd-fuse.x86_64 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:00:23.448 INFO:teuthology.orchestra.run.gibba013.stdout:--> Running transaction check 2021-02-10T11:00:23.448 INFO:teuthology.orchestra.run.gibba013.stdout:---> Package ceph-base.x86_64 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:00:23.452 INFO:teuthology.orchestra.run.gibba013.stdout:---> Package ceph-common.x86_64 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:00:23.454 INFO:teuthology.orchestra.run.gibba013.stdout:--> Processing Dependency: python-ceph-argparse = 2:14.2.16-280.g962fac13.el7 for package: 2:ceph-common-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:00:23.455 INFO:teuthology.orchestra.run.gibba013.stdout:--> Processing Dependency: python-prettytable for package: 2:ceph-common-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:00:23.456 INFO:teuthology.orchestra.run.gibba013.stdout:--> Processing Dependency: librdkafka.so.1()(64bit) for package: 2:ceph-common-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:00:23.457 INFO:teuthology.orchestra.run.gibba013.stdout:--> Processing Dependency: libbabeltrace-ctf.so.1()(64bit) for package: 2:ceph-common-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:00:23.458 INFO:teuthology.orchestra.run.gibba013.stdout:--> Processing Dependency: libbabeltrace.so.1()(64bit) for package: 2:ceph-common-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:00:23.458 INFO:teuthology.orchestra.run.gibba013.stdout:---> Package ceph-grafana-dashboards.noarch 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:00:23.458 INFO:teuthology.orchestra.run.gibba013.stdout:---> Package ceph-mds.x86_64 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:00:23.458 INFO:teuthology.orchestra.run.gibba013.stdout:---> Package ceph-mon.x86_64 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:00:23.459 INFO:teuthology.orchestra.run.gibba013.stdout:---> Package ceph-osd.x86_64 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:00:23.461 INFO:teuthology.orchestra.run.gibba013.stdout:---> Package ceph-selinux.x86_64 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:00:23.463 INFO:teuthology.orchestra.run.gibba013.stdout:---> Package jq.x86_64 0:1.6-2.el7 will be installed 2021-02-10T11:00:23.463 INFO:teuthology.orchestra.run.gibba013.stdout:--> Processing Dependency: libonig.so.5()(64bit) for package: jq-1.6-2.el7.x86_64 2021-02-10T11:00:23.463 INFO:teuthology.orchestra.run.gibba013.stdout:---> Package krb5-devel.x86_64 0:1.15.1-50.el7 will be installed 2021-02-10T11:00:23.465 INFO:teuthology.orchestra.run.gibba013.stdout:--> Processing Dependency: libverto-devel for package: krb5-devel-1.15.1-50.el7.x86_64 2021-02-10T11:00:23.467 INFO:teuthology.orchestra.run.gibba013.stdout:--> Processing Dependency: libselinux-devel for package: krb5-devel-1.15.1-50.el7.x86_64 2021-02-10T11:00:23.473 INFO:teuthology.orchestra.run.gibba013.stdout:--> Processing Dependency: libcom_err-devel for package: krb5-devel-1.15.1-50.el7.x86_64 2021-02-10T11:00:23.483 INFO:teuthology.orchestra.run.gibba013.stdout:--> Processing Dependency: keyutils-libs-devel for package: krb5-devel-1.15.1-50.el7.x86_64 2021-02-10T11:00:23.485 INFO:teuthology.orchestra.run.gibba013.stdout:---> Package liboath.x86_64 0:2.6.2-1.el7 will be installed 2021-02-10T11:00:23.485 INFO:teuthology.orchestra.run.gibba013.stdout:---> Package librabbitmq.x86_64 0:0.8.0-3.el7 will be installed 2021-02-10T11:00:23.485 INFO:teuthology.orchestra.run.gibba013.stdout:---> Package librados-devel.x86_64 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:00:23.486 INFO:teuthology.orchestra.run.gibba013.stdout:---> Package libradosstriper1.x86_64 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:00:23.486 INFO:teuthology.orchestra.run.gibba013.stdout:---> Package librgw2.x86_64 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:00:23.487 INFO:teuthology.orchestra.run.gibba013.stdout:---> Package lttng-ust.x86_64 0:2.4.1-4.el7 will be installed 2021-02-10T11:00:23.488 INFO:teuthology.orchestra.run.gibba013.stdout:--> Processing Dependency: liburcu-bp.so.1()(64bit) for package: lttng-ust-2.4.1-4.el7.x86_64 2021-02-10T11:00:23.488 INFO:teuthology.orchestra.run.gibba013.stdout:--> Processing Dependency: liburcu-cds.so.1()(64bit) for package: lttng-ust-2.4.1-4.el7.x86_64 2021-02-10T11:00:23.488 INFO:teuthology.orchestra.run.gibba013.stdout:---> Package openssl-libs.x86_64 1:1.0.2k-19.el7 will be updated 2021-02-10T11:00:23.489 INFO:teuthology.orchestra.run.gibba013.stdout:--> Processing Dependency: openssl-libs(x86-64) = 1:1.0.2k-19.el7 for package: 1:openssl-1.0.2k-19.el7.x86_64 2021-02-10T11:00:23.490 INFO:teuthology.orchestra.run.gibba013.stdout:---> Package openssl-libs.x86_64 1:1.0.2k-21.el7_9 will be an update 2021-02-10T11:00:23.491 INFO:teuthology.orchestra.run.gibba013.stdout:---> Package python-cephfs.x86_64 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:00:23.491 INFO:teuthology.orchestra.run.gibba013.stdout:---> Package python-cherrypy.noarch 0:3.2.2-4.el7 will be installed 2021-02-10T11:00:23.500 INFO:teuthology.orchestra.run.gibba013.stdout:---> Package python-pecan.noarch 0:0.4.5-2.el7 will be installed 2021-02-10T11:00:23.500 INFO:teuthology.orchestra.run.gibba013.stdout:--> Processing Dependency: python-mako >= 0.4.0 for package: python-pecan-0.4.5-2.el7.noarch 2021-02-10T11:00:23.501 INFO:teuthology.orchestra.run.gibba013.stdout:--> Processing Dependency: python-simplegeneric >= 0.8 for package: python-pecan-0.4.5-2.el7.noarch 2021-02-10T11:00:23.501 INFO:teuthology.orchestra.run.gibba013.stdout:--> Processing Dependency: python-webob >= 1.2 for package: python-pecan-0.4.5-2.el7.noarch 2021-02-10T11:00:23.503 INFO:teuthology.orchestra.run.gibba013.stdout:--> Processing Dependency: python-webtest >= 1.3.1 for package: python-pecan-0.4.5-2.el7.noarch 2021-02-10T11:00:23.503 INFO:teuthology.orchestra.run.gibba013.stdout:--> Processing Dependency: python-singledispatch for package: python-pecan-0.4.5-2.el7.noarch 2021-02-10T11:00:23.504 INFO:teuthology.orchestra.run.gibba013.stdout:---> Package python-rados.x86_64 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:00:23.504 INFO:teuthology.orchestra.run.gibba013.stdout:---> Package python-rbd.x86_64 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:00:23.505 INFO:teuthology.orchestra.run.gibba013.stdout:---> Package python-requests.noarch 0:2.6.0-10.el7 will be installed 2021-02-10T11:00:23.505 INFO:teuthology.orchestra.run.gibba013.stdout:--> Processing Dependency: python-urllib3 >= 1.10.2-1 for package: python-requests-2.6.0-10.el7.noarch 2021-02-10T11:00:23.506 INFO:teuthology.orchestra.run.gibba013.stdout:---> Package python-rgw.x86_64 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:00:23.507 INFO:teuthology.orchestra.run.gibba013.stdout:---> Package python-routes.noarch 0:1.13-2.el7 will be installed 2021-02-10T11:00:23.507 INFO:teuthology.orchestra.run.gibba013.stdout:--> Processing Dependency: python-repoze-lru for package: python-routes-1.13-2.el7.noarch 2021-02-10T11:00:23.507 INFO:teuthology.orchestra.run.gibba013.stdout:---> Package python-werkzeug.noarch 0:0.9.1-2.el7 will be installed 2021-02-10T11:00:23.508 INFO:teuthology.orchestra.run.gibba013.stdout:---> Package python2-bcrypt.x86_64 0:3.1.6-2.el7 will be installed 2021-02-10T11:00:23.508 INFO:teuthology.orchestra.run.gibba013.stdout:--> Processing Dependency: python2-six for package: python2-bcrypt-3.1.6-2.el7.x86_64 2021-02-10T11:00:23.508 INFO:teuthology.orchestra.run.gibba013.stdout:---> Package python2-kubernetes.noarch 1:11.0.0-6.el7 will be installed 2021-02-10T11:00:23.509 INFO:teuthology.orchestra.run.gibba013.stdout:--> Processing Dependency: python-certifi for package: 1:python2-kubernetes-11.0.0-6.el7.noarch 2021-02-10T11:00:23.510 INFO:teuthology.orchestra.run.gibba013.stdout:--> Processing Dependency: python-google-auth for package: 1:python2-kubernetes-11.0.0-6.el7.noarch 2021-02-10T11:00:23.510 INFO:teuthology.orchestra.run.gibba013.stdout:--> Processing Dependency: python-requests-oauthlib for package: 1:python2-kubernetes-11.0.0-6.el7.noarch 2021-02-10T11:00:23.511 INFO:teuthology.orchestra.run.gibba013.stdout:--> Processing Dependency: python-websocket-client for package: 1:python2-kubernetes-11.0.0-6.el7.noarch 2021-02-10T11:00:23.514 INFO:teuthology.orchestra.run.gibba013.stdout:---> Package python2-remoto.noarch 0:1.0.0-1.el7 will be installed 2021-02-10T11:00:23.514 INFO:teuthology.orchestra.run.gibba013.stdout:--> Processing Dependency: python2-execnet >= 1.2.0 for package: python2-remoto-1.0.0-1.el7.noarch 2021-02-10T11:00:23.515 INFO:teuthology.orchestra.run.gibba013.stdout:---> Package python3-ceph-argparse.x86_64 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:00:23.515 INFO:teuthology.orchestra.run.gibba013.stdout:---> Package python3-rados.x86_64 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:00:23.515 INFO:teuthology.orchestra.run.gibba013.stdout:---> Package scipy.x86_64 0:0.12.1-6.el7 will be installed 2021-02-10T11:00:23.517 INFO:teuthology.orchestra.run.gibba013.stdout:--> Processing Dependency: f2py for package: scipy-0.12.1-6.el7.x86_64 2021-02-10T11:00:23.519 INFO:teuthology.orchestra.run.gibba013.stdout:--> Processing Dependency: libumfpack.so.5()(64bit) for package: scipy-0.12.1-6.el7.x86_64 2021-02-10T11:00:23.520 INFO:teuthology.orchestra.run.gibba013.stdout:--> Processing Dependency: libamd.so.2()(64bit) for package: scipy-0.12.1-6.el7.x86_64 2021-02-10T11:00:23.520 INFO:teuthology.orchestra.run.gibba013.stdout:---> Package socat.x86_64 0:1.7.3.2-2.el7 will be installed 2021-02-10T11:00:23.521 INFO:teuthology.orchestra.run.gibba013.stdout:---> Package zlib-devel.x86_64 0:1.2.7-19.el7_9 will be installed 2021-02-10T11:00:23.522 INFO:teuthology.orchestra.run.gibba013.stdout:--> Processing Dependency: zlib = 1.2.7-19.el7_9 for package: zlib-devel-1.2.7-19.el7_9.x86_64 2021-02-10T11:00:23.524 INFO:teuthology.orchestra.run.gibba013.stdout:--> Running transaction check 2021-02-10T11:00:23.524 INFO:teuthology.orchestra.run.gibba013.stdout:---> Package keyutils-libs-devel.x86_64 0:1.5.8-3.el7 will be installed 2021-02-10T11:00:23.524 INFO:teuthology.orchestra.run.gibba013.stdout:---> Package libbabeltrace.x86_64 0:1.2.4-3.el7 will be installed 2021-02-10T11:00:23.525 INFO:teuthology.orchestra.run.gibba013.stdout:---> Package libcom_err-devel.x86_64 0:1.42.9-19.el7 will be installed 2021-02-10T11:00:23.525 INFO:teuthology.orchestra.run.gibba013.stdout:---> Package librdkafka.x86_64 0:0.11.5-1.el7 will be installed 2021-02-10T11:00:23.526 INFO:teuthology.orchestra.run.gibba013.stdout:---> Package libselinux-devel.x86_64 0:2.5-15.el7 will be installed 2021-02-10T11:00:23.526 INFO:teuthology.orchestra.run.gibba013.stdout:--> Processing Dependency: libsepol-devel(x86-64) >= 2.5-10 for package: libselinux-devel-2.5-15.el7.x86_64 2021-02-10T11:00:23.528 INFO:teuthology.orchestra.run.gibba013.stdout:--> Processing Dependency: pkgconfig(libsepol) for package: libselinux-devel-2.5-15.el7.x86_64 2021-02-10T11:00:23.528 INFO:teuthology.orchestra.run.gibba013.stdout:--> Processing Dependency: pkgconfig(libpcre) for package: libselinux-devel-2.5-15.el7.x86_64 2021-02-10T11:00:23.533 INFO:teuthology.orchestra.run.gibba013.stdout:---> Package libverto-devel.x86_64 0:0.2.5-4.el7 will be installed 2021-02-10T11:00:23.533 INFO:teuthology.orchestra.run.gibba013.stdout:---> Package numpy-f2py.x86_64 1:1.7.1-13.el7 will be installed 2021-02-10T11:00:23.533 INFO:teuthology.orchestra.run.gibba013.stdout:---> Package oniguruma.x86_64 0:6.8.2-1.el7 will be installed 2021-02-10T11:00:23.533 INFO:teuthology.orchestra.run.gibba013.stdout:---> Package openssl.x86_64 1:1.0.2k-19.el7 will be updated 2021-02-10T11:00:23.534 INFO:teuthology.orchestra.run.gibba013.stdout:---> Package openssl.x86_64 1:1.0.2k-21.el7_9 will be an update 2021-02-10T11:00:23.535 INFO:teuthology.orchestra.run.gibba013.stdout:---> Package python-ceph-argparse.x86_64 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:00:23.536 INFO:teuthology.orchestra.run.gibba013.stdout:---> Package python-execnet.noarch 0:1.7.1-3.el7 will be installed 2021-02-10T11:00:23.536 INFO:teuthology.orchestra.run.gibba013.stdout:--> Processing Dependency: python-apipkg for package: python-execnet-1.7.1-3.el7.noarch 2021-02-10T11:00:23.536 INFO:teuthology.orchestra.run.gibba013.stdout:---> Package python-mako.noarch 0:0.8.1-2.el7 will be installed 2021-02-10T11:00:23.536 INFO:teuthology.orchestra.run.gibba013.stdout:--> Processing Dependency: python-markupsafe for package: python-mako-0.8.1-2.el7.noarch 2021-02-10T11:00:23.537 INFO:teuthology.orchestra.run.gibba013.stdout:--> Processing Dependency: python-beaker for package: python-mako-0.8.1-2.el7.noarch 2021-02-10T11:00:23.538 INFO:teuthology.orchestra.run.gibba013.stdout:---> Package python-prettytable.noarch 0:0.7.2-3.el7 will be installed 2021-02-10T11:00:23.538 INFO:teuthology.orchestra.run.gibba013.stdout:---> Package python-repoze-lru.noarch 0:0.4-3.el7 will be installed 2021-02-10T11:00:23.538 INFO:teuthology.orchestra.run.gibba013.stdout:---> Package python-simplegeneric.noarch 0:0.8-7.el7 will be installed 2021-02-10T11:00:23.538 INFO:teuthology.orchestra.run.gibba013.stdout:---> Package python-singledispatch.noarch 0:3.4.0.2-2.el7 will be installed 2021-02-10T11:00:23.538 INFO:teuthology.orchestra.run.gibba013.stdout:---> Package python-urllib3.noarch 0:1.10.2-7.el7 will be installed 2021-02-10T11:00:23.539 INFO:teuthology.orchestra.run.gibba013.stdout:---> Package python-webob.noarch 0:1.2.3-7.el7 will be installed 2021-02-10T11:00:23.539 INFO:teuthology.orchestra.run.gibba013.stdout:---> Package python-websocket-client.noarch 0:0.56.0-3.git3c25814.el7 will be installed 2021-02-10T11:00:23.539 INFO:teuthology.orchestra.run.gibba013.stdout:---> Package python-webtest.noarch 0:1.3.4-6.el7 will be installed 2021-02-10T11:00:23.539 INFO:teuthology.orchestra.run.gibba013.stdout:---> Package python2-certifi.noarch 0:2018.10.15-5.el7 will be installed 2021-02-10T11:00:23.539 INFO:teuthology.orchestra.run.gibba013.stdout:---> Package python2-google-auth.noarch 1:1.1.1-5.el7 will be installed 2021-02-10T11:00:23.539 INFO:teuthology.orchestra.run.gibba013.stdout:--> Processing Dependency: python-cachetools for package: 1:python2-google-auth-1.1.1-5.el7.noarch 2021-02-10T11:00:23.540 INFO:teuthology.orchestra.run.gibba013.stdout:--> Processing Dependency: python2-pyasn1-modules for package: 1:python2-google-auth-1.1.1-5.el7.noarch 2021-02-10T11:00:23.541 INFO:teuthology.orchestra.run.gibba013.stdout:--> Processing Dependency: python2-rsa for package: 1:python2-google-auth-1.1.1-5.el7.noarch 2021-02-10T11:00:23.541 INFO:teuthology.orchestra.run.gibba013.stdout:---> Package python2-requests-oauthlib.noarch 0:0.8.0-5.el7 will be installed 2021-02-10T11:00:23.541 INFO:teuthology.orchestra.run.gibba013.stdout:--> Processing Dependency: python2-oauthlib for package: python2-requests-oauthlib-0.8.0-5.el7.noarch 2021-02-10T11:00:23.542 INFO:teuthology.orchestra.run.gibba013.stdout:---> Package python2-six.noarch 0:1.9.0-0.el7 will be installed 2021-02-10T11:00:23.542 INFO:teuthology.orchestra.run.gibba013.stdout:---> Package suitesparse.x86_64 0:4.0.2-10.el7 will be installed 2021-02-10T11:00:23.543 INFO:teuthology.orchestra.run.gibba013.stdout:--> Processing Dependency: libtbbmalloc.so.2()(64bit) for package: suitesparse-4.0.2-10.el7.x86_64 2021-02-10T11:00:23.544 INFO:teuthology.orchestra.run.gibba013.stdout:--> Processing Dependency: libtbb.so.2()(64bit) for package: suitesparse-4.0.2-10.el7.x86_64 2021-02-10T11:00:23.544 INFO:teuthology.orchestra.run.gibba013.stdout:---> Package userspace-rcu.x86_64 0:0.7.16-1.el7 will be installed 2021-02-10T11:00:23.545 INFO:teuthology.orchestra.run.gibba013.stdout:---> Package zlib.x86_64 0:1.2.7-18.el7 will be updated 2021-02-10T11:00:23.546 INFO:teuthology.orchestra.run.gibba013.stdout:---> Package zlib.x86_64 0:1.2.7-19.el7_9 will be an update 2021-02-10T11:00:23.546 INFO:teuthology.orchestra.run.gibba013.stdout:--> Running transaction check 2021-02-10T11:00:23.546 INFO:teuthology.orchestra.run.gibba013.stdout:---> Package libsepol-devel.x86_64 0:2.5-10.el7 will be installed 2021-02-10T11:00:23.547 INFO:teuthology.orchestra.run.gibba013.stdout:---> Package pcre-devel.x86_64 0:8.32-17.el7 will be installed 2021-02-10T11:00:23.547 INFO:teuthology.orchestra.run.gibba013.stdout:---> Package python-beaker.noarch 0:1.5.4-10.el7 will be installed 2021-02-10T11:00:23.547 INFO:teuthology.orchestra.run.gibba013.stdout:--> Processing Dependency: python-paste for package: python-beaker-1.5.4-10.el7.noarch 2021-02-10T11:00:23.548 INFO:teuthology.orchestra.run.gibba013.stdout:---> Package python-cachetools.noarch 0:1.0.3-1.el7 will be installed 2021-02-10T11:00:23.548 INFO:teuthology.orchestra.run.gibba013.stdout:---> Package python-markupsafe.x86_64 0:0.11-10.el7 will be installed 2021-02-10T11:00:23.548 INFO:teuthology.orchestra.run.gibba013.stdout:---> Package python2-apipkg.noarch 0:1.5-5.el7 will be installed 2021-02-10T11:00:23.548 INFO:teuthology.orchestra.run.gibba013.stdout:---> Package python2-oauthlib.noarch 0:2.0.1-8.el7 will be installed 2021-02-10T11:00:23.549 INFO:teuthology.orchestra.run.gibba013.stdout:--> Processing Dependency: python-jwcrypto for package: python2-oauthlib-2.0.1-8.el7.noarch 2021-02-10T11:00:23.550 INFO:teuthology.orchestra.run.gibba013.stdout:---> Package python2-pyasn1-modules.noarch 0:0.1.9-7.el7 will be installed 2021-02-10T11:00:23.550 INFO:teuthology.orchestra.run.gibba013.stdout:---> Package python2-rsa.noarch 0:3.4.2-2.el7 will be installed 2021-02-10T11:00:23.550 INFO:teuthology.orchestra.run.gibba013.stdout:---> Package tbb.x86_64 0:4.1-9.20130314.el7 will be installed 2021-02-10T11:00:23.551 INFO:teuthology.orchestra.run.gibba013.stdout:--> Running transaction check 2021-02-10T11:00:23.551 INFO:teuthology.orchestra.run.gibba013.stdout:---> Package python-jwcrypto.noarch 0:0.4.2-1.el7 will be installed 2021-02-10T11:00:23.551 INFO:teuthology.orchestra.run.gibba013.stdout:---> Package python-paste.noarch 0:1.7.5.1-9.20111221hg1498.el7 will be installed 2021-02-10T11:00:23.551 INFO:teuthology.orchestra.run.gibba013.stdout:--> Processing Dependency: python-tempita for package: python-paste-1.7.5.1-9.20111221hg1498.el7.noarch 2021-02-10T11:00:23.551 INFO:teuthology.orchestra.run.gibba013.stdout:--> Running transaction check 2021-02-10T11:00:23.551 INFO:teuthology.orchestra.run.gibba013.stdout:---> Package python-tempita.noarch 0:0.5.1-6.el7 will be installed 2021-02-10T11:00:23.645 INFO:teuthology.orchestra.run.gibba002.stdout:-------------------------------------------------------------------------------- 2021-02-10T11:00:23.645 INFO:teuthology.orchestra.run.gibba002.stdout:Total 26 MB/s | 130 MB 00:04 2021-02-10T11:00:23.679 INFO:teuthology.orchestra.run.gibba002.stdout:Running transaction check 2021-02-10T11:00:23.808 INFO:teuthology.orchestra.run.gibba002.stdout:Running transaction test 2021-02-10T11:00:24.142 INFO:teuthology.orchestra.run.gibba013.stdout:--> Finished Dependency Resolution 2021-02-10T11:00:24.198 INFO:teuthology.orchestra.run.gibba013.stdout: 2021-02-10T11:00:24.199 INFO:teuthology.orchestra.run.gibba013.stdout:Dependencies Resolved 2021-02-10T11:00:24.207 INFO:teuthology.orchestra.run.gibba013.stdout: 2021-02-10T11:00:24.207 INFO:teuthology.orchestra.run.gibba013.stdout:================================================================================ 2021-02-10T11:00:24.207 INFO:teuthology.orchestra.run.gibba013.stdout: Package Arch Version Repository Size 2021-02-10T11:00:24.208 INFO:teuthology.orchestra.run.gibba013.stdout:================================================================================ 2021-02-10T11:00:24.208 INFO:teuthology.orchestra.run.gibba013.stdout:Installing: 2021-02-10T11:00:24.208 INFO:teuthology.orchestra.run.gibba013.stdout: bison x86_64 3.0.4-2.el7 rhel-7-server-rpms 674 k 2021-02-10T11:00:24.208 INFO:teuthology.orchestra.run.gibba013.stdout: ceph x86_64 2:14.2.16-280.g962fac13.el7 ceph 2.0 k 2021-02-10T11:00:24.208 INFO:teuthology.orchestra.run.gibba013.stdout: ceph-fuse x86_64 2:14.2.16-280.g962fac13.el7 ceph 571 k 2021-02-10T11:00:24.208 INFO:teuthology.orchestra.run.gibba013.stdout: ceph-mgr x86_64 2:14.2.16-280.g962fac13.el7 ceph 1.8 M 2021-02-10T11:00:24.208 INFO:teuthology.orchestra.run.gibba013.stdout: ceph-mgr-dashboard noarch 2:14.2.16-280.g962fac13.el7 ceph-noarch 4.0 M 2021-02-10T11:00:24.208 INFO:teuthology.orchestra.run.gibba013.stdout: ceph-mgr-diskprediction-cloud 2021-02-10T11:00:24.208 INFO:teuthology.orchestra.run.gibba013.stdout: noarch 2:14.2.16-280.g962fac13.el7 ceph-noarch 83 k 2021-02-10T11:00:24.209 INFO:teuthology.orchestra.run.gibba013.stdout: ceph-mgr-diskprediction-local 2021-02-10T11:00:24.209 INFO:teuthology.orchestra.run.gibba013.stdout: noarch 2:14.2.16-280.g962fac13.el7 ceph-noarch 1.1 M 2021-02-10T11:00:24.209 INFO:teuthology.orchestra.run.gibba013.stdout: ceph-mgr-rook noarch 2:14.2.16-280.g962fac13.el7 ceph-noarch 21 k 2021-02-10T11:00:24.209 INFO:teuthology.orchestra.run.gibba013.stdout: ceph-mgr-ssh noarch 2:14.2.16-280.g962fac13.el7 ceph-noarch 21 k 2021-02-10T11:00:24.209 INFO:teuthology.orchestra.run.gibba013.stdout: ceph-radosgw x86_64 2:14.2.16-280.g962fac13.el7 ceph 5.8 M 2021-02-10T11:00:24.209 INFO:teuthology.orchestra.run.gibba013.stdout: ceph-test x86_64 2:14.2.16-280.g962fac13.el7 ceph 34 M 2021-02-10T11:00:24.209 INFO:teuthology.orchestra.run.gibba013.stdout: elfutils-libelf-devel 2021-02-10T11:00:24.209 INFO:teuthology.orchestra.run.gibba013.stdout: x86_64 0.176-5.el7 rhel-7-server-rpms 40 k 2021-02-10T11:00:24.209 INFO:teuthology.orchestra.run.gibba013.stdout: flex x86_64 2.5.37-6.el7 rhel-7-server-rpms 293 k 2021-02-10T11:00:24.209 INFO:teuthology.orchestra.run.gibba013.stdout: libcephfs-devel x86_64 2:14.2.16-280.g962fac13.el7 ceph 17 k 2021-02-10T11:00:24.210 INFO:teuthology.orchestra.run.gibba013.stdout: libcephfs2 x86_64 2:14.2.16-280.g962fac13.el7 ceph 500 k 2021-02-10T11:00:24.210 INFO:teuthology.orchestra.run.gibba013.stdout: openssl-devel x86_64 1:1.0.2k-21.el7_9 rhel-7-server-rpms 1.5 M 2021-02-10T11:00:24.210 INFO:teuthology.orchestra.run.gibba013.stdout: python-ceph-compat x86_64 2:14.2.16-280.g962fac13.el7 ceph 2.2 k 2021-02-10T11:00:24.210 INFO:teuthology.orchestra.run.gibba013.stdout: python3-cephfs x86_64 2:14.2.16-280.g962fac13.el7 ceph 151 k 2021-02-10T11:00:24.210 INFO:teuthology.orchestra.run.gibba013.stdout: rbd-fuse x86_64 2:14.2.16-280.g962fac13.el7 ceph 69 k 2021-02-10T11:00:24.210 INFO:teuthology.orchestra.run.gibba013.stdout:Updating: 2021-02-10T11:00:24.210 INFO:teuthology.orchestra.run.gibba013.stdout: librados2 x86_64 2:14.2.16-280.g962fac13.el7 ceph 3.4 M 2021-02-10T11:00:24.210 INFO:teuthology.orchestra.run.gibba013.stdout: librbd1 x86_64 2:14.2.16-280.g962fac13.el7 ceph 1.7 M 2021-02-10T11:00:24.210 INFO:teuthology.orchestra.run.gibba013.stdout:Installing for dependencies: 2021-02-10T11:00:24.210 INFO:teuthology.orchestra.run.gibba013.stdout: ceph-base x86_64 2:14.2.16-280.g962fac13.el7 ceph 5.4 M 2021-02-10T11:00:24.211 INFO:teuthology.orchestra.run.gibba013.stdout: ceph-common x86_64 2:14.2.16-280.g962fac13.el7 ceph 19 M 2021-02-10T11:00:24.211 INFO:teuthology.orchestra.run.gibba013.stdout: ceph-grafana-dashboards 2021-02-10T11:00:24.211 INFO:teuthology.orchestra.run.gibba013.stdout: noarch 2:14.2.16-280.g962fac13.el7 ceph-noarch 19 k 2021-02-10T11:00:24.211 INFO:teuthology.orchestra.run.gibba013.stdout: ceph-mds x86_64 2:14.2.16-280.g962fac13.el7 ceph 1.9 M 2021-02-10T11:00:24.211 INFO:teuthology.orchestra.run.gibba013.stdout: ceph-mon x86_64 2:14.2.16-280.g962fac13.el7 ceph 4.3 M 2021-02-10T11:00:24.211 INFO:teuthology.orchestra.run.gibba013.stdout: ceph-osd x86_64 2:14.2.16-280.g962fac13.el7 ceph 16 M 2021-02-10T11:00:24.211 INFO:teuthology.orchestra.run.gibba013.stdout: ceph-selinux x86_64 2:14.2.16-280.g962fac13.el7 ceph 20 k 2021-02-10T11:00:24.211 INFO:teuthology.orchestra.run.gibba013.stdout: jq x86_64 1.6-2.el7 epel 167 k 2021-02-10T11:00:24.211 INFO:teuthology.orchestra.run.gibba013.stdout: keyutils-libs-devel 2021-02-10T11:00:24.211 INFO:teuthology.orchestra.run.gibba013.stdout: x86_64 1.5.8-3.el7 rhel-7-server-rpms 37 k 2021-02-10T11:00:24.211 INFO:teuthology.orchestra.run.gibba013.stdout: krb5-devel x86_64 1.15.1-50.el7 rhel-7-server-rpms 273 k 2021-02-10T11:00:24.212 INFO:teuthology.orchestra.run.gibba013.stdout: libbabeltrace x86_64 1.2.4-3.el7 epel 147 k 2021-02-10T11:00:24.212 INFO:teuthology.orchestra.run.gibba013.stdout: libcom_err-devel x86_64 1.42.9-19.el7 rhel-7-server-rpms 32 k 2021-02-10T11:00:24.212 INFO:teuthology.orchestra.run.gibba013.stdout: liboath x86_64 2.6.2-1.el7 epel 51 k 2021-02-10T11:00:24.212 INFO:teuthology.orchestra.run.gibba013.stdout: librabbitmq x86_64 0.8.0-3.el7 rhel-7-server-rpms 37 k 2021-02-10T11:00:24.212 INFO:teuthology.orchestra.run.gibba013.stdout: librados-devel x86_64 2:14.2.16-280.g962fac13.el7 ceph 86 k 2021-02-10T11:00:24.212 INFO:teuthology.orchestra.run.gibba013.stdout: libradosstriper1 x86_64 2:14.2.16-280.g962fac13.el7 ceph 364 k 2021-02-10T11:00:24.212 INFO:teuthology.orchestra.run.gibba013.stdout: librdkafka x86_64 0.11.5-1.el7 epel 368 k 2021-02-10T11:00:24.212 INFO:teuthology.orchestra.run.gibba013.stdout: librgw2 x86_64 2:14.2.16-280.g962fac13.el7 ceph 5.1 M 2021-02-10T11:00:24.212 INFO:teuthology.orchestra.run.gibba013.stdout: libselinux-devel x86_64 2.5-15.el7 rhel-7-server-rpms 187 k 2021-02-10T11:00:24.212 INFO:teuthology.orchestra.run.gibba013.stdout: libsepol-devel x86_64 2.5-10.el7 rhel-7-server-rpms 77 k 2021-02-10T11:00:24.212 INFO:teuthology.orchestra.run.gibba013.stdout: libverto-devel x86_64 0.2.5-4.el7 rhel-7-server-rpms 12 k 2021-02-10T11:00:24.213 INFO:teuthology.orchestra.run.gibba013.stdout: lttng-ust x86_64 2.4.1-4.el7 epel 176 k 2021-02-10T11:00:24.213 INFO:teuthology.orchestra.run.gibba013.stdout: numpy-f2py x86_64 1:1.7.1-13.el7 rhel-7-server-rpms 206 k 2021-02-10T11:00:24.213 INFO:teuthology.orchestra.run.gibba013.stdout: oniguruma x86_64 6.8.2-1.el7 epel 181 k 2021-02-10T11:00:24.213 INFO:teuthology.orchestra.run.gibba013.stdout: pcre-devel x86_64 8.32-17.el7 rhel-7-server-rpms 480 k 2021-02-10T11:00:24.213 INFO:teuthology.orchestra.run.gibba013.stdout: python-beaker noarch 1.5.4-10.el7 rhel-7-server-rpms 80 k 2021-02-10T11:00:24.213 INFO:teuthology.orchestra.run.gibba013.stdout: python-cachetools noarch 1.0.3-1.el7 epel 22 k 2021-02-10T11:00:24.213 INFO:teuthology.orchestra.run.gibba013.stdout: python-ceph-argparse 2021-02-10T11:00:24.213 INFO:teuthology.orchestra.run.gibba013.stdout: x86_64 2:14.2.16-280.g962fac13.el7 ceph 35 k 2021-02-10T11:00:24.213 INFO:teuthology.orchestra.run.gibba013.stdout: python-cephfs x86_64 2:14.2.16-280.g962fac13.el7 ceph 113 k 2021-02-10T11:00:24.213 INFO:teuthology.orchestra.run.gibba013.stdout: python-cherrypy noarch 3.2.2-4.el7 rhel-7-server-rpms 422 k 2021-02-10T11:00:24.213 INFO:teuthology.orchestra.run.gibba013.stdout: python-execnet noarch 1.7.1-3.el7 epel 260 k 2021-02-10T11:00:24.214 INFO:teuthology.orchestra.run.gibba013.stdout: python-jwcrypto noarch 0.4.2-1.el7 rhel-7-server-rpms 57 k 2021-02-10T11:00:24.214 INFO:teuthology.orchestra.run.gibba013.stdout: python-mako noarch 0.8.1-2.el7 rhel-7-server-rpms 307 k 2021-02-10T11:00:24.214 INFO:teuthology.orchestra.run.gibba013.stdout: python-markupsafe x86_64 0.11-10.el7 rhel-7-server-rpms 25 k 2021-02-10T11:00:24.214 INFO:teuthology.orchestra.run.gibba013.stdout: python-paste noarch 1.7.5.1-9.20111221hg1498.el7 2021-02-10T11:00:24.214 INFO:teuthology.orchestra.run.gibba013.stdout: rhel-7-server-rpms 866 k 2021-02-10T11:00:24.214 INFO:teuthology.orchestra.run.gibba013.stdout: python-pecan noarch 0.4.5-2.el7 epel 255 k 2021-02-10T11:00:24.214 INFO:teuthology.orchestra.run.gibba013.stdout: python-prettytable noarch 0.7.2-3.el7 rhel-7-server-rpms 37 k 2021-02-10T11:00:24.214 INFO:teuthology.orchestra.run.gibba013.stdout: python-rados x86_64 2:14.2.16-280.g962fac13.el7 ceph 199 k 2021-02-10T11:00:24.214 INFO:teuthology.orchestra.run.gibba013.stdout: python-rbd x86_64 2:14.2.16-280.g962fac13.el7 ceph 194 k 2021-02-10T11:00:24.214 INFO:teuthology.orchestra.run.gibba013.stdout: python-repoze-lru noarch 0.4-3.el7 epel 13 k 2021-02-10T11:00:24.214 INFO:teuthology.orchestra.run.gibba013.stdout: python-requests noarch 2.6.0-10.el7 rhel-7-server-rpms 95 k 2021-02-10T11:00:24.215 INFO:teuthology.orchestra.run.gibba013.stdout: python-rgw x86_64 2:14.2.16-280.g962fac13.el7 ceph 77 k 2021-02-10T11:00:24.215 INFO:teuthology.orchestra.run.gibba013.stdout: python-routes noarch 1.13-2.el7 epel 640 k 2021-02-10T11:00:24.215 INFO:teuthology.orchestra.run.gibba013.stdout: python-simplegeneric 2021-02-10T11:00:24.215 INFO:teuthology.orchestra.run.gibba013.stdout: noarch 0.8-7.el7 epel 12 k 2021-02-10T11:00:24.215 INFO:teuthology.orchestra.run.gibba013.stdout: python-singledispatch 2021-02-10T11:00:24.215 INFO:teuthology.orchestra.run.gibba013.stdout: noarch 3.4.0.2-2.el7 epel 18 k 2021-02-10T11:00:24.215 INFO:teuthology.orchestra.run.gibba013.stdout: python-tempita noarch 0.5.1-6.el7 rhel-7-server-rpms 33 k 2021-02-10T11:00:24.215 INFO:teuthology.orchestra.run.gibba013.stdout: python-urllib3 noarch 1.10.2-7.el7 rhel-7-server-rpms 103 k 2021-02-10T11:00:24.215 INFO:teuthology.orchestra.run.gibba013.stdout: python-webob noarch 1.2.3-7.el7 rhel-7-server-optional-rpms 2021-02-10T11:00:24.215 INFO:teuthology.orchestra.run.gibba013.stdout: 202 k 2021-02-10T11:00:24.215 INFO:teuthology.orchestra.run.gibba013.stdout: python-websocket-client 2021-02-10T11:00:24.215 INFO:teuthology.orchestra.run.gibba013.stdout: noarch 0.56.0-3.git3c25814.el7 rhel-7-server-extras-rpms 2021-02-10T11:00:24.216 INFO:teuthology.orchestra.run.gibba013.stdout: 66 k 2021-02-10T11:00:24.216 INFO:teuthology.orchestra.run.gibba013.stdout: python-webtest noarch 1.3.4-6.el7 rhel-7-server-optional-rpms 2021-02-10T11:00:24.216 INFO:teuthology.orchestra.run.gibba013.stdout: 102 k 2021-02-10T11:00:24.216 INFO:teuthology.orchestra.run.gibba013.stdout: python-werkzeug noarch 0.9.1-2.el7 rhel-7-server-extras-rpms 2021-02-10T11:00:24.216 INFO:teuthology.orchestra.run.gibba013.stdout: 562 k 2021-02-10T11:00:24.216 INFO:teuthology.orchestra.run.gibba013.stdout: python2-apipkg noarch 1.5-5.el7 epel 17 k 2021-02-10T11:00:24.216 INFO:teuthology.orchestra.run.gibba013.stdout: python2-bcrypt x86_64 3.1.6-2.el7 epel 39 k 2021-02-10T11:00:24.216 INFO:teuthology.orchestra.run.gibba013.stdout: python2-certifi noarch 2018.10.15-5.el7 epel 12 k 2021-02-10T11:00:24.216 INFO:teuthology.orchestra.run.gibba013.stdout: python2-google-auth 2021-02-10T11:00:24.216 INFO:teuthology.orchestra.run.gibba013.stdout: noarch 1:1.1.1-5.el7 epel 76 k 2021-02-10T11:00:24.216 INFO:teuthology.orchestra.run.gibba013.stdout: python2-kubernetes noarch 1:11.0.0-6.el7 epel 1.1 M 2021-02-10T11:00:24.217 INFO:teuthology.orchestra.run.gibba013.stdout: python2-oauthlib noarch 2.0.1-8.el7 rhel-7-server-rpms 146 k 2021-02-10T11:00:24.217 INFO:teuthology.orchestra.run.gibba013.stdout: python2-pyasn1-modules 2021-02-10T11:00:24.217 INFO:teuthology.orchestra.run.gibba013.stdout: noarch 0.1.9-7.el7 rhel-7-server-optional-rpms 2021-02-10T11:00:24.217 INFO:teuthology.orchestra.run.gibba013.stdout: 59 k 2021-02-10T11:00:24.217 INFO:teuthology.orchestra.run.gibba013.stdout: python2-remoto noarch 1.0.0-1.el7 epel 42 k 2021-02-10T11:00:24.217 INFO:teuthology.orchestra.run.gibba013.stdout: python2-requests-oauthlib 2021-02-10T11:00:24.217 INFO:teuthology.orchestra.run.gibba013.stdout: noarch 0.8.0-5.el7 rhel-7-server-rpms 35 k 2021-02-10T11:00:24.217 INFO:teuthology.orchestra.run.gibba013.stdout: python2-rsa noarch 3.4.2-2.el7 epel 68 k 2021-02-10T11:00:24.217 INFO:teuthology.orchestra.run.gibba013.stdout: python2-six noarch 1.9.0-0.el7 epel 2.9 k 2021-02-10T11:00:24.217 INFO:teuthology.orchestra.run.gibba013.stdout: python3-ceph-argparse 2021-02-10T11:00:24.217 INFO:teuthology.orchestra.run.gibba013.stdout: x86_64 2:14.2.16-280.g962fac13.el7 ceph 35 k 2021-02-10T11:00:24.217 INFO:teuthology.orchestra.run.gibba013.stdout: python3-rados x86_64 2:14.2.16-280.g962fac13.el7 ceph 317 k 2021-02-10T11:00:24.218 INFO:teuthology.orchestra.run.gibba013.stdout: scipy x86_64 0.12.1-6.el7 rhel-7-server-rpms 9.3 M 2021-02-10T11:00:24.218 INFO:teuthology.orchestra.run.gibba013.stdout: socat x86_64 1.7.3.2-2.el7 rhel-7-server-rpms 290 k 2021-02-10T11:00:24.218 INFO:teuthology.orchestra.run.gibba013.stdout: suitesparse x86_64 4.0.2-10.el7 rhel-7-server-rpms 928 k 2021-02-10T11:00:24.218 INFO:teuthology.orchestra.run.gibba013.stdout: tbb x86_64 4.1-9.20130314.el7 rhel-7-server-rpms 124 k 2021-02-10T11:00:24.218 INFO:teuthology.orchestra.run.gibba013.stdout: userspace-rcu x86_64 0.7.16-1.el7 epel 73 k 2021-02-10T11:00:24.218 INFO:teuthology.orchestra.run.gibba013.stdout: zlib-devel x86_64 1.2.7-19.el7_9 rhel-7-server-rpms 50 k 2021-02-10T11:00:24.218 INFO:teuthology.orchestra.run.gibba013.stdout:Updating for dependencies: 2021-02-10T11:00:24.218 INFO:teuthology.orchestra.run.gibba013.stdout: openssl x86_64 1:1.0.2k-21.el7_9 rhel-7-server-rpms 493 k 2021-02-10T11:00:24.218 INFO:teuthology.orchestra.run.gibba013.stdout: openssl-libs x86_64 1:1.0.2k-21.el7_9 rhel-7-server-rpms 1.2 M 2021-02-10T11:00:24.218 INFO:teuthology.orchestra.run.gibba013.stdout: zlib x86_64 1.2.7-19.el7_9 rhel-7-server-rpms 90 k 2021-02-10T11:00:24.218 INFO:teuthology.orchestra.run.gibba013.stdout: 2021-02-10T11:00:24.219 INFO:teuthology.orchestra.run.gibba013.stdout:Transaction Summary 2021-02-10T11:00:24.219 INFO:teuthology.orchestra.run.gibba013.stdout:================================================================================ 2021-02-10T11:00:24.219 INFO:teuthology.orchestra.run.gibba013.stdout:Install 19 Packages (+70 Dependent packages) 2021-02-10T11:00:24.219 INFO:teuthology.orchestra.run.gibba013.stdout:Upgrade 2 Packages (+ 3 Dependent packages) 2021-02-10T11:00:24.219 INFO:teuthology.orchestra.run.gibba013.stdout: 2021-02-10T11:00:24.220 INFO:teuthology.orchestra.run.gibba013.stdout:Total download size: 130 M 2021-02-10T11:00:24.220 INFO:teuthology.orchestra.run.gibba013.stdout:Downloading packages: 2021-02-10T11:00:24.220 INFO:teuthology.orchestra.run.gibba013.stdout:Delta RPMs disabled because /usr/bin/applydeltarpm not installed. 2021-02-10T11:00:24.973 INFO:teuthology.orchestra.run.gibba002.stdout:Transaction test succeeded 2021-02-10T11:00:24.973 INFO:teuthology.orchestra.run.gibba002.stdout:Running transaction 2021-02-10T11:00:25.690 INFO:teuthology.orchestra.run.gibba002.stdout: Updating : zlib-1.2.7-19.el7_9.x86_64 1/99 2021-02-10T11:00:26.044 INFO:teuthology.orchestra.run.gibba002.stdout: Updating : 1:openssl-libs-1.0.2k-21.el7_9.x86_64 2/99 2021-02-10T11:00:26.353 INFO:teuthology.orchestra.run.gibba002.stdout: Installing : liboath-2.6.2-1.el7.x86_64 3/99 2021-02-10T11:00:26.623 INFO:teuthology.orchestra.run.gibba002.stdout: Installing : librabbitmq-0.8.0-3.el7.x86_64 4/99 2021-02-10T11:00:26.876 INFO:teuthology.orchestra.run.gibba002.stdout: Installing : zlib-devel-1.2.7-19.el7_9.x86_64 5/99 2021-02-10T11:00:27.106 INFO:teuthology.orchestra.run.gibba002.stdout: Installing : python2-six-1.9.0-0.el7.noarch 6/99 2021-02-10T11:00:27.443 INFO:teuthology.orchestra.run.gibba002.stdout: Installing : 2:python-ceph-argparse-14.2.16-280.g962fac13.el7.x86_64 7/99 2021-02-10T11:00:27.776 INFO:teuthology.orchestra.run.gibba002.stdout: Installing : python-werkzeug-0.9.1-2.el7.noarch 8/99 2021-02-10T11:00:28.104 INFO:teuthology.orchestra.run.gibba002.stdout: Installing : python-cherrypy-3.2.2-4.el7.noarch 9/99 2021-02-10T11:00:28.421 INFO:teuthology.orchestra.run.gibba002.stdout: Installing : python-webob-1.2.3-7.el7.noarch 10/99 2021-02-10T11:00:28.748 INFO:teuthology.orchestra.run.gibba002.stdout: Installing : python-urllib3-1.10.2-7.el7.noarch 11/99 2021-02-10T11:00:28.862 INFO:teuthology.orchestra.run.gibba013.stdout:-------------------------------------------------------------------------------- 2021-02-10T11:00:28.862 INFO:teuthology.orchestra.run.gibba013.stdout:Total 28 MB/s | 130 MB 00:04 2021-02-10T11:00:28.896 INFO:teuthology.orchestra.run.gibba013.stdout:Running transaction check 2021-02-10T11:00:29.026 INFO:teuthology.orchestra.run.gibba013.stdout:Running transaction test 2021-02-10T11:00:29.043 INFO:teuthology.orchestra.run.gibba002.stdout: Installing : python-requests-2.6.0-10.el7.noarch 12/99 2021-02-10T11:00:29.507 INFO:teuthology.orchestra.run.gibba002.stdout: Installing : python-webtest-1.3.4-6.el7.noarch 13/99 2021-02-10T11:00:29.621 INFO:teuthology.orchestra.run.gibba013.stdout:Transaction test succeeded 2021-02-10T11:00:29.621 INFO:teuthology.orchestra.run.gibba013.stdout:Running transaction 2021-02-10T11:00:29.786 INFO:teuthology.orchestra.run.gibba002.stdout: Installing : python2-bcrypt-3.1.6-2.el7.x86_64 14/99 2021-02-10T11:00:30.104 INFO:teuthology.orchestra.run.gibba002.stdout: Installing : socat-1.7.3.2-2.el7.x86_64 15/99 2021-02-10T11:00:30.351 INFO:teuthology.orchestra.run.gibba013.stdout: Updating : zlib-1.2.7-19.el7_9.x86_64 1/99 2021-02-10T11:00:30.452 INFO:teuthology.orchestra.run.gibba002.stdout: Installing : librdkafka-0.11.5-1.el7.x86_64 16/99 2021-02-10T11:00:30.679 INFO:teuthology.orchestra.run.gibba013.stdout: Updating : 1:openssl-libs-1.0.2k-21.el7_9.x86_64 2/99 2021-02-10T11:00:30.809 INFO:teuthology.orchestra.run.gibba002.stdout: Installing : oniguruma-6.8.2-1.el7.x86_64 17/99 2021-02-10T11:00:30.971 INFO:teuthology.orchestra.run.gibba013.stdout: Installing : liboath-2.6.2-1.el7.x86_64 3/99 2021-02-10T11:00:31.149 INFO:teuthology.orchestra.run.gibba002.stdout: Installing : jq-1.6-2.el7.x86_64 18/99 2021-02-10T11:00:31.241 INFO:teuthology.orchestra.run.gibba013.stdout: Installing : librabbitmq-0.8.0-3.el7.x86_64 4/99 2021-02-10T11:00:31.511 INFO:teuthology.orchestra.run.gibba013.stdout: Installing : zlib-devel-1.2.7-19.el7_9.x86_64 5/99 2021-02-10T11:00:31.517 INFO:teuthology.orchestra.run.gibba002.stdout: Installing : tbb-4.1-9.20130314.el7.x86_64 19/99 2021-02-10T11:00:31.732 INFO:teuthology.orchestra.run.gibba013.stdout: Installing : python2-six-1.9.0-0.el7.noarch 6/99 2021-02-10T11:00:31.900 INFO:teuthology.orchestra.run.gibba002.stdout: Installing : suitesparse-4.0.2-10.el7.x86_64 20/99 2021-02-10T11:00:32.052 INFO:teuthology.orchestra.run.gibba013.stdout: Installing : 2:python-ceph-argparse-14.2.16-280.g962fac13.el7.x86_64 7/99 2021-02-10T11:00:32.260 INFO:teuthology.orchestra.run.gibba002.stdout: Installing : python-tempita-0.5.1-6.el7.noarch 21/99 2021-02-10T11:00:32.411 INFO:teuthology.orchestra.run.gibba013.stdout: Installing : python-werkzeug-0.9.1-2.el7.noarch 8/99 2021-02-10T11:00:32.581 INFO:teuthology.orchestra.run.gibba002.stdout: Installing : python-paste-1.7.5.1-9.20111221hg1498.el7.noarch 22/99 2021-02-10T11:00:32.707 INFO:teuthology.orchestra.run.gibba013.stdout: Installing : python-cherrypy-3.2.2-4.el7.noarch 9/99 2021-02-10T11:00:32.871 INFO:teuthology.orchestra.run.gibba002.stdout: Installing : python-beaker-1.5.4-10.el7.noarch 23/99 2021-02-10T11:00:32.998 INFO:teuthology.orchestra.run.gibba013.stdout: Installing : python-webob-1.2.3-7.el7.noarch 10/99 2021-02-10T11:00:33.152 INFO:teuthology.orchestra.run.gibba002.stdout: Installing : python-cachetools-1.0.3-1.el7.noarch 24/99 2021-02-10T11:00:33.300 INFO:teuthology.orchestra.run.gibba013.stdout: Installing : python-urllib3-1.10.2-7.el7.noarch 11/99 2021-02-10T11:00:33.387 INFO:teuthology.orchestra.run.gibba002.stdout: Installing : 2:ceph-grafana-dashboards-14.2.16-280.g962fac13.el7.noar 25/99 2021-02-10T11:00:33.604 INFO:teuthology.orchestra.run.gibba013.stdout: Installing : python-requests-2.6.0-10.el7.noarch 12/99 2021-02-10T11:00:33.683 INFO:teuthology.orchestra.run.gibba002.stdout: Installing : libbabeltrace-1.2.4-3.el7.x86_64 26/99 2021-02-10T11:00:33.875 INFO:teuthology.orchestra.run.gibba013.stdout: Installing : python-webtest-1.3.4-6.el7.noarch 13/99 2021-02-10T11:00:33.974 INFO:teuthology.orchestra.run.gibba002.stdout: Installing : python-websocket-client-0.56.0-3.git3c25814.el7.noarch 27/99 2021-02-10T11:00:34.281 INFO:teuthology.orchestra.run.gibba013.stdout: Installing : python2-bcrypt-3.1.6-2.el7.x86_64 14/99 2021-02-10T11:00:34.323 INFO:teuthology.orchestra.run.gibba002.stdout: Installing : libcom_err-devel-1.42.9-19.el7.x86_64 28/99 2021-02-10T11:00:34.540 INFO:teuthology.orchestra.run.gibba002.stdout: Installing : 2:python3-ceph-argparse-14.2.16-280.g962fac13.el7.x86_64 29/99 2021-02-10T11:00:34.655 INFO:teuthology.orchestra.run.gibba013.stdout: Installing : socat-1.7.3.2-2.el7.x86_64 15/99 2021-02-10T11:00:34.918 INFO:teuthology.orchestra.run.gibba002.stdout: Installing : python2-certifi-2018.10.15-5.el7.noarch 30/99 2021-02-10T11:00:34.987 INFO:teuthology.orchestra.run.gibba013.stdout: Installing : librdkafka-0.11.5-1.el7.x86_64 16/99 2021-02-10T11:00:35.236 INFO:teuthology.orchestra.run.gibba002.stdout: Installing : pcre-devel-8.32-17.el7.x86_64 31/99 2021-02-10T11:00:35.294 INFO:teuthology.orchestra.run.gibba013.stdout: Installing : oniguruma-6.8.2-1.el7.x86_64 17/99 2021-02-10T11:00:35.475 INFO:teuthology.orchestra.run.gibba002.stdout: Installing : python-prettytable-0.7.2-3.el7.noarch 32/99 2021-02-10T11:00:35.602 INFO:teuthology.orchestra.run.gibba013.stdout: Installing : jq-1.6-2.el7.x86_64 18/99 2021-02-10T11:00:35.735 INFO:teuthology.orchestra.run.gibba002.stdout: Installing : python2-pyasn1-modules-0.1.9-7.el7.noarch 33/99 2021-02-10T11:00:35.926 INFO:teuthology.orchestra.run.gibba013.stdout: Installing : tbb-4.1-9.20130314.el7.x86_64 19/99 2021-02-10T11:00:36.034 INFO:teuthology.orchestra.run.gibba002.stdout: Installing : python-jwcrypto-0.4.2-1.el7.noarch 34/99 2021-02-10T11:00:36.280 INFO:teuthology.orchestra.run.gibba002.stdout: Installing : python2-oauthlib-2.0.1-8.el7.noarch 35/99 2021-02-10T11:00:36.301 INFO:teuthology.orchestra.run.gibba013.stdout: Installing : suitesparse-4.0.2-10.el7.x86_64 20/99 2021-02-10T11:00:36.572 INFO:teuthology.orchestra.run.gibba002.stdout: Installing : python2-requests-oauthlib-0.8.0-5.el7.noarch 36/99 2021-02-10T11:00:36.652 INFO:teuthology.orchestra.run.gibba013.stdout: Installing : python-tempita-0.5.1-6.el7.noarch 21/99 2021-02-10T11:00:36.903 INFO:teuthology.orchestra.run.gibba002.stdout: Installing : python2-rsa-3.4.2-2.el7.noarch 37/99 2021-02-10T11:00:36.984 INFO:teuthology.orchestra.run.gibba013.stdout: Installing : python-paste-1.7.5.1-9.20111221hg1498.el7.noarch 22/99 2021-02-10T11:00:37.239 INFO:teuthology.orchestra.run.gibba013.stdout: Installing : python-beaker-1.5.4-10.el7.noarch 23/99 2021-02-10T11:00:37.503 INFO:teuthology.orchestra.run.gibba013.stdout: Installing : python-cachetools-1.0.3-1.el7.noarch 24/99 2021-02-10T11:00:37.754 INFO:teuthology.orchestra.run.gibba013.stdout: Installing : 2:ceph-grafana-dashboards-14.2.16-280.g962fac13.el7.noar 25/99 2021-02-10T11:00:37.845 INFO:teuthology.orchestra.run.gibba002.stdout: Installing : 1:python2-google-auth-1.1.1-5.el7.noarch 38/99 2021-02-10T11:00:38.000 INFO:teuthology.orchestra.run.gibba013.stdout: Installing : libbabeltrace-1.2.4-3.el7.x86_64 26/99 2021-02-10T11:00:38.145 INFO:teuthology.orchestra.run.gibba002.stdout: Installing : 1:python2-kubernetes-11.0.0-6.el7.noarch 39/99 2021-02-10T11:00:38.281 INFO:teuthology.orchestra.run.gibba013.stdout: Installing : python-websocket-client-0.56.0-3.git3c25814.el7.noarch 27/99 2021-02-10T11:00:38.404 INFO:teuthology.orchestra.run.gibba002.stdout: Installing : python-singledispatch-3.4.0.2-2.el7.noarch 40/99 2021-02-10T11:00:38.596 INFO:teuthology.orchestra.run.gibba013.stdout: Installing : libcom_err-devel-1.42.9-19.el7.x86_64 28/99 2021-02-10T11:00:38.638 INFO:teuthology.orchestra.run.gibba002.stdout: Installing : keyutils-libs-devel-1.5.8-3.el7.x86_64 41/99 2021-02-10T11:00:38.849 INFO:teuthology.orchestra.run.gibba013.stdout: Installing : 2:python3-ceph-argparse-14.2.16-280.g962fac13.el7.x86_64 29/99 2021-02-10T11:00:38.912 INFO:teuthology.orchestra.run.gibba002.stdout: Installing : python-simplegeneric-0.8-7.el7.noarch 42/99 2021-02-10T11:00:39.215 INFO:teuthology.orchestra.run.gibba013.stdout: Installing : python2-certifi-2018.10.15-5.el7.noarch 30/99 2021-02-10T11:00:39.313 INFO:teuthology.orchestra.run.gibba002.stdout: Installing : userspace-rcu-0.7.16-1.el7.x86_64 43/99 2021-02-10T11:00:39.588 INFO:teuthology.orchestra.run.gibba013.stdout: Installing : pcre-devel-8.32-17.el7.x86_64 31/99 2021-02-10T11:00:39.818 INFO:teuthology.orchestra.run.gibba013.stdout: Installing : python-prettytable-0.7.2-3.el7.noarch 32/99 2021-02-10T11:00:39.875 INFO:teuthology.orchestra.run.gibba002.stdout: Installing : lttng-ust-2.4.1-4.el7.x86_64 44/99 2021-02-10T11:00:40.095 INFO:teuthology.orchestra.run.gibba013.stdout: Installing : python2-pyasn1-modules-0.1.9-7.el7.noarch 33/99 2021-02-10T11:00:40.232 INFO:teuthology.orchestra.run.gibba002.stdout: Updating : 2:librados2-14.2.16-280.g962fac13.el7.x86_64 45/99 2021-02-10T11:00:40.369 INFO:teuthology.orchestra.run.gibba013.stdout: Installing : python-jwcrypto-0.4.2-1.el7.noarch 34/99 2021-02-10T11:00:40.541 INFO:teuthology.orchestra.run.gibba002.stdout: Installing : 2:libcephfs2-14.2.16-280.g962fac13.el7.x86_64 46/99 2021-02-10T11:00:40.665 INFO:teuthology.orchestra.run.gibba013.stdout: Installing : python2-oauthlib-2.0.1-8.el7.noarch 35/99 2021-02-10T11:00:40.902 INFO:teuthology.orchestra.run.gibba002.stdout: Installing : 2:python-rados-14.2.16-280.g962fac13.el7.x86_64 47/99 2021-02-10T11:00:40.973 INFO:teuthology.orchestra.run.gibba013.stdout: Installing : python2-requests-oauthlib-0.8.0-5.el7.noarch 36/99 2021-02-10T11:00:41.308 INFO:teuthology.orchestra.run.gibba013.stdout: Installing : python2-rsa-3.4.2-2.el7.noarch 37/99 2021-02-10T11:00:41.486 INFO:teuthology.orchestra.run.gibba002.stdout: Updating : 2:librbd1-14.2.16-280.g962fac13.el7.x86_64 48/99 2021-02-10T11:00:41.760 INFO:teuthology.orchestra.run.gibba002.stdout: Installing : 2:librgw2-14.2.16-280.g962fac13.el7.x86_64 49/99 2021-02-10T11:00:42.078 INFO:teuthology.orchestra.run.gibba002.stdout: Installing : 2:python-rgw-14.2.16-280.g962fac13.el7.x86_64 50/99 2021-02-10T11:00:42.234 INFO:teuthology.orchestra.run.gibba013.stdout: Installing : 1:python2-google-auth-1.1.1-5.el7.noarch 38/99 2021-02-10T11:00:42.373 INFO:teuthology.orchestra.run.gibba002.stdout: Installing : 2:python-rbd-14.2.16-280.g962fac13.el7.x86_64 51/99 2021-02-10T11:00:42.554 INFO:teuthology.orchestra.run.gibba013.stdout: Installing : 1:python2-kubernetes-11.0.0-6.el7.noarch 39/99 2021-02-10T11:00:42.714 INFO:teuthology.orchestra.run.gibba002.stdout: Installing : 2:python-cephfs-14.2.16-280.g962fac13.el7.x86_64 52/99 2021-02-10T11:00:42.808 INFO:teuthology.orchestra.run.gibba013.stdout: Installing : python-singledispatch-3.4.0.2-2.el7.noarch 40/99 2021-02-10T11:00:43.108 INFO:teuthology.orchestra.run.gibba013.stdout: Installing : keyutils-libs-devel-1.5.8-3.el7.x86_64 41/99 2021-02-10T11:00:43.345 INFO:teuthology.orchestra.run.gibba013.stdout: Installing : python-simplegeneric-0.8-7.el7.noarch 42/99 2021-02-10T11:00:43.705 INFO:teuthology.orchestra.run.gibba013.stdout: Installing : userspace-rcu-0.7.16-1.el7.x86_64 43/99 2021-02-10T11:00:44.240 INFO:teuthology.orchestra.run.gibba013.stdout: Installing : lttng-ust-2.4.1-4.el7.x86_64 44/99 2021-02-10T11:00:44.495 INFO:teuthology.orchestra.run.gibba002.stdout: Installing : 2:libradosstriper1-14.2.16-280.g962fac13.el7.x86_64 53/99 2021-02-10T11:00:44.532 INFO:teuthology.orchestra.run.gibba013.stdout: Updating : 2:librados2-14.2.16-280.g962fac13.el7.x86_64 45/99 2021-02-10T11:00:44.808 INFO:teuthology.orchestra.run.gibba013.stdout: Installing : 2:libcephfs2-14.2.16-280.g962fac13.el7.x86_64 46/99 2021-02-10T11:00:45.160 INFO:teuthology.orchestra.run.gibba013.stdout: Installing : 2:python-rados-14.2.16-280.g962fac13.el7.x86_64 47/99 2021-02-10T11:00:45.245 INFO:teuthology.orchestra.run.gibba002.stdout: Installing : 2:ceph-common-14.2.16-280.g962fac13.el7.x86_64 54/99 2021-02-10T11:00:45.683 INFO:teuthology.orchestra.run.gibba002.stdout: Installing : 2:ceph-base-14.2.16-280.g962fac13.el7.x86_64 55/99 2021-02-10T11:00:45.786 INFO:teuthology.orchestra.run.gibba013.stdout: Updating : 2:librbd1-14.2.16-280.g962fac13.el7.x86_64 48/99 2021-02-10T11:00:46.102 INFO:teuthology.orchestra.run.gibba013.stdout: Installing : 2:librgw2-14.2.16-280.g962fac13.el7.x86_64 49/99 2021-02-10T11:00:46.421 INFO:teuthology.orchestra.run.gibba013.stdout: Installing : 2:python-rgw-14.2.16-280.g962fac13.el7.x86_64 50/99 2021-02-10T11:00:46.708 INFO:teuthology.orchestra.run.gibba013.stdout: Installing : 2:python-rbd-14.2.16-280.g962fac13.el7.x86_64 51/99 2021-02-10T11:00:47.039 INFO:teuthology.orchestra.run.gibba013.stdout: Installing : 2:python-cephfs-14.2.16-280.g962fac13.el7.x86_64 52/99 2021-02-10T11:00:48.889 INFO:teuthology.orchestra.run.gibba013.stdout: Installing : 2:libradosstriper1-14.2.16-280.g962fac13.el7.x86_64 53/99 2021-02-10T11:00:49.697 INFO:teuthology.orchestra.run.gibba013.stdout: Installing : 2:ceph-common-14.2.16-280.g962fac13.el7.x86_64 54/99 2021-02-10T11:00:50.160 INFO:teuthology.orchestra.run.gibba013.stdout: Installing : 2:ceph-base-14.2.16-280.g962fac13.el7.x86_64 55/99 2021-02-10T11:00:53.027 INFO:teuthology.orchestra.run.gibba002.stdout: Installing : 2:ceph-selinux-14.2.16-280.g962fac13.el7.x86_64 56/99 2021-02-10T11:00:53.433 INFO:teuthology.orchestra.run.gibba002.stdout: Installing : 2:ceph-osd-14.2.16-280.g962fac13.el7.x86_64 57/99 2021-02-10T11:00:54.016 INFO:teuthology.orchestra.run.gibba002.stdout: Installing : 2:ceph-mds-14.2.16-280.g962fac13.el7.x86_64 58/99 2021-02-10T11:00:54.903 INFO:teuthology.orchestra.run.gibba002.stdout: Installing : 2:ceph-mon-14.2.16-280.g962fac13.el7.x86_64 59/99 2021-02-10T11:00:55.219 INFO:teuthology.orchestra.run.gibba002.stdout: Installing : 2:librados-devel-14.2.16-280.g962fac13.el7.x86_64 60/99 2021-02-10T11:00:55.467 INFO:teuthology.orchestra.run.gibba002.stdout: Installing : 2:python3-rados-14.2.16-280.g962fac13.el7.x86_64 61/99 2021-02-10T11:00:55.812 INFO:teuthology.orchestra.run.gibba002.stdout: Installing : python-repoze-lru-0.4-3.el7.noarch 62/99 2021-02-10T11:00:56.087 INFO:teuthology.orchestra.run.gibba002.stdout: Installing : python-routes-1.13-2.el7.noarch 63/99 2021-02-10T11:00:56.359 INFO:teuthology.orchestra.run.gibba002.stdout: Installing : python-markupsafe-0.11-10.el7.x86_64 64/99 2021-02-10T11:00:56.674 INFO:teuthology.orchestra.run.gibba002.stdout: Installing : python-mako-0.8.1-2.el7.noarch 65/99 2021-02-10T11:00:57.184 INFO:teuthology.orchestra.run.gibba002.stdout: Installing : python-pecan-0.4.5-2.el7.noarch 66/99 2021-02-10T11:00:57.542 INFO:teuthology.orchestra.run.gibba013.stdout: Installing : 2:ceph-selinux-14.2.16-280.g962fac13.el7.x86_64 56/99 2021-02-10T11:00:57.589 INFO:teuthology.orchestra.run.gibba002.stdout: Installing : 2:ceph-mgr-14.2.16-280.g962fac13.el7.x86_64 67/99 2021-02-10T11:00:57.911 INFO:teuthology.orchestra.run.gibba002.stdout: Installing : python2-apipkg-1.5-5.el7.noarch 68/99 2021-02-10T11:00:57.997 INFO:teuthology.orchestra.run.gibba013.stdout: Installing : 2:ceph-osd-14.2.16-280.g962fac13.el7.x86_64 57/99 2021-02-10T11:00:58.266 INFO:teuthology.orchestra.run.gibba002.stdout: Installing : python-execnet-1.7.1-3.el7.noarch 69/99 2021-02-10T11:00:58.577 INFO:teuthology.orchestra.run.gibba002.stdout: Installing : python2-remoto-1.0.0-1.el7.noarch 70/99 2021-02-10T11:00:58.616 INFO:teuthology.orchestra.run.gibba013.stdout: Installing : 2:ceph-mds-14.2.16-280.g962fac13.el7.x86_64 58/99 2021-02-10T11:00:58.872 INFO:teuthology.orchestra.run.gibba002.stdout: Installing : libverto-devel-0.2.5-4.el7.x86_64 71/99 2021-02-10T11:00:58.944 INFO:teuthology.orchestra.run.gibba013.stdout: Installing : 2:ceph-mon-14.2.16-280.g962fac13.el7.x86_64 59/99 2021-02-10T11:00:59.198 INFO:teuthology.orchestra.run.gibba002.stdout: Installing : libsepol-devel-2.5-10.el7.x86_64 72/99 2021-02-10T11:00:59.253 INFO:teuthology.orchestra.run.gibba013.stdout: Installing : 2:librados-devel-14.2.16-280.g962fac13.el7.x86_64 60/99 2021-02-10T11:00:59.889 INFO:teuthology.orchestra.run.gibba002.stdout: Installing : libselinux-devel-2.5-15.el7.x86_64 73/99 2021-02-10T11:01:00.228 INFO:teuthology.orchestra.run.gibba002.stdout: Installing : krb5-devel-1.15.1-50.el7.x86_64 74/99 2021-02-10T11:01:00.376 INFO:teuthology.orchestra.run.gibba013.stdout: Installing : 2:python3-rados-14.2.16-280.g962fac13.el7.x86_64 61/99 2021-02-10T11:01:00.746 INFO:teuthology.orchestra.run.gibba013.stdout: Installing : python-repoze-lru-0.4-3.el7.noarch 62/99 2021-02-10T11:01:01.022 INFO:teuthology.orchestra.run.gibba013.stdout: Installing : python-routes-1.13-2.el7.noarch 63/99 2021-02-10T11:01:01.324 INFO:teuthology.orchestra.run.gibba013.stdout: Installing : python-markupsafe-0.11-10.el7.x86_64 64/99 2021-02-10T11:01:01.465 INFO:teuthology.orchestra.run.gibba002.stdout: Installing : 1:numpy-f2py-1.7.1-13.el7.x86_64 75/99 2021-02-10T11:01:01.744 INFO:teuthology.orchestra.run.gibba013.stdout: Installing : python-mako-0.8.1-2.el7.noarch 65/99 2021-02-10T11:01:02.110 INFO:teuthology.orchestra.run.gibba002.stdout: Installing : scipy-0.12.1-6.el7.x86_64 76/99 2021-02-10T11:01:02.270 INFO:teuthology.orchestra.run.gibba013.stdout: Installing : python-pecan-0.4.5-2.el7.noarch 66/99 2021-02-10T11:01:02.654 INFO:teuthology.orchestra.run.gibba002.stdout: Installing : 2:ceph-mgr-diskprediction-local-14.2.16-280.g962fac13.el 77/99 2021-02-10T11:01:02.673 INFO:teuthology.orchestra.run.gibba013.stdout: Installing : 2:ceph-mgr-14.2.16-280.g962fac13.el7.x86_64 67/99 2021-02-10T11:01:03.008 INFO:teuthology.orchestra.run.gibba002.stdout: Installing : 1:openssl-devel-1.0.2k-21.el7_9.x86_64 78/99 2021-02-10T11:01:03.029 INFO:teuthology.orchestra.run.gibba013.stdout: Installing : python2-apipkg-1.5-5.el7.noarch 68/99 2021-02-10T11:01:03.293 INFO:teuthology.orchestra.run.gibba002.stdout: Installing : 2:ceph-mgr-ssh-14.2.16-280.g962fac13.el7.noarch 79/99 2021-02-10T11:01:03.425 INFO:teuthology.orchestra.run.gibba013.stdout: Installing : python-execnet-1.7.1-3.el7.noarch 69/99 2021-02-10T11:01:03.598 INFO:teuthology.orchestra.run.gibba002.stdout: Installing : 2:ceph-mgr-diskprediction-cloud-14.2.16-280.g962fac13.el 80/99 2021-02-10T11:01:03.703 INFO:teuthology.orchestra.run.gibba013.stdout: Installing : python2-remoto-1.0.0-1.el7.noarch 70/99 2021-02-10T11:01:03.828 INFO:teuthology.orchestra.run.gibba002.stdout: Installing : 2:ceph-14.2.16-280.g962fac13.el7.x86_64 81/99 2021-02-10T11:01:03.972 INFO:teuthology.orchestra.run.gibba013.stdout: Installing : libverto-devel-0.2.5-4.el7.x86_64 71/99 2021-02-10T11:01:04.345 INFO:teuthology.orchestra.run.gibba013.stdout: Installing : libsepol-devel-2.5-10.el7.x86_64 72/99 2021-02-10T11:01:04.585 INFO:teuthology.orchestra.run.gibba002.stdout: Installing : 2:ceph-mgr-rook-14.2.16-280.g962fac13.el7.noarch 82/99 2021-02-10T11:01:05.036 INFO:teuthology.orchestra.run.gibba002.stdout: Installing : 2:ceph-mgr-dashboard-14.2.16-280.g962fac13.el7.noarch 83/99 2021-02-10T11:01:05.124 INFO:teuthology.orchestra.run.gibba013.stdout: Installing : libselinux-devel-2.5-15.el7.x86_64 73/99 2021-02-10T11:01:05.285 INFO:teuthology.orchestra.run.gibba002.stdout: Installing : 2:python3-cephfs-14.2.16-280.g962fac13.el7.x86_64 84/99 2021-02-10T11:01:05.478 INFO:teuthology.orchestra.run.gibba013.stdout: Installing : krb5-devel-1.15.1-50.el7.x86_64 74/99 2021-02-10T11:01:05.948 INFO:teuthology.orchestra.run.gibba002.stdout: Installing : 2:libcephfs-devel-14.2.16-280.g962fac13.el7.x86_64 85/99 2021-02-10T11:01:06.747 INFO:teuthology.orchestra.run.gibba013.stdout: Installing : 1:numpy-f2py-1.7.1-13.el7.x86_64 75/99 2021-02-10T11:01:07.418 INFO:teuthology.orchestra.run.gibba013.stdout: Installing : scipy-0.12.1-6.el7.x86_64 76/99 2021-02-10T11:01:07.952 INFO:teuthology.orchestra.run.gibba013.stdout: Installing : 2:ceph-mgr-diskprediction-local-14.2.16-280.g962fac13.el 77/99 2021-02-10T11:01:08.309 INFO:teuthology.orchestra.run.gibba013.stdout: Installing : 1:openssl-devel-1.0.2k-21.el7_9.x86_64 78/99 2021-02-10T11:01:08.601 INFO:teuthology.orchestra.run.gibba013.stdout: Installing : 2:ceph-mgr-ssh-14.2.16-280.g962fac13.el7.noarch 79/99 2021-02-10T11:01:08.749 INFO:teuthology.orchestra.run.gibba002.stdout: Installing : 2:ceph-radosgw-14.2.16-280.g962fac13.el7.x86_64 86/99 2021-02-10T11:01:08.910 INFO:teuthology.orchestra.run.gibba013.stdout: Installing : 2:ceph-mgr-diskprediction-cloud-14.2.16-280.g962fac13.el 80/99 2021-02-10T11:01:09.012 INFO:teuthology.orchestra.run.gibba002.stdout: Installing : 2:ceph-test-14.2.16-280.g962fac13.el7.x86_64 87/99 2021-02-10T11:01:09.094 INFO:teuthology.orchestra.run.gibba013.stdout: Installing : 2:ceph-14.2.16-280.g962fac13.el7.x86_64 81/99 2021-02-10T11:01:09.275 INFO:teuthology.orchestra.run.gibba002.stdout: Installing : 2:python-ceph-compat-14.2.16-280.g962fac13.el7.x86_64 88/99 2021-02-10T11:01:09.804 INFO:teuthology.orchestra.run.gibba002.stdout: Installing : 2:rbd-fuse-14.2.16-280.g962fac13.el7.x86_64 89/99 2021-02-10T11:01:09.871 INFO:teuthology.orchestra.run.gibba013.stdout: Installing : 2:ceph-mgr-rook-14.2.16-280.g962fac13.el7.noarch 82/99 2021-02-10T11:01:10.060 INFO:teuthology.orchestra.run.gibba002.stdout: Installing : 2:ceph-fuse-14.2.16-280.g962fac13.el7.x86_64 90/99 2021-02-10T11:01:10.311 INFO:teuthology.orchestra.run.gibba013.stdout: Installing : 2:ceph-mgr-dashboard-14.2.16-280.g962fac13.el7.noarch 83/99 2021-02-10T11:01:10.399 INFO:teuthology.orchestra.run.gibba002.stdout: Installing : elfutils-libelf-devel-0.176-5.el7.x86_64 91/99 2021-02-10T11:01:10.585 INFO:teuthology.orchestra.run.gibba013.stdout: Installing : 2:python3-cephfs-14.2.16-280.g962fac13.el7.x86_64 84/99 2021-02-10T11:01:10.800 INFO:teuthology.orchestra.run.gibba002.stdout: Updating : 1:openssl-1.0.2k-21.el7_9.x86_64 92/99 2021-02-10T11:01:11.215 INFO:teuthology.orchestra.run.gibba002.stdout: Installing : bison-3.0.4-2.el7.x86_64 93/99 2021-02-10T11:01:11.240 INFO:teuthology.orchestra.run.gibba013.stdout: Installing : 2:libcephfs-devel-14.2.16-280.g962fac13.el7.x86_64 85/99 2021-02-10T11:01:11.448 INFO:teuthology.orchestra.run.gibba002.stdout: Installing : flex-2.5.37-6.el7.x86_64 94/99 2021-02-10T11:01:11.716 INFO:teuthology.orchestra.run.gibba002.stdout: Cleanup : 1:openssl-1.0.2k-19.el7.x86_64 95/99 2021-02-10T11:01:12.024 INFO:teuthology.orchestra.run.gibba002.stdout: Cleanup : 1:openssl-libs-1.0.2k-19.el7.x86_64 96/99 2021-02-10T11:01:12.273 INFO:teuthology.orchestra.run.gibba002.stdout: Cleanup : 1:librbd1-10.2.5-4.el7.x86_64 97/99 2021-02-10T11:01:12.525 INFO:teuthology.orchestra.run.gibba002.stdout: Cleanup : 1:librados2-10.2.5-4.el7.x86_64 98/99 2021-02-10T11:01:12.943 INFO:teuthology.orchestra.run.gibba002.stdout: Cleanup : zlib-1.2.7-18.el7.x86_64 99/99 2021-02-10T11:01:12.943 INFO:teuthology.orchestra.run.gibba002.stdout:Loaded plugins: fastestmirror, langpacks, priorities, product-id, subscription- 2021-02-10T11:01:12.943 INFO:teuthology.orchestra.run.gibba002.stdout: : manager 2021-02-10T11:01:13.660 INFO:teuthology.orchestra.run.gibba002.stdout: Verifying : 2:libradosstriper1-14.2.16-280.g962fac13.el7.x86_64 1/99 2021-02-10T11:01:13.768 INFO:teuthology.orchestra.run.gibba002.stdout: Verifying : 2:ceph-selinux-14.2.16-280.g962fac13.el7.x86_64 2/99 2021-02-10T11:01:13.886 INFO:teuthology.orchestra.run.gibba002.stdout: Verifying : 2:python-rgw-14.2.16-280.g962fac13.el7.x86_64 3/99 2021-02-10T11:01:14.010 INFO:teuthology.orchestra.run.gibba013.stdout: Installing : 2:ceph-radosgw-14.2.16-280.g962fac13.el7.x86_64 86/99 2021-02-10T11:01:14.036 INFO:teuthology.orchestra.run.gibba002.stdout: Verifying : suitesparse-4.0.2-10.el7.x86_64 4/99 2021-02-10T11:01:14.153 INFO:teuthology.orchestra.run.gibba002.stdout: Verifying : jq-1.6-2.el7.x86_64 5/99 2021-02-10T11:01:14.295 INFO:teuthology.orchestra.run.gibba002.stdout: Verifying : python-routes-1.13-2.el7.noarch 6/99 2021-02-10T11:01:14.328 INFO:teuthology.orchestra.run.gibba013.stdout: Installing : 2:ceph-test-14.2.16-280.g962fac13.el7.x86_64 87/99 2021-02-10T11:01:14.446 INFO:teuthology.orchestra.run.gibba002.stdout: Verifying : 1:numpy-f2py-1.7.1-13.el7.x86_64 7/99 2021-02-10T11:01:14.542 INFO:teuthology.orchestra.run.gibba013.stdout: Installing : 2:python-ceph-compat-14.2.16-280.g962fac13.el7.x86_64 88/99 2021-02-10T11:01:15.120 INFO:teuthology.orchestra.run.gibba013.stdout: Installing : 2:rbd-fuse-14.2.16-280.g962fac13.el7.x86_64 89/99 2021-02-10T11:01:15.222 INFO:teuthology.orchestra.run.gibba002.stdout: Verifying : python-urllib3-1.10.2-7.el7.noarch 8/99 2021-02-10T11:01:15.326 INFO:teuthology.orchestra.run.gibba013.stdout: Installing : 2:ceph-fuse-14.2.16-280.g962fac13.el7.x86_64 90/99 2021-02-10T11:01:15.331 INFO:teuthology.orchestra.run.gibba002.stdout: Verifying : libsepol-devel-2.5-10.el7.x86_64 9/99 2021-02-10T11:01:15.423 INFO:teuthology.orchestra.run.gibba002.stdout: Verifying : python-mako-0.8.1-2.el7.noarch 10/99 2021-02-10T11:01:15.540 INFO:teuthology.orchestra.run.gibba002.stdout: Verifying : libverto-devel-0.2.5-4.el7.x86_64 11/99 2021-02-10T11:01:15.632 INFO:teuthology.orchestra.run.gibba002.stdout: Verifying : 2:libcephfs2-14.2.16-280.g962fac13.el7.x86_64 12/99 2021-02-10T11:01:15.649 INFO:teuthology.orchestra.run.gibba013.stdout: Installing : elfutils-libelf-devel-0.176-5.el7.x86_64 91/99 2021-02-10T11:01:15.724 INFO:teuthology.orchestra.run.gibba002.stdout: Verifying : 2:python-rbd-14.2.16-280.g962fac13.el7.x86_64 13/99 2021-02-10T11:01:15.825 INFO:teuthology.orchestra.run.gibba002.stdout: Verifying : 1:openssl-1.0.2k-21.el7_9.x86_64 14/99 2021-02-10T11:01:15.959 INFO:teuthology.orchestra.run.gibba002.stdout: Verifying : python2-apipkg-1.5-5.el7.noarch 15/99 2021-02-10T11:01:16.059 INFO:teuthology.orchestra.run.gibba002.stdout: Verifying : python-markupsafe-0.11-10.el7.x86_64 16/99 2021-02-10T11:01:16.084 INFO:teuthology.orchestra.run.gibba013.stdout: Updating : 1:openssl-1.0.2k-21.el7_9.x86_64 92/99 2021-02-10T11:01:16.185 INFO:teuthology.orchestra.run.gibba002.stdout: Verifying : python-webob-1.2.3-7.el7.noarch 17/99 2021-02-10T11:01:16.330 INFO:teuthology.orchestra.run.gibba002.stdout: Verifying : python-repoze-lru-0.4-3.el7.noarch 18/99 2021-02-10T11:01:16.452 INFO:teuthology.orchestra.run.gibba002.stdout: Verifying : 1:openssl-libs-1.0.2k-21.el7_9.x86_64 19/99 2021-02-10T11:01:16.485 INFO:teuthology.orchestra.run.gibba013.stdout: Installing : bison-3.0.4-2.el7.x86_64 93/99 2021-02-10T11:01:16.586 INFO:teuthology.orchestra.run.gibba002.stdout: Verifying : python-requests-2.6.0-10.el7.noarch 20/99 2021-02-10T11:01:16.712 INFO:teuthology.orchestra.run.gibba002.stdout: Verifying : python-cherrypy-3.2.2-4.el7.noarch 21/99 2021-02-10T11:01:16.797 INFO:teuthology.orchestra.run.gibba013.stdout: Installing : flex-2.5.37-6.el7.x86_64 94/99 2021-02-10T11:01:16.871 INFO:teuthology.orchestra.run.gibba002.stdout: Verifying : python-beaker-1.5.4-10.el7.noarch 22/99 2021-02-10T11:01:16.988 INFO:teuthology.orchestra.run.gibba002.stdout: Verifying : 1:python2-kubernetes-11.0.0-6.el7.noarch 23/99 2021-02-10T11:01:17.080 INFO:teuthology.orchestra.run.gibba002.stdout: Verifying : userspace-rcu-0.7.16-1.el7.x86_64 24/99 2021-02-10T11:01:17.117 INFO:teuthology.orchestra.run.gibba013.stdout: Cleanup : 1:openssl-1.0.2k-19.el7.x86_64 95/99 2021-02-10T11:01:17.205 INFO:teuthology.orchestra.run.gibba002.stdout: Verifying : 2:ceph-osd-14.2.16-280.g962fac13.el7.x86_64 25/99 2021-02-10T11:01:17.289 INFO:teuthology.orchestra.run.gibba002.stdout: Verifying : socat-1.7.3.2-2.el7.x86_64 26/99 2021-02-10T11:01:17.381 INFO:teuthology.orchestra.run.gibba002.stdout: Verifying : python2-remoto-1.0.0-1.el7.noarch 27/99 2021-02-10T11:01:17.459 INFO:teuthology.orchestra.run.gibba013.stdout: Cleanup : 1:openssl-libs-1.0.2k-19.el7.x86_64 96/99 2021-02-10T11:01:17.473 INFO:teuthology.orchestra.run.gibba002.stdout: Verifying : python-simplegeneric-0.8-7.el7.noarch 28/99 2021-02-10T11:01:17.616 INFO:teuthology.orchestra.run.gibba002.stdout: Verifying : keyutils-libs-devel-1.5.8-3.el7.x86_64 29/99 2021-02-10T11:01:17.716 INFO:teuthology.orchestra.run.gibba002.stdout: Verifying : 2:ceph-mgr-diskprediction-cloud-14.2.16-280.g962fac13.el 30/99 2021-02-10T11:01:17.733 INFO:teuthology.orchestra.run.gibba013.stdout: Cleanup : 1:librbd1-10.2.5-4.el7.x86_64 97/99 2021-02-10T11:01:17.792 INFO:teuthology.orchestra.run.gibba002.stdout: Verifying : 2:ceph-fuse-14.2.16-280.g962fac13.el7.x86_64 31/99 2021-02-10T11:01:17.900 INFO:teuthology.orchestra.run.gibba002.stdout: Verifying : 2:ceph-14.2.16-280.g962fac13.el7.x86_64 32/99 2021-02-10T11:01:17.983 INFO:teuthology.orchestra.run.gibba013.stdout: Cleanup : 1:librados2-10.2.5-4.el7.x86_64 98/99 2021-02-10T11:01:18.026 INFO:teuthology.orchestra.run.gibba002.stdout: Verifying : python-singledispatch-3.4.0.2-2.el7.noarch 33/99 2021-02-10T11:01:18.151 INFO:teuthology.orchestra.run.gibba002.stdout: Verifying : python2-requests-oauthlib-0.8.0-5.el7.noarch 34/99 2021-02-10T11:01:18.277 INFO:teuthology.orchestra.run.gibba002.stdout: Verifying : 2:ceph-mds-14.2.16-280.g962fac13.el7.x86_64 35/99 2021-02-10T11:01:18.366 INFO:teuthology.orchestra.run.gibba013.stdout: Cleanup : zlib-1.2.7-18.el7.x86_64 99/99 2021-02-10T11:01:18.366 INFO:teuthology.orchestra.run.gibba013.stdout:Loaded plugins: fastestmirror, langpacks, priorities, product-id, subscription- 2021-02-10T11:01:18.366 INFO:teuthology.orchestra.run.gibba013.stdout: : manager 2021-02-10T11:01:18.428 INFO:teuthology.orchestra.run.gibba002.stdout: Verifying : python-paste-1.7.5.1-9.20111221hg1498.el7.noarch 36/99 2021-02-10T11:01:18.553 INFO:teuthology.orchestra.run.gibba002.stdout: Verifying : python2-rsa-3.4.2-2.el7.noarch 37/99 2021-02-10T11:01:18.679 INFO:teuthology.orchestra.run.gibba002.stdout: Verifying : 2:ceph-mgr-ssh-14.2.16-280.g962fac13.el7.noarch 38/99 2021-02-10T11:01:18.838 INFO:teuthology.orchestra.run.gibba002.stdout: Verifying : python-jwcrypto-0.4.2-1.el7.noarch 39/99 2021-02-10T11:01:18.946 INFO:teuthology.orchestra.run.gibba002.stdout: Verifying : python-werkzeug-0.9.1-2.el7.noarch 40/99 2021-02-10T11:01:19.037 INFO:teuthology.orchestra.run.gibba013.stdout: Verifying : 2:libradosstriper1-14.2.16-280.g962fac13.el7.x86_64 1/99 2021-02-10T11:01:19.038 INFO:teuthology.orchestra.run.gibba002.stdout: Verifying : elfutils-libelf-devel-0.176-5.el7.x86_64 41/99 2021-02-10T11:01:19.122 INFO:teuthology.orchestra.run.gibba002.stdout: Verifying : python2-pyasn1-modules-0.1.9-7.el7.noarch 42/99 2021-02-10T11:01:19.126 INFO:teuthology.orchestra.run.gibba013.stdout: Verifying : 2:ceph-selinux-14.2.16-280.g962fac13.el7.x86_64 2/99 2021-02-10T11:01:19.210 INFO:teuthology.orchestra.run.gibba013.stdout: Verifying : 2:python-rgw-14.2.16-280.g962fac13.el7.x86_64 3/99 2021-02-10T11:01:19.248 INFO:teuthology.orchestra.run.gibba002.stdout: Verifying : python-prettytable-0.7.2-3.el7.noarch 43/99 2021-02-10T11:01:19.310 INFO:teuthology.orchestra.run.gibba013.stdout: Verifying : suitesparse-4.0.2-10.el7.x86_64 4/99 2021-02-10T11:01:19.357 INFO:teuthology.orchestra.run.gibba002.stdout: Verifying : librabbitmq-0.8.0-3.el7.x86_64 44/99 2021-02-10T11:01:19.449 INFO:teuthology.orchestra.run.gibba002.stdout: Verifying : 2:ceph-base-14.2.16-280.g962fac13.el7.x86_64 45/99 2021-02-10T11:01:19.452 INFO:teuthology.orchestra.run.gibba013.stdout: Verifying : jq-1.6-2.el7.x86_64 5/99 2021-02-10T11:01:19.549 INFO:teuthology.orchestra.run.gibba002.stdout: Verifying : scipy-0.12.1-6.el7.x86_64 46/99 2021-02-10T11:01:19.569 INFO:teuthology.orchestra.run.gibba013.stdout: Verifying : python-routes-1.13-2.el7.noarch 6/99 2021-02-10T11:01:19.678 INFO:teuthology.orchestra.run.gibba013.stdout: Verifying : 1:numpy-f2py-1.7.1-13.el7.x86_64 7/99 2021-02-10T11:01:19.683 INFO:teuthology.orchestra.run.gibba002.stdout: Verifying : 2:python-ceph-argparse-14.2.16-280.g962fac13.el7.x86_64 47/99 2021-02-10T11:01:20.053 INFO:teuthology.orchestra.run.gibba002.stdout: Verifying : libselinux-devel-2.5-15.el7.x86_64 48/99 2021-02-10T11:01:20.178 INFO:teuthology.orchestra.run.gibba002.stdout: Verifying : zlib-1.2.7-19.el7_9.x86_64 49/99 2021-02-10T11:01:20.304 INFO:teuthology.orchestra.run.gibba002.stdout: Verifying : python2-oauthlib-2.0.1-8.el7.noarch 50/99 2021-02-10T11:01:20.413 INFO:teuthology.orchestra.run.gibba002.stdout: Verifying : pcre-devel-8.32-17.el7.x86_64 51/99 2021-02-10T11:01:20.547 INFO:teuthology.orchestra.run.gibba002.stdout: Verifying : 2:ceph-radosgw-14.2.16-280.g962fac13.el7.x86_64 52/99 2021-02-10T11:01:20.631 INFO:teuthology.orchestra.run.gibba002.stdout: Verifying : 2:python-cephfs-14.2.16-280.g962fac13.el7.x86_64 53/99 2021-02-10T11:01:20.646 INFO:teuthology.orchestra.run.gibba013.stdout: Verifying : python-urllib3-1.10.2-7.el7.noarch 8/99 2021-02-10T11:01:20.714 INFO:teuthology.orchestra.run.gibba002.stdout: Verifying : 2:libcephfs-devel-14.2.16-280.g962fac13.el7.x86_64 54/99 2021-02-10T11:01:20.799 INFO:teuthology.orchestra.run.gibba013.stdout: Verifying : libsepol-devel-2.5-10.el7.x86_64 9/99 2021-02-10T11:01:20.806 INFO:teuthology.orchestra.run.gibba002.stdout: Verifying : 2:python-rados-14.2.16-280.g962fac13.el7.x86_64 55/99 2021-02-10T11:01:20.891 INFO:teuthology.orchestra.run.gibba013.stdout: Verifying : python-mako-0.8.1-2.el7.noarch 10/99 2021-02-10T11:01:20.931 INFO:teuthology.orchestra.run.gibba002.stdout: Verifying : 2:librados2-14.2.16-280.g962fac13.el7.x86_64 56/99 2021-02-10T11:01:20.975 INFO:teuthology.orchestra.run.gibba013.stdout: Verifying : libverto-devel-0.2.5-4.el7.x86_64 11/99 2021-02-10T11:01:21.024 INFO:teuthology.orchestra.run.gibba002.stdout: Verifying : librdkafka-0.11.5-1.el7.x86_64 57/99 2021-02-10T11:01:21.084 INFO:teuthology.orchestra.run.gibba013.stdout: Verifying : 2:libcephfs2-14.2.16-280.g962fac13.el7.x86_64 12/99 2021-02-10T11:01:21.116 INFO:teuthology.orchestra.run.gibba002.stdout: Verifying : flex-2.5.37-6.el7.x86_64 58/99 2021-02-10T11:01:21.167 INFO:teuthology.orchestra.run.gibba013.stdout: Verifying : 2:python-rbd-14.2.16-280.g962fac13.el7.x86_64 13/99 2021-02-10T11:01:21.191 INFO:teuthology.orchestra.run.gibba002.stdout: Verifying : python2-certifi-2018.10.15-5.el7.noarch 59/99 2021-02-10T11:01:21.251 INFO:teuthology.orchestra.run.gibba013.stdout: Verifying : 1:openssl-1.0.2k-21.el7_9.x86_64 14/99 2021-02-10T11:01:21.343 INFO:teuthology.orchestra.run.gibba013.stdout: Verifying : python2-apipkg-1.5-5.el7.noarch 15/99 2021-02-10T11:01:21.350 INFO:teuthology.orchestra.run.gibba002.stdout: Verifying : 2:python3-ceph-argparse-14.2.16-280.g962fac13.el7.x86_64 60/99 2021-02-10T11:01:21.459 INFO:teuthology.orchestra.run.gibba002.stdout: Verifying : 2:ceph-mgr-14.2.16-280.g962fac13.el7.x86_64 61/99 2021-02-10T11:01:21.469 INFO:teuthology.orchestra.run.gibba013.stdout: Verifying : python-markupsafe-0.11-10.el7.x86_64 16/99 2021-02-10T11:01:21.576 INFO:teuthology.orchestra.run.gibba002.stdout: Verifying : 2:librgw2-14.2.16-280.g962fac13.el7.x86_64 62/99 2021-02-10T11:01:21.577 INFO:teuthology.orchestra.run.gibba013.stdout: Verifying : python-webob-1.2.3-7.el7.noarch 17/99 2021-02-10T11:01:21.686 INFO:teuthology.orchestra.run.gibba013.stdout: Verifying : python-repoze-lru-0.4-3.el7.noarch 18/99 2021-02-10T11:01:21.727 INFO:teuthology.orchestra.run.gibba002.stdout: Verifying : krb5-devel-1.15.1-50.el7.x86_64 63/99 2021-02-10T11:01:21.837 INFO:teuthology.orchestra.run.gibba013.stdout: Verifying : 1:openssl-libs-1.0.2k-21.el7_9.x86_64 19/99 2021-02-10T11:01:21.852 INFO:teuthology.orchestra.run.gibba002.stdout: Verifying : 2:ceph-test-14.2.16-280.g962fac13.el7.x86_64 64/99 2021-02-10T11:01:21.954 INFO:teuthology.orchestra.run.gibba013.stdout: Verifying : python-requests-2.6.0-10.el7.noarch 20/99 2021-02-10T11:01:21.978 INFO:teuthology.orchestra.run.gibba002.stdout: Verifying : python-webtest-1.3.4-6.el7.noarch 65/99 2021-02-10T11:01:22.071 INFO:teuthology.orchestra.run.gibba013.stdout: Verifying : python-cherrypy-3.2.2-4.el7.noarch 21/99 2021-02-10T11:01:22.137 INFO:teuthology.orchestra.run.gibba002.stdout: Verifying : libcom_err-devel-1.42.9-19.el7.x86_64 66/99 2021-02-10T11:01:22.230 INFO:teuthology.orchestra.run.gibba013.stdout: Verifying : python-beaker-1.5.4-10.el7.noarch 22/99 2021-02-10T11:01:22.246 INFO:teuthology.orchestra.run.gibba002.stdout: Verifying : 2:rbd-fuse-14.2.16-280.g962fac13.el7.x86_64 67/99 2021-02-10T11:01:22.347 INFO:teuthology.orchestra.run.gibba013.stdout: Verifying : 1:python2-kubernetes-11.0.0-6.el7.noarch 23/99 2021-02-10T11:01:22.354 INFO:teuthology.orchestra.run.gibba002.stdout: Verifying : 2:librbd1-14.2.16-280.g962fac13.el7.x86_64 68/99 2021-02-10T11:01:22.447 INFO:teuthology.orchestra.run.gibba013.stdout: Verifying : userspace-rcu-0.7.16-1.el7.x86_64 24/99 2021-02-10T11:01:22.455 INFO:teuthology.orchestra.run.gibba002.stdout: Verifying : liboath-2.6.2-1.el7.x86_64 69/99 2021-02-10T11:01:22.572 INFO:teuthology.orchestra.run.gibba002.stdout: Verifying : python2-six-1.9.0-0.el7.noarch 70/99 2021-02-10T11:01:22.573 INFO:teuthology.orchestra.run.gibba013.stdout: Verifying : 2:ceph-osd-14.2.16-280.g962fac13.el7.x86_64 25/99 2021-02-10T11:01:22.656 INFO:teuthology.orchestra.run.gibba013.stdout: Verifying : socat-1.7.3.2-2.el7.x86_64 26/99 2021-02-10T11:01:22.664 INFO:teuthology.orchestra.run.gibba002.stdout: Verifying : 2:ceph-mgr-diskprediction-local-14.2.16-280.g962fac13.el 71/99 2021-02-10T11:01:22.748 INFO:teuthology.orchestra.run.gibba002.stdout: Verifying : python2-bcrypt-3.1.6-2.el7.x86_64 72/99 2021-02-10T11:01:22.749 INFO:teuthology.orchestra.run.gibba013.stdout: Verifying : python2-remoto-1.0.0-1.el7.noarch 27/99 2021-02-10T11:01:22.841 INFO:teuthology.orchestra.run.gibba013.stdout: Verifying : python-simplegeneric-0.8-7.el7.noarch 28/99 2021-02-10T11:01:22.874 INFO:teuthology.orchestra.run.gibba002.stdout: Verifying : python-websocket-client-0.56.0-3.git3c25814.el7.noarch 73/99 2021-02-10T11:01:22.983 INFO:teuthology.orchestra.run.gibba013.stdout: Verifying : keyutils-libs-devel-1.5.8-3.el7.x86_64 29/99 2021-02-10T11:01:23.033 INFO:teuthology.orchestra.run.gibba002.stdout: Verifying : 2:ceph-mgr-rook-14.2.16-280.g962fac13.el7.noarch 74/99 2021-02-10T11:01:23.067 INFO:teuthology.orchestra.run.gibba013.stdout: Verifying : 2:ceph-mgr-diskprediction-cloud-14.2.16-280.g962fac13.el 30/99 2021-02-10T11:01:23.133 INFO:teuthology.orchestra.run.gibba002.stdout: Verifying : 2:librados-devel-14.2.16-280.g962fac13.el7.x86_64 75/99 2021-02-10T11:01:23.143 INFO:teuthology.orchestra.run.gibba013.stdout: Verifying : 2:ceph-fuse-14.2.16-280.g962fac13.el7.x86_64 31/99 2021-02-10T11:01:23.234 INFO:teuthology.orchestra.run.gibba002.stdout: Verifying : libbabeltrace-1.2.4-3.el7.x86_64 76/99 2021-02-10T11:01:23.268 INFO:teuthology.orchestra.run.gibba013.stdout: Verifying : 2:ceph-14.2.16-280.g962fac13.el7.x86_64 32/99 2021-02-10T11:01:23.360 INFO:teuthology.orchestra.run.gibba013.stdout: Verifying : python-singledispatch-3.4.0.2-2.el7.noarch 33/99 2021-02-10T11:01:23.376 INFO:teuthology.orchestra.run.gibba002.stdout: Verifying : 2:ceph-mgr-dashboard-14.2.16-280.g962fac13.el7.noarch 77/99 2021-02-10T11:01:23.469 INFO:teuthology.orchestra.run.gibba013.stdout: Verifying : python2-requests-oauthlib-0.8.0-5.el7.noarch 34/99 2021-02-10T11:01:23.476 INFO:teuthology.orchestra.run.gibba002.stdout: Verifying : 2:python3-rados-14.2.16-280.g962fac13.el7.x86_64 78/99 2021-02-10T11:01:23.594 INFO:teuthology.orchestra.run.gibba002.stdout: Verifying : python-pecan-0.4.5-2.el7.noarch 79/99 2021-02-10T11:01:23.889 INFO:teuthology.orchestra.run.gibba013.stdout: Verifying : 2:ceph-mds-14.2.16-280.g962fac13.el7.x86_64 35/99 2021-02-10T11:01:23.890 INFO:teuthology.orchestra.run.gibba013.stdout: Verifying : python-paste-1.7.5.1-9.20111221hg1498.el7.noarch 36/99 2021-02-10T11:01:23.890 INFO:teuthology.orchestra.run.gibba013.stdout: Verifying : python2-rsa-3.4.2-2.el7.noarch 37/99 2021-02-10T11:01:23.890 INFO:teuthology.orchestra.run.gibba002.stdout: Verifying : 2:python-ceph-compat-14.2.16-280.g962fac13.el7.x86_64 80/99 2021-02-10T11:01:23.890 INFO:teuthology.orchestra.run.gibba002.stdout: Verifying : 2:ceph-common-14.2.16-280.g962fac13.el7.x86_64 81/99 2021-02-10T11:01:23.961 INFO:teuthology.orchestra.run.gibba013.stdout: Verifying : 2:ceph-mgr-ssh-14.2.16-280.g962fac13.el7.noarch 38/99 2021-02-10T11:01:23.986 INFO:teuthology.orchestra.run.gibba002.stdout: Verifying : 2:python3-cephfs-14.2.16-280.g962fac13.el7.x86_64 82/99 2021-02-10T11:01:24.096 INFO:teuthology.orchestra.run.gibba013.stdout: Verifying : python-jwcrypto-0.4.2-1.el7.noarch 39/99 2021-02-10T11:01:24.104 INFO:teuthology.orchestra.run.gibba002.stdout: Verifying : 2:ceph-grafana-dashboards-14.2.16-280.g962fac13.el7.noar 83/99 2021-02-10T11:01:24.222 INFO:teuthology.orchestra.run.gibba013.stdout: Verifying : python-werkzeug-0.9.1-2.el7.noarch 40/99 2021-02-10T11:01:24.255 INFO:teuthology.orchestra.run.gibba002.stdout: Verifying : python-execnet-1.7.1-3.el7.noarch 84/99 2021-02-10T11:01:24.347 INFO:teuthology.orchestra.run.gibba002.stdout: Verifying : lttng-ust-2.4.1-4.el7.x86_64 85/99 2021-02-10T11:01:24.364 INFO:teuthology.orchestra.run.gibba013.stdout: Verifying : elfutils-libelf-devel-0.176-5.el7.x86_64 41/99 2021-02-10T11:01:24.439 INFO:teuthology.orchestra.run.gibba002.stdout: Verifying : 2:ceph-mon-14.2.16-280.g962fac13.el7.x86_64 86/99 2021-02-10T11:01:24.464 INFO:teuthology.orchestra.run.gibba013.stdout: Verifying : python2-pyasn1-modules-0.1.9-7.el7.noarch 42/99 2021-02-10T11:01:24.531 INFO:teuthology.orchestra.run.gibba002.stdout: Verifying : bison-3.0.4-2.el7.x86_64 87/99 2021-02-10T11:01:24.548 INFO:teuthology.orchestra.run.gibba013.stdout: Verifying : python-prettytable-0.7.2-3.el7.noarch 43/99 2021-02-10T11:01:24.657 INFO:teuthology.orchestra.run.gibba002.stdout: Verifying : 1:openssl-devel-1.0.2k-21.el7_9.x86_64 88/99 2021-02-10T11:01:24.665 INFO:teuthology.orchestra.run.gibba013.stdout: Verifying : librabbitmq-0.8.0-3.el7.x86_64 44/99 2021-02-10T11:01:24.799 INFO:teuthology.orchestra.run.gibba013.stdout: Verifying : 2:ceph-base-14.2.16-280.g962fac13.el7.x86_64 45/99 2021-02-10T11:01:24.891 INFO:teuthology.orchestra.run.gibba013.stdout: Verifying : scipy-0.12.1-6.el7.x86_64 46/99 2021-02-10T11:01:24.983 INFO:teuthology.orchestra.run.gibba013.stdout: Verifying : 2:python-ceph-argparse-14.2.16-280.g962fac13.el7.x86_64 47/99 2021-02-10T11:01:25.084 INFO:teuthology.orchestra.run.gibba013.stdout: Verifying : libselinux-devel-2.5-15.el7.x86_64 48/99 2021-02-10T11:01:25.176 INFO:teuthology.orchestra.run.gibba013.stdout: Verifying : zlib-1.2.7-19.el7_9.x86_64 49/99 2021-02-10T11:01:25.232 INFO:teuthology.orchestra.run.gibba002.stdout: Verifying : python-cachetools-1.0.3-1.el7.noarch 89/99 2021-02-10T11:01:25.260 INFO:teuthology.orchestra.run.gibba013.stdout: Verifying : python2-oauthlib-2.0.1-8.el7.noarch 50/99 2021-02-10T11:01:25.333 INFO:teuthology.orchestra.run.gibba002.stdout: Verifying : python-tempita-0.5.1-6.el7.noarch 90/99 2021-02-10T11:01:25.361 INFO:teuthology.orchestra.run.gibba013.stdout: Verifying : pcre-devel-8.32-17.el7.x86_64 51/99 2021-02-10T11:01:25.463 INFO:teuthology.orchestra.run.gibba002.stdout: Verifying : zlib-devel-1.2.7-19.el7_9.x86_64 91/99 2021-02-10T11:01:25.503 INFO:teuthology.orchestra.run.gibba013.stdout: Verifying : 2:ceph-radosgw-14.2.16-280.g962fac13.el7.x86_64 52/99 2021-02-10T11:01:25.575 INFO:teuthology.orchestra.run.gibba002.stdout: Verifying : 1:python2-google-auth-1.1.1-5.el7.noarch 92/99 2021-02-10T11:01:25.611 INFO:teuthology.orchestra.run.gibba013.stdout: Verifying : 2:python-cephfs-14.2.16-280.g962fac13.el7.x86_64 53/99 2021-02-10T11:01:25.684 INFO:teuthology.orchestra.run.gibba002.stdout: Verifying : tbb-4.1-9.20130314.el7.x86_64 93/99 2021-02-10T11:01:25.729 INFO:teuthology.orchestra.run.gibba013.stdout: Verifying : 2:libcephfs-devel-14.2.16-280.g962fac13.el7.x86_64 54/99 2021-02-10T11:01:25.801 INFO:teuthology.orchestra.run.gibba002.stdout: Verifying : oniguruma-6.8.2-1.el7.x86_64 94/99 2021-02-10T11:01:25.802 INFO:teuthology.orchestra.run.gibba002.stdout: Verifying : 1:librbd1-10.2.5-4.el7.x86_64 95/99 2021-02-10T11:01:25.802 INFO:teuthology.orchestra.run.gibba002.stdout: Verifying : zlib-1.2.7-18.el7.x86_64 96/99 2021-02-10T11:01:25.802 INFO:teuthology.orchestra.run.gibba002.stdout: Verifying : 1:openssl-libs-1.0.2k-19.el7.x86_64 97/99 2021-02-10T11:01:25.803 INFO:teuthology.orchestra.run.gibba002.stdout: Verifying : 1:openssl-1.0.2k-19.el7.x86_64 98/99 2021-02-10T11:01:25.871 INFO:teuthology.orchestra.run.gibba013.stdout: Verifying : 2:python-rados-14.2.16-280.g962fac13.el7.x86_64 55/99 2021-02-10T11:01:25.996 INFO:teuthology.orchestra.run.gibba013.stdout: Verifying : 2:librados2-14.2.16-280.g962fac13.el7.x86_64 56/99 2021-02-10T11:01:26.077 INFO:teuthology.orchestra.run.gibba002.stdout: Verifying : 1:librados2-10.2.5-4.el7.x86_64 99/99 2021-02-10T11:01:26.077 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:01:26.078 INFO:teuthology.orchestra.run.gibba002.stdout:Installed: 2021-02-10T11:01:26.078 INFO:teuthology.orchestra.run.gibba002.stdout: bison.x86_64 0:3.0.4-2.el7 2021-02-10T11:01:26.078 INFO:teuthology.orchestra.run.gibba002.stdout: ceph.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:01:26.078 INFO:teuthology.orchestra.run.gibba002.stdout: ceph-fuse.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:01:26.078 INFO:teuthology.orchestra.run.gibba002.stdout: ceph-mgr.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:01:26.078 INFO:teuthology.orchestra.run.gibba002.stdout: ceph-mgr-dashboard.noarch 2:14.2.16-280.g962fac13.el7 2021-02-10T11:01:26.078 INFO:teuthology.orchestra.run.gibba002.stdout: ceph-mgr-diskprediction-cloud.noarch 2:14.2.16-280.g962fac13.el7 2021-02-10T11:01:26.078 INFO:teuthology.orchestra.run.gibba002.stdout: ceph-mgr-diskprediction-local.noarch 2:14.2.16-280.g962fac13.el7 2021-02-10T11:01:26.078 INFO:teuthology.orchestra.run.gibba002.stdout: ceph-mgr-rook.noarch 2:14.2.16-280.g962fac13.el7 2021-02-10T11:01:26.078 INFO:teuthology.orchestra.run.gibba002.stdout: ceph-mgr-ssh.noarch 2:14.2.16-280.g962fac13.el7 2021-02-10T11:01:26.078 INFO:teuthology.orchestra.run.gibba002.stdout: ceph-radosgw.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:01:26.078 INFO:teuthology.orchestra.run.gibba002.stdout: ceph-test.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:01:26.079 INFO:teuthology.orchestra.run.gibba002.stdout: elfutils-libelf-devel.x86_64 0:0.176-5.el7 2021-02-10T11:01:26.079 INFO:teuthology.orchestra.run.gibba002.stdout: flex.x86_64 0:2.5.37-6.el7 2021-02-10T11:01:26.079 INFO:teuthology.orchestra.run.gibba002.stdout: libcephfs-devel.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:01:26.079 INFO:teuthology.orchestra.run.gibba002.stdout: libcephfs2.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:01:26.079 INFO:teuthology.orchestra.run.gibba002.stdout: openssl-devel.x86_64 1:1.0.2k-21.el7_9 2021-02-10T11:01:26.079 INFO:teuthology.orchestra.run.gibba002.stdout: python-ceph-compat.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:01:26.079 INFO:teuthology.orchestra.run.gibba002.stdout: python3-cephfs.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:01:26.079 INFO:teuthology.orchestra.run.gibba002.stdout: rbd-fuse.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:01:26.079 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:01:26.079 INFO:teuthology.orchestra.run.gibba002.stdout:Dependency Installed: 2021-02-10T11:01:26.079 INFO:teuthology.orchestra.run.gibba002.stdout: ceph-base.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:01:26.079 INFO:teuthology.orchestra.run.gibba002.stdout: ceph-common.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:01:26.079 INFO:teuthology.orchestra.run.gibba002.stdout: ceph-grafana-dashboards.noarch 2:14.2.16-280.g962fac13.el7 2021-02-10T11:01:26.080 INFO:teuthology.orchestra.run.gibba002.stdout: ceph-mds.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:01:26.080 INFO:teuthology.orchestra.run.gibba002.stdout: ceph-mon.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:01:26.080 INFO:teuthology.orchestra.run.gibba002.stdout: ceph-osd.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:01:26.080 INFO:teuthology.orchestra.run.gibba002.stdout: ceph-selinux.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:01:26.080 INFO:teuthology.orchestra.run.gibba002.stdout: jq.x86_64 0:1.6-2.el7 2021-02-10T11:01:26.080 INFO:teuthology.orchestra.run.gibba002.stdout: keyutils-libs-devel.x86_64 0:1.5.8-3.el7 2021-02-10T11:01:26.080 INFO:teuthology.orchestra.run.gibba002.stdout: krb5-devel.x86_64 0:1.15.1-50.el7 2021-02-10T11:01:26.080 INFO:teuthology.orchestra.run.gibba002.stdout: libbabeltrace.x86_64 0:1.2.4-3.el7 2021-02-10T11:01:26.080 INFO:teuthology.orchestra.run.gibba002.stdout: libcom_err-devel.x86_64 0:1.42.9-19.el7 2021-02-10T11:01:26.080 INFO:teuthology.orchestra.run.gibba002.stdout: liboath.x86_64 0:2.6.2-1.el7 2021-02-10T11:01:26.080 INFO:teuthology.orchestra.run.gibba002.stdout: librabbitmq.x86_64 0:0.8.0-3.el7 2021-02-10T11:01:26.081 INFO:teuthology.orchestra.run.gibba002.stdout: librados-devel.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:01:26.081 INFO:teuthology.orchestra.run.gibba002.stdout: libradosstriper1.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:01:26.081 INFO:teuthology.orchestra.run.gibba002.stdout: librdkafka.x86_64 0:0.11.5-1.el7 2021-02-10T11:01:26.081 INFO:teuthology.orchestra.run.gibba002.stdout: librgw2.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:01:26.081 INFO:teuthology.orchestra.run.gibba002.stdout: libselinux-devel.x86_64 0:2.5-15.el7 2021-02-10T11:01:26.081 INFO:teuthology.orchestra.run.gibba002.stdout: libsepol-devel.x86_64 0:2.5-10.el7 2021-02-10T11:01:26.081 INFO:teuthology.orchestra.run.gibba002.stdout: libverto-devel.x86_64 0:0.2.5-4.el7 2021-02-10T11:01:26.081 INFO:teuthology.orchestra.run.gibba002.stdout: lttng-ust.x86_64 0:2.4.1-4.el7 2021-02-10T11:01:26.081 INFO:teuthology.orchestra.run.gibba002.stdout: numpy-f2py.x86_64 1:1.7.1-13.el7 2021-02-10T11:01:26.081 INFO:teuthology.orchestra.run.gibba002.stdout: oniguruma.x86_64 0:6.8.2-1.el7 2021-02-10T11:01:26.081 INFO:teuthology.orchestra.run.gibba002.stdout: pcre-devel.x86_64 0:8.32-17.el7 2021-02-10T11:01:26.081 INFO:teuthology.orchestra.run.gibba002.stdout: python-beaker.noarch 0:1.5.4-10.el7 2021-02-10T11:01:26.081 INFO:teuthology.orchestra.run.gibba002.stdout: python-cachetools.noarch 0:1.0.3-1.el7 2021-02-10T11:01:26.081 INFO:teuthology.orchestra.run.gibba002.stdout: python-ceph-argparse.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:01:26.082 INFO:teuthology.orchestra.run.gibba002.stdout: python-cephfs.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:01:26.082 INFO:teuthology.orchestra.run.gibba002.stdout: python-cherrypy.noarch 0:3.2.2-4.el7 2021-02-10T11:01:26.082 INFO:teuthology.orchestra.run.gibba002.stdout: python-execnet.noarch 0:1.7.1-3.el7 2021-02-10T11:01:26.082 INFO:teuthology.orchestra.run.gibba002.stdout: python-jwcrypto.noarch 0:0.4.2-1.el7 2021-02-10T11:01:26.082 INFO:teuthology.orchestra.run.gibba002.stdout: python-mako.noarch 0:0.8.1-2.el7 2021-02-10T11:01:26.082 INFO:teuthology.orchestra.run.gibba002.stdout: python-markupsafe.x86_64 0:0.11-10.el7 2021-02-10T11:01:26.082 INFO:teuthology.orchestra.run.gibba002.stdout: python-paste.noarch 0:1.7.5.1-9.20111221hg1498.el7 2021-02-10T11:01:26.082 INFO:teuthology.orchestra.run.gibba002.stdout: python-pecan.noarch 0:0.4.5-2.el7 2021-02-10T11:01:26.082 INFO:teuthology.orchestra.run.gibba002.stdout: python-prettytable.noarch 0:0.7.2-3.el7 2021-02-10T11:01:26.082 INFO:teuthology.orchestra.run.gibba002.stdout: python-rados.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:01:26.082 INFO:teuthology.orchestra.run.gibba002.stdout: python-rbd.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:01:26.082 INFO:teuthology.orchestra.run.gibba002.stdout: python-repoze-lru.noarch 0:0.4-3.el7 2021-02-10T11:01:26.083 INFO:teuthology.orchestra.run.gibba002.stdout: python-requests.noarch 0:2.6.0-10.el7 2021-02-10T11:01:26.083 INFO:teuthology.orchestra.run.gibba002.stdout: python-rgw.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:01:26.083 INFO:teuthology.orchestra.run.gibba002.stdout: python-routes.noarch 0:1.13-2.el7 2021-02-10T11:01:26.083 INFO:teuthology.orchestra.run.gibba002.stdout: python-simplegeneric.noarch 0:0.8-7.el7 2021-02-10T11:01:26.083 INFO:teuthology.orchestra.run.gibba002.stdout: python-singledispatch.noarch 0:3.4.0.2-2.el7 2021-02-10T11:01:26.083 INFO:teuthology.orchestra.run.gibba002.stdout: python-tempita.noarch 0:0.5.1-6.el7 2021-02-10T11:01:26.083 INFO:teuthology.orchestra.run.gibba002.stdout: python-urllib3.noarch 0:1.10.2-7.el7 2021-02-10T11:01:26.083 INFO:teuthology.orchestra.run.gibba002.stdout: python-webob.noarch 0:1.2.3-7.el7 2021-02-10T11:01:26.083 INFO:teuthology.orchestra.run.gibba002.stdout: python-websocket-client.noarch 0:0.56.0-3.git3c25814.el7 2021-02-10T11:01:26.083 INFO:teuthology.orchestra.run.gibba002.stdout: python-webtest.noarch 0:1.3.4-6.el7 2021-02-10T11:01:26.083 INFO:teuthology.orchestra.run.gibba002.stdout: python-werkzeug.noarch 0:0.9.1-2.el7 2021-02-10T11:01:26.083 INFO:teuthology.orchestra.run.gibba002.stdout: python2-apipkg.noarch 0:1.5-5.el7 2021-02-10T11:01:26.083 INFO:teuthology.orchestra.run.gibba002.stdout: python2-bcrypt.x86_64 0:3.1.6-2.el7 2021-02-10T11:01:26.084 INFO:teuthology.orchestra.run.gibba002.stdout: python2-certifi.noarch 0:2018.10.15-5.el7 2021-02-10T11:01:26.084 INFO:teuthology.orchestra.run.gibba002.stdout: python2-google-auth.noarch 1:1.1.1-5.el7 2021-02-10T11:01:26.084 INFO:teuthology.orchestra.run.gibba002.stdout: python2-kubernetes.noarch 1:11.0.0-6.el7 2021-02-10T11:01:26.084 INFO:teuthology.orchestra.run.gibba002.stdout: python2-oauthlib.noarch 0:2.0.1-8.el7 2021-02-10T11:01:26.084 INFO:teuthology.orchestra.run.gibba002.stdout: python2-pyasn1-modules.noarch 0:0.1.9-7.el7 2021-02-10T11:01:26.084 INFO:teuthology.orchestra.run.gibba002.stdout: python2-remoto.noarch 0:1.0.0-1.el7 2021-02-10T11:01:26.084 INFO:teuthology.orchestra.run.gibba002.stdout: python2-requests-oauthlib.noarch 0:0.8.0-5.el7 2021-02-10T11:01:26.084 INFO:teuthology.orchestra.run.gibba002.stdout: python2-rsa.noarch 0:3.4.2-2.el7 2021-02-10T11:01:26.084 INFO:teuthology.orchestra.run.gibba002.stdout: python2-six.noarch 0:1.9.0-0.el7 2021-02-10T11:01:26.084 INFO:teuthology.orchestra.run.gibba002.stdout: python3-ceph-argparse.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:01:26.084 INFO:teuthology.orchestra.run.gibba002.stdout: python3-rados.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:01:26.085 INFO:teuthology.orchestra.run.gibba002.stdout: scipy.x86_64 0:0.12.1-6.el7 2021-02-10T11:01:26.085 INFO:teuthology.orchestra.run.gibba002.stdout: socat.x86_64 0:1.7.3.2-2.el7 2021-02-10T11:01:26.085 INFO:teuthology.orchestra.run.gibba002.stdout: suitesparse.x86_64 0:4.0.2-10.el7 2021-02-10T11:01:26.085 INFO:teuthology.orchestra.run.gibba002.stdout: tbb.x86_64 0:4.1-9.20130314.el7 2021-02-10T11:01:26.085 INFO:teuthology.orchestra.run.gibba002.stdout: userspace-rcu.x86_64 0:0.7.16-1.el7 2021-02-10T11:01:26.085 INFO:teuthology.orchestra.run.gibba002.stdout: zlib-devel.x86_64 0:1.2.7-19.el7_9 2021-02-10T11:01:26.085 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:01:26.085 INFO:teuthology.orchestra.run.gibba002.stdout:Updated: 2021-02-10T11:01:26.085 INFO:teuthology.orchestra.run.gibba002.stdout: librados2.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:01:26.085 INFO:teuthology.orchestra.run.gibba002.stdout: librbd1.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:01:26.085 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:01:26.085 INFO:teuthology.orchestra.run.gibba002.stdout:Dependency Updated: 2021-02-10T11:01:26.085 INFO:teuthology.orchestra.run.gibba002.stdout: openssl.x86_64 1:1.0.2k-21.el7_9 openssl-libs.x86_64 1:1.0.2k-21.el7_9 2021-02-10T11:01:26.085 INFO:teuthology.orchestra.run.gibba002.stdout: zlib.x86_64 0:1.2.7-19.el7_9 2021-02-10T11:01:26.086 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:01:26.086 INFO:teuthology.orchestra.run.gibba002.stdout:Complete! 2021-02-10T11:01:26.144 INFO:teuthology.orchestra.run.gibba013.stdout: Verifying : librdkafka-0.11.5-1.el7.x86_64 57/99 2021-02-10T11:01:26.191 DEBUG:teuthology.parallel:result is None 2021-02-10T11:01:26.256 INFO:teuthology.orchestra.run.gibba013.stdout: Verifying : flex-2.5.37-6.el7.x86_64 58/99 2021-02-10T11:01:26.348 INFO:teuthology.orchestra.run.gibba013.stdout: Verifying : python2-certifi-2018.10.15-5.el7.noarch 59/99 2021-02-10T11:01:26.448 INFO:teuthology.orchestra.run.gibba013.stdout: Verifying : 2:python3-ceph-argparse-14.2.16-280.g962fac13.el7.x86_64 60/99 2021-02-10T11:01:26.582 INFO:teuthology.orchestra.run.gibba013.stdout: Verifying : 2:ceph-mgr-14.2.16-280.g962fac13.el7.x86_64 61/99 2021-02-10T11:01:26.674 INFO:teuthology.orchestra.run.gibba013.stdout: Verifying : 2:librgw2-14.2.16-280.g962fac13.el7.x86_64 62/99 2021-02-10T11:01:26.766 INFO:teuthology.orchestra.run.gibba013.stdout: Verifying : krb5-devel-1.15.1-50.el7.x86_64 63/99 2021-02-10T11:01:26.892 INFO:teuthology.orchestra.run.gibba013.stdout: Verifying : 2:ceph-test-14.2.16-280.g962fac13.el7.x86_64 64/99 2021-02-10T11:01:27.004 INFO:teuthology.orchestra.run.gibba013.stdout: Verifying : python-webtest-1.3.4-6.el7.noarch 65/99 2021-02-10T11:01:27.068 INFO:teuthology.orchestra.run.gibba013.stdout: Verifying : libcom_err-devel-1.42.9-19.el7.x86_64 66/99 2021-02-10T11:01:27.161 INFO:teuthology.orchestra.run.gibba013.stdout: Verifying : 2:rbd-fuse-14.2.16-280.g962fac13.el7.x86_64 67/99 2021-02-10T11:01:27.294 INFO:teuthology.orchestra.run.gibba013.stdout: Verifying : 2:librbd1-14.2.16-280.g962fac13.el7.x86_64 68/99 2021-02-10T11:01:27.403 INFO:teuthology.orchestra.run.gibba013.stdout: Verifying : liboath-2.6.2-1.el7.x86_64 69/99 2021-02-10T11:01:27.512 INFO:teuthology.orchestra.run.gibba013.stdout: Verifying : python2-six-1.9.0-0.el7.noarch 70/99 2021-02-10T11:01:27.662 INFO:teuthology.orchestra.run.gibba013.stdout: Verifying : 2:ceph-mgr-diskprediction-local-14.2.16-280.g962fac13.el 71/99 2021-02-10T11:01:27.779 INFO:teuthology.orchestra.run.gibba013.stdout: Verifying : python2-bcrypt-3.1.6-2.el7.x86_64 72/99 2021-02-10T11:01:27.940 INFO:teuthology.orchestra.run.gibba013.stdout: Verifying : python-websocket-client-0.56.0-3.git3c25814.el7.noarch 73/99 2021-02-10T11:01:28.072 INFO:teuthology.orchestra.run.gibba013.stdout: Verifying : 2:ceph-mgr-rook-14.2.16-280.g962fac13.el7.noarch 74/99 2021-02-10T11:01:28.173 INFO:teuthology.orchestra.run.gibba013.stdout: Verifying : 2:librados-devel-14.2.16-280.g962fac13.el7.x86_64 75/99 2021-02-10T11:01:28.274 INFO:teuthology.orchestra.run.gibba013.stdout: Verifying : libbabeltrace-1.2.4-3.el7.x86_64 76/99 2021-02-10T11:01:28.424 INFO:teuthology.orchestra.run.gibba013.stdout: Verifying : 2:ceph-mgr-dashboard-14.2.16-280.g962fac13.el7.noarch 77/99 2021-02-10T11:01:28.499 INFO:teuthology.orchestra.run.gibba013.stdout: Verifying : 2:python3-rados-14.2.16-280.g962fac13.el7.x86_64 78/99 2021-02-10T11:01:28.583 INFO:teuthology.orchestra.run.gibba013.stdout: Verifying : python-pecan-0.4.5-2.el7.noarch 79/99 2021-02-10T11:01:28.700 INFO:teuthology.orchestra.run.gibba013.stdout: Verifying : 2:python-ceph-compat-14.2.16-280.g962fac13.el7.x86_64 80/99 2021-02-10T11:01:28.794 INFO:teuthology.orchestra.run.gibba013.stdout: Verifying : 2:ceph-common-14.2.16-280.g962fac13.el7.x86_64 81/99 2021-02-10T11:01:28.876 INFO:teuthology.orchestra.run.gibba013.stdout: Verifying : 2:python3-cephfs-14.2.16-280.g962fac13.el7.x86_64 82/99 2021-02-10T11:01:28.968 INFO:teuthology.orchestra.run.gibba013.stdout: Verifying : 2:ceph-grafana-dashboards-14.2.16-280.g962fac13.el7.noar 83/99 2021-02-10T11:01:29.085 INFO:teuthology.orchestra.run.gibba013.stdout: Verifying : python-execnet-1.7.1-3.el7.noarch 84/99 2021-02-10T11:01:29.186 INFO:teuthology.orchestra.run.gibba013.stdout: Verifying : lttng-ust-2.4.1-4.el7.x86_64 85/99 2021-02-10T11:01:29.295 INFO:teuthology.orchestra.run.gibba013.stdout: Verifying : 2:ceph-mon-14.2.16-280.g962fac13.el7.x86_64 86/99 2021-02-10T11:01:29.437 INFO:teuthology.orchestra.run.gibba013.stdout: Verifying : bison-3.0.4-2.el7.x86_64 87/99 2021-02-10T11:01:29.554 INFO:teuthology.orchestra.run.gibba013.stdout: Verifying : 1:openssl-devel-1.0.2k-21.el7_9.x86_64 88/99 2021-02-10T11:01:29.671 INFO:teuthology.orchestra.run.gibba013.stdout: Verifying : python-cachetools-1.0.3-1.el7.noarch 89/99 2021-02-10T11:01:29.814 INFO:teuthology.orchestra.run.gibba013.stdout: Verifying : python-tempita-0.5.1-6.el7.noarch 90/99 2021-02-10T11:01:29.932 INFO:teuthology.orchestra.run.gibba013.stdout: Verifying : zlib-devel-1.2.7-19.el7_9.x86_64 91/99 2021-02-10T11:01:30.031 INFO:teuthology.orchestra.run.gibba013.stdout: Verifying : 1:python2-google-auth-1.1.1-5.el7.noarch 92/99 2021-02-10T11:01:30.173 INFO:teuthology.orchestra.run.gibba013.stdout: Verifying : tbb-4.1-9.20130314.el7.x86_64 93/99 2021-02-10T11:01:30.282 INFO:teuthology.orchestra.run.gibba013.stdout: Verifying : oniguruma-6.8.2-1.el7.x86_64 94/99 2021-02-10T11:01:30.282 INFO:teuthology.orchestra.run.gibba013.stdout: Verifying : 1:librbd1-10.2.5-4.el7.x86_64 95/99 2021-02-10T11:01:30.283 INFO:teuthology.orchestra.run.gibba013.stdout: Verifying : zlib-1.2.7-18.el7.x86_64 96/99 2021-02-10T11:01:30.283 INFO:teuthology.orchestra.run.gibba013.stdout: Verifying : 1:openssl-libs-1.0.2k-19.el7.x86_64 97/99 2021-02-10T11:01:30.283 INFO:teuthology.orchestra.run.gibba013.stdout: Verifying : 1:openssl-1.0.2k-19.el7.x86_64 98/99 2021-02-10T11:01:30.991 INFO:teuthology.orchestra.run.gibba013.stdout: Verifying : 1:librados2-10.2.5-4.el7.x86_64 99/99 2021-02-10T11:01:30.991 INFO:teuthology.orchestra.run.gibba013.stdout: 2021-02-10T11:01:30.991 INFO:teuthology.orchestra.run.gibba013.stdout:Installed: 2021-02-10T11:01:30.992 INFO:teuthology.orchestra.run.gibba013.stdout: bison.x86_64 0:3.0.4-2.el7 2021-02-10T11:01:30.992 INFO:teuthology.orchestra.run.gibba013.stdout: ceph.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:01:30.992 INFO:teuthology.orchestra.run.gibba013.stdout: ceph-fuse.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:01:30.992 INFO:teuthology.orchestra.run.gibba013.stdout: ceph-mgr.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:01:30.992 INFO:teuthology.orchestra.run.gibba013.stdout: ceph-mgr-dashboard.noarch 2:14.2.16-280.g962fac13.el7 2021-02-10T11:01:30.992 INFO:teuthology.orchestra.run.gibba013.stdout: ceph-mgr-diskprediction-cloud.noarch 2:14.2.16-280.g962fac13.el7 2021-02-10T11:01:30.992 INFO:teuthology.orchestra.run.gibba013.stdout: ceph-mgr-diskprediction-local.noarch 2:14.2.16-280.g962fac13.el7 2021-02-10T11:01:30.992 INFO:teuthology.orchestra.run.gibba013.stdout: ceph-mgr-rook.noarch 2:14.2.16-280.g962fac13.el7 2021-02-10T11:01:30.992 INFO:teuthology.orchestra.run.gibba013.stdout: ceph-mgr-ssh.noarch 2:14.2.16-280.g962fac13.el7 2021-02-10T11:01:30.992 INFO:teuthology.orchestra.run.gibba013.stdout: ceph-radosgw.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:01:30.993 INFO:teuthology.orchestra.run.gibba013.stdout: ceph-test.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:01:30.993 INFO:teuthology.orchestra.run.gibba013.stdout: elfutils-libelf-devel.x86_64 0:0.176-5.el7 2021-02-10T11:01:30.993 INFO:teuthology.orchestra.run.gibba013.stdout: flex.x86_64 0:2.5.37-6.el7 2021-02-10T11:01:30.993 INFO:teuthology.orchestra.run.gibba013.stdout: libcephfs-devel.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:01:30.993 INFO:teuthology.orchestra.run.gibba013.stdout: libcephfs2.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:01:30.993 INFO:teuthology.orchestra.run.gibba013.stdout: openssl-devel.x86_64 1:1.0.2k-21.el7_9 2021-02-10T11:01:30.993 INFO:teuthology.orchestra.run.gibba013.stdout: python-ceph-compat.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:01:30.993 INFO:teuthology.orchestra.run.gibba013.stdout: python3-cephfs.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:01:30.993 INFO:teuthology.orchestra.run.gibba013.stdout: rbd-fuse.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:01:30.994 INFO:teuthology.orchestra.run.gibba013.stdout: 2021-02-10T11:01:30.994 INFO:teuthology.orchestra.run.gibba013.stdout:Dependency Installed: 2021-02-10T11:01:30.994 INFO:teuthology.orchestra.run.gibba013.stdout: ceph-base.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:01:30.994 INFO:teuthology.orchestra.run.gibba013.stdout: ceph-common.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:01:30.994 INFO:teuthology.orchestra.run.gibba013.stdout: ceph-grafana-dashboards.noarch 2:14.2.16-280.g962fac13.el7 2021-02-10T11:01:30.994 INFO:teuthology.orchestra.run.gibba013.stdout: ceph-mds.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:01:30.994 INFO:teuthology.orchestra.run.gibba013.stdout: ceph-mon.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:01:30.994 INFO:teuthology.orchestra.run.gibba013.stdout: ceph-osd.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:01:30.994 INFO:teuthology.orchestra.run.gibba013.stdout: ceph-selinux.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:01:30.994 INFO:teuthology.orchestra.run.gibba013.stdout: jq.x86_64 0:1.6-2.el7 2021-02-10T11:01:30.995 INFO:teuthology.orchestra.run.gibba013.stdout: keyutils-libs-devel.x86_64 0:1.5.8-3.el7 2021-02-10T11:01:30.995 INFO:teuthology.orchestra.run.gibba013.stdout: krb5-devel.x86_64 0:1.15.1-50.el7 2021-02-10T11:01:30.995 INFO:teuthology.orchestra.run.gibba013.stdout: libbabeltrace.x86_64 0:1.2.4-3.el7 2021-02-10T11:01:30.995 INFO:teuthology.orchestra.run.gibba013.stdout: libcom_err-devel.x86_64 0:1.42.9-19.el7 2021-02-10T11:01:30.995 INFO:teuthology.orchestra.run.gibba013.stdout: liboath.x86_64 0:2.6.2-1.el7 2021-02-10T11:01:30.995 INFO:teuthology.orchestra.run.gibba013.stdout: librabbitmq.x86_64 0:0.8.0-3.el7 2021-02-10T11:01:30.995 INFO:teuthology.orchestra.run.gibba013.stdout: librados-devel.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:01:30.995 INFO:teuthology.orchestra.run.gibba013.stdout: libradosstriper1.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:01:30.995 INFO:teuthology.orchestra.run.gibba013.stdout: librdkafka.x86_64 0:0.11.5-1.el7 2021-02-10T11:01:30.995 INFO:teuthology.orchestra.run.gibba013.stdout: librgw2.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:01:30.996 INFO:teuthology.orchestra.run.gibba013.stdout: libselinux-devel.x86_64 0:2.5-15.el7 2021-02-10T11:01:30.996 INFO:teuthology.orchestra.run.gibba013.stdout: libsepol-devel.x86_64 0:2.5-10.el7 2021-02-10T11:01:30.996 INFO:teuthology.orchestra.run.gibba013.stdout: libverto-devel.x86_64 0:0.2.5-4.el7 2021-02-10T11:01:30.996 INFO:teuthology.orchestra.run.gibba013.stdout: lttng-ust.x86_64 0:2.4.1-4.el7 2021-02-10T11:01:30.996 INFO:teuthology.orchestra.run.gibba013.stdout: numpy-f2py.x86_64 1:1.7.1-13.el7 2021-02-10T11:01:30.996 INFO:teuthology.orchestra.run.gibba013.stdout: oniguruma.x86_64 0:6.8.2-1.el7 2021-02-10T11:01:30.996 INFO:teuthology.orchestra.run.gibba013.stdout: pcre-devel.x86_64 0:8.32-17.el7 2021-02-10T11:01:30.996 INFO:teuthology.orchestra.run.gibba013.stdout: python-beaker.noarch 0:1.5.4-10.el7 2021-02-10T11:01:30.996 INFO:teuthology.orchestra.run.gibba013.stdout: python-cachetools.noarch 0:1.0.3-1.el7 2021-02-10T11:01:30.996 INFO:teuthology.orchestra.run.gibba013.stdout: python-ceph-argparse.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:01:30.997 INFO:teuthology.orchestra.run.gibba013.stdout: python-cephfs.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:01:30.997 INFO:teuthology.orchestra.run.gibba013.stdout: python-cherrypy.noarch 0:3.2.2-4.el7 2021-02-10T11:01:30.997 INFO:teuthology.orchestra.run.gibba013.stdout: python-execnet.noarch 0:1.7.1-3.el7 2021-02-10T11:01:30.997 INFO:teuthology.orchestra.run.gibba013.stdout: python-jwcrypto.noarch 0:0.4.2-1.el7 2021-02-10T11:01:30.997 INFO:teuthology.orchestra.run.gibba013.stdout: python-mako.noarch 0:0.8.1-2.el7 2021-02-10T11:01:30.997 INFO:teuthology.orchestra.run.gibba013.stdout: python-markupsafe.x86_64 0:0.11-10.el7 2021-02-10T11:01:30.997 INFO:teuthology.orchestra.run.gibba013.stdout: python-paste.noarch 0:1.7.5.1-9.20111221hg1498.el7 2021-02-10T11:01:30.997 INFO:teuthology.orchestra.run.gibba013.stdout: python-pecan.noarch 0:0.4.5-2.el7 2021-02-10T11:01:30.997 INFO:teuthology.orchestra.run.gibba013.stdout: python-prettytable.noarch 0:0.7.2-3.el7 2021-02-10T11:01:30.997 INFO:teuthology.orchestra.run.gibba013.stdout: python-rados.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:01:30.998 INFO:teuthology.orchestra.run.gibba013.stdout: python-rbd.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:01:30.998 INFO:teuthology.orchestra.run.gibba013.stdout: python-repoze-lru.noarch 0:0.4-3.el7 2021-02-10T11:01:30.998 INFO:teuthology.orchestra.run.gibba013.stdout: python-requests.noarch 0:2.6.0-10.el7 2021-02-10T11:01:30.998 INFO:teuthology.orchestra.run.gibba013.stdout: python-rgw.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:01:30.998 INFO:teuthology.orchestra.run.gibba013.stdout: python-routes.noarch 0:1.13-2.el7 2021-02-10T11:01:30.998 INFO:teuthology.orchestra.run.gibba013.stdout: python-simplegeneric.noarch 0:0.8-7.el7 2021-02-10T11:01:30.998 INFO:teuthology.orchestra.run.gibba013.stdout: python-singledispatch.noarch 0:3.4.0.2-2.el7 2021-02-10T11:01:30.998 INFO:teuthology.orchestra.run.gibba013.stdout: python-tempita.noarch 0:0.5.1-6.el7 2021-02-10T11:01:30.999 INFO:teuthology.orchestra.run.gibba013.stdout: python-urllib3.noarch 0:1.10.2-7.el7 2021-02-10T11:01:30.999 INFO:teuthology.orchestra.run.gibba013.stdout: python-webob.noarch 0:1.2.3-7.el7 2021-02-10T11:01:30.999 INFO:teuthology.orchestra.run.gibba013.stdout: python-websocket-client.noarch 0:0.56.0-3.git3c25814.el7 2021-02-10T11:01:30.999 INFO:teuthology.orchestra.run.gibba013.stdout: python-webtest.noarch 0:1.3.4-6.el7 2021-02-10T11:01:30.999 INFO:teuthology.orchestra.run.gibba013.stdout: python-werkzeug.noarch 0:0.9.1-2.el7 2021-02-10T11:01:30.999 INFO:teuthology.orchestra.run.gibba013.stdout: python2-apipkg.noarch 0:1.5-5.el7 2021-02-10T11:01:31.000 INFO:teuthology.orchestra.run.gibba013.stdout: python2-bcrypt.x86_64 0:3.1.6-2.el7 2021-02-10T11:01:31.000 INFO:teuthology.orchestra.run.gibba013.stdout: python2-certifi.noarch 0:2018.10.15-5.el7 2021-02-10T11:01:31.000 INFO:teuthology.orchestra.run.gibba013.stdout: python2-google-auth.noarch 1:1.1.1-5.el7 2021-02-10T11:01:31.000 INFO:teuthology.orchestra.run.gibba013.stdout: python2-kubernetes.noarch 1:11.0.0-6.el7 2021-02-10T11:01:31.000 INFO:teuthology.orchestra.run.gibba013.stdout: python2-oauthlib.noarch 0:2.0.1-8.el7 2021-02-10T11:01:31.000 INFO:teuthology.orchestra.run.gibba013.stdout: python2-pyasn1-modules.noarch 0:0.1.9-7.el7 2021-02-10T11:01:31.000 INFO:teuthology.orchestra.run.gibba013.stdout: python2-remoto.noarch 0:1.0.0-1.el7 2021-02-10T11:01:31.000 INFO:teuthology.orchestra.run.gibba013.stdout: python2-requests-oauthlib.noarch 0:0.8.0-5.el7 2021-02-10T11:01:31.000 INFO:teuthology.orchestra.run.gibba013.stdout: python2-rsa.noarch 0:3.4.2-2.el7 2021-02-10T11:01:31.000 INFO:teuthology.orchestra.run.gibba013.stdout: python2-six.noarch 0:1.9.0-0.el7 2021-02-10T11:01:31.000 INFO:teuthology.orchestra.run.gibba013.stdout: python3-ceph-argparse.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:01:31.000 INFO:teuthology.orchestra.run.gibba013.stdout: python3-rados.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:01:31.001 INFO:teuthology.orchestra.run.gibba013.stdout: scipy.x86_64 0:0.12.1-6.el7 2021-02-10T11:01:31.001 INFO:teuthology.orchestra.run.gibba013.stdout: socat.x86_64 0:1.7.3.2-2.el7 2021-02-10T11:01:31.001 INFO:teuthology.orchestra.run.gibba013.stdout: suitesparse.x86_64 0:4.0.2-10.el7 2021-02-10T11:01:31.001 INFO:teuthology.orchestra.run.gibba013.stdout: tbb.x86_64 0:4.1-9.20130314.el7 2021-02-10T11:01:31.001 INFO:teuthology.orchestra.run.gibba013.stdout: userspace-rcu.x86_64 0:0.7.16-1.el7 2021-02-10T11:01:31.001 INFO:teuthology.orchestra.run.gibba013.stdout: zlib-devel.x86_64 0:1.2.7-19.el7_9 2021-02-10T11:01:31.001 INFO:teuthology.orchestra.run.gibba013.stdout: 2021-02-10T11:01:31.001 INFO:teuthology.orchestra.run.gibba013.stdout:Updated: 2021-02-10T11:01:31.001 INFO:teuthology.orchestra.run.gibba013.stdout: librados2.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:01:31.001 INFO:teuthology.orchestra.run.gibba013.stdout: librbd1.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:01:31.001 INFO:teuthology.orchestra.run.gibba013.stdout: 2021-02-10T11:01:31.002 INFO:teuthology.orchestra.run.gibba013.stdout:Dependency Updated: 2021-02-10T11:01:31.002 INFO:teuthology.orchestra.run.gibba013.stdout: openssl.x86_64 1:1.0.2k-21.el7_9 openssl-libs.x86_64 1:1.0.2k-21.el7_9 2021-02-10T11:01:31.002 INFO:teuthology.orchestra.run.gibba013.stdout: zlib.x86_64 0:1.2.7-19.el7_9 2021-02-10T11:01:31.002 INFO:teuthology.orchestra.run.gibba013.stdout: 2021-02-10T11:01:31.002 INFO:teuthology.orchestra.run.gibba013.stdout:Complete! 2021-02-10T11:01:31.114 DEBUG:teuthology.parallel:result is None 2021-02-10T11:01:31.115 DEBUG:teuthology.orchestra.run.gibba002:> sudo lsb_release -is 2021-02-10T11:01:31.487 INFO:teuthology.orchestra.run.gibba002.stdout:RedHatEnterpriseServer 2021-02-10T11:01:31.488 DEBUG:teuthology.misc:System to be installed: RedHatEnterpriseServer 2021-02-10T11:01:31.488 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F7%2Fx86_64&sha1=962fac13b1a781b8c7d25c1eed50d93813b8f5f0 2021-02-10T11:01:31.664 DEBUG:teuthology.orchestra.run.gibba002:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2021-02-10T11:01:32.017 INFO:teuthology.orchestra.run.gibba002.stdout:14.2.16-280.g962fac13.el7 2021-02-10T11:01:32.018 INFO:teuthology.packaging:The installed version of ceph is 14.2.16-280.g962fac13.el7 2021-02-10T11:01:32.018 INFO:teuthology.task.install:The correct ceph version 14.2.16-280.g962fac13 is installed. 2021-02-10T11:01:32.018 DEBUG:teuthology.orchestra.run.gibba013:> sudo lsb_release -is 2021-02-10T11:01:32.391 INFO:teuthology.orchestra.run.gibba013.stdout:RedHatEnterpriseServer 2021-02-10T11:01:32.392 DEBUG:teuthology.misc:System to be installed: RedHatEnterpriseServer 2021-02-10T11:01:32.393 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F7%2Fx86_64&sha1=962fac13b1a781b8c7d25c1eed50d93813b8f5f0 2021-02-10T11:01:32.557 DEBUG:teuthology.orchestra.run.gibba013:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2021-02-10T11:01:32.911 INFO:teuthology.orchestra.run.gibba013.stdout:14.2.16-280.g962fac13.el7 2021-02-10T11:01:32.911 INFO:teuthology.packaging:The installed version of ceph is 14.2.16-280.g962fac13.el7 2021-02-10T11:01:32.911 INFO:teuthology.task.install:The correct ceph version 14.2.16-280.g962fac13 is installed. 2021-02-10T11:01:32.912 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2021-02-10T11:01:32.914 DEBUG:teuthology.orchestra.run.gibba002:> set -ex 2021-02-10T11:01:32.914 DEBUG:teuthology.orchestra.run.gibba002:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2021-02-10T11:01:33.260 DEBUG:teuthology.orchestra.run.gibba013:> set -ex 2021-02-10T11:01:33.260 DEBUG:teuthology.orchestra.run.gibba013:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2021-02-10T11:01:33.604 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2021-02-10T11:01:33.606 DEBUG:teuthology.orchestra.run.gibba002:> set -ex 2021-02-10T11:01:33.606 DEBUG:teuthology.orchestra.run.gibba002:> sudo dd of=/usr/bin/daemon-helper 2021-02-10T11:01:33.949 DEBUG:teuthology.orchestra.run.gibba002:> sudo chmod a=rx -- /usr/bin/daemon-helper 2021-02-10T11:01:34.292 DEBUG:teuthology.orchestra.run.gibba013:> set -ex 2021-02-10T11:01:34.292 DEBUG:teuthology.orchestra.run.gibba013:> sudo dd of=/usr/bin/daemon-helper 2021-02-10T11:01:34.638 DEBUG:teuthology.orchestra.run.gibba013:> sudo chmod a=rx -- /usr/bin/daemon-helper 2021-02-10T11:01:34.981 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2021-02-10T11:01:34.981 DEBUG:teuthology.orchestra.run.gibba002:> set -ex 2021-02-10T11:01:34.981 DEBUG:teuthology.orchestra.run.gibba002:> sudo dd of=/usr/bin/adjust-ulimits 2021-02-10T11:01:35.323 DEBUG:teuthology.orchestra.run.gibba002:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2021-02-10T11:01:35.664 DEBUG:teuthology.orchestra.run.gibba013:> set -ex 2021-02-10T11:01:35.664 DEBUG:teuthology.orchestra.run.gibba013:> sudo dd of=/usr/bin/adjust-ulimits 2021-02-10T11:01:36.008 DEBUG:teuthology.orchestra.run.gibba013:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2021-02-10T11:01:36.351 INFO:teuthology.run_tasks:Running task ceph... 2021-02-10T11:01:36.423 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2021-02-10T11:01:36.424 DEBUG:teuthology.orchestra.run.gibba002:> sudo chmod 777 /var/log/ceph 2021-02-10T11:01:36.427 DEBUG:teuthology.orchestra.run.gibba013:> sudo chmod 777 /var/log/ceph 2021-02-10T11:01:36.770 INFO:tasks.ceph:Disabling ceph logrotate... 2021-02-10T11:01:36.770 DEBUG:teuthology.orchestra.run.gibba002:> sudo rm -f -- /etc/logrotate.d/ceph 2021-02-10T11:01:36.773 DEBUG:teuthology.orchestra.run.gibba013:> sudo rm -f -- /etc/logrotate.d/ceph 2021-02-10T11:01:37.115 INFO:tasks.ceph:Creating extra log directories... 2021-02-10T11:01:37.115 DEBUG:teuthology.orchestra.run.gibba002:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2021-02-10T11:01:37.118 DEBUG:teuthology.orchestra.run.gibba013:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2021-02-10T11:01:37.461 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2021-02-10T11:01:37.463 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2021-02-10T11:01:37.464 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2021-02-10T11:01:37.464 DEBUG:teuthology.orchestra.run.gibba002:> set -ex 2021-02-10T11:01:37.464 DEBUG:teuthology.orchestra.run.gibba002:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2021-02-10T11:01:37.797 DEBUG:teuthology.orchestra.run.gibba002:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2021-02-10T11:01:38.189 DEBUG:teuthology.orchestra.run.gibba002:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2021-02-10T11:01:38.532 DEBUG:teuthology.orchestra.run.gibba013:> set -ex 2021-02-10T11:01:38.532 DEBUG:teuthology.orchestra.run.gibba013:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2021-02-10T11:01:38.867 DEBUG:teuthology.orchestra.run.gibba013:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2021-02-10T11:01:39.260 DEBUG:teuthology.orchestra.run.gibba013:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2021-02-10T11:01:39.603 INFO:tasks.ceph:Creating ceph cluster ceph... 2021-02-10T11:01:39.603 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': ''}, 'mds': {'debug mds': 20, 'debug ms': 1, 'mds bal frag': True, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180}, 'mgr': {'debug client': 10, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(TOO_FEW_PGS\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'OSD full dropping all updates', 'OSD near full', 'pausewr flag', 'failsafe engaged, dropping updates', 'failsafe disengaged, no longer dropping', 'is full \\(reached quota', 'POOL_FULL', 'POOL_BACKFILLFULL'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2021-02-10T11:01:39.603 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/khiremat-2021-02-10_10:13:45-fs-wip-khiremat-mgr-authid-bp-7-nautilus-distro-basic-gibba/5872718', 'branch': 'wip-khiremat-mgr-authid-bp-7-nautilus', 'description': 'fs/basic_functional/{begin clusters/1-mds-4-client-coloc conf/{client mds mon osd} mount/fuse objectstore/bluestore-bitmap overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} supported-random-distros$/{rhel_7} tasks/volumes}', 'email': 'khiremat@redhat.com', 'first_in_suite': False, 'job_id': '5872718', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'gibba', 'name': 'khiremat-2021-02-10_10:13:45-fs-wip-khiremat-mgr-authid-bp-7-nautilus-distro-basic-gibba', 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 30}}, {'machine': {'disk': 200}}], 'os_type': 'rhel', 'os_version': '7.9', 'overrides': {'admin_socket': {'branch': 'wip-khiremat-mgr-authid-bp-7-nautilus'}, 'ceph': {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': ''}, 'mds': {'debug mds': 20, 'debug ms': 1, 'mds bal frag': True, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180}, 'mgr': {'debug client': 10, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(TOO_FEW_PGS\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'OSD full dropping all updates', 'OSD near full', 'pausewr flag', 'failsafe engaged, dropping updates', 'failsafe disengaged, no longer dropping', 'is full \\(reached quota', 'POOL_FULL', 'POOL_BACKFILLFULL'], 'sha1': '962fac13b1a781b8c7d25c1eed50d93813b8f5f0'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'sha1': '962fac13b1a781b8c7d25c1eed50d93813b8f5f0'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-khiremat-mgr-authid-bp-7-nautilus', 'sha1': '962fac13b1a781b8c7d25c1eed50d93813b8f5f0'}}, 'owner': 'scheduled_khiremat@teuthology', 'priority': 50, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.1'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.2', 'client.3']], 'sha1': '962fac13b1a781b8c7d25c1eed50d93813b8f5f0', 'sleep_before_teardown': 0, 'suite': 'fs', 'suite_branch': 'wip-khiremat-mgr-authid-bp-7-nautilus', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_962fac13b1a781b8c7d25c1eed50d93813b8f5f0/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': '962fac13b1a781b8c7d25c1eed50d93813b8f5f0', 'targets': {'gibba002.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2bmtFOE1KHoyhHloefVeAs5eOjJHc91Zkg84KoprB2NN75v56Zrnct2fAo7z4L3T2CG9e97gcvCwjA9SbXGMyASI8Qoinc2p9FRje3mos8DriJf+4yWh6E9sZUlAyVkF5xSKS8Z9eS/FceXbqWb42s+KOXCt3AVNZFrkBlu8jyeUFz5BCaEl+6CHQqxzJCWa/KpKGgqFCQf1N0DV9eAXBCBOwo5jpVcgvq8oEq28fZmx6b8Yr77QeFDj0fcn32I57e0aQTE13jmc2lAZbSUWM/HPyzVefFxIWfg/YXaA5MWE1+gEUXpc5k3cSmyq6ZNXdpI3EaqGHUuJKxNaDR2vT', 'gibba013.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBkalw17+zsGdqNpcFqATXsFUNbxrsotr+ltN6gwKELl9WUVYSNUgi/nlVJOdtCaJzywCFRbScdjK67BCw40xZ6ReZuBXgznlPP2VAyakp3rjQYVNZdQEuMhLOVaQiradixCkYypPJV3d+RwTL5yj29g/05PRqFiSDcsH1vqr4KwKzqd14lZa9XxMzUB3oKk0XlhF4N2+WWwc1waQ5Oy8pAifhoLcOaHC+eRwvTwzBbMlmJFMoDOq/ocQd/MgfcMR4wp3QwRbGz4eIJ753fuHtAhemPdv83QPpudbH+1/Q3sH5jL1D/KezMkgRG9CWngAHRGr7Ea0zHPspcxlIpV4N'}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell'], 'rpm': ['python3-cephfs']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel']}, 'sha1': '962fac13b1a781b8c7d25c1eed50d93813b8f5f0'}}, {'ceph': None}, {'ceph-fuse': None}, {'cephfs_test_runner': {'fail_on_skip': False, 'modules': ['tasks.cephfs.test_volumes']}}], 'teuthology_branch': 'master', 'timestamp': '2021-02-10_10:13:45', 'tube': 'gibba', 'user': 'khiremat', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.gibba.29737'} 2021-02-10T11:01:39.603 DEBUG:teuthology.orchestra.run.gibba002:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2021-02-10T11:01:39.608 DEBUG:teuthology.orchestra.run.gibba013:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2021-02-10T11:01:39.941 DEBUG:teuthology.orchestra.run.gibba002:> sudo install -d -m0777 -- /var/run/ceph 2021-02-10T11:01:39.944 DEBUG:teuthology.orchestra.run.gibba013:> sudo install -d -m0777 -- /var/run/ceph 2021-02-10T11:01:40.287 DEBUG:teuthology.orchestra.run.gibba002:> set -ex 2021-02-10T11:01:40.287 DEBUG:teuthology.orchestra.run.gibba002:> dd if=/scratch_devs of=/dev/stdout 2021-02-10T11:01:40.766 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2021-02-10T11:01:40.766 DEBUG:teuthology.orchestra.run.gibba002:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2021-02-10T11:01:41.097 INFO:teuthology.orchestra.run.gibba002.stdout: File: ‘/dev/vg_nvme/lv_1’ -> ‘../dm-0’ 2021-02-10T11:01:41.098 INFO:teuthology.orchestra.run.gibba002.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-02-10T11:01:41.098 INFO:teuthology.orchestra.run.gibba002.stdout:Device: 5h/5d Inode: 93435 Links: 1 2021-02-10T11:01:41.098 INFO:teuthology.orchestra.run.gibba002.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-02-10T11:01:41.098 INFO:teuthology.orchestra.run.gibba002.stdout:Context: system_u:object_r:device_t:s0 2021-02-10T11:01:41.098 INFO:teuthology.orchestra.run.gibba002.stdout:Access: 2021-02-10 11:01:06.140037413 +0000 2021-02-10T11:01:41.098 INFO:teuthology.orchestra.run.gibba002.stdout:Modify: 2021-02-10 10:58:27.364520187 +0000 2021-02-10T11:01:41.098 INFO:teuthology.orchestra.run.gibba002.stdout:Change: 2021-02-10 10:58:27.364520187 +0000 2021-02-10T11:01:41.098 INFO:teuthology.orchestra.run.gibba002.stdout: Birth: - 2021-02-10T11:01:41.107 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:01:41.107 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:01:41.107 INFO:teuthology.orchestra.run.gibba002.stderr:512 bytes (512 B) copied, 0.000202211 s, 2.5 MB/s 2021-02-10T11:01:41.110 DEBUG:teuthology.orchestra.run.gibba002:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2021-02-10T11:01:41.443 INFO:teuthology.orchestra.run.gibba002.stdout: File: ‘/dev/vg_nvme/lv_2’ -> ‘../dm-1’ 2021-02-10T11:01:41.443 INFO:teuthology.orchestra.run.gibba002.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-02-10T11:01:41.443 INFO:teuthology.orchestra.run.gibba002.stdout:Device: 5h/5d Inode: 100469 Links: 1 2021-02-10T11:01:41.443 INFO:teuthology.orchestra.run.gibba002.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-02-10T11:01:41.443 INFO:teuthology.orchestra.run.gibba002.stdout:Context: system_u:object_r:device_t:s0 2021-02-10T11:01:41.443 INFO:teuthology.orchestra.run.gibba002.stdout:Access: 2021-02-10 11:01:06.140037413 +0000 2021-02-10T11:01:41.443 INFO:teuthology.orchestra.run.gibba002.stdout:Modify: 2021-02-10 10:58:27.910516546 +0000 2021-02-10T11:01:41.443 INFO:teuthology.orchestra.run.gibba002.stdout:Change: 2021-02-10 10:58:27.910516546 +0000 2021-02-10T11:01:41.444 INFO:teuthology.orchestra.run.gibba002.stdout: Birth: - 2021-02-10T11:01:41.452 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:01:41.452 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:01:41.452 INFO:teuthology.orchestra.run.gibba002.stderr:512 bytes (512 B) copied, 0.000176008 s, 2.9 MB/s 2021-02-10T11:01:41.454 DEBUG:teuthology.orchestra.run.gibba002:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2021-02-10T11:01:41.788 INFO:teuthology.orchestra.run.gibba002.stdout: File: ‘/dev/vg_nvme/lv_3’ -> ‘../dm-2’ 2021-02-10T11:01:41.788 INFO:teuthology.orchestra.run.gibba002.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-02-10T11:01:41.788 INFO:teuthology.orchestra.run.gibba002.stdout:Device: 5h/5d Inode: 89760 Links: 1 2021-02-10T11:01:41.788 INFO:teuthology.orchestra.run.gibba002.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-02-10T11:01:41.788 INFO:teuthology.orchestra.run.gibba002.stdout:Context: system_u:object_r:device_t:s0 2021-02-10T11:01:41.788 INFO:teuthology.orchestra.run.gibba002.stdout:Access: 2021-02-10 11:01:06.140037413 +0000 2021-02-10T11:01:41.788 INFO:teuthology.orchestra.run.gibba002.stdout:Modify: 2021-02-10 10:58:28.461512872 +0000 2021-02-10T11:01:41.789 INFO:teuthology.orchestra.run.gibba002.stdout:Change: 2021-02-10 10:58:28.461512872 +0000 2021-02-10T11:01:41.789 INFO:teuthology.orchestra.run.gibba002.stdout: Birth: - 2021-02-10T11:01:41.797 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:01:41.797 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:01:41.797 INFO:teuthology.orchestra.run.gibba002.stderr:512 bytes (512 B) copied, 0.000168442 s, 3.0 MB/s 2021-02-10T11:01:41.799 DEBUG:teuthology.orchestra.run.gibba002:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2021-02-10T11:01:42.133 INFO:teuthology.orchestra.run.gibba002.stdout: File: ‘/dev/vg_nvme/lv_4’ -> ‘../dm-3’ 2021-02-10T11:01:42.133 INFO:teuthology.orchestra.run.gibba002.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-02-10T11:01:42.133 INFO:teuthology.orchestra.run.gibba002.stdout:Device: 5h/5d Inode: 102426 Links: 1 2021-02-10T11:01:42.133 INFO:teuthology.orchestra.run.gibba002.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-02-10T11:01:42.133 INFO:teuthology.orchestra.run.gibba002.stdout:Context: system_u:object_r:device_t:s0 2021-02-10T11:01:42.133 INFO:teuthology.orchestra.run.gibba002.stdout:Access: 2021-02-10 11:01:06.140037413 +0000 2021-02-10T11:01:42.134 INFO:teuthology.orchestra.run.gibba002.stdout:Modify: 2021-02-10 10:58:29.018509159 +0000 2021-02-10T11:01:42.134 INFO:teuthology.orchestra.run.gibba002.stdout:Change: 2021-02-10 10:58:29.018509159 +0000 2021-02-10T11:01:42.134 INFO:teuthology.orchestra.run.gibba002.stdout: Birth: - 2021-02-10T11:01:42.142 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:01:42.142 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:01:42.142 INFO:teuthology.orchestra.run.gibba002.stderr:512 bytes (512 B) copied, 0.000189455 s, 2.7 MB/s 2021-02-10T11:01:42.144 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2021-02-10T11:01:42.144 DEBUG:teuthology.orchestra.run.gibba013:> set -ex 2021-02-10T11:01:42.144 DEBUG:teuthology.orchestra.run.gibba013:> dd if=/scratch_devs of=/dev/stdout 2021-02-10T11:01:42.479 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2021-02-10T11:01:42.479 DEBUG:teuthology.orchestra.run.gibba013:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2021-02-10T11:01:42.813 INFO:teuthology.orchestra.run.gibba013.stdout: File: ‘/dev/vg_nvme/lv_1’ -> ‘../dm-0’ 2021-02-10T11:01:42.813 INFO:teuthology.orchestra.run.gibba013.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-02-10T11:01:42.813 INFO:teuthology.orchestra.run.gibba013.stdout:Device: 5h/5d Inode: 89987 Links: 1 2021-02-10T11:01:42.813 INFO:teuthology.orchestra.run.gibba013.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-02-10T11:01:42.813 INFO:teuthology.orchestra.run.gibba013.stdout:Context: system_u:object_r:device_t:s0 2021-02-10T11:01:42.813 INFO:teuthology.orchestra.run.gibba013.stdout:Access: 2021-02-10 11:01:11.464673216 +0000 2021-02-10T11:01:42.813 INFO:teuthology.orchestra.run.gibba013.stdout:Modify: 2021-02-10 10:58:27.330215405 +0000 2021-02-10T11:01:42.813 INFO:teuthology.orchestra.run.gibba013.stdout:Change: 2021-02-10 10:58:27.330215405 +0000 2021-02-10T11:01:42.813 INFO:teuthology.orchestra.run.gibba013.stdout: Birth: - 2021-02-10T11:01:42.822 INFO:teuthology.orchestra.run.gibba013.stderr:1+0 records in 2021-02-10T11:01:42.822 INFO:teuthology.orchestra.run.gibba013.stderr:1+0 records out 2021-02-10T11:01:42.822 INFO:teuthology.orchestra.run.gibba013.stderr:512 bytes (512 B) copied, 0.000206943 s, 2.5 MB/s 2021-02-10T11:01:42.824 DEBUG:teuthology.orchestra.run.gibba013:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2021-02-10T11:01:43.158 INFO:teuthology.orchestra.run.gibba013.stdout: File: ‘/dev/vg_nvme/lv_2’ -> ‘../dm-1’ 2021-02-10T11:01:43.158 INFO:teuthology.orchestra.run.gibba013.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-02-10T11:01:43.158 INFO:teuthology.orchestra.run.gibba013.stdout:Device: 5h/5d Inode: 62035 Links: 1 2021-02-10T11:01:43.158 INFO:teuthology.orchestra.run.gibba013.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-02-10T11:01:43.158 INFO:teuthology.orchestra.run.gibba013.stdout:Context: system_u:object_r:device_t:s0 2021-02-10T11:01:43.158 INFO:teuthology.orchestra.run.gibba013.stdout:Access: 2021-02-10 11:01:11.464673216 +0000 2021-02-10T11:01:43.158 INFO:teuthology.orchestra.run.gibba013.stdout:Modify: 2021-02-10 10:58:27.899217098 +0000 2021-02-10T11:01:43.158 INFO:teuthology.orchestra.run.gibba013.stdout:Change: 2021-02-10 10:58:27.899217098 +0000 2021-02-10T11:01:43.158 INFO:teuthology.orchestra.run.gibba013.stdout: Birth: - 2021-02-10T11:01:43.167 INFO:teuthology.orchestra.run.gibba013.stderr:1+0 records in 2021-02-10T11:01:43.167 INFO:teuthology.orchestra.run.gibba013.stderr:1+0 records out 2021-02-10T11:01:43.167 INFO:teuthology.orchestra.run.gibba013.stderr:512 bytes (512 B) copied, 0.000191014 s, 2.7 MB/s 2021-02-10T11:01:43.169 DEBUG:teuthology.orchestra.run.gibba013:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2021-02-10T11:01:43.503 INFO:teuthology.orchestra.run.gibba013.stdout: File: ‘/dev/vg_nvme/lv_3’ -> ‘../dm-2’ 2021-02-10T11:01:43.503 INFO:teuthology.orchestra.run.gibba013.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-02-10T11:01:43.503 INFO:teuthology.orchestra.run.gibba013.stdout:Device: 5h/5d Inode: 41734 Links: 1 2021-02-10T11:01:43.503 INFO:teuthology.orchestra.run.gibba013.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-02-10T11:01:43.503 INFO:teuthology.orchestra.run.gibba013.stdout:Context: system_u:object_r:device_t:s0 2021-02-10T11:01:43.503 INFO:teuthology.orchestra.run.gibba013.stdout:Access: 2021-02-10 11:01:11.464673216 +0000 2021-02-10T11:01:43.503 INFO:teuthology.orchestra.run.gibba013.stdout:Modify: 2021-02-10 10:58:28.485218842 +0000 2021-02-10T11:01:43.503 INFO:teuthology.orchestra.run.gibba013.stdout:Change: 2021-02-10 10:58:28.485218842 +0000 2021-02-10T11:01:43.504 INFO:teuthology.orchestra.run.gibba013.stdout: Birth: - 2021-02-10T11:01:43.512 INFO:teuthology.orchestra.run.gibba013.stderr:1+0 records in 2021-02-10T11:01:43.512 INFO:teuthology.orchestra.run.gibba013.stderr:1+0 records out 2021-02-10T11:01:43.512 INFO:teuthology.orchestra.run.gibba013.stderr:512 bytes (512 B) copied, 0.000195417 s, 2.6 MB/s 2021-02-10T11:01:43.514 DEBUG:teuthology.orchestra.run.gibba013:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2021-02-10T11:01:43.847 INFO:teuthology.orchestra.run.gibba013.stdout: File: ‘/dev/vg_nvme/lv_4’ -> ‘../dm-3’ 2021-02-10T11:01:43.847 INFO:teuthology.orchestra.run.gibba013.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-02-10T11:01:43.847 INFO:teuthology.orchestra.run.gibba013.stdout:Device: 5h/5d Inode: 62063 Links: 1 2021-02-10T11:01:43.847 INFO:teuthology.orchestra.run.gibba013.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-02-10T11:01:43.847 INFO:teuthology.orchestra.run.gibba013.stdout:Context: system_u:object_r:device_t:s0 2021-02-10T11:01:43.847 INFO:teuthology.orchestra.run.gibba013.stdout:Access: 2021-02-10 11:01:11.464673216 +0000 2021-02-10T11:01:43.847 INFO:teuthology.orchestra.run.gibba013.stdout:Modify: 2021-02-10 10:58:29.027220455 +0000 2021-02-10T11:01:43.847 INFO:teuthology.orchestra.run.gibba013.stdout:Change: 2021-02-10 10:58:29.027220455 +0000 2021-02-10T11:01:43.848 INFO:teuthology.orchestra.run.gibba013.stdout: Birth: - 2021-02-10T11:01:43.856 INFO:teuthology.orchestra.run.gibba013.stderr:1+0 records in 2021-02-10T11:01:43.856 INFO:teuthology.orchestra.run.gibba013.stderr:1+0 records out 2021-02-10T11:01:43.856 INFO:teuthology.orchestra.run.gibba013.stderr:512 bytes (512 B) copied, 0.000184634 s, 2.8 MB/s 2021-02-10T11:01:43.858 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2021-02-10T11:01:43.858 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@gibba002.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@gibba013.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2021-02-10T11:01:43.858 INFO:tasks.ceph:Generating config... 2021-02-10T11:01:43.869 INFO:tasks.ceph:[client] client mount timeout = 600 2021-02-10T11:01:43.869 INFO:tasks.ceph:[client] debug client = 20 2021-02-10T11:01:43.869 INFO:tasks.ceph:[client] debug ms = 1 2021-02-10T11:01:43.869 INFO:tasks.ceph:[client] pid file = 2021-02-10T11:01:43.869 INFO:tasks.ceph:[mds] debug mds = 20 2021-02-10T11:01:43.869 INFO:tasks.ceph:[mds] debug ms = 1 2021-02-10T11:01:43.869 INFO:tasks.ceph:[mds] mds bal frag = True 2021-02-10T11:01:43.869 INFO:tasks.ceph:[mds] mds bal fragment size max = 10000 2021-02-10T11:01:43.869 INFO:tasks.ceph:[mds] mds bal merge size = 5 2021-02-10T11:01:43.870 INFO:tasks.ceph:[mds] mds bal split bits = 3 2021-02-10T11:01:43.870 INFO:tasks.ceph:[mds] mds bal split size = 100 2021-02-10T11:01:43.870 INFO:tasks.ceph:[mds] mds debug frag = True 2021-02-10T11:01:43.870 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2021-02-10T11:01:43.870 INFO:tasks.ceph:[mds] mds op complaint time = 180 2021-02-10T11:01:43.870 INFO:tasks.ceph:[mds] mds verify scatter = True 2021-02-10T11:01:43.870 INFO:tasks.ceph:[mds] osd op complaint time = 180 2021-02-10T11:01:43.870 INFO:tasks.ceph:[mgr] debug client = 10 2021-02-10T11:01:43.870 INFO:tasks.ceph:[mgr] debug mgr = 20 2021-02-10T11:01:43.870 INFO:tasks.ceph:[mgr] debug ms = 1 2021-02-10T11:01:43.870 INFO:tasks.ceph:[mon] debug mon = 20 2021-02-10T11:01:43.871 INFO:tasks.ceph:[mon] debug ms = 1 2021-02-10T11:01:43.871 INFO:tasks.ceph:[mon] debug paxos = 20 2021-02-10T11:01:43.871 INFO:tasks.ceph:[mon] mon op complaint time = 120 2021-02-10T11:01:43.871 INFO:tasks.ceph:[osd] bdev async discard = True 2021-02-10T11:01:43.871 INFO:tasks.ceph:[osd] bdev enable discard = True 2021-02-10T11:01:43.871 INFO:tasks.ceph:[osd] bluestore allocator = bitmap 2021-02-10T11:01:43.871 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2021-02-10T11:01:43.871 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2021-02-10T11:01:43.871 INFO:tasks.ceph:[osd] debug bluefs = 20 2021-02-10T11:01:43.871 INFO:tasks.ceph:[osd] debug bluestore = 20 2021-02-10T11:01:43.871 INFO:tasks.ceph:[osd] debug filestore = 20 2021-02-10T11:01:43.871 INFO:tasks.ceph:[osd] debug journal = 20 2021-02-10T11:01:43.872 INFO:tasks.ceph:[osd] debug ms = 20 2021-02-10T11:01:43.872 INFO:tasks.ceph:[osd] debug osd = 25 2021-02-10T11:01:43.872 INFO:tasks.ceph:[osd] debug rocksdb = 10 2021-02-10T11:01:43.872 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2021-02-10T11:01:43.872 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2021-02-10T11:01:43.872 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2021-02-10T11:01:43.872 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2021-02-10T11:01:43.872 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2021-02-10T11:01:43.872 INFO:tasks.ceph:[osd] osd op complaint time = 180 2021-02-10T11:01:43.872 INFO:tasks.ceph:Setting up mon.a... 2021-02-10T11:01:43.873 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2021-02-10T11:01:44.268 INFO:teuthology.orchestra.run.gibba002.stdout:creating /etc/ceph/ceph.keyring 2021-02-10T11:01:44.270 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2021-02-10T11:01:44.638 DEBUG:teuthology.orchestra.run.gibba002:> sudo chmod 0644 /etc/ceph/ceph.keyring 2021-02-10T11:01:44.981 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.2.102'), ('mon.b', '172.21.2.113'), ('mon.c', '[v2:172.21.2.113:3301,v1:172.21.2.113:6790]')] 2021-02-10T11:01:44.981 DEBUG:teuthology.orchestra.run.gibba002:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool --create --clobber --enable-all-features --add a 172.21.2.102 --add b 172.21.2.113 --addv c '[v2:172.21.2.113:3301,v1:172.21.2.113:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2021-02-10T11:01:45.347 INFO:teuthology.orchestra.run.gibba002.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2021-02-10T11:01:45.347 INFO:teuthology.orchestra.run.gibba002.stdout:monmaptool: generated fsid 68db30ea-dbce-4aef-a6df-f01c6fea0513 2021-02-10T11:01:45.347 INFO:teuthology.orchestra.run.gibba002.stdout:epoch 0 2021-02-10T11:01:45.348 INFO:teuthology.orchestra.run.gibba002.stdout:fsid 68db30ea-dbce-4aef-a6df-f01c6fea0513 2021-02-10T11:01:45.348 INFO:teuthology.orchestra.run.gibba002.stdout:last_changed 2021-02-10 11:01:45.348992 2021-02-10T11:01:45.348 INFO:teuthology.orchestra.run.gibba002.stdout:created 2021-02-10 11:01:45.348992 2021-02-10T11:01:45.348 INFO:teuthology.orchestra.run.gibba002.stdout:min_mon_release 0 (unknown) 2021-02-10T11:01:45.348 INFO:teuthology.orchestra.run.gibba002.stdout:0: [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] mon.a 2021-02-10T11:01:45.348 INFO:teuthology.orchestra.run.gibba002.stdout:1: [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] mon.b 2021-02-10T11:01:45.348 INFO:teuthology.orchestra.run.gibba002.stdout:2: [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] mon.c 2021-02-10T11:01:45.348 INFO:teuthology.orchestra.run.gibba002.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2021-02-10T11:01:45.349 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 68db30ea-dbce-4aef-a6df-f01c6fea0513... 2021-02-10T11:01:45.350 DEBUG:teuthology.orchestra.run.gibba002:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2021-02-10T11:01:45.353 DEBUG:teuthology.orchestra.run.gibba013:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2021-02-10T11:01:45.712 INFO:teuthology.orchestra.run.gibba002.stdout:[global] 2021-02-10T11:01:45.712 INFO:teuthology.orchestra.run.gibba002.stdout: chdir = "" 2021-02-10T11:01:45.712 INFO:teuthology.orchestra.run.gibba002.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2021-02-10T11:01:45.712 INFO:teuthology.orchestra.run.gibba002.stdout: auth supported = cephx 2021-02-10T11:01:45.712 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:01:45.712 INFO:teuthology.orchestra.run.gibba002.stdout: filestore xattr use omap = true 2021-02-10T11:01:45.712 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:01:45.712 INFO:teuthology.orchestra.run.gibba002.stdout: mon clock drift allowed = 1.000 2021-02-10T11:01:45.712 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:01:45.713 INFO:teuthology.orchestra.run.gibba002.stdout: osd crush chooseleaf type = 0 2021-02-10T11:01:45.713 INFO:teuthology.orchestra.run.gibba002.stdout: auth debug = true 2021-02-10T11:01:45.713 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:01:45.713 INFO:teuthology.orchestra.run.gibba002.stdout: ms die on old message = true 2021-02-10T11:01:45.713 INFO:teuthology.orchestra.run.gibba002.stdout: ms die on bug = true 2021-02-10T11:01:45.713 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:01:45.713 INFO:teuthology.orchestra.run.gibba002.stdout: mon max pg per osd = 10000 # >= luminous 2021-02-10T11:01:45.713 INFO:teuthology.orchestra.run.gibba002.stdout: mon pg warn max object skew = 0 2021-02-10T11:01:45.713 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:01:45.713 INFO:teuthology.orchestra.run.gibba002.stdout: osd pool default size = 2 2021-02-10T11:01:45.713 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:01:45.713 INFO:teuthology.orchestra.run.gibba002.stdout: mon osd allow primary affinity = true 2021-02-10T11:01:45.713 INFO:teuthology.orchestra.run.gibba002.stdout: mon osd allow pg remap = true 2021-02-10T11:01:45.713 INFO:teuthology.orchestra.run.gibba002.stdout: mon warn on legacy crush tunables = false 2021-02-10T11:01:45.714 INFO:teuthology.orchestra.run.gibba002.stdout: mon warn on crush straw calc version zero = false 2021-02-10T11:01:45.714 INFO:teuthology.orchestra.run.gibba002.stdout: mon warn on no sortbitwise = false 2021-02-10T11:01:45.714 INFO:teuthology.orchestra.run.gibba002.stdout: mon warn on osd down out interval zero = false 2021-02-10T11:01:45.714 INFO:teuthology.orchestra.run.gibba002.stdout: mon warn on too few osds = false 2021-02-10T11:01:45.714 INFO:teuthology.orchestra.run.gibba002.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2021-02-10T11:01:45.714 INFO:teuthology.orchestra.run.gibba002.stdout: mon_warn_on_pool_no_redundancy = false 2021-02-10T11:01:45.714 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:01:45.714 INFO:teuthology.orchestra.run.gibba002.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2021-02-10T11:01:45.714 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:01:45.714 INFO:teuthology.orchestra.run.gibba002.stdout: osd default data pool replay window = 5 2021-02-10T11:01:45.714 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:01:45.714 INFO:teuthology.orchestra.run.gibba002.stdout: mon allow pool delete = true 2021-02-10T11:01:45.714 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:01:45.715 INFO:teuthology.orchestra.run.gibba002.stdout: mon cluster log file level = debug 2021-02-10T11:01:45.715 INFO:teuthology.orchestra.run.gibba002.stdout: debug asserts on shutdown = true 2021-02-10T11:01:45.715 INFO:teuthology.orchestra.run.gibba002.stdout: mon health detail to clog = false 2021-02-10T11:01:45.715 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:01:45.715 INFO:teuthology.orchestra.run.gibba002.stdout: # we see this fail in qa on *nautilus*; bump up retries 2021-02-10T11:01:45.715 INFO:teuthology.orchestra.run.gibba002.stdout: mon_client_directed_command_retry = 4 2021-02-10T11:01:45.715 INFO:teuthology.orchestra.run.gibba002.stdout: mon host = "172.21.2.102,172.21.2.113,[v2:172.21.2.113:3301,v1:172.21.2.113:6790]" 2021-02-10T11:01:45.715 INFO:teuthology.orchestra.run.gibba002.stdout: fsid = 68db30ea-dbce-4aef-a6df-f01c6fea0513 2021-02-10T11:01:45.715 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:01:45.715 INFO:teuthology.orchestra.run.gibba002.stdout:[osd] 2021-02-10T11:01:45.715 INFO:teuthology.orchestra.run.gibba002.stdout: osd journal size = 100 2021-02-10T11:01:45.715 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:01:45.715 INFO:teuthology.orchestra.run.gibba002.stdout: osd scrub load threshold = 5.0 2021-02-10T11:01:45.715 INFO:teuthology.orchestra.run.gibba002.stdout: osd scrub max interval = 600 2021-02-10T11:01:45.715 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:01:45.716 INFO:teuthology.orchestra.run.gibba002.stdout: osd recover clone overlap = true 2021-02-10T11:01:45.716 INFO:teuthology.orchestra.run.gibba002.stdout: osd recovery max chunk = 1048576 2021-02-10T11:01:45.716 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:01:45.716 INFO:teuthology.orchestra.run.gibba002.stdout: osd debug shutdown = true 2021-02-10T11:01:45.716 INFO:teuthology.orchestra.run.gibba002.stdout: osd debug op order = true 2021-02-10T11:01:45.716 INFO:teuthology.orchestra.run.gibba002.stdout: osd debug verify stray on activate = true 2021-02-10T11:01:45.716 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:01:45.716 INFO:teuthology.orchestra.run.gibba002.stdout: osd open classes on start = true 2021-02-10T11:01:45.716 INFO:teuthology.orchestra.run.gibba002.stdout: osd debug pg log writeout = true 2021-02-10T11:01:45.716 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:01:45.716 INFO:teuthology.orchestra.run.gibba002.stdout: osd deep scrub update digest min age = 30 2021-02-10T11:01:45.716 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:01:45.716 INFO:teuthology.orchestra.run.gibba002.stdout: osd map max advance = 10 2021-02-10T11:01:45.717 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:01:45.717 INFO:teuthology.orchestra.run.gibba002.stdout: journal zero on create = true 2021-02-10T11:01:45.717 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:01:45.717 INFO:teuthology.orchestra.run.gibba002.stdout: filestore ondisk finisher threads = 3 2021-02-10T11:01:45.717 INFO:teuthology.orchestra.run.gibba002.stdout: filestore apply finisher threads = 3 2021-02-10T11:01:45.717 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:01:45.717 INFO:teuthology.orchestra.run.gibba002.stdout: bdev debug aio = true 2021-02-10T11:01:45.717 INFO:teuthology.orchestra.run.gibba002.stdout: osd debug misdirected ops = true 2021-02-10T11:01:45.717 INFO:teuthology.orchestra.run.gibba002.stdout: bdev async discard = True 2021-02-10T11:01:45.717 INFO:teuthology.orchestra.run.gibba002.stdout: bdev enable discard = True 2021-02-10T11:01:45.717 INFO:teuthology.orchestra.run.gibba002.stdout: bluestore allocator = bitmap 2021-02-10T11:01:45.717 INFO:teuthology.orchestra.run.gibba002.stdout: bluestore block size = 96636764160 2021-02-10T11:01:45.717 INFO:teuthology.orchestra.run.gibba002.stdout: bluestore fsck on mount = True 2021-02-10T11:01:45.717 INFO:teuthology.orchestra.run.gibba002.stdout: debug bluefs = 20 2021-02-10T11:01:45.717 INFO:teuthology.orchestra.run.gibba002.stdout: debug bluestore = 20 2021-02-10T11:01:45.718 INFO:teuthology.orchestra.run.gibba002.stdout: debug filestore = 20 2021-02-10T11:01:45.718 INFO:teuthology.orchestra.run.gibba002.stdout: debug journal = 20 2021-02-10T11:01:45.718 INFO:teuthology.orchestra.run.gibba002.stdout: debug ms = 20 2021-02-10T11:01:45.718 INFO:teuthology.orchestra.run.gibba002.stdout: debug osd = 25 2021-02-10T11:01:45.718 INFO:teuthology.orchestra.run.gibba002.stdout: debug rocksdb = 10 2021-02-10T11:01:45.718 INFO:teuthology.orchestra.run.gibba002.stdout: mon osd backfillfull_ratio = 0.85 2021-02-10T11:01:45.718 INFO:teuthology.orchestra.run.gibba002.stdout: mon osd full ratio = 0.9 2021-02-10T11:01:45.718 INFO:teuthology.orchestra.run.gibba002.stdout: mon osd nearfull ratio = 0.8 2021-02-10T11:01:45.718 INFO:teuthology.orchestra.run.gibba002.stdout: osd failsafe full ratio = 0.95 2021-02-10T11:01:45.718 INFO:teuthology.orchestra.run.gibba002.stdout: osd objectstore = bluestore 2021-02-10T11:01:45.718 INFO:teuthology.orchestra.run.gibba002.stdout: osd op complaint time = 180 2021-02-10T11:01:45.718 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:01:45.718 INFO:teuthology.orchestra.run.gibba002.stdout:[mgr] 2021-02-10T11:01:45.718 INFO:teuthology.orchestra.run.gibba002.stdout: debug ms = 1 2021-02-10T11:01:45.719 INFO:teuthology.orchestra.run.gibba002.stdout: debug mgr = 20 2021-02-10T11:01:45.719 INFO:teuthology.orchestra.run.gibba002.stdout: debug mon = 20 2021-02-10T11:01:45.719 INFO:teuthology.orchestra.run.gibba002.stdout: debug auth = 20 2021-02-10T11:01:45.719 INFO:teuthology.orchestra.run.gibba002.stdout: mon reweight min pgs per osd = 4 2021-02-10T11:01:45.719 INFO:teuthology.orchestra.run.gibba002.stdout: mon reweight min bytes per osd = 10 2021-02-10T11:01:45.719 INFO:teuthology.orchestra.run.gibba002.stdout: debug client = 10 2021-02-10T11:01:45.719 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:01:45.719 INFO:teuthology.orchestra.run.gibba002.stdout:[mon] 2021-02-10T11:01:45.719 INFO:teuthology.orchestra.run.gibba002.stdout: debug ms = 1 2021-02-10T11:01:45.719 INFO:teuthology.orchestra.run.gibba002.stdout: debug mon = 20 2021-02-10T11:01:45.719 INFO:teuthology.orchestra.run.gibba002.stdout: debug paxos = 20 2021-02-10T11:01:45.719 INFO:teuthology.orchestra.run.gibba002.stdout: debug auth = 20 2021-02-10T11:01:45.719 INFO:teuthology.orchestra.run.gibba002.stdout: mon data avail warn = 5 2021-02-10T11:01:45.719 INFO:teuthology.orchestra.run.gibba002.stdout: mon mgr mkfs grace = 240 2021-02-10T11:01:45.719 INFO:teuthology.orchestra.run.gibba002.stdout: mon reweight min pgs per osd = 4 2021-02-10T11:01:45.720 INFO:teuthology.orchestra.run.gibba002.stdout: mon osd reporter subtree level = osd 2021-02-10T11:01:45.720 INFO:teuthology.orchestra.run.gibba002.stdout: mon osd prime pg temp = true 2021-02-10T11:01:45.720 INFO:teuthology.orchestra.run.gibba002.stdout: mon reweight min bytes per osd = 10 2021-02-10T11:01:45.720 INFO:teuthology.orchestra.run.gibba002.stdout: mon op complaint time = 120 2021-02-10T11:01:45.720 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:01:45.720 INFO:teuthology.orchestra.run.gibba002.stdout:[client] 2021-02-10T11:01:45.720 INFO:teuthology.orchestra.run.gibba002.stdout: rgw cache enabled = true 2021-02-10T11:01:45.720 INFO:teuthology.orchestra.run.gibba002.stdout: rgw enable ops log = true 2021-02-10T11:01:45.720 INFO:teuthology.orchestra.run.gibba002.stdout: rgw enable usage log = true 2021-02-10T11:01:45.720 INFO:teuthology.orchestra.run.gibba002.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2021-02-10T11:01:45.720 INFO:teuthology.orchestra.run.gibba002.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2021-02-10T11:01:45.720 INFO:teuthology.orchestra.run.gibba002.stdout: client mount timeout = 600 2021-02-10T11:01:45.720 INFO:teuthology.orchestra.run.gibba002.stdout: debug client = 20 2021-02-10T11:01:45.720 INFO:teuthology.orchestra.run.gibba002.stdout: debug ms = 1 2021-02-10T11:01:45.721 INFO:teuthology.orchestra.run.gibba002.stdout: pid file = "" 2021-02-10T11:01:45.721 INFO:teuthology.orchestra.run.gibba002.stdout:[mon.a] 2021-02-10T11:01:45.721 INFO:teuthology.orchestra.run.gibba002.stdout:[mon.b] 2021-02-10T11:01:45.721 INFO:teuthology.orchestra.run.gibba002.stdout:[mon.c] 2021-02-10T11:01:45.721 INFO:teuthology.orchestra.run.gibba002.stdout:[mds.a] 2021-02-10T11:01:45.721 INFO:teuthology.orchestra.run.gibba002.stdout:[mds.b] 2021-02-10T11:01:45.721 INFO:teuthology.orchestra.run.gibba002.stdout:[mds] 2021-02-10T11:01:45.721 INFO:teuthology.orchestra.run.gibba002.stdout: debug mds = 20 2021-02-10T11:01:45.721 INFO:teuthology.orchestra.run.gibba002.stdout: debug ms = 1 2021-02-10T11:01:45.721 INFO:teuthology.orchestra.run.gibba002.stdout: mds bal frag = True 2021-02-10T11:01:45.721 INFO:teuthology.orchestra.run.gibba002.stdout: mds bal fragment size max = 10000 2021-02-10T11:01:45.721 INFO:teuthology.orchestra.run.gibba002.stdout: mds bal merge size = 5 2021-02-10T11:01:45.721 INFO:teuthology.orchestra.run.gibba002.stdout: mds bal split bits = 3 2021-02-10T11:01:45.722 INFO:teuthology.orchestra.run.gibba002.stdout: mds bal split size = 100 2021-02-10T11:01:45.722 INFO:teuthology.orchestra.run.gibba002.stdout: mds debug frag = True 2021-02-10T11:01:45.722 INFO:teuthology.orchestra.run.gibba002.stdout: mds debug scatterstat = True 2021-02-10T11:01:45.722 INFO:teuthology.orchestra.run.gibba002.stdout: mds op complaint time = 180 2021-02-10T11:01:45.722 INFO:teuthology.orchestra.run.gibba002.stdout: mds verify scatter = True 2021-02-10T11:01:45.722 INFO:teuthology.orchestra.run.gibba002.stdout: osd op complaint time = 180 2021-02-10T11:01:45.723 INFO:teuthology.orchestra.run.gibba013.stdout:[global] 2021-02-10T11:01:45.723 INFO:teuthology.orchestra.run.gibba013.stdout: chdir = "" 2021-02-10T11:01:45.723 INFO:teuthology.orchestra.run.gibba013.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2021-02-10T11:01:45.723 INFO:teuthology.orchestra.run.gibba013.stdout: auth supported = cephx 2021-02-10T11:01:45.723 INFO:teuthology.orchestra.run.gibba013.stdout: 2021-02-10T11:01:45.723 INFO:teuthology.orchestra.run.gibba013.stdout: filestore xattr use omap = true 2021-02-10T11:01:45.723 INFO:teuthology.orchestra.run.gibba013.stdout: 2021-02-10T11:01:45.723 INFO:teuthology.orchestra.run.gibba013.stdout: mon clock drift allowed = 1.000 2021-02-10T11:01:45.723 INFO:teuthology.orchestra.run.gibba013.stdout: 2021-02-10T11:01:45.724 INFO:teuthology.orchestra.run.gibba013.stdout: osd crush chooseleaf type = 0 2021-02-10T11:01:45.724 INFO:teuthology.orchestra.run.gibba013.stdout: auth debug = true 2021-02-10T11:01:45.724 INFO:teuthology.orchestra.run.gibba013.stdout: 2021-02-10T11:01:45.724 INFO:teuthology.orchestra.run.gibba013.stdout: ms die on old message = true 2021-02-10T11:01:45.724 INFO:teuthology.orchestra.run.gibba013.stdout: ms die on bug = true 2021-02-10T11:01:45.724 INFO:teuthology.orchestra.run.gibba013.stdout: 2021-02-10T11:01:45.724 INFO:teuthology.orchestra.run.gibba013.stdout: mon max pg per osd = 10000 # >= luminous 2021-02-10T11:01:45.724 INFO:teuthology.orchestra.run.gibba013.stdout: mon pg warn max object skew = 0 2021-02-10T11:01:45.724 INFO:teuthology.orchestra.run.gibba013.stdout: 2021-02-10T11:01:45.724 INFO:teuthology.orchestra.run.gibba013.stdout: osd pool default size = 2 2021-02-10T11:01:45.724 INFO:teuthology.orchestra.run.gibba013.stdout: 2021-02-10T11:01:45.724 INFO:teuthology.orchestra.run.gibba013.stdout: mon osd allow primary affinity = true 2021-02-10T11:01:45.724 INFO:teuthology.orchestra.run.gibba013.stdout: mon osd allow pg remap = true 2021-02-10T11:01:45.724 INFO:teuthology.orchestra.run.gibba013.stdout: mon warn on legacy crush tunables = false 2021-02-10T11:01:45.725 INFO:teuthology.orchestra.run.gibba013.stdout: mon warn on crush straw calc version zero = false 2021-02-10T11:01:45.725 INFO:teuthology.orchestra.run.gibba013.stdout: mon warn on no sortbitwise = false 2021-02-10T11:01:45.725 INFO:teuthology.orchestra.run.gibba013.stdout: mon warn on osd down out interval zero = false 2021-02-10T11:01:45.725 INFO:teuthology.orchestra.run.gibba013.stdout: mon warn on too few osds = false 2021-02-10T11:01:45.725 INFO:teuthology.orchestra.run.gibba013.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2021-02-10T11:01:45.725 INFO:teuthology.orchestra.run.gibba013.stdout: mon_warn_on_pool_no_redundancy = false 2021-02-10T11:01:45.725 INFO:teuthology.orchestra.run.gibba013.stdout: 2021-02-10T11:01:45.725 INFO:teuthology.orchestra.run.gibba013.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2021-02-10T11:01:45.725 INFO:teuthology.orchestra.run.gibba013.stdout: 2021-02-10T11:01:45.725 INFO:teuthology.orchestra.run.gibba013.stdout: osd default data pool replay window = 5 2021-02-10T11:01:45.725 INFO:teuthology.orchestra.run.gibba013.stdout: 2021-02-10T11:01:45.725 INFO:teuthology.orchestra.run.gibba013.stdout: mon allow pool delete = true 2021-02-10T11:01:45.725 INFO:teuthology.orchestra.run.gibba013.stdout: 2021-02-10T11:01:45.725 INFO:teuthology.orchestra.run.gibba013.stdout: mon cluster log file level = debug 2021-02-10T11:01:45.726 INFO:teuthology.orchestra.run.gibba013.stdout: debug asserts on shutdown = true 2021-02-10T11:01:45.726 INFO:teuthology.orchestra.run.gibba013.stdout: mon health detail to clog = false 2021-02-10T11:01:45.726 INFO:teuthology.orchestra.run.gibba013.stdout: 2021-02-10T11:01:45.726 INFO:teuthology.orchestra.run.gibba013.stdout: # we see this fail in qa on *nautilus*; bump up retries 2021-02-10T11:01:45.726 INFO:teuthology.orchestra.run.gibba013.stdout: mon_client_directed_command_retry = 4 2021-02-10T11:01:45.726 INFO:teuthology.orchestra.run.gibba013.stdout: mon host = "172.21.2.102,172.21.2.113,[v2:172.21.2.113:3301,v1:172.21.2.113:6790]" 2021-02-10T11:01:45.726 INFO:teuthology.orchestra.run.gibba013.stdout: fsid = 68db30ea-dbce-4aef-a6df-f01c6fea0513 2021-02-10T11:01:45.726 INFO:teuthology.orchestra.run.gibba013.stdout: 2021-02-10T11:01:45.726 INFO:teuthology.orchestra.run.gibba013.stdout:[osd] 2021-02-10T11:01:45.726 INFO:teuthology.orchestra.run.gibba013.stdout: osd journal size = 100 2021-02-10T11:01:45.726 INFO:teuthology.orchestra.run.gibba013.stdout: 2021-02-10T11:01:45.726 INFO:teuthology.orchestra.run.gibba013.stdout: osd scrub load threshold = 5.0 2021-02-10T11:01:45.726 INFO:teuthology.orchestra.run.gibba013.stdout: osd scrub max interval = 600 2021-02-10T11:01:45.727 INFO:teuthology.orchestra.run.gibba013.stdout: 2021-02-10T11:01:45.727 INFO:teuthology.orchestra.run.gibba013.stdout: osd recover clone overlap = true 2021-02-10T11:01:45.727 INFO:teuthology.orchestra.run.gibba013.stdout: osd recovery max chunk = 1048576 2021-02-10T11:01:45.727 INFO:teuthology.orchestra.run.gibba013.stdout: 2021-02-10T11:01:45.727 INFO:teuthology.orchestra.run.gibba013.stdout: osd debug shutdown = true 2021-02-10T11:01:45.727 INFO:teuthology.orchestra.run.gibba013.stdout: osd debug op order = true 2021-02-10T11:01:45.727 INFO:teuthology.orchestra.run.gibba013.stdout: osd debug verify stray on activate = true 2021-02-10T11:01:45.727 INFO:teuthology.orchestra.run.gibba013.stdout: 2021-02-10T11:01:45.727 INFO:teuthology.orchestra.run.gibba013.stdout: osd open classes on start = true 2021-02-10T11:01:45.727 INFO:teuthology.orchestra.run.gibba013.stdout: osd debug pg log writeout = true 2021-02-10T11:01:45.727 INFO:teuthology.orchestra.run.gibba013.stdout: 2021-02-10T11:01:45.727 INFO:teuthology.orchestra.run.gibba013.stdout: osd deep scrub update digest min age = 30 2021-02-10T11:01:45.727 INFO:teuthology.orchestra.run.gibba013.stdout: 2021-02-10T11:01:45.728 INFO:teuthology.orchestra.run.gibba013.stdout: osd map max advance = 10 2021-02-10T11:01:45.728 INFO:teuthology.orchestra.run.gibba013.stdout: 2021-02-10T11:01:45.728 INFO:teuthology.orchestra.run.gibba013.stdout: journal zero on create = true 2021-02-10T11:01:45.728 INFO:teuthology.orchestra.run.gibba013.stdout: 2021-02-10T11:01:45.728 INFO:teuthology.orchestra.run.gibba013.stdout: filestore ondisk finisher threads = 3 2021-02-10T11:01:45.728 INFO:teuthology.orchestra.run.gibba013.stdout: filestore apply finisher threads = 3 2021-02-10T11:01:45.728 INFO:teuthology.orchestra.run.gibba013.stdout: 2021-02-10T11:01:45.728 INFO:teuthology.orchestra.run.gibba013.stdout: bdev debug aio = true 2021-02-10T11:01:45.728 INFO:teuthology.orchestra.run.gibba013.stdout: osd debug misdirected ops = true 2021-02-10T11:01:45.728 INFO:teuthology.orchestra.run.gibba013.stdout: bdev async discard = True 2021-02-10T11:01:45.728 INFO:teuthology.orchestra.run.gibba013.stdout: bdev enable discard = True 2021-02-10T11:01:45.728 INFO:teuthology.orchestra.run.gibba013.stdout: bluestore allocator = bitmap 2021-02-10T11:01:45.728 INFO:teuthology.orchestra.run.gibba013.stdout: bluestore block size = 96636764160 2021-02-10T11:01:45.728 INFO:teuthology.orchestra.run.gibba013.stdout: bluestore fsck on mount = True 2021-02-10T11:01:45.729 INFO:teuthology.orchestra.run.gibba013.stdout: debug bluefs = 20 2021-02-10T11:01:45.729 INFO:teuthology.orchestra.run.gibba013.stdout: debug bluestore = 20 2021-02-10T11:01:45.729 INFO:teuthology.orchestra.run.gibba013.stdout: debug filestore = 20 2021-02-10T11:01:45.729 INFO:teuthology.orchestra.run.gibba013.stdout: debug journal = 20 2021-02-10T11:01:45.729 INFO:teuthology.orchestra.run.gibba013.stdout: debug ms = 20 2021-02-10T11:01:45.729 INFO:teuthology.orchestra.run.gibba013.stdout: debug osd = 25 2021-02-10T11:01:45.729 INFO:teuthology.orchestra.run.gibba013.stdout: debug rocksdb = 10 2021-02-10T11:01:45.729 INFO:teuthology.orchestra.run.gibba013.stdout: mon osd backfillfull_ratio = 0.85 2021-02-10T11:01:45.729 INFO:teuthology.orchestra.run.gibba013.stdout: mon osd full ratio = 0.9 2021-02-10T11:01:45.729 INFO:teuthology.orchestra.run.gibba013.stdout: mon osd nearfull ratio = 0.8 2021-02-10T11:01:45.729 INFO:teuthology.orchestra.run.gibba013.stdout: osd failsafe full ratio = 0.95 2021-02-10T11:01:45.729 INFO:teuthology.orchestra.run.gibba013.stdout: osd objectstore = bluestore 2021-02-10T11:01:45.729 INFO:teuthology.orchestra.run.gibba013.stdout: osd op complaint time = 180 2021-02-10T11:01:45.729 INFO:teuthology.orchestra.run.gibba013.stdout: 2021-02-10T11:01:45.730 INFO:teuthology.orchestra.run.gibba013.stdout:[mgr] 2021-02-10T11:01:45.730 INFO:teuthology.orchestra.run.gibba013.stdout: debug ms = 1 2021-02-10T11:01:45.730 INFO:teuthology.orchestra.run.gibba013.stdout: debug mgr = 20 2021-02-10T11:01:45.730 INFO:teuthology.orchestra.run.gibba013.stdout: debug mon = 20 2021-02-10T11:01:45.730 INFO:teuthology.orchestra.run.gibba013.stdout: debug auth = 20 2021-02-10T11:01:45.730 INFO:teuthology.orchestra.run.gibba013.stdout: mon reweight min pgs per osd = 4 2021-02-10T11:01:45.730 INFO:teuthology.orchestra.run.gibba013.stdout: mon reweight min bytes per osd = 10 2021-02-10T11:01:45.730 INFO:teuthology.orchestra.run.gibba013.stdout: debug client = 10 2021-02-10T11:01:45.730 INFO:teuthology.orchestra.run.gibba013.stdout: 2021-02-10T11:01:45.730 INFO:teuthology.orchestra.run.gibba013.stdout:[mon] 2021-02-10T11:01:45.730 INFO:teuthology.orchestra.run.gibba013.stdout: debug ms = 1 2021-02-10T11:01:45.730 INFO:teuthology.orchestra.run.gibba013.stdout: debug mon = 20 2021-02-10T11:01:45.730 INFO:teuthology.orchestra.run.gibba013.stdout: debug paxos = 20 2021-02-10T11:01:45.731 INFO:teuthology.orchestra.run.gibba013.stdout: debug auth = 20 2021-02-10T11:01:45.731 INFO:teuthology.orchestra.run.gibba013.stdout: mon data avail warn = 5 2021-02-10T11:01:45.731 INFO:teuthology.orchestra.run.gibba013.stdout: mon mgr mkfs grace = 240 2021-02-10T11:01:45.731 INFO:teuthology.orchestra.run.gibba013.stdout: mon reweight min pgs per osd = 4 2021-02-10T11:01:45.731 INFO:teuthology.orchestra.run.gibba013.stdout: mon osd reporter subtree level = osd 2021-02-10T11:01:45.731 INFO:teuthology.orchestra.run.gibba013.stdout: mon osd prime pg temp = true 2021-02-10T11:01:45.731 INFO:teuthology.orchestra.run.gibba013.stdout: mon reweight min bytes per osd = 10 2021-02-10T11:01:45.731 INFO:teuthology.orchestra.run.gibba013.stdout: mon op complaint time = 120 2021-02-10T11:01:45.731 INFO:teuthology.orchestra.run.gibba013.stdout: 2021-02-10T11:01:45.731 INFO:teuthology.orchestra.run.gibba013.stdout:[client] 2021-02-10T11:01:45.731 INFO:teuthology.orchestra.run.gibba013.stdout: rgw cache enabled = true 2021-02-10T11:01:45.731 INFO:teuthology.orchestra.run.gibba013.stdout: rgw enable ops log = true 2021-02-10T11:01:45.731 INFO:teuthology.orchestra.run.gibba013.stdout: rgw enable usage log = true 2021-02-10T11:01:45.731 INFO:teuthology.orchestra.run.gibba013.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2021-02-10T11:01:45.731 INFO:teuthology.orchestra.run.gibba013.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2021-02-10T11:01:45.732 INFO:teuthology.orchestra.run.gibba013.stdout: client mount timeout = 600 2021-02-10T11:01:45.732 INFO:teuthology.orchestra.run.gibba013.stdout: debug client = 20 2021-02-10T11:01:45.732 INFO:teuthology.orchestra.run.gibba013.stdout: debug ms = 1 2021-02-10T11:01:45.732 INFO:teuthology.orchestra.run.gibba013.stdout: pid file = "" 2021-02-10T11:01:45.732 INFO:teuthology.orchestra.run.gibba013.stdout:[mon.a] 2021-02-10T11:01:45.732 INFO:teuthology.orchestra.run.gibba013.stdout:[mon.b] 2021-02-10T11:01:45.732 INFO:teuthology.orchestra.run.gibba013.stdout:[mon.c] 2021-02-10T11:01:45.732 INFO:teuthology.orchestra.run.gibba013.stdout:[mds.a] 2021-02-10T11:01:45.732 INFO:teuthology.orchestra.run.gibba013.stdout:[mds.b] 2021-02-10T11:01:45.732 INFO:teuthology.orchestra.run.gibba013.stdout:[mds] 2021-02-10T11:01:45.732 INFO:teuthology.orchestra.run.gibba013.stdout: debug mds = 20 2021-02-10T11:01:45.732 INFO:teuthology.orchestra.run.gibba013.stdout: debug ms = 1 2021-02-10T11:01:45.732 INFO:teuthology.orchestra.run.gibba013.stdout: mds bal frag = True 2021-02-10T11:01:45.732 INFO:teuthology.orchestra.run.gibba013.stdout: mds bal fragment size max = 10000 2021-02-10T11:01:45.733 INFO:teuthology.orchestra.run.gibba013.stdout: mds bal merge size = 5 2021-02-10T11:01:45.733 INFO:teuthology.orchestra.run.gibba013.stdout: mds bal split bits = 3 2021-02-10T11:01:45.733 INFO:teuthology.orchestra.run.gibba013.stdout: mds bal split size = 100 2021-02-10T11:01:45.733 INFO:teuthology.orchestra.run.gibba013.stdout: mds debug frag = True 2021-02-10T11:01:45.733 INFO:teuthology.orchestra.run.gibba013.stdout: mds debug scatterstat = True 2021-02-10T11:01:45.733 INFO:teuthology.orchestra.run.gibba013.stdout: mds op complaint time = 180 2021-02-10T11:01:45.733 INFO:teuthology.orchestra.run.gibba013.stdout: mds verify scatter = True 2021-02-10T11:01:45.733 INFO:teuthology.orchestra.run.gibba013.stdout: osd op complaint time = 180 2021-02-10T11:01:45.734 INFO:tasks.ceph:Creating admin key on mon.a... 2021-02-10T11:01:45.734 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2021-02-10T11:01:46.111 INFO:tasks.ceph:Copying monmap to all nodes... 2021-02-10T11:01:46.449 DEBUG:teuthology.orchestra.remote:gibba002:/etc/ceph/ceph.keyring is 206B 2021-02-10T11:01:47.126 DEBUG:teuthology.orchestra.remote:gibba002:/home/ubuntu/cephtest/ceph.monmap is 373B 2021-02-10T11:01:47.466 INFO:tasks.ceph:Sending monmap to node ubuntu@gibba002.front.sepia.ceph.com 2021-02-10T11:01:47.467 DEBUG:teuthology.orchestra.run.gibba002:> set -ex 2021-02-10T11:01:47.467 DEBUG:teuthology.orchestra.run.gibba002:> sudo dd of=/etc/ceph/ceph.keyring 2021-02-10T11:01:47.467 DEBUG:teuthology.orchestra.run.gibba002:> sudo chmod 0644 /etc/ceph/ceph.keyring 2021-02-10T11:01:47.819 DEBUG:teuthology.orchestra.run.gibba002:> set -ex 2021-02-10T11:01:47.819 DEBUG:teuthology.orchestra.run.gibba002:> dd of=/home/ubuntu/cephtest/ceph.monmap 2021-02-10T11:01:48.152 INFO:tasks.ceph:Sending monmap to node ubuntu@gibba013.front.sepia.ceph.com 2021-02-10T11:01:48.152 DEBUG:teuthology.orchestra.run.gibba013:> set -ex 2021-02-10T11:01:48.152 DEBUG:teuthology.orchestra.run.gibba013:> sudo dd of=/etc/ceph/ceph.keyring 2021-02-10T11:01:48.152 DEBUG:teuthology.orchestra.run.gibba013:> sudo chmod 0644 /etc/ceph/ceph.keyring 2021-02-10T11:01:48.505 DEBUG:teuthology.orchestra.run.gibba013:> set -ex 2021-02-10T11:01:48.506 DEBUG:teuthology.orchestra.run.gibba013:> dd of=/home/ubuntu/cephtest/ceph.monmap 2021-02-10T11:01:48.839 INFO:tasks.ceph:Setting up mon nodes... 2021-02-10T11:01:48.839 INFO:tasks.ceph:Setting up mgr nodes... 2021-02-10T11:01:48.839 DEBUG:teuthology.orchestra.run.gibba002:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2021-02-10T11:01:49.217 INFO:teuthology.orchestra.run.gibba002.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2021-02-10T11:01:49.219 DEBUG:teuthology.orchestra.run.gibba013:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2021-02-10T11:01:49.643 INFO:teuthology.orchestra.run.gibba013.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2021-02-10T11:01:49.646 INFO:tasks.ceph:Setting up mds nodes... 2021-02-10T11:01:49.646 DEBUG:teuthology.orchestra.run.gibba002:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2021-02-10T11:01:50.023 INFO:teuthology.orchestra.run.gibba002.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2021-02-10T11:01:50.107 DEBUG:teuthology.orchestra.run.gibba002:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2021-02-10T11:01:50.449 DEBUG:teuthology.orchestra.run.gibba013:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2021-02-10T11:01:50.826 INFO:teuthology.orchestra.run.gibba013.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2021-02-10T11:01:50.828 DEBUG:teuthology.orchestra.run.gibba013:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2021-02-10T11:01:51.171 INFO:tasks.ceph_client:Setting up client nodes... 2021-02-10T11:01:51.172 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2021-02-10T11:01:51.541 INFO:teuthology.orchestra.run.gibba002.stdout:creating /etc/ceph/ceph.client.0.keyring 2021-02-10T11:01:51.552 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2021-02-10T11:01:51.918 INFO:teuthology.orchestra.run.gibba002.stdout:creating /etc/ceph/ceph.client.1.keyring 2021-02-10T11:01:51.930 DEBUG:teuthology.orchestra.run.gibba013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.2 /etc/ceph/ceph.client.2.keyring && sudo chmod 0644 /etc/ceph/ceph.client.2.keyring 2021-02-10T11:01:52.297 INFO:teuthology.orchestra.run.gibba013.stdout:creating /etc/ceph/ceph.client.2.keyring 2021-02-10T11:01:52.309 DEBUG:teuthology.orchestra.run.gibba013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.3 /etc/ceph/ceph.client.3.keyring && sudo chmod 0644 /etc/ceph/ceph.client.3.keyring 2021-02-10T11:01:52.676 INFO:teuthology.orchestra.run.gibba013.stdout:creating /etc/ceph/ceph.client.3.keyring 2021-02-10T11:01:52.687 INFO:tasks.ceph:Running mkfs on osd nodes... 2021-02-10T11:01:52.688 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@gibba002.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@gibba013.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2021-02-10T11:01:52.688 DEBUG:teuthology.orchestra.run.gibba002:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2021-02-10T11:01:53.031 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2021-02-10T11:01:53.031 INFO:tasks.ceph:role: osd.0 2021-02-10T11:01:53.032 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@gibba002.front.sepia.ceph.com 2021-02-10T11:01:53.032 DEBUG:teuthology.orchestra.run.gibba002:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2021-02-10T11:01:53.420 INFO:teuthology.orchestra.run.gibba002.stdout:Discarding blocks...Done. 2021-02-10T11:01:53.420 INFO:teuthology.orchestra.run.gibba002.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5505024 blks 2021-02-10T11:01:53.420 INFO:teuthology.orchestra.run.gibba002.stdout: = sectsz=512 attr=2, projid32bit=1 2021-02-10T11:01:53.420 INFO:teuthology.orchestra.run.gibba002.stdout: = crc=1 finobt=0, sparse=0 2021-02-10T11:01:53.420 INFO:teuthology.orchestra.run.gibba002.stdout:data = bsize=4096 blocks=22020096, imaxpct=25 2021-02-10T11:01:53.421 INFO:teuthology.orchestra.run.gibba002.stdout: = sunit=0 swidth=0 blks 2021-02-10T11:01:53.421 INFO:teuthology.orchestra.run.gibba002.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2021-02-10T11:01:53.421 INFO:teuthology.orchestra.run.gibba002.stdout:log =internal log bsize=4096 blocks=10752, version=2 2021-02-10T11:01:53.421 INFO:teuthology.orchestra.run.gibba002.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2021-02-10T11:01:53.421 INFO:teuthology.orchestra.run.gibba002.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2021-02-10T11:01:53.422 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@gibba002.front.sepia.ceph.com -o noatime 2021-02-10T11:01:53.422 DEBUG:teuthology.orchestra.run.gibba002:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2021-02-10T11:01:53.781 DEBUG:teuthology.orchestra.run.gibba002:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2021-02-10T11:01:54.127 DEBUG:teuthology.orchestra.run.gibba002:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2021-02-10T11:01:54.470 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2021-02-10T11:01:54.470 INFO:tasks.ceph:role: osd.1 2021-02-10T11:01:54.470 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@gibba002.front.sepia.ceph.com 2021-02-10T11:01:54.471 DEBUG:teuthology.orchestra.run.gibba002:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2021-02-10T11:01:54.863 INFO:teuthology.orchestra.run.gibba002.stdout:Discarding blocks...Done. 2021-02-10T11:01:54.863 INFO:teuthology.orchestra.run.gibba002.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5505024 blks 2021-02-10T11:01:54.863 INFO:teuthology.orchestra.run.gibba002.stdout: = sectsz=512 attr=2, projid32bit=1 2021-02-10T11:01:54.863 INFO:teuthology.orchestra.run.gibba002.stdout: = crc=1 finobt=0, sparse=0 2021-02-10T11:01:54.863 INFO:teuthology.orchestra.run.gibba002.stdout:data = bsize=4096 blocks=22020096, imaxpct=25 2021-02-10T11:01:54.863 INFO:teuthology.orchestra.run.gibba002.stdout: = sunit=0 swidth=0 blks 2021-02-10T11:01:54.864 INFO:teuthology.orchestra.run.gibba002.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2021-02-10T11:01:54.864 INFO:teuthology.orchestra.run.gibba002.stdout:log =internal log bsize=4096 blocks=10752, version=2 2021-02-10T11:01:54.864 INFO:teuthology.orchestra.run.gibba002.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2021-02-10T11:01:54.864 INFO:teuthology.orchestra.run.gibba002.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2021-02-10T11:01:54.864 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@gibba002.front.sepia.ceph.com -o noatime 2021-02-10T11:01:54.864 DEBUG:teuthology.orchestra.run.gibba002:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2021-02-10T11:01:55.223 DEBUG:teuthology.orchestra.run.gibba002:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2021-02-10T11:01:55.572 DEBUG:teuthology.orchestra.run.gibba002:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2021-02-10T11:01:55.920 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2021-02-10T11:01:55.921 INFO:tasks.ceph:role: osd.2 2021-02-10T11:01:55.921 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@gibba002.front.sepia.ceph.com 2021-02-10T11:01:55.921 DEBUG:teuthology.orchestra.run.gibba002:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2021-02-10T11:01:56.305 INFO:teuthology.orchestra.run.gibba002.stdout:Discarding blocks...Done. 2021-02-10T11:01:56.306 INFO:teuthology.orchestra.run.gibba002.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5505024 blks 2021-02-10T11:01:56.306 INFO:teuthology.orchestra.run.gibba002.stdout: = sectsz=512 attr=2, projid32bit=1 2021-02-10T11:01:56.306 INFO:teuthology.orchestra.run.gibba002.stdout: = crc=1 finobt=0, sparse=0 2021-02-10T11:01:56.306 INFO:teuthology.orchestra.run.gibba002.stdout:data = bsize=4096 blocks=22020096, imaxpct=25 2021-02-10T11:01:56.306 INFO:teuthology.orchestra.run.gibba002.stdout: = sunit=0 swidth=0 blks 2021-02-10T11:01:56.306 INFO:teuthology.orchestra.run.gibba002.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2021-02-10T11:01:56.306 INFO:teuthology.orchestra.run.gibba002.stdout:log =internal log bsize=4096 blocks=10752, version=2 2021-02-10T11:01:56.306 INFO:teuthology.orchestra.run.gibba002.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2021-02-10T11:01:56.306 INFO:teuthology.orchestra.run.gibba002.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2021-02-10T11:01:56.307 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@gibba002.front.sepia.ceph.com -o noatime 2021-02-10T11:01:56.307 DEBUG:teuthology.orchestra.run.gibba002:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2021-02-10T11:01:56.670 DEBUG:teuthology.orchestra.run.gibba002:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2021-02-10T11:01:57.015 DEBUG:teuthology.orchestra.run.gibba002:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2021-02-10T11:01:57.359 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2021-02-10T11:01:57.359 INFO:tasks.ceph:role: osd.3 2021-02-10T11:01:57.359 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@gibba002.front.sepia.ceph.com 2021-02-10T11:01:57.359 DEBUG:teuthology.orchestra.run.gibba002:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2021-02-10T11:01:57.741 INFO:teuthology.orchestra.run.gibba002.stdout:Discarding blocks...Done. 2021-02-10T11:01:57.741 INFO:teuthology.orchestra.run.gibba002.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5505024 blks 2021-02-10T11:01:57.742 INFO:teuthology.orchestra.run.gibba002.stdout: = sectsz=512 attr=2, projid32bit=1 2021-02-10T11:01:57.742 INFO:teuthology.orchestra.run.gibba002.stdout: = crc=1 finobt=0, sparse=0 2021-02-10T11:01:57.742 INFO:teuthology.orchestra.run.gibba002.stdout:data = bsize=4096 blocks=22020096, imaxpct=25 2021-02-10T11:01:57.742 INFO:teuthology.orchestra.run.gibba002.stdout: = sunit=0 swidth=0 blks 2021-02-10T11:01:57.742 INFO:teuthology.orchestra.run.gibba002.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2021-02-10T11:01:57.742 INFO:teuthology.orchestra.run.gibba002.stdout:log =internal log bsize=4096 blocks=10752, version=2 2021-02-10T11:01:57.742 INFO:teuthology.orchestra.run.gibba002.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2021-02-10T11:01:57.742 INFO:teuthology.orchestra.run.gibba002.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2021-02-10T11:01:57.743 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@gibba002.front.sepia.ceph.com -o noatime 2021-02-10T11:01:57.743 DEBUG:teuthology.orchestra.run.gibba002:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2021-02-10T11:01:58.101 DEBUG:teuthology.orchestra.run.gibba002:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2021-02-10T11:01:58.446 DEBUG:teuthology.orchestra.run.gibba002:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2021-02-10T11:01:58.817 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:01:58.818 7f57979a9a80 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2021-02-10T11:01:58.817 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:01:58.818 7f57979a9a80 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2021-02-10T11:01:58.817 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:01:58.818 7f57979a9a80 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2021-02-10T11:01:58.817 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:01:58.818 7f57979a9a80 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2021-02-10T11:01:58.817 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:01:58.818 7f57979a9a80 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2021-02-10T11:01:58.818 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:01:58.819 7f57979a9a80 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2021-02-10T11:02:01.109 DEBUG:teuthology.orchestra.run.gibba002:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2021-02-10T11:02:01.454 DEBUG:teuthology.orchestra.run.gibba002:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2021-02-10T11:02:01.825 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:02:01.825 7fdc64378a80 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2021-02-10T11:02:01.825 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:02:01.826 7fdc64378a80 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2021-02-10T11:02:01.825 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:02:01.826 7fdc64378a80 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2021-02-10T11:02:01.825 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:02:01.826 7fdc64378a80 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2021-02-10T11:02:01.826 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:02:01.826 7fdc64378a80 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2021-02-10T11:02:01.826 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:02:01.827 7fdc64378a80 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2021-02-10T11:02:04.116 DEBUG:teuthology.orchestra.run.gibba002:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2021-02-10T11:02:04.461 DEBUG:teuthology.orchestra.run.gibba002:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2021-02-10T11:02:04.830 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:02:04.831 7fe2393f6a80 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2021-02-10T11:02:04.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:02:04.832 7fe2393f6a80 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2021-02-10T11:02:04.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:02:04.832 7fe2393f6a80 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2021-02-10T11:02:04.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:02:04.832 7fe2393f6a80 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2021-02-10T11:02:04.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:02:04.832 7fe2393f6a80 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2021-02-10T11:02:04.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:02:04.833 7fe2393f6a80 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2021-02-10T11:02:07.121 DEBUG:teuthology.orchestra.run.gibba002:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2021-02-10T11:02:07.465 DEBUG:teuthology.orchestra.run.gibba002:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2021-02-10T11:02:07.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:02:07.837 7ff332969a80 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2021-02-10T11:02:07.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:02:07.837 7ff332969a80 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2021-02-10T11:02:07.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:02:07.837 7ff332969a80 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2021-02-10T11:02:07.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:02:07.837 7ff332969a80 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2021-02-10T11:02:07.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:02:07.837 7ff332969a80 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2021-02-10T11:02:07.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:02:07.838 7ff332969a80 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2021-02-10T11:02:09.605 DEBUG:teuthology.orchestra.run.gibba002:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:02:09.608 DEBUG:teuthology.orchestra.run.gibba013:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:02:10.127 DEBUG:teuthology.orchestra.run.gibba002:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2021-02-10T11:02:10.471 DEBUG:teuthology.orchestra.run.gibba013:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2021-02-10T11:02:10.816 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2021-02-10T11:02:10.816 INFO:tasks.ceph:role: osd.4 2021-02-10T11:02:10.816 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@gibba013.front.sepia.ceph.com 2021-02-10T11:02:10.816 DEBUG:teuthology.orchestra.run.gibba013:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2021-02-10T11:02:11.207 INFO:teuthology.orchestra.run.gibba013.stdout:Discarding blocks...Done. 2021-02-10T11:02:11.207 INFO:teuthology.orchestra.run.gibba013.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5505024 blks 2021-02-10T11:02:11.207 INFO:teuthology.orchestra.run.gibba013.stdout: = sectsz=512 attr=2, projid32bit=1 2021-02-10T11:02:11.207 INFO:teuthology.orchestra.run.gibba013.stdout: = crc=1 finobt=0, sparse=0 2021-02-10T11:02:11.207 INFO:teuthology.orchestra.run.gibba013.stdout:data = bsize=4096 blocks=22020096, imaxpct=25 2021-02-10T11:02:11.208 INFO:teuthology.orchestra.run.gibba013.stdout: = sunit=0 swidth=0 blks 2021-02-10T11:02:11.208 INFO:teuthology.orchestra.run.gibba013.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2021-02-10T11:02:11.208 INFO:teuthology.orchestra.run.gibba013.stdout:log =internal log bsize=4096 blocks=10752, version=2 2021-02-10T11:02:11.208 INFO:teuthology.orchestra.run.gibba013.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2021-02-10T11:02:11.208 INFO:teuthology.orchestra.run.gibba013.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2021-02-10T11:02:11.209 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@gibba013.front.sepia.ceph.com -o noatime 2021-02-10T11:02:11.209 DEBUG:teuthology.orchestra.run.gibba013:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2021-02-10T11:02:11.570 DEBUG:teuthology.orchestra.run.gibba013:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2021-02-10T11:02:11.918 DEBUG:teuthology.orchestra.run.gibba013:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2021-02-10T11:02:12.261 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2021-02-10T11:02:12.261 INFO:tasks.ceph:role: osd.5 2021-02-10T11:02:12.262 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@gibba013.front.sepia.ceph.com 2021-02-10T11:02:12.262 DEBUG:teuthology.orchestra.run.gibba013:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2021-02-10T11:02:12.653 INFO:teuthology.orchestra.run.gibba013.stdout:Discarding blocks...Done. 2021-02-10T11:02:12.653 INFO:teuthology.orchestra.run.gibba013.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5505024 blks 2021-02-10T11:02:12.653 INFO:teuthology.orchestra.run.gibba013.stdout: = sectsz=512 attr=2, projid32bit=1 2021-02-10T11:02:12.653 INFO:teuthology.orchestra.run.gibba013.stdout: = crc=1 finobt=0, sparse=0 2021-02-10T11:02:12.653 INFO:teuthology.orchestra.run.gibba013.stdout:data = bsize=4096 blocks=22020096, imaxpct=25 2021-02-10T11:02:12.653 INFO:teuthology.orchestra.run.gibba013.stdout: = sunit=0 swidth=0 blks 2021-02-10T11:02:12.654 INFO:teuthology.orchestra.run.gibba013.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2021-02-10T11:02:12.654 INFO:teuthology.orchestra.run.gibba013.stdout:log =internal log bsize=4096 blocks=10752, version=2 2021-02-10T11:02:12.654 INFO:teuthology.orchestra.run.gibba013.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2021-02-10T11:02:12.654 INFO:teuthology.orchestra.run.gibba013.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2021-02-10T11:02:12.655 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@gibba013.front.sepia.ceph.com -o noatime 2021-02-10T11:02:12.655 DEBUG:teuthology.orchestra.run.gibba013:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2021-02-10T11:02:13.012 DEBUG:teuthology.orchestra.run.gibba013:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2021-02-10T11:02:13.358 DEBUG:teuthology.orchestra.run.gibba013:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2021-02-10T11:02:13.700 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2021-02-10T11:02:13.701 INFO:tasks.ceph:role: osd.6 2021-02-10T11:02:13.701 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@gibba013.front.sepia.ceph.com 2021-02-10T11:02:13.701 DEBUG:teuthology.orchestra.run.gibba013:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2021-02-10T11:02:14.092 INFO:teuthology.orchestra.run.gibba013.stdout:Discarding blocks...Done. 2021-02-10T11:02:14.092 INFO:teuthology.orchestra.run.gibba013.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5505024 blks 2021-02-10T11:02:14.092 INFO:teuthology.orchestra.run.gibba013.stdout: = sectsz=512 attr=2, projid32bit=1 2021-02-10T11:02:14.092 INFO:teuthology.orchestra.run.gibba013.stdout: = crc=1 finobt=0, sparse=0 2021-02-10T11:02:14.092 INFO:teuthology.orchestra.run.gibba013.stdout:data = bsize=4096 blocks=22020096, imaxpct=25 2021-02-10T11:02:14.092 INFO:teuthology.orchestra.run.gibba013.stdout: = sunit=0 swidth=0 blks 2021-02-10T11:02:14.092 INFO:teuthology.orchestra.run.gibba013.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2021-02-10T11:02:14.092 INFO:teuthology.orchestra.run.gibba013.stdout:log =internal log bsize=4096 blocks=10752, version=2 2021-02-10T11:02:14.092 INFO:teuthology.orchestra.run.gibba013.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2021-02-10T11:02:14.092 INFO:teuthology.orchestra.run.gibba013.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2021-02-10T11:02:14.093 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@gibba013.front.sepia.ceph.com -o noatime 2021-02-10T11:02:14.093 DEBUG:teuthology.orchestra.run.gibba013:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2021-02-10T11:02:14.451 DEBUG:teuthology.orchestra.run.gibba013:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2021-02-10T11:02:14.796 DEBUG:teuthology.orchestra.run.gibba013:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2021-02-10T11:02:15.139 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2021-02-10T11:02:15.139 INFO:tasks.ceph:role: osd.7 2021-02-10T11:02:15.140 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@gibba013.front.sepia.ceph.com 2021-02-10T11:02:15.140 DEBUG:teuthology.orchestra.run.gibba013:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2021-02-10T11:02:15.530 INFO:teuthology.orchestra.run.gibba013.stdout:Discarding blocks...Done. 2021-02-10T11:02:15.530 INFO:teuthology.orchestra.run.gibba013.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5505024 blks 2021-02-10T11:02:15.530 INFO:teuthology.orchestra.run.gibba013.stdout: = sectsz=512 attr=2, projid32bit=1 2021-02-10T11:02:15.530 INFO:teuthology.orchestra.run.gibba013.stdout: = crc=1 finobt=0, sparse=0 2021-02-10T11:02:15.531 INFO:teuthology.orchestra.run.gibba013.stdout:data = bsize=4096 blocks=22020096, imaxpct=25 2021-02-10T11:02:15.531 INFO:teuthology.orchestra.run.gibba013.stdout: = sunit=0 swidth=0 blks 2021-02-10T11:02:15.531 INFO:teuthology.orchestra.run.gibba013.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2021-02-10T11:02:15.531 INFO:teuthology.orchestra.run.gibba013.stdout:log =internal log bsize=4096 blocks=10752, version=2 2021-02-10T11:02:15.531 INFO:teuthology.orchestra.run.gibba013.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2021-02-10T11:02:15.531 INFO:teuthology.orchestra.run.gibba013.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2021-02-10T11:02:15.532 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@gibba013.front.sepia.ceph.com -o noatime 2021-02-10T11:02:15.532 DEBUG:teuthology.orchestra.run.gibba013:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2021-02-10T11:02:15.889 DEBUG:teuthology.orchestra.run.gibba013:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2021-02-10T11:02:16.234 DEBUG:teuthology.orchestra.run.gibba013:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2021-02-10T11:02:16.605 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:02:16.604 7fcd43b74a80 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2021-02-10T11:02:16.605 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:02:16.604 7fcd43b74a80 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2021-02-10T11:02:16.605 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:02:16.604 7fcd43b74a80 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2021-02-10T11:02:16.605 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:02:16.604 7fcd43b74a80 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2021-02-10T11:02:16.605 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:02:16.604 7fcd43b74a80 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2021-02-10T11:02:16.606 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:02:16.605 7fcd43b74a80 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2021-02-10T11:02:18.901 DEBUG:teuthology.orchestra.run.gibba013:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2021-02-10T11:02:19.244 DEBUG:teuthology.orchestra.run.gibba013:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2021-02-10T11:02:19.614 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:02:19.613 7fecfc597a80 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2021-02-10T11:02:19.614 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:02:19.614 7fecfc597a80 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2021-02-10T11:02:19.615 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:02:19.614 7fecfc597a80 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2021-02-10T11:02:19.615 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:02:19.614 7fecfc597a80 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2021-02-10T11:02:19.615 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:02:19.614 7fecfc597a80 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2021-02-10T11:02:19.615 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:02:19.615 7fecfc597a80 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2021-02-10T11:02:21.905 DEBUG:teuthology.orchestra.run.gibba013:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2021-02-10T11:02:22.250 DEBUG:teuthology.orchestra.run.gibba013:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2021-02-10T11:02:22.628 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:02:22.627 7ff37c652a80 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2021-02-10T11:02:22.628 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:02:22.627 7ff37c652a80 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2021-02-10T11:02:22.628 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:02:22.627 7ff37c652a80 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2021-02-10T11:02:22.628 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:02:22.627 7ff37c652a80 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2021-02-10T11:02:22.628 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:02:22.627 7ff37c652a80 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2021-02-10T11:02:22.629 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:02:22.628 7ff37c652a80 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2021-02-10T11:02:24.918 DEBUG:teuthology.orchestra.run.gibba013:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2021-02-10T11:02:25.262 DEBUG:teuthology.orchestra.run.gibba013:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2021-02-10T11:02:25.632 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:02:25.631 7fdce99a7a80 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2021-02-10T11:02:25.632 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:02:25.631 7fdce99a7a80 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2021-02-10T11:02:25.632 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:02:25.631 7fdce99a7a80 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2021-02-10T11:02:25.632 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:02:25.631 7fdce99a7a80 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2021-02-10T11:02:25.632 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:02:25.631 7fdce99a7a80 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2021-02-10T11:02:25.633 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:02:25.632 7fdce99a7a80 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2021-02-10T11:02:27.929 DEBUG:teuthology.orchestra.run.gibba013:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2021-02-10T11:02:28.272 INFO:tasks.ceph:Reading keys from all nodes... 2021-02-10T11:02:28.273 DEBUG:teuthology.orchestra.run.gibba002:> mktemp 2021-02-10T11:02:28.607 INFO:teuthology.orchestra.run.gibba002.stdout:/tmp/tmp.PF4ggLi3Uz 2021-02-10T11:02:28.608 DEBUG:teuthology.orchestra.run.gibba002:> sudo cp /var/lib/ceph/mgr/ceph-y/keyring /tmp/tmp.PF4ggLi3Uz 2021-02-10T11:02:28.952 DEBUG:teuthology.orchestra.run.gibba002:> sudo chmod 0666 /tmp/tmp.PF4ggLi3Uz 2021-02-10T11:02:29.640 DEBUG:teuthology.orchestra.remote:gibba002:/tmp/tmp.PF4ggLi3Uz is 56B 2021-02-10T11:02:29.982 DEBUG:teuthology.orchestra.run.gibba002:> rm -fr /tmp/tmp.PF4ggLi3Uz 2021-02-10T11:02:30.316 DEBUG:teuthology.orchestra.run.gibba002:> mktemp 2021-02-10T11:02:30.649 INFO:teuthology.orchestra.run.gibba002.stdout:/tmp/tmp.fx7VrZb0W2 2021-02-10T11:02:30.650 DEBUG:teuthology.orchestra.run.gibba002:> sudo cp /var/lib/ceph/mds/ceph-a/keyring /tmp/tmp.fx7VrZb0W2 2021-02-10T11:02:30.993 DEBUG:teuthology.orchestra.run.gibba002:> sudo chmod 0666 /tmp/tmp.fx7VrZb0W2 2021-02-10T11:02:31.674 DEBUG:teuthology.orchestra.remote:gibba002:/tmp/tmp.fx7VrZb0W2 is 56B 2021-02-10T11:02:32.014 DEBUG:teuthology.orchestra.run.gibba002:> rm -fr /tmp/tmp.fx7VrZb0W2 2021-02-10T11:02:32.350 DEBUG:teuthology.orchestra.run.gibba002:> mktemp 2021-02-10T11:02:32.685 INFO:teuthology.orchestra.run.gibba002.stdout:/tmp/tmp.bEwbbIJJGZ 2021-02-10T11:02:32.686 DEBUG:teuthology.orchestra.run.gibba002:> sudo cp /var/lib/ceph/osd/ceph-0/keyring /tmp/tmp.bEwbbIJJGZ 2021-02-10T11:02:33.030 DEBUG:teuthology.orchestra.run.gibba002:> sudo chmod 0666 /tmp/tmp.bEwbbIJJGZ 2021-02-10T11:02:33.711 DEBUG:teuthology.orchestra.remote:gibba002:/tmp/tmp.bEwbbIJJGZ is 56B 2021-02-10T11:02:34.051 DEBUG:teuthology.orchestra.run.gibba002:> rm -fr /tmp/tmp.bEwbbIJJGZ 2021-02-10T11:02:34.384 DEBUG:teuthology.orchestra.run.gibba002:> mktemp 2021-02-10T11:02:34.718 INFO:teuthology.orchestra.run.gibba002.stdout:/tmp/tmp.pA2vfRioeb 2021-02-10T11:02:34.719 DEBUG:teuthology.orchestra.run.gibba002:> sudo cp /var/lib/ceph/osd/ceph-1/keyring /tmp/tmp.pA2vfRioeb 2021-02-10T11:02:35.062 DEBUG:teuthology.orchestra.run.gibba002:> sudo chmod 0666 /tmp/tmp.pA2vfRioeb 2021-02-10T11:02:35.744 DEBUG:teuthology.orchestra.remote:gibba002:/tmp/tmp.pA2vfRioeb is 56B 2021-02-10T11:02:36.084 DEBUG:teuthology.orchestra.run.gibba002:> rm -fr /tmp/tmp.pA2vfRioeb 2021-02-10T11:02:36.418 DEBUG:teuthology.orchestra.run.gibba002:> mktemp 2021-02-10T11:02:36.752 INFO:teuthology.orchestra.run.gibba002.stdout:/tmp/tmp.CDF7cBrIis 2021-02-10T11:02:36.753 DEBUG:teuthology.orchestra.run.gibba002:> sudo cp /var/lib/ceph/osd/ceph-2/keyring /tmp/tmp.CDF7cBrIis 2021-02-10T11:02:37.096 DEBUG:teuthology.orchestra.run.gibba002:> sudo chmod 0666 /tmp/tmp.CDF7cBrIis 2021-02-10T11:02:37.776 DEBUG:teuthology.orchestra.remote:gibba002:/tmp/tmp.CDF7cBrIis is 56B 2021-02-10T11:02:38.117 DEBUG:teuthology.orchestra.run.gibba002:> rm -fr /tmp/tmp.CDF7cBrIis 2021-02-10T11:02:38.452 DEBUG:teuthology.orchestra.run.gibba002:> mktemp 2021-02-10T11:02:38.786 INFO:teuthology.orchestra.run.gibba002.stdout:/tmp/tmp.CA8ptkTIoP 2021-02-10T11:02:38.787 DEBUG:teuthology.orchestra.run.gibba002:> sudo cp /var/lib/ceph/osd/ceph-3/keyring /tmp/tmp.CA8ptkTIoP 2021-02-10T11:02:39.132 DEBUG:teuthology.orchestra.run.gibba002:> sudo chmod 0666 /tmp/tmp.CA8ptkTIoP 2021-02-10T11:02:39.814 DEBUG:teuthology.orchestra.remote:gibba002:/tmp/tmp.CA8ptkTIoP is 56B 2021-02-10T11:02:40.051 DEBUG:teuthology.orchestra.run.gibba002:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:02:40.056 DEBUG:teuthology.orchestra.run.gibba013:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:02:40.154 DEBUG:teuthology.orchestra.run.gibba002:> rm -fr /tmp/tmp.CA8ptkTIoP 2021-02-10T11:02:40.388 DEBUG:teuthology.orchestra.run.gibba013:> mktemp 2021-02-10T11:02:40.723 INFO:teuthology.orchestra.run.gibba013.stdout:/tmp/tmp.TPkjmLVUlx 2021-02-10T11:02:40.724 DEBUG:teuthology.orchestra.run.gibba013:> sudo cp /var/lib/ceph/mgr/ceph-x/keyring /tmp/tmp.TPkjmLVUlx 2021-02-10T11:02:41.067 DEBUG:teuthology.orchestra.run.gibba013:> sudo chmod 0666 /tmp/tmp.TPkjmLVUlx 2021-02-10T11:02:41.749 DEBUG:teuthology.orchestra.remote:gibba013:/tmp/tmp.TPkjmLVUlx is 56B 2021-02-10T11:02:42.092 DEBUG:teuthology.orchestra.run.gibba013:> rm -fr /tmp/tmp.TPkjmLVUlx 2021-02-10T11:02:42.427 DEBUG:teuthology.orchestra.run.gibba013:> mktemp 2021-02-10T11:02:42.762 INFO:teuthology.orchestra.run.gibba013.stdout:/tmp/tmp.vTU7ngzQPl 2021-02-10T11:02:42.763 DEBUG:teuthology.orchestra.run.gibba013:> sudo cp /var/lib/ceph/mds/ceph-b/keyring /tmp/tmp.vTU7ngzQPl 2021-02-10T11:02:43.108 DEBUG:teuthology.orchestra.run.gibba013:> sudo chmod 0666 /tmp/tmp.vTU7ngzQPl 2021-02-10T11:02:43.790 DEBUG:teuthology.orchestra.remote:gibba013:/tmp/tmp.vTU7ngzQPl is 56B 2021-02-10T11:02:44.131 DEBUG:teuthology.orchestra.run.gibba013:> rm -fr /tmp/tmp.vTU7ngzQPl 2021-02-10T11:02:44.465 DEBUG:teuthology.orchestra.run.gibba013:> mktemp 2021-02-10T11:02:44.806 INFO:teuthology.orchestra.run.gibba013.stdout:/tmp/tmp.KHSC30PFwx 2021-02-10T11:02:44.806 DEBUG:teuthology.orchestra.run.gibba013:> sudo cp /var/lib/ceph/osd/ceph-4/keyring /tmp/tmp.KHSC30PFwx 2021-02-10T11:02:45.151 DEBUG:teuthology.orchestra.run.gibba013:> sudo chmod 0666 /tmp/tmp.KHSC30PFwx 2021-02-10T11:02:45.832 DEBUG:teuthology.orchestra.remote:gibba013:/tmp/tmp.KHSC30PFwx is 56B 2021-02-10T11:02:46.178 DEBUG:teuthology.orchestra.run.gibba013:> rm -fr /tmp/tmp.KHSC30PFwx 2021-02-10T11:02:46.657 DEBUG:teuthology.orchestra.run.gibba013:> mktemp 2021-02-10T11:02:46.991 INFO:teuthology.orchestra.run.gibba013.stdout:/tmp/tmp.2pB8AU8f5o 2021-02-10T11:02:46.991 DEBUG:teuthology.orchestra.run.gibba013:> sudo cp /var/lib/ceph/osd/ceph-5/keyring /tmp/tmp.2pB8AU8f5o 2021-02-10T11:02:47.336 DEBUG:teuthology.orchestra.run.gibba013:> sudo chmod 0666 /tmp/tmp.2pB8AU8f5o 2021-02-10T11:02:48.027 DEBUG:teuthology.orchestra.remote:gibba013:/tmp/tmp.2pB8AU8f5o is 56B 2021-02-10T11:02:48.380 DEBUG:teuthology.orchestra.run.gibba013:> rm -fr /tmp/tmp.2pB8AU8f5o 2021-02-10T11:02:48.718 DEBUG:teuthology.orchestra.run.gibba013:> mktemp 2021-02-10T11:02:49.055 INFO:teuthology.orchestra.run.gibba013.stdout:/tmp/tmp.xwsj7BZxZg 2021-02-10T11:02:49.056 DEBUG:teuthology.orchestra.run.gibba013:> sudo cp /var/lib/ceph/osd/ceph-6/keyring /tmp/tmp.xwsj7BZxZg 2021-02-10T11:02:49.400 DEBUG:teuthology.orchestra.run.gibba013:> sudo chmod 0666 /tmp/tmp.xwsj7BZxZg 2021-02-10T11:02:50.084 DEBUG:teuthology.orchestra.remote:gibba013:/tmp/tmp.xwsj7BZxZg is 56B 2021-02-10T11:02:50.423 DEBUG:teuthology.orchestra.run.gibba013:> rm -fr /tmp/tmp.xwsj7BZxZg 2021-02-10T11:02:50.758 DEBUG:teuthology.orchestra.run.gibba013:> mktemp 2021-02-10T11:02:51.094 INFO:teuthology.orchestra.run.gibba013.stdout:/tmp/tmp.qRqBBM5zBK 2021-02-10T11:02:51.094 DEBUG:teuthology.orchestra.run.gibba013:> sudo cp /var/lib/ceph/osd/ceph-7/keyring /tmp/tmp.qRqBBM5zBK 2021-02-10T11:02:51.439 DEBUG:teuthology.orchestra.run.gibba013:> sudo chmod 0666 /tmp/tmp.qRqBBM5zBK 2021-02-10T11:02:52.119 DEBUG:teuthology.orchestra.remote:gibba013:/tmp/tmp.qRqBBM5zBK is 56B 2021-02-10T11:02:52.460 DEBUG:teuthology.orchestra.run.gibba013:> rm -fr /tmp/tmp.qRqBBM5zBK 2021-02-10T11:02:53.135 DEBUG:teuthology.orchestra.remote:gibba002:/etc/ceph/ceph.client.0.keyring is 59B 2021-02-10T11:02:53.816 DEBUG:teuthology.orchestra.remote:gibba002:/etc/ceph/ceph.client.1.keyring is 59B 2021-02-10T11:02:54.493 DEBUG:teuthology.orchestra.remote:gibba013:/etc/ceph/ceph.client.2.keyring is 59B 2021-02-10T11:02:55.173 DEBUG:teuthology.orchestra.remote:gibba013:/etc/ceph/ceph.client.3.keyring is 59B 2021-02-10T11:02:55.515 INFO:tasks.ceph:Adding keys to all mons... 2021-02-10T11:02:55.515 DEBUG:teuthology.orchestra.run.gibba002:> sudo tee -a /etc/ceph/ceph.keyring 2021-02-10T11:02:55.518 DEBUG:teuthology.orchestra.run.gibba013:> sudo tee -a /etc/ceph/ceph.keyring 2021-02-10T11:02:55.857 INFO:teuthology.orchestra.run.gibba002.stdout:[mgr.y] 2021-02-10T11:02:55.858 INFO:teuthology.orchestra.run.gibba002.stdout: key = AQAdvSNgVqQFDRAAwo6YEdWZCAr5GdOWVL/Qvg== 2021-02-10T11:02:55.858 INFO:teuthology.orchestra.run.gibba002.stdout:[mds.a] 2021-02-10T11:02:55.858 INFO:teuthology.orchestra.run.gibba002.stdout: key = AQAevSNgsgyEARAAvr0cUjjCoEHLxaZSZ2HtVA== 2021-02-10T11:02:55.858 INFO:teuthology.orchestra.run.gibba002.stdout:[osd.0] 2021-02-10T11:02:55.858 INFO:teuthology.orchestra.run.gibba002.stdout: key = AQAmvSNgj+bMMBAAD3HUD6fLiMAg+++iNzLJKQ== 2021-02-10T11:02:55.858 INFO:teuthology.orchestra.run.gibba002.stdout:[osd.1] 2021-02-10T11:02:55.858 INFO:teuthology.orchestra.run.gibba002.stdout: key = AQApvSNgnvc+MRAAAOxqlNwiyLHqNITOrYeFKQ== 2021-02-10T11:02:55.858 INFO:teuthology.orchestra.run.gibba002.stdout:[osd.2] 2021-02-10T11:02:55.858 INFO:teuthology.orchestra.run.gibba002.stdout: key = AQAsvSNgZ4SbMRAA76IOYWfbiAuBoM/4k81GxA== 2021-02-10T11:02:55.858 INFO:teuthology.orchestra.run.gibba002.stdout:[osd.3] 2021-02-10T11:02:55.858 INFO:teuthology.orchestra.run.gibba002.stdout: key = AQAvvSNgFk3rMRAAWCmKKyw0LDQ+UlAHZAITOw== 2021-02-10T11:02:55.859 INFO:teuthology.orchestra.run.gibba002.stdout:[mgr.x] 2021-02-10T11:02:55.859 INFO:teuthology.orchestra.run.gibba002.stdout: key = AQAdvSNgXZZgJhAA7dtBSGzH+DsBwnddZpOBJA== 2021-02-10T11:02:55.859 INFO:teuthology.orchestra.run.gibba002.stdout:[mds.b] 2021-02-10T11:02:55.859 INFO:teuthology.orchestra.run.gibba002.stdout: key = AQAevSNgwOFMMRAALfb1fbX56ao1TU8ClnTW1A== 2021-02-10T11:02:55.859 INFO:teuthology.orchestra.run.gibba002.stdout:[osd.4] 2021-02-10T11:02:55.859 INFO:teuthology.orchestra.run.gibba002.stdout: key = AQA4vSNgpKcNJBAApJMmvj5YxVHPDOmvmQONOg== 2021-02-10T11:02:55.859 INFO:teuthology.orchestra.run.gibba002.stdout:[osd.5] 2021-02-10T11:02:55.859 INFO:teuthology.orchestra.run.gibba002.stdout: key = AQA7vSNgf2ujJBAAfkuYpyb3WTzc8V5kw+Wktg== 2021-02-10T11:02:55.859 INFO:teuthology.orchestra.run.gibba002.stdout:[osd.6] 2021-02-10T11:02:55.859 INFO:teuthology.orchestra.run.gibba002.stdout: key = AQA+vSNgqw9uJRAAFMU0R+UxU9C3o/9byb3e0g== 2021-02-10T11:02:55.860 INFO:teuthology.orchestra.run.gibba002.stdout:[osd.7] 2021-02-10T11:02:55.860 INFO:teuthology.orchestra.run.gibba002.stdout: key = AQBBvSNgvWSrJRAA2KaCHmLpzovRgYzLvyofTQ== 2021-02-10T11:02:55.860 INFO:teuthology.orchestra.run.gibba002.stdout:[client.0] 2021-02-10T11:02:55.860 INFO:teuthology.orchestra.run.gibba002.stdout: key = AQAfvSNgZd9XIBAAvyrvl8fdKuHnJZfczfRoFw== 2021-02-10T11:02:55.860 INFO:teuthology.orchestra.run.gibba002.stdout:[client.1] 2021-02-10T11:02:55.860 INFO:teuthology.orchestra.run.gibba002.stdout: key = AQAfvSNgSyfXNhAAIduPnF7Vd/6HSkJ0MlcMSA== 2021-02-10T11:02:55.860 INFO:teuthology.orchestra.run.gibba002.stdout:[client.2] 2021-02-10T11:02:55.860 INFO:teuthology.orchestra.run.gibba002.stdout: key = AQAgvSNgiIG/ERAA9ZTamaTI7ZsPSnxBqjplCg== 2021-02-10T11:02:55.860 INFO:teuthology.orchestra.run.gibba002.stdout:[client.3] 2021-02-10T11:02:55.860 INFO:teuthology.orchestra.run.gibba002.stdout: key = AQAgvSNgOWpOKBAAlXsEB04Q/GvtL06PqRMLeA== 2021-02-10T11:02:55.861 INFO:teuthology.orchestra.run.gibba013.stdout:[mgr.y] 2021-02-10T11:02:55.862 INFO:teuthology.orchestra.run.gibba013.stdout: key = AQAdvSNgVqQFDRAAwo6YEdWZCAr5GdOWVL/Qvg== 2021-02-10T11:02:55.862 INFO:teuthology.orchestra.run.gibba013.stdout:[mds.a] 2021-02-10T11:02:55.862 INFO:teuthology.orchestra.run.gibba013.stdout: key = AQAevSNgsgyEARAAvr0cUjjCoEHLxaZSZ2HtVA== 2021-02-10T11:02:55.862 INFO:teuthology.orchestra.run.gibba013.stdout:[osd.0] 2021-02-10T11:02:55.862 INFO:teuthology.orchestra.run.gibba013.stdout: key = AQAmvSNgj+bMMBAAD3HUD6fLiMAg+++iNzLJKQ== 2021-02-10T11:02:55.862 INFO:teuthology.orchestra.run.gibba013.stdout:[osd.1] 2021-02-10T11:02:55.862 INFO:teuthology.orchestra.run.gibba013.stdout: key = AQApvSNgnvc+MRAAAOxqlNwiyLHqNITOrYeFKQ== 2021-02-10T11:02:55.862 INFO:teuthology.orchestra.run.gibba013.stdout:[osd.2] 2021-02-10T11:02:55.862 INFO:teuthology.orchestra.run.gibba013.stdout: key = AQAsvSNgZ4SbMRAA76IOYWfbiAuBoM/4k81GxA== 2021-02-10T11:02:55.862 INFO:teuthology.orchestra.run.gibba013.stdout:[osd.3] 2021-02-10T11:02:55.862 INFO:teuthology.orchestra.run.gibba013.stdout: key = AQAvvSNgFk3rMRAAWCmKKyw0LDQ+UlAHZAITOw== 2021-02-10T11:02:55.863 INFO:teuthology.orchestra.run.gibba013.stdout:[mgr.x] 2021-02-10T11:02:55.863 INFO:teuthology.orchestra.run.gibba013.stdout: key = AQAdvSNgXZZgJhAA7dtBSGzH+DsBwnddZpOBJA== 2021-02-10T11:02:55.863 INFO:teuthology.orchestra.run.gibba013.stdout:[mds.b] 2021-02-10T11:02:55.863 INFO:teuthology.orchestra.run.gibba013.stdout: key = AQAevSNgwOFMMRAALfb1fbX56ao1TU8ClnTW1A== 2021-02-10T11:02:55.863 INFO:teuthology.orchestra.run.gibba013.stdout:[osd.4] 2021-02-10T11:02:55.863 INFO:teuthology.orchestra.run.gibba013.stdout: key = AQA4vSNgpKcNJBAApJMmvj5YxVHPDOmvmQONOg== 2021-02-10T11:02:55.863 INFO:teuthology.orchestra.run.gibba013.stdout:[osd.5] 2021-02-10T11:02:55.863 INFO:teuthology.orchestra.run.gibba013.stdout: key = AQA7vSNgf2ujJBAAfkuYpyb3WTzc8V5kw+Wktg== 2021-02-10T11:02:55.863 INFO:teuthology.orchestra.run.gibba013.stdout:[osd.6] 2021-02-10T11:02:55.863 INFO:teuthology.orchestra.run.gibba013.stdout: key = AQA+vSNgqw9uJRAAFMU0R+UxU9C3o/9byb3e0g== 2021-02-10T11:02:55.863 INFO:teuthology.orchestra.run.gibba013.stdout:[osd.7] 2021-02-10T11:02:55.863 INFO:teuthology.orchestra.run.gibba013.stdout: key = AQBBvSNgvWSrJRAA2KaCHmLpzovRgYzLvyofTQ== 2021-02-10T11:02:55.863 INFO:teuthology.orchestra.run.gibba013.stdout:[client.0] 2021-02-10T11:02:55.864 INFO:teuthology.orchestra.run.gibba013.stdout: key = AQAfvSNgZd9XIBAAvyrvl8fdKuHnJZfczfRoFw== 2021-02-10T11:02:55.864 INFO:teuthology.orchestra.run.gibba013.stdout:[client.1] 2021-02-10T11:02:55.864 INFO:teuthology.orchestra.run.gibba013.stdout: key = AQAfvSNgSyfXNhAAIduPnF7Vd/6HSkJ0MlcMSA== 2021-02-10T11:02:55.864 INFO:teuthology.orchestra.run.gibba013.stdout:[client.2] 2021-02-10T11:02:55.864 INFO:teuthology.orchestra.run.gibba013.stdout: key = AQAgvSNgiIG/ERAA9ZTamaTI7ZsPSnxBqjplCg== 2021-02-10T11:02:55.864 INFO:teuthology.orchestra.run.gibba013.stdout:[client.3] 2021-02-10T11:02:55.864 INFO:teuthology.orchestra.run.gibba013.stdout: key = AQAgvSNgOWpOKBAAlXsEB04Q/GvtL06PqRMLeA== 2021-02-10T11:02:55.864 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2021-02-10T11:02:55.867 DEBUG:teuthology.orchestra.run.gibba013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2021-02-10T11:02:56.240 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2021-02-10T11:02:56.243 DEBUG:teuthology.orchestra.run.gibba013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2021-02-10T11:02:56.619 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-02-10T11:02:56.622 DEBUG:teuthology.orchestra.run.gibba013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-02-10T11:02:56.997 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-02-10T11:02:57.001 DEBUG:teuthology.orchestra.run.gibba013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-02-10T11:02:57.375 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-02-10T11:02:57.379 DEBUG:teuthology.orchestra.run.gibba013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-02-10T11:02:57.753 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-02-10T11:02:57.756 DEBUG:teuthology.orchestra.run.gibba013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-02-10T11:02:58.129 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2021-02-10T11:02:58.132 DEBUG:teuthology.orchestra.run.gibba013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2021-02-10T11:02:58.508 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2021-02-10T11:02:58.512 DEBUG:teuthology.orchestra.run.gibba013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2021-02-10T11:02:58.888 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-02-10T11:02:58.892 DEBUG:teuthology.orchestra.run.gibba013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-02-10T11:02:59.267 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-02-10T11:02:59.270 DEBUG:teuthology.orchestra.run.gibba013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-02-10T11:02:59.645 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-02-10T11:02:59.649 DEBUG:teuthology.orchestra.run.gibba013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-02-10T11:03:00.025 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-02-10T11:03:00.028 DEBUG:teuthology.orchestra.run.gibba013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-02-10T11:03:00.404 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2021-02-10T11:03:00.407 DEBUG:teuthology.orchestra.run.gibba013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2021-02-10T11:03:00.784 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2021-02-10T11:03:00.787 DEBUG:teuthology.orchestra.run.gibba013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2021-02-10T11:03:01.164 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2021-02-10T11:03:01.167 DEBUG:teuthology.orchestra.run.gibba013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2021-02-10T11:03:01.543 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2021-02-10T11:03:01.547 DEBUG:teuthology.orchestra.run.gibba013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2021-02-10T11:03:01.924 INFO:tasks.ceph:Running mkfs on mon nodes... 2021-02-10T11:03:01.924 DEBUG:teuthology.orchestra.run.gibba002:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2021-02-10T11:03:02.268 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2021-02-10T11:03:02.650 DEBUG:teuthology.orchestra.run.gibba002:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2021-02-10T11:03:02.995 DEBUG:teuthology.orchestra.run.gibba013:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2021-02-10T11:03:03.341 DEBUG:teuthology.orchestra.run.gibba013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2021-02-10T11:03:03.723 DEBUG:teuthology.orchestra.run.gibba013:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2021-02-10T11:03:04.065 DEBUG:teuthology.orchestra.run.gibba013:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2021-02-10T11:03:04.411 DEBUG:teuthology.orchestra.run.gibba013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2021-02-10T11:03:04.793 DEBUG:teuthology.orchestra.run.gibba013:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2021-02-10T11:03:05.137 DEBUG:teuthology.orchestra.run.gibba002:> rm -- /home/ubuntu/cephtest/ceph.monmap 2021-02-10T11:03:05.141 DEBUG:teuthology.orchestra.run.gibba013:> rm -- /home/ubuntu/cephtest/ceph.monmap 2021-02-10T11:03:05.476 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2021-02-10T11:03:05.476 INFO:tasks.ceph.mon.a:Restarting daemon 2021-02-10T11:03:05.476 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2021-02-10T11:03:05.479 INFO:tasks.ceph.mon.a:Started 2021-02-10T11:03:05.479 INFO:tasks.ceph.mon.b:Restarting daemon 2021-02-10T11:03:05.480 DEBUG:teuthology.orchestra.run.gibba013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2021-02-10T11:03:05.483 INFO:tasks.ceph.mon.b:Started 2021-02-10T11:03:05.483 INFO:tasks.ceph.mon.c:Restarting daemon 2021-02-10T11:03:05.483 DEBUG:teuthology.orchestra.run.gibba013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2021-02-10T11:03:05.485 INFO:tasks.ceph.mon.c:Started 2021-02-10T11:03:05.486 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2021-02-10T11:03:05.486 INFO:tasks.ceph.mgr.y:Restarting daemon 2021-02-10T11:03:05.486 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2021-02-10T11:03:05.489 INFO:tasks.ceph.mgr.y:Started 2021-02-10T11:03:05.489 INFO:tasks.ceph.mgr.x:Restarting daemon 2021-02-10T11:03:05.489 DEBUG:teuthology.orchestra.run.gibba013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2021-02-10T11:03:05.491 INFO:tasks.ceph.mgr.x:Started 2021-02-10T11:03:05.491 INFO:tasks.ceph:Setting crush tunables to default 2021-02-10T11:03:05.492 DEBUG:teuthology.orchestra.run.gibba002:> sudo ceph --cluster ceph osd crush tunables default 2021-02-10T11:03:06.510 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:05.892 7f48b80b5700 1 Processor -- start 2021-02-10T11:03:06.510 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:05.892 7f48b80b5700 1 -- start start 2021-02-10T11:03:06.511 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:05.892 7f48b80b5700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f48b00cff00 0x7f48b01b39d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:06.511 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:05.892 7f48b80b5700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f48b00d1bf0 0x7f48b00ef620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:06.511 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:05.892 7f48b80b5700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48b00f0b20 con 0x7f48b00d17f0 2021-02-10T11:03:06.511 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:05.892 7f48b80b5700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48b01b3ff0 con 0x7f48b00d1bf0 2021-02-10T11:03:06.511 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:05.892 7f48b80b5700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48b00f20b0 con 0x7f48b00cff00 2021-02-10T11:03:06.511 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:05.892 7f48b5ed1700 1 -- >> v1:172.21.2.113:6789/0 conn(0x7f48b00d17f0 legacy=0x7f48b01b31e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.2.113:6789/0 2021-02-10T11:03:06.511 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:05.892 7f48b56d0700 1 -- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f48b00cff00 msgr2=0x7f48b01b39d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.2.113:3301/0 2021-02-10T11:03:06.511 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:05.892 7f48b66d2700 1 -- >> v2:172.21.2.113:3300/0 conn(0x7f48b00d1bf0 msgr2=0x7f48b00ef620 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.2.113:3300/0 2021-02-10T11:03:06.511 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:05.892 7f48b56d0700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f48b00cff00 0x7f48b01b39d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-02-10T11:03:06.511 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:05.892 7f48b66d2700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f48b00d1bf0 0x7f48b00ef620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-02-10T11:03:06.511 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:06.093 7f48b56d0700 1 -- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f48b00cff00 msgr2=0x7f48b01b39d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.2.113:3301/0 2021-02-10T11:03:06.511 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:06.093 7f48b56d0700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f48b00cff00 0x7f48b01b39d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2021-02-10T11:03:06.512 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:06.093 7f48b66d2700 1 -- >> v2:172.21.2.113:3300/0 conn(0x7f48b00d1bf0 msgr2=0x7f48b00ef620 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.2.113:3300/0 2021-02-10T11:03:06.512 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:06.093 7f48b5ed1700 1 -- >> v1:172.21.2.113:6789/0 conn(0x7f48b00d17f0 legacy=0x7f48b01b31e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.2.113:6789/0 2021-02-10T11:03:06.512 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:06.093 7f48b66d2700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f48b00d1bf0 0x7f48b00ef620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2021-02-10T11:03:06.512 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:06.494 7f48b66d2700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f48b00d1bf0 0x7f48b00ef620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:06.512 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:06.494 7f48b56d0700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f48b00cff00 0x7f48b01b39d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:06.512 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:06.494 7f48b5ed1700 1 --1- >> v1:172.21.2.113:6789/0 conn(0x7f48b00d17f0 0x7f48b01b31e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.113:6789/0 says I am v1:172.21.2.102:48268/0 (socket says 172.21.2.102:48268) 2021-02-10T11:03:06.512 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:06.494 7f48b5ed1700 1 -- 172.21.2.102:0/1811734133 learned_addr learned my addr 172.21.2.102:0/1811734133 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:03:06.512 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:06.494 7f48b56d0700 1 -- 172.21.2.102:0/1811734133 >> v2:172.21.2.113:3300/0 conn(0x7f48b00d1bf0 msgr2=0x7f48b00ef620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:03:06.512 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:06.494 7f48b56d0700 1 --2- 172.21.2.102:0/1811734133 >> v2:172.21.2.113:3300/0 conn(0x7f48b00d1bf0 0x7f48b00ef620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:06.512 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:06.494 7f48b4ecf700 1 -- 172.21.2.102:0/1811734133 <== mon.1 v1:172.21.2.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3140661650 0 0) 0x7f48a8000fe0 con 0x7f48b00d17f0 2021-02-10T11:03:06.512 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:06.494 7f48b56d0700 1 -- 172.21.2.102:0/1811734133 >> v1:172.21.2.113:6789/0 conn(0x7f48b00d17f0 legacy=0x7f48b01b31e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:06.512 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:06.494 7f48b56d0700 1 -- 172.21.2.102:0/1811734133 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48b00f22a0 con 0x7f48b00cff00 2021-02-10T11:03:06.513 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:06.495 7f48b56d0700 1 --2- 172.21.2.102:0/1811734133 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f48b00cff00 0x7f48b01b39d0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f48a0006b50 tx=0x7f48a0009b40).ready entity=mon.2 client_cookie=d57fb28e8f4bf42d server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:03:06.513 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:06.495 7f48b4ecf700 1 -- 172.21.2.102:0/1811734133 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f48a00060c0 con 0x7f48b00cff00 2021-02-10T11:03:06.513 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:06.495 7f48b4ecf700 1 -- 172.21.2.102:0/1811734133 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48a0006490 con 0x7f48b00cff00 2021-02-10T11:03:06.513 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:06.495 7f48b4ecf700 1 -- 172.21.2.102:0/1811734133 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f48a0003680 con 0x7f48b00cff00 2021-02-10T11:03:06.513 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:06.495 7f48b80b5700 1 -- 172.21.2.102:0/1811734133 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f48b00cff00 msgr2=0x7f48b01b39d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:06.513 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:06.495 7f48b80b5700 1 --2- 172.21.2.102:0/1811734133 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f48b00cff00 0x7f48b01b39d0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f48a0006b50 tx=0x7f48a0009b40).stop 2021-02-10T11:03:06.513 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:06.496 7f48b80b5700 1 -- 172.21.2.102:0/1811734133 shutdown_connections 2021-02-10T11:03:06.513 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:06.496 7f48b80b5700 1 --2- 172.21.2.102:0/1811734133 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f48b00cff00 0x7f48b01b39d0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:06.513 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:06.496 7f48b80b5700 1 --2- 172.21.2.102:0/1811734133 >> v2:172.21.2.113:3300/0 conn(0x7f48b00d1bf0 0x7f48b00ef620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:06.513 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:06.496 7f48b80b5700 1 -- 172.21.2.102:0/1811734133 >> 172.21.2.102:0/1811734133 conn(0x7f48b00fde40 msgr2=0x7f48b00ce390 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:03:06.513 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:06.496 7f48b80b5700 1 -- 172.21.2.102:0/1811734133 shutdown_connections 2021-02-10T11:03:06.513 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:06.496 7f48b80b5700 1 -- 172.21.2.102:0/1811734133 wait complete. 2021-02-10T11:03:06.514 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:06.511 7f48b80b5700 1 Processor -- start 2021-02-10T11:03:06.514 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:06.511 7f48b80b5700 1 -- start start 2021-02-10T11:03:06.515 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:06.512 7f48b80b5700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f48b01a49b0 0x7f48b008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:06.515 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:06.512 7f48b80b5700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f48b009e080 0x7f48b00c3100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:06.515 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:06.512 7f48b80b5700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f48b009e480 0x7f48b00c3720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:06.515 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:06.512 7f48b80b5700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48b00c2b80 con 0x7f48b009e080 2021-02-10T11:03:06.515 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:06.512 7f48b80b5700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48b00bee50 con 0x7f48b009e480 2021-02-10T11:03:06.515 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:06.512 7f48b80b5700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48b00c13f0 con 0x7f48b01a49b0 2021-02-10T11:03:06.516 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:06.512 7f48b66d2700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f48b009e480 0x7f48b00c3720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:06.516 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:06.512 7f48b5ed1700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f48b01a49b0 0x7f48b008a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:06.516 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:06.512 7f48b66d2700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f48b009e480 0x7f48b00c3720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.113:3300/0 says I am v2:172.21.2.102:52154/0 (socket says 172.21.2.102:52154) 2021-02-10T11:03:06.516 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:06.512 7f48b66d2700 1 -- 172.21.2.102:0/1543460226 learned_addr learned my addr 172.21.2.102:0/1543460226 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:03:06.516 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:06.512 7f48b56d0700 1 --2- 172.21.2.102:0/1543460226 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f48b009e080 0x7f48b00c3100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:06.516 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:06.512 7f48b66d2700 1 -- 172.21.2.102:0/1543460226 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f48b01a49b0 msgr2=0x7f48b008a400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:06.516 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:06.512 7f48b66d2700 1 --2- 172.21.2.102:0/1543460226 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f48b01a49b0 0x7f48b008a400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:06.516 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:06.512 7f48b66d2700 1 -- 172.21.2.102:0/1543460226 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f48b009e080 msgr2=0x7f48b00c3100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:06.516 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:06.512 7f48b66d2700 1 --2- 172.21.2.102:0/1543460226 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f48b009e080 0x7f48b00c3100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:06.516 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:06.512 7f48b66d2700 1 -- 172.21.2.102:0/1543460226 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48b00c1610 con 0x7f48b009e480 2021-02-10T11:03:06.516 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:06.512 7f48b5ed1700 1 --2- 172.21.2.102:0/1543460226 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f48b01a49b0 0x7f48b008a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:03:06.516 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:06.513 7f48b66d2700 1 --2- 172.21.2.102:0/1543460226 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f48b009e480 0x7f48b00c3720 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f48ac00b590 tx=0x7f48ac00b930).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:03:06.516 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:06.513 7f48a67fc700 1 -- 172.21.2.102:0/1543460226 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f48ac00bd20 con 0x7f48b009e480 2021-02-10T11:03:06.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:06.513 7f48a67fc700 1 -- 172.21.2.102:0/1543460226 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48ac0060d0 con 0x7f48b009e480 2021-02-10T11:03:06.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:06.513 7f48b80b5700 1 -- 172.21.2.102:0/1543460226 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48b00c1610 con 0x7f48b009e480 2021-02-10T11:03:06.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:06.513 7f48a67fc700 1 -- 172.21.2.102:0/1543460226 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f48ac003820 con 0x7f48b009e480 2021-02-10T11:03:06.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:06.513 7f48b80b5700 1 -- 172.21.2.102:0/1543460226 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48b00c1610 con 0x7f48b009e480 2021-02-10T11:03:06.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:06.513 7f48a67fc700 1 -- 172.21.2.102:0/1543460226 <== mon.1 v2:172.21.2.113:3300/0 4 ==== mgrmap(e 1) v1 ==== 190+0+0 (secure 0 0 0) 0x7f48ac00bd20 con 0x7f48b009e480 2021-02-10T11:03:06.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:06.513 7f48a67fc700 1 -- 172.21.2.102:0/1543460226 <== mon.1 v2:172.21.2.113:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f48ac003910 con 0x7f48b009e480 2021-02-10T11:03:06.543 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:06.544 7f48b80b5700 1 -- 172.21.2.102:0/1543460226 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48b0103690 con 0x7f48b009e480 2021-02-10T11:03:06.545 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:06.546 7f48a67fc700 1 -- 172.21.2.102:0/1543460226 <== mon.1 v2:172.21.2.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+66016 (secure 0 0 0) 0x7f48ac000990 con 0x7f48b009e480 2021-02-10T11:03:06.616 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:06.616 7f48b80b5700 1 -- 172.21.2.102:0/1543460226 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_command({"profile": "default", "prefix": "osd crush tunables"} v 0) v1 -- 0x7f48b0103690 con 0x7f48b009e480 2021-02-10T11:03:06.619 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:06.620 7f48a67fc700 1 -- 172.21.2.102:0/1543460226 <== mon.1 v2:172.21.2.113:3300/0 7 ==== mon_command_ack([{"profile": "default", "prefix": "osd crush tunables"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7f48ac000990 con 0x7f48b009e480 2021-02-10T11:03:06.623 INFO:teuthology.orchestra.run.gibba002.stderr:adjusted tunables profile to default 2021-02-10T11:03:06.624 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:06.624 7f48b80b5700 1 -- 172.21.2.102:0/1543460226 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f48b009e480 msgr2=0x7f48b00c3720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:06.624 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:06.624 7f48b80b5700 1 --2- 172.21.2.102:0/1543460226 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f48b009e480 0x7f48b00c3720 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f48ac00b590 tx=0x7f48ac00b930).stop 2021-02-10T11:03:06.624 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:06.625 7f48b80b5700 1 -- 172.21.2.102:0/1543460226 shutdown_connections 2021-02-10T11:03:06.624 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:06.625 7f48b80b5700 1 --2- 172.21.2.102:0/1543460226 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f48b009e480 0x7f48b00c3720 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:06.624 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:06.625 7f48b80b5700 1 --2- 172.21.2.102:0/1543460226 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f48b009e080 0x7f48b00c3100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:06.624 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:06.625 7f48b80b5700 1 --2- 172.21.2.102:0/1543460226 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f48b01a49b0 0x7f48b008a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:06.624 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:06.625 7f48b80b5700 1 -- 172.21.2.102:0/1543460226 >> 172.21.2.102:0/1543460226 conn(0x7f48b00a1690 msgr2=0x7f48b00a94c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:03:06.624 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:06.625 7f48b80b5700 1 -- 172.21.2.102:0/1543460226 shutdown_connections 2021-02-10T11:03:06.625 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:06.625 7f48b80b5700 1 -- 172.21.2.102:0/1543460226 wait complete. 2021-02-10T11:03:06.632 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2021-02-10T11:03:06.632 DEBUG:teuthology.orchestra.run.gibba002:> mktemp 2021-02-10T11:03:06.965 INFO:teuthology.orchestra.run.gibba002.stdout:/tmp/tmp.J43dBSfISw 2021-02-10T11:03:06.965 DEBUG:teuthology.orchestra.run.gibba002:> sudo cp /var/lib/ceph/osd/ceph-0/fsid /tmp/tmp.J43dBSfISw 2021-02-10T11:03:07.308 DEBUG:teuthology.orchestra.run.gibba002:> sudo chmod 0666 /tmp/tmp.J43dBSfISw 2021-02-10T11:03:07.991 DEBUG:teuthology.orchestra.remote:gibba002:/tmp/tmp.J43dBSfISw is 37B 2021-02-10T11:03:08.339 DEBUG:teuthology.orchestra.run.gibba002:> rm -fr /tmp/tmp.J43dBSfISw 2021-02-10T11:03:08.679 DEBUG:teuthology.orchestra.run.gibba002:> mktemp 2021-02-10T11:03:09.016 INFO:teuthology.orchestra.run.gibba002.stdout:/tmp/tmp.g20W65Dch2 2021-02-10T11:03:09.016 DEBUG:teuthology.orchestra.run.gibba002:> sudo cp /var/lib/ceph/osd/ceph-1/fsid /tmp/tmp.g20W65Dch2 2021-02-10T11:03:09.367 DEBUG:teuthology.orchestra.run.gibba002:> sudo chmod 0666 /tmp/tmp.g20W65Dch2 2021-02-10T11:03:10.063 DEBUG:teuthology.orchestra.remote:gibba002:/tmp/tmp.g20W65Dch2 is 37B 2021-02-10T11:03:10.416 DEBUG:teuthology.orchestra.run.gibba002:> rm -fr /tmp/tmp.g20W65Dch2 2021-02-10T11:03:10.425 DEBUG:teuthology.orchestra.run.gibba002:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:03:10.472 DEBUG:teuthology.orchestra.run.gibba013:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:03:10.754 DEBUG:teuthology.orchestra.run.gibba002:> mktemp 2021-02-10T11:03:11.089 INFO:teuthology.orchestra.run.gibba002.stdout:/tmp/tmp.3irNBxtCzG 2021-02-10T11:03:11.090 DEBUG:teuthology.orchestra.run.gibba002:> sudo cp /var/lib/ceph/osd/ceph-2/fsid /tmp/tmp.3irNBxtCzG 2021-02-10T11:03:11.434 DEBUG:teuthology.orchestra.run.gibba002:> sudo chmod 0666 /tmp/tmp.3irNBxtCzG 2021-02-10T11:03:12.115 DEBUG:teuthology.orchestra.remote:gibba002:/tmp/tmp.3irNBxtCzG is 37B 2021-02-10T11:03:12.455 DEBUG:teuthology.orchestra.run.gibba002:> rm -fr /tmp/tmp.3irNBxtCzG 2021-02-10T11:03:12.792 DEBUG:teuthology.orchestra.run.gibba002:> mktemp 2021-02-10T11:03:13.125 INFO:teuthology.orchestra.run.gibba002.stdout:/tmp/tmp.CiY8FhB5lx 2021-02-10T11:03:13.126 DEBUG:teuthology.orchestra.run.gibba002:> sudo cp /var/lib/ceph/osd/ceph-3/fsid /tmp/tmp.CiY8FhB5lx 2021-02-10T11:03:13.469 DEBUG:teuthology.orchestra.run.gibba002:> sudo chmod 0666 /tmp/tmp.CiY8FhB5lx 2021-02-10T11:03:14.150 DEBUG:teuthology.orchestra.remote:gibba002:/tmp/tmp.CiY8FhB5lx is 37B 2021-02-10T11:03:14.493 DEBUG:teuthology.orchestra.run.gibba002:> rm -fr /tmp/tmp.CiY8FhB5lx 2021-02-10T11:03:14.830 DEBUG:teuthology.orchestra.run.gibba013:> mktemp 2021-02-10T11:03:15.165 INFO:teuthology.orchestra.run.gibba013.stdout:/tmp/tmp.r6B9OoImPz 2021-02-10T11:03:15.165 DEBUG:teuthology.orchestra.run.gibba013:> sudo cp /var/lib/ceph/osd/ceph-4/fsid /tmp/tmp.r6B9OoImPz 2021-02-10T11:03:15.509 DEBUG:teuthology.orchestra.run.gibba013:> sudo chmod 0666 /tmp/tmp.r6B9OoImPz 2021-02-10T11:03:16.190 DEBUG:teuthology.orchestra.remote:gibba013:/tmp/tmp.r6B9OoImPz is 37B 2021-02-10T11:03:16.529 DEBUG:teuthology.orchestra.run.gibba013:> rm -fr /tmp/tmp.r6B9OoImPz 2021-02-10T11:03:16.863 DEBUG:teuthology.orchestra.run.gibba013:> mktemp 2021-02-10T11:03:17.199 INFO:teuthology.orchestra.run.gibba013.stdout:/tmp/tmp.xSr9LoZQKW 2021-02-10T11:03:17.199 DEBUG:teuthology.orchestra.run.gibba013:> sudo cp /var/lib/ceph/osd/ceph-5/fsid /tmp/tmp.xSr9LoZQKW 2021-02-10T11:03:17.544 DEBUG:teuthology.orchestra.run.gibba013:> sudo chmod 0666 /tmp/tmp.xSr9LoZQKW 2021-02-10T11:03:18.224 DEBUG:teuthology.orchestra.remote:gibba013:/tmp/tmp.xSr9LoZQKW is 37B 2021-02-10T11:03:18.565 DEBUG:teuthology.orchestra.run.gibba013:> rm -fr /tmp/tmp.xSr9LoZQKW 2021-02-10T11:03:18.900 DEBUG:teuthology.orchestra.run.gibba013:> mktemp 2021-02-10T11:03:19.233 INFO:teuthology.orchestra.run.gibba013.stdout:/tmp/tmp.Uvo7qhMMJj 2021-02-10T11:03:19.233 DEBUG:teuthology.orchestra.run.gibba013:> sudo cp /var/lib/ceph/osd/ceph-6/fsid /tmp/tmp.Uvo7qhMMJj 2021-02-10T11:03:19.576 DEBUG:teuthology.orchestra.run.gibba013:> sudo chmod 0666 /tmp/tmp.Uvo7qhMMJj 2021-02-10T11:03:20.265 DEBUG:teuthology.orchestra.remote:gibba013:/tmp/tmp.Uvo7qhMMJj is 37B 2021-02-10T11:03:20.606 DEBUG:teuthology.orchestra.run.gibba013:> rm -fr /tmp/tmp.Uvo7qhMMJj 2021-02-10T11:03:20.939 DEBUG:teuthology.orchestra.run.gibba013:> mktemp 2021-02-10T11:03:21.273 INFO:teuthology.orchestra.run.gibba013.stdout:/tmp/tmp.J1ySsmZQli 2021-02-10T11:03:21.273 DEBUG:teuthology.orchestra.run.gibba013:> sudo cp /var/lib/ceph/osd/ceph-7/fsid /tmp/tmp.J1ySsmZQli 2021-02-10T11:03:21.615 DEBUG:teuthology.orchestra.run.gibba013:> sudo chmod 0666 /tmp/tmp.J1ySsmZQli 2021-02-10T11:03:22.298 DEBUG:teuthology.orchestra.remote:gibba013:/tmp/tmp.J1ySsmZQli is 37B 2021-02-10T11:03:22.638 DEBUG:teuthology.orchestra.run.gibba013:> rm -fr /tmp/tmp.J1ySsmZQli 2021-02-10T11:03:22.971 DEBUG:teuthology.orchestra.run.gibba013:> sudo ceph --cluster ceph osd new be691b4d-a039-408a-af80-9d53f02b6f1f 0 2021-02-10T11:03:23.397 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:23.380 7fa0fe687700 1 Processor -- start 2021-02-10T11:03:23.398 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:23.380 7fa0fe687700 1 -- start start 2021-02-10T11:03:23.398 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:23.381 7fa0fe687700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fa0f80cff00 0x7fa0f81b39d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:23.398 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:23.381 7fa0fe687700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fa0f80d1bf0 0x7fa0f80ef620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:23.398 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:23.381 7fa0fe687700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0f80f0b20 con 0x7fa0f80d17f0 2021-02-10T11:03:23.398 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:23.381 7fa0fe687700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0f80e5eb0 con 0x7fa0f80d1bf0 2021-02-10T11:03:23.398 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:23.381 7fa0fe687700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0f81b4020 con 0x7fa0f80cff00 2021-02-10T11:03:23.398 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:23.381 7fa0f77fe700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fa0f80cff00 0x7fa0f81b39d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:23.398 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:23.381 7fa0f77fe700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fa0f80cff00 0x7fa0f81b39d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.113:3300/0 says I am v2:172.21.2.113:60312/0 (socket says 172.21.2.113:60312) 2021-02-10T11:03:23.398 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:23.381 7fa0f77fe700 1 -- 172.21.2.113:0/2564524691 learned_addr learned my addr 172.21.2.113:0/2564524691 (peer_addr_for_me v2:172.21.2.113:0/0) 2021-02-10T11:03:23.398 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:23.381 7fa0fcca4700 1 --2- 172.21.2.113:0/2564524691 >> v2:172.21.2.102:3300/0 conn(0x7fa0f80d1bf0 0x7fa0f80ef620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:23.398 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:23.381 7fa0f77fe700 1 -- 172.21.2.113:0/2564524691 >> v2:172.21.2.102:3300/0 conn(0x7fa0f80d1bf0 msgr2=0x7fa0f80ef620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:03:23.398 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:23.381 7fa0f77fe700 1 --2- 172.21.2.113:0/2564524691 >> v2:172.21.2.102:3300/0 conn(0x7fa0f80d1bf0 0x7fa0f80ef620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:23.399 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:23.381 7fa0f77fe700 1 -- 172.21.2.113:0/2564524691 >> v1:172.21.2.102:6789/0 conn(0x7fa0f80d17f0 legacy=0x7fa0f81b31e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:23.399 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:23.381 7fa0f77fe700 1 -- 172.21.2.113:0/2564524691 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0f80f20b0 con 0x7fa0f80cff00 2021-02-10T11:03:23.399 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:23.382 7fa0f77fe700 1 --2- 172.21.2.113:0/2564524691 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa0f80cff00 0x7fa0f81b39d0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7fa0ec006700 tx=0x7fa0ec009b80).ready entity=mon.1 client_cookie=e1047f2c5863ffaa server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:03:23.399 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:23.382 7fa0f6ffd700 1 -- 172.21.2.113:0/2564524691 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa0ec00c0c0 con 0x7fa0f80cff00 2021-02-10T11:03:23.399 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:23.382 7fa0f6ffd700 1 -- 172.21.2.113:0/2564524691 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0ec00c3c0 con 0x7fa0f80cff00 2021-02-10T11:03:23.399 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:23.382 7fa0f6ffd700 1 -- 172.21.2.113:0/2564524691 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa0ec00c8e0 con 0x7fa0f80cff00 2021-02-10T11:03:23.399 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:23.382 7fa0fe687700 1 -- 172.21.2.113:0/2564524691 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa0f80cff00 msgr2=0x7fa0f81b39d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:23.399 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:23.382 7fa0fe687700 1 --2- 172.21.2.113:0/2564524691 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa0f80cff00 0x7fa0f81b39d0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7fa0ec006700 tx=0x7fa0ec009b80).stop 2021-02-10T11:03:23.399 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:23.382 7fa0fe687700 1 -- 172.21.2.113:0/2564524691 shutdown_connections 2021-02-10T11:03:23.399 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:23.382 7fa0fe687700 1 --2- 172.21.2.113:0/2564524691 >> v2:172.21.2.102:3300/0 conn(0x7fa0f80d1bf0 0x7fa0f80ef620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:23.399 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:23.382 7fa0fe687700 1 --2- 172.21.2.113:0/2564524691 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa0f80cff00 0x7fa0f81b39d0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:23.399 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:23.382 7fa0fe687700 1 -- 172.21.2.113:0/2564524691 >> 172.21.2.113:0/2564524691 conn(0x7fa0f80fde40 msgr2=0x7fa0f80ce390 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:03:23.399 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:23.382 7fa0fe687700 1 -- 172.21.2.113:0/2564524691 shutdown_connections 2021-02-10T11:03:23.400 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:23.382 7fa0fe687700 1 -- 172.21.2.113:0/2564524691 wait complete. 2021-02-10T11:03:23.400 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:23.397 7fa0fe687700 1 Processor -- start 2021-02-10T11:03:23.402 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:23.397 7fa0fe687700 1 -- start start 2021-02-10T11:03:23.402 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:23.397 7fa0fe687700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa0f81038f0 0x7fa0f81981c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:23.402 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:23.397 7fa0fe687700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa0f8198a00 0x7fa0f8198e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:23.402 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:23.397 7fa0fe687700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa0f8199420 0x7fa0f80b1330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:23.402 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:23.397 7fa0fe687700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0f80990c0 con 0x7fa0f8199420 2021-02-10T11:03:23.402 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:23.397 7fa0fe687700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0f80b1950 con 0x7fa0f81038f0 2021-02-10T11:03:23.402 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:23.397 7fa0fe687700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0f80b1b30 con 0x7fa0f8198a00 2021-02-10T11:03:23.402 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:23.398 7fa0f7fff700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa0f81038f0 0x7fa0f81981c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:23.402 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:23.398 7fa0f77fe700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa0f8198a00 0x7fa0f8198e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:23.402 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:23.398 7fa0f7fff700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa0f81038f0 0x7fa0f81981c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.113:3300/0 says I am v2:172.21.2.113:60318/0 (socket says 172.21.2.113:60318) 2021-02-10T11:03:23.402 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:23.398 7fa0f7fff700 1 -- 172.21.2.113:0/2428666090 learned_addr learned my addr 172.21.2.113:0/2428666090 (peer_addr_for_me v2:172.21.2.113:0/0) 2021-02-10T11:03:23.402 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:23.398 7fa0f77fe700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa0f8198a00 0x7fa0f8198e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.113:3301/0 says I am v2:172.21.2.113:60430/0 (socket says 172.21.2.113:60430) 2021-02-10T11:03:23.403 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:23.398 7fa0fcca4700 1 --2- 172.21.2.113:0/2428666090 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa0f8199420 0x7fa0f80b1330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:23.403 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:23.398 7fa0f7fff700 1 -- 172.21.2.113:0/2428666090 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa0f8198a00 msgr2=0x7fa0f8198e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:23.403 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:23.398 7fa0f7fff700 1 --2- 172.21.2.113:0/2428666090 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa0f8198a00 0x7fa0f8198e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:23.403 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:23.398 7fa0f7fff700 1 -- 172.21.2.113:0/2428666090 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa0f8199420 msgr2=0x7fa0f80b1330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:23.403 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:23.398 7fa0f7fff700 1 --2- 172.21.2.113:0/2428666090 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa0f8199420 0x7fa0f80b1330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:23.403 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:23.398 7fa0f7fff700 1 -- 172.21.2.113:0/2428666090 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0f80b1d50 con 0x7fa0f81038f0 2021-02-10T11:03:23.403 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:23.398 7fa0f77fe700 1 --2- 172.21.2.113:0/2428666090 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa0f8198a00 0x7fa0f8198e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:03:23.403 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:23.398 7fa0f7fff700 1 --2- 172.21.2.113:0/2428666090 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa0f81038f0 0x7fa0f81981c0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7fa0e800b5b0 tx=0x7fa0e800b950).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:03:23.403 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:23.399 7fa0f4ff9700 1 -- 172.21.2.113:0/2428666090 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa0e800bd40 con 0x7fa0f81038f0 2021-02-10T11:03:23.403 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:23.399 7fa0f4ff9700 1 -- 172.21.2.113:0/2428666090 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0e8000c60 con 0x7fa0f81038f0 2021-02-10T11:03:23.403 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:23.399 7fa0f4ff9700 1 -- 172.21.2.113:0/2428666090 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa0e8006210 con 0x7fa0f81038f0 2021-02-10T11:03:23.403 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:23.399 7fa0fe687700 1 -- 172.21.2.113:0/2428666090 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0f80b1d50 con 0x7fa0f81038f0 2021-02-10T11:03:23.403 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:23.399 7fa0fe687700 1 -- 172.21.2.113:0/2428666090 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0f8099640 con 0x7fa0f81038f0 2021-02-10T11:03:23.403 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:23.399 7fa0f4ff9700 1 -- 172.21.2.113:0/2428666090 <== mon.1 v2:172.21.2.113:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa0e8006020 con 0x7fa0f81038f0 2021-02-10T11:03:23.404 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:23.400 7fa0f4ff9700 1 --2- 172.21.2.113:0/2428666090 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa0d80186c0 0x7fa0d801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:23.404 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:23.400 7fa0f4ff9700 1 -- 172.21.2.113:0/2428666090 <== mon.1 v2:172.21.2.113:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 754+0+0 (secure 0 0 0) 0x7fa0e80310f0 con 0x7fa0f81038f0 2021-02-10T11:03:23.404 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:23.400 7fa0f77fe700 1 --2- 172.21.2.113:0/2428666090 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa0d80186c0 0x7fa0d801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:23.404 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:23.401 7fa0f77fe700 1 --2- 172.21.2.113:0/2428666090 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa0d80186c0 0x7fa0d801abc0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:03:23.431 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:23.431 7fa0fe687700 1 -- 172.21.2.113:0/2428666090 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0f8103cf0 con 0x7fa0f81038f0 2021-02-10T11:03:23.434 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:23.433 7fa0f4ff9700 1 -- 172.21.2.113:0/2428666090 <== mon.1 v2:172.21.2.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fa0e8006230 con 0x7fa0f81038f0 2021-02-10T11:03:23.569 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:23.568 7fa0fe687700 1 -- 172.21.2.113:0/2428666090 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_command({"prefix": "osd new", "uuid": "be691b4d-a039-408a-af80-9d53f02b6f1f", "id": 0} v 0) v1 -- 0x7fa0f8103cf0 con 0x7fa0f81038f0 2021-02-10T11:03:23.574 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:23.574 7fa0f4ff9700 1 -- 172.21.2.113:0/2428666090 <== mon.1 v2:172.21.2.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "be691b4d-a039-408a-af80-9d53f02b6f1f", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7fa0e8006000 con 0x7fa0f81038f0 2021-02-10T11:03:23.577 INFO:teuthology.orchestra.run.gibba013.stdout:0 2021-02-10T11:03:23.578 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:23.577 7fa0fe687700 1 -- 172.21.2.113:0/2428666090 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa0d80186c0 msgr2=0x7fa0d801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:23.578 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:23.577 7fa0fe687700 1 --2- 172.21.2.113:0/2428666090 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa0d80186c0 0x7fa0d801abc0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:23.578 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:23.577 7fa0fe687700 1 -- 172.21.2.113:0/2428666090 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa0f81038f0 msgr2=0x7fa0f81981c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:23.578 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:23.577 7fa0fe687700 1 --2- 172.21.2.113:0/2428666090 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa0f81038f0 0x7fa0f81981c0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7fa0e800b5b0 tx=0x7fa0e800b950).stop 2021-02-10T11:03:23.579 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:23.578 7fa0fe687700 1 -- 172.21.2.113:0/2428666090 shutdown_connections 2021-02-10T11:03:23.579 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:23.578 7fa0fe687700 1 --2- 172.21.2.113:0/2428666090 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa0f8199420 0x7fa0f80b1330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:23.579 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:23.578 7fa0fe687700 1 --2- 172.21.2.113:0/2428666090 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa0f8198a00 0x7fa0f8198e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:23.579 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:23.578 7fa0fe687700 1 --2- 172.21.2.113:0/2428666090 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa0d80186c0 0x7fa0d801abc0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:23.579 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:23.578 7fa0fe687700 1 --2- 172.21.2.113:0/2428666090 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa0f81038f0 0x7fa0f81981c0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:23.579 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:23.578 7fa0fe687700 1 -- 172.21.2.113:0/2428666090 >> 172.21.2.113:0/2428666090 conn(0x7fa0f808dd20 msgr2=0x7fa0f80959b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:03:23.579 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:23.578 7fa0fe687700 1 -- 172.21.2.113:0/2428666090 shutdown_connections 2021-02-10T11:03:23.580 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:23.578 7fa0fe687700 1 -- 172.21.2.113:0/2428666090 wait complete. 2021-02-10T11:03:23.586 DEBUG:teuthology.orchestra.run.gibba013:> sudo ceph --cluster ceph osd new 64e3ccc8-9c69-4d7c-865e-af8e83822635 1 2021-02-10T11:03:24.013 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:23.996 7fe533c8d700 1 Processor -- start 2021-02-10T11:03:24.014 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:23.996 7fe533c8d700 1 -- start start 2021-02-10T11:03:24.014 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:23.996 7fe533c8d700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fe52c0d17f0 0x7fe52c1b31e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:24.014 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:23.996 7fe533c8d700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe52c0cff00 0x7fe52c0ef620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:24.014 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:23.996 7fe533c8d700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe52c0f0b20 con 0x7fe52c0d1bf0 2021-02-10T11:03:24.014 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:23.996 7fe533c8d700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe52c1b3ff0 con 0x7fe52c0d17f0 2021-02-10T11:03:24.014 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:23.996 7fe533c8d700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe52c0f20b0 con 0x7fe52c0cff00 2021-02-10T11:03:24.014 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:23.997 7fe5322aa700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe52c0cff00 0x7fe52c0ef620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:24.014 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:23.997 7fe531aa9700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fe52c0d17f0 0x7fe52c1b31e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:24.014 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:23.997 7fe5322aa700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe52c0cff00 0x7fe52c0ef620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.113:3301/0 says I am v2:172.21.2.113:60438/0 (socket says 172.21.2.113:60438) 2021-02-10T11:03:24.014 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:23.997 7fe5322aa700 1 -- 172.21.2.113:0/3043808450 learned_addr learned my addr 172.21.2.113:0/3043808450 (peer_addr_for_me v2:172.21.2.113:0/0) 2021-02-10T11:03:24.014 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:23.997 7fe5322aa700 1 -- 172.21.2.113:0/3043808450 >> v2:172.21.2.113:3300/0 conn(0x7fe52c0d17f0 msgr2=0x7fe52c1b31e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:03:24.015 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:23.997 7fe5322aa700 1 --2- 172.21.2.113:0/3043808450 >> v2:172.21.2.113:3300/0 conn(0x7fe52c0d17f0 0x7fe52c1b31e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:24.015 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:23.997 7fe5322aa700 1 -- 172.21.2.113:0/3043808450 >> v1:172.21.2.102:6789/0 conn(0x7fe52c0d1bf0 legacy=0x7fe52c1b3800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:03:24.015 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:23.997 7fe5322aa700 1 -- 172.21.2.113:0/3043808450 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe52c0f22d0 con 0x7fe52c0cff00 2021-02-10T11:03:24.015 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:23.997 7fe531aa9700 1 --2- 172.21.2.113:0/3043808450 >> v2:172.21.2.113:3300/0 conn(0x7fe52c0d17f0 0x7fe52c1b31e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:03:24.015 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:23.997 7fe5322aa700 1 --2- 172.21.2.113:0/3043808450 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe52c0cff00 0x7fe52c0ef620 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7fe524006f00 tx=0x7fe524009f10).ready entity=mon.2 client_cookie=294be8b3c805cc0b server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:03:24.015 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:23.997 7fe530aa7700 1 -- 172.21.2.113:0/3043808450 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe52400b3b0 con 0x7fe52c0cff00 2021-02-10T11:03:24.015 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:23.997 7fe530aa7700 1 -- 172.21.2.113:0/3043808450 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe52400b730 con 0x7fe52c0cff00 2021-02-10T11:03:24.015 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:23.997 7fe530aa7700 1 -- 172.21.2.113:0/3043808450 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe52400bc50 con 0x7fe52c0cff00 2021-02-10T11:03:24.015 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:23.997 7fe533c8d700 1 -- 172.21.2.113:0/3043808450 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe52c0cff00 msgr2=0x7fe52c0ef620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:24.015 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:23.997 7fe533c8d700 1 --2- 172.21.2.113:0/3043808450 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe52c0cff00 0x7fe52c0ef620 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7fe524006f00 tx=0x7fe524009f10).stop 2021-02-10T11:03:24.015 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:23.998 7fe533c8d700 1 -- 172.21.2.113:0/3043808450 shutdown_connections 2021-02-10T11:03:24.015 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:23.998 7fe533c8d700 1 --2- 172.21.2.113:0/3043808450 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe52c0cff00 0x7fe52c0ef620 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:24.015 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:23.998 7fe533c8d700 1 --2- 172.21.2.113:0/3043808450 >> v2:172.21.2.113:3300/0 conn(0x7fe52c0d17f0 0x7fe52c1b31e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:24.016 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:23.998 7fe533c8d700 1 -- 172.21.2.113:0/3043808450 >> 172.21.2.113:0/3043808450 conn(0x7fe52c0fde40 msgr2=0x7fe52c0ce390 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:03:24.016 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:23.998 7fe533c8d700 1 -- 172.21.2.113:0/3043808450 shutdown_connections 2021-02-10T11:03:24.016 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:23.998 7fe533c8d700 1 -- 172.21.2.113:0/3043808450 wait complete. 2021-02-10T11:03:24.016 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:24.013 7fe533c8d700 1 Processor -- start 2021-02-10T11:03:24.017 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:24.013 7fe533c8d700 1 -- start start 2021-02-10T11:03:24.017 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:24.013 7fe533c8d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe52c0ae340 0x7fe52c0ae740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:24.017 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:24.013 7fe533c8d700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe52c106a40 0x7fe52c106e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:24.018 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:24.013 7fe533c8d700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe52c187e70 0x7fe52c188270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:24.018 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:24.013 7fe533c8d700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe52c1076e0 con 0x7fe52c0ae340 2021-02-10T11:03:24.018 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:24.013 7fe533c8d700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe52c095020 con 0x7fe52c187e70 2021-02-10T11:03:24.018 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:24.013 7fe533c8d700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe52c095200 con 0x7fe52c106a40 2021-02-10T11:03:24.018 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:24.014 7fe5322aa700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe52c187e70 0x7fe52c188270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:24.018 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:24.014 7fe5312a8700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe52c106a40 0x7fe52c106e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:24.018 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:24.014 7fe5322aa700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe52c187e70 0x7fe52c188270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.113:3300/0 says I am v2:172.21.2.113:60326/0 (socket says 172.21.2.113:60326) 2021-02-10T11:03:24.018 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:24.014 7fe5322aa700 1 -- 172.21.2.113:0/2160715294 learned_addr learned my addr 172.21.2.113:0/2160715294 (peer_addr_for_me v2:172.21.2.113:0/0) 2021-02-10T11:03:24.018 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:24.014 7fe5322aa700 1 -- 172.21.2.113:0/2160715294 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe52c106a40 msgr2=0x7fe52c106e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:24.018 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:24.014 7fe5322aa700 1 --2- 172.21.2.113:0/2160715294 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe52c106a40 0x7fe52c106e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:24.018 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:24.014 7fe5322aa700 1 -- 172.21.2.113:0/2160715294 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe52c0ae340 msgr2=0x7fe52c0ae740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:24.018 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:24.014 7fe5322aa700 1 --2- 172.21.2.113:0/2160715294 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe52c0ae340 0x7fe52c0ae740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:03:24.019 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:24.014 7fe5322aa700 1 -- 172.21.2.113:0/2160715294 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe52c1a8a40 con 0x7fe52c187e70 2021-02-10T11:03:24.019 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:24.014 7fe5322aa700 1 --2- 172.21.2.113:0/2160715294 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe52c187e70 0x7fe52c188270 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7fe524009970 tx=0x7fe52400bcc0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:03:24.019 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:24.014 7fe51e7fc700 1 -- 172.21.2.113:0/2160715294 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe524009450 con 0x7fe52c187e70 2021-02-10T11:03:24.019 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:24.014 7fe51e7fc700 1 -- 172.21.2.113:0/2160715294 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe524000cc0 con 0x7fe52c187e70 2021-02-10T11:03:24.019 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:24.014 7fe51e7fc700 1 -- 172.21.2.113:0/2160715294 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe52400c4e0 con 0x7fe52c187e70 2021-02-10T11:03:24.019 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:24.014 7fe533c8d700 1 -- 172.21.2.113:0/2160715294 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe52c1a8a40 con 0x7fe52c187e70 2021-02-10T11:03:24.019 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:24.015 7fe533c8d700 1 -- 172.21.2.113:0/2160715294 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe52c1a8a40 con 0x7fe52c187e70 2021-02-10T11:03:24.019 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:24.015 7fe51e7fc700 1 -- 172.21.2.113:0/2160715294 <== mon.1 v2:172.21.2.113:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fe524001270 con 0x7fe52c187e70 2021-02-10T11:03:24.019 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:24.015 7fe51e7fc700 1 --2- 172.21.2.113:0/2160715294 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe5140186c0 0x7fe51401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:24.019 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:24.015 7fe51e7fc700 1 -- 172.21.2.113:0/2160715294 <== mon.1 v2:172.21.2.113:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 857+0+0 (secure 0 0 0) 0x7fe5240318d0 con 0x7fe52c187e70 2021-02-10T11:03:24.019 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:24.016 7fe531aa9700 1 --2- 172.21.2.113:0/2160715294 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe5140186c0 0x7fe51401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:24.019 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:24.016 7fe531aa9700 1 --2- 172.21.2.113:0/2160715294 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe5140186c0 0x7fe51401abc0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:03:24.047 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:24.046 7fe533c8d700 1 -- 172.21.2.113:0/2160715294 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe52c103690 con 0x7fe52c187e70 2021-02-10T11:03:24.050 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:24.049 7fe51e7fc700 1 -- 172.21.2.113:0/2160715294 <== mon.1 v2:172.21.2.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fe52400be00 con 0x7fe52c187e70 2021-02-10T11:03:24.184 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:24.184 7fe533c8d700 1 -- 172.21.2.113:0/2160715294 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_command({"prefix": "osd new", "uuid": "64e3ccc8-9c69-4d7c-865e-af8e83822635", "id": 1} v 0) v1 -- 0x7fe52c0f0b20 con 0x7fe52c187e70 2021-02-10T11:03:24.189 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:24.189 7fe51e7fc700 1 -- 172.21.2.113:0/2160715294 <== mon.1 v2:172.21.2.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "64e3ccc8-9c69-4d7c-865e-af8e83822635", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7fe524000cc0 con 0x7fe52c187e70 2021-02-10T11:03:24.192 INFO:teuthology.orchestra.run.gibba013.stdout:1 2021-02-10T11:03:24.194 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:24.193 7fe533c8d700 1 -- 172.21.2.113:0/2160715294 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe5140186c0 msgr2=0x7fe51401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:24.194 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:24.193 7fe533c8d700 1 --2- 172.21.2.113:0/2160715294 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe5140186c0 0x7fe51401abc0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:24.194 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:24.193 7fe533c8d700 1 -- 172.21.2.113:0/2160715294 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe52c187e70 msgr2=0x7fe52c188270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:24.194 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:24.193 7fe533c8d700 1 --2- 172.21.2.113:0/2160715294 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe52c187e70 0x7fe52c188270 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7fe524009970 tx=0x7fe52400bcc0).stop 2021-02-10T11:03:24.194 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:24.193 7fe533c8d700 1 -- 172.21.2.113:0/2160715294 shutdown_connections 2021-02-10T11:03:24.194 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:24.193 7fe533c8d700 1 --2- 172.21.2.113:0/2160715294 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe5140186c0 0x7fe51401abc0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:24.194 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:24.193 7fe533c8d700 1 --2- 172.21.2.113:0/2160715294 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe52c187e70 0x7fe52c188270 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:24.194 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:24.193 7fe533c8d700 1 --2- 172.21.2.113:0/2160715294 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe52c106a40 0x7fe52c106e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:24.194 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:24.193 7fe533c8d700 1 --2- 172.21.2.113:0/2160715294 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe52c0ae340 0x7fe52c0ae740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:03:24.195 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:24.193 7fe533c8d700 1 -- 172.21.2.113:0/2160715294 >> 172.21.2.113:0/2160715294 conn(0x7fe52c1ba0f0 msgr2=0x7fe52c18c240 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:03:24.195 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:24.193 7fe533c8d700 1 -- 172.21.2.113:0/2160715294 shutdown_connections 2021-02-10T11:03:24.195 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:24.193 7fe533c8d700 1 -- 172.21.2.113:0/2160715294 wait complete. 2021-02-10T11:03:24.201 DEBUG:teuthology.orchestra.run.gibba013:> sudo ceph --cluster ceph osd new 153700af-747f-481f-bdf4-d6072d3a842f 2 2021-02-10T11:03:24.629 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:24.612 7fc65dc70700 1 Processor -- start 2021-02-10T11:03:24.630 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:24.612 7fc65dc70700 1 -- start start 2021-02-10T11:03:24.630 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:24.612 7fc65dc70700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fc6580f0630 0x7fc6580f10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:24.630 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:24.612 7fc65dc70700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6580f0b20 con 0x7fc6580d17f0 2021-02-10T11:03:24.630 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:24.612 7fc65dc70700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6580d1bf0 con 0x7fc6580cff00 2021-02-10T11:03:24.630 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:24.612 7fc65dc70700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6580e5eb0 con 0x7fc6580f0630 2021-02-10T11:03:24.630 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:24.612 7fc657fff700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fc6580f0630 0x7fc6580f10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:24.630 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:24.612 7fc657fff700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fc6580f0630 0x7fc6580f10a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.113:3300/0 says I am v2:172.21.2.113:60338/0 (socket says 172.21.2.113:60338) 2021-02-10T11:03:24.630 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:24.612 7fc657fff700 1 -- 172.21.2.113:0/1415599938 learned_addr learned my addr 172.21.2.113:0/1415599938 (peer_addr_for_me v2:172.21.2.113:0/0) 2021-02-10T11:03:24.630 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:24.612 7fc656ffd700 1 --1- 172.21.2.113:0/1415599938 >> v1:172.21.2.113:6789/0 conn(0x7fc6580cff00 0x7fc6581b39d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.113:6789/0 says I am v1:172.21.2.113:47614/0 (socket says 172.21.2.113:47614) 2021-02-10T11:03:24.630 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:24.613 7fc657fff700 1 -- 172.21.2.113:0/1415599938 >> v1:172.21.2.113:6789/0 conn(0x7fc6580cff00 legacy=0x7fc6581b39d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:24.631 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:24.613 7fc657fff700 1 -- 172.21.2.113:0/1415599938 >> v1:172.21.2.102:6789/0 conn(0x7fc6580d17f0 legacy=0x7fc6581b31e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:24.631 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:24.613 7fc657fff700 1 -- 172.21.2.113:0/1415599938 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6580d1e10 con 0x7fc6580f0630 2021-02-10T11:03:24.631 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:24.613 7fc657fff700 1 --2- 172.21.2.113:0/1415599938 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc6580f0630 0x7fc6580f10a0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7fc650006f00 tx=0x7fc650009f10).ready entity=mon.1 client_cookie=d2a85f9b0d9b5c60 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:03:24.631 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:24.613 7fc6567fc700 1 -- 172.21.2.113:0/1415599938 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc65000b3b0 con 0x7fc6580f0630 2021-02-10T11:03:24.631 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:24.613 7fc6567fc700 1 -- 172.21.2.113:0/1415599938 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc65000b630 con 0x7fc6580f0630 2021-02-10T11:03:24.631 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:24.613 7fc6567fc700 1 -- 172.21.2.113:0/1415599938 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc65000bb50 con 0x7fc6580f0630 2021-02-10T11:03:24.631 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:24.613 7fc65dc70700 1 -- 172.21.2.113:0/1415599938 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc6580f0630 msgr2=0x7fc6580f10a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:24.631 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:24.613 7fc65dc70700 1 --2- 172.21.2.113:0/1415599938 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc6580f0630 0x7fc6580f10a0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7fc650006f00 tx=0x7fc650009f10).stop 2021-02-10T11:03:24.631 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:24.614 7fc65dc70700 1 -- 172.21.2.113:0/1415599938 shutdown_connections 2021-02-10T11:03:24.631 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:24.614 7fc65dc70700 1 --2- 172.21.2.113:0/1415599938 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc6580f0630 0x7fc6580f10a0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:24.631 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:24.614 7fc65dc70700 1 -- 172.21.2.113:0/1415599938 >> 172.21.2.113:0/1415599938 conn(0x7fc6580fde40 msgr2=0x7fc6580ce390 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:03:24.631 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:24.614 7fc65dc70700 1 -- 172.21.2.113:0/1415599938 shutdown_connections 2021-02-10T11:03:24.631 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:24.614 7fc65dc70700 1 -- 172.21.2.113:0/1415599938 wait complete. 2021-02-10T11:03:24.633 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:24.629 7fc65dc70700 1 Processor -- start 2021-02-10T11:03:24.633 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:24.629 7fc65dc70700 1 -- start start 2021-02-10T11:03:24.633 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:24.630 7fc65dc70700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc6580c3de0 0x7fc65808c3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:24.633 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:24.630 7fc65dc70700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc65808ca10 0x7fc658092830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:24.633 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:24.630 7fc65dc70700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc658092e50 0x7fc6580c29d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:24.634 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:24.630 7fc65dc70700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6580c3860 con 0x7fc65808ca10 2021-02-10T11:03:24.634 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:24.630 7fc65dc70700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6580c41e0 con 0x7fc6580c3de0 2021-02-10T11:03:24.634 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:24.630 7fc65dc70700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc658093250 con 0x7fc658092e50 2021-02-10T11:03:24.634 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:24.630 7fc657fff700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc658092e50 0x7fc6580c29d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:24.634 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:24.630 7fc6577fe700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc6580c3de0 0x7fc65808c3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:24.634 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:24.630 7fc657fff700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc658092e50 0x7fc6580c29d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.113:3301/0 says I am v2:172.21.2.113:60454/0 (socket says 172.21.2.113:60454) 2021-02-10T11:03:24.634 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:24.630 7fc657fff700 1 -- 172.21.2.113:0/3641500574 learned_addr learned my addr 172.21.2.113:0/3641500574 (peer_addr_for_me v2:172.21.2.113:0/0) 2021-02-10T11:03:24.634 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:24.630 7fc656ffd700 1 --2- 172.21.2.113:0/3641500574 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc65808ca10 0x7fc658092830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:24.634 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:24.630 7fc657fff700 1 -- 172.21.2.113:0/3641500574 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc6580c3de0 msgr2=0x7fc65808c3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:24.634 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:24.630 7fc657fff700 1 --2- 172.21.2.113:0/3641500574 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc6580c3de0 0x7fc65808c3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:24.634 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:24.630 7fc657fff700 1 -- 172.21.2.113:0/3641500574 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc65808ca10 msgr2=0x7fc658092830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:24.634 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:24.630 7fc657fff700 1 --2- 172.21.2.113:0/3641500574 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc65808ca10 0x7fc658092830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:24.635 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:24.630 7fc657fff700 1 -- 172.21.2.113:0/3641500574 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc658093470 con 0x7fc658092e50 2021-02-10T11:03:24.635 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:24.630 7fc6577fe700 1 --2- 172.21.2.113:0/3641500574 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc6580c3de0 0x7fc65808c3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:03:24.635 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:24.630 7fc656ffd700 1 --2- 172.21.2.113:0/3641500574 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc65808ca10 0x7fc658092830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:03:24.635 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:24.630 7fc657fff700 1 --2- 172.21.2.113:0/3641500574 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc658092e50 0x7fc6580c29d0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7fc650009390 tx=0x7fc65000bd40).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:03:24.635 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:24.631 7fc63ffff700 1 -- 172.21.2.113:0/3641500574 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc650003030 con 0x7fc658092e50 2021-02-10T11:03:24.635 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:24.631 7fc63ffff700 1 -- 172.21.2.113:0/3641500574 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc650006c80 con 0x7fc658092e50 2021-02-10T11:03:24.635 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:24.631 7fc63ffff700 1 -- 172.21.2.113:0/3641500574 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc650001480 con 0x7fc658092e50 2021-02-10T11:03:24.635 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:24.631 7fc65dc70700 1 -- 172.21.2.113:0/3641500574 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc658093470 con 0x7fc658092e50 2021-02-10T11:03:24.635 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:24.631 7fc65dc70700 1 -- 172.21.2.113:0/3641500574 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc658093470 con 0x7fc658092e50 2021-02-10T11:03:24.635 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:24.631 7fc63ffff700 1 -- 172.21.2.113:0/3641500574 <== mon.2 v2:172.21.2.113:3301/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fc650001270 con 0x7fc658092e50 2021-02-10T11:03:24.635 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:24.631 7fc63ffff700 1 --2- 172.21.2.113:0/3641500574 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc638018730 0x7fc63801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:24.635 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:24.631 7fc63ffff700 1 -- 172.21.2.113:0/3641500574 <== mon.2 v2:172.21.2.113:3301/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 960+0+0 (secure 0 0 0) 0x7fc650031910 con 0x7fc658092e50 2021-02-10T11:03:24.635 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:24.632 7fc6577fe700 1 --2- 172.21.2.113:0/3641500574 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc638018730 0x7fc63801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:24.636 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:24.632 7fc6577fe700 1 --2- 172.21.2.113:0/3641500574 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc638018730 0x7fc63801abc0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:03:24.663 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:24.662 7fc65dc70700 1 -- 172.21.2.113:0/3641500574 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc65810a010 con 0x7fc658092e50 2021-02-10T11:03:24.665 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:24.665 7fc63ffff700 1 -- 172.21.2.113:0/3641500574 <== mon.2 v2:172.21.2.113:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fc650001ef0 con 0x7fc658092e50 2021-02-10T11:03:24.800 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:24.799 7fc65dc70700 1 -- 172.21.2.113:0/3641500574 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_command({"prefix": "osd new", "uuid": "153700af-747f-481f-bdf4-d6072d3a842f", "id": 2} v 0) v1 -- 0x7fc65810a010 con 0x7fc658092e50 2021-02-10T11:03:24.805 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:24.804 7fc63ffff700 1 -- 172.21.2.113:0/3641500574 <== mon.2 v2:172.21.2.113:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "153700af-747f-481f-bdf4-d6072d3a842f", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7fc650006c80 con 0x7fc658092e50 2021-02-10T11:03:24.807 INFO:teuthology.orchestra.run.gibba013.stdout:2 2021-02-10T11:03:24.809 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:24.808 7fc65dc70700 1 -- 172.21.2.113:0/3641500574 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc638018730 msgr2=0x7fc63801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:24.809 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:24.808 7fc65dc70700 1 --2- 172.21.2.113:0/3641500574 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc638018730 0x7fc63801abc0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:24.809 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:24.808 7fc65dc70700 1 -- 172.21.2.113:0/3641500574 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc658092e50 msgr2=0x7fc6580c29d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:24.809 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:24.808 7fc65dc70700 1 --2- 172.21.2.113:0/3641500574 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc658092e50 0x7fc6580c29d0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7fc650009390 tx=0x7fc65000bd40).stop 2021-02-10T11:03:24.809 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:24.809 7fc65dc70700 1 -- 172.21.2.113:0/3641500574 shutdown_connections 2021-02-10T11:03:24.810 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:24.809 7fc65dc70700 1 --2- 172.21.2.113:0/3641500574 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc658092e50 0x7fc6580c29d0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:24.810 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:24.809 7fc65dc70700 1 --2- 172.21.2.113:0/3641500574 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc65808ca10 0x7fc658092830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:24.810 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:24.809 7fc65dc70700 1 --2- 172.21.2.113:0/3641500574 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc638018730 0x7fc63801abc0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:24.810 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:24.809 7fc65dc70700 1 --2- 172.21.2.113:0/3641500574 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc6580c3de0 0x7fc65808c3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:24.810 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:24.809 7fc65dc70700 1 -- 172.21.2.113:0/3641500574 >> 172.21.2.113:0/3641500574 conn(0x7fc65809ea20 msgr2=0x7fc65810a240 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:03:24.810 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:24.809 7fc65dc70700 1 -- 172.21.2.113:0/3641500574 shutdown_connections 2021-02-10T11:03:24.810 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:24.809 7fc65dc70700 1 -- 172.21.2.113:0/3641500574 wait complete. 2021-02-10T11:03:24.818 DEBUG:teuthology.orchestra.run.gibba013:> sudo ceph --cluster ceph osd new cbc678f4-83a9-45e6-8830-2f7c232b1ea7 3 2021-02-10T11:03:25.244 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:25.226 7f29452e7700 1 Processor -- start 2021-02-10T11:03:25.244 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:25.226 7f29452e7700 1 -- start start 2021-02-10T11:03:25.244 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:25.227 7f29452e7700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f29400cff00 0x7f29401b39d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:25.244 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:25.227 7f29452e7700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f29400d1bf0 0x7f29400ef620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:25.244 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:25.227 7f29452e7700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29400f0b20 con 0x7f29400d17f0 2021-02-10T11:03:25.245 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:25.227 7f29452e7700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29401b3ff0 con 0x7f29400d1bf0 2021-02-10T11:03:25.245 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:25.227 7f29452e7700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29400f20b0 con 0x7f29400cff00 2021-02-10T11:03:25.245 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:25.227 7f293f87e700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f29400d1bf0 0x7f29400ef620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:25.245 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:25.227 7f293e87c700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f29400cff00 0x7f29401b39d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:25.245 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:25.227 7f293f87e700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f29400d1bf0 0x7f29400ef620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.113:3300/0 says I am v2:172.21.2.113:60352/0 (socket says 172.21.2.113:60352) 2021-02-10T11:03:25.245 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:25.227 7f293f87e700 1 -- 172.21.2.113:0/731914619 learned_addr learned my addr 172.21.2.113:0/731914619 (peer_addr_for_me v2:172.21.2.113:0/0) 2021-02-10T11:03:25.245 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:25.227 7f293e87c700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f29400cff00 0x7f29401b39d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.113:3301/0 says I am v2:172.21.2.113:60462/0 (socket says 172.21.2.113:60462) 2021-02-10T11:03:25.245 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:25.228 7f293f87e700 1 -- 172.21.2.113:0/731914619 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f29400cff00 msgr2=0x7f29401b39d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:03:25.245 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:25.228 7f293f87e700 1 --2- 172.21.2.113:0/731914619 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f29400cff00 0x7f29401b39d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:25.245 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:25.228 7f293f87e700 1 -- 172.21.2.113:0/731914619 >> v1:172.21.2.102:6789/0 conn(0x7f29400d17f0 legacy=0x7f29401b31e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:03:25.245 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:25.228 7f293f87e700 1 -- 172.21.2.113:0/731914619 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29400f22a0 con 0x7f29400d1bf0 2021-02-10T11:03:25.245 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:25.228 7f293f87e700 1 --2- 172.21.2.113:0/731914619 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f29400d1bf0 0x7f29400ef620 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f2938006f00 tx=0x7f2938009f10).ready entity=mon.1 client_cookie=a27977e41f7907c1 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:03:25.246 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:25.228 7f293e07b700 1 -- 172.21.2.113:0/731914619 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f293800b3b0 con 0x7f29400d1bf0 2021-02-10T11:03:25.246 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:25.228 7f293e07b700 1 -- 172.21.2.113:0/731914619 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f293800b630 con 0x7f29400d1bf0 2021-02-10T11:03:25.246 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:25.228 7f293e07b700 1 -- 172.21.2.113:0/731914619 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f293800bb50 con 0x7f29400d1bf0 2021-02-10T11:03:25.246 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:25.228 7f29452e7700 1 -- 172.21.2.113:0/731914619 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f29400d1bf0 msgr2=0x7f29400ef620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:25.246 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:25.228 7f29452e7700 1 --2- 172.21.2.113:0/731914619 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f29400d1bf0 0x7f29400ef620 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f2938006f00 tx=0x7f2938009f10).stop 2021-02-10T11:03:25.246 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:25.229 7f29452e7700 1 -- 172.21.2.113:0/731914619 shutdown_connections 2021-02-10T11:03:25.246 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:25.229 7f29452e7700 1 --2- 172.21.2.113:0/731914619 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f29400d1bf0 0x7f29400ef620 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:25.246 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:25.229 7f29452e7700 1 --2- 172.21.2.113:0/731914619 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f29400cff00 0x7f29401b39d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:25.246 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:25.229 7f29452e7700 1 -- 172.21.2.113:0/731914619 >> 172.21.2.113:0/731914619 conn(0x7f29400fde40 msgr2=0x7f29400ce390 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:03:25.246 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:25.229 7f29452e7700 1 -- 172.21.2.113:0/731914619 shutdown_connections 2021-02-10T11:03:25.246 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:25.229 7f29452e7700 1 -- 172.21.2.113:0/731914619 wait complete. 2021-02-10T11:03:25.248 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:25.244 7f29452e7700 1 Processor -- start 2021-02-10T11:03:25.248 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:25.244 7f29452e7700 1 -- start start 2021-02-10T11:03:25.248 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:25.244 7f29452e7700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f29401a49b0 0x7f294008a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:25.248 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:25.244 7f29452e7700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f294009e080 0x7f29400c3100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:25.249 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:25.244 7f29452e7700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f294009e480 0x7f29400c3720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:25.249 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:25.244 7f29452e7700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29400c2b80 con 0x7f294009e480 2021-02-10T11:03:25.249 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:25.244 7f29452e7700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29400bee50 con 0x7f294009e080 2021-02-10T11:03:25.249 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:25.244 7f29452e7700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29400c13f0 con 0x7f29401a49b0 2021-02-10T11:03:25.249 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:25.244 7f293e87c700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f294009e080 0x7f29400c3100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:25.249 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:25.244 7f293e87c700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f294009e080 0x7f29400c3100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.113:3300/0 says I am v2:172.21.2.113:60354/0 (socket says 172.21.2.113:60354) 2021-02-10T11:03:25.249 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:25.244 7f293e87c700 1 -- 172.21.2.113:0/4289683082 learned_addr learned my addr 172.21.2.113:0/4289683082 (peer_addr_for_me v2:172.21.2.113:0/0) 2021-02-10T11:03:25.249 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:25.244 7f293f07d700 1 --2- 172.21.2.113:0/4289683082 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f29401a49b0 0x7f294008a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:25.249 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:25.245 7f293e87c700 1 -- 172.21.2.113:0/4289683082 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f29401a49b0 msgr2=0x7f294008a400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:25.249 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:25.245 7f293e87c700 1 --2- 172.21.2.113:0/4289683082 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f29401a49b0 0x7f294008a400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:25.249 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:25.245 7f293e87c700 1 -- 172.21.2.113:0/4289683082 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f294009e480 msgr2=0x7f29400c3720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:25.249 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:25.245 7f293e87c700 1 --2- 172.21.2.113:0/4289683082 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f294009e480 0x7f29400c3720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:03:25.250 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:25.245 7f293e87c700 1 -- 172.21.2.113:0/4289683082 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29400c1610 con 0x7f294009e080 2021-02-10T11:03:25.250 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:25.245 7f293e87c700 1 --2- 172.21.2.113:0/4289683082 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f294009e080 0x7f29400c3100 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f293000b560 tx=0x7f293000b900).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:03:25.250 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:25.245 7f2937fff700 1 -- 172.21.2.113:0/4289683082 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f293000bcf0 con 0x7f294009e080 2021-02-10T11:03:25.250 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:25.245 7f2937fff700 1 -- 172.21.2.113:0/4289683082 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2930000c90 con 0x7f294009e080 2021-02-10T11:03:25.250 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:25.245 7f2937fff700 1 -- 172.21.2.113:0/4289683082 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f29300022b0 con 0x7f294009e080 2021-02-10T11:03:25.250 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:25.245 7f29452e7700 1 -- 172.21.2.113:0/4289683082 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29400c1610 con 0x7f294009e080 2021-02-10T11:03:25.250 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:25.245 7f29452e7700 1 -- 172.21.2.113:0/4289683082 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29400c1610 con 0x7f294009e080 2021-02-10T11:03:25.250 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:25.246 7f2937fff700 1 -- 172.21.2.113:0/4289683082 <== mon.1 v2:172.21.2.113:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f2930006b00 con 0x7f294009e080 2021-02-10T11:03:25.250 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:25.246 7f2937fff700 1 --2- 172.21.2.113:0/4289683082 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f29240186d0 0x7f292401abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:25.250 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:25.246 7f2937fff700 1 -- 172.21.2.113:0/4289683082 <== mon.1 v2:172.21.2.113:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1063+0+0 (secure 0 0 0) 0x7f2930031600 con 0x7f294009e080 2021-02-10T11:03:25.250 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:25.246 7f293f07d700 1 --2- 172.21.2.113:0/4289683082 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f29240186d0 0x7f292401abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:25.250 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:25.247 7f293f07d700 1 --2- 172.21.2.113:0/4289683082 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f29240186d0 0x7f292401abd0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:03:25.278 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:25.277 7f29452e7700 1 -- 172.21.2.113:0/4289683082 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2940103690 con 0x7f294009e080 2021-02-10T11:03:25.281 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:25.280 7f2937fff700 1 -- 172.21.2.113:0/4289683082 <== mon.1 v2:172.21.2.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f2930006340 con 0x7f294009e080 2021-02-10T11:03:25.416 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:25.415 7f29452e7700 1 -- 172.21.2.113:0/4289683082 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_command({"prefix": "osd new", "uuid": "cbc678f4-83a9-45e6-8830-2f7c232b1ea7", "id": 3} v 0) v1 -- 0x7f29400f0b20 con 0x7f294009e080 2021-02-10T11:03:25.421 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:25.420 7f2937fff700 1 -- 172.21.2.113:0/4289683082 <== mon.1 v2:172.21.2.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "cbc678f4-83a9-45e6-8830-2f7c232b1ea7", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7f2930000c90 con 0x7f294009e080 2021-02-10T11:03:25.423 INFO:teuthology.orchestra.run.gibba013.stdout:3 2021-02-10T11:03:25.425 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:25.424 7f29452e7700 1 -- 172.21.2.113:0/4289683082 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f29240186d0 msgr2=0x7f292401abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:25.425 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:25.424 7f29452e7700 1 --2- 172.21.2.113:0/4289683082 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f29240186d0 0x7f292401abd0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:25.425 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:25.424 7f29452e7700 1 -- 172.21.2.113:0/4289683082 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f294009e080 msgr2=0x7f29400c3100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:25.425 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:25.424 7f29452e7700 1 --2- 172.21.2.113:0/4289683082 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f294009e080 0x7f29400c3100 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f293000b560 tx=0x7f293000b900).stop 2021-02-10T11:03:25.425 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:25.424 7f29452e7700 1 -- 172.21.2.113:0/4289683082 shutdown_connections 2021-02-10T11:03:25.425 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:25.424 7f29452e7700 1 --2- 172.21.2.113:0/4289683082 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f294009e480 0x7f29400c3720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:03:25.426 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:25.424 7f29452e7700 1 --2- 172.21.2.113:0/4289683082 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f29240186d0 0x7f292401abd0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:25.426 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:25.424 7f29452e7700 1 --2- 172.21.2.113:0/4289683082 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f294009e080 0x7f29400c3100 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:25.426 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:25.424 7f29452e7700 1 --2- 172.21.2.113:0/4289683082 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f29401a49b0 0x7f294008a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:25.426 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:25.424 7f29452e7700 1 -- 172.21.2.113:0/4289683082 >> 172.21.2.113:0/4289683082 conn(0x7f29400a1690 msgr2=0x7f29400a94c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:03:25.426 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:25.425 7f29452e7700 1 -- 172.21.2.113:0/4289683082 shutdown_connections 2021-02-10T11:03:25.426 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:25.425 7f29452e7700 1 -- 172.21.2.113:0/4289683082 wait complete. 2021-02-10T11:03:25.433 DEBUG:teuthology.orchestra.run.gibba013:> sudo ceph --cluster ceph osd new c01e4008-b258-4f10-afd2-9e7f37214351 4 2021-02-10T11:03:25.859 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:25.842 7f79b7653700 1 Processor -- start 2021-02-10T11:03:25.859 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:25.842 7f79b7653700 1 -- start start 2021-02-10T11:03:25.860 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:25.843 7f79b7653700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f79b00f0630 0x7f79b00f10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:25.860 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:25.843 7f79b7653700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79b00f0b20 con 0x7f79b00d17f0 2021-02-10T11:03:25.860 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:25.843 7f79b7653700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79b00d1bf0 con 0x7f79b00cff00 2021-02-10T11:03:25.860 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:25.843 7f79b7653700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79b00e5eb0 con 0x7f79b00f0630 2021-02-10T11:03:25.860 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:25.843 7f79b5c70700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f79b00f0630 0x7f79b00f10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:25.860 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:25.843 7f79b5c70700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f79b00f0630 0x7f79b00f10a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.113:3300/0 says I am v2:172.21.2.113:60362/0 (socket says 172.21.2.113:60362) 2021-02-10T11:03:25.860 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:25.843 7f79b5c70700 1 -- 172.21.2.113:0/2510573275 learned_addr learned my addr 172.21.2.113:0/2510573275 (peer_addr_for_me v2:172.21.2.113:0/0) 2021-02-10T11:03:25.860 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:25.843 7f79b5c70700 1 -- 172.21.2.113:0/2510573275 >> v1:172.21.2.113:6789/0 conn(0x7f79b00cff00 legacy=0x7f79b01b39d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:25.860 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:25.843 7f79b5c70700 1 -- 172.21.2.113:0/2510573275 >> v1:172.21.2.102:6789/0 conn(0x7f79b00d17f0 legacy=0x7f79b01b31e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:03:25.860 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:25.843 7f79b5c70700 1 -- 172.21.2.113:0/2510573275 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79b00d1e10 con 0x7f79b00f0630 2021-02-10T11:03:25.860 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:25.844 7f79b5c70700 1 --2- 172.21.2.113:0/2510573275 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f79b00f0630 0x7f79b00f10a0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f79a8006f00 tx=0x7f79a8009f10).ready entity=mon.1 client_cookie=21d207104053e09d server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:03:25.860 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:25.844 7f79a3fff700 1 -- 172.21.2.113:0/2510573275 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f79a800b3b0 con 0x7f79b00f0630 2021-02-10T11:03:25.861 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:25.844 7f79a3fff700 1 -- 172.21.2.113:0/2510573275 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79a800b730 con 0x7f79b00f0630 2021-02-10T11:03:25.861 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:25.844 7f79a3fff700 1 -- 172.21.2.113:0/2510573275 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f79a800bc50 con 0x7f79b00f0630 2021-02-10T11:03:25.861 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:25.844 7f79b7653700 1 -- 172.21.2.113:0/2510573275 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f79b00f0630 msgr2=0x7f79b00f10a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:25.861 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:25.844 7f79b7653700 1 --2- 172.21.2.113:0/2510573275 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f79b00f0630 0x7f79b00f10a0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f79a8006f00 tx=0x7f79a8009f10).stop 2021-02-10T11:03:25.861 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:25.844 7f79b7653700 1 -- 172.21.2.113:0/2510573275 shutdown_connections 2021-02-10T11:03:25.861 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:25.844 7f79b7653700 1 --2- 172.21.2.113:0/2510573275 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f79b00f0630 0x7f79b00f10a0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:25.861 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:25.844 7f79b7653700 1 -- 172.21.2.113:0/2510573275 >> 172.21.2.113:0/2510573275 conn(0x7f79b00fde40 msgr2=0x7f79b00ce390 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:03:25.861 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:25.845 7f79b7653700 1 -- 172.21.2.113:0/2510573275 shutdown_connections 2021-02-10T11:03:25.861 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:25.845 7f79b7653700 1 -- 172.21.2.113:0/2510573275 wait complete. 2021-02-10T11:03:25.862 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:25.859 7f79b7653700 1 Processor -- start 2021-02-10T11:03:25.863 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:25.859 7f79b7653700 1 -- start start 2021-02-10T11:03:25.863 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:25.860 7f79b7653700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f79b0091360 0x7f79b00bfb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:25.863 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:25.860 7f79b7653700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f79b00c0140 0x7f79b00c0540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:25.863 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:25.860 7f79b7653700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f79b01b1190 0x7f79b01b1590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:25.863 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:25.860 7f79b7653700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79b01b2860 con 0x7f79b01b1190 2021-02-10T11:03:25.863 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:25.860 7f79b7653700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79b00c0b60 con 0x7f79b0091360 2021-02-10T11:03:25.863 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:25.860 7f79b7653700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79b00c0d40 con 0x7f79b00c0140 2021-02-10T11:03:25.863 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:25.860 7f79b546f700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f79b0091360 0x7f79b00bfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:25.863 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:25.860 7f79b546f700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f79b0091360 0x7f79b00bfb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.113:3300/0 says I am v2:172.21.2.113:60366/0 (socket says 172.21.2.113:60366) 2021-02-10T11:03:25.863 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:25.860 7f79b546f700 1 -- 172.21.2.113:0/1467224741 learned_addr learned my addr 172.21.2.113:0/1467224741 (peer_addr_for_me v2:172.21.2.113:0/0) 2021-02-10T11:03:25.864 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:25.860 7f79b4c6e700 1 --2- 172.21.2.113:0/1467224741 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f79b00c0140 0x7f79b00c0540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:25.864 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:25.860 7f79b546f700 1 -- 172.21.2.113:0/1467224741 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f79b00c0140 msgr2=0x7f79b00c0540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:25.864 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:25.860 7f79b546f700 1 --2- 172.21.2.113:0/1467224741 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f79b00c0140 0x7f79b00c0540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:25.864 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:25.860 7f79b546f700 1 -- 172.21.2.113:0/1467224741 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f79b01b1190 msgr2=0x7f79b01b1590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:25.864 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:25.860 7f79b546f700 1 --2- 172.21.2.113:0/1467224741 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f79b01b1190 0x7f79b01b1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:03:25.864 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:25.860 7f79b546f700 1 -- 172.21.2.113:0/1467224741 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79b01b1bb0 con 0x7f79b0091360 2021-02-10T11:03:25.864 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:25.860 7f79b546f700 1 --2- 172.21.2.113:0/1467224741 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f79b0091360 0x7f79b00bfb20 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7f79a400b5b0 tx=0x7f79a400b950).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:03:25.864 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:25.861 7f79a1ffb700 1 -- 172.21.2.113:0/1467224741 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f79a400bd40 con 0x7f79b0091360 2021-02-10T11:03:25.864 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:25.861 7f79a1ffb700 1 -- 172.21.2.113:0/1467224741 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79a4006020 con 0x7f79b0091360 2021-02-10T11:03:25.864 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:25.861 7f79b7653700 1 -- 172.21.2.113:0/1467224741 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79b01b1bb0 con 0x7f79b0091360 2021-02-10T11:03:25.864 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:25.861 7f79b7653700 1 -- 172.21.2.113:0/1467224741 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79b01b1bb0 con 0x7f79b0091360 2021-02-10T11:03:25.864 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:25.861 7f79a1ffb700 1 -- 172.21.2.113:0/1467224741 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f79a40054a0 con 0x7f79b0091360 2021-02-10T11:03:25.864 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:25.861 7f79a1ffb700 1 -- 172.21.2.113:0/1467224741 <== mon.1 v2:172.21.2.113:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f79a4006200 con 0x7f79b0091360 2021-02-10T11:03:25.865 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:25.862 7f79a1ffb700 1 --2- 172.21.2.113:0/1467224741 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f79900186c0 0x7f799001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:25.865 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:25.862 7f79a1ffb700 1 -- 172.21.2.113:0/1467224741 <== mon.1 v2:172.21.2.113:3300/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1166+0+0 (secure 0 0 0) 0x7f79a40316a0 con 0x7f79b0091360 2021-02-10T11:03:25.865 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:25.862 7f79b4c6e700 1 --2- 172.21.2.113:0/1467224741 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f79900186c0 0x7f799001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:25.865 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:25.863 7f79b4c6e700 1 --2- 172.21.2.113:0/1467224741 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f79900186c0 0x7f799001abc0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:03:25.894 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:25.893 7f79b7653700 1 -- 172.21.2.113:0/1467224741 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79b010a010 con 0x7f79b0091360 2021-02-10T11:03:25.896 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:25.895 7f79a1ffb700 1 -- 172.21.2.113:0/1467224741 <== mon.1 v2:172.21.2.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f79a4002020 con 0x7f79b0091360 2021-02-10T11:03:26.032 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:26.032 7f79b7653700 1 -- 172.21.2.113:0/1467224741 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_command({"prefix": "osd new", "uuid": "c01e4008-b258-4f10-afd2-9e7f37214351", "id": 4} v 0) v1 -- 0x7f79b010a010 con 0x7f79b0091360 2021-02-10T11:03:26.037 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:26.036 7f79a1ffb700 1 -- 172.21.2.113:0/1467224741 <== mon.1 v2:172.21.2.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "c01e4008-b258-4f10-afd2-9e7f37214351", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7f79a400bd40 con 0x7f79b0091360 2021-02-10T11:03:26.039 INFO:teuthology.orchestra.run.gibba013.stdout:4 2021-02-10T11:03:26.041 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:26.040 7f79b7653700 1 -- 172.21.2.113:0/1467224741 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f79900186c0 msgr2=0x7f799001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:26.041 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:26.040 7f79b7653700 1 --2- 172.21.2.113:0/1467224741 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f79900186c0 0x7f799001abc0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:26.041 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:26.040 7f79b7653700 1 -- 172.21.2.113:0/1467224741 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f79b0091360 msgr2=0x7f79b00bfb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:26.041 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:26.040 7f79b7653700 1 --2- 172.21.2.113:0/1467224741 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f79b0091360 0x7f79b00bfb20 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7f79a400b5b0 tx=0x7f79a400b950).stop 2021-02-10T11:03:26.041 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:26.040 7f79b7653700 1 -- 172.21.2.113:0/1467224741 shutdown_connections 2021-02-10T11:03:26.041 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:26.040 7f79b7653700 1 --2- 172.21.2.113:0/1467224741 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f79b01b1190 0x7f79b01b1590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:03:26.041 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:26.040 7f79b7653700 1 --2- 172.21.2.113:0/1467224741 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f79b00c0140 0x7f79b00c0540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:26.042 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:26.040 7f79b7653700 1 --2- 172.21.2.113:0/1467224741 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f79900186c0 0x7f799001abc0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:26.042 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:26.040 7f79b7653700 1 --2- 172.21.2.113:0/1467224741 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f79b0091360 0x7f79b00bfb20 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:26.042 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:26.040 7f79b7653700 1 -- 172.21.2.113:0/1467224741 >> 172.21.2.113:0/1467224741 conn(0x7f79b00ae1b0 msgr2=0x7f79b010d9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:03:26.042 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:26.041 7f79b7653700 1 -- 172.21.2.113:0/1467224741 shutdown_connections 2021-02-10T11:03:26.042 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:26.041 7f79b7653700 1 -- 172.21.2.113:0/1467224741 wait complete. 2021-02-10T11:03:26.049 DEBUG:teuthology.orchestra.run.gibba013:> sudo ceph --cluster ceph osd new 551f6007-daf8-4a98-8ee0-c2b16bcccb5e 5 2021-02-10T11:03:26.475 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:26.458 7f37f2dc4700 1 Processor -- start 2021-02-10T11:03:26.475 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:26.458 7f37f2dc4700 1 -- start start 2021-02-10T11:03:26.475 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:26.458 7f37f2dc4700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f37ec0cff00 0x7f37ec1b39d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:26.475 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:26.458 7f37f2dc4700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f37ec0d1bf0 0x7f37ec0ef620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:26.476 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:26.458 7f37f2dc4700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37ec0f0b20 con 0x7f37ec0d17f0 2021-02-10T11:03:26.476 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:26.458 7f37f2dc4700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37ec0e5eb0 con 0x7f37ec0d1bf0 2021-02-10T11:03:26.476 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:26.458 7f37f2dc4700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37ec1b4020 con 0x7f37ec0cff00 2021-02-10T11:03:26.476 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:26.458 7f37ebfff700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f37ec0cff00 0x7f37ec1b39d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:26.476 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:26.458 7f37ebfff700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f37ec0cff00 0x7f37ec1b39d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.113:3300/0 says I am v2:172.21.2.113:60374/0 (socket says 172.21.2.113:60374) 2021-02-10T11:03:26.476 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:26.458 7f37ebfff700 1 -- 172.21.2.113:0/2718091932 learned_addr learned my addr 172.21.2.113:0/2718091932 (peer_addr_for_me v2:172.21.2.113:0/0) 2021-02-10T11:03:26.476 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:26.459 7f37f13e1700 1 --2- 172.21.2.113:0/2718091932 >> v2:172.21.2.102:3300/0 conn(0x7f37ec0d1bf0 0x7f37ec0ef620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:26.476 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:26.459 7f37ebfff700 1 -- 172.21.2.113:0/2718091932 >> v2:172.21.2.102:3300/0 conn(0x7f37ec0d1bf0 msgr2=0x7f37ec0ef620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:03:26.476 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:26.459 7f37ebfff700 1 --2- 172.21.2.113:0/2718091932 >> v2:172.21.2.102:3300/0 conn(0x7f37ec0d1bf0 0x7f37ec0ef620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:26.476 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:26.459 7f37ebfff700 1 -- 172.21.2.113:0/2718091932 >> v1:172.21.2.102:6789/0 conn(0x7f37ec0d17f0 legacy=0x7f37ec1b31e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:26.476 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:26.459 7f37ebfff700 1 -- 172.21.2.113:0/2718091932 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37ec0f20b0 con 0x7f37ec0cff00 2021-02-10T11:03:26.476 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:26.459 7f37ebfff700 1 --2- 172.21.2.113:0/2718091932 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f37ec0cff00 0x7f37ec1b39d0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7f37dc006a80 tx=0x7f37dc009f00).ready entity=mon.1 client_cookie=49e30e090bd1fbfd server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:03:26.477 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:26.459 7f37eb7fe700 1 -- 172.21.2.113:0/2718091932 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f37dc00b5c0 con 0x7f37ec0cff00 2021-02-10T11:03:26.477 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:26.459 7f37eb7fe700 1 -- 172.21.2.113:0/2718091932 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37dc00b8c0 con 0x7f37ec0cff00 2021-02-10T11:03:26.477 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:26.459 7f37eb7fe700 1 -- 172.21.2.113:0/2718091932 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f37dc00bd20 con 0x7f37ec0cff00 2021-02-10T11:03:26.477 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:26.459 7f37f2dc4700 1 -- 172.21.2.113:0/2718091932 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f37ec0cff00 msgr2=0x7f37ec1b39d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:26.477 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:26.459 7f37f2dc4700 1 --2- 172.21.2.113:0/2718091932 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f37ec0cff00 0x7f37ec1b39d0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7f37dc006a80 tx=0x7f37dc009f00).stop 2021-02-10T11:03:26.477 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:26.460 7f37f2dc4700 1 -- 172.21.2.113:0/2718091932 shutdown_connections 2021-02-10T11:03:26.477 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:26.460 7f37f2dc4700 1 --2- 172.21.2.113:0/2718091932 >> v2:172.21.2.102:3300/0 conn(0x7f37ec0d1bf0 0x7f37ec0ef620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:26.477 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:26.460 7f37f2dc4700 1 --2- 172.21.2.113:0/2718091932 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f37ec0cff00 0x7f37ec1b39d0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:26.477 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:26.460 7f37f2dc4700 1 -- 172.21.2.113:0/2718091932 >> 172.21.2.113:0/2718091932 conn(0x7f37ec0fde40 msgr2=0x7f37ec0ce390 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:03:26.477 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:26.460 7f37f2dc4700 1 -- 172.21.2.113:0/2718091932 shutdown_connections 2021-02-10T11:03:26.477 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:26.460 7f37f2dc4700 1 -- 172.21.2.113:0/2718091932 wait complete. 2021-02-10T11:03:26.479 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:26.475 7f37f2dc4700 1 Processor -- start 2021-02-10T11:03:26.479 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:26.475 7f37f2dc4700 1 -- start start 2021-02-10T11:03:26.479 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:26.475 7f37f2dc4700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f37ec099ec0 0x7f37ec0ad480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:26.480 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:26.475 7f37f2dc4700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f37ec0adaa0 0x7f37ec106a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:26.480 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:26.475 7f37f2dc4700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f37ec095e80 0x7f37ec107060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:26.480 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:26.475 7f37f2dc4700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37ec0964f0 con 0x7f37ec099ec0 2021-02-10T11:03:26.480 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:26.475 7f37f2dc4700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37ec1076c0 con 0x7f37ec0adaa0 2021-02-10T11:03:26.480 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:26.475 7f37f2dc4700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37ec1078b0 con 0x7f37ec095e80 2021-02-10T11:03:26.480 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:26.476 7f37f0be0700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f37ec099ec0 0x7f37ec0ad480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:26.480 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:26.476 7f37ebfff700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f37ec0adaa0 0x7f37ec106a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:26.480 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:26.476 7f37f13e1700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f37ec095e80 0x7f37ec107060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:26.480 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:26.476 7f37f13e1700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f37ec095e80 0x7f37ec107060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.113:3301/0 says I am v2:172.21.2.113:60492/0 (socket says 172.21.2.113:60492) 2021-02-10T11:03:26.480 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:26.476 7f37ebfff700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f37ec0adaa0 0x7f37ec106a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.113:3300/0 says I am v2:172.21.2.113:60380/0 (socket says 172.21.2.113:60380) 2021-02-10T11:03:26.480 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:26.476 7f37f13e1700 1 -- 172.21.2.113:0/1123927876 learned_addr learned my addr 172.21.2.113:0/1123927876 (peer_addr_for_me v2:172.21.2.113:0/0) 2021-02-10T11:03:26.480 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:26.476 7f37ebfff700 1 -- 172.21.2.113:0/1123927876 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f37ec095e80 msgr2=0x7f37ec107060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:26.481 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:26.476 7f37ebfff700 1 --2- 172.21.2.113:0/1123927876 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f37ec095e80 0x7f37ec107060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:26.481 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:26.476 7f37ebfff700 1 -- 172.21.2.113:0/1123927876 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f37ec099ec0 msgr2=0x7f37ec0ad480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:26.481 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:26.476 7f37ebfff700 1 --2- 172.21.2.113:0/1123927876 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f37ec099ec0 0x7f37ec0ad480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:26.481 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:26.476 7f37ebfff700 1 -- 172.21.2.113:0/1123927876 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37ec0969f0 con 0x7f37ec0adaa0 2021-02-10T11:03:26.481 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:26.476 7f37f13e1700 1 --2- 172.21.2.113:0/1123927876 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f37ec095e80 0x7f37ec107060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:03:26.481 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:26.476 7f37f0be0700 1 --2- 172.21.2.113:0/1123927876 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f37ec099ec0 0x7f37ec0ad480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:03:26.481 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:26.476 7f37ebfff700 1 --2- 172.21.2.113:0/1123927876 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f37ec0adaa0 0x7f37ec106a40 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7f37dc00b8a0 tx=0x7f37dc00bc40).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:03:26.481 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:26.476 7f37e97fa700 1 -- 172.21.2.113:0/1123927876 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f37dc00c2f0 con 0x7f37ec0adaa0 2021-02-10T11:03:26.481 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:26.477 7f37e97fa700 1 -- 172.21.2.113:0/1123927876 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37dc00c670 con 0x7f37ec0adaa0 2021-02-10T11:03:26.481 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:26.477 7f37e97fa700 1 -- 172.21.2.113:0/1123927876 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f37dc00cb50 con 0x7f37ec0adaa0 2021-02-10T11:03:26.481 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:26.477 7f37f2dc4700 1 -- 172.21.2.113:0/1123927876 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37ec0969f0 con 0x7f37ec0adaa0 2021-02-10T11:03:26.481 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:26.477 7f37f2dc4700 1 -- 172.21.2.113:0/1123927876 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37ec096f80 con 0x7f37ec0adaa0 2021-02-10T11:03:26.481 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:26.477 7f37e97fa700 1 -- 172.21.2.113:0/1123927876 <== mon.1 v2:172.21.2.113:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f37dc001010 con 0x7f37ec0adaa0 2021-02-10T11:03:26.482 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:26.477 7f37e97fa700 1 --2- 172.21.2.113:0/1123927876 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f37d00186c0 0x7f37d001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:26.482 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:26.477 7f37e97fa700 1 -- 172.21.2.113:0/1123927876 <== mon.1 v2:172.21.2.113:3300/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1269+0+0 (secure 0 0 0) 0x7f37dc031690 con 0x7f37ec0adaa0 2021-02-10T11:03:26.482 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:26.478 7f37f0be0700 1 --2- 172.21.2.113:0/1123927876 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f37d00186c0 0x7f37d001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:26.482 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:26.478 7f37f0be0700 1 --2- 172.21.2.113:0/1123927876 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f37d00186c0 0x7f37d001abc0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:03:26.509 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:26.508 7f37f2dc4700 1 -- 172.21.2.113:0/1123927876 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37ec103690 con 0x7f37ec0adaa0 2021-02-10T11:03:26.511 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:26.510 7f37e97fa700 1 -- 172.21.2.113:0/1123927876 <== mon.1 v2:172.21.2.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f37dc000d30 con 0x7f37ec0adaa0 2021-02-10T11:03:26.649 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:26.648 7f37f2dc4700 1 -- 172.21.2.113:0/1123927876 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_command({"prefix": "osd new", "uuid": "551f6007-daf8-4a98-8ee0-c2b16bcccb5e", "id": 5} v 0) v1 -- 0x7f37ec0f0b20 con 0x7f37ec0adaa0 2021-02-10T11:03:26.653 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:26.652 7f37e97fa700 1 -- 172.21.2.113:0/1123927876 <== mon.1 v2:172.21.2.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "551f6007-daf8-4a98-8ee0-c2b16bcccb5e", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7f37dc001010 con 0x7f37ec0adaa0 2021-02-10T11:03:26.656 INFO:teuthology.orchestra.run.gibba013.stdout:5 2021-02-10T11:03:26.657 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:26.657 7f37f2dc4700 1 -- 172.21.2.113:0/1123927876 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f37d00186c0 msgr2=0x7f37d001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:26.657 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:26.657 7f37f2dc4700 1 --2- 172.21.2.113:0/1123927876 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f37d00186c0 0x7f37d001abc0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:26.658 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:26.657 7f37f2dc4700 1 -- 172.21.2.113:0/1123927876 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f37ec0adaa0 msgr2=0x7f37ec106a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:26.658 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:26.657 7f37f2dc4700 1 --2- 172.21.2.113:0/1123927876 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f37ec0adaa0 0x7f37ec106a40 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7f37dc00b8a0 tx=0x7f37dc00bc40).stop 2021-02-10T11:03:26.658 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:26.657 7f37f2dc4700 1 -- 172.21.2.113:0/1123927876 shutdown_connections 2021-02-10T11:03:26.658 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:26.657 7f37f2dc4700 1 --2- 172.21.2.113:0/1123927876 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f37ec095e80 0x7f37ec107060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:26.658 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:26.657 7f37f2dc4700 1 --2- 172.21.2.113:0/1123927876 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f37d00186c0 0x7f37d001abc0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:26.658 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:26.657 7f37f2dc4700 1 --2- 172.21.2.113:0/1123927876 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f37ec0adaa0 0x7f37ec106a40 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:26.659 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:26.657 7f37f2dc4700 1 --2- 172.21.2.113:0/1123927876 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f37ec099ec0 0x7f37ec0ad480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:26.659 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:26.657 7f37f2dc4700 1 -- 172.21.2.113:0/1123927876 >> 172.21.2.113:0/1123927876 conn(0x7f37ec1ba0f0 msgr2=0x7f37ec18c240 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:03:26.659 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:26.657 7f37f2dc4700 1 -- 172.21.2.113:0/1123927876 shutdown_connections 2021-02-10T11:03:26.659 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:26.657 7f37f2dc4700 1 -- 172.21.2.113:0/1123927876 wait complete. 2021-02-10T11:03:26.666 DEBUG:teuthology.orchestra.run.gibba013:> sudo ceph --cluster ceph osd new bf67c040-0e50-48be-b102-c3943bc9ebef 6 2021-02-10T11:03:27.091 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:27.074 7fa326d36700 1 Processor -- start 2021-02-10T11:03:27.139 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:27.074 7fa326d36700 1 -- start start 2021-02-10T11:03:27.139 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:27.074 7fa326d36700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa3200d17f0 0x7fa3201b31e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:27.139 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:27.074 7fa326d36700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fa3200cff00 0x7fa3200ef620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:27.140 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:27.074 7fa326d36700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3200f0b20 con 0x7fa3200d1bf0 2021-02-10T11:03:27.140 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:27.074 7fa326d36700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3200e5eb0 con 0x7fa3200cff00 2021-02-10T11:03:27.140 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:27.074 7fa326d36700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3201b3ff0 con 0x7fa3200d17f0 2021-02-10T11:03:27.140 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:27.074 7fa324b52700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa3200d17f0 0x7fa3201b31e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:27.140 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:27.074 7fa324b52700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa3200d17f0 0x7fa3201b31e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.113:3301/0 says I am v2:172.21.2.113:60498/0 (socket says 172.21.2.113:60498) 2021-02-10T11:03:27.141 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:27.074 7fa324b52700 1 -- 172.21.2.113:0/1543136353 learned_addr learned my addr 172.21.2.113:0/1543136353 (peer_addr_for_me v2:172.21.2.113:0/0) 2021-02-10T11:03:27.141 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:27.075 7fa324b52700 1 -- 172.21.2.113:0/1543136353 >> v2:172.21.2.102:3300/0 conn(0x7fa3200cff00 msgr2=0x7fa3200ef620 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:03:27.141 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:27.075 7fa324b52700 1 --2- 172.21.2.113:0/1543136353 >> v2:172.21.2.102:3300/0 conn(0x7fa3200cff00 0x7fa3200ef620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:03:27.141 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:27.075 7fa324b52700 1 -- 172.21.2.113:0/1543136353 >> v1:172.21.2.102:6789/0 conn(0x7fa3200d1bf0 legacy=0x7fa3201b3800 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:03:27.142 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:27.075 7fa324b52700 1 -- 172.21.2.113:0/1543136353 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3200f20b0 con 0x7fa3200d17f0 2021-02-10T11:03:27.142 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:27.075 7fa324b52700 1 --2- 172.21.2.113:0/1543136353 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa3200d17f0 0x7fa3201b31e0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7fa308006700 tx=0x7fa308009b80).ready entity=mon.2 client_cookie=8a280089e7d91f26 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:03:27.142 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:27.075 7fa31f7fe700 1 -- 172.21.2.113:0/1543136353 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa30800c0c0 con 0x7fa3200d17f0 2021-02-10T11:03:27.142 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:27.075 7fa31f7fe700 1 -- 172.21.2.113:0/1543136353 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa30800c440 con 0x7fa3200d17f0 2021-02-10T11:03:27.142 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:27.075 7fa31f7fe700 1 -- 172.21.2.113:0/1543136353 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa30800c8a0 con 0x7fa3200d17f0 2021-02-10T11:03:27.143 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:27.075 7fa326d36700 1 -- 172.21.2.113:0/1543136353 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa3200d17f0 msgr2=0x7fa3201b31e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:27.143 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:27.075 7fa326d36700 1 --2- 172.21.2.113:0/1543136353 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa3200d17f0 0x7fa3201b31e0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7fa308006700 tx=0x7fa308009b80).stop 2021-02-10T11:03:27.143 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:27.076 7fa326d36700 1 -- 172.21.2.113:0/1543136353 shutdown_connections 2021-02-10T11:03:27.143 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:27.076 7fa326d36700 1 --2- 172.21.2.113:0/1543136353 >> v2:172.21.2.102:3300/0 conn(0x7fa3200cff00 0x7fa3200ef620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:03:27.143 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:27.076 7fa326d36700 1 --2- 172.21.2.113:0/1543136353 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa3200d17f0 0x7fa3201b31e0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:27.144 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:27.076 7fa326d36700 1 -- 172.21.2.113:0/1543136353 >> 172.21.2.113:0/1543136353 conn(0x7fa3200fde40 msgr2=0x7fa3200ce390 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:03:27.144 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:27.076 7fa326d36700 1 -- 172.21.2.113:0/1543136353 shutdown_connections 2021-02-10T11:03:27.144 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:27.076 7fa326d36700 1 -- 172.21.2.113:0/1543136353 wait complete. 2021-02-10T11:03:27.144 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:27.090 7fa326d36700 1 Processor -- start 2021-02-10T11:03:27.144 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:27.090 7fa326d36700 1 -- start start 2021-02-10T11:03:27.147 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:27.091 7fa326d36700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa32009a370 0x7fa320096570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:27.147 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:27.091 7fa326d36700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa320096b90 0x7fa320097640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:27.147 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:27.091 7fa326d36700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa3201b4050 0x7fa320097c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:27.147 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:27.091 7fa326d36700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3200970c0 con 0x7fa32009a370 2021-02-10T11:03:27.148 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:27.091 7fa326d36700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa32009a770 con 0x7fa3201b4050 2021-02-10T11:03:27.148 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:27.091 7fa326d36700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3201b1bb0 con 0x7fa320096b90 2021-02-10T11:03:27.148 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:27.091 7fa31ffff700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa320096b90 0x7fa320097640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:27.148 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:27.091 7fa325353700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa3201b4050 0x7fa320097c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:27.148 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:27.091 7fa31ffff700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa320096b90 0x7fa320097640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.113:3301/0 says I am v2:172.21.2.113:60502/0 (socket says 172.21.2.113:60502) 2021-02-10T11:03:27.149 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:27.091 7fa31ffff700 1 -- 172.21.2.113:0/1477947397 learned_addr learned my addr 172.21.2.113:0/1477947397 (peer_addr_for_me v2:172.21.2.113:0/0) 2021-02-10T11:03:27.149 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:27.092 7fa31ffff700 1 -- 172.21.2.113:0/1477947397 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa3201b4050 msgr2=0x7fa320097c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:27.149 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:27.092 7fa31ffff700 1 --2- 172.21.2.113:0/1477947397 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa3201b4050 0x7fa320097c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:27.149 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:27.092 7fa31ffff700 1 -- 172.21.2.113:0/1477947397 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa32009a370 msgr2=0x7fa320096570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:27.149 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:27.092 7fa31ffff700 1 --2- 172.21.2.113:0/1477947397 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa32009a370 0x7fa320096570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:03:27.149 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:27.092 7fa31ffff700 1 -- 172.21.2.113:0/1477947397 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa320098280 con 0x7fa320096b90 2021-02-10T11:03:27.150 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:27.092 7fa31ffff700 1 --2- 172.21.2.113:0/1477947397 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa320096b90 0x7fa320097640 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7fa310009940 tx=0x7fa31000bcf0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:03:27.150 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:27.092 7fa31d7fa700 1 -- 172.21.2.113:0/1477947397 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa310000d30 con 0x7fa320096b90 2021-02-10T11:03:27.150 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:27.092 7fa31d7fa700 1 -- 172.21.2.113:0/1477947397 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa310000fe0 con 0x7fa320096b90 2021-02-10T11:03:27.150 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:27.092 7fa31d7fa700 1 -- 172.21.2.113:0/1477947397 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa31000c4e0 con 0x7fa320096b90 2021-02-10T11:03:27.152 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:27.092 7fa326d36700 1 -- 172.21.2.113:0/1477947397 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa320098280 con 0x7fa320096b90 2021-02-10T11:03:27.152 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:27.092 7fa326d36700 1 -- 172.21.2.113:0/1477947397 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa320098280 con 0x7fa320096b90 2021-02-10T11:03:27.152 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:27.093 7fa31d7fa700 1 -- 172.21.2.113:0/1477947397 <== mon.2 v2:172.21.2.113:3301/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa310001230 con 0x7fa320096b90 2021-02-10T11:03:27.152 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:27.093 7fa31d7fa700 1 --2- 172.21.2.113:0/1477947397 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa3040186c0 0x7fa30401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:27.152 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:27.093 7fa31d7fa700 1 -- 172.21.2.113:0/1477947397 <== mon.2 v2:172.21.2.113:3301/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1372+0+0 (secure 0 0 0) 0x7fa3100318a0 con 0x7fa320096b90 2021-02-10T11:03:27.152 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:27.093 7fa324b52700 1 --2- 172.21.2.113:0/1477947397 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa3040186c0 0x7fa30401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:27.152 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:27.094 7fa324b52700 1 --2- 172.21.2.113:0/1477947397 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa3040186c0 0x7fa30401abc0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:03:27.152 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:27.124 7fa326d36700 1 -- 172.21.2.113:0/1477947397 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3200f0b20 con 0x7fa320096b90 2021-02-10T11:03:27.152 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:27.126 7fa31d7fa700 1 -- 172.21.2.113:0/1477947397 <== mon.2 v2:172.21.2.113:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fa31000be30 con 0x7fa320096b90 2021-02-10T11:03:27.263 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:27.262 7fa326d36700 1 -- 172.21.2.113:0/1477947397 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_command({"prefix": "osd new", "uuid": "bf67c040-0e50-48be-b102-c3943bc9ebef", "id": 6} v 0) v1 -- 0x7fa3200f0b20 con 0x7fa320096b90 2021-02-10T11:03:27.268 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:27.268 7fa31d7fa700 1 -- 172.21.2.113:0/1477947397 <== mon.2 v2:172.21.2.113:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "bf67c040-0e50-48be-b102-c3943bc9ebef", "id": 6}]=0 v9) v1 ==== 112+0+2 (secure 0 0 0) 0x7fa31001f8a0 con 0x7fa320096b90 2021-02-10T11:03:27.270 INFO:teuthology.orchestra.run.gibba013.stdout:6 2021-02-10T11:03:27.272 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:27.271 7fa326d36700 1 -- 172.21.2.113:0/1477947397 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa3040186c0 msgr2=0x7fa30401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:27.272 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:27.271 7fa326d36700 1 --2- 172.21.2.113:0/1477947397 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa3040186c0 0x7fa30401abc0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:27.272 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:27.272 7fa326d36700 1 -- 172.21.2.113:0/1477947397 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa320096b90 msgr2=0x7fa320097640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:27.272 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:27.272 7fa326d36700 1 --2- 172.21.2.113:0/1477947397 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa320096b90 0x7fa320097640 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7fa310009940 tx=0x7fa31000bcf0).stop 2021-02-10T11:03:27.273 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:27.272 7fa326d36700 1 -- 172.21.2.113:0/1477947397 shutdown_connections 2021-02-10T11:03:27.273 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:27.272 7fa326d36700 1 --2- 172.21.2.113:0/1477947397 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa3040186c0 0x7fa30401abc0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:27.273 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:27.272 7fa326d36700 1 --2- 172.21.2.113:0/1477947397 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa3201b4050 0x7fa320097c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:27.273 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:27.272 7fa326d36700 1 --2- 172.21.2.113:0/1477947397 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa320096b90 0x7fa320097640 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:27.273 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:27.272 7fa326d36700 1 --2- 172.21.2.113:0/1477947397 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa32009a370 0x7fa320096570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:03:27.274 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:27.272 7fa326d36700 1 -- 172.21.2.113:0/1477947397 >> 172.21.2.113:0/1477947397 conn(0x7fa320105300 msgr2=0x7fa32018e5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:03:27.274 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:27.272 7fa326d36700 1 -- 172.21.2.113:0/1477947397 shutdown_connections 2021-02-10T11:03:27.274 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:27.272 7fa326d36700 1 -- 172.21.2.113:0/1477947397 wait complete. 2021-02-10T11:03:27.281 DEBUG:teuthology.orchestra.run.gibba013:> sudo ceph --cluster ceph osd new f5461f80-018d-428f-be89-2eb2bcbe7bb6 7 2021-02-10T11:03:27.709 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:27.691 7f1b3319b700 1 Processor -- start 2021-02-10T11:03:27.709 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:27.691 7f1b3319b700 1 -- start start 2021-02-10T11:03:27.709 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:27.692 7f1b3319b700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1b2c0d17f0 0x7f1b2c1b31e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:27.709 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:27.692 7f1b3319b700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f1b2c0cff00 0x7f1b2c0ef620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:27.709 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:27.692 7f1b3319b700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b2c0f0b20 con 0x7f1b2c0d1bf0 2021-02-10T11:03:27.709 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:27.692 7f1b3319b700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b2c0e5eb0 con 0x7f1b2c0cff00 2021-02-10T11:03:27.709 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:27.692 7f1b3319b700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b2c1b3ff0 con 0x7f1b2c0d17f0 2021-02-10T11:03:27.709 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:27.692 7f1b317b8700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f1b2c0cff00 0x7f1b2c0ef620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:27.709 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:27.692 7f1b317b8700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f1b2c0cff00 0x7f1b2c0ef620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.113:3300/0 says I am v2:172.21.2.113:60400/0 (socket says 172.21.2.113:60400) 2021-02-10T11:03:27.709 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:27.692 7f1b30fb7700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1b2c0d17f0 0x7f1b2c1b31e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:27.710 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:27.692 7f1b317b8700 1 -- 172.21.2.113:0/1025219274 learned_addr learned my addr 172.21.2.113:0/1025219274 (peer_addr_for_me v2:172.21.2.113:0/0) 2021-02-10T11:03:27.710 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:27.692 7f1b317b8700 1 -- 172.21.2.113:0/1025219274 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1b2c0d17f0 msgr2=0x7f1b2c1b31e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:03:27.710 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:27.692 7f1b317b8700 1 --2- 172.21.2.113:0/1025219274 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1b2c0d17f0 0x7f1b2c1b31e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:27.710 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:27.692 7f1b317b8700 1 -- 172.21.2.113:0/1025219274 >> v1:172.21.2.113:6789/0 conn(0x7f1b2c0d1bf0 legacy=0x7f1b2c1b3800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:27.710 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:27.692 7f1b317b8700 1 -- 172.21.2.113:0/1025219274 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b2c0f20b0 con 0x7f1b2c0cff00 2021-02-10T11:03:27.710 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:27.692 7f1b30fb7700 1 --2- 172.21.2.113:0/1025219274 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1b2c0d17f0 0x7f1b2c1b31e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:03:27.710 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:27.693 7f1b317b8700 1 --2- 172.21.2.113:0/1025219274 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1b2c0cff00 0x7f1b2c0ef620 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f1b24006b80 tx=0x7f1b24009b90).ready entity=mon.1 client_cookie=dec98e4b8a177afc server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:03:27.710 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:27.693 7f1b2b7fe700 1 -- 172.21.2.113:0/1025219274 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1b2400c0c0 con 0x7f1b2c0cff00 2021-02-10T11:03:27.710 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:27.693 7f1b2b7fe700 1 -- 172.21.2.113:0/1025219274 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b2400c380 con 0x7f1b2c0cff00 2021-02-10T11:03:27.710 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:27.693 7f1b2b7fe700 1 -- 172.21.2.113:0/1025219274 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1b2400c8a0 con 0x7f1b2c0cff00 2021-02-10T11:03:27.710 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:27.693 7f1b3319b700 1 -- 172.21.2.113:0/1025219274 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1b2c0cff00 msgr2=0x7f1b2c0ef620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:27.710 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:27.693 7f1b3319b700 1 --2- 172.21.2.113:0/1025219274 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1b2c0cff00 0x7f1b2c0ef620 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f1b24006b80 tx=0x7f1b24009b90).stop 2021-02-10T11:03:27.710 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:27.693 7f1b3319b700 1 -- 172.21.2.113:0/1025219274 shutdown_connections 2021-02-10T11:03:27.711 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:27.693 7f1b3319b700 1 --2- 172.21.2.113:0/1025219274 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1b2c0cff00 0x7f1b2c0ef620 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:27.711 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:27.693 7f1b3319b700 1 --2- 172.21.2.113:0/1025219274 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1b2c0d17f0 0x7f1b2c1b31e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:27.711 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:27.693 7f1b3319b700 1 -- 172.21.2.113:0/1025219274 >> 172.21.2.113:0/1025219274 conn(0x7f1b2c0fde40 msgr2=0x7f1b2c0ce390 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:03:27.711 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:27.694 7f1b3319b700 1 -- 172.21.2.113:0/1025219274 shutdown_connections 2021-02-10T11:03:27.711 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:27.694 7f1b3319b700 1 -- 172.21.2.113:0/1025219274 wait complete. 2021-02-10T11:03:27.713 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:27.708 7f1b3319b700 1 Processor -- start 2021-02-10T11:03:27.713 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:27.708 7f1b3319b700 1 -- start start 2021-02-10T11:03:27.713 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:27.709 7f1b3319b700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1b2c1038f0 0x7f1b2c1981c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:27.713 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:27.709 7f1b3319b700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1b2c198a00 0x7f1b2c198e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:27.713 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:27.709 7f1b3319b700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1b2c199420 0x7f1b2c0b1330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:27.713 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:27.709 7f1b3319b700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b2c0990c0 con 0x7f1b2c198a00 2021-02-10T11:03:27.713 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:27.709 7f1b3319b700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b2c0b1950 con 0x7f1b2c199420 2021-02-10T11:03:27.713 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:27.709 7f1b3319b700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b2c0b1b30 con 0x7f1b2c1038f0 2021-02-10T11:03:27.713 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:27.709 7f1b317b8700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1b2c199420 0x7f1b2c0b1330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:27.713 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:27.709 7f1b317b8700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1b2c199420 0x7f1b2c0b1330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.113:3300/0 says I am v2:172.21.2.113:60402/0 (socket says 172.21.2.113:60402) 2021-02-10T11:03:27.713 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:27.709 7f1b317b8700 1 -- 172.21.2.113:0/1255230149 learned_addr learned my addr 172.21.2.113:0/1255230149 (peer_addr_for_me v2:172.21.2.113:0/0) 2021-02-10T11:03:27.714 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:27.709 7f1b30fb7700 1 --2- 172.21.2.113:0/1255230149 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1b2c1038f0 0x7f1b2c1981c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:27.714 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:27.709 7f1b2bfff700 1 --2- 172.21.2.113:0/1255230149 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1b2c198a00 0x7f1b2c198e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:27.714 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:27.709 7f1b317b8700 1 -- 172.21.2.113:0/1255230149 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1b2c1038f0 msgr2=0x7f1b2c1981c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:27.714 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:27.709 7f1b317b8700 1 --2- 172.21.2.113:0/1255230149 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1b2c1038f0 0x7f1b2c1981c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:27.714 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:27.709 7f1b317b8700 1 -- 172.21.2.113:0/1255230149 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1b2c198a00 msgr2=0x7f1b2c198e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:27.714 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:27.709 7f1b317b8700 1 --2- 172.21.2.113:0/1255230149 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1b2c198a00 0x7f1b2c198e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:27.714 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:27.709 7f1b317b8700 1 -- 172.21.2.113:0/1255230149 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b2c0b1d50 con 0x7f1b2c199420 2021-02-10T11:03:27.714 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:27.710 7f1b317b8700 1 --2- 172.21.2.113:0/1255230149 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1b2c199420 0x7f1b2c0b1330 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f1b2400b630 tx=0x7f1b2400b9d0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:03:27.714 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:27.710 7f1b297fa700 1 -- 172.21.2.113:0/1255230149 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1b2400bdc0 con 0x7f1b2c199420 2021-02-10T11:03:27.714 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:27.710 7f1b297fa700 1 -- 172.21.2.113:0/1255230149 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b24000bc0 con 0x7f1b2c199420 2021-02-10T11:03:27.714 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:27.710 7f1b3319b700 1 -- 172.21.2.113:0/1255230149 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b2c0b1d50 con 0x7f1b2c199420 2021-02-10T11:03:27.714 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:27.710 7f1b297fa700 1 -- 172.21.2.113:0/1255230149 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1b240054a0 con 0x7f1b2c199420 2021-02-10T11:03:27.715 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:27.710 7f1b3319b700 1 -- 172.21.2.113:0/1255230149 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b2c099640 con 0x7f1b2c199420 2021-02-10T11:03:27.715 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:27.710 7f1b297fa700 1 -- 172.21.2.113:0/1255230149 <== mon.1 v2:172.21.2.113:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1b24009fa0 con 0x7f1b2c199420 2021-02-10T11:03:27.715 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:27.710 7f1b297fa700 1 --2- 172.21.2.113:0/1255230149 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1b100186d0 0x7f1b1001abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:27.715 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:27.711 7f1b297fa700 1 -- 172.21.2.113:0/1255230149 <== mon.1 v2:172.21.2.113:3300/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1475+0+0 (secure 0 0 0) 0x7f1b240317b0 con 0x7f1b2c199420 2021-02-10T11:03:27.715 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:27.711 7f1b30fb7700 1 --2- 172.21.2.113:0/1255230149 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1b100186d0 0x7f1b1001abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:27.715 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:27.711 7f1b30fb7700 1 --2- 172.21.2.113:0/1255230149 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1b100186d0 0x7f1b1001abd0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:03:27.742 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:27.741 7f1b3319b700 1 -- 172.21.2.113:0/1255230149 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b2c103cf0 con 0x7f1b2c199420 2021-02-10T11:03:27.745 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:27.744 7f1b297fa700 1 -- 172.21.2.113:0/1255230149 <== mon.1 v2:172.21.2.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f1b24009fa0 con 0x7f1b2c199420 2021-02-10T11:03:27.880 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:27.879 7f1b3319b700 1 -- 172.21.2.113:0/1255230149 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_command({"prefix": "osd new", "uuid": "f5461f80-018d-428f-be89-2eb2bcbe7bb6", "id": 7} v 0) v1 -- 0x7f1b2c103cf0 con 0x7f1b2c199420 2021-02-10T11:03:27.885 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:27.884 7f1b297fa700 1 -- 172.21.2.113:0/1255230149 <== mon.1 v2:172.21.2.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "f5461f80-018d-428f-be89-2eb2bcbe7bb6", "id": 7}]=0 v10) v1 ==== 112+0+2 (secure 0 0 0) 0x7f1b2400bdc0 con 0x7f1b2c199420 2021-02-10T11:03:27.887 INFO:teuthology.orchestra.run.gibba013.stdout:7 2021-02-10T11:03:27.888 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:27.888 7f1b3319b700 1 -- 172.21.2.113:0/1255230149 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1b100186d0 msgr2=0x7f1b1001abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:27.888 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:27.888 7f1b3319b700 1 --2- 172.21.2.113:0/1255230149 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1b100186d0 0x7f1b1001abd0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:27.889 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:27.888 7f1b3319b700 1 -- 172.21.2.113:0/1255230149 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1b2c199420 msgr2=0x7f1b2c0b1330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:27.889 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:27.888 7f1b3319b700 1 --2- 172.21.2.113:0/1255230149 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1b2c199420 0x7f1b2c0b1330 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f1b2400b630 tx=0x7f1b2400b9d0).stop 2021-02-10T11:03:27.889 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:27.888 7f1b3319b700 1 -- 172.21.2.113:0/1255230149 shutdown_connections 2021-02-10T11:03:27.890 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:27.888 7f1b3319b700 1 --2- 172.21.2.113:0/1255230149 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1b100186d0 0x7f1b1001abd0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:27.890 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:27.888 7f1b3319b700 1 --2- 172.21.2.113:0/1255230149 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1b2c199420 0x7f1b2c0b1330 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:27.890 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:27.888 7f1b3319b700 1 --2- 172.21.2.113:0/1255230149 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1b2c198a00 0x7f1b2c198e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:27.890 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:27.888 7f1b3319b700 1 --2- 172.21.2.113:0/1255230149 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1b2c1038f0 0x7f1b2c1981c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:27.890 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:27.888 7f1b3319b700 1 -- 172.21.2.113:0/1255230149 >> 172.21.2.113:0/1255230149 conn(0x7f1b2c08dd20 msgr2=0x7f1b2c0959b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:03:27.890 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:27.888 7f1b3319b700 1 -- 172.21.2.113:0/1255230149 shutdown_connections 2021-02-10T11:03:27.890 INFO:teuthology.orchestra.run.gibba013.stderr:2021-02-10 11:03:27.888 7f1b3319b700 1 -- 172.21.2.113:0/1255230149 wait complete. 2021-02-10T11:03:27.897 INFO:tasks.ceph.osd.0:Restarting daemon 2021-02-10T11:03:27.897 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2021-02-10T11:03:27.900 INFO:tasks.ceph.osd.0:Started 2021-02-10T11:03:27.900 INFO:tasks.ceph.osd.1:Restarting daemon 2021-02-10T11:03:27.900 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2021-02-10T11:03:27.902 INFO:tasks.ceph.osd.1:Started 2021-02-10T11:03:27.903 INFO:tasks.ceph.osd.2:Restarting daemon 2021-02-10T11:03:27.903 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2021-02-10T11:03:27.905 INFO:tasks.ceph.osd.2:Started 2021-02-10T11:03:27.905 INFO:tasks.ceph.osd.3:Restarting daemon 2021-02-10T11:03:27.905 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2021-02-10T11:03:27.908 INFO:tasks.ceph.osd.3:Started 2021-02-10T11:03:27.908 INFO:tasks.ceph.osd.4:Restarting daemon 2021-02-10T11:03:27.908 DEBUG:teuthology.orchestra.run.gibba013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2021-02-10T11:03:27.911 INFO:tasks.ceph.osd.4:Started 2021-02-10T11:03:27.911 INFO:tasks.ceph.osd.5:Restarting daemon 2021-02-10T11:03:27.911 DEBUG:teuthology.orchestra.run.gibba013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2021-02-10T11:03:27.913 INFO:tasks.ceph.osd.5:Started 2021-02-10T11:03:27.914 INFO:tasks.ceph.osd.6:Restarting daemon 2021-02-10T11:03:27.914 DEBUG:teuthology.orchestra.run.gibba013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2021-02-10T11:03:27.916 INFO:tasks.ceph.osd.6:Started 2021-02-10T11:03:27.916 INFO:tasks.ceph.osd.7:Restarting daemon 2021-02-10T11:03:27.916 DEBUG:teuthology.orchestra.run.gibba013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2021-02-10T11:03:27.918 INFO:tasks.ceph.osd.7:Started 2021-02-10T11:03:27.919 INFO:tasks.ceph:Waiting for OSDs to come up 2021-02-10T11:03:28.726 DEBUG:teuthology.orchestra.run.gibba002:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2021-02-10T11:03:28.818 INFO:tasks.ceph.osd.3.gibba002.stderr:2021-02-10 11:03:28.819 7f4aeb034a80 -1 Falling back to public interface 2021-02-10T11:03:28.820 INFO:tasks.ceph.osd.2.gibba002.stderr:2021-02-10 11:03:28.821 7f1aee6bba80 -1 Falling back to public interface 2021-02-10T11:03:28.832 INFO:tasks.ceph.osd.5.gibba013.stderr:2021-02-10 11:03:28.831 7f2965361a80 -1 Falling back to public interface 2021-02-10T11:03:28.833 INFO:tasks.ceph.osd.5.gibba013.stderr:2021-02-10 11:03:28.832 7f295e484700 -1 PosixStack listen unable to listen on v2:0.0.0.0:6800/0: (98) Address already in use 2021-02-10T11:03:28.833 INFO:tasks.ceph.osd.5.gibba013.stderr:2021-02-10 11:03:28.832 7f295e484700 -1 PosixStack listen unable to listen on v2:0.0.0.0:6801/0: (98) Address already in use 2021-02-10T11:03:28.833 INFO:tasks.ceph.osd.6.gibba013.stderr:2021-02-10 11:03:28.831 7fa2899a1a80 -1 Falling back to public interface 2021-02-10T11:03:28.833 INFO:tasks.ceph.osd.4.gibba013.stderr:2021-02-10 11:03:28.832 7f414aee7a80 -1 Falling back to public interface 2021-02-10T11:03:28.834 INFO:tasks.ceph.osd.6.gibba013.stderr:2021-02-10 11:03:28.833 7fa282ac4700 -1 PosixStack listen unable to listen on v2:0.0.0.0:6812/0: (98) Address already in use 2021-02-10T11:03:28.834 INFO:tasks.ceph.osd.7.gibba013.stderr:2021-02-10 11:03:28.833 7f56efdafa80 -1 Falling back to public interface 2021-02-10T11:03:28.835 INFO:tasks.ceph.osd.7.gibba013.stderr:2021-02-10 11:03:28.834 7f56e8ed2700 -1 PosixStack listen unable to listen on v1:0.0.0.0:6820/0: (98) Address already in use 2021-02-10T11:03:28.835 INFO:tasks.ceph.osd.0.gibba002.stderr:2021-02-10 11:03:28.836 7f95039cea80 -1 Falling back to public interface 2021-02-10T11:03:28.835 INFO:tasks.ceph.osd.1.gibba002.stderr:2021-02-10 11:03:28.836 7fb47fa8da80 -1 Falling back to public interface 2021-02-10T11:03:29.147 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:29.131 7f69b7fa8700 1 Processor -- start 2021-02-10T11:03:29.147 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:29.131 7f69b7fa8700 1 -- start start 2021-02-10T11:03:29.148 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:29.131 7f69b7fa8700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f69b00e5470 0x7f69b007e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:29.148 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:29.131 7f69b7fa8700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f69b007f300 0x7f69b007f700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:29.148 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:29.131 7f69b7fa8700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69b007ed80 con 0x7f69b00e5070 2021-02-10T11:03:29.148 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:29.131 7f69b7fa8700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69b01b4010 con 0x7f69b007f300 2021-02-10T11:03:29.148 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:29.131 7f69b7fa8700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69b00d1790 con 0x7f69b00e5470 2021-02-10T11:03:29.148 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:29.132 7f69b5dc4700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f69b00e5070 0x7f69b01b3820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:41000/0 (socket says 172.21.2.102:41000) 2021-02-10T11:03:29.148 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:29.132 7f69b5dc4700 1 -- 172.21.2.102:0/3895920326 learned_addr learned my addr 172.21.2.102:0/3895920326 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:03:29.148 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:29.132 7f69b55c3700 1 --2- 172.21.2.102:0/3895920326 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f69b00e5470 0x7f69b007e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:29.148 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:29.132 7f69b65c5700 1 --2- 172.21.2.102:0/3895920326 >> v2:172.21.2.113:3300/0 conn(0x7f69b007f300 0x7f69b007f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:29.148 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:29.132 7f69b4dc2700 1 -- 172.21.2.102:0/3895920326 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 734776372 0 0) 0x7f69a0001180 con 0x7f69b00e5070 2021-02-10T11:03:29.148 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:29.132 7f69b4dc2700 1 -- 172.21.2.102:0/3895920326 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f699c002fe0 con 0x7f69b00e5070 2021-02-10T11:03:29.148 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:29.132 7f69b55c3700 1 -- 172.21.2.102:0/3895920326 >> v2:172.21.2.113:3300/0 conn(0x7f69b007f300 msgr2=0x7f69b007f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:03:29.148 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:29.132 7f69b55c3700 1 --2- 172.21.2.102:0/3895920326 >> v2:172.21.2.113:3300/0 conn(0x7f69b007f300 0x7f69b007f700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:29.149 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:29.132 7f69b55c3700 1 -- 172.21.2.102:0/3895920326 >> v1:172.21.2.102:6789/0 conn(0x7f69b00e5070 legacy=0x7f69b01b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:29.149 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:29.132 7f69b55c3700 1 -- 172.21.2.102:0/3895920326 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69b00d1980 con 0x7f69b00e5470 2021-02-10T11:03:29.149 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:29.132 7f69b65c5700 1 --2- 172.21.2.102:0/3895920326 >> v2:172.21.2.113:3300/0 conn(0x7f69b007f300 0x7f69b007f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:03:29.149 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:29.132 7f69b55c3700 1 --2- 172.21.2.102:0/3895920326 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f69b00e5470 0x7f69b007e6a0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f69ac006650 tx=0x7f69ac009a80).ready entity=mon.2 client_cookie=2a0b0dc7cf3fc097 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:03:29.149 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:29.133 7f69b4dc2700 1 -- 172.21.2.102:0/3895920326 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f69ac0009b0 con 0x7f69b00e5470 2021-02-10T11:03:29.149 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:29.133 7f69b4dc2700 1 -- 172.21.2.102:0/3895920326 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69ac00c0c0 con 0x7f69b00e5470 2021-02-10T11:03:29.149 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:29.133 7f69b4dc2700 1 -- 172.21.2.102:0/3895920326 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f69ac00c520 con 0x7f69b00e5470 2021-02-10T11:03:29.149 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:29.133 7f69b7fa8700 1 -- 172.21.2.102:0/3895920326 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f69b00e5470 msgr2=0x7f69b007e6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:29.149 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:29.133 7f69b7fa8700 1 --2- 172.21.2.102:0/3895920326 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f69b00e5470 0x7f69b007e6a0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f69ac006650 tx=0x7f69ac009a80).stop 2021-02-10T11:03:29.149 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:29.133 7f69b7fa8700 1 -- 172.21.2.102:0/3895920326 shutdown_connections 2021-02-10T11:03:29.149 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:29.133 7f69b7fa8700 1 --2- 172.21.2.102:0/3895920326 >> v2:172.21.2.113:3300/0 conn(0x7f69b007f300 0x7f69b007f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:29.149 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:29.133 7f69b7fa8700 1 --2- 172.21.2.102:0/3895920326 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f69b00e5470 0x7f69b007e6a0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:29.149 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:29.133 7f69b7fa8700 1 -- 172.21.2.102:0/3895920326 >> 172.21.2.102:0/3895920326 conn(0x7f69b00ce690 msgr2=0x7f69b00e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:03:29.149 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:29.133 7f69b7fa8700 1 -- 172.21.2.102:0/3895920326 shutdown_connections 2021-02-10T11:03:29.150 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:29.133 7f69b7fa8700 1 -- 172.21.2.102:0/3895920326 wait complete. 2021-02-10T11:03:29.151 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:29.149 7f69b7fa8700 1 Processor -- start 2021-02-10T11:03:29.151 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:29.149 7f69b7fa8700 1 -- start start 2021-02-10T11:03:29.151 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:29.149 7f69b7fa8700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f69b01a6af0 0x7f69b009bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:29.152 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:29.149 7f69b7fa8700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f69b009c2d0 0x7f69b009c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:29.152 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:29.149 7f69b7fa8700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f69b008ca70 0x7f69b008ce70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:29.152 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:29.149 7f69b7fa8700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69b0187eb0 con 0x7f69b009c2d0 2021-02-10T11:03:29.152 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:29.149 7f69b7fa8700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69b009b430 con 0x7f69b008ca70 2021-02-10T11:03:29.152 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:29.149 7f69b7fa8700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69b01a6ef0 con 0x7f69b01a6af0 2021-02-10T11:03:29.152 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:29.149 7f69b55c3700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f69b009c2d0 0x7f69b009c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:29.152 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:29.149 7f69b55c3700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f69b009c2d0 0x7f69b009c6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:48782/0 (socket says 172.21.2.102:48782) 2021-02-10T11:03:29.152 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:29.149 7f69b55c3700 1 -- 172.21.2.102:0/2471727045 learned_addr learned my addr 172.21.2.102:0/2471727045 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:03:29.152 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:29.149 7f69b65c5700 1 --2- 172.21.2.102:0/2471727045 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f69b008ca70 0x7f69b008ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:29.152 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:29.150 7f69b5dc4700 1 --2- 172.21.2.102:0/2471727045 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f69b01a6af0 0x7f69b009bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:29.152 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:29.150 7f69b55c3700 1 -- 172.21.2.102:0/2471727045 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f69b01a6af0 msgr2=0x7f69b009bcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:29.152 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:29.150 7f69b55c3700 1 --2- 172.21.2.102:0/2471727045 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f69b01a6af0 0x7f69b009bcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:29.152 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:29.150 7f69b55c3700 1 -- 172.21.2.102:0/2471727045 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f69b008ca70 msgr2=0x7f69b008ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:29.153 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:29.150 7f69b55c3700 1 --2- 172.21.2.102:0/2471727045 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f69b008ca70 0x7f69b008ce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:29.153 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:29.150 7f69b55c3700 1 -- 172.21.2.102:0/2471727045 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69b008d490 con 0x7f69b009c2d0 2021-02-10T11:03:29.153 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:29.150 7f69b55c3700 1 --2- 172.21.2.102:0/2471727045 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f69b009c2d0 0x7f69b009c6d0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7f69ac009630 tx=0x7f69ac00b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:03:29.153 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:29.150 7f69a67fc700 1 -- 172.21.2.102:0/2471727045 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f69ac0093a0 con 0x7f69b009c2d0 2021-02-10T11:03:29.153 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:29.150 7f69a67fc700 1 -- 172.21.2.102:0/2471727045 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69ac00bd70 con 0x7f69b009c2d0 2021-02-10T11:03:29.153 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:29.150 7f69a67fc700 1 -- 172.21.2.102:0/2471727045 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f69ac0054d0 con 0x7f69b009c2d0 2021-02-10T11:03:29.153 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:29.150 7f69b7fa8700 1 -- 172.21.2.102:0/2471727045 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69b008d490 con 0x7f69b009c2d0 2021-02-10T11:03:29.153 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:29.150 7f69b7fa8700 1 -- 172.21.2.102:0/2471727045 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69b008d490 con 0x7f69b009c2d0 2021-02-10T11:03:29.153 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:29.150 7f69a67fc700 1 -- 172.21.2.102:0/2471727045 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f69ac0054f0 con 0x7f69b009c2d0 2021-02-10T11:03:29.153 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:29.151 7f69a67fc700 1 --2- 172.21.2.102:0/2471727045 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f69980186c0 0x7f699801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:29.153 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:29.151 7f69a67fc700 1 -- 172.21.2.102:0/2471727045 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1578+0+0 (secure 0 0 0) 0x7f69ac031490 con 0x7f69b009c2d0 2021-02-10T11:03:29.153 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:29.151 7f69b5dc4700 1 --2- 172.21.2.102:0/2471727045 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f69980186c0 0x7f699801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:29.153 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:29.151 7f69b5dc4700 1 --2- 172.21.2.102:0/2471727045 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f69980186c0 0x7f699801abc0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:03:29.179 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:29.180 7f69b7fa8700 1 -- 172.21.2.102:0/2471727045 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69b008daa0 con 0x7f69b009c2d0 2021-02-10T11:03:29.182 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:29.183 7f69a67fc700 1 -- 172.21.2.102:0/2471727045 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f69ac0052e0 con 0x7f69b009c2d0 2021-02-10T11:03:29.309 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:29.310 7f69b7fa8700 1 -- 172.21.2.102:0/2471727045 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f69b008daa0 con 0x7f69b009c2d0 2021-02-10T11:03:29.310 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:29.311 7f69a67fc700 1 -- 172.21.2.102:0/2471727045 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+5464 (secure 0 0 0) 0x7f69ac0054f0 con 0x7f69b009c2d0 2021-02-10T11:03:29.310 INFO:teuthology.misc.health.gibba002.stdout: 2021-02-10T11:03:29.310 INFO:teuthology.misc.health.gibba002.stdout:{"epoch":10,"fsid":"68db30ea-dbce-4aef-a6df-f01c6fea0513","created":"2021-02-10 11:03:06.331557","modified":"2021-02-10 11:03:27.883449","last_up_change":"0.000000","last_in_change":"0.000000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[],"osds":[{"osd":0,"uuid":"be691b4d-a039-408a-af80-9d53f02b6f1f","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":":/0","cluster_addr":":/0","heartbeat_back_addr":":/0","heartbeat_front_addr":":/0","state":["exists","new"]},{"osd":1,"uuid":"64e3ccc8-9c69-4d7c-865e-af8e83822635","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":":/0","cluster_addr":":/0","heartbeat_back_addr":":/0","heartbeat_front_addr":":/0","state":["exists","new"]},{"osd":2,"uuid":"153700af-747f-481f-bdf4-d6072d3a842f","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":":/0","cluster_addr":":/0","heartbeat_back_addr":":/0","heartbeat_front_addr":":/0","state":["exists","new"]},{"osd":3,"uuid":"cbc678f4-83a9-45e6-8830-2f7c232b1ea7","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":":/0","cluster_addr":":/0","heartbeat_back_addr":":/0","heartbeat_front_addr":":/0","state":["exists","new"]},{"osd":4,"uuid":"c01e4008-b258-4f10-afd2-9e7f37214351","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":":/0","cluster_addr":":/0","heartbeat_back_addr":":/0","heartbeat_front_addr":":/0","state":["exists","new"]},{"osd":5,"uuid":"551f6007-daf8-4a98-8ee0-c2b16bcccb5e","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":":/0","cluster_addr":":/0","heartbeat_back_addr":":/0","heartbeat_front_addr":":/0","state":["exists","new"]},{"osd":6,"uuid":"bf67c040-0e50-48be-b102-c3943bc9ebef","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":":/0","cluster_addr":":/0","heartbeat_back_addr":":/0","heartbeat_front_addr":":/0","state":["exists","new"]},{"osd":7,"uuid":"f5461f80-018d-428f-be89-2eb2bcbe7bb6","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":":/0","cluster_addr":":/0","heartbeat_back_addr":":/0","heartbeat_front_addr":":/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:03:29.312 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:29.313 7f69b7fa8700 1 -- 172.21.2.102:0/2471727045 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f69980186c0 msgr2=0x7f699801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:29.312 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:29.313 7f69b7fa8700 1 --2- 172.21.2.102:0/2471727045 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f69980186c0 0x7f699801abc0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:29.312 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:29.313 7f69b7fa8700 1 -- 172.21.2.102:0/2471727045 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f69b009c2d0 msgr2=0x7f69b009c6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:29.312 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:29.313 7f69b7fa8700 1 --2- 172.21.2.102:0/2471727045 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f69b009c2d0 0x7f69b009c6d0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7f69ac009630 tx=0x7f69ac00b980).stop 2021-02-10T11:03:29.315 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:29.313 7f69b7fa8700 1 -- 172.21.2.102:0/2471727045 shutdown_connections 2021-02-10T11:03:29.315 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:29.313 7f69b7fa8700 1 --2- 172.21.2.102:0/2471727045 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f69980186c0 0x7f699801abc0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:29.315 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:29.313 7f69b7fa8700 1 --2- 172.21.2.102:0/2471727045 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f69b008ca70 0x7f69b008ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:29.315 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:29.313 7f69b7fa8700 1 --2- 172.21.2.102:0/2471727045 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f69b009c2d0 0x7f69b009c6d0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:29.315 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:29.313 7f69b7fa8700 1 --2- 172.21.2.102:0/2471727045 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f69b01a6af0 0x7f69b009bcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:29.316 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:29.313 7f69b7fa8700 1 -- 172.21.2.102:0/2471727045 >> 172.21.2.102:0/2471727045 conn(0x7f69b009b790 msgr2=0x7f69b0093720 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:03:29.316 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:29.313 7f69b7fa8700 1 -- 172.21.2.102:0/2471727045 shutdown_connections 2021-02-10T11:03:29.316 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:29.313 7f69b7fa8700 1 -- 172.21.2.102:0/2471727045 wait complete. 2021-02-10T11:03:29.319 DEBUG:teuthology.misc:0 of 8 OSDs are up 2021-02-10T11:03:29.857 INFO:tasks.ceph.osd.2.gibba002.stderr:2021-02-10 11:03:29.857 7f1aee6bba80 -1 osd.2 0 log_to_monitors {default=true} 2021-02-10T11:03:29.859 INFO:tasks.ceph.osd.3.gibba002.stderr:2021-02-10 11:03:29.859 7f4aeb034a80 -1 osd.3 0 log_to_monitors {default=true} 2021-02-10T11:03:29.870 INFO:tasks.ceph.osd.4.gibba013.stderr:2021-02-10 11:03:29.869 7f414aee7a80 -1 osd.4 0 log_to_monitors {default=true} 2021-02-10T11:03:29.872 INFO:tasks.ceph.osd.5.gibba013.stderr:2021-02-10 11:03:29.871 7f2965361a80 -1 osd.5 0 log_to_monitors {default=true} 2021-02-10T11:03:29.874 INFO:tasks.ceph.osd.6.gibba013.stderr:2021-02-10 11:03:29.874 7fa2899a1a80 -1 osd.6 0 log_to_monitors {default=true} 2021-02-10T11:03:29.876 INFO:tasks.ceph.osd.0.gibba002.stderr:2021-02-10 11:03:29.877 7f95039cea80 -1 osd.0 0 log_to_monitors {default=true} 2021-02-10T11:03:29.880 INFO:tasks.ceph.osd.1.gibba002.stderr:2021-02-10 11:03:29.881 7fb47fa8da80 -1 osd.1 0 log_to_monitors {default=true} 2021-02-10T11:03:29.884 INFO:tasks.ceph.osd.7.gibba013.stderr:2021-02-10 11:03:29.883 7f56efdafa80 -1 osd.7 0 log_to_monitors {default=true} 2021-02-10T11:03:31.689 INFO:tasks.ceph.osd.2.gibba002.stderr:2021-02-10 11:03:31.689 7f1ad4d34700 -1 osd.2 0 waiting for initial osdmap 2021-02-10T11:03:31.689 INFO:tasks.ceph.osd.3.gibba002.stderr:2021-02-10 11:03:31.689 7f4ad16ad700 -1 osd.3 0 waiting for initial osdmap 2021-02-10T11:03:31.689 INFO:tasks.ceph.osd.0.gibba002.stderr:2021-02-10 11:03:31.689 7f94ea047700 -1 osd.0 0 waiting for initial osdmap 2021-02-10T11:03:31.689 INFO:tasks.ceph.osd.1.gibba002.stderr:2021-02-10 11:03:31.689 7fb466106700 -1 osd.1 0 waiting for initial osdmap 2021-02-10T11:03:31.690 INFO:tasks.ceph.osd.4.gibba013.stderr:2021-02-10 11:03:31.689 7f4131560700 -1 osd.4 0 waiting for initial osdmap 2021-02-10T11:03:31.690 INFO:tasks.ceph.osd.5.gibba013.stderr:2021-02-10 11:03:31.689 7f294b9da700 -1 osd.5 0 waiting for initial osdmap 2021-02-10T11:03:31.690 INFO:tasks.ceph.osd.6.gibba013.stderr:2021-02-10 11:03:31.689 7fa27001a700 -1 osd.6 0 waiting for initial osdmap 2021-02-10T11:03:31.690 INFO:tasks.ceph.osd.7.gibba013.stderr:2021-02-10 11:03:31.689 7f56d6428700 -1 osd.7 0 waiting for initial osdmap 2021-02-10T11:03:31.693 INFO:tasks.ceph.osd.4.gibba013.stderr:2021-02-10 11:03:31.692 7f413d77e700 -1 osd.4 12 set_numa_affinity unable to identify public interface 'enp1s0f0' numa node: (0) Success 2021-02-10T11:03:31.693 INFO:tasks.ceph.osd.5.gibba013.stderr:2021-02-10 11:03:31.692 7f2957bf8700 -1 osd.5 12 set_numa_affinity unable to identify public interface 'enp1s0f0' numa node: (0) Success 2021-02-10T11:03:31.693 INFO:tasks.ceph.osd.6.gibba013.stderr:2021-02-10 11:03:31.692 7fa27c238700 -1 osd.6 12 set_numa_affinity unable to identify public interface 'enp1s0f0' numa node: (0) Success 2021-02-10T11:03:31.693 INFO:tasks.ceph.osd.7.gibba013.stderr:2021-02-10 11:03:31.692 7f56e2646700 -1 osd.7 12 set_numa_affinity unable to identify public interface 'enp1s0f0' numa node: (0) Success 2021-02-10T11:03:31.694 INFO:tasks.ceph.osd.0.gibba002.stderr:2021-02-10 11:03:31.694 7f94f6265700 -1 osd.0 12 set_numa_affinity unable to identify public interface 'enp1s0f0' numa node: (0) Success 2021-02-10T11:03:31.694 INFO:tasks.ceph.osd.1.gibba002.stderr:2021-02-10 11:03:31.694 7fb472324700 -1 osd.1 12 set_numa_affinity unable to identify public interface 'enp1s0f0' numa node: (0) Success 2021-02-10T11:03:31.694 INFO:tasks.ceph.osd.2.gibba002.stderr:2021-02-10 11:03:31.694 7f1ae0f52700 -1 osd.2 12 set_numa_affinity unable to identify public interface 'enp1s0f0' numa node: (0) Success 2021-02-10T11:03:31.694 INFO:tasks.ceph.osd.3.gibba002.stderr:2021-02-10 11:03:31.694 7f4add8cb700 -1 osd.3 12 set_numa_affinity unable to identify public interface 'enp1s0f0' numa node: (0) Success 2021-02-10T11:03:36.126 DEBUG:teuthology.orchestra.run.gibba002:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2021-02-10T11:03:36.543 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:36.527 7f3faf5e1700 1 Processor -- start 2021-02-10T11:03:36.544 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:36.528 7f3faf5e1700 1 -- start start 2021-02-10T11:03:36.544 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:36.528 7f3faf5e1700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f3fa807e6a0 0x7f3fa80d1790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:36.544 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:36.528 7f3faf5e1700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fa807ed80 con 0x7f3fa80e5070 2021-02-10T11:03:36.544 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:36.528 7f3faf5e1700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fa807eaa0 con 0x7f3fa80e5470 2021-02-10T11:03:36.544 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:36.528 7f3faf5e1700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3fa81b4080 con 0x7f3fa807e6a0 2021-02-10T11:03:36.544 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:36.528 7f3fad3fd700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f3fa80e5070 0x7f3fa81b3820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:41236/0 (socket says 172.21.2.102:41236) 2021-02-10T11:03:36.544 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:36.528 7f3fad3fd700 1 -- 172.21.2.102:0/3340208839 learned_addr learned my addr 172.21.2.102:0/3340208839 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:03:36.544 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:36.528 7f3fadbfe700 1 --2- 172.21.2.102:0/3340208839 >> v2:172.21.2.113:3300/0 conn(0x7f3fa807e6a0 0x7f3fa80d1790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:36.544 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:36.529 7f3f9bfff700 1 -- 172.21.2.102:0/3340208839 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 430587771 0 0) 0x7f3f9c000ce0 con 0x7f3fa80e5070 2021-02-10T11:03:36.544 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:36.529 7f3f9bfff700 1 -- 172.21.2.102:0/3340208839 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f94002fe0 con 0x7f3fa80e5070 2021-02-10T11:03:36.544 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:36.529 7f3fadbfe700 1 -- 172.21.2.102:0/3340208839 >> v1:172.21.2.113:6789/0 conn(0x7f3fa80e5470 legacy=0x7f3fa807f300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:36.544 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:36.529 7f3fadbfe700 1 -- 172.21.2.102:0/3340208839 >> v1:172.21.2.102:6789/0 conn(0x7f3fa80e5070 legacy=0x7f3fa81b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:36.545 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:36.529 7f3fadbfe700 1 -- 172.21.2.102:0/3340208839 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fa80d1db0 con 0x7f3fa807e6a0 2021-02-10T11:03:36.545 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:36.529 7f3fadbfe700 1 --2- 172.21.2.102:0/3340208839 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3fa807e6a0 0x7f3fa80d1790 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7f3fa0006e50 tx=0x7f3fa0009e70).ready entity=mon.1 client_cookie=b06d86c68ba34b0a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:03:36.545 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:36.529 7f3f9bfff700 1 -- 172.21.2.102:0/3340208839 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3fa000b510 con 0x7f3fa807e6a0 2021-02-10T11:03:36.545 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:36.529 7f3f9bfff700 1 -- 172.21.2.102:0/3340208839 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fa0006790 con 0x7f3fa807e6a0 2021-02-10T11:03:36.545 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:36.529 7f3f9bfff700 1 -- 172.21.2.102:0/3340208839 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3fa000b940 con 0x7f3fa807e6a0 2021-02-10T11:03:36.545 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:36.529 7f3faf5e1700 1 -- 172.21.2.102:0/3340208839 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3fa807e6a0 msgr2=0x7f3fa80d1790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:36.545 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:36.529 7f3faf5e1700 1 --2- 172.21.2.102:0/3340208839 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3fa807e6a0 0x7f3fa80d1790 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7f3fa0006e50 tx=0x7f3fa0009e70).stop 2021-02-10T11:03:36.545 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:36.530 7f3faf5e1700 1 -- 172.21.2.102:0/3340208839 shutdown_connections 2021-02-10T11:03:36.545 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:36.530 7f3faf5e1700 1 --2- 172.21.2.102:0/3340208839 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3fa807e6a0 0x7f3fa80d1790 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:36.545 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:36.530 7f3faf5e1700 1 -- 172.21.2.102:0/3340208839 >> 172.21.2.102:0/3340208839 conn(0x7f3fa80ce690 msgr2=0x7f3fa80e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:03:36.545 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:36.530 7f3faf5e1700 1 -- 172.21.2.102:0/3340208839 shutdown_connections 2021-02-10T11:03:36.545 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:36.530 7f3faf5e1700 1 -- 172.21.2.102:0/3340208839 wait complete. 2021-02-10T11:03:36.545 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:36.544 7f3faf5e1700 1 Processor -- start 2021-02-10T11:03:36.547 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:36.545 7f3faf5e1700 1 -- start start 2021-02-10T11:03:36.547 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:36.545 7f3faf5e1700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3fa8093b80 0x7f3fa8093f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:36.547 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:36.545 7f3faf5e1700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3fa819efc0 0x7f3fa819f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:36.547 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:36.545 7f3faf5e1700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3fa81a0060 0x7f3fa81a0460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:36.547 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:36.545 7f3faf5e1700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fa819fae0 con 0x7f3fa8093b80 2021-02-10T11:03:36.547 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:36.545 7f3faf5e1700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fa80946b0 con 0x7f3fa819efc0 2021-02-10T11:03:36.547 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:36.545 7f3faf5e1700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fa81a0a80 con 0x7f3fa81a0060 2021-02-10T11:03:36.548 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:36.545 7f3fad3fd700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3fa8093b80 0x7f3fa8093f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:36.548 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:36.545 7f3fad3fd700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3fa8093b80 0x7f3fa8093f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:49018/0 (socket says 172.21.2.102:49018) 2021-02-10T11:03:36.548 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:36.545 7f3fad3fd700 1 -- 172.21.2.102:0/2080635881 learned_addr learned my addr 172.21.2.102:0/2080635881 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:03:36.548 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:36.546 7f3facbfc700 1 --2- 172.21.2.102:0/2080635881 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3fa819efc0 0x7f3fa819f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:36.548 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:36.546 7f3fadbfe700 1 --2- 172.21.2.102:0/2080635881 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3fa81a0060 0x7f3fa81a0460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:36.548 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:36.546 7f3fad3fd700 1 -- 172.21.2.102:0/2080635881 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3fa81a0060 msgr2=0x7f3fa81a0460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:36.548 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:36.546 7f3fad3fd700 1 --2- 172.21.2.102:0/2080635881 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3fa81a0060 0x7f3fa81a0460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:36.548 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:36.546 7f3fad3fd700 1 -- 172.21.2.102:0/2080635881 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3fa819efc0 msgr2=0x7f3fa819f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:36.548 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:36.546 7f3fad3fd700 1 --2- 172.21.2.102:0/2080635881 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3fa819efc0 0x7f3fa819f3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:36.548 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:36.546 7f3fad3fd700 1 -- 172.21.2.102:0/2080635881 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fa81a0ca0 con 0x7f3fa8093b80 2021-02-10T11:03:36.548 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:36.546 7f3fad3fd700 1 --2- 172.21.2.102:0/2080635881 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3fa8093b80 0x7f3fa8093f80 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f3f9c00b5c0 tx=0x7f3f9c00b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:03:36.548 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:36.546 7f3f99ffb700 1 -- 172.21.2.102:0/2080635881 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3f9c006030 con 0x7f3fa8093b80 2021-02-10T11:03:36.548 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:36.546 7f3f99ffb700 1 -- 172.21.2.102:0/2080635881 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f9c0043a0 con 0x7f3fa8093b80 2021-02-10T11:03:36.549 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:36.546 7f3faf5e1700 1 -- 172.21.2.102:0/2080635881 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fa81a0ca0 con 0x7f3fa8093b80 2021-02-10T11:03:36.549 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:36.546 7f3faf5e1700 1 -- 172.21.2.102:0/2080635881 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fa81a1230 con 0x7f3fa8093b80 2021-02-10T11:03:36.549 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:36.546 7f3f99ffb700 1 -- 172.21.2.102:0/2080635881 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3f9c0048c0 con 0x7f3fa8093b80 2021-02-10T11:03:36.549 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:36.547 7f3f99ffb700 1 -- 172.21.2.102:0/2080635881 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f3f9c009fb0 con 0x7f3fa8093b80 2021-02-10T11:03:36.549 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:36.547 7f3f99ffb700 1 --2- 172.21.2.102:0/2080635881 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3f880186c0 0x7f3f8801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:36.549 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:36.547 7f3f99ffb700 1 -- 172.21.2.102:0/2080635881 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(13..13 src has 1..13) v4 ==== 4400+0+0 (secure 0 0 0) 0x7f3f9c032300 con 0x7f3fa8093b80 2021-02-10T11:03:36.549 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:36.547 7f3facbfc700 1 --2- 172.21.2.102:0/2080635881 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3f880186c0 0x7f3f8801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:36.549 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:36.547 7f3facbfc700 1 --2- 172.21.2.102:0/2080635881 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3f880186c0 0x7f3f8801abc0 crc :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:03:36.577 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:36.578 7f3faf5e1700 1 -- 172.21.2.102:0/2080635881 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fa81a5150 con 0x7f3fa8093b80 2021-02-10T11:03:36.580 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:36.581 7f3f99ffb700 1 -- 172.21.2.102:0/2080635881 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f3f9c00a390 con 0x7f3fa8093b80 2021-02-10T11:03:36.704 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:36.705 7f3faf5e1700 1 -- 172.21.2.102:0/2080635881 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3fa81a5150 con 0x7f3fa8093b80 2021-02-10T11:03:36.704 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:36.706 7f3f99ffb700 1 -- 172.21.2.102:0/2080635881 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v13) v1 ==== 74+0+9772 (secure 0 0 0) 0x7f3f9c009fb0 con 0x7f3fa8093b80 2021-02-10T11:03:36.705 INFO:teuthology.misc.health.gibba002.stdout: 2021-02-10T11:03:36.705 INFO:teuthology.misc.health.gibba002.stdout:{"epoch":13,"fsid":"68db30ea-dbce-4aef-a6df-f01c6fea0513","created":"2021-02-10 11:03:06.331557","modified":"2021-02-10 11:03:32.690562","last_up_change":"2021-02-10 11:03:32.690562","last_in_change":"2021-02-10 11:03:32.690562","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[],"osds":[{"osd":0,"uuid":"be691b4d-a039-408a-af80-9d53f02b6f1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6818","nonce":36088},{"type":"v1","addr":"172.21.2.102:6820","nonce":36088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6822","nonce":36088},{"type":"v1","addr":"172.21.2.102:6824","nonce":36088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6830","nonce":36088},{"type":"v1","addr":"172.21.2.102:6832","nonce":36088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6826","nonce":36088},{"type":"v1","addr":"172.21.2.102:6828","nonce":36088}]},"public_addr":"172.21.2.102:6820/36088","cluster_addr":"172.21.2.102:6824/36088","heartbeat_back_addr":"172.21.2.102:6832/36088","heartbeat_front_addr":"172.21.2.102:6828/36088","state":["exists","up"]},{"osd":1,"uuid":"64e3ccc8-9c69-4d7c-865e-af8e83822635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6819","nonce":36087},{"type":"v1","addr":"172.21.2.102:6821","nonce":36087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6823","nonce":36087},{"type":"v1","addr":"172.21.2.102:6825","nonce":36087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6831","nonce":36087},{"type":"v1","addr":"172.21.2.102:6833","nonce":36087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6827","nonce":36087},{"type":"v1","addr":"172.21.2.102:6829","nonce":36087}]},"public_addr":"172.21.2.102:6821/36087","cluster_addr":"172.21.2.102:6825/36087","heartbeat_back_addr":"172.21.2.102:6833/36087","heartbeat_front_addr":"172.21.2.102:6829/36087","state":["exists","up"]},{"osd":2,"uuid":"153700af-747f-481f-bdf4-d6072d3a842f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6810","nonce":36089},{"type":"v1","addr":"172.21.2.102:6811","nonce":36089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6812","nonce":36089},{"type":"v1","addr":"172.21.2.102:6813","nonce":36089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6816","nonce":36089},{"type":"v1","addr":"172.21.2.102:6817","nonce":36089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6814","nonce":36089},{"type":"v1","addr":"172.21.2.102:6815","nonce":36089}]},"public_addr":"172.21.2.102:6811/36089","cluster_addr":"172.21.2.102:6813/36089","heartbeat_back_addr":"172.21.2.102:6817/36089","heartbeat_front_addr":"172.21.2.102:6815/36089","state":["exists","up"]},{"osd":3,"uuid":"cbc678f4-83a9-45e6-8830-2f7c232b1ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6802","nonce":36090},{"type":"v1","addr":"172.21.2.102:6803","nonce":36090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6804","nonce":36090},{"type":"v1","addr":"172.21.2.102:6805","nonce":36090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6808","nonce":36090},{"type":"v1","addr":"172.21.2.102:6809","nonce":36090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6806","nonce":36090},{"type":"v1","addr":"172.21.2.102:6807","nonce":36090}]},"public_addr":"172.21.2.102:6803/36090","cluster_addr":"172.21.2.102:6805/36090","heartbeat_back_addr":"172.21.2.102:6809/36090","heartbeat_front_addr":"172.21.2.102:6807/36090","state":["exists","up"]},{"osd":4,"uuid":"c01e4008-b258-4f10-afd2-9e7f37214351","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6808","nonce":36368},{"type":"v1","addr":"172.21.2.113:6811","nonce":36368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6814","nonce":36368},{"type":"v1","addr":"172.21.2.113:6818","nonce":36368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6826","nonce":36368},{"type":"v1","addr":"172.21.2.113:6828","nonce":36368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6820","nonce":36368},{"type":"v1","addr":"172.21.2.113:6824","nonce":36368}]},"public_addr":"172.21.2.113:6811/36368","cluster_addr":"172.21.2.113:6818/36368","heartbeat_back_addr":"172.21.2.113:6828/36368","heartbeat_front_addr":"172.21.2.113:6824/36368","state":["exists","up"]},{"osd":5,"uuid":"551f6007-daf8-4a98-8ee0-c2b16bcccb5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6802","nonce":36366},{"type":"v1","addr":"172.21.2.113:6805","nonce":36366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6807","nonce":36366},{"type":"v1","addr":"172.21.2.113:6810","nonce":36366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6819","nonce":36366},{"type":"v1","addr":"172.21.2.113:6822","nonce":36366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6812","nonce":36366},{"type":"v1","addr":"172.21.2.113:6815","nonce":36366}]},"public_addr":"172.21.2.113:6805/36366","cluster_addr":"172.21.2.113:6810/36366","heartbeat_back_addr":"172.21.2.113:6822/36366","heartbeat_front_addr":"172.21.2.113:6815/36366","state":["exists","up"]},{"osd":6,"uuid":"bf67c040-0e50-48be-b102-c3943bc9ebef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6800","nonce":36367},{"type":"v1","addr":"172.21.2.113:6801","nonce":36367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6803","nonce":36367},{"type":"v1","addr":"172.21.2.113:6804","nonce":36367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6813","nonce":36367},{"type":"v1","addr":"172.21.2.113:6816","nonce":36367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6806","nonce":36367},{"type":"v1","addr":"172.21.2.113:6809","nonce":36367}]},"public_addr":"172.21.2.113:6801/36367","cluster_addr":"172.21.2.113:6804/36367","heartbeat_back_addr":"172.21.2.113:6816/36367","heartbeat_front_addr":"172.21.2.113:6809/36367","state":["exists","up"]},{"osd":7,"uuid":"f5461f80-018d-428f-be89-2eb2bcbe7bb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6817","nonce":36365},{"type":"v1","addr":"172.21.2.113:6821","nonce":36365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6823","nonce":36365},{"type":"v1","addr":"172.21.2.113:6825","nonce":36365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6830","nonce":36365},{"type":"v1","addr":"172.21.2.113:6831","nonce":36365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6827","nonce":36365},{"type":"v1","addr":"172.21.2.113:6829","nonce":36365}]},"public_addr":"172.21.2.113:6821/36365","cluster_addr":"172.21.2.113:6825/36365","heartbeat_back_addr":"172.21.2.113:6831/36365","heartbeat_front_addr":"172.21.2.113:6829/36365","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:03:36.707 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:36.707 7f3faf5e1700 1 -- 172.21.2.102:0/2080635881 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3f880186c0 msgr2=0x7f3f8801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:36.707 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:36.707 7f3faf5e1700 1 --2- 172.21.2.102:0/2080635881 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3f880186c0 0x7f3f8801abc0 crc :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:36.707 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:36.707 7f3faf5e1700 1 -- 172.21.2.102:0/2080635881 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3fa8093b80 msgr2=0x7f3fa8093f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:36.707 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:36.707 7f3faf5e1700 1 --2- 172.21.2.102:0/2080635881 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3fa8093b80 0x7f3fa8093f80 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f3f9c00b5c0 tx=0x7f3f9c00b960).stop 2021-02-10T11:03:36.707 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:36.708 7f3faf5e1700 1 -- 172.21.2.102:0/2080635881 shutdown_connections 2021-02-10T11:03:36.707 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:36.708 7f3faf5e1700 1 --2- 172.21.2.102:0/2080635881 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3fa81a0060 0x7f3fa81a0460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:36.707 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:36.708 7f3faf5e1700 1 --2- 172.21.2.102:0/2080635881 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3f880186c0 0x7f3f8801abc0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:36.707 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:36.708 7f3faf5e1700 1 --2- 172.21.2.102:0/2080635881 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3fa819efc0 0x7f3fa819f3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:36.708 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:36.708 7f3faf5e1700 1 --2- 172.21.2.102:0/2080635881 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3fa8093b80 0x7f3fa8093f80 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:36.708 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:36.708 7f3faf5e1700 1 -- 172.21.2.102:0/2080635881 >> 172.21.2.102:0/2080635881 conn(0x7f3fa81a2430 msgr2=0x7f3fa8094ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:03:36.708 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:36.708 7f3faf5e1700 1 -- 172.21.2.102:0/2080635881 shutdown_connections 2021-02-10T11:03:36.708 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:36.708 7f3faf5e1700 1 -- 172.21.2.102:0/2080635881 wait complete. 2021-02-10T11:03:36.714 DEBUG:teuthology.misc:8 of 8 OSDs are up 2021-02-10T11:03:36.715 INFO:tasks.ceph:Creating RBD pool 2021-02-10T11:03:36.715 DEBUG:teuthology.orchestra.run.gibba002:> sudo ceph --cluster ceph osd pool create rbd 8 2021-02-10T11:03:37.143 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:37.126 7f2629ff6700 1 Processor -- start 2021-02-10T11:03:37.143 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:37.126 7f2629ff6700 1 -- start start 2021-02-10T11:03:37.143 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:37.127 7f2629ff6700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f26241b0470 0x7f26241b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:37.143 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:37.127 7f2629ff6700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f26240f20e0 0x7f26240f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:37.143 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:37.127 7f2629ff6700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26240f0b50 con 0x7f26240ef610 2021-02-10T11:03:37.143 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:37.127 7f2629ff6700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26240f0820 con 0x7f26241b0470 2021-02-10T11:03:37.143 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:37.127 7f2629ff6700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26240f24e0 con 0x7f26240f20e0 2021-02-10T11:03:37.143 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:37.128 7f2623fff700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f26240f20e0 0x7f26240f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:37.143 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:37.128 7f2623fff700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f26240f20e0 0x7f26240f0200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.113:3301/0 says I am v2:172.21.2.102:54564/0 (socket says 172.21.2.102:54564) 2021-02-10T11:03:37.143 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:37.128 7f2623fff700 1 -- 172.21.2.102:0/228487532 learned_addr learned my addr 172.21.2.102:0/228487532 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:03:37.144 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:37.128 7f26237fe700 1 --2- 172.21.2.102:0/228487532 >> v2:172.21.2.113:3300/0 conn(0x7f26241b0470 0x7f26241b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:37.144 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:37.128 7f2622ffd700 1 --1- 172.21.2.102:0/228487532 >> v1:172.21.2.113:6789/0 conn(0x7f26240ef610 0x7f26240efa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.113:6789/0 says I am v1:172.21.2.102:48572/0 (socket says 172.21.2.102:48572) 2021-02-10T11:03:37.144 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:37.128 7f2623fff700 1 -- 172.21.2.102:0/228487532 >> v2:172.21.2.113:3300/0 conn(0x7f26241b0470 msgr2=0x7f26241b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:03:37.144 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:37.128 7f2623fff700 1 --2- 172.21.2.102:0/228487532 >> v2:172.21.2.113:3300/0 conn(0x7f26241b0470 0x7f26241b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:37.144 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:37.128 7f2623fff700 1 -- 172.21.2.102:0/228487532 >> v1:172.21.2.113:6789/0 conn(0x7f26240ef610 legacy=0x7f26240efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:37.144 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:37.128 7f2623fff700 1 -- 172.21.2.102:0/228487532 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26240e52e0 con 0x7f26240f20e0 2021-02-10T11:03:37.144 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:37.128 7f2623fff700 1 --2- 172.21.2.102:0/228487532 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f26240f20e0 0x7f26240f0200 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f261c006aa0 tx=0x7f261c009ea0).ready entity=mon.2 client_cookie=72bb522705317234 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:03:37.144 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:37.129 7f26227fc700 1 -- 172.21.2.102:0/228487532 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f261c00b510 con 0x7f26240f20e0 2021-02-10T11:03:37.144 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:37.129 7f26227fc700 1 -- 172.21.2.102:0/228487532 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f261c003030 con 0x7f26240f20e0 2021-02-10T11:03:37.144 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:37.129 7f26227fc700 1 -- 172.21.2.102:0/228487532 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f261c00ba10 con 0x7f26240f20e0 2021-02-10T11:03:37.144 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:37.129 7f2629ff6700 1 -- 172.21.2.102:0/228487532 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f26240f20e0 msgr2=0x7f26240f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:37.144 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:37.129 7f2629ff6700 1 --2- 172.21.2.102:0/228487532 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f26240f20e0 0x7f26240f0200 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f261c006aa0 tx=0x7f261c009ea0).stop 2021-02-10T11:03:37.144 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:37.129 7f2629ff6700 1 -- 172.21.2.102:0/228487532 shutdown_connections 2021-02-10T11:03:37.145 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:37.129 7f2629ff6700 1 --2- 172.21.2.102:0/228487532 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f26240f20e0 0x7f26240f0200 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:37.145 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:37.129 7f2629ff6700 1 --2- 172.21.2.102:0/228487532 >> v2:172.21.2.113:3300/0 conn(0x7f26241b0470 0x7f26241b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:37.145 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:37.129 7f2629ff6700 1 -- 172.21.2.102:0/228487532 >> 172.21.2.102:0/228487532 conn(0x7f26240ce3e0 msgr2=0x7f26240e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:03:37.145 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:37.129 7f2629ff6700 1 -- 172.21.2.102:0/228487532 shutdown_connections 2021-02-10T11:03:37.145 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:37.129 7f2629ff6700 1 -- 172.21.2.102:0/228487532 wait complete. 2021-02-10T11:03:37.145 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:37.143 7f2629ff6700 1 Processor -- start 2021-02-10T11:03:37.145 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:37.144 7f2629ff6700 1 -- start start 2021-02-10T11:03:37.147 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:37.144 7f2629ff6700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2624190920 0x7f26240b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:37.147 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:37.144 7f2629ff6700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f26240b8200 0x7f26240b8600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:37.147 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:37.144 7f2629ff6700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f26240c1d80 0x7f26240c2180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:37.148 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:37.144 7f2629ff6700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2624190da0 con 0x7f2624190920 2021-02-10T11:03:37.148 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:37.144 7f2629ff6700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26240b8c20 con 0x7f26240b8200 2021-02-10T11:03:37.148 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:37.144 7f2629ff6700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26240b8e10 con 0x7f26240c1d80 2021-02-10T11:03:37.148 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:37.144 7f26237fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2624190920 0x7f26240b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:37.148 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:37.144 7f26237fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2624190920 0x7f26240b7be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:49030/0 (socket says 172.21.2.102:49030) 2021-02-10T11:03:37.148 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:37.144 7f26237fe700 1 -- 172.21.2.102:0/3700538077 learned_addr learned my addr 172.21.2.102:0/3700538077 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:03:37.148 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:37.144 7f2623fff700 1 --2- 172.21.2.102:0/3700538077 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f26240c1d80 0x7f26240c2180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:37.148 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:37.144 7f2622ffd700 1 --2- 172.21.2.102:0/3700538077 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f26240b8200 0x7f26240b8600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:37.148 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:37.145 7f26237fe700 1 -- 172.21.2.102:0/3700538077 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f26240c1d80 msgr2=0x7f26240c2180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:37.148 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:37.145 7f26237fe700 1 --2- 172.21.2.102:0/3700538077 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f26240c1d80 0x7f26240c2180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:37.148 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:37.145 7f26237fe700 1 -- 172.21.2.102:0/3700538077 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f26240b8200 msgr2=0x7f26240b8600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:37.148 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:37.145 7f26237fe700 1 --2- 172.21.2.102:0/3700538077 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f26240b8200 0x7f26240b8600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:37.149 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:37.145 7f26237fe700 1 -- 172.21.2.102:0/3700538077 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26240c27d0 con 0x7f2624190920 2021-02-10T11:03:37.149 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:37.145 7f2623fff700 1 --2- 172.21.2.102:0/3700538077 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f26240c1d80 0x7f26240c2180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:03:37.149 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:37.145 7f26237fe700 1 --2- 172.21.2.102:0/3700538077 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2624190920 0x7f26240b7be0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7f2614006f60 tx=0x7f261400b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:03:37.149 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:37.145 7f261bfff700 1 -- 172.21.2.102:0/3700538077 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f261400bd60 con 0x7f2624190920 2021-02-10T11:03:37.149 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:37.145 7f261bfff700 1 -- 172.21.2.102:0/3700538077 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2614000c90 con 0x7f2624190920 2021-02-10T11:03:37.149 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:37.145 7f2629ff6700 1 -- 172.21.2.102:0/3700538077 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26240c27d0 con 0x7f2624190920 2021-02-10T11:03:37.149 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:37.145 7f261bfff700 1 -- 172.21.2.102:0/3700538077 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f26140022b0 con 0x7f2624190920 2021-02-10T11:03:37.149 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:37.145 7f2629ff6700 1 -- 172.21.2.102:0/3700538077 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26240c2d60 con 0x7f2624190920 2021-02-10T11:03:37.149 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:37.146 7f261bfff700 1 -- 172.21.2.102:0/3700538077 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f2614006b00 con 0x7f2624190920 2021-02-10T11:03:37.149 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:37.146 7f261bfff700 1 --2- 172.21.2.102:0/3700538077 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f26080186c0 0x7f260801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:37.149 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:37.146 7f261bfff700 1 -- 172.21.2.102:0/3700538077 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(13..13 src has 1..13) v4 ==== 4400+0+0 (secure 0 0 0) 0x7f2614031f40 con 0x7f2624190920 2021-02-10T11:03:37.149 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:37.146 7f2622ffd700 1 --2- 172.21.2.102:0/3700538077 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f26080186c0 0x7f260801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:37.149 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:37.146 7f2622ffd700 1 --2- 172.21.2.102:0/3700538077 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f26080186c0 0x7f260801abc0 crc :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:03:37.176 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:37.177 7f2629ff6700 1 -- 172.21.2.102:0/3700538077 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2624105ae0 con 0x7f2624190920 2021-02-10T11:03:37.179 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:37.180 7f261bfff700 1 -- 172.21.2.102:0/3700538077 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f2614000960 con 0x7f2624190920 2021-02-10T11:03:37.305 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:37.305 7f2629ff6700 1 -- 172.21.2.102:0/3700538077 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "rbd"} v 0) v1 -- 0x7f26240f0b50 con 0x7f2624190920 2021-02-10T11:03:37.732 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:37.733 7f261bfff700 1 -- 172.21.2.102:0/3700538077 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "rbd"}]=0 pool 'rbd' created v14) v1 ==== 109+0+0 (secure 0 0 0) 0x7f2614006b00 con 0x7f2624190920 2021-02-10T11:03:37.771 INFO:teuthology.orchestra.run.gibba002.stderr:pool 'rbd' created 2021-02-10T11:03:37.773 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:37.774 7f2629ff6700 1 -- 172.21.2.102:0/3700538077 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f26080186c0 msgr2=0x7f260801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:37.773 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:37.774 7f2629ff6700 1 --2- 172.21.2.102:0/3700538077 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f26080186c0 0x7f260801abc0 crc :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:37.774 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:37.774 7f2629ff6700 1 -- 172.21.2.102:0/3700538077 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2624190920 msgr2=0x7f26240b7be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:37.774 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:37.774 7f2629ff6700 1 --2- 172.21.2.102:0/3700538077 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2624190920 0x7f26240b7be0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7f2614006f60 tx=0x7f261400b970).stop 2021-02-10T11:03:37.774 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:37.774 7f2629ff6700 1 -- 172.21.2.102:0/3700538077 shutdown_connections 2021-02-10T11:03:37.774 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:37.774 7f2629ff6700 1 --2- 172.21.2.102:0/3700538077 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f26240c1d80 0x7f26240c2180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:37.774 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:37.774 7f2629ff6700 1 --2- 172.21.2.102:0/3700538077 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f26080186c0 0x7f260801abc0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:37.774 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:37.774 7f2629ff6700 1 --2- 172.21.2.102:0/3700538077 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f26240b8200 0x7f26240b8600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:37.774 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:37.774 7f2629ff6700 1 --2- 172.21.2.102:0/3700538077 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2624190920 0x7f26240b7be0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:37.774 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:37.774 7f2629ff6700 1 -- 172.21.2.102:0/3700538077 >> 172.21.2.102:0/3700538077 conn(0x7f2624094e10 msgr2=0x7f26240a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:03:37.775 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:37.774 7f2629ff6700 1 -- 172.21.2.102:0/3700538077 shutdown_connections 2021-02-10T11:03:37.775 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:37.775 7f2629ff6700 1 -- 172.21.2.102:0/3700538077 wait complete. 2021-02-10T11:03:37.781 DEBUG:teuthology.orchestra.run.gibba002:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2021-02-10T11:03:38.211 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:38.194 7feee4ca1700 1 Processor -- start 2021-02-10T11:03:38.211 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:38.194 7feee4ca1700 1 -- start start 2021-02-10T11:03:38.211 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:38.194 7feee4ca1700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7feee00d1860 0x7feee01b03e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:38.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:38.194 7feee4ca1700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7feee00f0640 0x7feee00cfb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:38.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:38.194 7feee4ca1700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feee00f0b00 con 0x7feee01b0a00 2021-02-10T11:03:38.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:38.194 7feee4ca1700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feee00fe220 con 0x7feee00f0640 2021-02-10T11:03:38.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:38.194 7feee4ca1700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feee00e5ec0 con 0x7feee00d1860 2021-02-10T11:03:38.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:38.194 7feede05d700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7feee01b0a00 0x7feee00cf320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:41264/0 (socket says 172.21.2.102:41264) 2021-02-10T11:03:38.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:38.194 7feede05d700 1 -- 172.21.2.102:0/3293873132 learned_addr learned my addr 172.21.2.102:0/3293873132 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:03:38.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:38.195 7feedd85c700 1 -- 172.21.2.102:0/3293873132 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2777624703 0 0) 0x7feec80010b0 con 0x7feee01b0a00 2021-02-10T11:03:38.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:38.195 7feedd85c700 1 -- 172.21.2.102:0/3293873132 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feecc002fe0 con 0x7feee01b0a00 2021-02-10T11:03:38.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:38.195 7feede85e700 1 --2- 172.21.2.102:0/3293873132 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7feee00d1860 0x7feee01b03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:38.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:38.195 7feedf05f700 1 --2- 172.21.2.102:0/3293873132 >> v2:172.21.2.113:3300/0 conn(0x7feee00f0640 0x7feee00cfb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:38.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:38.195 7feedd85c700 1 -- 172.21.2.102:0/3293873132 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2974053352 0 0) 0x7feec80010b0 con 0x7feee01b0a00 2021-02-10T11:03:38.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:38.195 7feedd85c700 1 -- 172.21.2.102:0/3293873132 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7feee00d1860 msgr2=0x7feee01b03e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:03:38.213 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:38.195 7feedd85c700 1 --2- 172.21.2.102:0/3293873132 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7feee00d1860 0x7feee01b03e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:38.213 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:38.195 7feedd85c700 1 -- 172.21.2.102:0/3293873132 >> v2:172.21.2.113:3300/0 conn(0x7feee00f0640 msgr2=0x7feee00cfb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:03:38.213 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:38.195 7feedd85c700 1 --2- 172.21.2.102:0/3293873132 >> v2:172.21.2.113:3300/0 conn(0x7feee00f0640 0x7feee00cfb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:38.213 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:38.195 7feedd85c700 1 -- 172.21.2.102:0/3293873132 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feee00f2090 con 0x7feee01b0a00 2021-02-10T11:03:38.213 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:38.195 7feedd85c700 1 -- 172.21.2.102:0/3293873132 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7feec80015b0 con 0x7feee01b0a00 2021-02-10T11:03:38.213 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:38.195 7feedd85c700 1 -- 172.21.2.102:0/3293873132 >> v1:172.21.2.102:6789/0 conn(0x7feee01b0a00 legacy=0x7feee00cf320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:38.213 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:38.195 7feedd85c700 1 --2- 172.21.2.102:0/3293873132 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7feecc003ac0 0x7feecc005ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:38.213 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:38.195 7feedd85c700 1 --2- 172.21.2.102:0/3293873132 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7feecc006790 0x7feecc008c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:38.213 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:38.195 7feedd85c700 1 --2- 172.21.2.102:0/3293873132 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7feecc009500 0x7feecc00ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:38.213 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:38.195 7feedd85c700 1 -- 172.21.2.102:0/3293873132 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feecc00c2d0 con 0x7feecc003ac0 2021-02-10T11:03:38.213 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:38.195 7feedd85c700 1 -- 172.21.2.102:0/3293873132 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feecc0504c0 con 0x7feecc009500 2021-02-10T11:03:38.213 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:38.195 7feedd85c700 1 -- 172.21.2.102:0/3293873132 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feecc0506e0 con 0x7feecc006790 2021-02-10T11:03:38.213 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:38.195 7feede85e700 1 --2- 172.21.2.102:0/3293873132 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7feecc003ac0 0x7feecc005ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:38.214 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:38.195 7feedf05f700 1 --2- 172.21.2.102:0/3293873132 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7feecc009500 0x7feecc00ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:38.214 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:38.196 7feede05d700 1 --2- 172.21.2.102:0/3293873132 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7feecc006790 0x7feecc008c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:38.214 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:38.196 7feede85e700 1 -- 172.21.2.102:0/3293873132 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7feecc006790 msgr2=0x7feecc008c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:03:38.214 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:38.196 7feede85e700 1 --2- 172.21.2.102:0/3293873132 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7feecc006790 0x7feecc008c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:38.214 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:38.196 7feede85e700 1 -- 172.21.2.102:0/3293873132 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7feecc009500 msgr2=0x7feecc00ba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:03:38.214 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:38.196 7feede85e700 1 --2- 172.21.2.102:0/3293873132 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7feecc009500 0x7feecc00ba20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:38.214 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:38.196 7feede85e700 1 -- 172.21.2.102:0/3293873132 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feecc050980 con 0x7feecc003ac0 2021-02-10T11:03:38.214 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:38.196 7feede05d700 1 --2- 172.21.2.102:0/3293873132 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7feecc006790 0x7feecc008c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:03:38.214 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:38.196 7feede85e700 1 --2- 172.21.2.102:0/3293873132 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7feecc003ac0 0x7feecc005ef0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7feed0006ad0 tx=0x7feed0009b40).ready entity=mon.0 client_cookie=176b8d294e53642c server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:03:38.214 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:38.196 7feedd85c700 1 -- 172.21.2.102:0/3293873132 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7feed00066d0 con 0x7feecc003ac0 2021-02-10T11:03:38.214 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:38.196 7feedd85c700 1 -- 172.21.2.102:0/3293873132 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feed000c0c0 con 0x7feecc003ac0 2021-02-10T11:03:38.214 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:38.196 7feedd85c700 1 -- 172.21.2.102:0/3293873132 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7feed000c520 con 0x7feecc003ac0 2021-02-10T11:03:38.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:38.196 7feee4ca1700 1 -- 172.21.2.102:0/3293873132 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7feecc003ac0 msgr2=0x7feecc005ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:38.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:38.196 7feee4ca1700 1 --2- 172.21.2.102:0/3293873132 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7feecc003ac0 0x7feecc005ef0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7feed0006ad0 tx=0x7feed0009b40).stop 2021-02-10T11:03:38.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:38.196 7feedf05f700 1 -- 172.21.2.102:0/3293873132 reap_dead start 2021-02-10T11:03:38.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:38.197 7feee4ca1700 1 -- 172.21.2.102:0/3293873132 shutdown_connections 2021-02-10T11:03:38.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:38.197 7feee4ca1700 1 -- 172.21.2.102:0/3293873132 >> 172.21.2.102:0/3293873132 conn(0x7feee00ce440 msgr2=0x7feee00d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:03:38.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:38.197 7feee4ca1700 1 -- 172.21.2.102:0/3293873132 shutdown_connections 2021-02-10T11:03:38.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:38.197 7feee4ca1700 1 -- 172.21.2.102:0/3293873132 wait complete. 2021-02-10T11:03:38.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:38.212 7feee4ca1700 1 Processor -- start 2021-02-10T11:03:38.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:38.212 7feee4ca1700 1 -- start start 2021-02-10T11:03:38.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:38.212 7feee4ca1700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7feee00efc20 0x7feee00f0020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:38.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:38.212 7feee4ca1700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7feee00b8640 0x7feee00b8a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:38.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:38.212 7feee4ca1700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7feee00b9060 0x7feee008eca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:38.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:38.212 7feee4ca1700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feee008f2c0 con 0x7feee00b8640 2021-02-10T11:03:38.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:38.212 7feee4ca1700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feee008f4e0 con 0x7feee00b9060 2021-02-10T11:03:38.216 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:38.212 7feee4ca1700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feee008f700 con 0x7feee00efc20 2021-02-10T11:03:38.216 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:38.212 7feede05d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7feee00b8640 0x7feee00b8a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:38.216 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:38.212 7feede05d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7feee00b8640 0x7feee00b8a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:49050/0 (socket says 172.21.2.102:49050) 2021-02-10T11:03:38.216 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:38.212 7feede05d700 1 -- 172.21.2.102:0/230122942 learned_addr learned my addr 172.21.2.102:0/230122942 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:03:38.217 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:38.212 7feede05d700 1 -- 172.21.2.102:0/230122942 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7feee00efc20 msgr2=0x7feee00f0020 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:03:38.217 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:38.212 7feede05d700 1 --2- 172.21.2.102:0/230122942 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7feee00efc20 0x7feee00f0020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:03:38.217 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:38.213 7feede05d700 1 -- 172.21.2.102:0/230122942 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7feee00b9060 msgr2=0x7feee008eca0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:03:38.217 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:38.213 7feede05d700 1 --2- 172.21.2.102:0/230122942 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7feee00b9060 0x7feee008eca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:03:38.217 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:38.213 7feede05d700 1 -- 172.21.2.102:0/230122942 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feee008f920 con 0x7feee00b8640 2021-02-10T11:03:38.217 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:38.213 7feede05d700 1 --2- 172.21.2.102:0/230122942 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7feee00b8640 0x7feee00b8a40 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7feec8007650 tx=0x7feec800a880).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:03:38.217 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:38.213 7feed77fe700 1 -- 172.21.2.102:0/230122942 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7feec800ac70 con 0x7feee00b8640 2021-02-10T11:03:38.217 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:38.213 7feed77fe700 1 -- 172.21.2.102:0/230122942 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feec8004350 con 0x7feee00b8640 2021-02-10T11:03:38.217 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:38.213 7feee4ca1700 1 -- 172.21.2.102:0/230122942 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feee008f920 con 0x7feee00b8640 2021-02-10T11:03:38.217 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:38.213 7feed77fe700 1 -- 172.21.2.102:0/230122942 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7feec8004870 con 0x7feee00b8640 2021-02-10T11:03:38.218 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:38.213 7feee4ca1700 1 -- 172.21.2.102:0/230122942 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feee008feb0 con 0x7feee00b8640 2021-02-10T11:03:38.218 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:38.214 7feed77fe700 1 -- 172.21.2.102:0/230122942 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7feec80018c0 con 0x7feee00b8640 2021-02-10T11:03:38.218 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:38.214 7feed77fe700 1 --2- 172.21.2.102:0/230122942 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7feec4018730 0x7feec401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:38.218 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:38.214 7feed77fe700 1 -- 172.21.2.102:0/230122942 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 4742+0+0 (secure 0 0 0) 0x7feec8032470 con 0x7feee00b8640 2021-02-10T11:03:38.218 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:38.214 7feede85e700 1 --2- 172.21.2.102:0/230122942 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7feec4018730 0x7feec401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:38.218 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:38.214 7feede85e700 1 --2- 172.21.2.102:0/230122942 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7feec4018730 0x7feec401abc0 crc :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:03:38.243 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:38.244 7feee4ca1700 1 -- 172.21.2.102:0/230122942 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feee00ebc30 con 0x7feee00b8640 2021-02-10T11:03:38.246 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:38.247 7feed77fe700 1 -- 172.21.2.102:0/230122942 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7feec8000940 con 0x7feee00b8640 2021-02-10T11:03:38.370 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:38.371 7feee4ca1700 1 -- 172.21.2.102:0/230122942 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd pool application enable", "app": "rbd", "pool": "rbd", "yes_i_really_mean_it": true} v 0) v1 -- 0x7feee00f0b00 con 0x7feee00b8640 2021-02-10T11:03:38.735 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:38.735 7feed77fe700 1 -- 172.21.2.102:0/230122942 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool application enable", "app": "rbd", "pool": "rbd", "yes_i_really_mean_it": true}]=0 enabled application 'rbd' on pool 'rbd' v15) v1 ==== 173+0+0 (secure 0 0 0) 0x7feec80018c0 con 0x7feee00b8640 2021-02-10T11:03:38.737 INFO:teuthology.orchestra.run.gibba002.stderr:enabled application 'rbd' on pool 'rbd' 2021-02-10T11:03:38.739 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:38.740 7feee4ca1700 1 -- 172.21.2.102:0/230122942 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7feec4018730 msgr2=0x7feec401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:38.740 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:38.740 7feee4ca1700 1 --2- 172.21.2.102:0/230122942 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7feec4018730 0x7feec401abc0 crc :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:38.740 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:38.740 7feee4ca1700 1 -- 172.21.2.102:0/230122942 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7feee00b8640 msgr2=0x7feee00b8a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:38.740 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:38.740 7feee4ca1700 1 --2- 172.21.2.102:0/230122942 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7feee00b8640 0x7feee00b8a40 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7feec8007650 tx=0x7feec800a880).stop 2021-02-10T11:03:38.740 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:38.740 7feee4ca1700 1 -- 172.21.2.102:0/230122942 shutdown_connections 2021-02-10T11:03:38.740 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:38.740 7feee4ca1700 1 --2- 172.21.2.102:0/230122942 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7feec4018730 0x7feec401abc0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:38.740 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:38.740 7feee4ca1700 1 --2- 172.21.2.102:0/230122942 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7feee00b9060 0x7feee008eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:03:38.740 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:38.740 7feee4ca1700 1 --2- 172.21.2.102:0/230122942 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7feee00b8640 0x7feee00b8a40 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:38.740 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:38.740 7feee4ca1700 1 --2- 172.21.2.102:0/230122942 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7feee00efc20 0x7feee00f0020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:03:38.740 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:38.740 7feee4ca1700 1 -- 172.21.2.102:0/230122942 >> 172.21.2.102:0/230122942 conn(0x7feee009cc00 msgr2=0x7feee00c3cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:03:38.741 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:38.740 7feee4ca1700 1 -- 172.21.2.102:0/230122942 shutdown_connections 2021-02-10T11:03:38.741 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:38.740 7feee4ca1700 1 -- 172.21.2.102:0/230122942 wait complete. 2021-02-10T11:03:38.747 INFO:tasks.ceph:Setting up CephFS filesystem... 2021-02-10T11:03:38.748 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:03:39.172 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:39.156 7f72679a6700 1 Processor -- start 2021-02-10T11:03:39.172 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:39.156 7f72679a6700 1 -- start start 2021-02-10T11:03:39.172 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:39.157 7f72679a6700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f72600e5070 0x7f72601b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:39.173 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:39.157 7f72679a6700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f72601b3e40 0x7f726007d690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:39.173 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:39.157 7f72679a6700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f726007ed80 con 0x7f72600e5470 2021-02-10T11:03:39.173 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:39.157 7f72679a6700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7260080310 con 0x7f72600e5070 2021-02-10T11:03:39.173 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:39.157 7f72679a6700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7260080530 con 0x7f72601b3e40 2021-02-10T11:03:39.173 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:39.157 7f72657c2700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f72600e5070 0x7f72601b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:39.173 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:39.157 7f72657c2700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f72600e5070 0x7f72601b3820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:49058/0 (socket says 172.21.2.102:49058) 2021-02-10T11:03:39.173 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:39.157 7f72657c2700 1 -- 172.21.2.102:0/4067698021 learned_addr learned my addr 172.21.2.102:0/4067698021 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:03:39.173 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:39.157 7f72657c2700 1 -- 172.21.2.102:0/4067698021 >> v2:172.21.2.113:3300/0 conn(0x7f72601b3e40 msgr2=0x7f726007d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:03:39.173 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:39.157 7f7264fc1700 1 --2- 172.21.2.102:0/4067698021 >> v2:172.21.2.113:3300/0 conn(0x7f72601b3e40 0x7f726007d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:39.173 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:39.157 7f72657c2700 1 --2- 172.21.2.102:0/4067698021 >> v2:172.21.2.113:3300/0 conn(0x7f72601b3e40 0x7f726007d690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:39.173 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:39.157 7f72657c2700 1 -- 172.21.2.102:0/4067698021 >> v1:172.21.2.113:6789/0 conn(0x7f72600e5470 legacy=0x7f726007dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:03:39.173 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:39.158 7f72657c2700 1 -- 172.21.2.102:0/4067698021 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7260080750 con 0x7f72600e5070 2021-02-10T11:03:39.173 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:39.158 7f7264fc1700 1 --2- 172.21.2.102:0/4067698021 >> v2:172.21.2.113:3300/0 conn(0x7f72601b3e40 0x7f726007d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:03:39.174 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:39.158 7f72657c2700 1 --2- 172.21.2.102:0/4067698021 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f72600e5070 0x7f72601b3820 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f7254006700 tx=0x7f7254009b80).ready entity=mon.0 client_cookie=ca8d1454be327e2e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:03:39.174 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:39.158 7f7253fff700 1 -- 172.21.2.102:0/4067698021 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f725400c0c0 con 0x7f72600e5070 2021-02-10T11:03:39.174 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:39.158 7f7253fff700 1 -- 172.21.2.102:0/4067698021 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f725400c400 con 0x7f72600e5070 2021-02-10T11:03:39.174 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:39.158 7f7253fff700 1 -- 172.21.2.102:0/4067698021 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f725400c920 con 0x7f72600e5070 2021-02-10T11:03:39.174 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:39.158 7f72679a6700 1 -- 172.21.2.102:0/4067698021 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f72600e5070 msgr2=0x7f72601b3820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:39.174 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:39.158 7f72679a6700 1 --2- 172.21.2.102:0/4067698021 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f72600e5070 0x7f72601b3820 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f7254006700 tx=0x7f7254009b80).stop 2021-02-10T11:03:39.174 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:39.159 7f72679a6700 1 -- 172.21.2.102:0/4067698021 shutdown_connections 2021-02-10T11:03:39.174 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:39.159 7f72679a6700 1 --2- 172.21.2.102:0/4067698021 >> v2:172.21.2.113:3300/0 conn(0x7f72601b3e40 0x7f726007d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:39.174 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:39.159 7f72679a6700 1 --2- 172.21.2.102:0/4067698021 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f72600e5070 0x7f72601b3820 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:39.175 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:39.159 7f72679a6700 1 -- 172.21.2.102:0/4067698021 >> 172.21.2.102:0/4067698021 conn(0x7f72600ce690 msgr2=0x7f72600e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:03:39.175 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:39.159 7f72679a6700 1 -- 172.21.2.102:0/4067698021 shutdown_connections 2021-02-10T11:03:39.175 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:39.159 7f72679a6700 1 -- 172.21.2.102:0/4067698021 wait complete. 2021-02-10T11:03:39.176 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:39.174 7f72679a6700 1 Processor -- start 2021-02-10T11:03:39.176 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:39.174 7f72679a6700 1 -- start start 2021-02-10T11:03:39.176 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:39.174 7f72679a6700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f72600ae8c0 0x7f7260096c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:39.176 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:39.174 7f72679a6700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7260081aa0 0x7f7260187eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:39.176 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:39.174 7f72679a6700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7260081ea0 0x7f72601884d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:39.177 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:39.174 7f72679a6700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72600815a0 con 0x7f7260081ea0 2021-02-10T11:03:39.177 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:39.174 7f72679a6700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7260097290 con 0x7f7260081aa0 2021-02-10T11:03:39.177 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:39.174 7f72679a6700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7260081230 con 0x7f72600ae8c0 2021-02-10T11:03:39.177 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:39.174 7f7265fc3700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7260081ea0 0x7f72601884d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:39.177 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:39.174 7f7265fc3700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7260081ea0 0x7f72601884d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:49068/0 (socket says 172.21.2.102:49068) 2021-02-10T11:03:39.177 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:39.174 7f7265fc3700 1 -- 172.21.2.102:0/1417594508 learned_addr learned my addr 172.21.2.102:0/1417594508 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:03:39.177 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:39.175 7f7265fc3700 1 -- 172.21.2.102:0/1417594508 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f72600ae8c0 msgr2=0x7f7260096c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:39.177 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:39.175 7f7265fc3700 1 --2- 172.21.2.102:0/1417594508 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f72600ae8c0 0x7f7260096c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:03:39.177 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:39.175 7f7265fc3700 1 -- 172.21.2.102:0/1417594508 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7260081aa0 msgr2=0x7f7260187eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:39.177 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:39.175 7f7265fc3700 1 --2- 172.21.2.102:0/1417594508 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7260081aa0 0x7f7260187eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:03:39.177 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:39.175 7f7265fc3700 1 -- 172.21.2.102:0/1417594508 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f726008da00 con 0x7f7260081ea0 2021-02-10T11:03:39.177 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:39.175 7f7265fc3700 1 --2- 172.21.2.102:0/1417594508 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7260081ea0 0x7f72601884d0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f7258006ed0 tx=0x7f725800b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:03:39.177 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:39.175 7f7251ffb700 1 -- 172.21.2.102:0/1417594508 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f725800bd50 con 0x7f7260081ea0 2021-02-10T11:03:39.178 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:39.175 7f7251ffb700 1 -- 172.21.2.102:0/1417594508 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72580009b0 con 0x7f7260081ea0 2021-02-10T11:03:39.178 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:39.175 7f72679a6700 1 -- 172.21.2.102:0/1417594508 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f726008da00 con 0x7f7260081ea0 2021-02-10T11:03:39.178 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:39.175 7f7251ffb700 1 -- 172.21.2.102:0/1417594508 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f72580062a0 con 0x7f7260081ea0 2021-02-10T11:03:39.178 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:39.175 7f72679a6700 1 -- 172.21.2.102:0/1417594508 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f726008ced0 con 0x7f7260081ea0 2021-02-10T11:03:39.178 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:39.176 7f7251ffb700 1 -- 172.21.2.102:0/1417594508 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f7258006020 con 0x7f7260081ea0 2021-02-10T11:03:39.178 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:39.176 7f7251ffb700 1 --2- 172.21.2.102:0/1417594508 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f72400186c0 0x7f724001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:39.178 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:39.176 7f7251ffb700 1 -- 172.21.2.102:0/1417594508 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4753+0+0 (secure 0 0 0) 0x7f72580322e0 con 0x7f7260081ea0 2021-02-10T11:03:39.178 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:39.176 7f72657c2700 1 --2- 172.21.2.102:0/1417594508 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f72400186c0 0x7f724001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:39.178 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:39.176 7f72657c2700 1 --2- 172.21.2.102:0/1417594508 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f72400186c0 0x7f724001abc0 crc :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:03:39.206 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:39.207 7f72679a6700 1 -- 172.21.2.102:0/1417594508 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f726007ed80 con 0x7f7260081ea0 2021-02-10T11:03:39.209 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:39.210 7f7251ffb700 1 -- 172.21.2.102:0/1417594508 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f7258000bc0 con 0x7f7260081ea0 2021-02-10T11:03:39.334 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:39.335 7f72679a6700 1 -- 172.21.2.102:0/1417594508 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7260081520 con 0x7f7260081ea0 2021-02-10T11:03:39.334 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:39.335 7f7251ffb700 1 -- 172.21.2.102:0/1417594508 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v15) v1 ==== 74+0+11155 (secure 0 0 0) 0x7f7258000bc0 con 0x7f7260081ea0 2021-02-10T11:03:39.335 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:03:39.335 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":15,"fsid":"68db30ea-dbce-4aef-a6df-f01c6fea0513","created":"2021-02-10 11:03:06.331557","modified":"2021-02-10 11:03:38.733550","last_up_change":"2021-02-10 11:03:32.690562","last_in_change":"2021-02-10 11:03:32.690562","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:03:37.322519","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"be691b4d-a039-408a-af80-9d53f02b6f1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6818","nonce":36088},{"type":"v1","addr":"172.21.2.102:6820","nonce":36088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6822","nonce":36088},{"type":"v1","addr":"172.21.2.102:6824","nonce":36088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6830","nonce":36088},{"type":"v1","addr":"172.21.2.102:6832","nonce":36088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6826","nonce":36088},{"type":"v1","addr":"172.21.2.102:6828","nonce":36088}]},"public_addr":"172.21.2.102:6820/36088","cluster_addr":"172.21.2.102:6824/36088","heartbeat_back_addr":"172.21.2.102:6832/36088","heartbeat_front_addr":"172.21.2.102:6828/36088","state":["exists","up"]},{"osd":1,"uuid":"64e3ccc8-9c69-4d7c-865e-af8e83822635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6819","nonce":36087},{"type":"v1","addr":"172.21.2.102:6821","nonce":36087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6823","nonce":36087},{"type":"v1","addr":"172.21.2.102:6825","nonce":36087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6831","nonce":36087},{"type":"v1","addr":"172.21.2.102:6833","nonce":36087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6827","nonce":36087},{"type":"v1","addr":"172.21.2.102:6829","nonce":36087}]},"public_addr":"172.21.2.102:6821/36087","cluster_addr":"172.21.2.102:6825/36087","heartbeat_back_addr":"172.21.2.102:6833/36087","heartbeat_front_addr":"172.21.2.102:6829/36087","state":["exists","up"]},{"osd":2,"uuid":"153700af-747f-481f-bdf4-d6072d3a842f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6810","nonce":36089},{"type":"v1","addr":"172.21.2.102:6811","nonce":36089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6812","nonce":36089},{"type":"v1","addr":"172.21.2.102:6813","nonce":36089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6816","nonce":36089},{"type":"v1","addr":"172.21.2.102:6817","nonce":36089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6814","nonce":36089},{"type":"v1","addr":"172.21.2.102:6815","nonce":36089}]},"public_addr":"172.21.2.102:6811/36089","cluster_addr":"172.21.2.102:6813/36089","heartbeat_back_addr":"172.21.2.102:6817/36089","heartbeat_front_addr":"172.21.2.102:6815/36089","state":["exists","up"]},{"osd":3,"uuid":"cbc678f4-83a9-45e6-8830-2f7c232b1ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6802","nonce":36090},{"type":"v1","addr":"172.21.2.102:6803","nonce":36090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6804","nonce":36090},{"type":"v1","addr":"172.21.2.102:6805","nonce":36090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6808","nonce":36090},{"type":"v1","addr":"172.21.2.102:6809","nonce":36090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6806","nonce":36090},{"type":"v1","addr":"172.21.2.102:6807","nonce":36090}]},"public_addr":"172.21.2.102:6803/36090","cluster_addr":"172.21.2.102:6805/36090","heartbeat_back_addr":"172.21.2.102:6809/36090","heartbeat_front_addr":"172.21.2.102:6807/36090","state":["exists","up"]},{"osd":4,"uuid":"c01e4008-b258-4f10-afd2-9e7f37214351","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6808","nonce":36368},{"type":"v1","addr":"172.21.2.113:6811","nonce":36368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6814","nonce":36368},{"type":"v1","addr":"172.21.2.113:6818","nonce":36368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6826","nonce":36368},{"type":"v1","addr":"172.21.2.113:6828","nonce":36368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6820","nonce":36368},{"type":"v1","addr":"172.21.2.113:6824","nonce":36368}]},"public_addr":"172.21.2.113:6811/36368","cluster_addr":"172.21.2.113:6818/36368","heartbeat_back_addr":"172.21.2.113:6828/36368","heartbeat_front_addr":"172.21.2.113:6824/36368","state":["exists","up"]},{"osd":5,"uuid":"551f6007-daf8-4a98-8ee0-c2b16bcccb5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6802","nonce":36366},{"type":"v1","addr":"172.21.2.113:6805","nonce":36366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6807","nonce":36366},{"type":"v1","addr":"172.21.2.113:6810","nonce":36366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6819","nonce":36366},{"type":"v1","addr":"172.21.2.113:6822","nonce":36366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6812","nonce":36366},{"type":"v1","addr":"172.21.2.113:6815","nonce":36366}]},"public_addr":"172.21.2.113:6805/36366","cluster_addr":"172.21.2.113:6810/36366","heartbeat_back_addr":"172.21.2.113:6822/36366","heartbeat_front_addr":"172.21.2.113:6815/36366","state":["exists","up"]},{"osd":6,"uuid":"bf67c040-0e50-48be-b102-c3943bc9ebef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6800","nonce":36367},{"type":"v1","addr":"172.21.2.113:6801","nonce":36367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6803","nonce":36367},{"type":"v1","addr":"172.21.2.113:6804","nonce":36367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6813","nonce":36367},{"type":"v1","addr":"172.21.2.113:6816","nonce":36367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6806","nonce":36367},{"type":"v1","addr":"172.21.2.113:6809","nonce":36367}]},"public_addr":"172.21.2.113:6801/36367","cluster_addr":"172.21.2.113:6804/36367","heartbeat_back_addr":"172.21.2.113:6816/36367","heartbeat_front_addr":"172.21.2.113:6809/36367","state":["exists","up"]},{"osd":7,"uuid":"f5461f80-018d-428f-be89-2eb2bcbe7bb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6817","nonce":36365},{"type":"v1","addr":"172.21.2.113:6821","nonce":36365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6823","nonce":36365},{"type":"v1","addr":"172.21.2.113:6825","nonce":36365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6830","nonce":36365},{"type":"v1","addr":"172.21.2.113:6831","nonce":36365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6827","nonce":36365},{"type":"v1","addr":"172.21.2.113:6829","nonce":36365}]},"public_addr":"172.21.2.113:6821/36365","cluster_addr":"172.21.2.113:6825/36365","heartbeat_back_addr":"172.21.2.113:6831/36365","heartbeat_front_addr":"172.21.2.113:6829/36365","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:03:39.336 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:39.337 7f72679a6700 1 -- 172.21.2.102:0/1417594508 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f72400186c0 msgr2=0x7f724001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:39.336 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:39.337 7f72679a6700 1 --2- 172.21.2.102:0/1417594508 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f72400186c0 0x7f724001abc0 crc :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:39.336 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:39.337 7f72679a6700 1 -- 172.21.2.102:0/1417594508 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7260081ea0 msgr2=0x7f72601884d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:39.336 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:39.337 7f72679a6700 1 --2- 172.21.2.102:0/1417594508 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7260081ea0 0x7f72601884d0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f7258006ed0 tx=0x7f725800b960).stop 2021-02-10T11:03:39.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:39.337 7f72679a6700 1 -- 172.21.2.102:0/1417594508 shutdown_connections 2021-02-10T11:03:39.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:39.337 7f72679a6700 1 --2- 172.21.2.102:0/1417594508 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7260081ea0 0x7f72601884d0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:39.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:39.337 7f72679a6700 1 --2- 172.21.2.102:0/1417594508 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f72400186c0 0x7f724001abc0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:39.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:39.337 7f72679a6700 1 --2- 172.21.2.102:0/1417594508 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7260081aa0 0x7f7260187eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:03:39.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:39.337 7f72679a6700 1 --2- 172.21.2.102:0/1417594508 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f72600ae8c0 0x7f7260096c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:03:39.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:39.337 7f72679a6700 1 -- 172.21.2.102:0/1417594508 >> 172.21.2.102:0/1417594508 conn(0x7f726019b930 msgr2=0x7f72601a0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:03:39.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:39.338 7f72679a6700 1 -- 172.21.2.102:0/1417594508 shutdown_connections 2021-02-10T11:03:39.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:39.338 7f72679a6700 1 -- 172.21.2.102:0/1417594508 wait complete. 2021-02-10T11:03:39.344 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:03:37.322519', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2021-02-10T11:03:39.345 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:03:39.775 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:39.759 7f06fe867700 1 Processor -- start 2021-02-10T11:03:39.775 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:39.759 7f06fe867700 1 -- start start 2021-02-10T11:03:39.775 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:39.759 7f06fe867700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f06f81b0470 0x7f06f81b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:39.775 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:39.759 7f06fe867700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f06f80f20e0 0x7f06f80f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:39.775 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:39.759 7f06fe867700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06f80f0b50 con 0x7f06f80ef610 2021-02-10T11:03:39.775 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:39.759 7f06fe867700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06f80f0820 con 0x7f06f81b0470 2021-02-10T11:03:39.775 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:39.759 7f06fe867700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06f80f24e0 con 0x7f06f80f20e0 2021-02-10T11:03:39.776 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:39.759 7f06f77fe700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f06f80ef610 0x7f06f80efa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:41296/0 (socket says 172.21.2.102:41296) 2021-02-10T11:03:39.776 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:39.759 7f06f77fe700 1 -- 172.21.2.102:0/1290193426 learned_addr learned my addr 172.21.2.102:0/1290193426 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:03:39.776 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:39.759 7f06f7fff700 1 --2- 172.21.2.102:0/1290193426 >> v2:172.21.2.102:3300/0 conn(0x7f06f81b0470 0x7f06f81b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:39.776 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:39.759 7f06fce84700 1 --2- 172.21.2.102:0/1290193426 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f06f80f20e0 0x7f06f80f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:39.776 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:39.760 7f06f7fff700 1 -- 172.21.2.102:0/1290193426 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f06f80f20e0 msgr2=0x7f06f80f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:03:39.776 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:39.760 7f06f7fff700 1 --2- 172.21.2.102:0/1290193426 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f06f80f20e0 0x7f06f80f0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:39.776 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:39.760 7f06f7fff700 1 -- 172.21.2.102:0/1290193426 >> v1:172.21.2.102:6789/0 conn(0x7f06f80ef610 legacy=0x7f06f80efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:39.776 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:39.760 7f06f6ffd700 1 -- 172.21.2.102:0/1290193426 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1663351391 0 0) 0x7f06ec000ce0 con 0x7f06f80ef610 2021-02-10T11:03:39.776 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:39.760 7f06f7fff700 1 -- 172.21.2.102:0/1290193426 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06f80e52e0 con 0x7f06f81b0470 2021-02-10T11:03:39.776 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:39.760 7f06f7fff700 1 --2- 172.21.2.102:0/1290193426 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f06f81b0470 0x7f06f81b0870 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f06e8006650 tx=0x7f06e8009a80).ready entity=mon.0 client_cookie=9d4be666ced9bbaa server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:03:39.776 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:39.760 7f06f6ffd700 1 -- 172.21.2.102:0/1290193426 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f06e80009b0 con 0x7f06f81b0470 2021-02-10T11:03:39.776 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:39.760 7f06f6ffd700 1 -- 172.21.2.102:0/1290193426 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06e800c0c0 con 0x7f06f81b0470 2021-02-10T11:03:39.776 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:39.760 7f06f6ffd700 1 -- 172.21.2.102:0/1290193426 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f06e800c5e0 con 0x7f06f81b0470 2021-02-10T11:03:39.777 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:39.760 7f06fe867700 1 -- 172.21.2.102:0/1290193426 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f06f81b0470 msgr2=0x7f06f81b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:39.777 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:39.760 7f06fe867700 1 --2- 172.21.2.102:0/1290193426 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f06f81b0470 0x7f06f81b0870 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f06e8006650 tx=0x7f06e8009a80).stop 2021-02-10T11:03:39.777 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:39.761 7f06fe867700 1 -- 172.21.2.102:0/1290193426 shutdown_connections 2021-02-10T11:03:39.777 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:39.761 7f06fe867700 1 --2- 172.21.2.102:0/1290193426 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f06f80f20e0 0x7f06f80f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:39.777 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:39.761 7f06fe867700 1 --2- 172.21.2.102:0/1290193426 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f06f81b0470 0x7f06f81b0870 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:39.777 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:39.761 7f06fe867700 1 -- 172.21.2.102:0/1290193426 >> 172.21.2.102:0/1290193426 conn(0x7f06f80ce3e0 msgr2=0x7f06f80e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:03:39.777 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:39.761 7f06fe867700 1 -- 172.21.2.102:0/1290193426 shutdown_connections 2021-02-10T11:03:39.777 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:39.761 7f06fe867700 1 -- 172.21.2.102:0/1290193426 wait complete. 2021-02-10T11:03:39.779 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:39.776 7f06fe867700 1 Processor -- start 2021-02-10T11:03:39.779 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:39.776 7f06fe867700 1 -- start start 2021-02-10T11:03:39.779 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:39.777 7f06fe867700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f06f80c28d0 0x7f06f80c2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:39.779 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:39.777 7f06fe867700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f06f808e690 0x7f06f808ea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:39.779 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:39.777 7f06fe867700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f06f80bc380 0x7f06f80bc780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:39.779 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:39.777 7f06fe867700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06f808f340 con 0x7f06f80bc380 2021-02-10T11:03:39.779 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:39.777 7f06fe867700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06f808f0b0 con 0x7f06f80c28d0 2021-02-10T11:03:39.779 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:39.777 7f06fe867700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06f80c3400 con 0x7f06f808e690 2021-02-10T11:03:39.779 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:39.777 7f06fce84700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f06f80bc380 0x7f06f80bc780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:39.780 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:39.777 7f06fce84700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f06f80bc380 0x7f06f80bc780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:49078/0 (socket says 172.21.2.102:49078) 2021-02-10T11:03:39.780 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:39.777 7f06fce84700 1 -- 172.21.2.102:0/1794078956 learned_addr learned my addr 172.21.2.102:0/1794078956 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:03:39.780 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:39.777 7f06f7fff700 1 --2- 172.21.2.102:0/1794078956 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f06f80c28d0 0x7f06f80c2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:39.780 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:39.777 7f06f77fe700 1 --2- 172.21.2.102:0/1794078956 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f06f808e690 0x7f06f808ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:39.780 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:39.777 7f06fce84700 1 -- 172.21.2.102:0/1794078956 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f06f808e690 msgr2=0x7f06f808ea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:39.780 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:39.777 7f06fce84700 1 --2- 172.21.2.102:0/1794078956 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f06f808e690 0x7f06f808ea90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:39.780 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:39.777 7f06fce84700 1 -- 172.21.2.102:0/1794078956 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f06f80c28d0 msgr2=0x7f06f80c2cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:39.780 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:39.777 7f06fce84700 1 --2- 172.21.2.102:0/1794078956 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f06f80c28d0 0x7f06f80c2cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:39.780 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:39.777 7f06fce84700 1 -- 172.21.2.102:0/1794078956 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06f80bcda0 con 0x7f06f80bc380 2021-02-10T11:03:39.780 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:39.777 7f06f77fe700 1 --2- 172.21.2.102:0/1794078956 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f06f808e690 0x7f06f808ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:03:39.780 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:39.777 7f06fce84700 1 --2- 172.21.2.102:0/1794078956 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f06f80bc380 0x7f06f80bc780 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f06f000b970 tx=0x7f06f000bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:03:39.780 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:39.777 7f06f4ff9700 1 -- 172.21.2.102:0/1794078956 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f06f0000f90 con 0x7f06f80bc380 2021-02-10T11:03:39.781 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:39.777 7f06f4ff9700 1 -- 172.21.2.102:0/1794078956 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06f00066f0 con 0x7f06f80bc380 2021-02-10T11:03:39.781 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:39.777 7f06fe867700 1 -- 172.21.2.102:0/1794078956 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06f80bcda0 con 0x7f06f80bc380 2021-02-10T11:03:39.781 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:39.777 7f06f4ff9700 1 -- 172.21.2.102:0/1794078956 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f06f0001680 con 0x7f06f80bc380 2021-02-10T11:03:39.781 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:39.777 7f06fe867700 1 -- 172.21.2.102:0/1794078956 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06f80bcda0 con 0x7f06f80bc380 2021-02-10T11:03:39.781 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:39.778 7f06f4ff9700 1 -- 172.21.2.102:0/1794078956 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f06f0000df0 con 0x7f06f80bc380 2021-02-10T11:03:39.781 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:39.778 7f06f4ff9700 1 --2- 172.21.2.102:0/1794078956 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f06d8018730 0x7f06d801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:39.782 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:39.778 7f06f4ff9700 1 -- 172.21.2.102:0/1794078956 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4753+0+0 (secure 0 0 0) 0x7f06f0032660 con 0x7f06f80bc380 2021-02-10T11:03:39.782 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:39.778 7f06f7fff700 1 --2- 172.21.2.102:0/1794078956 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f06d8018730 0x7f06d801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:39.782 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:39.778 7f06f7fff700 1 --2- 172.21.2.102:0/1794078956 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f06d8018730 0x7f06d801ac10 crc :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:03:39.808 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:39.809 7f06fe867700 1 -- 172.21.2.102:0/1794078956 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06f8105ae0 con 0x7f06f80bc380 2021-02-10T11:03:39.810 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:39.811 7f06f4ff9700 1 -- 172.21.2.102:0/1794078956 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f06f0000df0 con 0x7f06f80bc380 2021-02-10T11:03:39.941 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:39.942 7f06fe867700 1 -- 172.21.2.102:0/1794078956 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f06f80f0b50 con 0x7f06f80bc380 2021-02-10T11:03:39.942 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:39.942 7f06f4ff9700 1 -- 172.21.2.102:0/1794078956 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v16) v1 ==== 92+0+10 (secure 0 0 0) 0x7f06f00066f0 con 0x7f06f80bc380 2021-02-10T11:03:39.942 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:03:39.943 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:39.944 7f06fe867700 1 -- 172.21.2.102:0/1794078956 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f06d8018730 msgr2=0x7f06d801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:39.944 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:39.944 7f06fe867700 1 --2- 172.21.2.102:0/1794078956 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f06d8018730 0x7f06d801ac10 crc :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:39.944 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:39.944 7f06fe867700 1 -- 172.21.2.102:0/1794078956 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f06f80bc380 msgr2=0x7f06f80bc780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:39.944 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:39.944 7f06fe867700 1 --2- 172.21.2.102:0/1794078956 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f06f80bc380 0x7f06f80bc780 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f06f000b970 tx=0x7f06f000bd10).stop 2021-02-10T11:03:39.944 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:39.944 7f06fe867700 1 -- 172.21.2.102:0/1794078956 shutdown_connections 2021-02-10T11:03:39.944 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:39.944 7f06fe867700 1 --2- 172.21.2.102:0/1794078956 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f06f80bc380 0x7f06f80bc780 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:39.944 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:39.944 7f06fe867700 1 --2- 172.21.2.102:0/1794078956 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f06f808e690 0x7f06f808ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:39.945 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:39.944 7f06fe867700 1 --2- 172.21.2.102:0/1794078956 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f06d8018730 0x7f06d801ac10 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:39.945 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:39.944 7f06fe867700 1 --2- 172.21.2.102:0/1794078956 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f06f80c28d0 0x7f06f80c2cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:39.945 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:39.944 7f06fe867700 1 -- 172.21.2.102:0/1794078956 >> 172.21.2.102:0/1794078956 conn(0x7f06f80bbf60 msgr2=0x7f06f8105d10 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:03:39.945 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:39.944 7f06fe867700 1 -- 172.21.2.102:0/1794078956 shutdown_connections 2021-02-10T11:03:39.945 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:39.944 7f06fe867700 1 -- 172.21.2.102:0/1794078956 wait complete. 2021-02-10T11:03:39.952 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --format=json-pretty osd lspools 2021-02-10T11:03:40.382 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:40.363 7efc7665e700 1 Processor -- start 2021-02-10T11:03:40.382 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:40.363 7efc7665e700 1 -- start start 2021-02-10T11:03:40.383 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:40.364 7efc7665e700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7efc700d17f0 0x7efc701b31e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:40.383 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:40.364 7efc7665e700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7efc700d1bf0 0x7efc701b3800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:40.383 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:40.364 7efc7665e700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7efc701b3e20 0x7efc700ceef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:40.383 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:40.364 7efc7665e700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc700e5eb0 con 0x7efc700d1bf0 2021-02-10T11:03:40.383 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:40.364 7efc7665e700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc700f0b20 con 0x7efc701b3e20 2021-02-10T11:03:40.383 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:40.364 7efc7665e700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc700cf510 con 0x7efc700d17f0 2021-02-10T11:03:40.384 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:40.364 7efc6f7fe700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7efc700d1bf0 0x7efc701b3800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:40.384 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:40.364 7efc6f7fe700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7efc700d1bf0 0x7efc701b3800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:49084/0 (socket says 172.21.2.102:49084) 2021-02-10T11:03:40.384 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:40.364 7efc6f7fe700 1 -- 172.21.2.102:0/2770883036 learned_addr learned my addr 172.21.2.102:0/2770883036 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:03:40.384 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:40.364 7efc6f7fe700 1 -- 172.21.2.102:0/2770883036 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7efc700d17f0 msgr2=0x7efc701b31e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:03:40.384 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:40.364 7efc6f7fe700 1 --2- 172.21.2.102:0/2770883036 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7efc700d17f0 0x7efc701b31e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:03:40.385 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:40.364 7efc6f7fe700 1 -- 172.21.2.102:0/2770883036 >> v2:172.21.2.113:3300/0 conn(0x7efc701b3e20 msgr2=0x7efc700ceef0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:03:40.385 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:40.364 7efc6f7fe700 1 --2- 172.21.2.102:0/2770883036 >> v2:172.21.2.113:3300/0 conn(0x7efc701b3e20 0x7efc700ceef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:03:40.385 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:40.364 7efc6f7fe700 1 -- 172.21.2.102:0/2770883036 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc700cf730 con 0x7efc700d1bf0 2021-02-10T11:03:40.385 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:40.364 7efc6f7fe700 1 --2- 172.21.2.102:0/2770883036 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efc700d1bf0 0x7efc701b3800 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7efc64006700 tx=0x7efc64009b80).ready entity=mon.0 client_cookie=ae0a6c79a4884755 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:03:40.385 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:40.364 7efc6effd700 1 -- 172.21.2.102:0/2770883036 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efc6400c0c0 con 0x7efc700d1bf0 2021-02-10T11:03:40.386 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:40.365 7efc6effd700 1 -- 172.21.2.102:0/2770883036 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc6400c340 con 0x7efc700d1bf0 2021-02-10T11:03:40.386 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:40.365 7efc6effd700 1 -- 172.21.2.102:0/2770883036 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efc6400c820 con 0x7efc700d1bf0 2021-02-10T11:03:40.386 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:40.365 7efc7665e700 1 -- 172.21.2.102:0/2770883036 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efc700d1bf0 msgr2=0x7efc701b3800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:40.386 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:40.365 7efc7665e700 1 --2- 172.21.2.102:0/2770883036 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efc700d1bf0 0x7efc701b3800 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7efc64006700 tx=0x7efc64009b80).stop 2021-02-10T11:03:40.386 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:40.365 7efc7665e700 1 -- 172.21.2.102:0/2770883036 shutdown_connections 2021-02-10T11:03:40.387 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:40.365 7efc7665e700 1 --2- 172.21.2.102:0/2770883036 >> v2:172.21.2.113:3300/0 conn(0x7efc701b3e20 0x7efc700ceef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:03:40.387 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:40.365 7efc7665e700 1 --2- 172.21.2.102:0/2770883036 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efc700d1bf0 0x7efc701b3800 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:40.387 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:40.365 7efc7665e700 1 --2- 172.21.2.102:0/2770883036 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7efc700d17f0 0x7efc701b31e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:03:40.387 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:40.365 7efc7665e700 1 -- 172.21.2.102:0/2770883036 >> 172.21.2.102:0/2770883036 conn(0x7efc700fde40 msgr2=0x7efc700ce390 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:03:40.387 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:40.365 7efc7665e700 1 -- 172.21.2.102:0/2770883036 shutdown_connections 2021-02-10T11:03:40.388 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:40.365 7efc7665e700 1 -- 172.21.2.102:0/2770883036 wait complete. 2021-02-10T11:03:40.388 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:40.380 7efc7665e700 1 Processor -- start 2021-02-10T11:03:40.388 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:40.380 7efc7665e700 1 -- start start 2021-02-10T11:03:40.388 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:40.380 7efc7665e700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7efc701041f0 0x7efc70103960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:40.388 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:40.380 7efc7665e700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7efc701a9a30 0x7efc700b1960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:40.388 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:40.380 7efc7665e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efc700b1f80 0x7efc701b2fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:40.389 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:40.380 7efc7665e700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc701a9ed0 con 0x7efc700b1f80 2021-02-10T11:03:40.389 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:40.380 7efc7665e700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc701b35f0 con 0x7efc701041f0 2021-02-10T11:03:40.389 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:40.380 7efc7665e700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc701b3810 con 0x7efc701a9a30 2021-02-10T11:03:40.389 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:40.381 7efc74c7b700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efc700b1f80 0x7efc701b2fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:40.389 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:40.381 7efc74c7b700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efc700b1f80 0x7efc701b2fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:49092/0 (socket says 172.21.2.102:49092) 2021-02-10T11:03:40.390 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:40.381 7efc74c7b700 1 -- 172.21.2.102:0/3967373008 learned_addr learned my addr 172.21.2.102:0/3967373008 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:03:40.390 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:40.381 7efc74c7b700 1 -- 172.21.2.102:0/3967373008 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7efc701a9a30 msgr2=0x7efc700b1960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:40.390 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:40.381 7efc74c7b700 1 --2- 172.21.2.102:0/3967373008 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7efc701a9a30 0x7efc700b1960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:03:40.390 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:40.381 7efc74c7b700 1 -- 172.21.2.102:0/3967373008 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7efc701041f0 msgr2=0x7efc70103960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:40.390 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:40.381 7efc74c7b700 1 --2- 172.21.2.102:0/3967373008 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7efc701041f0 0x7efc70103960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:03:40.391 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:40.381 7efc74c7b700 1 -- 172.21.2.102:0/3967373008 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc701b3a30 con 0x7efc700b1f80 2021-02-10T11:03:40.391 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:40.381 7efc74c7b700 1 --2- 172.21.2.102:0/3967373008 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efc700b1f80 0x7efc701b2fd0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7efc68009960 tx=0x7efc6800bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:03:40.391 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:40.381 7efc6cff9700 1 -- 172.21.2.102:0/3967373008 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efc68003030 con 0x7efc700b1f80 2021-02-10T11:03:40.391 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:40.381 7efc6cff9700 1 -- 172.21.2.102:0/3967373008 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc68001310 con 0x7efc700b1f80 2021-02-10T11:03:40.391 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:40.381 7efc6cff9700 1 -- 172.21.2.102:0/3967373008 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efc68001770 con 0x7efc700b1f80 2021-02-10T11:03:40.392 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:40.381 7efc7665e700 1 -- 172.21.2.102:0/3967373008 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc701b3a30 con 0x7efc700b1f80 2021-02-10T11:03:40.392 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:40.382 7efc7665e700 1 -- 172.21.2.102:0/3967373008 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc701b3a30 con 0x7efc700b1f80 2021-02-10T11:03:40.392 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:40.382 7efc6cff9700 1 -- 172.21.2.102:0/3967373008 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7efc68000d10 con 0x7efc700b1f80 2021-02-10T11:03:40.392 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:40.382 7efc6cff9700 1 --2- 172.21.2.102:0/3967373008 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7efc500186c0 0x7efc5001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:40.392 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:40.382 7efc6cff9700 1 -- 172.21.2.102:0/3967373008 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4753+0+0 (secure 0 0 0) 0x7efc680325a0 con 0x7efc700b1f80 2021-02-10T11:03:40.392 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:40.382 7efc6ffff700 1 --2- 172.21.2.102:0/3967373008 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7efc500186c0 0x7efc5001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:40.393 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:40.383 7efc6ffff700 1 --2- 172.21.2.102:0/3967373008 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7efc500186c0 0x7efc5001abc0 crc :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:03:40.413 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:40.413 7efc7665e700 1 -- 172.21.2.102:0/3967373008 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc701036a0 con 0x7efc700b1f80 2021-02-10T11:03:40.415 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:40.416 7efc6cff9700 1 -- 172.21.2.102:0/3967373008 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7efc68003020 con 0x7efc700b1f80 2021-02-10T11:03:40.539 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:40.540 7efc7665e700 1 -- 172.21.2.102:0/3967373008 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7efc700f0b20 con 0x7efc700b1f80 2021-02-10T11:03:40.540 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:40.540 7efc6cff9700 1 -- 172.21.2.102:0/3967373008 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v16) v1 ==== 84+0+64 (secure 0 0 0) 0x7efc68001310 con 0x7efc700b1f80 2021-02-10T11:03:40.540 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:03:40.540 INFO:teuthology.orchestra.run.gibba002.stdout:[ 2021-02-10T11:03:40.540 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:03:40.540 INFO:teuthology.orchestra.run.gibba002.stdout: "poolnum": 1, 2021-02-10T11:03:40.540 INFO:teuthology.orchestra.run.gibba002.stdout: "poolname": "rbd" 2021-02-10T11:03:40.541 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:03:40.541 INFO:teuthology.orchestra.run.gibba002.stdout:] 2021-02-10T11:03:40.542 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:40.542 7efc7665e700 1 -- 172.21.2.102:0/3967373008 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7efc500186c0 msgr2=0x7efc5001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:40.542 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:40.542 7efc7665e700 1 --2- 172.21.2.102:0/3967373008 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7efc500186c0 0x7efc5001abc0 crc :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:40.542 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:40.542 7efc7665e700 1 -- 172.21.2.102:0/3967373008 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efc700b1f80 msgr2=0x7efc701b2fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:40.542 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:40.542 7efc7665e700 1 --2- 172.21.2.102:0/3967373008 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efc700b1f80 0x7efc701b2fd0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7efc68009960 tx=0x7efc6800bcf0).stop 2021-02-10T11:03:40.542 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:40.542 7efc7665e700 1 -- 172.21.2.102:0/3967373008 shutdown_connections 2021-02-10T11:03:40.542 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:40.543 7efc7665e700 1 --2- 172.21.2.102:0/3967373008 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efc700b1f80 0x7efc701b2fd0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:40.543 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:40.543 7efc7665e700 1 --2- 172.21.2.102:0/3967373008 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7efc701a9a30 0x7efc700b1960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:03:40.543 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:40.543 7efc7665e700 1 --2- 172.21.2.102:0/3967373008 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7efc500186c0 0x7efc5001abc0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:40.543 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:40.543 7efc7665e700 1 --2- 172.21.2.102:0/3967373008 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7efc701041f0 0x7efc70103960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:03:40.543 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:40.543 7efc7665e700 1 -- 172.21.2.102:0/3967373008 >> 172.21.2.102:0/3967373008 conn(0x7efc701ad0f0 msgr2=0x7efc7008a460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:03:40.543 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:40.543 7efc7665e700 1 -- 172.21.2.102:0/3967373008 shutdown_connections 2021-02-10T11:03:40.543 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:40.543 7efc7665e700 1 -- 172.21.2.102:0/3967373008 wait complete. 2021-02-10T11:03:40.550 INFO:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2021-02-10T11:03:40.550 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool create cephfs_metadata 8 2021-02-10T11:03:40.842 DEBUG:teuthology.orchestra.run.gibba002:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:03:40.846 DEBUG:teuthology.orchestra.run.gibba013:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:03:40.979 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:40.963 7f7b2b456700 1 Processor -- start 2021-02-10T11:03:40.979 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:40.963 7f7b2b456700 1 -- start start 2021-02-10T11:03:40.979 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:40.963 7f7b2b456700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f7b241b0870 0x7f7b240f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:40.979 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:40.963 7f7b2b456700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f7b240efe00 0x7f7b240f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:40.980 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:40.963 7f7b2b456700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b241b0c70 con 0x7f7b241b0470 2021-02-10T11:03:40.980 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:40.963 7f7b2b456700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b240e5ea0 con 0x7f7b241b0870 2021-02-10T11:03:40.980 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:40.963 7f7b2b456700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b240f0b50 con 0x7f7b240efe00 2021-02-10T11:03:40.980 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:40.964 7f7b28a71700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f7b241b0870 0x7f7b240f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:40.980 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:40.964 7f7b29272700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f7b241b0470 0x7f7b240ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:41320/0 (socket says 172.21.2.102:41320) 2021-02-10T11:03:40.980 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:40.964 7f7b29272700 1 -- 172.21.2.102:0/504234644 learned_addr learned my addr 172.21.2.102:0/504234644 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:03:40.980 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:40.964 7f7b28a71700 1 -- 172.21.2.102:0/504234644 >> v2:172.21.2.113:3300/0 conn(0x7f7b240efe00 msgr2=0x7f7b240f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:03:40.980 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:40.964 7f7b28a71700 1 --2- 172.21.2.102:0/504234644 >> v2:172.21.2.113:3300/0 conn(0x7f7b240efe00 0x7f7b240f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:03:40.980 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:40.964 7f7b28a71700 1 -- 172.21.2.102:0/504234644 >> v1:172.21.2.102:6789/0 conn(0x7f7b241b0470 legacy=0x7f7b240ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:40.980 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:40.964 7f7b28a71700 1 -- 172.21.2.102:0/504234644 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b240f0820 con 0x7f7b241b0870 2021-02-10T11:03:40.980 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:40.964 7f7b28a71700 1 --2- 172.21.2.102:0/504234644 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7b241b0870 0x7f7b240f20e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f7b20006650 tx=0x7f7b20009a80).ready entity=mon.0 client_cookie=c7854d10e2fb5710 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:03:40.980 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:40.965 7f7b17fff700 1 -- 172.21.2.102:0/504234644 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7b200009b0 con 0x7f7b241b0870 2021-02-10T11:03:40.980 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:40.965 7f7b17fff700 1 -- 172.21.2.102:0/504234644 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b2000c0c0 con 0x7f7b241b0870 2021-02-10T11:03:40.980 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:40.965 7f7b17fff700 1 -- 172.21.2.102:0/504234644 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7b2000c5e0 con 0x7f7b241b0870 2021-02-10T11:03:40.981 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:40.965 7f7b2b456700 1 -- 172.21.2.102:0/504234644 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7b241b0870 msgr2=0x7f7b240f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:40.981 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:40.965 7f7b2b456700 1 --2- 172.21.2.102:0/504234644 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7b241b0870 0x7f7b240f20e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f7b20006650 tx=0x7f7b20009a80).stop 2021-02-10T11:03:40.981 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:40.965 7f7b2b456700 1 -- 172.21.2.102:0/504234644 shutdown_connections 2021-02-10T11:03:40.981 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:40.965 7f7b2b456700 1 --2- 172.21.2.102:0/504234644 >> v2:172.21.2.113:3300/0 conn(0x7f7b240efe00 0x7f7b240f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:03:40.981 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:40.965 7f7b2b456700 1 --2- 172.21.2.102:0/504234644 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7b241b0870 0x7f7b240f20e0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:40.981 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:40.965 7f7b2b456700 1 -- 172.21.2.102:0/504234644 >> 172.21.2.102:0/504234644 conn(0x7f7b240ce3e0 msgr2=0x7f7b240e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:03:40.981 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:40.966 7f7b2b456700 1 -- 172.21.2.102:0/504234644 shutdown_connections 2021-02-10T11:03:40.981 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:40.966 7f7b2b456700 1 -- 172.21.2.102:0/504234644 wait complete. 2021-02-10T11:03:40.982 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:40.980 7f7b2b456700 1 Processor -- start 2021-02-10T11:03:40.982 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:40.980 7f7b2b456700 1 -- start start 2021-02-10T11:03:40.983 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:40.981 7f7b2b456700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7b240c36a0 0x7f7b240c2b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:40.983 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:40.981 7f7b2b456700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7b240c3190 0x7f7b240b1d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:40.983 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:40.981 7f7b2b456700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7b240b2370 0x7f7b240b2d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:40.983 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:40.981 7f7b2b456700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b240b27c0 con 0x7f7b240c36a0 2021-02-10T11:03:40.983 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:40.981 7f7b2b456700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b240b3360 con 0x7f7b240c3190 2021-02-10T11:03:40.983 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:40.981 7f7b2b456700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b240b3540 con 0x7f7b240b2370 2021-02-10T11:03:40.983 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:40.981 7f7b29272700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7b240c36a0 0x7f7b240c2b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:40.983 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:40.981 7f7b29272700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7b240c36a0 0x7f7b240c2b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:49102/0 (socket says 172.21.2.102:49102) 2021-02-10T11:03:40.983 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:40.981 7f7b29272700 1 -- 172.21.2.102:0/2379166956 learned_addr learned my addr 172.21.2.102:0/2379166956 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:03:40.983 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:40.982 7f7b29272700 1 -- 172.21.2.102:0/2379166956 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7b240b2370 msgr2=0x7f7b240b2d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:40.983 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:40.982 7f7b29272700 1 --2- 172.21.2.102:0/2379166956 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7b240b2370 0x7f7b240b2d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:03:40.983 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:40.982 7f7b29272700 1 -- 172.21.2.102:0/2379166956 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7b240c3190 msgr2=0x7f7b240b1d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:40.983 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:40.982 7f7b29272700 1 --2- 172.21.2.102:0/2379166956 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7b240c3190 0x7f7b240b1d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:03:40.984 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:40.982 7f7b29272700 1 -- 172.21.2.102:0/2379166956 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b240b3760 con 0x7f7b240c36a0 2021-02-10T11:03:40.984 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:40.982 7f7b29272700 1 --2- 172.21.2.102:0/2379166956 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7b240c36a0 0x7f7b240c2b70 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f7b18001c60 tx=0x7f7b1800a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:03:40.984 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:40.982 7f7b15ffb700 1 -- 172.21.2.102:0/2379166956 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7b1800acc0 con 0x7f7b240c36a0 2021-02-10T11:03:40.984 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:40.982 7f7b15ffb700 1 -- 172.21.2.102:0/2379166956 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b18004140 con 0x7f7b240c36a0 2021-02-10T11:03:40.984 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:40.982 7f7b15ffb700 1 -- 172.21.2.102:0/2379166956 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7b18004590 con 0x7f7b240c36a0 2021-02-10T11:03:40.984 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:40.982 7f7b2b456700 1 -- 172.21.2.102:0/2379166956 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b240b3760 con 0x7f7b240c36a0 2021-02-10T11:03:40.984 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:40.982 7f7b2b456700 1 -- 172.21.2.102:0/2379166956 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b240b3cf0 con 0x7f7b240c36a0 2021-02-10T11:03:40.984 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:40.983 7f7b15ffb700 1 -- 172.21.2.102:0/2379166956 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f7b18004320 con 0x7f7b240c36a0 2021-02-10T11:03:40.984 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:40.983 7f7b15ffb700 1 --2- 172.21.2.102:0/2379166956 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7b040186c0 0x7f7b0401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:40.984 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:40.983 7f7b15ffb700 1 -- 172.21.2.102:0/2379166956 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4753+0+0 (secure 0 0 0) 0x7f7b18032210 con 0x7f7b240c36a0 2021-02-10T11:03:40.984 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:40.983 7f7b28a71700 1 --2- 172.21.2.102:0/2379166956 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7b040186c0 0x7f7b0401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:40.984 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:40.983 7f7b28a71700 1 --2- 172.21.2.102:0/2379166956 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7b040186c0 0x7f7b0401abc0 crc :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:03:41.013 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:41.014 7f7b2b456700 1 -- 172.21.2.102:0/2379166956 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b240edd10 con 0x7f7b240c36a0 2021-02-10T11:03:41.015 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:41.016 7f7b15ffb700 1 -- 172.21.2.102:0/2379166956 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f7b18000dd0 con 0x7f7b240c36a0 2021-02-10T11:03:41.143 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:41.143 7f7b2b456700 1 -- 172.21.2.102:0/2379166956 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"} v 0) v1 -- 0x7f7b240e52e0 con 0x7f7b240c36a0 2021-02-10T11:03:41.750 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:41.751 7f7b15ffb700 1 -- 172.21.2.102:0/2379166956 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"}]=0 pool 'cephfs_metadata' created v17) v1 ==== 133+0+0 (secure 0 0 0) 0x7f7b1801f2e0 con 0x7f7b240c36a0 2021-02-10T11:03:41.759 INFO:teuthology.orchestra.run.gibba002.stderr:pool 'cephfs_metadata' created 2021-02-10T11:03:41.761 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:41.761 7f7b2b456700 1 -- 172.21.2.102:0/2379166956 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7b040186c0 msgr2=0x7f7b0401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:41.761 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:41.761 7f7b2b456700 1 --2- 172.21.2.102:0/2379166956 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7b040186c0 0x7f7b0401abc0 crc :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:41.761 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:41.762 7f7b2b456700 1 -- 172.21.2.102:0/2379166956 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7b240c36a0 msgr2=0x7f7b240c2b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:41.761 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:41.762 7f7b2b456700 1 --2- 172.21.2.102:0/2379166956 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7b240c36a0 0x7f7b240c2b70 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f7b18001c60 tx=0x7f7b1800a8d0).stop 2021-02-10T11:03:41.761 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:41.762 7f7b2b456700 1 -- 172.21.2.102:0/2379166956 shutdown_connections 2021-02-10T11:03:41.761 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:41.762 7f7b2b456700 1 --2- 172.21.2.102:0/2379166956 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7b240b2370 0x7f7b240b2d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:03:41.761 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:41.762 7f7b2b456700 1 --2- 172.21.2.102:0/2379166956 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7b040186c0 0x7f7b0401abc0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:41.761 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:41.762 7f7b2b456700 1 --2- 172.21.2.102:0/2379166956 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7b240c3190 0x7f7b240b1d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:03:41.761 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:41.762 7f7b2b456700 1 --2- 172.21.2.102:0/2379166956 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7b240c36a0 0x7f7b240c2b70 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:41.762 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:41.762 7f7b2b456700 1 -- 172.21.2.102:0/2379166956 >> 172.21.2.102:0/2379166956 conn(0x7f7b2409f630 msgr2=0x7f7b240a0130 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:03:41.762 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:41.762 7f7b2b456700 1 -- 172.21.2.102:0/2379166956 shutdown_connections 2021-02-10T11:03:41.762 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:41.762 7f7b2b456700 1 -- 172.21.2.102:0/2379166956 wait complete. 2021-02-10T11:03:41.769 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool create cephfs_data 8 2021-02-10T11:03:42.197 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:42.181 7fe3775d9700 1 Processor -- start 2021-02-10T11:03:42.197 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:42.181 7fe3775d9700 1 -- start start 2021-02-10T11:03:42.197 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:42.182 7fe3775d9700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe3701b0470 0x7fe3701b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:42.198 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:42.182 7fe3775d9700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fe3700ef610 0x7fe3700efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:42.198 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:42.182 7fe3775d9700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fe3700f0030 0x7fe3700f0430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:42.198 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:42.182 7fe3775d9700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3700e5ea0 con 0x7fe3700f0030 2021-02-10T11:03:42.198 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:42.182 7fe3775d9700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3700f0b50 con 0x7fe3700ef610 2021-02-10T11:03:42.198 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:42.182 7fe3775d9700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3700e52e0 con 0x7fe3701b0470 2021-02-10T11:03:42.198 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:42.182 7fe375bf6700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fe3700f0030 0x7fe3700f0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:42.198 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:42.182 7fe375bf6700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fe3700f0030 0x7fe3700f0430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:49108/0 (socket says 172.21.2.102:49108) 2021-02-10T11:03:42.198 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:42.182 7fe375bf6700 1 -- 172.21.2.102:0/3401877782 learned_addr learned my addr 172.21.2.102:0/3401877782 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:03:42.198 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:42.182 7fe375bf6700 1 -- 172.21.2.102:0/3401877782 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe3701b0470 msgr2=0x7fe3701b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:03:42.199 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:42.182 7fe375bf6700 1 --2- 172.21.2.102:0/3401877782 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe3701b0470 0x7fe3701b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:03:42.199 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:42.182 7fe375bf6700 1 -- 172.21.2.102:0/3401877782 >> v2:172.21.2.113:3300/0 conn(0x7fe3700ef610 msgr2=0x7fe3700efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:03:42.199 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:42.182 7fe375bf6700 1 --2- 172.21.2.102:0/3401877782 >> v2:172.21.2.113:3300/0 conn(0x7fe3700ef610 0x7fe3700efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:03:42.199 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:42.182 7fe375bf6700 1 -- 172.21.2.102:0/3401877782 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3700f10d0 con 0x7fe3700f0030 2021-02-10T11:03:42.199 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:42.183 7fe375bf6700 1 --2- 172.21.2.102:0/3401877782 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe3700f0030 0x7fe3700f0430 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7fe368006f00 tx=0x7fe368009f10).ready entity=mon.0 client_cookie=8d227a49f08d93b4 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:03:42.199 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:42.183 7fe363fff700 1 -- 172.21.2.102:0/3401877782 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe368003030 con 0x7fe3700f0030 2021-02-10T11:03:42.199 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:42.183 7fe363fff700 1 -- 172.21.2.102:0/3401877782 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe36800b3b0 con 0x7fe3700f0030 2021-02-10T11:03:42.199 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:42.183 7fe363fff700 1 -- 172.21.2.102:0/3401877782 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe36800b8d0 con 0x7fe3700f0030 2021-02-10T11:03:42.199 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:42.183 7fe3775d9700 1 -- 172.21.2.102:0/3401877782 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe3700f0030 msgr2=0x7fe3700f0430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:42.199 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:42.183 7fe3775d9700 1 --2- 172.21.2.102:0/3401877782 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe3700f0030 0x7fe3700f0430 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7fe368006f00 tx=0x7fe368009f10).stop 2021-02-10T11:03:42.199 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:42.183 7fe3775d9700 1 -- 172.21.2.102:0/3401877782 shutdown_connections 2021-02-10T11:03:42.199 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:42.183 7fe3775d9700 1 --2- 172.21.2.102:0/3401877782 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe3700f0030 0x7fe3700f0430 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:42.200 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:42.183 7fe3775d9700 1 --2- 172.21.2.102:0/3401877782 >> v2:172.21.2.113:3300/0 conn(0x7fe3700ef610 0x7fe3700efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:03:42.200 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:42.183 7fe3775d9700 1 --2- 172.21.2.102:0/3401877782 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe3701b0470 0x7fe3701b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:03:42.200 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:42.183 7fe3775d9700 1 -- 172.21.2.102:0/3401877782 >> 172.21.2.102:0/3401877782 conn(0x7fe3700ce3e0 msgr2=0x7fe3700e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:03:42.200 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:42.184 7fe3775d9700 1 -- 172.21.2.102:0/3401877782 shutdown_connections 2021-02-10T11:03:42.200 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:42.184 7fe3775d9700 1 -- 172.21.2.102:0/3401877782 wait complete. 2021-02-10T11:03:42.200 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:42.198 7fe3775d9700 1 Processor -- start 2021-02-10T11:03:42.200 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:42.198 7fe3775d9700 1 -- start start 2021-02-10T11:03:42.202 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:42.199 7fe3775d9700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe3700b20b0 0x7fe3700ac970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:42.202 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:42.199 7fe3775d9700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe3700b24b0 0x7fe370191dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:42.202 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:42.199 7fe3775d9700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe3700acf90 0x7fe3701923e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:42.202 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:42.199 7fe3775d9700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3700b1b30 con 0x7fe3700b24b0 2021-02-10T11:03:42.202 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:42.199 7fe3775d9700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3701a04e0 con 0x7fe3700b20b0 2021-02-10T11:03:42.202 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:42.199 7fe3775d9700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3701a06c0 con 0x7fe3700acf90 2021-02-10T11:03:42.202 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:42.199 7fe374bf4700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe3700b24b0 0x7fe370191dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:42.202 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:42.199 7fe374bf4700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe3700b24b0 0x7fe370191dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:49116/0 (socket says 172.21.2.102:49116) 2021-02-10T11:03:42.203 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:42.199 7fe374bf4700 1 -- 172.21.2.102:0/2056775009 learned_addr learned my addr 172.21.2.102:0/2056775009 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:03:42.203 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:42.199 7fe374bf4700 1 -- 172.21.2.102:0/2056775009 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe3700acf90 msgr2=0x7fe3701923e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:42.203 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:42.199 7fe374bf4700 1 --2- 172.21.2.102:0/2056775009 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe3700acf90 0x7fe3701923e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:03:42.203 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:42.199 7fe374bf4700 1 -- 172.21.2.102:0/2056775009 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe3700b20b0 msgr2=0x7fe3700ac970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:42.203 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:42.199 7fe374bf4700 1 --2- 172.21.2.102:0/2056775009 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe3700b20b0 0x7fe3700ac970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:03:42.203 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:42.199 7fe374bf4700 1 -- 172.21.2.102:0/2056775009 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe370192be0 con 0x7fe3700b24b0 2021-02-10T11:03:42.203 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:42.199 7fe374bf4700 1 --2- 172.21.2.102:0/2056775009 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe3700b24b0 0x7fe370191dc0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7fe36c006f00 tx=0x7fe36c00b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:03:42.203 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:42.200 7fe361ffb700 1 -- 172.21.2.102:0/2056775009 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe36c00bd30 con 0x7fe3700b24b0 2021-02-10T11:03:42.203 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:42.200 7fe3775d9700 1 -- 172.21.2.102:0/2056775009 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe370192be0 con 0x7fe3700b24b0 2021-02-10T11:03:42.204 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:42.200 7fe361ffb700 1 -- 172.21.2.102:0/2056775009 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe36c00bd30 con 0x7fe3700b24b0 2021-02-10T11:03:42.204 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:42.200 7fe361ffb700 1 -- 172.21.2.102:0/2056775009 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe36c000c90 con 0x7fe3700b24b0 2021-02-10T11:03:42.204 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:42.200 7fe3775d9700 1 -- 172.21.2.102:0/2056775009 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe370192be0 con 0x7fe3700b24b0 2021-02-10T11:03:42.204 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:42.200 7fe361ffb700 1 -- 172.21.2.102:0/2056775009 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fe36c005310 con 0x7fe3700b24b0 2021-02-10T11:03:42.204 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:42.200 7fe361ffb700 1 --2- 172.21.2.102:0/2056775009 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe350018730 0x7fe35001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:42.204 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:42.200 7fe361ffb700 1 -- 172.21.2.102:0/2056775009 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 5107+0+0 (secure 0 0 0) 0x7fe36c032470 con 0x7fe3700b24b0 2021-02-10T11:03:42.204 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:42.200 7fe3753f5700 1 --2- 172.21.2.102:0/2056775009 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe350018730 0x7fe35001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:42.204 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:42.201 7fe3753f5700 1 --2- 172.21.2.102:0/2056775009 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe350018730 0x7fe35001abc0 crc :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:03:42.231 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:42.232 7fe3775d9700 1 -- 172.21.2.102:0/2056775009 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe370105af0 con 0x7fe3700b24b0 2021-02-10T11:03:42.233 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:42.234 7fe361ffb700 1 -- 172.21.2.102:0/2056775009 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fe36c006b90 con 0x7fe3700b24b0 2021-02-10T11:03:42.361 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:42.362 7fe3775d9700 1 -- 172.21.2.102:0/2056775009 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_data"} v 0) v1 -- 0x7fe3700f0b50 con 0x7fe3700b24b0 2021-02-10T11:03:42.753 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:42.754 7fe361ffb700 1 -- 172.21.2.102:0/2056775009 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_data"}]=0 pool 'cephfs_data' created v18) v1 ==== 125+0+0 (secure 0 0 0) 0x7fe36c005310 con 0x7fe3700b24b0 2021-02-10T11:03:42.778 INFO:teuthology.orchestra.run.gibba002.stderr:pool 'cephfs_data' created 2021-02-10T11:03:42.779 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:42.780 7fe3775d9700 1 -- 172.21.2.102:0/2056775009 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe350018730 msgr2=0x7fe35001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:42.780 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:42.780 7fe3775d9700 1 --2- 172.21.2.102:0/2056775009 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe350018730 0x7fe35001abc0 crc :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:42.780 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:42.780 7fe3775d9700 1 -- 172.21.2.102:0/2056775009 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe3700b24b0 msgr2=0x7fe370191dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:42.780 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:42.780 7fe3775d9700 1 --2- 172.21.2.102:0/2056775009 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe3700b24b0 0x7fe370191dc0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7fe36c006f00 tx=0x7fe36c00b940).stop 2021-02-10T11:03:42.780 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:42.780 7fe3775d9700 1 -- 172.21.2.102:0/2056775009 shutdown_connections 2021-02-10T11:03:42.780 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:42.780 7fe3775d9700 1 --2- 172.21.2.102:0/2056775009 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe3700acf90 0x7fe3701923e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:03:42.780 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:42.780 7fe3775d9700 1 --2- 172.21.2.102:0/2056775009 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe3700b24b0 0x7fe370191dc0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:42.780 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:42.780 7fe3775d9700 1 --2- 172.21.2.102:0/2056775009 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe350018730 0x7fe35001abc0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:42.780 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:42.780 7fe3775d9700 1 --2- 172.21.2.102:0/2056775009 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe3700b20b0 0x7fe3700ac970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:03:42.780 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:42.780 7fe3775d9700 1 -- 172.21.2.102:0/2056775009 >> 172.21.2.102:0/2056775009 conn(0x7fe3700bf4e0 msgr2=0x7fe3700be3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:03:42.780 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:42.780 7fe3775d9700 1 -- 172.21.2.102:0/2056775009 shutdown_connections 2021-02-10T11:03:42.780 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:42.780 7fe3775d9700 1 -- 172.21.2.102:0/2056775009 wait complete. 2021-02-10T11:03:42.787 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2021-02-10T11:03:43.219 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:43.202 7f2814cd7700 1 Processor -- start 2021-02-10T11:03:43.220 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:43.202 7f2814cd7700 1 -- start start 2021-02-10T11:03:43.220 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:43.203 7f2814cd7700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f28100d1870 0x7f28101b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:43.220 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:43.203 7f2814cd7700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28100fe1c0 con 0x7f28101b0ae0 2021-02-10T11:03:43.220 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:43.203 7f2814cd7700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2810109a60 con 0x7f28100cff70 2021-02-10T11:03:43.220 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:43.203 7f2814cd7700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2810088020 con 0x7f28100d1870 2021-02-10T11:03:43.220 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:43.203 7f280e2e7700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f28101b0ae0 0x7f2810108940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:41344/0 (socket says 172.21.2.102:41344) 2021-02-10T11:03:43.220 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:43.203 7f280e2e7700 1 -- 172.21.2.102:0/2524163196 learned_addr learned my addr 172.21.2.102:0/2524163196 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:03:43.220 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:43.204 7f280dae6700 1 -- 172.21.2.102:0/2524163196 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4220680146 0 0) 0x7f28000009b0 con 0x7f28101b0ae0 2021-02-10T11:03:43.220 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:43.204 7f280dae6700 1 -- 172.21.2.102:0/2524163196 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f27fc002fe0 con 0x7f28101b0ae0 2021-02-10T11:03:43.220 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:43.204 7f280dae6700 1 -- 172.21.2.102:0/2524163196 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1494168601 0 0) 0x7f28000010b0 con 0x7f28101b0ae0 2021-02-10T11:03:43.220 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:43.204 7f280eae8700 1 --2- 172.21.2.102:0/2524163196 >> v2:172.21.2.113:3300/0 conn(0x7f28100d1870 0x7f28101b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:43.220 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:43.204 7f280dae6700 1 -- 172.21.2.102:0/2524163196 >> v2:172.21.2.113:3300/0 conn(0x7f28100d1870 msgr2=0x7f28101b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:03:43.221 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:43.204 7f280dae6700 1 --2- 172.21.2.102:0/2524163196 >> v2:172.21.2.113:3300/0 conn(0x7f28100d1870 0x7f28101b04c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:43.221 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:43.204 7f280dae6700 1 -- 172.21.2.102:0/2524163196 >> v1:172.21.2.113:6789/0 conn(0x7f28100cff70 legacy=0x7f2810109130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:03:43.221 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:43.204 7f280dae6700 1 -- 172.21.2.102:0/2524163196 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2810088210 con 0x7f28101b0ae0 2021-02-10T11:03:43.221 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:43.204 7f280dae6700 1 -- 172.21.2.102:0/2524163196 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7f28000015b0 con 0x7f28101b0ae0 2021-02-10T11:03:43.221 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:43.204 7f280dae6700 1 -- 172.21.2.102:0/2524163196 >> v1:172.21.2.102:6789/0 conn(0x7f28101b0ae0 legacy=0x7f2810108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:43.221 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:43.204 7f280dae6700 1 --2- 172.21.2.102:0/2524163196 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f27fc003b90 0x7f27fc006000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:43.221 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:43.204 7f280dae6700 1 --2- 172.21.2.102:0/2524163196 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f27fc006820 0x7f27fc008d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:43.221 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:43.204 7f280dae6700 1 --2- 172.21.2.102:0/2524163196 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f27fc009640 0x7f27fc00bb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:43.221 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:43.204 7f280dae6700 1 -- 172.21.2.102:0/2524163196 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27fc00c390 con 0x7f27fc009640 2021-02-10T11:03:43.221 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:43.204 7f280dae6700 1 -- 172.21.2.102:0/2524163196 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27fc00c5b0 con 0x7f27fc006820 2021-02-10T11:03:43.221 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:43.204 7f280dae6700 1 -- 172.21.2.102:0/2524163196 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27fc00c7d0 con 0x7f27fc003b90 2021-02-10T11:03:43.221 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:43.204 7f280eae8700 1 --2- 172.21.2.102:0/2524163196 >> v2:172.21.2.113:3300/0 conn(0x7f28100d1870 0x7f28101b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:03:43.221 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:43.204 7f280f2e9700 1 --2- 172.21.2.102:0/2524163196 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f27fc009640 0x7f27fc00bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:43.222 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:43.204 7f280e2e7700 1 --2- 172.21.2.102:0/2524163196 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f27fc006820 0x7f27fc008d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:43.222 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:43.204 7f280eae8700 1 --2- 172.21.2.102:0/2524163196 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f27fc003b90 0x7f27fc006000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:43.222 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:43.204 7f280f2e9700 1 -- 172.21.2.102:0/2524163196 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f27fc003b90 msgr2=0x7f27fc006000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:03:43.222 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:43.204 7f280f2e9700 1 --2- 172.21.2.102:0/2524163196 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f27fc003b90 0x7f27fc006000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:43.222 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:43.205 7f280f2e9700 1 -- 172.21.2.102:0/2524163196 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f27fc006820 msgr2=0x7f27fc008d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:03:43.222 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:43.205 7f280f2e9700 1 --2- 172.21.2.102:0/2524163196 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f27fc006820 0x7f27fc008d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:43.222 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:43.205 7f280f2e9700 1 -- 172.21.2.102:0/2524163196 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27fc00ca70 con 0x7f27fc009640 2021-02-10T11:03:43.222 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:43.205 7f280f2e9700 1 -- 172.21.2.102:0/2524163196 reap_dead start 2021-02-10T11:03:43.222 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:43.205 7f280f2e9700 1 --2- 172.21.2.102:0/2524163196 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f27fc009640 0x7f27fc00bb40 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f2808006e30 tx=0x7f280804de60).ready entity=mon.0 client_cookie=292394bd9966a4c6 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:03:43.222 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:43.205 7f280dae6700 1 -- 172.21.2.102:0/2524163196 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2808003020 con 0x7f27fc009640 2021-02-10T11:03:43.222 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:43.205 7f280dae6700 1 -- 172.21.2.102:0/2524163196 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2808001400 con 0x7f27fc009640 2021-02-10T11:03:43.222 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:43.205 7f280dae6700 1 -- 172.21.2.102:0/2524163196 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2808006a70 con 0x7f27fc009640 2021-02-10T11:03:43.223 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:43.205 7f2814cd7700 1 -- 172.21.2.102:0/2524163196 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f27fc009640 msgr2=0x7f27fc00bb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:43.223 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:43.205 7f2814cd7700 1 --2- 172.21.2.102:0/2524163196 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f27fc009640 0x7f27fc00bb40 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f2808006e30 tx=0x7f280804de60).stop 2021-02-10T11:03:43.223 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:43.206 7f2814cd7700 1 -- 172.21.2.102:0/2524163196 shutdown_connections 2021-02-10T11:03:43.223 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:43.206 7f2814cd7700 1 --2- 172.21.2.102:0/2524163196 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f27fc009640 0x7f27fc00bb40 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:43.223 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:43.206 7f2814cd7700 1 -- 172.21.2.102:0/2524163196 >> 172.21.2.102:0/2524163196 conn(0x7f28100ce420 msgr2=0x7f28100d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:03:43.223 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:43.206 7f2814cd7700 1 -- 172.21.2.102:0/2524163196 shutdown_connections 2021-02-10T11:03:43.223 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:43.206 7f2814cd7700 1 -- 172.21.2.102:0/2524163196 wait complete. 2021-02-10T11:03:43.223 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:43.220 7f2814cd7700 1 Processor -- start 2021-02-10T11:03:43.223 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:43.220 7f2814cd7700 1 -- start start 2021-02-10T11:03:43.225 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:43.221 7f2814cd7700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f28101a4390 0x7f28100a3460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:43.225 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:43.221 7f2814cd7700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f28101a4790 0x7f28100a3a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:43.225 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:43.221 7f2814cd7700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f28100a40a0 0x7f281008adb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:43.225 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:43.221 7f2814cd7700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f281008a8b0 con 0x7f28101a4790 2021-02-10T11:03:43.225 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:43.221 7f2814cd7700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28100a44a0 con 0x7f28101a4390 2021-02-10T11:03:43.225 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:43.221 7f2814cd7700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28100b5360 con 0x7f28100a40a0 2021-02-10T11:03:43.225 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:43.221 7f280e2e7700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f28101a4790 0x7f28100a3a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:43.225 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:43.221 7f280e2e7700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f28101a4790 0x7f28100a3a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:49134/0 (socket says 172.21.2.102:49134) 2021-02-10T11:03:43.225 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:43.221 7f280e2e7700 1 -- 172.21.2.102:0/876066594 learned_addr learned my addr 172.21.2.102:0/876066594 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:03:43.225 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:43.221 7f280eae8700 1 --2- 172.21.2.102:0/876066594 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f28101a4390 0x7f28100a3460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:43.225 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:43.221 7f280f2e9700 1 --2- 172.21.2.102:0/876066594 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f28100a40a0 0x7f281008adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:43.225 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:43.221 7f280e2e7700 1 -- 172.21.2.102:0/876066594 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f28100a40a0 msgr2=0x7f281008adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:43.226 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:43.221 7f280e2e7700 1 --2- 172.21.2.102:0/876066594 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f28100a40a0 0x7f281008adb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:43.226 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:43.221 7f280e2e7700 1 -- 172.21.2.102:0/876066594 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f28101a4390 msgr2=0x7f28100a3460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:43.226 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:43.221 7f280e2e7700 1 --2- 172.21.2.102:0/876066594 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f28101a4390 0x7f28100a3460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:43.226 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:43.221 7f280e2e7700 1 -- 172.21.2.102:0/876066594 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f281008a460 con 0x7f28101a4790 2021-02-10T11:03:43.226 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:43.221 7f280e2e7700 1 --2- 172.21.2.102:0/876066594 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f28101a4790 0x7f28100a3a80 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7f2800001660 tx=0x7f280000a930).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:03:43.226 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:43.222 7f28077fe700 1 -- 172.21.2.102:0/876066594 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f280000ad20 con 0x7f28101a4790 2021-02-10T11:03:43.226 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:43.222 7f28077fe700 1 -- 172.21.2.102:0/876066594 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28000016a0 con 0x7f28101a4790 2021-02-10T11:03:43.226 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:43.222 7f28077fe700 1 -- 172.21.2.102:0/876066594 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f28000043c0 con 0x7f28101a4790 2021-02-10T11:03:43.226 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:43.222 7f2814cd7700 1 -- 172.21.2.102:0/876066594 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f281008a460 con 0x7f28101a4790 2021-02-10T11:03:43.226 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:43.222 7f2814cd7700 1 -- 172.21.2.102:0/876066594 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f281008b6c0 con 0x7f28101a4790 2021-02-10T11:03:43.226 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:43.222 7f28077fe700 1 -- 172.21.2.102:0/876066594 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f2800004140 con 0x7f28101a4790 2021-02-10T11:03:43.226 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:43.222 7f28077fe700 1 --2- 172.21.2.102:0/876066594 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f27f40186c0 0x7f27f401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:43.226 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:43.223 7f28077fe700 1 -- 172.21.2.102:0/876066594 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5457+0+0 (secure 0 0 0) 0x7f28000324c0 con 0x7f28101a4790 2021-02-10T11:03:43.227 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:43.223 7f280eae8700 1 --2- 172.21.2.102:0/876066594 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f27f40186c0 0x7f27f401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:43.227 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:43.223 7f280eae8700 1 --2- 172.21.2.102:0/876066594 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f27f40186c0 0x7f27f401abc0 crc :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:03:43.252 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:43.253 7f2814cd7700 1 -- 172.21.2.102:0/876066594 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28100828e0 con 0x7f28101a4790 2021-02-10T11:03:43.254 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:43.255 7f28077fe700 1 -- 172.21.2.102:0/876066594 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f2800014f50 con 0x7f28101a4790 2021-02-10T11:03:43.382 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:43.383 7f2814cd7700 1 -- 172.21.2.102:0/876066594 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"data": "cephfs_data", "prefix": "fs new", "force": true, "fs_name": "cephfs", "metadata": "cephfs_metadata"} v 0) v1 -- 0x7f281008a830 con 0x7f28101a4790 2021-02-10T11:03:43.382 INFO:tasks.ceph.mon.a.gibba002.stderr:2021-02-10 11:03:43.383 7f8dcff50700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-02-10T11:03:43.386 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:43.387 7f28077fe700 1 -- 172.21.2.102:0/876066594 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"data": "cephfs_data", "prefix": "fs new", "force": true, "fs_name": "cephfs", "metadata": "cephfs_metadata"}]=0 new fs with metadata pool 2 and data pool 3 v2) v1 ==== 187+0+0 (secure 0 0 0) 0x7f2800004140 con 0x7f28101a4790 2021-02-10T11:03:43.389 INFO:teuthology.orchestra.run.gibba002.stderr:new fs with metadata pool 2 and data pool 3 2021-02-10T11:03:43.391 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:43.392 7f2814cd7700 1 -- 172.21.2.102:0/876066594 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f27f40186c0 msgr2=0x7f27f401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:43.391 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:43.392 7f2814cd7700 1 --2- 172.21.2.102:0/876066594 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f27f40186c0 0x7f27f401abc0 crc :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:43.391 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:43.392 7f2814cd7700 1 -- 172.21.2.102:0/876066594 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f28101a4790 msgr2=0x7f28100a3a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:43.391 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:43.392 7f2814cd7700 1 --2- 172.21.2.102:0/876066594 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f28101a4790 0x7f28100a3a80 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7f2800001660 tx=0x7f280000a930).stop 2021-02-10T11:03:43.391 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:43.392 7f2814cd7700 1 -- 172.21.2.102:0/876066594 shutdown_connections 2021-02-10T11:03:43.391 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:43.392 7f2814cd7700 1 --2- 172.21.2.102:0/876066594 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f28100a40a0 0x7f281008adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:43.391 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:43.392 7f2814cd7700 1 --2- 172.21.2.102:0/876066594 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f28101a4790 0x7f28100a3a80 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:43.391 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:43.392 7f2814cd7700 1 --2- 172.21.2.102:0/876066594 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f27f40186c0 0x7f27f401abc0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:43.391 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:43.392 7f2814cd7700 1 --2- 172.21.2.102:0/876066594 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f28101a4390 0x7f28100a3460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:43.392 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:43.392 7f2814cd7700 1 -- 172.21.2.102:0/876066594 >> 172.21.2.102:0/876066594 conn(0x7f2810093520 msgr2=0x7f28100c3c40 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:03:43.392 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:43.392 7f2814cd7700 1 -- 172.21.2.102:0/876066594 shutdown_connections 2021-02-10T11:03:43.392 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:43.392 7f2814cd7700 1 -- 172.21.2.102:0/876066594 wait complete. 2021-02-10T11:03:43.399 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:03:43.826 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:43.810 7fac24a46700 1 Processor -- start 2021-02-10T11:03:43.826 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:43.810 7fac24a46700 1 -- start start 2021-02-10T11:03:43.826 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:43.811 7fac24a46700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fac200e5070 0x7fac201b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:43.827 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:43.811 7fac24a46700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fac200e5470 0x7fac20080310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:43.827 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:43.811 7fac24a46700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac2007ed80 con 0x7fac201b3e40 2021-02-10T11:03:43.827 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:43.811 7fac24a46700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac200d1790 con 0x7fac200e5070 2021-02-10T11:03:43.827 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:43.811 7fac24a46700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac200d19b0 con 0x7fac200e5470 2021-02-10T11:03:43.827 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:43.811 7fac1f05f700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fac200e5470 0x7fac20080310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:43.827 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:43.811 7fac1e85e700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fac200e5070 0x7fac201b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:43.827 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:43.811 7fac1e85e700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fac200e5070 0x7fac201b3820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:49138/0 (socket says 172.21.2.102:49138) 2021-02-10T11:03:43.827 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:43.811 7fac1f05f700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fac200e5470 0x7fac20080310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.113:3300/0 says I am v2:172.21.2.102:52572/0 (socket says 172.21.2.102:52572) 2021-02-10T11:03:43.827 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:43.811 7fac1e85e700 1 -- 172.21.2.102:0/157218203 learned_addr learned my addr 172.21.2.102:0/157218203 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:03:43.827 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:43.811 7fac1d85c700 1 -- 172.21.2.102:0/157218203 <== mon.1 v1:172.21.2.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1351300154 0 0) 0x7fac10000c60 con 0x7fac201b3e40 2021-02-10T11:03:43.827 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:43.811 7fac1d85c700 1 -- 172.21.2.102:0/157218203 --> v1:172.21.2.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac0c002fe0 con 0x7fac201b3e40 2021-02-10T11:03:43.827 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:43.812 7fac1f05f700 1 -- 172.21.2.102:0/157218203 >> v2:172.21.2.102:3300/0 conn(0x7fac200e5070 msgr2=0x7fac201b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:03:43.827 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:43.812 7fac1f05f700 1 --2- 172.21.2.102:0/157218203 >> v2:172.21.2.102:3300/0 conn(0x7fac200e5070 0x7fac201b3820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:43.828 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:43.812 7fac1f05f700 1 -- 172.21.2.102:0/157218203 >> v1:172.21.2.113:6789/0 conn(0x7fac201b3e40 legacy=0x7fac2007d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:43.828 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:43.812 7fac1f05f700 1 -- 172.21.2.102:0/157218203 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac200d1bd0 con 0x7fac200e5470 2021-02-10T11:03:43.828 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:43.812 7fac1e85e700 1 --2- 172.21.2.102:0/157218203 >> v2:172.21.2.102:3300/0 conn(0x7fac200e5070 0x7fac201b3820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:03:43.828 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:43.812 7fac1f05f700 1 --2- 172.21.2.102:0/157218203 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fac200e5470 0x7fac20080310 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7fac18006e50 tx=0x7fac18009e70).ready entity=mon.1 client_cookie=9884ef851d00b36f server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:03:43.828 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:43.812 7fac1d85c700 1 -- 172.21.2.102:0/157218203 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fac18000f90 con 0x7fac200e5470 2021-02-10T11:03:43.828 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:43.812 7fac1d85c700 1 -- 172.21.2.102:0/157218203 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac180066f0 con 0x7fac200e5470 2021-02-10T11:03:43.828 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:43.812 7fac1d85c700 1 -- 172.21.2.102:0/157218203 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fac1800b700 con 0x7fac200e5470 2021-02-10T11:03:43.828 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:43.812 7fac24a46700 1 -- 172.21.2.102:0/157218203 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fac200e5470 msgr2=0x7fac20080310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:43.828 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:43.812 7fac24a46700 1 --2- 172.21.2.102:0/157218203 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fac200e5470 0x7fac20080310 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7fac18006e50 tx=0x7fac18009e70).stop 2021-02-10T11:03:43.828 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:43.813 7fac24a46700 1 -- 172.21.2.102:0/157218203 shutdown_connections 2021-02-10T11:03:43.828 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:43.813 7fac24a46700 1 --2- 172.21.2.102:0/157218203 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fac200e5470 0x7fac20080310 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:43.828 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:43.813 7fac24a46700 1 --2- 172.21.2.102:0/157218203 >> v2:172.21.2.102:3300/0 conn(0x7fac200e5070 0x7fac201b3820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:43.828 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:43.813 7fac24a46700 1 -- 172.21.2.102:0/157218203 >> 172.21.2.102:0/157218203 conn(0x7fac200ce690 msgr2=0x7fac200e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:03:43.829 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:43.813 7fac24a46700 1 -- 172.21.2.102:0/157218203 shutdown_connections 2021-02-10T11:03:43.829 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:43.813 7fac24a46700 1 -- 172.21.2.102:0/157218203 wait complete. 2021-02-10T11:03:43.830 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:43.828 7fac24a46700 1 Processor -- start 2021-02-10T11:03:43.830 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:43.828 7fac24a46700 1 -- start start 2021-02-10T11:03:43.830 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:43.828 7fac24a46700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fac201918e0 0x7fac20189c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:43.830 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:43.828 7fac24a46700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fac2018a2a0 0x7fac2018a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:43.830 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:43.828 7fac24a46700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fac2018acc0 0x7fac20190070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:43.830 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:43.828 7fac24a46700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac201a2f00 con 0x7fac2018acc0 2021-02-10T11:03:43.830 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:43.828 7fac24a46700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac20191ce0 con 0x7fac2018a2a0 2021-02-10T11:03:43.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:43.828 7fac24a46700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac20191f00 con 0x7fac201918e0 2021-02-10T11:03:43.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:43.828 7fac1f05f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fac2018acc0 0x7fac20190070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:43.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:43.828 7fac1f05f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fac2018acc0 0x7fac20190070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:49148/0 (socket says 172.21.2.102:49148) 2021-02-10T11:03:43.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:43.828 7fac1f05f700 1 -- 172.21.2.102:0/857076210 learned_addr learned my addr 172.21.2.102:0/857076210 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:03:43.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:43.829 7fac1f05f700 1 -- 172.21.2.102:0/857076210 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fac201918e0 msgr2=0x7fac20189c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:43.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:43.829 7fac1f05f700 1 --2- 172.21.2.102:0/857076210 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fac201918e0 0x7fac20189c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:03:43.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:43.829 7fac1f05f700 1 -- 172.21.2.102:0/857076210 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fac2018a2a0 msgr2=0x7fac2018a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:43.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:43.829 7fac1f05f700 1 --2- 172.21.2.102:0/857076210 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fac2018a2a0 0x7fac2018a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:03:43.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:43.829 7fac1f05f700 1 -- 172.21.2.102:0/857076210 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac20190690 con 0x7fac2018acc0 2021-02-10T11:03:43.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:43.829 7fac1f05f700 1 --2- 172.21.2.102:0/857076210 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fac2018acc0 0x7fac20190070 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7fac18006830 tx=0x7fac1800bcc0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:03:43.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:43.829 7fac177fe700 1 -- 172.21.2.102:0/857076210 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fac18009390 con 0x7fac2018acc0 2021-02-10T11:03:43.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:43.829 7fac177fe700 1 -- 172.21.2.102:0/857076210 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac18002030 con 0x7fac2018acc0 2021-02-10T11:03:43.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:43.829 7fac24a46700 1 -- 172.21.2.102:0/857076210 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac20190690 con 0x7fac2018acc0 2021-02-10T11:03:43.832 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:43.829 7fac24a46700 1 -- 172.21.2.102:0/857076210 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac20190c20 con 0x7fac2018acc0 2021-02-10T11:03:43.832 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:43.829 7fac177fe700 1 -- 172.21.2.102:0/857076210 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fac18002550 con 0x7fac2018acc0 2021-02-10T11:03:43.832 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:43.830 7fac177fe700 1 -- 172.21.2.102:0/857076210 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fac18002a20 con 0x7fac2018acc0 2021-02-10T11:03:43.832 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:43.830 7fac177fe700 1 --2- 172.21.2.102:0/857076210 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fac040186d0 0x7fac0401abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:43.832 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:43.830 7fac177fe700 1 -- 172.21.2.102:0/857076210 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fac18032b60 con 0x7fac2018acc0 2021-02-10T11:03:43.832 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:43.830 7fac1e85e700 1 --2- 172.21.2.102:0/857076210 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fac040186d0 0x7fac0401abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:43.832 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:43.830 7fac1e85e700 1 --2- 172.21.2.102:0/857076210 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fac040186d0 0x7fac0401abd0 crc :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:03:43.859 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:43.860 7fac24a46700 1 -- 172.21.2.102:0/857076210 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac20189750 con 0x7fac2018acc0 2021-02-10T11:03:43.862 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:43.863 7fac177fe700 1 -- 172.21.2.102:0/857076210 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fac18002810 con 0x7fac2018acc0 2021-02-10T11:03:43.987 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:43.988 7fac24a46700 1 -- 172.21.2.102:0/857076210 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fac20189750 con 0x7fac2018acc0 2021-02-10T11:03:43.987 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:43.988 7fac177fe700 1 -- 172.21.2.102:0/857076210 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v19) v1 ==== 74+0+14006 (secure 0 0 0) 0x7fac18002a20 con 0x7fac2018acc0 2021-02-10T11:03:43.988 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:03:43.988 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":19,"fsid":"68db30ea-dbce-4aef-a6df-f01c6fea0513","created":"2021-02-10 11:03:06.331557","modified":"2021-02-10 11:03:43.384612","last_up_change":"2021-02-10 11:03:32.690562","last_in_change":"2021-02-10 11:03:32.690562","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:03:37.322519","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:03:41.161563","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2021-02-10 11:03:42.379651","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"be691b4d-a039-408a-af80-9d53f02b6f1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6818","nonce":36088},{"type":"v1","addr":"172.21.2.102:6820","nonce":36088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6822","nonce":36088},{"type":"v1","addr":"172.21.2.102:6824","nonce":36088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6830","nonce":36088},{"type":"v1","addr":"172.21.2.102:6832","nonce":36088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6826","nonce":36088},{"type":"v1","addr":"172.21.2.102:6828","nonce":36088}]},"public_addr":"172.21.2.102:6820/36088","cluster_addr":"172.21.2.102:6824/36088","heartbeat_back_addr":"172.21.2.102:6832/36088","heartbeat_front_addr":"172.21.2.102:6828/36088","state":["exists","up"]},{"osd":1,"uuid":"64e3ccc8-9c69-4d7c-865e-af8e83822635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6819","nonce":36087},{"type":"v1","addr":"172.21.2.102:6821","nonce":36087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6823","nonce":36087},{"type":"v1","addr":"172.21.2.102:6825","nonce":36087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6831","nonce":36087},{"type":"v1","addr":"172.21.2.102:6833","nonce":36087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6827","nonce":36087},{"type":"v1","addr":"172.21.2.102:6829","nonce":36087}]},"public_addr":"172.21.2.102:6821/36087","cluster_addr":"172.21.2.102:6825/36087","heartbeat_back_addr":"172.21.2.102:6833/36087","heartbeat_front_addr":"172.21.2.102:6829/36087","state":["exists","up"]},{"osd":2,"uuid":"153700af-747f-481f-bdf4-d6072d3a842f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6810","nonce":36089},{"type":"v1","addr":"172.21.2.102:6811","nonce":36089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6812","nonce":36089},{"type":"v1","addr":"172.21.2.102:6813","nonce":36089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6816","nonce":36089},{"type":"v1","addr":"172.21.2.102:6817","nonce":36089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6814","nonce":36089},{"type":"v1","addr":"172.21.2.102:6815","nonce":36089}]},"public_addr":"172.21.2.102:6811/36089","cluster_addr":"172.21.2.102:6813/36089","heartbeat_back_addr":"172.21.2.102:6817/36089","heartbeat_front_addr":"172.21.2.102:6815/36089","state":["exists","up"]},{"osd":3,"uuid":"cbc678f4-83a9-45e6-8830-2f7c232b1ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6802","nonce":36090},{"type":"v1","addr":"172.21.2.102:6803","nonce":36090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6804","nonce":36090},{"type":"v1","addr":"172.21.2.102:6805","nonce":36090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6808","nonce":36090},{"type":"v1","addr":"172.21.2.102:6809","nonce":36090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6806","nonce":36090},{"type":"v1","addr":"172.21.2.102:6807","nonce":36090}]},"public_addr":"172.21.2.102:6803/36090","cluster_addr":"172.21.2.102:6805/36090","heartbeat_back_addr":"172.21.2.102:6809/36090","heartbeat_front_addr":"172.21.2.102:6807/36090","state":["exists","up"]},{"osd":4,"uuid":"c01e4008-b258-4f10-afd2-9e7f37214351","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6808","nonce":36368},{"type":"v1","addr":"172.21.2.113:6811","nonce":36368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6814","nonce":36368},{"type":"v1","addr":"172.21.2.113:6818","nonce":36368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6826","nonce":36368},{"type":"v1","addr":"172.21.2.113:6828","nonce":36368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6820","nonce":36368},{"type":"v1","addr":"172.21.2.113:6824","nonce":36368}]},"public_addr":"172.21.2.113:6811/36368","cluster_addr":"172.21.2.113:6818/36368","heartbeat_back_addr":"172.21.2.113:6828/36368","heartbeat_front_addr":"172.21.2.113:6824/36368","state":["exists","up"]},{"osd":5,"uuid":"551f6007-daf8-4a98-8ee0-c2b16bcccb5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6802","nonce":36366},{"type":"v1","addr":"172.21.2.113:6805","nonce":36366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6807","nonce":36366},{"type":"v1","addr":"172.21.2.113:6810","nonce":36366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6819","nonce":36366},{"type":"v1","addr":"172.21.2.113:6822","nonce":36366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6812","nonce":36366},{"type":"v1","addr":"172.21.2.113:6815","nonce":36366}]},"public_addr":"172.21.2.113:6805/36366","cluster_addr":"172.21.2.113:6810/36366","heartbeat_back_addr":"172.21.2.113:6822/36366","heartbeat_front_addr":"172.21.2.113:6815/36366","state":["exists","up"]},{"osd":6,"uuid":"bf67c040-0e50-48be-b102-c3943bc9ebef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6800","nonce":36367},{"type":"v1","addr":"172.21.2.113:6801","nonce":36367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6803","nonce":36367},{"type":"v1","addr":"172.21.2.113:6804","nonce":36367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6813","nonce":36367},{"type":"v1","addr":"172.21.2.113:6816","nonce":36367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6806","nonce":36367},{"type":"v1","addr":"172.21.2.113:6809","nonce":36367}]},"public_addr":"172.21.2.113:6801/36367","cluster_addr":"172.21.2.113:6804/36367","heartbeat_back_addr":"172.21.2.113:6816/36367","heartbeat_front_addr":"172.21.2.113:6809/36367","state":["exists","up"]},{"osd":7,"uuid":"f5461f80-018d-428f-be89-2eb2bcbe7bb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6817","nonce":36365},{"type":"v1","addr":"172.21.2.113:6821","nonce":36365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6823","nonce":36365},{"type":"v1","addr":"172.21.2.113:6825","nonce":36365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6830","nonce":36365},{"type":"v1","addr":"172.21.2.113:6831","nonce":36365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6827","nonce":36365},{"type":"v1","addr":"172.21.2.113:6829","nonce":36365}]},"public_addr":"172.21.2.113:6821/36365","cluster_addr":"172.21.2.113:6825/36365","heartbeat_back_addr":"172.21.2.113:6831/36365","heartbeat_front_addr":"172.21.2.113:6829/36365","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:03:43.989 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:43.990 7fac24a46700 1 -- 172.21.2.102:0/857076210 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fac040186d0 msgr2=0x7fac0401abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:43.989 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:43.990 7fac24a46700 1 --2- 172.21.2.102:0/857076210 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fac040186d0 0x7fac0401abd0 crc :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:43.989 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:43.990 7fac24a46700 1 -- 172.21.2.102:0/857076210 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fac2018acc0 msgr2=0x7fac20190070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:43.989 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:43.990 7fac24a46700 1 --2- 172.21.2.102:0/857076210 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fac2018acc0 0x7fac20190070 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7fac18006830 tx=0x7fac1800bcc0).stop 2021-02-10T11:03:43.990 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:43.990 7fac24a46700 1 -- 172.21.2.102:0/857076210 shutdown_connections 2021-02-10T11:03:43.990 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:43.990 7fac24a46700 1 --2- 172.21.2.102:0/857076210 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fac2018acc0 0x7fac20190070 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:43.990 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:43.990 7fac24a46700 1 --2- 172.21.2.102:0/857076210 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fac040186d0 0x7fac0401abd0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:43.990 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:43.990 7fac24a46700 1 --2- 172.21.2.102:0/857076210 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fac2018a2a0 0x7fac2018a6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:03:43.990 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:43.990 7fac24a46700 1 --2- 172.21.2.102:0/857076210 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fac201918e0 0x7fac20189c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:03:43.990 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:43.990 7fac24a46700 1 -- 172.21.2.102:0/857076210 >> 172.21.2.102:0/857076210 conn(0x7fac201a31d0 msgr2=0x7fac2008b830 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:03:43.990 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:43.990 7fac24a46700 1 -- 172.21.2.102:0/857076210 shutdown_connections 2021-02-10T11:03:43.990 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:43.990 7fac24a46700 1 -- 172.21.2.102:0/857076210 wait complete. 2021-02-10T11:03:43.998 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:03:44.422 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:44.407 7f029f000700 1 Processor -- start 2021-02-10T11:03:44.423 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:44.407 7f029f000700 1 -- start start 2021-02-10T11:03:44.423 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:44.408 7f029f000700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f02980e5070 0x7f02981b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:44.423 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:44.408 7f029f000700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f02981b3e40 0x7f029807d690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:44.423 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:44.408 7f029f000700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f029807ed80 con 0x7f02980e5470 2021-02-10T11:03:44.423 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:44.408 7f029f000700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0298080310 con 0x7f02980e5070 2021-02-10T11:03:44.423 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:44.408 7f029f000700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0298080530 con 0x7f02981b3e40 2021-02-10T11:03:44.423 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:44.408 7f029ce1c700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f02980e5070 0x7f02981b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:44.423 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:44.408 7f029ce1c700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f02980e5070 0x7f02981b3820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:49152/0 (socket says 172.21.2.102:49152) 2021-02-10T11:03:44.423 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:44.408 7f029ce1c700 1 -- 172.21.2.102:0/3708218249 learned_addr learned my addr 172.21.2.102:0/3708218249 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:03:44.423 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:44.408 7f029d61d700 1 --1- 172.21.2.102:0/3708218249 >> v1:172.21.2.113:6789/0 conn(0x7f02980e5470 0x7f029807dcb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.113:6789/0 says I am v1:172.21.2.102:48702/0 (socket says 172.21.2.102:48702) 2021-02-10T11:03:44.424 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:44.408 7f028ffff700 1 --2- 172.21.2.102:0/3708218249 >> v2:172.21.2.113:3300/0 conn(0x7f02981b3e40 0x7f029807d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:44.424 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:44.408 7f029ce1c700 1 -- 172.21.2.102:0/3708218249 >> v2:172.21.2.113:3300/0 conn(0x7f02981b3e40 msgr2=0x7f029807d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:03:44.424 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:44.408 7f029ce1c700 1 --2- 172.21.2.102:0/3708218249 >> v2:172.21.2.113:3300/0 conn(0x7f02981b3e40 0x7f029807d690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:44.424 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:44.408 7f029ce1c700 1 -- 172.21.2.102:0/3708218249 >> v1:172.21.2.113:6789/0 conn(0x7f02980e5470 legacy=0x7f029807dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:44.424 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:44.408 7f029ce1c700 1 -- 172.21.2.102:0/3708218249 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0298080750 con 0x7f02980e5070 2021-02-10T11:03:44.424 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:44.408 7f028ffff700 1 --2- 172.21.2.102:0/3708218249 >> v2:172.21.2.113:3300/0 conn(0x7f02981b3e40 0x7f029807d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:03:44.424 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:44.408 7f029ce1c700 1 --2- 172.21.2.102:0/3708218249 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f02980e5070 0x7f02981b3820 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7f0288006700 tx=0x7f0288009b80).ready entity=mon.0 client_cookie=bf2ce180ba610394 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:03:44.424 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:44.408 7f028f7fe700 1 -- 172.21.2.102:0/3708218249 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f028800c0c0 con 0x7f02980e5070 2021-02-10T11:03:44.424 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:44.408 7f028f7fe700 1 -- 172.21.2.102:0/3708218249 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f028800c440 con 0x7f02980e5070 2021-02-10T11:03:44.424 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:44.408 7f028f7fe700 1 -- 172.21.2.102:0/3708218249 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f028800c920 con 0x7f02980e5070 2021-02-10T11:03:44.425 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:44.409 7f029f000700 1 -- 172.21.2.102:0/3708218249 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f02980e5070 msgr2=0x7f02981b3820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:44.425 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:44.409 7f029f000700 1 --2- 172.21.2.102:0/3708218249 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f02980e5070 0x7f02981b3820 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7f0288006700 tx=0x7f0288009b80).stop 2021-02-10T11:03:44.425 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:44.409 7f029f000700 1 -- 172.21.2.102:0/3708218249 shutdown_connections 2021-02-10T11:03:44.425 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:44.409 7f029f000700 1 --2- 172.21.2.102:0/3708218249 >> v2:172.21.2.113:3300/0 conn(0x7f02981b3e40 0x7f029807d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:44.425 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:44.409 7f029f000700 1 --2- 172.21.2.102:0/3708218249 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f02980e5070 0x7f02981b3820 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:44.425 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:44.409 7f029f000700 1 -- 172.21.2.102:0/3708218249 >> 172.21.2.102:0/3708218249 conn(0x7f02980ce690 msgr2=0x7f02980e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:03:44.425 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:44.409 7f029f000700 1 -- 172.21.2.102:0/3708218249 shutdown_connections 2021-02-10T11:03:44.425 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:44.409 7f029f000700 1 -- 172.21.2.102:0/3708218249 wait complete. 2021-02-10T11:03:44.428 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:44.424 7f029f000700 1 Processor -- start 2021-02-10T11:03:44.428 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:44.424 7f029f000700 1 -- start start 2021-02-10T11:03:44.428 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:44.424 7f029f000700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0298093250 0x7f029819c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:44.428 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:44.424 7f029f000700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0298197fd0 0x7f02980b80e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:44.428 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:44.424 7f029f000700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f02980b8700 0x7f029819cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:44.429 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:44.424 7f029f000700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02981a2170 con 0x7f0298197fd0 2021-02-10T11:03:44.429 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:44.424 7f029f000700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02980b8b00 con 0x7f02980b8700 2021-02-10T11:03:44.429 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:44.424 7f029f000700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02980b8d20 con 0x7f0298093250 2021-02-10T11:03:44.429 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:44.425 7f028ffff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0298197fd0 0x7f02980b80e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:44.429 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:44.425 7f029d61d700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f02980b8700 0x7f029819cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:44.429 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:44.425 7f028ffff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0298197fd0 0x7f02980b80e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:49158/0 (socket says 172.21.2.102:49158) 2021-02-10T11:03:44.429 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:44.425 7f028ffff700 1 -- 172.21.2.102:0/3207241936 learned_addr learned my addr 172.21.2.102:0/3207241936 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:03:44.429 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:44.425 7f029ce1c700 1 --2- 172.21.2.102:0/3207241936 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0298093250 0x7f029819c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:44.429 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:44.425 7f029d61d700 1 -- 172.21.2.102:0/3207241936 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0298093250 msgr2=0x7f029819c2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:44.430 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:44.425 7f029d61d700 1 --2- 172.21.2.102:0/3207241936 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0298093250 0x7f029819c2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:44.430 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:44.425 7f029d61d700 1 -- 172.21.2.102:0/3207241936 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0298197fd0 msgr2=0x7f02980b80e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:44.430 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:44.425 7f029d61d700 1 --2- 172.21.2.102:0/3207241936 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0298197fd0 0x7f02980b80e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:44.430 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:44.425 7f029d61d700 1 -- 172.21.2.102:0/3207241936 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f029819d600 con 0x7f02980b8700 2021-02-10T11:03:44.430 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:44.425 7f029d61d700 1 --2- 172.21.2.102:0/3207241936 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f02980b8700 0x7f029819cfe0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f0290001b10 tx=0x7f029000aba0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:03:44.430 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:44.425 7f028d7fa700 1 -- 172.21.2.102:0/3207241936 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0290007280 con 0x7f02980b8700 2021-02-10T11:03:44.430 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:44.425 7f028d7fa700 1 -- 172.21.2.102:0/3207241936 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02900046a0 con 0x7f02980b8700 2021-02-10T11:03:44.430 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:44.425 7f029f000700 1 -- 172.21.2.102:0/3207241936 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f029819d600 con 0x7f02980b8700 2021-02-10T11:03:44.431 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:44.425 7f028d7fa700 1 -- 172.21.2.102:0/3207241936 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0290004b80 con 0x7f02980b8700 2021-02-10T11:03:44.431 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:44.425 7f029f000700 1 -- 172.21.2.102:0/3207241936 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f029819d600 con 0x7f02980b8700 2021-02-10T11:03:44.431 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:44.426 7f028d7fa700 1 -- 172.21.2.102:0/3207241936 <== mon.1 v2:172.21.2.113:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f0290004880 con 0x7f02980b8700 2021-02-10T11:03:44.431 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:44.426 7f028d7fa700 1 --2- 172.21.2.102:0/3207241936 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f02780186d0 0x7f027801abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:44.431 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:44.426 7f028d7fa700 1 -- 172.21.2.102:0/3207241936 <== mon.1 v2:172.21.2.113:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f02900328d0 con 0x7f02980b8700 2021-02-10T11:03:44.431 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:44.426 7f029ce1c700 1 --2- 172.21.2.102:0/3207241936 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f02780186d0 0x7f027801abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:44.431 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:44.426 7f029ce1c700 1 --2- 172.21.2.102:0/3207241936 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f02780186d0 0x7f027801abd0 crc :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:03:44.456 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:44.457 7f029f000700 1 -- 172.21.2.102:0/3207241936 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0298084b20 con 0x7f02980b8700 2021-02-10T11:03:44.459 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:44.461 7f028d7fa700 1 -- 172.21.2.102:0/3207241936 <== mon.1 v2:172.21.2.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f0290000cc0 con 0x7f02980b8700 2021-02-10T11:03:44.588 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:44.589 7f029f000700 1 -- 172.21.2.102:0/3207241936 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f029807ed80 con 0x7f02980b8700 2021-02-10T11:03:44.589 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:44.590 7f028d7fa700 1 -- 172.21.2.102:0/3207241936 <== mon.1 v2:172.21.2.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+13993 (secure 0 0 0) 0x7f02900046a0 con 0x7f02980b8700 2021-02-10T11:03:44.591 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:03:44.591 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":20,"fsid":"68db30ea-dbce-4aef-a6df-f01c6fea0513","created":"2021-02-10 11:03:06.331557","modified":"2021-02-10 11:03:44.386945","last_up_change":"2021-02-10 11:03:32.690562","last_in_change":"2021-02-10 11:03:32.690562","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:03:37.322519","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:03:41.161563","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2021-02-10 11:03:42.379651","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"be691b4d-a039-408a-af80-9d53f02b6f1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6818","nonce":36088},{"type":"v1","addr":"172.21.2.102:6820","nonce":36088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6822","nonce":36088},{"type":"v1","addr":"172.21.2.102:6824","nonce":36088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6830","nonce":36088},{"type":"v1","addr":"172.21.2.102:6832","nonce":36088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6826","nonce":36088},{"type":"v1","addr":"172.21.2.102:6828","nonce":36088}]},"public_addr":"172.21.2.102:6820/36088","cluster_addr":"172.21.2.102:6824/36088","heartbeat_back_addr":"172.21.2.102:6832/36088","heartbeat_front_addr":"172.21.2.102:6828/36088","state":["exists","up"]},{"osd":1,"uuid":"64e3ccc8-9c69-4d7c-865e-af8e83822635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6819","nonce":36087},{"type":"v1","addr":"172.21.2.102:6821","nonce":36087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6823","nonce":36087},{"type":"v1","addr":"172.21.2.102:6825","nonce":36087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6831","nonce":36087},{"type":"v1","addr":"172.21.2.102:6833","nonce":36087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6827","nonce":36087},{"type":"v1","addr":"172.21.2.102:6829","nonce":36087}]},"public_addr":"172.21.2.102:6821/36087","cluster_addr":"172.21.2.102:6825/36087","heartbeat_back_addr":"172.21.2.102:6833/36087","heartbeat_front_addr":"172.21.2.102:6829/36087","state":["exists","up"]},{"osd":2,"uuid":"153700af-747f-481f-bdf4-d6072d3a842f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6810","nonce":36089},{"type":"v1","addr":"172.21.2.102:6811","nonce":36089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6812","nonce":36089},{"type":"v1","addr":"172.21.2.102:6813","nonce":36089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6816","nonce":36089},{"type":"v1","addr":"172.21.2.102:6817","nonce":36089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6814","nonce":36089},{"type":"v1","addr":"172.21.2.102:6815","nonce":36089}]},"public_addr":"172.21.2.102:6811/36089","cluster_addr":"172.21.2.102:6813/36089","heartbeat_back_addr":"172.21.2.102:6817/36089","heartbeat_front_addr":"172.21.2.102:6815/36089","state":["exists","up"]},{"osd":3,"uuid":"cbc678f4-83a9-45e6-8830-2f7c232b1ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6802","nonce":36090},{"type":"v1","addr":"172.21.2.102:6803","nonce":36090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6804","nonce":36090},{"type":"v1","addr":"172.21.2.102:6805","nonce":36090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6808","nonce":36090},{"type":"v1","addr":"172.21.2.102:6809","nonce":36090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6806","nonce":36090},{"type":"v1","addr":"172.21.2.102:6807","nonce":36090}]},"public_addr":"172.21.2.102:6803/36090","cluster_addr":"172.21.2.102:6805/36090","heartbeat_back_addr":"172.21.2.102:6809/36090","heartbeat_front_addr":"172.21.2.102:6807/36090","state":["exists","up"]},{"osd":4,"uuid":"c01e4008-b258-4f10-afd2-9e7f37214351","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6808","nonce":36368},{"type":"v1","addr":"172.21.2.113:6811","nonce":36368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6814","nonce":36368},{"type":"v1","addr":"172.21.2.113:6818","nonce":36368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6826","nonce":36368},{"type":"v1","addr":"172.21.2.113:6828","nonce":36368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6820","nonce":36368},{"type":"v1","addr":"172.21.2.113:6824","nonce":36368}]},"public_addr":"172.21.2.113:6811/36368","cluster_addr":"172.21.2.113:6818/36368","heartbeat_back_addr":"172.21.2.113:6828/36368","heartbeat_front_addr":"172.21.2.113:6824/36368","state":["exists","up"]},{"osd":5,"uuid":"551f6007-daf8-4a98-8ee0-c2b16bcccb5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6802","nonce":36366},{"type":"v1","addr":"172.21.2.113:6805","nonce":36366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6807","nonce":36366},{"type":"v1","addr":"172.21.2.113:6810","nonce":36366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6819","nonce":36366},{"type":"v1","addr":"172.21.2.113:6822","nonce":36366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6812","nonce":36366},{"type":"v1","addr":"172.21.2.113:6815","nonce":36366}]},"public_addr":"172.21.2.113:6805/36366","cluster_addr":"172.21.2.113:6810/36366","heartbeat_back_addr":"172.21.2.113:6822/36366","heartbeat_front_addr":"172.21.2.113:6815/36366","state":["exists","up"]},{"osd":6,"uuid":"bf67c040-0e50-48be-b102-c3943bc9ebef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6800","nonce":36367},{"type":"v1","addr":"172.21.2.113:6801","nonce":36367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6803","nonce":36367},{"type":"v1","addr":"172.21.2.113:6804","nonce":36367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6813","nonce":36367},{"type":"v1","addr":"172.21.2.113:6816","nonce":36367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6806","nonce":36367},{"type":"v1","addr":"172.21.2.113:6809","nonce":36367}]},"public_addr":"172.21.2.113:6801/36367","cluster_addr":"172.21.2.113:6804/36367","heartbeat_back_addr":"172.21.2.113:6816/36367","heartbeat_front_addr":"172.21.2.113:6809/36367","state":["exists","up"]},{"osd":7,"uuid":"f5461f80-018d-428f-be89-2eb2bcbe7bb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6817","nonce":36365},{"type":"v1","addr":"172.21.2.113:6821","nonce":36365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6823","nonce":36365},{"type":"v1","addr":"172.21.2.113:6825","nonce":36365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6830","nonce":36365},{"type":"v1","addr":"172.21.2.113:6831","nonce":36365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6827","nonce":36365},{"type":"v1","addr":"172.21.2.113:6829","nonce":36365}]},"public_addr":"172.21.2.113:6821/36365","cluster_addr":"172.21.2.113:6825/36365","heartbeat_back_addr":"172.21.2.113:6831/36365","heartbeat_front_addr":"172.21.2.113:6829/36365","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:03:44.592 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:44.593 7f029f000700 1 -- 172.21.2.102:0/3207241936 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f02780186d0 msgr2=0x7f027801abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:44.592 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:44.593 7f029f000700 1 --2- 172.21.2.102:0/3207241936 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f02780186d0 0x7f027801abd0 crc :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:44.592 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:44.593 7f029f000700 1 -- 172.21.2.102:0/3207241936 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f02980b8700 msgr2=0x7f029819cfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:44.593 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:44.593 7f029f000700 1 --2- 172.21.2.102:0/3207241936 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f02980b8700 0x7f029819cfe0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f0290001b10 tx=0x7f029000aba0).stop 2021-02-10T11:03:44.593 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:44.593 7f029f000700 1 -- 172.21.2.102:0/3207241936 shutdown_connections 2021-02-10T11:03:44.593 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:44.593 7f029f000700 1 --2- 172.21.2.102:0/3207241936 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f02780186d0 0x7f027801abd0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:44.593 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:44.593 7f029f000700 1 --2- 172.21.2.102:0/3207241936 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f02980b8700 0x7f029819cfe0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:44.593 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:44.593 7f029f000700 1 --2- 172.21.2.102:0/3207241936 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0298197fd0 0x7f02980b80e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:44.593 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:44.593 7f029f000700 1 --2- 172.21.2.102:0/3207241936 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0298093250 0x7f029819c2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:44.593 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:44.593 7f029f000700 1 -- 172.21.2.102:0/3207241936 >> 172.21.2.102:0/3207241936 conn(0x7f0298197aa0 msgr2=0x7f0298084d50 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:03:44.593 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:44.593 7f029f000700 1 -- 172.21.2.102:0/3207241936 shutdown_connections 2021-02-10T11:03:44.593 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:44.594 7f029f000700 1 -- 172.21.2.102:0/3207241936 wait complete. 2021-02-10T11:03:44.601 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2021-02-10T11:03:45.031 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:45.015 7f7217f55700 1 Processor -- start 2021-02-10T11:03:45.031 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:45.015 7f7217f55700 1 -- start start 2021-02-10T11:03:45.031 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:45.016 7f7217f55700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f72101b0470 0x7f72101b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:45.031 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:45.016 7f7217f55700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f72100f20e0 0x7f72100f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:45.031 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:45.016 7f7217f55700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72100f0b50 con 0x7f72100ef610 2021-02-10T11:03:45.031 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:45.016 7f7217f55700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72100f0820 con 0x7f72101b0470 2021-02-10T11:03:45.031 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:45.016 7f7217f55700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72100f24e0 con 0x7f72100f20e0 2021-02-10T11:03:45.031 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:45.016 7f7215d71700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f72101b0470 0x7f72101b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:45.032 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:45.016 7f7215d71700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f72101b0470 0x7f72101b0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:49166/0 (socket says 172.21.2.102:49166) 2021-02-10T11:03:45.032 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:45.016 7f7215d71700 1 -- 172.21.2.102:0/1858854151 learned_addr learned my addr 172.21.2.102:0/1858854151 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:03:45.032 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:45.016 7f7216572700 1 --2- 172.21.2.102:0/1858854151 >> v2:172.21.2.113:3300/0 conn(0x7f72100f20e0 0x7f72100f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:45.032 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:45.016 7f7215d71700 1 -- 172.21.2.102:0/1858854151 >> v2:172.21.2.113:3300/0 conn(0x7f72100f20e0 msgr2=0x7f72100f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:03:45.032 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:45.016 7f7215d71700 1 --2- 172.21.2.102:0/1858854151 >> v2:172.21.2.113:3300/0 conn(0x7f72100f20e0 0x7f72100f0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:45.032 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:45.016 7f7215d71700 1 -- 172.21.2.102:0/1858854151 >> v1:172.21.2.113:6789/0 conn(0x7f72100ef610 legacy=0x7f72100efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:45.032 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:45.016 7f7215d71700 1 -- 172.21.2.102:0/1858854151 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72100e52e0 con 0x7f72101b0470 2021-02-10T11:03:45.032 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:45.017 7f7215d71700 1 --2- 172.21.2.102:0/1858854151 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f72101b0470 0x7f72101b0870 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7f720c006700 tx=0x7f720c009b80).ready entity=mon.0 client_cookie=4bfb2b1a7daa59b2 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:03:45.032 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:45.017 7f7214d6f700 1 -- 172.21.2.102:0/1858854151 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f720c00c0c0 con 0x7f72101b0470 2021-02-10T11:03:45.032 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:45.017 7f7214d6f700 1 -- 172.21.2.102:0/1858854151 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f720c00c340 con 0x7f72101b0470 2021-02-10T11:03:45.032 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:45.017 7f7214d6f700 1 -- 172.21.2.102:0/1858854151 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f720c00c820 con 0x7f72101b0470 2021-02-10T11:03:45.033 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:45.017 7f7217f55700 1 -- 172.21.2.102:0/1858854151 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f72101b0470 msgr2=0x7f72101b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:45.033 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:45.017 7f7217f55700 1 --2- 172.21.2.102:0/1858854151 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f72101b0470 0x7f72101b0870 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7f720c006700 tx=0x7f720c009b80).stop 2021-02-10T11:03:45.033 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:45.017 7f7217f55700 1 -- 172.21.2.102:0/1858854151 shutdown_connections 2021-02-10T11:03:45.033 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:45.017 7f7217f55700 1 --2- 172.21.2.102:0/1858854151 >> v2:172.21.2.113:3300/0 conn(0x7f72100f20e0 0x7f72100f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:45.033 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:45.017 7f7217f55700 1 --2- 172.21.2.102:0/1858854151 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f72101b0470 0x7f72101b0870 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:45.033 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:45.017 7f7217f55700 1 -- 172.21.2.102:0/1858854151 >> 172.21.2.102:0/1858854151 conn(0x7f72100ce3e0 msgr2=0x7f72100e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:03:45.033 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:45.018 7f7217f55700 1 -- 172.21.2.102:0/1858854151 shutdown_connections 2021-02-10T11:03:45.033 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:45.018 7f7217f55700 1 -- 172.21.2.102:0/1858854151 wait complete. 2021-02-10T11:03:45.035 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:45.032 7f7217f55700 1 Processor -- start 2021-02-10T11:03:45.035 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:45.032 7f7217f55700 1 -- start start 2021-02-10T11:03:45.035 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:45.032 7f7217f55700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f72100c3580 0x7f72100a7080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:45.035 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:45.032 7f7217f55700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7210105d10 0x7f7210106110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:45.035 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:45.032 7f7217f55700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7210106730 0x7f72100bff90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:45.035 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:45.032 7f7217f55700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72100c3980 con 0x7f72100c3580 2021-02-10T11:03:45.035 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:45.032 7f7217f55700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72100c05b0 con 0x7f7210105d10 2021-02-10T11:03:45.035 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:45.032 7f7217f55700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72100c07d0 con 0x7f7210106730 2021-02-10T11:03:45.035 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:45.033 7f7215d71700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f72100c3580 0x7f72100a7080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:45.035 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:45.033 7f7215d71700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f72100c3580 0x7f72100a7080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:49172/0 (socket says 172.21.2.102:49172) 2021-02-10T11:03:45.036 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:45.033 7f7215d71700 1 -- 172.21.2.102:0/2413506893 learned_addr learned my addr 172.21.2.102:0/2413506893 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:03:45.036 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:45.033 7f7215570700 1 --2- 172.21.2.102:0/2413506893 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7210105d10 0x7f7210106110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:45.036 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:45.033 7f7216572700 1 --2- 172.21.2.102:0/2413506893 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7210106730 0x7f72100bff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:45.036 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:45.033 7f7215d71700 1 -- 172.21.2.102:0/2413506893 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7210106730 msgr2=0x7f72100bff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:45.036 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:45.033 7f7215d71700 1 --2- 172.21.2.102:0/2413506893 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7210106730 0x7f72100bff90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:45.036 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:45.033 7f7215d71700 1 -- 172.21.2.102:0/2413506893 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7210105d10 msgr2=0x7f7210106110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:45.036 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:45.033 7f7215d71700 1 --2- 172.21.2.102:0/2413506893 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7210105d10 0x7f7210106110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:45.036 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:45.033 7f7215d71700 1 -- 172.21.2.102:0/2413506893 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72100c09f0 con 0x7f72100c3580 2021-02-10T11:03:45.036 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:45.033 7f7215d71700 1 --2- 172.21.2.102:0/2413506893 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f72100c3580 0x7f72100a7080 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7f720c00b5e0 tx=0x7f720c00b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:03:45.036 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:45.033 7f72027fc700 1 -- 172.21.2.102:0/2413506893 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f720c00bd70 con 0x7f72100c3580 2021-02-10T11:03:45.036 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:45.033 7f72027fc700 1 -- 172.21.2.102:0/2413506893 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f720c002030 con 0x7f72100c3580 2021-02-10T11:03:45.036 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:45.033 7f72027fc700 1 -- 172.21.2.102:0/2413506893 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f720c0024d0 con 0x7f72100c3580 2021-02-10T11:03:45.036 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:45.033 7f7217f55700 1 -- 172.21.2.102:0/2413506893 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72100c09f0 con 0x7f72100c3580 2021-02-10T11:03:45.037 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:45.033 7f7217f55700 1 -- 172.21.2.102:0/2413506893 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72100c09f0 con 0x7f72100c3580 2021-02-10T11:03:45.037 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:45.034 7f72027fc700 1 -- 172.21.2.102:0/2413506893 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f720c000c90 con 0x7f72100c3580 2021-02-10T11:03:45.037 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:45.034 7f72027fc700 1 --2- 172.21.2.102:0/2413506893 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f71f80186c0 0x7f71f801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:45.037 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:45.034 7f72027fc700 1 -- 172.21.2.102:0/2413506893 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f720c032550 con 0x7f72100c3580 2021-02-10T11:03:45.037 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:45.034 7f7215570700 1 --2- 172.21.2.102:0/2413506893 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f71f80186c0 0x7f71f801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:45.037 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:45.034 7f7215570700 1 --2- 172.21.2.102:0/2413506893 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f71f80186c0 0x7f71f801abc0 crc :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:03:45.065 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:45.066 7f7217f55700 1 -- 172.21.2.102:0/2413506893 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72100c3980 con 0x7f72100c3580 2021-02-10T11:03:45.067 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:45.068 7f72027fc700 1 -- 172.21.2.102:0/2413506893 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f720c030d00 con 0x7f72100c3580 2021-02-10T11:03:45.202 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:45.203 7f7217f55700 1 -- 172.21.2.102:0/2413506893 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"} v 0) v1 -- 0x7f72100c3980 con 0x7f72100c3580 2021-02-10T11:03:45.767 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:45.768 7f72027fc700 1 -- 172.21.2.102:0/2413506893 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"}]=0 v3) v1 ==== 118+0+0 (secure 0 0 0) 0x7f720c030d00 con 0x7f72100c3580 2021-02-10T11:03:45.773 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:45.774 7f7217f55700 1 -- 172.21.2.102:0/2413506893 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f71f80186c0 msgr2=0x7f71f801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:45.774 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:45.774 7f7217f55700 1 --2- 172.21.2.102:0/2413506893 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f71f80186c0 0x7f71f801abc0 crc :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:45.774 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:45.774 7f7217f55700 1 -- 172.21.2.102:0/2413506893 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f72100c3580 msgr2=0x7f72100a7080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:45.774 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:45.774 7f7217f55700 1 --2- 172.21.2.102:0/2413506893 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f72100c3580 0x7f72100a7080 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7f720c00b5e0 tx=0x7f720c00b980).stop 2021-02-10T11:03:45.774 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:45.774 7f7217f55700 1 -- 172.21.2.102:0/2413506893 shutdown_connections 2021-02-10T11:03:45.774 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:45.774 7f7217f55700 1 --2- 172.21.2.102:0/2413506893 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7210106730 0x7f72100bff90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:45.774 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:45.774 7f7217f55700 1 --2- 172.21.2.102:0/2413506893 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f71f80186c0 0x7f71f801abc0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:45.774 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:45.774 7f7217f55700 1 --2- 172.21.2.102:0/2413506893 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7210105d10 0x7f7210106110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:45.774 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:45.774 7f7217f55700 1 --2- 172.21.2.102:0/2413506893 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f72100c3580 0x7f72100a7080 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:45.774 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:45.774 7f7217f55700 1 -- 172.21.2.102:0/2413506893 >> 172.21.2.102:0/2413506893 conn(0x7f721010a430 msgr2=0x7f72100bf050 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:03:45.774 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:45.774 7f7217f55700 1 -- 172.21.2.102:0/2413506893 shutdown_connections 2021-02-10T11:03:45.774 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:45.774 7f7217f55700 1 -- 172.21.2.102:0/2413506893 wait complete. 2021-02-10T11:03:45.781 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:03:46.208 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:46.192 7f65e3778700 1 Processor -- start 2021-02-10T11:03:46.208 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:46.192 7f65e3778700 1 -- start start 2021-02-10T11:03:46.208 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:46.193 7f65e3778700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f65dc0e5070 0x7f65dc1b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:46.208 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:46.193 7f65e3778700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f65dc0e5470 0x7f65dc080310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:46.208 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:46.193 7f65e3778700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65dc07ed80 con 0x7f65dc1b3e40 2021-02-10T11:03:46.208 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:46.193 7f65e3778700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65dc0d1790 con 0x7f65dc0e5470 2021-02-10T11:03:46.208 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:46.193 7f65e3778700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65dc0d19b0 con 0x7f65dc0e5070 2021-02-10T11:03:46.208 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:46.193 7f65e0d93700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f65dc1b3e40 0x7f65dc07d690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:41402/0 (socket says 172.21.2.102:41402) 2021-02-10T11:03:46.209 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:46.193 7f65e0d93700 1 -- 172.21.2.102:0/653764510 learned_addr learned my addr 172.21.2.102:0/653764510 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:03:46.209 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:46.193 7f65e1d95700 1 --2- 172.21.2.102:0/653764510 >> v2:172.21.2.102:3300/0 conn(0x7f65dc0e5470 0x7f65dc080310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:46.209 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:46.193 7f65cffff700 1 -- 172.21.2.102:0/653764510 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1766044239 0 0) 0x7f65d00010b0 con 0x7f65dc1b3e40 2021-02-10T11:03:46.209 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:46.193 7f65cffff700 1 -- 172.21.2.102:0/653764510 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f65c8002fe0 con 0x7f65dc1b3e40 2021-02-10T11:03:46.209 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:46.193 7f65e1d95700 1 -- 172.21.2.102:0/653764510 >> v2:172.21.2.113:3300/0 conn(0x7f65dc0e5070 msgr2=0x7f65dc1b3820 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:03:46.209 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:46.193 7f65e1d95700 1 --2- 172.21.2.102:0/653764510 >> v2:172.21.2.113:3300/0 conn(0x7f65dc0e5070 0x7f65dc1b3820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:03:46.209 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:46.193 7f65e1d95700 1 -- 172.21.2.102:0/653764510 >> v1:172.21.2.102:6789/0 conn(0x7f65dc1b3e40 legacy=0x7f65dc07d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:46.209 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:46.193 7f65e1d95700 1 -- 172.21.2.102:0/653764510 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65dc0d1bd0 con 0x7f65dc0e5470 2021-02-10T11:03:46.209 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:46.194 7f65e1d95700 1 --2- 172.21.2.102:0/653764510 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f65dc0e5470 0x7f65dc080310 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f65d4006e30 tx=0x7f65d4009e60).ready entity=mon.0 client_cookie=7fe115baa530c479 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:03:46.209 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:46.194 7f65cffff700 1 -- 172.21.2.102:0/653764510 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f65d4006a50 con 0x7f65dc0e5470 2021-02-10T11:03:46.209 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:46.194 7f65cffff700 1 -- 172.21.2.102:0/653764510 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65d400b300 con 0x7f65dc0e5470 2021-02-10T11:03:46.209 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:46.194 7f65cffff700 1 -- 172.21.2.102:0/653764510 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f65d400b820 con 0x7f65dc0e5470 2021-02-10T11:03:46.209 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:46.194 7f65e3778700 1 -- 172.21.2.102:0/653764510 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f65dc0e5470 msgr2=0x7f65dc080310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:46.210 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:46.194 7f65e3778700 1 --2- 172.21.2.102:0/653764510 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f65dc0e5470 0x7f65dc080310 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f65d4006e30 tx=0x7f65d4009e60).stop 2021-02-10T11:03:46.210 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:46.194 7f65e3778700 1 -- 172.21.2.102:0/653764510 shutdown_connections 2021-02-10T11:03:46.210 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:46.194 7f65e3778700 1 --2- 172.21.2.102:0/653764510 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f65dc0e5470 0x7f65dc080310 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:46.210 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:46.194 7f65e3778700 1 --2- 172.21.2.102:0/653764510 >> v2:172.21.2.113:3300/0 conn(0x7f65dc0e5070 0x7f65dc1b3820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:03:46.210 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:46.194 7f65e3778700 1 -- 172.21.2.102:0/653764510 >> 172.21.2.102:0/653764510 conn(0x7f65dc0ce690 msgr2=0x7f65dc0e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:03:46.210 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:46.195 7f65e3778700 1 -- 172.21.2.102:0/653764510 shutdown_connections 2021-02-10T11:03:46.210 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:46.195 7f65e3778700 1 -- 172.21.2.102:0/653764510 wait complete. 2021-02-10T11:03:46.211 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:46.209 7f65e3778700 1 Processor -- start 2021-02-10T11:03:46.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:46.209 7f65e3778700 1 -- start start 2021-02-10T11:03:46.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:46.210 7f65e3778700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f65dc08d400 0x7f65dc096c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:46.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:46.210 7f65e3778700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f65dc0b1450 0x7f65dc0b1850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:46.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:46.210 7f65e3778700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f65dc0b1e70 0x7f65dc1b1f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:46.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:46.210 7f65e3778700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65dc097310 con 0x7f65dc0b1450 2021-02-10T11:03:46.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:46.210 7f65e3778700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65dc08d800 con 0x7f65dc0b1e70 2021-02-10T11:03:46.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:46.210 7f65e3778700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65dc08d9f0 con 0x7f65dc08d400 2021-02-10T11:03:46.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:46.210 7f65e0d93700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f65dc0b1450 0x7f65dc0b1850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:46.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:46.210 7f65e0d93700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f65dc0b1450 0x7f65dc0b1850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:49186/0 (socket says 172.21.2.102:49186) 2021-02-10T11:03:46.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:46.210 7f65e0d93700 1 -- 172.21.2.102:0/3140958338 learned_addr learned my addr 172.21.2.102:0/3140958338 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:03:46.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:46.210 7f65e0d93700 1 -- 172.21.2.102:0/3140958338 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f65dc08d400 msgr2=0x7f65dc096c70 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:03:46.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:46.210 7f65e0d93700 1 --2- 172.21.2.102:0/3140958338 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f65dc08d400 0x7f65dc096c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:03:46.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:46.210 7f65e0d93700 1 -- 172.21.2.102:0/3140958338 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f65dc0b1e70 msgr2=0x7f65dc1b1f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:46.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:46.210 7f65e0d93700 1 --2- 172.21.2.102:0/3140958338 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f65dc0b1e70 0x7f65dc1b1f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:03:46.213 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:46.210 7f65e0d93700 1 -- 172.21.2.102:0/3140958338 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65dc1b25f0 con 0x7f65dc0b1450 2021-02-10T11:03:46.213 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:46.210 7f65e0d93700 1 --2- 172.21.2.102:0/3140958338 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f65dc0b1450 0x7f65dc0b1850 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f65d0001c60 tx=0x7f65d000a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:03:46.213 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:46.210 7f65cdffb700 1 -- 172.21.2.102:0/3140958338 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f65d000acc0 con 0x7f65dc0b1450 2021-02-10T11:03:46.213 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:46.211 7f65cdffb700 1 -- 172.21.2.102:0/3140958338 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65d0004350 con 0x7f65dc0b1450 2021-02-10T11:03:46.213 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:46.211 7f65cdffb700 1 -- 172.21.2.102:0/3140958338 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f65d0004870 con 0x7f65dc0b1450 2021-02-10T11:03:46.213 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:46.211 7f65e3778700 1 -- 172.21.2.102:0/3140958338 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65dc1b25f0 con 0x7f65dc0b1450 2021-02-10T11:03:46.213 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:46.211 7f65e3778700 1 -- 172.21.2.102:0/3140958338 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65dc1b25f0 con 0x7f65dc0b1450 2021-02-10T11:03:46.213 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:46.211 7f65cdffb700 1 -- 172.21.2.102:0/3140958338 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f65d0004530 con 0x7f65dc0b1450 2021-02-10T11:03:46.213 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:46.211 7f65cdffb700 1 --2- 172.21.2.102:0/3140958338 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f65bc0186c0 0x7f65bc01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:46.213 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:46.211 7f65cdffb700 1 -- 172.21.2.102:0/3140958338 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f65d00327f0 con 0x7f65dc0b1450 2021-02-10T11:03:46.213 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:46.211 7f65e1594700 1 --2- 172.21.2.102:0/3140958338 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f65bc0186c0 0x7f65bc01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:46.213 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:46.212 7f65e1594700 1 --2- 172.21.2.102:0/3140958338 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f65bc0186c0 0x7f65bc01abc0 crc :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:03:46.242 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:46.243 7f65e3778700 1 -- 172.21.2.102:0/3140958338 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65dc07ed80 con 0x7f65dc0b1450 2021-02-10T11:03:46.244 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:46.246 7f65cdffb700 1 -- 172.21.2.102:0/3140958338 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f65d0004140 con 0x7f65dc0b1450 2021-02-10T11:03:46.373 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:46.373 7f65e3778700 1 -- 172.21.2.102:0/3140958338 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f65dc07ed80 con 0x7f65dc0b1450 2021-02-10T11:03:46.373 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:46.374 7f65cdffb700 1 -- 172.21.2.102:0/3140958338 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+1415 (secure 0 0 0) 0x7f65d0004350 con 0x7f65dc0b1450 2021-02-10T11:03:46.373 INFO:teuthology.orchestra.run.gibba002.stderr:dumped fsmap epoch 3 2021-02-10T11:03:46.374 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:03:46.374 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":3,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":3,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:03:43.384709","modified":"2021-02-10 11:03:45.763508","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[],"up":{},"failed":[],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2021-02-10T11:03:46.375 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:46.376 7f65e3778700 1 -- 172.21.2.102:0/3140958338 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f65bc0186c0 msgr2=0x7f65bc01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:46.375 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:46.376 7f65e3778700 1 --2- 172.21.2.102:0/3140958338 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f65bc0186c0 0x7f65bc01abc0 crc :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:46.375 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:46.376 7f65e3778700 1 -- 172.21.2.102:0/3140958338 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f65dc0b1450 msgr2=0x7f65dc0b1850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:46.375 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:46.376 7f65e3778700 1 --2- 172.21.2.102:0/3140958338 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f65dc0b1450 0x7f65dc0b1850 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f65d0001c60 tx=0x7f65d000a8d0).stop 2021-02-10T11:03:46.375 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:46.376 7f65e3778700 1 -- 172.21.2.102:0/3140958338 shutdown_connections 2021-02-10T11:03:46.375 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:46.376 7f65e3778700 1 --2- 172.21.2.102:0/3140958338 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f65bc0186c0 0x7f65bc01abc0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:46.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:46.376 7f65e3778700 1 --2- 172.21.2.102:0/3140958338 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f65dc0b1e70 0x7f65dc1b1f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:03:46.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:46.376 7f65e3778700 1 --2- 172.21.2.102:0/3140958338 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f65dc0b1450 0x7f65dc0b1850 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:46.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:46.376 7f65e3778700 1 --2- 172.21.2.102:0/3140958338 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f65dc08d400 0x7f65dc096c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:03:46.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:46.376 7f65e3778700 1 -- 172.21.2.102:0/3140958338 >> 172.21.2.102:0/3140958338 conn(0x7f65dc1ac620 msgr2=0x7f65dc0896e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:03:46.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:46.376 7f65e3778700 1 -- 172.21.2.102:0/3140958338 shutdown_connections 2021-02-10T11:03:46.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:46.376 7f65e3778700 1 -- 172.21.2.102:0/3140958338 wait complete. 2021-02-10T11:03:46.384 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:03:46.810 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:46.795 7f2dbdd5d700 1 Processor -- start 2021-02-10T11:03:46.810 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:46.795 7f2dbdd5d700 1 -- start start 2021-02-10T11:03:46.810 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:46.795 7f2dbdd5d700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f2db80e5070 0x7f2db81b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:46.811 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:46.795 7f2dbdd5d700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f2db80e5470 0x7f2db8080310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:46.811 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:46.795 7f2dbdd5d700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2db807ed80 con 0x7f2db81b3e40 2021-02-10T11:03:46.811 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:46.795 7f2dbdd5d700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2db80d1790 con 0x7f2db80e5070 2021-02-10T11:03:46.811 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:46.795 7f2dbdd5d700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2db80d19b0 con 0x7f2db80e5470 2021-02-10T11:03:46.811 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:46.796 7f2db77fe700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f2db80e5070 0x7f2db81b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:46.811 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:46.796 7f2db77fe700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f2db80e5070 0x7f2db81b3820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:49190/0 (socket says 172.21.2.102:49190) 2021-02-10T11:03:46.811 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:46.796 7f2db77fe700 1 -- 172.21.2.102:0/3735138590 learned_addr learned my addr 172.21.2.102:0/3735138590 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:03:46.811 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:46.796 7f2db7fff700 1 --2- 172.21.2.102:0/3735138590 >> v2:172.21.2.113:3300/0 conn(0x7f2db80e5470 0x7f2db8080310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:46.811 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:46.796 7f2db77fe700 1 -- 172.21.2.102:0/3735138590 >> v2:172.21.2.113:3300/0 conn(0x7f2db80e5470 msgr2=0x7f2db8080310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:03:46.811 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:46.796 7f2db77fe700 1 --2- 172.21.2.102:0/3735138590 >> v2:172.21.2.113:3300/0 conn(0x7f2db80e5470 0x7f2db8080310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:46.811 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:46.796 7f2db77fe700 1 -- 172.21.2.102:0/3735138590 >> v1:172.21.2.113:6789/0 conn(0x7f2db81b3e40 legacy=0x7f2db807d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:03:46.811 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:46.796 7f2db77fe700 1 -- 172.21.2.102:0/3735138590 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2db80d1bd0 con 0x7f2db80e5070 2021-02-10T11:03:46.811 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:46.796 7f2db77fe700 1 --2- 172.21.2.102:0/3735138590 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2db80e5070 0x7f2db81b3820 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f2da8006700 tx=0x7f2da8009b00).ready entity=mon.0 client_cookie=98f38ec258773428 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:03:46.812 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:46.796 7f2db6ffd700 1 -- 172.21.2.102:0/3735138590 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2da800c0c0 con 0x7f2db80e5070 2021-02-10T11:03:46.812 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:46.796 7f2db6ffd700 1 -- 172.21.2.102:0/3735138590 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2da800c400 con 0x7f2db80e5070 2021-02-10T11:03:46.812 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:46.796 7f2db6ffd700 1 -- 172.21.2.102:0/3735138590 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2da800c8e0 con 0x7f2db80e5070 2021-02-10T11:03:46.812 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:46.797 7f2dbdd5d700 1 -- 172.21.2.102:0/3735138590 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2db80e5070 msgr2=0x7f2db81b3820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:46.812 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:46.797 7f2dbdd5d700 1 --2- 172.21.2.102:0/3735138590 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2db80e5070 0x7f2db81b3820 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f2da8006700 tx=0x7f2da8009b00).stop 2021-02-10T11:03:46.812 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:46.797 7f2dbdd5d700 1 -- 172.21.2.102:0/3735138590 shutdown_connections 2021-02-10T11:03:46.812 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:46.797 7f2dbdd5d700 1 --2- 172.21.2.102:0/3735138590 >> v2:172.21.2.113:3300/0 conn(0x7f2db80e5470 0x7f2db8080310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:46.812 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:46.797 7f2dbdd5d700 1 --2- 172.21.2.102:0/3735138590 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2db80e5070 0x7f2db81b3820 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:46.812 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:46.797 7f2dbdd5d700 1 -- 172.21.2.102:0/3735138590 >> 172.21.2.102:0/3735138590 conn(0x7f2db80ce690 msgr2=0x7f2db80e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:03:46.812 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:46.797 7f2dbdd5d700 1 -- 172.21.2.102:0/3735138590 shutdown_connections 2021-02-10T11:03:46.812 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:46.797 7f2dbdd5d700 1 -- 172.21.2.102:0/3735138590 wait complete. 2021-02-10T11:03:46.813 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:46.812 7f2dbdd5d700 1 Processor -- start 2021-02-10T11:03:46.814 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:46.812 7f2dbdd5d700 1 -- start start 2021-02-10T11:03:46.814 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:46.812 7f2dbdd5d700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2db8191bb0 0x7f2db80a2380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:46.814 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:46.812 7f2dbdd5d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2db80a29a0 0x7f2db80a2da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:46.814 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:46.812 7f2dbdd5d700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2db80a33c0 0x7f2db80a03d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:46.814 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:46.812 7f2dbdd5d700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2db80adf90 con 0x7f2db80a29a0 2021-02-10T11:03:46.814 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:46.812 7f2dbdd5d700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2db80a1e00 con 0x7f2db80a33c0 2021-02-10T11:03:46.814 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:46.812 7f2dbdd5d700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2db80a09f0 con 0x7f2db8191bb0 2021-02-10T11:03:46.814 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:46.812 7f2daeffd700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2db80a29a0 0x7f2db80a2da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:46.814 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:46.812 7f2daeffd700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2db80a29a0 0x7f2db80a2da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:49196/0 (socket says 172.21.2.102:49196) 2021-02-10T11:03:46.814 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:46.812 7f2daeffd700 1 -- 172.21.2.102:0/3115601193 learned_addr learned my addr 172.21.2.102:0/3115601193 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:03:46.814 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:46.813 7f2daeffd700 1 -- 172.21.2.102:0/3115601193 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2db8191bb0 msgr2=0x7f2db80a2380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:46.814 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:46.813 7f2daeffd700 1 --2- 172.21.2.102:0/3115601193 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2db8191bb0 0x7f2db80a2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:03:46.815 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:46.813 7f2daeffd700 1 -- 172.21.2.102:0/3115601193 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2db80a33c0 msgr2=0x7f2db80a03d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:46.815 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:46.813 7f2daeffd700 1 --2- 172.21.2.102:0/3115601193 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2db80a33c0 0x7f2db80a03d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:03:46.815 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:46.813 7f2daeffd700 1 -- 172.21.2.102:0/3115601193 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2db80a0bd0 con 0x7f2db80a29a0 2021-02-10T11:03:46.815 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:46.813 7f2daeffd700 1 --2- 172.21.2.102:0/3115601193 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2db80a29a0 0x7f2db80a2da0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f2da0006ed0 tx=0x7f2da000b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:03:46.815 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:46.813 7f2db4ff9700 1 -- 172.21.2.102:0/3115601193 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2da0006020 con 0x7f2db80a29a0 2021-02-10T11:03:46.815 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:46.813 7f2db4ff9700 1 -- 172.21.2.102:0/3115601193 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2da00009b0 con 0x7f2db80a29a0 2021-02-10T11:03:46.815 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:46.813 7f2dbdd5d700 1 -- 172.21.2.102:0/3115601193 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2db80a0bd0 con 0x7f2db80a29a0 2021-02-10T11:03:46.815 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:46.813 7f2db4ff9700 1 -- 172.21.2.102:0/3115601193 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2da00054b0 con 0x7f2db80a29a0 2021-02-10T11:03:46.815 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:46.813 7f2dbdd5d700 1 -- 172.21.2.102:0/3115601193 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2db80a1160 con 0x7f2db80a29a0 2021-02-10T11:03:46.815 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:46.814 7f2db4ff9700 1 -- 172.21.2.102:0/3115601193 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f2da0006020 con 0x7f2db80a29a0 2021-02-10T11:03:46.815 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:46.814 7f2db4ff9700 1 --2- 172.21.2.102:0/3115601193 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2d9c0186c0 0x7f2d9c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:46.815 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:46.814 7f2db4ff9700 1 -- 172.21.2.102:0/3115601193 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f2da00323f0 con 0x7f2db80a29a0 2021-02-10T11:03:46.815 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:46.814 7f2db77fe700 1 --2- 172.21.2.102:0/3115601193 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2d9c0186c0 0x7f2d9c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:46.816 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:46.814 7f2db77fe700 1 --2- 172.21.2.102:0/3115601193 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2d9c0186c0 0x7f2d9c01abc0 crc :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:03:46.845 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:46.846 7f2dbdd5d700 1 -- 172.21.2.102:0/3115601193 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2db807ed80 con 0x7f2db80a29a0 2021-02-10T11:03:46.847 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:46.848 7f2db4ff9700 1 -- 172.21.2.102:0/3115601193 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f2da002eef0 con 0x7f2db80a29a0 2021-02-10T11:03:46.971 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:46.972 7f2dbdd5d700 1 -- 172.21.2.102:0/3115601193 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2db80adf90 con 0x7f2db80a29a0 2021-02-10T11:03:46.972 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:46.973 7f2db4ff9700 1 -- 172.21.2.102:0/3115601193 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+13993 (secure 0 0 0) 0x7f2da002eef0 con 0x7f2db80a29a0 2021-02-10T11:03:46.972 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:03:46.972 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":20,"fsid":"68db30ea-dbce-4aef-a6df-f01c6fea0513","created":"2021-02-10 11:03:06.331557","modified":"2021-02-10 11:03:44.386945","last_up_change":"2021-02-10 11:03:32.690562","last_in_change":"2021-02-10 11:03:32.690562","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:03:37.322519","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:03:41.161563","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2021-02-10 11:03:42.379651","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"be691b4d-a039-408a-af80-9d53f02b6f1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6818","nonce":36088},{"type":"v1","addr":"172.21.2.102:6820","nonce":36088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6822","nonce":36088},{"type":"v1","addr":"172.21.2.102:6824","nonce":36088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6830","nonce":36088},{"type":"v1","addr":"172.21.2.102:6832","nonce":36088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6826","nonce":36088},{"type":"v1","addr":"172.21.2.102:6828","nonce":36088}]},"public_addr":"172.21.2.102:6820/36088","cluster_addr":"172.21.2.102:6824/36088","heartbeat_back_addr":"172.21.2.102:6832/36088","heartbeat_front_addr":"172.21.2.102:6828/36088","state":["exists","up"]},{"osd":1,"uuid":"64e3ccc8-9c69-4d7c-865e-af8e83822635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6819","nonce":36087},{"type":"v1","addr":"172.21.2.102:6821","nonce":36087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6823","nonce":36087},{"type":"v1","addr":"172.21.2.102:6825","nonce":36087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6831","nonce":36087},{"type":"v1","addr":"172.21.2.102:6833","nonce":36087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6827","nonce":36087},{"type":"v1","addr":"172.21.2.102:6829","nonce":36087}]},"public_addr":"172.21.2.102:6821/36087","cluster_addr":"172.21.2.102:6825/36087","heartbeat_back_addr":"172.21.2.102:6833/36087","heartbeat_front_addr":"172.21.2.102:6829/36087","state":["exists","up"]},{"osd":2,"uuid":"153700af-747f-481f-bdf4-d6072d3a842f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6810","nonce":36089},{"type":"v1","addr":"172.21.2.102:6811","nonce":36089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6812","nonce":36089},{"type":"v1","addr":"172.21.2.102:6813","nonce":36089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6816","nonce":36089},{"type":"v1","addr":"172.21.2.102:6817","nonce":36089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6814","nonce":36089},{"type":"v1","addr":"172.21.2.102:6815","nonce":36089}]},"public_addr":"172.21.2.102:6811/36089","cluster_addr":"172.21.2.102:6813/36089","heartbeat_back_addr":"172.21.2.102:6817/36089","heartbeat_front_addr":"172.21.2.102:6815/36089","state":["exists","up"]},{"osd":3,"uuid":"cbc678f4-83a9-45e6-8830-2f7c232b1ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6802","nonce":36090},{"type":"v1","addr":"172.21.2.102:6803","nonce":36090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6804","nonce":36090},{"type":"v1","addr":"172.21.2.102:6805","nonce":36090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6808","nonce":36090},{"type":"v1","addr":"172.21.2.102:6809","nonce":36090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6806","nonce":36090},{"type":"v1","addr":"172.21.2.102:6807","nonce":36090}]},"public_addr":"172.21.2.102:6803/36090","cluster_addr":"172.21.2.102:6805/36090","heartbeat_back_addr":"172.21.2.102:6809/36090","heartbeat_front_addr":"172.21.2.102:6807/36090","state":["exists","up"]},{"osd":4,"uuid":"c01e4008-b258-4f10-afd2-9e7f37214351","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6808","nonce":36368},{"type":"v1","addr":"172.21.2.113:6811","nonce":36368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6814","nonce":36368},{"type":"v1","addr":"172.21.2.113:6818","nonce":36368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6826","nonce":36368},{"type":"v1","addr":"172.21.2.113:6828","nonce":36368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6820","nonce":36368},{"type":"v1","addr":"172.21.2.113:6824","nonce":36368}]},"public_addr":"172.21.2.113:6811/36368","cluster_addr":"172.21.2.113:6818/36368","heartbeat_back_addr":"172.21.2.113:6828/36368","heartbeat_front_addr":"172.21.2.113:6824/36368","state":["exists","up"]},{"osd":5,"uuid":"551f6007-daf8-4a98-8ee0-c2b16bcccb5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6802","nonce":36366},{"type":"v1","addr":"172.21.2.113:6805","nonce":36366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6807","nonce":36366},{"type":"v1","addr":"172.21.2.113:6810","nonce":36366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6819","nonce":36366},{"type":"v1","addr":"172.21.2.113:6822","nonce":36366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6812","nonce":36366},{"type":"v1","addr":"172.21.2.113:6815","nonce":36366}]},"public_addr":"172.21.2.113:6805/36366","cluster_addr":"172.21.2.113:6810/36366","heartbeat_back_addr":"172.21.2.113:6822/36366","heartbeat_front_addr":"172.21.2.113:6815/36366","state":["exists","up"]},{"osd":6,"uuid":"bf67c040-0e50-48be-b102-c3943bc9ebef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6800","nonce":36367},{"type":"v1","addr":"172.21.2.113:6801","nonce":36367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6803","nonce":36367},{"type":"v1","addr":"172.21.2.113:6804","nonce":36367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6813","nonce":36367},{"type":"v1","addr":"172.21.2.113:6816","nonce":36367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6806","nonce":36367},{"type":"v1","addr":"172.21.2.113:6809","nonce":36367}]},"public_addr":"172.21.2.113:6801/36367","cluster_addr":"172.21.2.113:6804/36367","heartbeat_back_addr":"172.21.2.113:6816/36367","heartbeat_front_addr":"172.21.2.113:6809/36367","state":["exists","up"]},{"osd":7,"uuid":"f5461f80-018d-428f-be89-2eb2bcbe7bb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6817","nonce":36365},{"type":"v1","addr":"172.21.2.113:6821","nonce":36365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6823","nonce":36365},{"type":"v1","addr":"172.21.2.113:6825","nonce":36365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6830","nonce":36365},{"type":"v1","addr":"172.21.2.113:6831","nonce":36365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6827","nonce":36365},{"type":"v1","addr":"172.21.2.113:6829","nonce":36365}]},"public_addr":"172.21.2.113:6821/36365","cluster_addr":"172.21.2.113:6825/36365","heartbeat_back_addr":"172.21.2.113:6831/36365","heartbeat_front_addr":"172.21.2.113:6829/36365","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:03:46.974 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:46.974 7f2dbdd5d700 1 -- 172.21.2.102:0/3115601193 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2d9c0186c0 msgr2=0x7f2d9c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:46.974 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:46.974 7f2dbdd5d700 1 --2- 172.21.2.102:0/3115601193 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2d9c0186c0 0x7f2d9c01abc0 crc :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:46.974 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:46.974 7f2dbdd5d700 1 -- 172.21.2.102:0/3115601193 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2db80a29a0 msgr2=0x7f2db80a2da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:46.974 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:46.974 7f2dbdd5d700 1 --2- 172.21.2.102:0/3115601193 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2db80a29a0 0x7f2db80a2da0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f2da0006ed0 tx=0x7f2da000b960).stop 2021-02-10T11:03:46.974 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:46.975 7f2dbdd5d700 1 -- 172.21.2.102:0/3115601193 shutdown_connections 2021-02-10T11:03:46.974 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:46.975 7f2dbdd5d700 1 --2- 172.21.2.102:0/3115601193 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2d9c0186c0 0x7f2d9c01abc0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:46.974 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:46.975 7f2dbdd5d700 1 --2- 172.21.2.102:0/3115601193 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2db80a33c0 0x7f2db80a03d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:03:46.974 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:46.975 7f2dbdd5d700 1 --2- 172.21.2.102:0/3115601193 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2db80a29a0 0x7f2db80a2da0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:46.974 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:46.975 7f2dbdd5d700 1 --2- 172.21.2.102:0/3115601193 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2db8191bb0 0x7f2db80a2380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:03:46.975 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:46.975 7f2dbdd5d700 1 -- 172.21.2.102:0/3115601193 >> 172.21.2.102:0/3115601193 conn(0x7f2db81a31d0 msgr2=0x7f2db808b830 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:03:46.975 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:46.975 7f2dbdd5d700 1 -- 172.21.2.102:0/3115601193 shutdown_connections 2021-02-10T11:03:46.975 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:46.975 7f2dbdd5d700 1 -- 172.21.2.102:0/3115601193 wait complete. 2021-02-10T11:03:46.982 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2021-02-10T11:03:46.982 INFO:tasks.ceph.mds.a:Restarting daemon 2021-02-10T11:03:46.982 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2021-02-10T11:03:46.985 INFO:tasks.ceph.mds.a:Started 2021-02-10T11:03:46.985 INFO:tasks.ceph.mds.b:Restarting daemon 2021-02-10T11:03:46.985 DEBUG:teuthology.orchestra.run.gibba013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2021-02-10T11:03:46.988 INFO:tasks.ceph.mds.b:Started 2021-02-10T11:03:46.988 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:03:47.394 INFO:tasks.ceph.mds.b.gibba013.stdout:starting mds.b at 2021-02-10T11:03:47.395 INFO:tasks.ceph.mds.a.gibba002.stdout:starting mds.a at 2021-02-10T11:03:47.418 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:47.396 7f820ba3b700 1 Processor -- start 2021-02-10T11:03:47.418 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:47.397 7f820ba3b700 1 -- start start 2021-02-10T11:03:47.418 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:47.397 7f820ba3b700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f82040d17f0 0x7f82041b31e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:47.419 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:47.397 7f820ba3b700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f82040d1bf0 0x7f82041b3800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:47.419 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:47.397 7f820ba3b700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f82041b3e20 0x7f82040ceef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:47.419 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:47.397 7f820ba3b700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82040e5eb0 con 0x7f82040d1bf0 2021-02-10T11:03:47.419 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:47.397 7f820ba3b700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82040f0b20 con 0x7f82040d17f0 2021-02-10T11:03:47.419 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:47.397 7f820ba3b700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82040cf510 con 0x7f82041b3e20 2021-02-10T11:03:47.420 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:47.397 7f8209056700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f82040d1bf0 0x7f82041b3800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:47.420 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:47.397 7f8209056700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f82040d1bf0 0x7f82041b3800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:49212/0 (socket says 172.21.2.102:49212) 2021-02-10T11:03:47.420 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:47.397 7f8209056700 1 -- 172.21.2.102:0/785100593 learned_addr learned my addr 172.21.2.102:0/785100593 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:03:47.420 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:47.398 7f8209857700 1 --2- 172.21.2.102:0/785100593 >> v2:172.21.2.113:3300/0 conn(0x7f82040d17f0 0x7f82041b31e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:47.421 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:47.398 7f820a058700 1 --2- 172.21.2.102:0/785100593 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f82041b3e20 0x7f82040ceef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:47.421 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:47.398 7f8209056700 1 -- 172.21.2.102:0/785100593 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f82041b3e20 msgr2=0x7f82040ceef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:03:47.421 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:47.398 7f8209056700 1 --2- 172.21.2.102:0/785100593 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f82041b3e20 0x7f82040ceef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:47.421 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:47.398 7f8209056700 1 -- 172.21.2.102:0/785100593 >> v2:172.21.2.113:3300/0 conn(0x7f82040d17f0 msgr2=0x7f82041b31e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:03:47.421 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:47.398 7f8209056700 1 --2- 172.21.2.102:0/785100593 >> v2:172.21.2.113:3300/0 conn(0x7f82040d17f0 0x7f82041b31e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:47.421 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:47.398 7f8209056700 1 -- 172.21.2.102:0/785100593 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82040cf6f0 con 0x7f82040d1bf0 2021-02-10T11:03:47.422 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:47.398 7f8209056700 1 --2- 172.21.2.102:0/785100593 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f82040d1bf0 0x7f82041b3800 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f8200006700 tx=0x7f8200009b00).ready entity=mon.0 client_cookie=a97f3452168dd0c1 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:03:47.422 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:47.398 7f8208855700 1 -- 172.21.2.102:0/785100593 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f820000c0c0 con 0x7f82040d1bf0 2021-02-10T11:03:47.422 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:47.398 7f8208855700 1 -- 172.21.2.102:0/785100593 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f820000c400 con 0x7f82040d1bf0 2021-02-10T11:03:47.422 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:47.398 7f8208855700 1 -- 172.21.2.102:0/785100593 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f820000c8e0 con 0x7f82040d1bf0 2021-02-10T11:03:47.422 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:47.398 7f820ba3b700 1 -- 172.21.2.102:0/785100593 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f82040d1bf0 msgr2=0x7f82041b3800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:47.423 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:47.398 7f820ba3b700 1 --2- 172.21.2.102:0/785100593 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f82040d1bf0 0x7f82041b3800 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f8200006700 tx=0x7f8200009b00).stop 2021-02-10T11:03:47.423 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:47.398 7f820ba3b700 1 -- 172.21.2.102:0/785100593 shutdown_connections 2021-02-10T11:03:47.423 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:47.398 7f820ba3b700 1 --2- 172.21.2.102:0/785100593 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f82041b3e20 0x7f82040ceef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:47.423 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:47.398 7f820ba3b700 1 --2- 172.21.2.102:0/785100593 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f82040d1bf0 0x7f82041b3800 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:47.423 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:47.398 7f820ba3b700 1 --2- 172.21.2.102:0/785100593 >> v2:172.21.2.113:3300/0 conn(0x7f82040d17f0 0x7f82041b31e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:47.424 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:47.398 7f820ba3b700 1 -- 172.21.2.102:0/785100593 >> 172.21.2.102:0/785100593 conn(0x7f82040fde40 msgr2=0x7f82040ce390 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:03:47.424 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:47.399 7f820ba3b700 1 -- 172.21.2.102:0/785100593 shutdown_connections 2021-02-10T11:03:47.424 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:47.399 7f820ba3b700 1 -- 172.21.2.102:0/785100593 wait complete. 2021-02-10T11:03:47.425 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:47.413 7f820ba3b700 1 Processor -- start 2021-02-10T11:03:47.425 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:47.413 7f820ba3b700 1 -- start start 2021-02-10T11:03:47.425 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:47.414 7f820ba3b700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8204107ee0 0x7f820408e870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:47.425 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:47.414 7f820ba3b700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f820408ee90 0x7f820408f290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:47.425 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:47.414 7f820ba3b700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f82040b6d20 0x7f82040b7120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:47.425 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:47.414 7f820ba3b700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82040b67a0 con 0x7f8204107ee0 2021-02-10T11:03:47.426 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:47.414 7f820ba3b700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82041082e0 con 0x7f82040b6d20 2021-02-10T11:03:47.427 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:47.414 7f820ba3b700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f820408f8b0 con 0x7f820408ee90 2021-02-10T11:03:47.427 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:47.414 7f8209857700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8204107ee0 0x7f820408e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:47.427 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:47.414 7f8209857700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8204107ee0 0x7f820408e870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:49220/0 (socket says 172.21.2.102:49220) 2021-02-10T11:03:47.427 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:47.414 7f8209857700 1 -- 172.21.2.102:0/1613297181 learned_addr learned my addr 172.21.2.102:0/1613297181 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:03:47.427 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:47.414 7f8209056700 1 --2- 172.21.2.102:0/1613297181 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f820408ee90 0x7f820408f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:47.427 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:47.414 7f8209857700 1 -- 172.21.2.102:0/1613297181 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f820408ee90 msgr2=0x7f820408f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:47.427 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:47.414 7f8209857700 1 --2- 172.21.2.102:0/1613297181 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f820408ee90 0x7f820408f290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:47.427 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:47.414 7f8209857700 1 -- 172.21.2.102:0/1613297181 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f82040b6d20 msgr2=0x7f82040b7120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:47.427 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:47.414 7f8209857700 1 --2- 172.21.2.102:0/1613297181 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f82040b6d20 0x7f82040b7120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:03:47.427 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:47.414 7f8209857700 1 -- 172.21.2.102:0/1613297181 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82040b7740 con 0x7f8204107ee0 2021-02-10T11:03:47.428 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:47.414 7f8209056700 1 --2- 172.21.2.102:0/1613297181 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f820408ee90 0x7f820408f290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:03:47.428 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:47.414 7f8209857700 1 --2- 172.21.2.102:0/1613297181 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8204107ee0 0x7f820408e870 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f81f4006f00 tx=0x7f81f400b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:03:47.428 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:47.415 7f81fa7fc700 1 -- 172.21.2.102:0/1613297181 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f81f400bd30 con 0x7f8204107ee0 2021-02-10T11:03:47.428 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:47.415 7f81fa7fc700 1 -- 172.21.2.102:0/1613297181 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81f4002030 con 0x7f8204107ee0 2021-02-10T11:03:47.428 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:47.415 7f820ba3b700 1 -- 172.21.2.102:0/1613297181 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82040b7740 con 0x7f8204107ee0 2021-02-10T11:03:47.428 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:47.415 7f81fa7fc700 1 -- 172.21.2.102:0/1613297181 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f81f4002550 con 0x7f8204107ee0 2021-02-10T11:03:47.428 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:47.415 7f820ba3b700 1 -- 172.21.2.102:0/1613297181 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82040b7740 con 0x7f8204107ee0 2021-02-10T11:03:47.428 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:47.415 7f81fa7fc700 1 -- 172.21.2.102:0/1613297181 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f81f4002210 con 0x7f8204107ee0 2021-02-10T11:03:47.428 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:47.415 7f81fa7fc700 1 --2- 172.21.2.102:0/1613297181 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f81ec018730 0x7f81ec01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:47.428 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:47.415 7f81fa7fc700 1 -- 172.21.2.102:0/1613297181 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f81f40327e0 con 0x7f8204107ee0 2021-02-10T11:03:47.428 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:47.415 7f8209056700 1 --2- 172.21.2.102:0/1613297181 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f81ec018730 0x7f81ec01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:47.428 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:47.416 7f8209056700 1 --2- 172.21.2.102:0/1613297181 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f81ec018730 0x7f81ec01ac10 crc :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:03:47.446 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:47.446 7f820ba3b700 1 -- 172.21.2.102:0/1613297181 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82041036a0 con 0x7f8204107ee0 2021-02-10T11:03:47.448 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:47.449 7f81fa7fc700 1 -- 172.21.2.102:0/1613297181 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f81f4002240 con 0x7f8204107ee0 2021-02-10T11:03:47.575 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:47.576 7f820ba3b700 1 -- 172.21.2.102:0/1613297181 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f82041036a0 con 0x7f8204107ee0 2021-02-10T11:03:47.576 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:47.577 7f81fa7fc700 1 -- 172.21.2.102:0/1613297181 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+13993 (secure 0 0 0) 0x7f81f4002030 con 0x7f8204107ee0 2021-02-10T11:03:47.576 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:03:47.576 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":20,"fsid":"68db30ea-dbce-4aef-a6df-f01c6fea0513","created":"2021-02-10 11:03:06.331557","modified":"2021-02-10 11:03:44.386945","last_up_change":"2021-02-10 11:03:32.690562","last_in_change":"2021-02-10 11:03:32.690562","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:03:37.322519","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:03:41.161563","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2021-02-10 11:03:42.379651","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"be691b4d-a039-408a-af80-9d53f02b6f1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6818","nonce":36088},{"type":"v1","addr":"172.21.2.102:6820","nonce":36088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6822","nonce":36088},{"type":"v1","addr":"172.21.2.102:6824","nonce":36088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6830","nonce":36088},{"type":"v1","addr":"172.21.2.102:6832","nonce":36088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6826","nonce":36088},{"type":"v1","addr":"172.21.2.102:6828","nonce":36088}]},"public_addr":"172.21.2.102:6820/36088","cluster_addr":"172.21.2.102:6824/36088","heartbeat_back_addr":"172.21.2.102:6832/36088","heartbeat_front_addr":"172.21.2.102:6828/36088","state":["exists","up"]},{"osd":1,"uuid":"64e3ccc8-9c69-4d7c-865e-af8e83822635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6819","nonce":36087},{"type":"v1","addr":"172.21.2.102:6821","nonce":36087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6823","nonce":36087},{"type":"v1","addr":"172.21.2.102:6825","nonce":36087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6831","nonce":36087},{"type":"v1","addr":"172.21.2.102:6833","nonce":36087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6827","nonce":36087},{"type":"v1","addr":"172.21.2.102:6829","nonce":36087}]},"public_addr":"172.21.2.102:6821/36087","cluster_addr":"172.21.2.102:6825/36087","heartbeat_back_addr":"172.21.2.102:6833/36087","heartbeat_front_addr":"172.21.2.102:6829/36087","state":["exists","up"]},{"osd":2,"uuid":"153700af-747f-481f-bdf4-d6072d3a842f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6810","nonce":36089},{"type":"v1","addr":"172.21.2.102:6811","nonce":36089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6812","nonce":36089},{"type":"v1","addr":"172.21.2.102:6813","nonce":36089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6816","nonce":36089},{"type":"v1","addr":"172.21.2.102:6817","nonce":36089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6814","nonce":36089},{"type":"v1","addr":"172.21.2.102:6815","nonce":36089}]},"public_addr":"172.21.2.102:6811/36089","cluster_addr":"172.21.2.102:6813/36089","heartbeat_back_addr":"172.21.2.102:6817/36089","heartbeat_front_addr":"172.21.2.102:6815/36089","state":["exists","up"]},{"osd":3,"uuid":"cbc678f4-83a9-45e6-8830-2f7c232b1ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6802","nonce":36090},{"type":"v1","addr":"172.21.2.102:6803","nonce":36090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6804","nonce":36090},{"type":"v1","addr":"172.21.2.102:6805","nonce":36090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6808","nonce":36090},{"type":"v1","addr":"172.21.2.102:6809","nonce":36090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6806","nonce":36090},{"type":"v1","addr":"172.21.2.102:6807","nonce":36090}]},"public_addr":"172.21.2.102:6803/36090","cluster_addr":"172.21.2.102:6805/36090","heartbeat_back_addr":"172.21.2.102:6809/36090","heartbeat_front_addr":"172.21.2.102:6807/36090","state":["exists","up"]},{"osd":4,"uuid":"c01e4008-b258-4f10-afd2-9e7f37214351","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6808","nonce":36368},{"type":"v1","addr":"172.21.2.113:6811","nonce":36368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6814","nonce":36368},{"type":"v1","addr":"172.21.2.113:6818","nonce":36368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6826","nonce":36368},{"type":"v1","addr":"172.21.2.113:6828","nonce":36368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6820","nonce":36368},{"type":"v1","addr":"172.21.2.113:6824","nonce":36368}]},"public_addr":"172.21.2.113:6811/36368","cluster_addr":"172.21.2.113:6818/36368","heartbeat_back_addr":"172.21.2.113:6828/36368","heartbeat_front_addr":"172.21.2.113:6824/36368","state":["exists","up"]},{"osd":5,"uuid":"551f6007-daf8-4a98-8ee0-c2b16bcccb5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6802","nonce":36366},{"type":"v1","addr":"172.21.2.113:6805","nonce":36366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6807","nonce":36366},{"type":"v1","addr":"172.21.2.113:6810","nonce":36366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6819","nonce":36366},{"type":"v1","addr":"172.21.2.113:6822","nonce":36366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6812","nonce":36366},{"type":"v1","addr":"172.21.2.113:6815","nonce":36366}]},"public_addr":"172.21.2.113:6805/36366","cluster_addr":"172.21.2.113:6810/36366","heartbeat_back_addr":"172.21.2.113:6822/36366","heartbeat_front_addr":"172.21.2.113:6815/36366","state":["exists","up"]},{"osd":6,"uuid":"bf67c040-0e50-48be-b102-c3943bc9ebef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6800","nonce":36367},{"type":"v1","addr":"172.21.2.113:6801","nonce":36367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6803","nonce":36367},{"type":"v1","addr":"172.21.2.113:6804","nonce":36367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6813","nonce":36367},{"type":"v1","addr":"172.21.2.113:6816","nonce":36367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6806","nonce":36367},{"type":"v1","addr":"172.21.2.113:6809","nonce":36367}]},"public_addr":"172.21.2.113:6801/36367","cluster_addr":"172.21.2.113:6804/36367","heartbeat_back_addr":"172.21.2.113:6816/36367","heartbeat_front_addr":"172.21.2.113:6809/36367","state":["exists","up"]},{"osd":7,"uuid":"f5461f80-018d-428f-be89-2eb2bcbe7bb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6817","nonce":36365},{"type":"v1","addr":"172.21.2.113:6821","nonce":36365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6823","nonce":36365},{"type":"v1","addr":"172.21.2.113:6825","nonce":36365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6830","nonce":36365},{"type":"v1","addr":"172.21.2.113:6831","nonce":36365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6827","nonce":36365},{"type":"v1","addr":"172.21.2.113:6829","nonce":36365}]},"public_addr":"172.21.2.113:6821/36365","cluster_addr":"172.21.2.113:6825/36365","heartbeat_back_addr":"172.21.2.113:6831/36365","heartbeat_front_addr":"172.21.2.113:6829/36365","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:03:47.578 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:47.578 7f820ba3b700 1 -- 172.21.2.102:0/1613297181 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f81ec018730 msgr2=0x7f81ec01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:47.578 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:47.579 7f820ba3b700 1 --2- 172.21.2.102:0/1613297181 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f81ec018730 0x7f81ec01ac10 crc :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:47.578 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:47.579 7f820ba3b700 1 -- 172.21.2.102:0/1613297181 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8204107ee0 msgr2=0x7f820408e870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:47.578 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:47.579 7f820ba3b700 1 --2- 172.21.2.102:0/1613297181 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8204107ee0 0x7f820408e870 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f81f4006f00 tx=0x7f81f400b940).stop 2021-02-10T11:03:47.578 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:47.579 7f820ba3b700 1 -- 172.21.2.102:0/1613297181 shutdown_connections 2021-02-10T11:03:47.578 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:47.579 7f820ba3b700 1 --2- 172.21.2.102:0/1613297181 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f81ec018730 0x7f81ec01ac10 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:47.578 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:47.579 7f820ba3b700 1 --2- 172.21.2.102:0/1613297181 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f82040b6d20 0x7f82040b7120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:03:47.578 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:47.579 7f820ba3b700 1 --2- 172.21.2.102:0/1613297181 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f820408ee90 0x7f820408f290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:47.578 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:47.579 7f820ba3b700 1 --2- 172.21.2.102:0/1613297181 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8204107ee0 0x7f820408e870 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:47.578 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:47.579 7f820ba3b700 1 -- 172.21.2.102:0/1613297181 >> 172.21.2.102:0/1613297181 conn(0x7f820409ca80 msgr2=0x7f82040c3c60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:03:47.579 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:47.579 7f820ba3b700 1 -- 172.21.2.102:0/1613297181 shutdown_connections 2021-02-10T11:03:47.579 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:47.579 7f820ba3b700 1 -- 172.21.2.102:0/1613297181 wait complete. 2021-02-10T11:03:47.586 INFO:tasks.ceph.ceph_manager.ceph:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:03:37.322519', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2021-02-10 11:03:41.161563', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2021-02-10 11:03:42.379651', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-02-10T11:03:47.587 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:03:48.016 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:48.000 7f70901aa700 1 Processor -- start 2021-02-10T11:03:48.182 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:48.000 7f70901aa700 1 -- start start 2021-02-10T11:03:48.182 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:48.001 7f70901aa700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f70881b0870 0x7f70880f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:48.182 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:48.001 7f70901aa700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f70880efe00 0x7f70880f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:48.182 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:48.001 7f70901aa700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70881b0c70 con 0x7f70881b0470 2021-02-10T11:03:48.182 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:48.001 7f70901aa700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70880f0b50 con 0x7f70880efe00 2021-02-10T11:03:48.182 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:48.001 7f70901aa700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70880f0820 con 0x7f70881b0870 2021-02-10T11:03:48.182 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:48.001 7f708e7c7700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f70880efe00 0x7f70880f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:48.182 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:48.001 7f708e7c7700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f70880efe00 0x7f70880f0200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:49230/0 (socket says 172.21.2.102:49230) 2021-02-10T11:03:48.183 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:48.001 7f708e7c7700 1 -- 172.21.2.102:0/587528000 learned_addr learned my addr 172.21.2.102:0/587528000 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:03:48.183 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:48.001 7f708d7c5700 1 --2- 172.21.2.102:0/587528000 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f70881b0870 0x7f70880f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:48.183 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:48.002 7f708e7c7700 1 -- 172.21.2.102:0/587528000 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f70881b0870 msgr2=0x7f70880f20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:03:48.183 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:48.002 7f708e7c7700 1 --2- 172.21.2.102:0/587528000 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f70881b0870 0x7f70880f20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:48.183 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:48.002 7f708e7c7700 1 -- 172.21.2.102:0/587528000 >> v1:172.21.2.102:6789/0 conn(0x7f70881b0470 legacy=0x7f70880ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:48.183 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:48.002 7f708e7c7700 1 -- 172.21.2.102:0/587528000 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70880e52e0 con 0x7f70880efe00 2021-02-10T11:03:48.183 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:48.002 7f708d7c5700 1 --2- 172.21.2.102:0/587528000 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f70881b0870 0x7f70880f20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:03:48.183 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:48.002 7f708e7c7700 1 --2- 172.21.2.102:0/587528000 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f70880efe00 0x7f70880f0200 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f7080006f00 tx=0x7f7080009f10).ready entity=mon.0 client_cookie=f4cfe52f266319d6 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:03:48.183 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:48.002 7f708cfc4700 1 -- 172.21.2.102:0/587528000 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f708000b3b0 con 0x7f70880efe00 2021-02-10T11:03:48.183 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:48.002 7f708cfc4700 1 -- 172.21.2.102:0/587528000 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f708000b630 con 0x7f70880efe00 2021-02-10T11:03:48.183 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:48.002 7f708cfc4700 1 -- 172.21.2.102:0/587528000 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f708000bb50 con 0x7f70880efe00 2021-02-10T11:03:48.183 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:48.002 7f70901aa700 1 -- 172.21.2.102:0/587528000 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f70880efe00 msgr2=0x7f70880f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:48.183 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:48.002 7f70901aa700 1 --2- 172.21.2.102:0/587528000 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f70880efe00 0x7f70880f0200 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f7080006f00 tx=0x7f7080009f10).stop 2021-02-10T11:03:48.184 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:48.002 7f70901aa700 1 -- 172.21.2.102:0/587528000 shutdown_connections 2021-02-10T11:03:48.184 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:48.002 7f70901aa700 1 --2- 172.21.2.102:0/587528000 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f70880efe00 0x7f70880f0200 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:48.184 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:48.002 7f70901aa700 1 --2- 172.21.2.102:0/587528000 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f70881b0870 0x7f70880f20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:48.184 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:48.002 7f70901aa700 1 -- 172.21.2.102:0/587528000 >> 172.21.2.102:0/587528000 conn(0x7f70880ce3e0 msgr2=0x7f70880e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:03:48.184 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:48.003 7f70901aa700 1 -- 172.21.2.102:0/587528000 shutdown_connections 2021-02-10T11:03:48.184 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:48.003 7f70901aa700 1 -- 172.21.2.102:0/587528000 wait complete. 2021-02-10T11:03:48.186 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:48.017 7f70901aa700 1 Processor -- start 2021-02-10T11:03:48.186 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:48.017 7f70901aa700 1 -- start start 2021-02-10T11:03:48.186 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:48.018 7f70901aa700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f708819e260 0x7f708808e550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:48.187 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:48.018 7f70901aa700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7088091900 0x7f7088091d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:48.187 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:48.018 7f70901aa700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f70880b9c40 0x7f70880ba040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:48.187 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:48.018 7f70901aa700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7088092440 con 0x7f708819e260 2021-02-10T11:03:48.187 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:48.018 7f70901aa700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70880ba660 con 0x7f7088091900 2021-02-10T11:03:48.187 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:48.018 7f70901aa700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70880ba880 con 0x7f70880b9c40 2021-02-10T11:03:48.187 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:48.018 7f708dfc6700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f708819e260 0x7f708808e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:48.187 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:48.018 7f708dfc6700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f708819e260 0x7f708808e550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:49232/0 (socket says 172.21.2.102:49232) 2021-02-10T11:03:48.187 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:48.018 7f708dfc6700 1 -- 172.21.2.102:0/4162157434 learned_addr learned my addr 172.21.2.102:0/4162157434 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:03:48.187 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:48.018 7f708dfc6700 1 -- 172.21.2.102:0/4162157434 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f70880b9c40 msgr2=0x7f70880ba040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:48.187 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:48.018 7f708dfc6700 1 --2- 172.21.2.102:0/4162157434 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f70880b9c40 0x7f70880ba040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:03:48.187 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:48.018 7f708dfc6700 1 -- 172.21.2.102:0/4162157434 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7088091900 msgr2=0x7f7088091d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:48.187 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:48.018 7f708dfc6700 1 --2- 172.21.2.102:0/4162157434 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7088091900 0x7f7088091d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:03:48.188 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:48.018 7f708dfc6700 1 -- 172.21.2.102:0/4162157434 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70880baaa0 con 0x7f708819e260 2021-02-10T11:03:48.188 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:48.018 7f708dfc6700 1 --2- 172.21.2.102:0/4162157434 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f708819e260 0x7f708808e550 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f707c006ed0 tx=0x7f707c00b8e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:03:48.188 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:48.019 7f707a7fc700 1 -- 172.21.2.102:0/4162157434 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f707c006240 con 0x7f708819e260 2021-02-10T11:03:48.188 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:48.019 7f707a7fc700 1 -- 172.21.2.102:0/4162157434 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f707c0054b0 con 0x7f708819e260 2021-02-10T11:03:48.188 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:48.019 7f707a7fc700 1 -- 172.21.2.102:0/4162157434 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f707c0059d0 con 0x7f708819e260 2021-02-10T11:03:48.188 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:48.019 7f70901aa700 1 -- 172.21.2.102:0/4162157434 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70880baaa0 con 0x7f708819e260 2021-02-10T11:03:48.188 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:48.019 7f70901aa700 1 -- 172.21.2.102:0/4162157434 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70880baaa0 con 0x7f708819e260 2021-02-10T11:03:48.188 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:48.020 7f707a7fc700 1 -- 172.21.2.102:0/4162157434 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f707c006030 con 0x7f708819e260 2021-02-10T11:03:48.188 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:48.020 7f707a7fc700 1 --2- 172.21.2.102:0/4162157434 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f707001cad0 0x7f707001eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:48.188 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:48.020 7f707a7fc700 1 -- 172.21.2.102:0/4162157434 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f707c0325b0 con 0x7f708819e260 2021-02-10T11:03:48.188 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:48.020 7f708d7c5700 1 --2- 172.21.2.102:0/4162157434 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f707001cad0 0x7f707001eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:48.188 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:48.020 7f708d7c5700 1 --2- 172.21.2.102:0/4162157434 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f707001cad0 0x7f707001eff0 crc :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:03:48.188 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:48.051 7f70901aa700 1 -- 172.21.2.102:0/4162157434 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70880f0b50 con 0x7f708819e260 2021-02-10T11:03:48.189 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:48.053 7f707a7fc700 1 -- 172.21.2.102:0/4162157434 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f707c000de0 con 0x7f708819e260 2021-02-10T11:03:48.189 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:48.181 7f70901aa700 1 -- 172.21.2.102:0/4162157434 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f70880f0b50 con 0x7f708819e260 2021-02-10T11:03:48.189 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:48.182 7f707a7fc700 1 -- 172.21.2.102:0/4162157434 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v20) v1 ==== 92+0+10 (secure 0 0 0) 0x7f707c006030 con 0x7f708819e260 2021-02-10T11:03:48.189 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:48.184 7f70901aa700 1 -- 172.21.2.102:0/4162157434 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f707001cad0 msgr2=0x7f707001eff0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:48.189 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:48.184 7f70901aa700 1 --2- 172.21.2.102:0/4162157434 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f707001cad0 0x7f707001eff0 crc :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:48.189 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:48.184 7f70901aa700 1 -- 172.21.2.102:0/4162157434 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f708819e260 msgr2=0x7f708808e550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:48.189 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:48.184 7f70901aa700 1 --2- 172.21.2.102:0/4162157434 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f708819e260 0x7f708808e550 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f707c006ed0 tx=0x7f707c00b8e0).stop 2021-02-10T11:03:48.189 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:48.184 7f70901aa700 1 -- 172.21.2.102:0/4162157434 shutdown_connections 2021-02-10T11:03:48.189 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:48.184 7f70901aa700 1 --2- 172.21.2.102:0/4162157434 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f70880b9c40 0x7f70880ba040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:03:48.189 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:48.184 7f70901aa700 1 --2- 172.21.2.102:0/4162157434 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f707001cad0 0x7f707001eff0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:48.189 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:48.184 7f70901aa700 1 --2- 172.21.2.102:0/4162157434 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7088091900 0x7f7088091d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:03:48.189 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:48.184 7f70901aa700 1 --2- 172.21.2.102:0/4162157434 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f708819e260 0x7f708808e550 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:48.189 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:48.184 7f70901aa700 1 -- 172.21.2.102:0/4162157434 >> 172.21.2.102:0/4162157434 conn(0x7f70880bd060 msgr2=0x7f708809a020 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:03:48.190 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:48.184 7f70901aa700 1 -- 172.21.2.102:0/4162157434 shutdown_connections 2021-02-10T11:03:48.190 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:48.184 7f70901aa700 1 -- 172.21.2.102:0/4162157434 wait complete. 2021-02-10T11:03:48.190 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:03:48.191 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-02-10T11:03:48.628 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:48.613 7f4f6168e700 1 Processor -- start 2021-02-10T11:03:48.628 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:48.613 7f4f6168e700 1 -- start start 2021-02-10T11:03:48.628 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:48.613 7f4f6168e700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f4f5c1b0470 0x7f4f5c1b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:48.628 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:48.613 7f4f6168e700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f5c0f0b50 con 0x7f4f5c0ef610 2021-02-10T11:03:48.629 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:48.613 7f4f6168e700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f5c0f24e0 con 0x7f4f5c0f20e0 2021-02-10T11:03:48.629 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:48.613 7f4f6168e700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f5c0e5ea0 con 0x7f4f5c1b0470 2021-02-10T11:03:48.629 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:48.614 7f4f5b2bf700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f4f5c1b0470 0x7f4f5c1b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:48.629 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:48.614 7f4f5aabe700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f4f5c0ef610 0x7f4f5c0efa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:41464/0 (socket says 172.21.2.102:41464) 2021-02-10T11:03:48.629 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:48.614 7f4f5aabe700 1 -- 172.21.2.102:0/325549512 learned_addr learned my addr 172.21.2.102:0/325549512 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:03:48.629 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:48.614 7f4f5b2bf700 1 -- 172.21.2.102:0/325549512 >> v1:172.21.2.113:6789/0 conn(0x7f4f5c0f20e0 legacy=0x7f4f5c0f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:03:48.629 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:48.614 7f4f5b2bf700 1 -- 172.21.2.102:0/325549512 >> v1:172.21.2.102:6789/0 conn(0x7f4f5c0ef610 legacy=0x7f4f5c0efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:48.629 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:48.614 7f4f5a2bd700 1 -- 172.21.2.102:0/325549512 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3626735810 0 0) 0x7f4f4c0009b0 con 0x7f4f5c0ef610 2021-02-10T11:03:48.629 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:48.614 7f4f5b2bf700 1 -- 172.21.2.102:0/325549512 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f5c0e52e0 con 0x7f4f5c1b0470 2021-02-10T11:03:48.629 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:48.614 7f4f5b2bf700 1 --2- 172.21.2.102:0/325549512 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4f5c1b0470 0x7f4f5c1b0870 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7f4f44006650 tx=0x7f4f44009a00).ready entity=mon.0 client_cookie=8abd5f07026066e1 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:03:48.629 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:48.615 7f4f5a2bd700 1 -- 172.21.2.102:0/325549512 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4f44009df0 con 0x7f4f5c1b0470 2021-02-10T11:03:48.629 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:48.615 7f4f5a2bd700 1 -- 172.21.2.102:0/325549512 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f440009b0 con 0x7f4f5c1b0470 2021-02-10T11:03:48.629 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:48.615 7f4f5a2bd700 1 -- 172.21.2.102:0/325549512 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4f4400c2b0 con 0x7f4f5c1b0470 2021-02-10T11:03:48.629 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:48.615 7f4f6168e700 1 -- 172.21.2.102:0/325549512 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4f5c1b0470 msgr2=0x7f4f5c1b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:48.630 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:48.615 7f4f6168e700 1 --2- 172.21.2.102:0/325549512 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4f5c1b0470 0x7f4f5c1b0870 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7f4f44006650 tx=0x7f4f44009a00).stop 2021-02-10T11:03:48.630 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:48.615 7f4f6168e700 1 -- 172.21.2.102:0/325549512 shutdown_connections 2021-02-10T11:03:48.630 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:48.615 7f4f6168e700 1 --2- 172.21.2.102:0/325549512 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4f5c1b0470 0x7f4f5c1b0870 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:48.630 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:48.615 7f4f6168e700 1 -- 172.21.2.102:0/325549512 >> 172.21.2.102:0/325549512 conn(0x7f4f5c0ce3e0 msgr2=0x7f4f5c0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:03:48.630 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:48.615 7f4f6168e700 1 -- 172.21.2.102:0/325549512 shutdown_connections 2021-02-10T11:03:48.630 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:48.615 7f4f6168e700 1 -- 172.21.2.102:0/325549512 wait complete. 2021-02-10T11:03:48.631 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:48.629 7f4f6168e700 1 Processor -- start 2021-02-10T11:03:48.631 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:48.630 7f4f6168e700 1 -- start start 2021-02-10T11:03:48.632 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:48.630 7f4f6168e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4f5c0bdbf0 0x7f4f5c1a7450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:48.632 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:48.630 7f4f6168e700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4f5c0bdff0 0x7f4f5c1a7a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:48.632 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:48.630 7f4f6168e700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4f5c0abcc0 0x7f4f5c0ac0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:48.632 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:48.630 7f4f6168e700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f5c0be590 con 0x7f4f5c0bdbf0 2021-02-10T11:03:48.632 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:48.630 7f4f6168e700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f5c1a8090 con 0x7f4f5c0bdff0 2021-02-10T11:03:48.632 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:48.630 7f4f6168e700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f5c1a82b0 con 0x7f4f5c0abcc0 2021-02-10T11:03:48.632 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:48.630 7f4f5b2bf700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4f5c0bdbf0 0x7f4f5c1a7450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:48.632 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:48.630 7f4f5b2bf700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4f5c0bdbf0 0x7f4f5c1a7450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:49248/0 (socket says 172.21.2.102:49248) 2021-02-10T11:03:48.632 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:48.630 7f4f5b2bf700 1 -- 172.21.2.102:0/1764044986 learned_addr learned my addr 172.21.2.102:0/1764044986 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:03:48.632 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:48.630 7f4f5b2bf700 1 -- 172.21.2.102:0/1764044986 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4f5c0abcc0 msgr2=0x7f4f5c0ac0c0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:03:48.632 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:48.630 7f4f5b2bf700 1 --2- 172.21.2.102:0/1764044986 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4f5c0abcc0 0x7f4f5c0ac0c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:03:48.632 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:48.630 7f4f5b2bf700 1 -- 172.21.2.102:0/1764044986 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4f5c0bdff0 msgr2=0x7f4f5c1a7a70 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:03:48.632 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:48.630 7f4f5b2bf700 1 --2- 172.21.2.102:0/1764044986 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4f5c0bdff0 0x7f4f5c1a7a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:03:48.632 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:48.630 7f4f5b2bf700 1 -- 172.21.2.102:0/1764044986 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f5c0ac750 con 0x7f4f5c0bdbf0 2021-02-10T11:03:48.633 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:48.631 7f4f5b2bf700 1 --2- 172.21.2.102:0/1764044986 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4f5c0bdbf0 0x7f4f5c1a7450 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f4f440096a0 tx=0x7f4f4400b9b0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:03:48.633 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:48.631 7f4f53fff700 1 -- 172.21.2.102:0/1764044986 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4f440093a0 con 0x7f4f5c0bdbf0 2021-02-10T11:03:48.633 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:48.631 7f4f53fff700 1 -- 172.21.2.102:0/1764044986 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f4400bda0 con 0x7f4f5c0bdbf0 2021-02-10T11:03:48.633 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:48.631 7f4f53fff700 1 -- 172.21.2.102:0/1764044986 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4f44002080 con 0x7f4f5c0bdbf0 2021-02-10T11:03:48.633 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:48.631 7f4f6168e700 1 -- 172.21.2.102:0/1764044986 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f5c0ac750 con 0x7f4f5c0bdbf0 2021-02-10T11:03:48.633 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:48.631 7f4f6168e700 1 -- 172.21.2.102:0/1764044986 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f5c0ad2d0 con 0x7f4f5c0bdbf0 2021-02-10T11:03:48.633 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:48.632 7f4f53fff700 1 -- 172.21.2.102:0/1764044986 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4f44000c90 con 0x7f4f5c0bdbf0 2021-02-10T11:03:48.633 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:48.632 7f4f53fff700 1 --2- 172.21.2.102:0/1764044986 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4f400186d0 0x7f4f4001abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:48.633 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:48.632 7f4f53fff700 1 -- 172.21.2.102:0/1764044986 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f4f44032780 con 0x7f4f5c0bdbf0 2021-02-10T11:03:48.633 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:48.632 7f4f5aabe700 1 --2- 172.21.2.102:0/1764044986 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4f400186d0 0x7f4f4001abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:48.633 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:48.632 7f4f5aabe700 1 --2- 172.21.2.102:0/1764044986 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4f400186d0 0x7f4f4001abd0 crc :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:03:48.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:48.663 7f4f6168e700 1 -- 172.21.2.102:0/1764044986 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f5c0edcc0 con 0x7f4f5c0bdbf0 2021-02-10T11:03:48.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:48.666 7f4f53fff700 1 -- 172.21.2.102:0/1764044986 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f4f440093a0 con 0x7f4f5c0bdbf0 2021-02-10T11:03:48.792 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:48.793 7f4f6168e700 1 -- 172.21.2.102:0/1764044986 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7f4f5c0ad290 con 0x7f4f5c0bdbf0 2021-02-10T11:03:48.792 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:48.793 7f4f53fff700 1 -- 172.21.2.102:0/1764044986 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v20) v1 ==== 104+0+10 (secure 0 0 0) 0x7f4f440093a0 con 0x7f4f5c0bdbf0 2021-02-10T11:03:48.793 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:03:48.794 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:48.795 7f4f6168e700 1 -- 172.21.2.102:0/1764044986 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4f400186d0 msgr2=0x7f4f4001abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:48.794 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:48.795 7f4f6168e700 1 --2- 172.21.2.102:0/1764044986 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4f400186d0 0x7f4f4001abd0 crc :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:48.795 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:48.795 7f4f6168e700 1 -- 172.21.2.102:0/1764044986 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4f5c0bdbf0 msgr2=0x7f4f5c1a7450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:48.795 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:48.795 7f4f6168e700 1 --2- 172.21.2.102:0/1764044986 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4f5c0bdbf0 0x7f4f5c1a7450 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f4f440096a0 tx=0x7f4f4400b9b0).stop 2021-02-10T11:03:48.795 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:48.795 7f4f6168e700 1 -- 172.21.2.102:0/1764044986 shutdown_connections 2021-02-10T11:03:48.795 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:48.795 7f4f6168e700 1 --2- 172.21.2.102:0/1764044986 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4f5c0abcc0 0x7f4f5c0ac0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:03:48.795 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:48.795 7f4f6168e700 1 --2- 172.21.2.102:0/1764044986 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4f400186d0 0x7f4f4001abd0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:48.795 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:48.795 7f4f6168e700 1 --2- 172.21.2.102:0/1764044986 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4f5c0bdff0 0x7f4f5c1a7a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:03:48.795 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:48.795 7f4f6168e700 1 --2- 172.21.2.102:0/1764044986 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4f5c0bdbf0 0x7f4f5c1a7450 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:48.795 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:48.795 7f4f6168e700 1 -- 172.21.2.102:0/1764044986 >> 172.21.2.102:0/1764044986 conn(0x7f4f5c095a70 msgr2=0x7f4f5c0bf040 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:03:48.796 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:48.796 7f4f6168e700 1 -- 172.21.2.102:0/1764044986 shutdown_connections 2021-02-10T11:03:48.796 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:48.796 7f4f6168e700 1 -- 172.21.2.102:0/1764044986 wait complete. 2021-02-10T11:03:48.803 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-02-10T11:03:49.232 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:49.216 7fec1a6fa700 1 Processor -- start 2021-02-10T11:03:49.232 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:49.216 7fec1a6fa700 1 -- start start 2021-02-10T11:03:49.232 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:49.216 7fec1a6fa700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fec141b0470 0x7fec141b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:49.232 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:49.216 7fec1a6fa700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec140f0b50 con 0x7fec140ef610 2021-02-10T11:03:49.232 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:49.216 7fec1a6fa700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec140f24e0 con 0x7fec140f20e0 2021-02-10T11:03:49.232 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:49.216 7fec1a6fa700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec140e5ea0 con 0x7fec141b0470 2021-02-10T11:03:49.232 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:49.216 7fec137fe700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fec140ef610 0x7fec140efa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:41482/0 (socket says 172.21.2.102:41482) 2021-02-10T11:03:49.232 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:49.216 7fec137fe700 1 -- 172.21.2.102:0/2499614822 learned_addr learned my addr 172.21.2.102:0/2499614822 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:03:49.232 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:49.217 7fec12ffd700 1 -- 172.21.2.102:0/2499614822 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1929115222 0 0) 0x7fec080009b0 con 0x7fec140ef610 2021-02-10T11:03:49.232 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:49.217 7fec12ffd700 1 -- 172.21.2.102:0/2499614822 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7febfc002fe0 con 0x7fec140ef610 2021-02-10T11:03:49.233 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:49.217 7fec13fff700 1 --2- 172.21.2.102:0/2499614822 >> v2:172.21.2.113:3300/0 conn(0x7fec141b0470 0x7fec141b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:49.233 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:49.217 7fec12ffd700 1 -- 172.21.2.102:0/2499614822 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3540707368 0 0) 0x7fec08001010 con 0x7fec140ef610 2021-02-10T11:03:49.233 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:49.217 7fec12ffd700 1 -- 172.21.2.102:0/2499614822 >> v2:172.21.2.113:3300/0 conn(0x7fec141b0470 msgr2=0x7fec141b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:03:49.233 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:49.217 7fec12ffd700 1 --2- 172.21.2.102:0/2499614822 >> v2:172.21.2.113:3300/0 conn(0x7fec141b0470 0x7fec141b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:49.233 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:49.217 7fec12ffd700 1 -- 172.21.2.102:0/2499614822 >> v1:172.21.2.113:6789/0 conn(0x7fec140f20e0 legacy=0x7fec140f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:03:49.233 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:49.217 7fec12ffd700 1 -- 172.21.2.102:0/2499614822 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec140e52e0 con 0x7fec140ef610 2021-02-10T11:03:49.233 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:49.217 7fec12ffd700 1 -- 172.21.2.102:0/2499614822 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7fec08001590 con 0x7fec140ef610 2021-02-10T11:03:49.233 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:49.217 7fec12ffd700 1 -- 172.21.2.102:0/2499614822 >> v1:172.21.2.102:6789/0 conn(0x7fec140ef610 legacy=0x7fec140efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:49.233 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:49.217 7fec12ffd700 1 --2- 172.21.2.102:0/2499614822 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7febfc003a60 0x7febfc005ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:49.233 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:49.217 7fec12ffd700 1 --2- 172.21.2.102:0/2499614822 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7febfc006790 0x7febfc008c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:49.233 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:49.217 7fec12ffd700 1 --2- 172.21.2.102:0/2499614822 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7febfc009590 0x7febfc00bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:49.233 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:49.217 7fec12ffd700 1 -- 172.21.2.102:0/2499614822 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febfc00c300 con 0x7febfc003a60 2021-02-10T11:03:49.233 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:49.217 7fec12ffd700 1 -- 172.21.2.102:0/2499614822 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febfc00c520 con 0x7febfc009590 2021-02-10T11:03:49.233 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:49.217 7fec12ffd700 1 -- 172.21.2.102:0/2499614822 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febfc00c740 con 0x7febfc006790 2021-02-10T11:03:49.234 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:49.217 7fec13fff700 1 --2- 172.21.2.102:0/2499614822 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7febfc003a60 0x7febfc005ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:49.234 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:49.218 7fec13fff700 1 -- 172.21.2.102:0/2499614822 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7febfc006790 msgr2=0x7febfc008c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:03:49.234 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:49.218 7fec13fff700 1 --2- 172.21.2.102:0/2499614822 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7febfc006790 0x7febfc008c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:03:49.234 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:49.218 7fec13fff700 1 -- 172.21.2.102:0/2499614822 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7febfc009590 msgr2=0x7febfc00bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:03:49.234 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:49.218 7fec13fff700 1 --2- 172.21.2.102:0/2499614822 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7febfc009590 0x7febfc00bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:03:49.234 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:49.218 7fec13fff700 1 -- 172.21.2.102:0/2499614822 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febfc00c9e0 con 0x7febfc003a60 2021-02-10T11:03:49.234 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:49.218 7fec18d17700 1 -- 172.21.2.102:0/2499614822 reap_dead start 2021-02-10T11:03:49.234 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:49.218 7fec13fff700 1 --2- 172.21.2.102:0/2499614822 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7febfc003a60 0x7febfc005ef0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7fec04006360 tx=0x7fec0404dc30).ready entity=mon.0 client_cookie=21b48c97a222d02a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:03:49.234 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:49.218 7fec12ffd700 1 -- 172.21.2.102:0/2499614822 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fec04006610 con 0x7febfc003a60 2021-02-10T11:03:49.234 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:49.218 7fec12ffd700 1 -- 172.21.2.102:0/2499614822 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec04000c90 con 0x7febfc003a60 2021-02-10T11:03:49.234 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:49.218 7fec12ffd700 1 -- 172.21.2.102:0/2499614822 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fec040022b0 con 0x7febfc003a60 2021-02-10T11:03:49.234 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:49.218 7fec1a6fa700 1 -- 172.21.2.102:0/2499614822 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7febfc003a60 msgr2=0x7febfc005ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:49.234 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:49.218 7fec1a6fa700 1 --2- 172.21.2.102:0/2499614822 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7febfc003a60 0x7febfc005ef0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7fec04006360 tx=0x7fec0404dc30).stop 2021-02-10T11:03:49.235 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:49.218 7fec1a6fa700 1 -- 172.21.2.102:0/2499614822 shutdown_connections 2021-02-10T11:03:49.235 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:49.218 7fec1a6fa700 1 --2- 172.21.2.102:0/2499614822 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7febfc003a60 0x7febfc005ef0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:49.235 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:49.218 7fec1a6fa700 1 -- 172.21.2.102:0/2499614822 >> 172.21.2.102:0/2499614822 conn(0x7fec140ce3e0 msgr2=0x7fec140e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:03:49.235 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:49.218 7fec1a6fa700 1 -- 172.21.2.102:0/2499614822 shutdown_connections 2021-02-10T11:03:49.235 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:49.218 7fec1a6fa700 1 -- 172.21.2.102:0/2499614822 wait complete. 2021-02-10T11:03:49.236 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:49.233 7fec1a6fa700 1 Processor -- start 2021-02-10T11:03:49.236 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:49.233 7fec1a6fa700 1 -- start start 2021-02-10T11:03:49.236 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:49.233 7fec1a6fa700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fec1409b930 0x7fec140b7c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:49.236 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:49.233 7fec1a6fa700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fec140b8410 0x7fec140b8810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:49.236 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:49.233 7fec1a6fa700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fec140c1f80 0x7fec140c2380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:49.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:49.233 7fec1a6fa700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec140c1a00 con 0x7fec1409b930 2021-02-10T11:03:49.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:49.233 7fec1a6fa700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec140b8e30 con 0x7fec140c1f80 2021-02-10T11:03:49.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:49.233 7fec1a6fa700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec140b9050 con 0x7fec140b8410 2021-02-10T11:03:49.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:49.234 7fec13fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fec1409b930 0x7fec140b7c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:49.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:49.234 7fec13fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fec1409b930 0x7fec140b7c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:49268/0 (socket says 172.21.2.102:49268) 2021-02-10T11:03:49.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:49.234 7fec13fff700 1 -- 172.21.2.102:0/1351050880 learned_addr learned my addr 172.21.2.102:0/1351050880 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:03:49.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:49.234 7fec13fff700 1 -- 172.21.2.102:0/1351050880 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fec140b8410 msgr2=0x7fec140b8810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:49.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:49.234 7fec13fff700 1 --2- 172.21.2.102:0/1351050880 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fec140b8410 0x7fec140b8810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:03:49.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:49.234 7fec13fff700 1 -- 172.21.2.102:0/1351050880 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fec140c1f80 msgr2=0x7fec140c2380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:49.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:49.234 7fec13fff700 1 --2- 172.21.2.102:0/1351050880 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fec140c1f80 0x7fec140c2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:03:49.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:49.234 7fec13fff700 1 -- 172.21.2.102:0/1351050880 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec140c29a0 con 0x7fec1409b930 2021-02-10T11:03:49.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:49.234 7fec13fff700 1 --2- 172.21.2.102:0/1351050880 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fec1409b930 0x7fec140b7c20 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7fec04006920 tx=0x7fec0404f960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:03:49.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:49.234 7fec10ff9700 1 -- 172.21.2.102:0/1351050880 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fec040009b0 con 0x7fec1409b930 2021-02-10T11:03:49.238 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:49.234 7fec10ff9700 1 -- 172.21.2.102:0/1351050880 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec0404ab50 con 0x7fec1409b930 2021-02-10T11:03:49.238 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:49.234 7fec1a6fa700 1 -- 172.21.2.102:0/1351050880 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec140c29a0 con 0x7fec1409b930 2021-02-10T11:03:49.238 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:49.234 7fec10ff9700 1 -- 172.21.2.102:0/1351050880 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fec040052e0 con 0x7fec1409b930 2021-02-10T11:03:49.238 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:49.234 7fec1a6fa700 1 -- 172.21.2.102:0/1351050880 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec140c2f30 con 0x7fec1409b930 2021-02-10T11:03:49.238 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:49.235 7fec10ff9700 1 -- 172.21.2.102:0/1351050880 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fec0404e000 con 0x7fec1409b930 2021-02-10T11:03:49.238 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:49.235 7fec10ff9700 1 --2- 172.21.2.102:0/1351050880 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7febf40186c0 0x7febf401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:49.238 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:49.235 7fec10ff9700 1 -- 172.21.2.102:0/1351050880 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fec040767d0 con 0x7fec1409b930 2021-02-10T11:03:49.238 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:49.235 7fec137fe700 1 --2- 172.21.2.102:0/1351050880 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7febf40186c0 0x7febf401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:49.238 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:49.235 7fec137fe700 1 --2- 172.21.2.102:0/1351050880 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7febf40186c0 0x7febf401abc0 crc :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:03:49.265 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:49.266 7fec1a6fa700 1 -- 172.21.2.102:0/1351050880 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec14105b30 con 0x7fec1409b930 2021-02-10T11:03:49.268 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:49.269 7fec10ff9700 1 -- 172.21.2.102:0/1351050880 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fec040009b0 con 0x7fec1409b930 2021-02-10T11:03:49.395 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:49.396 7fec1a6fa700 1 -- 172.21.2.102:0/1351050880 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7fec140f0b50 con 0x7fec1409b930 2021-02-10T11:03:49.396 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:49.397 7fec10ff9700 1 -- 172.21.2.102:0/1351050880 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v20) v1 ==== 100+0+10 (secure 0 0 0) 0x7fec0404fd50 con 0x7fec1409b930 2021-02-10T11:03:49.396 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:03:49.397 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:49.398 7fec1a6fa700 1 -- 172.21.2.102:0/1351050880 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7febf40186c0 msgr2=0x7febf401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:49.398 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:49.398 7fec1a6fa700 1 --2- 172.21.2.102:0/1351050880 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7febf40186c0 0x7febf401abc0 crc :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:49.398 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:49.398 7fec1a6fa700 1 -- 172.21.2.102:0/1351050880 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fec1409b930 msgr2=0x7fec140b7c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:49.398 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:49.398 7fec1a6fa700 1 --2- 172.21.2.102:0/1351050880 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fec1409b930 0x7fec140b7c20 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7fec04006920 tx=0x7fec0404f960).stop 2021-02-10T11:03:49.398 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:49.399 7fec1a6fa700 1 -- 172.21.2.102:0/1351050880 shutdown_connections 2021-02-10T11:03:49.398 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:49.399 7fec1a6fa700 1 --2- 172.21.2.102:0/1351050880 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7febf40186c0 0x7febf401abc0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:49.398 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:49.399 7fec1a6fa700 1 --2- 172.21.2.102:0/1351050880 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fec140c1f80 0x7fec140c2380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:03:49.398 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:49.399 7fec1a6fa700 1 --2- 172.21.2.102:0/1351050880 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fec140b8410 0x7fec140b8810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:03:49.398 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:49.399 7fec1a6fa700 1 --2- 172.21.2.102:0/1351050880 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fec1409b930 0x7fec140b7c20 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:49.398 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:49.399 7fec1a6fa700 1 -- 172.21.2.102:0/1351050880 >> 172.21.2.102:0/1351050880 conn(0x7fec14094e10 msgr2=0x7fec140a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:03:49.399 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:49.399 7fec1a6fa700 1 -- 172.21.2.102:0/1351050880 shutdown_connections 2021-02-10T11:03:49.399 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:49.399 7fec1a6fa700 1 -- 172.21.2.102:0/1351050880 wait complete. 2021-02-10T11:03:49.406 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2021-02-10T11:03:49.406 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2021-02-10T11:03:49.406 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph mgr dump --format=json 2021-02-10T11:03:49.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:49.816 7ff35c5df700 1 Processor -- start 2021-02-10T11:03:49.832 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:49.816 7ff35c5df700 1 -- start start 2021-02-10T11:03:49.832 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:49.817 7ff35c5df700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7ff35407e6a0 0x7ff3540d1790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:49.832 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:49.817 7ff35c5df700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff35407ed80 con 0x7ff3540e5470 2021-02-10T11:03:49.832 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:49.817 7ff35c5df700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff35407eaa0 con 0x7ff3540e5070 2021-02-10T11:03:49.832 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:49.817 7ff35c5df700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3541b4080 con 0x7ff35407e6a0 2021-02-10T11:03:49.832 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:49.817 7ff35abfc700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7ff35407e6a0 0x7ff3540d1790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:49.832 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:49.817 7ff35abfc700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7ff35407e6a0 0x7ff3540d1790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:49278/0 (socket says 172.21.2.102:49278) 2021-02-10T11:03:49.832 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:49.817 7ff35abfc700 1 -- 172.21.2.102:0/2633693373 learned_addr learned my addr 172.21.2.102:0/2633693373 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:03:49.832 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:49.817 7ff35abfc700 1 -- 172.21.2.102:0/2633693373 >> v1:172.21.2.113:6789/0 conn(0x7ff3540e5070 legacy=0x7ff3541b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:03:49.832 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:49.817 7ff35abfc700 1 -- 172.21.2.102:0/2633693373 >> v1:172.21.2.102:6789/0 conn(0x7ff3540e5470 legacy=0x7ff35407f300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:49.832 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:49.817 7ff35abfc700 1 -- 172.21.2.102:0/2633693373 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3540d1db0 con 0x7ff35407e6a0 2021-02-10T11:03:49.832 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:49.817 7ff35abfc700 1 --2- 172.21.2.102:0/2633693373 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff35407e6a0 0x7ff3540d1790 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7ff34c006f00 tx=0x7ff34c009f10).ready entity=mon.0 client_cookie=cf6a38c34fc3d842 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:03:49.833 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:49.818 7ff3593f9700 1 -- 172.21.2.102:0/2633693373 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff34c00b3b0 con 0x7ff35407e6a0 2021-02-10T11:03:49.833 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:49.818 7ff3593f9700 1 -- 172.21.2.102:0/2633693373 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff34c00b730 con 0x7ff35407e6a0 2021-02-10T11:03:49.833 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:49.818 7ff3593f9700 1 -- 172.21.2.102:0/2633693373 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff34c00bc10 con 0x7ff35407e6a0 2021-02-10T11:03:49.833 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:49.818 7ff35c5df700 1 -- 172.21.2.102:0/2633693373 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff35407e6a0 msgr2=0x7ff3540d1790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:49.833 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:49.818 7ff35c5df700 1 --2- 172.21.2.102:0/2633693373 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff35407e6a0 0x7ff3540d1790 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7ff34c006f00 tx=0x7ff34c009f10).stop 2021-02-10T11:03:49.833 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:49.818 7ff35c5df700 1 -- 172.21.2.102:0/2633693373 shutdown_connections 2021-02-10T11:03:49.833 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:49.818 7ff35c5df700 1 --2- 172.21.2.102:0/2633693373 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff35407e6a0 0x7ff3540d1790 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:49.833 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:49.818 7ff35c5df700 1 -- 172.21.2.102:0/2633693373 >> 172.21.2.102:0/2633693373 conn(0x7ff3540ce690 msgr2=0x7ff3540e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:03:49.833 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:49.818 7ff35c5df700 1 -- 172.21.2.102:0/2633693373 shutdown_connections 2021-02-10T11:03:49.833 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:49.818 7ff35c5df700 1 -- 172.21.2.102:0/2633693373 wait complete. 2021-02-10T11:03:49.835 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:49.833 7ff35c5df700 1 Processor -- start 2021-02-10T11:03:49.835 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:49.833 7ff35c5df700 1 -- start start 2021-02-10T11:03:49.835 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:49.833 7ff35c5df700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff3540af850 0x7ff35410a450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:49.835 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:49.833 7ff35c5df700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff3540bfa70 0x7ff354096380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:49.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:49.833 7ff35c5df700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff3540bfe70 0x7ff35410b130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:49.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:49.833 7ff35c5df700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3540c0490 con 0x7ff3540bfa70 2021-02-10T11:03:49.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:49.833 7ff35c5df700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff35410abb0 con 0x7ff3540af850 2021-02-10T11:03:49.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:49.833 7ff35c5df700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3540afc50 con 0x7ff3540bfe70 2021-02-10T11:03:49.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:49.833 7ff359bfa700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff3540bfa70 0x7ff354096380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:49.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:49.833 7ff359bfa700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff3540bfa70 0x7ff354096380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:49282/0 (socket says 172.21.2.102:49282) 2021-02-10T11:03:49.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:49.833 7ff359bfa700 1 -- 172.21.2.102:0/2865836249 learned_addr learned my addr 172.21.2.102:0/2865836249 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:03:49.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:49.834 7ff35abfc700 1 --2- 172.21.2.102:0/2865836249 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff3540bfe70 0x7ff35410b130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:49.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:49.834 7ff35a3fb700 1 --2- 172.21.2.102:0/2865836249 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff3540af850 0x7ff35410a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:49.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:49.834 7ff359bfa700 1 -- 172.21.2.102:0/2865836249 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff3540bfe70 msgr2=0x7ff35410b130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:49.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:49.834 7ff359bfa700 1 --2- 172.21.2.102:0/2865836249 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff3540bfe70 0x7ff35410b130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:49.837 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:49.834 7ff359bfa700 1 -- 172.21.2.102:0/2865836249 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff3540af850 msgr2=0x7ff35410a450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:49.837 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:49.834 7ff359bfa700 1 --2- 172.21.2.102:0/2865836249 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff3540af850 0x7ff35410a450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:49.837 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:49.834 7ff359bfa700 1 -- 172.21.2.102:0/2865836249 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3540969a0 con 0x7ff3540bfa70 2021-02-10T11:03:49.837 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:49.834 7ff35abfc700 1 --2- 172.21.2.102:0/2865836249 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff3540bfe70 0x7ff35410b130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:03:49.837 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:49.834 7ff35a3fb700 1 --2- 172.21.2.102:0/2865836249 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff3540af850 0x7ff35410a450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:03:49.837 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:49.834 7ff359bfa700 1 --2- 172.21.2.102:0/2865836249 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff3540bfa70 0x7ff354096380 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7ff348006ed0 tx=0x7ff34800b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:03:49.837 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:49.834 7ff346ffd700 1 -- 172.21.2.102:0/2865836249 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff348006020 con 0x7ff3540bfa70 2021-02-10T11:03:49.837 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:49.834 7ff346ffd700 1 -- 172.21.2.102:0/2865836249 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3480063a0 con 0x7ff3540bfa70 2021-02-10T11:03:49.837 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:49.834 7ff35c5df700 1 -- 172.21.2.102:0/2865836249 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3540969a0 con 0x7ff3540bfa70 2021-02-10T11:03:49.837 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:49.834 7ff35c5df700 1 -- 172.21.2.102:0/2865836249 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3540969a0 con 0x7ff3540bfa70 2021-02-10T11:03:49.837 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:49.834 7ff346ffd700 1 -- 172.21.2.102:0/2865836249 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff348005730 con 0x7ff3540bfa70 2021-02-10T11:03:49.837 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:49.835 7ff346ffd700 1 -- 172.21.2.102:0/2865836249 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ff3480054b0 con 0x7ff3540bfa70 2021-02-10T11:03:49.837 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:49.835 7ff346ffd700 1 --2- 172.21.2.102:0/2865836249 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff33c018730 0x7ff33c01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:49.838 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:49.835 7ff346ffd700 1 -- 172.21.2.102:0/2865836249 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5573+0+0 (secure 0 0 0) 0x7ff3480327c0 con 0x7ff3540bfa70 2021-02-10T11:03:49.838 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:49.835 7ff35a3fb700 1 --2- 172.21.2.102:0/2865836249 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff33c018730 0x7ff33c01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:49.838 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:49.835 7ff35a3fb700 1 --2- 172.21.2.102:0/2865836249 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff33c018730 0x7ff33c01ac10 crc :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:03:49.865 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:49.866 7ff35c5df700 1 -- 172.21.2.102:0/2865836249 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3540891c0 con 0x7ff3540bfa70 2021-02-10T11:03:49.868 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:49.869 7ff346ffd700 1 -- 172.21.2.102:0/2865836249 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ff348006000 con 0x7ff3540bfa70 2021-02-10T11:03:50.033 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:50.034 7ff35c5df700 1 -- 172.21.2.102:0/2865836249 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7ff3540ce690 con 0x7ff3540bfa70 2021-02-10T11:03:50.034 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:50.035 7ff346ffd700 1 -- 172.21.2.102:0/2865836249 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v3) v1 ==== 74+0+70186 (secure 0 0 0) 0x7ff3480063a0 con 0x7ff3540bfa70 2021-02-10T11:03:50.037 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:03:50.038 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:50.038 7ff35c5df700 1 -- 172.21.2.102:0/2865836249 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff33c018730 msgr2=0x7ff33c01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:50.038 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:50.038 7ff35c5df700 1 --2- 172.21.2.102:0/2865836249 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff33c018730 0x7ff33c01ac10 crc :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:50.038 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:50.038 7ff35c5df700 1 -- 172.21.2.102:0/2865836249 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff3540bfa70 msgr2=0x7ff354096380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:50.038 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:50.038 7ff35c5df700 1 --2- 172.21.2.102:0/2865836249 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff3540bfa70 0x7ff354096380 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7ff348006ed0 tx=0x7ff34800b960).stop 2021-02-10T11:03:50.038 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:50.039 7ff35c5df700 1 -- 172.21.2.102:0/2865836249 shutdown_connections 2021-02-10T11:03:50.038 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:50.039 7ff35c5df700 1 --2- 172.21.2.102:0/2865836249 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff3540bfe70 0x7ff35410b130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:50.038 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:50.039 7ff35c5df700 1 --2- 172.21.2.102:0/2865836249 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff3540bfa70 0x7ff354096380 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:50.038 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:50.039 7ff35c5df700 1 --2- 172.21.2.102:0/2865836249 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff33c018730 0x7ff33c01ac10 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:50.039 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:50.039 7ff35c5df700 1 --2- 172.21.2.102:0/2865836249 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff3540af850 0x7ff35410a450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:50.039 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:50.039 7ff35c5df700 1 -- 172.21.2.102:0/2865836249 >> 172.21.2.102:0/2865836249 conn(0x7ff3540b07f0 msgr2=0x7ff354107250 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:03:50.039 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:50.039 7ff35c5df700 1 -- 172.21.2.102:0/2865836249 shutdown_connections 2021-02-10T11:03:50.039 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:50.039 7ff35c5df700 1 -- 172.21.2.102:0/2865836249 wait complete. 2021-02-10T11:03:50.046 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":3,"active_gid":4101,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6800","nonce":35161},{"type":"v1","addr":"172.21.2.102:6801","nonce":35161}]},"active_addr":"172.21.2.102:6801/35161","active_change":"2021-02-10 11:03:07.666576","available":true,"standbys":[{"gid":4105,"name":"x","available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"ansible","can_run":true,"error_string":"","module_options":{"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_url":{"name":"server_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_server":{"name":"verify_server","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"none","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_iterations":{"name":"upmap_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimization iterations","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_cephfs":{"name":"FEATURE_TOGGLE_cephfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_iscsi":{"name":"FEATURE_TOGGLE_iscsi","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_mirroring":{"name":"FEATURE_TOGGLE_mirroring","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rbd":{"name":"FEATURE_TOGGLE_rbd","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rgw":{"name":"FEATURE_TOGGLE_rgw","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_HOST":{"name":"RGW_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_PORT":{"name":"RGW_API_PORT","type":"int","level":"advanced","flags":0,"default_value":"80","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SCHEME":{"name":"RGW_API_SCHEME","type":"str","level":"advanced","flags":0,"default_value":"http","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_USER_ID":{"name":"RGW_API_USER_ID","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"deepsea","can_run":true,"error_string":"","module_options":{"salt_api_eauth":{"name":"salt_api_eauth","type":"str","level":"advanced","flags":0,"default_value":"sharedsecret","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"salt_api_password":{"name":"salt_api_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"salt_api_url":{"name":"salt_api_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"salt_api_username":{"name":"salt_api_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_cloud","can_run":true,"error_string":"","module_options":{"diskprediction_cert_context":{"name":"diskprediction_cert_context","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_default_authority":{"name":"diskprediction_default_authority","type":"str","level":"advanced","flags":0,"default_value":"localhost","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_password":{"name":"diskprediction_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_port":{"name":"diskprediction_port","type":"str","level":"advanced","flags":0,"default_value":"31400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_retrieve_prediction_interval":{"name":"diskprediction_retrieve_prediction_interval","type":"str","level":"advanced","flags":0,"default_value":"43200","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_server":{"name":"diskprediction_server","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_ssl_target_name_override":{"name":"diskprediction_ssl_target_name_override","type":"str","level":"advanced","flags":0,"default_value":"localhost","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_upload_metrics_interval":{"name":"diskprediction_upload_metrics_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_upload_smart_interval":{"name":"diskprediction_upload_smart_interval","type":"str","level":"advanced","flags":0,"default_value":"43200","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_user":{"name":"diskprediction_user","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{}},{"name":"iostat","can_run":true,"error_string":"","module_options":{}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator_cli","can_run":true,"error_string":"","module_options":{"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"persist_interval":{"name":"persist_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how frequently to persist completed events","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{}},{"name":"restful","can_run":true,"error_string":"","module_options":{"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"ssh","can_run":true,"error_string":"","module_options":{"inventory_cache_timeout_min":{"name":"inventory_cache_timeout_min","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"ansible","can_run":true,"error_string":"","module_options":{"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_url":{"name":"server_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_server":{"name":"verify_server","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"none","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_iterations":{"name":"upmap_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimization iterations","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_cephfs":{"name":"FEATURE_TOGGLE_cephfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_iscsi":{"name":"FEATURE_TOGGLE_iscsi","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_mirroring":{"name":"FEATURE_TOGGLE_mirroring","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rbd":{"name":"FEATURE_TOGGLE_rbd","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rgw":{"name":"FEATURE_TOGGLE_rgw","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_HOST":{"name":"RGW_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_PORT":{"name":"RGW_API_PORT","type":"int","level":"advanced","flags":0,"default_value":"80","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SCHEME":{"name":"RGW_API_SCHEME","type":"str","level":"advanced","flags":0,"default_value":"http","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_USER_ID":{"name":"RGW_API_USER_ID","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"deepsea","can_run":true,"error_string":"","module_options":{"salt_api_eauth":{"name":"salt_api_eauth","type":"str","level":"advanced","flags":0,"default_value":"sharedsecret","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"salt_api_password":{"name":"salt_api_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"salt_api_url":{"name":"salt_api_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"salt_api_username":{"name":"salt_api_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_cloud","can_run":true,"error_string":"","module_options":{"diskprediction_cert_context":{"name":"diskprediction_cert_context","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_default_authority":{"name":"diskprediction_default_authority","type":"str","level":"advanced","flags":0,"default_value":"localhost","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_password":{"name":"diskprediction_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_port":{"name":"diskprediction_port","type":"str","level":"advanced","flags":0,"default_value":"31400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_retrieve_prediction_interval":{"name":"diskprediction_retrieve_prediction_interval","type":"str","level":"advanced","flags":0,"default_value":"43200","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_server":{"name":"diskprediction_server","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_ssl_target_name_override":{"name":"diskprediction_ssl_target_name_override","type":"str","level":"advanced","flags":0,"default_value":"localhost","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_upload_metrics_interval":{"name":"diskprediction_upload_metrics_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_upload_smart_interval":{"name":"diskprediction_upload_smart_interval","type":"str","level":"advanced","flags":0,"default_value":"43200","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_user":{"name":"diskprediction_user","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{}},{"name":"iostat","can_run":true,"error_string":"","module_options":{}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator_cli","can_run":true,"error_string":"","module_options":{"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"persist_interval":{"name":"persist_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how frequently to persist completed events","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{}},{"name":"restful","can_run":true,"error_string":"","module_options":{"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"ssh","can_run":true,"error_string":"","module_options":{"inventory_cache_timeout_min":{"name":"inventory_cache_timeout_min","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"nautilus":["balancer","crash","devicehealth","orchestrator_cli","progress","rbd_support","status","volumes"]}} 2021-02-10T11:03:50.048 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2021-02-10T11:03:50.854 DEBUG:teuthology.orchestra.run.gibba002:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2021-02-10T11:03:51.271 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:51.256 7f99ab9c6700 1 Processor -- start 2021-02-10T11:03:51.271 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:51.256 7f99ab9c6700 1 -- start start 2021-02-10T11:03:51.271 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:51.256 7f99ab9c6700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f99a40e5070 0x7f99a41b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:51.271 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:51.256 7f99ab9c6700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f99a41b3e40 0x7f99a4080310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:51.271 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:51.256 7f99ab9c6700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99a407ed80 con 0x7f99a40e5470 2021-02-10T11:03:51.271 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:51.256 7f99ab9c6700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99a40d1790 con 0x7f99a41b3e40 2021-02-10T11:03:51.272 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:51.256 7f99ab9c6700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99a40d1980 con 0x7f99a40e5070 2021-02-10T11:03:51.272 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:51.256 7f99a9fe3700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f99a41b3e40 0x7f99a4080310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:51.272 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:51.256 7f99a9fe3700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f99a41b3e40 0x7f99a4080310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:49288/0 (socket says 172.21.2.102:49288) 2021-02-10T11:03:51.272 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:51.257 7f99a9fe3700 1 -- 172.21.2.102:0/365668415 learned_addr learned my addr 172.21.2.102:0/365668415 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:03:51.272 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:51.257 7f99a9fe3700 1 -- 172.21.2.102:0/365668415 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f99a40e5070 msgr2=0x7f99a41b3820 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:03:51.272 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:51.257 7f99a9fe3700 1 --2- 172.21.2.102:0/365668415 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f99a40e5070 0x7f99a41b3820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:03:51.272 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:51.257 7f99a9fe3700 1 -- 172.21.2.102:0/365668415 >> v1:172.21.2.113:6789/0 conn(0x7f99a40e5470 legacy=0x7f99a407d690 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:03:51.272 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:51.257 7f99a9fe3700 1 -- 172.21.2.102:0/365668415 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99a40d1ba0 con 0x7f99a41b3e40 2021-02-10T11:03:51.272 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:51.257 7f99a9fe3700 1 --2- 172.21.2.102:0/365668415 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f99a41b3e40 0x7f99a4080310 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7f999c006f00 tx=0x7f999c009f10).ready entity=mon.0 client_cookie=97d9fdf971fc7ded server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:03:51.272 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:51.257 7f9997fff700 1 -- 172.21.2.102:0/365668415 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f999c00b3b0 con 0x7f99a41b3e40 2021-02-10T11:03:51.272 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:51.257 7f9997fff700 1 -- 172.21.2.102:0/365668415 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f999c00b730 con 0x7f99a41b3e40 2021-02-10T11:03:51.272 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:51.257 7f9997fff700 1 -- 172.21.2.102:0/365668415 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f999c00bc10 con 0x7f99a41b3e40 2021-02-10T11:03:51.272 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:51.257 7f99ab9c6700 1 -- 172.21.2.102:0/365668415 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f99a41b3e40 msgr2=0x7f99a4080310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:51.273 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:51.257 7f99ab9c6700 1 --2- 172.21.2.102:0/365668415 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f99a41b3e40 0x7f99a4080310 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7f999c006f00 tx=0x7f999c009f10).stop 2021-02-10T11:03:51.273 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:51.258 7f99ab9c6700 1 -- 172.21.2.102:0/365668415 shutdown_connections 2021-02-10T11:03:51.273 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:51.258 7f99ab9c6700 1 --2- 172.21.2.102:0/365668415 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f99a41b3e40 0x7f99a4080310 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:51.273 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:51.258 7f99ab9c6700 1 --2- 172.21.2.102:0/365668415 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f99a40e5070 0x7f99a41b3820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:03:51.273 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:51.258 7f99ab9c6700 1 -- 172.21.2.102:0/365668415 >> 172.21.2.102:0/365668415 conn(0x7f99a40ce690 msgr2=0x7f99a40e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:03:51.273 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:51.258 7f99ab9c6700 1 -- 172.21.2.102:0/365668415 shutdown_connections 2021-02-10T11:03:51.273 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:51.258 7f99ab9c6700 1 -- 172.21.2.102:0/365668415 wait complete. 2021-02-10T11:03:51.274 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:51.273 7f99ab9c6700 1 Processor -- start 2021-02-10T11:03:51.275 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:51.273 7f99ab9c6700 1 -- start start 2021-02-10T11:03:51.275 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:51.273 7f99ab9c6700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f99a40b4920 0x7f99a409c640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:51.275 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:51.273 7f99ab9c6700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f99a409cc60 0x7f99a4086710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:51.275 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:51.273 7f99ab9c6700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f99a4086d30 0x7f99a4087130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:51.275 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:51.273 7f99ab9c6700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99a4087860 con 0x7f99a409cc60 2021-02-10T11:03:51.275 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:51.273 7f99ab9c6700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99a419dc70 con 0x7f99a40b4920 2021-02-10T11:03:51.275 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:51.273 7f99ab9c6700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99a419de90 con 0x7f99a4086d30 2021-02-10T11:03:51.275 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:51.273 7f99a8fe1700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f99a409cc60 0x7f99a4086710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:51.275 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:51.273 7f99a8fe1700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f99a409cc60 0x7f99a4086710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:49296/0 (socket says 172.21.2.102:49296) 2021-02-10T11:03:51.275 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:51.273 7f99a8fe1700 1 -- 172.21.2.102:0/1572332174 learned_addr learned my addr 172.21.2.102:0/1572332174 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:03:51.275 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:51.274 7f99a8fe1700 1 -- 172.21.2.102:0/1572332174 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f99a4086d30 msgr2=0x7f99a4087130 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:03:51.275 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:51.274 7f99a8fe1700 1 --2- 172.21.2.102:0/1572332174 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f99a4086d30 0x7f99a4087130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:03:51.276 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:51.274 7f99a8fe1700 1 -- 172.21.2.102:0/1572332174 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f99a40b4920 msgr2=0x7f99a409c640 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:03:51.276 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:51.274 7f99a8fe1700 1 --2- 172.21.2.102:0/1572332174 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f99a40b4920 0x7f99a409c640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:03:51.276 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:51.274 7f99a8fe1700 1 -- 172.21.2.102:0/1572332174 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99a419e0b0 con 0x7f99a409cc60 2021-02-10T11:03:51.276 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:51.274 7f99a8fe1700 1 --2- 172.21.2.102:0/1572332174 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f99a409cc60 0x7f99a4086710 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7f999800b5b0 tx=0x7f999800b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:03:51.276 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:51.274 7f9995ffb700 1 -- 172.21.2.102:0/1572332174 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f999800bd40 con 0x7f99a409cc60 2021-02-10T11:03:51.276 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:51.274 7f9995ffb700 1 -- 172.21.2.102:0/1572332174 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9998006240 con 0x7f99a409cc60 2021-02-10T11:03:51.276 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:51.274 7f9995ffb700 1 -- 172.21.2.102:0/1572332174 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9998005690 con 0x7f99a409cc60 2021-02-10T11:03:51.276 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:51.274 7f99ab9c6700 1 -- 172.21.2.102:0/1572332174 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99a419e0b0 con 0x7f99a409cc60 2021-02-10T11:03:51.276 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:51.274 7f99ab9c6700 1 -- 172.21.2.102:0/1572332174 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99a419e0b0 con 0x7f99a409cc60 2021-02-10T11:03:51.276 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:51.275 7f9995ffb700 1 -- 172.21.2.102:0/1572332174 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f99980054a0 con 0x7f99a409cc60 2021-02-10T11:03:51.276 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:51.275 7f9995ffb700 1 --2- 172.21.2.102:0/1572332174 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f9984018730 0x7f998401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:51.276 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:51.275 7f9995ffb700 1 -- 172.21.2.102:0/1572332174 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f9998032650 con 0x7f99a409cc60 2021-02-10T11:03:51.276 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:51.275 7f99a97e2700 1 --2- 172.21.2.102:0/1572332174 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f9984018730 0x7f998401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:51.276 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:51.275 7f99a97e2700 1 --2- 172.21.2.102:0/1572332174 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f9984018730 0x7f998401ac10 crc :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:03:51.305 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:51.306 7f99ab9c6700 1 -- 172.21.2.102:0/1572332174 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99a4083e20 con 0x7f99a409cc60 2021-02-10T11:03:51.307 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:51.309 7f9995ffb700 1 -- 172.21.2.102:0/1572332174 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f9998000c60 con 0x7f99a409cc60 2021-02-10T11:03:51.431 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:51.432 7f99ab9c6700 1 -- 172.21.2.102:0/1572332174 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f99a419dc30 con 0x7f99a409cc60 2021-02-10T11:03:51.432 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:51.433 7f9995ffb700 1 -- 172.21.2.102:0/1572332174 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+13993 (secure 0 0 0) 0x7f99980054a0 con 0x7f99a409cc60 2021-02-10T11:03:51.432 INFO:teuthology.misc.health.gibba002.stdout: 2021-02-10T11:03:51.432 INFO:teuthology.misc.health.gibba002.stdout:{"epoch":20,"fsid":"68db30ea-dbce-4aef-a6df-f01c6fea0513","created":"2021-02-10 11:03:06.331557","modified":"2021-02-10 11:03:44.386945","last_up_change":"2021-02-10 11:03:32.690562","last_in_change":"2021-02-10 11:03:32.690562","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:03:37.322519","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:03:41.161563","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2021-02-10 11:03:42.379651","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"be691b4d-a039-408a-af80-9d53f02b6f1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6818","nonce":36088},{"type":"v1","addr":"172.21.2.102:6820","nonce":36088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6822","nonce":36088},{"type":"v1","addr":"172.21.2.102:6824","nonce":36088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6830","nonce":36088},{"type":"v1","addr":"172.21.2.102:6832","nonce":36088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6826","nonce":36088},{"type":"v1","addr":"172.21.2.102:6828","nonce":36088}]},"public_addr":"172.21.2.102:6820/36088","cluster_addr":"172.21.2.102:6824/36088","heartbeat_back_addr":"172.21.2.102:6832/36088","heartbeat_front_addr":"172.21.2.102:6828/36088","state":["exists","up"]},{"osd":1,"uuid":"64e3ccc8-9c69-4d7c-865e-af8e83822635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6819","nonce":36087},{"type":"v1","addr":"172.21.2.102:6821","nonce":36087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6823","nonce":36087},{"type":"v1","addr":"172.21.2.102:6825","nonce":36087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6831","nonce":36087},{"type":"v1","addr":"172.21.2.102:6833","nonce":36087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6827","nonce":36087},{"type":"v1","addr":"172.21.2.102:6829","nonce":36087}]},"public_addr":"172.21.2.102:6821/36087","cluster_addr":"172.21.2.102:6825/36087","heartbeat_back_addr":"172.21.2.102:6833/36087","heartbeat_front_addr":"172.21.2.102:6829/36087","state":["exists","up"]},{"osd":2,"uuid":"153700af-747f-481f-bdf4-d6072d3a842f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6810","nonce":36089},{"type":"v1","addr":"172.21.2.102:6811","nonce":36089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6812","nonce":36089},{"type":"v1","addr":"172.21.2.102:6813","nonce":36089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6816","nonce":36089},{"type":"v1","addr":"172.21.2.102:6817","nonce":36089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6814","nonce":36089},{"type":"v1","addr":"172.21.2.102:6815","nonce":36089}]},"public_addr":"172.21.2.102:6811/36089","cluster_addr":"172.21.2.102:6813/36089","heartbeat_back_addr":"172.21.2.102:6817/36089","heartbeat_front_addr":"172.21.2.102:6815/36089","state":["exists","up"]},{"osd":3,"uuid":"cbc678f4-83a9-45e6-8830-2f7c232b1ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6802","nonce":36090},{"type":"v1","addr":"172.21.2.102:6803","nonce":36090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6804","nonce":36090},{"type":"v1","addr":"172.21.2.102:6805","nonce":36090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6808","nonce":36090},{"type":"v1","addr":"172.21.2.102:6809","nonce":36090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6806","nonce":36090},{"type":"v1","addr":"172.21.2.102:6807","nonce":36090}]},"public_addr":"172.21.2.102:6803/36090","cluster_addr":"172.21.2.102:6805/36090","heartbeat_back_addr":"172.21.2.102:6809/36090","heartbeat_front_addr":"172.21.2.102:6807/36090","state":["exists","up"]},{"osd":4,"uuid":"c01e4008-b258-4f10-afd2-9e7f37214351","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6808","nonce":36368},{"type":"v1","addr":"172.21.2.113:6811","nonce":36368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6814","nonce":36368},{"type":"v1","addr":"172.21.2.113:6818","nonce":36368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6826","nonce":36368},{"type":"v1","addr":"172.21.2.113:6828","nonce":36368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6820","nonce":36368},{"type":"v1","addr":"172.21.2.113:6824","nonce":36368}]},"public_addr":"172.21.2.113:6811/36368","cluster_addr":"172.21.2.113:6818/36368","heartbeat_back_addr":"172.21.2.113:6828/36368","heartbeat_front_addr":"172.21.2.113:6824/36368","state":["exists","up"]},{"osd":5,"uuid":"551f6007-daf8-4a98-8ee0-c2b16bcccb5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6802","nonce":36366},{"type":"v1","addr":"172.21.2.113:6805","nonce":36366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6807","nonce":36366},{"type":"v1","addr":"172.21.2.113:6810","nonce":36366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6819","nonce":36366},{"type":"v1","addr":"172.21.2.113:6822","nonce":36366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6812","nonce":36366},{"type":"v1","addr":"172.21.2.113:6815","nonce":36366}]},"public_addr":"172.21.2.113:6805/36366","cluster_addr":"172.21.2.113:6810/36366","heartbeat_back_addr":"172.21.2.113:6822/36366","heartbeat_front_addr":"172.21.2.113:6815/36366","state":["exists","up"]},{"osd":6,"uuid":"bf67c040-0e50-48be-b102-c3943bc9ebef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6800","nonce":36367},{"type":"v1","addr":"172.21.2.113:6801","nonce":36367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6803","nonce":36367},{"type":"v1","addr":"172.21.2.113:6804","nonce":36367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6813","nonce":36367},{"type":"v1","addr":"172.21.2.113:6816","nonce":36367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6806","nonce":36367},{"type":"v1","addr":"172.21.2.113:6809","nonce":36367}]},"public_addr":"172.21.2.113:6801/36367","cluster_addr":"172.21.2.113:6804/36367","heartbeat_back_addr":"172.21.2.113:6816/36367","heartbeat_front_addr":"172.21.2.113:6809/36367","state":["exists","up"]},{"osd":7,"uuid":"f5461f80-018d-428f-be89-2eb2bcbe7bb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6817","nonce":36365},{"type":"v1","addr":"172.21.2.113:6821","nonce":36365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6823","nonce":36365},{"type":"v1","addr":"172.21.2.113:6825","nonce":36365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6830","nonce":36365},{"type":"v1","addr":"172.21.2.113:6831","nonce":36365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6827","nonce":36365},{"type":"v1","addr":"172.21.2.113:6829","nonce":36365}]},"public_addr":"172.21.2.113:6821/36365","cluster_addr":"172.21.2.113:6825/36365","heartbeat_back_addr":"172.21.2.113:6831/36365","heartbeat_front_addr":"172.21.2.113:6829/36365","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:03:51.433 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:51.434 7f99ab9c6700 1 -- 172.21.2.102:0/1572332174 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f9984018730 msgr2=0x7f998401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:51.434 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:51.434 7f99ab9c6700 1 --2- 172.21.2.102:0/1572332174 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f9984018730 0x7f998401ac10 crc :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:51.434 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:51.434 7f99ab9c6700 1 -- 172.21.2.102:0/1572332174 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f99a409cc60 msgr2=0x7f99a4086710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:51.434 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:51.434 7f99ab9c6700 1 --2- 172.21.2.102:0/1572332174 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f99a409cc60 0x7f99a4086710 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7f999800b5b0 tx=0x7f999800b950).stop 2021-02-10T11:03:51.434 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:51.435 7f99ab9c6700 1 -- 172.21.2.102:0/1572332174 shutdown_connections 2021-02-10T11:03:51.434 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:51.435 7f99ab9c6700 1 --2- 172.21.2.102:0/1572332174 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f99a4086d30 0x7f99a4087130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:03:51.434 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:51.435 7f99ab9c6700 1 --2- 172.21.2.102:0/1572332174 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f99a409cc60 0x7f99a4086710 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:51.434 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:51.435 7f99ab9c6700 1 --2- 172.21.2.102:0/1572332174 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f9984018730 0x7f998401ac10 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:51.434 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:51.435 7f99ab9c6700 1 --2- 172.21.2.102:0/1572332174 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f99a40b4920 0x7f99a409c640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:03:51.435 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:51.435 7f99ab9c6700 1 -- 172.21.2.102:0/1572332174 >> 172.21.2.102:0/1572332174 conn(0x7f99a4089ce0 msgr2=0x7f99a419cd30 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:03:51.435 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:51.435 7f99ab9c6700 1 -- 172.21.2.102:0/1572332174 shutdown_connections 2021-02-10T11:03:51.435 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:03:51.435 7f99ab9c6700 1 -- 172.21.2.102:0/1572332174 wait complete. 2021-02-10T11:03:51.441 DEBUG:teuthology.misc:8 of 8 OSDs are up 2021-02-10T11:03:51.442 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:03:51.872 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:51.856 7f0b69c41700 1 Processor -- start 2021-02-10T11:03:51.872 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:51.856 7f0b69c41700 1 -- start start 2021-02-10T11:03:51.873 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:51.856 7f0b69c41700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0b640e5470 0x7f0b6407e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:51.873 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:51.856 7f0b69c41700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f0b6407f300 0x7f0b6407f700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:51.873 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:51.856 7f0b69c41700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b6407ed80 con 0x7f0b640e5070 2021-02-10T11:03:51.873 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:51.856 7f0b69c41700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b641b4010 con 0x7f0b6407f300 2021-02-10T11:03:51.873 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:51.856 7f0b69c41700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b640d1790 con 0x7f0b640e5470 2021-02-10T11:03:51.873 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:51.856 7f0b63fff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f0b6407f300 0x7f0b6407f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:51.873 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:51.856 7f0b63fff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f0b6407f300 0x7f0b6407f700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:49306/0 (socket says 172.21.2.102:49306) 2021-02-10T11:03:51.873 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:51.856 7f0b637fe700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f0b640e5070 0x7f0b641b3820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:41526/0 (socket says 172.21.2.102:41526) 2021-02-10T11:03:51.873 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:51.856 7f0b63fff700 1 -- 172.21.2.102:0/748602130 learned_addr learned my addr 172.21.2.102:0/748602130 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:03:51.873 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:51.856 7f0b62ffd700 1 --2- 172.21.2.102:0/748602130 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0b640e5470 0x7f0b6407e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:51.873 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:51.857 7f0b63fff700 1 -- 172.21.2.102:0/748602130 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0b640e5470 msgr2=0x7f0b6407e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:03:51.873 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:51.857 7f0b63fff700 1 --2- 172.21.2.102:0/748602130 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0b640e5470 0x7f0b6407e6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:51.873 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:51.857 7f0b63fff700 1 -- 172.21.2.102:0/748602130 >> v1:172.21.2.102:6789/0 conn(0x7f0b640e5070 legacy=0x7f0b641b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:51.874 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:51.857 7f0b63fff700 1 -- 172.21.2.102:0/748602130 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b640d1980 con 0x7f0b6407f300 2021-02-10T11:03:51.874 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:51.857 7f0b63fff700 1 --2- 172.21.2.102:0/748602130 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0b6407f300 0x7f0b6407f700 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f0b5c006f00 tx=0x7f0b5c009f10).ready entity=mon.0 client_cookie=3e009318b7711320 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:03:51.874 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:51.857 7f0b627fc700 1 -- 172.21.2.102:0/748602130 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0b5c00b3b0 con 0x7f0b6407f300 2021-02-10T11:03:51.874 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:51.857 7f0b627fc700 1 -- 172.21.2.102:0/748602130 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b5c00b730 con 0x7f0b6407f300 2021-02-10T11:03:51.874 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:51.857 7f0b627fc700 1 -- 172.21.2.102:0/748602130 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0b5c00bc10 con 0x7f0b6407f300 2021-02-10T11:03:51.874 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:51.857 7f0b69c41700 1 -- 172.21.2.102:0/748602130 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0b6407f300 msgr2=0x7f0b6407f700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:51.874 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:51.857 7f0b69c41700 1 --2- 172.21.2.102:0/748602130 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0b6407f300 0x7f0b6407f700 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f0b5c006f00 tx=0x7f0b5c009f10).stop 2021-02-10T11:03:51.874 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:51.858 7f0b69c41700 1 -- 172.21.2.102:0/748602130 shutdown_connections 2021-02-10T11:03:51.874 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:51.858 7f0b69c41700 1 --2- 172.21.2.102:0/748602130 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0b6407f300 0x7f0b6407f700 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:51.874 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:51.858 7f0b69c41700 1 --2- 172.21.2.102:0/748602130 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0b640e5470 0x7f0b6407e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:51.874 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:51.858 7f0b69c41700 1 -- 172.21.2.102:0/748602130 >> 172.21.2.102:0/748602130 conn(0x7f0b640ce690 msgr2=0x7f0b640e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:03:51.874 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:51.858 7f0b69c41700 1 -- 172.21.2.102:0/748602130 shutdown_connections 2021-02-10T11:03:51.874 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:51.858 7f0b69c41700 1 -- 172.21.2.102:0/748602130 wait complete. 2021-02-10T11:03:51.875 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:51.873 7f0b69c41700 1 Processor -- start 2021-02-10T11:03:51.875 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:51.873 7f0b69c41700 1 -- start start 2021-02-10T11:03:51.876 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:51.874 7f0b69c41700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0b64085480 0x7f0b6419dcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:51.876 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:51.874 7f0b69c41700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0b640a83c0 0x7f0b6407f220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:51.876 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:51.874 7f0b69c41700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0b6407f840 0x7f0b64085f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:51.876 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:51.874 7f0b69c41700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b64085990 con 0x7f0b64085480 2021-02-10T11:03:51.876 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:51.874 7f0b69c41700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b6419e310 con 0x7f0b6407f840 2021-02-10T11:03:51.876 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:51.874 7f0b69c41700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b6407fc40 con 0x7f0b640a83c0 2021-02-10T11:03:51.877 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:51.874 7f0b637fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0b64085480 0x7f0b6419dcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:51.877 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:51.874 7f0b637fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0b64085480 0x7f0b6419dcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:49308/0 (socket says 172.21.2.102:49308) 2021-02-10T11:03:51.877 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:51.874 7f0b637fe700 1 -- 172.21.2.102:0/2267879405 learned_addr learned my addr 172.21.2.102:0/2267879405 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:03:51.877 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:51.874 7f0b62ffd700 1 --2- 172.21.2.102:0/2267879405 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0b640a83c0 0x7f0b6407f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:51.877 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:51.874 7f0b63fff700 1 --2- 172.21.2.102:0/2267879405 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0b6407f840 0x7f0b64085f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:51.877 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:51.874 7f0b637fe700 1 -- 172.21.2.102:0/2267879405 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0b640a83c0 msgr2=0x7f0b6407f220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:51.877 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:51.874 7f0b637fe700 1 --2- 172.21.2.102:0/2267879405 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0b640a83c0 0x7f0b6407f220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:51.877 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:51.874 7f0b637fe700 1 -- 172.21.2.102:0/2267879405 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0b6407f840 msgr2=0x7f0b64085f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:51.877 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:51.874 7f0b637fe700 1 --2- 172.21.2.102:0/2267879405 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0b6407f840 0x7f0b64085f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:51.877 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:51.874 7f0b637fe700 1 -- 172.21.2.102:0/2267879405 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b64080010 con 0x7f0b64085480 2021-02-10T11:03:51.877 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:51.874 7f0b62ffd700 1 --2- 172.21.2.102:0/2267879405 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0b640a83c0 0x7f0b6407f220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:03:51.877 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:51.874 7f0b637fe700 1 --2- 172.21.2.102:0/2267879405 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0b64085480 0x7f0b6419dcf0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f0b5400a590 tx=0x7f0b5400a930).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:03:51.878 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:51.874 7f0b5bfff700 1 -- 172.21.2.102:0/2267879405 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0b54000ef0 con 0x7f0b64085480 2021-02-10T11:03:51.878 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:51.874 7f0b5bfff700 1 -- 172.21.2.102:0/2267879405 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b5400ad20 con 0x7f0b64085480 2021-02-10T11:03:51.878 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:51.875 7f0b69c41700 1 -- 172.21.2.102:0/2267879405 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b64080010 con 0x7f0b64085480 2021-02-10T11:03:51.878 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:51.875 7f0b5bfff700 1 -- 172.21.2.102:0/2267879405 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0b54004590 con 0x7f0b64085480 2021-02-10T11:03:51.878 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:51.875 7f0b69c41700 1 -- 172.21.2.102:0/2267879405 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b64080010 con 0x7f0b64085480 2021-02-10T11:03:51.878 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:51.875 7f0b5bfff700 1 -- 172.21.2.102:0/2267879405 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f0b540043a0 con 0x7f0b64085480 2021-02-10T11:03:51.878 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:51.875 7f0b5bfff700 1 --2- 172.21.2.102:0/2267879405 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0b48018730 0x7f0b4801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:51.878 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:51.875 7f0b5bfff700 1 -- 172.21.2.102:0/2267879405 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f0b54032630 con 0x7f0b64085480 2021-02-10T11:03:51.878 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:51.875 7f0b62ffd700 1 --2- 172.21.2.102:0/2267879405 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0b48018730 0x7f0b4801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:51.878 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:51.875 7f0b62ffd700 1 --2- 172.21.2.102:0/2267879405 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0b48018730 0x7f0b4801ac10 crc :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:03:51.905 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:51.906 7f0b69c41700 1 -- 172.21.2.102:0/2267879405 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b6407ed80 con 0x7f0b64085480 2021-02-10T11:03:51.907 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:51.908 7f0b5bfff700 1 -- 172.21.2.102:0/2267879405 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f0b54007070 con 0x7f0b64085480 2021-02-10T11:03:52.036 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:52.037 7f0b69c41700 1 -- 172.21.2.102:0/2267879405 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0b6407ed80 con 0x7f0b64085480 2021-02-10T11:03:52.036 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:52.037 7f0b5bfff700 1 -- 172.21.2.102:0/2267879405 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+13993 (secure 0 0 0) 0x7f0b540043a0 con 0x7f0b64085480 2021-02-10T11:03:52.037 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:03:52.037 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":20,"fsid":"68db30ea-dbce-4aef-a6df-f01c6fea0513","created":"2021-02-10 11:03:06.331557","modified":"2021-02-10 11:03:44.386945","last_up_change":"2021-02-10 11:03:32.690562","last_in_change":"2021-02-10 11:03:32.690562","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:03:37.322519","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:03:41.161563","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2021-02-10 11:03:42.379651","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"be691b4d-a039-408a-af80-9d53f02b6f1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6818","nonce":36088},{"type":"v1","addr":"172.21.2.102:6820","nonce":36088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6822","nonce":36088},{"type":"v1","addr":"172.21.2.102:6824","nonce":36088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6830","nonce":36088},{"type":"v1","addr":"172.21.2.102:6832","nonce":36088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6826","nonce":36088},{"type":"v1","addr":"172.21.2.102:6828","nonce":36088}]},"public_addr":"172.21.2.102:6820/36088","cluster_addr":"172.21.2.102:6824/36088","heartbeat_back_addr":"172.21.2.102:6832/36088","heartbeat_front_addr":"172.21.2.102:6828/36088","state":["exists","up"]},{"osd":1,"uuid":"64e3ccc8-9c69-4d7c-865e-af8e83822635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6819","nonce":36087},{"type":"v1","addr":"172.21.2.102:6821","nonce":36087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6823","nonce":36087},{"type":"v1","addr":"172.21.2.102:6825","nonce":36087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6831","nonce":36087},{"type":"v1","addr":"172.21.2.102:6833","nonce":36087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6827","nonce":36087},{"type":"v1","addr":"172.21.2.102:6829","nonce":36087}]},"public_addr":"172.21.2.102:6821/36087","cluster_addr":"172.21.2.102:6825/36087","heartbeat_back_addr":"172.21.2.102:6833/36087","heartbeat_front_addr":"172.21.2.102:6829/36087","state":["exists","up"]},{"osd":2,"uuid":"153700af-747f-481f-bdf4-d6072d3a842f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6810","nonce":36089},{"type":"v1","addr":"172.21.2.102:6811","nonce":36089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6812","nonce":36089},{"type":"v1","addr":"172.21.2.102:6813","nonce":36089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6816","nonce":36089},{"type":"v1","addr":"172.21.2.102:6817","nonce":36089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6814","nonce":36089},{"type":"v1","addr":"172.21.2.102:6815","nonce":36089}]},"public_addr":"172.21.2.102:6811/36089","cluster_addr":"172.21.2.102:6813/36089","heartbeat_back_addr":"172.21.2.102:6817/36089","heartbeat_front_addr":"172.21.2.102:6815/36089","state":["exists","up"]},{"osd":3,"uuid":"cbc678f4-83a9-45e6-8830-2f7c232b1ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6802","nonce":36090},{"type":"v1","addr":"172.21.2.102:6803","nonce":36090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6804","nonce":36090},{"type":"v1","addr":"172.21.2.102:6805","nonce":36090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6808","nonce":36090},{"type":"v1","addr":"172.21.2.102:6809","nonce":36090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6806","nonce":36090},{"type":"v1","addr":"172.21.2.102:6807","nonce":36090}]},"public_addr":"172.21.2.102:6803/36090","cluster_addr":"172.21.2.102:6805/36090","heartbeat_back_addr":"172.21.2.102:6809/36090","heartbeat_front_addr":"172.21.2.102:6807/36090","state":["exists","up"]},{"osd":4,"uuid":"c01e4008-b258-4f10-afd2-9e7f37214351","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6808","nonce":36368},{"type":"v1","addr":"172.21.2.113:6811","nonce":36368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6814","nonce":36368},{"type":"v1","addr":"172.21.2.113:6818","nonce":36368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6826","nonce":36368},{"type":"v1","addr":"172.21.2.113:6828","nonce":36368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6820","nonce":36368},{"type":"v1","addr":"172.21.2.113:6824","nonce":36368}]},"public_addr":"172.21.2.113:6811/36368","cluster_addr":"172.21.2.113:6818/36368","heartbeat_back_addr":"172.21.2.113:6828/36368","heartbeat_front_addr":"172.21.2.113:6824/36368","state":["exists","up"]},{"osd":5,"uuid":"551f6007-daf8-4a98-8ee0-c2b16bcccb5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6802","nonce":36366},{"type":"v1","addr":"172.21.2.113:6805","nonce":36366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6807","nonce":36366},{"type":"v1","addr":"172.21.2.113:6810","nonce":36366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6819","nonce":36366},{"type":"v1","addr":"172.21.2.113:6822","nonce":36366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6812","nonce":36366},{"type":"v1","addr":"172.21.2.113:6815","nonce":36366}]},"public_addr":"172.21.2.113:6805/36366","cluster_addr":"172.21.2.113:6810/36366","heartbeat_back_addr":"172.21.2.113:6822/36366","heartbeat_front_addr":"172.21.2.113:6815/36366","state":["exists","up"]},{"osd":6,"uuid":"bf67c040-0e50-48be-b102-c3943bc9ebef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6800","nonce":36367},{"type":"v1","addr":"172.21.2.113:6801","nonce":36367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6803","nonce":36367},{"type":"v1","addr":"172.21.2.113:6804","nonce":36367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6813","nonce":36367},{"type":"v1","addr":"172.21.2.113:6816","nonce":36367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6806","nonce":36367},{"type":"v1","addr":"172.21.2.113:6809","nonce":36367}]},"public_addr":"172.21.2.113:6801/36367","cluster_addr":"172.21.2.113:6804/36367","heartbeat_back_addr":"172.21.2.113:6816/36367","heartbeat_front_addr":"172.21.2.113:6809/36367","state":["exists","up"]},{"osd":7,"uuid":"f5461f80-018d-428f-be89-2eb2bcbe7bb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6817","nonce":36365},{"type":"v1","addr":"172.21.2.113:6821","nonce":36365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6823","nonce":36365},{"type":"v1","addr":"172.21.2.113:6825","nonce":36365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6830","nonce":36365},{"type":"v1","addr":"172.21.2.113:6831","nonce":36365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6827","nonce":36365},{"type":"v1","addr":"172.21.2.113:6829","nonce":36365}]},"public_addr":"172.21.2.113:6821/36365","cluster_addr":"172.21.2.113:6825/36365","heartbeat_back_addr":"172.21.2.113:6831/36365","heartbeat_front_addr":"172.21.2.113:6829/36365","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:03:52.038 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:52.039 7f0b69c41700 1 -- 172.21.2.102:0/2267879405 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0b48018730 msgr2=0x7f0b4801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:52.038 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:52.039 7f0b69c41700 1 --2- 172.21.2.102:0/2267879405 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0b48018730 0x7f0b4801ac10 crc :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:52.038 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:52.039 7f0b69c41700 1 -- 172.21.2.102:0/2267879405 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0b64085480 msgr2=0x7f0b6419dcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:52.038 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:52.039 7f0b69c41700 1 --2- 172.21.2.102:0/2267879405 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0b64085480 0x7f0b6419dcf0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f0b5400a590 tx=0x7f0b5400a930).stop 2021-02-10T11:03:52.038 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:52.039 7f0b69c41700 1 -- 172.21.2.102:0/2267879405 shutdown_connections 2021-02-10T11:03:52.039 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:52.039 7f0b69c41700 1 --2- 172.21.2.102:0/2267879405 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0b48018730 0x7f0b4801ac10 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:52.039 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:52.039 7f0b69c41700 1 --2- 172.21.2.102:0/2267879405 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0b6407f840 0x7f0b64085f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:52.039 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:52.039 7f0b69c41700 1 --2- 172.21.2.102:0/2267879405 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0b640a83c0 0x7f0b6407f220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:52.039 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:52.039 7f0b69c41700 1 --2- 172.21.2.102:0/2267879405 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0b64085480 0x7f0b6419dcf0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:52.039 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:52.039 7f0b69c41700 1 -- 172.21.2.102:0/2267879405 >> 172.21.2.102:0/2267879405 conn(0x7f0b6418f110 msgr2=0x7f0b64190370 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:03:52.039 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:52.039 7f0b69c41700 1 -- 172.21.2.102:0/2267879405 shutdown_connections 2021-02-10T11:03:52.039 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:52.039 7f0b69c41700 1 -- 172.21.2.102:0/2267879405 wait complete. 2021-02-10T11:03:52.047 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell osd.0 flush_pg_stats 2021-02-10T11:03:52.473 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:52.457 7f9b3743d700 1 Processor -- start 2021-02-10T11:03:52.473 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:52.458 7f9b3743d700 1 -- start start 2021-02-10T11:03:52.473 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:52.458 7f9b3743d700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9b300e5070 0x7f9b301b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:52.474 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:52.458 7f9b3743d700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f9b301b3e40 0x7f9b30080310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:52.474 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:52.458 7f9b3743d700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b3007ed80 con 0x7f9b300e5470 2021-02-10T11:03:52.474 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:52.458 7f9b3743d700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b300d1790 con 0x7f9b301b3e40 2021-02-10T11:03:52.474 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:52.458 7f9b3743d700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b300d1980 con 0x7f9b300e5070 2021-02-10T11:03:52.474 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:52.458 7f9b35a5a700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f9b301b3e40 0x7f9b30080310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:52.474 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:52.458 7f9b35259700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9b300e5070 0x7f9b301b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:52.474 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:52.458 7f9b34a58700 1 --1- >> v1:172.21.2.113:6789/0 conn(0x7f9b300e5470 0x7f9b3007d690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.113:6789/0 says I am v1:172.21.2.102:48864/0 (socket says 172.21.2.102:48864) 2021-02-10T11:03:52.474 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:52.458 7f9b34a58700 1 -- 172.21.2.102:0/3802846531 learned_addr learned my addr 172.21.2.102:0/3802846531 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:03:52.474 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:52.459 7f9b35259700 1 -- 172.21.2.102:0/3802846531 >> v2:172.21.2.113:3300/0 conn(0x7f9b301b3e40 msgr2=0x7f9b30080310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:03:52.474 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:52.459 7f9b35259700 1 --2- 172.21.2.102:0/3802846531 >> v2:172.21.2.113:3300/0 conn(0x7f9b301b3e40 0x7f9b30080310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:52.474 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:52.459 7f9b35259700 1 -- 172.21.2.102:0/3802846531 >> v1:172.21.2.113:6789/0 conn(0x7f9b300e5470 legacy=0x7f9b3007d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:52.474 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:52.459 7f9b35259700 1 -- 172.21.2.102:0/3802846531 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b300d1ba0 con 0x7f9b300e5070 2021-02-10T11:03:52.474 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:52.459 7f9b27fff700 1 -- 172.21.2.102:0/3802846531 <== mon.1 v1:172.21.2.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1337977452 0 0) 0x7f9b200010b0 con 0x7f9b300e5470 2021-02-10T11:03:52.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:52.459 7f9b35a5a700 1 --2- 172.21.2.102:0/3802846531 >> v2:172.21.2.113:3300/0 conn(0x7f9b301b3e40 0x7f9b30080310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:03:52.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:52.459 7f9b35259700 1 --2- 172.21.2.102:0/3802846531 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9b300e5070 0x7f9b301b3820 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f9b2c006650 tx=0x7f9b2c009a80).ready entity=mon.2 client_cookie=f0cbe5b3717f0985 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:03:52.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:52.459 7f9b27fff700 1 -- 172.21.2.102:0/3802846531 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9b2c00c0c0 con 0x7f9b300e5070 2021-02-10T11:03:52.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:52.459 7f9b27fff700 1 -- 172.21.2.102:0/3802846531 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b2c00c340 con 0x7f9b300e5070 2021-02-10T11:03:52.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:52.459 7f9b27fff700 1 -- 172.21.2.102:0/3802846531 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9b2c00c860 con 0x7f9b300e5070 2021-02-10T11:03:52.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:52.459 7f9b3743d700 1 -- 172.21.2.102:0/3802846531 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9b300e5070 msgr2=0x7f9b301b3820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:52.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:52.459 7f9b3743d700 1 --2- 172.21.2.102:0/3802846531 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9b300e5070 0x7f9b301b3820 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f9b2c006650 tx=0x7f9b2c009a80).stop 2021-02-10T11:03:52.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:52.460 7f9b3743d700 1 -- 172.21.2.102:0/3802846531 shutdown_connections 2021-02-10T11:03:52.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:52.460 7f9b3743d700 1 --2- 172.21.2.102:0/3802846531 >> v2:172.21.2.113:3300/0 conn(0x7f9b301b3e40 0x7f9b30080310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:52.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:52.460 7f9b3743d700 1 --2- 172.21.2.102:0/3802846531 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9b300e5070 0x7f9b301b3820 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:52.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:52.460 7f9b3743d700 1 -- 172.21.2.102:0/3802846531 >> 172.21.2.102:0/3802846531 conn(0x7f9b300ce690 msgr2=0x7f9b300e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:03:52.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:52.460 7f9b3743d700 1 -- 172.21.2.102:0/3802846531 shutdown_connections 2021-02-10T11:03:52.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:52.460 7f9b3743d700 1 -- 172.21.2.102:0/3802846531 wait complete. 2021-02-10T11:03:52.477 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:52.475 7f9b3743d700 1 Processor -- start 2021-02-10T11:03:52.477 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:52.475 7f9b3743d700 1 -- start start 2021-02-10T11:03:52.477 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:52.475 7f9b3743d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9b301a43e0 0x7f9b300b5180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:52.477 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:52.475 7f9b3743d700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9b300b57a0 0x7f9b300b5ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:52.477 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:52.475 7f9b3743d700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f9b300b61c0 0x7f9b300a5420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:52.477 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:52.475 7f9b3743d700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b300b4c00 con 0x7f9b301a43e0 2021-02-10T11:03:52.477 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:52.475 7f9b3743d700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b300a5a40 con 0x7f9b300b61c0 2021-02-10T11:03:52.477 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:52.475 7f9b3743d700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b300a5c20 con 0x7f9b300b57a0 2021-02-10T11:03:52.477 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:52.475 7f9b35259700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9b301a43e0 0x7f9b300b5180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:52.477 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:52.475 7f9b35259700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9b301a43e0 0x7f9b300b5180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:49320/0 (socket says 172.21.2.102:49320) 2021-02-10T11:03:52.477 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:52.475 7f9b35259700 1 -- 172.21.2.102:0/2201212460 learned_addr learned my addr 172.21.2.102:0/2201212460 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:03:52.477 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:52.476 7f9b35259700 1 -- 172.21.2.102:0/2201212460 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9b300b57a0 msgr2=0x7f9b300b5ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:52.478 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:52.476 7f9b35259700 1 --2- 172.21.2.102:0/2201212460 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9b300b57a0 0x7f9b300b5ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:03:52.478 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:52.476 7f9b35259700 1 -- 172.21.2.102:0/2201212460 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f9b300b61c0 msgr2=0x7f9b300a5420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:52.478 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:52.476 7f9b35259700 1 --2- 172.21.2.102:0/2201212460 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f9b300b61c0 0x7f9b300a5420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:03:52.478 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:52.476 7f9b35259700 1 -- 172.21.2.102:0/2201212460 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b300a5e40 con 0x7f9b301a43e0 2021-02-10T11:03:52.478 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:52.476 7f9b35259700 1 --2- 172.21.2.102:0/2201212460 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9b301a43e0 0x7f9b300b5180 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f9b2c006f60 tx=0x7f9b2c00b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:03:52.478 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:52.476 7f9b25ffb700 1 -- 172.21.2.102:0/2201212460 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9b2c0093a0 con 0x7f9b301a43e0 2021-02-10T11:03:52.478 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:52.476 7f9b25ffb700 1 -- 172.21.2.102:0/2201212460 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b2c00bd40 con 0x7f9b301a43e0 2021-02-10T11:03:52.478 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:52.476 7f9b3743d700 1 -- 172.21.2.102:0/2201212460 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b300a5e40 con 0x7f9b301a43e0 2021-02-10T11:03:52.478 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:52.476 7f9b25ffb700 1 -- 172.21.2.102:0/2201212460 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9b2c002270 con 0x7f9b301a43e0 2021-02-10T11:03:52.478 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:52.476 7f9b3743d700 1 -- 172.21.2.102:0/2201212460 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b300a63d0 con 0x7f9b301a43e0 2021-02-10T11:03:52.478 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:52.477 7f9b25ffb700 1 -- 172.21.2.102:0/2201212460 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f9b2c000c90 con 0x7f9b301a43e0 2021-02-10T11:03:52.478 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:52.477 7f9b25ffb700 1 --2- 172.21.2.102:0/2201212460 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f9b10018730 0x7f9b1001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:52.478 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:52.477 7f9b25ffb700 1 -- 172.21.2.102:0/2201212460 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f9b2c032800 con 0x7f9b301a43e0 2021-02-10T11:03:52.479 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:52.477 7f9b34a58700 1 --2- 172.21.2.102:0/2201212460 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f9b10018730 0x7f9b1001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:52.479 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:52.477 7f9b34a58700 1 --2- 172.21.2.102:0/2201212460 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f9b10018730 0x7f9b1001abc0 crc :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:03:52.507 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:52.508 7f9b3743d700 1 --2- 172.21.2.102:0/2201212460 >> [v2:172.21.2.102:6818/36088,v1:172.21.2.102:6820/36088] conn(0x7f9b300a5a40 0x7f9b300627c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:52.507 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:52.508 7f9b3743d700 1 -- 172.21.2.102:0/2201212460 --> [v2:172.21.2.102:6818/36088,v1:172.21.2.102:6820/36088] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9b300a63d0 con 0x7f9b300a5a40 2021-02-10T11:03:52.507 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:52.508 7f9b35a5a700 1 --2- 172.21.2.102:0/2201212460 >> [v2:172.21.2.102:6818/36088,v1:172.21.2.102:6820/36088] conn(0x7f9b300a5a40 0x7f9b300627c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:52.508 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:52.509 7f9b35a5a700 1 --2- 172.21.2.102:0/2201212460 >> [v2:172.21.2.102:6818/36088,v1:172.21.2.102:6820/36088] conn(0x7f9b300a5a40 0x7f9b300627c0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:03:52.508 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:52.509 7f9b25ffb700 1 -- 172.21.2.102:0/2201212460 <== osd.0 v2:172.21.2.102:6818/36088 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+5151 (crc 0 0 0) 0x7f9b28000f40 con 0x7f9b300a5a40 2021-02-10T11:03:52.520 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:52.516 7f9b3743d700 1 -- 172.21.2.102:0/2201212460 --> [v2:172.21.2.102:6818/36088,v1:172.21.2.102:6820/36088] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f9b300a63d0 con 0x7f9b300a5a40 2021-02-10T11:03:52.520 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:52.516 7f9b25ffb700 1 -- 172.21.2.102:0/2201212460 <== osd.0 v2:172.21.2.102:6818/36088 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f9b28000fa0 con 0x7f9b300a5a40 2021-02-10T11:03:52.520 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:52.517 7f9b3743d700 1 -- 172.21.2.102:0/2201212460 >> [v2:172.21.2.102:6818/36088,v1:172.21.2.102:6820/36088] conn(0x7f9b300a5a40 msgr2=0x7f9b300627c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:52.520 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:52.517 7f9b3743d700 1 --2- 172.21.2.102:0/2201212460 >> [v2:172.21.2.102:6818/36088,v1:172.21.2.102:6820/36088] conn(0x7f9b300a5a40 0x7f9b300627c0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:52.520 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:52.517 7f9b3743d700 1 -- 172.21.2.102:0/2201212460 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f9b10018730 msgr2=0x7f9b1001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:52.520 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:52.517 7f9b3743d700 1 --2- 172.21.2.102:0/2201212460 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f9b10018730 0x7f9b1001abc0 crc :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:52.520 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:52.517 7f9b3743d700 1 -- 172.21.2.102:0/2201212460 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9b301a43e0 msgr2=0x7f9b300b5180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:52.521 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:52.517 7f9b3743d700 1 --2- 172.21.2.102:0/2201212460 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9b301a43e0 0x7f9b300b5180 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f9b2c006f60 tx=0x7f9b2c00b950).stop 2021-02-10T11:03:52.521 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:52.517 7f9b35a5a700 1 -- 172.21.2.102:0/2201212460 reap_dead start 2021-02-10T11:03:52.521 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:52.518 7f9b3743d700 1 -- 172.21.2.102:0/2201212460 shutdown_connections 2021-02-10T11:03:52.521 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:52.518 7f9b3743d700 1 -- 172.21.2.102:0/2201212460 >> 172.21.2.102:0/2201212460 conn(0x7f9b300beb60 msgr2=0x7f9b30091c60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:03:52.521 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:52.518 7f9b3743d700 1 -- 172.21.2.102:0/2201212460 shutdown_connections 2021-02-10T11:03:52.521 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:52.518 7f9b3743d700 1 -- 172.21.2.102:0/2201212460 wait complete. 2021-02-10T11:03:52.522 INFO:teuthology.orchestra.run.gibba002.stdout:55834574854 2021-02-10T11:03:52.524 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell osd.1 flush_pg_stats 2021-02-10T11:03:52.951 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:52.935 7efe5e87b700 1 Processor -- start 2021-02-10T11:03:52.951 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:52.935 7efe5e87b700 1 -- start start 2021-02-10T11:03:52.951 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:52.935 7efe5e87b700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7efe580e5070 0x7efe581b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:52.952 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:52.935 7efe5e87b700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7efe581b3e40 0x7efe58080310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:52.952 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:52.935 7efe5e87b700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe5807ed80 con 0x7efe580e5470 2021-02-10T11:03:52.952 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:52.935 7efe5e87b700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe580d1790 con 0x7efe581b3e40 2021-02-10T11:03:52.952 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:52.935 7efe5e87b700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe580d1980 con 0x7efe580e5070 2021-02-10T11:03:52.952 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:52.936 7efe577fe700 1 --1- >> v1:172.21.2.113:6789/0 conn(0x7efe580e5470 0x7efe5807d690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.113:6789/0 says I am v1:172.21.2.102:48878/0 (socket says 172.21.2.102:48878) 2021-02-10T11:03:52.952 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:52.936 7efe577fe700 1 -- 172.21.2.102:0/2485428243 learned_addr learned my addr 172.21.2.102:0/2485428243 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:03:52.952 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:52.936 7efe57fff700 1 --2- 172.21.2.102:0/2485428243 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7efe580e5070 0x7efe581b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:52.952 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:52.936 7efe5ce98700 1 --2- 172.21.2.102:0/2485428243 >> v2:172.21.2.113:3300/0 conn(0x7efe581b3e40 0x7efe58080310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:52.952 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:52.937 7efe57fff700 1 -- 172.21.2.102:0/2485428243 >> v2:172.21.2.113:3300/0 conn(0x7efe581b3e40 msgr2=0x7efe58080310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:03:52.952 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:52.937 7efe57fff700 1 --2- 172.21.2.102:0/2485428243 >> v2:172.21.2.113:3300/0 conn(0x7efe581b3e40 0x7efe58080310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:52.952 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:52.937 7efe57fff700 1 -- 172.21.2.102:0/2485428243 >> v1:172.21.2.113:6789/0 conn(0x7efe580e5470 legacy=0x7efe5807d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:52.952 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:52.937 7efe57fff700 1 -- 172.21.2.102:0/2485428243 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe580d1ba0 con 0x7efe580e5070 2021-02-10T11:03:52.952 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:52.937 7efe5ce98700 1 --2- 172.21.2.102:0/2485428243 >> v2:172.21.2.113:3300/0 conn(0x7efe581b3e40 0x7efe58080310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:03:52.953 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:52.937 7efe57fff700 1 --2- 172.21.2.102:0/2485428243 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7efe580e5070 0x7efe581b3820 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7efe48006650 tx=0x7efe48009a80).ready entity=mon.2 client_cookie=7bd321c7d552e312 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:03:52.953 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:52.937 7efe56ffd700 1 -- 172.21.2.102:0/2485428243 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efe480009b0 con 0x7efe580e5070 2021-02-10T11:03:52.953 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:52.937 7efe56ffd700 1 -- 172.21.2.102:0/2485428243 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe4800c0c0 con 0x7efe580e5070 2021-02-10T11:03:52.953 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:52.937 7efe56ffd700 1 -- 172.21.2.102:0/2485428243 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efe4800c5e0 con 0x7efe580e5070 2021-02-10T11:03:52.953 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:52.937 7efe5e87b700 1 -- 172.21.2.102:0/2485428243 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7efe580e5070 msgr2=0x7efe581b3820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:52.953 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:52.937 7efe5e87b700 1 --2- 172.21.2.102:0/2485428243 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7efe580e5070 0x7efe581b3820 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7efe48006650 tx=0x7efe48009a80).stop 2021-02-10T11:03:52.953 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:52.938 7efe5e87b700 1 -- 172.21.2.102:0/2485428243 shutdown_connections 2021-02-10T11:03:52.953 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:52.938 7efe5e87b700 1 --2- 172.21.2.102:0/2485428243 >> v2:172.21.2.113:3300/0 conn(0x7efe581b3e40 0x7efe58080310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:52.953 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:52.938 7efe5e87b700 1 --2- 172.21.2.102:0/2485428243 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7efe580e5070 0x7efe581b3820 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:52.953 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:52.938 7efe5e87b700 1 -- 172.21.2.102:0/2485428243 >> 172.21.2.102:0/2485428243 conn(0x7efe580ce690 msgr2=0x7efe580e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:03:52.953 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:52.938 7efe5e87b700 1 -- 172.21.2.102:0/2485428243 shutdown_connections 2021-02-10T11:03:52.953 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:52.938 7efe5e87b700 1 -- 172.21.2.102:0/2485428243 wait complete. 2021-02-10T11:03:52.955 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:52.953 7efe5e87b700 1 Processor -- start 2021-02-10T11:03:52.955 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:52.953 7efe5e87b700 1 -- start start 2021-02-10T11:03:52.955 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:52.953 7efe5e87b700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efe580be120 0x7efe580be720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:52.955 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:52.953 7efe5e87b700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7efe580896d0 0x7efe58198b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:52.955 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:52.953 7efe5e87b700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7efe58199150 0x7efe580ba680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:52.955 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:52.953 7efe5e87b700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe580baca0 con 0x7efe580be120 2021-02-10T11:03:52.955 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:52.953 7efe5e87b700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe580baec0 con 0x7efe580896d0 2021-02-10T11:03:52.955 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:52.953 7efe5e87b700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe58089ad0 con 0x7efe58199150 2021-02-10T11:03:52.955 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:52.953 7efe57fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efe580be120 0x7efe580be720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:52.955 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:52.953 7efe57fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efe580be120 0x7efe580be720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:49336/0 (socket says 172.21.2.102:49336) 2021-02-10T11:03:52.955 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:52.953 7efe57fff700 1 -- 172.21.2.102:0/547533561 learned_addr learned my addr 172.21.2.102:0/547533561 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:03:52.955 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:52.954 7efe57fff700 1 -- 172.21.2.102:0/547533561 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7efe58199150 msgr2=0x7efe580ba680 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:03:52.956 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:52.954 7efe57fff700 1 --2- 172.21.2.102:0/547533561 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7efe58199150 0x7efe580ba680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:03:52.956 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:52.954 7efe57fff700 1 -- 172.21.2.102:0/547533561 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7efe580896d0 msgr2=0x7efe58198b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:52.956 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:52.954 7efe57fff700 1 --2- 172.21.2.102:0/547533561 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7efe580896d0 0x7efe58198b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:03:52.956 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:52.954 7efe57fff700 1 -- 172.21.2.102:0/547533561 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe58089cf0 con 0x7efe580be120 2021-02-10T11:03:52.956 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:52.954 7efe57fff700 1 --2- 172.21.2.102:0/547533561 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efe580be120 0x7efe580be720 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7efe48009630 tx=0x7efe4800b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:03:52.956 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:52.954 7efe54ff9700 1 -- 172.21.2.102:0/547533561 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efe480093a0 con 0x7efe580be120 2021-02-10T11:03:52.956 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:52.954 7efe54ff9700 1 -- 172.21.2.102:0/547533561 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe4800bd70 con 0x7efe580be120 2021-02-10T11:03:52.956 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:52.954 7efe54ff9700 1 -- 172.21.2.102:0/547533561 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efe48005560 con 0x7efe580be120 2021-02-10T11:03:52.956 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:52.954 7efe5e87b700 1 -- 172.21.2.102:0/547533561 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe58089cf0 con 0x7efe580be120 2021-02-10T11:03:52.956 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:52.954 7efe5e87b700 1 -- 172.21.2.102:0/547533561 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe580bf0e0 con 0x7efe580be120 2021-02-10T11:03:52.956 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:52.955 7efe54ff9700 1 -- 172.21.2.102:0/547533561 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7efe48007020 con 0x7efe580be120 2021-02-10T11:03:52.956 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:52.955 7efe54ff9700 1 --2- 172.21.2.102:0/547533561 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7efe380186c0 0x7efe3801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:52.956 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:52.955 7efe54ff9700 1 -- 172.21.2.102:0/547533561 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5573+0+0 (secure 0 0 0) 0x7efe48032560 con 0x7efe580be120 2021-02-10T11:03:52.957 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:52.955 7efe577fe700 1 --2- 172.21.2.102:0/547533561 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7efe380186c0 0x7efe3801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:52.957 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:52.955 7efe577fe700 1 --2- 172.21.2.102:0/547533561 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7efe380186c0 0x7efe3801abc0 crc :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:03:52.984 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:52.985 7efe5e87b700 1 --2- 172.21.2.102:0/547533561 >> [v2:172.21.2.102:6819/36087,v1:172.21.2.102:6821/36087] conn(0x7efe580c45a0 0x7efe580bf970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:52.984 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:52.985 7efe5e87b700 1 -- 172.21.2.102:0/547533561 --> [v2:172.21.2.102:6819/36087,v1:172.21.2.102:6821/36087] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7efe580bf0e0 con 0x7efe580c45a0 2021-02-10T11:03:52.984 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:52.985 7efe5ce98700 1 --2- 172.21.2.102:0/547533561 >> [v2:172.21.2.102:6819/36087,v1:172.21.2.102:6821/36087] conn(0x7efe580c45a0 0x7efe580bf970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:52.984 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:52.985 7efe5ce98700 1 --2- 172.21.2.102:0/547533561 >> [v2:172.21.2.102:6819/36087,v1:172.21.2.102:6821/36087] conn(0x7efe580c45a0 0x7efe580bf970 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:03:52.985 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:52.986 7efe54ff9700 1 -- 172.21.2.102:0/547533561 <== osd.1 v2:172.21.2.102:6819/36087 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+5151 (crc 0 0 0) 0x7efe500067b0 con 0x7efe580c45a0 2021-02-10T11:03:52.991 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:52.993 7efe5e87b700 1 -- 172.21.2.102:0/547533561 --> [v2:172.21.2.102:6819/36087,v1:172.21.2.102:6821/36087] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7efe580bf0e0 con 0x7efe580c45a0 2021-02-10T11:03:52.992 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:52.993 7efe54ff9700 1 -- 172.21.2.102:0/547533561 <== osd.1 v2:172.21.2.102:6819/36087 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7efe500067b0 con 0x7efe580c45a0 2021-02-10T11:03:52.993 INFO:teuthology.orchestra.run.gibba002.stdout:55834574855 2021-02-10T11:03:52.993 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:52.994 7efe5e87b700 1 -- 172.21.2.102:0/547533561 >> [v2:172.21.2.102:6819/36087,v1:172.21.2.102:6821/36087] conn(0x7efe580c45a0 msgr2=0x7efe580bf970 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:52.993 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:52.994 7efe5e87b700 1 --2- 172.21.2.102:0/547533561 >> [v2:172.21.2.102:6819/36087,v1:172.21.2.102:6821/36087] conn(0x7efe580c45a0 0x7efe580bf970 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:52.994 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:52.994 7efe5e87b700 1 -- 172.21.2.102:0/547533561 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7efe380186c0 msgr2=0x7efe3801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:52.994 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:52.994 7efe5e87b700 1 --2- 172.21.2.102:0/547533561 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7efe380186c0 0x7efe3801abc0 crc :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:52.994 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:52.994 7efe5e87b700 1 -- 172.21.2.102:0/547533561 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efe580be120 msgr2=0x7efe580be720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:52.994 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:52.994 7efe5e87b700 1 --2- 172.21.2.102:0/547533561 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efe580be120 0x7efe580be720 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7efe48009630 tx=0x7efe4800b980).stop 2021-02-10T11:03:52.994 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:52.994 7efe5ce98700 1 -- 172.21.2.102:0/547533561 reap_dead start 2021-02-10T11:03:52.994 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:52.994 7efe5e87b700 1 -- 172.21.2.102:0/547533561 shutdown_connections 2021-02-10T11:03:52.994 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:52.994 7efe5e87b700 1 -- 172.21.2.102:0/547533561 >> 172.21.2.102:0/547533561 conn(0x7efe58098740 msgr2=0x7efe580c3590 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:03:52.994 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:52.994 7efe5e87b700 1 -- 172.21.2.102:0/547533561 shutdown_connections 2021-02-10T11:03:52.995 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:52.994 7efe5e87b700 1 -- 172.21.2.102:0/547533561 wait complete. 2021-02-10T11:03:53.001 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell osd.2 flush_pg_stats 2021-02-10T11:03:53.427 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:53.411 7f57e4bb0700 1 Processor -- start 2021-02-10T11:03:53.427 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:53.411 7f57e4bb0700 1 -- start start 2021-02-10T11:03:53.427 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:53.411 7f57e4bb0700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f57e00e5470 0x7f57e007e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:53.427 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:53.411 7f57e4bb0700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f57e007f300 0x7f57e007f700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:53.427 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:53.411 7f57e4bb0700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57e007ed80 con 0x7f57e00e5070 2021-02-10T11:03:53.427 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:53.411 7f57e4bb0700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57e01b4010 con 0x7f57e00e5470 2021-02-10T11:03:53.428 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:53.411 7f57e4bb0700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57e00d1790 con 0x7f57e007f300 2021-02-10T11:03:53.428 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:53.411 7f57de05d700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f57e00e5470 0x7f57e007e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:53.428 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:53.411 7f57de05d700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f57e00e5470 0x7f57e007e6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:49346/0 (socket says 172.21.2.102:49346) 2021-02-10T11:03:53.428 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:53.411 7f57de05d700 1 -- 172.21.2.102:0/2618336562 learned_addr learned my addr 172.21.2.102:0/2618336562 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:03:53.428 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:53.412 7f57de05d700 1 -- 172.21.2.102:0/2618336562 >> v2:172.21.2.113:3300/0 conn(0x7f57e007f300 msgr2=0x7f57e007f700 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:03:53.428 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:53.412 7f57de05d700 1 --2- 172.21.2.102:0/2618336562 >> v2:172.21.2.113:3300/0 conn(0x7f57e007f300 0x7f57e007f700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:03:53.428 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:53.412 7f57de05d700 1 -- 172.21.2.102:0/2618336562 >> v1:172.21.2.102:6789/0 conn(0x7f57e00e5070 legacy=0x7f57e01b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:53.428 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:53.412 7f57de05d700 1 -- 172.21.2.102:0/2618336562 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57e00d19b0 con 0x7f57e00e5470 2021-02-10T11:03:53.428 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:53.412 7f57de05d700 1 --2- 172.21.2.102:0/2618336562 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f57e00e5470 0x7f57e007e6a0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7f57d0006700 tx=0x7f57d0009b00).ready entity=mon.0 client_cookie=5900f15835c76efa server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:03:53.428 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:53.412 7f57dd85c700 1 -- 172.21.2.102:0/2618336562 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f57d000c0c0 con 0x7f57e00e5470 2021-02-10T11:03:53.428 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:53.412 7f57dd85c700 1 -- 172.21.2.102:0/2618336562 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57d000c400 con 0x7f57e00e5470 2021-02-10T11:03:53.428 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:53.412 7f57dd85c700 1 -- 172.21.2.102:0/2618336562 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f57d000c920 con 0x7f57e00e5470 2021-02-10T11:03:53.429 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:53.412 7f57e4bb0700 1 -- 172.21.2.102:0/2618336562 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f57e00e5470 msgr2=0x7f57e007e6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:53.429 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:53.412 7f57e4bb0700 1 --2- 172.21.2.102:0/2618336562 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f57e00e5470 0x7f57e007e6a0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7f57d0006700 tx=0x7f57d0009b00).stop 2021-02-10T11:03:53.429 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:53.413 7f57e4bb0700 1 -- 172.21.2.102:0/2618336562 shutdown_connections 2021-02-10T11:03:53.429 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:53.413 7f57e4bb0700 1 --2- 172.21.2.102:0/2618336562 >> v2:172.21.2.113:3300/0 conn(0x7f57e007f300 0x7f57e007f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:03:53.429 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:53.413 7f57e4bb0700 1 --2- 172.21.2.102:0/2618336562 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f57e00e5470 0x7f57e007e6a0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:53.429 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:53.413 7f57e4bb0700 1 -- 172.21.2.102:0/2618336562 >> 172.21.2.102:0/2618336562 conn(0x7f57e00ce690 msgr2=0x7f57e00e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:03:53.429 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:53.413 7f57e4bb0700 1 -- 172.21.2.102:0/2618336562 shutdown_connections 2021-02-10T11:03:53.429 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:53.413 7f57e4bb0700 1 -- 172.21.2.102:0/2618336562 wait complete. 2021-02-10T11:03:53.430 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:53.428 7f57e4bb0700 1 Processor -- start 2021-02-10T11:03:53.430 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:53.428 7f57e4bb0700 1 -- start start 2021-02-10T11:03:53.430 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:53.428 7f57e4bb0700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f57e00a2b70 0x7f57e00c0590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:53.430 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:53.428 7f57e4bb0700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f57e00c0bb0 0x7f57e00a3850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:53.430 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:53.428 7f57e4bb0700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f57e00c0fb0 0x7f57e00a3e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:53.430 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:53.428 7f57e4bb0700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57e00a2f70 con 0x7f57e00c0bb0 2021-02-10T11:03:53.430 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:53.428 7f57e4bb0700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57e00a1a40 con 0x7f57e00c0fb0 2021-02-10T11:03:53.430 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:53.428 7f57e4bb0700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57e00a1c60 con 0x7f57e00a2b70 2021-02-10T11:03:53.430 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:53.428 7f57de05d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f57e00c0bb0 0x7f57e00a3850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:53.431 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:53.428 7f57de05d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f57e00c0bb0 0x7f57e00a3850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:49352/0 (socket says 172.21.2.102:49352) 2021-02-10T11:03:53.431 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:53.428 7f57de05d700 1 -- 172.21.2.102:0/2648678522 learned_addr learned my addr 172.21.2.102:0/2648678522 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:03:53.431 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:53.429 7f57de05d700 1 -- 172.21.2.102:0/2648678522 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f57e00a2b70 msgr2=0x7f57e00c0590 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:03:53.431 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:53.429 7f57de05d700 1 --2- 172.21.2.102:0/2648678522 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f57e00a2b70 0x7f57e00c0590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:03:53.431 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:53.429 7f57de05d700 1 -- 172.21.2.102:0/2648678522 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f57e00c0fb0 msgr2=0x7f57e00a3e70 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:03:53.431 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:53.429 7f57de05d700 1 --2- 172.21.2.102:0/2648678522 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f57e00c0fb0 0x7f57e00a3e70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:03:53.432 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:53.429 7f57de05d700 1 -- 172.21.2.102:0/2648678522 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57e00a4490 con 0x7f57e00c0bb0 2021-02-10T11:03:53.432 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:53.429 7f57de05d700 1 --2- 172.21.2.102:0/2648678522 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f57e00c0bb0 0x7f57e00a3850 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f57d000b660 tx=0x7f57d000ba00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:03:53.432 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:53.429 7f57d77fe700 1 -- 172.21.2.102:0/2648678522 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f57d0009440 con 0x7f57e00c0bb0 2021-02-10T11:03:53.432 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:53.429 7f57d77fe700 1 -- 172.21.2.102:0/2648678522 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57d0006390 con 0x7f57e00c0bb0 2021-02-10T11:03:53.432 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:53.429 7f57e4bb0700 1 -- 172.21.2.102:0/2648678522 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57e00a4490 con 0x7f57e00c0bb0 2021-02-10T11:03:53.432 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:53.429 7f57e4bb0700 1 -- 172.21.2.102:0/2648678522 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57e00a4490 con 0x7f57e00c0bb0 2021-02-10T11:03:53.433 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:53.429 7f57d77fe700 1 -- 172.21.2.102:0/2648678522 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f57d0002030 con 0x7f57e00c0bb0 2021-02-10T11:03:53.433 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:53.430 7f57d77fe700 1 -- 172.21.2.102:0/2648678522 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f57d00093e0 con 0x7f57e00c0bb0 2021-02-10T11:03:53.433 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:53.430 7f57d77fe700 1 --2- 172.21.2.102:0/2648678522 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f57c4018730 0x7f57c401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:53.433 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:53.430 7f57d77fe700 1 -- 172.21.2.102:0/2648678522 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f57d0032810 con 0x7f57e00c0bb0 2021-02-10T11:03:53.433 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:53.430 7f57de85e700 1 --2- 172.21.2.102:0/2648678522 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f57c4018730 0x7f57c401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:53.433 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:53.430 7f57de85e700 1 --2- 172.21.2.102:0/2648678522 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f57c4018730 0x7f57c401ac10 crc :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:03:53.460 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:53.461 7f57e4bb0700 1 --2- 172.21.2.102:0/2648678522 >> [v2:172.21.2.102:6810/36089,v1:172.21.2.102:6811/36089] conn(0x7f57e00c45a0 0x7f57e00627c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:53.460 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:53.461 7f57e4bb0700 1 -- 172.21.2.102:0/2648678522 --> [v2:172.21.2.102:6810/36089,v1:172.21.2.102:6811/36089] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f57e00a2f70 con 0x7f57e00c45a0 2021-02-10T11:03:53.461 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:53.461 7f57df05f700 1 --2- 172.21.2.102:0/2648678522 >> [v2:172.21.2.102:6810/36089,v1:172.21.2.102:6811/36089] conn(0x7f57e00c45a0 0x7f57e00627c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:53.461 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:53.462 7f57df05f700 1 --2- 172.21.2.102:0/2648678522 >> [v2:172.21.2.102:6810/36089,v1:172.21.2.102:6811/36089] conn(0x7f57e00c45a0 0x7f57e00627c0 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:03:53.461 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:53.462 7f57d77fe700 1 -- 172.21.2.102:0/2648678522 <== osd.2 v2:172.21.2.102:6810/36089 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+5151 (crc 0 0 0) 0x7f57d8006d10 con 0x7f57e00c45a0 2021-02-10T11:03:53.472 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:53.469 7f57e4bb0700 1 -- 172.21.2.102:0/2648678522 --> [v2:172.21.2.102:6810/36089,v1:172.21.2.102:6811/36089] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f57e0083da0 con 0x7f57e00c45a0 2021-02-10T11:03:53.472 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:53.469 7f57d77fe700 1 -- 172.21.2.102:0/2648678522 <== osd.2 v2:172.21.2.102:6810/36089 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f57d80058c0 con 0x7f57e00c45a0 2021-02-10T11:03:53.472 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:53.470 7f57e4bb0700 1 -- 172.21.2.102:0/2648678522 >> [v2:172.21.2.102:6810/36089,v1:172.21.2.102:6811/36089] conn(0x7f57e00c45a0 msgr2=0x7f57e00627c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:53.473 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:53.471 7f57e4bb0700 1 --2- 172.21.2.102:0/2648678522 >> [v2:172.21.2.102:6810/36089,v1:172.21.2.102:6811/36089] conn(0x7f57e00c45a0 0x7f57e00627c0 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:53.473 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:53.471 7f57e4bb0700 1 -- 172.21.2.102:0/2648678522 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f57c4018730 msgr2=0x7f57c401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:53.473 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:53.471 7f57e4bb0700 1 --2- 172.21.2.102:0/2648678522 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f57c4018730 0x7f57c401ac10 crc :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:53.473 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:53.471 7f57e4bb0700 1 -- 172.21.2.102:0/2648678522 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f57e00c0bb0 msgr2=0x7f57e00a3850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:53.473 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:53.471 7f57e4bb0700 1 --2- 172.21.2.102:0/2648678522 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f57e00c0bb0 0x7f57e00a3850 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f57d000b660 tx=0x7f57d000ba00).stop 2021-02-10T11:03:53.474 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:53.471 7f57df05f700 1 -- 172.21.2.102:0/2648678522 reap_dead start 2021-02-10T11:03:53.474 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:53.471 7f57e4bb0700 1 -- 172.21.2.102:0/2648678522 shutdown_connections 2021-02-10T11:03:53.474 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:53.471 7f57e4bb0700 1 -- 172.21.2.102:0/2648678522 >> 172.21.2.102:0/2648678522 conn(0x7f57e00bde70 msgr2=0x7f57e00c3970 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:03:53.474 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:53.471 7f57e4bb0700 1 -- 172.21.2.102:0/2648678522 shutdown_connections 2021-02-10T11:03:53.474 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:53.471 7f57e4bb0700 1 -- 172.21.2.102:0/2648678522 wait complete. 2021-02-10T11:03:53.475 INFO:teuthology.orchestra.run.gibba002.stdout:55834574854 2021-02-10T11:03:53.478 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell osd.3 flush_pg_stats 2021-02-10T11:03:53.913 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:53.897 7fb735dbe700 1 Processor -- start 2021-02-10T11:03:53.913 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:53.897 7fb735dbe700 1 -- start start 2021-02-10T11:03:53.913 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:53.898 7fb735dbe700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fb7300e5070 0x7fb7301b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:53.913 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:53.898 7fb735dbe700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fb7301b3e40 0x7fb73007d690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:53.913 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:53.898 7fb735dbe700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb73007ed80 con 0x7fb7300e5470 2021-02-10T11:03:53.913 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:53.898 7fb735dbe700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb730080310 con 0x7fb7300e5070 2021-02-10T11:03:53.913 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:53.898 7fb735dbe700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb730080530 con 0x7fb7301b3e40 2021-02-10T11:03:53.913 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:53.898 7fb72f7fe700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fb7300e5070 0x7fb7301b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:53.914 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:53.898 7fb72f7fe700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fb7300e5070 0x7fb7301b3820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:49360/0 (socket says 172.21.2.102:49360) 2021-02-10T11:03:53.914 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:53.898 7fb72f7fe700 1 -- 172.21.2.102:0/667150297 learned_addr learned my addr 172.21.2.102:0/667150297 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:03:53.914 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:53.898 7fb72f7fe700 1 -- 172.21.2.102:0/667150297 >> v2:172.21.2.113:3300/0 conn(0x7fb7301b3e40 msgr2=0x7fb73007d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:03:53.914 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:53.898 7fb72f7fe700 1 --2- 172.21.2.102:0/667150297 >> v2:172.21.2.113:3300/0 conn(0x7fb7301b3e40 0x7fb73007d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:03:53.914 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:53.898 7fb72f7fe700 1 -- 172.21.2.102:0/667150297 >> v1:172.21.2.113:6789/0 conn(0x7fb7300e5470 legacy=0x7fb73007dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:03:53.915 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:53.898 7fb72f7fe700 1 -- 172.21.2.102:0/667150297 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb730080750 con 0x7fb7300e5070 2021-02-10T11:03:53.915 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:53.898 7fb72f7fe700 1 --2- 172.21.2.102:0/667150297 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb7300e5070 0x7fb7301b3820 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7fb720006700 tx=0x7fb720009b80).ready entity=mon.0 client_cookie=e140c7fcc313bdad server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:03:53.915 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:53.899 7fb72e7fc700 1 -- 172.21.2.102:0/667150297 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb72000c0c0 con 0x7fb7300e5070 2021-02-10T11:03:53.915 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:53.899 7fb72e7fc700 1 -- 172.21.2.102:0/667150297 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb72000c340 con 0x7fb7300e5070 2021-02-10T11:03:53.915 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:53.899 7fb72e7fc700 1 -- 172.21.2.102:0/667150297 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb72000c860 con 0x7fb7300e5070 2021-02-10T11:03:53.915 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:53.899 7fb735dbe700 1 -- 172.21.2.102:0/667150297 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb7300e5070 msgr2=0x7fb7301b3820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:53.915 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:53.899 7fb735dbe700 1 --2- 172.21.2.102:0/667150297 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb7300e5070 0x7fb7301b3820 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7fb720006700 tx=0x7fb720009b80).stop 2021-02-10T11:03:53.915 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:53.899 7fb735dbe700 1 -- 172.21.2.102:0/667150297 shutdown_connections 2021-02-10T11:03:53.916 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:53.899 7fb735dbe700 1 --2- 172.21.2.102:0/667150297 >> v2:172.21.2.113:3300/0 conn(0x7fb7301b3e40 0x7fb73007d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:03:53.916 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:53.899 7fb735dbe700 1 --2- 172.21.2.102:0/667150297 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb7300e5070 0x7fb7301b3820 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:53.916 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:53.899 7fb735dbe700 1 -- 172.21.2.102:0/667150297 >> 172.21.2.102:0/667150297 conn(0x7fb7300ce690 msgr2=0x7fb7300e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:03:53.916 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:53.900 7fb735dbe700 1 -- 172.21.2.102:0/667150297 shutdown_connections 2021-02-10T11:03:53.916 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:53.900 7fb735dbe700 1 -- 172.21.2.102:0/667150297 wait complete. 2021-02-10T11:03:53.918 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:53.914 7fb735dbe700 1 Processor -- start 2021-02-10T11:03:53.918 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:53.914 7fb735dbe700 1 -- start start 2021-02-10T11:03:53.918 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:53.915 7fb735dbe700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb7300be580 0x7fb7300be980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:53.918 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:53.915 7fb735dbe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb730097ed0 0x7fb7300982d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:53.918 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:53.915 7fb735dbe700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb730099180 0x7fb730099580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:53.918 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:53.915 7fb735dbe700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7300988f0 con 0x7fb730097ed0 2021-02-10T11:03:53.919 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:53.915 7fb735dbe700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7300c1110 con 0x7fb730099180 2021-02-10T11:03:53.919 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:53.915 7fb735dbe700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb730099ba0 con 0x7fb7300be580 2021-02-10T11:03:53.919 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:53.915 7fb72effd700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb730097ed0 0x7fb7300982d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:53.919 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:53.915 7fb72effd700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb730097ed0 0x7fb7300982d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:49366/0 (socket says 172.21.2.102:49366) 2021-02-10T11:03:53.919 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:53.915 7fb72effd700 1 -- 172.21.2.102:0/1205047324 learned_addr learned my addr 172.21.2.102:0/1205047324 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:03:53.919 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:53.915 7fb72f7fe700 1 --2- 172.21.2.102:0/1205047324 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb7300be580 0x7fb7300be980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:53.920 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:53.915 7fb72ffff700 1 --2- 172.21.2.102:0/1205047324 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb730099180 0x7fb730099580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:53.920 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:53.915 7fb72effd700 1 -- 172.21.2.102:0/1205047324 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb7300be580 msgr2=0x7fb7300be980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:53.920 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:53.915 7fb72effd700 1 --2- 172.21.2.102:0/1205047324 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb7300be580 0x7fb7300be980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:53.920 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:53.915 7fb72effd700 1 -- 172.21.2.102:0/1205047324 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb730099180 msgr2=0x7fb730099580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:53.920 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:53.915 7fb72effd700 1 --2- 172.21.2.102:0/1205047324 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb730099180 0x7fb730099580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:53.920 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:53.915 7fb72effd700 1 -- 172.21.2.102:0/1205047324 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb730099dc0 con 0x7fb730097ed0 2021-02-10T11:03:53.920 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:53.915 7fb72ffff700 1 --2- 172.21.2.102:0/1205047324 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb730099180 0x7fb730099580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:03:53.920 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:53.915 7fb72effd700 1 --2- 172.21.2.102:0/1205047324 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb730097ed0 0x7fb7300982d0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7fb71800b530 tx=0x7fb71800b8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:03:53.921 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:53.916 7fb727fff700 1 -- 172.21.2.102:0/1205047324 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb718006350 con 0x7fb730097ed0 2021-02-10T11:03:53.921 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:53.916 7fb727fff700 1 -- 172.21.2.102:0/1205047324 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb718006c20 con 0x7fb730097ed0 2021-02-10T11:03:53.921 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:53.916 7fb735dbe700 1 -- 172.21.2.102:0/1205047324 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb730099dc0 con 0x7fb730097ed0 2021-02-10T11:03:53.921 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:53.916 7fb735dbe700 1 -- 172.21.2.102:0/1205047324 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb730099dc0 con 0x7fb730097ed0 2021-02-10T11:03:53.921 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:53.916 7fb727fff700 1 -- 172.21.2.102:0/1205047324 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb718000c90 con 0x7fb730097ed0 2021-02-10T11:03:53.921 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:53.916 7fb727fff700 1 -- 172.21.2.102:0/1205047324 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fb718007000 con 0x7fb730097ed0 2021-02-10T11:03:53.921 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:53.916 7fb727fff700 1 --2- 172.21.2.102:0/1205047324 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb7140186c0 0x7fb71401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:53.921 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:53.916 7fb727fff700 1 -- 172.21.2.102:0/1205047324 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fb7180327a0 con 0x7fb730097ed0 2021-02-10T11:03:53.921 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:53.916 7fb72f7fe700 1 --2- 172.21.2.102:0/1205047324 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb7140186c0 0x7fb71401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:53.921 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:53.917 7fb72f7fe700 1 --2- 172.21.2.102:0/1205047324 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb7140186c0 0x7fb71401abc0 crc :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:03:53.947 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:53.948 7fb735dbe700 1 --2- 172.21.2.102:0/1205047324 >> [v2:172.21.2.102:6802/36090,v1:172.21.2.102:6803/36090] conn(0x7fb7300c45a0 0x7fb7300627c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:53.947 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:53.948 7fb735dbe700 1 -- 172.21.2.102:0/1205047324 --> [v2:172.21.2.102:6802/36090,v1:172.21.2.102:6803/36090] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb730098c50 con 0x7fb7300c45a0 2021-02-10T11:03:53.947 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:53.948 7fb72ffff700 1 --2- 172.21.2.102:0/1205047324 >> [v2:172.21.2.102:6802/36090,v1:172.21.2.102:6803/36090] conn(0x7fb7300c45a0 0x7fb7300627c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:53.948 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:53.948 7fb72ffff700 1 --2- 172.21.2.102:0/1205047324 >> [v2:172.21.2.102:6802/36090,v1:172.21.2.102:6803/36090] conn(0x7fb7300c45a0 0x7fb7300627c0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:03:53.949 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:53.949 7fb727fff700 1 -- 172.21.2.102:0/1205047324 <== osd.3 v2:172.21.2.102:6802/36090 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+5151 (crc 0 0 0) 0x7fb728006c20 con 0x7fb7300c45a0 2021-02-10T11:03:53.956 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:53.956 7fb735dbe700 1 -- 172.21.2.102:0/1205047324 --> [v2:172.21.2.102:6802/36090,v1:172.21.2.102:6803/36090] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fb7300988f0 con 0x7fb7300c45a0 2021-02-10T11:03:53.956 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:53.956 7fb727fff700 1 -- 172.21.2.102:0/1205047324 <== osd.3 v2:172.21.2.102:6802/36090 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fb728006c20 con 0x7fb7300c45a0 2021-02-10T11:03:53.957 INFO:teuthology.orchestra.run.gibba002.stdout:55834574855 2021-02-10T11:03:53.957 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:53.957 7fb735dbe700 1 -- 172.21.2.102:0/1205047324 >> [v2:172.21.2.102:6802/36090,v1:172.21.2.102:6803/36090] conn(0x7fb7300c45a0 msgr2=0x7fb7300627c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:53.957 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:53.957 7fb735dbe700 1 --2- 172.21.2.102:0/1205047324 >> [v2:172.21.2.102:6802/36090,v1:172.21.2.102:6803/36090] conn(0x7fb7300c45a0 0x7fb7300627c0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:53.958 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:53.957 7fb735dbe700 1 -- 172.21.2.102:0/1205047324 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb7140186c0 msgr2=0x7fb71401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:53.958 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:53.957 7fb735dbe700 1 --2- 172.21.2.102:0/1205047324 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb7140186c0 0x7fb71401abc0 crc :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:53.958 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:53.957 7fb735dbe700 1 -- 172.21.2.102:0/1205047324 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb730097ed0 msgr2=0x7fb7300982d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:53.958 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:53.957 7fb735dbe700 1 --2- 172.21.2.102:0/1205047324 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb730097ed0 0x7fb7300982d0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7fb71800b530 tx=0x7fb71800b8d0).stop 2021-02-10T11:03:53.958 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:53.957 7fb72ffff700 1 -- 172.21.2.102:0/1205047324 reap_dead start 2021-02-10T11:03:53.958 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:53.957 7fb735dbe700 1 -- 172.21.2.102:0/1205047324 shutdown_connections 2021-02-10T11:03:53.958 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:53.957 7fb735dbe700 1 -- 172.21.2.102:0/1205047324 >> 172.21.2.102:0/1205047324 conn(0x7fb7300bde70 msgr2=0x7fb7300c3970 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:03:53.959 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:53.958 7fb735dbe700 1 -- 172.21.2.102:0/1205047324 shutdown_connections 2021-02-10T11:03:53.959 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:53.958 7fb735dbe700 1 -- 172.21.2.102:0/1205047324 wait complete. 2021-02-10T11:03:53.966 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell osd.4 flush_pg_stats 2021-02-10T11:03:54.394 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:54.378 7f900dacd700 1 Processor -- start 2021-02-10T11:03:54.394 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:54.378 7f900dacd700 1 -- start start 2021-02-10T11:03:54.394 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:54.379 7f900dacd700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f90080e5070 0x7f90081b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:54.394 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:54.379 7f900dacd700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f90081b3e40 0x7f900807d690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:54.394 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:54.379 7f900dacd700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f90080e5470 0x7f900807dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:54.394 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:54.379 7f900dacd700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f900807ed80 con 0x7f90081b3e40 2021-02-10T11:03:54.394 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:54.379 7f900dacd700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f900807e2d0 con 0x7f90080e5070 2021-02-10T11:03:54.394 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:54.379 7f900dacd700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f900807e4f0 con 0x7f90080e5470 2021-02-10T11:03:54.395 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:54.379 7f9006ffd700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f90081b3e40 0x7f900807d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:54.395 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:54.379 7f9006ffd700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f90081b3e40 0x7f900807d690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:49374/0 (socket says 172.21.2.102:49374) 2021-02-10T11:03:54.395 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:54.379 7f9006ffd700 1 -- 172.21.2.102:0/1510247744 learned_addr learned my addr 172.21.2.102:0/1510247744 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:03:54.395 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:54.379 7f9007fff700 1 --2- 172.21.2.102:0/1510247744 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f90080e5470 0x7f900807dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:54.395 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:54.379 7f90077fe700 1 --2- 172.21.2.102:0/1510247744 >> v2:172.21.2.113:3300/0 conn(0x7f90080e5070 0x7f90081b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:54.395 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:54.379 7f9006ffd700 1 -- 172.21.2.102:0/1510247744 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f90080e5470 msgr2=0x7f900807dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:03:54.395 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:54.379 7f9006ffd700 1 --2- 172.21.2.102:0/1510247744 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f90080e5470 0x7f900807dcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:54.395 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:54.379 7f9006ffd700 1 -- 172.21.2.102:0/1510247744 >> v2:172.21.2.113:3300/0 conn(0x7f90080e5070 msgr2=0x7f90081b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:03:54.395 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:54.379 7f9006ffd700 1 --2- 172.21.2.102:0/1510247744 >> v2:172.21.2.113:3300/0 conn(0x7f90080e5070 0x7f90081b3820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:54.395 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:54.379 7f9006ffd700 1 -- 172.21.2.102:0/1510247744 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f900807e710 con 0x7f90081b3e40 2021-02-10T11:03:54.395 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:54.380 7f9006ffd700 1 --2- 172.21.2.102:0/1510247744 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f90081b3e40 0x7f900807d690 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7f8ff8006700 tx=0x7f8ff8009b80).ready entity=mon.0 client_cookie=3af2d8eccf34bb46 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:03:54.395 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:54.380 7f90067fc700 1 -- 172.21.2.102:0/1510247744 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8ff800c0c0 con 0x7f90081b3e40 2021-02-10T11:03:54.396 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:54.380 7f90067fc700 1 -- 172.21.2.102:0/1510247744 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ff800c440 con 0x7f90081b3e40 2021-02-10T11:03:54.396 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:54.380 7f90067fc700 1 -- 172.21.2.102:0/1510247744 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8ff800c960 con 0x7f90081b3e40 2021-02-10T11:03:54.396 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:54.380 7f900dacd700 1 -- 172.21.2.102:0/1510247744 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f90081b3e40 msgr2=0x7f900807d690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:54.396 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:54.380 7f900dacd700 1 --2- 172.21.2.102:0/1510247744 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f90081b3e40 0x7f900807d690 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7f8ff8006700 tx=0x7f8ff8009b80).stop 2021-02-10T11:03:54.396 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:54.380 7f900dacd700 1 -- 172.21.2.102:0/1510247744 shutdown_connections 2021-02-10T11:03:54.396 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:54.380 7f900dacd700 1 --2- 172.21.2.102:0/1510247744 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f90080e5470 0x7f900807dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:54.396 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:54.380 7f900dacd700 1 --2- 172.21.2.102:0/1510247744 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f90081b3e40 0x7f900807d690 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:54.396 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:54.380 7f900dacd700 1 --2- 172.21.2.102:0/1510247744 >> v2:172.21.2.113:3300/0 conn(0x7f90080e5070 0x7f90081b3820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:54.396 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:54.380 7f900dacd700 1 -- 172.21.2.102:0/1510247744 >> 172.21.2.102:0/1510247744 conn(0x7f90080ce690 msgr2=0x7f90080e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:03:54.396 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:54.380 7f900dacd700 1 -- 172.21.2.102:0/1510247744 shutdown_connections 2021-02-10T11:03:54.396 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:54.380 7f900dacd700 1 -- 172.21.2.102:0/1510247744 wait complete. 2021-02-10T11:03:54.396 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:54.395 7f900dacd700 1 Processor -- start 2021-02-10T11:03:54.398 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:54.395 7f900dacd700 1 -- start start 2021-02-10T11:03:54.398 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:54.395 7f900dacd700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f90080b4ad0 0x7f90080a78d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:54.398 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:54.395 7f900dacd700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f90080a7ef0 0x7f90081a6d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:54.398 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:54.395 7f900dacd700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f90080a82f0 0x7f90081a73b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:54.398 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:54.395 7f900dacd700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90080aa3c0 con 0x7f90080a82f0 2021-02-10T11:03:54.398 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:54.395 7f900dacd700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90080a88c0 con 0x7f90080a7ef0 2021-02-10T11:03:54.398 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:54.395 7f900dacd700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90080b4ed0 con 0x7f90080b4ad0 2021-02-10T11:03:54.398 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:54.396 7f9007fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f90080a82f0 0x7f90081a73b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:54.398 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:54.396 7f9007fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f90080a82f0 0x7f90081a73b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:49382/0 (socket says 172.21.2.102:49382) 2021-02-10T11:03:54.398 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:54.396 7f9007fff700 1 -- 172.21.2.102:0/1477965715 learned_addr learned my addr 172.21.2.102:0/1477965715 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:03:54.399 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:54.396 7f9006ffd700 1 --2- 172.21.2.102:0/1477965715 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f90080a7ef0 0x7f90081a6d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:54.399 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:54.396 7f9007fff700 1 -- 172.21.2.102:0/1477965715 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f90080b4ad0 msgr2=0x7f90080a78d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:54.399 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:54.396 7f9007fff700 1 --2- 172.21.2.102:0/1477965715 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f90080b4ad0 0x7f90080a78d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:03:54.399 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:54.396 7f9007fff700 1 -- 172.21.2.102:0/1477965715 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f90080a7ef0 msgr2=0x7f90081a6d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:54.399 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:54.396 7f9007fff700 1 --2- 172.21.2.102:0/1477965715 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f90080a7ef0 0x7f90081a6d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:54.399 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:54.396 7f9007fff700 1 -- 172.21.2.102:0/1477965715 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90081a79d0 con 0x7f90080a82f0 2021-02-10T11:03:54.399 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:54.396 7f9007fff700 1 --2- 172.21.2.102:0/1477965715 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f90080a82f0 0x7f90081a73b0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f900000a940 tx=0x7f900000ace0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:03:54.399 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:54.396 7f8ffffff700 1 -- 172.21.2.102:0/1477965715 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f90000066f0 con 0x7f90080a82f0 2021-02-10T11:03:54.399 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:54.396 7f8ffffff700 1 -- 172.21.2.102:0/1477965715 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f900000ae20 con 0x7f90080a82f0 2021-02-10T11:03:54.399 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:54.397 7f900dacd700 1 -- 172.21.2.102:0/1477965715 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90081a79d0 con 0x7f90080a82f0 2021-02-10T11:03:54.400 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:54.397 7f8ffffff700 1 -- 172.21.2.102:0/1477965715 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f90000015f0 con 0x7f90080a82f0 2021-02-10T11:03:54.400 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:54.397 7f900dacd700 1 -- 172.21.2.102:0/1477965715 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90080a9130 con 0x7f90080a82f0 2021-02-10T11:03:54.400 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:54.397 7f8ffffff700 1 -- 172.21.2.102:0/1477965715 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f900000b030 con 0x7f90080a82f0 2021-02-10T11:03:54.400 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:54.397 7f8ffffff700 1 --2- 172.21.2.102:0/1477965715 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8fec0186c0 0x7f8fec01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:54.400 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:54.397 7f8ffffff700 1 -- 172.21.2.102:0/1477965715 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f90000328c0 con 0x7f90080a82f0 2021-02-10T11:03:54.400 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:54.397 7f90077fe700 1 --2- 172.21.2.102:0/1477965715 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8fec0186c0 0x7f8fec01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:54.400 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:54.398 7f90077fe700 1 --2- 172.21.2.102:0/1477965715 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8fec0186c0 0x7f8fec01abc0 crc :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:03:54.427 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:54.428 7f900dacd700 1 --2- 172.21.2.102:0/1477965715 >> [v2:172.21.2.113:6808/36368,v1:172.21.2.113:6811/36368] conn(0x7f90080c45a0 0x7f90080a99c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:54.427 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:54.428 7f900dacd700 1 -- 172.21.2.102:0/1477965715 --> [v2:172.21.2.113:6808/36368,v1:172.21.2.113:6811/36368] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f90080a9130 con 0x7f90080c45a0 2021-02-10T11:03:54.428 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:54.429 7f9006ffd700 1 --2- 172.21.2.102:0/1477965715 >> [v2:172.21.2.113:6808/36368,v1:172.21.2.113:6811/36368] conn(0x7f90080c45a0 0x7f90080a99c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:54.428 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:54.429 7f9006ffd700 1 --2- 172.21.2.102:0/1477965715 >> [v2:172.21.2.113:6808/36368,v1:172.21.2.113:6811/36368] conn(0x7f90080c45a0 0x7f90080a99c0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:03:54.428 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:54.430 7f8ffffff700 1 -- 172.21.2.102:0/1477965715 <== osd.4 v2:172.21.2.113:6808/36368 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+5151 (crc 0 0 0) 0x7f8ff8000940 con 0x7f90080c45a0 2021-02-10T11:03:54.435 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:54.436 7f900dacd700 1 -- 172.21.2.102:0/1477965715 --> [v2:172.21.2.113:6808/36368,v1:172.21.2.113:6811/36368] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f9008084a90 con 0x7f90080c45a0 2021-02-10T11:03:54.436 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:54.437 7f8ffffff700 1 -- 172.21.2.102:0/1477965715 <== osd.4 v2:172.21.2.113:6808/36368 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f8ff8000940 con 0x7f90080c45a0 2021-02-10T11:03:54.436 INFO:teuthology.orchestra.run.gibba002.stdout:55834574856 2021-02-10T11:03:54.437 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:54.438 7f900dacd700 1 -- 172.21.2.102:0/1477965715 >> [v2:172.21.2.113:6808/36368,v1:172.21.2.113:6811/36368] conn(0x7f90080c45a0 msgr2=0x7f90080a99c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:54.438 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:54.438 7f900dacd700 1 --2- 172.21.2.102:0/1477965715 >> [v2:172.21.2.113:6808/36368,v1:172.21.2.113:6811/36368] conn(0x7f90080c45a0 0x7f90080a99c0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:54.438 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:54.438 7f900dacd700 1 -- 172.21.2.102:0/1477965715 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8fec0186c0 msgr2=0x7f8fec01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:54.438 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:54.438 7f900dacd700 1 --2- 172.21.2.102:0/1477965715 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8fec0186c0 0x7f8fec01abc0 crc :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:54.438 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:54.438 7f900dacd700 1 -- 172.21.2.102:0/1477965715 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f90080a82f0 msgr2=0x7f90081a73b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:54.438 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:54.438 7f900dacd700 1 --2- 172.21.2.102:0/1477965715 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f90080a82f0 0x7f90081a73b0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f900000a940 tx=0x7f900000ace0).stop 2021-02-10T11:03:54.438 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:54.438 7f9007fff700 1 -- 172.21.2.102:0/1477965715 reap_dead start 2021-02-10T11:03:54.438 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:54.438 7f900dacd700 1 -- 172.21.2.102:0/1477965715 shutdown_connections 2021-02-10T11:03:54.438 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:54.438 7f900dacd700 1 -- 172.21.2.102:0/1477965715 >> 172.21.2.102:0/1477965715 conn(0x7f90080b5560 msgr2=0x7f90080a6200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:03:54.438 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:54.438 7f900dacd700 1 -- 172.21.2.102:0/1477965715 shutdown_connections 2021-02-10T11:03:54.438 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:54.438 7f900dacd700 1 -- 172.21.2.102:0/1477965715 wait complete. 2021-02-10T11:03:54.446 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell osd.5 flush_pg_stats 2021-02-10T11:03:54.874 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:54.858 7f6d956a5700 1 Processor -- start 2021-02-10T11:03:54.875 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:54.858 7f6d956a5700 1 -- start start 2021-02-10T11:03:54.875 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:54.859 7f6d956a5700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6d900e5470 0x7f6d9007e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:54.875 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:54.859 7f6d956a5700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f6d9007f300 0x7f6d9007f700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:54.875 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:54.859 7f6d956a5700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d9007ed80 con 0x7f6d900e5070 2021-02-10T11:03:54.875 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:54.859 7f6d956a5700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d901b4010 con 0x7f6d9007f300 2021-02-10T11:03:54.875 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:54.859 7f6d956a5700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d900d1790 con 0x7f6d900e5470 2021-02-10T11:03:54.875 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:54.860 7f6d8eabe700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6d900e5470 0x7f6d9007e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:54.875 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:54.860 7f6d8f2bf700 1 --1- >> v1:172.21.2.113:6789/0 conn(0x7f6d900e5070 0x7f6d901b3820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.113:6789/0 says I am v1:172.21.2.102:48936/0 (socket says 172.21.2.102:48936) 2021-02-10T11:03:54.875 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:54.860 7f6d8f2bf700 1 -- 172.21.2.102:0/3819609650 learned_addr learned my addr 172.21.2.102:0/3819609650 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:03:54.875 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:54.860 7f6d8fac0700 1 --2- 172.21.2.102:0/3819609650 >> v2:172.21.2.113:3300/0 conn(0x7f6d9007f300 0x7f6d9007f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:54.875 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:54.860 7f6d8eabe700 1 -- 172.21.2.102:0/3819609650 >> v2:172.21.2.113:3300/0 conn(0x7f6d9007f300 msgr2=0x7f6d9007f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:03:54.875 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:54.860 7f6d8eabe700 1 --2- 172.21.2.102:0/3819609650 >> v2:172.21.2.113:3300/0 conn(0x7f6d9007f300 0x7f6d9007f700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:54.875 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:54.860 7f6d8eabe700 1 -- 172.21.2.102:0/3819609650 >> v1:172.21.2.113:6789/0 conn(0x7f6d900e5070 legacy=0x7f6d901b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:54.876 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:54.860 7f6d8eabe700 1 -- 172.21.2.102:0/3819609650 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d900d1980 con 0x7f6d900e5470 2021-02-10T11:03:54.876 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:54.860 7f6d8eabe700 1 --2- 172.21.2.102:0/3819609650 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6d900e5470 0x7f6d9007e6a0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f6d84006650 tx=0x7f6d84009a80).ready entity=mon.2 client_cookie=dfc4c48adf776733 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:03:54.876 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:54.860 7f6d8e2bd700 1 -- 172.21.2.102:0/3819609650 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6d8400c0c0 con 0x7f6d900e5470 2021-02-10T11:03:54.876 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:54.860 7f6d8e2bd700 1 -- 172.21.2.102:0/3819609650 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d8400c340 con 0x7f6d900e5470 2021-02-10T11:03:54.876 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:54.860 7f6d8e2bd700 1 -- 172.21.2.102:0/3819609650 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6d8400c860 con 0x7f6d900e5470 2021-02-10T11:03:54.876 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:54.860 7f6d956a5700 1 -- 172.21.2.102:0/3819609650 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6d900e5470 msgr2=0x7f6d9007e6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:54.876 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:54.860 7f6d956a5700 1 --2- 172.21.2.102:0/3819609650 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6d900e5470 0x7f6d9007e6a0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f6d84006650 tx=0x7f6d84009a80).stop 2021-02-10T11:03:54.876 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:54.861 7f6d956a5700 1 -- 172.21.2.102:0/3819609650 shutdown_connections 2021-02-10T11:03:54.876 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:54.861 7f6d956a5700 1 --2- 172.21.2.102:0/3819609650 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6d900e5470 0x7f6d9007e6a0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:54.876 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:54.861 7f6d956a5700 1 --2- 172.21.2.102:0/3819609650 >> v2:172.21.2.113:3300/0 conn(0x7f6d9007f300 0x7f6d9007f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:54.876 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:54.861 7f6d956a5700 1 -- 172.21.2.102:0/3819609650 >> 172.21.2.102:0/3819609650 conn(0x7f6d900ce690 msgr2=0x7f6d900e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:03:54.876 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:54.861 7f6d956a5700 1 -- 172.21.2.102:0/3819609650 shutdown_connections 2021-02-10T11:03:54.876 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:54.861 7f6d956a5700 1 -- 172.21.2.102:0/3819609650 wait complete. 2021-02-10T11:03:54.877 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:54.875 7f6d956a5700 1 Processor -- start 2021-02-10T11:03:54.877 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:54.875 7f6d956a5700 1 -- start start 2021-02-10T11:03:54.878 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:54.876 7f6d956a5700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6d90096cc0 0x7f6d901073b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:54.878 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:54.876 7f6d956a5700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6d900a0780 0x7f6d901079d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:54.878 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:54.876 7f6d956a5700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6d9008f590 0x7f6d9008f990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:54.878 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:54.876 7f6d956a5700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d90090b10 con 0x7f6d900a0780 2021-02-10T11:03:54.878 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:54.876 7f6d956a5700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d9008f010 con 0x7f6d9008f590 2021-02-10T11:03:54.878 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:54.876 7f6d956a5700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d9019a070 con 0x7f6d90096cc0 2021-02-10T11:03:54.878 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:54.876 7f6d8eabe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6d900a0780 0x7f6d901079d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:54.878 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:54.876 7f6d8eabe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6d900a0780 0x7f6d901079d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:49394/0 (socket says 172.21.2.102:49394) 2021-02-10T11:03:54.878 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:54.876 7f6d8eabe700 1 -- 172.21.2.102:0/4205354611 learned_addr learned my addr 172.21.2.102:0/4205354611 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:03:54.878 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:54.876 7f6d8fac0700 1 --2- 172.21.2.102:0/4205354611 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6d9008f590 0x7f6d9008f990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:54.879 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:54.876 7f6d8f2bf700 1 --2- 172.21.2.102:0/4205354611 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6d90096cc0 0x7f6d901073b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:54.879 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:54.877 7f6d8eabe700 1 -- 172.21.2.102:0/4205354611 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6d90096cc0 msgr2=0x7f6d901073b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:54.879 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:54.877 7f6d8eabe700 1 --2- 172.21.2.102:0/4205354611 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6d90096cc0 0x7f6d901073b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:54.879 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:54.877 7f6d8eabe700 1 -- 172.21.2.102:0/4205354611 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6d9008f590 msgr2=0x7f6d9008f990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:54.879 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:54.877 7f6d8eabe700 1 --2- 172.21.2.102:0/4205354611 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6d9008f590 0x7f6d9008f990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:54.879 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:54.877 7f6d8eabe700 1 -- 172.21.2.102:0/4205354611 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d90107ff0 con 0x7f6d900a0780 2021-02-10T11:03:54.879 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:54.877 7f6d8f2bf700 1 --2- 172.21.2.102:0/4205354611 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6d90096cc0 0x7f6d901073b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:03:54.879 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:54.877 7f6d8eabe700 1 --2- 172.21.2.102:0/4205354611 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6d900a0780 0x7f6d901079d0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7f6d84009630 tx=0x7f6d8400b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:03:54.879 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:54.877 7f6d77fff700 1 -- 172.21.2.102:0/4205354611 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6d840093a0 con 0x7f6d900a0780 2021-02-10T11:03:54.879 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:54.877 7f6d77fff700 1 -- 172.21.2.102:0/4205354611 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d8400bd70 con 0x7f6d900a0780 2021-02-10T11:03:54.879 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:54.877 7f6d956a5700 1 -- 172.21.2.102:0/4205354611 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d90107ff0 con 0x7f6d900a0780 2021-02-10T11:03:54.879 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:54.877 7f6d956a5700 1 -- 172.21.2.102:0/4205354611 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d900902a0 con 0x7f6d900a0780 2021-02-10T11:03:54.879 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:54.877 7f6d77fff700 1 -- 172.21.2.102:0/4205354611 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6d84005560 con 0x7f6d900a0780 2021-02-10T11:03:54.880 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:54.878 7f6d77fff700 1 -- 172.21.2.102:0/4205354611 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f6d8400a000 con 0x7f6d900a0780 2021-02-10T11:03:54.880 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:54.878 7f6d77fff700 1 --2- 172.21.2.102:0/4205354611 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6d70018730 0x7f6d7001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:54.880 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:54.878 7f6d77fff700 1 -- 172.21.2.102:0/4205354611 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f6d84032800 con 0x7f6d900a0780 2021-02-10T11:03:54.880 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:54.878 7f6d8f2bf700 1 --2- 172.21.2.102:0/4205354611 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6d70018730 0x7f6d7001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:54.880 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:54.878 7f6d8f2bf700 1 --2- 172.21.2.102:0/4205354611 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6d70018730 0x7f6d7001abc0 crc :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:03:54.907 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:54.908 7f6d956a5700 1 --2- 172.21.2.102:0/4205354611 >> [v2:172.21.2.113:6802/36366,v1:172.21.2.113:6805/36366] conn(0x7f6d900c45a0 0x7f6d90062cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:54.907 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:54.908 7f6d956a5700 1 -- 172.21.2.102:0/4205354611 --> [v2:172.21.2.113:6802/36366,v1:172.21.2.113:6805/36366] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6d900902a0 con 0x7f6d900c45a0 2021-02-10T11:03:54.908 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:54.909 7f6d8fac0700 1 --2- 172.21.2.102:0/4205354611 >> [v2:172.21.2.113:6802/36366,v1:172.21.2.113:6805/36366] conn(0x7f6d900c45a0 0x7f6d90062cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:54.908 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:54.910 7f6d8fac0700 1 --2- 172.21.2.102:0/4205354611 >> [v2:172.21.2.113:6802/36366,v1:172.21.2.113:6805/36366] conn(0x7f6d900c45a0 0x7f6d90062cd0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:03:54.909 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:54.910 7f6d77fff700 1 -- 172.21.2.102:0/4205354611 <== osd.5 v2:172.21.2.113:6802/36366 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+5151 (crc 0 0 0) 0x7f6d88006a10 con 0x7f6d900c45a0 2021-02-10T11:03:54.915 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:54.916 7f6d956a5700 1 -- 172.21.2.102:0/4205354611 --> [v2:172.21.2.113:6802/36366,v1:172.21.2.113:6805/36366] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f6d90107ff0 con 0x7f6d900c45a0 2021-02-10T11:03:54.916 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:54.917 7f6d77fff700 1 -- 172.21.2.102:0/4205354611 <== osd.5 v2:172.21.2.113:6802/36366 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f6d88006a10 con 0x7f6d900c45a0 2021-02-10T11:03:54.917 INFO:teuthology.orchestra.run.gibba002.stdout:55834574856 2021-02-10T11:03:54.917 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:54.918 7f6d956a5700 1 -- 172.21.2.102:0/4205354611 >> [v2:172.21.2.113:6802/36366,v1:172.21.2.113:6805/36366] conn(0x7f6d900c45a0 msgr2=0x7f6d90062cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:54.917 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:54.918 7f6d956a5700 1 --2- 172.21.2.102:0/4205354611 >> [v2:172.21.2.113:6802/36366,v1:172.21.2.113:6805/36366] conn(0x7f6d900c45a0 0x7f6d90062cd0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:54.918 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:54.918 7f6d956a5700 1 -- 172.21.2.102:0/4205354611 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6d70018730 msgr2=0x7f6d7001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:54.918 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:54.918 7f6d956a5700 1 --2- 172.21.2.102:0/4205354611 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6d70018730 0x7f6d7001abc0 crc :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:54.918 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:54.918 7f6d956a5700 1 -- 172.21.2.102:0/4205354611 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6d900a0780 msgr2=0x7f6d901079d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:54.918 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:54.918 7f6d956a5700 1 --2- 172.21.2.102:0/4205354611 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6d900a0780 0x7f6d901079d0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7f6d84009630 tx=0x7f6d8400b980).stop 2021-02-10T11:03:54.918 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:54.918 7f6d8fac0700 1 -- 172.21.2.102:0/4205354611 reap_dead start 2021-02-10T11:03:54.918 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:54.918 7f6d956a5700 1 -- 172.21.2.102:0/4205354611 shutdown_connections 2021-02-10T11:03:54.918 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:54.918 7f6d956a5700 1 -- 172.21.2.102:0/4205354611 >> 172.21.2.102:0/4205354611 conn(0x7f6d90097190 msgr2=0x7f6d90086310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:03:54.918 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:54.919 7f6d956a5700 1 -- 172.21.2.102:0/4205354611 shutdown_connections 2021-02-10T11:03:54.918 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:54.919 7f6d956a5700 1 -- 172.21.2.102:0/4205354611 wait complete. 2021-02-10T11:03:54.925 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell osd.6 flush_pg_stats 2021-02-10T11:03:55.351 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:55.336 7f60b899e700 1 Processor -- start 2021-02-10T11:03:55.352 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:55.336 7f60b899e700 1 -- start start 2021-02-10T11:03:55.352 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:55.337 7f60b899e700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f60b40e5070 0x7f60b41b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:55.352 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:55.337 7f60b899e700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f60b40e5470 0x7f60b407d690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:55.352 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:55.337 7f60b899e700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60b407ed80 con 0x7f60b41b3e40 2021-02-10T11:03:55.352 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:55.337 7f60b899e700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60b4080310 con 0x7f60b40e5470 2021-02-10T11:03:55.352 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:55.337 7f60b899e700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60b4080500 con 0x7f60b40e5070 2021-02-10T11:03:55.352 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:55.337 7f60b1e1b700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f60b40e5470 0x7f60b407d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:55.352 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:55.337 7f60b1e1b700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f60b40e5470 0x7f60b407d690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:49404/0 (socket says 172.21.2.102:49404) 2021-02-10T11:03:55.352 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:55.337 7f60b1e1b700 1 -- 172.21.2.102:0/2314964512 learned_addr learned my addr 172.21.2.102:0/2314964512 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:03:55.352 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:55.337 7f60b261c700 1 --2- 172.21.2.102:0/2314964512 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f60b40e5070 0x7f60b41b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:55.352 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:55.337 7f60b1e1b700 1 -- 172.21.2.102:0/2314964512 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f60b40e5070 msgr2=0x7f60b41b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:03:55.352 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:55.337 7f60b1e1b700 1 --2- 172.21.2.102:0/2314964512 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f60b40e5070 0x7f60b41b3820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:55.353 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:55.337 7f60b1e1b700 1 -- 172.21.2.102:0/2314964512 >> v1:172.21.2.102:6789/0 conn(0x7f60b41b3e40 legacy=0x7f60b407dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:55.353 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:55.337 7f60b1e1b700 1 -- 172.21.2.102:0/2314964512 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60b40806f0 con 0x7f60b40e5470 2021-02-10T11:03:55.353 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:55.337 7f60b1e1b700 1 --2- 172.21.2.102:0/2314964512 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f60b40e5470 0x7f60b407d690 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7f60a4006700 tx=0x7f60a4009b80).ready entity=mon.0 client_cookie=136a24806d5ca436 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:03:55.353 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:55.338 7f60b161a700 1 -- 172.21.2.102:0/2314964512 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f60a400c0c0 con 0x7f60b40e5470 2021-02-10T11:03:55.353 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:55.338 7f60b161a700 1 -- 172.21.2.102:0/2314964512 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60a400c440 con 0x7f60b40e5470 2021-02-10T11:03:55.353 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:55.338 7f60b161a700 1 -- 172.21.2.102:0/2314964512 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f60a400c960 con 0x7f60b40e5470 2021-02-10T11:03:55.353 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:55.338 7f60b899e700 1 -- 172.21.2.102:0/2314964512 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f60b40e5470 msgr2=0x7f60b407d690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:55.353 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:55.338 7f60b899e700 1 --2- 172.21.2.102:0/2314964512 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f60b40e5470 0x7f60b407d690 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7f60a4006700 tx=0x7f60a4009b80).stop 2021-02-10T11:03:55.353 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:55.338 7f60b899e700 1 -- 172.21.2.102:0/2314964512 shutdown_connections 2021-02-10T11:03:55.353 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:55.338 7f60b899e700 1 --2- 172.21.2.102:0/2314964512 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f60b40e5470 0x7f60b407d690 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:55.353 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:55.338 7f60b899e700 1 --2- 172.21.2.102:0/2314964512 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f60b40e5070 0x7f60b41b3820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:55.353 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:55.338 7f60b899e700 1 -- 172.21.2.102:0/2314964512 >> 172.21.2.102:0/2314964512 conn(0x7f60b40ce690 msgr2=0x7f60b40e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:03:55.354 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:55.338 7f60b899e700 1 -- 172.21.2.102:0/2314964512 shutdown_connections 2021-02-10T11:03:55.354 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:55.339 7f60b899e700 1 -- 172.21.2.102:0/2314964512 wait complete. 2021-02-10T11:03:55.355 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:55.353 7f60b899e700 1 Processor -- start 2021-02-10T11:03:55.355 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:55.353 7f60b899e700 1 -- start start 2021-02-10T11:03:55.355 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:55.353 7f60b899e700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f60b409f6c0 0x7f60b40bc050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:55.355 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:55.353 7f60b899e700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f60b4081230 0x7f60b4198bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:55.356 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:55.353 7f60b899e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f60b41991d0 0x7f60b40c1580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:55.356 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:55.353 7f60b899e700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60b40bc670 con 0x7f60b41991d0 2021-02-10T11:03:55.356 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:55.353 7f60b899e700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60b4081bf0 con 0x7f60b409f6c0 2021-02-10T11:03:55.356 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:55.353 7f60b899e700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60b4081e10 con 0x7f60b4081230 2021-02-10T11:03:55.357 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:55.353 7f60b2e1d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f60b41991d0 0x7f60b40c1580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:55.357 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:55.353 7f60b2e1d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f60b41991d0 0x7f60b40c1580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:49412/0 (socket says 172.21.2.102:49412) 2021-02-10T11:03:55.357 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:55.353 7f60b2e1d700 1 -- 172.21.2.102:0/3062732449 learned_addr learned my addr 172.21.2.102:0/3062732449 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:03:55.357 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:55.353 7f60b1e1b700 1 --2- 172.21.2.102:0/3062732449 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f60b4081230 0x7f60b4198bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:55.357 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:55.354 7f60b261c700 1 --2- 172.21.2.102:0/3062732449 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f60b409f6c0 0x7f60b40bc050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:55.357 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:55.354 7f60b2e1d700 1 -- 172.21.2.102:0/3062732449 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f60b4081230 msgr2=0x7f60b4198bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:55.357 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:55.354 7f60b2e1d700 1 --2- 172.21.2.102:0/3062732449 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f60b4081230 0x7f60b4198bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:55.358 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:55.354 7f60b2e1d700 1 -- 172.21.2.102:0/3062732449 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f60b409f6c0 msgr2=0x7f60b40bc050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:55.358 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:55.354 7f60b2e1d700 1 --2- 172.21.2.102:0/3062732449 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f60b409f6c0 0x7f60b40bc050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:55.358 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:55.354 7f60b2e1d700 1 -- 172.21.2.102:0/3062732449 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60b4089550 con 0x7f60b41991d0 2021-02-10T11:03:55.358 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:55.354 7f60b1e1b700 1 --2- 172.21.2.102:0/3062732449 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f60b4081230 0x7f60b4198bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:03:55.358 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:55.354 7f60b2e1d700 1 --2- 172.21.2.102:0/3062732449 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f60b41991d0 0x7f60b40c1580 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f60ac009980 tx=0x7f60ac00bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:03:55.358 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:55.354 7f60aaffd700 1 -- 172.21.2.102:0/3062732449 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f60ac005870 con 0x7f60b41991d0 2021-02-10T11:03:55.358 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:55.354 7f60aaffd700 1 -- 172.21.2.102:0/3062732449 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60ac005ab0 con 0x7f60b41991d0 2021-02-10T11:03:55.359 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:55.354 7f60aaffd700 1 -- 172.21.2.102:0/3062732449 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f60ac006030 con 0x7f60b41991d0 2021-02-10T11:03:55.359 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:55.354 7f60b899e700 1 -- 172.21.2.102:0/3062732449 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60b4089550 con 0x7f60b41991d0 2021-02-10T11:03:55.359 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:55.354 7f60b899e700 1 -- 172.21.2.102:0/3062732449 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60b40c1ba0 con 0x7f60b41991d0 2021-02-10T11:03:55.359 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:55.355 7f60aaffd700 1 -- 172.21.2.102:0/3062732449 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f60ac006030 con 0x7f60b41991d0 2021-02-10T11:03:55.359 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:55.355 7f60aaffd700 1 --2- 172.21.2.102:0/3062732449 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f60980186c0 0x7f609801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:55.359 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:55.355 7f60aaffd700 1 -- 172.21.2.102:0/3062732449 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f60ac032920 con 0x7f60b41991d0 2021-02-10T11:03:55.359 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:55.355 7f60b261c700 1 --2- 172.21.2.102:0/3062732449 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f60980186c0 0x7f609801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:55.359 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:55.355 7f60b261c700 1 --2- 172.21.2.102:0/3062732449 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f60980186c0 0x7f609801abc0 crc :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:03:55.386 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:55.386 7f60b899e700 1 --2- 172.21.2.102:0/3062732449 >> [v2:172.21.2.113:6800/36367,v1:172.21.2.113:6801/36367] conn(0x7f60b40c45b0 0x7f60b40627c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:55.387 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:55.386 7f60b899e700 1 -- 172.21.2.102:0/3062732449 --> [v2:172.21.2.113:6800/36367,v1:172.21.2.113:6801/36367] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f60b40c1ba0 con 0x7f60b40c45b0 2021-02-10T11:03:55.387 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:55.387 7f60b1e1b700 1 --2- 172.21.2.102:0/3062732449 >> [v2:172.21.2.113:6800/36367,v1:172.21.2.113:6801/36367] conn(0x7f60b40c45b0 0x7f60b40627c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:55.387 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:55.387 7f60b1e1b700 1 --2- 172.21.2.102:0/3062732449 >> [v2:172.21.2.113:6800/36367,v1:172.21.2.113:6801/36367] conn(0x7f60b40c45b0 0x7f60b40627c0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:03:55.387 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:55.388 7f60aaffd700 1 -- 172.21.2.102:0/3062732449 <== osd.6 v2:172.21.2.113:6800/36367 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+5151 (crc 0 0 0) 0x7f60a40009b0 con 0x7f60b40c45b0 2021-02-10T11:03:55.393 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:55.394 7f60b899e700 1 -- 172.21.2.102:0/3062732449 --> [v2:172.21.2.113:6800/36367,v1:172.21.2.113:6801/36367] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f60b4089550 con 0x7f60b40c45b0 2021-02-10T11:03:55.394 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:55.395 7f60aaffd700 1 -- 172.21.2.102:0/3062732449 <== osd.6 v2:172.21.2.113:6800/36367 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f60a4000940 con 0x7f60b40c45b0 2021-02-10T11:03:55.395 INFO:teuthology.orchestra.run.gibba002.stdout:55834574856 2021-02-10T11:03:55.395 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:55.396 7f60b899e700 1 -- 172.21.2.102:0/3062732449 >> [v2:172.21.2.113:6800/36367,v1:172.21.2.113:6801/36367] conn(0x7f60b40c45b0 msgr2=0x7f60b40627c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:55.395 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:55.396 7f60b899e700 1 --2- 172.21.2.102:0/3062732449 >> [v2:172.21.2.113:6800/36367,v1:172.21.2.113:6801/36367] conn(0x7f60b40c45b0 0x7f60b40627c0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:55.395 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:55.396 7f60b899e700 1 -- 172.21.2.102:0/3062732449 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f60980186c0 msgr2=0x7f609801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:55.395 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:55.396 7f60b899e700 1 --2- 172.21.2.102:0/3062732449 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f60980186c0 0x7f609801abc0 crc :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:55.396 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:55.396 7f60b899e700 1 -- 172.21.2.102:0/3062732449 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f60b41991d0 msgr2=0x7f60b40c1580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:55.396 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:55.396 7f60b899e700 1 --2- 172.21.2.102:0/3062732449 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f60b41991d0 0x7f60b40c1580 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f60ac009980 tx=0x7f60ac00bcd0).stop 2021-02-10T11:03:55.396 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:55.396 7f60b2e1d700 1 -- 172.21.2.102:0/3062732449 reap_dead start 2021-02-10T11:03:55.396 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:55.396 7f60b899e700 1 -- 172.21.2.102:0/3062732449 shutdown_connections 2021-02-10T11:03:55.396 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:55.396 7f60b899e700 1 -- 172.21.2.102:0/3062732449 >> 172.21.2.102:0/3062732449 conn(0x7f60b40c0d40 msgr2=0x7f60b41073b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:03:55.396 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:55.396 7f60b899e700 1 -- 172.21.2.102:0/3062732449 shutdown_connections 2021-02-10T11:03:55.396 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:55.396 7f60b899e700 1 -- 172.21.2.102:0/3062732449 wait complete. 2021-02-10T11:03:55.403 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell osd.7 flush_pg_stats 2021-02-10T11:03:55.826 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:55.811 7f1dd0294700 1 Processor -- start 2021-02-10T11:03:55.827 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:55.811 7f1dd0294700 1 -- start start 2021-02-10T11:03:55.827 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:55.812 7f1dd0294700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1dc80e5070 0x7f1dc81b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:55.827 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:55.812 7f1dd0294700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f1dc81b3e40 0x7f1dc8080310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:55.827 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:55.812 7f1dd0294700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1dc807ed80 con 0x7f1dc80e5470 2021-02-10T11:03:55.827 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:55.812 7f1dd0294700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1dc80d1790 con 0x7f1dc81b3e40 2021-02-10T11:03:55.827 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:55.812 7f1dd0294700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1dc80d1980 con 0x7f1dc80e5070 2021-02-10T11:03:55.827 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:55.812 7f1dce8b1700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f1dc81b3e40 0x7f1dc8080310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:55.827 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:55.812 7f1dce8b1700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f1dc81b3e40 0x7f1dc8080310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:49418/0 (socket says 172.21.2.102:49418) 2021-02-10T11:03:55.827 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:55.812 7f1dce8b1700 1 -- 172.21.2.102:0/2667425826 learned_addr learned my addr 172.21.2.102:0/2667425826 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:03:55.827 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:55.812 7f1dce8b1700 1 -- 172.21.2.102:0/2667425826 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1dc80e5070 msgr2=0x7f1dc81b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:03:55.827 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:55.812 7f1dce8b1700 1 --2- 172.21.2.102:0/2667425826 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1dc80e5070 0x7f1dc81b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:03:55.827 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:55.812 7f1dce8b1700 1 -- 172.21.2.102:0/2667425826 >> v1:172.21.2.113:6789/0 conn(0x7f1dc80e5470 legacy=0x7f1dc807d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:03:55.827 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:55.812 7f1dce8b1700 1 -- 172.21.2.102:0/2667425826 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dc80d1ba0 con 0x7f1dc81b3e40 2021-02-10T11:03:55.828 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:55.813 7f1dce8b1700 1 --2- 172.21.2.102:0/2667425826 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1dc81b3e40 0x7f1dc8080310 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7f1dc0006f00 tx=0x7f1dc0009f10).ready entity=mon.0 client_cookie=5b41418c45693c51 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:03:55.828 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:55.813 7f1dcd0ae700 1 -- 172.21.2.102:0/2667425826 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1dc000b3b0 con 0x7f1dc81b3e40 2021-02-10T11:03:55.828 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:55.813 7f1dcd0ae700 1 -- 172.21.2.102:0/2667425826 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1dc000b630 con 0x7f1dc81b3e40 2021-02-10T11:03:55.828 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:55.813 7f1dcd0ae700 1 -- 172.21.2.102:0/2667425826 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1dc000bb50 con 0x7f1dc81b3e40 2021-02-10T11:03:55.828 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:55.813 7f1dd0294700 1 -- 172.21.2.102:0/2667425826 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1dc81b3e40 msgr2=0x7f1dc8080310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:55.828 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:55.813 7f1dd0294700 1 --2- 172.21.2.102:0/2667425826 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1dc81b3e40 0x7f1dc8080310 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7f1dc0006f00 tx=0x7f1dc0009f10).stop 2021-02-10T11:03:55.828 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:55.813 7f1dd0294700 1 -- 172.21.2.102:0/2667425826 shutdown_connections 2021-02-10T11:03:55.828 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:55.813 7f1dd0294700 1 --2- 172.21.2.102:0/2667425826 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1dc81b3e40 0x7f1dc8080310 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:55.828 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:55.813 7f1dd0294700 1 --2- 172.21.2.102:0/2667425826 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1dc80e5070 0x7f1dc81b3820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:03:55.828 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:55.813 7f1dd0294700 1 -- 172.21.2.102:0/2667425826 >> 172.21.2.102:0/2667425826 conn(0x7f1dc80ce690 msgr2=0x7f1dc80e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:03:55.828 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:55.814 7f1dd0294700 1 -- 172.21.2.102:0/2667425826 shutdown_connections 2021-02-10T11:03:55.828 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:55.814 7f1dd0294700 1 -- 172.21.2.102:0/2667425826 wait complete. 2021-02-10T11:03:55.830 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:55.828 7f1dd0294700 1 Processor -- start 2021-02-10T11:03:55.830 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:55.828 7f1dd0294700 1 -- start start 2021-02-10T11:03:55.830 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:55.828 7f1dd0294700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1dc80be580 0x7f1dc80be980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:55.830 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:55.828 7f1dd0294700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1dc8097ed0 0x7f1dc80982d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:55.830 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:55.828 7f1dd0294700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1dc8099180 0x7f1dc8099580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:55.830 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:55.828 7f1dd0294700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dc80988f0 con 0x7f1dc8097ed0 2021-02-10T11:03:55.830 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:55.828 7f1dd0294700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dc80c1110 con 0x7f1dc80be580 2021-02-10T11:03:55.830 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:55.828 7f1dd0294700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1dc8099ba0 con 0x7f1dc8099180 2021-02-10T11:03:55.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:55.828 7f1dcd8af700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1dc8097ed0 0x7f1dc80982d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:55.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:55.829 7f1dcd8af700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1dc8097ed0 0x7f1dc80982d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:49424/0 (socket says 172.21.2.102:49424) 2021-02-10T11:03:55.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:55.829 7f1dcd8af700 1 -- 172.21.2.102:0/2903403932 learned_addr learned my addr 172.21.2.102:0/2903403932 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:03:55.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:55.829 7f1dce0b0700 1 --2- 172.21.2.102:0/2903403932 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1dc80be580 0x7f1dc80be980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:55.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:55.829 7f1dce8b1700 1 --2- 172.21.2.102:0/2903403932 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1dc8099180 0x7f1dc8099580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:55.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:55.829 7f1dcd8af700 1 -- 172.21.2.102:0/2903403932 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1dc8099180 msgr2=0x7f1dc8099580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:55.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:55.829 7f1dcd8af700 1 --2- 172.21.2.102:0/2903403932 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1dc8099180 0x7f1dc8099580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:55.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:55.829 7f1dcd8af700 1 -- 172.21.2.102:0/2903403932 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1dc80be580 msgr2=0x7f1dc80be980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:55.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:55.829 7f1dcd8af700 1 --2- 172.21.2.102:0/2903403932 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1dc80be580 0x7f1dc80be980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:55.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:55.829 7f1dcd8af700 1 -- 172.21.2.102:0/2903403932 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dc8099dc0 con 0x7f1dc8097ed0 2021-02-10T11:03:55.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:55.829 7f1dcd8af700 1 --2- 172.21.2.102:0/2903403932 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1dc8097ed0 0x7f1dc80982d0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f1dc400b5b0 tx=0x7f1dc400b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:03:55.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:55.829 7f1dbaffd700 1 -- 172.21.2.102:0/2903403932 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1dc400bd40 con 0x7f1dc8097ed0 2021-02-10T11:03:55.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:55.829 7f1dbaffd700 1 -- 172.21.2.102:0/2903403932 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1dc4000e20 con 0x7f1dc8097ed0 2021-02-10T11:03:55.832 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:55.829 7f1dbaffd700 1 -- 172.21.2.102:0/2903403932 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1dc40054a0 con 0x7f1dc8097ed0 2021-02-10T11:03:55.832 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:55.829 7f1dd0294700 1 -- 172.21.2.102:0/2903403932 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1dc8099dc0 con 0x7f1dc8097ed0 2021-02-10T11:03:55.832 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:55.829 7f1dd0294700 1 -- 172.21.2.102:0/2903403932 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1dc809a350 con 0x7f1dc8097ed0 2021-02-10T11:03:55.832 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:55.830 7f1dbaffd700 1 -- 172.21.2.102:0/2903403932 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1dc4006030 con 0x7f1dc8097ed0 2021-02-10T11:03:55.832 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:55.830 7f1dbaffd700 1 --2- 172.21.2.102:0/2903403932 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1db00186c0 0x7f1db001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:55.832 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:55.830 7f1dbaffd700 1 -- 172.21.2.102:0/2903403932 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f1dc4032390 con 0x7f1dc8097ed0 2021-02-10T11:03:55.832 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:55.830 7f1dce0b0700 1 --2- 172.21.2.102:0/2903403932 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1db00186c0 0x7f1db001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:55.832 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:55.830 7f1dce0b0700 1 --2- 172.21.2.102:0/2903403932 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1db00186c0 0x7f1db001abc0 crc :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:03:55.860 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:55.862 7f1dd0294700 1 --2- 172.21.2.102:0/2903403932 >> [v2:172.21.2.113:6817/36365,v1:172.21.2.113:6821/36365] conn(0x7f1dc80c45a0 0x7f1dc80627c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:55.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:55.862 7f1dd0294700 1 -- 172.21.2.102:0/2903403932 --> [v2:172.21.2.113:6817/36365,v1:172.21.2.113:6821/36365] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1dc809a350 con 0x7f1dc80c45a0 2021-02-10T11:03:55.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:55.862 7f1dce8b1700 1 --2- 172.21.2.102:0/2903403932 >> [v2:172.21.2.113:6817/36365,v1:172.21.2.113:6821/36365] conn(0x7f1dc80c45a0 0x7f1dc80627c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:55.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:55.863 7f1dce8b1700 1 --2- 172.21.2.102:0/2903403932 >> [v2:172.21.2.113:6817/36365,v1:172.21.2.113:6821/36365] conn(0x7f1dc80c45a0 0x7f1dc80627c0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:03:55.862 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:55.863 7f1dbaffd700 1 -- 172.21.2.102:0/2903403932 <== osd.7 v2:172.21.2.113:6817/36365 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+5151 (crc 0 0 0) 0x7f1dc0000f90 con 0x7f1dc80c45a0 2021-02-10T11:03:55.868 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:55.869 7f1dd0294700 1 -- 172.21.2.102:0/2903403932 --> [v2:172.21.2.113:6817/36365,v1:172.21.2.113:6821/36365] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f1dc8098c50 con 0x7f1dc80c45a0 2021-02-10T11:03:55.869 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:55.870 7f1dbaffd700 1 -- 172.21.2.102:0/2903403932 <== osd.7 v2:172.21.2.113:6817/36365 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f1dc0006950 con 0x7f1dc80c45a0 2021-02-10T11:03:55.869 INFO:teuthology.orchestra.run.gibba002.stdout:55834574856 2021-02-10T11:03:55.870 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:55.871 7f1dd0294700 1 -- 172.21.2.102:0/2903403932 >> [v2:172.21.2.113:6817/36365,v1:172.21.2.113:6821/36365] conn(0x7f1dc80c45a0 msgr2=0x7f1dc80627c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:55.870 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:55.871 7f1dd0294700 1 --2- 172.21.2.102:0/2903403932 >> [v2:172.21.2.113:6817/36365,v1:172.21.2.113:6821/36365] conn(0x7f1dc80c45a0 0x7f1dc80627c0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:55.870 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:55.871 7f1dd0294700 1 -- 172.21.2.102:0/2903403932 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1db00186c0 msgr2=0x7f1db001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:55.870 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:55.871 7f1dd0294700 1 --2- 172.21.2.102:0/2903403932 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1db00186c0 0x7f1db001abc0 crc :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:55.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:55.871 7f1dd0294700 1 -- 172.21.2.102:0/2903403932 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1dc8097ed0 msgr2=0x7f1dc80982d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:55.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:55.871 7f1dd0294700 1 --2- 172.21.2.102:0/2903403932 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1dc8097ed0 0x7f1dc80982d0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f1dc400b5b0 tx=0x7f1dc400b950).stop 2021-02-10T11:03:55.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:55.871 7f1dce8b1700 1 -- 172.21.2.102:0/2903403932 reap_dead start 2021-02-10T11:03:55.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:55.871 7f1dd0294700 1 -- 172.21.2.102:0/2903403932 shutdown_connections 2021-02-10T11:03:55.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:55.871 7f1dd0294700 1 -- 172.21.2.102:0/2903403932 >> 172.21.2.102:0/2903403932 conn(0x7f1dc80bde70 msgr2=0x7f1dc80c3970 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:03:55.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:55.871 7f1dd0294700 1 -- 172.21.2.102:0/2903403932 shutdown_connections 2021-02-10T11:03:55.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:55.871 7f1dd0294700 1 -- 172.21.2.102:0/2903403932 wait complete. 2021-02-10T11:03:55.878 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd last-stat-seq osd.0 2021-02-10T11:03:56.306 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:56.289 7f65e5c91700 1 Processor -- start 2021-02-10T11:03:56.307 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:56.289 7f65e5c91700 1 -- start start 2021-02-10T11:03:56.307 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:56.289 7f65e5c91700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f65e00e5470 0x7f65e007e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:56.307 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:56.289 7f65e5c91700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f65e007f300 0x7f65e007f700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:56.307 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:56.289 7f65e5c91700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65e007ed80 con 0x7f65e00e5070 2021-02-10T11:03:56.307 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:56.289 7f65e5c91700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65e01b4010 con 0x7f65e00e5470 2021-02-10T11:03:56.307 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:56.289 7f65e5c91700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65e00d1790 con 0x7f65e007f300 2021-02-10T11:03:56.307 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:56.290 7f65deffd700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f65e00e5470 0x7f65e007e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:56.307 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:56.290 7f65deffd700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f65e00e5470 0x7f65e007e6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:49434/0 (socket says 172.21.2.102:49434) 2021-02-10T11:03:56.307 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:56.290 7f65deffd700 1 -- 172.21.2.102:0/156015178 learned_addr learned my addr 172.21.2.102:0/156015178 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:03:56.307 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:56.290 7f65deffd700 1 -- 172.21.2.102:0/156015178 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f65e007f300 msgr2=0x7f65e007f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:03:56.307 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:56.290 7f65deffd700 1 --2- 172.21.2.102:0/156015178 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f65e007f300 0x7f65e007f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:03:56.307 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:56.290 7f65deffd700 1 -- 172.21.2.102:0/156015178 >> v1:172.21.2.113:6789/0 conn(0x7f65e00e5070 legacy=0x7f65e01b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:03:56.307 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:56.290 7f65deffd700 1 -- 172.21.2.102:0/156015178 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65e00d19b0 con 0x7f65e00e5470 2021-02-10T11:03:56.308 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:56.290 7f65deffd700 1 --2- 172.21.2.102:0/156015178 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f65e00e5470 0x7f65e007e6a0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7f65d0006700 tx=0x7f65d0009b80).ready entity=mon.0 client_cookie=3168322099c5d4df server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:03:56.308 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:56.290 7f65de7fc700 1 -- 172.21.2.102:0/156015178 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f65d000c0c0 con 0x7f65e00e5470 2021-02-10T11:03:56.308 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:56.290 7f65de7fc700 1 -- 172.21.2.102:0/156015178 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65d000c3c0 con 0x7f65e00e5470 2021-02-10T11:03:56.308 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:56.290 7f65de7fc700 1 -- 172.21.2.102:0/156015178 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f65d000c8e0 con 0x7f65e00e5470 2021-02-10T11:03:56.308 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:56.290 7f65e5c91700 1 -- 172.21.2.102:0/156015178 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f65e00e5470 msgr2=0x7f65e007e6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:56.308 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:56.290 7f65e5c91700 1 --2- 172.21.2.102:0/156015178 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f65e00e5470 0x7f65e007e6a0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7f65d0006700 tx=0x7f65d0009b80).stop 2021-02-10T11:03:56.308 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:56.291 7f65e5c91700 1 -- 172.21.2.102:0/156015178 shutdown_connections 2021-02-10T11:03:56.308 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:56.291 7f65e5c91700 1 --2- 172.21.2.102:0/156015178 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f65e007f300 0x7f65e007f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:03:56.308 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:56.291 7f65e5c91700 1 --2- 172.21.2.102:0/156015178 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f65e00e5470 0x7f65e007e6a0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:56.308 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:56.291 7f65e5c91700 1 -- 172.21.2.102:0/156015178 >> 172.21.2.102:0/156015178 conn(0x7f65e00ce690 msgr2=0x7f65e00e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:03:56.308 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:56.291 7f65e5c91700 1 -- 172.21.2.102:0/156015178 shutdown_connections 2021-02-10T11:03:56.308 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:56.291 7f65e5c91700 1 -- 172.21.2.102:0/156015178 wait complete. 2021-02-10T11:03:56.308 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:56.306 7f65e5c91700 1 Processor -- start 2021-02-10T11:03:56.309 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:56.306 7f65e5c91700 1 -- start start 2021-02-10T11:03:56.309 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:56.306 7f65e5c91700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f65e00be580 0x7f65e00be980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:56.309 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:56.306 7f65e5c91700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f65e0097ed0 0x7f65e00982d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:56.309 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:56.306 7f65e5c91700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f65e0099180 0x7f65e0099580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:56.309 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:56.306 7f65e5c91700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65e00988f0 con 0x7f65e00be580 2021-02-10T11:03:56.309 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:56.306 7f65e5c91700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65e00c1110 con 0x7f65e0097ed0 2021-02-10T11:03:56.309 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:56.306 7f65e5c91700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65e0099ba0 con 0x7f65e0099180 2021-02-10T11:03:56.309 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:56.306 7f65df7fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f65e00be580 0x7f65e00be980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:56.309 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:56.306 7f65df7fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f65e00be580 0x7f65e00be980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:49440/0 (socket says 172.21.2.102:49440) 2021-02-10T11:03:56.309 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:56.306 7f65df7fe700 1 -- 172.21.2.102:0/133098574 learned_addr learned my addr 172.21.2.102:0/133098574 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:03:56.309 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:56.307 7f65df7fe700 1 -- 172.21.2.102:0/133098574 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f65e0099180 msgr2=0x7f65e0099580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:56.309 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:56.307 7f65df7fe700 1 --2- 172.21.2.102:0/133098574 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f65e0099180 0x7f65e0099580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:03:56.309 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:56.307 7f65df7fe700 1 -- 172.21.2.102:0/133098574 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f65e0097ed0 msgr2=0x7f65e00982d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:56.309 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:56.307 7f65df7fe700 1 --2- 172.21.2.102:0/133098574 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f65e0097ed0 0x7f65e00982d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:03:56.310 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:56.307 7f65df7fe700 1 -- 172.21.2.102:0/133098574 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65e0099dc0 con 0x7f65e00be580 2021-02-10T11:03:56.310 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:56.307 7f65df7fe700 1 --2- 172.21.2.102:0/133098574 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f65e00be580 0x7f65e00be980 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f65c800b5b0 tx=0x7f65c800b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:03:56.310 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:56.307 7f65d7fff700 1 -- 172.21.2.102:0/133098574 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f65c800bd40 con 0x7f65e00be580 2021-02-10T11:03:56.310 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:56.307 7f65d7fff700 1 -- 172.21.2.102:0/133098574 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65c8000e20 con 0x7f65e00be580 2021-02-10T11:03:56.310 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:56.307 7f65e5c91700 1 -- 172.21.2.102:0/133098574 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65e0099dc0 con 0x7f65e00be580 2021-02-10T11:03:56.310 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:56.307 7f65d7fff700 1 -- 172.21.2.102:0/133098574 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f65c80054a0 con 0x7f65e00be580 2021-02-10T11:03:56.310 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:56.307 7f65e5c91700 1 -- 172.21.2.102:0/133098574 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65e009a350 con 0x7f65e00be580 2021-02-10T11:03:56.311 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:56.308 7f65d7fff700 1 -- 172.21.2.102:0/133098574 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f65c8006030 con 0x7f65e00be580 2021-02-10T11:03:56.311 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:56.308 7f65d7fff700 1 --2- 172.21.2.102:0/133098574 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f65c40186c0 0x7f65c401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:56.311 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:56.308 7f65d7fff700 1 -- 172.21.2.102:0/133098574 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f65c80323e0 con 0x7f65e00be580 2021-02-10T11:03:56.311 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:56.308 7f65deffd700 1 --2- 172.21.2.102:0/133098574 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f65c40186c0 0x7f65c401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:56.311 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:56.308 7f65deffd700 1 --2- 172.21.2.102:0/133098574 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f65c40186c0 0x7f65c401abc0 crc :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:03:56.338 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:56.339 7f65e5c91700 1 -- 172.21.2.102:0/133098574 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65e0098c50 con 0x7f65e00be580 2021-02-10T11:03:56.341 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:56.342 7f65d7fff700 1 -- 172.21.2.102:0/133098574 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f65c80009b0 con 0x7f65e00be580 2021-02-10T11:03:56.480 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:56.481 7f65e5c91700 1 -- 172.21.2.102:0/133098574 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f65e0098c50 con 0x7f65e00be580 2021-02-10T11:03:56.481 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:56.482 7f65d7fff700 1 -- 172.21.2.102:0/133098574 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f65c800bd40 con 0x7f65e00be580 2021-02-10T11:03:56.481 INFO:teuthology.orchestra.run.gibba002.stdout:55834574855 2021-02-10T11:03:56.483 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:56.484 7f65e5c91700 1 -- 172.21.2.102:0/133098574 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f65c40186c0 msgr2=0x7f65c401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:56.483 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:56.484 7f65e5c91700 1 --2- 172.21.2.102:0/133098574 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f65c40186c0 0x7f65c401abc0 crc :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:56.483 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:56.484 7f65e5c91700 1 -- 172.21.2.102:0/133098574 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f65e00be580 msgr2=0x7f65e00be980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:56.483 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:56.484 7f65e5c91700 1 --2- 172.21.2.102:0/133098574 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f65e00be580 0x7f65e00be980 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f65c800b5b0 tx=0x7f65c800b950).stop 2021-02-10T11:03:56.483 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:56.484 7f65e5c91700 1 -- 172.21.2.102:0/133098574 shutdown_connections 2021-02-10T11:03:56.483 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:56.484 7f65e5c91700 1 --2- 172.21.2.102:0/133098574 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f65e0099180 0x7f65e0099580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:03:56.483 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:56.484 7f65e5c91700 1 --2- 172.21.2.102:0/133098574 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f65c40186c0 0x7f65c401abc0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:56.483 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:56.484 7f65e5c91700 1 --2- 172.21.2.102:0/133098574 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f65e0097ed0 0x7f65e00982d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:03:56.483 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:56.484 7f65e5c91700 1 --2- 172.21.2.102:0/133098574 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f65e00be580 0x7f65e00be980 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:56.483 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:56.484 7f65e5c91700 1 -- 172.21.2.102:0/133098574 >> 172.21.2.102:0/133098574 conn(0x7f65e00bde70 msgr2=0x7f65e00c3970 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:03:56.484 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:56.484 7f65e5c91700 1 -- 172.21.2.102:0/133098574 shutdown_connections 2021-02-10T11:03:56.484 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:56.484 7f65e5c91700 1 -- 172.21.2.102:0/133098574 wait complete. 2021-02-10T11:03:56.492 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574855 for osd.0 2021-02-10T11:03:56.492 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd last-stat-seq osd.1 2021-02-10T11:03:56.917 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:56.900 7fba5381f700 1 Processor -- start 2021-02-10T11:03:56.917 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:56.900 7fba5381f700 1 -- start start 2021-02-10T11:03:56.917 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:56.901 7fba5381f700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fba4c0e5070 0x7fba4c1b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:56.917 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:56.901 7fba5381f700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fba4c0e5470 0x7fba4c080310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:56.917 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:56.901 7fba5381f700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba4c07ed80 con 0x7fba4c1b3e40 2021-02-10T11:03:56.917 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:56.901 7fba5381f700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba4c0d1790 con 0x7fba4c0e5070 2021-02-10T11:03:56.918 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:56.901 7fba5381f700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba4c0d19b0 con 0x7fba4c0e5470 2021-02-10T11:03:56.918 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:56.901 7fba50e3a700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fba4c1b3e40 0x7fba4c07d690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:41670/0 (socket says 172.21.2.102:41670) 2021-02-10T11:03:56.918 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:56.901 7fba50e3a700 1 -- 172.21.2.102:0/1744413962 learned_addr learned my addr 172.21.2.102:0/1744413962 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:03:56.918 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:56.901 7fba3ffff700 1 -- 172.21.2.102:0/1744413962 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1334750751 0 0) 0x7fba480010b0 con 0x7fba4c1b3e40 2021-02-10T11:03:56.918 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:56.901 7fba3ffff700 1 -- 172.21.2.102:0/1744413962 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba38002fe0 con 0x7fba4c1b3e40 2021-02-10T11:03:56.918 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:56.901 7fba51e3c700 1 --2- 172.21.2.102:0/1744413962 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fba4c0e5470 0x7fba4c080310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:56.918 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:56.901 7fba5163b700 1 --2- 172.21.2.102:0/1744413962 >> v2:172.21.2.113:3300/0 conn(0x7fba4c0e5070 0x7fba4c1b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:56.918 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:56.902 7fba3ffff700 1 -- 172.21.2.102:0/1744413962 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2947874300 0 0) 0x7fba480010b0 con 0x7fba4c1b3e40 2021-02-10T11:03:56.918 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:56.902 7fba3ffff700 1 -- 172.21.2.102:0/1744413962 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fba4c0e5470 msgr2=0x7fba4c080310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:03:56.918 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:56.902 7fba3ffff700 1 --2- 172.21.2.102:0/1744413962 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fba4c0e5470 0x7fba4c080310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:56.918 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:56.902 7fba3ffff700 1 -- 172.21.2.102:0/1744413962 >> v2:172.21.2.113:3300/0 conn(0x7fba4c0e5070 msgr2=0x7fba4c1b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:03:56.918 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:56.902 7fba3ffff700 1 --2- 172.21.2.102:0/1744413962 >> v2:172.21.2.113:3300/0 conn(0x7fba4c0e5070 0x7fba4c1b3820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:56.919 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:56.902 7fba3ffff700 1 -- 172.21.2.102:0/1744413962 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba4c0d1bd0 con 0x7fba4c1b3e40 2021-02-10T11:03:56.919 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:56.902 7fba3ffff700 1 -- 172.21.2.102:0/1744413962 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7fba480015b0 con 0x7fba4c1b3e40 2021-02-10T11:03:56.919 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:56.902 7fba3ffff700 1 -- 172.21.2.102:0/1744413962 >> v1:172.21.2.102:6789/0 conn(0x7fba4c1b3e40 legacy=0x7fba4c07d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:56.919 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:56.902 7fba3ffff700 1 --2- 172.21.2.102:0/1744413962 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fba38003a70 0x7fba38005ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:56.919 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:56.902 7fba3ffff700 1 --2- 172.21.2.102:0/1744413962 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fba38006760 0x7fba38008c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:56.919 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:56.902 7fba3ffff700 1 --2- 172.21.2.102:0/1744413962 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fba38009500 0x7fba3800ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:56.919 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:56.902 7fba3ffff700 1 -- 172.21.2.102:0/1744413962 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba380502e0 con 0x7fba38006760 2021-02-10T11:03:56.919 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:56.902 7fba3ffff700 1 -- 172.21.2.102:0/1744413962 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba38050500 con 0x7fba38009500 2021-02-10T11:03:56.919 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:56.902 7fba3ffff700 1 -- 172.21.2.102:0/1744413962 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba38050720 con 0x7fba38003a70 2021-02-10T11:03:56.919 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:56.902 7fba50e3a700 1 --2- 172.21.2.102:0/1744413962 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fba38006760 0x7fba38008c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:56.919 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:56.902 7fba5163b700 1 --2- 172.21.2.102:0/1744413962 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fba38003a70 0x7fba38005ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:56.919 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:56.902 7fba50e3a700 1 -- 172.21.2.102:0/1744413962 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fba38003a70 msgr2=0x7fba38005ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:03:56.919 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:56.902 7fba50e3a700 1 --2- 172.21.2.102:0/1744413962 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fba38003a70 0x7fba38005ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:56.919 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:56.902 7fba51e3c700 1 --2- 172.21.2.102:0/1744413962 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fba38009500 0x7fba3800ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:56.920 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:56.902 7fba50e3a700 1 -- 172.21.2.102:0/1744413962 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fba38009500 msgr2=0x7fba3800ba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:03:56.920 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:56.902 7fba50e3a700 1 --2- 172.21.2.102:0/1744413962 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fba38009500 0x7fba3800ba20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:56.920 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:56.902 7fba50e3a700 1 -- 172.21.2.102:0/1744413962 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba380509c0 con 0x7fba38006760 2021-02-10T11:03:56.920 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:56.902 7fba50e3a700 1 --2- 172.21.2.102:0/1744413962 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fba38006760 0x7fba38008c60 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7fba48000b50 tx=0x7fba48007ce0).ready entity=mon.0 client_cookie=92f30c07596bb2f2 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:03:56.920 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:56.902 7fba3ffff700 1 -- 172.21.2.102:0/1744413962 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fba4800a310 con 0x7fba38006760 2021-02-10T11:03:56.920 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:56.902 7fba3ffff700 1 -- 172.21.2.102:0/1744413962 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba4800a650 con 0x7fba38006760 2021-02-10T11:03:56.920 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:56.902 7fba3ffff700 1 -- 172.21.2.102:0/1744413962 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fba4800ab30 con 0x7fba38006760 2021-02-10T11:03:56.920 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:56.903 7fba5381f700 1 -- 172.21.2.102:0/1744413962 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fba38006760 msgr2=0x7fba38008c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:56.920 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:56.903 7fba5381f700 1 --2- 172.21.2.102:0/1744413962 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fba38006760 0x7fba38008c60 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7fba48000b50 tx=0x7fba48007ce0).stop 2021-02-10T11:03:56.920 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:56.903 7fba51e3c700 1 -- 172.21.2.102:0/1744413962 reap_dead start 2021-02-10T11:03:56.920 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:56.903 7fba5381f700 1 -- 172.21.2.102:0/1744413962 shutdown_connections 2021-02-10T11:03:56.921 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:56.903 7fba5381f700 1 -- 172.21.2.102:0/1744413962 >> 172.21.2.102:0/1744413962 conn(0x7fba4c0ce690 msgr2=0x7fba4c0e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:03:56.921 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:56.903 7fba5381f700 1 -- 172.21.2.102:0/1744413962 shutdown_connections 2021-02-10T11:03:56.921 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:56.903 7fba5381f700 1 -- 172.21.2.102:0/1744413962 wait complete. 2021-02-10T11:03:56.921 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:56.918 7fba5381f700 1 Processor -- start 2021-02-10T11:03:56.921 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:56.918 7fba5381f700 1 -- start start 2021-02-10T11:03:56.922 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:56.918 7fba5381f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fba4c0b4ad0 0x7fba4c0a78d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:56.922 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:56.918 7fba5381f700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fba4c0a7ef0 0x7fba4c1a6d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:56.923 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:56.918 7fba5381f700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fba4c0a82f0 0x7fba4c1a73b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:56.923 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:56.918 7fba5381f700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba4c0aa3c0 con 0x7fba4c0b4ad0 2021-02-10T11:03:56.923 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:56.918 7fba5381f700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba4c0a88c0 con 0x7fba4c0a7ef0 2021-02-10T11:03:56.923 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:56.918 7fba5381f700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba4c0b4ed0 con 0x7fba4c0a82f0 2021-02-10T11:03:56.923 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:56.919 7fba5163b700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fba4c0b4ad0 0x7fba4c0a78d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:56.923 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:56.919 7fba5163b700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fba4c0b4ad0 0x7fba4c0a78d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:49456/0 (socket says 172.21.2.102:49456) 2021-02-10T11:03:56.923 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:56.919 7fba5163b700 1 -- 172.21.2.102:0/1093685921 learned_addr learned my addr 172.21.2.102:0/1093685921 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:03:56.923 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:56.919 7fba5163b700 1 -- 172.21.2.102:0/1093685921 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fba4c0a82f0 msgr2=0x7fba4c1a73b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:03:56.923 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:56.919 7fba5163b700 1 --2- 172.21.2.102:0/1093685921 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fba4c0a82f0 0x7fba4c1a73b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:03:56.923 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:56.919 7fba5163b700 1 -- 172.21.2.102:0/1093685921 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fba4c0a7ef0 msgr2=0x7fba4c1a6d90 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:03:56.923 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:56.919 7fba5163b700 1 --2- 172.21.2.102:0/1093685921 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fba4c0a7ef0 0x7fba4c1a6d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:03:56.923 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:56.919 7fba5163b700 1 -- 172.21.2.102:0/1093685921 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba4c1a79d0 con 0x7fba4c0b4ad0 2021-02-10T11:03:56.923 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:56.919 7fba5163b700 1 --2- 172.21.2.102:0/1093685921 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fba4c0b4ad0 0x7fba4c0a78d0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7fba40006f60 tx=0x7fba4000b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:03:56.924 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:56.919 7fba3dffb700 1 -- 172.21.2.102:0/1093685921 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fba4000bd60 con 0x7fba4c0b4ad0 2021-02-10T11:03:56.924 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:56.919 7fba3dffb700 1 -- 172.21.2.102:0/1093685921 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba40000c90 con 0x7fba4c0b4ad0 2021-02-10T11:03:56.924 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:56.919 7fba3dffb700 1 -- 172.21.2.102:0/1093685921 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fba400022b0 con 0x7fba4c0b4ad0 2021-02-10T11:03:56.924 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:56.919 7fba5381f700 1 -- 172.21.2.102:0/1093685921 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba4c1a79d0 con 0x7fba4c0b4ad0 2021-02-10T11:03:56.924 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:56.919 7fba5381f700 1 -- 172.21.2.102:0/1093685921 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba4c1a79d0 con 0x7fba4c0b4ad0 2021-02-10T11:03:56.924 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:56.920 7fba3dffb700 1 -- 172.21.2.102:0/1093685921 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fba40006b00 con 0x7fba4c0b4ad0 2021-02-10T11:03:56.924 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:56.920 7fba3dffb700 1 --2- 172.21.2.102:0/1093685921 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fba2c0186c0 0x7fba2c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:56.924 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:56.920 7fba3dffb700 1 -- 172.21.2.102:0/1093685921 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fba40032560 con 0x7fba4c0b4ad0 2021-02-10T11:03:56.924 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:56.920 7fba50e3a700 1 --2- 172.21.2.102:0/1093685921 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fba2c0186c0 0x7fba2c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:56.924 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:56.920 7fba50e3a700 1 --2- 172.21.2.102:0/1093685921 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fba2c0186c0 0x7fba2c01abc0 crc :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:03:56.950 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:56.951 7fba5381f700 1 -- 172.21.2.102:0/1093685921 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba4c084a90 con 0x7fba4c0b4ad0 2021-02-10T11:03:56.952 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:56.953 7fba3dffb700 1 -- 172.21.2.102:0/1093685921 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fba40030d10 con 0x7fba4c0b4ad0 2021-02-10T11:03:57.091 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:57.092 7fba5381f700 1 -- 172.21.2.102:0/1093685921 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fba4c07ed80 con 0x7fba4c0b4ad0 2021-02-10T11:03:57.091 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:57.092 7fba3dffb700 1 -- 172.21.2.102:0/1093685921 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fba4000bd60 con 0x7fba4c0b4ad0 2021-02-10T11:03:57.092 INFO:teuthology.orchestra.run.gibba002.stdout:55834574856 2021-02-10T11:03:57.093 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:57.094 7fba5381f700 1 -- 172.21.2.102:0/1093685921 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fba2c0186c0 msgr2=0x7fba2c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:57.093 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:57.094 7fba5381f700 1 --2- 172.21.2.102:0/1093685921 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fba2c0186c0 0x7fba2c01abc0 crc :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:57.093 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:57.094 7fba5381f700 1 -- 172.21.2.102:0/1093685921 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fba4c0b4ad0 msgr2=0x7fba4c0a78d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:57.093 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:57.094 7fba5381f700 1 --2- 172.21.2.102:0/1093685921 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fba4c0b4ad0 0x7fba4c0a78d0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7fba40006f60 tx=0x7fba4000b970).stop 2021-02-10T11:03:57.094 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:57.094 7fba5381f700 1 -- 172.21.2.102:0/1093685921 shutdown_connections 2021-02-10T11:03:57.094 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:57.094 7fba5381f700 1 --2- 172.21.2.102:0/1093685921 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fba4c0a82f0 0x7fba4c1a73b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:03:57.094 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:57.094 7fba5381f700 1 --2- 172.21.2.102:0/1093685921 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fba2c0186c0 0x7fba2c01abc0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:57.094 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:57.094 7fba5381f700 1 --2- 172.21.2.102:0/1093685921 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fba4c0a7ef0 0x7fba4c1a6d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:03:57.094 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:57.095 7fba5381f700 1 --2- 172.21.2.102:0/1093685921 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fba4c0b4ad0 0x7fba4c0a78d0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:57.094 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:57.095 7fba5381f700 1 -- 172.21.2.102:0/1093685921 >> 172.21.2.102:0/1093685921 conn(0x7fba4c0b5560 msgr2=0x7fba4c0a6200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:03:57.095 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:57.095 7fba5381f700 1 -- 172.21.2.102:0/1093685921 shutdown_connections 2021-02-10T11:03:57.095 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:57.095 7fba5381f700 1 -- 172.21.2.102:0/1093685921 wait complete. 2021-02-10T11:03:57.101 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574856 for osd.1 2021-02-10T11:03:57.102 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd last-stat-seq osd.2 2021-02-10T11:03:57.526 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:57.511 7f6016f0e700 1 Processor -- start 2021-02-10T11:03:57.526 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:57.511 7f6016f0e700 1 -- start start 2021-02-10T11:03:57.526 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:57.512 7f6016f0e700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f60100e5070 0x7f60101b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:57.527 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:57.512 7f6016f0e700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f60100e5470 0x7f6010080310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:57.527 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:57.512 7f6016f0e700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f601007ed80 con 0x7f60101b3e40 2021-02-10T11:03:57.527 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:57.512 7f6016f0e700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60100d1790 con 0x7f60100e5470 2021-02-10T11:03:57.527 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:57.512 7f6016f0e700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60100d19b0 con 0x7f60100e5070 2021-02-10T11:03:57.527 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:57.512 7f601552b700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f60100e5470 0x7f6010080310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:57.527 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:57.512 7f601552b700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f60100e5470 0x7f6010080310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:49468/0 (socket says 172.21.2.102:49468) 2021-02-10T11:03:57.527 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:57.512 7f601552b700 1 -- 172.21.2.102:0/3845921022 learned_addr learned my addr 172.21.2.102:0/3845921022 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:03:57.527 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:57.512 7f601552b700 1 -- 172.21.2.102:0/3845921022 >> v2:172.21.2.113:3300/0 conn(0x7f60100e5070 msgr2=0x7f60101b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:03:57.527 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:57.512 7f601552b700 1 --2- 172.21.2.102:0/3845921022 >> v2:172.21.2.113:3300/0 conn(0x7f60100e5070 0x7f60101b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:03:57.527 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:57.512 7f601552b700 1 -- 172.21.2.102:0/3845921022 >> v1:172.21.2.102:6789/0 conn(0x7f60101b3e40 legacy=0x7f601007d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:57.527 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:57.512 7f601552b700 1 -- 172.21.2.102:0/3845921022 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60100d1bd0 con 0x7f60100e5470 2021-02-10T11:03:57.527 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:57.512 7f601552b700 1 --2- 172.21.2.102:0/3845921022 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f60100e5470 0x7f6010080310 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7f6008006b80 tx=0x7f6008009b90).ready entity=mon.0 client_cookie=b790c045223dee13 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:03:57.527 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:57.513 7f600f7fe700 1 -- 172.21.2.102:0/3845921022 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f60080022e0 con 0x7f60100e5470 2021-02-10T11:03:57.528 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:57.513 7f600f7fe700 1 -- 172.21.2.102:0/3845921022 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6008002620 con 0x7f60100e5470 2021-02-10T11:03:57.528 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:57.513 7f600f7fe700 1 -- 172.21.2.102:0/3845921022 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6008002b00 con 0x7f60100e5470 2021-02-10T11:03:57.528 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:57.513 7f6016f0e700 1 -- 172.21.2.102:0/3845921022 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f60100e5470 msgr2=0x7f6010080310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:57.528 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:57.513 7f6016f0e700 1 --2- 172.21.2.102:0/3845921022 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f60100e5470 0x7f6010080310 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7f6008006b80 tx=0x7f6008009b90).stop 2021-02-10T11:03:57.528 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:57.513 7f6016f0e700 1 -- 172.21.2.102:0/3845921022 shutdown_connections 2021-02-10T11:03:57.528 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:57.513 7f6016f0e700 1 --2- 172.21.2.102:0/3845921022 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f60100e5470 0x7f6010080310 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:57.528 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:57.513 7f6016f0e700 1 --2- 172.21.2.102:0/3845921022 >> v2:172.21.2.113:3300/0 conn(0x7f60100e5070 0x7f60101b3820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:03:57.528 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:57.513 7f6016f0e700 1 -- 172.21.2.102:0/3845921022 >> 172.21.2.102:0/3845921022 conn(0x7f60100ce690 msgr2=0x7f60100e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:03:57.528 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:57.513 7f6016f0e700 1 -- 172.21.2.102:0/3845921022 shutdown_connections 2021-02-10T11:03:57.528 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:57.513 7f6016f0e700 1 -- 172.21.2.102:0/3845921022 wait complete. 2021-02-10T11:03:57.529 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:57.528 7f6016f0e700 1 Processor -- start 2021-02-10T11:03:57.530 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:57.528 7f6016f0e700 1 -- start start 2021-02-10T11:03:57.530 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:57.528 7f6016f0e700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f60100be580 0x7f60100be980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:57.530 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:57.528 7f6016f0e700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6010097ed0 0x7f60100982d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:57.530 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:57.528 7f6016f0e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6010099180 0x7f6010099580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:57.530 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:57.528 7f6016f0e700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60100988f0 con 0x7f6010099180 2021-02-10T11:03:57.530 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:57.528 7f6016f0e700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60100c1110 con 0x7f60100be580 2021-02-10T11:03:57.530 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:57.528 7f6016f0e700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6010099ba0 con 0x7f6010097ed0 2021-02-10T11:03:57.530 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:57.528 7f601552b700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6010099180 0x7f6010099580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:57.530 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:57.528 7f601552b700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6010099180 0x7f6010099580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:49472/0 (socket says 172.21.2.102:49472) 2021-02-10T11:03:57.530 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:57.528 7f601552b700 1 -- 172.21.2.102:0/439843154 learned_addr learned my addr 172.21.2.102:0/439843154 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:03:57.530 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:57.529 7f601552b700 1 -- 172.21.2.102:0/439843154 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6010097ed0 msgr2=0x7f60100982d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:57.530 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:57.529 7f601552b700 1 --2- 172.21.2.102:0/439843154 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6010097ed0 0x7f60100982d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:03:57.531 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:57.529 7f601552b700 1 -- 172.21.2.102:0/439843154 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f60100be580 msgr2=0x7f60100be980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:57.531 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:57.529 7f601552b700 1 --2- 172.21.2.102:0/439843154 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f60100be580 0x7f60100be980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:03:57.531 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:57.529 7f601552b700 1 -- 172.21.2.102:0/439843154 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6010099dc0 con 0x7f6010099180 2021-02-10T11:03:57.531 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:57.529 7f601552b700 1 --2- 172.21.2.102:0/439843154 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6010099180 0x7f6010099580 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f6008006f10 tx=0x7f600800b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:03:57.531 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:57.529 7f600d7fa700 1 -- 172.21.2.102:0/439843154 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6008000b90 con 0x7f6010099180 2021-02-10T11:03:57.531 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:57.529 7f600d7fa700 1 -- 172.21.2.102:0/439843154 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f600800bd50 con 0x7f6010099180 2021-02-10T11:03:57.531 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:57.529 7f6016f0e700 1 -- 172.21.2.102:0/439843154 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6010099dc0 con 0x7f6010099180 2021-02-10T11:03:57.531 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:57.529 7f600d7fa700 1 -- 172.21.2.102:0/439843154 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6008002300 con 0x7f6010099180 2021-02-10T11:03:57.531 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:57.529 7f6016f0e700 1 -- 172.21.2.102:0/439843154 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6010099dc0 con 0x7f6010099180 2021-02-10T11:03:57.531 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:57.530 7f600d7fa700 1 -- 172.21.2.102:0/439843154 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f6008009100 con 0x7f6010099180 2021-02-10T11:03:57.531 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:57.530 7f600d7fa700 1 --2- 172.21.2.102:0/439843154 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5ff4018730 0x7f5ff401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:57.531 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:57.530 7f600d7fa700 1 -- 172.21.2.102:0/439843154 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f6008032630 con 0x7f6010099180 2021-02-10T11:03:57.531 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:57.530 7f6014d2a700 1 --2- 172.21.2.102:0/439843154 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5ff4018730 0x7f5ff401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:57.531 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:57.530 7f6014d2a700 1 --2- 172.21.2.102:0/439843154 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5ff4018730 0x7f5ff401abc0 crc :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:03:57.561 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:57.562 7f6016f0e700 1 -- 172.21.2.102:0/439843154 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6010098c50 con 0x7f6010099180 2021-02-10T11:03:57.563 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:57.564 7f600d7fa700 1 -- 172.21.2.102:0/439843154 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f6008000b90 con 0x7f6010099180 2021-02-10T11:03:57.704 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:57.705 7f6016f0e700 1 -- 172.21.2.102:0/439843154 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f6010098c50 con 0x7f6010099180 2021-02-10T11:03:57.705 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:57.706 7f600d7fa700 1 -- 172.21.2.102:0/439843154 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f600800bd50 con 0x7f6010099180 2021-02-10T11:03:57.705 INFO:teuthology.orchestra.run.gibba002.stdout:55834574855 2021-02-10T11:03:57.706 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:57.707 7f6016f0e700 1 -- 172.21.2.102:0/439843154 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5ff4018730 msgr2=0x7f5ff401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:57.707 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:57.707 7f6016f0e700 1 --2- 172.21.2.102:0/439843154 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5ff4018730 0x7f5ff401abc0 crc :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:57.707 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:57.707 7f6016f0e700 1 -- 172.21.2.102:0/439843154 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6010099180 msgr2=0x7f6010099580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:57.707 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:57.707 7f6016f0e700 1 --2- 172.21.2.102:0/439843154 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6010099180 0x7f6010099580 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f6008006f10 tx=0x7f600800b960).stop 2021-02-10T11:03:57.707 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:57.708 7f6016f0e700 1 -- 172.21.2.102:0/439843154 shutdown_connections 2021-02-10T11:03:57.707 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:57.708 7f6016f0e700 1 --2- 172.21.2.102:0/439843154 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6010099180 0x7f6010099580 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:57.707 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:57.708 7f6016f0e700 1 --2- 172.21.2.102:0/439843154 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6010097ed0 0x7f60100982d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:03:57.707 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:57.708 7f6016f0e700 1 --2- 172.21.2.102:0/439843154 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5ff4018730 0x7f5ff401abc0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:57.707 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:57.708 7f6016f0e700 1 --2- 172.21.2.102:0/439843154 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f60100be580 0x7f60100be980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:03:57.707 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:57.708 7f6016f0e700 1 -- 172.21.2.102:0/439843154 >> 172.21.2.102:0/439843154 conn(0x7f60100bde70 msgr2=0x7f60100c3970 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:03:57.708 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:57.708 7f6016f0e700 1 -- 172.21.2.102:0/439843154 shutdown_connections 2021-02-10T11:03:57.708 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:57.708 7f6016f0e700 1 -- 172.21.2.102:0/439843154 wait complete. 2021-02-10T11:03:57.715 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574855 for osd.2 2021-02-10T11:03:57.715 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd last-stat-seq osd.3 2021-02-10T11:03:58.142 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:58.127 7fee06cb3700 1 Processor -- start 2021-02-10T11:03:58.142 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:58.127 7fee06cb3700 1 -- start start 2021-02-10T11:03:58.142 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:58.127 7fee06cb3700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fee000e5470 0x7fee0007e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:58.143 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:58.127 7fee06cb3700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fee0007f300 0x7fee0007f700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:58.143 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:58.127 7fee06cb3700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee0007ed80 con 0x7fee000e5070 2021-02-10T11:03:58.143 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:58.127 7fee06cb3700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee001b4010 con 0x7fee0007f300 2021-02-10T11:03:58.143 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:58.127 7fee06cb3700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee000d1790 con 0x7fee000e5470 2021-02-10T11:03:58.143 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:58.127 7fee052d0700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fee0007f300 0x7fee0007f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:58.143 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:58.128 7fee052d0700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fee0007f300 0x7fee0007f700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:49478/0 (socket says 172.21.2.102:49478) 2021-02-10T11:03:58.143 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:58.128 7fee052d0700 1 -- 172.21.2.102:0/674246296 learned_addr learned my addr 172.21.2.102:0/674246296 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:03:58.143 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:58.128 7fee052d0700 1 -- 172.21.2.102:0/674246296 >> v2:172.21.2.113:3300/0 conn(0x7fee000e5470 msgr2=0x7fee0007e6a0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:03:58.143 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:58.128 7fee052d0700 1 --2- 172.21.2.102:0/674246296 >> v2:172.21.2.113:3300/0 conn(0x7fee000e5470 0x7fee0007e6a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:03:58.143 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:58.128 7fee052d0700 1 -- 172.21.2.102:0/674246296 >> v1:172.21.2.113:6789/0 conn(0x7fee000e5070 legacy=0x7fee001b3820 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:03:58.143 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:58.128 7fee052d0700 1 -- 172.21.2.102:0/674246296 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee000d19b0 con 0x7fee0007f300 2021-02-10T11:03:58.143 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:58.128 7fee052d0700 1 --2- 172.21.2.102:0/674246296 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fee0007f300 0x7fee0007f700 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7fedf8006b80 tx=0x7fedf8009b90).ready entity=mon.0 client_cookie=bcd7a621f05c3ab9 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:03:58.143 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:58.128 7fedff7fe700 1 -- 172.21.2.102:0/674246296 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fedf80022e0 con 0x7fee0007f300 2021-02-10T11:03:58.144 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:58.128 7fedff7fe700 1 -- 172.21.2.102:0/674246296 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedf8002560 con 0x7fee0007f300 2021-02-10T11:03:58.144 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:58.128 7fedff7fe700 1 -- 172.21.2.102:0/674246296 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fedf80029c0 con 0x7fee0007f300 2021-02-10T11:03:58.144 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:58.128 7fee06cb3700 1 -- 172.21.2.102:0/674246296 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fee0007f300 msgr2=0x7fee0007f700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:58.144 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:58.128 7fee06cb3700 1 --2- 172.21.2.102:0/674246296 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fee0007f300 0x7fee0007f700 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7fedf8006b80 tx=0x7fedf8009b90).stop 2021-02-10T11:03:58.144 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:58.129 7fee06cb3700 1 -- 172.21.2.102:0/674246296 shutdown_connections 2021-02-10T11:03:58.144 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:58.129 7fee06cb3700 1 --2- 172.21.2.102:0/674246296 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fee0007f300 0x7fee0007f700 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:58.144 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:58.129 7fee06cb3700 1 --2- 172.21.2.102:0/674246296 >> v2:172.21.2.113:3300/0 conn(0x7fee000e5470 0x7fee0007e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:03:58.144 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:58.129 7fee06cb3700 1 -- 172.21.2.102:0/674246296 >> 172.21.2.102:0/674246296 conn(0x7fee000ce690 msgr2=0x7fee000e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:03:58.144 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:58.129 7fee06cb3700 1 -- 172.21.2.102:0/674246296 shutdown_connections 2021-02-10T11:03:58.144 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:58.129 7fee06cb3700 1 -- 172.21.2.102:0/674246296 wait complete. 2021-02-10T11:03:58.146 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:58.143 7fee06cb3700 1 Processor -- start 2021-02-10T11:03:58.146 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:58.144 7fee06cb3700 1 -- start start 2021-02-10T11:03:58.146 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:58.144 7fee06cb3700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fee000a2b70 0x7fee000c0590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:58.146 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:58.144 7fee06cb3700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fee000c0bb0 0x7fee000a3850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:58.147 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:58.144 7fee06cb3700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fee000c0fb0 0x7fee000a3e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:58.147 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:58.144 7fee06cb3700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee000a2f70 con 0x7fee000c0bb0 2021-02-10T11:03:58.147 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:58.144 7fee06cb3700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee000a1a40 con 0x7fee000c0fb0 2021-02-10T11:03:58.147 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:58.144 7fee06cb3700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee000a1c60 con 0x7fee000a2b70 2021-02-10T11:03:58.147 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:58.144 7fedfffff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fee000c0bb0 0x7fee000a3850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:58.147 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:58.144 7fedfffff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fee000c0bb0 0x7fee000a3850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:49482/0 (socket says 172.21.2.102:49482) 2021-02-10T11:03:58.147 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:58.144 7fedfffff700 1 -- 172.21.2.102:0/2993179704 learned_addr learned my addr 172.21.2.102:0/2993179704 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:03:58.147 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:58.145 7fedfffff700 1 -- 172.21.2.102:0/2993179704 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fee000a2b70 msgr2=0x7fee000c0590 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:03:58.147 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:58.145 7fedfffff700 1 --2- 172.21.2.102:0/2993179704 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fee000a2b70 0x7fee000c0590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:03:58.147 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:58.145 7fedfffff700 1 -- 172.21.2.102:0/2993179704 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fee000c0fb0 msgr2=0x7fee000a3e70 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:03:58.147 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:58.145 7fedfffff700 1 --2- 172.21.2.102:0/2993179704 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fee000c0fb0 0x7fee000a3e70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:03:58.147 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:58.145 7fedfffff700 1 -- 172.21.2.102:0/2993179704 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee000a4490 con 0x7fee000c0bb0 2021-02-10T11:03:58.147 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:58.145 7fedfffff700 1 --2- 172.21.2.102:0/2993179704 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fee000c0bb0 0x7fee000a3850 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7fedf0009960 tx=0x7fedf000bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:03:58.148 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:58.145 7fedfd7fa700 1 -- 172.21.2.102:0/2993179704 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fedf0000d30 con 0x7fee000c0bb0 2021-02-10T11:03:58.148 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:58.145 7fedfd7fa700 1 -- 172.21.2.102:0/2993179704 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedf0003030 con 0x7fee000c0bb0 2021-02-10T11:03:58.148 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:58.145 7fedfd7fa700 1 -- 172.21.2.102:0/2993179704 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fedf0001590 con 0x7fee000c0bb0 2021-02-10T11:03:58.148 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:58.145 7fee06cb3700 1 -- 172.21.2.102:0/2993179704 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee000a4490 con 0x7fee000c0bb0 2021-02-10T11:03:58.148 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:58.145 7fee06cb3700 1 -- 172.21.2.102:0/2993179704 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee000a4490 con 0x7fee000c0bb0 2021-02-10T11:03:58.148 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:58.146 7fedfd7fa700 1 -- 172.21.2.102:0/2993179704 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fedf0001310 con 0x7fee000c0bb0 2021-02-10T11:03:58.148 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:58.146 7fedfd7fa700 1 --2- 172.21.2.102:0/2993179704 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fede40186c0 0x7fede401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:58.148 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:58.146 7fedfd7fa700 1 -- 172.21.2.102:0/2993179704 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fedf00328f0 con 0x7fee000c0bb0 2021-02-10T11:03:58.148 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:58.146 7fee04acf700 1 --2- 172.21.2.102:0/2993179704 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fede40186c0 0x7fede401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:58.148 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:58.146 7fee04acf700 1 --2- 172.21.2.102:0/2993179704 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fede40186c0 0x7fede401abc0 crc :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:03:58.176 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:58.177 7fee06cb3700 1 -- 172.21.2.102:0/2993179704 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee000a2f70 con 0x7fee000c0bb0 2021-02-10T11:03:58.179 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:58.180 7fedfd7fa700 1 -- 172.21.2.102:0/2993179704 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fedf0000d30 con 0x7fee000c0bb0 2021-02-10T11:03:58.319 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:58.319 7fee06cb3700 1 -- 172.21.2.102:0/2993179704 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7fee000a2f70 con 0x7fee000c0bb0 2021-02-10T11:03:58.319 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:58.320 7fedfd7fa700 1 -- 172.21.2.102:0/2993179704 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fedf0001310 con 0x7fee000c0bb0 2021-02-10T11:03:58.320 INFO:teuthology.orchestra.run.gibba002.stdout:55834574855 2021-02-10T11:03:58.321 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:58.321 7fee06cb3700 1 -- 172.21.2.102:0/2993179704 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fede40186c0 msgr2=0x7fede401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:58.321 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:58.321 7fee06cb3700 1 --2- 172.21.2.102:0/2993179704 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fede40186c0 0x7fede401abc0 crc :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:58.321 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:58.321 7fee06cb3700 1 -- 172.21.2.102:0/2993179704 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fee000c0bb0 msgr2=0x7fee000a3850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:58.322 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:58.321 7fee06cb3700 1 --2- 172.21.2.102:0/2993179704 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fee000c0bb0 0x7fee000a3850 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7fedf0009960 tx=0x7fedf000bcf0).stop 2021-02-10T11:03:58.322 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:58.322 7fee06cb3700 1 -- 172.21.2.102:0/2993179704 shutdown_connections 2021-02-10T11:03:58.322 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:58.322 7fee06cb3700 1 --2- 172.21.2.102:0/2993179704 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fede40186c0 0x7fede401abc0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:58.322 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:58.322 7fee06cb3700 1 --2- 172.21.2.102:0/2993179704 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fee000c0fb0 0x7fee000a3e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:03:58.322 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:58.322 7fee06cb3700 1 --2- 172.21.2.102:0/2993179704 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fee000c0bb0 0x7fee000a3850 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:58.322 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:58.322 7fee06cb3700 1 --2- 172.21.2.102:0/2993179704 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fee000a2b70 0x7fee000c0590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:03:58.322 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:58.322 7fee06cb3700 1 -- 172.21.2.102:0/2993179704 >> 172.21.2.102:0/2993179704 conn(0x7fee000bde70 msgr2=0x7fee000c3970 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:03:58.323 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:58.322 7fee06cb3700 1 -- 172.21.2.102:0/2993179704 shutdown_connections 2021-02-10T11:03:58.323 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:58.322 7fee06cb3700 1 -- 172.21.2.102:0/2993179704 wait complete. 2021-02-10T11:03:58.330 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.3 2021-02-10T11:03:58.330 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd last-stat-seq osd.4 2021-02-10T11:03:58.758 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:58.742 7f9e542d6700 1 Processor -- start 2021-02-10T11:03:58.759 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:58.742 7f9e542d6700 1 -- start start 2021-02-10T11:03:58.759 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:58.743 7f9e542d6700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f9e4c0e5070 0x7f9e4c1b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:58.759 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:58.743 7f9e542d6700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9e4c1b3e40 0x7f9e4c07d690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:58.759 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:58.743 7f9e542d6700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e4c07ed80 con 0x7f9e4c0e5470 2021-02-10T11:03:58.759 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:58.743 7f9e542d6700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e4c080310 con 0x7f9e4c0e5070 2021-02-10T11:03:58.759 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:58.743 7f9e542d6700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e4c080500 con 0x7f9e4c1b3e40 2021-02-10T11:03:58.759 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:58.743 7f9e520f2700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f9e4c0e5070 0x7f9e4c1b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:58.759 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:58.743 7f9e520f2700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f9e4c0e5070 0x7f9e4c1b3820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:49490/0 (socket says 172.21.2.102:49490) 2021-02-10T11:03:58.759 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:58.743 7f9e520f2700 1 -- 172.21.2.102:0/377671121 learned_addr learned my addr 172.21.2.102:0/377671121 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:03:58.759 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:58.743 7f9e518f1700 1 --2- 172.21.2.102:0/377671121 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9e4c1b3e40 0x7f9e4c07d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:58.759 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:58.743 7f9e520f2700 1 -- 172.21.2.102:0/377671121 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9e4c1b3e40 msgr2=0x7f9e4c07d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:03:58.759 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:58.743 7f9e520f2700 1 --2- 172.21.2.102:0/377671121 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9e4c1b3e40 0x7f9e4c07d690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:58.760 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:58.744 7f9e520f2700 1 -- 172.21.2.102:0/377671121 >> v1:172.21.2.113:6789/0 conn(0x7f9e4c0e5470 legacy=0x7f9e4c07dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:03:58.760 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:58.744 7f9e520f2700 1 -- 172.21.2.102:0/377671121 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e4c0806f0 con 0x7f9e4c0e5070 2021-02-10T11:03:58.760 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:58.744 7f9e520f2700 1 --2- 172.21.2.102:0/377671121 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9e4c0e5070 0x7f9e4c1b3820 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7f9e48006700 tx=0x7f9e48009b80).ready entity=mon.0 client_cookie=9f82f7582070383f server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:03:58.760 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:58.744 7f9e510f0700 1 -- 172.21.2.102:0/377671121 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9e4800c0c0 con 0x7f9e4c0e5070 2021-02-10T11:03:58.760 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:58.744 7f9e510f0700 1 -- 172.21.2.102:0/377671121 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e4800c340 con 0x7f9e4c0e5070 2021-02-10T11:03:58.760 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:58.744 7f9e510f0700 1 -- 172.21.2.102:0/377671121 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9e4800c860 con 0x7f9e4c0e5070 2021-02-10T11:03:58.760 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:58.744 7f9e542d6700 1 -- 172.21.2.102:0/377671121 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9e4c0e5070 msgr2=0x7f9e4c1b3820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:58.760 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:58.744 7f9e542d6700 1 --2- 172.21.2.102:0/377671121 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9e4c0e5070 0x7f9e4c1b3820 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7f9e48006700 tx=0x7f9e48009b80).stop 2021-02-10T11:03:58.760 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:58.745 7f9e542d6700 1 -- 172.21.2.102:0/377671121 shutdown_connections 2021-02-10T11:03:58.760 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:58.745 7f9e542d6700 1 --2- 172.21.2.102:0/377671121 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9e4c1b3e40 0x7f9e4c07d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:58.760 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:58.745 7f9e542d6700 1 --2- 172.21.2.102:0/377671121 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9e4c0e5070 0x7f9e4c1b3820 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:58.760 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:58.745 7f9e542d6700 1 -- 172.21.2.102:0/377671121 >> 172.21.2.102:0/377671121 conn(0x7f9e4c0ce690 msgr2=0x7f9e4c0e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:03:58.761 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:58.745 7f9e542d6700 1 -- 172.21.2.102:0/377671121 shutdown_connections 2021-02-10T11:03:58.761 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:58.745 7f9e542d6700 1 -- 172.21.2.102:0/377671121 wait complete. 2021-02-10T11:03:58.762 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:58.760 7f9e542d6700 1 Processor -- start 2021-02-10T11:03:58.762 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:58.760 7f9e542d6700 1 -- start start 2021-02-10T11:03:58.762 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:58.760 7f9e542d6700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9e4c1a43e0 0x7f9e4c0b5180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:58.762 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:58.760 7f9e542d6700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9e4c0b57a0 0x7f9e4c0b5ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:58.762 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:58.760 7f9e542d6700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f9e4c0b61c0 0x7f9e4c0a5420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:58.762 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:58.760 7f9e542d6700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e4c0b4c00 con 0x7f9e4c1a43e0 2021-02-10T11:03:58.763 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:58.760 7f9e542d6700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e4c0a5a40 con 0x7f9e4c0b61c0 2021-02-10T11:03:58.763 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:58.760 7f9e542d6700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e4c0a5c20 con 0x7f9e4c0b57a0 2021-02-10T11:03:58.763 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:58.760 7f9e520f2700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9e4c1a43e0 0x7f9e4c0b5180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:58.763 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:58.760 7f9e520f2700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9e4c1a43e0 0x7f9e4c0b5180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:49494/0 (socket says 172.21.2.102:49494) 2021-02-10T11:03:58.763 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:58.760 7f9e520f2700 1 -- 172.21.2.102:0/1097173369 learned_addr learned my addr 172.21.2.102:0/1097173369 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:03:58.763 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:58.761 7f9e520f2700 1 -- 172.21.2.102:0/1097173369 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9e4c0b57a0 msgr2=0x7f9e4c0b5ba0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:03:58.763 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:58.761 7f9e520f2700 1 --2- 172.21.2.102:0/1097173369 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9e4c0b57a0 0x7f9e4c0b5ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:03:58.763 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:58.761 7f9e520f2700 1 -- 172.21.2.102:0/1097173369 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f9e4c0b61c0 msgr2=0x7f9e4c0a5420 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:03:58.763 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:58.761 7f9e520f2700 1 --2- 172.21.2.102:0/1097173369 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f9e4c0b61c0 0x7f9e4c0a5420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:03:58.763 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:58.761 7f9e520f2700 1 -- 172.21.2.102:0/1097173369 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e4c0a5e40 con 0x7f9e4c1a43e0 2021-02-10T11:03:58.764 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:58.761 7f9e520f2700 1 --2- 172.21.2.102:0/1097173369 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9e4c1a43e0 0x7f9e4c0b5180 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f9e4800b580 tx=0x7f9e4800b920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:03:58.764 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:58.761 7f9e42ffd700 1 -- 172.21.2.102:0/1097173369 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9e4800bd10 con 0x7f9e4c1a43e0 2021-02-10T11:03:58.764 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:58.761 7f9e42ffd700 1 -- 172.21.2.102:0/1097173369 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e48002030 con 0x7f9e4c1a43e0 2021-02-10T11:03:58.764 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:58.761 7f9e42ffd700 1 -- 172.21.2.102:0/1097173369 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9e48002550 con 0x7f9e4c1a43e0 2021-02-10T11:03:58.764 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:58.761 7f9e542d6700 1 -- 172.21.2.102:0/1097173369 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e4c0a5e40 con 0x7f9e4c1a43e0 2021-02-10T11:03:58.764 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:58.761 7f9e542d6700 1 -- 172.21.2.102:0/1097173369 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e4c0a5e40 con 0x7f9e4c1a43e0 2021-02-10T11:03:58.764 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:58.762 7f9e42ffd700 1 -- 172.21.2.102:0/1097173369 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f9e48000c90 con 0x7f9e4c1a43e0 2021-02-10T11:03:58.764 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:58.762 7f9e42ffd700 1 --2- 172.21.2.102:0/1097173369 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f9e340186c0 0x7f9e3401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:58.764 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:58.762 7f9e42ffd700 1 -- 172.21.2.102:0/1097173369 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f9e480325c0 con 0x7f9e4c1a43e0 2021-02-10T11:03:58.764 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:58.762 7f9e518f1700 1 --2- 172.21.2.102:0/1097173369 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f9e340186c0 0x7f9e3401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:58.764 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:58.762 7f9e518f1700 1 --2- 172.21.2.102:0/1097173369 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f9e340186c0 0x7f9e3401abc0 crc :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:03:58.790 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:58.792 7f9e542d6700 1 -- 172.21.2.102:0/1097173369 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e4c084a90 con 0x7f9e4c1a43e0 2021-02-10T11:03:58.793 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:58.794 7f9e42ffd700 1 -- 172.21.2.102:0/1097173369 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f9e48030d70 con 0x7f9e4c1a43e0 2021-02-10T11:03:58.934 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:58.935 7f9e542d6700 1 -- 172.21.2.102:0/1097173369 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f9e4c07ed80 con 0x7f9e4c1a43e0 2021-02-10T11:03:58.934 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:58.935 7f9e42ffd700 1 -- 172.21.2.102:0/1097173369 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f9e48030d70 con 0x7f9e4c1a43e0 2021-02-10T11:03:58.935 INFO:teuthology.orchestra.run.gibba002.stdout:55834574856 2021-02-10T11:03:58.936 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:58.937 7f9e542d6700 1 -- 172.21.2.102:0/1097173369 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f9e340186c0 msgr2=0x7f9e3401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:58.936 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:58.937 7f9e542d6700 1 --2- 172.21.2.102:0/1097173369 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f9e340186c0 0x7f9e3401abc0 crc :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:58.936 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:58.937 7f9e542d6700 1 -- 172.21.2.102:0/1097173369 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9e4c1a43e0 msgr2=0x7f9e4c0b5180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:58.936 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:58.937 7f9e542d6700 1 --2- 172.21.2.102:0/1097173369 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9e4c1a43e0 0x7f9e4c0b5180 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f9e4800b580 tx=0x7f9e4800b920).stop 2021-02-10T11:03:58.936 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:58.937 7f9e542d6700 1 -- 172.21.2.102:0/1097173369 shutdown_connections 2021-02-10T11:03:58.937 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:58.937 7f9e542d6700 1 --2- 172.21.2.102:0/1097173369 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f9e340186c0 0x7f9e3401abc0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:58.937 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:58.937 7f9e542d6700 1 --2- 172.21.2.102:0/1097173369 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f9e4c0b61c0 0x7f9e4c0a5420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:03:58.937 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:58.937 7f9e542d6700 1 --2- 172.21.2.102:0/1097173369 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9e4c0b57a0 0x7f9e4c0b5ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:03:58.937 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:58.937 7f9e542d6700 1 --2- 172.21.2.102:0/1097173369 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9e4c1a43e0 0x7f9e4c0b5180 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:58.937 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:58.937 7f9e542d6700 1 -- 172.21.2.102:0/1097173369 >> 172.21.2.102:0/1097173369 conn(0x7f9e4c0beb60 msgr2=0x7f9e4c091c60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:03:58.937 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:58.937 7f9e542d6700 1 -- 172.21.2.102:0/1097173369 shutdown_connections 2021-02-10T11:03:58.937 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:58.937 7f9e542d6700 1 -- 172.21.2.102:0/1097173369 wait complete. 2021-02-10T11:03:58.944 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574856 for osd.4 2021-02-10T11:03:58.944 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd last-stat-seq osd.5 2021-02-10T11:03:59.371 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:59.356 7fe361b38700 1 Processor -- start 2021-02-10T11:03:59.371 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:59.356 7fe361b38700 1 -- start start 2021-02-10T11:03:59.372 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:59.356 7fe361b38700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe35c0e5470 0x7fe35c07e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:59.372 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:59.356 7fe361b38700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe35c07ed80 con 0x7fe35c07f300 2021-02-10T11:03:59.372 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:59.356 7fe361b38700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe35c1b4010 con 0x7fe35c0e5070 2021-02-10T11:03:59.372 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:59.356 7fe361b38700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe35c0d1790 con 0x7fe35c0e5470 2021-02-10T11:03:59.372 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:59.356 7fe35bfff700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fe35c07f300 0x7fe35c07f700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:41726/0 (socket says 172.21.2.102:41726) 2021-02-10T11:03:59.372 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:59.356 7fe35bfff700 1 -- 172.21.2.102:0/2293742889 learned_addr learned my addr 172.21.2.102:0/2293742889 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:03:59.372 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:59.357 7fe35affd700 1 --2- 172.21.2.102:0/2293742889 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe35c0e5470 0x7fe35c07e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:59.372 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:59.357 7fe35a7fc700 1 -- 172.21.2.102:0/2293742889 <== mon.1 v1:172.21.2.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3758785203 0 0) 0x7fe34c000ff0 con 0x7fe35c0e5070 2021-02-10T11:03:59.372 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:59.357 7fe35a7fc700 1 -- 172.21.2.102:0/2293742889 --> v1:172.21.2.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe344002fe0 con 0x7fe35c0e5070 2021-02-10T11:03:59.372 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:59.357 7fe35a7fc700 1 -- 172.21.2.102:0/2293742889 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3635367066 0 0) 0x7fe354001500 con 0x7fe35c07f300 2021-02-10T11:03:59.372 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:59.357 7fe35a7fc700 1 -- 172.21.2.102:0/2293742889 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe344004370 con 0x7fe35c07f300 2021-02-10T11:03:59.372 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:59.357 7fe35affd700 1 -- 172.21.2.102:0/2293742889 >> v1:172.21.2.113:6789/0 conn(0x7fe35c0e5070 legacy=0x7fe35c1b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:59.373 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:59.357 7fe35affd700 1 -- 172.21.2.102:0/2293742889 >> v1:172.21.2.102:6789/0 conn(0x7fe35c07f300 legacy=0x7fe35c07f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:59.373 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:59.357 7fe35affd700 1 -- 172.21.2.102:0/2293742889 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe35c0d1980 con 0x7fe35c0e5470 2021-02-10T11:03:59.373 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:59.357 7fe35affd700 1 --2- 172.21.2.102:0/2293742889 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe35c0e5470 0x7fe35c07e6a0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7fe350006650 tx=0x7fe350009a80).ready entity=mon.2 client_cookie=7d4e2d2b39b819da server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:03:59.373 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:59.358 7fe35a7fc700 1 -- 172.21.2.102:0/2293742889 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe35000c0c0 con 0x7fe35c0e5470 2021-02-10T11:03:59.373 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:59.358 7fe35a7fc700 1 -- 172.21.2.102:0/2293742889 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe35000c340 con 0x7fe35c0e5470 2021-02-10T11:03:59.373 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:59.358 7fe35a7fc700 1 -- 172.21.2.102:0/2293742889 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe3500009b0 con 0x7fe35c0e5470 2021-02-10T11:03:59.373 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:59.358 7fe361b38700 1 -- 172.21.2.102:0/2293742889 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe35c0e5470 msgr2=0x7fe35c07e6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:59.373 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:59.358 7fe361b38700 1 --2- 172.21.2.102:0/2293742889 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe35c0e5470 0x7fe35c07e6a0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7fe350006650 tx=0x7fe350009a80).stop 2021-02-10T11:03:59.373 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:59.358 7fe361b38700 1 -- 172.21.2.102:0/2293742889 shutdown_connections 2021-02-10T11:03:59.373 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:59.358 7fe361b38700 1 --2- 172.21.2.102:0/2293742889 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe35c0e5470 0x7fe35c07e6a0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:59.373 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:59.358 7fe361b38700 1 -- 172.21.2.102:0/2293742889 >> 172.21.2.102:0/2293742889 conn(0x7fe35c0ce690 msgr2=0x7fe35c0e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:03:59.373 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:59.358 7fe361b38700 1 -- 172.21.2.102:0/2293742889 shutdown_connections 2021-02-10T11:03:59.374 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:59.358 7fe361b38700 1 -- 172.21.2.102:0/2293742889 wait complete. 2021-02-10T11:03:59.375 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:59.373 7fe361b38700 1 Processor -- start 2021-02-10T11:03:59.375 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:59.373 7fe361b38700 1 -- start start 2021-02-10T11:03:59.375 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:59.373 7fe361b38700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe35c0be890 0x7fe35c087500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:59.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:59.373 7fe361b38700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe35c0bf300 0x7fe35c09fc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:59.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:59.373 7fe361b38700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe35c07f220 0x7fe35c07f620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:59.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:59.373 7fe361b38700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe35c0a0920 con 0x7fe35c0bf300 2021-02-10T11:03:59.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:59.374 7fe361b38700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe35c0bed80 con 0x7fe35c07f220 2021-02-10T11:03:59.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:59.374 7fe361b38700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe35c0bf700 con 0x7fe35c0be890 2021-02-10T11:03:59.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:59.374 7fe35affd700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe35c0bf300 0x7fe35c09fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:59.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:59.374 7fe35affd700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe35c0bf300 0x7fe35c09fc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:49510/0 (socket says 172.21.2.102:49510) 2021-02-10T11:03:59.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:59.374 7fe35affd700 1 -- 172.21.2.102:0/3024935523 learned_addr learned my addr 172.21.2.102:0/3024935523 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:03:59.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:59.374 7fe35affd700 1 -- 172.21.2.102:0/3024935523 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe35c0be890 msgr2=0x7fe35c087500 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:03:59.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:59.374 7fe35affd700 1 --2- 172.21.2.102:0/3024935523 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe35c0be890 0x7fe35c087500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:03:59.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:59.374 7fe35affd700 1 -- 172.21.2.102:0/3024935523 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe35c07f220 msgr2=0x7fe35c07f620 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:03:59.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:59.374 7fe35affd700 1 --2- 172.21.2.102:0/3024935523 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe35c07f220 0x7fe35c07f620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:03:59.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:59.374 7fe35affd700 1 -- 172.21.2.102:0/3024935523 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe35c0a0480 con 0x7fe35c0bf300 2021-02-10T11:03:59.377 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:59.374 7fe35affd700 1 --2- 172.21.2.102:0/3024935523 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe35c0bf300 0x7fe35c09fc70 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7fe350009650 tx=0x7fe35000b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:03:59.377 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:59.374 7fe343fff700 1 -- 172.21.2.102:0/3024935523 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe3500093a0 con 0x7fe35c0bf300 2021-02-10T11:03:59.377 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:59.375 7fe343fff700 1 -- 172.21.2.102:0/3024935523 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe35000bd50 con 0x7fe35c0bf300 2021-02-10T11:03:59.377 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:59.375 7fe361b38700 1 -- 172.21.2.102:0/3024935523 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe35c0a0480 con 0x7fe35c0bf300 2021-02-10T11:03:59.377 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:59.375 7fe343fff700 1 -- 172.21.2.102:0/3024935523 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe3500052e0 con 0x7fe35c0bf300 2021-02-10T11:03:59.377 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:59.375 7fe361b38700 1 -- 172.21.2.102:0/3024935523 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe35c197760 con 0x7fe35c0bf300 2021-02-10T11:03:59.377 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:59.375 7fe343fff700 1 -- 172.21.2.102:0/3024935523 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fe350006030 con 0x7fe35c0bf300 2021-02-10T11:03:59.377 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:59.375 7fe343fff700 1 --2- 172.21.2.102:0/3024935523 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe33c018730 0x7fe33c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:59.377 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:59.375 7fe343fff700 1 -- 172.21.2.102:0/3024935523 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fe350032820 con 0x7fe35c0bf300 2021-02-10T11:03:59.377 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:59.375 7fe35b7fe700 1 --2- 172.21.2.102:0/3024935523 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe33c018730 0x7fe33c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:59.377 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:59.376 7fe35b7fe700 1 --2- 172.21.2.102:0/3024935523 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe33c018730 0x7fe33c01abc0 crc :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:03:59.406 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:59.406 7fe361b38700 1 -- 172.21.2.102:0/3024935523 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe35c107180 con 0x7fe35c0bf300 2021-02-10T11:03:59.408 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:59.409 7fe343fff700 1 -- 172.21.2.102:0/3024935523 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fe350009360 con 0x7fe35c0bf300 2021-02-10T11:03:59.545 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:59.546 7fe361b38700 1 -- 172.21.2.102:0/3024935523 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7fe35c07ed80 con 0x7fe35c0bf300 2021-02-10T11:03:59.545 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:59.546 7fe343fff700 1 -- 172.21.2.102:0/3024935523 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fe350006030 con 0x7fe35c0bf300 2021-02-10T11:03:59.546 INFO:teuthology.orchestra.run.gibba002.stdout:55834574856 2021-02-10T11:03:59.547 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:59.548 7fe361b38700 1 -- 172.21.2.102:0/3024935523 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe33c018730 msgr2=0x7fe33c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:59.547 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:59.548 7fe361b38700 1 --2- 172.21.2.102:0/3024935523 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe33c018730 0x7fe33c01abc0 crc :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:59.547 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:59.548 7fe361b38700 1 -- 172.21.2.102:0/3024935523 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe35c0bf300 msgr2=0x7fe35c09fc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:59.547 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:59.548 7fe361b38700 1 --2- 172.21.2.102:0/3024935523 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe35c0bf300 0x7fe35c09fc70 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7fe350009650 tx=0x7fe35000b960).stop 2021-02-10T11:03:59.548 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:59.548 7fe361b38700 1 -- 172.21.2.102:0/3024935523 shutdown_connections 2021-02-10T11:03:59.548 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:59.548 7fe361b38700 1 --2- 172.21.2.102:0/3024935523 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe33c018730 0x7fe33c01abc0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:59.548 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:59.548 7fe361b38700 1 --2- 172.21.2.102:0/3024935523 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe35c07f220 0x7fe35c07f620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:03:59.548 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:59.548 7fe361b38700 1 --2- 172.21.2.102:0/3024935523 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe35c0bf300 0x7fe35c09fc70 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:59.548 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:59.548 7fe361b38700 1 --2- 172.21.2.102:0/3024935523 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe35c0be890 0x7fe35c087500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:03:59.548 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:59.548 7fe361b38700 1 -- 172.21.2.102:0/3024935523 >> 172.21.2.102:0/3024935523 conn(0x7fe35c0c0d40 msgr2=0x7fe35c1073b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:03:59.549 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:59.549 7fe361b38700 1 -- 172.21.2.102:0/3024935523 shutdown_connections 2021-02-10T11:03:59.549 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:59.549 7fe361b38700 1 -- 172.21.2.102:0/3024935523 wait complete. 2021-02-10T11:03:59.556 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574856 for osd.5 2021-02-10T11:03:59.556 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd last-stat-seq osd.6 2021-02-10T11:03:59.980 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:59.965 7faab5c28700 1 Processor -- start 2021-02-10T11:03:59.981 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:59.965 7faab5c28700 1 -- start start 2021-02-10T11:03:59.981 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:59.965 7faab5c28700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7faab00e5070 0x7faab01b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:59.981 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:59.965 7faab5c28700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7faab00e5470 0x7faab0080310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:59.981 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:59.965 7faab5c28700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faab007ed80 con 0x7faab01b3e40 2021-02-10T11:03:59.981 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:59.965 7faab5c28700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faab00d1790 con 0x7faab00e5070 2021-02-10T11:03:59.981 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:59.965 7faab5c28700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faab00d19b0 con 0x7faab00e5470 2021-02-10T11:03:59.981 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:59.965 7faaaf7fe700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7faab00e5070 0x7faab01b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:59.981 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:59.965 7faaaeffd700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7faab01b3e40 0x7faab007d690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:41740/0 (socket says 172.21.2.102:41740) 2021-02-10T11:03:59.981 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:59.965 7faaaeffd700 1 -- 172.21.2.102:0/1802624254 learned_addr learned my addr 172.21.2.102:0/1802624254 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:03:59.981 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:59.965 7faaaf7fe700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7faab00e5070 0x7faab01b3820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:49516/0 (socket says 172.21.2.102:49516) 2021-02-10T11:03:59.981 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:59.966 7faaaffff700 1 --2- 172.21.2.102:0/1802624254 >> v2:172.21.2.113:3300/0 conn(0x7faab00e5470 0x7faab0080310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:59.981 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:59.966 7faaaf7fe700 1 -- 172.21.2.102:0/1802624254 >> v2:172.21.2.113:3300/0 conn(0x7faab00e5470 msgr2=0x7faab0080310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:03:59.982 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:59.966 7faaaf7fe700 1 --2- 172.21.2.102:0/1802624254 >> v2:172.21.2.113:3300/0 conn(0x7faab00e5470 0x7faab0080310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:59.982 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:59.966 7faaaf7fe700 1 -- 172.21.2.102:0/1802624254 >> v1:172.21.2.102:6789/0 conn(0x7faab01b3e40 legacy=0x7faab007d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:59.982 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:59.966 7faaaf7fe700 1 -- 172.21.2.102:0/1802624254 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faab00d1bd0 con 0x7faab00e5070 2021-02-10T11:03:59.982 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:59.966 7faaaffff700 1 --2- 172.21.2.102:0/1802624254 >> v2:172.21.2.113:3300/0 conn(0x7faab00e5470 0x7faab0080310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:03:59.982 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:59.966 7faaaf7fe700 1 --2- 172.21.2.102:0/1802624254 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7faab00e5070 0x7faab01b3820 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7faaa0006650 tx=0x7faaa0009a80).ready entity=mon.0 client_cookie=3f5b621a939ef118 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:03:59.982 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:59.966 7faaae7fc700 1 -- 172.21.2.102:0/1802624254 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faaa000c0c0 con 0x7faab00e5070 2021-02-10T11:03:59.982 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:59.966 7faaae7fc700 1 -- 172.21.2.102:0/1802624254 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faaa000c340 con 0x7faab00e5070 2021-02-10T11:03:59.982 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:59.966 7faaae7fc700 1 -- 172.21.2.102:0/1802624254 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faaa000c860 con 0x7faab00e5070 2021-02-10T11:03:59.982 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:59.966 7faab5c28700 1 -- 172.21.2.102:0/1802624254 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7faab00e5070 msgr2=0x7faab01b3820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:59.982 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:59.966 7faab5c28700 1 --2- 172.21.2.102:0/1802624254 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7faab00e5070 0x7faab01b3820 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7faaa0006650 tx=0x7faaa0009a80).stop 2021-02-10T11:03:59.982 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:59.967 7faab5c28700 1 -- 172.21.2.102:0/1802624254 shutdown_connections 2021-02-10T11:03:59.982 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:59.967 7faab5c28700 1 --2- 172.21.2.102:0/1802624254 >> v2:172.21.2.113:3300/0 conn(0x7faab00e5470 0x7faab0080310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:59.982 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:59.967 7faab5c28700 1 --2- 172.21.2.102:0/1802624254 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7faab00e5070 0x7faab01b3820 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:59.983 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:59.967 7faab5c28700 1 -- 172.21.2.102:0/1802624254 >> 172.21.2.102:0/1802624254 conn(0x7faab00ce690 msgr2=0x7faab00e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:03:59.983 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:59.967 7faab5c28700 1 -- 172.21.2.102:0/1802624254 shutdown_connections 2021-02-10T11:03:59.983 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:59.967 7faab5c28700 1 -- 172.21.2.102:0/1802624254 wait complete. 2021-02-10T11:03:59.984 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:59.982 7faab5c28700 1 Processor -- start 2021-02-10T11:03:59.985 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:59.982 7faab5c28700 1 -- start start 2021-02-10T11:03:59.985 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:59.982 7faab5c28700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7faab00be120 0x7faab00be720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:59.985 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:59.982 7faab5c28700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7faab00896d0 0x7faab0198b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:59.985 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:59.982 7faab5c28700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7faab0199150 0x7faab00ba680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:59.985 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:59.982 7faab5c28700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faab00baca0 con 0x7faab00896d0 2021-02-10T11:03:59.985 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:59.982 7faab5c28700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faab00baec0 con 0x7faab00be120 2021-02-10T11:03:59.985 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:59.982 7faab5c28700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faab0089ad0 con 0x7faab0199150 2021-02-10T11:03:59.985 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:59.983 7faaaeffd700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7faab00896d0 0x7faab0198b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:59.985 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:59.983 7faaaeffd700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7faab00896d0 0x7faab0198b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:49522/0 (socket says 172.21.2.102:49522) 2021-02-10T11:03:59.985 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:59.983 7faaaeffd700 1 -- 172.21.2.102:0/498036353 learned_addr learned my addr 172.21.2.102:0/498036353 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:03:59.986 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:59.983 7faaaffff700 1 --2- 172.21.2.102:0/498036353 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7faab0199150 0x7faab00ba680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:59.986 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:59.983 7faaaf7fe700 1 --2- 172.21.2.102:0/498036353 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7faab00be120 0x7faab00be720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:59.986 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:59.983 7faaaeffd700 1 -- 172.21.2.102:0/498036353 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7faab0199150 msgr2=0x7faab00ba680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:59.986 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:59.983 7faaaeffd700 1 --2- 172.21.2.102:0/498036353 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7faab0199150 0x7faab00ba680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:59.986 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:59.983 7faaaeffd700 1 -- 172.21.2.102:0/498036353 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7faab00be120 msgr2=0x7faab00be720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:03:59.986 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:59.983 7faaaeffd700 1 --2- 172.21.2.102:0/498036353 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7faab00be120 0x7faab00be720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:03:59.986 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:59.983 7faaaeffd700 1 -- 172.21.2.102:0/498036353 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faab0089cf0 con 0x7faab00896d0 2021-02-10T11:03:59.986 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:59.983 7faaaf7fe700 1 --2- 172.21.2.102:0/498036353 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7faab00be120 0x7faab00be720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:03:59.986 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:59.983 7faaaeffd700 1 --2- 172.21.2.102:0/498036353 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7faab00896d0 0x7faab0198b30 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7faaa4001c10 tx=0x7faaa400a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:03:59.987 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:59.983 7faa97fff700 1 -- 172.21.2.102:0/498036353 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faaa40010b0 con 0x7faab00896d0 2021-02-10T11:03:59.987 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:59.983 7faa97fff700 1 -- 172.21.2.102:0/498036353 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faaa400acc0 con 0x7faab00896d0 2021-02-10T11:03:59.987 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:59.983 7faab5c28700 1 -- 172.21.2.102:0/498036353 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faab0089cf0 con 0x7faab00896d0 2021-02-10T11:03:59.987 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:59.983 7faa97fff700 1 -- 172.21.2.102:0/498036353 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faaa40045d0 con 0x7faab00896d0 2021-02-10T11:03:59.987 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:59.983 7faab5c28700 1 -- 172.21.2.102:0/498036353 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faab0089cf0 con 0x7faab00896d0 2021-02-10T11:03:59.987 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:59.984 7faa97fff700 1 -- 172.21.2.102:0/498036353 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7faaa4004140 con 0x7faab00896d0 2021-02-10T11:03:59.987 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:59.984 7faa97fff700 1 --2- 172.21.2.102:0/498036353 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7faa90018700 0x7faa9001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:03:59.987 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:59.984 7faa97fff700 1 -- 172.21.2.102:0/498036353 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5573+0+0 (secure 0 0 0) 0x7faaa4032610 con 0x7faab00896d0 2021-02-10T11:03:59.987 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:59.984 7faaaf7fe700 1 --2- 172.21.2.102:0/498036353 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7faa90018700 0x7faa9001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:03:59.987 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:03:59.984 7faaaf7fe700 1 --2- 172.21.2.102:0/498036353 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7faa90018700 0x7faa9001abc0 crc :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:04:00.014 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:00.015 7faab5c28700 1 -- 172.21.2.102:0/498036353 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faab0107180 con 0x7faab00896d0 2021-02-10T11:04:00.017 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:00.018 7faa97fff700 1 -- 172.21.2.102:0/498036353 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7faaa4001050 con 0x7faab00896d0 2021-02-10T11:04:00.157 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:00.158 7faab5c28700 1 -- 172.21.2.102:0/498036353 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7faab0107180 con 0x7faab00896d0 2021-02-10T11:04:00.157 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:00.158 7faa97fff700 1 -- 172.21.2.102:0/498036353 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7faaa4004140 con 0x7faab00896d0 2021-02-10T11:04:00.158 INFO:teuthology.orchestra.run.gibba002.stdout:55834574857 2021-02-10T11:04:00.159 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:00.160 7faab5c28700 1 -- 172.21.2.102:0/498036353 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7faa90018700 msgr2=0x7faa9001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:00.159 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:00.160 7faab5c28700 1 --2- 172.21.2.102:0/498036353 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7faa90018700 0x7faa9001abc0 crc :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:00.159 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:00.160 7faab5c28700 1 -- 172.21.2.102:0/498036353 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7faab00896d0 msgr2=0x7faab0198b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:00.159 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:00.160 7faab5c28700 1 --2- 172.21.2.102:0/498036353 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7faab00896d0 0x7faab0198b30 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7faaa4001c10 tx=0x7faaa400a8d0).stop 2021-02-10T11:04:00.160 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:00.160 7faab5c28700 1 -- 172.21.2.102:0/498036353 shutdown_connections 2021-02-10T11:04:00.160 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:00.160 7faab5c28700 1 --2- 172.21.2.102:0/498036353 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7faab0199150 0x7faab00ba680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:00.160 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:00.160 7faab5c28700 1 --2- 172.21.2.102:0/498036353 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7faab00896d0 0x7faab0198b30 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:00.160 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:00.160 7faab5c28700 1 --2- 172.21.2.102:0/498036353 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7faa90018700 0x7faa9001abc0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:00.160 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:00.160 7faab5c28700 1 --2- 172.21.2.102:0/498036353 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7faab00be120 0x7faab00be720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:00.160 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:00.160 7faab5c28700 1 -- 172.21.2.102:0/498036353 >> 172.21.2.102:0/498036353 conn(0x7faab0098740 msgr2=0x7faab00c3590 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:04:00.160 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:00.160 7faab5c28700 1 -- 172.21.2.102:0/498036353 shutdown_connections 2021-02-10T11:04:00.160 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:00.161 7faab5c28700 1 -- 172.21.2.102:0/498036353 wait complete. 2021-02-10T11:04:00.167 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574857 for osd.6 2021-02-10T11:04:00.168 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd last-stat-seq osd.7 2021-02-10T11:04:00.593 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:00.577 7f0ce023d700 1 Processor -- start 2021-02-10T11:04:00.593 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:00.577 7f0ce023d700 1 -- start start 2021-02-10T11:04:00.593 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:00.578 7f0ce023d700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f0cd80e5470 0x7f0cd807e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:00.593 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:00.578 7f0ce023d700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f0cd807f300 0x7f0cd807f700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:00.593 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:00.578 7f0ce023d700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cd807ed80 con 0x7f0cd80e5070 2021-02-10T11:04:00.594 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:00.578 7f0ce023d700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0cd81b4010 con 0x7f0cd807f300 2021-02-10T11:04:00.594 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:00.578 7f0ce023d700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0cd80d1790 con 0x7f0cd80e5470 2021-02-10T11:04:00.594 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:00.578 7f0cde85a700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f0cd807f300 0x7f0cd807f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:00.594 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:00.578 7f0cde85a700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f0cd807f300 0x7f0cd807f700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:49532/0 (socket says 172.21.2.102:49532) 2021-02-10T11:04:00.594 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:00.578 7f0cde85a700 1 -- 172.21.2.102:0/3160834576 learned_addr learned my addr 172.21.2.102:0/3160834576 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:04:00.594 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:00.578 7f0cde85a700 1 -- 172.21.2.102:0/3160834576 >> v2:172.21.2.113:3300/0 conn(0x7f0cd80e5470 msgr2=0x7f0cd807e6a0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:04:00.594 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:00.578 7f0cde85a700 1 --2- 172.21.2.102:0/3160834576 >> v2:172.21.2.113:3300/0 conn(0x7f0cd80e5470 0x7f0cd807e6a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:00.594 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:00.578 7f0cde85a700 1 -- 172.21.2.102:0/3160834576 >> v1:172.21.2.113:6789/0 conn(0x7f0cd80e5070 legacy=0x7f0cd81b3820 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:04:00.594 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:00.578 7f0cde85a700 1 -- 172.21.2.102:0/3160834576 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cd80d19b0 con 0x7f0cd807f300 2021-02-10T11:04:00.594 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:00.579 7f0cde85a700 1 --2- 172.21.2.102:0/3160834576 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0cd807f300 0x7f0cd807f700 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f0cd0006f00 tx=0x7f0cd0009f10).ready entity=mon.0 client_cookie=b62b1441b3d46921 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:04:00.594 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:00.579 7f0cdd057700 1 -- 172.21.2.102:0/3160834576 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0cd000b3b0 con 0x7f0cd807f300 2021-02-10T11:04:00.594 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:00.579 7f0cdd057700 1 -- 172.21.2.102:0/3160834576 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cd000b730 con 0x7f0cd807f300 2021-02-10T11:04:00.595 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:00.579 7f0cdd057700 1 -- 172.21.2.102:0/3160834576 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0cd000bc50 con 0x7f0cd807f300 2021-02-10T11:04:00.595 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:00.579 7f0ce023d700 1 -- 172.21.2.102:0/3160834576 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0cd807f300 msgr2=0x7f0cd807f700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:00.595 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:00.579 7f0ce023d700 1 --2- 172.21.2.102:0/3160834576 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0cd807f300 0x7f0cd807f700 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f0cd0006f00 tx=0x7f0cd0009f10).stop 2021-02-10T11:04:00.595 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:00.579 7f0ce023d700 1 -- 172.21.2.102:0/3160834576 shutdown_connections 2021-02-10T11:04:00.595 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:00.579 7f0ce023d700 1 --2- 172.21.2.102:0/3160834576 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0cd807f300 0x7f0cd807f700 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:00.595 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:00.579 7f0ce023d700 1 --2- 172.21.2.102:0/3160834576 >> v2:172.21.2.113:3300/0 conn(0x7f0cd80e5470 0x7f0cd807e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:00.595 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:00.579 7f0ce023d700 1 -- 172.21.2.102:0/3160834576 >> 172.21.2.102:0/3160834576 conn(0x7f0cd80ce690 msgr2=0x7f0cd80e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:04:00.595 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:00.580 7f0ce023d700 1 -- 172.21.2.102:0/3160834576 shutdown_connections 2021-02-10T11:04:00.595 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:00.580 7f0ce023d700 1 -- 172.21.2.102:0/3160834576 wait complete. 2021-02-10T11:04:00.595 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:00.594 7f0ce023d700 1 Processor -- start 2021-02-10T11:04:00.595 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:00.594 7f0ce023d700 1 -- start start 2021-02-10T11:04:00.596 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:00.595 7f0ce023d700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0cd80be580 0x7f0cd80be980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:00.597 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:00.595 7f0ce023d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0cd8097ed0 0x7f0cd80982d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:00.597 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:00.595 7f0ce023d700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0cd8099180 0x7f0cd8099580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:00.597 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:00.595 7f0ce023d700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cd80988f0 con 0x7f0cd8097ed0 2021-02-10T11:04:00.597 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:00.595 7f0ce023d700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cd80c1110 con 0x7f0cd8099180 2021-02-10T11:04:00.597 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:00.595 7f0ce023d700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cd8099ba0 con 0x7f0cd80be580 2021-02-10T11:04:00.597 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:00.595 7f0cdd858700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0cd8097ed0 0x7f0cd80982d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:00.597 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:00.595 7f0cdd858700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0cd8097ed0 0x7f0cd80982d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:49536/0 (socket says 172.21.2.102:49536) 2021-02-10T11:04:00.597 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:00.595 7f0cdd858700 1 -- 172.21.2.102:0/1963753514 learned_addr learned my addr 172.21.2.102:0/1963753514 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:04:00.597 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:00.595 7f0cdd858700 1 -- 172.21.2.102:0/1963753514 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0cd80be580 msgr2=0x7f0cd80be980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:00.597 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:00.595 7f0cdd858700 1 --2- 172.21.2.102:0/1963753514 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0cd80be580 0x7f0cd80be980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:00.597 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:00.595 7f0cdd858700 1 -- 172.21.2.102:0/1963753514 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0cd8099180 msgr2=0x7f0cd8099580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:00.597 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:00.595 7f0cdd858700 1 --2- 172.21.2.102:0/1963753514 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0cd8099180 0x7f0cd8099580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:00.597 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:00.595 7f0cdd858700 1 -- 172.21.2.102:0/1963753514 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cd8099dc0 con 0x7f0cd8097ed0 2021-02-10T11:04:00.597 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:00.595 7f0cdd858700 1 --2- 172.21.2.102:0/1963753514 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0cd8097ed0 0x7f0cd80982d0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7f0cd400b5c0 tx=0x7f0cd400b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:04:00.598 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:00.595 7f0ccaffd700 1 -- 172.21.2.102:0/1963753514 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0cd4006b80 con 0x7f0cd8097ed0 2021-02-10T11:04:00.598 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:00.596 7f0ccaffd700 1 -- 172.21.2.102:0/1963753514 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cd400bd50 con 0x7f0cd8097ed0 2021-02-10T11:04:00.598 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:00.596 7f0ce023d700 1 -- 172.21.2.102:0/1963753514 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0cd8099dc0 con 0x7f0cd8097ed0 2021-02-10T11:04:00.598 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:00.596 7f0ccaffd700 1 -- 172.21.2.102:0/1963753514 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0cd40054d0 con 0x7f0cd8097ed0 2021-02-10T11:04:00.598 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:00.596 7f0ce023d700 1 -- 172.21.2.102:0/1963753514 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cd809a350 con 0x7f0cd8097ed0 2021-02-10T11:04:00.598 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:00.596 7f0ccaffd700 1 -- 172.21.2.102:0/1963753514 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f0cd4006030 con 0x7f0cd8097ed0 2021-02-10T11:04:00.598 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:00.596 7f0ccaffd700 1 --2- 172.21.2.102:0/1963753514 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0cc00186d0 0x7f0cc001abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:00.598 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:00.596 7f0ccaffd700 1 -- 172.21.2.102:0/1963753514 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f0cd4032630 con 0x7f0cd8097ed0 2021-02-10T11:04:00.598 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:00.596 7f0cde059700 1 --2- 172.21.2.102:0/1963753514 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0cc00186d0 0x7f0cc001abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:00.598 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:00.597 7f0cde059700 1 --2- 172.21.2.102:0/1963753514 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0cc00186d0 0x7f0cc001abd0 crc :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:04:00.627 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:00.628 7f0ce023d700 1 -- 172.21.2.102:0/1963753514 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0cd8098c50 con 0x7f0cd8097ed0 2021-02-10T11:04:00.629 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:00.630 7f0ccaffd700 1 -- 172.21.2.102:0/1963753514 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f0cd4000960 con 0x7f0cd8097ed0 2021-02-10T11:04:00.770 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:00.771 7f0ce023d700 1 -- 172.21.2.102:0/1963753514 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f0cd8098c50 con 0x7f0cd8097ed0 2021-02-10T11:04:00.771 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:00.772 7f0ccaffd700 1 -- 172.21.2.102:0/1963753514 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f0cd400bd50 con 0x7f0cd8097ed0 2021-02-10T11:04:00.771 INFO:teuthology.orchestra.run.gibba002.stdout:55834574857 2021-02-10T11:04:00.773 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:00.774 7f0ce023d700 1 -- 172.21.2.102:0/1963753514 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0cc00186d0 msgr2=0x7f0cc001abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:00.773 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:00.774 7f0ce023d700 1 --2- 172.21.2.102:0/1963753514 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0cc00186d0 0x7f0cc001abd0 crc :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:00.773 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:00.774 7f0ce023d700 1 -- 172.21.2.102:0/1963753514 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0cd8097ed0 msgr2=0x7f0cd80982d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:00.773 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:00.774 7f0ce023d700 1 --2- 172.21.2.102:0/1963753514 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0cd8097ed0 0x7f0cd80982d0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7f0cd400b5c0 tx=0x7f0cd400b960).stop 2021-02-10T11:04:00.773 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:00.774 7f0ce023d700 1 -- 172.21.2.102:0/1963753514 shutdown_connections 2021-02-10T11:04:00.774 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:00.774 7f0ce023d700 1 --2- 172.21.2.102:0/1963753514 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0cc00186d0 0x7f0cc001abd0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:00.774 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:00.774 7f0ce023d700 1 --2- 172.21.2.102:0/1963753514 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0cd8099180 0x7f0cd8099580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:00.774 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:00.774 7f0ce023d700 1 --2- 172.21.2.102:0/1963753514 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0cd8097ed0 0x7f0cd80982d0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:00.774 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:00.774 7f0ce023d700 1 --2- 172.21.2.102:0/1963753514 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0cd80be580 0x7f0cd80be980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:00.774 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:00.774 7f0ce023d700 1 -- 172.21.2.102:0/1963753514 >> 172.21.2.102:0/1963753514 conn(0x7f0cd80bde70 msgr2=0x7f0cd80c3970 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:04:00.774 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:00.774 7f0ce023d700 1 -- 172.21.2.102:0/1963753514 shutdown_connections 2021-02-10T11:04:00.774 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:00.774 7f0ce023d700 1 -- 172.21.2.102:0/1963753514 wait complete. 2021-02-10T11:04:00.781 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574857 for osd.7 2021-02-10T11:04:00.781 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2021-02-10T11:04:00.781 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph pg dump --format=json 2021-02-10T11:04:01.208 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:01.191 7f8fb2ffa700 1 Processor -- start 2021-02-10T11:04:01.208 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:01.191 7f8fb2ffa700 1 -- start start 2021-02-10T11:04:01.208 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:01.191 7f8fb2ffa700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8fac0e5470 0x7f8fac07e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:01.208 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:01.191 7f8fb2ffa700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f8fac07f300 0x7f8fac07f700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:01.208 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:01.192 7f8fb2ffa700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fac07ed80 con 0x7f8fac0e5070 2021-02-10T11:04:01.208 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:01.192 7f8fb2ffa700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8fac1b4010 con 0x7f8fac07f300 2021-02-10T11:04:01.209 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:01.192 7f8fb2ffa700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fac0d1790 con 0x7f8fac0e5470 2021-02-10T11:04:01.209 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:01.192 7f8fb0e16700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f8fac0e5070 0x7f8fac1b3820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:41766/0 (socket says 172.21.2.102:41766) 2021-02-10T11:04:01.209 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:01.192 7f8fb0e16700 1 -- 172.21.2.102:0/2882364838 learned_addr learned my addr 172.21.2.102:0/2882364838 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:04:01.209 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:01.192 7f8fab7fe700 1 -- 172.21.2.102:0/2882364838 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4241294107 0 0) 0x7f8f94000ce0 con 0x7f8fac0e5070 2021-02-10T11:04:01.209 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:01.192 7f8fab7fe700 1 -- 172.21.2.102:0/2882364838 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f98002fe0 con 0x7f8fac0e5070 2021-02-10T11:04:01.209 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:01.192 7f8fb1617700 1 --2- 172.21.2.102:0/2882364838 >> v2:172.21.2.113:3300/0 conn(0x7f8fac07f300 0x7f8fac07f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:01.209 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:01.192 7f8fabfff700 1 --2- 172.21.2.102:0/2882364838 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8fac0e5470 0x7f8fac07e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:01.209 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:01.193 7f8fab7fe700 1 -- 172.21.2.102:0/2882364838 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1713826484 0 0) 0x7f8f940010e0 con 0x7f8fac0e5070 2021-02-10T11:04:01.209 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:01.193 7f8fab7fe700 1 -- 172.21.2.102:0/2882364838 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8fac0e5470 msgr2=0x7f8fac07e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:04:01.209 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:01.193 7f8fab7fe700 1 --2- 172.21.2.102:0/2882364838 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8fac0e5470 0x7f8fac07e6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:01.209 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:01.193 7f8fab7fe700 1 -- 172.21.2.102:0/2882364838 >> v2:172.21.2.113:3300/0 conn(0x7f8fac07f300 msgr2=0x7f8fac07f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:04:01.209 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:01.193 7f8fab7fe700 1 --2- 172.21.2.102:0/2882364838 >> v2:172.21.2.113:3300/0 conn(0x7f8fac07f300 0x7f8fac07f700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:01.209 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:01.193 7f8fab7fe700 1 -- 172.21.2.102:0/2882364838 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fac0d1980 con 0x7f8fac0e5070 2021-02-10T11:04:01.210 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:01.193 7f8fab7fe700 1 -- 172.21.2.102:0/2882364838 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7f8f94001600 con 0x7f8fac0e5070 2021-02-10T11:04:01.210 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:01.193 7f8fab7fe700 1 -- 172.21.2.102:0/2882364838 >> v1:172.21.2.102:6789/0 conn(0x7f8fac0e5070 legacy=0x7f8fac1b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:01.210 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:01.193 7f8fab7fe700 1 --2- 172.21.2.102:0/2882364838 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8f98003ae0 0x7f8f98005f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:01.210 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:01.193 7f8fab7fe700 1 --2- 172.21.2.102:0/2882364838 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8f9804a820 0x7f8f9804cd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:01.210 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:01.193 7f8fab7fe700 1 --2- 172.21.2.102:0/2882364838 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8f9804d640 0x7f8f9804fb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:01.210 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:01.193 7f8fab7fe700 1 -- 172.21.2.102:0/2882364838 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f98050380 con 0x7f8f9804a820 2021-02-10T11:04:01.210 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:01.193 7f8fab7fe700 1 -- 172.21.2.102:0/2882364838 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f980505a0 con 0x7f8f98003ae0 2021-02-10T11:04:01.210 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:01.193 7f8fab7fe700 1 -- 172.21.2.102:0/2882364838 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f980507c0 con 0x7f8f9804d640 2021-02-10T11:04:01.210 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:01.193 7f8fabfff700 1 --2- 172.21.2.102:0/2882364838 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8fac0e5470 0x7f8fac07e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:04:01.210 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:01.193 7f8fb1617700 1 --2- 172.21.2.102:0/2882364838 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8f9804d640 0x7f8f9804fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:01.210 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:01.193 7f8fb0e16700 1 --2- 172.21.2.102:0/2882364838 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8f98003ae0 0x7f8f98005f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:01.210 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:01.193 7f8fabfff700 1 --2- 172.21.2.102:0/2882364838 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8f9804a820 0x7f8f9804cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:01.211 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:01.193 7f8fabfff700 1 -- 172.21.2.102:0/2882364838 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8f9804d640 msgr2=0x7f8f9804fb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:04:01.211 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:01.193 7f8fabfff700 1 --2- 172.21.2.102:0/2882364838 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8f9804d640 0x7f8f9804fb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:01.211 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:01.193 7f8fabfff700 1 -- 172.21.2.102:0/2882364838 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8f98003ae0 msgr2=0x7f8f98005f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:04:01.211 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:01.193 7f8fabfff700 1 --2- 172.21.2.102:0/2882364838 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8f98003ae0 0x7f8f98005f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:01.211 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:01.193 7f8fabfff700 1 -- 172.21.2.102:0/2882364838 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f98050a60 con 0x7f8f9804a820 2021-02-10T11:04:01.211 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:01.194 7f8fabfff700 1 --2- 172.21.2.102:0/2882364838 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8f9804a820 0x7f8f9804cd40 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7f8f9c003180 tx=0x7f8f9c009e70).ready entity=mon.0 client_cookie=430e50f0b58fc266 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:04:01.211 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:01.194 7f8fab7fe700 1 -- 172.21.2.102:0/2882364838 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8f9c006800 con 0x7f8f9804a820 2021-02-10T11:04:01.211 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:01.194 7f8fab7fe700 1 -- 172.21.2.102:0/2882364838 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f9c006800 con 0x7f8f9804a820 2021-02-10T11:04:01.211 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:01.194 7f8fab7fe700 1 -- 172.21.2.102:0/2882364838 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8f9c00b4f0 con 0x7f8f9804a820 2021-02-10T11:04:01.211 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:01.194 7f8fb2ffa700 1 -- 172.21.2.102:0/2882364838 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8f9804a820 msgr2=0x7f8f9804cd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:01.211 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:01.194 7f8fb2ffa700 1 --2- 172.21.2.102:0/2882364838 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8f9804a820 0x7f8f9804cd40 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7f8f9c003180 tx=0x7f8f9c009e70).stop 2021-02-10T11:04:01.211 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:01.194 7f8fb1617700 1 -- 172.21.2.102:0/2882364838 reap_dead start 2021-02-10T11:04:01.211 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:01.194 7f8fb2ffa700 1 -- 172.21.2.102:0/2882364838 shutdown_connections 2021-02-10T11:04:01.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:01.194 7f8fb2ffa700 1 -- 172.21.2.102:0/2882364838 >> 172.21.2.102:0/2882364838 conn(0x7f8fac0ce690 msgr2=0x7f8fac0e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:04:01.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:01.194 7f8fb2ffa700 1 -- 172.21.2.102:0/2882364838 shutdown_connections 2021-02-10T11:04:01.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:01.194 7f8fb2ffa700 1 -- 172.21.2.102:0/2882364838 wait complete. 2021-02-10T11:04:01.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:01.209 7f8fb2ffa700 1 Processor -- start 2021-02-10T11:04:01.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:01.209 7f8fb2ffa700 1 -- start start 2021-02-10T11:04:01.213 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:01.209 7f8fb2ffa700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8fac194000 0x7f8fac0b0d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:01.213 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:01.209 7f8fb2ffa700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8fac0b1330 0x7f8fac087050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:01.214 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:01.209 7f8fb2ffa700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8fac087670 0x7f8fac087a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:01.214 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:01.209 7f8fb2ffa700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fac09c0e0 con 0x7f8fac0b1330 2021-02-10T11:04:01.214 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:01.210 7f8fb2ffa700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fac194440 con 0x7f8fac087670 2021-02-10T11:04:01.214 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:01.210 7f8fb2ffa700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fac09c5e0 con 0x7f8fac194000 2021-02-10T11:04:01.214 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:01.210 7f8fabfff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8fac0b1330 0x7f8fac087050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:01.214 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:01.210 7f8fabfff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8fac0b1330 0x7f8fac087050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:49554/0 (socket says 172.21.2.102:49554) 2021-02-10T11:04:01.214 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:01.210 7f8fabfff700 1 -- 172.21.2.102:0/3776631858 learned_addr learned my addr 172.21.2.102:0/3776631858 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:04:01.214 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:01.210 7f8fabfff700 1 -- 172.21.2.102:0/3776631858 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8fac194000 msgr2=0x7f8fac0b0d10 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:04:01.214 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:01.210 7f8fabfff700 1 --2- 172.21.2.102:0/3776631858 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8fac194000 0x7f8fac0b0d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:01.214 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:01.210 7f8fabfff700 1 -- 172.21.2.102:0/3776631858 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8fac087670 msgr2=0x7f8fac087a70 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:04:01.214 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:01.210 7f8fabfff700 1 --2- 172.21.2.102:0/3776631858 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8fac087670 0x7f8fac087a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:01.214 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:01.210 7f8fabfff700 1 -- 172.21.2.102:0/3776631858 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fac09c7d0 con 0x7f8fac0b1330 2021-02-10T11:04:01.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:01.210 7f8fabfff700 1 --2- 172.21.2.102:0/3776631858 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8fac0b1330 0x7f8fac087050 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f8f9c009f20 tx=0x7f8f9c00bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:04:01.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:01.210 7f8fa97fa700 1 -- 172.21.2.102:0/3776631858 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8f9c000ce0 con 0x7f8fac0b1330 2021-02-10T11:04:01.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:01.210 7f8fa97fa700 1 -- 172.21.2.102:0/3776631858 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f9c00c380 con 0x7f8fac0b1330 2021-02-10T11:04:01.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:01.210 7f8fa97fa700 1 -- 172.21.2.102:0/3776631858 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8f9c00c750 con 0x7f8fac0b1330 2021-02-10T11:04:01.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:01.210 7f8fb2ffa700 1 -- 172.21.2.102:0/3776631858 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fac09c7d0 con 0x7f8fac0b1330 2021-02-10T11:04:01.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:01.211 7f8fb2ffa700 1 -- 172.21.2.102:0/3776631858 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fac09c7d0 con 0x7f8fac0b1330 2021-02-10T11:04:01.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:01.211 7f8fa97fa700 1 -- 172.21.2.102:0/3776631858 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f8f9c001010 con 0x7f8fac0b1330 2021-02-10T11:04:01.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:01.211 7f8fa97fa700 1 --2- 172.21.2.102:0/3776631858 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8f90018730 0x7f8f9001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:01.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:01.211 7f8fa97fa700 1 -- 172.21.2.102:0/3776631858 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f8f9c0329d0 con 0x7f8fac0b1330 2021-02-10T11:04:01.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:01.211 7f8fb0e16700 1 --2- 172.21.2.102:0/3776631858 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8f90018730 0x7f8f9001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:01.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:01.212 7f8fb0e16700 1 --2- 172.21.2.102:0/3776631858 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8f90018730 0x7f8f9001ac10 crc :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:04:01.240 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:01.241 7f8fb2ffa700 1 -- 172.21.2.102:0/3776631858 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fac084b20 con 0x7f8fac0b1330 2021-02-10T11:04:01.242 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:01.244 7f8fa97fa700 1 -- 172.21.2.102:0/3776631858 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f8f9c00be30 con 0x7f8fac0b1330 2021-02-10T11:04:01.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:01.377 7f8fb2ffa700 1 -- 172.21.2.102:0/3776631858 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "pg dump", "target": ["mgr", ""], "format": "json"}) v1 -- 0x7f8fac084b20 con 0x7f8f90018730 2021-02-10T11:04:01.378 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:01.379 7f8fa97fa700 1 -- 172.21.2.102:0/3776631858 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 dumped all) v1 ==== 18+0+77157 (crc 0 0 0) 0x7f8f94001ab0 con 0x7f8f90018730 2021-02-10T11:04:01.380 INFO:teuthology.orchestra.run.gibba002.stderr:dumped all 2021-02-10T11:04:01.380 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:04:01.380 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:01.381 7f8fb2ffa700 1 -- 172.21.2.102:0/3776631858 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8f90018730 msgr2=0x7f8f9001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:01.381 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:01.381 7f8fb2ffa700 1 --2- 172.21.2.102:0/3776631858 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8f90018730 0x7f8f9001ac10 crc :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:01.381 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:01.381 7f8fb2ffa700 1 -- 172.21.2.102:0/3776631858 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8fac0b1330 msgr2=0x7f8fac087050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:01.381 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:01.381 7f8fb2ffa700 1 --2- 172.21.2.102:0/3776631858 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8fac0b1330 0x7f8fac087050 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f8f9c009f20 tx=0x7f8f9c00bcf0).stop 2021-02-10T11:04:01.381 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:01.382 7f8fb2ffa700 1 -- 172.21.2.102:0/3776631858 shutdown_connections 2021-02-10T11:04:01.381 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:01.382 7f8fb2ffa700 1 --2- 172.21.2.102:0/3776631858 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8f90018730 0x7f8f9001ac10 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:01.381 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:01.382 7f8fb2ffa700 1 --2- 172.21.2.102:0/3776631858 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8fac087670 0x7f8fac087a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:01.381 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:01.382 7f8fb2ffa700 1 --2- 172.21.2.102:0/3776631858 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8fac0b1330 0x7f8fac087050 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:01.382 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:01.382 7f8fb2ffa700 1 --2- 172.21.2.102:0/3776631858 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8fac194000 0x7f8fac0b0d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:01.382 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:01.382 7f8fb2ffa700 1 -- 172.21.2.102:0/3776631858 >> 172.21.2.102:0/3776631858 conn(0x7f8fac092c20 msgr2=0x7f8fac195070 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:04:01.382 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:01.382 7f8fb2ffa700 1 -- 172.21.2.102:0/3776631858 shutdown_connections 2021-02-10T11:04:01.382 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:01.382 7f8fb2ffa700 1 -- 172.21.2.102:0/3776631858 wait complete. 2021-02-10T11:04:01.389 INFO:teuthology.orchestra.run.gibba002.stdout:{"pg_ready":true,"pg_map":{"version":37,"stamp":"2021-02-10 11:03:59.678795","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2286,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":8,"num_per_pool_osds":8,"kb":754974720,"kb_used":8390336,"kb_used_data":1664,"kb_used_omap":0,"kb_used_meta":8388608,"kb_avail":746584384,"statfs":{"total":773094113280,"available":764502409216,"internally_reserved":8589934592,"allocated":1703936,"data_stored":554356,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8589934592},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":760,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":33,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":11,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001585"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":"13","reported_epoch":"19","state":"active+clean","last_fresh":"2021-02-10 11:03:43.388602","last_change":"2021-02-10 11:03:38.737712","last_active":"2021-02-10 11:03:43.388602","last_peered":"2021-02-10 11:03:43.388602","last_clean":"2021-02-10 11:03:43.388602","last_became_active":"2021-02-10 11:03:38.737435","last_became_peered":"2021-02-10 11:03:38.737435","last_unstale":"2021-02-10 11:03:43.388602","last_undegraded":"2021-02-10 11:03:43.388602","last_fullsized":"2021-02-10 11:03:43.388602","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:03:37.729316","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:03:37.729316","last_clean_scrub_stamp":"2021-02-10 11:03:37.729316","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.4","version":"20'8","reported_seq":"18","reported_epoch":"20","state":"active+clean","last_fresh":"2021-02-10 11:03:51.798768","last_change":"2021-02-10 11:03:42.760313","last_active":"2021-02-10 11:03:51.798768","last_peered":"2021-02-10 11:03:51.798768","last_clean":"2021-02-10 11:03:51.798768","last_became_active":"2021-02-10 11:03:42.760064","last_became_peered":"2021-02-10 11:03:42.760064","last_unstale":"2021-02-10 11:03:51.798768","last_undegraded":"2021-02-10 11:03:51.798768","last_fullsized":"2021-02-10 11:03:51.798768","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:03:41.747519","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:03:41.747519","last_clean_scrub_stamp":"2021-02-10 11:03:41.747519","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1526,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":"9","reported_epoch":"19","state":"active+clean","last_fresh":"2021-02-10 11:03:43.391539","last_change":"2021-02-10 11:03:43.391081","last_active":"2021-02-10 11:03:43.391539","last_peered":"2021-02-10 11:03:43.391539","last_clean":"2021-02-10 11:03:43.391539","last_became_active":"2021-02-10 11:03:43.390918","last_became_peered":"2021-02-10 11:03:43.390918","last_unstale":"2021-02-10 11:03:43.391539","last_undegraded":"2021-02-10 11:03:43.391539","last_fullsized":"2021-02-10 11:03:43.391539","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:03:42.751373","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:03:42.751373","last_clean_scrub_stamp":"2021-02-10 11:03:42.751373","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":"13","reported_epoch":"19","state":"active+clean","last_fresh":"2021-02-10 11:03:43.388603","last_change":"2021-02-10 11:03:38.738386","last_active":"2021-02-10 11:03:43.388603","last_peered":"2021-02-10 11:03:43.388603","last_clean":"2021-02-10 11:03:43.388603","last_became_active":"2021-02-10 11:03:38.738144","last_became_peered":"2021-02-10 11:03:38.738144","last_unstale":"2021-02-10 11:03:43.388603","last_undegraded":"2021-02-10 11:03:43.388603","last_fullsized":"2021-02-10 11:03:43.388603","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:03:37.729316","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:03:37.729316","last_clean_scrub_stamp":"2021-02-10 11:03:37.729316","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"20'6","reported_seq":"16","reported_epoch":"20","state":"active+clean","last_fresh":"2021-02-10 11:03:51.796270","last_change":"2021-02-10 11:03:42.758151","last_active":"2021-02-10 11:03:51.796270","last_peered":"2021-02-10 11:03:51.796270","last_clean":"2021-02-10 11:03:51.796270","last_became_active":"2021-02-10 11:03:42.757926","last_became_peered":"2021-02-10 11:03:42.757926","last_unstale":"2021-02-10 11:03:51.796270","last_undegraded":"2021-02-10 11:03:51.796270","last_fullsized":"2021-02-10 11:03:51.796270","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:03:41.747519","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:03:41.747519","last_clean_scrub_stamp":"2021-02-10 11:03:41.747519","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":160,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":"9","reported_epoch":"19","state":"active+clean","last_fresh":"2021-02-10 11:03:43.393614","last_change":"2021-02-10 11:03:43.392999","last_active":"2021-02-10 11:03:43.393614","last_peered":"2021-02-10 11:03:43.393614","last_clean":"2021-02-10 11:03:43.393614","last_became_active":"2021-02-10 11:03:43.392742","last_became_peered":"2021-02-10 11:03:43.392742","last_unstale":"2021-02-10 11:03:43.393614","last_undegraded":"2021-02-10 11:03:43.393614","last_fullsized":"2021-02-10 11:03:43.393614","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:03:42.751373","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:03:42.751373","last_clean_scrub_stamp":"2021-02-10 11:03:42.751373","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":"9","reported_epoch":"19","state":"active+clean","last_fresh":"2021-02-10 11:03:43.394812","last_change":"2021-02-10 11:03:43.394157","last_active":"2021-02-10 11:03:43.394812","last_peered":"2021-02-10 11:03:43.394812","last_clean":"2021-02-10 11:03:43.394812","last_became_active":"2021-02-10 11:03:43.393997","last_became_peered":"2021-02-10 11:03:43.393997","last_unstale":"2021-02-10 11:03:43.394812","last_undegraded":"2021-02-10 11:03:43.394812","last_fullsized":"2021-02-10 11:03:43.394812","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:03:42.751373","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:03:42.751373","last_clean_scrub_stamp":"2021-02-10 11:03:42.751373","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":"13","reported_epoch":"19","state":"active+clean","last_fresh":"2021-02-10 11:03:43.388728","last_change":"2021-02-10 11:03:38.738326","last_active":"2021-02-10 11:03:43.388728","last_peered":"2021-02-10 11:03:43.388728","last_clean":"2021-02-10 11:03:43.388728","last_became_active":"2021-02-10 11:03:38.738088","last_became_peered":"2021-02-10 11:03:38.738088","last_unstale":"2021-02-10 11:03:43.388728","last_undegraded":"2021-02-10 11:03:43.388728","last_fullsized":"2021-02-10 11:03:43.388728","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:03:37.729316","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:03:37.729316","last_clean_scrub_stamp":"2021-02-10 11:03:37.729316","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"20'4","reported_seq":"14","reported_epoch":"20","state":"active+clean","last_fresh":"2021-02-10 11:03:51.798316","last_change":"2021-02-10 11:03:42.759030","last_active":"2021-02-10 11:03:51.798316","last_peered":"2021-02-10 11:03:51.798316","last_clean":"2021-02-10 11:03:51.798316","last_became_active":"2021-02-10 11:03:42.758801","last_became_peered":"2021-02-10 11:03:42.758801","last_unstale":"2021-02-10 11:03:51.798316","last_undegraded":"2021-02-10 11:03:51.798316","last_fullsized":"2021-02-10 11:03:51.798316","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:03:41.747519","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:03:41.747519","last_clean_scrub_stamp":"2021-02-10 11:03:41.747519","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":"9","reported_epoch":"19","state":"active+clean","last_fresh":"2021-02-10 11:03:43.392222","last_change":"2021-02-10 11:03:43.391781","last_active":"2021-02-10 11:03:43.392222","last_peered":"2021-02-10 11:03:43.392222","last_clean":"2021-02-10 11:03:43.392222","last_became_active":"2021-02-10 11:03:43.391533","last_became_peered":"2021-02-10 11:03:43.391533","last_unstale":"2021-02-10 11:03:43.392222","last_undegraded":"2021-02-10 11:03:43.392222","last_fullsized":"2021-02-10 11:03:43.392222","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:03:42.751373","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:03:42.751373","last_clean_scrub_stamp":"2021-02-10 11:03:42.751373","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":"13","reported_epoch":"19","state":"active+clean","last_fresh":"2021-02-10 11:03:43.390513","last_change":"2021-02-10 11:03:38.740643","last_active":"2021-02-10 11:03:43.390513","last_peered":"2021-02-10 11:03:43.390513","last_clean":"2021-02-10 11:03:43.390513","last_became_active":"2021-02-10 11:03:38.740334","last_became_peered":"2021-02-10 11:03:38.740334","last_unstale":"2021-02-10 11:03:43.390513","last_undegraded":"2021-02-10 11:03:43.390513","last_fullsized":"2021-02-10 11:03:43.390513","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:03:37.729316","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:03:37.729316","last_clean_scrub_stamp":"2021-02-10 11:03:37.729316","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"20'7","reported_seq":"17","reported_epoch":"20","state":"active+clean","last_fresh":"2021-02-10 11:03:51.796205","last_change":"2021-02-10 11:03:42.756551","last_active":"2021-02-10 11:03:51.796205","last_peered":"2021-02-10 11:03:51.796205","last_clean":"2021-02-10 11:03:51.796205","last_became_active":"2021-02-10 11:03:42.756332","last_became_peered":"2021-02-10 11:03:42.756332","last_unstale":"2021-02-10 11:03:51.796205","last_undegraded":"2021-02-10 11:03:51.796205","last_fullsized":"2021-02-10 11:03:51.796205","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:03:41.747519","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:03:41.747519","last_clean_scrub_stamp":"2021-02-10 11:03:41.747519","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":566,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"13","reported_epoch":"19","state":"active+clean","last_fresh":"2021-02-10 11:03:43.390551","last_change":"2021-02-10 11:03:38.739319","last_active":"2021-02-10 11:03:43.390551","last_peered":"2021-02-10 11:03:43.390551","last_clean":"2021-02-10 11:03:43.390551","last_became_active":"2021-02-10 11:03:38.739150","last_became_peered":"2021-02-10 11:03:38.739150","last_unstale":"2021-02-10 11:03:43.390551","last_undegraded":"2021-02-10 11:03:43.390551","last_fullsized":"2021-02-10 11:03:43.390551","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:03:37.729316","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:03:37.729316","last_clean_scrub_stamp":"2021-02-10 11:03:37.729316","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"20'5","reported_seq":"15","reported_epoch":"20","state":"active+clean","last_fresh":"2021-02-10 11:03:51.795109","last_change":"2021-02-10 11:03:42.757597","last_active":"2021-02-10 11:03:51.795109","last_peered":"2021-02-10 11:03:51.795109","last_clean":"2021-02-10 11:03:51.795109","last_became_active":"2021-02-10 11:03:42.757396","last_became_peered":"2021-02-10 11:03:42.757396","last_unstale":"2021-02-10 11:03:51.795109","last_undegraded":"2021-02-10 11:03:51.795109","last_fullsized":"2021-02-10 11:03:51.795109","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:03:41.747519","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:03:41.747519","last_clean_scrub_stamp":"2021-02-10 11:03:41.747519","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":"9","reported_epoch":"19","state":"active+clean","last_fresh":"2021-02-10 11:03:43.393904","last_change":"2021-02-10 11:03:43.393584","last_active":"2021-02-10 11:03:43.393904","last_peered":"2021-02-10 11:03:43.393904","last_clean":"2021-02-10 11:03:43.393904","last_became_active":"2021-02-10 11:03:43.393360","last_became_peered":"2021-02-10 11:03:43.393360","last_unstale":"2021-02-10 11:03:43.393904","last_undegraded":"2021-02-10 11:03:43.393904","last_fullsized":"2021-02-10 11:03:43.393904","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:03:42.751373","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:03:42.751373","last_clean_scrub_stamp":"2021-02-10 11:03:42.751373","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":"13","reported_epoch":"19","state":"active+clean","last_fresh":"2021-02-10 11:03:43.389737","last_change":"2021-02-10 11:03:38.738931","last_active":"2021-02-10 11:03:43.389737","last_peered":"2021-02-10 11:03:43.389737","last_clean":"2021-02-10 11:03:43.389737","last_became_active":"2021-02-10 11:03:38.738675","last_became_peered":"2021-02-10 11:03:38.738675","last_unstale":"2021-02-10 11:03:43.389737","last_undegraded":"2021-02-10 11:03:43.389737","last_fullsized":"2021-02-10 11:03:43.389737","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:03:37.729316","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:03:37.729316","last_clean_scrub_stamp":"2021-02-10 11:03:37.729316","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"20'1","reported_seq":"11","reported_epoch":"20","state":"active+clean","last_fresh":"2021-02-10 11:03:44.882543","last_change":"2021-02-10 11:03:42.757799","last_active":"2021-02-10 11:03:44.882543","last_peered":"2021-02-10 11:03:44.882543","last_clean":"2021-02-10 11:03:44.882543","last_became_active":"2021-02-10 11:03:42.757538","last_became_peered":"2021-02-10 11:03:42.757538","last_unstale":"2021-02-10 11:03:44.882543","last_undegraded":"2021-02-10 11:03:44.882543","last_fullsized":"2021-02-10 11:03:44.882543","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:03:41.747519","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:03:41.747519","last_clean_scrub_stamp":"2021-02-10 11:03:41.747519","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":"9","reported_epoch":"19","state":"active+clean","last_fresh":"2021-02-10 11:03:43.391121","last_change":"2021-02-10 11:03:43.390550","last_active":"2021-02-10 11:03:43.391121","last_peered":"2021-02-10 11:03:43.391121","last_clean":"2021-02-10 11:03:43.391121","last_became_active":"2021-02-10 11:03:43.390234","last_became_peered":"2021-02-10 11:03:43.390234","last_unstale":"2021-02-10 11:03:43.391121","last_undegraded":"2021-02-10 11:03:43.391121","last_fullsized":"2021-02-10 11:03:43.391121","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:03:42.751373","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:03:42.751373","last_clean_scrub_stamp":"2021-02-10 11:03:42.751373","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"10","reported_epoch":"19","state":"active+clean","last_fresh":"2021-02-10 11:03:43.391840","last_change":"2021-02-10 11:03:42.759394","last_active":"2021-02-10 11:03:43.391840","last_peered":"2021-02-10 11:03:43.391840","last_clean":"2021-02-10 11:03:43.391840","last_became_active":"2021-02-10 11:03:42.758964","last_became_peered":"2021-02-10 11:03:42.758964","last_unstale":"2021-02-10 11:03:43.391840","last_undegraded":"2021-02-10 11:03:43.391840","last_fullsized":"2021-02-10 11:03:43.391840","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:03:41.747519","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:03:41.747519","last_clean_scrub_stamp":"2021-02-10 11:03:41.747519","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":"13","reported_epoch":"19","state":"active+clean","last_fresh":"2021-02-10 11:03:43.390542","last_change":"2021-02-10 11:03:38.739203","last_active":"2021-02-10 11:03:43.390542","last_peered":"2021-02-10 11:03:43.390542","last_clean":"2021-02-10 11:03:43.390542","last_became_active":"2021-02-10 11:03:38.739032","last_became_peered":"2021-02-10 11:03:38.739032","last_unstale":"2021-02-10 11:03:43.390542","last_undegraded":"2021-02-10 11:03:43.390542","last_fullsized":"2021-02-10 11:03:43.390542","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:03:37.729316","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:03:37.729316","last_clean_scrub_stamp":"2021-02-10 11:03:37.729316","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":"9","reported_epoch":"19","state":"active+clean","last_fresh":"2021-02-10 11:03:43.393643","last_change":"2021-02-10 11:03:43.393073","last_active":"2021-02-10 11:03:43.393643","last_peered":"2021-02-10 11:03:43.393643","last_clean":"2021-02-10 11:03:43.393643","last_became_active":"2021-02-10 11:03:43.392808","last_became_peered":"2021-02-10 11:03:43.392808","last_unstale":"2021-02-10 11:03:43.393643","last_undegraded":"2021-02-10 11:03:43.393643","last_fullsized":"2021-02-10 11:03:43.393643","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:03:42.751373","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:03:42.751373","last_clean_scrub_stamp":"2021-02-10 11:03:42.751373","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"20'8","reported_seq":"18","reported_epoch":"20","state":"active+clean","last_fresh":"2021-02-10 11:03:51.796513","last_change":"2021-02-10 11:03:42.757859","last_active":"2021-02-10 11:03:51.796513","last_peered":"2021-02-10 11:03:51.796513","last_clean":"2021-02-10 11:03:51.796513","last_became_active":"2021-02-10 11:03:42.757610","last_became_peered":"2021-02-10 11:03:42.757610","last_unstale":"2021-02-10 11:03:51.796513","last_undegraded":"2021-02-10 11:03:51.796513","last_fullsized":"2021-02-10 11:03:51.796513","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:03:41.747519","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:03:41.747519","last_clean_scrub_stamp":"2021-02-10 11:03:41.747519","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":"13","reported_epoch":"19","state":"active+clean","last_fresh":"2021-02-10 11:03:43.390448","last_change":"2021-02-10 11:03:38.740597","last_active":"2021-02-10 11:03:43.390448","last_peered":"2021-02-10 11:03:43.390448","last_clean":"2021-02-10 11:03:43.390448","last_became_active":"2021-02-10 11:03:38.740308","last_became_peered":"2021-02-10 11:03:38.740308","last_unstale":"2021-02-10 11:03:43.390448","last_undegraded":"2021-02-10 11:03:43.390448","last_fullsized":"2021-02-10 11:03:43.390448","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:03:37.729316","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:03:37.729316","last_clean_scrub_stamp":"2021-02-10 11:03:37.729316","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":"9","reported_epoch":"19","state":"active+clean","last_fresh":"2021-02-10 11:03:43.393290","last_change":"2021-02-10 11:03:43.392735","last_active":"2021-02-10 11:03:43.393290","last_peered":"2021-02-10 11:03:43.393290","last_clean":"2021-02-10 11:03:43.393290","last_became_active":"2021-02-10 11:03:43.392539","last_became_peered":"2021-02-10 11:03:43.392539","last_unstale":"2021-02-10 11:03:43.393290","last_undegraded":"2021-02-10 11:03:43.393290","last_fullsized":"2021-02-10 11:03:43.393290","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:03:42.751373","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:03:42.751373","last_clean_scrub_stamp":"2021-02-10 11:03:42.751373","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2286,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574857,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1048800,"kb_used_data":216,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323040,"statfs":{"total":96636764160,"available":95562792960,"internally_reserved":1073741824,"allocated":221184,"data_stored":69449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47399999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40100000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52100000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.627}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77000000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59499999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89100000000000001}]}]},{"osd":6,"up_from":13,"seq":55834574857,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1048792,"kb_used_data":208,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323048,"statfs":{"total":96636764160,"available":95562801152,"internally_reserved":1073741824,"allocated":212992,"data_stored":69289,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53500000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.436}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42099999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63100000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71399999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55600000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73999999999999999}]}]},{"osd":5,"up_from":13,"seq":55834574857,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1048788,"kb_used_data":204,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323052,"statfs":{"total":96636764160,"available":95562805248,"internally_reserved":1073741824,"allocated":208896,"data_stored":68757,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.871}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57199999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56000000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77000000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93200000000000005}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.04}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88800000000000001}]}]},{"osd":4,"up_from":13,"seq":55834574857,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1048784,"kb_used_data":200,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323056,"statfs":{"total":96636764160,"available":95562809344,"internally_reserved":1073741824,"allocated":204800,"data_stored":68723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.502}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72399999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55100000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0680000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.044}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64300000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2589999999999999}]}]},{"osd":0,"up_from":13,"seq":55834574856,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1048804,"kb_used_data":220,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323036,"statfs":{"total":96636764160,"available":95562788864,"internally_reserved":1073741824,"allocated":225280,"data_stored":70409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45300000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59099999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55200000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66200000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89900000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91100000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.02}]}]},{"osd":1,"up_from":13,"seq":55834574857,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1048796,"kb_used_data":212,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323044,"statfs":{"total":96636764160,"available":95562797056,"internally_reserved":1073741824,"allocated":217088,"data_stored":70249,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34499999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35999999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56499999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54200000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61299999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84099999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75}]}]},{"osd":2,"up_from":13,"seq":55834574856,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1048788,"kb_used_data":204,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323052,"statfs":{"total":96636764160,"available":95562805248,"internally_reserved":1073741824,"allocated":208896,"data_stored":68757,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66800000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42499999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41699999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0609999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0289999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92400000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86199999999999999}]}]},{"osd":3,"up_from":13,"seq":55834574856,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1048784,"kb_used_data":200,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323056,"statfs":{"total":96636764160,"available":95562809344,"internally_reserved":1073741824,"allocated":204800,"data_stored":68723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.372}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36099999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53200000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1299999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0529999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0780000000000001}]}]}]}} 2021-02-10T11:04:01.391 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph pg dump --format=json 2021-02-10T11:04:01.820 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:01.805 7fb4fb0c5700 1 Processor -- start 2021-02-10T11:04:01.821 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:01.805 7fb4fb0c5700 1 -- start start 2021-02-10T11:04:01.821 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:01.805 7fb4fb0c5700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fb4f40e5070 0x7fb4f41b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:01.821 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:01.805 7fb4fb0c5700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fb4f41b3e40 0x7fb4f407d690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:01.821 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:01.805 7fb4fb0c5700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb4f40e5470 0x7fb4f407dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:01.821 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:01.805 7fb4fb0c5700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4f407ed80 con 0x7fb4f41b3e40 2021-02-10T11:04:01.821 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:01.805 7fb4fb0c5700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4f407e2d0 con 0x7fb4f40e5070 2021-02-10T11:04:01.821 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:01.805 7fb4fb0c5700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4f407e4f0 con 0x7fb4f40e5470 2021-02-10T11:04:01.821 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:01.805 7fb4f3fff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fb4f41b3e40 0x7fb4f407d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:01.821 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:01.806 7fb4f3fff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fb4f41b3e40 0x7fb4f407d690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:49562/0 (socket says 172.21.2.102:49562) 2021-02-10T11:04:01.821 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:01.806 7fb4f3fff700 1 -- 172.21.2.102:0/3859769871 learned_addr learned my addr 172.21.2.102:0/3859769871 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:04:01.821 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:01.806 7fb4f3fff700 1 -- 172.21.2.102:0/3859769871 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb4f40e5470 msgr2=0x7fb4f407dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:04:01.822 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:01.806 7fb4f3fff700 1 --2- 172.21.2.102:0/3859769871 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb4f40e5470 0x7fb4f407dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:01.822 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:01.806 7fb4f3fff700 1 -- 172.21.2.102:0/3859769871 >> v2:172.21.2.113:3300/0 conn(0x7fb4f40e5070 msgr2=0x7fb4f41b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:04:01.822 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:01.806 7fb4f3fff700 1 --2- 172.21.2.102:0/3859769871 >> v2:172.21.2.113:3300/0 conn(0x7fb4f40e5070 0x7fb4f41b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:01.822 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:01.806 7fb4f3fff700 1 -- 172.21.2.102:0/3859769871 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4f407e710 con 0x7fb4f41b3e40 2021-02-10T11:04:01.822 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:01.806 7fb4f3fff700 1 --2- 172.21.2.102:0/3859769871 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb4f41b3e40 0x7fb4f407d690 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7fb4e4006700 tx=0x7fb4e4009b80).ready entity=mon.0 client_cookie=687ec82095102e53 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:04:01.822 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:01.806 7fb4f37fe700 1 -- 172.21.2.102:0/3859769871 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb4e4000c90 con 0x7fb4f41b3e40 2021-02-10T11:04:01.822 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:01.806 7fb4f37fe700 1 -- 172.21.2.102:0/3859769871 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4e400c0c0 con 0x7fb4f41b3e40 2021-02-10T11:04:01.822 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:01.806 7fb4f37fe700 1 -- 172.21.2.102:0/3859769871 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb4e400c520 con 0x7fb4f41b3e40 2021-02-10T11:04:01.822 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:01.806 7fb4fb0c5700 1 -- 172.21.2.102:0/3859769871 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb4f41b3e40 msgr2=0x7fb4f407d690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:01.822 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:01.806 7fb4fb0c5700 1 --2- 172.21.2.102:0/3859769871 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb4f41b3e40 0x7fb4f407d690 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7fb4e4006700 tx=0x7fb4e4009b80).stop 2021-02-10T11:04:01.822 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:01.807 7fb4fb0c5700 1 -- 172.21.2.102:0/3859769871 shutdown_connections 2021-02-10T11:04:01.822 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:01.807 7fb4fb0c5700 1 --2- 172.21.2.102:0/3859769871 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb4f40e5470 0x7fb4f407dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:01.823 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:01.807 7fb4fb0c5700 1 --2- 172.21.2.102:0/3859769871 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb4f41b3e40 0x7fb4f407d690 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:01.823 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:01.807 7fb4fb0c5700 1 --2- 172.21.2.102:0/3859769871 >> v2:172.21.2.113:3300/0 conn(0x7fb4f40e5070 0x7fb4f41b3820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:01.823 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:01.807 7fb4fb0c5700 1 -- 172.21.2.102:0/3859769871 >> 172.21.2.102:0/3859769871 conn(0x7fb4f40ce690 msgr2=0x7fb4f40e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:04:01.823 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:01.807 7fb4fb0c5700 1 -- 172.21.2.102:0/3859769871 shutdown_connections 2021-02-10T11:04:01.823 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:01.807 7fb4fb0c5700 1 -- 172.21.2.102:0/3859769871 wait complete. 2021-02-10T11:04:01.823 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:01.822 7fb4fb0c5700 1 Processor -- start 2021-02-10T11:04:01.823 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:01.822 7fb4fb0c5700 1 -- start start 2021-02-10T11:04:01.825 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:01.822 7fb4fb0c5700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb4f4192960 0x7fb4f4192d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:01.825 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:01.822 7fb4fb0c5700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb4f40ad110 0x7fb4f40ad510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:01.825 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:01.822 7fb4fb0c5700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb4f40adb30 0x7fb4f408e460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:01.825 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:01.822 7fb4fb0c5700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4f41a03e0 con 0x7fb4f40ad110 2021-02-10T11:04:01.825 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:01.822 7fb4fb0c5700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4f4193380 con 0x7fb4f40adb30 2021-02-10T11:04:01.825 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:01.822 7fb4fb0c5700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4f408ea80 con 0x7fb4f4192960 2021-02-10T11:04:01.825 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:01.822 7fb4f3fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb4f40ad110 0x7fb4f40ad510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:01.825 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:01.822 7fb4f3fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb4f40ad110 0x7fb4f40ad510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:49568/0 (socket says 172.21.2.102:49568) 2021-02-10T11:04:01.825 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:01.822 7fb4f3fff700 1 -- 172.21.2.102:0/3621515526 learned_addr learned my addr 172.21.2.102:0/3621515526 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:04:01.826 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:01.823 7fb4f8ee1700 1 --2- 172.21.2.102:0/3621515526 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb4f4192960 0x7fb4f4192d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:01.826 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:01.823 7fb4f96e2700 1 --2- 172.21.2.102:0/3621515526 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb4f40adb30 0x7fb4f408e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:01.826 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:01.823 7fb4f3fff700 1 -- 172.21.2.102:0/3621515526 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb4f4192960 msgr2=0x7fb4f4192d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:01.826 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:01.823 7fb4f3fff700 1 --2- 172.21.2.102:0/3621515526 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb4f4192960 0x7fb4f4192d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:01.826 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:01.823 7fb4f3fff700 1 -- 172.21.2.102:0/3621515526 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb4f40adb30 msgr2=0x7fb4f408e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:01.826 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:01.823 7fb4f3fff700 1 --2- 172.21.2.102:0/3621515526 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb4f40adb30 0x7fb4f408e460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:01.826 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:01.823 7fb4f3fff700 1 -- 172.21.2.102:0/3621515526 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4f408eca0 con 0x7fb4f40ad110 2021-02-10T11:04:01.826 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:01.823 7fb4f8ee1700 1 --2- 172.21.2.102:0/3621515526 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb4f4192960 0x7fb4f4192d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:04:01.826 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:01.823 7fb4f96e2700 1 --2- 172.21.2.102:0/3621515526 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb4f40adb30 0x7fb4f408e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:04:01.826 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:01.823 7fb4f3fff700 1 --2- 172.21.2.102:0/3621515526 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb4f40ad110 0x7fb4f40ad510 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7fb4e4009fb0 tx=0x7fb4e400b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:04:01.826 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:01.823 7fb4f17fa700 1 -- 172.21.2.102:0/3621515526 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb4e400bd80 con 0x7fb4f40ad110 2021-02-10T11:04:01.826 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:01.823 7fb4f17fa700 1 -- 172.21.2.102:0/3621515526 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4e4006030 con 0x7fb4f40ad110 2021-02-10T11:04:01.826 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:01.823 7fb4f17fa700 1 -- 172.21.2.102:0/3621515526 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb4e40052e0 con 0x7fb4f40ad110 2021-02-10T11:04:01.827 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:01.823 7fb4fb0c5700 1 -- 172.21.2.102:0/3621515526 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4f408eca0 con 0x7fb4f40ad110 2021-02-10T11:04:01.827 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:01.823 7fb4fb0c5700 1 -- 172.21.2.102:0/3621515526 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4f408f230 con 0x7fb4f40ad110 2021-02-10T11:04:01.827 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:01.824 7fb4f17fa700 1 -- 172.21.2.102:0/3621515526 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fb4e4006210 con 0x7fb4f40ad110 2021-02-10T11:04:01.827 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:01.824 7fb4f17fa700 1 --2- 172.21.2.102:0/3621515526 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb4d80186c0 0x7fb4d801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:01.827 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:01.824 7fb4f17fa700 1 -- 172.21.2.102:0/3621515526 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fb4e4032640 con 0x7fb4f40ad110 2021-02-10T11:04:01.827 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:01.824 7fb4f8ee1700 1 --2- 172.21.2.102:0/3621515526 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb4d80186c0 0x7fb4d801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:01.827 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:01.824 7fb4f8ee1700 1 --2- 172.21.2.102:0/3621515526 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb4d80186c0 0x7fb4d801abc0 crc :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:04:01.854 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:01.855 7fb4fb0c5700 1 -- 172.21.2.102:0/3621515526 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4f4107120 con 0x7fb4f40ad110 2021-02-10T11:04:01.856 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:01.857 7fb4f17fa700 1 -- 172.21.2.102:0/3621515526 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fb4e4006810 con 0x7fb4f40ad110 2021-02-10T11:04:01.990 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:01.991 7fb4fb0c5700 1 -- 172.21.2.102:0/3621515526 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "pg dump", "target": ["mgr", ""], "format": "json"}) v1 -- 0x7fb4f408f230 con 0x7fb4d80186c0 2021-02-10T11:04:01.992 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:01.993 7fb4f17fa700 1 -- 172.21.2.102:0/3621515526 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 dumped all) v1 ==== 18+0+77157 (crc 0 0 0) 0x7fb4dc006dd0 con 0x7fb4d80186c0 2021-02-10T11:04:01.994 INFO:teuthology.orchestra.run.gibba002.stderr:dumped all 2021-02-10T11:04:01.995 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:04:01.995 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:01.996 7fb4fb0c5700 1 -- 172.21.2.102:0/3621515526 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb4d80186c0 msgr2=0x7fb4d801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:01.995 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:01.996 7fb4fb0c5700 1 --2- 172.21.2.102:0/3621515526 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb4d80186c0 0x7fb4d801abc0 crc :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:01.995 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:01.996 7fb4fb0c5700 1 -- 172.21.2.102:0/3621515526 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb4f40ad110 msgr2=0x7fb4f40ad510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:01.995 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:01.996 7fb4fb0c5700 1 --2- 172.21.2.102:0/3621515526 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb4f40ad110 0x7fb4f40ad510 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7fb4e4009fb0 tx=0x7fb4e400b990).stop 2021-02-10T11:04:01.996 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:01.996 7fb4fb0c5700 1 -- 172.21.2.102:0/3621515526 shutdown_connections 2021-02-10T11:04:01.996 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:01.996 7fb4fb0c5700 1 --2- 172.21.2.102:0/3621515526 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb4d80186c0 0x7fb4d801abc0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:01.996 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:01.996 7fb4fb0c5700 1 --2- 172.21.2.102:0/3621515526 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb4f40adb30 0x7fb4f408e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:01.996 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:01.996 7fb4fb0c5700 1 --2- 172.21.2.102:0/3621515526 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb4f40ad110 0x7fb4f40ad510 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:01.996 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:01.996 7fb4fb0c5700 1 --2- 172.21.2.102:0/3621515526 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb4f4192960 0x7fb4f4192d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:01.996 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:01.996 7fb4fb0c5700 1 -- 172.21.2.102:0/3621515526 >> 172.21.2.102:0/3621515526 conn(0x7fb4f40891f0 msgr2=0x7fb4f41a0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:04:01.996 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:01.997 7fb4fb0c5700 1 -- 172.21.2.102:0/3621515526 shutdown_connections 2021-02-10T11:04:01.996 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:01.997 7fb4fb0c5700 1 -- 172.21.2.102:0/3621515526 wait complete. 2021-02-10T11:04:02.004 INFO:teuthology.orchestra.run.gibba002.stdout:{"pg_ready":true,"pg_map":{"version":38,"stamp":"2021-02-10 11:04:01.679333","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2286,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":8,"num_per_pool_osds":8,"kb":754974720,"kb_used":8390336,"kb_used_data":1664,"kb_used_omap":0,"kb_used_meta":8388608,"kb_avail":746584384,"statfs":{"total":773094113280,"available":764502409216,"internally_reserved":8589934592,"allocated":1703936,"data_stored":554356,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8589934592},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":760,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":33,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":11,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002124"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":"13","reported_epoch":"19","state":"active+clean","last_fresh":"2021-02-10 11:03:43.388602","last_change":"2021-02-10 11:03:38.737712","last_active":"2021-02-10 11:03:43.388602","last_peered":"2021-02-10 11:03:43.388602","last_clean":"2021-02-10 11:03:43.388602","last_became_active":"2021-02-10 11:03:38.737435","last_became_peered":"2021-02-10 11:03:38.737435","last_unstale":"2021-02-10 11:03:43.388602","last_undegraded":"2021-02-10 11:03:43.388602","last_fullsized":"2021-02-10 11:03:43.388602","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:03:37.729316","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:03:37.729316","last_clean_scrub_stamp":"2021-02-10 11:03:37.729316","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.4","version":"20'8","reported_seq":"18","reported_epoch":"20","state":"active+clean","last_fresh":"2021-02-10 11:03:51.798768","last_change":"2021-02-10 11:03:42.760313","last_active":"2021-02-10 11:03:51.798768","last_peered":"2021-02-10 11:03:51.798768","last_clean":"2021-02-10 11:03:51.798768","last_became_active":"2021-02-10 11:03:42.760064","last_became_peered":"2021-02-10 11:03:42.760064","last_unstale":"2021-02-10 11:03:51.798768","last_undegraded":"2021-02-10 11:03:51.798768","last_fullsized":"2021-02-10 11:03:51.798768","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:03:41.747519","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:03:41.747519","last_clean_scrub_stamp":"2021-02-10 11:03:41.747519","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1526,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":"9","reported_epoch":"19","state":"active+clean","last_fresh":"2021-02-10 11:03:43.391539","last_change":"2021-02-10 11:03:43.391081","last_active":"2021-02-10 11:03:43.391539","last_peered":"2021-02-10 11:03:43.391539","last_clean":"2021-02-10 11:03:43.391539","last_became_active":"2021-02-10 11:03:43.390918","last_became_peered":"2021-02-10 11:03:43.390918","last_unstale":"2021-02-10 11:03:43.391539","last_undegraded":"2021-02-10 11:03:43.391539","last_fullsized":"2021-02-10 11:03:43.391539","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:03:42.751373","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:03:42.751373","last_clean_scrub_stamp":"2021-02-10 11:03:42.751373","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":"13","reported_epoch":"19","state":"active+clean","last_fresh":"2021-02-10 11:03:43.388603","last_change":"2021-02-10 11:03:38.738386","last_active":"2021-02-10 11:03:43.388603","last_peered":"2021-02-10 11:03:43.388603","last_clean":"2021-02-10 11:03:43.388603","last_became_active":"2021-02-10 11:03:38.738144","last_became_peered":"2021-02-10 11:03:38.738144","last_unstale":"2021-02-10 11:03:43.388603","last_undegraded":"2021-02-10 11:03:43.388603","last_fullsized":"2021-02-10 11:03:43.388603","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:03:37.729316","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:03:37.729316","last_clean_scrub_stamp":"2021-02-10 11:03:37.729316","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"20'6","reported_seq":"16","reported_epoch":"20","state":"active+clean","last_fresh":"2021-02-10 11:03:51.796270","last_change":"2021-02-10 11:03:42.758151","last_active":"2021-02-10 11:03:51.796270","last_peered":"2021-02-10 11:03:51.796270","last_clean":"2021-02-10 11:03:51.796270","last_became_active":"2021-02-10 11:03:42.757926","last_became_peered":"2021-02-10 11:03:42.757926","last_unstale":"2021-02-10 11:03:51.796270","last_undegraded":"2021-02-10 11:03:51.796270","last_fullsized":"2021-02-10 11:03:51.796270","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:03:41.747519","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:03:41.747519","last_clean_scrub_stamp":"2021-02-10 11:03:41.747519","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":160,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":"9","reported_epoch":"19","state":"active+clean","last_fresh":"2021-02-10 11:03:43.393614","last_change":"2021-02-10 11:03:43.392999","last_active":"2021-02-10 11:03:43.393614","last_peered":"2021-02-10 11:03:43.393614","last_clean":"2021-02-10 11:03:43.393614","last_became_active":"2021-02-10 11:03:43.392742","last_became_peered":"2021-02-10 11:03:43.392742","last_unstale":"2021-02-10 11:03:43.393614","last_undegraded":"2021-02-10 11:03:43.393614","last_fullsized":"2021-02-10 11:03:43.393614","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:03:42.751373","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:03:42.751373","last_clean_scrub_stamp":"2021-02-10 11:03:42.751373","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":"9","reported_epoch":"19","state":"active+clean","last_fresh":"2021-02-10 11:03:43.394812","last_change":"2021-02-10 11:03:43.394157","last_active":"2021-02-10 11:03:43.394812","last_peered":"2021-02-10 11:03:43.394812","last_clean":"2021-02-10 11:03:43.394812","last_became_active":"2021-02-10 11:03:43.393997","last_became_peered":"2021-02-10 11:03:43.393997","last_unstale":"2021-02-10 11:03:43.394812","last_undegraded":"2021-02-10 11:03:43.394812","last_fullsized":"2021-02-10 11:03:43.394812","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:03:42.751373","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:03:42.751373","last_clean_scrub_stamp":"2021-02-10 11:03:42.751373","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":"13","reported_epoch":"19","state":"active+clean","last_fresh":"2021-02-10 11:03:43.388728","last_change":"2021-02-10 11:03:38.738326","last_active":"2021-02-10 11:03:43.388728","last_peered":"2021-02-10 11:03:43.388728","last_clean":"2021-02-10 11:03:43.388728","last_became_active":"2021-02-10 11:03:38.738088","last_became_peered":"2021-02-10 11:03:38.738088","last_unstale":"2021-02-10 11:03:43.388728","last_undegraded":"2021-02-10 11:03:43.388728","last_fullsized":"2021-02-10 11:03:43.388728","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:03:37.729316","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:03:37.729316","last_clean_scrub_stamp":"2021-02-10 11:03:37.729316","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"20'4","reported_seq":"14","reported_epoch":"20","state":"active+clean","last_fresh":"2021-02-10 11:03:51.798316","last_change":"2021-02-10 11:03:42.759030","last_active":"2021-02-10 11:03:51.798316","last_peered":"2021-02-10 11:03:51.798316","last_clean":"2021-02-10 11:03:51.798316","last_became_active":"2021-02-10 11:03:42.758801","last_became_peered":"2021-02-10 11:03:42.758801","last_unstale":"2021-02-10 11:03:51.798316","last_undegraded":"2021-02-10 11:03:51.798316","last_fullsized":"2021-02-10 11:03:51.798316","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:03:41.747519","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:03:41.747519","last_clean_scrub_stamp":"2021-02-10 11:03:41.747519","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":"9","reported_epoch":"19","state":"active+clean","last_fresh":"2021-02-10 11:03:43.392222","last_change":"2021-02-10 11:03:43.391781","last_active":"2021-02-10 11:03:43.392222","last_peered":"2021-02-10 11:03:43.392222","last_clean":"2021-02-10 11:03:43.392222","last_became_active":"2021-02-10 11:03:43.391533","last_became_peered":"2021-02-10 11:03:43.391533","last_unstale":"2021-02-10 11:03:43.392222","last_undegraded":"2021-02-10 11:03:43.392222","last_fullsized":"2021-02-10 11:03:43.392222","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:03:42.751373","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:03:42.751373","last_clean_scrub_stamp":"2021-02-10 11:03:42.751373","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":"13","reported_epoch":"19","state":"active+clean","last_fresh":"2021-02-10 11:03:43.390513","last_change":"2021-02-10 11:03:38.740643","last_active":"2021-02-10 11:03:43.390513","last_peered":"2021-02-10 11:03:43.390513","last_clean":"2021-02-10 11:03:43.390513","last_became_active":"2021-02-10 11:03:38.740334","last_became_peered":"2021-02-10 11:03:38.740334","last_unstale":"2021-02-10 11:03:43.390513","last_undegraded":"2021-02-10 11:03:43.390513","last_fullsized":"2021-02-10 11:03:43.390513","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:03:37.729316","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:03:37.729316","last_clean_scrub_stamp":"2021-02-10 11:03:37.729316","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"20'7","reported_seq":"17","reported_epoch":"20","state":"active+clean","last_fresh":"2021-02-10 11:03:51.796205","last_change":"2021-02-10 11:03:42.756551","last_active":"2021-02-10 11:03:51.796205","last_peered":"2021-02-10 11:03:51.796205","last_clean":"2021-02-10 11:03:51.796205","last_became_active":"2021-02-10 11:03:42.756332","last_became_peered":"2021-02-10 11:03:42.756332","last_unstale":"2021-02-10 11:03:51.796205","last_undegraded":"2021-02-10 11:03:51.796205","last_fullsized":"2021-02-10 11:03:51.796205","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:03:41.747519","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:03:41.747519","last_clean_scrub_stamp":"2021-02-10 11:03:41.747519","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":566,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"13","reported_epoch":"19","state":"active+clean","last_fresh":"2021-02-10 11:03:43.390551","last_change":"2021-02-10 11:03:38.739319","last_active":"2021-02-10 11:03:43.390551","last_peered":"2021-02-10 11:03:43.390551","last_clean":"2021-02-10 11:03:43.390551","last_became_active":"2021-02-10 11:03:38.739150","last_became_peered":"2021-02-10 11:03:38.739150","last_unstale":"2021-02-10 11:03:43.390551","last_undegraded":"2021-02-10 11:03:43.390551","last_fullsized":"2021-02-10 11:03:43.390551","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:03:37.729316","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:03:37.729316","last_clean_scrub_stamp":"2021-02-10 11:03:37.729316","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"20'5","reported_seq":"15","reported_epoch":"20","state":"active+clean","last_fresh":"2021-02-10 11:03:51.795109","last_change":"2021-02-10 11:03:42.757597","last_active":"2021-02-10 11:03:51.795109","last_peered":"2021-02-10 11:03:51.795109","last_clean":"2021-02-10 11:03:51.795109","last_became_active":"2021-02-10 11:03:42.757396","last_became_peered":"2021-02-10 11:03:42.757396","last_unstale":"2021-02-10 11:03:51.795109","last_undegraded":"2021-02-10 11:03:51.795109","last_fullsized":"2021-02-10 11:03:51.795109","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:03:41.747519","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:03:41.747519","last_clean_scrub_stamp":"2021-02-10 11:03:41.747519","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":"9","reported_epoch":"19","state":"active+clean","last_fresh":"2021-02-10 11:03:43.393904","last_change":"2021-02-10 11:03:43.393584","last_active":"2021-02-10 11:03:43.393904","last_peered":"2021-02-10 11:03:43.393904","last_clean":"2021-02-10 11:03:43.393904","last_became_active":"2021-02-10 11:03:43.393360","last_became_peered":"2021-02-10 11:03:43.393360","last_unstale":"2021-02-10 11:03:43.393904","last_undegraded":"2021-02-10 11:03:43.393904","last_fullsized":"2021-02-10 11:03:43.393904","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:03:42.751373","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:03:42.751373","last_clean_scrub_stamp":"2021-02-10 11:03:42.751373","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":"13","reported_epoch":"19","state":"active+clean","last_fresh":"2021-02-10 11:03:43.389737","last_change":"2021-02-10 11:03:38.738931","last_active":"2021-02-10 11:03:43.389737","last_peered":"2021-02-10 11:03:43.389737","last_clean":"2021-02-10 11:03:43.389737","last_became_active":"2021-02-10 11:03:38.738675","last_became_peered":"2021-02-10 11:03:38.738675","last_unstale":"2021-02-10 11:03:43.389737","last_undegraded":"2021-02-10 11:03:43.389737","last_fullsized":"2021-02-10 11:03:43.389737","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:03:37.729316","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:03:37.729316","last_clean_scrub_stamp":"2021-02-10 11:03:37.729316","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"20'1","reported_seq":"11","reported_epoch":"20","state":"active+clean","last_fresh":"2021-02-10 11:03:44.882543","last_change":"2021-02-10 11:03:42.757799","last_active":"2021-02-10 11:03:44.882543","last_peered":"2021-02-10 11:03:44.882543","last_clean":"2021-02-10 11:03:44.882543","last_became_active":"2021-02-10 11:03:42.757538","last_became_peered":"2021-02-10 11:03:42.757538","last_unstale":"2021-02-10 11:03:44.882543","last_undegraded":"2021-02-10 11:03:44.882543","last_fullsized":"2021-02-10 11:03:44.882543","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:03:41.747519","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:03:41.747519","last_clean_scrub_stamp":"2021-02-10 11:03:41.747519","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":"9","reported_epoch":"19","state":"active+clean","last_fresh":"2021-02-10 11:03:43.391121","last_change":"2021-02-10 11:03:43.390550","last_active":"2021-02-10 11:03:43.391121","last_peered":"2021-02-10 11:03:43.391121","last_clean":"2021-02-10 11:03:43.391121","last_became_active":"2021-02-10 11:03:43.390234","last_became_peered":"2021-02-10 11:03:43.390234","last_unstale":"2021-02-10 11:03:43.391121","last_undegraded":"2021-02-10 11:03:43.391121","last_fullsized":"2021-02-10 11:03:43.391121","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:03:42.751373","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:03:42.751373","last_clean_scrub_stamp":"2021-02-10 11:03:42.751373","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"10","reported_epoch":"19","state":"active+clean","last_fresh":"2021-02-10 11:03:43.391840","last_change":"2021-02-10 11:03:42.759394","last_active":"2021-02-10 11:03:43.391840","last_peered":"2021-02-10 11:03:43.391840","last_clean":"2021-02-10 11:03:43.391840","last_became_active":"2021-02-10 11:03:42.758964","last_became_peered":"2021-02-10 11:03:42.758964","last_unstale":"2021-02-10 11:03:43.391840","last_undegraded":"2021-02-10 11:03:43.391840","last_fullsized":"2021-02-10 11:03:43.391840","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:03:41.747519","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:03:41.747519","last_clean_scrub_stamp":"2021-02-10 11:03:41.747519","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":"13","reported_epoch":"19","state":"active+clean","last_fresh":"2021-02-10 11:03:43.390542","last_change":"2021-02-10 11:03:38.739203","last_active":"2021-02-10 11:03:43.390542","last_peered":"2021-02-10 11:03:43.390542","last_clean":"2021-02-10 11:03:43.390542","last_became_active":"2021-02-10 11:03:38.739032","last_became_peered":"2021-02-10 11:03:38.739032","last_unstale":"2021-02-10 11:03:43.390542","last_undegraded":"2021-02-10 11:03:43.390542","last_fullsized":"2021-02-10 11:03:43.390542","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:03:37.729316","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:03:37.729316","last_clean_scrub_stamp":"2021-02-10 11:03:37.729316","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":"9","reported_epoch":"19","state":"active+clean","last_fresh":"2021-02-10 11:03:43.393643","last_change":"2021-02-10 11:03:43.393073","last_active":"2021-02-10 11:03:43.393643","last_peered":"2021-02-10 11:03:43.393643","last_clean":"2021-02-10 11:03:43.393643","last_became_active":"2021-02-10 11:03:43.392808","last_became_peered":"2021-02-10 11:03:43.392808","last_unstale":"2021-02-10 11:03:43.393643","last_undegraded":"2021-02-10 11:03:43.393643","last_fullsized":"2021-02-10 11:03:43.393643","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:03:42.751373","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:03:42.751373","last_clean_scrub_stamp":"2021-02-10 11:03:42.751373","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"20'8","reported_seq":"18","reported_epoch":"20","state":"active+clean","last_fresh":"2021-02-10 11:03:51.796513","last_change":"2021-02-10 11:03:42.757859","last_active":"2021-02-10 11:03:51.796513","last_peered":"2021-02-10 11:03:51.796513","last_clean":"2021-02-10 11:03:51.796513","last_became_active":"2021-02-10 11:03:42.757610","last_became_peered":"2021-02-10 11:03:42.757610","last_unstale":"2021-02-10 11:03:51.796513","last_undegraded":"2021-02-10 11:03:51.796513","last_fullsized":"2021-02-10 11:03:51.796513","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:03:41.747519","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:03:41.747519","last_clean_scrub_stamp":"2021-02-10 11:03:41.747519","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":"13","reported_epoch":"19","state":"active+clean","last_fresh":"2021-02-10 11:03:43.390448","last_change":"2021-02-10 11:03:38.740597","last_active":"2021-02-10 11:03:43.390448","last_peered":"2021-02-10 11:03:43.390448","last_clean":"2021-02-10 11:03:43.390448","last_became_active":"2021-02-10 11:03:38.740308","last_became_peered":"2021-02-10 11:03:38.740308","last_unstale":"2021-02-10 11:03:43.390448","last_undegraded":"2021-02-10 11:03:43.390448","last_fullsized":"2021-02-10 11:03:43.390448","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:03:37.729316","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:03:37.729316","last_clean_scrub_stamp":"2021-02-10 11:03:37.729316","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":"9","reported_epoch":"19","state":"active+clean","last_fresh":"2021-02-10 11:03:43.393290","last_change":"2021-02-10 11:03:43.392735","last_active":"2021-02-10 11:03:43.393290","last_peered":"2021-02-10 11:03:43.393290","last_clean":"2021-02-10 11:03:43.393290","last_became_active":"2021-02-10 11:03:43.392539","last_became_peered":"2021-02-10 11:03:43.392539","last_unstale":"2021-02-10 11:03:43.393290","last_undegraded":"2021-02-10 11:03:43.393290","last_fullsized":"2021-02-10 11:03:43.393290","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:03:42.751373","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:03:42.751373","last_clean_scrub_stamp":"2021-02-10 11:03:42.751373","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2286,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574857,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1048800,"kb_used_data":216,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323040,"statfs":{"total":96636764160,"available":95562792960,"internally_reserved":1073741824,"allocated":221184,"data_stored":69449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47399999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40100000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52100000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.627}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77000000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59499999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89100000000000001}]}]},{"osd":6,"up_from":13,"seq":55834574857,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1048792,"kb_used_data":208,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323048,"statfs":{"total":96636764160,"available":95562801152,"internally_reserved":1073741824,"allocated":212992,"data_stored":69289,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53500000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.436}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42099999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63100000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71399999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55600000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73999999999999999}]}]},{"osd":5,"up_from":13,"seq":55834574857,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1048788,"kb_used_data":204,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323052,"statfs":{"total":96636764160,"available":95562805248,"internally_reserved":1073741824,"allocated":208896,"data_stored":68757,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.871}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57199999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56000000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77000000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93200000000000005}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.04}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88800000000000001}]}]},{"osd":4,"up_from":13,"seq":55834574857,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1048784,"kb_used_data":200,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323056,"statfs":{"total":96636764160,"available":95562809344,"internally_reserved":1073741824,"allocated":204800,"data_stored":68723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.502}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72399999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55100000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0680000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.044}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64300000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2589999999999999}]}]},{"osd":0,"up_from":13,"seq":55834574856,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1048804,"kb_used_data":220,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323036,"statfs":{"total":96636764160,"available":95562788864,"internally_reserved":1073741824,"allocated":225280,"data_stored":70409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45300000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59099999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55200000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66200000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89900000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91100000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.02}]}]},{"osd":1,"up_from":13,"seq":55834574857,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1048796,"kb_used_data":212,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323044,"statfs":{"total":96636764160,"available":95562797056,"internally_reserved":1073741824,"allocated":217088,"data_stored":70249,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34499999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35999999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56499999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54200000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61299999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84099999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75}]}]},{"osd":2,"up_from":13,"seq":55834574856,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1048788,"kb_used_data":204,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323052,"statfs":{"total":96636764160,"available":95562805248,"internally_reserved":1073741824,"allocated":208896,"data_stored":68757,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66800000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42499999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41699999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0609999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0289999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92400000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86199999999999999}]}]},{"osd":3,"up_from":13,"seq":55834574856,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1048784,"kb_used_data":200,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323056,"statfs":{"total":96636764160,"available":95562809344,"internally_reserved":1073741824,"allocated":204800,"data_stored":68723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.372}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36099999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53200000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1299999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0529999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0780000000000001}]}]}]}} 2021-02-10T11:04:02.006 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph status --format=json-pretty 2021-02-10T11:04:02.431 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:02.416 7f69a73ad700 1 Processor -- start 2021-02-10T11:04:02.432 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:02.416 7f69a73ad700 1 -- start start 2021-02-10T11:04:02.432 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:02.416 7f69a73ad700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f69a01b0470 0x7f69a01b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:02.432 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:02.416 7f69a73ad700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f69a00f20e0 0x7f69a00f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:02.432 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:02.416 7f69a73ad700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69a00f0b50 con 0x7f69a00ef610 2021-02-10T11:04:02.432 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:02.416 7f69a73ad700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69a00f0820 con 0x7f69a00f20e0 2021-02-10T11:04:02.432 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:02.416 7f69a73ad700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69a00f24e0 con 0x7f69a01b0470 2021-02-10T11:04:02.432 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:02.416 7f69a59ca700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f69a00f20e0 0x7f69a00f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:02.432 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:02.417 7f69a59ca700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f69a00f20e0 0x7f69a00f0200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:49576/0 (socket says 172.21.2.102:49576) 2021-02-10T11:04:02.432 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:02.417 7f69a49c8700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f69a00ef610 0x7f69a00efa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:41800/0 (socket says 172.21.2.102:41800) 2021-02-10T11:04:02.432 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:02.417 7f69a59ca700 1 -- 172.21.2.102:0/4112745231 learned_addr learned my addr 172.21.2.102:0/4112745231 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:04:02.432 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:02.417 7f69a51c9700 1 --2- 172.21.2.102:0/4112745231 >> v2:172.21.2.113:3300/0 conn(0x7f69a01b0470 0x7f69a01b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:02.432 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:02.417 7f69a59ca700 1 -- 172.21.2.102:0/4112745231 >> v2:172.21.2.113:3300/0 conn(0x7f69a01b0470 msgr2=0x7f69a01b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:04:02.433 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:02.417 7f69a59ca700 1 --2- 172.21.2.102:0/4112745231 >> v2:172.21.2.113:3300/0 conn(0x7f69a01b0470 0x7f69a01b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:02.433 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:02.417 7f69a59ca700 1 -- 172.21.2.102:0/4112745231 >> v1:172.21.2.102:6789/0 conn(0x7f69a00ef610 legacy=0x7f69a00efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:02.433 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:02.417 7f69a59ca700 1 -- 172.21.2.102:0/4112745231 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69a00e52e0 con 0x7f69a00f20e0 2021-02-10T11:04:02.433 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:02.417 7f69a51c9700 1 --2- 172.21.2.102:0/4112745231 >> v2:172.21.2.113:3300/0 conn(0x7f69a01b0470 0x7f69a01b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:04:02.433 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:02.417 7f69a59ca700 1 --2- 172.21.2.102:0/4112745231 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f69a00f20e0 0x7f69a00f0200 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7f6998006f00 tx=0x7f6998009f10).ready entity=mon.0 client_cookie=87b7c575452c4993 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:04:02.433 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:02.417 7f6997fff700 1 -- 172.21.2.102:0/4112745231 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f699800b3b0 con 0x7f69a00f20e0 2021-02-10T11:04:02.433 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:02.417 7f6997fff700 1 -- 172.21.2.102:0/4112745231 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f699800b630 con 0x7f69a00f20e0 2021-02-10T11:04:02.433 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:02.417 7f6997fff700 1 -- 172.21.2.102:0/4112745231 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f699800bb50 con 0x7f69a00f20e0 2021-02-10T11:04:02.433 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:02.418 7f69a73ad700 1 -- 172.21.2.102:0/4112745231 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f69a00f20e0 msgr2=0x7f69a00f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:02.433 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:02.418 7f69a73ad700 1 --2- 172.21.2.102:0/4112745231 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f69a00f20e0 0x7f69a00f0200 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7f6998006f00 tx=0x7f6998009f10).stop 2021-02-10T11:04:02.433 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:02.418 7f69a73ad700 1 -- 172.21.2.102:0/4112745231 shutdown_connections 2021-02-10T11:04:02.433 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:02.418 7f69a73ad700 1 --2- 172.21.2.102:0/4112745231 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f69a00f20e0 0x7f69a00f0200 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:02.433 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:02.418 7f69a73ad700 1 --2- 172.21.2.102:0/4112745231 >> v2:172.21.2.113:3300/0 conn(0x7f69a01b0470 0x7f69a01b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:02.434 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:02.418 7f69a73ad700 1 -- 172.21.2.102:0/4112745231 >> 172.21.2.102:0/4112745231 conn(0x7f69a00ce3e0 msgr2=0x7f69a00e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:04:02.434 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:02.418 7f69a73ad700 1 -- 172.21.2.102:0/4112745231 shutdown_connections 2021-02-10T11:04:02.434 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:02.419 7f69a73ad700 1 -- 172.21.2.102:0/4112745231 wait complete. 2021-02-10T11:04:02.435 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:02.433 7f69a73ad700 1 Processor -- start 2021-02-10T11:04:02.435 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:02.433 7f69a73ad700 1 -- start start 2021-02-10T11:04:02.435 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:02.433 7f69a73ad700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f69a01885a0 0x7f69a01889a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:02.435 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:02.434 7f69a73ad700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f69a0188fc0 0x7f69a01893c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:02.436 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:02.434 7f69a73ad700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f69a00bf300 0x7f69a00bf700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:02.436 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:02.434 7f69a73ad700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69a0188020 con 0x7f69a01885a0 2021-02-10T11:04:02.436 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:02.434 7f69a73ad700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69a00bfd20 con 0x7f69a0188fc0 2021-02-10T11:04:02.436 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:02.434 7f69a73ad700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69a00bff40 con 0x7f69a00bf300 2021-02-10T11:04:02.436 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:02.434 7f69a51c9700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f69a01885a0 0x7f69a01889a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:02.436 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:02.434 7f69a51c9700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f69a01885a0 0x7f69a01889a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:49578/0 (socket says 172.21.2.102:49578) 2021-02-10T11:04:02.436 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:02.434 7f69a51c9700 1 -- 172.21.2.102:0/2906447467 learned_addr learned my addr 172.21.2.102:0/2906447467 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:04:02.436 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:02.434 7f69a59ca700 1 --2- 172.21.2.102:0/2906447467 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f69a00bf300 0x7f69a00bf700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:02.436 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:02.434 7f69a49c8700 1 --2- 172.21.2.102:0/2906447467 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f69a0188fc0 0x7f69a01893c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:02.436 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:02.434 7f69a51c9700 1 -- 172.21.2.102:0/2906447467 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f69a00bf300 msgr2=0x7f69a00bf700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:02.436 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:02.434 7f69a51c9700 1 --2- 172.21.2.102:0/2906447467 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f69a00bf300 0x7f69a00bf700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:02.436 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:02.434 7f69a51c9700 1 -- 172.21.2.102:0/2906447467 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f69a0188fc0 msgr2=0x7f69a01893c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:02.436 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:02.434 7f69a51c9700 1 --2- 172.21.2.102:0/2906447467 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f69a0188fc0 0x7f69a01893c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:02.436 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:02.434 7f69a51c9700 1 -- 172.21.2.102:0/2906447467 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69a00c0160 con 0x7f69a01885a0 2021-02-10T11:04:02.437 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:02.434 7f69a51c9700 1 --2- 172.21.2.102:0/2906447467 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f69a01885a0 0x7f69a01889a0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7f699c00b580 tx=0x7f699c00b920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:04:02.437 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:02.435 7f6995ffb700 1 -- 172.21.2.102:0/2906447467 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f699c00bd10 con 0x7f69a01885a0 2021-02-10T11:04:02.437 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:02.435 7f6995ffb700 1 -- 172.21.2.102:0/2906447467 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f699c000c90 con 0x7f69a01885a0 2021-02-10T11:04:02.437 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:02.435 7f69a73ad700 1 -- 172.21.2.102:0/2906447467 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69a00c0160 con 0x7f69a01885a0 2021-02-10T11:04:02.437 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:02.435 7f6995ffb700 1 -- 172.21.2.102:0/2906447467 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f699c0022b0 con 0x7f69a01885a0 2021-02-10T11:04:02.437 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:02.435 7f69a73ad700 1 -- 172.21.2.102:0/2906447467 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69a00c06f0 con 0x7f69a01885a0 2021-02-10T11:04:02.437 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:02.435 7f6995ffb700 1 -- 172.21.2.102:0/2906447467 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f699c006b00 con 0x7f69a01885a0 2021-02-10T11:04:02.437 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:02.435 7f6995ffb700 1 --2- 172.21.2.102:0/2906447467 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f69800186c0 0x7f698001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:02.437 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:02.435 7f6995ffb700 1 -- 172.21.2.102:0/2906447467 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f699c032570 con 0x7f69a01885a0 2021-02-10T11:04:02.437 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:02.435 7f69a49c8700 1 --2- 172.21.2.102:0/2906447467 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f69800186c0 0x7f698001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:02.437 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:02.436 7f69a49c8700 1 --2- 172.21.2.102:0/2906447467 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f69800186c0 0x7f698001abc0 crc :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:04:02.465 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:02.466 7f69a73ad700 1 -- 172.21.2.102:0/2906447467 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69a0105af0 con 0x7f69a01885a0 2021-02-10T11:04:02.467 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:02.469 7f6995ffb700 1 -- 172.21.2.102:0/2906447467 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f699c02cef0 con 0x7f69a01885a0 2021-02-10T11:04:02.604 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:02.602 7f69a73ad700 1 -- 172.21.2.102:0/2906447467 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f69a00f0b50 con 0x7f69a01885a0 2021-02-10T11:04:02.604 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:02.604 7f6995ffb700 1 -- 172.21.2.102:0/2906447467 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+211028 (secure 0 0 0) 0x7f699c02cef0 con 0x7f69a01885a0 2021-02-10T11:04:02.607 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:04:02.607 INFO:teuthology.orchestra.run.gibba002.stdout:{ 2021-02-10T11:04:02.607 INFO:teuthology.orchestra.run.gibba002.stdout: "fsid": "68db30ea-dbce-4aef-a6df-f01c6fea0513", 2021-02-10T11:04:02.608 INFO:teuthology.orchestra.run.gibba002.stdout: "health": { 2021-02-10T11:04:02.608 INFO:teuthology.orchestra.run.gibba002.stdout: "checks": {}, 2021-02-10T11:04:02.608 INFO:teuthology.orchestra.run.gibba002.stdout: "status": "HEALTH_OK" 2021-02-10T11:04:02.608 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.608 INFO:teuthology.orchestra.run.gibba002.stdout: "election_epoch": 4, 2021-02-10T11:04:02.608 INFO:teuthology.orchestra.run.gibba002.stdout: "quorum": [ 2021-02-10T11:04:02.608 INFO:teuthology.orchestra.run.gibba002.stdout: 0, 2021-02-10T11:04:02.608 INFO:teuthology.orchestra.run.gibba002.stdout: 1, 2021-02-10T11:04:02.608 INFO:teuthology.orchestra.run.gibba002.stdout: 2 2021-02-10T11:04:02.608 INFO:teuthology.orchestra.run.gibba002.stdout: ], 2021-02-10T11:04:02.608 INFO:teuthology.orchestra.run.gibba002.stdout: "quorum_names": [ 2021-02-10T11:04:02.608 INFO:teuthology.orchestra.run.gibba002.stdout: "a", 2021-02-10T11:04:02.608 INFO:teuthology.orchestra.run.gibba002.stdout: "b", 2021-02-10T11:04:02.609 INFO:teuthology.orchestra.run.gibba002.stdout: "c" 2021-02-10T11:04:02.609 INFO:teuthology.orchestra.run.gibba002.stdout: ], 2021-02-10T11:04:02.609 INFO:teuthology.orchestra.run.gibba002.stdout: "quorum_age": 56, 2021-02-10T11:04:02.609 INFO:teuthology.orchestra.run.gibba002.stdout: "monmap": { 2021-02-10T11:04:02.609 INFO:teuthology.orchestra.run.gibba002.stdout: "epoch": 1, 2021-02-10T11:04:02.609 INFO:teuthology.orchestra.run.gibba002.stdout: "min_mon_release_name": "14", 2021-02-10T11:04:02.609 INFO:teuthology.orchestra.run.gibba002.stdout: "num_mons": 3 2021-02-10T11:04:02.609 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.609 INFO:teuthology.orchestra.run.gibba002.stdout: "osdmap": { 2021-02-10T11:04:02.609 INFO:teuthology.orchestra.run.gibba002.stdout: "osdmap": { 2021-02-10T11:04:02.609 INFO:teuthology.orchestra.run.gibba002.stdout: "epoch": 20, 2021-02-10T11:04:02.609 INFO:teuthology.orchestra.run.gibba002.stdout: "num_osds": 8, 2021-02-10T11:04:02.609 INFO:teuthology.orchestra.run.gibba002.stdout: "num_up_osds": 8, 2021-02-10T11:04:02.609 INFO:teuthology.orchestra.run.gibba002.stdout: "num_in_osds": 8, 2021-02-10T11:04:02.610 INFO:teuthology.orchestra.run.gibba002.stdout: "num_remapped_pgs": 0 2021-02-10T11:04:02.610 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:04:02.610 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.610 INFO:teuthology.orchestra.run.gibba002.stdout: "pgmap": { 2021-02-10T11:04:02.610 INFO:teuthology.orchestra.run.gibba002.stdout: "pgs_by_state": [ 2021-02-10T11:04:02.610 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:04:02.610 INFO:teuthology.orchestra.run.gibba002.stdout: "state_name": "active+clean", 2021-02-10T11:04:02.610 INFO:teuthology.orchestra.run.gibba002.stdout: "count": 24 2021-02-10T11:04:02.610 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:04:02.610 INFO:teuthology.orchestra.run.gibba002.stdout: ], 2021-02-10T11:04:02.610 INFO:teuthology.orchestra.run.gibba002.stdout: "num_pgs": 24, 2021-02-10T11:04:02.610 INFO:teuthology.orchestra.run.gibba002.stdout: "num_pools": 3, 2021-02-10T11:04:02.610 INFO:teuthology.orchestra.run.gibba002.stdout: "num_objects": 22, 2021-02-10T11:04:02.611 INFO:teuthology.orchestra.run.gibba002.stdout: "data_bytes": 2286, 2021-02-10T11:04:02.611 INFO:teuthology.orchestra.run.gibba002.stdout: "bytes_used": 8591704064, 2021-02-10T11:04:02.611 INFO:teuthology.orchestra.run.gibba002.stdout: "bytes_avail": 764502409216, 2021-02-10T11:04:02.611 INFO:teuthology.orchestra.run.gibba002.stdout: "bytes_total": 773094113280, 2021-02-10T11:04:02.611 INFO:teuthology.orchestra.run.gibba002.stdout: "write_bytes_sec": 1279, 2021-02-10T11:04:02.611 INFO:teuthology.orchestra.run.gibba002.stdout: "read_op_per_sec": 0, 2021-02-10T11:04:02.611 INFO:teuthology.orchestra.run.gibba002.stdout: "write_op_per_sec": 4 2021-02-10T11:04:02.611 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.611 INFO:teuthology.orchestra.run.gibba002.stdout: "fsmap": { 2021-02-10T11:04:02.611 INFO:teuthology.orchestra.run.gibba002.stdout: "epoch": 6, 2021-02-10T11:04:02.611 INFO:teuthology.orchestra.run.gibba002.stdout: "id": 1, 2021-02-10T11:04:02.611 INFO:teuthology.orchestra.run.gibba002.stdout: "up": 1, 2021-02-10T11:04:02.611 INFO:teuthology.orchestra.run.gibba002.stdout: "in": 1, 2021-02-10T11:04:02.611 INFO:teuthology.orchestra.run.gibba002.stdout: "max": 1, 2021-02-10T11:04:02.612 INFO:teuthology.orchestra.run.gibba002.stdout: "by_rank": [ 2021-02-10T11:04:02.612 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:04:02.612 INFO:teuthology.orchestra.run.gibba002.stdout: "filesystem_id": 1, 2021-02-10T11:04:02.612 INFO:teuthology.orchestra.run.gibba002.stdout: "rank": 0, 2021-02-10T11:04:02.612 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "b", 2021-02-10T11:04:02.612 INFO:teuthology.orchestra.run.gibba002.stdout: "status": "up:active", 2021-02-10T11:04:02.612 INFO:teuthology.orchestra.run.gibba002.stdout: "gid": 4226 2021-02-10T11:04:02.612 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:04:02.612 INFO:teuthology.orchestra.run.gibba002.stdout: ], 2021-02-10T11:04:02.612 INFO:teuthology.orchestra.run.gibba002.stdout: "up:standby": 1 2021-02-10T11:04:02.612 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.612 INFO:teuthology.orchestra.run.gibba002.stdout: "mgrmap": { 2021-02-10T11:04:02.612 INFO:teuthology.orchestra.run.gibba002.stdout: "epoch": 3, 2021-02-10T11:04:02.613 INFO:teuthology.orchestra.run.gibba002.stdout: "active_gid": 4101, 2021-02-10T11:04:02.613 INFO:teuthology.orchestra.run.gibba002.stdout: "active_name": "y", 2021-02-10T11:04:02.613 INFO:teuthology.orchestra.run.gibba002.stdout: "active_addrs": { 2021-02-10T11:04:02.613 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:04:02.613 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:04:02.613 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:04:02.613 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6800", 2021-02-10T11:04:02.613 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 35161 2021-02-10T11:04:02.613 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.613 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:04:02.613 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:04:02.613 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6801", 2021-02-10T11:04:02.613 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 35161 2021-02-10T11:04:02.614 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:04:02.614 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:04:02.614 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.614 INFO:teuthology.orchestra.run.gibba002.stdout: "active_addr": "172.21.2.102:6801/35161", 2021-02-10T11:04:02.614 INFO:teuthology.orchestra.run.gibba002.stdout: "active_change": "2021-02-10 11:03:07.666576", 2021-02-10T11:04:02.614 INFO:teuthology.orchestra.run.gibba002.stdout: "available": true, 2021-02-10T11:04:02.614 INFO:teuthology.orchestra.run.gibba002.stdout: "standbys": [ 2021-02-10T11:04:02.614 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:04:02.614 INFO:teuthology.orchestra.run.gibba002.stdout: "gid": 4105, 2021-02-10T11:04:02.614 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "x", 2021-02-10T11:04:02.614 INFO:teuthology.orchestra.run.gibba002.stdout: "available_modules": [ 2021-02-10T11:04:02.614 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:04:02.614 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "alerts", 2021-02-10T11:04:02.615 INFO:teuthology.orchestra.run.gibba002.stdout: "can_run": true, 2021-02-10T11:04:02.615 INFO:teuthology.orchestra.run.gibba002.stdout: "error_string": "", 2021-02-10T11:04:02.615 INFO:teuthology.orchestra.run.gibba002.stdout: "module_options": { 2021-02-10T11:04:02.615 INFO:teuthology.orchestra.run.gibba002.stdout: "interval": { 2021-02-10T11:04:02.615 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "interval", 2021-02-10T11:04:02.615 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "secs", 2021-02-10T11:04:02.615 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.615 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 1, 2021-02-10T11:04:02.615 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "60", 2021-02-10T11:04:02.615 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.615 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.615 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.615 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "How frequently to reexamine health status", 2021-02-10T11:04:02.615 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.616 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.616 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.616 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.616 INFO:teuthology.orchestra.run.gibba002.stdout: "smtp_destination": { 2021-02-10T11:04:02.616 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "smtp_destination", 2021-02-10T11:04:02.616 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.616 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.616 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 1, 2021-02-10T11:04:02.616 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "", 2021-02-10T11:04:02.616 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.616 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.616 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.616 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "Email address to send alerts to", 2021-02-10T11:04:02.617 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.617 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.617 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.617 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.617 INFO:teuthology.orchestra.run.gibba002.stdout: "smtp_from_name": { 2021-02-10T11:04:02.617 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "smtp_from_name", 2021-02-10T11:04:02.617 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.617 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.617 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 1, 2021-02-10T11:04:02.617 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "Ceph", 2021-02-10T11:04:02.617 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.617 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.617 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.617 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "Email From: name", 2021-02-10T11:04:02.617 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.618 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.618 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.618 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.618 INFO:teuthology.orchestra.run.gibba002.stdout: "smtp_host": { 2021-02-10T11:04:02.618 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "smtp_host", 2021-02-10T11:04:02.618 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.618 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.618 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 1, 2021-02-10T11:04:02.618 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "", 2021-02-10T11:04:02.618 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.618 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.618 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.619 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "SMTP server", 2021-02-10T11:04:02.619 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.619 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.619 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.619 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.619 INFO:teuthology.orchestra.run.gibba002.stdout: "smtp_password": { 2021-02-10T11:04:02.619 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "smtp_password", 2021-02-10T11:04:02.619 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.619 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.619 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 1, 2021-02-10T11:04:02.619 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "", 2021-02-10T11:04:02.619 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.619 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.619 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.620 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "Password to authenticate with", 2021-02-10T11:04:02.620 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.620 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.620 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.620 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.620 INFO:teuthology.orchestra.run.gibba002.stdout: "smtp_port": { 2021-02-10T11:04:02.620 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "smtp_port", 2021-02-10T11:04:02.620 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "int", 2021-02-10T11:04:02.620 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.620 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 1, 2021-02-10T11:04:02.620 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "465", 2021-02-10T11:04:02.620 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.620 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.620 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.621 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "SMTP port", 2021-02-10T11:04:02.621 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.621 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.621 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.621 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.621 INFO:teuthology.orchestra.run.gibba002.stdout: "smtp_sender": { 2021-02-10T11:04:02.621 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "smtp_sender", 2021-02-10T11:04:02.621 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.621 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.621 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 1, 2021-02-10T11:04:02.621 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "", 2021-02-10T11:04:02.621 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.621 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.621 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.621 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "SMTP envelope sender", 2021-02-10T11:04:02.622 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.622 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.622 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.622 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.622 INFO:teuthology.orchestra.run.gibba002.stdout: "smtp_ssl": { 2021-02-10T11:04:02.622 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "smtp_ssl", 2021-02-10T11:04:02.622 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "bool", 2021-02-10T11:04:02.622 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.622 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 1, 2021-02-10T11:04:02.622 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "True", 2021-02-10T11:04:02.622 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.622 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.622 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.622 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "Use SSL to connect to SMTP server", 2021-02-10T11:04:02.623 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.623 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.623 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.623 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.623 INFO:teuthology.orchestra.run.gibba002.stdout: "smtp_user": { 2021-02-10T11:04:02.623 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "smtp_user", 2021-02-10T11:04:02.623 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.623 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.623 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 1, 2021-02-10T11:04:02.623 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "", 2021-02-10T11:04:02.623 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.623 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.623 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.624 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "User to authenticate as", 2021-02-10T11:04:02.624 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.624 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.624 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.624 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:04:02.624 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:04:02.624 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.624 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:04:02.624 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "ansible", 2021-02-10T11:04:02.624 INFO:teuthology.orchestra.run.gibba002.stdout: "can_run": true, 2021-02-10T11:04:02.624 INFO:teuthology.orchestra.run.gibba002.stdout: "error_string": "", 2021-02-10T11:04:02.624 INFO:teuthology.orchestra.run.gibba002.stdout: "module_options": { 2021-02-10T11:04:02.624 INFO:teuthology.orchestra.run.gibba002.stdout: "password": { 2021-02-10T11:04:02.624 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "password", 2021-02-10T11:04:02.625 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.625 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.625 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.625 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "", 2021-02-10T11:04:02.625 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.625 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.625 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.625 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.625 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.625 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.625 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.625 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.625 INFO:teuthology.orchestra.run.gibba002.stdout: "server_url": { 2021-02-10T11:04:02.626 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "server_url", 2021-02-10T11:04:02.626 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.626 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.626 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.626 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "", 2021-02-10T11:04:02.626 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.626 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.626 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.626 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.626 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.626 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.626 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.627 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.627 INFO:teuthology.orchestra.run.gibba002.stdout: "username": { 2021-02-10T11:04:02.627 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "username", 2021-02-10T11:04:02.627 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.627 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.627 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.627 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "", 2021-02-10T11:04:02.627 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.627 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.627 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.627 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.627 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.627 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.628 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.628 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.628 INFO:teuthology.orchestra.run.gibba002.stdout: "verify_server": { 2021-02-10T11:04:02.628 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "verify_server", 2021-02-10T11:04:02.628 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.628 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.628 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.628 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "", 2021-02-10T11:04:02.628 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.628 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.628 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.628 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.628 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.629 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.629 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.629 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:04:02.629 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:04:02.629 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.629 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:04:02.629 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "balancer", 2021-02-10T11:04:02.629 INFO:teuthology.orchestra.run.gibba002.stdout: "can_run": true, 2021-02-10T11:04:02.629 INFO:teuthology.orchestra.run.gibba002.stdout: "error_string": "", 2021-02-10T11:04:02.629 INFO:teuthology.orchestra.run.gibba002.stdout: "module_options": { 2021-02-10T11:04:02.629 INFO:teuthology.orchestra.run.gibba002.stdout: "active": { 2021-02-10T11:04:02.629 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "active", 2021-02-10T11:04:02.629 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "bool", 2021-02-10T11:04:02.630 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.630 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 1, 2021-02-10T11:04:02.630 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "False", 2021-02-10T11:04:02.630 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.630 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.630 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.630 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "automatically balance PGs across cluster", 2021-02-10T11:04:02.630 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.630 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.630 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.630 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.630 INFO:teuthology.orchestra.run.gibba002.stdout: "begin_time": { 2021-02-10T11:04:02.630 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "begin_time", 2021-02-10T11:04:02.630 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.631 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.631 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 1, 2021-02-10T11:04:02.631 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "0000", 2021-02-10T11:04:02.631 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.631 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.631 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.631 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "beginning time of day to automatically balance", 2021-02-10T11:04:02.631 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "This is a time of day in the format HHMM.", 2021-02-10T11:04:02.631 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.631 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.631 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.631 INFO:teuthology.orchestra.run.gibba002.stdout: "begin_weekday": { 2021-02-10T11:04:02.631 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "begin_weekday", 2021-02-10T11:04:02.631 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "uint", 2021-02-10T11:04:02.632 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.632 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 1, 2021-02-10T11:04:02.632 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "0", 2021-02-10T11:04:02.632 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "0", 2021-02-10T11:04:02.632 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "7", 2021-02-10T11:04:02.632 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.632 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2021-02-10T11:04:02.632 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "0 or 7 = Sunday, 1 = Monday, etc.", 2021-02-10T11:04:02.632 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.632 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.632 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.632 INFO:teuthology.orchestra.run.gibba002.stdout: "crush_compat_max_iterations": { 2021-02-10T11:04:02.632 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "crush_compat_max_iterations", 2021-02-10T11:04:02.633 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "uint", 2021-02-10T11:04:02.633 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.633 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 1, 2021-02-10T11:04:02.633 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "25", 2021-02-10T11:04:02.633 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "1", 2021-02-10T11:04:02.633 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "250", 2021-02-10T11:04:02.633 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.633 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "maximum number of iterations to attempt optimization", 2021-02-10T11:04:02.633 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.633 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.633 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.633 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.633 INFO:teuthology.orchestra.run.gibba002.stdout: "crush_compat_metrics": { 2021-02-10T11:04:02.634 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "crush_compat_metrics", 2021-02-10T11:04:02.634 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.634 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.634 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 1, 2021-02-10T11:04:02.634 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "pgs,objects,bytes", 2021-02-10T11:04:02.634 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.634 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.634 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.634 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "metrics with which to calculate OSD utilization", 2021-02-10T11:04:02.634 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2021-02-10T11:04:02.634 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.634 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.634 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.634 INFO:teuthology.orchestra.run.gibba002.stdout: "crush_compat_step": { 2021-02-10T11:04:02.635 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "crush_compat_step", 2021-02-10T11:04:02.635 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "float", 2021-02-10T11:04:02.635 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.635 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 1, 2021-02-10T11:04:02.635 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "0.5", 2021-02-10T11:04:02.635 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "0.001", 2021-02-10T11:04:02.635 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "0.999", 2021-02-10T11:04:02.635 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.635 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "aggressiveness of optimization", 2021-02-10T11:04:02.635 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2021-02-10T11:04:02.635 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.635 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.635 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.636 INFO:teuthology.orchestra.run.gibba002.stdout: "end_time": { 2021-02-10T11:04:02.636 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "end_time", 2021-02-10T11:04:02.636 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.636 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.636 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 1, 2021-02-10T11:04:02.636 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "2400", 2021-02-10T11:04:02.636 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.636 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.636 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.636 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "ending time of day to automatically balance", 2021-02-10T11:04:02.636 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "This is a time of day in the format HHMM.", 2021-02-10T11:04:02.636 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.637 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.637 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.637 INFO:teuthology.orchestra.run.gibba002.stdout: "end_weekday": { 2021-02-10T11:04:02.637 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "end_weekday", 2021-02-10T11:04:02.637 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "uint", 2021-02-10T11:04:02.637 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.637 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 1, 2021-02-10T11:04:02.637 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "7", 2021-02-10T11:04:02.637 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "0", 2021-02-10T11:04:02.637 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "7", 2021-02-10T11:04:02.637 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.637 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2021-02-10T11:04:02.637 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "0 or 7 = Sunday, 1 = Monday, etc.", 2021-02-10T11:04:02.637 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.638 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.638 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.638 INFO:teuthology.orchestra.run.gibba002.stdout: "min_score": { 2021-02-10T11:04:02.638 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "min_score", 2021-02-10T11:04:02.638 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "float", 2021-02-10T11:04:02.638 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.638 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 1, 2021-02-10T11:04:02.638 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "0", 2021-02-10T11:04:02.638 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.638 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.638 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.638 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "minimum score, below which no optimization is attempted", 2021-02-10T11:04:02.638 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.639 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.639 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.639 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.639 INFO:teuthology.orchestra.run.gibba002.stdout: "mode": { 2021-02-10T11:04:02.639 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "mode", 2021-02-10T11:04:02.639 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.639 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.639 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 1, 2021-02-10T11:04:02.639 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "none", 2021-02-10T11:04:02.639 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.639 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.639 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [ 2021-02-10T11:04:02.639 INFO:teuthology.orchestra.run.gibba002.stdout: "crush-compat", 2021-02-10T11:04:02.639 INFO:teuthology.orchestra.run.gibba002.stdout: "none", 2021-02-10T11:04:02.640 INFO:teuthology.orchestra.run.gibba002.stdout: "upmap" 2021-02-10T11:04:02.640 INFO:teuthology.orchestra.run.gibba002.stdout: ], 2021-02-10T11:04:02.640 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "Balancer mode", 2021-02-10T11:04:02.640 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.640 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.640 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.640 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.640 INFO:teuthology.orchestra.run.gibba002.stdout: "pool_ids": { 2021-02-10T11:04:02.640 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "pool_ids", 2021-02-10T11:04:02.640 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.640 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.640 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 1, 2021-02-10T11:04:02.640 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "", 2021-02-10T11:04:02.640 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.641 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.641 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.641 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "pools which the automatic balancing will be limited to", 2021-02-10T11:04:02.641 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.641 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.641 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.641 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.641 INFO:teuthology.orchestra.run.gibba002.stdout: "sleep_interval": { 2021-02-10T11:04:02.641 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "sleep_interval", 2021-02-10T11:04:02.641 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "secs", 2021-02-10T11:04:02.641 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.641 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 1, 2021-02-10T11:04:02.641 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "60", 2021-02-10T11:04:02.641 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.642 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.642 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.642 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "how frequently to wake up and attempt optimization", 2021-02-10T11:04:02.642 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.642 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.642 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.642 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.642 INFO:teuthology.orchestra.run.gibba002.stdout: "upmap_max_deviation": { 2021-02-10T11:04:02.642 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "upmap_max_deviation", 2021-02-10T11:04:02.642 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "int", 2021-02-10T11:04:02.642 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.642 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 1, 2021-02-10T11:04:02.642 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "5", 2021-02-10T11:04:02.642 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "1", 2021-02-10T11:04:02.643 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.643 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.643 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "deviation below which no optimization is attempted", 2021-02-10T11:04:02.643 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2021-02-10T11:04:02.643 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.643 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.643 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.643 INFO:teuthology.orchestra.run.gibba002.stdout: "upmap_max_iterations": { 2021-02-10T11:04:02.643 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "upmap_max_iterations", 2021-02-10T11:04:02.643 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "uint", 2021-02-10T11:04:02.643 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.643 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 1, 2021-02-10T11:04:02.643 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "10", 2021-02-10T11:04:02.643 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.644 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.644 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.644 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "maximum upmap optimization iterations", 2021-02-10T11:04:02.644 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.644 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.644 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.644 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:04:02.644 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:04:02.644 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.644 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:04:02.644 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "crash", 2021-02-10T11:04:02.644 INFO:teuthology.orchestra.run.gibba002.stdout: "can_run": true, 2021-02-10T11:04:02.644 INFO:teuthology.orchestra.run.gibba002.stdout: "error_string": "", 2021-02-10T11:04:02.645 INFO:teuthology.orchestra.run.gibba002.stdout: "module_options": { 2021-02-10T11:04:02.645 INFO:teuthology.orchestra.run.gibba002.stdout: "retain_interval": { 2021-02-10T11:04:02.645 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "retain_interval", 2021-02-10T11:04:02.645 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "secs", 2021-02-10T11:04:02.645 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.645 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 1, 2021-02-10T11:04:02.645 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "31536000", 2021-02-10T11:04:02.645 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.645 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.645 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.645 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "how long to retain crashes before pruning them", 2021-02-10T11:04:02.645 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.645 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.645 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.646 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.646 INFO:teuthology.orchestra.run.gibba002.stdout: "warn_recent_interval": { 2021-02-10T11:04:02.646 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "warn_recent_interval", 2021-02-10T11:04:02.646 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "secs", 2021-02-10T11:04:02.646 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.646 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 1, 2021-02-10T11:04:02.646 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "1209600", 2021-02-10T11:04:02.646 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.646 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.646 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.646 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "time interval in which to warn about recent crashes", 2021-02-10T11:04:02.646 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.646 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.647 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.647 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:04:02.647 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:04:02.647 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.647 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:04:02.647 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "dashboard", 2021-02-10T11:04:02.647 INFO:teuthology.orchestra.run.gibba002.stdout: "can_run": true, 2021-02-10T11:04:02.647 INFO:teuthology.orchestra.run.gibba002.stdout: "error_string": "", 2021-02-10T11:04:02.647 INFO:teuthology.orchestra.run.gibba002.stdout: "module_options": { 2021-02-10T11:04:02.647 INFO:teuthology.orchestra.run.gibba002.stdout: "ALERTMANAGER_API_HOST": { 2021-02-10T11:04:02.647 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "ALERTMANAGER_API_HOST", 2021-02-10T11:04:02.647 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.647 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.647 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.648 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "", 2021-02-10T11:04:02.648 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.648 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.648 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.648 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.648 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.648 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.648 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.648 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.648 INFO:teuthology.orchestra.run.gibba002.stdout: "AUDIT_API_ENABLED": { 2021-02-10T11:04:02.648 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "AUDIT_API_ENABLED", 2021-02-10T11:04:02.648 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "bool", 2021-02-10T11:04:02.648 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.649 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.649 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "False", 2021-02-10T11:04:02.649 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.649 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.649 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.649 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.649 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.649 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.649 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.649 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.649 INFO:teuthology.orchestra.run.gibba002.stdout: "AUDIT_API_LOG_PAYLOAD": { 2021-02-10T11:04:02.649 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2021-02-10T11:04:02.649 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "bool", 2021-02-10T11:04:02.650 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.650 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.650 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "True", 2021-02-10T11:04:02.650 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.650 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.650 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.650 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.650 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.650 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.650 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.650 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.650 INFO:teuthology.orchestra.run.gibba002.stdout: "ENABLE_BROWSABLE_API": { 2021-02-10T11:04:02.650 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "ENABLE_BROWSABLE_API", 2021-02-10T11:04:02.651 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "bool", 2021-02-10T11:04:02.651 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.651 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.651 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "True", 2021-02-10T11:04:02.651 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.651 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.651 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.651 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.651 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.651 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.651 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.651 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.651 INFO:teuthology.orchestra.run.gibba002.stdout: "FEATURE_TOGGLE_cephfs": { 2021-02-10T11:04:02.652 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "FEATURE_TOGGLE_cephfs", 2021-02-10T11:04:02.652 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "bool", 2021-02-10T11:04:02.652 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.652 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.652 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "True", 2021-02-10T11:04:02.652 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.652 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.652 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.652 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.652 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.652 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.652 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.652 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.652 INFO:teuthology.orchestra.run.gibba002.stdout: "FEATURE_TOGGLE_iscsi": { 2021-02-10T11:04:02.653 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "FEATURE_TOGGLE_iscsi", 2021-02-10T11:04:02.653 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "bool", 2021-02-10T11:04:02.653 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.653 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.653 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "True", 2021-02-10T11:04:02.653 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.653 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.653 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.653 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.653 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.653 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.653 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.653 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.653 INFO:teuthology.orchestra.run.gibba002.stdout: "FEATURE_TOGGLE_mirroring": { 2021-02-10T11:04:02.654 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "FEATURE_TOGGLE_mirroring", 2021-02-10T11:04:02.654 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "bool", 2021-02-10T11:04:02.654 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.654 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.654 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "True", 2021-02-10T11:04:02.654 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.654 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.654 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.654 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.654 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.654 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.654 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.654 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.655 INFO:teuthology.orchestra.run.gibba002.stdout: "FEATURE_TOGGLE_rbd": { 2021-02-10T11:04:02.655 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "FEATURE_TOGGLE_rbd", 2021-02-10T11:04:02.655 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "bool", 2021-02-10T11:04:02.655 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.655 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.655 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "True", 2021-02-10T11:04:02.655 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.655 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.655 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.655 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.655 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.655 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.655 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.655 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.656 INFO:teuthology.orchestra.run.gibba002.stdout: "FEATURE_TOGGLE_rgw": { 2021-02-10T11:04:02.656 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "FEATURE_TOGGLE_rgw", 2021-02-10T11:04:02.656 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "bool", 2021-02-10T11:04:02.656 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.656 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.656 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "True", 2021-02-10T11:04:02.656 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.656 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.656 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.656 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.656 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.656 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.657 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.657 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.657 INFO:teuthology.orchestra.run.gibba002.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2021-02-10T11:04:02.657 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2021-02-10T11:04:02.657 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.657 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.657 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.657 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "", 2021-02-10T11:04:02.657 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.657 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.657 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.657 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.657 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.658 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.658 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.658 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.658 INFO:teuthology.orchestra.run.gibba002.stdout: "GRAFANA_API_PASSWORD": { 2021-02-10T11:04:02.658 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "GRAFANA_API_PASSWORD", 2021-02-10T11:04:02.658 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.658 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.658 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.658 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "admin", 2021-02-10T11:04:02.658 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.658 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.658 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.658 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.658 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.659 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.659 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.659 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.659 INFO:teuthology.orchestra.run.gibba002.stdout: "GRAFANA_API_SSL_VERIFY": { 2021-02-10T11:04:02.659 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2021-02-10T11:04:02.659 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "bool", 2021-02-10T11:04:02.659 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.659 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.659 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "True", 2021-02-10T11:04:02.659 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.659 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.659 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.659 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.660 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.660 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.660 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.660 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.660 INFO:teuthology.orchestra.run.gibba002.stdout: "GRAFANA_API_URL": { 2021-02-10T11:04:02.660 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "GRAFANA_API_URL", 2021-02-10T11:04:02.660 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.660 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.660 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.660 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "", 2021-02-10T11:04:02.660 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.660 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.660 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.660 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.661 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.661 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.661 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.661 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.661 INFO:teuthology.orchestra.run.gibba002.stdout: "GRAFANA_API_USERNAME": { 2021-02-10T11:04:02.661 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "GRAFANA_API_USERNAME", 2021-02-10T11:04:02.661 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.661 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.661 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.661 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "admin", 2021-02-10T11:04:02.661 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.661 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.661 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.662 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.662 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.662 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.662 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.662 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.662 INFO:teuthology.orchestra.run.gibba002.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2021-02-10T11:04:02.662 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2021-02-10T11:04:02.662 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "bool", 2021-02-10T11:04:02.662 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.662 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.662 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "False", 2021-02-10T11:04:02.662 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.662 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.663 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.663 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.663 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.663 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.663 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.663 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.663 INFO:teuthology.orchestra.run.gibba002.stdout: "ISCSI_API_SSL_VERIFICATION": { 2021-02-10T11:04:02.663 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2021-02-10T11:04:02.663 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "bool", 2021-02-10T11:04:02.663 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.663 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.663 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "True", 2021-02-10T11:04:02.663 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.664 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.664 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.664 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.664 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.664 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.664 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.664 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.664 INFO:teuthology.orchestra.run.gibba002.stdout: "PROMETHEUS_API_HOST": { 2021-02-10T11:04:02.664 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "PROMETHEUS_API_HOST", 2021-02-10T11:04:02.664 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.664 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.664 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.664 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "", 2021-02-10T11:04:02.665 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.665 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.665 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.665 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.665 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.665 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.665 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.665 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.665 INFO:teuthology.orchestra.run.gibba002.stdout: "REST_REQUESTS_TIMEOUT": { 2021-02-10T11:04:02.665 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "REST_REQUESTS_TIMEOUT", 2021-02-10T11:04:02.665 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "int", 2021-02-10T11:04:02.665 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.665 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.666 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "45", 2021-02-10T11:04:02.666 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.666 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.666 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.666 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.666 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.666 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.666 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.666 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.666 INFO:teuthology.orchestra.run.gibba002.stdout: "RGW_API_ACCESS_KEY": { 2021-02-10T11:04:02.666 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "RGW_API_ACCESS_KEY", 2021-02-10T11:04:02.666 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.666 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.667 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.667 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "", 2021-02-10T11:04:02.667 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.667 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.667 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.667 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.667 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.667 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.667 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.667 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.667 INFO:teuthology.orchestra.run.gibba002.stdout: "RGW_API_ADMIN_RESOURCE": { 2021-02-10T11:04:02.667 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2021-02-10T11:04:02.667 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.668 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.668 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.668 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "admin", 2021-02-10T11:04:02.668 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.668 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.668 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.668 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.668 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.668 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.668 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.668 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.668 INFO:teuthology.orchestra.run.gibba002.stdout: "RGW_API_HOST": { 2021-02-10T11:04:02.668 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "RGW_API_HOST", 2021-02-10T11:04:02.668 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.669 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.669 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.669 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "", 2021-02-10T11:04:02.669 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.669 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.669 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.669 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.669 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.669 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.669 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.669 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.669 INFO:teuthology.orchestra.run.gibba002.stdout: "RGW_API_PORT": { 2021-02-10T11:04:02.669 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "RGW_API_PORT", 2021-02-10T11:04:02.670 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "int", 2021-02-10T11:04:02.670 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.670 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.670 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "80", 2021-02-10T11:04:02.670 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.670 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.670 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.670 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.670 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.670 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.670 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.670 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.670 INFO:teuthology.orchestra.run.gibba002.stdout: "RGW_API_SCHEME": { 2021-02-10T11:04:02.671 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "RGW_API_SCHEME", 2021-02-10T11:04:02.671 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.671 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.671 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.671 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "http", 2021-02-10T11:04:02.671 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.671 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.671 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.671 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.671 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.671 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.671 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.671 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.671 INFO:teuthology.orchestra.run.gibba002.stdout: "RGW_API_SECRET_KEY": { 2021-02-10T11:04:02.672 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "RGW_API_SECRET_KEY", 2021-02-10T11:04:02.672 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.672 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.672 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.672 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "", 2021-02-10T11:04:02.672 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.672 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.672 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.672 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.672 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.672 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.672 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.672 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.672 INFO:teuthology.orchestra.run.gibba002.stdout: "RGW_API_SSL_VERIFY": { 2021-02-10T11:04:02.673 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "RGW_API_SSL_VERIFY", 2021-02-10T11:04:02.673 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "bool", 2021-02-10T11:04:02.673 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.673 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.673 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "True", 2021-02-10T11:04:02.673 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.673 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.673 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.673 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.673 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.673 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.673 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.673 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.673 INFO:teuthology.orchestra.run.gibba002.stdout: "RGW_API_USER_ID": { 2021-02-10T11:04:02.673 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "RGW_API_USER_ID", 2021-02-10T11:04:02.674 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.674 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.674 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.674 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "", 2021-02-10T11:04:02.674 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.674 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.674 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.674 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.674 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.674 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.674 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.674 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.675 INFO:teuthology.orchestra.run.gibba002.stdout: "crt_file": { 2021-02-10T11:04:02.675 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "crt_file", 2021-02-10T11:04:02.675 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.675 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.675 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.675 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "", 2021-02-10T11:04:02.675 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.675 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.675 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.675 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.675 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.675 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.675 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.676 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.676 INFO:teuthology.orchestra.run.gibba002.stdout: "debug": { 2021-02-10T11:04:02.676 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "debug", 2021-02-10T11:04:02.676 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "bool", 2021-02-10T11:04:02.676 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.676 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.676 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "False", 2021-02-10T11:04:02.676 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.676 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.676 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.676 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "Enable/disable debug options", 2021-02-10T11:04:02.676 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.676 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.676 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.676 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.677 INFO:teuthology.orchestra.run.gibba002.stdout: "jwt_token_ttl": { 2021-02-10T11:04:02.677 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "jwt_token_ttl", 2021-02-10T11:04:02.677 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "int", 2021-02-10T11:04:02.677 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.677 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.677 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "28800", 2021-02-10T11:04:02.677 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.677 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.677 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.677 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.677 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.677 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.677 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.677 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.678 INFO:teuthology.orchestra.run.gibba002.stdout: "key_file": { 2021-02-10T11:04:02.678 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "key_file", 2021-02-10T11:04:02.678 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.678 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.678 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.678 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "", 2021-02-10T11:04:02.678 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.678 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.678 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.678 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.678 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.678 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.678 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.679 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.679 INFO:teuthology.orchestra.run.gibba002.stdout: "password": { 2021-02-10T11:04:02.679 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "password", 2021-02-10T11:04:02.679 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.679 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.679 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.679 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "", 2021-02-10T11:04:02.679 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.679 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.679 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.679 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.679 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.679 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.679 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.680 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.680 INFO:teuthology.orchestra.run.gibba002.stdout: "server_addr": { 2021-02-10T11:04:02.680 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "server_addr", 2021-02-10T11:04:02.680 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.680 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.680 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.680 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "::", 2021-02-10T11:04:02.680 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.680 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.680 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.680 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.680 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.680 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.680 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.681 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.681 INFO:teuthology.orchestra.run.gibba002.stdout: "server_port": { 2021-02-10T11:04:02.681 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "server_port", 2021-02-10T11:04:02.681 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "int", 2021-02-10T11:04:02.681 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.681 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.681 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "8080", 2021-02-10T11:04:02.681 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.681 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.681 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.681 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.681 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.681 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.681 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.682 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.682 INFO:teuthology.orchestra.run.gibba002.stdout: "ssl": { 2021-02-10T11:04:02.682 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "ssl", 2021-02-10T11:04:02.682 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "bool", 2021-02-10T11:04:02.682 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.682 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.682 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "True", 2021-02-10T11:04:02.682 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.682 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.682 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.682 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.682 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.682 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.683 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.683 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.683 INFO:teuthology.orchestra.run.gibba002.stdout: "ssl_server_port": { 2021-02-10T11:04:02.683 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "ssl_server_port", 2021-02-10T11:04:02.683 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "int", 2021-02-10T11:04:02.683 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.683 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.683 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "8443", 2021-02-10T11:04:02.683 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.683 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.683 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.683 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.683 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.683 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.684 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.684 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.684 INFO:teuthology.orchestra.run.gibba002.stdout: "standby_behaviour": { 2021-02-10T11:04:02.684 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "standby_behaviour", 2021-02-10T11:04:02.684 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.684 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.684 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.684 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "redirect", 2021-02-10T11:04:02.684 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.684 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.684 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [ 2021-02-10T11:04:02.684 INFO:teuthology.orchestra.run.gibba002.stdout: "error", 2021-02-10T11:04:02.684 INFO:teuthology.orchestra.run.gibba002.stdout: "redirect" 2021-02-10T11:04:02.685 INFO:teuthology.orchestra.run.gibba002.stdout: ], 2021-02-10T11:04:02.685 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.685 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.685 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.685 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.685 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.685 INFO:teuthology.orchestra.run.gibba002.stdout: "standby_error_status_code": { 2021-02-10T11:04:02.685 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "standby_error_status_code", 2021-02-10T11:04:02.685 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "int", 2021-02-10T11:04:02.685 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.685 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.685 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "500", 2021-02-10T11:04:02.685 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "400", 2021-02-10T11:04:02.685 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "599", 2021-02-10T11:04:02.686 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.686 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.686 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.686 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.686 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.686 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.686 INFO:teuthology.orchestra.run.gibba002.stdout: "url_prefix": { 2021-02-10T11:04:02.686 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "url_prefix", 2021-02-10T11:04:02.686 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.686 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.686 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.686 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "", 2021-02-10T11:04:02.686 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.687 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.687 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.687 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.687 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.687 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.687 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.687 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.687 INFO:teuthology.orchestra.run.gibba002.stdout: "username": { 2021-02-10T11:04:02.687 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "username", 2021-02-10T11:04:02.687 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.687 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.687 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.687 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "", 2021-02-10T11:04:02.688 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.688 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.688 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.688 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.688 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.688 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.688 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.688 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:04:02.688 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:04:02.688 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.688 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:04:02.688 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "deepsea", 2021-02-10T11:04:02.688 INFO:teuthology.orchestra.run.gibba002.stdout: "can_run": true, 2021-02-10T11:04:02.688 INFO:teuthology.orchestra.run.gibba002.stdout: "error_string": "", 2021-02-10T11:04:02.689 INFO:teuthology.orchestra.run.gibba002.stdout: "module_options": { 2021-02-10T11:04:02.689 INFO:teuthology.orchestra.run.gibba002.stdout: "salt_api_eauth": { 2021-02-10T11:04:02.689 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "salt_api_eauth", 2021-02-10T11:04:02.689 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.689 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.689 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.689 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "sharedsecret", 2021-02-10T11:04:02.689 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.689 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.689 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.689 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.689 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.689 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.690 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.690 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.690 INFO:teuthology.orchestra.run.gibba002.stdout: "salt_api_password": { 2021-02-10T11:04:02.690 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "salt_api_password", 2021-02-10T11:04:02.690 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.690 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.690 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.690 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "", 2021-02-10T11:04:02.690 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.690 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.690 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.690 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.690 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.691 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.691 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.691 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.691 INFO:teuthology.orchestra.run.gibba002.stdout: "salt_api_url": { 2021-02-10T11:04:02.691 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "salt_api_url", 2021-02-10T11:04:02.691 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.691 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.691 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.691 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "", 2021-02-10T11:04:02.691 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.691 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.691 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.691 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.691 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.692 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.692 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.692 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.692 INFO:teuthology.orchestra.run.gibba002.stdout: "salt_api_username": { 2021-02-10T11:04:02.692 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "salt_api_username", 2021-02-10T11:04:02.692 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.692 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.692 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.692 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "", 2021-02-10T11:04:02.692 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.692 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.692 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.692 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.693 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.693 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.693 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.693 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:04:02.693 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:04:02.693 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.693 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:04:02.693 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "devicehealth", 2021-02-10T11:04:02.693 INFO:teuthology.orchestra.run.gibba002.stdout: "can_run": true, 2021-02-10T11:04:02.693 INFO:teuthology.orchestra.run.gibba002.stdout: "error_string": "", 2021-02-10T11:04:02.693 INFO:teuthology.orchestra.run.gibba002.stdout: "module_options": { 2021-02-10T11:04:02.693 INFO:teuthology.orchestra.run.gibba002.stdout: "enable_monitoring": { 2021-02-10T11:04:02.693 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "enable_monitoring", 2021-02-10T11:04:02.694 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "bool", 2021-02-10T11:04:02.694 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.694 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 1, 2021-02-10T11:04:02.694 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "False", 2021-02-10T11:04:02.694 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.694 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.694 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.694 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "monitor device health metrics", 2021-02-10T11:04:02.694 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.694 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.694 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.694 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.694 INFO:teuthology.orchestra.run.gibba002.stdout: "mark_out_threshold": { 2021-02-10T11:04:02.694 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "mark_out_threshold", 2021-02-10T11:04:02.695 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "secs", 2021-02-10T11:04:02.695 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.695 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 1, 2021-02-10T11:04:02.695 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "2419200", 2021-02-10T11:04:02.695 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.695 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.695 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.695 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "automatically mark OSD if it may fail before this long", 2021-02-10T11:04:02.695 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.695 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.695 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.695 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.695 INFO:teuthology.orchestra.run.gibba002.stdout: "pool_name": { 2021-02-10T11:04:02.695 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "pool_name", 2021-02-10T11:04:02.696 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.696 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.696 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 1, 2021-02-10T11:04:02.696 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "device_health_metrics", 2021-02-10T11:04:02.696 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.696 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.696 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.696 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "name of pool in which to store device health metrics", 2021-02-10T11:04:02.696 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.696 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.696 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.696 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.696 INFO:teuthology.orchestra.run.gibba002.stdout: "retention_period": { 2021-02-10T11:04:02.696 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "retention_period", 2021-02-10T11:04:02.697 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "secs", 2021-02-10T11:04:02.697 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.697 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 1, 2021-02-10T11:04:02.697 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "15552000", 2021-02-10T11:04:02.697 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.697 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.697 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.697 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "how long to retain device health metrics", 2021-02-10T11:04:02.697 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.697 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.697 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.697 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.697 INFO:teuthology.orchestra.run.gibba002.stdout: "scrape_frequency": { 2021-02-10T11:04:02.697 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "scrape_frequency", 2021-02-10T11:04:02.698 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "secs", 2021-02-10T11:04:02.698 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.698 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 1, 2021-02-10T11:04:02.698 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "86400", 2021-02-10T11:04:02.698 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.698 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.698 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.698 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "how frequently to scrape device health metrics", 2021-02-10T11:04:02.698 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.698 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.698 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.698 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.698 INFO:teuthology.orchestra.run.gibba002.stdout: "self_heal": { 2021-02-10T11:04:02.699 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "self_heal", 2021-02-10T11:04:02.699 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "bool", 2021-02-10T11:04:02.699 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.699 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 1, 2021-02-10T11:04:02.699 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "True", 2021-02-10T11:04:02.699 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.699 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.699 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.699 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "preemptively heal cluster around devices that may fail", 2021-02-10T11:04:02.699 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.699 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.699 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.699 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.700 INFO:teuthology.orchestra.run.gibba002.stdout: "sleep_interval": { 2021-02-10T11:04:02.700 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "sleep_interval", 2021-02-10T11:04:02.700 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "secs", 2021-02-10T11:04:02.700 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.700 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 1, 2021-02-10T11:04:02.700 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "600", 2021-02-10T11:04:02.700 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.700 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.700 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.700 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "how frequently to wake up and check device health", 2021-02-10T11:04:02.700 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.700 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.700 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.700 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.700 INFO:teuthology.orchestra.run.gibba002.stdout: "warn_threshold": { 2021-02-10T11:04:02.701 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "warn_threshold", 2021-02-10T11:04:02.701 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "secs", 2021-02-10T11:04:02.701 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.701 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 1, 2021-02-10T11:04:02.701 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "7257600", 2021-02-10T11:04:02.701 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.701 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.701 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.701 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "raise health warning if OSD may fail before this long", 2021-02-10T11:04:02.701 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.701 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.701 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.701 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:04:02.701 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:04:02.702 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.702 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:04:02.702 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "diskprediction_cloud", 2021-02-10T11:04:02.702 INFO:teuthology.orchestra.run.gibba002.stdout: "can_run": true, 2021-02-10T11:04:02.702 INFO:teuthology.orchestra.run.gibba002.stdout: "error_string": "", 2021-02-10T11:04:02.702 INFO:teuthology.orchestra.run.gibba002.stdout: "module_options": { 2021-02-10T11:04:02.702 INFO:teuthology.orchestra.run.gibba002.stdout: "diskprediction_cert_context": { 2021-02-10T11:04:02.702 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "diskprediction_cert_context", 2021-02-10T11:04:02.702 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.702 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.702 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.702 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "", 2021-02-10T11:04:02.702 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.703 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.703 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.703 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.703 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.703 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.703 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.703 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.703 INFO:teuthology.orchestra.run.gibba002.stdout: "diskprediction_default_authority": { 2021-02-10T11:04:02.703 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "diskprediction_default_authority", 2021-02-10T11:04:02.703 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.703 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.703 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.703 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "localhost", 2021-02-10T11:04:02.703 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.704 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.704 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.704 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.704 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.704 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.704 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.704 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.704 INFO:teuthology.orchestra.run.gibba002.stdout: "diskprediction_password": { 2021-02-10T11:04:02.704 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "diskprediction_password", 2021-02-10T11:04:02.704 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.704 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.704 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.704 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "", 2021-02-10T11:04:02.705 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.705 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.705 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.705 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.705 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.705 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.705 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.705 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.705 INFO:teuthology.orchestra.run.gibba002.stdout: "diskprediction_port": { 2021-02-10T11:04:02.705 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "diskprediction_port", 2021-02-10T11:04:02.705 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.705 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.705 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.705 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "31400", 2021-02-10T11:04:02.706 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.706 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.706 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.706 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.706 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.706 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.706 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.706 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.706 INFO:teuthology.orchestra.run.gibba002.stdout: "diskprediction_retrieve_prediction_interval": { 2021-02-10T11:04:02.706 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "diskprediction_retrieve_prediction_interval", 2021-02-10T11:04:02.706 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.706 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.707 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.707 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "43200", 2021-02-10T11:04:02.707 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.707 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.707 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.707 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.707 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.707 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.707 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.707 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.707 INFO:teuthology.orchestra.run.gibba002.stdout: "diskprediction_server": { 2021-02-10T11:04:02.707 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "diskprediction_server", 2021-02-10T11:04:02.707 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.708 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.708 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.708 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "", 2021-02-10T11:04:02.708 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.708 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.708 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.708 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.708 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.708 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.708 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.708 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.708 INFO:teuthology.orchestra.run.gibba002.stdout: "diskprediction_ssl_target_name_override": { 2021-02-10T11:04:02.708 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "diskprediction_ssl_target_name_override", 2021-02-10T11:04:02.708 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.709 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.709 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.709 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "localhost", 2021-02-10T11:04:02.709 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.709 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.709 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.709 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.709 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.709 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.709 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.709 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.709 INFO:teuthology.orchestra.run.gibba002.stdout: "diskprediction_upload_metrics_interval": { 2021-02-10T11:04:02.709 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "diskprediction_upload_metrics_interval", 2021-02-10T11:04:02.709 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.710 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.710 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.710 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "600", 2021-02-10T11:04:02.710 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.710 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.710 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.710 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.710 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.710 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.710 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.710 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.710 INFO:teuthology.orchestra.run.gibba002.stdout: "diskprediction_upload_smart_interval": { 2021-02-10T11:04:02.710 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "diskprediction_upload_smart_interval", 2021-02-10T11:04:02.711 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.711 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.711 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.711 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "43200", 2021-02-10T11:04:02.711 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.711 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.711 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.711 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.711 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.711 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.711 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.711 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.711 INFO:teuthology.orchestra.run.gibba002.stdout: "diskprediction_user": { 2021-02-10T11:04:02.711 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "diskprediction_user", 2021-02-10T11:04:02.712 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.712 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.712 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.712 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "", 2021-02-10T11:04:02.712 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.712 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.712 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.712 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.712 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.712 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.712 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.712 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.712 INFO:teuthology.orchestra.run.gibba002.stdout: "sleep_interval": { 2021-02-10T11:04:02.713 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "sleep_interval", 2021-02-10T11:04:02.713 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.713 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.713 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.713 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "600", 2021-02-10T11:04:02.713 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.713 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.713 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.713 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.713 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.713 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.713 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.713 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:04:02.713 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:04:02.714 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.714 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:04:02.714 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "diskprediction_local", 2021-02-10T11:04:02.714 INFO:teuthology.orchestra.run.gibba002.stdout: "can_run": true, 2021-02-10T11:04:02.714 INFO:teuthology.orchestra.run.gibba002.stdout: "error_string": "", 2021-02-10T11:04:02.714 INFO:teuthology.orchestra.run.gibba002.stdout: "module_options": { 2021-02-10T11:04:02.714 INFO:teuthology.orchestra.run.gibba002.stdout: "predict_interval": { 2021-02-10T11:04:02.714 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "predict_interval", 2021-02-10T11:04:02.714 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.714 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.714 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.714 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "86400", 2021-02-10T11:04:02.714 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.715 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.715 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.715 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.715 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.715 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.715 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.715 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.715 INFO:teuthology.orchestra.run.gibba002.stdout: "sleep_interval": { 2021-02-10T11:04:02.715 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "sleep_interval", 2021-02-10T11:04:02.715 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.715 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.715 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.715 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "600", 2021-02-10T11:04:02.715 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.716 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.716 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.716 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.716 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.716 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.716 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.716 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:04:02.716 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:04:02.716 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.716 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:04:02.716 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "influx", 2021-02-10T11:04:02.716 INFO:teuthology.orchestra.run.gibba002.stdout: "can_run": false, 2021-02-10T11:04:02.716 INFO:teuthology.orchestra.run.gibba002.stdout: "error_string": "influxdb python module not found", 2021-02-10T11:04:02.716 INFO:teuthology.orchestra.run.gibba002.stdout: "module_options": { 2021-02-10T11:04:02.717 INFO:teuthology.orchestra.run.gibba002.stdout: "batch_size": { 2021-02-10T11:04:02.717 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "batch_size", 2021-02-10T11:04:02.717 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.717 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.717 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.717 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "5000", 2021-02-10T11:04:02.717 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.717 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.717 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.717 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.717 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.717 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.717 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.718 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.718 INFO:teuthology.orchestra.run.gibba002.stdout: "database": { 2021-02-10T11:04:02.718 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "database", 2021-02-10T11:04:02.718 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.718 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.718 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.718 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "ceph", 2021-02-10T11:04:02.718 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.718 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.718 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.718 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.718 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.718 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.718 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.719 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.719 INFO:teuthology.orchestra.run.gibba002.stdout: "hostname": { 2021-02-10T11:04:02.719 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "hostname", 2021-02-10T11:04:02.719 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.719 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.719 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.719 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "None", 2021-02-10T11:04:02.719 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.719 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.719 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.719 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.719 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.719 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.720 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.720 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.720 INFO:teuthology.orchestra.run.gibba002.stdout: "interval": { 2021-02-10T11:04:02.720 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "interval", 2021-02-10T11:04:02.720 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.720 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.720 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.720 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "30", 2021-02-10T11:04:02.720 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.720 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.720 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.720 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.720 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.720 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.721 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.721 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.721 INFO:teuthology.orchestra.run.gibba002.stdout: "password": { 2021-02-10T11:04:02.721 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "password", 2021-02-10T11:04:02.721 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.721 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.721 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.721 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "None", 2021-02-10T11:04:02.721 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.721 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.721 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.721 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.721 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.721 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.721 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.722 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.722 INFO:teuthology.orchestra.run.gibba002.stdout: "port": { 2021-02-10T11:04:02.722 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "port", 2021-02-10T11:04:02.722 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.722 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.722 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.722 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "8086", 2021-02-10T11:04:02.722 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.722 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.722 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.722 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.722 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.723 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.723 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.723 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.723 INFO:teuthology.orchestra.run.gibba002.stdout: "ssl": { 2021-02-10T11:04:02.723 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "ssl", 2021-02-10T11:04:02.723 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.723 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.723 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.723 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "false", 2021-02-10T11:04:02.723 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.723 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.723 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.723 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.723 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.724 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.724 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.724 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.724 INFO:teuthology.orchestra.run.gibba002.stdout: "threads": { 2021-02-10T11:04:02.724 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "threads", 2021-02-10T11:04:02.724 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.724 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.724 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.724 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "5", 2021-02-10T11:04:02.724 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.724 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.724 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.724 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.725 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.725 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.725 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.725 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.725 INFO:teuthology.orchestra.run.gibba002.stdout: "username": { 2021-02-10T11:04:02.725 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "username", 2021-02-10T11:04:02.725 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.725 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.725 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.725 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "None", 2021-02-10T11:04:02.725 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.725 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.725 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.725 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.725 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.726 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.726 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.726 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.726 INFO:teuthology.orchestra.run.gibba002.stdout: "verify_ssl": { 2021-02-10T11:04:02.726 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "verify_ssl", 2021-02-10T11:04:02.726 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.726 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.726 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.726 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "true", 2021-02-10T11:04:02.726 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.726 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.726 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.726 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.727 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.727 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.727 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.727 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:04:02.727 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:04:02.727 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.727 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:04:02.727 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "insights", 2021-02-10T11:04:02.727 INFO:teuthology.orchestra.run.gibba002.stdout: "can_run": true, 2021-02-10T11:04:02.727 INFO:teuthology.orchestra.run.gibba002.stdout: "error_string": "", 2021-02-10T11:04:02.727 INFO:teuthology.orchestra.run.gibba002.stdout: "module_options": {} 2021-02-10T11:04:02.727 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.727 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:04:02.727 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "iostat", 2021-02-10T11:04:02.728 INFO:teuthology.orchestra.run.gibba002.stdout: "can_run": true, 2021-02-10T11:04:02.728 INFO:teuthology.orchestra.run.gibba002.stdout: "error_string": "", 2021-02-10T11:04:02.728 INFO:teuthology.orchestra.run.gibba002.stdout: "module_options": {} 2021-02-10T11:04:02.728 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.728 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:04:02.728 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "localpool", 2021-02-10T11:04:02.728 INFO:teuthology.orchestra.run.gibba002.stdout: "can_run": true, 2021-02-10T11:04:02.728 INFO:teuthology.orchestra.run.gibba002.stdout: "error_string": "", 2021-02-10T11:04:02.728 INFO:teuthology.orchestra.run.gibba002.stdout: "module_options": { 2021-02-10T11:04:02.728 INFO:teuthology.orchestra.run.gibba002.stdout: "failure_domain": { 2021-02-10T11:04:02.728 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "failure_domain", 2021-02-10T11:04:02.728 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.728 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.729 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 1, 2021-02-10T11:04:02.729 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "host", 2021-02-10T11:04:02.729 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.729 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.729 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.729 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "failure domain for any created local pool", 2021-02-10T11:04:02.729 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.729 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.729 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.729 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.729 INFO:teuthology.orchestra.run.gibba002.stdout: "min_size": { 2021-02-10T11:04:02.729 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "min_size", 2021-02-10T11:04:02.729 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "int", 2021-02-10T11:04:02.730 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.730 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 1, 2021-02-10T11:04:02.730 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "", 2021-02-10T11:04:02.730 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.730 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.730 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.730 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "default min_size for any created local pool", 2021-02-10T11:04:02.730 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.730 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.730 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.730 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.730 INFO:teuthology.orchestra.run.gibba002.stdout: "num_rep": { 2021-02-10T11:04:02.730 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "num_rep", 2021-02-10T11:04:02.731 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "int", 2021-02-10T11:04:02.731 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.731 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 1, 2021-02-10T11:04:02.731 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "3", 2021-02-10T11:04:02.731 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.731 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.731 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.731 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "default replica count for any created local pool", 2021-02-10T11:04:02.731 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.731 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.731 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.731 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.731 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num": { 2021-02-10T11:04:02.731 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "pg_num", 2021-02-10T11:04:02.732 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "int", 2021-02-10T11:04:02.732 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.732 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 1, 2021-02-10T11:04:02.732 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "128", 2021-02-10T11:04:02.732 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.732 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.732 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.732 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "default pg_num for any created local pool", 2021-02-10T11:04:02.732 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.732 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.732 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.732 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.732 INFO:teuthology.orchestra.run.gibba002.stdout: "prefix": { 2021-02-10T11:04:02.733 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "prefix", 2021-02-10T11:04:02.733 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.733 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.733 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 1, 2021-02-10T11:04:02.733 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "", 2021-02-10T11:04:02.733 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.733 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.733 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.733 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "name prefix for any created local pool", 2021-02-10T11:04:02.733 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.733 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.733 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.733 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.733 INFO:teuthology.orchestra.run.gibba002.stdout: "subtree": { 2021-02-10T11:04:02.734 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "subtree", 2021-02-10T11:04:02.734 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.734 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.734 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 1, 2021-02-10T11:04:02.734 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "rack", 2021-02-10T11:04:02.734 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.734 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.734 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.734 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "CRUSH level for which to create a local pool", 2021-02-10T11:04:02.734 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.734 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.734 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.734 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:04:02.735 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:04:02.735 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.735 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:04:02.735 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "orchestrator_cli", 2021-02-10T11:04:02.735 INFO:teuthology.orchestra.run.gibba002.stdout: "can_run": true, 2021-02-10T11:04:02.735 INFO:teuthology.orchestra.run.gibba002.stdout: "error_string": "", 2021-02-10T11:04:02.735 INFO:teuthology.orchestra.run.gibba002.stdout: "module_options": { 2021-02-10T11:04:02.735 INFO:teuthology.orchestra.run.gibba002.stdout: "orchestrator": { 2021-02-10T11:04:02.735 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "orchestrator", 2021-02-10T11:04:02.735 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.735 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.735 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.735 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "", 2021-02-10T11:04:02.735 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.736 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.736 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.736 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.736 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.736 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.736 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.736 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:04:02.736 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:04:02.736 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.736 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:04:02.736 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "pg_autoscaler", 2021-02-10T11:04:02.736 INFO:teuthology.orchestra.run.gibba002.stdout: "can_run": true, 2021-02-10T11:04:02.736 INFO:teuthology.orchestra.run.gibba002.stdout: "error_string": "", 2021-02-10T11:04:02.736 INFO:teuthology.orchestra.run.gibba002.stdout: "module_options": { 2021-02-10T11:04:02.737 INFO:teuthology.orchestra.run.gibba002.stdout: "sleep_interval": { 2021-02-10T11:04:02.737 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "sleep_interval", 2021-02-10T11:04:02.737 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.737 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.737 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.737 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "60", 2021-02-10T11:04:02.737 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.737 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.737 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.737 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.737 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.737 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.737 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.738 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:04:02.738 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:04:02.738 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.738 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:04:02.738 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "progress", 2021-02-10T11:04:02.738 INFO:teuthology.orchestra.run.gibba002.stdout: "can_run": true, 2021-02-10T11:04:02.738 INFO:teuthology.orchestra.run.gibba002.stdout: "error_string": "", 2021-02-10T11:04:02.738 INFO:teuthology.orchestra.run.gibba002.stdout: "module_options": { 2021-02-10T11:04:02.738 INFO:teuthology.orchestra.run.gibba002.stdout: "enabled": { 2021-02-10T11:04:02.738 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "enabled", 2021-02-10T11:04:02.738 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "bool", 2021-02-10T11:04:02.738 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.738 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.739 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "True", 2021-02-10T11:04:02.739 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.739 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.739 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.739 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.739 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.739 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.739 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.739 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.739 INFO:teuthology.orchestra.run.gibba002.stdout: "max_completed_events": { 2021-02-10T11:04:02.739 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "max_completed_events", 2021-02-10T11:04:02.739 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "int", 2021-02-10T11:04:02.739 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.739 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 1, 2021-02-10T11:04:02.740 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "50", 2021-02-10T11:04:02.740 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.740 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.740 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.740 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "number of past completed events to remember", 2021-02-10T11:04:02.740 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.740 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.740 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.740 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.740 INFO:teuthology.orchestra.run.gibba002.stdout: "persist_interval": { 2021-02-10T11:04:02.740 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "persist_interval", 2021-02-10T11:04:02.740 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "secs", 2021-02-10T11:04:02.740 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.741 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 1, 2021-02-10T11:04:02.741 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "5", 2021-02-10T11:04:02.741 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.741 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.741 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.741 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "how frequently to persist completed events", 2021-02-10T11:04:02.741 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.741 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.741 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.741 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:04:02.741 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:04:02.741 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.741 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:04:02.742 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "prometheus", 2021-02-10T11:04:02.742 INFO:teuthology.orchestra.run.gibba002.stdout: "can_run": true, 2021-02-10T11:04:02.742 INFO:teuthology.orchestra.run.gibba002.stdout: "error_string": "", 2021-02-10T11:04:02.742 INFO:teuthology.orchestra.run.gibba002.stdout: "module_options": { 2021-02-10T11:04:02.742 INFO:teuthology.orchestra.run.gibba002.stdout: "rbd_stats_pools": { 2021-02-10T11:04:02.742 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "rbd_stats_pools", 2021-02-10T11:04:02.742 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.742 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.742 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.742 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "", 2021-02-10T11:04:02.742 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.742 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.742 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.742 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.743 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.743 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.743 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.743 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.743 INFO:teuthology.orchestra.run.gibba002.stdout: "rbd_stats_pools_refresh_interval": { 2021-02-10T11:04:02.743 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "rbd_stats_pools_refresh_interval", 2021-02-10T11:04:02.743 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "int", 2021-02-10T11:04:02.743 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.743 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.743 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "300", 2021-02-10T11:04:02.743 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.743 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.743 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.744 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.744 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.744 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.744 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.744 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.744 INFO:teuthology.orchestra.run.gibba002.stdout: "scrape_interval": { 2021-02-10T11:04:02.744 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "scrape_interval", 2021-02-10T11:04:02.744 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.744 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.744 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.744 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "", 2021-02-10T11:04:02.744 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.744 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.745 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.745 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.745 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.745 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.745 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.745 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.745 INFO:teuthology.orchestra.run.gibba002.stdout: "server_addr": { 2021-02-10T11:04:02.745 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "server_addr", 2021-02-10T11:04:02.745 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.745 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.745 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.745 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "", 2021-02-10T11:04:02.745 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.745 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.745 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.746 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.746 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.746 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.746 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.746 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.746 INFO:teuthology.orchestra.run.gibba002.stdout: "server_port": { 2021-02-10T11:04:02.746 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "server_port", 2021-02-10T11:04:02.746 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.746 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.746 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.746 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "", 2021-02-10T11:04:02.746 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.747 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.747 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.747 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.747 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.747 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.747 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.747 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.747 INFO:teuthology.orchestra.run.gibba002.stdout: "stale_cache_strategy": { 2021-02-10T11:04:02.747 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "stale_cache_strategy", 2021-02-10T11:04:02.747 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.747 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.747 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.747 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "", 2021-02-10T11:04:02.748 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.748 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.748 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.748 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.748 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.748 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.748 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.748 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:04:02.748 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:04:02.748 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.748 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:04:02.748 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "rbd_support", 2021-02-10T11:04:02.748 INFO:teuthology.orchestra.run.gibba002.stdout: "can_run": true, 2021-02-10T11:04:02.748 INFO:teuthology.orchestra.run.gibba002.stdout: "error_string": "", 2021-02-10T11:04:02.749 INFO:teuthology.orchestra.run.gibba002.stdout: "module_options": {} 2021-02-10T11:04:02.749 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.749 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:04:02.749 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "restful", 2021-02-10T11:04:02.749 INFO:teuthology.orchestra.run.gibba002.stdout: "can_run": true, 2021-02-10T11:04:02.749 INFO:teuthology.orchestra.run.gibba002.stdout: "error_string": "", 2021-02-10T11:04:02.749 INFO:teuthology.orchestra.run.gibba002.stdout: "module_options": { 2021-02-10T11:04:02.749 INFO:teuthology.orchestra.run.gibba002.stdout: "key_file": { 2021-02-10T11:04:02.749 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "key_file", 2021-02-10T11:04:02.749 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.749 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.749 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.749 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "", 2021-02-10T11:04:02.749 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.750 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.750 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.750 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.750 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.750 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.750 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.750 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.750 INFO:teuthology.orchestra.run.gibba002.stdout: "server_addr": { 2021-02-10T11:04:02.750 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "server_addr", 2021-02-10T11:04:02.750 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.750 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.750 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.750 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "", 2021-02-10T11:04:02.751 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.751 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.751 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.751 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.751 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.751 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.751 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.751 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.751 INFO:teuthology.orchestra.run.gibba002.stdout: "server_port": { 2021-02-10T11:04:02.751 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "server_port", 2021-02-10T11:04:02.751 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.751 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.751 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.752 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "", 2021-02-10T11:04:02.752 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.752 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.752 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.752 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.752 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.752 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.752 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.752 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:04:02.752 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:04:02.752 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.752 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:04:02.752 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "rook", 2021-02-10T11:04:02.752 INFO:teuthology.orchestra.run.gibba002.stdout: "can_run": true, 2021-02-10T11:04:02.753 INFO:teuthology.orchestra.run.gibba002.stdout: "error_string": "", 2021-02-10T11:04:02.753 INFO:teuthology.orchestra.run.gibba002.stdout: "module_options": {} 2021-02-10T11:04:02.753 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.753 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:04:02.753 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "selftest", 2021-02-10T11:04:02.753 INFO:teuthology.orchestra.run.gibba002.stdout: "can_run": true, 2021-02-10T11:04:02.753 INFO:teuthology.orchestra.run.gibba002.stdout: "error_string": "", 2021-02-10T11:04:02.753 INFO:teuthology.orchestra.run.gibba002.stdout: "module_options": { 2021-02-10T11:04:02.753 INFO:teuthology.orchestra.run.gibba002.stdout: "roption1": { 2021-02-10T11:04:02.753 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "roption1", 2021-02-10T11:04:02.753 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.753 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.753 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.754 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "", 2021-02-10T11:04:02.754 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.754 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.754 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.754 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.754 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.754 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.754 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.754 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.754 INFO:teuthology.orchestra.run.gibba002.stdout: "roption2": { 2021-02-10T11:04:02.754 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "roption2", 2021-02-10T11:04:02.754 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.754 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.755 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.755 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "xyz", 2021-02-10T11:04:02.755 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.755 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.755 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.755 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.755 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.755 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.755 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.755 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.755 INFO:teuthology.orchestra.run.gibba002.stdout: "rwoption1": { 2021-02-10T11:04:02.755 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "rwoption1", 2021-02-10T11:04:02.755 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.755 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.756 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.756 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "", 2021-02-10T11:04:02.756 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.756 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.756 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.756 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.756 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.756 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.756 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.756 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.756 INFO:teuthology.orchestra.run.gibba002.stdout: "rwoption2": { 2021-02-10T11:04:02.756 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "rwoption2", 2021-02-10T11:04:02.756 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "int", 2021-02-10T11:04:02.756 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.757 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.757 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "", 2021-02-10T11:04:02.757 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.757 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.757 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.757 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.757 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.757 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.757 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.757 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.757 INFO:teuthology.orchestra.run.gibba002.stdout: "rwoption3": { 2021-02-10T11:04:02.757 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "rwoption3", 2021-02-10T11:04:02.757 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "float", 2021-02-10T11:04:02.758 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.758 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.758 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "", 2021-02-10T11:04:02.758 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.758 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.758 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.758 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.758 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.758 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.758 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.758 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.758 INFO:teuthology.orchestra.run.gibba002.stdout: "rwoption4": { 2021-02-10T11:04:02.758 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "rwoption4", 2021-02-10T11:04:02.759 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.759 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.759 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.759 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "", 2021-02-10T11:04:02.759 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.759 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.759 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.759 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.759 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.759 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.759 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.759 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.759 INFO:teuthology.orchestra.run.gibba002.stdout: "rwoption5": { 2021-02-10T11:04:02.759 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "rwoption5", 2021-02-10T11:04:02.760 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "bool", 2021-02-10T11:04:02.760 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.760 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.760 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "", 2021-02-10T11:04:02.760 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.760 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.760 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.760 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.760 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.760 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.760 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.760 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.760 INFO:teuthology.orchestra.run.gibba002.stdout: "rwoption6": { 2021-02-10T11:04:02.760 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "rwoption6", 2021-02-10T11:04:02.761 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "bool", 2021-02-10T11:04:02.761 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.761 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.761 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "True", 2021-02-10T11:04:02.761 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.761 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.761 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.761 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.761 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.761 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.761 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.761 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.761 INFO:teuthology.orchestra.run.gibba002.stdout: "testkey": { 2021-02-10T11:04:02.761 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "testkey", 2021-02-10T11:04:02.762 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.762 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.762 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.762 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "", 2021-02-10T11:04:02.762 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.762 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.762 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.762 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.762 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.762 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.762 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.762 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.762 INFO:teuthology.orchestra.run.gibba002.stdout: "testlkey": { 2021-02-10T11:04:02.763 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "testlkey", 2021-02-10T11:04:02.763 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.763 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.763 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.763 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "", 2021-02-10T11:04:02.763 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.763 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.763 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.763 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.763 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.763 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.763 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.763 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.763 INFO:teuthology.orchestra.run.gibba002.stdout: "testnewline": { 2021-02-10T11:04:02.764 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "testnewline", 2021-02-10T11:04:02.764 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.764 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.764 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.764 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "", 2021-02-10T11:04:02.764 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.764 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.764 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.764 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.764 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.764 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.764 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.764 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:04:02.764 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:04:02.765 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.765 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:04:02.765 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "ssh", 2021-02-10T11:04:02.765 INFO:teuthology.orchestra.run.gibba002.stdout: "can_run": true, 2021-02-10T11:04:02.765 INFO:teuthology.orchestra.run.gibba002.stdout: "error_string": "", 2021-02-10T11:04:02.765 INFO:teuthology.orchestra.run.gibba002.stdout: "module_options": { 2021-02-10T11:04:02.765 INFO:teuthology.orchestra.run.gibba002.stdout: "inventory_cache_timeout_min": { 2021-02-10T11:04:02.765 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "inventory_cache_timeout_min", 2021-02-10T11:04:02.765 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.765 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.765 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.765 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "", 2021-02-10T11:04:02.765 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.765 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.766 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.766 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.766 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.766 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.766 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.766 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.766 INFO:teuthology.orchestra.run.gibba002.stdout: "ssh_config_file": { 2021-02-10T11:04:02.766 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "ssh_config_file", 2021-02-10T11:04:02.766 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.766 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.766 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.766 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "", 2021-02-10T11:04:02.766 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.767 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.767 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.767 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.767 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.767 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.767 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.767 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:04:02.767 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:04:02.767 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.767 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:04:02.767 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "status", 2021-02-10T11:04:02.767 INFO:teuthology.orchestra.run.gibba002.stdout: "can_run": true, 2021-02-10T11:04:02.767 INFO:teuthology.orchestra.run.gibba002.stdout: "error_string": "", 2021-02-10T11:04:02.767 INFO:teuthology.orchestra.run.gibba002.stdout: "module_options": {} 2021-02-10T11:04:02.768 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.768 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:04:02.768 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "telegraf", 2021-02-10T11:04:02.768 INFO:teuthology.orchestra.run.gibba002.stdout: "can_run": true, 2021-02-10T11:04:02.768 INFO:teuthology.orchestra.run.gibba002.stdout: "error_string": "", 2021-02-10T11:04:02.768 INFO:teuthology.orchestra.run.gibba002.stdout: "module_options": { 2021-02-10T11:04:02.768 INFO:teuthology.orchestra.run.gibba002.stdout: "address": { 2021-02-10T11:04:02.768 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "address", 2021-02-10T11:04:02.768 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.768 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.768 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.768 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2021-02-10T11:04:02.768 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.768 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.769 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.769 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.769 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.769 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.769 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.769 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.769 INFO:teuthology.orchestra.run.gibba002.stdout: "interval": { 2021-02-10T11:04:02.769 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "interval", 2021-02-10T11:04:02.769 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "secs", 2021-02-10T11:04:02.769 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.769 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.769 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "15", 2021-02-10T11:04:02.769 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.769 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.770 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.770 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.770 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.770 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.770 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.770 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:04:02.770 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:04:02.770 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.770 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:04:02.770 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "telemetry", 2021-02-10T11:04:02.770 INFO:teuthology.orchestra.run.gibba002.stdout: "can_run": true, 2021-02-10T11:04:02.770 INFO:teuthology.orchestra.run.gibba002.stdout: "error_string": "", 2021-02-10T11:04:02.770 INFO:teuthology.orchestra.run.gibba002.stdout: "module_options": { 2021-02-10T11:04:02.771 INFO:teuthology.orchestra.run.gibba002.stdout: "channel_basic": { 2021-02-10T11:04:02.771 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "channel_basic", 2021-02-10T11:04:02.771 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "bool", 2021-02-10T11:04:02.771 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.771 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.771 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "True", 2021-02-10T11:04:02.771 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.771 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.771 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.771 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "Share basic cluster information (size, version)", 2021-02-10T11:04:02.771 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.771 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.771 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.771 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.772 INFO:teuthology.orchestra.run.gibba002.stdout: "channel_crash": { 2021-02-10T11:04:02.772 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "channel_crash", 2021-02-10T11:04:02.772 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "bool", 2021-02-10T11:04:02.772 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.772 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.772 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "True", 2021-02-10T11:04:02.772 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.772 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.772 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.772 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.772 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.772 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.772 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.772 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.773 INFO:teuthology.orchestra.run.gibba002.stdout: "channel_device": { 2021-02-10T11:04:02.773 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "channel_device", 2021-02-10T11:04:02.773 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "bool", 2021-02-10T11:04:02.773 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.773 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.773 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "True", 2021-02-10T11:04:02.773 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.773 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.773 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.773 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.773 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.773 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.773 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.774 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.774 INFO:teuthology.orchestra.run.gibba002.stdout: "channel_ident": { 2021-02-10T11:04:02.774 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "channel_ident", 2021-02-10T11:04:02.774 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "bool", 2021-02-10T11:04:02.774 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.774 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.774 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "False", 2021-02-10T11:04:02.774 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.774 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.774 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.774 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.774 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.774 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.775 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.775 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.775 INFO:teuthology.orchestra.run.gibba002.stdout: "contact": { 2021-02-10T11:04:02.775 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "contact", 2021-02-10T11:04:02.775 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.775 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.775 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.775 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "None", 2021-02-10T11:04:02.775 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.775 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.775 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.775 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.775 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.775 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.776 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.776 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.776 INFO:teuthology.orchestra.run.gibba002.stdout: "description": { 2021-02-10T11:04:02.776 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "description", 2021-02-10T11:04:02.776 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.776 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.776 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.776 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "None", 2021-02-10T11:04:02.776 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.776 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.776 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.776 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.776 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.776 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.777 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.777 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.777 INFO:teuthology.orchestra.run.gibba002.stdout: "device_url": { 2021-02-10T11:04:02.777 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "device_url", 2021-02-10T11:04:02.777 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.777 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.777 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.777 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "https://telemetry.ceph.com/device", 2021-02-10T11:04:02.777 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.777 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.777 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.777 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.777 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.777 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.778 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.778 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.778 INFO:teuthology.orchestra.run.gibba002.stdout: "enabled": { 2021-02-10T11:04:02.778 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "enabled", 2021-02-10T11:04:02.778 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "bool", 2021-02-10T11:04:02.778 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.778 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.778 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "False", 2021-02-10T11:04:02.778 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.778 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.778 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.778 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.779 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.779 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.779 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.779 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.779 INFO:teuthology.orchestra.run.gibba002.stdout: "interval": { 2021-02-10T11:04:02.779 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "interval", 2021-02-10T11:04:02.779 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "int", 2021-02-10T11:04:02.779 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.779 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.779 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "24", 2021-02-10T11:04:02.779 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "8", 2021-02-10T11:04:02.779 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.779 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.779 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.780 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.780 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.780 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.780 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.780 INFO:teuthology.orchestra.run.gibba002.stdout: "last_opt_revision": { 2021-02-10T11:04:02.780 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "last_opt_revision", 2021-02-10T11:04:02.780 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "int", 2021-02-10T11:04:02.780 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.780 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.780 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "1", 2021-02-10T11:04:02.780 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.780 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.780 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.781 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.781 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.781 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.781 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.781 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.781 INFO:teuthology.orchestra.run.gibba002.stdout: "leaderboard": { 2021-02-10T11:04:02.781 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "leaderboard", 2021-02-10T11:04:02.781 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "bool", 2021-02-10T11:04:02.781 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.781 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.781 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "False", 2021-02-10T11:04:02.781 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.781 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.781 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.782 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.782 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.782 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.782 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.782 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.782 INFO:teuthology.orchestra.run.gibba002.stdout: "organization": { 2021-02-10T11:04:02.782 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "organization", 2021-02-10T11:04:02.782 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.782 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.782 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.782 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "None", 2021-02-10T11:04:02.782 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.782 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.783 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.783 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.783 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.783 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.783 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.783 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.783 INFO:teuthology.orchestra.run.gibba002.stdout: "proxy": { 2021-02-10T11:04:02.783 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "proxy", 2021-02-10T11:04:02.783 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.783 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.783 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.783 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "None", 2021-02-10T11:04:02.783 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.783 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.784 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.784 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.784 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.784 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.784 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.784 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.784 INFO:teuthology.orchestra.run.gibba002.stdout: "url": { 2021-02-10T11:04:02.784 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "url", 2021-02-10T11:04:02.784 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.784 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.784 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.784 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "https://telemetry.ceph.com/report", 2021-02-10T11:04:02.784 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.785 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.785 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.785 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.785 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.785 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.785 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.785 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:04:02.785 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:04:02.785 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.785 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:04:02.785 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "test_orchestrator", 2021-02-10T11:04:02.785 INFO:teuthology.orchestra.run.gibba002.stdout: "can_run": true, 2021-02-10T11:04:02.785 INFO:teuthology.orchestra.run.gibba002.stdout: "error_string": "", 2021-02-10T11:04:02.785 INFO:teuthology.orchestra.run.gibba002.stdout: "module_options": {} 2021-02-10T11:04:02.786 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.786 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:04:02.786 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "volumes", 2021-02-10T11:04:02.786 INFO:teuthology.orchestra.run.gibba002.stdout: "can_run": true, 2021-02-10T11:04:02.786 INFO:teuthology.orchestra.run.gibba002.stdout: "error_string": "", 2021-02-10T11:04:02.786 INFO:teuthology.orchestra.run.gibba002.stdout: "module_options": { 2021-02-10T11:04:02.786 INFO:teuthology.orchestra.run.gibba002.stdout: "max_concurrent_clones": { 2021-02-10T11:04:02.786 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "max_concurrent_clones", 2021-02-10T11:04:02.786 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "int", 2021-02-10T11:04:02.786 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.786 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.786 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "4", 2021-02-10T11:04:02.786 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.787 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.787 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.787 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "Number of asynchronous cloner threads", 2021-02-10T11:04:02.787 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.787 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.787 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.787 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:04:02.787 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:04:02.787 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.787 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:04:02.787 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "zabbix", 2021-02-10T11:04:02.787 INFO:teuthology.orchestra.run.gibba002.stdout: "can_run": true, 2021-02-10T11:04:02.787 INFO:teuthology.orchestra.run.gibba002.stdout: "error_string": "", 2021-02-10T11:04:02.787 INFO:teuthology.orchestra.run.gibba002.stdout: "module_options": { 2021-02-10T11:04:02.788 INFO:teuthology.orchestra.run.gibba002.stdout: "identifier": { 2021-02-10T11:04:02.788 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "identifier", 2021-02-10T11:04:02.788 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.788 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.788 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.788 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "", 2021-02-10T11:04:02.788 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.788 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.788 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.788 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.788 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.788 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.788 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.788 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.789 INFO:teuthology.orchestra.run.gibba002.stdout: "interval": { 2021-02-10T11:04:02.789 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "interval", 2021-02-10T11:04:02.789 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "secs", 2021-02-10T11:04:02.789 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.789 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.789 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "60", 2021-02-10T11:04:02.789 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.789 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.789 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.789 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.789 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.789 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.789 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.790 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.790 INFO:teuthology.orchestra.run.gibba002.stdout: "zabbix_host": { 2021-02-10T11:04:02.790 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "zabbix_host", 2021-02-10T11:04:02.790 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.790 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.790 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.790 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "None", 2021-02-10T11:04:02.790 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.790 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.790 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.790 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.790 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.790 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.791 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.791 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.791 INFO:teuthology.orchestra.run.gibba002.stdout: "zabbix_port": { 2021-02-10T11:04:02.791 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "zabbix_port", 2021-02-10T11:04:02.791 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "int", 2021-02-10T11:04:02.791 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.791 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.791 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "10051", 2021-02-10T11:04:02.791 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.791 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.791 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.791 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.791 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.792 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.792 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.792 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.792 INFO:teuthology.orchestra.run.gibba002.stdout: "zabbix_sender": { 2021-02-10T11:04:02.792 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "zabbix_sender", 2021-02-10T11:04:02.792 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.792 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.792 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.792 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "/usr/bin/zabbix_sender", 2021-02-10T11:04:02.792 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.792 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.792 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.792 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.793 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.793 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.793 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.793 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:04:02.793 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:04:02.793 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:04:02.793 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:04:02.793 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:04:02.793 INFO:teuthology.orchestra.run.gibba002.stdout: ], 2021-02-10T11:04:02.793 INFO:teuthology.orchestra.run.gibba002.stdout: "modules": [ 2021-02-10T11:04:02.793 INFO:teuthology.orchestra.run.gibba002.stdout: "iostat", 2021-02-10T11:04:02.793 INFO:teuthology.orchestra.run.gibba002.stdout: "restful" 2021-02-10T11:04:02.793 INFO:teuthology.orchestra.run.gibba002.stdout: ], 2021-02-10T11:04:02.793 INFO:teuthology.orchestra.run.gibba002.stdout: "available_modules": [ 2021-02-10T11:04:02.794 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:04:02.794 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "alerts", 2021-02-10T11:04:02.794 INFO:teuthology.orchestra.run.gibba002.stdout: "can_run": true, 2021-02-10T11:04:02.794 INFO:teuthology.orchestra.run.gibba002.stdout: "error_string": "", 2021-02-10T11:04:02.794 INFO:teuthology.orchestra.run.gibba002.stdout: "module_options": { 2021-02-10T11:04:02.794 INFO:teuthology.orchestra.run.gibba002.stdout: "interval": { 2021-02-10T11:04:02.794 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "interval", 2021-02-10T11:04:02.794 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "secs", 2021-02-10T11:04:02.794 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.794 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 1, 2021-02-10T11:04:02.794 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "60", 2021-02-10T11:04:02.794 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.794 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.795 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.795 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "How frequently to reexamine health status", 2021-02-10T11:04:02.795 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.795 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.795 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.795 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.795 INFO:teuthology.orchestra.run.gibba002.stdout: "smtp_destination": { 2021-02-10T11:04:02.795 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "smtp_destination", 2021-02-10T11:04:02.795 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.795 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.795 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 1, 2021-02-10T11:04:02.795 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "", 2021-02-10T11:04:02.795 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.795 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.796 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.796 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "Email address to send alerts to", 2021-02-10T11:04:02.796 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.796 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.796 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.796 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.796 INFO:teuthology.orchestra.run.gibba002.stdout: "smtp_from_name": { 2021-02-10T11:04:02.796 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "smtp_from_name", 2021-02-10T11:04:02.796 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.796 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.796 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 1, 2021-02-10T11:04:02.796 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "Ceph", 2021-02-10T11:04:02.796 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.797 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.797 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.797 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "Email From: name", 2021-02-10T11:04:02.797 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.797 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.797 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.797 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.797 INFO:teuthology.orchestra.run.gibba002.stdout: "smtp_host": { 2021-02-10T11:04:02.797 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "smtp_host", 2021-02-10T11:04:02.797 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.797 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.797 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 1, 2021-02-10T11:04:02.797 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "", 2021-02-10T11:04:02.797 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.798 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.798 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.798 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "SMTP server", 2021-02-10T11:04:02.798 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.798 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.798 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.798 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.798 INFO:teuthology.orchestra.run.gibba002.stdout: "smtp_password": { 2021-02-10T11:04:02.798 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "smtp_password", 2021-02-10T11:04:02.798 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.798 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.798 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 1, 2021-02-10T11:04:02.799 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "", 2021-02-10T11:04:02.799 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.799 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.799 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.799 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "Password to authenticate with", 2021-02-10T11:04:02.799 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.799 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.799 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.799 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.799 INFO:teuthology.orchestra.run.gibba002.stdout: "smtp_port": { 2021-02-10T11:04:02.799 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "smtp_port", 2021-02-10T11:04:02.799 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "int", 2021-02-10T11:04:02.799 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.799 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 1, 2021-02-10T11:04:02.800 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "465", 2021-02-10T11:04:02.800 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.800 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.800 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.800 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "SMTP port", 2021-02-10T11:04:02.800 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.800 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.800 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.800 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.800 INFO:teuthology.orchestra.run.gibba002.stdout: "smtp_sender": { 2021-02-10T11:04:02.800 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "smtp_sender", 2021-02-10T11:04:02.800 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.800 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.801 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 1, 2021-02-10T11:04:02.801 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "", 2021-02-10T11:04:02.801 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.801 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.801 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.801 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "SMTP envelope sender", 2021-02-10T11:04:02.801 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.801 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.801 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.801 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.801 INFO:teuthology.orchestra.run.gibba002.stdout: "smtp_ssl": { 2021-02-10T11:04:02.801 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "smtp_ssl", 2021-02-10T11:04:02.801 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "bool", 2021-02-10T11:04:02.801 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.802 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 1, 2021-02-10T11:04:02.802 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "True", 2021-02-10T11:04:02.802 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.802 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.802 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.802 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "Use SSL to connect to SMTP server", 2021-02-10T11:04:02.802 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.802 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.802 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.802 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.802 INFO:teuthology.orchestra.run.gibba002.stdout: "smtp_user": { 2021-02-10T11:04:02.802 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "smtp_user", 2021-02-10T11:04:02.802 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.803 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.803 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 1, 2021-02-10T11:04:02.803 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "", 2021-02-10T11:04:02.803 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.803 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.803 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.803 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "User to authenticate as", 2021-02-10T11:04:02.803 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.803 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.803 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.803 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:04:02.803 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:04:02.803 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.804 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:04:02.804 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "ansible", 2021-02-10T11:04:02.804 INFO:teuthology.orchestra.run.gibba002.stdout: "can_run": true, 2021-02-10T11:04:02.804 INFO:teuthology.orchestra.run.gibba002.stdout: "error_string": "", 2021-02-10T11:04:02.804 INFO:teuthology.orchestra.run.gibba002.stdout: "module_options": { 2021-02-10T11:04:02.804 INFO:teuthology.orchestra.run.gibba002.stdout: "password": { 2021-02-10T11:04:02.804 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "password", 2021-02-10T11:04:02.804 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.804 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.804 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.804 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "", 2021-02-10T11:04:02.804 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.804 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.805 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.805 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.805 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.805 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.805 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.805 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.805 INFO:teuthology.orchestra.run.gibba002.stdout: "server_url": { 2021-02-10T11:04:02.805 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "server_url", 2021-02-10T11:04:02.805 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.805 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.805 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.805 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "", 2021-02-10T11:04:02.806 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.806 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.806 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.806 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.806 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.806 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.806 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.806 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.806 INFO:teuthology.orchestra.run.gibba002.stdout: "username": { 2021-02-10T11:04:02.806 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "username", 2021-02-10T11:04:02.806 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.806 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.806 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.807 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "", 2021-02-10T11:04:02.807 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.807 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.807 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.807 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.807 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.807 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.807 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.807 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.807 INFO:teuthology.orchestra.run.gibba002.stdout: "verify_server": { 2021-02-10T11:04:02.807 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "verify_server", 2021-02-10T11:04:02.807 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.807 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.808 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.808 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "", 2021-02-10T11:04:02.808 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.808 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.808 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.808 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.808 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.808 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.808 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.808 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:04:02.808 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:04:02.808 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.808 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:04:02.808 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "balancer", 2021-02-10T11:04:02.809 INFO:teuthology.orchestra.run.gibba002.stdout: "can_run": true, 2021-02-10T11:04:02.809 INFO:teuthology.orchestra.run.gibba002.stdout: "error_string": "", 2021-02-10T11:04:02.809 INFO:teuthology.orchestra.run.gibba002.stdout: "module_options": { 2021-02-10T11:04:02.809 INFO:teuthology.orchestra.run.gibba002.stdout: "active": { 2021-02-10T11:04:02.809 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "active", 2021-02-10T11:04:02.809 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "bool", 2021-02-10T11:04:02.809 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.809 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 1, 2021-02-10T11:04:02.809 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "False", 2021-02-10T11:04:02.809 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.809 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.809 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.809 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "automatically balance PGs across cluster", 2021-02-10T11:04:02.810 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.810 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.810 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.810 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.810 INFO:teuthology.orchestra.run.gibba002.stdout: "begin_time": { 2021-02-10T11:04:02.810 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "begin_time", 2021-02-10T11:04:02.810 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.810 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.810 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 1, 2021-02-10T11:04:02.810 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "0000", 2021-02-10T11:04:02.810 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.810 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.810 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.810 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "beginning time of day to automatically balance", 2021-02-10T11:04:02.811 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "This is a time of day in the format HHMM.", 2021-02-10T11:04:02.811 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.811 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.811 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.811 INFO:teuthology.orchestra.run.gibba002.stdout: "begin_weekday": { 2021-02-10T11:04:02.811 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "begin_weekday", 2021-02-10T11:04:02.811 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "uint", 2021-02-10T11:04:02.811 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.811 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 1, 2021-02-10T11:04:02.811 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "0", 2021-02-10T11:04:02.811 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "0", 2021-02-10T11:04:02.811 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "7", 2021-02-10T11:04:02.811 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.812 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2021-02-10T11:04:02.812 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "0 or 7 = Sunday, 1 = Monday, etc.", 2021-02-10T11:04:02.812 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.812 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.812 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.812 INFO:teuthology.orchestra.run.gibba002.stdout: "crush_compat_max_iterations": { 2021-02-10T11:04:02.812 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "crush_compat_max_iterations", 2021-02-10T11:04:02.812 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "uint", 2021-02-10T11:04:02.812 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.812 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 1, 2021-02-10T11:04:02.812 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "25", 2021-02-10T11:04:02.812 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "1", 2021-02-10T11:04:02.812 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "250", 2021-02-10T11:04:02.812 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.813 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "maximum number of iterations to attempt optimization", 2021-02-10T11:04:02.813 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.813 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.813 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.813 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.813 INFO:teuthology.orchestra.run.gibba002.stdout: "crush_compat_metrics": { 2021-02-10T11:04:02.813 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "crush_compat_metrics", 2021-02-10T11:04:02.813 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.813 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.813 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 1, 2021-02-10T11:04:02.813 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "pgs,objects,bytes", 2021-02-10T11:04:02.813 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.813 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.813 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.814 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "metrics with which to calculate OSD utilization", 2021-02-10T11:04:02.814 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2021-02-10T11:04:02.814 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.814 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.814 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.814 INFO:teuthology.orchestra.run.gibba002.stdout: "crush_compat_step": { 2021-02-10T11:04:02.814 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "crush_compat_step", 2021-02-10T11:04:02.814 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "float", 2021-02-10T11:04:02.814 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.814 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 1, 2021-02-10T11:04:02.814 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "0.5", 2021-02-10T11:04:02.814 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "0.001", 2021-02-10T11:04:02.815 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "0.999", 2021-02-10T11:04:02.815 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.815 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "aggressiveness of optimization", 2021-02-10T11:04:02.815 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2021-02-10T11:04:02.815 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.815 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.815 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.815 INFO:teuthology.orchestra.run.gibba002.stdout: "end_time": { 2021-02-10T11:04:02.815 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "end_time", 2021-02-10T11:04:02.815 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.815 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.815 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 1, 2021-02-10T11:04:02.815 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "2400", 2021-02-10T11:04:02.815 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.816 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.816 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.816 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "ending time of day to automatically balance", 2021-02-10T11:04:02.816 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "This is a time of day in the format HHMM.", 2021-02-10T11:04:02.816 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.816 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.816 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.816 INFO:teuthology.orchestra.run.gibba002.stdout: "end_weekday": { 2021-02-10T11:04:02.816 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "end_weekday", 2021-02-10T11:04:02.816 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "uint", 2021-02-10T11:04:02.816 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.816 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 1, 2021-02-10T11:04:02.816 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "7", 2021-02-10T11:04:02.816 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "0", 2021-02-10T11:04:02.817 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "7", 2021-02-10T11:04:02.817 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.817 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2021-02-10T11:04:02.817 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "0 or 7 = Sunday, 1 = Monday, etc.", 2021-02-10T11:04:02.817 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.817 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.817 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.817 INFO:teuthology.orchestra.run.gibba002.stdout: "min_score": { 2021-02-10T11:04:02.817 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "min_score", 2021-02-10T11:04:02.817 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "float", 2021-02-10T11:04:02.817 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.817 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 1, 2021-02-10T11:04:02.817 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "0", 2021-02-10T11:04:02.818 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.818 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.818 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.818 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "minimum score, below which no optimization is attempted", 2021-02-10T11:04:02.818 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.818 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.818 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.818 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.818 INFO:teuthology.orchestra.run.gibba002.stdout: "mode": { 2021-02-10T11:04:02.818 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "mode", 2021-02-10T11:04:02.818 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.818 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.818 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 1, 2021-02-10T11:04:02.818 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "none", 2021-02-10T11:04:02.819 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.819 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.819 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [ 2021-02-10T11:04:02.819 INFO:teuthology.orchestra.run.gibba002.stdout: "crush-compat", 2021-02-10T11:04:02.819 INFO:teuthology.orchestra.run.gibba002.stdout: "none", 2021-02-10T11:04:02.819 INFO:teuthology.orchestra.run.gibba002.stdout: "upmap" 2021-02-10T11:04:02.819 INFO:teuthology.orchestra.run.gibba002.stdout: ], 2021-02-10T11:04:02.819 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "Balancer mode", 2021-02-10T11:04:02.819 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.819 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.819 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.819 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.819 INFO:teuthology.orchestra.run.gibba002.stdout: "pool_ids": { 2021-02-10T11:04:02.819 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "pool_ids", 2021-02-10T11:04:02.820 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.820 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.820 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 1, 2021-02-10T11:04:02.820 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "", 2021-02-10T11:04:02.820 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.820 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.820 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.820 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "pools which the automatic balancing will be limited to", 2021-02-10T11:04:02.820 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.820 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.820 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.820 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.820 INFO:teuthology.orchestra.run.gibba002.stdout: "sleep_interval": { 2021-02-10T11:04:02.820 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "sleep_interval", 2021-02-10T11:04:02.821 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "secs", 2021-02-10T11:04:02.821 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.821 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 1, 2021-02-10T11:04:02.821 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "60", 2021-02-10T11:04:02.821 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.821 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.821 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.821 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "how frequently to wake up and attempt optimization", 2021-02-10T11:04:02.821 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.821 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.821 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.821 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.822 INFO:teuthology.orchestra.run.gibba002.stdout: "upmap_max_deviation": { 2021-02-10T11:04:02.822 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "upmap_max_deviation", 2021-02-10T11:04:02.822 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "int", 2021-02-10T11:04:02.822 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.822 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 1, 2021-02-10T11:04:02.822 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "5", 2021-02-10T11:04:02.822 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "1", 2021-02-10T11:04:02.822 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.822 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.822 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "deviation below which no optimization is attempted", 2021-02-10T11:04:02.822 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2021-02-10T11:04:02.822 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.822 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.823 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.823 INFO:teuthology.orchestra.run.gibba002.stdout: "upmap_max_iterations": { 2021-02-10T11:04:02.823 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "upmap_max_iterations", 2021-02-10T11:04:02.823 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "uint", 2021-02-10T11:04:02.823 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.823 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 1, 2021-02-10T11:04:02.823 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "10", 2021-02-10T11:04:02.823 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.823 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.823 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.823 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "maximum upmap optimization iterations", 2021-02-10T11:04:02.823 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.823 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.823 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.824 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:04:02.824 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:04:02.824 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.824 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:04:02.824 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "crash", 2021-02-10T11:04:02.824 INFO:teuthology.orchestra.run.gibba002.stdout: "can_run": true, 2021-02-10T11:04:02.824 INFO:teuthology.orchestra.run.gibba002.stdout: "error_string": "", 2021-02-10T11:04:02.824 INFO:teuthology.orchestra.run.gibba002.stdout: "module_options": { 2021-02-10T11:04:02.824 INFO:teuthology.orchestra.run.gibba002.stdout: "retain_interval": { 2021-02-10T11:04:02.824 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "retain_interval", 2021-02-10T11:04:02.824 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "secs", 2021-02-10T11:04:02.824 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.824 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 1, 2021-02-10T11:04:02.824 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "31536000", 2021-02-10T11:04:02.825 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.825 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.825 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.825 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "how long to retain crashes before pruning them", 2021-02-10T11:04:02.825 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.825 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.825 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.825 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.825 INFO:teuthology.orchestra.run.gibba002.stdout: "warn_recent_interval": { 2021-02-10T11:04:02.825 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "warn_recent_interval", 2021-02-10T11:04:02.825 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "secs", 2021-02-10T11:04:02.825 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.825 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 1, 2021-02-10T11:04:02.826 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "1209600", 2021-02-10T11:04:02.826 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.826 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.826 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.826 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "time interval in which to warn about recent crashes", 2021-02-10T11:04:02.826 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.826 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.826 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.826 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:04:02.826 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:04:02.826 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.826 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:04:02.826 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "dashboard", 2021-02-10T11:04:02.826 INFO:teuthology.orchestra.run.gibba002.stdout: "can_run": true, 2021-02-10T11:04:02.827 INFO:teuthology.orchestra.run.gibba002.stdout: "error_string": "", 2021-02-10T11:04:02.827 INFO:teuthology.orchestra.run.gibba002.stdout: "module_options": { 2021-02-10T11:04:02.827 INFO:teuthology.orchestra.run.gibba002.stdout: "ALERTMANAGER_API_HOST": { 2021-02-10T11:04:02.827 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "ALERTMANAGER_API_HOST", 2021-02-10T11:04:02.827 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.827 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.827 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.827 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "", 2021-02-10T11:04:02.827 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.827 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.827 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.827 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.827 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.828 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.828 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.828 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.828 INFO:teuthology.orchestra.run.gibba002.stdout: "AUDIT_API_ENABLED": { 2021-02-10T11:04:02.828 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "AUDIT_API_ENABLED", 2021-02-10T11:04:02.828 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "bool", 2021-02-10T11:04:02.828 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.828 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.828 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "False", 2021-02-10T11:04:02.828 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.828 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.828 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.828 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.828 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.829 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.829 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.829 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.829 INFO:teuthology.orchestra.run.gibba002.stdout: "AUDIT_API_LOG_PAYLOAD": { 2021-02-10T11:04:02.829 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2021-02-10T11:04:02.829 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "bool", 2021-02-10T11:04:02.829 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.829 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.829 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "True", 2021-02-10T11:04:02.829 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.829 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.829 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.829 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.829 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.830 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.830 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.830 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.830 INFO:teuthology.orchestra.run.gibba002.stdout: "ENABLE_BROWSABLE_API": { 2021-02-10T11:04:02.830 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "ENABLE_BROWSABLE_API", 2021-02-10T11:04:02.830 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "bool", 2021-02-10T11:04:02.830 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.830 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.830 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "True", 2021-02-10T11:04:02.830 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.830 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.830 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.830 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.831 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.831 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.831 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.831 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.831 INFO:teuthology.orchestra.run.gibba002.stdout: "FEATURE_TOGGLE_cephfs": { 2021-02-10T11:04:02.831 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "FEATURE_TOGGLE_cephfs", 2021-02-10T11:04:02.831 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "bool", 2021-02-10T11:04:02.831 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.831 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.831 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "True", 2021-02-10T11:04:02.831 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.831 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.831 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.831 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.832 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.832 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.832 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.832 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.832 INFO:teuthology.orchestra.run.gibba002.stdout: "FEATURE_TOGGLE_iscsi": { 2021-02-10T11:04:02.832 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "FEATURE_TOGGLE_iscsi", 2021-02-10T11:04:02.832 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "bool", 2021-02-10T11:04:02.832 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.832 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.832 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "True", 2021-02-10T11:04:02.832 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.832 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.832 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.832 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.833 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.833 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.833 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.833 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.833 INFO:teuthology.orchestra.run.gibba002.stdout: "FEATURE_TOGGLE_mirroring": { 2021-02-10T11:04:02.833 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "FEATURE_TOGGLE_mirroring", 2021-02-10T11:04:02.833 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "bool", 2021-02-10T11:04:02.833 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.833 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.833 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "True", 2021-02-10T11:04:02.833 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.833 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.833 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.833 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.834 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.834 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.834 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.834 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.834 INFO:teuthology.orchestra.run.gibba002.stdout: "FEATURE_TOGGLE_rbd": { 2021-02-10T11:04:02.834 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "FEATURE_TOGGLE_rbd", 2021-02-10T11:04:02.834 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "bool", 2021-02-10T11:04:02.834 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.834 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.834 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "True", 2021-02-10T11:04:02.834 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.834 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.834 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.834 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.835 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.835 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.835 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.835 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.835 INFO:teuthology.orchestra.run.gibba002.stdout: "FEATURE_TOGGLE_rgw": { 2021-02-10T11:04:02.835 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "FEATURE_TOGGLE_rgw", 2021-02-10T11:04:02.835 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "bool", 2021-02-10T11:04:02.835 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.835 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.835 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "True", 2021-02-10T11:04:02.835 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.835 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.835 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.835 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.836 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.836 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.836 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.836 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.836 INFO:teuthology.orchestra.run.gibba002.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2021-02-10T11:04:02.836 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2021-02-10T11:04:02.836 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.836 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.836 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.836 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "", 2021-02-10T11:04:02.836 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.836 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.836 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.837 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.837 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.837 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.837 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.837 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.837 INFO:teuthology.orchestra.run.gibba002.stdout: "GRAFANA_API_PASSWORD": { 2021-02-10T11:04:02.837 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "GRAFANA_API_PASSWORD", 2021-02-10T11:04:02.837 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.837 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.837 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.837 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "admin", 2021-02-10T11:04:02.837 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.837 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.837 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.838 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.838 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.838 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.838 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.838 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.838 INFO:teuthology.orchestra.run.gibba002.stdout: "GRAFANA_API_SSL_VERIFY": { 2021-02-10T11:04:02.838 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2021-02-10T11:04:02.838 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "bool", 2021-02-10T11:04:02.838 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.838 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.838 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "True", 2021-02-10T11:04:02.838 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.838 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.838 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.839 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.839 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.839 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.839 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.839 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.839 INFO:teuthology.orchestra.run.gibba002.stdout: "GRAFANA_API_URL": { 2021-02-10T11:04:02.839 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "GRAFANA_API_URL", 2021-02-10T11:04:02.839 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.839 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.839 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.839 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "", 2021-02-10T11:04:02.839 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.839 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.839 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.840 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.840 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.840 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.840 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.840 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.840 INFO:teuthology.orchestra.run.gibba002.stdout: "GRAFANA_API_USERNAME": { 2021-02-10T11:04:02.840 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "GRAFANA_API_USERNAME", 2021-02-10T11:04:02.840 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.840 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.840 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.840 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "admin", 2021-02-10T11:04:02.840 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.840 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.840 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.841 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.841 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.841 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.841 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.841 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.841 INFO:teuthology.orchestra.run.gibba002.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2021-02-10T11:04:02.841 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2021-02-10T11:04:02.841 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "bool", 2021-02-10T11:04:02.841 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.841 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.841 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "False", 2021-02-10T11:04:02.841 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.841 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.842 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.842 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.842 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.842 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.842 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.842 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.842 INFO:teuthology.orchestra.run.gibba002.stdout: "ISCSI_API_SSL_VERIFICATION": { 2021-02-10T11:04:02.842 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2021-02-10T11:04:02.842 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "bool", 2021-02-10T11:04:02.842 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.842 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.842 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "True", 2021-02-10T11:04:02.842 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.842 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.843 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.843 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.843 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.843 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.843 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.843 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.843 INFO:teuthology.orchestra.run.gibba002.stdout: "PROMETHEUS_API_HOST": { 2021-02-10T11:04:02.843 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "PROMETHEUS_API_HOST", 2021-02-10T11:04:02.843 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.843 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.843 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.843 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "", 2021-02-10T11:04:02.843 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.843 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.844 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.844 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.844 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.844 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.844 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.844 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.844 INFO:teuthology.orchestra.run.gibba002.stdout: "REST_REQUESTS_TIMEOUT": { 2021-02-10T11:04:02.844 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "REST_REQUESTS_TIMEOUT", 2021-02-10T11:04:02.844 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "int", 2021-02-10T11:04:02.844 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.844 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.844 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "45", 2021-02-10T11:04:02.844 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.844 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.845 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.845 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.845 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.845 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.845 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.845 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.845 INFO:teuthology.orchestra.run.gibba002.stdout: "RGW_API_ACCESS_KEY": { 2021-02-10T11:04:02.845 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "RGW_API_ACCESS_KEY", 2021-02-10T11:04:02.845 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.845 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.845 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.845 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "", 2021-02-10T11:04:02.845 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.845 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.846 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.846 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.846 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.846 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.846 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.846 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.846 INFO:teuthology.orchestra.run.gibba002.stdout: "RGW_API_ADMIN_RESOURCE": { 2021-02-10T11:04:02.846 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2021-02-10T11:04:02.846 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.846 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.846 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.846 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "admin", 2021-02-10T11:04:02.846 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.846 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.847 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.847 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.847 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.847 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.847 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.847 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.847 INFO:teuthology.orchestra.run.gibba002.stdout: "RGW_API_HOST": { 2021-02-10T11:04:02.847 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "RGW_API_HOST", 2021-02-10T11:04:02.847 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.847 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.847 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.847 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "", 2021-02-10T11:04:02.847 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.847 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.848 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.848 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.848 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.848 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.848 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.848 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.848 INFO:teuthology.orchestra.run.gibba002.stdout: "RGW_API_PORT": { 2021-02-10T11:04:02.848 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "RGW_API_PORT", 2021-02-10T11:04:02.848 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "int", 2021-02-10T11:04:02.848 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.848 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.848 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "80", 2021-02-10T11:04:02.848 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.848 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.849 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.849 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.849 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.849 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.849 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.849 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.849 INFO:teuthology.orchestra.run.gibba002.stdout: "RGW_API_SCHEME": { 2021-02-10T11:04:02.849 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "RGW_API_SCHEME", 2021-02-10T11:04:02.849 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.849 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.849 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.849 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "http", 2021-02-10T11:04:02.849 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.849 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.850 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.850 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.850 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.850 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.850 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.850 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.850 INFO:teuthology.orchestra.run.gibba002.stdout: "RGW_API_SECRET_KEY": { 2021-02-10T11:04:02.850 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "RGW_API_SECRET_KEY", 2021-02-10T11:04:02.850 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.850 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.850 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.850 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "", 2021-02-10T11:04:02.850 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.850 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.851 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.851 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.851 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.851 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.851 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.851 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.851 INFO:teuthology.orchestra.run.gibba002.stdout: "RGW_API_SSL_VERIFY": { 2021-02-10T11:04:02.851 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "RGW_API_SSL_VERIFY", 2021-02-10T11:04:02.851 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "bool", 2021-02-10T11:04:02.851 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.851 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.851 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "True", 2021-02-10T11:04:02.851 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.852 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.852 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.852 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.852 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.852 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.852 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.852 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.852 INFO:teuthology.orchestra.run.gibba002.stdout: "RGW_API_USER_ID": { 2021-02-10T11:04:02.852 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "RGW_API_USER_ID", 2021-02-10T11:04:02.852 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.852 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.852 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.852 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "", 2021-02-10T11:04:02.853 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.853 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.853 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.853 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.853 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.853 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.853 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.853 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.853 INFO:teuthology.orchestra.run.gibba002.stdout: "crt_file": { 2021-02-10T11:04:02.853 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "crt_file", 2021-02-10T11:04:02.853 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.853 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.853 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.853 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "", 2021-02-10T11:04:02.854 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.854 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.854 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.854 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.854 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.854 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.854 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.854 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.854 INFO:teuthology.orchestra.run.gibba002.stdout: "debug": { 2021-02-10T11:04:02.854 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "debug", 2021-02-10T11:04:02.854 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "bool", 2021-02-10T11:04:02.854 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.854 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.854 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "False", 2021-02-10T11:04:02.855 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.855 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.855 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.855 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "Enable/disable debug options", 2021-02-10T11:04:02.855 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.855 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.855 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.855 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.855 INFO:teuthology.orchestra.run.gibba002.stdout: "jwt_token_ttl": { 2021-02-10T11:04:02.855 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "jwt_token_ttl", 2021-02-10T11:04:02.855 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "int", 2021-02-10T11:04:02.855 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.855 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.856 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "28800", 2021-02-10T11:04:02.856 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.856 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.856 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.856 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.856 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.856 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.856 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.856 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.856 INFO:teuthology.orchestra.run.gibba002.stdout: "key_file": { 2021-02-10T11:04:02.856 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "key_file", 2021-02-10T11:04:02.856 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.856 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.856 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.857 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "", 2021-02-10T11:04:02.857 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.857 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.857 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.857 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.857 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.857 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.857 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.857 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.857 INFO:teuthology.orchestra.run.gibba002.stdout: "password": { 2021-02-10T11:04:02.857 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "password", 2021-02-10T11:04:02.857 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.857 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.858 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.858 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "", 2021-02-10T11:04:02.858 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.858 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.858 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.858 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.858 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.858 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.858 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.858 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.858 INFO:teuthology.orchestra.run.gibba002.stdout: "server_addr": { 2021-02-10T11:04:02.858 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "server_addr", 2021-02-10T11:04:02.858 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.858 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.859 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.859 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "::", 2021-02-10T11:04:02.859 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.859 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.859 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.859 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.859 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.859 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.859 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.859 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.859 INFO:teuthology.orchestra.run.gibba002.stdout: "server_port": { 2021-02-10T11:04:02.859 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "server_port", 2021-02-10T11:04:02.859 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "int", 2021-02-10T11:04:02.859 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.860 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.860 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "8080", 2021-02-10T11:04:02.860 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.860 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.860 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.860 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.860 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.860 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.860 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.860 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.860 INFO:teuthology.orchestra.run.gibba002.stdout: "ssl": { 2021-02-10T11:04:02.860 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "ssl", 2021-02-10T11:04:02.860 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "bool", 2021-02-10T11:04:02.860 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.861 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.861 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "True", 2021-02-10T11:04:02.861 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.861 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.861 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.861 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.861 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.861 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.861 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.861 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.861 INFO:teuthology.orchestra.run.gibba002.stdout: "ssl_server_port": { 2021-02-10T11:04:02.861 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "ssl_server_port", 2021-02-10T11:04:02.861 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "int", 2021-02-10T11:04:02.861 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.862 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.862 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "8443", 2021-02-10T11:04:02.862 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.862 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.862 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.862 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.862 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.862 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.862 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.862 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.862 INFO:teuthology.orchestra.run.gibba002.stdout: "standby_behaviour": { 2021-02-10T11:04:02.862 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "standby_behaviour", 2021-02-10T11:04:02.862 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.863 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.863 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.863 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "redirect", 2021-02-10T11:04:02.863 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.863 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.863 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [ 2021-02-10T11:04:02.863 INFO:teuthology.orchestra.run.gibba002.stdout: "error", 2021-02-10T11:04:02.863 INFO:teuthology.orchestra.run.gibba002.stdout: "redirect" 2021-02-10T11:04:02.863 INFO:teuthology.orchestra.run.gibba002.stdout: ], 2021-02-10T11:04:02.863 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.863 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.863 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.863 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.863 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.864 INFO:teuthology.orchestra.run.gibba002.stdout: "standby_error_status_code": { 2021-02-10T11:04:02.864 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "standby_error_status_code", 2021-02-10T11:04:02.864 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "int", 2021-02-10T11:04:02.864 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.864 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.864 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "500", 2021-02-10T11:04:02.864 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "400", 2021-02-10T11:04:02.864 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "599", 2021-02-10T11:04:02.864 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.864 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.864 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.864 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.864 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.864 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.865 INFO:teuthology.orchestra.run.gibba002.stdout: "url_prefix": { 2021-02-10T11:04:02.865 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "url_prefix", 2021-02-10T11:04:02.865 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.865 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.865 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.865 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "", 2021-02-10T11:04:02.865 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.865 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.865 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.865 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.865 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.865 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.865 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.865 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.866 INFO:teuthology.orchestra.run.gibba002.stdout: "username": { 2021-02-10T11:04:02.866 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "username", 2021-02-10T11:04:02.866 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.866 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.866 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.866 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "", 2021-02-10T11:04:02.866 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.866 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.866 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.866 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.866 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.866 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.866 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.867 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:04:02.867 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:04:02.867 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.867 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:04:02.867 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "deepsea", 2021-02-10T11:04:02.867 INFO:teuthology.orchestra.run.gibba002.stdout: "can_run": true, 2021-02-10T11:04:02.867 INFO:teuthology.orchestra.run.gibba002.stdout: "error_string": "", 2021-02-10T11:04:02.867 INFO:teuthology.orchestra.run.gibba002.stdout: "module_options": { 2021-02-10T11:04:02.867 INFO:teuthology.orchestra.run.gibba002.stdout: "salt_api_eauth": { 2021-02-10T11:04:02.867 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "salt_api_eauth", 2021-02-10T11:04:02.867 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.867 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.867 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.867 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "sharedsecret", 2021-02-10T11:04:02.868 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.868 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.868 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.868 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.868 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.868 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.868 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.868 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.868 INFO:teuthology.orchestra.run.gibba002.stdout: "salt_api_password": { 2021-02-10T11:04:02.868 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "salt_api_password", 2021-02-10T11:04:02.868 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.868 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.868 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.868 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "", 2021-02-10T11:04:02.869 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.869 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.869 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.869 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.869 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.869 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.869 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.869 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.869 INFO:teuthology.orchestra.run.gibba002.stdout: "salt_api_url": { 2021-02-10T11:04:02.869 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "salt_api_url", 2021-02-10T11:04:02.869 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.869 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.869 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.870 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "", 2021-02-10T11:04:02.870 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.870 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.870 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.870 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.870 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.870 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.870 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.870 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.870 INFO:teuthology.orchestra.run.gibba002.stdout: "salt_api_username": { 2021-02-10T11:04:02.870 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "salt_api_username", 2021-02-10T11:04:02.870 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.870 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.871 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.871 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "", 2021-02-10T11:04:02.871 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.871 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.871 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.871 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.871 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.871 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.871 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.871 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:04:02.871 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:04:02.871 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.871 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:04:02.871 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "devicehealth", 2021-02-10T11:04:02.872 INFO:teuthology.orchestra.run.gibba002.stdout: "can_run": true, 2021-02-10T11:04:02.872 INFO:teuthology.orchestra.run.gibba002.stdout: "error_string": "", 2021-02-10T11:04:02.872 INFO:teuthology.orchestra.run.gibba002.stdout: "module_options": { 2021-02-10T11:04:02.872 INFO:teuthology.orchestra.run.gibba002.stdout: "enable_monitoring": { 2021-02-10T11:04:02.872 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "enable_monitoring", 2021-02-10T11:04:02.872 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "bool", 2021-02-10T11:04:02.872 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.872 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 1, 2021-02-10T11:04:02.872 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "False", 2021-02-10T11:04:02.872 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.872 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.872 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.872 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "monitor device health metrics", 2021-02-10T11:04:02.873 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.873 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.873 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.873 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.873 INFO:teuthology.orchestra.run.gibba002.stdout: "mark_out_threshold": { 2021-02-10T11:04:02.873 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "mark_out_threshold", 2021-02-10T11:04:02.873 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "secs", 2021-02-10T11:04:02.873 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.873 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 1, 2021-02-10T11:04:02.873 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "2419200", 2021-02-10T11:04:02.873 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.873 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.873 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.873 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "automatically mark OSD if it may fail before this long", 2021-02-10T11:04:02.874 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.874 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.874 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.874 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.874 INFO:teuthology.orchestra.run.gibba002.stdout: "pool_name": { 2021-02-10T11:04:02.874 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "pool_name", 2021-02-10T11:04:02.874 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.874 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.874 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 1, 2021-02-10T11:04:02.874 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "device_health_metrics", 2021-02-10T11:04:02.874 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.874 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.874 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.875 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "name of pool in which to store device health metrics", 2021-02-10T11:04:02.875 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.875 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.875 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.875 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.875 INFO:teuthology.orchestra.run.gibba002.stdout: "retention_period": { 2021-02-10T11:04:02.875 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "retention_period", 2021-02-10T11:04:02.875 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "secs", 2021-02-10T11:04:02.875 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.875 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 1, 2021-02-10T11:04:02.875 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "15552000", 2021-02-10T11:04:02.875 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.875 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.875 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.876 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "how long to retain device health metrics", 2021-02-10T11:04:02.876 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.876 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.876 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.876 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.876 INFO:teuthology.orchestra.run.gibba002.stdout: "scrape_frequency": { 2021-02-10T11:04:02.876 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "scrape_frequency", 2021-02-10T11:04:02.876 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "secs", 2021-02-10T11:04:02.876 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.876 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 1, 2021-02-10T11:04:02.876 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "86400", 2021-02-10T11:04:02.876 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.876 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.876 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.877 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "how frequently to scrape device health metrics", 2021-02-10T11:04:02.877 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.877 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.877 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.877 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.877 INFO:teuthology.orchestra.run.gibba002.stdout: "self_heal": { 2021-02-10T11:04:02.877 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "self_heal", 2021-02-10T11:04:02.877 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "bool", 2021-02-10T11:04:02.877 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.877 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 1, 2021-02-10T11:04:02.877 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "True", 2021-02-10T11:04:02.877 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.877 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.877 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.878 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "preemptively heal cluster around devices that may fail", 2021-02-10T11:04:02.878 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.878 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.878 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.878 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.878 INFO:teuthology.orchestra.run.gibba002.stdout: "sleep_interval": { 2021-02-10T11:04:02.878 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "sleep_interval", 2021-02-10T11:04:02.878 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "secs", 2021-02-10T11:04:02.878 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.878 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 1, 2021-02-10T11:04:02.878 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "600", 2021-02-10T11:04:02.878 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.878 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.878 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.879 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "how frequently to wake up and check device health", 2021-02-10T11:04:02.879 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.879 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.879 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.879 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.879 INFO:teuthology.orchestra.run.gibba002.stdout: "warn_threshold": { 2021-02-10T11:04:02.879 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "warn_threshold", 2021-02-10T11:04:02.879 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "secs", 2021-02-10T11:04:02.879 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.879 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 1, 2021-02-10T11:04:02.879 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "7257600", 2021-02-10T11:04:02.879 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.879 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.879 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.880 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "raise health warning if OSD may fail before this long", 2021-02-10T11:04:02.880 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.880 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.880 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.880 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:04:02.880 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:04:02.880 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.880 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:04:02.880 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "diskprediction_cloud", 2021-02-10T11:04:02.880 INFO:teuthology.orchestra.run.gibba002.stdout: "can_run": true, 2021-02-10T11:04:02.880 INFO:teuthology.orchestra.run.gibba002.stdout: "error_string": "", 2021-02-10T11:04:02.880 INFO:teuthology.orchestra.run.gibba002.stdout: "module_options": { 2021-02-10T11:04:02.880 INFO:teuthology.orchestra.run.gibba002.stdout: "diskprediction_cert_context": { 2021-02-10T11:04:02.880 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "diskprediction_cert_context", 2021-02-10T11:04:02.881 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.881 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.881 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.881 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "", 2021-02-10T11:04:02.881 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.881 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.881 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.881 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.881 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.881 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.881 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.881 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.881 INFO:teuthology.orchestra.run.gibba002.stdout: "diskprediction_default_authority": { 2021-02-10T11:04:02.881 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "diskprediction_default_authority", 2021-02-10T11:04:02.882 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.882 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.882 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.882 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "localhost", 2021-02-10T11:04:02.882 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.882 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.882 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.882 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.882 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.882 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.882 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.882 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.882 INFO:teuthology.orchestra.run.gibba002.stdout: "diskprediction_password": { 2021-02-10T11:04:02.883 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "diskprediction_password", 2021-02-10T11:04:02.883 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.883 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.883 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.883 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "", 2021-02-10T11:04:02.883 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.883 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.883 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.883 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.883 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.883 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.883 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.883 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.883 INFO:teuthology.orchestra.run.gibba002.stdout: "diskprediction_port": { 2021-02-10T11:04:02.884 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "diskprediction_port", 2021-02-10T11:04:02.884 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.884 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.884 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.884 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "31400", 2021-02-10T11:04:02.884 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.884 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.884 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.884 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.884 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.884 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.884 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.884 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.884 INFO:teuthology.orchestra.run.gibba002.stdout: "diskprediction_retrieve_prediction_interval": { 2021-02-10T11:04:02.885 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "diskprediction_retrieve_prediction_interval", 2021-02-10T11:04:02.885 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.885 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.885 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.885 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "43200", 2021-02-10T11:04:02.885 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.885 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.885 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.885 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.885 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.885 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.885 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.885 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.886 INFO:teuthology.orchestra.run.gibba002.stdout: "diskprediction_server": { 2021-02-10T11:04:02.886 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "diskprediction_server", 2021-02-10T11:04:02.886 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.886 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.886 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.886 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "", 2021-02-10T11:04:02.886 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.886 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.886 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.886 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.886 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.886 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.886 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.886 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.887 INFO:teuthology.orchestra.run.gibba002.stdout: "diskprediction_ssl_target_name_override": { 2021-02-10T11:04:02.887 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "diskprediction_ssl_target_name_override", 2021-02-10T11:04:02.887 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.887 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.887 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.887 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "localhost", 2021-02-10T11:04:02.887 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.887 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.887 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.887 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.887 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.887 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.887 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.888 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.888 INFO:teuthology.orchestra.run.gibba002.stdout: "diskprediction_upload_metrics_interval": { 2021-02-10T11:04:02.888 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "diskprediction_upload_metrics_interval", 2021-02-10T11:04:02.888 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.888 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.888 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.888 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "600", 2021-02-10T11:04:02.888 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.888 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.888 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.888 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.888 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.888 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.888 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.889 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.889 INFO:teuthology.orchestra.run.gibba002.stdout: "diskprediction_upload_smart_interval": { 2021-02-10T11:04:02.889 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "diskprediction_upload_smart_interval", 2021-02-10T11:04:02.889 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.889 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.889 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.889 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "43200", 2021-02-10T11:04:02.889 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.889 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.889 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.889 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.889 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.889 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.889 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.890 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.890 INFO:teuthology.orchestra.run.gibba002.stdout: "diskprediction_user": { 2021-02-10T11:04:02.890 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "diskprediction_user", 2021-02-10T11:04:02.890 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.890 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.890 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.890 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "", 2021-02-10T11:04:02.890 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.890 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.890 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.890 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.890 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.890 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.891 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.891 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.891 INFO:teuthology.orchestra.run.gibba002.stdout: "sleep_interval": { 2021-02-10T11:04:02.891 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "sleep_interval", 2021-02-10T11:04:02.891 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.891 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.891 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.891 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "600", 2021-02-10T11:04:02.891 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.891 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.891 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.891 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.891 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.891 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.892 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.892 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:04:02.892 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:04:02.892 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.892 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:04:02.892 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "diskprediction_local", 2021-02-10T11:04:02.892 INFO:teuthology.orchestra.run.gibba002.stdout: "can_run": true, 2021-02-10T11:04:02.892 INFO:teuthology.orchestra.run.gibba002.stdout: "error_string": "", 2021-02-10T11:04:02.892 INFO:teuthology.orchestra.run.gibba002.stdout: "module_options": { 2021-02-10T11:04:02.892 INFO:teuthology.orchestra.run.gibba002.stdout: "predict_interval": { 2021-02-10T11:04:02.892 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "predict_interval", 2021-02-10T11:04:02.892 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.892 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.892 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.893 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "86400", 2021-02-10T11:04:02.893 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.893 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.893 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.893 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.893 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.893 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.893 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.893 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.893 INFO:teuthology.orchestra.run.gibba002.stdout: "sleep_interval": { 2021-02-10T11:04:02.893 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "sleep_interval", 2021-02-10T11:04:02.893 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.893 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.893 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.894 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "600", 2021-02-10T11:04:02.894 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.894 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.894 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.894 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.894 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.894 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.894 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.894 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:04:02.894 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:04:02.894 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.894 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:04:02.894 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "influx", 2021-02-10T11:04:02.895 INFO:teuthology.orchestra.run.gibba002.stdout: "can_run": false, 2021-02-10T11:04:02.895 INFO:teuthology.orchestra.run.gibba002.stdout: "error_string": "influxdb python module not found", 2021-02-10T11:04:02.895 INFO:teuthology.orchestra.run.gibba002.stdout: "module_options": { 2021-02-10T11:04:02.895 INFO:teuthology.orchestra.run.gibba002.stdout: "batch_size": { 2021-02-10T11:04:02.895 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "batch_size", 2021-02-10T11:04:02.895 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.895 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.895 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.895 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "5000", 2021-02-10T11:04:02.895 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.895 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.895 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.895 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.895 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.896 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.896 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.896 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.896 INFO:teuthology.orchestra.run.gibba002.stdout: "database": { 2021-02-10T11:04:02.896 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "database", 2021-02-10T11:04:02.896 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.896 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.896 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.896 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "ceph", 2021-02-10T11:04:02.896 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.896 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.896 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.896 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.897 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.897 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.897 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.897 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.897 INFO:teuthology.orchestra.run.gibba002.stdout: "hostname": { 2021-02-10T11:04:02.897 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "hostname", 2021-02-10T11:04:02.897 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.897 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.897 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.897 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "None", 2021-02-10T11:04:02.897 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.897 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.897 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.897 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.898 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.898 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.898 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.898 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.898 INFO:teuthology.orchestra.run.gibba002.stdout: "interval": { 2021-02-10T11:04:02.898 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "interval", 2021-02-10T11:04:02.898 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.898 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.898 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.898 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "30", 2021-02-10T11:04:02.898 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.898 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.898 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.898 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.899 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.899 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.899 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.899 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.899 INFO:teuthology.orchestra.run.gibba002.stdout: "password": { 2021-02-10T11:04:02.899 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "password", 2021-02-10T11:04:02.899 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.899 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.899 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.899 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "None", 2021-02-10T11:04:02.899 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.899 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.899 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.900 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.900 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.900 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.900 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.900 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.900 INFO:teuthology.orchestra.run.gibba002.stdout: "port": { 2021-02-10T11:04:02.900 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "port", 2021-02-10T11:04:02.900 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.900 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.900 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.900 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "8086", 2021-02-10T11:04:02.900 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.900 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.900 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.901 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.901 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.901 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.901 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.901 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.901 INFO:teuthology.orchestra.run.gibba002.stdout: "ssl": { 2021-02-10T11:04:02.901 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "ssl", 2021-02-10T11:04:02.901 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.901 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.901 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.901 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "false", 2021-02-10T11:04:02.901 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.901 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.902 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.902 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.902 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.902 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.902 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.902 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.902 INFO:teuthology.orchestra.run.gibba002.stdout: "threads": { 2021-02-10T11:04:02.902 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "threads", 2021-02-10T11:04:02.902 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.902 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.902 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.902 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "5", 2021-02-10T11:04:02.902 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.902 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.903 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.903 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.903 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.903 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.903 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.903 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.903 INFO:teuthology.orchestra.run.gibba002.stdout: "username": { 2021-02-10T11:04:02.903 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "username", 2021-02-10T11:04:02.903 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.903 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.903 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.903 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "None", 2021-02-10T11:04:02.903 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.903 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.904 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.904 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.904 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.904 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.904 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.904 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.904 INFO:teuthology.orchestra.run.gibba002.stdout: "verify_ssl": { 2021-02-10T11:04:02.904 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "verify_ssl", 2021-02-10T11:04:02.904 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.904 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.904 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.904 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "true", 2021-02-10T11:04:02.904 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.905 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.905 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.905 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.905 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.905 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.905 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.905 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:04:02.905 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:04:02.905 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.905 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:04:02.905 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "insights", 2021-02-10T11:04:02.905 INFO:teuthology.orchestra.run.gibba002.stdout: "can_run": true, 2021-02-10T11:04:02.905 INFO:teuthology.orchestra.run.gibba002.stdout: "error_string": "", 2021-02-10T11:04:02.905 INFO:teuthology.orchestra.run.gibba002.stdout: "module_options": {} 2021-02-10T11:04:02.906 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.906 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:04:02.906 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "iostat", 2021-02-10T11:04:02.906 INFO:teuthology.orchestra.run.gibba002.stdout: "can_run": true, 2021-02-10T11:04:02.906 INFO:teuthology.orchestra.run.gibba002.stdout: "error_string": "", 2021-02-10T11:04:02.906 INFO:teuthology.orchestra.run.gibba002.stdout: "module_options": {} 2021-02-10T11:04:02.906 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.906 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:04:02.906 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "localpool", 2021-02-10T11:04:02.906 INFO:teuthology.orchestra.run.gibba002.stdout: "can_run": true, 2021-02-10T11:04:02.906 INFO:teuthology.orchestra.run.gibba002.stdout: "error_string": "", 2021-02-10T11:04:02.906 INFO:teuthology.orchestra.run.gibba002.stdout: "module_options": { 2021-02-10T11:04:02.906 INFO:teuthology.orchestra.run.gibba002.stdout: "failure_domain": { 2021-02-10T11:04:02.906 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "failure_domain", 2021-02-10T11:04:02.907 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.907 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.907 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 1, 2021-02-10T11:04:02.907 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "host", 2021-02-10T11:04:02.907 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.907 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.907 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.907 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "failure domain for any created local pool", 2021-02-10T11:04:02.907 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.907 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.907 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.907 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.907 INFO:teuthology.orchestra.run.gibba002.stdout: "min_size": { 2021-02-10T11:04:02.908 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "min_size", 2021-02-10T11:04:02.908 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "int", 2021-02-10T11:04:02.908 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.908 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 1, 2021-02-10T11:04:02.908 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "", 2021-02-10T11:04:02.908 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.908 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.908 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.908 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "default min_size for any created local pool", 2021-02-10T11:04:02.908 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.908 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.908 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.908 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.909 INFO:teuthology.orchestra.run.gibba002.stdout: "num_rep": { 2021-02-10T11:04:02.909 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "num_rep", 2021-02-10T11:04:02.909 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "int", 2021-02-10T11:04:02.909 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.909 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 1, 2021-02-10T11:04:02.909 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "3", 2021-02-10T11:04:02.909 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.909 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.909 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.909 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "default replica count for any created local pool", 2021-02-10T11:04:02.909 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.909 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.909 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.909 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.910 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num": { 2021-02-10T11:04:02.910 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "pg_num", 2021-02-10T11:04:02.910 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "int", 2021-02-10T11:04:02.910 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.910 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 1, 2021-02-10T11:04:02.910 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "128", 2021-02-10T11:04:02.910 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.910 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.910 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.910 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "default pg_num for any created local pool", 2021-02-10T11:04:02.910 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.910 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.910 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.911 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.911 INFO:teuthology.orchestra.run.gibba002.stdout: "prefix": { 2021-02-10T11:04:02.911 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "prefix", 2021-02-10T11:04:02.911 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.911 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.911 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 1, 2021-02-10T11:04:02.911 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "", 2021-02-10T11:04:02.911 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.911 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.911 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.911 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "name prefix for any created local pool", 2021-02-10T11:04:02.911 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.911 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.912 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.912 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.912 INFO:teuthology.orchestra.run.gibba002.stdout: "subtree": { 2021-02-10T11:04:02.912 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "subtree", 2021-02-10T11:04:02.912 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.912 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.912 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 1, 2021-02-10T11:04:02.912 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "rack", 2021-02-10T11:04:02.912 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.912 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.912 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.912 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "CRUSH level for which to create a local pool", 2021-02-10T11:04:02.912 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.913 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.913 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.913 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:04:02.913 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:04:02.913 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.913 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:04:02.913 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "orchestrator_cli", 2021-02-10T11:04:02.913 INFO:teuthology.orchestra.run.gibba002.stdout: "can_run": true, 2021-02-10T11:04:02.913 INFO:teuthology.orchestra.run.gibba002.stdout: "error_string": "", 2021-02-10T11:04:02.913 INFO:teuthology.orchestra.run.gibba002.stdout: "module_options": { 2021-02-10T11:04:02.913 INFO:teuthology.orchestra.run.gibba002.stdout: "orchestrator": { 2021-02-10T11:04:02.913 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "orchestrator", 2021-02-10T11:04:02.913 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.914 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.914 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.914 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "", 2021-02-10T11:04:02.914 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.914 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.914 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.914 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.914 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.914 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.914 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.914 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:04:02.914 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:04:02.914 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.914 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:04:02.915 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "pg_autoscaler", 2021-02-10T11:04:02.915 INFO:teuthology.orchestra.run.gibba002.stdout: "can_run": true, 2021-02-10T11:04:02.915 INFO:teuthology.orchestra.run.gibba002.stdout: "error_string": "", 2021-02-10T11:04:02.915 INFO:teuthology.orchestra.run.gibba002.stdout: "module_options": { 2021-02-10T11:04:02.915 INFO:teuthology.orchestra.run.gibba002.stdout: "sleep_interval": { 2021-02-10T11:04:02.915 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "sleep_interval", 2021-02-10T11:04:02.915 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.915 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.915 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.915 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "60", 2021-02-10T11:04:02.915 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.915 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.915 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.915 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.916 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.916 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.916 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.916 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:04:02.916 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:04:02.916 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.916 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:04:02.916 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "progress", 2021-02-10T11:04:02.916 INFO:teuthology.orchestra.run.gibba002.stdout: "can_run": true, 2021-02-10T11:04:02.916 INFO:teuthology.orchestra.run.gibba002.stdout: "error_string": "", 2021-02-10T11:04:02.916 INFO:teuthology.orchestra.run.gibba002.stdout: "module_options": { 2021-02-10T11:04:02.916 INFO:teuthology.orchestra.run.gibba002.stdout: "enabled": { 2021-02-10T11:04:02.916 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "enabled", 2021-02-10T11:04:02.916 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "bool", 2021-02-10T11:04:02.917 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.917 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.917 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "True", 2021-02-10T11:04:02.917 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.917 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.917 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.917 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.917 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.917 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.917 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.917 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.917 INFO:teuthology.orchestra.run.gibba002.stdout: "max_completed_events": { 2021-02-10T11:04:02.917 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "max_completed_events", 2021-02-10T11:04:02.918 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "int", 2021-02-10T11:04:02.918 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.918 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 1, 2021-02-10T11:04:02.918 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "50", 2021-02-10T11:04:02.918 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.918 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.918 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.918 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "number of past completed events to remember", 2021-02-10T11:04:02.918 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.918 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.918 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.918 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.918 INFO:teuthology.orchestra.run.gibba002.stdout: "persist_interval": { 2021-02-10T11:04:02.918 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "persist_interval", 2021-02-10T11:04:02.919 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "secs", 2021-02-10T11:04:02.919 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.919 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 1, 2021-02-10T11:04:02.919 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "5", 2021-02-10T11:04:02.919 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.919 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.919 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.919 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "how frequently to persist completed events", 2021-02-10T11:04:02.919 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.919 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.919 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.919 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:04:02.919 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:04:02.919 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.920 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:04:02.920 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "prometheus", 2021-02-10T11:04:02.920 INFO:teuthology.orchestra.run.gibba002.stdout: "can_run": true, 2021-02-10T11:04:02.920 INFO:teuthology.orchestra.run.gibba002.stdout: "error_string": "", 2021-02-10T11:04:02.920 INFO:teuthology.orchestra.run.gibba002.stdout: "module_options": { 2021-02-10T11:04:02.920 INFO:teuthology.orchestra.run.gibba002.stdout: "rbd_stats_pools": { 2021-02-10T11:04:02.920 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "rbd_stats_pools", 2021-02-10T11:04:02.920 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.920 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.920 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.920 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "", 2021-02-10T11:04:02.920 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.920 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.920 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.921 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.921 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.921 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.921 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.921 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.921 INFO:teuthology.orchestra.run.gibba002.stdout: "rbd_stats_pools_refresh_interval": { 2021-02-10T11:04:02.921 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "rbd_stats_pools_refresh_interval", 2021-02-10T11:04:02.921 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "int", 2021-02-10T11:04:02.921 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.921 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.921 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "300", 2021-02-10T11:04:02.921 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.921 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.921 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.922 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.922 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.922 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.922 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.922 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.922 INFO:teuthology.orchestra.run.gibba002.stdout: "scrape_interval": { 2021-02-10T11:04:02.922 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "scrape_interval", 2021-02-10T11:04:02.922 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.922 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.922 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.922 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "", 2021-02-10T11:04:02.922 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.922 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.923 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.923 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.923 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.923 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.923 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.923 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.923 INFO:teuthology.orchestra.run.gibba002.stdout: "server_addr": { 2021-02-10T11:04:02.923 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "server_addr", 2021-02-10T11:04:02.923 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.923 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.923 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.923 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "", 2021-02-10T11:04:02.923 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.923 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.924 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.924 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.924 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.924 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.924 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.924 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.924 INFO:teuthology.orchestra.run.gibba002.stdout: "server_port": { 2021-02-10T11:04:02.924 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "server_port", 2021-02-10T11:04:02.924 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.924 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.924 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.924 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "", 2021-02-10T11:04:02.924 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.925 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.925 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.925 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.925 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.925 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.925 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.925 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.925 INFO:teuthology.orchestra.run.gibba002.stdout: "stale_cache_strategy": { 2021-02-10T11:04:02.925 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "stale_cache_strategy", 2021-02-10T11:04:02.925 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.925 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.925 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.925 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "", 2021-02-10T11:04:02.925 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.926 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.926 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.926 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.926 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.926 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.926 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.926 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:04:02.926 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:04:02.926 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.926 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:04:02.926 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "rbd_support", 2021-02-10T11:04:02.926 INFO:teuthology.orchestra.run.gibba002.stdout: "can_run": true, 2021-02-10T11:04:02.926 INFO:teuthology.orchestra.run.gibba002.stdout: "error_string": "", 2021-02-10T11:04:02.926 INFO:teuthology.orchestra.run.gibba002.stdout: "module_options": {} 2021-02-10T11:04:02.927 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.927 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:04:02.927 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "restful", 2021-02-10T11:04:02.927 INFO:teuthology.orchestra.run.gibba002.stdout: "can_run": true, 2021-02-10T11:04:02.927 INFO:teuthology.orchestra.run.gibba002.stdout: "error_string": "", 2021-02-10T11:04:02.927 INFO:teuthology.orchestra.run.gibba002.stdout: "module_options": { 2021-02-10T11:04:02.927 INFO:teuthology.orchestra.run.gibba002.stdout: "key_file": { 2021-02-10T11:04:02.927 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "key_file", 2021-02-10T11:04:02.927 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.927 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.927 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.927 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "", 2021-02-10T11:04:02.927 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.927 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.928 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.928 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.928 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.928 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.928 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.928 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.928 INFO:teuthology.orchestra.run.gibba002.stdout: "server_addr": { 2021-02-10T11:04:02.928 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "server_addr", 2021-02-10T11:04:02.928 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.928 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.928 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.928 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "", 2021-02-10T11:04:02.928 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.929 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.929 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.929 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.929 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.929 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.929 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.929 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.929 INFO:teuthology.orchestra.run.gibba002.stdout: "server_port": { 2021-02-10T11:04:02.929 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "server_port", 2021-02-10T11:04:02.929 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.929 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.929 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.929 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "", 2021-02-10T11:04:02.929 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.930 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.930 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.930 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.930 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.930 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.930 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.930 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:04:02.930 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:04:02.930 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.930 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:04:02.930 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "rook", 2021-02-10T11:04:02.930 INFO:teuthology.orchestra.run.gibba002.stdout: "can_run": true, 2021-02-10T11:04:02.930 INFO:teuthology.orchestra.run.gibba002.stdout: "error_string": "", 2021-02-10T11:04:02.931 INFO:teuthology.orchestra.run.gibba002.stdout: "module_options": {} 2021-02-10T11:04:02.931 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.931 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:04:02.931 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "selftest", 2021-02-10T11:04:02.931 INFO:teuthology.orchestra.run.gibba002.stdout: "can_run": true, 2021-02-10T11:04:02.931 INFO:teuthology.orchestra.run.gibba002.stdout: "error_string": "", 2021-02-10T11:04:02.931 INFO:teuthology.orchestra.run.gibba002.stdout: "module_options": { 2021-02-10T11:04:02.931 INFO:teuthology.orchestra.run.gibba002.stdout: "roption1": { 2021-02-10T11:04:02.931 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "roption1", 2021-02-10T11:04:02.931 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.931 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.931 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.931 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "", 2021-02-10T11:04:02.931 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.932 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.932 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.932 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.932 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.932 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.932 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.932 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.932 INFO:teuthology.orchestra.run.gibba002.stdout: "roption2": { 2021-02-10T11:04:02.932 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "roption2", 2021-02-10T11:04:02.932 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.932 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.932 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.932 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "xyz", 2021-02-10T11:04:02.933 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.933 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.933 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.933 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.933 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.933 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.933 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.933 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.933 INFO:teuthology.orchestra.run.gibba002.stdout: "rwoption1": { 2021-02-10T11:04:02.933 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "rwoption1", 2021-02-10T11:04:02.933 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.933 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.933 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.934 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "", 2021-02-10T11:04:02.934 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.934 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.934 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.934 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.934 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.934 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.934 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.934 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.934 INFO:teuthology.orchestra.run.gibba002.stdout: "rwoption2": { 2021-02-10T11:04:02.934 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "rwoption2", 2021-02-10T11:04:02.934 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "int", 2021-02-10T11:04:02.934 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.934 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.935 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "", 2021-02-10T11:04:02.935 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.935 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.935 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.935 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.935 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.935 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.935 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.935 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.935 INFO:teuthology.orchestra.run.gibba002.stdout: "rwoption3": { 2021-02-10T11:04:02.935 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "rwoption3", 2021-02-10T11:04:02.935 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "float", 2021-02-10T11:04:02.935 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.936 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.936 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "", 2021-02-10T11:04:02.936 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.936 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.936 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.936 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.936 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.936 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.936 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.936 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.936 INFO:teuthology.orchestra.run.gibba002.stdout: "rwoption4": { 2021-02-10T11:04:02.936 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "rwoption4", 2021-02-10T11:04:02.936 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.936 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.937 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.937 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "", 2021-02-10T11:04:02.937 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.937 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.937 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.937 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.937 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.937 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.937 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.937 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.937 INFO:teuthology.orchestra.run.gibba002.stdout: "rwoption5": { 2021-02-10T11:04:02.937 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "rwoption5", 2021-02-10T11:04:02.937 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "bool", 2021-02-10T11:04:02.937 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.937 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.938 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "", 2021-02-10T11:04:02.938 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.938 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.938 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.938 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.938 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.938 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.938 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.938 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.938 INFO:teuthology.orchestra.run.gibba002.stdout: "rwoption6": { 2021-02-10T11:04:02.938 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "rwoption6", 2021-02-10T11:04:02.938 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "bool", 2021-02-10T11:04:02.938 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.939 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.939 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "True", 2021-02-10T11:04:02.939 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.939 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.939 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.939 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.939 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.939 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.939 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.939 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.939 INFO:teuthology.orchestra.run.gibba002.stdout: "testkey": { 2021-02-10T11:04:02.939 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "testkey", 2021-02-10T11:04:02.939 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.939 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.940 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.940 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "", 2021-02-10T11:04:02.940 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.940 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.940 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.940 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.940 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.940 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.940 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.940 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.940 INFO:teuthology.orchestra.run.gibba002.stdout: "testlkey": { 2021-02-10T11:04:02.940 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "testlkey", 2021-02-10T11:04:02.940 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.940 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.941 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.941 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "", 2021-02-10T11:04:02.941 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.941 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.941 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.941 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.941 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.941 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.941 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.941 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.941 INFO:teuthology.orchestra.run.gibba002.stdout: "testnewline": { 2021-02-10T11:04:02.941 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "testnewline", 2021-02-10T11:04:02.941 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.942 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.942 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.942 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "", 2021-02-10T11:04:02.942 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.942 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.942 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.942 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.942 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.942 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.942 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.942 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:04:02.942 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:04:02.942 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.942 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:04:02.943 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "ssh", 2021-02-10T11:04:02.943 INFO:teuthology.orchestra.run.gibba002.stdout: "can_run": true, 2021-02-10T11:04:02.943 INFO:teuthology.orchestra.run.gibba002.stdout: "error_string": "", 2021-02-10T11:04:02.943 INFO:teuthology.orchestra.run.gibba002.stdout: "module_options": { 2021-02-10T11:04:02.943 INFO:teuthology.orchestra.run.gibba002.stdout: "inventory_cache_timeout_min": { 2021-02-10T11:04:02.943 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "inventory_cache_timeout_min", 2021-02-10T11:04:02.943 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.943 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.943 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.943 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "", 2021-02-10T11:04:02.943 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.943 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.943 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.943 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.944 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.944 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.944 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.944 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.944 INFO:teuthology.orchestra.run.gibba002.stdout: "ssh_config_file": { 2021-02-10T11:04:02.944 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "ssh_config_file", 2021-02-10T11:04:02.944 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.944 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.944 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.944 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "", 2021-02-10T11:04:02.944 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.944 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.945 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.945 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.945 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.945 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.945 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.945 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:04:02.945 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:04:02.945 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.945 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:04:02.945 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "status", 2021-02-10T11:04:02.945 INFO:teuthology.orchestra.run.gibba002.stdout: "can_run": true, 2021-02-10T11:04:02.945 INFO:teuthology.orchestra.run.gibba002.stdout: "error_string": "", 2021-02-10T11:04:02.946 INFO:teuthology.orchestra.run.gibba002.stdout: "module_options": {} 2021-02-10T11:04:02.946 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.946 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:04:02.946 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "telegraf", 2021-02-10T11:04:02.946 INFO:teuthology.orchestra.run.gibba002.stdout: "can_run": true, 2021-02-10T11:04:02.946 INFO:teuthology.orchestra.run.gibba002.stdout: "error_string": "", 2021-02-10T11:04:02.946 INFO:teuthology.orchestra.run.gibba002.stdout: "module_options": { 2021-02-10T11:04:02.946 INFO:teuthology.orchestra.run.gibba002.stdout: "address": { 2021-02-10T11:04:02.946 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "address", 2021-02-10T11:04:02.946 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.946 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.946 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.946 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2021-02-10T11:04:02.947 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.947 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.947 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.947 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.947 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.947 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.947 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.947 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.947 INFO:teuthology.orchestra.run.gibba002.stdout: "interval": { 2021-02-10T11:04:02.947 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "interval", 2021-02-10T11:04:02.947 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "secs", 2021-02-10T11:04:02.947 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.947 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.948 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "15", 2021-02-10T11:04:02.948 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.948 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.948 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.948 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.948 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.948 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.948 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.948 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:04:02.948 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:04:02.948 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.948 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:04:02.948 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "telemetry", 2021-02-10T11:04:02.948 INFO:teuthology.orchestra.run.gibba002.stdout: "can_run": true, 2021-02-10T11:04:02.949 INFO:teuthology.orchestra.run.gibba002.stdout: "error_string": "", 2021-02-10T11:04:02.949 INFO:teuthology.orchestra.run.gibba002.stdout: "module_options": { 2021-02-10T11:04:02.949 INFO:teuthology.orchestra.run.gibba002.stdout: "channel_basic": { 2021-02-10T11:04:02.949 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "channel_basic", 2021-02-10T11:04:02.949 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "bool", 2021-02-10T11:04:02.949 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.949 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.949 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "True", 2021-02-10T11:04:02.949 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.949 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.949 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.949 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "Share basic cluster information (size, version)", 2021-02-10T11:04:02.949 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.949 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.950 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.950 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.950 INFO:teuthology.orchestra.run.gibba002.stdout: "channel_crash": { 2021-02-10T11:04:02.950 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "channel_crash", 2021-02-10T11:04:02.950 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "bool", 2021-02-10T11:04:02.950 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.950 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.950 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "True", 2021-02-10T11:04:02.950 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.950 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.950 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.950 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.950 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.951 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.951 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.951 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.951 INFO:teuthology.orchestra.run.gibba002.stdout: "channel_device": { 2021-02-10T11:04:02.951 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "channel_device", 2021-02-10T11:04:02.951 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "bool", 2021-02-10T11:04:02.951 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.951 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.951 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "True", 2021-02-10T11:04:02.951 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.951 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.951 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.951 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.952 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.952 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.952 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.952 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.952 INFO:teuthology.orchestra.run.gibba002.stdout: "channel_ident": { 2021-02-10T11:04:02.952 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "channel_ident", 2021-02-10T11:04:02.952 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "bool", 2021-02-10T11:04:02.952 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.952 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.952 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "False", 2021-02-10T11:04:02.952 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.952 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.952 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.952 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.953 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.953 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.953 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.953 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.953 INFO:teuthology.orchestra.run.gibba002.stdout: "contact": { 2021-02-10T11:04:02.953 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "contact", 2021-02-10T11:04:02.953 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.953 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.953 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.953 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "None", 2021-02-10T11:04:02.953 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.953 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.953 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.953 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.954 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.954 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.954 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.954 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.954 INFO:teuthology.orchestra.run.gibba002.stdout: "description": { 2021-02-10T11:04:02.954 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "description", 2021-02-10T11:04:02.954 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.954 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.954 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.954 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "None", 2021-02-10T11:04:02.954 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.954 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.954 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.954 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.955 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.955 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.955 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.955 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.955 INFO:teuthology.orchestra.run.gibba002.stdout: "device_url": { 2021-02-10T11:04:02.955 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "device_url", 2021-02-10T11:04:02.955 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.955 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.955 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.955 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "https://telemetry.ceph.com/device", 2021-02-10T11:04:02.955 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.955 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.955 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.955 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.956 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.956 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.956 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.956 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.956 INFO:teuthology.orchestra.run.gibba002.stdout: "enabled": { 2021-02-10T11:04:02.956 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "enabled", 2021-02-10T11:04:02.956 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "bool", 2021-02-10T11:04:02.956 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.956 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.956 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "False", 2021-02-10T11:04:02.956 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.956 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.956 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.956 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.957 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.957 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.957 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.957 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.957 INFO:teuthology.orchestra.run.gibba002.stdout: "interval": { 2021-02-10T11:04:02.957 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "interval", 2021-02-10T11:04:02.957 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "int", 2021-02-10T11:04:02.957 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.957 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.957 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "24", 2021-02-10T11:04:02.957 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "8", 2021-02-10T11:04:02.957 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.957 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.957 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.958 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.958 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.958 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.958 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.958 INFO:teuthology.orchestra.run.gibba002.stdout: "last_opt_revision": { 2021-02-10T11:04:02.958 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "last_opt_revision", 2021-02-10T11:04:02.958 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "int", 2021-02-10T11:04:02.958 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.958 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.958 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "1", 2021-02-10T11:04:02.958 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.958 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.958 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.958 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.959 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.959 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.959 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.959 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.959 INFO:teuthology.orchestra.run.gibba002.stdout: "leaderboard": { 2021-02-10T11:04:02.959 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "leaderboard", 2021-02-10T11:04:02.959 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "bool", 2021-02-10T11:04:02.959 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.959 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.959 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "False", 2021-02-10T11:04:02.959 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.959 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.959 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.959 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.960 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.960 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.960 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.960 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.960 INFO:teuthology.orchestra.run.gibba002.stdout: "organization": { 2021-02-10T11:04:02.960 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "organization", 2021-02-10T11:04:02.960 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.960 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.960 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.960 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "None", 2021-02-10T11:04:02.960 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.960 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.960 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.961 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.961 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.961 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.961 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.961 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.961 INFO:teuthology.orchestra.run.gibba002.stdout: "proxy": { 2021-02-10T11:04:02.961 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "proxy", 2021-02-10T11:04:02.961 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.961 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.961 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.961 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "None", 2021-02-10T11:04:02.961 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.961 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.961 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.962 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.962 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.962 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.962 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.962 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.962 INFO:teuthology.orchestra.run.gibba002.stdout: "url": { 2021-02-10T11:04:02.962 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "url", 2021-02-10T11:04:02.962 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.962 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.962 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.962 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "https://telemetry.ceph.com/report", 2021-02-10T11:04:02.962 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.962 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.963 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.963 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.963 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.963 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.963 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.963 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:04:02.963 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:04:02.963 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.963 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:04:02.963 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "test_orchestrator", 2021-02-10T11:04:02.963 INFO:teuthology.orchestra.run.gibba002.stdout: "can_run": true, 2021-02-10T11:04:02.963 INFO:teuthology.orchestra.run.gibba002.stdout: "error_string": "", 2021-02-10T11:04:02.963 INFO:teuthology.orchestra.run.gibba002.stdout: "module_options": {} 2021-02-10T11:04:02.963 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.964 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:04:02.964 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "volumes", 2021-02-10T11:04:02.964 INFO:teuthology.orchestra.run.gibba002.stdout: "can_run": true, 2021-02-10T11:04:02.964 INFO:teuthology.orchestra.run.gibba002.stdout: "error_string": "", 2021-02-10T11:04:02.964 INFO:teuthology.orchestra.run.gibba002.stdout: "module_options": { 2021-02-10T11:04:02.964 INFO:teuthology.orchestra.run.gibba002.stdout: "max_concurrent_clones": { 2021-02-10T11:04:02.964 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "max_concurrent_clones", 2021-02-10T11:04:02.964 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "int", 2021-02-10T11:04:02.964 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.964 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.964 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "4", 2021-02-10T11:04:02.964 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.964 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.964 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.965 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "Number of asynchronous cloner threads", 2021-02-10T11:04:02.965 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.965 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.965 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.965 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:04:02.965 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:04:02.965 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.965 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:04:02.965 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "zabbix", 2021-02-10T11:04:02.965 INFO:teuthology.orchestra.run.gibba002.stdout: "can_run": true, 2021-02-10T11:04:02.965 INFO:teuthology.orchestra.run.gibba002.stdout: "error_string": "", 2021-02-10T11:04:02.965 INFO:teuthology.orchestra.run.gibba002.stdout: "module_options": { 2021-02-10T11:04:02.965 INFO:teuthology.orchestra.run.gibba002.stdout: "identifier": { 2021-02-10T11:04:02.965 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "identifier", 2021-02-10T11:04:02.966 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.966 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.966 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.966 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "", 2021-02-10T11:04:02.966 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.966 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.966 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.966 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.966 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.966 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.966 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.966 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.966 INFO:teuthology.orchestra.run.gibba002.stdout: "interval": { 2021-02-10T11:04:02.967 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "interval", 2021-02-10T11:04:02.967 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "secs", 2021-02-10T11:04:02.967 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.967 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.967 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "60", 2021-02-10T11:04:02.967 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.967 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.967 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.967 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.967 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.967 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.967 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.967 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.967 INFO:teuthology.orchestra.run.gibba002.stdout: "zabbix_host": { 2021-02-10T11:04:02.967 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "zabbix_host", 2021-02-10T11:04:02.968 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.968 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.968 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.968 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "None", 2021-02-10T11:04:02.968 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.968 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.968 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.968 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.968 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.968 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.968 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.968 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.968 INFO:teuthology.orchestra.run.gibba002.stdout: "zabbix_port": { 2021-02-10T11:04:02.968 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "zabbix_port", 2021-02-10T11:04:02.969 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "int", 2021-02-10T11:04:02.969 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.969 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.969 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "10051", 2021-02-10T11:04:02.969 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.969 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.969 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.969 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.969 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.969 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.969 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.969 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.969 INFO:teuthology.orchestra.run.gibba002.stdout: "zabbix_sender": { 2021-02-10T11:04:02.969 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "zabbix_sender", 2021-02-10T11:04:02.970 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "str", 2021-02-10T11:04:02.970 INFO:teuthology.orchestra.run.gibba002.stdout: "level": "advanced", 2021-02-10T11:04:02.970 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 0, 2021-02-10T11:04:02.970 INFO:teuthology.orchestra.run.gibba002.stdout: "default_value": "/usr/bin/zabbix_sender", 2021-02-10T11:04:02.970 INFO:teuthology.orchestra.run.gibba002.stdout: "min": "", 2021-02-10T11:04:02.970 INFO:teuthology.orchestra.run.gibba002.stdout: "max": "", 2021-02-10T11:04:02.970 INFO:teuthology.orchestra.run.gibba002.stdout: "enum_allowed": [], 2021-02-10T11:04:02.970 INFO:teuthology.orchestra.run.gibba002.stdout: "desc": "", 2021-02-10T11:04:02.970 INFO:teuthology.orchestra.run.gibba002.stdout: "long_desc": "", 2021-02-10T11:04:02.970 INFO:teuthology.orchestra.run.gibba002.stdout: "tags": [], 2021-02-10T11:04:02.970 INFO:teuthology.orchestra.run.gibba002.stdout: "see_also": [] 2021-02-10T11:04:02.970 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:04:02.970 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:04:02.970 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:04:02.971 INFO:teuthology.orchestra.run.gibba002.stdout: ], 2021-02-10T11:04:02.971 INFO:teuthology.orchestra.run.gibba002.stdout: "services": {}, 2021-02-10T11:04:02.971 INFO:teuthology.orchestra.run.gibba002.stdout: "always_on_modules": { 2021-02-10T11:04:02.971 INFO:teuthology.orchestra.run.gibba002.stdout: "nautilus": [ 2021-02-10T11:04:02.971 INFO:teuthology.orchestra.run.gibba002.stdout: "balancer", 2021-02-10T11:04:02.971 INFO:teuthology.orchestra.run.gibba002.stdout: "crash", 2021-02-10T11:04:02.971 INFO:teuthology.orchestra.run.gibba002.stdout: "devicehealth", 2021-02-10T11:04:02.971 INFO:teuthology.orchestra.run.gibba002.stdout: "orchestrator_cli", 2021-02-10T11:04:02.971 INFO:teuthology.orchestra.run.gibba002.stdout: "progress", 2021-02-10T11:04:02.971 INFO:teuthology.orchestra.run.gibba002.stdout: "rbd_support", 2021-02-10T11:04:02.971 INFO:teuthology.orchestra.run.gibba002.stdout: "status", 2021-02-10T11:04:02.971 INFO:teuthology.orchestra.run.gibba002.stdout: "volumes" 2021-02-10T11:04:02.971 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:04:02.971 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:04:02.972 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.972 INFO:teuthology.orchestra.run.gibba002.stdout: "servicemap": { 2021-02-10T11:04:02.972 INFO:teuthology.orchestra.run.gibba002.stdout: "epoch": 2, 2021-02-10T11:04:02.972 INFO:teuthology.orchestra.run.gibba002.stdout: "modified": "2021-02-10 11:03:57.678215", 2021-02-10T11:04:02.972 INFO:teuthology.orchestra.run.gibba002.stdout: "services": { 2021-02-10T11:04:02.972 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": { 2021-02-10T11:04:02.972 INFO:teuthology.orchestra.run.gibba002.stdout: "daemons": { 2021-02-10T11:04:02.972 INFO:teuthology.orchestra.run.gibba002.stdout: "summary": "", 2021-02-10T11:04:02.972 INFO:teuthology.orchestra.run.gibba002.stdout: "b": { 2021-02-10T11:04:02.972 INFO:teuthology.orchestra.run.gibba002.stdout: "start_epoch": 0, 2021-02-10T11:04:02.972 INFO:teuthology.orchestra.run.gibba002.stdout: "start_stamp": "0.000000", 2021-02-10T11:04:02.972 INFO:teuthology.orchestra.run.gibba002.stdout: "gid": 0, 2021-02-10T11:04:02.972 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": ":/0", 2021-02-10T11:04:02.972 INFO:teuthology.orchestra.run.gibba002.stdout: "metadata": {}, 2021-02-10T11:04:02.973 INFO:teuthology.orchestra.run.gibba002.stdout: "task_status": { 2021-02-10T11:04:02.973 INFO:teuthology.orchestra.run.gibba002.stdout: "scrub status": "idle" 2021-02-10T11:04:02.973 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:04:02.973 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:04:02.973 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:04:02.973 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:04:02.973 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:04:02.973 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:02.973 INFO:teuthology.orchestra.run.gibba002.stdout: "progress_events": {} 2021-02-10T11:04:02.973 INFO:teuthology.orchestra.run.gibba002.stdout:} 2021-02-10T11:04:02.973 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:02.607 7f69a73ad700 1 -- 172.21.2.102:0/2906447467 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f69800186c0 msgr2=0x7f698001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:02.973 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:02.607 7f69a73ad700 1 --2- 172.21.2.102:0/2906447467 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f69800186c0 0x7f698001abc0 crc :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:02.973 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:02.607 7f69a73ad700 1 -- 172.21.2.102:0/2906447467 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f69a01885a0 msgr2=0x7f69a01889a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:02.974 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:02.607 7f69a73ad700 1 --2- 172.21.2.102:0/2906447467 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f69a01885a0 0x7f69a01889a0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7f699c00b580 tx=0x7f699c00b920).stop 2021-02-10T11:04:02.974 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:02.608 7f69a73ad700 1 -- 172.21.2.102:0/2906447467 shutdown_connections 2021-02-10T11:04:02.974 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:02.608 7f69a73ad700 1 --2- 172.21.2.102:0/2906447467 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f69a00bf300 0x7f69a00bf700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:02.974 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:02.608 7f69a73ad700 1 --2- 172.21.2.102:0/2906447467 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f69800186c0 0x7f698001abc0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:02.974 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:02.608 7f69a73ad700 1 --2- 172.21.2.102:0/2906447467 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f69a0188fc0 0x7f69a01893c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:02.974 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:02.608 7f69a73ad700 1 --2- 172.21.2.102:0/2906447467 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f69a01885a0 0x7f69a01889a0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:02.974 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:02.608 7f69a73ad700 1 -- 172.21.2.102:0/2906447467 >> 172.21.2.102:0/2906447467 conn(0x7f69a00a1740 msgr2=0x7f69a00c3c00 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:04:02.974 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:02.608 7f69a73ad700 1 -- 172.21.2.102:0/2906447467 shutdown_connections 2021-02-10T11:04:02.974 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:02.608 7f69a73ad700 1 -- 172.21.2.102:0/2906447467 wait complete. 2021-02-10T11:04:02.976 INFO:tasks.ceph.ceph_manager.ceph:{'fsid': '68db30ea-dbce-4aef-a6df-f01c6fea0513', 'health': {'checks': {}, 'status': 'HEALTH_OK'}, 'election_epoch': 4, 'quorum': [0, 1, 2], 'quorum_names': ['a', 'b', 'c'], 'quorum_age': 56, 'monmap': {'epoch': 1, 'min_mon_release_name': '14', 'num_mons': 3}, 'osdmap': {'osdmap': {'epoch': 20, 'num_osds': 8, 'num_up_osds': 8, 'num_in_osds': 8, 'num_remapped_pgs': 0}}, 'pgmap': {'pgs_by_state': [{'state_name': 'active+clean', 'count': 24}], 'num_pgs': 24, 'num_pools': 3, 'num_objects': 22, 'data_bytes': 2286, 'bytes_used': 8591704064, 'bytes_avail': 764502409216, 'bytes_total': 773094113280, 'write_bytes_sec': 1279, 'read_op_per_sec': 0, 'write_op_per_sec': 4}, 'fsmap': {'epoch': 6, 'id': 1, 'up': 1, 'in': 1, 'max': 1, 'by_rank': [{'filesystem_id': 1, 'rank': 0, 'name': 'b', 'status': 'up:active', 'gid': 4226}], 'up:standby': 1}, 'mgrmap': {'epoch': 3, 'active_gid': 4101, 'active_name': 'y', 'active_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.102:6800', 'nonce': 35161}, {'type': 'v1', 'addr': '172.21.2.102:6801', 'nonce': 35161}]}, 'active_addr': '172.21.2.102:6801/35161', 'active_change': '2021-02-10 11:03:07.666576', 'available': True, 'standbys': [{'gid': 4105, 'name': 'x', 'available_modules': [{'name': 'alerts', 'can_run': True, 'error_string': '', 'module_options': {'interval': {'name': 'interval', 'type': 'secs', 'level': 'advanced', 'flags': 1, 'default_value': '60', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'How frequently to reexamine health status', 'long_desc': '', 'tags': [], 'see_also': []}, 'smtp_destination': {'name': 'smtp_destination', 'type': 'str', 'level': 'advanced', 'flags': 1, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'Email address to send alerts to', 'long_desc': '', 'tags': [], 'see_also': []}, 'smtp_from_name': {'name': 'smtp_from_name', 'type': 'str', 'level': 'advanced', 'flags': 1, 'default_value': 'Ceph', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'Email From: name', 'long_desc': '', 'tags': [], 'see_also': []}, 'smtp_host': {'name': 'smtp_host', 'type': 'str', 'level': 'advanced', 'flags': 1, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'SMTP server', 'long_desc': '', 'tags': [], 'see_also': []}, 'smtp_password': {'name': 'smtp_password', 'type': 'str', 'level': 'advanced', 'flags': 1, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'Password to authenticate with', 'long_desc': '', 'tags': [], 'see_also': []}, 'smtp_port': {'name': 'smtp_port', 'type': 'int', 'level': 'advanced', 'flags': 1, 'default_value': '465', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'SMTP port', 'long_desc': '', 'tags': [], 'see_also': []}, 'smtp_sender': {'name': 'smtp_sender', 'type': 'str', 'level': 'advanced', 'flags': 1, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'SMTP envelope sender', 'long_desc': '', 'tags': [], 'see_also': []}, 'smtp_ssl': {'name': 'smtp_ssl', 'type': 'bool', 'level': 'advanced', 'flags': 1, 'default_value': 'True', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'Use SSL to connect to SMTP server', 'long_desc': '', 'tags': [], 'see_also': []}, 'smtp_user': {'name': 'smtp_user', 'type': 'str', 'level': 'advanced', 'flags': 1, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'User to authenticate as', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'ansible', 'can_run': True, 'error_string': '', 'module_options': {'password': {'name': 'password', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'server_url': {'name': 'server_url', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'username': {'name': 'username', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'verify_server': {'name': 'verify_server', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'balancer', 'can_run': True, 'error_string': '', 'module_options': {'active': {'name': 'active', 'type': 'bool', 'level': 'advanced', 'flags': 1, 'default_value': 'False', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'automatically balance PGs across cluster', 'long_desc': '', 'tags': [], 'see_also': []}, 'begin_time': {'name': 'begin_time', 'type': 'str', 'level': 'advanced', 'flags': 1, 'default_value': '0000', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'beginning time of day to automatically balance', 'long_desc': 'This is a time of day in the format HHMM.', 'tags': [], 'see_also': []}, 'begin_weekday': {'name': 'begin_weekday', 'type': 'uint', 'level': 'advanced', 'flags': 1, 'default_value': '0', 'min': '0', 'max': '7', 'enum_allowed': [], 'desc': 'Restrict automatic balancing to this day of the week or later', 'long_desc': '0 or 7 = Sunday, 1 = Monday, etc.', 'tags': [], 'see_also': []}, 'crush_compat_max_iterations': {'name': 'crush_compat_max_iterations', 'type': 'uint', 'level': 'advanced', 'flags': 1, 'default_value': '25', 'min': '1', 'max': '250', 'enum_allowed': [], 'desc': 'maximum number of iterations to attempt optimization', 'long_desc': '', 'tags': [], 'see_also': []}, 'crush_compat_metrics': {'name': 'crush_compat_metrics', 'type': 'str', 'level': 'advanced', 'flags': 1, 'default_value': 'pgs,objects,bytes', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'metrics with which to calculate OSD utilization', 'long_desc': 'Value is a list of one or more of "pgs", "objects", or "bytes", and indicates which metrics to use to balance utilization.', 'tags': [], 'see_also': []}, 'crush_compat_step': {'name': 'crush_compat_step', 'type': 'float', 'level': 'advanced', 'flags': 1, 'default_value': '0.5', 'min': '0.001', 'max': '0.999', 'enum_allowed': [], 'desc': 'aggressiveness of optimization', 'long_desc': '.99 is very aggressive, .01 is less aggressive', 'tags': [], 'see_also': []}, 'end_time': {'name': 'end_time', 'type': 'str', 'level': 'advanced', 'flags': 1, 'default_value': '2400', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'ending time of day to automatically balance', 'long_desc': 'This is a time of day in the format HHMM.', 'tags': [], 'see_also': []}, 'end_weekday': {'name': 'end_weekday', 'type': 'uint', 'level': 'advanced', 'flags': 1, 'default_value': '7', 'min': '0', 'max': '7', 'enum_allowed': [], 'desc': 'Restrict automatic balancing to days of the week earlier than this', 'long_desc': '0 or 7 = Sunday, 1 = Monday, etc.', 'tags': [], 'see_also': []}, 'min_score': {'name': 'min_score', 'type': 'float', 'level': 'advanced', 'flags': 1, 'default_value': '0', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'minimum score, below which no optimization is attempted', 'long_desc': '', 'tags': [], 'see_also': []}, 'mode': {'name': 'mode', 'type': 'str', 'level': 'advanced', 'flags': 1, 'default_value': 'none', 'min': '', 'max': '', 'enum_allowed': ['crush-compat', 'none', 'upmap'], 'desc': 'Balancer mode', 'long_desc': '', 'tags': [], 'see_also': []}, 'pool_ids': {'name': 'pool_ids', 'type': 'str', 'level': 'advanced', 'flags': 1, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'pools which the automatic balancing will be limited to', 'long_desc': '', 'tags': [], 'see_also': []}, 'sleep_interval': {'name': 'sleep_interval', 'type': 'secs', 'level': 'advanced', 'flags': 1, 'default_value': '60', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'how frequently to wake up and attempt optimization', 'long_desc': '', 'tags': [], 'see_also': []}, 'upmap_max_deviation': {'name': 'upmap_max_deviation', 'type': 'int', 'level': 'advanced', 'flags': 1, 'default_value': '5', 'min': '1', 'max': '', 'enum_allowed': [], 'desc': 'deviation below which no optimization is attempted', 'long_desc': 'If the number of PGs are within this count then no optimization is attempted', 'tags': [], 'see_also': []}, 'upmap_max_iterations': {'name': 'upmap_max_iterations', 'type': 'uint', 'level': 'advanced', 'flags': 1, 'default_value': '10', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'maximum upmap optimization iterations', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'crash', 'can_run': True, 'error_string': '', 'module_options': {'retain_interval': {'name': 'retain_interval', 'type': 'secs', 'level': 'advanced', 'flags': 1, 'default_value': '31536000', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'how long to retain crashes before pruning them', 'long_desc': '', 'tags': [], 'see_also': []}, 'warn_recent_interval': {'name': 'warn_recent_interval', 'type': 'secs', 'level': 'advanced', 'flags': 1, 'default_value': '1209600', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'time interval in which to warn about recent crashes', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'dashboard', 'can_run': True, 'error_string': '', 'module_options': {'ALERTMANAGER_API_HOST': {'name': 'ALERTMANAGER_API_HOST', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'AUDIT_API_ENABLED': {'name': 'AUDIT_API_ENABLED', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'False', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'AUDIT_API_LOG_PAYLOAD': {'name': 'AUDIT_API_LOG_PAYLOAD', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'True', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'ENABLE_BROWSABLE_API': {'name': 'ENABLE_BROWSABLE_API', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'True', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'FEATURE_TOGGLE_cephfs': {'name': 'FEATURE_TOGGLE_cephfs', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'True', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'FEATURE_TOGGLE_iscsi': {'name': 'FEATURE_TOGGLE_iscsi', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'True', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'FEATURE_TOGGLE_mirroring': {'name': 'FEATURE_TOGGLE_mirroring', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'True', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'FEATURE_TOGGLE_rbd': {'name': 'FEATURE_TOGGLE_rbd', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'True', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'FEATURE_TOGGLE_rgw': {'name': 'FEATURE_TOGGLE_rgw', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'True', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE': {'name': 'GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'GRAFANA_API_PASSWORD': {'name': 'GRAFANA_API_PASSWORD', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'admin', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'GRAFANA_API_SSL_VERIFY': {'name': 'GRAFANA_API_SSL_VERIFY', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'True', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'GRAFANA_API_URL': {'name': 'GRAFANA_API_URL', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'GRAFANA_API_USERNAME': {'name': 'GRAFANA_API_USERNAME', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'admin', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'GRAFANA_UPDATE_DASHBOARDS': {'name': 'GRAFANA_UPDATE_DASHBOARDS', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'False', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'ISCSI_API_SSL_VERIFICATION': {'name': 'ISCSI_API_SSL_VERIFICATION', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'True', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'PROMETHEUS_API_HOST': {'name': 'PROMETHEUS_API_HOST', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'REST_REQUESTS_TIMEOUT': {'name': 'REST_REQUESTS_TIMEOUT', 'type': 'int', 'level': 'advanced', 'flags': 0, 'default_value': '45', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'RGW_API_ACCESS_KEY': {'name': 'RGW_API_ACCESS_KEY', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'RGW_API_ADMIN_RESOURCE': {'name': 'RGW_API_ADMIN_RESOURCE', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'admin', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'RGW_API_HOST': {'name': 'RGW_API_HOST', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'RGW_API_PORT': {'name': 'RGW_API_PORT', 'type': 'int', 'level': 'advanced', 'flags': 0, 'default_value': '80', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'RGW_API_SCHEME': {'name': 'RGW_API_SCHEME', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'http', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'RGW_API_SECRET_KEY': {'name': 'RGW_API_SECRET_KEY', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'RGW_API_SSL_VERIFY': {'name': 'RGW_API_SSL_VERIFY', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'True', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'RGW_API_USER_ID': {'name': 'RGW_API_USER_ID', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'crt_file': {'name': 'crt_file', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'debug': {'name': 'debug', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'False', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'Enable/disable debug options', 'long_desc': '', 'tags': [], 'see_also': []}, 'jwt_token_ttl': {'name': 'jwt_token_ttl', 'type': 'int', 'level': 'advanced', 'flags': 0, 'default_value': '28800', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'key_file': {'name': 'key_file', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'password': {'name': 'password', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'server_addr': {'name': 'server_addr', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '::', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'server_port': {'name': 'server_port', 'type': 'int', 'level': 'advanced', 'flags': 0, 'default_value': '8080', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'ssl': {'name': 'ssl', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'True', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'ssl_server_port': {'name': 'ssl_server_port', 'type': 'int', 'level': 'advanced', 'flags': 0, 'default_value': '8443', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'standby_behaviour': {'name': 'standby_behaviour', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'redirect', 'min': '', 'max': '', 'enum_allowed': ['error', 'redirect'], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'standby_error_status_code': {'name': 'standby_error_status_code', 'type': 'int', 'level': 'advanced', 'flags': 0, 'default_value': '500', 'min': '400', 'max': '599', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'url_prefix': {'name': 'url_prefix', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'username': {'name': 'username', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'deepsea', 'can_run': True, 'error_string': '', 'module_options': {'salt_api_eauth': {'name': 'salt_api_eauth', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'sharedsecret', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'salt_api_password': {'name': 'salt_api_password', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'salt_api_url': {'name': 'salt_api_url', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'salt_api_username': {'name': 'salt_api_username', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'devicehealth', 'can_run': True, 'error_string': '', 'module_options': {'enable_monitoring': {'name': 'enable_monitoring', 'type': 'bool', 'level': 'advanced', 'flags': 1, 'default_value': 'False', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'monitor device health metrics', 'long_desc': '', 'tags': [], 'see_also': []}, 'mark_out_threshold': {'name': 'mark_out_threshold', 'type': 'secs', 'level': 'advanced', 'flags': 1, 'default_value': '2419200', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'automatically mark OSD if it may fail before this long', 'long_desc': '', 'tags': [], 'see_also': []}, 'pool_name': {'name': 'pool_name', 'type': 'str', 'level': 'advanced', 'flags': 1, 'default_value': 'device_health_metrics', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'name of pool in which to store device health metrics', 'long_desc': '', 'tags': [], 'see_also': []}, 'retention_period': {'name': 'retention_period', 'type': 'secs', 'level': 'advanced', 'flags': 1, 'default_value': '15552000', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'how long to retain device health metrics', 'long_desc': '', 'tags': [], 'see_also': []}, 'scrape_frequency': {'name': 'scrape_frequency', 'type': 'secs', 'level': 'advanced', 'flags': 1, 'default_value': '86400', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'how frequently to scrape device health metrics', 'long_desc': '', 'tags': [], 'see_also': []}, 'self_heal': {'name': 'self_heal', 'type': 'bool', 'level': 'advanced', 'flags': 1, 'default_value': 'True', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'preemptively heal cluster around devices that may fail', 'long_desc': '', 'tags': [], 'see_also': []}, 'sleep_interval': {'name': 'sleep_interval', 'type': 'secs', 'level': 'advanced', 'flags': 1, 'default_value': '600', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'how frequently to wake up and check device health', 'long_desc': '', 'tags': [], 'see_also': []}, 'warn_threshold': {'name': 'warn_threshold', 'type': 'secs', 'level': 'advanced', 'flags': 1, 'default_value': '7257600', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'raise health warning if OSD may fail before this long', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'diskprediction_cloud', 'can_run': True, 'error_string': '', 'module_options': {'diskprediction_cert_context': {'name': 'diskprediction_cert_context', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'diskprediction_default_authority': {'name': 'diskprediction_default_authority', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'localhost', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'diskprediction_password': {'name': 'diskprediction_password', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'diskprediction_port': {'name': 'diskprediction_port', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '31400', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'diskprediction_retrieve_prediction_interval': {'name': 'diskprediction_retrieve_prediction_interval', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '43200', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'diskprediction_server': {'name': 'diskprediction_server', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'diskprediction_ssl_target_name_override': {'name': 'diskprediction_ssl_target_name_override', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'localhost', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'diskprediction_upload_metrics_interval': {'name': 'diskprediction_upload_metrics_interval', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '600', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'diskprediction_upload_smart_interval': {'name': 'diskprediction_upload_smart_interval', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '43200', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'diskprediction_user': {'name': 'diskprediction_user', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'sleep_interval': {'name': 'sleep_interval', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '600', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'diskprediction_local', 'can_run': True, 'error_string': '', 'module_options': {'predict_interval': {'name': 'predict_interval', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '86400', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'sleep_interval': {'name': 'sleep_interval', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '600', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'influx', 'can_run': False, 'error_string': 'influxdb python module not found', 'module_options': {'batch_size': {'name': 'batch_size', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '5000', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'database': {'name': 'database', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'ceph', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'hostname': {'name': 'hostname', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'None', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'interval': {'name': 'interval', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '30', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'password': {'name': 'password', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'None', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'port': {'name': 'port', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '8086', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'ssl': {'name': 'ssl', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'false', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'threads': {'name': 'threads', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '5', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'username': {'name': 'username', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'None', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'verify_ssl': {'name': 'verify_ssl', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'true', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'insights', 'can_run': True, 'error_string': '', 'module_options': {}}, {'name': 'iostat', 'can_run': True, 'error_string': '', 'module_options': {}}, {'name': 'localpool', 'can_run': True, 'error_string': '', 'module_options': {'failure_domain': {'name': 'failure_domain', 'type': 'str', 'level': 'advanced', 'flags': 1, 'default_value': 'host', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'failure domain for any created local pool', 'long_desc': '', 'tags': [], 'see_also': []}, 'min_size': {'name': 'min_size', 'type': 'int', 'level': 'advanced', 'flags': 1, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'default min_size for any created local pool', 'long_desc': '', 'tags': [], 'see_also': []}, 'num_rep': {'name': 'num_rep', 'type': 'int', 'level': 'advanced', 'flags': 1, 'default_value': '3', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'default replica count for any created local pool', 'long_desc': '', 'tags': [], 'see_also': []}, 'pg_num': {'name': 'pg_num', 'type': 'int', 'level': 'advanced', 'flags': 1, 'default_value': '128', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'default pg_num for any created local pool', 'long_desc': '', 'tags': [], 'see_also': []}, 'prefix': {'name': 'prefix', 'type': 'str', 'level': 'advanced', 'flags': 1, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'name prefix for any created local pool', 'long_desc': '', 'tags': [], 'see_also': []}, 'subtree': {'name': 'subtree', 'type': 'str', 'level': 'advanced', 'flags': 1, 'default_value': 'rack', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'CRUSH level for which to create a local pool', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'orchestrator_cli', 'can_run': True, 'error_string': '', 'module_options': {'orchestrator': {'name': 'orchestrator', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'pg_autoscaler', 'can_run': True, 'error_string': '', 'module_options': {'sleep_interval': {'name': 'sleep_interval', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '60', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'progress', 'can_run': True, 'error_string': '', 'module_options': {'enabled': {'name': 'enabled', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'True', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'max_completed_events': {'name': 'max_completed_events', 'type': 'int', 'level': 'advanced', 'flags': 1, 'default_value': '50', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'number of past completed events to remember', 'long_desc': '', 'tags': [], 'see_also': []}, 'persist_interval': {'name': 'persist_interval', 'type': 'secs', 'level': 'advanced', 'flags': 1, 'default_value': '5', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'how frequently to persist completed events', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'prometheus', 'can_run': True, 'error_string': '', 'module_options': {'rbd_stats_pools': {'name': 'rbd_stats_pools', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'rbd_stats_pools_refresh_interval': {'name': 'rbd_stats_pools_refresh_interval', 'type': 'int', 'level': 'advanced', 'flags': 0, 'default_value': '300', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'scrape_interval': {'name': 'scrape_interval', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'server_addr': {'name': 'server_addr', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'server_port': {'name': 'server_port', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'stale_cache_strategy': {'name': 'stale_cache_strategy', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'rbd_support', 'can_run': True, 'error_string': '', 'module_options': {}}, {'name': 'restful', 'can_run': True, 'error_string': '', 'module_options': {'key_file': {'name': 'key_file', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'server_addr': {'name': 'server_addr', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'server_port': {'name': 'server_port', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'rook', 'can_run': True, 'error_string': '', 'module_options': {}}, {'name': 'selftest', 'can_run': True, 'error_string': '', 'module_options': {'roption1': {'name': 'roption1', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'roption2': {'name': 'roption2', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'xyz', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'rwoption1': {'name': 'rwoption1', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'rwoption2': {'name': 'rwoption2', 'type': 'int', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'rwoption3': {'name': 'rwoption3', 'type': 'float', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'rwoption4': {'name': 'rwoption4', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'rwoption5': {'name': 'rwoption5', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'rwoption6': {'name': 'rwoption6', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'True', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'testkey': {'name': 'testkey', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'testlkey': {'name': 'testlkey', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'testnewline': {'name': 'testnewline', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'ssh', 'can_run': True, 'error_string': '', 'module_options': {'inventory_cache_timeout_min': {'name': 'inventory_cache_timeout_min', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'ssh_config_file': {'name': 'ssh_config_file', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'status', 'can_run': True, 'error_string': '', 'module_options': {}}, {'name': 'telegraf', 'can_run': True, 'error_string': '', 'module_options': {'address': {'name': 'address', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'unixgram:///tmp/telegraf.sock', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'interval': {'name': 'interval', 'type': 'secs', 'level': 'advanced', 'flags': 0, 'default_value': '15', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'telemetry', 'can_run': True, 'error_string': '', 'module_options': {'channel_basic': {'name': 'channel_basic', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'True', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'Share basic cluster information (size, version)', 'long_desc': '', 'tags': [], 'see_also': []}, 'channel_crash': {'name': 'channel_crash', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'True', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'channel_device': {'name': 'channel_device', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'True', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'channel_ident': {'name': 'channel_ident', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'False', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'contact': {'name': 'contact', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'None', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'description': {'name': 'description', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'None', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'device_url': {'name': 'device_url', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'https://telemetry.ceph.com/device', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'enabled': {'name': 'enabled', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'False', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'interval': {'name': 'interval', 'type': 'int', 'level': 'advanced', 'flags': 0, 'default_value': '24', 'min': '8', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'last_opt_revision': {'name': 'last_opt_revision', 'type': 'int', 'level': 'advanced', 'flags': 0, 'default_value': '1', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'leaderboard': {'name': 'leaderboard', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'False', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'organization': {'name': 'organization', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'None', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'proxy': {'name': 'proxy', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'None', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'url': {'name': 'url', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'https://telemetry.ceph.com/report', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'test_orchestrator', 'can_run': True, 'error_string': '', 'module_options': {}}, {'name': 'volumes', 'can_run': True, 'error_string': '', 'module_options': {'max_concurrent_clones': {'name': 'max_concurrent_clones', 'type': 'int', 'level': 'advanced', 'flags': 0, 'default_value': '4', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'Number of asynchronous cloner threads', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'zabbix', 'can_run': True, 'error_string': '', 'module_options': {'identifier': {'name': 'identifier', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'interval': {'name': 'interval', 'type': 'secs', 'level': 'advanced', 'flags': 0, 'default_value': '60', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'zabbix_host': {'name': 'zabbix_host', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'None', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'zabbix_port': {'name': 'zabbix_port', 'type': 'int', 'level': 'advanced', 'flags': 0, 'default_value': '10051', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'zabbix_sender': {'name': 'zabbix_sender', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '/usr/bin/zabbix_sender', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}}}]}], 'modules': ['iostat', 'restful'], 'available_modules': [{'name': 'alerts', 'can_run': True, 'error_string': '', 'module_options': {'interval': {'name': 'interval', 'type': 'secs', 'level': 'advanced', 'flags': 1, 'default_value': '60', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'How frequently to reexamine health status', 'long_desc': '', 'tags': [], 'see_also': []}, 'smtp_destination': {'name': 'smtp_destination', 'type': 'str', 'level': 'advanced', 'flags': 1, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'Email address to send alerts to', 'long_desc': '', 'tags': [], 'see_also': []}, 'smtp_from_name': {'name': 'smtp_from_name', 'type': 'str', 'level': 'advanced', 'flags': 1, 'default_value': 'Ceph', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'Email From: name', 'long_desc': '', 'tags': [], 'see_also': []}, 'smtp_host': {'name': 'smtp_host', 'type': 'str', 'level': 'advanced', 'flags': 1, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'SMTP server', 'long_desc': '', 'tags': [], 'see_also': []}, 'smtp_password': {'name': 'smtp_password', 'type': 'str', 'level': 'advanced', 'flags': 1, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'Password to authenticate with', 'long_desc': '', 'tags': [], 'see_also': []}, 'smtp_port': {'name': 'smtp_port', 'type': 'int', 'level': 'advanced', 'flags': 1, 'default_value': '465', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'SMTP port', 'long_desc': '', 'tags': [], 'see_also': []}, 'smtp_sender': {'name': 'smtp_sender', 'type': 'str', 'level': 'advanced', 'flags': 1, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'SMTP envelope sender', 'long_desc': '', 'tags': [], 'see_also': []}, 'smtp_ssl': {'name': 'smtp_ssl', 'type': 'bool', 'level': 'advanced', 'flags': 1, 'default_value': 'True', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'Use SSL to connect to SMTP server', 'long_desc': '', 'tags': [], 'see_also': []}, 'smtp_user': {'name': 'smtp_user', 'type': 'str', 'level': 'advanced', 'flags': 1, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'User to authenticate as', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'ansible', 'can_run': True, 'error_string': '', 'module_options': {'password': {'name': 'password', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'server_url': {'name': 'server_url', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'username': {'name': 'username', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'verify_server': {'name': 'verify_server', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'balancer', 'can_run': True, 'error_string': '', 'module_options': {'active': {'name': 'active', 'type': 'bool', 'level': 'advanced', 'flags': 1, 'default_value': 'False', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'automatically balance PGs across cluster', 'long_desc': '', 'tags': [], 'see_also': []}, 'begin_time': {'name': 'begin_time', 'type': 'str', 'level': 'advanced', 'flags': 1, 'default_value': '0000', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'beginning time of day to automatically balance', 'long_desc': 'This is a time of day in the format HHMM.', 'tags': [], 'see_also': []}, 'begin_weekday': {'name': 'begin_weekday', 'type': 'uint', 'level': 'advanced', 'flags': 1, 'default_value': '0', 'min': '0', 'max': '7', 'enum_allowed': [], 'desc': 'Restrict automatic balancing to this day of the week or later', 'long_desc': '0 or 7 = Sunday, 1 = Monday, etc.', 'tags': [], 'see_also': []}, 'crush_compat_max_iterations': {'name': 'crush_compat_max_iterations', 'type': 'uint', 'level': 'advanced', 'flags': 1, 'default_value': '25', 'min': '1', 'max': '250', 'enum_allowed': [], 'desc': 'maximum number of iterations to attempt optimization', 'long_desc': '', 'tags': [], 'see_also': []}, 'crush_compat_metrics': {'name': 'crush_compat_metrics', 'type': 'str', 'level': 'advanced', 'flags': 1, 'default_value': 'pgs,objects,bytes', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'metrics with which to calculate OSD utilization', 'long_desc': 'Value is a list of one or more of "pgs", "objects", or "bytes", and indicates which metrics to use to balance utilization.', 'tags': [], 'see_also': []}, 'crush_compat_step': {'name': 'crush_compat_step', 'type': 'float', 'level': 'advanced', 'flags': 1, 'default_value': '0.5', 'min': '0.001', 'max': '0.999', 'enum_allowed': [], 'desc': 'aggressiveness of optimization', 'long_desc': '.99 is very aggressive, .01 is less aggressive', 'tags': [], 'see_also': []}, 'end_time': {'name': 'end_time', 'type': 'str', 'level': 'advanced', 'flags': 1, 'default_value': '2400', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'ending time of day to automatically balance', 'long_desc': 'This is a time of day in the format HHMM.', 'tags': [], 'see_also': []}, 'end_weekday': {'name': 'end_weekday', 'type': 'uint', 'level': 'advanced', 'flags': 1, 'default_value': '7', 'min': '0', 'max': '7', 'enum_allowed': [], 'desc': 'Restrict automatic balancing to days of the week earlier than this', 'long_desc': '0 or 7 = Sunday, 1 = Monday, etc.', 'tags': [], 'see_also': []}, 'min_score': {'name': 'min_score', 'type': 'float', 'level': 'advanced', 'flags': 1, 'default_value': '0', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'minimum score, below which no optimization is attempted', 'long_desc': '', 'tags': [], 'see_also': []}, 'mode': {'name': 'mode', 'type': 'str', 'level': 'advanced', 'flags': 1, 'default_value': 'none', 'min': '', 'max': '', 'enum_allowed': ['crush-compat', 'none', 'upmap'], 'desc': 'Balancer mode', 'long_desc': '', 'tags': [], 'see_also': []}, 'pool_ids': {'name': 'pool_ids', 'type': 'str', 'level': 'advanced', 'flags': 1, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'pools which the automatic balancing will be limited to', 'long_desc': '', 'tags': [], 'see_also': []}, 'sleep_interval': {'name': 'sleep_interval', 'type': 'secs', 'level': 'advanced', 'flags': 1, 'default_value': '60', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'how frequently to wake up and attempt optimization', 'long_desc': '', 'tags': [], 'see_also': []}, 'upmap_max_deviation': {'name': 'upmap_max_deviation', 'type': 'int', 'level': 'advanced', 'flags': 1, 'default_value': '5', 'min': '1', 'max': '', 'enum_allowed': [], 'desc': 'deviation below which no optimization is attempted', 'long_desc': 'If the number of PGs are within this count then no optimization is attempted', 'tags': [], 'see_also': []}, 'upmap_max_iterations': {'name': 'upmap_max_iterations', 'type': 'uint', 'level': 'advanced', 'flags': 1, 'default_value': '10', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'maximum upmap optimization iterations', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'crash', 'can_run': True, 'error_string': '', 'module_options': {'retain_interval': {'name': 'retain_interval', 'type': 'secs', 'level': 'advanced', 'flags': 1, 'default_value': '31536000', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'how long to retain crashes before pruning them', 'long_desc': '', 'tags': [], 'see_also': []}, 'warn_recent_interval': {'name': 'warn_recent_interval', 'type': 'secs', 'level': 'advanced', 'flags': 1, 'default_value': '1209600', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'time interval in which to warn about recent crashes', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'dashboard', 'can_run': True, 'error_string': '', 'module_options': {'ALERTMANAGER_API_HOST': {'name': 'ALERTMANAGER_API_HOST', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'AUDIT_API_ENABLED': {'name': 'AUDIT_API_ENABLED', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'False', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'AUDIT_API_LOG_PAYLOAD': {'name': 'AUDIT_API_LOG_PAYLOAD', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'True', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'ENABLE_BROWSABLE_API': {'name': 'ENABLE_BROWSABLE_API', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'True', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'FEATURE_TOGGLE_cephfs': {'name': 'FEATURE_TOGGLE_cephfs', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'True', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'FEATURE_TOGGLE_iscsi': {'name': 'FEATURE_TOGGLE_iscsi', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'True', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'FEATURE_TOGGLE_mirroring': {'name': 'FEATURE_TOGGLE_mirroring', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'True', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'FEATURE_TOGGLE_rbd': {'name': 'FEATURE_TOGGLE_rbd', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'True', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'FEATURE_TOGGLE_rgw': {'name': 'FEATURE_TOGGLE_rgw', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'True', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE': {'name': 'GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'GRAFANA_API_PASSWORD': {'name': 'GRAFANA_API_PASSWORD', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'admin', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'GRAFANA_API_SSL_VERIFY': {'name': 'GRAFANA_API_SSL_VERIFY', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'True', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'GRAFANA_API_URL': {'name': 'GRAFANA_API_URL', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'GRAFANA_API_USERNAME': {'name': 'GRAFANA_API_USERNAME', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'admin', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'GRAFANA_UPDATE_DASHBOARDS': {'name': 'GRAFANA_UPDATE_DASHBOARDS', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'False', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'ISCSI_API_SSL_VERIFICATION': {'name': 'ISCSI_API_SSL_VERIFICATION', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'True', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'PROMETHEUS_API_HOST': {'name': 'PROMETHEUS_API_HOST', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'REST_REQUESTS_TIMEOUT': {'name': 'REST_REQUESTS_TIMEOUT', 'type': 'int', 'level': 'advanced', 'flags': 0, 'default_value': '45', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'RGW_API_ACCESS_KEY': {'name': 'RGW_API_ACCESS_KEY', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'RGW_API_ADMIN_RESOURCE': {'name': 'RGW_API_ADMIN_RESOURCE', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'admin', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'RGW_API_HOST': {'name': 'RGW_API_HOST', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'RGW_API_PORT': {'name': 'RGW_API_PORT', 'type': 'int', 'level': 'advanced', 'flags': 0, 'default_value': '80', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'RGW_API_SCHEME': {'name': 'RGW_API_SCHEME', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'http', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'RGW_API_SECRET_KEY': {'name': 'RGW_API_SECRET_KEY', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'RGW_API_SSL_VERIFY': {'name': 'RGW_API_SSL_VERIFY', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'True', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'RGW_API_USER_ID': {'name': 'RGW_API_USER_ID', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'crt_file': {'name': 'crt_file', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'debug': {'name': 'debug', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'False', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'Enable/disable debug options', 'long_desc': '', 'tags': [], 'see_also': []}, 'jwt_token_ttl': {'name': 'jwt_token_ttl', 'type': 'int', 'level': 'advanced', 'flags': 0, 'default_value': '28800', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'key_file': {'name': 'key_file', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'password': {'name': 'password', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'server_addr': {'name': 'server_addr', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '::', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'server_port': {'name': 'server_port', 'type': 'int', 'level': 'advanced', 'flags': 0, 'default_value': '8080', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'ssl': {'name': 'ssl', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'True', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'ssl_server_port': {'name': 'ssl_server_port', 'type': 'int', 'level': 'advanced', 'flags': 0, 'default_value': '8443', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'standby_behaviour': {'name': 'standby_behaviour', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'redirect', 'min': '', 'max': '', 'enum_allowed': ['error', 'redirect'], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'standby_error_status_code': {'name': 'standby_error_status_code', 'type': 'int', 'level': 'advanced', 'flags': 0, 'default_value': '500', 'min': '400', 'max': '599', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'url_prefix': {'name': 'url_prefix', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'username': {'name': 'username', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'deepsea', 'can_run': True, 'error_string': '', 'module_options': {'salt_api_eauth': {'name': 'salt_api_eauth', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'sharedsecret', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'salt_api_password': {'name': 'salt_api_password', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'salt_api_url': {'name': 'salt_api_url', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'salt_api_username': {'name': 'salt_api_username', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'devicehealth', 'can_run': True, 'error_string': '', 'module_options': {'enable_monitoring': {'name': 'enable_monitoring', 'type': 'bool', 'level': 'advanced', 'flags': 1, 'default_value': 'False', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'monitor device health metrics', 'long_desc': '', 'tags': [], 'see_also': []}, 'mark_out_threshold': {'name': 'mark_out_threshold', 'type': 'secs', 'level': 'advanced', 'flags': 1, 'default_value': '2419200', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'automatically mark OSD if it may fail before this long', 'long_desc': '', 'tags': [], 'see_also': []}, 'pool_name': {'name': 'pool_name', 'type': 'str', 'level': 'advanced', 'flags': 1, 'default_value': 'device_health_metrics', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'name of pool in which to store device health metrics', 'long_desc': '', 'tags': [], 'see_also': []}, 'retention_period': {'name': 'retention_period', 'type': 'secs', 'level': 'advanced', 'flags': 1, 'default_value': '15552000', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'how long to retain device health metrics', 'long_desc': '', 'tags': [], 'see_also': []}, 'scrape_frequency': {'name': 'scrape_frequency', 'type': 'secs', 'level': 'advanced', 'flags': 1, 'default_value': '86400', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'how frequently to scrape device health metrics', 'long_desc': '', 'tags': [], 'see_also': []}, 'self_heal': {'name': 'self_heal', 'type': 'bool', 'level': 'advanced', 'flags': 1, 'default_value': 'True', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'preemptively heal cluster around devices that may fail', 'long_desc': '', 'tags': [], 'see_also': []}, 'sleep_interval': {'name': 'sleep_interval', 'type': 'secs', 'level': 'advanced', 'flags': 1, 'default_value': '600', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'how frequently to wake up and check device health', 'long_desc': '', 'tags': [], 'see_also': []}, 'warn_threshold': {'name': 'warn_threshold', 'type': 'secs', 'level': 'advanced', 'flags': 1, 'default_value': '7257600', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'raise health warning if OSD may fail before this long', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'diskprediction_cloud', 'can_run': True, 'error_string': '', 'module_options': {'diskprediction_cert_context': {'name': 'diskprediction_cert_context', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'diskprediction_default_authority': {'name': 'diskprediction_default_authority', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'localhost', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'diskprediction_password': {'name': 'diskprediction_password', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'diskprediction_port': {'name': 'diskprediction_port', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '31400', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'diskprediction_retrieve_prediction_interval': {'name': 'diskprediction_retrieve_prediction_interval', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '43200', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'diskprediction_server': {'name': 'diskprediction_server', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'diskprediction_ssl_target_name_override': {'name': 'diskprediction_ssl_target_name_override', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'localhost', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'diskprediction_upload_metrics_interval': {'name': 'diskprediction_upload_metrics_interval', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '600', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'diskprediction_upload_smart_interval': {'name': 'diskprediction_upload_smart_interval', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '43200', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'diskprediction_user': {'name': 'diskprediction_user', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'sleep_interval': {'name': 'sleep_interval', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '600', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'diskprediction_local', 'can_run': True, 'error_string': '', 'module_options': {'predict_interval': {'name': 'predict_interval', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '86400', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'sleep_interval': {'name': 'sleep_interval', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '600', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'influx', 'can_run': False, 'error_string': 'influxdb python module not found', 'module_options': {'batch_size': {'name': 'batch_size', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '5000', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'database': {'name': 'database', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'ceph', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'hostname': {'name': 'hostname', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'None', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'interval': {'name': 'interval', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '30', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'password': {'name': 'password', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'None', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'port': {'name': 'port', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '8086', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'ssl': {'name': 'ssl', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'false', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'threads': {'name': 'threads', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '5', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'username': {'name': 'username', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'None', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'verify_ssl': {'name': 'verify_ssl', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'true', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'insights', 'can_run': True, 'error_string': '', 'module_options': {}}, {'name': 'iostat', 'can_run': True, 'error_string': '', 'module_options': {}}, {'name': 'localpool', 'can_run': True, 'error_string': '', 'module_options': {'failure_domain': {'name': 'failure_domain', 'type': 'str', 'level': 'advanced', 'flags': 1, 'default_value': 'host', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'failure domain for any created local pool', 'long_desc': '', 'tags': [], 'see_also': []}, 'min_size': {'name': 'min_size', 'type': 'int', 'level': 'advanced', 'flags': 1, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'default min_size for any created local pool', 'long_desc': '', 'tags': [], 'see_also': []}, 'num_rep': {'name': 'num_rep', 'type': 'int', 'level': 'advanced', 'flags': 1, 'default_value': '3', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'default replica count for any created local pool', 'long_desc': '', 'tags': [], 'see_also': []}, 'pg_num': {'name': 'pg_num', 'type': 'int', 'level': 'advanced', 'flags': 1, 'default_value': '128', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'default pg_num for any created local pool', 'long_desc': '', 'tags': [], 'see_also': []}, 'prefix': {'name': 'prefix', 'type': 'str', 'level': 'advanced', 'flags': 1, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'name prefix for any created local pool', 'long_desc': '', 'tags': [], 'see_also': []}, 'subtree': {'name': 'subtree', 'type': 'str', 'level': 'advanced', 'flags': 1, 'default_value': 'rack', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'CRUSH level for which to create a local pool', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'orchestrator_cli', 'can_run': True, 'error_string': '', 'module_options': {'orchestrator': {'name': 'orchestrator', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'pg_autoscaler', 'can_run': True, 'error_string': '', 'module_options': {'sleep_interval': {'name': 'sleep_interval', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '60', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'progress', 'can_run': True, 'error_string': '', 'module_options': {'enabled': {'name': 'enabled', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'True', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'max_completed_events': {'name': 'max_completed_events', 'type': 'int', 'level': 'advanced', 'flags': 1, 'default_value': '50', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'number of past completed events to remember', 'long_desc': '', 'tags': [], 'see_also': []}, 'persist_interval': {'name': 'persist_interval', 'type': 'secs', 'level': 'advanced', 'flags': 1, 'default_value': '5', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'how frequently to persist completed events', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'prometheus', 'can_run': True, 'error_string': '', 'module_options': {'rbd_stats_pools': {'name': 'rbd_stats_pools', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'rbd_stats_pools_refresh_interval': {'name': 'rbd_stats_pools_refresh_interval', 'type': 'int', 'level': 'advanced', 'flags': 0, 'default_value': '300', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'scrape_interval': {'name': 'scrape_interval', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'server_addr': {'name': 'server_addr', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'server_port': {'name': 'server_port', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'stale_cache_strategy': {'name': 'stale_cache_strategy', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'rbd_support', 'can_run': True, 'error_string': '', 'module_options': {}}, {'name': 'restful', 'can_run': True, 'error_string': '', 'module_options': {'key_file': {'name': 'key_file', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'server_addr': {'name': 'server_addr', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'server_port': {'name': 'server_port', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'rook', 'can_run': True, 'error_string': '', 'module_options': {}}, {'name': 'selftest', 'can_run': True, 'error_string': '', 'module_options': {'roption1': {'name': 'roption1', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'roption2': {'name': 'roption2', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'xyz', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'rwoption1': {'name': 'rwoption1', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'rwoption2': {'name': 'rwoption2', 'type': 'int', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'rwoption3': {'name': 'rwoption3', 'type': 'float', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'rwoption4': {'name': 'rwoption4', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'rwoption5': {'name': 'rwoption5', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'rwoption6': {'name': 'rwoption6', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'True', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'testkey': {'name': 'testkey', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'testlkey': {'name': 'testlkey', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'testnewline': {'name': 'testnewline', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'ssh', 'can_run': True, 'error_string': '', 'module_options': {'inventory_cache_timeout_min': {'name': 'inventory_cache_timeout_min', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'ssh_config_file': {'name': 'ssh_config_file', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'status', 'can_run': True, 'error_string': '', 'module_options': {}}, {'name': 'telegraf', 'can_run': True, 'error_string': '', 'module_options': {'address': {'name': 'address', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'unixgram:///tmp/telegraf.sock', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'interval': {'name': 'interval', 'type': 'secs', 'level': 'advanced', 'flags': 0, 'default_value': '15', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'telemetry', 'can_run': True, 'error_string': '', 'module_options': {'channel_basic': {'name': 'channel_basic', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'True', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'Share basic cluster information (size, version)', 'long_desc': '', 'tags': [], 'see_also': []}, 'channel_crash': {'name': 'channel_crash', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'True', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'channel_device': {'name': 'channel_device', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'True', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'channel_ident': {'name': 'channel_ident', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'False', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'contact': {'name': 'contact', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'None', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'description': {'name': 'description', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'None', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'device_url': {'name': 'device_url', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'https://telemetry.ceph.com/device', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'enabled': {'name': 'enabled', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'False', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'interval': {'name': 'interval', 'type': 'int', 'level': 'advanced', 'flags': 0, 'default_value': '24', 'min': '8', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'last_opt_revision': {'name': 'last_opt_revision', 'type': 'int', 'level': 'advanced', 'flags': 0, 'default_value': '1', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'leaderboard': {'name': 'leaderboard', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'False', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'organization': {'name': 'organization', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'None', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'proxy': {'name': 'proxy', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'None', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'url': {'name': 'url', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'https://telemetry.ceph.com/report', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'test_orchestrator', 'can_run': True, 'error_string': '', 'module_options': {}}, {'name': 'volumes', 'can_run': True, 'error_string': '', 'module_options': {'max_concurrent_clones': {'name': 'max_concurrent_clones', 'type': 'int', 'level': 'advanced', 'flags': 0, 'default_value': '4', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'Number of asynchronous cloner threads', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'zabbix', 'can_run': True, 'error_string': '', 'module_options': {'identifier': {'name': 'identifier', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'interval': {'name': 'interval', 'type': 'secs', 'level': 'advanced', 'flags': 0, 'default_value': '60', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'zabbix_host': {'name': 'zabbix_host', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'None', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'zabbix_port': {'name': 'zabbix_port', 'type': 'int', 'level': 'advanced', 'flags': 0, 'default_value': '10051', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'zabbix_sender': {'name': 'zabbix_sender', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '/usr/bin/zabbix_sender', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}}}], 'services': {}, 'always_on_modules': {'nautilus': ['balancer', 'crash', 'devicehealth', 'orchestrator_cli', 'progress', 'rbd_support', 'status', 'volumes']}}, 'servicemap': {'epoch': 2, 'modified': '2021-02-10 11:03:57.678215', 'services': {'mds': {'daemons': {'summary': '', 'b': {'start_epoch': 0, 'start_stamp': '0.000000', 'gid': 0, 'addr': ':/0', 'metadata': {}, 'task_status': {'scrub status': 'idle'}}}}}}, 'progress_events': {}} 2021-02-10T11:04:02.979 INFO:tasks.ceph.ceph_manager.ceph:clean! 2021-02-10T11:04:02.979 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2021-02-10T11:04:02.979 DEBUG:teuthology.orchestra.run.gibba002:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph health 2021-02-10T11:04:03.395 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:04:03.380 7fc2dad35700 1 Processor -- start 2021-02-10T11:04:03.395 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:04:03.380 7fc2dad35700 1 -- start start 2021-02-10T11:04:03.396 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:04:03.381 7fc2dad35700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc2d41b0870 0x7fc2d40f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:03.396 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:04:03.381 7fc2dad35700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fc2d40efe00 0x7fc2d40f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:03.396 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:04:03.381 7fc2dad35700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2d41b0c70 con 0x7fc2d41b0470 2021-02-10T11:04:03.396 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:04:03.381 7fc2dad35700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2d40f0b50 con 0x7fc2d40efe00 2021-02-10T11:04:03.396 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:04:03.381 7fc2dad35700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2d40f0820 con 0x7fc2d41b0870 2021-02-10T11:04:03.396 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:04:03.381 7fc2d8b51700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fc2d41b0470 0x7fc2d40ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:41812/0 (socket says 172.21.2.102:41812) 2021-02-10T11:04:03.396 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:04:03.381 7fc2d8b51700 1 -- 172.21.2.102:0/2659845721 learned_addr learned my addr 172.21.2.102:0/2659845721 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:04:03.396 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:04:03.381 7fc2d9352700 1 --2- 172.21.2.102:0/2659845721 >> v2:172.21.2.113:3300/0 conn(0x7fc2d40efe00 0x7fc2d40f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:03.396 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:04:03.381 7fc2d3fff700 1 --2- 172.21.2.102:0/2659845721 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc2d41b0870 0x7fc2d40f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:03.396 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:04:03.381 7fc2d37fe700 1 -- 172.21.2.102:0/2659845721 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1427022341 0 0) 0x7fc2bc0009b0 con 0x7fc2d41b0470 2021-02-10T11:04:03.396 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:04:03.381 7fc2d37fe700 1 -- 172.21.2.102:0/2659845721 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2c0002fe0 con 0x7fc2d41b0470 2021-02-10T11:04:03.396 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:04:03.381 7fc2d9352700 1 -- 172.21.2.102:0/2659845721 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc2d41b0870 msgr2=0x7fc2d40f20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:04:03.396 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:04:03.381 7fc2d9352700 1 --2- 172.21.2.102:0/2659845721 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc2d41b0870 0x7fc2d40f20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:03.397 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:04:03.381 7fc2d9352700 1 -- 172.21.2.102:0/2659845721 >> v1:172.21.2.102:6789/0 conn(0x7fc2d41b0470 legacy=0x7fc2d40ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:03.397 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:04:03.382 7fc2d9352700 1 -- 172.21.2.102:0/2659845721 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2d40e52e0 con 0x7fc2d40efe00 2021-02-10T11:04:03.397 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:04:03.382 7fc2d3fff700 1 --2- 172.21.2.102:0/2659845721 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc2d41b0870 0x7fc2d40f20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:04:03.397 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:04:03.382 7fc2d9352700 1 --2- 172.21.2.102:0/2659845721 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc2d40efe00 0x7fc2d40f0200 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7fc2cc006e50 tx=0x7fc2cc009e70).ready entity=mon.1 client_cookie=54e93fd067158177 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:04:03.397 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:04:03.382 7fc2d37fe700 1 -- 172.21.2.102:0/2659845721 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc2cc000f90 con 0x7fc2d40efe00 2021-02-10T11:04:03.397 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:04:03.382 7fc2d37fe700 1 -- 172.21.2.102:0/2659845721 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2cc003030 con 0x7fc2d40efe00 2021-02-10T11:04:03.397 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:04:03.382 7fc2d37fe700 1 -- 172.21.2.102:0/2659845721 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc2cc000f90 con 0x7fc2d40efe00 2021-02-10T11:04:03.397 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:04:03.382 7fc2dad35700 1 -- 172.21.2.102:0/2659845721 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc2d40efe00 msgr2=0x7fc2d40f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:03.397 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:04:03.382 7fc2dad35700 1 --2- 172.21.2.102:0/2659845721 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc2d40efe00 0x7fc2d40f0200 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7fc2cc006e50 tx=0x7fc2cc009e70).stop 2021-02-10T11:04:03.397 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:04:03.382 7fc2dad35700 1 -- 172.21.2.102:0/2659845721 shutdown_connections 2021-02-10T11:04:03.397 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:04:03.382 7fc2dad35700 1 --2- 172.21.2.102:0/2659845721 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc2d40efe00 0x7fc2d40f0200 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:03.397 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:04:03.382 7fc2dad35700 1 --2- 172.21.2.102:0/2659845721 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc2d41b0870 0x7fc2d40f20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:03.397 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:04:03.382 7fc2dad35700 1 -- 172.21.2.102:0/2659845721 >> 172.21.2.102:0/2659845721 conn(0x7fc2d40ce3e0 msgr2=0x7fc2d40e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:04:03.397 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:04:03.383 7fc2dad35700 1 -- 172.21.2.102:0/2659845721 shutdown_connections 2021-02-10T11:04:03.398 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:04:03.383 7fc2dad35700 1 -- 172.21.2.102:0/2659845721 wait complete. 2021-02-10T11:04:03.400 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:04:03.397 7fc2dad35700 1 Processor -- start 2021-02-10T11:04:03.400 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:04:03.397 7fc2dad35700 1 -- start start 2021-02-10T11:04:03.400 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:04:03.397 7fc2dad35700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc2d4094240 0x7fc2d4189e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:03.400 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:04:03.397 7fc2dad35700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc2d418a450 0x7fc2d40b18b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:03.400 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:04:03.397 7fc2dad35700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc2d40b1ed0 0x7fc2d418b090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:03.400 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:04:03.397 7fc2dad35700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2d418a850 con 0x7fc2d4094240 2021-02-10T11:04:03.400 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:04:03.397 7fc2dad35700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2d4094640 con 0x7fc2d40b1ed0 2021-02-10T11:04:03.400 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:04:03.397 7fc2dad35700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2d4094ab0 con 0x7fc2d418a450 2021-02-10T11:04:03.401 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:04:03.398 7fc2d8b51700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc2d4094240 0x7fc2d4189e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:03.401 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:04:03.398 7fc2d8b51700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc2d4094240 0x7fc2d4189e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:49596/0 (socket says 172.21.2.102:49596) 2021-02-10T11:04:03.401 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:04:03.398 7fc2d8b51700 1 -- 172.21.2.102:0/1489552459 learned_addr learned my addr 172.21.2.102:0/1489552459 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:04:03.401 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:04:03.398 7fc2d9352700 1 --2- 172.21.2.102:0/1489552459 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc2d40b1ed0 0x7fc2d418b090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:03.401 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:04:03.398 7fc2d8b51700 1 -- 172.21.2.102:0/1489552459 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc2d418a450 msgr2=0x7fc2d40b18b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:03.401 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:04:03.398 7fc2d8b51700 1 --2- 172.21.2.102:0/1489552459 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc2d418a450 0x7fc2d40b18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:03.401 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:04:03.398 7fc2d8b51700 1 -- 172.21.2.102:0/1489552459 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc2d40b1ed0 msgr2=0x7fc2d418b090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:03.401 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:04:03.398 7fc2d8b51700 1 --2- 172.21.2.102:0/1489552459 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc2d40b1ed0 0x7fc2d418b090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:03.401 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:04:03.398 7fc2d8b51700 1 -- 172.21.2.102:0/1489552459 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2d41b4ba0 con 0x7fc2d4094240 2021-02-10T11:04:03.401 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:04:03.398 7fc2d8b51700 1 --2- 172.21.2.102:0/1489552459 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc2d4094240 0x7fc2d4189e30 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7fc2bc001c60 tx=0x7fc2bc00a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:04:03.401 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:04:03.398 7fc2d17fa700 1 -- 172.21.2.102:0/1489552459 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc2bc00acc0 con 0x7fc2d4094240 2021-02-10T11:04:03.401 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:04:03.398 7fc2d17fa700 1 -- 172.21.2.102:0/1489552459 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2bc004140 con 0x7fc2d4094240 2021-02-10T11:04:03.401 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:04:03.398 7fc2d17fa700 1 -- 172.21.2.102:0/1489552459 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc2bc0045a0 con 0x7fc2d4094240 2021-02-10T11:04:03.402 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:04:03.398 7fc2dad35700 1 -- 172.21.2.102:0/1489552459 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2d41b4ba0 con 0x7fc2d4094240 2021-02-10T11:04:03.402 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:04:03.398 7fc2dad35700 1 -- 172.21.2.102:0/1489552459 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2d41b4ba0 con 0x7fc2d4094240 2021-02-10T11:04:03.402 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:04:03.399 7fc2d17fa700 1 -- 172.21.2.102:0/1489552459 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fc2bc007f30 con 0x7fc2d4094240 2021-02-10T11:04:03.402 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:04:03.399 7fc2d17fa700 1 --2- 172.21.2.102:0/1489552459 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc2b8018730 0x7fc2b801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:03.402 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:04:03.399 7fc2d17fa700 1 -- 172.21.2.102:0/1489552459 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fc2bc0325f0 con 0x7fc2d4094240 2021-02-10T11:04:03.402 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:04:03.399 7fc2d3fff700 1 --2- 172.21.2.102:0/1489552459 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc2b8018730 0x7fc2b801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:03.402 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:04:03.400 7fc2d3fff700 1 --2- 172.21.2.102:0/1489552459 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc2b8018730 0x7fc2b801ac10 crc :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:04:03.429 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:04:03.430 7fc2dad35700 1 -- 172.21.2.102:0/1489552459 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2d418a850 con 0x7fc2d4094240 2021-02-10T11:04:03.432 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:04:03.433 7fc2d17fa700 1 -- 172.21.2.102:0/1489552459 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fc2bc000e20 con 0x7fc2d4094240 2021-02-10T11:04:03.564 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:04:03.565 7fc2dad35700 1 -- 172.21.2.102:0/1489552459 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "health"} v 0) v1 -- 0x7fc2d418a850 con 0x7fc2d4094240 2021-02-10T11:04:03.564 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:04:03.565 7fc2d17fa700 1 -- 172.21.2.102:0/1489552459 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "health"}]=0 v0) v1 ==== 54+0+10 (secure 0 0 0) 0x7fc2bc007f30 con 0x7fc2d4094240 2021-02-10T11:04:03.564 INFO:teuthology.misc.health.gibba002.stdout:HEALTH_OK 2021-02-10T11:04:03.566 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:04:03.567 7fc2dad35700 1 -- 172.21.2.102:0/1489552459 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc2b8018730 msgr2=0x7fc2b801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:03.566 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:04:03.567 7fc2dad35700 1 --2- 172.21.2.102:0/1489552459 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc2b8018730 0x7fc2b801ac10 crc :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:03.566 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:04:03.567 7fc2dad35700 1 -- 172.21.2.102:0/1489552459 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc2d4094240 msgr2=0x7fc2d4189e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:03.566 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:04:03.567 7fc2dad35700 1 --2- 172.21.2.102:0/1489552459 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc2d4094240 0x7fc2d4189e30 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7fc2bc001c60 tx=0x7fc2bc00a8d0).stop 2021-02-10T11:04:03.566 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:04:03.567 7fc2dad35700 1 -- 172.21.2.102:0/1489552459 shutdown_connections 2021-02-10T11:04:03.567 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:04:03.567 7fc2dad35700 1 --2- 172.21.2.102:0/1489552459 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc2b8018730 0x7fc2b801ac10 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:03.567 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:04:03.567 7fc2dad35700 1 --2- 172.21.2.102:0/1489552459 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc2d40b1ed0 0x7fc2d418b090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:03.567 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:04:03.567 7fc2dad35700 1 --2- 172.21.2.102:0/1489552459 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc2d418a450 0x7fc2d40b18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:03.567 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:04:03.567 7fc2dad35700 1 --2- 172.21.2.102:0/1489552459 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc2d4094240 0x7fc2d4189e30 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:03.567 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:04:03.567 7fc2dad35700 1 -- 172.21.2.102:0/1489552459 >> 172.21.2.102:0/1489552459 conn(0x7fc2d419f800 msgr2=0x7fc2d4105c40 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:04:03.567 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:04:03.567 7fc2dad35700 1 -- 172.21.2.102:0/1489552459 shutdown_connections 2021-02-10T11:04:03.567 INFO:teuthology.misc.health.gibba002.stderr:2021-02-10 11:04:03.567 7fc2dad35700 1 -- 172.21.2.102:0/1489552459 wait complete. 2021-02-10T11:04:03.573 DEBUG:teuthology.misc:Ceph health: HEALTH_OK 2021-02-10T11:04:03.573 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:04:03.998 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:03.983 7f4ead5af700 1 Processor -- start 2021-02-10T11:04:03.999 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:03.983 7f4ead5af700 1 -- start start 2021-02-10T11:04:03.999 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:03.983 7f4ead5af700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4ea80e5070 0x7f4ea81b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:03.999 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:03.983 7f4ead5af700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f4ea81b3e40 0x7f4ea8080310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:03.999 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:03.983 7f4ead5af700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ea807ed80 con 0x7f4ea80e5470 2021-02-10T11:04:03.999 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:03.983 7f4ead5af700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ea80d1790 con 0x7f4ea81b3e40 2021-02-10T11:04:03.999 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:03.983 7f4ead5af700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ea80d1980 con 0x7f4ea80e5070 2021-02-10T11:04:03.999 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:03.984 7f4ea72bf700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4ea80e5070 0x7f4ea81b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:04.000 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:03.984 7f4ea7ac0700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f4ea81b3e40 0x7f4ea8080310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:04.000 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:03.984 7f4ea6abe700 1 --1- >> v1:172.21.2.113:6789/0 conn(0x7f4ea80e5470 0x7f4ea807d690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.113:6789/0 says I am v1:172.21.2.102:49148/0 (socket says 172.21.2.102:49148) 2021-02-10T11:04:04.000 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:03.984 7f4ea6abe700 1 -- 172.21.2.102:0/927472353 learned_addr learned my addr 172.21.2.102:0/927472353 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:04:04.000 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:03.984 7f4ea72bf700 1 -- 172.21.2.102:0/927472353 >> v2:172.21.2.113:3300/0 conn(0x7f4ea81b3e40 msgr2=0x7f4ea8080310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:04:04.000 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:03.984 7f4ea72bf700 1 --2- 172.21.2.102:0/927472353 >> v2:172.21.2.113:3300/0 conn(0x7f4ea81b3e40 0x7f4ea8080310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:04.000 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:03.984 7f4ea72bf700 1 -- 172.21.2.102:0/927472353 >> v1:172.21.2.113:6789/0 conn(0x7f4ea80e5470 legacy=0x7f4ea807d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:04.001 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:03.984 7f4ea72bf700 1 -- 172.21.2.102:0/927472353 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ea80d1ba0 con 0x7f4ea80e5070 2021-02-10T11:04:04.001 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:03.985 7f4ea72bf700 1 --2- 172.21.2.102:0/927472353 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4ea80e5070 0x7f4ea81b3820 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7f4e98006650 tx=0x7f4e98009a80).ready entity=mon.2 client_cookie=95350f96c8532b3e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:04:04.001 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:03.985 7f4ea62bd700 1 -- 172.21.2.102:0/927472353 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4e980009b0 con 0x7f4ea80e5070 2021-02-10T11:04:04.001 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:03.985 7f4ea62bd700 1 -- 172.21.2.102:0/927472353 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e9800c0c0 con 0x7f4ea80e5070 2021-02-10T11:04:04.001 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:03.985 7f4ea62bd700 1 -- 172.21.2.102:0/927472353 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4e9800c5e0 con 0x7f4ea80e5070 2021-02-10T11:04:04.001 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:03.985 7f4ead5af700 1 -- 172.21.2.102:0/927472353 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4ea80e5070 msgr2=0x7f4ea81b3820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:04.002 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:03.985 7f4ead5af700 1 --2- 172.21.2.102:0/927472353 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4ea80e5070 0x7f4ea81b3820 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7f4e98006650 tx=0x7f4e98009a80).stop 2021-02-10T11:04:04.002 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:03.986 7f4ead5af700 1 -- 172.21.2.102:0/927472353 shutdown_connections 2021-02-10T11:04:04.002 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:03.986 7f4ead5af700 1 --2- 172.21.2.102:0/927472353 >> v2:172.21.2.113:3300/0 conn(0x7f4ea81b3e40 0x7f4ea8080310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:04.002 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:03.986 7f4ead5af700 1 --2- 172.21.2.102:0/927472353 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4ea80e5070 0x7f4ea81b3820 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:04.002 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:03.986 7f4ead5af700 1 -- 172.21.2.102:0/927472353 >> 172.21.2.102:0/927472353 conn(0x7f4ea80ce690 msgr2=0x7f4ea80e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:04:04.002 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:03.986 7f4ead5af700 1 -- 172.21.2.102:0/927472353 shutdown_connections 2021-02-10T11:04:04.003 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:03.986 7f4ead5af700 1 -- 172.21.2.102:0/927472353 wait complete. 2021-02-10T11:04:04.005 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:04.000 7f4ead5af700 1 Processor -- start 2021-02-10T11:04:04.005 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:04.000 7f4ead5af700 1 -- start start 2021-02-10T11:04:04.005 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:04.001 7f4ead5af700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4ea808d400 0x7f4ea8081230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:04.006 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:04.001 7f4ead5af700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4ea8081850 0x7f4ea80acf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:04.006 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:04.001 7f4ead5af700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4ea8081c50 0x7f4ea80ad5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:04.006 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:04.001 7f4ead5af700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ea8082110 con 0x7f4ea808d400 2021-02-10T11:04:04.006 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:04.001 7f4ead5af700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ea808d800 con 0x7f4ea8081850 2021-02-10T11:04:04.006 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:04.001 7f4ead5af700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ea80adbc0 con 0x7f4ea8081c50 2021-02-10T11:04:04.007 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:04.001 7f4ea72bf700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4ea808d400 0x7f4ea8081230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:04.007 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:04.001 7f4ea72bf700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4ea808d400 0x7f4ea8081230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:49608/0 (socket says 172.21.2.102:49608) 2021-02-10T11:04:04.007 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:04.001 7f4ea72bf700 1 -- 172.21.2.102:0/187627352 learned_addr learned my addr 172.21.2.102:0/187627352 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:04:04.007 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:04.001 7f4ea6abe700 1 --2- 172.21.2.102:0/187627352 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4ea8081850 0x7f4ea80acf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:04.007 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:04.001 7f4ea7ac0700 1 --2- 172.21.2.102:0/187627352 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4ea8081c50 0x7f4ea80ad5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:04.007 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:04.001 7f4ea72bf700 1 -- 172.21.2.102:0/187627352 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4ea8081c50 msgr2=0x7f4ea80ad5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:04.007 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:04.001 7f4ea72bf700 1 --2- 172.21.2.102:0/187627352 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4ea8081c50 0x7f4ea80ad5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:04.008 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:04.001 7f4ea72bf700 1 -- 172.21.2.102:0/187627352 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4ea8081850 msgr2=0x7f4ea80acf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:04.008 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:04.001 7f4ea72bf700 1 --2- 172.21.2.102:0/187627352 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4ea8081850 0x7f4ea80acf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:04.008 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:04.001 7f4ea72bf700 1 -- 172.21.2.102:0/187627352 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ea8187fc0 con 0x7f4ea808d400 2021-02-10T11:04:04.008 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:04.001 7f4ea72bf700 1 --2- 172.21.2.102:0/187627352 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4ea808d400 0x7f4ea8081230 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7f4e98009630 tx=0x7f4e9800b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:04:04.008 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:04.002 7f4e9ffff700 1 -- 172.21.2.102:0/187627352 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4e980093a0 con 0x7f4ea808d400 2021-02-10T11:04:04.008 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:04.002 7f4e9ffff700 1 -- 172.21.2.102:0/187627352 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e9800bd70 con 0x7f4ea808d400 2021-02-10T11:04:04.008 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:04.002 7f4e9ffff700 1 -- 172.21.2.102:0/187627352 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4e98005560 con 0x7f4ea808d400 2021-02-10T11:04:04.008 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:04.002 7f4ead5af700 1 -- 172.21.2.102:0/187627352 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ea8187fc0 con 0x7f4ea808d400 2021-02-10T11:04:04.008 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:04.002 7f4ead5af700 1 -- 172.21.2.102:0/187627352 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ea8187fc0 con 0x7f4ea808d400 2021-02-10T11:04:04.008 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:04.002 7f4e9ffff700 1 -- 172.21.2.102:0/187627352 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4e9800a000 con 0x7f4ea808d400 2021-02-10T11:04:04.008 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:04.002 7f4e9ffff700 1 --2- 172.21.2.102:0/187627352 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4e8c0186c0 0x7f4e8c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:04.009 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:04.002 7f4e9ffff700 1 -- 172.21.2.102:0/187627352 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f4e98032530 con 0x7f4ea808d400 2021-02-10T11:04:04.009 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:04.003 7f4ea6abe700 1 --2- 172.21.2.102:0/187627352 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4e8c0186c0 0x7f4e8c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:04.009 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:04.003 7f4ea6abe700 1 --2- 172.21.2.102:0/187627352 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4e8c0186c0 0x7f4e8c01abc0 crc :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:04:04.032 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:04.033 7f4ead5af700 1 -- 172.21.2.102:0/187627352 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ea808d800 con 0x7f4ea808d400 2021-02-10T11:04:04.035 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:04.036 7f4e9ffff700 1 -- 172.21.2.102:0/187627352 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f4e98030ce0 con 0x7f4ea808d400 2021-02-10T11:04:04.162 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:04.163 7f4ead5af700 1 -- 172.21.2.102:0/187627352 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4ea808d800 con 0x7f4ea808d400 2021-02-10T11:04:04.162 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:04.163 7f4e9ffff700 1 -- 172.21.2.102:0/187627352 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+13993 (secure 0 0 0) 0x7f4e9800a000 con 0x7f4ea808d400 2021-02-10T11:04:04.163 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:04:04.163 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":20,"fsid":"68db30ea-dbce-4aef-a6df-f01c6fea0513","created":"2021-02-10 11:03:06.331557","modified":"2021-02-10 11:03:44.386945","last_up_change":"2021-02-10 11:03:32.690562","last_in_change":"2021-02-10 11:03:32.690562","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:03:37.322519","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:03:41.161563","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2021-02-10 11:03:42.379651","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"be691b4d-a039-408a-af80-9d53f02b6f1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6818","nonce":36088},{"type":"v1","addr":"172.21.2.102:6820","nonce":36088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6822","nonce":36088},{"type":"v1","addr":"172.21.2.102:6824","nonce":36088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6830","nonce":36088},{"type":"v1","addr":"172.21.2.102:6832","nonce":36088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6826","nonce":36088},{"type":"v1","addr":"172.21.2.102:6828","nonce":36088}]},"public_addr":"172.21.2.102:6820/36088","cluster_addr":"172.21.2.102:6824/36088","heartbeat_back_addr":"172.21.2.102:6832/36088","heartbeat_front_addr":"172.21.2.102:6828/36088","state":["exists","up"]},{"osd":1,"uuid":"64e3ccc8-9c69-4d7c-865e-af8e83822635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6819","nonce":36087},{"type":"v1","addr":"172.21.2.102:6821","nonce":36087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6823","nonce":36087},{"type":"v1","addr":"172.21.2.102:6825","nonce":36087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6831","nonce":36087},{"type":"v1","addr":"172.21.2.102:6833","nonce":36087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6827","nonce":36087},{"type":"v1","addr":"172.21.2.102:6829","nonce":36087}]},"public_addr":"172.21.2.102:6821/36087","cluster_addr":"172.21.2.102:6825/36087","heartbeat_back_addr":"172.21.2.102:6833/36087","heartbeat_front_addr":"172.21.2.102:6829/36087","state":["exists","up"]},{"osd":2,"uuid":"153700af-747f-481f-bdf4-d6072d3a842f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6810","nonce":36089},{"type":"v1","addr":"172.21.2.102:6811","nonce":36089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6812","nonce":36089},{"type":"v1","addr":"172.21.2.102:6813","nonce":36089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6816","nonce":36089},{"type":"v1","addr":"172.21.2.102:6817","nonce":36089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6814","nonce":36089},{"type":"v1","addr":"172.21.2.102:6815","nonce":36089}]},"public_addr":"172.21.2.102:6811/36089","cluster_addr":"172.21.2.102:6813/36089","heartbeat_back_addr":"172.21.2.102:6817/36089","heartbeat_front_addr":"172.21.2.102:6815/36089","state":["exists","up"]},{"osd":3,"uuid":"cbc678f4-83a9-45e6-8830-2f7c232b1ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6802","nonce":36090},{"type":"v1","addr":"172.21.2.102:6803","nonce":36090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6804","nonce":36090},{"type":"v1","addr":"172.21.2.102:6805","nonce":36090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6808","nonce":36090},{"type":"v1","addr":"172.21.2.102:6809","nonce":36090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6806","nonce":36090},{"type":"v1","addr":"172.21.2.102:6807","nonce":36090}]},"public_addr":"172.21.2.102:6803/36090","cluster_addr":"172.21.2.102:6805/36090","heartbeat_back_addr":"172.21.2.102:6809/36090","heartbeat_front_addr":"172.21.2.102:6807/36090","state":["exists","up"]},{"osd":4,"uuid":"c01e4008-b258-4f10-afd2-9e7f37214351","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6808","nonce":36368},{"type":"v1","addr":"172.21.2.113:6811","nonce":36368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6814","nonce":36368},{"type":"v1","addr":"172.21.2.113:6818","nonce":36368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6826","nonce":36368},{"type":"v1","addr":"172.21.2.113:6828","nonce":36368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6820","nonce":36368},{"type":"v1","addr":"172.21.2.113:6824","nonce":36368}]},"public_addr":"172.21.2.113:6811/36368","cluster_addr":"172.21.2.113:6818/36368","heartbeat_back_addr":"172.21.2.113:6828/36368","heartbeat_front_addr":"172.21.2.113:6824/36368","state":["exists","up"]},{"osd":5,"uuid":"551f6007-daf8-4a98-8ee0-c2b16bcccb5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6802","nonce":36366},{"type":"v1","addr":"172.21.2.113:6805","nonce":36366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6807","nonce":36366},{"type":"v1","addr":"172.21.2.113:6810","nonce":36366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6819","nonce":36366},{"type":"v1","addr":"172.21.2.113:6822","nonce":36366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6812","nonce":36366},{"type":"v1","addr":"172.21.2.113:6815","nonce":36366}]},"public_addr":"172.21.2.113:6805/36366","cluster_addr":"172.21.2.113:6810/36366","heartbeat_back_addr":"172.21.2.113:6822/36366","heartbeat_front_addr":"172.21.2.113:6815/36366","state":["exists","up"]},{"osd":6,"uuid":"bf67c040-0e50-48be-b102-c3943bc9ebef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6800","nonce":36367},{"type":"v1","addr":"172.21.2.113:6801","nonce":36367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6803","nonce":36367},{"type":"v1","addr":"172.21.2.113:6804","nonce":36367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6813","nonce":36367},{"type":"v1","addr":"172.21.2.113:6816","nonce":36367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6806","nonce":36367},{"type":"v1","addr":"172.21.2.113:6809","nonce":36367}]},"public_addr":"172.21.2.113:6801/36367","cluster_addr":"172.21.2.113:6804/36367","heartbeat_back_addr":"172.21.2.113:6816/36367","heartbeat_front_addr":"172.21.2.113:6809/36367","state":["exists","up"]},{"osd":7,"uuid":"f5461f80-018d-428f-be89-2eb2bcbe7bb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6817","nonce":36365},{"type":"v1","addr":"172.21.2.113:6821","nonce":36365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6823","nonce":36365},{"type":"v1","addr":"172.21.2.113:6825","nonce":36365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6830","nonce":36365},{"type":"v1","addr":"172.21.2.113:6831","nonce":36365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6827","nonce":36365},{"type":"v1","addr":"172.21.2.113:6829","nonce":36365}]},"public_addr":"172.21.2.113:6821/36365","cluster_addr":"172.21.2.113:6825/36365","heartbeat_back_addr":"172.21.2.113:6831/36365","heartbeat_front_addr":"172.21.2.113:6829/36365","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:04:04.164 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:04.165 7f4ead5af700 1 -- 172.21.2.102:0/187627352 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4e8c0186c0 msgr2=0x7f4e8c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:04.164 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:04.165 7f4ead5af700 1 --2- 172.21.2.102:0/187627352 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4e8c0186c0 0x7f4e8c01abc0 crc :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:04.164 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:04.165 7f4ead5af700 1 -- 172.21.2.102:0/187627352 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4ea808d400 msgr2=0x7f4ea8081230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:04.165 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:04.165 7f4ead5af700 1 --2- 172.21.2.102:0/187627352 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4ea808d400 0x7f4ea8081230 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7f4e98009630 tx=0x7f4e9800b980).stop 2021-02-10T11:04:04.165 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:04.165 7f4ead5af700 1 -- 172.21.2.102:0/187627352 shutdown_connections 2021-02-10T11:04:04.165 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:04.165 7f4ead5af700 1 --2- 172.21.2.102:0/187627352 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4ea8081c50 0x7f4ea80ad5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:04.165 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:04.165 7f4ead5af700 1 --2- 172.21.2.102:0/187627352 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4e8c0186c0 0x7f4e8c01abc0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:04.165 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:04.165 7f4ead5af700 1 --2- 172.21.2.102:0/187627352 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4ea8081850 0x7f4ea80acf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:04.165 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:04.165 7f4ead5af700 1 --2- 172.21.2.102:0/187627352 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4ea808d400 0x7f4ea8081230 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:04.165 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:04.165 7f4ead5af700 1 -- 172.21.2.102:0/187627352 >> 172.21.2.102:0/187627352 conn(0x7f4ea81ac620 msgr2=0x7f4ea80896e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:04:04.165 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:04.165 7f4ead5af700 1 -- 172.21.2.102:0/187627352 shutdown_connections 2021-02-10T11:04:04.165 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:04.165 7f4ead5af700 1 -- 172.21.2.102:0/187627352 wait complete. 2021-02-10T11:04:04.173 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:03:37.322519', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2021-02-10 11:03:41.161563', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2021-02-10 11:03:42.379651', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-02-10T11:04:04.173 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:04:04.604 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:04.587 7f96f2c82700 1 Processor -- start 2021-02-10T11:04:04.604 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:04.587 7f96f2c82700 1 -- start start 2021-02-10T11:04:04.604 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:04.588 7f96f2c82700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f96ec0f05f0 0x7f96ec0f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:04.604 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:04.588 7f96f2c82700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96ec1b0c70 con 0x7f96ec1b0870 2021-02-10T11:04:04.604 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:04.588 7f96f2c82700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96ec0f0b50 con 0x7f96ec1b0470 2021-02-10T11:04:04.604 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:04.588 7f96f2c82700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96ec0e5ea0 con 0x7f96ec0f05f0 2021-02-10T11:04:04.604 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:04.588 7f96ebfff700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f96ec1b0870 0x7f96ec0efe00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:41840/0 (socket says 172.21.2.102:41840) 2021-02-10T11:04:04.605 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:04.588 7f96ebfff700 1 -- 172.21.2.102:0/2902956380 learned_addr learned my addr 172.21.2.102:0/2902956380 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:04:04.605 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:04.588 7f96f129f700 1 --2- 172.21.2.102:0/2902956380 >> v2:172.21.2.113:3300/0 conn(0x7f96ec0f05f0 0x7f96ec0f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:04.605 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:04.588 7f96eb7fe700 1 -- 172.21.2.102:0/2902956380 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3646243790 0 0) 0x7f96dc0009b0 con 0x7f96ec1b0870 2021-02-10T11:04:04.605 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:04.588 7f96eb7fe700 1 -- 172.21.2.102:0/2902956380 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96d8002fe0 con 0x7f96ec1b0870 2021-02-10T11:04:04.605 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:04.589 7f96f129f700 1 -- 172.21.2.102:0/2902956380 >> v1:172.21.2.113:6789/0 conn(0x7f96ec1b0470 legacy=0x7f96ec0ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:04.605 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:04.589 7f96eb7fe700 1 -- 172.21.2.102:0/2902956380 <== mon.1 v1:172.21.2.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1402534891 0 0) 0x7f96d4000ff0 con 0x7f96ec1b0470 2021-02-10T11:04:04.606 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:04.589 7f96f129f700 1 -- 172.21.2.102:0/2902956380 >> v1:172.21.2.102:6789/0 conn(0x7f96ec1b0870 legacy=0x7f96ec0efe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:04.606 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:04.589 7f96f129f700 1 -- 172.21.2.102:0/2902956380 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96ec0e52e0 con 0x7f96ec0f05f0 2021-02-10T11:04:04.606 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:04.589 7f96f129f700 1 --2- 172.21.2.102:0/2902956380 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f96ec0f05f0 0x7f96ec0f20e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7f96e4006e50 tx=0x7f96e4009e70).ready entity=mon.1 client_cookie=46453d753e77f17f server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:04:04.606 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:04.589 7f96eb7fe700 1 -- 172.21.2.102:0/2902956380 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f96e4006a20 con 0x7f96ec0f05f0 2021-02-10T11:04:04.606 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:04.589 7f96eb7fe700 1 -- 172.21.2.102:0/2902956380 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96e4006790 con 0x7f96ec0f05f0 2021-02-10T11:04:04.606 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:04.589 7f96eb7fe700 1 -- 172.21.2.102:0/2902956380 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f96e400b700 con 0x7f96ec0f05f0 2021-02-10T11:04:04.606 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:04.589 7f96f2c82700 1 -- 172.21.2.102:0/2902956380 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f96ec0f05f0 msgr2=0x7f96ec0f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:04.606 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:04.589 7f96f2c82700 1 --2- 172.21.2.102:0/2902956380 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f96ec0f05f0 0x7f96ec0f20e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7f96e4006e50 tx=0x7f96e4009e70).stop 2021-02-10T11:04:04.606 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:04.590 7f96f2c82700 1 -- 172.21.2.102:0/2902956380 shutdown_connections 2021-02-10T11:04:04.606 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:04.590 7f96f2c82700 1 --2- 172.21.2.102:0/2902956380 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f96ec0f05f0 0x7f96ec0f20e0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:04.606 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:04.590 7f96f2c82700 1 -- 172.21.2.102:0/2902956380 >> 172.21.2.102:0/2902956380 conn(0x7f96ec0ce3e0 msgr2=0x7f96ec0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:04:04.606 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:04.590 7f96f2c82700 1 -- 172.21.2.102:0/2902956380 shutdown_connections 2021-02-10T11:04:04.607 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:04.590 7f96f2c82700 1 -- 172.21.2.102:0/2902956380 wait complete. 2021-02-10T11:04:04.607 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:04.605 7f96f2c82700 1 Processor -- start 2021-02-10T11:04:04.607 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:04.605 7f96f2c82700 1 -- start start 2021-02-10T11:04:04.609 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:04.605 7f96f2c82700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f96ec190920 0x7f96ec08f600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:04.609 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:04.605 7f96f2c82700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f96ec190d20 0x7f96ec08fc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:04.609 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:04.605 7f96f2c82700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f96ec193270 0x7f96ec193670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:04.609 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:04.605 7f96f2c82700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96ec090470 con 0x7f96ec190920 2021-02-10T11:04:04.609 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:04.605 7f96f2c82700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96ec193c90 con 0x7f96ec190d20 2021-02-10T11:04:04.609 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:04.605 7f96f2c82700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96ec193e70 con 0x7f96ec193270 2021-02-10T11:04:04.609 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:04.605 7f96f0a9e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f96ec190920 0x7f96ec08f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:04.609 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:04.606 7f96f0a9e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f96ec190920 0x7f96ec08f600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:49618/0 (socket says 172.21.2.102:49618) 2021-02-10T11:04:04.610 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:04.606 7f96f0a9e700 1 -- 172.21.2.102:0/3304526497 learned_addr learned my addr 172.21.2.102:0/3304526497 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:04:04.610 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:04.606 7f96ebfff700 1 --2- 172.21.2.102:0/3304526497 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f96ec190d20 0x7f96ec08fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:04.610 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:04.606 7f96f129f700 1 --2- 172.21.2.102:0/3304526497 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f96ec193270 0x7f96ec193670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:04.610 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:04.606 7f96f0a9e700 1 -- 172.21.2.102:0/3304526497 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f96ec193270 msgr2=0x7f96ec193670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:04.610 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:04.606 7f96f0a9e700 1 --2- 172.21.2.102:0/3304526497 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f96ec193270 0x7f96ec193670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:04.610 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:04.606 7f96f0a9e700 1 -- 172.21.2.102:0/3304526497 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f96ec190d20 msgr2=0x7f96ec08fc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:04.610 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:04.606 7f96f0a9e700 1 --2- 172.21.2.102:0/3304526497 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f96ec190d20 0x7f96ec08fc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:04.610 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:04.606 7f96f0a9e700 1 -- 172.21.2.102:0/3304526497 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96ec194090 con 0x7f96ec190920 2021-02-10T11:04:04.610 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:04.606 7f96f0a9e700 1 --2- 172.21.2.102:0/3304526497 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f96ec190920 0x7f96ec08f600 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f96d400a570 tx=0x7f96d400a910).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:04:04.610 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:04.606 7f96e97fa700 1 -- 172.21.2.102:0/3304526497 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f96d400ad00 con 0x7f96ec190920 2021-02-10T11:04:04.610 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:04.606 7f96e97fa700 1 -- 172.21.2.102:0/3304526497 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96d400c0d0 con 0x7f96ec190920 2021-02-10T11:04:04.610 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:04.606 7f96e97fa700 1 -- 172.21.2.102:0/3304526497 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f96d400c520 con 0x7f96ec190920 2021-02-10T11:04:04.611 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:04.606 7f96f2c82700 1 -- 172.21.2.102:0/3304526497 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96ec194090 con 0x7f96ec190920 2021-02-10T11:04:04.611 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:04.606 7f96f2c82700 1 -- 172.21.2.102:0/3304526497 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96ec194090 con 0x7f96ec190920 2021-02-10T11:04:04.611 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:04.607 7f96e97fa700 1 -- 172.21.2.102:0/3304526497 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f96d4008030 con 0x7f96ec190920 2021-02-10T11:04:04.611 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:04.607 7f96e97fa700 1 --2- 172.21.2.102:0/3304526497 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f96d00186c0 0x7f96d001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:04.611 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:04.607 7f96e97fa700 1 -- 172.21.2.102:0/3304526497 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f96d40326a0 con 0x7f96ec190920 2021-02-10T11:04:04.611 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:04.607 7f96ebfff700 1 --2- 172.21.2.102:0/3304526497 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f96d00186c0 0x7f96d001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:04.611 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:04.607 7f96ebfff700 1 --2- 172.21.2.102:0/3304526497 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f96d00186c0 0x7f96d001abc0 crc :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:04:04.636 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:04.637 7f96f2c82700 1 -- 172.21.2.102:0/3304526497 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96ec105af0 con 0x7f96ec190920 2021-02-10T11:04:04.639 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:04.640 7f96e97fa700 1 -- 172.21.2.102:0/3304526497 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f96d4000d90 con 0x7f96ec190920 2021-02-10T11:04:04.768 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:04.769 7f96f2c82700 1 -- 172.21.2.102:0/3304526497 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f96ec0f0b50 con 0x7f96ec190920 2021-02-10T11:04:04.769 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:04.769 7f96e97fa700 1 -- 172.21.2.102:0/3304526497 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v20) v1 ==== 92+0+10 (secure 0 0 0) 0x7f96d4008030 con 0x7f96ec190920 2021-02-10T11:04:04.769 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:04:04.770 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:04.771 7f96f2c82700 1 -- 172.21.2.102:0/3304526497 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f96d00186c0 msgr2=0x7f96d001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:04.770 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:04.771 7f96f2c82700 1 --2- 172.21.2.102:0/3304526497 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f96d00186c0 0x7f96d001abc0 crc :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:04.770 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:04.771 7f96f2c82700 1 -- 172.21.2.102:0/3304526497 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f96ec190920 msgr2=0x7f96ec08f600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:04.771 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:04.771 7f96f2c82700 1 --2- 172.21.2.102:0/3304526497 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f96ec190920 0x7f96ec08f600 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f96d400a570 tx=0x7f96d400a910).stop 2021-02-10T11:04:04.771 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:04.772 7f96f2c82700 1 -- 172.21.2.102:0/3304526497 shutdown_connections 2021-02-10T11:04:04.771 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:04.772 7f96f2c82700 1 --2- 172.21.2.102:0/3304526497 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f96ec193270 0x7f96ec193670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:04.771 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:04.772 7f96f2c82700 1 --2- 172.21.2.102:0/3304526497 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f96d00186c0 0x7f96d001abc0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:04.771 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:04.772 7f96f2c82700 1 --2- 172.21.2.102:0/3304526497 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f96ec190d20 0x7f96ec08fc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:04.771 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:04.772 7f96f2c82700 1 --2- 172.21.2.102:0/3304526497 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f96ec190920 0x7f96ec08f600 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:04.771 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:04.772 7f96f2c82700 1 -- 172.21.2.102:0/3304526497 >> 172.21.2.102:0/3304526497 conn(0x7f96ec0bf4e0 msgr2=0x7f96ec0be3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:04:04.772 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:04.772 7f96f2c82700 1 -- 172.21.2.102:0/3304526497 shutdown_connections 2021-02-10T11:04:04.772 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:04.772 7f96f2c82700 1 -- 172.21.2.102:0/3304526497 wait complete. 2021-02-10T11:04:04.779 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-02-10T11:04:05.209 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:05.193 7f577b04c700 1 Processor -- start 2021-02-10T11:04:05.210 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:05.193 7f577b04c700 1 -- start start 2021-02-10T11:04:05.210 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:05.194 7f577b04c700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f57741b0470 0x7f57741b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:05.210 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:05.194 7f577b04c700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f57740ef610 0x7f57740efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:05.210 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:05.194 7f577b04c700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57740f0b50 con 0x7f57740f0030 2021-02-10T11:04:05.210 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:05.194 7f577b04c700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57740e52e0 con 0x7f57741b0470 2021-02-10T11:04:05.210 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:05.194 7f577b04c700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57740e54d0 con 0x7f57740ef610 2021-02-10T11:04:05.210 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:05.194 7f5779669700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f57740f0030 0x7f57740f10d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:41854/0 (socket says 172.21.2.102:41854) 2021-02-10T11:04:05.210 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:05.194 7f5779669700 1 -- 172.21.2.102:0/1862760865 learned_addr learned my addr 172.21.2.102:0/1862760865 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:04:05.210 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:05.194 7f57737fe700 1 -- 172.21.2.102:0/1862760865 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2249074903 0 0) 0x7f576c001180 con 0x7f57740f0030 2021-02-10T11:04:05.210 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:05.194 7f57737fe700 1 -- 172.21.2.102:0/1862760865 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5760002fe0 con 0x7f57740f0030 2021-02-10T11:04:05.210 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:05.194 7f5778e68700 1 --2- 172.21.2.102:0/1862760865 >> v2:172.21.2.113:3300/0 conn(0x7f57741b0470 0x7f57741b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:05.211 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:05.194 7f5773fff700 1 --2- 172.21.2.102:0/1862760865 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f57740ef610 0x7f57740efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:05.211 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:05.195 7f57737fe700 1 -- 172.21.2.102:0/1862760865 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4290731210 0 0) 0x7f576c0010e0 con 0x7f57740f0030 2021-02-10T11:04:05.211 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:05.195 7f57737fe700 1 -- 172.21.2.102:0/1862760865 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f57740ef610 msgr2=0x7f57740efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:04:05.211 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:05.195 7f57737fe700 1 --2- 172.21.2.102:0/1862760865 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f57740ef610 0x7f57740efa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:05.211 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:05.195 7f57737fe700 1 -- 172.21.2.102:0/1862760865 >> v2:172.21.2.113:3300/0 conn(0x7f57741b0470 msgr2=0x7f57741b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:04:05.211 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:05.195 7f57737fe700 1 --2- 172.21.2.102:0/1862760865 >> v2:172.21.2.113:3300/0 conn(0x7f57741b0470 0x7f57741b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:05.211 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:05.195 7f57737fe700 1 -- 172.21.2.102:0/1862760865 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57740f0430 con 0x7f57740f0030 2021-02-10T11:04:05.211 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:05.195 7f57737fe700 1 -- 172.21.2.102:0/1862760865 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7f576c001600 con 0x7f57740f0030 2021-02-10T11:04:05.211 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:05.195 7f57737fe700 1 -- 172.21.2.102:0/1862760865 >> v1:172.21.2.102:6789/0 conn(0x7f57740f0030 legacy=0x7f57740f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:05.211 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:05.195 7f57737fe700 1 --2- 172.21.2.102:0/1862760865 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5760003b10 0x7f5760005f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:05.211 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:05.195 7f57737fe700 1 --2- 172.21.2.102:0/1862760865 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5760006790 0x7f5760008cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:05.211 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:05.195 7f57737fe700 1 --2- 172.21.2.102:0/1862760865 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f57600095b0 0x7f576000ba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:05.211 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:05.195 7f57737fe700 1 -- 172.21.2.102:0/1862760865 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57600502e0 con 0x7f5760003b10 2021-02-10T11:04:05.211 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:05.195 7f57737fe700 1 -- 172.21.2.102:0/1862760865 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5760050500 con 0x7f5760006790 2021-02-10T11:04:05.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:05.195 7f57737fe700 1 -- 172.21.2.102:0/1862760865 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5760050720 con 0x7f57600095b0 2021-02-10T11:04:05.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:05.195 7f5778e68700 1 --2- 172.21.2.102:0/1862760865 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5760003b10 0x7f5760005f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:05.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:05.195 7f5773fff700 1 --2- 172.21.2.102:0/1862760865 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5760006790 0x7f5760008cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:05.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:05.195 7f5779669700 1 --2- 172.21.2.102:0/1862760865 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f57600095b0 0x7f576000ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:05.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:05.195 7f5778e68700 1 -- 172.21.2.102:0/1862760865 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f57600095b0 msgr2=0x7f576000ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:04:05.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:05.195 7f5778e68700 1 --2- 172.21.2.102:0/1862760865 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f57600095b0 0x7f576000ba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:05.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:05.195 7f5778e68700 1 -- 172.21.2.102:0/1862760865 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5760006790 msgr2=0x7f5760008cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:04:05.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:05.195 7f5778e68700 1 --2- 172.21.2.102:0/1862760865 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5760006790 0x7f5760008cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:05.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:05.195 7f5778e68700 1 -- 172.21.2.102:0/1862760865 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57600509c0 con 0x7f5760003b10 2021-02-10T11:04:05.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:05.195 7f5779669700 1 --2- 172.21.2.102:0/1862760865 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f57600095b0 0x7f576000ba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:04:05.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:05.195 7f5778e68700 1 --2- 172.21.2.102:0/1862760865 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5760003b10 0x7f5760005f80 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7f575c006360 tx=0x7f575c009c30).ready entity=mon.0 client_cookie=b6fa0ffdda7a624d server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:04:05.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:05.195 7f57737fe700 1 -- 172.21.2.102:0/1862760865 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f575c0009b0 con 0x7f5760003b10 2021-02-10T11:04:05.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:05.195 7f57737fe700 1 -- 172.21.2.102:0/1862760865 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f575c000c90 con 0x7f5760003b10 2021-02-10T11:04:05.213 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:05.195 7f57737fe700 1 -- 172.21.2.102:0/1862760865 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f575c002220 con 0x7f5760003b10 2021-02-10T11:04:05.213 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:05.196 7f577b04c700 1 -- 172.21.2.102:0/1862760865 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5760003b10 msgr2=0x7f5760005f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:05.213 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:05.196 7f577b04c700 1 --2- 172.21.2.102:0/1862760865 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5760003b10 0x7f5760005f80 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7f575c006360 tx=0x7f575c009c30).stop 2021-02-10T11:04:05.213 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:05.196 7f5779669700 1 -- 172.21.2.102:0/1862760865 reap_dead start 2021-02-10T11:04:05.213 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:05.196 7f577b04c700 1 -- 172.21.2.102:0/1862760865 shutdown_connections 2021-02-10T11:04:05.213 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:05.196 7f577b04c700 1 -- 172.21.2.102:0/1862760865 >> 172.21.2.102:0/1862760865 conn(0x7f57740ce3e0 msgr2=0x7f57740e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:04:05.213 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:05.196 7f577b04c700 1 -- 172.21.2.102:0/1862760865 shutdown_connections 2021-02-10T11:04:05.213 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:05.196 7f577b04c700 1 -- 172.21.2.102:0/1862760865 wait complete. 2021-02-10T11:04:05.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:05.211 7f577b04c700 1 Processor -- start 2021-02-10T11:04:05.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:05.211 7f577b04c700 1 -- start start 2021-02-10T11:04:05.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:05.211 7f577b04c700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f577409b930 0x7f57740b7c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:05.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:05.211 7f577b04c700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f57740b8410 0x7f57740b8810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:05.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:05.211 7f577b04c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f57740c1f80 0x7f57740c2380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:05.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:05.211 7f577b04c700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57740c1a00 con 0x7f57740c1f80 2021-02-10T11:04:05.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:05.211 7f577b04c700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57740b8e30 con 0x7f577409b930 2021-02-10T11:04:05.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:05.211 7f577b04c700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57740b9050 con 0x7f57740b8410 2021-02-10T11:04:05.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:05.212 7f5779669700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f57740c1f80 0x7f57740c2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:05.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:05.212 7f5779669700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f57740c1f80 0x7f57740c2380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:49640/0 (socket says 172.21.2.102:49640) 2021-02-10T11:04:05.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:05.212 7f5779669700 1 -- 172.21.2.102:0/79439423 learned_addr learned my addr 172.21.2.102:0/79439423 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:04:05.216 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:05.212 7f5773fff700 1 --2- 172.21.2.102:0/79439423 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f57740b8410 0x7f57740b8810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:05.216 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:05.212 7f5779669700 1 -- 172.21.2.102:0/79439423 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f57740b8410 msgr2=0x7f57740b8810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:05.216 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:05.212 7f5779669700 1 --2- 172.21.2.102:0/79439423 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f57740b8410 0x7f57740b8810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:05.216 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:05.212 7f5779669700 1 -- 172.21.2.102:0/79439423 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f577409b930 msgr2=0x7f57740b7c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:05.216 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:05.212 7f5779669700 1 --2- 172.21.2.102:0/79439423 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f577409b930 0x7f57740b7c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:05.216 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:05.212 7f5779669700 1 -- 172.21.2.102:0/79439423 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57740c29a0 con 0x7f57740c1f80 2021-02-10T11:04:05.216 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:05.212 7f5779669700 1 --2- 172.21.2.102:0/79439423 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f57740c1f80 0x7f57740c2380 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7f576c00a5a0 tx=0x7f576c001dc0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:04:05.216 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:05.213 7f57717fa700 1 -- 172.21.2.102:0/79439423 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f576c000fd0 con 0x7f57740c1f80 2021-02-10T11:04:05.216 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:05.213 7f57717fa700 1 -- 172.21.2.102:0/79439423 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f576c0070d0 con 0x7f57740c1f80 2021-02-10T11:04:05.216 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:05.213 7f57717fa700 1 -- 172.21.2.102:0/79439423 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f576c0045b0 con 0x7f57740c1f80 2021-02-10T11:04:05.216 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:05.213 7f577b04c700 1 -- 172.21.2.102:0/79439423 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57740c29a0 con 0x7f57740c1f80 2021-02-10T11:04:05.216 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:05.213 7f577b04c700 1 -- 172.21.2.102:0/79439423 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57740c2f30 con 0x7f57740c1f80 2021-02-10T11:04:05.216 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:05.213 7f57717fa700 1 -- 172.21.2.102:0/79439423 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f576c0043a0 con 0x7f57740c1f80 2021-02-10T11:04:05.216 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:05.214 7f57717fa700 1 --2- 172.21.2.102:0/79439423 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f57580186c0 0x7f575801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:05.217 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:05.214 7f57717fa700 1 -- 172.21.2.102:0/79439423 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f576c032540 con 0x7f57740c1f80 2021-02-10T11:04:05.217 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:05.214 7f5778e68700 1 --2- 172.21.2.102:0/79439423 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f57580186c0 0x7f575801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:05.217 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:05.214 7f5778e68700 1 --2- 172.21.2.102:0/79439423 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f57580186c0 0x7f575801abc0 crc :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:04:05.242 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:05.243 7f577b04c700 1 -- 172.21.2.102:0/79439423 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5774105b30 con 0x7f57740c1f80 2021-02-10T11:04:05.245 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:05.246 7f57717fa700 1 -- 172.21.2.102:0/79439423 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f576c001320 con 0x7f57740c1f80 2021-02-10T11:04:05.372 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:05.373 7f577b04c700 1 -- 172.21.2.102:0/79439423 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7f57740f0b50 con 0x7f57740c1f80 2021-02-10T11:04:05.372 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:05.373 7f57717fa700 1 -- 172.21.2.102:0/79439423 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v20) v1 ==== 104+0+10 (secure 0 0 0) 0x7f576c00aca0 con 0x7f57740c1f80 2021-02-10T11:04:05.373 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:04:05.374 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:05.375 7f577b04c700 1 -- 172.21.2.102:0/79439423 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f57580186c0 msgr2=0x7f575801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:05.374 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:05.375 7f577b04c700 1 --2- 172.21.2.102:0/79439423 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f57580186c0 0x7f575801abc0 crc :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:05.374 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:05.375 7f577b04c700 1 -- 172.21.2.102:0/79439423 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f57740c1f80 msgr2=0x7f57740c2380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:05.374 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:05.375 7f577b04c700 1 --2- 172.21.2.102:0/79439423 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f57740c1f80 0x7f57740c2380 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7f576c00a5a0 tx=0x7f576c001dc0).stop 2021-02-10T11:04:05.375 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:05.375 7f577b04c700 1 -- 172.21.2.102:0/79439423 shutdown_connections 2021-02-10T11:04:05.375 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:05.376 7f577b04c700 1 --2- 172.21.2.102:0/79439423 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f57740c1f80 0x7f57740c2380 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:05.375 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:05.376 7f577b04c700 1 --2- 172.21.2.102:0/79439423 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f57740b8410 0x7f57740b8810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:05.375 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:05.376 7f577b04c700 1 --2- 172.21.2.102:0/79439423 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f57580186c0 0x7f575801abc0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:05.375 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:05.376 7f577b04c700 1 --2- 172.21.2.102:0/79439423 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f577409b930 0x7f57740b7c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:05.375 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:05.376 7f577b04c700 1 -- 172.21.2.102:0/79439423 >> 172.21.2.102:0/79439423 conn(0x7f5774094e10 msgr2=0x7f57740a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:04:05.375 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:05.376 7f577b04c700 1 -- 172.21.2.102:0/79439423 shutdown_connections 2021-02-10T11:04:05.375 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:05.376 7f577b04c700 1 -- 172.21.2.102:0/79439423 wait complete. 2021-02-10T11:04:05.383 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-02-10T11:04:05.813 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:05.798 7fb3c1802700 1 Processor -- start 2021-02-10T11:04:05.813 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:05.798 7fb3c1802700 1 -- start start 2021-02-10T11:04:05.814 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:05.799 7fb3c1802700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb3bc0f05f0 0x7fb3bc0f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:05.814 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:05.799 7fb3c1802700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3bc1b0c70 con 0x7fb3bc1b0470 2021-02-10T11:04:05.814 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:05.799 7fb3c1802700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3bc0f0b50 con 0x7fb3bc1b0870 2021-02-10T11:04:05.814 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:05.799 7fb3c1802700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3bc0e5ea0 con 0x7fb3bc0f05f0 2021-02-10T11:04:05.814 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:05.799 7fb3bb574700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fb3bc1b0470 0x7fb3bc0ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:41870/0 (socket says 172.21.2.102:41870) 2021-02-10T11:04:05.814 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:05.799 7fb3bb574700 1 -- 172.21.2.102:0/3035199943 learned_addr learned my addr 172.21.2.102:0/3035199943 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:04:05.814 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:05.799 7fb3bbd75700 1 --2- 172.21.2.102:0/3035199943 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb3bc0f05f0 0x7fb3bc0f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:05.814 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:05.799 7fb3ba572700 1 -- 172.21.2.102:0/3035199943 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1062404633 0 0) 0x7fb3a40010b0 con 0x7fb3bc1b0470 2021-02-10T11:04:05.814 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:05.799 7fb3ba572700 1 -- 172.21.2.102:0/3035199943 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb3a8002fe0 con 0x7fb3bc1b0470 2021-02-10T11:04:05.814 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:05.799 7fb3bbd75700 1 -- 172.21.2.102:0/3035199943 >> v1:172.21.2.113:6789/0 conn(0x7fb3bc1b0870 legacy=0x7fb3bc0efe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:05.814 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:05.799 7fb3bbd75700 1 -- 172.21.2.102:0/3035199943 >> v1:172.21.2.102:6789/0 conn(0x7fb3bc1b0470 legacy=0x7fb3bc0ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:05.815 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:05.799 7fb3bbd75700 1 -- 172.21.2.102:0/3035199943 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3bc0e52e0 con 0x7fb3bc0f05f0 2021-02-10T11:04:05.815 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:05.799 7fb3bbd75700 1 --2- 172.21.2.102:0/3035199943 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb3bc0f05f0 0x7fb3bc0f20e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7fb3b4006e50 tx=0x7fb3b4009e70).ready entity=mon.2 client_cookie=c53ccfcea1ee17ff server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:04:05.815 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:05.800 7fb3ba572700 1 -- 172.21.2.102:0/3035199943 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb3b4006a20 con 0x7fb3bc0f05f0 2021-02-10T11:04:05.815 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:05.800 7fb3ba572700 1 -- 172.21.2.102:0/3035199943 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3b4006790 con 0x7fb3bc0f05f0 2021-02-10T11:04:05.815 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:05.800 7fb3ba572700 1 -- 172.21.2.102:0/3035199943 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb3b4000f90 con 0x7fb3bc0f05f0 2021-02-10T11:04:05.815 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:05.800 7fb3c1802700 1 -- 172.21.2.102:0/3035199943 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb3bc0f05f0 msgr2=0x7fb3bc0f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:05.815 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:05.800 7fb3c1802700 1 --2- 172.21.2.102:0/3035199943 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb3bc0f05f0 0x7fb3bc0f20e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7fb3b4006e50 tx=0x7fb3b4009e70).stop 2021-02-10T11:04:05.815 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:05.800 7fb3c1802700 1 -- 172.21.2.102:0/3035199943 shutdown_connections 2021-02-10T11:04:05.815 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:05.800 7fb3c1802700 1 --2- 172.21.2.102:0/3035199943 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb3bc0f05f0 0x7fb3bc0f20e0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:05.815 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:05.800 7fb3c1802700 1 -- 172.21.2.102:0/3035199943 >> 172.21.2.102:0/3035199943 conn(0x7fb3bc0ce3e0 msgr2=0x7fb3bc0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:04:05.815 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:05.800 7fb3c1802700 1 -- 172.21.2.102:0/3035199943 shutdown_connections 2021-02-10T11:04:05.815 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:05.800 7fb3c1802700 1 -- 172.21.2.102:0/3035199943 wait complete. 2021-02-10T11:04:05.817 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:05.815 7fb3c1802700 1 Processor -- start 2021-02-10T11:04:05.817 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:05.815 7fb3c1802700 1 -- start start 2021-02-10T11:04:05.817 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:05.815 7fb3c1802700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb3bc0b2500 0x7fb3bc0b2900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:05.817 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:05.815 7fb3c1802700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb3bc0b3080 0x7fb3bc0b3480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:05.817 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:05.815 7fb3c1802700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb3bc0b3aa0 0x7fb3bc0f0530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:05.817 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:05.815 7fb3c1802700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3bc0effb0 con 0x7fb3bc0b2500 2021-02-10T11:04:05.817 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:05.815 7fb3c1802700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3bc0f0b50 con 0x7fb3bc0b3080 2021-02-10T11:04:05.817 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:05.815 7fb3c1802700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3bc0f0d30 con 0x7fb3bc0b3aa0 2021-02-10T11:04:05.817 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:05.815 7fb3bb574700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb3bc0b2500 0x7fb3bc0b2900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:05.818 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:05.816 7fb3bb574700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb3bc0b2500 0x7fb3bc0b2900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:49648/0 (socket says 172.21.2.102:49648) 2021-02-10T11:04:05.818 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:05.816 7fb3bb574700 1 -- 172.21.2.102:0/1267113115 learned_addr learned my addr 172.21.2.102:0/1267113115 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:04:05.818 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:05.816 7fb3bb574700 1 -- 172.21.2.102:0/1267113115 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb3bc0b3aa0 msgr2=0x7fb3bc0f0530 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:04:05.818 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:05.816 7fb3bb574700 1 --2- 172.21.2.102:0/1267113115 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb3bc0b3aa0 0x7fb3bc0f0530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:05.818 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:05.816 7fb3bb574700 1 -- 172.21.2.102:0/1267113115 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb3bc0b3080 msgr2=0x7fb3bc0b3480 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:04:05.818 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:05.816 7fb3bb574700 1 --2- 172.21.2.102:0/1267113115 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb3bc0b3080 0x7fb3bc0b3480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:05.818 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:05.816 7fb3bb574700 1 -- 172.21.2.102:0/1267113115 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3bc0f0f50 con 0x7fb3bc0b2500 2021-02-10T11:04:05.818 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:05.816 7fb3bb574700 1 --2- 172.21.2.102:0/1267113115 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb3bc0b2500 0x7fb3bc0b2900 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7fb3a4001c60 tx=0x7fb3a400a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:04:05.818 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:05.816 7fb3b3fff700 1 -- 172.21.2.102:0/1267113115 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb3a4000e00 con 0x7fb3bc0b2500 2021-02-10T11:04:05.818 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:05.816 7fb3b3fff700 1 -- 172.21.2.102:0/1267113115 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3a400acc0 con 0x7fb3bc0b2500 2021-02-10T11:04:05.818 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:05.816 7fb3b3fff700 1 -- 172.21.2.102:0/1267113115 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb3a40045d0 con 0x7fb3bc0b2500 2021-02-10T11:04:05.818 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:05.816 7fb3c1802700 1 -- 172.21.2.102:0/1267113115 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3bc0f0f50 con 0x7fb3bc0b2500 2021-02-10T11:04:05.819 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:05.816 7fb3c1802700 1 -- 172.21.2.102:0/1267113115 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3bc0f0f50 con 0x7fb3bc0b2500 2021-02-10T11:04:05.819 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:05.817 7fb3b3fff700 1 -- 172.21.2.102:0/1267113115 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fb3a4004140 con 0x7fb3bc0b2500 2021-02-10T11:04:05.819 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:05.817 7fb3b3fff700 1 --2- 172.21.2.102:0/1267113115 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb3a00186c0 0x7fb3a001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:05.819 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:05.817 7fb3b3fff700 1 -- 172.21.2.102:0/1267113115 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fb3a40323d0 con 0x7fb3bc0b2500 2021-02-10T11:04:05.819 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:05.817 7fb3bad73700 1 --2- 172.21.2.102:0/1267113115 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb3a00186c0 0x7fb3a001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:05.819 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:05.817 7fb3bad73700 1 --2- 172.21.2.102:0/1267113115 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb3a00186c0 0x7fb3a001abc0 crc :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:04:05.848 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:05.848 7fb3c1802700 1 -- 172.21.2.102:0/1267113115 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3bc0edd10 con 0x7fb3bc0b2500 2021-02-10T11:04:05.850 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:05.851 7fb3b3fff700 1 -- 172.21.2.102:0/1267113115 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fb3a4028bc0 con 0x7fb3bc0b2500 2021-02-10T11:04:05.978 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:05.979 7fb3c1802700 1 -- 172.21.2.102:0/1267113115 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7fb3bc0e52e0 con 0x7fb3bc0b2500 2021-02-10T11:04:05.978 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:05.979 7fb3b3fff700 1 -- 172.21.2.102:0/1267113115 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v20) v1 ==== 100+0+10 (secure 0 0 0) 0x7fb3a4028bc0 con 0x7fb3bc0b2500 2021-02-10T11:04:05.979 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:04:05.981 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:05.981 7fb3c1802700 1 -- 172.21.2.102:0/1267113115 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb3a00186c0 msgr2=0x7fb3a001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:05.981 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:05.981 7fb3c1802700 1 --2- 172.21.2.102:0/1267113115 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb3a00186c0 0x7fb3a001abc0 crc :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:05.981 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:05.981 7fb3c1802700 1 -- 172.21.2.102:0/1267113115 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb3bc0b2500 msgr2=0x7fb3bc0b2900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:05.981 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:05.981 7fb3c1802700 1 --2- 172.21.2.102:0/1267113115 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb3bc0b2500 0x7fb3bc0b2900 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7fb3a4001c60 tx=0x7fb3a400a8d0).stop 2021-02-10T11:04:05.981 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:05.982 7fb3c1802700 1 -- 172.21.2.102:0/1267113115 shutdown_connections 2021-02-10T11:04:05.981 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:05.982 7fb3c1802700 1 --2- 172.21.2.102:0/1267113115 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb3bc0b3aa0 0x7fb3bc0f0530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:05.981 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:05.982 7fb3c1802700 1 --2- 172.21.2.102:0/1267113115 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb3a00186c0 0x7fb3a001abc0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:05.981 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:05.982 7fb3c1802700 1 --2- 172.21.2.102:0/1267113115 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb3bc0b3080 0x7fb3bc0b3480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:05.981 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:05.982 7fb3c1802700 1 --2- 172.21.2.102:0/1267113115 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb3bc0b2500 0x7fb3bc0b2900 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:05.981 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:05.982 7fb3c1802700 1 -- 172.21.2.102:0/1267113115 >> 172.21.2.102:0/1267113115 conn(0x7fb3bc0c3c70 msgr2=0x7fb3bc094c30 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:04:05.981 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:05.982 7fb3c1802700 1 -- 172.21.2.102:0/1267113115 shutdown_connections 2021-02-10T11:04:05.982 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:05.982 7fb3c1802700 1 -- 172.21.2.102:0/1267113115 wait complete. 2021-02-10T11:04:05.989 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:04:06.415 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:06.400 7f93ecd9e700 1 Processor -- start 2021-02-10T11:04:06.415 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:06.400 7f93ecd9e700 1 -- start start 2021-02-10T11:04:06.415 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:06.400 7f93ecd9e700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f93e80e5070 0x7f93e81b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:06.415 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:06.400 7f93ecd9e700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f93e81b3e40 0x7f93e807d690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:06.415 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:06.400 7f93ecd9e700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93e807ed80 con 0x7f93e80e5470 2021-02-10T11:04:06.416 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:06.400 7f93ecd9e700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93e8080310 con 0x7f93e80e5070 2021-02-10T11:04:06.416 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:06.400 7f93ecd9e700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93e8080500 con 0x7f93e81b3e40 2021-02-10T11:04:06.416 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:06.401 7f93e6b13700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f93e80e5070 0x7f93e81b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:06.416 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:06.401 7f93e6b13700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f93e80e5070 0x7f93e81b3820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:49656/0 (socket says 172.21.2.102:49656) 2021-02-10T11:04:06.416 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:06.401 7f93e6b13700 1 -- 172.21.2.102:0/176865521 learned_addr learned my addr 172.21.2.102:0/176865521 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:04:06.416 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:06.401 7f93e6312700 1 --2- 172.21.2.102:0/176865521 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f93e81b3e40 0x7f93e807d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:06.416 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:06.401 7f93e6b13700 1 -- 172.21.2.102:0/176865521 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f93e81b3e40 msgr2=0x7f93e807d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:04:06.416 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:06.401 7f93e6b13700 1 --2- 172.21.2.102:0/176865521 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f93e81b3e40 0x7f93e807d690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:06.416 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:06.401 7f93e6b13700 1 -- 172.21.2.102:0/176865521 >> v1:172.21.2.113:6789/0 conn(0x7f93e80e5470 legacy=0x7f93e807dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:04:06.416 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:06.401 7f93e6b13700 1 -- 172.21.2.102:0/176865521 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93e80806f0 con 0x7f93e80e5070 2021-02-10T11:04:06.416 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:06.401 7f93e6b13700 1 --2- 172.21.2.102:0/176865521 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f93e80e5070 0x7f93e81b3820 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7f93d0006700 tx=0x7f93d0009b80).ready entity=mon.0 client_cookie=dbc74f338527b150 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:04:06.416 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:06.401 7f93e5b11700 1 -- 172.21.2.102:0/176865521 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f93d000c0c0 con 0x7f93e80e5070 2021-02-10T11:04:06.416 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:06.401 7f93e5b11700 1 -- 172.21.2.102:0/176865521 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93d000c340 con 0x7f93e80e5070 2021-02-10T11:04:06.417 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:06.401 7f93e5b11700 1 -- 172.21.2.102:0/176865521 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f93d000c860 con 0x7f93e80e5070 2021-02-10T11:04:06.417 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:06.401 7f93ecd9e700 1 -- 172.21.2.102:0/176865521 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f93e80e5070 msgr2=0x7f93e81b3820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:06.417 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:06.401 7f93ecd9e700 1 --2- 172.21.2.102:0/176865521 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f93e80e5070 0x7f93e81b3820 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7f93d0006700 tx=0x7f93d0009b80).stop 2021-02-10T11:04:06.417 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:06.402 7f93ecd9e700 1 -- 172.21.2.102:0/176865521 shutdown_connections 2021-02-10T11:04:06.417 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:06.402 7f93ecd9e700 1 --2- 172.21.2.102:0/176865521 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f93e81b3e40 0x7f93e807d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:06.417 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:06.402 7f93ecd9e700 1 --2- 172.21.2.102:0/176865521 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f93e80e5070 0x7f93e81b3820 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:06.417 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:06.402 7f93ecd9e700 1 -- 172.21.2.102:0/176865521 >> 172.21.2.102:0/176865521 conn(0x7f93e80ce690 msgr2=0x7f93e80e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:04:06.417 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:06.402 7f93ecd9e700 1 -- 172.21.2.102:0/176865521 shutdown_connections 2021-02-10T11:04:06.417 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:06.402 7f93ecd9e700 1 -- 172.21.2.102:0/176865521 wait complete. 2021-02-10T11:04:06.418 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:06.417 7f93ecd9e700 1 Processor -- start 2021-02-10T11:04:06.419 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:06.417 7f93ecd9e700 1 -- start start 2021-02-10T11:04:06.419 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:06.417 7f93ecd9e700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f93e808d400 0x7f93e8081230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:06.419 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:06.417 7f93ecd9e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f93e8081850 0x7f93e80acf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:06.419 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:06.417 7f93ecd9e700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f93e8081c50 0x7f93e80ad5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:06.419 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:06.417 7f93ecd9e700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93e8082110 con 0x7f93e8081850 2021-02-10T11:04:06.419 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:06.417 7f93ecd9e700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93e808d800 con 0x7f93e8081c50 2021-02-10T11:04:06.419 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:06.417 7f93ecd9e700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93e80adbc0 con 0x7f93e808d400 2021-02-10T11:04:06.419 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:06.417 7f93e6312700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f93e8081850 0x7f93e80acf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:06.419 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:06.417 7f93e6312700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f93e8081850 0x7f93e80acf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:49662/0 (socket says 172.21.2.102:49662) 2021-02-10T11:04:06.419 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:06.417 7f93e6312700 1 -- 172.21.2.102:0/1007570927 learned_addr learned my addr 172.21.2.102:0/1007570927 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:04:06.419 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:06.417 7f93e7314700 1 --2- 172.21.2.102:0/1007570927 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f93e8081c50 0x7f93e80ad5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:06.419 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:06.417 7f93e6b13700 1 --2- 172.21.2.102:0/1007570927 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f93e808d400 0x7f93e8081230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:06.419 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:06.418 7f93e6312700 1 -- 172.21.2.102:0/1007570927 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f93e808d400 msgr2=0x7f93e8081230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:06.420 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:06.418 7f93e6312700 1 --2- 172.21.2.102:0/1007570927 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f93e808d400 0x7f93e8081230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:06.420 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:06.418 7f93e6312700 1 -- 172.21.2.102:0/1007570927 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f93e8081c50 msgr2=0x7f93e80ad5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:06.420 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:06.418 7f93e6312700 1 --2- 172.21.2.102:0/1007570927 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f93e8081c50 0x7f93e80ad5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:06.420 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:06.418 7f93e6312700 1 -- 172.21.2.102:0/1007570927 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93e8187fc0 con 0x7f93e8081850 2021-02-10T11:04:06.420 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:06.418 7f93e6312700 1 --2- 172.21.2.102:0/1007570927 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f93e8081850 0x7f93e80acf80 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7f93d8006f60 tx=0x7f93d800b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:04:06.420 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:06.418 7f93df7fe700 1 -- 172.21.2.102:0/1007570927 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f93d80009b0 con 0x7f93e8081850 2021-02-10T11:04:06.420 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:06.418 7f93df7fe700 1 -- 172.21.2.102:0/1007570927 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93d800bdf0 con 0x7f93e8081850 2021-02-10T11:04:06.420 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:06.418 7f93df7fe700 1 -- 172.21.2.102:0/1007570927 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f93d8002030 con 0x7f93e8081850 2021-02-10T11:04:06.420 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:06.418 7f93ecd9e700 1 -- 172.21.2.102:0/1007570927 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93e8187fc0 con 0x7f93e8081850 2021-02-10T11:04:06.420 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:06.418 7f93ecd9e700 1 -- 172.21.2.102:0/1007570927 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93e8187fc0 con 0x7f93e8081850 2021-02-10T11:04:06.420 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:06.419 7f93df7fe700 1 -- 172.21.2.102:0/1007570927 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f93d8006b00 con 0x7f93e8081850 2021-02-10T11:04:06.420 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:06.419 7f93df7fe700 1 --2- 172.21.2.102:0/1007570927 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f93cc0186c0 0x7f93cc01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:06.420 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:06.419 7f93df7fe700 1 -- 172.21.2.102:0/1007570927 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f93d80327b0 con 0x7f93e8081850 2021-02-10T11:04:06.421 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:06.419 7f93e6b13700 1 --2- 172.21.2.102:0/1007570927 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f93cc0186c0 0x7f93cc01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:06.421 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:06.419 7f93e6b13700 1 --2- 172.21.2.102:0/1007570927 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f93cc0186c0 0x7f93cc01abc0 crc :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:04:06.450 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:06.451 7f93ecd9e700 1 -- 172.21.2.102:0/1007570927 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93e808d800 con 0x7f93e8081850 2021-02-10T11:04:06.452 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:06.453 7f93df7fe700 1 -- 172.21.2.102:0/1007570927 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f93d8006340 con 0x7f93e8081850 2021-02-10T11:04:06.582 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:06.583 7f93ecd9e700 1 -- 172.21.2.102:0/1007570927 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f93e808d800 con 0x7f93e8081850 2021-02-10T11:04:06.582 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:06.583 7f93df7fe700 1 -- 172.21.2.102:0/1007570927 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+2094 (secure 0 0 0) 0x7f93d8006b00 con 0x7f93e8081850 2021-02-10T11:04:06.583 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:04:06.583 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":6,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4302,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.102:6835/3297596596","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6834","nonce":3297596596},{"type":"v1","addr":"172.21.2.102:6835","nonce":3297596596}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":4}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:03:43.384709","modified":"2021-02-10 11:03:52.791414","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4226},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4226":{"gid":4226,"name":"b","rank":0,"incarnation":5,"state":"up:active","state_seq":3,"addr":"172.21.2.113:6833/285807165","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6832","nonce":285807165},{"type":"v1","addr":"172.21.2.113:6833","nonce":285807165}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2021-02-10T11:04:06.583 INFO:teuthology.orchestra.run.gibba002.stderr:dumped fsmap epoch 6 2021-02-10T11:04:06.585 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:06.585 7f93ecd9e700 1 -- 172.21.2.102:0/1007570927 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f93cc0186c0 msgr2=0x7f93cc01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:06.585 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:06.585 7f93ecd9e700 1 --2- 172.21.2.102:0/1007570927 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f93cc0186c0 0x7f93cc01abc0 crc :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:06.585 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:06.585 7f93ecd9e700 1 -- 172.21.2.102:0/1007570927 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f93e8081850 msgr2=0x7f93e80acf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:06.585 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:06.585 7f93ecd9e700 1 --2- 172.21.2.102:0/1007570927 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f93e8081850 0x7f93e80acf80 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7f93d8006f60 tx=0x7f93d800b970).stop 2021-02-10T11:04:06.585 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:06.585 7f93ecd9e700 1 -- 172.21.2.102:0/1007570927 shutdown_connections 2021-02-10T11:04:06.585 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:06.585 7f93ecd9e700 1 --2- 172.21.2.102:0/1007570927 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f93cc0186c0 0x7f93cc01abc0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:06.585 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:06.585 7f93ecd9e700 1 --2- 172.21.2.102:0/1007570927 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f93e8081c50 0x7f93e80ad5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:06.585 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:06.585 7f93ecd9e700 1 --2- 172.21.2.102:0/1007570927 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f93e8081850 0x7f93e80acf80 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:06.585 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:06.585 7f93ecd9e700 1 --2- 172.21.2.102:0/1007570927 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f93e808d400 0x7f93e8081230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:06.586 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:06.585 7f93ecd9e700 1 -- 172.21.2.102:0/1007570927 >> 172.21.2.102:0/1007570927 conn(0x7f93e81ac620 msgr2=0x7f93e80896e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:04:06.586 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:06.585 7f93ecd9e700 1 -- 172.21.2.102:0/1007570927 shutdown_connections 2021-02-10T11:04:06.586 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:06.585 7f93ecd9e700 1 -- 172.21.2.102:0/1007570927 wait complete. 2021-02-10T11:04:06.794 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 6, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:03:43.384709', 'modified': '2021-02-10 11:03:52.791414', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4226}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4226': {'gid': 4226, 'name': 'b', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.113:6833/285807165', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.113:6832', 'nonce': 285807165}, {'type': 'v1', 'addr': '172.21.2.113:6833', 'nonce': 285807165}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:04:06.795 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:04:06.795 DEBUG:teuthology.orchestra.run.gibba013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:04:07.193 INFO:teuthology.orchestra.run.gibba013.stdout:{ 2021-02-10T11:04:07.194 INFO:teuthology.orchestra.run.gibba013.stdout: "cluster_fsid": "68db30ea-dbce-4aef-a6df-f01c6fea0513", 2021-02-10T11:04:07.194 INFO:teuthology.orchestra.run.gibba013.stdout: "whoami": 0, 2021-02-10T11:04:07.194 INFO:teuthology.orchestra.run.gibba013.stdout: "id": 4226, 2021-02-10T11:04:07.194 INFO:teuthology.orchestra.run.gibba013.stdout: "want_state": "up:active", 2021-02-10T11:04:07.194 INFO:teuthology.orchestra.run.gibba013.stdout: "state": "up:active", 2021-02-10T11:04:07.194 INFO:teuthology.orchestra.run.gibba013.stdout: "rank_uptime": 15.403986886, 2021-02-10T11:04:07.194 INFO:teuthology.orchestra.run.gibba013.stdout: "mdsmap_epoch": 6, 2021-02-10T11:04:07.194 INFO:teuthology.orchestra.run.gibba013.stdout: "osdmap_epoch": 20, 2021-02-10T11:04:07.194 INFO:teuthology.orchestra.run.gibba013.stdout: "osdmap_epoch_barrier": 20, 2021-02-10T11:04:07.194 INFO:teuthology.orchestra.run.gibba013.stdout: "uptime": 19.798422312 2021-02-10T11:04:07.195 INFO:teuthology.orchestra.run.gibba013.stdout:} 2021-02-10T11:04:07.198 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "68db30ea-dbce-4aef-a6df-f01c6fea0513", "whoami": 0, "id": 4226, "want_state": "up:active", "state": "up:active", "rank_uptime": 15.403986886, "mdsmap_epoch": 6, "osdmap_epoch": 20, "osdmap_epoch_barrier": 20, "uptime": 19.798422312 } 2021-02-10T11:04:07.198 INFO:teuthology.run_tasks:Running task ceph-fuse... 2021-02-10T11:04:07.236 INFO:tasks.ceph_fuse:Running ceph_fuse task... 2021-02-10T11:04:07.236 INFO:tasks.ceph_fuse:config is None 2021-02-10T11:04:07.236 INFO:tasks.ceph_fuse:new config is {'client.0': None, 'client.1': None, 'client.2': None, 'client.3': None} 2021-02-10T11:04:07.236 INFO:tasks.ceph_fuse:Mounting ceph-fuse clients... 2021-02-10T11:04:07.237 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:04:07.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:07.647 7f7274870700 1 Processor -- start 2021-02-10T11:04:07.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:07.647 7f7274870700 1 -- start start 2021-02-10T11:04:07.663 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:07.648 7f7274870700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f72700e5070 0x7f72701b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:07.663 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:07.648 7f7274870700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f72701b3e40 0x7f7270080310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:07.663 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:07.648 7f7274870700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f727007ed80 con 0x7f72700e5470 2021-02-10T11:04:07.663 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:07.648 7f7274870700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72700d1790 con 0x7f72701b3e40 2021-02-10T11:04:07.663 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:07.648 7f7274870700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72700d1980 con 0x7f72700e5070 2021-02-10T11:04:07.663 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:07.648 7f726ee1d700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f72701b3e40 0x7f7270080310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:07.663 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:07.648 7f726ee1d700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f72701b3e40 0x7f7270080310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:49668/0 (socket says 172.21.2.102:49668) 2021-02-10T11:04:07.663 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:07.648 7f726ee1d700 1 -- 172.21.2.102:0/798284640 learned_addr learned my addr 172.21.2.102:0/798284640 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:04:07.663 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:07.648 7f726ee1d700 1 -- 172.21.2.102:0/798284640 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f72700e5070 msgr2=0x7f72701b3820 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:04:07.663 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:07.648 7f726ee1d700 1 --2- 172.21.2.102:0/798284640 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f72700e5070 0x7f72701b3820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:07.663 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:07.648 7f726ee1d700 1 -- 172.21.2.102:0/798284640 >> v1:172.21.2.113:6789/0 conn(0x7f72700e5470 legacy=0x7f727007d690 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:04:07.663 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:07.648 7f726ee1d700 1 -- 172.21.2.102:0/798284640 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72700d1ba0 con 0x7f72701b3e40 2021-02-10T11:04:07.663 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:07.648 7f726ee1d700 1 --2- 172.21.2.102:0/798284640 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f72701b3e40 0x7f7270080310 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f7268006f00 tx=0x7f7268009f10).ready entity=mon.0 client_cookie=aadf5ebcb20e9473 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:04:07.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:07.648 7f726d61a700 1 -- 172.21.2.102:0/798284640 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f726800b3b0 con 0x7f72701b3e40 2021-02-10T11:04:07.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:07.648 7f726d61a700 1 -- 172.21.2.102:0/798284640 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f726800b730 con 0x7f72701b3e40 2021-02-10T11:04:07.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:07.648 7f726d61a700 1 -- 172.21.2.102:0/798284640 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f726800bc50 con 0x7f72701b3e40 2021-02-10T11:04:07.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:07.649 7f7274870700 1 -- 172.21.2.102:0/798284640 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f72701b3e40 msgr2=0x7f7270080310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:07.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:07.649 7f7274870700 1 --2- 172.21.2.102:0/798284640 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f72701b3e40 0x7f7270080310 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f7268006f00 tx=0x7f7268009f10).stop 2021-02-10T11:04:07.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:07.649 7f7274870700 1 -- 172.21.2.102:0/798284640 shutdown_connections 2021-02-10T11:04:07.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:07.649 7f7274870700 1 --2- 172.21.2.102:0/798284640 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f72701b3e40 0x7f7270080310 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:07.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:07.649 7f7274870700 1 --2- 172.21.2.102:0/798284640 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f72700e5070 0x7f72701b3820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:07.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:07.649 7f7274870700 1 -- 172.21.2.102:0/798284640 >> 172.21.2.102:0/798284640 conn(0x7f72700ce690 msgr2=0x7f72700e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:04:07.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:07.649 7f7274870700 1 -- 172.21.2.102:0/798284640 shutdown_connections 2021-02-10T11:04:07.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:07.649 7f7274870700 1 -- 172.21.2.102:0/798284640 wait complete. 2021-02-10T11:04:07.666 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:07.664 7f7274870700 1 Processor -- start 2021-02-10T11:04:07.666 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:07.664 7f7274870700 1 -- start start 2021-02-10T11:04:07.666 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:07.664 7f7274870700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f72701b19d0 0x7f727008cd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:07.666 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:07.664 7f7274870700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f72701b0d00 0x7f72701b1100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:07.666 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:07.664 7f7274870700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f72701a2b50 0x7f72701a2f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:07.666 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:07.664 7f7274870700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72701b1720 con 0x7f72701b0d00 2021-02-10T11:04:07.667 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:07.664 7f7274870700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f727008c470 con 0x7f72701a2b50 2021-02-10T11:04:07.667 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:07.664 7f7274870700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72701b1f50 con 0x7f72701b19d0 2021-02-10T11:04:07.667 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:07.664 7f726de1b700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f72701b0d00 0x7f72701b1100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:07.667 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:07.665 7f726de1b700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f72701b0d00 0x7f72701b1100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:49674/0 (socket says 172.21.2.102:49674) 2021-02-10T11:04:07.667 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:07.665 7f726de1b700 1 -- 172.21.2.102:0/2837489159 learned_addr learned my addr 172.21.2.102:0/2837489159 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:04:07.667 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:07.665 7f726de1b700 1 -- 172.21.2.102:0/2837489159 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f72701b19d0 msgr2=0x7f727008cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:07.667 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:07.665 7f726de1b700 1 --2- 172.21.2.102:0/2837489159 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f72701b19d0 0x7f727008cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:07.667 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:07.665 7f726de1b700 1 -- 172.21.2.102:0/2837489159 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f72701a2b50 msgr2=0x7f72701a2f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:07.667 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:07.665 7f726de1b700 1 --2- 172.21.2.102:0/2837489159 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f72701a2b50 0x7f72701a2f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:07.667 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:07.665 7f726de1b700 1 -- 172.21.2.102:0/2837489159 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72701a3570 con 0x7f72701b0d00 2021-02-10T11:04:07.667 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:07.665 7f726de1b700 1 --2- 172.21.2.102:0/2837489159 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f72701b0d00 0x7f72701b1100 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7f726400b5b0 tx=0x7f726400b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:04:07.667 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:07.665 7f725effd700 1 -- 172.21.2.102:0/2837489159 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f726400bd40 con 0x7f72701b0d00 2021-02-10T11:04:07.667 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:07.665 7f725effd700 1 -- 172.21.2.102:0/2837489159 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7264006020 con 0x7f72701b0d00 2021-02-10T11:04:07.667 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:07.665 7f7274870700 1 -- 172.21.2.102:0/2837489159 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72701a3570 con 0x7f72701b0d00 2021-02-10T11:04:07.668 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:07.665 7f725effd700 1 -- 172.21.2.102:0/2837489159 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f72640054a0 con 0x7f72701b0d00 2021-02-10T11:04:07.668 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:07.665 7f7274870700 1 -- 172.21.2.102:0/2837489159 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72701a3570 con 0x7f72701b0d00 2021-02-10T11:04:07.668 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:07.666 7f725effd700 1 -- 172.21.2.102:0/2837489159 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f7264006200 con 0x7f72701b0d00 2021-02-10T11:04:07.668 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:07.666 7f725effd700 1 --2- 172.21.2.102:0/2837489159 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7254018730 0x7f725401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:07.668 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:07.666 7f725effd700 1 -- 172.21.2.102:0/2837489159 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f7264032630 con 0x7f72701b0d00 2021-02-10T11:04:07.668 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:07.666 7f726e61c700 1 --2- 172.21.2.102:0/2837489159 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7254018730 0x7f725401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:07.668 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:07.667 7f726e61c700 1 --2- 172.21.2.102:0/2837489159 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7254018730 0x7f725401ac10 crc :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:04:07.697 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:07.698 7f7274870700 1 -- 172.21.2.102:0/2837489159 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72701b1720 con 0x7f72701b0d00 2021-02-10T11:04:07.699 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:07.700 7f725effd700 1 -- 172.21.2.102:0/2837489159 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f726400bd40 con 0x7f72701b0d00 2021-02-10T11:04:07.822 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:07.823 7f7274870700 1 -- 172.21.2.102:0/2837489159 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f72701b1720 con 0x7f72701b0d00 2021-02-10T11:04:07.822 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:07.824 7f725effd700 1 -- 172.21.2.102:0/2837489159 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+13993 (secure 0 0 0) 0x7f7264006000 con 0x7f72701b0d00 2021-02-10T11:04:07.823 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:04:07.823 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":20,"fsid":"68db30ea-dbce-4aef-a6df-f01c6fea0513","created":"2021-02-10 11:03:06.331557","modified":"2021-02-10 11:03:44.386945","last_up_change":"2021-02-10 11:03:32.690562","last_in_change":"2021-02-10 11:03:32.690562","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:03:37.322519","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:03:41.161563","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2021-02-10 11:03:42.379651","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"be691b4d-a039-408a-af80-9d53f02b6f1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6818","nonce":36088},{"type":"v1","addr":"172.21.2.102:6820","nonce":36088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6822","nonce":36088},{"type":"v1","addr":"172.21.2.102:6824","nonce":36088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6830","nonce":36088},{"type":"v1","addr":"172.21.2.102:6832","nonce":36088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6826","nonce":36088},{"type":"v1","addr":"172.21.2.102:6828","nonce":36088}]},"public_addr":"172.21.2.102:6820/36088","cluster_addr":"172.21.2.102:6824/36088","heartbeat_back_addr":"172.21.2.102:6832/36088","heartbeat_front_addr":"172.21.2.102:6828/36088","state":["exists","up"]},{"osd":1,"uuid":"64e3ccc8-9c69-4d7c-865e-af8e83822635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6819","nonce":36087},{"type":"v1","addr":"172.21.2.102:6821","nonce":36087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6823","nonce":36087},{"type":"v1","addr":"172.21.2.102:6825","nonce":36087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6831","nonce":36087},{"type":"v1","addr":"172.21.2.102:6833","nonce":36087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6827","nonce":36087},{"type":"v1","addr":"172.21.2.102:6829","nonce":36087}]},"public_addr":"172.21.2.102:6821/36087","cluster_addr":"172.21.2.102:6825/36087","heartbeat_back_addr":"172.21.2.102:6833/36087","heartbeat_front_addr":"172.21.2.102:6829/36087","state":["exists","up"]},{"osd":2,"uuid":"153700af-747f-481f-bdf4-d6072d3a842f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6810","nonce":36089},{"type":"v1","addr":"172.21.2.102:6811","nonce":36089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6812","nonce":36089},{"type":"v1","addr":"172.21.2.102:6813","nonce":36089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6816","nonce":36089},{"type":"v1","addr":"172.21.2.102:6817","nonce":36089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6814","nonce":36089},{"type":"v1","addr":"172.21.2.102:6815","nonce":36089}]},"public_addr":"172.21.2.102:6811/36089","cluster_addr":"172.21.2.102:6813/36089","heartbeat_back_addr":"172.21.2.102:6817/36089","heartbeat_front_addr":"172.21.2.102:6815/36089","state":["exists","up"]},{"osd":3,"uuid":"cbc678f4-83a9-45e6-8830-2f7c232b1ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6802","nonce":36090},{"type":"v1","addr":"172.21.2.102:6803","nonce":36090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6804","nonce":36090},{"type":"v1","addr":"172.21.2.102:6805","nonce":36090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6808","nonce":36090},{"type":"v1","addr":"172.21.2.102:6809","nonce":36090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6806","nonce":36090},{"type":"v1","addr":"172.21.2.102:6807","nonce":36090}]},"public_addr":"172.21.2.102:6803/36090","cluster_addr":"172.21.2.102:6805/36090","heartbeat_back_addr":"172.21.2.102:6809/36090","heartbeat_front_addr":"172.21.2.102:6807/36090","state":["exists","up"]},{"osd":4,"uuid":"c01e4008-b258-4f10-afd2-9e7f37214351","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6808","nonce":36368},{"type":"v1","addr":"172.21.2.113:6811","nonce":36368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6814","nonce":36368},{"type":"v1","addr":"172.21.2.113:6818","nonce":36368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6826","nonce":36368},{"type":"v1","addr":"172.21.2.113:6828","nonce":36368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6820","nonce":36368},{"type":"v1","addr":"172.21.2.113:6824","nonce":36368}]},"public_addr":"172.21.2.113:6811/36368","cluster_addr":"172.21.2.113:6818/36368","heartbeat_back_addr":"172.21.2.113:6828/36368","heartbeat_front_addr":"172.21.2.113:6824/36368","state":["exists","up"]},{"osd":5,"uuid":"551f6007-daf8-4a98-8ee0-c2b16bcccb5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6802","nonce":36366},{"type":"v1","addr":"172.21.2.113:6805","nonce":36366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6807","nonce":36366},{"type":"v1","addr":"172.21.2.113:6810","nonce":36366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6819","nonce":36366},{"type":"v1","addr":"172.21.2.113:6822","nonce":36366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6812","nonce":36366},{"type":"v1","addr":"172.21.2.113:6815","nonce":36366}]},"public_addr":"172.21.2.113:6805/36366","cluster_addr":"172.21.2.113:6810/36366","heartbeat_back_addr":"172.21.2.113:6822/36366","heartbeat_front_addr":"172.21.2.113:6815/36366","state":["exists","up"]},{"osd":6,"uuid":"bf67c040-0e50-48be-b102-c3943bc9ebef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6800","nonce":36367},{"type":"v1","addr":"172.21.2.113:6801","nonce":36367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6803","nonce":36367},{"type":"v1","addr":"172.21.2.113:6804","nonce":36367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6813","nonce":36367},{"type":"v1","addr":"172.21.2.113:6816","nonce":36367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6806","nonce":36367},{"type":"v1","addr":"172.21.2.113:6809","nonce":36367}]},"public_addr":"172.21.2.113:6801/36367","cluster_addr":"172.21.2.113:6804/36367","heartbeat_back_addr":"172.21.2.113:6816/36367","heartbeat_front_addr":"172.21.2.113:6809/36367","state":["exists","up"]},{"osd":7,"uuid":"f5461f80-018d-428f-be89-2eb2bcbe7bb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6817","nonce":36365},{"type":"v1","addr":"172.21.2.113:6821","nonce":36365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6823","nonce":36365},{"type":"v1","addr":"172.21.2.113:6825","nonce":36365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6830","nonce":36365},{"type":"v1","addr":"172.21.2.113:6831","nonce":36365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6827","nonce":36365},{"type":"v1","addr":"172.21.2.113:6829","nonce":36365}]},"public_addr":"172.21.2.113:6821/36365","cluster_addr":"172.21.2.113:6825/36365","heartbeat_back_addr":"172.21.2.113:6831/36365","heartbeat_front_addr":"172.21.2.113:6829/36365","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:04:07.824 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:07.826 7f7274870700 1 -- 172.21.2.102:0/2837489159 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7254018730 msgr2=0x7f725401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:07.825 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:07.826 7f7274870700 1 --2- 172.21.2.102:0/2837489159 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7254018730 0x7f725401ac10 crc :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:07.825 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:07.826 7f7274870700 1 -- 172.21.2.102:0/2837489159 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f72701b0d00 msgr2=0x7f72701b1100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:07.825 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:07.826 7f7274870700 1 --2- 172.21.2.102:0/2837489159 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f72701b0d00 0x7f72701b1100 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7f726400b5b0 tx=0x7f726400b950).stop 2021-02-10T11:04:07.825 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:07.826 7f7274870700 1 -- 172.21.2.102:0/2837489159 shutdown_connections 2021-02-10T11:04:07.825 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:07.826 7f7274870700 1 --2- 172.21.2.102:0/2837489159 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7254018730 0x7f725401ac10 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:07.825 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:07.826 7f7274870700 1 --2- 172.21.2.102:0/2837489159 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f72701a2b50 0x7f72701a2f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:07.825 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:07.826 7f7274870700 1 --2- 172.21.2.102:0/2837489159 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f72701b0d00 0x7f72701b1100 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:07.825 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:07.826 7f7274870700 1 --2- 172.21.2.102:0/2837489159 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f72701b19d0 0x7f727008cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:07.825 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:07.826 7f7274870700 1 -- 172.21.2.102:0/2837489159 >> 172.21.2.102:0/2837489159 conn(0x7f7270093720 msgr2=0x7f7270084b80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:04:07.826 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:07.826 7f7274870700 1 -- 172.21.2.102:0/2837489159 shutdown_connections 2021-02-10T11:04:07.826 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:07.826 7f7274870700 1 -- 172.21.2.102:0/2837489159 wait complete. 2021-02-10T11:04:07.833 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:03:37.322519', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2021-02-10 11:03:41.161563', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2021-02-10 11:03:42.379651', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-02-10T11:04:07.833 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:04:08.263 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:08.248 7f595355b700 1 Processor -- start 2021-02-10T11:04:08.264 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:08.248 7f595355b700 1 -- start start 2021-02-10T11:04:08.264 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:08.248 7f595355b700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f594c1b0470 0x7f594c1b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:08.264 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:08.248 7f595355b700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f594c0f20e0 0x7f594c0f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:08.264 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:08.248 7f595355b700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f594c0f0b50 con 0x7f594c0ef610 2021-02-10T11:04:08.264 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:08.248 7f595355b700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f594c0f0820 con 0x7f594c0f20e0 2021-02-10T11:04:08.264 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:08.248 7f595355b700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f594c0f24e0 con 0x7f594c1b0470 2021-02-10T11:04:08.264 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:08.248 7f5951b78700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f594c0f20e0 0x7f594c0f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:08.264 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:08.249 7f5951b78700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f594c0f20e0 0x7f594c0f0200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:49682/0 (socket says 172.21.2.102:49682) 2021-02-10T11:04:08.264 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:08.249 7f5951b78700 1 -- 172.21.2.102:0/26844539 learned_addr learned my addr 172.21.2.102:0/26844539 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:04:08.264 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:08.249 7f5950b76700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f594c0ef610 0x7f594c0efa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:41906/0 (socket says 172.21.2.102:41906) 2021-02-10T11:04:08.264 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:08.249 7f5951b78700 1 -- 172.21.2.102:0/26844539 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f594c1b0470 msgr2=0x7f594c1b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:04:08.265 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:08.249 7f5951b78700 1 --2- 172.21.2.102:0/26844539 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f594c1b0470 0x7f594c1b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:08.265 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:08.249 7f5951b78700 1 -- 172.21.2.102:0/26844539 >> v1:172.21.2.102:6789/0 conn(0x7f594c0ef610 legacy=0x7f594c0efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:08.265 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:08.249 7f5951b78700 1 -- 172.21.2.102:0/26844539 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f594c0e52e0 con 0x7f594c0f20e0 2021-02-10T11:04:08.265 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:08.249 7f5951b78700 1 --2- 172.21.2.102:0/26844539 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f594c0f20e0 0x7f594c0f0200 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7f5944006f00 tx=0x7f5944009f10).ready entity=mon.0 client_cookie=f3a916b4f582fb64 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:04:08.265 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:08.249 7f593ffff700 1 -- 172.21.2.102:0/26844539 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f594400b3b0 con 0x7f594c0f20e0 2021-02-10T11:04:08.265 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:08.249 7f593ffff700 1 -- 172.21.2.102:0/26844539 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f594400b730 con 0x7f594c0f20e0 2021-02-10T11:04:08.265 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:08.249 7f593ffff700 1 -- 172.21.2.102:0/26844539 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f594400bb90 con 0x7f594c0f20e0 2021-02-10T11:04:08.265 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:08.249 7f595355b700 1 -- 172.21.2.102:0/26844539 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f594c0f20e0 msgr2=0x7f594c0f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:08.265 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:08.249 7f595355b700 1 --2- 172.21.2.102:0/26844539 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f594c0f20e0 0x7f594c0f0200 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7f5944006f00 tx=0x7f5944009f10).stop 2021-02-10T11:04:08.265 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:08.250 7f595355b700 1 -- 172.21.2.102:0/26844539 shutdown_connections 2021-02-10T11:04:08.265 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:08.250 7f595355b700 1 --2- 172.21.2.102:0/26844539 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f594c0f20e0 0x7f594c0f0200 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:08.265 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:08.250 7f595355b700 1 --2- 172.21.2.102:0/26844539 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f594c1b0470 0x7f594c1b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:08.265 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:08.250 7f595355b700 1 -- 172.21.2.102:0/26844539 >> 172.21.2.102:0/26844539 conn(0x7f594c0ce3e0 msgr2=0x7f594c0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:04:08.266 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:08.250 7f595355b700 1 -- 172.21.2.102:0/26844539 shutdown_connections 2021-02-10T11:04:08.266 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:08.250 7f595355b700 1 -- 172.21.2.102:0/26844539 wait complete. 2021-02-10T11:04:08.266 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:08.265 7f595355b700 1 Processor -- start 2021-02-10T11:04:08.266 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:08.265 7f595355b700 1 -- start start 2021-02-10T11:04:08.268 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:08.265 7f595355b700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f594c193230 0x7f594c0a6a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:08.268 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:08.265 7f595355b700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f594c08f5b0 0x7f594c08f9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:08.268 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:08.265 7f595355b700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f594c08e6a0 0x7f594c08eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:08.268 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:08.265 7f595355b700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f594c08ffd0 con 0x7f594c08f5b0 2021-02-10T11:04:08.269 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:08.265 7f595355b700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f594c090240 con 0x7f594c193230 2021-02-10T11:04:08.269 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:08.265 7f595355b700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f594c08f0c0 con 0x7f594c08e6a0 2021-02-10T11:04:08.269 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:08.265 7f5950b76700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f594c08f5b0 0x7f594c08f9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:08.269 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:08.265 7f5950b76700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f594c08f5b0 0x7f594c08f9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:49686/0 (socket says 172.21.2.102:49686) 2021-02-10T11:04:08.269 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:08.265 7f5950b76700 1 -- 172.21.2.102:0/1685059163 learned_addr learned my addr 172.21.2.102:0/1685059163 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:04:08.269 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:08.266 7f5950b76700 1 -- 172.21.2.102:0/1685059163 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f594c08e6a0 msgr2=0x7f594c08eaa0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:04:08.269 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:08.266 7f5950b76700 1 --2- 172.21.2.102:0/1685059163 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f594c08e6a0 0x7f594c08eaa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:08.269 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:08.266 7f5950b76700 1 -- 172.21.2.102:0/1685059163 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f594c193230 msgr2=0x7f594c0a6a80 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:04:08.269 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:08.266 7f5950b76700 1 --2- 172.21.2.102:0/1685059163 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f594c193230 0x7f594c0a6a80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:08.269 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:08.266 7f5950b76700 1 -- 172.21.2.102:0/1685059163 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f594c08f2a0 con 0x7f594c08f5b0 2021-02-10T11:04:08.269 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:08.266 7f5950b76700 1 --2- 172.21.2.102:0/1685059163 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f594c08f5b0 0x7f594c08f9b0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7f5940001c60 tx=0x7f594000a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:04:08.269 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:08.266 7f593dffb700 1 -- 172.21.2.102:0/1685059163 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f594000acc0 con 0x7f594c08f5b0 2021-02-10T11:04:08.269 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:08.266 7f593dffb700 1 -- 172.21.2.102:0/1685059163 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5940004140 con 0x7f594c08f5b0 2021-02-10T11:04:08.269 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:08.266 7f593dffb700 1 -- 172.21.2.102:0/1685059163 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5940004620 con 0x7f594c08f5b0 2021-02-10T11:04:08.270 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:08.266 7f595355b700 1 -- 172.21.2.102:0/1685059163 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f594c08f2a0 con 0x7f594c08f5b0 2021-02-10T11:04:08.270 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:08.266 7f595355b700 1 -- 172.21.2.102:0/1685059163 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f594c08f2a0 con 0x7f594c08f5b0 2021-02-10T11:04:08.270 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:08.267 7f593dffb700 1 -- 172.21.2.102:0/1685059163 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f5940000db0 con 0x7f594c08f5b0 2021-02-10T11:04:08.270 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:08.267 7f593dffb700 1 --2- 172.21.2.102:0/1685059163 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f592c0186c0 0x7f592c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:08.270 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:08.267 7f593dffb700 1 -- 172.21.2.102:0/1685059163 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f5940032780 con 0x7f594c08f5b0 2021-02-10T11:04:08.270 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:08.267 7f5951377700 1 --2- 172.21.2.102:0/1685059163 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f592c0186c0 0x7f592c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:08.270 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:08.267 7f5951377700 1 --2- 172.21.2.102:0/1685059163 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f592c0186c0 0x7f592c01abc0 crc :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:04:08.297 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:08.298 7f595355b700 1 -- 172.21.2.102:0/1685059163 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f594c0f0b50 con 0x7f594c08f5b0 2021-02-10T11:04:08.301 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:08.301 7f593dffb700 1 -- 172.21.2.102:0/1685059163 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f59400009b0 con 0x7f594c08f5b0 2021-02-10T11:04:08.427 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:08.428 7f595355b700 1 -- 172.21.2.102:0/1685059163 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f594c0f0b50 con 0x7f594c08f5b0 2021-02-10T11:04:08.427 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:08.429 7f593dffb700 1 -- 172.21.2.102:0/1685059163 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v20) v1 ==== 92+0+10 (secure 0 0 0) 0x7f5940000db0 con 0x7f594c08f5b0 2021-02-10T11:04:08.428 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:04:08.430 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:08.430 7f595355b700 1 -- 172.21.2.102:0/1685059163 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f592c0186c0 msgr2=0x7f592c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:08.430 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:08.430 7f595355b700 1 --2- 172.21.2.102:0/1685059163 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f592c0186c0 0x7f592c01abc0 crc :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:08.430 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:08.430 7f595355b700 1 -- 172.21.2.102:0/1685059163 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f594c08f5b0 msgr2=0x7f594c08f9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:08.430 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:08.430 7f595355b700 1 --2- 172.21.2.102:0/1685059163 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f594c08f5b0 0x7f594c08f9b0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7f5940001c60 tx=0x7f594000a8d0).stop 2021-02-10T11:04:08.430 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:08.431 7f595355b700 1 -- 172.21.2.102:0/1685059163 shutdown_connections 2021-02-10T11:04:08.430 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:08.431 7f595355b700 1 --2- 172.21.2.102:0/1685059163 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f594c08e6a0 0x7f594c08eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:08.430 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:08.431 7f595355b700 1 --2- 172.21.2.102:0/1685059163 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f594c08f5b0 0x7f594c08f9b0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:08.430 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:08.431 7f595355b700 1 --2- 172.21.2.102:0/1685059163 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f592c0186c0 0x7f592c01abc0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:08.431 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:08.431 7f595355b700 1 --2- 172.21.2.102:0/1685059163 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f594c193230 0x7f594c0a6a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:08.431 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:08.431 7f595355b700 1 -- 172.21.2.102:0/1685059163 >> 172.21.2.102:0/1685059163 conn(0x7f594c0bf4e0 msgr2=0x7f594c0be160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:04:08.431 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:08.431 7f595355b700 1 -- 172.21.2.102:0/1685059163 shutdown_connections 2021-02-10T11:04:08.431 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:08.431 7f595355b700 1 -- 172.21.2.102:0/1685059163 wait complete. 2021-02-10T11:04:08.438 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-02-10T11:04:08.866 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:08.851 7faec5a32700 1 Processor -- start 2021-02-10T11:04:08.866 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:08.851 7faec5a32700 1 -- start start 2021-02-10T11:04:08.866 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:08.852 7faec5a32700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7faec01b0470 0x7faec01b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:08.866 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:08.852 7faec5a32700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7faec00f20e0 0x7faec00f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:08.867 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:08.852 7faec5a32700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faec00f0b50 con 0x7faec00ef610 2021-02-10T11:04:08.867 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:08.852 7faec5a32700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faec00f0820 con 0x7faec00f20e0 2021-02-10T11:04:08.867 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:08.852 7faec5a32700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faec00f24e0 con 0x7faec01b0470 2021-02-10T11:04:08.867 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:08.852 7faebffff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7faec00f20e0 0x7faec00f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:08.867 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:08.852 7faebffff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7faec00f20e0 0x7faec00f0200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:49698/0 (socket says 172.21.2.102:49698) 2021-02-10T11:04:08.867 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:08.852 7faebffff700 1 -- 172.21.2.102:0/1640769710 learned_addr learned my addr 172.21.2.102:0/1640769710 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:04:08.867 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:08.852 7faebffff700 1 -- 172.21.2.102:0/1640769710 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7faec01b0470 msgr2=0x7faec01b0870 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:04:08.867 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:08.852 7faebffff700 1 --2- 172.21.2.102:0/1640769710 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7faec01b0470 0x7faec01b0870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:08.867 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:08.852 7faebffff700 1 -- 172.21.2.102:0/1640769710 >> v1:172.21.2.113:6789/0 conn(0x7faec00ef610 legacy=0x7faec00efa10 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:04:08.867 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:08.852 7faebffff700 1 -- 172.21.2.102:0/1640769710 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faec00e52e0 con 0x7faec00f20e0 2021-02-10T11:04:08.867 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:08.852 7faebffff700 1 --2- 172.21.2.102:0/1640769710 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7faec00f20e0 0x7faec00f0200 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7faeb8006f00 tx=0x7faeb8009f10).ready entity=mon.0 client_cookie=b04fcf0bce07bf83 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:04:08.867 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:08.852 7faebe7fc700 1 -- 172.21.2.102:0/1640769710 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faeb800b3b0 con 0x7faec00f20e0 2021-02-10T11:04:08.868 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:08.853 7faebe7fc700 1 -- 172.21.2.102:0/1640769710 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faeb800b630 con 0x7faec00f20e0 2021-02-10T11:04:08.868 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:08.853 7faebe7fc700 1 -- 172.21.2.102:0/1640769710 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faeb800bb10 con 0x7faec00f20e0 2021-02-10T11:04:08.868 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:08.853 7faec5a32700 1 -- 172.21.2.102:0/1640769710 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7faec00f20e0 msgr2=0x7faec00f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:08.868 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:08.853 7faec5a32700 1 --2- 172.21.2.102:0/1640769710 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7faec00f20e0 0x7faec00f0200 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7faeb8006f00 tx=0x7faeb8009f10).stop 2021-02-10T11:04:08.868 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:08.853 7faec5a32700 1 -- 172.21.2.102:0/1640769710 shutdown_connections 2021-02-10T11:04:08.868 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:08.853 7faec5a32700 1 --2- 172.21.2.102:0/1640769710 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7faec00f20e0 0x7faec00f0200 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:08.868 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:08.853 7faec5a32700 1 --2- 172.21.2.102:0/1640769710 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7faec01b0470 0x7faec01b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:08.868 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:08.853 7faec5a32700 1 -- 172.21.2.102:0/1640769710 >> 172.21.2.102:0/1640769710 conn(0x7faec00ce3e0 msgr2=0x7faec00e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:04:08.868 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:08.853 7faec5a32700 1 -- 172.21.2.102:0/1640769710 shutdown_connections 2021-02-10T11:04:08.868 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:08.853 7faec5a32700 1 -- 172.21.2.102:0/1640769710 wait complete. 2021-02-10T11:04:08.870 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:08.868 7faec5a32700 1 Processor -- start 2021-02-10T11:04:08.870 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:08.868 7faec5a32700 1 -- start start 2021-02-10T11:04:08.870 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:08.868 7faec5a32700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7faec0094fb0 0x7faec009e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:08.870 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:08.868 7faec5a32700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7faec00c2be0 0x7faec00c2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:08.870 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:08.868 7faec5a32700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7faec00c3600 0x7faec00a9c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:08.870 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:08.868 7faec5a32700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faec009ebf0 con 0x7faec00c3600 2021-02-10T11:04:08.870 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:08.868 7faec5a32700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faec00aa2a0 con 0x7faec0094fb0 2021-02-10T11:04:08.870 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:08.868 7faec5a32700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faec00aa4c0 con 0x7faec00c2be0 2021-02-10T11:04:08.870 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:08.868 7faebffff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7faec00c3600 0x7faec00a9c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:08.870 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:08.868 7faebffff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7faec00c3600 0x7faec00a9c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:49702/0 (socket says 172.21.2.102:49702) 2021-02-10T11:04:08.870 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:08.868 7faebffff700 1 -- 172.21.2.102:0/45174483 learned_addr learned my addr 172.21.2.102:0/45174483 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:04:08.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:08.868 7faebffff700 1 -- 172.21.2.102:0/45174483 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7faec00c2be0 msgr2=0x7faec00c2fe0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:04:08.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:08.868 7faebffff700 1 --2- 172.21.2.102:0/45174483 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7faec00c2be0 0x7faec00c2fe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:08.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:08.868 7faebffff700 1 -- 172.21.2.102:0/45174483 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7faec0094fb0 msgr2=0x7faec009e5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:08.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:08.868 7faebffff700 1 --2- 172.21.2.102:0/45174483 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7faec0094fb0 0x7faec009e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:08.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:08.868 7faebffff700 1 -- 172.21.2.102:0/45174483 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faec00aa6e0 con 0x7faec00c3600 2021-02-10T11:04:08.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:08.869 7faebffff700 1 --2- 172.21.2.102:0/45174483 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7faec00c3600 0x7faec00a9c80 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7faeb8009390 tx=0x7faeb800bd40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:04:08.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:08.869 7faeb7fff700 1 -- 172.21.2.102:0/45174483 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faeb8003030 con 0x7faec00c3600 2021-02-10T11:04:08.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:08.869 7faeb7fff700 1 -- 172.21.2.102:0/45174483 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faeb80068e0 con 0x7faec00c3600 2021-02-10T11:04:08.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:08.869 7faec5a32700 1 -- 172.21.2.102:0/45174483 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faec00aa6e0 con 0x7faec00c3600 2021-02-10T11:04:08.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:08.869 7faec5a32700 1 -- 172.21.2.102:0/45174483 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faec00aa6e0 con 0x7faec00c3600 2021-02-10T11:04:08.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:08.869 7faeb7fff700 1 -- 172.21.2.102:0/45174483 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faeb800c570 con 0x7faec00c3600 2021-02-10T11:04:08.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:08.869 7faeb7fff700 1 -- 172.21.2.102:0/45174483 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7faeb8005820 con 0x7faec00c3600 2021-02-10T11:04:08.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:08.870 7faeb7fff700 1 --2- 172.21.2.102:0/45174483 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7faea40186c0 0x7faea401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:08.872 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:08.870 7faeb7fff700 1 -- 172.21.2.102:0/45174483 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5573+0+0 (secure 0 0 0) 0x7faeb8032b70 con 0x7faec00c3600 2021-02-10T11:04:08.872 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:08.870 7faebf7fe700 1 --2- 172.21.2.102:0/45174483 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7faea40186c0 0x7faea401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:08.872 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:08.870 7faebf7fe700 1 --2- 172.21.2.102:0/45174483 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7faea40186c0 0x7faea401abc0 crc :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:04:08.901 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:08.902 7faec5a32700 1 -- 172.21.2.102:0/45174483 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faec00edd10 con 0x7faec00c3600 2021-02-10T11:04:08.903 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:08.904 7faeb7fff700 1 -- 172.21.2.102:0/45174483 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7faeb8006c80 con 0x7faec00c3600 2021-02-10T11:04:09.030 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:09.031 7faec5a32700 1 -- 172.21.2.102:0/45174483 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7faec00e52e0 con 0x7faec00c3600 2021-02-10T11:04:09.031 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:09.032 7faeb7fff700 1 -- 172.21.2.102:0/45174483 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v20) v1 ==== 104+0+10 (secure 0 0 0) 0x7faeb80068e0 con 0x7faec00c3600 2021-02-10T11:04:09.031 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:04:09.033 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:09.034 7faec5a32700 1 -- 172.21.2.102:0/45174483 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7faea40186c0 msgr2=0x7faea401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:09.033 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:09.034 7faec5a32700 1 --2- 172.21.2.102:0/45174483 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7faea40186c0 0x7faea401abc0 crc :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:09.033 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:09.034 7faec5a32700 1 -- 172.21.2.102:0/45174483 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7faec00c3600 msgr2=0x7faec00a9c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:09.033 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:09.034 7faec5a32700 1 --2- 172.21.2.102:0/45174483 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7faec00c3600 0x7faec00a9c80 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7faeb8009390 tx=0x7faeb800bd40).stop 2021-02-10T11:04:09.033 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:09.034 7faec5a32700 1 -- 172.21.2.102:0/45174483 shutdown_connections 2021-02-10T11:04:09.033 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:09.034 7faec5a32700 1 --2- 172.21.2.102:0/45174483 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7faec00c3600 0x7faec00a9c80 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:09.033 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:09.034 7faec5a32700 1 --2- 172.21.2.102:0/45174483 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7faec00c2be0 0x7faec00c2fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:09.033 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:09.034 7faec5a32700 1 --2- 172.21.2.102:0/45174483 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7faea40186c0 0x7faea401abc0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:09.033 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:09.034 7faec5a32700 1 --2- 172.21.2.102:0/45174483 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7faec0094fb0 0x7faec009e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:09.033 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:09.034 7faec5a32700 1 -- 172.21.2.102:0/45174483 >> 172.21.2.102:0/45174483 conn(0x7faec009f630 msgr2=0x7faec00a0130 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:04:09.034 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:09.034 7faec5a32700 1 -- 172.21.2.102:0/45174483 shutdown_connections 2021-02-10T11:04:09.034 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:09.034 7faec5a32700 1 -- 172.21.2.102:0/45174483 wait complete. 2021-02-10T11:04:09.041 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-02-10T11:04:09.471 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:09.456 7fe30dbf4700 1 Processor -- start 2021-02-10T11:04:09.471 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:09.456 7fe30dbf4700 1 -- start start 2021-02-10T11:04:09.471 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:09.456 7fe30dbf4700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fe3080f05f0 0x7fe3080f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:09.471 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:09.456 7fe30dbf4700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3081b0c70 con 0x7fe3081b0870 2021-02-10T11:04:09.472 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:09.456 7fe30dbf4700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3080f0b50 con 0x7fe3081b0470 2021-02-10T11:04:09.472 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:09.456 7fe30dbf4700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3080e5ea0 con 0x7fe3080f05f0 2021-02-10T11:04:09.472 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:09.456 7fe307fff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fe3080f05f0 0x7fe3080f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:09.472 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:09.456 7fe307fff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fe3080f05f0 0x7fe3080f20e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:49712/0 (socket says 172.21.2.102:49712) 2021-02-10T11:04:09.472 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:09.456 7fe307fff700 1 -- 172.21.2.102:0/3546017640 learned_addr learned my addr 172.21.2.102:0/3546017640 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:04:09.472 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:09.456 7fe306ffd700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fe3081b0870 0x7fe3080efe00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:41932/0 (socket says 172.21.2.102:41932) 2021-02-10T11:04:09.472 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:09.457 7fe307fff700 1 -- 172.21.2.102:0/3546017640 >> v1:172.21.2.113:6789/0 conn(0x7fe3081b0470 legacy=0x7fe3080ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:04:09.472 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:09.457 7fe307fff700 1 -- 172.21.2.102:0/3546017640 >> v1:172.21.2.102:6789/0 conn(0x7fe3081b0870 legacy=0x7fe3080efe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:09.472 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:09.457 7fe307fff700 1 -- 172.21.2.102:0/3546017640 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3080e52e0 con 0x7fe3080f05f0 2021-02-10T11:04:09.472 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:09.457 7fe3067fc700 1 -- 172.21.2.102:0/3546017640 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2027438744 0 0) 0x7fe2fc001100 con 0x7fe3081b0870 2021-02-10T11:04:09.472 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:09.457 7fe307fff700 1 --2- 172.21.2.102:0/3546017640 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe3080f05f0 0x7fe3080f20e0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7fe300006f00 tx=0x7fe300009e90).ready entity=mon.0 client_cookie=fe5d942769e0e0f8 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:04:09.472 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:09.457 7fe3067fc700 1 -- 172.21.2.102:0/3546017640 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe30000b300 con 0x7fe3080f05f0 2021-02-10T11:04:09.473 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:09.458 7fe3067fc700 1 -- 172.21.2.102:0/3546017640 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe30000b580 con 0x7fe3080f05f0 2021-02-10T11:04:09.473 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:09.458 7fe3067fc700 1 -- 172.21.2.102:0/3546017640 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe30000b9d0 con 0x7fe3080f05f0 2021-02-10T11:04:09.473 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:09.458 7fe30dbf4700 1 -- 172.21.2.102:0/3546017640 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe3080f05f0 msgr2=0x7fe3080f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:09.473 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:09.458 7fe30dbf4700 1 --2- 172.21.2.102:0/3546017640 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe3080f05f0 0x7fe3080f20e0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7fe300006f00 tx=0x7fe300009e90).stop 2021-02-10T11:04:09.473 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:09.458 7fe30dbf4700 1 -- 172.21.2.102:0/3546017640 shutdown_connections 2021-02-10T11:04:09.473 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:09.458 7fe30dbf4700 1 --2- 172.21.2.102:0/3546017640 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe3080f05f0 0x7fe3080f20e0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:09.473 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:09.458 7fe30dbf4700 1 -- 172.21.2.102:0/3546017640 >> 172.21.2.102:0/3546017640 conn(0x7fe3080ce3e0 msgr2=0x7fe3080e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:04:09.473 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:09.458 7fe30dbf4700 1 -- 172.21.2.102:0/3546017640 shutdown_connections 2021-02-10T11:04:09.473 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:09.458 7fe30dbf4700 1 -- 172.21.2.102:0/3546017640 wait complete. 2021-02-10T11:04:09.474 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:09.473 7fe30dbf4700 1 Processor -- start 2021-02-10T11:04:09.474 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:09.473 7fe30dbf4700 1 -- start start 2021-02-10T11:04:09.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:09.473 7fe30dbf4700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe3081a0570 0x7fe3080a4590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:09.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:09.473 7fe30dbf4700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe3080f1fd0 0x7fe3080f23d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:09.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:09.473 7fe30dbf4700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe3080a5450 0x7fe3080a5850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:09.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:09.473 7fe30dbf4700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3080a4bb0 con 0x7fe3080a5450 2021-02-10T11:04:09.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:09.473 7fe30dbf4700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3080b35e0 con 0x7fe3081a0570 2021-02-10T11:04:09.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:09.473 7fe30dbf4700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3080a1f00 con 0x7fe3080f1fd0 2021-02-10T11:04:09.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:09.474 7fe307fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe3080a5450 0x7fe3080a5850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:09.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:09.474 7fe307fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe3080a5450 0x7fe3080a5850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:49714/0 (socket says 172.21.2.102:49714) 2021-02-10T11:04:09.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:09.474 7fe307fff700 1 -- 172.21.2.102:0/198071211 learned_addr learned my addr 172.21.2.102:0/198071211 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:04:09.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:09.474 7fe307fff700 1 -- 172.21.2.102:0/198071211 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe3080f1fd0 msgr2=0x7fe3080f23d0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:04:09.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:09.474 7fe307fff700 1 --2- 172.21.2.102:0/198071211 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe3080f1fd0 0x7fe3080f23d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:09.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:09.474 7fe307fff700 1 -- 172.21.2.102:0/198071211 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe3081a0570 msgr2=0x7fe3080a4590 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:04:09.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:09.474 7fe307fff700 1 --2- 172.21.2.102:0/198071211 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe3081a0570 0x7fe3080a4590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:09.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:09.474 7fe307fff700 1 -- 172.21.2.102:0/198071211 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3080a5e70 con 0x7fe3080a5450 2021-02-10T11:04:09.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:09.474 7fe307fff700 1 --2- 172.21.2.102:0/198071211 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe3080a5450 0x7fe3080a5850 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7fe30000b910 tx=0x7fe30000bcb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:04:09.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:09.474 7fe2effff700 1 -- 172.21.2.102:0/198071211 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe30000bdf0 con 0x7fe3080a5450 2021-02-10T11:04:09.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:09.474 7fe2effff700 1 -- 172.21.2.102:0/198071211 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe300005820 con 0x7fe3080a5450 2021-02-10T11:04:09.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:09.474 7fe2effff700 1 -- 172.21.2.102:0/198071211 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe300005d40 con 0x7fe3080a5450 2021-02-10T11:04:09.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:09.474 7fe30dbf4700 1 -- 172.21.2.102:0/198071211 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3080a5e70 con 0x7fe3080a5450 2021-02-10T11:04:09.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:09.474 7fe30dbf4700 1 -- 172.21.2.102:0/198071211 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3080a6400 con 0x7fe3080a5450 2021-02-10T11:04:09.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:09.475 7fe2effff700 1 -- 172.21.2.102:0/198071211 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fe300005a00 con 0x7fe3080a5450 2021-02-10T11:04:09.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:09.475 7fe2effff700 1 --2- 172.21.2.102:0/198071211 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe2e80186c0 0x7fe2e801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:09.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:09.475 7fe2effff700 1 -- 172.21.2.102:0/198071211 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fe3000328d0 con 0x7fe3080a5450 2021-02-10T11:04:09.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:09.475 7fe3077fe700 1 --2- 172.21.2.102:0/198071211 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe2e80186c0 0x7fe2e801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:09.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:09.476 7fe3077fe700 1 --2- 172.21.2.102:0/198071211 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe2e80186c0 0x7fe2e801abc0 crc :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:04:09.505 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:09.506 7fe30dbf4700 1 -- 172.21.2.102:0/198071211 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3080e52e0 con 0x7fe3080a5450 2021-02-10T11:04:09.507 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:09.509 7fe2effff700 1 -- 172.21.2.102:0/198071211 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fe300000f90 con 0x7fe3080a5450 2021-02-10T11:04:09.636 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:09.637 7fe30dbf4700 1 -- 172.21.2.102:0/198071211 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7fe3080e52e0 con 0x7fe3080a5450 2021-02-10T11:04:09.636 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:09.637 7fe2effff700 1 -- 172.21.2.102:0/198071211 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v20) v1 ==== 100+0+10 (secure 0 0 0) 0x7fe300006ef0 con 0x7fe3080a5450 2021-02-10T11:04:09.637 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:04:09.638 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:09.639 7fe30dbf4700 1 -- 172.21.2.102:0/198071211 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe2e80186c0 msgr2=0x7fe2e801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:09.638 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:09.639 7fe30dbf4700 1 --2- 172.21.2.102:0/198071211 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe2e80186c0 0x7fe2e801abc0 crc :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:09.639 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:09.639 7fe30dbf4700 1 -- 172.21.2.102:0/198071211 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe3080a5450 msgr2=0x7fe3080a5850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:09.639 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:09.639 7fe30dbf4700 1 --2- 172.21.2.102:0/198071211 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe3080a5450 0x7fe3080a5850 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7fe30000b910 tx=0x7fe30000bcb0).stop 2021-02-10T11:04:09.639 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:09.639 7fe30dbf4700 1 -- 172.21.2.102:0/198071211 shutdown_connections 2021-02-10T11:04:09.639 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:09.639 7fe30dbf4700 1 --2- 172.21.2.102:0/198071211 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe3080a5450 0x7fe3080a5850 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:09.639 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:09.639 7fe30dbf4700 1 --2- 172.21.2.102:0/198071211 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe3080f1fd0 0x7fe3080f23d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:09.639 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:09.639 7fe30dbf4700 1 --2- 172.21.2.102:0/198071211 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe2e80186c0 0x7fe2e801abc0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:09.639 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:09.639 7fe30dbf4700 1 --2- 172.21.2.102:0/198071211 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe3081a0570 0x7fe3080a4590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:09.639 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:09.639 7fe30dbf4700 1 -- 172.21.2.102:0/198071211 >> 172.21.2.102:0/198071211 conn(0x7fe3080958b0 msgr2=0x7fe3080bf040 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:04:09.639 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:09.640 7fe30dbf4700 1 -- 172.21.2.102:0/198071211 shutdown_connections 2021-02-10T11:04:09.639 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:09.640 7fe30dbf4700 1 -- 172.21.2.102:0/198071211 wait complete. 2021-02-10T11:04:09.647 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-02-10T11:04:09.647 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:04:10.075 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:10.059 7f54d7c3f700 1 Processor -- start 2021-02-10T11:04:10.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:10.059 7f54d7c3f700 1 -- start start 2021-02-10T11:04:10.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:10.060 7f54d7c3f700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f54d00e5070 0x7f54d01b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:10.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:10.060 7f54d7c3f700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f54d01b3e40 0x7f54d0080310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:10.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:10.060 7f54d7c3f700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54d007ed80 con 0x7f54d00e5470 2021-02-10T11:04:10.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:10.060 7f54d7c3f700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54d00d1790 con 0x7f54d01b3e40 2021-02-10T11:04:10.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:10.060 7f54d7c3f700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54d00d1980 con 0x7f54d00e5070 2021-02-10T11:04:10.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:10.061 7f54d5a5b700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f54d00e5070 0x7f54d01b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:10.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:10.061 7f54d525a700 1 --1- >> v1:172.21.2.113:6789/0 conn(0x7f54d00e5470 0x7f54d007d690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.113:6789/0 says I am v1:172.21.2.102:49270/0 (socket says 172.21.2.102:49270) 2021-02-10T11:04:10.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:10.061 7f54d525a700 1 -- 172.21.2.102:0/867242402 learned_addr learned my addr 172.21.2.102:0/867242402 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:04:10.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:10.061 7f54d625c700 1 --2- 172.21.2.102:0/867242402 >> v2:172.21.2.113:3300/0 conn(0x7f54d01b3e40 0x7f54d0080310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:10.077 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:10.061 7f54d5a5b700 1 -- 172.21.2.102:0/867242402 >> v2:172.21.2.113:3300/0 conn(0x7f54d01b3e40 msgr2=0x7f54d0080310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:04:10.077 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:10.061 7f54d5a5b700 1 --2- 172.21.2.102:0/867242402 >> v2:172.21.2.113:3300/0 conn(0x7f54d01b3e40 0x7f54d0080310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:10.077 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:10.061 7f54d5a5b700 1 -- 172.21.2.102:0/867242402 >> v1:172.21.2.113:6789/0 conn(0x7f54d00e5470 legacy=0x7f54d007d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:10.077 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:10.061 7f54d4a59700 1 -- 172.21.2.102:0/867242402 <== mon.1 v1:172.21.2.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2129903509 0 0) 0x7f54c00010b0 con 0x7f54d00e5470 2021-02-10T11:04:10.077 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:10.061 7f54d5a5b700 1 -- 172.21.2.102:0/867242402 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54d00d1ba0 con 0x7f54d00e5070 2021-02-10T11:04:10.077 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:10.061 7f54d5a5b700 1 --2- 172.21.2.102:0/867242402 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f54d00e5070 0x7f54d01b3820 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7f54cc006650 tx=0x7f54cc009a80).ready entity=mon.2 client_cookie=fe10424cc2a46800 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:04:10.077 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:10.062 7f54d4a59700 1 -- 172.21.2.102:0/867242402 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f54cc0009b0 con 0x7f54d00e5070 2021-02-10T11:04:10.077 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:10.062 7f54d4a59700 1 -- 172.21.2.102:0/867242402 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54cc00c0c0 con 0x7f54d00e5070 2021-02-10T11:04:10.077 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:10.062 7f54d4a59700 1 -- 172.21.2.102:0/867242402 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f54cc00c520 con 0x7f54d00e5070 2021-02-10T11:04:10.077 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:10.062 7f54d7c3f700 1 -- 172.21.2.102:0/867242402 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f54d00e5070 msgr2=0x7f54d01b3820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:10.077 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:10.062 7f54d7c3f700 1 --2- 172.21.2.102:0/867242402 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f54d00e5070 0x7f54d01b3820 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7f54cc006650 tx=0x7f54cc009a80).stop 2021-02-10T11:04:10.077 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:10.062 7f54d7c3f700 1 -- 172.21.2.102:0/867242402 shutdown_connections 2021-02-10T11:04:10.077 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:10.062 7f54d7c3f700 1 --2- 172.21.2.102:0/867242402 >> v2:172.21.2.113:3300/0 conn(0x7f54d01b3e40 0x7f54d0080310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:10.078 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:10.062 7f54d7c3f700 1 --2- 172.21.2.102:0/867242402 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f54d00e5070 0x7f54d01b3820 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:10.078 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:10.062 7f54d7c3f700 1 -- 172.21.2.102:0/867242402 >> 172.21.2.102:0/867242402 conn(0x7f54d00ce690 msgr2=0x7f54d00e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:04:10.078 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:10.062 7f54d7c3f700 1 -- 172.21.2.102:0/867242402 shutdown_connections 2021-02-10T11:04:10.078 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:10.063 7f54d7c3f700 1 -- 172.21.2.102:0/867242402 wait complete. 2021-02-10T11:04:10.079 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:10.077 7f54d7c3f700 1 Processor -- start 2021-02-10T11:04:10.079 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:10.077 7f54d7c3f700 1 -- start start 2021-02-10T11:04:10.080 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:10.077 7f54d7c3f700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f54d00959b0 0x7f54d00acf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:10.080 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:10.077 7f54d7c3f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f54d0088aa0 0x7f54d00ad570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:10.080 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:10.077 7f54d7c3f700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f54d007f220 0x7f54d007f620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:10.080 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:10.077 7f54d7c3f700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54d0094a70 con 0x7f54d0088aa0 2021-02-10T11:04:10.080 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:10.077 7f54d7c3f700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54d0094ff0 con 0x7f54d00959b0 2021-02-10T11:04:10.080 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:10.077 7f54d7c3f700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54d00951d0 con 0x7f54d007f220 2021-02-10T11:04:10.080 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:10.078 7f54d525a700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f54d0088aa0 0x7f54d00ad570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:10.080 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:10.078 7f54d525a700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f54d0088aa0 0x7f54d00ad570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:49730/0 (socket says 172.21.2.102:49730) 2021-02-10T11:04:10.080 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:10.078 7f54d525a700 1 -- 172.21.2.102:0/2256516979 learned_addr learned my addr 172.21.2.102:0/2256516979 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:04:10.080 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:10.078 7f54d525a700 1 -- 172.21.2.102:0/2256516979 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f54d007f220 msgr2=0x7f54d007f620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:10.080 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:10.078 7f54d525a700 1 --2- 172.21.2.102:0/2256516979 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f54d007f220 0x7f54d007f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:10.080 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:10.078 7f54d525a700 1 -- 172.21.2.102:0/2256516979 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f54d00959b0 msgr2=0x7f54d00acf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:10.080 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:10.078 7f54d525a700 1 --2- 172.21.2.102:0/2256516979 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f54d00959b0 0x7f54d00acf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:10.081 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:10.078 7f54d525a700 1 -- 172.21.2.102:0/2256516979 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54d00adb90 con 0x7f54d0088aa0 2021-02-10T11:04:10.081 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:10.078 7f54d525a700 1 --2- 172.21.2.102:0/2256516979 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f54d0088aa0 0x7f54d00ad570 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7f54c0001c10 tx=0x7f54c000a8f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:04:10.081 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:10.078 7f54c67fc700 1 -- 172.21.2.102:0/2256516979 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f54c00009b0 con 0x7f54d0088aa0 2021-02-10T11:04:10.081 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:10.078 7f54c67fc700 1 -- 172.21.2.102:0/2256516979 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54c00012c0 con 0x7f54d0088aa0 2021-02-10T11:04:10.081 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:10.078 7f54c67fc700 1 -- 172.21.2.102:0/2256516979 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f54c0004350 con 0x7f54d0088aa0 2021-02-10T11:04:10.081 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:10.078 7f54d7c3f700 1 -- 172.21.2.102:0/2256516979 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54d00adb90 con 0x7f54d0088aa0 2021-02-10T11:04:10.081 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:10.079 7f54d7c3f700 1 -- 172.21.2.102:0/2256516979 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54d00adb90 con 0x7f54d0088aa0 2021-02-10T11:04:10.081 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:10.079 7f54c67fc700 1 -- 172.21.2.102:0/2256516979 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f54c0004140 con 0x7f54d0088aa0 2021-02-10T11:04:10.081 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:10.079 7f54c67fc700 1 --2- 172.21.2.102:0/2256516979 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f54b80186c0 0x7f54b801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:10.081 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:10.079 7f54c67fc700 1 -- 172.21.2.102:0/2256516979 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f54c00323d0 con 0x7f54d0088aa0 2021-02-10T11:04:10.081 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:10.079 7f54d5a5b700 1 --2- 172.21.2.102:0/2256516979 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f54b80186c0 0x7f54b801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:10.081 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:10.080 7f54d5a5b700 1 --2- 172.21.2.102:0/2256516979 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f54b80186c0 0x7f54b801abc0 crc :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:04:10.109 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:10.110 7f54d7c3f700 1 -- 172.21.2.102:0/2256516979 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54d007ed80 con 0x7f54d0088aa0 2021-02-10T11:04:10.111 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:10.112 7f54c67fc700 1 -- 172.21.2.102:0/2256516979 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f54c0030d00 con 0x7f54d0088aa0 2021-02-10T11:04:10.240 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:10.241 7f54d7c3f700 1 -- 172.21.2.102:0/2256516979 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f54d007ed80 con 0x7f54d0088aa0 2021-02-10T11:04:10.241 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:10.241 7f54c67fc700 1 -- 172.21.2.102:0/2256516979 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+2094 (secure 0 0 0) 0x7f54c0004140 con 0x7f54d0088aa0 2021-02-10T11:04:10.241 INFO:teuthology.orchestra.run.gibba002.stderr:dumped fsmap epoch 6 2021-02-10T11:04:10.241 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:04:10.241 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":6,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4302,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.102:6835/3297596596","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6834","nonce":3297596596},{"type":"v1","addr":"172.21.2.102:6835","nonce":3297596596}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":4}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:03:43.384709","modified":"2021-02-10 11:03:52.791414","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4226},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4226":{"gid":4226,"name":"b","rank":0,"incarnation":5,"state":"up:active","state_seq":3,"addr":"172.21.2.113:6833/285807165","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6832","nonce":285807165},{"type":"v1","addr":"172.21.2.113:6833","nonce":285807165}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2021-02-10T11:04:10.242 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:10.243 7f54d7c3f700 1 -- 172.21.2.102:0/2256516979 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f54b80186c0 msgr2=0x7f54b801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:10.242 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:10.243 7f54d7c3f700 1 --2- 172.21.2.102:0/2256516979 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f54b80186c0 0x7f54b801abc0 crc :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:10.242 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:10.243 7f54d7c3f700 1 -- 172.21.2.102:0/2256516979 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f54d0088aa0 msgr2=0x7f54d00ad570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:10.242 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:10.243 7f54d7c3f700 1 --2- 172.21.2.102:0/2256516979 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f54d0088aa0 0x7f54d00ad570 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7f54c0001c10 tx=0x7f54c000a8f0).stop 2021-02-10T11:04:10.243 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:10.243 7f54d7c3f700 1 -- 172.21.2.102:0/2256516979 shutdown_connections 2021-02-10T11:04:10.243 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:10.243 7f54d7c3f700 1 --2- 172.21.2.102:0/2256516979 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f54d007f220 0x7f54d007f620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:10.243 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:10.243 7f54d7c3f700 1 --2- 172.21.2.102:0/2256516979 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f54d0088aa0 0x7f54d00ad570 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:10.243 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:10.243 7f54d7c3f700 1 --2- 172.21.2.102:0/2256516979 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f54b80186c0 0x7f54b801abc0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:10.243 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:10.243 7f54d7c3f700 1 --2- 172.21.2.102:0/2256516979 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f54d00959b0 0x7f54d00acf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:10.243 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:10.243 7f54d7c3f700 1 -- 172.21.2.102:0/2256516979 >> 172.21.2.102:0/2256516979 conn(0x7f54d01afc80 msgr2=0x7f54d0089400 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:04:10.243 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:10.243 7f54d7c3f700 1 -- 172.21.2.102:0/2256516979 shutdown_connections 2021-02-10T11:04:10.243 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:10.243 7f54d7c3f700 1 -- 172.21.2.102:0/2256516979 wait complete. 2021-02-10T11:04:10.452 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 6, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:03:43.384709', 'modified': '2021-02-10 11:03:52.791414', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4226}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4226': {'gid': 4226, 'name': 'b', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.113:6833/285807165', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.113:6832', 'nonce': 285807165}, {'type': 'v1', 'addr': '172.21.2.113:6833', 'nonce': 285807165}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:04:10.452 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:04:10.453 DEBUG:teuthology.orchestra.run.gibba013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:04:10.852 INFO:teuthology.orchestra.run.gibba013.stdout:{ 2021-02-10T11:04:10.852 INFO:teuthology.orchestra.run.gibba013.stdout: "cluster_fsid": "68db30ea-dbce-4aef-a6df-f01c6fea0513", 2021-02-10T11:04:10.852 INFO:teuthology.orchestra.run.gibba013.stdout: "whoami": 0, 2021-02-10T11:04:10.853 INFO:teuthology.orchestra.run.gibba013.stdout: "id": 4226, 2021-02-10T11:04:10.853 INFO:teuthology.orchestra.run.gibba013.stdout: "want_state": "up:active", 2021-02-10T11:04:10.853 INFO:teuthology.orchestra.run.gibba013.stdout: "state": "up:active", 2021-02-10T11:04:10.853 INFO:teuthology.orchestra.run.gibba013.stdout: "rank_uptime": 19.062682433999999, 2021-02-10T11:04:10.853 INFO:teuthology.orchestra.run.gibba013.stdout: "mdsmap_epoch": 6, 2021-02-10T11:04:10.853 INFO:teuthology.orchestra.run.gibba013.stdout: "osdmap_epoch": 20, 2021-02-10T11:04:10.853 INFO:teuthology.orchestra.run.gibba013.stdout: "osdmap_epoch_barrier": 20, 2021-02-10T11:04:10.853 INFO:teuthology.orchestra.run.gibba013.stdout: "uptime": 23.457116161999998 2021-02-10T11:04:10.853 INFO:teuthology.orchestra.run.gibba013.stdout:} 2021-02-10T11:04:10.856 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "68db30ea-dbce-4aef-a6df-f01c6fea0513", "whoami": 0, "id": 4226, "want_state": "up:active", "state": "up:active", "rank_uptime": 19.062682433999999, "mdsmap_epoch": 6, "osdmap_epoch": 20, "osdmap_epoch_barrier": 20, "uptime": 23.457116161999998 } 2021-02-10T11:04:10.856 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2021-02-10T11:04:10.856 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2021-02-10T11:04:10.857 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@gibba002.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2021-02-10T11:04:10.857 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:04:10.857 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec mkdir -p /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:04:11.191 DEBUG:teuthology.orchestra.run.gibba002:> sudo modprobe fuse 2021-02-10T11:04:11.211 DEBUG:teuthology.orchestra.run.gibba002:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:04:11.237 DEBUG:teuthology.orchestra.run.gibba013:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:04:11.583 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:04:11.583 DEBUG:teuthology.orchestra.run.gibba002:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:04:11.965 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:04:11.965 DEBUG:teuthology.orchestra.run.gibba002:> ls /sys/fs/fuse/connections 2021-02-10T11:04:12.298 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2021-02-10T11:04:12.298 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:04:12.301 DEBUG:teuthology.orchestra.run.gibba002:> sudo modprobe fuse 2021-02-10T11:04:12.647 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:04:12.648 DEBUG:teuthology.orchestra.run.gibba002:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:04:12.712 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba002.stderr:ceph-fuse[39866]: starting ceph client 2021-02-10T11:04:12.712 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba002.stderr:2021-02-10 11:04:12.713 7fc8b8f79f80 -1 init, newargv = 0x55ad3ce29b00 newargc=9 2021-02-10T11:04:12.718 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba002.stderr:ceph-fuse[39866]: starting fuse 2021-02-10T11:04:12.990 INFO:teuthology.orchestra.run.gibba002.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:04:12.990 INFO:teuthology.orchestra.run.gibba002.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:04:12.991 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:04:12.991 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:04:12.991 DEBUG:teuthology.orchestra.run.gibba002:> ls /sys/fs/fuse/connections 2021-02-10T11:04:13.326 INFO:teuthology.orchestra.run.gibba002.stdout:43 2021-02-10T11:04:13.326 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2021-02-10T11:04:13.326 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:04:13.327 DEBUG:teuthology.orchestra.run.gibba002:> sudo python3 -c ' 2021-02-10T11:04:13.327 DEBUG:teuthology.orchestra.run.gibba002:> import glob 2021-02-10T11:04:13.327 DEBUG:teuthology.orchestra.run.gibba002:> import re 2021-02-10T11:04:13.327 DEBUG:teuthology.orchestra.run.gibba002:> import os 2021-02-10T11:04:13.327 DEBUG:teuthology.orchestra.run.gibba002:> import subprocess 2021-02-10T11:04:13.327 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:04:13.327 DEBUG:teuthology.orchestra.run.gibba002:> def find_socket(client_name): 2021-02-10T11:04:13.327 DEBUG:teuthology.orchestra.run.gibba002:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2021-02-10T11:04:13.327 DEBUG:teuthology.orchestra.run.gibba002:> files = glob.glob(asok_path) 2021-02-10T11:04:13.327 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:04:13.327 DEBUG:teuthology.orchestra.run.gibba002:> # Given a non-glob path, it better be there 2021-02-10T11:04:13.328 DEBUG:teuthology.orchestra.run.gibba002:> if "*" not in asok_path: 2021-02-10T11:04:13.328 DEBUG:teuthology.orchestra.run.gibba002:> assert(len(files) == 1) 2021-02-10T11:04:13.328 DEBUG:teuthology.orchestra.run.gibba002:> return files[0] 2021-02-10T11:04:13.328 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:04:13.328 DEBUG:teuthology.orchestra.run.gibba002:> for f in files: 2021-02-10T11:04:13.328 DEBUG:teuthology.orchestra.run.gibba002:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2021-02-10T11:04:13.328 DEBUG:teuthology.orchestra.run.gibba002:> if os.path.exists("/proc/{0}".format(pid)): 2021-02-10T11:04:13.328 DEBUG:teuthology.orchestra.run.gibba002:> return f 2021-02-10T11:04:13.328 DEBUG:teuthology.orchestra.run.gibba002:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2021-02-10T11:04:13.328 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:04:13.328 DEBUG:teuthology.orchestra.run.gibba002:> print(find_socket("client.0")) 2021-02-10T11:04:13.328 DEBUG:teuthology.orchestra.run.gibba002:> ' 2021-02-10T11:04:13.700 INFO:teuthology.orchestra.run.gibba002.stdout:/var/run/ceph/ceph-client.0.39866.asok 2021-02-10T11:04:13.703 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.39866.asok 2021-02-10T11:04:13.703 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:04:13.703 DEBUG:teuthology.orchestra.run.gibba002:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.39866.asok status 2021-02-10T11:04:14.092 INFO:teuthology.orchestra.run.gibba002.stdout:{ 2021-02-10T11:04:14.269 INFO:teuthology.orchestra.run.gibba002.stdout: "metadata": { 2021-02-10T11:04:14.269 INFO:teuthology.orchestra.run.gibba002.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:04:14.269 INFO:teuthology.orchestra.run.gibba002.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:04:14.269 INFO:teuthology.orchestra.run.gibba002.stdout: "entity_id": "0", 2021-02-10T11:04:14.269 INFO:teuthology.orchestra.run.gibba002.stdout: "hostname": "gibba002", 2021-02-10T11:04:14.270 INFO:teuthology.orchestra.run.gibba002.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2021-02-10T11:04:14.270 INFO:teuthology.orchestra.run.gibba002.stdout: "pid": "39866", 2021-02-10T11:04:14.270 INFO:teuthology.orchestra.run.gibba002.stdout: "root": "/" 2021-02-10T11:04:14.270 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:14.270 INFO:teuthology.orchestra.run.gibba002.stdout: "dentry_count": 0, 2021-02-10T11:04:14.270 INFO:teuthology.orchestra.run.gibba002.stdout: "dentry_pinned_count": 0, 2021-02-10T11:04:14.270 INFO:teuthology.orchestra.run.gibba002.stdout: "id": 4554, 2021-02-10T11:04:14.270 INFO:teuthology.orchestra.run.gibba002.stdout: "inst": { 2021-02-10T11:04:14.270 INFO:teuthology.orchestra.run.gibba002.stdout: "name": { 2021-02-10T11:04:14.270 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "client", 2021-02-10T11:04:14.270 INFO:teuthology.orchestra.run.gibba002.stdout: "num": 4554 2021-02-10T11:04:14.270 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:14.270 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": { 2021-02-10T11:04:14.271 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:04:14.271 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:0", 2021-02-10T11:04:14.271 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 1672025766 2021-02-10T11:04:14.271 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:04:14.271 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:14.271 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": { 2021-02-10T11:04:14.271 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:04:14.271 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:0", 2021-02-10T11:04:14.271 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 1672025766 2021-02-10T11:04:14.271 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:14.271 INFO:teuthology.orchestra.run.gibba002.stdout: "inst_str": "client.4554 172.21.2.102:0/1672025766", 2021-02-10T11:04:14.271 INFO:teuthology.orchestra.run.gibba002.stdout: "addr_str": "172.21.2.102:0/1672025766", 2021-02-10T11:04:14.271 INFO:teuthology.orchestra.run.gibba002.stdout: "inode_count": 1, 2021-02-10T11:04:14.272 INFO:teuthology.orchestra.run.gibba002.stdout: "mds_epoch": 6, 2021-02-10T11:04:14.272 INFO:teuthology.orchestra.run.gibba002.stdout: "osd_epoch": 20, 2021-02-10T11:04:14.272 INFO:teuthology.orchestra.run.gibba002.stdout: "osd_epoch_barrier": 0, 2021-02-10T11:04:14.272 INFO:teuthology.orchestra.run.gibba002.stdout: "blacklisted": false 2021-02-10T11:04:14.272 INFO:teuthology.orchestra.run.gibba002.stdout:} 2021-02-10T11:04:14.273 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:04:14.700 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:14.685 7fd2289ca700 1 Processor -- start 2021-02-10T11:04:14.700 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:14.685 7fd2289ca700 1 -- start start 2021-02-10T11:04:14.700 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:14.685 7fd2289ca700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fd2240e5070 0x7fd2241b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:14.701 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:14.685 7fd2289ca700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fd2241b3e40 0x7fd22407d690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:14.701 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:14.685 7fd2289ca700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd22407ed80 con 0x7fd2240e5470 2021-02-10T11:04:14.701 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:14.685 7fd2289ca700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd224080310 con 0x7fd2240e5070 2021-02-10T11:04:14.701 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:14.685 7fd2289ca700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd224080530 con 0x7fd2241b3e40 2021-02-10T11:04:14.701 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:14.685 7fd22261c700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fd2240e5070 0x7fd2241b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:14.701 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:14.685 7fd22261c700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fd2240e5070 0x7fd2241b3820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:49750/0 (socket says 172.21.2.102:49750) 2021-02-10T11:04:14.701 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:14.685 7fd22261c700 1 -- 172.21.2.102:0/3294719108 learned_addr learned my addr 172.21.2.102:0/3294719108 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:04:14.701 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:14.686 7fd22261c700 1 -- 172.21.2.102:0/3294719108 >> v2:172.21.2.113:3300/0 conn(0x7fd2241b3e40 msgr2=0x7fd22407d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:04:14.701 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:14.686 7fd22261c700 1 --2- 172.21.2.102:0/3294719108 >> v2:172.21.2.113:3300/0 conn(0x7fd2241b3e40 0x7fd22407d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:14.701 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:14.686 7fd22261c700 1 -- 172.21.2.102:0/3294719108 >> v1:172.21.2.113:6789/0 conn(0x7fd2240e5470 legacy=0x7fd22407dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:04:14.701 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:14.686 7fd22261c700 1 -- 172.21.2.102:0/3294719108 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd224080750 con 0x7fd2240e5070 2021-02-10T11:04:14.701 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:14.686 7fd22261c700 1 --2- 172.21.2.102:0/3294719108 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd2240e5070 0x7fd2241b3820 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7fd20c006700 tx=0x7fd20c009b00).ready entity=mon.0 client_cookie=5cd4b77e2f642be3 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:04:14.701 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:14.686 7fd22161a700 1 -- 172.21.2.102:0/3294719108 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd20c00c0c0 con 0x7fd2240e5070 2021-02-10T11:04:14.702 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:14.686 7fd22161a700 1 -- 172.21.2.102:0/3294719108 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd20c00c340 con 0x7fd2240e5070 2021-02-10T11:04:14.702 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:14.686 7fd22161a700 1 -- 172.21.2.102:0/3294719108 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd20c00c820 con 0x7fd2240e5070 2021-02-10T11:04:14.702 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:14.686 7fd2289ca700 1 -- 172.21.2.102:0/3294719108 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd2240e5070 msgr2=0x7fd2241b3820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:14.702 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:14.686 7fd2289ca700 1 --2- 172.21.2.102:0/3294719108 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd2240e5070 0x7fd2241b3820 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7fd20c006700 tx=0x7fd20c009b00).stop 2021-02-10T11:04:14.702 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:14.687 7fd2289ca700 1 -- 172.21.2.102:0/3294719108 shutdown_connections 2021-02-10T11:04:14.702 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:14.687 7fd2289ca700 1 --2- 172.21.2.102:0/3294719108 >> v2:172.21.2.113:3300/0 conn(0x7fd2241b3e40 0x7fd22407d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:14.702 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:14.687 7fd2289ca700 1 --2- 172.21.2.102:0/3294719108 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd2240e5070 0x7fd2241b3820 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:14.702 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:14.687 7fd2289ca700 1 -- 172.21.2.102:0/3294719108 >> 172.21.2.102:0/3294719108 conn(0x7fd2240ce690 msgr2=0x7fd2240e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:04:14.702 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:14.687 7fd2289ca700 1 -- 172.21.2.102:0/3294719108 shutdown_connections 2021-02-10T11:04:14.702 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:14.687 7fd2289ca700 1 -- 172.21.2.102:0/3294719108 wait complete. 2021-02-10T11:04:14.704 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:14.702 7fd2289ca700 1 Processor -- start 2021-02-10T11:04:14.704 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:14.702 7fd2289ca700 1 -- start start 2021-02-10T11:04:14.704 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:14.702 7fd2289ca700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd2240a5770 0x7fd2240b72b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:14.704 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:14.702 7fd2289ca700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd2240b78d0 0x7fd2240b7cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:14.704 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:14.702 7fd2289ca700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd2241a2b50 0x7fd2241a2f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:14.704 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:14.702 7fd2289ca700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2241a3df0 con 0x7fd2240b78d0 2021-02-10T11:04:14.704 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:14.702 7fd2289ca700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd224092d90 con 0x7fd2240a5770 2021-02-10T11:04:14.704 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:14.702 7fd2289ca700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2241a3570 con 0x7fd2241a2b50 2021-02-10T11:04:14.704 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:14.702 7fd221e1b700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd2240b78d0 0x7fd2240b7cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:14.704 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:14.702 7fd221e1b700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd2240b78d0 0x7fd2240b7cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:49754/0 (socket says 172.21.2.102:49754) 2021-02-10T11:04:14.705 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:14.702 7fd221e1b700 1 -- 172.21.2.102:0/1784159336 learned_addr learned my addr 172.21.2.102:0/1784159336 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:04:14.705 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:14.703 7fd221e1b700 1 -- 172.21.2.102:0/1784159336 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd2241a2b50 msgr2=0x7fd2241a2f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:14.705 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:14.703 7fd222e1d700 1 --2- 172.21.2.102:0/1784159336 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd2241a2b50 0x7fd2241a2f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:14.705 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:14.703 7fd221e1b700 1 --2- 172.21.2.102:0/1784159336 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd2241a2b50 0x7fd2241a2f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:14.705 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:14.703 7fd22261c700 1 --2- 172.21.2.102:0/1784159336 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd2240a5770 0x7fd2240b72b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:14.705 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:14.703 7fd221e1b700 1 -- 172.21.2.102:0/1784159336 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd2240a5770 msgr2=0x7fd2240b72b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:14.705 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:14.703 7fd221e1b700 1 --2- 172.21.2.102:0/1784159336 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd2240a5770 0x7fd2240b72b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:14.705 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:14.703 7fd221e1b700 1 -- 172.21.2.102:0/1784159336 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2241a3790 con 0x7fd2240b78d0 2021-02-10T11:04:14.705 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:14.703 7fd22261c700 1 --2- 172.21.2.102:0/1784159336 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd2240a5770 0x7fd2240b72b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:04:14.705 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:14.703 7fd221e1b700 1 --2- 172.21.2.102:0/1784159336 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd2240b78d0 0x7fd2240b7cd0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7fd21c00b8c0 tx=0x7fd21c00bc60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:04:14.705 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:14.703 7fd21affd700 1 -- 172.21.2.102:0/1784159336 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd21c0066d0 con 0x7fd2240b78d0 2021-02-10T11:04:14.705 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:14.703 7fd21affd700 1 -- 172.21.2.102:0/1784159336 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd21c00c2f0 con 0x7fd2240b78d0 2021-02-10T11:04:14.705 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:14.703 7fd21affd700 1 -- 172.21.2.102:0/1784159336 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd21c00c750 con 0x7fd2240b78d0 2021-02-10T11:04:14.705 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:14.703 7fd2289ca700 1 -- 172.21.2.102:0/1784159336 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2241a3790 con 0x7fd2240b78d0 2021-02-10T11:04:14.706 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:14.703 7fd2289ca700 1 -- 172.21.2.102:0/1784159336 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2241a42f0 con 0x7fd2240b78d0 2021-02-10T11:04:14.706 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:14.704 7fd21affd700 1 -- 172.21.2.102:0/1784159336 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd21c001010 con 0x7fd2240b78d0 2021-02-10T11:04:14.706 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:14.704 7fd21affd700 1 --2- 172.21.2.102:0/1784159336 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd2080186c0 0x7fd20801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:14.706 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:14.704 7fd21affd700 1 -- 172.21.2.102:0/1784159336 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fd21c032930 con 0x7fd2240b78d0 2021-02-10T11:04:14.706 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:14.704 7fd22261c700 1 --2- 172.21.2.102:0/1784159336 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd2080186c0 0x7fd20801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:14.706 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:14.705 7fd22261c700 1 --2- 172.21.2.102:0/1784159336 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd2080186c0 0x7fd20801abc0 crc :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:04:14.734 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:14.735 7fd2289ca700 1 -- 172.21.2.102:0/1784159336 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd224083dd0 con 0x7fd2240b78d0 2021-02-10T11:04:14.737 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:14.738 7fd21affd700 1 -- 172.21.2.102:0/1784159336 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd21c02a000 con 0x7fd2240b78d0 2021-02-10T11:04:14.860 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:14.861 7fd2289ca700 1 -- 172.21.2.102:0/1784159336 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd22407ed80 con 0x7fd2240b78d0 2021-02-10T11:04:14.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:14.862 7fd21affd700 1 -- 172.21.2.102:0/1784159336 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+13993 (secure 0 0 0) 0x7fd21c02a000 con 0x7fd2240b78d0 2021-02-10T11:04:14.861 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:04:14.862 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":20,"fsid":"68db30ea-dbce-4aef-a6df-f01c6fea0513","created":"2021-02-10 11:03:06.331557","modified":"2021-02-10 11:03:44.386945","last_up_change":"2021-02-10 11:03:32.690562","last_in_change":"2021-02-10 11:03:32.690562","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:03:37.322519","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:03:41.161563","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2021-02-10 11:03:42.379651","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"be691b4d-a039-408a-af80-9d53f02b6f1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6818","nonce":36088},{"type":"v1","addr":"172.21.2.102:6820","nonce":36088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6822","nonce":36088},{"type":"v1","addr":"172.21.2.102:6824","nonce":36088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6830","nonce":36088},{"type":"v1","addr":"172.21.2.102:6832","nonce":36088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6826","nonce":36088},{"type":"v1","addr":"172.21.2.102:6828","nonce":36088}]},"public_addr":"172.21.2.102:6820/36088","cluster_addr":"172.21.2.102:6824/36088","heartbeat_back_addr":"172.21.2.102:6832/36088","heartbeat_front_addr":"172.21.2.102:6828/36088","state":["exists","up"]},{"osd":1,"uuid":"64e3ccc8-9c69-4d7c-865e-af8e83822635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6819","nonce":36087},{"type":"v1","addr":"172.21.2.102:6821","nonce":36087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6823","nonce":36087},{"type":"v1","addr":"172.21.2.102:6825","nonce":36087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6831","nonce":36087},{"type":"v1","addr":"172.21.2.102:6833","nonce":36087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6827","nonce":36087},{"type":"v1","addr":"172.21.2.102:6829","nonce":36087}]},"public_addr":"172.21.2.102:6821/36087","cluster_addr":"172.21.2.102:6825/36087","heartbeat_back_addr":"172.21.2.102:6833/36087","heartbeat_front_addr":"172.21.2.102:6829/36087","state":["exists","up"]},{"osd":2,"uuid":"153700af-747f-481f-bdf4-d6072d3a842f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6810","nonce":36089},{"type":"v1","addr":"172.21.2.102:6811","nonce":36089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6812","nonce":36089},{"type":"v1","addr":"172.21.2.102:6813","nonce":36089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6816","nonce":36089},{"type":"v1","addr":"172.21.2.102:6817","nonce":36089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6814","nonce":36089},{"type":"v1","addr":"172.21.2.102:6815","nonce":36089}]},"public_addr":"172.21.2.102:6811/36089","cluster_addr":"172.21.2.102:6813/36089","heartbeat_back_addr":"172.21.2.102:6817/36089","heartbeat_front_addr":"172.21.2.102:6815/36089","state":["exists","up"]},{"osd":3,"uuid":"cbc678f4-83a9-45e6-8830-2f7c232b1ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6802","nonce":36090},{"type":"v1","addr":"172.21.2.102:6803","nonce":36090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6804","nonce":36090},{"type":"v1","addr":"172.21.2.102:6805","nonce":36090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6808","nonce":36090},{"type":"v1","addr":"172.21.2.102:6809","nonce":36090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6806","nonce":36090},{"type":"v1","addr":"172.21.2.102:6807","nonce":36090}]},"public_addr":"172.21.2.102:6803/36090","cluster_addr":"172.21.2.102:6805/36090","heartbeat_back_addr":"172.21.2.102:6809/36090","heartbeat_front_addr":"172.21.2.102:6807/36090","state":["exists","up"]},{"osd":4,"uuid":"c01e4008-b258-4f10-afd2-9e7f37214351","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6808","nonce":36368},{"type":"v1","addr":"172.21.2.113:6811","nonce":36368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6814","nonce":36368},{"type":"v1","addr":"172.21.2.113:6818","nonce":36368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6826","nonce":36368},{"type":"v1","addr":"172.21.2.113:6828","nonce":36368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6820","nonce":36368},{"type":"v1","addr":"172.21.2.113:6824","nonce":36368}]},"public_addr":"172.21.2.113:6811/36368","cluster_addr":"172.21.2.113:6818/36368","heartbeat_back_addr":"172.21.2.113:6828/36368","heartbeat_front_addr":"172.21.2.113:6824/36368","state":["exists","up"]},{"osd":5,"uuid":"551f6007-daf8-4a98-8ee0-c2b16bcccb5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6802","nonce":36366},{"type":"v1","addr":"172.21.2.113:6805","nonce":36366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6807","nonce":36366},{"type":"v1","addr":"172.21.2.113:6810","nonce":36366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6819","nonce":36366},{"type":"v1","addr":"172.21.2.113:6822","nonce":36366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6812","nonce":36366},{"type":"v1","addr":"172.21.2.113:6815","nonce":36366}]},"public_addr":"172.21.2.113:6805/36366","cluster_addr":"172.21.2.113:6810/36366","heartbeat_back_addr":"172.21.2.113:6822/36366","heartbeat_front_addr":"172.21.2.113:6815/36366","state":["exists","up"]},{"osd":6,"uuid":"bf67c040-0e50-48be-b102-c3943bc9ebef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6800","nonce":36367},{"type":"v1","addr":"172.21.2.113:6801","nonce":36367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6803","nonce":36367},{"type":"v1","addr":"172.21.2.113:6804","nonce":36367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6813","nonce":36367},{"type":"v1","addr":"172.21.2.113:6816","nonce":36367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6806","nonce":36367},{"type":"v1","addr":"172.21.2.113:6809","nonce":36367}]},"public_addr":"172.21.2.113:6801/36367","cluster_addr":"172.21.2.113:6804/36367","heartbeat_back_addr":"172.21.2.113:6816/36367","heartbeat_front_addr":"172.21.2.113:6809/36367","state":["exists","up"]},{"osd":7,"uuid":"f5461f80-018d-428f-be89-2eb2bcbe7bb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6817","nonce":36365},{"type":"v1","addr":"172.21.2.113:6821","nonce":36365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6823","nonce":36365},{"type":"v1","addr":"172.21.2.113:6825","nonce":36365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6830","nonce":36365},{"type":"v1","addr":"172.21.2.113:6831","nonce":36365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6827","nonce":36365},{"type":"v1","addr":"172.21.2.113:6829","nonce":36365}]},"public_addr":"172.21.2.113:6821/36365","cluster_addr":"172.21.2.113:6825/36365","heartbeat_back_addr":"172.21.2.113:6831/36365","heartbeat_front_addr":"172.21.2.113:6829/36365","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:04:14.862 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:14.864 7fd2289ca700 1 -- 172.21.2.102:0/1784159336 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd2080186c0 msgr2=0x7fd20801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:14.863 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:14.864 7fd2289ca700 1 --2- 172.21.2.102:0/1784159336 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd2080186c0 0x7fd20801abc0 crc :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:14.863 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:14.864 7fd2289ca700 1 -- 172.21.2.102:0/1784159336 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd2240b78d0 msgr2=0x7fd2240b7cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:14.863 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:14.864 7fd2289ca700 1 --2- 172.21.2.102:0/1784159336 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd2240b78d0 0x7fd2240b7cd0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7fd21c00b8c0 tx=0x7fd21c00bc60).stop 2021-02-10T11:04:14.863 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:14.864 7fd2289ca700 1 -- 172.21.2.102:0/1784159336 shutdown_connections 2021-02-10T11:04:14.863 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:14.864 7fd2289ca700 1 --2- 172.21.2.102:0/1784159336 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd2241a2b50 0x7fd2241a2f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:14.863 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:14.864 7fd2289ca700 1 --2- 172.21.2.102:0/1784159336 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd2240b78d0 0x7fd2240b7cd0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:14.864 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:14.864 7fd2289ca700 1 --2- 172.21.2.102:0/1784159336 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd2080186c0 0x7fd20801abc0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:14.864 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:14.864 7fd2289ca700 1 --2- 172.21.2.102:0/1784159336 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd2240a5770 0x7fd2240b72b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:14.864 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:14.864 7fd2289ca700 1 -- 172.21.2.102:0/1784159336 >> 172.21.2.102:0/1784159336 conn(0x7fd224093720 msgr2=0x7fd224084b80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:04:14.864 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:14.864 7fd2289ca700 1 -- 172.21.2.102:0/1784159336 shutdown_connections 2021-02-10T11:04:14.864 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:14.864 7fd2289ca700 1 -- 172.21.2.102:0/1784159336 wait complete. 2021-02-10T11:04:14.871 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:03:37.322519', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2021-02-10 11:03:41.161563', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2021-02-10 11:03:42.379651', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-02-10T11:04:14.871 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:04:15.299 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:15.283 7fdd289c2700 1 Processor -- start 2021-02-10T11:04:15.299 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:15.283 7fdd289c2700 1 -- start start 2021-02-10T11:04:15.299 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:15.284 7fdd289c2700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fdd241b0470 0x7fdd241b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:15.300 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:15.284 7fdd289c2700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdd240ef610 0x7fdd240efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:15.300 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:15.284 7fdd289c2700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd240f0b50 con 0x7fdd240f0030 2021-02-10T11:04:15.300 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:15.284 7fdd289c2700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd240e52e0 con 0x7fdd241b0470 2021-02-10T11:04:15.300 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:15.284 7fdd289c2700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd240e54d0 con 0x7fdd240ef610 2021-02-10T11:04:15.300 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:15.284 7fdd2261c700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fdd241b0470 0x7fdd241b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:15.300 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:15.284 7fdd2261c700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fdd241b0470 0x7fdd241b0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:49762/0 (socket says 172.21.2.102:49762) 2021-02-10T11:04:15.300 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:15.284 7fdd22e1d700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fdd240f0030 0x7fdd240f10d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:41986/0 (socket says 172.21.2.102:41986) 2021-02-10T11:04:15.300 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:15.284 7fdd2261c700 1 -- 172.21.2.102:0/918952719 learned_addr learned my addr 172.21.2.102:0/918952719 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:04:15.300 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:15.285 7fdd2261c700 1 -- 172.21.2.102:0/918952719 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdd240ef610 msgr2=0x7fdd240efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:04:15.300 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:15.285 7fdd2261c700 1 --2- 172.21.2.102:0/918952719 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdd240ef610 0x7fdd240efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:15.301 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:15.285 7fdd2261c700 1 -- 172.21.2.102:0/918952719 >> v1:172.21.2.102:6789/0 conn(0x7fdd240f0030 legacy=0x7fdd240f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:15.301 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:15.285 7fdd2261c700 1 -- 172.21.2.102:0/918952719 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd240f0430 con 0x7fdd241b0470 2021-02-10T11:04:15.301 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:15.285 7fdd2261c700 1 --2- 172.21.2.102:0/918952719 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdd241b0470 0x7fdd241b0870 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7fdd14006700 tx=0x7fdd14009b80).ready entity=mon.0 client_cookie=530060582f6a3d2c server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:04:15.301 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:15.285 7fdd2161a700 1 -- 172.21.2.102:0/918952719 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdd1400c0c0 con 0x7fdd241b0470 2021-02-10T11:04:15.301 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:15.285 7fdd2161a700 1 -- 172.21.2.102:0/918952719 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd1400c340 con 0x7fdd241b0470 2021-02-10T11:04:15.301 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:15.285 7fdd2161a700 1 -- 172.21.2.102:0/918952719 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdd1400c860 con 0x7fdd241b0470 2021-02-10T11:04:15.301 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:15.285 7fdd289c2700 1 -- 172.21.2.102:0/918952719 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdd241b0470 msgr2=0x7fdd241b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:15.301 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:15.285 7fdd289c2700 1 --2- 172.21.2.102:0/918952719 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdd241b0470 0x7fdd241b0870 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7fdd14006700 tx=0x7fdd14009b80).stop 2021-02-10T11:04:15.301 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:15.286 7fdd289c2700 1 -- 172.21.2.102:0/918952719 shutdown_connections 2021-02-10T11:04:15.302 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:15.286 7fdd289c2700 1 --2- 172.21.2.102:0/918952719 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdd240ef610 0x7fdd240efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:15.302 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:15.286 7fdd289c2700 1 --2- 172.21.2.102:0/918952719 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdd241b0470 0x7fdd241b0870 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:15.302 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:15.286 7fdd289c2700 1 -- 172.21.2.102:0/918952719 >> 172.21.2.102:0/918952719 conn(0x7fdd240ce3e0 msgr2=0x7fdd240e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:04:15.302 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:15.286 7fdd289c2700 1 -- 172.21.2.102:0/918952719 shutdown_connections 2021-02-10T11:04:15.302 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:15.286 7fdd289c2700 1 -- 172.21.2.102:0/918952719 wait complete. 2021-02-10T11:04:15.302 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:15.300 7fdd289c2700 1 Processor -- start 2021-02-10T11:04:15.302 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:15.300 7fdd289c2700 1 -- start start 2021-02-10T11:04:15.303 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:15.301 7fdd289c2700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdd240bf910 0x7fdd240bfd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:15.303 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:15.301 7fdd289c2700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fdd240c0330 0x7fdd240c0730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:15.303 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:15.301 7fdd289c2700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdd240c13d0 0x7fdd240c17d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:15.303 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:15.301 7fdd289c2700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd24109930 con 0x7fdd240c13d0 2021-02-10T11:04:15.304 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:15.301 7fdd289c2700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd24109b50 con 0x7fdd240c0330 2021-02-10T11:04:15.304 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:15.301 7fdd289c2700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd240c1df0 con 0x7fdd240bf910 2021-02-10T11:04:15.304 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:15.301 7fdd22e1d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdd240c13d0 0x7fdd240c17d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:15.304 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:15.301 7fdd22e1d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdd240c13d0 0x7fdd240c17d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:49768/0 (socket says 172.21.2.102:49768) 2021-02-10T11:04:15.304 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:15.301 7fdd22e1d700 1 -- 172.21.2.102:0/3840484384 learned_addr learned my addr 172.21.2.102:0/3840484384 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:04:15.304 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:15.301 7fdd22e1d700 1 -- 172.21.2.102:0/3840484384 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdd240bf910 msgr2=0x7fdd240bfd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:15.304 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:15.301 7fdd22e1d700 1 --2- 172.21.2.102:0/3840484384 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdd240bf910 0x7fdd240bfd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:15.304 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:15.301 7fdd22e1d700 1 -- 172.21.2.102:0/3840484384 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fdd240c0330 msgr2=0x7fdd240c0730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:15.304 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:15.301 7fdd22e1d700 1 --2- 172.21.2.102:0/3840484384 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fdd240c0330 0x7fdd240c0730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:15.304 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:15.301 7fdd22e1d700 1 -- 172.21.2.102:0/3840484384 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd240c1fd0 con 0x7fdd240c13d0 2021-02-10T11:04:15.304 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:15.301 7fdd22e1d700 1 --2- 172.21.2.102:0/3840484384 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdd240c13d0 0x7fdd240c17d0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7fdd1c007f90 tx=0x7fdd1c00abd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:04:15.304 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:15.302 7fdd1affd700 1 -- 172.21.2.102:0/3840484384 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdd1c000ec0 con 0x7fdd240c13d0 2021-02-10T11:04:15.304 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:15.302 7fdd1affd700 1 -- 172.21.2.102:0/3840484384 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd1c0070d0 con 0x7fdd240c13d0 2021-02-10T11:04:15.304 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:15.302 7fdd289c2700 1 -- 172.21.2.102:0/3840484384 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd240c1fd0 con 0x7fdd240c13d0 2021-02-10T11:04:15.305 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:15.302 7fdd289c2700 1 -- 172.21.2.102:0/3840484384 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd240c2560 con 0x7fdd240c13d0 2021-02-10T11:04:15.305 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:15.302 7fdd1affd700 1 -- 172.21.2.102:0/3840484384 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdd1c004720 con 0x7fdd240c13d0 2021-02-10T11:04:15.305 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:15.302 7fdd1affd700 1 -- 172.21.2.102:0/3840484384 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fdd1c0072b0 con 0x7fdd240c13d0 2021-02-10T11:04:15.305 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:15.302 7fdd1affd700 1 --2- 172.21.2.102:0/3840484384 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fdd08018730 0x7fdd0801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:15.305 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:15.302 7fdd1affd700 1 -- 172.21.2.102:0/3840484384 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fdd1c032b00 con 0x7fdd240c13d0 2021-02-10T11:04:15.305 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:15.303 7fdd2261c700 1 --2- 172.21.2.102:0/3840484384 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fdd08018730 0x7fdd0801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:15.305 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:15.303 7fdd2261c700 1 --2- 172.21.2.102:0/3840484384 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fdd08018730 0x7fdd0801abc0 crc :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:04:15.333 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:15.334 7fdd289c2700 1 -- 172.21.2.102:0/3840484384 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd240f0b50 con 0x7fdd240c13d0 2021-02-10T11:04:15.335 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:15.336 7fdd1affd700 1 -- 172.21.2.102:0/3840484384 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fdd1c001270 con 0x7fdd240c13d0 2021-02-10T11:04:15.463 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:15.464 7fdd289c2700 1 -- 172.21.2.102:0/3840484384 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7fdd240f0b50 con 0x7fdd240c13d0 2021-02-10T11:04:15.463 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:15.464 7fdd1affd700 1 -- 172.21.2.102:0/3840484384 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v20) v1 ==== 92+0+10 (secure 0 0 0) 0x7fdd1c0070b0 con 0x7fdd240c13d0 2021-02-10T11:04:15.464 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:04:15.465 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:15.466 7fdd289c2700 1 -- 172.21.2.102:0/3840484384 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fdd08018730 msgr2=0x7fdd0801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:15.466 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:15.466 7fdd289c2700 1 --2- 172.21.2.102:0/3840484384 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fdd08018730 0x7fdd0801abc0 crc :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:15.466 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:15.466 7fdd289c2700 1 -- 172.21.2.102:0/3840484384 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdd240c13d0 msgr2=0x7fdd240c17d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:15.466 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:15.466 7fdd289c2700 1 --2- 172.21.2.102:0/3840484384 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdd240c13d0 0x7fdd240c17d0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7fdd1c007f90 tx=0x7fdd1c00abd0).stop 2021-02-10T11:04:15.466 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:15.466 7fdd289c2700 1 -- 172.21.2.102:0/3840484384 shutdown_connections 2021-02-10T11:04:15.466 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:15.466 7fdd289c2700 1 --2- 172.21.2.102:0/3840484384 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdd240c13d0 0x7fdd240c17d0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:15.466 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:15.466 7fdd289c2700 1 --2- 172.21.2.102:0/3840484384 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fdd08018730 0x7fdd0801abc0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:15.466 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:15.466 7fdd289c2700 1 --2- 172.21.2.102:0/3840484384 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fdd240c0330 0x7fdd240c0730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:15.466 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:15.466 7fdd289c2700 1 --2- 172.21.2.102:0/3840484384 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdd240bf910 0x7fdd240bfd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:15.466 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:15.466 7fdd289c2700 1 -- 172.21.2.102:0/3840484384 >> 172.21.2.102:0/3840484384 conn(0x7fdd240bf4e0 msgr2=0x7fdd24191190 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:04:15.466 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:15.466 7fdd289c2700 1 -- 172.21.2.102:0/3840484384 shutdown_connections 2021-02-10T11:04:15.466 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:15.466 7fdd289c2700 1 -- 172.21.2.102:0/3840484384 wait complete. 2021-02-10T11:04:15.473 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-02-10T11:04:15.902 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:15.887 7fa458cf4700 1 Processor -- start 2021-02-10T11:04:15.903 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:15.887 7fa458cf4700 1 -- start start 2021-02-10T11:04:15.903 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:15.887 7fa458cf4700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fa4541b0470 0x7fa4541b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:15.903 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:15.887 7fa458cf4700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4540f0b50 con 0x7fa4540f20e0 2021-02-10T11:04:15.903 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:15.887 7fa458cf4700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4540f24e0 con 0x7fa4540ef610 2021-02-10T11:04:15.903 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:15.887 7fa458cf4700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4540e5ea0 con 0x7fa4541b0470 2021-02-10T11:04:15.903 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:15.888 7fa452ae8700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fa4541b0470 0x7fa4541b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:15.903 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:15.888 7fa4532e9700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fa4540f20e0 0x7fa4540f0200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:42002/0 (socket says 172.21.2.102:42002) 2021-02-10T11:04:15.903 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:15.888 7fa4532e9700 1 -- 172.21.2.102:0/15006386 learned_addr learned my addr 172.21.2.102:0/15006386 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:04:15.903 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:15.888 7fa452ae8700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fa4541b0470 0x7fa4541b0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:49774/0 (socket says 172.21.2.102:49774) 2021-02-10T11:04:15.903 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:15.888 7fa452ae8700 1 -- 172.21.2.102:0/15006386 >> v1:172.21.2.113:6789/0 conn(0x7fa4540ef610 legacy=0x7fa4540efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:04:15.904 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:15.888 7fa452ae8700 1 -- 172.21.2.102:0/15006386 >> v1:172.21.2.102:6789/0 conn(0x7fa4540f20e0 legacy=0x7fa4540f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:15.904 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:15.888 7fa452ae8700 1 -- 172.21.2.102:0/15006386 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4540e52e0 con 0x7fa4541b0470 2021-02-10T11:04:15.904 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:15.888 7fa452ae8700 1 --2- 172.21.2.102:0/15006386 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa4541b0470 0x7fa4541b0870 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7fa43c006650 tx=0x7fa43c009a80).ready entity=mon.0 client_cookie=c1141ae519964bbc server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:04:15.904 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:15.888 7fa451ae6700 1 -- 172.21.2.102:0/15006386 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa43c0009b0 con 0x7fa4541b0470 2021-02-10T11:04:15.904 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:15.889 7fa451ae6700 1 -- 172.21.2.102:0/15006386 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa43c00c0c0 con 0x7fa4541b0470 2021-02-10T11:04:15.904 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:15.889 7fa451ae6700 1 -- 172.21.2.102:0/15006386 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa43c00c5e0 con 0x7fa4541b0470 2021-02-10T11:04:15.904 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:15.889 7fa458cf4700 1 -- 172.21.2.102:0/15006386 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa4541b0470 msgr2=0x7fa4541b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:15.904 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:15.889 7fa458cf4700 1 --2- 172.21.2.102:0/15006386 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa4541b0470 0x7fa4541b0870 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7fa43c006650 tx=0x7fa43c009a80).stop 2021-02-10T11:04:15.904 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:15.889 7fa458cf4700 1 -- 172.21.2.102:0/15006386 shutdown_connections 2021-02-10T11:04:15.904 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:15.889 7fa458cf4700 1 --2- 172.21.2.102:0/15006386 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa4541b0470 0x7fa4541b0870 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:15.904 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:15.889 7fa458cf4700 1 -- 172.21.2.102:0/15006386 >> 172.21.2.102:0/15006386 conn(0x7fa4540ce3e0 msgr2=0x7fa4540e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:04:15.904 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:15.889 7fa458cf4700 1 -- 172.21.2.102:0/15006386 shutdown_connections 2021-02-10T11:04:15.904 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:15.889 7fa458cf4700 1 -- 172.21.2.102:0/15006386 wait complete. 2021-02-10T11:04:15.906 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:15.904 7fa458cf4700 1 Processor -- start 2021-02-10T11:04:15.906 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:15.904 7fa458cf4700 1 -- start start 2021-02-10T11:04:15.906 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:15.905 7fa458cf4700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa4541c6910 0x7fa45408e840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:15.906 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:15.905 7fa458cf4700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa45408ee60 0x7fa4540b7db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:15.906 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:15.905 7fa458cf4700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa4540b83d0 0x7fa4540b2270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:15.907 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:15.905 7fa458cf4700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4540b87d0 con 0x7fa4541c6910 2021-02-10T11:04:15.907 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:15.905 7fa458cf4700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4540b89f0 con 0x7fa4540b83d0 2021-02-10T11:04:15.907 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:15.905 7fa458cf4700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4540b2890 con 0x7fa45408ee60 2021-02-10T11:04:15.907 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:15.905 7fa452ae8700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa4541c6910 0x7fa45408e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:15.907 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:15.905 7fa452ae8700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa4541c6910 0x7fa45408e840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:49784/0 (socket says 172.21.2.102:49784) 2021-02-10T11:04:15.907 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:15.905 7fa452ae8700 1 -- 172.21.2.102:0/1277497442 learned_addr learned my addr 172.21.2.102:0/1277497442 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:04:15.907 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:15.905 7fa452ae8700 1 -- 172.21.2.102:0/1277497442 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa45408ee60 msgr2=0x7fa4540b7db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:15.907 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:15.905 7fa452ae8700 1 --2- 172.21.2.102:0/1277497442 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa45408ee60 0x7fa4540b7db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:15.907 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:15.905 7fa452ae8700 1 -- 172.21.2.102:0/1277497442 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa4540b83d0 msgr2=0x7fa4540b2270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:15.907 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:15.905 7fa452ae8700 1 --2- 172.21.2.102:0/1277497442 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa4540b83d0 0x7fa4540b2270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:15.907 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:15.905 7fa452ae8700 1 -- 172.21.2.102:0/1277497442 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4540b2ab0 con 0x7fa4541c6910 2021-02-10T11:04:15.907 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:15.905 7fa452ae8700 1 --2- 172.21.2.102:0/1277497442 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa4541c6910 0x7fa45408e840 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7fa43c009630 tx=0x7fa43c00b900).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:04:15.907 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:15.905 7fa44b7fe700 1 -- 172.21.2.102:0/1277497442 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa43c0093a0 con 0x7fa4541c6910 2021-02-10T11:04:15.908 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:15.905 7fa44b7fe700 1 -- 172.21.2.102:0/1277497442 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa43c00bcf0 con 0x7fa4541c6910 2021-02-10T11:04:15.908 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:15.905 7fa44b7fe700 1 -- 172.21.2.102:0/1277497442 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa43c0054d0 con 0x7fa4541c6910 2021-02-10T11:04:15.908 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:15.905 7fa458cf4700 1 -- 172.21.2.102:0/1277497442 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4540b2ab0 con 0x7fa4541c6910 2021-02-10T11:04:15.908 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:15.905 7fa458cf4700 1 -- 172.21.2.102:0/1277497442 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4540b2ab0 con 0x7fa4541c6910 2021-02-10T11:04:15.908 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:15.906 7fa44b7fe700 1 -- 172.21.2.102:0/1277497442 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa43c0054f0 con 0x7fa4541c6910 2021-02-10T11:04:15.908 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:15.906 7fa44b7fe700 1 --2- 172.21.2.102:0/1277497442 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa4380186c0 0x7fa43801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:15.908 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:15.906 7fa44b7fe700 1 -- 172.21.2.102:0/1277497442 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fa43c0327b0 con 0x7fa4541c6910 2021-02-10T11:04:15.908 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:15.906 7fa4522e7700 1 --2- 172.21.2.102:0/1277497442 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa4380186c0 0x7fa43801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:15.908 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:15.906 7fa4522e7700 1 --2- 172.21.2.102:0/1277497442 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa4380186c0 0x7fa43801abc0 crc :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:04:15.936 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:15.937 7fa458cf4700 1 -- 172.21.2.102:0/1277497442 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa45407ed60 con 0x7fa4541c6910 2021-02-10T11:04:15.939 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:15.940 7fa44b7fe700 1 -- 172.21.2.102:0/1277497442 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fa43c0052e0 con 0x7fa4541c6910 2021-02-10T11:04:16.065 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:16.066 7fa458cf4700 1 -- 172.21.2.102:0/1277497442 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7fa45407ed60 con 0x7fa4541c6910 2021-02-10T11:04:16.065 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:16.066 7fa44b7fe700 1 -- 172.21.2.102:0/1277497442 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v20) v1 ==== 104+0+10 (secure 0 0 0) 0x7fa43c0054f0 con 0x7fa4541c6910 2021-02-10T11:04:16.065 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:04:16.067 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:16.068 7fa458cf4700 1 -- 172.21.2.102:0/1277497442 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa4380186c0 msgr2=0x7fa43801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:16.067 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:16.068 7fa458cf4700 1 --2- 172.21.2.102:0/1277497442 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa4380186c0 0x7fa43801abc0 crc :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:16.067 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:16.068 7fa458cf4700 1 -- 172.21.2.102:0/1277497442 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa4541c6910 msgr2=0x7fa45408e840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:16.067 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:16.068 7fa458cf4700 1 --2- 172.21.2.102:0/1277497442 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa4541c6910 0x7fa45408e840 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7fa43c009630 tx=0x7fa43c00b900).stop 2021-02-10T11:04:16.067 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:16.068 7fa458cf4700 1 -- 172.21.2.102:0/1277497442 shutdown_connections 2021-02-10T11:04:16.068 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:16.068 7fa458cf4700 1 --2- 172.21.2.102:0/1277497442 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa4380186c0 0x7fa43801abc0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:16.068 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:16.068 7fa458cf4700 1 --2- 172.21.2.102:0/1277497442 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa4540b83d0 0x7fa4540b2270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:16.068 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:16.068 7fa458cf4700 1 --2- 172.21.2.102:0/1277497442 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa45408ee60 0x7fa4540b7db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:16.068 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:16.068 7fa458cf4700 1 --2- 172.21.2.102:0/1277497442 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa4541c6910 0x7fa45408e840 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:16.068 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:16.068 7fa458cf4700 1 -- 172.21.2.102:0/1277497442 >> 172.21.2.102:0/1277497442 conn(0x7fa4540bb880 msgr2=0x7fa4540b3360 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:04:16.068 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:16.068 7fa458cf4700 1 -- 172.21.2.102:0/1277497442 shutdown_connections 2021-02-10T11:04:16.068 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:16.068 7fa458cf4700 1 -- 172.21.2.102:0/1277497442 wait complete. 2021-02-10T11:04:16.075 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-02-10T11:04:16.505 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:16.489 7fda6faf3700 1 Processor -- start 2021-02-10T11:04:16.505 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:16.489 7fda6faf3700 1 -- start start 2021-02-10T11:04:16.505 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:16.490 7fda6faf3700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fda681b0470 0x7fda681b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:16.505 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:16.490 7fda6faf3700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fda680ef610 0x7fda680efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:16.506 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:16.490 7fda6faf3700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda680f0b50 con 0x7fda680f0030 2021-02-10T11:04:16.506 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:16.490 7fda6faf3700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda680e52e0 con 0x7fda681b0470 2021-02-10T11:04:16.506 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:16.490 7fda6faf3700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda680e54d0 con 0x7fda680ef610 2021-02-10T11:04:16.506 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:16.490 7fda6d90f700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fda681b0470 0x7fda681b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:16.506 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:16.490 7fda6d90f700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fda681b0470 0x7fda681b0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:49788/0 (socket says 172.21.2.102:49788) 2021-02-10T11:04:16.506 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:16.490 7fda6d90f700 1 -- 172.21.2.102:0/522875618 learned_addr learned my addr 172.21.2.102:0/522875618 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:04:16.506 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:16.490 7fda6d10e700 1 --2- 172.21.2.102:0/522875618 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fda680ef610 0x7fda680efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:16.506 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:16.490 7fda6d90f700 1 -- 172.21.2.102:0/522875618 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fda680ef610 msgr2=0x7fda680efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:04:16.506 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:16.490 7fda6d90f700 1 --2- 172.21.2.102:0/522875618 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fda680ef610 0x7fda680efa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:16.506 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:16.490 7fda6d90f700 1 -- 172.21.2.102:0/522875618 >> v1:172.21.2.113:6789/0 conn(0x7fda680f0030 legacy=0x7fda680f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:16.506 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:16.490 7fda6d90f700 1 -- 172.21.2.102:0/522875618 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda680f0430 con 0x7fda681b0470 2021-02-10T11:04:16.506 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:16.491 7fda6d90f700 1 --2- 172.21.2.102:0/522875618 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fda681b0470 0x7fda681b0870 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7fda58006700 tx=0x7fda58009b80).ready entity=mon.0 client_cookie=8e2f753b62b8367 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:04:16.507 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:16.491 7fda6c90d700 1 -- 172.21.2.102:0/522875618 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fda5800c0c0 con 0x7fda681b0470 2021-02-10T11:04:16.507 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:16.491 7fda6c90d700 1 -- 172.21.2.102:0/522875618 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda5800c340 con 0x7fda681b0470 2021-02-10T11:04:16.507 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:16.491 7fda6c90d700 1 -- 172.21.2.102:0/522875618 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fda5800c860 con 0x7fda681b0470 2021-02-10T11:04:16.507 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:16.491 7fda6faf3700 1 -- 172.21.2.102:0/522875618 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fda681b0470 msgr2=0x7fda681b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:16.507 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:16.491 7fda6faf3700 1 --2- 172.21.2.102:0/522875618 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fda681b0470 0x7fda681b0870 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7fda58006700 tx=0x7fda58009b80).stop 2021-02-10T11:04:16.507 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:16.492 7fda6faf3700 1 -- 172.21.2.102:0/522875618 shutdown_connections 2021-02-10T11:04:16.507 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:16.492 7fda6faf3700 1 --2- 172.21.2.102:0/522875618 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fda680ef610 0x7fda680efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:16.507 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:16.492 7fda6faf3700 1 --2- 172.21.2.102:0/522875618 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fda681b0470 0x7fda681b0870 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:16.507 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:16.492 7fda6faf3700 1 -- 172.21.2.102:0/522875618 >> 172.21.2.102:0/522875618 conn(0x7fda680ce3e0 msgr2=0x7fda680e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:04:16.507 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:16.492 7fda6faf3700 1 -- 172.21.2.102:0/522875618 shutdown_connections 2021-02-10T11:04:16.507 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:16.492 7fda6faf3700 1 -- 172.21.2.102:0/522875618 wait complete. 2021-02-10T11:04:16.509 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:16.507 7fda6faf3700 1 Processor -- start 2021-02-10T11:04:16.509 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:16.507 7fda6faf3700 1 -- start start 2021-02-10T11:04:16.509 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:16.507 7fda6faf3700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fda680a0420 0x7fda680a0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:16.509 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:16.507 7fda6faf3700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fda68189d00 0x7fda6818a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:16.509 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:16.507 7fda6faf3700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fda6818a720 0x7fda680b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:16.509 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:16.507 7fda6faf3700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda68189780 con 0x7fda68189d00 2021-02-10T11:04:16.509 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:16.507 7fda6faf3700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda6818ab20 con 0x7fda680a0420 2021-02-10T11:04:16.509 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:16.507 7fda6faf3700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda680a1140 con 0x7fda6818a720 2021-02-10T11:04:16.509 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:16.507 7fda6d90f700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fda680a0420 0x7fda680a0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:16.509 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:16.507 7fda6d10e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fda68189d00 0x7fda6818a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:16.509 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:16.507 7fda6d10e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fda68189d00 0x7fda6818a100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:49796/0 (socket says 172.21.2.102:49796) 2021-02-10T11:04:16.509 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:16.507 7fda6d10e700 1 -- 172.21.2.102:0/2782293721 learned_addr learned my addr 172.21.2.102:0/2782293721 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:04:16.509 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:16.507 7fda6e110700 1 --2- 172.21.2.102:0/2782293721 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fda6818a720 0x7fda680b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:16.510 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:16.508 7fda6d10e700 1 -- 172.21.2.102:0/2782293721 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fda6818a720 msgr2=0x7fda680b7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:16.510 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:16.508 7fda6d10e700 1 --2- 172.21.2.102:0/2782293721 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fda6818a720 0x7fda680b7be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:16.510 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:16.508 7fda6d10e700 1 -- 172.21.2.102:0/2782293721 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fda680a0420 msgr2=0x7fda680a0820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:16.510 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:16.508 7fda6d10e700 1 --2- 172.21.2.102:0/2782293721 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fda680a0420 0x7fda680a0820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:16.510 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:16.508 7fda6d10e700 1 -- 172.21.2.102:0/2782293721 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda680b8200 con 0x7fda68189d00 2021-02-10T11:04:16.510 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:16.508 7fda6d10e700 1 --2- 172.21.2.102:0/2782293721 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fda68189d00 0x7fda6818a100 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7fda64006f60 tx=0x7fda6400b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:04:16.510 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:16.508 7fda5e7fc700 1 -- 172.21.2.102:0/2782293721 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fda6400bd60 con 0x7fda68189d00 2021-02-10T11:04:16.510 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:16.508 7fda5e7fc700 1 -- 172.21.2.102:0/2782293721 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda64000c90 con 0x7fda68189d00 2021-02-10T11:04:16.510 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:16.508 7fda5e7fc700 1 -- 172.21.2.102:0/2782293721 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fda64002220 con 0x7fda68189d00 2021-02-10T11:04:16.510 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:16.508 7fda6faf3700 1 -- 172.21.2.102:0/2782293721 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda680b8200 con 0x7fda68189d00 2021-02-10T11:04:16.510 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:16.508 7fda6faf3700 1 -- 172.21.2.102:0/2782293721 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda680b8200 con 0x7fda68189d00 2021-02-10T11:04:16.510 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:16.509 7fda5e7fc700 1 -- 172.21.2.102:0/2782293721 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fda640052e0 con 0x7fda68189d00 2021-02-10T11:04:16.511 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:16.509 7fda5e7fc700 1 --2- 172.21.2.102:0/2782293721 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fda50018730 0x7fda5001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:16.511 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:16.509 7fda5e7fc700 1 -- 172.21.2.102:0/2782293721 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fda640327c0 con 0x7fda68189d00 2021-02-10T11:04:16.511 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:16.509 7fda6d90f700 1 --2- 172.21.2.102:0/2782293721 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fda50018730 0x7fda5001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:16.511 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:16.509 7fda6d90f700 1 --2- 172.21.2.102:0/2782293721 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fda50018730 0x7fda5001ac10 crc :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:04:16.538 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:16.540 7fda6faf3700 1 -- 172.21.2.102:0/2782293721 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda68105ae0 con 0x7fda68189d00 2021-02-10T11:04:16.541 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:16.542 7fda5e7fc700 1 -- 172.21.2.102:0/2782293721 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fda64006b00 con 0x7fda68189d00 2021-02-10T11:04:16.668 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:16.669 7fda6faf3700 1 -- 172.21.2.102:0/2782293721 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7fda68105ae0 con 0x7fda68189d00 2021-02-10T11:04:16.668 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:16.669 7fda5e7fc700 1 -- 172.21.2.102:0/2782293721 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v20) v1 ==== 100+0+10 (secure 0 0 0) 0x7fda640052e0 con 0x7fda68189d00 2021-02-10T11:04:16.669 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:04:16.670 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:16.671 7fda6faf3700 1 -- 172.21.2.102:0/2782293721 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fda50018730 msgr2=0x7fda5001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:16.670 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:16.671 7fda6faf3700 1 --2- 172.21.2.102:0/2782293721 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fda50018730 0x7fda5001ac10 crc :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:16.670 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:16.671 7fda6faf3700 1 -- 172.21.2.102:0/2782293721 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fda68189d00 msgr2=0x7fda6818a100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:16.670 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:16.671 7fda6faf3700 1 --2- 172.21.2.102:0/2782293721 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fda68189d00 0x7fda6818a100 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7fda64006f60 tx=0x7fda6400b970).stop 2021-02-10T11:04:16.670 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:16.671 7fda6faf3700 1 -- 172.21.2.102:0/2782293721 shutdown_connections 2021-02-10T11:04:16.671 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:16.671 7fda6faf3700 1 --2- 172.21.2.102:0/2782293721 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fda6818a720 0x7fda680b7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:16.671 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:16.671 7fda6faf3700 1 --2- 172.21.2.102:0/2782293721 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fda68189d00 0x7fda6818a100 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:16.671 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:16.671 7fda6faf3700 1 --2- 172.21.2.102:0/2782293721 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fda50018730 0x7fda5001ac10 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:16.671 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:16.671 7fda6faf3700 1 --2- 172.21.2.102:0/2782293721 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fda680a0420 0x7fda680a0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:16.671 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:16.671 7fda6faf3700 1 -- 172.21.2.102:0/2782293721 >> 172.21.2.102:0/2782293721 conn(0x7fda68094e10 msgr2=0x7fda680a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:04:16.671 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:16.671 7fda6faf3700 1 -- 172.21.2.102:0/2782293721 shutdown_connections 2021-02-10T11:04:16.671 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:16.671 7fda6faf3700 1 -- 172.21.2.102:0/2782293721 wait complete. 2021-02-10T11:04:16.678 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-02-10T11:04:16.679 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:04:17.104 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:17.089 7f9cd80e2700 1 Processor -- start 2021-02-10T11:04:17.104 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:17.089 7f9cd80e2700 1 -- start start 2021-02-10T11:04:17.104 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:17.089 7f9cd80e2700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9cd00d17f0 0x7f9cd01b31e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:17.105 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:17.089 7f9cd80e2700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f9cd00d1bf0 0x7f9cd01b3800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:17.105 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:17.089 7f9cd80e2700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f9cd01b3e20 0x7f9cd00ceef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:17.105 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:17.089 7f9cd80e2700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9cd00e5eb0 con 0x7f9cd01b3e20 2021-02-10T11:04:17.105 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:17.089 7f9cd80e2700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9cd00f0b20 con 0x7f9cd00d1bf0 2021-02-10T11:04:17.105 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:17.089 7f9cd80e2700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9cd00cf510 con 0x7f9cd00d17f0 2021-02-10T11:04:17.105 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:17.089 7f9cd66ff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f9cd01b3e20 0x7f9cd00ceef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:17.105 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:17.089 7f9cd66ff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f9cd01b3e20 0x7f9cd00ceef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:49802/0 (socket says 172.21.2.102:49802) 2021-02-10T11:04:17.105 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:17.089 7f9cd66ff700 1 -- 172.21.2.102:0/3245682229 learned_addr learned my addr 172.21.2.102:0/3245682229 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:04:17.105 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:17.090 7f9cd66ff700 1 -- 172.21.2.102:0/3245682229 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9cd00d17f0 msgr2=0x7f9cd01b31e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:04:17.105 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:17.090 7f9cd66ff700 1 --2- 172.21.2.102:0/3245682229 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9cd00d17f0 0x7f9cd01b31e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:17.105 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:17.090 7f9cd66ff700 1 -- 172.21.2.102:0/3245682229 >> v2:172.21.2.113:3300/0 conn(0x7f9cd00d1bf0 msgr2=0x7f9cd01b3800 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:04:17.106 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:17.090 7f9cd66ff700 1 --2- 172.21.2.102:0/3245682229 >> v2:172.21.2.113:3300/0 conn(0x7f9cd00d1bf0 0x7f9cd01b3800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:17.106 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:17.090 7f9cd66ff700 1 -- 172.21.2.102:0/3245682229 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cd00cf730 con 0x7f9cd01b3e20 2021-02-10T11:04:17.106 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:17.090 7f9cd66ff700 1 --2- 172.21.2.102:0/3245682229 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9cd01b3e20 0x7f9cd00ceef0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f9cc8006f00 tx=0x7f9cc8009f10).ready entity=mon.0 client_cookie=1b1097285025c89f server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:04:17.106 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:17.090 7f9cd4efc700 1 -- 172.21.2.102:0/3245682229 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9cc800b3b0 con 0x7f9cd01b3e20 2021-02-10T11:04:17.106 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:17.090 7f9cd4efc700 1 -- 172.21.2.102:0/3245682229 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cc800b730 con 0x7f9cd01b3e20 2021-02-10T11:04:17.106 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:17.090 7f9cd4efc700 1 -- 172.21.2.102:0/3245682229 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9cc800bc50 con 0x7f9cd01b3e20 2021-02-10T11:04:17.106 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:17.090 7f9cd80e2700 1 -- 172.21.2.102:0/3245682229 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9cd01b3e20 msgr2=0x7f9cd00ceef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:17.106 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:17.090 7f9cd80e2700 1 --2- 172.21.2.102:0/3245682229 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9cd01b3e20 0x7f9cd00ceef0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f9cc8006f00 tx=0x7f9cc8009f10).stop 2021-02-10T11:04:17.106 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:17.091 7f9cd80e2700 1 -- 172.21.2.102:0/3245682229 shutdown_connections 2021-02-10T11:04:17.106 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:17.091 7f9cd80e2700 1 --2- 172.21.2.102:0/3245682229 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9cd01b3e20 0x7f9cd00ceef0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:17.106 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:17.091 7f9cd80e2700 1 --2- 172.21.2.102:0/3245682229 >> v2:172.21.2.113:3300/0 conn(0x7f9cd00d1bf0 0x7f9cd01b3800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:17.106 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:17.091 7f9cd80e2700 1 --2- 172.21.2.102:0/3245682229 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9cd00d17f0 0x7f9cd01b31e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:17.107 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:17.091 7f9cd80e2700 1 -- 172.21.2.102:0/3245682229 >> 172.21.2.102:0/3245682229 conn(0x7f9cd00fde40 msgr2=0x7f9cd00ce390 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:04:17.107 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:17.091 7f9cd80e2700 1 -- 172.21.2.102:0/3245682229 shutdown_connections 2021-02-10T11:04:17.107 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:17.091 7f9cd80e2700 1 -- 172.21.2.102:0/3245682229 wait complete. 2021-02-10T11:04:17.108 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:17.106 7f9cd80e2700 1 Processor -- start 2021-02-10T11:04:17.108 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:17.106 7f9cd80e2700 1 -- start start 2021-02-10T11:04:17.108 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:17.106 7f9cd80e2700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9cd00ae810 0x7f9cd018ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:17.108 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:17.106 7f9cd80e2700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f9cd019e390 0x7f9cd019e790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:17.108 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:17.106 7f9cd80e2700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9cd0197af0 0x7f9cd0197ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:17.109 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:17.106 7f9cd80e2700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cd019de10 con 0x7f9cd00ae810 2021-02-10T11:04:17.109 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:17.106 7f9cd80e2700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cd018d4d0 con 0x7f9cd019e390 2021-02-10T11:04:17.109 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:17.106 7f9cd80e2700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9cd0198510 con 0x7f9cd0197af0 2021-02-10T11:04:17.109 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:17.106 7f9cd5efe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9cd00ae810 0x7f9cd018ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:17.109 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:17.106 7f9cd5efe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9cd00ae810 0x7f9cd018ceb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:49806/0 (socket says 172.21.2.102:49806) 2021-02-10T11:04:17.109 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:17.106 7f9cd5efe700 1 -- 172.21.2.102:0/1734660225 learned_addr learned my addr 172.21.2.102:0/1734660225 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:04:17.109 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:17.107 7f9cd5efe700 1 -- 172.21.2.102:0/1734660225 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9cd0197af0 msgr2=0x7f9cd0197ef0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:04:17.109 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:17.107 7f9cd5efe700 1 --2- 172.21.2.102:0/1734660225 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9cd0197af0 0x7f9cd0197ef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:17.109 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:17.107 7f9cd5efe700 1 -- 172.21.2.102:0/1734660225 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f9cd019e390 msgr2=0x7f9cd019e790 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:04:17.109 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:17.107 7f9cd5efe700 1 --2- 172.21.2.102:0/1734660225 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f9cd019e390 0x7f9cd019e790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:17.109 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:17.107 7f9cd5efe700 1 -- 172.21.2.102:0/1734660225 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cd019cab0 con 0x7f9cd00ae810 2021-02-10T11:04:17.109 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:17.107 7f9cd5efe700 1 --2- 172.21.2.102:0/1734660225 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9cd00ae810 0x7f9cd018ceb0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7f9ccc00b5c0 tx=0x7f9ccc00b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:04:17.109 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:17.107 7f9cc67fc700 1 -- 172.21.2.102:0/1734660225 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9ccc00bd50 con 0x7f9cd00ae810 2021-02-10T11:04:17.109 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:17.107 7f9cc67fc700 1 -- 172.21.2.102:0/1734660225 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ccc006030 con 0x7f9cd00ae810 2021-02-10T11:04:17.110 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:17.107 7f9cc67fc700 1 -- 172.21.2.102:0/1734660225 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9ccc0054d0 con 0x7f9cd00ae810 2021-02-10T11:04:17.110 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:17.107 7f9cd80e2700 1 -- 172.21.2.102:0/1734660225 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9cd019cab0 con 0x7f9cd00ae810 2021-02-10T11:04:17.110 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:17.107 7f9cd80e2700 1 -- 172.21.2.102:0/1734660225 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9cd019cab0 con 0x7f9cd00ae810 2021-02-10T11:04:17.110 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:17.108 7f9cc67fc700 1 -- 172.21.2.102:0/1734660225 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f9ccc006210 con 0x7f9cd00ae810 2021-02-10T11:04:17.110 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:17.108 7f9cc67fc700 1 --2- 172.21.2.102:0/1734660225 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f9cb80186c0 0x7f9cb801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:17.110 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:17.108 7f9cc67fc700 1 -- 172.21.2.102:0/1734660225 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f9ccc032590 con 0x7f9cd00ae810 2021-02-10T11:04:17.110 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:17.108 7f9cd56fd700 1 --2- 172.21.2.102:0/1734660225 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f9cb80186c0 0x7f9cb801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:17.110 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:17.108 7f9cd56fd700 1 --2- 172.21.2.102:0/1734660225 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f9cb80186c0 0x7f9cb801abc0 crc :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:04:17.138 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:17.139 7f9cd80e2700 1 -- 172.21.2.102:0/1734660225 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9cd00f0b20 con 0x7f9cd00ae810 2021-02-10T11:04:17.140 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:17.141 7f9cc67fc700 1 -- 172.21.2.102:0/1734660225 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f9ccc028300 con 0x7f9cd00ae810 2021-02-10T11:04:17.268 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:17.269 7f9cd80e2700 1 -- 172.21.2.102:0/1734660225 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9cd00f0b20 con 0x7f9cd00ae810 2021-02-10T11:04:17.268 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:17.269 7f9cc67fc700 1 -- 172.21.2.102:0/1734660225 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+2094 (secure 0 0 0) 0x7f9ccc028300 con 0x7f9cd00ae810 2021-02-10T11:04:17.269 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:04:17.269 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":6,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4302,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.102:6835/3297596596","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6834","nonce":3297596596},{"type":"v1","addr":"172.21.2.102:6835","nonce":3297596596}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":4}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:03:43.384709","modified":"2021-02-10 11:03:52.791414","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4226},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4226":{"gid":4226,"name":"b","rank":0,"incarnation":5,"state":"up:active","state_seq":3,"addr":"172.21.2.113:6833/285807165","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6832","nonce":285807165},{"type":"v1","addr":"172.21.2.113:6833","nonce":285807165}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2021-02-10T11:04:17.269 INFO:teuthology.orchestra.run.gibba002.stderr:dumped fsmap epoch 6 2021-02-10T11:04:17.270 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:17.271 7f9cd80e2700 1 -- 172.21.2.102:0/1734660225 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f9cb80186c0 msgr2=0x7f9cb801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:17.271 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:17.271 7f9cd80e2700 1 --2- 172.21.2.102:0/1734660225 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f9cb80186c0 0x7f9cb801abc0 crc :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:17.271 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:17.271 7f9cd80e2700 1 -- 172.21.2.102:0/1734660225 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9cd00ae810 msgr2=0x7f9cd018ceb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:17.271 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:17.271 7f9cd80e2700 1 --2- 172.21.2.102:0/1734660225 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9cd00ae810 0x7f9cd018ceb0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7f9ccc00b5c0 tx=0x7f9ccc00b960).stop 2021-02-10T11:04:17.271 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:17.271 7f9cd80e2700 1 -- 172.21.2.102:0/1734660225 shutdown_connections 2021-02-10T11:04:17.271 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:17.271 7f9cd80e2700 1 --2- 172.21.2.102:0/1734660225 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9cd0197af0 0x7f9cd0197ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:17.271 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:17.271 7f9cd80e2700 1 --2- 172.21.2.102:0/1734660225 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f9cb80186c0 0x7f9cb801abc0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:17.271 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:17.271 7f9cd80e2700 1 --2- 172.21.2.102:0/1734660225 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f9cd019e390 0x7f9cd019e790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:17.271 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:17.271 7f9cd80e2700 1 --2- 172.21.2.102:0/1734660225 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9cd00ae810 0x7f9cd018ceb0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:17.271 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:17.271 7f9cd80e2700 1 -- 172.21.2.102:0/1734660225 >> 172.21.2.102:0/1734660225 conn(0x7f9cd01a3780 msgr2=0x7f9cd0198c50 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:04:17.271 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:17.272 7f9cd80e2700 1 -- 172.21.2.102:0/1734660225 shutdown_connections 2021-02-10T11:04:17.271 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:17.272 7f9cd80e2700 1 -- 172.21.2.102:0/1734660225 wait complete. 2021-02-10T11:04:17.480 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 6, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:03:43.384709', 'modified': '2021-02-10 11:03:52.791414', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4226}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4226': {'gid': 4226, 'name': 'b', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.113:6833/285807165', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.113:6832', 'nonce': 285807165}, {'type': 'v1', 'addr': '172.21.2.113:6833', 'nonce': 285807165}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:04:17.480 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:04:17.480 DEBUG:teuthology.orchestra.run.gibba013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:04:17.879 INFO:teuthology.orchestra.run.gibba013.stdout:{ 2021-02-10T11:04:17.879 INFO:teuthology.orchestra.run.gibba013.stdout: "cluster_fsid": "68db30ea-dbce-4aef-a6df-f01c6fea0513", 2021-02-10T11:04:17.879 INFO:teuthology.orchestra.run.gibba013.stdout: "whoami": 0, 2021-02-10T11:04:17.879 INFO:teuthology.orchestra.run.gibba013.stdout: "id": 4226, 2021-02-10T11:04:17.879 INFO:teuthology.orchestra.run.gibba013.stdout: "want_state": "up:active", 2021-02-10T11:04:17.879 INFO:teuthology.orchestra.run.gibba013.stdout: "state": "up:active", 2021-02-10T11:04:17.879 INFO:teuthology.orchestra.run.gibba013.stdout: "rank_uptime": 26.089388073999999, 2021-02-10T11:04:17.880 INFO:teuthology.orchestra.run.gibba013.stdout: "mdsmap_epoch": 6, 2021-02-10T11:04:17.880 INFO:teuthology.orchestra.run.gibba013.stdout: "osdmap_epoch": 20, 2021-02-10T11:04:17.880 INFO:teuthology.orchestra.run.gibba013.stdout: "osdmap_epoch_barrier": 20, 2021-02-10T11:04:17.880 INFO:teuthology.orchestra.run.gibba013.stdout: "uptime": 30.483823450999999 2021-02-10T11:04:17.880 INFO:teuthology.orchestra.run.gibba013.stdout:} 2021-02-10T11:04:17.883 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "68db30ea-dbce-4aef-a6df-f01c6fea0513", "whoami": 0, "id": 4226, "want_state": "up:active", "state": "up:active", "rank_uptime": 26.089388073999999, "mdsmap_epoch": 6, "osdmap_epoch": 20, "osdmap_epoch_barrier": 20, "uptime": 30.483823450999999 } 2021-02-10T11:04:17.884 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2021-02-10T11:04:17.884 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2021-02-10T11:04:17.884 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.1 at ubuntu@gibba002.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.1... 2021-02-10T11:04:17.884 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:04:17.884 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec mkdir -p /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:04:18.220 DEBUG:teuthology.orchestra.run.gibba002:> sudo modprobe fuse 2021-02-10T11:04:18.564 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:04:18.564 DEBUG:teuthology.orchestra.run.gibba002:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:04:18.906 INFO:teuthology.orchestra.run.gibba002.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:04:18.907 INFO:teuthology.orchestra.run.gibba002.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:04:18.907 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:04:18.907 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:04:18.908 DEBUG:teuthology.orchestra.run.gibba002:> ls /sys/fs/fuse/connections 2021-02-10T11:04:19.244 INFO:teuthology.orchestra.run.gibba002.stdout:43 2021-02-10T11:04:19.246 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [43] 2021-02-10T11:04:19.247 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.1 /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:04:19.266 DEBUG:teuthology.orchestra.run.gibba002:> sudo modprobe fuse 2021-02-10T11:04:19.607 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:04:19.607 DEBUG:teuthology.orchestra.run.gibba002:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:04:19.673 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba002.stderr:ceph-fuse[40446]: starting ceph client 2021-02-10T11:04:19.673 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba002.stderr:2021-02-10 11:04:19.674 7ff015175f80 -1 init, newargv = 0x55bb02448100 newargc=9 2021-02-10T11:04:19.679 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba002.stderr:ceph-fuse[40446]: starting fuse 2021-02-10T11:04:19.962 INFO:teuthology.orchestra.run.gibba002.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:04:19.962 INFO:teuthology.orchestra.run.gibba002.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:04:19.962 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:04:19.963 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:04:19.963 DEBUG:teuthology.orchestra.run.gibba002:> ls /sys/fs/fuse/connections 2021-02-10T11:04:20.308 INFO:teuthology.orchestra.run.gibba002.stdout:43 2021-02-10T11:04:20.308 INFO:teuthology.orchestra.run.gibba002.stdout:44 2021-02-10T11:04:20.309 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43, 44] 2021-02-10T11:04:20.309 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:04:20.309 DEBUG:teuthology.orchestra.run.gibba002:> sudo python3 -c ' 2021-02-10T11:04:20.309 DEBUG:teuthology.orchestra.run.gibba002:> import glob 2021-02-10T11:04:20.309 DEBUG:teuthology.orchestra.run.gibba002:> import re 2021-02-10T11:04:20.309 DEBUG:teuthology.orchestra.run.gibba002:> import os 2021-02-10T11:04:20.309 DEBUG:teuthology.orchestra.run.gibba002:> import subprocess 2021-02-10T11:04:20.309 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:04:20.309 DEBUG:teuthology.orchestra.run.gibba002:> def find_socket(client_name): 2021-02-10T11:04:20.309 DEBUG:teuthology.orchestra.run.gibba002:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2021-02-10T11:04:20.309 DEBUG:teuthology.orchestra.run.gibba002:> files = glob.glob(asok_path) 2021-02-10T11:04:20.309 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:04:20.310 DEBUG:teuthology.orchestra.run.gibba002:> # Given a non-glob path, it better be there 2021-02-10T11:04:20.310 DEBUG:teuthology.orchestra.run.gibba002:> if "*" not in asok_path: 2021-02-10T11:04:20.310 DEBUG:teuthology.orchestra.run.gibba002:> assert(len(files) == 1) 2021-02-10T11:04:20.310 DEBUG:teuthology.orchestra.run.gibba002:> return files[0] 2021-02-10T11:04:20.310 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:04:20.310 DEBUG:teuthology.orchestra.run.gibba002:> for f in files: 2021-02-10T11:04:20.310 DEBUG:teuthology.orchestra.run.gibba002:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2021-02-10T11:04:20.310 DEBUG:teuthology.orchestra.run.gibba002:> if os.path.exists("/proc/{0}".format(pid)): 2021-02-10T11:04:20.310 DEBUG:teuthology.orchestra.run.gibba002:> return f 2021-02-10T11:04:20.310 DEBUG:teuthology.orchestra.run.gibba002:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2021-02-10T11:04:20.310 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:04:20.310 DEBUG:teuthology.orchestra.run.gibba002:> print(find_socket("client.1")) 2021-02-10T11:04:20.310 DEBUG:teuthology.orchestra.run.gibba002:> ' 2021-02-10T11:04:20.674 INFO:teuthology.orchestra.run.gibba002.stdout:/var/run/ceph/ceph-client.1.40446.asok 2021-02-10T11:04:20.676 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.40446.asok 2021-02-10T11:04:20.678 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:04:20.678 DEBUG:teuthology.orchestra.run.gibba002:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.40446.asok status 2021-02-10T11:04:21.071 INFO:teuthology.orchestra.run.gibba002.stdout:{ 2021-02-10T11:04:21.071 INFO:teuthology.orchestra.run.gibba002.stdout: "metadata": { 2021-02-10T11:04:21.072 INFO:teuthology.orchestra.run.gibba002.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:04:21.072 INFO:teuthology.orchestra.run.gibba002.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:04:21.072 INFO:teuthology.orchestra.run.gibba002.stdout: "entity_id": "1", 2021-02-10T11:04:21.072 INFO:teuthology.orchestra.run.gibba002.stdout: "hostname": "gibba002", 2021-02-10T11:04:21.072 INFO:teuthology.orchestra.run.gibba002.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2021-02-10T11:04:21.072 INFO:teuthology.orchestra.run.gibba002.stdout: "pid": "40446", 2021-02-10T11:04:21.072 INFO:teuthology.orchestra.run.gibba002.stdout: "root": "/" 2021-02-10T11:04:21.072 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:21.072 INFO:teuthology.orchestra.run.gibba002.stdout: "dentry_count": 0, 2021-02-10T11:04:21.072 INFO:teuthology.orchestra.run.gibba002.stdout: "dentry_pinned_count": 0, 2021-02-10T11:04:21.072 INFO:teuthology.orchestra.run.gibba002.stdout: "id": 4596, 2021-02-10T11:04:21.072 INFO:teuthology.orchestra.run.gibba002.stdout: "inst": { 2021-02-10T11:04:21.073 INFO:teuthology.orchestra.run.gibba002.stdout: "name": { 2021-02-10T11:04:21.073 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "client", 2021-02-10T11:04:21.073 INFO:teuthology.orchestra.run.gibba002.stdout: "num": 4596 2021-02-10T11:04:21.073 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:21.073 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": { 2021-02-10T11:04:21.073 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:04:21.073 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:0", 2021-02-10T11:04:21.073 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 1964515609 2021-02-10T11:04:21.073 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:04:21.073 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:21.073 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": { 2021-02-10T11:04:21.073 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:04:21.073 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:0", 2021-02-10T11:04:21.074 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 1964515609 2021-02-10T11:04:21.074 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:21.074 INFO:teuthology.orchestra.run.gibba002.stdout: "inst_str": "client.4596 172.21.2.102:0/1964515609", 2021-02-10T11:04:21.074 INFO:teuthology.orchestra.run.gibba002.stdout: "addr_str": "172.21.2.102:0/1964515609", 2021-02-10T11:04:21.074 INFO:teuthology.orchestra.run.gibba002.stdout: "inode_count": 1, 2021-02-10T11:04:21.074 INFO:teuthology.orchestra.run.gibba002.stdout: "mds_epoch": 6, 2021-02-10T11:04:21.074 INFO:teuthology.orchestra.run.gibba002.stdout: "osd_epoch": 20, 2021-02-10T11:04:21.074 INFO:teuthology.orchestra.run.gibba002.stdout: "osd_epoch_barrier": 0, 2021-02-10T11:04:21.074 INFO:teuthology.orchestra.run.gibba002.stdout: "blacklisted": false 2021-02-10T11:04:21.074 INFO:teuthology.orchestra.run.gibba002.stdout:} 2021-02-10T11:04:21.075 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:04:21.501 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:21.486 7f7c27804700 1 Processor -- start 2021-02-10T11:04:21.501 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:21.486 7f7c27804700 1 -- start start 2021-02-10T11:04:21.501 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:21.486 7f7c27804700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7c200e5070 0x7f7c201b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:21.502 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:21.486 7f7c27804700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f7c201b3e40 0x7f7c20080310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:21.502 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:21.486 7f7c27804700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c2007ed80 con 0x7f7c200e5470 2021-02-10T11:04:21.502 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:21.486 7f7c27804700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c200d1790 con 0x7f7c201b3e40 2021-02-10T11:04:21.502 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:21.486 7f7c27804700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c200d1980 con 0x7f7c200e5070 2021-02-10T11:04:21.502 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:21.486 7f7c25e21700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f7c201b3e40 0x7f7c20080310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:21.502 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:21.486 7f7c25e21700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f7c201b3e40 0x7f7c20080310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:49826/0 (socket says 172.21.2.102:49826) 2021-02-10T11:04:21.502 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:21.486 7f7c25e21700 1 -- 172.21.2.102:0/2015517246 learned_addr learned my addr 172.21.2.102:0/2015517246 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:04:21.502 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:21.487 7f7c25e21700 1 -- 172.21.2.102:0/2015517246 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7c200e5070 msgr2=0x7f7c201b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:04:21.502 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:21.487 7f7c25e21700 1 --2- 172.21.2.102:0/2015517246 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7c200e5070 0x7f7c201b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:21.502 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:21.487 7f7c25e21700 1 -- 172.21.2.102:0/2015517246 >> v1:172.21.2.113:6789/0 conn(0x7f7c200e5470 legacy=0x7f7c2007d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:04:21.502 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:21.487 7f7c25e21700 1 -- 172.21.2.102:0/2015517246 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c200d1ba0 con 0x7f7c201b3e40 2021-02-10T11:04:21.502 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:21.487 7f7c25e21700 1 --2- 172.21.2.102:0/2015517246 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7c201b3e40 0x7f7c20080310 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f7c18006f00 tx=0x7f7c18009f10).ready entity=mon.0 client_cookie=9bf11749729dc33f server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:04:21.503 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:21.487 7f7c17fff700 1 -- 172.21.2.102:0/2015517246 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7c1800b3b0 con 0x7f7c201b3e40 2021-02-10T11:04:21.503 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:21.487 7f7c17fff700 1 -- 172.21.2.102:0/2015517246 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c1800b630 con 0x7f7c201b3e40 2021-02-10T11:04:21.503 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:21.487 7f7c17fff700 1 -- 172.21.2.102:0/2015517246 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7c1800bb50 con 0x7f7c201b3e40 2021-02-10T11:04:21.503 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:21.487 7f7c27804700 1 -- 172.21.2.102:0/2015517246 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7c201b3e40 msgr2=0x7f7c20080310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:21.503 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:21.487 7f7c27804700 1 --2- 172.21.2.102:0/2015517246 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7c201b3e40 0x7f7c20080310 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f7c18006f00 tx=0x7f7c18009f10).stop 2021-02-10T11:04:21.503 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:21.489 7f7c27804700 1 -- 172.21.2.102:0/2015517246 shutdown_connections 2021-02-10T11:04:21.503 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:21.489 7f7c27804700 1 --2- 172.21.2.102:0/2015517246 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7c201b3e40 0x7f7c20080310 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:21.503 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:21.489 7f7c27804700 1 --2- 172.21.2.102:0/2015517246 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7c200e5070 0x7f7c201b3820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:21.503 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:21.489 7f7c27804700 1 -- 172.21.2.102:0/2015517246 >> 172.21.2.102:0/2015517246 conn(0x7f7c200ce690 msgr2=0x7f7c200e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:04:21.503 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:21.489 7f7c27804700 1 -- 172.21.2.102:0/2015517246 shutdown_connections 2021-02-10T11:04:21.503 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:21.489 7f7c27804700 1 -- 172.21.2.102:0/2015517246 wait complete. 2021-02-10T11:04:21.505 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:21.503 7f7c27804700 1 Processor -- start 2021-02-10T11:04:21.505 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:21.503 7f7c27804700 1 -- start start 2021-02-10T11:04:21.505 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:21.503 7f7c27804700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7c200b7a90 0x7f7c200b7e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:21.505 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:21.503 7f7c27804700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7c20084180 0x7f7c20084580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:21.505 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:21.503 7f7c27804700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7c20084ba0 0x7f7c20084fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:21.505 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:21.503 7f7c27804700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c2009d0f0 con 0x7f7c200b7a90 2021-02-10T11:04:21.505 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:21.503 7f7c27804700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c20191ed0 con 0x7f7c20084ba0 2021-02-10T11:04:21.505 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:21.503 7f7c27804700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c201a9190 con 0x7f7c20084180 2021-02-10T11:04:21.505 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:21.503 7f7c25620700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7c200b7a90 0x7f7c200b7e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:21.506 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:21.503 7f7c25620700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7c200b7a90 0x7f7c200b7e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:49834/0 (socket says 172.21.2.102:49834) 2021-02-10T11:04:21.506 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:21.503 7f7c25620700 1 -- 172.21.2.102:0/1544891607 learned_addr learned my addr 172.21.2.102:0/1544891607 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:04:21.506 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:21.504 7f7c25620700 1 -- 172.21.2.102:0/1544891607 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7c20084180 msgr2=0x7f7c20084580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:21.506 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:21.504 7f7c25620700 1 --2- 172.21.2.102:0/1544891607 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7c20084180 0x7f7c20084580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:21.506 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:21.504 7f7c25620700 1 -- 172.21.2.102:0/1544891607 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7c20084ba0 msgr2=0x7f7c20084fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:21.506 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:21.504 7f7c25620700 1 --2- 172.21.2.102:0/1544891607 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7c20084ba0 0x7f7c20084fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:21.506 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:21.504 7f7c25620700 1 -- 172.21.2.102:0/1544891607 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c201a93b0 con 0x7f7c200b7a90 2021-02-10T11:04:21.506 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:21.504 7f7c25620700 1 --2- 172.21.2.102:0/1544891607 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7c200b7a90 0x7f7c200b7e90 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7f7c1c006f00 tx=0x7f7c1c00b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:04:21.506 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:21.504 7f7c15ffb700 1 -- 172.21.2.102:0/1544891607 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7c1c006350 con 0x7f7c200b7a90 2021-02-10T11:04:21.506 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:21.504 7f7c15ffb700 1 -- 172.21.2.102:0/1544891607 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c1c00be00 con 0x7f7c200b7a90 2021-02-10T11:04:21.506 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:21.504 7f7c15ffb700 1 -- 172.21.2.102:0/1544891607 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7c1c002020 con 0x7f7c200b7a90 2021-02-10T11:04:21.506 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:21.504 7f7c27804700 1 -- 172.21.2.102:0/1544891607 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c201a93b0 con 0x7f7c200b7a90 2021-02-10T11:04:21.507 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:21.504 7f7c27804700 1 -- 172.21.2.102:0/1544891607 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c201a9940 con 0x7f7c200b7a90 2021-02-10T11:04:21.507 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:21.505 7f7c15ffb700 1 -- 172.21.2.102:0/1544891607 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f7c1c005310 con 0x7f7c200b7a90 2021-02-10T11:04:21.507 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:21.505 7f7c15ffb700 1 --2- 172.21.2.102:0/1544891607 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7c000186c0 0x7f7c0001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:21.507 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:21.505 7f7c15ffb700 1 -- 172.21.2.102:0/1544891607 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f7c1c0325a0 con 0x7f7c200b7a90 2021-02-10T11:04:21.507 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:21.505 7f7c24e1f700 1 --2- 172.21.2.102:0/1544891607 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7c000186c0 0x7f7c0001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:21.507 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:21.505 7f7c24e1f700 1 --2- 172.21.2.102:0/1544891607 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7c000186c0 0x7f7c0001abc0 crc :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:04:21.535 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:21.537 7f7c27804700 1 -- 172.21.2.102:0/1544891607 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c201aa850 con 0x7f7c200b7a90 2021-02-10T11:04:21.537 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:21.540 7f7c15ffb700 1 -- 172.21.2.102:0/1544891607 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f7c1c025fd0 con 0x7f7c200b7a90 2021-02-10T11:04:21.661 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:21.662 7f7c27804700 1 -- 172.21.2.102:0/1544891607 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7c201aa850 con 0x7f7c200b7a90 2021-02-10T11:04:21.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:21.663 7f7c15ffb700 1 -- 172.21.2.102:0/1544891607 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+13993 (secure 0 0 0) 0x7f7c1c025fd0 con 0x7f7c200b7a90 2021-02-10T11:04:21.662 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:04:21.663 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":20,"fsid":"68db30ea-dbce-4aef-a6df-f01c6fea0513","created":"2021-02-10 11:03:06.331557","modified":"2021-02-10 11:03:44.386945","last_up_change":"2021-02-10 11:03:32.690562","last_in_change":"2021-02-10 11:03:32.690562","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:03:37.322519","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:03:41.161563","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2021-02-10 11:03:42.379651","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"be691b4d-a039-408a-af80-9d53f02b6f1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6818","nonce":36088},{"type":"v1","addr":"172.21.2.102:6820","nonce":36088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6822","nonce":36088},{"type":"v1","addr":"172.21.2.102:6824","nonce":36088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6830","nonce":36088},{"type":"v1","addr":"172.21.2.102:6832","nonce":36088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6826","nonce":36088},{"type":"v1","addr":"172.21.2.102:6828","nonce":36088}]},"public_addr":"172.21.2.102:6820/36088","cluster_addr":"172.21.2.102:6824/36088","heartbeat_back_addr":"172.21.2.102:6832/36088","heartbeat_front_addr":"172.21.2.102:6828/36088","state":["exists","up"]},{"osd":1,"uuid":"64e3ccc8-9c69-4d7c-865e-af8e83822635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6819","nonce":36087},{"type":"v1","addr":"172.21.2.102:6821","nonce":36087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6823","nonce":36087},{"type":"v1","addr":"172.21.2.102:6825","nonce":36087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6831","nonce":36087},{"type":"v1","addr":"172.21.2.102:6833","nonce":36087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6827","nonce":36087},{"type":"v1","addr":"172.21.2.102:6829","nonce":36087}]},"public_addr":"172.21.2.102:6821/36087","cluster_addr":"172.21.2.102:6825/36087","heartbeat_back_addr":"172.21.2.102:6833/36087","heartbeat_front_addr":"172.21.2.102:6829/36087","state":["exists","up"]},{"osd":2,"uuid":"153700af-747f-481f-bdf4-d6072d3a842f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6810","nonce":36089},{"type":"v1","addr":"172.21.2.102:6811","nonce":36089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6812","nonce":36089},{"type":"v1","addr":"172.21.2.102:6813","nonce":36089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6816","nonce":36089},{"type":"v1","addr":"172.21.2.102:6817","nonce":36089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6814","nonce":36089},{"type":"v1","addr":"172.21.2.102:6815","nonce":36089}]},"public_addr":"172.21.2.102:6811/36089","cluster_addr":"172.21.2.102:6813/36089","heartbeat_back_addr":"172.21.2.102:6817/36089","heartbeat_front_addr":"172.21.2.102:6815/36089","state":["exists","up"]},{"osd":3,"uuid":"cbc678f4-83a9-45e6-8830-2f7c232b1ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6802","nonce":36090},{"type":"v1","addr":"172.21.2.102:6803","nonce":36090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6804","nonce":36090},{"type":"v1","addr":"172.21.2.102:6805","nonce":36090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6808","nonce":36090},{"type":"v1","addr":"172.21.2.102:6809","nonce":36090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6806","nonce":36090},{"type":"v1","addr":"172.21.2.102:6807","nonce":36090}]},"public_addr":"172.21.2.102:6803/36090","cluster_addr":"172.21.2.102:6805/36090","heartbeat_back_addr":"172.21.2.102:6809/36090","heartbeat_front_addr":"172.21.2.102:6807/36090","state":["exists","up"]},{"osd":4,"uuid":"c01e4008-b258-4f10-afd2-9e7f37214351","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6808","nonce":36368},{"type":"v1","addr":"172.21.2.113:6811","nonce":36368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6814","nonce":36368},{"type":"v1","addr":"172.21.2.113:6818","nonce":36368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6826","nonce":36368},{"type":"v1","addr":"172.21.2.113:6828","nonce":36368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6820","nonce":36368},{"type":"v1","addr":"172.21.2.113:6824","nonce":36368}]},"public_addr":"172.21.2.113:6811/36368","cluster_addr":"172.21.2.113:6818/36368","heartbeat_back_addr":"172.21.2.113:6828/36368","heartbeat_front_addr":"172.21.2.113:6824/36368","state":["exists","up"]},{"osd":5,"uuid":"551f6007-daf8-4a98-8ee0-c2b16bcccb5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6802","nonce":36366},{"type":"v1","addr":"172.21.2.113:6805","nonce":36366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6807","nonce":36366},{"type":"v1","addr":"172.21.2.113:6810","nonce":36366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6819","nonce":36366},{"type":"v1","addr":"172.21.2.113:6822","nonce":36366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6812","nonce":36366},{"type":"v1","addr":"172.21.2.113:6815","nonce":36366}]},"public_addr":"172.21.2.113:6805/36366","cluster_addr":"172.21.2.113:6810/36366","heartbeat_back_addr":"172.21.2.113:6822/36366","heartbeat_front_addr":"172.21.2.113:6815/36366","state":["exists","up"]},{"osd":6,"uuid":"bf67c040-0e50-48be-b102-c3943bc9ebef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6800","nonce":36367},{"type":"v1","addr":"172.21.2.113:6801","nonce":36367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6803","nonce":36367},{"type":"v1","addr":"172.21.2.113:6804","nonce":36367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6813","nonce":36367},{"type":"v1","addr":"172.21.2.113:6816","nonce":36367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6806","nonce":36367},{"type":"v1","addr":"172.21.2.113:6809","nonce":36367}]},"public_addr":"172.21.2.113:6801/36367","cluster_addr":"172.21.2.113:6804/36367","heartbeat_back_addr":"172.21.2.113:6816/36367","heartbeat_front_addr":"172.21.2.113:6809/36367","state":["exists","up"]},{"osd":7,"uuid":"f5461f80-018d-428f-be89-2eb2bcbe7bb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6817","nonce":36365},{"type":"v1","addr":"172.21.2.113:6821","nonce":36365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6823","nonce":36365},{"type":"v1","addr":"172.21.2.113:6825","nonce":36365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6830","nonce":36365},{"type":"v1","addr":"172.21.2.113:6831","nonce":36365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6827","nonce":36365},{"type":"v1","addr":"172.21.2.113:6829","nonce":36365}]},"public_addr":"172.21.2.113:6821/36365","cluster_addr":"172.21.2.113:6825/36365","heartbeat_back_addr":"172.21.2.113:6831/36365","heartbeat_front_addr":"172.21.2.113:6829/36365","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:04:21.663 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:21.665 7f7c27804700 1 -- 172.21.2.102:0/1544891607 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7c000186c0 msgr2=0x7f7c0001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:21.663 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:21.665 7f7c27804700 1 --2- 172.21.2.102:0/1544891607 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7c000186c0 0x7f7c0001abc0 crc :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:21.663 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:21.665 7f7c27804700 1 -- 172.21.2.102:0/1544891607 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7c200b7a90 msgr2=0x7f7c200b7e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:21.663 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:21.665 7f7c27804700 1 --2- 172.21.2.102:0/1544891607 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7c200b7a90 0x7f7c200b7e90 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7f7c1c006f00 tx=0x7f7c1c00b940).stop 2021-02-10T11:04:21.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:21.665 7f7c27804700 1 -- 172.21.2.102:0/1544891607 shutdown_connections 2021-02-10T11:04:21.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:21.665 7f7c27804700 1 --2- 172.21.2.102:0/1544891607 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7c000186c0 0x7f7c0001abc0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:21.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:21.665 7f7c27804700 1 --2- 172.21.2.102:0/1544891607 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7c20084ba0 0x7f7c20084fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:21.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:21.665 7f7c27804700 1 --2- 172.21.2.102:0/1544891607 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7c20084180 0x7f7c20084580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:21.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:21.665 7f7c27804700 1 --2- 172.21.2.102:0/1544891607 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7c200b7a90 0x7f7c200b7e90 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:21.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:21.665 7f7c27804700 1 -- 172.21.2.102:0/1544891607 >> 172.21.2.102:0/1544891607 conn(0x7f7c201b2dc0 msgr2=0x7f7c20092c70 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:04:21.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:21.665 7f7c27804700 1 -- 172.21.2.102:0/1544891607 shutdown_connections 2021-02-10T11:04:21.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:21.665 7f7c27804700 1 -- 172.21.2.102:0/1544891607 wait complete. 2021-02-10T11:04:21.672 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:03:37.322519', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2021-02-10 11:03:41.161563', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2021-02-10 11:03:42.379651', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-02-10T11:04:21.672 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:04:22.101 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:22.085 7f00d9156700 1 Processor -- start 2021-02-10T11:04:22.101 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:22.085 7f00d9156700 1 -- start start 2021-02-10T11:04:22.101 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:22.086 7f00d9156700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f00d41b0470 0x7f00d41b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:22.101 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:22.086 7f00d9156700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f00d40ef610 0x7f00d40efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:22.101 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:22.086 7f00d9156700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00d40f0b50 con 0x7f00d40f0030 2021-02-10T11:04:22.101 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:22.086 7f00d9156700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00d40e52e0 con 0x7f00d40ef610 2021-02-10T11:04:22.101 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:22.086 7f00d9156700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00d40e5490 con 0x7f00d41b0470 2021-02-10T11:04:22.101 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:22.086 7f00d359e700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f00d40f0030 0x7f00d40f10d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:42068/0 (socket says 172.21.2.102:42068) 2021-02-10T11:04:22.101 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:22.086 7f00d359e700 1 -- 172.21.2.102:0/2702320276 learned_addr learned my addr 172.21.2.102:0/2702320276 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:04:22.102 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:22.086 7f00d259c700 1 --2- 172.21.2.102:0/2702320276 >> v2:172.21.2.102:3300/0 conn(0x7f00d40ef610 0x7f00d40efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:22.102 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:22.087 7f00d259c700 1 -- 172.21.2.102:0/2702320276 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f00d41b0470 msgr2=0x7f00d41b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:04:22.102 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:22.087 7f00d259c700 1 --2- 172.21.2.102:0/2702320276 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f00d41b0470 0x7f00d41b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:22.102 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:22.087 7f00d259c700 1 -- 172.21.2.102:0/2702320276 >> v1:172.21.2.102:6789/0 conn(0x7f00d40f0030 legacy=0x7f00d40f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:22.102 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:22.087 7f00d259c700 1 -- 172.21.2.102:0/2702320276 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00d40f0430 con 0x7f00d40ef610 2021-02-10T11:04:22.102 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:22.087 7f00d259c700 1 --2- 172.21.2.102:0/2702320276 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f00d40ef610 0x7f00d40efa10 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7f00c4006650 tx=0x7f00c4009a00).ready entity=mon.0 client_cookie=401564e478d11e3c server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:04:22.102 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:22.087 7f00d1d9b700 1 -- 172.21.2.102:0/2702320276 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f00c4009df0 con 0x7f00d40ef610 2021-02-10T11:04:22.102 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:22.087 7f00d1d9b700 1 -- 172.21.2.102:0/2702320276 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00c40009b0 con 0x7f00d40ef610 2021-02-10T11:04:22.102 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:22.087 7f00d1d9b700 1 -- 172.21.2.102:0/2702320276 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f00c400c2b0 con 0x7f00d40ef610 2021-02-10T11:04:22.102 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:22.087 7f00d9156700 1 -- 172.21.2.102:0/2702320276 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f00d40ef610 msgr2=0x7f00d40efa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:22.102 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:22.087 7f00d9156700 1 --2- 172.21.2.102:0/2702320276 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f00d40ef610 0x7f00d40efa10 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7f00c4006650 tx=0x7f00c4009a00).stop 2021-02-10T11:04:22.102 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:22.088 7f00d9156700 1 -- 172.21.2.102:0/2702320276 shutdown_connections 2021-02-10T11:04:22.102 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:22.088 7f00d9156700 1 --2- 172.21.2.102:0/2702320276 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f00d40ef610 0x7f00d40efa10 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:22.103 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:22.088 7f00d9156700 1 --2- 172.21.2.102:0/2702320276 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f00d41b0470 0x7f00d41b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:22.103 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:22.088 7f00d9156700 1 -- 172.21.2.102:0/2702320276 >> 172.21.2.102:0/2702320276 conn(0x7f00d40ce3e0 msgr2=0x7f00d40e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:04:22.103 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:22.088 7f00d9156700 1 -- 172.21.2.102:0/2702320276 shutdown_connections 2021-02-10T11:04:22.103 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:22.088 7f00d9156700 1 -- 172.21.2.102:0/2702320276 wait complete. 2021-02-10T11:04:22.104 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:22.102 7f00d9156700 1 Processor -- start 2021-02-10T11:04:22.104 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:22.102 7f00d9156700 1 -- start start 2021-02-10T11:04:22.104 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:22.103 7f00d9156700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f00d4094fb0 0x7f00d409e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:22.104 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:22.103 7f00d9156700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f00d40c2be0 0x7f00d40c2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:22.105 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:22.103 7f00d9156700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f00d40c3600 0x7f00d40a9c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:22.105 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:22.103 7f00d9156700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00d409ebf0 con 0x7f00d40c3600 2021-02-10T11:04:22.105 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:22.103 7f00d9156700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00d40aa2a0 con 0x7f00d4094fb0 2021-02-10T11:04:22.105 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:22.103 7f00d9156700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00d40aa4c0 con 0x7f00d40c2be0 2021-02-10T11:04:22.105 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:22.103 7f00d359e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f00d40c3600 0x7f00d40a9c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:22.105 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:22.103 7f00d359e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f00d40c3600 0x7f00d40a9c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:49850/0 (socket says 172.21.2.102:49850) 2021-02-10T11:04:22.105 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:22.103 7f00d359e700 1 -- 172.21.2.102:0/3193461273 learned_addr learned my addr 172.21.2.102:0/3193461273 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:04:22.105 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:22.103 7f00d359e700 1 -- 172.21.2.102:0/3193461273 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f00d40c2be0 msgr2=0x7f00d40c2fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:22.105 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:22.103 7f00d359e700 1 --2- 172.21.2.102:0/3193461273 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f00d40c2be0 0x7f00d40c2fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:22.105 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:22.103 7f00d359e700 1 -- 172.21.2.102:0/3193461273 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f00d4094fb0 msgr2=0x7f00d409e5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:22.105 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:22.103 7f00d359e700 1 --2- 172.21.2.102:0/3193461273 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f00d4094fb0 0x7f00d409e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:22.105 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:22.103 7f00d359e700 1 -- 172.21.2.102:0/3193461273 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00d40aa6e0 con 0x7f00d40c3600 2021-02-10T11:04:22.105 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:22.105 7f00d359e700 1 --2- 172.21.2.102:0/3193461273 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f00d40c3600 0x7f00d40a9c80 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7f00cc00a940 tx=0x7f00cc00ace0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:04:22.106 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:22.105 7f00cb7fe700 1 -- 172.21.2.102:0/3193461273 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f00cc001270 con 0x7f00d40c3600 2021-02-10T11:04:22.106 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:22.105 7f00cb7fe700 1 -- 172.21.2.102:0/3193461273 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00cc0070b0 con 0x7f00d40c3600 2021-02-10T11:04:22.106 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:22.105 7f00d9156700 1 -- 172.21.2.102:0/3193461273 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00d40aa6e0 con 0x7f00d40c3600 2021-02-10T11:04:22.106 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:22.105 7f00d9156700 1 -- 172.21.2.102:0/3193461273 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00d40aac70 con 0x7f00d40c3600 2021-02-10T11:04:22.106 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:22.105 7f00cb7fe700 1 -- 172.21.2.102:0/3193461273 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f00cc007540 con 0x7f00d40c3600 2021-02-10T11:04:22.106 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:22.105 7f00cb7fe700 1 -- 172.21.2.102:0/3193461273 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f00cc007290 con 0x7f00d40c3600 2021-02-10T11:04:22.106 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:22.105 7f00cb7fe700 1 --2- 172.21.2.102:0/3193461273 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f00b8018730 0x7f00b801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:22.106 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:22.105 7f00cb7fe700 1 -- 172.21.2.102:0/3193461273 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f00cc032b20 con 0x7f00d40c3600 2021-02-10T11:04:22.106 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:22.105 7f00d2d9d700 1 --2- 172.21.2.102:0/3193461273 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f00b8018730 0x7f00b801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:22.106 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:22.105 7f00d2d9d700 1 --2- 172.21.2.102:0/3193461273 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f00b8018730 0x7f00b801ac10 crc :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:04:22.135 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:22.136 7f00d9156700 1 -- 172.21.2.102:0/3193461273 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00d40a9750 con 0x7f00d40c3600 2021-02-10T11:04:22.137 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:22.138 7f00cb7fe700 1 -- 172.21.2.102:0/3193461273 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f00cc001270 con 0x7f00d40c3600 2021-02-10T11:04:22.264 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:22.266 7f00d9156700 1 -- 172.21.2.102:0/3193461273 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f00d40a9750 con 0x7f00d40c3600 2021-02-10T11:04:22.265 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:22.266 7f00cb7fe700 1 -- 172.21.2.102:0/3193461273 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v20) v1 ==== 92+0+10 (secure 0 0 0) 0x7f00cc0070b0 con 0x7f00d40c3600 2021-02-10T11:04:22.265 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:04:22.267 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:22.268 7f00d9156700 1 -- 172.21.2.102:0/3193461273 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f00b8018730 msgr2=0x7f00b801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:22.267 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:22.268 7f00d9156700 1 --2- 172.21.2.102:0/3193461273 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f00b8018730 0x7f00b801ac10 crc :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:22.267 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:22.268 7f00d9156700 1 -- 172.21.2.102:0/3193461273 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f00d40c3600 msgr2=0x7f00d40a9c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:22.267 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:22.268 7f00d9156700 1 --2- 172.21.2.102:0/3193461273 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f00d40c3600 0x7f00d40a9c80 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7f00cc00a940 tx=0x7f00cc00ace0).stop 2021-02-10T11:04:22.267 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:22.268 7f00d9156700 1 -- 172.21.2.102:0/3193461273 shutdown_connections 2021-02-10T11:04:22.267 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:22.268 7f00d9156700 1 --2- 172.21.2.102:0/3193461273 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f00d40c3600 0x7f00d40a9c80 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:22.267 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:22.268 7f00d9156700 1 --2- 172.21.2.102:0/3193461273 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f00d40c2be0 0x7f00d40c2fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:22.267 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:22.268 7f00d9156700 1 --2- 172.21.2.102:0/3193461273 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f00b8018730 0x7f00b801ac10 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:22.267 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:22.268 7f00d9156700 1 --2- 172.21.2.102:0/3193461273 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f00d4094fb0 0x7f00d409e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:22.267 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:22.268 7f00d9156700 1 -- 172.21.2.102:0/3193461273 >> 172.21.2.102:0/3193461273 conn(0x7f00d409f630 msgr2=0x7f00d40a0130 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:04:22.267 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:22.268 7f00d9156700 1 -- 172.21.2.102:0/3193461273 shutdown_connections 2021-02-10T11:04:22.268 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:22.268 7f00d9156700 1 -- 172.21.2.102:0/3193461273 wait complete. 2021-02-10T11:04:22.275 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-02-10T11:04:22.702 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:22.687 7fd4edcfe700 1 Processor -- start 2021-02-10T11:04:22.702 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:22.687 7fd4edcfe700 1 -- start start 2021-02-10T11:04:22.703 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:22.688 7fd4edcfe700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fd4e81b0470 0x7fd4e81b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:22.703 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:22.688 7fd4edcfe700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fd4e80ef610 0x7fd4e80efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:22.703 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:22.688 7fd4edcfe700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd4e80f0030 0x7fd4e80f0430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:22.703 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:22.688 7fd4edcfe700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4e80e5ea0 con 0x7fd4e81b0470 2021-02-10T11:04:22.703 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:22.688 7fd4edcfe700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4e80f0b50 con 0x7fd4e80ef610 2021-02-10T11:04:22.703 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:22.688 7fd4edcfe700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4e80e52e0 con 0x7fd4e80f0030 2021-02-10T11:04:22.703 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:22.688 7fd4e77fe700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fd4e81b0470 0x7fd4e81b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:22.703 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:22.688 7fd4e77fe700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fd4e81b0470 0x7fd4e81b0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:49854/0 (socket says 172.21.2.102:49854) 2021-02-10T11:04:22.703 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:22.688 7fd4e77fe700 1 -- 172.21.2.102:0/2326270684 learned_addr learned my addr 172.21.2.102:0/2326270684 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:04:22.703 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:22.688 7fd4e77fe700 1 -- 172.21.2.102:0/2326270684 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd4e80f0030 msgr2=0x7fd4e80f0430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:04:22.703 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:22.688 7fd4e77fe700 1 --2- 172.21.2.102:0/2326270684 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd4e80f0030 0x7fd4e80f0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:22.703 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:22.688 7fd4e77fe700 1 -- 172.21.2.102:0/2326270684 >> v2:172.21.2.113:3300/0 conn(0x7fd4e80ef610 msgr2=0x7fd4e80efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:04:22.703 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:22.688 7fd4e77fe700 1 --2- 172.21.2.102:0/2326270684 >> v2:172.21.2.113:3300/0 conn(0x7fd4e80ef610 0x7fd4e80efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:22.704 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:22.688 7fd4e77fe700 1 -- 172.21.2.102:0/2326270684 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4e80f10d0 con 0x7fd4e81b0470 2021-02-10T11:04:22.704 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:22.689 7fd4e77fe700 1 --2- 172.21.2.102:0/2326270684 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd4e81b0470 0x7fd4e81b0870 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7fd4d0006700 tx=0x7fd4d0009b80).ready entity=mon.0 client_cookie=2d3660c7e53f037d server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:04:22.704 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:22.689 7fd4e67fc700 1 -- 172.21.2.102:0/2326270684 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd4d000c0c0 con 0x7fd4e81b0470 2021-02-10T11:04:22.704 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:22.689 7fd4e67fc700 1 -- 172.21.2.102:0/2326270684 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4d000c340 con 0x7fd4e81b0470 2021-02-10T11:04:22.704 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:22.689 7fd4e67fc700 1 -- 172.21.2.102:0/2326270684 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd4d000c860 con 0x7fd4e81b0470 2021-02-10T11:04:22.704 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:22.689 7fd4edcfe700 1 -- 172.21.2.102:0/2326270684 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd4e81b0470 msgr2=0x7fd4e81b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:22.704 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:22.689 7fd4edcfe700 1 --2- 172.21.2.102:0/2326270684 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd4e81b0470 0x7fd4e81b0870 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7fd4d0006700 tx=0x7fd4d0009b80).stop 2021-02-10T11:04:22.704 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:22.689 7fd4edcfe700 1 -- 172.21.2.102:0/2326270684 shutdown_connections 2021-02-10T11:04:22.704 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:22.689 7fd4edcfe700 1 --2- 172.21.2.102:0/2326270684 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd4e80f0030 0x7fd4e80f0430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:22.704 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:22.689 7fd4edcfe700 1 --2- 172.21.2.102:0/2326270684 >> v2:172.21.2.113:3300/0 conn(0x7fd4e80ef610 0x7fd4e80efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:22.704 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:22.689 7fd4edcfe700 1 --2- 172.21.2.102:0/2326270684 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd4e81b0470 0x7fd4e81b0870 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:22.704 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:22.689 7fd4edcfe700 1 -- 172.21.2.102:0/2326270684 >> 172.21.2.102:0/2326270684 conn(0x7fd4e80ce3e0 msgr2=0x7fd4e80e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:04:22.704 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:22.689 7fd4edcfe700 1 -- 172.21.2.102:0/2326270684 shutdown_connections 2021-02-10T11:04:22.704 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:22.689 7fd4edcfe700 1 -- 172.21.2.102:0/2326270684 wait complete. 2021-02-10T11:04:22.706 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:22.704 7fd4edcfe700 1 Processor -- start 2021-02-10T11:04:22.706 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:22.704 7fd4edcfe700 1 -- start start 2021-02-10T11:04:22.706 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:22.704 7fd4edcfe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd4e80b20b0 0x7fd4e80ac970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:22.706 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:22.704 7fd4edcfe700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd4e80b24b0 0x7fd4e8191dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:22.706 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:22.704 7fd4edcfe700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd4e80acf90 0x7fd4e81923e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:22.706 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:22.704 7fd4edcfe700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4e80b1b30 con 0x7fd4e80b20b0 2021-02-10T11:04:22.706 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:22.704 7fd4edcfe700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4e81a04e0 con 0x7fd4e80acf90 2021-02-10T11:04:22.707 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:22.704 7fd4edcfe700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4e81a06c0 con 0x7fd4e80b24b0 2021-02-10T11:04:22.707 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:22.704 7fd4e77fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd4e80b20b0 0x7fd4e80ac970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:22.707 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:22.705 7fd4e77fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd4e80b20b0 0x7fd4e80ac970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:49858/0 (socket says 172.21.2.102:49858) 2021-02-10T11:04:22.707 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:22.705 7fd4e77fe700 1 -- 172.21.2.102:0/2545776772 learned_addr learned my addr 172.21.2.102:0/2545776772 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:04:22.707 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:22.705 7fd4e77fe700 1 -- 172.21.2.102:0/2545776772 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd4e80b24b0 msgr2=0x7fd4e8191dc0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:04:22.707 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:22.705 7fd4e77fe700 1 --2- 172.21.2.102:0/2545776772 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd4e80b24b0 0x7fd4e8191dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:22.707 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:22.705 7fd4e77fe700 1 -- 172.21.2.102:0/2545776772 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd4e80acf90 msgr2=0x7fd4e81923e0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:04:22.707 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:22.705 7fd4e77fe700 1 --2- 172.21.2.102:0/2545776772 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd4e80acf90 0x7fd4e81923e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:22.707 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:22.705 7fd4e77fe700 1 -- 172.21.2.102:0/2545776772 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4e8192be0 con 0x7fd4e80b20b0 2021-02-10T11:04:22.707 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:22.705 7fd4e77fe700 1 --2- 172.21.2.102:0/2545776772 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd4e80b20b0 0x7fd4e80ac970 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7fd4d000b5e0 tx=0x7fd4d000b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:04:22.707 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:22.705 7fd4dffff700 1 -- 172.21.2.102:0/2545776772 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd4d000bd70 con 0x7fd4e80b20b0 2021-02-10T11:04:22.707 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:22.705 7fd4dffff700 1 -- 172.21.2.102:0/2545776772 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4d0002030 con 0x7fd4e80b20b0 2021-02-10T11:04:22.707 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:22.705 7fd4dffff700 1 -- 172.21.2.102:0/2545776772 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd4d0002550 con 0x7fd4e80b20b0 2021-02-10T11:04:22.708 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:22.705 7fd4edcfe700 1 -- 172.21.2.102:0/2545776772 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4e8192be0 con 0x7fd4e80b20b0 2021-02-10T11:04:22.708 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:22.705 7fd4edcfe700 1 -- 172.21.2.102:0/2545776772 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4e8193170 con 0x7fd4e80b20b0 2021-02-10T11:04:22.708 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:22.706 7fd4dffff700 1 -- 172.21.2.102:0/2545776772 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd4d0000c90 con 0x7fd4e80b20b0 2021-02-10T11:04:22.708 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:22.706 7fd4dffff700 1 --2- 172.21.2.102:0/2545776772 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd4cc0186c0 0x7fd4cc01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:22.708 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:22.706 7fd4dffff700 1 -- 172.21.2.102:0/2545776772 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fd4d00325e0 con 0x7fd4e80b20b0 2021-02-10T11:04:22.708 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:22.706 7fd4e6ffd700 1 --2- 172.21.2.102:0/2545776772 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd4cc0186c0 0x7fd4cc01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:22.708 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:22.707 7fd4e6ffd700 1 --2- 172.21.2.102:0/2545776772 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd4cc0186c0 0x7fd4cc01abc0 crc :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:04:22.736 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:22.738 7fd4edcfe700 1 -- 172.21.2.102:0/2545776772 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4e8105af0 con 0x7fd4e80b20b0 2021-02-10T11:04:22.739 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:22.740 7fd4dffff700 1 -- 172.21.2.102:0/2545776772 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd4d00009b0 con 0x7fd4e80b20b0 2021-02-10T11:04:22.867 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:22.868 7fd4edcfe700 1 -- 172.21.2.102:0/2545776772 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7fd4e80f0b50 con 0x7fd4e80b20b0 2021-02-10T11:04:22.867 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:22.868 7fd4dffff700 1 -- 172.21.2.102:0/2545776772 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v20) v1 ==== 104+0+10 (secure 0 0 0) 0x7fd4d0000c90 con 0x7fd4e80b20b0 2021-02-10T11:04:22.868 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:04:22.869 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:22.871 7fd4edcfe700 1 -- 172.21.2.102:0/2545776772 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd4cc0186c0 msgr2=0x7fd4cc01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:22.869 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:22.871 7fd4edcfe700 1 --2- 172.21.2.102:0/2545776772 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd4cc0186c0 0x7fd4cc01abc0 crc :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:22.869 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:22.871 7fd4edcfe700 1 -- 172.21.2.102:0/2545776772 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd4e80b20b0 msgr2=0x7fd4e80ac970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:22.870 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:22.871 7fd4edcfe700 1 --2- 172.21.2.102:0/2545776772 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd4e80b20b0 0x7fd4e80ac970 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7fd4d000b5e0 tx=0x7fd4d000b980).stop 2021-02-10T11:04:22.870 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:22.871 7fd4edcfe700 1 -- 172.21.2.102:0/2545776772 shutdown_connections 2021-02-10T11:04:22.870 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:22.871 7fd4edcfe700 1 --2- 172.21.2.102:0/2545776772 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd4cc0186c0 0x7fd4cc01abc0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:22.870 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:22.871 7fd4edcfe700 1 --2- 172.21.2.102:0/2545776772 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd4e80acf90 0x7fd4e81923e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:22.870 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:22.871 7fd4edcfe700 1 --2- 172.21.2.102:0/2545776772 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd4e80b24b0 0x7fd4e8191dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:22.870 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:22.871 7fd4edcfe700 1 --2- 172.21.2.102:0/2545776772 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd4e80b20b0 0x7fd4e80ac970 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:22.870 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:22.871 7fd4edcfe700 1 -- 172.21.2.102:0/2545776772 >> 172.21.2.102:0/2545776772 conn(0x7fd4e80bf4e0 msgr2=0x7fd4e80be3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:04:22.870 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:22.871 7fd4edcfe700 1 -- 172.21.2.102:0/2545776772 shutdown_connections 2021-02-10T11:04:22.870 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:22.871 7fd4edcfe700 1 -- 172.21.2.102:0/2545776772 wait complete. 2021-02-10T11:04:22.877 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-02-10T11:04:23.305 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:23.290 7f7e1f46e700 1 Processor -- start 2021-02-10T11:04:23.306 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:23.290 7f7e1f46e700 1 -- start start 2021-02-10T11:04:23.306 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:23.291 7f7e1f46e700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f7e181b0470 0x7f7e181b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:23.306 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:23.291 7f7e1f46e700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f7e180ef610 0x7f7e180efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:23.306 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:23.291 7f7e1f46e700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e180f0b50 con 0x7f7e180f0030 2021-02-10T11:04:23.306 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:23.291 7f7e1f46e700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e180e52e0 con 0x7f7e180ef610 2021-02-10T11:04:23.306 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:23.291 7f7e1f46e700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e180e5490 con 0x7f7e181b0470 2021-02-10T11:04:23.306 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:23.291 7f7e1ca89700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f7e180ef610 0x7f7e180efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:23.306 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:23.291 7f7e1ca89700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f7e180ef610 0x7f7e180efa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:49866/0 (socket says 172.21.2.102:49866) 2021-02-10T11:04:23.306 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:23.291 7f7e1da8b700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f7e180f0030 0x7f7e180f10d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:42094/0 (socket says 172.21.2.102:42094) 2021-02-10T11:04:23.307 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:23.291 7f7e1ca89700 1 -- 172.21.2.102:0/17132245 learned_addr learned my addr 172.21.2.102:0/17132245 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:04:23.307 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:23.291 7f7e1d28a700 1 --2- 172.21.2.102:0/17132245 >> v2:172.21.2.113:3300/0 conn(0x7f7e181b0470 0x7f7e181b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:23.307 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:23.291 7f7e1ca89700 1 -- 172.21.2.102:0/17132245 >> v2:172.21.2.113:3300/0 conn(0x7f7e181b0470 msgr2=0x7f7e181b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:04:23.307 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:23.291 7f7e1ca89700 1 --2- 172.21.2.102:0/17132245 >> v2:172.21.2.113:3300/0 conn(0x7f7e181b0470 0x7f7e181b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:23.307 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:23.291 7f7e1ca89700 1 -- 172.21.2.102:0/17132245 >> v1:172.21.2.102:6789/0 conn(0x7f7e180f0030 legacy=0x7f7e180f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:23.307 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:23.291 7f7e1ca89700 1 -- 172.21.2.102:0/17132245 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e180f0430 con 0x7f7e180ef610 2021-02-10T11:04:23.307 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:23.291 7f7e1ca89700 1 --2- 172.21.2.102:0/17132245 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7e180ef610 0x7f7e180efa10 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f7e10006a80 tx=0x7f7e10009f00).ready entity=mon.0 client_cookie=75ba6d7fedede801 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:04:23.307 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:23.292 7f7e0ffff700 1 -- 172.21.2.102:0/17132245 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7e1000b5c0 con 0x7f7e180ef610 2021-02-10T11:04:23.307 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:23.292 7f7e0ffff700 1 -- 172.21.2.102:0/17132245 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e1000b8c0 con 0x7f7e180ef610 2021-02-10T11:04:23.307 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:23.292 7f7e0ffff700 1 -- 172.21.2.102:0/17132245 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7e1000bda0 con 0x7f7e180ef610 2021-02-10T11:04:23.307 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:23.292 7f7e1f46e700 1 -- 172.21.2.102:0/17132245 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7e180ef610 msgr2=0x7f7e180efa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:23.307 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:23.292 7f7e1f46e700 1 --2- 172.21.2.102:0/17132245 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7e180ef610 0x7f7e180efa10 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f7e10006a80 tx=0x7f7e10009f00).stop 2021-02-10T11:04:23.307 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:23.292 7f7e1f46e700 1 -- 172.21.2.102:0/17132245 shutdown_connections 2021-02-10T11:04:23.308 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:23.292 7f7e1f46e700 1 --2- 172.21.2.102:0/17132245 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7e180ef610 0x7f7e180efa10 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:23.308 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:23.292 7f7e1f46e700 1 --2- 172.21.2.102:0/17132245 >> v2:172.21.2.113:3300/0 conn(0x7f7e181b0470 0x7f7e181b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:23.308 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:23.292 7f7e1f46e700 1 -- 172.21.2.102:0/17132245 >> 172.21.2.102:0/17132245 conn(0x7f7e180ce3e0 msgr2=0x7f7e180e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:04:23.308 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:23.292 7f7e1f46e700 1 -- 172.21.2.102:0/17132245 shutdown_connections 2021-02-10T11:04:23.308 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:23.292 7f7e1f46e700 1 -- 172.21.2.102:0/17132245 wait complete. 2021-02-10T11:04:23.308 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:23.307 7f7e1f46e700 1 Processor -- start 2021-02-10T11:04:23.308 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:23.307 7f7e1f46e700 1 -- start start 2021-02-10T11:04:23.309 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:23.307 7f7e1f46e700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7e1808ece0 0x7f7e180a6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:23.309 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:23.307 7f7e1f46e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7e180a6860 0x7f7e180a6c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:23.309 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:23.307 7f7e1f46e700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7e180a7280 0x7f7e180a4590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:23.310 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:23.307 7f7e1f46e700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e180a5cc0 con 0x7f7e180a6860 2021-02-10T11:04:23.310 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:23.307 7f7e1f46e700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e1808f0e0 con 0x7f7e180a7280 2021-02-10T11:04:23.310 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:23.307 7f7e1f46e700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e1808f2c0 con 0x7f7e1808ece0 2021-02-10T11:04:23.310 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:23.307 7f7e1ca89700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7e180a6860 0x7f7e180a6c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:23.310 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:23.308 7f7e1ca89700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7e180a6860 0x7f7e180a6c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:49872/0 (socket says 172.21.2.102:49872) 2021-02-10T11:04:23.310 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:23.308 7f7e1ca89700 1 -- 172.21.2.102:0/1460890310 learned_addr learned my addr 172.21.2.102:0/1460890310 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:04:23.310 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:23.308 7f7e1da8b700 1 --2- 172.21.2.102:0/1460890310 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7e180a7280 0x7f7e180a4590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:23.310 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:23.308 7f7e1d28a700 1 --2- 172.21.2.102:0/1460890310 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7e1808ece0 0x7f7e180a6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:23.310 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:23.308 7f7e1ca89700 1 -- 172.21.2.102:0/1460890310 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7e1808ece0 msgr2=0x7f7e180a6240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:23.311 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:23.308 7f7e1ca89700 1 --2- 172.21.2.102:0/1460890310 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7e1808ece0 0x7f7e180a6240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:23.311 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:23.308 7f7e1ca89700 1 -- 172.21.2.102:0/1460890310 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7e180a7280 msgr2=0x7f7e180a4590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:23.311 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:23.308 7f7e1ca89700 1 --2- 172.21.2.102:0/1460890310 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7e180a7280 0x7f7e180a4590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:23.311 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:23.308 7f7e1ca89700 1 -- 172.21.2.102:0/1460890310 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e180a4bb0 con 0x7f7e180a6860 2021-02-10T11:04:23.311 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:23.308 7f7e1ca89700 1 --2- 172.21.2.102:0/1460890310 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7e180a6860 0x7f7e180a6c60 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7f7e10006a90 tx=0x7f7e1000bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:04:23.311 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:23.308 7f7e0dffb700 1 -- 172.21.2.102:0/1460890310 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7e10000d30 con 0x7f7e180a6860 2021-02-10T11:04:23.311 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:23.308 7f7e0dffb700 1 -- 172.21.2.102:0/1460890310 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e1000c2f0 con 0x7f7e180a6860 2021-02-10T11:04:23.311 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:23.308 7f7e1f46e700 1 -- 172.21.2.102:0/1460890310 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e180a4bb0 con 0x7f7e180a6860 2021-02-10T11:04:23.311 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:23.308 7f7e0dffb700 1 -- 172.21.2.102:0/1460890310 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7e1000c740 con 0x7f7e180a6860 2021-02-10T11:04:23.311 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:23.308 7f7e1f46e700 1 -- 172.21.2.102:0/1460890310 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e180a4bb0 con 0x7f7e180a6860 2021-02-10T11:04:23.311 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:23.309 7f7e0dffb700 1 -- 172.21.2.102:0/1460890310 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f7e10007000 con 0x7f7e180a6860 2021-02-10T11:04:23.311 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:23.309 7f7e0dffb700 1 --2- 172.21.2.102:0/1460890310 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7df8018730 0x7f7df801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:23.311 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:23.309 7f7e0dffb700 1 -- 172.21.2.102:0/1460890310 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f7e100329d0 con 0x7f7e180a6860 2021-02-10T11:04:23.311 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:23.309 7f7e1d28a700 1 --2- 172.21.2.102:0/1460890310 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7df8018730 0x7f7df801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:23.312 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:23.309 7f7e1d28a700 1 --2- 172.21.2.102:0/1460890310 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7df8018730 0x7f7df801ac10 crc :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:04:23.339 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:23.341 7f7e1f46e700 1 -- 172.21.2.102:0/1460890310 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e18105ae0 con 0x7f7e180a6860 2021-02-10T11:04:23.341 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:23.343 7f7e0dffb700 1 -- 172.21.2.102:0/1460890310 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f7e1001f890 con 0x7f7e180a6860 2021-02-10T11:04:23.469 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:23.470 7f7e1f46e700 1 -- 172.21.2.102:0/1460890310 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7f7e180f0b50 con 0x7f7e180a6860 2021-02-10T11:04:23.470 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:23.472 7f7e0dffb700 1 -- 172.21.2.102:0/1460890310 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v20) v1 ==== 100+0+10 (secure 0 0 0) 0x7f7e10007000 con 0x7f7e180a6860 2021-02-10T11:04:23.470 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:04:23.471 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:23.472 7f7e1f46e700 1 -- 172.21.2.102:0/1460890310 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7df8018730 msgr2=0x7f7df801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:23.471 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:23.472 7f7e1f46e700 1 --2- 172.21.2.102:0/1460890310 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7df8018730 0x7f7df801ac10 crc :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:23.471 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:23.472 7f7e1f46e700 1 -- 172.21.2.102:0/1460890310 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7e180a6860 msgr2=0x7f7e180a6c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:23.471 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:23.472 7f7e1f46e700 1 --2- 172.21.2.102:0/1460890310 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7e180a6860 0x7f7e180a6c60 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7f7e10006a90 tx=0x7f7e1000bd10).stop 2021-02-10T11:04:23.472 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:23.473 7f7e1f46e700 1 -- 172.21.2.102:0/1460890310 shutdown_connections 2021-02-10T11:04:23.472 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:23.473 7f7e1f46e700 1 --2- 172.21.2.102:0/1460890310 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7df8018730 0x7f7df801ac10 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:23.472 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:23.473 7f7e1f46e700 1 --2- 172.21.2.102:0/1460890310 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7e180a7280 0x7f7e180a4590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:23.472 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:23.473 7f7e1f46e700 1 --2- 172.21.2.102:0/1460890310 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7e180a6860 0x7f7e180a6c60 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:23.472 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:23.473 7f7e1f46e700 1 --2- 172.21.2.102:0/1460890310 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7e1808ece0 0x7f7e180a6240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:23.472 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:23.473 7f7e1f46e700 1 -- 172.21.2.102:0/1460890310 >> 172.21.2.102:0/1460890310 conn(0x7f7e1808ffe0 msgr2=0x7f7e180bf050 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:04:23.472 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:23.473 7f7e1f46e700 1 -- 172.21.2.102:0/1460890310 shutdown_connections 2021-02-10T11:04:23.472 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:23.473 7f7e1f46e700 1 -- 172.21.2.102:0/1460890310 wait complete. 2021-02-10T11:04:23.480 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-02-10T11:04:23.480 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:04:23.906 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:23.890 7fbb95d50700 1 Processor -- start 2021-02-10T11:04:23.907 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:23.890 7fbb95d50700 1 -- start start 2021-02-10T11:04:23.907 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:23.890 7fbb95d50700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbb900e5470 0x7fbb9007e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:23.907 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:23.890 7fbb95d50700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fbb9007f300 0x7fbb9007f700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:23.907 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:23.890 7fbb95d50700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb9007ed80 con 0x7fbb900e5070 2021-02-10T11:04:23.907 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:23.890 7fbb95d50700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb901b4010 con 0x7fbb9007f300 2021-02-10T11:04:23.907 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:23.890 7fbb95d50700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb900d1790 con 0x7fbb900e5470 2021-02-10T11:04:23.907 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:23.893 7fbb8effd700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbb900e5470 0x7fbb9007e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:23.907 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:23.893 7fbb8f7fe700 1 --1- >> v1:172.21.2.113:6789/0 conn(0x7fbb900e5070 0x7fbb901b3820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.113:6789/0 says I am v1:172.21.2.102:49428/0 (socket says 172.21.2.102:49428) 2021-02-10T11:04:23.907 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:23.893 7fbb8f7fe700 1 -- 172.21.2.102:0/4161052046 learned_addr learned my addr 172.21.2.102:0/4161052046 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:04:23.907 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:23.893 7fbb8ffff700 1 --2- 172.21.2.102:0/4161052046 >> v2:172.21.2.113:3300/0 conn(0x7fbb9007f300 0x7fbb9007f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:23.908 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:23.893 7fbb8effd700 1 -- 172.21.2.102:0/4161052046 >> v2:172.21.2.113:3300/0 conn(0x7fbb9007f300 msgr2=0x7fbb9007f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:04:23.908 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:23.893 7fbb8effd700 1 --2- 172.21.2.102:0/4161052046 >> v2:172.21.2.113:3300/0 conn(0x7fbb9007f300 0x7fbb9007f700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:23.908 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:23.893 7fbb8effd700 1 -- 172.21.2.102:0/4161052046 >> v1:172.21.2.113:6789/0 conn(0x7fbb900e5070 legacy=0x7fbb901b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:23.908 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:23.893 7fbb8effd700 1 -- 172.21.2.102:0/4161052046 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb900d1980 con 0x7fbb900e5470 2021-02-10T11:04:23.908 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:23.893 7fbb8ffff700 1 --2- 172.21.2.102:0/4161052046 >> v2:172.21.2.113:3300/0 conn(0x7fbb9007f300 0x7fbb9007f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:04:23.908 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:23.893 7fbb8effd700 1 --2- 172.21.2.102:0/4161052046 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbb900e5470 0x7fbb9007e6a0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7fbb80006650 tx=0x7fbb80009a80).ready entity=mon.2 client_cookie=953f7f85a29e91b9 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:04:23.908 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:23.894 7fbb8e7fc700 1 -- 172.21.2.102:0/4161052046 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbb800009b0 con 0x7fbb900e5470 2021-02-10T11:04:23.908 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:23.894 7fbb8e7fc700 1 -- 172.21.2.102:0/4161052046 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb8000c0c0 con 0x7fbb900e5470 2021-02-10T11:04:23.908 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:23.894 7fbb8e7fc700 1 -- 172.21.2.102:0/4161052046 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbb8000c5e0 con 0x7fbb900e5470 2021-02-10T11:04:23.908 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:23.894 7fbb95d50700 1 -- 172.21.2.102:0/4161052046 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbb900e5470 msgr2=0x7fbb9007e6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:23.908 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:23.894 7fbb95d50700 1 --2- 172.21.2.102:0/4161052046 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbb900e5470 0x7fbb9007e6a0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7fbb80006650 tx=0x7fbb80009a80).stop 2021-02-10T11:04:23.908 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:23.894 7fbb95d50700 1 -- 172.21.2.102:0/4161052046 shutdown_connections 2021-02-10T11:04:23.908 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:23.894 7fbb95d50700 1 --2- 172.21.2.102:0/4161052046 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbb900e5470 0x7fbb9007e6a0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:23.908 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:23.894 7fbb95d50700 1 --2- 172.21.2.102:0/4161052046 >> v2:172.21.2.113:3300/0 conn(0x7fbb9007f300 0x7fbb9007f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:23.909 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:23.894 7fbb95d50700 1 -- 172.21.2.102:0/4161052046 >> 172.21.2.102:0/4161052046 conn(0x7fbb900ce690 msgr2=0x7fbb900e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:04:23.909 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:23.894 7fbb95d50700 1 -- 172.21.2.102:0/4161052046 shutdown_connections 2021-02-10T11:04:23.909 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:23.894 7fbb95d50700 1 -- 172.21.2.102:0/4161052046 wait complete. 2021-02-10T11:04:23.910 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:23.908 7fbb95d50700 1 Processor -- start 2021-02-10T11:04:23.910 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:23.908 7fbb95d50700 1 -- start start 2021-02-10T11:04:23.910 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:23.909 7fbb95d50700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fbb90097250 0x7fbb901a1b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:23.910 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:23.909 7fbb95d50700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbb9008d5a0 0x7fbb900ae3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:23.910 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:23.909 7fbb95d50700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbb900ae9c0 0x7fbb901a2870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:23.910 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:23.909 7fbb95d50700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb901a22f0 con 0x7fbb900ae9c0 2021-02-10T11:04:23.911 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:23.909 7fbb95d50700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb901b2d40 con 0x7fbb90097250 2021-02-10T11:04:23.911 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:23.909 7fbb95d50700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb901b2f20 con 0x7fbb9008d5a0 2021-02-10T11:04:23.911 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:23.909 7fbb8ffff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbb900ae9c0 0x7fbb901a2870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:23.911 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:23.909 7fbb8ffff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbb900ae9c0 0x7fbb901a2870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:49890/0 (socket says 172.21.2.102:49890) 2021-02-10T11:04:23.911 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:23.909 7fbb8ffff700 1 -- 172.21.2.102:0/4006583732 learned_addr learned my addr 172.21.2.102:0/4006583732 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:04:23.911 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:23.909 7fbb8ffff700 1 -- 172.21.2.102:0/4006583732 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbb9008d5a0 msgr2=0x7fbb900ae3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:23.911 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:23.909 7fbb8ffff700 1 --2- 172.21.2.102:0/4006583732 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbb9008d5a0 0x7fbb900ae3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:23.911 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:23.909 7fbb8ffff700 1 -- 172.21.2.102:0/4006583732 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fbb90097250 msgr2=0x7fbb901a1b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:23.911 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:23.909 7fbb8ffff700 1 --2- 172.21.2.102:0/4006583732 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fbb90097250 0x7fbb901a1b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:23.911 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:23.909 7fbb8ffff700 1 -- 172.21.2.102:0/4006583732 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb9008d9d0 con 0x7fbb900ae9c0 2021-02-10T11:04:23.911 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:23.909 7fbb8ffff700 1 --2- 172.21.2.102:0/4006583732 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbb900ae9c0 0x7fbb901a2870 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7fbb8800b970 tx=0x7fbb8800bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:04:23.911 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:23.909 7fbb87fff700 1 -- 172.21.2.102:0/4006583732 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbb88000f90 con 0x7fbb900ae9c0 2021-02-10T11:04:23.912 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:23.909 7fbb87fff700 1 -- 172.21.2.102:0/4006583732 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb88006790 con 0x7fbb900ae9c0 2021-02-10T11:04:23.912 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:23.909 7fbb87fff700 1 -- 172.21.2.102:0/4006583732 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbb880015f0 con 0x7fbb900ae9c0 2021-02-10T11:04:23.912 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:23.909 7fbb95d50700 1 -- 172.21.2.102:0/4006583732 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb9008d9d0 con 0x7fbb900ae9c0 2021-02-10T11:04:23.912 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:23.909 7fbb95d50700 1 -- 172.21.2.102:0/4006583732 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb901a3180 con 0x7fbb900ae9c0 2021-02-10T11:04:23.912 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:23.911 7fbb87fff700 1 -- 172.21.2.102:0/4006583732 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fbb880011f0 con 0x7fbb900ae9c0 2021-02-10T11:04:23.912 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:23.911 7fbb87fff700 1 --2- 172.21.2.102:0/4006583732 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fbb740186c0 0x7fbb7401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:23.912 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:23.911 7fbb87fff700 1 -- 172.21.2.102:0/4006583732 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fbb880328e0 con 0x7fbb900ae9c0 2021-02-10T11:04:23.912 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:23.911 7fbb8f7fe700 1 --2- 172.21.2.102:0/4006583732 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fbb740186c0 0x7fbb7401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:23.912 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:23.911 7fbb8f7fe700 1 --2- 172.21.2.102:0/4006583732 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fbb740186c0 0x7fbb7401abc0 crc :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:04:23.939 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:23.940 7fbb95d50700 1 -- 172.21.2.102:0/4006583732 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb90107170 con 0x7fbb900ae9c0 2021-02-10T11:04:23.942 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:23.944 7fbb87fff700 1 -- 172.21.2.102:0/4006583732 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fbb88000f70 con 0x7fbb900ae9c0 2021-02-10T11:04:24.071 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:24.072 7fbb95d50700 1 -- 172.21.2.102:0/4006583732 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbb9007ed80 con 0x7fbb900ae9c0 2021-02-10T11:04:24.072 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:24.073 7fbb87fff700 1 -- 172.21.2.102:0/4006583732 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+2094 (secure 0 0 0) 0x7fbb880011f0 con 0x7fbb900ae9c0 2021-02-10T11:04:24.073 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:04:24.073 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":6,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4302,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.102:6835/3297596596","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6834","nonce":3297596596},{"type":"v1","addr":"172.21.2.102:6835","nonce":3297596596}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":4}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:03:43.384709","modified":"2021-02-10 11:03:52.791414","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4226},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4226":{"gid":4226,"name":"b","rank":0,"incarnation":5,"state":"up:active","state_seq":3,"addr":"172.21.2.113:6833/285807165","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6832","nonce":285807165},{"type":"v1","addr":"172.21.2.113:6833","nonce":285807165}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2021-02-10T11:04:24.073 INFO:teuthology.orchestra.run.gibba002.stderr:dumped fsmap epoch 6 2021-02-10T11:04:24.073 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:24.075 7fbb95d50700 1 -- 172.21.2.102:0/4006583732 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fbb740186c0 msgr2=0x7fbb7401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:24.074 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:24.075 7fbb95d50700 1 --2- 172.21.2.102:0/4006583732 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fbb740186c0 0x7fbb7401abc0 crc :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:24.074 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:24.075 7fbb95d50700 1 -- 172.21.2.102:0/4006583732 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbb900ae9c0 msgr2=0x7fbb901a2870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:24.074 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:24.076 7fbb95d50700 1 --2- 172.21.2.102:0/4006583732 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbb900ae9c0 0x7fbb901a2870 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7fbb8800b970 tx=0x7fbb8800bd10).stop 2021-02-10T11:04:24.074 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:24.076 7fbb95d50700 1 -- 172.21.2.102:0/4006583732 shutdown_connections 2021-02-10T11:04:24.074 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:24.076 7fbb95d50700 1 --2- 172.21.2.102:0/4006583732 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbb900ae9c0 0x7fbb901a2870 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:24.074 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:24.076 7fbb95d50700 1 --2- 172.21.2.102:0/4006583732 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbb9008d5a0 0x7fbb900ae3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:24.074 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:24.076 7fbb95d50700 1 --2- 172.21.2.102:0/4006583732 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fbb740186c0 0x7fbb7401abc0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:24.074 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:24.076 7fbb95d50700 1 --2- 172.21.2.102:0/4006583732 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fbb90097250 0x7fbb901a1b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:24.074 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:24.076 7fbb95d50700 1 -- 172.21.2.102:0/4006583732 >> 172.21.2.102:0/4006583732 conn(0x7fbb901ac620 msgr2=0x7fbb900896e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:04:24.075 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:24.076 7fbb95d50700 1 -- 172.21.2.102:0/4006583732 shutdown_connections 2021-02-10T11:04:24.075 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:24.076 7fbb95d50700 1 -- 172.21.2.102:0/4006583732 wait complete. 2021-02-10T11:04:24.284 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 6, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:03:43.384709', 'modified': '2021-02-10 11:03:52.791414', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4226}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4226': {'gid': 4226, 'name': 'b', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.113:6833/285807165', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.113:6832', 'nonce': 285807165}, {'type': 'v1', 'addr': '172.21.2.113:6833', 'nonce': 285807165}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:04:24.284 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:04:24.284 DEBUG:teuthology.orchestra.run.gibba013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:04:24.685 INFO:teuthology.orchestra.run.gibba013.stdout:{ 2021-02-10T11:04:24.685 INFO:teuthology.orchestra.run.gibba013.stdout: "cluster_fsid": "68db30ea-dbce-4aef-a6df-f01c6fea0513", 2021-02-10T11:04:24.685 INFO:teuthology.orchestra.run.gibba013.stdout: "whoami": 0, 2021-02-10T11:04:24.685 INFO:teuthology.orchestra.run.gibba013.stdout: "id": 4226, 2021-02-10T11:04:24.685 INFO:teuthology.orchestra.run.gibba013.stdout: "want_state": "up:active", 2021-02-10T11:04:24.685 INFO:teuthology.orchestra.run.gibba013.stdout: "state": "up:active", 2021-02-10T11:04:24.685 INFO:teuthology.orchestra.run.gibba013.stdout: "rank_uptime": 32.895151861999999, 2021-02-10T11:04:24.685 INFO:teuthology.orchestra.run.gibba013.stdout: "mdsmap_epoch": 6, 2021-02-10T11:04:24.685 INFO:teuthology.orchestra.run.gibba013.stdout: "osdmap_epoch": 20, 2021-02-10T11:04:24.685 INFO:teuthology.orchestra.run.gibba013.stdout: "osdmap_epoch_barrier": 20, 2021-02-10T11:04:24.685 INFO:teuthology.orchestra.run.gibba013.stdout: "uptime": 37.289587294999997 2021-02-10T11:04:24.686 INFO:teuthology.orchestra.run.gibba013.stdout:} 2021-02-10T11:04:24.690 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "68db30ea-dbce-4aef-a6df-f01c6fea0513", "whoami": 0, "id": 4226, "want_state": "up:active", "state": "up:active", "rank_uptime": 32.895151861999999, "mdsmap_epoch": 6, "osdmap_epoch": 20, "osdmap_epoch_barrier": 20, "uptime": 37.289587294999997 } 2021-02-10T11:04:24.690 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2021-02-10T11:04:24.690 INFO:tasks.cephfs.fuse_mount:Client client.2 config is {} 2021-02-10T11:04:24.690 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.2 at ubuntu@gibba013.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.2... 2021-02-10T11:04:24.690 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:04:24.690 DEBUG:teuthology.orchestra.run.gibba013:> (cd /home/ubuntu/cephtest && exec mkdir -p /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:04:25.025 DEBUG:teuthology.orchestra.run.gibba013:> sudo modprobe fuse 2021-02-10T11:04:25.410 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:04:25.410 DEBUG:teuthology.orchestra.run.gibba013:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:04:25.757 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:04:25.757 DEBUG:teuthology.orchestra.run.gibba013:> ls /sys/fs/fuse/connections 2021-02-10T11:04:26.091 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2021-02-10T11:04:26.091 DEBUG:teuthology.orchestra.run.gibba013:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.2 /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:04:26.094 DEBUG:teuthology.orchestra.run.gibba013:> sudo modprobe fuse 2021-02-10T11:04:26.437 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:04:26.437 DEBUG:teuthology.orchestra.run.gibba013:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:04:26.502 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.gibba013.stderr:ceph-fuse[37327]: starting ceph client 2021-02-10T11:04:26.502 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.gibba013.stderr:2021-02-10 11:04:26.501 7fd10ee6af80 -1 init, newargv = 0x55825feba100 newargc=9 2021-02-10T11:04:26.508 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.gibba013.stderr:ceph-fuse[37327]: starting fuse 2021-02-10T11:04:26.778 INFO:teuthology.orchestra.run.gibba013.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:04:26.778 INFO:teuthology.orchestra.run.gibba013.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:04:26.779 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:04:26.779 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:04:26.780 DEBUG:teuthology.orchestra.run.gibba013:> ls /sys/fs/fuse/connections 2021-02-10T11:04:27.112 INFO:teuthology.orchestra.run.gibba013.stdout:43 2021-02-10T11:04:27.112 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2021-02-10T11:04:27.112 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:04:27.113 DEBUG:teuthology.orchestra.run.gibba013:> sudo python3 -c ' 2021-02-10T11:04:27.113 DEBUG:teuthology.orchestra.run.gibba013:> import glob 2021-02-10T11:04:27.113 DEBUG:teuthology.orchestra.run.gibba013:> import re 2021-02-10T11:04:27.113 DEBUG:teuthology.orchestra.run.gibba013:> import os 2021-02-10T11:04:27.113 DEBUG:teuthology.orchestra.run.gibba013:> import subprocess 2021-02-10T11:04:27.113 DEBUG:teuthology.orchestra.run.gibba013:> 2021-02-10T11:04:27.113 DEBUG:teuthology.orchestra.run.gibba013:> def find_socket(client_name): 2021-02-10T11:04:27.113 DEBUG:teuthology.orchestra.run.gibba013:> asok_path = "/var/run/ceph/ceph-client.2.*.asok" 2021-02-10T11:04:27.113 DEBUG:teuthology.orchestra.run.gibba013:> files = glob.glob(asok_path) 2021-02-10T11:04:27.113 DEBUG:teuthology.orchestra.run.gibba013:> 2021-02-10T11:04:27.113 DEBUG:teuthology.orchestra.run.gibba013:> # Given a non-glob path, it better be there 2021-02-10T11:04:27.113 DEBUG:teuthology.orchestra.run.gibba013:> if "*" not in asok_path: 2021-02-10T11:04:27.113 DEBUG:teuthology.orchestra.run.gibba013:> assert(len(files) == 1) 2021-02-10T11:04:27.114 DEBUG:teuthology.orchestra.run.gibba013:> return files[0] 2021-02-10T11:04:27.114 DEBUG:teuthology.orchestra.run.gibba013:> 2021-02-10T11:04:27.114 DEBUG:teuthology.orchestra.run.gibba013:> for f in files: 2021-02-10T11:04:27.114 DEBUG:teuthology.orchestra.run.gibba013:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2021-02-10T11:04:27.114 DEBUG:teuthology.orchestra.run.gibba013:> if os.path.exists("/proc/{0}".format(pid)): 2021-02-10T11:04:27.114 DEBUG:teuthology.orchestra.run.gibba013:> return f 2021-02-10T11:04:27.114 DEBUG:teuthology.orchestra.run.gibba013:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2021-02-10T11:04:27.114 DEBUG:teuthology.orchestra.run.gibba013:> 2021-02-10T11:04:27.114 DEBUG:teuthology.orchestra.run.gibba013:> print(find_socket("client.2")) 2021-02-10T11:04:27.114 DEBUG:teuthology.orchestra.run.gibba013:> ' 2021-02-10T11:04:27.486 INFO:teuthology.orchestra.run.gibba013.stdout:/var/run/ceph/ceph-client.2.37327.asok 2021-02-10T11:04:27.489 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.2.37327.asok 2021-02-10T11:04:27.489 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:04:27.489 DEBUG:teuthology.orchestra.run.gibba013:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.2.37327.asok status 2021-02-10T11:04:27.878 INFO:teuthology.orchestra.run.gibba013.stdout:{ 2021-02-10T11:04:27.878 INFO:teuthology.orchestra.run.gibba013.stdout: "metadata": { 2021-02-10T11:04:27.878 INFO:teuthology.orchestra.run.gibba013.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:04:27.878 INFO:teuthology.orchestra.run.gibba013.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:04:27.878 INFO:teuthology.orchestra.run.gibba013.stdout: "entity_id": "2", 2021-02-10T11:04:27.878 INFO:teuthology.orchestra.run.gibba013.stdout: "hostname": "gibba013", 2021-02-10T11:04:27.878 INFO:teuthology.orchestra.run.gibba013.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.2", 2021-02-10T11:04:27.879 INFO:teuthology.orchestra.run.gibba013.stdout: "pid": "37327", 2021-02-10T11:04:27.879 INFO:teuthology.orchestra.run.gibba013.stdout: "root": "/" 2021-02-10T11:04:27.879 INFO:teuthology.orchestra.run.gibba013.stdout: }, 2021-02-10T11:04:27.879 INFO:teuthology.orchestra.run.gibba013.stdout: "dentry_count": 0, 2021-02-10T11:04:27.879 INFO:teuthology.orchestra.run.gibba013.stdout: "dentry_pinned_count": 0, 2021-02-10T11:04:27.879 INFO:teuthology.orchestra.run.gibba013.stdout: "id": 4328, 2021-02-10T11:04:27.879 INFO:teuthology.orchestra.run.gibba013.stdout: "inst": { 2021-02-10T11:04:27.879 INFO:teuthology.orchestra.run.gibba013.stdout: "name": { 2021-02-10T11:04:27.879 INFO:teuthology.orchestra.run.gibba013.stdout: "type": "client", 2021-02-10T11:04:27.879 INFO:teuthology.orchestra.run.gibba013.stdout: "num": 4328 2021-02-10T11:04:27.879 INFO:teuthology.orchestra.run.gibba013.stdout: }, 2021-02-10T11:04:27.879 INFO:teuthology.orchestra.run.gibba013.stdout: "addr": { 2021-02-10T11:04:27.879 INFO:teuthology.orchestra.run.gibba013.stdout: "type": "v1", 2021-02-10T11:04:27.879 INFO:teuthology.orchestra.run.gibba013.stdout: "addr": "172.21.2.113:0", 2021-02-10T11:04:27.880 INFO:teuthology.orchestra.run.gibba013.stdout: "nonce": 2724508509 2021-02-10T11:04:27.880 INFO:teuthology.orchestra.run.gibba013.stdout: } 2021-02-10T11:04:27.880 INFO:teuthology.orchestra.run.gibba013.stdout: }, 2021-02-10T11:04:27.880 INFO:teuthology.orchestra.run.gibba013.stdout: "addr": { 2021-02-10T11:04:27.880 INFO:teuthology.orchestra.run.gibba013.stdout: "type": "v1", 2021-02-10T11:04:27.880 INFO:teuthology.orchestra.run.gibba013.stdout: "addr": "172.21.2.113:0", 2021-02-10T11:04:27.880 INFO:teuthology.orchestra.run.gibba013.stdout: "nonce": 2724508509 2021-02-10T11:04:27.880 INFO:teuthology.orchestra.run.gibba013.stdout: }, 2021-02-10T11:04:27.880 INFO:teuthology.orchestra.run.gibba013.stdout: "inst_str": "client.4328 172.21.2.113:0/2724508509", 2021-02-10T11:04:27.880 INFO:teuthology.orchestra.run.gibba013.stdout: "addr_str": "172.21.2.113:0/2724508509", 2021-02-10T11:04:27.880 INFO:teuthology.orchestra.run.gibba013.stdout: "inode_count": 1, 2021-02-10T11:04:27.880 INFO:teuthology.orchestra.run.gibba013.stdout: "mds_epoch": 6, 2021-02-10T11:04:27.880 INFO:teuthology.orchestra.run.gibba013.stdout: "osd_epoch": 20, 2021-02-10T11:04:27.881 INFO:teuthology.orchestra.run.gibba013.stdout: "osd_epoch_barrier": 0, 2021-02-10T11:04:27.881 INFO:teuthology.orchestra.run.gibba013.stdout: "blacklisted": false 2021-02-10T11:04:27.881 INFO:teuthology.orchestra.run.gibba013.stdout:} 2021-02-10T11:04:27.882 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:04:28.310 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:28.295 7f86fca4a700 1 Processor -- start 2021-02-10T11:04:28.311 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:28.295 7f86fca4a700 1 -- start start 2021-02-10T11:04:28.311 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:28.296 7f86fca4a700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f86f807e6a0 0x7f86f80d1790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:28.311 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:28.296 7f86fca4a700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86f807ed80 con 0x7f86f80e5470 2021-02-10T11:04:28.311 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:28.296 7f86fca4a700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86f807eaa0 con 0x7f86f80e5070 2021-02-10T11:04:28.311 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:28.296 7f86fca4a700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86f81b4080 con 0x7f86f807e6a0 2021-02-10T11:04:28.311 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:28.296 7f86f605d700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f86f80e5470 0x7f86f807f300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:42118/0 (socket says 172.21.2.102:42118) 2021-02-10T11:04:28.311 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:28.296 7f86f605d700 1 -- 172.21.2.102:0/3198647095 learned_addr learned my addr 172.21.2.102:0/3198647095 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:04:28.311 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:28.296 7f86f585c700 1 -- 172.21.2.102:0/3198647095 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2544858530 0 0) 0x7f86e00010b0 con 0x7f86f80e5470 2021-02-10T11:04:28.311 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:28.296 7f86f585c700 1 -- 172.21.2.102:0/3198647095 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86e4002fe0 con 0x7f86f80e5470 2021-02-10T11:04:28.311 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:28.296 7f86f705f700 1 --2- 172.21.2.102:0/3198647095 >> v2:172.21.2.113:3300/0 conn(0x7f86f807e6a0 0x7f86f80d1790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:28.311 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:28.297 7f86f585c700 1 -- 172.21.2.102:0/3198647095 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4278303444 0 0) 0x7f86e00010b0 con 0x7f86f80e5470 2021-02-10T11:04:28.312 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:28.297 7f86f585c700 1 -- 172.21.2.102:0/3198647095 >> v2:172.21.2.113:3300/0 conn(0x7f86f807e6a0 msgr2=0x7f86f80d1790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:04:28.312 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:28.297 7f86f585c700 1 --2- 172.21.2.102:0/3198647095 >> v2:172.21.2.113:3300/0 conn(0x7f86f807e6a0 0x7f86f80d1790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:28.312 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:28.297 7f86f585c700 1 -- 172.21.2.102:0/3198647095 >> v1:172.21.2.113:6789/0 conn(0x7f86f80e5070 legacy=0x7f86f81b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:28.312 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:28.297 7f86f585c700 1 -- 172.21.2.102:0/3198647095 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86f80d1db0 con 0x7f86f80e5470 2021-02-10T11:04:28.312 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:28.297 7f86f585c700 1 -- 172.21.2.102:0/3198647095 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7f86e00015b0 con 0x7f86f80e5470 2021-02-10T11:04:28.312 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:28.297 7f86f585c700 1 -- 172.21.2.102:0/3198647095 >> v1:172.21.2.102:6789/0 conn(0x7f86f80e5470 legacy=0x7f86f807f300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:28.312 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:28.297 7f86f585c700 1 --2- 172.21.2.102:0/3198647095 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f86e4003aa0 0x7f86e4005f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:28.312 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:28.297 7f86f585c700 1 --2- 172.21.2.102:0/3198647095 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f86e4006780 0x7f86e4008ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:28.312 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:28.297 7f86f585c700 1 --2- 172.21.2.102:0/3198647095 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f86e40095a0 0x7f86e400baa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:28.312 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:28.297 7f86f585c700 1 -- 172.21.2.102:0/3198647095 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86e400c2f0 con 0x7f86e4003aa0 2021-02-10T11:04:28.312 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:28.297 7f86f585c700 1 -- 172.21.2.102:0/3198647095 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86e400c510 con 0x7f86e40095a0 2021-02-10T11:04:28.312 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:28.297 7f86f585c700 1 -- 172.21.2.102:0/3198647095 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86e400c730 con 0x7f86e4006780 2021-02-10T11:04:28.312 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:28.297 7f86f685e700 1 --2- 172.21.2.102:0/3198647095 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f86e4003aa0 0x7f86e4005f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:28.313 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:28.297 7f86f605d700 1 --2- 172.21.2.102:0/3198647095 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f86e4006780 0x7f86e4008ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:28.313 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:28.297 7f86f685e700 1 -- 172.21.2.102:0/3198647095 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f86e4006780 msgr2=0x7f86e4008ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:04:28.313 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:28.297 7f86f685e700 1 --2- 172.21.2.102:0/3198647095 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f86e4006780 0x7f86e4008ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:28.313 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:28.297 7f86f685e700 1 -- 172.21.2.102:0/3198647095 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f86e40095a0 msgr2=0x7f86e400baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:04:28.313 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:28.297 7f86f685e700 1 --2- 172.21.2.102:0/3198647095 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f86e40095a0 0x7f86e400baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:28.313 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:28.297 7f86f685e700 1 -- 172.21.2.102:0/3198647095 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86e40509a0 con 0x7f86e4003aa0 2021-02-10T11:04:28.313 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:28.297 7f86f705f700 1 -- 172.21.2.102:0/3198647095 reap_dead start 2021-02-10T11:04:28.313 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:28.297 7f86f685e700 1 --2- 172.21.2.102:0/3198647095 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f86e4003aa0 0x7f86e4005f60 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7f86e8006ab0 tx=0x7f86e8009b40).ready entity=mon.0 client_cookie=2fc311e7208d1610 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:04:28.313 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:28.297 7f86f585c700 1 -- 172.21.2.102:0/3198647095 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f86e800b2e0 con 0x7f86e4003aa0 2021-02-10T11:04:28.313 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:28.297 7f86f585c700 1 -- 172.21.2.102:0/3198647095 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86e800b660 con 0x7f86e4003aa0 2021-02-10T11:04:28.313 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:28.297 7f86f585c700 1 -- 172.21.2.102:0/3198647095 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f86e800bac0 con 0x7f86e4003aa0 2021-02-10T11:04:28.313 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:28.297 7f86fca4a700 1 -- 172.21.2.102:0/3198647095 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f86e4003aa0 msgr2=0x7f86e4005f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:28.314 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:28.297 7f86fca4a700 1 --2- 172.21.2.102:0/3198647095 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f86e4003aa0 0x7f86e4005f60 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7f86e8006ab0 tx=0x7f86e8009b40).stop 2021-02-10T11:04:28.314 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:28.298 7f86fca4a700 1 -- 172.21.2.102:0/3198647095 shutdown_connections 2021-02-10T11:04:28.314 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:28.298 7f86fca4a700 1 --2- 172.21.2.102:0/3198647095 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f86e4003aa0 0x7f86e4005f60 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:28.314 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:28.298 7f86fca4a700 1 -- 172.21.2.102:0/3198647095 >> 172.21.2.102:0/3198647095 conn(0x7f86f80ce690 msgr2=0x7f86f80e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:04:28.314 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:28.298 7f86fca4a700 1 -- 172.21.2.102:0/3198647095 shutdown_connections 2021-02-10T11:04:28.314 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:28.298 7f86fca4a700 1 -- 172.21.2.102:0/3198647095 wait complete. 2021-02-10T11:04:28.314 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:28.313 7f86fca4a700 1 Processor -- start 2021-02-10T11:04:28.314 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:28.313 7f86fca4a700 1 -- start start 2021-02-10T11:04:28.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:28.313 7f86fca4a700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f86f8194000 0x7f86f80b0d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:28.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:28.313 7f86fca4a700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f86f80b1330 0x7f86f8087050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:28.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:28.313 7f86fca4a700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f86f8087670 0x7f86f8087a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:28.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:28.313 7f86fca4a700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86f809c0e0 con 0x7f86f8087670 2021-02-10T11:04:28.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:28.313 7f86fca4a700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86f8194440 con 0x7f86f8194000 2021-02-10T11:04:28.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:28.313 7f86fca4a700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86f809c5e0 con 0x7f86f80b1330 2021-02-10T11:04:28.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:28.313 7f86f705f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f86f8087670 0x7f86f8087a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:28.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:28.313 7f86f705f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f86f8087670 0x7f86f8087a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:49906/0 (socket says 172.21.2.102:49906) 2021-02-10T11:04:28.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:28.313 7f86f705f700 1 -- 172.21.2.102:0/3829523346 learned_addr learned my addr 172.21.2.102:0/3829523346 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:04:28.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:28.314 7f86f705f700 1 -- 172.21.2.102:0/3829523346 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f86f80b1330 msgr2=0x7f86f8087050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:28.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:28.314 7f86f605d700 1 --2- 172.21.2.102:0/3829523346 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f86f80b1330 0x7f86f8087050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:28.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:28.314 7f86f685e700 1 --2- 172.21.2.102:0/3829523346 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f86f8194000 0x7f86f80b0d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:28.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:28.314 7f86f705f700 1 --2- 172.21.2.102:0/3829523346 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f86f80b1330 0x7f86f8087050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:28.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:28.314 7f86f705f700 1 -- 172.21.2.102:0/3829523346 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f86f8194000 msgr2=0x7f86f80b0d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:28.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:28.314 7f86f705f700 1 --2- 172.21.2.102:0/3829523346 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f86f8194000 0x7f86f80b0d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:28.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:28.314 7f86f705f700 1 -- 172.21.2.102:0/3829523346 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86f809c7d0 con 0x7f86f8087670 2021-02-10T11:04:28.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:28.314 7f86f605d700 1 --2- 172.21.2.102:0/3829523346 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f86f80b1330 0x7f86f8087050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:04:28.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:28.314 7f86f705f700 1 --2- 172.21.2.102:0/3829523346 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f86f8087670 0x7f86f8087a70 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f86f0002ac0 tx=0x7f86f000ac90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:04:28.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:28.314 7f86ef7fe700 1 -- 172.21.2.102:0/3829523346 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f86f0002020 con 0x7f86f8087670 2021-02-10T11:04:28.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:28.314 7f86ef7fe700 1 -- 172.21.2.102:0/3829523346 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86f0007030 con 0x7f86f8087670 2021-02-10T11:04:28.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:28.314 7f86fca4a700 1 -- 172.21.2.102:0/3829523346 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86f809c7d0 con 0x7f86f8087670 2021-02-10T11:04:28.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:28.314 7f86fca4a700 1 -- 172.21.2.102:0/3829523346 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86f809c7d0 con 0x7f86f8087670 2021-02-10T11:04:28.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:28.314 7f86ef7fe700 1 -- 172.21.2.102:0/3829523346 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f86f0007550 con 0x7f86f8087670 2021-02-10T11:04:28.318 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:28.315 7f86ef7fe700 1 -- 172.21.2.102:0/3829523346 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f86f0007210 con 0x7f86f8087670 2021-02-10T11:04:28.318 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:28.315 7f86ef7fe700 1 --2- 172.21.2.102:0/3829523346 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f86dc018730 0x7f86dc01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:28.318 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:28.315 7f86ef7fe700 1 -- 172.21.2.102:0/3829523346 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f86f0032bb0 con 0x7f86f8087670 2021-02-10T11:04:28.318 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:28.315 7f86f685e700 1 --2- 172.21.2.102:0/3829523346 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f86dc018730 0x7f86dc01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:28.318 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:28.315 7f86f685e700 1 --2- 172.21.2.102:0/3829523346 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f86dc018730 0x7f86dc01ac10 crc :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:04:28.343 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:28.345 7f86fca4a700 1 -- 172.21.2.102:0/3829523346 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86f8084b20 con 0x7f86f8087670 2021-02-10T11:04:28.345 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:28.348 7f86ef7fe700 1 -- 172.21.2.102:0/3829523346 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f86f0007240 con 0x7f86f8087670 2021-02-10T11:04:28.469 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:28.471 7f86fca4a700 1 -- 172.21.2.102:0/3829523346 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f86f8084b20 con 0x7f86f8087670 2021-02-10T11:04:28.469 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:28.472 7f86ef7fe700 1 -- 172.21.2.102:0/3829523346 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+13993 (secure 0 0 0) 0x7f86f0007030 con 0x7f86f8087670 2021-02-10T11:04:28.470 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:04:28.470 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":20,"fsid":"68db30ea-dbce-4aef-a6df-f01c6fea0513","created":"2021-02-10 11:03:06.331557","modified":"2021-02-10 11:03:44.386945","last_up_change":"2021-02-10 11:03:32.690562","last_in_change":"2021-02-10 11:03:32.690562","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:03:37.322519","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:03:41.161563","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2021-02-10 11:03:42.379651","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"be691b4d-a039-408a-af80-9d53f02b6f1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6818","nonce":36088},{"type":"v1","addr":"172.21.2.102:6820","nonce":36088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6822","nonce":36088},{"type":"v1","addr":"172.21.2.102:6824","nonce":36088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6830","nonce":36088},{"type":"v1","addr":"172.21.2.102:6832","nonce":36088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6826","nonce":36088},{"type":"v1","addr":"172.21.2.102:6828","nonce":36088}]},"public_addr":"172.21.2.102:6820/36088","cluster_addr":"172.21.2.102:6824/36088","heartbeat_back_addr":"172.21.2.102:6832/36088","heartbeat_front_addr":"172.21.2.102:6828/36088","state":["exists","up"]},{"osd":1,"uuid":"64e3ccc8-9c69-4d7c-865e-af8e83822635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6819","nonce":36087},{"type":"v1","addr":"172.21.2.102:6821","nonce":36087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6823","nonce":36087},{"type":"v1","addr":"172.21.2.102:6825","nonce":36087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6831","nonce":36087},{"type":"v1","addr":"172.21.2.102:6833","nonce":36087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6827","nonce":36087},{"type":"v1","addr":"172.21.2.102:6829","nonce":36087}]},"public_addr":"172.21.2.102:6821/36087","cluster_addr":"172.21.2.102:6825/36087","heartbeat_back_addr":"172.21.2.102:6833/36087","heartbeat_front_addr":"172.21.2.102:6829/36087","state":["exists","up"]},{"osd":2,"uuid":"153700af-747f-481f-bdf4-d6072d3a842f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6810","nonce":36089},{"type":"v1","addr":"172.21.2.102:6811","nonce":36089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6812","nonce":36089},{"type":"v1","addr":"172.21.2.102:6813","nonce":36089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6816","nonce":36089},{"type":"v1","addr":"172.21.2.102:6817","nonce":36089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6814","nonce":36089},{"type":"v1","addr":"172.21.2.102:6815","nonce":36089}]},"public_addr":"172.21.2.102:6811/36089","cluster_addr":"172.21.2.102:6813/36089","heartbeat_back_addr":"172.21.2.102:6817/36089","heartbeat_front_addr":"172.21.2.102:6815/36089","state":["exists","up"]},{"osd":3,"uuid":"cbc678f4-83a9-45e6-8830-2f7c232b1ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6802","nonce":36090},{"type":"v1","addr":"172.21.2.102:6803","nonce":36090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6804","nonce":36090},{"type":"v1","addr":"172.21.2.102:6805","nonce":36090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6808","nonce":36090},{"type":"v1","addr":"172.21.2.102:6809","nonce":36090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6806","nonce":36090},{"type":"v1","addr":"172.21.2.102:6807","nonce":36090}]},"public_addr":"172.21.2.102:6803/36090","cluster_addr":"172.21.2.102:6805/36090","heartbeat_back_addr":"172.21.2.102:6809/36090","heartbeat_front_addr":"172.21.2.102:6807/36090","state":["exists","up"]},{"osd":4,"uuid":"c01e4008-b258-4f10-afd2-9e7f37214351","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6808","nonce":36368},{"type":"v1","addr":"172.21.2.113:6811","nonce":36368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6814","nonce":36368},{"type":"v1","addr":"172.21.2.113:6818","nonce":36368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6826","nonce":36368},{"type":"v1","addr":"172.21.2.113:6828","nonce":36368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6820","nonce":36368},{"type":"v1","addr":"172.21.2.113:6824","nonce":36368}]},"public_addr":"172.21.2.113:6811/36368","cluster_addr":"172.21.2.113:6818/36368","heartbeat_back_addr":"172.21.2.113:6828/36368","heartbeat_front_addr":"172.21.2.113:6824/36368","state":["exists","up"]},{"osd":5,"uuid":"551f6007-daf8-4a98-8ee0-c2b16bcccb5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6802","nonce":36366},{"type":"v1","addr":"172.21.2.113:6805","nonce":36366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6807","nonce":36366},{"type":"v1","addr":"172.21.2.113:6810","nonce":36366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6819","nonce":36366},{"type":"v1","addr":"172.21.2.113:6822","nonce":36366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6812","nonce":36366},{"type":"v1","addr":"172.21.2.113:6815","nonce":36366}]},"public_addr":"172.21.2.113:6805/36366","cluster_addr":"172.21.2.113:6810/36366","heartbeat_back_addr":"172.21.2.113:6822/36366","heartbeat_front_addr":"172.21.2.113:6815/36366","state":["exists","up"]},{"osd":6,"uuid":"bf67c040-0e50-48be-b102-c3943bc9ebef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6800","nonce":36367},{"type":"v1","addr":"172.21.2.113:6801","nonce":36367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6803","nonce":36367},{"type":"v1","addr":"172.21.2.113:6804","nonce":36367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6813","nonce":36367},{"type":"v1","addr":"172.21.2.113:6816","nonce":36367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6806","nonce":36367},{"type":"v1","addr":"172.21.2.113:6809","nonce":36367}]},"public_addr":"172.21.2.113:6801/36367","cluster_addr":"172.21.2.113:6804/36367","heartbeat_back_addr":"172.21.2.113:6816/36367","heartbeat_front_addr":"172.21.2.113:6809/36367","state":["exists","up"]},{"osd":7,"uuid":"f5461f80-018d-428f-be89-2eb2bcbe7bb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6817","nonce":36365},{"type":"v1","addr":"172.21.2.113:6821","nonce":36365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6823","nonce":36365},{"type":"v1","addr":"172.21.2.113:6825","nonce":36365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6830","nonce":36365},{"type":"v1","addr":"172.21.2.113:6831","nonce":36365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6827","nonce":36365},{"type":"v1","addr":"172.21.2.113:6829","nonce":36365}]},"public_addr":"172.21.2.113:6821/36365","cluster_addr":"172.21.2.113:6825/36365","heartbeat_back_addr":"172.21.2.113:6831/36365","heartbeat_front_addr":"172.21.2.113:6829/36365","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:04:28.471 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:28.473 7f86fca4a700 1 -- 172.21.2.102:0/3829523346 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f86dc018730 msgr2=0x7f86dc01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:28.471 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:28.473 7f86fca4a700 1 --2- 172.21.2.102:0/3829523346 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f86dc018730 0x7f86dc01ac10 crc :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:28.471 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:28.473 7f86fca4a700 1 -- 172.21.2.102:0/3829523346 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f86f8087670 msgr2=0x7f86f8087a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:28.472 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:28.473 7f86fca4a700 1 --2- 172.21.2.102:0/3829523346 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f86f8087670 0x7f86f8087a70 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f86f0002ac0 tx=0x7f86f000ac90).stop 2021-02-10T11:04:28.472 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:28.473 7f86fca4a700 1 -- 172.21.2.102:0/3829523346 shutdown_connections 2021-02-10T11:04:28.472 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:28.473 7f86fca4a700 1 --2- 172.21.2.102:0/3829523346 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f86f8087670 0x7f86f8087a70 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:28.472 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:28.474 7f86fca4a700 1 --2- 172.21.2.102:0/3829523346 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f86f80b1330 0x7f86f8087050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:28.472 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:28.474 7f86fca4a700 1 --2- 172.21.2.102:0/3829523346 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f86dc018730 0x7f86dc01ac10 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:28.472 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:28.474 7f86fca4a700 1 --2- 172.21.2.102:0/3829523346 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f86f8194000 0x7f86f80b0d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:28.472 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:28.474 7f86fca4a700 1 -- 172.21.2.102:0/3829523346 >> 172.21.2.102:0/3829523346 conn(0x7f86f8092c20 msgr2=0x7f86f8195070 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:04:28.472 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:28.474 7f86fca4a700 1 -- 172.21.2.102:0/3829523346 shutdown_connections 2021-02-10T11:04:28.472 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:28.474 7f86fca4a700 1 -- 172.21.2.102:0/3829523346 wait complete. 2021-02-10T11:04:28.480 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:03:37.322519', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2021-02-10 11:03:41.161563', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2021-02-10 11:03:42.379651', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-02-10T11:04:28.480 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:04:28.909 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:28.894 7f8cc8b88700 1 Processor -- start 2021-02-10T11:04:28.910 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:28.894 7f8cc8b88700 1 -- start start 2021-02-10T11:04:28.910 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:28.895 7f8cc8b88700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8cc41b0870 0x7f8cc40f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:28.910 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:28.895 7f8cc8b88700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f8cc40efe00 0x7f8cc40f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:28.910 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:28.895 7f8cc8b88700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cc41b0c70 con 0x7f8cc41b0470 2021-02-10T11:04:28.910 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:28.895 7f8cc8b88700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8cc40f0b50 con 0x7f8cc40efe00 2021-02-10T11:04:28.910 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:28.895 7f8cc8b88700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cc40f0820 con 0x7f8cc41b0870 2021-02-10T11:04:28.910 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:28.895 7f8cc285e700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f8cc41b0470 0x7f8cc40ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:42134/0 (socket says 172.21.2.102:42134) 2021-02-10T11:04:28.910 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:28.895 7f8cc285e700 1 -- 172.21.2.102:0/4105204346 learned_addr learned my addr 172.21.2.102:0/4105204346 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:04:28.910 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:28.895 7f8cc185c700 1 -- 172.21.2.102:0/4105204346 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3111549046 0 0) 0x7f8cac001180 con 0x7f8cc41b0470 2021-02-10T11:04:28.910 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:28.895 7f8cc185c700 1 -- 172.21.2.102:0/4105204346 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8cb0002fe0 con 0x7f8cc41b0470 2021-02-10T11:04:28.910 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:28.895 7f8cc185c700 1 -- 172.21.2.102:0/4105204346 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3986789097 0 0) 0x7f8cac001180 con 0x7f8cc41b0470 2021-02-10T11:04:28.910 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:28.895 7f8cc205d700 1 --2- 172.21.2.102:0/4105204346 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8cc41b0870 0x7f8cc40f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:28.911 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:28.895 7f8cc185c700 1 -- 172.21.2.102:0/4105204346 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8cc41b0870 msgr2=0x7f8cc40f20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:04:28.911 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:28.895 7f8cc305f700 1 --2- 172.21.2.102:0/4105204346 >> v2:172.21.2.113:3300/0 conn(0x7f8cc40efe00 0x7f8cc40f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:28.911 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:28.895 7f8cc185c700 1 --2- 172.21.2.102:0/4105204346 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8cc41b0870 0x7f8cc40f20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:28.911 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:28.895 7f8cc185c700 1 -- 172.21.2.102:0/4105204346 >> v2:172.21.2.113:3300/0 conn(0x7f8cc40efe00 msgr2=0x7f8cc40f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:04:28.911 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:28.895 7f8cc185c700 1 --2- 172.21.2.102:0/4105204346 >> v2:172.21.2.113:3300/0 conn(0x7f8cc40efe00 0x7f8cc40f0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:28.911 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:28.895 7f8cc185c700 1 -- 172.21.2.102:0/4105204346 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cc40e52e0 con 0x7f8cc41b0470 2021-02-10T11:04:28.911 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:28.895 7f8cc185c700 1 -- 172.21.2.102:0/4105204346 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7f8cac001680 con 0x7f8cc41b0470 2021-02-10T11:04:28.911 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:28.895 7f8cc185c700 1 -- 172.21.2.102:0/4105204346 >> v1:172.21.2.102:6789/0 conn(0x7f8cc41b0470 legacy=0x7f8cc40ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:28.911 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:28.895 7f8cc185c700 1 --2- 172.21.2.102:0/4105204346 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8cb0003b90 0x7f8cb0006040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:28.911 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:28.895 7f8cc185c700 1 --2- 172.21.2.102:0/4105204346 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8cb00068e0 0x7f8cb0008dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:28.911 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:28.895 7f8cc185c700 1 --2- 172.21.2.102:0/4105204346 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8cb0009670 0x7f8cb000bb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:28.911 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:28.895 7f8cc185c700 1 -- 172.21.2.102:0/4105204346 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cb000c420 con 0x7f8cb0009670 2021-02-10T11:04:28.911 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:28.895 7f8cc185c700 1 -- 172.21.2.102:0/4105204346 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cb000c640 con 0x7f8cb0003b90 2021-02-10T11:04:28.912 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:28.895 7f8cc185c700 1 -- 172.21.2.102:0/4105204346 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cb000c860 con 0x7f8cb00068e0 2021-02-10T11:04:28.912 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:28.896 7f8cc305f700 1 --2- 172.21.2.102:0/4105204346 >> v2:172.21.2.113:3300/0 conn(0x7f8cc40efe00 0x7f8cc40f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:04:28.912 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:28.896 7f8cc205d700 1 --2- 172.21.2.102:0/4105204346 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8cc41b0870 0x7f8cc40f20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:04:28.912 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:28.896 7f8cc305f700 1 --2- 172.21.2.102:0/4105204346 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8cb0009670 0x7f8cb000bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:28.912 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:28.896 7f8cc305f700 1 -- 172.21.2.102:0/4105204346 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8cb00068e0 msgr2=0x7f8cb0008dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:04:28.912 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:28.896 7f8cc305f700 1 --2- 172.21.2.102:0/4105204346 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8cb00068e0 0x7f8cb0008dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:28.912 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:28.896 7f8cc305f700 1 -- 172.21.2.102:0/4105204346 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8cb0003b90 msgr2=0x7f8cb0006040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:04:28.912 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:28.896 7f8cc305f700 1 --2- 172.21.2.102:0/4105204346 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8cb0003b90 0x7f8cb0006040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:28.912 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:28.896 7f8cc305f700 1 -- 172.21.2.102:0/4105204346 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cb0050b10 con 0x7f8cb0009670 2021-02-10T11:04:28.912 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:28.896 7f8cc305f700 1 --2- 172.21.2.102:0/4105204346 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8cb0009670 0x7f8cb000bb70 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7f8cbc006cc0 tx=0x7f8cbc00b300).ready entity=mon.0 client_cookie=987dd491f828902a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:04:28.912 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:28.896 7f8cc185c700 1 -- 172.21.2.102:0/4105204346 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8cbc00b6f0 con 0x7f8cb0009670 2021-02-10T11:04:28.912 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:28.896 7f8cc185c700 1 -- 172.21.2.102:0/4105204346 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cbc003030 con 0x7f8cb0009670 2021-02-10T11:04:28.912 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:28.896 7f8cc185c700 1 -- 172.21.2.102:0/4105204346 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8cbc00bcb0 con 0x7f8cb0009670 2021-02-10T11:04:28.913 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:28.896 7f8cc8b88700 1 -- 172.21.2.102:0/4105204346 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8cb0009670 msgr2=0x7f8cb000bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:28.913 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:28.896 7f8cc8b88700 1 --2- 172.21.2.102:0/4105204346 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8cb0009670 0x7f8cb000bb70 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7f8cbc006cc0 tx=0x7f8cbc00b300).stop 2021-02-10T11:04:28.913 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:28.896 7f8cc305f700 1 -- 172.21.2.102:0/4105204346 reap_dead start 2021-02-10T11:04:28.913 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:28.897 7f8cc8b88700 1 -- 172.21.2.102:0/4105204346 shutdown_connections 2021-02-10T11:04:28.913 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:28.897 7f8cc8b88700 1 -- 172.21.2.102:0/4105204346 >> 172.21.2.102:0/4105204346 conn(0x7f8cc40ce3e0 msgr2=0x7f8cc40e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:04:28.913 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:28.897 7f8cc8b88700 1 -- 172.21.2.102:0/4105204346 shutdown_connections 2021-02-10T11:04:28.913 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:28.897 7f8cc8b88700 1 -- 172.21.2.102:0/4105204346 wait complete. 2021-02-10T11:04:28.913 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:28.912 7f8cc8b88700 1 Processor -- start 2021-02-10T11:04:28.913 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:28.912 7f8cc8b88700 1 -- start start 2021-02-10T11:04:28.914 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:28.912 7f8cc8b88700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8cc40b3940 0x7f8cc40b30b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:28.915 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:28.912 7f8cc8b88700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8cc419ba70 0x7f8cc40b3e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:28.915 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:28.912 7f8cc8b88700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8cc419be70 0x7f8cc40b4430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:28.915 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:28.912 7f8cc8b88700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cc40b7710 con 0x7f8cc40b3940 2021-02-10T11:04:28.915 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:28.912 7f8cc8b88700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cc40b4a50 con 0x7f8cc419ba70 2021-02-10T11:04:28.915 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:28.912 7f8cc8b88700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cc40b4c30 con 0x7f8cc419be70 2021-02-10T11:04:28.915 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:28.912 7f8cc285e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8cc40b3940 0x7f8cc40b30b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:28.915 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:28.912 7f8cc285e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8cc40b3940 0x7f8cc40b30b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:49922/0 (socket says 172.21.2.102:49922) 2021-02-10T11:04:28.915 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:28.912 7f8cc285e700 1 -- 172.21.2.102:0/4203282971 learned_addr learned my addr 172.21.2.102:0/4203282971 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:04:28.915 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:28.912 7f8cc305f700 1 --2- 172.21.2.102:0/4203282971 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8cc419be70 0x7f8cc40b4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:28.915 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:28.912 7f8cc285e700 1 -- 172.21.2.102:0/4203282971 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8cc419be70 msgr2=0x7f8cc40b4430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:28.915 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:28.912 7f8cc285e700 1 --2- 172.21.2.102:0/4203282971 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8cc419be70 0x7f8cc40b4430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:28.916 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:28.912 7f8cc285e700 1 -- 172.21.2.102:0/4203282971 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8cc419ba70 msgr2=0x7f8cc40b3e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:28.916 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:28.912 7f8cc285e700 1 --2- 172.21.2.102:0/4203282971 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8cc419ba70 0x7f8cc40b3e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:28.916 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:28.912 7f8cc285e700 1 -- 172.21.2.102:0/4203282971 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cc40b7c90 con 0x7f8cc40b3940 2021-02-10T11:04:28.916 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:28.913 7f8cc285e700 1 --2- 172.21.2.102:0/4203282971 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8cc40b3940 0x7f8cc40b30b0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7f8cac00b5c0 tx=0x7f8cac00b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:04:28.916 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:28.913 7f8cbb7fe700 1 -- 172.21.2.102:0/4203282971 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8cac006030 con 0x7f8cc40b3940 2021-02-10T11:04:28.916 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:28.913 7f8cbb7fe700 1 -- 172.21.2.102:0/4203282971 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cac000990 con 0x7f8cc40b3940 2021-02-10T11:04:28.916 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:28.913 7f8cbb7fe700 1 -- 172.21.2.102:0/4203282971 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8cac004590 con 0x7f8cc40b3940 2021-02-10T11:04:28.916 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:28.913 7f8cc8b88700 1 -- 172.21.2.102:0/4203282971 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cc40b7c90 con 0x7f8cc40b3940 2021-02-10T11:04:28.916 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:28.913 7f8cc8b88700 1 -- 172.21.2.102:0/4203282971 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cc40b7c90 con 0x7f8cc40b3940 2021-02-10T11:04:28.916 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:28.913 7f8cbb7fe700 1 -- 172.21.2.102:0/4203282971 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f8cac0043a0 con 0x7f8cc40b3940 2021-02-10T11:04:28.916 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:28.913 7f8cbb7fe700 1 --2- 172.21.2.102:0/4203282971 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8ca80186c0 0x7f8ca801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:28.916 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:28.913 7f8cbb7fe700 1 -- 172.21.2.102:0/4203282971 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f8cac0327e0 con 0x7f8cc40b3940 2021-02-10T11:04:28.916 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:28.914 7f8cc205d700 1 --2- 172.21.2.102:0/4203282971 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8ca80186c0 0x7f8ca801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:28.917 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:28.914 7f8cc205d700 1 --2- 172.21.2.102:0/4203282971 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8ca80186c0 0x7f8ca801abc0 crc :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:04:28.943 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:28.944 7f8cc8b88700 1 -- 172.21.2.102:0/4203282971 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8cc4105b30 con 0x7f8cc40b3940 2021-02-10T11:04:28.944 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:28.947 7f8cbb7fe700 1 -- 172.21.2.102:0/4203282971 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f8cac006b80 con 0x7f8cc40b3940 2021-02-10T11:04:29.073 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:29.075 7f8cc8b88700 1 -- 172.21.2.102:0/4203282971 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f8cc4105b30 con 0x7f8cc40b3940 2021-02-10T11:04:29.073 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:29.075 7f8cbb7fe700 1 -- 172.21.2.102:0/4203282971 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v20) v1 ==== 92+0+10 (secure 0 0 0) 0x7f8cac0043a0 con 0x7f8cc40b3940 2021-02-10T11:04:29.073 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:04:29.075 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:29.077 7f8cc8b88700 1 -- 172.21.2.102:0/4203282971 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8ca80186c0 msgr2=0x7f8ca801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:29.075 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:29.077 7f8cc8b88700 1 --2- 172.21.2.102:0/4203282971 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8ca80186c0 0x7f8ca801abc0 crc :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:29.075 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:29.077 7f8cc8b88700 1 -- 172.21.2.102:0/4203282971 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8cc40b3940 msgr2=0x7f8cc40b30b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:29.075 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:29.077 7f8cc8b88700 1 --2- 172.21.2.102:0/4203282971 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8cc40b3940 0x7f8cc40b30b0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7f8cac00b5c0 tx=0x7f8cac00b960).stop 2021-02-10T11:04:29.075 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:29.077 7f8cc8b88700 1 -- 172.21.2.102:0/4203282971 shutdown_connections 2021-02-10T11:04:29.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:29.077 7f8cc8b88700 1 --2- 172.21.2.102:0/4203282971 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8cc419be70 0x7f8cc40b4430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:29.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:29.077 7f8cc8b88700 1 --2- 172.21.2.102:0/4203282971 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8ca80186c0 0x7f8ca801abc0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:29.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:29.077 7f8cc8b88700 1 --2- 172.21.2.102:0/4203282971 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8cc419ba70 0x7f8cc40b3e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:29.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:29.077 7f8cc8b88700 1 --2- 172.21.2.102:0/4203282971 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8cc40b3940 0x7f8cc40b30b0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:29.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:29.077 7f8cc8b88700 1 -- 172.21.2.102:0/4203282971 >> 172.21.2.102:0/4203282971 conn(0x7f8cc4094e10 msgr2=0x7f8cc40a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:04:29.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:29.077 7f8cc8b88700 1 -- 172.21.2.102:0/4203282971 shutdown_connections 2021-02-10T11:04:29.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:29.078 7f8cc8b88700 1 -- 172.21.2.102:0/4203282971 wait complete. 2021-02-10T11:04:29.085 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-02-10T11:04:29.512 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:29.498 7f896be47700 1 Processor -- start 2021-02-10T11:04:29.513 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:29.498 7f896be47700 1 -- start start 2021-02-10T11:04:29.513 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:29.498 7f896be47700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f89641b0870 0x7f89640f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:29.513 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:29.498 7f896be47700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f89640efe00 0x7f89640f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:29.513 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:29.498 7f896be47700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89641b0c70 con 0x7f89641b0470 2021-02-10T11:04:29.513 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:29.498 7f896be47700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89640e5ea0 con 0x7f89640efe00 2021-02-10T11:04:29.513 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:29.499 7f896be47700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89640f0b50 con 0x7f89641b0870 2021-02-10T11:04:29.513 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:29.499 7f8969c63700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f89641b0470 0x7f89640ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:42154/0 (socket says 172.21.2.102:42154) 2021-02-10T11:04:29.513 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:29.499 7f8969c63700 1 -- 172.21.2.102:0/2447485088 learned_addr learned my addr 172.21.2.102:0/2447485088 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:04:29.513 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:29.499 7f896a464700 1 --2- 172.21.2.102:0/2447485088 >> v2:172.21.2.102:3300/0 conn(0x7f89640efe00 0x7f89640f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:29.513 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:29.499 7f8968c61700 1 -- 172.21.2.102:0/2447485088 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4121711040 0 0) 0x7f89600010b0 con 0x7f89641b0470 2021-02-10T11:04:29.513 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:29.499 7f896a464700 1 -- 172.21.2.102:0/2447485088 >> v2:172.21.2.113:3300/0 conn(0x7f89641b0870 msgr2=0x7f89640f20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:04:29.513 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:29.499 7f896a464700 1 --2- 172.21.2.102:0/2447485088 >> v2:172.21.2.113:3300/0 conn(0x7f89641b0870 0x7f89640f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:29.513 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:29.499 7f896a464700 1 -- 172.21.2.102:0/2447485088 >> v1:172.21.2.102:6789/0 conn(0x7f89641b0470 legacy=0x7f89640ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:29.514 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:29.499 7f896a464700 1 -- 172.21.2.102:0/2447485088 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89640f0820 con 0x7f89640efe00 2021-02-10T11:04:29.514 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:29.499 7f896a464700 1 --2- 172.21.2.102:0/2447485088 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f89640efe00 0x7f89640f0200 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7f895c006e30 tx=0x7f895c009e60).ready entity=mon.0 client_cookie=f669607d26d9ebac server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:04:29.514 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:29.500 7f8968c61700 1 -- 172.21.2.102:0/2447485088 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f895c006a50 con 0x7f89640efe00 2021-02-10T11:04:29.514 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:29.500 7f8968c61700 1 -- 172.21.2.102:0/2447485088 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f895c00b300 con 0x7f89640efe00 2021-02-10T11:04:29.514 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:29.500 7f8968c61700 1 -- 172.21.2.102:0/2447485088 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f895c00b820 con 0x7f89640efe00 2021-02-10T11:04:29.514 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:29.500 7f896be47700 1 -- 172.21.2.102:0/2447485088 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f89640efe00 msgr2=0x7f89640f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:29.514 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:29.500 7f896be47700 1 --2- 172.21.2.102:0/2447485088 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f89640efe00 0x7f89640f0200 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7f895c006e30 tx=0x7f895c009e60).stop 2021-02-10T11:04:29.514 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:29.500 7f896be47700 1 -- 172.21.2.102:0/2447485088 shutdown_connections 2021-02-10T11:04:29.514 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:29.500 7f896be47700 1 --2- 172.21.2.102:0/2447485088 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f89640efe00 0x7f89640f0200 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:29.514 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:29.500 7f896be47700 1 --2- 172.21.2.102:0/2447485088 >> v2:172.21.2.113:3300/0 conn(0x7f89641b0870 0x7f89640f20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:29.514 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:29.500 7f896be47700 1 -- 172.21.2.102:0/2447485088 >> 172.21.2.102:0/2447485088 conn(0x7f89640ce3e0 msgr2=0x7f89640e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:04:29.514 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:29.500 7f896be47700 1 -- 172.21.2.102:0/2447485088 shutdown_connections 2021-02-10T11:04:29.515 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:29.500 7f896be47700 1 -- 172.21.2.102:0/2447485088 wait complete. 2021-02-10T11:04:29.516 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:29.515 7f896be47700 1 Processor -- start 2021-02-10T11:04:29.516 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:29.515 7f896be47700 1 -- start start 2021-02-10T11:04:29.516 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:29.515 7f896be47700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8964193230 0x7f89640a6a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:29.516 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:29.515 7f896be47700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f896408f5b0 0x7f896408f9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:29.516 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:29.515 7f896be47700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f896408e6a0 0x7f896408eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:29.516 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:29.515 7f896be47700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f896408ffd0 con 0x7f8964193230 2021-02-10T11:04:29.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:29.515 7f896be47700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8964090240 con 0x7f896408f5b0 2021-02-10T11:04:29.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:29.515 7f896be47700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f896408f0c0 con 0x7f896408e6a0 2021-02-10T11:04:29.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:29.516 7f8969c63700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8964193230 0x7f89640a6a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:29.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:29.516 7f8969c63700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8964193230 0x7f89640a6a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:49936/0 (socket says 172.21.2.102:49936) 2021-02-10T11:04:29.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:29.516 7f8969c63700 1 -- 172.21.2.102:0/325772029 learned_addr learned my addr 172.21.2.102:0/325772029 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:04:29.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:29.516 7f896a464700 1 --2- 172.21.2.102:0/325772029 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f896408e6a0 0x7f896408eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:29.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:29.516 7f8969c63700 1 -- 172.21.2.102:0/325772029 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f896408e6a0 msgr2=0x7f896408eaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:29.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:29.516 7f8969c63700 1 --2- 172.21.2.102:0/325772029 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f896408e6a0 0x7f896408eaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:29.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:29.516 7f8969c63700 1 -- 172.21.2.102:0/325772029 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f896408f5b0 msgr2=0x7f896408f9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:29.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:29.516 7f8969c63700 1 --2- 172.21.2.102:0/325772029 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f896408f5b0 0x7f896408f9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:29.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:29.516 7f8969c63700 1 -- 172.21.2.102:0/325772029 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f896408f2a0 con 0x7f8964193230 2021-02-10T11:04:29.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:29.516 7f8969c63700 1 --2- 172.21.2.102:0/325772029 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8964193230 0x7f89640a6a80 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f896000a4c0 tx=0x7f896000a860).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:04:29.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:29.516 7f89567fc700 1 -- 172.21.2.102:0/325772029 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f896000ac50 con 0x7f8964193230 2021-02-10T11:04:29.518 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:29.516 7f89567fc700 1 -- 172.21.2.102:0/325772029 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8960004140 con 0x7f8964193230 2021-02-10T11:04:29.518 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:29.516 7f89567fc700 1 -- 172.21.2.102:0/325772029 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8960004660 con 0x7f8964193230 2021-02-10T11:04:29.518 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:29.517 7f896be47700 1 -- 172.21.2.102:0/325772029 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f896408f2a0 con 0x7f8964193230 2021-02-10T11:04:29.518 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:29.517 7f896be47700 1 -- 172.21.2.102:0/325772029 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f896408f2a0 con 0x7f8964193230 2021-02-10T11:04:29.518 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:29.517 7f89567fc700 1 -- 172.21.2.102:0/325772029 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f8960001090 con 0x7f8964193230 2021-02-10T11:04:29.518 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:29.517 7f89567fc700 1 --2- 172.21.2.102:0/325772029 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f894c0186c0 0x7f894c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:29.518 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:29.517 7f89567fc700 1 -- 172.21.2.102:0/325772029 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f8960032540 con 0x7f8964193230 2021-02-10T11:04:29.518 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:29.517 7f8969462700 1 --2- 172.21.2.102:0/325772029 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f894c0186c0 0x7f894c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:29.518 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:29.518 7f8969462700 1 --2- 172.21.2.102:0/325772029 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f894c0186c0 0x7f894c01abc0 crc :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:04:29.547 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:29.549 7f896be47700 1 -- 172.21.2.102:0/325772029 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89640f0b50 con 0x7f8964193230 2021-02-10T11:04:29.550 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:29.551 7f89567fc700 1 -- 172.21.2.102:0/325772029 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f8960000980 con 0x7f8964193230 2021-02-10T11:04:29.676 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:29.678 7f896be47700 1 -- 172.21.2.102:0/325772029 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7f89640f0b50 con 0x7f8964193230 2021-02-10T11:04:29.676 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:29.678 7f89567fc700 1 -- 172.21.2.102:0/325772029 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v20) v1 ==== 104+0+10 (secure 0 0 0) 0x7f8960001090 con 0x7f8964193230 2021-02-10T11:04:29.677 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:04:29.678 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:29.680 7f896be47700 1 -- 172.21.2.102:0/325772029 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f894c0186c0 msgr2=0x7f894c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:29.678 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:29.680 7f896be47700 1 --2- 172.21.2.102:0/325772029 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f894c0186c0 0x7f894c01abc0 crc :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:29.679 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:29.680 7f896be47700 1 -- 172.21.2.102:0/325772029 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8964193230 msgr2=0x7f89640a6a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:29.679 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:29.680 7f896be47700 1 --2- 172.21.2.102:0/325772029 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8964193230 0x7f89640a6a80 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f896000a4c0 tx=0x7f896000a860).stop 2021-02-10T11:04:29.679 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:29.680 7f896be47700 1 -- 172.21.2.102:0/325772029 shutdown_connections 2021-02-10T11:04:29.679 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:29.680 7f896be47700 1 --2- 172.21.2.102:0/325772029 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f896408e6a0 0x7f896408eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:29.679 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:29.680 7f896be47700 1 --2- 172.21.2.102:0/325772029 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f894c0186c0 0x7f894c01abc0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:29.679 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:29.680 7f896be47700 1 --2- 172.21.2.102:0/325772029 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f896408f5b0 0x7f896408f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:29.679 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:29.680 7f896be47700 1 --2- 172.21.2.102:0/325772029 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8964193230 0x7f89640a6a80 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:29.679 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:29.680 7f896be47700 1 -- 172.21.2.102:0/325772029 >> 172.21.2.102:0/325772029 conn(0x7f89640bf4e0 msgr2=0x7f89640be160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:04:29.679 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:29.681 7f896be47700 1 -- 172.21.2.102:0/325772029 shutdown_connections 2021-02-10T11:04:29.679 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:29.681 7f896be47700 1 -- 172.21.2.102:0/325772029 wait complete. 2021-02-10T11:04:29.687 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-02-10T11:04:30.116 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:30.101 7f999ad23700 1 Processor -- start 2021-02-10T11:04:30.117 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:30.101 7f999ad23700 1 -- start start 2021-02-10T11:04:30.117 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:30.102 7f999ad23700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f99941b0870 0x7f99940f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:30.117 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:30.102 7f999ad23700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f99940efe00 0x7f99940f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:30.117 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:30.102 7f999ad23700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99941b0c70 con 0x7f99941b0470 2021-02-10T11:04:30.117 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:30.102 7f999ad23700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99940f0b50 con 0x7f99941b0870 2021-02-10T11:04:30.117 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:30.102 7f999ad23700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99940f0820 con 0x7f99940efe00 2021-02-10T11:04:30.117 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:30.102 7f9998b3f700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f99941b0470 0x7f99940ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:42166/0 (socket says 172.21.2.102:42166) 2021-02-10T11:04:30.117 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:30.102 7f9998b3f700 1 -- 172.21.2.102:0/1474684386 learned_addr learned my addr 172.21.2.102:0/1474684386 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:04:30.117 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:30.102 7f99937fe700 1 -- 172.21.2.102:0/1474684386 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3180783037 0 0) 0x7f997c0009b0 con 0x7f99941b0470 2021-02-10T11:04:30.117 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:30.102 7f99937fe700 1 -- 172.21.2.102:0/1474684386 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9980002fe0 con 0x7f99941b0470 2021-02-10T11:04:30.118 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:30.102 7f9993fff700 1 --2- 172.21.2.102:0/1474684386 >> v2:172.21.2.113:3300/0 conn(0x7f99941b0870 0x7f99940f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:30.118 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:30.102 7f9999340700 1 --2- 172.21.2.102:0/1474684386 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f99940efe00 0x7f99940f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:30.118 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:30.103 7f99937fe700 1 -- 172.21.2.102:0/1474684386 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1038418854 0 0) 0x7f997c001010 con 0x7f99941b0470 2021-02-10T11:04:30.118 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:30.103 7f99937fe700 1 -- 172.21.2.102:0/1474684386 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f99940efe00 msgr2=0x7f99940f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:04:30.118 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:30.103 7f99937fe700 1 --2- 172.21.2.102:0/1474684386 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f99940efe00 0x7f99940f0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:30.118 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:30.103 7f99937fe700 1 -- 172.21.2.102:0/1474684386 >> v2:172.21.2.113:3300/0 conn(0x7f99941b0870 msgr2=0x7f99940f20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:04:30.118 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:30.103 7f99937fe700 1 --2- 172.21.2.102:0/1474684386 >> v2:172.21.2.113:3300/0 conn(0x7f99941b0870 0x7f99940f20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:30.118 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:30.103 7f99937fe700 1 -- 172.21.2.102:0/1474684386 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99940e52e0 con 0x7f99941b0470 2021-02-10T11:04:30.118 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:30.103 7f99937fe700 1 -- 172.21.2.102:0/1474684386 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7f997c001590 con 0x7f99941b0470 2021-02-10T11:04:30.118 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:30.103 7f99937fe700 1 -- 172.21.2.102:0/1474684386 >> v1:172.21.2.102:6789/0 conn(0x7f99941b0470 legacy=0x7f99940ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:30.118 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:30.103 7f99937fe700 1 --2- 172.21.2.102:0/1474684386 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f9980003ae0 0x7f9980005f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:30.118 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:30.103 7f99937fe700 1 --2- 172.21.2.102:0/1474684386 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9980006830 0x7f9980008d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:30.118 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:30.103 7f99937fe700 1 --2- 172.21.2.102:0/1474684386 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f99800095a0 0x7f998000bac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:30.119 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:30.103 7f99937fe700 1 -- 172.21.2.102:0/1474684386 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f998000c370 con 0x7f99800095a0 2021-02-10T11:04:30.119 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:30.103 7f99937fe700 1 -- 172.21.2.102:0/1474684386 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f998000c590 con 0x7f9980003ae0 2021-02-10T11:04:30.119 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:30.103 7f99937fe700 1 -- 172.21.2.102:0/1474684386 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99800507c0 con 0x7f9980006830 2021-02-10T11:04:30.119 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:30.103 7f9999340700 1 --2- 172.21.2.102:0/1474684386 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f99940efe00 0x7f99940f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:04:30.119 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:30.103 7f9999340700 1 --2- 172.21.2.102:0/1474684386 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f99800095a0 0x7f998000bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:30.119 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:30.103 7f9998b3f700 1 --2- 172.21.2.102:0/1474684386 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f9980003ae0 0x7f9980005f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:30.119 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:30.103 7f9993fff700 1 --2- 172.21.2.102:0/1474684386 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9980006830 0x7f9980008d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:30.119 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:30.103 7f9999340700 1 -- 172.21.2.102:0/1474684386 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9980006830 msgr2=0x7f9980008d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:04:30.119 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:30.103 7f9999340700 1 --2- 172.21.2.102:0/1474684386 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9980006830 0x7f9980008d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:30.119 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:30.103 7f9999340700 1 -- 172.21.2.102:0/1474684386 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f9980003ae0 msgr2=0x7f9980005f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:04:30.119 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:30.103 7f9999340700 1 --2- 172.21.2.102:0/1474684386 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f9980003ae0 0x7f9980005f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:30.119 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:30.103 7f9999340700 1 -- 172.21.2.102:0/1474684386 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9980050a20 con 0x7f99800095a0 2021-02-10T11:04:30.119 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:30.103 7f9999340700 1 --2- 172.21.2.102:0/1474684386 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f99800095a0 0x7f998000bac0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f998c001140 tx=0x7f998c009e50).ready entity=mon.0 client_cookie=85df126c58eaec14 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:04:30.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:30.103 7f99937fe700 1 -- 172.21.2.102:0/1474684386 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f998c00b510 con 0x7f99800095a0 2021-02-10T11:04:30.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:30.103 7f99937fe700 1 -- 172.21.2.102:0/1474684386 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f998c000f60 con 0x7f99800095a0 2021-02-10T11:04:30.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:30.103 7f99937fe700 1 -- 172.21.2.102:0/1474684386 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f998c0069a0 con 0x7f99800095a0 2021-02-10T11:04:30.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:30.103 7f999ad23700 1 -- 172.21.2.102:0/1474684386 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f99800095a0 msgr2=0x7f998000bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:30.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:30.103 7f999ad23700 1 --2- 172.21.2.102:0/1474684386 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f99800095a0 0x7f998000bac0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f998c001140 tx=0x7f998c009e50).stop 2021-02-10T11:04:30.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:30.103 7f9999340700 1 -- 172.21.2.102:0/1474684386 reap_dead start 2021-02-10T11:04:30.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:30.104 7f999ad23700 1 -- 172.21.2.102:0/1474684386 shutdown_connections 2021-02-10T11:04:30.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:30.104 7f999ad23700 1 -- 172.21.2.102:0/1474684386 >> 172.21.2.102:0/1474684386 conn(0x7f99940ce3e0 msgr2=0x7f99940e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:04:30.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:30.104 7f999ad23700 1 -- 172.21.2.102:0/1474684386 shutdown_connections 2021-02-10T11:04:30.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:30.104 7f999ad23700 1 -- 172.21.2.102:0/1474684386 wait complete. 2021-02-10T11:04:30.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:30.119 7f999ad23700 1 Processor -- start 2021-02-10T11:04:30.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:30.119 7f999ad23700 1 -- start start 2021-02-10T11:04:30.122 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:30.119 7f999ad23700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f99940b7db0 0x7f99940b8260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:30.122 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:30.119 7f999ad23700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f99940b8880 0x7f99941a7450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:30.122 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:30.119 7f999ad23700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f99940b8c80 0x7f99941a7a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:30.122 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:30.119 7f999ad23700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99940b79b0 con 0x7f99940b8880 2021-02-10T11:04:30.122 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:30.119 7f999ad23700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99941a8090 con 0x7f99940b7db0 2021-02-10T11:04:30.122 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:30.119 7f999ad23700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99941a82b0 con 0x7f99940b8c80 2021-02-10T11:04:30.122 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:30.120 7f9993fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f99940b8880 0x7f99941a7450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:30.122 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:30.120 7f9998b3f700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f99940b7db0 0x7f99940b8260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:30.123 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:30.120 7f9993fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f99940b8880 0x7f99941a7450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:49956/0 (socket says 172.21.2.102:49956) 2021-02-10T11:04:30.123 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:30.120 7f9993fff700 1 -- 172.21.2.102:0/3906416315 learned_addr learned my addr 172.21.2.102:0/3906416315 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:04:30.123 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:30.120 7f9998b3f700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f99940b7db0 0x7f99940b8260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.113:3300/0 says I am v2:172.21.2.102:53388/0 (socket says 172.21.2.102:53388) 2021-02-10T11:04:30.123 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:30.120 7f9999340700 1 --2- 172.21.2.102:0/3906416315 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f99940b8c80 0x7f99941a7a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:30.123 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:30.120 7f9998b3f700 1 -- 172.21.2.102:0/3906416315 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f99940b8c80 msgr2=0x7f99941a7a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:30.123 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:30.120 7f9998b3f700 1 --2- 172.21.2.102:0/3906416315 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f99940b8c80 0x7f99941a7a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:30.123 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:30.120 7f9998b3f700 1 -- 172.21.2.102:0/3906416315 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f99940b8880 msgr2=0x7f99941a7450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:30.123 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:30.120 7f9998b3f700 1 --2- 172.21.2.102:0/3906416315 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f99940b8880 0x7f99941a7450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:30.123 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:30.120 7f9998b3f700 1 -- 172.21.2.102:0/3906416315 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9994190c10 con 0x7f99940b7db0 2021-02-10T11:04:30.123 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:30.120 7f9993fff700 1 --2- 172.21.2.102:0/3906416315 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f99940b8880 0x7f99941a7450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:04:30.123 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:30.120 7f9998b3f700 1 --2- 172.21.2.102:0/3906416315 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f99940b7db0 0x7f99940b8260 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f997c00a4c0 tx=0x7f997c00a860).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:04:30.123 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:30.120 7f99917fa700 1 -- 172.21.2.102:0/3906416315 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f997c0010d0 con 0x7f99940b7db0 2021-02-10T11:04:30.123 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:30.120 7f99917fa700 1 -- 172.21.2.102:0/3906416315 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f997c00ac50 con 0x7f99940b7db0 2021-02-10T11:04:30.124 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:30.120 7f999ad23700 1 -- 172.21.2.102:0/3906416315 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9994190c10 con 0x7f99940b7db0 2021-02-10T11:04:30.124 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:30.120 7f99917fa700 1 -- 172.21.2.102:0/3906416315 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f997c0009b0 con 0x7f99940b7db0 2021-02-10T11:04:30.124 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:30.121 7f999ad23700 1 -- 172.21.2.102:0/3906416315 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9994190c10 con 0x7f99940b7db0 2021-02-10T11:04:30.124 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:30.121 7f99917fa700 1 -- 172.21.2.102:0/3906416315 <== mon.1 v2:172.21.2.113:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f997c0010b0 con 0x7f99940b7db0 2021-02-10T11:04:30.124 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:30.121 7f99917fa700 1 --2- 172.21.2.102:0/3906416315 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f99780186d0 0x7f997801abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:30.124 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:30.121 7f99917fa700 1 -- 172.21.2.102:0/3906416315 <== mon.1 v2:172.21.2.113:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f997c032510 con 0x7f99940b7db0 2021-02-10T11:04:30.124 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:30.121 7f9993fff700 1 --2- 172.21.2.102:0/3906416315 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f99780186d0 0x7f997801abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:30.124 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:30.121 7f9993fff700 1 --2- 172.21.2.102:0/3906416315 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f99780186d0 0x7f997801abd0 crc :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:04:30.149 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:30.151 7f999ad23700 1 -- 172.21.2.102:0/3906416315 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9994105b30 con 0x7f99940b7db0 2021-02-10T11:04:30.153 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:30.155 7f99917fa700 1 -- 172.21.2.102:0/3906416315 <== mon.1 v2:172.21.2.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f997c004140 con 0x7f99940b7db0 2021-02-10T11:04:30.283 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:30.285 7f999ad23700 1 -- 172.21.2.102:0/3906416315 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7f99940f0b50 con 0x7f99940b7db0 2021-02-10T11:04:30.284 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:30.286 7f99917fa700 1 -- 172.21.2.102:0/3906416315 <== mon.1 v2:172.21.2.113:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v20) v1 ==== 100+0+10 (secure 0 0 0) 0x7f997c0010b0 con 0x7f99940b7db0 2021-02-10T11:04:30.284 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:04:30.285 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:30.288 7f999ad23700 1 -- 172.21.2.102:0/3906416315 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f99780186d0 msgr2=0x7f997801abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:30.286 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:30.288 7f999ad23700 1 --2- 172.21.2.102:0/3906416315 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f99780186d0 0x7f997801abd0 crc :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:30.286 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:30.288 7f999ad23700 1 -- 172.21.2.102:0/3906416315 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f99940b7db0 msgr2=0x7f99940b8260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:30.286 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:30.288 7f999ad23700 1 --2- 172.21.2.102:0/3906416315 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f99940b7db0 0x7f99940b8260 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f997c00a4c0 tx=0x7f997c00a860).stop 2021-02-10T11:04:30.286 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:30.288 7f999ad23700 1 -- 172.21.2.102:0/3906416315 shutdown_connections 2021-02-10T11:04:30.286 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:30.288 7f999ad23700 1 --2- 172.21.2.102:0/3906416315 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f99940b8c80 0x7f99941a7a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:30.286 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:30.288 7f999ad23700 1 --2- 172.21.2.102:0/3906416315 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f99940b8880 0x7f99941a7450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:30.286 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:30.288 7f999ad23700 1 --2- 172.21.2.102:0/3906416315 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f99780186d0 0x7f997801abd0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:30.287 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:30.288 7f999ad23700 1 --2- 172.21.2.102:0/3906416315 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f99940b7db0 0x7f99940b8260 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:30.287 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:30.288 7f999ad23700 1 -- 172.21.2.102:0/3906416315 >> 172.21.2.102:0/3906416315 conn(0x7f9994094e10 msgr2=0x7f99940a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:04:30.287 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:30.288 7f999ad23700 1 -- 172.21.2.102:0/3906416315 shutdown_connections 2021-02-10T11:04:30.287 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:30.288 7f999ad23700 1 -- 172.21.2.102:0/3906416315 wait complete. 2021-02-10T11:04:30.294 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-02-10T11:04:30.294 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:04:30.720 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:30.706 7f1085206700 1 Processor -- start 2021-02-10T11:04:30.721 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:30.706 7f1085206700 1 -- start start 2021-02-10T11:04:30.721 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:30.707 7f1085206700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f10800e5070 0x7f10801b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:30.721 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:30.707 7f1085206700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f10800e5470 0x7f1080080310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:30.721 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:30.707 7f1085206700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f108007ed80 con 0x7f10801b3e40 2021-02-10T11:04:30.721 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:30.707 7f1085206700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10800d1790 con 0x7f10800e5070 2021-02-10T11:04:30.721 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:30.707 7f1085206700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10800d19b0 con 0x7f10800e5470 2021-02-10T11:04:30.721 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:30.707 7f107ed9d700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f10800e5070 0x7f10801b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:30.721 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:30.707 7f107ed9d700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f10800e5070 0x7f10801b3820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:49962/0 (socket says 172.21.2.102:49962) 2021-02-10T11:04:30.721 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:30.707 7f107ed9d700 1 -- 172.21.2.102:0/4010405467 learned_addr learned my addr 172.21.2.102:0/4010405467 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:04:30.721 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:30.707 7f107ed9d700 1 -- 172.21.2.102:0/4010405467 >> v2:172.21.2.113:3300/0 conn(0x7f10800e5470 msgr2=0x7f1080080310 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:04:30.721 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:30.707 7f107ed9d700 1 --2- 172.21.2.102:0/4010405467 >> v2:172.21.2.113:3300/0 conn(0x7f10800e5470 0x7f1080080310 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:30.721 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:30.707 7f107ed9d700 1 -- 172.21.2.102:0/4010405467 >> v1:172.21.2.113:6789/0 conn(0x7f10801b3e40 legacy=0x7f108007d690 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:04:30.722 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:30.707 7f107ed9d700 1 -- 172.21.2.102:0/4010405467 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10800d1bd0 con 0x7f10800e5070 2021-02-10T11:04:30.722 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:30.707 7f107ed9d700 1 --2- 172.21.2.102:0/4010405467 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f10800e5070 0x7f10801b3820 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7f1068006700 tx=0x7f1068009b80).ready entity=mon.0 client_cookie=5b452dc8c86e7b52 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:04:30.722 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:30.708 7f107dd9b700 1 -- 172.21.2.102:0/4010405467 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f106800c0c0 con 0x7f10800e5070 2021-02-10T11:04:30.722 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:30.708 7f107dd9b700 1 -- 172.21.2.102:0/4010405467 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f106800c340 con 0x7f10800e5070 2021-02-10T11:04:30.722 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:30.708 7f107dd9b700 1 -- 172.21.2.102:0/4010405467 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f106800c860 con 0x7f10800e5070 2021-02-10T11:04:30.722 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:30.708 7f1085206700 1 -- 172.21.2.102:0/4010405467 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f10800e5070 msgr2=0x7f10801b3820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:30.722 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:30.708 7f1085206700 1 --2- 172.21.2.102:0/4010405467 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f10800e5070 0x7f10801b3820 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7f1068006700 tx=0x7f1068009b80).stop 2021-02-10T11:04:30.722 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:30.708 7f1085206700 1 -- 172.21.2.102:0/4010405467 shutdown_connections 2021-02-10T11:04:30.722 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:30.708 7f1085206700 1 --2- 172.21.2.102:0/4010405467 >> v2:172.21.2.113:3300/0 conn(0x7f10800e5470 0x7f1080080310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:30.722 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:30.708 7f1085206700 1 --2- 172.21.2.102:0/4010405467 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f10800e5070 0x7f10801b3820 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:30.722 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:30.708 7f1085206700 1 -- 172.21.2.102:0/4010405467 >> 172.21.2.102:0/4010405467 conn(0x7f10800ce690 msgr2=0x7f10800e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:04:30.723 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:30.709 7f1085206700 1 -- 172.21.2.102:0/4010405467 shutdown_connections 2021-02-10T11:04:30.723 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:30.709 7f1085206700 1 -- 172.21.2.102:0/4010405467 wait complete. 2021-02-10T11:04:30.724 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:30.723 7f1085206700 1 Processor -- start 2021-02-10T11:04:30.724 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:30.723 7f1085206700 1 -- start start 2021-02-10T11:04:30.724 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:30.723 7f1085206700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f10800a5770 0x7f10800b72b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:30.724 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:30.723 7f1085206700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f10800b78d0 0x7f10800b7cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:30.724 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:30.723 7f1085206700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f10801a2b50 0x7f10801a2f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:30.724 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:30.723 7f1085206700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10801a3df0 con 0x7f10800a5770 2021-02-10T11:04:30.724 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:30.723 7f1085206700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1080092d90 con 0x7f10801a2b50 2021-02-10T11:04:30.725 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:30.723 7f1085206700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10801a3570 con 0x7f10800b78d0 2021-02-10T11:04:30.725 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:30.724 7f107ed9d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f10800a5770 0x7f10800b72b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:30.725 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:30.724 7f107ed9d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f10800a5770 0x7f10800b72b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:49970/0 (socket says 172.21.2.102:49970) 2021-02-10T11:04:30.725 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:30.724 7f107ed9d700 1 -- 172.21.2.102:0/3155092594 learned_addr learned my addr 172.21.2.102:0/3155092594 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:04:30.725 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:30.724 7f107ed9d700 1 -- 172.21.2.102:0/3155092594 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f10800b78d0 msgr2=0x7f10800b7cd0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:04:30.725 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:30.724 7f107ed9d700 1 --2- 172.21.2.102:0/3155092594 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f10800b78d0 0x7f10800b7cd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:30.725 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:30.724 7f107ed9d700 1 -- 172.21.2.102:0/3155092594 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f10801a2b50 msgr2=0x7f10801a2f50 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:04:30.725 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:30.724 7f107ed9d700 1 --2- 172.21.2.102:0/3155092594 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f10801a2b50 0x7f10801a2f50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:30.725 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:30.724 7f107ed9d700 1 -- 172.21.2.102:0/3155092594 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10801a3790 con 0x7f10800a5770 2021-02-10T11:04:30.725 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:30.724 7f107ed9d700 1 --2- 172.21.2.102:0/3155092594 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f10800a5770 0x7f10800b72b0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f106800b650 tx=0x7f106800b9f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:04:30.725 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:30.724 7f10777fe700 1 -- 172.21.2.102:0/3155092594 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f106800bde0 con 0x7f10800a5770 2021-02-10T11:04:30.725 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:30.724 7f10777fe700 1 -- 172.21.2.102:0/3155092594 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1068006520 con 0x7f10800a5770 2021-02-10T11:04:30.726 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:30.724 7f10777fe700 1 -- 172.21.2.102:0/3155092594 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f10680022b0 con 0x7f10800a5770 2021-02-10T11:04:30.726 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:30.724 7f1085206700 1 -- 172.21.2.102:0/3155092594 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10801a3790 con 0x7f10800a5770 2021-02-10T11:04:30.726 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:30.724 7f1085206700 1 -- 172.21.2.102:0/3155092594 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10801a3790 con 0x7f10800a5770 2021-02-10T11:04:30.726 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:30.725 7f10777fe700 1 -- 172.21.2.102:0/3155092594 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1068000c90 con 0x7f10800a5770 2021-02-10T11:04:30.726 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:30.725 7f10777fe700 1 --2- 172.21.2.102:0/3155092594 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f10640186c0 0x7f106401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:30.726 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:30.725 7f10777fe700 1 -- 172.21.2.102:0/3155092594 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f10680325c0 con 0x7f10800a5770 2021-02-10T11:04:30.726 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:30.725 7f107e59c700 1 --2- 172.21.2.102:0/3155092594 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f10640186c0 0x7f106401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:30.726 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:30.726 7f107e59c700 1 --2- 172.21.2.102:0/3155092594 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f10640186c0 0x7f106401abc0 crc :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:04:30.754 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:30.756 7f1085206700 1 -- 172.21.2.102:0/3155092594 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1080083dd0 con 0x7f10800a5770 2021-02-10T11:04:30.757 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:30.759 7f10777fe700 1 -- 172.21.2.102:0/3155092594 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f1068030d70 con 0x7f10800a5770 2021-02-10T11:04:30.886 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:30.888 7f1085206700 1 -- 172.21.2.102:0/3155092594 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1080092d50 con 0x7f10800a5770 2021-02-10T11:04:30.887 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:30.889 7f10777fe700 1 -- 172.21.2.102:0/3155092594 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+2094 (secure 0 0 0) 0x7f106800bde0 con 0x7f10800a5770 2021-02-10T11:04:30.887 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:04:30.887 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":6,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4302,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.102:6835/3297596596","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6834","nonce":3297596596},{"type":"v1","addr":"172.21.2.102:6835","nonce":3297596596}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":4}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:03:43.384709","modified":"2021-02-10 11:03:52.791414","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4226},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4226":{"gid":4226,"name":"b","rank":0,"incarnation":5,"state":"up:active","state_seq":3,"addr":"172.21.2.113:6833/285807165","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6832","nonce":285807165},{"type":"v1","addr":"172.21.2.113:6833","nonce":285807165}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2021-02-10T11:04:30.888 INFO:teuthology.orchestra.run.gibba002.stderr:dumped fsmap epoch 6 2021-02-10T11:04:30.889 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:30.891 7f1085206700 1 -- 172.21.2.102:0/3155092594 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f10640186c0 msgr2=0x7f106401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:30.889 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:30.891 7f1085206700 1 --2- 172.21.2.102:0/3155092594 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f10640186c0 0x7f106401abc0 crc :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:30.889 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:30.891 7f1085206700 1 -- 172.21.2.102:0/3155092594 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f10800a5770 msgr2=0x7f10800b72b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:30.889 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:30.891 7f1085206700 1 --2- 172.21.2.102:0/3155092594 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f10800a5770 0x7f10800b72b0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f106800b650 tx=0x7f106800b9f0).stop 2021-02-10T11:04:30.889 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:30.891 7f1085206700 1 -- 172.21.2.102:0/3155092594 shutdown_connections 2021-02-10T11:04:30.890 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:30.891 7f1085206700 1 --2- 172.21.2.102:0/3155092594 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f10640186c0 0x7f106401abc0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:30.890 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:30.891 7f1085206700 1 --2- 172.21.2.102:0/3155092594 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f10801a2b50 0x7f10801a2f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:30.890 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:30.891 7f1085206700 1 --2- 172.21.2.102:0/3155092594 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f10800b78d0 0x7f10800b7cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:30.890 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:30.891 7f1085206700 1 --2- 172.21.2.102:0/3155092594 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f10800a5770 0x7f10800b72b0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:30.890 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:30.891 7f1085206700 1 -- 172.21.2.102:0/3155092594 >> 172.21.2.102:0/3155092594 conn(0x7f1080093720 msgr2=0x7f1080084b80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:04:30.890 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:30.891 7f1085206700 1 -- 172.21.2.102:0/3155092594 shutdown_connections 2021-02-10T11:04:30.890 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:30.891 7f1085206700 1 -- 172.21.2.102:0/3155092594 wait complete. 2021-02-10T11:04:31.100 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 6, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:03:43.384709', 'modified': '2021-02-10 11:03:52.791414', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4226}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4226': {'gid': 4226, 'name': 'b', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.113:6833/285807165', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.113:6832', 'nonce': 285807165}, {'type': 'v1', 'addr': '172.21.2.113:6833', 'nonce': 285807165}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:04:31.100 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:04:31.100 DEBUG:teuthology.orchestra.run.gibba013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:04:31.499 INFO:teuthology.orchestra.run.gibba013.stdout:{ 2021-02-10T11:04:31.499 INFO:teuthology.orchestra.run.gibba013.stdout: "cluster_fsid": "68db30ea-dbce-4aef-a6df-f01c6fea0513", 2021-02-10T11:04:31.499 INFO:teuthology.orchestra.run.gibba013.stdout: "whoami": 0, 2021-02-10T11:04:31.500 INFO:teuthology.orchestra.run.gibba013.stdout: "id": 4226, 2021-02-10T11:04:31.500 INFO:teuthology.orchestra.run.gibba013.stdout: "want_state": "up:active", 2021-02-10T11:04:31.500 INFO:teuthology.orchestra.run.gibba013.stdout: "state": "up:active", 2021-02-10T11:04:31.500 INFO:teuthology.orchestra.run.gibba013.stdout: "rank_uptime": 39.709469112000001, 2021-02-10T11:04:31.500 INFO:teuthology.orchestra.run.gibba013.stdout: "mdsmap_epoch": 6, 2021-02-10T11:04:31.500 INFO:teuthology.orchestra.run.gibba013.stdout: "osdmap_epoch": 20, 2021-02-10T11:04:31.500 INFO:teuthology.orchestra.run.gibba013.stdout: "osdmap_epoch_barrier": 20, 2021-02-10T11:04:31.500 INFO:teuthology.orchestra.run.gibba013.stdout: "uptime": 44.103904086 2021-02-10T11:04:31.500 INFO:teuthology.orchestra.run.gibba013.stdout:} 2021-02-10T11:04:31.503 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "68db30ea-dbce-4aef-a6df-f01c6fea0513", "whoami": 0, "id": 4226, "want_state": "up:active", "state": "up:active", "rank_uptime": 39.709469112000001, "mdsmap_epoch": 6, "osdmap_epoch": 20, "osdmap_epoch_barrier": 20, "uptime": 44.103904086 } 2021-02-10T11:04:31.503 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2021-02-10T11:04:31.503 INFO:tasks.cephfs.fuse_mount:Client client.3 config is {} 2021-02-10T11:04:31.503 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.3 at ubuntu@gibba013.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.3... 2021-02-10T11:04:31.503 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:04:31.504 DEBUG:teuthology.orchestra.run.gibba013:> (cd /home/ubuntu/cephtest && exec mkdir -p /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:04:31.838 DEBUG:teuthology.orchestra.run.gibba013:> sudo modprobe fuse 2021-02-10T11:04:32.184 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:04:32.184 DEBUG:teuthology.orchestra.run.gibba013:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:04:32.527 INFO:teuthology.orchestra.run.gibba013.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:04:32.527 INFO:teuthology.orchestra.run.gibba013.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:04:32.528 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:04:32.528 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:04:32.528 DEBUG:teuthology.orchestra.run.gibba013:> ls /sys/fs/fuse/connections 2021-02-10T11:04:32.862 INFO:teuthology.orchestra.run.gibba013.stdout:43 2021-02-10T11:04:32.863 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [43] 2021-02-10T11:04:32.863 DEBUG:teuthology.orchestra.run.gibba013:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.3 /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:04:32.866 DEBUG:teuthology.orchestra.run.gibba013:> sudo modprobe fuse 2021-02-10T11:04:33.210 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:04:33.210 DEBUG:teuthology.orchestra.run.gibba013:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:04:33.276 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.gibba013.stderr:ceph-fuse[37660]: starting ceph client 2021-02-10T11:04:33.276 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.gibba013.stderr:2021-02-10 11:04:33.275 7f791b1f5f80 -1 init, newargv = 0x55edb19b4100 newargc=9 2021-02-10T11:04:33.282 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.gibba013.stderr:ceph-fuse[37660]: starting fuse 2021-02-10T11:04:33.553 INFO:teuthology.orchestra.run.gibba013.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:04:33.553 INFO:teuthology.orchestra.run.gibba013.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:04:33.553 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:04:33.554 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:04:33.554 DEBUG:teuthology.orchestra.run.gibba013:> ls /sys/fs/fuse/connections 2021-02-10T11:04:33.887 INFO:teuthology.orchestra.run.gibba013.stdout:43 2021-02-10T11:04:33.887 INFO:teuthology.orchestra.run.gibba013.stdout:44 2021-02-10T11:04:33.887 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43, 44] 2021-02-10T11:04:33.887 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:04:33.888 DEBUG:teuthology.orchestra.run.gibba013:> sudo python3 -c ' 2021-02-10T11:04:33.888 DEBUG:teuthology.orchestra.run.gibba013:> import glob 2021-02-10T11:04:33.888 DEBUG:teuthology.orchestra.run.gibba013:> import re 2021-02-10T11:04:33.888 DEBUG:teuthology.orchestra.run.gibba013:> import os 2021-02-10T11:04:33.888 DEBUG:teuthology.orchestra.run.gibba013:> import subprocess 2021-02-10T11:04:33.888 DEBUG:teuthology.orchestra.run.gibba013:> 2021-02-10T11:04:33.888 DEBUG:teuthology.orchestra.run.gibba013:> def find_socket(client_name): 2021-02-10T11:04:33.888 DEBUG:teuthology.orchestra.run.gibba013:> asok_path = "/var/run/ceph/ceph-client.3.*.asok" 2021-02-10T11:04:33.888 DEBUG:teuthology.orchestra.run.gibba013:> files = glob.glob(asok_path) 2021-02-10T11:04:33.888 DEBUG:teuthology.orchestra.run.gibba013:> 2021-02-10T11:04:33.888 DEBUG:teuthology.orchestra.run.gibba013:> # Given a non-glob path, it better be there 2021-02-10T11:04:33.888 DEBUG:teuthology.orchestra.run.gibba013:> if "*" not in asok_path: 2021-02-10T11:04:33.888 DEBUG:teuthology.orchestra.run.gibba013:> assert(len(files) == 1) 2021-02-10T11:04:33.888 DEBUG:teuthology.orchestra.run.gibba013:> return files[0] 2021-02-10T11:04:33.889 DEBUG:teuthology.orchestra.run.gibba013:> 2021-02-10T11:04:33.889 DEBUG:teuthology.orchestra.run.gibba013:> for f in files: 2021-02-10T11:04:33.889 DEBUG:teuthology.orchestra.run.gibba013:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2021-02-10T11:04:33.889 DEBUG:teuthology.orchestra.run.gibba013:> if os.path.exists("/proc/{0}".format(pid)): 2021-02-10T11:04:33.889 DEBUG:teuthology.orchestra.run.gibba013:> return f 2021-02-10T11:04:33.889 DEBUG:teuthology.orchestra.run.gibba013:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2021-02-10T11:04:33.889 DEBUG:teuthology.orchestra.run.gibba013:> 2021-02-10T11:04:33.889 DEBUG:teuthology.orchestra.run.gibba013:> print(find_socket("client.3")) 2021-02-10T11:04:33.889 DEBUG:teuthology.orchestra.run.gibba013:> ' 2021-02-10T11:04:34.251 INFO:teuthology.orchestra.run.gibba013.stdout:/var/run/ceph/ceph-client.3.37660.asok 2021-02-10T11:04:34.253 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.3.37660.asok 2021-02-10T11:04:34.253 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:04:34.254 DEBUG:teuthology.orchestra.run.gibba013:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.3.37660.asok status 2021-02-10T11:04:34.643 INFO:teuthology.orchestra.run.gibba013.stdout:{ 2021-02-10T11:04:34.643 INFO:teuthology.orchestra.run.gibba013.stdout: "metadata": { 2021-02-10T11:04:34.643 INFO:teuthology.orchestra.run.gibba013.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:04:34.643 INFO:teuthology.orchestra.run.gibba013.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:04:34.643 INFO:teuthology.orchestra.run.gibba013.stdout: "entity_id": "3", 2021-02-10T11:04:34.643 INFO:teuthology.orchestra.run.gibba013.stdout: "hostname": "gibba013", 2021-02-10T11:04:34.643 INFO:teuthology.orchestra.run.gibba013.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.3", 2021-02-10T11:04:34.643 INFO:teuthology.orchestra.run.gibba013.stdout: "pid": "37660", 2021-02-10T11:04:34.643 INFO:teuthology.orchestra.run.gibba013.stdout: "root": "/" 2021-02-10T11:04:34.643 INFO:teuthology.orchestra.run.gibba013.stdout: }, 2021-02-10T11:04:34.643 INFO:teuthology.orchestra.run.gibba013.stdout: "dentry_count": 0, 2021-02-10T11:04:34.644 INFO:teuthology.orchestra.run.gibba013.stdout: "dentry_pinned_count": 0, 2021-02-10T11:04:34.644 INFO:teuthology.orchestra.run.gibba013.stdout: "id": 4337, 2021-02-10T11:04:34.644 INFO:teuthology.orchestra.run.gibba013.stdout: "inst": { 2021-02-10T11:04:34.644 INFO:teuthology.orchestra.run.gibba013.stdout: "name": { 2021-02-10T11:04:34.644 INFO:teuthology.orchestra.run.gibba013.stdout: "type": "client", 2021-02-10T11:04:34.644 INFO:teuthology.orchestra.run.gibba013.stdout: "num": 4337 2021-02-10T11:04:34.644 INFO:teuthology.orchestra.run.gibba013.stdout: }, 2021-02-10T11:04:34.644 INFO:teuthology.orchestra.run.gibba013.stdout: "addr": { 2021-02-10T11:04:34.644 INFO:teuthology.orchestra.run.gibba013.stdout: "type": "v1", 2021-02-10T11:04:34.644 INFO:teuthology.orchestra.run.gibba013.stdout: "addr": "172.21.2.113:0", 2021-02-10T11:04:34.644 INFO:teuthology.orchestra.run.gibba013.stdout: "nonce": 399516333 2021-02-10T11:04:34.644 INFO:teuthology.orchestra.run.gibba013.stdout: } 2021-02-10T11:04:34.644 INFO:teuthology.orchestra.run.gibba013.stdout: }, 2021-02-10T11:04:34.644 INFO:teuthology.orchestra.run.gibba013.stdout: "addr": { 2021-02-10T11:04:34.645 INFO:teuthology.orchestra.run.gibba013.stdout: "type": "v1", 2021-02-10T11:04:34.645 INFO:teuthology.orchestra.run.gibba013.stdout: "addr": "172.21.2.113:0", 2021-02-10T11:04:34.645 INFO:teuthology.orchestra.run.gibba013.stdout: "nonce": 399516333 2021-02-10T11:04:34.645 INFO:teuthology.orchestra.run.gibba013.stdout: }, 2021-02-10T11:04:34.645 INFO:teuthology.orchestra.run.gibba013.stdout: "inst_str": "client.4337 172.21.2.113:0/399516333", 2021-02-10T11:04:34.645 INFO:teuthology.orchestra.run.gibba013.stdout: "addr_str": "172.21.2.113:0/399516333", 2021-02-10T11:04:34.645 INFO:teuthology.orchestra.run.gibba013.stdout: "inode_count": 1, 2021-02-10T11:04:34.645 INFO:teuthology.orchestra.run.gibba013.stdout: "mds_epoch": 6, 2021-02-10T11:04:34.645 INFO:teuthology.orchestra.run.gibba013.stdout: "osd_epoch": 20, 2021-02-10T11:04:34.645 INFO:teuthology.orchestra.run.gibba013.stdout: "osd_epoch_barrier": 0, 2021-02-10T11:04:34.645 INFO:teuthology.orchestra.run.gibba013.stdout: "blacklisted": false 2021-02-10T11:04:34.645 INFO:teuthology.orchestra.run.gibba013.stdout:} 2021-02-10T11:04:34.647 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:04:34.647 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:04:34.647 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:04:34.983 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:04:34.983 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:04:34.983 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:04:34.983 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:04:35.329 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:04:35.329 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:04:35.329 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:04:35.665 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:04:35.665 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2021-02-10T11:04:35.666 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:04:35.666 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:04:36.016 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:04:36.016 DEBUG:teuthology.orchestra.run.gibba013:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:04:36.016 DEBUG:teuthology.orchestra.run.gibba013:> ' -- /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:04:36.352 INFO:teuthology.orchestra.run.gibba013.stdout:fuseblk 2021-02-10T11:04:36.352 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.2 2021-02-10T11:04:36.352 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:04:36.353 DEBUG:teuthology.orchestra.run.gibba013:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:04:36.702 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:04:36.703 DEBUG:teuthology.orchestra.run.gibba013:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:04:36.703 DEBUG:teuthology.orchestra.run.gibba013:> ' -- /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:04:37.036 INFO:teuthology.orchestra.run.gibba013.stdout:fuseblk 2021-02-10T11:04:37.036 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.3 2021-02-10T11:04:37.036 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:04:37.036 DEBUG:teuthology.orchestra.run.gibba013:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:04:37.383 INFO:teuthology.run_tasks:Running task cephfs_test_runner... 2021-02-10T11:04:37.427 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:04:37.855 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:37.839 7f352669f700 1 Processor -- start 2021-02-10T11:04:37.855 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:37.839 7f352669f700 1 -- start start 2021-02-10T11:04:37.855 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:37.840 7f352669f700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f35200e5470 0x7f352007e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:37.855 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:37.840 7f352669f700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f352007ed80 con 0x7f352007f300 2021-02-10T11:04:37.855 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:37.840 7f352669f700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35201b4010 con 0x7f35200e5070 2021-02-10T11:04:37.855 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:37.840 7f352669f700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35200d1790 con 0x7f35200e5470 2021-02-10T11:04:37.855 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:37.840 7f3524cbc700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f352007f300 0x7f352007f700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:42202/0 (socket says 172.21.2.102:42202) 2021-02-10T11:04:37.855 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:37.840 7f3524cbc700 1 -- 172.21.2.102:0/1735190430 learned_addr learned my addr 172.21.2.102:0/1735190430 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:04:37.856 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:37.840 7f351effd700 1 -- 172.21.2.102:0/1735190430 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 849999826 0 0) 0x7f3518001500 con 0x7f352007f300 2021-02-10T11:04:37.856 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:37.840 7f351effd700 1 -- 172.21.2.102:0/1735190430 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3508002fe0 con 0x7f352007f300 2021-02-10T11:04:37.856 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:37.840 7f351f7fe700 1 --2- 172.21.2.102:0/1735190430 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f35200e5470 0x7f352007e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:37.856 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:37.840 7f351effd700 1 -- 172.21.2.102:0/1735190430 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 582821868 0 0) 0x7f3518001500 con 0x7f352007f300 2021-02-10T11:04:37.856 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:37.840 7f351effd700 1 -- 172.21.2.102:0/1735190430 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f35200e5470 msgr2=0x7f352007e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:04:37.856 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:37.841 7f351effd700 1 --2- 172.21.2.102:0/1735190430 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f35200e5470 0x7f352007e6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:37.856 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:37.841 7f351effd700 1 -- 172.21.2.102:0/1735190430 >> v1:172.21.2.113:6789/0 conn(0x7f35200e5070 legacy=0x7f35201b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:04:37.856 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:37.841 7f351effd700 1 -- 172.21.2.102:0/1735190430 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35200d1980 con 0x7f352007f300 2021-02-10T11:04:37.856 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:37.841 7f351effd700 1 -- 172.21.2.102:0/1735190430 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7f3518001a00 con 0x7f352007f300 2021-02-10T11:04:37.856 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:37.841 7f351effd700 1 -- 172.21.2.102:0/1735190430 >> v1:172.21.2.102:6789/0 conn(0x7f352007f300 legacy=0x7f352007f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:37.856 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:37.841 7f351effd700 1 --2- 172.21.2.102:0/1735190430 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3508003b90 0x7f3508006000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:37.856 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:37.841 7f351effd700 1 --2- 172.21.2.102:0/1735190430 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3508006820 0x7f3508008d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:37.856 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:37.841 7f351effd700 1 --2- 172.21.2.102:0/1735190430 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f35080095e0 0x7f350800bac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:37.857 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:37.841 7f351effd700 1 -- 172.21.2.102:0/1735190430 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f350800c370 con 0x7f3508006820 2021-02-10T11:04:37.857 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:37.841 7f351effd700 1 -- 172.21.2.102:0/1735190430 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f350800c590 con 0x7f35080095e0 2021-02-10T11:04:37.857 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:37.841 7f351effd700 1 -- 172.21.2.102:0/1735190430 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f350800c7b0 con 0x7f3508003b90 2021-02-10T11:04:37.857 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:37.841 7f351f7fe700 1 --2- 172.21.2.102:0/1735190430 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3508006820 0x7f3508008d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:37.857 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:37.841 7f351ffff700 1 --2- 172.21.2.102:0/1735190430 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3508003b90 0x7f3508006000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:37.857 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:37.841 7f3524cbc700 1 --2- 172.21.2.102:0/1735190430 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f35080095e0 0x7f350800bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:37.857 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:37.841 7f351f7fe700 1 -- 172.21.2.102:0/1735190430 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3508003b90 msgr2=0x7f3508006000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:04:37.857 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:37.841 7f351f7fe700 1 --2- 172.21.2.102:0/1735190430 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3508003b90 0x7f3508006000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:37.857 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:37.841 7f351f7fe700 1 -- 172.21.2.102:0/1735190430 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f35080095e0 msgr2=0x7f350800bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:04:37.857 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:37.841 7f351f7fe700 1 --2- 172.21.2.102:0/1735190430 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f35080095e0 0x7f350800bac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:37.857 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:37.841 7f351f7fe700 1 -- 172.21.2.102:0/1735190430 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f350800ca50 con 0x7f3508006820 2021-02-10T11:04:37.857 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:37.841 7f351f7fe700 1 --2- 172.21.2.102:0/1735190430 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3508006820 0x7f3508008d40 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f351404aaa0 tx=0x7f351404da20).ready entity=mon.0 client_cookie=5fa794a4e782cf69 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:04:37.858 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:37.841 7f351effd700 1 -- 172.21.2.102:0/1735190430 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f35140009b0 con 0x7f3508006820 2021-02-10T11:04:37.858 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:37.842 7f351effd700 1 -- 172.21.2.102:0/1735190430 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35140500c0 con 0x7f3508006820 2021-02-10T11:04:37.858 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:37.842 7f351effd700 1 -- 172.21.2.102:0/1735190430 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3514050550 con 0x7f3508006820 2021-02-10T11:04:37.858 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:37.842 7f352669f700 1 -- 172.21.2.102:0/1735190430 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3508006820 msgr2=0x7f3508008d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:37.858 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:37.842 7f352669f700 1 --2- 172.21.2.102:0/1735190430 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3508006820 0x7f3508008d40 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f351404aaa0 tx=0x7f351404da20).stop 2021-02-10T11:04:37.858 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:37.842 7f3524cbc700 1 -- 172.21.2.102:0/1735190430 reap_dead start 2021-02-10T11:04:37.858 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:37.842 7f352669f700 1 -- 172.21.2.102:0/1735190430 shutdown_connections 2021-02-10T11:04:37.858 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:37.842 7f352669f700 1 -- 172.21.2.102:0/1735190430 >> 172.21.2.102:0/1735190430 conn(0x7f35200ce690 msgr2=0x7f35200e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:04:37.858 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:37.842 7f352669f700 1 -- 172.21.2.102:0/1735190430 shutdown_connections 2021-02-10T11:04:37.858 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:37.842 7f352669f700 1 -- 172.21.2.102:0/1735190430 wait complete. 2021-02-10T11:04:37.858 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:37.857 7f352669f700 1 Processor -- start 2021-02-10T11:04:37.858 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:37.857 7f352669f700 1 -- start start 2021-02-10T11:04:37.860 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:37.857 7f352669f700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3520194000 0x7f35200b0d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:37.860 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:37.857 7f352669f700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f35200b1330 0x7f3520087050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:37.860 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:37.857 7f352669f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3520087670 0x7f3520087a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:37.860 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:37.857 7f352669f700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f352009c0e0 con 0x7f3520087670 2021-02-10T11:04:37.860 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:37.857 7f352669f700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3520194440 con 0x7f35200b1330 2021-02-10T11:04:37.860 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:37.857 7f352669f700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f352009c5e0 con 0x7f3520194000 2021-02-10T11:04:37.860 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:37.858 7f3524cbc700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3520087670 0x7f3520087a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:37.860 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:37.858 7f3524cbc700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3520087670 0x7f3520087a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:49992/0 (socket says 172.21.2.102:49992) 2021-02-10T11:04:37.860 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:37.858 7f3524cbc700 1 -- 172.21.2.102:0/754655271 learned_addr learned my addr 172.21.2.102:0/754655271 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:04:37.860 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:37.858 7f3524cbc700 1 -- 172.21.2.102:0/754655271 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3520194000 msgr2=0x7f35200b0d10 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:04:37.860 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:37.858 7f3524cbc700 1 --2- 172.21.2.102:0/754655271 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3520194000 0x7f35200b0d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:37.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:37.858 7f3524cbc700 1 -- 172.21.2.102:0/754655271 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f35200b1330 msgr2=0x7f3520087050 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:04:37.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:37.858 7f3524cbc700 1 --2- 172.21.2.102:0/754655271 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f35200b1330 0x7f3520087050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:37.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:37.858 7f3524cbc700 1 -- 172.21.2.102:0/754655271 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f352009c7d0 con 0x7f3520087670 2021-02-10T11:04:37.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:37.858 7f3524cbc700 1 --2- 172.21.2.102:0/754655271 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3520087670 0x7f3520087a70 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7f351800b990 tx=0x7f3518009cc0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:04:37.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:37.858 7f351cff9700 1 -- 172.21.2.102:0/754655271 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f35180079d0 con 0x7f3520087670 2021-02-10T11:04:37.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:37.858 7f351cff9700 1 -- 172.21.2.102:0/754655271 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35180070c0 con 0x7f3520087670 2021-02-10T11:04:37.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:37.858 7f351cff9700 1 -- 172.21.2.102:0/754655271 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3518004720 con 0x7f3520087670 2021-02-10T11:04:37.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:37.858 7f352669f700 1 -- 172.21.2.102:0/754655271 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f352009c7d0 con 0x7f3520087670 2021-02-10T11:04:37.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:37.858 7f352669f700 1 -- 172.21.2.102:0/754655271 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f352009cd60 con 0x7f3520087670 2021-02-10T11:04:37.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:37.859 7f351cff9700 1 -- 172.21.2.102:0/754655271 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f35180072a0 con 0x7f3520087670 2021-02-10T11:04:37.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:37.859 7f351cff9700 1 --2- 172.21.2.102:0/754655271 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f35000186d0 0x7f350001abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:37.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:37.859 7f351cff9700 1 -- 172.21.2.102:0/754655271 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f35180329b0 con 0x7f3520087670 2021-02-10T11:04:37.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:37.859 7f351ffff700 1 --2- 172.21.2.102:0/754655271 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f35000186d0 0x7f350001abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:37.862 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:37.859 7f351ffff700 1 --2- 172.21.2.102:0/754655271 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f35000186d0 0x7f350001abd0 crc :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:04:37.887 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:37.890 7f352669f700 1 -- 172.21.2.102:0/754655271 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3520084b20 con 0x7f3520087670 2021-02-10T11:04:37.890 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:37.892 7f351cff9700 1 -- 172.21.2.102:0/754655271 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f35180079d0 con 0x7f3520087670 2021-02-10T11:04:38.015 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:38.017 7f352669f700 1 -- 172.21.2.102:0/754655271 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f352007ed80 con 0x7f3520087670 2021-02-10T11:04:38.015 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:38.018 7f351cff9700 1 -- 172.21.2.102:0/754655271 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+13993 (secure 0 0 0) 0x7f3518006fa0 con 0x7f3520087670 2021-02-10T11:04:38.016 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:04:38.016 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":20,"fsid":"68db30ea-dbce-4aef-a6df-f01c6fea0513","created":"2021-02-10 11:03:06.331557","modified":"2021-02-10 11:03:44.386945","last_up_change":"2021-02-10 11:03:32.690562","last_in_change":"2021-02-10 11:03:32.690562","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:03:37.322519","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:03:41.161563","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2021-02-10 11:03:42.379651","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"be691b4d-a039-408a-af80-9d53f02b6f1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6818","nonce":36088},{"type":"v1","addr":"172.21.2.102:6820","nonce":36088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6822","nonce":36088},{"type":"v1","addr":"172.21.2.102:6824","nonce":36088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6830","nonce":36088},{"type":"v1","addr":"172.21.2.102:6832","nonce":36088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6826","nonce":36088},{"type":"v1","addr":"172.21.2.102:6828","nonce":36088}]},"public_addr":"172.21.2.102:6820/36088","cluster_addr":"172.21.2.102:6824/36088","heartbeat_back_addr":"172.21.2.102:6832/36088","heartbeat_front_addr":"172.21.2.102:6828/36088","state":["exists","up"]},{"osd":1,"uuid":"64e3ccc8-9c69-4d7c-865e-af8e83822635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6819","nonce":36087},{"type":"v1","addr":"172.21.2.102:6821","nonce":36087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6823","nonce":36087},{"type":"v1","addr":"172.21.2.102:6825","nonce":36087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6831","nonce":36087},{"type":"v1","addr":"172.21.2.102:6833","nonce":36087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6827","nonce":36087},{"type":"v1","addr":"172.21.2.102:6829","nonce":36087}]},"public_addr":"172.21.2.102:6821/36087","cluster_addr":"172.21.2.102:6825/36087","heartbeat_back_addr":"172.21.2.102:6833/36087","heartbeat_front_addr":"172.21.2.102:6829/36087","state":["exists","up"]},{"osd":2,"uuid":"153700af-747f-481f-bdf4-d6072d3a842f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6810","nonce":36089},{"type":"v1","addr":"172.21.2.102:6811","nonce":36089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6812","nonce":36089},{"type":"v1","addr":"172.21.2.102:6813","nonce":36089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6816","nonce":36089},{"type":"v1","addr":"172.21.2.102:6817","nonce":36089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6814","nonce":36089},{"type":"v1","addr":"172.21.2.102:6815","nonce":36089}]},"public_addr":"172.21.2.102:6811/36089","cluster_addr":"172.21.2.102:6813/36089","heartbeat_back_addr":"172.21.2.102:6817/36089","heartbeat_front_addr":"172.21.2.102:6815/36089","state":["exists","up"]},{"osd":3,"uuid":"cbc678f4-83a9-45e6-8830-2f7c232b1ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6802","nonce":36090},{"type":"v1","addr":"172.21.2.102:6803","nonce":36090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6804","nonce":36090},{"type":"v1","addr":"172.21.2.102:6805","nonce":36090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6808","nonce":36090},{"type":"v1","addr":"172.21.2.102:6809","nonce":36090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6806","nonce":36090},{"type":"v1","addr":"172.21.2.102:6807","nonce":36090}]},"public_addr":"172.21.2.102:6803/36090","cluster_addr":"172.21.2.102:6805/36090","heartbeat_back_addr":"172.21.2.102:6809/36090","heartbeat_front_addr":"172.21.2.102:6807/36090","state":["exists","up"]},{"osd":4,"uuid":"c01e4008-b258-4f10-afd2-9e7f37214351","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6808","nonce":36368},{"type":"v1","addr":"172.21.2.113:6811","nonce":36368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6814","nonce":36368},{"type":"v1","addr":"172.21.2.113:6818","nonce":36368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6826","nonce":36368},{"type":"v1","addr":"172.21.2.113:6828","nonce":36368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6820","nonce":36368},{"type":"v1","addr":"172.21.2.113:6824","nonce":36368}]},"public_addr":"172.21.2.113:6811/36368","cluster_addr":"172.21.2.113:6818/36368","heartbeat_back_addr":"172.21.2.113:6828/36368","heartbeat_front_addr":"172.21.2.113:6824/36368","state":["exists","up"]},{"osd":5,"uuid":"551f6007-daf8-4a98-8ee0-c2b16bcccb5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6802","nonce":36366},{"type":"v1","addr":"172.21.2.113:6805","nonce":36366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6807","nonce":36366},{"type":"v1","addr":"172.21.2.113:6810","nonce":36366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6819","nonce":36366},{"type":"v1","addr":"172.21.2.113:6822","nonce":36366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6812","nonce":36366},{"type":"v1","addr":"172.21.2.113:6815","nonce":36366}]},"public_addr":"172.21.2.113:6805/36366","cluster_addr":"172.21.2.113:6810/36366","heartbeat_back_addr":"172.21.2.113:6822/36366","heartbeat_front_addr":"172.21.2.113:6815/36366","state":["exists","up"]},{"osd":6,"uuid":"bf67c040-0e50-48be-b102-c3943bc9ebef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6800","nonce":36367},{"type":"v1","addr":"172.21.2.113:6801","nonce":36367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6803","nonce":36367},{"type":"v1","addr":"172.21.2.113:6804","nonce":36367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6813","nonce":36367},{"type":"v1","addr":"172.21.2.113:6816","nonce":36367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6806","nonce":36367},{"type":"v1","addr":"172.21.2.113:6809","nonce":36367}]},"public_addr":"172.21.2.113:6801/36367","cluster_addr":"172.21.2.113:6804/36367","heartbeat_back_addr":"172.21.2.113:6816/36367","heartbeat_front_addr":"172.21.2.113:6809/36367","state":["exists","up"]},{"osd":7,"uuid":"f5461f80-018d-428f-be89-2eb2bcbe7bb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6817","nonce":36365},{"type":"v1","addr":"172.21.2.113:6821","nonce":36365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6823","nonce":36365},{"type":"v1","addr":"172.21.2.113:6825","nonce":36365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6830","nonce":36365},{"type":"v1","addr":"172.21.2.113:6831","nonce":36365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6827","nonce":36365},{"type":"v1","addr":"172.21.2.113:6829","nonce":36365}]},"public_addr":"172.21.2.113:6821/36365","cluster_addr":"172.21.2.113:6825/36365","heartbeat_back_addr":"172.21.2.113:6831/36365","heartbeat_front_addr":"172.21.2.113:6829/36365","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:04:38.017 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:38.019 7f352669f700 1 -- 172.21.2.102:0/754655271 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f35000186d0 msgr2=0x7f350001abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:38.017 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:38.019 7f352669f700 1 --2- 172.21.2.102:0/754655271 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f35000186d0 0x7f350001abd0 crc :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:38.017 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:38.019 7f352669f700 1 -- 172.21.2.102:0/754655271 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3520087670 msgr2=0x7f3520087a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:38.017 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:38.019 7f352669f700 1 --2- 172.21.2.102:0/754655271 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3520087670 0x7f3520087a70 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7f351800b990 tx=0x7f3518009cc0).stop 2021-02-10T11:04:38.017 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:38.019 7f352669f700 1 -- 172.21.2.102:0/754655271 shutdown_connections 2021-02-10T11:04:38.018 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:38.019 7f352669f700 1 --2- 172.21.2.102:0/754655271 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3520087670 0x7f3520087a70 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:38.018 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:38.019 7f352669f700 1 --2- 172.21.2.102:0/754655271 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f35000186d0 0x7f350001abd0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:38.018 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:38.019 7f352669f700 1 --2- 172.21.2.102:0/754655271 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f35200b1330 0x7f3520087050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:38.018 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:38.019 7f352669f700 1 --2- 172.21.2.102:0/754655271 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3520194000 0x7f35200b0d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:38.018 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:38.019 7f352669f700 1 -- 172.21.2.102:0/754655271 >> 172.21.2.102:0/754655271 conn(0x7f3520092c20 msgr2=0x7f3520195070 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:04:38.018 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:38.020 7f352669f700 1 -- 172.21.2.102:0/754655271 shutdown_connections 2021-02-10T11:04:38.018 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:38.020 7f352669f700 1 -- 172.21.2.102:0/754655271 wait complete. 2021-02-10T11:04:38.026 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:03:37.322519', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2021-02-10 11:03:41.161563', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2021-02-10 11:03:42.379651', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-02-10T11:04:38.026 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:04:38.454 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:38.440 7ff04f0ad700 1 Processor -- start 2021-02-10T11:04:38.454 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:38.440 7ff04f0ad700 1 -- start start 2021-02-10T11:04:38.454 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:38.440 7ff04f0ad700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7ff0481b0470 0x7ff0481b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:38.454 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:38.440 7ff04f0ad700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7ff0480f20e0 0x7ff0480f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:38.454 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:38.440 7ff04f0ad700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0480f0b50 con 0x7ff0480ef610 2021-02-10T11:04:38.454 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:38.440 7ff04f0ad700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0480f0820 con 0x7ff0481b0470 2021-02-10T11:04:38.455 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:38.440 7ff04f0ad700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0480f24e0 con 0x7ff0480f20e0 2021-02-10T11:04:38.455 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:38.441 7ff047fff700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7ff0480ef610 0x7ff0480efa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:42220/0 (socket says 172.21.2.102:42220) 2021-02-10T11:04:38.455 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:38.441 7ff047fff700 1 -- 172.21.2.102:0/608606569 learned_addr learned my addr 172.21.2.102:0/608606569 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:04:38.455 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:38.441 7ff04cec9700 1 --2- 172.21.2.102:0/608606569 >> v2:172.21.2.102:3300/0 conn(0x7ff0481b0470 0x7ff0481b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:38.455 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:38.441 7ff04d6ca700 1 --2- 172.21.2.102:0/608606569 >> v2:172.21.2.113:3300/0 conn(0x7ff0480f20e0 0x7ff0480f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:38.455 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:38.441 7ff04cec9700 1 -- 172.21.2.102:0/608606569 >> v2:172.21.2.113:3300/0 conn(0x7ff0480f20e0 msgr2=0x7ff0480f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:04:38.455 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:38.441 7ff04cec9700 1 --2- 172.21.2.102:0/608606569 >> v2:172.21.2.113:3300/0 conn(0x7ff0480f20e0 0x7ff0480f0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:38.455 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:38.441 7ff04cec9700 1 -- 172.21.2.102:0/608606569 >> v1:172.21.2.102:6789/0 conn(0x7ff0480ef610 legacy=0x7ff0480efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:38.455 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:38.441 7ff04cec9700 1 -- 172.21.2.102:0/608606569 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0480e52e0 con 0x7ff0481b0470 2021-02-10T11:04:38.455 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:38.441 7ff04cec9700 1 --2- 172.21.2.102:0/608606569 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff0481b0470 0x7ff0481b0870 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7ff030006650 tx=0x7ff030009a80).ready entity=mon.0 client_cookie=c6024038638693c8 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:04:38.455 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:38.442 7ff0477fe700 1 -- 172.21.2.102:0/608606569 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff0300009b0 con 0x7ff0481b0470 2021-02-10T11:04:38.455 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:38.442 7ff0477fe700 1 -- 172.21.2.102:0/608606569 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff03000c0c0 con 0x7ff0481b0470 2021-02-10T11:04:38.456 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:38.442 7ff0477fe700 1 -- 172.21.2.102:0/608606569 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff03000c520 con 0x7ff0481b0470 2021-02-10T11:04:38.456 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:38.442 7ff04f0ad700 1 -- 172.21.2.102:0/608606569 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff0481b0470 msgr2=0x7ff0481b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:38.456 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:38.442 7ff04f0ad700 1 --2- 172.21.2.102:0/608606569 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff0481b0470 0x7ff0481b0870 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7ff030006650 tx=0x7ff030009a80).stop 2021-02-10T11:04:38.456 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:38.442 7ff04f0ad700 1 -- 172.21.2.102:0/608606569 shutdown_connections 2021-02-10T11:04:38.456 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:38.442 7ff04f0ad700 1 --2- 172.21.2.102:0/608606569 >> v2:172.21.2.113:3300/0 conn(0x7ff0480f20e0 0x7ff0480f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:38.456 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:38.442 7ff04f0ad700 1 --2- 172.21.2.102:0/608606569 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff0481b0470 0x7ff0481b0870 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:38.456 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:38.442 7ff04f0ad700 1 -- 172.21.2.102:0/608606569 >> 172.21.2.102:0/608606569 conn(0x7ff0480ce3e0 msgr2=0x7ff0480e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:04:38.456 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:38.442 7ff04f0ad700 1 -- 172.21.2.102:0/608606569 shutdown_connections 2021-02-10T11:04:38.456 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:38.442 7ff04f0ad700 1 -- 172.21.2.102:0/608606569 wait complete. 2021-02-10T11:04:38.458 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:38.457 7ff04f0ad700 1 Processor -- start 2021-02-10T11:04:38.458 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:38.457 7ff04f0ad700 1 -- start start 2021-02-10T11:04:38.458 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:38.457 7ff04f0ad700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff04808ece0 0x7ff0480a6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:38.458 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:38.457 7ff04f0ad700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff0480a6860 0x7ff0480a6c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:38.458 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:38.457 7ff04f0ad700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff0480a7280 0x7ff0480a4590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:38.458 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:38.457 7ff04f0ad700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0480a5cc0 con 0x7ff0480a6860 2021-02-10T11:04:38.458 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:38.457 7ff04f0ad700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff04808f0e0 con 0x7ff0480a7280 2021-02-10T11:04:38.458 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:38.457 7ff04f0ad700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff04808f2c0 con 0x7ff04808ece0 2021-02-10T11:04:38.458 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:38.457 7ff047fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff0480a6860 0x7ff0480a6c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:38.458 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:38.457 7ff047fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff0480a6860 0x7ff0480a6c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:50004/0 (socket says 172.21.2.102:50004) 2021-02-10T11:04:38.459 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:38.457 7ff047fff700 1 -- 172.21.2.102:0/1796132846 learned_addr learned my addr 172.21.2.102:0/1796132846 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:04:38.459 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:38.458 7ff047fff700 1 -- 172.21.2.102:0/1796132846 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff04808ece0 msgr2=0x7ff0480a6240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:38.459 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:38.458 7ff047fff700 1 --2- 172.21.2.102:0/1796132846 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff04808ece0 0x7ff0480a6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:38.459 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:38.458 7ff047fff700 1 -- 172.21.2.102:0/1796132846 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff0480a7280 msgr2=0x7ff0480a4590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:38.459 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:38.458 7ff047fff700 1 --2- 172.21.2.102:0/1796132846 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff0480a7280 0x7ff0480a4590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:38.459 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:38.458 7ff047fff700 1 -- 172.21.2.102:0/1796132846 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0480a4bb0 con 0x7ff0480a6860 2021-02-10T11:04:38.459 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:38.458 7ff047fff700 1 --2- 172.21.2.102:0/1796132846 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff0480a6860 0x7ff0480a6c60 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7ff03800a940 tx=0x7ff03800ace0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:04:38.459 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:38.458 7ff0457fa700 1 -- 172.21.2.102:0/1796132846 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff038007020 con 0x7ff0480a6860 2021-02-10T11:04:38.459 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:38.458 7ff0457fa700 1 -- 172.21.2.102:0/1796132846 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff038007220 con 0x7ff0480a6860 2021-02-10T11:04:38.459 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:38.458 7ff0457fa700 1 -- 172.21.2.102:0/1796132846 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff038004720 con 0x7ff0480a6860 2021-02-10T11:04:38.459 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:38.458 7ff04f0ad700 1 -- 172.21.2.102:0/1796132846 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0480a4bb0 con 0x7ff0480a6860 2021-02-10T11:04:38.459 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:38.458 7ff04f0ad700 1 -- 172.21.2.102:0/1796132846 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0480a5140 con 0x7ff0480a6860 2021-02-10T11:04:38.459 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:38.459 7ff0457fa700 1 -- 172.21.2.102:0/1796132846 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ff038007400 con 0x7ff0480a6860 2021-02-10T11:04:38.460 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:38.459 7ff0457fa700 1 --2- 172.21.2.102:0/1796132846 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff02c0186c0 0x7ff02c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:38.460 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:38.459 7ff0457fa700 1 -- 172.21.2.102:0/1796132846 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5573+0+0 (secure 0 0 0) 0x7ff0380328f0 con 0x7ff0480a6860 2021-02-10T11:04:38.460 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:38.459 7ff04cec9700 1 --2- 172.21.2.102:0/1796132846 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff02c0186c0 0x7ff02c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:38.460 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:38.459 7ff04cec9700 1 --2- 172.21.2.102:0/1796132846 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff02c0186c0 0x7ff02c01abc0 crc :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:04:38.488 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:38.490 7ff04f0ad700 1 -- 172.21.2.102:0/1796132846 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff048105ae0 con 0x7ff0480a6860 2021-02-10T11:04:38.490 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:38.493 7ff0457fa700 1 -- 172.21.2.102:0/1796132846 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ff038001250 con 0x7ff0480a6860 2021-02-10T11:04:38.616 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:38.619 7ff04f0ad700 1 -- 172.21.2.102:0/1796132846 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7ff0480f0b50 con 0x7ff0480a6860 2021-02-10T11:04:38.617 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:38.619 7ff0457fa700 1 -- 172.21.2.102:0/1796132846 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v20) v1 ==== 92+0+10 (secure 0 0 0) 0x7ff0380073e0 con 0x7ff0480a6860 2021-02-10T11:04:38.617 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:04:38.618 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:38.621 7ff04f0ad700 1 -- 172.21.2.102:0/1796132846 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff02c0186c0 msgr2=0x7ff02c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:38.619 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:38.621 7ff04f0ad700 1 --2- 172.21.2.102:0/1796132846 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff02c0186c0 0x7ff02c01abc0 crc :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:38.619 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:38.621 7ff04f0ad700 1 -- 172.21.2.102:0/1796132846 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff0480a6860 msgr2=0x7ff0480a6c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:38.619 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:38.621 7ff04f0ad700 1 --2- 172.21.2.102:0/1796132846 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff0480a6860 0x7ff0480a6c60 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7ff03800a940 tx=0x7ff03800ace0).stop 2021-02-10T11:04:38.619 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:38.621 7ff04f0ad700 1 -- 172.21.2.102:0/1796132846 shutdown_connections 2021-02-10T11:04:38.619 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:38.621 7ff04f0ad700 1 --2- 172.21.2.102:0/1796132846 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff02c0186c0 0x7ff02c01abc0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:38.619 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:38.621 7ff04f0ad700 1 --2- 172.21.2.102:0/1796132846 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff0480a7280 0x7ff0480a4590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:38.619 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:38.621 7ff04f0ad700 1 --2- 172.21.2.102:0/1796132846 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff0480a6860 0x7ff0480a6c60 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:38.620 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:38.621 7ff04f0ad700 1 --2- 172.21.2.102:0/1796132846 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff04808ece0 0x7ff0480a6240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:38.620 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:38.621 7ff04f0ad700 1 -- 172.21.2.102:0/1796132846 >> 172.21.2.102:0/1796132846 conn(0x7ff04808ffe0 msgr2=0x7ff0480bf050 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:04:38.620 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:38.621 7ff04f0ad700 1 -- 172.21.2.102:0/1796132846 shutdown_connections 2021-02-10T11:04:38.620 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:38.621 7ff04f0ad700 1 -- 172.21.2.102:0/1796132846 wait complete. 2021-02-10T11:04:38.627 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-02-10T11:04:39.057 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:39.042 7fe93f34e700 1 Processor -- start 2021-02-10T11:04:39.057 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:39.042 7fe93f34e700 1 -- start start 2021-02-10T11:04:39.057 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:39.043 7fe93f34e700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe9381b0470 0x7fe9381b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:39.057 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:39.043 7fe93f34e700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fe9380f20e0 0x7fe9380f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:39.057 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:39.043 7fe93f34e700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9380f0b50 con 0x7fe9380ef610 2021-02-10T11:04:39.057 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:39.043 7fe93f34e700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9380f0820 con 0x7fe9380f20e0 2021-02-10T11:04:39.058 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:39.043 7fe93f34e700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9380f24e0 con 0x7fe9381b0470 2021-02-10T11:04:39.058 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:39.043 7fe93c969700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fe9380ef610 0x7fe9380efa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:42232/0 (socket says 172.21.2.102:42232) 2021-02-10T11:04:39.058 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:39.043 7fe93c969700 1 -- 172.21.2.102:0/519877969 learned_addr learned my addr 172.21.2.102:0/519877969 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:04:39.058 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:39.043 7fe93d96b700 1 --2- 172.21.2.102:0/519877969 >> v2:172.21.2.102:3300/0 conn(0x7fe9380f20e0 0x7fe9380f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:39.058 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:39.044 7fe93d16a700 1 --2- 172.21.2.102:0/519877969 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe9381b0470 0x7fe9381b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:39.058 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:39.044 7fe93d96b700 1 -- 172.21.2.102:0/519877969 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe9381b0470 msgr2=0x7fe9381b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:04:39.058 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:39.044 7fe92ffff700 1 -- 172.21.2.102:0/519877969 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1524138910 0 0) 0x7fe9280010b0 con 0x7fe9380ef610 2021-02-10T11:04:39.058 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:39.044 7fe93d96b700 1 --2- 172.21.2.102:0/519877969 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe9381b0470 0x7fe9381b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:39.058 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:39.044 7fe93d96b700 1 -- 172.21.2.102:0/519877969 >> v1:172.21.2.102:6789/0 conn(0x7fe9380ef610 legacy=0x7fe9380efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:39.058 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:39.044 7fe93d96b700 1 -- 172.21.2.102:0/519877969 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9380e52e0 con 0x7fe9380f20e0 2021-02-10T11:04:39.058 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:39.044 7fe93d16a700 1 --2- 172.21.2.102:0/519877969 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe9381b0470 0x7fe9381b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:04:39.058 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:39.044 7fe93d96b700 1 --2- 172.21.2.102:0/519877969 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe9380f20e0 0x7fe9380f0200 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7fe930006ab0 tx=0x7fe930009b40).ready entity=mon.0 client_cookie=40f2d0cd070a8b7a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:04:39.059 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:39.044 7fe92ffff700 1 -- 172.21.2.102:0/519877969 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe9300066d0 con 0x7fe9380f20e0 2021-02-10T11:04:39.059 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:39.044 7fe92ffff700 1 -- 172.21.2.102:0/519877969 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9300022e0 con 0x7fe9380f20e0 2021-02-10T11:04:39.059 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:39.044 7fe92ffff700 1 -- 172.21.2.102:0/519877969 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe930002800 con 0x7fe9380f20e0 2021-02-10T11:04:39.059 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:39.044 7fe93f34e700 1 -- 172.21.2.102:0/519877969 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe9380f20e0 msgr2=0x7fe9380f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:39.059 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:39.044 7fe93f34e700 1 --2- 172.21.2.102:0/519877969 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe9380f20e0 0x7fe9380f0200 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7fe930006ab0 tx=0x7fe930009b40).stop 2021-02-10T11:04:39.059 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:39.045 7fe93f34e700 1 -- 172.21.2.102:0/519877969 shutdown_connections 2021-02-10T11:04:39.059 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:39.045 7fe93f34e700 1 --2- 172.21.2.102:0/519877969 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe9380f20e0 0x7fe9380f0200 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:39.059 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:39.045 7fe93f34e700 1 --2- 172.21.2.102:0/519877969 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe9381b0470 0x7fe9381b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:39.059 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:39.045 7fe93f34e700 1 -- 172.21.2.102:0/519877969 >> 172.21.2.102:0/519877969 conn(0x7fe9380ce3e0 msgr2=0x7fe9380e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:04:39.059 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:39.045 7fe93f34e700 1 -- 172.21.2.102:0/519877969 shutdown_connections 2021-02-10T11:04:39.059 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:39.045 7fe93f34e700 1 -- 172.21.2.102:0/519877969 wait complete. 2021-02-10T11:04:39.061 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:39.060 7fe93f34e700 1 Processor -- start 2021-02-10T11:04:39.061 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:39.060 7fe93f34e700 1 -- start start 2021-02-10T11:04:39.061 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:39.060 7fe93f34e700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe9380b7de0 0x7fe938190920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:39.061 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:39.060 7fe93f34e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe9380b82f0 0x7fe9380b86f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:39.061 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:39.060 7fe93f34e700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe9380b8d10 0x7fe9380bf9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:39.061 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:39.060 7fe93f34e700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9380b7930 con 0x7fe9380b82f0 2021-02-10T11:04:39.061 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:39.060 7fe93f34e700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9380b7b20 con 0x7fe9380b7de0 2021-02-10T11:04:39.062 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:39.060 7fe93f34e700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9380bffd0 con 0x7fe9380b8d10 2021-02-10T11:04:39.062 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:39.060 7fe93c969700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe9380b82f0 0x7fe9380b86f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:39.062 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:39.060 7fe93c969700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe9380b82f0 0x7fe9380b86f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:50016/0 (socket says 172.21.2.102:50016) 2021-02-10T11:04:39.062 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:39.060 7fe93c969700 1 -- 172.21.2.102:0/3628968165 learned_addr learned my addr 172.21.2.102:0/3628968165 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:04:39.062 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:39.061 7fe93c969700 1 -- 172.21.2.102:0/3628968165 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe9380b8d10 msgr2=0x7fe9380bf9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:39.062 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:39.061 7fe93c969700 1 --2- 172.21.2.102:0/3628968165 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe9380b8d10 0x7fe9380bf9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:39.062 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:39.061 7fe93c969700 1 -- 172.21.2.102:0/3628968165 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe9380b7de0 msgr2=0x7fe938190920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:39.062 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:39.061 7fe93c969700 1 --2- 172.21.2.102:0/3628968165 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe9380b7de0 0x7fe938190920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:39.062 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:39.061 7fe93c969700 1 -- 172.21.2.102:0/3628968165 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9380c01c0 con 0x7fe9380b82f0 2021-02-10T11:04:39.063 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:39.061 7fe93c969700 1 --2- 172.21.2.102:0/3628968165 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe9380b82f0 0x7fe9380b86f0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 rx=0x7fe928001c10 tx=0x7fe92800a8f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:04:39.063 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:39.061 7fe92dffb700 1 -- 172.21.2.102:0/3628968165 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe9280009b0 con 0x7fe9380b82f0 2021-02-10T11:04:39.063 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:39.061 7fe92dffb700 1 -- 172.21.2.102:0/3628968165 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9280012c0 con 0x7fe9380b82f0 2021-02-10T11:04:39.063 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:39.061 7fe92dffb700 1 -- 172.21.2.102:0/3628968165 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe928004350 con 0x7fe9380b82f0 2021-02-10T11:04:39.063 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:39.061 7fe93f34e700 1 -- 172.21.2.102:0/3628968165 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9380c01c0 con 0x7fe9380b82f0 2021-02-10T11:04:39.063 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:39.061 7fe93f34e700 1 -- 172.21.2.102:0/3628968165 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9380c01c0 con 0x7fe9380b82f0 2021-02-10T11:04:39.063 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:39.062 7fe92dffb700 1 -- 172.21.2.102:0/3628968165 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fe928004140 con 0x7fe9380b82f0 2021-02-10T11:04:39.064 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:39.062 7fe92dffb700 1 --2- 172.21.2.102:0/3628968165 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe9180186c0 0x7fe91801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:39.064 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:39.062 7fe92dffb700 1 -- 172.21.2.102:0/3628968165 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fe9280325b0 con 0x7fe9380b82f0 2021-02-10T11:04:39.064 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:39.062 7fe93d16a700 1 --2- 172.21.2.102:0/3628968165 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe9180186c0 0x7fe91801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:39.064 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:39.062 7fe93d16a700 1 --2- 172.21.2.102:0/3628968165 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe9180186c0 0x7fe91801abc0 crc :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:04:39.091 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:39.093 7fe93f34e700 1 -- 172.21.2.102:0/3628968165 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe938105b30 con 0x7fe9380b82f0 2021-02-10T11:04:39.093 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:39.095 7fe92dffb700 1 -- 172.21.2.102:0/3628968165 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fe928030d60 con 0x7fe9380b82f0 2021-02-10T11:04:39.220 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:39.222 7fe93f34e700 1 -- 172.21.2.102:0/3628968165 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7fe938105b30 con 0x7fe9380b82f0 2021-02-10T11:04:39.220 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:39.222 7fe92dffb700 1 -- 172.21.2.102:0/3628968165 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v20) v1 ==== 104+0+10 (secure 0 0 0) 0x7fe928004140 con 0x7fe9380b82f0 2021-02-10T11:04:39.221 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:04:39.222 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:39.224 7fe93f34e700 1 -- 172.21.2.102:0/3628968165 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe9180186c0 msgr2=0x7fe91801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:39.222 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:39.224 7fe93f34e700 1 --2- 172.21.2.102:0/3628968165 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe9180186c0 0x7fe91801abc0 crc :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:39.222 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:39.224 7fe93f34e700 1 -- 172.21.2.102:0/3628968165 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe9380b82f0 msgr2=0x7fe9380b86f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:39.222 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:39.224 7fe93f34e700 1 --2- 172.21.2.102:0/3628968165 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe9380b82f0 0x7fe9380b86f0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 rx=0x7fe928001c10 tx=0x7fe92800a8f0).stop 2021-02-10T11:04:39.223 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:39.225 7fe93f34e700 1 -- 172.21.2.102:0/3628968165 shutdown_connections 2021-02-10T11:04:39.223 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:39.225 7fe93f34e700 1 --2- 172.21.2.102:0/3628968165 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe9380b8d10 0x7fe9380bf9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:39.223 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:39.225 7fe93f34e700 1 --2- 172.21.2.102:0/3628968165 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe9380b82f0 0x7fe9380b86f0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:39.223 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:39.225 7fe93f34e700 1 --2- 172.21.2.102:0/3628968165 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe9180186c0 0x7fe91801abc0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:39.223 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:39.225 7fe93f34e700 1 --2- 172.21.2.102:0/3628968165 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe9380b7de0 0x7fe938190920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:39.223 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:39.225 7fe93f34e700 1 -- 172.21.2.102:0/3628968165 >> 172.21.2.102:0/3628968165 conn(0x7fe938094e10 msgr2=0x7fe9380a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:04:39.223 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:39.225 7fe93f34e700 1 -- 172.21.2.102:0/3628968165 shutdown_connections 2021-02-10T11:04:39.223 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:39.225 7fe93f34e700 1 -- 172.21.2.102:0/3628968165 wait complete. 2021-02-10T11:04:39.231 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-02-10T11:04:39.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:39.646 7feab36fa700 1 Processor -- start 2021-02-10T11:04:39.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:39.646 7feab36fa700 1 -- start start 2021-02-10T11:04:39.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:39.647 7feab36fa700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7feaac1b0470 0x7feaac1b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:39.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:39.647 7feab36fa700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7feaac0ef610 0x7feaac0efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:39.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:39.647 7feab36fa700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feaac0f0b50 con 0x7feaac0f0030 2021-02-10T11:04:39.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:39.647 7feab36fa700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feaac0e52e0 con 0x7feaac1b0470 2021-02-10T11:04:39.663 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:39.647 7feab36fa700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feaac0e54d0 con 0x7feaac0ef610 2021-02-10T11:04:39.663 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:39.648 7feab1d17700 1 --1- >> v1:172.21.2.113:6789/0 conn(0x7feaac0f0030 0x7feaac0f10d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.113:6789/0 says I am v1:172.21.2.102:49570/0 (socket says 172.21.2.102:49570) 2021-02-10T11:04:39.663 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:39.648 7feab1d17700 1 -- 172.21.2.102:0/3403576980 learned_addr learned my addr 172.21.2.102:0/3403576980 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:04:39.663 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:39.648 7feab1516700 1 --2- 172.21.2.102:0/3403576980 >> v2:172.21.2.113:3300/0 conn(0x7feaac1b0470 0x7feaac1b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:39.663 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:39.648 7feab0d15700 1 --2- 172.21.2.102:0/3403576980 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7feaac0ef610 0x7feaac0efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:39.663 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:39.648 7feab1516700 1 -- 172.21.2.102:0/3403576980 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7feaac0ef610 msgr2=0x7feaac0efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:04:39.663 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:39.648 7feab1516700 1 --2- 172.21.2.102:0/3403576980 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7feaac0ef610 0x7feaac0efa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:39.663 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:39.648 7feab1516700 1 -- 172.21.2.102:0/3403576980 >> v1:172.21.2.113:6789/0 conn(0x7feaac0f0030 legacy=0x7feaac0f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:39.663 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:39.648 7feab1516700 1 -- 172.21.2.102:0/3403576980 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feaac0f0430 con 0x7feaac1b0470 2021-02-10T11:04:39.663 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:39.648 7feab0d15700 1 --2- 172.21.2.102:0/3403576980 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7feaac0ef610 0x7feaac0efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:04:39.663 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:39.649 7feab1516700 1 --2- 172.21.2.102:0/3403576980 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7feaac1b0470 0x7feaac1b0870 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7feaa0006650 tx=0x7feaa0009a80).ready entity=mon.1 client_cookie=cfee6f931e07df44 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:04:39.663 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:39.649 7fea9ffff700 1 -- 172.21.2.102:0/3403576980 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7feaa000c0c0 con 0x7feaac1b0470 2021-02-10T11:04:39.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:39.649 7fea9ffff700 1 -- 172.21.2.102:0/3403576980 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feaa000c340 con 0x7feaac1b0470 2021-02-10T11:04:39.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:39.649 7fea9ffff700 1 -- 172.21.2.102:0/3403576980 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7feaa000c860 con 0x7feaac1b0470 2021-02-10T11:04:39.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:39.649 7feab36fa700 1 -- 172.21.2.102:0/3403576980 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7feaac1b0470 msgr2=0x7feaac1b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:39.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:39.649 7feab36fa700 1 --2- 172.21.2.102:0/3403576980 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7feaac1b0470 0x7feaac1b0870 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7feaa0006650 tx=0x7feaa0009a80).stop 2021-02-10T11:04:39.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:39.650 7feab36fa700 1 -- 172.21.2.102:0/3403576980 shutdown_connections 2021-02-10T11:04:39.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:39.650 7feab36fa700 1 --2- 172.21.2.102:0/3403576980 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7feaac0ef610 0x7feaac0efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:39.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:39.650 7feab36fa700 1 --2- 172.21.2.102:0/3403576980 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7feaac1b0470 0x7feaac1b0870 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:39.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:39.650 7feab36fa700 1 -- 172.21.2.102:0/3403576980 >> 172.21.2.102:0/3403576980 conn(0x7feaac0ce3e0 msgr2=0x7feaac0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:04:39.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:39.650 7feab36fa700 1 -- 172.21.2.102:0/3403576980 shutdown_connections 2021-02-10T11:04:39.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:39.650 7feab36fa700 1 -- 172.21.2.102:0/3403576980 wait complete. 2021-02-10T11:04:39.666 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:39.664 7feab36fa700 1 Processor -- start 2021-02-10T11:04:39.666 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:39.664 7feab36fa700 1 -- start start 2021-02-10T11:04:39.666 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:39.665 7feab36fa700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7feaac105d10 0x7feaac09eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:39.666 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:39.665 7feab36fa700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7feaac106110 0x7feaac09f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:39.666 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:39.665 7feab36fa700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7feaac0b2a00 0x7feaac0b2e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:39.666 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:39.665 7feab36fa700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feaac09e560 con 0x7feaac106110 2021-02-10T11:04:39.666 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:39.665 7feab36fa700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feaac09c090 con 0x7feaac105d10 2021-02-10T11:04:39.666 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:39.665 7feab36fa700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feaac0b3420 con 0x7feaac0b2a00 2021-02-10T11:04:39.666 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:39.665 7feab0d15700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7feaac106110 0x7feaac09f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:39.667 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:39.665 7feab0d15700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7feaac106110 0x7feaac09f100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:50030/0 (socket says 172.21.2.102:50030) 2021-02-10T11:04:39.667 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:39.665 7feab0d15700 1 -- 172.21.2.102:0/1215640843 learned_addr learned my addr 172.21.2.102:0/1215640843 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:04:39.667 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:39.666 7feab0d15700 1 -- 172.21.2.102:0/1215640843 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7feaac0b2a00 msgr2=0x7feaac0b2e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:39.667 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:39.666 7feab0d15700 1 --2- 172.21.2.102:0/1215640843 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7feaac0b2a00 0x7feaac0b2e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:39.667 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:39.666 7feab0d15700 1 -- 172.21.2.102:0/1215640843 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7feaac105d10 msgr2=0x7feaac09eae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:39.667 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:39.666 7feab1516700 1 --2- 172.21.2.102:0/1215640843 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7feaac105d10 0x7feaac09eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:39.667 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:39.666 7feab0d15700 1 --2- 172.21.2.102:0/1215640843 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7feaac105d10 0x7feaac09eae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:39.667 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:39.666 7feab0d15700 1 -- 172.21.2.102:0/1215640843 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feaac0b3640 con 0x7feaac106110 2021-02-10T11:04:39.667 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:39.666 7feab0d15700 1 --2- 172.21.2.102:0/1215640843 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7feaac106110 0x7feaac09f100 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7feaa800b580 tx=0x7feaa800b920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:04:39.667 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:39.666 7fea9dffb700 1 -- 172.21.2.102:0/1215640843 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7feaa80009b0 con 0x7feaac106110 2021-02-10T11:04:39.667 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:39.666 7fea9dffb700 1 -- 172.21.2.102:0/1215640843 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feaa800bd10 con 0x7feaac106110 2021-02-10T11:04:39.667 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:39.666 7fea9dffb700 1 -- 172.21.2.102:0/1215640843 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7feaa80022b0 con 0x7feaac106110 2021-02-10T11:04:39.667 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:39.666 7feab36fa700 1 -- 172.21.2.102:0/1215640843 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feaac0b3640 con 0x7feaac106110 2021-02-10T11:04:39.668 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:39.666 7feab36fa700 1 -- 172.21.2.102:0/1215640843 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feaac0b3bd0 con 0x7feaac106110 2021-02-10T11:04:39.668 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:39.667 7fea9dffb700 1 -- 172.21.2.102:0/1215640843 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7feaa8000c90 con 0x7feaac106110 2021-02-10T11:04:39.668 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:39.667 7fea9dffb700 1 --2- 172.21.2.102:0/1215640843 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fea8c0186c0 0x7fea8c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:39.668 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:39.667 7fea9dffb700 1 -- 172.21.2.102:0/1215640843 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5573+0+0 (secure 0 0 0) 0x7feaa80323e0 con 0x7feaac106110 2021-02-10T11:04:39.668 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:39.667 7feab1516700 1 --2- 172.21.2.102:0/1215640843 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fea8c0186c0 0x7fea8c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:39.668 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:39.667 7feab1516700 1 --2- 172.21.2.102:0/1215640843 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fea8c0186c0 0x7fea8c01abc0 crc :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:04:39.695 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:39.697 7feab36fa700 1 -- 172.21.2.102:0/1215640843 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feaac105ae0 con 0x7feaac106110 2021-02-10T11:04:39.697 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:39.699 7fea9dffb700 1 -- 172.21.2.102:0/1215640843 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7feaa8000940 con 0x7feaac106110 2021-02-10T11:04:39.827 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:39.829 7feab36fa700 1 -- 172.21.2.102:0/1215640843 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7feaac0f0b50 con 0x7feaac106110 2021-02-10T11:04:39.827 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:39.829 7fea9dffb700 1 -- 172.21.2.102:0/1215640843 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v20) v1 ==== 100+0+10 (secure 0 0 0) 0x7feaa8000c90 con 0x7feaac106110 2021-02-10T11:04:39.828 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:04:39.829 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:39.831 7feab36fa700 1 -- 172.21.2.102:0/1215640843 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fea8c0186c0 msgr2=0x7fea8c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:39.830 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:39.831 7feab36fa700 1 --2- 172.21.2.102:0/1215640843 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fea8c0186c0 0x7fea8c01abc0 crc :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:39.830 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:39.831 7feab36fa700 1 -- 172.21.2.102:0/1215640843 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7feaac106110 msgr2=0x7feaac09f100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:39.830 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:39.831 7feab36fa700 1 --2- 172.21.2.102:0/1215640843 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7feaac106110 0x7feaac09f100 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7feaa800b580 tx=0x7feaa800b920).stop 2021-02-10T11:04:39.830 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:39.831 7feab36fa700 1 -- 172.21.2.102:0/1215640843 shutdown_connections 2021-02-10T11:04:39.830 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:39.831 7feab36fa700 1 --2- 172.21.2.102:0/1215640843 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7feaac0b2a00 0x7feaac0b2e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:39.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:39.831 7feab36fa700 1 --2- 172.21.2.102:0/1215640843 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7feaac106110 0x7feaac09f100 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:39.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:39.831 7feab36fa700 1 --2- 172.21.2.102:0/1215640843 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fea8c0186c0 0x7fea8c01abc0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:39.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:39.831 7feab36fa700 1 --2- 172.21.2.102:0/1215640843 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7feaac105d10 0x7feaac09eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:39.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:39.831 7feab36fa700 1 -- 172.21.2.102:0/1215640843 >> 172.21.2.102:0/1215640843 conn(0x7feaac0c36f0 msgr2=0x7feaac0bd640 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:04:39.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:39.832 7feab36fa700 1 -- 172.21.2.102:0/1215640843 shutdown_connections 2021-02-10T11:04:39.832 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:39.832 7feab36fa700 1 -- 172.21.2.102:0/1215640843 wait complete. 2021-02-10T11:04:39.838 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:04:40.265 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:40.250 7ff404dfb700 1 Processor -- start 2021-02-10T11:04:40.265 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:40.250 7ff404dfb700 1 -- start start 2021-02-10T11:04:40.265 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:40.251 7ff404dfb700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7ff4000e5070 0x7ff4001b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:40.265 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:40.251 7ff404dfb700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7ff4001b3e40 0x7ff40007d690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:40.265 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:40.251 7ff404dfb700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff40007ed80 con 0x7ff4000e5470 2021-02-10T11:04:40.265 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:40.251 7ff404dfb700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff400080310 con 0x7ff4000e5070 2021-02-10T11:04:40.265 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:40.251 7ff404dfb700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff400080530 con 0x7ff4001b3e40 2021-02-10T11:04:40.266 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:40.251 7ff3feb13700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7ff4000e5070 0x7ff4001b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:40.266 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:40.251 7ff3ff314700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7ff4000e5470 0x7ff40007dcb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:42258/0 (socket says 172.21.2.102:42258) 2021-02-10T11:04:40.266 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:40.251 7ff3ff314700 1 -- 172.21.2.102:0/4069628593 learned_addr learned my addr 172.21.2.102:0/4069628593 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:04:40.266 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:40.251 7ff3fe312700 1 --2- 172.21.2.102:0/4069628593 >> v2:172.21.2.113:3300/0 conn(0x7ff4001b3e40 0x7ff40007d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:40.266 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:40.251 7ff3feb13700 1 -- 172.21.2.102:0/4069628593 >> v2:172.21.2.113:3300/0 conn(0x7ff4001b3e40 msgr2=0x7ff40007d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:04:40.266 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:40.251 7ff3feb13700 1 --2- 172.21.2.102:0/4069628593 >> v2:172.21.2.113:3300/0 conn(0x7ff4001b3e40 0x7ff40007d690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:40.266 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:40.251 7ff3feb13700 1 -- 172.21.2.102:0/4069628593 >> v1:172.21.2.102:6789/0 conn(0x7ff4000e5470 legacy=0x7ff40007dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:40.266 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:40.251 7ff3fdb11700 1 -- 172.21.2.102:0/4069628593 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1234217449 0 0) 0x7ff3f8001500 con 0x7ff4000e5470 2021-02-10T11:04:40.266 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:40.251 7ff3feb13700 1 -- 172.21.2.102:0/4069628593 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff400080750 con 0x7ff4000e5070 2021-02-10T11:04:40.266 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:40.252 7ff3feb13700 1 --2- 172.21.2.102:0/4069628593 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff4000e5070 0x7ff4001b3820 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7ff3f0006650 tx=0x7ff3f0009a00).ready entity=mon.0 client_cookie=6fc9de3c9d14ede6 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:04:40.266 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:40.252 7ff3fdb11700 1 -- 172.21.2.102:0/4069628593 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff3f00009b0 con 0x7ff4000e5070 2021-02-10T11:04:40.267 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:40.252 7ff3fdb11700 1 -- 172.21.2.102:0/4069628593 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3f000c0c0 con 0x7ff4000e5070 2021-02-10T11:04:40.267 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:40.252 7ff3fdb11700 1 -- 172.21.2.102:0/4069628593 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff3f000c5a0 con 0x7ff4000e5070 2021-02-10T11:04:40.267 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:40.252 7ff404dfb700 1 -- 172.21.2.102:0/4069628593 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff4000e5070 msgr2=0x7ff4001b3820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:40.267 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:40.252 7ff404dfb700 1 --2- 172.21.2.102:0/4069628593 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff4000e5070 0x7ff4001b3820 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7ff3f0006650 tx=0x7ff3f0009a00).stop 2021-02-10T11:04:40.267 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:40.253 7ff404dfb700 1 -- 172.21.2.102:0/4069628593 shutdown_connections 2021-02-10T11:04:40.267 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:40.253 7ff404dfb700 1 --2- 172.21.2.102:0/4069628593 >> v2:172.21.2.113:3300/0 conn(0x7ff4001b3e40 0x7ff40007d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:40.267 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:40.253 7ff404dfb700 1 --2- 172.21.2.102:0/4069628593 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff4000e5070 0x7ff4001b3820 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:40.267 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:40.253 7ff404dfb700 1 -- 172.21.2.102:0/4069628593 >> 172.21.2.102:0/4069628593 conn(0x7ff4000ce690 msgr2=0x7ff4000e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:04:40.267 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:40.253 7ff404dfb700 1 -- 172.21.2.102:0/4069628593 shutdown_connections 2021-02-10T11:04:40.267 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:40.253 7ff404dfb700 1 -- 172.21.2.102:0/4069628593 wait complete. 2021-02-10T11:04:40.269 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:40.268 7ff404dfb700 1 Processor -- start 2021-02-10T11:04:40.269 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:40.268 7ff404dfb700 1 -- start start 2021-02-10T11:04:40.269 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:40.268 7ff404dfb700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff40008d400 0x7ff400096c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:40.269 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:40.268 7ff404dfb700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff4000b1450 0x7ff4000b1850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:40.269 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:40.268 7ff404dfb700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff4000b1e70 0x7ff4001b1f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:40.269 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:40.268 7ff404dfb700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff400097310 con 0x7ff4000b1450 2021-02-10T11:04:40.269 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:40.268 7ff404dfb700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff40008d800 con 0x7ff4000b1e70 2021-02-10T11:04:40.269 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:40.268 7ff404dfb700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff40008d9f0 con 0x7ff40008d400 2021-02-10T11:04:40.269 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:40.268 7ff3fe312700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff4000b1450 0x7ff4000b1850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:40.269 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:40.268 7ff3fe312700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff4000b1450 0x7ff4000b1850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:50040/0 (socket says 172.21.2.102:50040) 2021-02-10T11:04:40.269 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:40.268 7ff3fe312700 1 -- 172.21.2.102:0/4163213786 learned_addr learned my addr 172.21.2.102:0/4163213786 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:04:40.269 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:40.269 7ff3fe312700 1 -- 172.21.2.102:0/4163213786 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff40008d400 msgr2=0x7ff400096c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:40.270 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:40.269 7ff3fe312700 1 --2- 172.21.2.102:0/4163213786 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff40008d400 0x7ff400096c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:40.270 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:40.269 7ff3fe312700 1 -- 172.21.2.102:0/4163213786 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff4000b1e70 msgr2=0x7ff4001b1f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:40.270 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:40.269 7ff3fe312700 1 --2- 172.21.2.102:0/4163213786 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff4000b1e70 0x7ff4001b1f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:40.270 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:40.269 7ff3fe312700 1 -- 172.21.2.102:0/4163213786 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4001b25f0 con 0x7ff4000b1450 2021-02-10T11:04:40.270 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:40.269 7ff3fe312700 1 --2- 172.21.2.102:0/4163213786 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff4000b1450 0x7ff4000b1850 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 rx=0x7ff3f4006f60 tx=0x7ff3f400b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:04:40.270 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:40.269 7ff3ef7fe700 1 -- 172.21.2.102:0/4163213786 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff3f400bd60 con 0x7ff4000b1450 2021-02-10T11:04:40.270 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:40.269 7ff3ef7fe700 1 -- 172.21.2.102:0/4163213786 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3f4000c90 con 0x7ff4000b1450 2021-02-10T11:04:40.270 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:40.269 7ff3ef7fe700 1 -- 172.21.2.102:0/4163213786 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff3f4002220 con 0x7ff4000b1450 2021-02-10T11:04:40.270 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:40.269 7ff404dfb700 1 -- 172.21.2.102:0/4163213786 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4001b25f0 con 0x7ff4000b1450 2021-02-10T11:04:40.270 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:40.269 7ff404dfb700 1 -- 172.21.2.102:0/4163213786 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4001b2b80 con 0x7ff4000b1450 2021-02-10T11:04:40.270 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:40.270 7ff3ef7fe700 1 -- 172.21.2.102:0/4163213786 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ff3f40052e0 con 0x7ff4000b1450 2021-02-10T11:04:40.270 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:40.270 7ff3ef7fe700 1 --2- 172.21.2.102:0/4163213786 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff3e40186c0 0x7ff3e401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:40.271 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:40.270 7ff3ef7fe700 1 -- 172.21.2.102:0/4163213786 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5573+0+0 (secure 0 0 0) 0x7ff3f40325b0 con 0x7ff4000b1450 2021-02-10T11:04:40.271 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:40.270 7ff3feb13700 1 --2- 172.21.2.102:0/4163213786 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff3e40186c0 0x7ff3e401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:40.271 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:40.270 7ff3feb13700 1 --2- 172.21.2.102:0/4163213786 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff3e40186c0 0x7ff3e401abc0 crc :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:04:40.298 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:40.300 7ff404dfb700 1 -- 172.21.2.102:0/4163213786 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff40007ed80 con 0x7ff4000b1450 2021-02-10T11:04:40.300 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:40.303 7ff3ef7fe700 1 -- 172.21.2.102:0/4163213786 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ff3f402df10 con 0x7ff4000b1450 2021-02-10T11:04:40.426 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:40.428 7ff404dfb700 1 -- 172.21.2.102:0/4163213786 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff40007ed80 con 0x7ff4000b1450 2021-02-10T11:04:40.426 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:40.429 7ff3ef7fe700 1 -- 172.21.2.102:0/4163213786 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+13993 (secure 0 0 0) 0x7ff3f402df10 con 0x7ff4000b1450 2021-02-10T11:04:40.427 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:04:40.427 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":20,"fsid":"68db30ea-dbce-4aef-a6df-f01c6fea0513","created":"2021-02-10 11:03:06.331557","modified":"2021-02-10 11:03:44.386945","last_up_change":"2021-02-10 11:03:32.690562","last_in_change":"2021-02-10 11:03:32.690562","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:03:37.322519","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:03:41.161563","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2021-02-10 11:03:42.379651","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"be691b4d-a039-408a-af80-9d53f02b6f1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6818","nonce":36088},{"type":"v1","addr":"172.21.2.102:6820","nonce":36088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6822","nonce":36088},{"type":"v1","addr":"172.21.2.102:6824","nonce":36088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6830","nonce":36088},{"type":"v1","addr":"172.21.2.102:6832","nonce":36088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6826","nonce":36088},{"type":"v1","addr":"172.21.2.102:6828","nonce":36088}]},"public_addr":"172.21.2.102:6820/36088","cluster_addr":"172.21.2.102:6824/36088","heartbeat_back_addr":"172.21.2.102:6832/36088","heartbeat_front_addr":"172.21.2.102:6828/36088","state":["exists","up"]},{"osd":1,"uuid":"64e3ccc8-9c69-4d7c-865e-af8e83822635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6819","nonce":36087},{"type":"v1","addr":"172.21.2.102:6821","nonce":36087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6823","nonce":36087},{"type":"v1","addr":"172.21.2.102:6825","nonce":36087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6831","nonce":36087},{"type":"v1","addr":"172.21.2.102:6833","nonce":36087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6827","nonce":36087},{"type":"v1","addr":"172.21.2.102:6829","nonce":36087}]},"public_addr":"172.21.2.102:6821/36087","cluster_addr":"172.21.2.102:6825/36087","heartbeat_back_addr":"172.21.2.102:6833/36087","heartbeat_front_addr":"172.21.2.102:6829/36087","state":["exists","up"]},{"osd":2,"uuid":"153700af-747f-481f-bdf4-d6072d3a842f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6810","nonce":36089},{"type":"v1","addr":"172.21.2.102:6811","nonce":36089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6812","nonce":36089},{"type":"v1","addr":"172.21.2.102:6813","nonce":36089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6816","nonce":36089},{"type":"v1","addr":"172.21.2.102:6817","nonce":36089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6814","nonce":36089},{"type":"v1","addr":"172.21.2.102:6815","nonce":36089}]},"public_addr":"172.21.2.102:6811/36089","cluster_addr":"172.21.2.102:6813/36089","heartbeat_back_addr":"172.21.2.102:6817/36089","heartbeat_front_addr":"172.21.2.102:6815/36089","state":["exists","up"]},{"osd":3,"uuid":"cbc678f4-83a9-45e6-8830-2f7c232b1ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6802","nonce":36090},{"type":"v1","addr":"172.21.2.102:6803","nonce":36090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6804","nonce":36090},{"type":"v1","addr":"172.21.2.102:6805","nonce":36090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6808","nonce":36090},{"type":"v1","addr":"172.21.2.102:6809","nonce":36090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6806","nonce":36090},{"type":"v1","addr":"172.21.2.102:6807","nonce":36090}]},"public_addr":"172.21.2.102:6803/36090","cluster_addr":"172.21.2.102:6805/36090","heartbeat_back_addr":"172.21.2.102:6809/36090","heartbeat_front_addr":"172.21.2.102:6807/36090","state":["exists","up"]},{"osd":4,"uuid":"c01e4008-b258-4f10-afd2-9e7f37214351","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6808","nonce":36368},{"type":"v1","addr":"172.21.2.113:6811","nonce":36368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6814","nonce":36368},{"type":"v1","addr":"172.21.2.113:6818","nonce":36368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6826","nonce":36368},{"type":"v1","addr":"172.21.2.113:6828","nonce":36368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6820","nonce":36368},{"type":"v1","addr":"172.21.2.113:6824","nonce":36368}]},"public_addr":"172.21.2.113:6811/36368","cluster_addr":"172.21.2.113:6818/36368","heartbeat_back_addr":"172.21.2.113:6828/36368","heartbeat_front_addr":"172.21.2.113:6824/36368","state":["exists","up"]},{"osd":5,"uuid":"551f6007-daf8-4a98-8ee0-c2b16bcccb5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6802","nonce":36366},{"type":"v1","addr":"172.21.2.113:6805","nonce":36366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6807","nonce":36366},{"type":"v1","addr":"172.21.2.113:6810","nonce":36366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6819","nonce":36366},{"type":"v1","addr":"172.21.2.113:6822","nonce":36366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6812","nonce":36366},{"type":"v1","addr":"172.21.2.113:6815","nonce":36366}]},"public_addr":"172.21.2.113:6805/36366","cluster_addr":"172.21.2.113:6810/36366","heartbeat_back_addr":"172.21.2.113:6822/36366","heartbeat_front_addr":"172.21.2.113:6815/36366","state":["exists","up"]},{"osd":6,"uuid":"bf67c040-0e50-48be-b102-c3943bc9ebef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6800","nonce":36367},{"type":"v1","addr":"172.21.2.113:6801","nonce":36367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6803","nonce":36367},{"type":"v1","addr":"172.21.2.113:6804","nonce":36367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6813","nonce":36367},{"type":"v1","addr":"172.21.2.113:6816","nonce":36367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6806","nonce":36367},{"type":"v1","addr":"172.21.2.113:6809","nonce":36367}]},"public_addr":"172.21.2.113:6801/36367","cluster_addr":"172.21.2.113:6804/36367","heartbeat_back_addr":"172.21.2.113:6816/36367","heartbeat_front_addr":"172.21.2.113:6809/36367","state":["exists","up"]},{"osd":7,"uuid":"f5461f80-018d-428f-be89-2eb2bcbe7bb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6817","nonce":36365},{"type":"v1","addr":"172.21.2.113:6821","nonce":36365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6823","nonce":36365},{"type":"v1","addr":"172.21.2.113:6825","nonce":36365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6830","nonce":36365},{"type":"v1","addr":"172.21.2.113:6831","nonce":36365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6827","nonce":36365},{"type":"v1","addr":"172.21.2.113:6829","nonce":36365}]},"public_addr":"172.21.2.113:6821/36365","cluster_addr":"172.21.2.113:6825/36365","heartbeat_back_addr":"172.21.2.113:6831/36365","heartbeat_front_addr":"172.21.2.113:6829/36365","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:04:40.428 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:40.430 7ff404dfb700 1 -- 172.21.2.102:0/4163213786 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff3e40186c0 msgr2=0x7ff3e401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:40.428 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:40.430 7ff404dfb700 1 --2- 172.21.2.102:0/4163213786 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff3e40186c0 0x7ff3e401abc0 crc :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:40.428 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:40.430 7ff404dfb700 1 -- 172.21.2.102:0/4163213786 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff4000b1450 msgr2=0x7ff4000b1850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:40.429 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:40.430 7ff404dfb700 1 --2- 172.21.2.102:0/4163213786 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff4000b1450 0x7ff4000b1850 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 rx=0x7ff3f4006f60 tx=0x7ff3f400b970).stop 2021-02-10T11:04:40.429 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:40.431 7ff404dfb700 1 -- 172.21.2.102:0/4163213786 shutdown_connections 2021-02-10T11:04:40.429 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:40.431 7ff404dfb700 1 --2- 172.21.2.102:0/4163213786 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff3e40186c0 0x7ff3e401abc0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:40.429 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:40.431 7ff404dfb700 1 --2- 172.21.2.102:0/4163213786 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff4000b1e70 0x7ff4001b1f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:40.429 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:40.431 7ff404dfb700 1 --2- 172.21.2.102:0/4163213786 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff4000b1450 0x7ff4000b1850 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:40.429 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:40.431 7ff404dfb700 1 --2- 172.21.2.102:0/4163213786 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff40008d400 0x7ff400096c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:40.429 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:40.431 7ff404dfb700 1 -- 172.21.2.102:0/4163213786 >> 172.21.2.102:0/4163213786 conn(0x7ff4001ac620 msgr2=0x7ff4000896e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:04:40.429 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:40.431 7ff404dfb700 1 -- 172.21.2.102:0/4163213786 shutdown_connections 2021-02-10T11:04:40.429 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:40.431 7ff404dfb700 1 -- 172.21.2.102:0/4163213786 wait complete. 2021-02-10T11:04:40.437 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:03:37.322519', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2021-02-10 11:03:41.161563', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2021-02-10 11:03:42.379651', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-02-10T11:04:40.437 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:04:40.867 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:40.852 7f22ab09d700 1 Processor -- start 2021-02-10T11:04:40.867 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:40.852 7f22ab09d700 1 -- start start 2021-02-10T11:04:40.867 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:40.853 7f22ab09d700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f22a41b0470 0x7f22a41b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:40.867 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:40.853 7f22ab09d700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f22a40ef610 0x7f22a40efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:40.867 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:40.853 7f22ab09d700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22a40f0b50 con 0x7f22a40f0030 2021-02-10T11:04:40.867 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:40.853 7f22ab09d700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22a40e52e0 con 0x7f22a40ef610 2021-02-10T11:04:40.868 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:40.853 7f22ab09d700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22a40e5490 con 0x7f22a41b0470 2021-02-10T11:04:40.868 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:40.853 7f22a96ba700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f22a40f0030 0x7f22a40f10d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:42268/0 (socket says 172.21.2.102:42268) 2021-02-10T11:04:40.868 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:40.853 7f22a96ba700 1 -- 172.21.2.102:0/806096212 learned_addr learned my addr 172.21.2.102:0/806096212 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:04:40.868 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:40.853 7f22a8eb9700 1 --2- 172.21.2.102:0/806096212 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f22a41b0470 0x7f22a41b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:40.868 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:40.853 7f22a3fff700 1 --2- 172.21.2.102:0/806096212 >> v2:172.21.2.113:3300/0 conn(0x7f22a40ef610 0x7f22a40efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:40.868 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:40.854 7f22a8eb9700 1 -- 172.21.2.102:0/806096212 >> v2:172.21.2.113:3300/0 conn(0x7f22a40ef610 msgr2=0x7f22a40efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:04:40.868 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:40.854 7f22a8eb9700 1 --2- 172.21.2.102:0/806096212 >> v2:172.21.2.113:3300/0 conn(0x7f22a40ef610 0x7f22a40efa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:40.868 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:40.854 7f22a8eb9700 1 -- 172.21.2.102:0/806096212 >> v1:172.21.2.102:6789/0 conn(0x7f22a40f0030 legacy=0x7f22a40f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:40.868 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:40.854 7f22a37fe700 1 -- 172.21.2.102:0/806096212 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3399013400 0 0) 0x7f229c001180 con 0x7f22a40f0030 2021-02-10T11:04:40.868 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:40.854 7f22a8eb9700 1 -- 172.21.2.102:0/806096212 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22a40f0430 con 0x7f22a41b0470 2021-02-10T11:04:40.868 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:40.854 7f22a8eb9700 1 --2- 172.21.2.102:0/806096212 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f22a41b0470 0x7f22a41b0870 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7f228c006650 tx=0x7f228c009a00).ready entity=mon.2 client_cookie=eba46ed14240bf42 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:04:40.868 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:40.854 7f22a37fe700 1 -- 172.21.2.102:0/806096212 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f228c009df0 con 0x7f22a41b0470 2021-02-10T11:04:40.868 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:40.854 7f22a37fe700 1 -- 172.21.2.102:0/806096212 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f228c0009b0 con 0x7f22a41b0470 2021-02-10T11:04:40.869 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:40.854 7f22a37fe700 1 -- 172.21.2.102:0/806096212 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f228c00c2b0 con 0x7f22a41b0470 2021-02-10T11:04:40.869 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:40.854 7f22ab09d700 1 -- 172.21.2.102:0/806096212 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f22a41b0470 msgr2=0x7f22a41b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:40.869 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:40.854 7f22ab09d700 1 --2- 172.21.2.102:0/806096212 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f22a41b0470 0x7f22a41b0870 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7f228c006650 tx=0x7f228c009a00).stop 2021-02-10T11:04:40.869 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:40.855 7f22ab09d700 1 -- 172.21.2.102:0/806096212 shutdown_connections 2021-02-10T11:04:40.869 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:40.855 7f22ab09d700 1 --2- 172.21.2.102:0/806096212 >> v2:172.21.2.113:3300/0 conn(0x7f22a40ef610 0x7f22a40efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:40.869 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:40.855 7f22ab09d700 1 --2- 172.21.2.102:0/806096212 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f22a41b0470 0x7f22a41b0870 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:40.869 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:40.855 7f22ab09d700 1 -- 172.21.2.102:0/806096212 >> 172.21.2.102:0/806096212 conn(0x7f22a40ce3e0 msgr2=0x7f22a40e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:04:40.869 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:40.855 7f22ab09d700 1 -- 172.21.2.102:0/806096212 shutdown_connections 2021-02-10T11:04:40.869 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:40.855 7f22ab09d700 1 -- 172.21.2.102:0/806096212 wait complete. 2021-02-10T11:04:40.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:40.870 7f22ab09d700 1 Processor -- start 2021-02-10T11:04:40.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:40.870 7f22ab09d700 1 -- start start 2021-02-10T11:04:40.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:40.870 7f22ab09d700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f22a408ece0 0x7f22a40a6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:40.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:40.870 7f22ab09d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f22a40a6860 0x7f22a40a6c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:40.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:40.870 7f22ab09d700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f22a40a7280 0x7f22a40a4590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:40.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:40.870 7f22ab09d700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22a40a5cc0 con 0x7f22a40a6860 2021-02-10T11:04:40.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:40.870 7f22ab09d700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22a408f0e0 con 0x7f22a40a7280 2021-02-10T11:04:40.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:40.870 7f22ab09d700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22a408f2c0 con 0x7f22a408ece0 2021-02-10T11:04:40.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:40.870 7f22a3fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f22a40a6860 0x7f22a40a6c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:40.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:40.870 7f22a8eb9700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f22a408ece0 0x7f22a40a6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:40.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:40.870 7f22a96ba700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f22a40a7280 0x7f22a40a4590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:40.872 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:40.870 7f22a3fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f22a40a6860 0x7f22a40a6c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:50052/0 (socket says 172.21.2.102:50052) 2021-02-10T11:04:40.872 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:40.870 7f22a3fff700 1 -- 172.21.2.102:0/355594571 learned_addr learned my addr 172.21.2.102:0/355594571 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:04:40.872 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:40.870 7f22a96ba700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f22a40a7280 0x7f22a40a4590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.113:3300/0 says I am v2:172.21.2.102:53478/0 (socket says 172.21.2.102:53478) 2021-02-10T11:04:40.872 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:40.871 7f22a96ba700 1 -- 172.21.2.102:0/355594571 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f22a408ece0 msgr2=0x7f22a40a6240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:40.872 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:40.871 7f22a96ba700 1 --2- 172.21.2.102:0/355594571 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f22a408ece0 0x7f22a40a6240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:40.872 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:40.871 7f22a96ba700 1 -- 172.21.2.102:0/355594571 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f22a40a6860 msgr2=0x7f22a40a6c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:40.872 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:40.871 7f22a96ba700 1 --2- 172.21.2.102:0/355594571 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f22a40a6860 0x7f22a40a6c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:40.872 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:40.871 7f22a96ba700 1 -- 172.21.2.102:0/355594571 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22a40a4bb0 con 0x7f22a40a7280 2021-02-10T11:04:40.872 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:40.871 7f22a96ba700 1 --2- 172.21.2.102:0/355594571 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f22a40a7280 0x7f22a40a4590 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f229c00a4a0 tx=0x7f229c00a840).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:04:40.872 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:40.871 7f22a17fa700 1 -- 172.21.2.102:0/355594571 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f229c000c10 con 0x7f22a40a7280 2021-02-10T11:04:40.872 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:40.871 7f22a17fa700 1 -- 172.21.2.102:0/355594571 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f229c007030 con 0x7f22a40a7280 2021-02-10T11:04:40.872 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:40.871 7f22ab09d700 1 -- 172.21.2.102:0/355594571 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22a40a4bb0 con 0x7f22a40a7280 2021-02-10T11:04:40.872 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:40.871 7f22a17fa700 1 -- 172.21.2.102:0/355594571 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f229c0043a0 con 0x7f22a40a7280 2021-02-10T11:04:40.872 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:40.871 7f22ab09d700 1 -- 172.21.2.102:0/355594571 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22a40a4bb0 con 0x7f22a40a7280 2021-02-10T11:04:40.873 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:40.872 7f22a17fa700 1 -- 172.21.2.102:0/355594571 <== mon.1 v2:172.21.2.113:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f229c0043a0 con 0x7f22a40a7280 2021-02-10T11:04:40.873 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:40.872 7f22a17fa700 1 --2- 172.21.2.102:0/355594571 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2288018730 0x7f228801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:40.873 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:40.872 7f22a17fa700 1 -- 172.21.2.102:0/355594571 <== mon.1 v2:172.21.2.113:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f229c032560 con 0x7f22a40a7280 2021-02-10T11:04:40.873 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:40.872 7f22a8eb9700 1 --2- 172.21.2.102:0/355594571 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2288018730 0x7f228801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:40.873 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:40.872 7f22a8eb9700 1 --2- 172.21.2.102:0/355594571 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2288018730 0x7f228801ac10 crc :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:04:40.907 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:40.903 7f22ab09d700 1 -- 172.21.2.102:0/355594571 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22a4105ae0 con 0x7f22a40a7280 2021-02-10T11:04:40.907 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:40.906 7f22a17fa700 1 -- 172.21.2.102:0/355594571 <== mon.1 v2:172.21.2.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f229c000940 con 0x7f22a40a7280 2021-02-10T11:04:41.034 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:41.036 7f22ab09d700 1 -- 172.21.2.102:0/355594571 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f22a40f0b50 con 0x7f22a40a7280 2021-02-10T11:04:41.035 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:41.037 7f22a17fa700 1 -- 172.21.2.102:0/355594571 <== mon.1 v2:172.21.2.113:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v20) v1 ==== 92+0+10 (secure 0 0 0) 0x7f229c0043a0 con 0x7f22a40a7280 2021-02-10T11:04:41.037 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:04:41.038 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:41.041 7f22ab09d700 1 -- 172.21.2.102:0/355594571 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2288018730 msgr2=0x7f228801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:41.038 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:41.041 7f22ab09d700 1 --2- 172.21.2.102:0/355594571 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2288018730 0x7f228801ac10 crc :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:41.038 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:41.041 7f22ab09d700 1 -- 172.21.2.102:0/355594571 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f22a40a7280 msgr2=0x7f22a40a4590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:41.039 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:41.041 7f22ab09d700 1 --2- 172.21.2.102:0/355594571 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f22a40a7280 0x7f22a40a4590 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f229c00a4a0 tx=0x7f229c00a840).stop 2021-02-10T11:04:41.039 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:41.041 7f22ab09d700 1 -- 172.21.2.102:0/355594571 shutdown_connections 2021-02-10T11:04:41.039 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:41.041 7f22ab09d700 1 --2- 172.21.2.102:0/355594571 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2288018730 0x7f228801ac10 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:41.039 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:41.041 7f22ab09d700 1 --2- 172.21.2.102:0/355594571 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f22a40a7280 0x7f22a40a4590 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:41.039 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:41.041 7f22ab09d700 1 --2- 172.21.2.102:0/355594571 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f22a40a6860 0x7f22a40a6c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:41.039 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:41.041 7f22ab09d700 1 --2- 172.21.2.102:0/355594571 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f22a408ece0 0x7f22a40a6240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:41.040 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:41.041 7f22ab09d700 1 -- 172.21.2.102:0/355594571 >> 172.21.2.102:0/355594571 conn(0x7f22a408ffe0 msgr2=0x7f22a40bf050 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:04:41.040 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:41.041 7f22ab09d700 1 -- 172.21.2.102:0/355594571 shutdown_connections 2021-02-10T11:04:41.040 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:41.041 7f22ab09d700 1 -- 172.21.2.102:0/355594571 wait complete. 2021-02-10T11:04:41.047 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-02-10T11:04:41.478 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:41.462 7fd50d57b700 1 Processor -- start 2021-02-10T11:04:41.479 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:41.462 7fd50d57b700 1 -- start start 2021-02-10T11:04:41.479 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:41.463 7fd50d57b700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fd5080f05f0 0x7fd5080f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:41.479 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:41.463 7fd50d57b700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5081b0c70 con 0x7fd5081b0870 2021-02-10T11:04:41.479 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:41.463 7fd50d57b700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5080f0b50 con 0x7fd5081b0470 2021-02-10T11:04:41.479 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:41.463 7fd50d57b700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5080e5ea0 con 0x7fd5080f05f0 2021-02-10T11:04:41.479 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:41.463 7fd506abe700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fd5081b0870 0x7fd5080efe00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:42280/0 (socket says 172.21.2.102:42280) 2021-02-10T11:04:41.479 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:41.463 7fd506abe700 1 -- 172.21.2.102:0/3905565702 learned_addr learned my addr 172.21.2.102:0/3905565702 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:04:41.479 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:41.464 7fd507ac0700 1 --2- 172.21.2.102:0/3905565702 >> v2:172.21.2.113:3300/0 conn(0x7fd5080f05f0 0x7fd5080f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:41.479 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:41.464 7fd5062bd700 1 -- 172.21.2.102:0/3905565702 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2743985437 0 0) 0x7fd4fc0009b0 con 0x7fd5081b0870 2021-02-10T11:04:41.479 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:41.464 7fd5062bd700 1 -- 172.21.2.102:0/3905565702 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd4f0002fe0 con 0x7fd5081b0870 2021-02-10T11:04:41.480 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:41.464 7fd5062bd700 1 -- 172.21.2.102:0/3905565702 <== mon.1 v1:172.21.2.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3333329307 0 0) 0x7fd4f8000fb0 con 0x7fd5081b0470 2021-02-10T11:04:41.480 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:41.464 7fd5062bd700 1 -- 172.21.2.102:0/3905565702 --> v1:172.21.2.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd4f0003210 con 0x7fd5081b0470 2021-02-10T11:04:41.480 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:41.464 7fd5062bd700 1 -- 172.21.2.102:0/3905565702 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 472504226 0 0) 0x7fd4fc0010b0 con 0x7fd5081b0870 2021-02-10T11:04:41.480 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:41.464 7fd5062bd700 1 -- 172.21.2.102:0/3905565702 >> v2:172.21.2.113:3300/0 conn(0x7fd5080f05f0 msgr2=0x7fd5080f20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:04:41.480 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:41.464 7fd5062bd700 1 --2- 172.21.2.102:0/3905565702 >> v2:172.21.2.113:3300/0 conn(0x7fd5080f05f0 0x7fd5080f20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:41.480 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:41.464 7fd5062bd700 1 -- 172.21.2.102:0/3905565702 >> v1:172.21.2.113:6789/0 conn(0x7fd5081b0470 legacy=0x7fd5080ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:41.480 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:41.464 7fd5062bd700 1 -- 172.21.2.102:0/3905565702 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5080e52e0 con 0x7fd5081b0870 2021-02-10T11:04:41.480 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:41.464 7fd5062bd700 1 -- 172.21.2.102:0/3905565702 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7fd4fc0015b0 con 0x7fd5081b0870 2021-02-10T11:04:41.480 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:41.464 7fd5062bd700 1 -- 172.21.2.102:0/3905565702 >> v1:172.21.2.102:6789/0 conn(0x7fd5081b0870 legacy=0x7fd5080efe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:41.480 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:41.464 7fd5062bd700 1 --2- 172.21.2.102:0/3905565702 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd4f00037e0 0x7fd4f0003c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:41.480 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:41.464 7fd5062bd700 1 --2- 172.21.2.102:0/3905565702 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd4f00067e0 0x7fd4f0008c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:41.480 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:41.464 7fd5062bd700 1 --2- 172.21.2.102:0/3905565702 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd4f0009580 0x7fd4f000baa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:41.480 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:41.464 7fd5062bd700 1 -- 172.21.2.102:0/3905565702 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4f000c2f0 con 0x7fd4f00067e0 2021-02-10T11:04:41.481 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:41.464 7fd5062bd700 1 -- 172.21.2.102:0/3905565702 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4f00504e0 con 0x7fd4f00037e0 2021-02-10T11:04:41.481 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:41.464 7fd5062bd700 1 -- 172.21.2.102:0/3905565702 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4f0050700 con 0x7fd4f0009580 2021-02-10T11:04:41.481 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:41.464 7fd506abe700 1 --2- 172.21.2.102:0/3905565702 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd4f00067e0 0x7fd4f0008c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:41.481 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:41.465 7fd507ac0700 1 --2- 172.21.2.102:0/3905565702 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd4f0009580 0x7fd4f000baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:41.481 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:41.465 7fd5072bf700 1 --2- 172.21.2.102:0/3905565702 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd4f00037e0 0x7fd4f0003c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:41.481 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:41.465 7fd506abe700 1 -- 172.21.2.102:0/3905565702 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd4f0009580 msgr2=0x7fd4f000baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:04:41.481 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:41.465 7fd506abe700 1 --2- 172.21.2.102:0/3905565702 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd4f0009580 0x7fd4f000baa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:41.481 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:41.465 7fd506abe700 1 -- 172.21.2.102:0/3905565702 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd4f00037e0 msgr2=0x7fd4f0003c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:04:41.481 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:41.465 7fd506abe700 1 --2- 172.21.2.102:0/3905565702 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd4f00037e0 0x7fd4f0003c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:41.481 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:41.465 7fd506abe700 1 -- 172.21.2.102:0/3905565702 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4f00509a0 con 0x7fd4f00067e0 2021-02-10T11:04:41.481 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:41.465 7fd507ac0700 1 -- 172.21.2.102:0/3905565702 reap_dead start 2021-02-10T11:04:41.481 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:41.465 7fd506abe700 1 --2- 172.21.2.102:0/3905565702 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd4f00067e0 0x7fd4f0008c80 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0x7fd4fc007990 tx=0x7fd4fc007d30).ready entity=mon.0 client_cookie=3bf79dd58b58a106 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:04:41.482 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:41.465 7fd5062bd700 1 -- 172.21.2.102:0/3905565702 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd4fc0009b0 con 0x7fd4f00067e0 2021-02-10T11:04:41.482 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:41.465 7fd5062bd700 1 -- 172.21.2.102:0/3905565702 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4fc00a310 con 0x7fd4f00067e0 2021-02-10T11:04:41.482 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:41.465 7fd5062bd700 1 -- 172.21.2.102:0/3905565702 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd4fc00a7a0 con 0x7fd4f00067e0 2021-02-10T11:04:41.482 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:41.465 7fd50d57b700 1 -- 172.21.2.102:0/3905565702 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd4f00067e0 msgr2=0x7fd4f0008c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:41.482 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:41.465 7fd50d57b700 1 --2- 172.21.2.102:0/3905565702 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd4f00067e0 0x7fd4f0008c80 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0x7fd4fc007990 tx=0x7fd4fc007d30).stop 2021-02-10T11:04:41.482 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:41.466 7fd50d57b700 1 -- 172.21.2.102:0/3905565702 shutdown_connections 2021-02-10T11:04:41.482 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:41.466 7fd50d57b700 1 --2- 172.21.2.102:0/3905565702 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd4f00067e0 0x7fd4f0008c80 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:41.482 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:41.466 7fd50d57b700 1 -- 172.21.2.102:0/3905565702 >> 172.21.2.102:0/3905565702 conn(0x7fd5080ce3e0 msgr2=0x7fd5080e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:04:41.482 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:41.466 7fd50d57b700 1 -- 172.21.2.102:0/3905565702 shutdown_connections 2021-02-10T11:04:41.482 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:41.466 7fd50d57b700 1 -- 172.21.2.102:0/3905565702 wait complete. 2021-02-10T11:04:41.482 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:41.481 7fd50d57b700 1 Processor -- start 2021-02-10T11:04:41.482 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:41.481 7fd50d57b700 1 -- start start 2021-02-10T11:04:41.484 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:41.481 7fd50d57b700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd50809b930 0x7fd5080b7c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:41.484 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:41.481 7fd50d57b700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd5080b8410 0x7fd5080b8810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:41.484 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:41.481 7fd50d57b700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd5080c1f80 0x7fd5080c2380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:41.484 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:41.481 7fd50d57b700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5080c1a00 con 0x7fd5080c1f80 2021-02-10T11:04:41.484 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:41.481 7fd50d57b700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5080b8e30 con 0x7fd5080b8410 2021-02-10T11:04:41.484 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:41.481 7fd50d57b700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5080b9050 con 0x7fd50809b930 2021-02-10T11:04:41.484 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:41.481 7fd507ac0700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd5080c1f80 0x7fd5080c2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:41.484 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:41.481 7fd507ac0700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd5080c1f80 0x7fd5080c2380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:50070/0 (socket says 172.21.2.102:50070) 2021-02-10T11:04:41.484 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:41.481 7fd507ac0700 1 -- 172.21.2.102:0/2747708119 learned_addr learned my addr 172.21.2.102:0/2747708119 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:04:41.485 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:41.482 7fd507ac0700 1 -- 172.21.2.102:0/2747708119 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd50809b930 msgr2=0x7fd5080b7c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:41.485 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:41.482 7fd507ac0700 1 --2- 172.21.2.102:0/2747708119 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd50809b930 0x7fd5080b7c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:41.485 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:41.482 7fd507ac0700 1 -- 172.21.2.102:0/2747708119 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd5080b8410 msgr2=0x7fd5080b8810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:41.485 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:41.482 7fd507ac0700 1 --2- 172.21.2.102:0/2747708119 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd5080b8410 0x7fd5080b8810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:41.485 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:41.482 7fd507ac0700 1 -- 172.21.2.102:0/2747708119 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5080c29a0 con 0x7fd5080c1f80 2021-02-10T11:04:41.485 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:41.482 7fd507ac0700 1 --2- 172.21.2.102:0/2747708119 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd5080c1f80 0x7fd5080c2380 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 rx=0x7fd5000099e0 tx=0x7fd50000bd20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:04:41.485 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:41.482 7fd4effff700 1 -- 172.21.2.102:0/2747708119 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd500001400 con 0x7fd5080c1f80 2021-02-10T11:04:41.485 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:41.482 7fd4effff700 1 -- 172.21.2.102:0/2747708119 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd50000be60 con 0x7fd5080c1f80 2021-02-10T11:04:41.485 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:41.482 7fd4effff700 1 -- 172.21.2.102:0/2747708119 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd500001700 con 0x7fd5080c1f80 2021-02-10T11:04:41.485 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:41.482 7fd50d57b700 1 -- 172.21.2.102:0/2747708119 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5080c29a0 con 0x7fd5080c1f80 2021-02-10T11:04:41.485 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:41.482 7fd50d57b700 1 -- 172.21.2.102:0/2747708119 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5080c2f30 con 0x7fd5080c1f80 2021-02-10T11:04:41.485 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:41.483 7fd4effff700 1 -- 172.21.2.102:0/2747708119 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd500001bd0 con 0x7fd5080c1f80 2021-02-10T11:04:41.486 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:41.483 7fd4effff700 1 --2- 172.21.2.102:0/2747708119 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd4e80186c0 0x7fd4e801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:41.486 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:41.483 7fd4effff700 1 -- 172.21.2.102:0/2747708119 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fd5000327d0 con 0x7fd5080c1f80 2021-02-10T11:04:41.486 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:41.483 7fd5072bf700 1 --2- 172.21.2.102:0/2747708119 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd4e80186c0 0x7fd4e801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:41.486 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:41.483 7fd5072bf700 1 --2- 172.21.2.102:0/2747708119 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd4e80186c0 0x7fd4e801abc0 crc :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:04:41.511 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:41.513 7fd50d57b700 1 -- 172.21.2.102:0/2747708119 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd508105b30 con 0x7fd5080c1f80 2021-02-10T11:04:41.513 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:41.516 7fd4effff700 1 -- 172.21.2.102:0/2747708119 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd5000019c0 con 0x7fd5080c1f80 2021-02-10T11:04:41.621 DEBUG:teuthology.orchestra.run.gibba002:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:04:41.623 DEBUG:teuthology.orchestra.run.gibba013:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:04:41.643 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:41.645 7fd50d57b700 1 -- 172.21.2.102:0/2747708119 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7fd5080f0b50 con 0x7fd5080c1f80 2021-02-10T11:04:41.643 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:41.645 7fd4effff700 1 -- 172.21.2.102:0/2747708119 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v20) v1 ==== 104+0+10 (secure 0 0 0) 0x7fd500001bd0 con 0x7fd5080c1f80 2021-02-10T11:04:41.644 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:04:41.646 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:41.648 7fd50d57b700 1 -- 172.21.2.102:0/2747708119 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd4e80186c0 msgr2=0x7fd4e801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:41.646 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:41.648 7fd50d57b700 1 --2- 172.21.2.102:0/2747708119 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd4e80186c0 0x7fd4e801abc0 crc :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:41.646 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:41.648 7fd50d57b700 1 -- 172.21.2.102:0/2747708119 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd5080c1f80 msgr2=0x7fd5080c2380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:41.646 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:41.648 7fd50d57b700 1 --2- 172.21.2.102:0/2747708119 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd5080c1f80 0x7fd5080c2380 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 rx=0x7fd5000099e0 tx=0x7fd50000bd20).stop 2021-02-10T11:04:41.646 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:41.648 7fd50d57b700 1 -- 172.21.2.102:0/2747708119 shutdown_connections 2021-02-10T11:04:41.646 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:41.648 7fd50d57b700 1 --2- 172.21.2.102:0/2747708119 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd5080c1f80 0x7fd5080c2380 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:41.646 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:41.648 7fd50d57b700 1 --2- 172.21.2.102:0/2747708119 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd4e80186c0 0x7fd4e801abc0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:41.646 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:41.648 7fd50d57b700 1 --2- 172.21.2.102:0/2747708119 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd5080b8410 0x7fd5080b8810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:41.647 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:41.648 7fd50d57b700 1 --2- 172.21.2.102:0/2747708119 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd50809b930 0x7fd5080b7c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:41.647 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:41.648 7fd50d57b700 1 -- 172.21.2.102:0/2747708119 >> 172.21.2.102:0/2747708119 conn(0x7fd508094e10 msgr2=0x7fd5080a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:04:41.647 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:41.648 7fd50d57b700 1 -- 172.21.2.102:0/2747708119 shutdown_connections 2021-02-10T11:04:41.647 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:41.648 7fd50d57b700 1 -- 172.21.2.102:0/2747708119 wait complete. 2021-02-10T11:04:41.654 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-02-10T11:04:42.052 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:42.038 7fcd54d22700 1 Processor -- start 2021-02-10T11:04:42.052 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:42.038 7fcd54d22700 1 -- start start 2021-02-10T11:04:42.052 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:42.038 7fcd54d22700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fcd501b0870 0x7fcd500f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:42.053 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:42.038 7fcd54d22700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcd500efe00 0x7fcd500f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:42.053 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:42.038 7fcd54d22700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd501b0c70 con 0x7fcd501b0470 2021-02-10T11:04:42.053 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:42.038 7fcd54d22700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd500f0b50 con 0x7fcd501b0870 2021-02-10T11:04:42.053 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:42.038 7fcd54d22700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd500f0820 con 0x7fcd500efe00 2021-02-10T11:04:42.053 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:42.039 7fcd4eb13700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fcd501b0470 0x7fcd500ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:42298/0 (socket says 172.21.2.102:42298) 2021-02-10T11:04:42.053 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:42.039 7fcd4eb13700 1 -- 172.21.2.102:0/2482833124 learned_addr learned my addr 172.21.2.102:0/2482833124 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:04:42.053 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:42.039 7fcd4e312700 1 --2- 172.21.2.102:0/2482833124 >> v2:172.21.2.102:3300/0 conn(0x7fcd501b0870 0x7fcd500f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:42.053 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:42.039 7fcd4db11700 1 -- 172.21.2.102:0/2482833124 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 776651685 0 0) 0x7fcd40000ce0 con 0x7fcd501b0470 2021-02-10T11:04:42.053 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:42.039 7fcd4f314700 1 --2- 172.21.2.102:0/2482833124 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcd500efe00 0x7fcd500f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:42.053 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:42.039 7fcd4db11700 1 -- 172.21.2.102:0/2482833124 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd3c002fe0 con 0x7fcd501b0470 2021-02-10T11:04:42.053 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:42.039 7fcd4e312700 1 -- 172.21.2.102:0/2482833124 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcd500efe00 msgr2=0x7fcd500f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:04:42.053 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:42.039 7fcd4e312700 1 --2- 172.21.2.102:0/2482833124 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcd500efe00 0x7fcd500f0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:42.054 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:42.039 7fcd4e312700 1 -- 172.21.2.102:0/2482833124 >> v1:172.21.2.102:6789/0 conn(0x7fcd501b0470 legacy=0x7fcd500ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:42.054 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:42.039 7fcd4e312700 1 -- 172.21.2.102:0/2482833124 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd500e52e0 con 0x7fcd501b0870 2021-02-10T11:04:42.054 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:42.039 7fcd4f314700 1 --2- 172.21.2.102:0/2482833124 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcd500efe00 0x7fcd500f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:04:42.054 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:42.039 7fcd4e312700 1 --2- 172.21.2.102:0/2482833124 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcd501b0870 0x7fcd500f20e0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 rx=0x7fcd38006650 tx=0x7fcd38009a80).ready entity=mon.0 client_cookie=80dbeb73491037f6 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:04:42.054 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:42.039 7fcd4db11700 1 -- 172.21.2.102:0/2482833124 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcd380009b0 con 0x7fcd501b0870 2021-02-10T11:04:42.054 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:42.039 7fcd4db11700 1 -- 172.21.2.102:0/2482833124 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd3800c0c0 con 0x7fcd501b0870 2021-02-10T11:04:42.054 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:42.039 7fcd4db11700 1 -- 172.21.2.102:0/2482833124 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcd3800c5e0 con 0x7fcd501b0870 2021-02-10T11:04:42.054 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:42.039 7fcd54d22700 1 -- 172.21.2.102:0/2482833124 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcd501b0870 msgr2=0x7fcd500f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:42.054 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:42.039 7fcd54d22700 1 --2- 172.21.2.102:0/2482833124 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcd501b0870 0x7fcd500f20e0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 rx=0x7fcd38006650 tx=0x7fcd38009a80).stop 2021-02-10T11:04:42.054 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:42.040 7fcd54d22700 1 -- 172.21.2.102:0/2482833124 shutdown_connections 2021-02-10T11:04:42.054 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:42.040 7fcd54d22700 1 --2- 172.21.2.102:0/2482833124 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcd501b0870 0x7fcd500f20e0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:42.054 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:42.040 7fcd54d22700 1 --2- 172.21.2.102:0/2482833124 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcd500efe00 0x7fcd500f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:42.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:42.040 7fcd54d22700 1 -- 172.21.2.102:0/2482833124 >> 172.21.2.102:0/2482833124 conn(0x7fcd500ce3e0 msgr2=0x7fcd500e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:04:42.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:42.040 7fcd54d22700 1 -- 172.21.2.102:0/2482833124 shutdown_connections 2021-02-10T11:04:42.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:42.040 7fcd54d22700 1 -- 172.21.2.102:0/2482833124 wait complete. 2021-02-10T11:04:42.056 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:42.055 7fcd54d22700 1 Processor -- start 2021-02-10T11:04:42.056 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:42.055 7fcd54d22700 1 -- start start 2021-02-10T11:04:42.056 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:42.055 7fcd54d22700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcd50105d10 0x7fcd5009ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:42.057 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:42.055 7fcd54d22700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcd50106110 0x7fcd500b2a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:42.057 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:42.055 7fcd54d22700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fcd5009e0a0 0x7fcd500b3040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:42.057 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:42.055 7fcd54d22700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd5009e640 con 0x7fcd50105d10 2021-02-10T11:04:42.057 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:42.055 7fcd54d22700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd501a1620 con 0x7fcd5009e0a0 2021-02-10T11:04:42.057 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:42.055 7fcd54d22700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd500b3a70 con 0x7fcd50106110 2021-02-10T11:04:42.057 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:42.055 7fcd4eb13700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcd50105d10 0x7fcd5009ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:42.057 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:42.055 7fcd4eb13700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcd50105d10 0x7fcd5009ebc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:50082/0 (socket says 172.21.2.102:50082) 2021-02-10T11:04:42.057 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:42.055 7fcd4eb13700 1 -- 172.21.2.102:0/2802170552 learned_addr learned my addr 172.21.2.102:0/2802170552 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:04:42.057 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:42.056 7fcd4f314700 1 --2- 172.21.2.102:0/2802170552 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fcd5009e0a0 0x7fcd500b3040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:42.057 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:42.056 7fcd4e312700 1 --2- 172.21.2.102:0/2802170552 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcd50106110 0x7fcd500b2a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:42.057 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:42.056 7fcd4eb13700 1 -- 172.21.2.102:0/2802170552 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcd50106110 msgr2=0x7fcd500b2a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:42.057 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:42.056 7fcd4eb13700 1 --2- 172.21.2.102:0/2802170552 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcd50106110 0x7fcd500b2a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:42.057 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:42.056 7fcd4eb13700 1 -- 172.21.2.102:0/2802170552 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fcd5009e0a0 msgr2=0x7fcd500b3040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:42.057 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:42.056 7fcd4eb13700 1 --2- 172.21.2.102:0/2802170552 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fcd5009e0a0 0x7fcd500b3040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:42.058 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:42.056 7fcd4eb13700 1 -- 172.21.2.102:0/2802170552 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd500b3660 con 0x7fcd50105d10 2021-02-10T11:04:42.058 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:42.056 7fcd4f314700 1 --2- 172.21.2.102:0/2802170552 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fcd5009e0a0 0x7fcd500b3040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:04:42.058 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:42.056 7fcd4e312700 1 --2- 172.21.2.102:0/2802170552 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcd50106110 0x7fcd500b2a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:04:42.058 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:42.056 7fcd4eb13700 1 --2- 172.21.2.102:0/2802170552 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcd50105d10 0x7fcd5009ebc0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7fcd4000b5c0 tx=0x7fcd4000b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:04:42.058 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:42.056 7fcd477fe700 1 -- 172.21.2.102:0/2802170552 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcd40006280 con 0x7fcd50105d10 2021-02-10T11:04:42.058 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:42.056 7fcd477fe700 1 -- 172.21.2.102:0/2802170552 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd400043a0 con 0x7fcd50105d10 2021-02-10T11:04:42.058 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:42.056 7fcd477fe700 1 -- 172.21.2.102:0/2802170552 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcd400048c0 con 0x7fcd50105d10 2021-02-10T11:04:42.058 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:42.056 7fcd54d22700 1 -- 172.21.2.102:0/2802170552 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd500b3660 con 0x7fcd50105d10 2021-02-10T11:04:42.058 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:42.056 7fcd54d22700 1 -- 172.21.2.102:0/2802170552 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd5009f4d0 con 0x7fcd50105d10 2021-02-10T11:04:42.058 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:42.057 7fcd477fe700 1 -- 172.21.2.102:0/2802170552 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fcd40004580 con 0x7fcd50105d10 2021-02-10T11:04:42.058 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:42.057 7fcd477fe700 1 --2- 172.21.2.102:0/2802170552 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcd340186c0 0x7fcd3401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:42.058 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:42.057 7fcd477fe700 1 -- 172.21.2.102:0/2802170552 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fcd40032590 con 0x7fcd50105d10 2021-02-10T11:04:42.059 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:42.057 7fcd4e312700 1 --2- 172.21.2.102:0/2802170552 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcd340186c0 0x7fcd3401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:42.059 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:42.057 7fcd4e312700 1 --2- 172.21.2.102:0/2802170552 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcd340186c0 0x7fcd3401abc0 crc :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:04:42.085 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:42.087 7fcd54d22700 1 -- 172.21.2.102:0/2802170552 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd50105ae0 con 0x7fcd50105d10 2021-02-10T11:04:42.088 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:42.090 7fcd477fe700 1 -- 172.21.2.102:0/2802170552 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fcd4002ff50 con 0x7fcd50105d10 2021-02-10T11:04:42.216 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:42.218 7fcd54d22700 1 -- 172.21.2.102:0/2802170552 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7fcd50105ae0 con 0x7fcd50105d10 2021-02-10T11:04:42.216 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:42.218 7fcd477fe700 1 -- 172.21.2.102:0/2802170552 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v20) v1 ==== 100+0+10 (secure 0 0 0) 0x7fcd4002ff50 con 0x7fcd50105d10 2021-02-10T11:04:42.216 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:04:42.218 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:42.220 7fcd54d22700 1 -- 172.21.2.102:0/2802170552 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcd340186c0 msgr2=0x7fcd3401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:42.218 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:42.220 7fcd54d22700 1 --2- 172.21.2.102:0/2802170552 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcd340186c0 0x7fcd3401abc0 crc :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:42.219 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:42.220 7fcd54d22700 1 -- 172.21.2.102:0/2802170552 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcd50105d10 msgr2=0x7fcd5009ebc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:42.219 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:42.220 7fcd54d22700 1 --2- 172.21.2.102:0/2802170552 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcd50105d10 0x7fcd5009ebc0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7fcd4000b5c0 tx=0x7fcd4000b960).stop 2021-02-10T11:04:42.219 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:42.220 7fcd54d22700 1 -- 172.21.2.102:0/2802170552 shutdown_connections 2021-02-10T11:04:42.219 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:42.220 7fcd54d22700 1 --2- 172.21.2.102:0/2802170552 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcd340186c0 0x7fcd3401abc0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:42.219 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:42.220 7fcd54d22700 1 --2- 172.21.2.102:0/2802170552 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fcd5009e0a0 0x7fcd500b3040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:42.219 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:42.220 7fcd54d22700 1 --2- 172.21.2.102:0/2802170552 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcd50106110 0x7fcd500b2a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:42.219 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:42.220 7fcd54d22700 1 --2- 172.21.2.102:0/2802170552 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcd50105d10 0x7fcd5009ebc0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:42.219 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:42.220 7fcd54d22700 1 -- 172.21.2.102:0/2802170552 >> 172.21.2.102:0/2802170552 conn(0x7fcd500c3c70 msgr2=0x7fcd500bd640 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:04:42.219 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:42.220 7fcd54d22700 1 -- 172.21.2.102:0/2802170552 shutdown_connections 2021-02-10T11:04:42.219 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:42.220 7fcd54d22700 1 -- 172.21.2.102:0/2802170552 wait complete. 2021-02-10T11:04:42.226 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:04:42.655 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:42.639 7f37db705700 1 Processor -- start 2021-02-10T11:04:42.655 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:42.639 7f37db705700 1 -- start start 2021-02-10T11:04:42.655 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:42.640 7f37db705700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f37d40e5470 0x7f37d407e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:42.655 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:42.640 7f37db705700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37d407ed80 con 0x7f37d407f300 2021-02-10T11:04:42.655 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:42.640 7f37db705700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37d41b4010 con 0x7f37d40e5070 2021-02-10T11:04:42.655 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:42.640 7f37db705700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37d40d1790 con 0x7f37d40e5470 2021-02-10T11:04:42.656 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:42.640 7f37d9d22700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f37d407f300 0x7f37d407f700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:42314/0 (socket says 172.21.2.102:42314) 2021-02-10T11:04:42.656 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:42.640 7f37d9d22700 1 -- 172.21.2.102:0/2677092122 learned_addr learned my addr 172.21.2.102:0/2677092122 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:04:42.656 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:42.640 7f37c7fff700 1 -- 172.21.2.102:0/2677092122 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 256367539 0 0) 0x7f37cc001500 con 0x7f37d407f300 2021-02-10T11:04:42.656 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:42.640 7f37c7fff700 1 -- 172.21.2.102:0/2677092122 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f37c0002fe0 con 0x7f37d407f300 2021-02-10T11:04:42.656 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:42.641 7f37d8d20700 1 --2- 172.21.2.102:0/2677092122 >> v2:172.21.2.113:3300/0 conn(0x7f37d40e5470 0x7f37d407e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:42.656 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:42.641 7f37c7fff700 1 -- 172.21.2.102:0/2677092122 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2490534234 0 0) 0x7f37cc001270 con 0x7f37d407f300 2021-02-10T11:04:42.656 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:42.641 7f37c7fff700 1 -- 172.21.2.102:0/2677092122 >> v2:172.21.2.113:3300/0 conn(0x7f37d40e5470 msgr2=0x7f37d407e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:04:42.656 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:42.641 7f37c7fff700 1 --2- 172.21.2.102:0/2677092122 >> v2:172.21.2.113:3300/0 conn(0x7f37d40e5470 0x7f37d407e6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:42.656 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:42.641 7f37c7fff700 1 -- 172.21.2.102:0/2677092122 >> v1:172.21.2.113:6789/0 conn(0x7f37d40e5070 legacy=0x7f37d41b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:42.656 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:42.641 7f37c7fff700 1 -- 172.21.2.102:0/2677092122 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37d40d19b0 con 0x7f37d407f300 2021-02-10T11:04:42.656 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:42.641 7f37c7fff700 1 -- 172.21.2.102:0/2677092122 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7f37cc001960 con 0x7f37d407f300 2021-02-10T11:04:42.657 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:42.641 7f37c7fff700 1 -- 172.21.2.102:0/2677092122 >> v1:172.21.2.102:6789/0 conn(0x7f37d407f300 legacy=0x7f37d407f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:42.657 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:42.641 7f37c7fff700 1 --2- 172.21.2.102:0/2677092122 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f37c0003b10 0x7f37c0005f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:42.657 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:42.641 7f37c7fff700 1 --2- 172.21.2.102:0/2677092122 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f37c00067a0 0x7f37c0008c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:42.657 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:42.641 7f37c7fff700 1 --2- 172.21.2.102:0/2677092122 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f37c0009580 0x7f37c000baa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:42.657 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:42.641 7f37c7fff700 1 -- 172.21.2.102:0/2677092122 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37c000c2f0 con 0x7f37c0009580 2021-02-10T11:04:42.657 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:42.641 7f37c7fff700 1 -- 172.21.2.102:0/2677092122 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37c000c510 con 0x7f37c0003b10 2021-02-10T11:04:42.657 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:42.641 7f37c7fff700 1 -- 172.21.2.102:0/2677092122 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37c000c730 con 0x7f37c00067a0 2021-02-10T11:04:42.657 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:42.641 7f37d9d22700 1 --2- 172.21.2.102:0/2677092122 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f37c0009580 0x7f37c000baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:42.657 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:42.641 7f37d9521700 1 --2- 172.21.2.102:0/2677092122 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f37c0003b10 0x7f37c0005f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:42.657 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:42.641 7f37d8d20700 1 --2- 172.21.2.102:0/2677092122 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f37c00067a0 0x7f37c0008c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:42.657 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:42.641 7f37d9d22700 1 -- 172.21.2.102:0/2677092122 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f37c00067a0 msgr2=0x7f37c0008c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:04:42.657 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:42.641 7f37d9d22700 1 --2- 172.21.2.102:0/2677092122 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f37c00067a0 0x7f37c0008c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:42.657 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:42.641 7f37d9d22700 1 -- 172.21.2.102:0/2677092122 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f37c0003b10 msgr2=0x7f37c0005f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:04:42.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:42.641 7f37d9d22700 1 --2- 172.21.2.102:0/2677092122 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f37c0003b10 0x7f37c0005f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:42.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:42.641 7f37d9d22700 1 -- 172.21.2.102:0/2677092122 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37c00509a0 con 0x7f37c0009580 2021-02-10T11:04:42.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:42.641 7f37d9d22700 1 -- 172.21.2.102:0/2677092122 reap_dead start 2021-02-10T11:04:42.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:42.642 7f37d9d22700 1 --2- 172.21.2.102:0/2677092122 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f37c0009580 0x7f37c000baa0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7f37cc001a90 tx=0x7f37cc00a310).ready entity=mon.0 client_cookie=ee8c22682a20f0e3 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:04:42.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:42.642 7f37c7fff700 1 -- 172.21.2.102:0/2677092122 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f37cc0070c0 con 0x7f37c0009580 2021-02-10T11:04:42.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:42.642 7f37c7fff700 1 -- 172.21.2.102:0/2677092122 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37cc0017d0 con 0x7f37c0009580 2021-02-10T11:04:42.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:42.642 7f37c7fff700 1 -- 172.21.2.102:0/2677092122 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f37cc004720 con 0x7f37c0009580 2021-02-10T11:04:42.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:42.642 7f37db705700 1 -- 172.21.2.102:0/2677092122 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f37c0009580 msgr2=0x7f37c000baa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:42.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:42.642 7f37db705700 1 --2- 172.21.2.102:0/2677092122 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f37c0009580 0x7f37c000baa0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7f37cc001a90 tx=0x7f37cc00a310).stop 2021-02-10T11:04:42.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:42.642 7f37db705700 1 -- 172.21.2.102:0/2677092122 shutdown_connections 2021-02-10T11:04:42.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:42.642 7f37db705700 1 --2- 172.21.2.102:0/2677092122 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f37c0009580 0x7f37c000baa0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:42.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:42.642 7f37db705700 1 -- 172.21.2.102:0/2677092122 >> 172.21.2.102:0/2677092122 conn(0x7f37d40ce690 msgr2=0x7f37d40e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:04:42.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:42.642 7f37db705700 1 -- 172.21.2.102:0/2677092122 shutdown_connections 2021-02-10T11:04:42.659 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:42.642 7f37db705700 1 -- 172.21.2.102:0/2677092122 wait complete. 2021-02-10T11:04:42.659 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:42.657 7f37db705700 1 Processor -- start 2021-02-10T11:04:42.659 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:42.657 7f37db705700 1 -- start start 2021-02-10T11:04:42.661 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:42.658 7f37db705700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f37d419f820 0x7f37d40863e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:42.661 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:42.658 7f37db705700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f37d4087120 0x7f37d4087520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:42.661 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:42.658 7f37db705700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f37d4087b40 0x7f37d4087f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:42.661 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:42.658 7f37db705700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37d4086ba0 con 0x7f37d4087120 2021-02-10T11:04:42.661 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:42.658 7f37db705700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37d408a750 con 0x7f37d419f820 2021-02-10T11:04:42.661 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:42.658 7f37db705700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37d408a930 con 0x7f37d4087b40 2021-02-10T11:04:42.661 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:42.658 7f37d8d20700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f37d4087120 0x7f37d4087520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:42.661 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:42.658 7f37d8d20700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f37d4087120 0x7f37d4087520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:50100/0 (socket says 172.21.2.102:50100) 2021-02-10T11:04:42.661 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:42.658 7f37d8d20700 1 -- 172.21.2.102:0/1790017781 learned_addr learned my addr 172.21.2.102:0/1790017781 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:04:42.661 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:42.658 7f37d9521700 1 --2- 172.21.2.102:0/1790017781 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f37d419f820 0x7f37d40863e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:42.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:42.658 7f37d9d22700 1 --2- 172.21.2.102:0/1790017781 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f37d4087b40 0x7f37d4087f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:42.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:42.658 7f37d8d20700 1 -- 172.21.2.102:0/1790017781 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f37d4087b40 msgr2=0x7f37d4087f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:42.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:42.658 7f37d8d20700 1 --2- 172.21.2.102:0/1790017781 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f37d4087b40 0x7f37d4087f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:42.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:42.658 7f37d8d20700 1 -- 172.21.2.102:0/1790017781 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f37d419f820 msgr2=0x7f37d40863e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:42.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:42.658 7f37d8d20700 1 --2- 172.21.2.102:0/1790017781 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f37d419f820 0x7f37d40863e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:42.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:42.658 7f37d8d20700 1 -- 172.21.2.102:0/1790017781 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37d408ab50 con 0x7f37d4087120 2021-02-10T11:04:42.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:42.659 7f37d8d20700 1 --2- 172.21.2.102:0/1790017781 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f37d4087120 0x7f37d4087520 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 rx=0x7f37c8006f60 tx=0x7f37c800b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:04:42.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:42.659 7f37c5ffb700 1 -- 172.21.2.102:0/1790017781 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f37c800bd60 con 0x7f37d4087120 2021-02-10T11:04:42.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:42.659 7f37c5ffb700 1 -- 172.21.2.102:0/1790017781 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37c8000c90 con 0x7f37d4087120 2021-02-10T11:04:42.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:42.659 7f37c5ffb700 1 -- 172.21.2.102:0/1790017781 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f37c80022b0 con 0x7f37d4087120 2021-02-10T11:04:42.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:42.659 7f37db705700 1 -- 172.21.2.102:0/1790017781 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37d408ab50 con 0x7f37d4087120 2021-02-10T11:04:42.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:42.659 7f37db705700 1 -- 172.21.2.102:0/1790017781 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37d408b0e0 con 0x7f37d4087120 2021-02-10T11:04:42.663 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:42.660 7f37c5ffb700 1 -- 172.21.2.102:0/1790017781 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f37c8006b00 con 0x7f37d4087120 2021-02-10T11:04:42.663 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:42.660 7f37c5ffb700 1 --2- 172.21.2.102:0/1790017781 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f37b401cad0 0x7f37b401eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:42.663 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:42.660 7f37c5ffb700 1 -- 172.21.2.102:0/1790017781 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f37c8032560 con 0x7f37d4087120 2021-02-10T11:04:42.663 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:42.660 7f37d9521700 1 --2- 172.21.2.102:0/1790017781 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f37b401cad0 0x7f37b401eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:42.663 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:42.660 7f37d9521700 1 --2- 172.21.2.102:0/1790017781 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f37b401cad0 0x7f37b401eff0 crc :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:04:42.687 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:42.689 7f37db705700 1 -- 172.21.2.102:0/1790017781 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37d4084b40 con 0x7f37d4087120 2021-02-10T11:04:42.690 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:42.692 7f37c5ffb700 1 -- 172.21.2.102:0/1790017781 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f37c8000940 con 0x7f37d4087120 2021-02-10T11:04:42.813 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:42.816 7f37db705700 1 -- 172.21.2.102:0/1790017781 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f37d407ed80 con 0x7f37d4087120 2021-02-10T11:04:42.814 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:42.816 7f37c5ffb700 1 -- 172.21.2.102:0/1790017781 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+13993 (secure 0 0 0) 0x7f37c8006b00 con 0x7f37d4087120 2021-02-10T11:04:42.814 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:04:42.814 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":20,"fsid":"68db30ea-dbce-4aef-a6df-f01c6fea0513","created":"2021-02-10 11:03:06.331557","modified":"2021-02-10 11:03:44.386945","last_up_change":"2021-02-10 11:03:32.690562","last_in_change":"2021-02-10 11:03:32.690562","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:03:37.322519","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:03:41.161563","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2021-02-10 11:03:42.379651","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"be691b4d-a039-408a-af80-9d53f02b6f1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6818","nonce":36088},{"type":"v1","addr":"172.21.2.102:6820","nonce":36088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6822","nonce":36088},{"type":"v1","addr":"172.21.2.102:6824","nonce":36088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6830","nonce":36088},{"type":"v1","addr":"172.21.2.102:6832","nonce":36088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6826","nonce":36088},{"type":"v1","addr":"172.21.2.102:6828","nonce":36088}]},"public_addr":"172.21.2.102:6820/36088","cluster_addr":"172.21.2.102:6824/36088","heartbeat_back_addr":"172.21.2.102:6832/36088","heartbeat_front_addr":"172.21.2.102:6828/36088","state":["exists","up"]},{"osd":1,"uuid":"64e3ccc8-9c69-4d7c-865e-af8e83822635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6819","nonce":36087},{"type":"v1","addr":"172.21.2.102:6821","nonce":36087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6823","nonce":36087},{"type":"v1","addr":"172.21.2.102:6825","nonce":36087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6831","nonce":36087},{"type":"v1","addr":"172.21.2.102:6833","nonce":36087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6827","nonce":36087},{"type":"v1","addr":"172.21.2.102:6829","nonce":36087}]},"public_addr":"172.21.2.102:6821/36087","cluster_addr":"172.21.2.102:6825/36087","heartbeat_back_addr":"172.21.2.102:6833/36087","heartbeat_front_addr":"172.21.2.102:6829/36087","state":["exists","up"]},{"osd":2,"uuid":"153700af-747f-481f-bdf4-d6072d3a842f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6810","nonce":36089},{"type":"v1","addr":"172.21.2.102:6811","nonce":36089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6812","nonce":36089},{"type":"v1","addr":"172.21.2.102:6813","nonce":36089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6816","nonce":36089},{"type":"v1","addr":"172.21.2.102:6817","nonce":36089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6814","nonce":36089},{"type":"v1","addr":"172.21.2.102:6815","nonce":36089}]},"public_addr":"172.21.2.102:6811/36089","cluster_addr":"172.21.2.102:6813/36089","heartbeat_back_addr":"172.21.2.102:6817/36089","heartbeat_front_addr":"172.21.2.102:6815/36089","state":["exists","up"]},{"osd":3,"uuid":"cbc678f4-83a9-45e6-8830-2f7c232b1ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6802","nonce":36090},{"type":"v1","addr":"172.21.2.102:6803","nonce":36090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6804","nonce":36090},{"type":"v1","addr":"172.21.2.102:6805","nonce":36090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6808","nonce":36090},{"type":"v1","addr":"172.21.2.102:6809","nonce":36090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6806","nonce":36090},{"type":"v1","addr":"172.21.2.102:6807","nonce":36090}]},"public_addr":"172.21.2.102:6803/36090","cluster_addr":"172.21.2.102:6805/36090","heartbeat_back_addr":"172.21.2.102:6809/36090","heartbeat_front_addr":"172.21.2.102:6807/36090","state":["exists","up"]},{"osd":4,"uuid":"c01e4008-b258-4f10-afd2-9e7f37214351","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6808","nonce":36368},{"type":"v1","addr":"172.21.2.113:6811","nonce":36368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6814","nonce":36368},{"type":"v1","addr":"172.21.2.113:6818","nonce":36368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6826","nonce":36368},{"type":"v1","addr":"172.21.2.113:6828","nonce":36368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6820","nonce":36368},{"type":"v1","addr":"172.21.2.113:6824","nonce":36368}]},"public_addr":"172.21.2.113:6811/36368","cluster_addr":"172.21.2.113:6818/36368","heartbeat_back_addr":"172.21.2.113:6828/36368","heartbeat_front_addr":"172.21.2.113:6824/36368","state":["exists","up"]},{"osd":5,"uuid":"551f6007-daf8-4a98-8ee0-c2b16bcccb5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6802","nonce":36366},{"type":"v1","addr":"172.21.2.113:6805","nonce":36366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6807","nonce":36366},{"type":"v1","addr":"172.21.2.113:6810","nonce":36366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6819","nonce":36366},{"type":"v1","addr":"172.21.2.113:6822","nonce":36366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6812","nonce":36366},{"type":"v1","addr":"172.21.2.113:6815","nonce":36366}]},"public_addr":"172.21.2.113:6805/36366","cluster_addr":"172.21.2.113:6810/36366","heartbeat_back_addr":"172.21.2.113:6822/36366","heartbeat_front_addr":"172.21.2.113:6815/36366","state":["exists","up"]},{"osd":6,"uuid":"bf67c040-0e50-48be-b102-c3943bc9ebef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6800","nonce":36367},{"type":"v1","addr":"172.21.2.113:6801","nonce":36367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6803","nonce":36367},{"type":"v1","addr":"172.21.2.113:6804","nonce":36367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6813","nonce":36367},{"type":"v1","addr":"172.21.2.113:6816","nonce":36367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6806","nonce":36367},{"type":"v1","addr":"172.21.2.113:6809","nonce":36367}]},"public_addr":"172.21.2.113:6801/36367","cluster_addr":"172.21.2.113:6804/36367","heartbeat_back_addr":"172.21.2.113:6816/36367","heartbeat_front_addr":"172.21.2.113:6809/36367","state":["exists","up"]},{"osd":7,"uuid":"f5461f80-018d-428f-be89-2eb2bcbe7bb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6817","nonce":36365},{"type":"v1","addr":"172.21.2.113:6821","nonce":36365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6823","nonce":36365},{"type":"v1","addr":"172.21.2.113:6825","nonce":36365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6830","nonce":36365},{"type":"v1","addr":"172.21.2.113:6831","nonce":36365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6827","nonce":36365},{"type":"v1","addr":"172.21.2.113:6829","nonce":36365}]},"public_addr":"172.21.2.113:6821/36365","cluster_addr":"172.21.2.113:6825/36365","heartbeat_back_addr":"172.21.2.113:6831/36365","heartbeat_front_addr":"172.21.2.113:6829/36365","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:04:42.816 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:42.818 7f37db705700 1 -- 172.21.2.102:0/1790017781 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f37b401cad0 msgr2=0x7f37b401eff0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:42.816 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:42.818 7f37db705700 1 --2- 172.21.2.102:0/1790017781 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f37b401cad0 0x7f37b401eff0 crc :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:42.816 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:42.818 7f37db705700 1 -- 172.21.2.102:0/1790017781 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f37d4087120 msgr2=0x7f37d4087520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:42.816 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:42.818 7f37db705700 1 --2- 172.21.2.102:0/1790017781 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f37d4087120 0x7f37d4087520 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 rx=0x7f37c8006f60 tx=0x7f37c800b970).stop 2021-02-10T11:04:42.816 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:42.818 7f37db705700 1 -- 172.21.2.102:0/1790017781 shutdown_connections 2021-02-10T11:04:42.816 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:42.818 7f37db705700 1 --2- 172.21.2.102:0/1790017781 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f37d4087b40 0x7f37d4087f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:42.816 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:42.818 7f37db705700 1 --2- 172.21.2.102:0/1790017781 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f37d4087120 0x7f37d4087520 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:42.817 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:42.818 7f37db705700 1 --2- 172.21.2.102:0/1790017781 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f37b401cad0 0x7f37b401eff0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:42.817 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:42.818 7f37db705700 1 --2- 172.21.2.102:0/1790017781 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f37d419f820 0x7f37d40863e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:42.817 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:42.818 7f37db705700 1 -- 172.21.2.102:0/1790017781 >> 172.21.2.102:0/1790017781 conn(0x7f37d40960d0 msgr2=0x7f37d4094980 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:04:42.817 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:42.818 7f37db705700 1 -- 172.21.2.102:0/1790017781 shutdown_connections 2021-02-10T11:04:42.817 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:42.819 7f37db705700 1 -- 172.21.2.102:0/1790017781 wait complete. 2021-02-10T11:04:42.825 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:03:37.322519', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2021-02-10 11:03:41.161563', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2021-02-10 11:03:42.379651', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-02-10T11:04:42.825 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:04:43.256 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:43.240 7fde99e90700 1 Processor -- start 2021-02-10T11:04:43.256 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:43.240 7fde99e90700 1 -- start start 2021-02-10T11:04:43.256 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:43.241 7fde99e90700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fde941b0870 0x7fde940f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:43.256 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:43.241 7fde99e90700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fde940efe00 0x7fde940f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:43.256 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:43.241 7fde99e90700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde941b0c70 con 0x7fde941b0470 2021-02-10T11:04:43.256 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:43.241 7fde99e90700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde940f0b50 con 0x7fde941b0870 2021-02-10T11:04:43.256 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:43.241 7fde99e90700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde940f0820 con 0x7fde940efe00 2021-02-10T11:04:43.256 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:43.241 7fde937fe700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fde941b0470 0x7fde940ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:42330/0 (socket says 172.21.2.102:42330) 2021-02-10T11:04:43.257 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:43.241 7fde937fe700 1 -- 172.21.2.102:0/2821275495 learned_addr learned my addr 172.21.2.102:0/2821275495 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:04:43.257 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:43.241 7fde93fff700 1 --2- 172.21.2.102:0/2821275495 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fde940efe00 0x7fde940f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:43.257 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:43.241 7fde92ffd700 1 --2- 172.21.2.102:0/2821275495 >> v2:172.21.2.113:3300/0 conn(0x7fde941b0870 0x7fde940f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:43.257 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:43.241 7fde927fc700 1 -- 172.21.2.102:0/2821275495 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2128686907 0 0) 0x7fde7c0010b0 con 0x7fde941b0470 2021-02-10T11:04:43.257 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:43.241 7fde927fc700 1 -- 172.21.2.102:0/2821275495 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde80002fe0 con 0x7fde941b0470 2021-02-10T11:04:43.257 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:43.242 7fde927fc700 1 -- 172.21.2.102:0/2821275495 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 279855021 0 0) 0x7fde7c0010b0 con 0x7fde941b0470 2021-02-10T11:04:43.257 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:43.242 7fde927fc700 1 -- 172.21.2.102:0/2821275495 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fde940efe00 msgr2=0x7fde940f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:04:43.257 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:43.242 7fde927fc700 1 --2- 172.21.2.102:0/2821275495 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fde940efe00 0x7fde940f0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:43.257 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:43.242 7fde927fc700 1 -- 172.21.2.102:0/2821275495 >> v2:172.21.2.113:3300/0 conn(0x7fde941b0870 msgr2=0x7fde940f20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:04:43.257 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:43.242 7fde927fc700 1 --2- 172.21.2.102:0/2821275495 >> v2:172.21.2.113:3300/0 conn(0x7fde941b0870 0x7fde940f20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:43.257 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:43.242 7fde927fc700 1 -- 172.21.2.102:0/2821275495 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde940e52e0 con 0x7fde941b0470 2021-02-10T11:04:43.257 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:43.242 7fde927fc700 1 -- 172.21.2.102:0/2821275495 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7fde7c0015b0 con 0x7fde941b0470 2021-02-10T11:04:43.257 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:43.242 7fde92ffd700 1 --2- 172.21.2.102:0/2821275495 >> v2:172.21.2.113:3300/0 conn(0x7fde941b0870 0x7fde940f20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:04:43.258 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:43.242 7fde93fff700 1 --2- 172.21.2.102:0/2821275495 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fde940efe00 0x7fde940f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:04:43.258 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:43.242 7fde927fc700 1 -- 172.21.2.102:0/2821275495 >> v1:172.21.2.102:6789/0 conn(0x7fde941b0470 legacy=0x7fde940ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:43.258 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:43.242 7fde927fc700 1 --2- 172.21.2.102:0/2821275495 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fde80047aa0 0x7fde80049f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:43.258 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:43.242 7fde927fc700 1 --2- 172.21.2.102:0/2821275495 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fde8004a810 0x7fde8004cd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:43.258 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:43.242 7fde927fc700 1 --2- 172.21.2.102:0/2821275495 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fde8004d520 0x7fde8004fa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:43.258 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:43.242 7fde927fc700 1 -- 172.21.2.102:0/2821275495 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde800502f0 con 0x7fde80047aa0 2021-02-10T11:04:43.258 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:43.242 7fde927fc700 1 -- 172.21.2.102:0/2821275495 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde80050510 con 0x7fde8004d520 2021-02-10T11:04:43.258 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:43.242 7fde927fc700 1 -- 172.21.2.102:0/2821275495 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde80050730 con 0x7fde8004a810 2021-02-10T11:04:43.258 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:43.242 7fde937fe700 1 --2- 172.21.2.102:0/2821275495 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fde80047aa0 0x7fde80049f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:43.258 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:43.242 7fde92ffd700 1 --2- 172.21.2.102:0/2821275495 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fde8004a810 0x7fde8004cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:43.258 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:43.242 7fde93fff700 1 --2- 172.21.2.102:0/2821275495 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fde8004d520 0x7fde8004fa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:43.258 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:43.242 7fde937fe700 1 -- 172.21.2.102:0/2821275495 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fde8004a810 msgr2=0x7fde8004cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:04:43.258 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:43.242 7fde937fe700 1 --2- 172.21.2.102:0/2821275495 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fde8004a810 0x7fde8004cd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:43.259 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:43.242 7fde937fe700 1 -- 172.21.2.102:0/2821275495 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fde8004d520 msgr2=0x7fde8004fa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:04:43.259 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:43.242 7fde937fe700 1 --2- 172.21.2.102:0/2821275495 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fde8004d520 0x7fde8004fa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:43.259 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:43.242 7fde937fe700 1 -- 172.21.2.102:0/2821275495 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde800509d0 con 0x7fde80047aa0 2021-02-10T11:04:43.259 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:43.242 7fde93fff700 1 --2- 172.21.2.102:0/2821275495 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fde8004d520 0x7fde8004fa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:04:43.259 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:43.242 7fde937fe700 1 --2- 172.21.2.102:0/2821275495 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fde80047aa0 0x7fde80049f70 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 rx=0x7fde7c0017d0 tx=0x7fde7c007cb0).ready entity=mon.0 client_cookie=a07b325e6e655af9 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:04:43.259 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:43.243 7fde927fc700 1 -- 172.21.2.102:0/2821275495 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fde7c0013a0 con 0x7fde80047aa0 2021-02-10T11:04:43.259 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:43.243 7fde927fc700 1 -- 172.21.2.102:0/2821275495 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde7c00a310 con 0x7fde80047aa0 2021-02-10T11:04:43.259 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:43.243 7fde927fc700 1 -- 172.21.2.102:0/2821275495 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fde7c00a830 con 0x7fde80047aa0 2021-02-10T11:04:43.259 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:43.243 7fde99e90700 1 -- 172.21.2.102:0/2821275495 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fde80047aa0 msgr2=0x7fde80049f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:43.259 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:43.243 7fde99e90700 1 --2- 172.21.2.102:0/2821275495 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fde80047aa0 0x7fde80049f70 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 rx=0x7fde7c0017d0 tx=0x7fde7c007cb0).stop 2021-02-10T11:04:43.259 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:43.243 7fde93fff700 1 -- 172.21.2.102:0/2821275495 reap_dead start 2021-02-10T11:04:43.259 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:43.243 7fde99e90700 1 -- 172.21.2.102:0/2821275495 shutdown_connections 2021-02-10T11:04:43.260 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:43.243 7fde99e90700 1 -- 172.21.2.102:0/2821275495 >> 172.21.2.102:0/2821275495 conn(0x7fde940ce3e0 msgr2=0x7fde940e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:04:43.260 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:43.243 7fde99e90700 1 -- 172.21.2.102:0/2821275495 shutdown_connections 2021-02-10T11:04:43.260 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:43.243 7fde99e90700 1 -- 172.21.2.102:0/2821275495 wait complete. 2021-02-10T11:04:43.260 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:43.258 7fde99e90700 1 Processor -- start 2021-02-10T11:04:43.260 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:43.258 7fde99e90700 1 -- start start 2021-02-10T11:04:43.261 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:43.258 7fde99e90700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fde9409b930 0x7fde940b7c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:43.261 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:43.258 7fde99e90700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fde940b8410 0x7fde940b8810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:43.261 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:43.258 7fde99e90700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fde940c1f80 0x7fde940c2380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:43.262 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:43.258 7fde99e90700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde940c1a00 con 0x7fde940c1f80 2021-02-10T11:04:43.262 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:43.258 7fde99e90700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde940b8e30 con 0x7fde9409b930 2021-02-10T11:04:43.262 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:43.258 7fde99e90700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde940b9050 con 0x7fde940b8410 2021-02-10T11:04:43.262 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:43.259 7fde93fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fde940c1f80 0x7fde940c2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:43.262 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:43.259 7fde93fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fde940c1f80 0x7fde940c2380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:50120/0 (socket says 172.21.2.102:50120) 2021-02-10T11:04:43.262 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:43.259 7fde93fff700 1 -- 172.21.2.102:0/1547110745 learned_addr learned my addr 172.21.2.102:0/1547110745 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:04:43.262 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:43.259 7fde93fff700 1 -- 172.21.2.102:0/1547110745 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fde940b8410 msgr2=0x7fde940b8810 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:04:43.262 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:43.259 7fde93fff700 1 --2- 172.21.2.102:0/1547110745 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fde940b8410 0x7fde940b8810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:43.262 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:43.259 7fde93fff700 1 -- 172.21.2.102:0/1547110745 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fde9409b930 msgr2=0x7fde940b7c20 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:04:43.262 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:43.259 7fde93fff700 1 --2- 172.21.2.102:0/1547110745 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fde9409b930 0x7fde940b7c20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:43.263 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:43.259 7fde93fff700 1 -- 172.21.2.102:0/1547110745 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde940c29a0 con 0x7fde940c1f80 2021-02-10T11:04:43.263 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:43.259 7fde93fff700 1 --2- 172.21.2.102:0/1547110745 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fde940c1f80 0x7fde940c2380 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7fde8c0099a0 tx=0x7fde8c00bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:04:43.263 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:43.259 7fde8bfff700 1 -- 172.21.2.102:0/1547110745 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fde8c0011f0 con 0x7fde940c1f80 2021-02-10T11:04:43.263 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:43.259 7fde8bfff700 1 -- 172.21.2.102:0/1547110745 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde8c003030 con 0x7fde940c1f80 2021-02-10T11:04:43.263 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:43.259 7fde8bfff700 1 -- 172.21.2.102:0/1547110745 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fde8c0016e0 con 0x7fde940c1f80 2021-02-10T11:04:43.263 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:43.259 7fde99e90700 1 -- 172.21.2.102:0/1547110745 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde940c29a0 con 0x7fde940c1f80 2021-02-10T11:04:43.263 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:43.259 7fde99e90700 1 -- 172.21.2.102:0/1547110745 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde940c2f30 con 0x7fde940c1f80 2021-02-10T11:04:43.263 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:43.260 7fde8bfff700 1 -- 172.21.2.102:0/1547110745 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fde8c0056b0 con 0x7fde940c1f80 2021-02-10T11:04:43.263 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:43.260 7fde8bfff700 1 --2- 172.21.2.102:0/1547110745 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fde780186c0 0x7fde7801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:43.263 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:43.260 7fde8bfff700 1 -- 172.21.2.102:0/1547110745 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fde8c0328e0 con 0x7fde940c1f80 2021-02-10T11:04:43.263 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:43.260 7fde937fe700 1 --2- 172.21.2.102:0/1547110745 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fde780186c0 0x7fde7801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:43.263 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:43.260 7fde937fe700 1 --2- 172.21.2.102:0/1547110745 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fde780186c0 0x7fde7801abc0 crc :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:04:43.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:43.290 7fde99e90700 1 -- 172.21.2.102:0/1547110745 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde94105b30 con 0x7fde940c1f80 2021-02-10T11:04:43.290 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:43.293 7fde8bfff700 1 -- 172.21.2.102:0/1547110745 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fde8c006ca0 con 0x7fde940c1f80 2021-02-10T11:04:43.420 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:43.422 7fde99e90700 1 -- 172.21.2.102:0/1547110745 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7fde940f0b50 con 0x7fde940c1f80 2021-02-10T11:04:43.420 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:43.422 7fde8bfff700 1 -- 172.21.2.102:0/1547110745 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v20) v1 ==== 92+0+10 (secure 0 0 0) 0x7fde8c006eb0 con 0x7fde940c1f80 2021-02-10T11:04:43.421 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:04:43.422 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:43.424 7fde99e90700 1 -- 172.21.2.102:0/1547110745 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fde780186c0 msgr2=0x7fde7801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:43.422 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:43.424 7fde99e90700 1 --2- 172.21.2.102:0/1547110745 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fde780186c0 0x7fde7801abc0 crc :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:43.422 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:43.424 7fde99e90700 1 -- 172.21.2.102:0/1547110745 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fde940c1f80 msgr2=0x7fde940c2380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:43.422 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:43.424 7fde99e90700 1 --2- 172.21.2.102:0/1547110745 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fde940c1f80 0x7fde940c2380 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7fde8c0099a0 tx=0x7fde8c00bcf0).stop 2021-02-10T11:04:43.422 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:43.424 7fde99e90700 1 -- 172.21.2.102:0/1547110745 shutdown_connections 2021-02-10T11:04:43.423 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:43.424 7fde99e90700 1 --2- 172.21.2.102:0/1547110745 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fde940c1f80 0x7fde940c2380 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:43.423 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:43.424 7fde99e90700 1 --2- 172.21.2.102:0/1547110745 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fde940b8410 0x7fde940b8810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:43.423 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:43.424 7fde99e90700 1 --2- 172.21.2.102:0/1547110745 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fde780186c0 0x7fde7801abc0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:43.423 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:43.424 7fde99e90700 1 --2- 172.21.2.102:0/1547110745 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fde9409b930 0x7fde940b7c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:43.423 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:43.424 7fde99e90700 1 -- 172.21.2.102:0/1547110745 >> 172.21.2.102:0/1547110745 conn(0x7fde94094e10 msgr2=0x7fde940a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:04:43.423 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:43.425 7fde99e90700 1 -- 172.21.2.102:0/1547110745 shutdown_connections 2021-02-10T11:04:43.423 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:43.425 7fde99e90700 1 -- 172.21.2.102:0/1547110745 wait complete. 2021-02-10T11:04:43.431 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-02-10T11:04:43.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:43.847 7f3b2a25a700 1 Processor -- start 2021-02-10T11:04:43.862 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:43.847 7f3b2a25a700 1 -- start start 2021-02-10T11:04:43.862 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:43.848 7f3b2a25a700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f3b241b0470 0x7f3b241b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:43.862 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:43.848 7f3b2a25a700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f3b240f20e0 0x7f3b240f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:43.862 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:43.848 7f3b2a25a700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b240f0b50 con 0x7f3b240ef610 2021-02-10T11:04:43.862 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:43.848 7f3b2a25a700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b240f0820 con 0x7f3b240f20e0 2021-02-10T11:04:43.862 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:43.848 7f3b2a25a700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b240f24e0 con 0x7f3b241b0470 2021-02-10T11:04:43.862 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:43.848 7f3b237fe700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f3b240ef610 0x7f3b240efa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:42348/0 (socket says 172.21.2.102:42348) 2021-02-10T11:04:43.862 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:43.848 7f3b237fe700 1 -- 172.21.2.102:0/1894530541 learned_addr learned my addr 172.21.2.102:0/1894530541 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:04:43.862 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:43.848 7f3b28877700 1 --2- 172.21.2.102:0/1894530541 >> v2:172.21.2.102:3300/0 conn(0x7f3b240f20e0 0x7f3b240f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:43.862 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:43.848 7f3b28877700 1 -- 172.21.2.102:0/1894530541 >> v2:172.21.2.113:3300/0 conn(0x7f3b241b0470 msgr2=0x7f3b241b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:04:43.863 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:43.848 7f3b23fff700 1 --2- 172.21.2.102:0/1894530541 >> v2:172.21.2.113:3300/0 conn(0x7f3b241b0470 0x7f3b241b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:43.863 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:43.848 7f3b28877700 1 --2- 172.21.2.102:0/1894530541 >> v2:172.21.2.113:3300/0 conn(0x7f3b241b0470 0x7f3b241b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:43.863 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:43.848 7f3b22ffd700 1 -- 172.21.2.102:0/1894530541 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 282471387 0 0) 0x7f3b180010b0 con 0x7f3b240ef610 2021-02-10T11:04:43.863 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:43.848 7f3b28877700 1 -- 172.21.2.102:0/1894530541 >> v1:172.21.2.102:6789/0 conn(0x7f3b240ef610 legacy=0x7f3b240efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:43.863 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:43.848 7f3b28877700 1 -- 172.21.2.102:0/1894530541 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b240e52e0 con 0x7f3b240f20e0 2021-02-10T11:04:43.863 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:43.848 7f3b23fff700 1 --2- 172.21.2.102:0/1894530541 >> v2:172.21.2.113:3300/0 conn(0x7f3b241b0470 0x7f3b241b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:04:43.863 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:43.849 7f3b28877700 1 --2- 172.21.2.102:0/1894530541 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3b240f20e0 0x7f3b240f0200 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7f3b1c006e30 tx=0x7f3b1c009e60).ready entity=mon.0 client_cookie=7224ad1a4330567d server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:04:43.863 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:43.849 7f3b22ffd700 1 -- 172.21.2.102:0/1894530541 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3b1c006a50 con 0x7f3b240f20e0 2021-02-10T11:04:43.863 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:43.849 7f3b22ffd700 1 -- 172.21.2.102:0/1894530541 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b1c00b300 con 0x7f3b240f20e0 2021-02-10T11:04:43.863 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:43.849 7f3b22ffd700 1 -- 172.21.2.102:0/1894530541 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3b1c00b820 con 0x7f3b240f20e0 2021-02-10T11:04:43.863 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:43.849 7f3b2a25a700 1 -- 172.21.2.102:0/1894530541 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3b240f20e0 msgr2=0x7f3b240f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:43.863 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:43.849 7f3b2a25a700 1 --2- 172.21.2.102:0/1894530541 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3b240f20e0 0x7f3b240f0200 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7f3b1c006e30 tx=0x7f3b1c009e60).stop 2021-02-10T11:04:43.864 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:43.849 7f3b2a25a700 1 -- 172.21.2.102:0/1894530541 shutdown_connections 2021-02-10T11:04:43.864 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:43.849 7f3b2a25a700 1 --2- 172.21.2.102:0/1894530541 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3b240f20e0 0x7f3b240f0200 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:43.864 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:43.849 7f3b2a25a700 1 --2- 172.21.2.102:0/1894530541 >> v2:172.21.2.113:3300/0 conn(0x7f3b241b0470 0x7f3b241b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:43.864 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:43.849 7f3b2a25a700 1 -- 172.21.2.102:0/1894530541 >> 172.21.2.102:0/1894530541 conn(0x7f3b240ce3e0 msgr2=0x7f3b240e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:04:43.864 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:43.850 7f3b2a25a700 1 -- 172.21.2.102:0/1894530541 shutdown_connections 2021-02-10T11:04:43.864 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:43.850 7f3b2a25a700 1 -- 172.21.2.102:0/1894530541 wait complete. 2021-02-10T11:04:43.865 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:43.864 7f3b2a25a700 1 Processor -- start 2021-02-10T11:04:43.865 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:43.864 7f3b2a25a700 1 -- start start 2021-02-10T11:04:43.865 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:43.865 7f3b2a25a700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3b24190920 0x7f3b240b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:43.866 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:43.865 7f3b2a25a700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3b240b8200 0x7f3b240b8600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:43.866 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:43.865 7f3b2a25a700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3b240c1d80 0x7f3b240c2180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:43.866 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:43.865 7f3b2a25a700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b24190d20 con 0x7f3b24190920 2021-02-10T11:04:43.866 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:43.865 7f3b2a25a700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b240b8c20 con 0x7f3b240b8200 2021-02-10T11:04:43.866 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:43.865 7f3b2a25a700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b240c27a0 con 0x7f3b240c1d80 2021-02-10T11:04:43.866 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:43.865 7f3b23fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3b24190920 0x7f3b240b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:43.866 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:43.865 7f3b23fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3b24190920 0x7f3b240b7be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:50128/0 (socket says 172.21.2.102:50128) 2021-02-10T11:04:43.866 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:43.865 7f3b23fff700 1 -- 172.21.2.102:0/611136024 learned_addr learned my addr 172.21.2.102:0/611136024 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:04:43.866 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:43.865 7f3b23fff700 1 -- 172.21.2.102:0/611136024 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3b240c1d80 msgr2=0x7f3b240c2180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:43.866 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:43.865 7f3b23fff700 1 --2- 172.21.2.102:0/611136024 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3b240c1d80 0x7f3b240c2180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:43.866 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:43.865 7f3b23fff700 1 -- 172.21.2.102:0/611136024 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3b240b8200 msgr2=0x7f3b240b8600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:43.866 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:43.865 7f3b23fff700 1 --2- 172.21.2.102:0/611136024 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3b240b8200 0x7f3b240b8600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:43.866 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:43.865 7f3b23fff700 1 -- 172.21.2.102:0/611136024 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b240c29c0 con 0x7f3b24190920 2021-02-10T11:04:43.867 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:43.865 7f3b23fff700 1 --2- 172.21.2.102:0/611136024 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3b24190920 0x7f3b240b7be0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 rx=0x7f3b1400b580 tx=0x7f3b1400b920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:04:43.867 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:43.865 7f3b20ff9700 1 -- 172.21.2.102:0/611136024 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3b1400bd10 con 0x7f3b24190920 2021-02-10T11:04:43.867 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:43.865 7f3b20ff9700 1 -- 172.21.2.102:0/611136024 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b14000c90 con 0x7f3b24190920 2021-02-10T11:04:43.867 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:43.865 7f3b2a25a700 1 -- 172.21.2.102:0/611136024 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b240c29c0 con 0x7f3b24190920 2021-02-10T11:04:43.867 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:43.865 7f3b2a25a700 1 -- 172.21.2.102:0/611136024 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b240c29c0 con 0x7f3b24190920 2021-02-10T11:04:43.867 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:43.865 7f3b20ff9700 1 -- 172.21.2.102:0/611136024 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3b140022b0 con 0x7f3b24190920 2021-02-10T11:04:43.867 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:43.866 7f3b20ff9700 1 -- 172.21.2.102:0/611136024 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f3b1400a000 con 0x7f3b24190920 2021-02-10T11:04:43.867 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:43.866 7f3b20ff9700 1 --2- 172.21.2.102:0/611136024 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3b04018730 0x7f3b0401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:43.867 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:43.866 7f3b20ff9700 1 -- 172.21.2.102:0/611136024 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f3b14032630 con 0x7f3b24190920 2021-02-10T11:04:43.867 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:43.866 7f3b237fe700 1 --2- 172.21.2.102:0/611136024 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3b04018730 0x7f3b0401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:43.867 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:43.866 7f3b237fe700 1 --2- 172.21.2.102:0/611136024 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3b04018730 0x7f3b0401ac10 crc :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:04:43.895 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:43.897 7f3b2a25a700 1 -- 172.21.2.102:0/611136024 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b24105ae0 con 0x7f3b24190920 2021-02-10T11:04:43.897 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:43.900 7f3b20ff9700 1 -- 172.21.2.102:0/611136024 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f3b14006bb0 con 0x7f3b24190920 2021-02-10T11:04:44.025 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:44.027 7f3b2a25a700 1 -- 172.21.2.102:0/611136024 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7f3b24105ae0 con 0x7f3b24190920 2021-02-10T11:04:44.026 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:44.028 7f3b20ff9700 1 -- 172.21.2.102:0/611136024 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v20) v1 ==== 104+0+10 (secure 0 0 0) 0x7f3b14000c90 con 0x7f3b24190920 2021-02-10T11:04:44.026 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:04:44.028 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:44.030 7f3b2a25a700 1 -- 172.21.2.102:0/611136024 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3b04018730 msgr2=0x7f3b0401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:44.028 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:44.030 7f3b2a25a700 1 --2- 172.21.2.102:0/611136024 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3b04018730 0x7f3b0401ac10 crc :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:44.028 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:44.030 7f3b2a25a700 1 -- 172.21.2.102:0/611136024 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3b24190920 msgr2=0x7f3b240b7be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:44.029 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:44.030 7f3b2a25a700 1 --2- 172.21.2.102:0/611136024 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3b24190920 0x7f3b240b7be0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 rx=0x7f3b1400b580 tx=0x7f3b1400b920).stop 2021-02-10T11:04:44.029 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:44.030 7f3b2a25a700 1 -- 172.21.2.102:0/611136024 shutdown_connections 2021-02-10T11:04:44.029 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:44.030 7f3b2a25a700 1 --2- 172.21.2.102:0/611136024 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3b240c1d80 0x7f3b240c2180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:44.029 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:44.030 7f3b2a25a700 1 --2- 172.21.2.102:0/611136024 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3b04018730 0x7f3b0401ac10 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:44.029 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:44.030 7f3b2a25a700 1 --2- 172.21.2.102:0/611136024 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3b240b8200 0x7f3b240b8600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:44.029 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:44.030 7f3b2a25a700 1 --2- 172.21.2.102:0/611136024 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3b24190920 0x7f3b240b7be0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:44.029 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:44.030 7f3b2a25a700 1 -- 172.21.2.102:0/611136024 >> 172.21.2.102:0/611136024 conn(0x7f3b24094e10 msgr2=0x7f3b240a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:04:44.029 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:44.030 7f3b2a25a700 1 -- 172.21.2.102:0/611136024 shutdown_connections 2021-02-10T11:04:44.029 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:44.030 7f3b2a25a700 1 -- 172.21.2.102:0/611136024 wait complete. 2021-02-10T11:04:44.036 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-02-10T11:04:44.465 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:44.450 7f0b8013f700 1 Processor -- start 2021-02-10T11:04:44.465 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:44.450 7f0b8013f700 1 -- start start 2021-02-10T11:04:44.465 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:44.450 7f0b8013f700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f0b781b0470 0x7f0b781b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:44.465 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:44.450 7f0b8013f700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0b780f20e0 0x7f0b780f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:44.466 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:44.450 7f0b8013f700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b780f0b50 con 0x7f0b780ef610 2021-02-10T11:04:44.466 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:44.450 7f0b8013f700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b780f0820 con 0x7f0b781b0470 2021-02-10T11:04:44.466 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:44.450 7f0b8013f700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b780f24e0 con 0x7f0b780f20e0 2021-02-10T11:04:44.466 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:44.451 7f0b7df5b700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f0b781b0470 0x7f0b781b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:44.466 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:44.451 7f0b7df5b700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f0b781b0470 0x7f0b781b0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:50134/0 (socket says 172.21.2.102:50134) 2021-02-10T11:04:44.466 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:44.451 7f0b7df5b700 1 -- 172.21.2.102:0/843411693 learned_addr learned my addr 172.21.2.102:0/843411693 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:04:44.466 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:44.451 7f0b7df5b700 1 -- 172.21.2.102:0/843411693 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0b780f20e0 msgr2=0x7f0b780f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:04:44.466 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:44.451 7f0b7df5b700 1 --2- 172.21.2.102:0/843411693 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0b780f20e0 0x7f0b780f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:44.467 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:44.451 7f0b7df5b700 1 -- 172.21.2.102:0/843411693 >> v1:172.21.2.113:6789/0 conn(0x7f0b780ef610 legacy=0x7f0b780efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:04:44.467 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:44.451 7f0b7df5b700 1 -- 172.21.2.102:0/843411693 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b780e52e0 con 0x7f0b781b0470 2021-02-10T11:04:44.467 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:44.451 7f0b7df5b700 1 --2- 172.21.2.102:0/843411693 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0b781b0470 0x7f0b781b0870 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0x7f0b74006700 tx=0x7f0b74009b80).ready entity=mon.0 client_cookie=4da7e0af5a82177c server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:04:44.467 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:44.451 7f0b7cf59700 1 -- 172.21.2.102:0/843411693 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0b7400c0c0 con 0x7f0b781b0470 2021-02-10T11:04:44.467 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:44.451 7f0b7cf59700 1 -- 172.21.2.102:0/843411693 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b7400c400 con 0x7f0b781b0470 2021-02-10T11:04:44.467 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:44.451 7f0b7cf59700 1 -- 172.21.2.102:0/843411693 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0b7400c8e0 con 0x7f0b781b0470 2021-02-10T11:04:44.467 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:44.451 7f0b8013f700 1 -- 172.21.2.102:0/843411693 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0b781b0470 msgr2=0x7f0b781b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:44.467 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:44.451 7f0b8013f700 1 --2- 172.21.2.102:0/843411693 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0b781b0470 0x7f0b781b0870 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0x7f0b74006700 tx=0x7f0b74009b80).stop 2021-02-10T11:04:44.467 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:44.452 7f0b8013f700 1 -- 172.21.2.102:0/843411693 shutdown_connections 2021-02-10T11:04:44.467 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:44.452 7f0b8013f700 1 --2- 172.21.2.102:0/843411693 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0b780f20e0 0x7f0b780f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:44.467 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:44.452 7f0b8013f700 1 --2- 172.21.2.102:0/843411693 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0b781b0470 0x7f0b781b0870 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:44.467 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:44.452 7f0b8013f700 1 -- 172.21.2.102:0/843411693 >> 172.21.2.102:0/843411693 conn(0x7f0b780ce3e0 msgr2=0x7f0b780e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:04:44.467 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:44.452 7f0b8013f700 1 -- 172.21.2.102:0/843411693 shutdown_connections 2021-02-10T11:04:44.467 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:44.452 7f0b8013f700 1 -- 172.21.2.102:0/843411693 wait complete. 2021-02-10T11:04:44.468 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:44.467 7f0b8013f700 1 Processor -- start 2021-02-10T11:04:44.468 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:44.467 7f0b8013f700 1 -- start start 2021-02-10T11:04:44.469 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:44.467 7f0b8013f700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0b780c36a0 0x7f0b780c2b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:44.469 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:44.467 7f0b8013f700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0b780c3190 0x7f0b780b1d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:44.469 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:44.467 7f0b8013f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0b780b2370 0x7f0b780b2d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:44.469 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:44.467 7f0b8013f700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b780b27c0 con 0x7f0b780b2370 2021-02-10T11:04:44.469 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:44.467 7f0b8013f700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b780b3360 con 0x7f0b780c3190 2021-02-10T11:04:44.469 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:44.467 7f0b8013f700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b780b3540 con 0x7f0b780c36a0 2021-02-10T11:04:44.469 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:44.467 7f0b7e75c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0b780b2370 0x7f0b780b2d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:44.469 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:44.468 7f0b7e75c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0b780b2370 0x7f0b780b2d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:50138/0 (socket says 172.21.2.102:50138) 2021-02-10T11:04:44.469 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:44.468 7f0b7e75c700 1 -- 172.21.2.102:0/2417480969 learned_addr learned my addr 172.21.2.102:0/2417480969 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:04:44.470 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:44.468 7f0b7e75c700 1 -- 172.21.2.102:0/2417480969 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0b780c36a0 msgr2=0x7f0b780c2b70 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:04:44.470 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:44.468 7f0b7e75c700 1 --2- 172.21.2.102:0/2417480969 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0b780c36a0 0x7f0b780c2b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:44.470 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:44.468 7f0b7e75c700 1 -- 172.21.2.102:0/2417480969 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0b780c3190 msgr2=0x7f0b780b1d50 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:04:44.470 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:44.468 7f0b7e75c700 1 --2- 172.21.2.102:0/2417480969 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0b780c3190 0x7f0b780b1d50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:44.470 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:44.468 7f0b7e75c700 1 -- 172.21.2.102:0/2417480969 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b780b3760 con 0x7f0b780b2370 2021-02-10T11:04:44.470 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:44.468 7f0b7e75c700 1 --2- 172.21.2.102:0/2417480969 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0b780b2370 0x7f0b780b2d40 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 rx=0x7f0b70009980 tx=0x7f0b7000bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:04:44.470 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:44.468 7f0b6a7fc700 1 -- 172.21.2.102:0/2417480969 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0b70005af0 con 0x7f0b780b2370 2021-02-10T11:04:44.470 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:44.468 7f0b6a7fc700 1 -- 172.21.2.102:0/2417480969 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b70005d70 con 0x7f0b780b2370 2021-02-10T11:04:44.470 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:44.468 7f0b8013f700 1 -- 172.21.2.102:0/2417480969 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b780b3760 con 0x7f0b780b2370 2021-02-10T11:04:44.470 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:44.468 7f0b6a7fc700 1 -- 172.21.2.102:0/2417480969 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0b70006220 con 0x7f0b780b2370 2021-02-10T11:04:44.470 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:44.468 7f0b8013f700 1 -- 172.21.2.102:0/2417480969 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b780b3760 con 0x7f0b780b2370 2021-02-10T11:04:44.470 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:44.469 7f0b6a7fc700 1 -- 172.21.2.102:0/2417480969 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f0b70005870 con 0x7f0b780b2370 2021-02-10T11:04:44.471 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:44.469 7f0b6a7fc700 1 --2- 172.21.2.102:0/2417480969 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0b60018730 0x7f0b6001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:44.471 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:44.469 7f0b6a7fc700 1 -- 172.21.2.102:0/2417480969 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f0b70032b70 con 0x7f0b780b2370 2021-02-10T11:04:44.471 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:44.469 7f0b7df5b700 1 --2- 172.21.2.102:0/2417480969 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0b60018730 0x7f0b6001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:44.471 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:44.469 7f0b7df5b700 1 --2- 172.21.2.102:0/2417480969 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0b60018730 0x7f0b6001ac10 crc :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:04:44.499 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:44.501 7f0b8013f700 1 -- 172.21.2.102:0/2417480969 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b780edd10 con 0x7f0b780b2370 2021-02-10T11:04:44.501 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:44.503 7f0b6a7fc700 1 -- 172.21.2.102:0/2417480969 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f0b70005d50 con 0x7f0b780b2370 2021-02-10T11:04:44.627 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:44.629 7f0b8013f700 1 -- 172.21.2.102:0/2417480969 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7f0b780edd10 con 0x7f0b780b2370 2021-02-10T11:04:44.628 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:44.630 7f0b6a7fc700 1 -- 172.21.2.102:0/2417480969 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v20) v1 ==== 100+0+10 (secure 0 0 0) 0x7f0b70005d50 con 0x7f0b780b2370 2021-02-10T11:04:44.628 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:04:44.630 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:44.632 7f0b8013f700 1 -- 172.21.2.102:0/2417480969 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0b60018730 msgr2=0x7f0b6001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:44.630 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:44.632 7f0b8013f700 1 --2- 172.21.2.102:0/2417480969 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0b60018730 0x7f0b6001ac10 crc :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:44.630 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:44.632 7f0b8013f700 1 -- 172.21.2.102:0/2417480969 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0b780b2370 msgr2=0x7f0b780b2d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:44.630 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:44.632 7f0b8013f700 1 --2- 172.21.2.102:0/2417480969 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0b780b2370 0x7f0b780b2d40 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 rx=0x7f0b70009980 tx=0x7f0b7000bcd0).stop 2021-02-10T11:04:44.630 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:44.632 7f0b8013f700 1 -- 172.21.2.102:0/2417480969 shutdown_connections 2021-02-10T11:04:44.630 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:44.632 7f0b8013f700 1 --2- 172.21.2.102:0/2417480969 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0b780b2370 0x7f0b780b2d40 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:44.630 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:44.632 7f0b8013f700 1 --2- 172.21.2.102:0/2417480969 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0b60018730 0x7f0b6001ac10 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:44.630 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:44.632 7f0b8013f700 1 --2- 172.21.2.102:0/2417480969 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0b780c3190 0x7f0b780b1d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:44.631 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:44.632 7f0b8013f700 1 --2- 172.21.2.102:0/2417480969 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0b780c36a0 0x7f0b780c2b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:44.631 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:44.632 7f0b8013f700 1 -- 172.21.2.102:0/2417480969 >> 172.21.2.102:0/2417480969 conn(0x7f0b7809f630 msgr2=0x7f0b780a0130 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:04:44.631 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:44.632 7f0b8013f700 1 -- 172.21.2.102:0/2417480969 shutdown_connections 2021-02-10T11:04:44.631 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:44.632 7f0b8013f700 1 -- 172.21.2.102:0/2417480969 wait complete. 2021-02-10T11:04:44.638 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:04:45.064 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:45.048 7f7558b71700 1 Processor -- start 2021-02-10T11:04:45.064 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:45.049 7f7558b71700 1 -- start start 2021-02-10T11:04:45.064 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:45.049 7f7558b71700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f75540e5070 0x7f75541b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:45.064 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:45.049 7f7558b71700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f75540e5470 0x7f7554080310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:45.064 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:45.049 7f7558b71700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f755407ed80 con 0x7f75541b3e40 2021-02-10T11:04:45.065 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:45.049 7f7558b71700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75540d1790 con 0x7f75540e5470 2021-02-10T11:04:45.065 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:45.049 7f7558b71700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75540d19b0 con 0x7f75540e5070 2021-02-10T11:04:45.065 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:45.049 7f755205d700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f75541b3e40 0x7f755407d690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:42372/0 (socket says 172.21.2.102:42372) 2021-02-10T11:04:45.065 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:45.049 7f755205d700 1 -- 172.21.2.102:0/2433408248 learned_addr learned my addr 172.21.2.102:0/2433408248 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:04:45.065 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:45.049 7f755305f700 1 --2- 172.21.2.102:0/2433408248 >> v2:172.21.2.102:3300/0 conn(0x7f75540e5470 0x7f7554080310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:45.066 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:45.050 7f755285e700 1 --2- 172.21.2.102:0/2433408248 >> v2:172.21.2.113:3300/0 conn(0x7f75540e5070 0x7f75541b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:45.066 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:45.050 7f755185c700 1 -- 172.21.2.102:0/2433408248 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 168397330 0 0) 0x7f75480010b0 con 0x7f75541b3e40 2021-02-10T11:04:45.066 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:45.050 7f755305f700 1 -- 172.21.2.102:0/2433408248 >> v2:172.21.2.113:3300/0 conn(0x7f75540e5070 msgr2=0x7f75541b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:04:45.066 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:45.050 7f755305f700 1 --2- 172.21.2.102:0/2433408248 >> v2:172.21.2.113:3300/0 conn(0x7f75540e5070 0x7f75541b3820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:45.066 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:45.050 7f755305f700 1 -- 172.21.2.102:0/2433408248 >> v1:172.21.2.102:6789/0 conn(0x7f75541b3e40 legacy=0x7f755407d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:45.067 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:45.050 7f755305f700 1 -- 172.21.2.102:0/2433408248 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75540d1bd0 con 0x7f75540e5470 2021-02-10T11:04:45.068 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:45.050 7f755285e700 1 --2- 172.21.2.102:0/2433408248 >> v2:172.21.2.113:3300/0 conn(0x7f75540e5070 0x7f75541b3820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:04:45.068 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:45.050 7f755305f700 1 --2- 172.21.2.102:0/2433408248 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f75540e5470 0x7f7554080310 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 rx=0x7f754c006e30 tx=0x7f754c009e60).ready entity=mon.0 client_cookie=c764bfba313ef6c8 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:04:45.068 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:45.050 7f755185c700 1 -- 172.21.2.102:0/2433408248 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f754c006a50 con 0x7f75540e5470 2021-02-10T11:04:45.068 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:45.050 7f755185c700 1 -- 172.21.2.102:0/2433408248 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f754c00b300 con 0x7f75540e5470 2021-02-10T11:04:45.068 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:45.051 7f755185c700 1 -- 172.21.2.102:0/2433408248 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f754c00b7e0 con 0x7f75540e5470 2021-02-10T11:04:45.069 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:45.051 7f7558b71700 1 -- 172.21.2.102:0/2433408248 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f75540e5470 msgr2=0x7f7554080310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:45.069 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:45.051 7f7558b71700 1 --2- 172.21.2.102:0/2433408248 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f75540e5470 0x7f7554080310 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 rx=0x7f754c006e30 tx=0x7f754c009e60).stop 2021-02-10T11:04:45.070 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:45.051 7f7558b71700 1 -- 172.21.2.102:0/2433408248 shutdown_connections 2021-02-10T11:04:45.070 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:45.051 7f7558b71700 1 --2- 172.21.2.102:0/2433408248 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f75540e5470 0x7f7554080310 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:45.070 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:45.051 7f7558b71700 1 --2- 172.21.2.102:0/2433408248 >> v2:172.21.2.113:3300/0 conn(0x7f75540e5070 0x7f75541b3820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:45.070 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:45.051 7f7558b71700 1 -- 172.21.2.102:0/2433408248 >> 172.21.2.102:0/2433408248 conn(0x7f75540ce690 msgr2=0x7f75540e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:04:45.070 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:45.051 7f7558b71700 1 -- 172.21.2.102:0/2433408248 shutdown_connections 2021-02-10T11:04:45.070 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:45.051 7f7558b71700 1 -- 172.21.2.102:0/2433408248 wait complete. 2021-02-10T11:04:45.073 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:45.066 7f7558b71700 1 Processor -- start 2021-02-10T11:04:45.073 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:45.066 7f7558b71700 1 -- start start 2021-02-10T11:04:45.073 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:45.066 7f7558b71700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f755408d400 0x7f7554081230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:45.073 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:45.066 7f7558b71700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7554081850 0x7f75540acf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:45.074 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:45.066 7f7558b71700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7554081c50 0x7f75540ad5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:45.074 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:45.066 7f7558b71700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7554082110 con 0x7f755408d400 2021-02-10T11:04:45.074 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:45.066 7f7558b71700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f755408d800 con 0x7f7554081850 2021-02-10T11:04:45.074 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:45.066 7f7558b71700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75540adbc0 con 0x7f7554081c50 2021-02-10T11:04:45.074 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:45.067 7f755285e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f755408d400 0x7f7554081230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:45.074 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:45.067 7f755285e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f755408d400 0x7f7554081230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:50152/0 (socket says 172.21.2.102:50152) 2021-02-10T11:04:45.075 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:45.067 7f755285e700 1 -- 172.21.2.102:0/1612503986 learned_addr learned my addr 172.21.2.102:0/1612503986 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:04:45.075 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:45.067 7f755285e700 1 -- 172.21.2.102:0/1612503986 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7554081c50 msgr2=0x7f75540ad5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:45.075 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:45.067 7f755285e700 1 --2- 172.21.2.102:0/1612503986 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7554081c50 0x7f75540ad5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:45.075 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:45.067 7f755285e700 1 -- 172.21.2.102:0/1612503986 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7554081850 msgr2=0x7f75540acf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:45.075 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:45.067 7f755285e700 1 --2- 172.21.2.102:0/1612503986 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7554081850 0x7f75540acf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:45.075 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:45.067 7f755285e700 1 -- 172.21.2.102:0/1612503986 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7554187fc0 con 0x7f755408d400 2021-02-10T11:04:45.075 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:45.067 7f755285e700 1 --2- 172.21.2.102:0/1612503986 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f755408d400 0x7f7554081230 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7f7544006f20 tx=0x7f754400b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:04:45.075 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:45.067 7f75437fe700 1 -- 172.21.2.102:0/1612503986 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f754400bd80 con 0x7f755408d400 2021-02-10T11:04:45.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:45.068 7f75437fe700 1 -- 172.21.2.102:0/1612503986 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7544000c90 con 0x7f755408d400 2021-02-10T11:04:45.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:45.068 7f75437fe700 1 -- 172.21.2.102:0/1612503986 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f75440022b0 con 0x7f755408d400 2021-02-10T11:04:45.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:45.068 7f7558b71700 1 -- 172.21.2.102:0/1612503986 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7554187fc0 con 0x7f755408d400 2021-02-10T11:04:45.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:45.068 7f7558b71700 1 -- 172.21.2.102:0/1612503986 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7554187fc0 con 0x7f755408d400 2021-02-10T11:04:45.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:45.068 7f75437fe700 1 -- 172.21.2.102:0/1612503986 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f754400a000 con 0x7f755408d400 2021-02-10T11:04:45.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:45.068 7f75437fe700 1 --2- 172.21.2.102:0/1612503986 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f75380186c0 0x7f753801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:45.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:45.068 7f75437fe700 1 -- 172.21.2.102:0/1612503986 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f7544032410 con 0x7f755408d400 2021-02-10T11:04:45.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:45.068 7f755205d700 1 --2- 172.21.2.102:0/1612503986 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f75380186c0 0x7f753801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:45.077 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:45.069 7f755205d700 1 --2- 172.21.2.102:0/1612503986 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f75380186c0 0x7f753801abc0 crc :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:04:45.097 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:45.099 7f7558b71700 1 -- 172.21.2.102:0/1612503986 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f755408d800 con 0x7f755408d400 2021-02-10T11:04:45.099 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:45.101 7f75437fe700 1 -- 172.21.2.102:0/1612503986 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f754402cf40 con 0x7f755408d400 2021-02-10T11:04:45.225 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:45.227 7f7558b71700 1 -- 172.21.2.102:0/1612503986 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f755408d800 con 0x7f755408d400 2021-02-10T11:04:45.226 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:45.228 7f75437fe700 1 -- 172.21.2.102:0/1612503986 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+13993 (secure 0 0 0) 0x7f754402cf40 con 0x7f755408d400 2021-02-10T11:04:45.226 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:04:45.226 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":20,"fsid":"68db30ea-dbce-4aef-a6df-f01c6fea0513","created":"2021-02-10 11:03:06.331557","modified":"2021-02-10 11:03:44.386945","last_up_change":"2021-02-10 11:03:32.690562","last_in_change":"2021-02-10 11:03:32.690562","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:03:37.322519","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:03:41.161563","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2021-02-10 11:03:42.379651","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"be691b4d-a039-408a-af80-9d53f02b6f1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6818","nonce":36088},{"type":"v1","addr":"172.21.2.102:6820","nonce":36088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6822","nonce":36088},{"type":"v1","addr":"172.21.2.102:6824","nonce":36088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6830","nonce":36088},{"type":"v1","addr":"172.21.2.102:6832","nonce":36088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6826","nonce":36088},{"type":"v1","addr":"172.21.2.102:6828","nonce":36088}]},"public_addr":"172.21.2.102:6820/36088","cluster_addr":"172.21.2.102:6824/36088","heartbeat_back_addr":"172.21.2.102:6832/36088","heartbeat_front_addr":"172.21.2.102:6828/36088","state":["exists","up"]},{"osd":1,"uuid":"64e3ccc8-9c69-4d7c-865e-af8e83822635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6819","nonce":36087},{"type":"v1","addr":"172.21.2.102:6821","nonce":36087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6823","nonce":36087},{"type":"v1","addr":"172.21.2.102:6825","nonce":36087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6831","nonce":36087},{"type":"v1","addr":"172.21.2.102:6833","nonce":36087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6827","nonce":36087},{"type":"v1","addr":"172.21.2.102:6829","nonce":36087}]},"public_addr":"172.21.2.102:6821/36087","cluster_addr":"172.21.2.102:6825/36087","heartbeat_back_addr":"172.21.2.102:6833/36087","heartbeat_front_addr":"172.21.2.102:6829/36087","state":["exists","up"]},{"osd":2,"uuid":"153700af-747f-481f-bdf4-d6072d3a842f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6810","nonce":36089},{"type":"v1","addr":"172.21.2.102:6811","nonce":36089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6812","nonce":36089},{"type":"v1","addr":"172.21.2.102:6813","nonce":36089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6816","nonce":36089},{"type":"v1","addr":"172.21.2.102:6817","nonce":36089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6814","nonce":36089},{"type":"v1","addr":"172.21.2.102:6815","nonce":36089}]},"public_addr":"172.21.2.102:6811/36089","cluster_addr":"172.21.2.102:6813/36089","heartbeat_back_addr":"172.21.2.102:6817/36089","heartbeat_front_addr":"172.21.2.102:6815/36089","state":["exists","up"]},{"osd":3,"uuid":"cbc678f4-83a9-45e6-8830-2f7c232b1ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6802","nonce":36090},{"type":"v1","addr":"172.21.2.102:6803","nonce":36090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6804","nonce":36090},{"type":"v1","addr":"172.21.2.102:6805","nonce":36090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6808","nonce":36090},{"type":"v1","addr":"172.21.2.102:6809","nonce":36090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6806","nonce":36090},{"type":"v1","addr":"172.21.2.102:6807","nonce":36090}]},"public_addr":"172.21.2.102:6803/36090","cluster_addr":"172.21.2.102:6805/36090","heartbeat_back_addr":"172.21.2.102:6809/36090","heartbeat_front_addr":"172.21.2.102:6807/36090","state":["exists","up"]},{"osd":4,"uuid":"c01e4008-b258-4f10-afd2-9e7f37214351","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6808","nonce":36368},{"type":"v1","addr":"172.21.2.113:6811","nonce":36368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6814","nonce":36368},{"type":"v1","addr":"172.21.2.113:6818","nonce":36368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6826","nonce":36368},{"type":"v1","addr":"172.21.2.113:6828","nonce":36368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6820","nonce":36368},{"type":"v1","addr":"172.21.2.113:6824","nonce":36368}]},"public_addr":"172.21.2.113:6811/36368","cluster_addr":"172.21.2.113:6818/36368","heartbeat_back_addr":"172.21.2.113:6828/36368","heartbeat_front_addr":"172.21.2.113:6824/36368","state":["exists","up"]},{"osd":5,"uuid":"551f6007-daf8-4a98-8ee0-c2b16bcccb5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6802","nonce":36366},{"type":"v1","addr":"172.21.2.113:6805","nonce":36366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6807","nonce":36366},{"type":"v1","addr":"172.21.2.113:6810","nonce":36366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6819","nonce":36366},{"type":"v1","addr":"172.21.2.113:6822","nonce":36366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6812","nonce":36366},{"type":"v1","addr":"172.21.2.113:6815","nonce":36366}]},"public_addr":"172.21.2.113:6805/36366","cluster_addr":"172.21.2.113:6810/36366","heartbeat_back_addr":"172.21.2.113:6822/36366","heartbeat_front_addr":"172.21.2.113:6815/36366","state":["exists","up"]},{"osd":6,"uuid":"bf67c040-0e50-48be-b102-c3943bc9ebef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6800","nonce":36367},{"type":"v1","addr":"172.21.2.113:6801","nonce":36367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6803","nonce":36367},{"type":"v1","addr":"172.21.2.113:6804","nonce":36367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6813","nonce":36367},{"type":"v1","addr":"172.21.2.113:6816","nonce":36367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6806","nonce":36367},{"type":"v1","addr":"172.21.2.113:6809","nonce":36367}]},"public_addr":"172.21.2.113:6801/36367","cluster_addr":"172.21.2.113:6804/36367","heartbeat_back_addr":"172.21.2.113:6816/36367","heartbeat_front_addr":"172.21.2.113:6809/36367","state":["exists","up"]},{"osd":7,"uuid":"f5461f80-018d-428f-be89-2eb2bcbe7bb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6817","nonce":36365},{"type":"v1","addr":"172.21.2.113:6821","nonce":36365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6823","nonce":36365},{"type":"v1","addr":"172.21.2.113:6825","nonce":36365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6830","nonce":36365},{"type":"v1","addr":"172.21.2.113:6831","nonce":36365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6827","nonce":36365},{"type":"v1","addr":"172.21.2.113:6829","nonce":36365}]},"public_addr":"172.21.2.113:6821/36365","cluster_addr":"172.21.2.113:6825/36365","heartbeat_back_addr":"172.21.2.113:6831/36365","heartbeat_front_addr":"172.21.2.113:6829/36365","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:04:45.227 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:45.230 7f7558b71700 1 -- 172.21.2.102:0/1612503986 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f75380186c0 msgr2=0x7f753801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:45.228 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:45.230 7f7558b71700 1 --2- 172.21.2.102:0/1612503986 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f75380186c0 0x7f753801abc0 crc :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:45.228 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:45.230 7f7558b71700 1 -- 172.21.2.102:0/1612503986 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f755408d400 msgr2=0x7f7554081230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:45.228 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:45.230 7f7558b71700 1 --2- 172.21.2.102:0/1612503986 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f755408d400 0x7f7554081230 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7f7544006f20 tx=0x7f754400b990).stop 2021-02-10T11:04:45.228 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:45.230 7f7558b71700 1 -- 172.21.2.102:0/1612503986 shutdown_connections 2021-02-10T11:04:45.228 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:45.230 7f7558b71700 1 --2- 172.21.2.102:0/1612503986 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7554081c50 0x7f75540ad5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:45.228 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:45.230 7f7558b71700 1 --2- 172.21.2.102:0/1612503986 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f75380186c0 0x7f753801abc0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:45.228 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:45.230 7f7558b71700 1 --2- 172.21.2.102:0/1612503986 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7554081850 0x7f75540acf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:45.228 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:45.230 7f7558b71700 1 --2- 172.21.2.102:0/1612503986 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f755408d400 0x7f7554081230 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:45.228 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:45.230 7f7558b71700 1 -- 172.21.2.102:0/1612503986 >> 172.21.2.102:0/1612503986 conn(0x7f75541ac620 msgr2=0x7f75540896e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:04:45.228 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:45.230 7f7558b71700 1 -- 172.21.2.102:0/1612503986 shutdown_connections 2021-02-10T11:04:45.228 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:45.230 7f7558b71700 1 -- 172.21.2.102:0/1612503986 wait complete. 2021-02-10T11:04:45.236 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:03:37.322519', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2021-02-10 11:03:41.161563', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2021-02-10 11:03:42.379651', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-02-10T11:04:45.237 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:04:45.666 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:45.652 7f2391eaf700 1 Processor -- start 2021-02-10T11:04:45.667 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:45.652 7f2391eaf700 1 -- start start 2021-02-10T11:04:45.667 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:45.652 7f2391eaf700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f238c1b0870 0x7f238c0f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:45.667 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:45.652 7f2391eaf700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f238c1b0c70 con 0x7f238c1b0470 2021-02-10T11:04:45.667 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:45.652 7f2391eaf700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f238c0f0b50 con 0x7f238c0efe00 2021-02-10T11:04:45.667 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:45.652 7f2391eaf700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f238c0e5ea0 con 0x7f238c1b0870 2021-02-10T11:04:45.667 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:45.653 7f238b7fe700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f238c1b0470 0x7f238c0ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:42384/0 (socket says 172.21.2.102:42384) 2021-02-10T11:04:45.667 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:45.653 7f238b7fe700 1 -- 172.21.2.102:0/1271299640 learned_addr learned my addr 172.21.2.102:0/1271299640 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:04:45.668 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:45.653 7f238affd700 1 --2- 172.21.2.102:0/1271299640 >> v2:172.21.2.102:3300/0 conn(0x7f238c1b0870 0x7f238c0f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:45.668 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:45.653 7f238a7fc700 1 -- 172.21.2.102:0/1271299640 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3495568829 0 0) 0x7f2374000ce0 con 0x7f238c1b0470 2021-02-10T11:04:45.668 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:45.653 7f238a7fc700 1 -- 172.21.2.102:0/1271299640 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2378002fe0 con 0x7f238c1b0470 2021-02-10T11:04:45.668 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:45.653 7f238affd700 1 -- 172.21.2.102:0/1271299640 >> v1:172.21.2.113:6789/0 conn(0x7f238c0efe00 legacy=0x7f238c0f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:04:45.668 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:45.653 7f238affd700 1 -- 172.21.2.102:0/1271299640 >> v1:172.21.2.102:6789/0 conn(0x7f238c1b0470 legacy=0x7f238c0ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:45.668 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:45.653 7f238affd700 1 -- 172.21.2.102:0/1271299640 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f238c0e52e0 con 0x7f238c1b0870 2021-02-10T11:04:45.668 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:45.653 7f238affd700 1 --2- 172.21.2.102:0/1271299640 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f238c1b0870 0x7f238c0f20e0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0x7f237c006650 tx=0x7f237c009a80).ready entity=mon.0 client_cookie=4a040b2a76e34c4e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:04:45.668 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:45.654 7f238a7fc700 1 -- 172.21.2.102:0/1271299640 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f237c0009b0 con 0x7f238c1b0870 2021-02-10T11:04:45.669 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:45.654 7f238a7fc700 1 -- 172.21.2.102:0/1271299640 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f237c00c0c0 con 0x7f238c1b0870 2021-02-10T11:04:45.669 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:45.654 7f238a7fc700 1 -- 172.21.2.102:0/1271299640 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f237c00c510 con 0x7f238c1b0870 2021-02-10T11:04:45.669 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:45.654 7f2391eaf700 1 -- 172.21.2.102:0/1271299640 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f238c1b0870 msgr2=0x7f238c0f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:45.669 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:45.654 7f2391eaf700 1 --2- 172.21.2.102:0/1271299640 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f238c1b0870 0x7f238c0f20e0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0x7f237c006650 tx=0x7f237c009a80).stop 2021-02-10T11:04:45.669 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:45.654 7f2391eaf700 1 -- 172.21.2.102:0/1271299640 shutdown_connections 2021-02-10T11:04:45.669 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:45.654 7f2391eaf700 1 --2- 172.21.2.102:0/1271299640 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f238c1b0870 0x7f238c0f20e0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:45.669 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:45.654 7f2391eaf700 1 -- 172.21.2.102:0/1271299640 >> 172.21.2.102:0/1271299640 conn(0x7f238c0ce3e0 msgr2=0x7f238c0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:04:45.670 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:45.654 7f2391eaf700 1 -- 172.21.2.102:0/1271299640 shutdown_connections 2021-02-10T11:04:45.670 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:45.654 7f2391eaf700 1 -- 172.21.2.102:0/1271299640 wait complete. 2021-02-10T11:04:45.672 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:45.669 7f2391eaf700 1 Processor -- start 2021-02-10T11:04:45.672 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:45.669 7f2391eaf700 1 -- start start 2021-02-10T11:04:45.672 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:45.670 7f2391eaf700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f238c0a6c90 0x7f238c1924c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:45.672 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:45.670 7f2391eaf700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f238c192ae0 0x7f238c192ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:45.672 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:45.670 7f2391eaf700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f238c0919e0 0x7f238c091de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:45.672 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:45.670 7f2391eaf700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f238c19d2e0 con 0x7f238c0919e0 2021-02-10T11:04:45.672 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:45.670 7f2391eaf700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f238c0914d0 con 0x7f238c0a6c90 2021-02-10T11:04:45.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:45.670 7f2391eaf700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f238c0a6820 con 0x7f238c192ae0 2021-02-10T11:04:45.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:45.670 7f238bfff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f238c0919e0 0x7f238c091de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:45.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:45.670 7f238bfff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f238c0919e0 0x7f238c091de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:50170/0 (socket says 172.21.2.102:50170) 2021-02-10T11:04:45.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:45.670 7f238bfff700 1 -- 172.21.2.102:0/1191848745 learned_addr learned my addr 172.21.2.102:0/1191848745 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:04:45.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:45.670 7f238bfff700 1 -- 172.21.2.102:0/1191848745 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f238c192ae0 msgr2=0x7f238c192ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:45.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:45.670 7f238bfff700 1 --2- 172.21.2.102:0/1191848745 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f238c192ae0 0x7f238c192ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:45.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:45.670 7f238bfff700 1 -- 172.21.2.102:0/1191848745 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f238c0a6c90 msgr2=0x7f238c1924c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:45.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:45.670 7f238bfff700 1 --2- 172.21.2.102:0/1191848745 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f238c0a6c90 0x7f238c1924c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:45.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:45.670 7f238bfff700 1 -- 172.21.2.102:0/1191848745 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f238c092400 con 0x7f238c0919e0 2021-02-10T11:04:45.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:45.670 7f238bfff700 1 --2- 172.21.2.102:0/1191848745 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f238c0919e0 0x7f238c091de0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 rx=0x7f238400b8e0 tx=0x7f238400bc80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:04:45.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:45.671 7f2383fff700 1 -- 172.21.2.102:0/1191848745 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2384000d30 con 0x7f238c0919e0 2021-02-10T11:04:45.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:45.671 7f2383fff700 1 -- 172.21.2.102:0/1191848745 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f238400c2f0 con 0x7f238c0919e0 2021-02-10T11:04:45.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:45.671 7f2383fff700 1 -- 172.21.2.102:0/1191848745 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f238400c810 con 0x7f238c0919e0 2021-02-10T11:04:45.674 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:45.671 7f2391eaf700 1 -- 172.21.2.102:0/1191848745 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f238c092400 con 0x7f238c0919e0 2021-02-10T11:04:45.674 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:45.671 7f2391eaf700 1 -- 172.21.2.102:0/1191848745 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f238c092400 con 0x7f238c0919e0 2021-02-10T11:04:45.674 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:45.672 7f2383fff700 1 -- 172.21.2.102:0/1191848745 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f2384001220 con 0x7f238c0919e0 2021-02-10T11:04:45.674 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:45.672 7f2383fff700 1 --2- 172.21.2.102:0/1191848745 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f23700186c0 0x7f237001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:45.674 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:45.672 7f2383fff700 1 -- 172.21.2.102:0/1191848745 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f2384032950 con 0x7f238c0919e0 2021-02-10T11:04:45.674 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:45.672 7f238b7fe700 1 --2- 172.21.2.102:0/1191848745 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f23700186c0 0x7f237001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:45.674 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:45.672 7f238b7fe700 1 --2- 172.21.2.102:0/1191848745 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f23700186c0 0x7f237001abc0 crc :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:04:45.700 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:45.702 7f2391eaf700 1 -- 172.21.2.102:0/1191848745 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f238c0e52e0 con 0x7f238c0919e0 2021-02-10T11:04:45.703 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:45.705 7f2383fff700 1 -- 172.21.2.102:0/1191848745 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f238400bdc0 con 0x7f238c0919e0 2021-02-10T11:04:45.832 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:45.834 7f2391eaf700 1 -- 172.21.2.102:0/1191848745 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f238c0e52e0 con 0x7f238c0919e0 2021-02-10T11:04:45.833 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:45.835 7f2383fff700 1 -- 172.21.2.102:0/1191848745 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v20) v1 ==== 92+0+10 (secure 0 0 0) 0x7f238401f8c0 con 0x7f238c0919e0 2021-02-10T11:04:45.833 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:04:45.835 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:45.837 7f2391eaf700 1 -- 172.21.2.102:0/1191848745 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f23700186c0 msgr2=0x7f237001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:45.835 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:45.837 7f2391eaf700 1 --2- 172.21.2.102:0/1191848745 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f23700186c0 0x7f237001abc0 crc :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:45.835 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:45.837 7f2391eaf700 1 -- 172.21.2.102:0/1191848745 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f238c0919e0 msgr2=0x7f238c091de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:45.835 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:45.837 7f2391eaf700 1 --2- 172.21.2.102:0/1191848745 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f238c0919e0 0x7f238c091de0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 rx=0x7f238400b8e0 tx=0x7f238400bc80).stop 2021-02-10T11:04:45.835 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:45.837 7f2391eaf700 1 -- 172.21.2.102:0/1191848745 shutdown_connections 2021-02-10T11:04:45.835 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:45.837 7f2391eaf700 1 --2- 172.21.2.102:0/1191848745 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f238c0919e0 0x7f238c091de0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:45.835 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:45.837 7f2391eaf700 1 --2- 172.21.2.102:0/1191848745 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f238c192ae0 0x7f238c192ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:45.835 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:45.837 7f2391eaf700 1 --2- 172.21.2.102:0/1191848745 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f23700186c0 0x7f237001abc0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:45.835 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:45.837 7f2391eaf700 1 --2- 172.21.2.102:0/1191848745 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f238c0a6c90 0x7f238c1924c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:45.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:45.837 7f2391eaf700 1 -- 172.21.2.102:0/1191848745 >> 172.21.2.102:0/1191848745 conn(0x7f238c0ac550 msgr2=0x7f238c0be1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:04:45.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:45.837 7f2391eaf700 1 -- 172.21.2.102:0/1191848745 shutdown_connections 2021-02-10T11:04:45.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:45.838 7f2391eaf700 1 -- 172.21.2.102:0/1191848745 wait complete. 2021-02-10T11:04:45.844 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-02-10T11:04:46.270 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:46.256 7f4e8ebd1700 1 Processor -- start 2021-02-10T11:04:46.271 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:46.256 7f4e8ebd1700 1 -- start start 2021-02-10T11:04:46.271 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:46.257 7f4e8ebd1700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f4e881b0870 0x7f4e880f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:46.271 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:46.257 7f4e8ebd1700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f4e880efe00 0x7f4e880f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:46.271 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:46.257 7f4e8ebd1700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e881b0c70 con 0x7f4e881b0470 2021-02-10T11:04:46.271 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:46.257 7f4e8ebd1700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e880e5ea0 con 0x7f4e881b0870 2021-02-10T11:04:46.271 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:46.257 7f4e8ebd1700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e880f0b50 con 0x7f4e880efe00 2021-02-10T11:04:46.271 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:46.257 7f4e87fff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f4e881b0870 0x7f4e880f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:46.271 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:46.257 7f4e87fff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f4e881b0870 0x7f4e880f20e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:50176/0 (socket says 172.21.2.102:50176) 2021-02-10T11:04:46.271 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:46.257 7f4e87fff700 1 -- 172.21.2.102:0/3160202608 learned_addr learned my addr 172.21.2.102:0/3160202608 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:04:46.271 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:46.257 7f4e87fff700 1 -- 172.21.2.102:0/3160202608 >> v2:172.21.2.113:3300/0 conn(0x7f4e880efe00 msgr2=0x7f4e880f0200 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:04:46.271 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:46.257 7f4e87fff700 1 --2- 172.21.2.102:0/3160202608 >> v2:172.21.2.113:3300/0 conn(0x7f4e880efe00 0x7f4e880f0200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:46.271 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:46.257 7f4e87fff700 1 -- 172.21.2.102:0/3160202608 >> v1:172.21.2.113:6789/0 conn(0x7f4e881b0470 legacy=0x7f4e880ef610 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:04:46.271 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:46.257 7f4e87fff700 1 -- 172.21.2.102:0/3160202608 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e880f0820 con 0x7f4e881b0870 2021-02-10T11:04:46.272 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:46.258 7f4e87fff700 1 --2- 172.21.2.102:0/3160202608 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4e881b0870 0x7f4e880f20e0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7f4e78006700 tx=0x7f4e78009b80).ready entity=mon.0 client_cookie=9539da843d8d5c6a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:04:46.272 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:46.258 7f4e877fe700 1 -- 172.21.2.102:0/3160202608 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4e7800c0c0 con 0x7f4e881b0870 2021-02-10T11:04:46.272 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:46.258 7f4e877fe700 1 -- 172.21.2.102:0/3160202608 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e7800c440 con 0x7f4e881b0870 2021-02-10T11:04:46.272 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:46.258 7f4e877fe700 1 -- 172.21.2.102:0/3160202608 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4e7800c960 con 0x7f4e881b0870 2021-02-10T11:04:46.272 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:46.258 7f4e8ebd1700 1 -- 172.21.2.102:0/3160202608 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4e881b0870 msgr2=0x7f4e880f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:46.272 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:46.258 7f4e8ebd1700 1 --2- 172.21.2.102:0/3160202608 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4e881b0870 0x7f4e880f20e0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7f4e78006700 tx=0x7f4e78009b80).stop 2021-02-10T11:04:46.272 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:46.258 7f4e8ebd1700 1 -- 172.21.2.102:0/3160202608 shutdown_connections 2021-02-10T11:04:46.272 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:46.258 7f4e8ebd1700 1 --2- 172.21.2.102:0/3160202608 >> v2:172.21.2.113:3300/0 conn(0x7f4e880efe00 0x7f4e880f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:46.272 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:46.258 7f4e8ebd1700 1 --2- 172.21.2.102:0/3160202608 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4e881b0870 0x7f4e880f20e0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:46.272 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:46.258 7f4e8ebd1700 1 -- 172.21.2.102:0/3160202608 >> 172.21.2.102:0/3160202608 conn(0x7f4e880ce3e0 msgr2=0x7f4e880e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:04:46.272 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:46.259 7f4e8ebd1700 1 -- 172.21.2.102:0/3160202608 shutdown_connections 2021-02-10T11:04:46.272 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:46.259 7f4e8ebd1700 1 -- 172.21.2.102:0/3160202608 wait complete. 2021-02-10T11:04:46.274 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:46.273 7f4e8ebd1700 1 Processor -- start 2021-02-10T11:04:46.274 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:46.273 7f4e8ebd1700 1 -- start start 2021-02-10T11:04:46.274 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:46.274 7f4e8ebd1700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4e880b2840 0x7f4e881a4300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:46.274 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:46.274 7f4e8ebd1700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4e880b30d0 0x7f4e880b34d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:46.274 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:46.274 7f4e8ebd1700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4e880b3af0 0x7f4e880f0580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:46.274 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:46.274 7f4e8ebd1700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e880b2c40 con 0x7f4e880b2840 2021-02-10T11:04:46.274 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:46.274 7f4e8ebd1700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e881a4920 con 0x7f4e880b3af0 2021-02-10T11:04:46.274 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:46.274 7f4e8ebd1700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e880f0ba0 con 0x7f4e880b30d0 2021-02-10T11:04:46.274 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:46.274 7f4e8c9ed700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4e880b2840 0x7f4e881a4300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:46.275 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:46.274 7f4e8c9ed700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4e880b2840 0x7f4e881a4300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:50184/0 (socket says 172.21.2.102:50184) 2021-02-10T11:04:46.275 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:46.274 7f4e8c9ed700 1 -- 172.21.2.102:0/517435719 learned_addr learned my addr 172.21.2.102:0/517435719 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:04:46.275 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:46.274 7f4e8c9ed700 1 -- 172.21.2.102:0/517435719 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4e880b30d0 msgr2=0x7f4e880b34d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:46.275 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:46.274 7f4e8c9ed700 1 --2- 172.21.2.102:0/517435719 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4e880b30d0 0x7f4e880b34d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:46.275 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:46.274 7f4e8c9ed700 1 -- 172.21.2.102:0/517435719 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4e880b3af0 msgr2=0x7f4e880f0580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:46.275 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:46.274 7f4e8c9ed700 1 --2- 172.21.2.102:0/517435719 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4e880b3af0 0x7f4e880f0580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:46.275 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:46.274 7f4e8c9ed700 1 -- 172.21.2.102:0/517435719 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e880f0dc0 con 0x7f4e880b2840 2021-02-10T11:04:46.275 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:46.274 7f4e8c9ed700 1 --2- 172.21.2.102:0/517435719 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4e880b2840 0x7f4e881a4300 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 rx=0x7f4e7000b5b0 tx=0x7f4e7000b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:04:46.275 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:46.275 7f4e857fa700 1 -- 172.21.2.102:0/517435719 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4e7000bd40 con 0x7f4e880b2840 2021-02-10T11:04:46.275 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:46.275 7f4e857fa700 1 -- 172.21.2.102:0/517435719 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e70006240 con 0x7f4e880b2840 2021-02-10T11:04:46.275 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:46.275 7f4e8ebd1700 1 -- 172.21.2.102:0/517435719 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e880f0dc0 con 0x7f4e880b2840 2021-02-10T11:04:46.275 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:46.275 7f4e8ebd1700 1 -- 172.21.2.102:0/517435719 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e880f1350 con 0x7f4e880b2840 2021-02-10T11:04:46.275 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:46.275 7f4e857fa700 1 -- 172.21.2.102:0/517435719 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4e70005720 con 0x7f4e880b2840 2021-02-10T11:04:46.276 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:46.276 7f4e857fa700 1 -- 172.21.2.102:0/517435719 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4e700054a0 con 0x7f4e880b2840 2021-02-10T11:04:46.276 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:46.276 7f4e857fa700 1 --2- 172.21.2.102:0/517435719 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4e6c0186c0 0x7f4e6c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:46.276 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:46.276 7f4e857fa700 1 -- 172.21.2.102:0/517435719 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f4e700325c0 con 0x7f4e880b2840 2021-02-10T11:04:46.276 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:46.276 7f4e87fff700 1 --2- 172.21.2.102:0/517435719 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4e6c0186c0 0x7f4e6c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:46.276 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:46.276 7f4e87fff700 1 --2- 172.21.2.102:0/517435719 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4e6c0186c0 0x7f4e6c01abc0 crc :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:04:46.305 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:46.307 7f4e8ebd1700 1 -- 172.21.2.102:0/517435719 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e880f0050 con 0x7f4e880b2840 2021-02-10T11:04:46.307 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:46.309 7f4e857fa700 1 -- 172.21.2.102:0/517435719 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f4e70000c60 con 0x7f4e880b2840 2021-02-10T11:04:46.433 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:46.435 7f4e8ebd1700 1 -- 172.21.2.102:0/517435719 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7f4e880f0050 con 0x7f4e880b2840 2021-02-10T11:04:46.433 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:46.435 7f4e857fa700 1 -- 172.21.2.102:0/517435719 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v20) v1 ==== 104+0+10 (secure 0 0 0) 0x7f4e700054a0 con 0x7f4e880b2840 2021-02-10T11:04:46.434 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:04:46.435 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:46.437 7f4e8ebd1700 1 -- 172.21.2.102:0/517435719 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4e6c0186c0 msgr2=0x7f4e6c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:46.435 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:46.437 7f4e8ebd1700 1 --2- 172.21.2.102:0/517435719 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4e6c0186c0 0x7f4e6c01abc0 crc :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:46.435 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:46.437 7f4e8ebd1700 1 -- 172.21.2.102:0/517435719 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4e880b2840 msgr2=0x7f4e881a4300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:46.435 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:46.437 7f4e8ebd1700 1 --2- 172.21.2.102:0/517435719 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4e880b2840 0x7f4e881a4300 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 rx=0x7f4e7000b5b0 tx=0x7f4e7000b950).stop 2021-02-10T11:04:46.436 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:46.437 7f4e8ebd1700 1 -- 172.21.2.102:0/517435719 shutdown_connections 2021-02-10T11:04:46.436 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:46.437 7f4e8ebd1700 1 --2- 172.21.2.102:0/517435719 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4e6c0186c0 0x7f4e6c01abc0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:46.436 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:46.438 7f4e8ebd1700 1 --2- 172.21.2.102:0/517435719 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4e880b3af0 0x7f4e880f0580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:46.436 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:46.438 7f4e8ebd1700 1 --2- 172.21.2.102:0/517435719 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4e880b30d0 0x7f4e880b34d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:46.436 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:46.438 7f4e8ebd1700 1 --2- 172.21.2.102:0/517435719 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4e880b2840 0x7f4e881a4300 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:46.436 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:46.438 7f4e8ebd1700 1 -- 172.21.2.102:0/517435719 >> 172.21.2.102:0/517435719 conn(0x7f4e880c36f0 msgr2=0x7f4e88094c30 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:04:46.436 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:46.438 7f4e8ebd1700 1 -- 172.21.2.102:0/517435719 shutdown_connections 2021-02-10T11:04:46.437 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:46.438 7f4e8ebd1700 1 -- 172.21.2.102:0/517435719 wait complete. 2021-02-10T11:04:46.443 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-02-10T11:04:46.872 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:46.858 7fab4dd14700 1 Processor -- start 2021-02-10T11:04:46.872 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:46.858 7fab4dd14700 1 -- start start 2021-02-10T11:04:46.873 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:46.859 7fab4dd14700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fab481b0470 0x7fab481b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:46.873 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:46.859 7fab4dd14700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fab480f20e0 0x7fab480f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:46.873 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:46.859 7fab4dd14700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab480f0b50 con 0x7fab480ef610 2021-02-10T11:04:46.873 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:46.859 7fab4dd14700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab480f0820 con 0x7fab480f20e0 2021-02-10T11:04:46.873 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:46.859 7fab4dd14700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab480f24e0 con 0x7fab481b0470 2021-02-10T11:04:46.873 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:46.859 7fab47fff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fab480f20e0 0x7fab480f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:46.873 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:46.859 7fab47fff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fab480f20e0 0x7fab480f0200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:50192/0 (socket says 172.21.2.102:50192) 2021-02-10T11:04:46.873 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:46.859 7fab47fff700 1 -- 172.21.2.102:0/182921411 learned_addr learned my addr 172.21.2.102:0/182921411 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:04:46.873 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:46.859 7fab477fe700 1 --2- 172.21.2.102:0/182921411 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fab481b0470 0x7fab481b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:46.873 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:46.859 7fab47fff700 1 -- 172.21.2.102:0/182921411 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fab481b0470 msgr2=0x7fab481b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:04:46.873 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:46.859 7fab47fff700 1 --2- 172.21.2.102:0/182921411 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fab481b0470 0x7fab481b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:46.874 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:46.859 7fab47fff700 1 -- 172.21.2.102:0/182921411 >> v1:172.21.2.113:6789/0 conn(0x7fab480ef610 legacy=0x7fab480efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:46.874 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:46.859 7fab47fff700 1 -- 172.21.2.102:0/182921411 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab480e52e0 con 0x7fab480f20e0 2021-02-10T11:04:46.874 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:46.860 7fab47fff700 1 --2- 172.21.2.102:0/182921411 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fab480f20e0 0x7fab480f0200 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0x7fab40006f00 tx=0x7fab40009f10).ready entity=mon.0 client_cookie=944fe3c7c0abe26 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:04:46.874 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:46.860 7fab467fc700 1 -- 172.21.2.102:0/182921411 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fab4000b3b0 con 0x7fab480f20e0 2021-02-10T11:04:46.874 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:46.860 7fab467fc700 1 -- 172.21.2.102:0/182921411 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab4000b3b0 con 0x7fab480f20e0 2021-02-10T11:04:46.874 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:46.860 7fab467fc700 1 -- 172.21.2.102:0/182921411 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fab4000b630 con 0x7fab480f20e0 2021-02-10T11:04:46.874 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:46.860 7fab4dd14700 1 -- 172.21.2.102:0/182921411 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fab480f20e0 msgr2=0x7fab480f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:46.874 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:46.860 7fab4dd14700 1 --2- 172.21.2.102:0/182921411 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fab480f20e0 0x7fab480f0200 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0x7fab40006f00 tx=0x7fab40009f10).stop 2021-02-10T11:04:46.874 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:46.860 7fab4dd14700 1 -- 172.21.2.102:0/182921411 shutdown_connections 2021-02-10T11:04:46.874 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:46.860 7fab4dd14700 1 --2- 172.21.2.102:0/182921411 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fab480f20e0 0x7fab480f0200 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:46.874 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:46.860 7fab4dd14700 1 --2- 172.21.2.102:0/182921411 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fab481b0470 0x7fab481b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:46.874 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:46.860 7fab4dd14700 1 -- 172.21.2.102:0/182921411 >> 172.21.2.102:0/182921411 conn(0x7fab480ce3e0 msgr2=0x7fab480e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:04:46.875 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:46.860 7fab4dd14700 1 -- 172.21.2.102:0/182921411 shutdown_connections 2021-02-10T11:04:46.875 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:46.860 7fab4dd14700 1 -- 172.21.2.102:0/182921411 wait complete. 2021-02-10T11:04:46.876 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:46.875 7fab4dd14700 1 Processor -- start 2021-02-10T11:04:46.876 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:46.875 7fab4dd14700 1 -- start start 2021-02-10T11:04:46.876 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:46.875 7fab4dd14700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fab480b2eb0 0x7fab480b2330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:46.876 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:46.875 7fab4dd14700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fab480b2950 0x7fab480b3380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:46.876 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:46.875 7fab4dd14700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fab480b39a0 0x7fab4809d340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:46.877 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:46.875 7fab4dd14700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab4809cdc0 con 0x7fab480b2950 2021-02-10T11:04:46.877 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:46.875 7fab4dd14700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab4809d960 con 0x7fab480b39a0 2021-02-10T11:04:46.877 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:46.875 7fab4dd14700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab4809db80 con 0x7fab480b2eb0 2021-02-10T11:04:46.877 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:46.876 7fab46ffd700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fab480b2950 0x7fab480b3380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:46.877 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:46.876 7fab46ffd700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fab480b2950 0x7fab480b3380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:50198/0 (socket says 172.21.2.102:50198) 2021-02-10T11:04:46.877 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:46.876 7fab46ffd700 1 -- 172.21.2.102:0/1969239256 learned_addr learned my addr 172.21.2.102:0/1969239256 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:04:46.877 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:46.876 7fab477fe700 1 --2- 172.21.2.102:0/1969239256 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fab480b2eb0 0x7fab480b2330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:46.877 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:46.876 7fab47fff700 1 --2- 172.21.2.102:0/1969239256 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fab480b39a0 0x7fab4809d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:46.877 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:46.876 7fab46ffd700 1 -- 172.21.2.102:0/1969239256 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fab480b2eb0 msgr2=0x7fab480b2330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:46.877 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:46.876 7fab46ffd700 1 --2- 172.21.2.102:0/1969239256 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fab480b2eb0 0x7fab480b2330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:46.877 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:46.876 7fab46ffd700 1 -- 172.21.2.102:0/1969239256 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fab480b39a0 msgr2=0x7fab4809d340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:46.877 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:46.876 7fab46ffd700 1 --2- 172.21.2.102:0/1969239256 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fab480b39a0 0x7fab4809d340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:46.877 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:46.876 7fab46ffd700 1 -- 172.21.2.102:0/1969239256 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab4809dda0 con 0x7fab480b2950 2021-02-10T11:04:46.878 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:46.876 7fab46ffd700 1 --2- 172.21.2.102:0/1969239256 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fab480b2950 0x7fab480b3380 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 rx=0x7fab38006ed0 tx=0x7fab3800b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:04:46.878 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:46.876 7fab3ffff700 1 -- 172.21.2.102:0/1969239256 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fab38006240 con 0x7fab480b2950 2021-02-10T11:04:46.878 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:46.876 7fab3ffff700 1 -- 172.21.2.102:0/1969239256 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab380009b0 con 0x7fab480b2950 2021-02-10T11:04:46.878 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:46.876 7fab4dd14700 1 -- 172.21.2.102:0/1969239256 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab4809dda0 con 0x7fab480b2950 2021-02-10T11:04:46.878 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:46.876 7fab3ffff700 1 -- 172.21.2.102:0/1969239256 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fab38005730 con 0x7fab480b2950 2021-02-10T11:04:46.878 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:46.877 7fab4dd14700 1 -- 172.21.2.102:0/1969239256 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab4809e330 con 0x7fab480b2950 2021-02-10T11:04:46.878 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:46.877 7fab3ffff700 1 -- 172.21.2.102:0/1969239256 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fab38006030 con 0x7fab480b2950 2021-02-10T11:04:46.878 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:46.877 7fab3ffff700 1 --2- 172.21.2.102:0/1969239256 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fab2c0186c0 0x7fab2c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:46.878 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:46.877 7fab3ffff700 1 -- 172.21.2.102:0/1969239256 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fab38032830 con 0x7fab480b2950 2021-02-10T11:04:46.878 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:46.877 7fab477fe700 1 --2- 172.21.2.102:0/1969239256 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fab2c0186c0 0x7fab2c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:46.878 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:46.878 7fab477fe700 1 --2- 172.21.2.102:0/1969239256 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fab2c0186c0 0x7fab2c01abc0 crc :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:04:46.906 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:46.909 7fab4dd14700 1 -- 172.21.2.102:0/1969239256 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab48105b30 con 0x7fab480b2950 2021-02-10T11:04:46.909 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:46.911 7fab3ffff700 1 -- 172.21.2.102:0/1969239256 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fab38006030 con 0x7fab480b2950 2021-02-10T11:04:47.038 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:47.040 7fab4dd14700 1 -- 172.21.2.102:0/1969239256 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7fab480f0b50 con 0x7fab480b2950 2021-02-10T11:04:47.039 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:47.041 7fab3ffff700 1 -- 172.21.2.102:0/1969239256 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v20) v1 ==== 100+0+10 (secure 0 0 0) 0x7fab380009b0 con 0x7fab480b2950 2021-02-10T11:04:47.039 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:04:47.041 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:47.042 7fab4dd14700 1 -- 172.21.2.102:0/1969239256 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fab2c0186c0 msgr2=0x7fab2c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:47.041 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:47.042 7fab4dd14700 1 --2- 172.21.2.102:0/1969239256 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fab2c0186c0 0x7fab2c01abc0 crc :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:47.041 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:47.042 7fab4dd14700 1 -- 172.21.2.102:0/1969239256 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fab480b2950 msgr2=0x7fab480b3380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:47.041 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:47.042 7fab4dd14700 1 --2- 172.21.2.102:0/1969239256 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fab480b2950 0x7fab480b3380 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 rx=0x7fab38006ed0 tx=0x7fab3800b960).stop 2021-02-10T11:04:47.041 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:47.043 7fab4dd14700 1 -- 172.21.2.102:0/1969239256 shutdown_connections 2021-02-10T11:04:47.041 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:47.043 7fab4dd14700 1 --2- 172.21.2.102:0/1969239256 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fab2c0186c0 0x7fab2c01abc0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:47.041 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:47.043 7fab4dd14700 1 --2- 172.21.2.102:0/1969239256 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fab480b39a0 0x7fab4809d340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:47.041 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:47.043 7fab4dd14700 1 --2- 172.21.2.102:0/1969239256 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fab480b2950 0x7fab480b3380 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:47.041 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:47.043 7fab4dd14700 1 --2- 172.21.2.102:0/1969239256 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fab480b2eb0 0x7fab480b2330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:47.041 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:47.043 7fab4dd14700 1 -- 172.21.2.102:0/1969239256 >> 172.21.2.102:0/1969239256 conn(0x7fab4809fc90 msgr2=0x7fab48094ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:04:47.041 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:47.043 7fab4dd14700 1 -- 172.21.2.102:0/1969239256 shutdown_connections 2021-02-10T11:04:47.042 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:47.043 7fab4dd14700 1 -- 172.21.2.102:0/1969239256 wait complete. 2021-02-10T11:04:47.091 INFO:tasks.cephfs_test_runner:Starting test: test_async_subvolume_rm (tasks.cephfs.test_volumes.TestVolumes) 2021-02-10T11:04:47.092 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestVolumes.test_async_subvolume_rm' 2021-02-10T11:04:47.519 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:47.503 7f639502d700 1 Processor -- start 2021-02-10T11:04:47.519 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:47.503 7f639502d700 1 -- start start 2021-02-10T11:04:47.519 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:47.504 7f639502d700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f63900e5070 0x7f63901b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:47.519 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:47.504 7f639502d700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f63900e5470 0x7f6390080310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:47.519 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:47.504 7f639502d700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f639007ed80 con 0x7f63901b3e40 2021-02-10T11:04:47.520 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:47.504 7f639502d700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63900d1790 con 0x7f63900e5070 2021-02-10T11:04:47.520 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:47.504 7f639502d700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63900d19b0 con 0x7f63900e5470 2021-02-10T11:04:47.520 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:47.505 7f638f59e700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f63900e5470 0x7f6390080310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:47.520 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:47.505 7f638ed9d700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f63900e5070 0x7f63901b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:47.520 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:47.505 7f638e59c700 1 --1- >> v1:172.21.2.113:6789/0 conn(0x7f63901b3e40 0x7f639007d690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.113:6789/0 says I am v1:172.21.2.102:49752/0 (socket says 172.21.2.102:49752) 2021-02-10T11:04:47.520 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:47.505 7f638e59c700 1 -- 172.21.2.102:0/4120069581 learned_addr learned my addr 172.21.2.102:0/4120069581 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:04:47.520 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:47.505 7f638f59e700 1 -- 172.21.2.102:0/4120069581 >> v2:172.21.2.113:3300/0 conn(0x7f63900e5070 msgr2=0x7f63901b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:04:47.520 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:47.505 7f638f59e700 1 --2- 172.21.2.102:0/4120069581 >> v2:172.21.2.113:3300/0 conn(0x7f63900e5070 0x7f63901b3820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:47.520 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:47.505 7f638f59e700 1 -- 172.21.2.102:0/4120069581 >> v1:172.21.2.113:6789/0 conn(0x7f63901b3e40 legacy=0x7f639007d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:47.520 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:47.505 7f638dd9b700 1 -- 172.21.2.102:0/4120069581 <== mon.1 v1:172.21.2.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2419023922 0 0) 0x7f63780010b0 con 0x7f63901b3e40 2021-02-10T11:04:47.520 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:47.505 7f638f59e700 1 -- 172.21.2.102:0/4120069581 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63900d1bd0 con 0x7f63900e5470 2021-02-10T11:04:47.520 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:47.506 7f638ed9d700 1 --2- 172.21.2.102:0/4120069581 >> v2:172.21.2.113:3300/0 conn(0x7f63900e5070 0x7f63901b3820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:04:47.520 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:47.506 7f638f59e700 1 --2- 172.21.2.102:0/4120069581 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f63900e5470 0x7f6390080310 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f6388006e50 tx=0x7f6388009e70).ready entity=mon.2 client_cookie=826e4997238bdbfb server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:04:47.520 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:47.507 7f638dd9b700 1 -- 172.21.2.102:0/4120069581 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6388006a20 con 0x7f63900e5470 2021-02-10T11:04:47.521 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:47.507 7f638dd9b700 1 -- 172.21.2.102:0/4120069581 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6388006790 con 0x7f63900e5470 2021-02-10T11:04:47.521 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:47.507 7f638dd9b700 1 -- 172.21.2.102:0/4120069581 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f638800b700 con 0x7f63900e5470 2021-02-10T11:04:47.521 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:47.507 7f639502d700 1 -- 172.21.2.102:0/4120069581 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f63900e5470 msgr2=0x7f6390080310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:47.521 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:47.507 7f639502d700 1 --2- 172.21.2.102:0/4120069581 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f63900e5470 0x7f6390080310 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f6388006e50 tx=0x7f6388009e70).stop 2021-02-10T11:04:47.521 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:47.507 7f639502d700 1 -- 172.21.2.102:0/4120069581 shutdown_connections 2021-02-10T11:04:47.521 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:47.507 7f639502d700 1 --2- 172.21.2.102:0/4120069581 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f63900e5470 0x7f6390080310 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:47.521 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:47.507 7f639502d700 1 --2- 172.21.2.102:0/4120069581 >> v2:172.21.2.113:3300/0 conn(0x7f63900e5070 0x7f63901b3820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:47.521 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:47.507 7f639502d700 1 -- 172.21.2.102:0/4120069581 >> 172.21.2.102:0/4120069581 conn(0x7f63900ce690 msgr2=0x7f63900e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:04:47.521 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:47.507 7f639502d700 1 -- 172.21.2.102:0/4120069581 shutdown_connections 2021-02-10T11:04:47.521 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:47.508 7f639502d700 1 -- 172.21.2.102:0/4120069581 wait complete. 2021-02-10T11:04:47.523 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:47.522 7f639502d700 1 Processor -- start 2021-02-10T11:04:47.523 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:47.522 7f639502d700 1 -- start start 2021-02-10T11:04:47.523 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:47.522 7f639502d700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6390191bb0 0x7f63900a2380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:47.523 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:47.522 7f639502d700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f63900a29a0 0x7f63900a2da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:47.523 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:47.522 7f639502d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f63900a33c0 0x7f63900a03d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:47.523 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:47.522 7f639502d700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63900adf90 con 0x7f63900a33c0 2021-02-10T11:04:47.523 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:47.522 7f639502d700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63900a1e00 con 0x7f63900a29a0 2021-02-10T11:04:47.523 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:47.522 7f639502d700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63900a09f0 con 0x7f6390191bb0 2021-02-10T11:04:47.523 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:47.522 7f638f59e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f63900a33c0 0x7f63900a03d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:47.523 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:47.522 7f638f59e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f63900a33c0 0x7f63900a03d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:50210/0 (socket says 172.21.2.102:50210) 2021-02-10T11:04:47.523 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:47.522 7f638f59e700 1 -- 172.21.2.102:0/4044529300 learned_addr learned my addr 172.21.2.102:0/4044529300 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:04:47.524 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:47.523 7f638ed9d700 1 --2- 172.21.2.102:0/4044529300 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6390191bb0 0x7f63900a2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:47.524 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:47.523 7f638f59e700 1 -- 172.21.2.102:0/4044529300 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6390191bb0 msgr2=0x7f63900a2380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:47.524 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:47.523 7f638f59e700 1 --2- 172.21.2.102:0/4044529300 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6390191bb0 0x7f63900a2380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:47.524 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:47.523 7f638e59c700 1 --2- 172.21.2.102:0/4044529300 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f63900a29a0 0x7f63900a2da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:47.524 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:47.523 7f638f59e700 1 -- 172.21.2.102:0/4044529300 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f63900a29a0 msgr2=0x7f63900a2da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:47.524 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:47.523 7f638f59e700 1 --2- 172.21.2.102:0/4044529300 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f63900a29a0 0x7f63900a2da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:47.524 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:47.523 7f638f59e700 1 -- 172.21.2.102:0/4044529300 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63900a0bd0 con 0x7f63900a33c0 2021-02-10T11:04:47.524 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:47.523 7f638ed9d700 1 --2- 172.21.2.102:0/4044529300 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6390191bb0 0x7f63900a2380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:04:47.524 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:47.523 7f638e59c700 1 --2- 172.21.2.102:0/4044529300 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f63900a29a0 0x7f63900a2da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:04:47.524 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:47.523 7f638f59e700 1 --2- 172.21.2.102:0/4044529300 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f63900a33c0 0x7f63900a03d0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7f6388006e60 tx=0x7f638800bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:04:47.524 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:47.523 7f63877fe700 1 -- 172.21.2.102:0/4044529300 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6388009130 con 0x7f63900a33c0 2021-02-10T11:04:47.524 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:47.523 7f63877fe700 1 -- 172.21.2.102:0/4044529300 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63880093b0 con 0x7f63900a33c0 2021-02-10T11:04:47.524 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:47.523 7f639502d700 1 -- 172.21.2.102:0/4044529300 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63900a0bd0 con 0x7f63900a33c0 2021-02-10T11:04:47.524 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:47.523 7f639502d700 1 -- 172.21.2.102:0/4044529300 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63900a1160 con 0x7f63900a33c0 2021-02-10T11:04:47.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:47.523 7f63877fe700 1 -- 172.21.2.102:0/4044529300 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6388001390 con 0x7f63900a33c0 2021-02-10T11:04:47.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:47.524 7f63877fe700 1 -- 172.21.2.102:0/4044529300 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f63880011a0 con 0x7f63900a33c0 2021-02-10T11:04:47.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:47.524 7f63877fe700 1 --2- 172.21.2.102:0/4044529300 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6374018730 0x7f637401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:47.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:47.524 7f63877fe700 1 -- 172.21.2.102:0/4044529300 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f6388032ba0 con 0x7f63900a33c0 2021-02-10T11:04:47.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:47.524 7f638ed9d700 1 --2- 172.21.2.102:0/4044529300 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6374018730 0x7f637401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:47.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:47.525 7f638ed9d700 1 --2- 172.21.2.102:0/4044529300 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6374018730 0x7f637401ac10 crc :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:04:47.552 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:47.554 7f639502d700 1 -- 172.21.2.102:0/4044529300 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63900a1160 con 0x7f63900a33c0 2021-02-10T11:04:47.554 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:47.556 7f63877fe700 1 -- 172.21.2.102:0/4044529300 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f6388009390 con 0x7f63900a33c0 2021-02-10T11:04:47.687 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:47.689 7f639502d700 1 -- 172.21.2.102:0/4044529300 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestVolumes.test_async_subvolume_rm"]} v 0) v1 -- 0x7f63900a1160 con 0x7f63900a33c0 2021-02-10T11:04:47.772 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:47.774 7f63877fe700 1 -- 172.21.2.102:0/4044529300 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestVolumes.test_async_subvolume_rm"]}]=0 v70) v1 ==== 143+0+0 (secure 0 0 0) 0x7f63880011a0 con 0x7f63900a33c0 2021-02-10T11:04:47.804 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:47.806 7f639502d700 1 -- 172.21.2.102:0/4044529300 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6374018730 msgr2=0x7f637401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:47.804 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:47.806 7f639502d700 1 --2- 172.21.2.102:0/4044529300 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6374018730 0x7f637401ac10 crc :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:47.804 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:47.806 7f639502d700 1 -- 172.21.2.102:0/4044529300 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f63900a33c0 msgr2=0x7f63900a03d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:47.804 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:47.806 7f639502d700 1 --2- 172.21.2.102:0/4044529300 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f63900a33c0 0x7f63900a03d0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7f6388006e60 tx=0x7f638800bcf0).stop 2021-02-10T11:04:47.804 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:47.806 7f639502d700 1 -- 172.21.2.102:0/4044529300 shutdown_connections 2021-02-10T11:04:47.804 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:47.806 7f639502d700 1 --2- 172.21.2.102:0/4044529300 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f63900a33c0 0x7f63900a03d0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:47.804 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:47.806 7f639502d700 1 --2- 172.21.2.102:0/4044529300 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6374018730 0x7f637401ac10 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:47.805 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:47.806 7f639502d700 1 --2- 172.21.2.102:0/4044529300 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f63900a29a0 0x7f63900a2da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:47.805 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:47.806 7f639502d700 1 --2- 172.21.2.102:0/4044529300 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6390191bb0 0x7f63900a2380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:47.805 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:47.806 7f639502d700 1 -- 172.21.2.102:0/4044529300 >> 172.21.2.102:0/4044529300 conn(0x7f63901a31d0 msgr2=0x7f639008b830 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:04:47.805 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:47.806 7f639502d700 1 -- 172.21.2.102:0/4044529300 shutdown_connections 2021-02-10T11:04:47.805 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:47.806 7f639502d700 1 -- 172.21.2.102:0/4044529300 wait complete. 2021-02-10T11:04:47.811 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config set mon mon_allow_pool_delete True 2021-02-10T11:04:48.241 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:48.225 7f537c84f700 1 Processor -- start 2021-02-10T11:04:48.241 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:48.225 7f537c84f700 1 -- start start 2021-02-10T11:04:48.241 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:48.226 7f537c84f700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f53781b0470 0x7f53781b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:48.241 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:48.226 7f537c84f700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f53780ef610 0x7f53780efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:48.241 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:48.226 7f537c84f700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53780f0b50 con 0x7f53780f0030 2021-02-10T11:04:48.241 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:48.226 7f537c84f700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53780e52e0 con 0x7f53781b0470 2021-02-10T11:04:48.241 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:48.226 7f537c84f700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53780e54d0 con 0x7f53780ef610 2021-02-10T11:04:48.241 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:48.227 7f5376e1d700 1 --1- >> v1:172.21.2.113:6789/0 conn(0x7f53780f0030 0x7f53780f10d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.113:6789/0 says I am v1:172.21.2.102:49764/0 (socket says 172.21.2.102:49764) 2021-02-10T11:04:48.241 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:48.227 7f5376e1d700 1 -- 172.21.2.102:0/264331837 learned_addr learned my addr 172.21.2.102:0/264331837 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:04:48.241 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:48.227 7f537661c700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f53781b0470 0x7f53781b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:48.242 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:48.227 7f5375e1b700 1 --2- 172.21.2.102:0/264331837 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f53780ef610 0x7f53780efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:48.242 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:48.227 7f5375e1b700 1 -- 172.21.2.102:0/264331837 >> v2:172.21.2.113:3300/0 conn(0x7f53781b0470 msgr2=0x7f53781b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:04:48.242 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:48.227 7f5375e1b700 1 --2- 172.21.2.102:0/264331837 >> v2:172.21.2.113:3300/0 conn(0x7f53781b0470 0x7f53781b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:48.242 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:48.227 7f5375e1b700 1 -- 172.21.2.102:0/264331837 >> v1:172.21.2.113:6789/0 conn(0x7f53780f0030 legacy=0x7f53780f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:48.242 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:48.227 7f5375e1b700 1 -- 172.21.2.102:0/264331837 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53780f0430 con 0x7f53780ef610 2021-02-10T11:04:48.242 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:48.227 7f5375e1b700 1 --2- 172.21.2.102:0/264331837 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f53780ef610 0x7f53780efa10 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f5368006650 tx=0x7f5368009a80).ready entity=mon.2 client_cookie=e626b6e0d6c97c9b server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:04:48.242 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:48.228 7f537561a700 1 -- 172.21.2.102:0/264331837 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f53680009b0 con 0x7f53780ef610 2021-02-10T11:04:48.242 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:48.228 7f537561a700 1 -- 172.21.2.102:0/264331837 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f536800c0c0 con 0x7f53780ef610 2021-02-10T11:04:48.242 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:48.228 7f537561a700 1 -- 172.21.2.102:0/264331837 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f536800c5e0 con 0x7f53780ef610 2021-02-10T11:04:48.242 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:48.228 7f537c84f700 1 -- 172.21.2.102:0/264331837 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f53780ef610 msgr2=0x7f53780efa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:48.242 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:48.228 7f537c84f700 1 --2- 172.21.2.102:0/264331837 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f53780ef610 0x7f53780efa10 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f5368006650 tx=0x7f5368009a80).stop 2021-02-10T11:04:48.243 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:48.229 7f537c84f700 1 -- 172.21.2.102:0/264331837 shutdown_connections 2021-02-10T11:04:48.243 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:48.229 7f537c84f700 1 --2- 172.21.2.102:0/264331837 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f53780ef610 0x7f53780efa10 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:48.243 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:48.229 7f537c84f700 1 --2- 172.21.2.102:0/264331837 >> v2:172.21.2.113:3300/0 conn(0x7f53781b0470 0x7f53781b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:48.243 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:48.229 7f537c84f700 1 -- 172.21.2.102:0/264331837 >> 172.21.2.102:0/264331837 conn(0x7f53780ce3e0 msgr2=0x7f53780e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:04:48.243 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:48.229 7f537c84f700 1 -- 172.21.2.102:0/264331837 shutdown_connections 2021-02-10T11:04:48.243 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:48.229 7f537c84f700 1 -- 172.21.2.102:0/264331837 wait complete. 2021-02-10T11:04:48.246 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:48.243 7f537c84f700 1 Processor -- start 2021-02-10T11:04:48.246 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:48.243 7f537c84f700 1 -- start start 2021-02-10T11:04:48.246 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:48.244 7f537c84f700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5378105d10 0x7f537809eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:48.246 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:48.244 7f537c84f700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5378106110 0x7f537809f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:48.246 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:48.244 7f537c84f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f53780b2a00 0x7f53780b2e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:48.246 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:48.244 7f537c84f700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f537809e560 con 0x7f53780b2a00 2021-02-10T11:04:48.246 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:48.244 7f537c84f700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f537809c090 con 0x7f5378105d10 2021-02-10T11:04:48.246 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:48.244 7f537c84f700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53780b3420 con 0x7f5378106110 2021-02-10T11:04:48.246 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:48.244 7f5376e1d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f53780b2a00 0x7f53780b2e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:48.246 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:48.244 7f5376e1d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f53780b2a00 0x7f53780b2e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:50224/0 (socket says 172.21.2.102:50224) 2021-02-10T11:04:48.246 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:48.244 7f5376e1d700 1 -- 172.21.2.102:0/1639966004 learned_addr learned my addr 172.21.2.102:0/1639966004 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:04:48.247 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:48.244 7f537661c700 1 --2- 172.21.2.102:0/1639966004 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5378105d10 0x7f537809eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:48.247 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:48.244 7f5375e1b700 1 --2- 172.21.2.102:0/1639966004 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5378106110 0x7f537809f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:48.247 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:48.245 7f5376e1d700 1 -- 172.21.2.102:0/1639966004 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5378106110 msgr2=0x7f537809f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:48.247 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:48.245 7f5376e1d700 1 --2- 172.21.2.102:0/1639966004 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5378106110 0x7f537809f100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:48.247 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:48.245 7f5376e1d700 1 -- 172.21.2.102:0/1639966004 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5378105d10 msgr2=0x7f537809eae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:48.247 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:48.245 7f5376e1d700 1 --2- 172.21.2.102:0/1639966004 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5378105d10 0x7f537809eae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:48.247 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:48.245 7f5376e1d700 1 -- 172.21.2.102:0/1639966004 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53780b3640 con 0x7f53780b2a00 2021-02-10T11:04:48.247 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:48.245 7f5376e1d700 1 --2- 172.21.2.102:0/1639966004 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f53780b2a00 0x7f53780b2e00 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 rx=0x7f537000a900 tx=0x7f537000aca0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:04:48.247 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:48.245 7f536effd700 1 -- 172.21.2.102:0/1639966004 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5370007020 con 0x7f53780b2a00 2021-02-10T11:04:48.247 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:48.245 7f536effd700 1 -- 172.21.2.102:0/1639966004 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53700072e0 con 0x7f53780b2a00 2021-02-10T11:04:48.247 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:48.245 7f536effd700 1 -- 172.21.2.102:0/1639966004 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5370004920 con 0x7f53780b2a00 2021-02-10T11:04:48.247 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:48.245 7f537c84f700 1 -- 172.21.2.102:0/1639966004 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53780b3640 con 0x7f53780b2a00 2021-02-10T11:04:48.247 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:48.245 7f537c84f700 1 -- 172.21.2.102:0/1639966004 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53780b3bd0 con 0x7f53780b2a00 2021-02-10T11:04:48.247 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:48.246 7f536effd700 1 -- 172.21.2.102:0/1639966004 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f537000ade0 con 0x7f53780b2a00 2021-02-10T11:04:48.248 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:48.246 7f536effd700 1 --2- 172.21.2.102:0/1639966004 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f535c0186c0 0x7f535c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:48.248 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:48.246 7f536effd700 1 -- 172.21.2.102:0/1639966004 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f5370032740 con 0x7f53780b2a00 2021-02-10T11:04:48.248 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:48.246 7f537661c700 1 --2- 172.21.2.102:0/1639966004 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f535c0186c0 0x7f535c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:48.248 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:48.246 7f537661c700 1 --2- 172.21.2.102:0/1639966004 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f535c0186c0 0x7f535c01abc0 crc :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:04:48.273 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:48.276 7f537c84f700 1 -- 172.21.2.102:0/1639966004 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5378105ae0 con 0x7f53780b2a00 2021-02-10T11:04:48.276 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:48.278 7f536effd700 1 -- 172.21.2.102:0/1639966004 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f5370001230 con 0x7f53780b2a00 2021-02-10T11:04:48.424 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:48.426 7f537c84f700 1 -- 172.21.2.102:0/1639966004 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f53780f0b50 con 0x7f53780b2a00 2021-02-10T11:04:48.427 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:48.430 7f536effd700 1 -- 172.21.2.102:0/1639966004 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v2)=0 v2) v1 ==== 122+0+0 (secure 0 0 0) 0x7f537000ade0 con 0x7f53780b2a00 2021-02-10T11:04:48.433 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:48.435 7f537c84f700 1 -- 172.21.2.102:0/1639966004 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f535c0186c0 msgr2=0x7f535c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:48.433 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:48.435 7f537c84f700 1 --2- 172.21.2.102:0/1639966004 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f535c0186c0 0x7f535c01abc0 crc :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:48.433 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:48.435 7f537c84f700 1 -- 172.21.2.102:0/1639966004 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f53780b2a00 msgr2=0x7f53780b2e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:48.433 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:48.435 7f537c84f700 1 --2- 172.21.2.102:0/1639966004 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f53780b2a00 0x7f53780b2e00 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 rx=0x7f537000a900 tx=0x7f537000aca0).stop 2021-02-10T11:04:48.433 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:48.435 7f537c84f700 1 -- 172.21.2.102:0/1639966004 shutdown_connections 2021-02-10T11:04:48.433 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:48.435 7f537c84f700 1 --2- 172.21.2.102:0/1639966004 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f53780b2a00 0x7f53780b2e00 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:48.433 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:48.435 7f537c84f700 1 --2- 172.21.2.102:0/1639966004 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5378106110 0x7f537809f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:48.433 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:48.435 7f537c84f700 1 --2- 172.21.2.102:0/1639966004 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f535c0186c0 0x7f535c01abc0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:48.433 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:48.435 7f537c84f700 1 --2- 172.21.2.102:0/1639966004 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5378105d10 0x7f537809eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:48.433 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:48.435 7f537c84f700 1 -- 172.21.2.102:0/1639966004 >> 172.21.2.102:0/1639966004 conn(0x7f53780c36f0 msgr2=0x7f53780bd640 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:04:48.434 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:48.435 7f537c84f700 1 -- 172.21.2.102:0/1639966004 shutdown_connections 2021-02-10T11:04:48.434 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:48.435 7f537c84f700 1 -- 172.21.2.102:0/1639966004 wait complete. 2021-02-10T11:04:48.441 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2021-02-10T11:04:48.441 DEBUG:teuthology.orchestra.run.gibba002:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:04:48.864 DEBUG:teuthology.orchestra.run.gibba013:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:04:49.225 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2021-02-10T11:04:49.225 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:04:49.225 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:04:49.225 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:04:49.561 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:04:49.561 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:04:49.561 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:04:49.562 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:04:49.562 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:04:49.896 INFO:teuthology.orchestra.run.gibba002.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.0’: Transport endpoint is not connected 2021-02-10T11:04:49.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:04:49.896 WARNING:tasks.cephfs.fuse_mount:Found stale moutn point at /home/ubuntu/cephtest/mnt.0 2021-02-10T11:04:49.896 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@gibba002.front.sepia.ceph.com... 2021-02-10T11:04:49.897 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-02-10T11:04:49.897 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:04:50.262 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:04:50.263 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:04:50.263 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:04:50.607 INFO:teuthology.orchestra.run.gibba002.stdout:ext2/ext3 2021-02-10T11:04:50.607 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2021-02-10T11:04:50.608 INFO:teuthology.orchestra.run:waiting for 900 2021-02-10T11:04:50.608 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:04:50.608 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:04:50.967 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:04:50.967 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:04:50.967 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:04:51.305 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:04:51.306 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2021-02-10T11:04:51.306 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:04:51.306 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:04:51.306 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:04:51.654 INFO:teuthology.orchestra.run.gibba002.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.1’: Transport endpoint is not connected 2021-02-10T11:04:51.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:04:51.654 WARNING:tasks.cephfs.fuse_mount:Found stale moutn point at /home/ubuntu/cephtest/mnt.1 2021-02-10T11:04:51.654 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@gibba002.front.sepia.ceph.com... 2021-02-10T11:04:51.655 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-02-10T11:04:51.655 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:04:52.015 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:04:52.016 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:04:52.016 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:04:52.350 INFO:teuthology.orchestra.run.gibba002.stdout:ext2/ext3 2021-02-10T11:04:52.350 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2021-02-10T11:04:52.350 INFO:teuthology.orchestra.run:waiting for 900 2021-02-10T11:04:52.351 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:04:52.351 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:04:52.685 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:04:52.685 DEBUG:teuthology.orchestra.run.gibba013:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:04:52.685 DEBUG:teuthology.orchestra.run.gibba013:> ' -- /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:04:53.020 INFO:teuthology.orchestra.run.gibba013.stdout:fuseblk 2021-02-10T11:04:53.020 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.2 2021-02-10T11:04:53.021 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:04:53.021 DEBUG:teuthology.orchestra.run.gibba013:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:04:53.021 DEBUG:teuthology.orchestra.run.gibba013:> ' -- /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:04:53.357 INFO:teuthology.orchestra.run.gibba013.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.2’: Transport endpoint is not connected 2021-02-10T11:04:53.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:04:53.357 WARNING:tasks.cephfs.fuse_mount:Found stale moutn point at /home/ubuntu/cephtest/mnt.2 2021-02-10T11:04:53.357 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@gibba013.front.sepia.ceph.com... 2021-02-10T11:04:53.357 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-02-10T11:04:53.358 DEBUG:teuthology.orchestra.run.gibba013:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:04:53.719 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:04:53.720 DEBUG:teuthology.orchestra.run.gibba013:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:04:53.720 DEBUG:teuthology.orchestra.run.gibba013:> ' -- /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:04:54.055 INFO:teuthology.orchestra.run.gibba013.stdout:ext2/ext3 2021-02-10T11:04:54.055 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2021-02-10T11:04:54.056 INFO:teuthology.orchestra.run:waiting for 900 2021-02-10T11:04:54.056 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:04:54.056 DEBUG:teuthology.orchestra.run.gibba013:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:04:54.400 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:04:54.400 DEBUG:teuthology.orchestra.run.gibba013:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:04:54.400 DEBUG:teuthology.orchestra.run.gibba013:> ' -- /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:04:54.734 INFO:teuthology.orchestra.run.gibba013.stdout:fuseblk 2021-02-10T11:04:54.734 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.3 2021-02-10T11:04:54.735 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:04:54.735 DEBUG:teuthology.orchestra.run.gibba013:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:04:54.735 DEBUG:teuthology.orchestra.run.gibba013:> ' -- /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:04:55.070 INFO:teuthology.orchestra.run.gibba013.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.3’: Transport endpoint is not connected 2021-02-10T11:04:55.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:04:55.070 WARNING:tasks.cephfs.fuse_mount:Found stale moutn point at /home/ubuntu/cephtest/mnt.3 2021-02-10T11:04:55.070 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@gibba013.front.sepia.ceph.com... 2021-02-10T11:04:55.071 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-02-10T11:04:55.071 DEBUG:teuthology.orchestra.run.gibba013:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:04:55.425 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:04:55.426 DEBUG:teuthology.orchestra.run.gibba013:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:04:55.426 DEBUG:teuthology.orchestra.run.gibba013:> ' -- /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:04:55.759 INFO:teuthology.orchestra.run.gibba013.stdout:ext2/ext3 2021-02-10T11:04:55.759 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2021-02-10T11:04:55.760 INFO:teuthology.orchestra.run:waiting for 900 2021-02-10T11:04:55.760 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:04:55.760 DEBUG:teuthology.orchestra.run.gibba013:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:04:56.094 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json-pretty 2021-02-10T11:04:56.520 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:56.505 7f9680ae5700 1 Processor -- start 2021-02-10T11:04:56.520 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:56.505 7f9680ae5700 1 -- start start 2021-02-10T11:04:56.520 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:56.506 7f9680ae5700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f967c0e5070 0x7f967c1b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:56.520 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:56.506 7f9680ae5700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f967c1b3e40 0x7f967c07d690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:56.520 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:56.506 7f9680ae5700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f967c0e5470 0x7f967c07dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:56.520 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:56.506 7f9680ae5700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f967c07ed80 con 0x7f967c0e5070 2021-02-10T11:04:56.520 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:56.506 7f9680ae5700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f967c07e2d0 con 0x7f967c0e5470 2021-02-10T11:04:56.520 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:56.506 7f9680ae5700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f967c07e4c0 con 0x7f967c1b3e40 2021-02-10T11:04:56.520 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:56.506 7f967a85e700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f967c0e5070 0x7f967c1b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:56.521 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:56.506 7f967a85e700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f967c0e5070 0x7f967c1b3820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:50228/0 (socket says 172.21.2.102:50228) 2021-02-10T11:04:56.521 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:56.506 7f967a85e700 1 -- 172.21.2.102:0/3694435660 learned_addr learned my addr 172.21.2.102:0/3694435660 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:04:56.521 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:56.506 7f967a05d700 1 --2- 172.21.2.102:0/3694435660 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f967c1b3e40 0x7f967c07d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:56.521 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:56.506 7f967a85e700 1 -- 172.21.2.102:0/3694435660 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f967c1b3e40 msgr2=0x7f967c07d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:04:56.521 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:56.506 7f967a85e700 1 --2- 172.21.2.102:0/3694435660 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f967c1b3e40 0x7f967c07d690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:56.521 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:56.506 7f967a85e700 1 -- 172.21.2.102:0/3694435660 >> v2:172.21.2.113:3300/0 conn(0x7f967c0e5470 msgr2=0x7f967c07dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:04:56.521 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:56.506 7f967a85e700 1 --2- 172.21.2.102:0/3694435660 >> v2:172.21.2.113:3300/0 conn(0x7f967c0e5470 0x7f967c07dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:56.521 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:56.506 7f967a85e700 1 -- 172.21.2.102:0/3694435660 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f967c07e6e0 con 0x7f967c0e5070 2021-02-10T11:04:56.521 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:56.507 7f967a85e700 1 --2- 172.21.2.102:0/3694435660 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f967c0e5070 0x7f967c1b3820 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0x7f9664006700 tx=0x7f9664009b80).ready entity=mon.0 client_cookie=5066f6386dc36ee7 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:04:56.521 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:56.507 7f967985c700 1 -- 172.21.2.102:0/3694435660 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f966400c0c0 con 0x7f967c0e5070 2021-02-10T11:04:56.521 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:56.507 7f967985c700 1 -- 172.21.2.102:0/3694435660 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f966400c340 con 0x7f967c0e5070 2021-02-10T11:04:56.521 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:56.507 7f967985c700 1 -- 172.21.2.102:0/3694435660 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f966400c860 con 0x7f967c0e5070 2021-02-10T11:04:56.521 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:56.507 7f9680ae5700 1 -- 172.21.2.102:0/3694435660 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f967c0e5070 msgr2=0x7f967c1b3820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:56.522 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:56.507 7f9680ae5700 1 --2- 172.21.2.102:0/3694435660 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f967c0e5070 0x7f967c1b3820 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0x7f9664006700 tx=0x7f9664009b80).stop 2021-02-10T11:04:56.522 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:56.508 7f9680ae5700 1 -- 172.21.2.102:0/3694435660 shutdown_connections 2021-02-10T11:04:56.522 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:56.508 7f9680ae5700 1 --2- 172.21.2.102:0/3694435660 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f967c1b3e40 0x7f967c07d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:56.522 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:56.508 7f9680ae5700 1 --2- 172.21.2.102:0/3694435660 >> v2:172.21.2.113:3300/0 conn(0x7f967c0e5470 0x7f967c07dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:56.522 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:56.508 7f9680ae5700 1 --2- 172.21.2.102:0/3694435660 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f967c0e5070 0x7f967c1b3820 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:56.522 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:56.508 7f9680ae5700 1 -- 172.21.2.102:0/3694435660 >> 172.21.2.102:0/3694435660 conn(0x7f967c0ce690 msgr2=0x7f967c0e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:04:56.522 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:56.508 7f9680ae5700 1 -- 172.21.2.102:0/3694435660 shutdown_connections 2021-02-10T11:04:56.522 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:56.508 7f9680ae5700 1 -- 172.21.2.102:0/3694435660 wait complete. 2021-02-10T11:04:56.523 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:56.522 7f9680ae5700 1 Processor -- start 2021-02-10T11:04:56.524 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:56.522 7f9680ae5700 1 -- start start 2021-02-10T11:04:56.524 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:56.523 7f9680ae5700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f967c0b1970 0x7f967c0b1d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:56.524 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:56.523 7f9680ae5700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f967c08acb0 0x7f967c08b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:56.524 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:56.523 7f9680ae5700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f967c08b6d0 0x7f967c1aac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:56.524 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:56.523 7f9680ae5700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f967c08bba0 con 0x7f967c0b1970 2021-02-10T11:04:56.524 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:56.523 7f9680ae5700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f967c1ab280 con 0x7f967c08acb0 2021-02-10T11:04:56.524 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:56.523 7f9680ae5700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f967c1ab4a0 con 0x7f967c08b6d0 2021-02-10T11:04:56.524 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:56.523 7f967a85e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f967c0b1970 0x7f967c0b1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:56.524 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:56.523 7f967a85e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f967c0b1970 0x7f967c0b1d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:50234/0 (socket says 172.21.2.102:50234) 2021-02-10T11:04:56.524 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:56.523 7f967a85e700 1 -- 172.21.2.102:0/1485864533 learned_addr learned my addr 172.21.2.102:0/1485864533 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:04:56.524 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:56.523 7f967a05d700 1 --2- 172.21.2.102:0/1485864533 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f967c08acb0 0x7f967c08b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:56.524 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:56.523 7f967b05f700 1 --2- 172.21.2.102:0/1485864533 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f967c08b6d0 0x7f967c1aac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:56.524 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:56.523 7f967a85e700 1 -- 172.21.2.102:0/1485864533 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f967c08b6d0 msgr2=0x7f967c1aac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:56.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:56.523 7f967a85e700 1 --2- 172.21.2.102:0/1485864533 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f967c08b6d0 0x7f967c1aac60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:56.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:56.523 7f967a85e700 1 -- 172.21.2.102:0/1485864533 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f967c08acb0 msgr2=0x7f967c08b0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:56.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:56.523 7f967a85e700 1 --2- 172.21.2.102:0/1485864533 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f967c08acb0 0x7f967c08b0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:56.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:56.523 7f967a85e700 1 -- 172.21.2.102:0/1485864533 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f967c1ab6c0 con 0x7f967c0b1970 2021-02-10T11:04:56.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:56.524 7f967a85e700 1 --2- 172.21.2.102:0/1485864533 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f967c0b1970 0x7f967c0b1d70 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 rx=0x7f966400b5e0 tx=0x7f966400b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:04:56.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:56.524 7f96737fe700 1 -- 172.21.2.102:0/1485864533 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f966400bd70 con 0x7f967c0b1970 2021-02-10T11:04:56.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:56.524 7f96737fe700 1 -- 172.21.2.102:0/1485864533 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9664002030 con 0x7f967c0b1970 2021-02-10T11:04:56.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:56.524 7f96737fe700 1 -- 172.21.2.102:0/1485864533 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9664002550 con 0x7f967c0b1970 2021-02-10T11:04:56.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:56.524 7f9680ae5700 1 -- 172.21.2.102:0/1485864533 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f967c1ab6c0 con 0x7f967c0b1970 2021-02-10T11:04:56.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:56.524 7f9680ae5700 1 -- 172.21.2.102:0/1485864533 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f967c08c0a0 con 0x7f967c0b1970 2021-02-10T11:04:56.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:56.525 7f96737fe700 1 -- 172.21.2.102:0/1485864533 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f9664000c90 con 0x7f967c0b1970 2021-02-10T11:04:56.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:56.525 7f96737fe700 1 --2- 172.21.2.102:0/1485864533 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f96600186c0 0x7f966001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:56.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:56.525 7f96737fe700 1 -- 172.21.2.102:0/1485864533 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f96640325c0 con 0x7f967c0b1970 2021-02-10T11:04:56.526 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:56.525 7f967a05d700 1 --2- 172.21.2.102:0/1485864533 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f96600186c0 0x7f966001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:56.526 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:56.525 7f967a05d700 1 --2- 172.21.2.102:0/1485864533 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f96600186c0 0x7f966001abc0 crc :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:04:56.554 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:56.556 7f9680ae5700 1 -- 172.21.2.102:0/1485864533 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f967c107180 con 0x7f967c0b1970 2021-02-10T11:04:56.556 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:56.559 7f96737fe700 1 -- 172.21.2.102:0/1485864533 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f9664030d70 con 0x7f967c0b1970 2021-02-10T11:04:56.681 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:56.683 7f9680ae5700 1 -- 172.21.2.102:0/1485864533 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7f967c08bb20 con 0x7f967c0b1970 2021-02-10T11:04:56.682 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:56.684 7f96737fe700 1 -- 172.21.2.102:0/1485864533 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v20) v1 ==== 81+0+30360 (secure 0 0 0) 0x7f9664030d70 con 0x7f967c0b1970 2021-02-10T11:04:56.682 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:04:56.683 INFO:teuthology.orchestra.run.gibba002.stdout:{ 2021-02-10T11:04:56.683 INFO:teuthology.orchestra.run.gibba002.stdout: "epoch": 20, 2021-02-10T11:04:56.683 INFO:teuthology.orchestra.run.gibba002.stdout: "fsid": "68db30ea-dbce-4aef-a6df-f01c6fea0513", 2021-02-10T11:04:56.683 INFO:teuthology.orchestra.run.gibba002.stdout: "created": "2021-02-10 11:03:06.331557", 2021-02-10T11:04:56.683 INFO:teuthology.orchestra.run.gibba002.stdout: "modified": "2021-02-10 11:03:44.386945", 2021-02-10T11:04:56.683 INFO:teuthology.orchestra.run.gibba002.stdout: "last_up_change": "2021-02-10 11:03:32.690562", 2021-02-10T11:04:56.684 INFO:teuthology.orchestra.run.gibba002.stdout: "last_in_change": "2021-02-10 11:03:32.690562", 2021-02-10T11:04:56.684 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2021-02-10T11:04:56.684 INFO:teuthology.orchestra.run.gibba002.stdout: "flags_num": 5799936, 2021-02-10T11:04:56.684 INFO:teuthology.orchestra.run.gibba002.stdout: "flags_set": [ 2021-02-10T11:04:56.684 INFO:teuthology.orchestra.run.gibba002.stdout: "pglog_hardlimit", 2021-02-10T11:04:56.685 INFO:teuthology.orchestra.run.gibba002.stdout: "purged_snapdirs", 2021-02-10T11:04:56.685 INFO:teuthology.orchestra.run.gibba002.stdout: "recovery_deletes", 2021-02-10T11:04:56.685 INFO:teuthology.orchestra.run.gibba002.stdout: "sortbitwise" 2021-02-10T11:04:56.685 INFO:teuthology.orchestra.run.gibba002.stdout: ], 2021-02-10T11:04:56.685 INFO:teuthology.orchestra.run.gibba002.stdout: "crush_version": 4, 2021-02-10T11:04:56.686 INFO:teuthology.orchestra.run.gibba002.stdout: "full_ratio": 0.94999998807907104, 2021-02-10T11:04:56.686 INFO:teuthology.orchestra.run.gibba002.stdout: "backfillfull_ratio": 0.89999997615814209, 2021-02-10T11:04:56.686 INFO:teuthology.orchestra.run.gibba002.stdout: "nearfull_ratio": 0.85000002384185791, 2021-02-10T11:04:56.687 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_snapshot": "", 2021-02-10T11:04:56.687 INFO:teuthology.orchestra.run.gibba002.stdout: "pool_max": 3, 2021-02-10T11:04:56.687 INFO:teuthology.orchestra.run.gibba002.stdout: "max_osd": 8, 2021-02-10T11:04:56.687 INFO:teuthology.orchestra.run.gibba002.stdout: "require_min_compat_client": "jewel", 2021-02-10T11:04:56.688 INFO:teuthology.orchestra.run.gibba002.stdout: "min_compat_client": "jewel", 2021-02-10T11:04:56.688 INFO:teuthology.orchestra.run.gibba002.stdout: "require_osd_release": "nautilus", 2021-02-10T11:04:56.688 INFO:teuthology.orchestra.run.gibba002.stdout: "pools": [ 2021-02-10T11:04:56.688 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:04:56.688 INFO:teuthology.orchestra.run.gibba002.stdout: "pool": 1, 2021-02-10T11:04:56.689 INFO:teuthology.orchestra.run.gibba002.stdout: "pool_name": "rbd", 2021-02-10T11:04:56.689 INFO:teuthology.orchestra.run.gibba002.stdout: "create_time": "2021-02-10 11:03:37.322519", 2021-02-10T11:04:56.689 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 1, 2021-02-10T11:04:56.690 INFO:teuthology.orchestra.run.gibba002.stdout: "flags_names": "hashpspool", 2021-02-10T11:04:56.690 INFO:teuthology.orchestra.run.gibba002.stdout: "type": 1, 2021-02-10T11:04:56.690 INFO:teuthology.orchestra.run.gibba002.stdout: "size": 2, 2021-02-10T11:04:56.690 INFO:teuthology.orchestra.run.gibba002.stdout: "min_size": 1, 2021-02-10T11:04:56.690 INFO:teuthology.orchestra.run.gibba002.stdout: "crush_rule": 0, 2021-02-10T11:04:56.691 INFO:teuthology.orchestra.run.gibba002.stdout: "object_hash": 2, 2021-02-10T11:04:56.691 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:04:56.691 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num": 8, 2021-02-10T11:04:56.691 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_placement_num": 8, 2021-02-10T11:04:56.691 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_placement_num_target": 8, 2021-02-10T11:04:56.692 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num_target": 8, 2021-02-10T11:04:56.692 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num_pending": 8, 2021-02-10T11:04:56.692 INFO:teuthology.orchestra.run.gibba002.stdout: "last_pg_merge_meta": { 2021-02-10T11:04:56.692 INFO:teuthology.orchestra.run.gibba002.stdout: "source_pgid": "0.0", 2021-02-10T11:04:56.692 INFO:teuthology.orchestra.run.gibba002.stdout: "ready_epoch": 0, 2021-02-10T11:04:56.692 INFO:teuthology.orchestra.run.gibba002.stdout: "last_epoch_started": 0, 2021-02-10T11:04:56.693 INFO:teuthology.orchestra.run.gibba002.stdout: "last_epoch_clean": 0, 2021-02-10T11:04:56.693 INFO:teuthology.orchestra.run.gibba002.stdout: "source_version": "0'0", 2021-02-10T11:04:56.693 INFO:teuthology.orchestra.run.gibba002.stdout: "target_version": "0'0" 2021-02-10T11:04:56.693 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:56.693 INFO:teuthology.orchestra.run.gibba002.stdout: "last_change": "16", 2021-02-10T11:04:56.694 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend": "0", 2021-02-10T11:04:56.694 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:04:56.694 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:04:56.694 INFO:teuthology.orchestra.run.gibba002.stdout: "auid": 0, 2021-02-10T11:04:56.694 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:04:56.695 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_seq": 0, 2021-02-10T11:04:56.695 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_epoch": 0, 2021-02-10T11:04:56.695 INFO:teuthology.orchestra.run.gibba002.stdout: "pool_snaps": [], 2021-02-10T11:04:56.695 INFO:teuthology.orchestra.run.gibba002.stdout: "removed_snaps": "[]", 2021-02-10T11:04:56.695 INFO:teuthology.orchestra.run.gibba002.stdout: "quota_max_bytes": 0, 2021-02-10T11:04:56.696 INFO:teuthology.orchestra.run.gibba002.stdout: "quota_max_objects": 0, 2021-02-10T11:04:56.696 INFO:teuthology.orchestra.run.gibba002.stdout: "tiers": [], 2021-02-10T11:04:56.696 INFO:teuthology.orchestra.run.gibba002.stdout: "tier_of": -1, 2021-02-10T11:04:56.696 INFO:teuthology.orchestra.run.gibba002.stdout: "read_tier": -1, 2021-02-10T11:04:56.696 INFO:teuthology.orchestra.run.gibba002.stdout: "write_tier": -1, 2021-02-10T11:04:56.697 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_mode": "none", 2021-02-10T11:04:56.697 INFO:teuthology.orchestra.run.gibba002.stdout: "target_max_bytes": 0, 2021-02-10T11:04:56.697 INFO:teuthology.orchestra.run.gibba002.stdout: "target_max_objects": 0, 2021-02-10T11:04:56.697 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:04:56.697 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:04:56.698 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:04:56.698 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_min_flush_age": 0, 2021-02-10T11:04:56.698 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_min_evict_age": 0, 2021-02-10T11:04:56.698 INFO:teuthology.orchestra.run.gibba002.stdout: "erasure_code_profile": "", 2021-02-10T11:04:56.698 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_params": { 2021-02-10T11:04:56.699 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "none" 2021-02-10T11:04:56.699 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:56.699 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_period": 0, 2021-02-10T11:04:56.699 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_count": 0, 2021-02-10T11:04:56.699 INFO:teuthology.orchestra.run.gibba002.stdout: "use_gmt_hitset": true, 2021-02-10T11:04:56.700 INFO:teuthology.orchestra.run.gibba002.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:04:56.700 INFO:teuthology.orchestra.run.gibba002.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:04:56.700 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:04:56.700 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:04:56.700 INFO:teuthology.orchestra.run.gibba002.stdout: "grade_table": [], 2021-02-10T11:04:56.701 INFO:teuthology.orchestra.run.gibba002.stdout: "stripe_width": 0, 2021-02-10T11:04:56.701 INFO:teuthology.orchestra.run.gibba002.stdout: "expected_num_objects": 0, 2021-02-10T11:04:56.701 INFO:teuthology.orchestra.run.gibba002.stdout: "fast_read": false, 2021-02-10T11:04:56.701 INFO:teuthology.orchestra.run.gibba002.stdout: "options": {}, 2021-02-10T11:04:56.701 INFO:teuthology.orchestra.run.gibba002.stdout: "application_metadata": { 2021-02-10T11:04:56.701 INFO:teuthology.orchestra.run.gibba002.stdout: "rbd": {} 2021-02-10T11:04:56.702 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:04:56.702 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:56.702 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:04:56.702 INFO:teuthology.orchestra.run.gibba002.stdout: "pool": 2, 2021-02-10T11:04:56.702 INFO:teuthology.orchestra.run.gibba002.stdout: "pool_name": "cephfs_metadata", 2021-02-10T11:04:56.703 INFO:teuthology.orchestra.run.gibba002.stdout: "create_time": "2021-02-10 11:03:41.161563", 2021-02-10T11:04:56.703 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 1, 2021-02-10T11:04:56.703 INFO:teuthology.orchestra.run.gibba002.stdout: "flags_names": "hashpspool", 2021-02-10T11:04:56.703 INFO:teuthology.orchestra.run.gibba002.stdout: "type": 1, 2021-02-10T11:04:56.703 INFO:teuthology.orchestra.run.gibba002.stdout: "size": 2, 2021-02-10T11:04:56.704 INFO:teuthology.orchestra.run.gibba002.stdout: "min_size": 1, 2021-02-10T11:04:56.704 INFO:teuthology.orchestra.run.gibba002.stdout: "crush_rule": 0, 2021-02-10T11:04:56.704 INFO:teuthology.orchestra.run.gibba002.stdout: "object_hash": 2, 2021-02-10T11:04:56.704 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:04:56.704 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num": 8, 2021-02-10T11:04:56.704 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_placement_num": 8, 2021-02-10T11:04:56.704 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_placement_num_target": 8, 2021-02-10T11:04:56.705 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num_target": 8, 2021-02-10T11:04:56.705 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num_pending": 8, 2021-02-10T11:04:56.705 INFO:teuthology.orchestra.run.gibba002.stdout: "last_pg_merge_meta": { 2021-02-10T11:04:56.705 INFO:teuthology.orchestra.run.gibba002.stdout: "source_pgid": "0.0", 2021-02-10T11:04:56.705 INFO:teuthology.orchestra.run.gibba002.stdout: "ready_epoch": 0, 2021-02-10T11:04:56.706 INFO:teuthology.orchestra.run.gibba002.stdout: "last_epoch_started": 0, 2021-02-10T11:04:56.706 INFO:teuthology.orchestra.run.gibba002.stdout: "last_epoch_clean": 0, 2021-02-10T11:04:56.706 INFO:teuthology.orchestra.run.gibba002.stdout: "source_version": "0'0", 2021-02-10T11:04:56.706 INFO:teuthology.orchestra.run.gibba002.stdout: "target_version": "0'0" 2021-02-10T11:04:56.707 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:56.707 INFO:teuthology.orchestra.run.gibba002.stdout: "last_change": "19", 2021-02-10T11:04:56.707 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend": "0", 2021-02-10T11:04:56.707 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:04:56.707 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:04:56.707 INFO:teuthology.orchestra.run.gibba002.stdout: "auid": 0, 2021-02-10T11:04:56.708 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:04:56.708 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_seq": 0, 2021-02-10T11:04:56.708 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_epoch": 0, 2021-02-10T11:04:56.708 INFO:teuthology.orchestra.run.gibba002.stdout: "pool_snaps": [], 2021-02-10T11:04:56.708 INFO:teuthology.orchestra.run.gibba002.stdout: "removed_snaps": "[]", 2021-02-10T11:04:56.709 INFO:teuthology.orchestra.run.gibba002.stdout: "quota_max_bytes": 0, 2021-02-10T11:04:56.709 INFO:teuthology.orchestra.run.gibba002.stdout: "quota_max_objects": 0, 2021-02-10T11:04:56.709 INFO:teuthology.orchestra.run.gibba002.stdout: "tiers": [], 2021-02-10T11:04:56.709 INFO:teuthology.orchestra.run.gibba002.stdout: "tier_of": -1, 2021-02-10T11:04:56.709 INFO:teuthology.orchestra.run.gibba002.stdout: "read_tier": -1, 2021-02-10T11:04:56.710 INFO:teuthology.orchestra.run.gibba002.stdout: "write_tier": -1, 2021-02-10T11:04:56.710 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_mode": "none", 2021-02-10T11:04:56.710 INFO:teuthology.orchestra.run.gibba002.stdout: "target_max_bytes": 0, 2021-02-10T11:04:56.710 INFO:teuthology.orchestra.run.gibba002.stdout: "target_max_objects": 0, 2021-02-10T11:04:56.710 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:04:56.711 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:04:56.711 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:04:56.711 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_min_flush_age": 0, 2021-02-10T11:04:56.711 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_min_evict_age": 0, 2021-02-10T11:04:56.711 INFO:teuthology.orchestra.run.gibba002.stdout: "erasure_code_profile": "", 2021-02-10T11:04:56.712 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_params": { 2021-02-10T11:04:56.712 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "none" 2021-02-10T11:04:56.712 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:56.712 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_period": 0, 2021-02-10T11:04:56.712 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_count": 0, 2021-02-10T11:04:56.713 INFO:teuthology.orchestra.run.gibba002.stdout: "use_gmt_hitset": true, 2021-02-10T11:04:56.713 INFO:teuthology.orchestra.run.gibba002.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:04:56.713 INFO:teuthology.orchestra.run.gibba002.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:04:56.713 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:04:56.713 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:04:56.714 INFO:teuthology.orchestra.run.gibba002.stdout: "grade_table": [], 2021-02-10T11:04:56.714 INFO:teuthology.orchestra.run.gibba002.stdout: "stripe_width": 0, 2021-02-10T11:04:56.714 INFO:teuthology.orchestra.run.gibba002.stdout: "expected_num_objects": 0, 2021-02-10T11:04:56.714 INFO:teuthology.orchestra.run.gibba002.stdout: "fast_read": false, 2021-02-10T11:04:56.714 INFO:teuthology.orchestra.run.gibba002.stdout: "options": { 2021-02-10T11:04:56.714 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_autoscale_bias": 4, 2021-02-10T11:04:56.714 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num_min": 16, 2021-02-10T11:04:56.715 INFO:teuthology.orchestra.run.gibba002.stdout: "recovery_priority": 5 2021-02-10T11:04:56.715 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:56.715 INFO:teuthology.orchestra.run.gibba002.stdout: "application_metadata": { 2021-02-10T11:04:56.715 INFO:teuthology.orchestra.run.gibba002.stdout: "cephfs": { 2021-02-10T11:04:56.715 INFO:teuthology.orchestra.run.gibba002.stdout: "metadata": "cephfs" 2021-02-10T11:04:56.716 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:04:56.716 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:04:56.716 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:56.716 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:04:56.716 INFO:teuthology.orchestra.run.gibba002.stdout: "pool": 3, 2021-02-10T11:04:56.716 INFO:teuthology.orchestra.run.gibba002.stdout: "pool_name": "cephfs_data", 2021-02-10T11:04:56.716 INFO:teuthology.orchestra.run.gibba002.stdout: "create_time": "2021-02-10 11:03:42.379651", 2021-02-10T11:04:56.717 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 1, 2021-02-10T11:04:56.717 INFO:teuthology.orchestra.run.gibba002.stdout: "flags_names": "hashpspool", 2021-02-10T11:04:56.717 INFO:teuthology.orchestra.run.gibba002.stdout: "type": 1, 2021-02-10T11:04:56.717 INFO:teuthology.orchestra.run.gibba002.stdout: "size": 2, 2021-02-10T11:04:56.717 INFO:teuthology.orchestra.run.gibba002.stdout: "min_size": 1, 2021-02-10T11:04:56.718 INFO:teuthology.orchestra.run.gibba002.stdout: "crush_rule": 0, 2021-02-10T11:04:56.718 INFO:teuthology.orchestra.run.gibba002.stdout: "object_hash": 2, 2021-02-10T11:04:56.718 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:04:56.718 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num": 8, 2021-02-10T11:04:56.718 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_placement_num": 8, 2021-02-10T11:04:56.719 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_placement_num_target": 8, 2021-02-10T11:04:56.719 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num_target": 8, 2021-02-10T11:04:56.719 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num_pending": 8, 2021-02-10T11:04:56.719 INFO:teuthology.orchestra.run.gibba002.stdout: "last_pg_merge_meta": { 2021-02-10T11:04:56.719 INFO:teuthology.orchestra.run.gibba002.stdout: "source_pgid": "0.0", 2021-02-10T11:04:56.720 INFO:teuthology.orchestra.run.gibba002.stdout: "ready_epoch": 0, 2021-02-10T11:04:56.720 INFO:teuthology.orchestra.run.gibba002.stdout: "last_epoch_started": 0, 2021-02-10T11:04:56.720 INFO:teuthology.orchestra.run.gibba002.stdout: "last_epoch_clean": 0, 2021-02-10T11:04:56.720 INFO:teuthology.orchestra.run.gibba002.stdout: "source_version": "0'0", 2021-02-10T11:04:56.720 INFO:teuthology.orchestra.run.gibba002.stdout: "target_version": "0'0" 2021-02-10T11:04:56.721 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:56.721 INFO:teuthology.orchestra.run.gibba002.stdout: "last_change": "20", 2021-02-10T11:04:56.721 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend": "0", 2021-02-10T11:04:56.721 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:04:56.721 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:04:56.722 INFO:teuthology.orchestra.run.gibba002.stdout: "auid": 0, 2021-02-10T11:04:56.722 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:04:56.722 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_seq": 0, 2021-02-10T11:04:56.722 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_epoch": 0, 2021-02-10T11:04:56.722 INFO:teuthology.orchestra.run.gibba002.stdout: "pool_snaps": [], 2021-02-10T11:04:56.723 INFO:teuthology.orchestra.run.gibba002.stdout: "removed_snaps": "[]", 2021-02-10T11:04:56.723 INFO:teuthology.orchestra.run.gibba002.stdout: "quota_max_bytes": 0, 2021-02-10T11:04:56.723 INFO:teuthology.orchestra.run.gibba002.stdout: "quota_max_objects": 0, 2021-02-10T11:04:56.723 INFO:teuthology.orchestra.run.gibba002.stdout: "tiers": [], 2021-02-10T11:04:56.723 INFO:teuthology.orchestra.run.gibba002.stdout: "tier_of": -1, 2021-02-10T11:04:56.724 INFO:teuthology.orchestra.run.gibba002.stdout: "read_tier": -1, 2021-02-10T11:04:56.724 INFO:teuthology.orchestra.run.gibba002.stdout: "write_tier": -1, 2021-02-10T11:04:56.724 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_mode": "none", 2021-02-10T11:04:56.724 INFO:teuthology.orchestra.run.gibba002.stdout: "target_max_bytes": 0, 2021-02-10T11:04:56.724 INFO:teuthology.orchestra.run.gibba002.stdout: "target_max_objects": 0, 2021-02-10T11:04:56.725 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:04:56.725 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:04:56.725 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:04:56.725 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_min_flush_age": 0, 2021-02-10T11:04:56.725 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_min_evict_age": 0, 2021-02-10T11:04:56.726 INFO:teuthology.orchestra.run.gibba002.stdout: "erasure_code_profile": "", 2021-02-10T11:04:56.726 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_params": { 2021-02-10T11:04:56.726 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "none" 2021-02-10T11:04:56.726 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:56.726 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_period": 0, 2021-02-10T11:04:56.726 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_count": 0, 2021-02-10T11:04:56.727 INFO:teuthology.orchestra.run.gibba002.stdout: "use_gmt_hitset": true, 2021-02-10T11:04:56.727 INFO:teuthology.orchestra.run.gibba002.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:04:56.727 INFO:teuthology.orchestra.run.gibba002.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:04:56.727 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:04:56.727 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:04:56.728 INFO:teuthology.orchestra.run.gibba002.stdout: "grade_table": [], 2021-02-10T11:04:56.728 INFO:teuthology.orchestra.run.gibba002.stdout: "stripe_width": 0, 2021-02-10T11:04:56.728 INFO:teuthology.orchestra.run.gibba002.stdout: "expected_num_objects": 0, 2021-02-10T11:04:56.728 INFO:teuthology.orchestra.run.gibba002.stdout: "fast_read": false, 2021-02-10T11:04:56.728 INFO:teuthology.orchestra.run.gibba002.stdout: "options": {}, 2021-02-10T11:04:56.729 INFO:teuthology.orchestra.run.gibba002.stdout: "application_metadata": { 2021-02-10T11:04:56.729 INFO:teuthology.orchestra.run.gibba002.stdout: "cephfs": { 2021-02-10T11:04:56.729 INFO:teuthology.orchestra.run.gibba002.stdout: "data": "cephfs" 2021-02-10T11:04:56.729 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:04:56.729 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:04:56.729 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:04:56.730 INFO:teuthology.orchestra.run.gibba002.stdout: ], 2021-02-10T11:04:56.730 INFO:teuthology.orchestra.run.gibba002.stdout: "osds": [ 2021-02-10T11:04:56.730 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:04:56.730 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 0, 2021-02-10T11:04:56.730 INFO:teuthology.orchestra.run.gibba002.stdout: "uuid": "be691b4d-a039-408a-af80-9d53f02b6f1f", 2021-02-10T11:04:56.731 INFO:teuthology.orchestra.run.gibba002.stdout: "up": 1, 2021-02-10T11:04:56.731 INFO:teuthology.orchestra.run.gibba002.stdout: "in": 1, 2021-02-10T11:04:56.731 INFO:teuthology.orchestra.run.gibba002.stdout: "weight": 1, 2021-02-10T11:04:56.731 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_affinity": 1, 2021-02-10T11:04:56.731 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_begin": 0, 2021-02-10T11:04:56.732 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_end": 0, 2021-02-10T11:04:56.732 INFO:teuthology.orchestra.run.gibba002.stdout: "up_from": 13, 2021-02-10T11:04:56.732 INFO:teuthology.orchestra.run.gibba002.stdout: "up_thru": 18, 2021-02-10T11:04:56.732 INFO:teuthology.orchestra.run.gibba002.stdout: "down_at": 0, 2021-02-10T11:04:56.732 INFO:teuthology.orchestra.run.gibba002.stdout: "lost_at": 0, 2021-02-10T11:04:56.732 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addrs": { 2021-02-10T11:04:56.732 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:04:56.732 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:04:56.732 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:04:56.733 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6818", 2021-02-10T11:04:56.733 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36088 2021-02-10T11:04:56.733 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:56.733 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:04:56.733 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:04:56.733 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6820", 2021-02-10T11:04:56.734 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36088 2021-02-10T11:04:56.734 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:04:56.734 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:04:56.734 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:56.734 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addrs": { 2021-02-10T11:04:56.735 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:04:56.735 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:04:56.735 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:04:56.735 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6822", 2021-02-10T11:04:56.735 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36088 2021-02-10T11:04:56.736 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:56.736 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:04:56.736 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:04:56.736 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6824", 2021-02-10T11:04:56.736 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36088 2021-02-10T11:04:56.737 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:04:56.737 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:04:56.737 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:56.737 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addrs": { 2021-02-10T11:04:56.737 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:04:56.737 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:04:56.738 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:04:56.738 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6830", 2021-02-10T11:04:56.738 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36088 2021-02-10T11:04:56.738 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:56.738 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:04:56.739 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:04:56.739 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6832", 2021-02-10T11:04:56.739 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36088 2021-02-10T11:04:56.739 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:04:56.739 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:04:56.739 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:56.739 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addrs": { 2021-02-10T11:04:56.740 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:04:56.740 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:04:56.740 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:04:56.740 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6826", 2021-02-10T11:04:56.740 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36088 2021-02-10T11:04:56.741 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:56.741 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:04:56.741 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:04:56.741 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6828", 2021-02-10T11:04:56.741 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36088 2021-02-10T11:04:56.742 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:04:56.742 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:04:56.742 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:56.742 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addr": "172.21.2.102:6820/36088", 2021-02-10T11:04:56.742 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addr": "172.21.2.102:6824/36088", 2021-02-10T11:04:56.743 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addr": "172.21.2.102:6832/36088", 2021-02-10T11:04:56.743 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addr": "172.21.2.102:6828/36088", 2021-02-10T11:04:56.743 INFO:teuthology.orchestra.run.gibba002.stdout: "state": [ 2021-02-10T11:04:56.743 INFO:teuthology.orchestra.run.gibba002.stdout: "exists", 2021-02-10T11:04:56.743 INFO:teuthology.orchestra.run.gibba002.stdout: "up" 2021-02-10T11:04:56.744 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:04:56.744 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:56.744 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:04:56.744 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 1, 2021-02-10T11:04:56.744 INFO:teuthology.orchestra.run.gibba002.stdout: "uuid": "64e3ccc8-9c69-4d7c-865e-af8e83822635", 2021-02-10T11:04:56.745 INFO:teuthology.orchestra.run.gibba002.stdout: "up": 1, 2021-02-10T11:04:56.745 INFO:teuthology.orchestra.run.gibba002.stdout: "in": 1, 2021-02-10T11:04:56.745 INFO:teuthology.orchestra.run.gibba002.stdout: "weight": 1, 2021-02-10T11:04:56.745 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_affinity": 1, 2021-02-10T11:04:56.745 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_begin": 0, 2021-02-10T11:04:56.746 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_end": 0, 2021-02-10T11:04:56.746 INFO:teuthology.orchestra.run.gibba002.stdout: "up_from": 13, 2021-02-10T11:04:56.746 INFO:teuthology.orchestra.run.gibba002.stdout: "up_thru": 18, 2021-02-10T11:04:56.746 INFO:teuthology.orchestra.run.gibba002.stdout: "down_at": 0, 2021-02-10T11:04:56.746 INFO:teuthology.orchestra.run.gibba002.stdout: "lost_at": 0, 2021-02-10T11:04:56.746 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addrs": { 2021-02-10T11:04:56.747 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:04:56.747 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:04:56.747 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:04:56.747 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6819", 2021-02-10T11:04:56.747 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36087 2021-02-10T11:04:56.747 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:56.747 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:04:56.747 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:04:56.747 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6821", 2021-02-10T11:04:56.747 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36087 2021-02-10T11:04:56.747 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:04:56.748 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:04:56.748 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:56.748 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addrs": { 2021-02-10T11:04:56.748 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:04:56.748 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:04:56.748 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:04:56.748 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6823", 2021-02-10T11:04:56.748 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36087 2021-02-10T11:04:56.748 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:56.748 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:04:56.748 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:04:56.748 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6825", 2021-02-10T11:04:56.748 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36087 2021-02-10T11:04:56.748 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:04:56.749 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:04:56.749 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:56.749 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addrs": { 2021-02-10T11:04:56.749 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:04:56.749 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:04:56.749 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:04:56.749 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6831", 2021-02-10T11:04:56.749 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36087 2021-02-10T11:04:56.749 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:56.749 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:04:56.749 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:04:56.749 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6833", 2021-02-10T11:04:56.749 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36087 2021-02-10T11:04:56.750 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:04:56.750 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:04:56.750 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:56.750 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addrs": { 2021-02-10T11:04:56.750 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:04:56.750 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:04:56.750 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:04:56.750 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6827", 2021-02-10T11:04:56.750 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36087 2021-02-10T11:04:56.750 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:56.750 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:04:56.750 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:04:56.750 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6829", 2021-02-10T11:04:56.751 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36087 2021-02-10T11:04:56.751 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:04:56.751 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:04:56.751 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:56.751 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addr": "172.21.2.102:6821/36087", 2021-02-10T11:04:56.751 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addr": "172.21.2.102:6825/36087", 2021-02-10T11:04:56.751 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addr": "172.21.2.102:6833/36087", 2021-02-10T11:04:56.751 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addr": "172.21.2.102:6829/36087", 2021-02-10T11:04:56.751 INFO:teuthology.orchestra.run.gibba002.stdout: "state": [ 2021-02-10T11:04:56.751 INFO:teuthology.orchestra.run.gibba002.stdout: "exists", 2021-02-10T11:04:56.751 INFO:teuthology.orchestra.run.gibba002.stdout: "up" 2021-02-10T11:04:56.751 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:04:56.751 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:56.752 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:04:56.752 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 2, 2021-02-10T11:04:56.752 INFO:teuthology.orchestra.run.gibba002.stdout: "uuid": "153700af-747f-481f-bdf4-d6072d3a842f", 2021-02-10T11:04:56.752 INFO:teuthology.orchestra.run.gibba002.stdout: "up": 1, 2021-02-10T11:04:56.752 INFO:teuthology.orchestra.run.gibba002.stdout: "in": 1, 2021-02-10T11:04:56.752 INFO:teuthology.orchestra.run.gibba002.stdout: "weight": 1, 2021-02-10T11:04:56.752 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_affinity": 1, 2021-02-10T11:04:56.752 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_begin": 0, 2021-02-10T11:04:56.752 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_end": 0, 2021-02-10T11:04:56.752 INFO:teuthology.orchestra.run.gibba002.stdout: "up_from": 13, 2021-02-10T11:04:56.752 INFO:teuthology.orchestra.run.gibba002.stdout: "up_thru": 17, 2021-02-10T11:04:56.752 INFO:teuthology.orchestra.run.gibba002.stdout: "down_at": 0, 2021-02-10T11:04:56.752 INFO:teuthology.orchestra.run.gibba002.stdout: "lost_at": 0, 2021-02-10T11:04:56.753 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addrs": { 2021-02-10T11:04:56.753 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:04:56.753 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:04:56.753 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:04:56.753 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6810", 2021-02-10T11:04:56.753 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36089 2021-02-10T11:04:56.753 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:56.753 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:04:56.753 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:04:56.753 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6811", 2021-02-10T11:04:56.753 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36089 2021-02-10T11:04:56.753 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:04:56.753 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:04:56.754 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:56.754 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addrs": { 2021-02-10T11:04:56.754 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:04:56.754 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:04:56.754 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:04:56.754 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6812", 2021-02-10T11:04:56.754 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36089 2021-02-10T11:04:56.754 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:56.754 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:04:56.754 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:04:56.754 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6813", 2021-02-10T11:04:56.754 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36089 2021-02-10T11:04:56.754 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:04:56.755 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:04:56.755 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:56.755 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addrs": { 2021-02-10T11:04:56.755 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:04:56.755 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:04:56.755 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:04:56.755 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6816", 2021-02-10T11:04:56.755 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36089 2021-02-10T11:04:56.755 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:56.755 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:04:56.755 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:04:56.755 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6817", 2021-02-10T11:04:56.755 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36089 2021-02-10T11:04:56.756 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:04:56.756 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:04:56.756 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:56.756 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addrs": { 2021-02-10T11:04:56.756 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:04:56.756 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:04:56.756 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:04:56.756 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6814", 2021-02-10T11:04:56.756 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36089 2021-02-10T11:04:56.756 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:56.756 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:04:56.756 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:04:56.756 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6815", 2021-02-10T11:04:56.757 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36089 2021-02-10T11:04:56.757 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:04:56.757 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:04:56.757 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:56.757 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addr": "172.21.2.102:6811/36089", 2021-02-10T11:04:56.757 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addr": "172.21.2.102:6813/36089", 2021-02-10T11:04:56.757 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addr": "172.21.2.102:6817/36089", 2021-02-10T11:04:56.757 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addr": "172.21.2.102:6815/36089", 2021-02-10T11:04:56.757 INFO:teuthology.orchestra.run.gibba002.stdout: "state": [ 2021-02-10T11:04:56.757 INFO:teuthology.orchestra.run.gibba002.stdout: "exists", 2021-02-10T11:04:56.757 INFO:teuthology.orchestra.run.gibba002.stdout: "up" 2021-02-10T11:04:56.757 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:04:56.757 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:56.758 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:04:56.758 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 3, 2021-02-10T11:04:56.758 INFO:teuthology.orchestra.run.gibba002.stdout: "uuid": "cbc678f4-83a9-45e6-8830-2f7c232b1ea7", 2021-02-10T11:04:56.758 INFO:teuthology.orchestra.run.gibba002.stdout: "up": 1, 2021-02-10T11:04:56.758 INFO:teuthology.orchestra.run.gibba002.stdout: "in": 1, 2021-02-10T11:04:56.758 INFO:teuthology.orchestra.run.gibba002.stdout: "weight": 1, 2021-02-10T11:04:56.758 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_affinity": 1, 2021-02-10T11:04:56.758 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_begin": 0, 2021-02-10T11:04:56.758 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_end": 0, 2021-02-10T11:04:56.758 INFO:teuthology.orchestra.run.gibba002.stdout: "up_from": 13, 2021-02-10T11:04:56.758 INFO:teuthology.orchestra.run.gibba002.stdout: "up_thru": 18, 2021-02-10T11:04:56.758 INFO:teuthology.orchestra.run.gibba002.stdout: "down_at": 0, 2021-02-10T11:04:56.758 INFO:teuthology.orchestra.run.gibba002.stdout: "lost_at": 0, 2021-02-10T11:04:56.759 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addrs": { 2021-02-10T11:04:56.759 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:04:56.759 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:04:56.759 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:04:56.759 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6802", 2021-02-10T11:04:56.759 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36090 2021-02-10T11:04:56.759 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:56.759 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:04:56.759 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:04:56.759 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6803", 2021-02-10T11:04:56.759 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36090 2021-02-10T11:04:56.759 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:04:56.759 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:04:56.760 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:56.760 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addrs": { 2021-02-10T11:04:56.760 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:04:56.760 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:04:56.760 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:04:56.760 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6804", 2021-02-10T11:04:56.760 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36090 2021-02-10T11:04:56.760 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:56.760 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:04:56.760 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:04:56.760 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6805", 2021-02-10T11:04:56.760 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36090 2021-02-10T11:04:56.760 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:04:56.761 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:04:56.761 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:56.761 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addrs": { 2021-02-10T11:04:56.761 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:04:56.761 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:04:56.761 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:04:56.761 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6808", 2021-02-10T11:04:56.761 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36090 2021-02-10T11:04:56.761 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:56.761 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:04:56.761 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:04:56.761 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6809", 2021-02-10T11:04:56.761 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36090 2021-02-10T11:04:56.762 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:04:56.762 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:04:56.762 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:56.762 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addrs": { 2021-02-10T11:04:56.762 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:04:56.762 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:04:56.762 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:04:56.762 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6806", 2021-02-10T11:04:56.762 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36090 2021-02-10T11:04:56.762 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:56.762 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:04:56.762 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:04:56.762 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6807", 2021-02-10T11:04:56.763 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36090 2021-02-10T11:04:56.763 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:04:56.763 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:04:56.763 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:56.763 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addr": "172.21.2.102:6803/36090", 2021-02-10T11:04:56.763 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addr": "172.21.2.102:6805/36090", 2021-02-10T11:04:56.763 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addr": "172.21.2.102:6809/36090", 2021-02-10T11:04:56.763 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addr": "172.21.2.102:6807/36090", 2021-02-10T11:04:56.763 INFO:teuthology.orchestra.run.gibba002.stdout: "state": [ 2021-02-10T11:04:56.763 INFO:teuthology.orchestra.run.gibba002.stdout: "exists", 2021-02-10T11:04:56.763 INFO:teuthology.orchestra.run.gibba002.stdout: "up" 2021-02-10T11:04:56.763 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:04:56.763 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:56.764 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:04:56.764 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 4, 2021-02-10T11:04:56.764 INFO:teuthology.orchestra.run.gibba002.stdout: "uuid": "c01e4008-b258-4f10-afd2-9e7f37214351", 2021-02-10T11:04:56.764 INFO:teuthology.orchestra.run.gibba002.stdout: "up": 1, 2021-02-10T11:04:56.764 INFO:teuthology.orchestra.run.gibba002.stdout: "in": 1, 2021-02-10T11:04:56.764 INFO:teuthology.orchestra.run.gibba002.stdout: "weight": 1, 2021-02-10T11:04:56.764 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_affinity": 1, 2021-02-10T11:04:56.764 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_begin": 0, 2021-02-10T11:04:56.764 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_end": 0, 2021-02-10T11:04:56.764 INFO:teuthology.orchestra.run.gibba002.stdout: "up_from": 13, 2021-02-10T11:04:56.764 INFO:teuthology.orchestra.run.gibba002.stdout: "up_thru": 18, 2021-02-10T11:04:56.764 INFO:teuthology.orchestra.run.gibba002.stdout: "down_at": 0, 2021-02-10T11:04:56.764 INFO:teuthology.orchestra.run.gibba002.stdout: "lost_at": 0, 2021-02-10T11:04:56.764 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addrs": { 2021-02-10T11:04:56.765 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:04:56.765 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:04:56.765 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:04:56.765 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6808", 2021-02-10T11:04:56.765 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36368 2021-02-10T11:04:56.765 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:56.765 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:04:56.765 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:04:56.765 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6811", 2021-02-10T11:04:56.765 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36368 2021-02-10T11:04:56.765 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:04:56.765 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:04:56.765 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:56.766 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addrs": { 2021-02-10T11:04:56.766 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:04:56.766 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:04:56.766 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:04:56.766 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6814", 2021-02-10T11:04:56.766 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36368 2021-02-10T11:04:56.766 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:56.766 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:04:56.766 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:04:56.766 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6818", 2021-02-10T11:04:56.766 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36368 2021-02-10T11:04:56.766 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:04:56.766 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:04:56.767 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:56.767 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addrs": { 2021-02-10T11:04:56.767 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:04:56.767 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:04:56.767 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:04:56.767 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6826", 2021-02-10T11:04:56.767 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36368 2021-02-10T11:04:56.767 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:56.767 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:04:56.767 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:04:56.767 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6828", 2021-02-10T11:04:56.767 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36368 2021-02-10T11:04:56.767 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:04:56.767 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:04:56.768 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:56.768 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addrs": { 2021-02-10T11:04:56.768 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:04:56.768 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:04:56.768 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:04:56.768 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6820", 2021-02-10T11:04:56.768 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36368 2021-02-10T11:04:56.768 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:56.768 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:04:56.768 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:04:56.768 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6824", 2021-02-10T11:04:56.768 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36368 2021-02-10T11:04:56.768 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:04:56.769 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:04:56.769 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:56.769 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addr": "172.21.2.113:6811/36368", 2021-02-10T11:04:56.769 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addr": "172.21.2.113:6818/36368", 2021-02-10T11:04:56.769 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addr": "172.21.2.113:6828/36368", 2021-02-10T11:04:56.769 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addr": "172.21.2.113:6824/36368", 2021-02-10T11:04:56.769 INFO:teuthology.orchestra.run.gibba002.stdout: "state": [ 2021-02-10T11:04:56.769 INFO:teuthology.orchestra.run.gibba002.stdout: "exists", 2021-02-10T11:04:56.769 INFO:teuthology.orchestra.run.gibba002.stdout: "up" 2021-02-10T11:04:56.769 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:04:56.769 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:56.769 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:04:56.769 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 5, 2021-02-10T11:04:56.769 INFO:teuthology.orchestra.run.gibba002.stdout: "uuid": "551f6007-daf8-4a98-8ee0-c2b16bcccb5e", 2021-02-10T11:04:56.770 INFO:teuthology.orchestra.run.gibba002.stdout: "up": 1, 2021-02-10T11:04:56.770 INFO:teuthology.orchestra.run.gibba002.stdout: "in": 1, 2021-02-10T11:04:56.770 INFO:teuthology.orchestra.run.gibba002.stdout: "weight": 1, 2021-02-10T11:04:56.770 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_affinity": 1, 2021-02-10T11:04:56.770 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_begin": 0, 2021-02-10T11:04:56.770 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_end": 0, 2021-02-10T11:04:56.770 INFO:teuthology.orchestra.run.gibba002.stdout: "up_from": 13, 2021-02-10T11:04:56.770 INFO:teuthology.orchestra.run.gibba002.stdout: "up_thru": 18, 2021-02-10T11:04:56.770 INFO:teuthology.orchestra.run.gibba002.stdout: "down_at": 0, 2021-02-10T11:04:56.770 INFO:teuthology.orchestra.run.gibba002.stdout: "lost_at": 0, 2021-02-10T11:04:56.770 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addrs": { 2021-02-10T11:04:56.770 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:04:56.770 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:04:56.771 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:04:56.771 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6802", 2021-02-10T11:04:56.771 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36366 2021-02-10T11:04:56.771 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:56.771 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:04:56.771 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:04:56.771 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6805", 2021-02-10T11:04:56.771 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36366 2021-02-10T11:04:56.771 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:04:56.771 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:04:56.771 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:56.771 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addrs": { 2021-02-10T11:04:56.771 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:04:56.771 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:04:56.772 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:04:56.772 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6807", 2021-02-10T11:04:56.772 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36366 2021-02-10T11:04:56.772 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:56.772 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:04:56.772 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:04:56.772 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6810", 2021-02-10T11:04:56.772 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36366 2021-02-10T11:04:56.772 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:04:56.772 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:04:56.772 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:56.772 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addrs": { 2021-02-10T11:04:56.772 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:04:56.773 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:04:56.773 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:04:56.773 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6819", 2021-02-10T11:04:56.773 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36366 2021-02-10T11:04:56.773 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:56.773 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:04:56.773 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:04:56.773 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6822", 2021-02-10T11:04:56.773 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36366 2021-02-10T11:04:56.773 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:04:56.773 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:04:56.773 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:56.773 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addrs": { 2021-02-10T11:04:56.773 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:04:56.774 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:04:56.774 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:04:56.774 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6812", 2021-02-10T11:04:56.774 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36366 2021-02-10T11:04:56.774 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:56.774 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:04:56.774 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:04:56.774 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6815", 2021-02-10T11:04:56.774 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36366 2021-02-10T11:04:56.774 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:04:56.774 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:04:56.774 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:56.774 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addr": "172.21.2.113:6805/36366", 2021-02-10T11:04:56.775 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addr": "172.21.2.113:6810/36366", 2021-02-10T11:04:56.775 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addr": "172.21.2.113:6822/36366", 2021-02-10T11:04:56.775 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addr": "172.21.2.113:6815/36366", 2021-02-10T11:04:56.775 INFO:teuthology.orchestra.run.gibba002.stdout: "state": [ 2021-02-10T11:04:56.775 INFO:teuthology.orchestra.run.gibba002.stdout: "exists", 2021-02-10T11:04:56.775 INFO:teuthology.orchestra.run.gibba002.stdout: "up" 2021-02-10T11:04:56.775 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:04:56.775 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:56.775 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:04:56.775 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 6, 2021-02-10T11:04:56.775 INFO:teuthology.orchestra.run.gibba002.stdout: "uuid": "bf67c040-0e50-48be-b102-c3943bc9ebef", 2021-02-10T11:04:56.775 INFO:teuthology.orchestra.run.gibba002.stdout: "up": 1, 2021-02-10T11:04:56.775 INFO:teuthology.orchestra.run.gibba002.stdout: "in": 1, 2021-02-10T11:04:56.776 INFO:teuthology.orchestra.run.gibba002.stdout: "weight": 1, 2021-02-10T11:04:56.776 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_affinity": 1, 2021-02-10T11:04:56.776 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_begin": 0, 2021-02-10T11:04:56.776 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_end": 0, 2021-02-10T11:04:56.776 INFO:teuthology.orchestra.run.gibba002.stdout: "up_from": 13, 2021-02-10T11:04:56.776 INFO:teuthology.orchestra.run.gibba002.stdout: "up_thru": 17, 2021-02-10T11:04:56.776 INFO:teuthology.orchestra.run.gibba002.stdout: "down_at": 0, 2021-02-10T11:04:56.776 INFO:teuthology.orchestra.run.gibba002.stdout: "lost_at": 0, 2021-02-10T11:04:56.776 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addrs": { 2021-02-10T11:04:56.776 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:04:56.776 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:04:56.776 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:04:56.776 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6800", 2021-02-10T11:04:56.776 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36367 2021-02-10T11:04:56.777 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:56.777 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:04:56.777 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:04:56.777 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6801", 2021-02-10T11:04:56.777 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36367 2021-02-10T11:04:56.777 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:04:56.777 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:04:56.777 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:56.777 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addrs": { 2021-02-10T11:04:56.777 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:04:56.777 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:04:56.777 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:04:56.777 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6803", 2021-02-10T11:04:56.778 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36367 2021-02-10T11:04:56.778 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:56.778 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:04:56.778 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:04:56.778 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6804", 2021-02-10T11:04:56.778 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36367 2021-02-10T11:04:56.778 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:04:56.778 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:04:56.778 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:56.778 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addrs": { 2021-02-10T11:04:56.778 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:04:56.778 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:04:56.778 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:04:56.779 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6813", 2021-02-10T11:04:56.779 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36367 2021-02-10T11:04:56.779 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:56.779 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:04:56.779 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:04:56.779 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6816", 2021-02-10T11:04:56.779 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36367 2021-02-10T11:04:56.779 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:04:56.779 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:04:56.779 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:56.779 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addrs": { 2021-02-10T11:04:56.779 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:04:56.779 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:04:56.780 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:04:56.780 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6806", 2021-02-10T11:04:56.780 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36367 2021-02-10T11:04:56.780 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:56.780 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:04:56.780 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:04:56.780 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6809", 2021-02-10T11:04:56.780 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36367 2021-02-10T11:04:56.780 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:04:56.780 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:04:56.780 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:56.780 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addr": "172.21.2.113:6801/36367", 2021-02-10T11:04:56.781 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addr": "172.21.2.113:6804/36367", 2021-02-10T11:04:56.781 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addr": "172.21.2.113:6816/36367", 2021-02-10T11:04:56.781 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addr": "172.21.2.113:6809/36367", 2021-02-10T11:04:56.781 INFO:teuthology.orchestra.run.gibba002.stdout: "state": [ 2021-02-10T11:04:56.781 INFO:teuthology.orchestra.run.gibba002.stdout: "exists", 2021-02-10T11:04:56.781 INFO:teuthology.orchestra.run.gibba002.stdout: "up" 2021-02-10T11:04:56.781 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:04:56.781 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:56.781 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:04:56.781 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 7, 2021-02-10T11:04:56.781 INFO:teuthology.orchestra.run.gibba002.stdout: "uuid": "f5461f80-018d-428f-be89-2eb2bcbe7bb6", 2021-02-10T11:04:56.781 INFO:teuthology.orchestra.run.gibba002.stdout: "up": 1, 2021-02-10T11:04:56.781 INFO:teuthology.orchestra.run.gibba002.stdout: "in": 1, 2021-02-10T11:04:56.782 INFO:teuthology.orchestra.run.gibba002.stdout: "weight": 1, 2021-02-10T11:04:56.782 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_affinity": 1, 2021-02-10T11:04:56.782 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_begin": 0, 2021-02-10T11:04:56.782 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_end": 0, 2021-02-10T11:04:56.782 INFO:teuthology.orchestra.run.gibba002.stdout: "up_from": 13, 2021-02-10T11:04:56.782 INFO:teuthology.orchestra.run.gibba002.stdout: "up_thru": 17, 2021-02-10T11:04:56.782 INFO:teuthology.orchestra.run.gibba002.stdout: "down_at": 0, 2021-02-10T11:04:56.782 INFO:teuthology.orchestra.run.gibba002.stdout: "lost_at": 0, 2021-02-10T11:04:56.782 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addrs": { 2021-02-10T11:04:56.782 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:04:56.782 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:04:56.782 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:04:56.782 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6817", 2021-02-10T11:04:56.783 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36365 2021-02-10T11:04:56.783 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:56.783 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:04:56.783 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:04:56.783 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6821", 2021-02-10T11:04:56.783 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36365 2021-02-10T11:04:56.783 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:04:56.783 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:04:56.783 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:56.783 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addrs": { 2021-02-10T11:04:56.783 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:04:56.783 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:04:56.783 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:04:56.784 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6823", 2021-02-10T11:04:56.784 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36365 2021-02-10T11:04:56.784 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:56.784 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:04:56.784 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:04:56.784 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6825", 2021-02-10T11:04:56.784 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36365 2021-02-10T11:04:56.784 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:04:56.784 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:04:56.784 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:56.784 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addrs": { 2021-02-10T11:04:56.784 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:04:56.784 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:04:56.785 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:04:56.785 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6830", 2021-02-10T11:04:56.785 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36365 2021-02-10T11:04:56.785 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:56.785 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:04:56.785 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:04:56.785 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6831", 2021-02-10T11:04:56.785 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36365 2021-02-10T11:04:56.785 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:04:56.785 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:04:56.785 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:56.785 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addrs": { 2021-02-10T11:04:56.785 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:04:56.785 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:04:56.786 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:04:56.786 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6827", 2021-02-10T11:04:56.786 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36365 2021-02-10T11:04:56.786 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:56.786 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:04:56.786 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:04:56.786 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6829", 2021-02-10T11:04:56.786 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36365 2021-02-10T11:04:56.786 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:04:56.786 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:04:56.786 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:56.786 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addr": "172.21.2.113:6821/36365", 2021-02-10T11:04:56.786 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addr": "172.21.2.113:6825/36365", 2021-02-10T11:04:56.787 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addr": "172.21.2.113:6831/36365", 2021-02-10T11:04:56.787 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addr": "172.21.2.113:6829/36365", 2021-02-10T11:04:56.787 INFO:teuthology.orchestra.run.gibba002.stdout: "state": [ 2021-02-10T11:04:56.787 INFO:teuthology.orchestra.run.gibba002.stdout: "exists", 2021-02-10T11:04:56.787 INFO:teuthology.orchestra.run.gibba002.stdout: "up" 2021-02-10T11:04:56.787 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:04:56.787 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:04:56.787 INFO:teuthology.orchestra.run.gibba002.stdout: ], 2021-02-10T11:04:56.787 INFO:teuthology.orchestra.run.gibba002.stdout: "osd_xinfo": [ 2021-02-10T11:04:56.787 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:04:56.787 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 0, 2021-02-10T11:04:56.787 INFO:teuthology.orchestra.run.gibba002.stdout: "down_stamp": "0.000000", 2021-02-10T11:04:56.787 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_probability": 0, 2021-02-10T11:04:56.787 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_interval": 0, 2021-02-10T11:04:56.788 INFO:teuthology.orchestra.run.gibba002.stdout: "features": 4611087854035861503, 2021-02-10T11:04:56.788 INFO:teuthology.orchestra.run.gibba002.stdout: "old_weight": 0 2021-02-10T11:04:56.788 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:56.788 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:04:56.788 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 1, 2021-02-10T11:04:56.788 INFO:teuthology.orchestra.run.gibba002.stdout: "down_stamp": "0.000000", 2021-02-10T11:04:56.788 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_probability": 0, 2021-02-10T11:04:56.788 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_interval": 0, 2021-02-10T11:04:56.788 INFO:teuthology.orchestra.run.gibba002.stdout: "features": 4611087854035861503, 2021-02-10T11:04:56.788 INFO:teuthology.orchestra.run.gibba002.stdout: "old_weight": 0 2021-02-10T11:04:56.788 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:56.788 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:04:56.788 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 2, 2021-02-10T11:04:56.789 INFO:teuthology.orchestra.run.gibba002.stdout: "down_stamp": "0.000000", 2021-02-10T11:04:56.789 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_probability": 0, 2021-02-10T11:04:56.789 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_interval": 0, 2021-02-10T11:04:56.789 INFO:teuthology.orchestra.run.gibba002.stdout: "features": 4611087854035861503, 2021-02-10T11:04:56.789 INFO:teuthology.orchestra.run.gibba002.stdout: "old_weight": 0 2021-02-10T11:04:56.789 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:56.789 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:04:56.789 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 3, 2021-02-10T11:04:56.789 INFO:teuthology.orchestra.run.gibba002.stdout: "down_stamp": "0.000000", 2021-02-10T11:04:56.789 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_probability": 0, 2021-02-10T11:04:56.789 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_interval": 0, 2021-02-10T11:04:56.789 INFO:teuthology.orchestra.run.gibba002.stdout: "features": 4611087854035861503, 2021-02-10T11:04:56.789 INFO:teuthology.orchestra.run.gibba002.stdout: "old_weight": 0 2021-02-10T11:04:56.789 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:56.790 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:04:56.790 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 4, 2021-02-10T11:04:56.790 INFO:teuthology.orchestra.run.gibba002.stdout: "down_stamp": "0.000000", 2021-02-10T11:04:56.790 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_probability": 0, 2021-02-10T11:04:56.790 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_interval": 0, 2021-02-10T11:04:56.790 INFO:teuthology.orchestra.run.gibba002.stdout: "features": 4611087854035861503, 2021-02-10T11:04:56.790 INFO:teuthology.orchestra.run.gibba002.stdout: "old_weight": 0 2021-02-10T11:04:56.790 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:56.790 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:04:56.790 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 5, 2021-02-10T11:04:56.790 INFO:teuthology.orchestra.run.gibba002.stdout: "down_stamp": "0.000000", 2021-02-10T11:04:56.790 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_probability": 0, 2021-02-10T11:04:56.790 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_interval": 0, 2021-02-10T11:04:56.791 INFO:teuthology.orchestra.run.gibba002.stdout: "features": 4611087854035861503, 2021-02-10T11:04:56.791 INFO:teuthology.orchestra.run.gibba002.stdout: "old_weight": 0 2021-02-10T11:04:56.791 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:56.791 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:04:56.791 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 6, 2021-02-10T11:04:56.791 INFO:teuthology.orchestra.run.gibba002.stdout: "down_stamp": "0.000000", 2021-02-10T11:04:56.791 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_probability": 0, 2021-02-10T11:04:56.791 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_interval": 0, 2021-02-10T11:04:56.791 INFO:teuthology.orchestra.run.gibba002.stdout: "features": 4611087854035861503, 2021-02-10T11:04:56.791 INFO:teuthology.orchestra.run.gibba002.stdout: "old_weight": 0 2021-02-10T11:04:56.791 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:56.791 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:04:56.791 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 7, 2021-02-10T11:04:56.791 INFO:teuthology.orchestra.run.gibba002.stdout: "down_stamp": "0.000000", 2021-02-10T11:04:56.792 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_probability": 0, 2021-02-10T11:04:56.792 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_interval": 0, 2021-02-10T11:04:56.792 INFO:teuthology.orchestra.run.gibba002.stdout: "features": 4611087854035861503, 2021-02-10T11:04:56.792 INFO:teuthology.orchestra.run.gibba002.stdout: "old_weight": 0 2021-02-10T11:04:56.792 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:04:56.792 INFO:teuthology.orchestra.run.gibba002.stdout: ], 2021-02-10T11:04:56.792 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_upmap": [], 2021-02-10T11:04:56.792 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_upmap_items": [], 2021-02-10T11:04:56.792 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_temp": [], 2021-02-10T11:04:56.792 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_temp": [], 2021-02-10T11:04:56.792 INFO:teuthology.orchestra.run.gibba002.stdout: "blacklist": {}, 2021-02-10T11:04:56.792 INFO:teuthology.orchestra.run.gibba002.stdout: "erasure_code_profiles": { 2021-02-10T11:04:56.792 INFO:teuthology.orchestra.run.gibba002.stdout: "default": { 2021-02-10T11:04:56.793 INFO:teuthology.orchestra.run.gibba002.stdout: "crush-failure-domain": "osd", 2021-02-10T11:04:56.793 INFO:teuthology.orchestra.run.gibba002.stdout: "k": "2", 2021-02-10T11:04:56.793 INFO:teuthology.orchestra.run.gibba002.stdout: "m": "1", 2021-02-10T11:04:56.793 INFO:teuthology.orchestra.run.gibba002.stdout: "plugin": "jerasure", 2021-02-10T11:04:56.793 INFO:teuthology.orchestra.run.gibba002.stdout: "ruleset-failure-domain": "osd", 2021-02-10T11:04:56.793 INFO:teuthology.orchestra.run.gibba002.stdout: "technique": "reed_sol_van" 2021-02-10T11:04:56.793 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:04:56.793 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:04:56.793 INFO:teuthology.orchestra.run.gibba002.stdout: "removed_snaps_queue": [], 2021-02-10T11:04:56.793 INFO:teuthology.orchestra.run.gibba002.stdout: "new_removed_snaps": [], 2021-02-10T11:04:56.793 INFO:teuthology.orchestra.run.gibba002.stdout: "new_purged_snaps": [], 2021-02-10T11:04:56.793 INFO:teuthology.orchestra.run.gibba002.stdout: "crush_node_flags": {}, 2021-02-10T11:04:56.793 INFO:teuthology.orchestra.run.gibba002.stdout: "device_class_flags": {} 2021-02-10T11:04:56.793 INFO:teuthology.orchestra.run.gibba002.stdout:} 2021-02-10T11:04:56.795 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:56.686 7f9680ae5700 1 -- 172.21.2.102:0/1485864533 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f96600186c0 msgr2=0x7f966001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:56.795 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:56.686 7f9680ae5700 1 --2- 172.21.2.102:0/1485864533 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f96600186c0 0x7f966001abc0 crc :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:56.795 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:56.686 7f9680ae5700 1 -- 172.21.2.102:0/1485864533 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f967c0b1970 msgr2=0x7f967c0b1d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:56.795 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:56.686 7f9680ae5700 1 --2- 172.21.2.102:0/1485864533 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f967c0b1970 0x7f967c0b1d70 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 rx=0x7f966400b5e0 tx=0x7f966400b980).stop 2021-02-10T11:04:56.795 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:56.686 7f9680ae5700 1 -- 172.21.2.102:0/1485864533 shutdown_connections 2021-02-10T11:04:56.795 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:56.686 7f9680ae5700 1 --2- 172.21.2.102:0/1485864533 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f967c08b6d0 0x7f967c1aac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:56.795 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:56.686 7f9680ae5700 1 --2- 172.21.2.102:0/1485864533 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f96600186c0 0x7f966001abc0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:56.795 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:56.686 7f9680ae5700 1 --2- 172.21.2.102:0/1485864533 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f967c08acb0 0x7f967c08b0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:56.795 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:56.686 7f9680ae5700 1 --2- 172.21.2.102:0/1485864533 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f967c0b1970 0x7f967c0b1d70 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:56.795 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:56.686 7f9680ae5700 1 -- 172.21.2.102:0/1485864533 >> 172.21.2.102:0/1485864533 conn(0x7f967c1ac620 msgr2=0x7f967c0893d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:04:56.795 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:56.686 7f9680ae5700 1 -- 172.21.2.102:0/1485864533 shutdown_connections 2021-02-10T11:04:56.796 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:56.686 7f9680ae5700 1 -- 172.21.2.102:0/1485864533 wait complete. 2021-02-10T11:04:56.796 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:04:57.223 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:57.208 7fc7540d3700 1 Processor -- start 2021-02-10T11:04:57.223 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:57.208 7fc7540d3700 1 -- start start 2021-02-10T11:04:57.223 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:57.208 7fc7540d3700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc74c0e5070 0x7fc74c1b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:57.223 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:57.208 7fc7540d3700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fc74c0e5470 0x7fc74c07d690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:57.223 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:57.208 7fc7540d3700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc74c07ed80 con 0x7fc74c1b3e40 2021-02-10T11:04:57.223 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:57.208 7fc7540d3700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc74c080310 con 0x7fc74c0e5470 2021-02-10T11:04:57.223 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:57.208 7fc7540d3700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc74c080500 con 0x7fc74c0e5070 2021-02-10T11:04:57.224 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:57.209 7fc751eef700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc74c0e5070 0x7fc74c1b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:57.224 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:57.209 7fc751eef700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc74c0e5070 0x7fc74c1b3820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.113:3301/0 says I am v2:172.21.2.102:55778/0 (socket says 172.21.2.102:55778) 2021-02-10T11:04:57.224 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:57.209 7fc7526f0700 1 --1- >> v1:172.21.2.113:6789/0 conn(0x7fc74c1b3e40 0x7fc74c07dcb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.113:6789/0 says I am v1:172.21.2.102:49794/0 (socket says 172.21.2.102:49794) 2021-02-10T11:04:57.224 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:57.209 7fc751eef700 1 -- 172.21.2.102:0/1805395031 learned_addr learned my addr 172.21.2.102:0/1805395031 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:04:57.224 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:57.209 7fc7516ee700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fc74c0e5470 0x7fc74c07d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:57.224 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:57.209 7fc7516ee700 1 -- 172.21.2.102:0/1805395031 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc74c0e5070 msgr2=0x7fc74c1b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:04:57.224 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:57.209 7fc7516ee700 1 --2- 172.21.2.102:0/1805395031 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc74c0e5070 0x7fc74c1b3820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:57.224 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:57.209 7fc7516ee700 1 -- 172.21.2.102:0/1805395031 >> v1:172.21.2.113:6789/0 conn(0x7fc74c1b3e40 legacy=0x7fc74c07dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:57.224 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:57.209 7fc7516ee700 1 -- 172.21.2.102:0/1805395031 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc74c0806f0 con 0x7fc74c0e5470 2021-02-10T11:04:57.224 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:57.210 7fc751eef700 1 --2- 172.21.2.102:0/1805395031 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc74c0e5070 0x7fc74c1b3820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:04:57.224 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:57.210 7fc7516ee700 1 --2- 172.21.2.102:0/1805395031 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc74c0e5470 0x7fc74c07d690 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7fc73c006650 tx=0x7fc73c009a80).ready entity=mon.1 client_cookie=8c50aabdd6b645e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:04:57.224 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:57.210 7fc750eed700 1 -- 172.21.2.102:0/1805395031 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc73c0009b0 con 0x7fc74c0e5470 2021-02-10T11:04:57.224 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:57.210 7fc750eed700 1 -- 172.21.2.102:0/1805395031 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc73c00c0c0 con 0x7fc74c0e5470 2021-02-10T11:04:57.225 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:57.210 7fc750eed700 1 -- 172.21.2.102:0/1805395031 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc73c00c5e0 con 0x7fc74c0e5470 2021-02-10T11:04:57.225 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:57.210 7fc7540d3700 1 -- 172.21.2.102:0/1805395031 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc74c0e5470 msgr2=0x7fc74c07d690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:57.225 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:57.210 7fc7540d3700 1 --2- 172.21.2.102:0/1805395031 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc74c0e5470 0x7fc74c07d690 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7fc73c006650 tx=0x7fc73c009a80).stop 2021-02-10T11:04:57.225 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:57.211 7fc7540d3700 1 -- 172.21.2.102:0/1805395031 shutdown_connections 2021-02-10T11:04:57.225 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:57.211 7fc7540d3700 1 --2- 172.21.2.102:0/1805395031 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc74c0e5470 0x7fc74c07d690 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:57.225 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:57.211 7fc7540d3700 1 --2- 172.21.2.102:0/1805395031 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc74c0e5070 0x7fc74c1b3820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:57.225 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:57.211 7fc7540d3700 1 -- 172.21.2.102:0/1805395031 >> 172.21.2.102:0/1805395031 conn(0x7fc74c0ce690 msgr2=0x7fc74c0e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:04:57.225 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:57.211 7fc7540d3700 1 -- 172.21.2.102:0/1805395031 shutdown_connections 2021-02-10T11:04:57.225 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:57.211 7fc7540d3700 1 -- 172.21.2.102:0/1805395031 wait complete. 2021-02-10T11:04:57.225 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:57.225 7fc7540d3700 1 Processor -- start 2021-02-10T11:04:57.225 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:57.225 7fc7540d3700 1 -- start start 2021-02-10T11:04:57.227 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:57.226 7fc7540d3700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc74c1918e0 0x7fc74c189c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:57.227 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:57.226 7fc7540d3700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc74c18a2a0 0x7fc74c18a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:57.227 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:57.226 7fc7540d3700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc74c18acc0 0x7fc74c190070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:57.227 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:57.226 7fc7540d3700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc74c1a2f00 con 0x7fc74c1918e0 2021-02-10T11:04:57.227 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:57.226 7fc7540d3700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc74c191ce0 con 0x7fc74c18a2a0 2021-02-10T11:04:57.227 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:57.226 7fc7540d3700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc74c191f00 con 0x7fc74c18acc0 2021-02-10T11:04:57.227 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:57.226 7fc751eef700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc74c1918e0 0x7fc74c189c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:57.228 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:57.226 7fc751eef700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc74c1918e0 0x7fc74c189c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:50250/0 (socket says 172.21.2.102:50250) 2021-02-10T11:04:57.228 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:57.226 7fc751eef700 1 -- 172.21.2.102:0/3289974542 learned_addr learned my addr 172.21.2.102:0/3289974542 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:04:57.228 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:57.226 7fc751eef700 1 -- 172.21.2.102:0/3289974542 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc74c18acc0 msgr2=0x7fc74c190070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:57.228 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:57.226 7fc751eef700 1 --2- 172.21.2.102:0/3289974542 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc74c18acc0 0x7fc74c190070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:57.228 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:57.226 7fc751eef700 1 -- 172.21.2.102:0/3289974542 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc74c18a2a0 msgr2=0x7fc74c18a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:57.228 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:57.226 7fc751eef700 1 --2- 172.21.2.102:0/3289974542 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc74c18a2a0 0x7fc74c18a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:57.228 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:57.226 7fc751eef700 1 -- 172.21.2.102:0/3289974542 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc74c190690 con 0x7fc74c1918e0 2021-02-10T11:04:57.228 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:57.226 7fc751eef700 1 --2- 172.21.2.102:0/3289974542 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc74c1918e0 0x7fc74c189c80 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0x7fc74400b970 tx=0x7fc74400bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:04:57.228 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:57.227 7fc7427fc700 1 -- 172.21.2.102:0/3289974542 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc744000d30 con 0x7fc74c1918e0 2021-02-10T11:04:57.228 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:57.227 7fc7427fc700 1 -- 172.21.2.102:0/3289974542 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc744006020 con 0x7fc74c1918e0 2021-02-10T11:04:57.228 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:57.227 7fc7427fc700 1 -- 172.21.2.102:0/3289974542 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc7440063f0 con 0x7fc74c1918e0 2021-02-10T11:04:57.228 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:57.227 7fc7540d3700 1 -- 172.21.2.102:0/3289974542 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc74c190690 con 0x7fc74c1918e0 2021-02-10T11:04:57.228 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:57.227 7fc7540d3700 1 -- 172.21.2.102:0/3289974542 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc74c190690 con 0x7fc74c1918e0 2021-02-10T11:04:57.229 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:57.227 7fc7427fc700 1 -- 172.21.2.102:0/3289974542 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fc744006410 con 0x7fc74c1918e0 2021-02-10T11:04:57.229 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:57.227 7fc7427fc700 1 --2- 172.21.2.102:0/3289974542 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc7340186c0 0x7fc73401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:57.229 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:57.227 7fc7427fc700 1 -- 172.21.2.102:0/3289974542 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fc744032940 con 0x7fc74c1918e0 2021-02-10T11:04:57.229 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:57.228 7fc7516ee700 1 --2- 172.21.2.102:0/3289974542 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc7340186c0 0x7fc73401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:57.229 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:57.228 7fc7516ee700 1 --2- 172.21.2.102:0/3289974542 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc7340186c0 0x7fc73401abc0 crc :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:04:57.255 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:57.258 7fc7540d3700 1 -- 172.21.2.102:0/3289974542 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc74c189750 con 0x7fc74c1918e0 2021-02-10T11:04:57.258 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:57.260 7fc7427fc700 1 -- 172.21.2.102:0/3289974542 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fc7440263d0 con 0x7fc74c1918e0 2021-02-10T11:04:57.386 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:57.388 7fc7540d3700 1 -- 172.21.2.102:0/3289974542 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc74c189750 con 0x7fc74c1918e0 2021-02-10T11:04:57.386 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:57.389 7fc7427fc700 1 -- 172.21.2.102:0/3289974542 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+2094 (secure 0 0 0) 0x7fc7440263d0 con 0x7fc74c1918e0 2021-02-10T11:04:57.387 INFO:teuthology.orchestra.run.gibba002.stderr:dumped fsmap epoch 6 2021-02-10T11:04:57.387 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:04:57.388 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":6,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4302,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.102:6835/3297596596","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6834","nonce":3297596596},{"type":"v1","addr":"172.21.2.102:6835","nonce":3297596596}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":4}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:03:43.384709","modified":"2021-02-10 11:03:52.791414","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4226},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4226":{"gid":4226,"name":"b","rank":0,"incarnation":5,"state":"up:active","state_seq":3,"addr":"172.21.2.113:6833/285807165","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6832","nonce":285807165},{"type":"v1","addr":"172.21.2.113:6833","nonce":285807165}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2021-02-10T11:04:57.389 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:57.391 7fc7540d3700 1 -- 172.21.2.102:0/3289974542 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc7340186c0 msgr2=0x7fc73401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:57.389 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:57.391 7fc7540d3700 1 --2- 172.21.2.102:0/3289974542 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc7340186c0 0x7fc73401abc0 crc :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:57.389 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:57.391 7fc7540d3700 1 -- 172.21.2.102:0/3289974542 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc74c1918e0 msgr2=0x7fc74c189c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:57.389 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:57.391 7fc7540d3700 1 --2- 172.21.2.102:0/3289974542 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc74c1918e0 0x7fc74c189c80 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0x7fc74400b970 tx=0x7fc74400bd10).stop 2021-02-10T11:04:57.390 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:57.391 7fc7540d3700 1 -- 172.21.2.102:0/3289974542 shutdown_connections 2021-02-10T11:04:57.390 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:57.391 7fc7540d3700 1 --2- 172.21.2.102:0/3289974542 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc74c18acc0 0x7fc74c190070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:57.390 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:57.391 7fc7540d3700 1 --2- 172.21.2.102:0/3289974542 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc7340186c0 0x7fc73401abc0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:57.390 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:57.391 7fc7540d3700 1 --2- 172.21.2.102:0/3289974542 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc74c18a2a0 0x7fc74c18a6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:57.390 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:57.391 7fc7540d3700 1 --2- 172.21.2.102:0/3289974542 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc74c1918e0 0x7fc74c189c80 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:57.390 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:57.391 7fc7540d3700 1 -- 172.21.2.102:0/3289974542 >> 172.21.2.102:0/3289974542 conn(0x7fc74c1a31d0 msgr2=0x7fc74c08b830 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:04:57.390 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:57.391 7fc7540d3700 1 -- 172.21.2.102:0/3289974542 shutdown_connections 2021-02-10T11:04:57.390 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:57.391 7fc7540d3700 1 -- 172.21.2.102:0/3289974542 wait complete. 2021-02-10T11:04:57.397 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs fail cephfs 2021-02-10T11:04:57.824 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:57.809 7f3db924c700 1 Processor -- start 2021-02-10T11:04:57.824 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:57.809 7f3db924c700 1 -- start start 2021-02-10T11:04:57.824 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:57.810 7f3db924c700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f3db40e5070 0x7f3db41b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:57.824 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:57.810 7f3db924c700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3db407ed80 con 0x7f3db40e5470 2021-02-10T11:04:57.825 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:57.810 7f3db924c700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3db407e670 con 0x7f3db41b3e40 2021-02-10T11:04:57.825 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:57.810 7f3db924c700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3db407e890 con 0x7f3db40e5070 2021-02-10T11:04:57.825 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:57.810 7f3db359e700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f3db40e5470 0x7f3db407de80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:42484/0 (socket says 172.21.2.102:42484) 2021-02-10T11:04:57.825 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:57.810 7f3db359e700 1 -- 172.21.2.102:0/3900831201 learned_addr learned my addr 172.21.2.102:0/3900831201 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:04:57.825 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:57.810 7f3db1d9b700 1 -- 172.21.2.102:0/3900831201 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4245263916 0 0) 0x7f3dac001500 con 0x7f3db40e5470 2021-02-10T11:04:57.825 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:57.810 7f3db1d9b700 1 -- 172.21.2.102:0/3900831201 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3da0002fe0 con 0x7f3db40e5470 2021-02-10T11:04:57.825 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:57.810 7f3db2d9d700 1 --2- 172.21.2.102:0/3900831201 >> v2:172.21.2.113:3300/0 conn(0x7f3db40e5070 0x7f3db41b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:57.825 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:57.810 7f3db1d9b700 1 -- 172.21.2.102:0/3900831201 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2280066294 0 0) 0x7f3dac001460 con 0x7f3db40e5470 2021-02-10T11:04:57.825 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:57.810 7f3db1d9b700 1 -- 172.21.2.102:0/3900831201 >> v2:172.21.2.113:3300/0 conn(0x7f3db40e5070 msgr2=0x7f3db41b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:04:57.825 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:57.811 7f3db1d9b700 1 --2- 172.21.2.102:0/3900831201 >> v2:172.21.2.113:3300/0 conn(0x7f3db40e5070 0x7f3db41b3820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:57.825 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:57.811 7f3db1d9b700 1 -- 172.21.2.102:0/3900831201 >> v1:172.21.2.113:6789/0 conn(0x7f3db41b3e40 legacy=0x7f3db407d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:04:57.825 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:57.811 7f3db1d9b700 1 -- 172.21.2.102:0/3900831201 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3db407eab0 con 0x7f3db40e5470 2021-02-10T11:04:57.826 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:57.811 7f3db1d9b700 1 -- 172.21.2.102:0/3900831201 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7f3dac001980 con 0x7f3db40e5470 2021-02-10T11:04:57.826 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:57.811 7f3db1d9b700 1 -- 172.21.2.102:0/3900831201 >> v1:172.21.2.102:6789/0 conn(0x7f3db40e5470 legacy=0x7f3db407de80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:57.826 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:57.811 7f3db1d9b700 1 --2- 172.21.2.102:0/3900831201 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3da0003a60 0x7f3da0005ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:57.826 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:57.811 7f3db1d9b700 1 --2- 172.21.2.102:0/3900831201 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3da0006790 0x7f3da0008c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:57.826 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:57.811 7f3db1d9b700 1 --2- 172.21.2.102:0/3900831201 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3da0009590 0x7f3da000bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:57.826 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:57.811 7f3db1d9b700 1 -- 172.21.2.102:0/3900831201 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3da000c300 con 0x7f3da0006790 2021-02-10T11:04:57.826 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:57.811 7f3db1d9b700 1 -- 172.21.2.102:0/3900831201 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3da000c520 con 0x7f3da0003a60 2021-02-10T11:04:57.826 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:57.811 7f3db1d9b700 1 -- 172.21.2.102:0/3900831201 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3da000c740 con 0x7f3da0009590 2021-02-10T11:04:57.826 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:57.811 7f3db259c700 1 --2- 172.21.2.102:0/3900831201 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3da0006790 0x7f3da0008c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:57.826 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:57.811 7f3db2d9d700 1 --2- 172.21.2.102:0/3900831201 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3da0003a60 0x7f3da0005ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:57.826 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:57.811 7f3db359e700 1 --2- 172.21.2.102:0/3900831201 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3da0009590 0x7f3da000bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:57.826 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:57.811 7f3db259c700 1 -- 172.21.2.102:0/3900831201 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3da0009590 msgr2=0x7f3da000bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:04:57.827 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:57.811 7f3db259c700 1 --2- 172.21.2.102:0/3900831201 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3da0009590 0x7f3da000bab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:57.827 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:57.811 7f3db259c700 1 -- 172.21.2.102:0/3900831201 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3da0003a60 msgr2=0x7f3da0005ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:04:57.827 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:57.811 7f3db259c700 1 --2- 172.21.2.102:0/3900831201 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3da0003a60 0x7f3da0005ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:57.827 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:57.811 7f3db259c700 1 -- 172.21.2.102:0/3900831201 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3da000c9e0 con 0x7f3da0006790 2021-02-10T11:04:57.827 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:57.811 7f3db359e700 1 -- 172.21.2.102:0/3900831201 reap_dead start 2021-02-10T11:04:57.827 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:57.811 7f3db259c700 1 --2- 172.21.2.102:0/3900831201 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3da0006790 0x7f3da0008c90 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 rx=0x7f3da404aac0 tx=0x7f3da404daf0).ready entity=mon.0 client_cookie=9f80f38d17e9bb5 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:04:57.827 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:57.811 7f3db1d9b700 1 -- 172.21.2.102:0/3900831201 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3da404a6b0 con 0x7f3da0006790 2021-02-10T11:04:57.827 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:57.811 7f3db1d9b700 1 -- 172.21.2.102:0/3900831201 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3da4000e20 con 0x7f3da0006790 2021-02-10T11:04:57.827 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:57.811 7f3db1d9b700 1 -- 172.21.2.102:0/3900831201 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3da404f4d0 con 0x7f3da0006790 2021-02-10T11:04:57.827 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:57.811 7f3db924c700 1 -- 172.21.2.102:0/3900831201 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3da0006790 msgr2=0x7f3da0008c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:57.827 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:57.811 7f3db924c700 1 --2- 172.21.2.102:0/3900831201 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3da0006790 0x7f3da0008c90 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 rx=0x7f3da404aac0 tx=0x7f3da404daf0).stop 2021-02-10T11:04:57.827 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:57.812 7f3db924c700 1 -- 172.21.2.102:0/3900831201 shutdown_connections 2021-02-10T11:04:57.827 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:57.812 7f3db924c700 1 --2- 172.21.2.102:0/3900831201 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3da0006790 0x7f3da0008c90 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:57.828 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:57.812 7f3db924c700 1 -- 172.21.2.102:0/3900831201 >> 172.21.2.102:0/3900831201 conn(0x7f3db40ce690 msgr2=0x7f3db40e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:04:57.828 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:57.812 7f3db924c700 1 -- 172.21.2.102:0/3900831201 shutdown_connections 2021-02-10T11:04:57.828 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:57.812 7f3db924c700 1 -- 172.21.2.102:0/3900831201 wait complete. 2021-02-10T11:04:57.828 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:57.827 7f3db924c700 1 Processor -- start 2021-02-10T11:04:57.828 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:57.827 7f3db924c700 1 -- start start 2021-02-10T11:04:57.829 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:57.827 7f3db924c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3db408f500 0x7f3db419f900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:57.829 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:57.827 7f3db924c700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3db408fd90 0x7f3db4090190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:57.830 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:57.827 7f3db924c700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3db40907b0 0x7f3db4094370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:57.830 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:57.827 7f3db924c700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3db408f900 con 0x7f3db408f500 2021-02-10T11:04:57.830 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:57.827 7f3db924c700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3db419ff20 con 0x7f3db408fd90 2021-02-10T11:04:57.830 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:57.827 7f3db924c700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3db41a0140 con 0x7f3db40907b0 2021-02-10T11:04:57.830 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:57.827 7f3db2d9d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3db408f500 0x7f3db419f900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:57.830 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:57.827 7f3db2d9d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3db408f500 0x7f3db419f900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:50268/0 (socket says 172.21.2.102:50268) 2021-02-10T11:04:57.830 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:57.827 7f3db2d9d700 1 -- 172.21.2.102:0/366803446 learned_addr learned my addr 172.21.2.102:0/366803446 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:04:57.830 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:57.827 7f3db2d9d700 1 -- 172.21.2.102:0/366803446 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3db40907b0 msgr2=0x7f3db4094370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:57.830 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:57.827 7f3db2d9d700 1 --2- 172.21.2.102:0/366803446 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3db40907b0 0x7f3db4094370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:57.830 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:57.827 7f3db2d9d700 1 -- 172.21.2.102:0/366803446 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3db408fd90 msgr2=0x7f3db4090190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:57.830 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:57.827 7f3db2d9d700 1 --2- 172.21.2.102:0/366803446 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3db408fd90 0x7f3db4090190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:57.830 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:57.827 7f3db2d9d700 1 -- 172.21.2.102:0/366803446 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3db4094990 con 0x7f3db408f500 2021-02-10T11:04:57.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:57.828 7f3db2d9d700 1 --2- 172.21.2.102:0/366803446 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3db408f500 0x7f3db419f900 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 rx=0x7f3d9c006950 tx=0x7f3d9c00b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:04:57.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:57.828 7f3dab7fe700 1 -- 172.21.2.102:0/366803446 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3d9c0065e0 con 0x7f3db408f500 2021-02-10T11:04:57.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:57.828 7f3dab7fe700 1 -- 172.21.2.102:0/366803446 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d9c00bd50 con 0x7f3db408f500 2021-02-10T11:04:57.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:57.828 7f3db924c700 1 -- 172.21.2.102:0/366803446 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3db4094990 con 0x7f3db408f500 2021-02-10T11:04:57.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:57.828 7f3db924c700 1 -- 172.21.2.102:0/366803446 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3db4094f20 con 0x7f3db408f500 2021-02-10T11:04:57.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:57.828 7f3dab7fe700 1 -- 172.21.2.102:0/366803446 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3d9c0022b0 con 0x7f3db408f500 2021-02-10T11:04:57.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:57.829 7f3dab7fe700 1 -- 172.21.2.102:0/366803446 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f3d9c000c90 con 0x7f3db408f500 2021-02-10T11:04:57.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:57.829 7f3dab7fe700 1 --2- 172.21.2.102:0/366803446 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3d98018730 0x7f3d9801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:57.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:57.829 7f3dab7fe700 1 -- 172.21.2.102:0/366803446 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f3d9c0327e0 con 0x7f3db408f500 2021-02-10T11:04:57.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:57.829 7f3db259c700 1 --2- 172.21.2.102:0/366803446 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3d98018730 0x7f3d9801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:57.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:57.829 7f3db259c700 1 --2- 172.21.2.102:0/366803446 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3d98018730 0x7f3d9801ac10 crc :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:04:57.857 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:57.860 7f3db924c700 1 -- 172.21.2.102:0/366803446 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3db4093e40 con 0x7f3db408f500 2021-02-10T11:04:57.860 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:57.862 7f3dab7fe700 1 -- 172.21.2.102:0/366803446 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f3d9c000940 con 0x7f3db408f500 2021-02-10T11:04:57.990 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:57.992 7f3db924c700 1 -- 172.21.2.102:0/366803446 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f3db4093e40 con 0x7f3db408f500 2021-02-10T11:04:58.747 INFO:tasks.ceph.mon.a.gibba002.stderr:2021-02-10 11:04:58.750 7f8dd2755700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-02-10T11:04:58.753 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:58.755 7f3dab7fe700 1 -- 172.21.2.102:0/366803446 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v7) v1 ==== 161+0+0 (secure 0 0 0) 0x7f3d9c000c90 con 0x7f3db408f500 2021-02-10T11:04:58.755 INFO:teuthology.orchestra.run.gibba002.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2021-02-10T11:04:58.757 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:58.759 7f3db924c700 1 -- 172.21.2.102:0/366803446 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3d98018730 msgr2=0x7f3d9801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:58.757 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:58.759 7f3db924c700 1 --2- 172.21.2.102:0/366803446 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3d98018730 0x7f3d9801ac10 crc :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:58.757 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:58.759 7f3db924c700 1 -- 172.21.2.102:0/366803446 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3db408f500 msgr2=0x7f3db419f900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:58.757 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:58.759 7f3db924c700 1 --2- 172.21.2.102:0/366803446 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3db408f500 0x7f3db419f900 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 rx=0x7f3d9c006950 tx=0x7f3d9c00b960).stop 2021-02-10T11:04:58.758 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:58.760 7f3db924c700 1 -- 172.21.2.102:0/366803446 shutdown_connections 2021-02-10T11:04:58.758 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:58.760 7f3db924c700 1 --2- 172.21.2.102:0/366803446 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3db40907b0 0x7f3db4094370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:58.758 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:58.760 7f3db924c700 1 --2- 172.21.2.102:0/366803446 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3d98018730 0x7f3d9801ac10 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:58.758 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:58.760 7f3db924c700 1 --2- 172.21.2.102:0/366803446 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3db408fd90 0x7f3db4090190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:58.758 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:58.760 7f3db924c700 1 --2- 172.21.2.102:0/366803446 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3db408f500 0x7f3db419f900 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:58.758 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:58.760 7f3db924c700 1 -- 172.21.2.102:0/366803446 >> 172.21.2.102:0/366803446 conn(0x7f3db40b5560 msgr2=0x7f3db40a6200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:04:58.758 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:58.760 7f3db924c700 1 -- 172.21.2.102:0/366803446 shutdown_connections 2021-02-10T11:04:58.758 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:58.760 7f3db924c700 1 -- 172.21.2.102:0/366803446 wait complete. 2021-02-10T11:04:58.766 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:04:58.797 INFO:tasks.ceph.mds.b.gibba013.stdout:starting mds.b at 2021-02-10T11:04:59.191 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:59.176 7f6b3a15d700 1 Processor -- start 2021-02-10T11:04:59.191 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:59.176 7f6b3a15d700 1 -- start start 2021-02-10T11:04:59.191 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:59.176 7f6b3a15d700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f6b340e5470 0x7f6b3407e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:59.191 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:59.176 7f6b3a15d700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6b3407f300 0x7f6b3407f700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:59.192 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:59.176 7f6b3a15d700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b3407ed80 con 0x7f6b340e5070 2021-02-10T11:04:59.192 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:59.176 7f6b3a15d700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b341b4010 con 0x7f6b340e5470 2021-02-10T11:04:59.192 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:59.176 7f6b3a15d700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b340d1790 con 0x7f6b3407f300 2021-02-10T11:04:59.192 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:59.177 7f6b32ffd700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f6b340e5470 0x7f6b3407e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:59.192 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:59.177 7f6b32ffd700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f6b340e5470 0x7f6b3407e6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:50276/0 (socket says 172.21.2.102:50276) 2021-02-10T11:04:59.192 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:59.177 7f6b32ffd700 1 -- 172.21.2.102:0/1681887229 learned_addr learned my addr 172.21.2.102:0/1681887229 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:04:59.192 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:59.177 7f6b32ffd700 1 -- 172.21.2.102:0/1681887229 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6b3407f300 msgr2=0x7f6b3407f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:04:59.193 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:59.177 7f6b32ffd700 1 --2- 172.21.2.102:0/1681887229 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6b3407f300 0x7f6b3407f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:59.193 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:59.177 7f6b32ffd700 1 -- 172.21.2.102:0/1681887229 >> v1:172.21.2.113:6789/0 conn(0x7f6b340e5070 legacy=0x7f6b341b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:04:59.193 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:59.177 7f6b32ffd700 1 -- 172.21.2.102:0/1681887229 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b340d19b0 con 0x7f6b340e5470 2021-02-10T11:04:59.193 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:59.177 7f6b32ffd700 1 --2- 172.21.2.102:0/1681887229 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6b340e5470 0x7f6b3407e6a0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 rx=0x7f6b1c006700 tx=0x7f6b1c009b00).ready entity=mon.0 client_cookie=5ae9ae43dbba8f82 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:04:59.193 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:59.177 7f6b327fc700 1 -- 172.21.2.102:0/1681887229 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6b1c00c0c0 con 0x7f6b340e5470 2021-02-10T11:04:59.193 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:59.177 7f6b327fc700 1 -- 172.21.2.102:0/1681887229 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b1c00c340 con 0x7f6b340e5470 2021-02-10T11:04:59.193 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:59.177 7f6b327fc700 1 -- 172.21.2.102:0/1681887229 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6b1c00c820 con 0x7f6b340e5470 2021-02-10T11:04:59.193 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:59.178 7f6b3a15d700 1 -- 172.21.2.102:0/1681887229 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6b340e5470 msgr2=0x7f6b3407e6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:59.193 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:59.178 7f6b3a15d700 1 --2- 172.21.2.102:0/1681887229 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6b340e5470 0x7f6b3407e6a0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 rx=0x7f6b1c006700 tx=0x7f6b1c009b00).stop 2021-02-10T11:04:59.193 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:59.178 7f6b3a15d700 1 -- 172.21.2.102:0/1681887229 shutdown_connections 2021-02-10T11:04:59.193 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:59.178 7f6b3a15d700 1 --2- 172.21.2.102:0/1681887229 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6b3407f300 0x7f6b3407f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:59.193 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:59.178 7f6b3a15d700 1 --2- 172.21.2.102:0/1681887229 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6b340e5470 0x7f6b3407e6a0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:59.194 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:59.178 7f6b3a15d700 1 -- 172.21.2.102:0/1681887229 >> 172.21.2.102:0/1681887229 conn(0x7f6b340ce690 msgr2=0x7f6b340e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:04:59.194 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:59.178 7f6b3a15d700 1 -- 172.21.2.102:0/1681887229 shutdown_connections 2021-02-10T11:04:59.194 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:59.178 7f6b3a15d700 1 -- 172.21.2.102:0/1681887229 wait complete. 2021-02-10T11:04:59.194 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:59.193 7f6b3a15d700 1 Processor -- start 2021-02-10T11:04:59.194 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:59.193 7f6b3a15d700 1 -- start start 2021-02-10T11:04:59.196 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:59.193 7f6b3a15d700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6b340a5770 0x7f6b340b72b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:59.196 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:59.193 7f6b3a15d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6b340b78d0 0x7f6b340b7cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:59.196 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:59.193 7f6b3a15d700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6b341a2b50 0x7f6b341a2f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:59.196 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:59.193 7f6b3a15d700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b341a3df0 con 0x7f6b340b78d0 2021-02-10T11:04:59.196 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:59.193 7f6b3a15d700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b34092d90 con 0x7f6b341a2b50 2021-02-10T11:04:59.196 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:59.193 7f6b3a15d700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b341a3570 con 0x7f6b340a5770 2021-02-10T11:04:59.196 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:59.194 7f6b32ffd700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6b340b78d0 0x7f6b340b7cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:59.196 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:59.194 7f6b32ffd700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6b340b78d0 0x7f6b340b7cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:50286/0 (socket says 172.21.2.102:50286) 2021-02-10T11:04:59.196 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:59.194 7f6b32ffd700 1 -- 172.21.2.102:0/65766046 learned_addr learned my addr 172.21.2.102:0/65766046 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:04:59.196 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:59.194 7f6b32ffd700 1 -- 172.21.2.102:0/65766046 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6b340a5770 msgr2=0x7f6b340b72b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:04:59.196 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:59.194 7f6b32ffd700 1 --2- 172.21.2.102:0/65766046 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6b340a5770 0x7f6b340b72b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:59.197 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:59.194 7f6b32ffd700 1 -- 172.21.2.102:0/65766046 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6b341a2b50 msgr2=0x7f6b341a2f50 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:04:59.197 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:59.194 7f6b32ffd700 1 --2- 172.21.2.102:0/65766046 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6b341a2b50 0x7f6b341a2f50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:59.197 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:59.194 7f6b32ffd700 1 -- 172.21.2.102:0/65766046 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b341a3790 con 0x7f6b340b78d0 2021-02-10T11:04:59.197 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:59.194 7f6b32ffd700 1 --2- 172.21.2.102:0/65766046 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6b340b78d0 0x7f6b340b7cd0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0x7f6b1c006f20 tx=0x7f6b1c00b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:04:59.197 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:59.194 7f6b2bfff700 1 -- 172.21.2.102:0/65766046 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6b1c00bd80 con 0x7f6b340b78d0 2021-02-10T11:04:59.197 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:59.194 7f6b2bfff700 1 -- 172.21.2.102:0/65766046 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b1c000c90 con 0x7f6b340b78d0 2021-02-10T11:04:59.197 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:59.194 7f6b2bfff700 1 -- 172.21.2.102:0/65766046 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6b1c0022b0 con 0x7f6b340b78d0 2021-02-10T11:04:59.197 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:59.194 7f6b3a15d700 1 -- 172.21.2.102:0/65766046 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b341a3790 con 0x7f6b340b78d0 2021-02-10T11:04:59.197 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:59.194 7f6b3a15d700 1 -- 172.21.2.102:0/65766046 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b341a3790 con 0x7f6b340b78d0 2021-02-10T11:04:59.197 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:59.194 7f6b2bfff700 1 -- 172.21.2.102:0/65766046 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f6b1c002020 con 0x7f6b340b78d0 2021-02-10T11:04:59.197 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:59.195 7f6b2bfff700 1 --2- 172.21.2.102:0/65766046 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6b18018730 0x7f6b1801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:59.197 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:59.195 7f6b2bfff700 1 -- 172.21.2.102:0/65766046 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5659+0+0 (secure 0 0 0) 0x7f6b1c032840 con 0x7f6b340b78d0 2021-02-10T11:04:59.197 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:59.195 7f6b337fe700 1 --2- 172.21.2.102:0/65766046 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6b18018730 0x7f6b1801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:59.198 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:59.195 7f6b337fe700 1 --2- 172.21.2.102:0/65766046 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6b18018730 0x7f6b1801abc0 crc :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:04:59.225 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:59.227 7f6b3a15d700 1 -- 172.21.2.102:0/65766046 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b34083dd0 con 0x7f6b340b78d0 2021-02-10T11:04:59.227 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:59.229 7f6b2bfff700 1 -- 172.21.2.102:0/65766046 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f6b1c006400 con 0x7f6b340b78d0 2021-02-10T11:04:59.357 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:59.358 7f6b3a15d700 1 -- 172.21.2.102:0/65766046 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6b34083dd0 con 0x7f6b340b78d0 2021-02-10T11:04:59.357 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:59.359 7f6b2bfff700 1 -- 172.21.2.102:0/65766046 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+1753 (secure 0 0 0) 0x7f6b1c00bd80 con 0x7f6b340b78d0 2021-02-10T11:04:59.358 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:04:59.358 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4302,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.102:6835/3297596596","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6834","nonce":3297596596},{"type":"v1","addr":"172.21.2.102:6835","nonce":3297596596}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":4}],"filesystems":[{"mdsmap":{"epoch":7,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:03:43.384709","modified":"2021-02-10 11:04:58.750306","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":21,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2021-02-10T11:04:59.358 INFO:teuthology.orchestra.run.gibba002.stderr:dumped fsmap epoch 7 2021-02-10T11:04:59.359 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:59.361 7f6b3a15d700 1 -- 172.21.2.102:0/65766046 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6b18018730 msgr2=0x7f6b1801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:59.359 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:59.361 7f6b3a15d700 1 --2- 172.21.2.102:0/65766046 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6b18018730 0x7f6b1801abc0 crc :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:59.360 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:59.361 7f6b3a15d700 1 -- 172.21.2.102:0/65766046 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6b340b78d0 msgr2=0x7f6b340b7cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:59.360 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:59.361 7f6b3a15d700 1 --2- 172.21.2.102:0/65766046 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6b340b78d0 0x7f6b340b7cd0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0x7f6b1c006f20 tx=0x7f6b1c00b990).stop 2021-02-10T11:04:59.360 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:59.361 7f6b3a15d700 1 -- 172.21.2.102:0/65766046 shutdown_connections 2021-02-10T11:04:59.360 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:59.361 7f6b3a15d700 1 --2- 172.21.2.102:0/65766046 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6b18018730 0x7f6b1801abc0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:59.360 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:59.361 7f6b3a15d700 1 --2- 172.21.2.102:0/65766046 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6b341a2b50 0x7f6b341a2f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:59.361 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:59.361 7f6b3a15d700 1 --2- 172.21.2.102:0/65766046 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6b340b78d0 0x7f6b340b7cd0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:59.361 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:59.361 7f6b3a15d700 1 --2- 172.21.2.102:0/65766046 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6b340a5770 0x7f6b340b72b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:59.361 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:59.361 7f6b3a15d700 1 -- 172.21.2.102:0/65766046 >> 172.21.2.102:0/65766046 conn(0x7f6b34093720 msgr2=0x7f6b34084b80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:04:59.361 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:59.361 7f6b3a15d700 1 -- 172.21.2.102:0/65766046 shutdown_connections 2021-02-10T11:04:59.361 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:59.361 7f6b3a15d700 1 -- 172.21.2.102:0/65766046 wait complete. 2021-02-10T11:04:59.367 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2021-02-10T11:04:59.799 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:59.785 7f04d5f74700 1 Processor -- start 2021-02-10T11:04:59.800 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:59.785 7f04d5f74700 1 -- start start 2021-02-10T11:04:59.800 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:59.786 7f04d5f74700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f04d00d17f0 0x7f04d01b31e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:59.800 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:59.786 7f04d5f74700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f04d00d1bf0 0x7f04d01b3800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:59.800 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:59.786 7f04d5f74700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04d00f0b20 con 0x7f04d01b3e20 2021-02-10T11:04:59.800 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:59.786 7f04d5f74700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04d00e5eb0 con 0x7f04d00d1bf0 2021-02-10T11:04:59.800 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:59.786 7f04d5f74700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04d00f0630 con 0x7f04d00d17f0 2021-02-10T11:04:59.800 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:59.786 7f04ceffd700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f04d00d1bf0 0x7f04d01b3800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:59.800 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:59.786 7f04ceffd700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f04d00d1bf0 0x7f04d01b3800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:50290/0 (socket says 172.21.2.102:50290) 2021-02-10T11:04:59.800 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:59.786 7f04ceffd700 1 -- 172.21.2.102:0/2777283474 learned_addr learned my addr 172.21.2.102:0/2777283474 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:04:59.800 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:59.786 7f04ceffd700 1 -- 172.21.2.102:0/2777283474 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f04d00d17f0 msgr2=0x7f04d01b31e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:04:59.800 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:59.786 7f04ceffd700 1 --2- 172.21.2.102:0/2777283474 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f04d00d17f0 0x7f04d01b31e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:59.801 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:59.786 7f04ceffd700 1 -- 172.21.2.102:0/2777283474 >> v1:172.21.2.113:6789/0 conn(0x7f04d01b3e20 legacy=0x7f04d00ceef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:04:59.801 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:59.786 7f04ceffd700 1 -- 172.21.2.102:0/2777283474 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04d00f0820 con 0x7f04d00d1bf0 2021-02-10T11:04:59.801 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:59.786 7f04ceffd700 1 --2- 172.21.2.102:0/2777283474 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f04d00d1bf0 0x7f04d01b3800 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 rx=0x7f04b8006700 tx=0x7f04b8009b80).ready entity=mon.0 client_cookie=ad085758c91e47c2 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:04:59.801 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:59.787 7f04ce7fc700 1 -- 172.21.2.102:0/2777283474 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f04b800c0c0 con 0x7f04d00d1bf0 2021-02-10T11:04:59.801 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:59.787 7f04ce7fc700 1 -- 172.21.2.102:0/2777283474 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04b800c440 con 0x7f04d00d1bf0 2021-02-10T11:04:59.801 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:59.787 7f04ce7fc700 1 -- 172.21.2.102:0/2777283474 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f04b800c8a0 con 0x7f04d00d1bf0 2021-02-10T11:04:59.801 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:59.787 7f04d5f74700 1 -- 172.21.2.102:0/2777283474 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f04d00d1bf0 msgr2=0x7f04d01b3800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:04:59.801 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:59.787 7f04d5f74700 1 --2- 172.21.2.102:0/2777283474 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f04d00d1bf0 0x7f04d01b3800 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 rx=0x7f04b8006700 tx=0x7f04b8009b80).stop 2021-02-10T11:04:59.801 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:59.787 7f04d5f74700 1 -- 172.21.2.102:0/2777283474 shutdown_connections 2021-02-10T11:04:59.801 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:59.787 7f04d5f74700 1 --2- 172.21.2.102:0/2777283474 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f04d00d1bf0 0x7f04d01b3800 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:04:59.801 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:59.787 7f04d5f74700 1 --2- 172.21.2.102:0/2777283474 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f04d00d17f0 0x7f04d01b31e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:59.801 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:59.787 7f04d5f74700 1 -- 172.21.2.102:0/2777283474 >> 172.21.2.102:0/2777283474 conn(0x7f04d00fde40 msgr2=0x7f04d00ce390 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:04:59.801 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:59.787 7f04d5f74700 1 -- 172.21.2.102:0/2777283474 shutdown_connections 2021-02-10T11:04:59.802 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:59.787 7f04d5f74700 1 -- 172.21.2.102:0/2777283474 wait complete. 2021-02-10T11:04:59.802 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:59.802 7f04d5f74700 1 Processor -- start 2021-02-10T11:04:59.802 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:59.802 7f04d5f74700 1 -- start start 2021-02-10T11:04:59.803 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:59.802 7f04d5f74700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f04d01960f0 0x7f04d019b6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:59.803 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:59.802 7f04d5f74700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f04d018ecc0 0x7f04d0195740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:59.803 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:59.802 7f04d5f74700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f04d019c280 0x7f04d019c680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:59.803 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:59.802 7f04d5f74700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04d018f0c0 con 0x7f04d019c280 2021-02-10T11:04:59.803 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:59.802 7f04d5f74700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04d0195d60 con 0x7f04d018ecc0 2021-02-10T11:04:59.803 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:59.802 7f04d5f74700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04d019cca0 con 0x7f04d01960f0 2021-02-10T11:04:59.803 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:59.803 7f04cffff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f04d019c280 0x7f04d019c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:59.803 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:59.803 7f04cffff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f04d019c280 0x7f04d019c680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:50296/0 (socket says 172.21.2.102:50296) 2021-02-10T11:04:59.803 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:59.803 7f04cffff700 1 -- 172.21.2.102:0/4148713405 learned_addr learned my addr 172.21.2.102:0/4148713405 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:04:59.804 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:59.803 7f04cffff700 1 -- 172.21.2.102:0/4148713405 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f04d01960f0 msgr2=0x7f04d019b6a0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:04:59.804 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:59.803 7f04cffff700 1 --2- 172.21.2.102:0/4148713405 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f04d01960f0 0x7f04d019b6a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:59.804 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:59.803 7f04cffff700 1 -- 172.21.2.102:0/4148713405 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f04d018ecc0 msgr2=0x7f04d0195740 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:04:59.804 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:59.803 7f04cffff700 1 --2- 172.21.2.102:0/4148713405 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f04d018ecc0 0x7f04d0195740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:04:59.804 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:59.803 7f04cffff700 1 -- 172.21.2.102:0/4148713405 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04d019cec0 con 0x7f04d019c280 2021-02-10T11:04:59.804 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:59.803 7f04cffff700 1 --2- 172.21.2.102:0/4148713405 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f04d019c280 0x7f04d019c680 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0x7f04c8009940 tx=0x7f04c800bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:04:59.804 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:59.803 7f04c7fff700 1 -- 172.21.2.102:0/4148713405 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f04c8000f90 con 0x7f04d019c280 2021-02-10T11:04:59.804 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:59.803 7f04c7fff700 1 -- 172.21.2.102:0/4148713405 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04c8000d30 con 0x7f04d019c280 2021-02-10T11:04:59.804 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:59.803 7f04c7fff700 1 -- 172.21.2.102:0/4148713405 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f04c800c570 con 0x7f04d019c280 2021-02-10T11:04:59.804 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:59.803 7f04d5f74700 1 -- 172.21.2.102:0/4148713405 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04d019cec0 con 0x7f04d019c280 2021-02-10T11:04:59.804 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:59.803 7f04d5f74700 1 -- 172.21.2.102:0/4148713405 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04d019d450 con 0x7f04d019c280 2021-02-10T11:04:59.804 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:59.804 7f04c7fff700 1 -- 172.21.2.102:0/4148713405 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f04c8001220 con 0x7f04d019c280 2021-02-10T11:04:59.804 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:59.804 7f04c7fff700 1 --2- 172.21.2.102:0/4148713405 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f04b40186c0 0x7f04b401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:04:59.805 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:59.804 7f04c7fff700 1 -- 172.21.2.102:0/4148713405 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5659+0+0 (secure 0 0 0) 0x7f04c8032970 con 0x7f04d019c280 2021-02-10T11:04:59.805 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:59.804 7f04cf7fe700 1 --2- 172.21.2.102:0/4148713405 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f04b40186c0 0x7f04b401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:04:59.805 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:59.804 7f04cf7fe700 1 --2- 172.21.2.102:0/4148713405 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f04b40186c0 0x7f04b401abc0 crc :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:04:59.833 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:59.835 7f04d5f74700 1 -- 172.21.2.102:0/4148713405 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04d019bd50 con 0x7f04d019c280 2021-02-10T11:04:59.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:59.838 7f04c7fff700 1 -- 172.21.2.102:0/4148713405 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f04c8000f40 con 0x7f04d019c280 2021-02-10T11:04:59.963 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:04:59.965 7f04d5f74700 1 -- 172.21.2.102:0/4148713405 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f04d019bd50 con 0x7f04d019c280 2021-02-10T11:05:00.761 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:00.764 7f04c7fff700 1 -- 172.21.2.102:0/4148713405 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v8) v1 ==== 104+0+0 (secure 0 0 0) 0x7f04c801f880 con 0x7f04d019c280 2021-02-10T11:05:00.785 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:00.787 7f04d5f74700 1 -- 172.21.2.102:0/4148713405 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f04b40186c0 msgr2=0x7f04b401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:00.785 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:00.787 7f04d5f74700 1 --2- 172.21.2.102:0/4148713405 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f04b40186c0 0x7f04b401abc0 crc :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:00.785 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:00.787 7f04d5f74700 1 -- 172.21.2.102:0/4148713405 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f04d019c280 msgr2=0x7f04d019c680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:00.785 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:00.787 7f04d5f74700 1 --2- 172.21.2.102:0/4148713405 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f04d019c280 0x7f04d019c680 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0x7f04c8009940 tx=0x7f04c800bcf0).stop 2021-02-10T11:05:00.786 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:00.787 7f04d5f74700 1 -- 172.21.2.102:0/4148713405 shutdown_connections 2021-02-10T11:05:00.786 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:00.787 7f04d5f74700 1 --2- 172.21.2.102:0/4148713405 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f04d019c280 0x7f04d019c680 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:00.786 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:00.787 7f04d5f74700 1 --2- 172.21.2.102:0/4148713405 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f04b40186c0 0x7f04b401abc0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:00.786 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:00.787 7f04d5f74700 1 --2- 172.21.2.102:0/4148713405 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f04d018ecc0 0x7f04d0195740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:05:00.786 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:00.787 7f04d5f74700 1 --2- 172.21.2.102:0/4148713405 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f04d01960f0 0x7f04d019b6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:05:00.786 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:00.787 7f04d5f74700 1 -- 172.21.2.102:0/4148713405 >> 172.21.2.102:0/4148713405 conn(0x7f04d019f7e0 msgr2=0x7f04d018c240 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:05:00.787 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:00.787 7f04d5f74700 1 -- 172.21.2.102:0/4148713405 shutdown_connections 2021-02-10T11:05:00.787 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:00.787 7f04d5f74700 1 -- 172.21.2.102:0/4148713405 wait complete. 2021-02-10T11:05:00.793 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool delete cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2021-02-10T11:05:01.222 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:01.208 7fce146eb700 1 Processor -- start 2021-02-10T11:05:01.222 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:01.208 7fce146eb700 1 -- start start 2021-02-10T11:05:01.222 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:01.209 7fce146eb700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fce0c0d1c70 0x7fce0c0cef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:01.222 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:01.209 7fce146eb700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fce0c0cf580 0x7fce0c0cf980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:01.222 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:01.209 7fce146eb700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce0c0fe1c0 con 0x7fce0c0d1870 2021-02-10T11:05:01.222 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:01.209 7fce146eb700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce0c1b0cb0 con 0x7fce0c0d1c70 2021-02-10T11:05:01.223 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:01.209 7fce146eb700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce0c109a60 con 0x7fce0c0cf580 2021-02-10T11:05:01.223 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:01.209 7fce11d06700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fce0c0d1c70 0x7fce0c0cef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:01.223 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:01.209 7fce11d06700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fce0c0d1c70 0x7fce0c0cef60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:50302/0 (socket says 172.21.2.102:50302) 2021-02-10T11:05:01.223 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:01.209 7fce11d06700 1 -- 172.21.2.102:0/3129082988 learned_addr learned my addr 172.21.2.102:0/3129082988 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:05:01.223 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:01.209 7fce11d06700 1 -- 172.21.2.102:0/3129082988 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fce0c0cf580 msgr2=0x7fce0c0cf980 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:05:01.223 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:01.209 7fce11d06700 1 --2- 172.21.2.102:0/3129082988 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fce0c0cf580 0x7fce0c0cf980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:05:01.223 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:01.209 7fce11d06700 1 -- 172.21.2.102:0/3129082988 >> v1:172.21.2.113:6789/0 conn(0x7fce0c0d1870 legacy=0x7fce0c1b04c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:05:01.223 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:01.209 7fce11d06700 1 -- 172.21.2.102:0/3129082988 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce0c088020 con 0x7fce0c0d1c70 2021-02-10T11:05:01.223 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:01.209 7fce11d06700 1 --2- 172.21.2.102:0/3129082988 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fce0c0d1c70 0x7fce0c0cef60 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 rx=0x7fce08006700 tx=0x7fce08009b80).ready entity=mon.0 client_cookie=1f085507bbec0dcc server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:05:01.223 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:01.209 7fce11505700 1 -- 172.21.2.102:0/3129082988 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fce0800c0c0 con 0x7fce0c0d1c70 2021-02-10T11:05:01.223 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:01.210 7fce11505700 1 -- 172.21.2.102:0/3129082988 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce0800c440 con 0x7fce0c0d1c70 2021-02-10T11:05:01.223 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:01.210 7fce11505700 1 -- 172.21.2.102:0/3129082988 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fce0800c920 con 0x7fce0c0d1c70 2021-02-10T11:05:01.223 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:01.210 7fce146eb700 1 -- 172.21.2.102:0/3129082988 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fce0c0d1c70 msgr2=0x7fce0c0cef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:01.224 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:01.210 7fce146eb700 1 --2- 172.21.2.102:0/3129082988 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fce0c0d1c70 0x7fce0c0cef60 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 rx=0x7fce08006700 tx=0x7fce08009b80).stop 2021-02-10T11:05:01.224 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:01.210 7fce146eb700 1 -- 172.21.2.102:0/3129082988 shutdown_connections 2021-02-10T11:05:01.224 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:01.210 7fce146eb700 1 --2- 172.21.2.102:0/3129082988 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fce0c0cf580 0x7fce0c0cf980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:05:01.224 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:01.210 7fce146eb700 1 --2- 172.21.2.102:0/3129082988 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fce0c0d1c70 0x7fce0c0cef60 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:01.224 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:01.210 7fce146eb700 1 -- 172.21.2.102:0/3129082988 >> 172.21.2.102:0/3129082988 conn(0x7fce0c0ce420 msgr2=0x7fce0c0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:05:01.224 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:01.210 7fce146eb700 1 -- 172.21.2.102:0/3129082988 shutdown_connections 2021-02-10T11:05:01.224 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:01.210 7fce146eb700 1 -- 172.21.2.102:0/3129082988 wait complete. 2021-02-10T11:05:01.225 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:01.225 7fce146eb700 1 Processor -- start 2021-02-10T11:05:01.226 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:01.225 7fce146eb700 1 -- start start 2021-02-10T11:05:01.226 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:01.225 7fce146eb700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fce0c0b7d30 0x7fce0c0b8130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:01.226 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:01.225 7fce146eb700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fce0c095780 0x7fce0c095b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:01.226 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:01.225 7fce146eb700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fce0c0961a0 0x7fce0c0965a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:01.226 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:01.225 7fce146eb700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce0c19f110 con 0x7fce0c0b7d30 2021-02-10T11:05:01.226 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:01.225 7fce146eb700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce0c1abb60 con 0x7fce0c095780 2021-02-10T11:05:01.226 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:01.225 7fce146eb700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce0c194a70 con 0x7fce0c0961a0 2021-02-10T11:05:01.226 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:01.225 7fce12507700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fce0c0b7d30 0x7fce0c0b8130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:01.226 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:01.225 7fce12507700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fce0c0b7d30 0x7fce0c0b8130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:50310/0 (socket says 172.21.2.102:50310) 2021-02-10T11:05:01.226 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:01.225 7fce12507700 1 -- 172.21.2.102:0/71940789 learned_addr learned my addr 172.21.2.102:0/71940789 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:05:01.226 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:01.226 7fce12507700 1 -- 172.21.2.102:0/71940789 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fce0c0961a0 msgr2=0x7fce0c0965a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:01.226 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:01.226 7fce12507700 1 --2- 172.21.2.102:0/71940789 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fce0c0961a0 0x7fce0c0965a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:05:01.226 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:01.226 7fce12507700 1 -- 172.21.2.102:0/71940789 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fce0c095780 msgr2=0x7fce0c095b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:01.227 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:01.226 7fce12507700 1 --2- 172.21.2.102:0/71940789 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fce0c095780 0x7fce0c095b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:05:01.227 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:01.226 7fce12507700 1 -- 172.21.2.102:0/71940789 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce0c194c90 con 0x7fce0c0b7d30 2021-02-10T11:05:01.227 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:01.226 7fce12507700 1 --2- 172.21.2.102:0/71940789 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fce0c0b7d30 0x7fce0c0b8130 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7fce0000b5b0 tx=0x7fce0000b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:05:01.227 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:01.226 7fcdfeffd700 1 -- 172.21.2.102:0/71940789 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fce0000bd40 con 0x7fce0c0b7d30 2021-02-10T11:05:01.227 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:01.226 7fcdfeffd700 1 -- 172.21.2.102:0/71940789 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce00006240 con 0x7fce0c0b7d30 2021-02-10T11:05:01.227 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:01.226 7fcdfeffd700 1 -- 172.21.2.102:0/71940789 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fce00005690 con 0x7fce0c0b7d30 2021-02-10T11:05:01.227 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:01.226 7fce146eb700 1 -- 172.21.2.102:0/71940789 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce0c194c90 con 0x7fce0c0b7d30 2021-02-10T11:05:01.227 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:01.226 7fce146eb700 1 -- 172.21.2.102:0/71940789 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce0c195220 con 0x7fce0c0b7d30 2021-02-10T11:05:01.227 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:01.227 7fcdfeffd700 1 -- 172.21.2.102:0/71940789 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fce000054a0 con 0x7fce0c0b7d30 2021-02-10T11:05:01.227 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:01.227 7fcdfeffd700 1 --2- 172.21.2.102:0/71940789 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcdf40186c0 0x7fcdf401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:01.227 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:01.227 7fcdfeffd700 1 -- 172.21.2.102:0/71940789 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5659+0+0 (secure 0 0 0) 0x7fce00032460 con 0x7fce0c0b7d30 2021-02-10T11:05:01.227 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:01.227 7fce11d06700 1 --2- 172.21.2.102:0/71940789 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcdf40186c0 0x7fcdf401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:01.227 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:01.227 7fce11d06700 1 --2- 172.21.2.102:0/71940789 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcdf40186c0 0x7fcdf401abc0 crc :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:05:01.257 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:01.259 7fce146eb700 1 -- 172.21.2.102:0/71940789 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce0c1961b0 con 0x7fce0c0b7d30 2021-02-10T11:05:01.259 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:01.261 7fcdfeffd700 1 -- 172.21.2.102:0/71940789 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fce00006320 con 0x7fce0c0b7d30 2021-02-10T11:05:01.388 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:01.391 7fce146eb700 1 -- 172.21.2.102:0/71940789 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"pool2": "cephfs_metadata", "prefix": "osd pool delete", "pool": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fce0c1961b0 con 0x7fce0c0b7d30 2021-02-10T11:05:01.766 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:01.768 7fcdfeffd700 1 -- 172.21.2.102:0/71940789 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"pool2": "cephfs_metadata", "prefix": "osd pool delete", "pool": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v22) v1 ==== 185+0+0 (secure 0 0 0) 0x7fce000054a0 con 0x7fce0c0b7d30 2021-02-10T11:05:01.803 INFO:teuthology.orchestra.run.gibba002.stderr:pool 'cephfs_metadata' removed 2021-02-10T11:05:01.805 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:01.807 7fce146eb700 1 -- 172.21.2.102:0/71940789 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcdf40186c0 msgr2=0x7fcdf401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:01.805 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:01.807 7fce146eb700 1 --2- 172.21.2.102:0/71940789 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcdf40186c0 0x7fcdf401abc0 crc :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:01.805 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:01.807 7fce146eb700 1 -- 172.21.2.102:0/71940789 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fce0c0b7d30 msgr2=0x7fce0c0b8130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:01.805 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:01.807 7fce146eb700 1 --2- 172.21.2.102:0/71940789 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fce0c0b7d30 0x7fce0c0b8130 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7fce0000b5b0 tx=0x7fce0000b950).stop 2021-02-10T11:05:01.806 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:01.807 7fce146eb700 1 -- 172.21.2.102:0/71940789 shutdown_connections 2021-02-10T11:05:01.806 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:01.807 7fce146eb700 1 --2- 172.21.2.102:0/71940789 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fce0c0961a0 0x7fce0c0965a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:05:01.806 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:01.807 7fce146eb700 1 --2- 172.21.2.102:0/71940789 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcdf40186c0 0x7fcdf401abc0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:01.806 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:01.807 7fce146eb700 1 --2- 172.21.2.102:0/71940789 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fce0c095780 0x7fce0c095b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:05:01.806 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:01.807 7fce146eb700 1 --2- 172.21.2.102:0/71940789 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fce0c0b7d30 0x7fce0c0b8130 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:01.806 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:01.807 7fce146eb700 1 -- 172.21.2.102:0/71940789 >> 172.21.2.102:0/71940789 conn(0x7fce0c19f820 msgr2=0x7fce0c082b10 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:05:01.806 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:01.807 7fce146eb700 1 -- 172.21.2.102:0/71940789 shutdown_connections 2021-02-10T11:05:01.806 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:01.807 7fce146eb700 1 -- 172.21.2.102:0/71940789 wait complete. 2021-02-10T11:05:01.814 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool delete cephfs_data cephfs_data --yes-i-really-really-mean-it 2021-02-10T11:05:02.244 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:02.229 7f075acd8700 1 Processor -- start 2021-02-10T11:05:02.244 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:02.229 7f075acd8700 1 -- start start 2021-02-10T11:05:02.244 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:02.229 7f075acd8700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f07540d1870 0x7f07541b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:02.244 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:02.229 7f075acd8700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f07541b0ae0 0x7f0754108940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:02.245 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:02.229 7f075acd8700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0754109a60 con 0x7f07540d1c70 2021-02-10T11:05:02.245 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:02.229 7f075acd8700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07540e5ef0 con 0x7f07541b0ae0 2021-02-10T11:05:02.245 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:02.229 7f075acd8700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0754109750 con 0x7f07540d1870 2021-02-10T11:05:02.245 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:02.230 7f0753fff700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f07541b0ae0 0x7f0754108940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:02.245 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:02.230 7f0758af4700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f07540d1870 0x7f07541b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:02.245 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:02.230 7f07592f5700 1 --1- >> v1:172.21.2.113:6789/0 conn(0x7f07540d1c70 0x7f0754108f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.113:6789/0 says I am v1:172.21.2.102:49868/0 (socket says 172.21.2.102:49868) 2021-02-10T11:05:02.245 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:02.230 7f07592f5700 1 -- 172.21.2.102:0/1826067954 learned_addr learned my addr 172.21.2.102:0/1826067954 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:05:02.245 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:02.231 7f0753fff700 1 -- 172.21.2.102:0/1826067954 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f07540d1870 msgr2=0x7f07541b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:05:02.245 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:02.231 7f0753fff700 1 --2- 172.21.2.102:0/1826067954 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f07540d1870 0x7f07541b04c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:02.245 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:02.231 7f0753fff700 1 -- 172.21.2.102:0/1826067954 >> v1:172.21.2.113:6789/0 conn(0x7f07540d1c70 legacy=0x7f0754108f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:02.245 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:02.231 7f0753fff700 1 -- 172.21.2.102:0/1826067954 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0754088020 con 0x7f07541b0ae0 2021-02-10T11:05:02.245 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:02.231 7f0758af4700 1 --2- 172.21.2.102:0/1826067954 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f07540d1870 0x7f07541b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:05:02.245 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:02.231 7f0753fff700 1 --2- 172.21.2.102:0/1826067954 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f07541b0ae0 0x7f0754108940 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f0748006650 tx=0x7f0748009a80).ready entity=mon.1 client_cookie=67fbc463e447a3d0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:05:02.246 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:02.231 7f07537fe700 1 -- 172.21.2.102:0/1826067954 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f07480009b0 con 0x7f07541b0ae0 2021-02-10T11:05:02.246 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:02.231 7f07537fe700 1 -- 172.21.2.102:0/1826067954 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f074800c0c0 con 0x7f07541b0ae0 2021-02-10T11:05:02.246 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:02.231 7f07537fe700 1 -- 172.21.2.102:0/1826067954 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f074800c520 con 0x7f07541b0ae0 2021-02-10T11:05:02.246 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:02.231 7f075acd8700 1 -- 172.21.2.102:0/1826067954 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f07541b0ae0 msgr2=0x7f0754108940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:02.246 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:02.231 7f075acd8700 1 --2- 172.21.2.102:0/1826067954 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f07541b0ae0 0x7f0754108940 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f0748006650 tx=0x7f0748009a80).stop 2021-02-10T11:05:02.246 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:02.232 7f075acd8700 1 -- 172.21.2.102:0/1826067954 shutdown_connections 2021-02-10T11:05:02.246 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:02.232 7f075acd8700 1 --2- 172.21.2.102:0/1826067954 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f07541b0ae0 0x7f0754108940 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:02.246 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:02.232 7f075acd8700 1 --2- 172.21.2.102:0/1826067954 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f07540d1870 0x7f07541b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:02.246 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:02.232 7f075acd8700 1 -- 172.21.2.102:0/1826067954 >> 172.21.2.102:0/1826067954 conn(0x7f07540ce420 msgr2=0x7f07540d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:05:02.246 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:02.232 7f075acd8700 1 -- 172.21.2.102:0/1826067954 shutdown_connections 2021-02-10T11:05:02.246 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:02.232 7f075acd8700 1 -- 172.21.2.102:0/1826067954 wait complete. 2021-02-10T11:05:02.246 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:02.246 7f075acd8700 1 Processor -- start 2021-02-10T11:05:02.246 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:02.246 7f075acd8700 1 -- start start 2021-02-10T11:05:02.248 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:02.247 7f075acd8700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f07540aa9e0 0x7f075418ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:02.248 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:02.247 7f075acd8700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f075418d4d0 0x7f0754098df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:02.248 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:02.247 7f075acd8700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0754099410 0x7f07541a6cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:02.248 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:02.247 7f075acd8700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0754098870 con 0x7f075418d4d0 2021-02-10T11:05:02.248 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:02.247 7f075acd8700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07541a72e0 con 0x7f07540aa9e0 2021-02-10T11:05:02.248 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:02.247 7f075acd8700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07541a74c0 con 0x7f0754099410 2021-02-10T11:05:02.248 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:02.247 7f0753fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f075418d4d0 0x7f0754098df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:02.249 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:02.247 7f0753fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f075418d4d0 0x7f0754098df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:50324/0 (socket says 172.21.2.102:50324) 2021-02-10T11:05:02.249 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:02.247 7f0753fff700 1 -- 172.21.2.102:0/2097210613 learned_addr learned my addr 172.21.2.102:0/2097210613 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:05:02.249 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:02.247 7f07592f5700 1 --2- 172.21.2.102:0/2097210613 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0754099410 0x7f07541a6cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:02.249 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:02.247 7f0758af4700 1 --2- 172.21.2.102:0/2097210613 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f07540aa9e0 0x7f075418ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:02.249 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:02.248 7f0753fff700 1 -- 172.21.2.102:0/2097210613 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0754099410 msgr2=0x7f07541a6cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:02.249 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:02.248 7f0753fff700 1 --2- 172.21.2.102:0/2097210613 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0754099410 0x7f07541a6cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:02.249 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:02.248 7f0753fff700 1 -- 172.21.2.102:0/2097210613 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f07540aa9e0 msgr2=0x7f075418ceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:02.249 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:02.248 7f0753fff700 1 --2- 172.21.2.102:0/2097210613 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f07540aa9e0 0x7f075418ceb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:02.249 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:02.248 7f0753fff700 1 -- 172.21.2.102:0/2097210613 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07541a76e0 con 0x7f075418d4d0 2021-02-10T11:05:02.249 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:02.248 7f0753fff700 1 --2- 172.21.2.102:0/2097210613 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f075418d4d0 0x7f0754098df0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 rx=0x7f0748006630 tx=0x7f074800a9a0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:05:02.249 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:02.248 7f07517fa700 1 -- 172.21.2.102:0/2097210613 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f07480009b0 con 0x7f075418d4d0 2021-02-10T11:05:02.249 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:02.248 7f07517fa700 1 -- 172.21.2.102:0/2097210613 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f074800ad90 con 0x7f075418d4d0 2021-02-10T11:05:02.249 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:02.248 7f07517fa700 1 -- 172.21.2.102:0/2097210613 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0748006ee0 con 0x7f075418d4d0 2021-02-10T11:05:02.249 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:02.248 7f075acd8700 1 -- 172.21.2.102:0/2097210613 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07541a76e0 con 0x7f075418d4d0 2021-02-10T11:05:02.250 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:02.248 7f075acd8700 1 -- 172.21.2.102:0/2097210613 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07541a76e0 con 0x7f075418d4d0 2021-02-10T11:05:02.250 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:02.248 7f07517fa700 1 -- 172.21.2.102:0/2097210613 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f0748001240 con 0x7f075418d4d0 2021-02-10T11:05:02.250 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:02.248 7f07517fa700 1 --2- 172.21.2.102:0/2097210613 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f07340186c0 0x7f073401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:02.250 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:02.248 7f07517fa700 1 -- 172.21.2.102:0/2097210613 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5221+0+0 (secure 0 0 0) 0x7f0748032260 con 0x7f075418d4d0 2021-02-10T11:05:02.250 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:02.249 7f0758af4700 1 --2- 172.21.2.102:0/2097210613 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f07340186c0 0x7f073401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:02.250 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:02.249 7f0758af4700 1 --2- 172.21.2.102:0/2097210613 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f07340186c0 0x7f073401abc0 crc :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:05:02.277 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:02.279 7f075acd8700 1 -- 172.21.2.102:0/2097210613 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07540828f0 con 0x7f075418d4d0 2021-02-10T11:05:02.279 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:02.282 7f07517fa700 1 -- 172.21.2.102:0/2097210613 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f0748026020 con 0x7f075418d4d0 2021-02-10T11:05:02.404 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:02.407 7f075acd8700 1 -- 172.21.2.102:0/2097210613 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"pool2": "cephfs_data", "prefix": "osd pool delete", "pool": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f0754109a60 con 0x7f075418d4d0 2021-02-10T11:05:02.771 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:02.773 7f07517fa700 1 -- 172.21.2.102:0/2097210613 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"pool2": "cephfs_data", "prefix": "osd pool delete", "pool": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v23) v1 ==== 173+0+0 (secure 0 0 0) 0x7f0748026020 con 0x7f075418d4d0 2021-02-10T11:05:02.819 INFO:teuthology.orchestra.run.gibba002.stderr:pool 'cephfs_data' removed 2021-02-10T11:05:02.821 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:02.824 7f075acd8700 1 -- 172.21.2.102:0/2097210613 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f07340186c0 msgr2=0x7f073401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:02.821 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:02.824 7f075acd8700 1 --2- 172.21.2.102:0/2097210613 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f07340186c0 0x7f073401abc0 crc :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:02.821 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:02.824 7f075acd8700 1 -- 172.21.2.102:0/2097210613 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f075418d4d0 msgr2=0x7f0754098df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:02.821 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:02.824 7f075acd8700 1 --2- 172.21.2.102:0/2097210613 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f075418d4d0 0x7f0754098df0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 rx=0x7f0748006630 tx=0x7f074800a9a0).stop 2021-02-10T11:05:02.822 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:02.824 7f075acd8700 1 -- 172.21.2.102:0/2097210613 shutdown_connections 2021-02-10T11:05:02.822 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:02.824 7f075acd8700 1 --2- 172.21.2.102:0/2097210613 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0754099410 0x7f07541a6cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:02.822 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:02.824 7f075acd8700 1 --2- 172.21.2.102:0/2097210613 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f075418d4d0 0x7f0754098df0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:02.822 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:02.824 7f075acd8700 1 --2- 172.21.2.102:0/2097210613 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f07340186c0 0x7f073401abc0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:02.822 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:02.824 7f075acd8700 1 --2- 172.21.2.102:0/2097210613 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f07540aa9e0 0x7f075418ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:02.822 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:02.824 7f075acd8700 1 -- 172.21.2.102:0/2097210613 >> 172.21.2.102:0/2097210613 conn(0x7f075419af60 msgr2=0x7f075418c280 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:05:02.822 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:02.824 7f075acd8700 1 -- 172.21.2.102:0/2097210613 shutdown_connections 2021-02-10T11:05:02.822 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:02.824 7f075acd8700 1 -- 172.21.2.102:0/2097210613 wait complete. 2021-02-10T11:05:02.830 INFO:tasks.ceph.mds.a:Restarting daemon 2021-02-10T11:05:02.831 INFO:tasks.ceph.mds.a:Stopping old one... 2021-02-10T11:05:02.831 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2021-02-10T11:05:02.831 INFO:teuthology.orchestra.run:waiting for 300 2021-02-10T11:05:02.831 INFO:tasks.ceph.mds.b:Restarting daemon 2021-02-10T11:05:02.831 INFO:tasks.ceph.mds.b:Stopping old one... 2021-02-10T11:05:02.831 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2021-02-10T11:05:02.832 INFO:teuthology.orchestra.run:waiting for 300 2021-02-10T11:05:02.837 INFO:tasks.ceph.mds.a:Stopped 2021-02-10T11:05:02.837 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2021-02-10T11:05:02.838 INFO:tasks.ceph.mds.b:Stopped 2021-02-10T11:05:02.838 DEBUG:teuthology.orchestra.run.gibba013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2021-02-10T11:05:02.878 INFO:tasks.ceph.mds.a:Started 2021-02-10T11:05:02.878 DEBUG:teuthology.parallel:result is None 2021-02-10T11:05:02.879 INFO:tasks.ceph.mds.b:Started 2021-02-10T11:05:02.879 DEBUG:teuthology.parallel:result is None 2021-02-10T11:05:02.880 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd blacklist clear 2021-02-10T11:05:03.282 INFO:tasks.ceph.mds.b.gibba013.stdout:starting mds.b at 2021-02-10T11:05:03.283 INFO:tasks.ceph.mds.a.gibba002.stdout:starting mds.a at 2021-02-10T11:05:03.305 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:03.291 7f7662ca4700 1 Processor -- start 2021-02-10T11:05:03.305 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:03.291 7f7662ca4700 1 -- start start 2021-02-10T11:05:03.306 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:03.292 7f7662ca4700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f765c0e5070 0x7f765c1b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:03.306 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:03.292 7f7662ca4700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f765c1b3e40 0x7f765c080310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:03.306 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:03.292 7f7662ca4700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f765c07ed80 con 0x7f765c0e5470 2021-02-10T11:05:03.306 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:03.292 7f7662ca4700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f765c0d1790 con 0x7f765c1b3e40 2021-02-10T11:05:03.306 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:03.292 7f7662ca4700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f765c0d1980 con 0x7f765c0e5070 2021-02-10T11:05:03.306 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:03.292 7f76612c1700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f765c1b3e40 0x7f765c080310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:03.306 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:03.292 7f76612c1700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f765c1b3e40 0x7f765c080310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:50342/0 (socket says 172.21.2.102:50342) 2021-02-10T11:05:03.306 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:03.292 7f76612c1700 1 -- 172.21.2.102:0/352005042 learned_addr learned my addr 172.21.2.102:0/352005042 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:05:03.306 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:03.292 7f7660ac0700 1 --2- 172.21.2.102:0/352005042 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f765c0e5070 0x7f765c1b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:03.306 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:03.292 7f76612c1700 1 -- 172.21.2.102:0/352005042 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f765c0e5070 msgr2=0x7f765c1b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:05:03.306 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:03.292 7f76612c1700 1 --2- 172.21.2.102:0/352005042 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f765c0e5070 0x7f765c1b3820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:03.306 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:03.292 7f76612c1700 1 -- 172.21.2.102:0/352005042 >> v1:172.21.2.113:6789/0 conn(0x7f765c0e5470 legacy=0x7f765c07d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:05:03.307 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:03.292 7f76612c1700 1 -- 172.21.2.102:0/352005042 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f765c0d1ba0 con 0x7f765c1b3e40 2021-02-10T11:05:03.307 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:03.292 7f76612c1700 1 --2- 172.21.2.102:0/352005042 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f765c1b3e40 0x7f765c080310 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 rx=0x7f7654006f00 tx=0x7f7654009f10).ready entity=mon.0 client_cookie=a0be14ee3baa88de server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:05:03.307 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:03.292 7f765b7fe700 1 -- 172.21.2.102:0/352005042 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f765400b3b0 con 0x7f765c1b3e40 2021-02-10T11:05:03.307 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:03.292 7f765b7fe700 1 -- 172.21.2.102:0/352005042 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f765400b730 con 0x7f765c1b3e40 2021-02-10T11:05:03.307 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:03.292 7f765b7fe700 1 -- 172.21.2.102:0/352005042 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f765400bc50 con 0x7f765c1b3e40 2021-02-10T11:05:03.307 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:03.292 7f7662ca4700 1 -- 172.21.2.102:0/352005042 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f765c1b3e40 msgr2=0x7f765c080310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:03.307 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:03.292 7f7662ca4700 1 --2- 172.21.2.102:0/352005042 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f765c1b3e40 0x7f765c080310 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 rx=0x7f7654006f00 tx=0x7f7654009f10).stop 2021-02-10T11:05:03.307 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:03.293 7f7662ca4700 1 -- 172.21.2.102:0/352005042 shutdown_connections 2021-02-10T11:05:03.307 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:03.293 7f7662ca4700 1 --2- 172.21.2.102:0/352005042 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f765c1b3e40 0x7f765c080310 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:03.307 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:03.293 7f7662ca4700 1 --2- 172.21.2.102:0/352005042 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f765c0e5070 0x7f765c1b3820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:03.307 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:03.293 7f7662ca4700 1 -- 172.21.2.102:0/352005042 >> 172.21.2.102:0/352005042 conn(0x7f765c0ce690 msgr2=0x7f765c0e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:05:03.307 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:03.293 7f7662ca4700 1 -- 172.21.2.102:0/352005042 shutdown_connections 2021-02-10T11:05:03.307 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:03.293 7f7662ca4700 1 -- 172.21.2.102:0/352005042 wait complete. 2021-02-10T11:05:03.308 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:03.308 7f7662ca4700 1 Processor -- start 2021-02-10T11:05:03.308 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:03.308 7f7662ca4700 1 -- start start 2021-02-10T11:05:03.309 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:03.308 7f7662ca4700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f765c082520 0x7f765c087ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:03.310 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:03.308 7f7662ca4700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f765c0880d0 0x7f765c0b1d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:03.310 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:03.308 7f7662ca4700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f765c0884d0 0x7f765c0b2370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:03.310 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:03.308 7f7662ca4700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f765c0b17d0 con 0x7f765c0884d0 2021-02-10T11:05:03.311 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:03.308 7f7662ca4700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f765c0b2990 con 0x7f765c082520 2021-02-10T11:05:03.311 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:03.308 7f7662ca4700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f765c0b2b70 con 0x7f765c0880d0 2021-02-10T11:05:03.311 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:03.308 7f76612c1700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f765c0884d0 0x7f765c0b2370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:03.311 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:03.308 7f76612c1700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f765c0884d0 0x7f765c0b2370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:50348/0 (socket says 172.21.2.102:50348) 2021-02-10T11:05:03.311 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:03.308 7f76612c1700 1 -- 172.21.2.102:0/3572286575 learned_addr learned my addr 172.21.2.102:0/3572286575 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:05:03.311 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:03.309 7f76612c1700 1 -- 172.21.2.102:0/3572286575 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f765c0880d0 msgr2=0x7f765c0b1d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:03.311 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:03.309 7f76612c1700 1 --2- 172.21.2.102:0/3572286575 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f765c0880d0 0x7f765c0b1d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:05:03.311 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:03.309 7f76612c1700 1 -- 172.21.2.102:0/3572286575 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f765c082520 msgr2=0x7f765c087ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:03.311 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:03.309 7f7660ac0700 1 --2- 172.21.2.102:0/3572286575 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f765c082520 0x7f765c087ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:03.311 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:03.309 7f76612c1700 1 --2- 172.21.2.102:0/3572286575 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f765c082520 0x7f765c087ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:03.311 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:03.309 7f76612c1700 1 -- 172.21.2.102:0/3572286575 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f765c088a70 con 0x7f765c0884d0 2021-02-10T11:05:03.311 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:03.309 7f76612c1700 1 --2- 172.21.2.102:0/3572286575 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f765c0884d0 0x7f765c0b2370 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 rx=0x7f765400b930 tx=0x7f765400bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:05:03.312 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:03.309 7f76597fa700 1 -- 172.21.2.102:0/3572286575 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7654009450 con 0x7f765c0884d0 2021-02-10T11:05:03.312 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:03.309 7f76597fa700 1 -- 172.21.2.102:0/3572286575 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76540069a0 con 0x7f765c0884d0 2021-02-10T11:05:03.312 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:03.309 7f76597fa700 1 -- 172.21.2.102:0/3572286575 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f765400c2f0 con 0x7f765c0884d0 2021-02-10T11:05:03.312 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:03.309 7f7662ca4700 1 -- 172.21.2.102:0/3572286575 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f765c088a70 con 0x7f765c0884d0 2021-02-10T11:05:03.312 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:03.309 7f7662ca4700 1 -- 172.21.2.102:0/3572286575 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f765c088fd0 con 0x7f765c0884d0 2021-02-10T11:05:03.312 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:03.310 7f76597fa700 1 -- 172.21.2.102:0/3572286575 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f7654001270 con 0x7f765c0884d0 2021-02-10T11:05:03.312 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:03.310 7f76597fa700 1 --2- 172.21.2.102:0/3572286575 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f763c0186c0 0x7f763c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:03.312 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:03.310 7f76597fa700 1 -- 172.21.2.102:0/3572286575 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 4839+0+0 (secure 0 0 0) 0x7f7654032610 con 0x7f765c0884d0 2021-02-10T11:05:03.312 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:03.310 7f7660ac0700 1 --2- 172.21.2.102:0/3572286575 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f763c0186c0 0x7f763c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:03.312 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:03.310 7f7660ac0700 1 --2- 172.21.2.102:0/3572286575 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f763c0186c0 0x7f763c01abc0 crc :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:05:03.339 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:03.340 7f7662ca4700 1 -- 172.21.2.102:0/3572286575 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f765c107180 con 0x7f765c0884d0 2021-02-10T11:05:03.342 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:03.343 7f76597fa700 1 -- 172.21.2.102:0/3572286575 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f7654003000 con 0x7f765c0884d0 2021-02-10T11:05:03.480 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:03.481 7f7662ca4700 1 -- 172.21.2.102:0/3572286575 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd blacklist clear"} v 0) v1 -- 0x7f765c07ed80 con 0x7f765c0884d0 2021-02-10T11:05:03.809 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:03.810 7f76597fa700 1 -- 172.21.2.102:0/3572286575 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd blacklist clear"}]=0 removed all blacklist entries v24) v1 ==== 97+0+0 (secure 0 0 0) 0x7f7654001270 con 0x7f765c0884d0 2021-02-10T11:05:03.846 INFO:teuthology.orchestra.run.gibba002.stderr: removed all blacklist entries 2021-02-10T11:05:03.847 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:03.849 7f7662ca4700 1 -- 172.21.2.102:0/3572286575 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f763c0186c0 msgr2=0x7f763c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:03.848 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:03.849 7f7662ca4700 1 --2- 172.21.2.102:0/3572286575 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f763c0186c0 0x7f763c01abc0 crc :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:03.848 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:03.849 7f7662ca4700 1 -- 172.21.2.102:0/3572286575 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f765c0884d0 msgr2=0x7f765c0b2370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:03.848 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:03.849 7f7662ca4700 1 --2- 172.21.2.102:0/3572286575 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f765c0884d0 0x7f765c0b2370 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 rx=0x7f765400b930 tx=0x7f765400bcd0).stop 2021-02-10T11:05:03.848 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:03.849 7f7662ca4700 1 -- 172.21.2.102:0/3572286575 shutdown_connections 2021-02-10T11:05:03.848 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:03.849 7f7662ca4700 1 --2- 172.21.2.102:0/3572286575 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f765c0884d0 0x7f765c0b2370 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:03.848 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:03.849 7f7662ca4700 1 --2- 172.21.2.102:0/3572286575 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f765c0880d0 0x7f765c0b1d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:05:03.848 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:03.849 7f7662ca4700 1 --2- 172.21.2.102:0/3572286575 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f763c0186c0 0x7f763c01abc0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:03.849 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:03.849 7f7662ca4700 1 --2- 172.21.2.102:0/3572286575 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f765c082520 0x7f765c087ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:03.849 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:03.849 7f7662ca4700 1 -- 172.21.2.102:0/3572286575 >> 172.21.2.102:0/3572286575 conn(0x7f765c098740 msgr2=0x7f765c0c3590 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:05:03.849 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:03.850 7f7662ca4700 1 -- 172.21.2.102:0/3572286575 shutdown_connections 2021-02-10T11:05:03.849 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:03.850 7f7662ca4700 1 -- 172.21.2.102:0/3572286575 wait complete. 2021-02-10T11:05:03.856 INFO:tasks.cephfs.cephfs_test_case:['0', '1', '2', '3'] 2021-02-10T11:05:03.856 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth ls --format=json-pretty 2021-02-10T11:05:04.282 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:04.266 7fd1e9851700 1 Processor -- start 2021-02-10T11:05:04.282 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:04.266 7fd1e9851700 1 -- start start 2021-02-10T11:05:04.282 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:04.267 7fd1e9851700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fd1e40e5070 0x7fd1e41b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:04.282 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:04.267 7fd1e9851700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd1e41b3e40 0x7fd1e407d690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:04.282 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:04.267 7fd1e9851700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1e407ed80 con 0x7fd1e40e5470 2021-02-10T11:05:04.282 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:04.267 7fd1e9851700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1e4080310 con 0x7fd1e40e5070 2021-02-10T11:05:04.282 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:04.267 7fd1e9851700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1e4080500 con 0x7fd1e41b3e40 2021-02-10T11:05:04.282 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:04.267 7fd1e3574700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fd1e40e5070 0x7fd1e41b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:04.283 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:04.267 7fd1e3574700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fd1e40e5070 0x7fd1e41b3820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:50356/0 (socket says 172.21.2.102:50356) 2021-02-10T11:05:04.283 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:04.267 7fd1e3574700 1 -- 172.21.2.102:0/2948627051 learned_addr learned my addr 172.21.2.102:0/2948627051 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:05:04.283 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:04.267 7fd1e3d75700 1 --1- 172.21.2.102:0/2948627051 >> v1:172.21.2.102:6789/0 conn(0x7fd1e40e5470 0x7fd1e407dcb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:42580/0 (socket says 172.21.2.102:42580) 2021-02-10T11:05:04.283 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:04.267 7fd1e2d73700 1 --2- 172.21.2.102:0/2948627051 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd1e41b3e40 0x7fd1e407d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:04.283 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:04.268 7fd1e3574700 1 -- 172.21.2.102:0/2948627051 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd1e41b3e40 msgr2=0x7fd1e407d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:05:04.283 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:04.268 7fd1e3574700 1 --2- 172.21.2.102:0/2948627051 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd1e41b3e40 0x7fd1e407d690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:04.283 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:04.268 7fd1e3574700 1 -- 172.21.2.102:0/2948627051 >> v1:172.21.2.102:6789/0 conn(0x7fd1e40e5470 legacy=0x7fd1e407dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:04.283 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:04.268 7fd1e3574700 1 -- 172.21.2.102:0/2948627051 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1e40806f0 con 0x7fd1e40e5070 2021-02-10T11:05:04.283 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:04.268 7fd1e2d73700 1 --2- 172.21.2.102:0/2948627051 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd1e41b3e40 0x7fd1e407d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:05:04.283 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:04.268 7fd1e3574700 1 --2- 172.21.2.102:0/2948627051 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd1e40e5070 0x7fd1e41b3820 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 rx=0x7fd1d4006700 tx=0x7fd1d4009b80).ready entity=mon.0 client_cookie=a23df6419aef670b server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:05:04.283 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:04.268 7fd1e2572700 1 -- 172.21.2.102:0/2948627051 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd1d400c0c0 con 0x7fd1e40e5070 2021-02-10T11:05:04.283 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:04.268 7fd1e2572700 1 -- 172.21.2.102:0/2948627051 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1d400c340 con 0x7fd1e40e5070 2021-02-10T11:05:04.283 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:04.268 7fd1e2572700 1 -- 172.21.2.102:0/2948627051 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd1d400c860 con 0x7fd1e40e5070 2021-02-10T11:05:04.284 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:04.268 7fd1e9851700 1 -- 172.21.2.102:0/2948627051 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd1e40e5070 msgr2=0x7fd1e41b3820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:04.284 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:04.268 7fd1e9851700 1 --2- 172.21.2.102:0/2948627051 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd1e40e5070 0x7fd1e41b3820 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 rx=0x7fd1d4006700 tx=0x7fd1d4009b80).stop 2021-02-10T11:05:04.284 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:04.269 7fd1e9851700 1 -- 172.21.2.102:0/2948627051 shutdown_connections 2021-02-10T11:05:04.284 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:04.269 7fd1e9851700 1 --2- 172.21.2.102:0/2948627051 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd1e41b3e40 0x7fd1e407d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:04.284 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:04.269 7fd1e9851700 1 --2- 172.21.2.102:0/2948627051 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd1e40e5070 0x7fd1e41b3820 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:04.284 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:04.269 7fd1e9851700 1 -- 172.21.2.102:0/2948627051 >> 172.21.2.102:0/2948627051 conn(0x7fd1e40ce690 msgr2=0x7fd1e40e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:05:04.284 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:04.269 7fd1e9851700 1 -- 172.21.2.102:0/2948627051 shutdown_connections 2021-02-10T11:05:04.284 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:04.269 7fd1e9851700 1 -- 172.21.2.102:0/2948627051 wait complete. 2021-02-10T11:05:04.286 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:04.284 7fd1e9851700 1 Processor -- start 2021-02-10T11:05:04.286 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:04.284 7fd1e9851700 1 -- start start 2021-02-10T11:05:04.286 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:04.284 7fd1e9851700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd1e41900e0 0x7fd1e40b0c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:04.286 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:04.284 7fd1e9851700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd1e418f410 0x7fd1e418f810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:04.286 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:04.284 7fd1e9851700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd1e41ada10 0x7fd1e41ade10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:04.286 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:04.284 7fd1e9851700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1e408e100 con 0x7fd1e41ada10 2021-02-10T11:05:04.286 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:04.284 7fd1e9851700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1e418fe30 con 0x7fd1e418f410 2021-02-10T11:05:04.286 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:04.284 7fd1e9851700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1e41ad090 con 0x7fd1e41900e0 2021-02-10T11:05:04.286 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:04.284 7fd1e3d75700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd1e41ada10 0x7fd1e41ade10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:04.287 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:04.284 7fd1e3d75700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd1e41ada10 0x7fd1e41ade10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:50362/0 (socket says 172.21.2.102:50362) 2021-02-10T11:05:04.287 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:04.284 7fd1e3d75700 1 -- 172.21.2.102:0/3419195826 learned_addr learned my addr 172.21.2.102:0/3419195826 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:05:04.287 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:04.285 7fd1e2d73700 1 --2- 172.21.2.102:0/3419195826 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd1e418f410 0x7fd1e418f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:04.287 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:04.285 7fd1e3574700 1 --2- 172.21.2.102:0/3419195826 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd1e41900e0 0x7fd1e40b0c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:04.287 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:04.285 7fd1e3d75700 1 -- 172.21.2.102:0/3419195826 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd1e41900e0 msgr2=0x7fd1e40b0c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:04.287 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:04.285 7fd1e3d75700 1 --2- 172.21.2.102:0/3419195826 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd1e41900e0 0x7fd1e40b0c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:04.287 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:04.285 7fd1e3d75700 1 -- 172.21.2.102:0/3419195826 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd1e418f410 msgr2=0x7fd1e418f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:04.287 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:04.285 7fd1e3d75700 1 --2- 172.21.2.102:0/3419195826 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd1e418f410 0x7fd1e418f810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:04.287 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:04.285 7fd1e3d75700 1 -- 172.21.2.102:0/3419195826 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1e40b1240 con 0x7fd1e41ada10 2021-02-10T11:05:04.287 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:04.285 7fd1e3574700 1 --2- 172.21.2.102:0/3419195826 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd1e41900e0 0x7fd1e40b0c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:05:04.287 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:04.285 7fd1e3d75700 1 --2- 172.21.2.102:0/3419195826 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd1e41ada10 0x7fd1e41ade10 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 rx=0x7fd1dc007e80 tx=0x7fd1dc00aca0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:05:04.287 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:04.285 7fd1cbfff700 1 -- 172.21.2.102:0/3419195826 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd1dc001480 con 0x7fd1e41ada10 2021-02-10T11:05:04.287 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:04.285 7fd1cbfff700 1 -- 172.21.2.102:0/3419195826 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1dc0070b0 con 0x7fd1e41ada10 2021-02-10T11:05:04.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:04.285 7fd1cbfff700 1 -- 172.21.2.102:0/3419195826 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd1dc007540 con 0x7fd1e41ada10 2021-02-10T11:05:04.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:04.285 7fd1e9851700 1 -- 172.21.2.102:0/3419195826 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1e40b1240 con 0x7fd1e41ada10 2021-02-10T11:05:04.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:04.285 7fd1e9851700 1 -- 172.21.2.102:0/3419195826 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1e40b1240 con 0x7fd1e41ada10 2021-02-10T11:05:04.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:04.286 7fd1cbfff700 1 -- 172.21.2.102:0/3419195826 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd1dc007290 con 0x7fd1e41ada10 2021-02-10T11:05:04.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:04.286 7fd1cbfff700 1 --2- 172.21.2.102:0/3419195826 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd1c40186c0 0x7fd1c401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:04.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:04.286 7fd1cbfff700 1 -- 172.21.2.102:0/3419195826 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 4753+0+0 (secure 0 0 0) 0x7fd1dc032420 con 0x7fd1e41ada10 2021-02-10T11:05:04.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:04.286 7fd1e3574700 1 --2- 172.21.2.102:0/3419195826 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd1c40186c0 0x7fd1c401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:04.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:04.286 7fd1e3574700 1 --2- 172.21.2.102:0/3419195826 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd1c40186c0 0x7fd1c401abc0 crc :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:05:04.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:04.317 7fd1e9851700 1 -- 172.21.2.102:0/3419195826 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1e418fe30 con 0x7fd1e41ada10 2021-02-10T11:05:04.318 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:04.319 7fd1cbfff700 1 -- 172.21.2.102:0/3419195826 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd1dc0074a0 con 0x7fd1e41ada10 2021-02-10T11:05:04.452 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:04.453 7fd1e9851700 1 -- 172.21.2.102:0/3419195826 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fd1e418fe30 con 0x7fd1e41ada10 2021-02-10T11:05:04.452 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:04.453 7fd1cbfff700 1 -- 172.21.2.102:0/3419195826 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v4) v1 ==== 80+0+5917 (secure 0 0 0) 0x7fd1dc007270 con 0x7fd1e41ada10 2021-02-10T11:05:04.453 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:05:04.453 INFO:teuthology.orchestra.run.gibba002.stdout:{ 2021-02-10T11:05:04.453 INFO:teuthology.orchestra.run.gibba002.stdout: "auth_dump": [ 2021-02-10T11:05:04.453 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:05:04.453 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "mds.a", 2021-02-10T11:05:04.453 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAevSNgsgyEARAAvr0cUjjCoEHLxaZSZ2HtVA==", 2021-02-10T11:05:04.453 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:05:04.453 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow", 2021-02-10T11:05:04.453 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:05:04.453 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:05:04.453 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:05:04.454 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:05:04.454 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:05:04.454 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:05:04.454 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "mds.b", 2021-02-10T11:05:04.454 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAevSNgwOFMMRAALfb1fbX56ao1TU8ClnTW1A==", 2021-02-10T11:05:04.454 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:05:04.454 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow", 2021-02-10T11:05:04.454 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:05:04.454 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:05:04.454 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:05:04.454 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:05:04.454 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:05:04.454 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:05:04.455 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.0", 2021-02-10T11:05:04.455 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAmvSNgj+bMMBAAD3HUD6fLiMAg+++iNzLJKQ==", 2021-02-10T11:05:04.455 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:05:04.455 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:05:04.455 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:05:04.455 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:05:04.455 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:05:04.455 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:05:04.455 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:05:04.455 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.1", 2021-02-10T11:05:04.455 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQApvSNgnvc+MRAAAOxqlNwiyLHqNITOrYeFKQ==", 2021-02-10T11:05:04.455 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:05:04.455 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:05:04.456 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:05:04.456 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:05:04.456 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:05:04.456 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:05:04.456 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:05:04.456 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.2", 2021-02-10T11:05:04.456 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAsvSNgZ4SbMRAA76IOYWfbiAuBoM/4k81GxA==", 2021-02-10T11:05:04.456 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:05:04.456 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:05:04.456 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:05:04.456 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:05:04.456 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:05:04.456 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:05:04.457 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:05:04.457 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.3", 2021-02-10T11:05:04.457 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAvvSNgFk3rMRAAWCmKKyw0LDQ+UlAHZAITOw==", 2021-02-10T11:05:04.457 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:05:04.457 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:05:04.457 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:05:04.457 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:05:04.457 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:05:04.457 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:05:04.457 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:05:04.457 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.4", 2021-02-10T11:05:04.457 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQA4vSNgpKcNJBAApJMmvj5YxVHPDOmvmQONOg==", 2021-02-10T11:05:04.457 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:05:04.457 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:05:04.458 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:05:04.458 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:05:04.458 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:05:04.458 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:05:04.458 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:05:04.458 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.5", 2021-02-10T11:05:04.458 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQA7vSNgf2ujJBAAfkuYpyb3WTzc8V5kw+Wktg==", 2021-02-10T11:05:04.458 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:05:04.458 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:05:04.458 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:05:04.458 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:05:04.458 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:05:04.458 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:05:04.459 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:05:04.459 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.6", 2021-02-10T11:05:04.459 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQA+vSNgqw9uJRAAFMU0R+UxU9C3o/9byb3e0g==", 2021-02-10T11:05:04.459 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:05:04.459 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:05:04.459 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:05:04.459 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:05:04.459 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:05:04.459 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:05:04.459 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:05:04.459 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.7", 2021-02-10T11:05:04.459 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQBBvSNgvWSrJRAA2KaCHmLpzovRgYzLvyofTQ==", 2021-02-10T11:05:04.459 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:05:04.459 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:05:04.460 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:05:04.460 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:05:04.460 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:05:04.460 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:05:04.460 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:05:04.460 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "client.0", 2021-02-10T11:05:04.460 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAfvSNgZd9XIBAAvyrvl8fdKuHnJZfczfRoFw==", 2021-02-10T11:05:04.460 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:05:04.460 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow", 2021-02-10T11:05:04.460 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow r", 2021-02-10T11:05:04.460 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow rw", 2021-02-10T11:05:04.460 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow rwx" 2021-02-10T11:05:04.460 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:05:04.460 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:05:04.461 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:05:04.461 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "client.1", 2021-02-10T11:05:04.461 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAfvSNgSyfXNhAAIduPnF7Vd/6HSkJ0MlcMSA==", 2021-02-10T11:05:04.461 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:05:04.461 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow", 2021-02-10T11:05:04.461 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow r", 2021-02-10T11:05:04.461 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow rw", 2021-02-10T11:05:04.461 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow rwx" 2021-02-10T11:05:04.461 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:05:04.461 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:05:04.461 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:05:04.461 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "client.2", 2021-02-10T11:05:04.461 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAgvSNgiIG/ERAA9ZTamaTI7ZsPSnxBqjplCg==", 2021-02-10T11:05:04.462 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:05:04.462 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow", 2021-02-10T11:05:04.462 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow r", 2021-02-10T11:05:04.462 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow rw", 2021-02-10T11:05:04.462 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow rwx" 2021-02-10T11:05:04.462 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:05:04.462 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:05:04.462 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:05:04.462 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "client.3", 2021-02-10T11:05:04.462 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAgvSNgOWpOKBAAlXsEB04Q/GvtL06PqRMLeA==", 2021-02-10T11:05:04.462 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:05:04.462 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow", 2021-02-10T11:05:04.462 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow r", 2021-02-10T11:05:04.463 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow rw", 2021-02-10T11:05:04.463 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow rwx" 2021-02-10T11:05:04.463 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:05:04.463 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:05:04.463 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:05:04.463 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "client.admin", 2021-02-10T11:05:04.463 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAavSNgO3GQBhAAL+NVnYskQRKVWLgyDPY/xw==", 2021-02-10T11:05:04.463 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:05:04.463 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow *", 2021-02-10T11:05:04.463 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:05:04.463 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:05:04.463 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:05:04.463 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:05:04.464 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:05:04.464 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:05:04.464 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "client.bootstrap-mds", 2021-02-10T11:05:04.464 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQBqvSNgmDrIExAAk51cXOdsF0CkXjmsfzwueg==", 2021-02-10T11:05:04.464 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:05:04.464 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow profile bootstrap-mds" 2021-02-10T11:05:04.464 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:05:04.464 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:05:04.464 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:05:04.464 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "client.bootstrap-mgr", 2021-02-10T11:05:04.464 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQBqvSNgy0PIExAAHXnwCYNUXFYASPOAlDhoCg==", 2021-02-10T11:05:04.464 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:05:04.464 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow profile bootstrap-mgr" 2021-02-10T11:05:04.464 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:05:04.465 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:05:04.465 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:05:04.465 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "client.bootstrap-osd", 2021-02-10T11:05:04.465 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQBqvSNgkkzIExAAcYQzxPPGp3ohqLscYSQ1Fg==", 2021-02-10T11:05:04.465 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:05:04.465 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow profile bootstrap-osd" 2021-02-10T11:05:04.465 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:05:04.465 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:05:04.465 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:05:04.465 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "client.bootstrap-rbd", 2021-02-10T11:05:04.465 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQBqvSNgXlXIExAAyla09zy4TM4iR4OINFuoNg==", 2021-02-10T11:05:04.465 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:05:04.465 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow profile bootstrap-rbd" 2021-02-10T11:05:04.465 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:05:04.466 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:05:04.466 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:05:04.466 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "client.bootstrap-rbd-mirror", 2021-02-10T11:05:04.466 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQBqvSNgVl7IExAAdbAcTuhAZaeFOwD2dBNpJg==", 2021-02-10T11:05:04.466 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:05:04.466 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow profile bootstrap-rbd-mirror" 2021-02-10T11:05:04.466 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:05:04.466 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:05:04.466 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:05:04.466 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "client.bootstrap-rgw", 2021-02-10T11:05:04.466 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQBqvSNgW2fIExAAXF2TTXvjhw5/GuSjXuNczA==", 2021-02-10T11:05:04.466 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:05:04.466 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow profile bootstrap-rgw" 2021-02-10T11:05:04.467 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:05:04.467 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:05:04.467 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:05:04.467 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "mgr.x", 2021-02-10T11:05:04.467 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAdvSNgXZZgJhAA7dtBSGzH+DsBwnddZpOBJA==", 2021-02-10T11:05:04.467 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:05:04.467 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow *", 2021-02-10T11:05:04.467 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow profile mgr", 2021-02-10T11:05:04.467 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:05:04.467 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:05:04.467 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:05:04.467 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:05:04.467 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "mgr.y", 2021-02-10T11:05:04.467 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAdvSNgVqQFDRAAwo6YEdWZCAr5GdOWVL/Qvg==", 2021-02-10T11:05:04.468 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:05:04.468 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow *", 2021-02-10T11:05:04.468 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow profile mgr", 2021-02-10T11:05:04.468 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:05:04.468 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:05:04.468 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:05:04.468 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:05:04.468 INFO:teuthology.orchestra.run.gibba002.stdout:} 2021-02-10T11:05:04.469 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:04.455 7fd1e9851700 1 -- 172.21.2.102:0/3419195826 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd1c40186c0 msgr2=0x7fd1c401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:04.469 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:04.455 7fd1e9851700 1 --2- 172.21.2.102:0/3419195826 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd1c40186c0 0x7fd1c401abc0 crc :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:04.469 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:04.455 7fd1e9851700 1 -- 172.21.2.102:0/3419195826 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd1e41ada10 msgr2=0x7fd1e41ade10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:04.469 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:04.455 7fd1e9851700 1 --2- 172.21.2.102:0/3419195826 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd1e41ada10 0x7fd1e41ade10 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 rx=0x7fd1dc007e80 tx=0x7fd1dc00aca0).stop 2021-02-10T11:05:04.469 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:04.455 7fd1e9851700 1 -- 172.21.2.102:0/3419195826 shutdown_connections 2021-02-10T11:05:04.470 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:04.455 7fd1e9851700 1 --2- 172.21.2.102:0/3419195826 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd1e41ada10 0x7fd1e41ade10 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:04.470 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:04.455 7fd1e9851700 1 --2- 172.21.2.102:0/3419195826 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd1c40186c0 0x7fd1c401abc0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:04.470 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:04.455 7fd1e9851700 1 --2- 172.21.2.102:0/3419195826 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd1e418f410 0x7fd1e418f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:04.470 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:04.455 7fd1e9851700 1 --2- 172.21.2.102:0/3419195826 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd1e41900e0 0x7fd1e40b0c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:04.470 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:04.455 7fd1e9851700 1 -- 172.21.2.102:0/3419195826 >> 172.21.2.102:0/3419195826 conn(0x7fd1e41a31c0 msgr2=0x7fd1e408fa80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:05:04.470 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:04.456 7fd1e9851700 1 -- 172.21.2.102:0/3419195826 shutdown_connections 2021-02-10T11:05:04.470 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:04.456 7fd1e9851700 1 -- 172.21.2.102:0/3419195826 wait complete. 2021-02-10T11:05:04.470 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth del client.bootstrap-mds 2021-02-10T11:05:04.895 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:04.880 7f9412949700 1 Processor -- start 2021-02-10T11:05:04.895 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:04.880 7f9412949700 1 -- start start 2021-02-10T11:05:04.895 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:04.881 7f9412949700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f940c0e5070 0x7f940c1b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:04.895 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:04.881 7f9412949700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f940c07ed80 con 0x7f940c1b3e40 2021-02-10T11:05:04.895 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:04.881 7f9412949700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f940c07e670 con 0x7f940c0e5470 2021-02-10T11:05:04.895 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:04.881 7f9412949700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f940c07e890 con 0x7f940c0e5070 2021-02-10T11:05:04.895 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:04.881 7f940b7fe700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f940c1b3e40 0x7f940c07d690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:42596/0 (socket says 172.21.2.102:42596) 2021-02-10T11:05:04.895 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:04.881 7f940b7fe700 1 -- 172.21.2.102:0/1115971851 learned_addr learned my addr 172.21.2.102:0/1115971851 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:05:04.896 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:04.881 7f940bfff700 1 --2- 172.21.2.102:0/1115971851 >> v2:172.21.2.113:3300/0 conn(0x7f940c0e5070 0x7f940c1b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:04.896 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:04.881 7f940affd700 1 -- 172.21.2.102:0/1115971851 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2391323788 0 0) 0x7f94000010b0 con 0x7f940c1b3e40 2021-02-10T11:05:04.896 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:04.881 7f940bfff700 1 -- 172.21.2.102:0/1115971851 >> v1:172.21.2.113:6789/0 conn(0x7f940c0e5470 legacy=0x7f940c07de80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:04.896 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:04.881 7f940bfff700 1 -- 172.21.2.102:0/1115971851 >> v1:172.21.2.102:6789/0 conn(0x7f940c1b3e40 legacy=0x7f940c07d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:04.896 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:04.881 7f940bfff700 1 -- 172.21.2.102:0/1115971851 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f940c07eab0 con 0x7f940c0e5070 2021-02-10T11:05:04.896 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:04.881 7f940bfff700 1 --2- 172.21.2.102:0/1115971851 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f940c0e5070 0x7f940c1b3820 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f93fc006650 tx=0x7f93fc009a00).ready entity=mon.1 client_cookie=9c8b829ea42d8b15 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:05:04.896 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:04.882 7f940affd700 1 -- 172.21.2.102:0/1115971851 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f93fc0009b0 con 0x7f940c0e5070 2021-02-10T11:05:04.896 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:04.882 7f940affd700 1 -- 172.21.2.102:0/1115971851 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93fc00c0c0 con 0x7f940c0e5070 2021-02-10T11:05:04.896 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:04.882 7f940affd700 1 -- 172.21.2.102:0/1115971851 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f93fc00c5a0 con 0x7f940c0e5070 2021-02-10T11:05:04.896 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:04.882 7f9412949700 1 -- 172.21.2.102:0/1115971851 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f940c0e5070 msgr2=0x7f940c1b3820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:04.896 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:04.882 7f9412949700 1 --2- 172.21.2.102:0/1115971851 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f940c0e5070 0x7f940c1b3820 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f93fc006650 tx=0x7f93fc009a00).stop 2021-02-10T11:05:04.896 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:04.882 7f9412949700 1 -- 172.21.2.102:0/1115971851 shutdown_connections 2021-02-10T11:05:04.897 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:04.882 7f9412949700 1 --2- 172.21.2.102:0/1115971851 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f940c0e5070 0x7f940c1b3820 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:04.897 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:04.882 7f9412949700 1 -- 172.21.2.102:0/1115971851 >> 172.21.2.102:0/1115971851 conn(0x7f940c0ce690 msgr2=0x7f940c0e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:05:04.897 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:04.882 7f9412949700 1 -- 172.21.2.102:0/1115971851 shutdown_connections 2021-02-10T11:05:04.897 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:04.882 7f9412949700 1 -- 172.21.2.102:0/1115971851 wait complete. 2021-02-10T11:05:04.899 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:04.897 7f9412949700 1 Processor -- start 2021-02-10T11:05:04.899 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:04.897 7f9412949700 1 -- start start 2021-02-10T11:05:04.899 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:04.897 7f9412949700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f940c1952f0 0x7f940c0be2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:04.899 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:04.897 7f9412949700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f940c1a35e0 0x7f940c0be900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:04.899 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:04.897 7f9412949700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f940c0bef20 0x7f940c0c1a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:04.899 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:04.897 7f9412949700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f940c0bf4b0 con 0x7f940c1a35e0 2021-02-10T11:05:04.899 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:04.897 7f9412949700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f940c0bf6d0 con 0x7f940c0bef20 2021-02-10T11:05:04.899 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:04.897 7f9412949700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f940c0c2080 con 0x7f940c1952f0 2021-02-10T11:05:04.899 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:04.897 7f940b7fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f940c1a35e0 0x7f940c0be900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:04.900 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:04.897 7f940b7fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f940c1a35e0 0x7f940c0be900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:50380/0 (socket says 172.21.2.102:50380) 2021-02-10T11:05:04.900 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:04.897 7f940b7fe700 1 -- 172.21.2.102:0/3778159394 learned_addr learned my addr 172.21.2.102:0/3778159394 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:05:04.900 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:04.897 7f940bfff700 1 --2- 172.21.2.102:0/3778159394 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f940c1952f0 0x7f940c0be2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:04.900 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:04.897 7f9410f66700 1 --2- 172.21.2.102:0/3778159394 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f940c0bef20 0x7f940c0c1a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:04.900 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:04.898 7f940bfff700 1 -- 172.21.2.102:0/3778159394 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f940c0bef20 msgr2=0x7f940c0c1a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:04.900 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:04.898 7f940bfff700 1 --2- 172.21.2.102:0/3778159394 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f940c0bef20 0x7f940c0c1a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:04.900 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:04.898 7f940bfff700 1 -- 172.21.2.102:0/3778159394 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f940c1a35e0 msgr2=0x7f940c0be900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:04.900 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:04.898 7f940bfff700 1 --2- 172.21.2.102:0/3778159394 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f940c1a35e0 0x7f940c0be900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:04.900 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:04.898 7f940bfff700 1 -- 172.21.2.102:0/3778159394 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f940c0c2260 con 0x7f940c1952f0 2021-02-10T11:05:04.900 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:04.898 7f940b7fe700 1 --2- 172.21.2.102:0/3778159394 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f940c1a35e0 0x7f940c0be900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:05:04.900 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:04.898 7f940bfff700 1 --2- 172.21.2.102:0/3778159394 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f940c1952f0 0x7f940c0be2e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f93fc006f10 tx=0x7f93fc00b8c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:05:04.900 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:04.898 7f9408ff9700 1 -- 172.21.2.102:0/3778159394 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f93fc006390 con 0x7f940c1952f0 2021-02-10T11:05:04.901 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:04.898 7f9408ff9700 1 -- 172.21.2.102:0/3778159394 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93fc002030 con 0x7f940c1952f0 2021-02-10T11:05:04.901 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:04.898 7f9408ff9700 1 -- 172.21.2.102:0/3778159394 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f93fc0024d0 con 0x7f940c1952f0 2021-02-10T11:05:04.901 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:04.898 7f9412949700 1 -- 172.21.2.102:0/3778159394 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f940c0c2260 con 0x7f940c1952f0 2021-02-10T11:05:04.901 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:04.898 7f9412949700 1 -- 172.21.2.102:0/3778159394 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f940c0c2260 con 0x7f940c1952f0 2021-02-10T11:05:04.901 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:04.899 7f9408ff9700 1 -- 172.21.2.102:0/3778159394 <== mon.2 v2:172.21.2.113:3301/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f93fc000c90 con 0x7f940c1952f0 2021-02-10T11:05:04.901 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:04.899 7f9408ff9700 1 --2- 172.21.2.102:0/3778159394 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f93ec018730 0x7f93ec01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:04.901 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:04.899 7f9408ff9700 1 -- 172.21.2.102:0/3778159394 <== mon.2 v2:172.21.2.113:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 4753+0+0 (secure 0 0 0) 0x7f93fc0323c0 con 0x7f940c1952f0 2021-02-10T11:05:04.901 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:04.899 7f940b7fe700 1 --2- 172.21.2.102:0/3778159394 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f93ec018730 0x7f93ec01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:04.901 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:04.899 7f940b7fe700 1 --2- 172.21.2.102:0/3778159394 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f93ec018730 0x7f93ec01ac10 crc :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:05:04.928 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:04.930 7f9412949700 1 -- 172.21.2.102:0/3778159394 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f940c083e60 con 0x7f940c1952f0 2021-02-10T11:05:04.932 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:04.934 7f9408ff9700 1 -- 172.21.2.102:0/3778159394 <== mon.2 v2:172.21.2.113:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f93fc009020 con 0x7f940c1952f0 2021-02-10T11:05:05.064 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:05.065 7f9412949700 1 -- 172.21.2.102:0/3778159394 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mds"} v 0) v1 -- 0x7f940c083e60 con 0x7f940c1952f0 2021-02-10T11:05:05.069 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:05.070 7f9408ff9700 1 -- 172.21.2.102:0/3778159394 <== mon.2 v2:172.21.2.113:3301/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mds"}]=0 updated v5) v1 ==== 97+0+0 (secure 0 0 0) 0x7f93fc000c90 con 0x7f940c1952f0 2021-02-10T11:05:05.071 INFO:teuthology.orchestra.run.gibba002.stderr:updated 2021-02-10T11:05:05.072 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:05.074 7f9412949700 1 -- 172.21.2.102:0/3778159394 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f93ec018730 msgr2=0x7f93ec01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:05.072 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:05.074 7f9412949700 1 --2- 172.21.2.102:0/3778159394 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f93ec018730 0x7f93ec01ac10 crc :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:05.072 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:05.074 7f9412949700 1 -- 172.21.2.102:0/3778159394 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f940c1952f0 msgr2=0x7f940c0be2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:05.072 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:05.074 7f9412949700 1 --2- 172.21.2.102:0/3778159394 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f940c1952f0 0x7f940c0be2e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f93fc006f10 tx=0x7f93fc00b8c0).stop 2021-02-10T11:05:05.073 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:05.074 7f9412949700 1 -- 172.21.2.102:0/3778159394 shutdown_connections 2021-02-10T11:05:05.073 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:05.074 7f9412949700 1 --2- 172.21.2.102:0/3778159394 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f93ec018730 0x7f93ec01ac10 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:05.073 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:05.074 7f9412949700 1 --2- 172.21.2.102:0/3778159394 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f940c0bef20 0x7f940c0c1a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:05.073 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:05.074 7f9412949700 1 --2- 172.21.2.102:0/3778159394 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f940c1a35e0 0x7f940c0be900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:05.073 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:05.074 7f9412949700 1 --2- 172.21.2.102:0/3778159394 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f940c1952f0 0x7f940c0be2e0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:05.073 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:05.074 7f9412949700 1 -- 172.21.2.102:0/3778159394 >> 172.21.2.102:0/3778159394 conn(0x7f940c085a70 msgr2=0x7f940c0a5790 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:05:05.074 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:05.074 7f9412949700 1 -- 172.21.2.102:0/3778159394 shutdown_connections 2021-02-10T11:05:05.074 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:05.074 7f9412949700 1 -- 172.21.2.102:0/3778159394 wait complete. 2021-02-10T11:05:05.081 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth del client.bootstrap-mgr 2021-02-10T11:05:05.507 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:05.491 7f4113d32700 1 Processor -- start 2021-02-10T11:05:05.507 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:05.491 7f4113d32700 1 -- start start 2021-02-10T11:05:05.507 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:05.491 7f4113d32700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f410c0e5070 0x7f410c1b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:05.507 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:05.491 7f4113d32700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f410c1b3e40 0x7f410c07d690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:05.507 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:05.491 7f4113d32700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f410c07ed80 con 0x7f410c0e5470 2021-02-10T11:05:05.507 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:05.491 7f4113d32700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f410c080310 con 0x7f410c0e5070 2021-02-10T11:05:05.507 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:05.491 7f4113d32700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f410c080500 con 0x7f410c1b3e40 2021-02-10T11:05:05.507 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:05.491 7f411234f700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f410c0e5470 0x7f410c07dcb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:42608/0 (socket says 172.21.2.102:42608) 2021-02-10T11:05:05.507 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:05.491 7f411234f700 1 -- 172.21.2.102:0/4169128572 learned_addr learned my addr 172.21.2.102:0/4169128572 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:05:05.508 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:05.492 7f4110b4c700 1 -- 172.21.2.102:0/4169128572 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 945007409 0 0) 0x7f4104001500 con 0x7f410c0e5470 2021-02-10T11:05:05.508 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:05.492 7f4110b4c700 1 -- 172.21.2.102:0/4169128572 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40f8002fe0 con 0x7f410c0e5470 2021-02-10T11:05:05.508 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:05.492 7f4111b4e700 1 --2- 172.21.2.102:0/4169128572 >> v2:172.21.2.113:3300/0 conn(0x7f410c0e5070 0x7f410c1b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:05.508 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:05.492 7f411134d700 1 --2- 172.21.2.102:0/4169128572 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f410c1b3e40 0x7f410c07d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:05.508 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:05.492 7f4110b4c700 1 -- 172.21.2.102:0/4169128572 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 855427691 0 0) 0x7f4104001460 con 0x7f410c0e5470 2021-02-10T11:05:05.508 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:05.492 7f4110b4c700 1 -- 172.21.2.102:0/4169128572 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f410c1b3e40 msgr2=0x7f410c07d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:05:05.508 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:05.492 7f4110b4c700 1 --2- 172.21.2.102:0/4169128572 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f410c1b3e40 0x7f410c07d690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:05.508 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:05.492 7f4110b4c700 1 -- 172.21.2.102:0/4169128572 >> v2:172.21.2.113:3300/0 conn(0x7f410c0e5070 msgr2=0x7f410c1b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:05:05.508 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:05.492 7f4110b4c700 1 --2- 172.21.2.102:0/4169128572 >> v2:172.21.2.113:3300/0 conn(0x7f410c0e5070 0x7f410c1b3820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:05.508 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:05.492 7f4110b4c700 1 -- 172.21.2.102:0/4169128572 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f410c0806f0 con 0x7f410c0e5470 2021-02-10T11:05:05.508 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:05.492 7f4110b4c700 1 -- 172.21.2.102:0/4169128572 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7f4104001980 con 0x7f410c0e5470 2021-02-10T11:05:05.508 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:05.492 7f4110b4c700 1 -- 172.21.2.102:0/4169128572 >> v1:172.21.2.102:6789/0 conn(0x7f410c0e5470 legacy=0x7f410c07dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:05.508 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:05.492 7f4110b4c700 1 --2- 172.21.2.102:0/4169128572 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f40f8003b90 0x7f40f8006040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:05.509 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:05.492 7f4110b4c700 1 --2- 172.21.2.102:0/4169128572 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f40f804a7e0 0x7f40f804cca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:05.509 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:05.492 7f4110b4c700 1 --2- 172.21.2.102:0/4169128572 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f40f804d5a0 0x7f40f804fa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:05.509 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:05.492 7f4110b4c700 1 -- 172.21.2.102:0/4169128572 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40f80502e0 con 0x7f40f804a7e0 2021-02-10T11:05:05.509 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:05.492 7f4110b4c700 1 -- 172.21.2.102:0/4169128572 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40f8050500 con 0x7f40f8003b90 2021-02-10T11:05:05.509 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:05.492 7f4110b4c700 1 -- 172.21.2.102:0/4169128572 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40f8050720 con 0x7f40f804d5a0 2021-02-10T11:05:05.509 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:05.492 7f411134d700 1 --2- 172.21.2.102:0/4169128572 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f40f804a7e0 0x7f40f804cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:05.509 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:05.493 7f4111b4e700 1 --2- 172.21.2.102:0/4169128572 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f40f8003b90 0x7f40f8006040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:05.509 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:05.493 7f411134d700 1 -- 172.21.2.102:0/4169128572 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f40f804d5a0 msgr2=0x7f40f804fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:05:05.509 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:05.493 7f411134d700 1 --2- 172.21.2.102:0/4169128572 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f40f804d5a0 0x7f40f804fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:05:05.509 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:05.493 7f411134d700 1 -- 172.21.2.102:0/4169128572 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f40f8003b90 msgr2=0x7f40f8006040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:05:05.509 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:05.493 7f411134d700 1 --2- 172.21.2.102:0/4169128572 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f40f8003b90 0x7f40f8006040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:05.509 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:05.493 7f411134d700 1 -- 172.21.2.102:0/4169128572 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40f80509c0 con 0x7f40f804a7e0 2021-02-10T11:05:05.509 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:05.493 7f411134d700 1 --2- 172.21.2.102:0/4169128572 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f40f804a7e0 0x7f40f804cca0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 rx=0x7f4108006a00 tx=0x7f4108009b00).ready entity=mon.0 client_cookie=7d44485da6c6cde9 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:05:05.510 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:05.493 7f4110b4c700 1 -- 172.21.2.102:0/4169128572 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f410800c0c0 con 0x7f40f804a7e0 2021-02-10T11:05:05.510 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:05.493 7f4110b4c700 1 -- 172.21.2.102:0/4169128572 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f410800c340 con 0x7f40f804a7e0 2021-02-10T11:05:05.510 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:05.493 7f4110b4c700 1 -- 172.21.2.102:0/4169128572 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f410800c860 con 0x7f40f804a7e0 2021-02-10T11:05:05.510 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:05.493 7f4113d32700 1 -- 172.21.2.102:0/4169128572 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f40f804a7e0 msgr2=0x7f40f804cca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:05.510 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:05.493 7f4113d32700 1 --2- 172.21.2.102:0/4169128572 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f40f804a7e0 0x7f40f804cca0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 rx=0x7f4108006a00 tx=0x7f4108009b00).stop 2021-02-10T11:05:05.510 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:05.493 7f411234f700 1 -- 172.21.2.102:0/4169128572 reap_dead start 2021-02-10T11:05:05.510 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:05.493 7f4113d32700 1 -- 172.21.2.102:0/4169128572 shutdown_connections 2021-02-10T11:05:05.510 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:05.493 7f4113d32700 1 -- 172.21.2.102:0/4169128572 >> 172.21.2.102:0/4169128572 conn(0x7f410c0ce690 msgr2=0x7f410c0e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:05:05.510 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:05.494 7f4113d32700 1 -- 172.21.2.102:0/4169128572 shutdown_connections 2021-02-10T11:05:05.510 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:05.494 7f4113d32700 1 -- 172.21.2.102:0/4169128572 wait complete. 2021-02-10T11:05:05.510 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:05.508 7f4113d32700 1 Processor -- start 2021-02-10T11:05:05.510 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:05.508 7f4113d32700 1 -- start start 2021-02-10T11:05:05.513 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:05.509 7f4113d32700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f410c0b4ad0 0x7f410c0a78d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:05.513 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:05.509 7f4113d32700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f410c0a7ef0 0x7f410c1a6d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:05.513 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:05.509 7f4113d32700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f410c0a82f0 0x7f410c1a73b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:05.513 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:05.509 7f4113d32700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f410c0aa3c0 con 0x7f410c0b4ad0 2021-02-10T11:05:05.513 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:05.509 7f4113d32700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f410c0a88c0 con 0x7f410c0a82f0 2021-02-10T11:05:05.513 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:05.509 7f4113d32700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f410c0b4ed0 con 0x7f410c0a7ef0 2021-02-10T11:05:05.513 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:05.509 7f4111b4e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f410c0b4ad0 0x7f410c0a78d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:05.513 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:05.509 7f4111b4e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f410c0b4ad0 0x7f410c0a78d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:50394/0 (socket says 172.21.2.102:50394) 2021-02-10T11:05:05.513 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:05.509 7f4111b4e700 1 -- 172.21.2.102:0/1828184871 learned_addr learned my addr 172.21.2.102:0/1828184871 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:05:05.513 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:05.509 7f411234f700 1 --2- 172.21.2.102:0/1828184871 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f410c0a82f0 0x7f410c1a73b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:05.513 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:05.509 7f411134d700 1 --2- 172.21.2.102:0/1828184871 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f410c0a7ef0 0x7f410c1a6d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:05.513 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:05.509 7f4111b4e700 1 -- 172.21.2.102:0/1828184871 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f410c0a7ef0 msgr2=0x7f410c1a6d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:05.513 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:05.509 7f4111b4e700 1 --2- 172.21.2.102:0/1828184871 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f410c0a7ef0 0x7f410c1a6d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:05.514 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:05.509 7f4111b4e700 1 -- 172.21.2.102:0/1828184871 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f410c0a82f0 msgr2=0x7f410c1a73b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:05.514 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:05.509 7f4111b4e700 1 --2- 172.21.2.102:0/1828184871 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f410c0a82f0 0x7f410c1a73b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:05.514 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:05.509 7f4111b4e700 1 -- 172.21.2.102:0/1828184871 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f410c1a79d0 con 0x7f410c0b4ad0 2021-02-10T11:05:05.514 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:05.509 7f4111b4e700 1 --2- 172.21.2.102:0/1828184871 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f410c0b4ad0 0x7f410c0a78d0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 rx=0x7f4100006f60 tx=0x7f410000b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:05:05.514 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:05.510 7f40fe7fc700 1 -- 172.21.2.102:0/1828184871 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f410000bd60 con 0x7f410c0b4ad0 2021-02-10T11:05:05.514 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:05.510 7f40fe7fc700 1 -- 172.21.2.102:0/1828184871 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4100000c90 con 0x7f410c0b4ad0 2021-02-10T11:05:05.514 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:05.510 7f40fe7fc700 1 -- 172.21.2.102:0/1828184871 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f41000022b0 con 0x7f410c0b4ad0 2021-02-10T11:05:05.514 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:05.510 7f4113d32700 1 -- 172.21.2.102:0/1828184871 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f410c1a79d0 con 0x7f410c0b4ad0 2021-02-10T11:05:05.514 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:05.510 7f4113d32700 1 -- 172.21.2.102:0/1828184871 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f410c1a79d0 con 0x7f410c0b4ad0 2021-02-10T11:05:05.514 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:05.510 7f40fe7fc700 1 -- 172.21.2.102:0/1828184871 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f410000a000 con 0x7f410c0b4ad0 2021-02-10T11:05:05.514 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:05.510 7f40fe7fc700 1 --2- 172.21.2.102:0/1828184871 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f40f4018730 0x7f40f401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:05.514 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:05.510 7f411134d700 1 --2- 172.21.2.102:0/1828184871 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f40f4018730 0x7f40f401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:05.514 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:05.510 7f40fe7fc700 1 -- 172.21.2.102:0/1828184871 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 4753+0+0 (secure 0 0 0) 0x7f4100000940 con 0x7f410c0b4ad0 2021-02-10T11:05:05.515 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:05.511 7f411134d700 1 --2- 172.21.2.102:0/1828184871 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f40f4018730 0x7f40f401ac10 crc :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:05:05.540 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:05.541 7f4113d32700 1 -- 172.21.2.102:0/1828184871 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f410c084a90 con 0x7f410c0b4ad0 2021-02-10T11:05:05.542 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:05.544 7f40fe7fc700 1 -- 172.21.2.102:0/1828184871 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f4100000940 con 0x7f410c0b4ad0 2021-02-10T11:05:05.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:05.674 7f4113d32700 1 -- 172.21.2.102:0/1828184871 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mgr"} v 0) v1 -- 0x7f410c07ed80 con 0x7f410c0b4ad0 2021-02-10T11:05:05.677 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:05.678 7f40fe7fc700 1 -- 172.21.2.102:0/1828184871 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mgr"}]=0 updated v6) v1 ==== 97+0+0 (secure 0 0 0) 0x7f410000a000 con 0x7f410c0b4ad0 2021-02-10T11:05:05.680 INFO:teuthology.orchestra.run.gibba002.stderr:updated 2021-02-10T11:05:05.682 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:05.683 7f4113d32700 1 -- 172.21.2.102:0/1828184871 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f40f4018730 msgr2=0x7f40f401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:05.682 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:05.683 7f4113d32700 1 --2- 172.21.2.102:0/1828184871 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f40f4018730 0x7f40f401ac10 crc :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:05.683 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:05.683 7f4113d32700 1 -- 172.21.2.102:0/1828184871 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f410c0b4ad0 msgr2=0x7f410c0a78d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:05.683 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:05.683 7f4113d32700 1 --2- 172.21.2.102:0/1828184871 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f410c0b4ad0 0x7f410c0a78d0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 rx=0x7f4100006f60 tx=0x7f410000b970).stop 2021-02-10T11:05:05.683 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:05.683 7f4113d32700 1 -- 172.21.2.102:0/1828184871 shutdown_connections 2021-02-10T11:05:05.683 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:05.683 7f4113d32700 1 --2- 172.21.2.102:0/1828184871 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f40f4018730 0x7f40f401ac10 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:05.683 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:05.683 7f4113d32700 1 --2- 172.21.2.102:0/1828184871 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f410c0a82f0 0x7f410c1a73b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:05.683 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:05.683 7f4113d32700 1 --2- 172.21.2.102:0/1828184871 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f410c0a7ef0 0x7f410c1a6d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:05.684 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:05.683 7f4113d32700 1 --2- 172.21.2.102:0/1828184871 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f410c0b4ad0 0x7f410c0a78d0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:05.684 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:05.683 7f4113d32700 1 -- 172.21.2.102:0/1828184871 >> 172.21.2.102:0/1828184871 conn(0x7f410c0b5560 msgr2=0x7f410c0a6200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:05:05.684 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:05.683 7f4113d32700 1 -- 172.21.2.102:0/1828184871 shutdown_connections 2021-02-10T11:05:05.684 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:05.683 7f4113d32700 1 -- 172.21.2.102:0/1828184871 wait complete. 2021-02-10T11:05:05.691 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth del client.bootstrap-osd 2021-02-10T11:05:06.118 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:06.103 7fbfa8121700 1 Processor -- start 2021-02-10T11:05:06.118 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:06.103 7fbfa8121700 1 -- start start 2021-02-10T11:05:06.118 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:06.103 7fbfa8121700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fbfa00e5070 0x7fbfa01b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:06.118 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:06.103 7fbfa8121700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fbfa01b3e40 0x7fbfa007d690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:06.119 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:06.103 7fbfa8121700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfa007ed80 con 0x7fbfa00e5470 2021-02-10T11:05:06.119 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:06.103 7fbfa8121700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbfa0080310 con 0x7fbfa01b3e40 2021-02-10T11:05:06.119 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:06.103 7fbfa8121700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbfa0080530 con 0x7fbfa00e5070 2021-02-10T11:05:06.119 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:06.103 7fbfa573c700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fbfa01b3e40 0x7fbfa007d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:06.119 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:06.103 7fbfa573c700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fbfa01b3e40 0x7fbfa007d690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:50402/0 (socket says 172.21.2.102:50402) 2021-02-10T11:05:06.119 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:06.103 7fbfa573c700 1 -- 172.21.2.102:0/3979842389 learned_addr learned my addr 172.21.2.102:0/3979842389 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:05:06.119 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:06.103 7fbfa673e700 1 --1- 172.21.2.102:0/3979842389 >> v1:172.21.2.102:6789/0 conn(0x7fbfa00e5470 0x7fbfa007dcb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:42626/0 (socket says 172.21.2.102:42626) 2021-02-10T11:05:06.119 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:06.104 7fbfa573c700 1 -- 172.21.2.102:0/3979842389 >> v2:172.21.2.113:3300/0 conn(0x7fbfa00e5070 msgr2=0x7fbfa01b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:05:06.119 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:06.104 7fbfa573c700 1 --2- 172.21.2.102:0/3979842389 >> v2:172.21.2.113:3300/0 conn(0x7fbfa00e5070 0x7fbfa01b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:05:06.119 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:06.104 7fbfa573c700 1 -- 172.21.2.102:0/3979842389 >> v1:172.21.2.102:6789/0 conn(0x7fbfa00e5470 legacy=0x7fbfa007dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:06.119 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:06.104 7fbfa573c700 1 -- 172.21.2.102:0/3979842389 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfa0080750 con 0x7fbfa01b3e40 2021-02-10T11:05:06.119 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:06.104 7fbfa573c700 1 --2- 172.21.2.102:0/3979842389 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbfa01b3e40 0x7fbfa007d690 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 rx=0x7fbf94006700 tx=0x7fbf94009b00).ready entity=mon.0 client_cookie=3d1ca866d47624c2 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:05:06.119 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:06.104 7fbfa4f3b700 1 -- 172.21.2.102:0/3979842389 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbf9400c0c0 con 0x7fbfa01b3e40 2021-02-10T11:05:06.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:06.104 7fbfa4f3b700 1 -- 172.21.2.102:0/3979842389 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf9400c340 con 0x7fbfa01b3e40 2021-02-10T11:05:06.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:06.104 7fbfa4f3b700 1 -- 172.21.2.102:0/3979842389 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbf9400c820 con 0x7fbfa01b3e40 2021-02-10T11:05:06.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:06.104 7fbfa8121700 1 -- 172.21.2.102:0/3979842389 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbfa01b3e40 msgr2=0x7fbfa007d690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:06.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:06.104 7fbfa8121700 1 --2- 172.21.2.102:0/3979842389 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbfa01b3e40 0x7fbfa007d690 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 rx=0x7fbf94006700 tx=0x7fbf94009b00).stop 2021-02-10T11:05:06.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:06.105 7fbfa8121700 1 -- 172.21.2.102:0/3979842389 shutdown_connections 2021-02-10T11:05:06.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:06.105 7fbfa8121700 1 --2- 172.21.2.102:0/3979842389 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbfa01b3e40 0x7fbfa007d690 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:06.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:06.105 7fbfa8121700 1 --2- 172.21.2.102:0/3979842389 >> v2:172.21.2.113:3300/0 conn(0x7fbfa00e5070 0x7fbfa01b3820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:05:06.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:06.105 7fbfa8121700 1 -- 172.21.2.102:0/3979842389 >> 172.21.2.102:0/3979842389 conn(0x7fbfa00ce690 msgr2=0x7fbfa00e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:05:06.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:06.105 7fbfa8121700 1 -- 172.21.2.102:0/3979842389 shutdown_connections 2021-02-10T11:05:06.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:06.105 7fbfa8121700 1 -- 172.21.2.102:0/3979842389 wait complete. 2021-02-10T11:05:06.122 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:06.120 7fbfa8121700 1 Processor -- start 2021-02-10T11:05:06.122 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:06.120 7fbfa8121700 1 -- start start 2021-02-10T11:05:06.122 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:06.120 7fbfa8121700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fbfa0080f60 0x7fbfa01946e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:06.122 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:06.120 7fbfa8121700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbfa0194d00 0x7fbfa00a3450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:06.123 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:06.120 7fbfa8121700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbfa00a3a70 0x7fbfa00c15c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:06.123 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:06.120 7fbfa8121700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfa00a3ef0 con 0x7fbfa0194d00 2021-02-10T11:05:06.123 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:06.120 7fbfa8121700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfa00a40e0 con 0x7fbfa0080f60 2021-02-10T11:05:06.123 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:06.120 7fbfa8121700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfa00c1c20 con 0x7fbfa00a3a70 2021-02-10T11:05:06.123 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:06.121 7fbfa573c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbfa0194d00 0x7fbfa00a3450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:06.123 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:06.121 7fbfa573c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbfa0194d00 0x7fbfa00a3450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:50406/0 (socket says 172.21.2.102:50406) 2021-02-10T11:05:06.123 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:06.121 7fbfa573c700 1 -- 172.21.2.102:0/1295736819 learned_addr learned my addr 172.21.2.102:0/1295736819 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:05:06.123 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:06.121 7fbfa573c700 1 -- 172.21.2.102:0/1295736819 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbfa00a3a70 msgr2=0x7fbfa00c15c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:06.123 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:06.121 7fbfa573c700 1 --2- 172.21.2.102:0/1295736819 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbfa00a3a70 0x7fbfa00c15c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:05:06.123 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:06.121 7fbfa573c700 1 -- 172.21.2.102:0/1295736819 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fbfa0080f60 msgr2=0x7fbfa01946e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:06.123 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:06.121 7fbfa573c700 1 --2- 172.21.2.102:0/1295736819 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fbfa0080f60 0x7fbfa01946e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:05:06.123 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:06.121 7fbfa573c700 1 -- 172.21.2.102:0/1295736819 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfa00c1e40 con 0x7fbfa0194d00 2021-02-10T11:05:06.123 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:06.121 7fbfa573c700 1 --2- 172.21.2.102:0/1295736819 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbfa0194d00 0x7fbfa00a3450 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 rx=0x7fbf9400b580 tx=0x7fbf9400b920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:05:06.124 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:06.121 7fbf927fc700 1 -- 172.21.2.102:0/1295736819 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbf9400bd10 con 0x7fbfa0194d00 2021-02-10T11:05:06.124 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:06.121 7fbf927fc700 1 -- 172.21.2.102:0/1295736819 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf94006030 con 0x7fbfa0194d00 2021-02-10T11:05:06.124 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:06.121 7fbfa8121700 1 -- 172.21.2.102:0/1295736819 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfa00c1e40 con 0x7fbfa0194d00 2021-02-10T11:05:06.124 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:06.121 7fbf927fc700 1 -- 172.21.2.102:0/1295736819 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbf94005560 con 0x7fbfa0194d00 2021-02-10T11:05:06.124 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:06.122 7fbfa8121700 1 -- 172.21.2.102:0/1295736819 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfa00c1e40 con 0x7fbfa0194d00 2021-02-10T11:05:06.124 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:06.122 7fbf927fc700 1 -- 172.21.2.102:0/1295736819 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fbf940052e0 con 0x7fbfa0194d00 2021-02-10T11:05:06.124 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:06.122 7fbf927fc700 1 --2- 172.21.2.102:0/1295736819 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fbf88018730 0x7fbf8801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:06.124 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:06.122 7fbf927fc700 1 -- 172.21.2.102:0/1295736819 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 4753+0+0 (secure 0 0 0) 0x7fbf940324c0 con 0x7fbfa0194d00 2021-02-10T11:05:06.124 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:06.122 7fbfa5f3d700 1 --2- 172.21.2.102:0/1295736819 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fbf88018730 0x7fbf8801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:06.124 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:06.123 7fbfa5f3d700 1 --2- 172.21.2.102:0/1295736819 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fbf88018730 0x7fbf8801ac10 crc :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:05:06.152 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:06.153 7fbfa8121700 1 -- 172.21.2.102:0/1295736819 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbfa0107160 con 0x7fbfa0194d00 2021-02-10T11:05:06.154 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:06.156 7fbf927fc700 1 -- 172.21.2.102:0/1295736819 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fbf94006030 con 0x7fbfa0194d00 2021-02-10T11:05:06.284 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:06.285 7fbfa8121700 1 -- 172.21.2.102:0/1295736819 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-osd"} v 0) v1 -- 0x7fbfa0107160 con 0x7fbfa0194d00 2021-02-10T11:05:06.287 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:06.289 7fbf927fc700 1 -- 172.21.2.102:0/1295736819 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-osd"}]=0 updated v7) v1 ==== 97+0+0 (secure 0 0 0) 0x7fbf940052e0 con 0x7fbfa0194d00 2021-02-10T11:05:06.291 INFO:teuthology.orchestra.run.gibba002.stderr:updated 2021-02-10T11:05:06.292 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:06.294 7fbfa8121700 1 -- 172.21.2.102:0/1295736819 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fbf88018730 msgr2=0x7fbf8801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:06.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:06.294 7fbfa8121700 1 --2- 172.21.2.102:0/1295736819 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fbf88018730 0x7fbf8801ac10 crc :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:06.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:06.294 7fbfa8121700 1 -- 172.21.2.102:0/1295736819 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbfa0194d00 msgr2=0x7fbfa00a3450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:06.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:06.294 7fbfa8121700 1 --2- 172.21.2.102:0/1295736819 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbfa0194d00 0x7fbfa00a3450 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 rx=0x7fbf9400b580 tx=0x7fbf9400b920).stop 2021-02-10T11:05:06.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:06.294 7fbfa8121700 1 -- 172.21.2.102:0/1295736819 shutdown_connections 2021-02-10T11:05:06.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:06.294 7fbfa8121700 1 --2- 172.21.2.102:0/1295736819 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbfa00a3a70 0x7fbfa00c15c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:05:06.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:06.294 7fbfa8121700 1 --2- 172.21.2.102:0/1295736819 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbfa0194d00 0x7fbfa00a3450 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:06.294 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:06.294 7fbfa8121700 1 --2- 172.21.2.102:0/1295736819 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fbf88018730 0x7fbf8801ac10 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:06.294 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:06.294 7fbfa8121700 1 --2- 172.21.2.102:0/1295736819 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fbfa0080f60 0x7fbfa01946e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:05:06.294 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:06.294 7fbfa8121700 1 -- 172.21.2.102:0/1295736819 >> 172.21.2.102:0/1295736819 conn(0x7fbfa0081b50 msgr2=0x7fbfa00a2850 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:05:06.294 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:06.294 7fbfa8121700 1 -- 172.21.2.102:0/1295736819 shutdown_connections 2021-02-10T11:05:06.294 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:06.294 7fbfa8121700 1 -- 172.21.2.102:0/1295736819 wait complete. 2021-02-10T11:05:06.301 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth del client.bootstrap-rbd 2021-02-10T11:05:06.727 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:06.711 7f6ebb3e8700 1 Processor -- start 2021-02-10T11:05:06.727 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:06.711 7f6ebb3e8700 1 -- start start 2021-02-10T11:05:06.728 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:06.712 7f6ebb3e8700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f6eb407e6a0 0x7f6eb40d1790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:06.728 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:06.712 7f6ebb3e8700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6eb407ed80 con 0x7f6eb40e5470 2021-02-10T11:05:06.728 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:06.712 7f6ebb3e8700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6eb407eaa0 con 0x7f6eb40e5070 2021-02-10T11:05:06.728 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:06.712 7f6ebb3e8700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6eb41b4080 con 0x7f6eb407e6a0 2021-02-10T11:05:06.728 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:06.712 7f6eb9a05700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f6eb407e6a0 0x7f6eb40d1790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:06.728 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:06.712 7f6eb9a05700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f6eb407e6a0 0x7f6eb40d1790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:50416/0 (socket says 172.21.2.102:50416) 2021-02-10T11:05:06.728 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:06.712 7f6eb9a05700 1 -- 172.21.2.102:0/1354037887 learned_addr learned my addr 172.21.2.102:0/1354037887 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:05:06.728 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:06.713 7f6eabfff700 1 -- 172.21.2.102:0/1354037887 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1618578420 0 0) 0x7f6ea4000ce0 con 0x7f6eb40e5470 2021-02-10T11:05:06.728 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:06.713 7f6eabfff700 1 -- 172.21.2.102:0/1354037887 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ea0002fe0 con 0x7f6eb40e5470 2021-02-10T11:05:06.728 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:06.713 7f6eb9a05700 1 -- 172.21.2.102:0/1354037887 >> v1:172.21.2.113:6789/0 conn(0x7f6eb40e5070 legacy=0x7f6eb41b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:06.728 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:06.713 7f6eb9a05700 1 -- 172.21.2.102:0/1354037887 >> v1:172.21.2.102:6789/0 conn(0x7f6eb40e5470 legacy=0x7f6eb407f300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:06.728 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:06.713 7f6eb9a05700 1 -- 172.21.2.102:0/1354037887 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6eb40e6090 con 0x7f6eb407e6a0 2021-02-10T11:05:06.729 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:06.713 7f6eb9a05700 1 --2- 172.21.2.102:0/1354037887 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6eb407e6a0 0x7f6eb40d1790 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 rx=0x7f6eac006f00 tx=0x7f6eac009f10).ready entity=mon.0 client_cookie=db9371a36bc8f57b server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:05:06.729 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:06.713 7f6eabfff700 1 -- 172.21.2.102:0/1354037887 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6eac00b3b0 con 0x7f6eb407e6a0 2021-02-10T11:05:06.729 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:06.713 7f6eabfff700 1 -- 172.21.2.102:0/1354037887 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6eac00b730 con 0x7f6eb407e6a0 2021-02-10T11:05:06.729 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:06.713 7f6eabfff700 1 -- 172.21.2.102:0/1354037887 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6eac00bc50 con 0x7f6eb407e6a0 2021-02-10T11:05:06.729 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:06.713 7f6ebb3e8700 1 -- 172.21.2.102:0/1354037887 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6eb407e6a0 msgr2=0x7f6eb40d1790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:06.729 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:06.713 7f6ebb3e8700 1 --2- 172.21.2.102:0/1354037887 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6eb407e6a0 0x7f6eb40d1790 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 rx=0x7f6eac006f00 tx=0x7f6eac009f10).stop 2021-02-10T11:05:06.729 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:06.714 7f6ebb3e8700 1 -- 172.21.2.102:0/1354037887 shutdown_connections 2021-02-10T11:05:06.729 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:06.714 7f6ebb3e8700 1 --2- 172.21.2.102:0/1354037887 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6eb407e6a0 0x7f6eb40d1790 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:06.729 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:06.714 7f6ebb3e8700 1 -- 172.21.2.102:0/1354037887 >> 172.21.2.102:0/1354037887 conn(0x7f6eb40ce690 msgr2=0x7f6eb40e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:05:06.729 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:06.714 7f6ebb3e8700 1 -- 172.21.2.102:0/1354037887 shutdown_connections 2021-02-10T11:05:06.729 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:06.714 7f6ebb3e8700 1 -- 172.21.2.102:0/1354037887 wait complete. 2021-02-10T11:05:06.729 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:06.729 7f6ebb3e8700 1 Processor -- start 2021-02-10T11:05:06.729 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:06.729 7f6ebb3e8700 1 -- start start 2021-02-10T11:05:06.731 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:06.729 7f6ebb3e8700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6eb40ab640 0x7f6eb40bf7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:06.731 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:06.729 7f6ebb3e8700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6eb40bfde0 0x7f6eb40a9bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:06.731 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:06.729 7f6ebb3e8700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6eb40c01e0 0x7f6eb40aa1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:06.731 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:06.729 7f6ebb3e8700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6eb40aba40 con 0x7f6eb40ab640 2021-02-10T11:05:06.732 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:06.729 7f6ebb3e8700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6eb40ab0c0 con 0x7f6eb40c01e0 2021-02-10T11:05:06.732 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:06.729 7f6ebb3e8700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6eb40899f0 con 0x7f6eb40bfde0 2021-02-10T11:05:06.732 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:06.729 7f6eb9204700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6eb40ab640 0x7f6eb40bf7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:06.732 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:06.729 7f6eb9204700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6eb40ab640 0x7f6eb40bf7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:50422/0 (socket says 172.21.2.102:50422) 2021-02-10T11:05:06.732 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:06.729 7f6eb9204700 1 -- 172.21.2.102:0/2395113588 learned_addr learned my addr 172.21.2.102:0/2395113588 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:05:06.732 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:06.730 7f6eb9a05700 1 --2- 172.21.2.102:0/2395113588 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6eb40c01e0 0x7f6eb40aa1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:06.732 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:06.730 7f6eb8a03700 1 --2- 172.21.2.102:0/2395113588 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6eb40bfde0 0x7f6eb40a9bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:06.732 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:06.730 7f6eb9204700 1 -- 172.21.2.102:0/2395113588 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6eb40bfde0 msgr2=0x7f6eb40a9bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:06.732 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:06.730 7f6eb9204700 1 --2- 172.21.2.102:0/2395113588 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6eb40bfde0 0x7f6eb40a9bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:06.732 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:06.730 7f6eb9204700 1 -- 172.21.2.102:0/2395113588 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6eb40c01e0 msgr2=0x7f6eb40aa1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:06.732 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:06.730 7f6eb9204700 1 --2- 172.21.2.102:0/2395113588 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6eb40c01e0 0x7f6eb40aa1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:06.732 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:06.730 7f6eb9204700 1 -- 172.21.2.102:0/2395113588 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6eb40aa810 con 0x7f6eb40ab640 2021-02-10T11:05:06.732 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:06.730 7f6eb9204700 1 --2- 172.21.2.102:0/2395113588 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6eb40ab640 0x7f6eb40bf7c0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 rx=0x7f6eb000b5b0 tx=0x7f6eb000b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:05:06.733 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:06.730 7f6ea9ffb700 1 -- 172.21.2.102:0/2395113588 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6eb000bd40 con 0x7f6eb40ab640 2021-02-10T11:05:06.733 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:06.730 7f6ea9ffb700 1 -- 172.21.2.102:0/2395113588 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6eb0006240 con 0x7f6eb40ab640 2021-02-10T11:05:06.733 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:06.730 7f6ea9ffb700 1 -- 172.21.2.102:0/2395113588 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6eb0005720 con 0x7f6eb40ab640 2021-02-10T11:05:06.733 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:06.730 7f6ebb3e8700 1 -- 172.21.2.102:0/2395113588 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6eb40aa810 con 0x7f6eb40ab640 2021-02-10T11:05:06.733 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:06.730 7f6ebb3e8700 1 -- 172.21.2.102:0/2395113588 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6eb40aada0 con 0x7f6eb40ab640 2021-02-10T11:05:06.733 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:06.731 7f6ea9ffb700 1 -- 172.21.2.102:0/2395113588 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f6eb00054a0 con 0x7f6eb40ab640 2021-02-10T11:05:06.733 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:06.731 7f6ea9ffb700 1 --2- 172.21.2.102:0/2395113588 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6e940186c0 0x7f6e9401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:06.733 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:06.731 7f6ea9ffb700 1 -- 172.21.2.102:0/2395113588 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 4753+0+0 (secure 0 0 0) 0x7f6eb0032140 con 0x7f6eb40ab640 2021-02-10T11:05:06.734 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:06.731 7f6eb8a03700 1 --2- 172.21.2.102:0/2395113588 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6e940186c0 0x7f6e9401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:06.734 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:06.731 7f6eb8a03700 1 --2- 172.21.2.102:0/2395113588 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6e940186c0 0x7f6e9401abc0 crc :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:05:06.760 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:06.761 7f6ebb3e8700 1 -- 172.21.2.102:0/2395113588 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6eb40aba40 con 0x7f6eb40ab640 2021-02-10T11:05:06.763 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:06.764 7f6ea9ffb700 1 -- 172.21.2.102:0/2395113588 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f6eb0000940 con 0x7f6eb40ab640 2021-02-10T11:05:06.893 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:06.894 7f6ebb3e8700 1 -- 172.21.2.102:0/2395113588 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd"} v 0) v1 -- 0x7f6eb40aba40 con 0x7f6eb40ab640 2021-02-10T11:05:06.896 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:06.897 7f6ea9ffb700 1 -- 172.21.2.102:0/2395113588 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd"}]=0 updated v8) v1 ==== 97+0+0 (secure 0 0 0) 0x7f6eb00054a0 con 0x7f6eb40ab640 2021-02-10T11:05:06.896 INFO:teuthology.orchestra.run.gibba002.stderr:updated 2021-02-10T11:05:06.898 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:06.899 7f6ebb3e8700 1 -- 172.21.2.102:0/2395113588 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6e940186c0 msgr2=0x7f6e9401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:06.898 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:06.899 7f6ebb3e8700 1 --2- 172.21.2.102:0/2395113588 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6e940186c0 0x7f6e9401abc0 crc :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:06.898 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:06.899 7f6ebb3e8700 1 -- 172.21.2.102:0/2395113588 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6eb40ab640 msgr2=0x7f6eb40bf7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:06.898 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:06.899 7f6ebb3e8700 1 --2- 172.21.2.102:0/2395113588 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6eb40ab640 0x7f6eb40bf7c0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 rx=0x7f6eb000b5b0 tx=0x7f6eb000b950).stop 2021-02-10T11:05:06.899 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:06.899 7f6ebb3e8700 1 -- 172.21.2.102:0/2395113588 shutdown_connections 2021-02-10T11:05:06.899 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:06.899 7f6ebb3e8700 1 --2- 172.21.2.102:0/2395113588 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6e940186c0 0x7f6e9401abc0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:06.899 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:06.899 7f6ebb3e8700 1 --2- 172.21.2.102:0/2395113588 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6eb40c01e0 0x7f6eb40aa1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:06.899 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:06.899 7f6ebb3e8700 1 --2- 172.21.2.102:0/2395113588 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6eb40bfde0 0x7f6eb40a9bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:06.899 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:06.899 7f6ebb3e8700 1 --2- 172.21.2.102:0/2395113588 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6eb40ab640 0x7f6eb40bf7c0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:06.899 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:06.899 7f6ebb3e8700 1 -- 172.21.2.102:0/2395113588 >> 172.21.2.102:0/2395113588 conn(0x7f6eb40a4400 msgr2=0x7f6eb40a6d40 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:05:06.899 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:06.899 7f6ebb3e8700 1 -- 172.21.2.102:0/2395113588 shutdown_connections 2021-02-10T11:05:06.899 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:06.900 7f6ebb3e8700 1 -- 172.21.2.102:0/2395113588 wait complete. 2021-02-10T11:05:06.906 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth del client.bootstrap-rbd-mirror 2021-02-10T11:05:07.332 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:07.317 7fa5728aa700 1 Processor -- start 2021-02-10T11:05:07.332 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:07.317 7fa5728aa700 1 -- start start 2021-02-10T11:05:07.333 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:07.318 7fa5728aa700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa56c0e5470 0x7fa56c07e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:07.333 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:07.318 7fa5728aa700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa56c07ed80 con 0x7fa56c07f300 2021-02-10T11:05:07.333 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:07.318 7fa5728aa700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa56c1b4010 con 0x7fa56c0e5070 2021-02-10T11:05:07.333 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:07.318 7fa5728aa700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa56c0d1790 con 0x7fa56c0e5470 2021-02-10T11:05:07.333 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:07.318 7fa570ec7700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fa56c07f300 0x7fa56c07f700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:42656/0 (socket says 172.21.2.102:42656) 2021-02-10T11:05:07.333 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:07.318 7fa570ec7700 1 -- 172.21.2.102:0/2625556744 learned_addr learned my addr 172.21.2.102:0/2625556744 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:05:07.333 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:07.318 7fa56b7fe700 1 --2- 172.21.2.102:0/2625556744 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa56c0e5470 0x7fa56c07e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:07.333 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:07.318 7fa56affd700 1 -- 172.21.2.102:0/2625556744 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2777731674 0 0) 0x7fa564001500 con 0x7fa56c07f300 2021-02-10T11:05:07.333 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:07.318 7fa56affd700 1 -- 172.21.2.102:0/2625556744 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa554002fe0 con 0x7fa56c07f300 2021-02-10T11:05:07.333 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:07.318 7fa56affd700 1 -- 172.21.2.102:0/2625556744 <== mon.1 v1:172.21.2.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3367788887 0 0) 0x7fa55c000ff0 con 0x7fa56c0e5070 2021-02-10T11:05:07.333 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:07.318 7fa56affd700 1 -- 172.21.2.102:0/2625556744 --> v1:172.21.2.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa554003210 con 0x7fa56c0e5070 2021-02-10T11:05:07.333 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:07.319 7fa56b7fe700 1 -- 172.21.2.102:0/2625556744 >> v1:172.21.2.113:6789/0 conn(0x7fa56c0e5070 legacy=0x7fa56c1b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:07.333 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:07.319 7fa56b7fe700 1 -- 172.21.2.102:0/2625556744 >> v1:172.21.2.102:6789/0 conn(0x7fa56c07f300 legacy=0x7fa56c07f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:07.334 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:07.319 7fa56affd700 1 -- 172.21.2.102:0/2625556744 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3190834803 0 0) 0x7fa564001460 con 0x7fa56c07f300 2021-02-10T11:05:07.334 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:07.319 7fa56b7fe700 1 -- 172.21.2.102:0/2625556744 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa56c0d1980 con 0x7fa56c0e5470 2021-02-10T11:05:07.334 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:07.319 7fa56b7fe700 1 --2- 172.21.2.102:0/2625556744 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa56c0e5470 0x7fa56c07e6a0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7fa560006650 tx=0x7fa560009a80).ready entity=mon.2 client_cookie=813dcc9cfb93cc7f server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:05:07.334 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:07.319 7fa56affd700 1 -- 172.21.2.102:0/2625556744 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa5600009b0 con 0x7fa56c0e5470 2021-02-10T11:05:07.334 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:07.319 7fa56affd700 1 -- 172.21.2.102:0/2625556744 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa560006740 con 0x7fa56c0e5470 2021-02-10T11:05:07.334 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:07.319 7fa56affd700 1 -- 172.21.2.102:0/2625556744 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa56000c0c0 con 0x7fa56c0e5470 2021-02-10T11:05:07.334 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:07.319 7fa5728aa700 1 -- 172.21.2.102:0/2625556744 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa56c0e5470 msgr2=0x7fa56c07e6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:07.334 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:07.319 7fa5728aa700 1 --2- 172.21.2.102:0/2625556744 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa56c0e5470 0x7fa56c07e6a0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7fa560006650 tx=0x7fa560009a80).stop 2021-02-10T11:05:07.334 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:07.319 7fa5728aa700 1 -- 172.21.2.102:0/2625556744 shutdown_connections 2021-02-10T11:05:07.334 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:07.319 7fa5728aa700 1 --2- 172.21.2.102:0/2625556744 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa56c0e5470 0x7fa56c07e6a0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:07.334 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:07.319 7fa5728aa700 1 -- 172.21.2.102:0/2625556744 >> 172.21.2.102:0/2625556744 conn(0x7fa56c0ce690 msgr2=0x7fa56c0e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:05:07.334 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:07.320 7fa5728aa700 1 -- 172.21.2.102:0/2625556744 shutdown_connections 2021-02-10T11:05:07.334 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:07.320 7fa5728aa700 1 -- 172.21.2.102:0/2625556744 wait complete. 2021-02-10T11:05:07.336 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:07.334 7fa5728aa700 1 Processor -- start 2021-02-10T11:05:07.336 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:07.334 7fa5728aa700 1 -- start start 2021-02-10T11:05:07.336 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:07.334 7fa5728aa700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa56c1a3150 0x7fa56c1a3550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:07.336 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:07.335 7fa5728aa700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa56c187eb0 0x7fa56c1882b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:07.336 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:07.335 7fa5728aa700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa56c0bd8c0 0x7fa56c0bdcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:07.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:07.335 7fa5728aa700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa56c188c80 con 0x7fa56c0bd8c0 2021-02-10T11:05:07.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:07.335 7fa5728aa700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa56c1888d0 con 0x7fa56c1a3150 2021-02-10T11:05:07.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:07.335 7fa5728aa700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa56c0be2e0 con 0x7fa56c187eb0 2021-02-10T11:05:07.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:07.335 7fa570ec7700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa56c0bd8c0 0x7fa56c0bdcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:07.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:07.335 7fa570ec7700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa56c0bd8c0 0x7fa56c0bdcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:50436/0 (socket says 172.21.2.102:50436) 2021-02-10T11:05:07.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:07.335 7fa570ec7700 1 -- 172.21.2.102:0/1960626515 learned_addr learned my addr 172.21.2.102:0/1960626515 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:05:07.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:07.335 7fa56bfff700 1 --2- 172.21.2.102:0/1960626515 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa56c1a3150 0x7fa56c1a3550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:07.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:07.335 7fa56b7fe700 1 --2- 172.21.2.102:0/1960626515 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa56c187eb0 0x7fa56c1882b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:07.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:07.335 7fa570ec7700 1 -- 172.21.2.102:0/1960626515 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa56c187eb0 msgr2=0x7fa56c1882b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:07.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:07.335 7fa570ec7700 1 --2- 172.21.2.102:0/1960626515 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa56c187eb0 0x7fa56c1882b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:07.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:07.335 7fa570ec7700 1 -- 172.21.2.102:0/1960626515 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa56c1a3150 msgr2=0x7fa56c1a3550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:07.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:07.335 7fa570ec7700 1 --2- 172.21.2.102:0/1960626515 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa56c1a3150 0x7fa56c1a3550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:07.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:07.335 7fa570ec7700 1 -- 172.21.2.102:0/1960626515 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa56c0be500 con 0x7fa56c0bd8c0 2021-02-10T11:05:07.338 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:07.335 7fa56b7fe700 1 --2- 172.21.2.102:0/1960626515 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa56c187eb0 0x7fa56c1882b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:05:07.338 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:07.335 7fa570ec7700 1 --2- 172.21.2.102:0/1960626515 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa56c0bd8c0 0x7fa56c0bdcc0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 rx=0x7fa564009980 tx=0x7fa56400bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:05:07.338 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:07.336 7fa568ff9700 1 -- 172.21.2.102:0/1960626515 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa564006030 con 0x7fa56c0bd8c0 2021-02-10T11:05:07.338 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:07.336 7fa568ff9700 1 -- 172.21.2.102:0/1960626515 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa564006370 con 0x7fa56c0bd8c0 2021-02-10T11:05:07.338 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:07.336 7fa568ff9700 1 -- 172.21.2.102:0/1960626515 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa564004720 con 0x7fa56c0bd8c0 2021-02-10T11:05:07.338 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:07.336 7fa5728aa700 1 -- 172.21.2.102:0/1960626515 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa56c0be500 con 0x7fa56c0bd8c0 2021-02-10T11:05:07.338 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:07.336 7fa5728aa700 1 -- 172.21.2.102:0/1960626515 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa56c1894f0 con 0x7fa56c0bd8c0 2021-02-10T11:05:07.338 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:07.336 7fa568ff9700 1 -- 172.21.2.102:0/1960626515 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa564006550 con 0x7fa56c0bd8c0 2021-02-10T11:05:07.338 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:07.336 7fa568ff9700 1 --2- 172.21.2.102:0/1960626515 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa54c0186c0 0x7fa54c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:07.338 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:07.336 7fa568ff9700 1 -- 172.21.2.102:0/1960626515 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 4753+0+0 (secure 0 0 0) 0x7fa5640325c0 con 0x7fa56c0bd8c0 2021-02-10T11:05:07.338 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:07.336 7fa56bfff700 1 --2- 172.21.2.102:0/1960626515 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa54c0186c0 0x7fa54c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:07.338 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:07.337 7fa56bfff700 1 --2- 172.21.2.102:0/1960626515 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa54c0186c0 0x7fa54c01abc0 crc :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:05:07.366 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:07.367 7fa5728aa700 1 -- 172.21.2.102:0/1960626515 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa56c07ed80 con 0x7fa56c0bd8c0 2021-02-10T11:05:07.369 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:07.370 7fa568ff9700 1 -- 172.21.2.102:0/1960626515 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fa564000ee0 con 0x7fa56c0bd8c0 2021-02-10T11:05:07.500 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:07.501 7fa5728aa700 1 -- 172.21.2.102:0/1960626515 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"} v 0) v1 -- 0x7fa56c07ed80 con 0x7fa56c0bd8c0 2021-02-10T11:05:07.503 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:07.505 7fa568ff9700 1 -- 172.21.2.102:0/1960626515 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"}]=0 updated v9) v1 ==== 104+0+0 (secure 0 0 0) 0x7fa564006550 con 0x7fa56c0bd8c0 2021-02-10T11:05:07.507 INFO:teuthology.orchestra.run.gibba002.stderr:updated 2021-02-10T11:05:07.508 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:07.510 7fa5728aa700 1 -- 172.21.2.102:0/1960626515 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa54c0186c0 msgr2=0x7fa54c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:07.509 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:07.510 7fa5728aa700 1 --2- 172.21.2.102:0/1960626515 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa54c0186c0 0x7fa54c01abc0 crc :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:07.509 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:07.510 7fa5728aa700 1 -- 172.21.2.102:0/1960626515 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa56c0bd8c0 msgr2=0x7fa56c0bdcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:07.509 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:07.510 7fa5728aa700 1 --2- 172.21.2.102:0/1960626515 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa56c0bd8c0 0x7fa56c0bdcc0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 rx=0x7fa564009980 tx=0x7fa56400bcd0).stop 2021-02-10T11:05:07.509 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:07.510 7fa5728aa700 1 -- 172.21.2.102:0/1960626515 shutdown_connections 2021-02-10T11:05:07.509 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:07.510 7fa5728aa700 1 --2- 172.21.2.102:0/1960626515 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa56c0bd8c0 0x7fa56c0bdcc0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:07.509 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:07.510 7fa5728aa700 1 --2- 172.21.2.102:0/1960626515 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa56c187eb0 0x7fa56c1882b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:07.509 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:07.510 7fa5728aa700 1 --2- 172.21.2.102:0/1960626515 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa54c0186c0 0x7fa54c01abc0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:07.509 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:07.510 7fa5728aa700 1 --2- 172.21.2.102:0/1960626515 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa56c1a3150 0x7fa56c1a3550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:07.509 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:07.510 7fa5728aa700 1 -- 172.21.2.102:0/1960626515 >> 172.21.2.102:0/1960626515 conn(0x7fa56c098e00 msgr2=0x7fa56c0adca0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:05:07.510 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:07.510 7fa5728aa700 1 -- 172.21.2.102:0/1960626515 shutdown_connections 2021-02-10T11:05:07.510 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:07.510 7fa5728aa700 1 -- 172.21.2.102:0/1960626515 wait complete. 2021-02-10T11:05:07.517 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth del client.bootstrap-rgw 2021-02-10T11:05:07.943 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:07.928 7f31da749700 1 Processor -- start 2021-02-10T11:05:07.943 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:07.928 7f31da749700 1 -- start start 2021-02-10T11:05:07.943 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:07.929 7f31da749700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f31d40e5070 0x7f31d41b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:07.944 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:07.929 7f31da749700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f31d40e5470 0x7f31d407d690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:07.944 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:07.929 7f31da749700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31d407ed80 con 0x7f31d41b3e40 2021-02-10T11:05:07.944 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:07.929 7f31da749700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31d4080310 con 0x7f31d40e5470 2021-02-10T11:05:07.944 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:07.929 7f31da749700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31d4080500 con 0x7f31d40e5070 2021-02-10T11:05:07.944 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:07.929 7f31d37fe700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f31d40e5470 0x7f31d407d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:07.944 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:07.929 7f31d37fe700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f31d40e5470 0x7f31d407d690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:50444/0 (socket says 172.21.2.102:50444) 2021-02-10T11:05:07.944 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:07.929 7f31d37fe700 1 -- 172.21.2.102:0/3047258711 learned_addr learned my addr 172.21.2.102:0/3047258711 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:05:07.944 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:07.929 7f31d37fe700 1 -- 172.21.2.102:0/3047258711 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f31d40e5070 msgr2=0x7f31d41b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:05:07.944 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:07.929 7f31d37fe700 1 --2- 172.21.2.102:0/3047258711 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f31d40e5070 0x7f31d41b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:05:07.944 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:07.929 7f31d37fe700 1 -- 172.21.2.102:0/3047258711 >> v1:172.21.2.113:6789/0 conn(0x7f31d41b3e40 legacy=0x7f31d407dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:05:07.944 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:07.929 7f31d37fe700 1 -- 172.21.2.102:0/3047258711 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31d40806f0 con 0x7f31d40e5470 2021-02-10T11:05:07.944 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:07.929 7f31d37fe700 1 --2- 172.21.2.102:0/3047258711 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f31d40e5470 0x7f31d407d690 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 rx=0x7f31c8006700 tx=0x7f31c8009b00).ready entity=mon.0 client_cookie=161fe78fb8b7f7f9 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:05:07.944 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:07.929 7f31d2ffd700 1 -- 172.21.2.102:0/3047258711 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f31c800c0c0 con 0x7f31d40e5470 2021-02-10T11:05:07.945 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:07.929 7f31d2ffd700 1 -- 172.21.2.102:0/3047258711 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31c800c340 con 0x7f31d40e5470 2021-02-10T11:05:07.945 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:07.929 7f31d2ffd700 1 -- 172.21.2.102:0/3047258711 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f31c800c7a0 con 0x7f31d40e5470 2021-02-10T11:05:07.945 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:07.929 7f31da749700 1 -- 172.21.2.102:0/3047258711 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f31d40e5470 msgr2=0x7f31d407d690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:07.945 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:07.929 7f31da749700 1 --2- 172.21.2.102:0/3047258711 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f31d40e5470 0x7f31d407d690 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 rx=0x7f31c8006700 tx=0x7f31c8009b00).stop 2021-02-10T11:05:07.945 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:07.930 7f31da749700 1 -- 172.21.2.102:0/3047258711 shutdown_connections 2021-02-10T11:05:07.945 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:07.930 7f31da749700 1 --2- 172.21.2.102:0/3047258711 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f31d40e5470 0x7f31d407d690 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:07.945 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:07.930 7f31da749700 1 --2- 172.21.2.102:0/3047258711 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f31d40e5070 0x7f31d41b3820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:05:07.945 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:07.930 7f31da749700 1 -- 172.21.2.102:0/3047258711 >> 172.21.2.102:0/3047258711 conn(0x7f31d40ce690 msgr2=0x7f31d40e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:05:07.945 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:07.930 7f31da749700 1 -- 172.21.2.102:0/3047258711 shutdown_connections 2021-02-10T11:05:07.945 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:07.930 7f31da749700 1 -- 172.21.2.102:0/3047258711 wait complete. 2021-02-10T11:05:07.945 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:07.945 7f31da749700 1 Processor -- start 2021-02-10T11:05:07.947 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:07.945 7f31da749700 1 -- start start 2021-02-10T11:05:07.947 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:07.945 7f31da749700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f31d409acf0 0x7f31d40a98e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:07.947 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:07.945 7f31da749700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f31d40a9f00 0x7f31d40aa980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:07.947 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:07.945 7f31da749700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f31d409b0f0 0x7f31d40aafa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:07.947 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:07.945 7f31da749700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31d40b45c0 con 0x7f31d409b0f0 2021-02-10T11:05:07.948 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:07.945 7f31da749700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31d408a430 con 0x7f31d409acf0 2021-02-10T11:05:07.948 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:07.945 7f31da749700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31d40b4990 con 0x7f31d40a9f00 2021-02-10T11:05:07.948 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:07.945 7f31d8d66700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f31d409b0f0 0x7f31d40aafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:07.948 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:07.945 7f31d8d66700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f31d409b0f0 0x7f31d40aafa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:50454/0 (socket says 172.21.2.102:50454) 2021-02-10T11:05:07.948 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:07.945 7f31d8d66700 1 -- 172.21.2.102:0/2250034398 learned_addr learned my addr 172.21.2.102:0/2250034398 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:05:07.948 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:07.946 7f31d8d66700 1 -- 172.21.2.102:0/2250034398 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f31d40a9f00 msgr2=0x7f31d40aa980 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:05:07.948 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:07.946 7f31d8d66700 1 --2- 172.21.2.102:0/2250034398 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f31d40a9f00 0x7f31d40aa980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:05:07.948 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:07.946 7f31d8d66700 1 -- 172.21.2.102:0/2250034398 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f31d409acf0 msgr2=0x7f31d40a98e0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:05:07.948 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:07.946 7f31d8d66700 1 --2- 172.21.2.102:0/2250034398 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f31d409acf0 0x7f31d40a98e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:05:07.948 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:07.946 7f31d8d66700 1 -- 172.21.2.102:0/2250034398 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31d40ab5c0 con 0x7f31d409b0f0 2021-02-10T11:05:07.948 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:07.946 7f31d8d66700 1 --2- 172.21.2.102:0/2250034398 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f31d409b0f0 0x7f31d40aafa0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 rx=0x7f31cc009940 tx=0x7f31cc00bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:05:07.949 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:07.946 7f31d0ff9700 1 -- 172.21.2.102:0/2250034398 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f31cc000f90 con 0x7f31d409b0f0 2021-02-10T11:05:07.949 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:07.946 7f31d0ff9700 1 -- 172.21.2.102:0/2250034398 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31cc00c2f0 con 0x7f31d409b0f0 2021-02-10T11:05:07.949 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:07.946 7f31d0ff9700 1 -- 172.21.2.102:0/2250034398 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f31cc00c7d0 con 0x7f31d409b0f0 2021-02-10T11:05:07.949 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:07.946 7f31da749700 1 -- 172.21.2.102:0/2250034398 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31d40ab5c0 con 0x7f31d409b0f0 2021-02-10T11:05:07.949 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:07.946 7f31da749700 1 -- 172.21.2.102:0/2250034398 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31d40ab5c0 con 0x7f31d409b0f0 2021-02-10T11:05:07.949 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:07.947 7f31d0ff9700 1 -- 172.21.2.102:0/2250034398 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f31cc001220 con 0x7f31d409b0f0 2021-02-10T11:05:07.949 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:07.947 7f31d0ff9700 1 --2- 172.21.2.102:0/2250034398 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f31b4018730 0x7f31b401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:07.949 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:07.947 7f31d0ff9700 1 -- 172.21.2.102:0/2250034398 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 4753+0+0 (secure 0 0 0) 0x7f31cc032690 con 0x7f31d409b0f0 2021-02-10T11:05:07.949 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:07.947 7f31d3fff700 1 --2- 172.21.2.102:0/2250034398 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f31b4018730 0x7f31b401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:07.949 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:07.947 7f31d3fff700 1 --2- 172.21.2.102:0/2250034398 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f31b4018730 0x7f31b401ac10 crc :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:05:07.977 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:07.978 7f31da749700 1 -- 172.21.2.102:0/2250034398 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31d407ed80 con 0x7f31d409b0f0 2021-02-10T11:05:07.979 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:07.981 7f31d0ff9700 1 -- 172.21.2.102:0/2250034398 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f31cc00be30 con 0x7f31d409b0f0 2021-02-10T11:05:08.108 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:08.109 7f31da749700 1 -- 172.21.2.102:0/2250034398 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rgw"} v 0) v1 -- 0x7f31d407ed80 con 0x7f31d409b0f0 2021-02-10T11:05:08.111 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:08.112 7f31d0ff9700 1 -- 172.21.2.102:0/2250034398 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rgw"}]=0 updated v10) v1 ==== 97+0+0 (secure 0 0 0) 0x7f31cc001220 con 0x7f31d409b0f0 2021-02-10T11:05:08.115 INFO:teuthology.orchestra.run.gibba002.stderr:updated 2021-02-10T11:05:08.117 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:08.117 7f31da749700 1 -- 172.21.2.102:0/2250034398 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f31b4018730 msgr2=0x7f31b401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:08.117 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:08.117 7f31da749700 1 --2- 172.21.2.102:0/2250034398 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f31b4018730 0x7f31b401ac10 crc :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:08.117 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:08.117 7f31da749700 1 -- 172.21.2.102:0/2250034398 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f31d409b0f0 msgr2=0x7f31d40aafa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:08.117 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:08.117 7f31da749700 1 --2- 172.21.2.102:0/2250034398 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f31d409b0f0 0x7f31d40aafa0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 rx=0x7f31cc009940 tx=0x7f31cc00bcf0).stop 2021-02-10T11:05:08.117 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:08.117 7f31da749700 1 -- 172.21.2.102:0/2250034398 shutdown_connections 2021-02-10T11:05:08.117 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:08.118 7f31da749700 1 --2- 172.21.2.102:0/2250034398 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f31d409b0f0 0x7f31d40aafa0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:08.117 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:08.118 7f31da749700 1 --2- 172.21.2.102:0/2250034398 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f31d40a9f00 0x7f31d40aa980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:05:08.117 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:08.118 7f31da749700 1 --2- 172.21.2.102:0/2250034398 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f31b4018730 0x7f31b401ac10 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:08.118 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:08.118 7f31da749700 1 --2- 172.21.2.102:0/2250034398 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f31d409acf0 0x7f31d40a98e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:05:08.118 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:08.118 7f31da749700 1 -- 172.21.2.102:0/2250034398 >> 172.21.2.102:0/2250034398 conn(0x7f31d4085960 msgr2=0x7f31d40a1c70 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:05:08.118 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:08.118 7f31da749700 1 -- 172.21.2.102:0/2250034398 shutdown_connections 2021-02-10T11:05:08.118 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:08.118 7f31da749700 1 -- 172.21.2.102:0/2250034398 wait complete. 2021-02-10T11:05:08.125 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:05:08.550 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:08.536 7ffa57ceb700 1 Processor -- start 2021-02-10T11:05:08.550 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:08.536 7ffa57ceb700 1 -- start start 2021-02-10T11:05:08.550 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:08.536 7ffa57ceb700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ffa5007e6a0 0x7ffa500d1790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:08.550 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:08.536 7ffa57ceb700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa5007ed80 con 0x7ffa500e5470 2021-02-10T11:05:08.551 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:08.536 7ffa57ceb700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa5007eb10 con 0x7ffa500e5070 2021-02-10T11:05:08.551 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:08.536 7ffa57ceb700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa501b4080 con 0x7ffa5007e6a0 2021-02-10T11:05:08.551 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:08.536 7ffa55306700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7ffa500e5470 0x7ffa5007f300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:42682/0 (socket says 172.21.2.102:42682) 2021-02-10T11:05:08.551 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:08.536 7ffa55306700 1 -- 172.21.2.102:0/2010316791 learned_addr learned my addr 172.21.2.102:0/2010316791 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:05:08.551 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:08.536 7ffa56308700 1 --2- 172.21.2.102:0/2010316791 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ffa5007e6a0 0x7ffa500d1790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:08.551 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:08.537 7ffa54b05700 1 -- 172.21.2.102:0/2010316791 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3624156276 0 0) 0x7ffa4c0010b0 con 0x7ffa500e5470 2021-02-10T11:05:08.551 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:08.537 7ffa54b05700 1 -- 172.21.2.102:0/2010316791 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa3c002fe0 con 0x7ffa500e5470 2021-02-10T11:05:08.551 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:08.537 7ffa56308700 1 -- 172.21.2.102:0/2010316791 >> v1:172.21.2.113:6789/0 conn(0x7ffa500e5070 legacy=0x7ffa501b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:08.551 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:08.537 7ffa56308700 1 -- 172.21.2.102:0/2010316791 >> v1:172.21.2.102:6789/0 conn(0x7ffa500e5470 legacy=0x7ffa5007f300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:08.551 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:08.537 7ffa56308700 1 -- 172.21.2.102:0/2010316791 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa500d1db0 con 0x7ffa5007e6a0 2021-02-10T11:05:08.551 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:08.537 7ffa56308700 1 --2- 172.21.2.102:0/2010316791 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ffa5007e6a0 0x7ffa500d1790 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7ffa48006e50 tx=0x7ffa48009e70).ready entity=mon.2 client_cookie=14b721c733393112 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:05:08.551 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:08.537 7ffa54b05700 1 -- 172.21.2.102:0/2010316791 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ffa480066f0 con 0x7ffa5007e6a0 2021-02-10T11:05:08.552 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:08.537 7ffa54b05700 1 -- 172.21.2.102:0/2010316791 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa48003030 con 0x7ffa5007e6a0 2021-02-10T11:05:08.552 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:08.537 7ffa54b05700 1 -- 172.21.2.102:0/2010316791 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ffa4800b700 con 0x7ffa5007e6a0 2021-02-10T11:05:08.552 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:08.537 7ffa57ceb700 1 -- 172.21.2.102:0/2010316791 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ffa5007e6a0 msgr2=0x7ffa500d1790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:08.552 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:08.537 7ffa57ceb700 1 --2- 172.21.2.102:0/2010316791 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ffa5007e6a0 0x7ffa500d1790 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7ffa48006e50 tx=0x7ffa48009e70).stop 2021-02-10T11:05:08.552 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:08.537 7ffa57ceb700 1 -- 172.21.2.102:0/2010316791 shutdown_connections 2021-02-10T11:05:08.552 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:08.537 7ffa57ceb700 1 --2- 172.21.2.102:0/2010316791 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ffa5007e6a0 0x7ffa500d1790 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:08.552 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:08.537 7ffa57ceb700 1 -- 172.21.2.102:0/2010316791 >> 172.21.2.102:0/2010316791 conn(0x7ffa500ce690 msgr2=0x7ffa500e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:05:08.552 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:08.538 7ffa57ceb700 1 -- 172.21.2.102:0/2010316791 shutdown_connections 2021-02-10T11:05:08.552 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:08.538 7ffa57ceb700 1 -- 172.21.2.102:0/2010316791 wait complete. 2021-02-10T11:05:08.553 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:08.552 7ffa57ceb700 1 Processor -- start 2021-02-10T11:05:08.554 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:08.552 7ffa57ceb700 1 -- start start 2021-02-10T11:05:08.554 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:08.552 7ffa57ceb700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ffa501a2a90 0x7ffa501aa0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:08.554 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:08.552 7ffa57ceb700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ffa501aa700 0x7ffa501a3a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:08.554 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:08.552 7ffa57ceb700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ffa500a55e0 0x7ffa501a4040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:08.554 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:08.552 7ffa57ceb700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa501a34a0 con 0x7ffa500a55e0 2021-02-10T11:05:08.554 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:08.552 7ffa57ceb700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa501a2e90 con 0x7ffa501a2a90 2021-02-10T11:05:08.554 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:08.552 7ffa57ceb700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa501a30b0 con 0x7ffa501aa700 2021-02-10T11:05:08.554 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:08.552 7ffa56308700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ffa500a55e0 0x7ffa501a4040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:08.554 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:08.552 7ffa56308700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ffa500a55e0 0x7ffa501a4040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:50464/0 (socket says 172.21.2.102:50464) 2021-02-10T11:05:08.554 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:08.552 7ffa56308700 1 -- 172.21.2.102:0/2718675072 learned_addr learned my addr 172.21.2.102:0/2718675072 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:05:08.554 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:08.553 7ffa56308700 1 -- 172.21.2.102:0/2718675072 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ffa501aa700 msgr2=0x7ffa501a3a20 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:05:08.555 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:08.553 7ffa56308700 1 --2- 172.21.2.102:0/2718675072 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ffa501aa700 0x7ffa501a3a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:05:08.555 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:08.553 7ffa56308700 1 -- 172.21.2.102:0/2718675072 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ffa501a2a90 msgr2=0x7ffa501aa0e0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:05:08.555 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:08.553 7ffa56308700 1 --2- 172.21.2.102:0/2718675072 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ffa501a2a90 0x7ffa501aa0e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:05:08.555 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:08.553 7ffa56308700 1 -- 172.21.2.102:0/2718675072 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa501a4660 con 0x7ffa500a55e0 2021-02-10T11:05:08.555 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:08.553 7ffa56308700 1 --2- 172.21.2.102:0/2718675072 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ffa500a55e0 0x7ffa501a4040 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0x7ffa480099f0 tx=0x7ffa4800bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:05:08.555 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:08.553 7ffa427fc700 1 -- 172.21.2.102:0/2718675072 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ffa480091b0 con 0x7ffa500a55e0 2021-02-10T11:05:08.555 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:08.553 7ffa427fc700 1 -- 172.21.2.102:0/2718675072 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa48006880 con 0x7ffa500a55e0 2021-02-10T11:05:08.555 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:08.553 7ffa427fc700 1 -- 172.21.2.102:0/2718675072 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ffa4800c2f0 con 0x7ffa500a55e0 2021-02-10T11:05:08.555 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:08.553 7ffa57ceb700 1 -- 172.21.2.102:0/2718675072 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa501a4660 con 0x7ffa500a55e0 2021-02-10T11:05:08.555 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:08.553 7ffa57ceb700 1 -- 172.21.2.102:0/2718675072 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa501a4660 con 0x7ffa500a55e0 2021-02-10T11:05:08.555 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:08.554 7ffa427fc700 1 -- 172.21.2.102:0/2718675072 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ffa48009170 con 0x7ffa500a55e0 2021-02-10T11:05:08.555 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:08.554 7ffa427fc700 1 --2- 172.21.2.102:0/2718675072 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ffa380186c0 0x7ffa3801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:08.555 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:08.554 7ffa427fc700 1 -- 172.21.2.102:0/2718675072 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 4753+0+0 (secure 0 0 0) 0x7ffa480323f0 con 0x7ffa500a55e0 2021-02-10T11:05:08.556 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:08.554 7ffa55b07700 1 --2- 172.21.2.102:0/2718675072 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ffa380186c0 0x7ffa3801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:08.556 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:08.554 7ffa55b07700 1 --2- 172.21.2.102:0/2718675072 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ffa380186c0 0x7ffa3801abc0 crc :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:05:08.585 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:08.586 7ffa57ceb700 1 -- 172.21.2.102:0/2718675072 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa50083da0 con 0x7ffa500a55e0 2021-02-10T11:05:08.587 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:08.589 7ffa427fc700 1 -- 172.21.2.102:0/2718675072 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ffa4801dd80 con 0x7ffa500a55e0 2021-02-10T11:05:08.715 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:08.716 7ffa57ceb700 1 -- 172.21.2.102:0/2718675072 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ffa50083da0 con 0x7ffa500a55e0 2021-02-10T11:05:08.715 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:08.717 7ffa427fc700 1 -- 172.21.2.102:0/2718675072 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) v1 ==== 74+0+11144 (secure 0 0 0) 0x7ffa4801dd80 con 0x7ffa500a55e0 2021-02-10T11:05:08.716 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:05:08.716 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":26,"fsid":"68db30ea-dbce-4aef-a6df-f01c6fea0513","created":"2021-02-10 11:03:06.331557","modified":"2021-02-10 11:05:07.292486","last_up_change":"2021-02-10 11:03:32.690562","last_in_change":"2021-02-10 11:03:32.690562","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:03:37.322519","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"be691b4d-a039-408a-af80-9d53f02b6f1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6818","nonce":36088},{"type":"v1","addr":"172.21.2.102:6820","nonce":36088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6822","nonce":36088},{"type":"v1","addr":"172.21.2.102:6824","nonce":36088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6830","nonce":36088},{"type":"v1","addr":"172.21.2.102:6832","nonce":36088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6826","nonce":36088},{"type":"v1","addr":"172.21.2.102:6828","nonce":36088}]},"public_addr":"172.21.2.102:6820/36088","cluster_addr":"172.21.2.102:6824/36088","heartbeat_back_addr":"172.21.2.102:6832/36088","heartbeat_front_addr":"172.21.2.102:6828/36088","state":["exists","up"]},{"osd":1,"uuid":"64e3ccc8-9c69-4d7c-865e-af8e83822635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6819","nonce":36087},{"type":"v1","addr":"172.21.2.102:6821","nonce":36087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6823","nonce":36087},{"type":"v1","addr":"172.21.2.102:6825","nonce":36087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6831","nonce":36087},{"type":"v1","addr":"172.21.2.102:6833","nonce":36087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6827","nonce":36087},{"type":"v1","addr":"172.21.2.102:6829","nonce":36087}]},"public_addr":"172.21.2.102:6821/36087","cluster_addr":"172.21.2.102:6825/36087","heartbeat_back_addr":"172.21.2.102:6833/36087","heartbeat_front_addr":"172.21.2.102:6829/36087","state":["exists","up"]},{"osd":2,"uuid":"153700af-747f-481f-bdf4-d6072d3a842f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6810","nonce":36089},{"type":"v1","addr":"172.21.2.102:6811","nonce":36089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6812","nonce":36089},{"type":"v1","addr":"172.21.2.102:6813","nonce":36089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6816","nonce":36089},{"type":"v1","addr":"172.21.2.102:6817","nonce":36089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6814","nonce":36089},{"type":"v1","addr":"172.21.2.102:6815","nonce":36089}]},"public_addr":"172.21.2.102:6811/36089","cluster_addr":"172.21.2.102:6813/36089","heartbeat_back_addr":"172.21.2.102:6817/36089","heartbeat_front_addr":"172.21.2.102:6815/36089","state":["exists","up"]},{"osd":3,"uuid":"cbc678f4-83a9-45e6-8830-2f7c232b1ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6802","nonce":36090},{"type":"v1","addr":"172.21.2.102:6803","nonce":36090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6804","nonce":36090},{"type":"v1","addr":"172.21.2.102:6805","nonce":36090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6808","nonce":36090},{"type":"v1","addr":"172.21.2.102:6809","nonce":36090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6806","nonce":36090},{"type":"v1","addr":"172.21.2.102:6807","nonce":36090}]},"public_addr":"172.21.2.102:6803/36090","cluster_addr":"172.21.2.102:6805/36090","heartbeat_back_addr":"172.21.2.102:6809/36090","heartbeat_front_addr":"172.21.2.102:6807/36090","state":["exists","up"]},{"osd":4,"uuid":"c01e4008-b258-4f10-afd2-9e7f37214351","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6808","nonce":36368},{"type":"v1","addr":"172.21.2.113:6811","nonce":36368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6814","nonce":36368},{"type":"v1","addr":"172.21.2.113:6818","nonce":36368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6826","nonce":36368},{"type":"v1","addr":"172.21.2.113:6828","nonce":36368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6820","nonce":36368},{"type":"v1","addr":"172.21.2.113:6824","nonce":36368}]},"public_addr":"172.21.2.113:6811/36368","cluster_addr":"172.21.2.113:6818/36368","heartbeat_back_addr":"172.21.2.113:6828/36368","heartbeat_front_addr":"172.21.2.113:6824/36368","state":["exists","up"]},{"osd":5,"uuid":"551f6007-daf8-4a98-8ee0-c2b16bcccb5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6802","nonce":36366},{"type":"v1","addr":"172.21.2.113:6805","nonce":36366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6807","nonce":36366},{"type":"v1","addr":"172.21.2.113:6810","nonce":36366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6819","nonce":36366},{"type":"v1","addr":"172.21.2.113:6822","nonce":36366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6812","nonce":36366},{"type":"v1","addr":"172.21.2.113:6815","nonce":36366}]},"public_addr":"172.21.2.113:6805/36366","cluster_addr":"172.21.2.113:6810/36366","heartbeat_back_addr":"172.21.2.113:6822/36366","heartbeat_front_addr":"172.21.2.113:6815/36366","state":["exists","up"]},{"osd":6,"uuid":"bf67c040-0e50-48be-b102-c3943bc9ebef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6800","nonce":36367},{"type":"v1","addr":"172.21.2.113:6801","nonce":36367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6803","nonce":36367},{"type":"v1","addr":"172.21.2.113:6804","nonce":36367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6813","nonce":36367},{"type":"v1","addr":"172.21.2.113:6816","nonce":36367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6806","nonce":36367},{"type":"v1","addr":"172.21.2.113:6809","nonce":36367}]},"public_addr":"172.21.2.113:6801/36367","cluster_addr":"172.21.2.113:6804/36367","heartbeat_back_addr":"172.21.2.113:6816/36367","heartbeat_front_addr":"172.21.2.113:6809/36367","state":["exists","up"]},{"osd":7,"uuid":"f5461f80-018d-428f-be89-2eb2bcbe7bb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6817","nonce":36365},{"type":"v1","addr":"172.21.2.113:6821","nonce":36365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6823","nonce":36365},{"type":"v1","addr":"172.21.2.113:6825","nonce":36365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6830","nonce":36365},{"type":"v1","addr":"172.21.2.113:6831","nonce":36365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6827","nonce":36365},{"type":"v1","addr":"172.21.2.113:6829","nonce":36365}]},"public_addr":"172.21.2.113:6821/36365","cluster_addr":"172.21.2.113:6825/36365","heartbeat_back_addr":"172.21.2.113:6831/36365","heartbeat_front_addr":"172.21.2.113:6829/36365","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:05:08.717 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:08.718 7ffa57ceb700 1 -- 172.21.2.102:0/2718675072 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ffa380186c0 msgr2=0x7ffa3801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:08.717 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:08.718 7ffa57ceb700 1 --2- 172.21.2.102:0/2718675072 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ffa380186c0 0x7ffa3801abc0 crc :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:08.718 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:08.718 7ffa57ceb700 1 -- 172.21.2.102:0/2718675072 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ffa500a55e0 msgr2=0x7ffa501a4040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:08.718 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:08.718 7ffa57ceb700 1 --2- 172.21.2.102:0/2718675072 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ffa500a55e0 0x7ffa501a4040 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0x7ffa480099f0 tx=0x7ffa4800bd10).stop 2021-02-10T11:05:08.718 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:08.719 7ffa57ceb700 1 -- 172.21.2.102:0/2718675072 shutdown_connections 2021-02-10T11:05:08.718 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:08.719 7ffa57ceb700 1 --2- 172.21.2.102:0/2718675072 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ffa500a55e0 0x7ffa501a4040 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:08.718 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:08.719 7ffa57ceb700 1 --2- 172.21.2.102:0/2718675072 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ffa501aa700 0x7ffa501a3a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:05:08.718 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:08.719 7ffa57ceb700 1 --2- 172.21.2.102:0/2718675072 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ffa380186c0 0x7ffa3801abc0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:08.718 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:08.719 7ffa57ceb700 1 --2- 172.21.2.102:0/2718675072 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ffa501a2a90 0x7ffa501aa0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:05:08.718 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:08.719 7ffa57ceb700 1 -- 172.21.2.102:0/2718675072 >> 172.21.2.102:0/2718675072 conn(0x7ffa5008fe10 msgr2=0x7ffa5019c020 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:05:08.718 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:08.719 7ffa57ceb700 1 -- 172.21.2.102:0/2718675072 shutdown_connections 2021-02-10T11:05:08.718 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:08.719 7ffa57ceb700 1 -- 172.21.2.102:0/2718675072 wait complete. 2021-02-10T11:05:08.726 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:03:37.322519', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2021-02-10T11:05:08.726 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:05:09.155 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:09.139 7f8464862700 1 Processor -- start 2021-02-10T11:05:09.155 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:09.139 7f8464862700 1 -- start start 2021-02-10T11:05:09.155 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:09.140 7f8464862700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f84601b0470 0x7f84601b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:09.155 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:09.140 7f8464862700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f84600f20e0 0x7f84600f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:09.155 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:09.140 7f8464862700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84600f0b50 con 0x7f84600ef610 2021-02-10T11:05:09.155 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:09.140 7f8464862700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84600f0820 con 0x7f84601b0470 2021-02-10T11:05:09.155 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:09.140 7f8464862700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84600f24e0 con 0x7f84600f20e0 2021-02-10T11:05:09.155 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:09.140 7f845e61c700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f84601b0470 0x7f84601b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:09.155 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:09.140 7f845e61c700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f84601b0470 0x7f84601b0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.113:3300/0 says I am v2:172.21.2.102:53900/0 (socket says 172.21.2.102:53900) 2021-02-10T11:05:09.155 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:09.140 7f845e61c700 1 -- 172.21.2.102:0/398148104 learned_addr learned my addr 172.21.2.102:0/398148104 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:05:09.155 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:09.140 7f845ee1d700 1 --2- 172.21.2.102:0/398148104 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f84600f20e0 0x7f84600f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:09.156 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:09.141 7f845ee1d700 1 -- 172.21.2.102:0/398148104 >> v2:172.21.2.113:3300/0 conn(0x7f84601b0470 msgr2=0x7f84601b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:05:09.156 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:09.141 7f845ee1d700 1 --2- 172.21.2.102:0/398148104 >> v2:172.21.2.113:3300/0 conn(0x7f84601b0470 0x7f84601b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:09.156 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:09.141 7f845ee1d700 1 -- 172.21.2.102:0/398148104 >> v1:172.21.2.113:6789/0 conn(0x7f84600ef610 legacy=0x7f84600efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:09.156 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:09.141 7f845ee1d700 1 -- 172.21.2.102:0/398148104 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84600e52e0 con 0x7f84600f20e0 2021-02-10T11:05:09.156 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:09.141 7f845ee1d700 1 --2- 172.21.2.102:0/398148104 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f84600f20e0 0x7f84600f0200 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f8458006e50 tx=0x7f8458009e70).ready entity=mon.2 client_cookie=8b300f7c53d389ce server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:05:09.156 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:09.141 7f845d61a700 1 -- 172.21.2.102:0/398148104 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f845800b510 con 0x7f84600f20e0 2021-02-10T11:05:09.156 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:09.141 7f845d61a700 1 -- 172.21.2.102:0/398148104 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8458006790 con 0x7f84600f20e0 2021-02-10T11:05:09.156 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:09.141 7f845d61a700 1 -- 172.21.2.102:0/398148104 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f845800b940 con 0x7f84600f20e0 2021-02-10T11:05:09.156 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:09.141 7f8464862700 1 -- 172.21.2.102:0/398148104 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f84600f20e0 msgr2=0x7f84600f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:09.156 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:09.141 7f8464862700 1 --2- 172.21.2.102:0/398148104 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f84600f20e0 0x7f84600f0200 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f8458006e50 tx=0x7f8458009e70).stop 2021-02-10T11:05:09.156 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:09.142 7f8464862700 1 -- 172.21.2.102:0/398148104 shutdown_connections 2021-02-10T11:05:09.156 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:09.142 7f8464862700 1 --2- 172.21.2.102:0/398148104 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f84600f20e0 0x7f84600f0200 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:09.156 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:09.142 7f8464862700 1 --2- 172.21.2.102:0/398148104 >> v2:172.21.2.113:3300/0 conn(0x7f84601b0470 0x7f84601b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:09.157 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:09.142 7f8464862700 1 -- 172.21.2.102:0/398148104 >> 172.21.2.102:0/398148104 conn(0x7f84600ce3e0 msgr2=0x7f84600e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:05:09.157 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:09.142 7f8464862700 1 -- 172.21.2.102:0/398148104 shutdown_connections 2021-02-10T11:05:09.157 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:09.142 7f8464862700 1 -- 172.21.2.102:0/398148104 wait complete. 2021-02-10T11:05:09.158 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:09.156 7f8464862700 1 Processor -- start 2021-02-10T11:05:09.158 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:09.156 7f8464862700 1 -- start start 2021-02-10T11:05:09.158 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:09.157 7f8464862700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f84600a0420 0x7f84600a0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:09.158 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:09.157 7f8464862700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8460189d00 0x7f846018a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:09.158 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:09.157 7f8464862700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f846018a720 0x7f84600b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:09.159 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:09.157 7f8464862700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8460189780 con 0x7f846018a720 2021-02-10T11:05:09.159 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:09.157 7f8464862700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f846018ab20 con 0x7f8460189d00 2021-02-10T11:05:09.159 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:09.157 7f8464862700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84600a1140 con 0x7f84600a0420 2021-02-10T11:05:09.159 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:09.157 7f845ee1d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f846018a720 0x7f84600b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:09.159 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:09.157 7f845ee1d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f846018a720 0x7f84600b7be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:50480/0 (socket says 172.21.2.102:50480) 2021-02-10T11:05:09.159 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:09.157 7f845de1b700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8460189d00 0x7f846018a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:09.159 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:09.157 7f845ee1d700 1 -- 172.21.2.102:0/3920847915 learned_addr learned my addr 172.21.2.102:0/3920847915 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:05:09.159 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:09.157 7f845e61c700 1 --2- 172.21.2.102:0/3920847915 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f84600a0420 0x7f84600a0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:09.159 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:09.158 7f845ee1d700 1 -- 172.21.2.102:0/3920847915 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f84600a0420 msgr2=0x7f84600a0820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:09.159 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:09.158 7f845ee1d700 1 --2- 172.21.2.102:0/3920847915 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f84600a0420 0x7f84600a0820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:09.159 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:09.158 7f845ee1d700 1 -- 172.21.2.102:0/3920847915 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8460189d00 msgr2=0x7f846018a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:09.159 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:09.158 7f845ee1d700 1 --2- 172.21.2.102:0/3920847915 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8460189d00 0x7f846018a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:09.159 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:09.158 7f845ee1d700 1 -- 172.21.2.102:0/3920847915 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84600b8200 con 0x7f846018a720 2021-02-10T11:05:09.160 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:09.158 7f845de1b700 1 --2- 172.21.2.102:0/3920847915 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8460189d00 0x7f846018a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:05:09.160 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:09.158 7f845ee1d700 1 --2- 172.21.2.102:0/3920847915 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f846018a720 0x7f84600b7be0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 rx=0x7f8458006c50 tx=0x7f845800bd20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:05:09.160 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:09.158 7f8456ffd700 1 -- 172.21.2.102:0/3920847915 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f84580093d0 con 0x7f846018a720 2021-02-10T11:05:09.160 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:09.158 7f8456ffd700 1 -- 172.21.2.102:0/3920847915 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f845800be60 con 0x7f846018a720 2021-02-10T11:05:09.160 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:09.158 7f8456ffd700 1 -- 172.21.2.102:0/3920847915 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f845800c4e0 con 0x7f846018a720 2021-02-10T11:05:09.160 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:09.158 7f8464862700 1 -- 172.21.2.102:0/3920847915 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84600b8200 con 0x7f846018a720 2021-02-10T11:05:09.160 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:09.158 7f8464862700 1 -- 172.21.2.102:0/3920847915 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84600b8200 con 0x7f846018a720 2021-02-10T11:05:09.160 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:09.159 7f8456ffd700 1 -- 172.21.2.102:0/3920847915 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f84580093d0 con 0x7f846018a720 2021-02-10T11:05:09.160 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:09.159 7f8456ffd700 1 --2- 172.21.2.102:0/3920847915 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f84440186c0 0x7f844401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:09.160 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:09.159 7f8456ffd700 1 -- 172.21.2.102:0/3920847915 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 4753+0+0 (secure 0 0 0) 0x7f84580325c0 con 0x7f846018a720 2021-02-10T11:05:09.160 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:09.159 7f845e61c700 1 --2- 172.21.2.102:0/3920847915 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f84440186c0 0x7f844401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:09.160 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:09.159 7f845e61c700 1 --2- 172.21.2.102:0/3920847915 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f84440186c0 0x7f844401abc0 crc :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:05:09.189 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:09.190 7f8464862700 1 -- 172.21.2.102:0/3920847915 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8460105ae0 con 0x7f846018a720 2021-02-10T11:05:09.191 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:09.193 7f8456ffd700 1 -- 172.21.2.102:0/3920847915 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f8458024f30 con 0x7f846018a720 2021-02-10T11:05:09.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:09.319 7f8464862700 1 -- 172.21.2.102:0/3920847915 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f84600f0b50 con 0x7f846018a720 2021-02-10T11:05:09.318 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:09.319 7f8456ffd700 1 -- 172.21.2.102:0/3920847915 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v26) v1 ==== 92+0+10 (secure 0 0 0) 0x7f8458024f30 con 0x7f846018a720 2021-02-10T11:05:09.318 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:05:09.320 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:09.321 7f8464862700 1 -- 172.21.2.102:0/3920847915 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f84440186c0 msgr2=0x7f844401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:09.320 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:09.321 7f8464862700 1 --2- 172.21.2.102:0/3920847915 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f84440186c0 0x7f844401abc0 crc :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:09.320 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:09.321 7f8464862700 1 -- 172.21.2.102:0/3920847915 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f846018a720 msgr2=0x7f84600b7be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:09.320 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:09.321 7f8464862700 1 --2- 172.21.2.102:0/3920847915 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f846018a720 0x7f84600b7be0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 rx=0x7f8458006c50 tx=0x7f845800bd20).stop 2021-02-10T11:05:09.320 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:09.321 7f8464862700 1 -- 172.21.2.102:0/3920847915 shutdown_connections 2021-02-10T11:05:09.320 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:09.321 7f8464862700 1 --2- 172.21.2.102:0/3920847915 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f846018a720 0x7f84600b7be0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:09.320 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:09.321 7f8464862700 1 --2- 172.21.2.102:0/3920847915 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f84440186c0 0x7f844401abc0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:09.320 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:09.321 7f8464862700 1 --2- 172.21.2.102:0/3920847915 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8460189d00 0x7f846018a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:09.321 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:09.321 7f8464862700 1 --2- 172.21.2.102:0/3920847915 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f84600a0420 0x7f84600a0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:09.321 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:09.321 7f8464862700 1 -- 172.21.2.102:0/3920847915 >> 172.21.2.102:0/3920847915 conn(0x7f8460094e10 msgr2=0x7f84600a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:05:09.321 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:09.321 7f8464862700 1 -- 172.21.2.102:0/3920847915 shutdown_connections 2021-02-10T11:05:09.321 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:09.321 7f8464862700 1 -- 172.21.2.102:0/3920847915 wait complete. 2021-02-10T11:05:09.329 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --format=json-pretty osd lspools 2021-02-10T11:05:09.754 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:09.739 7f7c3545c700 1 Processor -- start 2021-02-10T11:05:09.754 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:09.739 7f7c3545c700 1 -- start start 2021-02-10T11:05:09.754 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:09.740 7f7c3545c700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f7c300e5470 0x7f7c3007e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:09.754 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:09.740 7f7c3545c700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c3007ed80 con 0x7f7c300e5070 2021-02-10T11:05:09.754 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:09.740 7f7c3545c700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c301b4010 con 0x7f7c3007f300 2021-02-10T11:05:09.754 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:09.740 7f7c3545c700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c300d1790 con 0x7f7c300e5470 2021-02-10T11:05:09.754 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:09.740 7f7c2e87c700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f7c300e5470 0x7f7c3007e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:09.754 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:09.740 7f7c2f07d700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f7c300e5070 0x7f7c301b3820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:42708/0 (socket says 172.21.2.102:42708) 2021-02-10T11:05:09.755 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:09.740 7f7c2f07d700 1 -- 172.21.2.102:0/1365769528 learned_addr learned my addr 172.21.2.102:0/1365769528 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:05:09.755 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:09.740 7f7c2e87c700 1 -- 172.21.2.102:0/1365769528 >> v1:172.21.2.113:6789/0 conn(0x7f7c3007f300 legacy=0x7f7c3007f700 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:05:09.755 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:09.740 7f7c2e87c700 1 -- 172.21.2.102:0/1365769528 >> v1:172.21.2.102:6789/0 conn(0x7f7c300e5070 legacy=0x7f7c301b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:09.755 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:09.740 7f7c2e87c700 1 -- 172.21.2.102:0/1365769528 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c300d19b0 con 0x7f7c300e5470 2021-02-10T11:05:09.755 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:09.740 7f7c2e87c700 1 --2- 172.21.2.102:0/1365769528 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7c300e5470 0x7f7c3007e6a0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 rx=0x7f7c180069d0 tx=0x7f7c18009e40).ready entity=mon.0 client_cookie=6f5d5b8876d5096f server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:05:09.755 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:09.741 7f7c2e07b700 1 -- 172.21.2.102:0/1365769528 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7c1800b510 con 0x7f7c300e5470 2021-02-10T11:05:09.755 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:09.741 7f7c2e07b700 1 -- 172.21.2.102:0/1365769528 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c1800b790 con 0x7f7c300e5470 2021-02-10T11:05:09.755 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:09.741 7f7c2e07b700 1 -- 172.21.2.102:0/1365769528 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7c1800bcb0 con 0x7f7c300e5470 2021-02-10T11:05:09.755 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:09.741 7f7c3545c700 1 -- 172.21.2.102:0/1365769528 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7c300e5470 msgr2=0x7f7c3007e6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:09.755 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:09.741 7f7c3545c700 1 --2- 172.21.2.102:0/1365769528 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7c300e5470 0x7f7c3007e6a0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 rx=0x7f7c180069d0 tx=0x7f7c18009e40).stop 2021-02-10T11:05:09.755 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:09.741 7f7c3545c700 1 -- 172.21.2.102:0/1365769528 shutdown_connections 2021-02-10T11:05:09.755 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:09.741 7f7c3545c700 1 --2- 172.21.2.102:0/1365769528 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7c300e5470 0x7f7c3007e6a0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:09.755 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:09.741 7f7c3545c700 1 -- 172.21.2.102:0/1365769528 >> 172.21.2.102:0/1365769528 conn(0x7f7c300ce690 msgr2=0x7f7c300e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:05:09.756 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:09.741 7f7c3545c700 1 -- 172.21.2.102:0/1365769528 shutdown_connections 2021-02-10T11:05:09.756 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:09.741 7f7c3545c700 1 -- 172.21.2.102:0/1365769528 wait complete. 2021-02-10T11:05:09.757 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:09.756 7f7c3545c700 1 Processor -- start 2021-02-10T11:05:09.757 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:09.756 7f7c3545c700 1 -- start start 2021-02-10T11:05:09.757 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:09.756 7f7c3545c700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7c301c55a0 0x7f7c301c59a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:09.757 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:09.756 7f7c3545c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7c300bfdc0 0x7f7c300c01c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:09.757 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:09.756 7f7c3545c700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7c30099900 0x7f7c30099d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:09.757 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:09.756 7f7c3545c700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c300c09f0 con 0x7f7c300bfdc0 2021-02-10T11:05:09.758 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:09.756 7f7c3545c700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c3009a320 con 0x7f7c30099900 2021-02-10T11:05:09.758 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:09.756 7f7c3545c700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c3009a540 con 0x7f7c301c55a0 2021-02-10T11:05:09.758 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:09.756 7f7c2e87c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7c300bfdc0 0x7f7c300c01c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:09.758 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:09.756 7f7c2e87c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7c300bfdc0 0x7f7c300c01c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:50488/0 (socket says 172.21.2.102:50488) 2021-02-10T11:05:09.758 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:09.756 7f7c2e87c700 1 -- 172.21.2.102:0/4159546787 learned_addr learned my addr 172.21.2.102:0/4159546787 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:05:09.758 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:09.757 7f7c2e87c700 1 -- 172.21.2.102:0/4159546787 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7c301c55a0 msgr2=0x7f7c301c59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:09.758 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:09.757 7f7c2e87c700 1 --2- 172.21.2.102:0/4159546787 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7c301c55a0 0x7f7c301c59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:05:09.758 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:09.757 7f7c2e87c700 1 -- 172.21.2.102:0/4159546787 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7c30099900 msgr2=0x7f7c30099d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:09.758 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:09.757 7f7c2e87c700 1 --2- 172.21.2.102:0/4159546787 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7c30099900 0x7f7c30099d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:05:09.758 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:09.757 7f7c2e87c700 1 -- 172.21.2.102:0/4159546787 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c3009a760 con 0x7f7c300bfdc0 2021-02-10T11:05:09.758 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:09.757 7f7c2e87c700 1 --2- 172.21.2.102:0/4159546787 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7c300bfdc0 0x7f7c300c01c0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 rx=0x7f7c18009f40 tx=0x7f7c1800bd50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:05:09.758 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:09.757 7f7c27fff700 1 -- 172.21.2.102:0/4159546787 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7c18006240 con 0x7f7c300bfdc0 2021-02-10T11:05:09.758 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:09.757 7f7c27fff700 1 -- 172.21.2.102:0/4159546787 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c180065c0 con 0x7f7c300bfdc0 2021-02-10T11:05:09.759 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:09.757 7f7c3545c700 1 -- 172.21.2.102:0/4159546787 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c3009a760 con 0x7f7c300bfdc0 2021-02-10T11:05:09.759 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:09.757 7f7c27fff700 1 -- 172.21.2.102:0/4159546787 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7c180058e0 con 0x7f7c300bfdc0 2021-02-10T11:05:09.759 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:09.757 7f7c3545c700 1 -- 172.21.2.102:0/4159546787 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c3009a760 con 0x7f7c300bfdc0 2021-02-10T11:05:09.759 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:09.758 7f7c27fff700 1 -- 172.21.2.102:0/4159546787 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f7c18006ca0 con 0x7f7c300bfdc0 2021-02-10T11:05:09.759 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:09.758 7f7c27fff700 1 --2- 172.21.2.102:0/4159546787 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7c14018730 0x7f7c1401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:09.759 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:09.758 7f7c27fff700 1 -- 172.21.2.102:0/4159546787 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 4753+0+0 (secure 0 0 0) 0x7f7c180326c0 con 0x7f7c300bfdc0 2021-02-10T11:05:09.759 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:09.758 7f7c2f07d700 1 --2- 172.21.2.102:0/4159546787 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7c14018730 0x7f7c1401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:09.759 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:09.758 7f7c2f07d700 1 --2- 172.21.2.102:0/4159546787 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7c14018730 0x7f7c1401ac10 crc :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:05:09.788 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:09.790 7f7c3545c700 1 -- 172.21.2.102:0/4159546787 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c300ce690 con 0x7f7c300bfdc0 2021-02-10T11:05:09.791 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:09.792 7f7c27fff700 1 -- 172.21.2.102:0/4159546787 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f7c18000cc0 con 0x7f7c300bfdc0 2021-02-10T11:05:09.915 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:09.917 7f7c3545c700 1 -- 172.21.2.102:0/4159546787 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f7c300ce690 con 0x7f7c300bfdc0 2021-02-10T11:05:09.916 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:09.917 7f7c27fff700 1 -- 172.21.2.102:0/4159546787 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v26) v1 ==== 84+0+64 (secure 0 0 0) 0x7f7c18006ca0 con 0x7f7c300bfdc0 2021-02-10T11:05:09.916 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:05:09.917 INFO:teuthology.orchestra.run.gibba002.stdout:[ 2021-02-10T11:05:09.917 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:05:09.917 INFO:teuthology.orchestra.run.gibba002.stdout: "poolnum": 1, 2021-02-10T11:05:09.917 INFO:teuthology.orchestra.run.gibba002.stdout: "poolname": "rbd" 2021-02-10T11:05:09.917 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:05:09.917 INFO:teuthology.orchestra.run.gibba002.stdout:] 2021-02-10T11:05:09.918 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:09.919 7f7c3545c700 1 -- 172.21.2.102:0/4159546787 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7c14018730 msgr2=0x7f7c1401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:09.918 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:09.919 7f7c3545c700 1 --2- 172.21.2.102:0/4159546787 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7c14018730 0x7f7c1401ac10 crc :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:09.919 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:09.919 7f7c3545c700 1 -- 172.21.2.102:0/4159546787 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7c300bfdc0 msgr2=0x7f7c300c01c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:09.919 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:09.919 7f7c3545c700 1 --2- 172.21.2.102:0/4159546787 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7c300bfdc0 0x7f7c300c01c0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 rx=0x7f7c18009f40 tx=0x7f7c1800bd50).stop 2021-02-10T11:05:09.919 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:09.920 7f7c3545c700 1 -- 172.21.2.102:0/4159546787 shutdown_connections 2021-02-10T11:05:09.919 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:09.920 7f7c3545c700 1 --2- 172.21.2.102:0/4159546787 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7c14018730 0x7f7c1401ac10 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:09.919 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:09.920 7f7c3545c700 1 --2- 172.21.2.102:0/4159546787 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7c30099900 0x7f7c30099d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:05:09.919 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:09.920 7f7c3545c700 1 --2- 172.21.2.102:0/4159546787 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7c300bfdc0 0x7f7c300c01c0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:09.919 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:09.920 7f7c3545c700 1 --2- 172.21.2.102:0/4159546787 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7c301c55a0 0x7f7c301c59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:05:09.919 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:09.920 7f7c3545c700 1 -- 172.21.2.102:0/4159546787 >> 172.21.2.102:0/4159546787 conn(0x7f7c300976b0 msgr2=0x7f7c300962b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:05:09.919 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:09.920 7f7c3545c700 1 -- 172.21.2.102:0/4159546787 shutdown_connections 2021-02-10T11:05:09.919 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:09.920 7f7c3545c700 1 -- 172.21.2.102:0/4159546787 wait complete. 2021-02-10T11:05:09.927 INFO:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2021-02-10T11:05:09.927 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool create cephfs_metadata 8 2021-02-10T11:05:10.355 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:10.339 7f1710f96700 1 Processor -- start 2021-02-10T11:05:10.355 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:10.339 7f1710f96700 1 -- start start 2021-02-10T11:05:10.355 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:10.340 7f1710f96700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f170c1b0870 0x7f170c0f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:10.355 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:10.340 7f1710f96700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f170c1b0c70 con 0x7f170c0efe00 2021-02-10T11:05:10.355 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:10.340 7f1710f96700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f170c0f0b50 con 0x7f170c1b0470 2021-02-10T11:05:10.355 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:10.340 7f1710f96700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f170c0e5ea0 con 0x7f170c1b0870 2021-02-10T11:05:10.355 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:10.340 7f170a59c700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f170c1b0870 0x7f170c0f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:10.355 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:10.340 7f170b59e700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f170c0efe00 0x7f170c0f0200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:42722/0 (socket says 172.21.2.102:42722) 2021-02-10T11:05:10.355 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:10.340 7f170b59e700 1 -- 172.21.2.102:0/2835364692 learned_addr learned my addr 172.21.2.102:0/2835364692 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:05:10.356 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:10.340 7f170a59c700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f170c1b0870 0x7f170c0f20e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:50496/0 (socket says 172.21.2.102:50496) 2021-02-10T11:05:10.356 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:10.341 7f170a59c700 1 -- 172.21.2.102:0/2835364692 >> v1:172.21.2.113:6789/0 conn(0x7f170c1b0470 legacy=0x7f170c0ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:05:10.356 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:10.341 7f170a59c700 1 -- 172.21.2.102:0/2835364692 >> v1:172.21.2.102:6789/0 conn(0x7f170c0efe00 legacy=0x7f170c0f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:10.356 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:10.341 7f170a59c700 1 -- 172.21.2.102:0/2835364692 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f170c0e52e0 con 0x7f170c1b0870 2021-02-10T11:05:10.356 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:10.341 7f170a59c700 1 --2- 172.21.2.102:0/2835364692 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f170c1b0870 0x7f170c0f20e0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 rx=0x7f16fc006650 tx=0x7f16fc009a80).ready entity=mon.0 client_cookie=cb594f757b7d798a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:05:10.356 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:10.341 7f1709d9b700 1 -- 172.21.2.102:0/2835364692 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f16fc00c0c0 con 0x7f170c1b0870 2021-02-10T11:05:10.356 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:10.341 7f1709d9b700 1 -- 172.21.2.102:0/2835364692 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16fc00c340 con 0x7f170c1b0870 2021-02-10T11:05:10.356 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:10.341 7f1709d9b700 1 -- 172.21.2.102:0/2835364692 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f16fc00c860 con 0x7f170c1b0870 2021-02-10T11:05:10.356 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:10.341 7f1710f96700 1 -- 172.21.2.102:0/2835364692 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f170c1b0870 msgr2=0x7f170c0f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:10.356 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:10.341 7f1710f96700 1 --2- 172.21.2.102:0/2835364692 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f170c1b0870 0x7f170c0f20e0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 rx=0x7f16fc006650 tx=0x7f16fc009a80).stop 2021-02-10T11:05:10.356 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:10.342 7f1710f96700 1 -- 172.21.2.102:0/2835364692 shutdown_connections 2021-02-10T11:05:10.356 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:10.342 7f1710f96700 1 --2- 172.21.2.102:0/2835364692 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f170c1b0870 0x7f170c0f20e0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:10.356 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:10.342 7f1710f96700 1 -- 172.21.2.102:0/2835364692 >> 172.21.2.102:0/2835364692 conn(0x7f170c0ce3e0 msgr2=0x7f170c0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:05:10.357 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:10.342 7f1710f96700 1 -- 172.21.2.102:0/2835364692 shutdown_connections 2021-02-10T11:05:10.357 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:10.342 7f1710f96700 1 -- 172.21.2.102:0/2835364692 wait complete. 2021-02-10T11:05:10.358 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:10.356 7f1710f96700 1 Processor -- start 2021-02-10T11:05:10.358 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:10.356 7f1710f96700 1 -- start start 2021-02-10T11:05:10.358 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:10.357 7f1710f96700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f170c099d70 0x7f170c081e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:10.358 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:10.357 7f1710f96700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f170c0a9bf0 0x7f170c0a9ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:10.359 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:10.357 7f1710f96700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f170c0aa610 0x7f170c0c1830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:10.359 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:10.357 7f1710f96700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f170c082480 con 0x7f170c0a9bf0 2021-02-10T11:05:10.359 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:10.357 7f1710f96700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f170c0c1e80 con 0x7f170c099d70 2021-02-10T11:05:10.359 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:10.357 7f1710f96700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f170c0c20a0 con 0x7f170c0aa610 2021-02-10T11:05:10.359 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:10.357 7f170a59c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f170c0a9bf0 0x7f170c0a9ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:10.359 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:10.357 7f170a59c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f170c0a9bf0 0x7f170c0a9ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:50502/0 (socket says 172.21.2.102:50502) 2021-02-10T11:05:10.359 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:10.357 7f170a59c700 1 -- 172.21.2.102:0/3375451917 learned_addr learned my addr 172.21.2.102:0/3375451917 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:05:10.359 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:10.357 7f170a59c700 1 -- 172.21.2.102:0/3375451917 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f170c0aa610 msgr2=0x7f170c0c1830 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:05:10.359 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:10.357 7f170a59c700 1 --2- 172.21.2.102:0/3375451917 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f170c0aa610 0x7f170c0c1830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:05:10.359 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:10.357 7f170a59c700 1 -- 172.21.2.102:0/3375451917 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f170c099d70 msgr2=0x7f170c081e60 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:05:10.359 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:10.357 7f170a59c700 1 --2- 172.21.2.102:0/3375451917 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f170c099d70 0x7f170c081e60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:05:10.359 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:10.357 7f170a59c700 1 -- 172.21.2.102:0/3375451917 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f170c0c22c0 con 0x7f170c0a9bf0 2021-02-10T11:05:10.359 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:10.358 7f170a59c700 1 --2- 172.21.2.102:0/3375451917 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f170c0a9bf0 0x7f170c0a9ff0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 rx=0x7f16fc009630 tx=0x7f16fc00b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:05:10.360 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:10.358 7f17037fe700 1 -- 172.21.2.102:0/3375451917 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f16fc0093a0 con 0x7f170c0a9bf0 2021-02-10T11:05:10.360 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:10.358 7f17037fe700 1 -- 172.21.2.102:0/3375451917 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16fc00bd70 con 0x7f170c0a9bf0 2021-02-10T11:05:10.360 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:10.358 7f1710f96700 1 -- 172.21.2.102:0/3375451917 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f170c0c22c0 con 0x7f170c0a9bf0 2021-02-10T11:05:10.360 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:10.358 7f1710f96700 1 -- 172.21.2.102:0/3375451917 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f170c0c2850 con 0x7f170c0a9bf0 2021-02-10T11:05:10.360 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:10.358 7f17037fe700 1 -- 172.21.2.102:0/3375451917 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f16fc0054d0 con 0x7f170c0a9bf0 2021-02-10T11:05:10.360 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:10.359 7f17037fe700 1 -- 172.21.2.102:0/3375451917 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f16fc007000 con 0x7f170c0a9bf0 2021-02-10T11:05:10.360 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:10.359 7f17037fe700 1 --2- 172.21.2.102:0/3375451917 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f16f0018730 0x7f16f001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:10.360 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:10.359 7f17037fe700 1 -- 172.21.2.102:0/3375451917 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 4753+0+0 (secure 0 0 0) 0x7f16fc0324a0 con 0x7f170c0a9bf0 2021-02-10T11:05:10.360 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:10.359 7f170ad9d700 1 --2- 172.21.2.102:0/3375451917 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f16f0018730 0x7f16f001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:10.360 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:10.359 7f170ad9d700 1 --2- 172.21.2.102:0/3375451917 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f16f0018730 0x7f16f001ac10 crc :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:05:10.389 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:10.390 7f1710f96700 1 -- 172.21.2.102:0/3375451917 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f170c07e6b0 con 0x7f170c0a9bf0 2021-02-10T11:05:10.391 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:10.393 7f17037fe700 1 -- 172.21.2.102:0/3375451917 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f16fc009340 con 0x7f170c0a9bf0 2021-02-10T11:05:10.520 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:10.521 7f1710f96700 1 -- 172.21.2.102:0/3375451917 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"} v 0) v1 -- 0x7f170c0ce3e0 con 0x7f170c0a9bf0 2021-02-10T11:05:10.911 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:10.912 7f17037fe700 1 -- 172.21.2.102:0/3375451917 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"}]=0 pool 'cephfs_metadata' created v27) v1 ==== 133+0+0 (secure 0 0 0) 0x7f16fc007000 con 0x7f170c0a9bf0 2021-02-10T11:05:10.935 INFO:teuthology.orchestra.run.gibba002.stderr:pool 'cephfs_metadata' created 2021-02-10T11:05:10.937 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:10.937 7f1710f96700 1 -- 172.21.2.102:0/3375451917 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f16f0018730 msgr2=0x7f16f001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:10.937 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:10.937 7f1710f96700 1 --2- 172.21.2.102:0/3375451917 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f16f0018730 0x7f16f001ac10 crc :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:10.937 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:10.937 7f1710f96700 1 -- 172.21.2.102:0/3375451917 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f170c0a9bf0 msgr2=0x7f170c0a9ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:10.937 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:10.937 7f1710f96700 1 --2- 172.21.2.102:0/3375451917 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f170c0a9bf0 0x7f170c0a9ff0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 rx=0x7f16fc009630 tx=0x7f16fc00b980).stop 2021-02-10T11:05:10.938 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:10.937 7f1710f96700 1 -- 172.21.2.102:0/3375451917 shutdown_connections 2021-02-10T11:05:10.938 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:10.937 7f1710f96700 1 --2- 172.21.2.102:0/3375451917 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f170c0aa610 0x7f170c0c1830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:05:10.938 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:10.937 7f1710f96700 1 --2- 172.21.2.102:0/3375451917 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f170c0a9bf0 0x7f170c0a9ff0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:10.938 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:10.937 7f1710f96700 1 --2- 172.21.2.102:0/3375451917 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f16f0018730 0x7f16f001ac10 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:10.938 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:10.937 7f1710f96700 1 --2- 172.21.2.102:0/3375451917 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f170c099d70 0x7f170c081e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:05:10.939 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:10.937 7f1710f96700 1 -- 172.21.2.102:0/3375451917 >> 172.21.2.102:0/3375451917 conn(0x7f170c088220 msgr2=0x7f170c0a2d50 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:05:10.939 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:10.938 7f1710f96700 1 -- 172.21.2.102:0/3375451917 shutdown_connections 2021-02-10T11:05:10.939 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:10.938 7f1710f96700 1 -- 172.21.2.102:0/3375451917 wait complete. 2021-02-10T11:05:10.945 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool create cephfs_data 8 2021-02-10T11:05:11.375 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:11.360 7f192eb90700 1 Processor -- start 2021-02-10T11:05:11.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:11.360 7f192eb90700 1 -- start start 2021-02-10T11:05:11.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:11.360 7f192eb90700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f19281b0470 0x7f19281b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:11.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:11.360 7f192eb90700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f19280ef610 0x7f19280efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:11.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:11.361 7f192eb90700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19280f0b50 con 0x7f19280f0030 2021-02-10T11:05:11.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:11.361 7f192eb90700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19280e52e0 con 0x7f19280ef610 2021-02-10T11:05:11.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:11.361 7f192eb90700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19280e5490 con 0x7f19281b0470 2021-02-10T11:05:11.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:11.361 7f1927fff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f19280ef610 0x7f19280efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:11.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:11.361 7f1927fff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f19280ef610 0x7f19280efa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:50508/0 (socket says 172.21.2.102:50508) 2021-02-10T11:05:11.377 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:11.361 7f192d1ad700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f19280f0030 0x7f19280f10d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:42734/0 (socket says 172.21.2.102:42734) 2021-02-10T11:05:11.377 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:11.361 7f1927fff700 1 -- 172.21.2.102:0/3195645213 learned_addr learned my addr 172.21.2.102:0/3195645213 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:05:11.377 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:11.361 7f1927fff700 1 -- 172.21.2.102:0/3195645213 >> v2:172.21.2.113:3300/0 conn(0x7f19281b0470 msgr2=0x7f19281b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:05:11.377 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:11.361 7f1927fff700 1 --2- 172.21.2.102:0/3195645213 >> v2:172.21.2.113:3300/0 conn(0x7f19281b0470 0x7f19281b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:05:11.377 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:11.361 7f1927fff700 1 -- 172.21.2.102:0/3195645213 >> v1:172.21.2.102:6789/0 conn(0x7f19280f0030 legacy=0x7f19280f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:11.377 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:11.361 7f1927fff700 1 -- 172.21.2.102:0/3195645213 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19280f0430 con 0x7f19280ef610 2021-02-10T11:05:11.377 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:11.361 7f1927fff700 1 --2- 172.21.2.102:0/3195645213 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f19280ef610 0x7f19280efa10 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 rx=0x7f1918006700 tx=0x7f1918009b00).ready entity=mon.0 client_cookie=b37af5bdcd4e0837 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:05:11.377 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:11.361 7f19277fe700 1 -- 172.21.2.102:0/3195645213 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f191800c0c0 con 0x7f19280ef610 2021-02-10T11:05:11.377 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:11.361 7f19277fe700 1 -- 172.21.2.102:0/3195645213 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f191800c340 con 0x7f19280ef610 2021-02-10T11:05:11.377 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:11.361 7f19277fe700 1 -- 172.21.2.102:0/3195645213 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f191800c820 con 0x7f19280ef610 2021-02-10T11:05:11.378 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:11.361 7f192eb90700 1 -- 172.21.2.102:0/3195645213 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f19280ef610 msgr2=0x7f19280efa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:11.378 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:11.361 7f192eb90700 1 --2- 172.21.2.102:0/3195645213 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f19280ef610 0x7f19280efa10 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 rx=0x7f1918006700 tx=0x7f1918009b00).stop 2021-02-10T11:05:11.378 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:11.362 7f192eb90700 1 -- 172.21.2.102:0/3195645213 shutdown_connections 2021-02-10T11:05:11.378 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:11.362 7f192eb90700 1 --2- 172.21.2.102:0/3195645213 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f19280ef610 0x7f19280efa10 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:11.378 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:11.362 7f192eb90700 1 --2- 172.21.2.102:0/3195645213 >> v2:172.21.2.113:3300/0 conn(0x7f19281b0470 0x7f19281b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:05:11.378 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:11.362 7f192eb90700 1 -- 172.21.2.102:0/3195645213 >> 172.21.2.102:0/3195645213 conn(0x7f19280ce3e0 msgr2=0x7f19280e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:05:11.378 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:11.362 7f192eb90700 1 -- 172.21.2.102:0/3195645213 shutdown_connections 2021-02-10T11:05:11.378 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:11.362 7f192eb90700 1 -- 172.21.2.102:0/3195645213 wait complete. 2021-02-10T11:05:11.378 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:11.377 7f192eb90700 1 Processor -- start 2021-02-10T11:05:11.378 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:11.377 7f192eb90700 1 -- start start 2021-02-10T11:05:11.380 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:11.377 7f192eb90700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f19280b20b0 0x7f19280ac970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:11.381 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:11.377 7f192eb90700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f19280b24b0 0x7f1928191dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:11.381 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:11.377 7f192eb90700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f19280acf90 0x7f19281923e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:11.381 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:11.377 7f192eb90700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19280b1b30 con 0x7f19280b20b0 2021-02-10T11:05:11.381 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:11.377 7f192eb90700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19281a04e0 con 0x7f19280b24b0 2021-02-10T11:05:11.381 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:11.377 7f192eb90700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19281a06c0 con 0x7f19280acf90 2021-02-10T11:05:11.381 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:11.378 7f192c9ac700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f19280b20b0 0x7f19280ac970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:11.381 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:11.378 7f192c9ac700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f19280b20b0 0x7f19280ac970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:50514/0 (socket says 172.21.2.102:50514) 2021-02-10T11:05:11.382 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:11.378 7f192c9ac700 1 -- 172.21.2.102:0/3747440587 learned_addr learned my addr 172.21.2.102:0/3747440587 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:05:11.382 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:11.378 7f192c9ac700 1 -- 172.21.2.102:0/3747440587 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f19280acf90 msgr2=0x7f19281923e0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:05:11.382 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:11.378 7f192c9ac700 1 --2- 172.21.2.102:0/3747440587 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f19280acf90 0x7f19281923e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:05:11.382 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:11.378 7f192c9ac700 1 -- 172.21.2.102:0/3747440587 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f19280b24b0 msgr2=0x7f1928191dc0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:05:11.382 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:11.378 7f192c9ac700 1 --2- 172.21.2.102:0/3747440587 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f19280b24b0 0x7f1928191dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:05:11.382 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:11.378 7f192c9ac700 1 -- 172.21.2.102:0/3747440587 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1928192be0 con 0x7f19280b20b0 2021-02-10T11:05:11.382 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:11.378 7f192c9ac700 1 --2- 172.21.2.102:0/3747440587 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f19280b20b0 0x7f19280ac970 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 rx=0x7f1910006f00 tx=0x7f191000b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:05:11.382 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:11.378 7f19257fa700 1 -- 172.21.2.102:0/3747440587 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f191000bd30 con 0x7f19280b20b0 2021-02-10T11:05:11.382 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:11.378 7f19257fa700 1 -- 172.21.2.102:0/3747440587 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1910000c90 con 0x7f19280b20b0 2021-02-10T11:05:11.382 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:11.378 7f19257fa700 1 -- 172.21.2.102:0/3747440587 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1910002020 con 0x7f19280b20b0 2021-02-10T11:05:11.382 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:11.378 7f192eb90700 1 -- 172.21.2.102:0/3747440587 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1928192be0 con 0x7f19280b20b0 2021-02-10T11:05:11.382 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:11.378 7f192eb90700 1 -- 172.21.2.102:0/3747440587 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1928193170 con 0x7f19280b20b0 2021-02-10T11:05:11.383 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:11.379 7f19257fa700 1 -- 172.21.2.102:0/3747440587 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1910005310 con 0x7f19280b20b0 2021-02-10T11:05:11.383 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:11.379 7f19257fa700 1 --2- 172.21.2.102:0/3747440587 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f190c0186c0 0x7f190c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:11.383 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:11.379 7f19257fa700 1 -- 172.21.2.102:0/3747440587 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 5107+0+0 (secure 0 0 0) 0x7f1910032410 con 0x7f19280b20b0 2021-02-10T11:05:11.383 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:11.379 7f1927fff700 1 --2- 172.21.2.102:0/3747440587 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f190c0186c0 0x7f190c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:11.383 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:11.379 7f1927fff700 1 --2- 172.21.2.102:0/3747440587 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f190c0186c0 0x7f190c01abc0 crc :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:05:11.409 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:11.410 7f192eb90700 1 -- 172.21.2.102:0/3747440587 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1928105af0 con 0x7f19280b20b0 2021-02-10T11:05:11.412 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:11.413 7f19257fa700 1 -- 172.21.2.102:0/3747440587 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f1910030d90 con 0x7f19280b20b0 2021-02-10T11:05:11.540 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:11.541 7f192eb90700 1 -- 172.21.2.102:0/3747440587 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_data"} v 0) v1 -- 0x7f19280f0b50 con 0x7f19280b20b0 2021-02-10T11:05:11.917 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:11.918 7f19257fa700 1 -- 172.21.2.102:0/3747440587 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_data"}]=0 pool 'cephfs_data' created v28) v1 ==== 125+0+0 (secure 0 0 0) 0x7f1910005310 con 0x7f19280b20b0 2021-02-10T11:05:11.955 INFO:teuthology.orchestra.run.gibba002.stderr:pool 'cephfs_data' created 2021-02-10T11:05:11.957 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:11.958 7f192eb90700 1 -- 172.21.2.102:0/3747440587 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f190c0186c0 msgr2=0x7f190c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:11.958 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:11.958 7f192eb90700 1 --2- 172.21.2.102:0/3747440587 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f190c0186c0 0x7f190c01abc0 crc :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:11.958 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:11.958 7f192eb90700 1 -- 172.21.2.102:0/3747440587 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f19280b20b0 msgr2=0x7f19280ac970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:11.958 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:11.958 7f192eb90700 1 --2- 172.21.2.102:0/3747440587 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f19280b20b0 0x7f19280ac970 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 rx=0x7f1910006f00 tx=0x7f191000b940).stop 2021-02-10T11:05:11.958 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:11.958 7f192eb90700 1 -- 172.21.2.102:0/3747440587 shutdown_connections 2021-02-10T11:05:11.958 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:11.958 7f192eb90700 1 --2- 172.21.2.102:0/3747440587 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f19280acf90 0x7f19281923e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:05:11.958 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:11.958 7f192eb90700 1 --2- 172.21.2.102:0/3747440587 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f190c0186c0 0x7f190c01abc0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:11.958 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:11.958 7f192eb90700 1 --2- 172.21.2.102:0/3747440587 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f19280b24b0 0x7f1928191dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:05:11.958 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:11.958 7f192eb90700 1 --2- 172.21.2.102:0/3747440587 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f19280b20b0 0x7f19280ac970 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:11.958 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:11.958 7f192eb90700 1 -- 172.21.2.102:0/3747440587 >> 172.21.2.102:0/3747440587 conn(0x7f19280bf4e0 msgr2=0x7f19280be3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:05:11.958 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:11.958 7f192eb90700 1 -- 172.21.2.102:0/3747440587 shutdown_connections 2021-02-10T11:05:11.958 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:11.959 7f192eb90700 1 -- 172.21.2.102:0/3747440587 wait complete. 2021-02-10T11:05:11.966 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2021-02-10T11:05:11.987 DEBUG:teuthology.orchestra.run.gibba002:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:05:12.011 DEBUG:teuthology.orchestra.run.gibba013:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:05:12.400 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:12.384 7fbb8a66d700 1 Processor -- start 2021-02-10T11:05:12.400 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:12.384 7fbb8a66d700 1 -- start start 2021-02-10T11:05:12.400 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:12.384 7fbb8a66d700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbb840d1c70 0x7fbb840cef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:12.400 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:12.384 7fbb8a66d700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb840d0170 con 0x7fbb840cf580 2021-02-10T11:05:12.400 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:12.384 7fbb8a66d700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb840fe1c0 con 0x7fbb840d1870 2021-02-10T11:05:12.400 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:12.384 7fbb8a66d700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb841b0cb0 con 0x7fbb840d1c70 2021-02-10T11:05:12.400 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:12.384 7fbb88c8a700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fbb840cf580 0x7fbb840cf980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:42748/0 (socket says 172.21.2.102:42748) 2021-02-10T11:05:12.400 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:12.384 7fbb88c8a700 1 -- 172.21.2.102:0/3356854225 learned_addr learned my addr 172.21.2.102:0/3356854225 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:05:12.400 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:12.385 7fbb82ffd700 1 -- 172.21.2.102:0/3356854225 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1053759921 0 0) 0x7fbb7c001500 con 0x7fbb840cf580 2021-02-10T11:05:12.400 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:12.385 7fbb82ffd700 1 -- 172.21.2.102:0/3356854225 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb6c002fe0 con 0x7fbb840cf580 2021-02-10T11:05:12.400 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:12.385 7fbb837fe700 1 --2- 172.21.2.102:0/3356854225 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbb840d1c70 0x7fbb840cef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:12.401 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:12.385 7fbb82ffd700 1 -- 172.21.2.102:0/3356854225 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3856944329 0 0) 0x7fbb7c001460 con 0x7fbb840cf580 2021-02-10T11:05:12.401 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:12.385 7fbb82ffd700 1 -- 172.21.2.102:0/3356854225 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbb840d1c70 msgr2=0x7fbb840cef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:05:12.401 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:12.385 7fbb82ffd700 1 --2- 172.21.2.102:0/3356854225 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbb840d1c70 0x7fbb840cef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:12.401 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:12.385 7fbb82ffd700 1 -- 172.21.2.102:0/3356854225 >> v1:172.21.2.113:6789/0 conn(0x7fbb840d1870 legacy=0x7fbb841b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:12.401 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:12.385 7fbb82ffd700 1 -- 172.21.2.102:0/3356854225 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb84109a60 con 0x7fbb840cf580 2021-02-10T11:05:12.401 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:12.385 7fbb82ffd700 1 -- 172.21.2.102:0/3356854225 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7fbb7c001980 con 0x7fbb840cf580 2021-02-10T11:05:12.401 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:12.385 7fbb82ffd700 1 -- 172.21.2.102:0/3356854225 >> v1:172.21.2.102:6789/0 conn(0x7fbb840cf580 legacy=0x7fbb840cf980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:12.401 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:12.385 7fbb82ffd700 1 --2- 172.21.2.102:0/3356854225 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fbb6c003a60 0x7fbb6c005ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:12.401 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:12.385 7fbb82ffd700 1 --2- 172.21.2.102:0/3356854225 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbb6c006790 0x7fbb6c008c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:12.401 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:12.385 7fbb82ffd700 1 --2- 172.21.2.102:0/3356854225 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbb6c009590 0x7fbb6c00ba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:12.401 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:12.385 7fbb82ffd700 1 -- 172.21.2.102:0/3356854225 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb6c00c2d0 con 0x7fbb6c006790 2021-02-10T11:05:12.401 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:12.385 7fbb82ffd700 1 -- 172.21.2.102:0/3356854225 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb6c00c4f0 con 0x7fbb6c003a60 2021-02-10T11:05:12.401 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:12.385 7fbb82ffd700 1 -- 172.21.2.102:0/3356854225 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb6c050720 con 0x7fbb6c009590 2021-02-10T11:05:12.402 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:12.385 7fbb837fe700 1 --2- 172.21.2.102:0/3356854225 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbb6c006790 0x7fbb6c008c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:12.402 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:12.385 7fbb88c8a700 1 --2- 172.21.2.102:0/3356854225 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbb6c009590 0x7fbb6c00ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:12.402 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:12.385 7fbb83fff700 1 --2- 172.21.2.102:0/3356854225 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fbb6c003a60 0x7fbb6c005ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:12.402 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:12.385 7fbb837fe700 1 -- 172.21.2.102:0/3356854225 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbb6c009590 msgr2=0x7fbb6c00ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:05:12.402 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:12.385 7fbb837fe700 1 --2- 172.21.2.102:0/3356854225 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbb6c009590 0x7fbb6c00ba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:12.402 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:12.385 7fbb837fe700 1 -- 172.21.2.102:0/3356854225 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fbb6c003a60 msgr2=0x7fbb6c005ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:05:12.402 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:12.385 7fbb837fe700 1 --2- 172.21.2.102:0/3356854225 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fbb6c003a60 0x7fbb6c005ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:12.402 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:12.385 7fbb837fe700 1 -- 172.21.2.102:0/3356854225 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb6c050980 con 0x7fbb6c006790 2021-02-10T11:05:12.402 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:12.385 7fbb837fe700 1 --2- 172.21.2.102:0/3356854225 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbb6c006790 0x7fbb6c008c90 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 rx=0x7fbb780069e0 tx=0x7fbb78009b00).ready entity=mon.0 client_cookie=ef54e6e920b7c1fb server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:05:12.402 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:12.386 7fbb82ffd700 1 -- 172.21.2.102:0/3356854225 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbb7800c0c0 con 0x7fbb6c006790 2021-02-10T11:05:12.402 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:12.386 7fbb82ffd700 1 -- 172.21.2.102:0/3356854225 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb7800c340 con 0x7fbb6c006790 2021-02-10T11:05:12.402 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:12.386 7fbb82ffd700 1 -- 172.21.2.102:0/3356854225 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbb7800c860 con 0x7fbb6c006790 2021-02-10T11:05:12.403 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:12.386 7fbb8a66d700 1 -- 172.21.2.102:0/3356854225 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbb6c006790 msgr2=0x7fbb6c008c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:12.403 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:12.386 7fbb8a66d700 1 --2- 172.21.2.102:0/3356854225 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbb6c006790 0x7fbb6c008c90 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 rx=0x7fbb780069e0 tx=0x7fbb78009b00).stop 2021-02-10T11:05:12.403 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:12.386 7fbb88c8a700 1 -- 172.21.2.102:0/3356854225 reap_dead start 2021-02-10T11:05:12.403 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:12.386 7fbb8a66d700 1 -- 172.21.2.102:0/3356854225 shutdown_connections 2021-02-10T11:05:12.403 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:12.386 7fbb8a66d700 1 -- 172.21.2.102:0/3356854225 >> 172.21.2.102:0/3356854225 conn(0x7fbb840ce420 msgr2=0x7fbb840d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:05:12.403 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:12.386 7fbb8a66d700 1 -- 172.21.2.102:0/3356854225 shutdown_connections 2021-02-10T11:05:12.403 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:12.386 7fbb8a66d700 1 -- 172.21.2.102:0/3356854225 wait complete. 2021-02-10T11:05:12.403 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:12.401 7fbb8a66d700 1 Processor -- start 2021-02-10T11:05:12.403 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:12.401 7fbb8a66d700 1 -- start start 2021-02-10T11:05:12.405 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:12.401 7fbb8a66d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbb840982f0 0x7fbb841932e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:12.405 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:12.401 7fbb8a66d700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fbb840b1f00 0x7fbb840b2300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:12.405 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:12.401 7fbb8a66d700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbb840b2920 0x7fbb840b32e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:12.405 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:12.401 7fbb8a66d700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb840b2d60 con 0x7fbb840982f0 2021-02-10T11:05:12.405 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:12.401 7fbb8a66d700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb840b3900 con 0x7fbb840b1f00 2021-02-10T11:05:12.405 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:12.401 7fbb8a66d700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb840b3ae0 con 0x7fbb840b2920 2021-02-10T11:05:12.405 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:12.402 7fbb83fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbb840982f0 0x7fbb841932e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:12.405 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:12.402 7fbb83fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbb840982f0 0x7fbb841932e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:50532/0 (socket says 172.21.2.102:50532) 2021-02-10T11:05:12.405 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:12.402 7fbb83fff700 1 -- 172.21.2.102:0/2047939857 learned_addr learned my addr 172.21.2.102:0/2047939857 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:05:12.405 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:12.402 7fbb837fe700 1 --2- 172.21.2.102:0/2047939857 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fbb840b1f00 0x7fbb840b2300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:12.405 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:12.402 7fbb88c8a700 1 --2- 172.21.2.102:0/2047939857 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbb840b2920 0x7fbb840b32e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:12.405 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:12.402 7fbb83fff700 1 -- 172.21.2.102:0/2047939857 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbb840b2920 msgr2=0x7fbb840b32e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:12.406 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:12.402 7fbb83fff700 1 --2- 172.21.2.102:0/2047939857 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbb840b2920 0x7fbb840b32e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:12.406 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:12.402 7fbb83fff700 1 -- 172.21.2.102:0/2047939857 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fbb840b1f00 msgr2=0x7fbb840b2300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:12.406 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:12.402 7fbb83fff700 1 --2- 172.21.2.102:0/2047939857 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fbb840b1f00 0x7fbb840b2300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:12.406 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:12.402 7fbb83fff700 1 -- 172.21.2.102:0/2047939857 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb840b3d00 con 0x7fbb840982f0 2021-02-10T11:05:12.406 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:12.402 7fbb837fe700 1 --2- 172.21.2.102:0/2047939857 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fbb840b1f00 0x7fbb840b2300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:05:12.406 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:12.402 7fbb83fff700 1 --2- 172.21.2.102:0/2047939857 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbb840982f0 0x7fbb841932e0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 rx=0x7fbb7400b650 tx=0x7fbb7400b9f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:05:12.406 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:12.402 7fbb80ff9700 1 -- 172.21.2.102:0/2047939857 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbb7400bde0 con 0x7fbb840982f0 2021-02-10T11:05:12.406 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:12.402 7fbb80ff9700 1 -- 172.21.2.102:0/2047939857 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb74006d90 con 0x7fbb840982f0 2021-02-10T11:05:12.406 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:12.402 7fbb8a66d700 1 -- 172.21.2.102:0/2047939857 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb840b3d00 con 0x7fbb840982f0 2021-02-10T11:05:12.406 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:12.402 7fbb80ff9700 1 -- 172.21.2.102:0/2047939857 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbb74002220 con 0x7fbb840982f0 2021-02-10T11:05:12.406 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:12.402 7fbb8a66d700 1 -- 172.21.2.102:0/2047939857 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb840b4290 con 0x7fbb840982f0 2021-02-10T11:05:12.406 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:12.403 7fbb80ff9700 1 -- 172.21.2.102:0/2047939857 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fbb74000d30 con 0x7fbb840982f0 2021-02-10T11:05:12.406 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:12.403 7fbb80ff9700 1 --2- 172.21.2.102:0/2047939857 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fbb64018730 0x7fbb6401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:12.407 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:12.403 7fbb80ff9700 1 -- 172.21.2.102:0/2047939857 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 5457+0+0 (secure 0 0 0) 0x7fbb74032590 con 0x7fbb840982f0 2021-02-10T11:05:12.407 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:12.403 7fbb837fe700 1 --2- 172.21.2.102:0/2047939857 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fbb64018730 0x7fbb6401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:12.407 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:12.403 7fbb837fe700 1 --2- 172.21.2.102:0/2047939857 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fbb64018730 0x7fbb6401abc0 crc :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:05:12.433 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:12.434 7fbb8a66d700 1 -- 172.21.2.102:0/2047939857 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb840fe1c0 con 0x7fbb840982f0 2021-02-10T11:05:12.435 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:12.437 7fbb80ff9700 1 -- 172.21.2.102:0/2047939857 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fbb74006b60 con 0x7fbb840982f0 2021-02-10T11:05:12.564 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:12.565 7fbb8a66d700 1 -- 172.21.2.102:0/2047939857 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"data": "cephfs_data", "prefix": "fs new", "force": true, "fs_name": "cephfs", "metadata": "cephfs_metadata"} v 0) v1 -- 0x7fbb840fe1c0 con 0x7fbb840982f0 2021-02-10T11:05:12.567 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:12.569 7fbb80ff9700 1 -- 172.21.2.102:0/2047939857 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"data": "cephfs_data", "prefix": "fs new", "force": true, "fs_name": "cephfs", "metadata": "cephfs_metadata"}]=0 new fs with metadata pool 4 and data pool 5 v11) v1 ==== 187+0+0 (secure 0 0 0) 0x7fbb74000d30 con 0x7fbb840982f0 2021-02-10T11:05:12.574 INFO:teuthology.orchestra.run.gibba002.stderr:new fs with metadata pool 4 and data pool 5 2021-02-10T11:05:12.574 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:12.575 7fbb8a66d700 1 -- 172.21.2.102:0/2047939857 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fbb64018730 msgr2=0x7fbb6401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:12.574 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:12.575 7fbb8a66d700 1 --2- 172.21.2.102:0/2047939857 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fbb64018730 0x7fbb6401abc0 crc :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:12.574 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:12.575 7fbb8a66d700 1 -- 172.21.2.102:0/2047939857 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbb840982f0 msgr2=0x7fbb841932e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:12.574 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:12.575 7fbb8a66d700 1 --2- 172.21.2.102:0/2047939857 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbb840982f0 0x7fbb841932e0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 rx=0x7fbb7400b650 tx=0x7fbb7400b9f0).stop 2021-02-10T11:05:12.574 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:12.575 7fbb8a66d700 1 -- 172.21.2.102:0/2047939857 shutdown_connections 2021-02-10T11:05:12.574 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:12.575 7fbb8a66d700 1 --2- 172.21.2.102:0/2047939857 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbb840b2920 0x7fbb840b32e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:12.574 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:12.575 7fbb8a66d700 1 --2- 172.21.2.102:0/2047939857 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fbb64018730 0x7fbb6401abc0 crc :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:12.575 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:12.575 7fbb8a66d700 1 --2- 172.21.2.102:0/2047939857 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fbb840b1f00 0x7fbb840b2300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:12.575 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:12.575 7fbb8a66d700 1 --2- 172.21.2.102:0/2047939857 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbb840982f0 0x7fbb841932e0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:12.575 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:12.575 7fbb8a66d700 1 -- 172.21.2.102:0/2047939857 >> 172.21.2.102:0/2047939857 conn(0x7fbb84084570 msgr2=0x7fbb8409a310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:05:12.575 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:12.575 7fbb8a66d700 1 -- 172.21.2.102:0/2047939857 shutdown_connections 2021-02-10T11:05:12.575 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:12.575 7fbb8a66d700 1 -- 172.21.2.102:0/2047939857 wait complete. 2021-02-10T11:05:12.582 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:05:13.008 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:12.992 7f3a17bed700 1 Processor -- start 2021-02-10T11:05:13.008 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:12.993 7f3a17bed700 1 -- start start 2021-02-10T11:05:13.008 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:12.993 7f3a17bed700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3a100e5070 0x7f3a101b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:13.008 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:12.993 7f3a17bed700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f3a100e5470 0x7f3a1007d690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:13.008 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:12.993 7f3a17bed700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a1007ed80 con 0x7f3a101b3e40 2021-02-10T11:05:13.008 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:12.993 7f3a17bed700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a10080310 con 0x7f3a100e5470 2021-02-10T11:05:13.008 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:12.993 7f3a17bed700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a10080500 con 0x7f3a100e5070 2021-02-10T11:05:13.009 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:12.993 7f3a15208700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f3a100e5470 0x7f3a1007d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:13.009 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:12.993 7f3a15208700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f3a100e5470 0x7f3a1007d690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:50540/0 (socket says 172.21.2.102:50540) 2021-02-10T11:05:13.009 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:12.993 7f3a1620a700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f3a101b3e40 0x7f3a1007dcb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:42764/0 (socket says 172.21.2.102:42764) 2021-02-10T11:05:13.009 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:12.993 7f3a15208700 1 -- 172.21.2.102:0/1187646357 learned_addr learned my addr 172.21.2.102:0/1187646357 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:05:13.009 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:12.994 7f3a15208700 1 -- 172.21.2.102:0/1187646357 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3a100e5070 msgr2=0x7f3a101b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:05:13.009 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:12.994 7f3a15a09700 1 --2- 172.21.2.102:0/1187646357 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3a100e5070 0x7f3a101b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:13.009 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:12.994 7f3a15208700 1 --2- 172.21.2.102:0/1187646357 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3a100e5070 0x7f3a101b3820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:13.009 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:12.994 7f3a15208700 1 -- 172.21.2.102:0/1187646357 >> v1:172.21.2.102:6789/0 conn(0x7f3a101b3e40 legacy=0x7f3a1007dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:13.009 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:12.994 7f3a15208700 1 -- 172.21.2.102:0/1187646357 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a100806f0 con 0x7f3a100e5470 2021-02-10T11:05:13.009 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:12.994 7f3a15208700 1 --2- 172.21.2.102:0/1187646357 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3a100e5470 0x7f3a1007d690 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 rx=0x7f3a04006700 tx=0x7f3a04009b80).ready entity=mon.0 client_cookie=4e84e046b12e3f04 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:05:13.009 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:12.994 7f3a14a07700 1 -- 172.21.2.102:0/1187646357 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3a0400c0c0 con 0x7f3a100e5470 2021-02-10T11:05:13.009 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:12.994 7f3a14a07700 1 -- 172.21.2.102:0/1187646357 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a0400c340 con 0x7f3a100e5470 2021-02-10T11:05:13.009 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:12.994 7f3a14a07700 1 -- 172.21.2.102:0/1187646357 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3a0400c860 con 0x7f3a100e5470 2021-02-10T11:05:13.010 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:12.994 7f3a17bed700 1 -- 172.21.2.102:0/1187646357 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3a100e5470 msgr2=0x7f3a1007d690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:13.010 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:12.994 7f3a17bed700 1 --2- 172.21.2.102:0/1187646357 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3a100e5470 0x7f3a1007d690 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 rx=0x7f3a04006700 tx=0x7f3a04009b80).stop 2021-02-10T11:05:13.010 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:12.995 7f3a17bed700 1 -- 172.21.2.102:0/1187646357 shutdown_connections 2021-02-10T11:05:13.010 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:12.995 7f3a17bed700 1 --2- 172.21.2.102:0/1187646357 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3a100e5470 0x7f3a1007d690 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:13.010 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:12.995 7f3a17bed700 1 --2- 172.21.2.102:0/1187646357 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3a100e5070 0x7f3a101b3820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:13.010 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:12.995 7f3a17bed700 1 -- 172.21.2.102:0/1187646357 >> 172.21.2.102:0/1187646357 conn(0x7f3a100ce690 msgr2=0x7f3a100e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:05:13.010 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:12.995 7f3a17bed700 1 -- 172.21.2.102:0/1187646357 shutdown_connections 2021-02-10T11:05:13.010 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:12.995 7f3a17bed700 1 -- 172.21.2.102:0/1187646357 wait complete. 2021-02-10T11:05:13.012 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:13.010 7f3a17bed700 1 Processor -- start 2021-02-10T11:05:13.012 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:13.010 7f3a17bed700 1 -- start start 2021-02-10T11:05:13.012 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:13.010 7f3a17bed700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3a10192fc0 0x7f3a10192780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:13.012 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:13.010 7f3a17bed700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3a100b1430 0x7f3a100b1830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:13.012 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:13.010 7f3a17bed700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3a100b1e50 0x7f3a1008a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:13.012 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:13.010 7f3a17bed700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a100b2250 con 0x7f3a100b1e50 2021-02-10T11:05:13.012 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:13.010 7f3a17bed700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a1008b5d0 con 0x7f3a10192fc0 2021-02-10T11:05:13.012 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:13.010 7f3a17bed700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a1008aa80 con 0x7f3a100b1430 2021-02-10T11:05:13.012 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:13.010 7f3a1620a700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3a100b1e50 0x7f3a1008a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:13.012 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:13.010 7f3a1620a700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3a100b1e50 0x7f3a1008a460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:50546/0 (socket says 172.21.2.102:50546) 2021-02-10T11:05:13.013 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:13.010 7f3a1620a700 1 -- 172.21.2.102:0/4229486981 learned_addr learned my addr 172.21.2.102:0/4229486981 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:05:13.013 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:13.010 7f3a1620a700 1 -- 172.21.2.102:0/4229486981 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3a100b1430 msgr2=0x7f3a100b1830 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:05:13.013 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:13.010 7f3a1620a700 1 --2- 172.21.2.102:0/4229486981 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3a100b1430 0x7f3a100b1830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:05:13.013 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:13.010 7f3a1620a700 1 -- 172.21.2.102:0/4229486981 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3a10192fc0 msgr2=0x7f3a10192780 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:05:13.013 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:13.010 7f3a1620a700 1 --2- 172.21.2.102:0/4229486981 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3a10192fc0 0x7f3a10192780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:05:13.013 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:13.010 7f3a1620a700 1 -- 172.21.2.102:0/4229486981 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a1008ac60 con 0x7f3a100b1e50 2021-02-10T11:05:13.013 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:13.011 7f3a1620a700 1 --2- 172.21.2.102:0/4229486981 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3a100b1e50 0x7f3a1008a460 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 rx=0x7f3a0800a8f0 tx=0x7f3a0800ac90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:05:13.013 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:13.011 7f3a027fc700 1 -- 172.21.2.102:0/4229486981 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3a080070d0 con 0x7f3a100b1e50 2021-02-10T11:05:13.013 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:13.011 7f3a027fc700 1 -- 172.21.2.102:0/4229486981 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a08007310 con 0x7f3a100b1e50 2021-02-10T11:05:13.013 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:13.011 7f3a027fc700 1 -- 172.21.2.102:0/4229486981 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3a08004720 con 0x7f3a100b1e50 2021-02-10T11:05:13.013 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:13.011 7f3a17bed700 1 -- 172.21.2.102:0/4229486981 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a1008ac60 con 0x7f3a100b1e50 2021-02-10T11:05:13.014 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:13.011 7f3a17bed700 1 -- 172.21.2.102:0/4229486981 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a1008ac60 con 0x7f3a100b1e50 2021-02-10T11:05:13.014 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:13.011 7f3a027fc700 1 -- 172.21.2.102:0/4229486981 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f3a080074f0 con 0x7f3a100b1e50 2021-02-10T11:05:13.014 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:13.012 7f3a027fc700 1 --2- 172.21.2.102:0/4229486981 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f39f8018730 0x7f39f801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:13.014 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:13.012 7f3a027fc700 1 -- 172.21.2.102:0/4229486981 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f3a08032bc0 con 0x7f3a100b1e50 2021-02-10T11:05:13.014 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:13.012 7f3a15a09700 1 --2- 172.21.2.102:0/4229486981 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f39f8018730 0x7f39f801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:13.014 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:13.012 7f3a15a09700 1 --2- 172.21.2.102:0/4229486981 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f39f8018730 0x7f39f801ac10 crc :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:05:13.041 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:13.043 7f3a17bed700 1 -- 172.21.2.102:0/4229486981 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a10107120 con 0x7f3a100b1e50 2021-02-10T11:05:13.044 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:13.045 7f3a027fc700 1 -- 172.21.2.102:0/4229486981 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f3a08001270 con 0x7f3a100b1e50 2021-02-10T11:05:13.169 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:13.170 7f3a17bed700 1 -- 172.21.2.102:0/4229486981 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3a1007ed80 con 0x7f3a100b1e50 2021-02-10T11:05:13.169 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:13.171 7f3a027fc700 1 -- 172.21.2.102:0/4229486981 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v29) v1 ==== 74+0+14006 (secure 0 0 0) 0x7f3a08005ef0 con 0x7f3a100b1e50 2021-02-10T11:05:13.170 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:05:13.170 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":29,"fsid":"68db30ea-dbce-4aef-a6df-f01c6fea0513","created":"2021-02-10 11:03:06.331557","modified":"2021-02-10 11:05:12.567054","last_up_change":"2021-02-10 11:03:32.690562","last_in_change":"2021-02-10 11:03:32.690562","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:03:37.322519","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:05:10.538786","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":5,"pool_name":"cephfs_data","create_time":"2021-02-10 11:05:11.559128","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"be691b4d-a039-408a-af80-9d53f02b6f1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6818","nonce":36088},{"type":"v1","addr":"172.21.2.102:6820","nonce":36088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6822","nonce":36088},{"type":"v1","addr":"172.21.2.102:6824","nonce":36088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6830","nonce":36088},{"type":"v1","addr":"172.21.2.102:6832","nonce":36088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6826","nonce":36088},{"type":"v1","addr":"172.21.2.102:6828","nonce":36088}]},"public_addr":"172.21.2.102:6820/36088","cluster_addr":"172.21.2.102:6824/36088","heartbeat_back_addr":"172.21.2.102:6832/36088","heartbeat_front_addr":"172.21.2.102:6828/36088","state":["exists","up"]},{"osd":1,"uuid":"64e3ccc8-9c69-4d7c-865e-af8e83822635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6819","nonce":36087},{"type":"v1","addr":"172.21.2.102:6821","nonce":36087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6823","nonce":36087},{"type":"v1","addr":"172.21.2.102:6825","nonce":36087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6831","nonce":36087},{"type":"v1","addr":"172.21.2.102:6833","nonce":36087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6827","nonce":36087},{"type":"v1","addr":"172.21.2.102:6829","nonce":36087}]},"public_addr":"172.21.2.102:6821/36087","cluster_addr":"172.21.2.102:6825/36087","heartbeat_back_addr":"172.21.2.102:6833/36087","heartbeat_front_addr":"172.21.2.102:6829/36087","state":["exists","up"]},{"osd":2,"uuid":"153700af-747f-481f-bdf4-d6072d3a842f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6810","nonce":36089},{"type":"v1","addr":"172.21.2.102:6811","nonce":36089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6812","nonce":36089},{"type":"v1","addr":"172.21.2.102:6813","nonce":36089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6816","nonce":36089},{"type":"v1","addr":"172.21.2.102:6817","nonce":36089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6814","nonce":36089},{"type":"v1","addr":"172.21.2.102:6815","nonce":36089}]},"public_addr":"172.21.2.102:6811/36089","cluster_addr":"172.21.2.102:6813/36089","heartbeat_back_addr":"172.21.2.102:6817/36089","heartbeat_front_addr":"172.21.2.102:6815/36089","state":["exists","up"]},{"osd":3,"uuid":"cbc678f4-83a9-45e6-8830-2f7c232b1ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6802","nonce":36090},{"type":"v1","addr":"172.21.2.102:6803","nonce":36090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6804","nonce":36090},{"type":"v1","addr":"172.21.2.102:6805","nonce":36090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6808","nonce":36090},{"type":"v1","addr":"172.21.2.102:6809","nonce":36090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6806","nonce":36090},{"type":"v1","addr":"172.21.2.102:6807","nonce":36090}]},"public_addr":"172.21.2.102:6803/36090","cluster_addr":"172.21.2.102:6805/36090","heartbeat_back_addr":"172.21.2.102:6809/36090","heartbeat_front_addr":"172.21.2.102:6807/36090","state":["exists","up"]},{"osd":4,"uuid":"c01e4008-b258-4f10-afd2-9e7f37214351","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6808","nonce":36368},{"type":"v1","addr":"172.21.2.113:6811","nonce":36368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6814","nonce":36368},{"type":"v1","addr":"172.21.2.113:6818","nonce":36368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6826","nonce":36368},{"type":"v1","addr":"172.21.2.113:6828","nonce":36368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6820","nonce":36368},{"type":"v1","addr":"172.21.2.113:6824","nonce":36368}]},"public_addr":"172.21.2.113:6811/36368","cluster_addr":"172.21.2.113:6818/36368","heartbeat_back_addr":"172.21.2.113:6828/36368","heartbeat_front_addr":"172.21.2.113:6824/36368","state":["exists","up"]},{"osd":5,"uuid":"551f6007-daf8-4a98-8ee0-c2b16bcccb5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6802","nonce":36366},{"type":"v1","addr":"172.21.2.113:6805","nonce":36366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6807","nonce":36366},{"type":"v1","addr":"172.21.2.113:6810","nonce":36366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6819","nonce":36366},{"type":"v1","addr":"172.21.2.113:6822","nonce":36366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6812","nonce":36366},{"type":"v1","addr":"172.21.2.113:6815","nonce":36366}]},"public_addr":"172.21.2.113:6805/36366","cluster_addr":"172.21.2.113:6810/36366","heartbeat_back_addr":"172.21.2.113:6822/36366","heartbeat_front_addr":"172.21.2.113:6815/36366","state":["exists","up"]},{"osd":6,"uuid":"bf67c040-0e50-48be-b102-c3943bc9ebef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6800","nonce":36367},{"type":"v1","addr":"172.21.2.113:6801","nonce":36367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6803","nonce":36367},{"type":"v1","addr":"172.21.2.113:6804","nonce":36367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6813","nonce":36367},{"type":"v1","addr":"172.21.2.113:6816","nonce":36367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6806","nonce":36367},{"type":"v1","addr":"172.21.2.113:6809","nonce":36367}]},"public_addr":"172.21.2.113:6801/36367","cluster_addr":"172.21.2.113:6804/36367","heartbeat_back_addr":"172.21.2.113:6816/36367","heartbeat_front_addr":"172.21.2.113:6809/36367","state":["exists","up"]},{"osd":7,"uuid":"f5461f80-018d-428f-be89-2eb2bcbe7bb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6817","nonce":36365},{"type":"v1","addr":"172.21.2.113:6821","nonce":36365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6823","nonce":36365},{"type":"v1","addr":"172.21.2.113:6825","nonce":36365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6830","nonce":36365},{"type":"v1","addr":"172.21.2.113:6831","nonce":36365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6827","nonce":36365},{"type":"v1","addr":"172.21.2.113:6829","nonce":36365}]},"public_addr":"172.21.2.113:6821/36365","cluster_addr":"172.21.2.113:6825/36365","heartbeat_back_addr":"172.21.2.113:6831/36365","heartbeat_front_addr":"172.21.2.113:6829/36365","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:05:13.171 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:13.173 7f3a17bed700 1 -- 172.21.2.102:0/4229486981 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f39f8018730 msgr2=0x7f39f801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:13.171 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:13.173 7f3a17bed700 1 --2- 172.21.2.102:0/4229486981 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f39f8018730 0x7f39f801ac10 crc :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:13.171 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:13.173 7f3a17bed700 1 -- 172.21.2.102:0/4229486981 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3a100b1e50 msgr2=0x7f3a1008a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:13.171 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:13.173 7f3a17bed700 1 --2- 172.21.2.102:0/4229486981 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3a100b1e50 0x7f3a1008a460 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 rx=0x7f3a0800a8f0 tx=0x7f3a0800ac90).stop 2021-02-10T11:05:13.171 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:13.173 7f3a17bed700 1 -- 172.21.2.102:0/4229486981 shutdown_connections 2021-02-10T11:05:13.172 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:13.173 7f3a17bed700 1 --2- 172.21.2.102:0/4229486981 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3a100b1e50 0x7f3a1008a460 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:13.172 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:13.173 7f3a17bed700 1 --2- 172.21.2.102:0/4229486981 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3a100b1430 0x7f3a100b1830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:05:13.172 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:13.173 7f3a17bed700 1 --2- 172.21.2.102:0/4229486981 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f39f8018730 0x7f39f801ac10 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:13.172 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:13.173 7f3a17bed700 1 --2- 172.21.2.102:0/4229486981 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3a10192fc0 0x7f3a10192780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:05:13.172 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:13.173 7f3a17bed700 1 -- 172.21.2.102:0/4229486981 >> 172.21.2.102:0/4229486981 conn(0x7f3a101ac620 msgr2=0x7f3a1019fee0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:05:13.172 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:13.173 7f3a17bed700 1 -- 172.21.2.102:0/4229486981 shutdown_connections 2021-02-10T11:05:13.172 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:13.173 7f3a17bed700 1 -- 172.21.2.102:0/4229486981 wait complete. 2021-02-10T11:05:13.180 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:05:13.607 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:13.591 7f7b2644e700 1 Processor -- start 2021-02-10T11:05:13.607 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:13.591 7f7b2644e700 1 -- start start 2021-02-10T11:05:13.607 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:13.592 7f7b2644e700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f7b200e5070 0x7f7b201b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:13.607 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:13.592 7f7b2644e700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f7b201b3e40 0x7f7b2007d690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:13.608 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:13.592 7f7b2644e700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b2007ed80 con 0x7f7b200e5470 2021-02-10T11:05:13.608 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:13.592 7f7b2644e700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b20080310 con 0x7f7b201b3e40 2021-02-10T11:05:13.608 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:13.592 7f7b2644e700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b20080530 con 0x7f7b200e5070 2021-02-10T11:05:13.608 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:13.592 7f7b24a6b700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f7b200e5470 0x7f7b2007dcb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:42778/0 (socket says 172.21.2.102:42778) 2021-02-10T11:05:13.608 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:13.592 7f7b24a6b700 1 -- 172.21.2.102:0/597213808 learned_addr learned my addr 172.21.2.102:0/597213808 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:05:13.608 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:13.592 7f7b1f7fe700 1 --2- 172.21.2.102:0/597213808 >> v2:172.21.2.102:3300/0 conn(0x7f7b201b3e40 0x7f7b2007d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:13.608 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:13.592 7f7b1effd700 1 -- 172.21.2.102:0/597213808 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1750215273 0 0) 0x7f7b18001500 con 0x7f7b200e5470 2021-02-10T11:05:13.608 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:13.592 7f7b1effd700 1 -- 172.21.2.102:0/597213808 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b08002fe0 con 0x7f7b200e5470 2021-02-10T11:05:13.609 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:13.592 7f7b1f7fe700 1 -- 172.21.2.102:0/597213808 >> v2:172.21.2.113:3300/0 conn(0x7f7b200e5070 msgr2=0x7f7b201b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:05:13.609 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:13.592 7f7b1f7fe700 1 --2- 172.21.2.102:0/597213808 >> v2:172.21.2.113:3300/0 conn(0x7f7b200e5070 0x7f7b201b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:05:13.609 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:13.592 7f7b1f7fe700 1 -- 172.21.2.102:0/597213808 >> v1:172.21.2.102:6789/0 conn(0x7f7b200e5470 legacy=0x7f7b2007dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:13.609 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:13.592 7f7b1f7fe700 1 -- 172.21.2.102:0/597213808 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b20080750 con 0x7f7b201b3e40 2021-02-10T11:05:13.609 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:13.593 7f7b1f7fe700 1 --2- 172.21.2.102:0/597213808 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7b201b3e40 0x7f7b2007d690 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 rx=0x7f7b14006650 tx=0x7f7b14009a80).ready entity=mon.0 client_cookie=528cf2aefcef0edc server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:05:13.609 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:13.593 7f7b1effd700 1 -- 172.21.2.102:0/597213808 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7b1400c0c0 con 0x7f7b201b3e40 2021-02-10T11:05:13.609 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:13.593 7f7b1effd700 1 -- 172.21.2.102:0/597213808 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b1400c340 con 0x7f7b201b3e40 2021-02-10T11:05:13.610 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:13.593 7f7b1effd700 1 -- 172.21.2.102:0/597213808 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7b1400c860 con 0x7f7b201b3e40 2021-02-10T11:05:13.610 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:13.593 7f7b2644e700 1 -- 172.21.2.102:0/597213808 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7b201b3e40 msgr2=0x7f7b2007d690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:13.610 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:13.593 7f7b2644e700 1 --2- 172.21.2.102:0/597213808 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7b201b3e40 0x7f7b2007d690 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 rx=0x7f7b14006650 tx=0x7f7b14009a80).stop 2021-02-10T11:05:13.610 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:13.593 7f7b2644e700 1 -- 172.21.2.102:0/597213808 shutdown_connections 2021-02-10T11:05:13.610 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:13.593 7f7b2644e700 1 --2- 172.21.2.102:0/597213808 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7b201b3e40 0x7f7b2007d690 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:13.610 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:13.593 7f7b2644e700 1 --2- 172.21.2.102:0/597213808 >> v2:172.21.2.113:3300/0 conn(0x7f7b200e5070 0x7f7b201b3820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:05:13.610 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:13.593 7f7b2644e700 1 -- 172.21.2.102:0/597213808 >> 172.21.2.102:0/597213808 conn(0x7f7b200ce690 msgr2=0x7f7b200e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:05:13.610 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:13.593 7f7b2644e700 1 -- 172.21.2.102:0/597213808 shutdown_connections 2021-02-10T11:05:13.610 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:13.593 7f7b2644e700 1 -- 172.21.2.102:0/597213808 wait complete. 2021-02-10T11:05:13.612 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:13.609 7f7b2644e700 1 Processor -- start 2021-02-10T11:05:13.613 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:13.609 7f7b2644e700 1 -- start start 2021-02-10T11:05:13.613 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:13.609 7f7b2644e700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7b201918e0 0x7f7b20189c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:13.613 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:13.609 7f7b2644e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7b2018a2a0 0x7f7b2018a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:13.613 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:13.609 7f7b2644e700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7b2018acc0 0x7f7b20190070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:13.613 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:13.609 7f7b2644e700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b201a2f00 con 0x7f7b2018a2a0 2021-02-10T11:05:13.613 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:13.609 7f7b2644e700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b20191ce0 con 0x7f7b201918e0 2021-02-10T11:05:13.613 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:13.609 7f7b2644e700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b20191f00 con 0x7f7b2018acc0 2021-02-10T11:05:13.613 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:13.609 7f7b1f7fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7b2018a2a0 0x7f7b2018a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:13.613 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:13.609 7f7b1f7fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7b2018a2a0 0x7f7b2018a6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:50560/0 (socket says 172.21.2.102:50560) 2021-02-10T11:05:13.613 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:13.609 7f7b1f7fe700 1 -- 172.21.2.102:0/2795920789 learned_addr learned my addr 172.21.2.102:0/2795920789 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:05:13.613 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:13.609 7f7b1ffff700 1 --2- 172.21.2.102:0/2795920789 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7b201918e0 0x7f7b20189c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:13.613 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:13.609 7f7b24a6b700 1 --2- 172.21.2.102:0/2795920789 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7b2018acc0 0x7f7b20190070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:13.614 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:13.610 7f7b1f7fe700 1 -- 172.21.2.102:0/2795920789 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7b2018acc0 msgr2=0x7f7b20190070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:13.614 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:13.610 7f7b1f7fe700 1 --2- 172.21.2.102:0/2795920789 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7b2018acc0 0x7f7b20190070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:13.614 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:13.610 7f7b1f7fe700 1 -- 172.21.2.102:0/2795920789 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7b201918e0 msgr2=0x7f7b20189c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:13.614 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:13.610 7f7b1f7fe700 1 --2- 172.21.2.102:0/2795920789 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7b201918e0 0x7f7b20189c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:13.614 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:13.610 7f7b1f7fe700 1 -- 172.21.2.102:0/2795920789 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b20190690 con 0x7f7b2018a2a0 2021-02-10T11:05:13.614 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:13.610 7f7b1ffff700 1 --2- 172.21.2.102:0/2795920789 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7b201918e0 0x7f7b20189c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:05:13.614 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:13.610 7f7b1f7fe700 1 --2- 172.21.2.102:0/2795920789 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7b2018a2a0 0x7f7b2018a6a0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 rx=0x7f7b14009630 tx=0x7f7b1400b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:05:13.614 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:13.610 7f7b1cff9700 1 -- 172.21.2.102:0/2795920789 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7b140093a0 con 0x7f7b2018a2a0 2021-02-10T11:05:13.614 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:13.610 7f7b1cff9700 1 -- 172.21.2.102:0/2795920789 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b1400bd70 con 0x7f7b2018a2a0 2021-02-10T11:05:13.614 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:13.610 7f7b2644e700 1 -- 172.21.2.102:0/2795920789 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b20190690 con 0x7f7b2018a2a0 2021-02-10T11:05:13.615 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:13.610 7f7b2644e700 1 -- 172.21.2.102:0/2795920789 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b20190690 con 0x7f7b2018a2a0 2021-02-10T11:05:13.615 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:13.610 7f7b1cff9700 1 -- 172.21.2.102:0/2795920789 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7b14005560 con 0x7f7b2018a2a0 2021-02-10T11:05:13.615 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:13.610 7f7b1cff9700 1 -- 172.21.2.102:0/2795920789 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f7b140052e0 con 0x7f7b2018a2a0 2021-02-10T11:05:13.615 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:13.611 7f7b1cff9700 1 --2- 172.21.2.102:0/2795920789 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7b00018730 0x7f7b0001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:13.615 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:13.611 7f7b1cff9700 1 -- 172.21.2.102:0/2795920789 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f7b140327d0 con 0x7f7b2018a2a0 2021-02-10T11:05:13.615 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:13.611 7f7b1ffff700 1 --2- 172.21.2.102:0/2795920789 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7b00018730 0x7f7b0001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:13.615 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:13.611 7f7b1ffff700 1 --2- 172.21.2.102:0/2795920789 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7b00018730 0x7f7b0001ac10 crc :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:05:13.640 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:13.641 7f7b2644e700 1 -- 172.21.2.102:0/2795920789 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b2007ed80 con 0x7f7b2018a2a0 2021-02-10T11:05:13.643 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:13.644 7f7b1cff9700 1 -- 172.21.2.102:0/2795920789 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f7b1401f510 con 0x7f7b2018a2a0 2021-02-10T11:05:13.770 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:13.771 7f7b2644e700 1 -- 172.21.2.102:0/2795920789 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7b2007ed80 con 0x7f7b2018a2a0 2021-02-10T11:05:13.771 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:13.772 7f7b1cff9700 1 -- 172.21.2.102:0/2795920789 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v30) v1 ==== 74+0+13993 (secure 0 0 0) 0x7f7b140052e0 con 0x7f7b2018a2a0 2021-02-10T11:05:13.771 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:05:13.772 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":30,"fsid":"68db30ea-dbce-4aef-a6df-f01c6fea0513","created":"2021-02-10 11:03:06.331557","modified":"2021-02-10 11:05:13.569263","last_up_change":"2021-02-10 11:03:32.690562","last_in_change":"2021-02-10 11:03:32.690562","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:03:37.322519","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:05:10.538786","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":5,"pool_name":"cephfs_data","create_time":"2021-02-10 11:05:11.559128","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"30","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"be691b4d-a039-408a-af80-9d53f02b6f1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6818","nonce":36088},{"type":"v1","addr":"172.21.2.102:6820","nonce":36088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6822","nonce":36088},{"type":"v1","addr":"172.21.2.102:6824","nonce":36088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6830","nonce":36088},{"type":"v1","addr":"172.21.2.102:6832","nonce":36088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6826","nonce":36088},{"type":"v1","addr":"172.21.2.102:6828","nonce":36088}]},"public_addr":"172.21.2.102:6820/36088","cluster_addr":"172.21.2.102:6824/36088","heartbeat_back_addr":"172.21.2.102:6832/36088","heartbeat_front_addr":"172.21.2.102:6828/36088","state":["exists","up"]},{"osd":1,"uuid":"64e3ccc8-9c69-4d7c-865e-af8e83822635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6819","nonce":36087},{"type":"v1","addr":"172.21.2.102:6821","nonce":36087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6823","nonce":36087},{"type":"v1","addr":"172.21.2.102:6825","nonce":36087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6831","nonce":36087},{"type":"v1","addr":"172.21.2.102:6833","nonce":36087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6827","nonce":36087},{"type":"v1","addr":"172.21.2.102:6829","nonce":36087}]},"public_addr":"172.21.2.102:6821/36087","cluster_addr":"172.21.2.102:6825/36087","heartbeat_back_addr":"172.21.2.102:6833/36087","heartbeat_front_addr":"172.21.2.102:6829/36087","state":["exists","up"]},{"osd":2,"uuid":"153700af-747f-481f-bdf4-d6072d3a842f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6810","nonce":36089},{"type":"v1","addr":"172.21.2.102:6811","nonce":36089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6812","nonce":36089},{"type":"v1","addr":"172.21.2.102:6813","nonce":36089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6816","nonce":36089},{"type":"v1","addr":"172.21.2.102:6817","nonce":36089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6814","nonce":36089},{"type":"v1","addr":"172.21.2.102:6815","nonce":36089}]},"public_addr":"172.21.2.102:6811/36089","cluster_addr":"172.21.2.102:6813/36089","heartbeat_back_addr":"172.21.2.102:6817/36089","heartbeat_front_addr":"172.21.2.102:6815/36089","state":["exists","up"]},{"osd":3,"uuid":"cbc678f4-83a9-45e6-8830-2f7c232b1ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6802","nonce":36090},{"type":"v1","addr":"172.21.2.102:6803","nonce":36090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6804","nonce":36090},{"type":"v1","addr":"172.21.2.102:6805","nonce":36090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6808","nonce":36090},{"type":"v1","addr":"172.21.2.102:6809","nonce":36090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6806","nonce":36090},{"type":"v1","addr":"172.21.2.102:6807","nonce":36090}]},"public_addr":"172.21.2.102:6803/36090","cluster_addr":"172.21.2.102:6805/36090","heartbeat_back_addr":"172.21.2.102:6809/36090","heartbeat_front_addr":"172.21.2.102:6807/36090","state":["exists","up"]},{"osd":4,"uuid":"c01e4008-b258-4f10-afd2-9e7f37214351","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6808","nonce":36368},{"type":"v1","addr":"172.21.2.113:6811","nonce":36368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6814","nonce":36368},{"type":"v1","addr":"172.21.2.113:6818","nonce":36368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6826","nonce":36368},{"type":"v1","addr":"172.21.2.113:6828","nonce":36368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6820","nonce":36368},{"type":"v1","addr":"172.21.2.113:6824","nonce":36368}]},"public_addr":"172.21.2.113:6811/36368","cluster_addr":"172.21.2.113:6818/36368","heartbeat_back_addr":"172.21.2.113:6828/36368","heartbeat_front_addr":"172.21.2.113:6824/36368","state":["exists","up"]},{"osd":5,"uuid":"551f6007-daf8-4a98-8ee0-c2b16bcccb5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6802","nonce":36366},{"type":"v1","addr":"172.21.2.113:6805","nonce":36366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6807","nonce":36366},{"type":"v1","addr":"172.21.2.113:6810","nonce":36366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6819","nonce":36366},{"type":"v1","addr":"172.21.2.113:6822","nonce":36366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6812","nonce":36366},{"type":"v1","addr":"172.21.2.113:6815","nonce":36366}]},"public_addr":"172.21.2.113:6805/36366","cluster_addr":"172.21.2.113:6810/36366","heartbeat_back_addr":"172.21.2.113:6822/36366","heartbeat_front_addr":"172.21.2.113:6815/36366","state":["exists","up"]},{"osd":6,"uuid":"bf67c040-0e50-48be-b102-c3943bc9ebef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6800","nonce":36367},{"type":"v1","addr":"172.21.2.113:6801","nonce":36367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6803","nonce":36367},{"type":"v1","addr":"172.21.2.113:6804","nonce":36367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6813","nonce":36367},{"type":"v1","addr":"172.21.2.113:6816","nonce":36367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6806","nonce":36367},{"type":"v1","addr":"172.21.2.113:6809","nonce":36367}]},"public_addr":"172.21.2.113:6801/36367","cluster_addr":"172.21.2.113:6804/36367","heartbeat_back_addr":"172.21.2.113:6816/36367","heartbeat_front_addr":"172.21.2.113:6809/36367","state":["exists","up"]},{"osd":7,"uuid":"f5461f80-018d-428f-be89-2eb2bcbe7bb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6817","nonce":36365},{"type":"v1","addr":"172.21.2.113:6821","nonce":36365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6823","nonce":36365},{"type":"v1","addr":"172.21.2.113:6825","nonce":36365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6830","nonce":36365},{"type":"v1","addr":"172.21.2.113:6831","nonce":36365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6827","nonce":36365},{"type":"v1","addr":"172.21.2.113:6829","nonce":36365}]},"public_addr":"172.21.2.113:6821/36365","cluster_addr":"172.21.2.113:6825/36365","heartbeat_back_addr":"172.21.2.113:6831/36365","heartbeat_front_addr":"172.21.2.113:6829/36365","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:05:13.773 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:13.773 7f7b2644e700 1 -- 172.21.2.102:0/2795920789 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7b00018730 msgr2=0x7f7b0001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:13.773 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:13.773 7f7b2644e700 1 --2- 172.21.2.102:0/2795920789 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7b00018730 0x7f7b0001ac10 crc :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:13.773 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:13.773 7f7b2644e700 1 -- 172.21.2.102:0/2795920789 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7b2018a2a0 msgr2=0x7f7b2018a6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:13.773 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:13.774 7f7b2644e700 1 --2- 172.21.2.102:0/2795920789 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7b2018a2a0 0x7f7b2018a6a0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 rx=0x7f7b14009630 tx=0x7f7b1400b980).stop 2021-02-10T11:05:13.773 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:13.774 7f7b2644e700 1 -- 172.21.2.102:0/2795920789 shutdown_connections 2021-02-10T11:05:13.773 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:13.774 7f7b2644e700 1 --2- 172.21.2.102:0/2795920789 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7b2018acc0 0x7f7b20190070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:13.773 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:13.774 7f7b2644e700 1 --2- 172.21.2.102:0/2795920789 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7b2018a2a0 0x7f7b2018a6a0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:13.773 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:13.774 7f7b2644e700 1 --2- 172.21.2.102:0/2795920789 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7b00018730 0x7f7b0001ac10 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:13.773 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:13.774 7f7b2644e700 1 --2- 172.21.2.102:0/2795920789 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7b201918e0 0x7f7b20189c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:13.773 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:13.774 7f7b2644e700 1 -- 172.21.2.102:0/2795920789 >> 172.21.2.102:0/2795920789 conn(0x7f7b201a31d0 msgr2=0x7f7b2008b830 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:05:13.773 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:13.774 7f7b2644e700 1 -- 172.21.2.102:0/2795920789 shutdown_connections 2021-02-10T11:05:13.774 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:13.774 7f7b2644e700 1 -- 172.21.2.102:0/2795920789 wait complete. 2021-02-10T11:05:13.781 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2021-02-10T11:05:14.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:14.197 7efe7038e700 1 Processor -- start 2021-02-10T11:05:14.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:14.197 7efe7038e700 1 -- start start 2021-02-10T11:05:14.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:14.197 7efe7038e700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7efe680f05f0 0x7efe680f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:14.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:14.197 7efe7038e700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe681b0c70 con 0x7efe681b0870 2021-02-10T11:05:14.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:14.197 7efe7038e700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe680f0b50 con 0x7efe681b0470 2021-02-10T11:05:14.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:14.197 7efe7038e700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe680e5ea0 con 0x7efe680f05f0 2021-02-10T11:05:14.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:14.198 7efe6e9ab700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7efe680f05f0 0x7efe680f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:14.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:14.198 7efe6d9a9700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7efe681b0870 0x7efe680efe00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:42792/0 (socket says 172.21.2.102:42792) 2021-02-10T11:05:14.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:14.198 7efe6d9a9700 1 -- 172.21.2.102:0/1526052909 learned_addr learned my addr 172.21.2.102:0/1526052909 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:05:14.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:14.198 7efe6e9ab700 1 -- 172.21.2.102:0/1526052909 >> v1:172.21.2.113:6789/0 conn(0x7efe681b0470 legacy=0x7efe680ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:05:14.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:14.198 7efe6e9ab700 1 -- 172.21.2.102:0/1526052909 >> v1:172.21.2.102:6789/0 conn(0x7efe681b0870 legacy=0x7efe680efe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:14.213 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:14.198 7efe6d1a8700 1 -- 172.21.2.102:0/1526052909 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3001027662 0 0) 0x7efe64001180 con 0x7efe681b0870 2021-02-10T11:05:14.213 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:14.198 7efe6e9ab700 1 -- 172.21.2.102:0/1526052909 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe680e52e0 con 0x7efe680f05f0 2021-02-10T11:05:14.213 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:14.198 7efe6e9ab700 1 --2- 172.21.2.102:0/1526052909 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efe680f05f0 0x7efe680f20e0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 rx=0x7efe60006e30 tx=0x7efe60009de0).ready entity=mon.0 client_cookie=f61008f8553d8fde server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:05:14.213 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:14.198 7efe6d1a8700 1 -- 172.21.2.102:0/1526052909 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efe6000b300 con 0x7efe680f05f0 2021-02-10T11:05:14.213 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:14.198 7efe6d1a8700 1 -- 172.21.2.102:0/1526052909 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe6000b680 con 0x7efe680f05f0 2021-02-10T11:05:14.213 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:14.198 7efe6d1a8700 1 -- 172.21.2.102:0/1526052909 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efe6000bba0 con 0x7efe680f05f0 2021-02-10T11:05:14.213 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:14.198 7efe7038e700 1 -- 172.21.2.102:0/1526052909 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efe680f05f0 msgr2=0x7efe680f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:14.213 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:14.198 7efe7038e700 1 --2- 172.21.2.102:0/1526052909 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efe680f05f0 0x7efe680f20e0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 rx=0x7efe60006e30 tx=0x7efe60009de0).stop 2021-02-10T11:05:14.213 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:14.199 7efe7038e700 1 -- 172.21.2.102:0/1526052909 shutdown_connections 2021-02-10T11:05:14.213 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:14.199 7efe7038e700 1 --2- 172.21.2.102:0/1526052909 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efe680f05f0 0x7efe680f20e0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:14.213 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:14.199 7efe7038e700 1 -- 172.21.2.102:0/1526052909 >> 172.21.2.102:0/1526052909 conn(0x7efe680ce3e0 msgr2=0x7efe680e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:05:14.213 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:14.199 7efe7038e700 1 -- 172.21.2.102:0/1526052909 shutdown_connections 2021-02-10T11:05:14.213 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:14.199 7efe7038e700 1 -- 172.21.2.102:0/1526052909 wait complete. 2021-02-10T11:05:14.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:14.213 7efe7038e700 1 Processor -- start 2021-02-10T11:05:14.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:14.213 7efe7038e700 1 -- start start 2021-02-10T11:05:14.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:14.214 7efe7038e700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7efe680bdbf0 0x7efe681a7450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:14.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:14.214 7efe7038e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efe680bdff0 0x7efe681a7a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:14.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:14.214 7efe7038e700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7efe680abcc0 0x7efe680ac0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:14.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:14.214 7efe7038e700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe680be590 con 0x7efe680bdff0 2021-02-10T11:05:14.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:14.214 7efe7038e700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe681a8090 con 0x7efe680bdbf0 2021-02-10T11:05:14.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:14.214 7efe7038e700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe681a82b0 con 0x7efe680abcc0 2021-02-10T11:05:14.216 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:14.214 7efe6d9a9700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efe680bdff0 0x7efe681a7a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:14.216 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:14.214 7efe6d9a9700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efe680bdff0 0x7efe681a7a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:50578/0 (socket says 172.21.2.102:50578) 2021-02-10T11:05:14.216 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:14.214 7efe6d9a9700 1 -- 172.21.2.102:0/935356909 learned_addr learned my addr 172.21.2.102:0/935356909 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:05:14.216 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:14.214 7efe6d9a9700 1 -- 172.21.2.102:0/935356909 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7efe680abcc0 msgr2=0x7efe680ac0c0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:05:14.216 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:14.214 7efe6d9a9700 1 --2- 172.21.2.102:0/935356909 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7efe680abcc0 0x7efe680ac0c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:05:14.216 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:14.214 7efe6d9a9700 1 -- 172.21.2.102:0/935356909 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7efe680bdbf0 msgr2=0x7efe681a7450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:14.216 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:14.214 7efe6d9a9700 1 --2- 172.21.2.102:0/935356909 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7efe680bdbf0 0x7efe681a7450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:05:14.216 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:14.214 7efe6d9a9700 1 -- 172.21.2.102:0/935356909 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe680ac750 con 0x7efe680bdff0 2021-02-10T11:05:14.216 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:14.214 7efe6d9a9700 1 --2- 172.21.2.102:0/935356909 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efe680bdff0 0x7efe681a7a70 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 rx=0x7efe6400a570 tx=0x7efe6400a910).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:05:14.216 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:14.215 7efe5affd700 1 -- 172.21.2.102:0/935356909 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efe6400ad00 con 0x7efe680bdff0 2021-02-10T11:05:14.216 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:14.215 7efe5affd700 1 -- 172.21.2.102:0/935356909 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe64001180 con 0x7efe680bdff0 2021-02-10T11:05:14.216 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:14.215 7efe7038e700 1 -- 172.21.2.102:0/935356909 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe680ac750 con 0x7efe680bdff0 2021-02-10T11:05:14.216 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:14.215 7efe7038e700 1 -- 172.21.2.102:0/935356909 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe680ac750 con 0x7efe680bdff0 2021-02-10T11:05:14.217 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:14.215 7efe5affd700 1 -- 172.21.2.102:0/935356909 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efe64004590 con 0x7efe680bdff0 2021-02-10T11:05:14.217 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:14.215 7efe5affd700 1 -- 172.21.2.102:0/935356909 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7efe640043a0 con 0x7efe680bdff0 2021-02-10T11:05:14.217 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:14.216 7efe5affd700 1 --2- 172.21.2.102:0/935356909 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7efe50018730 0x7efe5001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:14.217 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:14.216 7efe5affd700 1 -- 172.21.2.102:0/935356909 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7efe640327c0 con 0x7efe680bdff0 2021-02-10T11:05:14.217 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:14.216 7efe6e1aa700 1 --2- 172.21.2.102:0/935356909 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7efe50018730 0x7efe5001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:14.217 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:14.216 7efe6e1aa700 1 --2- 172.21.2.102:0/935356909 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7efe50018730 0x7efe5001abc0 crc :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:05:14.246 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:14.247 7efe7038e700 1 -- 172.21.2.102:0/935356909 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe681a8090 con 0x7efe680bdff0 2021-02-10T11:05:14.248 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:14.250 7efe5affd700 1 -- 172.21.2.102:0/935356909 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7efe64000c90 con 0x7efe680bdff0 2021-02-10T11:05:14.387 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:14.388 7efe7038e700 1 -- 172.21.2.102:0/935356909 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"} v 0) v1 -- 0x7efe681a8090 con 0x7efe680bdff0 2021-02-10T11:05:14.926 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:14.927 7efe5affd700 1 -- 172.21.2.102:0/935356909 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"}]=0 v14) v1 ==== 118+0+0 (secure 0 0 0) 0x7efe64001160 con 0x7efe680bdff0 2021-02-10T11:05:14.956 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:14.957 7efe7038e700 1 -- 172.21.2.102:0/935356909 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7efe50018730 msgr2=0x7efe5001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:14.956 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:14.957 7efe7038e700 1 --2- 172.21.2.102:0/935356909 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7efe50018730 0x7efe5001abc0 crc :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:14.956 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:14.957 7efe7038e700 1 -- 172.21.2.102:0/935356909 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efe680bdff0 msgr2=0x7efe681a7a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:14.956 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:14.957 7efe7038e700 1 --2- 172.21.2.102:0/935356909 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efe680bdff0 0x7efe681a7a70 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 rx=0x7efe6400a570 tx=0x7efe6400a910).stop 2021-02-10T11:05:14.956 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:14.957 7efe7038e700 1 -- 172.21.2.102:0/935356909 shutdown_connections 2021-02-10T11:05:14.956 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:14.957 7efe7038e700 1 --2- 172.21.2.102:0/935356909 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7efe680abcc0 0x7efe680ac0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:05:14.956 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:14.957 7efe7038e700 1 --2- 172.21.2.102:0/935356909 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efe680bdff0 0x7efe681a7a70 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:14.956 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:14.957 7efe7038e700 1 --2- 172.21.2.102:0/935356909 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7efe50018730 0x7efe5001abc0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:14.956 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:14.957 7efe7038e700 1 --2- 172.21.2.102:0/935356909 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7efe680bdbf0 0x7efe681a7450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:05:14.957 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:14.957 7efe7038e700 1 -- 172.21.2.102:0/935356909 >> 172.21.2.102:0/935356909 conn(0x7efe68095a70 msgr2=0x7efe680bf040 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:05:14.957 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:14.957 7efe7038e700 1 -- 172.21.2.102:0/935356909 shutdown_connections 2021-02-10T11:05:14.957 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:14.957 7efe7038e700 1 -- 172.21.2.102:0/935356909 wait complete. 2021-02-10T11:05:14.964 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:05:15.393 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:15.376 7f699342d700 1 Processor -- start 2021-02-10T11:05:15.393 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:15.376 7f699342d700 1 -- start start 2021-02-10T11:05:15.393 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:15.376 7f699342d700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f698c0e5070 0x7f698c1b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:15.393 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:15.376 7f699342d700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f698c0e5470 0x7f698c07d690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:15.393 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:15.376 7f699342d700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f698c07ed80 con 0x7f698c1b3e40 2021-02-10T11:05:15.393 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:15.376 7f699342d700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f698c080310 con 0x7f698c0e5470 2021-02-10T11:05:15.393 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:15.376 7f699342d700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f698c080500 con 0x7f698c0e5070 2021-02-10T11:05:15.393 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:15.378 7f6991a4a700 1 --1- >> v1:172.21.2.113:6789/0 conn(0x7f698c1b3e40 0x7f698c07dcb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.113:6789/0 says I am v1:172.21.2.102:50130/0 (socket says 172.21.2.102:50130) 2021-02-10T11:05:15.393 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:15.378 7f6991a4a700 1 -- 172.21.2.102:0/3078776081 learned_addr learned my addr 172.21.2.102:0/3078776081 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:05:15.394 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:15.378 7f6990a48700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f698c0e5470 0x7f698c07d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:15.394 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:15.378 7f6991249700 1 --2- 172.21.2.102:0/3078776081 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f698c0e5070 0x7f698c1b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:15.394 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:15.378 7f6991249700 1 -- 172.21.2.102:0/3078776081 >> v2:172.21.2.113:3300/0 conn(0x7f698c0e5470 msgr2=0x7f698c07d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:05:15.394 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:15.378 7f6991249700 1 --2- 172.21.2.102:0/3078776081 >> v2:172.21.2.113:3300/0 conn(0x7f698c0e5470 0x7f698c07d690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:15.394 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:15.378 7f6991249700 1 -- 172.21.2.102:0/3078776081 >> v1:172.21.2.113:6789/0 conn(0x7f698c1b3e40 legacy=0x7f698c07dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:15.394 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:15.378 7f6991249700 1 -- 172.21.2.102:0/3078776081 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f698c0806f0 con 0x7f698c0e5070 2021-02-10T11:05:15.394 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:15.379 7f6990a48700 1 --2- 172.21.2.102:0/3078776081 >> v2:172.21.2.113:3300/0 conn(0x7f698c0e5470 0x7f698c07d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:05:15.394 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:15.378 7f697ffff700 1 -- 172.21.2.102:0/3078776081 <== mon.1 v1:172.21.2.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2975685799 0 0) 0x7f6984001510 con 0x7f698c1b3e40 2021-02-10T11:05:15.394 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:15.379 7f6991249700 1 --2- 172.21.2.102:0/3078776081 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f698c0e5070 0x7f698c1b3820 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f6988006650 tx=0x7f6988009a80).ready entity=mon.2 client_cookie=97e08d8bcf59f5cd server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:05:15.394 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:15.379 7f697ffff700 1 -- 172.21.2.102:0/3078776081 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f698800c0c0 con 0x7f698c0e5070 2021-02-10T11:05:15.394 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:15.379 7f697ffff700 1 -- 172.21.2.102:0/3078776081 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f698800c340 con 0x7f698c0e5070 2021-02-10T11:05:15.394 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:15.379 7f697ffff700 1 -- 172.21.2.102:0/3078776081 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f698800c860 con 0x7f698c0e5070 2021-02-10T11:05:15.395 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:15.379 7f699342d700 1 -- 172.21.2.102:0/3078776081 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f698c0e5070 msgr2=0x7f698c1b3820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:15.395 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:15.379 7f699342d700 1 --2- 172.21.2.102:0/3078776081 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f698c0e5070 0x7f698c1b3820 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f6988006650 tx=0x7f6988009a80).stop 2021-02-10T11:05:15.395 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:15.380 7f699342d700 1 -- 172.21.2.102:0/3078776081 shutdown_connections 2021-02-10T11:05:15.395 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:15.380 7f699342d700 1 --2- 172.21.2.102:0/3078776081 >> v2:172.21.2.113:3300/0 conn(0x7f698c0e5470 0x7f698c07d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:15.395 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:15.380 7f699342d700 1 --2- 172.21.2.102:0/3078776081 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f698c0e5070 0x7f698c1b3820 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:15.395 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:15.380 7f699342d700 1 -- 172.21.2.102:0/3078776081 >> 172.21.2.102:0/3078776081 conn(0x7f698c0ce690 msgr2=0x7f698c0e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:05:15.396 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:15.380 7f699342d700 1 -- 172.21.2.102:0/3078776081 shutdown_connections 2021-02-10T11:05:15.396 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:15.380 7f699342d700 1 -- 172.21.2.102:0/3078776081 wait complete. 2021-02-10T11:05:15.397 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:15.394 7f699342d700 1 Processor -- start 2021-02-10T11:05:15.397 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:15.394 7f699342d700 1 -- start start 2021-02-10T11:05:15.397 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:15.395 7f699342d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f698c18ee60 0x7f698c0a03a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:15.397 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:15.395 7f699342d700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f698c095a70 0x7f698c095e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:15.397 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:15.395 7f699342d700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f698c096490 0x7f698c18f820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:15.397 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:15.395 7f699342d700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f698c096890 con 0x7f698c18ee60 2021-02-10T11:05:15.397 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:15.395 7f699342d700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f698c096a80 con 0x7f698c096490 2021-02-10T11:05:15.398 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:15.395 7f699342d700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f698c18fe40 con 0x7f698c095a70 2021-02-10T11:05:15.398 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:15.395 7f6991249700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f698c18ee60 0x7f698c0a03a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:15.398 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:15.395 7f6991249700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f698c18ee60 0x7f698c0a03a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:50588/0 (socket says 172.21.2.102:50588) 2021-02-10T11:05:15.398 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:15.395 7f6991249700 1 -- 172.21.2.102:0/2826704564 learned_addr learned my addr 172.21.2.102:0/2826704564 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:05:15.398 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:15.395 7f6991249700 1 -- 172.21.2.102:0/2826704564 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f698c095a70 msgr2=0x7f698c095e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:15.398 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:15.395 7f6991249700 1 --2- 172.21.2.102:0/2826704564 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f698c095a70 0x7f698c095e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:05:15.398 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:15.395 7f6991249700 1 -- 172.21.2.102:0/2826704564 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f698c096490 msgr2=0x7f698c18f820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:15.398 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:15.395 7f6991249700 1 --2- 172.21.2.102:0/2826704564 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f698c096490 0x7f698c18f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:05:15.398 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:15.395 7f6991249700 1 -- 172.21.2.102:0/2826704564 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f698c190060 con 0x7f698c18ee60 2021-02-10T11:05:15.398 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:15.396 7f6991249700 1 --2- 172.21.2.102:0/2826704564 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f698c18ee60 0x7f698c0a03a0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 rx=0x7f6988009630 tx=0x7f698800b900).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:05:15.398 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:15.396 7f697dffb700 1 -- 172.21.2.102:0/2826704564 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f69880093a0 con 0x7f698c18ee60 2021-02-10T11:05:15.398 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:15.396 7f697dffb700 1 -- 172.21.2.102:0/2826704564 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f698800bcf0 con 0x7f698c18ee60 2021-02-10T11:05:15.398 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:15.396 7f697dffb700 1 -- 172.21.2.102:0/2826704564 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6988005560 con 0x7f698c18ee60 2021-02-10T11:05:15.399 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:15.396 7f699342d700 1 -- 172.21.2.102:0/2826704564 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f698c190060 con 0x7f698c18ee60 2021-02-10T11:05:15.399 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:15.396 7f699342d700 1 -- 172.21.2.102:0/2826704564 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f698c190060 con 0x7f698c18ee60 2021-02-10T11:05:15.399 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:15.396 7f697dffb700 1 -- 172.21.2.102:0/2826704564 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f6988008240 con 0x7f698c18ee60 2021-02-10T11:05:15.399 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:15.396 7f697dffb700 1 --2- 172.21.2.102:0/2826704564 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f696c0186d0 0x7f696c01abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:15.399 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:15.396 7f697dffb700 1 -- 172.21.2.102:0/2826704564 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f6988032600 con 0x7f698c18ee60 2021-02-10T11:05:15.399 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:15.397 7f6990a48700 1 --2- 172.21.2.102:0/2826704564 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f696c0186d0 0x7f696c01abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:15.399 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:15.397 7f6990a48700 1 --2- 172.21.2.102:0/2826704564 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f696c0186d0 0x7f696c01abd0 crc :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:05:15.425 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:15.426 7f699342d700 1 -- 172.21.2.102:0/2826704564 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f698c084b80 con 0x7f698c18ee60 2021-02-10T11:05:15.427 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:15.429 7f697dffb700 1 -- 172.21.2.102:0/2826704564 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f6988008030 con 0x7f698c18ee60 2021-02-10T11:05:15.557 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:15.559 7f699342d700 1 -- 172.21.2.102:0/2826704564 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f698c07ed80 con 0x7f698c18ee60 2021-02-10T11:05:15.559 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:15.559 7f697dffb700 1 -- 172.21.2.102:0/2826704564 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) v1 ==== 94+0+2101 (secure 0 0 0) 0x7f6988008240 con 0x7f698c18ee60 2021-02-10T11:05:15.559 INFO:teuthology.orchestra.run.gibba002.stderr:dumped fsmap epoch 14 2021-02-10T11:05:15.559 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:05:15.559 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":14,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4845,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.102:6835/3747080008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6834","nonce":3747080008},{"type":"v1","addr":"172.21.2.102:6835","nonce":3747080008}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":10}],"filesystems":[{"mdsmap":{"epoch":14,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:05:12.567145","modified":"2021-02-10 11:05:14.922542","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4510},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4510":{"gid":4510,"name":"b","rank":0,"incarnation":11,"state":"up:active","state_seq":4,"addr":"172.21.2.113:6833/3898434901","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6832","nonce":3898434901},{"type":"v1","addr":"172.21.2.113:6833","nonce":3898434901}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":2}]} 2021-02-10T11:05:15.560 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:15.561 7f699342d700 1 -- 172.21.2.102:0/2826704564 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f696c0186d0 msgr2=0x7f696c01abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:15.561 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:15.561 7f699342d700 1 --2- 172.21.2.102:0/2826704564 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f696c0186d0 0x7f696c01abd0 crc :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:15.561 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:15.561 7f699342d700 1 -- 172.21.2.102:0/2826704564 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f698c18ee60 msgr2=0x7f698c0a03a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:15.561 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:15.561 7f699342d700 1 --2- 172.21.2.102:0/2826704564 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f698c18ee60 0x7f698c0a03a0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 rx=0x7f6988009630 tx=0x7f698800b900).stop 2021-02-10T11:05:15.561 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:15.561 7f699342d700 1 -- 172.21.2.102:0/2826704564 shutdown_connections 2021-02-10T11:05:15.561 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:15.561 7f699342d700 1 --2- 172.21.2.102:0/2826704564 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f696c0186d0 0x7f696c01abd0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:15.561 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:15.561 7f699342d700 1 --2- 172.21.2.102:0/2826704564 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f698c096490 0x7f698c18f820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:05:15.562 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:15.561 7f699342d700 1 --2- 172.21.2.102:0/2826704564 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f698c095a70 0x7f698c095e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:05:15.562 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:15.561 7f699342d700 1 --2- 172.21.2.102:0/2826704564 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f698c18ee60 0x7f698c0a03a0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:15.562 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:15.561 7f699342d700 1 -- 172.21.2.102:0/2826704564 >> 172.21.2.102:0/2826704564 conn(0x7f698c1b2710 msgr2=0x7f698c084db0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:05:15.562 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:15.562 7f699342d700 1 -- 172.21.2.102:0/2826704564 shutdown_connections 2021-02-10T11:05:15.562 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:15.562 7f699342d700 1 -- 172.21.2.102:0/2826704564 wait complete. 2021-02-10T11:05:15.568 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:05:15.998 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:15.982 7fe7b8872700 1 Processor -- start 2021-02-10T11:05:16.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:15.982 7fe7b8872700 1 -- start start 2021-02-10T11:05:16.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:15.983 7fe7b8872700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe7b40e5070 0x7fe7b41b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:16.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:15.983 7fe7b8872700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fe7b40e5470 0x7fe7b407d690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:16.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:15.983 7fe7b8872700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7b407ed80 con 0x7fe7b41b3e40 2021-02-10T11:05:16.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:15.983 7fe7b8872700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7b4080310 con 0x7fe7b40e5470 2021-02-10T11:05:16.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:15.983 7fe7b8872700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7b4080500 con 0x7fe7b40e5070 2021-02-10T11:05:16.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:15.983 7fe7b2e1d700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fe7b41b3e40 0x7fe7b407dcb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:42820/0 (socket says 172.21.2.102:42820) 2021-02-10T11:05:16.052 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:15.983 7fe7b2e1d700 1 -- 172.21.2.102:0/408261447 learned_addr learned my addr 172.21.2.102:0/408261447 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:05:16.052 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:15.983 7fe7b1e1b700 1 --2- 172.21.2.102:0/408261447 >> v2:172.21.2.113:3300/0 conn(0x7fe7b40e5470 0x7fe7b407d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:16.052 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:15.983 7fe7b261c700 1 --2- 172.21.2.102:0/408261447 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe7b40e5070 0x7fe7b41b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:16.052 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:15.983 7fe7b161a700 1 -- 172.21.2.102:0/408261447 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1262092427 0 0) 0x7fe7ac000f90 con 0x7fe7b41b3e40 2021-02-10T11:05:16.052 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:15.983 7fe7b161a700 1 -- 172.21.2.102:0/408261447 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7a0002fe0 con 0x7fe7b41b3e40 2021-02-10T11:05:16.052 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:15.983 7fe7b1e1b700 1 -- 172.21.2.102:0/408261447 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe7b40e5070 msgr2=0x7fe7b41b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:05:16.052 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:15.983 7fe7b1e1b700 1 --2- 172.21.2.102:0/408261447 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe7b40e5070 0x7fe7b41b3820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:16.052 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:15.983 7fe7b1e1b700 1 -- 172.21.2.102:0/408261447 >> v1:172.21.2.102:6789/0 conn(0x7fe7b41b3e40 legacy=0x7fe7b407dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:16.052 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:15.983 7fe7b1e1b700 1 -- 172.21.2.102:0/408261447 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7b40806f0 con 0x7fe7b40e5470 2021-02-10T11:05:16.052 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:15.984 7fe7b261c700 1 --2- 172.21.2.102:0/408261447 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe7b40e5070 0x7fe7b41b3820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:05:16.052 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:15.984 7fe7b1e1b700 1 --2- 172.21.2.102:0/408261447 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe7b40e5470 0x7fe7b407d690 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7fe79c006650 tx=0x7fe79c009a80).ready entity=mon.1 client_cookie=301ad2bda19897eb server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:05:16.052 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:15.984 7fe7b161a700 1 -- 172.21.2.102:0/408261447 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe79c00c0c0 con 0x7fe7b40e5470 2021-02-10T11:05:16.052 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:15.984 7fe7b161a700 1 -- 172.21.2.102:0/408261447 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe79c00c340 con 0x7fe7b40e5470 2021-02-10T11:05:16.053 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:15.984 7fe7b161a700 1 -- 172.21.2.102:0/408261447 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe79c00c860 con 0x7fe7b40e5470 2021-02-10T11:05:16.053 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:15.984 7fe7b8872700 1 -- 172.21.2.102:0/408261447 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe7b40e5470 msgr2=0x7fe7b407d690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:16.053 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:15.984 7fe7b8872700 1 --2- 172.21.2.102:0/408261447 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe7b40e5470 0x7fe7b407d690 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7fe79c006650 tx=0x7fe79c009a80).stop 2021-02-10T11:05:16.053 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:15.985 7fe7b8872700 1 -- 172.21.2.102:0/408261447 shutdown_connections 2021-02-10T11:05:16.053 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:15.985 7fe7b8872700 1 --2- 172.21.2.102:0/408261447 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe7b40e5470 0x7fe7b407d690 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:16.053 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:15.985 7fe7b8872700 1 --2- 172.21.2.102:0/408261447 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe7b40e5070 0x7fe7b41b3820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:16.053 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:15.985 7fe7b8872700 1 -- 172.21.2.102:0/408261447 >> 172.21.2.102:0/408261447 conn(0x7fe7b40ce690 msgr2=0x7fe7b40e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:05:16.053 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:15.985 7fe7b8872700 1 -- 172.21.2.102:0/408261447 shutdown_connections 2021-02-10T11:05:16.053 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:15.985 7fe7b8872700 1 -- 172.21.2.102:0/408261447 wait complete. 2021-02-10T11:05:16.053 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:15.999 7fe7b8872700 1 Processor -- start 2021-02-10T11:05:16.053 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:15.999 7fe7b8872700 1 -- start start 2021-02-10T11:05:16.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:16.000 7fe7b8872700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe7b4198980 0x7fe7b40ad1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:16.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:16.000 7fe7b8872700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe7b40ad7f0 0x7fe7b41af390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:16.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:16.000 7fe7b8872700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe7b4198d80 0x7fe7b41af9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:16.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:16.000 7fe7b8872700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7b4199320 con 0x7fe7b4198980 2021-02-10T11:05:16.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:16.000 7fe7b8872700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7b41b05e0 con 0x7fe7b40ad7f0 2021-02-10T11:05:16.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:16.000 7fe7b8872700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7b41b0800 con 0x7fe7b4198d80 2021-02-10T11:05:16.056 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:16.000 7fe7b261c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe7b4198980 0x7fe7b40ad1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:16.056 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:16.000 7fe7b261c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe7b4198980 0x7fe7b40ad1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:50600/0 (socket says 172.21.2.102:50600) 2021-02-10T11:05:16.056 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:16.000 7fe7b261c700 1 -- 172.21.2.102:0/1431541703 learned_addr learned my addr 172.21.2.102:0/1431541703 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:05:16.056 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:16.000 7fe7b261c700 1 -- 172.21.2.102:0/1431541703 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe7b4198d80 msgr2=0x7fe7b41af9b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:05:16.056 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:16.000 7fe7b261c700 1 --2- 172.21.2.102:0/1431541703 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe7b4198d80 0x7fe7b41af9b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:05:16.056 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:16.000 7fe7b261c700 1 -- 172.21.2.102:0/1431541703 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe7b40ad7f0 msgr2=0x7fe7b41af390 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:05:16.056 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:16.000 7fe7b261c700 1 --2- 172.21.2.102:0/1431541703 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe7b40ad7f0 0x7fe7b41af390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:05:16.056 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:16.000 7fe7b261c700 1 -- 172.21.2.102:0/1431541703 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7b41b0a20 con 0x7fe7b4198980 2021-02-10T11:05:16.056 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:16.000 7fe7b261c700 1 --2- 172.21.2.102:0/1431541703 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe7b4198980 0x7fe7b40ad1d0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 rx=0x7fe7a400b560 tx=0x7fe7a400b900).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:05:16.056 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:16.000 7fe7aaffd700 1 -- 172.21.2.102:0/1431541703 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe7a4006bb0 con 0x7fe7b4198980 2021-02-10T11:05:16.056 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:16.000 7fe7aaffd700 1 -- 172.21.2.102:0/1431541703 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7a4000c90 con 0x7fe7b4198980 2021-02-10T11:05:16.056 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:16.000 7fe7aaffd700 1 -- 172.21.2.102:0/1431541703 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe7a4002220 con 0x7fe7b4198980 2021-02-10T11:05:16.056 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:16.000 7fe7b8872700 1 -- 172.21.2.102:0/1431541703 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7b41b0a20 con 0x7fe7b4198980 2021-02-10T11:05:16.057 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:16.000 7fe7b8872700 1 -- 172.21.2.102:0/1431541703 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7b41b0a20 con 0x7fe7b4198980 2021-02-10T11:05:16.057 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:16.001 7fe7aaffd700 1 -- 172.21.2.102:0/1431541703 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fe7a4006b00 con 0x7fe7b4198980 2021-02-10T11:05:16.057 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:16.001 7fe7aaffd700 1 --2- 172.21.2.102:0/1431541703 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe79801a920 0x7fe79801cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:16.057 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:16.001 7fe7aaffd700 1 -- 172.21.2.102:0/1431541703 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fe7a4032600 con 0x7fe7b4198980 2021-02-10T11:05:16.057 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:16.001 7fe7b1e1b700 1 --2- 172.21.2.102:0/1431541703 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe79801a920 0x7fe79801cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:16.057 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:16.001 7fe7b1e1b700 1 --2- 172.21.2.102:0/1431541703 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe79801a920 0x7fe79801cdd0 crc :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:05:16.057 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:16.033 7fe7b8872700 1 -- 172.21.2.102:0/1431541703 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7b4084b50 con 0x7fe7b4198980 2021-02-10T11:05:16.057 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:16.035 7fe7aaffd700 1 -- 172.21.2.102:0/1431541703 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fe7a4006b00 con 0x7fe7b4198980 2021-02-10T11:05:16.159 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:16.160 7fe7b8872700 1 -- 172.21.2.102:0/1431541703 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe7b4084b50 con 0x7fe7b4198980 2021-02-10T11:05:16.160 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:16.161 7fe7aaffd700 1 -- 172.21.2.102:0/1431541703 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v30) v1 ==== 74+0+13993 (secure 0 0 0) 0x7fe7a4000c90 con 0x7fe7b4198980 2021-02-10T11:05:16.160 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:05:16.161 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":30,"fsid":"68db30ea-dbce-4aef-a6df-f01c6fea0513","created":"2021-02-10 11:03:06.331557","modified":"2021-02-10 11:05:13.569263","last_up_change":"2021-02-10 11:03:32.690562","last_in_change":"2021-02-10 11:03:32.690562","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:03:37.322519","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:05:10.538786","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":5,"pool_name":"cephfs_data","create_time":"2021-02-10 11:05:11.559128","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"30","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"be691b4d-a039-408a-af80-9d53f02b6f1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6818","nonce":36088},{"type":"v1","addr":"172.21.2.102:6820","nonce":36088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6822","nonce":36088},{"type":"v1","addr":"172.21.2.102:6824","nonce":36088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6830","nonce":36088},{"type":"v1","addr":"172.21.2.102:6832","nonce":36088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6826","nonce":36088},{"type":"v1","addr":"172.21.2.102:6828","nonce":36088}]},"public_addr":"172.21.2.102:6820/36088","cluster_addr":"172.21.2.102:6824/36088","heartbeat_back_addr":"172.21.2.102:6832/36088","heartbeat_front_addr":"172.21.2.102:6828/36088","state":["exists","up"]},{"osd":1,"uuid":"64e3ccc8-9c69-4d7c-865e-af8e83822635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6819","nonce":36087},{"type":"v1","addr":"172.21.2.102:6821","nonce":36087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6823","nonce":36087},{"type":"v1","addr":"172.21.2.102:6825","nonce":36087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6831","nonce":36087},{"type":"v1","addr":"172.21.2.102:6833","nonce":36087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6827","nonce":36087},{"type":"v1","addr":"172.21.2.102:6829","nonce":36087}]},"public_addr":"172.21.2.102:6821/36087","cluster_addr":"172.21.2.102:6825/36087","heartbeat_back_addr":"172.21.2.102:6833/36087","heartbeat_front_addr":"172.21.2.102:6829/36087","state":["exists","up"]},{"osd":2,"uuid":"153700af-747f-481f-bdf4-d6072d3a842f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6810","nonce":36089},{"type":"v1","addr":"172.21.2.102:6811","nonce":36089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6812","nonce":36089},{"type":"v1","addr":"172.21.2.102:6813","nonce":36089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6816","nonce":36089},{"type":"v1","addr":"172.21.2.102:6817","nonce":36089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6814","nonce":36089},{"type":"v1","addr":"172.21.2.102:6815","nonce":36089}]},"public_addr":"172.21.2.102:6811/36089","cluster_addr":"172.21.2.102:6813/36089","heartbeat_back_addr":"172.21.2.102:6817/36089","heartbeat_front_addr":"172.21.2.102:6815/36089","state":["exists","up"]},{"osd":3,"uuid":"cbc678f4-83a9-45e6-8830-2f7c232b1ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6802","nonce":36090},{"type":"v1","addr":"172.21.2.102:6803","nonce":36090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6804","nonce":36090},{"type":"v1","addr":"172.21.2.102:6805","nonce":36090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6808","nonce":36090},{"type":"v1","addr":"172.21.2.102:6809","nonce":36090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6806","nonce":36090},{"type":"v1","addr":"172.21.2.102:6807","nonce":36090}]},"public_addr":"172.21.2.102:6803/36090","cluster_addr":"172.21.2.102:6805/36090","heartbeat_back_addr":"172.21.2.102:6809/36090","heartbeat_front_addr":"172.21.2.102:6807/36090","state":["exists","up"]},{"osd":4,"uuid":"c01e4008-b258-4f10-afd2-9e7f37214351","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6808","nonce":36368},{"type":"v1","addr":"172.21.2.113:6811","nonce":36368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6814","nonce":36368},{"type":"v1","addr":"172.21.2.113:6818","nonce":36368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6826","nonce":36368},{"type":"v1","addr":"172.21.2.113:6828","nonce":36368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6820","nonce":36368},{"type":"v1","addr":"172.21.2.113:6824","nonce":36368}]},"public_addr":"172.21.2.113:6811/36368","cluster_addr":"172.21.2.113:6818/36368","heartbeat_back_addr":"172.21.2.113:6828/36368","heartbeat_front_addr":"172.21.2.113:6824/36368","state":["exists","up"]},{"osd":5,"uuid":"551f6007-daf8-4a98-8ee0-c2b16bcccb5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6802","nonce":36366},{"type":"v1","addr":"172.21.2.113:6805","nonce":36366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6807","nonce":36366},{"type":"v1","addr":"172.21.2.113:6810","nonce":36366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6819","nonce":36366},{"type":"v1","addr":"172.21.2.113:6822","nonce":36366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6812","nonce":36366},{"type":"v1","addr":"172.21.2.113:6815","nonce":36366}]},"public_addr":"172.21.2.113:6805/36366","cluster_addr":"172.21.2.113:6810/36366","heartbeat_back_addr":"172.21.2.113:6822/36366","heartbeat_front_addr":"172.21.2.113:6815/36366","state":["exists","up"]},{"osd":6,"uuid":"bf67c040-0e50-48be-b102-c3943bc9ebef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6800","nonce":36367},{"type":"v1","addr":"172.21.2.113:6801","nonce":36367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6803","nonce":36367},{"type":"v1","addr":"172.21.2.113:6804","nonce":36367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6813","nonce":36367},{"type":"v1","addr":"172.21.2.113:6816","nonce":36367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6806","nonce":36367},{"type":"v1","addr":"172.21.2.113:6809","nonce":36367}]},"public_addr":"172.21.2.113:6801/36367","cluster_addr":"172.21.2.113:6804/36367","heartbeat_back_addr":"172.21.2.113:6816/36367","heartbeat_front_addr":"172.21.2.113:6809/36367","state":["exists","up"]},{"osd":7,"uuid":"f5461f80-018d-428f-be89-2eb2bcbe7bb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6817","nonce":36365},{"type":"v1","addr":"172.21.2.113:6821","nonce":36365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6823","nonce":36365},{"type":"v1","addr":"172.21.2.113:6825","nonce":36365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6830","nonce":36365},{"type":"v1","addr":"172.21.2.113:6831","nonce":36365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6827","nonce":36365},{"type":"v1","addr":"172.21.2.113:6829","nonce":36365}]},"public_addr":"172.21.2.113:6821/36365","cluster_addr":"172.21.2.113:6825/36365","heartbeat_back_addr":"172.21.2.113:6831/36365","heartbeat_front_addr":"172.21.2.113:6829/36365","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:05:16.162 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:16.163 7fe7b8872700 1 -- 172.21.2.102:0/1431541703 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe79801a920 msgr2=0x7fe79801cdd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:16.162 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:16.163 7fe7b8872700 1 --2- 172.21.2.102:0/1431541703 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe79801a920 0x7fe79801cdd0 crc :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:16.162 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:16.163 7fe7b8872700 1 -- 172.21.2.102:0/1431541703 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe7b4198980 msgr2=0x7fe7b40ad1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:16.162 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:16.163 7fe7b8872700 1 --2- 172.21.2.102:0/1431541703 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe7b4198980 0x7fe7b40ad1d0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 rx=0x7fe7a400b560 tx=0x7fe7a400b900).stop 2021-02-10T11:05:16.162 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:16.163 7fe7b8872700 1 -- 172.21.2.102:0/1431541703 shutdown_connections 2021-02-10T11:05:16.163 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:16.163 7fe7b8872700 1 --2- 172.21.2.102:0/1431541703 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe7b4198d80 0x7fe7b41af9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:05:16.163 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:16.163 7fe7b8872700 1 --2- 172.21.2.102:0/1431541703 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe79801a920 0x7fe79801cdd0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:16.163 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:16.163 7fe7b8872700 1 --2- 172.21.2.102:0/1431541703 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe7b40ad7f0 0x7fe7b41af390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:05:16.163 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:16.163 7fe7b8872700 1 --2- 172.21.2.102:0/1431541703 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe7b4198980 0x7fe7b40ad1d0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:16.163 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:16.163 7fe7b8872700 1 -- 172.21.2.102:0/1431541703 >> 172.21.2.102:0/1431541703 conn(0x7fe7b4187eb0 msgr2=0x7fe7b41abed0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:05:16.163 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:16.163 7fe7b8872700 1 -- 172.21.2.102:0/1431541703 shutdown_connections 2021-02-10T11:05:16.164 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:16.163 7fe7b8872700 1 -- 172.21.2.102:0/1431541703 wait complete. 2021-02-10T11:05:16.170 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:05:16.609 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:16.593 7f8e34bed700 1 Processor -- start 2021-02-10T11:05:16.609 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:16.593 7f8e34bed700 1 -- start start 2021-02-10T11:05:16.609 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:16.594 7f8e34bed700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f8e301b0470 0x7f8e301b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:16.609 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:16.594 7f8e34bed700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f8e300f20e0 0x7f8e300f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:16.609 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:16.594 7f8e34bed700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e300f0b50 con 0x7f8e300ef610 2021-02-10T11:05:16.609 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:16.594 7f8e34bed700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e300f0820 con 0x7f8e301b0470 2021-02-10T11:05:16.609 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:16.594 7f8e34bed700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e300f24e0 con 0x7f8e300f20e0 2021-02-10T11:05:16.609 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:16.594 7f8e2e85e700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f8e301b0470 0x7f8e301b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:16.610 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:16.594 7f8e2e85e700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f8e301b0470 0x7f8e301b0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:50608/0 (socket says 172.21.2.102:50608) 2021-02-10T11:05:16.610 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:16.594 7f8e2e85e700 1 -- 172.21.2.102:0/1148973455 learned_addr learned my addr 172.21.2.102:0/1148973455 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:05:16.610 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:16.594 7f8e2e05d700 1 --1- 172.21.2.102:0/1148973455 >> v1:172.21.2.102:6789/0 conn(0x7f8e300ef610 0x7f8e300efa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:42832/0 (socket says 172.21.2.102:42832) 2021-02-10T11:05:16.610 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:16.594 7f8e2e85e700 1 -- 172.21.2.102:0/1148973455 >> v2:172.21.2.113:3300/0 conn(0x7f8e300f20e0 msgr2=0x7f8e300f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:05:16.610 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:16.594 7f8e2e85e700 1 --2- 172.21.2.102:0/1148973455 >> v2:172.21.2.113:3300/0 conn(0x7f8e300f20e0 0x7f8e300f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:05:16.610 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:16.594 7f8e2e85e700 1 -- 172.21.2.102:0/1148973455 >> v1:172.21.2.102:6789/0 conn(0x7f8e300ef610 legacy=0x7f8e300efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:16.610 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:16.594 7f8e2e85e700 1 -- 172.21.2.102:0/1148973455 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e300e52e0 con 0x7f8e301b0470 2021-02-10T11:05:16.610 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:16.595 7f8e2e85e700 1 --2- 172.21.2.102:0/1148973455 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8e301b0470 0x7f8e301b0870 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 rx=0x7f8e18006700 tx=0x7f8e18009b80).ready entity=mon.0 client_cookie=6928d79ecc26236d server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:05:16.610 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:16.595 7f8e2d85c700 1 -- 172.21.2.102:0/1148973455 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8e1800c0c0 con 0x7f8e301b0470 2021-02-10T11:05:16.610 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:16.595 7f8e2d85c700 1 -- 172.21.2.102:0/1148973455 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e1800c340 con 0x7f8e301b0470 2021-02-10T11:05:16.610 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:16.595 7f8e2d85c700 1 -- 172.21.2.102:0/1148973455 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8e1800c860 con 0x7f8e301b0470 2021-02-10T11:05:16.611 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:16.595 7f8e34bed700 1 -- 172.21.2.102:0/1148973455 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8e301b0470 msgr2=0x7f8e301b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:16.611 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:16.595 7f8e34bed700 1 --2- 172.21.2.102:0/1148973455 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8e301b0470 0x7f8e301b0870 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 rx=0x7f8e18006700 tx=0x7f8e18009b80).stop 2021-02-10T11:05:16.611 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:16.596 7f8e34bed700 1 -- 172.21.2.102:0/1148973455 shutdown_connections 2021-02-10T11:05:16.611 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:16.596 7f8e34bed700 1 --2- 172.21.2.102:0/1148973455 >> v2:172.21.2.113:3300/0 conn(0x7f8e300f20e0 0x7f8e300f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:05:16.611 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:16.596 7f8e34bed700 1 --2- 172.21.2.102:0/1148973455 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8e301b0470 0x7f8e301b0870 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:16.611 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:16.596 7f8e34bed700 1 -- 172.21.2.102:0/1148973455 >> 172.21.2.102:0/1148973455 conn(0x7f8e300ce3e0 msgr2=0x7f8e300e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:05:16.611 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:16.596 7f8e34bed700 1 -- 172.21.2.102:0/1148973455 shutdown_connections 2021-02-10T11:05:16.611 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:16.596 7f8e34bed700 1 -- 172.21.2.102:0/1148973455 wait complete. 2021-02-10T11:05:16.612 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:16.611 7f8e34bed700 1 Processor -- start 2021-02-10T11:05:16.612 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:16.611 7f8e34bed700 1 -- start start 2021-02-10T11:05:16.613 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:16.611 7f8e34bed700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8e3018ea30 0x7f8e300b17c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:16.613 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:16.611 7f8e34bed700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8e3018ee30 0x7f8e300b1de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:16.613 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:16.611 7f8e34bed700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8e30092f70 0x7f8e30093370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:16.613 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:16.611 7f8e34bed700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e300b2400 con 0x7f8e30092f70 2021-02-10T11:05:16.613 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:16.611 7f8e34bed700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e300b25e0 con 0x7f8e3018ea30 2021-02-10T11:05:16.613 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:16.611 7f8e34bed700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e300939c0 con 0x7f8e3018ee30 2021-02-10T11:05:16.613 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:16.611 7f8e2f05f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8e30092f70 0x7f8e30093370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:16.613 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:16.611 7f8e2f05f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8e30092f70 0x7f8e30093370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:50612/0 (socket says 172.21.2.102:50612) 2021-02-10T11:05:16.613 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:16.611 7f8e2f05f700 1 -- 172.21.2.102:0/220680340 learned_addr learned my addr 172.21.2.102:0/220680340 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:05:16.613 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:16.611 7f8e2f05f700 1 -- 172.21.2.102:0/220680340 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8e3018ee30 msgr2=0x7f8e300b1de0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:05:16.613 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:16.611 7f8e2f05f700 1 --2- 172.21.2.102:0/220680340 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8e3018ee30 0x7f8e300b1de0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:05:16.613 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:16.611 7f8e2f05f700 1 -- 172.21.2.102:0/220680340 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8e3018ea30 msgr2=0x7f8e300b17c0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:05:16.613 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:16.611 7f8e2f05f700 1 --2- 172.21.2.102:0/220680340 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8e3018ea30 0x7f8e300b17c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:05:16.614 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:16.611 7f8e2f05f700 1 -- 172.21.2.102:0/220680340 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e30093be0 con 0x7f8e30092f70 2021-02-10T11:05:16.614 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:16.611 7f8e2f05f700 1 --2- 172.21.2.102:0/220680340 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8e30092f70 0x7f8e30093370 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 rx=0x7f8e28009980 tx=0x7f8e2800bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:05:16.614 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:16.612 7f8e277fe700 1 -- 172.21.2.102:0/220680340 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8e28005870 con 0x7f8e30092f70 2021-02-10T11:05:16.614 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:16.612 7f8e277fe700 1 -- 172.21.2.102:0/220680340 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e28005a70 con 0x7f8e30092f70 2021-02-10T11:05:16.614 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:16.612 7f8e34bed700 1 -- 172.21.2.102:0/220680340 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e30093be0 con 0x7f8e30092f70 2021-02-10T11:05:16.614 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:16.612 7f8e34bed700 1 -- 172.21.2.102:0/220680340 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e30093be0 con 0x7f8e30092f70 2021-02-10T11:05:16.614 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:16.612 7f8e277fe700 1 -- 172.21.2.102:0/220680340 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8e28006030 con 0x7f8e30092f70 2021-02-10T11:05:16.614 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:16.612 7f8e277fe700 1 -- 172.21.2.102:0/220680340 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f8e28005c50 con 0x7f8e30092f70 2021-02-10T11:05:16.614 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:16.612 7f8e277fe700 1 --2- 172.21.2.102:0/220680340 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8e14018730 0x7f8e1401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:16.614 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:16.612 7f8e277fe700 1 -- 172.21.2.102:0/220680340 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f8e28032b30 con 0x7f8e30092f70 2021-02-10T11:05:16.614 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:16.612 7f8e2e85e700 1 --2- 172.21.2.102:0/220680340 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8e14018730 0x7f8e1401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:16.614 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:16.613 7f8e2e85e700 1 --2- 172.21.2.102:0/220680340 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8e14018730 0x7f8e1401ac10 crc :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:05:16.642 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:16.643 7f8e34bed700 1 -- 172.21.2.102:0/220680340 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e30105a30 con 0x7f8e30092f70 2021-02-10T11:05:16.644 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:16.646 7f8e277fe700 1 -- 172.21.2.102:0/220680340 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f8e2800be10 con 0x7f8e30092f70 2021-02-10T11:05:16.777 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:16.778 7f8e34bed700 1 -- 172.21.2.102:0/220680340 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f8e30105a30 con 0x7f8e30092f70 2021-02-10T11:05:16.780 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:16.781 7f8e277fe700 1 -- 172.21.2.102:0/220680340 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.0 v11) v1 ==== 184+0+0 (secure 0 0 0) 0x7f8e28006f20 con 0x7f8e30092f70 2021-02-10T11:05:16.784 INFO:teuthology.orchestra.run.gibba002.stderr:updated caps for client.0 2021-02-10T11:05:16.785 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:16.786 7f8e34bed700 1 -- 172.21.2.102:0/220680340 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8e14018730 msgr2=0x7f8e1401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:16.785 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:16.786 7f8e34bed700 1 --2- 172.21.2.102:0/220680340 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8e14018730 0x7f8e1401ac10 crc :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:16.785 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:16.786 7f8e34bed700 1 -- 172.21.2.102:0/220680340 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8e30092f70 msgr2=0x7f8e30093370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:16.785 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:16.786 7f8e34bed700 1 --2- 172.21.2.102:0/220680340 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8e30092f70 0x7f8e30093370 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 rx=0x7f8e28009980 tx=0x7f8e2800bcd0).stop 2021-02-10T11:05:16.785 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:16.787 7f8e34bed700 1 -- 172.21.2.102:0/220680340 shutdown_connections 2021-02-10T11:05:16.786 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:16.787 7f8e34bed700 1 --2- 172.21.2.102:0/220680340 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8e30092f70 0x7f8e30093370 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:16.786 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:16.787 7f8e34bed700 1 --2- 172.21.2.102:0/220680340 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8e3018ee30 0x7f8e300b1de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:05:16.786 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:16.787 7f8e34bed700 1 --2- 172.21.2.102:0/220680340 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8e14018730 0x7f8e1401ac10 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:16.786 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:16.787 7f8e34bed700 1 --2- 172.21.2.102:0/220680340 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8e3018ea30 0x7f8e300b17c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:05:16.786 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:16.787 7f8e34bed700 1 -- 172.21.2.102:0/220680340 >> 172.21.2.102:0/220680340 conn(0x7f8e301a3790 msgr2=0x7f8e301aff80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:05:16.786 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:16.787 7f8e34bed700 1 -- 172.21.2.102:0/220680340 shutdown_connections 2021-02-10T11:05:16.786 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:16.787 7f8e34bed700 1 -- 172.21.2.102:0/220680340 wait complete. 2021-02-10T11:05:16.793 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.1 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:05:17.233 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:17.216 7f3ebaed8700 1 Processor -- start 2021-02-10T11:05:17.233 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:17.216 7f3ebaed8700 1 -- start start 2021-02-10T11:05:17.233 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:17.216 7f3ebaed8700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f3eb41b0870 0x7f3eb40f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:17.233 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:17.216 7f3ebaed8700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3eb40efe00 0x7f3eb40f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:17.233 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:17.216 7f3ebaed8700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3eb41b0c70 con 0x7f3eb41b0470 2021-02-10T11:05:17.233 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:17.216 7f3ebaed8700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3eb40f0b50 con 0x7f3eb41b0870 2021-02-10T11:05:17.233 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:17.216 7f3ebaed8700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3eb40f0820 con 0x7f3eb40efe00 2021-02-10T11:05:17.233 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:17.216 7f3eb8cf4700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f3eb41b0470 0x7f3eb40ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:42844/0 (socket says 172.21.2.102:42844) 2021-02-10T11:05:17.233 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:17.216 7f3eb8cf4700 1 -- 172.21.2.102:0/3249545959 learned_addr learned my addr 172.21.2.102:0/3249545959 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:05:17.233 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:17.217 7f3eb37fe700 1 -- 172.21.2.102:0/3249545959 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2081626865 0 0) 0x7f3e9c0009b0 con 0x7f3eb41b0470 2021-02-10T11:05:17.234 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:17.217 7f3eb37fe700 1 -- 172.21.2.102:0/3249545959 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ea0002fe0 con 0x7f3eb41b0470 2021-02-10T11:05:17.234 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:17.217 7f3eb3fff700 1 --2- 172.21.2.102:0/3249545959 >> v2:172.21.2.113:3300/0 conn(0x7f3eb41b0870 0x7f3eb40f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:17.234 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:17.217 7f3eb37fe700 1 -- 172.21.2.102:0/3249545959 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3158792574 0 0) 0x7f3e9c001010 con 0x7f3eb41b0470 2021-02-10T11:05:17.234 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:17.217 7f3eb37fe700 1 -- 172.21.2.102:0/3249545959 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3eb40efe00 msgr2=0x7f3eb40f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:05:17.234 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:17.217 7f3eb94f5700 1 --2- 172.21.2.102:0/3249545959 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3eb40efe00 0x7f3eb40f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:17.234 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:17.217 7f3eb37fe700 1 --2- 172.21.2.102:0/3249545959 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3eb40efe00 0x7f3eb40f0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:17.234 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:17.217 7f3eb37fe700 1 -- 172.21.2.102:0/3249545959 >> v2:172.21.2.113:3300/0 conn(0x7f3eb41b0870 msgr2=0x7f3eb40f20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:05:17.234 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:17.217 7f3eb37fe700 1 --2- 172.21.2.102:0/3249545959 >> v2:172.21.2.113:3300/0 conn(0x7f3eb41b0870 0x7f3eb40f20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:17.234 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:17.217 7f3eb37fe700 1 -- 172.21.2.102:0/3249545959 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3eb40e52e0 con 0x7f3eb41b0470 2021-02-10T11:05:17.234 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:17.217 7f3eb37fe700 1 -- 172.21.2.102:0/3249545959 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7f3e9c001530 con 0x7f3eb41b0470 2021-02-10T11:05:17.234 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:17.217 7f3eb37fe700 1 -- 172.21.2.102:0/3249545959 >> v1:172.21.2.102:6789/0 conn(0x7f3eb41b0470 legacy=0x7f3eb40ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:17.235 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:17.217 7f3eb37fe700 1 --2- 172.21.2.102:0/3249545959 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3ea0003b90 0x7f3ea00060b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:17.235 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:17.217 7f3eb37fe700 1 --2- 172.21.2.102:0/3249545959 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3ea00068d0 0x7f3ea0008df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:17.235 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:17.217 7f3eb37fe700 1 --2- 172.21.2.102:0/3249545959 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3ea00096f0 0x7f3ea000bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:17.235 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:17.217 7f3eb37fe700 1 -- 172.21.2.102:0/3249545959 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ea000c430 con 0x7f3ea00068d0 2021-02-10T11:05:17.235 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:17.217 7f3eb37fe700 1 -- 172.21.2.102:0/3249545959 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ea000c650 con 0x7f3ea0003b90 2021-02-10T11:05:17.235 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:17.217 7f3eb37fe700 1 -- 172.21.2.102:0/3249545959 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ea000c870 con 0x7f3ea00096f0 2021-02-10T11:05:17.235 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:17.217 7f3eb94f5700 1 --2- 172.21.2.102:0/3249545959 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3eb40efe00 0x7f3eb40f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:05:17.235 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:17.217 7f3eb3fff700 1 --2- 172.21.2.102:0/3249545959 >> v2:172.21.2.113:3300/0 conn(0x7f3eb41b0870 0x7f3eb40f20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:05:17.235 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:17.217 7f3eb3fff700 1 --2- 172.21.2.102:0/3249545959 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3ea00068d0 0x7f3ea0008df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:17.235 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:17.217 7f3eb8cf4700 1 --2- 172.21.2.102:0/3249545959 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3ea0003b90 0x7f3ea00060b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:17.235 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:17.218 7f3eb94f5700 1 --2- 172.21.2.102:0/3249545959 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3ea00096f0 0x7f3ea000bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:17.235 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:17.218 7f3eb3fff700 1 -- 172.21.2.102:0/3249545959 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3ea00096f0 msgr2=0x7f3ea000bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:05:17.235 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:17.218 7f3eb3fff700 1 --2- 172.21.2.102:0/3249545959 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3ea00096f0 0x7f3ea000bbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:17.236 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:17.218 7f3eb3fff700 1 -- 172.21.2.102:0/3249545959 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3ea0003b90 msgr2=0x7f3ea00060b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:05:17.236 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:17.218 7f3eb3fff700 1 --2- 172.21.2.102:0/3249545959 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3ea0003b90 0x7f3ea00060b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:17.236 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:17.218 7f3eb3fff700 1 -- 172.21.2.102:0/3249545959 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ea000cb10 con 0x7f3ea00068d0 2021-02-10T11:05:17.236 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:17.218 7f3eb3fff700 1 --2- 172.21.2.102:0/3249545959 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3ea00068d0 0x7f3ea0008df0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 rx=0x7f3ea4006fc0 tx=0x7f3ea400b050).ready entity=mon.0 client_cookie=353db344f692dadb server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:05:17.236 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:17.218 7f3eb37fe700 1 -- 172.21.2.102:0/3249545959 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3ea4006c90 con 0x7f3ea00068d0 2021-02-10T11:05:17.236 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:17.218 7f3eb37fe700 1 -- 172.21.2.102:0/3249545959 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ea4006240 con 0x7f3ea00068d0 2021-02-10T11:05:17.236 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:17.218 7f3eb37fe700 1 -- 172.21.2.102:0/3249545959 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3ea4005660 con 0x7f3ea00068d0 2021-02-10T11:05:17.236 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:17.218 7f3ebaed8700 1 -- 172.21.2.102:0/3249545959 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3ea00068d0 msgr2=0x7f3ea0008df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:17.236 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:17.218 7f3ebaed8700 1 --2- 172.21.2.102:0/3249545959 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3ea00068d0 0x7f3ea0008df0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 rx=0x7f3ea4006fc0 tx=0x7f3ea400b050).stop 2021-02-10T11:05:17.236 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:17.218 7f3eb94f5700 1 -- 172.21.2.102:0/3249545959 reap_dead start 2021-02-10T11:05:17.236 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:17.219 7f3ebaed8700 1 -- 172.21.2.102:0/3249545959 shutdown_connections 2021-02-10T11:05:17.236 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:17.219 7f3ebaed8700 1 -- 172.21.2.102:0/3249545959 >> 172.21.2.102:0/3249545959 conn(0x7f3eb40ce3e0 msgr2=0x7f3eb40e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:05:17.236 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:17.219 7f3ebaed8700 1 -- 172.21.2.102:0/3249545959 shutdown_connections 2021-02-10T11:05:17.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:17.219 7f3ebaed8700 1 -- 172.21.2.102:0/3249545959 wait complete. 2021-02-10T11:05:17.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:17.234 7f3ebaed8700 1 Processor -- start 2021-02-10T11:05:17.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:17.234 7f3ebaed8700 1 -- start start 2021-02-10T11:05:17.238 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:17.234 7f3ebaed8700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3eb40b8530 0x7f3eb40b8930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:17.238 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:17.234 7f3ebaed8700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3eb40b8f50 0x7f3eb40b9350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:17.238 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:17.234 7f3ebaed8700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3eb40b65c0 0x7f3eb40b69c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:17.238 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:17.234 7f3ebaed8700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3eb40b99f0 con 0x7f3eb40b65c0 2021-02-10T11:05:17.239 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:17.234 7f3ebaed8700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3eb40b8020 con 0x7f3eb40b8f50 2021-02-10T11:05:17.239 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:17.234 7f3ebaed8700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3eb4091650 con 0x7f3eb40b8530 2021-02-10T11:05:17.239 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:17.235 7f3eb94f5700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3eb40b65c0 0x7f3eb40b69c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:17.239 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:17.235 7f3eb94f5700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3eb40b65c0 0x7f3eb40b69c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:50630/0 (socket says 172.21.2.102:50630) 2021-02-10T11:05:17.239 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:17.235 7f3eb94f5700 1 -- 172.21.2.102:0/3874342479 learned_addr learned my addr 172.21.2.102:0/3874342479 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:05:17.239 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:17.235 7f3eb94f5700 1 -- 172.21.2.102:0/3874342479 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3eb40b8530 msgr2=0x7f3eb40b8930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:17.239 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:17.235 7f3eb94f5700 1 --2- 172.21.2.102:0/3874342479 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3eb40b8530 0x7f3eb40b8930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:05:17.239 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:17.235 7f3eb94f5700 1 -- 172.21.2.102:0/3874342479 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3eb40b8f50 msgr2=0x7f3eb40b9350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:17.239 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:17.235 7f3eb94f5700 1 --2- 172.21.2.102:0/3874342479 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3eb40b8f50 0x7f3eb40b9350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:05:17.239 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:17.235 7f3eb94f5700 1 -- 172.21.2.102:0/3874342479 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3eb40b6fe0 con 0x7f3eb40b65c0 2021-02-10T11:05:17.239 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:17.235 7f3eb94f5700 1 --2- 172.21.2.102:0/3874342479 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3eb40b65c0 0x7f3eb40b69c0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 rx=0x7f3eac0067f0 tx=0x7f3eac050960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:05:17.239 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:17.235 7f3eb17fa700 1 -- 172.21.2.102:0/3874342479 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3eac050d50 con 0x7f3eb40b65c0 2021-02-10T11:05:17.240 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:17.235 7f3eb17fa700 1 -- 172.21.2.102:0/3874342479 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3eac000c40 con 0x7f3eb40b65c0 2021-02-10T11:05:17.240 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:17.235 7f3eb17fa700 1 -- 172.21.2.102:0/3874342479 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3eac005330 con 0x7f3eb40b65c0 2021-02-10T11:05:17.240 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:17.235 7f3ebaed8700 1 -- 172.21.2.102:0/3874342479 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3eb40b6fe0 con 0x7f3eb40b65c0 2021-02-10T11:05:17.240 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:17.236 7f3ebaed8700 1 -- 172.21.2.102:0/3874342479 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3eb40b6fe0 con 0x7f3eb40b65c0 2021-02-10T11:05:17.240 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:17.236 7f3eb17fa700 1 -- 172.21.2.102:0/3874342479 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f3eac006030 con 0x7f3eb40b65c0 2021-02-10T11:05:17.240 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:17.236 7f3eb17fa700 1 --2- 172.21.2.102:0/3874342479 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3e980186c0 0x7f3e9801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:17.240 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:17.236 7f3eb17fa700 1 -- 172.21.2.102:0/3874342479 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f3eac076570 con 0x7f3eb40b65c0 2021-02-10T11:05:17.240 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:17.236 7f3eb8cf4700 1 --2- 172.21.2.102:0/3874342479 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3e980186c0 0x7f3e9801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:17.240 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:17.237 7f3eb8cf4700 1 --2- 172.21.2.102:0/3874342479 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3e980186c0 0x7f3e9801abc0 crc :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:05:17.264 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:17.266 7f3ebaed8700 1 -- 172.21.2.102:0/3874342479 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3eb4105a30 con 0x7f3eb40b65c0 2021-02-10T11:05:17.267 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:17.268 7f3eb17fa700 1 -- 172.21.2.102:0/3874342479 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f3eac070ab0 con 0x7f3eb40b65c0 2021-02-10T11:05:17.396 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:17.397 7f3ebaed8700 1 -- 172.21.2.102:0/3874342479 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f3eb40f0b50 con 0x7f3eb40b65c0 2021-02-10T11:05:17.399 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:17.400 7f3eb17fa700 1 -- 172.21.2.102:0/3874342479 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.1 v12) v1 ==== 184+0+0 (secure 0 0 0) 0x7f3eac070ab0 con 0x7f3eb40b65c0 2021-02-10T11:05:17.403 INFO:teuthology.orchestra.run.gibba002.stderr:updated caps for client.1 2021-02-10T11:05:17.404 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:17.405 7f3ebaed8700 1 -- 172.21.2.102:0/3874342479 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3e980186c0 msgr2=0x7f3e9801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:17.404 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:17.405 7f3ebaed8700 1 --2- 172.21.2.102:0/3874342479 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3e980186c0 0x7f3e9801abc0 crc :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:17.404 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:17.405 7f3ebaed8700 1 -- 172.21.2.102:0/3874342479 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3eb40b65c0 msgr2=0x7f3eb40b69c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:17.404 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:17.405 7f3ebaed8700 1 --2- 172.21.2.102:0/3874342479 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3eb40b65c0 0x7f3eb40b69c0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 rx=0x7f3eac0067f0 tx=0x7f3eac050960).stop 2021-02-10T11:05:17.405 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:17.406 7f3ebaed8700 1 -- 172.21.2.102:0/3874342479 shutdown_connections 2021-02-10T11:05:17.405 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:17.406 7f3ebaed8700 1 --2- 172.21.2.102:0/3874342479 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3eb40b65c0 0x7f3eb40b69c0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:17.405 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:17.406 7f3ebaed8700 1 --2- 172.21.2.102:0/3874342479 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3e980186c0 0x7f3e9801abc0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:17.405 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:17.406 7f3ebaed8700 1 --2- 172.21.2.102:0/3874342479 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3eb40b8f50 0x7f3eb40b9350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:05:17.405 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:17.406 7f3ebaed8700 1 --2- 172.21.2.102:0/3874342479 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3eb40b8530 0x7f3eb40b8930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:05:17.405 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:17.406 7f3ebaed8700 1 -- 172.21.2.102:0/3874342479 >> 172.21.2.102:0/3874342479 conn(0x7f3eb40b2800 msgr2=0x7f3eb4091a50 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:05:17.405 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:17.406 7f3ebaed8700 1 -- 172.21.2.102:0/3874342479 shutdown_connections 2021-02-10T11:05:17.405 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:17.406 7f3ebaed8700 1 -- 172.21.2.102:0/3874342479 wait complete. 2021-02-10T11:05:17.413 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.2 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:05:17.849 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:17.834 7fab8da3c700 1 Processor -- start 2021-02-10T11:05:17.850 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:17.835 7fab8da3c700 1 -- start start 2021-02-10T11:05:17.850 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:17.835 7fab8da3c700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fab881b0470 0x7fab881b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:17.850 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:17.835 7fab8da3c700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fab880ef610 0x7fab880efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:17.850 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:17.835 7fab8da3c700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab880f0b50 con 0x7fab880f0030 2021-02-10T11:05:17.850 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:17.835 7fab8da3c700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab880e52e0 con 0x7fab880ef610 2021-02-10T11:05:17.850 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:17.835 7fab8da3c700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab880e5490 con 0x7fab881b0470 2021-02-10T11:05:17.850 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:17.835 7fab86ffd700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fab880ef610 0x7fab880efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:17.850 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:17.835 7fab87fff700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fab880f0030 0x7fab880f10d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:42862/0 (socket says 172.21.2.102:42862) 2021-02-10T11:05:17.851 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:17.835 7fab86ffd700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fab880ef610 0x7fab880efa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:50638/0 (socket says 172.21.2.102:50638) 2021-02-10T11:05:17.851 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:17.835 7fab87fff700 1 -- 172.21.2.102:0/1198439033 learned_addr learned my addr 172.21.2.102:0/1198439033 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:05:17.851 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:17.835 7fab867fc700 1 -- 172.21.2.102:0/1198439033 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 109849648 0 0) 0x7fab80001500 con 0x7fab880f0030 2021-02-10T11:05:17.851 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:17.835 7fab867fc700 1 -- 172.21.2.102:0/1198439033 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab74002fe0 con 0x7fab880f0030 2021-02-10T11:05:17.851 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:17.836 7fab86ffd700 1 -- 172.21.2.102:0/1198439033 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fab881b0470 msgr2=0x7fab881b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:05:17.851 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:17.836 7fab86ffd700 1 --2- 172.21.2.102:0/1198439033 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fab881b0470 0x7fab881b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:05:17.851 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:17.836 7fab86ffd700 1 -- 172.21.2.102:0/1198439033 >> v1:172.21.2.102:6789/0 conn(0x7fab880f0030 legacy=0x7fab880f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:17.851 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:17.836 7fab86ffd700 1 -- 172.21.2.102:0/1198439033 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab880f0430 con 0x7fab880ef610 2021-02-10T11:05:17.851 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:17.836 7fab86ffd700 1 --2- 172.21.2.102:0/1198439033 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fab880ef610 0x7fab880efa10 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 rx=0x7fab70006650 tx=0x7fab70009a80).ready entity=mon.0 client_cookie=be5823241f78a67d server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:05:17.851 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:17.836 7fab867fc700 1 -- 172.21.2.102:0/1198439033 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fab700009b0 con 0x7fab880ef610 2021-02-10T11:05:17.852 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:17.836 7fab867fc700 1 -- 172.21.2.102:0/1198439033 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab7000c0c0 con 0x7fab880ef610 2021-02-10T11:05:17.852 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:17.836 7fab867fc700 1 -- 172.21.2.102:0/1198439033 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fab7000c5e0 con 0x7fab880ef610 2021-02-10T11:05:17.852 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:17.836 7fab8da3c700 1 -- 172.21.2.102:0/1198439033 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fab880ef610 msgr2=0x7fab880efa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:17.852 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:17.836 7fab8da3c700 1 --2- 172.21.2.102:0/1198439033 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fab880ef610 0x7fab880efa10 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 rx=0x7fab70006650 tx=0x7fab70009a80).stop 2021-02-10T11:05:17.852 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:17.837 7fab8da3c700 1 -- 172.21.2.102:0/1198439033 shutdown_connections 2021-02-10T11:05:17.852 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:17.837 7fab8da3c700 1 --2- 172.21.2.102:0/1198439033 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fab880ef610 0x7fab880efa10 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:17.852 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:17.837 7fab8da3c700 1 --2- 172.21.2.102:0/1198439033 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fab881b0470 0x7fab881b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:05:17.852 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:17.837 7fab8da3c700 1 -- 172.21.2.102:0/1198439033 >> 172.21.2.102:0/1198439033 conn(0x7fab880ce3e0 msgr2=0x7fab880e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:05:17.852 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:17.837 7fab8da3c700 1 -- 172.21.2.102:0/1198439033 shutdown_connections 2021-02-10T11:05:17.852 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:17.837 7fab8da3c700 1 -- 172.21.2.102:0/1198439033 wait complete. 2021-02-10T11:05:17.854 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:17.851 7fab8da3c700 1 Processor -- start 2021-02-10T11:05:17.854 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:17.851 7fab8da3c700 1 -- start start 2021-02-10T11:05:17.854 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:17.852 7fab8da3c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fab8818ed50 0x7fab880ad800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:17.854 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:17.852 7fab8da3c700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fab880ade20 0x7fab881b6dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:17.854 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:17.852 7fab8da3c700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fab8819c2d0 0x7fab881b73f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:17.855 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:17.852 7fab8da3c700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab8818e7d0 con 0x7fab8818ed50 2021-02-10T11:05:17.855 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:17.852 7fab8da3c700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab881b7a10 con 0x7fab880ade20 2021-02-10T11:05:17.855 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:17.852 7fab8da3c700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab881b7bf0 con 0x7fab8819c2d0 2021-02-10T11:05:17.855 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:17.852 7fab877fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fab8818ed50 0x7fab880ad800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:17.855 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:17.852 7fab877fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fab8818ed50 0x7fab880ad800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:50646/0 (socket says 172.21.2.102:50646) 2021-02-10T11:05:17.855 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:17.852 7fab877fe700 1 -- 172.21.2.102:0/177370445 learned_addr learned my addr 172.21.2.102:0/177370445 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:05:17.855 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:17.852 7fab877fe700 1 -- 172.21.2.102:0/177370445 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fab8819c2d0 msgr2=0x7fab881b73f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:17.855 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:17.852 7fab877fe700 1 --2- 172.21.2.102:0/177370445 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fab8819c2d0 0x7fab881b73f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:05:17.855 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:17.852 7fab877fe700 1 -- 172.21.2.102:0/177370445 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fab880ade20 msgr2=0x7fab881b6dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:17.855 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:17.852 7fab877fe700 1 --2- 172.21.2.102:0/177370445 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fab880ade20 0x7fab881b6dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:05:17.855 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:17.852 7fab877fe700 1 -- 172.21.2.102:0/177370445 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab8818cee0 con 0x7fab8818ed50 2021-02-10T11:05:17.856 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:17.852 7fab877fe700 1 --2- 172.21.2.102:0/177370445 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fab8818ed50 0x7fab880ad800 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 rx=0x7fab78006f00 tx=0x7fab7800b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:05:17.856 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:17.853 7fab7ffff700 1 -- 172.21.2.102:0/177370445 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fab78006350 con 0x7fab8818ed50 2021-02-10T11:05:17.856 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:17.853 7fab7ffff700 1 -- 172.21.2.102:0/177370445 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab78006b90 con 0x7fab8818ed50 2021-02-10T11:05:17.856 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:17.853 7fab7ffff700 1 -- 172.21.2.102:0/177370445 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fab78000c90 con 0x7fab8818ed50 2021-02-10T11:05:17.856 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:17.853 7fab8da3c700 1 -- 172.21.2.102:0/177370445 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab8818cee0 con 0x7fab8818ed50 2021-02-10T11:05:17.856 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:17.853 7fab8da3c700 1 -- 172.21.2.102:0/177370445 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab8818d470 con 0x7fab8818ed50 2021-02-10T11:05:17.856 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:17.853 7fab7ffff700 1 -- 172.21.2.102:0/177370445 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fab78007000 con 0x7fab8818ed50 2021-02-10T11:05:17.856 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:17.853 7fab7ffff700 1 --2- 172.21.2.102:0/177370445 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fab6c0186c0 0x7fab6c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:17.856 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:17.853 7fab7ffff700 1 -- 172.21.2.102:0/177370445 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fab78032560 con 0x7fab8818ed50 2021-02-10T11:05:17.856 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:17.854 7fab86ffd700 1 --2- 172.21.2.102:0/177370445 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fab6c0186c0 0x7fab6c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:17.856 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:17.854 7fab86ffd700 1 --2- 172.21.2.102:0/177370445 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fab6c0186c0 0x7fab6c01abc0 crc :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:05:17.883 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:17.885 7fab8da3c700 1 -- 172.21.2.102:0/177370445 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab881059f0 con 0x7fab8818ed50 2021-02-10T11:05:17.886 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:17.887 7fab7ffff700 1 -- 172.21.2.102:0/177370445 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fab78007ef0 con 0x7fab8818ed50 2021-02-10T11:05:18.016 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:18.018 7fab8da3c700 1 -- 172.21.2.102:0/177370445 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7fab880f0b50 con 0x7fab8818ed50 2021-02-10T11:05:18.020 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:18.022 7fab7ffff700 1 -- 172.21.2.102:0/177370445 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.2 v13) v1 ==== 184+0+0 (secure 0 0 0) 0x7fab78007000 con 0x7fab8818ed50 2021-02-10T11:05:18.023 INFO:teuthology.orchestra.run.gibba002.stderr:updated caps for client.2 2021-02-10T11:05:18.025 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:18.027 7fab8da3c700 1 -- 172.21.2.102:0/177370445 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fab6c0186c0 msgr2=0x7fab6c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:18.025 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:18.027 7fab8da3c700 1 --2- 172.21.2.102:0/177370445 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fab6c0186c0 0x7fab6c01abc0 crc :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:18.025 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:18.027 7fab8da3c700 1 -- 172.21.2.102:0/177370445 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fab8818ed50 msgr2=0x7fab880ad800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:18.025 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:18.027 7fab8da3c700 1 --2- 172.21.2.102:0/177370445 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fab8818ed50 0x7fab880ad800 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 rx=0x7fab78006f00 tx=0x7fab7800b940).stop 2021-02-10T11:05:18.026 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:18.027 7fab8da3c700 1 -- 172.21.2.102:0/177370445 shutdown_connections 2021-02-10T11:05:18.026 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:18.027 7fab8da3c700 1 --2- 172.21.2.102:0/177370445 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fab8819c2d0 0x7fab881b73f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:05:18.026 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:18.027 7fab8da3c700 1 --2- 172.21.2.102:0/177370445 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fab6c0186c0 0x7fab6c01abc0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:18.026 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:18.027 7fab8da3c700 1 --2- 172.21.2.102:0/177370445 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fab880ade20 0x7fab881b6dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:05:18.026 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:18.027 7fab8da3c700 1 --2- 172.21.2.102:0/177370445 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fab8818ed50 0x7fab880ad800 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:18.027 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:18.027 7fab8da3c700 1 -- 172.21.2.102:0/177370445 >> 172.21.2.102:0/177370445 conn(0x7fab881a3790 msgr2=0x7fab8819f670 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:05:18.027 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:18.027 7fab8da3c700 1 -- 172.21.2.102:0/177370445 shutdown_connections 2021-02-10T11:05:18.027 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:18.027 7fab8da3c700 1 -- 172.21.2.102:0/177370445 wait complete. 2021-02-10T11:05:18.034 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.3 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:05:18.470 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:18.456 7f550ca61700 1 Processor -- start 2021-02-10T11:05:18.471 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:18.456 7f550ca61700 1 -- start start 2021-02-10T11:05:18.471 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:18.456 7f550ca61700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f55081b0470 0x7f55081b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:18.471 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:18.456 7f550ca61700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f55080ef610 0x7f55080efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:18.471 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:18.456 7f550ca61700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f55080f0030 0x7f55080f0430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:18.471 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:18.456 7f550ca61700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55080e5ea0 con 0x7f55081b0470 2021-02-10T11:05:18.471 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:18.456 7f550ca61700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55080f0b50 con 0x7f55080ef610 2021-02-10T11:05:18.471 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:18.456 7f550ca61700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55080e52e0 con 0x7f55080f0030 2021-02-10T11:05:18.471 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:18.456 7f550685e700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f55081b0470 0x7f55081b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:18.471 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:18.456 7f550685e700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f55081b0470 0x7f55081b0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:50650/0 (socket says 172.21.2.102:50650) 2021-02-10T11:05:18.471 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:18.456 7f550685e700 1 -- 172.21.2.102:0/856554337 learned_addr learned my addr 172.21.2.102:0/856554337 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:05:18.472 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:18.457 7f550605d700 1 --2- 172.21.2.102:0/856554337 >> v2:172.21.2.113:3300/0 conn(0x7f55080ef610 0x7f55080efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:18.472 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:18.457 7f550685e700 1 -- 172.21.2.102:0/856554337 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f55080f0030 msgr2=0x7f55080f0430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:05:18.472 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:18.457 7f550685e700 1 --2- 172.21.2.102:0/856554337 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f55080f0030 0x7f55080f0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:05:18.472 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:18.457 7f550685e700 1 -- 172.21.2.102:0/856554337 >> v2:172.21.2.113:3300/0 conn(0x7f55080ef610 msgr2=0x7f55080efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:05:18.472 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:18.457 7f550685e700 1 --2- 172.21.2.102:0/856554337 >> v2:172.21.2.113:3300/0 conn(0x7f55080ef610 0x7f55080efa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:18.472 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:18.457 7f550685e700 1 -- 172.21.2.102:0/856554337 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55080f10d0 con 0x7f55081b0470 2021-02-10T11:05:18.472 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:18.457 7f550605d700 1 --2- 172.21.2.102:0/856554337 >> v2:172.21.2.113:3300/0 conn(0x7f55080ef610 0x7f55080efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:05:18.472 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:18.457 7f550685e700 1 --2- 172.21.2.102:0/856554337 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f55081b0470 0x7f55081b0870 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 rx=0x7f54f8006700 tx=0x7f54f8009b80).ready entity=mon.0 client_cookie=ed447b7ca1002d00 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:05:18.472 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:18.457 7f550585c700 1 -- 172.21.2.102:0/856554337 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f54f800c0c0 con 0x7f55081b0470 2021-02-10T11:05:18.472 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:18.457 7f550585c700 1 -- 172.21.2.102:0/856554337 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54f800c340 con 0x7f55081b0470 2021-02-10T11:05:18.472 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:18.457 7f550585c700 1 -- 172.21.2.102:0/856554337 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f54f800c860 con 0x7f55081b0470 2021-02-10T11:05:18.472 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:18.457 7f550ca61700 1 -- 172.21.2.102:0/856554337 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f55081b0470 msgr2=0x7f55081b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:18.473 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:18.457 7f550ca61700 1 --2- 172.21.2.102:0/856554337 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f55081b0470 0x7f55081b0870 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 rx=0x7f54f8006700 tx=0x7f54f8009b80).stop 2021-02-10T11:05:18.473 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:18.458 7f550ca61700 1 -- 172.21.2.102:0/856554337 shutdown_connections 2021-02-10T11:05:18.473 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:18.458 7f550ca61700 1 --2- 172.21.2.102:0/856554337 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f55080f0030 0x7f55080f0430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:05:18.473 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:18.458 7f550ca61700 1 --2- 172.21.2.102:0/856554337 >> v2:172.21.2.113:3300/0 conn(0x7f55080ef610 0x7f55080efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:18.473 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:18.458 7f550ca61700 1 --2- 172.21.2.102:0/856554337 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f55081b0470 0x7f55081b0870 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:18.473 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:18.458 7f550ca61700 1 -- 172.21.2.102:0/856554337 >> 172.21.2.102:0/856554337 conn(0x7f55080ce3e0 msgr2=0x7f55080e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:05:18.473 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:18.458 7f550ca61700 1 -- 172.21.2.102:0/856554337 shutdown_connections 2021-02-10T11:05:18.473 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:18.458 7f550ca61700 1 -- 172.21.2.102:0/856554337 wait complete. 2021-02-10T11:05:18.473 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:18.472 7f550ca61700 1 Processor -- start 2021-02-10T11:05:18.473 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:18.472 7f550ca61700 1 -- start start 2021-02-10T11:05:18.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:18.472 7f550ca61700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f550818ea30 0x7f55080b17c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:18.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:18.472 7f550ca61700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f550818ee30 0x7f55080b1de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:18.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:18.472 7f550ca61700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5508092f70 0x7f5508093370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:18.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:18.472 7f550ca61700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55080b2400 con 0x7f550818ea30 2021-02-10T11:05:18.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:18.472 7f550ca61700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55080b25e0 con 0x7f550818ee30 2021-02-10T11:05:18.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:18.472 7f550ca61700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55080939c0 con 0x7f5508092f70 2021-02-10T11:05:18.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:18.473 7f550685e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f550818ea30 0x7f55080b17c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:18.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:18.473 7f550685e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f550818ea30 0x7f55080b17c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:50656/0 (socket says 172.21.2.102:50656) 2021-02-10T11:05:18.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:18.473 7f550685e700 1 -- 172.21.2.102:0/18063485 learned_addr learned my addr 172.21.2.102:0/18063485 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:05:18.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:18.473 7f550605d700 1 --2- 172.21.2.102:0/18063485 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f550818ee30 0x7f55080b1de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:18.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:18.473 7f550705f700 1 --2- 172.21.2.102:0/18063485 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5508092f70 0x7f5508093370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:18.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:18.473 7f550685e700 1 -- 172.21.2.102:0/18063485 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5508092f70 msgr2=0x7f5508093370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:18.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:18.473 7f550685e700 1 --2- 172.21.2.102:0/18063485 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5508092f70 0x7f5508093370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:18.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:18.473 7f550685e700 1 -- 172.21.2.102:0/18063485 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f550818ee30 msgr2=0x7f55080b1de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:18.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:18.473 7f550685e700 1 --2- 172.21.2.102:0/18063485 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f550818ee30 0x7f55080b1de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:18.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:18.473 7f550685e700 1 -- 172.21.2.102:0/18063485 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5508093be0 con 0x7f550818ea30 2021-02-10T11:05:18.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:18.473 7f550705f700 1 --2- 172.21.2.102:0/18063485 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5508092f70 0x7f5508093370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:05:18.477 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:18.473 7f550605d700 1 --2- 172.21.2.102:0/18063485 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f550818ee30 0x7f55080b1de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:05:18.477 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:18.473 7f550685e700 1 --2- 172.21.2.102:0/18063485 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f550818ea30 0x7f55080b17c0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 rx=0x7f54f800b5e0 tx=0x7f54f800b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:05:18.477 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:18.473 7f54ff7fe700 1 -- 172.21.2.102:0/18063485 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f54f800bd70 con 0x7f550818ea30 2021-02-10T11:05:18.477 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:18.474 7f54ff7fe700 1 -- 172.21.2.102:0/18063485 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54f8002030 con 0x7f550818ea30 2021-02-10T11:05:18.477 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:18.474 7f550ca61700 1 -- 172.21.2.102:0/18063485 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5508093be0 con 0x7f550818ea30 2021-02-10T11:05:18.477 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:18.474 7f550ca61700 1 -- 172.21.2.102:0/18063485 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5508093be0 con 0x7f550818ea30 2021-02-10T11:05:18.477 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:18.474 7f54ff7fe700 1 -- 172.21.2.102:0/18063485 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f54f8002550 con 0x7f550818ea30 2021-02-10T11:05:18.477 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:18.474 7f54ff7fe700 1 -- 172.21.2.102:0/18063485 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f54f8000c90 con 0x7f550818ea30 2021-02-10T11:05:18.477 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:18.474 7f54ff7fe700 1 --2- 172.21.2.102:0/18063485 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f54ec018730 0x7f54ec01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:18.477 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:18.474 7f54ff7fe700 1 -- 172.21.2.102:0/18063485 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f54f80327d0 con 0x7f550818ea30 2021-02-10T11:05:18.477 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:18.474 7f550605d700 1 --2- 172.21.2.102:0/18063485 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f54ec018730 0x7f54ec01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:18.477 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:18.475 7f550605d700 1 --2- 172.21.2.102:0/18063485 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f54ec018730 0x7f54ec01ac10 crc :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:05:18.504 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:18.506 7f550ca61700 1 -- 172.21.2.102:0/18063485 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5508105a30 con 0x7f550818ea30 2021-02-10T11:05:18.507 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:18.508 7f54ff7fe700 1 -- 172.21.2.102:0/18063485 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f54f801f510 con 0x7f550818ea30 2021-02-10T11:05:18.637 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:18.638 7f550ca61700 1 -- 172.21.2.102:0/18063485 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.3", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f5508105a30 con 0x7f550818ea30 2021-02-10T11:05:18.640 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:18.641 7f54ff7fe700 1 -- 172.21.2.102:0/18063485 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.3", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.3 v14) v1 ==== 184+0+0 (secure 0 0 0) 0x7f54f8000c90 con 0x7f550818ea30 2021-02-10T11:05:18.640 INFO:teuthology.orchestra.run.gibba002.stderr:updated caps for client.3 2021-02-10T11:05:18.641 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:18.643 7f550ca61700 1 -- 172.21.2.102:0/18063485 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f54ec018730 msgr2=0x7f54ec01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:18.641 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:18.643 7f550ca61700 1 --2- 172.21.2.102:0/18063485 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f54ec018730 0x7f54ec01ac10 crc :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:18.641 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:18.643 7f550ca61700 1 -- 172.21.2.102:0/18063485 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f550818ea30 msgr2=0x7f55080b17c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:18.641 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:18.643 7f550ca61700 1 --2- 172.21.2.102:0/18063485 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f550818ea30 0x7f55080b17c0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 rx=0x7f54f800b5e0 tx=0x7f54f800b980).stop 2021-02-10T11:05:18.642 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:18.643 7f550ca61700 1 -- 172.21.2.102:0/18063485 shutdown_connections 2021-02-10T11:05:18.642 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:18.643 7f550ca61700 1 --2- 172.21.2.102:0/18063485 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5508092f70 0x7f5508093370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:18.642 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:18.643 7f550ca61700 1 --2- 172.21.2.102:0/18063485 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f54ec018730 0x7f54ec01ac10 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:18.642 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:18.643 7f550ca61700 1 --2- 172.21.2.102:0/18063485 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f550818ee30 0x7f55080b1de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:18.642 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:18.643 7f550ca61700 1 --2- 172.21.2.102:0/18063485 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f550818ea30 0x7f55080b17c0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:18.642 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:18.643 7f550ca61700 1 -- 172.21.2.102:0/18063485 >> 172.21.2.102:0/18063485 conn(0x7f55081a3790 msgr2=0x7f55081aff80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:05:18.642 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:18.643 7f550ca61700 1 -- 172.21.2.102:0/18063485 shutdown_connections 2021-02-10T11:05:18.643 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:18.643 7f550ca61700 1 -- 172.21.2.102:0/18063485 wait complete. 2021-02-10T11:05:18.650 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:05:19.075 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:19.060 7fa056fce700 1 Processor -- start 2021-02-10T11:05:19.075 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:19.060 7fa056fce700 1 -- start start 2021-02-10T11:05:19.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:19.060 7fa056fce700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fa0500e5070 0x7fa0501b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:19.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:19.060 7fa056fce700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fa0501b3e40 0x7fa05007d690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:19.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:19.060 7fa056fce700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa05007ed80 con 0x7fa0500e5470 2021-02-10T11:05:19.077 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:19.060 7fa056fce700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa050080310 con 0x7fa0500e5070 2021-02-10T11:05:19.077 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:19.060 7fa056fce700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa050080530 con 0x7fa0501b3e40 2021-02-10T11:05:19.077 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:19.060 7fa054dea700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fa0500e5070 0x7fa0501b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:19.077 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:19.060 7fa054dea700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fa0500e5070 0x7fa0501b3820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:50664/0 (socket says 172.21.2.102:50664) 2021-02-10T11:05:19.077 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:19.060 7fa054dea700 1 -- 172.21.2.102:0/433475426 learned_addr learned my addr 172.21.2.102:0/433475426 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:05:19.077 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:19.061 7fa054dea700 1 -- 172.21.2.102:0/433475426 >> v2:172.21.2.113:3300/0 conn(0x7fa0501b3e40 msgr2=0x7fa05007d690 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:05:19.077 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:19.061 7fa054dea700 1 --2- 172.21.2.102:0/433475426 >> v2:172.21.2.113:3300/0 conn(0x7fa0501b3e40 0x7fa05007d690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:05:19.077 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:19.061 7fa054dea700 1 -- 172.21.2.102:0/433475426 >> v1:172.21.2.113:6789/0 conn(0x7fa0500e5470 legacy=0x7fa05007dcb0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:05:19.078 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:19.061 7fa054dea700 1 -- 172.21.2.102:0/433475426 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa050080750 con 0x7fa0500e5070 2021-02-10T11:05:19.078 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:19.061 7fa054dea700 1 --2- 172.21.2.102:0/433475426 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa0500e5070 0x7fa0501b3820 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 rx=0x7fa038006700 tx=0x7fa038009b80).ready entity=mon.0 client_cookie=b6ac7fa81332a721 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:05:19.078 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:19.061 7fa04f7fe700 1 -- 172.21.2.102:0/433475426 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa03800c0c0 con 0x7fa0500e5070 2021-02-10T11:05:19.078 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:19.061 7fa04f7fe700 1 -- 172.21.2.102:0/433475426 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa03800c340 con 0x7fa0500e5070 2021-02-10T11:05:19.078 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:19.061 7fa04f7fe700 1 -- 172.21.2.102:0/433475426 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa03800c860 con 0x7fa0500e5070 2021-02-10T11:05:19.078 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:19.061 7fa056fce700 1 -- 172.21.2.102:0/433475426 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa0500e5070 msgr2=0x7fa0501b3820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:19.079 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:19.061 7fa056fce700 1 --2- 172.21.2.102:0/433475426 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa0500e5070 0x7fa0501b3820 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 rx=0x7fa038006700 tx=0x7fa038009b80).stop 2021-02-10T11:05:19.079 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:19.062 7fa056fce700 1 -- 172.21.2.102:0/433475426 shutdown_connections 2021-02-10T11:05:19.079 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:19.062 7fa056fce700 1 --2- 172.21.2.102:0/433475426 >> v2:172.21.2.113:3300/0 conn(0x7fa0501b3e40 0x7fa05007d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:05:19.079 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:19.062 7fa056fce700 1 --2- 172.21.2.102:0/433475426 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa0500e5070 0x7fa0501b3820 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:19.079 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:19.062 7fa056fce700 1 -- 172.21.2.102:0/433475426 >> 172.21.2.102:0/433475426 conn(0x7fa0500ce690 msgr2=0x7fa0500e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:05:19.080 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:19.062 7fa056fce700 1 -- 172.21.2.102:0/433475426 shutdown_connections 2021-02-10T11:05:19.080 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:19.062 7fa056fce700 1 -- 172.21.2.102:0/433475426 wait complete. 2021-02-10T11:05:19.080 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:19.076 7fa056fce700 1 Processor -- start 2021-02-10T11:05:19.080 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:19.077 7fa056fce700 1 -- start start 2021-02-10T11:05:19.084 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:19.077 7fa056fce700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa0501a2a90 0x7fa0501aa0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:19.084 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:19.077 7fa056fce700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa0501aa700 0x7fa0501a3a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:19.084 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:19.077 7fa056fce700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa0500a55e0 0x7fa0501a4040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:19.084 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:19.077 7fa056fce700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0501a34a0 con 0x7fa0500a55e0 2021-02-10T11:05:19.084 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:19.077 7fa056fce700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0501a2e90 con 0x7fa0501a2a90 2021-02-10T11:05:19.085 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:19.077 7fa056fce700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0501a30b0 con 0x7fa0501aa700 2021-02-10T11:05:19.085 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:19.077 7fa0555eb700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa0500a55e0 0x7fa0501a4040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:19.085 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:19.077 7fa0555eb700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa0500a55e0 0x7fa0501a4040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:50668/0 (socket says 172.21.2.102:50668) 2021-02-10T11:05:19.085 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:19.077 7fa0555eb700 1 -- 172.21.2.102:0/2997793233 learned_addr learned my addr 172.21.2.102:0/2997793233 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:05:19.085 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:19.077 7fa054dea700 1 --2- 172.21.2.102:0/2997793233 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa0501a2a90 0x7fa0501aa0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:19.085 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:19.077 7fa04ffff700 1 --2- 172.21.2.102:0/2997793233 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa0501aa700 0x7fa0501a3a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:19.086 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:19.077 7fa0555eb700 1 -- 172.21.2.102:0/2997793233 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa0501aa700 msgr2=0x7fa0501a3a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:19.086 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:19.077 7fa0555eb700 1 --2- 172.21.2.102:0/2997793233 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa0501aa700 0x7fa0501a3a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:19.086 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:19.077 7fa0555eb700 1 -- 172.21.2.102:0/2997793233 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa0501a2a90 msgr2=0x7fa0501aa0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:19.086 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:19.077 7fa0555eb700 1 --2- 172.21.2.102:0/2997793233 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa0501a2a90 0x7fa0501aa0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:19.086 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:19.078 7fa0555eb700 1 -- 172.21.2.102:0/2997793233 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0501a4660 con 0x7fa0500a55e0 2021-02-10T11:05:19.086 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:19.078 7fa04ffff700 1 --2- 172.21.2.102:0/2997793233 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa0501aa700 0x7fa0501a3a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:05:19.086 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:19.078 7fa0555eb700 1 --2- 172.21.2.102:0/2997793233 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa0500a55e0 0x7fa0501a4040 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 rx=0x7fa048006ed0 tx=0x7fa04800b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:05:19.086 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:19.078 7fa04d7fa700 1 -- 172.21.2.102:0/2997793233 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa04800bd50 con 0x7fa0500a55e0 2021-02-10T11:05:19.086 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:19.078 7fa04d7fa700 1 -- 172.21.2.102:0/2997793233 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa048006020 con 0x7fa0500a55e0 2021-02-10T11:05:19.087 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:19.078 7fa04d7fa700 1 -- 172.21.2.102:0/2997793233 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa0480054b0 con 0x7fa0500a55e0 2021-02-10T11:05:19.087 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:19.078 7fa056fce700 1 -- 172.21.2.102:0/2997793233 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0501a4660 con 0x7fa0500a55e0 2021-02-10T11:05:19.087 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:19.078 7fa056fce700 1 -- 172.21.2.102:0/2997793233 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0501a4660 con 0x7fa0500a55e0 2021-02-10T11:05:19.087 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:19.078 7fa04d7fa700 1 -- 172.21.2.102:0/2997793233 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa048006200 con 0x7fa0500a55e0 2021-02-10T11:05:19.087 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:19.079 7fa04d7fa700 1 --2- 172.21.2.102:0/2997793233 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa0340186c0 0x7fa03401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:19.087 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:19.079 7fa04d7fa700 1 -- 172.21.2.102:0/2997793233 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fa048032660 con 0x7fa0500a55e0 2021-02-10T11:05:19.087 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:19.079 7fa054dea700 1 --2- 172.21.2.102:0/2997793233 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa0340186c0 0x7fa03401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:19.087 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:19.079 7fa054dea700 1 --2- 172.21.2.102:0/2997793233 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa0340186c0 0x7fa03401abc0 crc :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:05:19.109 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:19.110 7fa056fce700 1 -- 172.21.2.102:0/2997793233 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa050083da0 con 0x7fa0500a55e0 2021-02-10T11:05:19.111 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:19.113 7fa04d7fa700 1 -- 172.21.2.102:0/2997793233 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fa04801f520 con 0x7fa0500a55e0 2021-02-10T11:05:19.239 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:19.240 7fa056fce700 1 -- 172.21.2.102:0/2997793233 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa05007ed80 con 0x7fa0500a55e0 2021-02-10T11:05:19.239 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:19.241 7fa04d7fa700 1 -- 172.21.2.102:0/2997793233 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) v1 ==== 94+0+2101 (secure 0 0 0) 0x7fa04800bd50 con 0x7fa0500a55e0 2021-02-10T11:05:19.240 INFO:teuthology.orchestra.run.gibba002.stderr:dumped fsmap epoch 14 2021-02-10T11:05:19.240 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:05:19.240 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":14,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4845,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.102:6835/3747080008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6834","nonce":3747080008},{"type":"v1","addr":"172.21.2.102:6835","nonce":3747080008}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":10}],"filesystems":[{"mdsmap":{"epoch":14,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:05:12.567145","modified":"2021-02-10 11:05:14.922542","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4510},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4510":{"gid":4510,"name":"b","rank":0,"incarnation":11,"state":"up:active","state_seq":4,"addr":"172.21.2.113:6833/3898434901","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6832","nonce":3898434901},{"type":"v1","addr":"172.21.2.113:6833","nonce":3898434901}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":2}]} 2021-02-10T11:05:19.241 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:19.243 7fa056fce700 1 -- 172.21.2.102:0/2997793233 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa0340186c0 msgr2=0x7fa03401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:19.241 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:19.243 7fa056fce700 1 --2- 172.21.2.102:0/2997793233 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa0340186c0 0x7fa03401abc0 crc :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:19.241 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:19.243 7fa056fce700 1 -- 172.21.2.102:0/2997793233 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa0500a55e0 msgr2=0x7fa0501a4040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:19.241 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:19.243 7fa056fce700 1 --2- 172.21.2.102:0/2997793233 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa0500a55e0 0x7fa0501a4040 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 rx=0x7fa048006ed0 tx=0x7fa04800b960).stop 2021-02-10T11:05:19.242 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:19.243 7fa056fce700 1 -- 172.21.2.102:0/2997793233 shutdown_connections 2021-02-10T11:05:19.242 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:19.243 7fa056fce700 1 --2- 172.21.2.102:0/2997793233 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa0500a55e0 0x7fa0501a4040 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:19.242 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:19.243 7fa056fce700 1 --2- 172.21.2.102:0/2997793233 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa0501aa700 0x7fa0501a3a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:19.242 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:19.243 7fa056fce700 1 --2- 172.21.2.102:0/2997793233 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa0340186c0 0x7fa03401abc0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:19.242 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:19.243 7fa056fce700 1 --2- 172.21.2.102:0/2997793233 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa0501a2a90 0x7fa0501aa0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:19.242 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:19.243 7fa056fce700 1 -- 172.21.2.102:0/2997793233 >> 172.21.2.102:0/2997793233 conn(0x7fa05008fe10 msgr2=0x7fa05019c020 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:05:19.243 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:19.243 7fa056fce700 1 -- 172.21.2.102:0/2997793233 shutdown_connections 2021-02-10T11:05:19.243 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:19.243 7fa056fce700 1 -- 172.21.2.102:0/2997793233 wait complete. 2021-02-10T11:05:19.452 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 14, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:05:12.567145', 'modified': '2021-02-10 11:05:14.922542', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4510}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4510': {'gid': 4510, 'name': 'b', 'rank': 0, 'incarnation': 11, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.2.113:6833/3898434901', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.113:6832', 'nonce': 3898434901}, {'type': 'v1', 'addr': '172.21.2.113:6833', 'nonce': 3898434901}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [5], 'metadata_pool': 4, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:05:19.452 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:05:19.452 DEBUG:teuthology.orchestra.run.gibba013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:05:19.850 INFO:teuthology.orchestra.run.gibba013.stdout:{ 2021-02-10T11:05:19.851 INFO:teuthology.orchestra.run.gibba013.stdout: "cluster_fsid": "68db30ea-dbce-4aef-a6df-f01c6fea0513", 2021-02-10T11:05:19.851 INFO:teuthology.orchestra.run.gibba013.stdout: "whoami": 0, 2021-02-10T11:05:19.851 INFO:teuthology.orchestra.run.gibba013.stdout: "id": 4510, 2021-02-10T11:05:19.851 INFO:teuthology.orchestra.run.gibba013.stdout: "want_state": "up:active", 2021-02-10T11:05:19.851 INFO:teuthology.orchestra.run.gibba013.stdout: "state": "up:active", 2021-02-10T11:05:19.851 INFO:teuthology.orchestra.run.gibba013.stdout: "rank_uptime": 7.2834480360000002, 2021-02-10T11:05:19.851 INFO:teuthology.orchestra.run.gibba013.stdout: "mdsmap_epoch": 14, 2021-02-10T11:05:19.851 INFO:teuthology.orchestra.run.gibba013.stdout: "osdmap_epoch": 30, 2021-02-10T11:05:19.851 INFO:teuthology.orchestra.run.gibba013.stdout: "osdmap_epoch_barrier": 30, 2021-02-10T11:05:19.851 INFO:teuthology.orchestra.run.gibba013.stdout: "uptime": 16.567227233000001 2021-02-10T11:05:19.851 INFO:teuthology.orchestra.run.gibba013.stdout:} 2021-02-10T11:05:19.855 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "68db30ea-dbce-4aef-a6df-f01c6fea0513", "whoami": 0, "id": 4510, "want_state": "up:active", "state": "up:active", "rank_uptime": 7.2834480360000002, "mdsmap_epoch": 14, "osdmap_epoch": 30, "osdmap_epoch_barrier": 30, "uptime": 16.567227233000001 } 2021-02-10T11:05:19.855 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:05:20.281 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:20.265 7f259e5ab700 1 Processor -- start 2021-02-10T11:05:21.508 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:20.265 7f259e5ab700 1 -- start start 2021-02-10T11:05:21.509 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:20.266 7f259e5ab700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f25980e5070 0x7f25981b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:21.509 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:20.266 7f259e5ab700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f25980e5470 0x7f259807d690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:21.510 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:20.266 7f259e5ab700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f25981b3e40 0x7f259807dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:21.510 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:20.266 7f259e5ab700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f259807ed80 con 0x7f25981b3e40 2021-02-10T11:05:21.510 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:20.266 7f259e5ab700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f259807e2d0 con 0x7f25980e5470 2021-02-10T11:05:21.510 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:20.266 7f259e5ab700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f259807e4c0 con 0x7f25980e5070 2021-02-10T11:05:21.510 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:20.266 7f259cbc8700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f25981b3e40 0x7f259807dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:21.510 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:20.266 7f259cbc8700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f25981b3e40 0x7f259807dcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:50676/0 (socket says 172.21.2.102:50676) 2021-02-10T11:05:21.510 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:20.266 7f259cbc8700 1 -- 172.21.2.102:0/4148667116 learned_addr learned my addr 172.21.2.102:0/4148667116 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:05:21.510 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:20.266 7f259cbc8700 1 -- 172.21.2.102:0/4148667116 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f25980e5070 msgr2=0x7f25981b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:05:21.510 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:20.266 7f259cbc8700 1 --2- 172.21.2.102:0/4148667116 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f25980e5070 0x7f25981b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:05:21.510 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:20.266 7f259cbc8700 1 -- 172.21.2.102:0/4148667116 >> v2:172.21.2.113:3300/0 conn(0x7f25980e5470 msgr2=0x7f259807d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:05:21.511 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:20.266 7f259cbc8700 1 --2- 172.21.2.102:0/4148667116 >> v2:172.21.2.113:3300/0 conn(0x7f25980e5470 0x7f259807d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:05:21.511 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:20.266 7f259cbc8700 1 -- 172.21.2.102:0/4148667116 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f259807e6e0 con 0x7f25981b3e40 2021-02-10T11:05:21.511 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:20.267 7f259cbc8700 1 --2- 172.21.2.102:0/4148667116 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f25981b3e40 0x7f259807dcb0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 rx=0x7f2590006f00 tx=0x7f2590009f10).ready entity=mon.0 client_cookie=5c9e07b1239f7369 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:05:21.511 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:20.267 7f25977fe700 1 -- 172.21.2.102:0/4148667116 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f259000b3b0 con 0x7f25981b3e40 2021-02-10T11:05:21.511 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:20.267 7f25977fe700 1 -- 172.21.2.102:0/4148667116 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f259000b630 con 0x7f25981b3e40 2021-02-10T11:05:21.511 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:20.267 7f25977fe700 1 -- 172.21.2.102:0/4148667116 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f259000bb50 con 0x7f25981b3e40 2021-02-10T11:05:21.511 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:20.267 7f259e5ab700 1 -- 172.21.2.102:0/4148667116 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f25981b3e40 msgr2=0x7f259807dcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:21.511 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:20.267 7f259e5ab700 1 --2- 172.21.2.102:0/4148667116 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f25981b3e40 0x7f259807dcb0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 rx=0x7f2590006f00 tx=0x7f2590009f10).stop 2021-02-10T11:05:21.511 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:20.267 7f259e5ab700 1 -- 172.21.2.102:0/4148667116 shutdown_connections 2021-02-10T11:05:21.511 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:20.268 7f259e5ab700 1 --2- 172.21.2.102:0/4148667116 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f25981b3e40 0x7f259807dcb0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:21.511 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:20.268 7f259e5ab700 1 --2- 172.21.2.102:0/4148667116 >> v2:172.21.2.113:3300/0 conn(0x7f25980e5470 0x7f259807d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:05:21.511 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:20.268 7f259e5ab700 1 --2- 172.21.2.102:0/4148667116 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f25980e5070 0x7f25981b3820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:05:21.512 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:20.268 7f259e5ab700 1 -- 172.21.2.102:0/4148667116 >> 172.21.2.102:0/4148667116 conn(0x7f25980ce690 msgr2=0x7f25980e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:05:21.512 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:20.268 7f259e5ab700 1 -- 172.21.2.102:0/4148667116 shutdown_connections 2021-02-10T11:05:21.512 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:20.268 7f259e5ab700 1 -- 172.21.2.102:0/4148667116 wait complete. 2021-02-10T11:05:21.512 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:20.282 7f259e5ab700 1 Processor -- start 2021-02-10T11:05:21.512 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:20.283 7f259e5ab700 1 -- start start 2021-02-10T11:05:21.515 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:20.283 7f259e5ab700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2598192fc0 0x7f2598192780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:21.515 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:20.283 7f259e5ab700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f25980b1430 0x7f25980b1830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:21.515 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:20.283 7f259e5ab700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f25980b1e50 0x7f259808a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:21.515 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:20.283 7f259e5ab700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25980b2250 con 0x7f25980b1430 2021-02-10T11:05:21.515 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:20.283 7f259e5ab700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f259808b5d0 con 0x7f25980b1e50 2021-02-10T11:05:21.515 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:20.283 7f259e5ab700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f259808aa80 con 0x7f2598192fc0 2021-02-10T11:05:21.516 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:20.283 7f258ffff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f25980b1430 0x7f25980b1830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:21.516 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:20.283 7f258ffff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f25980b1430 0x7f25980b1830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:50682/0 (socket says 172.21.2.102:50682) 2021-02-10T11:05:21.516 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:20.283 7f258ffff700 1 -- 172.21.2.102:0/3384180910 learned_addr learned my addr 172.21.2.102:0/3384180910 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:05:21.516 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:20.283 7f258ffff700 1 -- 172.21.2.102:0/3384180910 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2598192fc0 msgr2=0x7f2598192780 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:05:21.516 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:20.283 7f258ffff700 1 --2- 172.21.2.102:0/3384180910 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2598192fc0 0x7f2598192780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:05:21.516 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:20.283 7f258ffff700 1 -- 172.21.2.102:0/3384180910 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f25980b1e50 msgr2=0x7f259808a460 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:05:21.516 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:20.283 7f258ffff700 1 --2- 172.21.2.102:0/3384180910 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f25980b1e50 0x7f259808a460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:05:21.516 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:20.283 7f258ffff700 1 -- 172.21.2.102:0/3384180910 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f259808ac60 con 0x7f25980b1430 2021-02-10T11:05:21.516 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:20.284 7f258ffff700 1 --2- 172.21.2.102:0/3384180910 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f25980b1430 0x7f25980b1830 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 rx=0x7f2588006f00 tx=0x7f258800b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:05:21.516 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:20.284 7f25957fa700 1 -- 172.21.2.102:0/3384180910 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f258800bd30 con 0x7f25980b1430 2021-02-10T11:05:21.516 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:20.284 7f25957fa700 1 -- 172.21.2.102:0/3384180910 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2588000c90 con 0x7f25980b1430 2021-02-10T11:05:21.516 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:20.284 7f259e5ab700 1 -- 172.21.2.102:0/3384180910 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f259808ac60 con 0x7f25980b1430 2021-02-10T11:05:21.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:20.284 7f259e5ab700 1 -- 172.21.2.102:0/3384180910 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f259808b1f0 con 0x7f25980b1430 2021-02-10T11:05:21.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:20.284 7f25957fa700 1 -- 172.21.2.102:0/3384180910 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f25880022a0 con 0x7f25980b1430 2021-02-10T11:05:21.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:20.284 7f25957fa700 1 -- 172.21.2.102:0/3384180910 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f2588005310 con 0x7f25980b1430 2021-02-10T11:05:21.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:20.284 7f25957fa700 1 --2- 172.21.2.102:0/3384180910 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f257c0186c0 0x7f257c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:21.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:20.284 7f25957fa700 1 -- 172.21.2.102:0/3384180910 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f2588032770 con 0x7f25980b1430 2021-02-10T11:05:21.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:20.284 7f2597fff700 1 --2- 172.21.2.102:0/3384180910 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f257c0186c0 0x7f257c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:21.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:20.285 7f2597fff700 1 --2- 172.21.2.102:0/3384180910 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f257c0186c0 0x7f257c01abc0 crc :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:05:21.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:20.316 7f259e5ab700 1 -- 172.21.2.102:0/3384180910 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2598107120 con 0x7f25980b1430 2021-02-10T11:05:21.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:20.319 7f25957fa700 1 -- 172.21.2.102:0/3384180910 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f2588000960 con 0x7f25980b1430 2021-02-10T11:05:21.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:20.445 7f259e5ab700 1 -- 172.21.2.102:0/3384180910 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f259807ed80 con 0x7f25980b1430 2021-02-10T11:05:21.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:20.446 7f25957fa700 1 -- 172.21.2.102:0/3384180910 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v30) v1 ==== 74+0+13993 (secure 0 0 0) 0x7f258800bd30 con 0x7f25980b1430 2021-02-10T11:05:21.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:20.447 7f259e5ab700 1 -- 172.21.2.102:0/3384180910 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f257c0186c0 msgr2=0x7f257c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:21.518 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:20.447 7f259e5ab700 1 --2- 172.21.2.102:0/3384180910 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f257c0186c0 0x7f257c01abc0 crc :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:21.518 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:20.447 7f259e5ab700 1 -- 172.21.2.102:0/3384180910 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f25980b1430 msgr2=0x7f25980b1830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:21.518 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:20.447 7f259e5ab700 1 --2- 172.21.2.102:0/3384180910 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f25980b1430 0x7f25980b1830 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 rx=0x7f2588006f00 tx=0x7f258800b940).stop 2021-02-10T11:05:21.518 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:20.448 7f259e5ab700 1 -- 172.21.2.102:0/3384180910 shutdown_connections 2021-02-10T11:05:21.518 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:20.448 7f259e5ab700 1 --2- 172.21.2.102:0/3384180910 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f257c0186c0 0x7f257c01abc0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:21.518 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:20.448 7f259e5ab700 1 --2- 172.21.2.102:0/3384180910 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f25980b1e50 0x7f259808a460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:05:21.518 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:20.448 7f259e5ab700 1 --2- 172.21.2.102:0/3384180910 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f25980b1430 0x7f25980b1830 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:21.518 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:20.448 7f259e5ab700 1 --2- 172.21.2.102:0/3384180910 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2598192fc0 0x7f2598192780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:05:21.518 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:20.448 7f259e5ab700 1 -- 172.21.2.102:0/3384180910 >> 172.21.2.102:0/3384180910 conn(0x7f25981ac620 msgr2=0x7f259819fee0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:05:21.518 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:20.448 7f259e5ab700 1 -- 172.21.2.102:0/3384180910 shutdown_connections 2021-02-10T11:05:21.518 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:20.448 7f259e5ab700 1 -- 172.21.2.102:0/3384180910 wait complete. 2021-02-10T11:05:21.519 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:05:21.519 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":30,"fsid":"68db30ea-dbce-4aef-a6df-f01c6fea0513","created":"2021-02-10 11:03:06.331557","modified":"2021-02-10 11:05:13.569263","last_up_change":"2021-02-10 11:03:32.690562","last_in_change":"2021-02-10 11:03:32.690562","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:03:37.322519","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:05:10.538786","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":5,"pool_name":"cephfs_data","create_time":"2021-02-10 11:05:11.559128","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"30","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"be691b4d-a039-408a-af80-9d53f02b6f1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6818","nonce":36088},{"type":"v1","addr":"172.21.2.102:6820","nonce":36088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6822","nonce":36088},{"type":"v1","addr":"172.21.2.102:6824","nonce":36088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6830","nonce":36088},{"type":"v1","addr":"172.21.2.102:6832","nonce":36088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6826","nonce":36088},{"type":"v1","addr":"172.21.2.102:6828","nonce":36088}]},"public_addr":"172.21.2.102:6820/36088","cluster_addr":"172.21.2.102:6824/36088","heartbeat_back_addr":"172.21.2.102:6832/36088","heartbeat_front_addr":"172.21.2.102:6828/36088","state":["exists","up"]},{"osd":1,"uuid":"64e3ccc8-9c69-4d7c-865e-af8e83822635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6819","nonce":36087},{"type":"v1","addr":"172.21.2.102:6821","nonce":36087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6823","nonce":36087},{"type":"v1","addr":"172.21.2.102:6825","nonce":36087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6831","nonce":36087},{"type":"v1","addr":"172.21.2.102:6833","nonce":36087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6827","nonce":36087},{"type":"v1","addr":"172.21.2.102:6829","nonce":36087}]},"public_addr":"172.21.2.102:6821/36087","cluster_addr":"172.21.2.102:6825/36087","heartbeat_back_addr":"172.21.2.102:6833/36087","heartbeat_front_addr":"172.21.2.102:6829/36087","state":["exists","up"]},{"osd":2,"uuid":"153700af-747f-481f-bdf4-d6072d3a842f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6810","nonce":36089},{"type":"v1","addr":"172.21.2.102:6811","nonce":36089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6812","nonce":36089},{"type":"v1","addr":"172.21.2.102:6813","nonce":36089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6816","nonce":36089},{"type":"v1","addr":"172.21.2.102:6817","nonce":36089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6814","nonce":36089},{"type":"v1","addr":"172.21.2.102:6815","nonce":36089}]},"public_addr":"172.21.2.102:6811/36089","cluster_addr":"172.21.2.102:6813/36089","heartbeat_back_addr":"172.21.2.102:6817/36089","heartbeat_front_addr":"172.21.2.102:6815/36089","state":["exists","up"]},{"osd":3,"uuid":"cbc678f4-83a9-45e6-8830-2f7c232b1ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6802","nonce":36090},{"type":"v1","addr":"172.21.2.102:6803","nonce":36090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6804","nonce":36090},{"type":"v1","addr":"172.21.2.102:6805","nonce":36090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6808","nonce":36090},{"type":"v1","addr":"172.21.2.102:6809","nonce":36090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6806","nonce":36090},{"type":"v1","addr":"172.21.2.102:6807","nonce":36090}]},"public_addr":"172.21.2.102:6803/36090","cluster_addr":"172.21.2.102:6805/36090","heartbeat_back_addr":"172.21.2.102:6809/36090","heartbeat_front_addr":"172.21.2.102:6807/36090","state":["exists","up"]},{"osd":4,"uuid":"c01e4008-b258-4f10-afd2-9e7f37214351","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6808","nonce":36368},{"type":"v1","addr":"172.21.2.113:6811","nonce":36368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6814","nonce":36368},{"type":"v1","addr":"172.21.2.113:6818","nonce":36368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6826","nonce":36368},{"type":"v1","addr":"172.21.2.113:6828","nonce":36368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6820","nonce":36368},{"type":"v1","addr":"172.21.2.113:6824","nonce":36368}]},"public_addr":"172.21.2.113:6811/36368","cluster_addr":"172.21.2.113:6818/36368","heartbeat_back_addr":"172.21.2.113:6828/36368","heartbeat_front_addr":"172.21.2.113:6824/36368","state":["exists","up"]},{"osd":5,"uuid":"551f6007-daf8-4a98-8ee0-c2b16bcccb5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6802","nonce":36366},{"type":"v1","addr":"172.21.2.113:6805","nonce":36366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6807","nonce":36366},{"type":"v1","addr":"172.21.2.113:6810","nonce":36366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6819","nonce":36366},{"type":"v1","addr":"172.21.2.113:6822","nonce":36366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6812","nonce":36366},{"type":"v1","addr":"172.21.2.113:6815","nonce":36366}]},"public_addr":"172.21.2.113:6805/36366","cluster_addr":"172.21.2.113:6810/36366","heartbeat_back_addr":"172.21.2.113:6822/36366","heartbeat_front_addr":"172.21.2.113:6815/36366","state":["exists","up"]},{"osd":6,"uuid":"bf67c040-0e50-48be-b102-c3943bc9ebef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6800","nonce":36367},{"type":"v1","addr":"172.21.2.113:6801","nonce":36367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6803","nonce":36367},{"type":"v1","addr":"172.21.2.113:6804","nonce":36367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6813","nonce":36367},{"type":"v1","addr":"172.21.2.113:6816","nonce":36367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6806","nonce":36367},{"type":"v1","addr":"172.21.2.113:6809","nonce":36367}]},"public_addr":"172.21.2.113:6801/36367","cluster_addr":"172.21.2.113:6804/36367","heartbeat_back_addr":"172.21.2.113:6816/36367","heartbeat_front_addr":"172.21.2.113:6809/36367","state":["exists","up"]},{"osd":7,"uuid":"f5461f80-018d-428f-be89-2eb2bcbe7bb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6817","nonce":36365},{"type":"v1","addr":"172.21.2.113:6821","nonce":36365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6823","nonce":36365},{"type":"v1","addr":"172.21.2.113:6825","nonce":36365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6830","nonce":36365},{"type":"v1","addr":"172.21.2.113:6831","nonce":36365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6827","nonce":36365},{"type":"v1","addr":"172.21.2.113:6829","nonce":36365}]},"public_addr":"172.21.2.113:6821/36365","cluster_addr":"172.21.2.113:6825/36365","heartbeat_back_addr":"172.21.2.113:6831/36365","heartbeat_front_addr":"172.21.2.113:6829/36365","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:05:21.519 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:03:37.322519', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 4, 'pool_name': 'cephfs_metadata', 'create_time': '2021-02-10 11:05:10.538786', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '29', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 5, 'pool_name': 'cephfs_data', 'create_time': '2021-02-10 11:05:11.559128', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '30', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-02-10T11:05:21.520 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:05:21.949 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:21.933 7f5643551700 1 Processor -- start 2021-02-10T11:05:21.950 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:21.933 7f5643551700 1 -- start start 2021-02-10T11:05:21.950 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:21.934 7f5643551700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f563c1b0470 0x7f563c1b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:21.950 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:21.934 7f5643551700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f563c0ef610 0x7f563c0efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:21.951 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:21.934 7f5643551700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f563c0f0b50 con 0x7f563c0f0030 2021-02-10T11:05:21.951 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:21.934 7f5643551700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f563c0e52e0 con 0x7f563c1b0470 2021-02-10T11:05:21.951 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:21.934 7f5643551700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f563c0e5490 con 0x7f563c0ef610 2021-02-10T11:05:21.951 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:21.934 7f564136d700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f563c1b0470 0x7f563c1b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:21.952 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:21.934 7f564136d700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f563c1b0470 0x7f563c1b0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:50688/0 (socket says 172.21.2.102:50688) 2021-02-10T11:05:21.952 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:21.934 7f564136d700 1 -- 172.21.2.102:0/470980442 learned_addr learned my addr 172.21.2.102:0/470980442 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:05:21.952 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:21.934 7f5640b6c700 1 --2- 172.21.2.102:0/470980442 >> v2:172.21.2.113:3300/0 conn(0x7f563c0ef610 0x7f563c0efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:21.953 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:21.934 7f564136d700 1 -- 172.21.2.102:0/470980442 >> v2:172.21.2.113:3300/0 conn(0x7f563c0ef610 msgr2=0x7f563c0efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:05:21.953 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:21.934 7f564136d700 1 --2- 172.21.2.102:0/470980442 >> v2:172.21.2.113:3300/0 conn(0x7f563c0ef610 0x7f563c0efa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:21.953 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:21.934 7f564136d700 1 -- 172.21.2.102:0/470980442 >> v1:172.21.2.113:6789/0 conn(0x7f563c0f0030 legacy=0x7f563c0f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:05:21.953 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:21.934 7f564136d700 1 -- 172.21.2.102:0/470980442 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f563c0f0430 con 0x7f563c1b0470 2021-02-10T11:05:21.953 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:21.934 7f564136d700 1 --2- 172.21.2.102:0/470980442 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f563c1b0470 0x7f563c1b0870 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 rx=0x7f5630006700 tx=0x7f5630009b80).ready entity=mon.0 client_cookie=8e838c6202df180e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:05:21.953 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:21.934 7f562ffff700 1 -- 172.21.2.102:0/470980442 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f563000c0c0 con 0x7f563c1b0470 2021-02-10T11:05:21.953 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:21.934 7f562ffff700 1 -- 172.21.2.102:0/470980442 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f563000c340 con 0x7f563c1b0470 2021-02-10T11:05:21.954 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:21.935 7f562ffff700 1 -- 172.21.2.102:0/470980442 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f563000c860 con 0x7f563c1b0470 2021-02-10T11:05:21.954 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:21.935 7f5643551700 1 -- 172.21.2.102:0/470980442 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f563c1b0470 msgr2=0x7f563c1b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:21.954 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:21.935 7f5643551700 1 --2- 172.21.2.102:0/470980442 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f563c1b0470 0x7f563c1b0870 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 rx=0x7f5630006700 tx=0x7f5630009b80).stop 2021-02-10T11:05:21.954 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:21.935 7f5643551700 1 -- 172.21.2.102:0/470980442 shutdown_connections 2021-02-10T11:05:21.954 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:21.935 7f5643551700 1 --2- 172.21.2.102:0/470980442 >> v2:172.21.2.113:3300/0 conn(0x7f563c0ef610 0x7f563c0efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:21.954 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:21.935 7f5643551700 1 --2- 172.21.2.102:0/470980442 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f563c1b0470 0x7f563c1b0870 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:21.955 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:21.935 7f5643551700 1 -- 172.21.2.102:0/470980442 >> 172.21.2.102:0/470980442 conn(0x7f563c0ce3e0 msgr2=0x7f563c0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:05:21.955 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:21.935 7f5643551700 1 -- 172.21.2.102:0/470980442 shutdown_connections 2021-02-10T11:05:21.955 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:21.935 7f5643551700 1 -- 172.21.2.102:0/470980442 wait complete. 2021-02-10T11:05:21.955 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:21.950 7f5643551700 1 Processor -- start 2021-02-10T11:05:21.955 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:21.950 7f5643551700 1 -- start start 2021-02-10T11:05:21.955 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:21.950 7f5643551700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f563c0a0420 0x7f563c0a0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:21.955 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:21.950 7f5643551700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f563c189d00 0x7f563c18a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:21.955 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:21.950 7f5643551700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f563c18a720 0x7f563c0b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:21.955 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:21.950 7f5643551700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f563c189800 con 0x7f563c0a0420 2021-02-10T11:05:21.955 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:21.950 7f5643551700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f563c18ab20 con 0x7f563c189d00 2021-02-10T11:05:21.958 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:21.950 7f5643551700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f563c0a1140 con 0x7f563c18a720 2021-02-10T11:05:21.958 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:21.950 7f564136d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f563c0a0420 0x7f563c0a0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:21.959 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:21.950 7f564136d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f563c0a0420 0x7f563c0a0820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:50692/0 (socket says 172.21.2.102:50692) 2021-02-10T11:05:21.959 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:21.951 7f564136d700 1 -- 172.21.2.102:0/2820313077 learned_addr learned my addr 172.21.2.102:0/2820313077 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:05:21.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:21.951 7f5640b6c700 1 --2- 172.21.2.102:0/2820313077 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f563c189d00 0x7f563c18a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:21.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:21.951 7f564136d700 1 -- 172.21.2.102:0/2820313077 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f563c18a720 msgr2=0x7f563c0b7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:21.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:21.951 7f564136d700 1 --2- 172.21.2.102:0/2820313077 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f563c18a720 0x7f563c0b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:05:21.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:21.951 7f564136d700 1 -- 172.21.2.102:0/2820313077 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f563c189d00 msgr2=0x7f563c18a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:21.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:21.951 7f564136d700 1 --2- 172.21.2.102:0/2820313077 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f563c189d00 0x7f563c18a100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:21.966 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:21.951 7f564136d700 1 -- 172.21.2.102:0/2820313077 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f563c0b8200 con 0x7f563c0a0420 2021-02-10T11:05:21.966 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:21.951 7f564136d700 1 --2- 172.21.2.102:0/2820313077 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f563c0a0420 0x7f563c0a0820 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 rx=0x7f563000b5e0 tx=0x7f563000b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:05:21.966 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:21.951 7f562dffb700 1 -- 172.21.2.102:0/2820313077 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f563000bd70 con 0x7f563c0a0420 2021-02-10T11:05:21.966 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:21.951 7f562dffb700 1 -- 172.21.2.102:0/2820313077 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5630002030 con 0x7f563c0a0420 2021-02-10T11:05:21.966 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:21.951 7f562dffb700 1 -- 172.21.2.102:0/2820313077 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5630002550 con 0x7f563c0a0420 2021-02-10T11:05:21.966 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:21.951 7f5643551700 1 -- 172.21.2.102:0/2820313077 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f563c0b8200 con 0x7f563c0a0420 2021-02-10T11:05:21.966 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:21.951 7f5643551700 1 -- 172.21.2.102:0/2820313077 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f563c0b8200 con 0x7f563c0a0420 2021-02-10T11:05:21.966 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:21.951 7f562dffb700 1 -- 172.21.2.102:0/2820313077 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f5630000c90 con 0x7f563c0a0420 2021-02-10T11:05:21.967 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:21.952 7f562dffb700 1 --2- 172.21.2.102:0/2820313077 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f561c0186c0 0x7f561c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:21.967 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:21.952 7f562dffb700 1 -- 172.21.2.102:0/2820313077 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f5630032860 con 0x7f563c0a0420 2021-02-10T11:05:21.967 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:21.952 7f5640b6c700 1 --2- 172.21.2.102:0/2820313077 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f561c0186c0 0x7f561c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:21.967 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:21.952 7f5640b6c700 1 --2- 172.21.2.102:0/2820313077 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f561c0186c0 0x7f561c01abc0 crc :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:05:21.984 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:21.983 7f5643551700 1 -- 172.21.2.102:0/2820313077 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f563c105ae0 con 0x7f563c0a0420 2021-02-10T11:05:21.985 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:21.986 7f562dffb700 1 -- 172.21.2.102:0/2820313077 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f563001f5a0 con 0x7f563c0a0420 2021-02-10T11:05:22.112 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:22.112 7f5643551700 1 -- 172.21.2.102:0/2820313077 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f563c0f0b50 con 0x7f563c0a0420 2021-02-10T11:05:22.112 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:22.113 7f562dffb700 1 -- 172.21.2.102:0/2820313077 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v30) v1 ==== 92+0+10 (secure 0 0 0) 0x7f563000bd70 con 0x7f563c0a0420 2021-02-10T11:05:22.112 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:05:22.114 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:22.115 7f5643551700 1 -- 172.21.2.102:0/2820313077 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f561c0186c0 msgr2=0x7f561c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:22.114 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:22.115 7f5643551700 1 --2- 172.21.2.102:0/2820313077 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f561c0186c0 0x7f561c01abc0 crc :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:22.114 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:22.115 7f5643551700 1 -- 172.21.2.102:0/2820313077 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f563c0a0420 msgr2=0x7f563c0a0820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:22.114 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:22.115 7f5643551700 1 --2- 172.21.2.102:0/2820313077 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f563c0a0420 0x7f563c0a0820 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 rx=0x7f563000b5e0 tx=0x7f563000b980).stop 2021-02-10T11:05:22.114 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:22.115 7f5643551700 1 -- 172.21.2.102:0/2820313077 shutdown_connections 2021-02-10T11:05:22.114 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:22.115 7f5643551700 1 --2- 172.21.2.102:0/2820313077 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f563c18a720 0x7f563c0b7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:05:22.114 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:22.115 7f5643551700 1 --2- 172.21.2.102:0/2820313077 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f561c0186c0 0x7f561c01abc0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:22.114 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:22.115 7f5643551700 1 --2- 172.21.2.102:0/2820313077 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f563c189d00 0x7f563c18a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:22.114 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:22.115 7f5643551700 1 --2- 172.21.2.102:0/2820313077 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f563c0a0420 0x7f563c0a0820 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:22.115 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:22.115 7f5643551700 1 -- 172.21.2.102:0/2820313077 >> 172.21.2.102:0/2820313077 conn(0x7f563c094e10 msgr2=0x7f563c0a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:05:22.115 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:22.115 7f5643551700 1 -- 172.21.2.102:0/2820313077 shutdown_connections 2021-02-10T11:05:22.115 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:22.115 7f5643551700 1 -- 172.21.2.102:0/2820313077 wait complete. 2021-02-10T11:05:22.122 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-02-10T11:05:22.569 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:22.554 7f38623c4700 1 Processor -- start 2021-02-10T11:05:22.569 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:22.554 7f38623c4700 1 -- start start 2021-02-10T11:05:22.569 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:22.554 7f38623c4700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f385c1b0870 0x7f385c0f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:22.569 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:22.554 7f38623c4700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f385c1b0c70 con 0x7f385c0efe00 2021-02-10T11:05:22.569 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:22.554 7f38623c4700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f385c0f0b50 con 0x7f385c1b0470 2021-02-10T11:05:22.569 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:22.554 7f38623c4700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f385c0e5ea0 con 0x7f385c1b0870 2021-02-10T11:05:22.569 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:22.554 7f38609e1700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f385c0efe00 0x7f385c0f0200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:42928/0 (socket says 172.21.2.102:42928) 2021-02-10T11:05:22.569 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:22.554 7f38609e1700 1 -- 172.21.2.102:0/427053410 learned_addr learned my addr 172.21.2.102:0/427053410 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:05:22.570 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:22.555 7f385b7fe700 1 --2- 172.21.2.102:0/427053410 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f385c1b0870 0x7f385c0f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:22.570 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:22.555 7f385affd700 1 -- 172.21.2.102:0/427053410 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 156665183 0 0) 0x7f3854001500 con 0x7f385c0efe00 2021-02-10T11:05:22.570 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:22.555 7f385b7fe700 1 -- 172.21.2.102:0/427053410 >> v1:172.21.2.113:6789/0 conn(0x7f385c1b0470 legacy=0x7f385c0ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:22.570 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:22.555 7f385b7fe700 1 -- 172.21.2.102:0/427053410 >> v1:172.21.2.102:6789/0 conn(0x7f385c0efe00 legacy=0x7f385c0f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:22.570 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:22.555 7f385b7fe700 1 -- 172.21.2.102:0/427053410 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f385c0e52e0 con 0x7f385c1b0870 2021-02-10T11:05:22.570 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:22.555 7f385b7fe700 1 --2- 172.21.2.102:0/427053410 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f385c1b0870 0x7f385c0f20e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f3850006650 tx=0x7f3850009a00).ready entity=mon.2 client_cookie=1493da259b5f9cc server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:05:22.570 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:22.556 7f385affd700 1 -- 172.21.2.102:0/427053410 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3850009df0 con 0x7f385c1b0870 2021-02-10T11:05:22.570 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:22.556 7f385affd700 1 -- 172.21.2.102:0/427053410 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38500009b0 con 0x7f385c1b0870 2021-02-10T11:05:22.570 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:22.556 7f385affd700 1 -- 172.21.2.102:0/427053410 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f385000c2b0 con 0x7f385c1b0870 2021-02-10T11:05:22.570 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:22.556 7f38623c4700 1 -- 172.21.2.102:0/427053410 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f385c1b0870 msgr2=0x7f385c0f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:22.570 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:22.556 7f38623c4700 1 --2- 172.21.2.102:0/427053410 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f385c1b0870 0x7f385c0f20e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f3850006650 tx=0x7f3850009a00).stop 2021-02-10T11:05:22.570 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:22.556 7f38623c4700 1 -- 172.21.2.102:0/427053410 shutdown_connections 2021-02-10T11:05:22.571 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:22.556 7f38623c4700 1 --2- 172.21.2.102:0/427053410 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f385c1b0870 0x7f385c0f20e0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:22.571 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:22.556 7f38623c4700 1 -- 172.21.2.102:0/427053410 >> 172.21.2.102:0/427053410 conn(0x7f385c0ce3e0 msgr2=0x7f385c0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:05:22.571 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:22.556 7f38623c4700 1 -- 172.21.2.102:0/427053410 shutdown_connections 2021-02-10T11:05:22.571 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:22.556 7f38623c4700 1 -- 172.21.2.102:0/427053410 wait complete. 2021-02-10T11:05:22.571 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:22.571 7f38623c4700 1 Processor -- start 2021-02-10T11:05:22.571 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:22.571 7f38623c4700 1 -- start start 2021-02-10T11:05:22.571 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:22.571 7f38623c4700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f385c1a0570 0x7f385c0a4590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:22.572 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:22.571 7f38623c4700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f385c0f1fd0 0x7f385c0f23d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:22.572 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:22.571 7f38623c4700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f385c0a5450 0x7f385c0a5850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:22.573 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:22.571 7f38623c4700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f385c0a4bb0 con 0x7f385c1a0570 2021-02-10T11:05:22.573 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:22.571 7f38623c4700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f385c0b35e0 con 0x7f385c0a5450 2021-02-10T11:05:22.573 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:22.571 7f38623c4700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f385c0a1f00 con 0x7f385c0f1fd0 2021-02-10T11:05:22.573 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:22.571 7f385bfff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f385c1a0570 0x7f385c0a4590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:22.573 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:22.571 7f385bfff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f385c1a0570 0x7f385c0a4590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:50706/0 (socket says 172.21.2.102:50706) 2021-02-10T11:05:22.573 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:22.571 7f385bfff700 1 -- 172.21.2.102:0/1175043695 learned_addr learned my addr 172.21.2.102:0/1175043695 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:05:22.573 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:22.572 7f385b7fe700 1 --2- 172.21.2.102:0/1175043695 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f385c0f1fd0 0x7f385c0f23d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:22.573 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:22.572 7f38609e1700 1 --2- 172.21.2.102:0/1175043695 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f385c0a5450 0x7f385c0a5850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:22.574 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:22.572 7f385bfff700 1 -- 172.21.2.102:0/1175043695 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f385c0f1fd0 msgr2=0x7f385c0f23d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:22.574 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:22.572 7f385bfff700 1 --2- 172.21.2.102:0/1175043695 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f385c0f1fd0 0x7f385c0f23d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:22.574 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:22.572 7f385bfff700 1 -- 172.21.2.102:0/1175043695 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f385c0a5450 msgr2=0x7f385c0a5850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:22.574 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:22.572 7f385bfff700 1 --2- 172.21.2.102:0/1175043695 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f385c0a5450 0x7f385c0a5850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:22.575 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:22.572 7f385bfff700 1 -- 172.21.2.102:0/1175043695 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f385c0a5e70 con 0x7f385c1a0570 2021-02-10T11:05:22.575 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:22.572 7f385bfff700 1 --2- 172.21.2.102:0/1175043695 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f385c1a0570 0x7f385c0a4590 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 rx=0x7f384c006ed0 tx=0x7f384c00b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:05:22.575 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:22.572 7f3858ff9700 1 -- 172.21.2.102:0/1175043695 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f384c000cd0 con 0x7f385c1a0570 2021-02-10T11:05:22.575 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:22.572 7f3858ff9700 1 -- 172.21.2.102:0/1175043695 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f384c006020 con 0x7f385c1a0570 2021-02-10T11:05:22.575 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:22.572 7f38623c4700 1 -- 172.21.2.102:0/1175043695 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f385c0a5e70 con 0x7f385c1a0570 2021-02-10T11:05:22.575 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:22.572 7f38623c4700 1 -- 172.21.2.102:0/1175043695 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f385c0a6400 con 0x7f385c1a0570 2021-02-10T11:05:22.575 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:22.572 7f3858ff9700 1 -- 172.21.2.102:0/1175043695 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f384c0054b0 con 0x7f385c1a0570 2021-02-10T11:05:22.575 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:22.573 7f3858ff9700 1 -- 172.21.2.102:0/1175043695 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f384c006200 con 0x7f385c1a0570 2021-02-10T11:05:22.575 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:22.573 7f3858ff9700 1 --2- 172.21.2.102:0/1175043695 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f383c0186c0 0x7f383c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:22.575 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:22.573 7f3858ff9700 1 -- 172.21.2.102:0/1175043695 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f384c032590 con 0x7f385c1a0570 2021-02-10T11:05:22.575 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:22.573 7f385b7fe700 1 --2- 172.21.2.102:0/1175043695 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f383c0186c0 0x7f383c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:22.575 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:22.574 7f385b7fe700 1 --2- 172.21.2.102:0/1175043695 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f383c0186c0 0x7f383c01abc0 crc :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:05:22.603 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:22.605 7f38623c4700 1 -- 172.21.2.102:0/1175043695 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3834002fd0 con 0x7f385c1a0570 2021-02-10T11:05:22.606 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:22.607 7f3858ff9700 1 -- 172.21.2.102:0/1175043695 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f384c030d40 con 0x7f385c1a0570 2021-02-10T11:05:22.735 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:22.736 7f38623c4700 1 -- 172.21.2.102:0/1175043695 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7f3834002ef0 con 0x7f385c1a0570 2021-02-10T11:05:22.735 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:22.737 7f3858ff9700 1 -- 172.21.2.102:0/1175043695 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v30) v1 ==== 104+0+10 (secure 0 0 0) 0x7f384c006200 con 0x7f385c1a0570 2021-02-10T11:05:22.736 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:05:22.737 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:22.738 7f38623c4700 1 -- 172.21.2.102:0/1175043695 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f383c0186c0 msgr2=0x7f383c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:22.737 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:22.738 7f38623c4700 1 --2- 172.21.2.102:0/1175043695 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f383c0186c0 0x7f383c01abc0 crc :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:22.737 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:22.738 7f38623c4700 1 -- 172.21.2.102:0/1175043695 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f385c1a0570 msgr2=0x7f385c0a4590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:22.737 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:22.738 7f38623c4700 1 --2- 172.21.2.102:0/1175043695 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f385c1a0570 0x7f385c0a4590 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 rx=0x7f384c006ed0 tx=0x7f384c00b960).stop 2021-02-10T11:05:22.738 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:22.739 7f38623c4700 1 -- 172.21.2.102:0/1175043695 shutdown_connections 2021-02-10T11:05:22.738 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:22.739 7f38623c4700 1 --2- 172.21.2.102:0/1175043695 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f383c0186c0 0x7f383c01abc0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:22.738 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:22.739 7f38623c4700 1 --2- 172.21.2.102:0/1175043695 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f385c0a5450 0x7f385c0a5850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:22.738 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:22.739 7f38623c4700 1 --2- 172.21.2.102:0/1175043695 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f385c0f1fd0 0x7f385c0f23d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:22.738 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:22.739 7f38623c4700 1 --2- 172.21.2.102:0/1175043695 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f385c1a0570 0x7f385c0a4590 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:22.738 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:22.739 7f38623c4700 1 -- 172.21.2.102:0/1175043695 >> 172.21.2.102:0/1175043695 conn(0x7f385c0958b0 msgr2=0x7f385c0bf040 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:05:22.738 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:22.739 7f38623c4700 1 -- 172.21.2.102:0/1175043695 shutdown_connections 2021-02-10T11:05:22.738 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:22.739 7f38623c4700 1 -- 172.21.2.102:0/1175043695 wait complete. 2021-02-10T11:05:22.746 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-02-10T11:05:23.178 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:23.162 7f2297cd1700 1 Processor -- start 2021-02-10T11:05:23.178 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:23.162 7f2297cd1700 1 -- start start 2021-02-10T11:05:23.178 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:23.163 7f2297cd1700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f22901b0870 0x7f22900f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:23.178 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:23.163 7f2297cd1700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22901b0c70 con 0x7f22901b0470 2021-02-10T11:05:23.178 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:23.163 7f2297cd1700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22900f0b50 con 0x7f22900efe00 2021-02-10T11:05:23.178 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:23.163 7f2297cd1700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22900e5ea0 con 0x7f22901b0870 2021-02-10T11:05:23.178 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:23.163 7f22952ec700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f22901b0870 0x7f22900f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:23.178 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:23.163 7f22952ec700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f22901b0870 0x7f22900f20e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:50714/0 (socket says 172.21.2.102:50714) 2021-02-10T11:05:23.179 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:23.163 7f22952ec700 1 -- 172.21.2.102:0/2365860480 learned_addr learned my addr 172.21.2.102:0/2365860480 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:05:23.179 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:23.163 7f2294aeb700 1 -- 172.21.2.102:0/2365860480 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2030231136 0 0) 0x7f228c000ff0 con 0x7f22901b0470 2021-02-10T11:05:23.179 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:23.163 7f22952ec700 1 -- 172.21.2.102:0/2365860480 >> v1:172.21.2.113:6789/0 conn(0x7f22900efe00 legacy=0x7f22900f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:05:23.179 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:23.163 7f22952ec700 1 -- 172.21.2.102:0/2365860480 >> v1:172.21.2.102:6789/0 conn(0x7f22901b0470 legacy=0x7f22900ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:23.179 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:23.163 7f22952ec700 1 -- 172.21.2.102:0/2365860480 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22900e52e0 con 0x7f22901b0870 2021-02-10T11:05:23.179 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:23.164 7f22952ec700 1 --2- 172.21.2.102:0/2365860480 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f22901b0870 0x7f22900f20e0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 rx=0x7f2280006700 tx=0x7f2280009b80).ready entity=mon.0 client_cookie=85f6cfbb05a71203 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:05:23.179 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:23.164 7f2294aeb700 1 -- 172.21.2.102:0/2365860480 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f228000c0c0 con 0x7f22901b0870 2021-02-10T11:05:23.179 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:23.164 7f2294aeb700 1 -- 172.21.2.102:0/2365860480 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f228000c340 con 0x7f22901b0870 2021-02-10T11:05:23.179 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:23.164 7f2294aeb700 1 -- 172.21.2.102:0/2365860480 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f228000c860 con 0x7f22901b0870 2021-02-10T11:05:23.179 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:23.164 7f2297cd1700 1 -- 172.21.2.102:0/2365860480 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f22901b0870 msgr2=0x7f22900f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:23.179 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:23.164 7f2297cd1700 1 --2- 172.21.2.102:0/2365860480 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f22901b0870 0x7f22900f20e0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 rx=0x7f2280006700 tx=0x7f2280009b80).stop 2021-02-10T11:05:23.179 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:23.165 7f2297cd1700 1 -- 172.21.2.102:0/2365860480 shutdown_connections 2021-02-10T11:05:23.179 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:23.165 7f2297cd1700 1 --2- 172.21.2.102:0/2365860480 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f22901b0870 0x7f22900f20e0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:23.180 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:23.165 7f2297cd1700 1 -- 172.21.2.102:0/2365860480 >> 172.21.2.102:0/2365860480 conn(0x7f22900ce3e0 msgr2=0x7f22900e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:05:23.180 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:23.165 7f2297cd1700 1 -- 172.21.2.102:0/2365860480 shutdown_connections 2021-02-10T11:05:23.180 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:23.165 7f2297cd1700 1 -- 172.21.2.102:0/2365860480 wait complete. 2021-02-10T11:05:23.181 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:23.180 7f2297cd1700 1 Processor -- start 2021-02-10T11:05:23.181 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:23.180 7f2297cd1700 1 -- start start 2021-02-10T11:05:23.181 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:23.180 7f2297cd1700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f229019c090 0x7f229009d090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:23.181 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:23.180 7f2297cd1700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f22900be650 0x7f22900bd830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:23.182 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:23.180 7f2297cd1700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f22900bde50 0x7f229009de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:23.182 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:23.180 7f2297cd1700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f229009d6b0 con 0x7f22900bde50 2021-02-10T11:05:23.182 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:23.180 7f2297cd1700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f229009d900 con 0x7f22900be650 2021-02-10T11:05:23.182 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:23.180 7f2297cd1700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22900be250 con 0x7f229019c090 2021-02-10T11:05:23.182 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:23.180 7f22962ee700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f22900bde50 0x7f229009de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:23.182 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:23.180 7f22962ee700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f22900bde50 0x7f229009de80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:50720/0 (socket says 172.21.2.102:50720) 2021-02-10T11:05:23.182 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:23.180 7f22962ee700 1 -- 172.21.2.102:0/3704976309 learned_addr learned my addr 172.21.2.102:0/3704976309 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:05:23.182 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:23.181 7f22962ee700 1 -- 172.21.2.102:0/3704976309 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f229019c090 msgr2=0x7f229009d090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:23.182 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:23.181 7f22962ee700 1 --2- 172.21.2.102:0/3704976309 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f229019c090 0x7f229009d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:05:23.182 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:23.181 7f22962ee700 1 -- 172.21.2.102:0/3704976309 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f22900be650 msgr2=0x7f22900bd830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:23.182 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:23.181 7f22962ee700 1 --2- 172.21.2.102:0/3704976309 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f22900be650 0x7f22900bd830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:05:23.182 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:23.181 7f22962ee700 1 -- 172.21.2.102:0/3704976309 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f229009e4a0 con 0x7f22900bde50 2021-02-10T11:05:23.182 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:23.181 7f22962ee700 1 --2- 172.21.2.102:0/3704976309 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f22900bde50 0x7f229009de80 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 rx=0x7f2288009940 tx=0x7f228800bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:05:23.183 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:23.181 7f22867fc700 1 -- 172.21.2.102:0/3704976309 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2288000f90 con 0x7f22900bde50 2021-02-10T11:05:23.183 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:23.181 7f22867fc700 1 -- 172.21.2.102:0/3704976309 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f228800c2f0 con 0x7f22900bde50 2021-02-10T11:05:23.183 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:23.181 7f2297cd1700 1 -- 172.21.2.102:0/3704976309 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f229009e4a0 con 0x7f22900bde50 2021-02-10T11:05:23.183 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:23.181 7f22867fc700 1 -- 172.21.2.102:0/3704976309 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f228800c750 con 0x7f22900bde50 2021-02-10T11:05:23.183 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:23.181 7f2297cd1700 1 -- 172.21.2.102:0/3704976309 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f229009e4a0 con 0x7f22900bde50 2021-02-10T11:05:23.183 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:23.182 7f22867fc700 1 -- 172.21.2.102:0/3704976309 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f2288001220 con 0x7f22900bde50 2021-02-10T11:05:23.183 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:23.182 7f22867fc700 1 --2- 172.21.2.102:0/3704976309 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2278018730 0x7f227801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:23.183 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:23.182 7f22867fc700 1 -- 172.21.2.102:0/3704976309 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f2288032b80 con 0x7f22900bde50 2021-02-10T11:05:23.183 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:23.182 7f2295aed700 1 --2- 172.21.2.102:0/3704976309 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2278018730 0x7f227801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:23.183 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:23.182 7f2295aed700 1 --2- 172.21.2.102:0/3704976309 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2278018730 0x7f227801ac10 crc :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:05:23.211 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:23.212 7f2297cd1700 1 -- 172.21.2.102:0/3704976309 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2290097e00 con 0x7f22900bde50 2021-02-10T11:05:23.213 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:23.215 7f22867fc700 1 -- 172.21.2.102:0/3704976309 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f228800be30 con 0x7f22900bde50 2021-02-10T11:05:23.340 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:23.341 7f2297cd1700 1 -- 172.21.2.102:0/3704976309 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7f2290097e00 con 0x7f22900bde50 2021-02-10T11:05:23.341 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:23.342 7f22867fc700 1 -- 172.21.2.102:0/3704976309 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v30) v1 ==== 100+0+10 (secure 0 0 0) 0x7f228801f8c0 con 0x7f22900bde50 2021-02-10T11:05:23.341 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:05:23.342 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:23.344 7f2297cd1700 1 -- 172.21.2.102:0/3704976309 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2278018730 msgr2=0x7f227801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:23.342 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:23.344 7f2297cd1700 1 --2- 172.21.2.102:0/3704976309 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2278018730 0x7f227801ac10 crc :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:23.342 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:23.344 7f2297cd1700 1 -- 172.21.2.102:0/3704976309 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f22900bde50 msgr2=0x7f229009de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:23.343 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:23.344 7f2297cd1700 1 --2- 172.21.2.102:0/3704976309 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f22900bde50 0x7f229009de80 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 rx=0x7f2288009940 tx=0x7f228800bcf0).stop 2021-02-10T11:05:23.343 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:23.344 7f2297cd1700 1 -- 172.21.2.102:0/3704976309 shutdown_connections 2021-02-10T11:05:23.343 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:23.344 7f2297cd1700 1 --2- 172.21.2.102:0/3704976309 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f22900bde50 0x7f229009de80 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:23.343 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:23.344 7f2297cd1700 1 --2- 172.21.2.102:0/3704976309 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2278018730 0x7f227801ac10 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:23.343 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:23.344 7f2297cd1700 1 --2- 172.21.2.102:0/3704976309 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f22900be650 0x7f22900bd830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:05:23.343 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:23.344 7f2297cd1700 1 --2- 172.21.2.102:0/3704976309 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f229019c090 0x7f229009d090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:05:23.344 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:23.344 7f2297cd1700 1 -- 172.21.2.102:0/3704976309 >> 172.21.2.102:0/3704976309 conn(0x7f22900bf4f0 msgr2=0x7f229018f460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:05:23.344 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:23.344 7f2297cd1700 1 -- 172.21.2.102:0/3704976309 shutdown_connections 2021-02-10T11:05:23.344 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:23.344 7f2297cd1700 1 -- 172.21.2.102:0/3704976309 wait complete. 2021-02-10T11:05:23.351 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-02-10T11:05:23.351 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:05:23.784 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:23.768 7f032bcbf700 1 Processor -- start 2021-02-10T11:05:23.784 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:23.769 7f032bcbf700 1 -- start start 2021-02-10T11:05:23.784 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:23.769 7f032bcbf700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f03240e5070 0x7f03241b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:23.784 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:23.769 7f032bcbf700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f03241b3e40 0x7f032407d690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:23.784 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:23.769 7f032bcbf700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f03240e5470 0x7f032407dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:23.784 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:23.769 7f032bcbf700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f032407ed80 con 0x7f03240e5470 2021-02-10T11:05:23.784 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:23.769 7f032bcbf700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f032407e2d0 con 0x7f03240e5070 2021-02-10T11:05:23.785 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:23.769 7f032bcbf700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f032407e4c0 con 0x7f03241b3e40 2021-02-10T11:05:23.785 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:23.769 7f032a2dc700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f03240e5470 0x7f032407dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:23.785 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:23.769 7f032a2dc700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f03240e5470 0x7f032407dcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:50732/0 (socket says 172.21.2.102:50732) 2021-02-10T11:05:23.785 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:23.769 7f032a2dc700 1 -- 172.21.2.102:0/4234897833 learned_addr learned my addr 172.21.2.102:0/4234897833 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:05:23.785 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:23.770 7f032a2dc700 1 -- 172.21.2.102:0/4234897833 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f03241b3e40 msgr2=0x7f032407d690 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:05:23.785 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:23.770 7f032a2dc700 1 --2- 172.21.2.102:0/4234897833 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f03241b3e40 0x7f032407d690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:05:23.785 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:23.770 7f032a2dc700 1 -- 172.21.2.102:0/4234897833 >> v2:172.21.2.113:3300/0 conn(0x7f03240e5070 msgr2=0x7f03241b3820 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:05:23.785 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:23.770 7f032a2dc700 1 --2- 172.21.2.102:0/4234897833 >> v2:172.21.2.113:3300/0 conn(0x7f03240e5070 0x7f03241b3820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:05:23.785 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:23.770 7f032a2dc700 1 -- 172.21.2.102:0/4234897833 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f032407e6e0 con 0x7f03240e5470 2021-02-10T11:05:23.785 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:23.770 7f032a2dc700 1 --2- 172.21.2.102:0/4234897833 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f03240e5470 0x7f032407dcb0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 rx=0x7f031c006f00 tx=0x7f031c009f10).ready entity=mon.0 client_cookie=aa21c1fbf408c28c server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:05:23.785 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:23.770 7f0328ad9700 1 -- 172.21.2.102:0/4234897833 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f031c00b3b0 con 0x7f03240e5470 2021-02-10T11:05:23.785 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:23.770 7f0328ad9700 1 -- 172.21.2.102:0/4234897833 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f031c00b730 con 0x7f03240e5470 2021-02-10T11:05:23.785 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:23.770 7f0328ad9700 1 -- 172.21.2.102:0/4234897833 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f031c00bc50 con 0x7f03240e5470 2021-02-10T11:05:23.785 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:23.770 7f032bcbf700 1 -- 172.21.2.102:0/4234897833 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f03240e5470 msgr2=0x7f032407dcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:23.786 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:23.770 7f032bcbf700 1 --2- 172.21.2.102:0/4234897833 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f03240e5470 0x7f032407dcb0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 rx=0x7f031c006f00 tx=0x7f031c009f10).stop 2021-02-10T11:05:23.786 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:23.771 7f032bcbf700 1 -- 172.21.2.102:0/4234897833 shutdown_connections 2021-02-10T11:05:23.786 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:23.771 7f032bcbf700 1 --2- 172.21.2.102:0/4234897833 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f03241b3e40 0x7f032407d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:05:23.786 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:23.771 7f032bcbf700 1 --2- 172.21.2.102:0/4234897833 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f03240e5470 0x7f032407dcb0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:23.786 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:23.771 7f032bcbf700 1 --2- 172.21.2.102:0/4234897833 >> v2:172.21.2.113:3300/0 conn(0x7f03240e5070 0x7f03241b3820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:05:23.786 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:23.771 7f032bcbf700 1 -- 172.21.2.102:0/4234897833 >> 172.21.2.102:0/4234897833 conn(0x7f03240ce690 msgr2=0x7f03240e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:05:23.786 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:23.771 7f032bcbf700 1 -- 172.21.2.102:0/4234897833 shutdown_connections 2021-02-10T11:05:23.786 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:23.771 7f032bcbf700 1 -- 172.21.2.102:0/4234897833 wait complete. 2021-02-10T11:05:23.788 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:23.786 7f032bcbf700 1 Processor -- start 2021-02-10T11:05:23.788 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:23.786 7f032bcbf700 1 -- start start 2021-02-10T11:05:23.788 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:23.786 7f032bcbf700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f032418c700 0x7f0324096c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:23.788 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:23.786 7f032bcbf700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f03240ae400 0x7f03240ae800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:23.788 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:23.786 7f032bcbf700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0324192380 0x7f0324192780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:23.788 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:23.786 7f032bcbf700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0324097290 con 0x7f032418c700 2021-02-10T11:05:23.789 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:23.786 7f032bcbf700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0324192da0 con 0x7f03240ae400 2021-02-10T11:05:23.789 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:23.786 7f032bcbf700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0324192f80 con 0x7f0324192380 2021-02-10T11:05:23.789 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:23.786 7f0329adb700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f032418c700 0x7f0324096c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:23.789 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:23.786 7f0329adb700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f032418c700 0x7f0324096c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:50734/0 (socket says 172.21.2.102:50734) 2021-02-10T11:05:23.789 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:23.786 7f0329adb700 1 -- 172.21.2.102:0/1032555877 learned_addr learned my addr 172.21.2.102:0/1032555877 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:05:23.789 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:23.787 7f0329adb700 1 -- 172.21.2.102:0/1032555877 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0324192380 msgr2=0x7f0324192780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:23.789 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:23.787 7f0329adb700 1 --2- 172.21.2.102:0/1032555877 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0324192380 0x7f0324192780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:05:23.789 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:23.787 7f0329adb700 1 -- 172.21.2.102:0/1032555877 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f03240ae400 msgr2=0x7f03240ae800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:23.789 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:23.787 7f0329adb700 1 --2- 172.21.2.102:0/1032555877 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f03240ae400 0x7f03240ae800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:05:23.789 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:23.787 7f0329adb700 1 -- 172.21.2.102:0/1032555877 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03241931a0 con 0x7f032418c700 2021-02-10T11:05:23.789 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:23.787 7f0329adb700 1 --2- 172.21.2.102:0/1032555877 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f032418c700 0x7f0324096c70 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 rx=0x7f032000b5c0 tx=0x7f032000b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:05:23.789 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:23.787 7f031a7fc700 1 -- 172.21.2.102:0/1032555877 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0320000c40 con 0x7f032418c700 2021-02-10T11:05:23.789 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:23.787 7f031a7fc700 1 -- 172.21.2.102:0/1032555877 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f032000be20 con 0x7f032418c700 2021-02-10T11:05:23.790 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:23.787 7f032bcbf700 1 -- 172.21.2.102:0/1032555877 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03241931a0 con 0x7f032418c700 2021-02-10T11:05:23.790 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:23.787 7f031a7fc700 1 -- 172.21.2.102:0/1032555877 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0320006220 con 0x7f032418c700 2021-02-10T11:05:23.790 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:23.787 7f032bcbf700 1 -- 172.21.2.102:0/1032555877 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03241931a0 con 0x7f032418c700 2021-02-10T11:05:23.790 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:23.787 7f031a7fc700 1 -- 172.21.2.102:0/1032555877 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f0320006030 con 0x7f032418c700 2021-02-10T11:05:23.790 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:23.787 7f031a7fc700 1 --2- 172.21.2.102:0/1032555877 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f030c018730 0x7f030c01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:23.790 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:23.787 7f031a7fc700 1 -- 172.21.2.102:0/1032555877 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f0320032840 con 0x7f032418c700 2021-02-10T11:05:23.790 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:23.788 7f03292da700 1 --2- 172.21.2.102:0/1032555877 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f030c018730 0x7f030c01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:23.790 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:23.788 7f03292da700 1 --2- 172.21.2.102:0/1032555877 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f030c018730 0x7f030c01ac10 crc :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:05:23.818 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:23.819 7f032bcbf700 1 -- 172.21.2.102:0/1032555877 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0324107120 con 0x7f032418c700 2021-02-10T11:05:23.820 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:23.822 7f031a7fc700 1 -- 172.21.2.102:0/1032555877 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f0320006b80 con 0x7f032418c700 2021-02-10T11:05:23.947 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:23.949 7f032bcbf700 1 -- 172.21.2.102:0/1032555877 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0324107120 con 0x7f032418c700 2021-02-10T11:05:23.948 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:23.949 7f031a7fc700 1 -- 172.21.2.102:0/1032555877 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) v1 ==== 94+0+2101 (secure 0 0 0) 0x7f0320006030 con 0x7f032418c700 2021-02-10T11:05:23.948 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:05:23.948 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":14,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4845,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.102:6835/3747080008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6834","nonce":3747080008},{"type":"v1","addr":"172.21.2.102:6835","nonce":3747080008}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":10}],"filesystems":[{"mdsmap":{"epoch":14,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:05:12.567145","modified":"2021-02-10 11:05:14.922542","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4510},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4510":{"gid":4510,"name":"b","rank":0,"incarnation":11,"state":"up:active","state_seq":4,"addr":"172.21.2.113:6833/3898434901","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6832","nonce":3898434901},{"type":"v1","addr":"172.21.2.113:6833","nonce":3898434901}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":2}]} 2021-02-10T11:05:23.949 INFO:teuthology.orchestra.run.gibba002.stderr:dumped fsmap epoch 14 2021-02-10T11:05:23.950 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:23.951 7f032bcbf700 1 -- 172.21.2.102:0/1032555877 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f030c018730 msgr2=0x7f030c01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:23.950 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:23.951 7f032bcbf700 1 --2- 172.21.2.102:0/1032555877 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f030c018730 0x7f030c01ac10 crc :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:23.950 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:23.951 7f032bcbf700 1 -- 172.21.2.102:0/1032555877 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f032418c700 msgr2=0x7f0324096c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:23.950 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:23.951 7f032bcbf700 1 --2- 172.21.2.102:0/1032555877 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f032418c700 0x7f0324096c70 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 rx=0x7f032000b5c0 tx=0x7f032000b960).stop 2021-02-10T11:05:23.951 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:23.951 7f032bcbf700 1 -- 172.21.2.102:0/1032555877 shutdown_connections 2021-02-10T11:05:23.951 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:23.951 7f032bcbf700 1 --2- 172.21.2.102:0/1032555877 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0324192380 0x7f0324192780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:05:23.951 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:23.951 7f032bcbf700 1 --2- 172.21.2.102:0/1032555877 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f030c018730 0x7f030c01ac10 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:23.951 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:23.951 7f032bcbf700 1 --2- 172.21.2.102:0/1032555877 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f03240ae400 0x7f03240ae800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:05:23.951 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:23.951 7f032bcbf700 1 --2- 172.21.2.102:0/1032555877 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f032418c700 0x7f0324096c70 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:23.951 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:23.951 7f032bcbf700 1 -- 172.21.2.102:0/1032555877 >> 172.21.2.102:0/1032555877 conn(0x7f03240891f0 msgr2=0x7f03241a0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:05:23.951 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:23.951 7f032bcbf700 1 -- 172.21.2.102:0/1032555877 shutdown_connections 2021-02-10T11:05:23.951 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:23.951 7f032bcbf700 1 -- 172.21.2.102:0/1032555877 wait complete. 2021-02-10T11:05:24.160 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 14, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:05:12.567145', 'modified': '2021-02-10 11:05:14.922542', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4510}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4510': {'gid': 4510, 'name': 'b', 'rank': 0, 'incarnation': 11, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.2.113:6833/3898434901', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.113:6832', 'nonce': 3898434901}, {'type': 'v1', 'addr': '172.21.2.113:6833', 'nonce': 3898434901}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [5], 'metadata_pool': 4, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:05:24.160 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:05:24.160 DEBUG:teuthology.orchestra.run.gibba013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:05:24.557 INFO:teuthology.orchestra.run.gibba013.stdout:{ 2021-02-10T11:05:24.558 INFO:teuthology.orchestra.run.gibba013.stdout: "cluster_fsid": "68db30ea-dbce-4aef-a6df-f01c6fea0513", 2021-02-10T11:05:24.558 INFO:teuthology.orchestra.run.gibba013.stdout: "whoami": 0, 2021-02-10T11:05:24.558 INFO:teuthology.orchestra.run.gibba013.stdout: "id": 4510, 2021-02-10T11:05:24.558 INFO:teuthology.orchestra.run.gibba013.stdout: "want_state": "up:active", 2021-02-10T11:05:24.558 INFO:teuthology.orchestra.run.gibba013.stdout: "state": "up:active", 2021-02-10T11:05:24.558 INFO:teuthology.orchestra.run.gibba013.stdout: "rank_uptime": 11.990496265000001, 2021-02-10T11:05:24.558 INFO:teuthology.orchestra.run.gibba013.stdout: "mdsmap_epoch": 14, 2021-02-10T11:05:24.558 INFO:teuthology.orchestra.run.gibba013.stdout: "osdmap_epoch": 30, 2021-02-10T11:05:24.558 INFO:teuthology.orchestra.run.gibba013.stdout: "osdmap_epoch_barrier": 30, 2021-02-10T11:05:24.558 INFO:teuthology.orchestra.run.gibba013.stdout: "uptime": 21.274275904 2021-02-10T11:05:24.558 INFO:teuthology.orchestra.run.gibba013.stdout:} 2021-02-10T11:05:24.562 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "68db30ea-dbce-4aef-a6df-f01c6fea0513", "whoami": 0, "id": 4510, "want_state": "up:active", "state": "up:active", "rank_uptime": 11.990496265000001, "mdsmap_epoch": 14, "osdmap_epoch": 30, "osdmap_epoch_barrier": 30, "uptime": 21.274275904 } 2021-02-10T11:05:24.562 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2021-02-10T11:05:24.562 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2021-02-10T11:05:24.562 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@gibba002.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2021-02-10T11:05:24.562 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:05:24.563 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec mkdir -p /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:05:24.898 DEBUG:teuthology.orchestra.run.gibba002:> sudo modprobe fuse 2021-02-10T11:05:25.242 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:05:25.242 DEBUG:teuthology.orchestra.run.gibba002:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:05:25.585 INFO:teuthology.orchestra.run.gibba002.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:05:25.585 INFO:teuthology.orchestra.run.gibba002.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:05:25.586 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:05:25.586 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:05:25.587 DEBUG:teuthology.orchestra.run.gibba002:> ls /sys/fs/fuse/connections 2021-02-10T11:05:25.922 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2021-02-10T11:05:25.922 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:05:25.925 DEBUG:teuthology.orchestra.run.gibba002:> sudo modprobe fuse 2021-02-10T11:05:26.268 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:05:26.268 DEBUG:teuthology.orchestra.run.gibba002:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:05:26.334 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba002.stderr:2021-02-10 11:05:26.335 7fe370a4ef80 -1 init, newargv = 0x55d042140100 newargc=9 2021-02-10T11:05:26.334 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba002.stderr:ceph-fuse[44913]: starting ceph client 2021-02-10T11:05:26.340 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba002.stderr:ceph-fuse[44913]: starting fuse 2021-02-10T11:05:26.611 INFO:teuthology.orchestra.run.gibba002.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:05:26.611 INFO:teuthology.orchestra.run.gibba002.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:05:26.612 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:05:26.612 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:05:26.612 DEBUG:teuthology.orchestra.run.gibba002:> ls /sys/fs/fuse/connections 2021-02-10T11:05:26.947 INFO:teuthology.orchestra.run.gibba002.stdout:43 2021-02-10T11:05:26.947 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2021-02-10T11:05:26.947 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:05:26.947 DEBUG:teuthology.orchestra.run.gibba002:> sudo python3 -c ' 2021-02-10T11:05:26.947 DEBUG:teuthology.orchestra.run.gibba002:> import glob 2021-02-10T11:05:26.947 DEBUG:teuthology.orchestra.run.gibba002:> import re 2021-02-10T11:05:26.948 DEBUG:teuthology.orchestra.run.gibba002:> import os 2021-02-10T11:05:26.948 DEBUG:teuthology.orchestra.run.gibba002:> import subprocess 2021-02-10T11:05:26.948 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:05:26.948 DEBUG:teuthology.orchestra.run.gibba002:> def find_socket(client_name): 2021-02-10T11:05:26.948 DEBUG:teuthology.orchestra.run.gibba002:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2021-02-10T11:05:26.948 DEBUG:teuthology.orchestra.run.gibba002:> files = glob.glob(asok_path) 2021-02-10T11:05:26.948 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:05:26.948 DEBUG:teuthology.orchestra.run.gibba002:> # Given a non-glob path, it better be there 2021-02-10T11:05:26.948 DEBUG:teuthology.orchestra.run.gibba002:> if "*" not in asok_path: 2021-02-10T11:05:26.948 DEBUG:teuthology.orchestra.run.gibba002:> assert(len(files) == 1) 2021-02-10T11:05:26.948 DEBUG:teuthology.orchestra.run.gibba002:> return files[0] 2021-02-10T11:05:26.948 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:05:26.948 DEBUG:teuthology.orchestra.run.gibba002:> for f in files: 2021-02-10T11:05:26.948 DEBUG:teuthology.orchestra.run.gibba002:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2021-02-10T11:05:26.948 DEBUG:teuthology.orchestra.run.gibba002:> if os.path.exists("/proc/{0}".format(pid)): 2021-02-10T11:05:26.949 DEBUG:teuthology.orchestra.run.gibba002:> return f 2021-02-10T11:05:26.949 DEBUG:teuthology.orchestra.run.gibba002:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2021-02-10T11:05:26.949 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:05:26.949 DEBUG:teuthology.orchestra.run.gibba002:> print(find_socket("client.0")) 2021-02-10T11:05:26.949 DEBUG:teuthology.orchestra.run.gibba002:> ' 2021-02-10T11:05:27.310 INFO:teuthology.orchestra.run.gibba002.stdout:/var/run/ceph/ceph-client.0.44913.asok 2021-02-10T11:05:27.312 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.44913.asok 2021-02-10T11:05:27.313 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:05:27.313 DEBUG:teuthology.orchestra.run.gibba002:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.44913.asok status 2021-02-10T11:05:27.702 INFO:teuthology.orchestra.run.gibba002.stdout:{ 2021-02-10T11:05:27.702 INFO:teuthology.orchestra.run.gibba002.stdout: "metadata": { 2021-02-10T11:05:27.702 INFO:teuthology.orchestra.run.gibba002.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:05:27.702 INFO:teuthology.orchestra.run.gibba002.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:05:27.702 INFO:teuthology.orchestra.run.gibba002.stdout: "entity_id": "0", 2021-02-10T11:05:27.703 INFO:teuthology.orchestra.run.gibba002.stdout: "hostname": "gibba002", 2021-02-10T11:05:27.703 INFO:teuthology.orchestra.run.gibba002.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2021-02-10T11:05:27.703 INFO:teuthology.orchestra.run.gibba002.stdout: "pid": "44913", 2021-02-10T11:05:27.703 INFO:teuthology.orchestra.run.gibba002.stdout: "root": "/" 2021-02-10T11:05:27.703 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:05:27.703 INFO:teuthology.orchestra.run.gibba002.stdout: "dentry_count": 0, 2021-02-10T11:05:27.703 INFO:teuthology.orchestra.run.gibba002.stdout: "dentry_pinned_count": 0, 2021-02-10T11:05:27.703 INFO:teuthology.orchestra.run.gibba002.stdout: "id": 5058, 2021-02-10T11:05:27.703 INFO:teuthology.orchestra.run.gibba002.stdout: "inst": { 2021-02-10T11:05:27.703 INFO:teuthology.orchestra.run.gibba002.stdout: "name": { 2021-02-10T11:05:27.703 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "client", 2021-02-10T11:05:27.703 INFO:teuthology.orchestra.run.gibba002.stdout: "num": 5058 2021-02-10T11:05:27.703 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:05:27.704 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": { 2021-02-10T11:05:27.704 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:05:27.704 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:0", 2021-02-10T11:05:27.704 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 621687058 2021-02-10T11:05:27.704 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:05:27.704 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:05:27.704 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": { 2021-02-10T11:05:27.704 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:05:27.704 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:0", 2021-02-10T11:05:27.704 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 621687058 2021-02-10T11:05:27.704 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:05:27.704 INFO:teuthology.orchestra.run.gibba002.stdout: "inst_str": "client.5058 172.21.2.102:0/621687058", 2021-02-10T11:05:27.704 INFO:teuthology.orchestra.run.gibba002.stdout: "addr_str": "172.21.2.102:0/621687058", 2021-02-10T11:05:27.704 INFO:teuthology.orchestra.run.gibba002.stdout: "inode_count": 1, 2021-02-10T11:05:27.705 INFO:teuthology.orchestra.run.gibba002.stdout: "mds_epoch": 14, 2021-02-10T11:05:27.705 INFO:teuthology.orchestra.run.gibba002.stdout: "osd_epoch": 30, 2021-02-10T11:05:27.705 INFO:teuthology.orchestra.run.gibba002.stdout: "osd_epoch_barrier": 0, 2021-02-10T11:05:27.705 INFO:teuthology.orchestra.run.gibba002.stdout: "blacklisted": false 2021-02-10T11:05:27.705 INFO:teuthology.orchestra.run.gibba002.stdout:} 2021-02-10T11:05:27.707 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:05:27.707 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:05:27.707 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:05:28.042 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:05:28.042 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:05:28.042 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:05:28.043 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:05:28.389 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:05:28.815 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:28.800 7f995d711700 1 Processor -- start 2021-02-10T11:05:28.815 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:28.800 7f995d711700 1 -- start start 2021-02-10T11:05:28.815 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:28.800 7f995d711700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f99580e5470 0x7f995807e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:28.815 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:28.800 7f995d711700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f995807ed80 con 0x7f995807f300 2021-02-10T11:05:28.815 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:28.800 7f995d711700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99581b4010 con 0x7f99580e5070 2021-02-10T11:05:28.816 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:28.800 7f995d711700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99580d1790 con 0x7f99580e5470 2021-02-10T11:05:28.816 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:28.801 7f9957ac0700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f995807f300 0x7f995807f700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:42980/0 (socket says 172.21.2.102:42980) 2021-02-10T11:05:28.816 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:28.801 7f9957ac0700 1 -- 172.21.2.102:0/2318464127 learned_addr learned my addr 172.21.2.102:0/2318464127 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:05:28.816 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:28.801 7f9956abe700 1 --2- 172.21.2.102:0/2318464127 >> v2:172.21.2.113:3300/0 conn(0x7f99580e5470 0x7f995807e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:28.816 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:28.801 7f99562bd700 1 -- 172.21.2.102:0/2318464127 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 949651315 0 0) 0x7f9950001500 con 0x7f995807f300 2021-02-10T11:05:28.816 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:28.801 7f99562bd700 1 -- 172.21.2.102:0/2318464127 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9944002fe0 con 0x7f995807f300 2021-02-10T11:05:28.816 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:28.801 7f9956abe700 1 -- 172.21.2.102:0/2318464127 >> v1:172.21.2.113:6789/0 conn(0x7f99580e5070 legacy=0x7f99581b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:28.816 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:28.801 7f9956abe700 1 -- 172.21.2.102:0/2318464127 >> v1:172.21.2.102:6789/0 conn(0x7f995807f300 legacy=0x7f995807f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:28.816 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:28.801 7f99562bd700 1 -- 172.21.2.102:0/2318464127 <== mon.1 v1:172.21.2.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1394682916 0 0) 0x7f9940001080 con 0x7f99580e5070 2021-02-10T11:05:28.816 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:28.801 7f9956abe700 1 -- 172.21.2.102:0/2318464127 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99580d19b0 con 0x7f99580e5470 2021-02-10T11:05:28.816 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:28.801 7f9956abe700 1 --2- 172.21.2.102:0/2318464127 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f99580e5470 0x7f995807e6a0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7f9948006650 tx=0x7f9948009a80).ready entity=mon.1 client_cookie=fba96ec6f2adaecd server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:05:28.816 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:28.802 7f99562bd700 1 -- 172.21.2.102:0/2318464127 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f99480009b0 con 0x7f99580e5470 2021-02-10T11:05:28.817 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:28.802 7f99562bd700 1 -- 172.21.2.102:0/2318464127 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f994800c0c0 con 0x7f99580e5470 2021-02-10T11:05:28.817 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:28.802 7f99562bd700 1 -- 172.21.2.102:0/2318464127 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f994800c5e0 con 0x7f99580e5470 2021-02-10T11:05:28.817 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:28.802 7f995d711700 1 -- 172.21.2.102:0/2318464127 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f99580e5470 msgr2=0x7f995807e6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:28.817 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:28.802 7f995d711700 1 --2- 172.21.2.102:0/2318464127 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f99580e5470 0x7f995807e6a0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7f9948006650 tx=0x7f9948009a80).stop 2021-02-10T11:05:28.817 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:28.802 7f995d711700 1 -- 172.21.2.102:0/2318464127 shutdown_connections 2021-02-10T11:05:28.817 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:28.802 7f995d711700 1 --2- 172.21.2.102:0/2318464127 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f99580e5470 0x7f995807e6a0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:28.817 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:28.802 7f995d711700 1 -- 172.21.2.102:0/2318464127 >> 172.21.2.102:0/2318464127 conn(0x7f99580ce690 msgr2=0x7f99580e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:05:28.817 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:28.802 7f995d711700 1 -- 172.21.2.102:0/2318464127 shutdown_connections 2021-02-10T11:05:28.817 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:28.802 7f995d711700 1 -- 172.21.2.102:0/2318464127 wait complete. 2021-02-10T11:05:28.819 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:28.817 7f995d711700 1 Processor -- start 2021-02-10T11:05:28.819 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:28.817 7f995d711700 1 -- start start 2021-02-10T11:05:28.819 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:28.817 7f995d711700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f9958084100 0x7f995818f730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:28.819 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:28.817 7f995d711700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f995818fd50 0x7f99580a9cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:28.819 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:28.817 7f995d711700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f99580aa310 0x7f9958086c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:28.819 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:28.817 7f995d711700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9958190400 con 0x7f995818fd50 2021-02-10T11:05:28.819 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:28.817 7f995d711700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9958083c60 con 0x7f9958084100 2021-02-10T11:05:28.819 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:28.817 7f995d711700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99580aa710 con 0x7f99580aa310 2021-02-10T11:05:28.819 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:28.818 7f9956abe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f995818fd50 0x7f99580a9cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:28.819 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:28.818 7f9956abe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f995818fd50 0x7f99580a9cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:50762/0 (socket says 172.21.2.102:50762) 2021-02-10T11:05:28.819 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:28.818 7f9956abe700 1 -- 172.21.2.102:0/1684493116 learned_addr learned my addr 172.21.2.102:0/1684493116 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:05:28.820 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:28.818 7f9956abe700 1 -- 172.21.2.102:0/1684493116 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f99580aa310 msgr2=0x7f9958086c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:28.820 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:28.818 7f9956abe700 1 --2- 172.21.2.102:0/1684493116 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f99580aa310 0x7f9958086c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:05:28.820 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:28.818 7f9956abe700 1 -- 172.21.2.102:0/1684493116 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f9958084100 msgr2=0x7f995818f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:28.820 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:28.818 7f9956abe700 1 --2- 172.21.2.102:0/1684493116 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f9958084100 0x7f995818f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:05:28.820 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:28.818 7f9956abe700 1 -- 172.21.2.102:0/1684493116 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99580aa930 con 0x7f995818fd50 2021-02-10T11:05:28.820 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:28.818 7f9956abe700 1 --2- 172.21.2.102:0/1684493116 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f995818fd50 0x7f99580a9cf0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 rx=0x7f9948009630 tx=0x7f994800b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:05:28.820 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:28.818 7f994ffff700 1 -- 172.21.2.102:0/1684493116 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f99480093a0 con 0x7f995818fd50 2021-02-10T11:05:28.820 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:28.818 7f994ffff700 1 -- 172.21.2.102:0/1684493116 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f994800bd70 con 0x7f995818fd50 2021-02-10T11:05:28.821 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:28.818 7f995d711700 1 -- 172.21.2.102:0/1684493116 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99580aa930 con 0x7f995818fd50 2021-02-10T11:05:28.821 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:28.818 7f995d711700 1 -- 172.21.2.102:0/1684493116 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9958087590 con 0x7f995818fd50 2021-02-10T11:05:28.821 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:28.818 7f994ffff700 1 -- 172.21.2.102:0/1684493116 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f99480054d0 con 0x7f995818fd50 2021-02-10T11:05:28.821 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:28.819 7f994ffff700 1 -- 172.21.2.102:0/1684493116 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f994800a000 con 0x7f995818fd50 2021-02-10T11:05:28.821 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:28.819 7f994ffff700 1 --2- 172.21.2.102:0/1684493116 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f993c0186c0 0x7f993c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:28.821 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:28.819 7f994ffff700 1 -- 172.21.2.102:0/1684493116 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f99480325a0 con 0x7f995818fd50 2021-02-10T11:05:28.821 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:28.819 7f99572bf700 1 --2- 172.21.2.102:0/1684493116 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f993c0186c0 0x7f993c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:28.821 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:28.819 7f99572bf700 1 --2- 172.21.2.102:0/1684493116 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f993c0186c0 0x7f993c01abc0 crc :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:05:28.849 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:28.850 7f995d711700 1 -- 172.21.2.102:0/1684493116 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9958107150 con 0x7f995818fd50 2021-02-10T11:05:28.851 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:28.853 7f994ffff700 1 -- 172.21.2.102:0/1684493116 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f9948009340 con 0x7f995818fd50 2021-02-10T11:05:28.977 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:28.978 7f995d711700 1 -- 172.21.2.102:0/1684493116 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f995807ed80 con 0x7f995818fd50 2021-02-10T11:05:28.977 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:28.979 7f994ffff700 1 -- 172.21.2.102:0/1684493116 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v30) v1 ==== 74+0+13993 (secure 0 0 0) 0x7f994800a000 con 0x7f995818fd50 2021-02-10T11:05:28.978 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:05:28.978 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":30,"fsid":"68db30ea-dbce-4aef-a6df-f01c6fea0513","created":"2021-02-10 11:03:06.331557","modified":"2021-02-10 11:05:13.569263","last_up_change":"2021-02-10 11:03:32.690562","last_in_change":"2021-02-10 11:03:32.690562","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:03:37.322519","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:05:10.538786","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":5,"pool_name":"cephfs_data","create_time":"2021-02-10 11:05:11.559128","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"30","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"be691b4d-a039-408a-af80-9d53f02b6f1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6818","nonce":36088},{"type":"v1","addr":"172.21.2.102:6820","nonce":36088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6822","nonce":36088},{"type":"v1","addr":"172.21.2.102:6824","nonce":36088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6830","nonce":36088},{"type":"v1","addr":"172.21.2.102:6832","nonce":36088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6826","nonce":36088},{"type":"v1","addr":"172.21.2.102:6828","nonce":36088}]},"public_addr":"172.21.2.102:6820/36088","cluster_addr":"172.21.2.102:6824/36088","heartbeat_back_addr":"172.21.2.102:6832/36088","heartbeat_front_addr":"172.21.2.102:6828/36088","state":["exists","up"]},{"osd":1,"uuid":"64e3ccc8-9c69-4d7c-865e-af8e83822635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6819","nonce":36087},{"type":"v1","addr":"172.21.2.102:6821","nonce":36087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6823","nonce":36087},{"type":"v1","addr":"172.21.2.102:6825","nonce":36087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6831","nonce":36087},{"type":"v1","addr":"172.21.2.102:6833","nonce":36087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6827","nonce":36087},{"type":"v1","addr":"172.21.2.102:6829","nonce":36087}]},"public_addr":"172.21.2.102:6821/36087","cluster_addr":"172.21.2.102:6825/36087","heartbeat_back_addr":"172.21.2.102:6833/36087","heartbeat_front_addr":"172.21.2.102:6829/36087","state":["exists","up"]},{"osd":2,"uuid":"153700af-747f-481f-bdf4-d6072d3a842f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6810","nonce":36089},{"type":"v1","addr":"172.21.2.102:6811","nonce":36089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6812","nonce":36089},{"type":"v1","addr":"172.21.2.102:6813","nonce":36089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6816","nonce":36089},{"type":"v1","addr":"172.21.2.102:6817","nonce":36089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6814","nonce":36089},{"type":"v1","addr":"172.21.2.102:6815","nonce":36089}]},"public_addr":"172.21.2.102:6811/36089","cluster_addr":"172.21.2.102:6813/36089","heartbeat_back_addr":"172.21.2.102:6817/36089","heartbeat_front_addr":"172.21.2.102:6815/36089","state":["exists","up"]},{"osd":3,"uuid":"cbc678f4-83a9-45e6-8830-2f7c232b1ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6802","nonce":36090},{"type":"v1","addr":"172.21.2.102:6803","nonce":36090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6804","nonce":36090},{"type":"v1","addr":"172.21.2.102:6805","nonce":36090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6808","nonce":36090},{"type":"v1","addr":"172.21.2.102:6809","nonce":36090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6806","nonce":36090},{"type":"v1","addr":"172.21.2.102:6807","nonce":36090}]},"public_addr":"172.21.2.102:6803/36090","cluster_addr":"172.21.2.102:6805/36090","heartbeat_back_addr":"172.21.2.102:6809/36090","heartbeat_front_addr":"172.21.2.102:6807/36090","state":["exists","up"]},{"osd":4,"uuid":"c01e4008-b258-4f10-afd2-9e7f37214351","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6808","nonce":36368},{"type":"v1","addr":"172.21.2.113:6811","nonce":36368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6814","nonce":36368},{"type":"v1","addr":"172.21.2.113:6818","nonce":36368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6826","nonce":36368},{"type":"v1","addr":"172.21.2.113:6828","nonce":36368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6820","nonce":36368},{"type":"v1","addr":"172.21.2.113:6824","nonce":36368}]},"public_addr":"172.21.2.113:6811/36368","cluster_addr":"172.21.2.113:6818/36368","heartbeat_back_addr":"172.21.2.113:6828/36368","heartbeat_front_addr":"172.21.2.113:6824/36368","state":["exists","up"]},{"osd":5,"uuid":"551f6007-daf8-4a98-8ee0-c2b16bcccb5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6802","nonce":36366},{"type":"v1","addr":"172.21.2.113:6805","nonce":36366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6807","nonce":36366},{"type":"v1","addr":"172.21.2.113:6810","nonce":36366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6819","nonce":36366},{"type":"v1","addr":"172.21.2.113:6822","nonce":36366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6812","nonce":36366},{"type":"v1","addr":"172.21.2.113:6815","nonce":36366}]},"public_addr":"172.21.2.113:6805/36366","cluster_addr":"172.21.2.113:6810/36366","heartbeat_back_addr":"172.21.2.113:6822/36366","heartbeat_front_addr":"172.21.2.113:6815/36366","state":["exists","up"]},{"osd":6,"uuid":"bf67c040-0e50-48be-b102-c3943bc9ebef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6800","nonce":36367},{"type":"v1","addr":"172.21.2.113:6801","nonce":36367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6803","nonce":36367},{"type":"v1","addr":"172.21.2.113:6804","nonce":36367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6813","nonce":36367},{"type":"v1","addr":"172.21.2.113:6816","nonce":36367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6806","nonce":36367},{"type":"v1","addr":"172.21.2.113:6809","nonce":36367}]},"public_addr":"172.21.2.113:6801/36367","cluster_addr":"172.21.2.113:6804/36367","heartbeat_back_addr":"172.21.2.113:6816/36367","heartbeat_front_addr":"172.21.2.113:6809/36367","state":["exists","up"]},{"osd":7,"uuid":"f5461f80-018d-428f-be89-2eb2bcbe7bb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6817","nonce":36365},{"type":"v1","addr":"172.21.2.113:6821","nonce":36365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6823","nonce":36365},{"type":"v1","addr":"172.21.2.113:6825","nonce":36365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6830","nonce":36365},{"type":"v1","addr":"172.21.2.113:6831","nonce":36365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6827","nonce":36365},{"type":"v1","addr":"172.21.2.113:6829","nonce":36365}]},"public_addr":"172.21.2.113:6821/36365","cluster_addr":"172.21.2.113:6825/36365","heartbeat_back_addr":"172.21.2.113:6831/36365","heartbeat_front_addr":"172.21.2.113:6829/36365","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:05:28.979 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:28.980 7f995d711700 1 -- 172.21.2.102:0/1684493116 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f993c0186c0 msgr2=0x7f993c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:28.979 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:28.980 7f995d711700 1 --2- 172.21.2.102:0/1684493116 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f993c0186c0 0x7f993c01abc0 crc :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:28.979 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:28.980 7f995d711700 1 -- 172.21.2.102:0/1684493116 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f995818fd50 msgr2=0x7f99580a9cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:28.979 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:28.980 7f995d711700 1 --2- 172.21.2.102:0/1684493116 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f995818fd50 0x7f99580a9cf0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 rx=0x7f9948009630 tx=0x7f994800b980).stop 2021-02-10T11:05:28.980 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:28.981 7f995d711700 1 -- 172.21.2.102:0/1684493116 shutdown_connections 2021-02-10T11:05:28.980 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:28.981 7f995d711700 1 --2- 172.21.2.102:0/1684493116 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f99580aa310 0x7f9958086c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:05:28.980 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:28.981 7f995d711700 1 --2- 172.21.2.102:0/1684493116 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f995818fd50 0x7f99580a9cf0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:28.980 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:28.981 7f995d711700 1 --2- 172.21.2.102:0/1684493116 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f993c0186c0 0x7f993c01abc0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:28.980 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:28.981 7f995d711700 1 --2- 172.21.2.102:0/1684493116 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f9958084100 0x7f995818f730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:05:28.980 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:28.981 7f995d711700 1 -- 172.21.2.102:0/1684493116 >> 172.21.2.102:0/1684493116 conn(0x7f99581a8840 msgr2=0x7f9958107690 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:05:28.980 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:28.981 7f995d711700 1 -- 172.21.2.102:0/1684493116 shutdown_connections 2021-02-10T11:05:28.980 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:28.981 7f995d711700 1 -- 172.21.2.102:0/1684493116 wait complete. 2021-02-10T11:05:28.988 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:03:37.322519', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 4, 'pool_name': 'cephfs_metadata', 'create_time': '2021-02-10 11:05:10.538786', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '29', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 5, 'pool_name': 'cephfs_data', 'create_time': '2021-02-10 11:05:11.559128', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '30', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-02-10T11:05:28.988 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:05:29.422 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:29.403 7ff35e162700 1 Processor -- start 2021-02-10T11:05:29.422 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:29.403 7ff35e162700 1 -- start start 2021-02-10T11:05:29.422 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:29.404 7ff35e162700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7ff3580f05f0 0x7ff3580f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:29.422 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:29.404 7ff35e162700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3581b0c70 con 0x7ff3581b0870 2021-02-10T11:05:29.422 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:29.404 7ff35e162700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3580f0b50 con 0x7ff3581b0470 2021-02-10T11:05:29.422 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:29.404 7ff35e162700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3580e5ea0 con 0x7ff3580f05f0 2021-02-10T11:05:29.422 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:29.404 7ff356ffd700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7ff3581b0870 0x7ff3580efe00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:42992/0 (socket says 172.21.2.102:42992) 2021-02-10T11:05:29.422 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:29.404 7ff356ffd700 1 -- 172.21.2.102:0/3006294930 learned_addr learned my addr 172.21.2.102:0/3006294930 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:05:29.423 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:29.404 7ff3567fc700 1 -- 172.21.2.102:0/3006294930 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2615509823 0 0) 0x7ff348001180 con 0x7ff3581b0870 2021-02-10T11:05:29.423 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:29.404 7ff3567fc700 1 -- 172.21.2.102:0/3006294930 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff344002fe0 con 0x7ff3581b0870 2021-02-10T11:05:29.423 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:29.405 7ff357fff700 1 --2- 172.21.2.102:0/3006294930 >> v2:172.21.2.113:3300/0 conn(0x7ff3580f05f0 0x7ff3580f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:29.423 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:29.405 7ff3567fc700 1 -- 172.21.2.102:0/3006294930 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 366742605 0 0) 0x7ff348001180 con 0x7ff3581b0870 2021-02-10T11:05:29.423 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:29.405 7ff3567fc700 1 -- 172.21.2.102:0/3006294930 >> v2:172.21.2.113:3300/0 conn(0x7ff3580f05f0 msgr2=0x7ff3580f20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:05:29.423 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:29.405 7ff3567fc700 1 --2- 172.21.2.102:0/3006294930 >> v2:172.21.2.113:3300/0 conn(0x7ff3580f05f0 0x7ff3580f20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:29.423 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:29.405 7ff3567fc700 1 -- 172.21.2.102:0/3006294930 >> v1:172.21.2.113:6789/0 conn(0x7ff3581b0470 legacy=0x7ff3580ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:29.423 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:29.405 7ff3567fc700 1 -- 172.21.2.102:0/3006294930 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3580e52e0 con 0x7ff3581b0870 2021-02-10T11:05:29.423 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:29.405 7ff3567fc700 1 -- 172.21.2.102:0/3006294930 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7ff348001680 con 0x7ff3581b0870 2021-02-10T11:05:29.423 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:29.405 7ff3567fc700 1 -- 172.21.2.102:0/3006294930 >> v1:172.21.2.102:6789/0 conn(0x7ff3581b0870 legacy=0x7ff3580efe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:29.423 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:29.405 7ff3567fc700 1 --2- 172.21.2.102:0/3006294930 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff344003a60 0x7ff344005f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:29.424 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:29.405 7ff3567fc700 1 --2- 172.21.2.102:0/3006294930 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff3440067b0 0x7ff344008cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:29.424 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:29.405 7ff3567fc700 1 --2- 172.21.2.102:0/3006294930 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff344009590 0x7ff34400bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:29.424 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:29.405 7ff3567fc700 1 -- 172.21.2.102:0/3006294930 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff34400c300 con 0x7ff344009590 2021-02-10T11:05:29.424 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:29.405 7ff3567fc700 1 -- 172.21.2.102:0/3006294930 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff34400c520 con 0x7ff3440067b0 2021-02-10T11:05:29.424 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:29.405 7ff3567fc700 1 -- 172.21.2.102:0/3006294930 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff34400c740 con 0x7ff344003a60 2021-02-10T11:05:29.424 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:29.405 7ff357fff700 1 --2- 172.21.2.102:0/3006294930 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff344009590 0x7ff34400bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:29.424 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:29.405 7ff356ffd700 1 --2- 172.21.2.102:0/3006294930 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff3440067b0 0x7ff344008cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:29.424 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:29.405 7ff3577fe700 1 --2- 172.21.2.102:0/3006294930 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff344003a60 0x7ff344005f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:29.424 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:29.405 7ff357fff700 1 -- 172.21.2.102:0/3006294930 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff344003a60 msgr2=0x7ff344005f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:05:29.424 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:29.405 7ff357fff700 1 --2- 172.21.2.102:0/3006294930 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff344003a60 0x7ff344005f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:29.424 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:29.405 7ff357fff700 1 -- 172.21.2.102:0/3006294930 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff3440067b0 msgr2=0x7ff344008cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:05:29.424 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:29.405 7ff357fff700 1 --2- 172.21.2.102:0/3006294930 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff3440067b0 0x7ff344008cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:29.425 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:29.405 7ff357fff700 1 -- 172.21.2.102:0/3006294930 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff344050980 con 0x7ff344009590 2021-02-10T11:05:29.425 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:29.405 7ff357fff700 1 -- 172.21.2.102:0/3006294930 reap_dead start 2021-02-10T11:05:29.425 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:29.406 7ff357fff700 1 --2- 172.21.2.102:0/3006294930 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff344009590 0x7ff34400bab0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 rx=0x7ff350006a40 tx=0x7ff35000c0c0).ready entity=mon.0 client_cookie=3e2052fe7fd0fc17 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:05:29.425 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:29.406 7ff3567fc700 1 -- 172.21.2.102:0/3006294930 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff350006ab0 con 0x7ff344009590 2021-02-10T11:05:29.425 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:29.406 7ff3567fc700 1 -- 172.21.2.102:0/3006294930 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff350007070 con 0x7ff344009590 2021-02-10T11:05:29.425 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:29.406 7ff3567fc700 1 -- 172.21.2.102:0/3006294930 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff350007550 con 0x7ff344009590 2021-02-10T11:05:29.425 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:29.406 7ff35e162700 1 -- 172.21.2.102:0/3006294930 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff344009590 msgr2=0x7ff34400bab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:29.425 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:29.406 7ff35e162700 1 --2- 172.21.2.102:0/3006294930 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff344009590 0x7ff34400bab0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 rx=0x7ff350006a40 tx=0x7ff35000c0c0).stop 2021-02-10T11:05:29.425 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:29.406 7ff35e162700 1 -- 172.21.2.102:0/3006294930 shutdown_connections 2021-02-10T11:05:29.425 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:29.406 7ff35e162700 1 --2- 172.21.2.102:0/3006294930 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff344009590 0x7ff34400bab0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:29.425 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:29.406 7ff35e162700 1 -- 172.21.2.102:0/3006294930 >> 172.21.2.102:0/3006294930 conn(0x7ff3580ce3e0 msgr2=0x7ff3580e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:05:29.426 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:29.406 7ff35e162700 1 -- 172.21.2.102:0/3006294930 shutdown_connections 2021-02-10T11:05:29.426 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:29.406 7ff35e162700 1 -- 172.21.2.102:0/3006294930 wait complete. 2021-02-10T11:05:29.426 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:29.421 7ff35e162700 1 Processor -- start 2021-02-10T11:05:29.426 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:29.421 7ff35e162700 1 -- start start 2021-02-10T11:05:29.426 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:29.422 7ff35e162700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff35809e560 0x7ff3580a0130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:29.426 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:29.422 7ff35e162700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff35809e960 0x7ff3580a0750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:29.426 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:29.422 7ff35e162700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff3580a0d70 0x7ff3580a1af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:29.426 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:29.422 7ff35e162700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3580c38f0 con 0x7ff35809e560 2021-02-10T11:05:29.426 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:29.422 7ff35e162700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3580a1170 con 0x7ff35809e960 2021-02-10T11:05:29.426 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:29.422 7ff35e162700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3580a1390 con 0x7ff3580a0d70 2021-02-10T11:05:29.426 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:29.422 7ff3577fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff35809e560 0x7ff3580a0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:29.427 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:29.422 7ff3577fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff35809e560 0x7ff3580a0130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:50782/0 (socket says 172.21.2.102:50782) 2021-02-10T11:05:29.427 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:29.422 7ff3577fe700 1 -- 172.21.2.102:0/1898990421 learned_addr learned my addr 172.21.2.102:0/1898990421 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:05:29.427 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:29.422 7ff356ffd700 1 --2- 172.21.2.102:0/1898990421 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff35809e960 0x7ff3580a0750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:29.427 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:29.422 7ff3577fe700 1 -- 172.21.2.102:0/1898990421 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff3580a0d70 msgr2=0x7ff3580a1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:29.427 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:29.422 7ff357fff700 1 --2- 172.21.2.102:0/1898990421 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff3580a0d70 0x7ff3580a1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:29.427 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:29.422 7ff3577fe700 1 --2- 172.21.2.102:0/1898990421 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff3580a0d70 0x7ff3580a1af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:29.427 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:29.422 7ff3577fe700 1 -- 172.21.2.102:0/1898990421 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff35809e960 msgr2=0x7ff3580a0750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:29.427 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:29.422 7ff3577fe700 1 --2- 172.21.2.102:0/1898990421 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff35809e960 0x7ff3580a0750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:29.427 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:29.422 7ff3577fe700 1 -- 172.21.2.102:0/1898990421 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3580a2110 con 0x7ff35809e560 2021-02-10T11:05:29.427 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:29.422 7ff357fff700 1 --2- 172.21.2.102:0/1898990421 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff3580a0d70 0x7ff3580a1af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:05:29.427 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:29.422 7ff356ffd700 1 --2- 172.21.2.102:0/1898990421 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff35809e960 0x7ff3580a0750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:05:29.428 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:29.423 7ff3577fe700 1 --2- 172.21.2.102:0/1898990421 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff35809e560 0x7ff3580a0130 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 rx=0x7ff340006f10 tx=0x7ff34000b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:05:29.428 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:29.423 7ff34ffff700 1 -- 172.21.2.102:0/1898990421 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff34000bd80 con 0x7ff35809e560 2021-02-10T11:05:29.428 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:29.423 7ff35e162700 1 -- 172.21.2.102:0/1898990421 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3580a2110 con 0x7ff35809e560 2021-02-10T11:05:29.428 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:29.423 7ff34ffff700 1 -- 172.21.2.102:0/1898990421 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff34000bd80 con 0x7ff35809e560 2021-02-10T11:05:29.429 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:29.423 7ff34ffff700 1 -- 172.21.2.102:0/1898990421 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff340006220 con 0x7ff35809e560 2021-02-10T11:05:29.429 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:29.423 7ff35e162700 1 -- 172.21.2.102:0/1898990421 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3580a2110 con 0x7ff35809e560 2021-02-10T11:05:29.431 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:29.424 7ff34ffff700 1 -- 172.21.2.102:0/1898990421 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ff340006030 con 0x7ff35809e560 2021-02-10T11:05:29.432 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:29.424 7ff34ffff700 1 --2- 172.21.2.102:0/1898990421 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff33c018730 0x7ff33c01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:29.432 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:29.424 7ff34ffff700 1 -- 172.21.2.102:0/1898990421 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7ff340032660 con 0x7ff35809e560 2021-02-10T11:05:29.432 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:29.424 7ff356ffd700 1 --2- 172.21.2.102:0/1898990421 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff33c018730 0x7ff33c01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:29.432 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:29.424 7ff356ffd700 1 --2- 172.21.2.102:0/1898990421 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff33c018730 0x7ff33c01ac10 crc :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:05:29.452 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:29.454 7ff35e162700 1 -- 172.21.2.102:0/1898990421 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff358105b30 con 0x7ff35809e560 2021-02-10T11:05:29.455 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:29.456 7ff34ffff700 1 -- 172.21.2.102:0/1898990421 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ff340006ba0 con 0x7ff35809e560 2021-02-10T11:05:29.585 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:29.585 7ff35e162700 1 -- 172.21.2.102:0/1898990421 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7ff3580f0b50 con 0x7ff35809e560 2021-02-10T11:05:29.585 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:29.586 7ff34ffff700 1 -- 172.21.2.102:0/1898990421 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v30) v1 ==== 92+0+10 (secure 0 0 0) 0x7ff34000bd80 con 0x7ff35809e560 2021-02-10T11:05:29.585 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:05:29.587 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:29.588 7ff35e162700 1 -- 172.21.2.102:0/1898990421 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff33c018730 msgr2=0x7ff33c01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:29.587 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:29.588 7ff35e162700 1 --2- 172.21.2.102:0/1898990421 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff33c018730 0x7ff33c01ac10 crc :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:29.587 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:29.588 7ff35e162700 1 -- 172.21.2.102:0/1898990421 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff35809e560 msgr2=0x7ff3580a0130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:29.587 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:29.588 7ff35e162700 1 --2- 172.21.2.102:0/1898990421 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff35809e560 0x7ff3580a0130 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 rx=0x7ff340006f10 tx=0x7ff34000b990).stop 2021-02-10T11:05:29.588 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:29.588 7ff35e162700 1 -- 172.21.2.102:0/1898990421 shutdown_connections 2021-02-10T11:05:29.588 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:29.588 7ff35e162700 1 --2- 172.21.2.102:0/1898990421 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff3580a0d70 0x7ff3580a1af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:29.588 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:29.588 7ff35e162700 1 --2- 172.21.2.102:0/1898990421 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff33c018730 0x7ff33c01ac10 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:29.588 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:29.588 7ff35e162700 1 --2- 172.21.2.102:0/1898990421 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff35809e960 0x7ff3580a0750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:29.588 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:29.588 7ff35e162700 1 --2- 172.21.2.102:0/1898990421 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff35809e560 0x7ff3580a0130 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:29.588 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:29.588 7ff35e162700 1 -- 172.21.2.102:0/1898990421 >> 172.21.2.102:0/1898990421 conn(0x7ff35809fc10 msgr2=0x7ff358094ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:05:29.589 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:29.588 7ff35e162700 1 -- 172.21.2.102:0/1898990421 shutdown_connections 2021-02-10T11:05:29.589 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:29.588 7ff35e162700 1 -- 172.21.2.102:0/1898990421 wait complete. 2021-02-10T11:05:29.595 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-02-10T11:05:30.025 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:30.009 7fb4fa8f5700 1 Processor -- start 2021-02-10T11:05:30.025 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:30.009 7fb4fa8f5700 1 -- start start 2021-02-10T11:05:30.025 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:30.009 7fb4fa8f5700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fb4f40f05f0 0x7fb4f40f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:30.026 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:30.009 7fb4fa8f5700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4f41b0c70 con 0x7fb4f41b0870 2021-02-10T11:05:30.026 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:30.009 7fb4fa8f5700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4f40f0b50 con 0x7fb4f41b0470 2021-02-10T11:05:30.026 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:30.009 7fb4fa8f5700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4f40e5ea0 con 0x7fb4f40f05f0 2021-02-10T11:05:30.026 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:30.010 7fb4f37fe700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fb4f41b0870 0x7fb4f40efe00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:43012/0 (socket says 172.21.2.102:43012) 2021-02-10T11:05:30.026 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:30.010 7fb4f37fe700 1 -- 172.21.2.102:0/756048168 learned_addr learned my addr 172.21.2.102:0/756048168 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:05:30.026 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:30.010 7fb4f2ffd700 1 -- 172.21.2.102:0/756048168 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1511138274 0 0) 0x7fb4e80010b0 con 0x7fb4f41b0870 2021-02-10T11:05:30.026 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:30.010 7fb4f2ffd700 1 -- 172.21.2.102:0/756048168 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb4dc002fe0 con 0x7fb4f41b0870 2021-02-10T11:05:30.026 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:30.010 7fb4f8f12700 1 --2- 172.21.2.102:0/756048168 >> v2:172.21.2.113:3300/0 conn(0x7fb4f40f05f0 0x7fb4f40f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:30.026 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:30.010 7fb4f2ffd700 1 -- 172.21.2.102:0/756048168 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 436235924 0 0) 0x7fb4e80010b0 con 0x7fb4f41b0870 2021-02-10T11:05:30.026 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:30.010 7fb4f2ffd700 1 -- 172.21.2.102:0/756048168 >> v2:172.21.2.113:3300/0 conn(0x7fb4f40f05f0 msgr2=0x7fb4f40f20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:05:30.026 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:30.010 7fb4f2ffd700 1 --2- 172.21.2.102:0/756048168 >> v2:172.21.2.113:3300/0 conn(0x7fb4f40f05f0 0x7fb4f40f20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:30.026 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:30.010 7fb4f2ffd700 1 -- 172.21.2.102:0/756048168 >> v1:172.21.2.113:6789/0 conn(0x7fb4f41b0470 legacy=0x7fb4f40ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:30.027 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:30.010 7fb4f2ffd700 1 -- 172.21.2.102:0/756048168 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4f40e52e0 con 0x7fb4f41b0870 2021-02-10T11:05:30.027 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:30.010 7fb4f2ffd700 1 -- 172.21.2.102:0/756048168 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7fb4e80015b0 con 0x7fb4f41b0870 2021-02-10T11:05:30.027 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:30.010 7fb4f2ffd700 1 -- 172.21.2.102:0/756048168 >> v1:172.21.2.102:6789/0 conn(0x7fb4f41b0870 legacy=0x7fb4f40efe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:30.027 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:30.010 7fb4f2ffd700 1 --2- 172.21.2.102:0/756048168 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb4dc003aa0 0x7fb4dc005ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:30.027 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:30.010 7fb4f2ffd700 1 --2- 172.21.2.102:0/756048168 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb4dc006770 0x7fb4dc008c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:30.027 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:30.011 7fb4f2ffd700 1 --2- 172.21.2.102:0/756048168 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb4dc009590 0x7fb4dc00bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:30.027 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:30.011 7fb4f2ffd700 1 -- 172.21.2.102:0/756048168 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4dc00c300 con 0x7fb4dc003aa0 2021-02-10T11:05:30.027 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:30.011 7fb4f2ffd700 1 -- 172.21.2.102:0/756048168 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4dc00c520 con 0x7fb4dc006770 2021-02-10T11:05:30.027 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:30.011 7fb4f2ffd700 1 -- 172.21.2.102:0/756048168 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4dc050750 con 0x7fb4dc009590 2021-02-10T11:05:30.027 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:30.011 7fb4f3fff700 1 --2- 172.21.2.102:0/756048168 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb4dc003aa0 0x7fb4dc005ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:30.027 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:30.011 7fb4f37fe700 1 --2- 172.21.2.102:0/756048168 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb4dc006770 0x7fb4dc008c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:30.027 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:30.011 7fb4f3fff700 1 -- 172.21.2.102:0/756048168 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb4dc009590 msgr2=0x7fb4dc00bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:05:30.027 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:30.011 7fb4f3fff700 1 --2- 172.21.2.102:0/756048168 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb4dc009590 0x7fb4dc00bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:05:30.028 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:30.011 7fb4f3fff700 1 -- 172.21.2.102:0/756048168 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb4dc006770 msgr2=0x7fb4dc008c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:05:30.028 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:30.011 7fb4f3fff700 1 --2- 172.21.2.102:0/756048168 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb4dc006770 0x7fb4dc008c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:30.028 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:30.011 7fb4f3fff700 1 -- 172.21.2.102:0/756048168 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4dc0509b0 con 0x7fb4dc003aa0 2021-02-10T11:05:30.028 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:30.011 7fb4f8f12700 1 -- 172.21.2.102:0/756048168 reap_dead start 2021-02-10T11:05:30.028 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:30.011 7fb4f3fff700 1 --2- 172.21.2.102:0/756048168 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb4dc003aa0 0x7fb4dc005ef0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 rx=0x7fb4e4006ac0 tx=0x7fb4e4009af0).ready entity=mon.0 client_cookie=2d5cc3a514eeae86 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:05:30.028 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:30.011 7fb4f2ffd700 1 -- 172.21.2.102:0/756048168 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb4e4007280 con 0x7fb4dc003aa0 2021-02-10T11:05:30.028 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:30.011 7fb4f2ffd700 1 -- 172.21.2.102:0/756048168 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4e4007600 con 0x7fb4dc003aa0 2021-02-10T11:05:30.028 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:30.011 7fb4f2ffd700 1 -- 172.21.2.102:0/756048168 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb4e4007ae0 con 0x7fb4dc003aa0 2021-02-10T11:05:30.028 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:30.011 7fb4fa8f5700 1 -- 172.21.2.102:0/756048168 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb4dc003aa0 msgr2=0x7fb4dc005ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:30.028 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:30.011 7fb4fa8f5700 1 --2- 172.21.2.102:0/756048168 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb4dc003aa0 0x7fb4dc005ef0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 rx=0x7fb4e4006ac0 tx=0x7fb4e4009af0).stop 2021-02-10T11:05:30.028 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:30.012 7fb4fa8f5700 1 -- 172.21.2.102:0/756048168 shutdown_connections 2021-02-10T11:05:30.028 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:30.012 7fb4fa8f5700 1 --2- 172.21.2.102:0/756048168 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb4dc003aa0 0x7fb4dc005ef0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:30.028 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:30.012 7fb4fa8f5700 1 -- 172.21.2.102:0/756048168 >> 172.21.2.102:0/756048168 conn(0x7fb4f40ce3e0 msgr2=0x7fb4f40e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:05:30.029 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:30.012 7fb4fa8f5700 1 -- 172.21.2.102:0/756048168 shutdown_connections 2021-02-10T11:05:30.029 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:30.012 7fb4fa8f5700 1 -- 172.21.2.102:0/756048168 wait complete. 2021-02-10T11:05:30.029 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:30.027 7fb4fa8f5700 1 Processor -- start 2021-02-10T11:05:30.029 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:30.027 7fb4fa8f5700 1 -- start start 2021-02-10T11:05:30.031 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:30.027 7fb4fa8f5700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb4f409b930 0x7fb4f40b7c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:30.031 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:30.027 7fb4fa8f5700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb4f40b8410 0x7fb4f40b8810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:30.031 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:30.027 7fb4fa8f5700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb4f40c1f80 0x7fb4f40c2380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:30.031 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:30.027 7fb4fa8f5700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4f40c1a00 con 0x7fb4f40c1f80 2021-02-10T11:05:30.031 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:30.027 7fb4fa8f5700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4f40b8e30 con 0x7fb4f409b930 2021-02-10T11:05:30.031 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:30.027 7fb4fa8f5700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4f40b9050 con 0x7fb4f40b8410 2021-02-10T11:05:30.031 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:30.028 7fb4f8f12700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb4f40c1f80 0x7fb4f40c2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:30.031 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:30.028 7fb4f3fff700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb4f409b930 0x7fb4f40b7c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:30.031 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:30.028 7fb4f8f12700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb4f40c1f80 0x7fb4f40c2380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:50802/0 (socket says 172.21.2.102:50802) 2021-02-10T11:05:30.031 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:30.028 7fb4f3fff700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb4f409b930 0x7fb4f40b7c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.113:3300/0 says I am v2:172.21.2.102:54230/0 (socket says 172.21.2.102:54230) 2021-02-10T11:05:30.032 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:30.028 7fb4f8f12700 1 -- 172.21.2.102:0/140432485 learned_addr learned my addr 172.21.2.102:0/140432485 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:05:30.032 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:30.028 7fb4f37fe700 1 --2- 172.21.2.102:0/140432485 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb4f40b8410 0x7fb4f40b8810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:30.032 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:30.028 7fb4f8f12700 1 -- 172.21.2.102:0/140432485 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb4f40b8410 msgr2=0x7fb4f40b8810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:30.032 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:30.028 7fb4f8f12700 1 --2- 172.21.2.102:0/140432485 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb4f40b8410 0x7fb4f40b8810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:30.032 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:30.028 7fb4f8f12700 1 -- 172.21.2.102:0/140432485 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb4f409b930 msgr2=0x7fb4f40b7c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:30.032 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:30.028 7fb4f8f12700 1 --2- 172.21.2.102:0/140432485 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb4f409b930 0x7fb4f40b7c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:30.032 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:30.028 7fb4f8f12700 1 -- 172.21.2.102:0/140432485 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4f40c29a0 con 0x7fb4f40c1f80 2021-02-10T11:05:30.032 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:30.028 7fb4f8f12700 1 --2- 172.21.2.102:0/140432485 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb4f40c1f80 0x7fb4f40c2380 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 rx=0x7fb4ec0099a0 tx=0x7fb4ec00bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:05:30.032 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:30.028 7fb4f0ff9700 1 -- 172.21.2.102:0/140432485 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb4ec0066f0 con 0x7fb4f40c1f80 2021-02-10T11:05:30.033 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:30.028 7fb4f0ff9700 1 -- 172.21.2.102:0/140432485 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4ec00c2f0 con 0x7fb4f40c1f80 2021-02-10T11:05:30.033 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:30.028 7fb4f0ff9700 1 -- 172.21.2.102:0/140432485 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb4ec00c7d0 con 0x7fb4f40c1f80 2021-02-10T11:05:30.033 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:30.028 7fb4fa8f5700 1 -- 172.21.2.102:0/140432485 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4f40c29a0 con 0x7fb4f40c1f80 2021-02-10T11:05:30.033 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:30.028 7fb4fa8f5700 1 -- 172.21.2.102:0/140432485 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4f40c2f30 con 0x7fb4f40c1f80 2021-02-10T11:05:30.033 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:30.029 7fb4f0ff9700 1 -- 172.21.2.102:0/140432485 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fb4ec0011f0 con 0x7fb4f40c1f80 2021-02-10T11:05:30.033 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:30.029 7fb4f0ff9700 1 --2- 172.21.2.102:0/140432485 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb4d40186d0 0x7fb4d401abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:30.033 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:30.029 7fb4f0ff9700 1 -- 172.21.2.102:0/140432485 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fb4ec0329c0 con 0x7fb4f40c1f80 2021-02-10T11:05:30.033 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:30.029 7fb4f3fff700 1 --2- 172.21.2.102:0/140432485 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb4d40186d0 0x7fb4d401abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:30.033 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:30.029 7fb4f3fff700 1 --2- 172.21.2.102:0/140432485 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb4d40186d0 0x7fb4d401abd0 crc :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:05:30.058 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:30.059 7fb4fa8f5700 1 -- 172.21.2.102:0/140432485 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4f4105b30 con 0x7fb4f40c1f80 2021-02-10T11:05:30.060 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:30.062 7fb4f0ff9700 1 -- 172.21.2.102:0/140432485 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fb4ec000f90 con 0x7fb4f40c1f80 2021-02-10T11:05:30.190 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:30.191 7fb4fa8f5700 1 -- 172.21.2.102:0/140432485 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7fb4f40f0b50 con 0x7fb4f40c1f80 2021-02-10T11:05:30.190 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:30.191 7fb4f0ff9700 1 -- 172.21.2.102:0/140432485 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v30) v1 ==== 104+0+10 (secure 0 0 0) 0x7fb4ec01f880 con 0x7fb4f40c1f80 2021-02-10T11:05:30.190 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:05:30.192 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:30.193 7fb4fa8f5700 1 -- 172.21.2.102:0/140432485 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb4d40186d0 msgr2=0x7fb4d401abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:30.192 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:30.193 7fb4fa8f5700 1 --2- 172.21.2.102:0/140432485 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb4d40186d0 0x7fb4d401abd0 crc :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:30.192 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:30.193 7fb4fa8f5700 1 -- 172.21.2.102:0/140432485 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb4f40c1f80 msgr2=0x7fb4f40c2380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:30.192 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:30.193 7fb4fa8f5700 1 --2- 172.21.2.102:0/140432485 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb4f40c1f80 0x7fb4f40c2380 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 rx=0x7fb4ec0099a0 tx=0x7fb4ec00bcf0).stop 2021-02-10T11:05:30.192 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:30.193 7fb4fa8f5700 1 -- 172.21.2.102:0/140432485 shutdown_connections 2021-02-10T11:05:30.192 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:30.193 7fb4fa8f5700 1 --2- 172.21.2.102:0/140432485 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb4f40c1f80 0x7fb4f40c2380 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:30.192 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:30.193 7fb4fa8f5700 1 --2- 172.21.2.102:0/140432485 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb4f40b8410 0x7fb4f40b8810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:30.192 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:30.193 7fb4fa8f5700 1 --2- 172.21.2.102:0/140432485 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb4d40186d0 0x7fb4d401abd0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:30.192 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:30.193 7fb4fa8f5700 1 --2- 172.21.2.102:0/140432485 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb4f409b930 0x7fb4f40b7c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:30.192 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:30.193 7fb4fa8f5700 1 -- 172.21.2.102:0/140432485 >> 172.21.2.102:0/140432485 conn(0x7fb4f4094e10 msgr2=0x7fb4f40a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:05:30.193 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:30.194 7fb4fa8f5700 1 -- 172.21.2.102:0/140432485 shutdown_connections 2021-02-10T11:05:30.193 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:30.194 7fb4fa8f5700 1 -- 172.21.2.102:0/140432485 wait complete. 2021-02-10T11:05:30.200 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-02-10T11:05:30.630 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:30.615 7fee3a245700 1 Processor -- start 2021-02-10T11:05:30.630 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:30.615 7fee3a245700 1 -- start start 2021-02-10T11:05:30.630 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:30.615 7fee3a245700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fee341b0470 0x7fee341b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:30.630 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:30.615 7fee3a245700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fee340ef610 0x7fee340efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:30.630 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:30.615 7fee3a245700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee340f0b50 con 0x7fee340f0030 2021-02-10T11:05:30.630 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:30.615 7fee3a245700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee340e52e0 con 0x7fee341b0470 2021-02-10T11:05:30.630 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:30.615 7fee3a245700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee340e54d0 con 0x7fee340ef610 2021-02-10T11:05:30.630 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:30.616 7fee38862700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fee340f0030 0x7fee340f10d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:43034/0 (socket says 172.21.2.102:43034) 2021-02-10T11:05:30.631 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:30.616 7fee38862700 1 -- 172.21.2.102:0/217781612 learned_addr learned my addr 172.21.2.102:0/217781612 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:05:30.631 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:30.616 7fee33fff700 1 --2- 172.21.2.102:0/217781612 >> v2:172.21.2.113:3300/0 conn(0x7fee341b0470 0x7fee341b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:30.631 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:30.616 7fee337fe700 1 --2- 172.21.2.102:0/217781612 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fee340ef610 0x7fee340efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:30.631 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:30.616 7fee32ffd700 1 -- 172.21.2.102:0/217781612 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3890105936 0 0) 0x7fee2c001500 con 0x7fee340f0030 2021-02-10T11:05:30.631 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:30.616 7fee32ffd700 1 -- 172.21.2.102:0/217781612 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee1c002fe0 con 0x7fee340f0030 2021-02-10T11:05:30.631 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:30.616 7fee33fff700 1 -- 172.21.2.102:0/217781612 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fee340ef610 msgr2=0x7fee340efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:05:30.631 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:30.616 7fee32ffd700 1 -- 172.21.2.102:0/217781612 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1662850720 0 0) 0x7fee2c001500 con 0x7fee340f0030 2021-02-10T11:05:30.631 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:30.616 7fee33fff700 1 --2- 172.21.2.102:0/217781612 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fee340ef610 0x7fee340efa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:30.631 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:30.616 7fee33fff700 1 -- 172.21.2.102:0/217781612 >> v1:172.21.2.102:6789/0 conn(0x7fee340f0030 legacy=0x7fee340f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:30.631 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:30.616 7fee33fff700 1 -- 172.21.2.102:0/217781612 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee340f0430 con 0x7fee341b0470 2021-02-10T11:05:30.631 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:30.616 7fee337fe700 1 --2- 172.21.2.102:0/217781612 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fee340ef610 0x7fee340efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:05:30.631 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:30.616 7fee33fff700 1 --2- 172.21.2.102:0/217781612 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fee341b0470 0x7fee341b0870 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7fee24006650 tx=0x7fee24009a80).ready entity=mon.1 client_cookie=a2cacd5cd5b86669 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:05:30.632 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:30.616 7fee32ffd700 1 -- 172.21.2.102:0/217781612 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fee24006390 con 0x7fee341b0470 2021-02-10T11:05:30.632 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:30.617 7fee32ffd700 1 -- 172.21.2.102:0/217781612 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee24006740 con 0x7fee341b0470 2021-02-10T11:05:30.632 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:30.617 7fee32ffd700 1 -- 172.21.2.102:0/217781612 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fee24006390 con 0x7fee341b0470 2021-02-10T11:05:30.632 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:30.617 7fee3a245700 1 -- 172.21.2.102:0/217781612 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fee341b0470 msgr2=0x7fee341b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:30.632 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:30.617 7fee3a245700 1 --2- 172.21.2.102:0/217781612 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fee341b0470 0x7fee341b0870 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7fee24006650 tx=0x7fee24009a80).stop 2021-02-10T11:05:30.632 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:30.617 7fee3a245700 1 -- 172.21.2.102:0/217781612 shutdown_connections 2021-02-10T11:05:30.632 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:30.617 7fee3a245700 1 --2- 172.21.2.102:0/217781612 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fee340ef610 0x7fee340efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:30.632 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:30.617 7fee3a245700 1 --2- 172.21.2.102:0/217781612 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fee341b0470 0x7fee341b0870 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:30.632 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:30.617 7fee3a245700 1 -- 172.21.2.102:0/217781612 >> 172.21.2.102:0/217781612 conn(0x7fee340ce3e0 msgr2=0x7fee340e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:05:30.632 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:30.617 7fee3a245700 1 -- 172.21.2.102:0/217781612 shutdown_connections 2021-02-10T11:05:30.632 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:30.617 7fee3a245700 1 -- 172.21.2.102:0/217781612 wait complete. 2021-02-10T11:05:30.634 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:30.632 7fee3a245700 1 Processor -- start 2021-02-10T11:05:30.634 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:30.632 7fee3a245700 1 -- start start 2021-02-10T11:05:30.634 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:30.632 7fee3a245700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fee340a0420 0x7fee340a0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:30.634 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:30.632 7fee3a245700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fee34189d00 0x7fee3418a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:30.634 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:30.632 7fee3a245700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fee3418a720 0x7fee340b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:30.634 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:30.632 7fee3a245700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee34189800 con 0x7fee34189d00 2021-02-10T11:05:30.634 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:30.632 7fee3a245700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee3418ab20 con 0x7fee3418a720 2021-02-10T11:05:30.634 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:30.632 7fee3a245700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee340a1140 con 0x7fee340a0420 2021-02-10T11:05:30.635 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:30.632 7fee337fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fee34189d00 0x7fee3418a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:30.635 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:30.632 7fee38862700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fee3418a720 0x7fee340b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:30.635 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:30.632 7fee337fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fee34189d00 0x7fee3418a100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:50818/0 (socket says 172.21.2.102:50818) 2021-02-10T11:05:30.635 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:30.632 7fee337fe700 1 -- 172.21.2.102:0/2846247133 learned_addr learned my addr 172.21.2.102:0/2846247133 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:05:30.635 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:30.632 7fee38862700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fee3418a720 0x7fee340b7be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.113:3300/0 says I am v2:172.21.2.102:54244/0 (socket says 172.21.2.102:54244) 2021-02-10T11:05:30.635 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:30.632 7fee33fff700 1 --2- 172.21.2.102:0/2846247133 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fee340a0420 0x7fee340a0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:30.635 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:30.633 7fee337fe700 1 -- 172.21.2.102:0/2846247133 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fee340a0420 msgr2=0x7fee340a0820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:30.635 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:30.633 7fee337fe700 1 --2- 172.21.2.102:0/2846247133 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fee340a0420 0x7fee340a0820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:30.635 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:30.633 7fee337fe700 1 -- 172.21.2.102:0/2846247133 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fee3418a720 msgr2=0x7fee340b7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:30.635 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:30.633 7fee337fe700 1 --2- 172.21.2.102:0/2846247133 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fee3418a720 0x7fee340b7be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:30.635 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:30.633 7fee337fe700 1 -- 172.21.2.102:0/2846247133 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee340b8200 con 0x7fee34189d00 2021-02-10T11:05:30.635 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:30.633 7fee337fe700 1 --2- 172.21.2.102:0/2846247133 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fee34189d00 0x7fee3418a100 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 rx=0x7fee28006f40 tx=0x7fee2800b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:05:30.635 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:30.633 7fee30ff9700 1 -- 172.21.2.102:0/2846247133 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fee2800bd80 con 0x7fee34189d00 2021-02-10T11:05:30.636 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:30.633 7fee30ff9700 1 -- 172.21.2.102:0/2846247133 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee28000c90 con 0x7fee34189d00 2021-02-10T11:05:30.636 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:30.633 7fee30ff9700 1 -- 172.21.2.102:0/2846247133 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fee280022b0 con 0x7fee34189d00 2021-02-10T11:05:30.636 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:30.633 7fee3a245700 1 -- 172.21.2.102:0/2846247133 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee340b8200 con 0x7fee34189d00 2021-02-10T11:05:30.636 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:30.633 7fee3a245700 1 -- 172.21.2.102:0/2846247133 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee340b8200 con 0x7fee34189d00 2021-02-10T11:05:30.636 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:30.634 7fee30ff9700 1 -- 172.21.2.102:0/2846247133 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fee28006b00 con 0x7fee34189d00 2021-02-10T11:05:30.636 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:30.634 7fee30ff9700 1 --2- 172.21.2.102:0/2846247133 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fee140186c0 0x7fee1401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:30.636 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:30.634 7fee30ff9700 1 -- 172.21.2.102:0/2846247133 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fee28032490 con 0x7fee34189d00 2021-02-10T11:05:30.636 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:30.634 7fee33fff700 1 --2- 172.21.2.102:0/2846247133 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fee140186c0 0x7fee1401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:30.636 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:30.634 7fee33fff700 1 --2- 172.21.2.102:0/2846247133 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fee140186c0 0x7fee1401abc0 crc :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:05:30.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:30.665 7fee3a245700 1 -- 172.21.2.102:0/2846247133 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee34105ae0 con 0x7fee34189d00 2021-02-10T11:05:30.667 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:30.668 7fee30ff9700 1 -- 172.21.2.102:0/2846247133 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fee28030dc0 con 0x7fee34189d00 2021-02-10T11:05:30.796 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:30.797 7fee3a245700 1 -- 172.21.2.102:0/2846247133 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7fee340f0b50 con 0x7fee34189d00 2021-02-10T11:05:30.796 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:30.797 7fee30ff9700 1 -- 172.21.2.102:0/2846247133 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v30) v1 ==== 100+0+10 (secure 0 0 0) 0x7fee2800bd80 con 0x7fee34189d00 2021-02-10T11:05:30.796 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:05:30.798 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:30.799 7fee3a245700 1 -- 172.21.2.102:0/2846247133 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fee140186c0 msgr2=0x7fee1401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:30.798 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:30.799 7fee3a245700 1 --2- 172.21.2.102:0/2846247133 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fee140186c0 0x7fee1401abc0 crc :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:30.798 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:30.799 7fee3a245700 1 -- 172.21.2.102:0/2846247133 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fee34189d00 msgr2=0x7fee3418a100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:30.798 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:30.799 7fee3a245700 1 --2- 172.21.2.102:0/2846247133 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fee34189d00 0x7fee3418a100 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 rx=0x7fee28006f40 tx=0x7fee2800b990).stop 2021-02-10T11:05:30.798 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:30.799 7fee3a245700 1 -- 172.21.2.102:0/2846247133 shutdown_connections 2021-02-10T11:05:30.798 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:30.799 7fee3a245700 1 --2- 172.21.2.102:0/2846247133 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fee140186c0 0x7fee1401abc0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:30.798 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:30.799 7fee3a245700 1 --2- 172.21.2.102:0/2846247133 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fee3418a720 0x7fee340b7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:30.799 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:30.799 7fee3a245700 1 --2- 172.21.2.102:0/2846247133 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fee34189d00 0x7fee3418a100 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:30.799 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:30.799 7fee3a245700 1 --2- 172.21.2.102:0/2846247133 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fee340a0420 0x7fee340a0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:30.799 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:30.799 7fee3a245700 1 -- 172.21.2.102:0/2846247133 >> 172.21.2.102:0/2846247133 conn(0x7fee34094e10 msgr2=0x7fee340a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:05:30.799 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:30.800 7fee3a245700 1 -- 172.21.2.102:0/2846247133 shutdown_connections 2021-02-10T11:05:30.799 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:30.800 7fee3a245700 1 -- 172.21.2.102:0/2846247133 wait complete. 2021-02-10T11:05:30.806 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-02-10T11:05:30.807 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:05:31.233 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:31.217 7f79bfeb2700 1 Processor -- start 2021-02-10T11:05:31.234 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:31.217 7f79bfeb2700 1 -- start start 2021-02-10T11:05:31.234 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:31.217 7f79bfeb2700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f79b807e6a0 0x7f79b80d1790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:31.234 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:31.217 7f79bfeb2700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79b807ed80 con 0x7f79b80e5470 2021-02-10T11:05:31.234 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:31.217 7f79bfeb2700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79b807eb10 con 0x7f79b80e5070 2021-02-10T11:05:31.234 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:31.217 7f79bfeb2700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79b81b4080 con 0x7f79b807e6a0 2021-02-10T11:05:31.235 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:31.218 7f79bd4cd700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f79b80e5470 0x7f79b807f300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:43046/0 (socket says 172.21.2.102:43046) 2021-02-10T11:05:31.235 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:31.218 7f79bd4cd700 1 -- 172.21.2.102:0/893165756 learned_addr learned my addr 172.21.2.102:0/893165756 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:05:31.235 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:31.218 7f79be4cf700 1 --2- 172.21.2.102:0/893165756 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f79b807e6a0 0x7f79b80d1790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:31.235 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:31.218 7f79bcccc700 1 -- 172.21.2.102:0/893165756 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2088653748 0 0) 0x7f79b40010b0 con 0x7f79b80e5470 2021-02-10T11:05:31.235 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:31.218 7f79bcccc700 1 -- 172.21.2.102:0/893165756 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f79a4002fe0 con 0x7f79b80e5470 2021-02-10T11:05:31.235 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:31.218 7f79bcccc700 1 -- 172.21.2.102:0/893165756 <== mon.1 v1:172.21.2.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1869920629 0 0) 0x7f79ac000ff0 con 0x7f79b80e5070 2021-02-10T11:05:31.236 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:31.218 7f79bcccc700 1 -- 172.21.2.102:0/893165756 --> v1:172.21.2.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f79a4004370 con 0x7f79b80e5070 2021-02-10T11:05:31.236 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:31.218 7f79be4cf700 1 -- 172.21.2.102:0/893165756 >> v1:172.21.2.113:6789/0 conn(0x7f79b80e5070 legacy=0x7f79b81b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:31.236 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:31.218 7f79be4cf700 1 -- 172.21.2.102:0/893165756 >> v1:172.21.2.102:6789/0 conn(0x7f79b80e5470 legacy=0x7f79b807f300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:31.236 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:31.218 7f79be4cf700 1 -- 172.21.2.102:0/893165756 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79b80d1db0 con 0x7f79b807e6a0 2021-02-10T11:05:31.236 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:31.218 7f79be4cf700 1 --2- 172.21.2.102:0/893165756 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f79b807e6a0 0x7f79b80d1790 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f79b0006e50 tx=0x7f79b0009e70).ready entity=mon.2 client_cookie=92a0a09a149f9c76 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:05:31.236 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:31.219 7f79bcccc700 1 -- 172.21.2.102:0/893165756 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f79b000b510 con 0x7f79b807e6a0 2021-02-10T11:05:31.236 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:31.219 7f79bcccc700 1 -- 172.21.2.102:0/893165756 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79b0006790 con 0x7f79b807e6a0 2021-02-10T11:05:31.236 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:31.219 7f79bcccc700 1 -- 172.21.2.102:0/893165756 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f79b000b940 con 0x7f79b807e6a0 2021-02-10T11:05:31.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:31.219 7f79bfeb2700 1 -- 172.21.2.102:0/893165756 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f79b807e6a0 msgr2=0x7f79b80d1790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:31.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:31.219 7f79bfeb2700 1 --2- 172.21.2.102:0/893165756 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f79b807e6a0 0x7f79b80d1790 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f79b0006e50 tx=0x7f79b0009e70).stop 2021-02-10T11:05:31.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:31.220 7f79bfeb2700 1 -- 172.21.2.102:0/893165756 shutdown_connections 2021-02-10T11:05:31.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:31.220 7f79bfeb2700 1 --2- 172.21.2.102:0/893165756 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f79b807e6a0 0x7f79b80d1790 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:31.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:31.220 7f79bfeb2700 1 -- 172.21.2.102:0/893165756 >> 172.21.2.102:0/893165756 conn(0x7f79b80ce690 msgr2=0x7f79b80e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:05:31.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:31.220 7f79bfeb2700 1 -- 172.21.2.102:0/893165756 shutdown_connections 2021-02-10T11:05:31.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:31.220 7f79bfeb2700 1 -- 172.21.2.102:0/893165756 wait complete. 2021-02-10T11:05:31.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:31.235 7f79bfeb2700 1 Processor -- start 2021-02-10T11:05:31.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:31.235 7f79bfeb2700 1 -- start start 2021-02-10T11:05:31.239 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:31.235 7f79bfeb2700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f79b808d400 0x7f79b8081230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:31.239 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:31.235 7f79bfeb2700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f79b8081850 0x7f79b80acf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:31.239 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:31.235 7f79bfeb2700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f79b8081c50 0x7f79b80ad5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:31.239 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:31.235 7f79bfeb2700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79b8082110 con 0x7f79b8081c50 2021-02-10T11:05:31.239 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:31.235 7f79bfeb2700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79b808d800 con 0x7f79b808d400 2021-02-10T11:05:31.239 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:31.235 7f79bfeb2700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79b80adbc0 con 0x7f79b8081850 2021-02-10T11:05:31.239 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:31.235 7f79be4cf700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f79b8081c50 0x7f79b80ad5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:31.239 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:31.235 7f79be4cf700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f79b8081c50 0x7f79b80ad5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:50830/0 (socket says 172.21.2.102:50830) 2021-02-10T11:05:31.239 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:31.235 7f79be4cf700 1 -- 172.21.2.102:0/1125908268 learned_addr learned my addr 172.21.2.102:0/1125908268 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:05:31.239 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:31.235 7f79bdcce700 1 --2- 172.21.2.102:0/1125908268 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f79b808d400 0x7f79b8081230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:31.240 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:31.236 7f79bd4cd700 1 --2- 172.21.2.102:0/1125908268 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f79b8081850 0x7f79b80acf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:31.240 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:31.236 7f79be4cf700 1 -- 172.21.2.102:0/1125908268 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f79b8081850 msgr2=0x7f79b80acf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:31.240 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:31.236 7f79be4cf700 1 --2- 172.21.2.102:0/1125908268 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f79b8081850 0x7f79b80acf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:31.240 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:31.236 7f79be4cf700 1 -- 172.21.2.102:0/1125908268 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f79b808d400 msgr2=0x7f79b8081230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:31.240 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:31.236 7f79be4cf700 1 --2- 172.21.2.102:0/1125908268 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f79b808d400 0x7f79b8081230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:31.240 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:31.236 7f79be4cf700 1 -- 172.21.2.102:0/1125908268 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79b8187fc0 con 0x7f79b8081c50 2021-02-10T11:05:31.240 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:31.236 7f79bd4cd700 1 --2- 172.21.2.102:0/1125908268 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f79b8081850 0x7f79b80acf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:05:31.240 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:31.236 7f79be4cf700 1 --2- 172.21.2.102:0/1125908268 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f79b8081c50 0x7f79b80ad5a0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 rx=0x7f79b0006c50 tx=0x7f79b000bd20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:05:31.240 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:31.236 7f79aa7fc700 1 -- 172.21.2.102:0/1125908268 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f79b00093d0 con 0x7f79b8081c50 2021-02-10T11:05:31.240 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:31.236 7f79aa7fc700 1 -- 172.21.2.102:0/1125908268 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79b000be60 con 0x7f79b8081c50 2021-02-10T11:05:31.240 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:31.236 7f79aa7fc700 1 -- 172.21.2.102:0/1125908268 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f79b000c570 con 0x7f79b8081c50 2021-02-10T11:05:31.240 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:31.236 7f79bfeb2700 1 -- 172.21.2.102:0/1125908268 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79b8187fc0 con 0x7f79b8081c50 2021-02-10T11:05:31.241 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:31.236 7f79bfeb2700 1 -- 172.21.2.102:0/1125908268 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79b8187fc0 con 0x7f79b8081c50 2021-02-10T11:05:31.241 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:31.237 7f79aa7fc700 1 -- 172.21.2.102:0/1125908268 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f79b0002ef0 con 0x7f79b8081c50 2021-02-10T11:05:31.241 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:31.237 7f79aa7fc700 1 --2- 172.21.2.102:0/1125908268 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f79a00186d0 0x7f79a001abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:31.241 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:31.237 7f79aa7fc700 1 -- 172.21.2.102:0/1125908268 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f79b0032990 con 0x7f79b8081c50 2021-02-10T11:05:31.241 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:31.237 7f79bdcce700 1 --2- 172.21.2.102:0/1125908268 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f79a00186d0 0x7f79a001abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:31.241 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:31.237 7f79bdcce700 1 --2- 172.21.2.102:0/1125908268 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f79a00186d0 0x7f79a001abd0 crc :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:05:31.265 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:31.267 7f79bfeb2700 1 -- 172.21.2.102:0/1125908268 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79b807ed80 con 0x7f79b8081c50 2021-02-10T11:05:31.268 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:31.269 7f79aa7fc700 1 -- 172.21.2.102:0/1125908268 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f79b0000df0 con 0x7f79b8081c50 2021-02-10T11:05:31.396 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:31.397 7f79bfeb2700 1 -- 172.21.2.102:0/1125908268 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f79b807ed80 con 0x7f79b8081c50 2021-02-10T11:05:31.396 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:31.398 7f79aa7fc700 1 -- 172.21.2.102:0/1125908268 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) v1 ==== 94+0+2101 (secure 0 0 0) 0x7f79b0002ef0 con 0x7f79b8081c50 2021-02-10T11:05:31.397 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:05:31.397 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":14,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4845,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.102:6835/3747080008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6834","nonce":3747080008},{"type":"v1","addr":"172.21.2.102:6835","nonce":3747080008}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":10}],"filesystems":[{"mdsmap":{"epoch":14,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:05:12.567145","modified":"2021-02-10 11:05:14.922542","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4510},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4510":{"gid":4510,"name":"b","rank":0,"incarnation":11,"state":"up:active","state_seq":4,"addr":"172.21.2.113:6833/3898434901","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6832","nonce":3898434901},{"type":"v1","addr":"172.21.2.113:6833","nonce":3898434901}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":2}]} 2021-02-10T11:05:31.397 INFO:teuthology.orchestra.run.gibba002.stderr:dumped fsmap epoch 14 2021-02-10T11:05:31.398 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:31.399 7f79bfeb2700 1 -- 172.21.2.102:0/1125908268 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f79a00186d0 msgr2=0x7f79a001abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:31.398 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:31.399 7f79bfeb2700 1 --2- 172.21.2.102:0/1125908268 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f79a00186d0 0x7f79a001abd0 crc :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:31.398 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:31.400 7f79bfeb2700 1 -- 172.21.2.102:0/1125908268 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f79b8081c50 msgr2=0x7f79b80ad5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:31.398 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:31.400 7f79bfeb2700 1 --2- 172.21.2.102:0/1125908268 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f79b8081c50 0x7f79b80ad5a0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 rx=0x7f79b0006c50 tx=0x7f79b000bd20).stop 2021-02-10T11:05:31.399 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:31.400 7f79bfeb2700 1 -- 172.21.2.102:0/1125908268 shutdown_connections 2021-02-10T11:05:31.399 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:31.400 7f79bfeb2700 1 --2- 172.21.2.102:0/1125908268 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f79b8081c50 0x7f79b80ad5a0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:31.399 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:31.400 7f79bfeb2700 1 --2- 172.21.2.102:0/1125908268 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f79b8081850 0x7f79b80acf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:31.399 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:31.400 7f79bfeb2700 1 --2- 172.21.2.102:0/1125908268 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f79a00186d0 0x7f79a001abd0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:31.399 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:31.400 7f79bfeb2700 1 --2- 172.21.2.102:0/1125908268 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f79b808d400 0x7f79b8081230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:31.399 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:31.400 7f79bfeb2700 1 -- 172.21.2.102:0/1125908268 >> 172.21.2.102:0/1125908268 conn(0x7f79b81ac620 msgr2=0x7f79b80896e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:05:31.399 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:31.400 7f79bfeb2700 1 -- 172.21.2.102:0/1125908268 shutdown_connections 2021-02-10T11:05:31.399 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:31.400 7f79bfeb2700 1 -- 172.21.2.102:0/1125908268 wait complete. 2021-02-10T11:05:31.608 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 14, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:05:12.567145', 'modified': '2021-02-10 11:05:14.922542', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4510}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4510': {'gid': 4510, 'name': 'b', 'rank': 0, 'incarnation': 11, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.2.113:6833/3898434901', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.113:6832', 'nonce': 3898434901}, {'type': 'v1', 'addr': '172.21.2.113:6833', 'nonce': 3898434901}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [5], 'metadata_pool': 4, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:05:31.608 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:05:31.609 DEBUG:teuthology.orchestra.run.gibba013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:05:32.006 INFO:teuthology.orchestra.run.gibba013.stdout:{ 2021-02-10T11:05:32.006 INFO:teuthology.orchestra.run.gibba013.stdout: "cluster_fsid": "68db30ea-dbce-4aef-a6df-f01c6fea0513", 2021-02-10T11:05:32.006 INFO:teuthology.orchestra.run.gibba013.stdout: "whoami": 0, 2021-02-10T11:05:32.006 INFO:teuthology.orchestra.run.gibba013.stdout: "id": 4510, 2021-02-10T11:05:32.006 INFO:teuthology.orchestra.run.gibba013.stdout: "want_state": "up:active", 2021-02-10T11:05:32.006 INFO:teuthology.orchestra.run.gibba013.stdout: "state": "up:active", 2021-02-10T11:05:32.007 INFO:teuthology.orchestra.run.gibba013.stdout: "rank_uptime": 19.439070301000001, 2021-02-10T11:05:32.007 INFO:teuthology.orchestra.run.gibba013.stdout: "mdsmap_epoch": 14, 2021-02-10T11:05:32.007 INFO:teuthology.orchestra.run.gibba013.stdout: "osdmap_epoch": 30, 2021-02-10T11:05:32.007 INFO:teuthology.orchestra.run.gibba013.stdout: "osdmap_epoch_barrier": 30, 2021-02-10T11:05:32.007 INFO:teuthology.orchestra.run.gibba013.stdout: "uptime": 28.722848527 2021-02-10T11:05:32.007 INFO:teuthology.orchestra.run.gibba013.stdout:} 2021-02-10T11:05:32.010 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "68db30ea-dbce-4aef-a6df-f01c6fea0513", "whoami": 0, "id": 4510, "want_state": "up:active", "state": "up:active", "rank_uptime": 19.439070301000001, "mdsmap_epoch": 14, "osdmap_epoch": 30, "osdmap_epoch_barrier": 30, "uptime": 28.722848527 } 2021-02-10T11:05:32.010 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2021-02-10T11:05:32.011 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2021-02-10T11:05:32.011 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.1 at ubuntu@gibba002.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.1... 2021-02-10T11:05:32.011 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:05:32.011 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec mkdir -p /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:05:32.345 DEBUG:teuthology.orchestra.run.gibba002:> sudo modprobe fuse 2021-02-10T11:05:32.689 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:05:32.689 DEBUG:teuthology.orchestra.run.gibba002:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:05:33.031 INFO:teuthology.orchestra.run.gibba002.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:05:33.031 INFO:teuthology.orchestra.run.gibba002.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:05:33.032 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:05:33.032 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:05:33.033 DEBUG:teuthology.orchestra.run.gibba002:> ls /sys/fs/fuse/connections 2021-02-10T11:05:33.367 INFO:teuthology.orchestra.run.gibba002.stdout:43 2021-02-10T11:05:33.368 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [43] 2021-02-10T11:05:33.368 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.1 /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:05:33.371 DEBUG:teuthology.orchestra.run.gibba002:> sudo modprobe fuse 2021-02-10T11:05:33.715 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:05:33.715 DEBUG:teuthology.orchestra.run.gibba002:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:05:33.781 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba002.stderr:ceph-fuse[45554]: starting ceph client 2021-02-10T11:05:33.781 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba002.stderr:2021-02-10 11:05:33.783 7f853847ef80 -1 init, newargv = 0x55bfbb0e3590 newargc=9 2021-02-10T11:05:33.786 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba002.stderr:ceph-fuse[45554]: starting fuse 2021-02-10T11:05:34.057 INFO:teuthology.orchestra.run.gibba002.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:05:34.057 INFO:teuthology.orchestra.run.gibba002.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:05:34.058 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:05:34.058 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:05:34.058 DEBUG:teuthology.orchestra.run.gibba002:> ls /sys/fs/fuse/connections 2021-02-10T11:05:34.392 INFO:teuthology.orchestra.run.gibba002.stdout:43 2021-02-10T11:05:34.392 INFO:teuthology.orchestra.run.gibba002.stdout:44 2021-02-10T11:05:34.392 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43, 44] 2021-02-10T11:05:34.392 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:05:34.392 DEBUG:teuthology.orchestra.run.gibba002:> sudo python3 -c ' 2021-02-10T11:05:34.393 DEBUG:teuthology.orchestra.run.gibba002:> import glob 2021-02-10T11:05:34.393 DEBUG:teuthology.orchestra.run.gibba002:> import re 2021-02-10T11:05:34.393 DEBUG:teuthology.orchestra.run.gibba002:> import os 2021-02-10T11:05:34.393 DEBUG:teuthology.orchestra.run.gibba002:> import subprocess 2021-02-10T11:05:34.393 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:05:34.393 DEBUG:teuthology.orchestra.run.gibba002:> def find_socket(client_name): 2021-02-10T11:05:34.393 DEBUG:teuthology.orchestra.run.gibba002:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2021-02-10T11:05:34.393 DEBUG:teuthology.orchestra.run.gibba002:> files = glob.glob(asok_path) 2021-02-10T11:05:34.393 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:05:34.393 DEBUG:teuthology.orchestra.run.gibba002:> # Given a non-glob path, it better be there 2021-02-10T11:05:34.393 DEBUG:teuthology.orchestra.run.gibba002:> if "*" not in asok_path: 2021-02-10T11:05:34.393 DEBUG:teuthology.orchestra.run.gibba002:> assert(len(files) == 1) 2021-02-10T11:05:34.393 DEBUG:teuthology.orchestra.run.gibba002:> return files[0] 2021-02-10T11:05:34.393 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:05:34.393 DEBUG:teuthology.orchestra.run.gibba002:> for f in files: 2021-02-10T11:05:34.394 DEBUG:teuthology.orchestra.run.gibba002:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2021-02-10T11:05:34.394 DEBUG:teuthology.orchestra.run.gibba002:> if os.path.exists("/proc/{0}".format(pid)): 2021-02-10T11:05:34.394 DEBUG:teuthology.orchestra.run.gibba002:> return f 2021-02-10T11:05:34.394 DEBUG:teuthology.orchestra.run.gibba002:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2021-02-10T11:05:34.394 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:05:34.394 DEBUG:teuthology.orchestra.run.gibba002:> print(find_socket("client.1")) 2021-02-10T11:05:34.394 DEBUG:teuthology.orchestra.run.gibba002:> ' 2021-02-10T11:05:34.755 INFO:teuthology.orchestra.run.gibba002.stdout:/var/run/ceph/ceph-client.1.45554.asok 2021-02-10T11:05:34.758 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.45554.asok 2021-02-10T11:05:34.758 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:05:34.758 DEBUG:teuthology.orchestra.run.gibba002:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.45554.asok status 2021-02-10T11:05:35.146 INFO:teuthology.orchestra.run.gibba002.stdout:{ 2021-02-10T11:05:35.147 INFO:teuthology.orchestra.run.gibba002.stdout: "metadata": { 2021-02-10T11:05:35.147 INFO:teuthology.orchestra.run.gibba002.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:05:35.147 INFO:teuthology.orchestra.run.gibba002.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:05:35.147 INFO:teuthology.orchestra.run.gibba002.stdout: "entity_id": "1", 2021-02-10T11:05:35.147 INFO:teuthology.orchestra.run.gibba002.stdout: "hostname": "gibba002", 2021-02-10T11:05:35.147 INFO:teuthology.orchestra.run.gibba002.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2021-02-10T11:05:35.147 INFO:teuthology.orchestra.run.gibba002.stdout: "pid": "45554", 2021-02-10T11:05:35.147 INFO:teuthology.orchestra.run.gibba002.stdout: "root": "/" 2021-02-10T11:05:35.147 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:05:35.148 INFO:teuthology.orchestra.run.gibba002.stdout: "dentry_count": 0, 2021-02-10T11:05:35.148 INFO:teuthology.orchestra.run.gibba002.stdout: "dentry_pinned_count": 0, 2021-02-10T11:05:35.148 INFO:teuthology.orchestra.run.gibba002.stdout: "id": 5094, 2021-02-10T11:05:35.148 INFO:teuthology.orchestra.run.gibba002.stdout: "inst": { 2021-02-10T11:05:35.148 INFO:teuthology.orchestra.run.gibba002.stdout: "name": { 2021-02-10T11:05:35.148 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "client", 2021-02-10T11:05:35.148 INFO:teuthology.orchestra.run.gibba002.stdout: "num": 5094 2021-02-10T11:05:35.148 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:05:35.148 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": { 2021-02-10T11:05:35.148 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:05:35.148 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:0", 2021-02-10T11:05:35.148 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 2359848887 2021-02-10T11:05:35.149 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:05:35.149 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:05:35.149 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": { 2021-02-10T11:05:35.149 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:05:35.149 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:0", 2021-02-10T11:05:35.149 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 2359848887 2021-02-10T11:05:35.149 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:05:35.149 INFO:teuthology.orchestra.run.gibba002.stdout: "inst_str": "client.5094 172.21.2.102:0/2359848887", 2021-02-10T11:05:35.149 INFO:teuthology.orchestra.run.gibba002.stdout: "addr_str": "172.21.2.102:0/2359848887", 2021-02-10T11:05:35.149 INFO:teuthology.orchestra.run.gibba002.stdout: "inode_count": 1, 2021-02-10T11:05:35.149 INFO:teuthology.orchestra.run.gibba002.stdout: "mds_epoch": 14, 2021-02-10T11:05:35.149 INFO:teuthology.orchestra.run.gibba002.stdout: "osd_epoch": 30, 2021-02-10T11:05:35.150 INFO:teuthology.orchestra.run.gibba002.stdout: "osd_epoch_barrier": 0, 2021-02-10T11:05:35.150 INFO:teuthology.orchestra.run.gibba002.stdout: "blacklisted": false 2021-02-10T11:05:35.150 INFO:teuthology.orchestra.run.gibba002.stdout:} 2021-02-10T11:05:35.151 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:05:35.151 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:05:35.151 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:05:35.486 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:05:35.487 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2021-02-10T11:05:35.487 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:05:35.487 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:05:35.833 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2021-02-10T11:05:36.261 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:36.246 7fee80bd1700 1 Processor -- start 2021-02-10T11:05:36.262 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:36.246 7fee80bd1700 1 -- start start 2021-02-10T11:05:36.262 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:36.247 7fee80bd1700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fee7c0d1860 0x7fee7c1b04b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:36.262 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:36.247 7fee80bd1700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fee7c1b0ad0 0x7fee7c108930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:36.262 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:36.247 7fee80bd1700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fee7c0d1c60 0x7fee7c108f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:36.262 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:36.247 7fee80bd1700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee7c0fe1b0 con 0x7fee7c0d1860 2021-02-10T11:05:36.262 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:36.247 7fee80bd1700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee7c109a50 con 0x7fee7c0d1c60 2021-02-10T11:05:36.262 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:36.247 7fee80bd1700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee7c088020 con 0x7fee7c1b0ad0 2021-02-10T11:05:36.262 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:36.247 7fee7a85e700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fee7c0d1860 0x7fee7c1b04b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:36.262 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:36.247 7fee7a85e700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fee7c0d1860 0x7fee7c1b04b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:50848/0 (socket says 172.21.2.102:50848) 2021-02-10T11:05:36.262 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:36.247 7fee7a85e700 1 -- 172.21.2.102:0/344686167 learned_addr learned my addr 172.21.2.102:0/344686167 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:05:36.262 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:36.247 7fee7a85e700 1 -- 172.21.2.102:0/344686167 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fee7c1b0ad0 msgr2=0x7fee7c108930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:05:36.263 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:36.247 7fee7a85e700 1 --2- 172.21.2.102:0/344686167 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fee7c1b0ad0 0x7fee7c108930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:05:36.263 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:36.247 7fee7a85e700 1 -- 172.21.2.102:0/344686167 >> v2:172.21.2.113:3300/0 conn(0x7fee7c0d1c60 msgr2=0x7fee7c108f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:05:36.263 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:36.247 7fee7a85e700 1 --2- 172.21.2.102:0/344686167 >> v2:172.21.2.113:3300/0 conn(0x7fee7c0d1c60 0x7fee7c108f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:05:36.263 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:36.247 7fee7a85e700 1 -- 172.21.2.102:0/344686167 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee7c109570 con 0x7fee7c0d1860 2021-02-10T11:05:36.263 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:36.247 7fee7a85e700 1 --2- 172.21.2.102:0/344686167 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fee7c0d1860 0x7fee7c1b04b0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 rx=0x7fee64006700 tx=0x7fee64009b80).ready entity=mon.0 client_cookie=c9e5e285ff616c52 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:05:36.263 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:36.248 7fee7985c700 1 -- 172.21.2.102:0/344686167 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fee6400c0c0 con 0x7fee7c0d1860 2021-02-10T11:05:36.263 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:36.248 7fee7985c700 1 -- 172.21.2.102:0/344686167 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee6400c340 con 0x7fee7c0d1860 2021-02-10T11:05:36.263 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:36.248 7fee7985c700 1 -- 172.21.2.102:0/344686167 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fee6400c860 con 0x7fee7c0d1860 2021-02-10T11:05:36.263 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:36.248 7fee80bd1700 1 -- 172.21.2.102:0/344686167 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fee7c0d1860 msgr2=0x7fee7c1b04b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:36.263 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:36.248 7fee80bd1700 1 --2- 172.21.2.102:0/344686167 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fee7c0d1860 0x7fee7c1b04b0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 rx=0x7fee64006700 tx=0x7fee64009b80).stop 2021-02-10T11:05:36.263 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:36.248 7fee80bd1700 1 -- 172.21.2.102:0/344686167 shutdown_connections 2021-02-10T11:05:36.263 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:36.249 7fee80bd1700 1 --2- 172.21.2.102:0/344686167 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fee7c1b0ad0 0x7fee7c108930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:05:36.264 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:36.249 7fee80bd1700 1 --2- 172.21.2.102:0/344686167 >> v2:172.21.2.113:3300/0 conn(0x7fee7c0d1c60 0x7fee7c108f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:05:36.264 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:36.249 7fee80bd1700 1 --2- 172.21.2.102:0/344686167 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fee7c0d1860 0x7fee7c1b04b0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:36.264 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:36.249 7fee80bd1700 1 -- 172.21.2.102:0/344686167 >> 172.21.2.102:0/344686167 conn(0x7fee7c0ce410 msgr2=0x7fee7c0d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:05:36.264 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:36.249 7fee80bd1700 1 -- 172.21.2.102:0/344686167 shutdown_connections 2021-02-10T11:05:36.264 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:36.249 7fee80bd1700 1 -- 172.21.2.102:0/344686167 wait complete. 2021-02-10T11:05:36.266 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:36.263 7fee80bd1700 1 Processor -- start 2021-02-10T11:05:36.266 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:36.263 7fee80bd1700 1 -- start start 2021-02-10T11:05:36.266 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:36.264 7fee80bd1700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fee7c093fb0 0x7fee7c1a5660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:36.266 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:36.264 7fee80bd1700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fee7c1a1c40 0x7fee7c1a2040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:36.266 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:36.264 7fee80bd1700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fee7c19bbc0 0x7fee7c19bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:36.266 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:36.264 7fee80bd1700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee7c1a2730 con 0x7fee7c19bbc0 2021-02-10T11:05:36.267 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:36.264 7fee80bd1700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee7c0a76c0 con 0x7fee7c1a1c40 2021-02-10T11:05:36.267 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:36.264 7fee80bd1700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee7c1a5c80 con 0x7fee7c093fb0 2021-02-10T11:05:36.267 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:36.264 7fee7b05f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fee7c19bbc0 0x7fee7c19bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:36.267 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:36.264 7fee7b05f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fee7c19bbc0 0x7fee7c19bfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:50856/0 (socket says 172.21.2.102:50856) 2021-02-10T11:05:36.267 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:36.264 7fee7b05f700 1 -- 172.21.2.102:0/964943520 learned_addr learned my addr 172.21.2.102:0/964943520 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:05:36.267 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:36.264 7fee7b05f700 1 -- 172.21.2.102:0/964943520 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fee7c093fb0 msgr2=0x7fee7c1a5660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:36.267 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:36.264 7fee7b05f700 1 --2- 172.21.2.102:0/964943520 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fee7c093fb0 0x7fee7c1a5660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:05:36.267 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:36.264 7fee7b05f700 1 -- 172.21.2.102:0/964943520 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fee7c1a1c40 msgr2=0x7fee7c1a2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:36.267 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:36.264 7fee7b05f700 1 --2- 172.21.2.102:0/964943520 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fee7c1a1c40 0x7fee7c1a2040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:05:36.267 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:36.264 7fee7b05f700 1 -- 172.21.2.102:0/964943520 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee7c19c5e0 con 0x7fee7c19bbc0 2021-02-10T11:05:36.267 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:36.265 7fee7b05f700 1 --2- 172.21.2.102:0/964943520 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fee7c19bbc0 0x7fee7c19bfc0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 rx=0x7fee7400b8c0 tx=0x7fee7400bc60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:05:36.267 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:36.265 7fee737fe700 1 -- 172.21.2.102:0/964943520 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fee74005870 con 0x7fee7c19bbc0 2021-02-10T11:05:36.267 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:36.265 7fee737fe700 1 -- 172.21.2.102:0/964943520 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee74005b30 con 0x7fee7c19bbc0 2021-02-10T11:05:36.267 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:36.265 7fee737fe700 1 -- 172.21.2.102:0/964943520 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fee74006030 con 0x7fee7c19bbc0 2021-02-10T11:05:36.268 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:36.265 7fee80bd1700 1 -- 172.21.2.102:0/964943520 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee7c19c5e0 con 0x7fee7c19bbc0 2021-02-10T11:05:36.268 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:36.265 7fee80bd1700 1 -- 172.21.2.102:0/964943520 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee7c19cb70 con 0x7fee7c19bbc0 2021-02-10T11:05:36.268 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:36.266 7fee737fe700 1 -- 172.21.2.102:0/964943520 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fee74005d10 con 0x7fee7c19bbc0 2021-02-10T11:05:36.268 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:36.266 7fee737fe700 1 --2- 172.21.2.102:0/964943520 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fee600186c0 0x7fee6001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:36.268 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:36.266 7fee737fe700 1 -- 172.21.2.102:0/964943520 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fee74032770 con 0x7fee7c19bbc0 2021-02-10T11:05:36.268 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:36.266 7fee7a85e700 1 --2- 172.21.2.102:0/964943520 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fee600186c0 0x7fee6001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:36.268 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:36.266 7fee7a85e700 1 --2- 172.21.2.102:0/964943520 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fee600186c0 0x7fee6001abc0 crc :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:05:36.295 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:36.297 7fee80bd1700 1 -- 172.21.2.102:0/964943520 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee7c0fe1b0 con 0x7fee7c19bbc0 2021-02-10T11:05:36.298 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:36.299 7fee737fe700 1 -- 172.21.2.102:0/964943520 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fee7400bda0 con 0x7fee7c19bbc0 2021-02-10T11:05:36.434 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:36.435 7fee80bd1700 1 -- 172.21.2.102:0/964943520 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fee7c0fe1b0 con 0x7fee7c19bbc0 2021-02-10T11:05:36.755 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:36.756 7fee737fe700 1 -- 172.21.2.102:0/964943520 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v15) v1 ==== 136+0+0 (secure 0 0 0) 0x7fee74006f20 con 0x7fee7c19bbc0 2021-02-10T11:05:36.801 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:36.803 7fee80bd1700 1 -- 172.21.2.102:0/964943520 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fee600186c0 msgr2=0x7fee6001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:36.802 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:36.803 7fee80bd1700 1 --2- 172.21.2.102:0/964943520 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fee600186c0 0x7fee6001abc0 crc :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:36.802 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:36.803 7fee80bd1700 1 -- 172.21.2.102:0/964943520 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fee7c19bbc0 msgr2=0x7fee7c19bfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:36.802 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:36.803 7fee80bd1700 1 --2- 172.21.2.102:0/964943520 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fee7c19bbc0 0x7fee7c19bfc0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 rx=0x7fee7400b8c0 tx=0x7fee7400bc60).stop 2021-02-10T11:05:36.802 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:36.803 7fee80bd1700 1 -- 172.21.2.102:0/964943520 shutdown_connections 2021-02-10T11:05:36.802 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:36.803 7fee80bd1700 1 --2- 172.21.2.102:0/964943520 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fee7c19bbc0 0x7fee7c19bfc0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:36.802 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:36.803 7fee80bd1700 1 --2- 172.21.2.102:0/964943520 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fee600186c0 0x7fee6001abc0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:36.802 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:36.803 7fee80bd1700 1 --2- 172.21.2.102:0/964943520 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fee7c1a1c40 0x7fee7c1a2040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:05:36.802 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:36.803 7fee80bd1700 1 --2- 172.21.2.102:0/964943520 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fee7c093fb0 0x7fee7c1a5660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:05:36.802 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:36.803 7fee80bd1700 1 -- 172.21.2.102:0/964943520 >> 172.21.2.102:0/964943520 conn(0x7fee7c094ea0 msgr2=0x7fee7c19ace0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:05:36.802 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:36.803 7fee80bd1700 1 -- 172.21.2.102:0/964943520 shutdown_connections 2021-02-10T11:05:36.802 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:36.803 7fee80bd1700 1 -- 172.21.2.102:0/964943520 wait complete. 2021-02-10T11:05:36.809 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs volume ls 2021-02-10T11:05:37.236 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:37.221 7f8b553f6700 1 Processor -- start 2021-02-10T11:05:37.236 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:37.221 7f8b553f6700 1 -- start start 2021-02-10T11:05:37.236 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:37.222 7f8b553f6700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8b500e5070 0x7f8b501b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:37.236 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:37.222 7f8b553f6700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f8b500e5470 0x7f8b5007d690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:37.236 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:37.222 7f8b553f6700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b5007ed80 con 0x7f8b501b3e40 2021-02-10T11:05:37.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:37.222 7f8b553f6700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b50080310 con 0x7f8b500e5470 2021-02-10T11:05:37.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:37.222 7f8b553f6700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b50080500 con 0x7f8b500e5070 2021-02-10T11:05:37.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:37.222 7f8b4e87c700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f8b500e5470 0x7f8b5007d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:37.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:37.222 7f8b4e87c700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f8b500e5470 0x7f8b5007d690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:50860/0 (socket says 172.21.2.102:50860) 2021-02-10T11:05:37.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:37.222 7f8b4e87c700 1 -- 172.21.2.102:0/110330289 learned_addr learned my addr 172.21.2.102:0/110330289 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:05:37.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:37.222 7f8b4e87c700 1 -- 172.21.2.102:0/110330289 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8b500e5070 msgr2=0x7f8b501b3820 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:05:37.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:37.222 7f8b4e87c700 1 --2- 172.21.2.102:0/110330289 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8b500e5070 0x7f8b501b3820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:05:37.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:37.222 7f8b4e87c700 1 -- 172.21.2.102:0/110330289 >> v1:172.21.2.113:6789/0 conn(0x7f8b501b3e40 legacy=0x7f8b5007dcb0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:05:37.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:37.222 7f8b4e87c700 1 -- 172.21.2.102:0/110330289 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b500806f0 con 0x7f8b500e5470 2021-02-10T11:05:37.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:37.222 7f8b4e87c700 1 --2- 172.21.2.102:0/110330289 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8b500e5470 0x7f8b5007d690 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 rx=0x7f8b40006700 tx=0x7f8b40009b00).ready entity=mon.0 client_cookie=90777336e27ed2a1 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:05:37.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:37.223 7f8b4e07b700 1 -- 172.21.2.102:0/110330289 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8b4000c0c0 con 0x7f8b500e5470 2021-02-10T11:05:37.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:37.223 7f8b4e07b700 1 -- 172.21.2.102:0/110330289 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b4000c400 con 0x7f8b500e5470 2021-02-10T11:05:37.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:37.223 7f8b4e07b700 1 -- 172.21.2.102:0/110330289 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8b4000c920 con 0x7f8b500e5470 2021-02-10T11:05:37.238 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:37.223 7f8b553f6700 1 -- 172.21.2.102:0/110330289 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8b500e5470 msgr2=0x7f8b5007d690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:37.238 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:37.223 7f8b553f6700 1 --2- 172.21.2.102:0/110330289 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8b500e5470 0x7f8b5007d690 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 rx=0x7f8b40006700 tx=0x7f8b40009b00).stop 2021-02-10T11:05:37.238 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:37.223 7f8b553f6700 1 -- 172.21.2.102:0/110330289 shutdown_connections 2021-02-10T11:05:37.238 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:37.223 7f8b553f6700 1 --2- 172.21.2.102:0/110330289 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8b500e5470 0x7f8b5007d690 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:37.238 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:37.223 7f8b553f6700 1 --2- 172.21.2.102:0/110330289 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8b500e5070 0x7f8b501b3820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:05:37.238 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:37.223 7f8b553f6700 1 -- 172.21.2.102:0/110330289 >> 172.21.2.102:0/110330289 conn(0x7f8b500ce690 msgr2=0x7f8b500e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:05:37.238 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:37.223 7f8b553f6700 1 -- 172.21.2.102:0/110330289 shutdown_connections 2021-02-10T11:05:37.238 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:37.223 7f8b553f6700 1 -- 172.21.2.102:0/110330289 wait complete. 2021-02-10T11:05:37.239 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:37.238 7f8b553f6700 1 Processor -- start 2021-02-10T11:05:37.240 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:37.238 7f8b553f6700 1 -- start start 2021-02-10T11:05:37.240 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:37.238 7f8b553f6700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8b500be580 0x7f8b500be980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:37.240 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:37.238 7f8b553f6700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8b50097ed0 0x7f8b500982d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:37.240 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:37.238 7f8b553f6700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8b50099180 0x7f8b50099580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:37.240 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:37.238 7f8b553f6700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b500988f0 con 0x7f8b50097ed0 2021-02-10T11:05:37.240 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:37.238 7f8b553f6700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b500c1110 con 0x7f8b500be580 2021-02-10T11:05:37.240 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:37.238 7f8b553f6700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b50099ba0 con 0x7f8b50099180 2021-02-10T11:05:37.240 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:37.239 7f8b4e87c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8b50097ed0 0x7f8b500982d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:37.240 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:37.239 7f8b4e87c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8b50097ed0 0x7f8b500982d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:50864/0 (socket says 172.21.2.102:50864) 2021-02-10T11:05:37.240 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:37.239 7f8b4e87c700 1 -- 172.21.2.102:0/3175078424 learned_addr learned my addr 172.21.2.102:0/3175078424 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:05:37.240 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:37.239 7f8b4e87c700 1 -- 172.21.2.102:0/3175078424 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8b50099180 msgr2=0x7f8b50099580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:05:37.241 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:37.239 7f8b4e87c700 1 --2- 172.21.2.102:0/3175078424 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8b50099180 0x7f8b50099580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:05:37.241 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:37.239 7f8b4e87c700 1 -- 172.21.2.102:0/3175078424 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8b500be580 msgr2=0x7f8b500be980 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:05:37.241 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:37.239 7f8b4e87c700 1 --2- 172.21.2.102:0/3175078424 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8b500be580 0x7f8b500be980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:05:37.241 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:37.239 7f8b4e87c700 1 -- 172.21.2.102:0/3175078424 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b50099dc0 con 0x7f8b50097ed0 2021-02-10T11:05:37.241 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:37.239 7f8b4e87c700 1 --2- 172.21.2.102:0/3175078424 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8b50097ed0 0x7f8b500982d0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 rx=0x7f8b4000b660 tx=0x7f8b4000ba00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:05:37.241 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:37.239 7f8b47fff700 1 -- 172.21.2.102:0/3175078424 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8b40009440 con 0x7f8b50097ed0 2021-02-10T11:05:37.241 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:37.239 7f8b47fff700 1 -- 172.21.2.102:0/3175078424 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b40000c90 con 0x7f8b50097ed0 2021-02-10T11:05:37.241 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:37.239 7f8b553f6700 1 -- 172.21.2.102:0/3175078424 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b50099dc0 con 0x7f8b50097ed0 2021-02-10T11:05:37.241 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:37.239 7f8b47fff700 1 -- 172.21.2.102:0/3175078424 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8b40002030 con 0x7f8b50097ed0 2021-02-10T11:05:37.241 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:37.240 7f8b553f6700 1 -- 172.21.2.102:0/3175078424 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b50099dc0 con 0x7f8b50097ed0 2021-02-10T11:05:37.241 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:37.240 7f8b47fff700 1 -- 172.21.2.102:0/3175078424 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f8b400093e0 con 0x7f8b50097ed0 2021-02-10T11:05:37.241 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:37.240 7f8b47fff700 1 --2- 172.21.2.102:0/3175078424 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8b340186c0 0x7f8b3401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:37.241 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:37.240 7f8b47fff700 1 -- 172.21.2.102:0/3175078424 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f8b40032550 con 0x7f8b50097ed0 2021-02-10T11:05:37.242 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:37.240 7f8b4f07d700 1 --2- 172.21.2.102:0/3175078424 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8b340186c0 0x7f8b3401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:37.242 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:37.241 7f8b4f07d700 1 --2- 172.21.2.102:0/3175078424 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8b340186c0 0x7f8b3401abc0 crc :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:05:37.270 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:37.272 7f8b553f6700 1 -- 172.21.2.102:0/3175078424 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b50098c50 con 0x7f8b50097ed0 2021-02-10T11:05:37.273 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:37.274 7f8b47fff700 1 -- 172.21.2.102:0/3175078424 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f8b400090c0 con 0x7f8b50097ed0 2021-02-10T11:05:37.407 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:37.409 7f8b553f6700 1 -- 172.21.2.102:0/3175078424 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs volume ls", "target": ["mgr", ""]}) v1 -- 0x7f8b50098c50 con 0x7f8b340186c0 2021-02-10T11:05:37.408 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:37.409 7f8b47fff700 1 -- 172.21.2.102:0/3175078424 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+41 (crc 0 0 0) 0x7f8b38006970 con 0x7f8b340186c0 2021-02-10T11:05:37.408 INFO:teuthology.orchestra.run.gibba002.stdout:[ 2021-02-10T11:05:37.409 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:05:37.409 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "cephfs" 2021-02-10T11:05:37.409 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:05:37.409 INFO:teuthology.orchestra.run.gibba002.stdout:] 2021-02-10T11:05:37.410 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:37.411 7f8b553f6700 1 -- 172.21.2.102:0/3175078424 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8b340186c0 msgr2=0x7f8b3401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:37.410 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:37.411 7f8b553f6700 1 --2- 172.21.2.102:0/3175078424 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8b340186c0 0x7f8b3401abc0 crc :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:37.410 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:37.411 7f8b553f6700 1 -- 172.21.2.102:0/3175078424 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8b50097ed0 msgr2=0x7f8b500982d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:37.411 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:37.411 7f8b553f6700 1 --2- 172.21.2.102:0/3175078424 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8b50097ed0 0x7f8b500982d0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 rx=0x7f8b4000b660 tx=0x7f8b4000ba00).stop 2021-02-10T11:05:37.411 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:37.411 7f8b553f6700 1 -- 172.21.2.102:0/3175078424 shutdown_connections 2021-02-10T11:05:37.411 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:37.411 7f8b553f6700 1 --2- 172.21.2.102:0/3175078424 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8b50099180 0x7f8b50099580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:05:37.411 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:37.411 7f8b553f6700 1 --2- 172.21.2.102:0/3175078424 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8b50097ed0 0x7f8b500982d0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:37.411 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:37.411 7f8b553f6700 1 --2- 172.21.2.102:0/3175078424 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8b340186c0 0x7f8b3401abc0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:37.411 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:37.411 7f8b553f6700 1 --2- 172.21.2.102:0/3175078424 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8b500be580 0x7f8b500be980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:05:37.411 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:37.411 7f8b553f6700 1 -- 172.21.2.102:0/3175078424 >> 172.21.2.102:0/3175078424 conn(0x7f8b500bde70 msgr2=0x7f8b500c3970 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:05:37.411 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:37.411 7f8b553f6700 1 -- 172.21.2.102:0/3175078424 shutdown_connections 2021-02-10T11:05:37.412 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:37.411 7f8b553f6700 1 -- 172.21.2.102:0/3175078424 wait complete. 2021-02-10T11:05:37.418 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config set mon mon_allow_pool_delete True 2021-02-10T11:05:37.849 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:37.834 7f4c68f90700 1 Processor -- start 2021-02-10T11:05:37.849 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:37.834 7f4c68f90700 1 -- start start 2021-02-10T11:05:37.849 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:37.834 7f4c68f90700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f4c641b0870 0x7f4c640f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:37.849 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:37.834 7f4c68f90700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4c640efe00 0x7f4c640f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:37.849 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:37.834 7f4c68f90700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c641b0c70 con 0x7f4c641b0470 2021-02-10T11:05:37.849 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:37.834 7f4c68f90700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c640f0b50 con 0x7f4c641b0870 2021-02-10T11:05:37.850 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:37.834 7f4c68f90700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c640f0820 con 0x7f4c640efe00 2021-02-10T11:05:37.850 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:37.834 7f4c62d9d700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f4c641b0470 0x7f4c640ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:43096/0 (socket says 172.21.2.102:43096) 2021-02-10T11:05:37.850 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:37.834 7f4c62d9d700 1 -- 172.21.2.102:0/2538456821 learned_addr learned my addr 172.21.2.102:0/2538456821 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:05:37.850 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:37.834 7f4c6259c700 1 --2- 172.21.2.102:0/2538456821 >> v2:172.21.2.102:3300/0 conn(0x7f4c641b0870 0x7f4c640f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:37.850 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:37.835 7f4c6359e700 1 --2- 172.21.2.102:0/2538456821 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4c640efe00 0x7f4c640f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:37.850 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:37.835 7f4c6259c700 1 -- 172.21.2.102:0/2538456821 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4c640efe00 msgr2=0x7f4c640f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:05:37.850 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:37.835 7f4c6259c700 1 --2- 172.21.2.102:0/2538456821 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4c640efe00 0x7f4c640f0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:37.850 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:37.835 7f4c6259c700 1 -- 172.21.2.102:0/2538456821 >> v1:172.21.2.102:6789/0 conn(0x7f4c641b0470 legacy=0x7f4c640ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:37.850 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:37.835 7f4c6259c700 1 -- 172.21.2.102:0/2538456821 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c640e52e0 con 0x7f4c641b0870 2021-02-10T11:05:37.850 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:37.835 7f4c6259c700 1 --2- 172.21.2.102:0/2538456821 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4c641b0870 0x7f4c640f20e0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 rx=0x7f4c54006650 tx=0x7f4c54009a80).ready entity=mon.0 client_cookie=9892b2012b80a0fb server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:05:37.850 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:37.835 7f4c61d9b700 1 -- 172.21.2.102:0/2538456821 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4c5400c0c0 con 0x7f4c641b0870 2021-02-10T11:05:37.850 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:37.835 7f4c61d9b700 1 -- 172.21.2.102:0/2538456821 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c5400c340 con 0x7f4c641b0870 2021-02-10T11:05:37.850 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:37.835 7f4c61d9b700 1 -- 172.21.2.102:0/2538456821 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4c5400c860 con 0x7f4c641b0870 2021-02-10T11:05:37.851 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:37.835 7f4c68f90700 1 -- 172.21.2.102:0/2538456821 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4c641b0870 msgr2=0x7f4c640f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:37.851 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:37.835 7f4c68f90700 1 --2- 172.21.2.102:0/2538456821 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4c641b0870 0x7f4c640f20e0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 rx=0x7f4c54006650 tx=0x7f4c54009a80).stop 2021-02-10T11:05:37.851 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:37.836 7f4c68f90700 1 -- 172.21.2.102:0/2538456821 shutdown_connections 2021-02-10T11:05:37.851 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:37.836 7f4c68f90700 1 --2- 172.21.2.102:0/2538456821 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4c641b0870 0x7f4c640f20e0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:37.851 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:37.836 7f4c68f90700 1 --2- 172.21.2.102:0/2538456821 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4c640efe00 0x7f4c640f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:37.851 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:37.836 7f4c68f90700 1 -- 172.21.2.102:0/2538456821 >> 172.21.2.102:0/2538456821 conn(0x7f4c640ce3e0 msgr2=0x7f4c640e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:05:37.851 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:37.836 7f4c68f90700 1 -- 172.21.2.102:0/2538456821 shutdown_connections 2021-02-10T11:05:37.851 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:37.836 7f4c68f90700 1 -- 172.21.2.102:0/2538456821 wait complete. 2021-02-10T11:05:37.853 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:37.851 7f4c68f90700 1 Processor -- start 2021-02-10T11:05:37.853 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:37.851 7f4c68f90700 1 -- start start 2021-02-10T11:05:37.853 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:37.851 7f4c68f90700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4c640a0420 0x7f4c640a0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:37.853 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:37.851 7f4c68f90700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4c64189d00 0x7f4c6418a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:37.853 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:37.851 7f4c68f90700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4c6418a720 0x7f4c640b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:37.853 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:37.851 7f4c68f90700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c64189800 con 0x7f4c64189d00 2021-02-10T11:05:37.853 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:37.851 7f4c68f90700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c6418ab20 con 0x7f4c6418a720 2021-02-10T11:05:37.854 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:37.851 7f4c68f90700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c640a1140 con 0x7f4c640a0420 2021-02-10T11:05:37.854 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:37.852 7f4c6259c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4c64189d00 0x7f4c6418a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:37.854 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:37.852 7f4c6259c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4c64189d00 0x7f4c6418a100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:50878/0 (socket says 172.21.2.102:50878) 2021-02-10T11:05:37.854 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:37.852 7f4c6259c700 1 -- 172.21.2.102:0/463739720 learned_addr learned my addr 172.21.2.102:0/463739720 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:05:37.854 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:37.852 7f4c6259c700 1 -- 172.21.2.102:0/463739720 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4c640a0420 msgr2=0x7f4c640a0820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:37.854 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:37.852 7f4c62d9d700 1 --2- 172.21.2.102:0/463739720 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4c640a0420 0x7f4c640a0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:37.854 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:37.852 7f4c6359e700 1 --2- 172.21.2.102:0/463739720 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4c6418a720 0x7f4c640b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:37.854 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:37.852 7f4c6259c700 1 --2- 172.21.2.102:0/463739720 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4c640a0420 0x7f4c640a0820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:37.854 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:37.852 7f4c6259c700 1 -- 172.21.2.102:0/463739720 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4c6418a720 msgr2=0x7f4c640b7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:37.854 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:37.852 7f4c6259c700 1 --2- 172.21.2.102:0/463739720 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4c6418a720 0x7f4c640b7be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:37.854 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:37.852 7f4c6259c700 1 -- 172.21.2.102:0/463739720 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c640b8200 con 0x7f4c64189d00 2021-02-10T11:05:37.855 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:37.852 7f4c62d9d700 1 --2- 172.21.2.102:0/463739720 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4c640a0420 0x7f4c640a0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:05:37.855 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:37.852 7f4c6359e700 1 --2- 172.21.2.102:0/463739720 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4c6418a720 0x7f4c640b7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:05:37.855 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:37.852 7f4c6259c700 1 --2- 172.21.2.102:0/463739720 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4c64189d00 0x7f4c6418a100 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 rx=0x7f4c54009630 tx=0x7f4c5400b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:05:37.855 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:37.852 7f4c5b7fe700 1 -- 172.21.2.102:0/463739720 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4c540093a0 con 0x7f4c64189d00 2021-02-10T11:05:37.855 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:37.852 7f4c5b7fe700 1 -- 172.21.2.102:0/463739720 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c5400bd70 con 0x7f4c64189d00 2021-02-10T11:05:37.855 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:37.852 7f4c5b7fe700 1 -- 172.21.2.102:0/463739720 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4c54005560 con 0x7f4c64189d00 2021-02-10T11:05:37.855 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:37.852 7f4c68f90700 1 -- 172.21.2.102:0/463739720 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c640b8200 con 0x7f4c64189d00 2021-02-10T11:05:37.855 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:37.853 7f4c68f90700 1 -- 172.21.2.102:0/463739720 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c640b8200 con 0x7f4c64189d00 2021-02-10T11:05:37.855 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:37.853 7f4c5b7fe700 1 -- 172.21.2.102:0/463739720 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4c54005770 con 0x7f4c64189d00 2021-02-10T11:05:37.855 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:37.853 7f4c5b7fe700 1 --2- 172.21.2.102:0/463739720 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4c480186c0 0x7f4c4801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:37.855 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:37.853 7f4c5b7fe700 1 -- 172.21.2.102:0/463739720 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f4c54032520 con 0x7f4c64189d00 2021-02-10T11:05:37.855 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:37.853 7f4c62d9d700 1 --2- 172.21.2.102:0/463739720 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4c480186c0 0x7f4c4801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:37.856 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:37.854 7f4c62d9d700 1 --2- 172.21.2.102:0/463739720 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4c480186c0 0x7f4c4801abc0 crc :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:05:37.883 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:37.884 7f4c68f90700 1 -- 172.21.2.102:0/463739720 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c6418ab20 con 0x7f4c64189d00 2021-02-10T11:05:37.885 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:37.887 7f4c5b7fe700 1 -- 172.21.2.102:0/463739720 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f4c540246a0 con 0x7f4c64189d00 2021-02-10T11:05:38.033 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:38.035 7f4c68f90700 1 -- 172.21.2.102:0/463739720 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f4c6418ab20 con 0x7f4c64189d00 2021-02-10T11:05:38.034 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:38.035 7f4c5b7fe700 1 -- 172.21.2.102:0/463739720 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v2)=0 v2) v1 ==== 122+0+0 (secure 0 0 0) 0x7f4c540246a0 con 0x7f4c64189d00 2021-02-10T11:05:38.036 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:38.037 7f4c68f90700 1 -- 172.21.2.102:0/463739720 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4c480186c0 msgr2=0x7f4c4801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:38.036 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:38.037 7f4c68f90700 1 --2- 172.21.2.102:0/463739720 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4c480186c0 0x7f4c4801abc0 crc :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:38.036 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:38.037 7f4c68f90700 1 -- 172.21.2.102:0/463739720 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4c64189d00 msgr2=0x7f4c6418a100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:38.036 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:38.037 7f4c68f90700 1 --2- 172.21.2.102:0/463739720 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4c64189d00 0x7f4c6418a100 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 rx=0x7f4c54009630 tx=0x7f4c5400b980).stop 2021-02-10T11:05:38.036 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:38.037 7f4c68f90700 1 -- 172.21.2.102:0/463739720 shutdown_connections 2021-02-10T11:05:38.036 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:38.037 7f4c68f90700 1 --2- 172.21.2.102:0/463739720 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4c480186c0 0x7f4c4801abc0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:38.036 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:38.037 7f4c68f90700 1 --2- 172.21.2.102:0/463739720 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4c6418a720 0x7f4c640b7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:38.036 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:38.037 7f4c68f90700 1 --2- 172.21.2.102:0/463739720 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4c64189d00 0x7f4c6418a100 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:38.037 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:38.037 7f4c68f90700 1 --2- 172.21.2.102:0/463739720 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4c640a0420 0x7f4c640a0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:38.037 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:38.037 7f4c68f90700 1 -- 172.21.2.102:0/463739720 >> 172.21.2.102:0/463739720 conn(0x7f4c64094e10 msgr2=0x7f4c640a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:05:38.037 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:38.038 7f4c68f90700 1 -- 172.21.2.102:0/463739720 shutdown_connections 2021-02-10T11:05:38.037 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:38.038 7f4c68f90700 1 -- 172.21.2.102:0/463739720 wait complete. 2021-02-10T11:05:38.044 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000344364 2021-02-10T11:05:38.474 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:38.457 7f1aa55af700 1 Processor -- start 2021-02-10T11:05:38.474 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:38.457 7f1aa55af700 1 -- start start 2021-02-10T11:05:38.474 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:38.458 7f1aa55af700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1aa01b0470 0x7f1aa01b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:38.474 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:38.458 7f1aa55af700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1aa00f0b50 con 0x7f1aa00ef610 2021-02-10T11:05:38.474 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:38.458 7f1aa55af700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1aa00f24e0 con 0x7f1aa00f20e0 2021-02-10T11:05:38.474 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:38.458 7f1aa55af700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1aa00e5ea0 con 0x7f1aa01b0470 2021-02-10T11:05:38.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:38.458 7f1a9eabe700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f1aa00ef610 0x7f1aa00efa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:43108/0 (socket says 172.21.2.102:43108) 2021-02-10T11:05:38.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:38.458 7f1a9eabe700 1 -- 172.21.2.102:0/3222547699 learned_addr learned my addr 172.21.2.102:0/3222547699 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:05:38.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:38.458 7f1a9e2bd700 1 -- 172.21.2.102:0/3222547699 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1566207952 0 0) 0x7f1a900010b0 con 0x7f1aa00ef610 2021-02-10T11:05:38.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:38.459 7f1a9e2bd700 1 -- 172.21.2.102:0/3222547699 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a8c002fe0 con 0x7f1aa00ef610 2021-02-10T11:05:38.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:38.459 7f1a9f2bf700 1 --2- 172.21.2.102:0/3222547699 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1aa01b0470 0x7f1aa01b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:38.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:38.459 7f1a9e2bd700 1 -- 172.21.2.102:0/3222547699 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 379176100 0 0) 0x7f1a900010b0 con 0x7f1aa00ef610 2021-02-10T11:05:38.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:38.459 7f1a9e2bd700 1 -- 172.21.2.102:0/3222547699 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1aa01b0470 msgr2=0x7f1aa01b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:05:38.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:38.459 7f1a9e2bd700 1 --2- 172.21.2.102:0/3222547699 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1aa01b0470 0x7f1aa01b0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:38.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:38.459 7f1a9e2bd700 1 -- 172.21.2.102:0/3222547699 >> v1:172.21.2.113:6789/0 conn(0x7f1aa00f20e0 legacy=0x7f1aa00f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:38.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:38.459 7f1a9e2bd700 1 -- 172.21.2.102:0/3222547699 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1aa00e52e0 con 0x7f1aa00ef610 2021-02-10T11:05:38.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:38.459 7f1a9e2bd700 1 -- 172.21.2.102:0/3222547699 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7f1a900015b0 con 0x7f1aa00ef610 2021-02-10T11:05:38.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:38.459 7f1a9e2bd700 1 -- 172.21.2.102:0/3222547699 >> v1:172.21.2.102:6789/0 conn(0x7f1aa00ef610 legacy=0x7f1aa00efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:38.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:38.459 7f1a9e2bd700 1 --2- 172.21.2.102:0/3222547699 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1a8c003a60 0x7f1a8c005f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:38.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:38.459 7f1a9e2bd700 1 --2- 172.21.2.102:0/3222547699 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1a8c006750 0x7f1a8c008c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:38.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:38.459 7f1a9e2bd700 1 --2- 172.21.2.102:0/3222547699 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1a8c009510 0x7f1a8c00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:38.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:38.459 7f1a9e2bd700 1 -- 172.21.2.102:0/3222547699 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a8c00c2c0 con 0x7f1a8c009510 2021-02-10T11:05:38.477 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:38.459 7f1a9e2bd700 1 -- 172.21.2.102:0/3222547699 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a8c00c4e0 con 0x7f1a8c006750 2021-02-10T11:05:38.477 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:38.459 7f1a9e2bd700 1 -- 172.21.2.102:0/3222547699 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a8c00c700 con 0x7f1a8c003a60 2021-02-10T11:05:38.477 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:38.459 7f1a9fac0700 1 --2- 172.21.2.102:0/3222547699 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1a8c009510 0x7f1a8c00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:38.477 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:38.459 7f1a9eabe700 1 --2- 172.21.2.102:0/3222547699 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1a8c006750 0x7f1a8c008c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:38.477 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:38.459 7f1a9f2bf700 1 --2- 172.21.2.102:0/3222547699 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1a8c003a60 0x7f1a8c005f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:38.477 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:38.459 7f1a9fac0700 1 -- 172.21.2.102:0/3222547699 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1a8c003a60 msgr2=0x7f1a8c005f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:05:38.477 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:38.459 7f1a9fac0700 1 --2- 172.21.2.102:0/3222547699 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1a8c003a60 0x7f1a8c005f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:38.477 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:38.459 7f1a9fac0700 1 -- 172.21.2.102:0/3222547699 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1a8c006750 msgr2=0x7f1a8c008c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:05:38.477 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:38.459 7f1a9fac0700 1 --2- 172.21.2.102:0/3222547699 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1a8c006750 0x7f1a8c008c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:38.477 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:38.459 7f1a9fac0700 1 -- 172.21.2.102:0/3222547699 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a8c00c9a0 con 0x7f1a8c009510 2021-02-10T11:05:38.477 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:38.459 7f1a9fac0700 1 --2- 172.21.2.102:0/3222547699 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1a8c009510 0x7f1a8c00ba10 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 rx=0x7f1a98006e40 tx=0x7f1a98009e30).ready entity=mon.0 client_cookie=4934291c5a1e87c7 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:05:38.477 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:38.460 7f1a9e2bd700 1 -- 172.21.2.102:0/3222547699 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1a98003030 con 0x7f1a8c009510 2021-02-10T11:05:38.477 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:38.460 7f1a9e2bd700 1 -- 172.21.2.102:0/3222547699 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a9800b300 con 0x7f1a8c009510 2021-02-10T11:05:38.478 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:38.460 7f1a9e2bd700 1 -- 172.21.2.102:0/3222547699 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1a9800b750 con 0x7f1a8c009510 2021-02-10T11:05:38.478 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:38.460 7f1aa55af700 1 -- 172.21.2.102:0/3222547699 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1a8c009510 msgr2=0x7f1a8c00ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:38.478 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:38.460 7f1aa55af700 1 --2- 172.21.2.102:0/3222547699 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1a8c009510 0x7f1a8c00ba10 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 rx=0x7f1a98006e40 tx=0x7f1a98009e30).stop 2021-02-10T11:05:38.478 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:38.460 7f1a9fac0700 1 -- 172.21.2.102:0/3222547699 reap_dead start 2021-02-10T11:05:38.478 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:38.460 7f1aa55af700 1 -- 172.21.2.102:0/3222547699 shutdown_connections 2021-02-10T11:05:38.478 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:38.460 7f1aa55af700 1 -- 172.21.2.102:0/3222547699 >> 172.21.2.102:0/3222547699 conn(0x7f1aa00ce3e0 msgr2=0x7f1aa00e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:05:38.478 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:38.460 7f1aa55af700 1 -- 172.21.2.102:0/3222547699 shutdown_connections 2021-02-10T11:05:38.479 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:38.460 7f1aa55af700 1 -- 172.21.2.102:0/3222547699 wait complete. 2021-02-10T11:05:38.479 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:38.475 7f1aa55af700 1 Processor -- start 2021-02-10T11:05:38.479 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:38.475 7f1aa55af700 1 -- start start 2021-02-10T11:05:38.479 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:38.475 7f1aa55af700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1aa00bf920 0x7f1aa00bfd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:38.479 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:38.475 7f1aa55af700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1aa00b6570 0x7f1aa00b6970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:38.479 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:38.475 7f1aa55af700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1aa01a7430 0x7f1aa01a7830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:38.479 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:38.475 7f1aa55af700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1aa00c0440 con 0x7f1aa00bf920 2021-02-10T11:05:38.479 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:38.475 7f1aa55af700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1aa00924b0 con 0x7f1aa01a7430 2021-02-10T11:05:38.479 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:38.475 7f1aa55af700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1aa0092970 con 0x7f1aa00b6570 2021-02-10T11:05:38.479 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:38.475 7f1a9f2bf700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1aa00bf920 0x7f1aa00bfd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:38.479 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:38.475 7f1a9f2bf700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1aa00bf920 0x7f1aa00bfd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:50894/0 (socket says 172.21.2.102:50894) 2021-02-10T11:05:38.479 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:38.475 7f1a9f2bf700 1 -- 172.21.2.102:0/2319495833 learned_addr learned my addr 172.21.2.102:0/2319495833 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:05:38.482 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:38.476 7f1a9f2bf700 1 -- 172.21.2.102:0/2319495833 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1aa00b6570 msgr2=0x7f1aa00b6970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:38.482 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:38.476 7f1a9f2bf700 1 --2- 172.21.2.102:0/2319495833 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1aa00b6570 0x7f1aa00b6970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:05:38.482 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:38.476 7f1a9f2bf700 1 -- 172.21.2.102:0/2319495833 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1aa01a7430 msgr2=0x7f1aa01a7830 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:05:38.482 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:38.476 7f1a9f2bf700 1 --2- 172.21.2.102:0/2319495833 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1aa01a7430 0x7f1aa01a7830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:05:38.482 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:38.476 7f1a9f2bf700 1 -- 172.21.2.102:0/2319495833 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1aa01a7e50 con 0x7f1aa00bf920 2021-02-10T11:05:38.483 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:38.476 7f1a9f2bf700 1 --2- 172.21.2.102:0/2319495833 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1aa00bf920 0x7f1aa00bfd20 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 rx=0x7f1a88006f10 tx=0x7f1a8800b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:05:38.483 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:38.476 7f1a97fff700 1 -- 172.21.2.102:0/2319495833 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1a8800bd80 con 0x7f1aa00bf920 2021-02-10T11:05:38.483 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:38.476 7f1a97fff700 1 -- 172.21.2.102:0/2319495833 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a88000c90 con 0x7f1aa00bf920 2021-02-10T11:05:38.483 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:38.476 7f1aa55af700 1 -- 172.21.2.102:0/2319495833 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1aa01a7e50 con 0x7f1aa00bf920 2021-02-10T11:05:38.483 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:38.476 7f1a97fff700 1 -- 172.21.2.102:0/2319495833 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1a880022b0 con 0x7f1aa00bf920 2021-02-10T11:05:38.483 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:38.476 7f1aa55af700 1 -- 172.21.2.102:0/2319495833 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1aa00c09c0 con 0x7f1aa00bf920 2021-02-10T11:05:38.483 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:38.477 7f1a97fff700 1 -- 172.21.2.102:0/2319495833 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1a880052e0 con 0x7f1aa00bf920 2021-02-10T11:05:38.484 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:38.477 7f1a97fff700 1 --2- 172.21.2.102:0/2319495833 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1a840186c0 0x7f1a8401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:38.484 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:38.477 7f1a97fff700 1 -- 172.21.2.102:0/2319495833 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f1a880325f0 con 0x7f1aa00bf920 2021-02-10T11:05:38.484 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:38.477 7f1a9eabe700 1 --2- 172.21.2.102:0/2319495833 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1a840186c0 0x7f1a8401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:38.484 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:38.477 7f1a9eabe700 1 --2- 172.21.2.102:0/2319495833 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1a840186c0 0x7f1a8401abc0 crc :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:05:38.506 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:38.508 7f1aa55af700 1 -- 172.21.2.102:0/2319495833 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1aa0105b40 con 0x7f1aa00bf920 2021-02-10T11:05:38.509 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:38.510 7f1a97fff700 1 -- 172.21.2.102:0/2319495833 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f1a88000940 con 0x7f1aa00bf920 2021-02-10T11:05:38.637 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:38.638 7f1aa55af700 1 -- 172.21.2.102:0/2319495833 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344364", "target": ["mgr", ""]}) v1 -- 0x7f1aa00c0440 con 0x7f1a840186c0 2021-02-10T11:05:38.640 INFO:tasks.ceph.mgr.y.gibba002.stderr:2021-02-10 11:05:38.641 7fd94bc53700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:05:38.640 INFO:tasks.ceph.mgr.y.gibba002.stderr:2021-02-10 11:05:38.641 7fd94bc53700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:05:38.640 INFO:tasks.ceph.mgr.y.gibba002.stderr:2021-02-10 11:05:38.641 7fd94bc53700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:05:38.640 INFO:tasks.ceph.mgr.y.gibba002.stderr:2021-02-10 11:05:38.641 7fd94bc53700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:05:38.640 INFO:tasks.ceph.mgr.y.gibba002.stderr:2021-02-10 11:05:38.641 7fd94bc53700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:05:38.670 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:38.672 7f1a97fff700 1 -- 172.21.2.102:0/2319495833 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f1a900009b0 con 0x7f1a840186c0 2021-02-10T11:05:38.702 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:38.703 7f1aa55af700 1 -- 172.21.2.102:0/2319495833 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1a840186c0 msgr2=0x7f1a8401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:38.703 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:38.703 7f1aa55af700 1 --2- 172.21.2.102:0/2319495833 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1a840186c0 0x7f1a8401abc0 crc :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:38.703 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:38.703 7f1aa55af700 1 -- 172.21.2.102:0/2319495833 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1aa00bf920 msgr2=0x7f1aa00bfd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:38.703 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:38.703 7f1aa55af700 1 --2- 172.21.2.102:0/2319495833 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1aa00bf920 0x7f1aa00bfd20 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 rx=0x7f1a88006f10 tx=0x7f1a8800b990).stop 2021-02-10T11:05:38.703 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:38.703 7f1aa55af700 1 -- 172.21.2.102:0/2319495833 shutdown_connections 2021-02-10T11:05:38.703 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:38.703 7f1aa55af700 1 --2- 172.21.2.102:0/2319495833 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1a840186c0 0x7f1a8401abc0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:38.703 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:38.703 7f1aa55af700 1 --2- 172.21.2.102:0/2319495833 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1aa01a7430 0x7f1aa01a7830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:05:38.703 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:38.703 7f1aa55af700 1 --2- 172.21.2.102:0/2319495833 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1aa00b6570 0x7f1aa00b6970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:05:38.703 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:38.703 7f1aa55af700 1 --2- 172.21.2.102:0/2319495833 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1aa00bf920 0x7f1aa00bfd20 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:38.703 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:38.703 7f1aa55af700 1 -- 172.21.2.102:0/2319495833 >> 172.21.2.102:0/2319495833 conn(0x7f1aa00b50c0 msgr2=0x7f1aa0092d90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:05:38.703 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:38.704 7f1aa55af700 1 -- 172.21.2.102:0/2319495833 shutdown_connections 2021-02-10T11:05:38.703 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:38.704 7f1aa55af700 1 -- 172.21.2.102:0/2319495833 wait complete. 2021-02-10T11:05:38.711 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000344364 2021-02-10T11:05:39.140 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:39.124 7f8215ecb700 1 Processor -- start 2021-02-10T11:05:39.140 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:39.124 7f8215ecb700 1 -- start start 2021-02-10T11:05:39.140 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:39.125 7f8215ecb700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f82101b0470 0x7f82101b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:39.140 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:39.125 7f8215ecb700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f82100ef610 0x7f82100efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:39.140 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:39.125 7f8215ecb700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82100f0b50 con 0x7f82100f0030 2021-02-10T11:05:39.140 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:39.125 7f8215ecb700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82100e52e0 con 0x7f82101b0470 2021-02-10T11:05:39.140 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:39.125 7f8215ecb700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82100e54d0 con 0x7f82100ef610 2021-02-10T11:05:39.140 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:39.125 7f820f7fe700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f82101b0470 0x7f82101b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:39.140 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:39.125 7f820effd700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f82100ef610 0x7f82100efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:39.141 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:39.125 7f820f7fe700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f82101b0470 0x7f82101b0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.113:3300/0 says I am v2:172.21.2.102:54348/0 (socket says 172.21.2.102:54348) 2021-02-10T11:05:39.141 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:39.125 7f820effd700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f82100ef610 0x7f82100efa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.113:3301/0 says I am v2:172.21.2.102:56454/0 (socket says 172.21.2.102:56454) 2021-02-10T11:05:39.141 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:39.125 7f820f7fe700 1 -- 172.21.2.102:0/2939946804 learned_addr learned my addr 172.21.2.102:0/2939946804 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:05:39.141 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:39.126 7f820f7fe700 1 -- 172.21.2.102:0/2939946804 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f82100ef610 msgr2=0x7f82100efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:05:39.141 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:39.126 7f820f7fe700 1 --2- 172.21.2.102:0/2939946804 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f82100ef610 0x7f82100efa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:39.141 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:39.126 7f820f7fe700 1 -- 172.21.2.102:0/2939946804 >> v1:172.21.2.113:6789/0 conn(0x7f82100f0030 legacy=0x7f82100f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:39.141 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:39.126 7f820f7fe700 1 -- 172.21.2.102:0/2939946804 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82100f0430 con 0x7f82101b0470 2021-02-10T11:05:39.141 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:39.126 7f820effd700 1 --2- 172.21.2.102:0/2939946804 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f82100ef610 0x7f82100efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:05:39.141 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:39.126 7f820f7fe700 1 --2- 172.21.2.102:0/2939946804 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f82101b0470 0x7f82101b0870 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7f81f8006700 tx=0x7f81f8009b80).ready entity=mon.1 client_cookie=a0b95f0251362ea1 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:05:39.141 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:39.126 7f820e7fc700 1 -- 172.21.2.102:0/2939946804 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f81f800c0c0 con 0x7f82101b0470 2021-02-10T11:05:39.141 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:39.126 7f820e7fc700 1 -- 172.21.2.102:0/2939946804 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81f800c3c0 con 0x7f82101b0470 2021-02-10T11:05:39.141 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:39.126 7f820e7fc700 1 -- 172.21.2.102:0/2939946804 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f81f800c810 con 0x7f82101b0470 2021-02-10T11:05:39.141 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:39.126 7f8215ecb700 1 -- 172.21.2.102:0/2939946804 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f82101b0470 msgr2=0x7f82101b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:39.142 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:39.126 7f8215ecb700 1 --2- 172.21.2.102:0/2939946804 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f82101b0470 0x7f82101b0870 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7f81f8006700 tx=0x7f81f8009b80).stop 2021-02-10T11:05:39.142 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:39.127 7f8215ecb700 1 -- 172.21.2.102:0/2939946804 shutdown_connections 2021-02-10T11:05:39.142 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:39.127 7f8215ecb700 1 --2- 172.21.2.102:0/2939946804 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f82100ef610 0x7f82100efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:39.142 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:39.127 7f8215ecb700 1 --2- 172.21.2.102:0/2939946804 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f82101b0470 0x7f82101b0870 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:39.142 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:39.127 7f8215ecb700 1 -- 172.21.2.102:0/2939946804 >> 172.21.2.102:0/2939946804 conn(0x7f82100ce3e0 msgr2=0x7f82100e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:05:39.142 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:39.127 7f8215ecb700 1 -- 172.21.2.102:0/2939946804 shutdown_connections 2021-02-10T11:05:39.142 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:39.127 7f8215ecb700 1 -- 172.21.2.102:0/2939946804 wait complete. 2021-02-10T11:05:39.142 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:39.141 7f8215ecb700 1 Processor -- start 2021-02-10T11:05:39.142 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:39.141 7f8215ecb700 1 -- start start 2021-02-10T11:05:39.143 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:39.142 7f8215ecb700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f82100a0420 0x7f82100a0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:39.143 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:39.142 7f8215ecb700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8210189d00 0x7f821018a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:39.144 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:39.142 7f8215ecb700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f821018a720 0x7f82100b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:39.144 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:39.142 7f8215ecb700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8210189780 con 0x7f821018a720 2021-02-10T11:05:39.144 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:39.142 7f8215ecb700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f821018ab20 con 0x7f82100a0420 2021-02-10T11:05:39.144 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:39.142 7f8215ecb700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82100a1140 con 0x7f8210189d00 2021-02-10T11:05:39.144 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:39.142 7f820ffff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f821018a720 0x7f82100b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:39.144 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:39.142 7f820ffff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f821018a720 0x7f82100b7be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:50926/0 (socket says 172.21.2.102:50926) 2021-02-10T11:05:39.144 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:39.142 7f820ffff700 1 -- 172.21.2.102:0/660382194 learned_addr learned my addr 172.21.2.102:0/660382194 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:05:39.144 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:39.142 7f820ffff700 1 -- 172.21.2.102:0/660382194 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8210189d00 msgr2=0x7f821018a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:39.144 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:39.142 7f820ffff700 1 --2- 172.21.2.102:0/660382194 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8210189d00 0x7f821018a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:05:39.144 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:39.142 7f820ffff700 1 -- 172.21.2.102:0/660382194 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f82100a0420 msgr2=0x7f82100a0820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:39.144 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:39.142 7f820ffff700 1 --2- 172.21.2.102:0/660382194 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f82100a0420 0x7f82100a0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:05:39.144 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:39.142 7f820ffff700 1 -- 172.21.2.102:0/660382194 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82100b8200 con 0x7f821018a720 2021-02-10T11:05:39.144 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:39.143 7f820ffff700 1 --2- 172.21.2.102:0/660382194 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f821018a720 0x7f82100b7be0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 rx=0x7f8208009940 tx=0x7f820800bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:05:39.144 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:39.143 7f8207fff700 1 -- 172.21.2.102:0/660382194 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8208000ce0 con 0x7f821018a720 2021-02-10T11:05:39.145 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:39.143 7f8207fff700 1 -- 172.21.2.102:0/660382194 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f820800c2f0 con 0x7f821018a720 2021-02-10T11:05:39.145 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:39.143 7f8207fff700 1 -- 172.21.2.102:0/660382194 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f820800c750 con 0x7f821018a720 2021-02-10T11:05:39.145 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:39.143 7f8215ecb700 1 -- 172.21.2.102:0/660382194 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82100b8200 con 0x7f821018a720 2021-02-10T11:05:39.145 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:39.143 7f8215ecb700 1 -- 172.21.2.102:0/660382194 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82100b8200 con 0x7f821018a720 2021-02-10T11:05:39.145 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:39.144 7f8207fff700 1 -- 172.21.2.102:0/660382194 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f8208001220 con 0x7f821018a720 2021-02-10T11:05:39.145 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:39.144 7f8207fff700 1 --2- 172.21.2.102:0/660382194 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f81f40186d0 0x7f81f401abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:39.145 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:39.144 7f8207fff700 1 -- 172.21.2.102:0/660382194 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f82080329a0 con 0x7f821018a720 2021-02-10T11:05:39.145 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:39.144 7f820f7fe700 1 --2- 172.21.2.102:0/660382194 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f81f40186d0 0x7f81f401abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:39.145 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:39.144 7f820f7fe700 1 --2- 172.21.2.102:0/660382194 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f81f40186d0 0x7f81f401abd0 crc :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:05:39.173 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:39.175 7f8215ecb700 1 -- 172.21.2.102:0/660382194 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8210105ae0 con 0x7f821018a720 2021-02-10T11:05:39.176 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:39.177 7f8207fff700 1 -- 172.21.2.102:0/660382194 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f820800be30 con 0x7f821018a720 2021-02-10T11:05:39.304 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:39.305 7f8215ecb700 1 -- 172.21.2.102:0/660382194 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344364", "target": ["mgr", ""]}) v1 -- 0x7f82100f0b50 con 0x7f81f40186d0 2021-02-10T11:05:39.307 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:39.308 7f8207fff700 1 -- 172.21.2.102:0/660382194 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f81f8000c90 con 0x7f81f40186d0 2021-02-10T11:05:39.307 INFO:teuthology.orchestra.run.gibba002.stdout:/volumes/_nogroup/subvolume_0000000000344364/3360921a-0c90-4665-a0c6-cc630eba523e 2021-02-10T11:05:39.308 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:39.310 7f8215ecb700 1 -- 172.21.2.102:0/660382194 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f81f40186d0 msgr2=0x7f81f401abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:39.308 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:39.310 7f8215ecb700 1 --2- 172.21.2.102:0/660382194 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f81f40186d0 0x7f81f401abd0 crc :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:39.308 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:39.310 7f8215ecb700 1 -- 172.21.2.102:0/660382194 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f821018a720 msgr2=0x7f82100b7be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:39.309 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:39.310 7f8215ecb700 1 --2- 172.21.2.102:0/660382194 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f821018a720 0x7f82100b7be0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 rx=0x7f8208009940 tx=0x7f820800bcf0).stop 2021-02-10T11:05:39.309 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:39.310 7f8215ecb700 1 -- 172.21.2.102:0/660382194 shutdown_connections 2021-02-10T11:05:39.309 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:39.310 7f8215ecb700 1 --2- 172.21.2.102:0/660382194 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f821018a720 0x7f82100b7be0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:39.309 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:39.310 7f8215ecb700 1 --2- 172.21.2.102:0/660382194 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8210189d00 0x7f821018a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:05:39.309 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:39.310 7f8215ecb700 1 --2- 172.21.2.102:0/660382194 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f81f40186d0 0x7f81f401abd0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:39.309 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:39.310 7f8215ecb700 1 --2- 172.21.2.102:0/660382194 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f82100a0420 0x7f82100a0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:05:39.309 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:39.310 7f8215ecb700 1 -- 172.21.2.102:0/660382194 >> 172.21.2.102:0/660382194 conn(0x7f8210094e10 msgr2=0x7f82100a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:05:39.309 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:39.310 7f8215ecb700 1 -- 172.21.2.102:0/660382194 shutdown_connections 2021-02-10T11:05:39.309 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:39.310 7f8215ecb700 1 -- 172.21.2.102:0/660382194 wait complete. 2021-02-10T11:05:39.317 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000344364 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000344364/3360921a-0c90-4665-a0c6-cc630eba523e 2021-02-10T11:05:39.317 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:05:39.317 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:05:39.317 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:05:39.651 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:05:39.652 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:05:39.652 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344364/3360921a-0c90-4665-a0c6-cc630eba523e/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:05:40.036 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:05:40.036 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:05:40.036 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0258978 s, 40.5 MB/s 2021-02-10T11:05:40.037 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:05:40.037 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:05:40.037 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:05:40.371 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:05:40.372 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:05:40.372 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344364/3360921a-0c90-4665-a0c6-cc630eba523e/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:05:40.741 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:05:40.742 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:05:40.742 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0236707 s, 44.3 MB/s 2021-02-10T11:05:40.743 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:05:40.743 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:05:40.743 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:05:41.078 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:05:41.079 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:05:41.079 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344364/3360921a-0c90-4665-a0c6-cc630eba523e/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:05:41.454 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:05:41.454 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:05:41.454 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0241317 s, 43.5 MB/s 2021-02-10T11:05:41.456 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:05:41.456 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:05:41.456 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:05:41.792 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:05:41.792 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:05:41.793 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344364/3360921a-0c90-4665-a0c6-cc630eba523e/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:05:42.168 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:05:42.168 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:05:42.168 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0244205 s, 42.9 MB/s 2021-02-10T11:05:42.169 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:05:42.170 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:05:42.170 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:05:42.377 DEBUG:teuthology.orchestra.run.gibba002:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:05:42.381 DEBUG:teuthology.orchestra.run.gibba013:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:05:42.504 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:05:42.505 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:05:42.505 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344364/3360921a-0c90-4665-a0c6-cc630eba523e/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:05:42.878 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:05:44.489 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:05:44.489 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0226352 s, 46.3 MB/s 2021-02-10T11:05:44.491 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:05:44.491 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:05:44.491 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:05:44.826 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:05:44.827 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:05:44.827 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344364/3360921a-0c90-4665-a0c6-cc630eba523e/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:05:45.202 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:05:45.202 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:05:45.202 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0260549 s, 40.2 MB/s 2021-02-10T11:05:45.203 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:05:45.203 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:05:45.203 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:05:45.538 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:05:45.538 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:05:45.539 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344364/3360921a-0c90-4665-a0c6-cc630eba523e/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:05:45.913 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:05:45.913 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:05:45.913 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0233755 s, 44.9 MB/s 2021-02-10T11:05:45.914 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:05:45.914 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:05:45.914 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:05:46.250 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:05:46.251 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:05:46.251 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344364/3360921a-0c90-4665-a0c6-cc630eba523e/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:05:46.627 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:05:46.627 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:05:46.628 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0264048 s, 39.7 MB/s 2021-02-10T11:05:46.628 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:05:46.629 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:05:46.629 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:05:46.962 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:05:46.962 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:05:46.963 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344364/3360921a-0c90-4665-a0c6-cc630eba523e/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:05:47.335 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:05:47.335 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:05:47.335 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0231536 s, 45.3 MB/s 2021-02-10T11:05:47.336 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:05:47.337 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:05:47.337 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:05:47.670 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:05:47.670 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:05:47.671 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344364/3360921a-0c90-4665-a0c6-cc630eba523e/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:05:48.048 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:05:48.048 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:05:48.048 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0265751 s, 39.5 MB/s 2021-02-10T11:05:48.049 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000344365 2021-02-10T11:05:48.488 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:48.473 7efe62afc700 1 Processor -- start 2021-02-10T11:05:48.488 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:48.473 7efe62afc700 1 -- start start 2021-02-10T11:05:48.488 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:48.473 7efe62afc700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7efe5c1b0870 0x7efe5c0f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:48.488 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:48.473 7efe62afc700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe5c1b0c70 con 0x7efe5c1b0470 2021-02-10T11:05:48.488 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:48.473 7efe62afc700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe5c0f0b50 con 0x7efe5c0efe00 2021-02-10T11:05:48.488 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:48.473 7efe62afc700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe5c0e5ea0 con 0x7efe5c1b0870 2021-02-10T11:05:48.488 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:48.474 7efe60918700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7efe5c1b0470 0x7efe5c0ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:43164/0 (socket says 172.21.2.102:43164) 2021-02-10T11:05:48.489 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:48.474 7efe60918700 1 -- 172.21.2.102:0/2776833174 learned_addr learned my addr 172.21.2.102:0/2776833174 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:05:48.489 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:48.474 7efe5bfff700 1 --2- 172.21.2.102:0/2776833174 >> v2:172.21.2.113:3300/0 conn(0x7efe5c1b0870 0x7efe5c0f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:48.489 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:48.474 7efe5b7fe700 1 -- 172.21.2.102:0/2776833174 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4258888178 0 0) 0x7efe44000ce0 con 0x7efe5c1b0470 2021-02-10T11:05:48.489 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:48.474 7efe5b7fe700 1 -- 172.21.2.102:0/2776833174 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe48002fe0 con 0x7efe5c1b0470 2021-02-10T11:05:48.489 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:48.474 7efe5bfff700 1 -- 172.21.2.102:0/2776833174 >> v1:172.21.2.113:6789/0 conn(0x7efe5c0efe00 legacy=0x7efe5c0f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:48.489 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:48.474 7efe5bfff700 1 -- 172.21.2.102:0/2776833174 >> v1:172.21.2.102:6789/0 conn(0x7efe5c1b0470 legacy=0x7efe5c0ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:48.489 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:48.474 7efe5b7fe700 1 -- 172.21.2.102:0/2776833174 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 230042789 0 0) 0x7efe44000ed0 con 0x7efe5c1b0470 2021-02-10T11:05:48.489 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:48.474 7efe5bfff700 1 -- 172.21.2.102:0/2776833174 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe5c0e52e0 con 0x7efe5c1b0870 2021-02-10T11:05:48.489 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:48.474 7efe5bfff700 1 --2- 172.21.2.102:0/2776833174 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7efe5c1b0870 0x7efe5c0f20e0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7efe4c006650 tx=0x7efe4c009a80).ready entity=mon.1 client_cookie=6afc3dd98091e35b server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:05:48.489 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:48.474 7efe5b7fe700 1 -- 172.21.2.102:0/2776833174 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efe4c00c0c0 con 0x7efe5c1b0870 2021-02-10T11:05:48.489 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:48.474 7efe5b7fe700 1 -- 172.21.2.102:0/2776833174 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe4c00c340 con 0x7efe5c1b0870 2021-02-10T11:05:48.489 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:48.474 7efe5b7fe700 1 -- 172.21.2.102:0/2776833174 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efe4c00c820 con 0x7efe5c1b0870 2021-02-10T11:05:48.490 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:48.474 7efe62afc700 1 -- 172.21.2.102:0/2776833174 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7efe5c1b0870 msgr2=0x7efe5c0f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:48.490 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:48.474 7efe62afc700 1 --2- 172.21.2.102:0/2776833174 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7efe5c1b0870 0x7efe5c0f20e0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7efe4c006650 tx=0x7efe4c009a80).stop 2021-02-10T11:05:48.490 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:48.475 7efe62afc700 1 -- 172.21.2.102:0/2776833174 shutdown_connections 2021-02-10T11:05:48.490 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:48.475 7efe62afc700 1 --2- 172.21.2.102:0/2776833174 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7efe5c1b0870 0x7efe5c0f20e0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:48.490 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:48.475 7efe62afc700 1 -- 172.21.2.102:0/2776833174 >> 172.21.2.102:0/2776833174 conn(0x7efe5c0ce3e0 msgr2=0x7efe5c0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:05:48.490 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:48.475 7efe62afc700 1 -- 172.21.2.102:0/2776833174 shutdown_connections 2021-02-10T11:05:48.490 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:48.475 7efe62afc700 1 -- 172.21.2.102:0/2776833174 wait complete. 2021-02-10T11:05:48.491 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:48.490 7efe62afc700 1 Processor -- start 2021-02-10T11:05:48.491 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:48.490 7efe62afc700 1 -- start start 2021-02-10T11:05:48.491 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:48.490 7efe62afc700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7efe5c0b20b0 0x7efe5c0ac970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:48.491 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:48.490 7efe62afc700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7efe5c0b24b0 0x7efe5c191dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:48.492 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:48.490 7efe62afc700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efe5c0acf90 0x7efe5c1923e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:48.492 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:48.490 7efe62afc700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe5c0b1b30 con 0x7efe5c0acf90 2021-02-10T11:05:48.492 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:48.490 7efe62afc700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe5c1a04e0 con 0x7efe5c0b24b0 2021-02-10T11:05:48.492 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:48.490 7efe62afc700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe5c1a06c0 con 0x7efe5c0b20b0 2021-02-10T11:05:48.492 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:48.491 7efe61119700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efe5c0acf90 0x7efe5c1923e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:48.492 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:48.491 7efe61119700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efe5c0acf90 0x7efe5c1923e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:50948/0 (socket says 172.21.2.102:50948) 2021-02-10T11:05:48.492 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:48.491 7efe61119700 1 -- 172.21.2.102:0/1836765964 learned_addr learned my addr 172.21.2.102:0/1836765964 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:05:48.492 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:48.491 7efe5bfff700 1 --2- 172.21.2.102:0/1836765964 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7efe5c0b24b0 0x7efe5c191dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:48.492 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:48.491 7efe60918700 1 --2- 172.21.2.102:0/1836765964 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7efe5c0b20b0 0x7efe5c0ac970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:48.492 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:48.491 7efe61119700 1 -- 172.21.2.102:0/1836765964 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7efe5c0b20b0 msgr2=0x7efe5c0ac970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:48.492 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:48.491 7efe61119700 1 --2- 172.21.2.102:0/1836765964 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7efe5c0b20b0 0x7efe5c0ac970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:48.492 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:48.491 7efe61119700 1 -- 172.21.2.102:0/1836765964 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7efe5c0b24b0 msgr2=0x7efe5c191dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:48.493 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:48.491 7efe61119700 1 --2- 172.21.2.102:0/1836765964 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7efe5c0b24b0 0x7efe5c191dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:48.493 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:48.491 7efe61119700 1 -- 172.21.2.102:0/1836765964 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe5c192be0 con 0x7efe5c0acf90 2021-02-10T11:05:48.493 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:48.491 7efe5bfff700 1 --2- 172.21.2.102:0/1836765964 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7efe5c0b24b0 0x7efe5c191dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:05:48.493 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:48.491 7efe61119700 1 --2- 172.21.2.102:0/1836765964 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efe5c0acf90 0x7efe5c1923e0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 rx=0x7efe54009940 tx=0x7efe5400bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:05:48.493 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:48.492 7efe597fa700 1 -- 172.21.2.102:0/1836765964 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efe54000d30 con 0x7efe5c0acf90 2021-02-10T11:05:48.493 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:48.492 7efe597fa700 1 -- 172.21.2.102:0/1836765964 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe5400c2f0 con 0x7efe5c0acf90 2021-02-10T11:05:48.493 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:48.492 7efe597fa700 1 -- 172.21.2.102:0/1836765964 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efe5400c810 con 0x7efe5c0acf90 2021-02-10T11:05:48.493 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:48.492 7efe62afc700 1 -- 172.21.2.102:0/1836765964 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe5c192be0 con 0x7efe5c0acf90 2021-02-10T11:05:48.494 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:48.492 7efe62afc700 1 -- 172.21.2.102:0/1836765964 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe5c193170 con 0x7efe5c0acf90 2021-02-10T11:05:48.494 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:48.492 7efe597fa700 1 -- 172.21.2.102:0/1836765964 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7efe54000d30 con 0x7efe5c0acf90 2021-02-10T11:05:48.494 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:48.492 7efe597fa700 1 --2- 172.21.2.102:0/1836765964 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7efe400186c0 0x7efe4001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:48.494 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:48.492 7efe597fa700 1 -- 172.21.2.102:0/1836765964 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7efe54032920 con 0x7efe5c0acf90 2021-02-10T11:05:48.494 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:48.493 7efe60918700 1 --2- 172.21.2.102:0/1836765964 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7efe400186c0 0x7efe4001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:48.494 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:48.493 7efe60918700 1 --2- 172.21.2.102:0/1836765964 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7efe400186c0 0x7efe4001abc0 crc :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:05:48.521 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:48.522 7efe62afc700 1 -- 172.21.2.102:0/1836765964 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe5c105af0 con 0x7efe5c0acf90 2021-02-10T11:05:48.524 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:48.525 7efe597fa700 1 -- 172.21.2.102:0/1836765964 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7efe5402df90 con 0x7efe5c0acf90 2021-02-10T11:05:48.651 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:48.652 7efe62afc700 1 -- 172.21.2.102:0/1836765964 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344365", "target": ["mgr", ""]}) v1 -- 0x7efe5c1a06c0 con 0x7efe400186c0 2021-02-10T11:05:48.672 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:48.674 7efe597fa700 1 -- 172.21.2.102:0/1836765964 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7efe44001200 con 0x7efe400186c0 2021-02-10T11:05:48.684 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:48.685 7efe62afc700 1 -- 172.21.2.102:0/1836765964 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7efe400186c0 msgr2=0x7efe4001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:48.684 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:48.685 7efe62afc700 1 --2- 172.21.2.102:0/1836765964 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7efe400186c0 0x7efe4001abc0 crc :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:48.684 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:48.685 7efe62afc700 1 -- 172.21.2.102:0/1836765964 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efe5c0acf90 msgr2=0x7efe5c1923e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:48.684 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:48.685 7efe62afc700 1 --2- 172.21.2.102:0/1836765964 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efe5c0acf90 0x7efe5c1923e0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 rx=0x7efe54009940 tx=0x7efe5400bcf0).stop 2021-02-10T11:05:48.684 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:48.685 7efe62afc700 1 -- 172.21.2.102:0/1836765964 shutdown_connections 2021-02-10T11:05:48.685 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:48.685 7efe62afc700 1 --2- 172.21.2.102:0/1836765964 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efe5c0acf90 0x7efe5c1923e0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:48.685 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:48.685 7efe62afc700 1 --2- 172.21.2.102:0/1836765964 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7efe400186c0 0x7efe4001abc0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:48.685 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:48.685 7efe62afc700 1 --2- 172.21.2.102:0/1836765964 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7efe5c0b24b0 0x7efe5c191dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:48.685 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:48.685 7efe62afc700 1 --2- 172.21.2.102:0/1836765964 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7efe5c0b20b0 0x7efe5c0ac970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:48.685 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:48.685 7efe62afc700 1 -- 172.21.2.102:0/1836765964 >> 172.21.2.102:0/1836765964 conn(0x7efe5c0bf4e0 msgr2=0x7efe5c0be3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:05:48.685 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:48.685 7efe62afc700 1 -- 172.21.2.102:0/1836765964 shutdown_connections 2021-02-10T11:05:48.685 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:48.686 7efe62afc700 1 -- 172.21.2.102:0/1836765964 wait complete. 2021-02-10T11:05:48.692 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000344365 2021-02-10T11:05:49.129 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:49.113 7f54b7e18700 1 Processor -- start 2021-02-10T11:05:49.129 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:49.114 7f54b7e18700 1 -- start start 2021-02-10T11:05:49.129 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:49.114 7f54b7e18700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f54b01b0470 0x7f54b01b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:49.129 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:49.114 7f54b7e18700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f54b00ef610 0x7f54b00efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:49.130 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:49.114 7f54b7e18700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f54b00f0030 0x7f54b00f0430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:49.130 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:49.114 7f54b7e18700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54b00e5ea0 con 0x7f54b00ef610 2021-02-10T11:05:49.130 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:49.114 7f54b7e18700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54b00f0b50 con 0x7f54b01b0470 2021-02-10T11:05:49.130 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:49.114 7f54b7e18700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54b00e52e0 con 0x7f54b00f0030 2021-02-10T11:05:49.130 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:49.114 7f54b5433700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f54b00ef610 0x7f54b00efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:49.130 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:49.114 7f54b5c34700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f54b01b0470 0x7f54b01b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:49.130 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:49.115 7f54b5433700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f54b00ef610 0x7f54b00efa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:50954/0 (socket says 172.21.2.102:50954) 2021-02-10T11:05:49.130 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:49.115 7f54b5c34700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f54b01b0470 0x7f54b01b0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.113:3300/0 says I am v2:172.21.2.102:54382/0 (socket says 172.21.2.102:54382) 2021-02-10T11:05:49.130 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:49.115 7f54b5433700 1 -- 172.21.2.102:0/3622441810 learned_addr learned my addr 172.21.2.102:0/3622441810 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:05:49.130 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:49.115 7f54b6435700 1 --2- 172.21.2.102:0/3622441810 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f54b00f0030 0x7f54b00f0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:49.130 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:49.115 7f54b5c34700 1 -- 172.21.2.102:0/3622441810 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f54b00f0030 msgr2=0x7f54b00f0430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:05:49.130 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:49.115 7f54b5c34700 1 --2- 172.21.2.102:0/3622441810 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f54b00f0030 0x7f54b00f0430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:49.131 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:49.115 7f54b5c34700 1 -- 172.21.2.102:0/3622441810 >> v2:172.21.2.102:3300/0 conn(0x7f54b00ef610 msgr2=0x7f54b00efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:05:49.131 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:49.115 7f54b5c34700 1 --2- 172.21.2.102:0/3622441810 >> v2:172.21.2.102:3300/0 conn(0x7f54b00ef610 0x7f54b00efa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:49.131 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:49.115 7f54b5c34700 1 -- 172.21.2.102:0/3622441810 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54b00f10d0 con 0x7f54b01b0470 2021-02-10T11:05:49.131 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:49.115 7f54b5433700 1 --2- 172.21.2.102:0/3622441810 >> v2:172.21.2.102:3300/0 conn(0x7f54b00ef610 0x7f54b00efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:05:49.131 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:49.115 7f54b5c34700 1 --2- 172.21.2.102:0/3622441810 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f54b01b0470 0x7f54b01b0870 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7f54a80069d0 tx=0x7f54a8009e40).ready entity=mon.1 client_cookie=877e409b45905454 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:05:49.131 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:49.115 7f54b4c32700 1 -- 172.21.2.102:0/3622441810 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f54a80066c0 con 0x7f54b01b0470 2021-02-10T11:05:49.131 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:49.115 7f54b4c32700 1 -- 172.21.2.102:0/3622441810 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54a800b510 con 0x7f54b01b0470 2021-02-10T11:05:49.131 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:49.115 7f54b4c32700 1 -- 172.21.2.102:0/3622441810 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f54a800ba30 con 0x7f54b01b0470 2021-02-10T11:05:49.131 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:49.116 7f54b7e18700 1 -- 172.21.2.102:0/3622441810 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f54b01b0470 msgr2=0x7f54b01b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:49.131 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:49.116 7f54b7e18700 1 --2- 172.21.2.102:0/3622441810 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f54b01b0470 0x7f54b01b0870 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7f54a80069d0 tx=0x7f54a8009e40).stop 2021-02-10T11:05:49.131 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:49.116 7f54b7e18700 1 -- 172.21.2.102:0/3622441810 shutdown_connections 2021-02-10T11:05:49.131 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:49.116 7f54b7e18700 1 --2- 172.21.2.102:0/3622441810 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f54b00f0030 0x7f54b00f0430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:49.132 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:49.116 7f54b7e18700 1 --2- 172.21.2.102:0/3622441810 >> v2:172.21.2.102:3300/0 conn(0x7f54b00ef610 0x7f54b00efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:49.132 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:49.116 7f54b7e18700 1 --2- 172.21.2.102:0/3622441810 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f54b01b0470 0x7f54b01b0870 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:49.132 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:49.116 7f54b7e18700 1 -- 172.21.2.102:0/3622441810 >> 172.21.2.102:0/3622441810 conn(0x7f54b00ce3e0 msgr2=0x7f54b00e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:05:49.132 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:49.116 7f54b7e18700 1 -- 172.21.2.102:0/3622441810 shutdown_connections 2021-02-10T11:05:49.132 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:49.117 7f54b7e18700 1 -- 172.21.2.102:0/3622441810 wait complete. 2021-02-10T11:05:49.132 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:49.131 7f54b7e18700 1 Processor -- start 2021-02-10T11:05:49.134 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:49.131 7f54b7e18700 1 -- start start 2021-02-10T11:05:49.134 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:49.132 7f54b7e18700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f54b00b7db0 0x7f54b00b8260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:49.134 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:49.132 7f54b7e18700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f54b00b8880 0x7f54b01a7450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:49.134 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:49.132 7f54b7e18700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f54b00b8c80 0x7f54b01a7a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:49.134 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:49.132 7f54b7e18700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54b00b79b0 con 0x7f54b00b7db0 2021-02-10T11:05:49.134 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:49.132 7f54b7e18700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54b01a8090 con 0x7f54b00b8c80 2021-02-10T11:05:49.134 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:49.132 7f54b7e18700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54b01a82b0 con 0x7f54b00b8880 2021-02-10T11:05:49.134 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:49.132 7f54b6435700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f54b00b8c80 0x7f54b01a7a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:49.134 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:49.132 7f54b5c34700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f54b00b7db0 0x7f54b00b8260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:49.134 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:49.132 7f54b5c34700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f54b00b7db0 0x7f54b00b8260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:50958/0 (socket says 172.21.2.102:50958) 2021-02-10T11:05:49.135 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:49.132 7f54b5c34700 1 -- 172.21.2.102:0/3626180934 learned_addr learned my addr 172.21.2.102:0/3626180934 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:05:49.135 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:49.132 7f54b5433700 1 --2- 172.21.2.102:0/3626180934 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f54b00b8880 0x7f54b01a7450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:49.135 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:49.132 7f54b5c34700 1 -- 172.21.2.102:0/3626180934 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f54b00b8880 msgr2=0x7f54b01a7450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:49.135 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:49.132 7f54b5c34700 1 --2- 172.21.2.102:0/3626180934 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f54b00b8880 0x7f54b01a7450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:49.135 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:49.132 7f54b5c34700 1 -- 172.21.2.102:0/3626180934 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f54b00b8c80 msgr2=0x7f54b01a7a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:49.135 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:49.132 7f54b5c34700 1 --2- 172.21.2.102:0/3626180934 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f54b00b8c80 0x7f54b01a7a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:49.135 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:49.132 7f54b5c34700 1 -- 172.21.2.102:0/3626180934 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54b0190c10 con 0x7f54b00b7db0 2021-02-10T11:05:49.135 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:49.133 7f54b5c34700 1 --2- 172.21.2.102:0/3626180934 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f54b00b7db0 0x7f54b00b8260 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 rx=0x7f54a8009f40 tx=0x7f54a800bd50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:05:49.137 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:49.133 7f54ae7fc700 1 -- 172.21.2.102:0/3626180934 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f54a8006240 con 0x7f54b00b7db0 2021-02-10T11:05:49.137 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:49.133 7f54ae7fc700 1 -- 172.21.2.102:0/3626180934 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54a80065c0 con 0x7f54b00b7db0 2021-02-10T11:05:49.137 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:49.133 7f54ae7fc700 1 -- 172.21.2.102:0/3626180934 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f54a80058e0 con 0x7f54b00b7db0 2021-02-10T11:05:49.137 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:49.133 7f54b7e18700 1 -- 172.21.2.102:0/3626180934 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54b0190c10 con 0x7f54b00b7db0 2021-02-10T11:05:49.137 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:49.133 7f54b7e18700 1 -- 172.21.2.102:0/3626180934 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54b0190c10 con 0x7f54b00b7db0 2021-02-10T11:05:49.138 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:49.134 7f54ae7fc700 1 -- 172.21.2.102:0/3626180934 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f54a8005660 con 0x7f54b00b7db0 2021-02-10T11:05:49.138 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:49.134 7f54ae7fc700 1 --2- 172.21.2.102:0/3626180934 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5490018730 0x7f549001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:49.138 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:49.134 7f54ae7fc700 1 -- 172.21.2.102:0/3626180934 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f54a8032b30 con 0x7f54b00b7db0 2021-02-10T11:05:49.138 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:49.134 7f54b5433700 1 --2- 172.21.2.102:0/3626180934 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5490018730 0x7f549001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:49.138 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:49.134 7f54b5433700 1 --2- 172.21.2.102:0/3626180934 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5490018730 0x7f549001ac10 crc :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:05:49.162 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:49.164 7f54b7e18700 1 -- 172.21.2.102:0/3626180934 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54b0105b30 con 0x7f54b00b7db0 2021-02-10T11:05:49.165 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:49.166 7f54ae7fc700 1 -- 172.21.2.102:0/3626180934 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f54a8006ca0 con 0x7f54b00b7db0 2021-02-10T11:05:49.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:49.295 7f54b7e18700 1 -- 172.21.2.102:0/3626180934 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344365", "target": ["mgr", ""]}) v1 -- 0x7f54b00f0b50 con 0x7f5490018730 2021-02-10T11:05:49.298 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:49.299 7f54ae7fc700 1 -- 172.21.2.102:0/3626180934 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f54a4000960 con 0x7f5490018730 2021-02-10T11:05:49.300 INFO:teuthology.orchestra.run.gibba002.stdout:/volumes/_nogroup/subvolume_0000000000344365/81f30dd4-3513-42f8-82b2-24c0a3e8206b 2021-02-10T11:05:49.302 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:49.303 7f54b7e18700 1 -- 172.21.2.102:0/3626180934 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5490018730 msgr2=0x7f549001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:49.302 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:49.303 7f54b7e18700 1 --2- 172.21.2.102:0/3626180934 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5490018730 0x7f549001ac10 crc :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:49.302 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:49.303 7f54b7e18700 1 -- 172.21.2.102:0/3626180934 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f54b00b7db0 msgr2=0x7f54b00b8260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:49.302 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:49.303 7f54b7e18700 1 --2- 172.21.2.102:0/3626180934 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f54b00b7db0 0x7f54b00b8260 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 rx=0x7f54a8009f40 tx=0x7f54a800bd50).stop 2021-02-10T11:05:49.303 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:49.304 7f54b7e18700 1 -- 172.21.2.102:0/3626180934 shutdown_connections 2021-02-10T11:05:49.303 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:49.304 7f54b7e18700 1 --2- 172.21.2.102:0/3626180934 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5490018730 0x7f549001ac10 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:49.303 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:49.304 7f54b7e18700 1 --2- 172.21.2.102:0/3626180934 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f54b00b8c80 0x7f54b01a7a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:49.303 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:49.304 7f54b7e18700 1 --2- 172.21.2.102:0/3626180934 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f54b00b8880 0x7f54b01a7450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:49.303 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:49.304 7f54b7e18700 1 --2- 172.21.2.102:0/3626180934 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f54b00b7db0 0x7f54b00b8260 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:49.303 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:49.304 7f54b7e18700 1 -- 172.21.2.102:0/3626180934 >> 172.21.2.102:0/3626180934 conn(0x7f54b0094e10 msgr2=0x7f54b00a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:05:49.303 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:49.304 7f54b7e18700 1 -- 172.21.2.102:0/3626180934 shutdown_connections 2021-02-10T11:05:49.303 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:49.304 7f54b7e18700 1 -- 172.21.2.102:0/3626180934 wait complete. 2021-02-10T11:05:49.310 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000344365 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000344365/81f30dd4-3513-42f8-82b2-24c0a3e8206b 2021-02-10T11:05:49.310 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:05:49.310 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:05:49.310 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:05:49.652 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:05:49.653 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:05:49.653 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344365/81f30dd4-3513-42f8-82b2-24c0a3e8206b/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:05:50.035 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:05:50.035 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:05:50.035 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0254295 s, 41.2 MB/s 2021-02-10T11:05:50.036 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:05:50.036 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:05:50.036 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:05:50.371 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:05:50.371 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:05:50.372 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344365/81f30dd4-3513-42f8-82b2-24c0a3e8206b/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:05:50.745 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:05:50.746 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:05:50.746 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0235849 s, 44.5 MB/s 2021-02-10T11:05:50.747 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:05:50.747 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:05:50.747 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:05:51.081 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:05:51.081 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:05:51.082 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344365/81f30dd4-3513-42f8-82b2-24c0a3e8206b/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:05:51.456 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:05:51.457 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:05:51.457 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0238865 s, 43.9 MB/s 2021-02-10T11:05:51.458 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:05:51.458 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:05:51.458 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:05:51.797 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:05:51.797 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:05:51.798 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344365/81f30dd4-3513-42f8-82b2-24c0a3e8206b/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:05:52.172 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:05:52.172 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:05:52.172 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.022686 s, 46.2 MB/s 2021-02-10T11:05:52.173 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:05:52.173 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:05:52.173 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:05:52.508 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:05:52.508 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:05:52.509 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344365/81f30dd4-3513-42f8-82b2-24c0a3e8206b/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:05:52.885 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:05:52.885 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:05:52.885 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0257562 s, 40.7 MB/s 2021-02-10T11:05:52.886 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:05:52.886 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:05:52.886 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:05:53.227 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:05:53.227 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:05:53.228 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344365/81f30dd4-3513-42f8-82b2-24c0a3e8206b/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:05:53.600 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:05:53.600 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:05:53.600 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0243316 s, 43.1 MB/s 2021-02-10T11:05:53.601 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:05:53.602 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:05:53.602 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:05:53.936 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:05:53.937 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:05:53.937 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344365/81f30dd4-3513-42f8-82b2-24c0a3e8206b/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:05:54.311 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:05:54.311 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:05:54.311 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0248501 s, 42.2 MB/s 2021-02-10T11:05:54.312 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:05:54.313 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:05:54.313 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:05:54.647 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:05:54.647 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:05:54.648 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344365/81f30dd4-3513-42f8-82b2-24c0a3e8206b/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:05:55.020 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:05:55.020 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:05:55.020 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0232564 s, 45.1 MB/s 2021-02-10T11:05:55.021 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:05:55.021 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:05:55.022 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:05:55.356 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:05:55.356 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:05:55.357 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344365/81f30dd4-3513-42f8-82b2-24c0a3e8206b/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:05:55.731 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:05:55.731 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:05:55.731 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0252307 s, 41.6 MB/s 2021-02-10T11:05:55.732 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:05:55.732 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:05:55.733 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:05:56.068 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:05:56.069 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:05:56.069 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344365/81f30dd4-3513-42f8-82b2-24c0a3e8206b/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:05:56.445 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:05:56.445 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:05:56.445 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0261394 s, 40.1 MB/s 2021-02-10T11:05:56.446 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000344366 2021-02-10T11:05:56.875 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:56.860 7f1ebe921700 1 Processor -- start 2021-02-10T11:05:56.876 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:56.860 7f1ebe921700 1 -- start start 2021-02-10T11:05:56.876 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:56.861 7f1ebe921700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f1eb81b0870 0x7f1eb80f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:56.876 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:56.861 7f1ebe921700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1eb80efe00 0x7f1eb80f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:56.876 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:56.861 7f1ebe921700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1eb81b0c70 con 0x7f1eb81b0470 2021-02-10T11:05:56.876 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:56.861 7f1ebe921700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1eb80f0b50 con 0x7f1eb81b0870 2021-02-10T11:05:56.876 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:56.861 7f1ebe921700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1eb80f0820 con 0x7f1eb80efe00 2021-02-10T11:05:56.876 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:56.861 7f1eb77fe700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f1eb81b0870 0x7f1eb80f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:56.876 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:56.861 7f1eb77fe700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f1eb81b0870 0x7f1eb80f20e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:50968/0 (socket says 172.21.2.102:50968) 2021-02-10T11:05:56.876 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:56.861 7f1eb77fe700 1 -- 172.21.2.102:0/2725604465 learned_addr learned my addr 172.21.2.102:0/2725604465 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:05:56.876 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:56.861 7f1ebcf3e700 1 --2- 172.21.2.102:0/2725604465 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1eb80efe00 0x7f1eb80f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:56.877 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:56.862 7f1eb77fe700 1 -- 172.21.2.102:0/2725604465 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1eb80efe00 msgr2=0x7f1eb80f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:05:56.877 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:56.862 7f1eb77fe700 1 --2- 172.21.2.102:0/2725604465 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1eb80efe00 0x7f1eb80f0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:56.877 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:56.862 7f1eb77fe700 1 -- 172.21.2.102:0/2725604465 >> v1:172.21.2.113:6789/0 conn(0x7f1eb81b0470 legacy=0x7f1eb80ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:56.877 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:56.862 7f1eb77fe700 1 -- 172.21.2.102:0/2725604465 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1eb80e52e0 con 0x7f1eb81b0870 2021-02-10T11:05:56.877 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:56.862 7f1eb77fe700 1 --2- 172.21.2.102:0/2725604465 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1eb81b0870 0x7f1eb80f20e0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 rx=0x7f1eac006700 tx=0x7f1eac009b80).ready entity=mon.0 client_cookie=9b99fbadc64f3573 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:05:56.877 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:56.862 7f1eb6ffd700 1 -- 172.21.2.102:0/2725604465 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1eac00c0c0 con 0x7f1eb81b0870 2021-02-10T11:05:56.877 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:56.862 7f1eb6ffd700 1 -- 172.21.2.102:0/2725604465 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1eac00c340 con 0x7f1eb81b0870 2021-02-10T11:05:56.877 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:56.862 7f1eb6ffd700 1 -- 172.21.2.102:0/2725604465 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1eac00c860 con 0x7f1eb81b0870 2021-02-10T11:05:56.877 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:56.862 7f1ebe921700 1 -- 172.21.2.102:0/2725604465 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1eb81b0870 msgr2=0x7f1eb80f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:56.877 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:56.862 7f1ebe921700 1 --2- 172.21.2.102:0/2725604465 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1eb81b0870 0x7f1eb80f20e0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 rx=0x7f1eac006700 tx=0x7f1eac009b80).stop 2021-02-10T11:05:56.877 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:56.863 7f1ebe921700 1 -- 172.21.2.102:0/2725604465 shutdown_connections 2021-02-10T11:05:56.877 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:56.863 7f1ebe921700 1 --2- 172.21.2.102:0/2725604465 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1eb80efe00 0x7f1eb80f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:56.877 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:56.863 7f1ebe921700 1 --2- 172.21.2.102:0/2725604465 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1eb81b0870 0x7f1eb80f20e0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:56.878 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:56.863 7f1ebe921700 1 -- 172.21.2.102:0/2725604465 >> 172.21.2.102:0/2725604465 conn(0x7f1eb80ce3e0 msgr2=0x7f1eb80e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:05:56.878 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:56.863 7f1ebe921700 1 -- 172.21.2.102:0/2725604465 shutdown_connections 2021-02-10T11:05:56.878 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:56.863 7f1ebe921700 1 -- 172.21.2.102:0/2725604465 wait complete. 2021-02-10T11:05:56.879 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:56.878 7f1ebe921700 1 Processor -- start 2021-02-10T11:05:56.880 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:56.878 7f1ebe921700 1 -- start start 2021-02-10T11:05:56.880 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:56.878 7f1ebe921700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1eb808ece0 0x7f1eb80a6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:56.880 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:56.878 7f1ebe921700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1eb80a6860 0x7f1eb80a6c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:56.880 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:56.878 7f1ebe921700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1eb80a7280 0x7f1eb80a4590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:56.880 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:56.878 7f1ebe921700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1eb80a5cc0 con 0x7f1eb80a7280 2021-02-10T11:05:56.880 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:56.878 7f1ebe921700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1eb808f0e0 con 0x7f1eb80a6860 2021-02-10T11:05:56.880 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:56.878 7f1ebe921700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1eb808f2c0 con 0x7f1eb808ece0 2021-02-10T11:05:56.880 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:56.878 7f1ebcf3e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1eb80a7280 0x7f1eb80a4590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:56.880 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:56.878 7f1ebcf3e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1eb80a7280 0x7f1eb80a4590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:50976/0 (socket says 172.21.2.102:50976) 2021-02-10T11:05:56.880 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:56.878 7f1ebcf3e700 1 -- 172.21.2.102:0/2654040397 learned_addr learned my addr 172.21.2.102:0/2654040397 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:05:56.880 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:56.878 7f1eb77fe700 1 --2- 172.21.2.102:0/2654040397 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1eb80a6860 0x7f1eb80a6c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:56.880 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:56.879 7f1eb7fff700 1 --2- 172.21.2.102:0/2654040397 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1eb808ece0 0x7f1eb80a6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:56.881 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:56.879 7f1ebcf3e700 1 -- 172.21.2.102:0/2654040397 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1eb808ece0 msgr2=0x7f1eb80a6240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:56.881 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:56.879 7f1ebcf3e700 1 --2- 172.21.2.102:0/2654040397 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1eb808ece0 0x7f1eb80a6240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:56.881 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:56.879 7f1ebcf3e700 1 -- 172.21.2.102:0/2654040397 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1eb80a6860 msgr2=0x7f1eb80a6c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:56.881 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:56.879 7f1ebcf3e700 1 --2- 172.21.2.102:0/2654040397 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1eb80a6860 0x7f1eb80a6c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:56.881 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:56.879 7f1ebcf3e700 1 -- 172.21.2.102:0/2654040397 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1eb80a4bb0 con 0x7f1eb80a7280 2021-02-10T11:05:56.881 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:56.879 7f1ebcf3e700 1 --2- 172.21.2.102:0/2654040397 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1eb80a7280 0x7f1eb80a4590 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 rx=0x7f1eb000b970 tx=0x7f1eb000bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:05:56.881 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:56.879 7f1eb4ff9700 1 -- 172.21.2.102:0/2654040397 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1eb0000f90 con 0x7f1eb80a7280 2021-02-10T11:05:56.881 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:56.879 7f1eb4ff9700 1 -- 172.21.2.102:0/2654040397 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1eb0003030 con 0x7f1eb80a7280 2021-02-10T11:05:56.881 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:56.879 7f1eb4ff9700 1 -- 172.21.2.102:0/2654040397 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1eb0001400 con 0x7f1eb80a7280 2021-02-10T11:05:56.881 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:56.879 7f1ebe921700 1 -- 172.21.2.102:0/2654040397 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1eb80a4bb0 con 0x7f1eb80a7280 2021-02-10T11:05:56.881 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:56.879 7f1ebe921700 1 -- 172.21.2.102:0/2654040397 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1eb80a5140 con 0x7f1eb80a7280 2021-02-10T11:05:56.881 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:56.880 7f1eb4ff9700 1 -- 172.21.2.102:0/2654040397 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1eb00056b0 con 0x7f1eb80a7280 2021-02-10T11:05:56.881 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:56.880 7f1eb4ff9700 1 --2- 172.21.2.102:0/2654040397 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1e980186d0 0x7f1e9801abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:56.882 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:56.880 7f1eb4ff9700 1 -- 172.21.2.102:0/2654040397 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f1eb0032a20 con 0x7f1eb80a7280 2021-02-10T11:05:56.882 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:56.880 7f1eb7fff700 1 --2- 172.21.2.102:0/2654040397 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1e980186d0 0x7f1e9801abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:56.882 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:56.880 7f1eb7fff700 1 --2- 172.21.2.102:0/2654040397 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1e980186d0 0x7f1e9801abd0 crc :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:05:56.910 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:56.910 7f1ebe921700 1 -- 172.21.2.102:0/2654040397 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1eb8105ae0 con 0x7f1eb80a7280 2021-02-10T11:05:56.911 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:56.913 7f1eb4ff9700 1 -- 172.21.2.102:0/2654040397 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f1eb0003010 con 0x7f1eb80a7280 2021-02-10T11:05:57.037 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:57.038 7f1ebe921700 1 -- 172.21.2.102:0/2654040397 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344366", "target": ["mgr", ""]}) v1 -- 0x7f1eb80a5cc0 con 0x7f1e980186d0 2021-02-10T11:05:57.061 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:57.063 7f1eb4ff9700 1 -- 172.21.2.102:0/2654040397 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f1ea8006a90 con 0x7f1e980186d0 2021-02-10T11:05:57.070 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:57.071 7f1ebe921700 1 -- 172.21.2.102:0/2654040397 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1e980186d0 msgr2=0x7f1e9801abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:57.070 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:57.071 7f1ebe921700 1 --2- 172.21.2.102:0/2654040397 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1e980186d0 0x7f1e9801abd0 crc :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:57.070 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:57.071 7f1ebe921700 1 -- 172.21.2.102:0/2654040397 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1eb80a7280 msgr2=0x7f1eb80a4590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:57.070 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:57.071 7f1ebe921700 1 --2- 172.21.2.102:0/2654040397 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1eb80a7280 0x7f1eb80a4590 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 rx=0x7f1eb000b970 tx=0x7f1eb000bd10).stop 2021-02-10T11:05:57.071 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:57.071 7f1ebe921700 1 -- 172.21.2.102:0/2654040397 shutdown_connections 2021-02-10T11:05:57.071 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:57.071 7f1ebe921700 1 --2- 172.21.2.102:0/2654040397 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1eb80a7280 0x7f1eb80a4590 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:57.071 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:57.071 7f1ebe921700 1 --2- 172.21.2.102:0/2654040397 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1e980186d0 0x7f1e9801abd0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:57.071 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:57.071 7f1ebe921700 1 --2- 172.21.2.102:0/2654040397 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1eb80a6860 0x7f1eb80a6c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:57.071 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:57.071 7f1ebe921700 1 --2- 172.21.2.102:0/2654040397 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1eb808ece0 0x7f1eb80a6240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:57.071 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:57.071 7f1ebe921700 1 -- 172.21.2.102:0/2654040397 >> 172.21.2.102:0/2654040397 conn(0x7f1eb808ffe0 msgr2=0x7f1eb80bf050 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:05:57.071 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:57.072 7f1ebe921700 1 -- 172.21.2.102:0/2654040397 shutdown_connections 2021-02-10T11:05:57.071 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:57.072 7f1ebe921700 1 -- 172.21.2.102:0/2654040397 wait complete. 2021-02-10T11:05:57.078 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000344366 2021-02-10T11:05:57.512 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:57.494 7f49a68c8700 1 Processor -- start 2021-02-10T11:05:57.513 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:57.494 7f49a68c8700 1 -- start start 2021-02-10T11:05:57.513 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:57.494 7f49a68c8700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f49a01b0470 0x7f49a01b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:57.513 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:57.494 7f49a68c8700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f49a00f20e0 0x7f49a00f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:57.513 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:57.494 7f49a68c8700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49a00f0b50 con 0x7f49a00ef610 2021-02-10T11:05:57.513 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:57.494 7f49a68c8700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49a00f0820 con 0x7f49a01b0470 2021-02-10T11:05:57.513 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:57.494 7f49a68c8700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49a00f24e0 con 0x7f49a00f20e0 2021-02-10T11:05:57.513 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:57.494 7f499ffff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f49a01b0470 0x7f49a01b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:57.513 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:57.494 7f499f7fe700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f49a00ef610 0x7f49a00efa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:43206/0 (socket says 172.21.2.102:43206) 2021-02-10T11:05:57.513 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:57.494 7f499f7fe700 1 -- 172.21.2.102:0/3244865617 learned_addr learned my addr 172.21.2.102:0/3244865617 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:05:57.513 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:57.495 7f49a4ee5700 1 --2- 172.21.2.102:0/3244865617 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f49a00f20e0 0x7f49a00f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:57.514 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:57.495 7f499effd700 1 -- 172.21.2.102:0/3244865617 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3463360999 0 0) 0x7f49940010b0 con 0x7f49a00ef610 2021-02-10T11:05:57.514 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:57.495 7f499effd700 1 -- 172.21.2.102:0/3244865617 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4988002fe0 con 0x7f49a00ef610 2021-02-10T11:05:57.514 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:57.495 7f499ffff700 1 -- 172.21.2.102:0/3244865617 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f49a00f20e0 msgr2=0x7f49a00f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:05:57.514 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:57.495 7f499ffff700 1 --2- 172.21.2.102:0/3244865617 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f49a00f20e0 0x7f49a00f0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:57.514 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:57.495 7f499ffff700 1 -- 172.21.2.102:0/3244865617 >> v1:172.21.2.102:6789/0 conn(0x7f49a00ef610 legacy=0x7f49a00efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:57.514 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:57.495 7f499ffff700 1 -- 172.21.2.102:0/3244865617 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49a00e52e0 con 0x7f49a01b0470 2021-02-10T11:05:57.514 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:57.495 7f499ffff700 1 --2- 172.21.2.102:0/3244865617 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f49a01b0470 0x7f49a01b0870 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 rx=0x7f4990006650 tx=0x7f4990009a80).ready entity=mon.0 client_cookie=9437293c920e4490 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:05:57.514 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:57.495 7f499effd700 1 -- 172.21.2.102:0/3244865617 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f499000c0c0 con 0x7f49a01b0470 2021-02-10T11:05:57.514 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:57.495 7f499effd700 1 -- 172.21.2.102:0/3244865617 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f499000c340 con 0x7f49a01b0470 2021-02-10T11:05:57.514 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:57.495 7f499effd700 1 -- 172.21.2.102:0/3244865617 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f499000c860 con 0x7f49a01b0470 2021-02-10T11:05:57.514 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:57.495 7f49a68c8700 1 -- 172.21.2.102:0/3244865617 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f49a01b0470 msgr2=0x7f49a01b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:57.515 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:57.495 7f49a68c8700 1 --2- 172.21.2.102:0/3244865617 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f49a01b0470 0x7f49a01b0870 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 rx=0x7f4990006650 tx=0x7f4990009a80).stop 2021-02-10T11:05:57.515 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:57.496 7f49a68c8700 1 -- 172.21.2.102:0/3244865617 shutdown_connections 2021-02-10T11:05:57.515 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:57.496 7f49a68c8700 1 --2- 172.21.2.102:0/3244865617 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f49a00f20e0 0x7f49a00f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:57.515 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:57.496 7f49a68c8700 1 --2- 172.21.2.102:0/3244865617 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f49a01b0470 0x7f49a01b0870 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:57.515 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:57.496 7f49a68c8700 1 -- 172.21.2.102:0/3244865617 >> 172.21.2.102:0/3244865617 conn(0x7f49a00ce3e0 msgr2=0x7f49a00e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:05:57.515 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:57.496 7f49a68c8700 1 -- 172.21.2.102:0/3244865617 shutdown_connections 2021-02-10T11:05:57.515 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:57.496 7f49a68c8700 1 -- 172.21.2.102:0/3244865617 wait complete. 2021-02-10T11:05:57.515 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:57.511 7f49a68c8700 1 Processor -- start 2021-02-10T11:05:57.515 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:57.511 7f49a68c8700 1 -- start start 2021-02-10T11:05:57.515 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:57.511 7f49a68c8700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f49a01a42e0 0x7f49a009ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:57.515 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:57.511 7f49a68c8700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f49a01a46e0 0x7f49a009f310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:57.516 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:57.511 7f49a68c8700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f49a008ad30 0x7f49a008b130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:57.516 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:57.511 7f49a68c8700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49a008a7b0 con 0x7f49a01a46e0 2021-02-10T11:05:57.516 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:57.511 7f49a68c8700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49a00a65f0 con 0x7f49a01a42e0 2021-02-10T11:05:57.516 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:57.511 7f49a68c8700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49a008a430 con 0x7f49a008ad30 2021-02-10T11:05:57.516 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:57.512 7f499f7fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f49a01a46e0 0x7f49a009f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:57.516 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:57.512 7f499f7fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f49a01a46e0 0x7f49a009f310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:50992/0 (socket says 172.21.2.102:50992) 2021-02-10T11:05:57.516 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:57.512 7f499f7fe700 1 -- 172.21.2.102:0/3605175055 learned_addr learned my addr 172.21.2.102:0/3605175055 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:05:57.516 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:57.512 7f499ffff700 1 --2- 172.21.2.102:0/3605175055 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f49a01a42e0 0x7f49a009ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:57.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:57.512 7f49a4ee5700 1 --2- 172.21.2.102:0/3605175055 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f49a008ad30 0x7f49a008b130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:57.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:57.512 7f499f7fe700 1 -- 172.21.2.102:0/3605175055 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f49a008ad30 msgr2=0x7f49a008b130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:57.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:57.512 7f499f7fe700 1 --2- 172.21.2.102:0/3605175055 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f49a008ad30 0x7f49a008b130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:57.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:57.512 7f499f7fe700 1 -- 172.21.2.102:0/3605175055 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f49a01a42e0 msgr2=0x7f49a009ecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:57.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:57.512 7f499f7fe700 1 --2- 172.21.2.102:0/3605175055 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f49a01a42e0 0x7f49a009ecf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:57.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:57.512 7f499f7fe700 1 -- 172.21.2.102:0/3605175055 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49a009fa50 con 0x7f49a01a46e0 2021-02-10T11:05:57.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:57.512 7f499ffff700 1 --2- 172.21.2.102:0/3605175055 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f49a01a42e0 0x7f49a009ecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:05:57.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:57.513 7f499f7fe700 1 --2- 172.21.2.102:0/3605175055 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f49a01a46e0 0x7f49a009f310 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 rx=0x7f4994001c60 tx=0x7f499400a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:05:57.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:57.513 7f499cff9700 1 -- 172.21.2.102:0/3605175055 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f499400acc0 con 0x7f49a01a46e0 2021-02-10T11:05:57.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:57.513 7f499cff9700 1 -- 172.21.2.102:0/3605175055 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4994004140 con 0x7f49a01a46e0 2021-02-10T11:05:57.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:57.513 7f499cff9700 1 -- 172.21.2.102:0/3605175055 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4994004620 con 0x7f49a01a46e0 2021-02-10T11:05:57.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:57.513 7f49a68c8700 1 -- 172.21.2.102:0/3605175055 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49a009fa50 con 0x7f49a01a46e0 2021-02-10T11:05:57.518 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:57.513 7f49a68c8700 1 -- 172.21.2.102:0/3605175055 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49a008ba40 con 0x7f49a01a46e0 2021-02-10T11:05:57.518 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:57.513 7f499cff9700 1 -- 172.21.2.102:0/3605175055 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4994004320 con 0x7f49a01a46e0 2021-02-10T11:05:57.518 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:57.514 7f499cff9700 1 --2- 172.21.2.102:0/3605175055 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f49800186c0 0x7f498001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:05:57.518 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:57.514 7f499cff9700 1 -- 172.21.2.102:0/3605175055 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f49940323e0 con 0x7f49a01a46e0 2021-02-10T11:05:57.518 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:57.514 7f499ffff700 1 --2- 172.21.2.102:0/3605175055 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f49800186c0 0x7f498001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:05:57.518 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:57.514 7f499ffff700 1 --2- 172.21.2.102:0/3605175055 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f49800186c0 0x7f498001abc0 crc :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:05:57.542 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:57.544 7f49a68c8700 1 -- 172.21.2.102:0/3605175055 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49a0105ae0 con 0x7f49a01a46e0 2021-02-10T11:05:57.544 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:57.546 7f499cff9700 1 -- 172.21.2.102:0/3605175055 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f4994016a90 con 0x7f49a01a46e0 2021-02-10T11:05:57.677 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:57.677 7f49a68c8700 1 -- 172.21.2.102:0/3605175055 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344366", "target": ["mgr", ""]}) v1 -- 0x7f49a008ba40 con 0x7f49800186c0 2021-02-10T11:05:57.678 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:57.679 7f499cff9700 1 -- 172.21.2.102:0/3605175055 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f49900090e0 con 0x7f49800186c0 2021-02-10T11:05:57.678 INFO:teuthology.orchestra.run.gibba002.stdout:/volumes/_nogroup/subvolume_0000000000344366/2e75b09c-3c68-4a91-bd9d-7ea007bafe2d 2021-02-10T11:05:57.680 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:57.681 7f49a68c8700 1 -- 172.21.2.102:0/3605175055 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f49800186c0 msgr2=0x7f498001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:57.680 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:57.681 7f49a68c8700 1 --2- 172.21.2.102:0/3605175055 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f49800186c0 0x7f498001abc0 crc :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:57.680 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:57.681 7f49a68c8700 1 -- 172.21.2.102:0/3605175055 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f49a01a46e0 msgr2=0x7f49a009f310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:05:57.680 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:57.681 7f49a68c8700 1 --2- 172.21.2.102:0/3605175055 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f49a01a46e0 0x7f49a009f310 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 rx=0x7f4994001c60 tx=0x7f499400a8d0).stop 2021-02-10T11:05:57.680 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:57.681 7f49a68c8700 1 -- 172.21.2.102:0/3605175055 shutdown_connections 2021-02-10T11:05:57.681 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:57.681 7f49a68c8700 1 --2- 172.21.2.102:0/3605175055 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f49a008ad30 0x7f49a008b130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:57.681 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:57.681 7f49a68c8700 1 --2- 172.21.2.102:0/3605175055 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f49a01a46e0 0x7f49a009f310 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:57.681 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:57.681 7f49a68c8700 1 --2- 172.21.2.102:0/3605175055 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f49800186c0 0x7f498001abc0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:57.681 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:57.681 7f49a68c8700 1 --2- 172.21.2.102:0/3605175055 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f49a01a42e0 0x7f49a009ecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:05:57.681 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:57.681 7f49a68c8700 1 -- 172.21.2.102:0/3605175055 >> 172.21.2.102:0/3605175055 conn(0x7f49a0095a70 msgr2=0x7f49a00bf050 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:05:57.681 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:57.681 7f49a68c8700 1 -- 172.21.2.102:0/3605175055 shutdown_connections 2021-02-10T11:05:57.682 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:05:57.682 7f49a68c8700 1 -- 172.21.2.102:0/3605175055 wait complete. 2021-02-10T11:05:57.688 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000344366 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000344366/2e75b09c-3c68-4a91-bd9d-7ea007bafe2d 2021-02-10T11:05:57.688 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:05:57.688 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:05:57.688 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:05:58.023 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:05:58.023 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:05:58.023 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344366/2e75b09c-3c68-4a91-bd9d-7ea007bafe2d/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:05:58.401 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:05:58.402 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:05:58.402 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0246837 s, 42.5 MB/s 2021-02-10T11:05:58.403 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:05:58.403 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:05:58.403 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:05:58.737 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:05:58.737 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:05:58.737 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344366/2e75b09c-3c68-4a91-bd9d-7ea007bafe2d/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:05:59.117 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:05:59.117 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:05:59.118 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.028389 s, 36.9 MB/s 2021-02-10T11:05:59.119 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:05:59.119 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:05:59.119 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:05:59.454 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:05:59.454 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:05:59.454 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344366/2e75b09c-3c68-4a91-bd9d-7ea007bafe2d/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:05:59.829 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:05:59.830 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:05:59.830 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0250924 s, 41.8 MB/s 2021-02-10T11:05:59.830 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:05:59.831 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:05:59.831 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:06:00.166 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:06:00.166 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:06:00.166 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344366/2e75b09c-3c68-4a91-bd9d-7ea007bafe2d/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:06:00.540 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:06:00.540 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:06:00.541 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0243876 s, 43.0 MB/s 2021-02-10T11:06:00.541 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:06:00.542 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:06:00.542 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:06:00.876 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:06:00.876 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:06:00.876 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344366/2e75b09c-3c68-4a91-bd9d-7ea007bafe2d/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:06:01.252 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:06:01.252 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:06:01.253 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0242898 s, 43.2 MB/s 2021-02-10T11:06:01.254 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:06:01.254 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:06:01.254 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:06:01.590 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:06:01.591 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:06:01.591 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344366/2e75b09c-3c68-4a91-bd9d-7ea007bafe2d/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:06:01.965 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:06:01.965 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:06:01.965 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0233948 s, 44.8 MB/s 2021-02-10T11:06:01.967 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:06:01.967 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:06:01.967 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:06:02.303 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:06:02.303 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:06:02.303 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344366/2e75b09c-3c68-4a91-bd9d-7ea007bafe2d/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:06:02.680 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:06:02.681 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:06:02.681 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0263795 s, 39.7 MB/s 2021-02-10T11:06:02.682 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:06:02.683 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:06:02.683 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:06:03.019 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:06:03.019 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:06:03.019 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344366/2e75b09c-3c68-4a91-bd9d-7ea007bafe2d/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:06:03.392 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:06:03.392 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:06:03.392 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0243453 s, 43.1 MB/s 2021-02-10T11:06:03.393 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:06:03.393 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:06:03.393 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:06:03.729 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:06:03.729 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:06:03.729 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344366/2e75b09c-3c68-4a91-bd9d-7ea007bafe2d/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:06:04.106 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:06:04.106 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:06:04.106 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0259818 s, 40.4 MB/s 2021-02-10T11:06:04.107 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:06:04.107 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:06:04.107 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:06:04.441 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:06:04.442 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:06:04.442 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344366/2e75b09c-3c68-4a91-bd9d-7ea007bafe2d/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:06:04.815 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:06:04.816 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:06:04.816 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0228725 s, 45.8 MB/s 2021-02-10T11:06:04.817 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000344367 2021-02-10T11:06:05.247 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:05.231 7ff1c2353700 1 Processor -- start 2021-02-10T11:06:05.247 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:05.231 7ff1c2353700 1 -- start start 2021-02-10T11:06:05.247 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:05.231 7ff1c2353700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7ff1bc1b0470 0x7ff1bc1b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:06:05.247 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:05.231 7ff1c2353700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff1bc0f20e0 0x7ff1bc0f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:06:05.247 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:05.231 7ff1c2353700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1bc0f0b50 con 0x7ff1bc0ef610 2021-02-10T11:06:05.248 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:05.231 7ff1c2353700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1bc0f0820 con 0x7ff1bc1b0470 2021-02-10T11:06:05.248 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:05.231 7ff1c2353700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1bc0f24e0 con 0x7ff1bc0f20e0 2021-02-10T11:06:05.248 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:05.232 7ff1bb7fe700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7ff1bc0ef610 0x7ff1bc0efa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:43222/0 (socket says 172.21.2.102:43222) 2021-02-10T11:06:05.248 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:05.232 7ff1bb7fe700 1 -- 172.21.2.102:0/1273044972 learned_addr learned my addr 172.21.2.102:0/1273044972 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:06:05.248 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:05.232 7ff1bbfff700 1 --2- 172.21.2.102:0/1273044972 >> v2:172.21.2.113:3300/0 conn(0x7ff1bc1b0470 0x7ff1bc1b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:06:05.248 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:05.232 7ff1c0970700 1 --2- 172.21.2.102:0/1273044972 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff1bc0f20e0 0x7ff1bc0f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:06:05.248 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:05.232 7ff1baffd700 1 -- 172.21.2.102:0/1273044972 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3984733722 0 0) 0x7ff1b00009b0 con 0x7ff1bc0ef610 2021-02-10T11:06:05.248 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:05.232 7ff1baffd700 1 -- 172.21.2.102:0/1273044972 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1a4002fe0 con 0x7ff1bc0ef610 2021-02-10T11:06:05.248 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:05.232 7ff1baffd700 1 -- 172.21.2.102:0/1273044972 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4244074224 0 0) 0x7ff1b0001010 con 0x7ff1bc0ef610 2021-02-10T11:06:05.248 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:05.232 7ff1baffd700 1 -- 172.21.2.102:0/1273044972 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff1bc0f20e0 msgr2=0x7ff1bc0f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:06:05.248 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:05.232 7ff1baffd700 1 --2- 172.21.2.102:0/1273044972 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff1bc0f20e0 0x7ff1bc0f0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:06:05.248 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:05.232 7ff1baffd700 1 -- 172.21.2.102:0/1273044972 >> v2:172.21.2.113:3300/0 conn(0x7ff1bc1b0470 msgr2=0x7ff1bc1b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:06:05.249 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:05.232 7ff1baffd700 1 --2- 172.21.2.102:0/1273044972 >> v2:172.21.2.113:3300/0 conn(0x7ff1bc1b0470 0x7ff1bc1b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:06:05.249 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:05.232 7ff1baffd700 1 -- 172.21.2.102:0/1273044972 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1bc0e52e0 con 0x7ff1bc0ef610 2021-02-10T11:06:05.249 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:05.232 7ff1baffd700 1 -- 172.21.2.102:0/1273044972 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7ff1b0001530 con 0x7ff1bc0ef610 2021-02-10T11:06:05.249 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:05.232 7ff1baffd700 1 -- 172.21.2.102:0/1273044972 >> v1:172.21.2.102:6789/0 conn(0x7ff1bc0ef610 legacy=0x7ff1bc0efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:06:05.249 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:05.232 7ff1baffd700 1 --2- 172.21.2.102:0/1273044972 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff1a4003af0 0x7ff1a4005f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:06:05.249 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:05.232 7ff1baffd700 1 --2- 172.21.2.102:0/1273044972 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff1a404a830 0x7ff1a404cd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:06:05.249 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:05.232 7ff1baffd700 1 --2- 172.21.2.102:0/1273044972 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff1a404d5c0 0x7ff1a404fae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:06:05.249 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:05.232 7ff1baffd700 1 -- 172.21.2.102:0/1273044972 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1a4050390 con 0x7ff1a4003af0 2021-02-10T11:06:05.249 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:05.232 7ff1baffd700 1 -- 172.21.2.102:0/1273044972 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1a40505b0 con 0x7ff1a404d5c0 2021-02-10T11:06:05.249 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:05.232 7ff1baffd700 1 -- 172.21.2.102:0/1273044972 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1a40507d0 con 0x7ff1a404a830 2021-02-10T11:06:05.249 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:05.232 7ff1bbfff700 1 --2- 172.21.2.102:0/1273044972 >> v2:172.21.2.113:3300/0 conn(0x7ff1bc1b0470 0x7ff1bc1b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:06:05.249 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:05.233 7ff1bbfff700 1 --2- 172.21.2.102:0/1273044972 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff1a4003af0 0x7ff1a4005f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:06:05.249 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:05.233 7ff1bb7fe700 1 --2- 172.21.2.102:0/1273044972 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff1a404a830 0x7ff1a404cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:06:05.249 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:05.233 7ff1c0970700 1 --2- 172.21.2.102:0/1273044972 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff1a404d5c0 0x7ff1a404fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:06:05.250 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:05.233 7ff1bbfff700 1 -- 172.21.2.102:0/1273044972 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff1a404a830 msgr2=0x7ff1a404cd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:06:05.250 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:05.233 7ff1bbfff700 1 --2- 172.21.2.102:0/1273044972 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff1a404a830 0x7ff1a404cd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:06:05.250 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:05.233 7ff1bbfff700 1 -- 172.21.2.102:0/1273044972 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff1a404d5c0 msgr2=0x7ff1a404fae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:06:05.250 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:05.233 7ff1bbfff700 1 --2- 172.21.2.102:0/1273044972 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff1a404d5c0 0x7ff1a404fae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:06:05.250 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:05.233 7ff1bbfff700 1 -- 172.21.2.102:0/1273044972 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1a4050a70 con 0x7ff1a4003af0 2021-02-10T11:06:05.250 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:05.233 7ff1c0970700 1 --2- 172.21.2.102:0/1273044972 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff1a404d5c0 0x7ff1a404fae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:06:05.250 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:05.233 7ff1bbfff700 1 --2- 172.21.2.102:0/1273044972 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff1a4003af0 0x7ff1a4005f80 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 rx=0x7ff1ac000b80 tx=0x7ff1ac009e80).ready entity=mon.0 client_cookie=6b987b76a1430a5d server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:06:05.250 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:05.233 7ff1baffd700 1 -- 172.21.2.102:0/1273044972 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff1ac00c5f0 con 0x7ff1a4003af0 2021-02-10T11:06:05.250 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:05.233 7ff1baffd700 1 -- 172.21.2.102:0/1273044972 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1ac00c8f0 con 0x7ff1a4003af0 2021-02-10T11:06:05.250 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:05.233 7ff1baffd700 1 -- 172.21.2.102:0/1273044972 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff1ac007070 con 0x7ff1a4003af0 2021-02-10T11:06:05.250 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:05.233 7ff1c2353700 1 -- 172.21.2.102:0/1273044972 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff1a4003af0 msgr2=0x7ff1a4005f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:06:05.250 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:05.233 7ff1c2353700 1 --2- 172.21.2.102:0/1273044972 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff1a4003af0 0x7ff1a4005f80 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 rx=0x7ff1ac000b80 tx=0x7ff1ac009e80).stop 2021-02-10T11:06:05.250 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:05.233 7ff1c0970700 1 -- 172.21.2.102:0/1273044972 reap_dead start 2021-02-10T11:06:05.251 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:05.234 7ff1c2353700 1 -- 172.21.2.102:0/1273044972 shutdown_connections 2021-02-10T11:06:05.251 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:05.234 7ff1c2353700 1 -- 172.21.2.102:0/1273044972 >> 172.21.2.102:0/1273044972 conn(0x7ff1bc0ce3e0 msgr2=0x7ff1bc0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:06:05.251 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:05.234 7ff1c2353700 1 -- 172.21.2.102:0/1273044972 shutdown_connections 2021-02-10T11:06:05.251 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:05.234 7ff1c2353700 1 -- 172.21.2.102:0/1273044972 wait complete. 2021-02-10T11:06:05.251 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:05.249 7ff1c2353700 1 Processor -- start 2021-02-10T11:06:05.251 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:05.249 7ff1c2353700 1 -- start start 2021-02-10T11:06:05.252 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:05.249 7ff1c2353700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff1bc09b930 0x7ff1bc0b7c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:06:05.253 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:05.249 7ff1c2353700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff1bc0b8410 0x7ff1bc0b8810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:06:05.253 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:05.249 7ff1c2353700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff1bc0c1f80 0x7ff1bc0c2380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:06:05.253 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:05.249 7ff1c2353700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1bc0c1a00 con 0x7ff1bc0c1f80 2021-02-10T11:06:05.253 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:05.249 7ff1c2353700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1bc0b8e30 con 0x7ff1bc09b930 2021-02-10T11:06:05.253 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:05.249 7ff1c2353700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1bc0b9050 con 0x7ff1bc0b8410 2021-02-10T11:06:05.253 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:05.250 7ff1c0970700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff1bc0c1f80 0x7ff1bc0c2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:06:05.253 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:05.250 7ff1c0970700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff1bc0c1f80 0x7ff1bc0c2380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:51012/0 (socket says 172.21.2.102:51012) 2021-02-10T11:06:05.253 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:05.250 7ff1c0970700 1 -- 172.21.2.102:0/3561284981 learned_addr learned my addr 172.21.2.102:0/3561284981 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:06:05.253 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:05.250 7ff1bbfff700 1 --2- 172.21.2.102:0/3561284981 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff1bc09b930 0x7ff1bc0b7c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:06:05.253 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:05.250 7ff1bb7fe700 1 --2- 172.21.2.102:0/3561284981 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff1bc0b8410 0x7ff1bc0b8810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:06:05.253 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:05.250 7ff1c0970700 1 -- 172.21.2.102:0/3561284981 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff1bc0b8410 msgr2=0x7ff1bc0b8810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:06:05.253 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:05.250 7ff1c0970700 1 --2- 172.21.2.102:0/3561284981 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff1bc0b8410 0x7ff1bc0b8810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:06:05.254 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:05.250 7ff1c0970700 1 -- 172.21.2.102:0/3561284981 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff1bc09b930 msgr2=0x7ff1bc0b7c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:06:05.254 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:05.250 7ff1c0970700 1 --2- 172.21.2.102:0/3561284981 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff1bc09b930 0x7ff1bc0b7c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:06:05.254 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:05.250 7ff1c0970700 1 -- 172.21.2.102:0/3561284981 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1bc0c29a0 con 0x7ff1bc0c1f80 2021-02-10T11:06:05.254 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:05.250 7ff1bbfff700 1 --2- 172.21.2.102:0/3561284981 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff1bc09b930 0x7ff1bc0b7c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:06:05.254 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:05.250 7ff1bb7fe700 1 --2- 172.21.2.102:0/3561284981 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff1bc0b8410 0x7ff1bc0b8810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:06:05.254 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:05.250 7ff1c0970700 1 --2- 172.21.2.102:0/3561284981 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff1bc0c1f80 0x7ff1bc0c2380 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 rx=0x7ff1b4009a00 tx=0x7ff1b400bd00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:06:05.254 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:05.250 7ff1b8ff9700 1 -- 172.21.2.102:0/3561284981 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff1b40069e0 con 0x7ff1bc0c1f80 2021-02-10T11:06:05.254 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:05.250 7ff1b8ff9700 1 -- 172.21.2.102:0/3561284981 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1b40011f0 con 0x7ff1bc0c1f80 2021-02-10T11:06:05.254 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:05.250 7ff1b8ff9700 1 -- 172.21.2.102:0/3561284981 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff1b4001710 con 0x7ff1bc0c1f80 2021-02-10T11:06:05.254 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:05.250 7ff1c2353700 1 -- 172.21.2.102:0/3561284981 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1bc0c29a0 con 0x7ff1bc0c1f80 2021-02-10T11:06:05.254 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:05.250 7ff1c2353700 1 -- 172.21.2.102:0/3561284981 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1bc0c2f30 con 0x7ff1bc0c1f80 2021-02-10T11:06:05.254 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:05.251 7ff1b8ff9700 1 -- 172.21.2.102:0/3561284981 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ff1b40069e0 con 0x7ff1bc0c1f80 2021-02-10T11:06:05.255 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:05.251 7ff1b8ff9700 1 --2- 172.21.2.102:0/3561284981 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff19c0186c0 0x7ff19c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:06:05.255 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:05.251 7ff1b8ff9700 1 -- 172.21.2.102:0/3561284981 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7ff1b40328f0 con 0x7ff1bc0c1f80 2021-02-10T11:06:05.255 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:05.251 7ff1bbfff700 1 --2- 172.21.2.102:0/3561284981 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff19c0186c0 0x7ff19c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:06:05.255 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:05.251 7ff1bbfff700 1 --2- 172.21.2.102:0/3561284981 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff19c0186c0 0x7ff19c01abc0 crc :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:06:05.280 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:05.281 7ff1c2353700 1 -- 172.21.2.102:0/3561284981 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1bc105b30 con 0x7ff1bc0c1f80 2021-02-10T11:06:05.283 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:05.284 7ff1b8ff9700 1 -- 172.21.2.102:0/3561284981 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ff1b401f690 con 0x7ff1bc0c1f80 2021-02-10T11:06:05.408 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:05.410 7ff1c2353700 1 -- 172.21.2.102:0/3561284981 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344367", "target": ["mgr", ""]}) v1 -- 0x7ff1bc0c2f30 con 0x7ff19c0186c0 2021-02-10T11:06:05.432 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:05.434 7ff1b8ff9700 1 -- 172.21.2.102:0/3561284981 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7ff1ac006550 con 0x7ff19c0186c0 2021-02-10T11:06:05.441 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:05.442 7ff1c2353700 1 -- 172.21.2.102:0/3561284981 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff19c0186c0 msgr2=0x7ff19c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:06:05.441 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:05.442 7ff1c2353700 1 --2- 172.21.2.102:0/3561284981 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff19c0186c0 0x7ff19c01abc0 crc :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:06:05.441 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:05.442 7ff1c2353700 1 -- 172.21.2.102:0/3561284981 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff1bc0c1f80 msgr2=0x7ff1bc0c2380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:06:05.441 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:05.442 7ff1c2353700 1 --2- 172.21.2.102:0/3561284981 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff1bc0c1f80 0x7ff1bc0c2380 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 rx=0x7ff1b4009a00 tx=0x7ff1b400bd00).stop 2021-02-10T11:06:05.441 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:05.443 7ff1c2353700 1 -- 172.21.2.102:0/3561284981 shutdown_connections 2021-02-10T11:06:05.441 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:05.443 7ff1c2353700 1 --2- 172.21.2.102:0/3561284981 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff1bc0c1f80 0x7ff1bc0c2380 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:06:05.441 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:05.443 7ff1c2353700 1 --2- 172.21.2.102:0/3561284981 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff1bc0b8410 0x7ff1bc0b8810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:06:05.442 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:05.443 7ff1c2353700 1 --2- 172.21.2.102:0/3561284981 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff19c0186c0 0x7ff19c01abc0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:06:05.442 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:05.443 7ff1c2353700 1 --2- 172.21.2.102:0/3561284981 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff1bc09b930 0x7ff1bc0b7c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:06:05.442 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:05.443 7ff1c2353700 1 -- 172.21.2.102:0/3561284981 >> 172.21.2.102:0/3561284981 conn(0x7ff1bc094e10 msgr2=0x7ff1bc0a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:06:05.442 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:05.443 7ff1c2353700 1 -- 172.21.2.102:0/3561284981 shutdown_connections 2021-02-10T11:06:05.442 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:05.443 7ff1c2353700 1 -- 172.21.2.102:0/3561284981 wait complete. 2021-02-10T11:06:05.450 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000344367 2021-02-10T11:06:05.880 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:05.865 7f87214d7700 1 Processor -- start 2021-02-10T11:06:05.880 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:05.865 7f87214d7700 1 -- start start 2021-02-10T11:06:05.880 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:05.865 7f87214d7700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f871c1b0470 0x7f871c1b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:06:05.880 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:05.865 7f87214d7700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f871c0ef610 0x7f871c0efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:06:05.880 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:05.865 7f87214d7700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f871c0f0030 0x7f871c0f0430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:06:05.880 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:05.865 7f87214d7700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f871c0e5ea0 con 0x7f871c0f0030 2021-02-10T11:06:05.880 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:05.865 7f87214d7700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f871c0f0b50 con 0x7f871c0ef610 2021-02-10T11:06:05.880 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:05.865 7f87214d7700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f871c0e52e0 con 0x7f871c1b0470 2021-02-10T11:06:05.880 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:05.865 7f871bac0700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f871c0f0030 0x7f871c0f0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:06:05.880 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:05.865 7f871bac0700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f871c0f0030 0x7f871c0f0430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:51016/0 (socket says 172.21.2.102:51016) 2021-02-10T11:06:05.880 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:05.866 7f871bac0700 1 -- 172.21.2.102:0/26167069 learned_addr learned my addr 172.21.2.102:0/26167069 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:06:05.881 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:05.866 7f871aabe700 1 --2- 172.21.2.102:0/26167069 >> v2:172.21.2.113:3300/0 conn(0x7f871c0ef610 0x7f871c0efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:06:05.881 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:05.866 7f871bac0700 1 -- 172.21.2.102:0/26167069 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f871c1b0470 msgr2=0x7f871c1b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:06:05.881 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:05.866 7f871bac0700 1 --2- 172.21.2.102:0/26167069 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f871c1b0470 0x7f871c1b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:06:05.881 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:05.866 7f871bac0700 1 -- 172.21.2.102:0/26167069 >> v2:172.21.2.113:3300/0 conn(0x7f871c0ef610 msgr2=0x7f871c0efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:06:05.881 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:05.866 7f871bac0700 1 --2- 172.21.2.102:0/26167069 >> v2:172.21.2.113:3300/0 conn(0x7f871c0ef610 0x7f871c0efa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:06:05.881 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:05.866 7f871bac0700 1 -- 172.21.2.102:0/26167069 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f871c0f10d0 con 0x7f871c0f0030 2021-02-10T11:06:05.881 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:05.866 7f871aabe700 1 --2- 172.21.2.102:0/26167069 >> v2:172.21.2.113:3300/0 conn(0x7f871c0ef610 0x7f871c0efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:06:05.881 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:05.866 7f871bac0700 1 --2- 172.21.2.102:0/26167069 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f871c0f0030 0x7f871c0f0430 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 rx=0x7f8714006f00 tx=0x7f8714009f10).ready entity=mon.0 client_cookie=f96a004956ce071 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:06:05.881 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:05.866 7f871a2bd700 1 -- 172.21.2.102:0/26167069 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f871400b3b0 con 0x7f871c0f0030 2021-02-10T11:06:05.881 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:05.866 7f871a2bd700 1 -- 172.21.2.102:0/26167069 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f871400b730 con 0x7f871c0f0030 2021-02-10T11:06:05.881 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:05.866 7f871a2bd700 1 -- 172.21.2.102:0/26167069 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f871400bc50 con 0x7f871c0f0030 2021-02-10T11:06:05.881 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:05.866 7f87214d7700 1 -- 172.21.2.102:0/26167069 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f871c0f0030 msgr2=0x7f871c0f0430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:06:05.882 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:05.867 7f87214d7700 1 --2- 172.21.2.102:0/26167069 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f871c0f0030 0x7f871c0f0430 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 rx=0x7f8714006f00 tx=0x7f8714009f10).stop 2021-02-10T11:06:05.882 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:05.867 7f87214d7700 1 -- 172.21.2.102:0/26167069 shutdown_connections 2021-02-10T11:06:05.882 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:05.867 7f87214d7700 1 --2- 172.21.2.102:0/26167069 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f871c0f0030 0x7f871c0f0430 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:06:05.882 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:05.867 7f87214d7700 1 --2- 172.21.2.102:0/26167069 >> v2:172.21.2.113:3300/0 conn(0x7f871c0ef610 0x7f871c0efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:06:05.882 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:05.867 7f87214d7700 1 --2- 172.21.2.102:0/26167069 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f871c1b0470 0x7f871c1b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:06:05.882 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:05.867 7f87214d7700 1 -- 172.21.2.102:0/26167069 >> 172.21.2.102:0/26167069 conn(0x7f871c0ce3e0 msgr2=0x7f871c0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:06:05.882 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:05.867 7f87214d7700 1 -- 172.21.2.102:0/26167069 shutdown_connections 2021-02-10T11:06:05.882 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:05.867 7f87214d7700 1 -- 172.21.2.102:0/26167069 wait complete. 2021-02-10T11:06:05.884 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:05.882 7f87214d7700 1 Processor -- start 2021-02-10T11:06:05.884 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:05.882 7f87214d7700 1 -- start start 2021-02-10T11:06:05.884 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:05.882 7f87214d7700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f871c0b7db0 0x7f871c0b8260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:06:05.884 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:05.882 7f87214d7700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f871c0b8880 0x7f871c1a7450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:06:05.884 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:05.882 7f87214d7700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f871c0b8c80 0x7f871c1a7a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:06:05.884 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:05.882 7f87214d7700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f871c0b79b0 con 0x7f871c0b8880 2021-02-10T11:06:05.884 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:05.882 7f87214d7700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f871c1a8090 con 0x7f871c0b8c80 2021-02-10T11:06:05.885 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:05.882 7f87214d7700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f871c1a82b0 con 0x7f871c0b7db0 2021-02-10T11:06:05.885 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:05.882 7f871aabe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f871c0b8880 0x7f871c1a7450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:06:05.885 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:05.882 7f871aabe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f871c0b8880 0x7f871c1a7450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:51020/0 (socket says 172.21.2.102:51020) 2021-02-10T11:06:05.885 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:05.882 7f871aabe700 1 -- 172.21.2.102:0/3966486794 learned_addr learned my addr 172.21.2.102:0/3966486794 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:06:05.885 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:05.882 7f871bac0700 1 --2- 172.21.2.102:0/3966486794 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f871c0b8c80 0x7f871c1a7a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:06:05.885 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:05.883 7f871b2bf700 1 --2- 172.21.2.102:0/3966486794 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f871c0b7db0 0x7f871c0b8260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:06:05.885 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:05.883 7f871aabe700 1 -- 172.21.2.102:0/3966486794 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f871c0b7db0 msgr2=0x7f871c0b8260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:06:05.885 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:05.883 7f871aabe700 1 --2- 172.21.2.102:0/3966486794 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f871c0b7db0 0x7f871c0b8260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:06:05.885 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:05.883 7f871aabe700 1 -- 172.21.2.102:0/3966486794 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f871c0b8c80 msgr2=0x7f871c1a7a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:06:05.885 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:05.883 7f871aabe700 1 --2- 172.21.2.102:0/3966486794 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f871c0b8c80 0x7f871c1a7a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:06:05.886 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:05.883 7f871aabe700 1 -- 172.21.2.102:0/3966486794 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f871c190c10 con 0x7f871c0b8880 2021-02-10T11:06:05.886 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:05.883 7f871b2bf700 1 --2- 172.21.2.102:0/3966486794 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f871c0b7db0 0x7f871c0b8260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:06:05.886 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:05.883 7f871aabe700 1 --2- 172.21.2.102:0/3966486794 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f871c0b8880 0x7f871c1a7450 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 rx=0x7f8704006f20 tx=0x7f870400b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:06:05.886 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:05.883 7f8713fff700 1 -- 172.21.2.102:0/3966486794 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f870400bd80 con 0x7f871c0b8880 2021-02-10T11:06:05.886 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:05.883 7f8713fff700 1 -- 172.21.2.102:0/3966486794 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8704000c90 con 0x7f871c0b8880 2021-02-10T11:06:05.886 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:05.883 7f8713fff700 1 -- 172.21.2.102:0/3966486794 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f87040022b0 con 0x7f871c0b8880 2021-02-10T11:06:05.886 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:05.883 7f87214d7700 1 -- 172.21.2.102:0/3966486794 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f871c190c10 con 0x7f871c0b8880 2021-02-10T11:06:05.886 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:05.883 7f87214d7700 1 -- 172.21.2.102:0/3966486794 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f871c190c10 con 0x7f871c0b8880 2021-02-10T11:06:05.886 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:05.884 7f8713fff700 1 -- 172.21.2.102:0/3966486794 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f870400a000 con 0x7f871c0b8880 2021-02-10T11:06:05.886 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:05.884 7f8713fff700 1 --2- 172.21.2.102:0/3966486794 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8700018730 0x7f870001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:06:05.887 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:05.884 7f8713fff700 1 -- 172.21.2.102:0/3966486794 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f87040327b0 con 0x7f871c0b8880 2021-02-10T11:06:05.887 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:05.884 7f871b2bf700 1 --2- 172.21.2.102:0/3966486794 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8700018730 0x7f870001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:06:05.887 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:05.884 7f871b2bf700 1 --2- 172.21.2.102:0/3966486794 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8700018730 0x7f870001abc0 crc :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:06:05.913 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:05.915 7f87214d7700 1 -- 172.21.2.102:0/3966486794 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f871c105b30 con 0x7f871c0b8880 2021-02-10T11:06:05.916 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:05.918 7f8713fff700 1 -- 172.21.2.102:0/3966486794 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f8704006bb0 con 0x7f871c0b8880 2021-02-10T11:06:06.044 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:06.046 7f87214d7700 1 -- 172.21.2.102:0/3966486794 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344367", "target": ["mgr", ""]}) v1 -- 0x7f871c1a8090 con 0x7f8700018730 2021-02-10T11:06:06.047 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:06.049 7f8713fff700 1 -- 172.21.2.102:0/3966486794 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f870c006930 con 0x7f8700018730 2021-02-10T11:06:06.051 INFO:teuthology.orchestra.run.gibba002.stdout:/volumes/_nogroup/subvolume_0000000000344367/b4da0bd3-e78f-4995-abee-ca1bd935a6ac 2021-02-10T11:06:06.053 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:06.054 7f87214d7700 1 -- 172.21.2.102:0/3966486794 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8700018730 msgr2=0x7f870001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:06:06.053 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:06.054 7f87214d7700 1 --2- 172.21.2.102:0/3966486794 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8700018730 0x7f870001abc0 crc :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:06:06.053 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:06.054 7f87214d7700 1 -- 172.21.2.102:0/3966486794 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f871c0b8880 msgr2=0x7f871c1a7450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:06:06.053 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:06.054 7f87214d7700 1 --2- 172.21.2.102:0/3966486794 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f871c0b8880 0x7f871c1a7450 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 rx=0x7f8704006f20 tx=0x7f870400b990).stop 2021-02-10T11:06:06.053 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:06.055 7f87214d7700 1 -- 172.21.2.102:0/3966486794 shutdown_connections 2021-02-10T11:06:06.053 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:06.055 7f87214d7700 1 --2- 172.21.2.102:0/3966486794 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8700018730 0x7f870001abc0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:06:06.053 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:06.055 7f87214d7700 1 --2- 172.21.2.102:0/3966486794 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f871c0b8c80 0x7f871c1a7a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:06:06.053 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:06.055 7f87214d7700 1 --2- 172.21.2.102:0/3966486794 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f871c0b8880 0x7f871c1a7450 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:06:06.053 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:06.055 7f87214d7700 1 --2- 172.21.2.102:0/3966486794 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f871c0b7db0 0x7f871c0b8260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:06:06.053 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:06.055 7f87214d7700 1 -- 172.21.2.102:0/3966486794 >> 172.21.2.102:0/3966486794 conn(0x7f871c094e10 msgr2=0x7f871c0a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:06:06.054 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:06.055 7f87214d7700 1 -- 172.21.2.102:0/3966486794 shutdown_connections 2021-02-10T11:06:06.054 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:06.055 7f87214d7700 1 -- 172.21.2.102:0/3966486794 wait complete. 2021-02-10T11:06:06.061 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000344367 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000344367/b4da0bd3-e78f-4995-abee-ca1bd935a6ac 2021-02-10T11:06:06.061 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:06:06.061 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:06:06.061 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:06:06.397 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:06:06.397 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:06:06.398 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344367/b4da0bd3-e78f-4995-abee-ca1bd935a6ac/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:06:06.777 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:06:06.777 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:06:06.777 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0241795 s, 43.4 MB/s 2021-02-10T11:06:06.779 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:06:06.779 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:06:06.779 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:06:07.113 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:06:07.113 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:06:07.114 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344367/b4da0bd3-e78f-4995-abee-ca1bd935a6ac/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:06:07.487 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:06:07.487 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:06:07.488 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0236746 s, 44.3 MB/s 2021-02-10T11:06:07.488 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:06:07.489 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:06:07.489 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:06:07.822 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:06:07.822 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:06:07.823 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344367/b4da0bd3-e78f-4995-abee-ca1bd935a6ac/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:06:08.198 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:06:08.198 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:06:08.198 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0254901 s, 41.1 MB/s 2021-02-10T11:06:08.199 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:06:08.199 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:06:08.199 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:06:08.534 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:06:08.534 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:06:08.534 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344367/b4da0bd3-e78f-4995-abee-ca1bd935a6ac/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:06:08.905 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:06:08.905 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:06:08.906 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0236965 s, 44.3 MB/s 2021-02-10T11:06:08.906 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:06:08.907 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:06:08.907 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:06:09.241 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:06:09.241 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:06:09.241 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344367/b4da0bd3-e78f-4995-abee-ca1bd935a6ac/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:06:09.619 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:06:09.619 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:06:09.619 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0262103 s, 40.0 MB/s 2021-02-10T11:06:09.620 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:06:09.620 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:06:09.620 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:06:09.955 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:06:09.955 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:06:09.955 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344367/b4da0bd3-e78f-4995-abee-ca1bd935a6ac/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:06:10.331 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:06:10.331 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:06:10.331 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0257619 s, 40.7 MB/s 2021-02-10T11:06:10.332 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:06:10.332 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:06:10.332 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:06:10.667 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:06:10.667 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:06:10.668 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344367/b4da0bd3-e78f-4995-abee-ca1bd935a6ac/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:06:11.042 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:06:11.043 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:06:11.043 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0253689 s, 41.3 MB/s 2021-02-10T11:06:11.044 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:06:11.044 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:06:11.044 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:06:11.379 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:06:11.379 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:06:11.379 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344367/b4da0bd3-e78f-4995-abee-ca1bd935a6ac/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:06:11.753 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:06:11.753 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:06:11.753 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.023651 s, 44.3 MB/s 2021-02-10T11:06:11.754 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:06:11.754 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:06:11.754 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:06:12.090 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:06:12.090 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:06:12.090 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344367/b4da0bd3-e78f-4995-abee-ca1bd935a6ac/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:06:12.468 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:06:12.468 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:06:12.468 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0273593 s, 38.3 MB/s 2021-02-10T11:06:12.469 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:06:12.470 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:06:12.470 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:06:12.753 DEBUG:teuthology.orchestra.run.gibba002:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:06:12.756 DEBUG:teuthology.orchestra.run.gibba013:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:06:12.804 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:06:12.804 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:06:12.804 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344367/b4da0bd3-e78f-4995-abee-ca1bd935a6ac/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:06:13.130 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:06:13.131 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:06:13.131 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0258022 s, 40.6 MB/s 2021-02-10T11:06:13.132 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000344368 2021-02-10T11:06:13.601 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:13.586 7fd49e2b0700 1 Processor -- start 2021-02-10T11:06:15.828 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:13.586 7fd49e2b0700 1 -- start start 2021-02-10T11:06:15.828 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:13.587 7fd49e2b0700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fd4981b0470 0x7fd4981b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:06:15.828 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:13.587 7fd49e2b0700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fd4980ef610 0x7fd4980efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:06:15.828 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:13.587 7fd49e2b0700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd4980f0030 0x7fd4980f0430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:06:15.828 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:13.587 7fd49e2b0700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4980e5ea0 con 0x7fd4980ef610 2021-02-10T11:06:15.828 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:13.587 7fd49e2b0700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4980f0b50 con 0x7fd4981b0470 2021-02-10T11:06:15.829 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:13.587 7fd49e2b0700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4980e52e0 con 0x7fd4980f0030 2021-02-10T11:06:15.829 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:13.587 7fd4977fe700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fd4980ef610 0x7fd4980efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:06:15.829 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:13.587 7fd4977fe700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fd4980ef610 0x7fd4980efa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:51028/0 (socket says 172.21.2.102:51028) 2021-02-10T11:06:15.829 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:13.587 7fd4977fe700 1 -- 172.21.2.102:0/799113377 learned_addr learned my addr 172.21.2.102:0/799113377 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:06:15.829 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:13.587 7fd4977fe700 1 -- 172.21.2.102:0/799113377 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd4980f0030 msgr2=0x7fd4980f0430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:06:15.829 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:13.587 7fd4977fe700 1 --2- 172.21.2.102:0/799113377 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd4980f0030 0x7fd4980f0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:06:15.829 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:13.587 7fd4977fe700 1 -- 172.21.2.102:0/799113377 >> v2:172.21.2.113:3300/0 conn(0x7fd4981b0470 msgr2=0x7fd4981b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:06:15.829 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:13.587 7fd4977fe700 1 --2- 172.21.2.102:0/799113377 >> v2:172.21.2.113:3300/0 conn(0x7fd4981b0470 0x7fd4981b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:06:15.829 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:13.587 7fd4977fe700 1 -- 172.21.2.102:0/799113377 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4980f10d0 con 0x7fd4980ef610 2021-02-10T11:06:15.829 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:13.587 7fd4977fe700 1 --2- 172.21.2.102:0/799113377 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd4980ef610 0x7fd4980efa10 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 rx=0x7fd48c006700 tx=0x7fd48c009b80).ready entity=mon.0 client_cookie=87988ec4c32a28ee server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:06:15.829 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:13.588 7fd496ffd700 1 -- 172.21.2.102:0/799113377 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd48c00c0c0 con 0x7fd4980ef610 2021-02-10T11:06:15.829 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:13.588 7fd496ffd700 1 -- 172.21.2.102:0/799113377 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd48c00c0c0 con 0x7fd4980ef610 2021-02-10T11:06:15.829 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:13.588 7fd496ffd700 1 -- 172.21.2.102:0/799113377 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd48c00c440 con 0x7fd4980ef610 2021-02-10T11:06:15.830 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:13.588 7fd49e2b0700 1 -- 172.21.2.102:0/799113377 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd4980ef610 msgr2=0x7fd4980efa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:06:15.830 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:13.588 7fd49e2b0700 1 --2- 172.21.2.102:0/799113377 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd4980ef610 0x7fd4980efa10 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 rx=0x7fd48c006700 tx=0x7fd48c009b80).stop 2021-02-10T11:06:15.830 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:13.588 7fd49e2b0700 1 -- 172.21.2.102:0/799113377 shutdown_connections 2021-02-10T11:06:15.830 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:13.588 7fd49e2b0700 1 --2- 172.21.2.102:0/799113377 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd4980f0030 0x7fd4980f0430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:06:15.830 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:13.588 7fd49e2b0700 1 --2- 172.21.2.102:0/799113377 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd4980ef610 0x7fd4980efa10 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:06:15.830 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:13.588 7fd49e2b0700 1 --2- 172.21.2.102:0/799113377 >> v2:172.21.2.113:3300/0 conn(0x7fd4981b0470 0x7fd4981b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:06:15.830 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:13.588 7fd49e2b0700 1 -- 172.21.2.102:0/799113377 >> 172.21.2.102:0/799113377 conn(0x7fd4980ce3e0 msgr2=0x7fd4980e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:06:15.830 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:13.588 7fd49e2b0700 1 -- 172.21.2.102:0/799113377 shutdown_connections 2021-02-10T11:06:15.830 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:13.589 7fd49e2b0700 1 -- 172.21.2.102:0/799113377 wait complete. 2021-02-10T11:06:15.830 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:13.603 7fd49e2b0700 1 Processor -- start 2021-02-10T11:06:15.830 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:13.603 7fd49e2b0700 1 -- start start 2021-02-10T11:06:15.833 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:13.603 7fd49e2b0700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd4980b20b0 0x7fd4980ac970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:06:15.833 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:13.603 7fd49e2b0700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd4980b24b0 0x7fd498191dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:06:15.833 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:13.603 7fd49e2b0700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd4980acf90 0x7fd4981923e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:06:15.833 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:13.603 7fd49e2b0700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4980b1b30 con 0x7fd4980acf90 2021-02-10T11:06:15.833 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:13.603 7fd49e2b0700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4981a04e0 con 0x7fd4980b24b0 2021-02-10T11:06:15.834 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:13.603 7fd49e2b0700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4981a06c0 con 0x7fd4980b20b0 2021-02-10T11:06:15.834 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:13.603 7fd49c8cd700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd4980acf90 0x7fd4981923e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:06:15.834 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:13.603 7fd49c8cd700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd4980acf90 0x7fd4981923e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:51032/0 (socket says 172.21.2.102:51032) 2021-02-10T11:06:15.834 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:13.603 7fd49c8cd700 1 -- 172.21.2.102:0/402084987 learned_addr learned my addr 172.21.2.102:0/402084987 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:06:15.834 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:13.604 7fd49c8cd700 1 -- 172.21.2.102:0/402084987 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd4980b20b0 msgr2=0x7fd4980ac970 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:06:15.834 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:13.604 7fd49c8cd700 1 --2- 172.21.2.102:0/402084987 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd4980b20b0 0x7fd4980ac970 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:06:15.834 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:13.604 7fd49c8cd700 1 -- 172.21.2.102:0/402084987 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd4980b24b0 msgr2=0x7fd498191dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:06:15.834 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:13.604 7fd49c8cd700 1 --2- 172.21.2.102:0/402084987 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd4980b24b0 0x7fd498191dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:06:15.834 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:13.604 7fd49c8cd700 1 -- 172.21.2.102:0/402084987 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd498192be0 con 0x7fd4980acf90 2021-02-10T11:06:15.834 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:13.604 7fd49c8cd700 1 --2- 172.21.2.102:0/402084987 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd4980acf90 0x7fd4981923e0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 rx=0x7fd49000b8c0 tx=0x7fd49000bc60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:06:15.834 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:13.604 7fd494ff9700 1 -- 172.21.2.102:0/402084987 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd490005af0 con 0x7fd4980acf90 2021-02-10T11:06:15.834 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:13.604 7fd494ff9700 1 -- 172.21.2.102:0/402084987 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd490005df0 con 0x7fd4980acf90 2021-02-10T11:06:15.834 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:13.604 7fd494ff9700 1 -- 172.21.2.102:0/402084987 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd4900062b0 con 0x7fd4980acf90 2021-02-10T11:06:15.835 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:13.604 7fd49e2b0700 1 -- 172.21.2.102:0/402084987 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd498192be0 con 0x7fd4980acf90 2021-02-10T11:06:15.835 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:13.604 7fd49e2b0700 1 -- 172.21.2.102:0/402084987 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd498192be0 con 0x7fd4980acf90 2021-02-10T11:06:15.835 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:13.605 7fd494ff9700 1 -- 172.21.2.102:0/402084987 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd490005af0 con 0x7fd4980acf90 2021-02-10T11:06:15.835 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:13.605 7fd494ff9700 1 --2- 172.21.2.102:0/402084987 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd478018730 0x7fd47801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:06:15.835 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:13.605 7fd494ff9700 1 -- 172.21.2.102:0/402084987 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fd4900329d0 con 0x7fd4980acf90 2021-02-10T11:06:15.835 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:13.605 7fd497fff700 1 --2- 172.21.2.102:0/402084987 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd478018730 0x7fd47801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:06:15.835 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:13.605 7fd497fff700 1 --2- 172.21.2.102:0/402084987 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd478018730 0x7fd47801abc0 crc :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:06:15.835 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:13.637 7fd49e2b0700 1 -- 172.21.2.102:0/402084987 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd498105af0 con 0x7fd4980acf90 2021-02-10T11:06:15.835 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:13.639 7fd494ff9700 1 -- 172.21.2.102:0/402084987 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd490005df0 con 0x7fd4980acf90 2021-02-10T11:06:15.835 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:13.766 7fd49e2b0700 1 -- 172.21.2.102:0/402084987 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344368", "target": ["mgr", ""]}) v1 -- 0x7fd4981a04e0 con 0x7fd478018730 2021-02-10T11:06:15.835 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:13.789 7fd494ff9700 1 -- 172.21.2.102:0/402084987 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fd488006960 con 0x7fd478018730 2021-02-10T11:06:15.835 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:13.799 7fd49e2b0700 1 -- 172.21.2.102:0/402084987 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd478018730 msgr2=0x7fd47801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:06:15.835 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:13.799 7fd49e2b0700 1 --2- 172.21.2.102:0/402084987 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd478018730 0x7fd47801abc0 crc :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:06:15.835 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:13.799 7fd49e2b0700 1 -- 172.21.2.102:0/402084987 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd4980acf90 msgr2=0x7fd4981923e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:06:15.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:13.799 7fd49e2b0700 1 --2- 172.21.2.102:0/402084987 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd4980acf90 0x7fd4981923e0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 rx=0x7fd49000b8c0 tx=0x7fd49000bc60).stop 2021-02-10T11:06:15.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:13.799 7fd49e2b0700 1 -- 172.21.2.102:0/402084987 shutdown_connections 2021-02-10T11:06:15.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:13.799 7fd49e2b0700 1 --2- 172.21.2.102:0/402084987 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd4980acf90 0x7fd4981923e0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:06:15.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:13.799 7fd49e2b0700 1 --2- 172.21.2.102:0/402084987 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd478018730 0x7fd47801abc0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:06:15.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:13.799 7fd49e2b0700 1 --2- 172.21.2.102:0/402084987 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd4980b24b0 0x7fd498191dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:06:15.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:13.799 7fd49e2b0700 1 --2- 172.21.2.102:0/402084987 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd4980b20b0 0x7fd4980ac970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:06:15.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:13.799 7fd49e2b0700 1 -- 172.21.2.102:0/402084987 >> 172.21.2.102:0/402084987 conn(0x7fd4980bf4e0 msgr2=0x7fd4980be3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:06:15.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:13.799 7fd49e2b0700 1 -- 172.21.2.102:0/402084987 shutdown_connections 2021-02-10T11:06:15.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:13.799 7fd49e2b0700 1 -- 172.21.2.102:0/402084987 wait complete. 2021-02-10T11:06:15.837 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000344368 2021-02-10T11:06:16.267 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:16.251 7efe5ce38700 1 Processor -- start 2021-02-10T11:06:16.268 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:16.251 7efe5ce38700 1 -- start start 2021-02-10T11:06:16.268 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:16.252 7efe5ce38700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7efe581b0470 0x7efe581b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:06:16.268 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:16.252 7efe5ce38700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7efe580ef610 0x7efe580efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:06:16.268 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:16.252 7efe5ce38700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe580f0b50 con 0x7efe580f0030 2021-02-10T11:06:16.268 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:16.252 7efe5ce38700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe580e52e0 con 0x7efe580ef610 2021-02-10T11:06:16.268 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:16.252 7efe5ce38700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe580e5490 con 0x7efe581b0470 2021-02-10T11:06:16.269 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:16.253 7efe56312700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7efe580ef610 0x7efe580efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:06:16.269 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:16.253 7efe56b13700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7efe581b0470 0x7efe581b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:06:16.269 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:16.253 7efe56312700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7efe580ef610 0x7efe580efa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.113:3300/0 says I am v2:172.21.2.102:54470/0 (socket says 172.21.2.102:54470) 2021-02-10T11:06:16.269 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:16.253 7efe56312700 1 -- 172.21.2.102:0/52520815 learned_addr learned my addr 172.21.2.102:0/52520815 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:06:16.269 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:16.253 7efe56312700 1 -- 172.21.2.102:0/52520815 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7efe581b0470 msgr2=0x7efe581b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:06:16.269 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:16.253 7efe56312700 1 --2- 172.21.2.102:0/52520815 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7efe581b0470 0x7efe581b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:06:16.269 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:16.253 7efe56312700 1 -- 172.21.2.102:0/52520815 >> v1:172.21.2.113:6789/0 conn(0x7efe580f0030 legacy=0x7efe580f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:06:16.270 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:16.253 7efe56312700 1 -- 172.21.2.102:0/52520815 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe580f0430 con 0x7efe580ef610 2021-02-10T11:06:16.270 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:16.253 7efe56312700 1 --2- 172.21.2.102:0/52520815 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7efe580ef610 0x7efe580efa10 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7efe48006700 tx=0x7efe48009b80).ready entity=mon.1 client_cookie=8d505c227862a5f8 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:06:16.270 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:16.254 7efe55b11700 1 -- 172.21.2.102:0/52520815 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efe4800c0c0 con 0x7efe580ef610 2021-02-10T11:06:16.270 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:16.254 7efe55b11700 1 -- 172.21.2.102:0/52520815 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe4800c340 con 0x7efe580ef610 2021-02-10T11:06:16.270 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:16.254 7efe55b11700 1 -- 172.21.2.102:0/52520815 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efe4800c0c0 con 0x7efe580ef610 2021-02-10T11:06:16.270 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:16.254 7efe5ce38700 1 -- 172.21.2.102:0/52520815 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7efe580ef610 msgr2=0x7efe580efa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:06:16.270 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:16.254 7efe5ce38700 1 --2- 172.21.2.102:0/52520815 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7efe580ef610 0x7efe580efa10 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7efe48006700 tx=0x7efe48009b80).stop 2021-02-10T11:06:16.271 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:16.254 7efe5ce38700 1 -- 172.21.2.102:0/52520815 shutdown_connections 2021-02-10T11:06:16.271 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:16.254 7efe5ce38700 1 --2- 172.21.2.102:0/52520815 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7efe580ef610 0x7efe580efa10 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:06:16.271 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:16.254 7efe5ce38700 1 --2- 172.21.2.102:0/52520815 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7efe581b0470 0x7efe581b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:06:16.271 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:16.254 7efe5ce38700 1 -- 172.21.2.102:0/52520815 >> 172.21.2.102:0/52520815 conn(0x7efe580ce3e0 msgr2=0x7efe580e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:06:16.271 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:16.255 7efe5ce38700 1 -- 172.21.2.102:0/52520815 shutdown_connections 2021-02-10T11:06:16.271 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:16.255 7efe5ce38700 1 -- 172.21.2.102:0/52520815 wait complete. 2021-02-10T11:06:16.272 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:16.269 7efe5ce38700 1 Processor -- start 2021-02-10T11:06:16.272 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:16.269 7efe5ce38700 1 -- start start 2021-02-10T11:06:16.274 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:16.269 7efe5ce38700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7efe580a0420 0x7efe580a0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:06:16.274 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:16.269 7efe5ce38700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7efe58189d00 0x7efe5818a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:06:16.274 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:16.269 7efe5ce38700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efe5818a720 0x7efe580b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:06:16.274 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:16.269 7efe5ce38700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe58189780 con 0x7efe5818a720 2021-02-10T11:06:16.274 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:16.269 7efe5ce38700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe5818ab20 con 0x7efe58189d00 2021-02-10T11:06:16.275 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:16.269 7efe5ce38700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe580a1140 con 0x7efe580a0420 2021-02-10T11:06:16.275 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:16.270 7efe57314700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efe5818a720 0x7efe580b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:06:16.275 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:16.270 7efe57314700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efe5818a720 0x7efe580b7be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:51050/0 (socket says 172.21.2.102:51050) 2021-02-10T11:06:16.275 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:16.270 7efe57314700 1 -- 172.21.2.102:0/3306207496 learned_addr learned my addr 172.21.2.102:0/3306207496 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:06:16.275 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:16.270 7efe56312700 1 --2- 172.21.2.102:0/3306207496 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7efe58189d00 0x7efe5818a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:06:16.275 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:16.270 7efe57314700 1 -- 172.21.2.102:0/3306207496 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7efe580a0420 msgr2=0x7efe580a0820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:06:16.275 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:16.270 7efe57314700 1 --2- 172.21.2.102:0/3306207496 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7efe580a0420 0x7efe580a0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:06:16.275 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:16.270 7efe57314700 1 -- 172.21.2.102:0/3306207496 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7efe58189d00 msgr2=0x7efe5818a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:06:16.275 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:16.270 7efe57314700 1 --2- 172.21.2.102:0/3306207496 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7efe58189d00 0x7efe5818a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:06:16.275 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:16.270 7efe57314700 1 -- 172.21.2.102:0/3306207496 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe580b8200 con 0x7efe5818a720 2021-02-10T11:06:16.276 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:16.270 7efe57314700 1 --2- 172.21.2.102:0/3306207496 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efe5818a720 0x7efe580b7be0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 rx=0x7efe50009940 tx=0x7efe5000bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:06:16.276 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:16.270 7efe4f7fe700 1 -- 172.21.2.102:0/3306207496 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efe50000ce0 con 0x7efe5818a720 2021-02-10T11:06:16.276 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:16.270 7efe4f7fe700 1 -- 172.21.2.102:0/3306207496 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe5000c2f0 con 0x7efe5818a720 2021-02-10T11:06:16.276 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:16.270 7efe5ce38700 1 -- 172.21.2.102:0/3306207496 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe580b8200 con 0x7efe5818a720 2021-02-10T11:06:16.276 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:16.271 7efe4f7fe700 1 -- 172.21.2.102:0/3306207496 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efe5000c7d0 con 0x7efe5818a720 2021-02-10T11:06:16.276 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:16.271 7efe5ce38700 1 -- 172.21.2.102:0/3306207496 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe580b8790 con 0x7efe5818a720 2021-02-10T11:06:16.276 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:16.271 7efe4f7fe700 1 -- 172.21.2.102:0/3306207496 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7efe50001220 con 0x7efe5818a720 2021-02-10T11:06:16.276 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:16.271 7efe4f7fe700 1 --2- 172.21.2.102:0/3306207496 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7efe3c018730 0x7efe3c01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:06:16.276 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:16.271 7efe4f7fe700 1 -- 172.21.2.102:0/3306207496 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7efe50032b50 con 0x7efe5818a720 2021-02-10T11:06:16.276 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:16.271 7efe56b13700 1 --2- 172.21.2.102:0/3306207496 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7efe3c018730 0x7efe3c01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:06:16.276 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:16.272 7efe56b13700 1 --2- 172.21.2.102:0/3306207496 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7efe3c018730 0x7efe3c01ac10 crc :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:06:16.300 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:16.302 7efe5ce38700 1 -- 172.21.2.102:0/3306207496 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe58105ae0 con 0x7efe5818a720 2021-02-10T11:06:16.303 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:16.305 7efe4f7fe700 1 -- 172.21.2.102:0/3306207496 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7efe5000be30 con 0x7efe5818a720 2021-02-10T11:06:16.432 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:16.433 7efe5ce38700 1 -- 172.21.2.102:0/3306207496 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344368", "target": ["mgr", ""]}) v1 -- 0x7efe58189780 con 0x7efe3c018730 2021-02-10T11:06:16.434 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:16.436 7efe4f7fe700 1 -- 172.21.2.102:0/3306207496 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7efe40006e70 con 0x7efe3c018730 2021-02-10T11:06:16.434 INFO:teuthology.orchestra.run.gibba002.stdout:/volumes/_nogroup/subvolume_0000000000344368/38b5b2ba-be5b-4617-b434-d3a9328f4354 2021-02-10T11:06:16.436 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:16.438 7efe5ce38700 1 -- 172.21.2.102:0/3306207496 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7efe3c018730 msgr2=0x7efe3c01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:06:16.436 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:16.438 7efe5ce38700 1 --2- 172.21.2.102:0/3306207496 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7efe3c018730 0x7efe3c01ac10 crc :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:06:16.436 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:16.438 7efe5ce38700 1 -- 172.21.2.102:0/3306207496 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efe5818a720 msgr2=0x7efe580b7be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:06:16.436 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:16.438 7efe5ce38700 1 --2- 172.21.2.102:0/3306207496 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efe5818a720 0x7efe580b7be0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 rx=0x7efe50009940 tx=0x7efe5000bcf0).stop 2021-02-10T11:06:16.436 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:16.438 7efe5ce38700 1 -- 172.21.2.102:0/3306207496 shutdown_connections 2021-02-10T11:06:16.436 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:16.438 7efe5ce38700 1 --2- 172.21.2.102:0/3306207496 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efe5818a720 0x7efe580b7be0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:06:16.437 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:16.438 7efe5ce38700 1 --2- 172.21.2.102:0/3306207496 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7efe3c018730 0x7efe3c01ac10 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:06:16.437 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:16.438 7efe5ce38700 1 --2- 172.21.2.102:0/3306207496 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7efe58189d00 0x7efe5818a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:06:16.437 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:16.438 7efe5ce38700 1 --2- 172.21.2.102:0/3306207496 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7efe580a0420 0x7efe580a0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:06:16.437 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:16.438 7efe5ce38700 1 -- 172.21.2.102:0/3306207496 >> 172.21.2.102:0/3306207496 conn(0x7efe58094e10 msgr2=0x7efe580a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:06:16.437 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:16.438 7efe5ce38700 1 -- 172.21.2.102:0/3306207496 shutdown_connections 2021-02-10T11:06:16.437 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:16.438 7efe5ce38700 1 -- 172.21.2.102:0/3306207496 wait complete. 2021-02-10T11:06:16.445 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000344368 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000344368/38b5b2ba-be5b-4617-b434-d3a9328f4354 2021-02-10T11:06:16.445 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:06:16.445 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:06:16.445 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:06:16.780 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:06:16.780 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:06:16.780 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344368/38b5b2ba-be5b-4617-b434-d3a9328f4354/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:06:17.164 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:06:17.164 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:06:17.164 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0257102 s, 40.8 MB/s 2021-02-10T11:06:17.165 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:06:17.165 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:06:17.165 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:06:17.499 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:06:17.499 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:06:17.500 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344368/38b5b2ba-be5b-4617-b434-d3a9328f4354/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:06:17.876 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:06:17.877 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:06:17.877 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0245482 s, 42.7 MB/s 2021-02-10T11:06:17.878 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:06:17.878 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:06:17.878 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:06:18.211 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:06:18.211 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:06:18.211 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344368/38b5b2ba-be5b-4617-b434-d3a9328f4354/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:06:18.583 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:06:18.584 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:06:18.584 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.025262 s, 41.5 MB/s 2021-02-10T11:06:18.585 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:06:18.585 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:06:18.585 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:06:18.922 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:06:18.923 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:06:18.924 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344368/38b5b2ba-be5b-4617-b434-d3a9328f4354/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:06:19.304 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:06:19.304 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:06:19.304 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.028412 s, 36.9 MB/s 2021-02-10T11:06:19.305 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:06:19.305 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:06:19.305 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:06:19.639 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:06:19.639 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:06:19.639 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344368/38b5b2ba-be5b-4617-b434-d3a9328f4354/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:06:20.018 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:06:20.018 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:06:20.018 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0270211 s, 38.8 MB/s 2021-02-10T11:06:20.019 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:06:20.019 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:06:20.020 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:06:20.354 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:06:20.354 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:06:20.354 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344368/38b5b2ba-be5b-4617-b434-d3a9328f4354/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:06:20.727 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:06:20.727 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:06:20.727 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0224583 s, 46.7 MB/s 2021-02-10T11:06:20.728 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:06:20.728 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:06:20.728 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:06:21.063 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:06:21.063 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:06:21.064 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344368/38b5b2ba-be5b-4617-b434-d3a9328f4354/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:06:21.442 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:06:21.443 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:06:21.443 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0279838 s, 37.5 MB/s 2021-02-10T11:06:21.444 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:06:21.444 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:06:21.444 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:06:21.778 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:06:21.778 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:06:21.779 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344368/38b5b2ba-be5b-4617-b434-d3a9328f4354/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:06:22.152 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:06:22.152 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:06:22.152 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0236129 s, 44.4 MB/s 2021-02-10T11:06:22.153 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:06:22.153 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:06:22.153 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:06:22.488 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:06:22.488 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:06:22.488 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344368/38b5b2ba-be5b-4617-b434-d3a9328f4354/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:06:22.863 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:06:22.863 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:06:22.863 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0247039 s, 42.4 MB/s 2021-02-10T11:06:22.865 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:06:22.865 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:06:22.865 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:06:23.199 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:06:23.199 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:06:23.199 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344368/38b5b2ba-be5b-4617-b434-d3a9328f4354/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:06:23.569 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:06:23.569 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:06:23.569 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0230537 s, 45.5 MB/s 2021-02-10T11:06:23.570 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000344369 2021-02-10T11:06:23.999 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:23.984 7fcf028c5700 1 Processor -- start 2021-02-10T11:06:24.000 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:23.984 7fcf028c5700 1 -- start start 2021-02-10T11:06:24.000 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:23.984 7fcf028c5700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcefc1b0470 0x7fcefc1b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:06:24.000 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:23.984 7fcf028c5700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fcefc0f20e0 0x7fcefc0f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:06:24.000 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:23.984 7fcf028c5700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcefc0f0b50 con 0x7fcefc0ef610 2021-02-10T11:06:24.000 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:23.984 7fcf028c5700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcefc0f0820 con 0x7fcefc0f20e0 2021-02-10T11:06:24.000 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:23.984 7fcf028c5700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcefc0f24e0 con 0x7fcefc1b0470 2021-02-10T11:06:24.000 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:23.985 7fcf00ee2700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fcefc0f20e0 0x7fcefc0f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:06:24.000 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:23.985 7fcf00ee2700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fcefc0f20e0 0x7fcefc0f0200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.113:3300/0 says I am v2:172.21.2.102:54486/0 (socket says 172.21.2.102:54486) 2021-02-10T11:06:24.000 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:23.985 7fcf00ee2700 1 -- 172.21.2.102:0/582993891 learned_addr learned my addr 172.21.2.102:0/582993891 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:06:24.000 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:23.985 7fcefbfff700 1 --2- 172.21.2.102:0/582993891 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcefc1b0470 0x7fcefc1b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:06:24.000 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:23.986 7fcf00ee2700 1 -- 172.21.2.102:0/582993891 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcefc1b0470 msgr2=0x7fcefc1b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:06:24.000 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:23.986 7fcf00ee2700 1 --2- 172.21.2.102:0/582993891 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcefc1b0470 0x7fcefc1b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:06:24.001 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:23.986 7fcf00ee2700 1 -- 172.21.2.102:0/582993891 >> v1:172.21.2.113:6789/0 conn(0x7fcefc0ef610 legacy=0x7fcefc0efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:06:24.001 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:23.986 7fcf00ee2700 1 -- 172.21.2.102:0/582993891 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcefc0e52e0 con 0x7fcefc0f20e0 2021-02-10T11:06:24.001 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:23.986 7fcf00ee2700 1 --2- 172.21.2.102:0/582993891 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fcefc0f20e0 0x7fcefc0f0200 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7fcef4006aa0 tx=0x7fcef4009f20).ready entity=mon.1 client_cookie=ae8da00aedd8a46b server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:06:24.001 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:23.986 7fcefaffd700 1 -- 172.21.2.102:0/582993891 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcef400b5c0 con 0x7fcefc0f20e0 2021-02-10T11:06:24.001 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:23.986 7fcefaffd700 1 -- 172.21.2.102:0/582993891 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcef4003030 con 0x7fcefc0f20e0 2021-02-10T11:06:24.001 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:23.986 7fcefaffd700 1 -- 172.21.2.102:0/582993891 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcef400b9f0 con 0x7fcefc0f20e0 2021-02-10T11:06:24.001 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:23.986 7fcf028c5700 1 -- 172.21.2.102:0/582993891 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fcefc0f20e0 msgr2=0x7fcefc0f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:06:24.001 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:23.986 7fcf028c5700 1 --2- 172.21.2.102:0/582993891 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fcefc0f20e0 0x7fcefc0f0200 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7fcef4006aa0 tx=0x7fcef4009f20).stop 2021-02-10T11:06:24.001 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:23.987 7fcf028c5700 1 -- 172.21.2.102:0/582993891 shutdown_connections 2021-02-10T11:06:24.001 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:23.987 7fcf028c5700 1 --2- 172.21.2.102:0/582993891 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fcefc0f20e0 0x7fcefc0f0200 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:06:24.001 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:23.987 7fcf028c5700 1 --2- 172.21.2.102:0/582993891 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcefc1b0470 0x7fcefc1b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:06:24.001 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:23.987 7fcf028c5700 1 -- 172.21.2.102:0/582993891 >> 172.21.2.102:0/582993891 conn(0x7fcefc0ce3e0 msgr2=0x7fcefc0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:06:24.001 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:23.987 7fcf028c5700 1 -- 172.21.2.102:0/582993891 shutdown_connections 2021-02-10T11:06:24.002 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:23.987 7fcf028c5700 1 -- 172.21.2.102:0/582993891 wait complete. 2021-02-10T11:06:24.003 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:24.002 7fcf028c5700 1 Processor -- start 2021-02-10T11:06:24.003 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:24.002 7fcf028c5700 1 -- start start 2021-02-10T11:06:24.003 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:24.002 7fcf028c5700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcefc0b7c20 0x7fcefc0b8410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:06:24.004 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:24.002 7fcf028c5700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fcefc0b8a30 0x7fcefc0c1f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:06:24.004 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:24.002 7fcf028c5700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcefc0c25a0 0x7fcefc0c29a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:06:24.004 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:24.002 7fcf028c5700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcefc0c1a80 con 0x7fcefc0b7c20 2021-02-10T11:06:24.004 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:24.002 7fcf028c5700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcefc0b8020 con 0x7fcefc0b8a30 2021-02-10T11:06:24.004 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:24.002 7fcf028c5700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcefc09b9c0 con 0x7fcefc0c25a0 2021-02-10T11:06:24.004 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:24.002 7fcefbfff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcefc0b7c20 0x7fcefc0b8410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:06:24.004 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:24.002 7fcefbfff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcefc0b7c20 0x7fcefc0b8410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:51062/0 (socket says 172.21.2.102:51062) 2021-02-10T11:06:24.004 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:24.002 7fcefbfff700 1 -- 172.21.2.102:0/624354805 learned_addr learned my addr 172.21.2.102:0/624354805 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:06:24.004 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:24.002 7fcf00ee2700 1 --2- 172.21.2.102:0/624354805 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcefc0c25a0 0x7fcefc0c29a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:06:24.004 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:24.002 7fcefb7fe700 1 --2- 172.21.2.102:0/624354805 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fcefc0b8a30 0x7fcefc0c1f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:06:24.004 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:24.003 7fcefbfff700 1 -- 172.21.2.102:0/624354805 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcefc0c25a0 msgr2=0x7fcefc0c29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:06:24.004 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:24.003 7fcefbfff700 1 --2- 172.21.2.102:0/624354805 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcefc0c25a0 0x7fcefc0c29a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:06:24.005 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:24.003 7fcefbfff700 1 -- 172.21.2.102:0/624354805 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fcefc0b8a30 msgr2=0x7fcefc0c1f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:06:24.005 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:24.003 7fcefbfff700 1 --2- 172.21.2.102:0/624354805 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fcefc0b8a30 0x7fcefc0c1f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:06:24.005 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:24.003 7fcefbfff700 1 -- 172.21.2.102:0/624354805 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcefc0b8e30 con 0x7fcefc0b7c20 2021-02-10T11:06:24.005 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:24.003 7fcefb7fe700 1 --2- 172.21.2.102:0/624354805 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fcefc0b8a30 0x7fcefc0c1f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:06:24.005 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:24.003 7fcefbfff700 1 --2- 172.21.2.102:0/624354805 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcefc0b7c20 0x7fcefc0b8410 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 rx=0x7fceec006f60 tx=0x7fceec00b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:06:24.005 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:24.003 7fcef8ff9700 1 -- 172.21.2.102:0/624354805 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fceec00bd60 con 0x7fcefc0b7c20 2021-02-10T11:06:24.005 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:24.003 7fcef8ff9700 1 -- 172.21.2.102:0/624354805 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fceec000c90 con 0x7fcefc0b7c20 2021-02-10T11:06:24.005 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:24.003 7fcf028c5700 1 -- 172.21.2.102:0/624354805 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcefc0b8e30 con 0x7fcefc0b7c20 2021-02-10T11:06:24.005 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:24.003 7fcf028c5700 1 -- 172.21.2.102:0/624354805 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcefc0c32b0 con 0x7fcefc0b7c20 2021-02-10T11:06:24.005 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:24.003 7fcef8ff9700 1 -- 172.21.2.102:0/624354805 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fceec0022b0 con 0x7fcefc0b7c20 2021-02-10T11:06:24.005 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:24.004 7fcef8ff9700 1 -- 172.21.2.102:0/624354805 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fceec006b00 con 0x7fcefc0b7c20 2021-02-10T11:06:24.005 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:24.004 7fcef8ff9700 1 --2- 172.21.2.102:0/624354805 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcedc0186c0 0x7fcedc01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:06:24.005 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:24.004 7fcef8ff9700 1 -- 172.21.2.102:0/624354805 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fceec032520 con 0x7fcefc0b7c20 2021-02-10T11:06:24.005 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:24.004 7fcefb7fe700 1 --2- 172.21.2.102:0/624354805 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcedc0186c0 0x7fcedc01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:06:24.006 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:24.005 7fcefb7fe700 1 --2- 172.21.2.102:0/624354805 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcedc0186c0 0x7fcedc01abc0 crc :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:06:24.033 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:24.034 7fcf028c5700 1 -- 172.21.2.102:0/624354805 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcefc105b30 con 0x7fcefc0b7c20 2021-02-10T11:06:24.035 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:24.037 7fcef8ff9700 1 -- 172.21.2.102:0/624354805 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fceec000940 con 0x7fcefc0b7c20 2021-02-10T11:06:24.163 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:24.164 7fcf028c5700 1 -- 172.21.2.102:0/624354805 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344369", "target": ["mgr", ""]}) v1 -- 0x7fcefc0b8020 con 0x7fcedc0186c0 2021-02-10T11:06:24.187 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:24.189 7fcef8ff9700 1 -- 172.21.2.102:0/624354805 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fcef00094c0 con 0x7fcedc0186c0 2021-02-10T11:06:24.196 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:24.197 7fcf028c5700 1 -- 172.21.2.102:0/624354805 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcedc0186c0 msgr2=0x7fcedc01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:06:24.196 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:24.197 7fcf028c5700 1 --2- 172.21.2.102:0/624354805 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcedc0186c0 0x7fcedc01abc0 crc :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:06:24.196 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:24.197 7fcf028c5700 1 -- 172.21.2.102:0/624354805 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcefc0b7c20 msgr2=0x7fcefc0b8410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:06:24.196 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:24.197 7fcf028c5700 1 --2- 172.21.2.102:0/624354805 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcefc0b7c20 0x7fcefc0b8410 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 rx=0x7fceec006f60 tx=0x7fceec00b970).stop 2021-02-10T11:06:24.196 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:24.197 7fcf028c5700 1 -- 172.21.2.102:0/624354805 shutdown_connections 2021-02-10T11:06:24.196 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:24.197 7fcf028c5700 1 --2- 172.21.2.102:0/624354805 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcefc0c25a0 0x7fcefc0c29a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:06:24.196 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:24.197 7fcf028c5700 1 --2- 172.21.2.102:0/624354805 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcedc0186c0 0x7fcedc01abc0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:06:24.196 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:24.197 7fcf028c5700 1 --2- 172.21.2.102:0/624354805 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fcefc0b8a30 0x7fcefc0c1f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:06:24.196 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:24.197 7fcf028c5700 1 --2- 172.21.2.102:0/624354805 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcefc0b7c20 0x7fcefc0b8410 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:06:24.196 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:24.197 7fcf028c5700 1 -- 172.21.2.102:0/624354805 >> 172.21.2.102:0/624354805 conn(0x7fcefc094e10 msgr2=0x7fcefc0a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:06:24.197 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:24.198 7fcf028c5700 1 -- 172.21.2.102:0/624354805 shutdown_connections 2021-02-10T11:06:24.197 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:24.198 7fcf028c5700 1 -- 172.21.2.102:0/624354805 wait complete. 2021-02-10T11:06:24.204 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000344369 2021-02-10T11:06:24.633 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:24.618 7f48e0243700 1 Processor -- start 2021-02-10T11:06:24.633 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:24.618 7f48e0243700 1 -- start start 2021-02-10T11:06:24.633 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:24.619 7f48e0243700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f48d80f05f0 0x7f48d80f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:06:24.633 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:24.619 7f48e0243700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48d81b0c70 con 0x7f48d81b0470 2021-02-10T11:06:24.633 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:24.619 7f48e0243700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48d80f0b50 con 0x7f48d81b0870 2021-02-10T11:06:24.633 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:24.619 7f48e0243700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48d80e5ea0 con 0x7f48d80f05f0 2021-02-10T11:06:24.633 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:24.619 7f48de05f700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f48d81b0470 0x7f48d80ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:43292/0 (socket says 172.21.2.102:43292) 2021-02-10T11:06:24.633 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:24.619 7f48de05f700 1 -- 172.21.2.102:0/1283361748 learned_addr learned my addr 172.21.2.102:0/1283361748 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:06:24.633 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:24.619 7f48de860700 1 --2- 172.21.2.102:0/1283361748 >> v2:172.21.2.102:3300/0 conn(0x7f48d80f05f0 0x7f48d80f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:06:24.634 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:24.619 7f48de860700 1 -- 172.21.2.102:0/1283361748 >> v1:172.21.2.113:6789/0 conn(0x7f48d81b0870 legacy=0x7f48d80efe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:06:24.634 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:24.619 7f48de860700 1 -- 172.21.2.102:0/1283361748 >> v1:172.21.2.102:6789/0 conn(0x7f48d81b0470 legacy=0x7f48d80ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:06:24.634 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:24.619 7f48de860700 1 -- 172.21.2.102:0/1283361748 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48d80e52e0 con 0x7f48d80f05f0 2021-02-10T11:06:24.634 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:24.620 7f48de860700 1 --2- 172.21.2.102:0/1283361748 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f48d80f05f0 0x7f48d80f20e0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 rx=0x7f48d0006e30 tx=0x7f48d0009e60).ready entity=mon.0 client_cookie=be13f274a6d8c4fc server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:06:24.634 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:24.620 7f48dd05d700 1 -- 172.21.2.102:0/1283361748 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f48d000b300 con 0x7f48d80f05f0 2021-02-10T11:06:24.634 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:24.620 7f48dd05d700 1 -- 172.21.2.102:0/1283361748 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48d0000d30 con 0x7f48d80f05f0 2021-02-10T11:06:24.634 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:24.620 7f48dd05d700 1 -- 172.21.2.102:0/1283361748 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f48d000b800 con 0x7f48d80f05f0 2021-02-10T11:06:24.634 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:24.620 7f48e0243700 1 -- 172.21.2.102:0/1283361748 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f48d80f05f0 msgr2=0x7f48d80f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:06:24.634 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:24.620 7f48e0243700 1 --2- 172.21.2.102:0/1283361748 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f48d80f05f0 0x7f48d80f20e0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 rx=0x7f48d0006e30 tx=0x7f48d0009e60).stop 2021-02-10T11:06:24.634 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:24.620 7f48e0243700 1 -- 172.21.2.102:0/1283361748 shutdown_connections 2021-02-10T11:06:24.634 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:24.620 7f48e0243700 1 --2- 172.21.2.102:0/1283361748 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f48d80f05f0 0x7f48d80f20e0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:06:24.634 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:24.620 7f48e0243700 1 -- 172.21.2.102:0/1283361748 >> 172.21.2.102:0/1283361748 conn(0x7f48d80ce3e0 msgr2=0x7f48d80e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:06:24.634 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:24.620 7f48e0243700 1 -- 172.21.2.102:0/1283361748 shutdown_connections 2021-02-10T11:06:24.635 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:24.620 7f48e0243700 1 -- 172.21.2.102:0/1283361748 wait complete. 2021-02-10T11:06:24.636 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:24.635 7f48e0243700 1 Processor -- start 2021-02-10T11:06:24.636 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:24.635 7f48e0243700 1 -- start start 2021-02-10T11:06:24.636 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:24.635 7f48e0243700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f48d80a3b50 0x7f48d81b3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:06:24.636 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:24.635 7f48e0243700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f48d81c25b0 0x7f48d81b3d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:06:24.636 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:24.635 7f48e0243700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f48d81b4350 0x7f48d81b18a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:06:24.636 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:24.635 7f48e0243700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48d81b3190 con 0x7f48d81c25b0 2021-02-10T11:06:24.636 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:24.635 7f48e0243700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48d8087750 con 0x7f48d81b4350 2021-02-10T11:06:24.636 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:24.635 7f48e0243700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48d80a6bf0 con 0x7f48d80a3b50 2021-02-10T11:06:24.637 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:24.636 7f48dd85e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f48d81c25b0 0x7f48d81b3d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:06:24.637 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:24.636 7f48dd85e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f48d81c25b0 0x7f48d81b3d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:51076/0 (socket says 172.21.2.102:51076) 2021-02-10T11:06:24.637 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:24.636 7f48dd85e700 1 -- 172.21.2.102:0/2769981480 learned_addr learned my addr 172.21.2.102:0/2769981480 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:06:24.637 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:24.636 7f48dd85e700 1 -- 172.21.2.102:0/2769981480 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f48d80a3b50 msgr2=0x7f48d81b3710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:06:24.637 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:24.636 7f48dd85e700 1 --2- 172.21.2.102:0/2769981480 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f48d80a3b50 0x7f48d81b3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:06:24.637 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:24.636 7f48dd85e700 1 -- 172.21.2.102:0/2769981480 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f48d81b4350 msgr2=0x7f48d81b18a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:06:24.637 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:24.636 7f48dd85e700 1 --2- 172.21.2.102:0/2769981480 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f48d81b4350 0x7f48d81b18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:06:24.637 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:24.636 7f48dd85e700 1 -- 172.21.2.102:0/2769981480 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48d81b1ec0 con 0x7f48d81c25b0 2021-02-10T11:06:24.637 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:24.636 7f48dd85e700 1 --2- 172.21.2.102:0/2769981480 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f48d81c25b0 0x7f48d81b3d30 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 rx=0x7f48cc00b530 tx=0x7f48cc00b8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:06:24.637 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:24.636 7f48caffd700 1 -- 172.21.2.102:0/2769981480 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f48cc00bcc0 con 0x7f48d81c25b0 2021-02-10T11:06:24.637 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:24.636 7f48caffd700 1 -- 172.21.2.102:0/2769981480 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48cc000e20 con 0x7f48d81c25b0 2021-02-10T11:06:24.637 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:24.636 7f48e0243700 1 -- 172.21.2.102:0/2769981480 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48d81b1ec0 con 0x7f48d81c25b0 2021-02-10T11:06:24.637 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:24.637 7f48e0243700 1 -- 172.21.2.102:0/2769981480 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48d81b2450 con 0x7f48d81c25b0 2021-02-10T11:06:24.638 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:24.637 7f48caffd700 1 -- 172.21.2.102:0/2769981480 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f48cc0062a0 con 0x7f48d81c25b0 2021-02-10T11:06:24.638 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:24.637 7f48caffd700 1 -- 172.21.2.102:0/2769981480 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f48cc006020 con 0x7f48d81c25b0 2021-02-10T11:06:24.638 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:24.637 7f48caffd700 1 --2- 172.21.2.102:0/2769981480 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f48c0018730 0x7f48c001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:06:24.638 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:24.637 7f48caffd700 1 -- 172.21.2.102:0/2769981480 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f48cc032830 con 0x7f48d81c25b0 2021-02-10T11:06:24.638 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:24.637 7f48de05f700 1 --2- 172.21.2.102:0/2769981480 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f48c0018730 0x7f48c001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:06:24.638 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:24.638 7f48de05f700 1 --2- 172.21.2.102:0/2769981480 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f48c0018730 0x7f48c001ac10 crc :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:06:24.667 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:24.669 7f48e0243700 1 -- 172.21.2.102:0/2769981480 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48d807e6b0 con 0x7f48d81c25b0 2021-02-10T11:06:24.669 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:24.671 7f48caffd700 1 -- 172.21.2.102:0/2769981480 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f48cc000940 con 0x7f48d81c25b0 2021-02-10T11:06:24.798 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:24.799 7f48e0243700 1 -- 172.21.2.102:0/2769981480 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344369", "target": ["mgr", ""]}) v1 -- 0x7f48d81b2450 con 0x7f48c0018730 2021-02-10T11:06:24.801 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:24.803 7f48caffd700 1 -- 172.21.2.102:0/2769981480 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f48d4001590 con 0x7f48c0018730 2021-02-10T11:06:24.804 INFO:teuthology.orchestra.run.gibba002.stdout:/volumes/_nogroup/subvolume_0000000000344369/62948779-a796-468d-ad6a-c3c5619e305f 2021-02-10T11:06:24.806 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:24.808 7f48e0243700 1 -- 172.21.2.102:0/2769981480 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f48c0018730 msgr2=0x7f48c001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:06:24.806 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:24.808 7f48e0243700 1 --2- 172.21.2.102:0/2769981480 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f48c0018730 0x7f48c001ac10 crc :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:06:24.806 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:24.808 7f48e0243700 1 -- 172.21.2.102:0/2769981480 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f48d81c25b0 msgr2=0x7f48d81b3d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:06:24.806 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:24.808 7f48e0243700 1 --2- 172.21.2.102:0/2769981480 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f48d81c25b0 0x7f48d81b3d30 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 rx=0x7f48cc00b530 tx=0x7f48cc00b8d0).stop 2021-02-10T11:06:24.807 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:24.808 7f48e0243700 1 -- 172.21.2.102:0/2769981480 shutdown_connections 2021-02-10T11:06:24.807 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:24.808 7f48e0243700 1 --2- 172.21.2.102:0/2769981480 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f48c0018730 0x7f48c001ac10 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:06:24.807 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:24.808 7f48e0243700 1 --2- 172.21.2.102:0/2769981480 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f48d81b4350 0x7f48d81b18a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:06:24.807 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:24.808 7f48e0243700 1 --2- 172.21.2.102:0/2769981480 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f48d81c25b0 0x7f48d81b3d30 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:06:24.807 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:24.808 7f48e0243700 1 --2- 172.21.2.102:0/2769981480 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f48d80a3b50 0x7f48d81b3710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:06:24.807 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:24.808 7f48e0243700 1 -- 172.21.2.102:0/2769981480 >> 172.21.2.102:0/2769981480 conn(0x7f48d80886d0 msgr2=0x7f48d8081c40 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:06:24.807 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:24.808 7f48e0243700 1 -- 172.21.2.102:0/2769981480 shutdown_connections 2021-02-10T11:06:24.807 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:24.808 7f48e0243700 1 -- 172.21.2.102:0/2769981480 wait complete. 2021-02-10T11:06:24.814 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000344369 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000344369/62948779-a796-468d-ad6a-c3c5619e305f 2021-02-10T11:06:24.814 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:06:24.815 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:06:24.815 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:06:25.151 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:06:25.151 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:06:25.152 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344369/62948779-a796-468d-ad6a-c3c5619e305f/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:06:25.531 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:06:25.531 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:06:25.531 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0231213 s, 45.4 MB/s 2021-02-10T11:06:25.532 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:06:25.532 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:06:25.533 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:06:25.868 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:06:25.868 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:06:25.868 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344369/62948779-a796-468d-ad6a-c3c5619e305f/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:06:26.242 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:06:26.242 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:06:26.242 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0233445 s, 44.9 MB/s 2021-02-10T11:06:26.243 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:06:26.244 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:06:26.244 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:06:26.578 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:06:26.578 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:06:26.578 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344369/62948779-a796-468d-ad6a-c3c5619e305f/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:06:26.953 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:06:26.954 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:06:26.954 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0249356 s, 42.1 MB/s 2021-02-10T11:06:26.955 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:06:26.955 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:06:26.955 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:06:27.290 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:06:27.290 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:06:27.291 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344369/62948779-a796-468d-ad6a-c3c5619e305f/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:06:27.665 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:06:27.666 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:06:27.666 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0251902 s, 41.6 MB/s 2021-02-10T11:06:27.667 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:06:27.667 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:06:27.667 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:06:28.000 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:06:28.000 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:06:28.001 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344369/62948779-a796-468d-ad6a-c3c5619e305f/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:06:28.374 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:06:28.374 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:06:28.374 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0261447 s, 40.1 MB/s 2021-02-10T11:06:28.375 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:06:28.375 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:06:28.375 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:06:28.711 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:06:28.711 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:06:28.711 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344369/62948779-a796-468d-ad6a-c3c5619e305f/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:06:29.086 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:06:29.086 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:06:29.086 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0253239 s, 41.4 MB/s 2021-02-10T11:06:29.087 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:06:29.087 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:06:29.088 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:06:29.443 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:06:29.444 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:06:29.444 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344369/62948779-a796-468d-ad6a-c3c5619e305f/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:06:29.833 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:06:29.834 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:06:29.834 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0233033 s, 45.0 MB/s 2021-02-10T11:06:29.835 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:06:29.836 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:06:29.836 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:06:30.172 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:06:30.173 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:06:30.173 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344369/62948779-a796-468d-ad6a-c3c5619e305f/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:06:30.547 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:06:30.547 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:06:30.547 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0227345 s, 46.1 MB/s 2021-02-10T11:06:30.548 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:06:30.548 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:06:30.549 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:06:30.885 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:06:30.885 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:06:30.885 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344369/62948779-a796-468d-ad6a-c3c5619e305f/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:06:31.264 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:06:31.264 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:06:31.264 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0279848 s, 37.5 MB/s 2021-02-10T11:06:31.265 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:06:31.265 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:06:31.265 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:06:31.600 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:06:31.600 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:06:31.601 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344369/62948779-a796-468d-ad6a-c3c5619e305f/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:06:31.973 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:06:31.973 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:06:31.973 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0232261 s, 45.1 MB/s 2021-02-10T11:06:31.974 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000344370 2021-02-10T11:06:32.405 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:32.390 7fd83b30e700 1 Processor -- start 2021-02-10T11:06:32.405 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:32.390 7fd83b30e700 1 -- start start 2021-02-10T11:06:32.405 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:32.390 7fd83b30e700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fd8340f05f0 0x7fd8340f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:06:32.405 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:32.390 7fd83b30e700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8341b0c70 con 0x7fd8341b0870 2021-02-10T11:06:32.405 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:32.390 7fd83b30e700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8340f0b50 con 0x7fd8341b0470 2021-02-10T11:06:32.405 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:32.390 7fd83b30e700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8340e5ea0 con 0x7fd8340f05f0 2021-02-10T11:06:32.405 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:32.391 7fd83992b700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fd8340f05f0 0x7fd8340f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:06:32.405 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:32.391 7fd838929700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fd8341b0870 0x7fd8340efe00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:43304/0 (socket says 172.21.2.102:43304) 2021-02-10T11:06:32.406 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:32.391 7fd83992b700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fd8340f05f0 0x7fd8340f20e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:51084/0 (socket says 172.21.2.102:51084) 2021-02-10T11:06:32.406 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:32.391 7fd83912a700 1 --1- >> v1:172.21.2.113:6789/0 conn(0x7fd8341b0470 0x7fd8340ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.113:6789/0 says I am v1:172.21.2.102:50628/0 (socket says 172.21.2.102:50628) 2021-02-10T11:06:32.406 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:32.391 7fd838929700 1 -- 172.21.2.102:0/1648381825 learned_addr learned my addr 172.21.2.102:0/1648381825 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:06:32.406 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:32.391 7fd83992b700 1 -- 172.21.2.102:0/1648381825 >> v1:172.21.2.113:6789/0 conn(0x7fd8341b0470 legacy=0x7fd8340ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:06:32.406 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:32.391 7fd83992b700 1 -- 172.21.2.102:0/1648381825 >> v1:172.21.2.102:6789/0 conn(0x7fd8341b0870 legacy=0x7fd8340efe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:06:32.406 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:32.391 7fd827fff700 1 -- 172.21.2.102:0/1648381825 <== mon.1 v1:172.21.2.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 942308892 0 0) 0x7fd830000fb0 con 0x7fd8341b0470 2021-02-10T11:06:32.406 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:32.391 7fd83992b700 1 -- 172.21.2.102:0/1648381825 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8340e52e0 con 0x7fd8340f05f0 2021-02-10T11:06:32.406 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:32.391 7fd83992b700 1 --2- 172.21.2.102:0/1648381825 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd8340f05f0 0x7fd8340f20e0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 rx=0x7fd82c006e30 tx=0x7fd82c009e60).ready entity=mon.0 client_cookie=a6bc01b7673c1dc3 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:06:32.406 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:32.392 7fd827fff700 1 -- 172.21.2.102:0/1648381825 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd82c006a50 con 0x7fd8340f05f0 2021-02-10T11:06:32.406 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:32.392 7fd827fff700 1 -- 172.21.2.102:0/1648381825 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd82c00b300 con 0x7fd8340f05f0 2021-02-10T11:06:32.406 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:32.392 7fd827fff700 1 -- 172.21.2.102:0/1648381825 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd82c00b7e0 con 0x7fd8340f05f0 2021-02-10T11:06:32.406 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:32.392 7fd83b30e700 1 -- 172.21.2.102:0/1648381825 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd8340f05f0 msgr2=0x7fd8340f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:06:32.407 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:32.392 7fd83b30e700 1 --2- 172.21.2.102:0/1648381825 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd8340f05f0 0x7fd8340f20e0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 rx=0x7fd82c006e30 tx=0x7fd82c009e60).stop 2021-02-10T11:06:32.407 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:32.392 7fd83b30e700 1 -- 172.21.2.102:0/1648381825 shutdown_connections 2021-02-10T11:06:32.407 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:32.392 7fd83b30e700 1 --2- 172.21.2.102:0/1648381825 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd8340f05f0 0x7fd8340f20e0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:06:32.407 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:32.392 7fd83b30e700 1 -- 172.21.2.102:0/1648381825 >> 172.21.2.102:0/1648381825 conn(0x7fd8340ce3e0 msgr2=0x7fd8340e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:06:32.407 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:32.393 7fd83b30e700 1 -- 172.21.2.102:0/1648381825 shutdown_connections 2021-02-10T11:06:32.407 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:32.393 7fd83b30e700 1 -- 172.21.2.102:0/1648381825 wait complete. 2021-02-10T11:06:32.408 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:32.407 7fd83b30e700 1 Processor -- start 2021-02-10T11:06:32.409 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:32.407 7fd83b30e700 1 -- start start 2021-02-10T11:06:32.409 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:32.407 7fd83b30e700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd834094fb0 0x7fd83409e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:06:32.409 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:32.407 7fd83b30e700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd8340c2be0 0x7fd8340c2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:06:32.409 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:32.408 7fd83b30e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd8340c3600 0x7fd8340a9c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:06:32.409 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:32.408 7fd83b30e700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd83409ebf0 con 0x7fd8340c3600 2021-02-10T11:06:32.409 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:32.408 7fd83b30e700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8340aa2a0 con 0x7fd834094fb0 2021-02-10T11:06:32.409 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:32.408 7fd83b30e700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8340aa4c0 con 0x7fd8340c2be0 2021-02-10T11:06:32.409 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:32.408 7fd83992b700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd8340c3600 0x7fd8340a9c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:06:32.409 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:32.408 7fd83992b700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd8340c3600 0x7fd8340a9c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:51086/0 (socket says 172.21.2.102:51086) 2021-02-10T11:06:32.409 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:32.408 7fd83992b700 1 -- 172.21.2.102:0/3581795134 learned_addr learned my addr 172.21.2.102:0/3581795134 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:06:32.409 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:32.408 7fd83992b700 1 -- 172.21.2.102:0/3581795134 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd8340c2be0 msgr2=0x7fd8340c2fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:06:32.409 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:32.408 7fd83992b700 1 --2- 172.21.2.102:0/3581795134 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd8340c2be0 0x7fd8340c2fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:06:32.409 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:32.408 7fd83992b700 1 -- 172.21.2.102:0/3581795134 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd834094fb0 msgr2=0x7fd83409e5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:06:32.410 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:32.408 7fd83992b700 1 --2- 172.21.2.102:0/3581795134 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd834094fb0 0x7fd83409e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:06:32.410 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:32.408 7fd83992b700 1 -- 172.21.2.102:0/3581795134 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8340aa6e0 con 0x7fd8340c3600 2021-02-10T11:06:32.410 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:32.408 7fd83992b700 1 --2- 172.21.2.102:0/3581795134 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd8340c3600 0x7fd8340a9c80 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 rx=0x7fd82c009f40 tx=0x7fd82c00bd40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:06:32.410 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:32.408 7fd825ffb700 1 -- 172.21.2.102:0/3581795134 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd82c001270 con 0x7fd8340c3600 2021-02-10T11:06:32.410 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:32.408 7fd825ffb700 1 -- 172.21.2.102:0/3581795134 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd82c006df0 con 0x7fd8340c3600 2021-02-10T11:06:32.410 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:32.408 7fd83b30e700 1 -- 172.21.2.102:0/3581795134 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8340aa6e0 con 0x7fd8340c3600 2021-02-10T11:06:32.410 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:32.408 7fd83b30e700 1 -- 172.21.2.102:0/3581795134 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8340aac70 con 0x7fd8340c3600 2021-02-10T11:06:32.410 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:32.409 7fd825ffb700 1 -- 172.21.2.102:0/3581795134 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd82c001770 con 0x7fd8340c3600 2021-02-10T11:06:32.410 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:32.409 7fd825ffb700 1 -- 172.21.2.102:0/3581795134 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd82c001c40 con 0x7fd8340c3600 2021-02-10T11:06:32.410 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:32.409 7fd825ffb700 1 --2- 172.21.2.102:0/3581795134 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd8140186c0 0x7fd81401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:06:32.410 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:32.409 7fd825ffb700 1 -- 172.21.2.102:0/3581795134 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fd82c032910 con 0x7fd8340c3600 2021-02-10T11:06:32.410 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:32.409 7fd83912a700 1 --2- 172.21.2.102:0/3581795134 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd8140186c0 0x7fd81401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:06:32.411 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:32.410 7fd83912a700 1 --2- 172.21.2.102:0/3581795134 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd8140186c0 0x7fd81401abc0 crc :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:06:32.438 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:32.440 7fd83b30e700 1 -- 172.21.2.102:0/3581795134 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8340a9750 con 0x7fd8340c3600 2021-02-10T11:06:32.440 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:32.442 7fd825ffb700 1 -- 172.21.2.102:0/3581795134 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd82c001930 con 0x7fd8340c3600 2021-02-10T11:06:32.569 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:32.571 7fd83b30e700 1 -- 172.21.2.102:0/3581795134 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344370", "target": ["mgr", ""]}) v1 -- 0x7fd8340aa2a0 con 0x7fd8140186c0 2021-02-10T11:06:32.595 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:32.597 7fd825ffb700 1 -- 172.21.2.102:0/3581795134 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fd830000fb0 con 0x7fd8140186c0 2021-02-10T11:06:32.602 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:32.603 7fd83b30e700 1 -- 172.21.2.102:0/3581795134 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd8140186c0 msgr2=0x7fd81401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:06:32.602 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:32.603 7fd83b30e700 1 --2- 172.21.2.102:0/3581795134 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd8140186c0 0x7fd81401abc0 crc :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:06:32.602 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:32.603 7fd83b30e700 1 -- 172.21.2.102:0/3581795134 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd8340c3600 msgr2=0x7fd8340a9c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:06:32.602 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:32.603 7fd83b30e700 1 --2- 172.21.2.102:0/3581795134 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd8340c3600 0x7fd8340a9c80 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 rx=0x7fd82c009f40 tx=0x7fd82c00bd40).stop 2021-02-10T11:06:32.603 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:32.604 7fd83b30e700 1 -- 172.21.2.102:0/3581795134 shutdown_connections 2021-02-10T11:06:32.603 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:32.604 7fd83b30e700 1 --2- 172.21.2.102:0/3581795134 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd8340c3600 0x7fd8340a9c80 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:06:32.603 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:32.604 7fd83b30e700 1 --2- 172.21.2.102:0/3581795134 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd8340c2be0 0x7fd8340c2fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:06:32.603 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:32.604 7fd83b30e700 1 --2- 172.21.2.102:0/3581795134 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd8140186c0 0x7fd81401abc0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:06:32.603 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:32.604 7fd83b30e700 1 --2- 172.21.2.102:0/3581795134 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd834094fb0 0x7fd83409e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:06:32.603 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:32.604 7fd83b30e700 1 -- 172.21.2.102:0/3581795134 >> 172.21.2.102:0/3581795134 conn(0x7fd83409f630 msgr2=0x7fd8340a0130 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:06:32.603 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:32.604 7fd83b30e700 1 -- 172.21.2.102:0/3581795134 shutdown_connections 2021-02-10T11:06:32.603 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:32.604 7fd83b30e700 1 -- 172.21.2.102:0/3581795134 wait complete. 2021-02-10T11:06:32.610 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000344370 2021-02-10T11:06:33.041 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:33.025 7fd718b97700 1 Processor -- start 2021-02-10T11:06:33.041 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:33.025 7fd718b97700 1 -- start start 2021-02-10T11:06:33.041 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:33.026 7fd718b97700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fd7141b0470 0x7fd7141b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:06:33.042 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:33.026 7fd718b97700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7140f0b50 con 0x7fd7140ef610 2021-02-10T11:06:33.042 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:33.026 7fd718b97700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7140f24e0 con 0x7fd7140f20e0 2021-02-10T11:06:33.042 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:33.026 7fd718b97700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7140e5ea0 con 0x7fd7141b0470 2021-02-10T11:06:33.042 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:33.026 7fd71205d700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fd7140ef610 0x7fd7140efa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:43322/0 (socket says 172.21.2.102:43322) 2021-02-10T11:06:33.042 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:33.026 7fd71205d700 1 -- 172.21.2.102:0/779650499 learned_addr learned my addr 172.21.2.102:0/779650499 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:06:33.042 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:33.027 7fd71185c700 1 -- 172.21.2.102:0/779650499 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4275584736 0 0) 0x7fd6fc0010b0 con 0x7fd7140ef610 2021-02-10T11:06:33.042 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:33.027 7fd71185c700 1 -- 172.21.2.102:0/779650499 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd700002fe0 con 0x7fd7140ef610 2021-02-10T11:06:33.042 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:33.027 7fd71285e700 1 --2- 172.21.2.102:0/779650499 >> v2:172.21.2.113:3300/0 conn(0x7fd7141b0470 0x7fd7141b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:06:33.042 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:33.027 7fd71185c700 1 -- 172.21.2.102:0/779650499 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3594249697 0 0) 0x7fd6fc0010b0 con 0x7fd7140ef610 2021-02-10T11:06:33.042 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:33.027 7fd71185c700 1 -- 172.21.2.102:0/779650499 >> v2:172.21.2.113:3300/0 conn(0x7fd7141b0470 msgr2=0x7fd7141b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:06:33.042 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:33.027 7fd71185c700 1 --2- 172.21.2.102:0/779650499 >> v2:172.21.2.113:3300/0 conn(0x7fd7141b0470 0x7fd7141b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:06:33.042 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:33.027 7fd71185c700 1 -- 172.21.2.102:0/779650499 >> v1:172.21.2.113:6789/0 conn(0x7fd7140f20e0 legacy=0x7fd7140f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:06:33.043 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:33.027 7fd71185c700 1 -- 172.21.2.102:0/779650499 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7140e52e0 con 0x7fd7140ef610 2021-02-10T11:06:33.043 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:33.027 7fd71185c700 1 -- 172.21.2.102:0/779650499 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7fd6fc0015b0 con 0x7fd7140ef610 2021-02-10T11:06:33.043 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:33.027 7fd71185c700 1 -- 172.21.2.102:0/779650499 >> v1:172.21.2.102:6789/0 conn(0x7fd7140ef610 legacy=0x7fd7140efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:06:33.043 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:33.027 7fd71185c700 1 --2- 172.21.2.102:0/779650499 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd700003b10 0x7fd700006010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:06:33.043 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:33.027 7fd71185c700 1 --2- 172.21.2.102:0/779650499 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd7000067a0 0x7fd700008ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:06:33.043 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:33.027 7fd71185c700 1 --2- 172.21.2.102:0/779650499 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd700009580 0x7fd70000baa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:06:33.043 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:33.027 7fd71185c700 1 -- 172.21.2.102:0/779650499 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd70000c2f0 con 0x7fd700009580 2021-02-10T11:06:33.043 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:33.027 7fd71185c700 1 -- 172.21.2.102:0/779650499 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd70000c510 con 0x7fd700003b10 2021-02-10T11:06:33.043 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:33.027 7fd71185c700 1 -- 172.21.2.102:0/779650499 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd70000c730 con 0x7fd7000067a0 2021-02-10T11:06:33.043 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:33.027 7fd71305f700 1 --2- 172.21.2.102:0/779650499 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd700009580 0x7fd70000baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:06:33.043 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:33.027 7fd71285e700 1 --2- 172.21.2.102:0/779650499 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd700003b10 0x7fd700006010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:06:33.043 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:33.027 7fd71205d700 1 --2- 172.21.2.102:0/779650499 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd7000067a0 0x7fd700008ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:06:33.043 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:33.027 7fd71305f700 1 -- 172.21.2.102:0/779650499 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd7000067a0 msgr2=0x7fd700008ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:06:33.044 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:33.027 7fd71305f700 1 --2- 172.21.2.102:0/779650499 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd7000067a0 0x7fd700008ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:06:33.044 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:33.027 7fd71305f700 1 -- 172.21.2.102:0/779650499 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd700003b10 msgr2=0x7fd700006010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:06:33.044 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:33.027 7fd71305f700 1 --2- 172.21.2.102:0/779650499 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd700003b10 0x7fd700006010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:06:33.044 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:33.027 7fd71305f700 1 -- 172.21.2.102:0/779650499 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7000509a0 con 0x7fd700009580 2021-02-10T11:06:33.044 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:33.027 7fd71285e700 1 --2- 172.21.2.102:0/779650499 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd700003b10 0x7fd700006010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:06:33.044 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:33.027 7fd71305f700 1 -- 172.21.2.102:0/779650499 reap_dead start 2021-02-10T11:06:33.044 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:33.027 7fd71305f700 1 --2- 172.21.2.102:0/779650499 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd700009580 0x7fd70000baa0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 rx=0x7fd70c006e40 tx=0x7fd70c009e30).ready entity=mon.0 client_cookie=23eb5244cd1f7d71 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:06:33.044 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:33.028 7fd71185c700 1 -- 172.21.2.102:0/779650499 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd70c006a30 con 0x7fd700009580 2021-02-10T11:06:33.044 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:33.028 7fd71185c700 1 -- 172.21.2.102:0/779650499 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd70c00b300 con 0x7fd700009580 2021-02-10T11:06:33.044 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:33.028 7fd71185c700 1 -- 172.21.2.102:0/779650499 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd70c00b7e0 con 0x7fd700009580 2021-02-10T11:06:33.044 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:33.028 7fd718b97700 1 -- 172.21.2.102:0/779650499 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd700009580 msgr2=0x7fd70000baa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:06:33.044 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:33.028 7fd718b97700 1 --2- 172.21.2.102:0/779650499 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd700009580 0x7fd70000baa0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 rx=0x7fd70c006e40 tx=0x7fd70c009e30).stop 2021-02-10T11:06:33.045 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:33.028 7fd718b97700 1 -- 172.21.2.102:0/779650499 shutdown_connections 2021-02-10T11:06:33.046 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:33.028 7fd718b97700 1 --2- 172.21.2.102:0/779650499 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd700009580 0x7fd70000baa0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:06:33.046 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:33.028 7fd718b97700 1 -- 172.21.2.102:0/779650499 >> 172.21.2.102:0/779650499 conn(0x7fd7140ce3e0 msgr2=0x7fd7140e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:06:33.046 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:33.028 7fd718b97700 1 -- 172.21.2.102:0/779650499 shutdown_connections 2021-02-10T11:06:33.046 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:33.028 7fd718b97700 1 -- 172.21.2.102:0/779650499 wait complete. 2021-02-10T11:06:33.046 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:33.043 7fd718b97700 1 Processor -- start 2021-02-10T11:06:33.046 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:33.043 7fd718b97700 1 -- start start 2021-02-10T11:06:33.047 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:33.043 7fd718b97700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd7140b7db0 0x7fd7140b8260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:06:33.047 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:33.043 7fd718b97700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd7140b8880 0x7fd7141a7450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:06:33.048 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:33.043 7fd718b97700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd7140b8c80 0x7fd7141a7a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:06:33.048 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:33.043 7fd718b97700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7140b79b0 con 0x7fd7140b7db0 2021-02-10T11:06:33.048 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:33.043 7fd718b97700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7141a8090 con 0x7fd7140b8c80 2021-02-10T11:06:33.048 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:33.043 7fd718b97700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7141a82b0 con 0x7fd7140b8880 2021-02-10T11:06:33.049 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:33.043 7fd71285e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd7140b7db0 0x7fd7140b8260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:06:33.049 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:33.044 7fd71285e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd7140b7db0 0x7fd7140b8260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:51108/0 (socket says 172.21.2.102:51108) 2021-02-10T11:06:33.049 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:33.044 7fd71285e700 1 -- 172.21.2.102:0/2640540449 learned_addr learned my addr 172.21.2.102:0/2640540449 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:06:33.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:33.044 7fd71305f700 1 --2- 172.21.2.102:0/2640540449 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd7140b8c80 0x7fd7141a7a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:06:33.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:33.044 7fd71285e700 1 -- 172.21.2.102:0/2640540449 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd7140b8880 msgr2=0x7fd7141a7450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:06:33.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:33.044 7fd71205d700 1 --2- 172.21.2.102:0/2640540449 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd7140b8880 0x7fd7141a7450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:06:33.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:33.044 7fd71285e700 1 --2- 172.21.2.102:0/2640540449 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd7140b8880 0x7fd7141a7450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:06:33.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:33.044 7fd71285e700 1 -- 172.21.2.102:0/2640540449 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd7140b8c80 msgr2=0x7fd7141a7a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:06:33.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:33.044 7fd71285e700 1 --2- 172.21.2.102:0/2640540449 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd7140b8c80 0x7fd7141a7a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:06:33.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:33.044 7fd71285e700 1 -- 172.21.2.102:0/2640540449 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd714190c10 con 0x7fd7140b7db0 2021-02-10T11:06:33.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:33.044 7fd71205d700 1 --2- 172.21.2.102:0/2640540449 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd7140b8880 0x7fd7141a7450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:06:33.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:33.044 7fd71305f700 1 --2- 172.21.2.102:0/2640540449 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd7140b8c80 0x7fd7141a7a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:06:33.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:33.044 7fd71285e700 1 --2- 172.21.2.102:0/2640540449 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd7140b7db0 0x7fd7140b8260 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 rx=0x7fd70400b5e0 tx=0x7fd70400b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:06:33.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:33.044 7fd70b7fe700 1 -- 172.21.2.102:0/2640540449 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd70400bd70 con 0x7fd7140b7db0 2021-02-10T11:06:33.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:33.044 7fd70b7fe700 1 -- 172.21.2.102:0/2640540449 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd704000c90 con 0x7fd7140b7db0 2021-02-10T11:06:33.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:33.044 7fd70b7fe700 1 -- 172.21.2.102:0/2640540449 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd7040022b0 con 0x7fd7140b7db0 2021-02-10T11:06:33.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:33.044 7fd718b97700 1 -- 172.21.2.102:0/2640540449 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd714190c10 con 0x7fd7140b7db0 2021-02-10T11:06:33.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:33.044 7fd718b97700 1 -- 172.21.2.102:0/2640540449 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7140c1eb0 con 0x7fd7140b7db0 2021-02-10T11:06:33.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:33.045 7fd70b7fe700 1 -- 172.21.2.102:0/2640540449 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd7040052e0 con 0x7fd7140b7db0 2021-02-10T11:06:33.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:33.045 7fd70b7fe700 1 --2- 172.21.2.102:0/2640540449 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd6f80186c0 0x7fd6f801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:06:33.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:33.045 7fd70b7fe700 1 -- 172.21.2.102:0/2640540449 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fd7040325c0 con 0x7fd7140b7db0 2021-02-10T11:06:33.052 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:33.045 7fd71205d700 1 --2- 172.21.2.102:0/2640540449 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd6f80186c0 0x7fd6f801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:06:33.052 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:33.046 7fd71205d700 1 --2- 172.21.2.102:0/2640540449 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd6f80186c0 0x7fd6f801abc0 crc :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:06:33.074 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:33.076 7fd718b97700 1 -- 172.21.2.102:0/2640540449 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd714105b30 con 0x7fd7140b7db0 2021-02-10T11:06:33.077 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:33.079 7fd70b7fe700 1 -- 172.21.2.102:0/2640540449 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd7040009b0 con 0x7fd7140b7db0 2021-02-10T11:06:33.206 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:33.208 7fd718b97700 1 -- 172.21.2.102:0/2640540449 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344370", "target": ["mgr", ""]}) v1 -- 0x7fd7140c1eb0 con 0x7fd6f80186c0 2021-02-10T11:06:33.211 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:33.211 7fd70b7fe700 1 -- 172.21.2.102:0/2640540449 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7fd6fc000960 con 0x7fd6f80186c0 2021-02-10T11:06:33.211 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:33.212 7fd718b97700 1 -- 172.21.2.102:0/2640540449 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd6f80186c0 msgr2=0x7fd6f801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:06:33.211 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:33.212 7fd718b97700 1 --2- 172.21.2.102:0/2640540449 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd6f80186c0 0x7fd6f801abc0 crc :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:06:33.211 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:33.212 7fd718b97700 1 -- 172.21.2.102:0/2640540449 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd7140b7db0 msgr2=0x7fd7140b8260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:06:33.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:33.212 7fd718b97700 1 --2- 172.21.2.102:0/2640540449 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd7140b7db0 0x7fd7140b8260 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 rx=0x7fd70400b5e0 tx=0x7fd70400b980).stop 2021-02-10T11:06:33.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:33.212 7fd718b97700 1 -- 172.21.2.102:0/2640540449 shutdown_connections 2021-02-10T11:06:33.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:33.212 7fd718b97700 1 --2- 172.21.2.102:0/2640540449 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd6f80186c0 0x7fd6f801abc0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:06:33.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:33.212 7fd718b97700 1 --2- 172.21.2.102:0/2640540449 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd7140b8c80 0x7fd7141a7a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:06:33.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:33.212 7fd718b97700 1 --2- 172.21.2.102:0/2640540449 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd7140b8880 0x7fd7141a7450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:06:33.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:33.212 7fd718b97700 1 --2- 172.21.2.102:0/2640540449 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd7140b7db0 0x7fd7140b8260 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:06:33.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:33.212 7fd718b97700 1 -- 172.21.2.102:0/2640540449 >> 172.21.2.102:0/2640540449 conn(0x7fd714094e10 msgr2=0x7fd7140a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:06:33.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:33.212 7fd718b97700 1 -- 172.21.2.102:0/2640540449 shutdown_connections 2021-02-10T11:06:33.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:33.212 7fd718b97700 1 -- 172.21.2.102:0/2640540449 wait complete. 2021-02-10T11:06:33.213 INFO:teuthology.orchestra.run.gibba002.stdout:/volumes/_nogroup/subvolume_0000000000344370/4b22c5c5-0975-44c3-af3a-eb02d4668c78 2021-02-10T11:06:33.219 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000344370 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000344370/4b22c5c5-0975-44c3-af3a-eb02d4668c78 2021-02-10T11:06:33.219 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:06:33.219 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:06:33.220 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:06:33.553 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:06:33.553 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:06:33.554 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344370/4b22c5c5-0975-44c3-af3a-eb02d4668c78/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:06:33.932 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:06:33.932 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:06:33.932 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0240716 s, 43.6 MB/s 2021-02-10T11:06:33.933 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:06:33.933 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:06:33.933 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:06:34.268 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:06:34.268 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:06:34.269 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344370/4b22c5c5-0975-44c3-af3a-eb02d4668c78/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:06:34.645 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:06:34.646 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:06:34.646 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0258281 s, 40.6 MB/s 2021-02-10T11:06:34.647 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:06:34.647 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:06:34.647 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:06:34.981 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:06:34.981 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:06:34.981 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344370/4b22c5c5-0975-44c3-af3a-eb02d4668c78/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:06:35.355 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:06:35.355 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:06:35.355 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0245342 s, 42.7 MB/s 2021-02-10T11:06:35.356 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:06:35.356 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:06:35.356 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:06:35.691 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:06:35.691 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:06:35.692 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344370/4b22c5c5-0975-44c3-af3a-eb02d4668c78/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:06:36.065 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:06:36.065 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:06:36.065 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0240163 s, 43.7 MB/s 2021-02-10T11:06:36.066 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:06:36.066 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:06:36.066 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:06:36.401 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:06:36.401 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:06:36.402 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344370/4b22c5c5-0975-44c3-af3a-eb02d4668c78/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:06:36.776 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:06:36.776 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:06:36.777 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.024715 s, 42.4 MB/s 2021-02-10T11:06:36.778 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:06:36.778 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:06:36.778 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:06:37.111 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:06:37.112 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:06:37.112 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344370/4b22c5c5-0975-44c3-af3a-eb02d4668c78/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:06:37.487 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:06:37.488 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:06:37.488 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0250355 s, 41.9 MB/s 2021-02-10T11:06:37.488 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:06:37.489 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:06:37.489 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:06:37.822 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:06:37.823 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:06:37.823 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344370/4b22c5c5-0975-44c3-af3a-eb02d4668c78/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:06:38.195 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:06:38.196 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:06:38.196 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0219975 s, 47.7 MB/s 2021-02-10T11:06:38.197 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:06:38.198 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:06:38.198 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:06:38.532 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:06:38.533 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:06:38.534 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344370/4b22c5c5-0975-44c3-af3a-eb02d4668c78/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:06:38.909 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:06:38.909 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:06:38.910 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.027264 s, 38.5 MB/s 2021-02-10T11:06:38.911 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:06:38.911 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:06:38.912 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:06:39.247 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:06:39.247 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:06:39.247 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344370/4b22c5c5-0975-44c3-af3a-eb02d4668c78/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:06:39.625 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:06:39.625 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:06:39.625 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0276654 s, 37.9 MB/s 2021-02-10T11:06:39.626 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:06:39.626 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:06:39.626 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:06:39.961 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:06:39.962 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:06:39.962 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344370/4b22c5c5-0975-44c3-af3a-eb02d4668c78/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:06:40.337 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:06:40.337 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:06:40.337 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0233044 s, 45.0 MB/s 2021-02-10T11:06:40.338 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000344371 2021-02-10T11:06:40.766 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:40.751 7f0d8ff5d700 1 Processor -- start 2021-02-10T11:06:40.767 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:40.751 7f0d8ff5d700 1 -- start start 2021-02-10T11:06:40.767 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:40.752 7f0d8ff5d700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f0d881b0470 0x7f0d881b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:06:40.767 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:40.752 7f0d8ff5d700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0d880f20e0 0x7f0d880f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:06:40.767 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:40.752 7f0d8ff5d700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d880f0b50 con 0x7f0d880ef610 2021-02-10T11:06:40.767 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:40.752 7f0d8ff5d700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d880f0820 con 0x7f0d881b0470 2021-02-10T11:06:40.767 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:40.752 7f0d8ff5d700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d880f24e0 con 0x7f0d880f20e0 2021-02-10T11:06:40.767 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:40.752 7f0d8dd79700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f0d881b0470 0x7f0d881b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:06:40.767 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:40.752 7f0d8dd79700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f0d881b0470 0x7f0d881b0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:51116/0 (socket says 172.21.2.102:51116) 2021-02-10T11:06:40.767 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:40.752 7f0d8d578700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f0d880ef610 0x7f0d880efa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:43340/0 (socket says 172.21.2.102:43340) 2021-02-10T11:06:40.767 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:40.752 7f0d8dd79700 1 -- 172.21.2.102:0/2995414685 learned_addr learned my addr 172.21.2.102:0/2995414685 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:06:40.767 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:40.752 7f0d8e57a700 1 --2- 172.21.2.102:0/2995414685 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0d880f20e0 0x7f0d880f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:06:40.768 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:40.753 7f0d8dd79700 1 -- 172.21.2.102:0/2995414685 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0d880f20e0 msgr2=0x7f0d880f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:06:40.768 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:40.753 7f0d8dd79700 1 --2- 172.21.2.102:0/2995414685 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0d880f20e0 0x7f0d880f0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:06:40.768 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:40.753 7f0d8dd79700 1 -- 172.21.2.102:0/2995414685 >> v1:172.21.2.102:6789/0 conn(0x7f0d880ef610 legacy=0x7f0d880efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:06:40.768 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:40.753 7f0d8dd79700 1 -- 172.21.2.102:0/2995414685 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d880e52e0 con 0x7f0d881b0470 2021-02-10T11:06:40.768 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:40.753 7f0d8cd77700 1 -- 172.21.2.102:0/2995414685 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1401542117 0 0) 0x7f0d7c0010b0 con 0x7f0d880ef610 2021-02-10T11:06:40.768 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:40.753 7f0d8dd79700 1 --2- 172.21.2.102:0/2995414685 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0d881b0470 0x7f0d881b0870 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 rx=0x7f0d84006700 tx=0x7f0d84009b00).ready entity=mon.0 client_cookie=a1fc432fe94f6d93 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:06:40.768 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:40.753 7f0d8cd77700 1 -- 172.21.2.102:0/2995414685 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0d8400c0c0 con 0x7f0d881b0470 2021-02-10T11:06:40.768 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:40.753 7f0d8cd77700 1 -- 172.21.2.102:0/2995414685 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d8400c340 con 0x7f0d881b0470 2021-02-10T11:06:40.768 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:40.753 7f0d8cd77700 1 -- 172.21.2.102:0/2995414685 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0d8400c820 con 0x7f0d881b0470 2021-02-10T11:06:40.768 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:40.753 7f0d8ff5d700 1 -- 172.21.2.102:0/2995414685 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0d881b0470 msgr2=0x7f0d881b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:06:40.768 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:40.753 7f0d8ff5d700 1 --2- 172.21.2.102:0/2995414685 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0d881b0470 0x7f0d881b0870 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 rx=0x7f0d84006700 tx=0x7f0d84009b00).stop 2021-02-10T11:06:40.768 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:40.754 7f0d8ff5d700 1 -- 172.21.2.102:0/2995414685 shutdown_connections 2021-02-10T11:06:40.768 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:40.754 7f0d8ff5d700 1 --2- 172.21.2.102:0/2995414685 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0d880f20e0 0x7f0d880f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:06:40.769 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:40.754 7f0d8ff5d700 1 --2- 172.21.2.102:0/2995414685 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0d881b0470 0x7f0d881b0870 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:06:40.769 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:40.754 7f0d8ff5d700 1 -- 172.21.2.102:0/2995414685 >> 172.21.2.102:0/2995414685 conn(0x7f0d880ce3e0 msgr2=0x7f0d880e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:06:40.769 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:40.754 7f0d8ff5d700 1 -- 172.21.2.102:0/2995414685 shutdown_connections 2021-02-10T11:06:40.769 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:40.754 7f0d8ff5d700 1 -- 172.21.2.102:0/2995414685 wait complete. 2021-02-10T11:06:40.772 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:40.769 7f0d8ff5d700 1 Processor -- start 2021-02-10T11:06:40.772 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:40.769 7f0d8ff5d700 1 -- start start 2021-02-10T11:06:40.772 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:40.769 7f0d8ff5d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0d88190920 0x7f0d880b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:06:40.772 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:40.769 7f0d8ff5d700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0d880b8200 0x7f0d880b8600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:06:40.772 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:40.769 7f0d8ff5d700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0d880c1d80 0x7f0d880c2180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:06:40.772 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:40.769 7f0d8ff5d700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d88190da0 con 0x7f0d88190920 2021-02-10T11:06:40.773 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:40.769 7f0d8ff5d700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d880b8c20 con 0x7f0d880c1d80 2021-02-10T11:06:40.773 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:40.769 7f0d8ff5d700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d880b8e10 con 0x7f0d880b8200 2021-02-10T11:06:40.773 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:40.770 7f0d8dd79700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0d88190920 0x7f0d880b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:06:40.773 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:40.770 7f0d8dd79700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0d88190920 0x7f0d880b7be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:51118/0 (socket says 172.21.2.102:51118) 2021-02-10T11:06:40.773 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:40.770 7f0d8dd79700 1 -- 172.21.2.102:0/3459823247 learned_addr learned my addr 172.21.2.102:0/3459823247 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:06:40.773 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:40.770 7f0d8dd79700 1 -- 172.21.2.102:0/3459823247 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0d880b8200 msgr2=0x7f0d880b8600 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:06:40.773 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:40.770 7f0d8dd79700 1 --2- 172.21.2.102:0/3459823247 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0d880b8200 0x7f0d880b8600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:06:40.773 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:40.770 7f0d8dd79700 1 -- 172.21.2.102:0/3459823247 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0d880c1d80 msgr2=0x7f0d880c2180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:06:40.773 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:40.770 7f0d8dd79700 1 --2- 172.21.2.102:0/3459823247 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0d880c1d80 0x7f0d880c2180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:06:40.773 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:40.770 7f0d8dd79700 1 -- 172.21.2.102:0/3459823247 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d880c27d0 con 0x7f0d88190920 2021-02-10T11:06:40.773 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:40.770 7f0d8dd79700 1 --2- 172.21.2.102:0/3459823247 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0d88190920 0x7f0d880b7be0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 rx=0x7f0d84006f20 tx=0x7f0d8400b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:06:40.773 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:40.770 7f0d7a7fc700 1 -- 172.21.2.102:0/3459823247 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0d84000c90 con 0x7f0d88190920 2021-02-10T11:06:40.773 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:40.770 7f0d7a7fc700 1 -- 172.21.2.102:0/3459823247 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d84009100 con 0x7f0d88190920 2021-02-10T11:06:40.773 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:40.770 7f0d8ff5d700 1 -- 172.21.2.102:0/3459823247 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d880c27d0 con 0x7f0d88190920 2021-02-10T11:06:40.774 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:40.770 7f0d7a7fc700 1 -- 172.21.2.102:0/3459823247 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0d84002220 con 0x7f0d88190920 2021-02-10T11:06:40.774 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:40.770 7f0d8ff5d700 1 -- 172.21.2.102:0/3459823247 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d880c27d0 con 0x7f0d88190920 2021-02-10T11:06:40.774 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:40.771 7f0d7a7fc700 1 -- 172.21.2.102:0/3459823247 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f0d8400a000 con 0x7f0d88190920 2021-02-10T11:06:40.774 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:40.771 7f0d7a7fc700 1 --2- 172.21.2.102:0/3459823247 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0d70018730 0x7f0d7001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:06:40.774 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:40.771 7f0d7a7fc700 1 -- 172.21.2.102:0/3459823247 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f0d84032620 con 0x7f0d88190920 2021-02-10T11:06:40.774 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:40.771 7f0d8d578700 1 --2- 172.21.2.102:0/3459823247 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0d70018730 0x7f0d7001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:06:40.774 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:40.771 7f0d8d578700 1 --2- 172.21.2.102:0/3459823247 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0d70018730 0x7f0d7001ac10 crc :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:06:40.800 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:40.802 7f0d8ff5d700 1 -- 172.21.2.102:0/3459823247 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d88105ae0 con 0x7f0d88190920 2021-02-10T11:06:40.802 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:40.804 7f0d7a7fc700 1 -- 172.21.2.102:0/3459823247 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f0d84000c90 con 0x7f0d88190920 2021-02-10T11:06:40.930 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:40.932 7f0d8ff5d700 1 -- 172.21.2.102:0/3459823247 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344371", "target": ["mgr", ""]}) v1 -- 0x7f0d88105ae0 con 0x7f0d70018730 2021-02-10T11:06:40.955 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:40.957 7f0d7a7fc700 1 -- 172.21.2.102:0/3459823247 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f0d7c0014b0 con 0x7f0d70018730 2021-02-10T11:06:40.963 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:40.965 7f0d8ff5d700 1 -- 172.21.2.102:0/3459823247 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0d70018730 msgr2=0x7f0d7001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:06:40.963 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:40.965 7f0d8ff5d700 1 --2- 172.21.2.102:0/3459823247 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0d70018730 0x7f0d7001ac10 crc :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:06:40.963 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:40.965 7f0d8ff5d700 1 -- 172.21.2.102:0/3459823247 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0d88190920 msgr2=0x7f0d880b7be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:06:40.964 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:40.965 7f0d8ff5d700 1 --2- 172.21.2.102:0/3459823247 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0d88190920 0x7f0d880b7be0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 rx=0x7f0d84006f20 tx=0x7f0d8400b990).stop 2021-02-10T11:06:40.964 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:40.965 7f0d8ff5d700 1 -- 172.21.2.102:0/3459823247 shutdown_connections 2021-02-10T11:06:40.964 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:40.965 7f0d8ff5d700 1 --2- 172.21.2.102:0/3459823247 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0d70018730 0x7f0d7001ac10 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:06:40.964 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:40.965 7f0d8ff5d700 1 --2- 172.21.2.102:0/3459823247 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0d880c1d80 0x7f0d880c2180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:06:40.964 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:40.965 7f0d8ff5d700 1 --2- 172.21.2.102:0/3459823247 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0d880b8200 0x7f0d880b8600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:06:40.964 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:40.965 7f0d8ff5d700 1 --2- 172.21.2.102:0/3459823247 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0d88190920 0x7f0d880b7be0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:06:40.964 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:40.965 7f0d8ff5d700 1 -- 172.21.2.102:0/3459823247 >> 172.21.2.102:0/3459823247 conn(0x7f0d88094e10 msgr2=0x7f0d880a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:06:40.964 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:40.965 7f0d8ff5d700 1 -- 172.21.2.102:0/3459823247 shutdown_connections 2021-02-10T11:06:40.964 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:40.965 7f0d8ff5d700 1 -- 172.21.2.102:0/3459823247 wait complete. 2021-02-10T11:06:40.972 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000344371 2021-02-10T11:06:41.401 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:41.387 7fa55c288700 1 Processor -- start 2021-02-10T11:06:41.402 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:41.387 7fa55c288700 1 -- start start 2021-02-10T11:06:41.402 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:41.387 7fa55c288700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fa5541b0470 0x7fa5541b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:06:41.402 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:41.387 7fa55c288700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fa5540f20e0 0x7fa5540f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:06:41.402 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:41.387 7fa55c288700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5540f0b50 con 0x7fa5540ef610 2021-02-10T11:06:41.402 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:41.387 7fa55c288700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5540f0820 con 0x7fa5541b0470 2021-02-10T11:06:41.402 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:41.387 7fa55c288700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5540f24e0 con 0x7fa5540f20e0 2021-02-10T11:06:41.402 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:41.387 7fa55a0a4700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fa5541b0470 0x7fa5541b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:06:41.402 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:41.387 7fa5598a3700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fa5540ef610 0x7fa5540efa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:43350/0 (socket says 172.21.2.102:43350) 2021-02-10T11:06:41.402 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:41.388 7fa5598a3700 1 -- 172.21.2.102:0/3466181647 learned_addr learned my addr 172.21.2.102:0/3466181647 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:06:41.402 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:41.388 7fa55a8a5700 1 --2- 172.21.2.102:0/3466181647 >> v2:172.21.2.113:3300/0 conn(0x7fa5540f20e0 0x7fa5540f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:06:41.403 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:41.388 7fa55a0a4700 1 -- 172.21.2.102:0/3466181647 >> v2:172.21.2.113:3300/0 conn(0x7fa5540f20e0 msgr2=0x7fa5540f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:06:41.403 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:41.388 7fa55a0a4700 1 --2- 172.21.2.102:0/3466181647 >> v2:172.21.2.113:3300/0 conn(0x7fa5540f20e0 0x7fa5540f0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:06:41.403 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:41.388 7fa55a0a4700 1 -- 172.21.2.102:0/3466181647 >> v1:172.21.2.102:6789/0 conn(0x7fa5540ef610 legacy=0x7fa5540efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:06:41.403 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:41.388 7fa55a0a4700 1 -- 172.21.2.102:0/3466181647 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5540e52e0 con 0x7fa5541b0470 2021-02-10T11:06:41.403 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:41.388 7fa55a0a4700 1 --2- 172.21.2.102:0/3466181647 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa5541b0470 0x7fa5541b0870 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 rx=0x7fa548006650 tx=0x7fa548009a80).ready entity=mon.0 client_cookie=9ce35b593179a0d8 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:06:41.403 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:41.388 7fa5590a2700 1 -- 172.21.2.102:0/3466181647 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa54800c0c0 con 0x7fa5541b0470 2021-02-10T11:06:41.403 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:41.388 7fa5590a2700 1 -- 172.21.2.102:0/3466181647 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa54800c340 con 0x7fa5541b0470 2021-02-10T11:06:41.403 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:41.388 7fa5590a2700 1 -- 172.21.2.102:0/3466181647 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa54800c860 con 0x7fa5541b0470 2021-02-10T11:06:41.403 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:41.388 7fa55c288700 1 -- 172.21.2.102:0/3466181647 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa5541b0470 msgr2=0x7fa5541b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:06:41.403 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:41.388 7fa55c288700 1 --2- 172.21.2.102:0/3466181647 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa5541b0470 0x7fa5541b0870 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 rx=0x7fa548006650 tx=0x7fa548009a80).stop 2021-02-10T11:06:41.403 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:41.389 7fa55c288700 1 -- 172.21.2.102:0/3466181647 shutdown_connections 2021-02-10T11:06:41.403 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:41.389 7fa55c288700 1 --2- 172.21.2.102:0/3466181647 >> v2:172.21.2.113:3300/0 conn(0x7fa5540f20e0 0x7fa5540f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:06:41.404 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:41.389 7fa55c288700 1 --2- 172.21.2.102:0/3466181647 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa5541b0470 0x7fa5541b0870 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:06:41.404 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:41.389 7fa55c288700 1 -- 172.21.2.102:0/3466181647 >> 172.21.2.102:0/3466181647 conn(0x7fa5540ce3e0 msgr2=0x7fa5540e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:06:41.404 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:41.389 7fa55c288700 1 -- 172.21.2.102:0/3466181647 shutdown_connections 2021-02-10T11:06:41.404 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:41.389 7fa55c288700 1 -- 172.21.2.102:0/3466181647 wait complete. 2021-02-10T11:06:41.405 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:41.404 7fa55c288700 1 Processor -- start 2021-02-10T11:06:41.406 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:41.404 7fa55c288700 1 -- start start 2021-02-10T11:06:41.406 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:41.404 7fa55c288700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa5541a0470 0x7fa55408a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:06:41.406 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:41.404 7fa55c288700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa55408aa50 0x7fa554099180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:06:41.406 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:41.404 7fa55c288700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa5540997a0 0x7fa554099ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:06:41.406 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:41.404 7fa55c288700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa554095610 con 0x7fa5541a0470 2021-02-10T11:06:41.406 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:41.404 7fa55c288700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa554095800 con 0x7fa55408aa50 2021-02-10T11:06:41.406 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:41.404 7fa55c288700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa55409a1c0 con 0x7fa5540997a0 2021-02-10T11:06:41.406 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:41.405 7fa5598a3700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa55408aa50 0x7fa554099180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:06:41.406 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:41.405 7fa55a0a4700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa5541a0470 0x7fa55408a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:06:41.406 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:41.405 7fa5598a3700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa55408aa50 0x7fa554099180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.113:3300/0 says I am v2:172.21.2.102:54562/0 (socket says 172.21.2.102:54562) 2021-02-10T11:06:41.406 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:41.405 7fa5598a3700 1 -- 172.21.2.102:0/264362430 learned_addr learned my addr 172.21.2.102:0/264362430 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:06:41.407 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:41.405 7fa55a8a5700 1 --2- 172.21.2.102:0/264362430 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa5540997a0 0x7fa554099ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:06:41.407 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:41.405 7fa55a0a4700 1 -- 172.21.2.102:0/264362430 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa5540997a0 msgr2=0x7fa554099ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:06:41.407 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:41.405 7fa55a0a4700 1 --2- 172.21.2.102:0/264362430 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa5540997a0 0x7fa554099ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:06:41.407 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:41.405 7fa55a0a4700 1 -- 172.21.2.102:0/264362430 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa55408aa50 msgr2=0x7fa554099180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:06:41.407 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:41.405 7fa55a0a4700 1 --2- 172.21.2.102:0/264362430 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa55408aa50 0x7fa554099180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:06:41.407 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:41.405 7fa55a0a4700 1 -- 172.21.2.102:0/264362430 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa55409a3b0 con 0x7fa5541a0470 2021-02-10T11:06:41.407 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:41.405 7fa55a0a4700 1 --2- 172.21.2.102:0/264362430 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa5541a0470 0x7fa55408a430 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 rx=0x7fa54800b510 tx=0x7fa54800b8b0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:06:41.407 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:41.405 7fa546ffd700 1 -- 172.21.2.102:0/264362430 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa548006370 con 0x7fa5541a0470 2021-02-10T11:06:41.407 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:41.405 7fa546ffd700 1 -- 172.21.2.102:0/264362430 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa54800bca0 con 0x7fa5541a0470 2021-02-10T11:06:41.407 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:41.405 7fa546ffd700 1 -- 172.21.2.102:0/264362430 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa5480022b0 con 0x7fa5541a0470 2021-02-10T11:06:41.407 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:41.405 7fa55c288700 1 -- 172.21.2.102:0/264362430 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa55409a3b0 con 0x7fa5541a0470 2021-02-10T11:06:41.408 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:41.405 7fa55c288700 1 -- 172.21.2.102:0/264362430 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa55408b850 con 0x7fa5541a0470 2021-02-10T11:06:41.408 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:41.406 7fa546ffd700 1 -- 172.21.2.102:0/264362430 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa5480052e0 con 0x7fa5541a0470 2021-02-10T11:06:41.408 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:41.406 7fa546ffd700 1 --2- 172.21.2.102:0/264362430 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa53c0186c0 0x7fa53c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:06:41.408 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:41.406 7fa546ffd700 1 -- 172.21.2.102:0/264362430 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fa548032480 con 0x7fa5541a0470 2021-02-10T11:06:41.408 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:41.406 7fa5598a3700 1 --2- 172.21.2.102:0/264362430 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa53c0186c0 0x7fa53c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:06:41.408 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:41.406 7fa5598a3700 1 --2- 172.21.2.102:0/264362430 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa53c0186c0 0x7fa53c01abc0 crc :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:06:41.435 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:41.437 7fa55c288700 1 -- 172.21.2.102:0/264362430 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa554105ae0 con 0x7fa5541a0470 2021-02-10T11:06:41.438 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:41.440 7fa546ffd700 1 -- 172.21.2.102:0/264362430 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fa548006740 con 0x7fa5541a0470 2021-02-10T11:06:41.567 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:41.569 7fa55c288700 1 -- 172.21.2.102:0/264362430 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344371", "target": ["mgr", ""]}) v1 -- 0x7fa554105ae0 con 0x7fa53c0186c0 2021-02-10T11:06:41.570 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:41.572 7fa546ffd700 1 -- 172.21.2.102:0/264362430 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7fa5500019e0 con 0x7fa53c0186c0 2021-02-10T11:06:41.570 INFO:teuthology.orchestra.run.gibba002.stdout:/volumes/_nogroup/subvolume_0000000000344371/b6ee90b2-2527-436b-b334-319e6a88fb9b 2021-02-10T11:06:41.572 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:41.573 7fa55c288700 1 -- 172.21.2.102:0/264362430 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa53c0186c0 msgr2=0x7fa53c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:06:41.572 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:41.573 7fa55c288700 1 --2- 172.21.2.102:0/264362430 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa53c0186c0 0x7fa53c01abc0 crc :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:06:41.572 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:41.573 7fa55c288700 1 -- 172.21.2.102:0/264362430 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa5541a0470 msgr2=0x7fa55408a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:06:41.572 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:41.573 7fa55c288700 1 --2- 172.21.2.102:0/264362430 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa5541a0470 0x7fa55408a430 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 rx=0x7fa54800b510 tx=0x7fa54800b8b0).stop 2021-02-10T11:06:41.572 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:41.573 7fa55c288700 1 -- 172.21.2.102:0/264362430 shutdown_connections 2021-02-10T11:06:41.572 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:41.573 7fa55c288700 1 --2- 172.21.2.102:0/264362430 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa5540997a0 0x7fa554099ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:06:41.572 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:41.573 7fa55c288700 1 --2- 172.21.2.102:0/264362430 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa53c0186c0 0x7fa53c01abc0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:06:41.572 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:41.573 7fa55c288700 1 --2- 172.21.2.102:0/264362430 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa55408aa50 0x7fa554099180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:06:41.572 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:41.573 7fa55c288700 1 --2- 172.21.2.102:0/264362430 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa5541a0470 0x7fa55408a430 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:06:41.572 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:41.573 7fa55c288700 1 -- 172.21.2.102:0/264362430 >> 172.21.2.102:0/264362430 conn(0x7fa5540bbf60 msgr2=0x7fa554105d10 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:06:41.572 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:41.573 7fa55c288700 1 -- 172.21.2.102:0/264362430 shutdown_connections 2021-02-10T11:06:41.573 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:41.573 7fa55c288700 1 -- 172.21.2.102:0/264362430 wait complete. 2021-02-10T11:06:41.580 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000344371 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000344371/b6ee90b2-2527-436b-b334-319e6a88fb9b 2021-02-10T11:06:41.580 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:06:41.580 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:06:41.580 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:06:41.915 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:06:41.915 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:06:41.915 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344371/b6ee90b2-2527-436b-b334-319e6a88fb9b/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:06:42.295 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:06:42.296 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:06:42.296 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0243979 s, 43.0 MB/s 2021-02-10T11:06:42.297 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:06:42.297 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:06:42.298 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:06:42.632 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:06:42.632 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:06:42.632 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344371/b6ee90b2-2527-436b-b334-319e6a88fb9b/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:06:43.008 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:06:43.008 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:06:43.008 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.023987 s, 43.7 MB/s 2021-02-10T11:06:43.009 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:06:43.009 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:06:43.009 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:06:43.131 DEBUG:teuthology.orchestra.run.gibba002:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:06:43.134 DEBUG:teuthology.orchestra.run.gibba013:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:06:43.343 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:06:43.344 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:06:43.344 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344371/b6ee90b2-2527-436b-b334-319e6a88fb9b/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:06:43.715 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:06:43.716 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:06:43.716 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0241695 s, 43.4 MB/s 2021-02-10T11:06:43.717 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:06:43.717 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:06:43.717 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:06:44.052 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:06:44.052 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:06:44.052 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344371/b6ee90b2-2527-436b-b334-319e6a88fb9b/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:06:44.427 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:06:44.427 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:06:44.427 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0234914 s, 44.6 MB/s 2021-02-10T11:06:44.428 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:06:44.428 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:06:44.428 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:06:44.762 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:06:44.762 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:06:44.762 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344371/b6ee90b2-2527-436b-b334-319e6a88fb9b/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:06:45.139 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:06:45.140 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:06:45.140 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0263279 s, 39.8 MB/s 2021-02-10T11:06:45.141 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:06:45.141 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:06:45.141 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:06:45.475 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:06:45.476 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:06:45.476 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344371/b6ee90b2-2527-436b-b334-319e6a88fb9b/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:06:45.848 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:06:45.849 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:06:45.849 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0226136 s, 46.4 MB/s 2021-02-10T11:06:45.850 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:06:45.850 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:06:45.850 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:06:46.184 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:06:46.185 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:06:46.185 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344371/b6ee90b2-2527-436b-b334-319e6a88fb9b/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:06:46.560 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:06:46.561 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:06:46.561 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0249757 s, 42.0 MB/s 2021-02-10T11:06:46.562 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:06:46.562 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:06:46.562 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:06:46.896 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:06:46.896 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:06:46.896 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344371/b6ee90b2-2527-436b-b334-319e6a88fb9b/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:06:47.271 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:06:47.271 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:06:47.271 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0248275 s, 42.2 MB/s 2021-02-10T11:06:47.272 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:06:47.272 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:06:47.272 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:06:47.607 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:06:47.607 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:06:47.608 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344371/b6ee90b2-2527-436b-b334-319e6a88fb9b/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:06:47.980 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:06:47.980 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:06:47.981 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0224886 s, 46.6 MB/s 2021-02-10T11:06:47.981 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:06:47.982 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:06:47.982 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:06:48.316 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:06:48.316 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:06:48.316 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344371/b6ee90b2-2527-436b-b334-319e6a88fb9b/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:06:48.686 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:06:48.687 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:06:48.687 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0230612 s, 45.5 MB/s 2021-02-10T11:06:48.688 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000344372 2021-02-10T11:06:49.116 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:49.102 7fe12d8f8700 1 Processor -- start 2021-02-10T11:06:49.116 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:49.102 7fe12d8f8700 1 -- start start 2021-02-10T11:06:49.117 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:49.102 7fe12d8f8700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fe1281b0470 0x7fe1281b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:06:49.117 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:49.102 7fe12d8f8700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe1280ef610 0x7fe1280efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:06:49.117 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:49.102 7fe12d8f8700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fe1280f0030 0x7fe1280f0430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:06:49.117 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:49.102 7fe12d8f8700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1280e5ea0 con 0x7fe1280f0030 2021-02-10T11:06:49.117 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:49.102 7fe12d8f8700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1280f0b50 con 0x7fe1281b0470 2021-02-10T11:06:49.117 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:49.102 7fe12d8f8700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1280e52e0 con 0x7fe1280ef610 2021-02-10T11:06:49.117 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:49.102 7fe127d75700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fe1280f0030 0x7fe1280f0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:06:49.117 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:49.102 7fe127d75700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fe1280f0030 0x7fe1280f0430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:51138/0 (socket says 172.21.2.102:51138) 2021-02-10T11:06:49.117 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:49.102 7fe127d75700 1 -- 172.21.2.102:0/4200400935 learned_addr learned my addr 172.21.2.102:0/4200400935 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:06:49.117 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:49.103 7fe127d75700 1 -- 172.21.2.102:0/4200400935 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe1280ef610 msgr2=0x7fe1280efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:06:49.117 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:49.103 7fe127d75700 1 --2- 172.21.2.102:0/4200400935 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe1280ef610 0x7fe1280efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:06:49.117 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:49.103 7fe127d75700 1 -- 172.21.2.102:0/4200400935 >> v2:172.21.2.113:3300/0 conn(0x7fe1281b0470 msgr2=0x7fe1281b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:06:49.118 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:49.103 7fe127d75700 1 --2- 172.21.2.102:0/4200400935 >> v2:172.21.2.113:3300/0 conn(0x7fe1281b0470 0x7fe1281b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:06:49.118 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:49.103 7fe127d75700 1 -- 172.21.2.102:0/4200400935 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1280f10d0 con 0x7fe1280f0030 2021-02-10T11:06:49.118 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:49.103 7fe127d75700 1 --2- 172.21.2.102:0/4200400935 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe1280f0030 0x7fe1280f0430 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 rx=0x7fe120006f00 tx=0x7fe120009f10).ready entity=mon.0 client_cookie=d28722cb81e199c3 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:06:49.118 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:49.103 7fe126572700 1 -- 172.21.2.102:0/4200400935 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe12000b3b0 con 0x7fe1280f0030 2021-02-10T11:06:49.118 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:49.103 7fe126572700 1 -- 172.21.2.102:0/4200400935 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe12000b630 con 0x7fe1280f0030 2021-02-10T11:06:49.118 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:49.103 7fe126572700 1 -- 172.21.2.102:0/4200400935 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe12000bb50 con 0x7fe1280f0030 2021-02-10T11:06:49.118 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:49.103 7fe12d8f8700 1 -- 172.21.2.102:0/4200400935 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe1280f0030 msgr2=0x7fe1280f0430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:06:49.118 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:49.103 7fe12d8f8700 1 --2- 172.21.2.102:0/4200400935 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe1280f0030 0x7fe1280f0430 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 rx=0x7fe120006f00 tx=0x7fe120009f10).stop 2021-02-10T11:06:49.118 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:49.104 7fe12d8f8700 1 -- 172.21.2.102:0/4200400935 shutdown_connections 2021-02-10T11:06:49.118 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:49.104 7fe12d8f8700 1 --2- 172.21.2.102:0/4200400935 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe1280ef610 0x7fe1280efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:06:49.118 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:49.104 7fe12d8f8700 1 --2- 172.21.2.102:0/4200400935 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe1280f0030 0x7fe1280f0430 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:06:49.118 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:49.104 7fe12d8f8700 1 --2- 172.21.2.102:0/4200400935 >> v2:172.21.2.113:3300/0 conn(0x7fe1281b0470 0x7fe1281b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:06:49.119 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:49.104 7fe12d8f8700 1 -- 172.21.2.102:0/4200400935 >> 172.21.2.102:0/4200400935 conn(0x7fe1280ce3e0 msgr2=0x7fe1280e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:06:49.119 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:49.104 7fe12d8f8700 1 -- 172.21.2.102:0/4200400935 shutdown_connections 2021-02-10T11:06:49.119 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:49.104 7fe12d8f8700 1 -- 172.21.2.102:0/4200400935 wait complete. 2021-02-10T11:06:49.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:49.119 7fe12d8f8700 1 Processor -- start 2021-02-10T11:06:49.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:49.119 7fe12d8f8700 1 -- start start 2021-02-10T11:06:49.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:49.119 7fe12d8f8700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe128190920 0x7fe12808f600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:06:49.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:49.119 7fe12d8f8700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe128190d20 0x7fe12808fc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:06:49.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:49.119 7fe12d8f8700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe128193270 0x7fe128193670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:06:49.121 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:49.119 7fe12d8f8700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe128090470 con 0x7fe128193270 2021-02-10T11:06:49.121 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:49.119 7fe12d8f8700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe128193c90 con 0x7fe128190d20 2021-02-10T11:06:49.121 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:49.119 7fe12d8f8700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe128193e70 con 0x7fe128190920 2021-02-10T11:06:49.121 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:49.119 7fe127d75700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe128193270 0x7fe128193670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:06:49.121 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:49.119 7fe127d75700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe128193270 0x7fe128193670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:51146/0 (socket says 172.21.2.102:51146) 2021-02-10T11:06:49.121 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:49.119 7fe127d75700 1 -- 172.21.2.102:0/1180593242 learned_addr learned my addr 172.21.2.102:0/1180593242 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:06:49.121 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:49.120 7fe127d75700 1 -- 172.21.2.102:0/1180593242 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe128190920 msgr2=0x7fe12808f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:06:49.121 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:49.120 7fe127d75700 1 --2- 172.21.2.102:0/1180593242 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe128190920 0x7fe12808f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:06:49.121 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:49.120 7fe127d75700 1 -- 172.21.2.102:0/1180593242 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe128190d20 msgr2=0x7fe12808fc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:06:49.121 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:49.120 7fe127d75700 1 --2- 172.21.2.102:0/1180593242 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe128190d20 0x7fe12808fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:06:49.121 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:49.120 7fe127d75700 1 -- 172.21.2.102:0/1180593242 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe128194090 con 0x7fe128193270 2021-02-10T11:06:49.121 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:49.120 7fe127d75700 1 --2- 172.21.2.102:0/1180593242 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe128193270 0x7fe128193670 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 rx=0x7fe120009390 tx=0x7fe12000bd40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:06:49.121 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:49.120 7fe10ffff700 1 -- 172.21.2.102:0/1180593242 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe120003030 con 0x7fe128193270 2021-02-10T11:06:49.121 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:49.120 7fe10ffff700 1 -- 172.21.2.102:0/1180593242 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe120006c80 con 0x7fe128193270 2021-02-10T11:06:49.122 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:49.120 7fe10ffff700 1 -- 172.21.2.102:0/1180593242 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe120001480 con 0x7fe128193270 2021-02-10T11:06:49.122 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:49.120 7fe12d8f8700 1 -- 172.21.2.102:0/1180593242 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe128194090 con 0x7fe128193270 2021-02-10T11:06:49.122 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:49.120 7fe12d8f8700 1 -- 172.21.2.102:0/1180593242 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1281920b0 con 0x7fe128193270 2021-02-10T11:06:49.122 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:49.121 7fe10ffff700 1 -- 172.21.2.102:0/1180593242 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fe120001270 con 0x7fe128193270 2021-02-10T11:06:49.122 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:49.121 7fe10ffff700 1 --2- 172.21.2.102:0/1180593242 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe1080186c0 0x7fe10801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:06:49.122 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:49.121 7fe10ffff700 1 -- 172.21.2.102:0/1180593242 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fe1200328a0 con 0x7fe128193270 2021-02-10T11:06:49.122 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:49.121 7fe127574700 1 --2- 172.21.2.102:0/1180593242 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe1080186c0 0x7fe10801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:06:49.122 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:49.121 7fe127574700 1 --2- 172.21.2.102:0/1180593242 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe1080186c0 0x7fe10801abc0 crc :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:06:49.150 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:49.152 7fe12d8f8700 1 -- 172.21.2.102:0/1180593242 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe128105af0 con 0x7fe128193270 2021-02-10T11:06:49.153 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:49.155 7fe10ffff700 1 -- 172.21.2.102:0/1180593242 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fe120005d60 con 0x7fe128193270 2021-02-10T11:06:49.281 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:49.283 7fe12d8f8700 1 -- 172.21.2.102:0/1180593242 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344372", "target": ["mgr", ""]}) v1 -- 0x7fe1281920b0 con 0x7fe1080186c0 2021-02-10T11:06:49.305 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:49.307 7fe10ffff700 1 -- 172.21.2.102:0/1180593242 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fe1180009b0 con 0x7fe1080186c0 2021-02-10T11:06:49.314 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:49.316 7fe12d8f8700 1 -- 172.21.2.102:0/1180593242 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe1080186c0 msgr2=0x7fe10801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:06:49.314 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:49.316 7fe12d8f8700 1 --2- 172.21.2.102:0/1180593242 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe1080186c0 0x7fe10801abc0 crc :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:06:49.314 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:49.316 7fe12d8f8700 1 -- 172.21.2.102:0/1180593242 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe128193270 msgr2=0x7fe128193670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:06:49.314 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:49.316 7fe12d8f8700 1 --2- 172.21.2.102:0/1180593242 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe128193270 0x7fe128193670 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 rx=0x7fe120009390 tx=0x7fe12000bd40).stop 2021-02-10T11:06:49.314 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:49.316 7fe12d8f8700 1 -- 172.21.2.102:0/1180593242 shutdown_connections 2021-02-10T11:06:49.314 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:49.316 7fe12d8f8700 1 --2- 172.21.2.102:0/1180593242 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe128193270 0x7fe128193670 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:06:49.314 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:49.316 7fe12d8f8700 1 --2- 172.21.2.102:0/1180593242 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe1080186c0 0x7fe10801abc0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:06:49.314 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:49.316 7fe12d8f8700 1 --2- 172.21.2.102:0/1180593242 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe128190d20 0x7fe12808fc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:06:49.314 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:49.316 7fe12d8f8700 1 --2- 172.21.2.102:0/1180593242 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe128190920 0x7fe12808f600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:06:49.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:49.316 7fe12d8f8700 1 -- 172.21.2.102:0/1180593242 >> 172.21.2.102:0/1180593242 conn(0x7fe1280bf4e0 msgr2=0x7fe1280be3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:06:49.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:49.316 7fe12d8f8700 1 -- 172.21.2.102:0/1180593242 shutdown_connections 2021-02-10T11:06:49.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:49.316 7fe12d8f8700 1 -- 172.21.2.102:0/1180593242 wait complete. 2021-02-10T11:06:49.322 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000344372 2021-02-10T11:06:49.750 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:49.736 7f2d1bfcc700 1 Processor -- start 2021-02-10T11:06:49.750 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:49.736 7f2d1bfcc700 1 -- start start 2021-02-10T11:06:49.751 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:49.737 7f2d1bfcc700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f2d141b0870 0x7f2d140f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:06:49.751 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:49.737 7f2d1bfcc700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2d140efe00 0x7f2d140f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:06:49.751 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:49.737 7f2d1bfcc700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d141b0c70 con 0x7f2d141b0470 2021-02-10T11:06:49.751 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:49.737 7f2d1bfcc700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d140f0b50 con 0x7f2d141b0870 2021-02-10T11:06:49.751 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:49.737 7f2d1bfcc700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d140f0820 con 0x7f2d140efe00 2021-02-10T11:06:49.751 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:49.737 7f2d195e7700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f2d141b0870 0x7f2d140f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:06:49.751 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:49.737 7f2d195e7700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f2d141b0870 0x7f2d140f20e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:51152/0 (socket says 172.21.2.102:51152) 2021-02-10T11:06:49.751 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:49.737 7f2d195e7700 1 -- 172.21.2.102:0/4234152892 learned_addr learned my addr 172.21.2.102:0/4234152892 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:06:49.751 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:49.737 7f2d195e7700 1 -- 172.21.2.102:0/4234152892 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2d140efe00 msgr2=0x7f2d140f0200 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:06:49.751 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:49.737 7f2d195e7700 1 --2- 172.21.2.102:0/4234152892 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2d140efe00 0x7f2d140f0200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:06:49.751 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:49.737 7f2d195e7700 1 -- 172.21.2.102:0/4234152892 >> v1:172.21.2.113:6789/0 conn(0x7f2d141b0470 legacy=0x7f2d140ef610 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:06:49.752 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:49.737 7f2d195e7700 1 -- 172.21.2.102:0/4234152892 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d140e52e0 con 0x7f2d141b0870 2021-02-10T11:06:49.752 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:49.737 7f2d195e7700 1 --2- 172.21.2.102:0/4234152892 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2d141b0870 0x7f2d140f20e0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 rx=0x7f2d10006700 tx=0x7f2d10009b80).ready entity=mon.0 client_cookie=cfd2d6c1f6dabfbe server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:06:49.752 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:49.737 7f2d18de6700 1 -- 172.21.2.102:0/4234152892 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2d1000c0c0 con 0x7f2d141b0870 2021-02-10T11:06:49.752 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:49.737 7f2d18de6700 1 -- 172.21.2.102:0/4234152892 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d1000c340 con 0x7f2d141b0870 2021-02-10T11:06:49.752 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:49.737 7f2d18de6700 1 -- 172.21.2.102:0/4234152892 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2d1000c860 con 0x7f2d141b0870 2021-02-10T11:06:49.752 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:49.737 7f2d1bfcc700 1 -- 172.21.2.102:0/4234152892 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2d141b0870 msgr2=0x7f2d140f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:06:49.752 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:49.737 7f2d1bfcc700 1 --2- 172.21.2.102:0/4234152892 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2d141b0870 0x7f2d140f20e0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 rx=0x7f2d10006700 tx=0x7f2d10009b80).stop 2021-02-10T11:06:49.752 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:49.738 7f2d1bfcc700 1 -- 172.21.2.102:0/4234152892 shutdown_connections 2021-02-10T11:06:49.752 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:49.738 7f2d1bfcc700 1 --2- 172.21.2.102:0/4234152892 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2d140efe00 0x7f2d140f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:06:49.752 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:49.738 7f2d1bfcc700 1 --2- 172.21.2.102:0/4234152892 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2d141b0870 0x7f2d140f20e0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:06:49.752 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:49.738 7f2d1bfcc700 1 -- 172.21.2.102:0/4234152892 >> 172.21.2.102:0/4234152892 conn(0x7f2d140ce3e0 msgr2=0x7f2d140e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:06:49.752 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:49.738 7f2d1bfcc700 1 -- 172.21.2.102:0/4234152892 shutdown_connections 2021-02-10T11:06:49.752 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:49.738 7f2d1bfcc700 1 -- 172.21.2.102:0/4234152892 wait complete. 2021-02-10T11:06:49.753 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:49.752 7f2d1bfcc700 1 Processor -- start 2021-02-10T11:06:49.753 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:49.753 7f2d1bfcc700 1 -- start start 2021-02-10T11:06:49.754 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:49.753 7f2d1bfcc700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2d140a6c90 0x7f2d141924c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:06:49.754 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:49.753 7f2d1bfcc700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2d14192ae0 0x7f2d14192ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:06:49.754 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:49.753 7f2d1bfcc700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2d140919e0 0x7f2d14091de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:06:49.754 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:49.753 7f2d1bfcc700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d1419d2e0 con 0x7f2d14192ae0 2021-02-10T11:06:49.755 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:49.753 7f2d1bfcc700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d140914d0 con 0x7f2d140a6c90 2021-02-10T11:06:49.755 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:49.753 7f2d1bfcc700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d140a6820 con 0x7f2d140919e0 2021-02-10T11:06:49.755 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:49.753 7f2d195e7700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2d14192ae0 0x7f2d14192ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:06:49.755 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:49.753 7f2d195e7700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2d14192ae0 0x7f2d14192ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:51160/0 (socket says 172.21.2.102:51160) 2021-02-10T11:06:49.755 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:49.753 7f2d195e7700 1 -- 172.21.2.102:0/199730101 learned_addr learned my addr 172.21.2.102:0/199730101 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:06:49.755 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:49.753 7f2d195e7700 1 -- 172.21.2.102:0/199730101 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2d140919e0 msgr2=0x7f2d14091de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:06:49.755 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:49.753 7f2d195e7700 1 --2- 172.21.2.102:0/199730101 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2d140919e0 0x7f2d14091de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:06:49.755 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:49.753 7f2d195e7700 1 -- 172.21.2.102:0/199730101 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2d140a6c90 msgr2=0x7f2d141924c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:06:49.755 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:49.753 7f2d195e7700 1 --2- 172.21.2.102:0/199730101 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2d140a6c90 0x7f2d141924c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:06:49.756 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:49.753 7f2d195e7700 1 -- 172.21.2.102:0/199730101 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d14092400 con 0x7f2d14192ae0 2021-02-10T11:06:49.756 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:49.754 7f2d195e7700 1 --2- 172.21.2.102:0/199730101 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2d14192ae0 0x7f2d14192ee0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 rx=0x7f2d1000b5e0 tx=0x7f2d1000b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:06:49.756 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:49.754 7f2d067fc700 1 -- 172.21.2.102:0/199730101 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2d1000bd70 con 0x7f2d14192ae0 2021-02-10T11:06:49.756 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:49.754 7f2d067fc700 1 -- 172.21.2.102:0/199730101 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d10002030 con 0x7f2d14192ae0 2021-02-10T11:06:49.756 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:49.754 7f2d067fc700 1 -- 172.21.2.102:0/199730101 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2d100024d0 con 0x7f2d14192ae0 2021-02-10T11:06:49.756 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:49.754 7f2d1bfcc700 1 -- 172.21.2.102:0/199730101 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d14092400 con 0x7f2d14192ae0 2021-02-10T11:06:49.756 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:49.754 7f2d1bfcc700 1 -- 172.21.2.102:0/199730101 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d14092400 con 0x7f2d14192ae0 2021-02-10T11:06:49.756 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:49.755 7f2d067fc700 1 -- 172.21.2.102:0/199730101 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f2d10000c90 con 0x7f2d14192ae0 2021-02-10T11:06:49.756 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:49.755 7f2d067fc700 1 --2- 172.21.2.102:0/199730101 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2cfc0186c0 0x7f2cfc01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:06:49.756 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:49.755 7f2d067fc700 1 -- 172.21.2.102:0/199730101 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f2d10032530 con 0x7f2d14192ae0 2021-02-10T11:06:49.757 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:49.755 7f2d19de8700 1 --2- 172.21.2.102:0/199730101 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2cfc0186c0 0x7f2cfc01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:06:49.757 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:49.755 7f2d19de8700 1 --2- 172.21.2.102:0/199730101 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2cfc0186c0 0x7f2cfc01abc0 crc :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:06:49.784 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:49.786 7f2d1bfcc700 1 -- 172.21.2.102:0/199730101 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d140e52e0 con 0x7f2d14192ae0 2021-02-10T11:06:49.787 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:49.789 7f2d067fc700 1 -- 172.21.2.102:0/199730101 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f2d10030ce0 con 0x7f2d14192ae0 2021-02-10T11:06:49.915 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:49.917 7f2d1bfcc700 1 -- 172.21.2.102:0/199730101 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344372", "target": ["mgr", ""]}) v1 -- 0x7f2d140e52e0 con 0x7f2cfc0186c0 2021-02-10T11:06:49.918 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:49.920 7f2d067fc700 1 -- 172.21.2.102:0/199730101 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f2d08006e40 con 0x7f2cfc0186c0 2021-02-10T11:06:49.922 INFO:teuthology.orchestra.run.gibba002.stdout:/volumes/_nogroup/subvolume_0000000000344372/d28fc962-fe67-4f31-a40c-4014b872d968 2021-02-10T11:06:49.928 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:49.925 7f2d1bfcc700 1 -- 172.21.2.102:0/199730101 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2cfc0186c0 msgr2=0x7f2cfc01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:06:49.929 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:49.925 7f2d1bfcc700 1 --2- 172.21.2.102:0/199730101 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2cfc0186c0 0x7f2cfc01abc0 crc :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:06:49.929 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:49.925 7f2d1bfcc700 1 -- 172.21.2.102:0/199730101 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2d14192ae0 msgr2=0x7f2d14192ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:06:49.929 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:49.925 7f2d1bfcc700 1 --2- 172.21.2.102:0/199730101 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2d14192ae0 0x7f2d14192ee0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 rx=0x7f2d1000b5e0 tx=0x7f2d1000b980).stop 2021-02-10T11:06:49.929 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:49.925 7f2d1bfcc700 1 -- 172.21.2.102:0/199730101 shutdown_connections 2021-02-10T11:06:49.929 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:49.925 7f2d1bfcc700 1 --2- 172.21.2.102:0/199730101 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2d140919e0 0x7f2d14091de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:06:49.929 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:49.925 7f2d1bfcc700 1 --2- 172.21.2.102:0/199730101 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2d14192ae0 0x7f2d14192ee0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:06:49.930 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:49.925 7f2d1bfcc700 1 --2- 172.21.2.102:0/199730101 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2cfc0186c0 0x7f2cfc01abc0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:06:49.930 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:49.925 7f2d1bfcc700 1 --2- 172.21.2.102:0/199730101 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2d140a6c90 0x7f2d141924c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:06:49.930 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:49.925 7f2d1bfcc700 1 -- 172.21.2.102:0/199730101 >> 172.21.2.102:0/199730101 conn(0x7f2d140ac550 msgr2=0x7f2d140be1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:06:49.930 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:49.925 7f2d1bfcc700 1 -- 172.21.2.102:0/199730101 shutdown_connections 2021-02-10T11:06:49.930 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:49.925 7f2d1bfcc700 1 -- 172.21.2.102:0/199730101 wait complete. 2021-02-10T11:06:49.931 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000344372 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000344372/d28fc962-fe67-4f31-a40c-4014b872d968 2021-02-10T11:06:49.932 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:06:49.932 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:06:49.932 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:06:50.269 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:06:50.269 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:06:50.269 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344372/d28fc962-fe67-4f31-a40c-4014b872d968/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:06:50.652 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:06:50.652 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:06:50.652 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0253518 s, 41.4 MB/s 2021-02-10T11:06:50.653 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:06:50.653 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:06:50.653 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:06:50.989 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:06:50.989 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:06:50.989 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344372/d28fc962-fe67-4f31-a40c-4014b872d968/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:06:51.365 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:06:51.366 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:06:51.366 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0257399 s, 40.7 MB/s 2021-02-10T11:06:51.367 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:06:51.367 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:06:51.367 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:06:51.701 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:06:51.701 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:06:51.702 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344372/d28fc962-fe67-4f31-a40c-4014b872d968/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:06:52.078 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:06:52.078 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:06:52.078 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0255597 s, 41.0 MB/s 2021-02-10T11:06:52.079 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:06:52.079 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:06:52.079 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:06:52.414 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:06:52.414 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:06:52.415 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344372/d28fc962-fe67-4f31-a40c-4014b872d968/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:06:52.790 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:06:52.790 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:06:52.790 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0253738 s, 41.3 MB/s 2021-02-10T11:06:52.791 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:06:52.791 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:06:52.791 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:06:53.125 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:06:53.125 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:06:53.125 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344372/d28fc962-fe67-4f31-a40c-4014b872d968/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:06:53.497 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:06:53.498 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:06:53.498 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.025255 s, 41.5 MB/s 2021-02-10T11:06:53.499 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:06:53.499 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:06:53.499 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:06:53.833 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:06:53.834 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:06:53.834 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344372/d28fc962-fe67-4f31-a40c-4014b872d968/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:06:54.209 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:06:54.209 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:06:54.209 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0243346 s, 43.1 MB/s 2021-02-10T11:06:54.210 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:06:54.210 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:06:54.211 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:06:54.545 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:06:54.546 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:06:54.546 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344372/d28fc962-fe67-4f31-a40c-4014b872d968/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:06:54.921 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:06:54.921 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:06:54.921 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0250877 s, 41.8 MB/s 2021-02-10T11:06:54.922 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:06:54.922 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:06:54.922 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:06:55.257 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:06:55.257 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:06:55.257 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344372/d28fc962-fe67-4f31-a40c-4014b872d968/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:06:55.631 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:06:55.631 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:06:55.631 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0230421 s, 45.5 MB/s 2021-02-10T11:06:55.632 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:06:55.632 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:06:55.633 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:06:55.968 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:06:55.968 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:06:55.968 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344372/d28fc962-fe67-4f31-a40c-4014b872d968/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:06:56.343 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:06:56.343 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:06:56.343 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0241648 s, 43.4 MB/s 2021-02-10T11:06:56.344 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:06:56.344 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:06:56.344 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:06:56.679 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:06:56.679 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:06:56.679 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344372/d28fc962-fe67-4f31-a40c-4014b872d968/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:06:57.053 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:06:57.053 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:06:57.053 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0244843 s, 42.8 MB/s 2021-02-10T11:06:57.054 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000344373 2021-02-10T11:06:57.483 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:57.468 7fe4b4715700 1 Processor -- start 2021-02-10T11:06:57.484 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:57.468 7fe4b4715700 1 -- start start 2021-02-10T11:06:57.484 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:57.468 7fe4b4715700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe4ac1b0870 0x7fe4ac0f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:06:57.484 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:57.468 7fe4b4715700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fe4ac0efe00 0x7fe4ac0f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:06:57.484 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:57.469 7fe4b4715700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4ac1b0c70 con 0x7fe4ac1b0470 2021-02-10T11:06:57.484 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:57.469 7fe4b4715700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4ac0f0b50 con 0x7fe4ac0efe00 2021-02-10T11:06:57.484 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:57.469 7fe4b4715700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4ac0f0820 con 0x7fe4ac1b0870 2021-02-10T11:06:57.484 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:57.469 7fe4b2531700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fe4ac1b0470 0x7fe4ac0ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:43390/0 (socket says 172.21.2.102:43390) 2021-02-10T11:06:57.484 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:57.469 7fe4b2531700 1 -- 172.21.2.102:0/2110480847 learned_addr learned my addr 172.21.2.102:0/2110480847 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:06:57.484 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:57.469 7fe4b2d32700 1 --2- 172.21.2.102:0/2110480847 >> v2:172.21.2.113:3300/0 conn(0x7fe4ac0efe00 0x7fe4ac0f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:06:57.484 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:57.469 7fe4b1d30700 1 --2- 172.21.2.102:0/2110480847 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe4ac1b0870 0x7fe4ac0f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:06:57.484 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:57.469 7fe4b1d30700 1 -- 172.21.2.102:0/2110480847 >> v2:172.21.2.113:3300/0 conn(0x7fe4ac0efe00 msgr2=0x7fe4ac0f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:06:57.484 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:57.469 7fe4b1d30700 1 --2- 172.21.2.102:0/2110480847 >> v2:172.21.2.113:3300/0 conn(0x7fe4ac0efe00 0x7fe4ac0f0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:06:57.485 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:57.469 7fe4b1d30700 1 -- 172.21.2.102:0/2110480847 >> v1:172.21.2.102:6789/0 conn(0x7fe4ac1b0470 legacy=0x7fe4ac0ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:06:57.485 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:57.469 7fe4b1d30700 1 -- 172.21.2.102:0/2110480847 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4ac0e52e0 con 0x7fe4ac1b0870 2021-02-10T11:06:57.485 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:57.470 7fe4b1d30700 1 --2- 172.21.2.102:0/2110480847 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe4ac1b0870 0x7fe4ac0f20e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7fe49c006650 tx=0x7fe49c009a00).ready entity=mon.2 client_cookie=fbcfdf57a9646bdf server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:06:57.485 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:57.470 7fe4b152f700 1 -- 172.21.2.102:0/2110480847 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe49c0009b0 con 0x7fe4ac1b0870 2021-02-10T11:06:57.485 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:57.470 7fe4b152f700 1 -- 172.21.2.102:0/2110480847 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe49c009df0 con 0x7fe4ac1b0870 2021-02-10T11:06:57.485 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:57.470 7fe4b152f700 1 -- 172.21.2.102:0/2110480847 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe49c00c0c0 con 0x7fe4ac1b0870 2021-02-10T11:06:57.485 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:57.470 7fe4b4715700 1 -- 172.21.2.102:0/2110480847 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe4ac1b0870 msgr2=0x7fe4ac0f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:06:57.485 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:57.470 7fe4b4715700 1 --2- 172.21.2.102:0/2110480847 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe4ac1b0870 0x7fe4ac0f20e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7fe49c006650 tx=0x7fe49c009a00).stop 2021-02-10T11:06:57.485 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:57.471 7fe4b4715700 1 -- 172.21.2.102:0/2110480847 shutdown_connections 2021-02-10T11:06:57.485 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:57.471 7fe4b4715700 1 --2- 172.21.2.102:0/2110480847 >> v2:172.21.2.113:3300/0 conn(0x7fe4ac0efe00 0x7fe4ac0f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:06:57.485 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:57.471 7fe4b4715700 1 --2- 172.21.2.102:0/2110480847 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe4ac1b0870 0x7fe4ac0f20e0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:06:57.485 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:57.471 7fe4b4715700 1 -- 172.21.2.102:0/2110480847 >> 172.21.2.102:0/2110480847 conn(0x7fe4ac0ce3e0 msgr2=0x7fe4ac0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:06:57.485 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:57.471 7fe4b4715700 1 -- 172.21.2.102:0/2110480847 shutdown_connections 2021-02-10T11:06:57.486 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:57.471 7fe4b4715700 1 -- 172.21.2.102:0/2110480847 wait complete. 2021-02-10T11:06:57.487 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:57.486 7fe4b4715700 1 Processor -- start 2021-02-10T11:06:57.488 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:57.486 7fe4b4715700 1 -- start start 2021-02-10T11:06:57.488 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:57.486 7fe4b4715700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe4ac0a0420 0x7fe4ac0a0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:06:57.488 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:57.486 7fe4b4715700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe4ac189d00 0x7fe4ac18a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:06:57.488 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:57.486 7fe4b4715700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe4ac18a720 0x7fe4ac0b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:06:57.488 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:57.486 7fe4b4715700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4ac189780 con 0x7fe4ac18a720 2021-02-10T11:06:57.488 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:57.486 7fe4b4715700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4ac18ab20 con 0x7fe4ac0a0420 2021-02-10T11:06:57.488 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:57.486 7fe4b4715700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4ac0a1140 con 0x7fe4ac189d00 2021-02-10T11:06:57.488 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:57.486 7fe4b2d32700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe4ac18a720 0x7fe4ac0b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:06:57.489 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:57.486 7fe4b2d32700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe4ac18a720 0x7fe4ac0b7be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:51176/0 (socket says 172.21.2.102:51176) 2021-02-10T11:06:57.489 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:57.486 7fe4b2d32700 1 -- 172.21.2.102:0/2398880235 learned_addr learned my addr 172.21.2.102:0/2398880235 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:06:57.489 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:57.486 7fe4b2531700 1 --2- 172.21.2.102:0/2398880235 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe4ac0a0420 0x7fe4ac0a0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:06:57.489 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:57.487 7fe4b1d30700 1 --2- 172.21.2.102:0/2398880235 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe4ac189d00 0x7fe4ac18a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:06:57.490 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:57.487 7fe4b2531700 1 -- 172.21.2.102:0/2398880235 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe4ac189d00 msgr2=0x7fe4ac18a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:06:57.490 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:57.487 7fe4b2531700 1 --2- 172.21.2.102:0/2398880235 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe4ac189d00 0x7fe4ac18a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:06:57.490 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:57.487 7fe4b2531700 1 -- 172.21.2.102:0/2398880235 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe4ac18a720 msgr2=0x7fe4ac0b7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:06:57.490 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:57.487 7fe4b2531700 1 --2- 172.21.2.102:0/2398880235 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe4ac18a720 0x7fe4ac0b7be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:06:57.490 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:57.487 7fe4b2531700 1 -- 172.21.2.102:0/2398880235 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4ac0b8200 con 0x7fe4ac0a0420 2021-02-10T11:06:57.490 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:57.487 7fe4b2531700 1 --2- 172.21.2.102:0/2398880235 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe4ac0a0420 0x7fe4ac0a0820 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7fe4a8001e30 tx=0x7fe4a800a880).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:06:57.491 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:57.487 7fe4a2ffd700 1 -- 172.21.2.102:0/2398880235 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe4a80070d0 con 0x7fe4ac0a0420 2021-02-10T11:06:57.491 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:57.487 7fe4a2ffd700 1 -- 172.21.2.102:0/2398880235 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4a800ac70 con 0x7fe4ac0a0420 2021-02-10T11:06:57.491 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:57.487 7fe4a2ffd700 1 -- 172.21.2.102:0/2398880235 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe4a8004620 con 0x7fe4ac0a0420 2021-02-10T11:06:57.491 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:57.487 7fe4b4715700 1 -- 172.21.2.102:0/2398880235 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4ac0b8200 con 0x7fe4ac0a0420 2021-02-10T11:06:57.491 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:57.487 7fe4b4715700 1 -- 172.21.2.102:0/2398880235 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4ac0b8200 con 0x7fe4ac0a0420 2021-02-10T11:06:57.491 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:57.488 7fe4a2ffd700 1 -- 172.21.2.102:0/2398880235 <== mon.1 v2:172.21.2.113:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fe4a8007ec0 con 0x7fe4ac0a0420 2021-02-10T11:06:57.491 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:57.488 7fe4a2ffd700 1 --2- 172.21.2.102:0/2398880235 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe4940186c0 0x7fe49401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:06:57.492 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:57.488 7fe4a2ffd700 1 -- 172.21.2.102:0/2398880235 <== mon.1 v2:172.21.2.113:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fe4a8032450 con 0x7fe4ac0a0420 2021-02-10T11:06:57.492 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:57.488 7fe4b1d30700 1 --2- 172.21.2.102:0/2398880235 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe4940186c0 0x7fe49401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:06:57.492 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:57.488 7fe4b1d30700 1 --2- 172.21.2.102:0/2398880235 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe4940186c0 0x7fe49401abc0 crc :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:06:57.516 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:57.518 7fe4b4715700 1 -- 172.21.2.102:0/2398880235 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4ac105ae0 con 0x7fe4ac0a0420 2021-02-10T11:06:57.519 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:57.521 7fe4a2ffd700 1 -- 172.21.2.102:0/2398880235 <== mon.1 v2:172.21.2.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fe4a8001580 con 0x7fe4ac0a0420 2021-02-10T11:06:57.646 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:57.648 7fe4b4715700 1 -- 172.21.2.102:0/2398880235 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344373", "target": ["mgr", ""]}) v1 -- 0x7fe4ac0f0b50 con 0x7fe4940186c0 2021-02-10T11:06:57.670 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:57.672 7fe4a2ffd700 1 -- 172.21.2.102:0/2398880235 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fe49c000c90 con 0x7fe4940186c0 2021-02-10T11:06:57.680 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:57.681 7fe4b4715700 1 -- 172.21.2.102:0/2398880235 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe4940186c0 msgr2=0x7fe49401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:06:57.681 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:57.681 7fe4b4715700 1 --2- 172.21.2.102:0/2398880235 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe4940186c0 0x7fe49401abc0 crc :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:06:57.681 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:57.681 7fe4b4715700 1 -- 172.21.2.102:0/2398880235 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe4ac0a0420 msgr2=0x7fe4ac0a0820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:06:57.681 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:57.681 7fe4b4715700 1 --2- 172.21.2.102:0/2398880235 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe4ac0a0420 0x7fe4ac0a0820 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7fe4a8001e30 tx=0x7fe4a800a880).stop 2021-02-10T11:06:57.681 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:57.682 7fe4b4715700 1 -- 172.21.2.102:0/2398880235 shutdown_connections 2021-02-10T11:06:57.681 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:57.682 7fe4b4715700 1 --2- 172.21.2.102:0/2398880235 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe4ac18a720 0x7fe4ac0b7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:06:57.681 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:57.682 7fe4b4715700 1 --2- 172.21.2.102:0/2398880235 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe4ac189d00 0x7fe4ac18a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:06:57.681 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:57.682 7fe4b4715700 1 --2- 172.21.2.102:0/2398880235 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe4940186c0 0x7fe49401abc0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:06:57.681 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:57.682 7fe4b4715700 1 --2- 172.21.2.102:0/2398880235 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe4ac0a0420 0x7fe4ac0a0820 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:06:57.681 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:57.682 7fe4b4715700 1 -- 172.21.2.102:0/2398880235 >> 172.21.2.102:0/2398880235 conn(0x7fe4ac094e10 msgr2=0x7fe4ac0a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:06:57.681 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:57.682 7fe4b4715700 1 -- 172.21.2.102:0/2398880235 shutdown_connections 2021-02-10T11:06:57.681 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:57.682 7fe4b4715700 1 -- 172.21.2.102:0/2398880235 wait complete. 2021-02-10T11:06:57.688 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000344373 2021-02-10T11:06:58.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:58.104 7f396e264700 1 Processor -- start 2021-02-10T11:06:58.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:58.104 7f396e264700 1 -- start start 2021-02-10T11:06:58.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:58.104 7f396e264700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f39681b0870 0x7f39680f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:06:58.121 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:58.104 7f396e264700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39681b0c70 con 0x7f39681b0470 2021-02-10T11:06:58.121 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:58.104 7f396e264700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39680f0b50 con 0x7f39680efe00 2021-02-10T11:06:58.121 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:58.104 7f396e264700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39680e5ea0 con 0x7f39681b0870 2021-02-10T11:06:58.121 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:58.105 7f3967fff700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f39681b0470 0x7f39680ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:43404/0 (socket says 172.21.2.102:43404) 2021-02-10T11:06:58.121 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:58.105 7f3967fff700 1 -- 172.21.2.102:0/2056572441 learned_addr learned my addr 172.21.2.102:0/2056572441 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:06:58.121 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:58.105 7f3966ffd700 1 -- 172.21.2.102:0/2056572441 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 286031327 0 0) 0x7f3958001180 con 0x7f39681b0470 2021-02-10T11:06:58.121 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:58.105 7f3966ffd700 1 -- 172.21.2.102:0/2056572441 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3950002fe0 con 0x7f39681b0470 2021-02-10T11:06:58.121 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:58.105 7f39677fe700 1 --2- 172.21.2.102:0/2056572441 >> v2:172.21.2.113:3300/0 conn(0x7f39681b0870 0x7f39680f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:06:58.121 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:58.105 7f3966ffd700 1 -- 172.21.2.102:0/2056572441 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 232073675 0 0) 0x7f39580010e0 con 0x7f39681b0470 2021-02-10T11:06:58.121 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:58.105 7f3966ffd700 1 -- 172.21.2.102:0/2056572441 >> v2:172.21.2.113:3300/0 conn(0x7f39681b0870 msgr2=0x7f39680f20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:06:58.121 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:58.105 7f3966ffd700 1 --2- 172.21.2.102:0/2056572441 >> v2:172.21.2.113:3300/0 conn(0x7f39681b0870 0x7f39680f20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:06:58.121 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:58.105 7f3966ffd700 1 -- 172.21.2.102:0/2056572441 >> v1:172.21.2.113:6789/0 conn(0x7f39680efe00 legacy=0x7f39680f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:06:58.121 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:58.105 7f3966ffd700 1 -- 172.21.2.102:0/2056572441 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39680e52e0 con 0x7f39681b0470 2021-02-10T11:06:58.122 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:58.105 7f3966ffd700 1 -- 172.21.2.102:0/2056572441 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7f3958001600 con 0x7f39681b0470 2021-02-10T11:06:58.122 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:58.106 7f3966ffd700 1 -- 172.21.2.102:0/2056572441 >> v1:172.21.2.102:6789/0 conn(0x7f39681b0470 legacy=0x7f39680ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:06:58.122 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:58.106 7f3966ffd700 1 --2- 172.21.2.102:0/2056572441 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3950003a60 0x7f3950005ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:06:58.122 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:58.106 7f3966ffd700 1 --2- 172.21.2.102:0/2056572441 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3950006770 0x7f3950008c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:06:58.122 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:58.106 7f3966ffd700 1 --2- 172.21.2.102:0/2056572441 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3950009590 0x7f395000bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:06:58.122 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:58.106 7f3966ffd700 1 -- 172.21.2.102:0/2056572441 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f395000c300 con 0x7f3950006770 2021-02-10T11:06:58.122 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:58.106 7f3966ffd700 1 -- 172.21.2.102:0/2056572441 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f395000c520 con 0x7f3950003a60 2021-02-10T11:06:58.122 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:58.106 7f3966ffd700 1 -- 172.21.2.102:0/2056572441 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f395000c740 con 0x7f3950009590 2021-02-10T11:06:58.122 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:58.106 7f39677fe700 1 --2- 172.21.2.102:0/2056572441 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3950006770 0x7f3950008c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:06:58.122 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:58.106 7f3967fff700 1 --2- 172.21.2.102:0/2056572441 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3950003a60 0x7f3950005ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:06:58.122 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:58.106 7f396c881700 1 --2- 172.21.2.102:0/2056572441 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3950009590 0x7f395000bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:06:58.122 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:58.106 7f39677fe700 1 -- 172.21.2.102:0/2056572441 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3950009590 msgr2=0x7f395000bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:06:58.122 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:58.106 7f39677fe700 1 --2- 172.21.2.102:0/2056572441 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3950009590 0x7f395000bab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:06:58.123 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:58.106 7f39677fe700 1 -- 172.21.2.102:0/2056572441 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3950003a60 msgr2=0x7f3950005ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:06:58.123 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:58.106 7f39677fe700 1 --2- 172.21.2.102:0/2056572441 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3950003a60 0x7f3950005ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:06:58.123 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:58.106 7f39677fe700 1 -- 172.21.2.102:0/2056572441 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f395000c9e0 con 0x7f3950006770 2021-02-10T11:06:58.123 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:58.106 7f396c881700 1 -- 172.21.2.102:0/2056572441 reap_dead start 2021-02-10T11:06:58.123 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:58.106 7f39677fe700 1 --2- 172.21.2.102:0/2056572441 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3950006770 0x7f3950008c90 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 rx=0x7f395c006360 tx=0x7f395c006820).ready entity=mon.0 client_cookie=2243c21193e5b401 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:06:58.123 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:58.106 7f3966ffd700 1 -- 172.21.2.102:0/2056572441 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f395c006560 con 0x7f3950006770 2021-02-10T11:06:58.123 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:58.106 7f3966ffd700 1 -- 172.21.2.102:0/2056572441 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f395c000c90 con 0x7f3950006770 2021-02-10T11:06:58.123 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:58.106 7f3966ffd700 1 -- 172.21.2.102:0/2056572441 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f395c0502b0 con 0x7f3950006770 2021-02-10T11:06:58.123 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:58.107 7f396e264700 1 -- 172.21.2.102:0/2056572441 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3950006770 msgr2=0x7f3950008c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:06:58.123 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:58.107 7f396e264700 1 --2- 172.21.2.102:0/2056572441 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3950006770 0x7f3950008c90 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 rx=0x7f395c006360 tx=0x7f395c006820).stop 2021-02-10T11:06:58.123 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:58.107 7f396e264700 1 -- 172.21.2.102:0/2056572441 shutdown_connections 2021-02-10T11:06:58.123 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:58.107 7f396e264700 1 --2- 172.21.2.102:0/2056572441 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3950006770 0x7f3950008c90 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:06:58.123 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:58.107 7f396e264700 1 -- 172.21.2.102:0/2056572441 >> 172.21.2.102:0/2056572441 conn(0x7f39680ce3e0 msgr2=0x7f39680e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:06:58.123 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:58.107 7f396e264700 1 -- 172.21.2.102:0/2056572441 shutdown_connections 2021-02-10T11:06:58.124 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:58.107 7f396e264700 1 -- 172.21.2.102:0/2056572441 wait complete. 2021-02-10T11:06:58.124 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:58.122 7f396e264700 1 Processor -- start 2021-02-10T11:06:58.124 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:58.122 7f396e264700 1 -- start start 2021-02-10T11:06:58.124 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:58.122 7f396e264700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f39680ae340 0x7f39680ad780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:06:58.124 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:58.122 7f396e264700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f39680ae7d0 0x7f39680aebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:06:58.124 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:58.122 7f396e264700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f396809fc70 0x7f39680a0070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:06:58.124 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:58.122 7f396e264700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39680af1f0 con 0x7f39680ae340 2021-02-10T11:06:58.124 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:58.122 7f396e264700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39680ade30 con 0x7f39680ae7d0 2021-02-10T11:06:58.124 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:58.122 7f396e264700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39680a0690 con 0x7f396809fc70 2021-02-10T11:06:58.124 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:58.122 7f3967fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f39680ae340 0x7f39680ad780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:06:58.124 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:58.122 7f3967fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f39680ae340 0x7f39680ad780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:51196/0 (socket says 172.21.2.102:51196) 2021-02-10T11:06:58.124 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:58.122 7f3967fff700 1 -- 172.21.2.102:0/1213652810 learned_addr learned my addr 172.21.2.102:0/1213652810 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:06:58.125 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:58.123 7f3967fff700 1 -- 172.21.2.102:0/1213652810 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f396809fc70 msgr2=0x7f39680a0070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:06:58.125 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:58.123 7f3967fff700 1 --2- 172.21.2.102:0/1213652810 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f396809fc70 0x7f39680a0070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:06:58.126 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:58.123 7f3967fff700 1 -- 172.21.2.102:0/1213652810 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f39680ae7d0 msgr2=0x7f39680aebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:06:58.126 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:58.123 7f3967fff700 1 --2- 172.21.2.102:0/1213652810 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f39680ae7d0 0x7f39680aebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:06:58.126 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:58.123 7f3967fff700 1 -- 172.21.2.102:0/1213652810 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39680a08b0 con 0x7f39680ae340 2021-02-10T11:06:58.126 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:58.123 7f3967fff700 1 --2- 172.21.2.102:0/1213652810 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f39680ae340 0x7f39680ad780 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 rx=0x7f3958007680 tx=0x7f395800a940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:06:58.126 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:58.123 7f3964ff9700 1 -- 172.21.2.102:0/1213652810 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3958001600 con 0x7f39680ae340 2021-02-10T11:06:58.126 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:58.123 7f3964ff9700 1 -- 172.21.2.102:0/1213652810 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f395800ad30 con 0x7f39680ae340 2021-02-10T11:06:58.126 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:58.123 7f3964ff9700 1 -- 172.21.2.102:0/1213652810 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f39580043a0 con 0x7f39680ae340 2021-02-10T11:06:58.126 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:58.123 7f396e264700 1 -- 172.21.2.102:0/1213652810 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39680a08b0 con 0x7f39680ae340 2021-02-10T11:06:58.126 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:58.123 7f396e264700 1 -- 172.21.2.102:0/1213652810 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39680a08b0 con 0x7f39680ae340 2021-02-10T11:06:58.126 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:58.124 7f3964ff9700 1 -- 172.21.2.102:0/1213652810 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f3958001af0 con 0x7f39680ae340 2021-02-10T11:06:58.126 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:58.124 7f3964ff9700 1 --2- 172.21.2.102:0/1213652810 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3948018730 0x7f394801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:06:58.126 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:58.124 7f3964ff9700 1 -- 172.21.2.102:0/1213652810 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f3958032600 con 0x7f39680ae340 2021-02-10T11:06:58.126 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:58.124 7f39677fe700 1 --2- 172.21.2.102:0/1213652810 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3948018730 0x7f394801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:06:58.127 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:58.124 7f39677fe700 1 --2- 172.21.2.102:0/1213652810 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3948018730 0x7f394801abc0 crc :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:06:58.152 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:58.154 7f396e264700 1 -- 172.21.2.102:0/1213652810 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39680af1f0 con 0x7f39680ae340 2021-02-10T11:06:58.155 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:58.157 7f3964ff9700 1 -- 172.21.2.102:0/1213652810 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f39580018c0 con 0x7f39680ae340 2021-02-10T11:06:58.285 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:58.287 7f396e264700 1 -- 172.21.2.102:0/1213652810 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344373", "target": ["mgr", ""]}) v1 -- 0x7f39680af1f0 con 0x7f3948018730 2021-02-10T11:06:58.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:58.290 7f3964ff9700 1 -- 172.21.2.102:0/1213652810 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f395c04acc0 con 0x7f3948018730 2021-02-10T11:06:58.292 INFO:teuthology.orchestra.run.gibba002.stdout:/volumes/_nogroup/subvolume_0000000000344373/da217f3e-d0cf-4f72-9340-922b0ba7d735 2021-02-10T11:06:58.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:58.295 7f396e264700 1 -- 172.21.2.102:0/1213652810 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3948018730 msgr2=0x7f394801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:06:58.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:58.295 7f396e264700 1 --2- 172.21.2.102:0/1213652810 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3948018730 0x7f394801abc0 crc :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:06:58.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:58.295 7f396e264700 1 -- 172.21.2.102:0/1213652810 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f39680ae340 msgr2=0x7f39680ad780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:06:58.294 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:58.295 7f396e264700 1 --2- 172.21.2.102:0/1213652810 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f39680ae340 0x7f39680ad780 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 rx=0x7f3958007680 tx=0x7f395800a940).stop 2021-02-10T11:06:58.294 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:58.295 7f396e264700 1 -- 172.21.2.102:0/1213652810 shutdown_connections 2021-02-10T11:06:58.294 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:58.295 7f396e264700 1 --2- 172.21.2.102:0/1213652810 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f396809fc70 0x7f39680a0070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:06:58.294 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:58.295 7f396e264700 1 --2- 172.21.2.102:0/1213652810 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3948018730 0x7f394801abc0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:06:58.294 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:58.295 7f396e264700 1 --2- 172.21.2.102:0/1213652810 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f39680ae7d0 0x7f39680aebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:06:58.294 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:58.295 7f396e264700 1 --2- 172.21.2.102:0/1213652810 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f39680ae340 0x7f39680ad780 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:06:58.294 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:58.295 7f396e264700 1 -- 172.21.2.102:0/1213652810 >> 172.21.2.102:0/1213652810 conn(0x7f39680af720 msgr2=0x7f39680c3280 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:06:58.294 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:58.295 7f396e264700 1 -- 172.21.2.102:0/1213652810 shutdown_connections 2021-02-10T11:06:58.294 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:06:58.295 7f396e264700 1 -- 172.21.2.102:0/1213652810 wait complete. 2021-02-10T11:06:58.301 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000344373 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000344373/da217f3e-d0cf-4f72-9340-922b0ba7d735 2021-02-10T11:06:58.301 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:06:58.302 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:06:58.302 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:06:58.636 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:06:58.636 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:06:58.636 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344373/da217f3e-d0cf-4f72-9340-922b0ba7d735/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:06:59.013 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:06:59.013 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:06:59.014 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0251216 s, 41.7 MB/s 2021-02-10T11:06:59.015 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:06:59.015 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:06:59.015 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:06:59.350 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:06:59.350 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:06:59.350 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344373/da217f3e-d0cf-4f72-9340-922b0ba7d735/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:06:59.726 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:06:59.726 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:06:59.726 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0247448 s, 42.4 MB/s 2021-02-10T11:06:59.727 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:06:59.727 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:06:59.727 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:07:00.062 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:07:00.062 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:07:00.062 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344373/da217f3e-d0cf-4f72-9340-922b0ba7d735/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:07:00.439 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:07:00.439 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:07:00.439 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0258485 s, 40.6 MB/s 2021-02-10T11:07:00.440 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:07:00.441 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:07:00.441 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:07:00.776 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:07:00.776 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:07:00.776 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344373/da217f3e-d0cf-4f72-9340-922b0ba7d735/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:07:01.151 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:07:01.151 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:07:01.152 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0243896 s, 43.0 MB/s 2021-02-10T11:07:01.152 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:07:01.153 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:07:01.153 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:07:01.489 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:07:01.490 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:07:01.490 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344373/da217f3e-d0cf-4f72-9340-922b0ba7d735/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:07:01.870 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:07:01.871 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:07:01.871 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0250057 s, 41.9 MB/s 2021-02-10T11:07:01.872 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:07:01.872 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:07:01.872 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:07:02.210 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:07:02.210 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:07:02.210 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344373/da217f3e-d0cf-4f72-9340-922b0ba7d735/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:07:02.588 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:07:02.588 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:07:02.589 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0263026 s, 39.9 MB/s 2021-02-10T11:07:02.590 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:07:02.590 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:07:02.590 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:07:02.924 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:07:02.925 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:07:02.925 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344373/da217f3e-d0cf-4f72-9340-922b0ba7d735/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:07:03.299 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:07:03.299 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:07:03.300 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0255214 s, 41.1 MB/s 2021-02-10T11:07:03.300 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:07:03.301 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:07:03.301 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:07:03.635 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:07:03.636 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:07:03.636 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344373/da217f3e-d0cf-4f72-9340-922b0ba7d735/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:07:04.010 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:07:04.010 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:07:04.011 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0258535 s, 40.6 MB/s 2021-02-10T11:07:04.012 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:07:04.012 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:07:04.012 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:07:04.347 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:07:04.347 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:07:04.348 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344373/da217f3e-d0cf-4f72-9340-922b0ba7d735/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:07:04.723 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:07:04.723 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:07:04.723 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0251576 s, 41.7 MB/s 2021-02-10T11:07:04.724 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:07:04.725 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:07:04.725 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:07:05.059 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:07:05.059 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:07:05.059 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344373/da217f3e-d0cf-4f72-9340-922b0ba7d735/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:07:05.437 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:07:05.438 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:07:05.438 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0263036 s, 39.9 MB/s 2021-02-10T11:07:05.439 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000344374 2021-02-10T11:07:05.875 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:05.859 7f1f1f754700 1 Processor -- start 2021-02-10T11:07:05.875 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:05.859 7f1f1f754700 1 -- start start 2021-02-10T11:07:05.875 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:05.860 7f1f1f754700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f1f181b0870 0x7f1f180f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:07:05.875 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:05.860 7f1f1f754700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f181b0c70 con 0x7f1f180efe00 2021-02-10T11:07:05.876 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:05.860 7f1f1f754700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f180f0b50 con 0x7f1f181b0470 2021-02-10T11:07:05.876 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:05.860 7f1f1f754700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f180e5ea0 con 0x7f1f181b0870 2021-02-10T11:07:05.876 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:05.860 7f1f1cd6f700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f1f181b0870 0x7f1f180f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:07:05.876 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:05.860 7f1f1dd71700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f1f180efe00 0x7f1f180f0200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:43424/0 (socket says 172.21.2.102:43424) 2021-02-10T11:07:05.876 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:05.860 7f1f1cd6f700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f1f181b0870 0x7f1f180f20e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:51200/0 (socket says 172.21.2.102:51200) 2021-02-10T11:07:05.876 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:05.860 7f1f1dd71700 1 -- 172.21.2.102:0/113282009 learned_addr learned my addr 172.21.2.102:0/113282009 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:07:05.876 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:05.861 7f1f1cd6f700 1 -- 172.21.2.102:0/113282009 >> v1:172.21.2.113:6789/0 conn(0x7f1f181b0470 legacy=0x7f1f180ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:07:05.876 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:05.861 7f1f1cd6f700 1 -- 172.21.2.102:0/113282009 >> v1:172.21.2.102:6789/0 conn(0x7f1f180efe00 legacy=0x7f1f180f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:07:05.876 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:05.861 7f1f1cd6f700 1 -- 172.21.2.102:0/113282009 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f180e52e0 con 0x7f1f181b0870 2021-02-10T11:07:05.876 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:05.861 7f1f1cd6f700 1 --2- 172.21.2.102:0/113282009 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1f181b0870 0x7f1f180f20e0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 rx=0x7f1f0c006650 tx=0x7f1f0c009a80).ready entity=mon.0 client_cookie=4ad23403b16226db server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:07:05.876 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:05.861 7f1f0bfff700 1 -- 172.21.2.102:0/113282009 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1f0c00c0c0 con 0x7f1f181b0870 2021-02-10T11:07:05.876 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:05.861 7f1f0bfff700 1 -- 172.21.2.102:0/113282009 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f0c00c340 con 0x7f1f181b0870 2021-02-10T11:07:05.877 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:05.861 7f1f0bfff700 1 -- 172.21.2.102:0/113282009 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1f0c00c860 con 0x7f1f181b0870 2021-02-10T11:07:05.877 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:05.861 7f1f1f754700 1 -- 172.21.2.102:0/113282009 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1f181b0870 msgr2=0x7f1f180f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:07:05.877 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:05.861 7f1f1f754700 1 --2- 172.21.2.102:0/113282009 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1f181b0870 0x7f1f180f20e0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 rx=0x7f1f0c006650 tx=0x7f1f0c009a80).stop 2021-02-10T11:07:05.877 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:05.862 7f1f1f754700 1 -- 172.21.2.102:0/113282009 shutdown_connections 2021-02-10T11:07:05.877 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:05.862 7f1f1f754700 1 --2- 172.21.2.102:0/113282009 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1f181b0870 0x7f1f180f20e0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:07:05.877 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:05.862 7f1f1f754700 1 -- 172.21.2.102:0/113282009 >> 172.21.2.102:0/113282009 conn(0x7f1f180ce3e0 msgr2=0x7f1f180e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:07:05.877 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:05.862 7f1f1f754700 1 -- 172.21.2.102:0/113282009 shutdown_connections 2021-02-10T11:07:05.877 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:05.862 7f1f1f754700 1 -- 172.21.2.102:0/113282009 wait complete. 2021-02-10T11:07:05.879 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:05.878 7f1f1f754700 1 Processor -- start 2021-02-10T11:07:05.879 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:05.878 7f1f1f754700 1 -- start start 2021-02-10T11:07:05.879 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:05.878 7f1f1f754700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1f180a3b50 0x7f1f181b3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:07:05.879 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:05.878 7f1f1f754700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1f181c25b0 0x7f1f181b3d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:07:05.879 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:05.878 7f1f1f754700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1f181b4350 0x7f1f181b18a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:07:05.879 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:05.878 7f1f1f754700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f181b3190 con 0x7f1f180a3b50 2021-02-10T11:07:05.879 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:05.878 7f1f1f754700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f18087750 con 0x7f1f181b4350 2021-02-10T11:07:05.879 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:05.878 7f1f1f754700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f180a6bf0 con 0x7f1f181c25b0 2021-02-10T11:07:05.879 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:05.878 7f1f1d570700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1f180a3b50 0x7f1f181b3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:07:05.879 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:05.878 7f1f1d570700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1f180a3b50 0x7f1f181b3710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:51208/0 (socket says 172.21.2.102:51208) 2021-02-10T11:07:05.880 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:05.878 7f1f1d570700 1 -- 172.21.2.102:0/2333190452 learned_addr learned my addr 172.21.2.102:0/2333190452 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:07:05.880 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:05.879 7f1f1d570700 1 -- 172.21.2.102:0/2333190452 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1f181c25b0 msgr2=0x7f1f181b3d30 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:07:05.880 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:05.879 7f1f1d570700 1 --2- 172.21.2.102:0/2333190452 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1f181c25b0 0x7f1f181b3d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:07:05.880 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:05.879 7f1f1d570700 1 -- 172.21.2.102:0/2333190452 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1f181b4350 msgr2=0x7f1f181b18a0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:07:05.880 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:05.879 7f1f1d570700 1 --2- 172.21.2.102:0/2333190452 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1f181b4350 0x7f1f181b18a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:07:05.880 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:05.879 7f1f1d570700 1 -- 172.21.2.102:0/2333190452 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f181b1ec0 con 0x7f1f180a3b50 2021-02-10T11:07:05.880 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:05.879 7f1f1d570700 1 --2- 172.21.2.102:0/2333190452 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1f180a3b50 0x7f1f181b3710 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 rx=0x7f1f1400b5b0 tx=0x7f1f1400b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:07:05.880 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:05.879 7f1f09ffb700 1 -- 172.21.2.102:0/2333190452 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1f1400bd40 con 0x7f1f180a3b50 2021-02-10T11:07:05.880 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:05.879 7f1f09ffb700 1 -- 172.21.2.102:0/2333190452 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f14006020 con 0x7f1f180a3b50 2021-02-10T11:07:05.880 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:05.879 7f1f09ffb700 1 -- 172.21.2.102:0/2333190452 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1f140054a0 con 0x7f1f180a3b50 2021-02-10T11:07:05.880 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:05.879 7f1f1f754700 1 -- 172.21.2.102:0/2333190452 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f181b1ec0 con 0x7f1f180a3b50 2021-02-10T11:07:05.880 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:05.879 7f1f1f754700 1 -- 172.21.2.102:0/2333190452 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f181b2450 con 0x7f1f180a3b50 2021-02-10T11:07:05.881 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:05.880 7f1f09ffb700 1 -- 172.21.2.102:0/2333190452 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1f14006200 con 0x7f1f180a3b50 2021-02-10T11:07:05.881 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:05.880 7f1f09ffb700 1 --2- 172.21.2.102:0/2333190452 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1ef80186d0 0x7f1ef801abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:07:05.881 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:05.880 7f1f09ffb700 1 -- 172.21.2.102:0/2333190452 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f1f14032670 con 0x7f1f180a3b50 2021-02-10T11:07:05.881 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:05.880 7f1f1cd6f700 1 --2- 172.21.2.102:0/2333190452 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1ef80186d0 0x7f1ef801abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:07:05.881 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:05.880 7f1f1cd6f700 1 --2- 172.21.2.102:0/2333190452 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1ef80186d0 0x7f1ef801abd0 crc :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:07:05.907 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:05.909 7f1f1f754700 1 -- 172.21.2.102:0/2333190452 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f1807e6b0 con 0x7f1f180a3b50 2021-02-10T11:07:05.909 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:05.911 7f1f09ffb700 1 -- 172.21.2.102:0/2333190452 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f1f14000c10 con 0x7f1f180a3b50 2021-02-10T11:07:06.039 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:06.041 7f1f1f754700 1 -- 172.21.2.102:0/2333190452 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344374", "target": ["mgr", ""]}) v1 -- 0x7f1f180ce3e0 con 0x7f1ef80186d0 2021-02-10T11:07:06.064 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:06.066 7f1f09ffb700 1 -- 172.21.2.102:0/2333190452 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f1f0c0092b0 con 0x7f1ef80186d0 2021-02-10T11:07:06.072 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:06.073 7f1f1f754700 1 -- 172.21.2.102:0/2333190452 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1ef80186d0 msgr2=0x7f1ef801abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:07:06.072 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:06.073 7f1f1f754700 1 --2- 172.21.2.102:0/2333190452 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1ef80186d0 0x7f1ef801abd0 crc :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:07:06.072 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:06.073 7f1f1f754700 1 -- 172.21.2.102:0/2333190452 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1f180a3b50 msgr2=0x7f1f181b3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:07:06.072 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:06.073 7f1f1f754700 1 --2- 172.21.2.102:0/2333190452 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1f180a3b50 0x7f1f181b3710 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 rx=0x7f1f1400b5b0 tx=0x7f1f1400b950).stop 2021-02-10T11:07:06.073 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:06.074 7f1f1f754700 1 -- 172.21.2.102:0/2333190452 shutdown_connections 2021-02-10T11:07:06.073 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:06.074 7f1f1f754700 1 --2- 172.21.2.102:0/2333190452 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1ef80186d0 0x7f1ef801abd0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:07:06.073 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:06.074 7f1f1f754700 1 --2- 172.21.2.102:0/2333190452 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1f181b4350 0x7f1f181b18a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:07:06.073 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:06.074 7f1f1f754700 1 --2- 172.21.2.102:0/2333190452 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1f181c25b0 0x7f1f181b3d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:07:06.073 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:06.074 7f1f1f754700 1 --2- 172.21.2.102:0/2333190452 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1f180a3b50 0x7f1f181b3710 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:07:06.073 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:06.074 7f1f1f754700 1 -- 172.21.2.102:0/2333190452 >> 172.21.2.102:0/2333190452 conn(0x7f1f180886d0 msgr2=0x7f1f18081c40 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:07:06.073 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:06.074 7f1f1f754700 1 -- 172.21.2.102:0/2333190452 shutdown_connections 2021-02-10T11:07:06.073 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:06.074 7f1f1f754700 1 -- 172.21.2.102:0/2333190452 wait complete. 2021-02-10T11:07:06.080 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000344374 2021-02-10T11:07:06.511 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:06.496 7f84e1420700 1 Processor -- start 2021-02-10T11:07:06.511 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:06.496 7f84e1420700 1 -- start start 2021-02-10T11:07:06.511 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:06.496 7f84e1420700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f84dc1b0470 0x7f84dc1b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:07:06.511 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:06.496 7f84e1420700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f84dc0ef610 0x7f84dc0efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:07:06.511 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:06.496 7f84e1420700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f84dc0f0030 0x7f84dc0f0430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:07:06.511 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:06.496 7f84e1420700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84dc0e5ea0 con 0x7f84dc0f0030 2021-02-10T11:07:06.511 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:06.496 7f84e1420700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84dc0f0b50 con 0x7f84dc1b0470 2021-02-10T11:07:06.511 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:06.496 7f84e1420700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84dc0e52e0 con 0x7f84dc0ef610 2021-02-10T11:07:06.511 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:06.497 7f84db87e700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f84dc0f0030 0x7f84dc0f0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:07:06.512 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:06.497 7f84db87e700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f84dc0f0030 0x7f84dc0f0430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:51216/0 (socket says 172.21.2.102:51216) 2021-02-10T11:07:06.512 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:06.497 7f84db87e700 1 -- 172.21.2.102:0/513701890 learned_addr learned my addr 172.21.2.102:0/513701890 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:07:06.512 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:06.497 7f84db87e700 1 -- 172.21.2.102:0/513701890 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f84dc0ef610 msgr2=0x7f84dc0efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:07:06.512 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:06.497 7f84db87e700 1 --2- 172.21.2.102:0/513701890 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f84dc0ef610 0x7f84dc0efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:07:06.512 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:06.497 7f84db87e700 1 -- 172.21.2.102:0/513701890 >> v2:172.21.2.113:3300/0 conn(0x7f84dc1b0470 msgr2=0x7f84dc1b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:07:06.512 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:06.497 7f84db87e700 1 --2- 172.21.2.102:0/513701890 >> v2:172.21.2.113:3300/0 conn(0x7f84dc1b0470 0x7f84dc1b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:07:06.512 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:06.497 7f84db87e700 1 -- 172.21.2.102:0/513701890 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84dc0f10d0 con 0x7f84dc0f0030 2021-02-10T11:07:06.512 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:06.497 7f84db87e700 1 --2- 172.21.2.102:0/513701890 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f84dc0f0030 0x7f84dc0f0430 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 rx=0x7f84cc006b80 tx=0x7f84cc009b90).ready entity=mon.0 client_cookie=6c59f01357db1786 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:07:06.512 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:06.497 7f84da07b700 1 -- 172.21.2.102:0/513701890 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f84cc0022e0 con 0x7f84dc0f0030 2021-02-10T11:07:06.512 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:06.497 7f84da07b700 1 -- 172.21.2.102:0/513701890 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84cc002560 con 0x7f84dc0f0030 2021-02-10T11:07:06.512 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:06.497 7f84da07b700 1 -- 172.21.2.102:0/513701890 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f84cc002a80 con 0x7f84dc0f0030 2021-02-10T11:07:06.512 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:06.498 7f84e1420700 1 -- 172.21.2.102:0/513701890 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f84dc0f0030 msgr2=0x7f84dc0f0430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:07:06.513 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:06.498 7f84e1420700 1 --2- 172.21.2.102:0/513701890 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f84dc0f0030 0x7f84dc0f0430 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 rx=0x7f84cc006b80 tx=0x7f84cc009b90).stop 2021-02-10T11:07:06.513 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:06.498 7f84e1420700 1 -- 172.21.2.102:0/513701890 shutdown_connections 2021-02-10T11:07:06.513 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:06.498 7f84e1420700 1 --2- 172.21.2.102:0/513701890 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f84dc0ef610 0x7f84dc0efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:07:06.513 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:06.498 7f84e1420700 1 --2- 172.21.2.102:0/513701890 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f84dc0f0030 0x7f84dc0f0430 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:07:06.513 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:06.498 7f84e1420700 1 --2- 172.21.2.102:0/513701890 >> v2:172.21.2.113:3300/0 conn(0x7f84dc1b0470 0x7f84dc1b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:07:06.513 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:06.498 7f84e1420700 1 -- 172.21.2.102:0/513701890 >> 172.21.2.102:0/513701890 conn(0x7f84dc0ce3e0 msgr2=0x7f84dc0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:07:06.513 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:06.498 7f84e1420700 1 -- 172.21.2.102:0/513701890 shutdown_connections 2021-02-10T11:07:06.513 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:06.498 7f84e1420700 1 -- 172.21.2.102:0/513701890 wait complete. 2021-02-10T11:07:06.515 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:06.513 7f84e1420700 1 Processor -- start 2021-02-10T11:07:06.515 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:06.513 7f84e1420700 1 -- start start 2021-02-10T11:07:06.515 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:06.514 7f84e1420700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f84dc190920 0x7f84dc08f600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:07:06.515 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:06.514 7f84e1420700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f84dc190d20 0x7f84dc08fc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:07:06.515 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:06.514 7f84e1420700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f84dc193270 0x7f84dc193670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:07:06.515 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:06.514 7f84e1420700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84dc090470 con 0x7f84dc190920 2021-02-10T11:07:06.515 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:06.514 7f84e1420700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84dc193c90 con 0x7f84dc190d20 2021-02-10T11:07:06.515 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:06.514 7f84e1420700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84dc193e70 con 0x7f84dc193270 2021-02-10T11:07:06.516 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:06.514 7f84db07d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f84dc190920 0x7f84dc08f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:07:06.516 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:06.514 7f84db07d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f84dc190920 0x7f84dc08f600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:51218/0 (socket says 172.21.2.102:51218) 2021-02-10T11:07:06.516 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:06.514 7f84db07d700 1 -- 172.21.2.102:0/3327728412 learned_addr learned my addr 172.21.2.102:0/3327728412 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:07:06.516 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:06.514 7f84db07d700 1 -- 172.21.2.102:0/3327728412 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f84dc193270 msgr2=0x7f84dc193670 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:07:06.516 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:06.514 7f84db07d700 1 --2- 172.21.2.102:0/3327728412 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f84dc193270 0x7f84dc193670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:07:06.516 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:06.514 7f84db07d700 1 -- 172.21.2.102:0/3327728412 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f84dc190d20 msgr2=0x7f84dc08fc20 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:07:06.516 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:06.514 7f84db07d700 1 --2- 172.21.2.102:0/3327728412 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f84dc190d20 0x7f84dc08fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:07:06.516 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:06.514 7f84db07d700 1 -- 172.21.2.102:0/3327728412 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84dc194090 con 0x7f84dc190920 2021-02-10T11:07:06.516 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:06.514 7f84db07d700 1 --2- 172.21.2.102:0/3327728412 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f84dc190920 0x7f84dc08f600 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 rx=0x7f84d400b8c0 tx=0x7f84d400bc60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:07:06.516 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:06.515 7f84c3fff700 1 -- 172.21.2.102:0/3327728412 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f84d400c2f0 con 0x7f84dc190920 2021-02-10T11:07:06.516 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:06.515 7f84c3fff700 1 -- 172.21.2.102:0/3327728412 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84d400c2f0 con 0x7f84dc190920 2021-02-10T11:07:06.516 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:06.515 7f84e1420700 1 -- 172.21.2.102:0/3327728412 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84dc194090 con 0x7f84dc190920 2021-02-10T11:07:06.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:06.515 7f84c3fff700 1 -- 172.21.2.102:0/3327728412 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f84d400c570 con 0x7f84dc190920 2021-02-10T11:07:06.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:06.515 7f84e1420700 1 -- 172.21.2.102:0/3327728412 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84dc194090 con 0x7f84dc190920 2021-02-10T11:07:06.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:06.515 7f84c3fff700 1 -- 172.21.2.102:0/3327728412 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f84d4001010 con 0x7f84dc190920 2021-02-10T11:07:06.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:06.515 7f84c3fff700 1 --2- 172.21.2.102:0/3327728412 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f84bc018730 0x7f84bc01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:07:06.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:06.515 7f84c3fff700 1 -- 172.21.2.102:0/3327728412 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f84d4032b70 con 0x7f84dc190920 2021-02-10T11:07:06.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:06.516 7f84da87c700 1 --2- 172.21.2.102:0/3327728412 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f84bc018730 0x7f84bc01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:07:06.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:06.516 7f84da87c700 1 --2- 172.21.2.102:0/3327728412 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f84bc018730 0x7f84bc01ac10 crc :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:07:06.544 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:06.546 7f84e1420700 1 -- 172.21.2.102:0/3327728412 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84dc105af0 con 0x7f84dc190920 2021-02-10T11:07:06.546 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:06.549 7f84c3fff700 1 -- 172.21.2.102:0/3327728412 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f84d401f8b0 con 0x7f84dc190920 2021-02-10T11:07:06.678 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:06.680 7f84e1420700 1 -- 172.21.2.102:0/3327728412 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344374", "target": ["mgr", ""]}) v1 -- 0x7f84dc0f0b50 con 0x7f84bc018730 2021-02-10T11:07:06.681 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:06.683 7f84c3fff700 1 -- 172.21.2.102:0/3327728412 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f84d0006e40 con 0x7f84bc018730 2021-02-10T11:07:06.685 INFO:teuthology.orchestra.run.gibba002.stdout:/volumes/_nogroup/subvolume_0000000000344374/6e0bdff8-4610-4f9c-9e4c-777e1d4c4ea8 2021-02-10T11:07:06.686 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:06.689 7f84e1420700 1 -- 172.21.2.102:0/3327728412 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f84bc018730 msgr2=0x7f84bc01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:07:06.687 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:06.689 7f84e1420700 1 --2- 172.21.2.102:0/3327728412 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f84bc018730 0x7f84bc01ac10 crc :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:07:06.687 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:06.689 7f84e1420700 1 -- 172.21.2.102:0/3327728412 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f84dc190920 msgr2=0x7f84dc08f600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:07:06.687 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:06.689 7f84e1420700 1 --2- 172.21.2.102:0/3327728412 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f84dc190920 0x7f84dc08f600 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 rx=0x7f84d400b8c0 tx=0x7f84d400bc60).stop 2021-02-10T11:07:06.687 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:06.689 7f84e1420700 1 -- 172.21.2.102:0/3327728412 shutdown_connections 2021-02-10T11:07:06.687 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:06.689 7f84e1420700 1 --2- 172.21.2.102:0/3327728412 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f84dc193270 0x7f84dc193670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:07:06.687 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:06.689 7f84e1420700 1 --2- 172.21.2.102:0/3327728412 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f84bc018730 0x7f84bc01ac10 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:07:06.687 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:06.689 7f84e1420700 1 --2- 172.21.2.102:0/3327728412 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f84dc190d20 0x7f84dc08fc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:07:06.687 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:06.689 7f84e1420700 1 --2- 172.21.2.102:0/3327728412 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f84dc190920 0x7f84dc08f600 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:07:06.687 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:06.689 7f84e1420700 1 -- 172.21.2.102:0/3327728412 >> 172.21.2.102:0/3327728412 conn(0x7f84dc0bf4e0 msgr2=0x7f84dc0be3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:07:06.688 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:06.689 7f84e1420700 1 -- 172.21.2.102:0/3327728412 shutdown_connections 2021-02-10T11:07:06.688 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:06.689 7f84e1420700 1 -- 172.21.2.102:0/3327728412 wait complete. 2021-02-10T11:07:06.695 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000344374 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000344374/6e0bdff8-4610-4f9c-9e4c-777e1d4c4ea8 2021-02-10T11:07:06.695 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:07:06.695 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:07:06.695 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:07:07.030 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:07:07.031 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:07:07.031 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344374/6e0bdff8-4610-4f9c-9e4c-777e1d4c4ea8/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:07:07.414 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:07:07.415 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:07:07.415 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.027264 s, 38.5 MB/s 2021-02-10T11:07:07.416 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:07:07.416 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:07:07.416 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:07:07.753 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:07:07.753 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:07:07.753 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344374/6e0bdff8-4610-4f9c-9e4c-777e1d4c4ea8/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:07:08.127 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:07:08.128 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:07:08.128 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.023917 s, 43.8 MB/s 2021-02-10T11:07:08.129 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:07:08.129 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:07:08.129 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:07:08.464 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:07:08.465 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:07:08.465 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344374/6e0bdff8-4610-4f9c-9e4c-777e1d4c4ea8/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:07:08.837 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:07:08.837 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:07:08.838 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0252515 s, 41.5 MB/s 2021-02-10T11:07:08.838 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:07:08.839 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:07:08.839 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:07:09.173 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:07:09.174 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:07:09.174 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344374/6e0bdff8-4610-4f9c-9e4c-777e1d4c4ea8/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:07:09.553 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:07:09.553 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:07:09.553 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0269451 s, 38.9 MB/s 2021-02-10T11:07:09.554 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:07:09.554 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:07:09.554 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:07:09.888 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:07:09.888 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:07:09.888 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344374/6e0bdff8-4610-4f9c-9e4c-777e1d4c4ea8/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:07:10.263 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:07:10.264 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:07:10.264 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0256846 s, 40.8 MB/s 2021-02-10T11:07:10.264 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:07:10.265 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:07:10.265 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:07:10.598 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:07:10.599 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:07:10.599 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344374/6e0bdff8-4610-4f9c-9e4c-777e1d4c4ea8/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:07:10.973 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:07:10.973 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:07:10.973 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0250993 s, 41.8 MB/s 2021-02-10T11:07:10.974 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:07:10.974 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:07:10.974 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:07:11.307 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:07:11.308 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:07:11.308 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344374/6e0bdff8-4610-4f9c-9e4c-777e1d4c4ea8/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:07:11.682 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:07:11.682 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:07:11.683 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0248056 s, 42.3 MB/s 2021-02-10T11:07:11.683 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:07:11.684 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:07:11.684 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:07:12.017 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:07:12.017 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:07:12.018 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344374/6e0bdff8-4610-4f9c-9e4c-777e1d4c4ea8/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:07:12.389 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:07:12.390 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:07:12.390 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.022301 s, 47.0 MB/s 2021-02-10T11:07:12.391 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:07:12.391 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:07:12.391 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:07:12.725 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:07:12.726 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:07:12.726 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344374/6e0bdff8-4610-4f9c-9e4c-777e1d4c4ea8/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:07:13.101 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:07:13.101 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:07:13.101 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0246137 s, 42.6 MB/s 2021-02-10T11:07:13.102 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:07:13.102 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:07:13.102 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:07:13.436 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:07:13.437 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:07:13.437 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344374/6e0bdff8-4610-4f9c-9e4c-777e1d4c4ea8/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:07:13.499 DEBUG:teuthology.orchestra.run.gibba002:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:07:13.502 DEBUG:teuthology.orchestra.run.gibba013:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:07:13.810 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:07:13.811 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:07:13.811 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0244963 s, 42.8 MB/s 2021-02-10T11:07:13.812 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000344375 2021-02-10T11:07:14.242 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:14.228 7f3a8a564700 1 Processor -- start 2021-02-10T11:07:14.243 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:14.228 7f3a8a564700 1 -- start start 2021-02-10T11:07:14.243 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:14.228 7f3a8a564700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f3a841b0870 0x7f3a840f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:07:14.243 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:14.228 7f3a8a564700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3a840efe00 0x7f3a840f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:07:14.243 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:14.228 7f3a8a564700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a841b0c70 con 0x7f3a841b0470 2021-02-10T11:07:14.243 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:14.228 7f3a8a564700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a840f0b50 con 0x7f3a841b0870 2021-02-10T11:07:14.243 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:14.228 7f3a8a564700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a840f0820 con 0x7f3a840efe00 2021-02-10T11:07:14.243 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:14.229 7f3a83fff700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f3a841b0470 0x7f3a840ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:43450/0 (socket says 172.21.2.102:43450) 2021-02-10T11:07:14.243 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:14.229 7f3a83fff700 1 -- 172.21.2.102:0/1564552252 learned_addr learned my addr 172.21.2.102:0/1564552252 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:07:14.243 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:14.229 7f3a837fe700 1 --2- 172.21.2.102:0/1564552252 >> v2:172.21.2.102:3300/0 conn(0x7f3a841b0870 0x7f3a840f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:07:14.243 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:14.229 7f3a88b81700 1 --2- 172.21.2.102:0/1564552252 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3a840efe00 0x7f3a840f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:07:14.243 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:14.229 7f3a82ffd700 1 -- 172.21.2.102:0/1564552252 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 68907908 0 0) 0x7f3a74000ce0 con 0x7f3a841b0470 2021-02-10T11:07:14.243 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:14.229 7f3a82ffd700 1 -- 172.21.2.102:0/1564552252 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a6c002fe0 con 0x7f3a841b0470 2021-02-10T11:07:14.244 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:14.229 7f3a837fe700 1 -- 172.21.2.102:0/1564552252 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3a840efe00 msgr2=0x7f3a840f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:07:14.244 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:14.229 7f3a837fe700 1 --2- 172.21.2.102:0/1564552252 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3a840efe00 0x7f3a840f0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:07:14.244 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:14.229 7f3a837fe700 1 -- 172.21.2.102:0/1564552252 >> v1:172.21.2.102:6789/0 conn(0x7f3a841b0470 legacy=0x7f3a840ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:07:14.244 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:14.229 7f3a837fe700 1 -- 172.21.2.102:0/1564552252 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a840e52e0 con 0x7f3a841b0870 2021-02-10T11:07:14.244 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:14.229 7f3a837fe700 1 --2- 172.21.2.102:0/1564552252 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3a841b0870 0x7f3a840f20e0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 rx=0x7f3a78006650 tx=0x7f3a78009a80).ready entity=mon.0 client_cookie=c08e881498d9296e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:07:14.244 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:14.229 7f3a82ffd700 1 -- 172.21.2.102:0/1564552252 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3a780009b0 con 0x7f3a841b0870 2021-02-10T11:07:14.244 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:14.229 7f3a82ffd700 1 -- 172.21.2.102:0/1564552252 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a7800c0c0 con 0x7f3a841b0870 2021-02-10T11:07:14.244 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:14.229 7f3a82ffd700 1 -- 172.21.2.102:0/1564552252 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3a7800c5e0 con 0x7f3a841b0870 2021-02-10T11:07:14.244 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:14.229 7f3a8a564700 1 -- 172.21.2.102:0/1564552252 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3a841b0870 msgr2=0x7f3a840f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:07:14.244 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:14.229 7f3a8a564700 1 --2- 172.21.2.102:0/1564552252 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3a841b0870 0x7f3a840f20e0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 rx=0x7f3a78006650 tx=0x7f3a78009a80).stop 2021-02-10T11:07:14.244 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:14.230 7f3a8a564700 1 -- 172.21.2.102:0/1564552252 shutdown_connections 2021-02-10T11:07:14.244 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:14.230 7f3a8a564700 1 --2- 172.21.2.102:0/1564552252 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3a841b0870 0x7f3a840f20e0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:07:14.244 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:14.230 7f3a8a564700 1 --2- 172.21.2.102:0/1564552252 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3a840efe00 0x7f3a840f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:07:14.245 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:14.230 7f3a8a564700 1 -- 172.21.2.102:0/1564552252 >> 172.21.2.102:0/1564552252 conn(0x7f3a840ce3e0 msgr2=0x7f3a840e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:07:14.245 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:14.230 7f3a8a564700 1 -- 172.21.2.102:0/1564552252 shutdown_connections 2021-02-10T11:07:14.245 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:14.230 7f3a8a564700 1 -- 172.21.2.102:0/1564552252 wait complete. 2021-02-10T11:07:14.245 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:14.245 7f3a8a564700 1 Processor -- start 2021-02-10T11:07:14.246 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:14.245 7f3a8a564700 1 -- start start 2021-02-10T11:07:14.246 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:14.245 7f3a8a564700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3a84190920 0x7f3a840b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:07:14.246 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:14.245 7f3a8a564700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3a840b8200 0x7f3a840b8600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:07:14.246 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:14.245 7f3a8a564700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3a840c1d80 0x7f3a840c2180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:07:14.246 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:14.245 7f3a8a564700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a84190d20 con 0x7f3a840b8200 2021-02-10T11:07:14.246 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:14.245 7f3a8a564700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a840b8c20 con 0x7f3a84190920 2021-02-10T11:07:14.247 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:14.245 7f3a8a564700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a840c27a0 con 0x7f3a840c1d80 2021-02-10T11:07:14.247 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:14.245 7f3a837fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3a840b8200 0x7f3a840b8600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:07:14.247 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:14.245 7f3a837fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3a840b8200 0x7f3a840b8600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:51234/0 (socket says 172.21.2.102:51234) 2021-02-10T11:07:14.247 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:14.245 7f3a837fe700 1 -- 172.21.2.102:0/422214296 learned_addr learned my addr 172.21.2.102:0/422214296 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:07:14.247 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:14.246 7f3a837fe700 1 -- 172.21.2.102:0/422214296 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3a840c1d80 msgr2=0x7f3a840c2180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:07:14.247 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:14.246 7f3a837fe700 1 --2- 172.21.2.102:0/422214296 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3a840c1d80 0x7f3a840c2180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:07:14.247 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:14.246 7f3a837fe700 1 -- 172.21.2.102:0/422214296 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3a84190920 msgr2=0x7f3a840b7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:07:14.247 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:14.246 7f3a837fe700 1 --2- 172.21.2.102:0/422214296 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3a84190920 0x7f3a840b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:07:14.247 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:14.246 7f3a837fe700 1 -- 172.21.2.102:0/422214296 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a840c29c0 con 0x7f3a840b8200 2021-02-10T11:07:14.247 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:14.246 7f3a837fe700 1 --2- 172.21.2.102:0/422214296 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3a840b8200 0x7f3a840b8600 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 rx=0x7f3a78009630 tx=0x7f3a7800b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:07:14.247 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:14.246 7f3a80ff9700 1 -- 172.21.2.102:0/422214296 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3a780093a0 con 0x7f3a840b8200 2021-02-10T11:07:14.247 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:14.246 7f3a80ff9700 1 -- 172.21.2.102:0/422214296 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a7800bd70 con 0x7f3a840b8200 2021-02-10T11:07:14.248 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:14.246 7f3a8a564700 1 -- 172.21.2.102:0/422214296 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a840c29c0 con 0x7f3a840b8200 2021-02-10T11:07:14.248 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:14.246 7f3a80ff9700 1 -- 172.21.2.102:0/422214296 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3a78005560 con 0x7f3a840b8200 2021-02-10T11:07:14.248 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:14.246 7f3a8a564700 1 -- 172.21.2.102:0/422214296 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a840c29c0 con 0x7f3a840b8200 2021-02-10T11:07:14.248 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:14.247 7f3a80ff9700 1 -- 172.21.2.102:0/422214296 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f3a78005700 con 0x7f3a840b8200 2021-02-10T11:07:14.248 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:14.247 7f3a80ff9700 1 --2- 172.21.2.102:0/422214296 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3a640186c0 0x7f3a6401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:07:14.248 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:14.247 7f3a80ff9700 1 -- 172.21.2.102:0/422214296 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f3a78032450 con 0x7f3a840b8200 2021-02-10T11:07:14.248 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:14.247 7f3a83fff700 1 --2- 172.21.2.102:0/422214296 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3a640186c0 0x7f3a6401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:07:14.248 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:14.247 7f3a83fff700 1 --2- 172.21.2.102:0/422214296 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3a640186c0 0x7f3a6401abc0 crc :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:07:14.276 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:14.278 7f3a8a564700 1 -- 172.21.2.102:0/422214296 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a84105ae0 con 0x7f3a840b8200 2021-02-10T11:07:14.279 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:14.281 7f3a80ff9700 1 -- 172.21.2.102:0/422214296 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f3a780054f0 con 0x7f3a840b8200 2021-02-10T11:07:14.406 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:14.408 7f3a8a564700 1 -- 172.21.2.102:0/422214296 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344375", "target": ["mgr", ""]}) v1 -- 0x7f3a84105ae0 con 0x7f3a640186c0 2021-02-10T11:07:14.430 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:14.432 7f3a80ff9700 1 -- 172.21.2.102:0/422214296 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f3a740043a0 con 0x7f3a640186c0 2021-02-10T11:07:14.439 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:14.441 7f3a8a564700 1 -- 172.21.2.102:0/422214296 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3a640186c0 msgr2=0x7f3a6401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:07:14.439 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:14.441 7f3a8a564700 1 --2- 172.21.2.102:0/422214296 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3a640186c0 0x7f3a6401abc0 crc :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:07:14.439 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:14.441 7f3a8a564700 1 -- 172.21.2.102:0/422214296 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3a840b8200 msgr2=0x7f3a840b8600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:07:14.439 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:14.441 7f3a8a564700 1 --2- 172.21.2.102:0/422214296 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3a840b8200 0x7f3a840b8600 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 rx=0x7f3a78009630 tx=0x7f3a7800b980).stop 2021-02-10T11:07:14.439 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:14.441 7f3a8a564700 1 -- 172.21.2.102:0/422214296 shutdown_connections 2021-02-10T11:07:14.439 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:14.441 7f3a8a564700 1 --2- 172.21.2.102:0/422214296 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3a840c1d80 0x7f3a840c2180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:07:14.439 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:14.441 7f3a8a564700 1 --2- 172.21.2.102:0/422214296 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3a840b8200 0x7f3a840b8600 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:07:14.439 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:14.441 7f3a8a564700 1 --2- 172.21.2.102:0/422214296 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3a640186c0 0x7f3a6401abc0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:07:14.439 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:14.441 7f3a8a564700 1 --2- 172.21.2.102:0/422214296 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3a84190920 0x7f3a840b7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:07:14.439 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:14.441 7f3a8a564700 1 -- 172.21.2.102:0/422214296 >> 172.21.2.102:0/422214296 conn(0x7f3a84094e10 msgr2=0x7f3a840a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:07:14.440 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:14.441 7f3a8a564700 1 -- 172.21.2.102:0/422214296 shutdown_connections 2021-02-10T11:07:14.440 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:14.441 7f3a8a564700 1 -- 172.21.2.102:0/422214296 wait complete. 2021-02-10T11:07:14.447 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000344375 2021-02-10T11:07:14.877 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:14.863 7f8fe471c700 1 Processor -- start 2021-02-10T11:07:14.878 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:14.863 7f8fe471c700 1 -- start start 2021-02-10T11:07:14.878 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:14.863 7f8fe471c700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f8fdc1b0470 0x7f8fdc1b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:07:14.878 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:14.863 7f8fe471c700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f8fdc0ef610 0x7f8fdc0efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:07:14.878 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:14.863 7f8fe471c700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fdc0f0b50 con 0x7f8fdc0f0030 2021-02-10T11:07:14.878 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:14.863 7f8fe471c700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8fdc0e52e0 con 0x7f8fdc1b0470 2021-02-10T11:07:14.878 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:14.864 7f8fe471c700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8fdc0e5490 con 0x7f8fdc0ef610 2021-02-10T11:07:14.878 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:14.864 7f8fe2538700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f8fdc1b0470 0x7f8fdc1b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:07:14.878 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:14.864 7f8fe2538700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f8fdc1b0470 0x7f8fdc1b0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:51240/0 (socket says 172.21.2.102:51240) 2021-02-10T11:07:14.878 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:14.864 7f8fe2538700 1 -- 172.21.2.102:0/2980751680 learned_addr learned my addr 172.21.2.102:0/2980751680 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:07:14.879 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:14.864 7f8fe2538700 1 -- 172.21.2.102:0/2980751680 >> v2:172.21.2.113:3300/0 conn(0x7f8fdc0ef610 msgr2=0x7f8fdc0efa10 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:07:14.879 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:14.864 7f8fe2538700 1 --2- 172.21.2.102:0/2980751680 >> v2:172.21.2.113:3300/0 conn(0x7f8fdc0ef610 0x7f8fdc0efa10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:07:14.879 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:14.864 7f8fe2538700 1 -- 172.21.2.102:0/2980751680 >> v1:172.21.2.113:6789/0 conn(0x7f8fdc0f0030 legacy=0x7f8fdc0f10d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:07:14.879 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:14.864 7f8fe2538700 1 -- 172.21.2.102:0/2980751680 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fdc0f0430 con 0x7f8fdc1b0470 2021-02-10T11:07:14.879 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:14.864 7f8fe2538700 1 --2- 172.21.2.102:0/2980751680 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8fdc1b0470 0x7f8fdc1b0870 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 rx=0x7f8fd0006700 tx=0x7f8fd0009b80).ready entity=mon.0 client_cookie=6fc09d4ade2c938e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:07:14.879 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:14.864 7f8fe1536700 1 -- 172.21.2.102:0/2980751680 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8fd000c0c0 con 0x7f8fdc1b0470 2021-02-10T11:07:14.879 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:14.865 7f8fe1536700 1 -- 172.21.2.102:0/2980751680 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fd000c340 con 0x7f8fdc1b0470 2021-02-10T11:07:14.879 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:14.865 7f8fe1536700 1 -- 172.21.2.102:0/2980751680 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8fd000c820 con 0x7f8fdc1b0470 2021-02-10T11:07:14.879 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:14.865 7f8fe471c700 1 -- 172.21.2.102:0/2980751680 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8fdc1b0470 msgr2=0x7f8fdc1b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:07:14.879 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:14.865 7f8fe471c700 1 --2- 172.21.2.102:0/2980751680 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8fdc1b0470 0x7f8fdc1b0870 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 rx=0x7f8fd0006700 tx=0x7f8fd0009b80).stop 2021-02-10T11:07:14.879 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:14.865 7f8fe471c700 1 -- 172.21.2.102:0/2980751680 shutdown_connections 2021-02-10T11:07:14.879 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:14.865 7f8fe471c700 1 --2- 172.21.2.102:0/2980751680 >> v2:172.21.2.113:3300/0 conn(0x7f8fdc0ef610 0x7f8fdc0efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:07:14.880 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:14.865 7f8fe471c700 1 --2- 172.21.2.102:0/2980751680 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8fdc1b0470 0x7f8fdc1b0870 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:07:14.880 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:14.865 7f8fe471c700 1 -- 172.21.2.102:0/2980751680 >> 172.21.2.102:0/2980751680 conn(0x7f8fdc0ce3e0 msgr2=0x7f8fdc0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:07:14.880 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:14.865 7f8fe471c700 1 -- 172.21.2.102:0/2980751680 shutdown_connections 2021-02-10T11:07:14.880 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:14.865 7f8fe471c700 1 -- 172.21.2.102:0/2980751680 wait complete. 2021-02-10T11:07:14.882 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:14.880 7f8fe471c700 1 Processor -- start 2021-02-10T11:07:14.882 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:14.880 7f8fe471c700 1 -- start start 2021-02-10T11:07:14.882 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:14.881 7f8fe471c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8fdc094fb0 0x7f8fdc09e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:07:14.882 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:14.881 7f8fe471c700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8fdc0c2be0 0x7f8fdc0c2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:07:14.882 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:14.881 7f8fe471c700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8fdc0c3600 0x7f8fdc0a9c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:07:14.882 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:14.881 7f8fe471c700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fdc09ebf0 con 0x7f8fdc094fb0 2021-02-10T11:07:14.882 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:14.881 7f8fe471c700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fdc0aa2a0 con 0x7f8fdc0c3600 2021-02-10T11:07:14.882 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:14.881 7f8fe471c700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fdc0aa4c0 con 0x7f8fdc0c2be0 2021-02-10T11:07:14.882 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:14.881 7f8fe2538700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8fdc094fb0 0x7f8fdc09e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:07:14.882 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:14.881 7f8fe2538700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8fdc094fb0 0x7f8fdc09e5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:51248/0 (socket says 172.21.2.102:51248) 2021-02-10T11:07:14.882 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:14.881 7f8fe2538700 1 -- 172.21.2.102:0/511376064 learned_addr learned my addr 172.21.2.102:0/511376064 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:07:14.883 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:14.881 7f8fe2d39700 1 --2- 172.21.2.102:0/511376064 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8fdc0c3600 0x7f8fdc0a9c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:07:14.883 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:14.881 7f8fe1d37700 1 --2- 172.21.2.102:0/511376064 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8fdc0c2be0 0x7f8fdc0c2fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:07:14.883 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:14.881 7f8fe2538700 1 -- 172.21.2.102:0/511376064 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8fdc0c2be0 msgr2=0x7f8fdc0c2fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:07:14.883 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:14.881 7f8fe2538700 1 --2- 172.21.2.102:0/511376064 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8fdc0c2be0 0x7f8fdc0c2fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:07:14.883 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:14.881 7f8fe2538700 1 -- 172.21.2.102:0/511376064 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8fdc0c3600 msgr2=0x7f8fdc0a9c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:07:14.883 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:14.881 7f8fe2538700 1 --2- 172.21.2.102:0/511376064 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8fdc0c3600 0x7f8fdc0a9c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:07:14.883 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:14.881 7f8fe2538700 1 -- 172.21.2.102:0/511376064 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fdc0aa6e0 con 0x7f8fdc094fb0 2021-02-10T11:07:14.883 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:14.881 7f8fe1d37700 1 --2- 172.21.2.102:0/511376064 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8fdc0c2be0 0x7f8fdc0c2fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:07:14.883 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:14.882 7f8fe2538700 1 --2- 172.21.2.102:0/511376064 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8fdc094fb0 0x7f8fdc09e5d0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 rx=0x7f8fd000b5e0 tx=0x7f8fd000b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:07:14.883 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:14.882 7f8fceffd700 1 -- 172.21.2.102:0/511376064 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8fd000bd70 con 0x7f8fdc094fb0 2021-02-10T11:07:14.883 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:14.882 7f8fceffd700 1 -- 172.21.2.102:0/511376064 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fd0002030 con 0x7f8fdc094fb0 2021-02-10T11:07:14.883 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:14.882 7f8fceffd700 1 -- 172.21.2.102:0/511376064 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8fd0002550 con 0x7f8fdc094fb0 2021-02-10T11:07:14.884 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:14.882 7f8fe471c700 1 -- 172.21.2.102:0/511376064 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fdc0aa6e0 con 0x7f8fdc094fb0 2021-02-10T11:07:14.884 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:14.882 7f8fe471c700 1 -- 172.21.2.102:0/511376064 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fdc0aa6e0 con 0x7f8fdc094fb0 2021-02-10T11:07:14.884 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:14.883 7f8fceffd700 1 -- 172.21.2.102:0/511376064 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f8fd0000c90 con 0x7f8fdc094fb0 2021-02-10T11:07:14.884 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:14.883 7f8fceffd700 1 --2- 172.21.2.102:0/511376064 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8fc40186c0 0x7f8fc401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:07:14.884 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:14.883 7f8fceffd700 1 -- 172.21.2.102:0/511376064 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f8fd00325a0 con 0x7f8fdc094fb0 2021-02-10T11:07:14.884 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:14.883 7f8fe1d37700 1 --2- 172.21.2.102:0/511376064 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8fc40186c0 0x7f8fc401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:07:14.884 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:14.883 7f8fe1d37700 1 --2- 172.21.2.102:0/511376064 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8fc40186c0 0x7f8fc401abc0 crc :-1 s=READY pgs=195 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:07:14.911 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:14.913 7f8fe471c700 1 -- 172.21.2.102:0/511376064 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fdc0edd10 con 0x7f8fdc094fb0 2021-02-10T11:07:14.913 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:14.916 7f8fceffd700 1 -- 172.21.2.102:0/511376064 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f8fd0030d50 con 0x7f8fdc094fb0 2021-02-10T11:07:15.041 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:15.043 7f8fe471c700 1 -- 172.21.2.102:0/511376064 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344375", "target": ["mgr", ""]}) v1 -- 0x7f8fdc09ebf0 con 0x7f8fc40186c0 2021-02-10T11:07:15.044 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:15.046 7f8fceffd700 1 -- 172.21.2.102:0/511376064 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f8fd8006580 con 0x7f8fc40186c0 2021-02-10T11:07:15.044 INFO:teuthology.orchestra.run.gibba002.stdout:/volumes/_nogroup/subvolume_0000000000344375/eaa4f9ad-46da-40c6-8429-9e8bb47e33e3 2021-02-10T11:07:15.046 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:15.048 7f8fe471c700 1 -- 172.21.2.102:0/511376064 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8fc40186c0 msgr2=0x7f8fc401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:07:15.046 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:15.048 7f8fe471c700 1 --2- 172.21.2.102:0/511376064 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8fc40186c0 0x7f8fc401abc0 crc :-1 s=READY pgs=195 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:07:15.046 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:15.048 7f8fe471c700 1 -- 172.21.2.102:0/511376064 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8fdc094fb0 msgr2=0x7f8fdc09e5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:07:15.046 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:15.048 7f8fe471c700 1 --2- 172.21.2.102:0/511376064 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8fdc094fb0 0x7f8fdc09e5d0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 rx=0x7f8fd000b5e0 tx=0x7f8fd000b980).stop 2021-02-10T11:07:15.046 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:15.048 7f8fe471c700 1 -- 172.21.2.102:0/511376064 shutdown_connections 2021-02-10T11:07:15.046 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:15.048 7f8fe471c700 1 --2- 172.21.2.102:0/511376064 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8fc40186c0 0x7f8fc401abc0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:07:15.046 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:15.048 7f8fe471c700 1 --2- 172.21.2.102:0/511376064 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8fdc0c3600 0x7f8fdc0a9c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:07:15.046 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:15.048 7f8fe471c700 1 --2- 172.21.2.102:0/511376064 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8fdc0c2be0 0x7f8fdc0c2fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:07:15.046 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:15.048 7f8fe471c700 1 --2- 172.21.2.102:0/511376064 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8fdc094fb0 0x7f8fdc09e5d0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:07:15.046 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:15.048 7f8fe471c700 1 -- 172.21.2.102:0/511376064 >> 172.21.2.102:0/511376064 conn(0x7f8fdc09f630 msgr2=0x7f8fdc0a0130 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:07:15.046 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:15.048 7f8fe471c700 1 -- 172.21.2.102:0/511376064 shutdown_connections 2021-02-10T11:07:15.047 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:15.048 7f8fe471c700 1 -- 172.21.2.102:0/511376064 wait complete. 2021-02-10T11:07:15.053 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000344375 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000344375/eaa4f9ad-46da-40c6-8429-9e8bb47e33e3 2021-02-10T11:07:15.054 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:07:15.054 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:07:15.054 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:07:15.389 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:07:15.389 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:07:15.389 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344375/eaa4f9ad-46da-40c6-8429-9e8bb47e33e3/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:07:15.776 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:07:15.776 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:07:15.776 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0246701 s, 42.5 MB/s 2021-02-10T11:07:15.777 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:07:15.778 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:07:15.778 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:07:16.115 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:07:16.115 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:07:16.115 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344375/eaa4f9ad-46da-40c6-8429-9e8bb47e33e3/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:07:16.489 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:07:16.489 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:07:16.489 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0231543 s, 45.3 MB/s 2021-02-10T11:07:16.490 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:07:16.490 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:07:16.490 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:07:16.824 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:07:16.825 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:07:16.825 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344375/eaa4f9ad-46da-40c6-8429-9e8bb47e33e3/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:07:17.201 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:07:17.201 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:07:17.201 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.026003 s, 40.3 MB/s 2021-02-10T11:07:17.202 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:07:17.202 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:07:17.202 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:07:17.537 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:07:17.538 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:07:17.538 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344375/eaa4f9ad-46da-40c6-8429-9e8bb47e33e3/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:07:17.914 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:07:17.914 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:07:17.914 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0260173 s, 40.3 MB/s 2021-02-10T11:07:17.915 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:07:17.915 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:07:17.915 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:07:18.250 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:07:18.250 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:07:18.250 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344375/eaa4f9ad-46da-40c6-8429-9e8bb47e33e3/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:07:18.623 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:07:18.623 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:07:18.624 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0260081 s, 40.3 MB/s 2021-02-10T11:07:18.624 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:07:18.625 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:07:18.625 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:07:18.961 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:07:18.961 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:07:18.961 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344375/eaa4f9ad-46da-40c6-8429-9e8bb47e33e3/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:07:19.339 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:07:19.339 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:07:19.339 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0253996 s, 41.3 MB/s 2021-02-10T11:07:19.340 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:07:19.340 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:07:19.340 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:07:19.674 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:07:19.675 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:07:19.675 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344375/eaa4f9ad-46da-40c6-8429-9e8bb47e33e3/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:07:20.052 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:07:20.053 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:07:20.053 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0263262 s, 39.8 MB/s 2021-02-10T11:07:20.054 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:07:20.054 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:07:20.054 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:07:20.389 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:07:20.390 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:07:20.390 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344375/eaa4f9ad-46da-40c6-8429-9e8bb47e33e3/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:07:20.764 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:07:20.764 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:07:20.764 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0235205 s, 44.6 MB/s 2021-02-10T11:07:20.765 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:07:20.765 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:07:20.765 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:07:21.102 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:07:21.102 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:07:21.102 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344375/eaa4f9ad-46da-40c6-8429-9e8bb47e33e3/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:07:21.478 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:07:21.478 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:07:21.478 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0247778 s, 42.3 MB/s 2021-02-10T11:07:21.479 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:07:21.479 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:07:21.479 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:07:21.814 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:07:21.815 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:07:21.815 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344375/eaa4f9ad-46da-40c6-8429-9e8bb47e33e3/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:07:22.193 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:07:22.193 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:07:22.194 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0265605 s, 39.5 MB/s 2021-02-10T11:07:22.195 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000344376 2021-02-10T11:07:22.626 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:22.612 7f1b8acde700 1 Processor -- start 2021-02-10T11:07:22.627 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:22.612 7f1b8acde700 1 -- start start 2021-02-10T11:07:22.627 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:22.612 7f1b8acde700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f1b841b0870 0x7f1b840f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:07:22.627 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:22.612 7f1b8acde700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b841b0c70 con 0x7f1b840efe00 2021-02-10T11:07:22.627 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:22.612 7f1b8acde700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b840f0b50 con 0x7f1b841b0470 2021-02-10T11:07:22.627 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:22.612 7f1b8acde700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b840e5ea0 con 0x7f1b841b0870 2021-02-10T11:07:22.627 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:22.612 7f1b892fb700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f1b840efe00 0x7f1b840f0200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:43482/0 (socket says 172.21.2.102:43482) 2021-02-10T11:07:22.627 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:22.612 7f1b892fb700 1 -- 172.21.2.102:0/1900665241 learned_addr learned my addr 172.21.2.102:0/1900665241 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:07:22.627 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:22.613 7f1b83fff700 1 --2- 172.21.2.102:0/1900665241 >> v2:172.21.2.102:3300/0 conn(0x7f1b841b0870 0x7f1b840f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:07:22.627 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:22.613 7f1b837fe700 1 -- 172.21.2.102:0/1900665241 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1401838100 0 0) 0x7f1b7c000f90 con 0x7f1b840efe00 2021-02-10T11:07:22.627 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:22.613 7f1b837fe700 1 -- 172.21.2.102:0/1900665241 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b70002fe0 con 0x7f1b840efe00 2021-02-10T11:07:22.627 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:22.613 7f1b83fff700 1 -- 172.21.2.102:0/1900665241 >> v1:172.21.2.113:6789/0 conn(0x7f1b841b0470 legacy=0x7f1b840ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:07:22.627 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:22.613 7f1b83fff700 1 -- 172.21.2.102:0/1900665241 >> v1:172.21.2.102:6789/0 conn(0x7f1b840efe00 legacy=0x7f1b840f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:07:22.627 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:22.613 7f1b83fff700 1 -- 172.21.2.102:0/1900665241 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b840e52e0 con 0x7f1b841b0870 2021-02-10T11:07:22.628 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:22.613 7f1b83fff700 1 --2- 172.21.2.102:0/1900665241 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1b841b0870 0x7f1b840f20e0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 rx=0x7f1b74006650 tx=0x7f1b74009a80).ready entity=mon.0 client_cookie=b5fc1d4e9de26c0d server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:07:22.628 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:22.613 7f1b837fe700 1 -- 172.21.2.102:0/1900665241 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1b7400c0c0 con 0x7f1b841b0870 2021-02-10T11:07:22.628 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:22.613 7f1b837fe700 1 -- 172.21.2.102:0/1900665241 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b7400c340 con 0x7f1b841b0870 2021-02-10T11:07:22.628 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:22.613 7f1b837fe700 1 -- 172.21.2.102:0/1900665241 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1b7400c860 con 0x7f1b841b0870 2021-02-10T11:07:22.628 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:22.613 7f1b8acde700 1 -- 172.21.2.102:0/1900665241 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1b841b0870 msgr2=0x7f1b840f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:07:22.628 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:22.613 7f1b8acde700 1 --2- 172.21.2.102:0/1900665241 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1b841b0870 0x7f1b840f20e0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 rx=0x7f1b74006650 tx=0x7f1b74009a80).stop 2021-02-10T11:07:22.628 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:22.614 7f1b8acde700 1 -- 172.21.2.102:0/1900665241 shutdown_connections 2021-02-10T11:07:22.628 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:22.614 7f1b8acde700 1 --2- 172.21.2.102:0/1900665241 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1b841b0870 0x7f1b840f20e0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:07:22.628 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:22.614 7f1b8acde700 1 -- 172.21.2.102:0/1900665241 >> 172.21.2.102:0/1900665241 conn(0x7f1b840ce3e0 msgr2=0x7f1b840e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:07:22.628 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:22.614 7f1b8acde700 1 -- 172.21.2.102:0/1900665241 shutdown_connections 2021-02-10T11:07:22.628 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:22.614 7f1b8acde700 1 -- 172.21.2.102:0/1900665241 wait complete. 2021-02-10T11:07:22.628 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:22.629 7f1b8acde700 1 Processor -- start 2021-02-10T11:07:22.628 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:22.629 7f1b8acde700 1 -- start start 2021-02-10T11:07:22.630 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:22.629 7f1b8acde700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1b840ee230 0x7f1b840bfc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:07:22.630 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:22.629 7f1b8acde700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1b840ee630 0x7f1b840eea30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:07:22.630 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:22.629 7f1b8acde700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1b840c09e0 0x7f1b840c0de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:07:22.630 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:22.629 7f1b8acde700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b840c1f60 con 0x7f1b840ee630 2021-02-10T11:07:22.630 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:22.629 7f1b8acde700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b8418bdf0 con 0x7f1b840c09e0 2021-02-10T11:07:22.630 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:22.629 7f1b8acde700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b840c0460 con 0x7f1b840ee230 2021-02-10T11:07:22.631 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:22.629 7f1b83fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1b840ee630 0x7f1b840eea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:07:22.631 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:22.629 7f1b83fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1b840ee630 0x7f1b840eea30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:51262/0 (socket says 172.21.2.102:51262) 2021-02-10T11:07:22.631 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:22.629 7f1b83fff700 1 -- 172.21.2.102:0/546861569 learned_addr learned my addr 172.21.2.102:0/546861569 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:07:22.631 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:22.630 7f1b88afa700 1 --2- 172.21.2.102:0/546861569 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1b840ee230 0x7f1b840bfc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:07:22.631 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:22.630 7f1b892fb700 1 --2- 172.21.2.102:0/546861569 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1b840c09e0 0x7f1b840c0de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:07:22.631 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:22.630 7f1b83fff700 1 -- 172.21.2.102:0/546861569 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1b840ee230 msgr2=0x7f1b840bfc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:07:22.631 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:22.630 7f1b83fff700 1 --2- 172.21.2.102:0/546861569 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1b840ee230 0x7f1b840bfc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:07:22.631 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:22.630 7f1b83fff700 1 -- 172.21.2.102:0/546861569 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1b840c09e0 msgr2=0x7f1b840c0de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:07:22.631 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:22.630 7f1b83fff700 1 --2- 172.21.2.102:0/546861569 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1b840c09e0 0x7f1b840c0de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:07:22.631 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:22.630 7f1b83fff700 1 -- 172.21.2.102:0/546861569 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b840c1400 con 0x7f1b840ee630 2021-02-10T11:07:22.631 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:22.630 7f1b83fff700 1 --2- 172.21.2.102:0/546861569 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1b840ee630 0x7f1b840eea30 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 rx=0x7f1b74009630 tx=0x7f1b7400b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:07:22.631 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:22.630 7f1b817fa700 1 -- 172.21.2.102:0/546861569 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1b740093a0 con 0x7f1b840ee630 2021-02-10T11:07:22.631 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:22.630 7f1b817fa700 1 -- 172.21.2.102:0/546861569 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b7400bd70 con 0x7f1b840ee630 2021-02-10T11:07:22.632 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:22.630 7f1b817fa700 1 -- 172.21.2.102:0/546861569 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1b74005560 con 0x7f1b840ee630 2021-02-10T11:07:22.632 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:22.630 7f1b8acde700 1 -- 172.21.2.102:0/546861569 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b840c1400 con 0x7f1b840ee630 2021-02-10T11:07:22.632 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:22.630 7f1b8acde700 1 -- 172.21.2.102:0/546861569 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b840c1400 con 0x7f1b840ee630 2021-02-10T11:07:22.632 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:22.631 7f1b817fa700 1 -- 172.21.2.102:0/546861569 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1b74005770 con 0x7f1b840ee630 2021-02-10T11:07:22.632 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:22.631 7f1b817fa700 1 --2- 172.21.2.102:0/546861569 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1b680186d0 0x7f1b6801abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:07:22.632 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:22.631 7f1b817fa700 1 -- 172.21.2.102:0/546861569 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f1b74032830 con 0x7f1b840ee630 2021-02-10T11:07:22.632 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:22.631 7f1b88afa700 1 --2- 172.21.2.102:0/546861569 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1b680186d0 0x7f1b6801abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:07:22.632 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:22.631 7f1b88afa700 1 --2- 172.21.2.102:0/546861569 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1b680186d0 0x7f1b6801abd0 crc :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:07:22.660 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:22.662 7f1b8acde700 1 -- 172.21.2.102:0/546861569 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b840eda20 con 0x7f1b840ee630 2021-02-10T11:07:22.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:22.664 7f1b817fa700 1 -- 172.21.2.102:0/546861569 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f1b74005560 con 0x7f1b840ee630 2021-02-10T11:07:22.789 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:22.791 7f1b8acde700 1 -- 172.21.2.102:0/546861569 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344376", "target": ["mgr", ""]}) v1 -- 0x7f1b840eda20 con 0x7f1b680186d0 2021-02-10T11:07:22.813 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:22.815 7f1b817fa700 1 -- 172.21.2.102:0/546861569 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f1b6c006a90 con 0x7f1b680186d0 2021-02-10T11:07:22.821 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:22.824 7f1b8acde700 1 -- 172.21.2.102:0/546861569 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1b680186d0 msgr2=0x7f1b6801abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:07:22.821 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:22.824 7f1b8acde700 1 --2- 172.21.2.102:0/546861569 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1b680186d0 0x7f1b6801abd0 crc :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:07:22.822 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:22.824 7f1b8acde700 1 -- 172.21.2.102:0/546861569 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1b840ee630 msgr2=0x7f1b840eea30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:07:22.822 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:22.824 7f1b8acde700 1 --2- 172.21.2.102:0/546861569 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1b840ee630 0x7f1b840eea30 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 rx=0x7f1b74009630 tx=0x7f1b7400b980).stop 2021-02-10T11:07:22.822 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:22.824 7f1b8acde700 1 -- 172.21.2.102:0/546861569 shutdown_connections 2021-02-10T11:07:22.822 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:22.824 7f1b8acde700 1 --2- 172.21.2.102:0/546861569 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1b680186d0 0x7f1b6801abd0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:07:22.822 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:22.824 7f1b8acde700 1 --2- 172.21.2.102:0/546861569 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1b840c09e0 0x7f1b840c0de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:07:22.823 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:22.824 7f1b8acde700 1 --2- 172.21.2.102:0/546861569 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1b840ee630 0x7f1b840eea30 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:07:22.823 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:22.824 7f1b8acde700 1 --2- 172.21.2.102:0/546861569 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1b840ee230 0x7f1b840bfc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:07:22.823 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:22.824 7f1b8acde700 1 -- 172.21.2.102:0/546861569 >> 172.21.2.102:0/546861569 conn(0x7f1b8409dfd0 msgr2=0x7f1b8409abf0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:07:22.823 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:22.824 7f1b8acde700 1 -- 172.21.2.102:0/546861569 shutdown_connections 2021-02-10T11:07:22.823 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:22.824 7f1b8acde700 1 -- 172.21.2.102:0/546861569 wait complete. 2021-02-10T11:07:22.830 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000344376 2021-02-10T11:07:23.260 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:23.246 7ff589eac700 1 Processor -- start 2021-02-10T11:07:23.261 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:23.246 7ff589eac700 1 -- start start 2021-02-10T11:07:23.261 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:23.247 7ff589eac700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7ff5841b0470 0x7ff5841b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:07:23.261 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:23.247 7ff589eac700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7ff5840f20e0 0x7ff5840f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:07:23.261 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:23.247 7ff589eac700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5840f0b50 con 0x7ff5840ef610 2021-02-10T11:07:23.261 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:23.247 7ff589eac700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5840f0820 con 0x7ff5840f20e0 2021-02-10T11:07:23.261 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:23.247 7ff589eac700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5840f24e0 con 0x7ff5841b0470 2021-02-10T11:07:23.261 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:23.247 7ff583fff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7ff5840f20e0 0x7ff5840f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:07:23.261 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:23.247 7ff582ffd700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7ff5840ef610 0x7ff5840efa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:43494/0 (socket says 172.21.2.102:43494) 2021-02-10T11:07:23.261 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:23.247 7ff582ffd700 1 -- 172.21.2.102:0/4105877147 learned_addr learned my addr 172.21.2.102:0/4105877147 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:07:23.261 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:23.247 7ff583fff700 1 -- 172.21.2.102:0/4105877147 >> v2:172.21.2.113:3300/0 conn(0x7ff5841b0470 msgr2=0x7ff5841b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:07:23.261 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:23.247 7ff5837fe700 1 --2- 172.21.2.102:0/4105877147 >> v2:172.21.2.113:3300/0 conn(0x7ff5841b0470 0x7ff5841b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:07:23.261 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:23.247 7ff583fff700 1 --2- 172.21.2.102:0/4105877147 >> v2:172.21.2.113:3300/0 conn(0x7ff5841b0470 0x7ff5841b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:07:23.261 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:23.248 7ff583fff700 1 -- 172.21.2.102:0/4105877147 >> v1:172.21.2.102:6789/0 conn(0x7ff5840ef610 legacy=0x7ff5840efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:07:23.262 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:23.248 7ff583fff700 1 -- 172.21.2.102:0/4105877147 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5840e52e0 con 0x7ff5840f20e0 2021-02-10T11:07:23.262 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:23.248 7ff5837fe700 1 --2- 172.21.2.102:0/4105877147 >> v2:172.21.2.113:3300/0 conn(0x7ff5841b0470 0x7ff5841b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:07:23.262 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:23.248 7ff583fff700 1 --2- 172.21.2.102:0/4105877147 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff5840f20e0 0x7ff5840f0200 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 rx=0x7ff57c006ab0 tx=0x7ff57c009b40).ready entity=mon.0 client_cookie=ff9c1fb3475ddcb7 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:07:23.262 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:23.248 7ff5827fc700 1 -- 172.21.2.102:0/4105877147 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff57c0022e0 con 0x7ff5840f20e0 2021-02-10T11:07:23.262 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:23.248 7ff5827fc700 1 -- 172.21.2.102:0/4105877147 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff57c002560 con 0x7ff5840f20e0 2021-02-10T11:07:23.262 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:23.248 7ff5827fc700 1 -- 172.21.2.102:0/4105877147 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff57c0029f0 con 0x7ff5840f20e0 2021-02-10T11:07:23.262 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:23.248 7ff589eac700 1 -- 172.21.2.102:0/4105877147 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff5840f20e0 msgr2=0x7ff5840f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:07:23.262 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:23.248 7ff589eac700 1 --2- 172.21.2.102:0/4105877147 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff5840f20e0 0x7ff5840f0200 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 rx=0x7ff57c006ab0 tx=0x7ff57c009b40).stop 2021-02-10T11:07:23.262 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:23.248 7ff589eac700 1 -- 172.21.2.102:0/4105877147 shutdown_connections 2021-02-10T11:07:23.262 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:23.249 7ff589eac700 1 --2- 172.21.2.102:0/4105877147 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff5840f20e0 0x7ff5840f0200 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:07:23.262 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:23.249 7ff589eac700 1 --2- 172.21.2.102:0/4105877147 >> v2:172.21.2.113:3300/0 conn(0x7ff5841b0470 0x7ff5841b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:07:23.262 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:23.249 7ff589eac700 1 -- 172.21.2.102:0/4105877147 >> 172.21.2.102:0/4105877147 conn(0x7ff5840ce3e0 msgr2=0x7ff5840e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:07:23.262 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:23.249 7ff589eac700 1 -- 172.21.2.102:0/4105877147 shutdown_connections 2021-02-10T11:07:23.263 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:23.249 7ff589eac700 1 -- 172.21.2.102:0/4105877147 wait complete. 2021-02-10T11:07:23.264 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:23.263 7ff589eac700 1 Processor -- start 2021-02-10T11:07:23.264 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:23.263 7ff589eac700 1 -- start start 2021-02-10T11:07:23.264 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:23.264 7ff589eac700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff5841a42f0 0x7ff5841a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:07:23.264 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:23.264 7ff589eac700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff584196170 0x7ff584196570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:07:23.265 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:23.264 7ff589eac700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff58408f5b0 0x7ff58408f9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:07:23.265 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:23.264 7ff589eac700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff584195bf0 con 0x7ff584196170 2021-02-10T11:07:23.265 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:23.264 7ff589eac700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5840c4130 con 0x7ff5841a42f0 2021-02-10T11:07:23.265 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:23.264 7ff589eac700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5840a6e70 con 0x7ff58408f5b0 2021-02-10T11:07:23.265 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:23.264 7ff582ffd700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff584196170 0x7ff584196570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:07:23.265 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:23.264 7ff582ffd700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff584196170 0x7ff584196570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:51276/0 (socket says 172.21.2.102:51276) 2021-02-10T11:07:23.265 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:23.264 7ff582ffd700 1 -- 172.21.2.102:0/2760252731 learned_addr learned my addr 172.21.2.102:0/2760252731 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:07:23.265 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:23.264 7ff5837fe700 1 --2- 172.21.2.102:0/2760252731 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff5841a42f0 0x7ff5841a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:07:23.265 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:23.264 7ff583fff700 1 --2- 172.21.2.102:0/2760252731 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff58408f5b0 0x7ff58408f9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:07:23.265 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:23.264 7ff582ffd700 1 -- 172.21.2.102:0/2760252731 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff58408f5b0 msgr2=0x7ff58408f9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:07:23.265 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:23.264 7ff582ffd700 1 --2- 172.21.2.102:0/2760252731 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff58408f5b0 0x7ff58408f9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:07:23.265 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:23.264 7ff582ffd700 1 -- 172.21.2.102:0/2760252731 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff5841a42f0 msgr2=0x7ff5841a46f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:07:23.265 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:23.264 7ff582ffd700 1 --2- 172.21.2.102:0/2760252731 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff5841a42f0 0x7ff5841a46f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:07:23.266 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:23.264 7ff582ffd700 1 -- 172.21.2.102:0/2760252731 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5840a6a80 con 0x7ff584196170 2021-02-10T11:07:23.266 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:23.264 7ff5837fe700 1 --2- 172.21.2.102:0/2760252731 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff5841a42f0 0x7ff5841a46f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:07:23.266 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:23.265 7ff582ffd700 1 --2- 172.21.2.102:0/2760252731 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff584196170 0x7ff584196570 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 rx=0x7ff574001c60 tx=0x7ff57400a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:07:23.266 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:23.265 7ff57bfff700 1 -- 172.21.2.102:0/2760252731 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff57400acc0 con 0x7ff584196170 2021-02-10T11:07:23.266 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:23.265 7ff57bfff700 1 -- 172.21.2.102:0/2760252731 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff574004140 con 0x7ff584196170 2021-02-10T11:07:23.266 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:23.265 7ff57bfff700 1 -- 172.21.2.102:0/2760252731 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff574004660 con 0x7ff584196170 2021-02-10T11:07:23.266 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:23.265 7ff589eac700 1 -- 172.21.2.102:0/2760252731 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5840a6a80 con 0x7ff584196170 2021-02-10T11:07:23.266 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:23.265 7ff589eac700 1 -- 172.21.2.102:0/2760252731 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5840a6a80 con 0x7ff584196170 2021-02-10T11:07:23.266 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:23.266 7ff57bfff700 1 -- 172.21.2.102:0/2760252731 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ff574004320 con 0x7ff584196170 2021-02-10T11:07:23.266 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:23.266 7ff57bfff700 1 --2- 172.21.2.102:0/2760252731 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff5680186c0 0x7ff56801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:07:23.266 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:23.266 7ff57bfff700 1 -- 172.21.2.102:0/2760252731 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7ff574032570 con 0x7ff584196170 2021-02-10T11:07:23.266 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:23.266 7ff5837fe700 1 --2- 172.21.2.102:0/2760252731 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff5680186c0 0x7ff56801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:07:23.266 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:23.266 7ff5837fe700 1 --2- 172.21.2.102:0/2760252731 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff5680186c0 0x7ff56801abc0 crc :-1 s=READY pgs=197 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:07:23.295 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:23.297 7ff589eac700 1 -- 172.21.2.102:0/2760252731 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5840f0b50 con 0x7ff584196170 2021-02-10T11:07:23.297 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:23.300 7ff57bfff700 1 -- 172.21.2.102:0/2760252731 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ff574001090 con 0x7ff584196170 2021-02-10T11:07:23.427 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:23.429 7ff589eac700 1 -- 172.21.2.102:0/2760252731 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344376", "target": ["mgr", ""]}) v1 -- 0x7ff5840f0b50 con 0x7ff5680186c0 2021-02-10T11:07:23.429 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:23.432 7ff57bfff700 1 -- 172.21.2.102:0/2760252731 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7ff56c000cc0 con 0x7ff5680186c0 2021-02-10T11:07:23.430 INFO:teuthology.orchestra.run.gibba002.stdout:/volumes/_nogroup/subvolume_0000000000344376/18f80d18-1f87-406d-af42-d2ab1848fe80 2021-02-10T11:07:23.432 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:23.433 7ff589eac700 1 -- 172.21.2.102:0/2760252731 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff5680186c0 msgr2=0x7ff56801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:07:23.432 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:23.433 7ff589eac700 1 --2- 172.21.2.102:0/2760252731 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff5680186c0 0x7ff56801abc0 crc :-1 s=READY pgs=197 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:07:23.432 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:23.433 7ff589eac700 1 -- 172.21.2.102:0/2760252731 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff584196170 msgr2=0x7ff584196570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:07:23.432 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:23.433 7ff589eac700 1 --2- 172.21.2.102:0/2760252731 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff584196170 0x7ff584196570 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 rx=0x7ff574001c60 tx=0x7ff57400a8d0).stop 2021-02-10T11:07:23.432 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:23.433 7ff589eac700 1 -- 172.21.2.102:0/2760252731 shutdown_connections 2021-02-10T11:07:23.432 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:23.433 7ff589eac700 1 --2- 172.21.2.102:0/2760252731 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff58408f5b0 0x7ff58408f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:07:23.432 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:23.433 7ff589eac700 1 --2- 172.21.2.102:0/2760252731 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff584196170 0x7ff584196570 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:07:23.433 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:23.433 7ff589eac700 1 --2- 172.21.2.102:0/2760252731 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff5680186c0 0x7ff56801abc0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:07:23.433 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:23.433 7ff589eac700 1 --2- 172.21.2.102:0/2760252731 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff5841a42f0 0x7ff5841a46f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:07:23.433 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:23.433 7ff589eac700 1 -- 172.21.2.102:0/2760252731 >> 172.21.2.102:0/2760252731 conn(0x7ff5840bf4e0 msgr2=0x7ff5840be160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:07:23.433 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:23.434 7ff589eac700 1 -- 172.21.2.102:0/2760252731 shutdown_connections 2021-02-10T11:07:23.433 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:23.434 7ff589eac700 1 -- 172.21.2.102:0/2760252731 wait complete. 2021-02-10T11:07:23.439 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000344376 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000344376/18f80d18-1f87-406d-af42-d2ab1848fe80 2021-02-10T11:07:23.439 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:07:23.440 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:07:23.440 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:07:23.775 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:07:23.776 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:07:23.776 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344376/18f80d18-1f87-406d-af42-d2ab1848fe80/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:07:24.155 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:07:24.155 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:07:24.156 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0255808 s, 41.0 MB/s 2021-02-10T11:07:24.156 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:07:24.157 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:07:24.157 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:07:24.492 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:07:24.492 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:07:24.492 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344376/18f80d18-1f87-406d-af42-d2ab1848fe80/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:07:24.867 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:07:24.868 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:07:24.868 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.024561 s, 42.7 MB/s 2021-02-10T11:07:24.869 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:07:24.869 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:07:24.869 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:07:25.205 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:07:25.205 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:07:25.206 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344376/18f80d18-1f87-406d-af42-d2ab1848fe80/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:07:25.584 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:07:25.585 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:07:25.585 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0258829 s, 40.5 MB/s 2021-02-10T11:07:25.586 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:07:25.587 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:07:25.587 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:07:25.923 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:07:25.923 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:07:25.924 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344376/18f80d18-1f87-406d-af42-d2ab1848fe80/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:07:26.295 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:07:26.296 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:07:26.296 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.02249 s, 46.6 MB/s 2021-02-10T11:07:26.297 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:07:26.297 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:07:26.297 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:07:26.630 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:07:26.631 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:07:26.631 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344376/18f80d18-1f87-406d-af42-d2ab1848fe80/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:07:27.008 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:07:27.008 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:07:27.009 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0244379 s, 42.9 MB/s 2021-02-10T11:07:27.010 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:07:27.010 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:07:27.011 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:07:27.345 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:07:27.346 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:07:27.346 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344376/18f80d18-1f87-406d-af42-d2ab1848fe80/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:07:27.720 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:07:27.720 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:07:27.720 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0242489 s, 43.2 MB/s 2021-02-10T11:07:27.721 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:07:27.722 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:07:27.722 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:07:28.058 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:07:28.058 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:07:28.059 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344376/18f80d18-1f87-406d-af42-d2ab1848fe80/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:07:28.428 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:07:28.428 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:07:28.428 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0235584 s, 44.5 MB/s 2021-02-10T11:07:28.429 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:07:28.430 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:07:28.430 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:07:28.764 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:07:28.764 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:07:28.765 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344376/18f80d18-1f87-406d-af42-d2ab1848fe80/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:07:29.139 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:07:29.139 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:07:29.139 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0241644 s, 43.4 MB/s 2021-02-10T11:07:29.140 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:07:29.140 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:07:29.140 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:07:29.476 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:07:29.477 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:07:29.477 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344376/18f80d18-1f87-406d-af42-d2ab1848fe80/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:07:29.855 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:07:29.856 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:07:29.856 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0250987 s, 41.8 MB/s 2021-02-10T11:07:29.857 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:07:29.858 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:07:29.858 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:07:30.195 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:07:30.196 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:07:30.196 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344376/18f80d18-1f87-406d-af42-d2ab1848fe80/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:07:30.571 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:07:30.571 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:07:30.571 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0239802 s, 43.7 MB/s 2021-02-10T11:07:30.572 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000344377 2021-02-10T11:07:31.002 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:30.987 7f4642e08700 1 Processor -- start 2021-02-10T11:07:31.002 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:30.987 7f4642e08700 1 -- start start 2021-02-10T11:07:31.002 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:30.988 7f4642e08700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f463c1b0470 0x7f463c1b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:07:31.002 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:30.988 7f4642e08700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f463c0ef610 0x7f463c0efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:07:31.002 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:30.988 7f4642e08700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f463c0f0030 0x7f463c0f0430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:07:31.002 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:30.988 7f4642e08700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f463c0e5ea0 con 0x7f463c1b0470 2021-02-10T11:07:31.002 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:30.988 7f4642e08700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f463c0f0b50 con 0x7f463c0ef610 2021-02-10T11:07:31.002 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:30.988 7f4642e08700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f463c0e52e0 con 0x7f463c0f0030 2021-02-10T11:07:31.003 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:30.988 7f4640c24700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f463c1b0470 0x7f463c1b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:07:31.003 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:30.988 7f4640c24700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f463c1b0470 0x7f463c1b0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:51284/0 (socket says 172.21.2.102:51284) 2021-02-10T11:07:31.003 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:30.988 7f4640c24700 1 -- 172.21.2.102:0/1632533331 learned_addr learned my addr 172.21.2.102:0/1632533331 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:07:31.003 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:30.988 7f463bfff700 1 --2- 172.21.2.102:0/1632533331 >> v2:172.21.2.113:3300/0 conn(0x7f463c0ef610 0x7f463c0efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:07:31.003 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:30.988 7f4641425700 1 --2- 172.21.2.102:0/1632533331 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f463c0f0030 0x7f463c0f0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:07:31.003 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:30.989 7f4640c24700 1 -- 172.21.2.102:0/1632533331 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f463c0f0030 msgr2=0x7f463c0f0430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:07:31.003 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:30.989 7f4640c24700 1 --2- 172.21.2.102:0/1632533331 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f463c0f0030 0x7f463c0f0430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:07:31.003 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:30.989 7f4640c24700 1 -- 172.21.2.102:0/1632533331 >> v2:172.21.2.113:3300/0 conn(0x7f463c0ef610 msgr2=0x7f463c0efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:07:31.003 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:30.989 7f4640c24700 1 --2- 172.21.2.102:0/1632533331 >> v2:172.21.2.113:3300/0 conn(0x7f463c0ef610 0x7f463c0efa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:07:31.003 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:30.989 7f4640c24700 1 -- 172.21.2.102:0/1632533331 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f463c0f10d0 con 0x7f463c1b0470 2021-02-10T11:07:31.003 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:30.989 7f4640c24700 1 --2- 172.21.2.102:0/1632533331 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f463c1b0470 0x7f463c1b0870 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 rx=0x7f4624006700 tx=0x7f4624009b80).ready entity=mon.0 client_cookie=4f5b9166cbe114c6 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:07:31.003 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:30.989 7f463b7fe700 1 -- 172.21.2.102:0/1632533331 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f462400c0c0 con 0x7f463c1b0470 2021-02-10T11:07:31.003 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:30.989 7f463b7fe700 1 -- 172.21.2.102:0/1632533331 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f462400c440 con 0x7f463c1b0470 2021-02-10T11:07:31.004 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:30.989 7f463b7fe700 1 -- 172.21.2.102:0/1632533331 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f462400c920 con 0x7f463c1b0470 2021-02-10T11:07:31.004 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:30.989 7f4642e08700 1 -- 172.21.2.102:0/1632533331 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f463c1b0470 msgr2=0x7f463c1b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:07:31.004 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:30.989 7f4642e08700 1 --2- 172.21.2.102:0/1632533331 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f463c1b0470 0x7f463c1b0870 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 rx=0x7f4624006700 tx=0x7f4624009b80).stop 2021-02-10T11:07:31.004 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:30.990 7f4642e08700 1 -- 172.21.2.102:0/1632533331 shutdown_connections 2021-02-10T11:07:31.004 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:30.990 7f4642e08700 1 --2- 172.21.2.102:0/1632533331 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f463c0f0030 0x7f463c0f0430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:07:31.004 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:30.990 7f4642e08700 1 --2- 172.21.2.102:0/1632533331 >> v2:172.21.2.113:3300/0 conn(0x7f463c0ef610 0x7f463c0efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:07:31.004 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:30.990 7f4642e08700 1 --2- 172.21.2.102:0/1632533331 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f463c1b0470 0x7f463c1b0870 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:07:31.004 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:30.990 7f4642e08700 1 -- 172.21.2.102:0/1632533331 >> 172.21.2.102:0/1632533331 conn(0x7f463c0ce3e0 msgr2=0x7f463c0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:07:31.004 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:30.990 7f4642e08700 1 -- 172.21.2.102:0/1632533331 shutdown_connections 2021-02-10T11:07:31.004 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:30.990 7f4642e08700 1 -- 172.21.2.102:0/1632533331 wait complete. 2021-02-10T11:07:31.006 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:31.004 7f4642e08700 1 Processor -- start 2021-02-10T11:07:31.006 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:31.004 7f4642e08700 1 -- start start 2021-02-10T11:07:31.006 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:31.005 7f4642e08700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f463c0924b0 0x7f463c1a3590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:07:31.006 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:31.005 7f4642e08700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f463c1a3bb0 0x7f463c0bf920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:07:31.006 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:31.005 7f4642e08700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f463c0bff40 0x7f463c0c0340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:07:31.006 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:31.005 7f4642e08700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f463c09a0a0 con 0x7f463c0924b0 2021-02-10T11:07:31.006 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:31.005 7f4642e08700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f463c0c0960 con 0x7f463c0bff40 2021-02-10T11:07:31.007 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:31.005 7f4642e08700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f463c098420 con 0x7f463c1a3bb0 2021-02-10T11:07:31.007 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:31.005 7f4640c24700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f463c0924b0 0x7f463c1a3590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:07:31.007 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:31.005 7f4640c24700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f463c0924b0 0x7f463c1a3590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:51290/0 (socket says 172.21.2.102:51290) 2021-02-10T11:07:31.007 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:31.005 7f4640c24700 1 -- 172.21.2.102:0/639026421 learned_addr learned my addr 172.21.2.102:0/639026421 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:07:31.007 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:31.005 7f463bfff700 1 --2- 172.21.2.102:0/639026421 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f463c1a3bb0 0x7f463c0bf920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:07:31.007 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:31.005 7f4641425700 1 --2- 172.21.2.102:0/639026421 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f463c0bff40 0x7f463c0c0340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:07:31.007 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:31.006 7f4640c24700 1 -- 172.21.2.102:0/639026421 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f463c1a3bb0 msgr2=0x7f463c0bf920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:07:31.007 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:31.006 7f4640c24700 1 --2- 172.21.2.102:0/639026421 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f463c1a3bb0 0x7f463c0bf920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:07:31.007 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:31.006 7f4640c24700 1 -- 172.21.2.102:0/639026421 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f463c0bff40 msgr2=0x7f463c0c0340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:07:31.007 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:31.006 7f4640c24700 1 --2- 172.21.2.102:0/639026421 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f463c0bff40 0x7f463c0c0340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:07:31.007 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:31.006 7f4640c24700 1 -- 172.21.2.102:0/639026421 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f463c098640 con 0x7f463c0924b0 2021-02-10T11:07:31.007 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:31.006 7f463bfff700 1 --2- 172.21.2.102:0/639026421 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f463c1a3bb0 0x7f463c0bf920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:07:31.007 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:31.006 7f4641425700 1 --2- 172.21.2.102:0/639026421 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f463c0bff40 0x7f463c0c0340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:07:31.008 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:31.006 7f4640c24700 1 --2- 172.21.2.102:0/639026421 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f463c0924b0 0x7f463c1a3590 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 rx=0x7f462400b5e0 tx=0x7f462400b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:07:31.008 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:31.006 7f46397fa700 1 -- 172.21.2.102:0/639026421 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f462400bd70 con 0x7f463c0924b0 2021-02-10T11:07:31.008 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:31.006 7f46397fa700 1 -- 172.21.2.102:0/639026421 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4624002030 con 0x7f463c0924b0 2021-02-10T11:07:31.008 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:31.006 7f46397fa700 1 -- 172.21.2.102:0/639026421 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4624002550 con 0x7f463c0924b0 2021-02-10T11:07:31.008 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:31.006 7f4642e08700 1 -- 172.21.2.102:0/639026421 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f463c098640 con 0x7f463c0924b0 2021-02-10T11:07:31.008 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:31.006 7f4642e08700 1 -- 172.21.2.102:0/639026421 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f463c098bd0 con 0x7f463c0924b0 2021-02-10T11:07:31.008 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:31.007 7f46397fa700 1 -- 172.21.2.102:0/639026421 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4624000c90 con 0x7f463c0924b0 2021-02-10T11:07:31.008 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:31.007 7f46397fa700 1 --2- 172.21.2.102:0/639026421 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f46200186c0 0x7f462001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:07:31.008 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:31.007 7f46397fa700 1 -- 172.21.2.102:0/639026421 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f46240325c0 con 0x7f463c0924b0 2021-02-10T11:07:31.008 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:31.007 7f463bfff700 1 --2- 172.21.2.102:0/639026421 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f46200186c0 0x7f462001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:07:31.008 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:31.007 7f463bfff700 1 --2- 172.21.2.102:0/639026421 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f46200186c0 0x7f462001abc0 crc :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:07:31.035 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:31.038 7f4642e08700 1 -- 172.21.2.102:0/639026421 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f463c105b40 con 0x7f463c0924b0 2021-02-10T11:07:31.038 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:31.040 7f46397fa700 1 -- 172.21.2.102:0/639026421 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f46240009b0 con 0x7f463c0924b0 2021-02-10T11:07:31.163 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:31.166 7f4642e08700 1 -- 172.21.2.102:0/639026421 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344377", "target": ["mgr", ""]}) v1 -- 0x7f463c098bd0 con 0x7f46200186c0 2021-02-10T11:07:31.190 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:31.193 7f46397fa700 1 -- 172.21.2.102:0/639026421 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f462c006b30 con 0x7f46200186c0 2021-02-10T11:07:31.197 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:31.199 7f4642e08700 1 -- 172.21.2.102:0/639026421 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f46200186c0 msgr2=0x7f462001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:07:31.197 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:31.199 7f4642e08700 1 --2- 172.21.2.102:0/639026421 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f46200186c0 0x7f462001abc0 crc :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:07:31.198 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:31.199 7f4642e08700 1 -- 172.21.2.102:0/639026421 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f463c0924b0 msgr2=0x7f463c1a3590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:07:31.198 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:31.199 7f4642e08700 1 --2- 172.21.2.102:0/639026421 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f463c0924b0 0x7f463c1a3590 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 rx=0x7f462400b5e0 tx=0x7f462400b980).stop 2021-02-10T11:07:31.198 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:31.199 7f4642e08700 1 -- 172.21.2.102:0/639026421 shutdown_connections 2021-02-10T11:07:31.198 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:31.199 7f4642e08700 1 --2- 172.21.2.102:0/639026421 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f46200186c0 0x7f462001abc0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:07:31.198 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:31.199 7f4642e08700 1 --2- 172.21.2.102:0/639026421 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f463c0bff40 0x7f463c0c0340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:07:31.199 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:31.199 7f4642e08700 1 --2- 172.21.2.102:0/639026421 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f463c1a3bb0 0x7f463c0bf920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:07:31.199 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:31.199 7f4642e08700 1 --2- 172.21.2.102:0/639026421 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f463c0924b0 0x7f463c1a3590 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:07:31.199 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:31.199 7f4642e08700 1 -- 172.21.2.102:0/639026421 >> 172.21.2.102:0/639026421 conn(0x7f463c0b50c0 msgr2=0x7f463c092d90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:07:31.199 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:31.199 7f4642e08700 1 -- 172.21.2.102:0/639026421 shutdown_connections 2021-02-10T11:07:31.199 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:31.199 7f4642e08700 1 -- 172.21.2.102:0/639026421 wait complete. 2021-02-10T11:07:31.206 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000344377 2021-02-10T11:07:31.637 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:31.621 7f0bf13e8700 1 Processor -- start 2021-02-10T11:07:31.637 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:31.621 7f0bf13e8700 1 -- start start 2021-02-10T11:07:31.637 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:31.622 7f0bf13e8700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f0bec1b0870 0x7f0bec0f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:07:31.637 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:31.622 7f0bf13e8700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0bec0efe00 0x7f0bec0f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:07:31.637 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:31.622 7f0bf13e8700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bec1b0c70 con 0x7f0bec1b0470 2021-02-10T11:07:31.638 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:31.622 7f0bf13e8700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0bec0f0b50 con 0x7f0bec1b0870 2021-02-10T11:07:31.638 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:31.622 7f0bf13e8700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bec0f0820 con 0x7f0bec0efe00 2021-02-10T11:07:31.638 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:31.622 7f0beb07d700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f0bec1b0470 0x7f0bec0ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:43520/0 (socket says 172.21.2.102:43520) 2021-02-10T11:07:31.638 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:31.622 7f0beb07d700 1 -- 172.21.2.102:0/4185719230 learned_addr learned my addr 172.21.2.102:0/4185719230 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:07:31.638 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:31.623 7f0bea87c700 1 --2- 172.21.2.102:0/4185719230 >> v2:172.21.2.113:3300/0 conn(0x7f0bec1b0870 0x7f0bec0f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:07:31.638 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:31.623 7f0beb87e700 1 --2- 172.21.2.102:0/4185719230 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0bec0efe00 0x7f0bec0f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:07:31.638 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:31.623 7f0bea07b700 1 -- 172.21.2.102:0/4185719230 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1609760742 0 0) 0x7f0bd4001180 con 0x7f0bec1b0470 2021-02-10T11:07:31.639 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:31.623 7f0bea07b700 1 -- 172.21.2.102:0/4185719230 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0bd8002fe0 con 0x7f0bec1b0470 2021-02-10T11:07:31.639 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:31.623 7f0bea87c700 1 -- 172.21.2.102:0/4185719230 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0bec0efe00 msgr2=0x7f0bec0f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:07:31.639 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:31.623 7f0bea87c700 1 --2- 172.21.2.102:0/4185719230 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0bec0efe00 0x7f0bec0f0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:07:31.639 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:31.623 7f0bea87c700 1 -- 172.21.2.102:0/4185719230 >> v1:172.21.2.102:6789/0 conn(0x7f0bec1b0470 legacy=0x7f0bec0ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:07:31.639 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:31.623 7f0bea87c700 1 -- 172.21.2.102:0/4185719230 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bec0e52e0 con 0x7f0bec1b0870 2021-02-10T11:07:31.639 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:31.623 7f0bea87c700 1 --2- 172.21.2.102:0/4185719230 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0bec1b0870 0x7f0bec0f20e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f0bdc006650 tx=0x7f0bdc009a80).ready entity=mon.1 client_cookie=ce6d205a031ccc3f server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:07:31.639 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:31.624 7f0bea07b700 1 -- 172.21.2.102:0/4185719230 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0bdc00c0c0 con 0x7f0bec1b0870 2021-02-10T11:07:31.640 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:31.624 7f0bea07b700 1 -- 172.21.2.102:0/4185719230 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bdc00c340 con 0x7f0bec1b0870 2021-02-10T11:07:31.640 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:31.624 7f0bea07b700 1 -- 172.21.2.102:0/4185719230 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0bdc00c860 con 0x7f0bec1b0870 2021-02-10T11:07:31.640 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:31.624 7f0bf13e8700 1 -- 172.21.2.102:0/4185719230 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0bec1b0870 msgr2=0x7f0bec0f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:07:31.640 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:31.624 7f0bf13e8700 1 --2- 172.21.2.102:0/4185719230 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0bec1b0870 0x7f0bec0f20e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f0bdc006650 tx=0x7f0bdc009a80).stop 2021-02-10T11:07:31.640 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:31.624 7f0bf13e8700 1 -- 172.21.2.102:0/4185719230 shutdown_connections 2021-02-10T11:07:31.640 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:31.624 7f0bf13e8700 1 --2- 172.21.2.102:0/4185719230 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0bec1b0870 0x7f0bec0f20e0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:07:31.641 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:31.624 7f0bf13e8700 1 --2- 172.21.2.102:0/4185719230 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0bec0efe00 0x7f0bec0f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:07:31.641 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:31.624 7f0bf13e8700 1 -- 172.21.2.102:0/4185719230 >> 172.21.2.102:0/4185719230 conn(0x7f0bec0ce3e0 msgr2=0x7f0bec0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:07:31.641 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:31.624 7f0bf13e8700 1 -- 172.21.2.102:0/4185719230 shutdown_connections 2021-02-10T11:07:31.641 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:31.624 7f0bf13e8700 1 -- 172.21.2.102:0/4185719230 wait complete. 2021-02-10T11:07:31.641 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:31.639 7f0bf13e8700 1 Processor -- start 2021-02-10T11:07:31.641 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:31.639 7f0bf13e8700 1 -- start start 2021-02-10T11:07:31.643 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:31.639 7f0bf13e8700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0bec0a0420 0x7f0bec0a0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:07:31.643 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:31.639 7f0bf13e8700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0bec189d00 0x7f0bec18a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:07:31.643 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:31.639 7f0bf13e8700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0bec18a720 0x7f0bec0b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:07:31.643 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:31.639 7f0bf13e8700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bec189800 con 0x7f0bec0a0420 2021-02-10T11:07:31.643 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:31.639 7f0bf13e8700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bec18ab20 con 0x7f0bec189d00 2021-02-10T11:07:31.643 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:31.639 7f0bf13e8700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bec0a1140 con 0x7f0bec18a720 2021-02-10T11:07:31.643 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:31.640 7f0beb07d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0bec0a0420 0x7f0bec0a0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:07:31.643 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:31.640 7f0beb07d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0bec0a0420 0x7f0bec0a0820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:51302/0 (socket says 172.21.2.102:51302) 2021-02-10T11:07:31.643 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:31.640 7f0beb07d700 1 -- 172.21.2.102:0/2191982691 learned_addr learned my addr 172.21.2.102:0/2191982691 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:07:31.643 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:31.640 7f0beb07d700 1 -- 172.21.2.102:0/2191982691 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0bec18a720 msgr2=0x7f0bec0b7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:07:31.644 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:31.640 7f0beb07d700 1 --2- 172.21.2.102:0/2191982691 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0bec18a720 0x7f0bec0b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:07:31.644 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:31.640 7f0beb07d700 1 -- 172.21.2.102:0/2191982691 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0bec189d00 msgr2=0x7f0bec18a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:07:31.644 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:31.640 7f0beb07d700 1 --2- 172.21.2.102:0/2191982691 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0bec189d00 0x7f0bec18a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:07:31.644 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:31.640 7f0beb07d700 1 -- 172.21.2.102:0/2191982691 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bec0b8200 con 0x7f0bec0a0420 2021-02-10T11:07:31.644 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:31.640 7f0beb07d700 1 --2- 172.21.2.102:0/2191982691 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0bec0a0420 0x7f0bec0a0820 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 rx=0x7f0bd4001e90 tx=0x7f0bd400a940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:07:31.644 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:31.640 7f0be3fff700 1 -- 172.21.2.102:0/2191982691 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0bd4001490 con 0x7f0bec0a0420 2021-02-10T11:07:31.644 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:31.640 7f0be3fff700 1 -- 172.21.2.102:0/2191982691 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bd400ad30 con 0x7f0bec0a0420 2021-02-10T11:07:31.644 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:31.640 7f0be3fff700 1 -- 172.21.2.102:0/2191982691 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0bd40043a0 con 0x7f0bec0a0420 2021-02-10T11:07:31.644 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:31.640 7f0bf13e8700 1 -- 172.21.2.102:0/2191982691 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bec0b8200 con 0x7f0bec0a0420 2021-02-10T11:07:31.644 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:31.641 7f0bf13e8700 1 -- 172.21.2.102:0/2191982691 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bec0b8200 con 0x7f0bec0a0420 2021-02-10T11:07:31.644 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:31.641 7f0be3fff700 1 -- 172.21.2.102:0/2191982691 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f0bd4007fa0 con 0x7f0bec0a0420 2021-02-10T11:07:31.644 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:31.641 7f0be3fff700 1 --2- 172.21.2.102:0/2191982691 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0bd0018730 0x7f0bd001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:07:31.644 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:31.641 7f0be3fff700 1 -- 172.21.2.102:0/2191982691 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f0bd4032610 con 0x7f0bec0a0420 2021-02-10T11:07:31.644 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:31.641 7f0bea87c700 1 --2- 172.21.2.102:0/2191982691 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0bd0018730 0x7f0bd001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:07:31.645 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:31.642 7f0bea87c700 1 --2- 172.21.2.102:0/2191982691 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0bd0018730 0x7f0bd001ac10 crc :-1 s=READY pgs=199 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:07:31.670 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:31.672 7f0bf13e8700 1 -- 172.21.2.102:0/2191982691 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bec105ae0 con 0x7f0bec0a0420 2021-02-10T11:07:31.672 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:31.674 7f0be3fff700 1 -- 172.21.2.102:0/2191982691 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f0bd4000990 con 0x7f0bec0a0420 2021-02-10T11:07:31.800 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:31.802 7f0bf13e8700 1 -- 172.21.2.102:0/2191982691 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344377", "target": ["mgr", ""]}) v1 -- 0x7f0bec18ab20 con 0x7f0bd0018730 2021-02-10T11:07:31.804 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:31.806 7f0be3fff700 1 -- 172.21.2.102:0/2191982691 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f0bdc000c90 con 0x7f0bd0018730 2021-02-10T11:07:31.807 INFO:teuthology.orchestra.run.gibba002.stdout:/volumes/_nogroup/subvolume_0000000000344377/1a6fd3ac-f605-4bd8-bef5-3266d766d27c 2021-02-10T11:07:31.809 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:31.811 7f0bf13e8700 1 -- 172.21.2.102:0/2191982691 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0bd0018730 msgr2=0x7f0bd001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:07:31.809 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:31.811 7f0bf13e8700 1 --2- 172.21.2.102:0/2191982691 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0bd0018730 0x7f0bd001ac10 crc :-1 s=READY pgs=199 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:07:31.809 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:31.811 7f0bf13e8700 1 -- 172.21.2.102:0/2191982691 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0bec0a0420 msgr2=0x7f0bec0a0820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:07:31.809 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:31.811 7f0bf13e8700 1 --2- 172.21.2.102:0/2191982691 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0bec0a0420 0x7f0bec0a0820 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 rx=0x7f0bd4001e90 tx=0x7f0bd400a940).stop 2021-02-10T11:07:31.809 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:31.811 7f0bf13e8700 1 -- 172.21.2.102:0/2191982691 shutdown_connections 2021-02-10T11:07:31.810 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:31.811 7f0bf13e8700 1 --2- 172.21.2.102:0/2191982691 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0bec18a720 0x7f0bec0b7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:07:31.810 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:31.811 7f0bf13e8700 1 --2- 172.21.2.102:0/2191982691 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0bd0018730 0x7f0bd001ac10 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:07:31.810 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:31.811 7f0bf13e8700 1 --2- 172.21.2.102:0/2191982691 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0bec189d00 0x7f0bec18a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:07:31.810 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:31.811 7f0bf13e8700 1 --2- 172.21.2.102:0/2191982691 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0bec0a0420 0x7f0bec0a0820 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:07:31.810 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:31.811 7f0bf13e8700 1 -- 172.21.2.102:0/2191982691 >> 172.21.2.102:0/2191982691 conn(0x7f0bec094e10 msgr2=0x7f0bec0a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:07:31.811 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:31.811 7f0bf13e8700 1 -- 172.21.2.102:0/2191982691 shutdown_connections 2021-02-10T11:07:31.811 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:31.811 7f0bf13e8700 1 -- 172.21.2.102:0/2191982691 wait complete. 2021-02-10T11:07:31.817 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000344377 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000344377/1a6fd3ac-f605-4bd8-bef5-3266d766d27c 2021-02-10T11:07:31.818 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:07:31.818 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:07:31.818 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:07:32.155 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:07:32.229 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:07:32.230 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344377/1a6fd3ac-f605-4bd8-bef5-3266d766d27c/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:07:32.623 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:07:32.623 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:07:32.623 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0233306 s, 44.9 MB/s 2021-02-10T11:07:32.624 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:07:32.624 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:07:32.624 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:07:32.970 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:07:32.970 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:07:32.970 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344377/1a6fd3ac-f605-4bd8-bef5-3266d766d27c/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:07:33.365 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:07:33.366 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:07:33.367 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0266558 s, 39.3 MB/s 2021-02-10T11:07:33.370 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:07:33.370 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:07:33.370 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:07:33.705 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:07:33.706 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:07:33.706 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344377/1a6fd3ac-f605-4bd8-bef5-3266d766d27c/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:07:34.084 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:07:34.084 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:07:34.084 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.02429 s, 43.2 MB/s 2021-02-10T11:07:34.085 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:07:34.085 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:07:34.085 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:07:34.420 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:07:34.420 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:07:34.421 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344377/1a6fd3ac-f605-4bd8-bef5-3266d766d27c/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:07:34.796 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:07:34.796 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:07:34.796 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0234485 s, 44.7 MB/s 2021-02-10T11:07:34.797 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:07:34.797 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:07:34.797 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:07:35.132 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:07:35.132 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:07:35.132 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344377/1a6fd3ac-f605-4bd8-bef5-3266d766d27c/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:07:35.509 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:07:35.509 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:07:35.509 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0255185 s, 41.1 MB/s 2021-02-10T11:07:35.510 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:07:35.510 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:07:35.510 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:07:35.846 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:07:35.846 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:07:35.847 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344377/1a6fd3ac-f605-4bd8-bef5-3266d766d27c/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:07:36.223 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:07:36.223 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:07:36.223 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0243013 s, 43.1 MB/s 2021-02-10T11:07:36.224 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:07:36.224 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:07:36.224 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:07:36.559 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:07:36.560 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:07:36.560 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344377/1a6fd3ac-f605-4bd8-bef5-3266d766d27c/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:07:36.935 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:07:36.935 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:07:36.935 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0253938 s, 41.3 MB/s 2021-02-10T11:07:36.936 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:07:36.936 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:07:36.937 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:07:37.272 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:07:37.273 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:07:37.273 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344377/1a6fd3ac-f605-4bd8-bef5-3266d766d27c/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:07:37.646 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:07:37.646 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:07:37.646 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0227742 s, 46.0 MB/s 2021-02-10T11:07:37.647 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:07:37.647 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:07:37.647 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:07:37.981 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:07:37.982 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:07:37.982 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344377/1a6fd3ac-f605-4bd8-bef5-3266d766d27c/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:07:38.354 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:07:38.355 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:07:38.355 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0244827 s, 42.8 MB/s 2021-02-10T11:07:38.356 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:07:38.356 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:07:38.356 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:07:38.690 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:07:38.691 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:07:38.691 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344377/1a6fd3ac-f605-4bd8-bef5-3266d766d27c/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:07:39.066 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:07:39.066 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:07:39.066 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0244586 s, 42.9 MB/s 2021-02-10T11:07:39.067 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000344378 2021-02-10T11:07:39.499 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:39.484 7fddfa0dd700 1 Processor -- start 2021-02-10T11:07:39.499 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:39.484 7fddfa0dd700 1 -- start start 2021-02-10T11:07:39.499 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:39.484 7fddfa0dd700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fddf40f05f0 0x7fddf40f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:07:39.499 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:39.484 7fddfa0dd700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddf41b0c70 con 0x7fddf41b0470 2021-02-10T11:07:39.499 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:39.484 7fddfa0dd700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddf40f0b50 con 0x7fddf41b0870 2021-02-10T11:07:39.500 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:39.484 7fddfa0dd700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fddf40e5ea0 con 0x7fddf40f05f0 2021-02-10T11:07:39.500 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:39.484 7fddf37fe700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fddf41b0470 0x7fddf40ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:43534/0 (socket says 172.21.2.102:43534) 2021-02-10T11:07:39.500 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:39.484 7fddf37fe700 1 -- 172.21.2.102:0/732770243 learned_addr learned my addr 172.21.2.102:0/732770243 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:07:39.500 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:39.485 7fddf27fc700 1 -- 172.21.2.102:0/732770243 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1308208285 0 0) 0x7fdddc0010b0 con 0x7fddf41b0470 2021-02-10T11:07:39.500 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:39.485 7fddf27fc700 1 -- 172.21.2.102:0/732770243 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdde0002fe0 con 0x7fddf41b0470 2021-02-10T11:07:39.500 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:39.485 7fddf3fff700 1 --2- 172.21.2.102:0/732770243 >> v2:172.21.2.113:3300/0 conn(0x7fddf40f05f0 0x7fddf40f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:07:39.500 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:39.485 7fddf27fc700 1 -- 172.21.2.102:0/732770243 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1794622953 0 0) 0x7fdddc0010b0 con 0x7fddf41b0470 2021-02-10T11:07:39.500 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:39.485 7fddf27fc700 1 -- 172.21.2.102:0/732770243 >> v2:172.21.2.113:3300/0 conn(0x7fddf40f05f0 msgr2=0x7fddf40f20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:07:39.500 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:39.485 7fddf27fc700 1 --2- 172.21.2.102:0/732770243 >> v2:172.21.2.113:3300/0 conn(0x7fddf40f05f0 0x7fddf40f20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:07:39.500 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:39.485 7fddf27fc700 1 -- 172.21.2.102:0/732770243 >> v1:172.21.2.113:6789/0 conn(0x7fddf41b0870 legacy=0x7fddf40efe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:07:39.500 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:39.485 7fddf27fc700 1 -- 172.21.2.102:0/732770243 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddf40e52e0 con 0x7fddf41b0470 2021-02-10T11:07:39.500 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:39.485 7fddf27fc700 1 -- 172.21.2.102:0/732770243 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7fdddc0015b0 con 0x7fddf41b0470 2021-02-10T11:07:39.500 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:39.485 7fddf27fc700 1 -- 172.21.2.102:0/732770243 >> v1:172.21.2.102:6789/0 conn(0x7fddf41b0470 legacy=0x7fddf40ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:07:39.501 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:39.485 7fddf27fc700 1 --2- 172.21.2.102:0/732770243 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fdde0003b10 0x7fdde0006010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:07:39.501 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:39.485 7fddf27fc700 1 --2- 172.21.2.102:0/732770243 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdde00067a0 0x7fdde0008cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:07:39.501 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:39.485 7fddf27fc700 1 --2- 172.21.2.102:0/732770243 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdde00095a0 0x7fdde000baa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:07:39.501 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:39.485 7fddf27fc700 1 -- 172.21.2.102:0/732770243 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdde000c2f0 con 0x7fdde00095a0 2021-02-10T11:07:39.501 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:39.485 7fddf27fc700 1 -- 172.21.2.102:0/732770243 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdde000c510 con 0x7fdde0003b10 2021-02-10T11:07:39.501 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:39.485 7fddf27fc700 1 -- 172.21.2.102:0/732770243 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdde000c730 con 0x7fdde00067a0 2021-02-10T11:07:39.501 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:39.485 7fddf3fff700 1 --2- 172.21.2.102:0/732770243 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdde00095a0 0x7fdde000baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:07:39.501 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:39.486 7fddf37fe700 1 --2- 172.21.2.102:0/732770243 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fdde0003b10 0x7fdde0006010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:07:39.501 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:39.486 7fddf2ffd700 1 --2- 172.21.2.102:0/732770243 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdde00067a0 0x7fdde0008cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:07:39.501 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:39.486 7fddf3fff700 1 -- 172.21.2.102:0/732770243 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdde00067a0 msgr2=0x7fdde0008cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:07:39.501 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:39.486 7fddf3fff700 1 --2- 172.21.2.102:0/732770243 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdde00067a0 0x7fdde0008cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:07:39.501 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:39.486 7fddf3fff700 1 -- 172.21.2.102:0/732770243 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fdde0003b10 msgr2=0x7fdde0006010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:07:39.501 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:39.486 7fddf3fff700 1 --2- 172.21.2.102:0/732770243 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fdde0003b10 0x7fdde0006010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:07:39.501 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:39.486 7fddf3fff700 1 -- 172.21.2.102:0/732770243 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdde00509a0 con 0x7fdde00095a0 2021-02-10T11:07:39.502 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:39.486 7fddf3fff700 1 -- 172.21.2.102:0/732770243 reap_dead start 2021-02-10T11:07:39.502 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:39.486 7fddf37fe700 1 --2- 172.21.2.102:0/732770243 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fdde0003b10 0x7fdde0006010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:07:39.502 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:39.486 7fddf3fff700 1 --2- 172.21.2.102:0/732770243 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdde00095a0 0x7fdde000baa0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 rx=0x7fddec006a40 tx=0x7fddec00c0c0).ready entity=mon.0 client_cookie=c79d60833be44c1c server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:07:39.502 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:39.486 7fddf27fc700 1 -- 172.21.2.102:0/732770243 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fddec0066f0 con 0x7fdde00095a0 2021-02-10T11:07:39.502 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:39.486 7fddf27fc700 1 -- 172.21.2.102:0/732770243 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddec007070 con 0x7fdde00095a0 2021-02-10T11:07:39.502 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:39.486 7fddf27fc700 1 -- 172.21.2.102:0/732770243 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fddec007590 con 0x7fdde00095a0 2021-02-10T11:07:39.502 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:39.486 7fddfa0dd700 1 -- 172.21.2.102:0/732770243 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdde00095a0 msgr2=0x7fdde000baa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:07:39.502 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:39.486 7fddfa0dd700 1 --2- 172.21.2.102:0/732770243 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdde00095a0 0x7fdde000baa0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 rx=0x7fddec006a40 tx=0x7fddec00c0c0).stop 2021-02-10T11:07:39.502 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:39.487 7fddfa0dd700 1 -- 172.21.2.102:0/732770243 shutdown_connections 2021-02-10T11:07:39.502 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:39.487 7fddfa0dd700 1 --2- 172.21.2.102:0/732770243 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdde00095a0 0x7fdde000baa0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:07:39.502 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:39.487 7fddfa0dd700 1 -- 172.21.2.102:0/732770243 >> 172.21.2.102:0/732770243 conn(0x7fddf40ce3e0 msgr2=0x7fddf40e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:07:39.503 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:39.487 7fddfa0dd700 1 -- 172.21.2.102:0/732770243 shutdown_connections 2021-02-10T11:07:39.503 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:39.487 7fddfa0dd700 1 -- 172.21.2.102:0/732770243 wait complete. 2021-02-10T11:07:39.503 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:39.501 7fddfa0dd700 1 Processor -- start 2021-02-10T11:07:39.503 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:39.502 7fddfa0dd700 1 -- start start 2021-02-10T11:07:39.504 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:39.502 7fddfa0dd700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fddf409e560 0x7fddf40a0130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:07:39.504 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:39.502 7fddfa0dd700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fddf409e960 0x7fddf40a0750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:07:39.504 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:39.502 7fddfa0dd700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fddf40a0d70 0x7fddf40a1af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:07:39.504 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:39.502 7fddfa0dd700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddf40c38f0 con 0x7fddf409e960 2021-02-10T11:07:39.504 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:39.502 7fddfa0dd700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddf40a1170 con 0x7fddf409e560 2021-02-10T11:07:39.504 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:39.502 7fddfa0dd700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fddf40a1390 con 0x7fddf40a0d70 2021-02-10T11:07:39.505 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:39.502 7fddf2ffd700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fddf409e960 0x7fddf40a0750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:07:39.505 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:39.502 7fddf2ffd700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fddf409e960 0x7fddf40a0750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:51322/0 (socket says 172.21.2.102:51322) 2021-02-10T11:07:39.505 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:39.502 7fddf2ffd700 1 -- 172.21.2.102:0/2245703425 learned_addr learned my addr 172.21.2.102:0/2245703425 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:07:39.505 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:39.502 7fddf2ffd700 1 -- 172.21.2.102:0/2245703425 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fddf40a0d70 msgr2=0x7fddf40a1af0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:07:39.505 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:39.502 7fddf2ffd700 1 --2- 172.21.2.102:0/2245703425 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fddf40a0d70 0x7fddf40a1af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:07:39.505 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:39.502 7fddf2ffd700 1 -- 172.21.2.102:0/2245703425 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fddf409e560 msgr2=0x7fddf40a0130 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:07:39.505 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:39.502 7fddf2ffd700 1 --2- 172.21.2.102:0/2245703425 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fddf409e560 0x7fddf40a0130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:07:39.505 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:39.502 7fddf2ffd700 1 -- 172.21.2.102:0/2245703425 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddf40a2110 con 0x7fddf409e960 2021-02-10T11:07:39.505 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:39.502 7fddf2ffd700 1 --2- 172.21.2.102:0/2245703425 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fddf409e960 0x7fddf40a0750 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 rx=0x7fdde400b650 tx=0x7fdde400b9f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:07:39.505 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:39.503 7fddebfff700 1 -- 172.21.2.102:0/2245703425 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdde4006b60 con 0x7fddf409e960 2021-02-10T11:07:39.505 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:39.503 7fddebfff700 1 -- 172.21.2.102:0/2245703425 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdde400bde0 con 0x7fddf409e960 2021-02-10T11:07:39.505 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:39.503 7fddfa0dd700 1 -- 172.21.2.102:0/2245703425 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fddf40a2110 con 0x7fddf409e960 2021-02-10T11:07:39.506 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:39.503 7fddfa0dd700 1 -- 172.21.2.102:0/2245703425 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fddf40a2110 con 0x7fddf409e960 2021-02-10T11:07:39.506 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:39.503 7fddebfff700 1 -- 172.21.2.102:0/2245703425 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdde4002220 con 0x7fddf409e960 2021-02-10T11:07:39.506 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:39.503 7fddebfff700 1 -- 172.21.2.102:0/2245703425 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fdde400a000 con 0x7fddf409e960 2021-02-10T11:07:39.506 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:39.503 7fddebfff700 1 --2- 172.21.2.102:0/2245703425 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fddd8018730 0x7fddd801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:07:39.506 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:39.503 7fddebfff700 1 -- 172.21.2.102:0/2245703425 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fdde4032620 con 0x7fddf409e960 2021-02-10T11:07:39.506 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:39.503 7fddf37fe700 1 --2- 172.21.2.102:0/2245703425 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fddd8018730 0x7fddd801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:07:39.506 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:39.504 7fddf37fe700 1 --2- 172.21.2.102:0/2245703425 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fddd8018730 0x7fddd801abc0 crc :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:07:39.532 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:39.534 7fddfa0dd700 1 -- 172.21.2.102:0/2245703425 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fddf4105b30 con 0x7fddf409e960 2021-02-10T11:07:39.535 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:39.537 7fddebfff700 1 -- 172.21.2.102:0/2245703425 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fdde4000de0 con 0x7fddf409e960 2021-02-10T11:07:39.661 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:39.663 7fddfa0dd700 1 -- 172.21.2.102:0/2245703425 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344378", "target": ["mgr", ""]}) v1 -- 0x7fddf40f0b50 con 0x7fddd8018730 2021-02-10T11:07:39.686 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:39.689 7fddebfff700 1 -- 172.21.2.102:0/2245703425 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fdddc0009b0 con 0x7fddd8018730 2021-02-10T11:07:39.694 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:39.696 7fddfa0dd700 1 -- 172.21.2.102:0/2245703425 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fddd8018730 msgr2=0x7fddd801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:07:39.694 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:39.696 7fddfa0dd700 1 --2- 172.21.2.102:0/2245703425 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fddd8018730 0x7fddd801abc0 crc :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:07:39.694 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:39.696 7fddfa0dd700 1 -- 172.21.2.102:0/2245703425 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fddf409e960 msgr2=0x7fddf40a0750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:07:39.694 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:39.696 7fddfa0dd700 1 --2- 172.21.2.102:0/2245703425 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fddf409e960 0x7fddf40a0750 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 rx=0x7fdde400b650 tx=0x7fdde400b9f0).stop 2021-02-10T11:07:39.695 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:39.696 7fddfa0dd700 1 -- 172.21.2.102:0/2245703425 shutdown_connections 2021-02-10T11:07:39.695 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:39.696 7fddfa0dd700 1 --2- 172.21.2.102:0/2245703425 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fddf40a0d70 0x7fddf40a1af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:07:39.695 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:39.696 7fddfa0dd700 1 --2- 172.21.2.102:0/2245703425 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fddf409e960 0x7fddf40a0750 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:07:39.695 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:39.696 7fddfa0dd700 1 --2- 172.21.2.102:0/2245703425 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fddd8018730 0x7fddd801abc0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:07:39.695 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:39.696 7fddfa0dd700 1 --2- 172.21.2.102:0/2245703425 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fddf409e560 0x7fddf40a0130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:07:39.695 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:39.696 7fddfa0dd700 1 -- 172.21.2.102:0/2245703425 >> 172.21.2.102:0/2245703425 conn(0x7fddf409fc10 msgr2=0x7fddf4094ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:07:39.695 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:39.696 7fddfa0dd700 1 -- 172.21.2.102:0/2245703425 shutdown_connections 2021-02-10T11:07:39.695 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:39.697 7fddfa0dd700 1 -- 172.21.2.102:0/2245703425 wait complete. 2021-02-10T11:07:39.702 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000344378 2021-02-10T11:07:40.131 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:40.117 7f40c36c1700 1 Processor -- start 2021-02-10T11:07:40.131 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:40.117 7f40c36c1700 1 -- start start 2021-02-10T11:07:40.131 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:40.117 7f40c36c1700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f40bc1b0470 0x7f40bc1b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:07:40.131 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:40.117 7f40c36c1700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f40bc0ef610 0x7f40bc0efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:07:40.131 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:40.117 7f40c36c1700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40bc0f0b50 con 0x7f40bc0f0030 2021-02-10T11:07:40.131 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:40.117 7f40c36c1700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40bc0e52e0 con 0x7f40bc0ef610 2021-02-10T11:07:40.131 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:40.117 7f40c36c1700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40bc0e5490 con 0x7f40bc1b0470 2021-02-10T11:07:40.131 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:40.118 7f40c0cdc700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f40bc0ef610 0x7f40bc0efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:07:40.132 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:40.118 7f40c0cdc700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f40bc0ef610 0x7f40bc0efa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:51330/0 (socket says 172.21.2.102:51330) 2021-02-10T11:07:40.132 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:40.118 7f40c0cdc700 1 -- 172.21.2.102:0/154021328 learned_addr learned my addr 172.21.2.102:0/154021328 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:07:40.132 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:40.118 7f40c0cdc700 1 -- 172.21.2.102:0/154021328 >> v2:172.21.2.113:3300/0 conn(0x7f40bc1b0470 msgr2=0x7f40bc1b0870 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:07:40.132 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:40.118 7f40c0cdc700 1 --2- 172.21.2.102:0/154021328 >> v2:172.21.2.113:3300/0 conn(0x7f40bc1b0470 0x7f40bc1b0870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:07:40.132 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:40.118 7f40c0cdc700 1 -- 172.21.2.102:0/154021328 >> v1:172.21.2.113:6789/0 conn(0x7f40bc0f0030 legacy=0x7f40bc0f10d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:07:40.132 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:40.118 7f40c0cdc700 1 -- 172.21.2.102:0/154021328 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40bc0f0430 con 0x7f40bc0ef610 2021-02-10T11:07:40.132 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:40.118 7f40c0cdc700 1 --2- 172.21.2.102:0/154021328 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f40bc0ef610 0x7f40bc0efa10 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 rx=0x7f40b8006700 tx=0x7f40b8009b00).ready entity=mon.0 client_cookie=8e0ab66c0a745c76 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:07:40.132 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:40.118 7f40affff700 1 -- 172.21.2.102:0/154021328 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f40b800c0c0 con 0x7f40bc0ef610 2021-02-10T11:07:40.132 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:40.118 7f40affff700 1 -- 172.21.2.102:0/154021328 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40b800c400 con 0x7f40bc0ef610 2021-02-10T11:07:40.132 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:40.118 7f40affff700 1 -- 172.21.2.102:0/154021328 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f40b800c860 con 0x7f40bc0ef610 2021-02-10T11:07:40.132 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:40.118 7f40c36c1700 1 -- 172.21.2.102:0/154021328 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f40bc0ef610 msgr2=0x7f40bc0efa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:07:40.132 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:40.118 7f40c36c1700 1 --2- 172.21.2.102:0/154021328 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f40bc0ef610 0x7f40bc0efa10 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 rx=0x7f40b8006700 tx=0x7f40b8009b00).stop 2021-02-10T11:07:40.133 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:40.119 7f40c36c1700 1 -- 172.21.2.102:0/154021328 shutdown_connections 2021-02-10T11:07:40.133 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:40.119 7f40c36c1700 1 --2- 172.21.2.102:0/154021328 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f40bc0ef610 0x7f40bc0efa10 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:07:40.133 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:40.119 7f40c36c1700 1 --2- 172.21.2.102:0/154021328 >> v2:172.21.2.113:3300/0 conn(0x7f40bc1b0470 0x7f40bc1b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:07:40.133 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:40.119 7f40c36c1700 1 -- 172.21.2.102:0/154021328 >> 172.21.2.102:0/154021328 conn(0x7f40bc0ce3e0 msgr2=0x7f40bc0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:07:40.133 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:40.119 7f40c36c1700 1 -- 172.21.2.102:0/154021328 shutdown_connections 2021-02-10T11:07:40.133 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:40.119 7f40c36c1700 1 -- 172.21.2.102:0/154021328 wait complete. 2021-02-10T11:07:40.134 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:40.134 7f40c36c1700 1 Processor -- start 2021-02-10T11:07:40.134 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:40.134 7f40c36c1700 1 -- start start 2021-02-10T11:07:40.134 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:40.134 7f40c36c1700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f40bc0c36a0 0x7f40bc0c2b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:07:40.134 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:40.134 7f40c36c1700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f40bc0c3190 0x7f40bc0b1d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:07:40.135 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:40.134 7f40c36c1700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f40bc0b2370 0x7f40bc0b2d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:07:40.135 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:40.134 7f40c36c1700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40bc0b27c0 con 0x7f40bc0c3190 2021-02-10T11:07:40.135 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:40.134 7f40c36c1700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40bc0b3360 con 0x7f40bc0c36a0 2021-02-10T11:07:40.135 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:40.134 7f40c36c1700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40bc0b3540 con 0x7f40bc0b2370 2021-02-10T11:07:40.135 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:40.134 7f40c0cdc700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f40bc0c3190 0x7f40bc0b1d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:07:40.135 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:40.134 7f40c0cdc700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f40bc0c3190 0x7f40bc0b1d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:51332/0 (socket says 172.21.2.102:51332) 2021-02-10T11:07:40.135 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:40.134 7f40c0cdc700 1 -- 172.21.2.102:0/1411189352 learned_addr learned my addr 172.21.2.102:0/1411189352 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:07:40.135 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:40.135 7f40c0cdc700 1 -- 172.21.2.102:0/1411189352 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f40bc0b2370 msgr2=0x7f40bc0b2d40 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:07:40.135 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:40.135 7f40c0cdc700 1 --2- 172.21.2.102:0/1411189352 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f40bc0b2370 0x7f40bc0b2d40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:07:40.135 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:40.135 7f40c0cdc700 1 -- 172.21.2.102:0/1411189352 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f40bc0c36a0 msgr2=0x7f40bc0c2b70 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:07:40.135 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:40.135 7f40c0cdc700 1 --2- 172.21.2.102:0/1411189352 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f40bc0c36a0 0x7f40bc0c2b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:07:40.135 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:40.135 7f40c0cdc700 1 -- 172.21.2.102:0/1411189352 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40bc0b3760 con 0x7f40bc0c3190 2021-02-10T11:07:40.136 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:40.135 7f40c0cdc700 1 --2- 172.21.2.102:0/1411189352 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f40bc0c3190 0x7f40bc0b1d50 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 rx=0x7f40b800b660 tx=0x7f40b800ba00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:07:40.136 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:40.135 7f40adffb700 1 -- 172.21.2.102:0/1411189352 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f40b8009440 con 0x7f40bc0c3190 2021-02-10T11:07:40.136 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:40.135 7f40adffb700 1 -- 172.21.2.102:0/1411189352 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40b8000c90 con 0x7f40bc0c3190 2021-02-10T11:07:40.136 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:40.135 7f40adffb700 1 -- 172.21.2.102:0/1411189352 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f40b8002030 con 0x7f40bc0c3190 2021-02-10T11:07:40.136 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:40.135 7f40c36c1700 1 -- 172.21.2.102:0/1411189352 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40bc0b3760 con 0x7f40bc0c3190 2021-02-10T11:07:40.136 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:40.135 7f40c36c1700 1 -- 172.21.2.102:0/1411189352 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40bc0b3cf0 con 0x7f40bc0c3190 2021-02-10T11:07:40.136 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:40.136 7f40adffb700 1 -- 172.21.2.102:0/1411189352 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f40b80093e0 con 0x7f40bc0c3190 2021-02-10T11:07:40.136 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:40.136 7f40adffb700 1 --2- 172.21.2.102:0/1411189352 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f409c0186c0 0x7f409c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:07:40.136 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:40.136 7f40adffb700 1 -- 172.21.2.102:0/1411189352 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f40b8032460 con 0x7f40bc0c3190 2021-02-10T11:07:40.136 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:40.136 7f40c14dd700 1 --2- 172.21.2.102:0/1411189352 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f409c0186c0 0x7f409c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:07:40.136 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:40.136 7f40c14dd700 1 --2- 172.21.2.102:0/1411189352 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f409c0186c0 0x7f409c01abc0 crc :-1 s=READY pgs=201 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:07:40.165 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:40.167 7f40c36c1700 1 -- 172.21.2.102:0/1411189352 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40bc0edd10 con 0x7f40bc0c3190 2021-02-10T11:07:40.167 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:40.170 7f40adffb700 1 -- 172.21.2.102:0/1411189352 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f40b80090c0 con 0x7f40bc0c3190 2021-02-10T11:07:40.296 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:40.298 7f40c36c1700 1 -- 172.21.2.102:0/1411189352 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344378", "target": ["mgr", ""]}) v1 -- 0x7f40bc0b3cf0 con 0x7f409c0186c0 2021-02-10T11:07:40.299 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:40.301 7f40adffb700 1 -- 172.21.2.102:0/1411189352 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f40b0006970 con 0x7f409c0186c0 2021-02-10T11:07:40.303 INFO:teuthology.orchestra.run.gibba002.stdout:/volumes/_nogroup/subvolume_0000000000344378/ced49ff6-ec9a-493f-8f27-28a1d12b893b 2021-02-10T11:07:40.304 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:40.306 7f40c36c1700 1 -- 172.21.2.102:0/1411189352 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f409c0186c0 msgr2=0x7f409c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:07:40.304 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:40.306 7f40c36c1700 1 --2- 172.21.2.102:0/1411189352 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f409c0186c0 0x7f409c01abc0 crc :-1 s=READY pgs=201 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:07:40.304 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:40.306 7f40c36c1700 1 -- 172.21.2.102:0/1411189352 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f40bc0c3190 msgr2=0x7f40bc0b1d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:07:40.304 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:40.306 7f40c36c1700 1 --2- 172.21.2.102:0/1411189352 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f40bc0c3190 0x7f40bc0b1d50 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 rx=0x7f40b800b660 tx=0x7f40b800ba00).stop 2021-02-10T11:07:40.304 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:40.306 7f40c36c1700 1 -- 172.21.2.102:0/1411189352 shutdown_connections 2021-02-10T11:07:40.304 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:40.306 7f40c36c1700 1 --2- 172.21.2.102:0/1411189352 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f40bc0b2370 0x7f40bc0b2d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:07:40.304 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:40.306 7f40c36c1700 1 --2- 172.21.2.102:0/1411189352 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f40bc0c3190 0x7f40bc0b1d50 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:07:40.305 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:40.306 7f40c36c1700 1 --2- 172.21.2.102:0/1411189352 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f409c0186c0 0x7f409c01abc0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:07:40.305 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:40.306 7f40c36c1700 1 --2- 172.21.2.102:0/1411189352 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f40bc0c36a0 0x7f40bc0c2b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:07:40.305 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:40.306 7f40c36c1700 1 -- 172.21.2.102:0/1411189352 >> 172.21.2.102:0/1411189352 conn(0x7f40bc09f630 msgr2=0x7f40bc0a0130 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:07:40.305 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:40.306 7f40c36c1700 1 -- 172.21.2.102:0/1411189352 shutdown_connections 2021-02-10T11:07:40.305 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:40.306 7f40c36c1700 1 -- 172.21.2.102:0/1411189352 wait complete. 2021-02-10T11:07:40.312 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000344378 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000344378/ced49ff6-ec9a-493f-8f27-28a1d12b893b 2021-02-10T11:07:40.312 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:07:40.313 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:07:40.313 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:07:40.648 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:07:40.648 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:07:40.648 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344378/ced49ff6-ec9a-493f-8f27-28a1d12b893b/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:07:41.028 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:07:41.028 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:07:41.028 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0240261 s, 43.6 MB/s 2021-02-10T11:07:41.029 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:07:41.029 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:07:41.029 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:07:41.364 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:07:41.364 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:07:41.365 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344378/ced49ff6-ec9a-493f-8f27-28a1d12b893b/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:07:41.741 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:07:41.741 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:07:41.741 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.023087 s, 45.4 MB/s 2021-02-10T11:07:41.742 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:07:41.742 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:07:41.742 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:07:42.077 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:07:42.077 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:07:42.078 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344378/ced49ff6-ec9a-493f-8f27-28a1d12b893b/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:07:42.453 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:07:42.453 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:07:42.453 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0245878 s, 42.6 MB/s 2021-02-10T11:07:42.454 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:07:42.454 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:07:42.454 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:07:42.789 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:07:42.790 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:07:42.790 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344378/ced49ff6-ec9a-493f-8f27-28a1d12b893b/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:07:43.168 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:07:43.168 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:07:43.168 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0266927 s, 39.3 MB/s 2021-02-10T11:07:43.169 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:07:43.169 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:07:43.169 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:07:43.504 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:07:43.505 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:07:43.505 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344378/ced49ff6-ec9a-493f-8f27-28a1d12b893b/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:07:43.859 DEBUG:teuthology.orchestra.run.gibba002:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:07:43.861 DEBUG:teuthology.orchestra.run.gibba013:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:07:43.877 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:07:43.877 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:07:43.877 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.024506 s, 42.8 MB/s 2021-02-10T11:07:43.878 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:07:43.878 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:07:43.879 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:07:44.199 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:07:44.199 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:07:44.199 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344378/ced49ff6-ec9a-493f-8f27-28a1d12b893b/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:07:44.577 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:07:44.577 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:07:44.577 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0250338 s, 41.9 MB/s 2021-02-10T11:07:44.578 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:07:44.578 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:07:44.578 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:07:44.913 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:07:44.913 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:07:44.913 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344378/ced49ff6-ec9a-493f-8f27-28a1d12b893b/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:07:45.289 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:07:45.289 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:07:45.289 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.02454 s, 42.7 MB/s 2021-02-10T11:07:45.290 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:07:45.290 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:07:45.290 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:07:45.624 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:07:45.625 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:07:45.625 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344378/ced49ff6-ec9a-493f-8f27-28a1d12b893b/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:07:46.000 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:07:46.001 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:07:46.001 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.025056 s, 41.8 MB/s 2021-02-10T11:07:46.002 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:07:46.003 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:07:46.003 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:07:46.337 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:07:46.338 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:07:46.338 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344378/ced49ff6-ec9a-493f-8f27-28a1d12b893b/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:07:46.714 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:07:46.714 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:07:46.714 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0258274 s, 40.6 MB/s 2021-02-10T11:07:46.715 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:07:46.715 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:07:46.716 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:07:47.050 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:07:47.051 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:07:47.051 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344378/ced49ff6-ec9a-493f-8f27-28a1d12b893b/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:07:47.428 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:07:47.428 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:07:47.428 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0253184 s, 41.4 MB/s 2021-02-10T11:07:47.429 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000344379 2021-02-10T11:07:47.859 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:47.845 7fe6b9c7d700 1 Processor -- start 2021-02-10T11:07:47.860 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:47.845 7fe6b9c7d700 1 -- start start 2021-02-10T11:07:47.860 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:47.846 7fe6b9c7d700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe6b41b0470 0x7fe6b41b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:07:47.860 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:47.846 7fe6b9c7d700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6b40f0b50 con 0x7fe6b40f20e0 2021-02-10T11:07:47.860 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:47.846 7fe6b9c7d700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6b40f24e0 con 0x7fe6b40ef610 2021-02-10T11:07:47.860 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:47.846 7fe6b9c7d700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6b40e5ea0 con 0x7fe6b41b0470 2021-02-10T11:07:47.860 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:47.846 7fe6b3fff700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fe6b40f20e0 0x7fe6b40f0200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:43568/0 (socket says 172.21.2.102:43568) 2021-02-10T11:07:47.860 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:47.846 7fe6b3fff700 1 -- 172.21.2.102:0/1003513350 learned_addr learned my addr 172.21.2.102:0/1003513350 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:07:47.860 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:47.846 7fe6b37fe700 1 --2- 172.21.2.102:0/1003513350 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe6b41b0470 0x7fe6b41b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:07:47.860 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:47.847 7fe6b27fc700 1 -- 172.21.2.102:0/1003513350 <== mon.1 v1:172.21.2.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2517556557 0 0) 0x7fe6ac000fe0 con 0x7fe6b40ef610 2021-02-10T11:07:47.860 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:47.847 7fe6b27fc700 1 -- 172.21.2.102:0/1003513350 --> v1:172.21.2.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe698002fe0 con 0x7fe6b40ef610 2021-02-10T11:07:47.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:47.847 7fe6b27fc700 1 -- 172.21.2.102:0/1003513350 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2943880084 0 0) 0x7fe6a4001180 con 0x7fe6b40f20e0 2021-02-10T11:07:47.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:47.847 7fe6b27fc700 1 -- 172.21.2.102:0/1003513350 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe698004370 con 0x7fe6b40f20e0 2021-02-10T11:07:47.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:47.847 7fe6b37fe700 1 -- 172.21.2.102:0/1003513350 >> v1:172.21.2.113:6789/0 conn(0x7fe6b40ef610 legacy=0x7fe6b40efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:07:47.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:47.847 7fe6b37fe700 1 -- 172.21.2.102:0/1003513350 >> v1:172.21.2.102:6789/0 conn(0x7fe6b40f20e0 legacy=0x7fe6b40f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:07:47.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:47.847 7fe6b37fe700 1 -- 172.21.2.102:0/1003513350 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6b40e52e0 con 0x7fe6b41b0470 2021-02-10T11:07:47.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:47.847 7fe6b37fe700 1 --2- 172.21.2.102:0/1003513350 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe6b41b0470 0x7fe6b41b0870 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7fe6a0006650 tx=0x7fe6a0009a80).ready entity=mon.2 client_cookie=37f3f12480ea1ebe server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:07:47.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:47.847 7fe6b27fc700 1 -- 172.21.2.102:0/1003513350 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe6a000c0c0 con 0x7fe6b41b0470 2021-02-10T11:07:47.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:47.847 7fe6b27fc700 1 -- 172.21.2.102:0/1003513350 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6a000c340 con 0x7fe6b41b0470 2021-02-10T11:07:47.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:47.847 7fe6b27fc700 1 -- 172.21.2.102:0/1003513350 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe6a000c860 con 0x7fe6b41b0470 2021-02-10T11:07:47.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:47.847 7fe6b9c7d700 1 -- 172.21.2.102:0/1003513350 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe6b41b0470 msgr2=0x7fe6b41b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:07:47.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:47.847 7fe6b9c7d700 1 --2- 172.21.2.102:0/1003513350 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe6b41b0470 0x7fe6b41b0870 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7fe6a0006650 tx=0x7fe6a0009a80).stop 2021-02-10T11:07:47.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:47.848 7fe6b9c7d700 1 -- 172.21.2.102:0/1003513350 shutdown_connections 2021-02-10T11:07:47.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:47.848 7fe6b9c7d700 1 --2- 172.21.2.102:0/1003513350 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe6b41b0470 0x7fe6b41b0870 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:07:47.862 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:47.848 7fe6b9c7d700 1 -- 172.21.2.102:0/1003513350 >> 172.21.2.102:0/1003513350 conn(0x7fe6b40ce3e0 msgr2=0x7fe6b40e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:07:47.862 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:47.848 7fe6b9c7d700 1 -- 172.21.2.102:0/1003513350 shutdown_connections 2021-02-10T11:07:47.862 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:47.848 7fe6b9c7d700 1 -- 172.21.2.102:0/1003513350 wait complete. 2021-02-10T11:07:47.863 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:47.862 7fe6b9c7d700 1 Processor -- start 2021-02-10T11:07:47.863 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:47.863 7fe6b9c7d700 1 -- start start 2021-02-10T11:07:47.863 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:47.863 7fe6b9c7d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe6b40a0420 0x7fe6b40a0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:07:47.864 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:47.863 7fe6b9c7d700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe6b4189d00 0x7fe6b418a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:07:47.864 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:47.863 7fe6b9c7d700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe6b418a720 0x7fe6b40b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:07:47.864 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:47.863 7fe6b9c7d700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6b4189780 con 0x7fe6b40a0420 2021-02-10T11:07:47.864 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:47.863 7fe6b9c7d700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6b418ab20 con 0x7fe6b418a720 2021-02-10T11:07:47.864 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:47.863 7fe6b9c7d700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6b40a1140 con 0x7fe6b4189d00 2021-02-10T11:07:47.864 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:47.863 7fe6b37fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe6b40a0420 0x7fe6b40a0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:07:47.864 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:47.863 7fe6b37fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe6b40a0420 0x7fe6b40a0820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:51348/0 (socket says 172.21.2.102:51348) 2021-02-10T11:07:47.864 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:47.863 7fe6b37fe700 1 -- 172.21.2.102:0/1379879584 learned_addr learned my addr 172.21.2.102:0/1379879584 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:07:47.864 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:47.863 7fe6b37fe700 1 -- 172.21.2.102:0/1379879584 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe6b4189d00 msgr2=0x7fe6b418a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:07:47.864 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:47.863 7fe6b37fe700 1 --2- 172.21.2.102:0/1379879584 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe6b4189d00 0x7fe6b418a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:07:47.865 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:47.863 7fe6b37fe700 1 -- 172.21.2.102:0/1379879584 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe6b418a720 msgr2=0x7fe6b40b7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:07:47.865 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:47.863 7fe6b37fe700 1 --2- 172.21.2.102:0/1379879584 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe6b418a720 0x7fe6b40b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:07:47.865 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:47.863 7fe6b37fe700 1 -- 172.21.2.102:0/1379879584 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6b40b8200 con 0x7fe6b40a0420 2021-02-10T11:07:47.865 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:47.864 7fe6b37fe700 1 --2- 172.21.2.102:0/1379879584 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe6b40a0420 0x7fe6b40a0820 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 rx=0x7fe6a0009650 tx=0x7fe6a000b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:07:47.865 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:47.864 7fe6abfff700 1 -- 172.21.2.102:0/1379879584 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe6a000bd50 con 0x7fe6b40a0420 2021-02-10T11:07:47.865 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:47.864 7fe6abfff700 1 -- 172.21.2.102:0/1379879584 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6a0006030 con 0x7fe6b40a0420 2021-02-10T11:07:47.865 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:47.864 7fe6abfff700 1 -- 172.21.2.102:0/1379879584 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe6a0005560 con 0x7fe6b40a0420 2021-02-10T11:07:47.865 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:47.864 7fe6b9c7d700 1 -- 172.21.2.102:0/1379879584 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6b40b8200 con 0x7fe6b40a0420 2021-02-10T11:07:47.865 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:47.864 7fe6b9c7d700 1 -- 172.21.2.102:0/1379879584 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6b40b8790 con 0x7fe6b40a0420 2021-02-10T11:07:47.865 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:47.865 7fe6abfff700 1 -- 172.21.2.102:0/1379879584 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fe6a0009300 con 0x7fe6b40a0420 2021-02-10T11:07:47.865 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:47.865 7fe6abfff700 1 --2- 172.21.2.102:0/1379879584 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe6900186c0 0x7fe69001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:07:47.865 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:47.865 7fe6abfff700 1 -- 172.21.2.102:0/1379879584 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fe6a0032580 con 0x7fe6b40a0420 2021-02-10T11:07:47.865 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:47.865 7fe6b2ffd700 1 --2- 172.21.2.102:0/1379879584 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe6900186c0 0x7fe69001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:07:47.866 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:47.865 7fe6b2ffd700 1 --2- 172.21.2.102:0/1379879584 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe6900186c0 0x7fe69001abc0 crc :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:07:47.893 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:47.895 7fe6b9c7d700 1 -- 172.21.2.102:0/1379879584 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6b4105ae0 con 0x7fe6b40a0420 2021-02-10T11:07:47.895 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:47.898 7fe6abfff700 1 -- 172.21.2.102:0/1379879584 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fe6a0030d30 con 0x7fe6b40a0420 2021-02-10T11:07:48.023 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:48.026 7fe6b9c7d700 1 -- 172.21.2.102:0/1379879584 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344379", "target": ["mgr", ""]}) v1 -- 0x7fe6b40b8790 con 0x7fe6900186c0 2021-02-10T11:07:48.049 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:48.051 7fe6abfff700 1 -- 172.21.2.102:0/1379879584 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fe6ac000f00 con 0x7fe6900186c0 2021-02-10T11:07:48.056 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:48.059 7fe6b9c7d700 1 -- 172.21.2.102:0/1379879584 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe6900186c0 msgr2=0x7fe69001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:07:48.056 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:48.059 7fe6b9c7d700 1 --2- 172.21.2.102:0/1379879584 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe6900186c0 0x7fe69001abc0 crc :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:07:48.057 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:48.059 7fe6b9c7d700 1 -- 172.21.2.102:0/1379879584 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe6b40a0420 msgr2=0x7fe6b40a0820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:07:48.057 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:48.059 7fe6b9c7d700 1 --2- 172.21.2.102:0/1379879584 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe6b40a0420 0x7fe6b40a0820 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 rx=0x7fe6a0009650 tx=0x7fe6a000b960).stop 2021-02-10T11:07:48.057 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:48.059 7fe6b9c7d700 1 -- 172.21.2.102:0/1379879584 shutdown_connections 2021-02-10T11:07:48.057 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:48.059 7fe6b9c7d700 1 --2- 172.21.2.102:0/1379879584 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe6900186c0 0x7fe69001abc0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:07:48.057 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:48.059 7fe6b9c7d700 1 --2- 172.21.2.102:0/1379879584 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe6b418a720 0x7fe6b40b7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:07:48.057 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:48.059 7fe6b9c7d700 1 --2- 172.21.2.102:0/1379879584 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe6b4189d00 0x7fe6b418a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:07:48.057 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:48.059 7fe6b9c7d700 1 --2- 172.21.2.102:0/1379879584 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe6b40a0420 0x7fe6b40a0820 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:07:48.057 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:48.059 7fe6b9c7d700 1 -- 172.21.2.102:0/1379879584 >> 172.21.2.102:0/1379879584 conn(0x7fe6b4094e10 msgr2=0x7fe6b40a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:07:48.057 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:48.059 7fe6b9c7d700 1 -- 172.21.2.102:0/1379879584 shutdown_connections 2021-02-10T11:07:48.058 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:48.059 7fe6b9c7d700 1 -- 172.21.2.102:0/1379879584 wait complete. 2021-02-10T11:07:48.065 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000344379 2021-02-10T11:07:48.497 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:48.483 7fcc31b61700 1 Processor -- start 2021-02-10T11:07:48.497 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:48.483 7fcc31b61700 1 -- start start 2021-02-10T11:07:48.497 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:48.483 7fcc31b61700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fcc2c1b0470 0x7fcc2c1b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:07:48.497 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:48.483 7fcc31b61700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fcc2c0ef610 0x7fcc2c0efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:07:48.497 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:48.483 7fcc31b61700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc2c0f0b50 con 0x7fcc2c0f0030 2021-02-10T11:07:48.497 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:48.483 7fcc31b61700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc2c0e52e0 con 0x7fcc2c0ef610 2021-02-10T11:07:48.497 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:48.483 7fcc31b61700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc2c0e5490 con 0x7fcc2c1b0470 2021-02-10T11:07:48.497 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:48.483 7fcc2affd700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fcc2c0ef610 0x7fcc2c0efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:07:48.498 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:48.483 7fcc2affd700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fcc2c0ef610 0x7fcc2c0efa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:51354/0 (socket says 172.21.2.102:51354) 2021-02-10T11:07:48.498 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:48.483 7fcc2affd700 1 -- 172.21.2.102:0/378012512 learned_addr learned my addr 172.21.2.102:0/378012512 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:07:48.498 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:48.484 7fcc2affd700 1 -- 172.21.2.102:0/378012512 >> v2:172.21.2.113:3300/0 conn(0x7fcc2c1b0470 msgr2=0x7fcc2c1b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:07:48.498 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:48.484 7fcc2affd700 1 --2- 172.21.2.102:0/378012512 >> v2:172.21.2.113:3300/0 conn(0x7fcc2c1b0470 0x7fcc2c1b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:07:48.498 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:48.484 7fcc2affd700 1 -- 172.21.2.102:0/378012512 >> v1:172.21.2.102:6789/0 conn(0x7fcc2c0f0030 legacy=0x7fcc2c0f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:07:48.498 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:48.484 7fcc2affd700 1 -- 172.21.2.102:0/378012512 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc2c0f0430 con 0x7fcc2c0ef610 2021-02-10T11:07:48.498 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:48.484 7fcc2affd700 1 --2- 172.21.2.102:0/378012512 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcc2c0ef610 0x7fcc2c0efa10 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 rx=0x7fcc1c006700 tx=0x7fcc1c009b80).ready entity=mon.0 client_cookie=f23f2ee6db478e65 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:07:48.498 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:48.484 7fcc2a7fc700 1 -- 172.21.2.102:0/378012512 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcc1c00c0c0 con 0x7fcc2c0ef610 2021-02-10T11:07:48.498 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:48.484 7fcc2a7fc700 1 -- 172.21.2.102:0/378012512 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc1c00c440 con 0x7fcc2c0ef610 2021-02-10T11:07:48.498 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:48.484 7fcc2a7fc700 1 -- 172.21.2.102:0/378012512 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcc1c00c8a0 con 0x7fcc2c0ef610 2021-02-10T11:07:48.498 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:48.484 7fcc31b61700 1 -- 172.21.2.102:0/378012512 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcc2c0ef610 msgr2=0x7fcc2c0efa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:07:48.498 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:48.484 7fcc31b61700 1 --2- 172.21.2.102:0/378012512 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcc2c0ef610 0x7fcc2c0efa10 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 rx=0x7fcc1c006700 tx=0x7fcc1c009b80).stop 2021-02-10T11:07:48.498 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:48.485 7fcc31b61700 1 -- 172.21.2.102:0/378012512 shutdown_connections 2021-02-10T11:07:48.499 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:48.485 7fcc31b61700 1 --2- 172.21.2.102:0/378012512 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcc2c0ef610 0x7fcc2c0efa10 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:07:48.499 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:48.485 7fcc31b61700 1 --2- 172.21.2.102:0/378012512 >> v2:172.21.2.113:3300/0 conn(0x7fcc2c1b0470 0x7fcc2c1b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:07:48.499 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:48.485 7fcc31b61700 1 -- 172.21.2.102:0/378012512 >> 172.21.2.102:0/378012512 conn(0x7fcc2c0ce3e0 msgr2=0x7fcc2c0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:07:48.499 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:48.485 7fcc31b61700 1 -- 172.21.2.102:0/378012512 shutdown_connections 2021-02-10T11:07:48.499 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:48.485 7fcc31b61700 1 -- 172.21.2.102:0/378012512 wait complete. 2021-02-10T11:07:48.500 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:48.500 7fcc31b61700 1 Processor -- start 2021-02-10T11:07:48.500 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:48.500 7fcc31b61700 1 -- start start 2021-02-10T11:07:48.500 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:48.500 7fcc31b61700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fcc2c0a0960 0x7fcc2c0a67f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:07:48.500 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:48.500 7fcc31b61700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcc2c1924c0 0x7fcc2c1928c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:07:48.500 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:48.500 7fcc31b61700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcc2c192ee0 0x7fcc2c0a29f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:07:48.501 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:48.500 7fcc31b61700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc2c0a6e10 con 0x7fcc2c1924c0 2021-02-10T11:07:48.501 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:48.500 7fcc31b61700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc2c190630 con 0x7fcc2c0a0960 2021-02-10T11:07:48.501 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:48.500 7fcc31b61700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc2c1932e0 con 0x7fcc2c192ee0 2021-02-10T11:07:48.501 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:48.500 7fcc2affd700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcc2c1924c0 0x7fcc2c1928c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:07:48.501 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:48.500 7fcc2affd700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcc2c1924c0 0x7fcc2c1928c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:51362/0 (socket says 172.21.2.102:51362) 2021-02-10T11:07:48.501 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:48.500 7fcc2affd700 1 -- 172.21.2.102:0/2936599382 learned_addr learned my addr 172.21.2.102:0/2936599382 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:07:48.501 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:48.501 7fcc2affd700 1 -- 172.21.2.102:0/2936599382 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcc2c192ee0 msgr2=0x7fcc2c0a29f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:07:48.501 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:48.501 7fcc2affd700 1 --2- 172.21.2.102:0/2936599382 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcc2c192ee0 0x7fcc2c0a29f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:07:48.501 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:48.501 7fcc2affd700 1 -- 172.21.2.102:0/2936599382 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fcc2c0a0960 msgr2=0x7fcc2c0a67f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:07:48.501 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:48.501 7fcc2affd700 1 --2- 172.21.2.102:0/2936599382 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fcc2c0a0960 0x7fcc2c0a67f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:07:48.501 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:48.501 7fcc2affd700 1 -- 172.21.2.102:0/2936599382 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc2c193500 con 0x7fcc2c1924c0 2021-02-10T11:07:48.501 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:48.501 7fcc2affd700 1 --2- 172.21.2.102:0/2936599382 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcc2c1924c0 0x7fcc2c1928c0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 rx=0x7fcc1c00b5e0 tx=0x7fcc1c00b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:07:48.501 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:48.501 7fcc23fff700 1 -- 172.21.2.102:0/2936599382 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcc1c00bd70 con 0x7fcc2c1924c0 2021-02-10T11:07:48.502 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:48.501 7fcc23fff700 1 -- 172.21.2.102:0/2936599382 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc1c002030 con 0x7fcc2c1924c0 2021-02-10T11:07:48.502 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:48.501 7fcc23fff700 1 -- 172.21.2.102:0/2936599382 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcc1c002550 con 0x7fcc2c1924c0 2021-02-10T11:07:48.502 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:48.501 7fcc31b61700 1 -- 172.21.2.102:0/2936599382 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc2c193500 con 0x7fcc2c1924c0 2021-02-10T11:07:48.502 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:48.501 7fcc31b61700 1 -- 172.21.2.102:0/2936599382 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc2c0a3300 con 0x7fcc2c1924c0 2021-02-10T11:07:48.502 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:48.502 7fcc23fff700 1 -- 172.21.2.102:0/2936599382 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fcc1c000c90 con 0x7fcc2c1924c0 2021-02-10T11:07:48.502 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:48.502 7fcc23fff700 1 --2- 172.21.2.102:0/2936599382 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcc100186c0 0x7fcc1001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:07:48.502 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:48.502 7fcc23fff700 1 -- 172.21.2.102:0/2936599382 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fcc1c032590 con 0x7fcc2c1924c0 2021-02-10T11:07:48.502 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:48.502 7fcc2b7fe700 1 --2- 172.21.2.102:0/2936599382 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcc100186c0 0x7fcc1001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:07:48.502 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:48.502 7fcc2b7fe700 1 --2- 172.21.2.102:0/2936599382 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcc100186c0 0x7fcc1001abc0 crc :-1 s=READY pgs=203 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:07:48.530 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:48.533 7fcc31b61700 1 -- 172.21.2.102:0/2936599382 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc2c0a24c0 con 0x7fcc2c1924c0 2021-02-10T11:07:48.533 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:48.535 7fcc23fff700 1 -- 172.21.2.102:0/2936599382 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fcc1c0009b0 con 0x7fcc2c1924c0 2021-02-10T11:07:48.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:48.664 7fcc31b61700 1 -- 172.21.2.102:0/2936599382 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344379", "target": ["mgr", ""]}) v1 -- 0x7fcc2c0a6e10 con 0x7fcc100186c0 2021-02-10T11:07:48.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:48.667 7fcc23fff700 1 -- 172.21.2.102:0/2936599382 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7fcc14006e40 con 0x7fcc100186c0 2021-02-10T11:07:48.669 INFO:teuthology.orchestra.run.gibba002.stdout:/volumes/_nogroup/subvolume_0000000000344379/d8f639ec-a1a7-4d83-91d3-92bfaf329ea6 2021-02-10T11:07:48.670 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:48.673 7fcc31b61700 1 -- 172.21.2.102:0/2936599382 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcc100186c0 msgr2=0x7fcc1001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:07:48.670 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:48.673 7fcc31b61700 1 --2- 172.21.2.102:0/2936599382 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcc100186c0 0x7fcc1001abc0 crc :-1 s=READY pgs=203 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:07:48.670 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:48.673 7fcc31b61700 1 -- 172.21.2.102:0/2936599382 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcc2c1924c0 msgr2=0x7fcc2c1928c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:07:48.670 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:48.673 7fcc31b61700 1 --2- 172.21.2.102:0/2936599382 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcc2c1924c0 0x7fcc2c1928c0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 rx=0x7fcc1c00b5e0 tx=0x7fcc1c00b980).stop 2021-02-10T11:07:48.671 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:48.673 7fcc31b61700 1 -- 172.21.2.102:0/2936599382 shutdown_connections 2021-02-10T11:07:48.671 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:48.673 7fcc31b61700 1 --2- 172.21.2.102:0/2936599382 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcc2c192ee0 0x7fcc2c0a29f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:07:48.671 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:48.673 7fcc31b61700 1 --2- 172.21.2.102:0/2936599382 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcc2c1924c0 0x7fcc2c1928c0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:07:48.671 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:48.673 7fcc31b61700 1 --2- 172.21.2.102:0/2936599382 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcc100186c0 0x7fcc1001abc0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:07:48.671 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:48.673 7fcc31b61700 1 --2- 172.21.2.102:0/2936599382 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fcc2c0a0960 0x7fcc2c0a67f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:07:48.671 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:48.673 7fcc31b61700 1 -- 172.21.2.102:0/2936599382 >> 172.21.2.102:0/2936599382 conn(0x7fcc2c0ac550 msgr2=0x7fcc2c0be1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:07:48.671 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:48.673 7fcc31b61700 1 -- 172.21.2.102:0/2936599382 shutdown_connections 2021-02-10T11:07:48.671 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:48.673 7fcc31b61700 1 -- 172.21.2.102:0/2936599382 wait complete. 2021-02-10T11:07:48.679 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000344379 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000344379/d8f639ec-a1a7-4d83-91d3-92bfaf329ea6 2021-02-10T11:07:48.679 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:07:48.679 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:07:48.679 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:07:49.015 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:07:49.016 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:07:49.016 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344379/d8f639ec-a1a7-4d83-91d3-92bfaf329ea6/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:07:49.396 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:07:49.396 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:07:49.396 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0251255 s, 41.7 MB/s 2021-02-10T11:07:49.397 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:07:49.397 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:07:49.398 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:07:49.732 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:07:49.733 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:07:49.733 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344379/d8f639ec-a1a7-4d83-91d3-92bfaf329ea6/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:07:50.108 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:07:50.109 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:07:50.109 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0242841 s, 43.2 MB/s 2021-02-10T11:07:50.110 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:07:50.110 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:07:50.110 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:07:50.446 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:07:50.447 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:07:50.447 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344379/d8f639ec-a1a7-4d83-91d3-92bfaf329ea6/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:07:50.825 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:07:50.826 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:07:50.826 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0267531 s, 39.2 MB/s 2021-02-10T11:07:50.827 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:07:50.827 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:07:50.827 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:07:51.162 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:07:51.163 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:07:51.163 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344379/d8f639ec-a1a7-4d83-91d3-92bfaf329ea6/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:07:51.539 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:07:51.539 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:07:51.540 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0252269 s, 41.6 MB/s 2021-02-10T11:07:51.540 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:07:51.541 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:07:51.541 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:07:51.875 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:07:51.876 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:07:51.876 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344379/d8f639ec-a1a7-4d83-91d3-92bfaf329ea6/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:07:52.251 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:07:52.251 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:07:52.251 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0235734 s, 44.5 MB/s 2021-02-10T11:07:52.252 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:07:52.252 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:07:52.252 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:07:52.586 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:07:52.586 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:07:52.586 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344379/d8f639ec-a1a7-4d83-91d3-92bfaf329ea6/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:07:52.960 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:07:52.961 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:07:52.961 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0250186 s, 41.9 MB/s 2021-02-10T11:07:52.962 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:07:52.962 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:07:52.962 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:07:53.298 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:07:53.298 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:07:53.298 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344379/d8f639ec-a1a7-4d83-91d3-92bfaf329ea6/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:07:53.672 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:07:53.672 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:07:53.672 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0248267 s, 42.2 MB/s 2021-02-10T11:07:53.673 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:07:53.674 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:07:53.674 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:07:54.008 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:07:54.009 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:07:54.009 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344379/d8f639ec-a1a7-4d83-91d3-92bfaf329ea6/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:07:54.386 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:07:54.386 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:07:54.386 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0248862 s, 42.1 MB/s 2021-02-10T11:07:54.387 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:07:54.387 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:07:54.387 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:07:54.726 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:07:54.727 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:07:54.728 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344379/d8f639ec-a1a7-4d83-91d3-92bfaf329ea6/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:07:55.114 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:07:55.114 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:07:55.115 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0256994 s, 40.8 MB/s 2021-02-10T11:07:55.115 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:07:55.115 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:07:55.116 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:07:55.453 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:07:55.453 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:07:55.454 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344379/d8f639ec-a1a7-4d83-91d3-92bfaf329ea6/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:07:55.836 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:07:55.837 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:07:55.837 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0238999 s, 43.9 MB/s 2021-02-10T11:07:55.839 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000344380 2021-02-10T11:07:56.278 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:56.262 7fb4a3d00700 1 Processor -- start 2021-02-10T11:07:56.278 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:56.262 7fb4a3d00700 1 -- start start 2021-02-10T11:07:56.278 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:56.262 7fb4a3d00700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb49c1b0470 0x7fb49c1b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:07:56.278 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:56.262 7fb4a3d00700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fb49c0f20e0 0x7fb49c0f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:07:56.278 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:56.262 7fb4a3d00700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb49c0f0b50 con 0x7fb49c0ef610 2021-02-10T11:07:56.278 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:56.262 7fb4a3d00700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb49c0f0820 con 0x7fb49c0f20e0 2021-02-10T11:07:56.278 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:56.262 7fb4a3d00700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb49c0f24e0 con 0x7fb49c1b0470 2021-02-10T11:07:56.278 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:56.265 7fb4a231d700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fb49c0f20e0 0x7fb49c0f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:07:56.278 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:56.265 7fb4a1b1c700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb49c1b0470 0x7fb49c1b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:07:56.278 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:56.265 7fb4a131b700 1 --1- >> v1:172.21.2.113:6789/0 conn(0x7fb49c0ef610 0x7fb49c0efa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.113:6789/0 says I am v1:172.21.2.102:50914/0 (socket says 172.21.2.102:50914) 2021-02-10T11:07:56.279 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:56.265 7fb4a131b700 1 -- 172.21.2.102:0/2804266418 learned_addr learned my addr 172.21.2.102:0/2804266418 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:07:56.279 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:56.265 7fb4a231d700 1 -- 172.21.2.102:0/2804266418 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb49c1b0470 msgr2=0x7fb49c1b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:07:56.279 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:56.265 7fb4a231d700 1 --2- 172.21.2.102:0/2804266418 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb49c1b0470 0x7fb49c1b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:07:56.279 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:56.265 7fb4a231d700 1 -- 172.21.2.102:0/2804266418 >> v1:172.21.2.113:6789/0 conn(0x7fb49c0ef610 legacy=0x7fb49c0efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:07:56.279 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:56.265 7fb4a231d700 1 -- 172.21.2.102:0/2804266418 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb49c0e52e0 con 0x7fb49c0f20e0 2021-02-10T11:07:56.279 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:56.265 7fb4a1b1c700 1 --2- 172.21.2.102:0/2804266418 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb49c1b0470 0x7fb49c1b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:07:56.279 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:56.265 7fb4a231d700 1 --2- 172.21.2.102:0/2804266418 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb49c0f20e0 0x7fb49c0f0200 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7fb494006e50 tx=0x7fb494009e70).ready entity=mon.1 client_cookie=c8902dd8412626a7 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:07:56.279 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:56.265 7fb4a0b1a700 1 -- 172.21.2.102:0/2804266418 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb494006a20 con 0x7fb49c0f20e0 2021-02-10T11:07:56.279 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:56.265 7fb4a0b1a700 1 -- 172.21.2.102:0/2804266418 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb494006790 con 0x7fb49c0f20e0 2021-02-10T11:07:56.279 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:56.265 7fb4a0b1a700 1 -- 172.21.2.102:0/2804266418 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb49400b700 con 0x7fb49c0f20e0 2021-02-10T11:07:56.279 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:56.265 7fb4a3d00700 1 -- 172.21.2.102:0/2804266418 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb49c0f20e0 msgr2=0x7fb49c0f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:07:56.279 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:56.265 7fb4a3d00700 1 --2- 172.21.2.102:0/2804266418 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb49c0f20e0 0x7fb49c0f0200 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7fb494006e50 tx=0x7fb494009e70).stop 2021-02-10T11:07:56.279 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:56.266 7fb4a3d00700 1 -- 172.21.2.102:0/2804266418 shutdown_connections 2021-02-10T11:07:56.280 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:56.266 7fb4a3d00700 1 --2- 172.21.2.102:0/2804266418 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb49c0f20e0 0x7fb49c0f0200 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:07:56.280 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:56.266 7fb4a3d00700 1 --2- 172.21.2.102:0/2804266418 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb49c1b0470 0x7fb49c1b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:07:56.280 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:56.266 7fb4a3d00700 1 -- 172.21.2.102:0/2804266418 >> 172.21.2.102:0/2804266418 conn(0x7fb49c0ce3e0 msgr2=0x7fb49c0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:07:56.280 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:56.266 7fb4a3d00700 1 -- 172.21.2.102:0/2804266418 shutdown_connections 2021-02-10T11:07:56.280 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:56.266 7fb4a3d00700 1 -- 172.21.2.102:0/2804266418 wait complete. 2021-02-10T11:07:56.281 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:56.280 7fb4a3d00700 1 Processor -- start 2021-02-10T11:07:56.281 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:56.280 7fb4a3d00700 1 -- start start 2021-02-10T11:07:56.281 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:56.281 7fb4a3d00700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb49c08edf0 0x7fb49c192870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:07:56.281 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:56.281 7fb4a3d00700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb49c192e90 0x7fb49c08ca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:07:56.282 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:56.281 7fb4a3d00700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb49c193290 0x7fb49c08d040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:07:56.282 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:56.281 7fb4a3d00700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb49c08c4a0 con 0x7fb49c08edf0 2021-02-10T11:07:56.282 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:56.281 7fb4a3d00700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb49c08f1f0 con 0x7fb49c192e90 2021-02-10T11:07:56.282 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:56.281 7fb4a3d00700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb49c09fd30 con 0x7fb49c193290 2021-02-10T11:07:56.282 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:56.281 7fb4a1b1c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb49c08edf0 0x7fb49c192870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:07:56.283 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:56.281 7fb4a1b1c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb49c08edf0 0x7fb49c192870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:51372/0 (socket says 172.21.2.102:51372) 2021-02-10T11:07:56.283 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:56.281 7fb4a1b1c700 1 -- 172.21.2.102:0/2108706934 learned_addr learned my addr 172.21.2.102:0/2108706934 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:07:56.283 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:56.281 7fb4a1b1c700 1 -- 172.21.2.102:0/2108706934 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb49c193290 msgr2=0x7fb49c08d040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:07:56.283 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:56.281 7fb4a1b1c700 1 --2- 172.21.2.102:0/2108706934 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb49c193290 0x7fb49c08d040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:07:56.283 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:56.281 7fb4a1b1c700 1 -- 172.21.2.102:0/2108706934 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb49c192e90 msgr2=0x7fb49c08ca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:07:56.284 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:56.281 7fb4a1b1c700 1 --2- 172.21.2.102:0/2108706934 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb49c192e90 0x7fb49c08ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:07:56.284 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:56.281 7fb4a1b1c700 1 -- 172.21.2.102:0/2108706934 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb49c08d660 con 0x7fb49c08edf0 2021-02-10T11:07:56.284 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:56.282 7fb4a1b1c700 1 --2- 172.21.2.102:0/2108706934 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb49c08edf0 0x7fb49c192870 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 rx=0x7fb49800b580 tx=0x7fb49800b920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:07:56.284 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:56.282 7fb48e7fc700 1 -- 172.21.2.102:0/2108706934 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb49800bd10 con 0x7fb49c08edf0 2021-02-10T11:07:56.284 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:56.282 7fb48e7fc700 1 -- 172.21.2.102:0/2108706934 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb498000c90 con 0x7fb49c08edf0 2021-02-10T11:07:56.284 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:56.282 7fb48e7fc700 1 -- 172.21.2.102:0/2108706934 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb4980022b0 con 0x7fb49c08edf0 2021-02-10T11:07:56.285 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:56.282 7fb4a3d00700 1 -- 172.21.2.102:0/2108706934 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb49c08d660 con 0x7fb49c08edf0 2021-02-10T11:07:56.285 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:56.282 7fb4a3d00700 1 -- 172.21.2.102:0/2108706934 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb49c08d660 con 0x7fb49c08edf0 2021-02-10T11:07:56.285 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:56.283 7fb48e7fc700 1 -- 172.21.2.102:0/2108706934 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fb498006b00 con 0x7fb49c08edf0 2021-02-10T11:07:56.285 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:56.283 7fb48e7fc700 1 --2- 172.21.2.102:0/2108706934 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb484018730 0x7fb48401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:07:56.285 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:56.283 7fb48e7fc700 1 -- 172.21.2.102:0/2108706934 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fb4980327a0 con 0x7fb49c08edf0 2021-02-10T11:07:56.285 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:56.283 7fb4a131b700 1 --2- 172.21.2.102:0/2108706934 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb484018730 0x7fb48401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:07:56.285 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:56.283 7fb4a131b700 1 --2- 172.21.2.102:0/2108706934 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb484018730 0x7fb48401ac10 crc :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:07:56.310 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:56.312 7fb4a3d00700 1 -- 172.21.2.102:0/2108706934 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb49c19d290 con 0x7fb49c08edf0 2021-02-10T11:07:56.312 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:56.315 7fb48e7fc700 1 -- 172.21.2.102:0/2108706934 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fb49800bd10 con 0x7fb49c08edf0 2021-02-10T11:07:56.438 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:56.441 7fb4a3d00700 1 -- 172.21.2.102:0/2108706934 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344380", "target": ["mgr", ""]}) v1 -- 0x7fb49c19d290 con 0x7fb484018730 2021-02-10T11:07:56.463 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:56.466 7fb48e7fc700 1 -- 172.21.2.102:0/2108706934 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fb490001590 con 0x7fb484018730 2021-02-10T11:07:56.472 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:56.474 7fb4a3d00700 1 -- 172.21.2.102:0/2108706934 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb484018730 msgr2=0x7fb48401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:07:56.472 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:56.474 7fb4a3d00700 1 --2- 172.21.2.102:0/2108706934 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb484018730 0x7fb48401ac10 crc :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:07:56.472 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:56.474 7fb4a3d00700 1 -- 172.21.2.102:0/2108706934 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb49c08edf0 msgr2=0x7fb49c192870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:07:56.472 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:56.474 7fb4a3d00700 1 --2- 172.21.2.102:0/2108706934 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb49c08edf0 0x7fb49c192870 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 rx=0x7fb49800b580 tx=0x7fb49800b920).stop 2021-02-10T11:07:56.472 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:56.474 7fb4a3d00700 1 -- 172.21.2.102:0/2108706934 shutdown_connections 2021-02-10T11:07:56.472 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:56.474 7fb4a3d00700 1 --2- 172.21.2.102:0/2108706934 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb49c193290 0x7fb49c08d040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:07:56.472 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:56.474 7fb4a3d00700 1 --2- 172.21.2.102:0/2108706934 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb484018730 0x7fb48401ac10 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:07:56.472 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:56.474 7fb4a3d00700 1 --2- 172.21.2.102:0/2108706934 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb49c192e90 0x7fb49c08ca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:07:56.472 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:56.474 7fb4a3d00700 1 --2- 172.21.2.102:0/2108706934 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb49c08edf0 0x7fb49c192870 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:07:56.473 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:56.474 7fb4a3d00700 1 -- 172.21.2.102:0/2108706934 >> 172.21.2.102:0/2108706934 conn(0x7fb49c0bf4f0 msgr2=0x7fb49c0be250 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:07:56.473 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:56.474 7fb4a3d00700 1 -- 172.21.2.102:0/2108706934 shutdown_connections 2021-02-10T11:07:56.473 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:56.474 7fb4a3d00700 1 -- 172.21.2.102:0/2108706934 wait complete. 2021-02-10T11:07:56.480 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000344380 2021-02-10T11:07:56.913 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:56.898 7f01c6695700 1 Processor -- start 2021-02-10T11:07:56.913 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:56.899 7f01c6695700 1 -- start start 2021-02-10T11:07:56.913 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:56.899 7f01c6695700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f01c01b0470 0x7f01c01b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:07:56.913 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:56.899 7f01c6695700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f01c00f20e0 0x7f01c00f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:07:56.913 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:56.899 7f01c6695700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01c00f0b50 con 0x7f01c00ef610 2021-02-10T11:07:56.913 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:56.899 7f01c6695700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01c00f0820 con 0x7f01c00f20e0 2021-02-10T11:07:56.913 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:56.899 7f01c6695700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01c00f24e0 con 0x7f01c01b0470 2021-02-10T11:07:56.913 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:56.899 7f01bf7fe700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f01c00ef610 0x7f01c00efa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:43602/0 (socket says 172.21.2.102:43602) 2021-02-10T11:07:56.913 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:56.899 7f01bf7fe700 1 -- 172.21.2.102:0/2537281870 learned_addr learned my addr 172.21.2.102:0/2537281870 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:07:56.913 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:56.899 7f01c4cb2700 1 --2- 172.21.2.102:0/2537281870 >> v2:172.21.2.102:3300/0 conn(0x7f01c00f20e0 0x7f01c00f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:07:56.914 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:56.899 7f01bffff700 1 --2- 172.21.2.102:0/2537281870 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f01c01b0470 0x7f01c01b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:07:56.914 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:56.900 7f01beffd700 1 -- 172.21.2.102:0/2537281870 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3144745501 0 0) 0x7f01b40010b0 con 0x7f01c00ef610 2021-02-10T11:07:56.914 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:56.900 7f01c4cb2700 1 -- 172.21.2.102:0/2537281870 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f01c01b0470 msgr2=0x7f01c01b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:07:56.914 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:56.900 7f01c4cb2700 1 --2- 172.21.2.102:0/2537281870 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f01c01b0470 0x7f01c01b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:07:56.914 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:56.900 7f01c4cb2700 1 -- 172.21.2.102:0/2537281870 >> v1:172.21.2.102:6789/0 conn(0x7f01c00ef610 legacy=0x7f01c00efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:07:56.914 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:56.900 7f01c4cb2700 1 -- 172.21.2.102:0/2537281870 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01c00e52e0 con 0x7f01c00f20e0 2021-02-10T11:07:56.914 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:56.900 7f01c4cb2700 1 --2- 172.21.2.102:0/2537281870 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f01c00f20e0 0x7f01c00f0200 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 rx=0x7f01b8006e30 tx=0x7f01b8009e60).ready entity=mon.0 client_cookie=7fee3b36632bccb9 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:07:56.914 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:56.900 7f01beffd700 1 -- 172.21.2.102:0/2537281870 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f01b800b300 con 0x7f01c00f20e0 2021-02-10T11:07:56.914 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:56.900 7f01beffd700 1 -- 172.21.2.102:0/2537281870 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01b8000d30 con 0x7f01c00f20e0 2021-02-10T11:07:56.914 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:56.900 7f01beffd700 1 -- 172.21.2.102:0/2537281870 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f01b800b830 con 0x7f01c00f20e0 2021-02-10T11:07:56.914 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:56.900 7f01c6695700 1 -- 172.21.2.102:0/2537281870 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f01c00f20e0 msgr2=0x7f01c00f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:07:56.914 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:56.900 7f01c6695700 1 --2- 172.21.2.102:0/2537281870 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f01c00f20e0 0x7f01c00f0200 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 rx=0x7f01b8006e30 tx=0x7f01b8009e60).stop 2021-02-10T11:07:56.915 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:56.901 7f01c6695700 1 -- 172.21.2.102:0/2537281870 shutdown_connections 2021-02-10T11:07:56.915 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:56.901 7f01c6695700 1 --2- 172.21.2.102:0/2537281870 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f01c00f20e0 0x7f01c00f0200 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:07:56.915 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:56.901 7f01c6695700 1 --2- 172.21.2.102:0/2537281870 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f01c01b0470 0x7f01c01b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:07:56.915 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:56.901 7f01c6695700 1 -- 172.21.2.102:0/2537281870 >> 172.21.2.102:0/2537281870 conn(0x7f01c00ce3e0 msgr2=0x7f01c00e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:07:56.915 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:56.901 7f01c6695700 1 -- 172.21.2.102:0/2537281870 shutdown_connections 2021-02-10T11:07:56.915 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:56.901 7f01c6695700 1 -- 172.21.2.102:0/2537281870 wait complete. 2021-02-10T11:07:56.916 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:56.916 7f01c6695700 1 Processor -- start 2021-02-10T11:07:56.917 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:56.916 7f01c6695700 1 -- start start 2021-02-10T11:07:56.917 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:56.916 7f01c6695700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f01c0192ad0 0x7f01c0192ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:07:56.917 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:56.916 7f01c6695700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f01c00a78f0 0x7f01c00a7cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:07:56.917 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:56.916 7f01c6695700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f01c00a8310 0x7f01c00a9060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:07:56.917 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:56.916 7f01c6695700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01c00a8ae0 con 0x7f01c0192ad0 2021-02-10T11:07:56.917 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:56.916 7f01c6695700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01c00a8710 con 0x7f01c00a78f0 2021-02-10T11:07:56.918 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:56.916 7f01c6695700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01c00a88f0 con 0x7f01c00a8310 2021-02-10T11:07:56.918 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:56.916 7f01bffff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f01c0192ad0 0x7f01c0192ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:07:56.918 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:56.916 7f01bffff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f01c0192ad0 0x7f01c0192ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:51382/0 (socket says 172.21.2.102:51382) 2021-02-10T11:07:56.918 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:56.916 7f01bffff700 1 -- 172.21.2.102:0/2797080064 learned_addr learned my addr 172.21.2.102:0/2797080064 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:07:56.918 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:56.916 7f01bf7fe700 1 --2- 172.21.2.102:0/2797080064 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f01c00a78f0 0x7f01c00a7cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:07:56.918 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:56.916 7f01c4cb2700 1 --2- 172.21.2.102:0/2797080064 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f01c00a8310 0x7f01c00a9060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:07:56.918 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:56.917 7f01bffff700 1 -- 172.21.2.102:0/2797080064 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f01c00a8310 msgr2=0x7f01c00a9060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:07:56.918 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:56.917 7f01bffff700 1 --2- 172.21.2.102:0/2797080064 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f01c00a8310 0x7f01c00a9060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:07:56.918 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:56.917 7f01bffff700 1 -- 172.21.2.102:0/2797080064 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f01c00a78f0 msgr2=0x7f01c00a7cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:07:56.918 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:56.917 7f01bffff700 1 --2- 172.21.2.102:0/2797080064 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f01c00a78f0 0x7f01c00a7cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:07:56.918 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:56.917 7f01bffff700 1 -- 172.21.2.102:0/2797080064 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01c00a9680 con 0x7f01c0192ad0 2021-02-10T11:07:56.918 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:56.917 7f01bf7fe700 1 --2- 172.21.2.102:0/2797080064 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f01c00a78f0 0x7f01c00a7cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:07:56.919 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:56.917 7f01bffff700 1 --2- 172.21.2.102:0/2797080064 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f01c0192ad0 0x7f01c0192ed0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 rx=0x7f01b0006f60 tx=0x7f01b000b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:07:56.919 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:56.917 7f01bcff9700 1 -- 172.21.2.102:0/2797080064 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f01b000bd60 con 0x7f01c0192ad0 2021-02-10T11:07:56.919 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:56.917 7f01bcff9700 1 -- 172.21.2.102:0/2797080064 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01b0000c90 con 0x7f01c0192ad0 2021-02-10T11:07:56.919 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:56.917 7f01c6695700 1 -- 172.21.2.102:0/2797080064 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01c00a9680 con 0x7f01c0192ad0 2021-02-10T11:07:56.919 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:56.917 7f01bcff9700 1 -- 172.21.2.102:0/2797080064 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f01b0002220 con 0x7f01c0192ad0 2021-02-10T11:07:56.919 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:56.917 7f01c6695700 1 -- 172.21.2.102:0/2797080064 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01c00a9c10 con 0x7f01c0192ad0 2021-02-10T11:07:56.919 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:56.917 7f01bcff9700 1 -- 172.21.2.102:0/2797080064 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f01b00052e0 con 0x7f01c0192ad0 2021-02-10T11:07:56.919 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:56.917 7f01bcff9700 1 --2- 172.21.2.102:0/2797080064 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f01a00186c0 0x7f01a001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:07:56.919 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:56.918 7f01bcff9700 1 -- 172.21.2.102:0/2797080064 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f01b0032410 con 0x7f01c0192ad0 2021-02-10T11:07:56.919 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:56.918 7f01bf7fe700 1 --2- 172.21.2.102:0/2797080064 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f01a00186c0 0x7f01a001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:07:56.919 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:56.918 7f01bf7fe700 1 --2- 172.21.2.102:0/2797080064 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f01a00186c0 0x7f01a001abc0 crc :-1 s=READY pgs=205 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:07:56.946 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:56.949 7f01c6695700 1 -- 172.21.2.102:0/2797080064 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01c00f0b50 con 0x7f01c0192ad0 2021-02-10T11:07:56.949 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:56.951 7f01bcff9700 1 -- 172.21.2.102:0/2797080064 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f01b0000960 con 0x7f01c0192ad0 2021-02-10T11:07:57.077 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:57.079 7f01c6695700 1 -- 172.21.2.102:0/2797080064 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344380", "target": ["mgr", ""]}) v1 -- 0x7f01c00f0b50 con 0x7f01a00186c0 2021-02-10T11:07:57.079 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:57.082 7f01bcff9700 1 -- 172.21.2.102:0/2797080064 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f01b40009b0 con 0x7f01a00186c0 2021-02-10T11:07:57.080 INFO:teuthology.orchestra.run.gibba002.stdout:/volumes/_nogroup/subvolume_0000000000344380/9e24e5b8-5ea2-46c8-8a36-37bc9e0f1bed 2021-02-10T11:07:57.082 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:57.083 7f01c6695700 1 -- 172.21.2.102:0/2797080064 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f01a00186c0 msgr2=0x7f01a001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:07:57.082 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:57.083 7f01c6695700 1 --2- 172.21.2.102:0/2797080064 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f01a00186c0 0x7f01a001abc0 crc :-1 s=READY pgs=205 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:07:57.082 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:57.083 7f01c6695700 1 -- 172.21.2.102:0/2797080064 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f01c0192ad0 msgr2=0x7f01c0192ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:07:57.082 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:57.083 7f01c6695700 1 --2- 172.21.2.102:0/2797080064 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f01c0192ad0 0x7f01c0192ed0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 rx=0x7f01b0006f60 tx=0x7f01b000b970).stop 2021-02-10T11:07:57.082 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:57.083 7f01c6695700 1 -- 172.21.2.102:0/2797080064 shutdown_connections 2021-02-10T11:07:57.082 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:57.083 7f01c6695700 1 --2- 172.21.2.102:0/2797080064 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f01c00a8310 0x7f01c00a9060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:07:57.082 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:57.083 7f01c6695700 1 --2- 172.21.2.102:0/2797080064 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f01a00186c0 0x7f01a001abc0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:07:57.083 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:57.083 7f01c6695700 1 --2- 172.21.2.102:0/2797080064 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f01c00a78f0 0x7f01c00a7cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:07:57.083 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:57.083 7f01c6695700 1 --2- 172.21.2.102:0/2797080064 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f01c0192ad0 0x7f01c0192ed0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:07:57.083 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:57.083 7f01c6695700 1 -- 172.21.2.102:0/2797080064 >> 172.21.2.102:0/2797080064 conn(0x7f01c00bf4f0 msgr2=0x7f01c00be250 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:07:57.083 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:57.084 7f01c6695700 1 -- 172.21.2.102:0/2797080064 shutdown_connections 2021-02-10T11:07:57.083 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:07:57.084 7f01c6695700 1 -- 172.21.2.102:0/2797080064 wait complete. 2021-02-10T11:07:57.090 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000344380 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000344380/9e24e5b8-5ea2-46c8-8a36-37bc9e0f1bed 2021-02-10T11:07:57.090 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:07:57.090 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:07:57.090 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:07:57.428 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:07:57.429 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:07:57.429 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344380/9e24e5b8-5ea2-46c8-8a36-37bc9e0f1bed/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:07:57.812 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:07:57.813 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:07:57.813 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0254656 s, 41.2 MB/s 2021-02-10T11:07:57.814 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:07:57.815 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:07:57.815 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:07:58.151 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:07:58.151 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:07:58.152 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344380/9e24e5b8-5ea2-46c8-8a36-37bc9e0f1bed/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:07:58.524 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:07:58.524 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:07:58.524 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0238289 s, 44.0 MB/s 2021-02-10T11:07:58.525 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:07:58.525 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:07:58.525 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:07:58.861 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:07:58.861 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:07:58.861 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344380/9e24e5b8-5ea2-46c8-8a36-37bc9e0f1bed/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:07:59.237 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:07:59.238 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:07:59.238 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0242006 s, 43.3 MB/s 2021-02-10T11:07:59.239 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:07:59.240 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:07:59.240 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:07:59.574 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:07:59.574 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:07:59.575 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344380/9e24e5b8-5ea2-46c8-8a36-37bc9e0f1bed/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:07:59.951 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:07:59.951 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:07:59.951 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0240756 s, 43.6 MB/s 2021-02-10T11:07:59.952 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:07:59.952 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:07:59.952 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:08:00.289 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:08:00.289 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:08:00.289 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344380/9e24e5b8-5ea2-46c8-8a36-37bc9e0f1bed/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:08:00.665 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:08:00.665 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:08:00.665 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0255501 s, 41.0 MB/s 2021-02-10T11:08:00.667 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:08:00.667 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:08:00.667 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:08:01.003 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:08:01.004 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:08:01.004 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344380/9e24e5b8-5ea2-46c8-8a36-37bc9e0f1bed/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:08:01.379 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:08:01.379 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:08:01.379 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0245585 s, 42.7 MB/s 2021-02-10T11:08:01.380 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:08:01.380 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:08:01.380 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:08:01.715 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:08:01.716 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:08:01.716 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344380/9e24e5b8-5ea2-46c8-8a36-37bc9e0f1bed/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:08:02.089 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:08:02.090 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:08:02.090 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0233018 s, 45.0 MB/s 2021-02-10T11:08:02.090 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:08:02.091 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:08:02.091 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:08:02.426 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:08:02.426 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:08:02.427 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344380/9e24e5b8-5ea2-46c8-8a36-37bc9e0f1bed/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:08:02.802 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:08:02.802 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:08:02.803 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0246872 s, 42.5 MB/s 2021-02-10T11:08:02.803 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:08:02.804 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:08:02.804 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:08:03.138 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:08:03.138 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:08:03.138 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344380/9e24e5b8-5ea2-46c8-8a36-37bc9e0f1bed/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:08:03.509 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:08:03.510 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:08:03.510 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0233773 s, 44.9 MB/s 2021-02-10T11:08:03.511 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:08:03.511 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:08:03.511 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:08:03.848 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:08:03.848 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:08:03.849 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344380/9e24e5b8-5ea2-46c8-8a36-37bc9e0f1bed/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:08:04.225 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:08:04.225 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:08:04.225 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0258483 s, 40.6 MB/s 2021-02-10T11:08:04.226 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000344381 2021-02-10T11:08:04.657 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:04.644 7f5d17ffa700 1 Processor -- start 2021-02-10T11:08:04.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:04.644 7f5d17ffa700 1 -- start start 2021-02-10T11:08:04.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:04.644 7f5d17ffa700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f5d101b0870 0x7f5d100f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:08:04.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:04.644 7f5d17ffa700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5d100efe00 0x7f5d100f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:08:04.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:04.644 7f5d17ffa700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d101b0c70 con 0x7f5d101b0470 2021-02-10T11:08:04.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:04.644 7f5d17ffa700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d100f0b50 con 0x7f5d101b0870 2021-02-10T11:08:04.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:04.644 7f5d17ffa700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d100f0820 con 0x7f5d100efe00 2021-02-10T11:08:04.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:04.644 7f5d15615700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f5d101b0870 0x7f5d100f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:08:04.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:04.644 7f5d15e16700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f5d101b0470 0x7f5d100ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:43614/0 (socket says 172.21.2.102:43614) 2021-02-10T11:08:04.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:04.645 7f5d15e16700 1 -- 172.21.2.102:0/349238854 learned_addr learned my addr 172.21.2.102:0/349238854 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:08:04.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:04.645 7f5d16617700 1 --2- 172.21.2.102:0/349238854 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5d100efe00 0x7f5d100f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:08:04.659 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:04.645 7f5d15615700 1 -- 172.21.2.102:0/349238854 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5d100efe00 msgr2=0x7f5d100f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:08:04.659 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:04.645 7f5d15615700 1 --2- 172.21.2.102:0/349238854 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5d100efe00 0x7f5d100f0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:08:04.659 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:04.645 7f5d15615700 1 -- 172.21.2.102:0/349238854 >> v1:172.21.2.102:6789/0 conn(0x7f5d101b0470 legacy=0x7f5d100ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:08:04.659 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:04.645 7f5d15615700 1 -- 172.21.2.102:0/349238854 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d100e52e0 con 0x7f5d101b0870 2021-02-10T11:08:04.659 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:04.645 7f5d14e14700 1 -- 172.21.2.102:0/349238854 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3482807699 0 0) 0x7f5d0c000ce0 con 0x7f5d101b0470 2021-02-10T11:08:04.659 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:04.645 7f5d15615700 1 --2- 172.21.2.102:0/349238854 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5d101b0870 0x7f5d100f20e0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 rx=0x7f5d04006650 tx=0x7f5d04009a00).ready entity=mon.0 client_cookie=f926c22c5a94b811 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:08:04.659 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:04.645 7f5d14e14700 1 -- 172.21.2.102:0/349238854 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5d04009df0 con 0x7f5d101b0870 2021-02-10T11:08:04.659 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:04.645 7f5d14e14700 1 -- 172.21.2.102:0/349238854 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d040009b0 con 0x7f5d101b0870 2021-02-10T11:08:04.659 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:04.645 7f5d14e14700 1 -- 172.21.2.102:0/349238854 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5d0400c2b0 con 0x7f5d101b0870 2021-02-10T11:08:04.659 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:04.645 7f5d17ffa700 1 -- 172.21.2.102:0/349238854 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5d101b0870 msgr2=0x7f5d100f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:08:04.659 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:04.645 7f5d17ffa700 1 --2- 172.21.2.102:0/349238854 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5d101b0870 0x7f5d100f20e0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 rx=0x7f5d04006650 tx=0x7f5d04009a00).stop 2021-02-10T11:08:04.659 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:04.646 7f5d17ffa700 1 -- 172.21.2.102:0/349238854 shutdown_connections 2021-02-10T11:08:04.660 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:04.646 7f5d17ffa700 1 --2- 172.21.2.102:0/349238854 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5d101b0870 0x7f5d100f20e0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:08:04.660 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:04.646 7f5d17ffa700 1 --2- 172.21.2.102:0/349238854 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5d100efe00 0x7f5d100f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:08:04.660 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:04.646 7f5d17ffa700 1 -- 172.21.2.102:0/349238854 >> 172.21.2.102:0/349238854 conn(0x7f5d100ce3e0 msgr2=0x7f5d100e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:08:04.660 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:04.646 7f5d17ffa700 1 -- 172.21.2.102:0/349238854 shutdown_connections 2021-02-10T11:08:04.660 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:04.646 7f5d17ffa700 1 -- 172.21.2.102:0/349238854 wait complete. 2021-02-10T11:08:04.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:04.660 7f5d17ffa700 1 Processor -- start 2021-02-10T11:08:04.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:04.661 7f5d17ffa700 1 -- start start 2021-02-10T11:08:04.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:04.661 7f5d17ffa700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5d1008ece0 0x7f5d100a6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:08:04.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:04.661 7f5d17ffa700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5d100a6860 0x7f5d100a6c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:08:04.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:04.661 7f5d17ffa700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5d100a7280 0x7f5d100a4590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:08:04.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:04.661 7f5d17ffa700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d100a5cc0 con 0x7f5d100a6860 2021-02-10T11:08:04.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:04.661 7f5d17ffa700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d1008f0e0 con 0x7f5d100a7280 2021-02-10T11:08:04.663 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:04.661 7f5d17ffa700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d1008f2c0 con 0x7f5d1008ece0 2021-02-10T11:08:04.663 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:04.661 7f5d15615700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5d100a6860 0x7f5d100a6c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:08:04.663 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:04.661 7f5d16617700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5d100a7280 0x7f5d100a4590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:08:04.663 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:04.661 7f5d15615700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5d100a6860 0x7f5d100a6c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:51396/0 (socket says 172.21.2.102:51396) 2021-02-10T11:08:04.663 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:04.661 7f5d15615700 1 -- 172.21.2.102:0/2145865037 learned_addr learned my addr 172.21.2.102:0/2145865037 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:08:04.663 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:04.661 7f5d15e16700 1 --2- 172.21.2.102:0/2145865037 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5d1008ece0 0x7f5d100a6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:08:04.663 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:04.662 7f5d16617700 1 -- 172.21.2.102:0/2145865037 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5d1008ece0 msgr2=0x7f5d100a6240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:08:04.663 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:04.662 7f5d16617700 1 --2- 172.21.2.102:0/2145865037 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5d1008ece0 0x7f5d100a6240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:08:04.663 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:04.662 7f5d16617700 1 -- 172.21.2.102:0/2145865037 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5d100a6860 msgr2=0x7f5d100a6c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:08:04.663 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:04.662 7f5d16617700 1 --2- 172.21.2.102:0/2145865037 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5d100a6860 0x7f5d100a6c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:08:04.663 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:04.662 7f5d16617700 1 -- 172.21.2.102:0/2145865037 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d100a4bb0 con 0x7f5d100a7280 2021-02-10T11:08:04.663 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:04.662 7f5d15e16700 1 --2- 172.21.2.102:0/2145865037 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5d1008ece0 0x7f5d100a6240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:08:04.663 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:04.662 7f5d16617700 1 --2- 172.21.2.102:0/2145865037 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5d100a7280 0x7f5d100a4590 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7f5d0800b8a0 tx=0x7f5d0800bc40).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:08:04.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:04.662 7f5d027fc700 1 -- 172.21.2.102:0/2145865037 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5d0800bd80 con 0x7f5d100a7280 2021-02-10T11:08:04.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:04.662 7f5d027fc700 1 -- 172.21.2.102:0/2145865037 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d08003030 con 0x7f5d100a7280 2021-02-10T11:08:04.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:04.662 7f5d17ffa700 1 -- 172.21.2.102:0/2145865037 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d100a4bb0 con 0x7f5d100a7280 2021-02-10T11:08:04.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:04.662 7f5d027fc700 1 -- 172.21.2.102:0/2145865037 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5d0800bd80 con 0x7f5d100a7280 2021-02-10T11:08:04.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:04.662 7f5d17ffa700 1 -- 172.21.2.102:0/2145865037 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d100a4bb0 con 0x7f5d100a7280 2021-02-10T11:08:04.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:04.663 7f5d027fc700 1 -- 172.21.2.102:0/2145865037 <== mon.1 v2:172.21.2.113:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f5d080056b0 con 0x7f5d100a7280 2021-02-10T11:08:04.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:04.663 7f5d027fc700 1 --2- 172.21.2.102:0/2145865037 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5cf8018730 0x7f5cf801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:08:04.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:04.663 7f5d027fc700 1 -- 172.21.2.102:0/2145865037 <== mon.1 v2:172.21.2.113:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f5d080328e0 con 0x7f5d100a7280 2021-02-10T11:08:04.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:04.663 7f5d15e16700 1 --2- 172.21.2.102:0/2145865037 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5cf8018730 0x7f5cf801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:08:04.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:04.663 7f5d15e16700 1 --2- 172.21.2.102:0/2145865037 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5cf8018730 0x7f5cf801ac10 crc :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:08:04.692 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:04.694 7f5d17ffa700 1 -- 172.21.2.102:0/2145865037 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d1019d1a0 con 0x7f5d100a7280 2021-02-10T11:08:04.695 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:04.699 7f5d027fc700 1 -- 172.21.2.102:0/2145865037 <== mon.1 v2:172.21.2.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f5d08006710 con 0x7f5d100a7280 2021-02-10T11:08:04.825 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:04.827 7f5d17ffa700 1 -- 172.21.2.102:0/2145865037 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344381", "target": ["mgr", ""]}) v1 -- 0x7f5d1019d1a0 con 0x7f5cf8018730 2021-02-10T11:08:04.852 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:04.855 7f5d027fc700 1 -- 172.21.2.102:0/2145865037 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f5d0c006bc0 con 0x7f5cf8018730 2021-02-10T11:08:04.859 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:04.861 7f5d17ffa700 1 -- 172.21.2.102:0/2145865037 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5cf8018730 msgr2=0x7f5cf801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:08:04.859 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:04.861 7f5d17ffa700 1 --2- 172.21.2.102:0/2145865037 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5cf8018730 0x7f5cf801ac10 crc :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:08:04.859 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:04.861 7f5d17ffa700 1 -- 172.21.2.102:0/2145865037 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5d100a7280 msgr2=0x7f5d100a4590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:08:04.859 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:04.861 7f5d17ffa700 1 --2- 172.21.2.102:0/2145865037 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5d100a7280 0x7f5d100a4590 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7f5d0800b8a0 tx=0x7f5d0800bc40).stop 2021-02-10T11:08:04.859 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:04.861 7f5d17ffa700 1 -- 172.21.2.102:0/2145865037 shutdown_connections 2021-02-10T11:08:04.859 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:04.861 7f5d17ffa700 1 --2- 172.21.2.102:0/2145865037 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5cf8018730 0x7f5cf801ac10 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:08:04.859 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:04.861 7f5d17ffa700 1 --2- 172.21.2.102:0/2145865037 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5d100a7280 0x7f5d100a4590 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:08:04.859 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:04.861 7f5d17ffa700 1 --2- 172.21.2.102:0/2145865037 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5d100a6860 0x7f5d100a6c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:08:04.860 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:04.861 7f5d17ffa700 1 --2- 172.21.2.102:0/2145865037 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5d1008ece0 0x7f5d100a6240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:08:04.860 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:04.861 7f5d17ffa700 1 -- 172.21.2.102:0/2145865037 >> 172.21.2.102:0/2145865037 conn(0x7f5d1008ffe0 msgr2=0x7f5d100bf050 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:08:04.860 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:04.861 7f5d17ffa700 1 -- 172.21.2.102:0/2145865037 shutdown_connections 2021-02-10T11:08:04.860 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:04.861 7f5d17ffa700 1 -- 172.21.2.102:0/2145865037 wait complete. 2021-02-10T11:08:04.867 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000344381 2021-02-10T11:08:05.297 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:05.283 7f7d9c7b8700 1 Processor -- start 2021-02-10T11:08:05.297 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:05.283 7f7d9c7b8700 1 -- start start 2021-02-10T11:08:05.297 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:05.283 7f7d9c7b8700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7d941b0870 0x7f7d940f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:08:05.297 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:05.283 7f7d9c7b8700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f7d940efe00 0x7f7d940f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:08:05.297 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:05.283 7f7d9c7b8700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d941b0c70 con 0x7f7d941b0470 2021-02-10T11:08:05.298 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:05.283 7f7d9c7b8700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d940f0b50 con 0x7f7d940efe00 2021-02-10T11:08:05.298 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:05.283 7f7d9c7b8700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d940f0820 con 0x7f7d941b0870 2021-02-10T11:08:05.298 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:05.284 7f7d9add5700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f7d940efe00 0x7f7d940f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:08:05.298 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:05.284 7f7d9add5700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f7d940efe00 0x7f7d940f0200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:51402/0 (socket says 172.21.2.102:51402) 2021-02-10T11:08:05.298 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:05.284 7f7d9add5700 1 -- 172.21.2.102:0/746148734 learned_addr learned my addr 172.21.2.102:0/746148734 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:08:05.298 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:05.284 7f7d99dd3700 1 --2- 172.21.2.102:0/746148734 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7d941b0870 0x7f7d940f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:08:05.298 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:05.284 7f7d9add5700 1 -- 172.21.2.102:0/746148734 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7d941b0870 msgr2=0x7f7d940f20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:08:05.299 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:05.284 7f7d9add5700 1 --2- 172.21.2.102:0/746148734 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7d941b0870 0x7f7d940f20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:08:05.299 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:05.284 7f7d9add5700 1 -- 172.21.2.102:0/746148734 >> v1:172.21.2.113:6789/0 conn(0x7f7d941b0470 legacy=0x7f7d940ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:08:05.299 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:05.284 7f7d9add5700 1 -- 172.21.2.102:0/746148734 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d940e52e0 con 0x7f7d940efe00 2021-02-10T11:08:05.299 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:05.284 7f7d99dd3700 1 --2- 172.21.2.102:0/746148734 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7d941b0870 0x7f7d940f20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:08:05.299 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:05.284 7f7d9add5700 1 --2- 172.21.2.102:0/746148734 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7d940efe00 0x7f7d940f0200 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 rx=0x7f7d8c006f00 tx=0x7f7d8c009f10).ready entity=mon.0 client_cookie=6d5a8a8a7112a3f3 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:08:05.299 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:05.284 7f7d995d2700 1 -- 172.21.2.102:0/746148734 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7d8c00b3b0 con 0x7f7d940efe00 2021-02-10T11:08:05.299 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:05.284 7f7d995d2700 1 -- 172.21.2.102:0/746148734 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d8c00b6b0 con 0x7f7d940efe00 2021-02-10T11:08:05.300 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:05.284 7f7d995d2700 1 -- 172.21.2.102:0/746148734 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7d8c00bb90 con 0x7f7d940efe00 2021-02-10T11:08:05.300 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:05.284 7f7d9c7b8700 1 -- 172.21.2.102:0/746148734 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7d940efe00 msgr2=0x7f7d940f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:08:05.300 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:05.284 7f7d9c7b8700 1 --2- 172.21.2.102:0/746148734 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7d940efe00 0x7f7d940f0200 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 rx=0x7f7d8c006f00 tx=0x7f7d8c009f10).stop 2021-02-10T11:08:05.300 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:05.285 7f7d9c7b8700 1 -- 172.21.2.102:0/746148734 shutdown_connections 2021-02-10T11:08:05.300 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:05.285 7f7d9c7b8700 1 --2- 172.21.2.102:0/746148734 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7d941b0870 0x7f7d940f20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:08:05.300 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:05.285 7f7d9c7b8700 1 --2- 172.21.2.102:0/746148734 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7d940efe00 0x7f7d940f0200 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:08:05.301 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:05.285 7f7d9c7b8700 1 -- 172.21.2.102:0/746148734 >> 172.21.2.102:0/746148734 conn(0x7f7d940ce3e0 msgr2=0x7f7d940e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:08:05.301 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:05.285 7f7d9c7b8700 1 -- 172.21.2.102:0/746148734 shutdown_connections 2021-02-10T11:08:05.301 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:05.285 7f7d9c7b8700 1 -- 172.21.2.102:0/746148734 wait complete. 2021-02-10T11:08:05.301 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:05.299 7f7d9c7b8700 1 Processor -- start 2021-02-10T11:08:05.301 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:05.299 7f7d9c7b8700 1 -- start start 2021-02-10T11:08:05.304 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:05.300 7f7d9c7b8700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7d940b20b0 0x7f7d940ac970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:08:05.304 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:05.300 7f7d9c7b8700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7d940b24b0 0x7f7d94191dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:08:05.304 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:05.300 7f7d9c7b8700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7d940acf90 0x7f7d941923e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:08:05.304 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:05.300 7f7d9c7b8700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d940b1b30 con 0x7f7d940b20b0 2021-02-10T11:08:05.304 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:05.300 7f7d9c7b8700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d941a04e0 con 0x7f7d940acf90 2021-02-10T11:08:05.304 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:05.300 7f7d9c7b8700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d941a06c0 con 0x7f7d940b24b0 2021-02-10T11:08:05.305 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:05.300 7f7d9a5d4700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7d940b20b0 0x7f7d940ac970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:08:05.305 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:05.300 7f7d9a5d4700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7d940b20b0 0x7f7d940ac970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:51406/0 (socket says 172.21.2.102:51406) 2021-02-10T11:08:05.305 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:05.300 7f7d9a5d4700 1 -- 172.21.2.102:0/2775988080 learned_addr learned my addr 172.21.2.102:0/2775988080 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:08:05.305 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:05.300 7f7d9add5700 1 --2- 172.21.2.102:0/2775988080 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7d940acf90 0x7f7d941923e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:08:05.305 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:05.300 7f7d99dd3700 1 --2- 172.21.2.102:0/2775988080 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7d940b24b0 0x7f7d94191dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:08:05.305 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:05.300 7f7d9a5d4700 1 -- 172.21.2.102:0/2775988080 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7d940b24b0 msgr2=0x7f7d94191dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:08:05.306 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:05.300 7f7d9a5d4700 1 --2- 172.21.2.102:0/2775988080 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7d940b24b0 0x7f7d94191dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:08:05.306 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:05.300 7f7d9a5d4700 1 -- 172.21.2.102:0/2775988080 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7d940acf90 msgr2=0x7f7d941923e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:08:05.306 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:05.300 7f7d9a5d4700 1 --2- 172.21.2.102:0/2775988080 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7d940acf90 0x7f7d941923e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:08:05.306 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:05.300 7f7d9a5d4700 1 -- 172.21.2.102:0/2775988080 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d94192be0 con 0x7f7d940b20b0 2021-02-10T11:08:05.306 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:05.300 7f7d9a5d4700 1 --2- 172.21.2.102:0/2775988080 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7d940b20b0 0x7f7d940ac970 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 rx=0x7f7d88006ed0 tx=0x7f7d8800b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:08:05.306 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:05.301 7f7d86ffd700 1 -- 172.21.2.102:0/2775988080 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7d88006240 con 0x7f7d940b20b0 2021-02-10T11:08:05.306 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:05.301 7f7d86ffd700 1 -- 172.21.2.102:0/2775988080 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d880054b0 con 0x7f7d940b20b0 2021-02-10T11:08:05.307 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:05.301 7f7d86ffd700 1 -- 172.21.2.102:0/2775988080 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7d88005990 con 0x7f7d940b20b0 2021-02-10T11:08:05.307 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:05.301 7f7d9c7b8700 1 -- 172.21.2.102:0/2775988080 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d94192be0 con 0x7f7d940b20b0 2021-02-10T11:08:05.307 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:05.301 7f7d9c7b8700 1 -- 172.21.2.102:0/2775988080 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d94193170 con 0x7f7d940b20b0 2021-02-10T11:08:05.307 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:05.301 7f7d86ffd700 1 -- 172.21.2.102:0/2775988080 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f7d88006030 con 0x7f7d940b20b0 2021-02-10T11:08:05.307 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:05.301 7f7d86ffd700 1 --2- 172.21.2.102:0/2775988080 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7d7c0186c0 0x7f7d7c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:08:05.307 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:05.301 7f7d86ffd700 1 -- 172.21.2.102:0/2775988080 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f7d880325b0 con 0x7f7d940b20b0 2021-02-10T11:08:05.307 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:05.303 7f7d99dd3700 1 --2- 172.21.2.102:0/2775988080 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7d7c0186c0 0x7f7d7c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:08:05.307 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:05.303 7f7d99dd3700 1 --2- 172.21.2.102:0/2775988080 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7d7c0186c0 0x7f7d7c01abc0 crc :-1 s=READY pgs=207 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:08:05.332 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:05.334 7f7d9c7b8700 1 -- 172.21.2.102:0/2775988080 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d94105af0 con 0x7f7d940b20b0 2021-02-10T11:08:05.334 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:05.336 7f7d86ffd700 1 -- 172.21.2.102:0/2775988080 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f7d88030d60 con 0x7f7d940b20b0 2021-02-10T11:08:05.464 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:05.465 7f7d9c7b8700 1 -- 172.21.2.102:0/2775988080 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344381", "target": ["mgr", ""]}) v1 -- 0x7f7d940b1b30 con 0x7f7d7c0186c0 2021-02-10T11:08:05.465 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:05.468 7f7d86ffd700 1 -- 172.21.2.102:0/2775988080 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f7d90000940 con 0x7f7d7c0186c0 2021-02-10T11:08:05.465 INFO:teuthology.orchestra.run.gibba002.stdout:/volumes/_nogroup/subvolume_0000000000344381/483a3076-1bdc-4022-a514-eae57ddc0218 2021-02-10T11:08:05.467 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:05.469 7f7d9c7b8700 1 -- 172.21.2.102:0/2775988080 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7d7c0186c0 msgr2=0x7f7d7c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:08:05.467 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:05.469 7f7d9c7b8700 1 --2- 172.21.2.102:0/2775988080 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7d7c0186c0 0x7f7d7c01abc0 crc :-1 s=READY pgs=207 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:08:05.467 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:05.469 7f7d9c7b8700 1 -- 172.21.2.102:0/2775988080 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7d940b20b0 msgr2=0x7f7d940ac970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:08:05.467 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:05.469 7f7d9c7b8700 1 --2- 172.21.2.102:0/2775988080 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7d940b20b0 0x7f7d940ac970 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 rx=0x7f7d88006ed0 tx=0x7f7d8800b960).stop 2021-02-10T11:08:05.467 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:05.469 7f7d9c7b8700 1 -- 172.21.2.102:0/2775988080 shutdown_connections 2021-02-10T11:08:05.467 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:05.469 7f7d9c7b8700 1 --2- 172.21.2.102:0/2775988080 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7d7c0186c0 0x7f7d7c01abc0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:08:05.467 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:05.469 7f7d9c7b8700 1 --2- 172.21.2.102:0/2775988080 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7d940acf90 0x7f7d941923e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:08:05.467 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:05.469 7f7d9c7b8700 1 --2- 172.21.2.102:0/2775988080 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7d940b24b0 0x7f7d94191dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:08:05.468 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:05.469 7f7d9c7b8700 1 --2- 172.21.2.102:0/2775988080 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7d940b20b0 0x7f7d940ac970 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:08:05.468 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:05.469 7f7d9c7b8700 1 -- 172.21.2.102:0/2775988080 >> 172.21.2.102:0/2775988080 conn(0x7f7d940bf4e0 msgr2=0x7f7d940be3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:08:05.468 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:05.469 7f7d9c7b8700 1 -- 172.21.2.102:0/2775988080 shutdown_connections 2021-02-10T11:08:05.468 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:05.469 7f7d9c7b8700 1 -- 172.21.2.102:0/2775988080 wait complete. 2021-02-10T11:08:05.475 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000344381 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000344381/483a3076-1bdc-4022-a514-eae57ddc0218 2021-02-10T11:08:05.475 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:08:05.475 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:08:05.475 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:08:05.810 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:08:05.810 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:08:05.810 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344381/483a3076-1bdc-4022-a514-eae57ddc0218/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:08:06.192 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:08:06.192 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:08:06.192 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0239777 s, 43.7 MB/s 2021-02-10T11:08:06.193 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:08:06.193 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:08:06.193 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:08:06.527 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:08:06.527 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:08:06.527 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344381/483a3076-1bdc-4022-a514-eae57ddc0218/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:08:06.901 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:08:06.901 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:08:06.902 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0253564 s, 41.4 MB/s 2021-02-10T11:08:06.902 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:08:06.903 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:08:06.903 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:08:07.238 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:08:07.238 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:08:07.239 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344381/483a3076-1bdc-4022-a514-eae57ddc0218/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:08:07.615 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:08:07.615 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:08:07.615 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0251453 s, 41.7 MB/s 2021-02-10T11:08:07.617 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:08:07.617 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:08:07.617 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:08:07.952 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:08:07.952 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:08:07.953 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344381/483a3076-1bdc-4022-a514-eae57ddc0218/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:08:08.328 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:08:08.328 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:08:08.328 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0235525 s, 44.5 MB/s 2021-02-10T11:08:08.329 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:08:08.329 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:08:08.329 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:08:08.664 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:08:08.664 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:08:08.664 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344381/483a3076-1bdc-4022-a514-eae57ddc0218/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:08:09.039 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:08:09.040 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:08:09.040 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0247629 s, 42.3 MB/s 2021-02-10T11:08:09.041 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:08:09.041 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:08:09.041 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:08:09.376 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:08:09.376 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:08:09.376 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344381/483a3076-1bdc-4022-a514-eae57ddc0218/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:08:09.752 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:08:09.752 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:08:09.752 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0259218 s, 40.5 MB/s 2021-02-10T11:08:09.753 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:08:09.753 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:08:09.753 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:08:10.089 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:08:10.090 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:08:10.090 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344381/483a3076-1bdc-4022-a514-eae57ddc0218/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:08:10.466 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:08:10.466 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:08:10.466 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0243913 s, 43.0 MB/s 2021-02-10T11:08:10.467 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:08:10.467 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:08:10.467 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:08:10.801 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:08:10.802 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:08:10.802 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344381/483a3076-1bdc-4022-a514-eae57ddc0218/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:08:11.176 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:08:11.176 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:08:11.176 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.023216 s, 45.2 MB/s 2021-02-10T11:08:11.177 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:08:11.177 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:08:11.177 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:08:11.511 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:08:11.511 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:08:11.511 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344381/483a3076-1bdc-4022-a514-eae57ddc0218/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:08:11.888 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:08:11.888 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:08:11.888 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0258779 s, 40.5 MB/s 2021-02-10T11:08:11.889 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:08:11.889 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:08:11.889 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:08:12.224 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:08:12.224 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:08:12.224 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344381/483a3076-1bdc-4022-a514-eae57ddc0218/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:08:12.601 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:08:12.825 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:08:12.825 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0264499 s, 39.6 MB/s 2021-02-10T11:08:12.827 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000344382 2021-02-10T11:08:13.257 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:13.244 7f495e7b5700 1 Processor -- start 2021-02-10T11:08:13.257 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:13.244 7f495e7b5700 1 -- start start 2021-02-10T11:08:13.257 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:13.244 7f495e7b5700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f49581b0470 0x7f49581b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:08:13.257 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:13.244 7f495e7b5700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f49580f20e0 0x7f49580f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:08:13.257 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:13.244 7f495e7b5700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49580f0b50 con 0x7f49580ef610 2021-02-10T11:08:13.257 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:13.244 7f495e7b5700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49580f0820 con 0x7f49581b0470 2021-02-10T11:08:13.257 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:13.244 7f495e7b5700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49580f24e0 con 0x7f49580f20e0 2021-02-10T11:08:13.257 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:13.244 7f4957fff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f49581b0470 0x7f49581b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:08:13.257 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:13.244 7f49577fe700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f49580ef610 0x7f49580efa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:43638/0 (socket says 172.21.2.102:43638) 2021-02-10T11:08:13.257 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:13.244 7f49577fe700 1 -- 172.21.2.102:0/687434382 learned_addr learned my addr 172.21.2.102:0/687434382 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:08:13.258 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:13.245 7f495cdd2700 1 --2- 172.21.2.102:0/687434382 >> v2:172.21.2.113:3300/0 conn(0x7f49580f20e0 0x7f49580f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:08:13.258 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:13.245 7f4956ffd700 1 -- 172.21.2.102:0/687434382 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 596671805 0 0) 0x7f494c0010b0 con 0x7f49580ef610 2021-02-10T11:08:13.258 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:13.245 7f4956ffd700 1 -- 172.21.2.102:0/687434382 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4940002fe0 con 0x7f49580ef610 2021-02-10T11:08:13.258 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:13.245 7f4957fff700 1 -- 172.21.2.102:0/687434382 >> v2:172.21.2.113:3300/0 conn(0x7f49580f20e0 msgr2=0x7f49580f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:08:13.258 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:13.245 7f4957fff700 1 --2- 172.21.2.102:0/687434382 >> v2:172.21.2.113:3300/0 conn(0x7f49580f20e0 0x7f49580f0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:08:13.258 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:13.245 7f4957fff700 1 -- 172.21.2.102:0/687434382 >> v1:172.21.2.102:6789/0 conn(0x7f49580ef610 legacy=0x7f49580efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:08:13.258 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:13.245 7f4957fff700 1 -- 172.21.2.102:0/687434382 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49580e52e0 con 0x7f49581b0470 2021-02-10T11:08:13.258 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:13.245 7f4957fff700 1 --2- 172.21.2.102:0/687434382 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f49581b0470 0x7f49581b0870 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 rx=0x7f4948006650 tx=0x7f4948009a80).ready entity=mon.0 client_cookie=f9a7152baa2574f0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:08:13.258 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:13.245 7f4956ffd700 1 -- 172.21.2.102:0/687434382 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f494800c0c0 con 0x7f49581b0470 2021-02-10T11:08:13.258 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:13.245 7f4956ffd700 1 -- 172.21.2.102:0/687434382 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f494800c340 con 0x7f49581b0470 2021-02-10T11:08:13.258 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:13.245 7f4956ffd700 1 -- 172.21.2.102:0/687434382 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f494800c860 con 0x7f49581b0470 2021-02-10T11:08:13.259 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:13.245 7f495e7b5700 1 -- 172.21.2.102:0/687434382 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f49581b0470 msgr2=0x7f49581b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:08:13.259 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:13.245 7f495e7b5700 1 --2- 172.21.2.102:0/687434382 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f49581b0470 0x7f49581b0870 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 rx=0x7f4948006650 tx=0x7f4948009a80).stop 2021-02-10T11:08:13.259 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:13.246 7f495e7b5700 1 -- 172.21.2.102:0/687434382 shutdown_connections 2021-02-10T11:08:13.259 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:13.246 7f495e7b5700 1 --2- 172.21.2.102:0/687434382 >> v2:172.21.2.113:3300/0 conn(0x7f49580f20e0 0x7f49580f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:08:13.259 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:13.246 7f495e7b5700 1 --2- 172.21.2.102:0/687434382 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f49581b0470 0x7f49581b0870 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:08:13.259 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:13.246 7f495e7b5700 1 -- 172.21.2.102:0/687434382 >> 172.21.2.102:0/687434382 conn(0x7f49580ce3e0 msgr2=0x7f49580e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:08:13.259 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:13.246 7f495e7b5700 1 -- 172.21.2.102:0/687434382 shutdown_connections 2021-02-10T11:08:13.259 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:13.246 7f495e7b5700 1 -- 172.21.2.102:0/687434382 wait complete. 2021-02-10T11:08:13.261 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:13.260 7f495e7b5700 1 Processor -- start 2021-02-10T11:08:13.261 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:13.261 7f495e7b5700 1 -- start start 2021-02-10T11:08:13.261 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:13.261 7f495e7b5700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f49580a0420 0x7f49580a0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:08:13.261 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:13.261 7f495e7b5700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4958189d00 0x7f495818a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:08:13.261 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:13.261 7f495e7b5700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f495818a720 0x7f49580b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:08:13.261 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:13.261 7f495e7b5700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4958189800 con 0x7f49580a0420 2021-02-10T11:08:13.261 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:13.261 7f495e7b5700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f495818ab20 con 0x7f4958189d00 2021-02-10T11:08:13.261 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:13.261 7f495e7b5700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49580a1140 con 0x7f495818a720 2021-02-10T11:08:13.261 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:13.261 7f4957fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f49580a0420 0x7f49580a0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:08:13.262 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:13.261 7f4957fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f49580a0420 0x7f49580a0820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:51422/0 (socket says 172.21.2.102:51422) 2021-02-10T11:08:13.262 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:13.261 7f4957fff700 1 -- 172.21.2.102:0/403389581 learned_addr learned my addr 172.21.2.102:0/403389581 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:08:13.262 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:13.261 7f49577fe700 1 --2- 172.21.2.102:0/403389581 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4958189d00 0x7f495818a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:08:13.262 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:13.262 7f495cdd2700 1 --2- 172.21.2.102:0/403389581 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f495818a720 0x7f49580b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:08:13.262 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:13.262 7f4957fff700 1 -- 172.21.2.102:0/403389581 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f495818a720 msgr2=0x7f49580b7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:08:13.262 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:13.262 7f4957fff700 1 --2- 172.21.2.102:0/403389581 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f495818a720 0x7f49580b7be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:08:13.262 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:13.262 7f4957fff700 1 -- 172.21.2.102:0/403389581 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4958189d00 msgr2=0x7f495818a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:08:13.262 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:13.262 7f4957fff700 1 --2- 172.21.2.102:0/403389581 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4958189d00 0x7f495818a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:08:13.262 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:13.262 7f4957fff700 1 -- 172.21.2.102:0/403389581 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49580b8200 con 0x7f49580a0420 2021-02-10T11:08:13.262 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:13.262 7f4957fff700 1 --2- 172.21.2.102:0/403389581 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f49580a0420 0x7f49580a0820 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 rx=0x7f4948009650 tx=0x7f494800b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:08:13.262 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:13.262 7f4954ff9700 1 -- 172.21.2.102:0/403389581 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f494800bd50 con 0x7f49580a0420 2021-02-10T11:08:13.262 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:13.262 7f4954ff9700 1 -- 172.21.2.102:0/403389581 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4948006030 con 0x7f49580a0420 2021-02-10T11:08:13.262 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:13.262 7f4954ff9700 1 -- 172.21.2.102:0/403389581 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f49480054d0 con 0x7f49580a0420 2021-02-10T11:08:13.263 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:13.262 7f495e7b5700 1 -- 172.21.2.102:0/403389581 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49580b8200 con 0x7f49580a0420 2021-02-10T11:08:13.263 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:13.263 7f495e7b5700 1 -- 172.21.2.102:0/403389581 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49580b8200 con 0x7f49580a0420 2021-02-10T11:08:13.263 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:13.263 7f4954ff9700 1 -- 172.21.2.102:0/403389581 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4948009300 con 0x7f49580a0420 2021-02-10T11:08:13.263 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:13.263 7f4954ff9700 1 --2- 172.21.2.102:0/403389581 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4938018730 0x7f493801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:08:13.263 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:13.263 7f4954ff9700 1 -- 172.21.2.102:0/403389581 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f4948032790 con 0x7f49580a0420 2021-02-10T11:08:13.263 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:13.263 7f49577fe700 1 --2- 172.21.2.102:0/403389581 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4938018730 0x7f493801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:08:13.263 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:13.264 7f49577fe700 1 --2- 172.21.2.102:0/403389581 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4938018730 0x7f493801abc0 crc :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:08:13.291 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:13.294 7f495e7b5700 1 -- 172.21.2.102:0/403389581 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4958105ae0 con 0x7f49580a0420 2021-02-10T11:08:13.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:13.297 7f4954ff9700 1 -- 172.21.2.102:0/403389581 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f4948000940 con 0x7f49580a0420 2021-02-10T11:08:13.419 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:13.423 7f495e7b5700 1 -- 172.21.2.102:0/403389581 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344382", "target": ["mgr", ""]}) v1 -- 0x7f495818ab20 con 0x7f4938018730 2021-02-10T11:08:13.442 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:13.446 7f4954ff9700 1 -- 172.21.2.102:0/403389581 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f494c0014b0 con 0x7f4938018730 2021-02-10T11:08:13.452 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:13.456 7f495e7b5700 1 -- 172.21.2.102:0/403389581 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4938018730 msgr2=0x7f493801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:08:13.452 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:13.456 7f495e7b5700 1 --2- 172.21.2.102:0/403389581 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4938018730 0x7f493801abc0 crc :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:08:13.453 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:13.456 7f495e7b5700 1 -- 172.21.2.102:0/403389581 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f49580a0420 msgr2=0x7f49580a0820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:08:13.453 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:13.456 7f495e7b5700 1 --2- 172.21.2.102:0/403389581 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f49580a0420 0x7f49580a0820 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 rx=0x7f4948009650 tx=0x7f494800b960).stop 2021-02-10T11:08:13.453 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:13.456 7f495e7b5700 1 -- 172.21.2.102:0/403389581 shutdown_connections 2021-02-10T11:08:13.453 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:13.456 7f495e7b5700 1 --2- 172.21.2.102:0/403389581 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f495818a720 0x7f49580b7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:08:13.453 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:13.456 7f495e7b5700 1 --2- 172.21.2.102:0/403389581 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4938018730 0x7f493801abc0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:08:13.453 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:13.456 7f495e7b5700 1 --2- 172.21.2.102:0/403389581 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4958189d00 0x7f495818a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:08:13.453 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:13.456 7f495e7b5700 1 --2- 172.21.2.102:0/403389581 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f49580a0420 0x7f49580a0820 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:08:13.453 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:13.456 7f495e7b5700 1 -- 172.21.2.102:0/403389581 >> 172.21.2.102:0/403389581 conn(0x7f4958094e10 msgr2=0x7f49580a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:08:13.454 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:13.456 7f495e7b5700 1 -- 172.21.2.102:0/403389581 shutdown_connections 2021-02-10T11:08:13.454 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:13.457 7f495e7b5700 1 -- 172.21.2.102:0/403389581 wait complete. 2021-02-10T11:08:13.460 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000344382 2021-02-10T11:08:13.891 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:13.877 7fc05a2f1700 1 Processor -- start 2021-02-10T11:08:13.891 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:13.877 7fc05a2f1700 1 -- start start 2021-02-10T11:08:13.892 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:13.878 7fc05a2f1700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc0541b0470 0x7fc0541b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:08:13.892 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:13.878 7fc05a2f1700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fc0540f20e0 0x7fc0540f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:08:13.892 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:13.878 7fc05a2f1700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0540f0b50 con 0x7fc0540ef610 2021-02-10T11:08:13.892 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:13.878 7fc05a2f1700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0540f0820 con 0x7fc0540f20e0 2021-02-10T11:08:13.892 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:13.878 7fc05a2f1700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0540f24e0 con 0x7fc0541b0470 2021-02-10T11:08:13.892 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:13.879 7fc05890e700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fc0540f20e0 0x7fc0540f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:08:13.892 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:13.879 7fc05890e700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fc0540f20e0 0x7fc0540f0200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.113:3300/0 says I am v2:172.21.2.102:54860/0 (socket says 172.21.2.102:54860) 2021-02-10T11:08:13.892 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:13.879 7fc05890e700 1 -- 172.21.2.102:0/3573293231 learned_addr learned my addr 172.21.2.102:0/3573293231 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:08:13.892 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:13.879 7fc053fff700 1 --2- 172.21.2.102:0/3573293231 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc0541b0470 0x7fc0541b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:08:13.892 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:13.879 7fc04bfff700 1 --1- 172.21.2.102:0/3573293231 >> v1:172.21.2.113:6789/0 conn(0x7fc0540ef610 0x7fc0540efa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.113:6789/0 says I am v1:172.21.2.102:50974/0 (socket says 172.21.2.102:50974) 2021-02-10T11:08:13.892 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:13.879 7fc05890e700 1 -- 172.21.2.102:0/3573293231 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc0541b0470 msgr2=0x7fc0541b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:08:13.892 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:13.879 7fc05890e700 1 --2- 172.21.2.102:0/3573293231 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc0541b0470 0x7fc0541b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:08:13.893 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:13.879 7fc05890e700 1 -- 172.21.2.102:0/3573293231 >> v1:172.21.2.113:6789/0 conn(0x7fc0540ef610 legacy=0x7fc0540efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:08:13.893 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:13.879 7fc05890e700 1 -- 172.21.2.102:0/3573293231 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0540e52e0 con 0x7fc0540f20e0 2021-02-10T11:08:13.893 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:13.879 7fc05890e700 1 --2- 172.21.2.102:0/3573293231 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc0540f20e0 0x7fc0540f0200 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7fc04c006aa0 tx=0x7fc04c009f20).ready entity=mon.1 client_cookie=a80a3ab74f035774 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:08:13.893 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:13.880 7fc0537fe700 1 -- 172.21.2.102:0/3573293231 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc04c00b5c0 con 0x7fc0540f20e0 2021-02-10T11:08:13.893 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:13.880 7fc0537fe700 1 -- 172.21.2.102:0/3573293231 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc04c003030 con 0x7fc0540f20e0 2021-02-10T11:08:13.893 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:13.880 7fc0537fe700 1 -- 172.21.2.102:0/3573293231 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc04c00b9f0 con 0x7fc0540f20e0 2021-02-10T11:08:13.893 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:13.880 7fc05a2f1700 1 -- 172.21.2.102:0/3573293231 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc0540f20e0 msgr2=0x7fc0540f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:08:13.893 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:13.880 7fc05a2f1700 1 --2- 172.21.2.102:0/3573293231 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc0540f20e0 0x7fc0540f0200 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7fc04c006aa0 tx=0x7fc04c009f20).stop 2021-02-10T11:08:13.893 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:13.881 7fc05a2f1700 1 -- 172.21.2.102:0/3573293231 shutdown_connections 2021-02-10T11:08:13.893 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:13.881 7fc05a2f1700 1 --2- 172.21.2.102:0/3573293231 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc0540f20e0 0x7fc0540f0200 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:08:13.893 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:13.881 7fc05a2f1700 1 --2- 172.21.2.102:0/3573293231 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc0541b0470 0x7fc0541b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:08:13.893 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:13.881 7fc05a2f1700 1 -- 172.21.2.102:0/3573293231 >> 172.21.2.102:0/3573293231 conn(0x7fc0540ce3e0 msgr2=0x7fc0540e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:08:13.894 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:13.881 7fc05a2f1700 1 -- 172.21.2.102:0/3573293231 shutdown_connections 2021-02-10T11:08:13.894 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:13.881 7fc05a2f1700 1 -- 172.21.2.102:0/3573293231 wait complete. 2021-02-10T11:08:13.895 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:13.895 7fc05a2f1700 1 Processor -- start 2021-02-10T11:08:13.895 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:13.895 7fc05a2f1700 1 -- start start 2021-02-10T11:08:13.895 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:13.896 7fc05a2f1700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc0540b7de0 0x7fc054190920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:08:13.895 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:13.896 7fc05a2f1700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc0540b82f0 0x7fc0540b86f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:08:13.895 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:13.896 7fc05a2f1700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc0540b8d10 0x7fc0540bf9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:08:13.896 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:13.896 7fc05a2f1700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0540b7930 con 0x7fc0540b8d10 2021-02-10T11:08:13.896 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:13.896 7fc05a2f1700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0540b7b20 con 0x7fc0540b7de0 2021-02-10T11:08:13.896 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:13.896 7fc05a2f1700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0540bffd0 con 0x7fc0540b82f0 2021-02-10T11:08:13.896 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:13.896 7fc05890e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc0540b8d10 0x7fc0540bf9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:08:13.896 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:13.896 7fc05890e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc0540b8d10 0x7fc0540bf9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:51436/0 (socket says 172.21.2.102:51436) 2021-02-10T11:08:13.896 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:13.896 7fc05890e700 1 -- 172.21.2.102:0/3807502393 learned_addr learned my addr 172.21.2.102:0/3807502393 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:08:13.896 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:13.896 7fc053fff700 1 --2- 172.21.2.102:0/3807502393 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc0540b7de0 0x7fc054190920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:08:13.896 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:13.896 7fc04bfff700 1 --2- 172.21.2.102:0/3807502393 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc0540b82f0 0x7fc0540b86f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:08:13.896 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:13.896 7fc05890e700 1 -- 172.21.2.102:0/3807502393 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc0540b82f0 msgr2=0x7fc0540b86f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:08:13.896 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:13.896 7fc05890e700 1 --2- 172.21.2.102:0/3807502393 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc0540b82f0 0x7fc0540b86f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:08:13.896 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:13.896 7fc05890e700 1 -- 172.21.2.102:0/3807502393 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc0540b7de0 msgr2=0x7fc054190920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:08:13.896 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:13.896 7fc05890e700 1 --2- 172.21.2.102:0/3807502393 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc0540b7de0 0x7fc054190920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:08:13.896 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:13.896 7fc05890e700 1 -- 172.21.2.102:0/3807502393 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0540c01c0 con 0x7fc0540b8d10 2021-02-10T11:08:13.897 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:13.896 7fc04bfff700 1 --2- 172.21.2.102:0/3807502393 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc0540b82f0 0x7fc0540b86f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:08:13.897 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:13.897 7fc05890e700 1 --2- 172.21.2.102:0/3807502393 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc0540b8d10 0x7fc0540bf9b0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 rx=0x7fc04c00b970 tx=0x7fc04c00bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:08:13.897 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:13.897 7fc0517fa700 1 -- 172.21.2.102:0/3807502393 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc04c009450 con 0x7fc0540b8d10 2021-02-10T11:08:13.897 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:13.897 7fc0517fa700 1 -- 172.21.2.102:0/3807502393 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc04c00c2f0 con 0x7fc0540b8d10 2021-02-10T11:08:13.897 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:13.897 7fc0517fa700 1 -- 172.21.2.102:0/3807502393 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc04c00c750 con 0x7fc0540b8d10 2021-02-10T11:08:13.897 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:13.897 7fc05a2f1700 1 -- 172.21.2.102:0/3807502393 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0540c01c0 con 0x7fc0540b8d10 2021-02-10T11:08:13.897 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:13.897 7fc05a2f1700 1 -- 172.21.2.102:0/3807502393 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0540c0750 con 0x7fc0540b8d10 2021-02-10T11:08:13.897 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:13.897 7fc0517fa700 1 -- 172.21.2.102:0/3807502393 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fc04c00a000 con 0x7fc0540b8d10 2021-02-10T11:08:13.897 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:13.897 7fc0517fa700 1 --2- 172.21.2.102:0/3807502393 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc0380186c0 0x7fc03801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:08:13.897 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:13.898 7fc0517fa700 1 -- 172.21.2.102:0/3807502393 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fc04c032950 con 0x7fc0540b8d10 2021-02-10T11:08:13.897 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:13.898 7fc053fff700 1 --2- 172.21.2.102:0/3807502393 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc0380186c0 0x7fc03801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:08:13.897 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:13.898 7fc053fff700 1 --2- 172.21.2.102:0/3807502393 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc0380186c0 0x7fc03801abc0 crc :-1 s=READY pgs=209 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:08:13.924 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:13.928 7fc05a2f1700 1 -- 172.21.2.102:0/3807502393 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc054105b30 con 0x7fc0540b8d10 2021-02-10T11:08:13.927 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:13.930 7fc0517fa700 1 -- 172.21.2.102:0/3807502393 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fc04c009450 con 0x7fc0540b8d10 2021-02-10T11:08:14.056 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:14.060 7fc05a2f1700 1 -- 172.21.2.102:0/3807502393 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344382", "target": ["mgr", ""]}) v1 -- 0x7fc0540c0750 con 0x7fc0380186c0 2021-02-10T11:08:14.059 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:14.063 7fc0517fa700 1 -- 172.21.2.102:0/3807502393 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7fc03c000960 con 0x7fc0380186c0 2021-02-10T11:08:14.059 INFO:teuthology.orchestra.run.gibba002.stdout:/volumes/_nogroup/subvolume_0000000000344382/fc9c8f04-6aa0-4fc2-9feb-496b1abf8388 2021-02-10T11:08:14.060 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:14.064 7fc05a2f1700 1 -- 172.21.2.102:0/3807502393 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc0380186c0 msgr2=0x7fc03801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:08:14.060 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:14.064 7fc05a2f1700 1 --2- 172.21.2.102:0/3807502393 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc0380186c0 0x7fc03801abc0 crc :-1 s=READY pgs=209 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:08:14.060 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:14.064 7fc05a2f1700 1 -- 172.21.2.102:0/3807502393 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc0540b8d10 msgr2=0x7fc0540bf9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:08:14.061 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:14.064 7fc05a2f1700 1 --2- 172.21.2.102:0/3807502393 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc0540b8d10 0x7fc0540bf9b0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 rx=0x7fc04c00b970 tx=0x7fc04c00bd10).stop 2021-02-10T11:08:14.061 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:14.064 7fc05a2f1700 1 -- 172.21.2.102:0/3807502393 shutdown_connections 2021-02-10T11:08:14.061 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:14.064 7fc05a2f1700 1 --2- 172.21.2.102:0/3807502393 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc0540b8d10 0x7fc0540bf9b0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:08:14.061 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:14.064 7fc05a2f1700 1 --2- 172.21.2.102:0/3807502393 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc0540b82f0 0x7fc0540b86f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:08:14.061 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:14.064 7fc05a2f1700 1 --2- 172.21.2.102:0/3807502393 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc0380186c0 0x7fc03801abc0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:08:14.061 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:14.064 7fc05a2f1700 1 --2- 172.21.2.102:0/3807502393 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc0540b7de0 0x7fc054190920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:08:14.061 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:14.064 7fc05a2f1700 1 -- 172.21.2.102:0/3807502393 >> 172.21.2.102:0/3807502393 conn(0x7fc054094e10 msgr2=0x7fc0540a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:08:14.062 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:14.064 7fc05a2f1700 1 -- 172.21.2.102:0/3807502393 shutdown_connections 2021-02-10T11:08:14.062 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:14.064 7fc05a2f1700 1 -- 172.21.2.102:0/3807502393 wait complete. 2021-02-10T11:08:14.069 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000344382 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000344382/fc9c8f04-6aa0-4fc2-9feb-496b1abf8388 2021-02-10T11:08:14.069 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:08:14.069 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:08:14.069 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:08:14.235 DEBUG:teuthology.orchestra.run.gibba002:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:08:14.238 DEBUG:teuthology.orchestra.run.gibba013:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:08:14.405 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:08:14.405 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:08:14.406 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344382/fc9c8f04-6aa0-4fc2-9feb-496b1abf8388/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:08:14.783 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:08:14.783 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:08:14.783 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0227877 s, 46.0 MB/s 2021-02-10T11:08:14.784 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:08:14.784 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:08:14.784 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:08:15.118 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:08:15.118 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:08:15.119 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344382/fc9c8f04-6aa0-4fc2-9feb-496b1abf8388/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:08:15.492 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:08:15.492 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:08:15.492 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0235971 s, 44.4 MB/s 2021-02-10T11:08:15.493 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:08:15.493 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:08:15.493 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:08:15.828 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:08:15.828 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:08:15.829 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344382/fc9c8f04-6aa0-4fc2-9feb-496b1abf8388/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:08:16.203 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:08:16.203 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:08:16.203 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0246334 s, 42.6 MB/s 2021-02-10T11:08:16.204 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:08:16.204 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:08:16.204 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:08:16.558 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:08:16.559 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:08:16.559 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344382/fc9c8f04-6aa0-4fc2-9feb-496b1abf8388/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:08:16.954 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:08:16.955 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:08:16.955 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0241346 s, 43.4 MB/s 2021-02-10T11:08:16.956 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:08:16.956 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:08:16.957 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:08:17.293 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:08:17.293 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:08:17.294 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344382/fc9c8f04-6aa0-4fc2-9feb-496b1abf8388/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:08:17.677 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:08:17.677 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:08:17.677 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0252239 s, 41.6 MB/s 2021-02-10T11:08:17.679 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:08:17.679 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:08:17.679 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:08:18.016 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:08:18.016 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:08:18.016 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344382/fc9c8f04-6aa0-4fc2-9feb-496b1abf8388/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:08:18.390 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:08:18.390 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:08:18.390 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0248708 s, 42.2 MB/s 2021-02-10T11:08:18.391 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:08:18.391 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:08:18.391 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:08:18.726 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:08:18.727 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:08:18.727 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344382/fc9c8f04-6aa0-4fc2-9feb-496b1abf8388/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:08:19.099 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:08:19.100 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:08:19.100 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0222886 s, 47.0 MB/s 2021-02-10T11:08:19.100 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:08:19.101 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:08:19.101 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:08:19.435 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:08:19.435 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:08:19.435 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344382/fc9c8f04-6aa0-4fc2-9feb-496b1abf8388/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:08:19.813 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:08:19.813 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:08:19.813 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0253068 s, 41.4 MB/s 2021-02-10T11:08:19.814 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:08:19.815 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:08:19.815 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:08:20.150 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:08:20.150 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:08:20.150 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344382/fc9c8f04-6aa0-4fc2-9feb-496b1abf8388/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:08:20.526 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:08:20.526 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:08:20.526 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0249535 s, 42.0 MB/s 2021-02-10T11:08:20.527 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:08:20.527 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:08:20.527 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:08:20.863 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:08:20.864 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:08:20.864 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344382/fc9c8f04-6aa0-4fc2-9feb-496b1abf8388/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:08:21.238 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:08:21.238 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:08:21.238 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0239089 s, 43.9 MB/s 2021-02-10T11:08:21.239 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000344383 2021-02-10T11:08:21.670 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:21.656 7f3aa882d700 1 Processor -- start 2021-02-10T11:08:21.670 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:21.656 7f3aa882d700 1 -- start start 2021-02-10T11:08:21.670 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:21.656 7f3aa882d700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3aa41b0470 0x7f3aa41b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:08:21.670 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:21.656 7f3aa882d700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3aa40f0b50 con 0x7f3aa40f20e0 2021-02-10T11:08:21.670 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:21.656 7f3aa882d700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3aa40f24e0 con 0x7f3aa40ef610 2021-02-10T11:08:21.670 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:21.656 7f3aa882d700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3aa40e5ea0 con 0x7f3aa41b0470 2021-02-10T11:08:21.671 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:21.657 7f3aa2e1d700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f3aa40f20e0 0x7f3aa40f0200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:43664/0 (socket says 172.21.2.102:43664) 2021-02-10T11:08:21.671 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:21.657 7f3aa2e1d700 1 -- 172.21.2.102:0/796021009 learned_addr learned my addr 172.21.2.102:0/796021009 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:08:21.671 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:21.657 7f3aa161a700 1 -- 172.21.2.102:0/796021009 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3793613123 0 0) 0x7f3a9c001500 con 0x7f3aa40f20e0 2021-02-10T11:08:21.671 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:21.657 7f3aa161a700 1 -- 172.21.2.102:0/796021009 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a90002fe0 con 0x7f3aa40f20e0 2021-02-10T11:08:21.671 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:21.657 7f3aa161a700 1 -- 172.21.2.102:0/796021009 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 330149941 0 0) 0x7f3a9c001460 con 0x7f3aa40f20e0 2021-02-10T11:08:21.671 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:21.657 7f3aa261c700 1 --2- 172.21.2.102:0/796021009 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3aa41b0470 0x7f3aa41b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:08:21.671 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:21.657 7f3aa161a700 1 -- 172.21.2.102:0/796021009 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3aa41b0470 msgr2=0x7f3aa41b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:08:21.671 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:21.657 7f3aa161a700 1 --2- 172.21.2.102:0/796021009 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3aa41b0470 0x7f3aa41b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:08:21.671 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:21.657 7f3aa161a700 1 -- 172.21.2.102:0/796021009 >> v1:172.21.2.113:6789/0 conn(0x7f3aa40ef610 legacy=0x7f3aa40efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:08:21.671 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:21.657 7f3aa161a700 1 -- 172.21.2.102:0/796021009 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3aa40e52e0 con 0x7f3aa40f20e0 2021-02-10T11:08:21.672 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:21.657 7f3aa161a700 1 -- 172.21.2.102:0/796021009 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7f3a9c001980 con 0x7f3aa40f20e0 2021-02-10T11:08:21.672 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:21.657 7f3aa261c700 1 --2- 172.21.2.102:0/796021009 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3aa41b0470 0x7f3aa41b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:08:21.672 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:21.657 7f3aa161a700 1 -- 172.21.2.102:0/796021009 >> v1:172.21.2.102:6789/0 conn(0x7f3aa40f20e0 legacy=0x7f3aa40f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:08:21.672 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:21.657 7f3aa161a700 1 --2- 172.21.2.102:0/796021009 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3a90003a60 0x7f3a90005ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:08:21.672 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:21.657 7f3aa161a700 1 --2- 172.21.2.102:0/796021009 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3a90006770 0x7f3a90008c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:08:21.672 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:21.657 7f3aa161a700 1 --2- 172.21.2.102:0/796021009 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3a90009500 0x7f3a9000ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:08:21.672 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:21.657 7f3aa161a700 1 -- 172.21.2.102:0/796021009 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a9000c2d0 con 0x7f3a90003a60 2021-02-10T11:08:21.672 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:21.657 7f3aa161a700 1 -- 172.21.2.102:0/796021009 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a9000c4f0 con 0x7f3a90009500 2021-02-10T11:08:21.672 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:21.657 7f3aa161a700 1 -- 172.21.2.102:0/796021009 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a9000c710 con 0x7f3a90006770 2021-02-10T11:08:21.672 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:21.658 7f3aa261c700 1 --2- 172.21.2.102:0/796021009 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3a90003a60 0x7f3a90005ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:08:21.672 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:21.658 7f3aa2e1d700 1 --2- 172.21.2.102:0/796021009 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3a90009500 0x7f3a9000ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:08:21.672 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:21.658 7f3aa1e1b700 1 --2- 172.21.2.102:0/796021009 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3a90006770 0x7f3a90008c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:08:21.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:21.658 7f3aa261c700 1 -- 172.21.2.102:0/796021009 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3a90006770 msgr2=0x7f3a90008c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:08:21.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:21.658 7f3aa261c700 1 --2- 172.21.2.102:0/796021009 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3a90006770 0x7f3a90008c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:08:21.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:21.658 7f3aa261c700 1 -- 172.21.2.102:0/796021009 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3a90009500 msgr2=0x7f3a9000ba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:08:21.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:21.658 7f3aa261c700 1 --2- 172.21.2.102:0/796021009 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3a90009500 0x7f3a9000ba20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:08:21.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:21.658 7f3aa261c700 1 -- 172.21.2.102:0/796021009 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a9000c9b0 con 0x7f3a90003a60 2021-02-10T11:08:21.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:21.658 7f3aa261c700 1 --2- 172.21.2.102:0/796021009 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3a90003a60 0x7f3a90005ef0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 rx=0x7f3a94006a90 tx=0x7f3a9404daa0).ready entity=mon.0 client_cookie=a005b2872c03615b server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:08:21.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:21.658 7f3aa161a700 1 -- 172.21.2.102:0/796021009 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3a94002030 con 0x7f3a90003a60 2021-02-10T11:08:21.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:21.658 7f3aa161a700 1 -- 172.21.2.102:0/796021009 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a94002270 con 0x7f3a90003a60 2021-02-10T11:08:21.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:21.658 7f3aa161a700 1 -- 172.21.2.102:0/796021009 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3a94002790 con 0x7f3a90003a60 2021-02-10T11:08:21.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:21.658 7f3aa882d700 1 -- 172.21.2.102:0/796021009 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3a90003a60 msgr2=0x7f3a90005ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:08:21.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:21.658 7f3aa882d700 1 --2- 172.21.2.102:0/796021009 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3a90003a60 0x7f3a90005ef0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 rx=0x7f3a94006a90 tx=0x7f3a9404daa0).stop 2021-02-10T11:08:21.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:21.658 7f3aa2e1d700 1 -- 172.21.2.102:0/796021009 reap_dead start 2021-02-10T11:08:21.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:21.659 7f3aa882d700 1 -- 172.21.2.102:0/796021009 shutdown_connections 2021-02-10T11:08:21.674 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:21.659 7f3aa882d700 1 -- 172.21.2.102:0/796021009 >> 172.21.2.102:0/796021009 conn(0x7f3aa40ce3e0 msgr2=0x7f3aa40e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:08:21.674 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:21.659 7f3aa882d700 1 -- 172.21.2.102:0/796021009 shutdown_connections 2021-02-10T11:08:21.674 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:21.659 7f3aa882d700 1 -- 172.21.2.102:0/796021009 wait complete. 2021-02-10T11:08:21.674 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:21.674 7f3aa882d700 1 Processor -- start 2021-02-10T11:08:21.674 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:21.674 7f3aa882d700 1 -- start start 2021-02-10T11:08:21.676 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:21.674 7f3aa882d700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3aa40b7db0 0x7f3aa40b8260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:08:21.676 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:21.674 7f3aa882d700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3aa40b8880 0x7f3aa41a7450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:08:21.676 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:21.674 7f3aa882d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3aa40b8c80 0x7f3aa41a7a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:08:21.676 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:21.674 7f3aa882d700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3aa40b79b0 con 0x7f3aa40b8c80 2021-02-10T11:08:21.676 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:21.674 7f3aa882d700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3aa41a8090 con 0x7f3aa40b8880 2021-02-10T11:08:21.676 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:21.674 7f3aa882d700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3aa41a82b0 con 0x7f3aa40b7db0 2021-02-10T11:08:21.676 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:21.674 7f3aa2e1d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3aa40b8c80 0x7f3aa41a7a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:08:21.676 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:21.674 7f3aa2e1d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3aa40b8c80 0x7f3aa41a7a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:51452/0 (socket says 172.21.2.102:51452) 2021-02-10T11:08:21.676 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:21.674 7f3aa2e1d700 1 -- 172.21.2.102:0/4211222749 learned_addr learned my addr 172.21.2.102:0/4211222749 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:08:21.677 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:21.675 7f3aa2e1d700 1 -- 172.21.2.102:0/4211222749 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3aa40b7db0 msgr2=0x7f3aa40b8260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:08:21.677 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:21.675 7f3aa2e1d700 1 --2- 172.21.2.102:0/4211222749 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3aa40b7db0 0x7f3aa40b8260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:08:21.677 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:21.675 7f3aa2e1d700 1 -- 172.21.2.102:0/4211222749 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3aa40b8880 msgr2=0x7f3aa41a7450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:08:21.677 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:21.675 7f3aa2e1d700 1 --2- 172.21.2.102:0/4211222749 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3aa40b8880 0x7f3aa41a7450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:08:21.677 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:21.675 7f3aa2e1d700 1 -- 172.21.2.102:0/4211222749 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3aa4190c10 con 0x7f3aa40b8c80 2021-02-10T11:08:21.677 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:21.675 7f3aa2e1d700 1 --2- 172.21.2.102:0/4211222749 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3aa40b8c80 0x7f3aa41a7a70 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 rx=0x7f3a9c00a8f0 tx=0x7f3a9c00ac90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:08:21.677 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:21.675 7f3a9affd700 1 -- 172.21.2.102:0/4211222749 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3a9c001e70 con 0x7f3aa40b8c80 2021-02-10T11:08:21.677 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:21.675 7f3a9affd700 1 -- 172.21.2.102:0/4211222749 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a9c004930 con 0x7f3aa40b8c80 2021-02-10T11:08:21.677 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:21.675 7f3a9affd700 1 -- 172.21.2.102:0/4211222749 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3a9c004d80 con 0x7f3aa40b8c80 2021-02-10T11:08:21.678 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:21.675 7f3aa882d700 1 -- 172.21.2.102:0/4211222749 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3aa4190c10 con 0x7f3aa40b8c80 2021-02-10T11:08:21.678 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:21.675 7f3aa882d700 1 -- 172.21.2.102:0/4211222749 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3aa40c1eb0 con 0x7f3aa40b8c80 2021-02-10T11:08:21.678 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:21.676 7f3a9affd700 1 -- 172.21.2.102:0/4211222749 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f3a9c001e50 con 0x7f3aa40b8c80 2021-02-10T11:08:21.678 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:21.676 7f3a9affd700 1 --2- 172.21.2.102:0/4211222749 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3a880186c0 0x7f3a8801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:08:21.678 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:21.676 7f3a9affd700 1 -- 172.21.2.102:0/4211222749 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f3a9c032770 con 0x7f3aa40b8c80 2021-02-10T11:08:21.678 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:21.676 7f3aa261c700 1 --2- 172.21.2.102:0/4211222749 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3a880186c0 0x7f3a8801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:08:21.678 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:21.676 7f3aa261c700 1 --2- 172.21.2.102:0/4211222749 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3a880186c0 0x7f3a8801abc0 crc :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:08:21.703 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:21.706 7f3aa882d700 1 -- 172.21.2.102:0/4211222749 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3aa4105b30 con 0x7f3aa40b8c80 2021-02-10T11:08:21.705 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:21.709 7f3a9affd700 1 -- 172.21.2.102:0/4211222749 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f3a9c007020 con 0x7f3aa40b8c80 2021-02-10T11:08:21.833 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:21.836 7f3aa882d700 1 -- 172.21.2.102:0/4211222749 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344383", "target": ["mgr", ""]}) v1 -- 0x7f3aa40c1eb0 con 0x7f3a880186c0 2021-02-10T11:08:21.859 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:21.862 7f3a9affd700 1 -- 172.21.2.102:0/4211222749 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f3a94000990 con 0x7f3a880186c0 2021-02-10T11:08:21.867 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:21.870 7f3aa882d700 1 -- 172.21.2.102:0/4211222749 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3a880186c0 msgr2=0x7f3a8801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:08:21.867 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:21.870 7f3aa882d700 1 --2- 172.21.2.102:0/4211222749 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3a880186c0 0x7f3a8801abc0 crc :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:08:21.867 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:21.870 7f3aa882d700 1 -- 172.21.2.102:0/4211222749 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3aa40b8c80 msgr2=0x7f3aa41a7a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:08:21.867 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:21.870 7f3aa882d700 1 --2- 172.21.2.102:0/4211222749 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3aa40b8c80 0x7f3aa41a7a70 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 rx=0x7f3a9c00a8f0 tx=0x7f3a9c00ac90).stop 2021-02-10T11:08:21.867 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:21.870 7f3aa882d700 1 -- 172.21.2.102:0/4211222749 shutdown_connections 2021-02-10T11:08:21.867 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:21.870 7f3aa882d700 1 --2- 172.21.2.102:0/4211222749 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3aa40b8c80 0x7f3aa41a7a70 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:08:21.867 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:21.870 7f3aa882d700 1 --2- 172.21.2.102:0/4211222749 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3a880186c0 0x7f3a8801abc0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:08:21.867 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:21.870 7f3aa882d700 1 --2- 172.21.2.102:0/4211222749 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3aa40b8880 0x7f3aa41a7450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:08:21.867 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:21.870 7f3aa882d700 1 --2- 172.21.2.102:0/4211222749 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3aa40b7db0 0x7f3aa40b8260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:08:21.867 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:21.870 7f3aa882d700 1 -- 172.21.2.102:0/4211222749 >> 172.21.2.102:0/4211222749 conn(0x7f3aa4094e10 msgr2=0x7f3aa40a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:08:21.867 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:21.870 7f3aa882d700 1 -- 172.21.2.102:0/4211222749 shutdown_connections 2021-02-10T11:08:21.868 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:21.870 7f3aa882d700 1 -- 172.21.2.102:0/4211222749 wait complete. 2021-02-10T11:08:21.875 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000344383 2021-02-10T11:08:22.307 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:22.293 7f9814b16700 1 Processor -- start 2021-02-10T11:08:22.307 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:22.293 7f9814b16700 1 -- start start 2021-02-10T11:08:22.307 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:22.294 7f9814b16700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f98101b0870 0x7f98100f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:08:22.307 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:22.294 7f9814b16700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f98100efe00 0x7f98100f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:08:22.307 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:22.294 7f9814b16700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98101b0c70 con 0x7f98101b0470 2021-02-10T11:08:22.307 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:22.294 7f9814b16700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98100e5ea0 con 0x7f98100efe00 2021-02-10T11:08:22.307 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:22.294 7f9814b16700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98100f0b50 con 0x7f98101b0870 2021-02-10T11:08:22.308 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:22.294 7f980f05f700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f98100efe00 0x7f98100f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:08:22.308 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:22.294 7f980e85e700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f98101b0470 0x7f98100ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:43682/0 (socket says 172.21.2.102:43682) 2021-02-10T11:08:22.308 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:22.294 7f980e85e700 1 -- 172.21.2.102:0/2642050108 learned_addr learned my addr 172.21.2.102:0/2642050108 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:08:22.308 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:22.294 7f980e05d700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f98101b0870 0x7f98100f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:08:22.308 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:22.295 7f980f05f700 1 -- 172.21.2.102:0/2642050108 >> v2:172.21.2.113:3300/0 conn(0x7f98101b0870 msgr2=0x7f98100f20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:08:22.308 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:22.295 7f980f05f700 1 --2- 172.21.2.102:0/2642050108 >> v2:172.21.2.113:3300/0 conn(0x7f98101b0870 0x7f98100f20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:08:22.308 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:22.295 7f980f05f700 1 -- 172.21.2.102:0/2642050108 >> v1:172.21.2.102:6789/0 conn(0x7f98101b0470 legacy=0x7f98100ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:08:22.308 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:22.295 7f980f05f700 1 -- 172.21.2.102:0/2642050108 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98100f0820 con 0x7f98100efe00 2021-02-10T11:08:22.308 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:22.295 7f980f05f700 1 --2- 172.21.2.102:0/2642050108 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f98100efe00 0x7f98100f0200 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 rx=0x7f9808006e30 tx=0x7f9808009e60).ready entity=mon.0 client_cookie=2bf72ef3327a9ba3 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:08:22.308 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:22.295 7f980d85c700 1 -- 172.21.2.102:0/2642050108 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f980800b300 con 0x7f98100efe00 2021-02-10T11:08:22.308 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:22.295 7f980d85c700 1 -- 172.21.2.102:0/2642050108 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f980800b580 con 0x7f98100efe00 2021-02-10T11:08:22.308 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:22.295 7f980d85c700 1 -- 172.21.2.102:0/2642050108 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f980800baa0 con 0x7f98100efe00 2021-02-10T11:08:22.309 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:22.295 7f9814b16700 1 -- 172.21.2.102:0/2642050108 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f98100efe00 msgr2=0x7f98100f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:08:22.309 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:22.295 7f9814b16700 1 --2- 172.21.2.102:0/2642050108 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f98100efe00 0x7f98100f0200 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 rx=0x7f9808006e30 tx=0x7f9808009e60).stop 2021-02-10T11:08:22.309 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:22.296 7f9814b16700 1 -- 172.21.2.102:0/2642050108 shutdown_connections 2021-02-10T11:08:22.309 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:22.296 7f9814b16700 1 --2- 172.21.2.102:0/2642050108 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f98100efe00 0x7f98100f0200 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:08:22.309 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:22.296 7f9814b16700 1 --2- 172.21.2.102:0/2642050108 >> v2:172.21.2.113:3300/0 conn(0x7f98101b0870 0x7f98100f20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:08:22.309 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:22.296 7f9814b16700 1 -- 172.21.2.102:0/2642050108 >> 172.21.2.102:0/2642050108 conn(0x7f98100ce3e0 msgr2=0x7f98100e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:08:22.309 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:22.296 7f9814b16700 1 -- 172.21.2.102:0/2642050108 shutdown_connections 2021-02-10T11:08:22.309 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:22.296 7f9814b16700 1 -- 172.21.2.102:0/2642050108 wait complete. 2021-02-10T11:08:22.309 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:22.311 7f9814b16700 1 Processor -- start 2021-02-10T11:08:22.310 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:22.311 7f9814b16700 1 -- start start 2021-02-10T11:08:22.310 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:22.311 7f9814b16700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9810190920 0x7f98100b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:08:22.310 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:22.311 7f9814b16700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f98100b8200 0x7f98100b8600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:08:22.310 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:22.311 7f9814b16700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f98100c1d80 0x7f98100c2180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:08:22.310 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:22.311 7f9814b16700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9810190d20 con 0x7f9810190920 2021-02-10T11:08:22.310 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:22.311 7f9814b16700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98100b8c20 con 0x7f98100c1d80 2021-02-10T11:08:22.310 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:22.311 7f9814b16700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98100c27a0 con 0x7f98100b8200 2021-02-10T11:08:22.311 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:22.311 7f980e85e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9810190920 0x7f98100b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:08:22.312 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:22.311 7f980e85e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9810190920 0x7f98100b7be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:51464/0 (socket says 172.21.2.102:51464) 2021-02-10T11:08:22.312 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:22.311 7f980e85e700 1 -- 172.21.2.102:0/1239378105 learned_addr learned my addr 172.21.2.102:0/1239378105 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:08:22.312 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:22.312 7f980e85e700 1 -- 172.21.2.102:0/1239378105 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f98100b8200 msgr2=0x7f98100b8600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:08:22.312 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:22.312 7f980e85e700 1 --2- 172.21.2.102:0/1239378105 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f98100b8200 0x7f98100b8600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:08:22.312 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:22.312 7f980e85e700 1 -- 172.21.2.102:0/1239378105 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f98100c1d80 msgr2=0x7f98100c2180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:08:22.312 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:22.312 7f980e85e700 1 --2- 172.21.2.102:0/1239378105 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f98100c1d80 0x7f98100c2180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:08:22.312 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:22.312 7f980e85e700 1 -- 172.21.2.102:0/1239378105 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98100c29c0 con 0x7f9810190920 2021-02-10T11:08:22.312 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:22.312 7f980e85e700 1 --2- 172.21.2.102:0/1239378105 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9810190920 0x7f98100b7be0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 rx=0x7f9800001410 tx=0x7f980000a950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:08:22.312 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:22.312 7f98077fe700 1 -- 172.21.2.102:0/1239378105 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f980000ad40 con 0x7f9810190920 2021-02-10T11:08:22.312 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:22.312 7f98077fe700 1 -- 172.21.2.102:0/1239378105 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9800000c90 con 0x7f9810190920 2021-02-10T11:08:22.312 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:22.312 7f98077fe700 1 -- 172.21.2.102:0/1239378105 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9800004590 con 0x7f9810190920 2021-02-10T11:08:22.312 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:22.312 7f9814b16700 1 -- 172.21.2.102:0/1239378105 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98100c29c0 con 0x7f9810190920 2021-02-10T11:08:22.313 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:22.312 7f9814b16700 1 -- 172.21.2.102:0/1239378105 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98100c29c0 con 0x7f9810190920 2021-02-10T11:08:22.313 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:22.313 7f98077fe700 1 -- 172.21.2.102:0/1239378105 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f98000045b0 con 0x7f9810190920 2021-02-10T11:08:22.313 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:22.313 7f98077fe700 1 --2- 172.21.2.102:0/1239378105 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f97f40186c0 0x7f97f401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:08:22.313 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:22.313 7f98077fe700 1 -- 172.21.2.102:0/1239378105 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f98000326a0 con 0x7f9810190920 2021-02-10T11:08:22.313 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:22.313 7f980e05d700 1 --2- 172.21.2.102:0/1239378105 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f97f40186c0 0x7f97f401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:08:22.313 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:22.313 7f980e05d700 1 --2- 172.21.2.102:0/1239378105 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f97f40186c0 0x7f97f401abc0 crc :-1 s=READY pgs=211 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:08:22.341 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:22.344 7f9814b16700 1 -- 172.21.2.102:0/1239378105 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9810105ae0 con 0x7f9810190920 2021-02-10T11:08:22.343 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:22.347 7f98077fe700 1 -- 172.21.2.102:0/1239378105 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f98000043a0 con 0x7f9810190920 2021-02-10T11:08:22.473 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:22.476 7f9814b16700 1 -- 172.21.2.102:0/1239378105 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344383", "target": ["mgr", ""]}) v1 -- 0x7f98100b8c20 con 0x7f97f40186c0 2021-02-10T11:08:22.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:22.479 7f98077fe700 1 -- 172.21.2.102:0/1239378105 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f97f8006e70 con 0x7f97f40186c0 2021-02-10T11:08:22.476 INFO:teuthology.orchestra.run.gibba002.stdout:/volumes/_nogroup/subvolume_0000000000344383/66beaa64-11fb-476c-a5e2-19e76192c644 2021-02-10T11:08:22.478 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:22.481 7f9814b16700 1 -- 172.21.2.102:0/1239378105 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f97f40186c0 msgr2=0x7f97f401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:08:22.478 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:22.481 7f9814b16700 1 --2- 172.21.2.102:0/1239378105 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f97f40186c0 0x7f97f401abc0 crc :-1 s=READY pgs=211 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:08:22.478 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:22.481 7f9814b16700 1 -- 172.21.2.102:0/1239378105 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9810190920 msgr2=0x7f98100b7be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:08:22.478 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:22.481 7f9814b16700 1 --2- 172.21.2.102:0/1239378105 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9810190920 0x7f98100b7be0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 rx=0x7f9800001410 tx=0x7f980000a950).stop 2021-02-10T11:08:22.479 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:22.481 7f9814b16700 1 -- 172.21.2.102:0/1239378105 shutdown_connections 2021-02-10T11:08:22.479 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:22.481 7f9814b16700 1 --2- 172.21.2.102:0/1239378105 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f97f40186c0 0x7f97f401abc0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:08:22.479 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:22.481 7f9814b16700 1 --2- 172.21.2.102:0/1239378105 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f98100c1d80 0x7f98100c2180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:08:22.479 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:22.481 7f9814b16700 1 --2- 172.21.2.102:0/1239378105 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f98100b8200 0x7f98100b8600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:08:22.479 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:22.481 7f9814b16700 1 --2- 172.21.2.102:0/1239378105 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9810190920 0x7f98100b7be0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:08:22.479 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:22.481 7f9814b16700 1 -- 172.21.2.102:0/1239378105 >> 172.21.2.102:0/1239378105 conn(0x7f9810094e10 msgr2=0x7f98100a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:08:22.480 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:22.481 7f9814b16700 1 -- 172.21.2.102:0/1239378105 shutdown_connections 2021-02-10T11:08:22.480 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:22.481 7f9814b16700 1 -- 172.21.2.102:0/1239378105 wait complete. 2021-02-10T11:08:22.486 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000344383 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000344383/66beaa64-11fb-476c-a5e2-19e76192c644 2021-02-10T11:08:22.486 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:08:22.487 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:08:22.487 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:08:22.824 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:08:22.824 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:08:22.824 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344383/66beaa64-11fb-476c-a5e2-19e76192c644/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:08:23.205 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:08:23.206 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:08:23.206 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0247328 s, 42.4 MB/s 2021-02-10T11:08:23.207 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:08:23.207 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:08:23.207 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:08:23.541 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:08:23.542 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:08:23.542 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344383/66beaa64-11fb-476c-a5e2-19e76192c644/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:08:23.913 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:08:23.914 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:08:23.914 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0239757 s, 43.7 MB/s 2021-02-10T11:08:23.915 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:08:23.915 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:08:23.915 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:08:24.250 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:08:24.251 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:08:24.251 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344383/66beaa64-11fb-476c-a5e2-19e76192c644/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:08:24.627 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:08:24.628 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:08:24.628 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0247912 s, 42.3 MB/s 2021-02-10T11:08:24.628 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:08:24.629 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:08:24.629 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:08:24.964 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:08:24.964 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:08:24.964 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344383/66beaa64-11fb-476c-a5e2-19e76192c644/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:08:25.339 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:08:25.339 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:08:25.339 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0241342 s, 43.4 MB/s 2021-02-10T11:08:25.340 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:08:25.340 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:08:25.340 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:08:25.675 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:08:25.675 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:08:25.676 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344383/66beaa64-11fb-476c-a5e2-19e76192c644/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:08:26.051 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:08:26.052 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:08:26.052 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0254411 s, 41.2 MB/s 2021-02-10T11:08:26.053 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:08:26.054 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:08:26.054 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:08:26.389 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:08:26.390 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:08:26.390 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344383/66beaa64-11fb-476c-a5e2-19e76192c644/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:08:26.766 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:08:26.766 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:08:26.766 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0241944 s, 43.3 MB/s 2021-02-10T11:08:26.767 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:08:26.767 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:08:26.767 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:08:27.103 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:08:27.103 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:08:27.104 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344383/66beaa64-11fb-476c-a5e2-19e76192c644/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:08:27.480 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:08:27.480 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:08:27.480 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0244168 s, 42.9 MB/s 2021-02-10T11:08:27.481 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:08:27.481 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:08:27.481 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:08:27.819 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:08:27.819 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:08:27.819 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344383/66beaa64-11fb-476c-a5e2-19e76192c644/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:08:28.193 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:08:28.194 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:08:28.194 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0235888 s, 44.5 MB/s 2021-02-10T11:08:28.195 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:08:28.195 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:08:28.195 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:08:28.529 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:08:28.529 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:08:28.530 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344383/66beaa64-11fb-476c-a5e2-19e76192c644/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:08:28.903 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:08:28.903 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:08:28.904 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0257344 s, 40.7 MB/s 2021-02-10T11:08:28.905 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:08:28.905 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:08:28.905 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:08:29.239 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:08:29.240 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:08:29.240 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344383/66beaa64-11fb-476c-a5e2-19e76192c644/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:08:29.616 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:08:29.616 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:08:29.616 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0253872 s, 41.3 MB/s 2021-02-10T11:08:29.617 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000344384 2021-02-10T11:08:30.049 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.034 7fc5a0f5c700 1 Processor -- start 2021-02-10T11:08:30.049 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.034 7fc5a0f5c700 1 -- start start 2021-02-10T11:08:30.049 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.035 7fc5a0f5c700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc59c0f05f0 0x7fc59c0f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:08:30.049 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.035 7fc5a0f5c700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc59c1b0c70 con 0x7fc59c1b0470 2021-02-10T11:08:30.049 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.035 7fc5a0f5c700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc59c0f0b50 con 0x7fc59c1b0870 2021-02-10T11:08:30.049 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.035 7fc5a0f5c700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc59c0e5ea0 con 0x7fc59c0f05f0 2021-02-10T11:08:30.049 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.035 7fc59ab13700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fc59c1b0470 0x7fc59c0ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:43696/0 (socket says 172.21.2.102:43696) 2021-02-10T11:08:30.049 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.035 7fc59ab13700 1 -- 172.21.2.102:0/3204138782 learned_addr learned my addr 172.21.2.102:0/3204138782 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:08:30.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.036 7fc59b314700 1 --2- 172.21.2.102:0/3204138782 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc59c0f05f0 0x7fc59c0f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:08:30.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.036 7fc599b11700 1 -- 172.21.2.102:0/3204138782 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1520378695 0 0) 0x7fc584000ce0 con 0x7fc59c1b0470 2021-02-10T11:08:30.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.036 7fc599b11700 1 -- 172.21.2.102:0/3204138782 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc588002fe0 con 0x7fc59c1b0470 2021-02-10T11:08:30.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.036 7fc599b11700 1 -- 172.21.2.102:0/3204138782 <== mon.1 v1:172.21.2.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3650863900 0 0) 0x7fc58c000ff0 con 0x7fc59c1b0870 2021-02-10T11:08:30.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.036 7fc599b11700 1 -- 172.21.2.102:0/3204138782 --> v1:172.21.2.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc588003210 con 0x7fc59c1b0870 2021-02-10T11:08:30.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.036 7fc599b11700 1 -- 172.21.2.102:0/3204138782 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3963363109 0 0) 0x7fc584001180 con 0x7fc59c1b0470 2021-02-10T11:08:30.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.036 7fc599b11700 1 -- 172.21.2.102:0/3204138782 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc59c0f05f0 msgr2=0x7fc59c0f20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:08:30.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.036 7fc599b11700 1 --2- 172.21.2.102:0/3204138782 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc59c0f05f0 0x7fc59c0f20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:08:30.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.036 7fc599b11700 1 -- 172.21.2.102:0/3204138782 >> v1:172.21.2.113:6789/0 conn(0x7fc59c1b0870 legacy=0x7fc59c0efe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:08:30.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.036 7fc599b11700 1 -- 172.21.2.102:0/3204138782 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc59c0e52e0 con 0x7fc59c1b0470 2021-02-10T11:08:30.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.036 7fc599b11700 1 -- 172.21.2.102:0/3204138782 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7fc584001680 con 0x7fc59c1b0470 2021-02-10T11:08:30.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.036 7fc599b11700 1 -- 172.21.2.102:0/3204138782 >> v1:172.21.2.102:6789/0 conn(0x7fc59c1b0470 legacy=0x7fc59c0ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:08:30.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.036 7fc599b11700 1 --2- 172.21.2.102:0/3204138782 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc5880037e0 0x7fc588003c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:08:30.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.036 7fc599b11700 1 --2- 172.21.2.102:0/3204138782 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc5880067e0 0x7fc588008c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:08:30.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.036 7fc599b11700 1 --2- 172.21.2.102:0/3204138782 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc588009510 0x7fc58800ba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:08:30.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.036 7fc599b11700 1 -- 172.21.2.102:0/3204138782 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc58800c2e0 con 0x7fc588009510 2021-02-10T11:08:30.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.036 7fc599b11700 1 -- 172.21.2.102:0/3204138782 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5880504d0 con 0x7fc5880067e0 2021-02-10T11:08:30.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.036 7fc599b11700 1 -- 172.21.2.102:0/3204138782 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5880506f0 con 0x7fc5880037e0 2021-02-10T11:08:30.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.036 7fc59ab13700 1 --2- 172.21.2.102:0/3204138782 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc5880037e0 0x7fc588003c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:08:30.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.036 7fc59b314700 1 --2- 172.21.2.102:0/3204138782 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc588009510 0x7fc58800ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:08:30.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.036 7fc59a312700 1 --2- 172.21.2.102:0/3204138782 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc5880067e0 0x7fc588008c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:08:30.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.036 7fc59b314700 1 -- 172.21.2.102:0/3204138782 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc5880037e0 msgr2=0x7fc588003c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:08:30.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.036 7fc59b314700 1 --2- 172.21.2.102:0/3204138782 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc5880037e0 0x7fc588003c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:08:30.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.036 7fc59b314700 1 -- 172.21.2.102:0/3204138782 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc5880067e0 msgr2=0x7fc588008c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:08:30.052 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.036 7fc59b314700 1 --2- 172.21.2.102:0/3204138782 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc5880067e0 0x7fc588008c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:08:30.052 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.036 7fc59b314700 1 -- 172.21.2.102:0/3204138782 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc588050990 con 0x7fc588009510 2021-02-10T11:08:30.052 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.037 7fc59b314700 1 --2- 172.21.2.102:0/3204138782 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc588009510 0x7fc58800ba30 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 rx=0x7fc594009b10 tx=0x7fc594009eb0).ready entity=mon.0 client_cookie=64aa96cc647baa78 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:08:30.052 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.037 7fc599b11700 1 -- 172.21.2.102:0/3204138782 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc59400b510 con 0x7fc588009510 2021-02-10T11:08:30.052 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.037 7fc599b11700 1 -- 172.21.2.102:0/3204138782 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc594003030 con 0x7fc588009510 2021-02-10T11:08:30.052 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.037 7fc599b11700 1 -- 172.21.2.102:0/3204138782 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc59400b790 con 0x7fc588009510 2021-02-10T11:08:30.052 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.037 7fc5a0f5c700 1 -- 172.21.2.102:0/3204138782 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc588009510 msgr2=0x7fc58800ba30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:08:30.052 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.037 7fc5a0f5c700 1 --2- 172.21.2.102:0/3204138782 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc588009510 0x7fc58800ba30 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 rx=0x7fc594009b10 tx=0x7fc594009eb0).stop 2021-02-10T11:08:30.052 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.037 7fc59b314700 1 -- 172.21.2.102:0/3204138782 reap_dead start 2021-02-10T11:08:30.052 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.038 7fc5a0f5c700 1 -- 172.21.2.102:0/3204138782 shutdown_connections 2021-02-10T11:08:30.052 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.038 7fc5a0f5c700 1 -- 172.21.2.102:0/3204138782 >> 172.21.2.102:0/3204138782 conn(0x7fc59c0ce3e0 msgr2=0x7fc59c0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:08:30.052 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.038 7fc5a0f5c700 1 -- 172.21.2.102:0/3204138782 shutdown_connections 2021-02-10T11:08:30.053 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.038 7fc5a0f5c700 1 -- 172.21.2.102:0/3204138782 wait complete. 2021-02-10T11:08:30.053 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.052 7fc5a0f5c700 1 Processor -- start 2021-02-10T11:08:30.053 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.052 7fc5a0f5c700 1 -- start start 2021-02-10T11:08:30.054 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.053 7fc5a0f5c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc59c0bf920 0x7fc59c0bfd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:08:30.054 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.053 7fc5a0f5c700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc59c0b6570 0x7fc59c0b6970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:08:30.054 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.053 7fc5a0f5c700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc59c1a7430 0x7fc59c1a7830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:08:30.054 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.053 7fc5a0f5c700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc59c0c0440 con 0x7fc59c0bf920 2021-02-10T11:08:30.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.053 7fc5a0f5c700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc59c0924b0 con 0x7fc59c0b6570 2021-02-10T11:08:30.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.053 7fc5a0f5c700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc59c092970 con 0x7fc59c1a7430 2021-02-10T11:08:30.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.053 7fc59ab13700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc59c0bf920 0x7fc59c0bfd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:08:30.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.053 7fc59ab13700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc59c0bf920 0x7fc59c0bfd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:51486/0 (socket says 172.21.2.102:51486) 2021-02-10T11:08:30.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.053 7fc59ab13700 1 -- 172.21.2.102:0/3677762028 learned_addr learned my addr 172.21.2.102:0/3677762028 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:08:30.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.053 7fc59ab13700 1 -- 172.21.2.102:0/3677762028 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc59c1a7430 msgr2=0x7fc59c1a7830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:08:30.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.053 7fc59ab13700 1 --2- 172.21.2.102:0/3677762028 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc59c1a7430 0x7fc59c1a7830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:08:30.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.053 7fc59ab13700 1 -- 172.21.2.102:0/3677762028 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc59c0b6570 msgr2=0x7fc59c0b6970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:08:30.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.053 7fc59ab13700 1 --2- 172.21.2.102:0/3677762028 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc59c0b6570 0x7fc59c0b6970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:08:30.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.053 7fc59ab13700 1 -- 172.21.2.102:0/3677762028 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc59c1a7e50 con 0x7fc59c0bf920 2021-02-10T11:08:30.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.054 7fc59ab13700 1 --2- 172.21.2.102:0/3677762028 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc59c0bf920 0x7fc59c0bfd20 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 rx=0x7fc584000d70 tx=0x7fc58400a930).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:08:30.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.054 7fc5937fe700 1 -- 172.21.2.102:0/3677762028 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc58400ad20 con 0x7fc59c0bf920 2021-02-10T11:08:30.056 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.054 7fc5937fe700 1 -- 172.21.2.102:0/3677762028 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5840014f0 con 0x7fc59c0bf920 2021-02-10T11:08:30.056 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.054 7fc5a0f5c700 1 -- 172.21.2.102:0/3677762028 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc59c1a7e50 con 0x7fc59c0bf920 2021-02-10T11:08:30.056 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.054 7fc5937fe700 1 -- 172.21.2.102:0/3677762028 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc5840045b0 con 0x7fc59c0bf920 2021-02-10T11:08:30.056 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.054 7fc5a0f5c700 1 -- 172.21.2.102:0/3677762028 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc59c1a7e50 con 0x7fc59c0bf920 2021-02-10T11:08:30.056 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.055 7fc5937fe700 1 -- 172.21.2.102:0/3677762028 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fc5840043a0 con 0x7fc59c0bf920 2021-02-10T11:08:30.056 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.055 7fc5937fe700 1 --2- 172.21.2.102:0/3677762028 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc580018700 0x7fc58001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:08:30.056 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.055 7fc5937fe700 1 -- 172.21.2.102:0/3677762028 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fc584032640 con 0x7fc59c0bf920 2021-02-10T11:08:30.056 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.055 7fc59a312700 1 --2- 172.21.2.102:0/3677762028 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc580018700 0x7fc58001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:08:30.057 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.055 7fc59a312700 1 --2- 172.21.2.102:0/3677762028 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc580018700 0x7fc58001abc0 crc :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:08:30.081 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.085 7fc5a0f5c700 1 -- 172.21.2.102:0/3677762028 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc59c105b40 con 0x7fc59c0bf920 2021-02-10T11:08:30.084 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.087 7fc5937fe700 1 -- 172.21.2.102:0/3677762028 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fc584004f00 con 0x7fc59c0bf920 2021-02-10T11:08:30.209 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.213 7fc5a0f5c700 1 -- 172.21.2.102:0/3677762028 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344384", "target": ["mgr", ""]}) v1 -- 0x7fc59c105b40 con 0x7fc580018700 2021-02-10T11:08:30.234 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.237 7fc5937fe700 1 -- 172.21.2.102:0/3677762028 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fc58c005530 con 0x7fc580018700 2021-02-10T11:08:30.243 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.246 7fc5a0f5c700 1 -- 172.21.2.102:0/3677762028 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc580018700 msgr2=0x7fc58001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:08:30.243 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.246 7fc5a0f5c700 1 --2- 172.21.2.102:0/3677762028 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc580018700 0x7fc58001abc0 crc :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:08:30.243 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.246 7fc5a0f5c700 1 -- 172.21.2.102:0/3677762028 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc59c0bf920 msgr2=0x7fc59c0bfd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:08:30.243 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.246 7fc5a0f5c700 1 --2- 172.21.2.102:0/3677762028 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc59c0bf920 0x7fc59c0bfd20 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 rx=0x7fc584000d70 tx=0x7fc58400a930).stop 2021-02-10T11:08:30.243 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.246 7fc5a0f5c700 1 -- 172.21.2.102:0/3677762028 shutdown_connections 2021-02-10T11:08:30.243 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.246 7fc5a0f5c700 1 --2- 172.21.2.102:0/3677762028 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc59c1a7430 0x7fc59c1a7830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:08:30.243 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.246 7fc5a0f5c700 1 --2- 172.21.2.102:0/3677762028 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc580018700 0x7fc58001abc0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:08:30.244 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.246 7fc5a0f5c700 1 --2- 172.21.2.102:0/3677762028 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc59c0b6570 0x7fc59c0b6970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:08:30.244 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.246 7fc5a0f5c700 1 --2- 172.21.2.102:0/3677762028 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc59c0bf920 0x7fc59c0bfd20 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:08:30.244 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.246 7fc5a0f5c700 1 -- 172.21.2.102:0/3677762028 >> 172.21.2.102:0/3677762028 conn(0x7fc59c0b50c0 msgr2=0x7fc59c092d90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:08:30.244 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.246 7fc5a0f5c700 1 -- 172.21.2.102:0/3677762028 shutdown_connections 2021-02-10T11:08:30.244 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.246 7fc5a0f5c700 1 -- 172.21.2.102:0/3677762028 wait complete. 2021-02-10T11:08:30.251 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000344384 2021-02-10T11:08:30.682 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.668 7f35cb756700 1 Processor -- start 2021-02-10T11:08:30.683 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.668 7f35cb756700 1 -- start start 2021-02-10T11:08:30.683 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.668 7f35cb756700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f35c41b0470 0x7f35c41b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:08:30.683 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.668 7f35cb756700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35c40f0b50 con 0x7f35c40f20e0 2021-02-10T11:08:30.683 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.668 7f35cb756700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35c40f24e0 con 0x7f35c40ef610 2021-02-10T11:08:30.683 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.668 7f35cb756700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35c40e5ea0 con 0x7f35c41b0470 2021-02-10T11:08:30.683 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.669 7f35c9d73700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f35c40f20e0 0x7f35c40f0200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:43716/0 (socket says 172.21.2.102:43716) 2021-02-10T11:08:30.683 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.669 7f35c9d73700 1 -- 172.21.2.102:0/695433096 learned_addr learned my addr 172.21.2.102:0/695433096 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:08:30.683 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.669 7f35bbfff700 1 -- 172.21.2.102:0/695433096 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4050687532 0 0) 0x7f35bc001500 con 0x7f35c40f20e0 2021-02-10T11:08:30.683 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.669 7f35bbfff700 1 -- 172.21.2.102:0/695433096 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35b0002fe0 con 0x7f35c40f20e0 2021-02-10T11:08:30.683 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.669 7f35c9572700 1 --2- 172.21.2.102:0/695433096 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f35c41b0470 0x7f35c41b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:08:30.683 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.670 7f35bbfff700 1 -- 172.21.2.102:0/695433096 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1908117577 0 0) 0x7f35bc001460 con 0x7f35c40f20e0 2021-02-10T11:08:30.683 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.670 7f35bbfff700 1 -- 172.21.2.102:0/695433096 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f35c41b0470 msgr2=0x7f35c41b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:08:30.684 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.670 7f35bbfff700 1 --2- 172.21.2.102:0/695433096 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f35c41b0470 0x7f35c41b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:08:30.684 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.670 7f35bbfff700 1 -- 172.21.2.102:0/695433096 >> v1:172.21.2.113:6789/0 conn(0x7f35c40ef610 legacy=0x7f35c40efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:08:30.684 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.670 7f35bbfff700 1 -- 172.21.2.102:0/695433096 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35c40e52e0 con 0x7f35c40f20e0 2021-02-10T11:08:30.684 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.670 7f35bbfff700 1 -- 172.21.2.102:0/695433096 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7f35bc001980 con 0x7f35c40f20e0 2021-02-10T11:08:30.684 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.670 7f35bbfff700 1 -- 172.21.2.102:0/695433096 >> v1:172.21.2.102:6789/0 conn(0x7f35c40f20e0 legacy=0x7f35c40f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:08:30.684 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.670 7f35bbfff700 1 --2- 172.21.2.102:0/695433096 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f35b0003ac0 0x7f35b0005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:08:30.684 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.670 7f35bbfff700 1 --2- 172.21.2.102:0/695433096 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f35b00067a0 0x7f35b0008c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:08:30.684 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.670 7f35bbfff700 1 --2- 172.21.2.102:0/695433096 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f35b0009510 0x7f35b000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:08:30.684 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.670 7f35bbfff700 1 -- 172.21.2.102:0/695433096 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35b000c2c0 con 0x7f35b0003ac0 2021-02-10T11:08:30.684 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.670 7f35bbfff700 1 -- 172.21.2.102:0/695433096 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35b000c4e0 con 0x7f35b0009510 2021-02-10T11:08:30.684 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.670 7f35bbfff700 1 -- 172.21.2.102:0/695433096 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35b000c700 con 0x7f35b00067a0 2021-02-10T11:08:30.684 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.670 7f35c9572700 1 --2- 172.21.2.102:0/695433096 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f35b0003ac0 0x7f35b0005fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:08:30.685 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.670 7f35c9d73700 1 --2- 172.21.2.102:0/695433096 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f35b0009510 0x7f35b000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:08:30.685 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.670 7f35c8d71700 1 --2- 172.21.2.102:0/695433096 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f35b00067a0 0x7f35b0008c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:08:30.685 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.670 7f35c9572700 1 -- 172.21.2.102:0/695433096 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f35b00067a0 msgr2=0x7f35b0008c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:08:30.685 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.670 7f35c9572700 1 --2- 172.21.2.102:0/695433096 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f35b00067a0 0x7f35b0008c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:08:30.685 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.670 7f35c9572700 1 -- 172.21.2.102:0/695433096 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f35b0009510 msgr2=0x7f35b000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:08:30.685 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.670 7f35c9572700 1 --2- 172.21.2.102:0/695433096 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f35b0009510 0x7f35b000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:08:30.685 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.670 7f35c9572700 1 -- 172.21.2.102:0/695433096 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35b0050970 con 0x7f35b0003ac0 2021-02-10T11:08:30.685 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.670 7f35c9572700 1 --2- 172.21.2.102:0/695433096 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f35b0003ac0 0x7f35b0005fa0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 rx=0x7f35c0006ac0 tx=0x7f35c0009ab0).ready entity=mon.0 client_cookie=5e974b36c33a4a86 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:08:30.685 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.671 7f35bbfff700 1 -- 172.21.2.102:0/695433096 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f35c000c0c0 con 0x7f35b0003ac0 2021-02-10T11:08:30.685 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.671 7f35bbfff700 1 -- 172.21.2.102:0/695433096 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35c000c3c0 con 0x7f35b0003ac0 2021-02-10T11:08:30.685 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.671 7f35bbfff700 1 -- 172.21.2.102:0/695433096 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f35c000c8e0 con 0x7f35b0003ac0 2021-02-10T11:08:30.685 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.671 7f35cb756700 1 -- 172.21.2.102:0/695433096 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f35b0003ac0 msgr2=0x7f35b0005fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:08:30.685 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.671 7f35cb756700 1 --2- 172.21.2.102:0/695433096 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f35b0003ac0 0x7f35b0005fa0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 rx=0x7f35c0006ac0 tx=0x7f35c0009ab0).stop 2021-02-10T11:08:30.686 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.671 7f35c9d73700 1 -- 172.21.2.102:0/695433096 reap_dead start 2021-02-10T11:08:30.686 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.671 7f35cb756700 1 -- 172.21.2.102:0/695433096 shutdown_connections 2021-02-10T11:08:30.686 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.671 7f35cb756700 1 -- 172.21.2.102:0/695433096 >> 172.21.2.102:0/695433096 conn(0x7f35c40ce3e0 msgr2=0x7f35c40e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:08:30.686 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.672 7f35cb756700 1 -- 172.21.2.102:0/695433096 shutdown_connections 2021-02-10T11:08:30.686 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.672 7f35cb756700 1 -- 172.21.2.102:0/695433096 wait complete. 2021-02-10T11:08:30.686 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.686 7f35cb756700 1 Processor -- start 2021-02-10T11:08:30.686 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.686 7f35cb756700 1 -- start start 2021-02-10T11:08:30.687 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.687 7f35cb756700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f35c40c30c0 0x7f35c41a7450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:08:30.688 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.687 7f35cb756700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f35c40c34c0 0x7f35c41a7a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:08:30.689 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.687 7f35cb756700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f35c40a93e0 0x7f35c40a97e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:08:30.689 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.687 7f35cb756700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35c41a8090 con 0x7f35c40c30c0 2021-02-10T11:08:30.689 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.687 7f35cb756700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35c41a82b0 con 0x7f35c40c34c0 2021-02-10T11:08:30.689 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.687 7f35cb756700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35c40a9e00 con 0x7f35c40a93e0 2021-02-10T11:08:30.689 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.687 7f35c9572700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f35c40c30c0 0x7f35c41a7450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:08:30.689 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.687 7f35c9572700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f35c40c30c0 0x7f35c41a7450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:51502/0 (socket says 172.21.2.102:51502) 2021-02-10T11:08:30.689 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.687 7f35c9572700 1 -- 172.21.2.102:0/4272837410 learned_addr learned my addr 172.21.2.102:0/4272837410 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:08:30.690 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.687 7f35c8d71700 1 --2- 172.21.2.102:0/4272837410 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f35c40c34c0 0x7f35c41a7a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:08:30.690 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.687 7f35c9d73700 1 --2- 172.21.2.102:0/4272837410 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f35c40a93e0 0x7f35c40a97e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:08:30.690 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.687 7f35c9572700 1 -- 172.21.2.102:0/4272837410 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f35c40a93e0 msgr2=0x7f35c40a97e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:08:30.690 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.687 7f35c9572700 1 --2- 172.21.2.102:0/4272837410 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f35c40a93e0 0x7f35c40a97e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:08:30.690 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.687 7f35c9572700 1 -- 172.21.2.102:0/4272837410 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f35c40c34c0 msgr2=0x7f35c41a7a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:08:30.690 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.687 7f35c9572700 1 --2- 172.21.2.102:0/4272837410 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f35c40c34c0 0x7f35c41a7a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:08:30.690 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.687 7f35c9572700 1 -- 172.21.2.102:0/4272837410 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35c40aa020 con 0x7f35c40c30c0 2021-02-10T11:08:30.691 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.687 7f35c8d71700 1 --2- 172.21.2.102:0/4272837410 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f35c40c34c0 0x7f35c41a7a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:08:30.691 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.688 7f35c9572700 1 --2- 172.21.2.102:0/4272837410 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f35c40c30c0 0x7f35c41a7450 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 rx=0x7f35c00097a0 tx=0x7f35c000b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:08:30.691 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.688 7f35b9ffb700 1 -- 172.21.2.102:0/4272837410 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f35c000bd60 con 0x7f35c40c30c0 2021-02-10T11:08:30.691 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.688 7f35b9ffb700 1 -- 172.21.2.102:0/4272837410 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35c0000c90 con 0x7f35c40c30c0 2021-02-10T11:08:30.691 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.688 7f35b9ffb700 1 -- 172.21.2.102:0/4272837410 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f35c0002220 con 0x7f35c40c30c0 2021-02-10T11:08:30.691 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.688 7f35cb756700 1 -- 172.21.2.102:0/4272837410 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35c40aa020 con 0x7f35c40c30c0 2021-02-10T11:08:30.691 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.688 7f35cb756700 1 -- 172.21.2.102:0/4272837410 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35c40aa5b0 con 0x7f35c40c30c0 2021-02-10T11:08:30.691 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.689 7f35b9ffb700 1 -- 172.21.2.102:0/4272837410 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f35c00052e0 con 0x7f35c40c30c0 2021-02-10T11:08:30.692 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.689 7f35b9ffb700 1 --2- 172.21.2.102:0/4272837410 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f35a40186c0 0x7f35a401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:08:30.692 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.689 7f35b9ffb700 1 -- 172.21.2.102:0/4272837410 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f35c00323c0 con 0x7f35c40c30c0 2021-02-10T11:08:30.692 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.689 7f35c8d71700 1 --2- 172.21.2.102:0/4272837410 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f35a40186c0 0x7f35a401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:08:30.692 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.689 7f35c8d71700 1 --2- 172.21.2.102:0/4272837410 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f35a40186c0 0x7f35a401abc0 crc :-1 s=READY pgs=213 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:08:30.715 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.718 7f35cb756700 1 -- 172.21.2.102:0/4272837410 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35c4105b30 con 0x7f35c40c30c0 2021-02-10T11:08:30.717 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.721 7f35b9ffb700 1 -- 172.21.2.102:0/4272837410 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f35c0009360 con 0x7f35c40c30c0 2021-02-10T11:08:30.846 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.850 7f35cb756700 1 -- 172.21.2.102:0/4272837410 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344384", "target": ["mgr", ""]}) v1 -- 0x7f35c40aa5b0 con 0x7f35a40186c0 2021-02-10T11:08:30.849 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.853 7f35b9ffb700 1 -- 172.21.2.102:0/4272837410 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f35b4006ab0 con 0x7f35a40186c0 2021-02-10T11:08:30.850 INFO:teuthology.orchestra.run.gibba002.stdout:/volumes/_nogroup/subvolume_0000000000344384/b7af9a74-1021-4517-8067-b1845c99fb78 2021-02-10T11:08:30.851 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.854 7f35cb756700 1 -- 172.21.2.102:0/4272837410 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f35a40186c0 msgr2=0x7f35a401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:08:30.851 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.854 7f35cb756700 1 --2- 172.21.2.102:0/4272837410 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f35a40186c0 0x7f35a401abc0 crc :-1 s=READY pgs=213 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:08:30.851 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.854 7f35cb756700 1 -- 172.21.2.102:0/4272837410 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f35c40c30c0 msgr2=0x7f35c41a7450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:08:30.851 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.854 7f35cb756700 1 --2- 172.21.2.102:0/4272837410 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f35c40c30c0 0x7f35c41a7450 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 rx=0x7f35c00097a0 tx=0x7f35c000b970).stop 2021-02-10T11:08:30.851 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.854 7f35cb756700 1 -- 172.21.2.102:0/4272837410 shutdown_connections 2021-02-10T11:08:30.851 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.854 7f35cb756700 1 --2- 172.21.2.102:0/4272837410 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f35c40a93e0 0x7f35c40a97e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:08:30.851 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.854 7f35cb756700 1 --2- 172.21.2.102:0/4272837410 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f35a40186c0 0x7f35a401abc0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:08:30.851 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.854 7f35cb756700 1 --2- 172.21.2.102:0/4272837410 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f35c40c34c0 0x7f35c41a7a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:08:30.852 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.854 7f35cb756700 1 --2- 172.21.2.102:0/4272837410 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f35c40c30c0 0x7f35c41a7450 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:08:30.852 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.854 7f35cb756700 1 -- 172.21.2.102:0/4272837410 >> 172.21.2.102:0/4272837410 conn(0x7f35c409fc10 msgr2=0x7f35c40b3120 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:08:30.852 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.855 7f35cb756700 1 -- 172.21.2.102:0/4272837410 shutdown_connections 2021-02-10T11:08:30.852 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:30.855 7f35cb756700 1 -- 172.21.2.102:0/4272837410 wait complete. 2021-02-10T11:08:30.859 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000344384 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000344384/b7af9a74-1021-4517-8067-b1845c99fb78 2021-02-10T11:08:30.860 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:08:30.860 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:08:30.860 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:08:31.197 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:08:31.197 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:08:31.198 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344384/b7af9a74-1021-4517-8067-b1845c99fb78/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:08:31.579 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:08:31.579 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:08:31.579 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0247635 s, 42.3 MB/s 2021-02-10T11:08:31.580 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:08:31.580 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:08:31.581 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:08:31.915 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:08:31.916 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:08:31.916 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344384/b7af9a74-1021-4517-8067-b1845c99fb78/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:08:32.291 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:08:32.291 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:08:32.291 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0251126 s, 41.8 MB/s 2021-02-10T11:08:32.292 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:08:32.292 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:08:32.292 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:08:32.628 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:08:32.628 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:08:32.629 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344384/b7af9a74-1021-4517-8067-b1845c99fb78/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:08:33.005 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:08:33.005 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:08:33.005 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.025353 s, 41.4 MB/s 2021-02-10T11:08:33.006 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:08:33.006 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:08:33.006 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:08:33.342 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:08:33.342 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:08:33.342 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344384/b7af9a74-1021-4517-8067-b1845c99fb78/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:08:33.715 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:08:33.715 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:08:33.716 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0247077 s, 42.4 MB/s 2021-02-10T11:08:33.716 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:08:33.717 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:08:33.717 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:08:34.052 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:08:34.052 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:08:34.053 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344384/b7af9a74-1021-4517-8067-b1845c99fb78/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:08:34.431 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:08:34.431 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:08:34.431 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0265503 s, 39.5 MB/s 2021-02-10T11:08:34.432 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:08:34.432 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:08:34.432 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:08:34.767 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:08:34.768 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:08:34.768 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344384/b7af9a74-1021-4517-8067-b1845c99fb78/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:08:35.144 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:08:35.144 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:08:35.144 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0257522 s, 40.7 MB/s 2021-02-10T11:08:35.145 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:08:35.145 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:08:35.145 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:08:35.480 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:08:35.480 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:08:35.481 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344384/b7af9a74-1021-4517-8067-b1845c99fb78/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:08:35.856 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:08:35.856 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:08:35.856 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0240311 s, 43.6 MB/s 2021-02-10T11:08:35.857 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:08:35.857 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:08:35.857 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:08:36.192 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:08:36.250 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:08:36.251 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344384/b7af9a74-1021-4517-8067-b1845c99fb78/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:08:36.626 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:08:36.627 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:08:36.627 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0240003 s, 43.7 MB/s 2021-02-10T11:08:36.627 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:08:36.628 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:08:36.628 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:08:36.962 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:08:36.962 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:08:36.962 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344384/b7af9a74-1021-4517-8067-b1845c99fb78/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:08:37.336 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:08:37.336 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:08:37.336 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0231899 s, 45.2 MB/s 2021-02-10T11:08:37.337 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:08:37.337 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:08:37.337 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:08:37.673 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:08:37.673 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:08:37.673 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344384/b7af9a74-1021-4517-8067-b1845c99fb78/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:08:38.048 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:08:38.048 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:08:38.048 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0235819 s, 44.5 MB/s 2021-02-10T11:08:38.049 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000344385 2021-02-10T11:08:38.481 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:38.468 7f807a101700 1 Processor -- start 2021-02-10T11:08:38.481 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:38.468 7f807a101700 1 -- start start 2021-02-10T11:08:38.481 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:38.468 7f807a101700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f80741b0470 0x7f80741b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:08:38.481 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:38.468 7f807a101700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f80740ef610 0x7f80740efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:08:38.481 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:38.468 7f807a101700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80740f0b50 con 0x7f80740f0030 2021-02-10T11:08:38.481 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:38.468 7f807a101700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80740e52e0 con 0x7f80741b0470 2021-02-10T11:08:38.481 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:38.468 7f807a101700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80740e5490 con 0x7f80740ef610 2021-02-10T11:08:38.481 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:38.469 7f80737fe700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f80741b0470 0x7f80741b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:08:38.481 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:38.469 7f80737fe700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f80741b0470 0x7f80741b0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:51510/0 (socket says 172.21.2.102:51510) 2021-02-10T11:08:38.481 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:38.469 7f80737fe700 1 -- 172.21.2.102:0/603269317 learned_addr learned my addr 172.21.2.102:0/603269317 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:08:38.482 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:38.469 7f8072ffd700 1 --2- 172.21.2.102:0/603269317 >> v2:172.21.2.113:3300/0 conn(0x7f80740ef610 0x7f80740efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:08:38.482 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:38.469 7f80737fe700 1 -- 172.21.2.102:0/603269317 >> v2:172.21.2.113:3300/0 conn(0x7f80740ef610 msgr2=0x7f80740efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:08:38.482 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:38.469 7f80737fe700 1 --2- 172.21.2.102:0/603269317 >> v2:172.21.2.113:3300/0 conn(0x7f80740ef610 0x7f80740efa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:08:38.482 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:38.469 7f80737fe700 1 -- 172.21.2.102:0/603269317 >> v1:172.21.2.113:6789/0 conn(0x7f80740f0030 legacy=0x7f80740f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:08:38.482 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:38.469 7f80737fe700 1 -- 172.21.2.102:0/603269317 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80740f0430 con 0x7f80741b0470 2021-02-10T11:08:38.482 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:38.469 7f80737fe700 1 --2- 172.21.2.102:0/603269317 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f80741b0470 0x7f80741b0870 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 rx=0x7f8064006700 tx=0x7f8064009b00).ready entity=mon.0 client_cookie=8457b3c3a259544e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:08:38.482 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:38.469 7f80727fc700 1 -- 172.21.2.102:0/603269317 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f806400c0c0 con 0x7f80741b0470 2021-02-10T11:08:38.482 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:38.469 7f80727fc700 1 -- 172.21.2.102:0/603269317 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f806400c400 con 0x7f80741b0470 2021-02-10T11:08:38.482 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:38.469 7f80727fc700 1 -- 172.21.2.102:0/603269317 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f806400c860 con 0x7f80741b0470 2021-02-10T11:08:38.482 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:38.469 7f807a101700 1 -- 172.21.2.102:0/603269317 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f80741b0470 msgr2=0x7f80741b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:08:38.482 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:38.469 7f807a101700 1 --2- 172.21.2.102:0/603269317 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f80741b0470 0x7f80741b0870 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 rx=0x7f8064006700 tx=0x7f8064009b00).stop 2021-02-10T11:08:38.482 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:38.470 7f807a101700 1 -- 172.21.2.102:0/603269317 shutdown_connections 2021-02-10T11:08:38.482 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:38.470 7f807a101700 1 --2- 172.21.2.102:0/603269317 >> v2:172.21.2.113:3300/0 conn(0x7f80740ef610 0x7f80740efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:08:38.483 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:38.470 7f807a101700 1 --2- 172.21.2.102:0/603269317 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f80741b0470 0x7f80741b0870 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:08:38.483 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:38.470 7f807a101700 1 -- 172.21.2.102:0/603269317 >> 172.21.2.102:0/603269317 conn(0x7f80740ce3e0 msgr2=0x7f80740e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:08:38.483 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:38.470 7f807a101700 1 -- 172.21.2.102:0/603269317 shutdown_connections 2021-02-10T11:08:38.483 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:38.470 7f807a101700 1 -- 172.21.2.102:0/603269317 wait complete. 2021-02-10T11:08:38.484 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:38.485 7f807a101700 1 Processor -- start 2021-02-10T11:08:38.484 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:38.485 7f807a101700 1 -- start start 2021-02-10T11:08:38.484 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:38.485 7f807a101700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f80740a0420 0x7f80740a0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:08:38.485 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:38.485 7f807a101700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8074189d00 0x7f807418a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:08:38.485 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:38.485 7f807a101700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f807418a720 0x7f80740b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:08:38.485 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:38.485 7f807a101700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8074189800 con 0x7f80740a0420 2021-02-10T11:08:38.485 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:38.485 7f807a101700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f807418ab20 con 0x7f807418a720 2021-02-10T11:08:38.485 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:38.485 7f807a101700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80740a1140 con 0x7f8074189d00 2021-02-10T11:08:38.485 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:38.486 7f80737fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f80740a0420 0x7f80740a0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:08:38.485 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:38.486 7f80737fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f80740a0420 0x7f80740a0820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:51514/0 (socket says 172.21.2.102:51514) 2021-02-10T11:08:38.485 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:38.486 7f80737fe700 1 -- 172.21.2.102:0/3906902939 learned_addr learned my addr 172.21.2.102:0/3906902939 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:08:38.485 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:38.486 7f8073fff700 1 --2- 172.21.2.102:0/3906902939 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f807418a720 0x7f80740b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:08:38.485 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:38.486 7f8072ffd700 1 --2- 172.21.2.102:0/3906902939 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8074189d00 0x7f807418a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:08:38.485 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:38.486 7f80737fe700 1 -- 172.21.2.102:0/3906902939 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8074189d00 msgr2=0x7f807418a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:08:38.485 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:38.486 7f80737fe700 1 --2- 172.21.2.102:0/3906902939 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8074189d00 0x7f807418a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:08:38.486 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:38.486 7f80737fe700 1 -- 172.21.2.102:0/3906902939 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f807418a720 msgr2=0x7f80740b7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:08:38.486 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:38.486 7f80737fe700 1 --2- 172.21.2.102:0/3906902939 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f807418a720 0x7f80740b7be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:08:38.486 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:38.486 7f80737fe700 1 -- 172.21.2.102:0/3906902939 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80740b8200 con 0x7f80740a0420 2021-02-10T11:08:38.486 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:38.486 7f80737fe700 1 --2- 172.21.2.102:0/3906902939 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f80740a0420 0x7f80740a0820 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 rx=0x7f806400b660 tx=0x7f806400ba00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:08:38.486 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:38.486 7f806bfff700 1 -- 172.21.2.102:0/3906902939 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8064009440 con 0x7f80740a0420 2021-02-10T11:08:38.486 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:38.486 7f806bfff700 1 -- 172.21.2.102:0/3906902939 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80640090c0 con 0x7f80740a0420 2021-02-10T11:08:38.486 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:38.486 7f806bfff700 1 -- 172.21.2.102:0/3906902939 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f80640054d0 con 0x7f80740a0420 2021-02-10T11:08:38.486 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:38.486 7f807a101700 1 -- 172.21.2.102:0/3906902939 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80740b8200 con 0x7f80740a0420 2021-02-10T11:08:38.486 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:38.486 7f807a101700 1 -- 172.21.2.102:0/3906902939 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80740b8790 con 0x7f80740a0420 2021-02-10T11:08:38.486 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:38.487 7f806bfff700 1 -- 172.21.2.102:0/3906902939 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f80640093e0 con 0x7f80740a0420 2021-02-10T11:08:38.486 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:38.487 7f806bfff700 1 --2- 172.21.2.102:0/3906902939 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f80580186c0 0x7f805801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:08:38.486 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:38.487 7f806bfff700 1 -- 172.21.2.102:0/3906902939 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f8064032430 con 0x7f80740a0420 2021-02-10T11:08:38.487 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:38.487 7f8072ffd700 1 --2- 172.21.2.102:0/3906902939 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f80580186c0 0x7f805801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:08:38.487 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:38.488 7f8072ffd700 1 --2- 172.21.2.102:0/3906902939 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f80580186c0 0x7f805801abc0 crc :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:08:38.514 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:38.518 7f807a101700 1 -- 172.21.2.102:0/3906902939 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8074105ae0 con 0x7f80740a0420 2021-02-10T11:08:38.516 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:38.520 7f806bfff700 1 -- 172.21.2.102:0/3906902939 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f8064030d60 con 0x7f80740a0420 2021-02-10T11:08:38.643 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:38.646 7f807a101700 1 -- 172.21.2.102:0/3906902939 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344385", "target": ["mgr", ""]}) v1 -- 0x7f80740b8790 con 0x7f80580186c0 2021-02-10T11:08:38.669 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:38.672 7f806bfff700 1 -- 172.21.2.102:0/3906902939 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f805c0009b0 con 0x7f80580186c0 2021-02-10T11:08:38.677 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:38.679 7f807a101700 1 -- 172.21.2.102:0/3906902939 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f80580186c0 msgr2=0x7f805801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:08:38.677 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:38.679 7f807a101700 1 --2- 172.21.2.102:0/3906902939 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f80580186c0 0x7f805801abc0 crc :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:08:38.677 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:38.679 7f807a101700 1 -- 172.21.2.102:0/3906902939 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f80740a0420 msgr2=0x7f80740a0820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:08:38.677 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:38.679 7f807a101700 1 --2- 172.21.2.102:0/3906902939 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f80740a0420 0x7f80740a0820 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 rx=0x7f806400b660 tx=0x7f806400ba00).stop 2021-02-10T11:08:38.677 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:38.680 7f807a101700 1 -- 172.21.2.102:0/3906902939 shutdown_connections 2021-02-10T11:08:38.678 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:38.680 7f807a101700 1 --2- 172.21.2.102:0/3906902939 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f80580186c0 0x7f805801abc0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:08:38.678 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:38.680 7f807a101700 1 --2- 172.21.2.102:0/3906902939 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f807418a720 0x7f80740b7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:08:38.678 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:38.680 7f807a101700 1 --2- 172.21.2.102:0/3906902939 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8074189d00 0x7f807418a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:08:38.678 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:38.680 7f807a101700 1 --2- 172.21.2.102:0/3906902939 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f80740a0420 0x7f80740a0820 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:08:38.678 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:38.680 7f807a101700 1 -- 172.21.2.102:0/3906902939 >> 172.21.2.102:0/3906902939 conn(0x7f8074094e10 msgr2=0x7f80740a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:08:38.678 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:38.680 7f807a101700 1 -- 172.21.2.102:0/3906902939 shutdown_connections 2021-02-10T11:08:38.678 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:38.680 7f807a101700 1 -- 172.21.2.102:0/3906902939 wait complete. 2021-02-10T11:08:38.684 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000344385 2021-02-10T11:08:39.118 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:39.106 7f522a651700 1 Processor -- start 2021-02-10T11:08:39.119 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:39.106 7f522a651700 1 -- start start 2021-02-10T11:08:39.119 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:39.106 7f522a651700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f52241b0470 0x7f52241b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:08:39.119 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:39.106 7f522a651700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f52240ef610 0x7f52240efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:08:39.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:39.106 7f522a651700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f52240f0030 0x7f52240f0430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:08:39.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:39.106 7f522a651700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52240e5ea0 con 0x7f52241b0470 2021-02-10T11:08:39.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:39.106 7f522a651700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52240f0b50 con 0x7f52240ef610 2021-02-10T11:08:39.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:39.106 7f522a651700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52240e52e0 con 0x7f52240f0030 2021-02-10T11:08:39.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:39.107 7f5223fff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f52241b0470 0x7f52241b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:08:39.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:39.107 7f5223fff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f52241b0470 0x7f52241b0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:51526/0 (socket says 172.21.2.102:51526) 2021-02-10T11:08:39.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:39.107 7f5223fff700 1 -- 172.21.2.102:0/1671079660 learned_addr learned my addr 172.21.2.102:0/1671079660 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:08:39.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:39.107 7f5223fff700 1 -- 172.21.2.102:0/1671079660 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f52240f0030 msgr2=0x7f52240f0430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:08:39.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:39.107 7f5223fff700 1 --2- 172.21.2.102:0/1671079660 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f52240f0030 0x7f52240f0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:08:39.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:39.107 7f5223fff700 1 -- 172.21.2.102:0/1671079660 >> v2:172.21.2.113:3300/0 conn(0x7f52240ef610 msgr2=0x7f52240efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:08:39.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:39.107 7f5223fff700 1 --2- 172.21.2.102:0/1671079660 >> v2:172.21.2.113:3300/0 conn(0x7f52240ef610 0x7f52240efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:08:39.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:39.107 7f5223fff700 1 -- 172.21.2.102:0/1671079660 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52240f10d0 con 0x7f52241b0470 2021-02-10T11:08:39.121 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:39.107 7f5223fff700 1 --2- 172.21.2.102:0/1671079660 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f52241b0470 0x7f52241b0870 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 rx=0x7f521c006a80 tx=0x7f521c009f00).ready entity=mon.0 client_cookie=fa0117b453760a41 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:08:39.121 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:39.107 7f5222ffd700 1 -- 172.21.2.102:0/1671079660 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f521c00b5c0 con 0x7f52241b0470 2021-02-10T11:08:39.121 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:39.107 7f5222ffd700 1 -- 172.21.2.102:0/1671079660 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f521c00b8c0 con 0x7f52241b0470 2021-02-10T11:08:39.121 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:39.107 7f5222ffd700 1 -- 172.21.2.102:0/1671079660 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f521c00bd20 con 0x7f52241b0470 2021-02-10T11:08:39.121 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:39.107 7f522a651700 1 -- 172.21.2.102:0/1671079660 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f52241b0470 msgr2=0x7f52241b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:08:39.121 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:39.107 7f522a651700 1 --2- 172.21.2.102:0/1671079660 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f52241b0470 0x7f52241b0870 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 rx=0x7f521c006a80 tx=0x7f521c009f00).stop 2021-02-10T11:08:39.121 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:39.108 7f522a651700 1 -- 172.21.2.102:0/1671079660 shutdown_connections 2021-02-10T11:08:39.121 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:39.108 7f522a651700 1 --2- 172.21.2.102:0/1671079660 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f52240f0030 0x7f52240f0430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:08:39.121 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:39.108 7f522a651700 1 --2- 172.21.2.102:0/1671079660 >> v2:172.21.2.113:3300/0 conn(0x7f52240ef610 0x7f52240efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:08:39.121 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:39.108 7f522a651700 1 --2- 172.21.2.102:0/1671079660 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f52241b0470 0x7f52241b0870 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:08:39.121 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:39.108 7f522a651700 1 -- 172.21.2.102:0/1671079660 >> 172.21.2.102:0/1671079660 conn(0x7f52240ce3e0 msgr2=0x7f52240e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:08:39.121 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:39.108 7f522a651700 1 -- 172.21.2.102:0/1671079660 shutdown_connections 2021-02-10T11:08:39.121 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:39.108 7f522a651700 1 -- 172.21.2.102:0/1671079660 wait complete. 2021-02-10T11:08:39.121 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:39.123 7f522a651700 1 Processor -- start 2021-02-10T11:08:39.122 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:39.123 7f522a651700 1 -- start start 2021-02-10T11:08:39.123 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:39.123 7f522a651700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f52240b20b0 0x7f52240ac970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:08:39.123 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:39.123 7f522a651700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f52240b24b0 0x7f5224191dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:08:39.123 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:39.123 7f522a651700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f52240acf90 0x7f52241923e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:08:39.123 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:39.123 7f522a651700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52240b1b30 con 0x7f52240acf90 2021-02-10T11:08:39.123 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:39.123 7f522a651700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52241a04e0 con 0x7f52240b20b0 2021-02-10T11:08:39.123 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:39.123 7f522a651700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52241a06c0 con 0x7f52240b24b0 2021-02-10T11:08:39.123 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:39.124 7f5228c6e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f52240acf90 0x7f52241923e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:08:39.123 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:39.124 7f5228c6e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f52240acf90 0x7f52241923e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:51530/0 (socket says 172.21.2.102:51530) 2021-02-10T11:08:39.123 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:39.124 7f5228c6e700 1 -- 172.21.2.102:0/2066649057 learned_addr learned my addr 172.21.2.102:0/2066649057 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:08:39.123 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:39.124 7f5228c6e700 1 -- 172.21.2.102:0/2066649057 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f52240b24b0 msgr2=0x7f5224191dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:08:39.124 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:39.124 7f5228c6e700 1 --2- 172.21.2.102:0/2066649057 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f52240b24b0 0x7f5224191dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:08:39.124 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:39.124 7f5228c6e700 1 -- 172.21.2.102:0/2066649057 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f52240b20b0 msgr2=0x7f52240ac970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:08:39.124 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:39.124 7f5228c6e700 1 --2- 172.21.2.102:0/2066649057 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f52240b20b0 0x7f52240ac970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:08:39.124 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:39.124 7f5228c6e700 1 -- 172.21.2.102:0/2066649057 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5224192be0 con 0x7f52240acf90 2021-02-10T11:08:39.124 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:39.124 7f5228c6e700 1 --2- 172.21.2.102:0/2066649057 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f52240acf90 0x7f52241923e0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 rx=0x7f5214006f10 tx=0x7f521400b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:08:39.124 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:39.124 7f5220ff9700 1 -- 172.21.2.102:0/2066649057 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5214006240 con 0x7f52240acf90 2021-02-10T11:08:39.124 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:39.124 7f5220ff9700 1 -- 172.21.2.102:0/2066649057 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5214000c10 con 0x7f52240acf90 2021-02-10T11:08:39.124 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:39.124 7f5220ff9700 1 -- 172.21.2.102:0/2066649057 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f52140054f0 con 0x7f52240acf90 2021-02-10T11:08:39.124 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:39.124 7f522a651700 1 -- 172.21.2.102:0/2066649057 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5224192be0 con 0x7f52240acf90 2021-02-10T11:08:39.124 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:39.124 7f522a651700 1 -- 172.21.2.102:0/2066649057 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5224193170 con 0x7f52240acf90 2021-02-10T11:08:39.125 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:39.125 7f5220ff9700 1 -- 172.21.2.102:0/2066649057 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f5214006030 con 0x7f52240acf90 2021-02-10T11:08:39.125 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:39.125 7f5220ff9700 1 --2- 172.21.2.102:0/2066649057 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f52040186c0 0x7f520401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:08:39.125 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:39.125 7f5220ff9700 1 -- 172.21.2.102:0/2066649057 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f5214032560 con 0x7f52240acf90 2021-02-10T11:08:39.125 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:39.125 7f5223fff700 1 --2- 172.21.2.102:0/2066649057 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f52040186c0 0x7f520401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:08:39.125 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:39.125 7f5223fff700 1 --2- 172.21.2.102:0/2066649057 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f52040186c0 0x7f520401abc0 crc :-1 s=READY pgs=215 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:08:39.152 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:39.156 7f522a651700 1 -- 172.21.2.102:0/2066649057 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5224105af0 con 0x7f52240acf90 2021-02-10T11:08:39.154 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:39.158 7f5220ff9700 1 -- 172.21.2.102:0/2066649057 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f521402c8e0 con 0x7f52240acf90 2021-02-10T11:08:39.281 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:39.284 7f522a651700 1 -- 172.21.2.102:0/2066649057 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344385", "target": ["mgr", ""]}) v1 -- 0x7f52241a06c0 con 0x7f52040186c0 2021-02-10T11:08:39.285 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:39.289 7f5220ff9700 1 -- 172.21.2.102:0/2066649057 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f521c006e80 con 0x7f52040186c0 2021-02-10T11:08:39.288 INFO:teuthology.orchestra.run.gibba002.stdout:/volumes/_nogroup/subvolume_0000000000344385/0089d263-9343-43e0-893c-ae15ef6625d1 2021-02-10T11:08:39.289 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:39.293 7f522a651700 1 -- 172.21.2.102:0/2066649057 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f52040186c0 msgr2=0x7f520401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:08:39.289 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:39.293 7f522a651700 1 --2- 172.21.2.102:0/2066649057 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f52040186c0 0x7f520401abc0 crc :-1 s=READY pgs=215 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:08:39.290 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:39.293 7f522a651700 1 -- 172.21.2.102:0/2066649057 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f52240acf90 msgr2=0x7f52241923e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:08:39.290 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:39.293 7f522a651700 1 --2- 172.21.2.102:0/2066649057 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f52240acf90 0x7f52241923e0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 rx=0x7f5214006f10 tx=0x7f521400b990).stop 2021-02-10T11:08:39.290 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:39.293 7f522a651700 1 -- 172.21.2.102:0/2066649057 shutdown_connections 2021-02-10T11:08:39.290 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:39.293 7f522a651700 1 --2- 172.21.2.102:0/2066649057 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f52240acf90 0x7f52241923e0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:08:39.290 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:39.293 7f522a651700 1 --2- 172.21.2.102:0/2066649057 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f52240b24b0 0x7f5224191dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:08:39.290 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:39.293 7f522a651700 1 --2- 172.21.2.102:0/2066649057 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f52040186c0 0x7f520401abc0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:08:39.290 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:39.293 7f522a651700 1 --2- 172.21.2.102:0/2066649057 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f52240b20b0 0x7f52240ac970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:08:39.290 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:39.293 7f522a651700 1 -- 172.21.2.102:0/2066649057 >> 172.21.2.102:0/2066649057 conn(0x7f52240bf4e0 msgr2=0x7f52240be3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:08:39.291 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:39.294 7f522a651700 1 -- 172.21.2.102:0/2066649057 shutdown_connections 2021-02-10T11:08:39.291 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:39.294 7f522a651700 1 -- 172.21.2.102:0/2066649057 wait complete. 2021-02-10T11:08:39.298 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000344385 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000344385/0089d263-9343-43e0-893c-ae15ef6625d1 2021-02-10T11:08:39.298 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:08:39.298 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:08:39.298 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:08:39.634 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:08:39.635 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:08:39.635 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344385/0089d263-9343-43e0-893c-ae15ef6625d1/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:08:40.017 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:08:40.018 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:08:40.018 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0230346 s, 45.5 MB/s 2021-02-10T11:08:40.019 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:08:40.019 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:08:40.019 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:08:40.355 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:08:40.355 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:08:40.356 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344385/0089d263-9343-43e0-893c-ae15ef6625d1/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:08:40.730 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:08:40.731 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:08:40.731 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0234314 s, 44.8 MB/s 2021-02-10T11:08:40.731 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:08:40.732 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:08:40.732 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:08:41.067 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:08:41.067 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:08:41.068 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344385/0089d263-9343-43e0-893c-ae15ef6625d1/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:08:41.443 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:08:41.443 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:08:41.443 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0243513 s, 43.1 MB/s 2021-02-10T11:08:41.444 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:08:41.444 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:08:41.444 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:08:41.780 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:08:41.781 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:08:41.781 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344385/0089d263-9343-43e0-893c-ae15ef6625d1/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:08:42.154 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:08:42.154 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:08:42.154 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.022778 s, 46.0 MB/s 2021-02-10T11:08:42.156 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:08:42.156 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:08:42.156 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:08:42.491 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:08:42.491 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:08:42.492 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344385/0089d263-9343-43e0-893c-ae15ef6625d1/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:08:42.866 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:08:42.867 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:08:42.867 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0243404 s, 43.1 MB/s 2021-02-10T11:08:42.868 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:08:42.869 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:08:42.869 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:08:43.205 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:08:43.205 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:08:43.205 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344385/0089d263-9343-43e0-893c-ae15ef6625d1/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:08:43.576 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:08:43.576 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:08:43.576 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0234039 s, 44.8 MB/s 2021-02-10T11:08:43.577 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:08:43.577 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:08:43.577 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:08:43.912 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:08:43.912 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:08:43.912 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344385/0089d263-9343-43e0-893c-ae15ef6625d1/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:08:44.288 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:08:44.288 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:08:44.288 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0248048 s, 42.3 MB/s 2021-02-10T11:08:44.289 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:08:44.290 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:08:44.290 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:08:44.606 DEBUG:teuthology.orchestra.run.gibba002:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:08:44.610 DEBUG:teuthology.orchestra.run.gibba013:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:08:44.625 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:08:44.625 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:08:44.625 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344385/0089d263-9343-43e0-893c-ae15ef6625d1/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:08:44.982 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:08:44.982 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:08:44.982 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0234174 s, 44.8 MB/s 2021-02-10T11:08:44.983 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:08:44.983 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:08:44.983 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:08:45.358 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:08:45.358 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:08:45.358 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344385/0089d263-9343-43e0-893c-ae15ef6625d1/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:08:45.732 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:08:45.733 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:08:45.733 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0231961 s, 45.2 MB/s 2021-02-10T11:08:45.734 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:08:45.735 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:08:45.735 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:08:46.071 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:08:46.071 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:08:46.071 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344385/0089d263-9343-43e0-893c-ae15ef6625d1/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:08:46.445 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:08:46.446 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:08:46.446 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0228569 s, 45.9 MB/s 2021-02-10T11:08:46.447 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000344386 2021-02-10T11:08:46.878 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:46.865 7f466e78f700 1 Processor -- start 2021-02-10T11:08:46.879 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:46.865 7f466e78f700 1 -- start start 2021-02-10T11:08:46.879 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:46.866 7f466e78f700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f46681b0470 0x7f46681b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:08:46.879 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:46.866 7f466e78f700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46680f0b50 con 0x7f46680f20e0 2021-02-10T11:08:46.879 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:46.866 7f466e78f700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46680f24e0 con 0x7f46680ef610 2021-02-10T11:08:46.879 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:46.866 7f466e78f700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46680e5ea0 con 0x7f46681b0470 2021-02-10T11:08:46.879 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:46.866 7f466cdac700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f46680f20e0 0x7f46680f0200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:43762/0 (socket says 172.21.2.102:43762) 2021-02-10T11:08:46.879 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:46.866 7f466cdac700 1 -- 172.21.2.102:0/3367518999 learned_addr learned my addr 172.21.2.102:0/3367518999 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:08:46.879 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:46.866 7f4667fff700 1 --2- 172.21.2.102:0/3367518999 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f46681b0470 0x7f46681b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:08:46.879 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:46.866 7f4666ffd700 1 -- 172.21.2.102:0/3367518999 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2311345358 0 0) 0x7f4660001500 con 0x7f46680f20e0 2021-02-10T11:08:46.879 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:46.866 7f4666ffd700 1 -- 172.21.2.102:0/3367518999 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4650002fe0 con 0x7f46680f20e0 2021-02-10T11:08:46.879 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:46.866 7f4666ffd700 1 -- 172.21.2.102:0/3367518999 <== mon.1 v1:172.21.2.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2370325835 0 0) 0x7f465c000f30 con 0x7f46680ef610 2021-02-10T11:08:46.880 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:46.866 7f4666ffd700 1 -- 172.21.2.102:0/3367518999 --> v1:172.21.2.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46500042c0 con 0x7f46680ef610 2021-02-10T11:08:46.880 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:46.867 7f4667fff700 1 -- 172.21.2.102:0/3367518999 >> v1:172.21.2.113:6789/0 conn(0x7f46680ef610 legacy=0x7f46680efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:08:46.880 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:46.867 7f4667fff700 1 -- 172.21.2.102:0/3367518999 >> v1:172.21.2.102:6789/0 conn(0x7f46680f20e0 legacy=0x7f46680f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:08:46.880 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:46.867 7f4667fff700 1 -- 172.21.2.102:0/3367518999 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46680e52e0 con 0x7f46681b0470 2021-02-10T11:08:46.880 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:46.867 7f4667fff700 1 --2- 172.21.2.102:0/3367518999 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f46681b0470 0x7f46681b0870 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7f4658006650 tx=0x7f4658009a00).ready entity=mon.2 client_cookie=6760378d1e3ee80a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:08:46.880 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:46.867 7f4666ffd700 1 -- 172.21.2.102:0/3367518999 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4658009df0 con 0x7f46681b0470 2021-02-10T11:08:46.880 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:46.867 7f4666ffd700 1 -- 172.21.2.102:0/3367518999 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46580009b0 con 0x7f46681b0470 2021-02-10T11:08:46.880 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:46.867 7f4666ffd700 1 -- 172.21.2.102:0/3367518999 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f465800c2b0 con 0x7f46681b0470 2021-02-10T11:08:46.880 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:46.867 7f466e78f700 1 -- 172.21.2.102:0/3367518999 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f46681b0470 msgr2=0x7f46681b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:08:46.880 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:46.867 7f466e78f700 1 --2- 172.21.2.102:0/3367518999 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f46681b0470 0x7f46681b0870 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7f4658006650 tx=0x7f4658009a00).stop 2021-02-10T11:08:46.880 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:46.868 7f466e78f700 1 -- 172.21.2.102:0/3367518999 shutdown_connections 2021-02-10T11:08:46.880 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:46.868 7f466e78f700 1 --2- 172.21.2.102:0/3367518999 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f46681b0470 0x7f46681b0870 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:08:46.880 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:46.868 7f466e78f700 1 -- 172.21.2.102:0/3367518999 >> 172.21.2.102:0/3367518999 conn(0x7f46680ce3e0 msgr2=0x7f46680e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:08:46.881 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:46.868 7f466e78f700 1 -- 172.21.2.102:0/3367518999 shutdown_connections 2021-02-10T11:08:46.881 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:46.868 7f466e78f700 1 -- 172.21.2.102:0/3367518999 wait complete. 2021-02-10T11:08:46.882 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:46.883 7f466e78f700 1 Processor -- start 2021-02-10T11:08:46.882 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:46.883 7f466e78f700 1 -- start start 2021-02-10T11:08:46.883 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:46.883 7f466e78f700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f46680a0420 0x7f46680a0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:08:46.883 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:46.883 7f466e78f700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4668189d00 0x7f466818a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:08:46.883 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:46.883 7f466e78f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f466818a720 0x7f46680b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:08:46.883 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:46.883 7f466e78f700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4668189780 con 0x7f466818a720 2021-02-10T11:08:46.883 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:46.883 7f466e78f700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f466818ab20 con 0x7f4668189d00 2021-02-10T11:08:46.883 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:46.883 7f466e78f700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46680a1140 con 0x7f46680a0420 2021-02-10T11:08:46.883 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:46.883 7f466cdac700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f466818a720 0x7f46680b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:08:46.883 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:46.883 7f466cdac700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f466818a720 0x7f46680b7be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:51542/0 (socket says 172.21.2.102:51542) 2021-02-10T11:08:46.883 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:46.883 7f466cdac700 1 -- 172.21.2.102:0/316980671 learned_addr learned my addr 172.21.2.102:0/316980671 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:08:46.883 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:46.883 7f46677fe700 1 --2- 172.21.2.102:0/316980671 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4668189d00 0x7f466818a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:08:46.883 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:46.884 7f4667fff700 1 --2- 172.21.2.102:0/316980671 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f46680a0420 0x7f46680a0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:08:46.883 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:46.884 7f466cdac700 1 -- 172.21.2.102:0/316980671 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f46680a0420 msgr2=0x7f46680a0820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:08:46.883 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:46.884 7f466cdac700 1 --2- 172.21.2.102:0/316980671 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f46680a0420 0x7f46680a0820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:08:46.884 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:46.884 7f466cdac700 1 -- 172.21.2.102:0/316980671 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4668189d00 msgr2=0x7f466818a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:08:46.884 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:46.884 7f466cdac700 1 --2- 172.21.2.102:0/316980671 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4668189d00 0x7f466818a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:08:46.884 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:46.884 7f466cdac700 1 -- 172.21.2.102:0/316980671 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46680b8200 con 0x7f466818a720 2021-02-10T11:08:46.884 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:46.884 7f46677fe700 1 --2- 172.21.2.102:0/316980671 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4668189d00 0x7f466818a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:08:46.884 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:46.884 7f4667fff700 1 --2- 172.21.2.102:0/316980671 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f46680a0420 0x7f46680a0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:08:46.884 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:46.884 7f466cdac700 1 --2- 172.21.2.102:0/316980671 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f466818a720 0x7f46680b7be0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 rx=0x7f4660009980 tx=0x7f466000bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:08:46.884 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:46.884 7f4664ff9700 1 -- 172.21.2.102:0/316980671 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4660004720 con 0x7f466818a720 2021-02-10T11:08:46.884 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:46.884 7f4664ff9700 1 -- 172.21.2.102:0/316980671 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4660000d30 con 0x7f466818a720 2021-02-10T11:08:46.884 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:46.884 7f4664ff9700 1 -- 172.21.2.102:0/316980671 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4660004b90 con 0x7f466818a720 2021-02-10T11:08:46.884 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:46.884 7f466e78f700 1 -- 172.21.2.102:0/316980671 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46680b8200 con 0x7f466818a720 2021-02-10T11:08:46.884 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:46.884 7f466e78f700 1 -- 172.21.2.102:0/316980671 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46680b8200 con 0x7f466818a720 2021-02-10T11:08:46.884 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:46.885 7f4664ff9700 1 -- 172.21.2.102:0/316980671 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4660001270 con 0x7f466818a720 2021-02-10T11:08:46.884 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:46.885 7f4664ff9700 1 --2- 172.21.2.102:0/316980671 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4648018730 0x7f464801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:08:46.885 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:46.885 7f4664ff9700 1 -- 172.21.2.102:0/316980671 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f4660032970 con 0x7f466818a720 2021-02-10T11:08:46.885 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:46.885 7f4667fff700 1 --2- 172.21.2.102:0/316980671 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4648018730 0x7f464801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:08:46.885 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:46.885 7f4667fff700 1 --2- 172.21.2.102:0/316980671 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4648018730 0x7f464801ac10 crc :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:08:46.912 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:46.916 7f466e78f700 1 -- 172.21.2.102:0/316980671 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4668105ae0 con 0x7f466818a720 2021-02-10T11:08:46.914 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:46.918 7f4664ff9700 1 -- 172.21.2.102:0/316980671 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f466000be10 con 0x7f466818a720 2021-02-10T11:08:47.039 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:47.043 7f466e78f700 1 -- 172.21.2.102:0/316980671 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344386", "target": ["mgr", ""]}) v1 -- 0x7f46680f0b50 con 0x7f4648018730 2021-02-10T11:08:47.065 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:47.069 7f4664ff9700 1 -- 172.21.2.102:0/316980671 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f4658006540 con 0x7f4648018730 2021-02-10T11:08:47.072 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:47.076 7f466e78f700 1 -- 172.21.2.102:0/316980671 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4648018730 msgr2=0x7f464801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:08:47.072 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:47.076 7f466e78f700 1 --2- 172.21.2.102:0/316980671 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4648018730 0x7f464801ac10 crc :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:08:47.072 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:47.076 7f466e78f700 1 -- 172.21.2.102:0/316980671 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f466818a720 msgr2=0x7f46680b7be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:08:47.073 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:47.076 7f466e78f700 1 --2- 172.21.2.102:0/316980671 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f466818a720 0x7f46680b7be0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 rx=0x7f4660009980 tx=0x7f466000bcd0).stop 2021-02-10T11:08:47.073 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:47.076 7f466e78f700 1 -- 172.21.2.102:0/316980671 shutdown_connections 2021-02-10T11:08:47.073 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:47.076 7f466e78f700 1 --2- 172.21.2.102:0/316980671 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f466818a720 0x7f46680b7be0 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:08:47.073 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:47.076 7f466e78f700 1 --2- 172.21.2.102:0/316980671 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4648018730 0x7f464801ac10 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:08:47.073 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:47.076 7f466e78f700 1 --2- 172.21.2.102:0/316980671 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4668189d00 0x7f466818a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:08:47.073 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:47.076 7f466e78f700 1 --2- 172.21.2.102:0/316980671 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f46680a0420 0x7f46680a0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:08:47.073 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:47.076 7f466e78f700 1 -- 172.21.2.102:0/316980671 >> 172.21.2.102:0/316980671 conn(0x7f4668094e10 msgr2=0x7f46680a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:08:47.074 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:47.077 7f466e78f700 1 -- 172.21.2.102:0/316980671 shutdown_connections 2021-02-10T11:08:47.074 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:47.077 7f466e78f700 1 -- 172.21.2.102:0/316980671 wait complete. 2021-02-10T11:08:47.080 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000344386 2021-02-10T11:08:47.511 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:47.498 7f2fc725a700 1 Processor -- start 2021-02-10T11:08:47.511 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:47.498 7f2fc725a700 1 -- start start 2021-02-10T11:08:47.511 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:47.499 7f2fc725a700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f2fc00f05f0 0x7f2fc00f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:08:47.511 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:47.499 7f2fc725a700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fc01b0c70 con 0x7f2fc01b0470 2021-02-10T11:08:47.511 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:47.499 7f2fc725a700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fc00f0b50 con 0x7f2fc01b0870 2021-02-10T11:08:47.511 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:47.499 7f2fc725a700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fc00e5ea0 con 0x7f2fc00f05f0 2021-02-10T11:08:47.511 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:47.499 7f2fc5877700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f2fc00f05f0 0x7f2fc00f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:08:47.511 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:47.499 7f2fc5076700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f2fc01b0470 0x7f2fc00ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:43772/0 (socket says 172.21.2.102:43772) 2021-02-10T11:08:47.511 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:47.499 7f2fc5076700 1 -- 172.21.2.102:0/3393363820 learned_addr learned my addr 172.21.2.102:0/3393363820 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:08:47.511 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:47.499 7f2fc5877700 1 -- 172.21.2.102:0/3393363820 >> v1:172.21.2.113:6789/0 conn(0x7f2fc01b0870 legacy=0x7f2fc00efe00 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:08:47.512 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:47.499 7f2fc5877700 1 -- 172.21.2.102:0/3393363820 >> v1:172.21.2.102:6789/0 conn(0x7f2fc01b0470 legacy=0x7f2fc00ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:08:47.512 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:47.499 7f2fc5877700 1 -- 172.21.2.102:0/3393363820 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fc00e52e0 con 0x7f2fc00f05f0 2021-02-10T11:08:47.512 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:47.499 7f2fc5877700 1 --2- 172.21.2.102:0/3393363820 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2fc00f05f0 0x7f2fc00f20e0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 rx=0x7f2fb8006e30 tx=0x7f2fb8009e60).ready entity=mon.0 client_cookie=9a229264beea69d8 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:08:47.512 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:47.499 7f2fb3fff700 1 -- 172.21.2.102:0/3393363820 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2fb8003030 con 0x7f2fc00f05f0 2021-02-10T11:08:47.512 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:47.500 7f2fb3fff700 1 -- 172.21.2.102:0/3393363820 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fb800b300 con 0x7f2fc00f05f0 2021-02-10T11:08:47.512 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:47.500 7f2fb3fff700 1 -- 172.21.2.102:0/3393363820 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2fb800b7e0 con 0x7f2fc00f05f0 2021-02-10T11:08:47.512 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:47.500 7f2fc725a700 1 -- 172.21.2.102:0/3393363820 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2fc00f05f0 msgr2=0x7f2fc00f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:08:47.512 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:47.500 7f2fc725a700 1 --2- 172.21.2.102:0/3393363820 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2fc00f05f0 0x7f2fc00f20e0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 rx=0x7f2fb8006e30 tx=0x7f2fb8009e60).stop 2021-02-10T11:08:47.512 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:47.500 7f2fc725a700 1 -- 172.21.2.102:0/3393363820 shutdown_connections 2021-02-10T11:08:47.512 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:47.500 7f2fc725a700 1 --2- 172.21.2.102:0/3393363820 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2fc00f05f0 0x7f2fc00f20e0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:08:47.512 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:47.500 7f2fc725a700 1 -- 172.21.2.102:0/3393363820 >> 172.21.2.102:0/3393363820 conn(0x7f2fc00ce3e0 msgr2=0x7f2fc00e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:08:47.512 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:47.500 7f2fc725a700 1 -- 172.21.2.102:0/3393363820 shutdown_connections 2021-02-10T11:08:47.513 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:47.500 7f2fc725a700 1 -- 172.21.2.102:0/3393363820 wait complete. 2021-02-10T11:08:47.513 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:47.515 7f2fc725a700 1 Processor -- start 2021-02-10T11:08:47.513 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:47.515 7f2fc725a700 1 -- start start 2021-02-10T11:08:47.513 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:47.516 7f2fc725a700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2fc0082170 0x7f2fc00a6400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:08:47.514 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:47.516 7f2fc725a700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2fc00894a0 0x7f2fc00898a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:08:47.514 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:47.516 7f2fc725a700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2fc0082f20 0x7f2fc0083320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:08:47.514 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:47.516 7f2fc725a700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fc0089ec0 con 0x7f2fc0082170 2021-02-10T11:08:47.514 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:47.516 7f2fc725a700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fc0087fa0 con 0x7f2fc0082f20 2021-02-10T11:08:47.514 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:47.516 7f2fc725a700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fc0088520 con 0x7f2fc00894a0 2021-02-10T11:08:47.515 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:47.516 7f2fc5076700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2fc0082170 0x7f2fc00a6400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:08:47.515 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:47.516 7f2fc5076700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2fc0082170 0x7f2fc00a6400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:51554/0 (socket says 172.21.2.102:51554) 2021-02-10T11:08:47.516 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:47.516 7f2fc5076700 1 -- 172.21.2.102:0/4203901591 learned_addr learned my addr 172.21.2.102:0/4203901591 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:08:47.516 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:47.516 7f2fc5076700 1 -- 172.21.2.102:0/4203901591 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2fc00894a0 msgr2=0x7f2fc00898a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:08:47.516 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:47.516 7f2fc5076700 1 --2- 172.21.2.102:0/4203901591 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2fc00894a0 0x7f2fc00898a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:08:47.516 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:47.516 7f2fc5076700 1 -- 172.21.2.102:0/4203901591 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2fc0082f20 msgr2=0x7f2fc0083320 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:08:47.516 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:47.516 7f2fc5076700 1 --2- 172.21.2.102:0/4203901591 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2fc0082f20 0x7f2fc0083320 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:08:47.516 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:47.516 7f2fc5076700 1 -- 172.21.2.102:0/4203901591 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fc00a6a20 con 0x7f2fc0082170 2021-02-10T11:08:47.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:47.516 7f2fc5076700 1 --2- 172.21.2.102:0/4203901591 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2fc0082170 0x7f2fc00a6400 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 rx=0x7f2fbc001c60 tx=0x7f2fbc00a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:08:47.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:47.517 7f2fb1ffb700 1 -- 172.21.2.102:0/4203901591 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2fbc00acc0 con 0x7f2fc0082170 2021-02-10T11:08:47.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:47.517 7f2fb1ffb700 1 -- 172.21.2.102:0/4203901591 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fbc004140 con 0x7f2fc0082170 2021-02-10T11:08:47.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:47.517 7f2fb1ffb700 1 -- 172.21.2.102:0/4203901591 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2fbc004660 con 0x7f2fc0082170 2021-02-10T11:08:47.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:47.517 7f2fc725a700 1 -- 172.21.2.102:0/4203901591 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fc00a6a20 con 0x7f2fc0082170 2021-02-10T11:08:47.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:47.517 7f2fc725a700 1 -- 172.21.2.102:0/4203901591 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fc00a6a20 con 0x7f2fc0082170 2021-02-10T11:08:47.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:47.517 7f2fb1ffb700 1 -- 172.21.2.102:0/4203901591 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f2fbc000db0 con 0x7f2fc0082170 2021-02-10T11:08:47.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:47.517 7f2fb1ffb700 1 --2- 172.21.2.102:0/4203901591 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2fa00186c0 0x7f2fa001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:08:47.518 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:47.517 7f2fb1ffb700 1 -- 172.21.2.102:0/4203901591 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f2fbc032560 con 0x7f2fc0082170 2021-02-10T11:08:47.518 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:47.517 7f2fc4875700 1 --2- 172.21.2.102:0/4203901591 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2fa00186c0 0x7f2fa001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:08:47.518 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:47.518 7f2fc4875700 1 --2- 172.21.2.102:0/4203901591 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2fa00186c0 0x7f2fa001abc0 crc :-1 s=READY pgs=217 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:08:47.545 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:47.549 7f2fc725a700 1 -- 172.21.2.102:0/4203901591 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fc0089ec0 con 0x7f2fc0082170 2021-02-10T11:08:47.547 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:47.551 7f2fb1ffb700 1 -- 172.21.2.102:0/4203901591 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f2fbc0009b0 con 0x7f2fc0082170 2021-02-10T11:08:47.677 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:47.680 7f2fc725a700 1 -- 172.21.2.102:0/4203901591 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344386", "target": ["mgr", ""]}) v1 -- 0x7f2fc0089ec0 con 0x7f2fa00186c0 2021-02-10T11:08:47.679 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:47.683 7f2fb1ffb700 1 -- 172.21.2.102:0/4203901591 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f2fb4006e40 con 0x7f2fa00186c0 2021-02-10T11:08:47.684 INFO:teuthology.orchestra.run.gibba002.stdout:/volumes/_nogroup/subvolume_0000000000344386/8642c421-c035-4392-955e-88abe3a75348 2021-02-10T11:08:47.685 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:47.689 7f2fc725a700 1 -- 172.21.2.102:0/4203901591 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2fa00186c0 msgr2=0x7f2fa001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:08:47.685 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:47.689 7f2fc725a700 1 --2- 172.21.2.102:0/4203901591 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2fa00186c0 0x7f2fa001abc0 crc :-1 s=READY pgs=217 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:08:47.685 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:47.689 7f2fc725a700 1 -- 172.21.2.102:0/4203901591 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2fc0082170 msgr2=0x7f2fc00a6400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:08:47.685 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:47.689 7f2fc725a700 1 --2- 172.21.2.102:0/4203901591 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2fc0082170 0x7f2fc00a6400 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 rx=0x7f2fbc001c60 tx=0x7f2fbc00a8d0).stop 2021-02-10T11:08:47.686 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:47.689 7f2fc725a700 1 -- 172.21.2.102:0/4203901591 shutdown_connections 2021-02-10T11:08:47.686 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:47.689 7f2fc725a700 1 --2- 172.21.2.102:0/4203901591 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2fa00186c0 0x7f2fa001abc0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:08:47.686 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:47.689 7f2fc725a700 1 --2- 172.21.2.102:0/4203901591 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2fc0082f20 0x7f2fc0083320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:08:47.686 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:47.689 7f2fc725a700 1 --2- 172.21.2.102:0/4203901591 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2fc00894a0 0x7f2fc00898a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:08:47.686 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:47.689 7f2fc725a700 1 --2- 172.21.2.102:0/4203901591 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2fc0082170 0x7f2fc00a6400 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:08:47.686 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:47.689 7f2fc725a700 1 -- 172.21.2.102:0/4203901591 >> 172.21.2.102:0/4203901591 conn(0x7f2fc0088d10 msgr2=0x7f2fc00afb80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:08:47.686 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:47.689 7f2fc725a700 1 -- 172.21.2.102:0/4203901591 shutdown_connections 2021-02-10T11:08:47.686 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:47.689 7f2fc725a700 1 -- 172.21.2.102:0/4203901591 wait complete. 2021-02-10T11:08:47.693 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000344386 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000344386/8642c421-c035-4392-955e-88abe3a75348 2021-02-10T11:08:47.693 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:08:47.694 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:08:47.694 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:08:48.030 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:08:48.030 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:08:48.031 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344386/8642c421-c035-4392-955e-88abe3a75348/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:08:48.404 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:08:48.405 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:08:48.405 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0214985 s, 48.8 MB/s 2021-02-10T11:08:48.406 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:08:48.406 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:08:48.406 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:08:48.742 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:08:48.742 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:08:48.742 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344386/8642c421-c035-4392-955e-88abe3a75348/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:08:49.118 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:08:49.119 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:08:49.119 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0251362 s, 41.7 MB/s 2021-02-10T11:08:49.120 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:08:49.120 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:08:49.120 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:08:49.456 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:08:49.456 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:08:49.456 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344386/8642c421-c035-4392-955e-88abe3a75348/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:08:49.834 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:08:49.834 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:08:49.834 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0255025 s, 41.1 MB/s 2021-02-10T11:08:49.835 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:08:49.835 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:08:49.835 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:08:50.172 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:08:50.173 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:08:50.173 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344386/8642c421-c035-4392-955e-88abe3a75348/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:08:50.550 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:08:50.550 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:08:50.550 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0242735 s, 43.2 MB/s 2021-02-10T11:08:50.551 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:08:50.551 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:08:50.551 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:08:50.888 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:08:50.888 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:08:50.888 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344386/8642c421-c035-4392-955e-88abe3a75348/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:08:51.265 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:08:51.266 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:08:51.266 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0252215 s, 41.6 MB/s 2021-02-10T11:08:51.267 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:08:51.267 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:08:51.267 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:08:51.601 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:08:51.602 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:08:51.602 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344386/8642c421-c035-4392-955e-88abe3a75348/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:08:51.977 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:08:51.978 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:08:51.978 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0251951 s, 41.6 MB/s 2021-02-10T11:08:51.979 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:08:51.979 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:08:51.979 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:08:52.313 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:08:52.314 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:08:52.314 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344386/8642c421-c035-4392-955e-88abe3a75348/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:08:52.690 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:08:52.690 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:08:52.690 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.025084 s, 41.8 MB/s 2021-02-10T11:08:52.691 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:08:52.691 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:08:52.691 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:08:53.027 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:08:53.027 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:08:53.027 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344386/8642c421-c035-4392-955e-88abe3a75348/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:08:53.398 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:08:53.398 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:08:53.398 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0223013 s, 47.0 MB/s 2021-02-10T11:08:53.399 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:08:53.399 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:08:53.399 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:08:53.733 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:08:53.734 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:08:53.734 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344386/8642c421-c035-4392-955e-88abe3a75348/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:08:54.109 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:08:54.109 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:08:54.109 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0237121 s, 44.2 MB/s 2021-02-10T11:08:54.110 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:08:54.110 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:08:54.111 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:08:54.447 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:08:54.447 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:08:54.447 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344386/8642c421-c035-4392-955e-88abe3a75348/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:08:54.821 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:08:54.821 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:08:54.821 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0233454 s, 44.9 MB/s 2021-02-10T11:08:54.822 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000344387 2021-02-10T11:08:55.253 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:55.240 7fba9d54f700 1 Processor -- start 2021-02-10T11:08:55.253 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:55.240 7fba9d54f700 1 -- start start 2021-02-10T11:08:55.253 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:55.240 7fba9d54f700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fba981b0470 0x7fba981b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:08:55.254 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:55.240 7fba9d54f700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba980f0b50 con 0x7fba980f20e0 2021-02-10T11:08:55.254 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:55.240 7fba9d54f700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba980f24e0 con 0x7fba980ef610 2021-02-10T11:08:55.254 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:55.240 7fba9d54f700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba980e5ea0 con 0x7fba981b0470 2021-02-10T11:08:55.254 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:55.240 7fba97ac0700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fba980f20e0 0x7fba980f0200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:43784/0 (socket says 172.21.2.102:43784) 2021-02-10T11:08:55.254 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:55.240 7fba97ac0700 1 -- 172.21.2.102:0/620570223 learned_addr learned my addr 172.21.2.102:0/620570223 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:08:55.254 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:55.241 7fba962bd700 1 -- 172.21.2.102:0/620570223 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 61102826 0 0) 0x7fba90000d30 con 0x7fba980f20e0 2021-02-10T11:08:55.254 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:55.241 7fba962bd700 1 -- 172.21.2.102:0/620570223 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba84002fe0 con 0x7fba980f20e0 2021-02-10T11:08:55.254 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:55.241 7fba972bf700 1 --2- 172.21.2.102:0/620570223 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fba981b0470 0x7fba981b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:08:55.254 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:55.241 7fba962bd700 1 -- 172.21.2.102:0/620570223 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 728056874 0 0) 0x7fba90001460 con 0x7fba980f20e0 2021-02-10T11:08:55.254 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:55.241 7fba962bd700 1 -- 172.21.2.102:0/620570223 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fba981b0470 msgr2=0x7fba981b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:08:55.254 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:55.241 7fba962bd700 1 --2- 172.21.2.102:0/620570223 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fba981b0470 0x7fba981b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:08:55.254 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:55.241 7fba962bd700 1 -- 172.21.2.102:0/620570223 >> v1:172.21.2.113:6789/0 conn(0x7fba980ef610 legacy=0x7fba980efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:08:55.255 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:55.241 7fba962bd700 1 -- 172.21.2.102:0/620570223 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba980e52e0 con 0x7fba980f20e0 2021-02-10T11:08:55.255 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:55.241 7fba962bd700 1 -- 172.21.2.102:0/620570223 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7fba900019e0 con 0x7fba980f20e0 2021-02-10T11:08:55.255 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:55.241 7fba962bd700 1 -- 172.21.2.102:0/620570223 >> v1:172.21.2.102:6789/0 conn(0x7fba980f20e0 legacy=0x7fba980f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:08:55.255 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:55.241 7fba962bd700 1 --2- 172.21.2.102:0/620570223 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fba84003ac0 0x7fba84005ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:08:55.255 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:55.241 7fba962bd700 1 --2- 172.21.2.102:0/620570223 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fba84006790 0x7fba84008c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:08:55.255 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:55.241 7fba962bd700 1 --2- 172.21.2.102:0/620570223 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fba84009590 0x7fba8400ba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:08:55.255 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:55.241 7fba962bd700 1 -- 172.21.2.102:0/620570223 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba8400c2d0 con 0x7fba84006790 2021-02-10T11:08:55.255 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:55.241 7fba962bd700 1 -- 172.21.2.102:0/620570223 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba8400c4f0 con 0x7fba84003ac0 2021-02-10T11:08:55.255 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:55.241 7fba962bd700 1 -- 172.21.2.102:0/620570223 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba8400c710 con 0x7fba84009590 2021-02-10T11:08:55.255 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:55.241 7fba96abe700 1 --2- 172.21.2.102:0/620570223 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fba84006790 0x7fba84008c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:08:55.255 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:55.241 7fba972bf700 1 --2- 172.21.2.102:0/620570223 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fba84003ac0 0x7fba84005ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:08:55.255 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:55.242 7fba97ac0700 1 --2- 172.21.2.102:0/620570223 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fba84009590 0x7fba8400ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:08:55.255 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:55.242 7fba96abe700 1 -- 172.21.2.102:0/620570223 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fba84009590 msgr2=0x7fba8400ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:08:55.256 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:55.242 7fba96abe700 1 --2- 172.21.2.102:0/620570223 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fba84009590 0x7fba8400ba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:08:55.256 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:55.242 7fba96abe700 1 -- 172.21.2.102:0/620570223 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fba84003ac0 msgr2=0x7fba84005ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:08:55.256 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:55.242 7fba96abe700 1 --2- 172.21.2.102:0/620570223 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fba84003ac0 0x7fba84005ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:08:55.256 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:55.242 7fba96abe700 1 -- 172.21.2.102:0/620570223 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba8400c9b0 con 0x7fba84006790 2021-02-10T11:08:55.256 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:55.242 7fba96abe700 1 --2- 172.21.2.102:0/620570223 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fba84006790 0x7fba84008c90 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 rx=0x7fba80006ad0 tx=0x7fba8004db50).ready entity=mon.0 client_cookie=4e3ed3dac4936384 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:08:55.256 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:55.242 7fba962bd700 1 -- 172.21.2.102:0/620570223 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fba800066d0 con 0x7fba84006790 2021-02-10T11:08:55.256 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:55.242 7fba962bd700 1 -- 172.21.2.102:0/620570223 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba80002280 con 0x7fba84006790 2021-02-10T11:08:55.256 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:55.242 7fba962bd700 1 -- 172.21.2.102:0/620570223 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fba800026d0 con 0x7fba84006790 2021-02-10T11:08:55.256 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:55.242 7fba9d54f700 1 -- 172.21.2.102:0/620570223 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fba84006790 msgr2=0x7fba84008c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:08:55.256 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:55.242 7fba9d54f700 1 --2- 172.21.2.102:0/620570223 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fba84006790 0x7fba84008c90 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 rx=0x7fba80006ad0 tx=0x7fba8004db50).stop 2021-02-10T11:08:55.256 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:55.242 7fba97ac0700 1 -- 172.21.2.102:0/620570223 reap_dead start 2021-02-10T11:08:55.256 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:55.243 7fba9d54f700 1 -- 172.21.2.102:0/620570223 shutdown_connections 2021-02-10T11:08:55.256 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:55.243 7fba9d54f700 1 -- 172.21.2.102:0/620570223 >> 172.21.2.102:0/620570223 conn(0x7fba980ce3e0 msgr2=0x7fba980e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:08:55.256 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:55.243 7fba9d54f700 1 -- 172.21.2.102:0/620570223 shutdown_connections 2021-02-10T11:08:55.257 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:55.243 7fba9d54f700 1 -- 172.21.2.102:0/620570223 wait complete. 2021-02-10T11:08:55.257 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:55.257 7fba9d54f700 1 Processor -- start 2021-02-10T11:08:55.257 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:55.258 7fba9d54f700 1 -- start start 2021-02-10T11:08:55.257 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:55.258 7fba9d54f700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fba9809b930 0x7fba980b7c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:08:55.257 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:55.258 7fba9d54f700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fba980b8410 0x7fba980b8810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:08:55.258 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:55.258 7fba9d54f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fba980c1f80 0x7fba980c2380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:08:55.258 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:55.258 7fba9d54f700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba980c1a00 con 0x7fba980c1f80 2021-02-10T11:08:55.258 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:55.258 7fba9d54f700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba980b8e30 con 0x7fba9809b930 2021-02-10T11:08:55.258 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:55.258 7fba9d54f700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba980b9050 con 0x7fba980b8410 2021-02-10T11:08:55.259 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:55.258 7fba97ac0700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fba980c1f80 0x7fba980c2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:08:55.259 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:55.258 7fba97ac0700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fba980c1f80 0x7fba980c2380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:51574/0 (socket says 172.21.2.102:51574) 2021-02-10T11:08:55.259 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:55.258 7fba97ac0700 1 -- 172.21.2.102:0/3781561269 learned_addr learned my addr 172.21.2.102:0/3781561269 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:08:55.259 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:55.259 7fba972bf700 1 --2- 172.21.2.102:0/3781561269 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fba9809b930 0x7fba980b7c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:08:55.259 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:55.259 7fba96abe700 1 --2- 172.21.2.102:0/3781561269 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fba980b8410 0x7fba980b8810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:08:55.259 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:55.259 7fba97ac0700 1 -- 172.21.2.102:0/3781561269 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fba980b8410 msgr2=0x7fba980b8810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:08:55.260 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:55.259 7fba97ac0700 1 --2- 172.21.2.102:0/3781561269 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fba980b8410 0x7fba980b8810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:08:55.260 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:55.259 7fba97ac0700 1 -- 172.21.2.102:0/3781561269 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fba9809b930 msgr2=0x7fba980b7c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:08:55.260 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:55.259 7fba97ac0700 1 --2- 172.21.2.102:0/3781561269 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fba9809b930 0x7fba980b7c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:08:55.260 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:55.259 7fba97ac0700 1 -- 172.21.2.102:0/3781561269 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba980c29a0 con 0x7fba980c1f80 2021-02-10T11:08:55.260 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:55.259 7fba96abe700 1 --2- 172.21.2.102:0/3781561269 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fba980b8410 0x7fba980b8810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:08:55.260 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:55.259 7fba97ac0700 1 --2- 172.21.2.102:0/3781561269 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fba980c1f80 0x7fba980c2380 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 rx=0x7fba90002240 tx=0x7fba9000acd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:08:55.260 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:55.259 7fba8ffff700 1 -- 172.21.2.102:0/3781561269 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fba90007ad0 con 0x7fba980c1f80 2021-02-10T11:08:55.260 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:55.259 7fba8ffff700 1 -- 172.21.2.102:0/3781561269 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba90004930 con 0x7fba980c1f80 2021-02-10T11:08:55.260 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:55.259 7fba9d54f700 1 -- 172.21.2.102:0/3781561269 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba980c29a0 con 0x7fba980c1f80 2021-02-10T11:08:55.260 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:55.259 7fba8ffff700 1 -- 172.21.2.102:0/3781561269 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fba900070d0 con 0x7fba980c1f80 2021-02-10T11:08:55.260 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:55.259 7fba9d54f700 1 -- 172.21.2.102:0/3781561269 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba980c2f30 con 0x7fba980c1f80 2021-02-10T11:08:55.260 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:55.260 7fba8ffff700 1 -- 172.21.2.102:0/3781561269 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fba9000ae10 con 0x7fba980c1f80 2021-02-10T11:08:55.260 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:55.260 7fba8ffff700 1 --2- 172.21.2.102:0/3781561269 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fba7c0186c0 0x7fba7c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:08:55.261 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:55.260 7fba8ffff700 1 -- 172.21.2.102:0/3781561269 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fba90032940 con 0x7fba980c1f80 2021-02-10T11:08:55.261 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:55.260 7fba972bf700 1 --2- 172.21.2.102:0/3781561269 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fba7c0186c0 0x7fba7c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:08:55.261 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:55.260 7fba972bf700 1 --2- 172.21.2.102:0/3781561269 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fba7c0186c0 0x7fba7c01abc0 crc :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:08:55.287 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:55.290 7fba9d54f700 1 -- 172.21.2.102:0/3781561269 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba98105b30 con 0x7fba980c1f80 2021-02-10T11:08:55.289 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:55.293 7fba8ffff700 1 -- 172.21.2.102:0/3781561269 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fba900049c0 con 0x7fba980c1f80 2021-02-10T11:08:55.416 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:55.420 7fba9d54f700 1 -- 172.21.2.102:0/3781561269 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344387", "target": ["mgr", ""]}) v1 -- 0x7fba980c2f30 con 0x7fba7c0186c0 2021-02-10T11:08:55.444 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:55.447 7fba8ffff700 1 -- 172.21.2.102:0/3781561269 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fba880009b0 con 0x7fba7c0186c0 2021-02-10T11:08:55.449 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:55.453 7fba9d54f700 1 -- 172.21.2.102:0/3781561269 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fba7c0186c0 msgr2=0x7fba7c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:08:55.449 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:55.453 7fba9d54f700 1 --2- 172.21.2.102:0/3781561269 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fba7c0186c0 0x7fba7c01abc0 crc :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:08:55.449 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:55.453 7fba9d54f700 1 -- 172.21.2.102:0/3781561269 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fba980c1f80 msgr2=0x7fba980c2380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:08:55.449 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:55.453 7fba9d54f700 1 --2- 172.21.2.102:0/3781561269 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fba980c1f80 0x7fba980c2380 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 rx=0x7fba90002240 tx=0x7fba9000acd0).stop 2021-02-10T11:08:55.449 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:55.453 7fba9d54f700 1 -- 172.21.2.102:0/3781561269 shutdown_connections 2021-02-10T11:08:55.449 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:55.453 7fba9d54f700 1 --2- 172.21.2.102:0/3781561269 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fba980c1f80 0x7fba980c2380 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:08:55.450 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:55.453 7fba9d54f700 1 --2- 172.21.2.102:0/3781561269 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fba980b8410 0x7fba980b8810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:08:55.450 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:55.453 7fba9d54f700 1 --2- 172.21.2.102:0/3781561269 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fba7c0186c0 0x7fba7c01abc0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:08:55.450 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:55.453 7fba9d54f700 1 --2- 172.21.2.102:0/3781561269 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fba9809b930 0x7fba980b7c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:08:55.450 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:55.453 7fba9d54f700 1 -- 172.21.2.102:0/3781561269 >> 172.21.2.102:0/3781561269 conn(0x7fba98094e10 msgr2=0x7fba980a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:08:55.450 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:55.453 7fba9d54f700 1 -- 172.21.2.102:0/3781561269 shutdown_connections 2021-02-10T11:08:55.450 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:55.453 7fba9d54f700 1 -- 172.21.2.102:0/3781561269 wait complete. 2021-02-10T11:08:55.457 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000344387 2021-02-10T11:08:55.889 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:55.875 7f8e77ac0700 1 Processor -- start 2021-02-10T11:08:55.889 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:55.875 7f8e77ac0700 1 -- start start 2021-02-10T11:08:55.889 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:55.875 7f8e77ac0700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f8e701b0870 0x7f8e700f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:08:55.889 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:55.875 7f8e77ac0700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e701b0c70 con 0x7f8e701b0470 2021-02-10T11:08:55.889 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:55.875 7f8e77ac0700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e700f0b50 con 0x7f8e700efe00 2021-02-10T11:08:55.889 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:55.875 7f8e77ac0700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e700e5ea0 con 0x7f8e701b0870 2021-02-10T11:08:55.890 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:55.876 7f8e758dc700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f8e701b0470 0x7f8e700ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:43802/0 (socket says 172.21.2.102:43802) 2021-02-10T11:08:55.890 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:55.876 7f8e758dc700 1 -- 172.21.2.102:0/2281170034 learned_addr learned my addr 172.21.2.102:0/2281170034 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:08:55.890 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:55.876 7f8e748da700 1 -- 172.21.2.102:0/2281170034 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2248682879 0 0) 0x7f8e6c001180 con 0x7f8e701b0470 2021-02-10T11:08:55.890 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:55.876 7f8e748da700 1 -- 172.21.2.102:0/2281170034 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e5c002fe0 con 0x7f8e701b0470 2021-02-10T11:08:55.890 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:55.876 7f8e750db700 1 --2- 172.21.2.102:0/2281170034 >> v2:172.21.2.113:3300/0 conn(0x7f8e701b0870 0x7f8e700f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:08:55.890 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:55.877 7f8e748da700 1 -- 172.21.2.102:0/2281170034 <== mon.1 v1:172.21.2.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3314022686 0 0) 0x7f8e68000d30 con 0x7f8e700efe00 2021-02-10T11:08:55.890 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:55.877 7f8e748da700 1 -- 172.21.2.102:0/2281170034 --> v1:172.21.2.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e5c003210 con 0x7f8e700efe00 2021-02-10T11:08:55.890 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:55.877 7f8e748da700 1 -- 172.21.2.102:0/2281170034 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2257746061 0 0) 0x7f8e6c001180 con 0x7f8e701b0470 2021-02-10T11:08:55.890 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:55.877 7f8e748da700 1 -- 172.21.2.102:0/2281170034 >> v2:172.21.2.113:3300/0 conn(0x7f8e701b0870 msgr2=0x7f8e700f20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:08:55.890 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:55.877 7f8e748da700 1 --2- 172.21.2.102:0/2281170034 >> v2:172.21.2.113:3300/0 conn(0x7f8e701b0870 0x7f8e700f20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:08:55.890 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:55.877 7f8e748da700 1 -- 172.21.2.102:0/2281170034 >> v1:172.21.2.113:6789/0 conn(0x7f8e700efe00 legacy=0x7f8e700f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:08:55.890 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:55.877 7f8e748da700 1 -- 172.21.2.102:0/2281170034 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e700e52e0 con 0x7f8e701b0470 2021-02-10T11:08:55.890 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:55.877 7f8e748da700 1 -- 172.21.2.102:0/2281170034 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7f8e6c001680 con 0x7f8e701b0470 2021-02-10T11:08:55.891 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:55.877 7f8e748da700 1 -- 172.21.2.102:0/2281170034 >> v1:172.21.2.102:6789/0 conn(0x7f8e701b0470 legacy=0x7f8e700ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:08:55.891 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:55.877 7f8e748da700 1 --2- 172.21.2.102:0/2281170034 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8e5c0038e0 0x7f8e5c003ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:08:55.891 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:55.877 7f8e748da700 1 --2- 172.21.2.102:0/2281170034 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8e5c0067c0 0x7f8e5c008c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:08:55.891 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:55.877 7f8e748da700 1 --2- 172.21.2.102:0/2281170034 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8e5c009580 0x7f8e5c00baa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:08:55.891 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:55.877 7f8e748da700 1 -- 172.21.2.102:0/2281170034 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e5c050300 con 0x7f8e5c009580 2021-02-10T11:08:55.891 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:55.877 7f8e748da700 1 -- 172.21.2.102:0/2281170034 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e5c0504e0 con 0x7f8e5c0038e0 2021-02-10T11:08:55.891 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:55.877 7f8e748da700 1 -- 172.21.2.102:0/2281170034 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e5c050700 con 0x7f8e5c0067c0 2021-02-10T11:08:55.891 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:55.877 7f8e760dd700 1 --2- 172.21.2.102:0/2281170034 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8e5c009580 0x7f8e5c00baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:08:55.891 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:55.877 7f8e750db700 1 --2- 172.21.2.102:0/2281170034 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8e5c0067c0 0x7f8e5c008c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:08:55.891 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:55.877 7f8e758dc700 1 --2- 172.21.2.102:0/2281170034 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8e5c0038e0 0x7f8e5c003ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:08:55.891 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:55.877 7f8e760dd700 1 -- 172.21.2.102:0/2281170034 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8e5c0067c0 msgr2=0x7f8e5c008c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:08:55.891 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:55.877 7f8e760dd700 1 --2- 172.21.2.102:0/2281170034 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8e5c0067c0 0x7f8e5c008c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:08:55.891 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:55.877 7f8e760dd700 1 -- 172.21.2.102:0/2281170034 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8e5c0038e0 msgr2=0x7f8e5c003ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:08:55.892 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:55.877 7f8e760dd700 1 --2- 172.21.2.102:0/2281170034 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8e5c0038e0 0x7f8e5c003ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:08:55.892 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:55.877 7f8e760dd700 1 -- 172.21.2.102:0/2281170034 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e5c0509a0 con 0x7f8e5c009580 2021-02-10T11:08:55.892 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:55.877 7f8e760dd700 1 -- 172.21.2.102:0/2281170034 reap_dead start 2021-02-10T11:08:55.892 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:55.877 7f8e760dd700 1 --2- 172.21.2.102:0/2281170034 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8e5c009580 0x7f8e5c00baa0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 rx=0x7f8e680013c0 tx=0x7f8e68009da0).ready entity=mon.0 client_cookie=9279db7432882d52 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:08:55.892 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:55.878 7f8e748da700 1 -- 172.21.2.102:0/2281170034 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8e68001400 con 0x7f8e5c009580 2021-02-10T11:08:55.892 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:55.878 7f8e748da700 1 -- 172.21.2.102:0/2281170034 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e68003210 con 0x7f8e5c009580 2021-02-10T11:08:55.892 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:55.878 7f8e748da700 1 -- 172.21.2.102:0/2281170034 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8e680019a0 con 0x7f8e5c009580 2021-02-10T11:08:55.892 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:55.878 7f8e77ac0700 1 -- 172.21.2.102:0/2281170034 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8e5c009580 msgr2=0x7f8e5c00baa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:08:55.892 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:55.878 7f8e77ac0700 1 --2- 172.21.2.102:0/2281170034 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8e5c009580 0x7f8e5c00baa0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 rx=0x7f8e680013c0 tx=0x7f8e68009da0).stop 2021-02-10T11:08:55.892 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:55.878 7f8e77ac0700 1 -- 172.21.2.102:0/2281170034 shutdown_connections 2021-02-10T11:08:55.892 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:55.878 7f8e77ac0700 1 --2- 172.21.2.102:0/2281170034 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8e5c009580 0x7f8e5c00baa0 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:08:55.892 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:55.878 7f8e77ac0700 1 -- 172.21.2.102:0/2281170034 >> 172.21.2.102:0/2281170034 conn(0x7f8e700ce3e0 msgr2=0x7f8e700e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:08:55.892 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:55.878 7f8e77ac0700 1 -- 172.21.2.102:0/2281170034 shutdown_connections 2021-02-10T11:08:55.893 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:55.879 7f8e77ac0700 1 -- 172.21.2.102:0/2281170034 wait complete. 2021-02-10T11:08:55.893 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:55.893 7f8e77ac0700 1 Processor -- start 2021-02-10T11:08:55.893 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:55.893 7f8e77ac0700 1 -- start start 2021-02-10T11:08:55.894 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:55.893 7f8e77ac0700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8e700c30c0 0x7f8e701a7450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:08:55.894 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:55.893 7f8e77ac0700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8e700c34c0 0x7f8e701a7a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:08:55.894 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:55.893 7f8e77ac0700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8e700a93e0 0x7f8e700a97e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:08:55.894 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:55.893 7f8e77ac0700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e701a8090 con 0x7f8e700c30c0 2021-02-10T11:08:55.894 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:55.893 7f8e77ac0700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e701a82b0 con 0x7f8e700a93e0 2021-02-10T11:08:55.894 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:55.893 7f8e77ac0700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e700a9e00 con 0x7f8e700c34c0 2021-02-10T11:08:55.894 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:55.894 7f8e758dc700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8e700c30c0 0x7f8e701a7450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:08:55.894 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:55.894 7f8e758dc700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8e700c30c0 0x7f8e701a7450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:51588/0 (socket says 172.21.2.102:51588) 2021-02-10T11:08:55.895 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:55.894 7f8e758dc700 1 -- 172.21.2.102:0/404038471 learned_addr learned my addr 172.21.2.102:0/404038471 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:08:55.895 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:55.894 7f8e758dc700 1 -- 172.21.2.102:0/404038471 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8e700c34c0 msgr2=0x7f8e701a7a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:08:55.895 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:55.894 7f8e758dc700 1 --2- 172.21.2.102:0/404038471 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8e700c34c0 0x7f8e701a7a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:08:55.895 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:55.894 7f8e758dc700 1 -- 172.21.2.102:0/404038471 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8e700a93e0 msgr2=0x7f8e700a97e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:08:55.895 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:55.894 7f8e758dc700 1 --2- 172.21.2.102:0/404038471 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8e700a93e0 0x7f8e700a97e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:08:55.895 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:55.894 7f8e758dc700 1 -- 172.21.2.102:0/404038471 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e700aa020 con 0x7f8e700c30c0 2021-02-10T11:08:55.895 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:55.894 7f8e758dc700 1 --2- 172.21.2.102:0/404038471 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8e700c30c0 0x7f8e701a7450 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 rx=0x7f8e6c001ec0 tx=0x7f8e6c00a9a0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:08:55.895 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:55.894 7f8e667fc700 1 -- 172.21.2.102:0/404038471 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8e6c00ad90 con 0x7f8e700c30c0 2021-02-10T11:08:55.895 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:55.894 7f8e667fc700 1 -- 172.21.2.102:0/404038471 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e6c0071a0 con 0x7f8e700c30c0 2021-02-10T11:08:55.895 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:55.894 7f8e667fc700 1 -- 172.21.2.102:0/404038471 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8e6c0043a0 con 0x7f8e700c30c0 2021-02-10T11:08:55.895 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:55.894 7f8e77ac0700 1 -- 172.21.2.102:0/404038471 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e700aa020 con 0x7f8e700c30c0 2021-02-10T11:08:55.895 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:55.894 7f8e77ac0700 1 -- 172.21.2.102:0/404038471 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e700aa020 con 0x7f8e700c30c0 2021-02-10T11:08:55.895 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:55.895 7f8e667fc700 1 -- 172.21.2.102:0/404038471 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f8e6c001a00 con 0x7f8e700c30c0 2021-02-10T11:08:55.896 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:55.895 7f8e667fc700 1 --2- 172.21.2.102:0/404038471 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8e58018730 0x7f8e5801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:08:55.896 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:55.895 7f8e667fc700 1 -- 172.21.2.102:0/404038471 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f8e6c032630 con 0x7f8e700c30c0 2021-02-10T11:08:55.896 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:55.895 7f8e750db700 1 --2- 172.21.2.102:0/404038471 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8e58018730 0x7f8e5801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:08:55.896 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:55.896 7f8e750db700 1 --2- 172.21.2.102:0/404038471 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8e58018730 0x7f8e5801ac10 crc :-1 s=READY pgs=219 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:08:55.921 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:55.925 7f8e77ac0700 1 -- 172.21.2.102:0/404038471 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e70105b30 con 0x7f8e700c30c0 2021-02-10T11:08:55.924 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:55.928 7f8e667fc700 1 -- 172.21.2.102:0/404038471 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f8e6c007180 con 0x7f8e700c30c0 2021-02-10T11:08:56.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:56.055 7f8e77ac0700 1 -- 172.21.2.102:0/404038471 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344387", "target": ["mgr", ""]}) v1 -- 0x7f8e700f0b50 con 0x7f8e58018730 2021-02-10T11:08:56.054 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:56.058 7f8e667fc700 1 -- 172.21.2.102:0/404038471 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f8e60006e70 con 0x7f8e58018730 2021-02-10T11:08:56.055 INFO:teuthology.orchestra.run.gibba002.stdout:/volumes/_nogroup/subvolume_0000000000344387/b7341304-e2ca-4202-b374-5ae33489f351 2021-02-10T11:08:56.056 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:56.059 7f8e77ac0700 1 -- 172.21.2.102:0/404038471 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8e58018730 msgr2=0x7f8e5801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:08:56.056 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:56.059 7f8e77ac0700 1 --2- 172.21.2.102:0/404038471 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8e58018730 0x7f8e5801ac10 crc :-1 s=READY pgs=219 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:08:56.057 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:56.059 7f8e77ac0700 1 -- 172.21.2.102:0/404038471 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8e700c30c0 msgr2=0x7f8e701a7450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:08:56.057 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:56.059 7f8e77ac0700 1 --2- 172.21.2.102:0/404038471 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8e700c30c0 0x7f8e701a7450 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 rx=0x7f8e6c001ec0 tx=0x7f8e6c00a9a0).stop 2021-02-10T11:08:56.057 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:56.060 7f8e77ac0700 1 -- 172.21.2.102:0/404038471 shutdown_connections 2021-02-10T11:08:56.057 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:56.060 7f8e77ac0700 1 --2- 172.21.2.102:0/404038471 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8e58018730 0x7f8e5801ac10 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:08:56.057 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:56.060 7f8e77ac0700 1 --2- 172.21.2.102:0/404038471 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8e700a93e0 0x7f8e700a97e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:08:56.057 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:56.060 7f8e77ac0700 1 --2- 172.21.2.102:0/404038471 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8e700c34c0 0x7f8e701a7a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:08:56.058 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:56.060 7f8e77ac0700 1 --2- 172.21.2.102:0/404038471 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8e700c30c0 0x7f8e701a7450 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:08:56.058 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:56.060 7f8e77ac0700 1 -- 172.21.2.102:0/404038471 >> 172.21.2.102:0/404038471 conn(0x7f8e7009fc10 msgr2=0x7f8e700b3120 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:08:56.058 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:56.060 7f8e77ac0700 1 -- 172.21.2.102:0/404038471 shutdown_connections 2021-02-10T11:08:56.058 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:08:56.060 7f8e77ac0700 1 -- 172.21.2.102:0/404038471 wait complete. 2021-02-10T11:08:56.064 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000344387 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000344387/b7341304-e2ca-4202-b374-5ae33489f351 2021-02-10T11:08:56.065 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:08:56.065 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:08:56.065 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:08:56.401 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:08:56.401 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:08:56.401 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344387/b7341304-e2ca-4202-b374-5ae33489f351/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:08:56.782 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:08:56.783 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:08:56.783 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0237824 s, 44.1 MB/s 2021-02-10T11:08:56.784 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:08:56.784 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:08:56.784 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:08:57.119 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:08:57.120 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:08:57.120 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344387/b7341304-e2ca-4202-b374-5ae33489f351/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:08:57.494 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:08:57.494 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:08:57.495 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0228062 s, 46.0 MB/s 2021-02-10T11:08:57.495 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:08:57.496 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:08:57.496 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:08:57.830 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:08:57.830 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:08:57.830 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344387/b7341304-e2ca-4202-b374-5ae33489f351/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:08:58.206 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:08:58.207 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:08:58.207 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.024696 s, 42.5 MB/s 2021-02-10T11:08:58.208 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:08:58.208 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:08:58.208 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:08:58.542 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:08:58.543 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:08:58.543 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344387/b7341304-e2ca-4202-b374-5ae33489f351/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:08:58.914 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:08:58.915 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:08:58.915 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0233702 s, 44.9 MB/s 2021-02-10T11:08:58.916 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:08:58.916 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:08:58.916 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:08:59.250 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:08:59.250 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:08:59.250 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344387/b7341304-e2ca-4202-b374-5ae33489f351/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:08:59.631 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:08:59.685 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:08:59.685 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0237869 s, 44.1 MB/s 2021-02-10T11:08:59.685 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:08:59.686 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:08:59.686 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:09:00.020 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:09:00.020 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:09:00.021 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344387/b7341304-e2ca-4202-b374-5ae33489f351/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:09:00.396 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:09:00.397 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:09:00.397 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0247085 s, 42.4 MB/s 2021-02-10T11:09:00.398 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:09:00.398 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:09:00.398 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:09:00.732 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:09:00.732 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:09:00.733 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344387/b7341304-e2ca-4202-b374-5ae33489f351/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:09:01.108 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:09:01.108 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:09:01.108 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0254562 s, 41.2 MB/s 2021-02-10T11:09:01.109 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:09:01.110 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:09:01.110 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:09:01.443 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:09:01.444 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:09:01.444 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344387/b7341304-e2ca-4202-b374-5ae33489f351/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:09:01.820 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:09:01.820 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:09:01.820 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0246579 s, 42.5 MB/s 2021-02-10T11:09:01.821 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:09:01.821 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:09:01.821 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:09:02.156 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:09:02.156 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:09:02.156 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344387/b7341304-e2ca-4202-b374-5ae33489f351/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:09:02.529 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:09:02.530 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:09:02.530 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0230557 s, 45.5 MB/s 2021-02-10T11:09:02.531 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:09:02.531 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:09:02.531 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:09:02.866 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:09:02.866 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:09:02.866 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344387/b7341304-e2ca-4202-b374-5ae33489f351/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:09:03.240 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:09:03.240 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:09:03.241 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0240871 s, 43.5 MB/s 2021-02-10T11:09:03.241 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000344388 2021-02-10T11:09:03.672 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:03.660 7f4b2b577700 1 Processor -- start 2021-02-10T11:09:03.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:03.660 7f4b2b577700 1 -- start start 2021-02-10T11:09:03.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:03.660 7f4b2b577700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f4b241b0470 0x7f4b241b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:09:03.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:03.660 7f4b2b577700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f4b240ef610 0x7f4b240efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:09:03.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:03.660 7f4b2b577700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4b240f0030 0x7f4b240f0430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:09:03.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:03.660 7f4b2b577700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b240e5ea0 con 0x7f4b241b0470 2021-02-10T11:09:03.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:03.660 7f4b2b577700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b240f0b50 con 0x7f4b240ef610 2021-02-10T11:09:03.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:03.660 7f4b2b577700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b240e52e0 con 0x7f4b240f0030 2021-02-10T11:09:03.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:03.661 7f4b29393700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f4b241b0470 0x7f4b241b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:09:03.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:03.661 7f4b29393700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f4b241b0470 0x7f4b241b0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:51596/0 (socket says 172.21.2.102:51596) 2021-02-10T11:09:03.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:03.661 7f4b29393700 1 -- 172.21.2.102:0/3371311135 learned_addr learned my addr 172.21.2.102:0/3371311135 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:09:03.674 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:03.661 7f4b28b92700 1 --2- 172.21.2.102:0/3371311135 >> v2:172.21.2.113:3300/0 conn(0x7f4b240ef610 0x7f4b240efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:09:03.674 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:03.661 7f4b29b94700 1 --2- 172.21.2.102:0/3371311135 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4b240f0030 0x7f4b240f0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:09:03.674 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:03.661 7f4b29393700 1 -- 172.21.2.102:0/3371311135 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4b240f0030 msgr2=0x7f4b240f0430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:09:03.674 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:03.661 7f4b29393700 1 --2- 172.21.2.102:0/3371311135 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4b240f0030 0x7f4b240f0430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:09:03.674 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:03.661 7f4b29393700 1 -- 172.21.2.102:0/3371311135 >> v2:172.21.2.113:3300/0 conn(0x7f4b240ef610 msgr2=0x7f4b240efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:09:03.674 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:03.661 7f4b29393700 1 --2- 172.21.2.102:0/3371311135 >> v2:172.21.2.113:3300/0 conn(0x7f4b240ef610 0x7f4b240efa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:09:03.674 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:03.661 7f4b29393700 1 -- 172.21.2.102:0/3371311135 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b240f10d0 con 0x7f4b241b0470 2021-02-10T11:09:03.674 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:03.661 7f4b29393700 1 --2- 172.21.2.102:0/3371311135 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4b241b0470 0x7f4b241b0870 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 rx=0x7f4b14006a80 tx=0x7f4b14009f00).ready entity=mon.0 client_cookie=a420350e6c783596 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:09:03.674 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:03.661 7f4b23fff700 1 -- 172.21.2.102:0/3371311135 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4b1400b5c0 con 0x7f4b241b0470 2021-02-10T11:09:03.674 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:03.661 7f4b23fff700 1 -- 172.21.2.102:0/3371311135 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b1400b8c0 con 0x7f4b241b0470 2021-02-10T11:09:03.674 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:03.661 7f4b23fff700 1 -- 172.21.2.102:0/3371311135 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4b1400bda0 con 0x7f4b241b0470 2021-02-10T11:09:03.674 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:03.661 7f4b2b577700 1 -- 172.21.2.102:0/3371311135 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4b241b0470 msgr2=0x7f4b241b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:09:03.674 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:03.661 7f4b2b577700 1 --2- 172.21.2.102:0/3371311135 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4b241b0470 0x7f4b241b0870 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 rx=0x7f4b14006a80 tx=0x7f4b14009f00).stop 2021-02-10T11:09:03.675 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:03.662 7f4b2b577700 1 -- 172.21.2.102:0/3371311135 shutdown_connections 2021-02-10T11:09:03.675 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:03.662 7f4b2b577700 1 --2- 172.21.2.102:0/3371311135 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4b240f0030 0x7f4b240f0430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:09:03.675 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:03.662 7f4b2b577700 1 --2- 172.21.2.102:0/3371311135 >> v2:172.21.2.113:3300/0 conn(0x7f4b240ef610 0x7f4b240efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:09:03.675 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:03.662 7f4b2b577700 1 --2- 172.21.2.102:0/3371311135 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4b241b0470 0x7f4b241b0870 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:09:03.675 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:03.662 7f4b2b577700 1 -- 172.21.2.102:0/3371311135 >> 172.21.2.102:0/3371311135 conn(0x7f4b240ce3e0 msgr2=0x7f4b240e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:09:03.675 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:03.662 7f4b2b577700 1 -- 172.21.2.102:0/3371311135 shutdown_connections 2021-02-10T11:09:03.675 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:03.662 7f4b2b577700 1 -- 172.21.2.102:0/3371311135 wait complete. 2021-02-10T11:09:03.677 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:03.677 7f4b2b577700 1 Processor -- start 2021-02-10T11:09:03.677 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:03.677 7f4b2b577700 1 -- start start 2021-02-10T11:09:03.677 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:03.677 7f4b2b577700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4b240b7db0 0x7f4b240b8260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:09:03.677 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:03.677 7f4b2b577700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4b240b8880 0x7f4b241a7450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:09:03.677 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:03.677 7f4b2b577700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4b240b8c80 0x7f4b241a7a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:09:03.677 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:03.677 7f4b2b577700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b240b79b0 con 0x7f4b240b8880 2021-02-10T11:09:03.677 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:03.677 7f4b2b577700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b241a8090 con 0x7f4b240b8c80 2021-02-10T11:09:03.677 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:03.677 7f4b2b577700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b241a82b0 con 0x7f4b240b7db0 2021-02-10T11:09:03.677 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:03.678 7f4b28b92700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4b240b8880 0x7f4b241a7450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:09:03.677 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:03.678 7f4b28b92700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4b240b8880 0x7f4b241a7450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:51606/0 (socket says 172.21.2.102:51606) 2021-02-10T11:09:03.678 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:03.678 7f4b28b92700 1 -- 172.21.2.102:0/2345354748 learned_addr learned my addr 172.21.2.102:0/2345354748 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:09:03.678 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:03.678 7f4b28b92700 1 -- 172.21.2.102:0/2345354748 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4b240b7db0 msgr2=0x7f4b240b8260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:09:03.678 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:03.678 7f4b28b92700 1 --2- 172.21.2.102:0/2345354748 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4b240b7db0 0x7f4b240b8260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:09:03.678 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:03.678 7f4b28b92700 1 -- 172.21.2.102:0/2345354748 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4b240b8c80 msgr2=0x7f4b241a7a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:09:03.678 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:03.678 7f4b28b92700 1 --2- 172.21.2.102:0/2345354748 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4b240b8c80 0x7f4b241a7a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:09:03.678 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:03.678 7f4b28b92700 1 -- 172.21.2.102:0/2345354748 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b24190c10 con 0x7f4b240b8880 2021-02-10T11:09:03.678 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:03.678 7f4b28b92700 1 --2- 172.21.2.102:0/2345354748 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4b240b8880 0x7f4b241a7450 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 rx=0x7f4b18006f60 tx=0x7f4b1800b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:09:03.678 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:03.678 7f4b21ffb700 1 -- 172.21.2.102:0/2345354748 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4b1800bd60 con 0x7f4b240b8880 2021-02-10T11:09:03.678 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:03.678 7f4b21ffb700 1 -- 172.21.2.102:0/2345354748 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b18000c90 con 0x7f4b240b8880 2021-02-10T11:09:03.678 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:03.678 7f4b21ffb700 1 -- 172.21.2.102:0/2345354748 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4b18002220 con 0x7f4b240b8880 2021-02-10T11:09:03.678 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:03.678 7f4b2b577700 1 -- 172.21.2.102:0/2345354748 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b24190c10 con 0x7f4b240b8880 2021-02-10T11:09:03.678 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:03.678 7f4b2b577700 1 -- 172.21.2.102:0/2345354748 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b240c1eb0 con 0x7f4b240b8880 2021-02-10T11:09:03.678 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:03.679 7f4b21ffb700 1 -- 172.21.2.102:0/2345354748 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4b180052e0 con 0x7f4b240b8880 2021-02-10T11:09:03.679 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:03.679 7f4b21ffb700 1 --2- 172.21.2.102:0/2345354748 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4b040186c0 0x7f4b0401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:09:03.679 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:03.679 7f4b21ffb700 1 -- 172.21.2.102:0/2345354748 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f4b180325b0 con 0x7f4b240b8880 2021-02-10T11:09:03.679 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:03.679 7f4b29393700 1 --2- 172.21.2.102:0/2345354748 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4b040186c0 0x7f4b0401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:09:03.679 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:03.680 7f4b29393700 1 --2- 172.21.2.102:0/2345354748 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4b040186c0 0x7f4b0401abc0 crc :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:09:03.707 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:03.710 7f4b2b577700 1 -- 172.21.2.102:0/2345354748 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b24105b30 con 0x7f4b240b8880 2021-02-10T11:09:03.709 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:03.713 7f4b21ffb700 1 -- 172.21.2.102:0/2345354748 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f4b18000960 con 0x7f4b240b8880 2021-02-10T11:09:03.838 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:03.842 7f4b2b577700 1 -- 172.21.2.102:0/2345354748 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344388", "target": ["mgr", ""]}) v1 -- 0x7f4b240c1eb0 con 0x7f4b040186c0 2021-02-10T11:09:03.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:03.865 7f4b21ffb700 1 -- 172.21.2.102:0/2345354748 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f4b14006710 con 0x7f4b040186c0 2021-02-10T11:09:03.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:03.875 7f4b2b577700 1 -- 172.21.2.102:0/2345354748 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4b040186c0 msgr2=0x7f4b0401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:09:03.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:03.875 7f4b2b577700 1 --2- 172.21.2.102:0/2345354748 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4b040186c0 0x7f4b0401abc0 crc :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:09:03.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:03.875 7f4b2b577700 1 -- 172.21.2.102:0/2345354748 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4b240b8880 msgr2=0x7f4b241a7450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:09:03.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:03.875 7f4b2b577700 1 --2- 172.21.2.102:0/2345354748 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4b240b8880 0x7f4b241a7450 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 rx=0x7f4b18006f60 tx=0x7f4b1800b970).stop 2021-02-10T11:09:03.872 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:03.875 7f4b2b577700 1 -- 172.21.2.102:0/2345354748 shutdown_connections 2021-02-10T11:09:03.872 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:03.875 7f4b2b577700 1 --2- 172.21.2.102:0/2345354748 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4b040186c0 0x7f4b0401abc0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:09:03.872 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:03.875 7f4b2b577700 1 --2- 172.21.2.102:0/2345354748 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4b240b8c80 0x7f4b241a7a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:09:03.872 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:03.875 7f4b2b577700 1 --2- 172.21.2.102:0/2345354748 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4b240b8880 0x7f4b241a7450 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:09:03.872 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:03.875 7f4b2b577700 1 --2- 172.21.2.102:0/2345354748 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4b240b7db0 0x7f4b240b8260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:09:03.872 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:03.875 7f4b2b577700 1 -- 172.21.2.102:0/2345354748 >> 172.21.2.102:0/2345354748 conn(0x7f4b24094e10 msgr2=0x7f4b240a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:09:03.872 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:03.875 7f4b2b577700 1 -- 172.21.2.102:0/2345354748 shutdown_connections 2021-02-10T11:09:03.872 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:03.875 7f4b2b577700 1 -- 172.21.2.102:0/2345354748 wait complete. 2021-02-10T11:09:03.879 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000344388 2021-02-10T11:09:04.311 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:04.297 7fac3129c700 1 Processor -- start 2021-02-10T11:09:04.311 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:04.297 7fac3129c700 1 -- start start 2021-02-10T11:09:04.311 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:04.298 7fac3129c700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fac2c1b0470 0x7fac2c1b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:09:04.311 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:04.298 7fac3129c700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac2c0f0b50 con 0x7fac2c0ef610 2021-02-10T11:09:04.311 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:04.298 7fac3129c700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac2c0f24e0 con 0x7fac2c0f20e0 2021-02-10T11:09:04.312 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:04.298 7fac3129c700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac2c0e5ea0 con 0x7fac2c1b0470 2021-02-10T11:09:04.312 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:04.298 7fac2a87c700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fac2c0ef610 0x7fac2c0efa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:43834/0 (socket says 172.21.2.102:43834) 2021-02-10T11:09:04.312 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:04.298 7fac2a87c700 1 -- 172.21.2.102:0/47258544 learned_addr learned my addr 172.21.2.102:0/47258544 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:09:04.312 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:04.298 7fac2a07b700 1 -- 172.21.2.102:0/47258544 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3102260169 0 0) 0x7fac140010b0 con 0x7fac2c0ef610 2021-02-10T11:09:04.312 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:04.298 7fac2a07b700 1 -- 172.21.2.102:0/47258544 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac18002fe0 con 0x7fac2c0ef610 2021-02-10T11:09:04.312 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:04.299 7fac2b07d700 1 --2- 172.21.2.102:0/47258544 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fac2c1b0470 0x7fac2c1b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:09:04.312 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:04.299 7fac2a07b700 1 -- 172.21.2.102:0/47258544 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1753749214 0 0) 0x7fac140010b0 con 0x7fac2c0ef610 2021-02-10T11:09:04.312 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:04.299 7fac2a07b700 1 -- 172.21.2.102:0/47258544 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fac2c1b0470 msgr2=0x7fac2c1b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:09:04.312 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:04.299 7fac2a07b700 1 --2- 172.21.2.102:0/47258544 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fac2c1b0470 0x7fac2c1b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:09:04.312 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:04.299 7fac2a07b700 1 -- 172.21.2.102:0/47258544 >> v1:172.21.2.113:6789/0 conn(0x7fac2c0f20e0 legacy=0x7fac2c0f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:09:04.312 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:04.299 7fac2a07b700 1 -- 172.21.2.102:0/47258544 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac2c0e52e0 con 0x7fac2c0ef610 2021-02-10T11:09:04.312 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:04.299 7fac2a07b700 1 -- 172.21.2.102:0/47258544 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7fac140015b0 con 0x7fac2c0ef610 2021-02-10T11:09:04.312 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:04.299 7fac2a07b700 1 -- 172.21.2.102:0/47258544 >> v1:172.21.2.102:6789/0 conn(0x7fac2c0ef610 legacy=0x7fac2c0efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:09:04.313 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:04.299 7fac2a07b700 1 --2- 172.21.2.102:0/47258544 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fac18003b20 0x7fac18005f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:09:04.313 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:04.299 7fac2a07b700 1 --2- 172.21.2.102:0/47258544 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fac18006830 0x7fac18008d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:09:04.313 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:04.299 7fac2a07b700 1 --2- 172.21.2.102:0/47258544 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fac18009630 0x7fac1800bb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:09:04.313 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:04.299 7fac2a07b700 1 -- 172.21.2.102:0/47258544 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac1800c370 con 0x7fac18006830 2021-02-10T11:09:04.313 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:04.299 7fac2a07b700 1 -- 172.21.2.102:0/47258544 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac1800c590 con 0x7fac18003b20 2021-02-10T11:09:04.313 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:04.299 7fac2a07b700 1 -- 172.21.2.102:0/47258544 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac1800c7b0 con 0x7fac18009630 2021-02-10T11:09:04.313 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:04.299 7fac2b07d700 1 --2- 172.21.2.102:0/47258544 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fac2c1b0470 0x7fac2c1b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:09:04.313 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:04.299 7fac2a87c700 1 --2- 172.21.2.102:0/47258544 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fac18006830 0x7fac18008d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:09:04.313 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:04.299 7fac2b87e700 1 --2- 172.21.2.102:0/47258544 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fac18009630 0x7fac1800bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:09:04.313 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:04.299 7fac2b07d700 1 --2- 172.21.2.102:0/47258544 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fac18003b20 0x7fac18005f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:09:04.313 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:04.299 7fac2a87c700 1 -- 172.21.2.102:0/47258544 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fac18009630 msgr2=0x7fac1800bb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:09:04.313 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:04.299 7fac2a87c700 1 --2- 172.21.2.102:0/47258544 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fac18009630 0x7fac1800bb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:09:04.313 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:04.299 7fac2a87c700 1 -- 172.21.2.102:0/47258544 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fac18003b20 msgr2=0x7fac18005f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:09:04.314 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:04.299 7fac2a87c700 1 --2- 172.21.2.102:0/47258544 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fac18003b20 0x7fac18005f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:09:04.314 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:04.299 7fac2a87c700 1 -- 172.21.2.102:0/47258544 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac1800ca50 con 0x7fac18006830 2021-02-10T11:09:04.314 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:04.299 7fac2a87c700 1 --2- 172.21.2.102:0/47258544 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fac18006830 0x7fac18008d50 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 rx=0x7fac14001c60 tx=0x7fac14007d40).ready entity=mon.0 client_cookie=a39b53a6c6b716a3 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:09:04.314 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:04.300 7fac2a07b700 1 -- 172.21.2.102:0/47258544 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fac140009b0 con 0x7fac18006830 2021-02-10T11:09:04.314 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:04.300 7fac2a07b700 1 -- 172.21.2.102:0/47258544 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac1404e320 con 0x7fac18006830 2021-02-10T11:09:04.314 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:04.300 7fac2a07b700 1 -- 172.21.2.102:0/47258544 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fac1404e7b0 con 0x7fac18006830 2021-02-10T11:09:04.314 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:04.300 7fac3129c700 1 -- 172.21.2.102:0/47258544 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fac18006830 msgr2=0x7fac18008d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:09:04.314 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:04.300 7fac3129c700 1 --2- 172.21.2.102:0/47258544 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fac18006830 0x7fac18008d50 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 rx=0x7fac14001c60 tx=0x7fac14007d40).stop 2021-02-10T11:09:04.314 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:04.300 7fac2b87e700 1 -- 172.21.2.102:0/47258544 reap_dead start 2021-02-10T11:09:04.314 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:04.300 7fac3129c700 1 -- 172.21.2.102:0/47258544 shutdown_connections 2021-02-10T11:09:04.314 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:04.300 7fac3129c700 1 -- 172.21.2.102:0/47258544 >> 172.21.2.102:0/47258544 conn(0x7fac2c0ce3e0 msgr2=0x7fac2c0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:09:04.314 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:04.300 7fac3129c700 1 -- 172.21.2.102:0/47258544 shutdown_connections 2021-02-10T11:09:04.314 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:04.300 7fac3129c700 1 -- 172.21.2.102:0/47258544 wait complete. 2021-02-10T11:09:04.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:04.315 7fac3129c700 1 Processor -- start 2021-02-10T11:09:04.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:04.315 7fac3129c700 1 -- start start 2021-02-10T11:09:04.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:04.315 7fac3129c700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fac2c0b7db0 0x7fac2c0b8260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:09:04.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:04.315 7fac3129c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fac2c0b8880 0x7fac2c1a7450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:09:04.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:04.315 7fac3129c700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fac2c0b8c80 0x7fac2c1a7a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:09:04.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:04.315 7fac3129c700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac2c0b79b0 con 0x7fac2c0b8880 2021-02-10T11:09:04.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:04.315 7fac3129c700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac2c1a8090 con 0x7fac2c0b8c80 2021-02-10T11:09:04.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:04.315 7fac3129c700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac2c1a82b0 con 0x7fac2c0b7db0 2021-02-10T11:09:04.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:04.316 7fac2b07d700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fac2c0b7db0 0x7fac2c0b8260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:09:04.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:04.316 7fac2b07d700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fac2c0b7db0 0x7fac2c0b8260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.113:3301/0 says I am v2:172.21.2.102:57156/0 (socket says 172.21.2.102:57156) 2021-02-10T11:09:04.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:04.316 7fac2b07d700 1 -- 172.21.2.102:0/3982245393 learned_addr learned my addr 172.21.2.102:0/3982245393 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:09:04.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:04.316 7fac2a87c700 1 --2- 172.21.2.102:0/3982245393 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fac2c0b8880 0x7fac2c1a7450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:09:04.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:04.316 7fac2b87e700 1 --2- 172.21.2.102:0/3982245393 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fac2c0b8c80 0x7fac2c1a7a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:09:04.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:04.316 7fac2b07d700 1 -- 172.21.2.102:0/3982245393 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fac2c0b8c80 msgr2=0x7fac2c1a7a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:09:04.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:04.316 7fac2b07d700 1 --2- 172.21.2.102:0/3982245393 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fac2c0b8c80 0x7fac2c1a7a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:09:04.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:04.316 7fac2b07d700 1 -- 172.21.2.102:0/3982245393 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fac2c0b8880 msgr2=0x7fac2c1a7450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:09:04.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:04.316 7fac2b07d700 1 --2- 172.21.2.102:0/3982245393 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fac2c0b8880 0x7fac2c1a7450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:09:04.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:04.316 7fac2b07d700 1 -- 172.21.2.102:0/3982245393 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac2c190c10 con 0x7fac2c0b7db0 2021-02-10T11:09:04.318 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:04.316 7fac2b07d700 1 --2- 172.21.2.102:0/3982245393 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fac2c0b7db0 0x7fac2c0b8260 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7fac1c00b590 tx=0x7fac1c00b930).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:09:04.318 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:04.316 7fac23fff700 1 -- 172.21.2.102:0/3982245393 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fac1c0065d0 con 0x7fac2c0b7db0 2021-02-10T11:09:04.318 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:04.316 7fac23fff700 1 -- 172.21.2.102:0/3982245393 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac1c0052e0 con 0x7fac2c0b7db0 2021-02-10T11:09:04.318 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:04.316 7fac3129c700 1 -- 172.21.2.102:0/3982245393 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac2c190c10 con 0x7fac2c0b7db0 2021-02-10T11:09:04.318 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:04.317 7fac23fff700 1 -- 172.21.2.102:0/3982245393 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fac1c005730 con 0x7fac2c0b7db0 2021-02-10T11:09:04.318 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:04.317 7fac3129c700 1 -- 172.21.2.102:0/3982245393 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac2c190c10 con 0x7fac2c0b7db0 2021-02-10T11:09:04.318 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:04.317 7fac23fff700 1 -- 172.21.2.102:0/3982245393 <== mon.2 v2:172.21.2.113:3301/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fac1c006030 con 0x7fac2c0b7db0 2021-02-10T11:09:04.318 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:04.317 7fac23fff700 1 --2- 172.21.2.102:0/3982245393 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fac100186c0 0x7fac1001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:09:04.318 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:04.317 7fac23fff700 1 -- 172.21.2.102:0/3982245393 <== mon.2 v2:172.21.2.113:3301/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fac1c032580 con 0x7fac2c0b7db0 2021-02-10T11:09:04.318 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:04.317 7fac2a87c700 1 --2- 172.21.2.102:0/3982245393 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fac100186c0 0x7fac1001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:09:04.318 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:04.318 7fac2a87c700 1 --2- 172.21.2.102:0/3982245393 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fac100186c0 0x7fac1001abc0 crc :-1 s=READY pgs=221 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:09:04.344 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:04.348 7fac3129c700 1 -- 172.21.2.102:0/3982245393 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac2c105b30 con 0x7fac2c0b7db0 2021-02-10T11:09:04.347 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:04.352 7fac23fff700 1 -- 172.21.2.102:0/3982245393 <== mon.2 v2:172.21.2.113:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fac1c02cf00 con 0x7fac2c0b7db0 2021-02-10T11:09:04.479 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:04.483 7fac3129c700 1 -- 172.21.2.102:0/3982245393 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344388", "target": ["mgr", ""]}) v1 -- 0x7fac2c0f0b50 con 0x7fac100186c0 2021-02-10T11:09:04.482 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:04.487 7fac23fff700 1 -- 172.21.2.102:0/3982245393 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7fac14000c90 con 0x7fac100186c0 2021-02-10T11:09:04.486 INFO:teuthology.orchestra.run.gibba002.stdout:/volumes/_nogroup/subvolume_0000000000344388/bd15c132-f86c-40b9-8218-f5a93446ac01 2021-02-10T11:09:04.488 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:04.492 7fac3129c700 1 -- 172.21.2.102:0/3982245393 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fac100186c0 msgr2=0x7fac1001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:09:04.488 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:04.492 7fac3129c700 1 --2- 172.21.2.102:0/3982245393 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fac100186c0 0x7fac1001abc0 crc :-1 s=READY pgs=221 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:09:04.488 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:04.492 7fac3129c700 1 -- 172.21.2.102:0/3982245393 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fac2c0b7db0 msgr2=0x7fac2c0b8260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:09:04.488 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:04.492 7fac3129c700 1 --2- 172.21.2.102:0/3982245393 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fac2c0b7db0 0x7fac2c0b8260 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7fac1c00b590 tx=0x7fac1c00b930).stop 2021-02-10T11:09:04.488 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:04.492 7fac3129c700 1 -- 172.21.2.102:0/3982245393 shutdown_connections 2021-02-10T11:09:04.488 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:04.492 7fac3129c700 1 --2- 172.21.2.102:0/3982245393 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fac100186c0 0x7fac1001abc0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:09:04.488 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:04.492 7fac3129c700 1 --2- 172.21.2.102:0/3982245393 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fac2c0b8c80 0x7fac2c1a7a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:09:04.488 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:04.492 7fac3129c700 1 --2- 172.21.2.102:0/3982245393 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fac2c0b8880 0x7fac2c1a7450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:09:04.488 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:04.492 7fac3129c700 1 --2- 172.21.2.102:0/3982245393 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fac2c0b7db0 0x7fac2c0b8260 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:09:04.488 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:04.492 7fac3129c700 1 -- 172.21.2.102:0/3982245393 >> 172.21.2.102:0/3982245393 conn(0x7fac2c094e10 msgr2=0x7fac2c0a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:09:04.489 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:04.492 7fac3129c700 1 -- 172.21.2.102:0/3982245393 shutdown_connections 2021-02-10T11:09:04.489 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:04.492 7fac3129c700 1 -- 172.21.2.102:0/3982245393 wait complete. 2021-02-10T11:09:04.496 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000344388 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000344388/bd15c132-f86c-40b9-8218-f5a93446ac01 2021-02-10T11:09:04.496 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:09:04.497 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:09:04.497 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:09:04.834 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:09:04.836 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:09:04.837 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344388/bd15c132-f86c-40b9-8218-f5a93446ac01/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:09:05.221 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:09:05.221 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:09:05.221 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0247554 s, 42.4 MB/s 2021-02-10T11:09:05.222 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:09:05.222 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:09:05.223 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:09:05.559 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:09:05.559 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:09:05.560 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344388/bd15c132-f86c-40b9-8218-f5a93446ac01/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:09:05.937 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:09:05.937 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:09:05.937 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0242294 s, 43.3 MB/s 2021-02-10T11:09:05.938 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:09:05.938 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:09:05.938 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:09:06.275 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:09:06.275 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:09:06.275 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344388/bd15c132-f86c-40b9-8218-f5a93446ac01/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:09:06.652 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:09:06.652 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:09:06.652 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0252577 s, 41.5 MB/s 2021-02-10T11:09:06.653 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:09:06.653 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:09:06.653 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:09:06.988 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:09:06.988 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:09:06.988 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344388/bd15c132-f86c-40b9-8218-f5a93446ac01/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:09:07.365 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:09:07.365 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:09:07.365 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0248743 s, 42.2 MB/s 2021-02-10T11:09:07.366 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:09:07.366 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:09:07.366 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:09:07.702 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:09:07.702 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:09:07.702 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344388/bd15c132-f86c-40b9-8218-f5a93446ac01/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:09:08.077 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:09:08.078 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:09:08.078 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0241338 s, 43.4 MB/s 2021-02-10T11:09:08.079 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:09:08.079 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:09:08.080 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:09:08.415 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:09:08.415 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:09:08.415 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344388/bd15c132-f86c-40b9-8218-f5a93446ac01/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:09:08.787 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:09:08.787 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:09:08.787 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0241459 s, 43.4 MB/s 2021-02-10T11:09:08.788 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:09:08.788 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:09:08.788 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:09:09.124 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:09:09.124 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:09:09.124 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344388/bd15c132-f86c-40b9-8218-f5a93446ac01/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:09:09.498 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:09:09.499 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:09:09.499 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0240035 s, 43.7 MB/s 2021-02-10T11:09:09.500 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:09:09.500 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:09:09.500 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:09:09.834 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:09:09.835 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:09:09.835 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344388/bd15c132-f86c-40b9-8218-f5a93446ac01/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:09:10.209 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:09:10.209 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:09:10.209 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0227151 s, 46.2 MB/s 2021-02-10T11:09:10.210 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:09:10.210 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:09:10.210 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:09:10.545 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:09:10.545 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:09:10.545 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344388/bd15c132-f86c-40b9-8218-f5a93446ac01/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:09:10.922 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:09:10.922 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:09:10.922 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.025295 s, 41.5 MB/s 2021-02-10T11:09:10.923 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:09:10.923 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:09:10.923 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:09:11.259 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:09:11.260 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:09:11.260 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344388/bd15c132-f86c-40b9-8218-f5a93446ac01/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:09:11.634 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:09:11.635 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:09:11.635 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0238501 s, 44.0 MB/s 2021-02-10T11:09:11.636 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000344389 2021-02-10T11:09:12.065 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:12.052 7fc1e3f79700 1 Processor -- start 2021-02-10T11:09:12.066 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:12.052 7fc1e3f79700 1 -- start start 2021-02-10T11:09:12.066 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:12.053 7fc1e3f79700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fc1dc1b0470 0x7fc1dc1b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:09:12.066 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:12.053 7fc1e3f79700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc1dc0ef610 0x7fc1dc0efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:09:12.066 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:12.053 7fc1e3f79700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fc1dc0f0030 0x7fc1dc0f0430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:09:12.066 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:12.053 7fc1e3f79700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1dc0e5ea0 con 0x7fc1dc1b0470 2021-02-10T11:09:12.066 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:12.053 7fc1e3f79700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1dc0f0b50 con 0x7fc1dc0f0030 2021-02-10T11:09:12.066 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:12.053 7fc1e3f79700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1dc0e52e0 con 0x7fc1dc0ef610 2021-02-10T11:09:12.066 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:12.053 7fc1e1d95700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fc1dc1b0470 0x7fc1dc1b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:09:12.066 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:12.053 7fc1e1d95700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fc1dc1b0470 0x7fc1dc1b0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:51628/0 (socket says 172.21.2.102:51628) 2021-02-10T11:09:12.066 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:12.053 7fc1e1d95700 1 -- 172.21.2.102:0/1742323746 learned_addr learned my addr 172.21.2.102:0/1742323746 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:09:12.067 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:12.054 7fc1e1594700 1 --2- 172.21.2.102:0/1742323746 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc1dc0ef610 0x7fc1dc0efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:09:12.067 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:12.054 7fc1e1d95700 1 -- 172.21.2.102:0/1742323746 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc1dc0ef610 msgr2=0x7fc1dc0efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:09:12.067 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:12.054 7fc1e1d95700 1 --2- 172.21.2.102:0/1742323746 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc1dc0ef610 0x7fc1dc0efa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:09:12.067 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:12.054 7fc1e1d95700 1 -- 172.21.2.102:0/1742323746 >> v2:172.21.2.113:3300/0 conn(0x7fc1dc0f0030 msgr2=0x7fc1dc0f0430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:09:12.067 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:12.054 7fc1e2596700 1 --2- 172.21.2.102:0/1742323746 >> v2:172.21.2.113:3300/0 conn(0x7fc1dc0f0030 0x7fc1dc0f0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:09:12.067 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:12.054 7fc1e1d95700 1 --2- 172.21.2.102:0/1742323746 >> v2:172.21.2.113:3300/0 conn(0x7fc1dc0f0030 0x7fc1dc0f0430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:09:12.067 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:12.054 7fc1e1d95700 1 -- 172.21.2.102:0/1742323746 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1dc0f10d0 con 0x7fc1dc1b0470 2021-02-10T11:09:12.067 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:12.054 7fc1e1594700 1 --2- 172.21.2.102:0/1742323746 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc1dc0ef610 0x7fc1dc0efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:09:12.067 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:12.054 7fc1e2596700 1 --2- 172.21.2.102:0/1742323746 >> v2:172.21.2.113:3300/0 conn(0x7fc1dc0f0030 0x7fc1dc0f0430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:09:12.067 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:12.054 7fc1e1d95700 1 --2- 172.21.2.102:0/1742323746 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc1dc1b0470 0x7fc1dc1b0870 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 rx=0x7fc1d8006700 tx=0x7fc1d8009b80).ready entity=mon.0 client_cookie=bef8d463889f61fd server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:09:12.067 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:12.054 7fc1e0d93700 1 -- 172.21.2.102:0/1742323746 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc1d800c0c0 con 0x7fc1dc1b0470 2021-02-10T11:09:12.068 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:12.054 7fc1e0d93700 1 -- 172.21.2.102:0/1742323746 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1d800c400 con 0x7fc1dc1b0470 2021-02-10T11:09:12.068 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:12.054 7fc1e0d93700 1 -- 172.21.2.102:0/1742323746 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc1d800c920 con 0x7fc1dc1b0470 2021-02-10T11:09:12.068 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:12.054 7fc1e3f79700 1 -- 172.21.2.102:0/1742323746 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc1dc1b0470 msgr2=0x7fc1dc1b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:09:12.068 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:12.054 7fc1e3f79700 1 --2- 172.21.2.102:0/1742323746 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc1dc1b0470 0x7fc1dc1b0870 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 rx=0x7fc1d8006700 tx=0x7fc1d8009b80).stop 2021-02-10T11:09:12.068 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:12.055 7fc1e3f79700 1 -- 172.21.2.102:0/1742323746 shutdown_connections 2021-02-10T11:09:12.068 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:12.055 7fc1e3f79700 1 --2- 172.21.2.102:0/1742323746 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc1dc0ef610 0x7fc1dc0efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:09:12.068 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:12.055 7fc1e3f79700 1 --2- 172.21.2.102:0/1742323746 >> v2:172.21.2.113:3300/0 conn(0x7fc1dc0f0030 0x7fc1dc0f0430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:09:12.068 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:12.055 7fc1e3f79700 1 --2- 172.21.2.102:0/1742323746 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc1dc1b0470 0x7fc1dc1b0870 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:09:12.068 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:12.055 7fc1e3f79700 1 -- 172.21.2.102:0/1742323746 >> 172.21.2.102:0/1742323746 conn(0x7fc1dc0ce3e0 msgr2=0x7fc1dc0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:09:12.068 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:12.055 7fc1e3f79700 1 -- 172.21.2.102:0/1742323746 shutdown_connections 2021-02-10T11:09:12.068 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:12.055 7fc1e3f79700 1 -- 172.21.2.102:0/1742323746 wait complete. 2021-02-10T11:09:12.069 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:12.070 7fc1e3f79700 1 Processor -- start 2021-02-10T11:09:12.069 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:12.070 7fc1e3f79700 1 -- start start 2021-02-10T11:09:12.070 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:12.070 7fc1e3f79700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc1dc192ad0 0x7fc1dc192ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:09:12.070 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:12.070 7fc1e3f79700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc1dc0a78f0 0x7fc1dc0a7cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:09:12.070 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:12.070 7fc1e3f79700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc1dc0a8310 0x7fc1dc0a9060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:09:12.070 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:12.070 7fc1e3f79700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1dc0a8ae0 con 0x7fc1dc192ad0 2021-02-10T11:09:12.070 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:12.070 7fc1e3f79700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1dc0a8710 con 0x7fc1dc0a8310 2021-02-10T11:09:12.070 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:12.070 7fc1e3f79700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1dc0a88f0 con 0x7fc1dc0a78f0 2021-02-10T11:09:12.070 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:12.070 7fc1e1d95700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc1dc192ad0 0x7fc1dc192ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:09:12.071 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:12.070 7fc1e1d95700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc1dc192ad0 0x7fc1dc192ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:51632/0 (socket says 172.21.2.102:51632) 2021-02-10T11:09:12.071 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:12.070 7fc1e1d95700 1 -- 172.21.2.102:0/4098116093 learned_addr learned my addr 172.21.2.102:0/4098116093 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:09:12.071 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:12.071 7fc1e1d95700 1 -- 172.21.2.102:0/4098116093 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc1dc0a78f0 msgr2=0x7fc1dc0a7cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:09:12.071 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:12.071 7fc1e1d95700 1 --2- 172.21.2.102:0/4098116093 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc1dc0a78f0 0x7fc1dc0a7cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:09:12.071 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:12.071 7fc1e1d95700 1 -- 172.21.2.102:0/4098116093 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc1dc0a8310 msgr2=0x7fc1dc0a9060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:09:12.071 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:12.071 7fc1e1d95700 1 --2- 172.21.2.102:0/4098116093 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc1dc0a8310 0x7fc1dc0a9060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:09:12.072 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:12.071 7fc1e1d95700 1 -- 172.21.2.102:0/4098116093 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1dc0a9680 con 0x7fc1dc192ad0 2021-02-10T11:09:12.072 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:12.071 7fc1e1d95700 1 --2- 172.21.2.102:0/4098116093 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc1dc192ad0 0x7fc1dc192ed0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 rx=0x7fc1d8006420 tx=0x7fc1d800b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:09:12.072 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:12.071 7fc1ce7fc700 1 -- 172.21.2.102:0/4098116093 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc1d80090d0 con 0x7fc1dc192ad0 2021-02-10T11:09:12.072 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:12.071 7fc1ce7fc700 1 -- 172.21.2.102:0/4098116093 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1d800bd50 con 0x7fc1dc192ad0 2021-02-10T11:09:12.072 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:12.071 7fc1ce7fc700 1 -- 172.21.2.102:0/4098116093 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc1d8005560 con 0x7fc1dc192ad0 2021-02-10T11:09:12.072 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:12.071 7fc1e3f79700 1 -- 172.21.2.102:0/4098116093 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1dc0a9680 con 0x7fc1dc192ad0 2021-02-10T11:09:12.072 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:12.071 7fc1e3f79700 1 -- 172.21.2.102:0/4098116093 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1dc0a9680 con 0x7fc1dc192ad0 2021-02-10T11:09:12.073 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:12.072 7fc1ce7fc700 1 -- 172.21.2.102:0/4098116093 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fc1d80054f0 con 0x7fc1dc192ad0 2021-02-10T11:09:12.073 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:12.072 7fc1ce7fc700 1 --2- 172.21.2.102:0/4098116093 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc1c40186c0 0x7fc1c401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:09:12.073 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:12.072 7fc1ce7fc700 1 -- 172.21.2.102:0/4098116093 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fc1d8032390 con 0x7fc1dc192ad0 2021-02-10T11:09:12.073 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:12.072 7fc1e1594700 1 --2- 172.21.2.102:0/4098116093 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc1c40186c0 0x7fc1c401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:09:12.073 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:12.073 7fc1e1594700 1 --2- 172.21.2.102:0/4098116093 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc1c40186c0 0x7fc1c401abc0 crc :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:09:12.099 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:12.103 7fc1e3f79700 1 -- 172.21.2.102:0/4098116093 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1dc0f0b50 con 0x7fc1dc192ad0 2021-02-10T11:09:12.102 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:12.106 7fc1ce7fc700 1 -- 172.21.2.102:0/4098116093 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fc1d80052e0 con 0x7fc1dc192ad0 2021-02-10T11:09:12.227 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:12.231 7fc1e3f79700 1 -- 172.21.2.102:0/4098116093 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344389", "target": ["mgr", ""]}) v1 -- 0x7fc1dc0f0b50 con 0x7fc1c40186c0 2021-02-10T11:09:12.252 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:12.256 7fc1ce7fc700 1 -- 172.21.2.102:0/4098116093 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fc1d00009b0 con 0x7fc1c40186c0 2021-02-10T11:09:12.261 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:12.264 7fc1e3f79700 1 -- 172.21.2.102:0/4098116093 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc1c40186c0 msgr2=0x7fc1c401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:09:12.261 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:12.264 7fc1e3f79700 1 --2- 172.21.2.102:0/4098116093 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc1c40186c0 0x7fc1c401abc0 crc :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:09:12.261 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:12.264 7fc1e3f79700 1 -- 172.21.2.102:0/4098116093 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc1dc192ad0 msgr2=0x7fc1dc192ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:09:12.261 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:12.264 7fc1e3f79700 1 --2- 172.21.2.102:0/4098116093 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc1dc192ad0 0x7fc1dc192ed0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 rx=0x7fc1d8006420 tx=0x7fc1d800b960).stop 2021-02-10T11:09:12.261 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:12.265 7fc1e3f79700 1 -- 172.21.2.102:0/4098116093 shutdown_connections 2021-02-10T11:09:12.261 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:12.265 7fc1e3f79700 1 --2- 172.21.2.102:0/4098116093 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc1c40186c0 0x7fc1c401abc0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:09:12.261 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:12.265 7fc1e3f79700 1 --2- 172.21.2.102:0/4098116093 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc1dc0a8310 0x7fc1dc0a9060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:09:12.261 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:12.265 7fc1e3f79700 1 --2- 172.21.2.102:0/4098116093 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc1dc0a78f0 0x7fc1dc0a7cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:09:12.261 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:12.265 7fc1e3f79700 1 --2- 172.21.2.102:0/4098116093 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc1dc192ad0 0x7fc1dc192ed0 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:09:12.262 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:12.265 7fc1e3f79700 1 -- 172.21.2.102:0/4098116093 >> 172.21.2.102:0/4098116093 conn(0x7fc1dc0bf4f0 msgr2=0x7fc1dc0be250 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:09:12.262 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:12.265 7fc1e3f79700 1 -- 172.21.2.102:0/4098116093 shutdown_connections 2021-02-10T11:09:12.262 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:12.265 7fc1e3f79700 1 -- 172.21.2.102:0/4098116093 wait complete. 2021-02-10T11:09:12.269 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000344389 2021-02-10T11:09:12.700 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:12.686 7f1792d29700 1 Processor -- start 2021-02-10T11:09:12.700 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:12.686 7f1792d29700 1 -- start start 2021-02-10T11:09:12.701 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:12.687 7f1792d29700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f178c1b0470 0x7f178c1b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:09:12.701 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:12.687 7f1792d29700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f178c0ef610 0x7f178c0efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:09:12.701 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:12.687 7f1792d29700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f178c0f0b50 con 0x7f178c0f0030 2021-02-10T11:09:12.701 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:12.687 7f1792d29700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f178c0e52e0 con 0x7f178c1b0470 2021-02-10T11:09:12.701 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:12.687 7f1792d29700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f178c0e54d0 con 0x7f178c0ef610 2021-02-10T11:09:12.701 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:12.687 7f1791346700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f178c0f0030 0x7f178c0f10d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:43868/0 (socket says 172.21.2.102:43868) 2021-02-10T11:09:12.701 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:12.687 7f1791346700 1 -- 172.21.2.102:0/1753782981 learned_addr learned my addr 172.21.2.102:0/1753782981 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:09:12.701 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:12.687 7f178b7fe700 1 -- 172.21.2.102:0/1753782981 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1989561403 0 0) 0x7f1784001180 con 0x7f178c0f0030 2021-02-10T11:09:12.701 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:12.687 7f178b7fe700 1 -- 172.21.2.102:0/1753782981 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1778002fe0 con 0x7f178c0f0030 2021-02-10T11:09:12.701 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:12.687 7f1790b45700 1 --2- 172.21.2.102:0/1753782981 >> v2:172.21.2.113:3300/0 conn(0x7f178c1b0470 0x7f178c1b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:09:12.701 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:12.687 7f178bfff700 1 --2- 172.21.2.102:0/1753782981 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f178c0ef610 0x7f178c0efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:09:12.701 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:12.687 7f178b7fe700 1 -- 172.21.2.102:0/1753782981 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1019128017 0 0) 0x7f17840010e0 con 0x7f178c0f0030 2021-02-10T11:09:12.701 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:12.688 7f178b7fe700 1 -- 172.21.2.102:0/1753782981 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f178c0ef610 msgr2=0x7f178c0efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:09:12.702 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:12.688 7f178b7fe700 1 --2- 172.21.2.102:0/1753782981 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f178c0ef610 0x7f178c0efa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:09:12.702 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:12.688 7f178b7fe700 1 -- 172.21.2.102:0/1753782981 >> v2:172.21.2.113:3300/0 conn(0x7f178c1b0470 msgr2=0x7f178c1b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:09:12.702 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:12.688 7f178b7fe700 1 --2- 172.21.2.102:0/1753782981 >> v2:172.21.2.113:3300/0 conn(0x7f178c1b0470 0x7f178c1b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:09:12.702 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:12.688 7f178b7fe700 1 -- 172.21.2.102:0/1753782981 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f178c0f0430 con 0x7f178c0f0030 2021-02-10T11:09:12.702 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:12.688 7f178b7fe700 1 -- 172.21.2.102:0/1753782981 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7f1784001600 con 0x7f178c0f0030 2021-02-10T11:09:12.702 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:12.688 7f178b7fe700 1 -- 172.21.2.102:0/1753782981 >> v1:172.21.2.102:6789/0 conn(0x7f178c0f0030 legacy=0x7f178c0f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:09:12.702 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:12.688 7f178b7fe700 1 --2- 172.21.2.102:0/1753782981 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1778003a60 0x7f1778005eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:09:12.702 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:12.688 7f178b7fe700 1 --2- 172.21.2.102:0/1753782981 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f177804a760 0x7f177804cc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:09:12.702 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:12.688 7f178b7fe700 1 --2- 172.21.2.102:0/1753782981 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f177804d500 0x7f177804fa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:09:12.702 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:12.688 7f178b7fe700 1 -- 172.21.2.102:0/1753782981 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17780502d0 con 0x7f177804d500 2021-02-10T11:09:12.703 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:12.688 7f178b7fe700 1 -- 172.21.2.102:0/1753782981 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17780504f0 con 0x7f177804a760 2021-02-10T11:09:12.703 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:12.688 7f178b7fe700 1 -- 172.21.2.102:0/1753782981 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1778050710 con 0x7f1778003a60 2021-02-10T11:09:12.703 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:12.688 7f1791346700 1 --2- 172.21.2.102:0/1753782981 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f177804d500 0x7f177804fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:09:12.703 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:12.688 7f178bfff700 1 --2- 172.21.2.102:0/1753782981 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f177804a760 0x7f177804cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:09:12.703 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:12.688 7f1790b45700 1 --2- 172.21.2.102:0/1753782981 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1778003a60 0x7f1778005eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:09:12.703 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:12.688 7f1791346700 1 -- 172.21.2.102:0/1753782981 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1778003a60 msgr2=0x7f1778005eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:09:12.703 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:12.688 7f1791346700 1 --2- 172.21.2.102:0/1753782981 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1778003a60 0x7f1778005eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:09:12.703 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:12.688 7f1791346700 1 -- 172.21.2.102:0/1753782981 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f177804a760 msgr2=0x7f177804cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:09:12.703 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:12.688 7f1791346700 1 --2- 172.21.2.102:0/1753782981 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f177804a760 0x7f177804cc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:09:12.703 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:12.688 7f1791346700 1 -- 172.21.2.102:0/1753782981 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17780509b0 con 0x7f177804d500 2021-02-10T11:09:12.703 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:12.688 7f1791346700 1 --2- 172.21.2.102:0/1753782981 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f177804d500 0x7f177804fa20 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 rx=0x7f1784000de0 tx=0x7f1784007cf0).ready entity=mon.0 client_cookie=5aa1acb6ee1ef576 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:09:12.703 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:12.688 7f178b7fe700 1 -- 172.21.2.102:0/1753782981 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f17840070c0 con 0x7f177804d500 2021-02-10T11:09:12.703 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:12.688 7f178b7fe700 1 -- 172.21.2.102:0/1753782981 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17840043a0 con 0x7f177804d500 2021-02-10T11:09:12.704 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:12.688 7f178b7fe700 1 -- 172.21.2.102:0/1753782981 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f17840010e0 con 0x7f177804d500 2021-02-10T11:09:12.704 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:12.689 7f1792d29700 1 -- 172.21.2.102:0/1753782981 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f177804d500 msgr2=0x7f177804fa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:09:12.704 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:12.689 7f1792d29700 1 --2- 172.21.2.102:0/1753782981 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f177804d500 0x7f177804fa20 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 rx=0x7f1784000de0 tx=0x7f1784007cf0).stop 2021-02-10T11:09:12.704 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:12.689 7f1791346700 1 -- 172.21.2.102:0/1753782981 reap_dead start 2021-02-10T11:09:12.704 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:12.689 7f1792d29700 1 -- 172.21.2.102:0/1753782981 shutdown_connections 2021-02-10T11:09:12.704 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:12.689 7f1792d29700 1 -- 172.21.2.102:0/1753782981 >> 172.21.2.102:0/1753782981 conn(0x7f178c0ce3e0 msgr2=0x7f178c0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:09:12.704 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:12.689 7f1792d29700 1 -- 172.21.2.102:0/1753782981 shutdown_connections 2021-02-10T11:09:12.704 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:12.689 7f1792d29700 1 -- 172.21.2.102:0/1753782981 wait complete. 2021-02-10T11:09:12.704 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:12.704 7f1792d29700 1 Processor -- start 2021-02-10T11:09:12.704 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:12.704 7f1792d29700 1 -- start start 2021-02-10T11:09:12.704 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:12.704 7f1792d29700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f178c09b930 0x7f178c0b7c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:09:12.704 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:12.704 7f1792d29700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f178c0b8410 0x7f178c0b8810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:09:12.704 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:12.704 7f1792d29700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f178c0c1f80 0x7f178c0c2380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:09:12.704 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:12.704 7f1792d29700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f178c0c1a00 con 0x7f178c0c1f80 2021-02-10T11:09:12.705 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:12.704 7f1792d29700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f178c0b8e30 con 0x7f178c0b8410 2021-02-10T11:09:12.705 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:12.704 7f1792d29700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f178c0b9050 con 0x7f178c09b930 2021-02-10T11:09:12.705 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:12.705 7f1791346700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f178c0c1f80 0x7f178c0c2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:09:12.705 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:12.705 7f1791346700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f178c0c1f80 0x7f178c0c2380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:51654/0 (socket says 172.21.2.102:51654) 2021-02-10T11:09:12.705 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:12.705 7f1791346700 1 -- 172.21.2.102:0/1852457227 learned_addr learned my addr 172.21.2.102:0/1852457227 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:09:12.706 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:12.705 7f1791346700 1 -- 172.21.2.102:0/1852457227 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f178c09b930 msgr2=0x7f178c0b7c20 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:09:12.706 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:12.705 7f1791346700 1 --2- 172.21.2.102:0/1852457227 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f178c09b930 0x7f178c0b7c20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:09:12.706 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:12.705 7f1791346700 1 -- 172.21.2.102:0/1852457227 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f178c0b8410 msgr2=0x7f178c0b8810 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:09:12.706 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:12.705 7f1791346700 1 --2- 172.21.2.102:0/1852457227 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f178c0b8410 0x7f178c0b8810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:09:12.706 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:12.705 7f1791346700 1 -- 172.21.2.102:0/1852457227 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f178c0c29a0 con 0x7f178c0c1f80 2021-02-10T11:09:12.706 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:12.705 7f1791346700 1 --2- 172.21.2.102:0/1852457227 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f178c0c1f80 0x7f178c0c2380 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 rx=0x7f1784007680 tx=0x7f178400a950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:09:12.706 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:12.705 7f17897fa700 1 -- 172.21.2.102:0/1852457227 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f178400ad40 con 0x7f178c0c1f80 2021-02-10T11:09:12.706 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:12.705 7f17897fa700 1 -- 172.21.2.102:0/1852457227 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1784001ad0 con 0x7f178c0c1f80 2021-02-10T11:09:12.706 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:12.705 7f17897fa700 1 -- 172.21.2.102:0/1852457227 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f17840043a0 con 0x7f178c0c1f80 2021-02-10T11:09:12.706 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:12.705 7f1792d29700 1 -- 172.21.2.102:0/1852457227 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f178c0c29a0 con 0x7f178c0c1f80 2021-02-10T11:09:12.707 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:12.705 7f1792d29700 1 -- 172.21.2.102:0/1852457227 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f178c0c29a0 con 0x7f178c0c1f80 2021-02-10T11:09:12.707 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:12.706 7f17897fa700 1 -- 172.21.2.102:0/1852457227 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1784000990 con 0x7f178c0c1f80 2021-02-10T11:09:12.707 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:12.706 7f17897fa700 1 --2- 172.21.2.102:0/1852457227 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f17700186c0 0x7f177001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:09:12.707 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:12.706 7f17897fa700 1 -- 172.21.2.102:0/1852457227 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f1784032580 con 0x7f178c0c1f80 2021-02-10T11:09:12.707 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:12.706 7f1790b45700 1 --2- 172.21.2.102:0/1852457227 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f17700186c0 0x7f177001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:09:12.707 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:12.706 7f1790b45700 1 --2- 172.21.2.102:0/1852457227 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f17700186c0 0x7f177001abc0 crc :-1 s=READY pgs=223 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:09:12.733 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:12.737 7f1792d29700 1 -- 172.21.2.102:0/1852457227 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f178c105b30 con 0x7f178c0c1f80 2021-02-10T11:09:12.735 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:12.739 7f17897fa700 1 -- 172.21.2.102:0/1852457227 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f178400ad40 con 0x7f178c0c1f80 2021-02-10T11:09:12.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:12.865 7f1792d29700 1 -- 172.21.2.102:0/1852457227 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344389", "target": ["mgr", ""]}) v1 -- 0x7f178c105b30 con 0x7f17700186c0 2021-02-10T11:09:12.865 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:12.869 7f17897fa700 1 -- 172.21.2.102:0/1852457227 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f1774000940 con 0x7f17700186c0 2021-02-10T11:09:12.869 INFO:teuthology.orchestra.run.gibba002.stdout:/volumes/_nogroup/subvolume_0000000000344389/b4648cfd-573b-46d1-a006-50c8372e487c 2021-02-10T11:09:12.869 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:12.874 7f1792d29700 1 -- 172.21.2.102:0/1852457227 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f17700186c0 msgr2=0x7f177001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:09:12.870 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:12.874 7f1792d29700 1 --2- 172.21.2.102:0/1852457227 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f17700186c0 0x7f177001abc0 crc :-1 s=READY pgs=223 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:09:12.870 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:12.874 7f1792d29700 1 -- 172.21.2.102:0/1852457227 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f178c0c1f80 msgr2=0x7f178c0c2380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:09:12.870 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:12.874 7f1792d29700 1 --2- 172.21.2.102:0/1852457227 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f178c0c1f80 0x7f178c0c2380 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 rx=0x7f1784007680 tx=0x7f178400a950).stop 2021-02-10T11:09:12.870 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:12.874 7f1792d29700 1 -- 172.21.2.102:0/1852457227 shutdown_connections 2021-02-10T11:09:12.870 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:12.874 7f1792d29700 1 --2- 172.21.2.102:0/1852457227 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f178c0c1f80 0x7f178c0c2380 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:09:12.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:12.874 7f1792d29700 1 --2- 172.21.2.102:0/1852457227 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f17700186c0 0x7f177001abc0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:09:12.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:12.874 7f1792d29700 1 --2- 172.21.2.102:0/1852457227 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f178c0b8410 0x7f178c0b8810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:09:12.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:12.874 7f1792d29700 1 --2- 172.21.2.102:0/1852457227 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f178c09b930 0x7f178c0b7c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:09:12.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:12.874 7f1792d29700 1 -- 172.21.2.102:0/1852457227 >> 172.21.2.102:0/1852457227 conn(0x7f178c094e10 msgr2=0x7f178c0a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:09:12.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:12.874 7f1792d29700 1 -- 172.21.2.102:0/1852457227 shutdown_connections 2021-02-10T11:09:12.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:12.874 7f1792d29700 1 -- 172.21.2.102:0/1852457227 wait complete. 2021-02-10T11:09:12.878 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000344389 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000344389/b4648cfd-573b-46d1-a006-50c8372e487c 2021-02-10T11:09:12.878 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:09:12.878 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:09:12.878 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:09:13.214 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:09:13.214 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:09:13.214 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344389/b4648cfd-573b-46d1-a006-50c8372e487c/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:09:13.590 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:09:13.591 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:09:13.591 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0217267 s, 48.3 MB/s 2021-02-10T11:09:13.592 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:09:13.592 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:09:13.592 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:09:13.927 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:09:13.927 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:09:13.927 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344389/b4648cfd-573b-46d1-a006-50c8372e487c/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:09:14.302 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:09:14.302 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:09:14.302 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0229692 s, 45.7 MB/s 2021-02-10T11:09:14.303 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:09:14.303 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:09:14.303 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:09:14.639 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:09:14.639 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:09:14.639 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344389/b4648cfd-573b-46d1-a006-50c8372e487c/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:09:14.980 DEBUG:teuthology.orchestra.run.gibba002:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:09:14.984 DEBUG:teuthology.orchestra.run.gibba013:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:09:15.016 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:09:15.016 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:09:15.016 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0240863 s, 43.5 MB/s 2021-02-10T11:09:15.017 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:09:15.017 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:09:15.017 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:09:15.321 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:09:15.322 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:09:15.322 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344389/b4648cfd-573b-46d1-a006-50c8372e487c/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:09:15.696 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:09:15.697 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:09:15.697 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0226268 s, 46.3 MB/s 2021-02-10T11:09:15.698 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:09:15.698 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:09:15.698 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:09:16.033 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:09:16.033 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:09:16.033 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344389/b4648cfd-573b-46d1-a006-50c8372e487c/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:09:16.407 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:09:16.407 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:09:16.407 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0222702 s, 47.1 MB/s 2021-02-10T11:09:16.408 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:09:16.408 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:09:16.409 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:09:16.744 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:09:16.744 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:09:16.744 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344389/b4648cfd-573b-46d1-a006-50c8372e487c/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:09:17.120 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:09:17.120 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:09:17.120 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0241872 s, 43.4 MB/s 2021-02-10T11:09:17.121 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:09:17.121 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:09:17.121 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:09:17.456 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:09:17.456 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:09:17.456 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344389/b4648cfd-573b-46d1-a006-50c8372e487c/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:09:17.833 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:09:17.833 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:09:17.833 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0249517 s, 42.0 MB/s 2021-02-10T11:09:17.834 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:09:17.834 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:09:17.834 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:09:18.169 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:09:18.170 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:09:18.170 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344389/b4648cfd-573b-46d1-a006-50c8372e487c/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:09:18.541 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:09:18.542 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:09:18.542 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.023868 s, 43.9 MB/s 2021-02-10T11:09:18.543 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:09:18.543 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:09:18.543 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:09:18.878 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:09:18.878 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:09:18.878 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344389/b4648cfd-573b-46d1-a006-50c8372e487c/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:09:19.252 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:09:19.252 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:09:19.252 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0238586 s, 43.9 MB/s 2021-02-10T11:09:19.253 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:09:19.253 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:09:19.253 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:09:19.589 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:09:19.589 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:09:19.589 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344389/b4648cfd-573b-46d1-a006-50c8372e487c/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:09:19.966 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:09:19.966 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:09:19.966 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0255798 s, 41.0 MB/s 2021-02-10T11:09:19.967 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000344390 2021-02-10T11:09:20.397 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:20.385 7f157de0e700 1 Processor -- start 2021-02-10T11:09:20.397 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:20.385 7f157de0e700 1 -- start start 2021-02-10T11:09:20.397 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:20.386 7f157de0e700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f15780f05f0 0x7f15780f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:09:20.398 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:20.386 7f157de0e700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15781b0c70 con 0x7f15781b0870 2021-02-10T11:09:20.398 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:20.386 7f157de0e700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15780f0b50 con 0x7f15781b0470 2021-02-10T11:09:20.398 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:20.386 7f157de0e700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15780e5ea0 con 0x7f15780f05f0 2021-02-10T11:09:20.398 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:20.386 7f1577fff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f15780f05f0 0x7f15780f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:09:20.398 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:20.386 7f1577fff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f15780f05f0 0x7f15780f20e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:51664/0 (socket says 172.21.2.102:51664) 2021-02-10T11:09:20.398 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:20.386 7f1577fff700 1 -- 172.21.2.102:0/1242838520 learned_addr learned my addr 172.21.2.102:0/1242838520 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:09:20.398 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:20.386 7f1577fff700 1 -- 172.21.2.102:0/1242838520 >> v1:172.21.2.113:6789/0 conn(0x7f15781b0470 legacy=0x7f15780ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:09:20.398 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:20.386 7f1577fff700 1 -- 172.21.2.102:0/1242838520 >> v1:172.21.2.102:6789/0 conn(0x7f15781b0870 legacy=0x7f15780efe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:09:20.398 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:20.386 7f1577fff700 1 -- 172.21.2.102:0/1242838520 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15780e52e0 con 0x7f15780f05f0 2021-02-10T11:09:20.398 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:20.386 7f1577fff700 1 --2- 172.21.2.102:0/1242838520 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f15780f05f0 0x7f15780f20e0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 rx=0x7f1570006f00 tx=0x7f1570009f10).ready entity=mon.0 client_cookie=866a04dc9b2492a8 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:09:20.398 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:20.386 7f15767fc700 1 -- 172.21.2.102:0/1242838520 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f157000b3b0 con 0x7f15780f05f0 2021-02-10T11:09:20.399 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:20.386 7f15767fc700 1 -- 172.21.2.102:0/1242838520 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f157000b630 con 0x7f15780f05f0 2021-02-10T11:09:20.399 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:20.386 7f15767fc700 1 -- 172.21.2.102:0/1242838520 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f157000bb10 con 0x7f15780f05f0 2021-02-10T11:09:20.399 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:20.386 7f157de0e700 1 -- 172.21.2.102:0/1242838520 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f15780f05f0 msgr2=0x7f15780f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:09:20.399 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:20.386 7f157de0e700 1 --2- 172.21.2.102:0/1242838520 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f15780f05f0 0x7f15780f20e0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 rx=0x7f1570006f00 tx=0x7f1570009f10).stop 2021-02-10T11:09:20.399 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:20.387 7f157de0e700 1 -- 172.21.2.102:0/1242838520 shutdown_connections 2021-02-10T11:09:20.399 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:20.387 7f157de0e700 1 --2- 172.21.2.102:0/1242838520 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f15780f05f0 0x7f15780f20e0 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:09:20.399 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:20.387 7f157de0e700 1 -- 172.21.2.102:0/1242838520 >> 172.21.2.102:0/1242838520 conn(0x7f15780ce3e0 msgr2=0x7f15780e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:09:20.399 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:20.387 7f157de0e700 1 -- 172.21.2.102:0/1242838520 shutdown_connections 2021-02-10T11:09:20.399 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:20.387 7f157de0e700 1 -- 172.21.2.102:0/1242838520 wait complete. 2021-02-10T11:09:20.401 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:20.402 7f157de0e700 1 Processor -- start 2021-02-10T11:09:20.401 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:20.402 7f157de0e700 1 -- start start 2021-02-10T11:09:20.401 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:20.402 7f157de0e700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1578081860 0x7f15780a9bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:09:20.401 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:20.402 7f157de0e700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f15780aa210 0x7f15780c19a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:09:20.401 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:20.402 7f157de0e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f15780aa610 0x7f15780c1fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:09:20.401 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:20.402 7f157de0e700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15780c2f20 con 0x7f15780aa610 2021-02-10T11:09:20.401 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:20.402 7f157de0e700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15780c1420 con 0x7f15780aa210 2021-02-10T11:09:20.401 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:20.402 7f157de0e700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15781b56f0 con 0x7f1578081860 2021-02-10T11:09:20.401 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:20.402 7f1577fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f15780aa610 0x7f15780c1fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:09:20.402 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:20.402 7f1577fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f15780aa610 0x7f15780c1fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:51666/0 (socket says 172.21.2.102:51666) 2021-02-10T11:09:20.402 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:20.403 7f1577fff700 1 -- 172.21.2.102:0/63006547 learned_addr learned my addr 172.21.2.102:0/63006547 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:09:20.402 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:20.403 7f1576ffd700 1 --2- 172.21.2.102:0/63006547 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f15780aa210 0x7f15780c19a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:09:20.402 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:20.403 7f15777fe700 1 --2- 172.21.2.102:0/63006547 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1578081860 0x7f15780a9bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:09:20.402 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:20.403 7f1577fff700 1 -- 172.21.2.102:0/63006547 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1578081860 msgr2=0x7f15780a9bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:09:20.402 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:20.403 7f1577fff700 1 --2- 172.21.2.102:0/63006547 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1578081860 0x7f15780a9bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:09:20.402 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:20.403 7f1577fff700 1 -- 172.21.2.102:0/63006547 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f15780aa210 msgr2=0x7f15780c19a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:09:20.402 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:20.403 7f1577fff700 1 --2- 172.21.2.102:0/63006547 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f15780aa210 0x7f15780c19a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:09:20.403 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:20.403 7f1577fff700 1 -- 172.21.2.102:0/63006547 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15780c25e0 con 0x7f15780aa610 2021-02-10T11:09:20.403 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:20.403 7f1576ffd700 1 --2- 172.21.2.102:0/63006547 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f15780aa210 0x7f15780c19a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:09:20.403 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:20.403 7f1577fff700 1 --2- 172.21.2.102:0/63006547 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f15780aa610 0x7f15780c1fc0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 rx=0x7f157000b9b0 tx=0x7f157000bd50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:09:20.403 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:20.403 7f156ffff700 1 -- 172.21.2.102:0/63006547 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1570005820 con 0x7f15780aa610 2021-02-10T11:09:20.403 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:20.403 7f156ffff700 1 -- 172.21.2.102:0/63006547 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1570005b20 con 0x7f15780aa610 2021-02-10T11:09:20.403 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:20.403 7f156ffff700 1 -- 172.21.2.102:0/63006547 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1570006030 con 0x7f15780aa610 2021-02-10T11:09:20.403 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:20.403 7f157de0e700 1 -- 172.21.2.102:0/63006547 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15780c25e0 con 0x7f15780aa610 2021-02-10T11:09:20.403 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:20.403 7f157de0e700 1 -- 172.21.2.102:0/63006547 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15780c25e0 con 0x7f15780aa610 2021-02-10T11:09:20.403 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:20.404 7f156ffff700 1 -- 172.21.2.102:0/63006547 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1570009450 con 0x7f15780aa610 2021-02-10T11:09:20.403 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:20.404 7f156ffff700 1 --2- 172.21.2.102:0/63006547 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f155c0186d0 0x7f155c01abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:09:20.403 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:20.404 7f156ffff700 1 -- 172.21.2.102:0/63006547 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f1570032b70 con 0x7f15780aa610 2021-02-10T11:09:20.403 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:20.404 7f15777fe700 1 --2- 172.21.2.102:0/63006547 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f155c0186d0 0x7f155c01abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:09:20.404 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:20.404 7f15777fe700 1 --2- 172.21.2.102:0/63006547 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f155c0186d0 0x7f155c01abd0 crc :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:09:20.431 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:20.435 7f157de0e700 1 -- 172.21.2.102:0/63006547 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f157807e6b0 con 0x7f15780aa610 2021-02-10T11:09:20.433 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:20.438 7f156ffff700 1 -- 172.21.2.102:0/63006547 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f1570000f90 con 0x7f15780aa610 2021-02-10T11:09:20.559 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:20.563 7f157de0e700 1 -- 172.21.2.102:0/63006547 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344390", "target": ["mgr", ""]}) v1 -- 0x7f15780ce3e0 con 0x7f155c0186d0 2021-02-10T11:09:20.585 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:20.590 7f156ffff700 1 -- 172.21.2.102:0/63006547 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f15680009b0 con 0x7f155c0186d0 2021-02-10T11:09:20.592 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:20.596 7f157de0e700 1 -- 172.21.2.102:0/63006547 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f155c0186d0 msgr2=0x7f155c01abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:09:20.592 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:20.596 7f157de0e700 1 --2- 172.21.2.102:0/63006547 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f155c0186d0 0x7f155c01abd0 crc :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:09:20.592 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:20.596 7f157de0e700 1 -- 172.21.2.102:0/63006547 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f15780aa610 msgr2=0x7f15780c1fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:09:20.592 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:20.596 7f157de0e700 1 --2- 172.21.2.102:0/63006547 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f15780aa610 0x7f15780c1fc0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 rx=0x7f157000b9b0 tx=0x7f157000bd50).stop 2021-02-10T11:09:20.593 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:20.596 7f157de0e700 1 -- 172.21.2.102:0/63006547 shutdown_connections 2021-02-10T11:09:20.593 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:20.596 7f157de0e700 1 --2- 172.21.2.102:0/63006547 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f15780aa610 0x7f15780c1fc0 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:09:20.593 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:20.596 7f157de0e700 1 --2- 172.21.2.102:0/63006547 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f155c0186d0 0x7f155c01abd0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:09:20.593 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:20.596 7f157de0e700 1 --2- 172.21.2.102:0/63006547 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f15780aa210 0x7f15780c19a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:09:20.593 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:20.596 7f157de0e700 1 --2- 172.21.2.102:0/63006547 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1578081860 0x7f15780a9bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:09:20.593 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:20.596 7f157de0e700 1 -- 172.21.2.102:0/63006547 >> 172.21.2.102:0/63006547 conn(0x7f1578088220 msgr2=0x7f1578081e30 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:09:20.593 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:20.596 7f157de0e700 1 -- 172.21.2.102:0/63006547 shutdown_connections 2021-02-10T11:09:20.593 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:20.596 7f157de0e700 1 -- 172.21.2.102:0/63006547 wait complete. 2021-02-10T11:09:20.600 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000344390 2021-02-10T11:09:21.032 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:21.018 7fc90c3a2700 1 Processor -- start 2021-02-10T11:09:21.032 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:21.018 7fc90c3a2700 1 -- start start 2021-02-10T11:09:21.032 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:21.018 7fc90c3a2700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc9041b0870 0x7fc9040f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:09:21.032 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:21.018 7fc90c3a2700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fc9040efe00 0x7fc9040f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:09:21.032 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:21.018 7fc90c3a2700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9041b0c70 con 0x7fc9041b0470 2021-02-10T11:09:21.032 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:21.018 7fc90c3a2700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9040f0b50 con 0x7fc9040efe00 2021-02-10T11:09:21.032 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:21.018 7fc90c3a2700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9040f0820 con 0x7fc9041b0870 2021-02-10T11:09:21.032 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:21.018 7fc90a1be700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fc9041b0470 0x7fc9040ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:43898/0 (socket says 172.21.2.102:43898) 2021-02-10T11:09:21.032 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:21.019 7fc90a1be700 1 -- 172.21.2.102:0/872936886 learned_addr learned my addr 172.21.2.102:0/872936886 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:09:21.033 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:21.019 7fc9091bc700 1 -- 172.21.2.102:0/872936886 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2654625111 0 0) 0x7fc9000010b0 con 0x7fc9041b0470 2021-02-10T11:09:21.033 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:21.019 7fc9091bc700 1 -- 172.21.2.102:0/872936886 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8f0002fe0 con 0x7fc9041b0470 2021-02-10T11:09:21.033 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:21.019 7fc90a9bf700 1 --2- 172.21.2.102:0/872936886 >> v2:172.21.2.113:3300/0 conn(0x7fc9040efe00 0x7fc9040f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:09:21.033 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:21.019 7fc9099bd700 1 --2- 172.21.2.102:0/872936886 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc9041b0870 0x7fc9040f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:09:21.033 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:21.019 7fc9091bc700 1 -- 172.21.2.102:0/872936886 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2709824652 0 0) 0x7fc9000010b0 con 0x7fc9041b0470 2021-02-10T11:09:21.033 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:21.019 7fc9091bc700 1 -- 172.21.2.102:0/872936886 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc9041b0870 msgr2=0x7fc9040f20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:09:21.033 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:21.019 7fc9091bc700 1 --2- 172.21.2.102:0/872936886 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc9041b0870 0x7fc9040f20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:09:21.033 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:21.019 7fc9091bc700 1 -- 172.21.2.102:0/872936886 >> v2:172.21.2.113:3300/0 conn(0x7fc9040efe00 msgr2=0x7fc9040f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:09:21.033 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:21.019 7fc9091bc700 1 --2- 172.21.2.102:0/872936886 >> v2:172.21.2.113:3300/0 conn(0x7fc9040efe00 0x7fc9040f0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:09:21.033 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:21.019 7fc9091bc700 1 -- 172.21.2.102:0/872936886 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9040e52e0 con 0x7fc9041b0470 2021-02-10T11:09:21.033 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:21.019 7fc9091bc700 1 -- 172.21.2.102:0/872936886 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7fc9000015b0 con 0x7fc9041b0470 2021-02-10T11:09:21.033 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:21.019 7fc9091bc700 1 -- 172.21.2.102:0/872936886 >> v1:172.21.2.102:6789/0 conn(0x7fc9041b0470 legacy=0x7fc9040ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:09:21.033 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:21.019 7fc9091bc700 1 --2- 172.21.2.102:0/872936886 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc8f0003b90 0x7fc8f0005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:09:21.034 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:21.019 7fc9091bc700 1 --2- 172.21.2.102:0/872936886 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc8f0006820 0x7fc8f0008d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:09:21.034 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:21.019 7fc9091bc700 1 --2- 172.21.2.102:0/872936886 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc8f00095a0 0x7fc8f000bac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:09:21.034 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:21.019 7fc9091bc700 1 -- 172.21.2.102:0/872936886 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8f000c370 con 0x7fc8f00095a0 2021-02-10T11:09:21.034 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:21.019 7fc9091bc700 1 -- 172.21.2.102:0/872936886 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8f0050560 con 0x7fc8f0006820 2021-02-10T11:09:21.034 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:21.019 7fc9091bc700 1 -- 172.21.2.102:0/872936886 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8f0050780 con 0x7fc8f0003b90 2021-02-10T11:09:21.034 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:21.020 7fc9099bd700 1 --2- 172.21.2.102:0/872936886 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc9041b0870 0x7fc9040f20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:09:21.034 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:21.020 7fc90a9bf700 1 --2- 172.21.2.102:0/872936886 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc8f00095a0 0x7fc8f000bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:09:21.034 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:21.020 7fc90a1be700 1 --2- 172.21.2.102:0/872936886 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc8f0003b90 0x7fc8f0005fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:09:21.034 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:21.020 7fc9099bd700 1 --2- 172.21.2.102:0/872936886 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc8f0006820 0x7fc8f0008d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:09:21.034 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:21.020 7fc90a9bf700 1 -- 172.21.2.102:0/872936886 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc8f0003b90 msgr2=0x7fc8f0005fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:09:21.034 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:21.020 7fc90a9bf700 1 --2- 172.21.2.102:0/872936886 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc8f0003b90 0x7fc8f0005fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:09:21.034 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:21.020 7fc90a9bf700 1 -- 172.21.2.102:0/872936886 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc8f0006820 msgr2=0x7fc8f0008d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:09:21.035 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:21.020 7fc90a9bf700 1 --2- 172.21.2.102:0/872936886 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc8f0006820 0x7fc8f0008d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:09:21.035 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:21.020 7fc90a9bf700 1 -- 172.21.2.102:0/872936886 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8f0050a20 con 0x7fc8f00095a0 2021-02-10T11:09:21.035 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:21.020 7fc9099bd700 1 --2- 172.21.2.102:0/872936886 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc8f0006820 0x7fc8f0008d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:09:21.035 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:21.020 7fc90a9bf700 1 --2- 172.21.2.102:0/872936886 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc8f00095a0 0x7fc8f000bac0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 rx=0x7fc8fc006a40 tx=0x7fc8fc00c0c0).ready entity=mon.0 client_cookie=81d3cf85ea6d9dcd server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:09:21.035 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:21.020 7fc9091bc700 1 -- 172.21.2.102:0/872936886 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc8fc00c4b0 con 0x7fc8f00095a0 2021-02-10T11:09:21.035 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:21.020 7fc9091bc700 1 -- 172.21.2.102:0/872936886 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8fc00c7b0 con 0x7fc8f00095a0 2021-02-10T11:09:21.035 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:21.020 7fc9091bc700 1 -- 172.21.2.102:0/872936886 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc8fc00ccd0 con 0x7fc8f00095a0 2021-02-10T11:09:21.035 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:21.020 7fc90c3a2700 1 -- 172.21.2.102:0/872936886 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc8f00095a0 msgr2=0x7fc8f000bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:09:21.035 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:21.020 7fc90c3a2700 1 --2- 172.21.2.102:0/872936886 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc8f00095a0 0x7fc8f000bac0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 rx=0x7fc8fc006a40 tx=0x7fc8fc00c0c0).stop 2021-02-10T11:09:21.035 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:21.020 7fc90a9bf700 1 -- 172.21.2.102:0/872936886 reap_dead start 2021-02-10T11:09:21.035 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:21.021 7fc90c3a2700 1 -- 172.21.2.102:0/872936886 shutdown_connections 2021-02-10T11:09:21.035 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:21.021 7fc90c3a2700 1 -- 172.21.2.102:0/872936886 >> 172.21.2.102:0/872936886 conn(0x7fc9040ce3e0 msgr2=0x7fc9040e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:09:21.035 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:21.021 7fc90c3a2700 1 -- 172.21.2.102:0/872936886 shutdown_connections 2021-02-10T11:09:21.036 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:21.021 7fc90c3a2700 1 -- 172.21.2.102:0/872936886 wait complete. 2021-02-10T11:09:21.036 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:21.036 7fc90c3a2700 1 Processor -- start 2021-02-10T11:09:21.036 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:21.036 7fc90c3a2700 1 -- start start 2021-02-10T11:09:21.037 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:21.036 7fc90c3a2700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc9040b7db0 0x7fc9040b8260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:09:21.037 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:21.036 7fc90c3a2700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc9040b8880 0x7fc9041a7450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:09:21.037 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:21.036 7fc90c3a2700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc9040b8c80 0x7fc9041a7a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:09:21.038 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:21.036 7fc90c3a2700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9040b79b0 con 0x7fc9040b7db0 2021-02-10T11:09:21.038 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:21.036 7fc90c3a2700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9041a8090 con 0x7fc9040b8c80 2021-02-10T11:09:21.038 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:21.036 7fc90c3a2700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9041a82b0 con 0x7fc9040b8880 2021-02-10T11:09:21.038 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:21.036 7fc90a1be700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc9040b7db0 0x7fc9040b8260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:09:21.038 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:21.036 7fc90a9bf700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc9040b8c80 0x7fc9041a7a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:09:21.038 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:21.037 7fc90a1be700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc9040b7db0 0x7fc9040b8260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:51688/0 (socket says 172.21.2.102:51688) 2021-02-10T11:09:21.038 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:21.037 7fc90a1be700 1 -- 172.21.2.102:0/957575355 learned_addr learned my addr 172.21.2.102:0/957575355 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:09:21.038 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:21.037 7fc9099bd700 1 --2- 172.21.2.102:0/957575355 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc9040b8880 0x7fc9041a7450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:09:21.038 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:21.037 7fc90a9bf700 1 -- 172.21.2.102:0/957575355 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc9040b8880 msgr2=0x7fc9041a7450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:09:21.038 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:21.037 7fc90a9bf700 1 --2- 172.21.2.102:0/957575355 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc9040b8880 0x7fc9041a7450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:09:21.038 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:21.037 7fc90a9bf700 1 -- 172.21.2.102:0/957575355 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc9040b7db0 msgr2=0x7fc9040b8260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:09:21.038 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:21.037 7fc90a9bf700 1 --2- 172.21.2.102:0/957575355 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc9040b7db0 0x7fc9040b8260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:09:21.038 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:21.037 7fc90a9bf700 1 -- 172.21.2.102:0/957575355 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc904190c10 con 0x7fc9040b8c80 2021-02-10T11:09:21.039 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:21.037 7fc90a9bf700 1 --2- 172.21.2.102:0/957575355 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc9040b8c80 0x7fc9041a7a70 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7fc8fc00b8e0 tx=0x7fc8fc00bc80).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:09:21.039 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:21.037 7fc8f6ffd700 1 -- 172.21.2.102:0/957575355 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc8fc009500 con 0x7fc9040b8c80 2021-02-10T11:09:21.039 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:21.037 7fc8f6ffd700 1 -- 172.21.2.102:0/957575355 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8fc002030 con 0x7fc9040b8c80 2021-02-10T11:09:21.039 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:21.037 7fc8f6ffd700 1 -- 172.21.2.102:0/957575355 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc8fc002550 con 0x7fc9040b8c80 2021-02-10T11:09:21.039 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:21.037 7fc90c3a2700 1 -- 172.21.2.102:0/957575355 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc904190c10 con 0x7fc9040b8c80 2021-02-10T11:09:21.039 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:21.037 7fc90c3a2700 1 -- 172.21.2.102:0/957575355 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc904190c10 con 0x7fc9040b8c80 2021-02-10T11:09:21.039 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:21.038 7fc8f6ffd700 1 -- 172.21.2.102:0/957575355 <== mon.1 v2:172.21.2.113:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fc8fc009500 con 0x7fc9040b8c80 2021-02-10T11:09:21.039 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:21.038 7fc8f6ffd700 1 --2- 172.21.2.102:0/957575355 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc8ec0186c0 0x7fc8ec01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:09:21.039 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:21.038 7fc8f6ffd700 1 -- 172.21.2.102:0/957575355 <== mon.1 v2:172.21.2.113:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fc8fc0326d0 con 0x7fc9040b8c80 2021-02-10T11:09:21.039 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:21.038 7fc90a1be700 1 --2- 172.21.2.102:0/957575355 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc8ec0186c0 0x7fc8ec01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:09:21.039 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:21.038 7fc90a1be700 1 --2- 172.21.2.102:0/957575355 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc8ec0186c0 0x7fc8ec01abc0 crc :-1 s=READY pgs=225 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:09:21.064 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:21.068 7fc90c3a2700 1 -- 172.21.2.102:0/957575355 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc904105b30 con 0x7fc9040b8c80 2021-02-10T11:09:21.067 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:21.071 7fc8f6ffd700 1 -- 172.21.2.102:0/957575355 <== mon.1 v2:172.21.2.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fc8fc02f390 con 0x7fc9040b8c80 2021-02-10T11:09:21.199 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:21.203 7fc90c3a2700 1 -- 172.21.2.102:0/957575355 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344390", "target": ["mgr", ""]}) v1 -- 0x7fc9040f0b50 con 0x7fc8ec0186c0 2021-02-10T11:09:21.202 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:21.207 7fc8f6ffd700 1 -- 172.21.2.102:0/957575355 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7fc900000940 con 0x7fc8ec0186c0 2021-02-10T11:09:21.203 INFO:teuthology.orchestra.run.gibba002.stdout:/volumes/_nogroup/subvolume_0000000000344390/73c6c0c1-0078-4fe0-abb7-b8998fd32107 2021-02-10T11:09:21.204 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:21.208 7fc90c3a2700 1 -- 172.21.2.102:0/957575355 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc8ec0186c0 msgr2=0x7fc8ec01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:09:21.204 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:21.208 7fc90c3a2700 1 --2- 172.21.2.102:0/957575355 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc8ec0186c0 0x7fc8ec01abc0 crc :-1 s=READY pgs=225 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:09:21.204 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:21.208 7fc90c3a2700 1 -- 172.21.2.102:0/957575355 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc9040b8c80 msgr2=0x7fc9041a7a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:09:21.204 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:21.208 7fc90c3a2700 1 --2- 172.21.2.102:0/957575355 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc9040b8c80 0x7fc9041a7a70 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7fc8fc00b8e0 tx=0x7fc8fc00bc80).stop 2021-02-10T11:09:21.204 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:21.208 7fc90c3a2700 1 -- 172.21.2.102:0/957575355 shutdown_connections 2021-02-10T11:09:21.205 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:21.208 7fc90c3a2700 1 --2- 172.21.2.102:0/957575355 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc8ec0186c0 0x7fc8ec01abc0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:09:21.205 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:21.208 7fc90c3a2700 1 --2- 172.21.2.102:0/957575355 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc9040b8c80 0x7fc9041a7a70 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:09:21.205 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:21.208 7fc90c3a2700 1 --2- 172.21.2.102:0/957575355 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc9040b8880 0x7fc9041a7450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:09:21.205 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:21.208 7fc90c3a2700 1 --2- 172.21.2.102:0/957575355 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc9040b7db0 0x7fc9040b8260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:09:21.205 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:21.208 7fc90c3a2700 1 -- 172.21.2.102:0/957575355 >> 172.21.2.102:0/957575355 conn(0x7fc904094e10 msgr2=0x7fc9040a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:09:21.205 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:21.208 7fc90c3a2700 1 -- 172.21.2.102:0/957575355 shutdown_connections 2021-02-10T11:09:21.205 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:21.208 7fc90c3a2700 1 -- 172.21.2.102:0/957575355 wait complete. 2021-02-10T11:09:21.212 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000344390 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000344390/73c6c0c1-0078-4fe0-abb7-b8998fd32107 2021-02-10T11:09:21.212 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:09:21.212 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:09:21.212 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:09:21.549 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:09:21.550 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:09:21.550 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344390/73c6c0c1-0078-4fe0-abb7-b8998fd32107/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:09:21.933 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:09:21.933 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:09:21.933 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0251006 s, 41.8 MB/s 2021-02-10T11:09:21.934 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:09:21.935 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:09:21.935 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:09:22.270 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:09:22.270 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:09:22.270 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344390/73c6c0c1-0078-4fe0-abb7-b8998fd32107/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:09:22.647 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:09:22.647 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:09:22.647 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0251767 s, 41.6 MB/s 2021-02-10T11:09:22.648 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:09:22.648 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:09:22.648 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:09:22.983 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:09:22.983 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:09:22.983 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344390/73c6c0c1-0078-4fe0-abb7-b8998fd32107/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:09:23.355 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:09:23.356 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:09:23.356 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.023731 s, 44.2 MB/s 2021-02-10T11:09:23.357 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:09:23.357 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:09:23.357 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:09:23.692 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:09:23.693 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:09:23.693 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344390/73c6c0c1-0078-4fe0-abb7-b8998fd32107/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:09:24.068 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:09:24.068 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:09:24.068 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0237883 s, 44.1 MB/s 2021-02-10T11:09:24.069 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:09:24.069 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:09:24.069 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:09:24.404 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:09:24.405 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:09:24.405 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344390/73c6c0c1-0078-4fe0-abb7-b8998fd32107/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:09:24.780 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:09:24.780 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:09:24.780 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0243353 s, 43.1 MB/s 2021-02-10T11:09:24.781 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:09:24.782 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:09:24.782 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:09:25.117 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:09:25.117 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:09:25.117 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344390/73c6c0c1-0078-4fe0-abb7-b8998fd32107/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:09:25.493 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:09:25.660 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:09:25.660 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0256985 s, 40.8 MB/s 2021-02-10T11:09:25.661 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:09:25.661 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:09:25.661 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:09:25.997 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:09:25.998 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:09:25.998 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344390/73c6c0c1-0078-4fe0-abb7-b8998fd32107/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:09:26.375 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:09:26.375 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:09:26.375 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0250115 s, 41.9 MB/s 2021-02-10T11:09:26.376 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:09:26.376 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:09:26.376 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:09:26.711 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:09:26.711 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:09:26.711 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344390/73c6c0c1-0078-4fe0-abb7-b8998fd32107/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:09:27.084 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:09:27.084 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:09:27.085 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0229838 s, 45.6 MB/s 2021-02-10T11:09:27.086 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:09:27.086 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:09:27.086 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:09:27.422 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:09:27.422 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:09:27.422 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344390/73c6c0c1-0078-4fe0-abb7-b8998fd32107/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:09:27.803 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:09:27.804 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:09:27.804 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0257032 s, 40.8 MB/s 2021-02-10T11:09:27.805 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:09:27.805 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:09:27.805 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:09:28.143 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:09:28.143 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:09:28.143 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344390/73c6c0c1-0078-4fe0-abb7-b8998fd32107/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:09:28.519 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:09:28.519 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:09:28.519 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0253204 s, 41.4 MB/s 2021-02-10T11:09:28.520 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000344391 2021-02-10T11:09:28.951 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:28.937 7fdb632a5700 1 Processor -- start 2021-02-10T11:09:28.951 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:28.938 7fdb632a5700 1 -- start start 2021-02-10T11:09:28.952 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:28.938 7fdb632a5700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdb5c1b0870 0x7fdb5c0f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:09:28.952 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:28.938 7fdb632a5700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fdb5c0efe00 0x7fdb5c0f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:09:28.952 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:28.938 7fdb632a5700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb5c1b0c70 con 0x7fdb5c1b0470 2021-02-10T11:09:28.952 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:28.938 7fdb632a5700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb5c0f0b50 con 0x7fdb5c0efe00 2021-02-10T11:09:28.952 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:28.938 7fdb632a5700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb5c0f0820 con 0x7fdb5c1b0870 2021-02-10T11:09:28.953 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:28.939 7fdb618c2700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fdb5c0efe00 0x7fdb5c0f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:09:28.953 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:28.939 7fdb608c0700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdb5c1b0870 0x7fdb5c0f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:09:28.953 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:28.939 7fdb610c1700 1 --1- >> v1:172.21.2.113:6789/0 conn(0x7fdb5c1b0470 0x7fdb5c0ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.113:6789/0 says I am v1:172.21.2.102:51242/0 (socket says 172.21.2.102:51242) 2021-02-10T11:09:28.953 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:28.939 7fdb610c1700 1 -- 172.21.2.102:0/1316127505 learned_addr learned my addr 172.21.2.102:0/1316127505 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:09:28.953 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:28.939 7fdb618c2700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fdb5c0efe00 0x7fdb5c0f0200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.113:3300/0 says I am v2:172.21.2.102:55128/0 (socket says 172.21.2.102:55128) 2021-02-10T11:09:28.954 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:28.939 7fdb618c2700 1 -- 172.21.2.102:0/1316127505 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdb5c1b0870 msgr2=0x7fdb5c0f20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:09:28.954 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:28.939 7fdb618c2700 1 --2- 172.21.2.102:0/1316127505 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdb5c1b0870 0x7fdb5c0f20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:09:28.954 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:28.939 7fdb618c2700 1 -- 172.21.2.102:0/1316127505 >> v1:172.21.2.113:6789/0 conn(0x7fdb5c1b0470 legacy=0x7fdb5c0ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:09:28.954 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:28.939 7fdb618c2700 1 -- 172.21.2.102:0/1316127505 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb5c0e52e0 con 0x7fdb5c0efe00 2021-02-10T11:09:28.954 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:28.939 7fdb608c0700 1 --2- 172.21.2.102:0/1316127505 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdb5c1b0870 0x7fdb5c0f20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:09:28.954 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:28.939 7fdb618c2700 1 --2- 172.21.2.102:0/1316127505 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fdb5c0efe00 0x7fdb5c0f0200 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7fdb54006e50 tx=0x7fdb54009e70).ready entity=mon.1 client_cookie=df3c8b5402cfb7c1 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:09:28.955 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:28.940 7fdb53fff700 1 -- 172.21.2.102:0/1316127505 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdb540066f0 con 0x7fdb5c0efe00 2021-02-10T11:09:28.958 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:28.940 7fdb53fff700 1 -- 172.21.2.102:0/1316127505 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb54003030 con 0x7fdb5c0efe00 2021-02-10T11:09:28.958 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:28.940 7fdb53fff700 1 -- 172.21.2.102:0/1316127505 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdb5400b700 con 0x7fdb5c0efe00 2021-02-10T11:09:28.959 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:28.940 7fdb632a5700 1 -- 172.21.2.102:0/1316127505 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fdb5c0efe00 msgr2=0x7fdb5c0f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:09:28.959 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:28.940 7fdb632a5700 1 --2- 172.21.2.102:0/1316127505 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fdb5c0efe00 0x7fdb5c0f0200 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7fdb54006e50 tx=0x7fdb54009e70).stop 2021-02-10T11:09:28.959 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:28.940 7fdb632a5700 1 -- 172.21.2.102:0/1316127505 shutdown_connections 2021-02-10T11:09:28.959 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:28.940 7fdb632a5700 1 --2- 172.21.2.102:0/1316127505 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdb5c1b0870 0x7fdb5c0f20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:09:28.959 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:28.940 7fdb632a5700 1 --2- 172.21.2.102:0/1316127505 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fdb5c0efe00 0x7fdb5c0f0200 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:09:28.959 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:28.940 7fdb632a5700 1 -- 172.21.2.102:0/1316127505 >> 172.21.2.102:0/1316127505 conn(0x7fdb5c0ce3e0 msgr2=0x7fdb5c0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:09:28.960 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:28.941 7fdb632a5700 1 -- 172.21.2.102:0/1316127505 shutdown_connections 2021-02-10T11:09:28.960 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:28.941 7fdb632a5700 1 -- 172.21.2.102:0/1316127505 wait complete. 2021-02-10T11:09:28.960 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:28.955 7fdb632a5700 1 Processor -- start 2021-02-10T11:09:28.960 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:28.955 7fdb632a5700 1 -- start start 2021-02-10T11:09:28.960 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:28.955 7fdb632a5700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdb5c190920 0x7fdb5c0b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:09:28.960 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:28.956 7fdb632a5700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fdb5c0b8200 0x7fdb5c0b8600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:09:28.960 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:28.956 7fdb632a5700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdb5c0c1d80 0x7fdb5c0c2180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:09:28.961 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:28.956 7fdb632a5700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb5c190d20 con 0x7fdb5c190920 2021-02-10T11:09:28.961 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:28.956 7fdb632a5700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb5c0b8c20 con 0x7fdb5c0b8200 2021-02-10T11:09:28.961 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:28.956 7fdb632a5700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb5c0c27a0 con 0x7fdb5c0c1d80 2021-02-10T11:09:28.961 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:28.956 7fdb610c1700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdb5c190920 0x7fdb5c0b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:09:28.961 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:28.956 7fdb610c1700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdb5c190920 0x7fdb5c0b7be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:51702/0 (socket says 172.21.2.102:51702) 2021-02-10T11:09:28.961 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:28.956 7fdb610c1700 1 -- 172.21.2.102:0/992737915 learned_addr learned my addr 172.21.2.102:0/992737915 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:09:28.961 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:28.956 7fdb610c1700 1 -- 172.21.2.102:0/992737915 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdb5c0c1d80 msgr2=0x7fdb5c0c2180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:09:28.962 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:28.956 7fdb610c1700 1 --2- 172.21.2.102:0/992737915 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdb5c0c1d80 0x7fdb5c0c2180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:09:28.962 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:28.956 7fdb610c1700 1 -- 172.21.2.102:0/992737915 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fdb5c0b8200 msgr2=0x7fdb5c0b8600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:09:28.962 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:28.956 7fdb610c1700 1 --2- 172.21.2.102:0/992737915 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fdb5c0b8200 0x7fdb5c0b8600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:09:28.962 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:28.956 7fdb610c1700 1 -- 172.21.2.102:0/992737915 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb5c0c29c0 con 0x7fdb5c190920 2021-02-10T11:09:28.962 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:28.956 7fdb610c1700 1 --2- 172.21.2.102:0/992737915 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdb5c190920 0x7fdb5c0b7be0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 rx=0x7fdb5800a580 tx=0x7fdb5800a920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:09:28.962 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:28.957 7fdb51ffb700 1 -- 172.21.2.102:0/992737915 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdb5800ad10 con 0x7fdb5c190920 2021-02-10T11:09:28.963 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:28.957 7fdb51ffb700 1 -- 172.21.2.102:0/992737915 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb58004320 con 0x7fdb5c190920 2021-02-10T11:09:28.963 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:28.957 7fdb51ffb700 1 -- 172.21.2.102:0/992737915 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdb58004840 con 0x7fdb5c190920 2021-02-10T11:09:28.963 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:28.957 7fdb632a5700 1 -- 172.21.2.102:0/992737915 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb5c0c29c0 con 0x7fdb5c190920 2021-02-10T11:09:28.963 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:28.957 7fdb632a5700 1 -- 172.21.2.102:0/992737915 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb5c0c2f50 con 0x7fdb5c190920 2021-02-10T11:09:28.963 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:28.957 7fdb51ffb700 1 -- 172.21.2.102:0/992737915 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fdb58004500 con 0x7fdb5c190920 2021-02-10T11:09:28.963 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:28.957 7fdb51ffb700 1 --2- 172.21.2.102:0/992737915 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fdb3c0186c0 0x7fdb3c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:09:28.963 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:28.957 7fdb51ffb700 1 -- 172.21.2.102:0/992737915 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fdb58032520 con 0x7fdb5c190920 2021-02-10T11:09:28.964 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:28.958 7fdb608c0700 1 --2- 172.21.2.102:0/992737915 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fdb3c0186c0 0x7fdb3c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:09:28.964 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:28.958 7fdb608c0700 1 --2- 172.21.2.102:0/992737915 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fdb3c0186c0 0x7fdb3c01abc0 crc :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:09:28.985 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:28.988 7fdb632a5700 1 -- 172.21.2.102:0/992737915 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb5c105ae0 con 0x7fdb5c190920 2021-02-10T11:09:28.986 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:28.991 7fdb51ffb700 1 -- 172.21.2.102:0/992737915 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fdb58000c60 con 0x7fdb5c190920 2021-02-10T11:09:29.112 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:29.117 7fdb632a5700 1 -- 172.21.2.102:0/992737915 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344391", "target": ["mgr", ""]}) v1 -- 0x7fdb5c190d20 con 0x7fdb3c0186c0 2021-02-10T11:09:29.140 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:29.144 7fdb51ffb700 1 -- 172.21.2.102:0/992737915 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fdb4c0009b0 con 0x7fdb3c0186c0 2021-02-10T11:09:29.146 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:29.149 7fdb632a5700 1 -- 172.21.2.102:0/992737915 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fdb3c0186c0 msgr2=0x7fdb3c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:09:29.146 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:29.149 7fdb632a5700 1 --2- 172.21.2.102:0/992737915 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fdb3c0186c0 0x7fdb3c01abc0 crc :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:09:29.146 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:29.149 7fdb632a5700 1 -- 172.21.2.102:0/992737915 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdb5c190920 msgr2=0x7fdb5c0b7be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:09:29.147 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:29.149 7fdb632a5700 1 --2- 172.21.2.102:0/992737915 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdb5c190920 0x7fdb5c0b7be0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 rx=0x7fdb5800a580 tx=0x7fdb5800a920).stop 2021-02-10T11:09:29.147 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:29.150 7fdb632a5700 1 -- 172.21.2.102:0/992737915 shutdown_connections 2021-02-10T11:09:29.147 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:29.150 7fdb632a5700 1 --2- 172.21.2.102:0/992737915 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdb5c0c1d80 0x7fdb5c0c2180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:09:29.147 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:29.150 7fdb632a5700 1 --2- 172.21.2.102:0/992737915 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fdb3c0186c0 0x7fdb3c01abc0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:09:29.147 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:29.150 7fdb632a5700 1 --2- 172.21.2.102:0/992737915 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fdb5c0b8200 0x7fdb5c0b8600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:09:29.147 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:29.150 7fdb632a5700 1 --2- 172.21.2.102:0/992737915 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdb5c190920 0x7fdb5c0b7be0 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:09:29.147 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:29.150 7fdb632a5700 1 -- 172.21.2.102:0/992737915 >> 172.21.2.102:0/992737915 conn(0x7fdb5c094e10 msgr2=0x7fdb5c0a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:09:29.147 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:29.150 7fdb632a5700 1 -- 172.21.2.102:0/992737915 shutdown_connections 2021-02-10T11:09:29.148 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:29.150 7fdb632a5700 1 -- 172.21.2.102:0/992737915 wait complete. 2021-02-10T11:09:29.154 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000344391 2021-02-10T11:09:29.589 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:29.577 7ff33f324700 1 Processor -- start 2021-02-10T11:09:29.589 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:29.577 7ff33f324700 1 -- start start 2021-02-10T11:09:29.589 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:29.578 7ff33f324700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7ff3381b0870 0x7ff3380f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:09:29.589 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:29.578 7ff33f324700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7ff3380efe00 0x7ff3380f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:09:29.589 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:29.578 7ff33f324700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3381b0c70 con 0x7ff3381b0470 2021-02-10T11:09:29.589 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:29.578 7ff33f324700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3380e5ea0 con 0x7ff3380efe00 2021-02-10T11:09:29.590 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:29.578 7ff33f324700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3380f0b50 con 0x7ff3381b0870 2021-02-10T11:09:29.590 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:29.578 7ff33d941700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7ff3380efe00 0x7ff3380f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:09:29.590 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:29.578 7ff33d941700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7ff3380efe00 0x7ff3380f0200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:51712/0 (socket says 172.21.2.102:51712) 2021-02-10T11:09:29.590 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:29.578 7ff33d941700 1 -- 172.21.2.102:0/3706620688 learned_addr learned my addr 172.21.2.102:0/3706620688 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:09:29.590 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:29.578 7ff33d941700 1 -- 172.21.2.102:0/3706620688 >> v2:172.21.2.113:3300/0 conn(0x7ff3381b0870 msgr2=0x7ff3380f20e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:09:29.590 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:29.578 7ff33d941700 1 --2- 172.21.2.102:0/3706620688 >> v2:172.21.2.113:3300/0 conn(0x7ff3381b0870 0x7ff3380f20e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:09:29.590 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:29.578 7ff33d941700 1 -- 172.21.2.102:0/3706620688 >> v1:172.21.2.113:6789/0 conn(0x7ff3381b0470 legacy=0x7ff3380ef610 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:09:29.590 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:29.578 7ff33d941700 1 -- 172.21.2.102:0/3706620688 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3380f0820 con 0x7ff3380efe00 2021-02-10T11:09:29.590 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:29.578 7ff33d941700 1 --2- 172.21.2.102:0/3706620688 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff3380efe00 0x7ff3380f0200 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 rx=0x7ff330006f00 tx=0x7ff330009f10).ready entity=mon.0 client_cookie=fb6b201a7e68ab30 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:09:29.590 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:29.578 7ff32bfff700 1 -- 172.21.2.102:0/3706620688 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff33000b3b0 con 0x7ff3380efe00 2021-02-10T11:09:29.590 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:29.578 7ff32bfff700 1 -- 172.21.2.102:0/3706620688 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff33000b630 con 0x7ff3380efe00 2021-02-10T11:09:29.590 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:29.578 7ff32bfff700 1 -- 172.21.2.102:0/3706620688 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff33000bb50 con 0x7ff3380efe00 2021-02-10T11:09:29.590 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:29.578 7ff33f324700 1 -- 172.21.2.102:0/3706620688 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff3380efe00 msgr2=0x7ff3380f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:09:29.591 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:29.578 7ff33f324700 1 --2- 172.21.2.102:0/3706620688 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff3380efe00 0x7ff3380f0200 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 rx=0x7ff330006f00 tx=0x7ff330009f10).stop 2021-02-10T11:09:29.591 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:29.579 7ff33f324700 1 -- 172.21.2.102:0/3706620688 shutdown_connections 2021-02-10T11:09:29.591 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:29.579 7ff33f324700 1 --2- 172.21.2.102:0/3706620688 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff3380efe00 0x7ff3380f0200 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:09:29.591 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:29.579 7ff33f324700 1 --2- 172.21.2.102:0/3706620688 >> v2:172.21.2.113:3300/0 conn(0x7ff3381b0870 0x7ff3380f20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:09:29.591 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:29.579 7ff33f324700 1 -- 172.21.2.102:0/3706620688 >> 172.21.2.102:0/3706620688 conn(0x7ff3380ce3e0 msgr2=0x7ff3380e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:09:29.591 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:29.579 7ff33f324700 1 -- 172.21.2.102:0/3706620688 shutdown_connections 2021-02-10T11:09:29.591 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:29.579 7ff33f324700 1 -- 172.21.2.102:0/3706620688 wait complete. 2021-02-10T11:09:29.591 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:29.593 7ff33f324700 1 Processor -- start 2021-02-10T11:09:29.592 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:29.593 7ff33f324700 1 -- start start 2021-02-10T11:09:29.592 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:29.594 7ff33f324700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff3380c36a0 0x7ff3380c2b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:09:29.593 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:29.594 7ff33f324700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff3380c3190 0x7ff3380b1d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:09:29.593 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:29.594 7ff33f324700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff3380b2370 0x7ff3380b2d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:09:29.593 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:29.594 7ff33f324700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3380b27c0 con 0x7ff3380c36a0 2021-02-10T11:09:29.593 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:29.594 7ff33f324700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3380b3360 con 0x7ff3380b2370 2021-02-10T11:09:29.593 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:29.594 7ff33f324700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3380b3540 con 0x7ff3380c3190 2021-02-10T11:09:29.593 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:29.594 7ff33d140700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff3380c36a0 0x7ff3380c2b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:09:29.593 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:29.594 7ff33d140700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff3380c36a0 0x7ff3380c2b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:51714/0 (socket says 172.21.2.102:51714) 2021-02-10T11:09:29.593 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:29.594 7ff33d140700 1 -- 172.21.2.102:0/3650303630 learned_addr learned my addr 172.21.2.102:0/3650303630 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:09:29.593 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:29.594 7ff33d140700 1 -- 172.21.2.102:0/3650303630 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff3380c3190 msgr2=0x7ff3380b1d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:09:29.593 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:29.594 7ff33d140700 1 --2- 172.21.2.102:0/3650303630 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff3380c3190 0x7ff3380b1d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:09:29.593 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:29.594 7ff33d140700 1 -- 172.21.2.102:0/3650303630 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff3380b2370 msgr2=0x7ff3380b2d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:09:29.593 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:29.594 7ff33d941700 1 --2- 172.21.2.102:0/3650303630 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff3380b2370 0x7ff3380b2d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:09:29.594 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:29.594 7ff33d140700 1 --2- 172.21.2.102:0/3650303630 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff3380b2370 0x7ff3380b2d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:09:29.594 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:29.594 7ff33d140700 1 -- 172.21.2.102:0/3650303630 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3380b3760 con 0x7ff3380c36a0 2021-02-10T11:09:29.594 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:29.594 7ff33d941700 1 --2- 172.21.2.102:0/3650303630 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff3380b2370 0x7ff3380b2d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:09:29.594 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:29.594 7ff33d140700 1 --2- 172.21.2.102:0/3650303630 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff3380c36a0 0x7ff3380c2b70 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 rx=0x7ff32c00b530 tx=0x7ff32c00b8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:09:29.594 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:29.595 7ff329ffb700 1 -- 172.21.2.102:0/3650303630 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff32c00bcc0 con 0x7ff3380c36a0 2021-02-10T11:09:29.594 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:29.595 7ff329ffb700 1 -- 172.21.2.102:0/3650303630 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff32c006020 con 0x7ff3380c36a0 2021-02-10T11:09:29.594 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:29.595 7ff329ffb700 1 -- 172.21.2.102:0/3650303630 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff32c0054a0 con 0x7ff3380c36a0 2021-02-10T11:09:29.594 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:29.595 7ff33f324700 1 -- 172.21.2.102:0/3650303630 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3380b3760 con 0x7ff3380c36a0 2021-02-10T11:09:29.594 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:29.595 7ff33f324700 1 -- 172.21.2.102:0/3650303630 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3380b3cf0 con 0x7ff3380c36a0 2021-02-10T11:09:29.594 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:29.595 7ff329ffb700 1 -- 172.21.2.102:0/3650303630 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ff32c006200 con 0x7ff3380c36a0 2021-02-10T11:09:29.595 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:29.595 7ff329ffb700 1 --2- 172.21.2.102:0/3650303630 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff3180186c0 0x7ff31801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:09:29.595 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:29.595 7ff329ffb700 1 -- 172.21.2.102:0/3650303630 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7ff32c032540 con 0x7ff3380c36a0 2021-02-10T11:09:29.596 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:29.596 7ff33c93f700 1 --2- 172.21.2.102:0/3650303630 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff3180186c0 0x7ff31801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:09:29.596 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:29.596 7ff33c93f700 1 --2- 172.21.2.102:0/3650303630 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff3180186c0 0x7ff31801abc0 crc :-1 s=READY pgs=227 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:09:29.624 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:29.628 7ff33f324700 1 -- 172.21.2.102:0/3650303630 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3380edd10 con 0x7ff3380c36a0 2021-02-10T11:09:29.626 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:29.631 7ff329ffb700 1 -- 172.21.2.102:0/3650303630 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ff32c000c10 con 0x7ff3380c36a0 2021-02-10T11:09:29.756 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:29.760 7ff33f324700 1 -- 172.21.2.102:0/3650303630 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344391", "target": ["mgr", ""]}) v1 -- 0x7ff3380b27c0 con 0x7ff3180186c0 2021-02-10T11:09:29.759 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:29.763 7ff329ffb700 1 -- 172.21.2.102:0/3650303630 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7ff334006e40 con 0x7ff3180186c0 2021-02-10T11:09:29.763 INFO:teuthology.orchestra.run.gibba002.stdout:/volumes/_nogroup/subvolume_0000000000344391/6d29d995-3b4f-4f77-803e-9fe18dca4f7f 2021-02-10T11:09:29.764 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:29.768 7ff33f324700 1 -- 172.21.2.102:0/3650303630 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff3180186c0 msgr2=0x7ff31801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:09:29.764 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:29.768 7ff33f324700 1 --2- 172.21.2.102:0/3650303630 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff3180186c0 0x7ff31801abc0 crc :-1 s=READY pgs=227 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:09:29.764 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:29.768 7ff33f324700 1 -- 172.21.2.102:0/3650303630 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff3380c36a0 msgr2=0x7ff3380c2b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:09:29.764 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:29.768 7ff33f324700 1 --2- 172.21.2.102:0/3650303630 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff3380c36a0 0x7ff3380c2b70 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 rx=0x7ff32c00b530 tx=0x7ff32c00b8d0).stop 2021-02-10T11:09:29.765 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:29.769 7ff33f324700 1 -- 172.21.2.102:0/3650303630 shutdown_connections 2021-02-10T11:09:29.765 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:29.769 7ff33f324700 1 --2- 172.21.2.102:0/3650303630 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff3180186c0 0x7ff31801abc0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:09:29.765 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:29.769 7ff33f324700 1 --2- 172.21.2.102:0/3650303630 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff3380b2370 0x7ff3380b2d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:09:29.765 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:29.769 7ff33f324700 1 --2- 172.21.2.102:0/3650303630 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff3380c3190 0x7ff3380b1d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:09:29.765 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:29.769 7ff33f324700 1 --2- 172.21.2.102:0/3650303630 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff3380c36a0 0x7ff3380c2b70 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:09:29.765 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:29.769 7ff33f324700 1 -- 172.21.2.102:0/3650303630 >> 172.21.2.102:0/3650303630 conn(0x7ff33809f630 msgr2=0x7ff3380a0130 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:09:29.765 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:29.769 7ff33f324700 1 -- 172.21.2.102:0/3650303630 shutdown_connections 2021-02-10T11:09:29.765 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:29.769 7ff33f324700 1 -- 172.21.2.102:0/3650303630 wait complete. 2021-02-10T11:09:29.773 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000344391 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000344391/6d29d995-3b4f-4f77-803e-9fe18dca4f7f 2021-02-10T11:09:29.773 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:09:29.773 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:09:29.773 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:09:30.111 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:09:30.111 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:09:30.111 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344391/6d29d995-3b4f-4f77-803e-9fe18dca4f7f/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:09:30.490 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:09:30.490 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:09:30.490 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0221672 s, 47.3 MB/s 2021-02-10T11:09:30.491 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:09:30.491 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:09:30.491 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:09:30.825 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:09:30.826 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:09:30.826 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344391/6d29d995-3b4f-4f77-803e-9fe18dca4f7f/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:09:31.200 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:09:31.201 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:09:31.201 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0231119 s, 45.4 MB/s 2021-02-10T11:09:31.202 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:09:31.202 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:09:31.202 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:09:31.543 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:09:31.544 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:09:31.544 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344391/6d29d995-3b4f-4f77-803e-9fe18dca4f7f/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:09:31.922 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:09:31.922 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:09:31.922 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.025938 s, 40.4 MB/s 2021-02-10T11:09:31.923 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:09:31.923 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:09:31.923 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:09:32.257 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:09:32.258 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:09:32.258 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344391/6d29d995-3b4f-4f77-803e-9fe18dca4f7f/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:09:32.631 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:09:32.631 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:09:32.632 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0229081 s, 45.8 MB/s 2021-02-10T11:09:32.632 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:09:32.633 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:09:32.633 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:09:32.967 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:09:32.968 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:09:32.968 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344391/6d29d995-3b4f-4f77-803e-9fe18dca4f7f/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:09:33.341 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:09:33.341 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:09:33.341 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0247536 s, 42.4 MB/s 2021-02-10T11:09:33.342 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:09:33.343 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:09:33.343 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:09:33.678 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:09:33.678 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:09:33.678 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344391/6d29d995-3b4f-4f77-803e-9fe18dca4f7f/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:09:34.052 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:09:34.052 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:09:34.052 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0232933 s, 45.0 MB/s 2021-02-10T11:09:34.053 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:09:34.053 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:09:34.053 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:09:34.389 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:09:34.389 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:09:34.390 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344391/6d29d995-3b4f-4f77-803e-9fe18dca4f7f/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:09:34.767 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:09:34.767 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:09:34.767 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0251187 s, 41.7 MB/s 2021-02-10T11:09:34.768 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:09:34.768 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:09:34.769 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:09:35.104 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:09:35.105 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:09:35.105 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344391/6d29d995-3b4f-4f77-803e-9fe18dca4f7f/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:09:35.480 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:09:35.480 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:09:35.480 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0236092 s, 44.4 MB/s 2021-02-10T11:09:35.481 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:09:35.481 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:09:35.481 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:09:35.817 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:09:35.818 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:09:35.818 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344391/6d29d995-3b4f-4f77-803e-9fe18dca4f7f/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:09:36.194 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:09:36.194 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:09:36.194 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.023806 s, 44.0 MB/s 2021-02-10T11:09:36.195 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:09:36.195 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:09:36.195 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:09:36.531 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:09:36.532 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:09:36.532 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344391/6d29d995-3b4f-4f77-803e-9fe18dca4f7f/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:09:36.909 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:09:36.910 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:09:36.910 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0256285 s, 40.9 MB/s 2021-02-10T11:09:36.911 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000344392 2021-02-10T11:09:37.341 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:37.329 7f7347cb0700 1 Processor -- start 2021-02-10T11:09:37.342 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:37.329 7f7347cb0700 1 -- start start 2021-02-10T11:09:37.342 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:37.330 7f7347cb0700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f73401b0870 0x7f73400f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:09:37.342 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:37.330 7f7347cb0700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f73400efe00 0x7f73400f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:09:37.342 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:37.330 7f7347cb0700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73401b0c70 con 0x7f73401b0470 2021-02-10T11:09:37.342 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:37.330 7f7347cb0700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73400e5ea0 con 0x7f73401b0870 2021-02-10T11:09:37.342 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:37.330 7f7347cb0700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73400f0b50 con 0x7f73400efe00 2021-02-10T11:09:37.342 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:37.330 7f7345acc700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f73401b0470 0x7f73400ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:43946/0 (socket says 172.21.2.102:43946) 2021-02-10T11:09:37.342 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:37.330 7f7345acc700 1 -- 172.21.2.102:0/2759474028 learned_addr learned my addr 172.21.2.102:0/2759474028 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:09:37.342 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:37.330 7f73452cb700 1 --2- 172.21.2.102:0/2759474028 >> v2:172.21.2.102:3300/0 conn(0x7f73401b0870 0x7f73400f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:09:37.342 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:37.330 7f73462cd700 1 --2- 172.21.2.102:0/2759474028 >> v2:172.21.2.113:3300/0 conn(0x7f73400efe00 0x7f73400f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:09:37.343 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:37.330 7f7344aca700 1 -- 172.21.2.102:0/2759474028 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3210513804 0 0) 0x7f7338001480 con 0x7f73401b0470 2021-02-10T11:09:37.343 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:37.330 7f7344aca700 1 -- 172.21.2.102:0/2759474028 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f732c002fe0 con 0x7f73401b0470 2021-02-10T11:09:37.343 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:37.330 7f73452cb700 1 -- 172.21.2.102:0/2759474028 >> v2:172.21.2.113:3300/0 conn(0x7f73400efe00 msgr2=0x7f73400f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:09:37.343 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:37.330 7f73452cb700 1 --2- 172.21.2.102:0/2759474028 >> v2:172.21.2.113:3300/0 conn(0x7f73400efe00 0x7f73400f0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:09:37.343 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:37.330 7f73452cb700 1 -- 172.21.2.102:0/2759474028 >> v1:172.21.2.102:6789/0 conn(0x7f73401b0470 legacy=0x7f73400ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:09:37.343 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:37.330 7f73452cb700 1 -- 172.21.2.102:0/2759474028 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73400f0820 con 0x7f73401b0870 2021-02-10T11:09:37.343 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:37.330 7f73462cd700 1 --2- 172.21.2.102:0/2759474028 >> v2:172.21.2.113:3300/0 conn(0x7f73400efe00 0x7f73400f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:09:37.343 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:37.330 7f73452cb700 1 --2- 172.21.2.102:0/2759474028 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f73401b0870 0x7f73400f20e0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 rx=0x7f7330006650 tx=0x7f7330009a80).ready entity=mon.0 client_cookie=c3414309d62e9604 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:09:37.343 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:37.331 7f7344aca700 1 -- 172.21.2.102:0/2759474028 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f73300009b0 con 0x7f73401b0870 2021-02-10T11:09:37.343 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:37.331 7f7344aca700 1 -- 172.21.2.102:0/2759474028 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f733000c0c0 con 0x7f73401b0870 2021-02-10T11:09:37.343 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:37.331 7f7344aca700 1 -- 172.21.2.102:0/2759474028 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f733000c5e0 con 0x7f73401b0870 2021-02-10T11:09:37.344 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:37.331 7f7347cb0700 1 -- 172.21.2.102:0/2759474028 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f73401b0870 msgr2=0x7f73400f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:09:37.344 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:37.331 7f7347cb0700 1 --2- 172.21.2.102:0/2759474028 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f73401b0870 0x7f73400f20e0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 rx=0x7f7330006650 tx=0x7f7330009a80).stop 2021-02-10T11:09:37.344 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:37.331 7f7347cb0700 1 -- 172.21.2.102:0/2759474028 shutdown_connections 2021-02-10T11:09:37.344 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:37.331 7f7347cb0700 1 --2- 172.21.2.102:0/2759474028 >> v2:172.21.2.113:3300/0 conn(0x7f73400efe00 0x7f73400f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:09:37.344 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:37.331 7f7347cb0700 1 --2- 172.21.2.102:0/2759474028 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f73401b0870 0x7f73400f20e0 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:09:37.344 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:37.331 7f7347cb0700 1 -- 172.21.2.102:0/2759474028 >> 172.21.2.102:0/2759474028 conn(0x7f73400ce3e0 msgr2=0x7f73400e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:09:37.344 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:37.332 7f7347cb0700 1 -- 172.21.2.102:0/2759474028 shutdown_connections 2021-02-10T11:09:37.344 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:37.332 7f7347cb0700 1 -- 172.21.2.102:0/2759474028 wait complete. 2021-02-10T11:09:37.346 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:37.346 7f7347cb0700 1 Processor -- start 2021-02-10T11:09:37.346 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:37.346 7f7347cb0700 1 -- start start 2021-02-10T11:09:37.346 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:37.346 7f7347cb0700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f73400a0420 0x7f73400a0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:09:37.346 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:37.346 7f7347cb0700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7340189d00 0x7f734018a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:09:37.347 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:37.346 7f7347cb0700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f734018a720 0x7f73400b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:09:37.347 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:37.346 7f7347cb0700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7340189800 con 0x7f73400a0420 2021-02-10T11:09:37.347 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:37.346 7f7347cb0700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f734018ab20 con 0x7f7340189d00 2021-02-10T11:09:37.347 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:37.346 7f7347cb0700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73400a1140 con 0x7f734018a720 2021-02-10T11:09:37.347 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:37.347 7f7345acc700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f73400a0420 0x7f73400a0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:09:37.347 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:37.347 7f7345acc700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f73400a0420 0x7f73400a0820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:51728/0 (socket says 172.21.2.102:51728) 2021-02-10T11:09:37.347 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:37.347 7f7345acc700 1 -- 172.21.2.102:0/3935899597 learned_addr learned my addr 172.21.2.102:0/3935899597 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:09:37.347 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:37.347 7f73452cb700 1 --2- 172.21.2.102:0/3935899597 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7340189d00 0x7f734018a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:09:37.347 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:37.347 7f73462cd700 1 --2- 172.21.2.102:0/3935899597 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f734018a720 0x7f73400b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:09:37.347 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:37.347 7f7345acc700 1 -- 172.21.2.102:0/3935899597 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f734018a720 msgr2=0x7f73400b7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:09:37.347 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:37.347 7f7345acc700 1 --2- 172.21.2.102:0/3935899597 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f734018a720 0x7f73400b7be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:09:37.348 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:37.347 7f7345acc700 1 -- 172.21.2.102:0/3935899597 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7340189d00 msgr2=0x7f734018a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:09:37.348 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:37.347 7f7345acc700 1 --2- 172.21.2.102:0/3935899597 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7340189d00 0x7f734018a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:09:37.348 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:37.347 7f7345acc700 1 -- 172.21.2.102:0/3935899597 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73400b8200 con 0x7f73400a0420 2021-02-10T11:09:37.348 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:37.347 7f7345acc700 1 --2- 172.21.2.102:0/3935899597 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f73400a0420 0x7f73400a0820 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 rx=0x7f7338009960 tx=0x7f733800bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:09:37.348 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:37.347 7f73367fc700 1 -- 172.21.2.102:0/3935899597 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f73380062b0 con 0x7f73400a0420 2021-02-10T11:09:37.348 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:37.347 7f73367fc700 1 -- 172.21.2.102:0/3935899597 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7338000ce0 con 0x7f73400a0420 2021-02-10T11:09:37.348 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:37.347 7f7347cb0700 1 -- 172.21.2.102:0/3935899597 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73400b8200 con 0x7f73400a0420 2021-02-10T11:09:37.348 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:37.347 7f73367fc700 1 -- 172.21.2.102:0/3935899597 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7338004910 con 0x7f73400a0420 2021-02-10T11:09:37.348 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:37.348 7f7347cb0700 1 -- 172.21.2.102:0/3935899597 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73400b8200 con 0x7f73400a0420 2021-02-10T11:09:37.348 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:37.348 7f73367fc700 1 -- 172.21.2.102:0/3935899597 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f7338004720 con 0x7f73400a0420 2021-02-10T11:09:37.348 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:37.348 7f73367fc700 1 --2- 172.21.2.102:0/3935899597 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7328018730 0x7f732801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:09:37.348 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:37.348 7f73367fc700 1 -- 172.21.2.102:0/3935899597 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f73380329f0 con 0x7f73400a0420 2021-02-10T11:09:37.349 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:37.348 7f73452cb700 1 --2- 172.21.2.102:0/3935899597 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7328018730 0x7f732801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:09:37.349 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:37.349 7f73452cb700 1 --2- 172.21.2.102:0/3935899597 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7328018730 0x7f732801ac10 crc :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:09:37.375 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:37.379 7f7347cb0700 1 -- 172.21.2.102:0/3935899597 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7340105ae0 con 0x7f73400a0420 2021-02-10T11:09:37.377 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:37.382 7f73367fc700 1 -- 172.21.2.102:0/3935899597 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f7338000cc0 con 0x7f73400a0420 2021-02-10T11:09:37.503 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:37.507 7f7347cb0700 1 -- 172.21.2.102:0/3935899597 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344392", "target": ["mgr", ""]}) v1 -- 0x7f734018ab20 con 0x7f7328018730 2021-02-10T11:09:37.529 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:37.532 7f73367fc700 1 -- 172.21.2.102:0/3935899597 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f7330000c90 con 0x7f7328018730 2021-02-10T11:09:37.536 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:37.540 7f7347cb0700 1 -- 172.21.2.102:0/3935899597 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7328018730 msgr2=0x7f732801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:09:37.536 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:37.540 7f7347cb0700 1 --2- 172.21.2.102:0/3935899597 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7328018730 0x7f732801ac10 crc :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:09:37.536 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:37.540 7f7347cb0700 1 -- 172.21.2.102:0/3935899597 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f73400a0420 msgr2=0x7f73400a0820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:09:37.536 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:37.540 7f7347cb0700 1 --2- 172.21.2.102:0/3935899597 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f73400a0420 0x7f73400a0820 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 rx=0x7f7338009960 tx=0x7f733800bcf0).stop 2021-02-10T11:09:37.536 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:37.540 7f7347cb0700 1 -- 172.21.2.102:0/3935899597 shutdown_connections 2021-02-10T11:09:37.536 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:37.540 7f7347cb0700 1 --2- 172.21.2.102:0/3935899597 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f734018a720 0x7f73400b7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:09:37.537 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:37.540 7f7347cb0700 1 --2- 172.21.2.102:0/3935899597 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7328018730 0x7f732801ac10 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:09:37.537 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:37.540 7f7347cb0700 1 --2- 172.21.2.102:0/3935899597 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7340189d00 0x7f734018a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:09:37.537 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:37.540 7f7347cb0700 1 --2- 172.21.2.102:0/3935899597 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f73400a0420 0x7f73400a0820 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:09:37.537 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:37.540 7f7347cb0700 1 -- 172.21.2.102:0/3935899597 >> 172.21.2.102:0/3935899597 conn(0x7f7340094e10 msgr2=0x7f73400a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:09:37.537 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:37.540 7f7347cb0700 1 -- 172.21.2.102:0/3935899597 shutdown_connections 2021-02-10T11:09:37.537 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:37.540 7f7347cb0700 1 -- 172.21.2.102:0/3935899597 wait complete. 2021-02-10T11:09:37.544 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000344392 2021-02-10T11:09:37.975 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:37.962 7f1437dd8700 1 Processor -- start 2021-02-10T11:09:37.975 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:37.962 7f1437dd8700 1 -- start start 2021-02-10T11:09:37.975 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:37.963 7f1437dd8700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f14301b0870 0x7f14300f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:09:37.975 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:37.963 7f1437dd8700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14301b0c70 con 0x7f14300efe00 2021-02-10T11:09:37.976 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:37.963 7f1437dd8700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14300f0b50 con 0x7f14301b0470 2021-02-10T11:09:37.976 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:37.963 7f1437dd8700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14300e5ea0 con 0x7f14301b0870 2021-02-10T11:09:37.976 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:37.963 7f14363f5700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f14300efe00 0x7f14300f0200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:43964/0 (socket says 172.21.2.102:43964) 2021-02-10T11:09:37.976 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:37.963 7f14363f5700 1 -- 172.21.2.102:0/3778579143 learned_addr learned my addr 172.21.2.102:0/3778579143 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:09:37.976 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:37.963 7f1434bf2700 1 -- 172.21.2.102:0/3778579143 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4093311124 0 0) 0x7f1428001500 con 0x7f14300efe00 2021-02-10T11:09:37.976 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:37.963 7f1434bf2700 1 -- 172.21.2.102:0/3778579143 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f141c002fe0 con 0x7f14300efe00 2021-02-10T11:09:37.976 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:37.964 7f14353f3700 1 --2- 172.21.2.102:0/3778579143 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f14301b0870 0x7f14300f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:09:37.976 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:37.964 7f1434bf2700 1 -- 172.21.2.102:0/3778579143 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3831946856 0 0) 0x7f1428001500 con 0x7f14300efe00 2021-02-10T11:09:37.976 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:37.964 7f1434bf2700 1 -- 172.21.2.102:0/3778579143 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f14301b0870 msgr2=0x7f14300f20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:09:37.976 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:37.964 7f1434bf2700 1 --2- 172.21.2.102:0/3778579143 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f14301b0870 0x7f14300f20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:09:37.976 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:37.964 7f1434bf2700 1 -- 172.21.2.102:0/3778579143 >> v1:172.21.2.113:6789/0 conn(0x7f14301b0470 legacy=0x7f14300ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:09:37.976 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:37.964 7f1434bf2700 1 -- 172.21.2.102:0/3778579143 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14300e52e0 con 0x7f14300efe00 2021-02-10T11:09:37.976 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:37.964 7f1434bf2700 1 -- 172.21.2.102:0/3778579143 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7f1428001a00 con 0x7f14300efe00 2021-02-10T11:09:37.977 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:37.964 7f1434bf2700 1 -- 172.21.2.102:0/3778579143 >> v1:172.21.2.102:6789/0 conn(0x7f14300efe00 legacy=0x7f14300f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:09:37.977 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:37.964 7f1434bf2700 1 --2- 172.21.2.102:0/3778579143 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f141c003a60 0x7f141c005ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:09:37.977 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:37.964 7f1434bf2700 1 --2- 172.21.2.102:0/3778579143 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f141c006790 0x7f141c008c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:09:37.977 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:37.964 7f1434bf2700 1 --2- 172.21.2.102:0/3778579143 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f141c009590 0x7f141c00ba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:09:37.977 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:37.964 7f1434bf2700 1 -- 172.21.2.102:0/3778579143 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f141c00c2d0 con 0x7f141c006790 2021-02-10T11:09:37.977 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:37.964 7f1434bf2700 1 -- 172.21.2.102:0/3778579143 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f141c00c4f0 con 0x7f141c003a60 2021-02-10T11:09:37.977 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:37.964 7f1434bf2700 1 -- 172.21.2.102:0/3778579143 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f141c00c710 con 0x7f141c009590 2021-02-10T11:09:37.977 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:37.964 7f14353f3700 1 --2- 172.21.2.102:0/3778579143 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f141c006790 0x7f141c008c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:09:37.977 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:37.964 7f1435bf4700 1 --2- 172.21.2.102:0/3778579143 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f141c003a60 0x7f141c005ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:09:37.977 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:37.964 7f14363f5700 1 --2- 172.21.2.102:0/3778579143 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f141c009590 0x7f141c00ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:09:37.977 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:37.964 7f14353f3700 1 -- 172.21.2.102:0/3778579143 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f141c009590 msgr2=0x7f141c00ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:09:37.977 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:37.964 7f14353f3700 1 --2- 172.21.2.102:0/3778579143 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f141c009590 0x7f141c00ba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:09:37.977 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:37.964 7f14353f3700 1 -- 172.21.2.102:0/3778579143 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f141c003a60 msgr2=0x7f141c005ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:09:37.978 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:37.964 7f14353f3700 1 --2- 172.21.2.102:0/3778579143 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f141c003a60 0x7f141c005ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:09:37.978 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:37.964 7f14353f3700 1 -- 172.21.2.102:0/3778579143 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f141c050980 con 0x7f141c006790 2021-02-10T11:09:37.978 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:37.965 7f14353f3700 1 --2- 172.21.2.102:0/3778579143 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f141c006790 0x7f141c008c90 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 rx=0x7f142c009310 tx=0x7f142c009a90).ready entity=mon.0 client_cookie=94ff794fccef6581 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:09:37.978 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:37.965 7f1434bf2700 1 -- 172.21.2.102:0/3778579143 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f142c00c0c0 con 0x7f141c006790 2021-02-10T11:09:37.978 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:37.965 7f1434bf2700 1 -- 172.21.2.102:0/3778579143 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f142c006400 con 0x7f141c006790 2021-02-10T11:09:37.978 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:37.965 7f1434bf2700 1 -- 172.21.2.102:0/3778579143 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f142c00c340 con 0x7f141c006790 2021-02-10T11:09:37.978 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:37.965 7f1437dd8700 1 -- 172.21.2.102:0/3778579143 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f141c006790 msgr2=0x7f141c008c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:09:37.978 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:37.965 7f1437dd8700 1 --2- 172.21.2.102:0/3778579143 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f141c006790 0x7f141c008c90 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 rx=0x7f142c009310 tx=0x7f142c009a90).stop 2021-02-10T11:09:37.978 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:37.965 7f14363f5700 1 -- 172.21.2.102:0/3778579143 reap_dead start 2021-02-10T11:09:37.978 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:37.965 7f1437dd8700 1 -- 172.21.2.102:0/3778579143 shutdown_connections 2021-02-10T11:09:37.978 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:37.965 7f1437dd8700 1 -- 172.21.2.102:0/3778579143 >> 172.21.2.102:0/3778579143 conn(0x7f14300ce3e0 msgr2=0x7f14300e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:09:37.978 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:37.965 7f1437dd8700 1 -- 172.21.2.102:0/3778579143 shutdown_connections 2021-02-10T11:09:37.979 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:37.965 7f1437dd8700 1 -- 172.21.2.102:0/3778579143 wait complete. 2021-02-10T11:09:37.980 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:37.980 7f1437dd8700 1 Processor -- start 2021-02-10T11:09:37.980 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:37.980 7f1437dd8700 1 -- start start 2021-02-10T11:09:37.980 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:37.980 7f1437dd8700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f143009b930 0x7f14300b7c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:09:37.980 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:37.980 7f1437dd8700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f14300b8410 0x7f14300b8810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:09:37.980 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:37.980 7f1437dd8700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f14300c1f80 0x7f14300c2380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:09:37.980 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:37.980 7f1437dd8700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14300c1a00 con 0x7f14300c1f80 2021-02-10T11:09:37.980 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:37.980 7f1437dd8700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14300b8e30 con 0x7f143009b930 2021-02-10T11:09:37.980 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:37.980 7f1437dd8700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14300b9050 con 0x7f14300b8410 2021-02-10T11:09:37.980 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:37.980 7f14363f5700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f14300c1f80 0x7f14300c2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:09:37.980 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:37.980 7f14363f5700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f14300c1f80 0x7f14300c2380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:51750/0 (socket says 172.21.2.102:51750) 2021-02-10T11:09:37.981 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:37.980 7f14363f5700 1 -- 172.21.2.102:0/2593028566 learned_addr learned my addr 172.21.2.102:0/2593028566 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:09:37.981 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:37.981 7f14363f5700 1 -- 172.21.2.102:0/2593028566 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f14300b8410 msgr2=0x7f14300b8810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:09:37.981 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:37.981 7f14363f5700 1 --2- 172.21.2.102:0/2593028566 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f14300b8410 0x7f14300b8810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:09:37.981 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:37.981 7f14363f5700 1 -- 172.21.2.102:0/2593028566 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f143009b930 msgr2=0x7f14300b7c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:09:37.981 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:37.981 7f14363f5700 1 --2- 172.21.2.102:0/2593028566 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f143009b930 0x7f14300b7c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:09:37.981 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:37.981 7f14363f5700 1 -- 172.21.2.102:0/2593028566 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14300c29a0 con 0x7f14300c1f80 2021-02-10T11:09:37.981 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:37.981 7f14363f5700 1 --2- 172.21.2.102:0/2593028566 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f14300c1f80 0x7f14300c2380 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 rx=0x7f1428009920 tx=0x7f1428009cc0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:09:37.981 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:37.981 7f14227fc700 1 -- 172.21.2.102:0/2593028566 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f14280079d0 con 0x7f14300c1f80 2021-02-10T11:09:37.981 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:37.981 7f14227fc700 1 -- 172.21.2.102:0/2593028566 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14280070d0 con 0x7f14300c1f80 2021-02-10T11:09:37.981 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:37.981 7f14227fc700 1 -- 172.21.2.102:0/2593028566 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1428004720 con 0x7f14300c1f80 2021-02-10T11:09:37.981 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:37.981 7f1437dd8700 1 -- 172.21.2.102:0/2593028566 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14300c29a0 con 0x7f14300c1f80 2021-02-10T11:09:37.981 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:37.981 7f1437dd8700 1 -- 172.21.2.102:0/2593028566 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14300c29a0 con 0x7f14300c1f80 2021-02-10T11:09:37.981 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:37.982 7f14227fc700 1 -- 172.21.2.102:0/2593028566 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f14280072b0 con 0x7f14300c1f80 2021-02-10T11:09:37.982 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:37.982 7f14227fc700 1 --2- 172.21.2.102:0/2593028566 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f14180186c0 0x7f141801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:09:37.982 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:37.982 7f14227fc700 1 -- 172.21.2.102:0/2593028566 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f14280328a0 con 0x7f14300c1f80 2021-02-10T11:09:37.982 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:37.982 7f1435bf4700 1 --2- 172.21.2.102:0/2593028566 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f14180186c0 0x7f141801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:09:37.982 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:37.982 7f1435bf4700 1 --2- 172.21.2.102:0/2593028566 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f14180186c0 0x7f141801abc0 crc :-1 s=READY pgs=229 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:09:38.008 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:38.013 7f1437dd8700 1 -- 172.21.2.102:0/2593028566 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1430105b30 con 0x7f14300c1f80 2021-02-10T11:09:38.011 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:38.015 7f14227fc700 1 -- 172.21.2.102:0/2593028566 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f14280079d0 con 0x7f14300c1f80 2021-02-10T11:09:38.141 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:38.145 7f1437dd8700 1 -- 172.21.2.102:0/2593028566 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344392", "target": ["mgr", ""]}) v1 -- 0x7f1430105b30 con 0x7f14180186c0 2021-02-10T11:09:38.143 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:38.148 7f14227fc700 1 -- 172.21.2.102:0/2593028566 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f1424000960 con 0x7f14180186c0 2021-02-10T11:09:38.144 INFO:teuthology.orchestra.run.gibba002.stdout:/volumes/_nogroup/subvolume_0000000000344392/15447767-59a3-4ceb-8432-b35af705a5e3 2021-02-10T11:09:38.145 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:38.149 7f1437dd8700 1 -- 172.21.2.102:0/2593028566 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f14180186c0 msgr2=0x7f141801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:09:38.145 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:38.149 7f1437dd8700 1 --2- 172.21.2.102:0/2593028566 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f14180186c0 0x7f141801abc0 crc :-1 s=READY pgs=229 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:09:38.146 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:38.149 7f1437dd8700 1 -- 172.21.2.102:0/2593028566 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f14300c1f80 msgr2=0x7f14300c2380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:09:38.146 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:38.149 7f1437dd8700 1 --2- 172.21.2.102:0/2593028566 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f14300c1f80 0x7f14300c2380 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 rx=0x7f1428009920 tx=0x7f1428009cc0).stop 2021-02-10T11:09:38.146 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:38.149 7f1437dd8700 1 -- 172.21.2.102:0/2593028566 shutdown_connections 2021-02-10T11:09:38.146 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:38.149 7f1437dd8700 1 --2- 172.21.2.102:0/2593028566 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f14300c1f80 0x7f14300c2380 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:09:38.146 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:38.149 7f1437dd8700 1 --2- 172.21.2.102:0/2593028566 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f14300b8410 0x7f14300b8810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:09:38.146 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:38.149 7f1437dd8700 1 --2- 172.21.2.102:0/2593028566 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f14180186c0 0x7f141801abc0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:09:38.146 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:38.149 7f1437dd8700 1 --2- 172.21.2.102:0/2593028566 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f143009b930 0x7f14300b7c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:09:38.146 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:38.149 7f1437dd8700 1 -- 172.21.2.102:0/2593028566 >> 172.21.2.102:0/2593028566 conn(0x7f1430094e10 msgr2=0x7f14300a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:09:38.146 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:38.149 7f1437dd8700 1 -- 172.21.2.102:0/2593028566 shutdown_connections 2021-02-10T11:09:38.146 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:38.149 7f1437dd8700 1 -- 172.21.2.102:0/2593028566 wait complete. 2021-02-10T11:09:38.153 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000344392 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000344392/15447767-59a3-4ceb-8432-b35af705a5e3 2021-02-10T11:09:38.154 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:09:38.154 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:09:38.154 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:09:38.489 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:09:38.490 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:09:38.490 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344392/15447767-59a3-4ceb-8432-b35af705a5e3/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:09:38.869 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:09:38.869 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:09:38.869 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0231797 s, 45.2 MB/s 2021-02-10T11:09:38.870 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:09:38.870 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:09:38.870 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:09:39.205 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:09:39.206 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:09:39.206 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344392/15447767-59a3-4ceb-8432-b35af705a5e3/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:09:39.580 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:09:39.580 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:09:39.580 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0232952 s, 45.0 MB/s 2021-02-10T11:09:39.581 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:09:39.581 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:09:39.582 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:09:39.917 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:09:39.917 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:09:39.918 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344392/15447767-59a3-4ceb-8432-b35af705a5e3/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:09:40.294 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:09:40.294 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:09:40.294 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0236903 s, 44.3 MB/s 2021-02-10T11:09:40.295 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:09:40.295 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:09:40.295 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:09:40.630 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:09:40.631 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:09:40.631 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344392/15447767-59a3-4ceb-8432-b35af705a5e3/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:09:41.008 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:09:41.008 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:09:41.008 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0246609 s, 42.5 MB/s 2021-02-10T11:09:41.009 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:09:41.009 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:09:41.009 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:09:41.345 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:09:41.346 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:09:41.346 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344392/15447767-59a3-4ceb-8432-b35af705a5e3/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:09:41.724 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:09:41.724 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:09:41.724 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0249959 s, 41.9 MB/s 2021-02-10T11:09:41.726 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:09:41.726 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:09:41.726 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:09:42.062 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:09:42.063 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:09:42.063 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344392/15447767-59a3-4ceb-8432-b35af705a5e3/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:09:42.440 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:09:42.440 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:09:42.440 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0254377 s, 41.2 MB/s 2021-02-10T11:09:42.441 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:09:42.441 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:09:42.441 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:09:42.777 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:09:42.778 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:09:42.778 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344392/15447767-59a3-4ceb-8432-b35af705a5e3/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:09:43.153 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:09:43.154 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:09:43.154 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0243604 s, 43.0 MB/s 2021-02-10T11:09:43.155 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:09:43.155 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:09:43.155 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:09:43.490 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:09:43.491 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:09:43.491 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344392/15447767-59a3-4ceb-8432-b35af705a5e3/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:09:43.862 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:09:43.862 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:09:43.863 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0235729 s, 44.5 MB/s 2021-02-10T11:09:43.863 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:09:43.864 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:09:43.864 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:09:44.198 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:09:44.199 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:09:44.199 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344392/15447767-59a3-4ceb-8432-b35af705a5e3/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:09:44.573 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:09:44.573 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:09:44.574 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0235892 s, 44.5 MB/s 2021-02-10T11:09:44.574 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:09:44.575 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:09:44.575 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:09:44.910 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:09:44.911 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:09:44.911 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344392/15447767-59a3-4ceb-8432-b35af705a5e3/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:09:45.286 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:09:45.286 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:09:45.286 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.022913 s, 45.8 MB/s 2021-02-10T11:09:45.287 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000344393 2021-02-10T11:09:45.367 DEBUG:teuthology.orchestra.run.gibba002:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:09:45.371 DEBUG:teuthology.orchestra.run.gibba013:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:09:45.719 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:45.706 7f1f17398700 1 Processor -- start 2021-02-10T11:09:45.719 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:45.706 7f1f17398700 1 -- start start 2021-02-10T11:09:45.720 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:45.707 7f1f17398700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f1f101b0470 0x7f1f101b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:09:45.720 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:45.707 7f1f17398700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f1f100f20e0 0x7f1f100f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:09:45.720 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:45.707 7f1f17398700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f100f0b50 con 0x7f1f100ef610 2021-02-10T11:09:45.720 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:45.707 7f1f17398700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f100f0820 con 0x7f1f100f20e0 2021-02-10T11:09:45.720 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:45.707 7f1f17398700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f100f24e0 con 0x7f1f101b0470 2021-02-10T11:09:45.720 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:45.707 7f1f159b5700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f1f100f20e0 0x7f1f100f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:09:45.720 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:45.707 7f1f159b5700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f1f100f20e0 0x7f1f100f0200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:51756/0 (socket says 172.21.2.102:51756) 2021-02-10T11:09:45.720 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:45.707 7f1f159b5700 1 -- 172.21.2.102:0/3057887981 learned_addr learned my addr 172.21.2.102:0/3057887981 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:09:45.720 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:45.707 7f1f151b4700 1 --2- 172.21.2.102:0/3057887981 >> v2:172.21.2.113:3300/0 conn(0x7f1f101b0470 0x7f1f101b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:09:45.720 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:45.707 7f1f159b5700 1 -- 172.21.2.102:0/3057887981 >> v2:172.21.2.113:3300/0 conn(0x7f1f101b0470 msgr2=0x7f1f101b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:09:45.720 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:45.707 7f1f159b5700 1 --2- 172.21.2.102:0/3057887981 >> v2:172.21.2.113:3300/0 conn(0x7f1f101b0470 0x7f1f101b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:09:45.720 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:45.707 7f1f159b5700 1 -- 172.21.2.102:0/3057887981 >> v1:172.21.2.113:6789/0 conn(0x7f1f100ef610 legacy=0x7f1f100efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:09:45.720 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:45.707 7f1f159b5700 1 -- 172.21.2.102:0/3057887981 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f100e52e0 con 0x7f1f100f20e0 2021-02-10T11:09:45.721 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:45.707 7f1f159b5700 1 --2- 172.21.2.102:0/3057887981 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1f100f20e0 0x7f1f100f0200 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 rx=0x7f1f08006f00 tx=0x7f1f08009e90).ready entity=mon.0 client_cookie=4468fca07fa074aa server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:09:45.721 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:45.707 7f1f03fff700 1 -- 172.21.2.102:0/3057887981 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1f0800b300 con 0x7f1f100f20e0 2021-02-10T11:09:45.721 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:45.708 7f1f03fff700 1 -- 172.21.2.102:0/3057887981 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f0800b580 con 0x7f1f100f20e0 2021-02-10T11:09:45.721 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:45.708 7f1f03fff700 1 -- 172.21.2.102:0/3057887981 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1f0800b9d0 con 0x7f1f100f20e0 2021-02-10T11:09:45.721 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:45.708 7f1f17398700 1 -- 172.21.2.102:0/3057887981 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1f100f20e0 msgr2=0x7f1f100f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:09:45.721 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:45.708 7f1f17398700 1 --2- 172.21.2.102:0/3057887981 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1f100f20e0 0x7f1f100f0200 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 rx=0x7f1f08006f00 tx=0x7f1f08009e90).stop 2021-02-10T11:09:45.721 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:45.708 7f1f17398700 1 -- 172.21.2.102:0/3057887981 shutdown_connections 2021-02-10T11:09:45.721 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:45.708 7f1f17398700 1 --2- 172.21.2.102:0/3057887981 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1f100f20e0 0x7f1f100f0200 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:09:45.721 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:45.708 7f1f17398700 1 --2- 172.21.2.102:0/3057887981 >> v2:172.21.2.113:3300/0 conn(0x7f1f101b0470 0x7f1f101b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:09:45.721 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:45.708 7f1f17398700 1 -- 172.21.2.102:0/3057887981 >> 172.21.2.102:0/3057887981 conn(0x7f1f100ce3e0 msgr2=0x7f1f100e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:09:45.721 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:45.708 7f1f17398700 1 -- 172.21.2.102:0/3057887981 shutdown_connections 2021-02-10T11:09:45.721 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:45.708 7f1f17398700 1 -- 172.21.2.102:0/3057887981 wait complete. 2021-02-10T11:09:45.722 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:45.723 7f1f17398700 1 Processor -- start 2021-02-10T11:09:45.722 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:45.723 7f1f17398700 1 -- start start 2021-02-10T11:09:45.724 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:45.723 7f1f17398700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1f100a0420 0x7f1f100a0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:09:45.724 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:45.723 7f1f17398700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1f10189d00 0x7f1f1018a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:09:45.724 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:45.723 7f1f17398700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1f1018a720 0x7f1f100b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:09:45.724 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:45.723 7f1f17398700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f10189780 con 0x7f1f100a0420 2021-02-10T11:09:45.724 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:45.723 7f1f17398700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f1018ab20 con 0x7f1f1018a720 2021-02-10T11:09:45.724 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:45.723 7f1f17398700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f100a1140 con 0x7f1f10189d00 2021-02-10T11:09:45.725 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:45.723 7f1f151b4700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1f100a0420 0x7f1f100a0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:09:45.725 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:45.723 7f1f151b4700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1f100a0420 0x7f1f100a0820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:51762/0 (socket says 172.21.2.102:51762) 2021-02-10T11:09:45.725 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:45.723 7f1f151b4700 1 -- 172.21.2.102:0/3834857784 learned_addr learned my addr 172.21.2.102:0/3834857784 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:09:45.725 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:45.724 7f1f159b5700 1 --2- 172.21.2.102:0/3834857784 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1f1018a720 0x7f1f100b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:09:45.725 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:45.724 7f1f149b3700 1 --2- 172.21.2.102:0/3834857784 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1f10189d00 0x7f1f1018a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:09:45.725 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:45.724 7f1f151b4700 1 -- 172.21.2.102:0/3834857784 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1f10189d00 msgr2=0x7f1f1018a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:09:45.725 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:45.724 7f1f151b4700 1 --2- 172.21.2.102:0/3834857784 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1f10189d00 0x7f1f1018a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:09:45.725 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:45.724 7f1f151b4700 1 -- 172.21.2.102:0/3834857784 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1f1018a720 msgr2=0x7f1f100b7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:09:45.725 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:45.724 7f1f151b4700 1 --2- 172.21.2.102:0/3834857784 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1f1018a720 0x7f1f100b7be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:09:45.725 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:45.724 7f1f151b4700 1 -- 172.21.2.102:0/3834857784 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f100b8200 con 0x7f1f100a0420 2021-02-10T11:09:45.725 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:45.724 7f1f151b4700 1 --2- 172.21.2.102:0/3834857784 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1f100a0420 0x7f1f100a0820 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 rx=0x7f1f04006f00 tx=0x7f1f0400b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:09:45.725 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:45.724 7f1f01ffb700 1 -- 172.21.2.102:0/3834857784 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1f0400bd30 con 0x7f1f100a0420 2021-02-10T11:09:45.725 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:45.724 7f1f01ffb700 1 -- 172.21.2.102:0/3834857784 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f04002030 con 0x7f1f100a0420 2021-02-10T11:09:45.725 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:45.724 7f1f01ffb700 1 -- 172.21.2.102:0/3834857784 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1f04002480 con 0x7f1f100a0420 2021-02-10T11:09:45.726 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:45.724 7f1f17398700 1 -- 172.21.2.102:0/3834857784 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f100b8200 con 0x7f1f100a0420 2021-02-10T11:09:45.726 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:45.724 7f1f17398700 1 -- 172.21.2.102:0/3834857784 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f100b8790 con 0x7f1f100a0420 2021-02-10T11:09:45.726 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:45.725 7f1f01ffb700 1 -- 172.21.2.102:0/3834857784 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1f04000c90 con 0x7f1f100a0420 2021-02-10T11:09:45.726 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:45.725 7f1f01ffb700 1 --2- 172.21.2.102:0/3834857784 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1ef00186c0 0x7f1ef001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:09:45.726 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:45.725 7f1f01ffb700 1 -- 172.21.2.102:0/3834857784 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f1f040327e0 con 0x7f1f100a0420 2021-02-10T11:09:45.726 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:45.725 7f1f149b3700 1 --2- 172.21.2.102:0/3834857784 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1ef00186c0 0x7f1ef001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:09:45.726 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:45.725 7f1f149b3700 1 --2- 172.21.2.102:0/3834857784 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1ef00186c0 0x7f1ef001abc0 crc :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:09:45.753 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:45.756 7f1f17398700 1 -- 172.21.2.102:0/3834857784 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f10105ae0 con 0x7f1f100a0420 2021-02-10T11:09:45.755 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:45.759 7f1f01ffb700 1 -- 172.21.2.102:0/3834857784 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f1f040009b0 con 0x7f1f100a0420 2021-02-10T11:09:45.883 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:45.886 7f1f17398700 1 -- 172.21.2.102:0/3834857784 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344393", "target": ["mgr", ""]}) v1 -- 0x7f1f100b8790 con 0x7f1ef00186c0 2021-02-10T11:09:45.907 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:45.910 7f1f01ffb700 1 -- 172.21.2.102:0/3834857784 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f1f0c0009b0 con 0x7f1ef00186c0 2021-02-10T11:09:45.916 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:45.919 7f1f17398700 1 -- 172.21.2.102:0/3834857784 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1ef00186c0 msgr2=0x7f1ef001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:09:45.916 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:45.919 7f1f17398700 1 --2- 172.21.2.102:0/3834857784 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1ef00186c0 0x7f1ef001abc0 crc :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:09:45.917 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:45.919 7f1f17398700 1 -- 172.21.2.102:0/3834857784 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1f100a0420 msgr2=0x7f1f100a0820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:09:45.917 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:45.919 7f1f17398700 1 --2- 172.21.2.102:0/3834857784 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1f100a0420 0x7f1f100a0820 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 rx=0x7f1f04006f00 tx=0x7f1f0400b940).stop 2021-02-10T11:09:45.917 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:45.920 7f1f17398700 1 -- 172.21.2.102:0/3834857784 shutdown_connections 2021-02-10T11:09:45.917 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:45.920 7f1f17398700 1 --2- 172.21.2.102:0/3834857784 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1ef00186c0 0x7f1ef001abc0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:09:45.917 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:45.920 7f1f17398700 1 --2- 172.21.2.102:0/3834857784 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1f1018a720 0x7f1f100b7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:09:45.917 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:45.920 7f1f17398700 1 --2- 172.21.2.102:0/3834857784 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1f10189d00 0x7f1f1018a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:09:45.917 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:45.920 7f1f17398700 1 --2- 172.21.2.102:0/3834857784 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1f100a0420 0x7f1f100a0820 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:09:45.917 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:45.920 7f1f17398700 1 -- 172.21.2.102:0/3834857784 >> 172.21.2.102:0/3834857784 conn(0x7f1f10094e10 msgr2=0x7f1f100a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:09:45.917 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:45.920 7f1f17398700 1 -- 172.21.2.102:0/3834857784 shutdown_connections 2021-02-10T11:09:45.917 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:45.920 7f1f17398700 1 -- 172.21.2.102:0/3834857784 wait complete. 2021-02-10T11:09:45.924 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000344393 2021-02-10T11:09:46.354 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:46.341 7f3aed75c700 1 Processor -- start 2021-02-10T11:09:46.355 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:46.341 7f3aed75c700 1 -- start start 2021-02-10T11:09:46.355 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:46.342 7f3aed75c700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f3ae81b0470 0x7f3ae81b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:09:46.355 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:46.342 7f3aed75c700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f3ae80f20e0 0x7f3ae80f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:09:46.355 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:46.342 7f3aed75c700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ae80f0b50 con 0x7f3ae80ef610 2021-02-10T11:09:46.355 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:46.342 7f3aed75c700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ae80f0820 con 0x7f3ae80f20e0 2021-02-10T11:09:46.355 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:46.342 7f3aed75c700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ae80f24e0 con 0x7f3ae81b0470 2021-02-10T11:09:46.355 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:46.342 7f3ae7d75700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f3ae80f20e0 0x7f3ae80f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:09:46.355 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:46.342 7f3ae6d73700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f3ae80ef610 0x7f3ae80efa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:43992/0 (socket says 172.21.2.102:43992) 2021-02-10T11:09:46.355 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:46.342 7f3ae6d73700 1 -- 172.21.2.102:0/4001084790 learned_addr learned my addr 172.21.2.102:0/4001084790 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:09:46.355 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:46.342 7f3ae7d75700 1 -- 172.21.2.102:0/4001084790 >> v2:172.21.2.113:3300/0 conn(0x7f3ae81b0470 msgr2=0x7f3ae81b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:09:46.355 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:46.342 7f3ae7d75700 1 --2- 172.21.2.102:0/4001084790 >> v2:172.21.2.113:3300/0 conn(0x7f3ae81b0470 0x7f3ae81b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:09:46.356 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:46.342 7f3ae7d75700 1 -- 172.21.2.102:0/4001084790 >> v1:172.21.2.102:6789/0 conn(0x7f3ae80ef610 legacy=0x7f3ae80efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:09:46.356 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:46.342 7f3ae7d75700 1 -- 172.21.2.102:0/4001084790 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ae80e52e0 con 0x7f3ae80f20e0 2021-02-10T11:09:46.356 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:46.343 7f3ae7d75700 1 --2- 172.21.2.102:0/4001084790 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3ae80f20e0 0x7f3ae80f0200 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 rx=0x7f3ae0006e30 tx=0x7f3ae0009e60).ready entity=mon.0 client_cookie=90fccef8c6ff9f3e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:09:46.356 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:46.343 7f3ae6572700 1 -- 172.21.2.102:0/4001084790 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3ae0006a50 con 0x7f3ae80f20e0 2021-02-10T11:09:46.356 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:46.343 7f3ae6572700 1 -- 172.21.2.102:0/4001084790 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ae000b300 con 0x7f3ae80f20e0 2021-02-10T11:09:46.356 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:46.343 7f3ae6572700 1 -- 172.21.2.102:0/4001084790 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3ae000b7e0 con 0x7f3ae80f20e0 2021-02-10T11:09:46.356 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:46.343 7f3aed75c700 1 -- 172.21.2.102:0/4001084790 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3ae80f20e0 msgr2=0x7f3ae80f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:09:46.356 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:46.343 7f3aed75c700 1 --2- 172.21.2.102:0/4001084790 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3ae80f20e0 0x7f3ae80f0200 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 rx=0x7f3ae0006e30 tx=0x7f3ae0009e60).stop 2021-02-10T11:09:46.356 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:46.343 7f3aed75c700 1 -- 172.21.2.102:0/4001084790 shutdown_connections 2021-02-10T11:09:46.356 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:46.343 7f3aed75c700 1 --2- 172.21.2.102:0/4001084790 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3ae80f20e0 0x7f3ae80f0200 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:09:46.356 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:46.343 7f3aed75c700 1 --2- 172.21.2.102:0/4001084790 >> v2:172.21.2.113:3300/0 conn(0x7f3ae81b0470 0x7f3ae81b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:09:46.356 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:46.343 7f3aed75c700 1 -- 172.21.2.102:0/4001084790 >> 172.21.2.102:0/4001084790 conn(0x7f3ae80ce3e0 msgr2=0x7f3ae80e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:09:46.356 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:46.343 7f3aed75c700 1 -- 172.21.2.102:0/4001084790 shutdown_connections 2021-02-10T11:09:46.357 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:46.343 7f3aed75c700 1 -- 172.21.2.102:0/4001084790 wait complete. 2021-02-10T11:09:46.357 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:46.358 7f3aed75c700 1 Processor -- start 2021-02-10T11:09:46.357 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:46.358 7f3aed75c700 1 -- start start 2021-02-10T11:09:46.358 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:46.358 7f3aed75c700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3ae809cee0 0x7f3ae819bb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:09:46.358 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:46.358 7f3aed75c700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3ae819c120 0x7f3ae80be040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:09:46.358 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:46.359 7f3aed75c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3ae80be660 0x7f3ae80bcec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:09:46.358 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:46.359 7f3aed75c700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ae809d2e0 con 0x7f3ae80be660 2021-02-10T11:09:46.359 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:46.359 7f3aed75c700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ae80a9610 con 0x7f3ae819c120 2021-02-10T11:09:46.359 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:46.359 7f3aed75c700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ae80bd4e0 con 0x7f3ae809cee0 2021-02-10T11:09:46.359 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:46.359 7f3ae7d75700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3ae80be660 0x7f3ae80bcec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:09:46.359 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:46.359 7f3ae7d75700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3ae80be660 0x7f3ae80bcec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:51778/0 (socket says 172.21.2.102:51778) 2021-02-10T11:09:46.359 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:46.359 7f3ae7d75700 1 -- 172.21.2.102:0/1740176856 learned_addr learned my addr 172.21.2.102:0/1740176856 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:09:46.359 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:46.359 7f3ae7d75700 1 -- 172.21.2.102:0/1740176856 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3ae809cee0 msgr2=0x7f3ae819bb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:09:46.359 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:46.359 7f3ae7d75700 1 --2- 172.21.2.102:0/1740176856 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3ae809cee0 0x7f3ae819bb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:09:46.359 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:46.359 7f3ae7d75700 1 -- 172.21.2.102:0/1740176856 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3ae819c120 msgr2=0x7f3ae80be040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:09:46.359 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:46.359 7f3ae7d75700 1 --2- 172.21.2.102:0/1740176856 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3ae819c120 0x7f3ae80be040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:09:46.359 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:46.359 7f3ae7d75700 1 -- 172.21.2.102:0/1740176856 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ae80bd700 con 0x7f3ae80be660 2021-02-10T11:09:46.359 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:46.359 7f3ae7d75700 1 --2- 172.21.2.102:0/1740176856 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3ae80be660 0x7f3ae80bcec0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 rx=0x7f3ae000b970 tx=0x7f3ae000bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:09:46.359 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:46.359 7f3adffff700 1 -- 172.21.2.102:0/1740176856 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3ae00093d0 con 0x7f3ae80be660 2021-02-10T11:09:46.359 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:46.360 7f3adffff700 1 -- 172.21.2.102:0/1740176856 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ae0000f60 con 0x7f3ae80be660 2021-02-10T11:09:46.359 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:46.360 7f3aed75c700 1 -- 172.21.2.102:0/1740176856 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ae80bd700 con 0x7f3ae80be660 2021-02-10T11:09:46.360 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:46.360 7f3adffff700 1 -- 172.21.2.102:0/1740176856 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3ae0005aa0 con 0x7f3ae80be660 2021-02-10T11:09:46.360 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:46.360 7f3aed75c700 1 -- 172.21.2.102:0/1740176856 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ae80bd700 con 0x7f3ae80be660 2021-02-10T11:09:46.360 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:46.360 7f3adffff700 1 -- 172.21.2.102:0/1740176856 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f3ae0005c80 con 0x7f3ae80be660 2021-02-10T11:09:46.360 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:46.360 7f3adffff700 1 --2- 172.21.2.102:0/1740176856 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3acc0186c0 0x7f3acc01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:09:46.360 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:46.360 7f3adffff700 1 -- 172.21.2.102:0/1740176856 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f3ae00327b0 con 0x7f3ae80be660 2021-02-10T11:09:46.360 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:46.360 7f3ae7574700 1 --2- 172.21.2.102:0/1740176856 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3acc0186c0 0x7f3acc01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:09:46.360 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:46.361 7f3ae7574700 1 --2- 172.21.2.102:0/1740176856 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3acc0186c0 0x7f3acc01abc0 crc :-1 s=READY pgs=231 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:09:46.388 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:46.391 7f3aed75c700 1 -- 172.21.2.102:0/1740176856 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ae80edd40 con 0x7f3ae80be660 2021-02-10T11:09:46.391 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:46.394 7f3adffff700 1 -- 172.21.2.102:0/1740176856 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f3ae0005a70 con 0x7f3ae80be660 2021-02-10T11:09:46.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:46.520 7f3aed75c700 1 -- 172.21.2.102:0/1740176856 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344393", "target": ["mgr", ""]}) v1 -- 0x7f3ae809d2e0 con 0x7f3acc0186c0 2021-02-10T11:09:46.520 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:46.523 7f3adffff700 1 -- 172.21.2.102:0/1740176856 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f3ad0006e40 con 0x7f3acc0186c0 2021-02-10T11:09:46.524 INFO:teuthology.orchestra.run.gibba002.stdout:/volumes/_nogroup/subvolume_0000000000344393/37d25a1c-29d4-4dc9-a7b4-b6a0348cbb59 2021-02-10T11:09:46.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:46.528 7f3aed75c700 1 -- 172.21.2.102:0/1740176856 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3acc0186c0 msgr2=0x7f3acc01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:09:46.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:46.528 7f3aed75c700 1 --2- 172.21.2.102:0/1740176856 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3acc0186c0 0x7f3acc01abc0 crc :-1 s=READY pgs=231 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:09:46.526 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:46.528 7f3aed75c700 1 -- 172.21.2.102:0/1740176856 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3ae80be660 msgr2=0x7f3ae80bcec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:09:46.526 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:46.528 7f3aed75c700 1 --2- 172.21.2.102:0/1740176856 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3ae80be660 0x7f3ae80bcec0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 rx=0x7f3ae000b970 tx=0x7f3ae000bd10).stop 2021-02-10T11:09:46.526 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:46.528 7f3aed75c700 1 -- 172.21.2.102:0/1740176856 shutdown_connections 2021-02-10T11:09:46.526 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:46.528 7f3aed75c700 1 --2- 172.21.2.102:0/1740176856 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3ae80be660 0x7f3ae80bcec0 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:09:46.526 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:46.528 7f3aed75c700 1 --2- 172.21.2.102:0/1740176856 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3acc0186c0 0x7f3acc01abc0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:09:46.526 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:46.528 7f3aed75c700 1 --2- 172.21.2.102:0/1740176856 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3ae819c120 0x7f3ae80be040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:09:46.526 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:46.528 7f3aed75c700 1 --2- 172.21.2.102:0/1740176856 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3ae809cee0 0x7f3ae819bb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:09:46.526 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:46.528 7f3aed75c700 1 -- 172.21.2.102:0/1740176856 >> 172.21.2.102:0/1740176856 conn(0x7f3ae80ac550 msgr2=0x7f3ae80f0c00 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:09:46.526 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:46.529 7f3aed75c700 1 -- 172.21.2.102:0/1740176856 shutdown_connections 2021-02-10T11:09:46.526 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:46.529 7f3aed75c700 1 -- 172.21.2.102:0/1740176856 wait complete. 2021-02-10T11:09:46.534 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000344393 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000344393/37d25a1c-29d4-4dc9-a7b4-b6a0348cbb59 2021-02-10T11:09:46.534 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:09:46.534 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:09:46.534 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:09:46.869 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:09:46.869 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:09:46.870 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344393/37d25a1c-29d4-4dc9-a7b4-b6a0348cbb59/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:09:47.254 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:09:47.254 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:09:47.254 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0254851 s, 41.1 MB/s 2021-02-10T11:09:47.255 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:09:47.255 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:09:47.255 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:09:47.591 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:09:47.592 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:09:47.592 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344393/37d25a1c-29d4-4dc9-a7b4-b6a0348cbb59/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:09:47.968 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:09:47.968 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:09:47.968 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0241788 s, 43.4 MB/s 2021-02-10T11:09:47.969 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:09:47.969 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:09:47.969 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:09:48.305 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:09:48.305 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:09:48.305 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344393/37d25a1c-29d4-4dc9-a7b4-b6a0348cbb59/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:09:48.679 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:09:48.680 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:09:48.680 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0236322 s, 44.4 MB/s 2021-02-10T11:09:48.681 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:09:48.681 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:09:48.681 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:09:49.015 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:09:49.016 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:09:49.016 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344393/37d25a1c-29d4-4dc9-a7b4-b6a0348cbb59/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:09:49.392 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:09:49.393 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:09:49.393 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0249134 s, 42.1 MB/s 2021-02-10T11:09:49.394 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:09:49.394 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:09:49.394 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:09:49.729 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:09:49.729 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:09:49.730 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344393/37d25a1c-29d4-4dc9-a7b4-b6a0348cbb59/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:09:50.103 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:09:50.103 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:09:50.103 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0232303 s, 45.1 MB/s 2021-02-10T11:09:50.104 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:09:50.104 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:09:50.104 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:09:50.440 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:09:50.440 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:09:50.441 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344393/37d25a1c-29d4-4dc9-a7b4-b6a0348cbb59/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:09:50.816 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:09:50.817 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:09:50.817 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0243743 s, 43.0 MB/s 2021-02-10T11:09:50.817 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:09:50.818 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:09:50.818 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:09:51.154 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:09:51.155 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:09:51.155 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344393/37d25a1c-29d4-4dc9-a7b4-b6a0348cbb59/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:09:51.527 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:09:51.528 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:09:51.528 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0220437 s, 47.6 MB/s 2021-02-10T11:09:51.529 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:09:51.529 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:09:51.529 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:09:51.864 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:09:51.864 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:09:51.865 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344393/37d25a1c-29d4-4dc9-a7b4-b6a0348cbb59/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:09:52.242 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:09:52.242 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:09:52.242 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.023606 s, 44.4 MB/s 2021-02-10T11:09:52.243 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:09:52.243 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:09:52.243 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:09:52.581 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:09:52.582 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:09:52.582 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344393/37d25a1c-29d4-4dc9-a7b4-b6a0348cbb59/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:09:52.960 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:09:52.960 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:09:52.960 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0234254 s, 44.8 MB/s 2021-02-10T11:09:52.961 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:09:52.961 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:09:52.961 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:09:53.298 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:09:53.298 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:09:53.299 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344393/37d25a1c-29d4-4dc9-a7b4-b6a0348cbb59/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:09:53.676 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:09:53.676 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:09:53.676 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0254524 s, 41.2 MB/s 2021-02-10T11:09:53.678 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000344394 2021-02-10T11:09:54.114 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:54.097 7f013891f700 1 Processor -- start 2021-02-10T11:09:54.114 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:54.097 7f013891f700 1 -- start start 2021-02-10T11:09:54.114 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:54.098 7f013891f700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f01340f05f0 0x7f01340f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:09:54.114 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:54.098 7f013891f700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01341b0c70 con 0x7f01341b0470 2021-02-10T11:09:54.115 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:54.098 7f013891f700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01340f0b50 con 0x7f01341b0870 2021-02-10T11:09:54.115 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:54.098 7f013891f700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01340e5ea0 con 0x7f01340f05f0 2021-02-10T11:09:54.115 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:54.098 7f0132e1d700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f01340f05f0 0x7f01340f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:09:54.115 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:54.098 7f013261c700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f01341b0470 0x7f01340ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:44006/0 (socket says 172.21.2.102:44006) 2021-02-10T11:09:54.115 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:54.098 7f013261c700 1 -- 172.21.2.102:0/3234360297 learned_addr learned my addr 172.21.2.102:0/3234360297 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:09:54.115 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:54.098 7f0132e1d700 1 -- 172.21.2.102:0/3234360297 >> v1:172.21.2.113:6789/0 conn(0x7f01341b0870 legacy=0x7f01340efe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:09:54.116 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:54.098 7f0132e1d700 1 -- 172.21.2.102:0/3234360297 >> v1:172.21.2.102:6789/0 conn(0x7f01341b0470 legacy=0x7f01340ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:09:54.116 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:54.098 7f0132e1d700 1 -- 172.21.2.102:0/3234360297 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01340e52e0 con 0x7f01340f05f0 2021-02-10T11:09:54.116 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:54.098 7f0132e1d700 1 --2- 172.21.2.102:0/3234360297 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f01340f05f0 0x7f01340f20e0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 rx=0x7f012c006e30 tx=0x7f012c009de0).ready entity=mon.0 client_cookie=ed50a9595ffd85ef server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:09:54.116 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:54.099 7f013161a700 1 -- 172.21.2.102:0/3234360297 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f012c00b300 con 0x7f01340f05f0 2021-02-10T11:09:54.116 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:54.099 7f013161a700 1 -- 172.21.2.102:0/3234360297 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f012c00b680 con 0x7f01340f05f0 2021-02-10T11:09:54.116 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:54.099 7f013161a700 1 -- 172.21.2.102:0/3234360297 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f012c00bba0 con 0x7f01340f05f0 2021-02-10T11:09:54.117 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:54.099 7f013891f700 1 -- 172.21.2.102:0/3234360297 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f01340f05f0 msgr2=0x7f01340f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:09:54.117 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:54.099 7f013891f700 1 --2- 172.21.2.102:0/3234360297 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f01340f05f0 0x7f01340f20e0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 rx=0x7f012c006e30 tx=0x7f012c009de0).stop 2021-02-10T11:09:54.117 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:54.099 7f013891f700 1 -- 172.21.2.102:0/3234360297 shutdown_connections 2021-02-10T11:09:54.117 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:54.099 7f013891f700 1 --2- 172.21.2.102:0/3234360297 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f01340f05f0 0x7f01340f20e0 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:09:54.117 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:54.099 7f013891f700 1 -- 172.21.2.102:0/3234360297 >> 172.21.2.102:0/3234360297 conn(0x7f01340ce3e0 msgr2=0x7f01340e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:09:54.117 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:54.100 7f013891f700 1 -- 172.21.2.102:0/3234360297 shutdown_connections 2021-02-10T11:09:54.118 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:54.100 7f013891f700 1 -- 172.21.2.102:0/3234360297 wait complete. 2021-02-10T11:09:54.118 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:54.114 7f013891f700 1 Processor -- start 2021-02-10T11:09:54.118 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:54.114 7f013891f700 1 -- start start 2021-02-10T11:09:54.118 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:54.115 7f013891f700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0134087510 0x7f01340aedc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:09:54.118 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:54.115 7f013891f700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f01341c68f0 0x7f01341b1f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:09:54.118 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:54.115 7f013891f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f013408c4d0 0x7f01341b18a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:09:54.119 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:54.115 7f013891f700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01340a2d50 con 0x7f013408c4d0 2021-02-10T11:09:54.119 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:54.115 7f013891f700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01340a2f70 con 0x7f0134087510 2021-02-10T11:09:54.119 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:54.115 7f013891f700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01340a3190 con 0x7f01341c68f0 2021-02-10T11:09:54.119 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:54.115 7f0132e1d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f013408c4d0 0x7f01341b18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:09:54.119 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:54.115 7f0132e1d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f013408c4d0 0x7f01341b18a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:51790/0 (socket says 172.21.2.102:51790) 2021-02-10T11:09:54.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:54.115 7f0132e1d700 1 -- 172.21.2.102:0/2655974695 learned_addr learned my addr 172.21.2.102:0/2655974695 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:09:54.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:54.115 7f0131e1b700 1 --2- 172.21.2.102:0/2655974695 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f01341c68f0 0x7f01341b1f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:09:54.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:54.115 7f013261c700 1 --2- 172.21.2.102:0/2655974695 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0134087510 0x7f01340aedc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:09:54.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:54.115 7f0132e1d700 1 -- 172.21.2.102:0/2655974695 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f01341c68f0 msgr2=0x7f01341b1f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:09:54.121 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:54.115 7f0132e1d700 1 --2- 172.21.2.102:0/2655974695 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f01341c68f0 0x7f01341b1f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:09:54.121 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:54.115 7f0132e1d700 1 -- 172.21.2.102:0/2655974695 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0134087510 msgr2=0x7f01340aedc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:09:54.121 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:54.115 7f0132e1d700 1 --2- 172.21.2.102:0/2655974695 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0134087510 0x7f01340aedc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:09:54.121 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:54.115 7f0132e1d700 1 -- 172.21.2.102:0/2655974695 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01340a33b0 con 0x7f013408c4d0 2021-02-10T11:09:54.121 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:54.115 7f0132e1d700 1 --2- 172.21.2.102:0/2655974695 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f013408c4d0 0x7f01341b18a0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 rx=0x7f012c009fc0 tx=0x7f012c00bd40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:09:54.121 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:54.116 7f012affd700 1 -- 172.21.2.102:0/2655974695 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f012c009760 con 0x7f013408c4d0 2021-02-10T11:09:54.122 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:54.116 7f012affd700 1 -- 172.21.2.102:0/2655974695 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f012c0093d0 con 0x7f013408c4d0 2021-02-10T11:09:54.122 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:54.116 7f012affd700 1 -- 172.21.2.102:0/2655974695 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f012c005b20 con 0x7f013408c4d0 2021-02-10T11:09:54.122 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:54.116 7f013891f700 1 -- 172.21.2.102:0/2655974695 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01340a33b0 con 0x7f013408c4d0 2021-02-10T11:09:54.122 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:54.116 7f013891f700 1 -- 172.21.2.102:0/2655974695 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01340afca0 con 0x7f013408c4d0 2021-02-10T11:09:54.122 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:54.116 7f012affd700 1 -- 172.21.2.102:0/2655974695 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f012c005820 con 0x7f013408c4d0 2021-02-10T11:09:54.123 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:54.116 7f012affd700 1 --2- 172.21.2.102:0/2655974695 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f01180186c0 0x7f011801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:09:54.123 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:54.116 7f012affd700 1 -- 172.21.2.102:0/2655974695 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f012c032960 con 0x7f013408c4d0 2021-02-10T11:09:54.123 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:54.116 7f013261c700 1 --2- 172.21.2.102:0/2655974695 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f01180186c0 0x7f011801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:09:54.123 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:54.117 7f013261c700 1 --2- 172.21.2.102:0/2655974695 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f01180186c0 0x7f011801abc0 crc :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:09:54.144 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:54.147 7f013891f700 1 -- 172.21.2.102:0/2655974695 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f013407e6b0 con 0x7f013408c4d0 2021-02-10T11:09:54.146 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:54.150 7f012affd700 1 -- 172.21.2.102:0/2655974695 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f012c0162b0 con 0x7f013408c4d0 2021-02-10T11:09:54.272 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:54.276 7f013891f700 1 -- 172.21.2.102:0/2655974695 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344394", "target": ["mgr", ""]}) v1 -- 0x7f013407e6b0 con 0x7f01180186c0 2021-02-10T11:09:54.300 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:54.303 7f012affd700 1 -- 172.21.2.102:0/2655974695 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f011c001a00 con 0x7f01180186c0 2021-02-10T11:09:54.306 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:54.309 7f013891f700 1 -- 172.21.2.102:0/2655974695 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f01180186c0 msgr2=0x7f011801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:09:54.306 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:54.309 7f013891f700 1 --2- 172.21.2.102:0/2655974695 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f01180186c0 0x7f011801abc0 crc :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:09:54.306 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:54.309 7f013891f700 1 -- 172.21.2.102:0/2655974695 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f013408c4d0 msgr2=0x7f01341b18a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:09:54.306 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:54.309 7f013891f700 1 --2- 172.21.2.102:0/2655974695 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f013408c4d0 0x7f01341b18a0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 rx=0x7f012c009fc0 tx=0x7f012c00bd40).stop 2021-02-10T11:09:54.306 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:54.309 7f013891f700 1 -- 172.21.2.102:0/2655974695 shutdown_connections 2021-02-10T11:09:54.306 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:54.309 7f013891f700 1 --2- 172.21.2.102:0/2655974695 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f013408c4d0 0x7f01341b18a0 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:09:54.306 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:54.309 7f013891f700 1 --2- 172.21.2.102:0/2655974695 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f01341c68f0 0x7f01341b1f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:09:54.306 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:54.309 7f013891f700 1 --2- 172.21.2.102:0/2655974695 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f01180186c0 0x7f011801abc0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:09:54.307 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:54.309 7f013891f700 1 --2- 172.21.2.102:0/2655974695 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0134087510 0x7f01340aedc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:09:54.307 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:54.309 7f013891f700 1 -- 172.21.2.102:0/2655974695 >> 172.21.2.102:0/2655974695 conn(0x7f01340886d0 msgr2=0x7f0134081c40 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:09:54.307 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:54.309 7f013891f700 1 -- 172.21.2.102:0/2655974695 shutdown_connections 2021-02-10T11:09:54.307 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:54.309 7f013891f700 1 -- 172.21.2.102:0/2655974695 wait complete. 2021-02-10T11:09:54.314 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000344394 2021-02-10T11:09:54.747 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:54.734 7f241d9be700 1 Processor -- start 2021-02-10T11:09:54.748 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:54.734 7f241d9be700 1 -- start start 2021-02-10T11:09:54.748 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:54.735 7f241d9be700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f24181b0470 0x7f24181b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:09:54.748 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:54.735 7f241d9be700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f24180f20e0 0x7f24180f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:09:54.748 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:54.735 7f241d9be700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24180f0b50 con 0x7f24180ef610 2021-02-10T11:09:54.748 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:54.735 7f241d9be700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24180f0820 con 0x7f24180f20e0 2021-02-10T11:09:54.748 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:54.735 7f241d9be700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24180f24e0 con 0x7f24181b0470 2021-02-10T11:09:54.748 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:54.735 7f2416d73700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f24180ef610 0x7f24180efa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:44020/0 (socket says 172.21.2.102:44020) 2021-02-10T11:09:54.748 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:54.735 7f2416d73700 1 -- 172.21.2.102:0/1714356001 learned_addr learned my addr 172.21.2.102:0/1714356001 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:09:54.748 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:54.735 7f2417d75700 1 --2- 172.21.2.102:0/1714356001 >> v2:172.21.2.102:3300/0 conn(0x7f24180f20e0 0x7f24180f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:09:54.748 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:54.735 7f2416572700 1 -- 172.21.2.102:0/1714356001 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1998561709 0 0) 0x7f24000010b0 con 0x7f24180ef610 2021-02-10T11:09:54.749 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:54.735 7f2416572700 1 -- 172.21.2.102:0/1714356001 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2404002fe0 con 0x7f24180ef610 2021-02-10T11:09:54.749 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:54.735 7f2417d75700 1 -- 172.21.2.102:0/1714356001 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f24181b0470 msgr2=0x7f24181b0870 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:09:54.749 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:54.735 7f2417d75700 1 --2- 172.21.2.102:0/1714356001 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f24181b0470 0x7f24181b0870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:09:54.749 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:54.735 7f2417d75700 1 -- 172.21.2.102:0/1714356001 >> v1:172.21.2.102:6789/0 conn(0x7f24180ef610 legacy=0x7f24180efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:09:54.749 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:54.735 7f2417d75700 1 -- 172.21.2.102:0/1714356001 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24180e52e0 con 0x7f24180f20e0 2021-02-10T11:09:54.749 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:54.736 7f2417d75700 1 --2- 172.21.2.102:0/1714356001 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f24180f20e0 0x7f24180f0200 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 rx=0x7f2410006e30 tx=0x7f2410009e60).ready entity=mon.0 client_cookie=bb996e313dbffc6e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:09:54.749 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:54.736 7f2416572700 1 -- 172.21.2.102:0/1714356001 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2410006a50 con 0x7f24180f20e0 2021-02-10T11:09:54.749 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:54.736 7f2416572700 1 -- 172.21.2.102:0/1714356001 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f241000b300 con 0x7f24180f20e0 2021-02-10T11:09:54.749 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:54.736 7f2416572700 1 -- 172.21.2.102:0/1714356001 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f241000b7e0 con 0x7f24180f20e0 2021-02-10T11:09:54.749 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:54.736 7f241d9be700 1 -- 172.21.2.102:0/1714356001 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f24180f20e0 msgr2=0x7f24180f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:09:54.749 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:54.736 7f241d9be700 1 --2- 172.21.2.102:0/1714356001 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f24180f20e0 0x7f24180f0200 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 rx=0x7f2410006e30 tx=0x7f2410009e60).stop 2021-02-10T11:09:54.749 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:54.736 7f241d9be700 1 -- 172.21.2.102:0/1714356001 shutdown_connections 2021-02-10T11:09:54.749 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:54.736 7f241d9be700 1 --2- 172.21.2.102:0/1714356001 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f24180f20e0 0x7f24180f0200 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:09:54.750 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:54.736 7f241d9be700 1 --2- 172.21.2.102:0/1714356001 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f24181b0470 0x7f24181b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:09:54.750 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:54.736 7f241d9be700 1 -- 172.21.2.102:0/1714356001 >> 172.21.2.102:0/1714356001 conn(0x7f24180ce3e0 msgr2=0x7f24180e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:09:54.750 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:54.736 7f241d9be700 1 -- 172.21.2.102:0/1714356001 shutdown_connections 2021-02-10T11:09:54.750 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:54.736 7f241d9be700 1 -- 172.21.2.102:0/1714356001 wait complete. 2021-02-10T11:09:54.750 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:54.751 7f241d9be700 1 Processor -- start 2021-02-10T11:09:54.750 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:54.751 7f241d9be700 1 -- start start 2021-02-10T11:09:54.751 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:54.751 7f241d9be700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f24180b7de0 0x7f2418190920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:09:54.751 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:54.751 7f241d9be700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f24180b82f0 0x7f24180b86f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:09:54.752 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:54.751 7f241d9be700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f24180b8d10 0x7f24180bf9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:09:54.752 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:54.751 7f241d9be700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24180b7930 con 0x7f24180b8d10 2021-02-10T11:09:54.752 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:54.751 7f241d9be700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24180b7b20 con 0x7f24180b82f0 2021-02-10T11:09:54.752 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:54.751 7f241d9be700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24180bffd0 con 0x7f24180b7de0 2021-02-10T11:09:54.752 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:54.752 7f2417d75700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f24180b8d10 0x7f24180bf9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:09:54.752 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:54.752 7f2417d75700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f24180b8d10 0x7f24180bf9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:51800/0 (socket says 172.21.2.102:51800) 2021-02-10T11:09:54.752 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:54.752 7f2417d75700 1 -- 172.21.2.102:0/104480816 learned_addr learned my addr 172.21.2.102:0/104480816 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:09:54.752 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:54.752 7f2417d75700 1 -- 172.21.2.102:0/104480816 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f24180b7de0 msgr2=0x7f2418190920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:09:54.752 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:54.752 7f2417d75700 1 --2- 172.21.2.102:0/104480816 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f24180b7de0 0x7f2418190920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:09:54.752 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:54.752 7f2417d75700 1 -- 172.21.2.102:0/104480816 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f24180b82f0 msgr2=0x7f24180b86f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:09:54.752 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:54.752 7f2417d75700 1 --2- 172.21.2.102:0/104480816 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f24180b82f0 0x7f24180b86f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:09:54.752 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:54.752 7f2417d75700 1 -- 172.21.2.102:0/104480816 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24180c01c0 con 0x7f24180b8d10 2021-02-10T11:09:54.753 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:54.752 7f2417d75700 1 --2- 172.21.2.102:0/104480816 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f24180b8d10 0x7f24180bf9b0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 rx=0x7f241000b970 tx=0x7f241000bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:09:54.753 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:54.752 7f240ffff700 1 -- 172.21.2.102:0/104480816 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2410009380 con 0x7f24180b8d10 2021-02-10T11:09:54.753 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:54.752 7f240ffff700 1 -- 172.21.2.102:0/104480816 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2410005820 con 0x7f24180b8d10 2021-02-10T11:09:54.753 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:54.752 7f240ffff700 1 -- 172.21.2.102:0/104480816 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2410005d40 con 0x7f24180b8d10 2021-02-10T11:09:54.753 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:54.752 7f241d9be700 1 -- 172.21.2.102:0/104480816 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24180c01c0 con 0x7f24180b8d10 2021-02-10T11:09:54.753 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:54.753 7f241d9be700 1 -- 172.21.2.102:0/104480816 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24180c0750 con 0x7f24180b8d10 2021-02-10T11:09:54.753 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:54.753 7f240ffff700 1 -- 172.21.2.102:0/104480816 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f2410005cd0 con 0x7f24180b8d10 2021-02-10T11:09:54.753 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:54.753 7f240ffff700 1 --2- 172.21.2.102:0/104480816 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f23fc0186c0 0x7f23fc01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:09:54.753 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:54.753 7f240ffff700 1 -- 172.21.2.102:0/104480816 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f24100327c0 con 0x7f24180b8d10 2021-02-10T11:09:54.753 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:54.753 7f2417574700 1 --2- 172.21.2.102:0/104480816 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f23fc0186c0 0x7f23fc01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:09:54.753 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:54.754 7f2417574700 1 --2- 172.21.2.102:0/104480816 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f23fc0186c0 0x7f23fc01abc0 crc :-1 s=READY pgs=233 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:09:54.782 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:54.785 7f241d9be700 1 -- 172.21.2.102:0/104480816 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2418105b30 con 0x7f24180b8d10 2021-02-10T11:09:54.784 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:54.788 7f240ffff700 1 -- 172.21.2.102:0/104480816 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f2410025f50 con 0x7f24180b8d10 2021-02-10T11:09:54.911 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:54.915 7f241d9be700 1 -- 172.21.2.102:0/104480816 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344394", "target": ["mgr", ""]}) v1 -- 0x7f24180c0750 con 0x7f23fc0186c0 2021-02-10T11:09:54.914 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:54.918 7f240ffff700 1 -- 172.21.2.102:0/104480816 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f2408006e40 con 0x7f23fc0186c0 2021-02-10T11:09:54.918 INFO:teuthology.orchestra.run.gibba002.stdout:/volumes/_nogroup/subvolume_0000000000344394/45c7afda-8396-4bd5-9203-add4b7c0f592 2021-02-10T11:09:54.920 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:54.923 7f241d9be700 1 -- 172.21.2.102:0/104480816 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f23fc0186c0 msgr2=0x7f23fc01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:09:54.920 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:54.923 7f241d9be700 1 --2- 172.21.2.102:0/104480816 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f23fc0186c0 0x7f23fc01abc0 crc :-1 s=READY pgs=233 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:09:54.920 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:54.923 7f241d9be700 1 -- 172.21.2.102:0/104480816 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f24180b8d10 msgr2=0x7f24180bf9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:09:54.920 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:54.923 7f241d9be700 1 --2- 172.21.2.102:0/104480816 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f24180b8d10 0x7f24180bf9b0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 rx=0x7f241000b970 tx=0x7f241000bd10).stop 2021-02-10T11:09:54.920 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:54.923 7f241d9be700 1 -- 172.21.2.102:0/104480816 shutdown_connections 2021-02-10T11:09:54.921 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:54.923 7f241d9be700 1 --2- 172.21.2.102:0/104480816 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f24180b8d10 0x7f24180bf9b0 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:09:54.921 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:54.923 7f241d9be700 1 --2- 172.21.2.102:0/104480816 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f23fc0186c0 0x7f23fc01abc0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:09:54.921 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:54.923 7f241d9be700 1 --2- 172.21.2.102:0/104480816 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f24180b82f0 0x7f24180b86f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:09:54.921 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:54.923 7f241d9be700 1 --2- 172.21.2.102:0/104480816 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f24180b7de0 0x7f2418190920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:09:54.921 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:54.923 7f241d9be700 1 -- 172.21.2.102:0/104480816 >> 172.21.2.102:0/104480816 conn(0x7f2418094e10 msgr2=0x7f24180a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:09:54.921 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:54.924 7f241d9be700 1 -- 172.21.2.102:0/104480816 shutdown_connections 2021-02-10T11:09:54.921 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:09:54.924 7f241d9be700 1 -- 172.21.2.102:0/104480816 wait complete. 2021-02-10T11:09:54.928 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000344394 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000344394/45c7afda-8396-4bd5-9203-add4b7c0f592 2021-02-10T11:09:54.928 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:09:54.928 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:09:54.929 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:09:55.265 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:09:55.266 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:09:55.266 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344394/45c7afda-8396-4bd5-9203-add4b7c0f592/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:09:55.647 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:09:55.647 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:09:55.648 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0234236 s, 44.8 MB/s 2021-02-10T11:09:55.648 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:09:55.649 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:09:55.649 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:09:55.983 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:09:55.984 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:09:55.984 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344394/45c7afda-8396-4bd5-9203-add4b7c0f592/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:09:56.360 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:09:56.360 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:09:56.360 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0242906 s, 43.2 MB/s 2021-02-10T11:09:56.361 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:09:56.361 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:09:56.361 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:09:56.695 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:09:56.696 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:09:56.696 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344394/45c7afda-8396-4bd5-9203-add4b7c0f592/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:09:57.070 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:09:57.070 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:09:57.070 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0236308 s, 44.4 MB/s 2021-02-10T11:09:57.071 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:09:57.071 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:09:57.071 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:09:57.407 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:09:57.408 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:09:57.408 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344394/45c7afda-8396-4bd5-9203-add4b7c0f592/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:09:57.782 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:09:57.782 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:09:57.782 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.02199 s, 47.7 MB/s 2021-02-10T11:09:57.783 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:09:57.784 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:09:57.784 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:09:58.118 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:09:58.119 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:09:58.119 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344394/45c7afda-8396-4bd5-9203-add4b7c0f592/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:09:58.492 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:09:58.492 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:09:58.492 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0244149 s, 42.9 MB/s 2021-02-10T11:09:58.493 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:09:58.493 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:09:58.494 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:09:58.828 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:09:58.828 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:09:58.829 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344394/45c7afda-8396-4bd5-9203-add4b7c0f592/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:09:59.205 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:09:59.206 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:09:59.206 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0256809 s, 40.8 MB/s 2021-02-10T11:09:59.207 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:09:59.207 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:09:59.207 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:09:59.543 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:09:59.543 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:09:59.543 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344394/45c7afda-8396-4bd5-9203-add4b7c0f592/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:09:59.917 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:09:59.917 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:09:59.917 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0226289 s, 46.3 MB/s 2021-02-10T11:09:59.918 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:09:59.918 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:09:59.918 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:10:00.253 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:10:00.254 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:10:00.254 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344394/45c7afda-8396-4bd5-9203-add4b7c0f592/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:10:00.629 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:10:00.629 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:10:00.630 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.023436 s, 44.7 MB/s 2021-02-10T11:10:00.630 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:10:00.631 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:10:00.631 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:10:00.965 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:10:00.965 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:10:00.965 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344394/45c7afda-8396-4bd5-9203-add4b7c0f592/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:10:01.343 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:10:01.343 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:10:01.343 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0244264 s, 42.9 MB/s 2021-02-10T11:10:01.344 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:10:01.344 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:10:01.344 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:10:01.682 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:10:01.682 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:10:01.683 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344394/45c7afda-8396-4bd5-9203-add4b7c0f592/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:10:02.061 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:10:02.061 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:10:02.062 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0228859 s, 45.8 MB/s 2021-02-10T11:10:02.063 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000344395 2021-02-10T11:10:02.496 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:02.483 7fe847c99700 1 Processor -- start 2021-02-10T11:10:02.496 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:02.483 7fe847c99700 1 -- start start 2021-02-10T11:10:02.496 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:02.483 7fe847c99700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe8401b0470 0x7fe8401b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:10:02.497 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:02.483 7fe847c99700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fe8400f20e0 0x7fe8400f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:10:02.497 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:02.483 7fe847c99700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8400f0b50 con 0x7fe8400ef610 2021-02-10T11:10:02.497 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:02.483 7fe847c99700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8400f0820 con 0x7fe8400f20e0 2021-02-10T11:10:02.497 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:02.483 7fe847c99700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8400f24e0 con 0x7fe8401b0470 2021-02-10T11:10:02.497 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:02.484 7fe8462b6700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fe8400f20e0 0x7fe8400f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:10:02.497 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:02.484 7fe8462b6700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fe8400f20e0 0x7fe8400f0200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:51812/0 (socket says 172.21.2.102:51812) 2021-02-10T11:10:02.497 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:02.484 7fe8462b6700 1 -- 172.21.2.102:0/2417153921 learned_addr learned my addr 172.21.2.102:0/2417153921 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:10:02.497 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:02.484 7fe845ab5700 1 --2- 172.21.2.102:0/2417153921 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe8401b0470 0x7fe8401b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:10:02.497 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:02.484 7fe8462b6700 1 -- 172.21.2.102:0/2417153921 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe8401b0470 msgr2=0x7fe8401b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:10:02.497 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:02.484 7fe8462b6700 1 --2- 172.21.2.102:0/2417153921 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe8401b0470 0x7fe8401b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:10:02.497 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:02.484 7fe8462b6700 1 -- 172.21.2.102:0/2417153921 >> v1:172.21.2.113:6789/0 conn(0x7fe8400ef610 legacy=0x7fe8400efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:10:02.497 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:02.484 7fe8462b6700 1 -- 172.21.2.102:0/2417153921 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8400e52e0 con 0x7fe8400f20e0 2021-02-10T11:10:02.498 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:02.484 7fe8462b6700 1 --2- 172.21.2.102:0/2417153921 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe8400f20e0 0x7fe8400f0200 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 rx=0x7fe838006f00 tx=0x7fe838009f10).ready entity=mon.0 client_cookie=1511f9de4a0f6d15 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:10:02.498 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:02.484 7fe844ab3700 1 -- 172.21.2.102:0/2417153921 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe83800b3b0 con 0x7fe8400f20e0 2021-02-10T11:10:02.498 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:02.484 7fe844ab3700 1 -- 172.21.2.102:0/2417153921 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe83800b730 con 0x7fe8400f20e0 2021-02-10T11:10:02.498 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:02.484 7fe844ab3700 1 -- 172.21.2.102:0/2417153921 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe83800bc50 con 0x7fe8400f20e0 2021-02-10T11:10:02.498 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:02.484 7fe847c99700 1 -- 172.21.2.102:0/2417153921 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe8400f20e0 msgr2=0x7fe8400f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:10:02.498 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:02.484 7fe847c99700 1 --2- 172.21.2.102:0/2417153921 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe8400f20e0 0x7fe8400f0200 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 rx=0x7fe838006f00 tx=0x7fe838009f10).stop 2021-02-10T11:10:02.498 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:02.485 7fe847c99700 1 -- 172.21.2.102:0/2417153921 shutdown_connections 2021-02-10T11:10:02.498 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:02.485 7fe847c99700 1 --2- 172.21.2.102:0/2417153921 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe8400f20e0 0x7fe8400f0200 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:10:02.498 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:02.485 7fe847c99700 1 --2- 172.21.2.102:0/2417153921 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe8401b0470 0x7fe8401b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:10:02.498 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:02.485 7fe847c99700 1 -- 172.21.2.102:0/2417153921 >> 172.21.2.102:0/2417153921 conn(0x7fe8400ce3e0 msgr2=0x7fe8400e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:10:02.498 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:02.485 7fe847c99700 1 -- 172.21.2.102:0/2417153921 shutdown_connections 2021-02-10T11:10:02.498 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:02.485 7fe847c99700 1 -- 172.21.2.102:0/2417153921 wait complete. 2021-02-10T11:10:02.500 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:02.500 7fe847c99700 1 Processor -- start 2021-02-10T11:10:02.500 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:02.500 7fe847c99700 1 -- start start 2021-02-10T11:10:02.500 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:02.501 7fe847c99700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe8401084a0 0x7fe84009e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:10:02.500 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:02.501 7fe847c99700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe8401071e0 0x7fe8401075e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:10:02.500 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:02.501 7fe847c99700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe840107c00 0x7fe840189bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:10:02.500 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:02.501 7fe847c99700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe840108000 con 0x7fe8401084a0 2021-02-10T11:10:02.500 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:02.501 7fe847c99700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe84009eb10 con 0x7fe840107c00 2021-02-10T11:10:02.500 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:02.501 7fe847c99700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8401088a0 con 0x7fe8401071e0 2021-02-10T11:10:02.500 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:02.501 7fe845ab5700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe8401084a0 0x7fe84009e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:10:02.500 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:02.501 7fe845ab5700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe8401084a0 0x7fe84009e4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:51814/0 (socket says 172.21.2.102:51814) 2021-02-10T11:10:02.501 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:02.501 7fe845ab5700 1 -- 172.21.2.102:0/1634196939 learned_addr learned my addr 172.21.2.102:0/1634196939 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:10:02.501 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:02.501 7fe845ab5700 1 -- 172.21.2.102:0/1634196939 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe8401071e0 msgr2=0x7fe8401075e0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:10:02.501 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:02.501 7fe845ab5700 1 --2- 172.21.2.102:0/1634196939 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe8401071e0 0x7fe8401075e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:10:02.501 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:02.501 7fe845ab5700 1 -- 172.21.2.102:0/1634196939 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe840107c00 msgr2=0x7fe840189bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:10:02.501 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:02.501 7fe845ab5700 1 --2- 172.21.2.102:0/1634196939 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe840107c00 0x7fe840189bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:10:02.501 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:02.501 7fe845ab5700 1 -- 172.21.2.102:0/1634196939 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe84018a210 con 0x7fe8401084a0 2021-02-10T11:10:02.501 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:02.501 7fe845ab5700 1 --2- 172.21.2.102:0/1634196939 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe8401084a0 0x7fe84009e4f0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 rx=0x7fe83c006f60 tx=0x7fe83c00b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:10:02.501 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:02.501 7fe8327fc700 1 -- 172.21.2.102:0/1634196939 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe83c0009b0 con 0x7fe8401084a0 2021-02-10T11:10:02.501 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:02.501 7fe8327fc700 1 -- 172.21.2.102:0/1634196939 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe83c00bdf0 con 0x7fe8401084a0 2021-02-10T11:10:02.501 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:02.501 7fe8327fc700 1 -- 172.21.2.102:0/1634196939 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe83c002030 con 0x7fe8401084a0 2021-02-10T11:10:02.501 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:02.501 7fe847c99700 1 -- 172.21.2.102:0/1634196939 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe84018a210 con 0x7fe8401084a0 2021-02-10T11:10:02.501 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:02.501 7fe847c99700 1 -- 172.21.2.102:0/1634196939 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe84018a210 con 0x7fe8401084a0 2021-02-10T11:10:02.501 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:02.502 7fe8327fc700 1 -- 172.21.2.102:0/1634196939 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fe83c006b00 con 0x7fe8401084a0 2021-02-10T11:10:02.501 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:02.502 7fe8327fc700 1 --2- 172.21.2.102:0/1634196939 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe828018730 0x7fe82801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:10:02.502 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:02.502 7fe8327fc700 1 -- 172.21.2.102:0/1634196939 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fe83c0327b0 con 0x7fe8401084a0 2021-02-10T11:10:02.502 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:02.502 7fe8452b4700 1 --2- 172.21.2.102:0/1634196939 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe828018730 0x7fe82801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:10:02.502 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:02.502 7fe8452b4700 1 --2- 172.21.2.102:0/1634196939 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe828018730 0x7fe82801ac10 crc :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:10:02.529 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:02.533 7fe847c99700 1 -- 172.21.2.102:0/1634196939 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8400f0b50 con 0x7fe8401084a0 2021-02-10T11:10:02.532 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:02.535 7fe8327fc700 1 -- 172.21.2.102:0/1634196939 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fe83c006340 con 0x7fe8401084a0 2021-02-10T11:10:02.660 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:02.663 7fe847c99700 1 -- 172.21.2.102:0/1634196939 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344395", "target": ["mgr", ""]}) v1 -- 0x7fe84009eb10 con 0x7fe828018730 2021-02-10T11:10:02.685 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:02.688 7fe8327fc700 1 -- 172.21.2.102:0/1634196939 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fe834006b80 con 0x7fe828018730 2021-02-10T11:10:02.693 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:02.697 7fe847c99700 1 -- 172.21.2.102:0/1634196939 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe828018730 msgr2=0x7fe82801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:10:02.693 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:02.697 7fe847c99700 1 --2- 172.21.2.102:0/1634196939 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe828018730 0x7fe82801ac10 crc :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:10:02.694 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:02.697 7fe847c99700 1 -- 172.21.2.102:0/1634196939 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe8401084a0 msgr2=0x7fe84009e4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:10:02.694 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:02.697 7fe847c99700 1 --2- 172.21.2.102:0/1634196939 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe8401084a0 0x7fe84009e4f0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 rx=0x7fe83c006f60 tx=0x7fe83c00b970).stop 2021-02-10T11:10:02.694 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:02.697 7fe847c99700 1 -- 172.21.2.102:0/1634196939 shutdown_connections 2021-02-10T11:10:02.694 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:02.697 7fe847c99700 1 --2- 172.21.2.102:0/1634196939 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe828018730 0x7fe82801ac10 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:10:02.694 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:02.697 7fe847c99700 1 --2- 172.21.2.102:0/1634196939 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe840107c00 0x7fe840189bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:10:02.694 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:02.697 7fe847c99700 1 --2- 172.21.2.102:0/1634196939 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe8401071e0 0x7fe8401075e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:10:02.694 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:02.697 7fe847c99700 1 --2- 172.21.2.102:0/1634196939 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe8401084a0 0x7fe84009e4f0 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:10:02.694 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:02.697 7fe847c99700 1 -- 172.21.2.102:0/1634196939 >> 172.21.2.102:0/1634196939 conn(0x7fe840091780 msgr2=0x7fe8400c3200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:10:02.694 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:02.697 7fe847c99700 1 -- 172.21.2.102:0/1634196939 shutdown_connections 2021-02-10T11:10:02.694 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:02.697 7fe847c99700 1 -- 172.21.2.102:0/1634196939 wait complete. 2021-02-10T11:10:02.701 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000344395 2021-02-10T11:10:03.133 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:03.119 7efeb8874700 1 Processor -- start 2021-02-10T11:10:03.133 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:03.119 7efeb8874700 1 -- start start 2021-02-10T11:10:03.133 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:03.119 7efeb8874700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7efeb40f05f0 0x7efeb40f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:10:03.133 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:03.119 7efeb8874700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efeb41b0c70 con 0x7efeb41b0870 2021-02-10T11:10:03.133 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:03.119 7efeb8874700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efeb40f0b50 con 0x7efeb41b0470 2021-02-10T11:10:03.133 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:03.119 7efeb8874700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efeb40e5ea0 con 0x7efeb40f05f0 2021-02-10T11:10:03.133 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:03.120 7efeb1e1b700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7efeb41b0870 0x7efeb40efe00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:44046/0 (socket says 172.21.2.102:44046) 2021-02-10T11:10:03.134 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:03.120 7efeb1e1b700 1 -- 172.21.2.102:0/405557649 learned_addr learned my addr 172.21.2.102:0/405557649 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:10:03.134 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:03.120 7efeb2e1d700 1 --2- 172.21.2.102:0/405557649 >> v2:172.21.2.113:3300/0 conn(0x7efeb40f05f0 0x7efeb40f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:10:03.134 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:03.120 7efeb161a700 1 -- 172.21.2.102:0/405557649 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2594421396 0 0) 0x7efea40010b0 con 0x7efeb41b0870 2021-02-10T11:10:03.134 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:03.120 7efeb161a700 1 -- 172.21.2.102:0/405557649 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efea0002fe0 con 0x7efeb41b0870 2021-02-10T11:10:03.134 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:03.120 7efeb161a700 1 -- 172.21.2.102:0/405557649 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3783230804 0 0) 0x7efea40010b0 con 0x7efeb41b0870 2021-02-10T11:10:03.134 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:03.120 7efeb161a700 1 -- 172.21.2.102:0/405557649 >> v2:172.21.2.113:3300/0 conn(0x7efeb40f05f0 msgr2=0x7efeb40f20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:10:03.134 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:03.120 7efeb161a700 1 --2- 172.21.2.102:0/405557649 >> v2:172.21.2.113:3300/0 conn(0x7efeb40f05f0 0x7efeb40f20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:10:03.134 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:03.120 7efeb161a700 1 -- 172.21.2.102:0/405557649 >> v1:172.21.2.113:6789/0 conn(0x7efeb41b0470 legacy=0x7efeb40ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:10:03.134 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:03.120 7efeb161a700 1 -- 172.21.2.102:0/405557649 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efeb40e52e0 con 0x7efeb41b0870 2021-02-10T11:10:03.134 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:03.120 7efeb161a700 1 -- 172.21.2.102:0/405557649 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7efea40015b0 con 0x7efeb41b0870 2021-02-10T11:10:03.134 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:03.120 7efeb161a700 1 -- 172.21.2.102:0/405557649 >> v1:172.21.2.102:6789/0 conn(0x7efeb41b0870 legacy=0x7efeb40efe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:10:03.134 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:03.120 7efeb161a700 1 --2- 172.21.2.102:0/405557649 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efea0003ac0 0x7efea0005f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:10:03.134 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:03.120 7efeb161a700 1 --2- 172.21.2.102:0/405557649 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7efea0006780 0x7efea0008ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:10:03.135 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:03.120 7efeb161a700 1 --2- 172.21.2.102:0/405557649 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7efea00095a0 0x7efea000baa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:10:03.135 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:03.120 7efeb161a700 1 -- 172.21.2.102:0/405557649 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efea000c2f0 con 0x7efea0003ac0 2021-02-10T11:10:03.135 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:03.120 7efeb161a700 1 -- 172.21.2.102:0/405557649 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efea000c510 con 0x7efea0006780 2021-02-10T11:10:03.135 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:03.120 7efeb161a700 1 -- 172.21.2.102:0/405557649 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efea000c730 con 0x7efea00095a0 2021-02-10T11:10:03.135 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:03.120 7efeb261c700 1 --2- 172.21.2.102:0/405557649 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efea0003ac0 0x7efea0005f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:10:03.135 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:03.121 7efeb1e1b700 1 --2- 172.21.2.102:0/405557649 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7efea0006780 0x7efea0008ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:10:03.135 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:03.121 7efeb2e1d700 1 --2- 172.21.2.102:0/405557649 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7efea00095a0 0x7efea000baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:10:03.135 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:03.121 7efeb261c700 1 -- 172.21.2.102:0/405557649 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7efea00095a0 msgr2=0x7efea000baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:10:03.135 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:03.121 7efeb261c700 1 --2- 172.21.2.102:0/405557649 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7efea00095a0 0x7efea000baa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:10:03.135 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:03.121 7efeb261c700 1 -- 172.21.2.102:0/405557649 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7efea0006780 msgr2=0x7efea0008ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:10:03.135 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:03.121 7efeb261c700 1 --2- 172.21.2.102:0/405557649 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7efea0006780 0x7efea0008ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:10:03.135 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:03.121 7efeb261c700 1 -- 172.21.2.102:0/405557649 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efea00509a0 con 0x7efea0003ac0 2021-02-10T11:10:03.135 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:03.121 7efeb2e1d700 1 --2- 172.21.2.102:0/405557649 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7efea00095a0 0x7efea000baa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:10:03.135 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:03.121 7efeb2e1d700 1 -- 172.21.2.102:0/405557649 reap_dead start 2021-02-10T11:10:03.136 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:03.121 7efeb261c700 1 --2- 172.21.2.102:0/405557649 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efea0003ac0 0x7efea0005f60 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 rx=0x7efe9c006ac0 tx=0x7efe9c009af0).ready entity=mon.0 client_cookie=e7861b7b0cd7a22d server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:10:03.136 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:03.121 7efeb161a700 1 -- 172.21.2.102:0/405557649 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efe9c007280 con 0x7efea0003ac0 2021-02-10T11:10:03.136 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:03.121 7efeb161a700 1 -- 172.21.2.102:0/405557649 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe9c007500 con 0x7efea0003ac0 2021-02-10T11:10:03.136 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:03.121 7efeb161a700 1 -- 172.21.2.102:0/405557649 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efe9c007a20 con 0x7efea0003ac0 2021-02-10T11:10:03.136 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:03.121 7efeb8874700 1 -- 172.21.2.102:0/405557649 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efea0003ac0 msgr2=0x7efea0005f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:10:03.136 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:03.121 7efeb8874700 1 --2- 172.21.2.102:0/405557649 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efea0003ac0 0x7efea0005f60 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 rx=0x7efe9c006ac0 tx=0x7efe9c009af0).stop 2021-02-10T11:10:03.136 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:03.122 7efeb8874700 1 -- 172.21.2.102:0/405557649 shutdown_connections 2021-02-10T11:10:03.136 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:03.122 7efeb8874700 1 --2- 172.21.2.102:0/405557649 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efea0003ac0 0x7efea0005f60 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:10:03.136 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:03.122 7efeb8874700 1 -- 172.21.2.102:0/405557649 >> 172.21.2.102:0/405557649 conn(0x7efeb40ce3e0 msgr2=0x7efeb40e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:10:03.136 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:03.122 7efeb8874700 1 -- 172.21.2.102:0/405557649 shutdown_connections 2021-02-10T11:10:03.136 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:03.122 7efeb8874700 1 -- 172.21.2.102:0/405557649 wait complete. 2021-02-10T11:10:03.136 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:03.136 7efeb8874700 1 Processor -- start 2021-02-10T11:10:03.136 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:03.136 7efeb8874700 1 -- start start 2021-02-10T11:10:03.137 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:03.137 7efeb8874700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7efeb409b930 0x7efeb40b7c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:10:03.138 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:03.137 7efeb8874700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efeb40b8410 0x7efeb40b8810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:10:03.138 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:03.137 7efeb8874700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7efeb40c1f80 0x7efeb40c2380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:10:03.138 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:03.137 7efeb8874700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efeb40c1a00 con 0x7efeb40b8410 2021-02-10T11:10:03.138 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:03.137 7efeb8874700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efeb40b8e30 con 0x7efeb409b930 2021-02-10T11:10:03.138 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:03.137 7efeb8874700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efeb40b9050 con 0x7efeb40c1f80 2021-02-10T11:10:03.138 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:03.137 7efeb1e1b700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efeb40b8410 0x7efeb40b8810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:10:03.138 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:03.137 7efeb1e1b700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efeb40b8410 0x7efeb40b8810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:51832/0 (socket says 172.21.2.102:51832) 2021-02-10T11:10:03.138 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:03.137 7efeb1e1b700 1 -- 172.21.2.102:0/469087757 learned_addr learned my addr 172.21.2.102:0/469087757 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:10:03.138 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:03.137 7efeb1e1b700 1 -- 172.21.2.102:0/469087757 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7efeb40c1f80 msgr2=0x7efeb40c2380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:10:03.138 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:03.137 7efeb1e1b700 1 --2- 172.21.2.102:0/469087757 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7efeb40c1f80 0x7efeb40c2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:10:03.138 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:03.137 7efeb1e1b700 1 -- 172.21.2.102:0/469087757 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7efeb409b930 msgr2=0x7efeb40b7c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:10:03.138 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:03.137 7efeb1e1b700 1 --2- 172.21.2.102:0/469087757 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7efeb409b930 0x7efeb40b7c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:10:03.138 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:03.137 7efeb1e1b700 1 -- 172.21.2.102:0/469087757 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efeb40c29a0 con 0x7efeb40b8410 2021-02-10T11:10:03.139 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:03.138 7efeb1e1b700 1 --2- 172.21.2.102:0/469087757 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efeb40b8410 0x7efeb40b8810 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 rx=0x7efea4001c00 tx=0x7efea400a8e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:10:03.139 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:03.138 7efeaaffd700 1 -- 172.21.2.102:0/469087757 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efea40009b0 con 0x7efeb40b8410 2021-02-10T11:10:03.139 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:03.138 7efeaaffd700 1 -- 172.21.2.102:0/469087757 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efea400acd0 con 0x7efeb40b8410 2021-02-10T11:10:03.139 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:03.138 7efeaaffd700 1 -- 172.21.2.102:0/469087757 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efea40045d0 con 0x7efeb40b8410 2021-02-10T11:10:03.139 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:03.138 7efeb8874700 1 -- 172.21.2.102:0/469087757 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efeb40c29a0 con 0x7efeb40b8410 2021-02-10T11:10:03.139 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:03.138 7efeb8874700 1 -- 172.21.2.102:0/469087757 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efeb40c2f30 con 0x7efeb40b8410 2021-02-10T11:10:03.139 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:03.138 7efeaaffd700 1 -- 172.21.2.102:0/469087757 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7efea4004140 con 0x7efeb40b8410 2021-02-10T11:10:03.139 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:03.138 7efeaaffd700 1 --2- 172.21.2.102:0/469087757 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7efe980186c0 0x7efe9801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:10:03.139 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:03.138 7efeaaffd700 1 -- 172.21.2.102:0/469087757 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7efea4032410 con 0x7efeb40b8410 2021-02-10T11:10:03.139 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:03.139 7efeb261c700 1 --2- 172.21.2.102:0/469087757 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7efe980186c0 0x7efe9801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:10:03.139 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:03.139 7efeb261c700 1 --2- 172.21.2.102:0/469087757 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7efe980186c0 0x7efe9801abc0 crc :-1 s=READY pgs=235 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:10:03.165 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:03.169 7efeb8874700 1 -- 172.21.2.102:0/469087757 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efeb4105b30 con 0x7efeb40b8410 2021-02-10T11:10:03.168 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:03.172 7efeaaffd700 1 -- 172.21.2.102:0/469087757 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7efea4000940 con 0x7efeb40b8410 2021-02-10T11:10:03.294 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:03.297 7efeb8874700 1 -- 172.21.2.102:0/469087757 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344395", "target": ["mgr", ""]}) v1 -- 0x7efeb40c2f30 con 0x7efe980186c0 2021-02-10T11:10:03.297 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:03.301 7efeaaffd700 1 -- 172.21.2.102:0/469087757 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7efe9c009360 con 0x7efe980186c0 2021-02-10T11:10:03.301 INFO:teuthology.orchestra.run.gibba002.stdout:/volumes/_nogroup/subvolume_0000000000344395/62d68bd1-13ae-423b-bd87-7a3633520930 2021-02-10T11:10:03.303 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:03.306 7efeb8874700 1 -- 172.21.2.102:0/469087757 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7efe980186c0 msgr2=0x7efe9801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:10:03.303 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:03.306 7efeb8874700 1 --2- 172.21.2.102:0/469087757 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7efe980186c0 0x7efe9801abc0 crc :-1 s=READY pgs=235 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:10:03.303 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:03.306 7efeb8874700 1 -- 172.21.2.102:0/469087757 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efeb40b8410 msgr2=0x7efeb40b8810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:10:03.303 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:03.306 7efeb8874700 1 --2- 172.21.2.102:0/469087757 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efeb40b8410 0x7efeb40b8810 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 rx=0x7efea4001c00 tx=0x7efea400a8e0).stop 2021-02-10T11:10:03.303 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:03.306 7efeb8874700 1 -- 172.21.2.102:0/469087757 shutdown_connections 2021-02-10T11:10:03.303 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:03.306 7efeb8874700 1 --2- 172.21.2.102:0/469087757 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7efeb40c1f80 0x7efeb40c2380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:10:03.303 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:03.306 7efeb8874700 1 --2- 172.21.2.102:0/469087757 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efeb40b8410 0x7efeb40b8810 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:10:03.304 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:03.306 7efeb8874700 1 --2- 172.21.2.102:0/469087757 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7efe980186c0 0x7efe9801abc0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:10:03.304 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:03.306 7efeb8874700 1 --2- 172.21.2.102:0/469087757 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7efeb409b930 0x7efeb40b7c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:10:03.304 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:03.306 7efeb8874700 1 -- 172.21.2.102:0/469087757 >> 172.21.2.102:0/469087757 conn(0x7efeb4094e10 msgr2=0x7efeb40a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:10:03.304 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:03.306 7efeb8874700 1 -- 172.21.2.102:0/469087757 shutdown_connections 2021-02-10T11:10:03.304 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:03.306 7efeb8874700 1 -- 172.21.2.102:0/469087757 wait complete. 2021-02-10T11:10:03.311 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000344395 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000344395/62d68bd1-13ae-423b-bd87-7a3633520930 2021-02-10T11:10:03.311 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:10:03.311 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:10:03.311 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:10:03.646 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:10:03.647 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:10:03.647 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344395/62d68bd1-13ae-423b-bd87-7a3633520930/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:10:04.023 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:10:04.024 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:10:04.024 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0234078 s, 44.8 MB/s 2021-02-10T11:10:04.025 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:10:04.025 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:10:04.025 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:10:04.359 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:10:04.359 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:10:04.359 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344395/62d68bd1-13ae-423b-bd87-7a3633520930/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:10:04.735 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:10:04.735 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:10:04.735 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0246353 s, 42.6 MB/s 2021-02-10T11:10:04.736 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:10:04.736 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:10:04.736 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:10:05.072 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:10:05.073 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:10:05.073 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344395/62d68bd1-13ae-423b-bd87-7a3633520930/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:10:05.448 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:10:05.448 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:10:05.448 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0250576 s, 41.8 MB/s 2021-02-10T11:10:05.449 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:10:05.449 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:10:05.450 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:10:05.784 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:10:05.784 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:10:05.785 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344395/62d68bd1-13ae-423b-bd87-7a3633520930/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:10:06.161 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:10:06.162 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:10:06.162 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0250016 s, 41.9 MB/s 2021-02-10T11:10:06.163 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:10:06.163 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:10:06.163 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:10:06.498 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:10:06.499 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:10:06.499 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344395/62d68bd1-13ae-423b-bd87-7a3633520930/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:10:06.875 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:10:06.876 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:10:06.876 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.024711 s, 42.4 MB/s 2021-02-10T11:10:06.877 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:10:06.877 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:10:06.877 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:10:07.212 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:10:07.212 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:10:07.213 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344395/62d68bd1-13ae-423b-bd87-7a3633520930/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:10:07.585 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:10:07.585 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:10:07.585 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0225389 s, 46.5 MB/s 2021-02-10T11:10:07.586 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:10:07.586 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:10:07.587 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:10:07.922 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:10:07.922 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:10:07.923 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344395/62d68bd1-13ae-423b-bd87-7a3633520930/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:10:08.299 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:10:08.299 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:10:08.299 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0245024 s, 42.8 MB/s 2021-02-10T11:10:08.300 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:10:08.300 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:10:08.300 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:10:08.635 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:10:08.636 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:10:08.636 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344395/62d68bd1-13ae-423b-bd87-7a3633520930/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:10:09.009 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:10:09.009 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:10:09.009 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0255555 s, 41.0 MB/s 2021-02-10T11:10:09.010 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:10:09.010 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:10:09.010 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:10:09.346 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:10:09.347 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:10:09.347 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344395/62d68bd1-13ae-423b-bd87-7a3633520930/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:10:09.722 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:10:09.722 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:10:09.722 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.023051 s, 45.5 MB/s 2021-02-10T11:10:09.723 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:10:09.723 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:10:09.723 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:10:10.059 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:10:10.060 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:10:10.060 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344395/62d68bd1-13ae-423b-bd87-7a3633520930/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:10:10.438 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:10:10.438 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:10:10.438 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0270358 s, 38.8 MB/s 2021-02-10T11:10:10.439 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000344396 2021-02-10T11:10:10.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:10.858 7f7d6b152700 1 Processor -- start 2021-02-10T11:10:10.979 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:10.858 7f7d6b152700 1 -- start start 2021-02-10T11:10:10.979 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:10.858 7f7d6b152700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f7d641b0470 0x7f7d641b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:10:10.980 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:10.858 7f7d6b152700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7d640ef610 0x7f7d640efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:10:10.980 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:10.858 7f7d6b152700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f7d640f0030 0x7f7d640f0430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:10:10.980 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:10.858 7f7d6b152700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d640e5ea0 con 0x7f7d641b0470 2021-02-10T11:10:10.980 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:10.858 7f7d6b152700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d640f0b50 con 0x7f7d640f0030 2021-02-10T11:10:10.980 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:10.858 7f7d6b152700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d640e52e0 con 0x7f7d640ef610 2021-02-10T11:10:10.980 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:10.858 7f7d68f6e700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f7d641b0470 0x7f7d641b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:10:10.980 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:10.859 7f7d68f6e700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f7d641b0470 0x7f7d641b0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:51840/0 (socket says 172.21.2.102:51840) 2021-02-10T11:10:10.980 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:10.859 7f7d68f6e700 1 -- 172.21.2.102:0/930584115 learned_addr learned my addr 172.21.2.102:0/930584115 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:10:10.980 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:10.859 7f7d63fff700 1 --2- 172.21.2.102:0/930584115 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7d640ef610 0x7f7d640efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:10:10.980 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:10.859 7f7d6976f700 1 --2- 172.21.2.102:0/930584115 >> v2:172.21.2.113:3300/0 conn(0x7f7d640f0030 0x7f7d640f0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:10:10.980 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:10.859 7f7d68f6e700 1 -- 172.21.2.102:0/930584115 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7d640ef610 msgr2=0x7f7d640efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:10:10.980 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:10.859 7f7d68f6e700 1 --2- 172.21.2.102:0/930584115 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7d640ef610 0x7f7d640efa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:10:10.981 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:10.859 7f7d68f6e700 1 -- 172.21.2.102:0/930584115 >> v2:172.21.2.113:3300/0 conn(0x7f7d640f0030 msgr2=0x7f7d640f0430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:10:10.981 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:10.859 7f7d68f6e700 1 --2- 172.21.2.102:0/930584115 >> v2:172.21.2.113:3300/0 conn(0x7f7d640f0030 0x7f7d640f0430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:10:10.981 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:10.859 7f7d68f6e700 1 -- 172.21.2.102:0/930584115 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d640f10d0 con 0x7f7d641b0470 2021-02-10T11:10:10.981 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:10.859 7f7d68f6e700 1 --2- 172.21.2.102:0/930584115 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7d641b0470 0x7f7d641b0870 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 rx=0x7f7d4c006700 tx=0x7f7d4c009b80).ready entity=mon.0 client_cookie=247906ac7968246 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:10:10.981 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:10.860 7f7d637fe700 1 -- 172.21.2.102:0/930584115 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7d4c00c0c0 con 0x7f7d641b0470 2021-02-10T11:10:10.981 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:10.860 7f7d637fe700 1 -- 172.21.2.102:0/930584115 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d4c00c400 con 0x7f7d641b0470 2021-02-10T11:10:10.981 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:10.860 7f7d637fe700 1 -- 172.21.2.102:0/930584115 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7d4c00c920 con 0x7f7d641b0470 2021-02-10T11:10:10.981 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:10.860 7f7d6b152700 1 -- 172.21.2.102:0/930584115 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7d641b0470 msgr2=0x7f7d641b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:10:10.981 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:10.860 7f7d6b152700 1 --2- 172.21.2.102:0/930584115 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7d641b0470 0x7f7d641b0870 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 rx=0x7f7d4c006700 tx=0x7f7d4c009b80).stop 2021-02-10T11:10:10.981 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:10.860 7f7d6b152700 1 -- 172.21.2.102:0/930584115 shutdown_connections 2021-02-10T11:10:10.981 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:10.860 7f7d6b152700 1 --2- 172.21.2.102:0/930584115 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7d640ef610 0x7f7d640efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:10:10.981 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:10.860 7f7d6b152700 1 --2- 172.21.2.102:0/930584115 >> v2:172.21.2.113:3300/0 conn(0x7f7d640f0030 0x7f7d640f0430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:10:10.981 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:10.860 7f7d6b152700 1 --2- 172.21.2.102:0/930584115 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7d641b0470 0x7f7d641b0870 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:10:10.982 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:10.860 7f7d6b152700 1 -- 172.21.2.102:0/930584115 >> 172.21.2.102:0/930584115 conn(0x7f7d640ce3e0 msgr2=0x7f7d640e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:10:10.982 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:10.860 7f7d6b152700 1 -- 172.21.2.102:0/930584115 shutdown_connections 2021-02-10T11:10:10.982 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:10.860 7f7d6b152700 1 -- 172.21.2.102:0/930584115 wait complete. 2021-02-10T11:10:10.984 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:10.875 7f7d6b152700 1 Processor -- start 2021-02-10T11:10:10.984 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:10.875 7f7d6b152700 1 -- start start 2021-02-10T11:10:10.984 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:10.875 7f7d6b152700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7d640b7db0 0x7f7d640b8260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:10:10.984 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:10.876 7f7d6b152700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7d640b8880 0x7f7d641a7450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:10:10.984 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:10.876 7f7d6b152700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7d640b8c80 0x7f7d641a7a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:10:10.984 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:10.876 7f7d6b152700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d640b79b0 con 0x7f7d640b8880 2021-02-10T11:10:10.984 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:10.876 7f7d6b152700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d641a8090 con 0x7f7d640b7db0 2021-02-10T11:10:10.984 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:10.876 7f7d6b152700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d641a82b0 con 0x7f7d640b8c80 2021-02-10T11:10:10.984 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:10.876 7f7d63fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7d640b8880 0x7f7d641a7450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:10:10.984 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:10.876 7f7d63fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7d640b8880 0x7f7d641a7450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:51846/0 (socket says 172.21.2.102:51846) 2021-02-10T11:10:10.984 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:10.876 7f7d63fff700 1 -- 172.21.2.102:0/475207351 learned_addr learned my addr 172.21.2.102:0/475207351 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:10:10.984 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:10.876 7f7d6976f700 1 --2- 172.21.2.102:0/475207351 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7d640b8c80 0x7f7d641a7a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:10:10.985 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:10.876 7f7d63fff700 1 -- 172.21.2.102:0/475207351 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7d640b8c80 msgr2=0x7f7d641a7a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:10:10.985 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:10.876 7f7d68f6e700 1 --2- 172.21.2.102:0/475207351 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7d640b7db0 0x7f7d640b8260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:10:10.985 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:10.876 7f7d63fff700 1 --2- 172.21.2.102:0/475207351 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7d640b8c80 0x7f7d641a7a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:10:10.985 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:10.876 7f7d63fff700 1 -- 172.21.2.102:0/475207351 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7d640b7db0 msgr2=0x7f7d640b8260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:10:10.985 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:10.876 7f7d63fff700 1 --2- 172.21.2.102:0/475207351 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7d640b7db0 0x7f7d640b8260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:10:10.985 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:10.876 7f7d63fff700 1 -- 172.21.2.102:0/475207351 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d64190c10 con 0x7f7d640b8880 2021-02-10T11:10:10.985 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:10.876 7f7d68f6e700 1 --2- 172.21.2.102:0/475207351 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7d640b7db0 0x7f7d640b8260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:10:10.985 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:10.876 7f7d63fff700 1 --2- 172.21.2.102:0/475207351 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7d640b8880 0x7f7d641a7450 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 rx=0x7f7d5c0099e0 tx=0x7f7d5c00bd40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:10:10.985 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:10.877 7f7d617fa700 1 -- 172.21.2.102:0/475207351 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7d5c00c2f0 con 0x7f7d640b8880 2021-02-10T11:10:10.985 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:10.877 7f7d617fa700 1 -- 172.21.2.102:0/475207351 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d5c00c5b0 con 0x7f7d640b8880 2021-02-10T11:10:10.985 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:10.877 7f7d617fa700 1 -- 172.21.2.102:0/475207351 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7d5c00ca10 con 0x7f7d640b8880 2021-02-10T11:10:10.985 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:10.877 7f7d6b152700 1 -- 172.21.2.102:0/475207351 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d64190c10 con 0x7f7d640b8880 2021-02-10T11:10:10.985 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:10.877 7f7d6b152700 1 -- 172.21.2.102:0/475207351 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d64190c10 con 0x7f7d640b8880 2021-02-10T11:10:10.986 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:10.877 7f7d617fa700 1 -- 172.21.2.102:0/475207351 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f7d5c005660 con 0x7f7d640b8880 2021-02-10T11:10:10.986 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:10.877 7f7d617fa700 1 --2- 172.21.2.102:0/475207351 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7d480186c0 0x7f7d4801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:10:10.986 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:10.877 7f7d617fa700 1 -- 172.21.2.102:0/475207351 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f7d5c0328e0 con 0x7f7d640b8880 2021-02-10T11:10:10.986 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:10.877 7f7d68f6e700 1 --2- 172.21.2.102:0/475207351 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7d480186c0 0x7f7d4801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:10:10.986 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:10.878 7f7d68f6e700 1 --2- 172.21.2.102:0/475207351 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7d480186c0 0x7f7d4801abc0 crc :-1 s=READY pgs=236 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:10:10.986 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:10.908 7f7d6b152700 1 -- 172.21.2.102:0/475207351 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d64105b30 con 0x7f7d640b8880 2021-02-10T11:10:10.986 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:10.911 7f7d617fa700 1 -- 172.21.2.102:0/475207351 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f7d5c00c2f0 con 0x7f7d640b8880 2021-02-10T11:10:11.036 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:11.040 7f7d6b152700 1 -- 172.21.2.102:0/475207351 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344396", "target": ["mgr", ""]}) v1 -- 0x7f7d641a8090 con 0x7f7d480186c0 2021-02-10T11:10:11.063 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:11.066 7f7d617fa700 1 -- 172.21.2.102:0/475207351 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f7d4c0009d0 con 0x7f7d480186c0 2021-02-10T11:10:11.069 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:11.073 7f7d6b152700 1 -- 172.21.2.102:0/475207351 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7d480186c0 msgr2=0x7f7d4801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:10:11.069 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:11.073 7f7d6b152700 1 --2- 172.21.2.102:0/475207351 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7d480186c0 0x7f7d4801abc0 crc :-1 s=READY pgs=236 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:10:11.069 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:11.073 7f7d6b152700 1 -- 172.21.2.102:0/475207351 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7d640b8880 msgr2=0x7f7d641a7450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:10:11.069 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:11.073 7f7d6b152700 1 --2- 172.21.2.102:0/475207351 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7d640b8880 0x7f7d641a7450 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 rx=0x7f7d5c0099e0 tx=0x7f7d5c00bd40).stop 2021-02-10T11:10:11.070 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:11.073 7f7d6b152700 1 -- 172.21.2.102:0/475207351 shutdown_connections 2021-02-10T11:10:11.070 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:11.073 7f7d6b152700 1 --2- 172.21.2.102:0/475207351 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7d640b8c80 0x7f7d641a7a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:10:11.070 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:11.073 7f7d6b152700 1 --2- 172.21.2.102:0/475207351 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7d640b8880 0x7f7d641a7450 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:10:11.070 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:11.073 7f7d6b152700 1 --2- 172.21.2.102:0/475207351 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7d480186c0 0x7f7d4801abc0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:10:11.070 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:11.073 7f7d6b152700 1 --2- 172.21.2.102:0/475207351 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7d640b7db0 0x7f7d640b8260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:10:11.070 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:11.073 7f7d6b152700 1 -- 172.21.2.102:0/475207351 >> 172.21.2.102:0/475207351 conn(0x7f7d64094e10 msgr2=0x7f7d640a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:10:11.070 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:11.073 7f7d6b152700 1 -- 172.21.2.102:0/475207351 shutdown_connections 2021-02-10T11:10:11.070 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:11.073 7f7d6b152700 1 -- 172.21.2.102:0/475207351 wait complete. 2021-02-10T11:10:11.078 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000344396 2021-02-10T11:10:11.508 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:11.495 7f17987f9700 1 Processor -- start 2021-02-10T11:10:11.508 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:11.495 7f17987f9700 1 -- start start 2021-02-10T11:10:11.508 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:11.496 7f17987f9700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f17901b0470 0x7f17901b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:10:11.508 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:11.496 7f17987f9700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f17900f20e0 0x7f17900f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:10:11.508 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:11.496 7f17987f9700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17900f0b50 con 0x7f17900ef610 2021-02-10T11:10:11.508 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:11.496 7f17987f9700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17900f0820 con 0x7f17900f20e0 2021-02-10T11:10:11.508 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:11.496 7f17987f9700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17900f24e0 con 0x7f17901b0470 2021-02-10T11:10:11.508 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:11.496 7f1796e16700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f17900f20e0 0x7f17900f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:10:11.508 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:11.496 7f1796e16700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f17900f20e0 0x7f17900f0200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:51854/0 (socket says 172.21.2.102:51854) 2021-02-10T11:10:11.508 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:11.496 7f1796e16700 1 -- 172.21.2.102:0/2068877611 learned_addr learned my addr 172.21.2.102:0/2068877611 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:10:11.509 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:11.496 7f1796e16700 1 -- 172.21.2.102:0/2068877611 >> v2:172.21.2.113:3300/0 conn(0x7f17901b0470 msgr2=0x7f17901b0870 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:10:11.509 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:11.496 7f1796e16700 1 --2- 172.21.2.102:0/2068877611 >> v2:172.21.2.113:3300/0 conn(0x7f17901b0470 0x7f17901b0870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:10:11.509 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:11.496 7f1796e16700 1 -- 172.21.2.102:0/2068877611 >> v1:172.21.2.113:6789/0 conn(0x7f17900ef610 legacy=0x7f17900efa10 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:10:11.509 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:11.496 7f1796e16700 1 -- 172.21.2.102:0/2068877611 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17900e52e0 con 0x7f17900f20e0 2021-02-10T11:10:11.509 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:11.496 7f1796e16700 1 --2- 172.21.2.102:0/2068877611 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f17900f20e0 0x7f17900f0200 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 rx=0x7f1788006f00 tx=0x7f1788009f10).ready entity=mon.0 client_cookie=b3082ec1f04d8ee9 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:10:11.509 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:11.496 7f1795613700 1 -- 172.21.2.102:0/2068877611 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f178800b3b0 con 0x7f17900f20e0 2021-02-10T11:10:11.509 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:11.497 7f1795613700 1 -- 172.21.2.102:0/2068877611 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f178800b630 con 0x7f17900f20e0 2021-02-10T11:10:11.509 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:11.497 7f1795613700 1 -- 172.21.2.102:0/2068877611 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f178800bb50 con 0x7f17900f20e0 2021-02-10T11:10:11.509 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:11.497 7f17987f9700 1 -- 172.21.2.102:0/2068877611 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f17900f20e0 msgr2=0x7f17900f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:10:11.509 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:11.497 7f17987f9700 1 --2- 172.21.2.102:0/2068877611 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f17900f20e0 0x7f17900f0200 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 rx=0x7f1788006f00 tx=0x7f1788009f10).stop 2021-02-10T11:10:11.509 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:11.497 7f17987f9700 1 -- 172.21.2.102:0/2068877611 shutdown_connections 2021-02-10T11:10:11.509 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:11.497 7f17987f9700 1 --2- 172.21.2.102:0/2068877611 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f17900f20e0 0x7f17900f0200 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:10:11.510 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:11.497 7f17987f9700 1 --2- 172.21.2.102:0/2068877611 >> v2:172.21.2.113:3300/0 conn(0x7f17901b0470 0x7f17901b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:10:11.510 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:11.497 7f17987f9700 1 -- 172.21.2.102:0/2068877611 >> 172.21.2.102:0/2068877611 conn(0x7f17900ce3e0 msgr2=0x7f17900e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:10:11.510 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:11.497 7f17987f9700 1 -- 172.21.2.102:0/2068877611 shutdown_connections 2021-02-10T11:10:11.510 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:11.497 7f17987f9700 1 -- 172.21.2.102:0/2068877611 wait complete. 2021-02-10T11:10:11.511 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:11.512 7f17987f9700 1 Processor -- start 2021-02-10T11:10:11.511 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:11.512 7f17987f9700 1 -- start start 2021-02-10T11:10:11.511 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:11.512 7f17987f9700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f17900a3950 0x7f17900a3d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:10:11.511 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:11.512 7f17987f9700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f17900a4480 0x7f17900a4880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:10:11.511 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:11.512 7f17987f9700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f17900a4ea0 0x7f17900beb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:10:11.512 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:11.512 7f17987f9700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17900a5320 con 0x7f17900a4480 2021-02-10T11:10:11.512 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:11.512 7f17987f9700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17900bf190 con 0x7f17900a3950 2021-02-10T11:10:11.512 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:11.512 7f17987f9700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17900bf3b0 con 0x7f17900a4ea0 2021-02-10T11:10:11.512 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:11.512 7f1795e14700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f17900a4480 0x7f17900a4880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:10:11.512 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:11.512 7f1795e14700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f17900a4480 0x7f17900a4880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:51858/0 (socket says 172.21.2.102:51858) 2021-02-10T11:10:11.512 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:11.512 7f1795e14700 1 -- 172.21.2.102:0/3493929734 learned_addr learned my addr 172.21.2.102:0/3493929734 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:10:11.512 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:11.513 7f1795e14700 1 -- 172.21.2.102:0/3493929734 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f17900a4ea0 msgr2=0x7f17900beb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:10:11.512 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:11.513 7f1795e14700 1 --2- 172.21.2.102:0/3493929734 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f17900a4ea0 0x7f17900beb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:10:11.512 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:11.513 7f1795e14700 1 -- 172.21.2.102:0/3493929734 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f17900a3950 msgr2=0x7f17900a3d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:10:11.512 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:11.513 7f1795e14700 1 --2- 172.21.2.102:0/3493929734 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f17900a3950 0x7f17900a3d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:10:11.512 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:11.513 7f1795e14700 1 -- 172.21.2.102:0/3493929734 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17900bf5d0 con 0x7f17900a4480 2021-02-10T11:10:11.512 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:11.513 7f1795e14700 1 --2- 172.21.2.102:0/3493929734 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f17900a4480 0x7f17900a4880 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 rx=0x7f178c00b5b0 tx=0x7f178c00b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:10:11.513 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:11.513 7f1782ffd700 1 -- 172.21.2.102:0/3493929734 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f178c00bd40 con 0x7f17900a4480 2021-02-10T11:10:11.513 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:11.513 7f1782ffd700 1 -- 172.21.2.102:0/3493929734 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f178c006240 con 0x7f17900a4480 2021-02-10T11:10:11.513 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:11.513 7f1782ffd700 1 -- 172.21.2.102:0/3493929734 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f178c005720 con 0x7f17900a4480 2021-02-10T11:10:11.513 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:11.513 7f17987f9700 1 -- 172.21.2.102:0/3493929734 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17900bf5d0 con 0x7f17900a4480 2021-02-10T11:10:11.513 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:11.513 7f17987f9700 1 -- 172.21.2.102:0/3493929734 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17900bf5d0 con 0x7f17900a4480 2021-02-10T11:10:11.513 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:11.514 7f1782ffd700 1 -- 172.21.2.102:0/3493929734 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f178c0054a0 con 0x7f17900a4480 2021-02-10T11:10:11.513 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:11.514 7f1782ffd700 1 --2- 172.21.2.102:0/3493929734 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f17780186c0 0x7f177801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:10:11.513 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:11.514 7f1782ffd700 1 -- 172.21.2.102:0/3493929734 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f178c032550 con 0x7f17900a4480 2021-02-10T11:10:11.513 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:11.514 7f1796615700 1 --2- 172.21.2.102:0/3493929734 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f17780186c0 0x7f177801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:10:11.513 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:11.514 7f1796615700 1 --2- 172.21.2.102:0/3493929734 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f17780186c0 0x7f177801abc0 crc :-1 s=READY pgs=237 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:10:11.542 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:11.546 7f17987f9700 1 -- 172.21.2.102:0/3493929734 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17900e52e0 con 0x7f17900a4480 2021-02-10T11:10:11.545 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:11.548 7f1782ffd700 1 -- 172.21.2.102:0/3493929734 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f178c000c60 con 0x7f17900a4480 2021-02-10T11:10:11.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:11.677 7f17987f9700 1 -- 172.21.2.102:0/3493929734 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344396", "target": ["mgr", ""]}) v1 -- 0x7f17900e52e0 con 0x7f17780186c0 2021-02-10T11:10:11.676 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:11.680 7f1782ffd700 1 -- 172.21.2.102:0/3493929734 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f1784006e40 con 0x7f17780186c0 2021-02-10T11:10:11.681 INFO:teuthology.orchestra.run.gibba002.stdout:/volumes/_nogroup/subvolume_0000000000344396/ef4e9e9f-8499-45ae-9614-b513148d5470 2021-02-10T11:10:11.682 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:11.686 7f17987f9700 1 -- 172.21.2.102:0/3493929734 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f17780186c0 msgr2=0x7f177801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:10:11.682 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:11.686 7f17987f9700 1 --2- 172.21.2.102:0/3493929734 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f17780186c0 0x7f177801abc0 crc :-1 s=READY pgs=237 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:10:11.682 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:11.686 7f17987f9700 1 -- 172.21.2.102:0/3493929734 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f17900a4480 msgr2=0x7f17900a4880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:10:11.682 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:11.686 7f17987f9700 1 --2- 172.21.2.102:0/3493929734 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f17900a4480 0x7f17900a4880 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 rx=0x7f178c00b5b0 tx=0x7f178c00b950).stop 2021-02-10T11:10:11.683 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:11.686 7f17987f9700 1 -- 172.21.2.102:0/3493929734 shutdown_connections 2021-02-10T11:10:11.683 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:11.686 7f17987f9700 1 --2- 172.21.2.102:0/3493929734 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f17900a4ea0 0x7f17900beb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:10:11.683 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:11.686 7f17987f9700 1 --2- 172.21.2.102:0/3493929734 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f17900a4480 0x7f17900a4880 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:10:11.683 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:11.686 7f17987f9700 1 --2- 172.21.2.102:0/3493929734 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f17780186c0 0x7f177801abc0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:10:11.683 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:11.686 7f17987f9700 1 --2- 172.21.2.102:0/3493929734 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f17900a3950 0x7f17900a3d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:10:11.683 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:11.686 7f17987f9700 1 -- 172.21.2.102:0/3493929734 >> 172.21.2.102:0/3493929734 conn(0x7f1790091b40 msgr2=0x7f1790097140 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:10:11.683 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:11.686 7f17987f9700 1 -- 172.21.2.102:0/3493929734 shutdown_connections 2021-02-10T11:10:11.683 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:11.686 7f17987f9700 1 -- 172.21.2.102:0/3493929734 wait complete. 2021-02-10T11:10:11.690 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000344396 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000344396/ef4e9e9f-8499-45ae-9614-b513148d5470 2021-02-10T11:10:11.691 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:10:11.691 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:10:11.691 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:10:12.026 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:10:12.026 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:10:12.027 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344396/ef4e9e9f-8499-45ae-9614-b513148d5470/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:10:12.408 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:10:12.408 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:10:12.408 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0233725 s, 44.9 MB/s 2021-02-10T11:10:12.409 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:10:12.409 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:10:12.410 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:10:12.743 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:10:12.744 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:10:12.744 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344396/ef4e9e9f-8499-45ae-9614-b513148d5470/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:10:13.120 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:10:13.120 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:10:13.121 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.025027 s, 41.9 MB/s 2021-02-10T11:10:13.121 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:10:13.122 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:10:13.122 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:10:13.471 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:10:13.471 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:10:13.472 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344396/ef4e9e9f-8499-45ae-9614-b513148d5470/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:10:13.853 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:10:13.853 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:10:13.853 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0244914 s, 42.8 MB/s 2021-02-10T11:10:13.854 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:10:13.854 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:10:13.854 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:10:14.200 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:10:14.201 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:10:14.201 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344396/ef4e9e9f-8499-45ae-9614-b513148d5470/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:10:14.579 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:10:14.580 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:10:14.580 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0252101 s, 41.6 MB/s 2021-02-10T11:10:14.581 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:10:14.581 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:10:14.581 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:10:14.921 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:10:14.921 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:10:14.921 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344396/ef4e9e9f-8499-45ae-9614-b513148d5470/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:10:15.300 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:10:15.300 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:10:15.300 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0246997 s, 42.5 MB/s 2021-02-10T11:10:15.301 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:10:15.301 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:10:15.301 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:10:15.636 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:10:15.636 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:10:15.636 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344396/ef4e9e9f-8499-45ae-9614-b513148d5470/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:10:15.733 DEBUG:teuthology.orchestra.run.gibba002:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:10:15.736 DEBUG:teuthology.orchestra.run.gibba013:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:10:16.013 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:10:16.013 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:10:16.013 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0261067 s, 40.2 MB/s 2021-02-10T11:10:16.014 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:10:16.015 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:10:16.015 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:10:16.389 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:10:16.389 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:10:16.389 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344396/ef4e9e9f-8499-45ae-9614-b513148d5470/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:10:16.764 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:10:16.764 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:10:16.764 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0250164 s, 41.9 MB/s 2021-02-10T11:10:16.765 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:10:16.765 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:10:16.766 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:10:17.101 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:10:17.101 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:10:17.102 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344396/ef4e9e9f-8499-45ae-9614-b513148d5470/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:10:17.476 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:10:17.477 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:10:17.477 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0226058 s, 46.4 MB/s 2021-02-10T11:10:17.477 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:10:17.478 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:10:17.478 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:10:17.813 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:10:17.814 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:10:17.814 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344396/ef4e9e9f-8499-45ae-9614-b513148d5470/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:10:18.189 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:10:18.190 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:10:18.190 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0239203 s, 43.8 MB/s 2021-02-10T11:10:18.191 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:10:18.191 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:10:18.191 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:10:18.525 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:10:18.525 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:10:18.526 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344396/ef4e9e9f-8499-45ae-9614-b513148d5470/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:10:18.900 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:10:18.901 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:10:18.901 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0260432 s, 40.3 MB/s 2021-02-10T11:10:18.902 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000344397 2021-02-10T11:10:19.332 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:19.318 7f22545b4700 1 Processor -- start 2021-02-10T11:10:19.332 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:19.319 7f22545b4700 1 -- start start 2021-02-10T11:10:19.332 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:19.319 7f22545b4700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f224c1b0470 0x7f224c1b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:10:19.333 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:19.319 7f22545b4700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f224c0f0b50 con 0x7f224c0f20e0 2021-02-10T11:10:19.333 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:19.319 7f22545b4700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f224c0f24e0 con 0x7f224c0ef610 2021-02-10T11:10:19.333 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:19.319 7f22545b4700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f224c0e5ea0 con 0x7f224c1b0470 2021-02-10T11:10:19.333 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:19.319 7f2251bcf700 1 --1- >> v1:172.21.2.113:6789/0 conn(0x7f224c0ef610 0x7f224c0efa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.113:6789/0 says I am v1:172.21.2.102:51416/0 (socket says 172.21.2.102:51416) 2021-02-10T11:10:19.333 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:19.319 7f2251bcf700 1 -- 172.21.2.102:0/2714914542 learned_addr learned my addr 172.21.2.102:0/2714914542 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:10:19.333 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:19.319 7f2252bd1700 1 --1- 172.21.2.102:0/2714914542 >> v1:172.21.2.102:6789/0 conn(0x7f224c0f20e0 0x7f224c0f0200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:44094/0 (socket says 172.21.2.102:44094) 2021-02-10T11:10:19.333 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:19.319 7f22523d0700 1 --2- 172.21.2.102:0/2714914542 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f224c1b0470 0x7f224c1b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:10:19.333 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:19.320 7f22513ce700 1 -- 172.21.2.102:0/2714914542 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3742931836 0 0) 0x7f2248000c10 con 0x7f224c0f20e0 2021-02-10T11:10:19.333 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:19.320 7f22513ce700 1 -- 172.21.2.102:0/2714914542 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2238002fe0 con 0x7f224c0f20e0 2021-02-10T11:10:19.333 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:19.320 7f22513ce700 1 -- 172.21.2.102:0/2714914542 <== mon.1 v1:172.21.2.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3976722001 0 0) 0x7f223c0009b0 con 0x7f224c0ef610 2021-02-10T11:10:19.333 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:19.320 7f22513ce700 1 -- 172.21.2.102:0/2714914542 --> v1:172.21.2.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2238004370 con 0x7f224c0ef610 2021-02-10T11:10:19.333 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:19.320 7f22523d0700 1 -- 172.21.2.102:0/2714914542 >> v1:172.21.2.113:6789/0 conn(0x7f224c0ef610 legacy=0x7f224c0efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:10:19.334 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:19.320 7f22523d0700 1 -- 172.21.2.102:0/2714914542 >> v1:172.21.2.102:6789/0 conn(0x7f224c0f20e0 legacy=0x7f224c0f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:10:19.334 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:19.320 7f22523d0700 1 -- 172.21.2.102:0/2714914542 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f224c0e52e0 con 0x7f224c1b0470 2021-02-10T11:10:19.334 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:19.320 7f22523d0700 1 --2- 172.21.2.102:0/2714914542 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f224c1b0470 0x7f224c1b0870 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f22440069d0 tx=0x7f2244009e40).ready entity=mon.2 client_cookie=18353242381b79c server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:10:19.334 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:19.320 7f22513ce700 1 -- 172.21.2.102:0/2714914542 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f224400b510 con 0x7f224c1b0470 2021-02-10T11:10:19.334 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:19.320 7f22513ce700 1 -- 172.21.2.102:0/2714914542 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f224400b790 con 0x7f224c1b0470 2021-02-10T11:10:19.334 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:19.320 7f22513ce700 1 -- 172.21.2.102:0/2714914542 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f224400bcb0 con 0x7f224c1b0470 2021-02-10T11:10:19.334 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:19.320 7f22545b4700 1 -- 172.21.2.102:0/2714914542 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f224c1b0470 msgr2=0x7f224c1b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:10:19.334 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:19.320 7f22545b4700 1 --2- 172.21.2.102:0/2714914542 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f224c1b0470 0x7f224c1b0870 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f22440069d0 tx=0x7f2244009e40).stop 2021-02-10T11:10:19.334 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:19.321 7f22545b4700 1 -- 172.21.2.102:0/2714914542 shutdown_connections 2021-02-10T11:10:19.334 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:19.321 7f22545b4700 1 --2- 172.21.2.102:0/2714914542 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f224c1b0470 0x7f224c1b0870 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:10:19.334 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:19.321 7f22545b4700 1 -- 172.21.2.102:0/2714914542 >> 172.21.2.102:0/2714914542 conn(0x7f224c0ce3e0 msgr2=0x7f224c0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:10:19.334 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:19.321 7f22545b4700 1 -- 172.21.2.102:0/2714914542 shutdown_connections 2021-02-10T11:10:19.335 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:19.321 7f22545b4700 1 -- 172.21.2.102:0/2714914542 wait complete. 2021-02-10T11:10:19.335 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:19.336 7f22545b4700 1 Processor -- start 2021-02-10T11:10:19.335 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:19.336 7f22545b4700 1 -- start start 2021-02-10T11:10:19.336 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:19.336 7f22545b4700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f224c0c28d0 0x7f224c0c2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:10:19.336 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:19.336 7f22545b4700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f224c08e690 0x7f224c08ea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:10:19.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:19.336 7f22545b4700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f224c0bc380 0x7f224c0bc780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:10:19.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:19.336 7f22545b4700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f224c08f340 con 0x7f224c0bc380 2021-02-10T11:10:19.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:19.336 7f22545b4700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f224c08f0b0 con 0x7f224c0c28d0 2021-02-10T11:10:19.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:19.336 7f22545b4700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f224c0c3400 con 0x7f224c08e690 2021-02-10T11:10:19.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:19.337 7f2252bd1700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f224c0bc380 0x7f224c0bc780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:10:19.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:19.337 7f2252bd1700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f224c0bc380 0x7f224c0bc780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:51872/0 (socket says 172.21.2.102:51872) 2021-02-10T11:10:19.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:19.337 7f2252bd1700 1 -- 172.21.2.102:0/2470048392 learned_addr learned my addr 172.21.2.102:0/2470048392 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:10:19.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:19.337 7f2252bd1700 1 -- 172.21.2.102:0/2470048392 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f224c08e690 msgr2=0x7f224c08ea90 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:10:19.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:19.337 7f2252bd1700 1 --2- 172.21.2.102:0/2470048392 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f224c08e690 0x7f224c08ea90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:10:19.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:19.337 7f2252bd1700 1 -- 172.21.2.102:0/2470048392 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f224c0c28d0 msgr2=0x7f224c0c2cd0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:10:19.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:19.337 7f2252bd1700 1 --2- 172.21.2.102:0/2470048392 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f224c0c28d0 0x7f224c0c2cd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:10:19.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:19.337 7f2252bd1700 1 -- 172.21.2.102:0/2470048392 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f224c0bcda0 con 0x7f224c0bc380 2021-02-10T11:10:19.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:19.337 7f2252bd1700 1 --2- 172.21.2.102:0/2470048392 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f224c0bc380 0x7f224c0bc780 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 rx=0x7f2248006f10 tx=0x7f224800dac0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:10:19.338 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:19.337 7f2242ffd700 1 -- 172.21.2.102:0/2470048392 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f22480043a0 con 0x7f224c0bc380 2021-02-10T11:10:19.338 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:19.337 7f2242ffd700 1 -- 172.21.2.102:0/2470048392 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2248004620 con 0x7f224c0bc380 2021-02-10T11:10:19.338 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:19.337 7f2242ffd700 1 -- 172.21.2.102:0/2470048392 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2248004b40 con 0x7f224c0bc380 2021-02-10T11:10:19.338 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:19.337 7f22545b4700 1 -- 172.21.2.102:0/2470048392 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f224c0bcda0 con 0x7f224c0bc380 2021-02-10T11:10:19.338 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:19.337 7f22545b4700 1 -- 172.21.2.102:0/2470048392 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f224c0bcda0 con 0x7f224c0bc380 2021-02-10T11:10:19.338 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:19.338 7f2242ffd700 1 -- 172.21.2.102:0/2470048392 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f22480009e0 con 0x7f224c0bc380 2021-02-10T11:10:19.338 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:19.338 7f2242ffd700 1 --2- 172.21.2.102:0/2470048392 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f22340186c0 0x7f223401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:10:19.338 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:19.338 7f2242ffd700 1 -- 172.21.2.102:0/2470048392 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f2248036900 con 0x7f224c0bc380 2021-02-10T11:10:19.338 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:19.338 7f22523d0700 1 --2- 172.21.2.102:0/2470048392 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f22340186c0 0x7f223401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:10:19.338 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:19.338 7f22523d0700 1 --2- 172.21.2.102:0/2470048392 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f22340186c0 0x7f223401abc0 crc :-1 s=READY pgs=238 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:10:19.365 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:19.369 7f22545b4700 1 -- 172.21.2.102:0/2470048392 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f224c105ae0 con 0x7f224c0bc380 2021-02-10T11:10:19.368 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:19.372 7f2242ffd700 1 -- 172.21.2.102:0/2470048392 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f224802aa20 con 0x7f224c0bc380 2021-02-10T11:10:19.495 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:19.498 7f22545b4700 1 -- 172.21.2.102:0/2470048392 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344397", "target": ["mgr", ""]}) v1 -- 0x7f224c08f0b0 con 0x7f22340186c0 2021-02-10T11:10:19.522 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:19.526 7f2242ffd700 1 -- 172.21.2.102:0/2470048392 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f22440093d0 con 0x7f22340186c0 2021-02-10T11:10:19.528 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:19.532 7f22545b4700 1 -- 172.21.2.102:0/2470048392 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f22340186c0 msgr2=0x7f223401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:10:19.528 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:19.532 7f22545b4700 1 --2- 172.21.2.102:0/2470048392 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f22340186c0 0x7f223401abc0 crc :-1 s=READY pgs=238 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:10:19.529 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:19.532 7f22545b4700 1 -- 172.21.2.102:0/2470048392 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f224c0bc380 msgr2=0x7f224c0bc780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:10:19.529 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:19.532 7f22545b4700 1 --2- 172.21.2.102:0/2470048392 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f224c0bc380 0x7f224c0bc780 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 rx=0x7f2248006f10 tx=0x7f224800dac0).stop 2021-02-10T11:10:19.529 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:19.532 7f22545b4700 1 -- 172.21.2.102:0/2470048392 shutdown_connections 2021-02-10T11:10:19.529 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:19.532 7f22545b4700 1 --2- 172.21.2.102:0/2470048392 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f224c0bc380 0x7f224c0bc780 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:10:19.529 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:19.532 7f22545b4700 1 --2- 172.21.2.102:0/2470048392 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f224c08e690 0x7f224c08ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:10:19.529 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:19.532 7f22545b4700 1 --2- 172.21.2.102:0/2470048392 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f22340186c0 0x7f223401abc0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:10:19.529 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:19.532 7f22545b4700 1 --2- 172.21.2.102:0/2470048392 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f224c0c28d0 0x7f224c0c2cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:10:19.529 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:19.532 7f22545b4700 1 -- 172.21.2.102:0/2470048392 >> 172.21.2.102:0/2470048392 conn(0x7f224c0bbf60 msgr2=0x7f224c105d10 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:10:19.529 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:19.532 7f22545b4700 1 -- 172.21.2.102:0/2470048392 shutdown_connections 2021-02-10T11:10:19.529 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:19.532 7f22545b4700 1 -- 172.21.2.102:0/2470048392 wait complete. 2021-02-10T11:10:19.536 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000344397 2021-02-10T11:10:19.968 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:19.954 7f902bf89700 1 Processor -- start 2021-02-10T11:10:19.968 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:19.954 7f902bf89700 1 -- start start 2021-02-10T11:10:19.968 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:19.954 7f902bf89700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f90241b0870 0x7f90240f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:10:19.968 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:19.954 7f902bf89700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f90240efe00 0x7f90240f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:10:19.968 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:19.954 7f902bf89700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90241b0c70 con 0x7f90241b0470 2021-02-10T11:10:19.968 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:19.954 7f902bf89700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90240f0b50 con 0x7f90240efe00 2021-02-10T11:10:19.969 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:19.954 7f902bf89700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90240f0820 con 0x7f90241b0870 2021-02-10T11:10:19.969 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:19.955 7f90295a4700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f90241b0870 0x7f90240f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:10:19.969 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:19.955 7f902a5a6700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f90240efe00 0x7f90240f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:10:19.969 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:19.955 7f90295a4700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f90241b0870 0x7f90240f20e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.113:3301/0 says I am v2:172.21.2.102:57416/0 (socket says 172.21.2.102:57416) 2021-02-10T11:10:19.969 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:19.955 7f90295a4700 1 -- 172.21.2.102:0/1215994461 learned_addr learned my addr 172.21.2.102:0/1215994461 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:10:19.969 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:19.956 7f90295a4700 1 -- 172.21.2.102:0/1215994461 >> v2:172.21.2.113:3300/0 conn(0x7f90240efe00 msgr2=0x7f90240f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:10:19.969 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:19.956 7f90295a4700 1 --2- 172.21.2.102:0/1215994461 >> v2:172.21.2.113:3300/0 conn(0x7f90240efe00 0x7f90240f0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:10:19.969 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:19.956 7f90295a4700 1 -- 172.21.2.102:0/1215994461 >> v1:172.21.2.113:6789/0 conn(0x7f90241b0470 legacy=0x7f90240ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:10:19.969 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:19.956 7f90295a4700 1 -- 172.21.2.102:0/1215994461 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90240e52e0 con 0x7f90241b0870 2021-02-10T11:10:19.969 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:19.956 7f902a5a6700 1 --2- 172.21.2.102:0/1215994461 >> v2:172.21.2.113:3300/0 conn(0x7f90240efe00 0x7f90240f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:10:19.969 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:19.956 7f90295a4700 1 --2- 172.21.2.102:0/1215994461 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f90241b0870 0x7f90240f20e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7f9014006700 tx=0x7f9014009b80).ready entity=mon.2 client_cookie=f31d8d66e5ea6049 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:10:19.969 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:19.957 7f9028da3700 1 -- 172.21.2.102:0/1215994461 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f901400c0c0 con 0x7f90241b0870 2021-02-10T11:10:19.969 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:19.957 7f9028da3700 1 -- 172.21.2.102:0/1215994461 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f901400c440 con 0x7f90241b0870 2021-02-10T11:10:19.970 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:19.957 7f9028da3700 1 -- 172.21.2.102:0/1215994461 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f901400c960 con 0x7f90241b0870 2021-02-10T11:10:19.970 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:19.957 7f902bf89700 1 -- 172.21.2.102:0/1215994461 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f90241b0870 msgr2=0x7f90240f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:10:19.970 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:19.957 7f902bf89700 1 --2- 172.21.2.102:0/1215994461 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f90241b0870 0x7f90240f20e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7f9014006700 tx=0x7f9014009b80).stop 2021-02-10T11:10:19.970 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:19.957 7f902bf89700 1 -- 172.21.2.102:0/1215994461 shutdown_connections 2021-02-10T11:10:19.970 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:19.957 7f902bf89700 1 --2- 172.21.2.102:0/1215994461 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f90241b0870 0x7f90240f20e0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:10:19.970 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:19.957 7f902bf89700 1 --2- 172.21.2.102:0/1215994461 >> v2:172.21.2.113:3300/0 conn(0x7f90240efe00 0x7f90240f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:10:19.970 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:19.957 7f902bf89700 1 -- 172.21.2.102:0/1215994461 >> 172.21.2.102:0/1215994461 conn(0x7f90240ce3e0 msgr2=0x7f90240e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:10:19.970 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:19.958 7f902bf89700 1 -- 172.21.2.102:0/1215994461 shutdown_connections 2021-02-10T11:10:19.970 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:19.958 7f902bf89700 1 -- 172.21.2.102:0/1215994461 wait complete. 2021-02-10T11:10:19.972 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:19.972 7f902bf89700 1 Processor -- start 2021-02-10T11:10:19.972 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:19.972 7f902bf89700 1 -- start start 2021-02-10T11:10:19.972 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:19.973 7f902bf89700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f90240a0420 0x7f90240a0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:10:19.972 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:19.973 7f902bf89700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9024189d00 0x7f902418a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:10:19.972 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:19.973 7f902bf89700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f902418a720 0x7f90240b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:10:19.972 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:19.973 7f902bf89700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9024189780 con 0x7f9024189d00 2021-02-10T11:10:19.972 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:19.973 7f902bf89700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f902418ab20 con 0x7f902418a720 2021-02-10T11:10:19.972 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:19.973 7f902bf89700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90240a1140 con 0x7f90240a0420 2021-02-10T11:10:19.972 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:19.973 7f90295a4700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9024189d00 0x7f902418a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:10:19.972 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:19.973 7f90295a4700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9024189d00 0x7f902418a100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:51890/0 (socket says 172.21.2.102:51890) 2021-02-10T11:10:19.973 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:19.973 7f90295a4700 1 -- 172.21.2.102:0/1379965879 learned_addr learned my addr 172.21.2.102:0/1379965879 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:10:19.973 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:19.973 7f9029da5700 1 --2- 172.21.2.102:0/1379965879 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f90240a0420 0x7f90240a0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:10:19.973 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:19.973 7f902a5a6700 1 --2- 172.21.2.102:0/1379965879 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f902418a720 0x7f90240b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:10:19.973 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:19.973 7f90295a4700 1 -- 172.21.2.102:0/1379965879 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f90240a0420 msgr2=0x7f90240a0820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:10:19.973 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:19.973 7f90295a4700 1 --2- 172.21.2.102:0/1379965879 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f90240a0420 0x7f90240a0820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:10:19.973 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:19.973 7f90295a4700 1 -- 172.21.2.102:0/1379965879 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f902418a720 msgr2=0x7f90240b7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:10:19.973 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:19.973 7f90295a4700 1 --2- 172.21.2.102:0/1379965879 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f902418a720 0x7f90240b7be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:10:19.973 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:19.973 7f90295a4700 1 -- 172.21.2.102:0/1379965879 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90240b8200 con 0x7f9024189d00 2021-02-10T11:10:19.973 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:19.974 7f90295a4700 1 --2- 172.21.2.102:0/1379965879 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9024189d00 0x7f902418a100 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 rx=0x7f901400b5e0 tx=0x7f901400b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:10:19.973 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:19.974 7f901a7fc700 1 -- 172.21.2.102:0/1379965879 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f901400bd70 con 0x7f9024189d00 2021-02-10T11:10:19.973 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:19.974 7f901a7fc700 1 -- 172.21.2.102:0/1379965879 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9014002030 con 0x7f9024189d00 2021-02-10T11:10:19.973 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:19.974 7f901a7fc700 1 -- 172.21.2.102:0/1379965879 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9014002550 con 0x7f9024189d00 2021-02-10T11:10:19.973 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:19.974 7f902bf89700 1 -- 172.21.2.102:0/1379965879 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90240b8200 con 0x7f9024189d00 2021-02-10T11:10:19.973 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:19.974 7f902bf89700 1 -- 172.21.2.102:0/1379965879 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90240b8200 con 0x7f9024189d00 2021-02-10T11:10:19.974 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:19.975 7f901a7fc700 1 -- 172.21.2.102:0/1379965879 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f9014002210 con 0x7f9024189d00 2021-02-10T11:10:19.974 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:19.975 7f901a7fc700 1 --2- 172.21.2.102:0/1379965879 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f900c0186c0 0x7f900c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:10:19.974 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:19.975 7f901a7fc700 1 -- 172.21.2.102:0/1379965879 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f90140323e0 con 0x7f9024189d00 2021-02-10T11:10:19.974 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:19.975 7f9029da5700 1 --2- 172.21.2.102:0/1379965879 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f900c0186c0 0x7f900c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:10:19.974 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:19.975 7f9029da5700 1 --2- 172.21.2.102:0/1379965879 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f900c0186c0 0x7f900c01abc0 crc :-1 s=READY pgs=239 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:10:20.001 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:20.005 7f902bf89700 1 -- 172.21.2.102:0/1379965879 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9024105ae0 con 0x7f9024189d00 2021-02-10T11:10:20.004 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:20.007 7f901a7fc700 1 -- 172.21.2.102:0/1379965879 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f9014030d10 con 0x7f9024189d00 2021-02-10T11:10:20.132 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:20.136 7f902bf89700 1 -- 172.21.2.102:0/1379965879 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344397", "target": ["mgr", ""]}) v1 -- 0x7f9024105ae0 con 0x7f900c0186c0 2021-02-10T11:10:20.136 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:20.140 7f901a7fc700 1 -- 172.21.2.102:0/1379965879 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f9020006550 con 0x7f900c0186c0 2021-02-10T11:10:20.139 INFO:teuthology.orchestra.run.gibba002.stdout:/volumes/_nogroup/subvolume_0000000000344397/d1638857-f939-452b-9179-10011c860226 2021-02-10T11:10:20.141 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:20.144 7f902bf89700 1 -- 172.21.2.102:0/1379965879 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f900c0186c0 msgr2=0x7f900c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:10:20.141 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:20.144 7f902bf89700 1 --2- 172.21.2.102:0/1379965879 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f900c0186c0 0x7f900c01abc0 crc :-1 s=READY pgs=239 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:10:20.141 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:20.144 7f902bf89700 1 -- 172.21.2.102:0/1379965879 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9024189d00 msgr2=0x7f902418a100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:10:20.141 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:20.144 7f902bf89700 1 --2- 172.21.2.102:0/1379965879 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9024189d00 0x7f902418a100 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 rx=0x7f901400b5e0 tx=0x7f901400b980).stop 2021-02-10T11:10:20.141 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:20.144 7f902bf89700 1 -- 172.21.2.102:0/1379965879 shutdown_connections 2021-02-10T11:10:20.141 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:20.144 7f902bf89700 1 --2- 172.21.2.102:0/1379965879 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f900c0186c0 0x7f900c01abc0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:10:20.141 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:20.144 7f902bf89700 1 --2- 172.21.2.102:0/1379965879 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f902418a720 0x7f90240b7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:10:20.141 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:20.144 7f902bf89700 1 --2- 172.21.2.102:0/1379965879 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9024189d00 0x7f902418a100 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:10:20.141 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:20.144 7f902bf89700 1 --2- 172.21.2.102:0/1379965879 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f90240a0420 0x7f90240a0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:10:20.141 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:20.144 7f902bf89700 1 -- 172.21.2.102:0/1379965879 >> 172.21.2.102:0/1379965879 conn(0x7f9024094e10 msgr2=0x7f90240a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:10:20.142 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:20.144 7f902bf89700 1 -- 172.21.2.102:0/1379965879 shutdown_connections 2021-02-10T11:10:20.142 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:20.144 7f902bf89700 1 -- 172.21.2.102:0/1379965879 wait complete. 2021-02-10T11:10:20.148 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000344397 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000344397/d1638857-f939-452b-9179-10011c860226 2021-02-10T11:10:20.149 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:10:20.149 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:10:20.149 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:10:20.485 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:10:20.485 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:10:20.486 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344397/d1638857-f939-452b-9179-10011c860226/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:10:20.867 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:10:20.867 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:10:20.867 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0239495 s, 43.8 MB/s 2021-02-10T11:10:20.868 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:10:20.868 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:10:20.868 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:10:21.204 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:10:21.204 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:10:21.204 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344397/d1638857-f939-452b-9179-10011c860226/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:10:21.577 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:10:21.578 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:10:21.578 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.022862 s, 45.9 MB/s 2021-02-10T11:10:21.579 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:10:21.579 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:10:21.579 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:10:21.914 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:10:21.914 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:10:21.914 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344397/d1638857-f939-452b-9179-10011c860226/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:10:22.289 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:10:22.289 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:10:22.289 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.024232 s, 43.3 MB/s 2021-02-10T11:10:22.290 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:10:22.290 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:10:22.290 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:10:22.626 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:10:22.626 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:10:22.627 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344397/d1638857-f939-452b-9179-10011c860226/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:10:23.002 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:10:23.003 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:10:23.003 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0248009 s, 42.3 MB/s 2021-02-10T11:10:23.004 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:10:23.004 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:10:23.004 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:10:23.340 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:10:23.340 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:10:23.340 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344397/d1638857-f939-452b-9179-10011c860226/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:10:23.711 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:10:23.711 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:10:23.711 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0234745 s, 44.7 MB/s 2021-02-10T11:10:23.712 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:10:23.712 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:10:23.712 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:10:24.047 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:10:24.047 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:10:24.048 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344397/d1638857-f939-452b-9179-10011c860226/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:10:24.424 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:10:24.424 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:10:24.424 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0244007 s, 43.0 MB/s 2021-02-10T11:10:24.425 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:10:24.425 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:10:24.425 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:10:24.761 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:10:24.761 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:10:24.762 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344397/d1638857-f939-452b-9179-10011c860226/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:10:25.138 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:10:25.138 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:10:25.138 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.025234 s, 41.6 MB/s 2021-02-10T11:10:25.139 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:10:25.139 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:10:25.139 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:10:25.474 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:10:25.474 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:10:25.475 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344397/d1638857-f939-452b-9179-10011c860226/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:10:25.849 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:10:25.850 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:10:25.850 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0233119 s, 45.0 MB/s 2021-02-10T11:10:25.851 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:10:25.851 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:10:25.851 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:10:26.186 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:10:26.186 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:10:26.187 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344397/d1638857-f939-452b-9179-10011c860226/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:10:26.562 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:10:26.563 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:10:26.563 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0235998 s, 44.4 MB/s 2021-02-10T11:10:26.564 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:10:26.564 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:10:26.564 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:10:26.899 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:10:26.899 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:10:26.900 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344397/d1638857-f939-452b-9179-10011c860226/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:10:27.276 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:10:27.277 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:10:27.277 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.02502 s, 41.9 MB/s 2021-02-10T11:10:27.278 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000344398 2021-02-10T11:10:27.708 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:27.696 7f8320517700 1 Processor -- start 2021-02-10T11:10:27.708 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:27.696 7f8320517700 1 -- start start 2021-02-10T11:10:27.708 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:27.696 7f8320517700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f83181b0470 0x7f83181b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:10:27.709 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:27.696 7f8320517700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f83180ef610 0x7f83180efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:10:27.709 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:27.696 7f8320517700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f83180f0030 0x7f83180f0430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:10:27.709 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:27.696 7f8320517700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83180e5ea0 con 0x7f83180f0030 2021-02-10T11:10:27.709 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:27.696 7f8320517700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83180f0b50 con 0x7f83180ef610 2021-02-10T11:10:27.709 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:27.696 7f8320517700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83180e52e0 con 0x7f83181b0470 2021-02-10T11:10:27.709 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:27.696 7f831eb34700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f83180f0030 0x7f83180f0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:10:27.709 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:27.696 7f831eb34700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f83180f0030 0x7f83180f0430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:51894/0 (socket says 172.21.2.102:51894) 2021-02-10T11:10:27.709 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:27.696 7f831eb34700 1 -- 172.21.2.102:0/1917250633 learned_addr learned my addr 172.21.2.102:0/1917250633 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:10:27.709 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:27.697 7f831db32700 1 --2- 172.21.2.102:0/1917250633 >> v2:172.21.2.113:3300/0 conn(0x7f83180ef610 0x7f83180efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:10:27.709 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:27.697 7f831e333700 1 --2- 172.21.2.102:0/1917250633 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f83181b0470 0x7f83181b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:10:27.709 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:27.697 7f831eb34700 1 -- 172.21.2.102:0/1917250633 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f83181b0470 msgr2=0x7f83181b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:10:27.709 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:27.697 7f831eb34700 1 --2- 172.21.2.102:0/1917250633 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f83181b0470 0x7f83181b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:10:27.710 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:27.697 7f831eb34700 1 -- 172.21.2.102:0/1917250633 >> v2:172.21.2.113:3300/0 conn(0x7f83180ef610 msgr2=0x7f83180efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:10:27.710 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:27.697 7f831eb34700 1 --2- 172.21.2.102:0/1917250633 >> v2:172.21.2.113:3300/0 conn(0x7f83180ef610 0x7f83180efa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:10:27.710 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:27.697 7f831eb34700 1 -- 172.21.2.102:0/1917250633 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83180f10d0 con 0x7f83180f0030 2021-02-10T11:10:27.710 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:27.697 7f831eb34700 1 --2- 172.21.2.102:0/1917250633 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f83180f0030 0x7f83180f0430 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 rx=0x7f8310006f00 tx=0x7f8310009f10).ready entity=mon.0 client_cookie=e2ef3203ffc03c12 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:10:27.710 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:27.697 7f831d331700 1 -- 172.21.2.102:0/1917250633 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f831000b3b0 con 0x7f83180f0030 2021-02-10T11:10:27.710 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:27.697 7f831d331700 1 -- 172.21.2.102:0/1917250633 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f831000b730 con 0x7f83180f0030 2021-02-10T11:10:27.710 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:27.697 7f8320517700 1 -- 172.21.2.102:0/1917250633 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f83180f0030 msgr2=0x7f83180f0430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:10:27.710 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:27.697 7f8320517700 1 --2- 172.21.2.102:0/1917250633 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f83180f0030 0x7f83180f0430 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 rx=0x7f8310006f00 tx=0x7f8310009f10).stop 2021-02-10T11:10:27.710 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:27.697 7f831d331700 1 -- 172.21.2.102:0/1917250633 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f831000bc50 con 0x7f83180f0030 2021-02-10T11:10:27.710 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:27.698 7f8320517700 1 -- 172.21.2.102:0/1917250633 shutdown_connections 2021-02-10T11:10:27.710 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:27.698 7f8320517700 1 --2- 172.21.2.102:0/1917250633 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f83180f0030 0x7f83180f0430 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:10:27.710 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:27.698 7f8320517700 1 --2- 172.21.2.102:0/1917250633 >> v2:172.21.2.113:3300/0 conn(0x7f83180ef610 0x7f83180efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:10:27.710 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:27.698 7f8320517700 1 --2- 172.21.2.102:0/1917250633 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f83181b0470 0x7f83181b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:10:27.711 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:27.698 7f8320517700 1 -- 172.21.2.102:0/1917250633 >> 172.21.2.102:0/1917250633 conn(0x7f83180ce3e0 msgr2=0x7f83180e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:10:27.711 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:27.698 7f8320517700 1 -- 172.21.2.102:0/1917250633 shutdown_connections 2021-02-10T11:10:27.711 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:27.698 7f8320517700 1 -- 172.21.2.102:0/1917250633 wait complete. 2021-02-10T11:10:27.712 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:27.712 7f8320517700 1 Processor -- start 2021-02-10T11:10:27.712 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:27.712 7f8320517700 1 -- start start 2021-02-10T11:10:27.713 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:27.713 7f8320517700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f83180b7de0 0x7f8318190920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:10:27.713 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:27.713 7f8320517700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f83180b82f0 0x7f83180b86f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:10:27.713 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:27.713 7f8320517700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f83180b8d10 0x7f83180bf9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:10:27.713 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:27.713 7f8320517700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83180b7930 con 0x7f83180b8d10 2021-02-10T11:10:27.713 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:27.713 7f8320517700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83180b7b20 con 0x7f83180b82f0 2021-02-10T11:10:27.713 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:27.713 7f8320517700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83180bffd0 con 0x7f83180b7de0 2021-02-10T11:10:27.713 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:27.713 7f831eb34700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f83180b8d10 0x7f83180bf9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:10:27.713 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:27.713 7f831eb34700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f83180b8d10 0x7f83180bf9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:51902/0 (socket says 172.21.2.102:51902) 2021-02-10T11:10:27.713 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:27.713 7f831eb34700 1 -- 172.21.2.102:0/2107202245 learned_addr learned my addr 172.21.2.102:0/2107202245 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:10:27.713 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:27.713 7f831db32700 1 --2- 172.21.2.102:0/2107202245 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f83180b82f0 0x7f83180b86f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:10:27.713 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:27.713 7f831e333700 1 --2- 172.21.2.102:0/2107202245 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f83180b7de0 0x7f8318190920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:10:27.713 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:27.713 7f831eb34700 1 -- 172.21.2.102:0/2107202245 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f83180b7de0 msgr2=0x7f8318190920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:10:27.714 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:27.713 7f831eb34700 1 --2- 172.21.2.102:0/2107202245 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f83180b7de0 0x7f8318190920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:10:27.714 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:27.713 7f831eb34700 1 -- 172.21.2.102:0/2107202245 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f83180b82f0 msgr2=0x7f83180b86f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:10:27.714 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:27.713 7f831eb34700 1 --2- 172.21.2.102:0/2107202245 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f83180b82f0 0x7f83180b86f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:10:27.714 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:27.714 7f831eb34700 1 -- 172.21.2.102:0/2107202245 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83180c01c0 con 0x7f83180b8d10 2021-02-10T11:10:27.714 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:27.714 7f831eb34700 1 --2- 172.21.2.102:0/2107202245 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f83180b8d10 0x7f83180bf9b0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 rx=0x7f8310009970 tx=0x7f831000bcc0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:10:27.714 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:27.714 7f830effd700 1 -- 172.21.2.102:0/2107202245 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8310009450 con 0x7f83180b8d10 2021-02-10T11:10:27.714 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:27.714 7f830effd700 1 -- 172.21.2.102:0/2107202245 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8310009140 con 0x7f83180b8d10 2021-02-10T11:10:27.714 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:27.714 7f830effd700 1 -- 172.21.2.102:0/2107202245 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f831000c4e0 con 0x7f83180b8d10 2021-02-10T11:10:27.714 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:27.714 7f8320517700 1 -- 172.21.2.102:0/2107202245 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83180c01c0 con 0x7f83180b8d10 2021-02-10T11:10:27.714 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:27.714 7f8320517700 1 -- 172.21.2.102:0/2107202245 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83180c0750 con 0x7f83180b8d10 2021-02-10T11:10:27.714 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:27.715 7f830effd700 1 -- 172.21.2.102:0/2107202245 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f8310001270 con 0x7f83180b8d10 2021-02-10T11:10:27.714 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:27.715 7f830effd700 1 --2- 172.21.2.102:0/2107202245 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f83000186c0 0x7f830001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:10:27.714 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:27.715 7f830effd700 1 -- 172.21.2.102:0/2107202245 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f83100328d0 con 0x7f83180b8d10 2021-02-10T11:10:27.715 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:27.715 7f831e333700 1 --2- 172.21.2.102:0/2107202245 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f83000186c0 0x7f830001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:10:27.715 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:27.715 7f831e333700 1 --2- 172.21.2.102:0/2107202245 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f83000186c0 0x7f830001abc0 crc :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:10:27.742 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:27.746 7f8320517700 1 -- 172.21.2.102:0/2107202245 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8318105b30 con 0x7f83180b8d10 2021-02-10T11:10:27.745 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:27.749 7f830effd700 1 -- 172.21.2.102:0/2107202245 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f83100259f0 con 0x7f83180b8d10 2021-02-10T11:10:27.872 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:27.875 7f8320517700 1 -- 172.21.2.102:0/2107202245 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344398", "target": ["mgr", ""]}) v1 -- 0x7f83180f0b50 con 0x7f83000186c0 2021-02-10T11:10:27.897 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:27.901 7f830effd700 1 -- 172.21.2.102:0/2107202245 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f83140009b0 con 0x7f83000186c0 2021-02-10T11:10:27.905 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:27.908 7f8320517700 1 -- 172.21.2.102:0/2107202245 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f83000186c0 msgr2=0x7f830001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:10:27.905 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:27.908 7f8320517700 1 --2- 172.21.2.102:0/2107202245 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f83000186c0 0x7f830001abc0 crc :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:10:27.906 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:27.908 7f8320517700 1 -- 172.21.2.102:0/2107202245 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f83180b8d10 msgr2=0x7f83180bf9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:10:27.906 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:27.908 7f8320517700 1 --2- 172.21.2.102:0/2107202245 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f83180b8d10 0x7f83180bf9b0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 rx=0x7f8310009970 tx=0x7f831000bcc0).stop 2021-02-10T11:10:27.906 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:27.909 7f8320517700 1 -- 172.21.2.102:0/2107202245 shutdown_connections 2021-02-10T11:10:27.906 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:27.909 7f8320517700 1 --2- 172.21.2.102:0/2107202245 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f83180b8d10 0x7f83180bf9b0 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:10:27.906 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:27.909 7f8320517700 1 --2- 172.21.2.102:0/2107202245 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f83000186c0 0x7f830001abc0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:10:27.906 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:27.909 7f8320517700 1 --2- 172.21.2.102:0/2107202245 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f83180b82f0 0x7f83180b86f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:10:27.906 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:27.909 7f8320517700 1 --2- 172.21.2.102:0/2107202245 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f83180b7de0 0x7f8318190920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:10:27.906 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:27.909 7f8320517700 1 -- 172.21.2.102:0/2107202245 >> 172.21.2.102:0/2107202245 conn(0x7f8318094e10 msgr2=0x7f83180a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:10:27.906 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:27.909 7f8320517700 1 -- 172.21.2.102:0/2107202245 shutdown_connections 2021-02-10T11:10:27.906 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:27.909 7f8320517700 1 -- 172.21.2.102:0/2107202245 wait complete. 2021-02-10T11:10:27.913 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000344398 2021-02-10T11:10:28.345 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:28.331 7fc913763700 1 Processor -- start 2021-02-10T11:10:28.345 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:28.332 7fc913763700 1 -- start start 2021-02-10T11:10:28.345 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:28.332 7fc913763700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc90c1b0470 0x7fc90c1b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:10:28.345 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:28.332 7fc913763700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc90c0f0b50 con 0x7fc90c0f20e0 2021-02-10T11:10:28.345 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:28.332 7fc913763700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc90c0f24e0 con 0x7fc90c0ef610 2021-02-10T11:10:28.345 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:28.332 7fc913763700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc90c0e5ea0 con 0x7fc90c1b0470 2021-02-10T11:10:28.345 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:28.332 7fc911d80700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fc90c0f20e0 0x7fc90c0f0200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:44130/0 (socket says 172.21.2.102:44130) 2021-02-10T11:10:28.345 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:28.332 7fc911d80700 1 -- 172.21.2.102:0/122973361 learned_addr learned my addr 172.21.2.102:0/122973361 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:10:28.345 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:28.332 7fc91157f700 1 --2- 172.21.2.102:0/122973361 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc90c1b0470 0x7fc90c1b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:10:28.346 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:28.333 7fc903fff700 1 -- 172.21.2.102:0/122973361 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2629371427 0 0) 0x7fc8fc000c10 con 0x7fc90c0f20e0 2021-02-10T11:10:28.346 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:28.333 7fc903fff700 1 -- 172.21.2.102:0/122973361 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8f8002fe0 con 0x7fc90c0f20e0 2021-02-10T11:10:28.346 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:28.333 7fc903fff700 1 -- 172.21.2.102:0/122973361 <== mon.1 v1:172.21.2.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3109535362 0 0) 0x7fc908000ff0 con 0x7fc90c0ef610 2021-02-10T11:10:28.346 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:28.333 7fc903fff700 1 -- 172.21.2.102:0/122973361 --> v1:172.21.2.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8f8003210 con 0x7fc90c0ef610 2021-02-10T11:10:28.346 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:28.333 7fc91157f700 1 -- 172.21.2.102:0/122973361 >> v1:172.21.2.113:6789/0 conn(0x7fc90c0ef610 legacy=0x7fc90c0efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:10:28.346 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:28.333 7fc91157f700 1 -- 172.21.2.102:0/122973361 >> v1:172.21.2.102:6789/0 conn(0x7fc90c0f20e0 legacy=0x7fc90c0f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:10:28.346 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:28.333 7fc91157f700 1 -- 172.21.2.102:0/122973361 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc90c0e52e0 con 0x7fc90c1b0470 2021-02-10T11:10:28.346 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:28.333 7fc91157f700 1 --2- 172.21.2.102:0/122973361 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc90c1b0470 0x7fc90c1b0870 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7fc9040069d0 tx=0x7fc904009e40).ready entity=mon.2 client_cookie=3a666919c9636eb8 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:10:28.346 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:28.333 7fc903fff700 1 -- 172.21.2.102:0/122973361 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc90400b510 con 0x7fc90c1b0470 2021-02-10T11:10:28.346 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:28.333 7fc903fff700 1 -- 172.21.2.102:0/122973361 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc90400b510 con 0x7fc90c1b0470 2021-02-10T11:10:28.346 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:28.333 7fc903fff700 1 -- 172.21.2.102:0/122973361 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc90400b890 con 0x7fc90c1b0470 2021-02-10T11:10:28.346 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:28.333 7fc913763700 1 -- 172.21.2.102:0/122973361 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc90c1b0470 msgr2=0x7fc90c1b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:10:28.347 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:28.333 7fc913763700 1 --2- 172.21.2.102:0/122973361 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc90c1b0470 0x7fc90c1b0870 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7fc9040069d0 tx=0x7fc904009e40).stop 2021-02-10T11:10:28.347 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:28.334 7fc913763700 1 -- 172.21.2.102:0/122973361 shutdown_connections 2021-02-10T11:10:28.347 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:28.334 7fc913763700 1 --2- 172.21.2.102:0/122973361 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc90c1b0470 0x7fc90c1b0870 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:10:28.347 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:28.334 7fc913763700 1 -- 172.21.2.102:0/122973361 >> 172.21.2.102:0/122973361 conn(0x7fc90c0ce3e0 msgr2=0x7fc90c0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:10:28.347 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:28.334 7fc913763700 1 -- 172.21.2.102:0/122973361 shutdown_connections 2021-02-10T11:10:28.347 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:28.334 7fc913763700 1 -- 172.21.2.102:0/122973361 wait complete. 2021-02-10T11:10:28.347 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:28.348 7fc913763700 1 Processor -- start 2021-02-10T11:10:28.347 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:28.348 7fc913763700 1 -- start start 2021-02-10T11:10:28.348 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:28.349 7fc913763700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc90c0a0420 0x7fc90c0a0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:10:28.348 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:28.349 7fc913763700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc90c189d00 0x7fc90c18a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:10:28.348 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:28.349 7fc913763700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc90c18a720 0x7fc90c0b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:10:28.348 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:28.349 7fc913763700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc90c189780 con 0x7fc90c0a0420 2021-02-10T11:10:28.349 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:28.349 7fc913763700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc90c18ab20 con 0x7fc90c189d00 2021-02-10T11:10:28.349 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:28.349 7fc913763700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc90c0a1140 con 0x7fc90c18a720 2021-02-10T11:10:28.349 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:28.349 7fc91157f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc90c0a0420 0x7fc90c0a0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:10:28.349 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:28.349 7fc91157f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc90c0a0420 0x7fc90c0a0820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:51908/0 (socket says 172.21.2.102:51908) 2021-02-10T11:10:28.349 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:28.349 7fc91157f700 1 -- 172.21.2.102:0/3181146183 learned_addr learned my addr 172.21.2.102:0/3181146183 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:10:28.349 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:28.349 7fc91157f700 1 -- 172.21.2.102:0/3181146183 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc90c18a720 msgr2=0x7fc90c0b7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:10:28.349 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:28.349 7fc91157f700 1 --2- 172.21.2.102:0/3181146183 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc90c18a720 0x7fc90c0b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:10:28.349 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:28.349 7fc91157f700 1 -- 172.21.2.102:0/3181146183 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc90c189d00 msgr2=0x7fc90c18a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:10:28.349 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:28.349 7fc91157f700 1 --2- 172.21.2.102:0/3181146183 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc90c189d00 0x7fc90c18a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:10:28.349 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:28.349 7fc91157f700 1 -- 172.21.2.102:0/3181146183 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc90c0b8200 con 0x7fc90c0a0420 2021-02-10T11:10:28.349 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:28.350 7fc91157f700 1 --2- 172.21.2.102:0/3181146183 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc90c0a0420 0x7fc90c0a0820 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 rx=0x7fc9040099e0 tx=0x7fc90400bd20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:10:28.349 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:28.350 7fc901ffb700 1 -- 172.21.2.102:0/3181146183 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc904006240 con 0x7fc90c0a0420 2021-02-10T11:10:28.349 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:28.350 7fc901ffb700 1 -- 172.21.2.102:0/3181146183 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc90400be60 con 0x7fc90c0a0420 2021-02-10T11:10:28.350 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:28.350 7fc901ffb700 1 -- 172.21.2.102:0/3181146183 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc904005660 con 0x7fc90c0a0420 2021-02-10T11:10:28.350 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:28.350 7fc913763700 1 -- 172.21.2.102:0/3181146183 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc90c0b8200 con 0x7fc90c0a0420 2021-02-10T11:10:28.350 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:28.350 7fc913763700 1 -- 172.21.2.102:0/3181146183 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc90c0b8200 con 0x7fc90c0a0420 2021-02-10T11:10:28.350 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:28.350 7fc901ffb700 1 -- 172.21.2.102:0/3181146183 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fc90400a000 con 0x7fc90c0a0420 2021-02-10T11:10:28.350 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:28.350 7fc901ffb700 1 --2- 172.21.2.102:0/3181146183 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc8ec0186c0 0x7fc8ec01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:10:28.350 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:28.350 7fc901ffb700 1 -- 172.21.2.102:0/3181146183 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fc904032900 con 0x7fc90c0a0420 2021-02-10T11:10:28.350 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:28.351 7fc910d7e700 1 --2- 172.21.2.102:0/3181146183 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc8ec0186c0 0x7fc8ec01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:10:28.350 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:28.351 7fc910d7e700 1 --2- 172.21.2.102:0/3181146183 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc8ec0186c0 0x7fc8ec01abc0 crc :-1 s=READY pgs=241 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:10:28.378 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:28.382 7fc913763700 1 -- 172.21.2.102:0/3181146183 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc90c105ae0 con 0x7fc90c0a0420 2021-02-10T11:10:28.381 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:28.385 7fc901ffb700 1 -- 172.21.2.102:0/3181146183 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fc904020ed0 con 0x7fc90c0a0420 2021-02-10T11:10:28.513 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:28.516 7fc913763700 1 -- 172.21.2.102:0/3181146183 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344398", "target": ["mgr", ""]}) v1 -- 0x7fc90c189780 con 0x7fc8ec0186c0 2021-02-10T11:10:28.516 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:28.520 7fc901ffb700 1 -- 172.21.2.102:0/3181146183 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7fc908006a20 con 0x7fc8ec0186c0 2021-02-10T11:10:28.520 INFO:teuthology.orchestra.run.gibba002.stdout:/volumes/_nogroup/subvolume_0000000000344398/3cffb902-1ca6-4002-b899-f206be55921c 2021-02-10T11:10:28.521 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:28.525 7fc913763700 1 -- 172.21.2.102:0/3181146183 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc8ec0186c0 msgr2=0x7fc8ec01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:10:28.521 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:28.525 7fc913763700 1 --2- 172.21.2.102:0/3181146183 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc8ec0186c0 0x7fc8ec01abc0 crc :-1 s=READY pgs=241 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:10:28.521 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:28.525 7fc913763700 1 -- 172.21.2.102:0/3181146183 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc90c0a0420 msgr2=0x7fc90c0a0820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:10:28.521 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:28.525 7fc913763700 1 --2- 172.21.2.102:0/3181146183 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc90c0a0420 0x7fc90c0a0820 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 rx=0x7fc9040099e0 tx=0x7fc90400bd20).stop 2021-02-10T11:10:28.522 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:28.525 7fc913763700 1 -- 172.21.2.102:0/3181146183 shutdown_connections 2021-02-10T11:10:28.522 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:28.525 7fc913763700 1 --2- 172.21.2.102:0/3181146183 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc90c18a720 0x7fc90c0b7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:10:28.522 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:28.525 7fc913763700 1 --2- 172.21.2.102:0/3181146183 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc8ec0186c0 0x7fc8ec01abc0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:10:28.522 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:28.525 7fc913763700 1 --2- 172.21.2.102:0/3181146183 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc90c189d00 0x7fc90c18a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:10:28.522 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:28.525 7fc913763700 1 --2- 172.21.2.102:0/3181146183 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc90c0a0420 0x7fc90c0a0820 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:10:28.522 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:28.525 7fc913763700 1 -- 172.21.2.102:0/3181146183 >> 172.21.2.102:0/3181146183 conn(0x7fc90c094e10 msgr2=0x7fc90c0a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:10:28.522 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:28.525 7fc913763700 1 -- 172.21.2.102:0/3181146183 shutdown_connections 2021-02-10T11:10:28.522 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:28.525 7fc913763700 1 -- 172.21.2.102:0/3181146183 wait complete. 2021-02-10T11:10:28.529 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000344398 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000344398/3cffb902-1ca6-4002-b899-f206be55921c 2021-02-10T11:10:28.530 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:10:28.530 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:10:28.530 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:10:28.866 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:10:28.866 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:10:28.866 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344398/3cffb902-1ca6-4002-b899-f206be55921c/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:10:29.245 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:10:29.245 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:10:29.245 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0253262 s, 41.4 MB/s 2021-02-10T11:10:29.247 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:10:29.247 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:10:29.247 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:10:29.582 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:10:29.583 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:10:29.583 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344398/3cffb902-1ca6-4002-b899-f206be55921c/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:10:29.960 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:10:29.960 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:10:29.960 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.023675 s, 44.3 MB/s 2021-02-10T11:10:29.961 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:10:29.961 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:10:29.961 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:10:30.298 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:10:30.298 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:10:30.299 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344398/3cffb902-1ca6-4002-b899-f206be55921c/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:10:30.678 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:10:30.678 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:10:30.678 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0246128 s, 42.6 MB/s 2021-02-10T11:10:30.679 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:10:30.679 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:10:30.679 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:10:31.024 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:10:31.025 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:10:31.025 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344398/3cffb902-1ca6-4002-b899-f206be55921c/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:10:31.401 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:10:31.401 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:10:31.401 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0224967 s, 46.6 MB/s 2021-02-10T11:10:31.402 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:10:31.402 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:10:31.403 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:10:31.740 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:10:31.741 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:10:31.741 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344398/3cffb902-1ca6-4002-b899-f206be55921c/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:10:32.118 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:10:32.118 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:10:32.118 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0241661 s, 43.4 MB/s 2021-02-10T11:10:32.119 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:10:32.119 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:10:32.119 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:10:32.454 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:10:32.454 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:10:32.454 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344398/3cffb902-1ca6-4002-b899-f206be55921c/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:10:32.827 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:10:32.828 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:10:32.828 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0222153 s, 47.2 MB/s 2021-02-10T11:10:32.829 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:10:32.829 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:10:32.829 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:10:33.165 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:10:33.165 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:10:33.166 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344398/3cffb902-1ca6-4002-b899-f206be55921c/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:10:33.539 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:10:33.539 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:10:33.539 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0245866 s, 42.6 MB/s 2021-02-10T11:10:33.540 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:10:33.540 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:10:33.540 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:10:33.876 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:10:33.876 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:10:33.876 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344398/3cffb902-1ca6-4002-b899-f206be55921c/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:10:34.250 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:10:34.250 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:10:34.250 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0226526 s, 46.3 MB/s 2021-02-10T11:10:34.251 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:10:34.252 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:10:34.252 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:10:34.587 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:10:34.587 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:10:34.587 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344398/3cffb902-1ca6-4002-b899-f206be55921c/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:10:34.965 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:10:34.966 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:10:34.966 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.02577 s, 40.7 MB/s 2021-02-10T11:10:34.967 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:10:34.967 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:10:34.967 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:10:35.302 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:10:35.303 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:10:35.303 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344398/3cffb902-1ca6-4002-b899-f206be55921c/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:10:35.680 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:10:35.680 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:10:35.680 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0250048 s, 41.9 MB/s 2021-02-10T11:10:35.681 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000344399 2021-02-10T11:10:36.114 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:36.098 7f9777607700 1 Processor -- start 2021-02-10T11:10:36.114 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:36.098 7f9777607700 1 -- start start 2021-02-10T11:10:36.114 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:36.099 7f9777607700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f97701b0470 0x7f97701b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:10:36.114 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:36.099 7f9777607700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97700f0b50 con 0x7f97700ef610 2021-02-10T11:10:36.114 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:36.099 7f9777607700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97700f24e0 con 0x7f97700f20e0 2021-02-10T11:10:36.114 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:36.099 7f9777607700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97700e5ea0 con 0x7f97701b0470 2021-02-10T11:10:36.115 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:36.099 7f9774c22700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f97700ef610 0x7f97700efa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:44138/0 (socket says 172.21.2.102:44138) 2021-02-10T11:10:36.115 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:36.099 7f9774c22700 1 -- 172.21.2.102:0/322489958 learned_addr learned my addr 172.21.2.102:0/322489958 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:10:36.115 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:36.099 7f9763fff700 1 -- 172.21.2.102:0/322489958 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2926890388 0 0) 0x7f976c0010b0 con 0x7f97700ef610 2021-02-10T11:10:36.115 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:36.099 7f9763fff700 1 -- 172.21.2.102:0/322489958 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f975c002fe0 con 0x7f97700ef610 2021-02-10T11:10:36.115 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:36.100 7f9763fff700 1 -- 172.21.2.102:0/322489958 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3643875114 0 0) 0x7f976c0010b0 con 0x7f97700ef610 2021-02-10T11:10:36.115 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:36.100 7f9763fff700 1 -- 172.21.2.102:0/322489958 >> v2:172.21.2.113:3300/0 conn(0x7f97701b0470 msgr2=0x7f97701b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:10:36.115 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:36.100 7f9775423700 1 --2- 172.21.2.102:0/322489958 >> v2:172.21.2.113:3300/0 conn(0x7f97701b0470 0x7f97701b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:10:36.115 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:36.100 7f9763fff700 1 --2- 172.21.2.102:0/322489958 >> v2:172.21.2.113:3300/0 conn(0x7f97701b0470 0x7f97701b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:10:36.115 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:36.100 7f9763fff700 1 -- 172.21.2.102:0/322489958 >> v1:172.21.2.113:6789/0 conn(0x7f97700f20e0 legacy=0x7f97700f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:10:36.115 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:36.100 7f9763fff700 1 -- 172.21.2.102:0/322489958 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97700e52e0 con 0x7f97700ef610 2021-02-10T11:10:36.115 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:36.100 7f9763fff700 1 -- 172.21.2.102:0/322489958 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7f976c0015b0 con 0x7f97700ef610 2021-02-10T11:10:36.115 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:36.100 7f9763fff700 1 -- 172.21.2.102:0/322489958 >> v1:172.21.2.102:6789/0 conn(0x7f97700ef610 legacy=0x7f97700efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:10:36.115 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:36.100 7f9763fff700 1 --2- 172.21.2.102:0/322489958 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f975c003ae0 0x7f975c005f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:10:36.116 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:36.100 7f9763fff700 1 --2- 172.21.2.102:0/322489958 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f975c006810 0x7f975c008d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:10:36.116 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:36.100 7f9763fff700 1 --2- 172.21.2.102:0/322489958 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f975c009630 0x7f975c00bb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:10:36.116 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:36.100 7f9763fff700 1 -- 172.21.2.102:0/322489958 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f975c00c3a0 con 0x7f975c003ae0 2021-02-10T11:10:36.116 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:36.100 7f9763fff700 1 -- 172.21.2.102:0/322489958 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f975c00c5c0 con 0x7f975c006810 2021-02-10T11:10:36.116 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:36.100 7f9763fff700 1 -- 172.21.2.102:0/322489958 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f975c00c7e0 con 0x7f975c009630 2021-02-10T11:10:36.116 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:36.100 7f9775423700 1 --2- 172.21.2.102:0/322489958 >> v2:172.21.2.113:3300/0 conn(0x7f97701b0470 0x7f97701b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:10:36.116 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:36.100 7f9774c22700 1 --2- 172.21.2.102:0/322489958 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f975c006810 0x7f975c008d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:10:36.116 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:36.100 7f9775423700 1 --2- 172.21.2.102:0/322489958 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f975c003ae0 0x7f975c005f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:10:36.116 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:36.100 7f9775c24700 1 --2- 172.21.2.102:0/322489958 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f975c009630 0x7f975c00bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:10:36.116 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:36.101 7f9775423700 1 -- 172.21.2.102:0/322489958 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f975c009630 msgr2=0x7f975c00bb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:10:36.116 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:36.101 7f9775423700 1 --2- 172.21.2.102:0/322489958 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f975c009630 0x7f975c00bb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:10:36.116 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:36.101 7f9775423700 1 -- 172.21.2.102:0/322489958 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f975c006810 msgr2=0x7f975c008d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:10:36.116 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:36.101 7f9775423700 1 --2- 172.21.2.102:0/322489958 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f975c006810 0x7f975c008d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:10:36.117 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:36.101 7f9775423700 1 -- 172.21.2.102:0/322489958 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f975c00ca80 con 0x7f975c003ae0 2021-02-10T11:10:36.117 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:36.101 7f9775c24700 1 --2- 172.21.2.102:0/322489958 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f975c009630 0x7f975c00bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:10:36.117 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:36.101 7f9774c22700 1 --2- 172.21.2.102:0/322489958 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f975c006810 0x7f975c008d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:10:36.117 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:36.101 7f9775c24700 1 -- 172.21.2.102:0/322489958 reap_dead start 2021-02-10T11:10:36.117 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:36.101 7f9775423700 1 --2- 172.21.2.102:0/322489958 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f975c003ae0 0x7f975c005f90 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 rx=0x7f9764006cf0 tx=0x7f976404dd20).ready entity=mon.0 client_cookie=a5cda272a79443dc server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:10:36.117 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:36.101 7f9763fff700 1 -- 172.21.2.102:0/322489958 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9764006910 con 0x7f975c003ae0 2021-02-10T11:10:36.117 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:36.101 7f9763fff700 1 -- 172.21.2.102:0/322489958 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9764000960 con 0x7f975c003ae0 2021-02-10T11:10:36.117 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:36.101 7f9763fff700 1 -- 172.21.2.102:0/322489958 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f97640020d0 con 0x7f975c003ae0 2021-02-10T11:10:36.117 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:36.101 7f9777607700 1 -- 172.21.2.102:0/322489958 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f975c003ae0 msgr2=0x7f975c005f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:10:36.117 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:36.101 7f9777607700 1 --2- 172.21.2.102:0/322489958 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f975c003ae0 0x7f975c005f90 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 rx=0x7f9764006cf0 tx=0x7f976404dd20).stop 2021-02-10T11:10:36.117 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:36.101 7f9777607700 1 -- 172.21.2.102:0/322489958 shutdown_connections 2021-02-10T11:10:36.117 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:36.101 7f9777607700 1 --2- 172.21.2.102:0/322489958 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f975c003ae0 0x7f975c005f90 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:10:36.117 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:36.102 7f9777607700 1 -- 172.21.2.102:0/322489958 >> 172.21.2.102:0/322489958 conn(0x7f97700ce3e0 msgr2=0x7f97700e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:10:36.118 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:36.102 7f9777607700 1 -- 172.21.2.102:0/322489958 shutdown_connections 2021-02-10T11:10:36.118 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:36.102 7f9777607700 1 -- 172.21.2.102:0/322489958 wait complete. 2021-02-10T11:10:36.118 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:36.116 7f9777607700 1 Processor -- start 2021-02-10T11:10:36.118 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:36.116 7f9777607700 1 -- start start 2021-02-10T11:10:36.118 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:36.117 7f9777607700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f977009b930 0x7f97700b7c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:10:36.118 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:36.117 7f9777607700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f97700b8410 0x7f97700b8810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:10:36.118 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:36.117 7f9777607700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f97700c1f80 0x7f97700c2380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:10:36.118 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:36.117 7f9777607700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97700c1a00 con 0x7f97700c1f80 2021-02-10T11:10:36.118 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:36.117 7f9777607700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97700b8e30 con 0x7f97700b8410 2021-02-10T11:10:36.118 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:36.117 7f9777607700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97700b9050 con 0x7f977009b930 2021-02-10T11:10:36.118 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:36.117 7f9775c24700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f97700c1f80 0x7f97700c2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:10:36.118 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:36.117 7f9775c24700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f97700c1f80 0x7f97700c2380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:51924/0 (socket says 172.21.2.102:51924) 2021-02-10T11:10:36.118 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:36.117 7f9775c24700 1 -- 172.21.2.102:0/3905853653 learned_addr learned my addr 172.21.2.102:0/3905853653 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:10:36.118 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:36.117 7f9775c24700 1 -- 172.21.2.102:0/3905853653 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f977009b930 msgr2=0x7f97700b7c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:10:36.119 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:36.117 7f9775c24700 1 --2- 172.21.2.102:0/3905853653 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f977009b930 0x7f97700b7c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:10:36.119 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:36.117 7f9775c24700 1 -- 172.21.2.102:0/3905853653 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f97700b8410 msgr2=0x7f97700b8810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:10:36.119 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:36.117 7f9775c24700 1 --2- 172.21.2.102:0/3905853653 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f97700b8410 0x7f97700b8810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:10:36.119 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:36.117 7f9775c24700 1 -- 172.21.2.102:0/3905853653 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97700c29a0 con 0x7f97700c1f80 2021-02-10T11:10:36.119 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:36.117 7f9775c24700 1 --2- 172.21.2.102:0/3905853653 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f97700c1f80 0x7f97700c2380 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 rx=0x7f97680099a0 tx=0x7f976800bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:10:36.119 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:36.118 7f9761ffb700 1 -- 172.21.2.102:0/3905853653 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f976800c2f0 con 0x7f97700c1f80 2021-02-10T11:10:36.119 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:36.118 7f9761ffb700 1 -- 172.21.2.102:0/3905853653 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f976800c670 con 0x7f97700c1f80 2021-02-10T11:10:36.119 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:36.118 7f9761ffb700 1 -- 172.21.2.102:0/3905853653 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f976800cb90 con 0x7f97700c1f80 2021-02-10T11:10:36.119 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:36.118 7f9777607700 1 -- 172.21.2.102:0/3905853653 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97700c29a0 con 0x7f97700c1f80 2021-02-10T11:10:36.119 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:36.118 7f9777607700 1 -- 172.21.2.102:0/3905853653 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97700c29a0 con 0x7f97700c1f80 2021-02-10T11:10:36.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:36.118 7f9761ffb700 1 -- 172.21.2.102:0/3905853653 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f976800c2f0 con 0x7f97700c1f80 2021-02-10T11:10:36.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:36.118 7f9761ffb700 1 --2- 172.21.2.102:0/3905853653 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f97500186c0 0x7f975001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:10:36.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:36.118 7f9761ffb700 1 -- 172.21.2.102:0/3905853653 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f97680328f0 con 0x7f97700c1f80 2021-02-10T11:10:36.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:36.118 7f9775423700 1 --2- 172.21.2.102:0/3905853653 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f97500186c0 0x7f975001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:10:36.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:36.119 7f9775423700 1 --2- 172.21.2.102:0/3905853653 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f97500186c0 0x7f975001abc0 crc :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:10:36.145 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:36.149 7f9777607700 1 -- 172.21.2.102:0/3905853653 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9770105b30 con 0x7f97700c1f80 2021-02-10T11:10:36.147 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:36.151 7f9761ffb700 1 -- 172.21.2.102:0/3905853653 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f976800be30 con 0x7f97700c1f80 2021-02-10T11:10:36.273 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:36.277 7f9777607700 1 -- 172.21.2.102:0/3905853653 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344399", "target": ["mgr", ""]}) v1 -- 0x7f9770105b30 con 0x7f97500186c0 2021-02-10T11:10:36.301 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:36.304 7f9761ffb700 1 -- 172.21.2.102:0/3905853653 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f9764000c90 con 0x7f97500186c0 2021-02-10T11:10:36.306 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:36.310 7f9777607700 1 -- 172.21.2.102:0/3905853653 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f97500186c0 msgr2=0x7f975001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:10:36.306 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:36.310 7f9777607700 1 --2- 172.21.2.102:0/3905853653 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f97500186c0 0x7f975001abc0 crc :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:10:36.306 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:36.310 7f9777607700 1 -- 172.21.2.102:0/3905853653 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f97700c1f80 msgr2=0x7f97700c2380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:10:36.306 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:36.310 7f9777607700 1 --2- 172.21.2.102:0/3905853653 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f97700c1f80 0x7f97700c2380 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 rx=0x7f97680099a0 tx=0x7f976800bcf0).stop 2021-02-10T11:10:36.307 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:36.310 7f9777607700 1 -- 172.21.2.102:0/3905853653 shutdown_connections 2021-02-10T11:10:36.307 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:36.310 7f9777607700 1 --2- 172.21.2.102:0/3905853653 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f97700c1f80 0x7f97700c2380 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:10:36.307 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:36.310 7f9777607700 1 --2- 172.21.2.102:0/3905853653 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f97500186c0 0x7f975001abc0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:10:36.307 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:36.310 7f9777607700 1 --2- 172.21.2.102:0/3905853653 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f97700b8410 0x7f97700b8810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:10:36.307 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:36.310 7f9777607700 1 --2- 172.21.2.102:0/3905853653 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f977009b930 0x7f97700b7c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:10:36.307 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:36.310 7f9777607700 1 -- 172.21.2.102:0/3905853653 >> 172.21.2.102:0/3905853653 conn(0x7f9770094e10 msgr2=0x7f97700a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:10:36.307 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:36.310 7f9777607700 1 -- 172.21.2.102:0/3905853653 shutdown_connections 2021-02-10T11:10:36.308 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:36.310 7f9777607700 1 -- 172.21.2.102:0/3905853653 wait complete. 2021-02-10T11:10:36.315 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000344399 2021-02-10T11:10:36.746 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:36.733 7f84e0ce3700 1 Processor -- start 2021-02-10T11:10:36.746 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:36.733 7f84e0ce3700 1 -- start start 2021-02-10T11:10:36.746 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:36.733 7f84e0ce3700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f84dc1b0470 0x7f84dc1b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:10:36.746 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:36.733 7f84e0ce3700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f84dc0ef610 0x7f84dc0efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:10:36.746 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:36.733 7f84e0ce3700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f84dc0f0030 0x7f84dc0f0430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:10:36.746 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:36.733 7f84e0ce3700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84dc0e5ea0 con 0x7f84dc0f0030 2021-02-10T11:10:36.746 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:36.733 7f84e0ce3700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84dc0f0b50 con 0x7f84dc1b0470 2021-02-10T11:10:36.746 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:36.733 7f84e0ce3700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84dc0e52e0 con 0x7f84dc0ef610 2021-02-10T11:10:36.747 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:36.733 7f84db2e9700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f84dc0f0030 0x7f84dc0f0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:10:36.747 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:36.734 7f84db2e9700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f84dc0f0030 0x7f84dc0f0430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:51934/0 (socket says 172.21.2.102:51934) 2021-02-10T11:10:36.747 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:36.734 7f84db2e9700 1 -- 172.21.2.102:0/1122919145 learned_addr learned my addr 172.21.2.102:0/1122919145 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:10:36.747 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:36.734 7f84db2e9700 1 -- 172.21.2.102:0/1122919145 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f84dc0ef610 msgr2=0x7f84dc0efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:10:36.747 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:36.734 7f84db2e9700 1 --2- 172.21.2.102:0/1122919145 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f84dc0ef610 0x7f84dc0efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:10:36.747 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:36.734 7f84db2e9700 1 -- 172.21.2.102:0/1122919145 >> v2:172.21.2.113:3300/0 conn(0x7f84dc1b0470 msgr2=0x7f84dc1b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:10:36.747 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:36.734 7f84db2e9700 1 --2- 172.21.2.102:0/1122919145 >> v2:172.21.2.113:3300/0 conn(0x7f84dc1b0470 0x7f84dc1b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:10:36.747 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:36.734 7f84db2e9700 1 -- 172.21.2.102:0/1122919145 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84dc0f10d0 con 0x7f84dc0f0030 2021-02-10T11:10:36.747 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:36.734 7f84db2e9700 1 --2- 172.21.2.102:0/1122919145 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f84dc0f0030 0x7f84dc0f0430 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 rx=0x7f84d4006f00 tx=0x7f84d4009f10).ready entity=mon.0 client_cookie=74e4c4705aa81a2b server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:10:36.747 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:36.734 7f84d9ae6700 1 -- 172.21.2.102:0/1122919145 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f84d400b3b0 con 0x7f84dc0f0030 2021-02-10T11:10:36.747 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:36.734 7f84d9ae6700 1 -- 172.21.2.102:0/1122919145 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84d400b630 con 0x7f84dc0f0030 2021-02-10T11:10:36.747 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:36.734 7f84d9ae6700 1 -- 172.21.2.102:0/1122919145 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f84d400bb10 con 0x7f84dc0f0030 2021-02-10T11:10:36.747 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:36.734 7f84e0ce3700 1 -- 172.21.2.102:0/1122919145 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f84dc0f0030 msgr2=0x7f84dc0f0430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:10:36.748 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:36.734 7f84e0ce3700 1 --2- 172.21.2.102:0/1122919145 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f84dc0f0030 0x7f84dc0f0430 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 rx=0x7f84d4006f00 tx=0x7f84d4009f10).stop 2021-02-10T11:10:36.748 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:36.735 7f84e0ce3700 1 -- 172.21.2.102:0/1122919145 shutdown_connections 2021-02-10T11:10:36.748 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:36.735 7f84e0ce3700 1 --2- 172.21.2.102:0/1122919145 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f84dc0ef610 0x7f84dc0efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:10:36.748 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:36.735 7f84e0ce3700 1 --2- 172.21.2.102:0/1122919145 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f84dc0f0030 0x7f84dc0f0430 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:10:36.748 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:36.735 7f84e0ce3700 1 --2- 172.21.2.102:0/1122919145 >> v2:172.21.2.113:3300/0 conn(0x7f84dc1b0470 0x7f84dc1b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:10:36.748 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:36.735 7f84e0ce3700 1 -- 172.21.2.102:0/1122919145 >> 172.21.2.102:0/1122919145 conn(0x7f84dc0ce3e0 msgr2=0x7f84dc0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:10:36.748 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:36.735 7f84e0ce3700 1 -- 172.21.2.102:0/1122919145 shutdown_connections 2021-02-10T11:10:36.748 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:36.735 7f84e0ce3700 1 -- 172.21.2.102:0/1122919145 wait complete. 2021-02-10T11:10:36.748 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:36.750 7f84e0ce3700 1 Processor -- start 2021-02-10T11:10:36.748 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:36.750 7f84e0ce3700 1 -- start start 2021-02-10T11:10:36.750 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:36.750 7f84e0ce3700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f84dc0b20b0 0x7f84dc0ac970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:10:36.750 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:36.750 7f84e0ce3700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f84dc0b24b0 0x7f84dc191dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:10:36.750 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:36.750 7f84e0ce3700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f84dc0acf90 0x7f84dc1923e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:10:36.750 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:36.750 7f84e0ce3700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84dc0b1b30 con 0x7f84dc0acf90 2021-02-10T11:10:36.750 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:36.750 7f84e0ce3700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84dc1a04e0 con 0x7f84dc0b20b0 2021-02-10T11:10:36.750 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:36.750 7f84e0ce3700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84dc1a06c0 con 0x7f84dc0b24b0 2021-02-10T11:10:36.750 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:36.750 7f84db2e9700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f84dc0acf90 0x7f84dc1923e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:10:36.750 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:36.750 7f84db2e9700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f84dc0acf90 0x7f84dc1923e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:51936/0 (socket says 172.21.2.102:51936) 2021-02-10T11:10:36.751 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:36.750 7f84db2e9700 1 -- 172.21.2.102:0/96717715 learned_addr learned my addr 172.21.2.102:0/96717715 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:10:36.751 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:36.751 7f84daae8700 1 --2- 172.21.2.102:0/96717715 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f84dc0b20b0 0x7f84dc0ac970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:10:36.751 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:36.751 7f84da2e7700 1 --2- 172.21.2.102:0/96717715 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f84dc0b24b0 0x7f84dc191dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:10:36.751 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:36.751 7f84db2e9700 1 -- 172.21.2.102:0/96717715 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f84dc0b24b0 msgr2=0x7f84dc191dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:10:36.751 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:36.751 7f84db2e9700 1 --2- 172.21.2.102:0/96717715 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f84dc0b24b0 0x7f84dc191dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:10:36.751 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:36.751 7f84db2e9700 1 -- 172.21.2.102:0/96717715 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f84dc0b20b0 msgr2=0x7f84dc0ac970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:10:36.751 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:36.751 7f84db2e9700 1 --2- 172.21.2.102:0/96717715 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f84dc0b20b0 0x7f84dc0ac970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:10:36.751 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:36.751 7f84db2e9700 1 -- 172.21.2.102:0/96717715 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84dc192be0 con 0x7f84dc0acf90 2021-02-10T11:10:36.751 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:36.751 7f84da2e7700 1 --2- 172.21.2.102:0/96717715 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f84dc0b24b0 0x7f84dc191dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:10:36.751 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:36.751 7f84daae8700 1 --2- 172.21.2.102:0/96717715 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f84dc0b20b0 0x7f84dc0ac970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:10:36.751 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:36.751 7f84db2e9700 1 --2- 172.21.2.102:0/96717715 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f84dc0acf90 0x7f84dc1923e0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 rx=0x7f84d40099f0 tx=0x7f84d400bcc0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:10:36.751 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:36.751 7f84d37fe700 1 -- 172.21.2.102:0/96717715 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f84d4000f90 con 0x7f84dc0acf90 2021-02-10T11:10:36.751 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:36.751 7f84d37fe700 1 -- 172.21.2.102:0/96717715 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84d4000cc0 con 0x7f84dc0acf90 2021-02-10T11:10:36.751 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:36.751 7f84d37fe700 1 -- 172.21.2.102:0/96717715 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f84d400c570 con 0x7f84dc0acf90 2021-02-10T11:10:36.752 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:36.751 7f84e0ce3700 1 -- 172.21.2.102:0/96717715 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84dc192be0 con 0x7f84dc0acf90 2021-02-10T11:10:36.752 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:36.751 7f84e0ce3700 1 -- 172.21.2.102:0/96717715 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84dc193170 con 0x7f84dc0acf90 2021-02-10T11:10:36.752 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:36.752 7f84d37fe700 1 -- 172.21.2.102:0/96717715 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f84d4001270 con 0x7f84dc0acf90 2021-02-10T11:10:36.752 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:36.752 7f84d37fe700 1 --2- 172.21.2.102:0/96717715 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f84c00186c0 0x7f84c001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:10:36.752 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:36.752 7f84d37fe700 1 -- 172.21.2.102:0/96717715 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f84d40328f0 con 0x7f84dc0acf90 2021-02-10T11:10:36.752 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:36.752 7f84daae8700 1 --2- 172.21.2.102:0/96717715 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f84c00186c0 0x7f84c001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:10:36.752 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:36.752 7f84daae8700 1 --2- 172.21.2.102:0/96717715 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f84c00186c0 0x7f84c001abc0 crc :-1 s=READY pgs=243 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:10:36.779 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:36.783 7f84e0ce3700 1 -- 172.21.2.102:0/96717715 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84dc105af0 con 0x7f84dc0acf90 2021-02-10T11:10:36.782 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:36.785 7f84d37fe700 1 -- 172.21.2.102:0/96717715 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f84d4003080 con 0x7f84dc0acf90 2021-02-10T11:10:36.909 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:36.912 7f84e0ce3700 1 -- 172.21.2.102:0/96717715 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344399", "target": ["mgr", ""]}) v1 -- 0x7f84dc1a06c0 con 0x7f84c00186c0 2021-02-10T11:10:36.911 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:36.915 7f84d37fe700 1 -- 172.21.2.102:0/96717715 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f84cc006930 con 0x7f84c00186c0 2021-02-10T11:10:36.912 INFO:teuthology.orchestra.run.gibba002.stdout:/volumes/_nogroup/subvolume_0000000000344399/f7df4536-50de-4883-b148-ea671266032a 2021-02-10T11:10:36.913 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:36.917 7f84e0ce3700 1 -- 172.21.2.102:0/96717715 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f84c00186c0 msgr2=0x7f84c001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:10:36.913 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:36.917 7f84e0ce3700 1 --2- 172.21.2.102:0/96717715 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f84c00186c0 0x7f84c001abc0 crc :-1 s=READY pgs=243 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:10:36.913 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:36.917 7f84e0ce3700 1 -- 172.21.2.102:0/96717715 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f84dc0acf90 msgr2=0x7f84dc1923e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:10:36.913 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:36.917 7f84e0ce3700 1 --2- 172.21.2.102:0/96717715 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f84dc0acf90 0x7f84dc1923e0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 rx=0x7f84d40099f0 tx=0x7f84d400bcc0).stop 2021-02-10T11:10:36.913 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:36.917 7f84e0ce3700 1 -- 172.21.2.102:0/96717715 shutdown_connections 2021-02-10T11:10:36.913 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:36.917 7f84e0ce3700 1 --2- 172.21.2.102:0/96717715 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f84dc0acf90 0x7f84dc1923e0 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:10:36.914 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:36.917 7f84e0ce3700 1 --2- 172.21.2.102:0/96717715 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f84dc0b24b0 0x7f84dc191dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:10:36.914 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:36.917 7f84e0ce3700 1 --2- 172.21.2.102:0/96717715 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f84c00186c0 0x7f84c001abc0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:10:36.914 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:36.917 7f84e0ce3700 1 --2- 172.21.2.102:0/96717715 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f84dc0b20b0 0x7f84dc0ac970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:10:36.914 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:36.917 7f84e0ce3700 1 -- 172.21.2.102:0/96717715 >> 172.21.2.102:0/96717715 conn(0x7f84dc0bf4e0 msgr2=0x7f84dc0be3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:10:36.914 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:36.917 7f84e0ce3700 1 -- 172.21.2.102:0/96717715 shutdown_connections 2021-02-10T11:10:36.914 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:36.917 7f84e0ce3700 1 -- 172.21.2.102:0/96717715 wait complete. 2021-02-10T11:10:36.921 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000344399 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000344399/f7df4536-50de-4883-b148-ea671266032a 2021-02-10T11:10:36.921 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:10:36.922 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:10:36.922 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:10:37.258 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:10:37.258 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:10:37.259 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344399/f7df4536-50de-4883-b148-ea671266032a/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:10:37.640 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:10:37.640 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:10:37.640 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0243097 s, 43.1 MB/s 2021-02-10T11:10:37.641 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:10:37.641 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:10:37.641 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:10:37.976 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:10:37.976 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:10:37.976 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344399/f7df4536-50de-4883-b148-ea671266032a/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:10:38.348 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:10:38.348 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:10:38.348 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0240041 s, 43.7 MB/s 2021-02-10T11:10:38.349 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:10:38.349 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:10:38.349 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:10:38.684 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:10:38.685 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:10:38.685 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344399/f7df4536-50de-4883-b148-ea671266032a/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:10:39.063 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:10:39.064 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:10:39.064 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0261767 s, 40.1 MB/s 2021-02-10T11:10:39.065 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:10:39.065 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:10:39.065 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:10:39.400 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:10:39.400 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:10:39.401 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344399/f7df4536-50de-4883-b148-ea671266032a/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:10:39.777 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:10:39.777 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:10:39.777 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0230914 s, 45.4 MB/s 2021-02-10T11:10:39.778 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:10:39.778 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:10:39.778 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:10:40.113 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:10:40.113 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:10:40.113 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344399/f7df4536-50de-4883-b148-ea671266032a/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:10:40.488 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:10:40.488 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:10:40.488 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0248481 s, 42.2 MB/s 2021-02-10T11:10:40.489 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:10:40.489 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:10:40.490 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:10:40.824 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:10:40.824 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:10:40.824 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344399/f7df4536-50de-4883-b148-ea671266032a/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:10:41.201 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:10:41.201 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:10:41.201 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0257225 s, 40.8 MB/s 2021-02-10T11:10:41.202 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:10:41.202 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:10:41.202 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:10:41.538 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:10:41.538 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:10:41.539 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344399/f7df4536-50de-4883-b148-ea671266032a/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:10:41.912 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:10:41.913 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:10:41.913 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0230796 s, 45.4 MB/s 2021-02-10T11:10:41.914 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:10:41.914 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:10:41.914 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:10:42.248 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:10:42.249 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:10:42.249 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344399/f7df4536-50de-4883-b148-ea671266032a/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:10:42.625 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:10:42.626 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:10:42.626 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0253064 s, 41.4 MB/s 2021-02-10T11:10:42.627 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:10:42.627 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:10:42.627 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:10:42.963 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:10:42.963 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:10:42.964 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344399/f7df4536-50de-4883-b148-ea671266032a/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:10:43.337 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:10:43.338 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:10:43.338 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0259026 s, 40.5 MB/s 2021-02-10T11:10:43.339 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:10:43.339 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:10:43.339 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:10:43.674 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:10:43.674 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:10:43.675 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344399/f7df4536-50de-4883-b148-ea671266032a/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:10:44.049 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:10:44.049 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:10:44.049 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0239434 s, 43.8 MB/s 2021-02-10T11:10:44.050 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000344400 2021-02-10T11:10:44.482 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:44.468 7f32d3876700 1 Processor -- start 2021-02-10T11:10:44.482 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:44.468 7f32d3876700 1 -- start start 2021-02-10T11:10:44.482 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:44.469 7f32d3876700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f32cc1b0870 0x7f32cc0f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:10:44.482 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:44.469 7f32d3876700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32cc1b0c70 con 0x7f32cc1b0470 2021-02-10T11:10:44.483 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:44.469 7f32d3876700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32cc0f0b50 con 0x7f32cc0efe00 2021-02-10T11:10:44.483 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:44.469 7f32d3876700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32cc0e5ea0 con 0x7f32cc1b0870 2021-02-10T11:10:44.483 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:44.469 7f32d0e91700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f32cc1b0870 0x7f32cc0f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:10:44.483 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:44.469 7f32d1692700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f32cc1b0470 0x7f32cc0ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:44168/0 (socket says 172.21.2.102:44168) 2021-02-10T11:10:44.483 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:44.469 7f32d1692700 1 -- 172.21.2.102:0/2729981940 learned_addr learned my addr 172.21.2.102:0/2729981940 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:10:44.483 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:44.470 7f32d0e91700 1 -- 172.21.2.102:0/2729981940 >> v1:172.21.2.113:6789/0 conn(0x7f32cc0efe00 legacy=0x7f32cc0f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:10:44.483 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:44.470 7f32d0e91700 1 -- 172.21.2.102:0/2729981940 >> v1:172.21.2.102:6789/0 conn(0x7f32cc1b0470 legacy=0x7f32cc0ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:10:44.483 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:44.470 7f32d0e91700 1 -- 172.21.2.102:0/2729981940 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32cc0e52e0 con 0x7f32cc1b0870 2021-02-10T11:10:44.483 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:44.470 7f32d0e91700 1 --2- 172.21.2.102:0/2729981940 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f32cc1b0870 0x7f32cc0f20e0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 rx=0x7f32bc006650 tx=0x7f32bc009a80).ready entity=mon.0 client_cookie=df08ac4a68af624d server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:10:44.483 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:44.470 7f32c3fff700 1 -- 172.21.2.102:0/2729981940 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f32bc00c0c0 con 0x7f32cc1b0870 2021-02-10T11:10:44.483 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:44.470 7f32c3fff700 1 -- 172.21.2.102:0/2729981940 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32bc00c340 con 0x7f32cc1b0870 2021-02-10T11:10:44.483 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:44.470 7f32c3fff700 1 -- 172.21.2.102:0/2729981940 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f32bc00c860 con 0x7f32cc1b0870 2021-02-10T11:10:44.484 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:44.470 7f32d3876700 1 -- 172.21.2.102:0/2729981940 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f32cc1b0870 msgr2=0x7f32cc0f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:10:44.484 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:44.470 7f32d3876700 1 --2- 172.21.2.102:0/2729981940 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f32cc1b0870 0x7f32cc0f20e0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 rx=0x7f32bc006650 tx=0x7f32bc009a80).stop 2021-02-10T11:10:44.484 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:44.471 7f32d3876700 1 -- 172.21.2.102:0/2729981940 shutdown_connections 2021-02-10T11:10:44.484 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:44.471 7f32d3876700 1 --2- 172.21.2.102:0/2729981940 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f32cc1b0870 0x7f32cc0f20e0 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:10:44.484 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:44.471 7f32d3876700 1 -- 172.21.2.102:0/2729981940 >> 172.21.2.102:0/2729981940 conn(0x7f32cc0ce3e0 msgr2=0x7f32cc0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:10:44.484 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:44.471 7f32d3876700 1 -- 172.21.2.102:0/2729981940 shutdown_connections 2021-02-10T11:10:44.484 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:44.471 7f32d3876700 1 -- 172.21.2.102:0/2729981940 wait complete. 2021-02-10T11:10:44.485 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:44.486 7f32d3876700 1 Processor -- start 2021-02-10T11:10:44.485 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:44.486 7f32d3876700 1 -- start start 2021-02-10T11:10:44.485 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:44.487 7f32d3876700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f32cc0a3b50 0x7f32cc1b3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:10:44.485 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:44.487 7f32d3876700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f32cc1c25b0 0x7f32cc1b3d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:10:44.485 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:44.487 7f32d3876700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f32cc1b4350 0x7f32cc1b18a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:10:44.485 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:44.487 7f32d3876700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32cc1b3190 con 0x7f32cc0a3b50 2021-02-10T11:10:44.486 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:44.487 7f32d3876700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32cc087750 con 0x7f32cc1b4350 2021-02-10T11:10:44.487 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:44.487 7f32d3876700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32cc0a6bf0 con 0x7f32cc1c25b0 2021-02-10T11:10:44.487 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:44.487 7f32d1692700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f32cc0a3b50 0x7f32cc1b3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:10:44.487 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:44.487 7f32d1692700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f32cc0a3b50 0x7f32cc1b3710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:51948/0 (socket says 172.21.2.102:51948) 2021-02-10T11:10:44.487 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:44.487 7f32d1692700 1 -- 172.21.2.102:0/363637985 learned_addr learned my addr 172.21.2.102:0/363637985 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:10:44.487 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:44.487 7f32d1e93700 1 --2- 172.21.2.102:0/363637985 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f32cc1b4350 0x7f32cc1b18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:10:44.488 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:44.488 7f32d0e91700 1 --2- 172.21.2.102:0/363637985 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f32cc1c25b0 0x7f32cc1b3d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:10:44.488 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:44.488 7f32d1692700 1 -- 172.21.2.102:0/363637985 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f32cc1c25b0 msgr2=0x7f32cc1b3d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:10:44.488 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:44.488 7f32d1692700 1 --2- 172.21.2.102:0/363637985 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f32cc1c25b0 0x7f32cc1b3d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:10:44.488 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:44.488 7f32d1692700 1 -- 172.21.2.102:0/363637985 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f32cc1b4350 msgr2=0x7f32cc1b18a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:10:44.488 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:44.488 7f32d1692700 1 --2- 172.21.2.102:0/363637985 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f32cc1b4350 0x7f32cc1b18a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:10:44.488 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:44.488 7f32d1692700 1 -- 172.21.2.102:0/363637985 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32cc1b1ec0 con 0x7f32cc0a3b50 2021-02-10T11:10:44.488 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:44.488 7f32d1692700 1 --2- 172.21.2.102:0/363637985 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f32cc0a3b50 0x7f32cc1b3710 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 rx=0x7f32c8001c60 tx=0x7f32c800a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:10:44.488 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:44.488 7f32c1ffb700 1 -- 172.21.2.102:0/363637985 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f32c800acc0 con 0x7f32cc0a3b50 2021-02-10T11:10:44.488 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:44.488 7f32c1ffb700 1 -- 172.21.2.102:0/363637985 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32c8004140 con 0x7f32cc0a3b50 2021-02-10T11:10:44.489 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:44.488 7f32c1ffb700 1 -- 172.21.2.102:0/363637985 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f32c8004660 con 0x7f32cc0a3b50 2021-02-10T11:10:44.489 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:44.488 7f32d3876700 1 -- 172.21.2.102:0/363637985 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32cc1b1ec0 con 0x7f32cc0a3b50 2021-02-10T11:10:44.489 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:44.488 7f32d3876700 1 -- 172.21.2.102:0/363637985 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32cc1b2450 con 0x7f32cc0a3b50 2021-02-10T11:10:44.489 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:44.489 7f32c1ffb700 1 -- 172.21.2.102:0/363637985 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f32c8004380 con 0x7f32cc0a3b50 2021-02-10T11:10:44.489 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:44.489 7f32c1ffb700 1 --2- 172.21.2.102:0/363637985 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f32ac0186c0 0x7f32ac01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:10:44.489 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:44.489 7f32c1ffb700 1 -- 172.21.2.102:0/363637985 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f32c8032530 con 0x7f32cc0a3b50 2021-02-10T11:10:44.490 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:44.489 7f32d0e91700 1 --2- 172.21.2.102:0/363637985 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f32ac0186c0 0x7f32ac01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:10:44.490 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:44.489 7f32d0e91700 1 --2- 172.21.2.102:0/363637985 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f32ac0186c0 0x7f32ac01abc0 crc :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:10:44.515 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:44.519 7f32d3876700 1 -- 172.21.2.102:0/363637985 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32cc07e6b0 con 0x7f32cc0a3b50 2021-02-10T11:10:44.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:44.521 7f32c1ffb700 1 -- 172.21.2.102:0/363637985 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f32c80009b0 con 0x7f32cc0a3b50 2021-02-10T11:10:44.644 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:44.648 7f32d3876700 1 -- 172.21.2.102:0/363637985 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344400", "target": ["mgr", ""]}) v1 -- 0x7f32cc1b3190 con 0x7f32ac0186c0 2021-02-10T11:10:46.933 DEBUG:teuthology.orchestra.run.gibba002:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:10:46.933 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:44.672 7f32c1ffb700 1 -- 172.21.2.102:0/363637985 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f32bc000c90 con 0x7f32ac0186c0 2021-02-10T11:10:46.933 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:44.681 7f32d3876700 1 -- 172.21.2.102:0/363637985 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f32ac0186c0 msgr2=0x7f32ac01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:10:46.933 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:44.681 7f32d3876700 1 --2- 172.21.2.102:0/363637985 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f32ac0186c0 0x7f32ac01abc0 crc :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:10:46.933 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:44.681 7f32d3876700 1 -- 172.21.2.102:0/363637985 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f32cc0a3b50 msgr2=0x7f32cc1b3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:10:46.933 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:44.681 7f32d3876700 1 --2- 172.21.2.102:0/363637985 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f32cc0a3b50 0x7f32cc1b3710 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 rx=0x7f32c8001c60 tx=0x7f32c800a8d0).stop 2021-02-10T11:10:46.933 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:44.681 7f32d3876700 1 -- 172.21.2.102:0/363637985 shutdown_connections 2021-02-10T11:10:46.934 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:44.681 7f32d3876700 1 --2- 172.21.2.102:0/363637985 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f32ac0186c0 0x7f32ac01abc0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:10:46.934 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:44.681 7f32d3876700 1 --2- 172.21.2.102:0/363637985 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f32cc1b4350 0x7f32cc1b18a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:10:46.934 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:44.681 7f32d3876700 1 --2- 172.21.2.102:0/363637985 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f32cc1c25b0 0x7f32cc1b3d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:10:46.934 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:44.681 7f32d3876700 1 --2- 172.21.2.102:0/363637985 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f32cc0a3b50 0x7f32cc1b3710 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:10:46.934 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:44.681 7f32d3876700 1 -- 172.21.2.102:0/363637985 >> 172.21.2.102:0/363637985 conn(0x7f32cc0886d0 msgr2=0x7f32cc081c40 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:10:46.934 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:44.681 7f32d3876700 1 -- 172.21.2.102:0/363637985 shutdown_connections 2021-02-10T11:10:46.934 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:44.681 7f32d3876700 1 -- 172.21.2.102:0/363637985 wait complete. 2021-02-10T11:10:46.934 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000344400 2021-02-10T11:10:46.937 DEBUG:teuthology.orchestra.run.gibba013:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:10:47.370 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:47.357 7f08b2407700 1 Processor -- start 2021-02-10T11:10:47.371 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:47.357 7f08b2407700 1 -- start start 2021-02-10T11:10:47.371 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:47.358 7f08b2407700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f08ac1b0870 0x7f08ac0f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:10:47.371 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:47.358 7f08b2407700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f08ac0efe00 0x7f08ac0f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:10:47.371 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:47.358 7f08b2407700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08ac1b0c70 con 0x7f08ac1b0470 2021-02-10T11:10:47.371 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:47.358 7f08b2407700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08ac0e5ea0 con 0x7f08ac0efe00 2021-02-10T11:10:47.371 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:47.358 7f08b2407700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08ac0f0b50 con 0x7f08ac1b0870 2021-02-10T11:10:47.371 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:47.358 7f08b0a24700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f08ac0efe00 0x7f08ac0f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:10:47.371 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:47.358 7f08abfff700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f08ac1b0470 0x7f08ac0ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:44180/0 (socket says 172.21.2.102:44180) 2021-02-10T11:10:47.371 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:47.358 7f08abfff700 1 -- 172.21.2.102:0/2984180088 learned_addr learned my addr 172.21.2.102:0/2984180088 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:10:47.371 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:47.358 7f08ab7fe700 1 --2- 172.21.2.102:0/2984180088 >> v2:172.21.2.113:3300/0 conn(0x7f08ac1b0870 0x7f08ac0f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:10:47.371 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:47.359 7f08b0a24700 1 -- 172.21.2.102:0/2984180088 >> v2:172.21.2.113:3300/0 conn(0x7f08ac1b0870 msgr2=0x7f08ac0f20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:10:47.371 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:47.359 7f08b0a24700 1 --2- 172.21.2.102:0/2984180088 >> v2:172.21.2.113:3300/0 conn(0x7f08ac1b0870 0x7f08ac0f20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:10:47.372 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:47.359 7f08b0a24700 1 -- 172.21.2.102:0/2984180088 >> v1:172.21.2.102:6789/0 conn(0x7f08ac1b0470 legacy=0x7f08ac0ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:10:47.372 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:47.359 7f08b0a24700 1 -- 172.21.2.102:0/2984180088 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08ac0f0820 con 0x7f08ac0efe00 2021-02-10T11:10:47.372 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:47.359 7f08b0a24700 1 --2- 172.21.2.102:0/2984180088 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f08ac0efe00 0x7f08ac0f0200 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 rx=0x7f08a4006e30 tx=0x7f08a4009e60).ready entity=mon.0 client_cookie=b982e010187fc01a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:10:47.372 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:47.359 7f08aaffd700 1 -- 172.21.2.102:0/2984180088 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f08a4006a50 con 0x7f08ac0efe00 2021-02-10T11:10:47.372 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:47.359 7f08aaffd700 1 -- 172.21.2.102:0/2984180088 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08a400b300 con 0x7f08ac0efe00 2021-02-10T11:10:47.372 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:47.359 7f08aaffd700 1 -- 172.21.2.102:0/2984180088 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f08a400b760 con 0x7f08ac0efe00 2021-02-10T11:10:47.372 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:47.359 7f08b2407700 1 -- 172.21.2.102:0/2984180088 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f08ac0efe00 msgr2=0x7f08ac0f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:10:47.372 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:47.359 7f08b2407700 1 --2- 172.21.2.102:0/2984180088 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f08ac0efe00 0x7f08ac0f0200 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 rx=0x7f08a4006e30 tx=0x7f08a4009e60).stop 2021-02-10T11:10:47.372 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:47.360 7f08b2407700 1 -- 172.21.2.102:0/2984180088 shutdown_connections 2021-02-10T11:10:47.372 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:47.360 7f08b2407700 1 --2- 172.21.2.102:0/2984180088 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f08ac0efe00 0x7f08ac0f0200 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:10:47.372 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:47.360 7f08b2407700 1 --2- 172.21.2.102:0/2984180088 >> v2:172.21.2.113:3300/0 conn(0x7f08ac1b0870 0x7f08ac0f20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:10:47.372 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:47.360 7f08b2407700 1 -- 172.21.2.102:0/2984180088 >> 172.21.2.102:0/2984180088 conn(0x7f08ac0ce3e0 msgr2=0x7f08ac0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:10:47.373 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:47.360 7f08b2407700 1 -- 172.21.2.102:0/2984180088 shutdown_connections 2021-02-10T11:10:47.373 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:47.360 7f08b2407700 1 -- 172.21.2.102:0/2984180088 wait complete. 2021-02-10T11:10:47.375 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:47.375 7f08b2407700 1 Processor -- start 2021-02-10T11:10:47.375 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:47.375 7f08b2407700 1 -- start start 2021-02-10T11:10:47.375 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:47.376 7f08b2407700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f08ac0c30e0 0x7f08ac10a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:10:47.375 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:47.376 7f08b2407700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f08ac1069f0 0x7f08ac106df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:10:47.375 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:47.376 7f08b2407700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f08ac107410 0x7f08ac08bd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:10:47.375 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:47.376 7f08b2407700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08ac10a7f0 con 0x7f08ac107410 2021-02-10T11:10:47.375 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:47.376 7f08b2407700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08ac107810 con 0x7f08ac0c30e0 2021-02-10T11:10:47.375 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:47.376 7f08b2407700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08ac0c2c40 con 0x7f08ac1069f0 2021-02-10T11:10:47.375 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:47.376 7f08b0a24700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f08ac107410 0x7f08ac08bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:10:47.375 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:47.376 7f08b0a24700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f08ac107410 0x7f08ac08bd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:51966/0 (socket says 172.21.2.102:51966) 2021-02-10T11:10:47.375 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:47.376 7f08b0a24700 1 -- 172.21.2.102:0/3418805478 learned_addr learned my addr 172.21.2.102:0/3418805478 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:10:47.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:47.376 7f08ab7fe700 1 --2- 172.21.2.102:0/3418805478 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f08ac1069f0 0x7f08ac106df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:10:47.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:47.376 7f08abfff700 1 --2- 172.21.2.102:0/3418805478 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f08ac0c30e0 0x7f08ac10a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:10:47.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:47.376 7f08ab7fe700 1 -- 172.21.2.102:0/3418805478 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f08ac0c30e0 msgr2=0x7f08ac10a1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:10:47.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:47.376 7f08ab7fe700 1 --2- 172.21.2.102:0/3418805478 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f08ac0c30e0 0x7f08ac10a1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:10:47.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:47.376 7f08ab7fe700 1 -- 172.21.2.102:0/3418805478 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f08ac107410 msgr2=0x7f08ac08bd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:10:47.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:47.376 7f08ab7fe700 1 --2- 172.21.2.102:0/3418805478 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f08ac107410 0x7f08ac08bd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:10:47.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:47.376 7f08ab7fe700 1 -- 172.21.2.102:0/3418805478 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08ac0c34e0 con 0x7f08ac1069f0 2021-02-10T11:10:47.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:47.376 7f08ab7fe700 1 --2- 172.21.2.102:0/3418805478 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f08ac1069f0 0x7f08ac106df0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7f08a0009540 tx=0x7f08a000b8e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:10:47.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:47.376 7f08a8ff9700 1 -- 172.21.2.102:0/3418805478 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f08a00063c0 con 0x7f08ac1069f0 2021-02-10T11:10:47.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:47.377 7f08a8ff9700 1 -- 172.21.2.102:0/3418805478 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08a000bd60 con 0x7f08ac1069f0 2021-02-10T11:10:47.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:47.377 7f08b2407700 1 -- 172.21.2.102:0/3418805478 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08ac0c34e0 con 0x7f08ac1069f0 2021-02-10T11:10:47.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:47.377 7f08a8ff9700 1 -- 172.21.2.102:0/3418805478 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f08a0002030 con 0x7f08ac1069f0 2021-02-10T11:10:47.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:47.377 7f08b2407700 1 -- 172.21.2.102:0/3418805478 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08ac0c34e0 con 0x7f08ac1069f0 2021-02-10T11:10:47.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:47.377 7f08a8ff9700 1 -- 172.21.2.102:0/3418805478 <== mon.2 v2:172.21.2.113:3301/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f08a0000c90 con 0x7f08ac1069f0 2021-02-10T11:10:47.377 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:47.377 7f08a8ff9700 1 --2- 172.21.2.102:0/3418805478 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f088c0186c0 0x7f088c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:10:47.377 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:47.377 7f08a8ff9700 1 -- 172.21.2.102:0/3418805478 <== mon.2 v2:172.21.2.113:3301/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f08a0032490 con 0x7f08ac1069f0 2021-02-10T11:10:47.377 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:47.377 7f08abfff700 1 --2- 172.21.2.102:0/3418805478 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f088c0186c0 0x7f088c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:10:47.377 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:47.378 7f08abfff700 1 --2- 172.21.2.102:0/3418805478 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f088c0186c0 0x7f088c01abc0 crc :-1 s=READY pgs=245 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:10:47.404 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:47.408 7f08b2407700 1 -- 172.21.2.102:0/3418805478 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08ac10a7f0 con 0x7f08ac1069f0 2021-02-10T11:10:47.407 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:47.411 7f08a8ff9700 1 -- 172.21.2.102:0/3418805478 <== mon.2 v2:172.21.2.113:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f08a0006ae0 con 0x7f08ac1069f0 2021-02-10T11:10:47.541 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:47.545 7f08b2407700 1 -- 172.21.2.102:0/3418805478 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344400", "target": ["mgr", ""]}) v1 -- 0x7f08ac107810 con 0x7f088c0186c0 2021-02-10T11:10:47.544 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:47.548 7f08a8ff9700 1 -- 172.21.2.102:0/3418805478 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f089c000ef0 con 0x7f088c0186c0 2021-02-10T11:10:47.544 INFO:teuthology.orchestra.run.gibba002.stdout:/volumes/_nogroup/subvolume_0000000000344400/21e507fe-f1bc-4024-bcc5-a3240ad36338 2021-02-10T11:10:47.546 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:47.550 7f08b2407700 1 -- 172.21.2.102:0/3418805478 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f088c0186c0 msgr2=0x7f088c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:10:47.546 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:47.550 7f08b2407700 1 --2- 172.21.2.102:0/3418805478 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f088c0186c0 0x7f088c01abc0 crc :-1 s=READY pgs=245 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:10:47.546 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:47.550 7f08b2407700 1 -- 172.21.2.102:0/3418805478 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f08ac1069f0 msgr2=0x7f08ac106df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:10:47.546 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:47.550 7f08b2407700 1 --2- 172.21.2.102:0/3418805478 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f08ac1069f0 0x7f08ac106df0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7f08a0009540 tx=0x7f08a000b8e0).stop 2021-02-10T11:10:47.546 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:47.550 7f08b2407700 1 -- 172.21.2.102:0/3418805478 shutdown_connections 2021-02-10T11:10:47.546 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:47.550 7f08b2407700 1 --2- 172.21.2.102:0/3418805478 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f08ac107410 0x7f08ac08bd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:10:47.546 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:47.550 7f08b2407700 1 --2- 172.21.2.102:0/3418805478 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f08ac1069f0 0x7f08ac106df0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:10:47.547 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:47.550 7f08b2407700 1 --2- 172.21.2.102:0/3418805478 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f088c0186c0 0x7f088c01abc0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:10:47.547 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:47.550 7f08b2407700 1 --2- 172.21.2.102:0/3418805478 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f08ac0c30e0 0x7f08ac10a1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:10:47.547 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:47.550 7f08b2407700 1 -- 172.21.2.102:0/3418805478 >> 172.21.2.102:0/3418805478 conn(0x7f08ac0bbf60 msgr2=0x7f08ac105d10 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:10:47.547 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:47.550 7f08b2407700 1 -- 172.21.2.102:0/3418805478 shutdown_connections 2021-02-10T11:10:47.547 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:47.550 7f08b2407700 1 -- 172.21.2.102:0/3418805478 wait complete. 2021-02-10T11:10:47.554 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000344400 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000344400/21e507fe-f1bc-4024-bcc5-a3240ad36338 2021-02-10T11:10:47.554 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:10:47.555 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:10:47.555 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:10:47.928 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:10:47.929 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:10:47.929 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344400/21e507fe-f1bc-4024-bcc5-a3240ad36338/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:10:48.312 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:10:48.312 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:10:48.312 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0256909 s, 40.8 MB/s 2021-02-10T11:10:48.313 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:10:48.314 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:10:48.314 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:10:48.650 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:10:48.650 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:10:48.651 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344400/21e507fe-f1bc-4024-bcc5-a3240ad36338/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:10:49.041 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:10:49.041 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:10:49.041 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0258956 s, 40.5 MB/s 2021-02-10T11:10:49.042 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:10:49.042 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:10:49.042 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:10:49.387 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:10:49.387 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:10:49.387 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344400/21e507fe-f1bc-4024-bcc5-a3240ad36338/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:10:49.777 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:10:49.777 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:10:49.777 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0242301 s, 43.3 MB/s 2021-02-10T11:10:49.779 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:10:49.779 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:10:49.779 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:10:50.129 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:10:50.129 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:10:50.130 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344400/21e507fe-f1bc-4024-bcc5-a3240ad36338/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:10:50.516 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:10:50.516 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:10:50.516 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0252062 s, 41.6 MB/s 2021-02-10T11:10:50.518 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:10:50.518 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:10:50.518 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:10:50.858 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:10:50.858 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:10:50.859 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344400/21e507fe-f1bc-4024-bcc5-a3240ad36338/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:10:51.234 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:10:51.234 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:10:51.235 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0252587 s, 41.5 MB/s 2021-02-10T11:10:51.236 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:10:51.236 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:10:51.236 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:10:51.572 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:10:51.572 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:10:51.572 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344400/21e507fe-f1bc-4024-bcc5-a3240ad36338/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:10:51.948 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:10:51.948 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:10:51.948 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0225243 s, 46.6 MB/s 2021-02-10T11:10:51.949 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:10:51.949 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:10:51.949 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:10:52.284 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:10:52.285 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:10:52.285 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344400/21e507fe-f1bc-4024-bcc5-a3240ad36338/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:10:52.663 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:10:52.663 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:10:52.663 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0260344 s, 40.3 MB/s 2021-02-10T11:10:52.664 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:10:52.664 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:10:52.665 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:10:52.998 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:10:52.999 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:10:52.999 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344400/21e507fe-f1bc-4024-bcc5-a3240ad36338/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:10:53.374 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:10:53.375 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:10:53.375 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0261324 s, 40.1 MB/s 2021-02-10T11:10:53.376 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:10:53.376 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:10:53.377 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:10:53.712 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:10:53.712 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:10:53.712 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344400/21e507fe-f1bc-4024-bcc5-a3240ad36338/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:10:54.087 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:10:54.087 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:10:54.088 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0230135 s, 45.6 MB/s 2021-02-10T11:10:54.089 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:10:54.089 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:10:54.089 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:10:54.425 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:10:54.425 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:10:54.425 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344400/21e507fe-f1bc-4024-bcc5-a3240ad36338/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:10:54.800 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:10:54.801 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:10:54.801 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0238576 s, 44.0 MB/s 2021-02-10T11:10:54.802 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000344401 2021-02-10T11:10:55.232 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:55.219 7f9a247d0700 1 Processor -- start 2021-02-10T11:10:55.233 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:55.219 7f9a247d0700 1 -- start start 2021-02-10T11:10:55.233 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:55.220 7f9a247d0700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f9a1c1b0470 0x7f9a1c1b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:10:55.233 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:55.220 7f9a247d0700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9a1c0ef610 0x7f9a1c0efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:10:55.233 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:55.220 7f9a247d0700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a1c0f0b50 con 0x7f9a1c0f0030 2021-02-10T11:10:55.233 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:55.220 7f9a247d0700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a1c0e52e0 con 0x7f9a1c1b0470 2021-02-10T11:10:55.233 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:55.220 7f9a247d0700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a1c0e54d0 con 0x7f9a1c0ef610 2021-02-10T11:10:55.233 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:55.220 7f9a22ded700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f9a1c0f0030 0x7f9a1c0f10d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:44196/0 (socket says 172.21.2.102:44196) 2021-02-10T11:10:55.233 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:55.220 7f9a22ded700 1 -- 172.21.2.102:0/764661566 learned_addr learned my addr 172.21.2.102:0/764661566 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:10:55.233 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:55.220 7f9a225ec700 1 --2- 172.21.2.102:0/764661566 >> v2:172.21.2.113:3300/0 conn(0x7f9a1c1b0470 0x7f9a1c1b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:10:55.233 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:55.220 7f9a21deb700 1 --2- 172.21.2.102:0/764661566 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9a1c0ef610 0x7f9a1c0efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:10:55.234 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:55.221 7f9a215ea700 1 -- 172.21.2.102:0/764661566 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2042377905 0 0) 0x7f9a14001500 con 0x7f9a1c0f0030 2021-02-10T11:10:55.234 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:55.221 7f9a215ea700 1 -- 172.21.2.102:0/764661566 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a08002fe0 con 0x7f9a1c0f0030 2021-02-10T11:10:55.234 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:55.221 7f9a21deb700 1 -- 172.21.2.102:0/764661566 >> v2:172.21.2.113:3300/0 conn(0x7f9a1c1b0470 msgr2=0x7f9a1c1b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:10:55.234 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:55.221 7f9a21deb700 1 --2- 172.21.2.102:0/764661566 >> v2:172.21.2.113:3300/0 conn(0x7f9a1c1b0470 0x7f9a1c1b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:10:55.234 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:55.221 7f9a21deb700 1 -- 172.21.2.102:0/764661566 >> v1:172.21.2.102:6789/0 conn(0x7f9a1c0f0030 legacy=0x7f9a1c0f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:10:55.234 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:55.221 7f9a21deb700 1 -- 172.21.2.102:0/764661566 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a1c0f0430 con 0x7f9a1c0ef610 2021-02-10T11:10:55.234 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:55.221 7f9a225ec700 1 --2- 172.21.2.102:0/764661566 >> v2:172.21.2.113:3300/0 conn(0x7f9a1c1b0470 0x7f9a1c1b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:10:55.234 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:55.221 7f9a21deb700 1 --2- 172.21.2.102:0/764661566 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9a1c0ef610 0x7f9a1c0efa10 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7f9a18006650 tx=0x7f9a18009a80).ready entity=mon.2 client_cookie=a03e08d590767fc6 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:10:55.234 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:55.221 7f9a215ea700 1 -- 172.21.2.102:0/764661566 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9a180009b0 con 0x7f9a1c0ef610 2021-02-10T11:10:55.234 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:55.221 7f9a215ea700 1 -- 172.21.2.102:0/764661566 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a1800c0c0 con 0x7f9a1c0ef610 2021-02-10T11:10:55.234 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:55.221 7f9a247d0700 1 -- 172.21.2.102:0/764661566 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9a1c0ef610 msgr2=0x7f9a1c0efa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:10:55.234 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:55.221 7f9a247d0700 1 --2- 172.21.2.102:0/764661566 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9a1c0ef610 0x7f9a1c0efa10 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7f9a18006650 tx=0x7f9a18009a80).stop 2021-02-10T11:10:55.235 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:55.221 7f9a215ea700 1 -- 172.21.2.102:0/764661566 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9a1800c5e0 con 0x7f9a1c0ef610 2021-02-10T11:10:55.235 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:55.222 7f9a247d0700 1 -- 172.21.2.102:0/764661566 shutdown_connections 2021-02-10T11:10:55.235 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:55.222 7f9a247d0700 1 --2- 172.21.2.102:0/764661566 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9a1c0ef610 0x7f9a1c0efa10 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:10:55.235 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:55.222 7f9a247d0700 1 --2- 172.21.2.102:0/764661566 >> v2:172.21.2.113:3300/0 conn(0x7f9a1c1b0470 0x7f9a1c1b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:10:55.235 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:55.222 7f9a247d0700 1 -- 172.21.2.102:0/764661566 >> 172.21.2.102:0/764661566 conn(0x7f9a1c0ce3e0 msgr2=0x7f9a1c0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:10:55.235 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:55.222 7f9a247d0700 1 -- 172.21.2.102:0/764661566 shutdown_connections 2021-02-10T11:10:55.235 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:55.222 7f9a247d0700 1 -- 172.21.2.102:0/764661566 wait complete. 2021-02-10T11:10:55.236 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:55.237 7f9a247d0700 1 Processor -- start 2021-02-10T11:10:55.236 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:55.237 7f9a247d0700 1 -- start start 2021-02-10T11:10:55.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:55.237 7f9a247d0700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9a1c190920 0x7f9a1c0b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:10:55.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:55.237 7f9a247d0700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f9a1c0b8200 0x7f9a1c0b8600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:10:55.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:55.237 7f9a247d0700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9a1c0c1d80 0x7f9a1c0c2180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:10:55.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:55.237 7f9a247d0700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a1c190d20 con 0x7f9a1c190920 2021-02-10T11:10:55.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:55.237 7f9a247d0700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a1c0b8c20 con 0x7f9a1c0b8200 2021-02-10T11:10:55.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:55.237 7f9a247d0700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a1c0c27a0 con 0x7f9a1c0c1d80 2021-02-10T11:10:55.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:55.237 7f9a225ec700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9a1c190920 0x7f9a1c0b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:10:55.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:55.237 7f9a225ec700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9a1c190920 0x7f9a1c0b7be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:51974/0 (socket says 172.21.2.102:51974) 2021-02-10T11:10:55.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:55.238 7f9a225ec700 1 -- 172.21.2.102:0/2559764738 learned_addr learned my addr 172.21.2.102:0/2559764738 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:10:55.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:55.238 7f9a225ec700 1 -- 172.21.2.102:0/2559764738 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9a1c0c1d80 msgr2=0x7f9a1c0c2180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:10:55.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:55.238 7f9a225ec700 1 --2- 172.21.2.102:0/2559764738 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9a1c0c1d80 0x7f9a1c0c2180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:10:55.238 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:55.238 7f9a225ec700 1 -- 172.21.2.102:0/2559764738 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f9a1c0b8200 msgr2=0x7f9a1c0b8600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:10:55.238 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:55.238 7f9a225ec700 1 --2- 172.21.2.102:0/2559764738 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f9a1c0b8200 0x7f9a1c0b8600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:10:55.238 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:55.238 7f9a225ec700 1 -- 172.21.2.102:0/2559764738 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a1c0c29c0 con 0x7f9a1c190920 2021-02-10T11:10:55.238 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:55.238 7f9a225ec700 1 --2- 172.21.2.102:0/2559764738 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9a1c190920 0x7f9a1c0b7be0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 rx=0x7f9a10006f40 tx=0x7f9a1000b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:10:55.238 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:55.238 7f9a0effd700 1 -- 172.21.2.102:0/2559764738 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9a1000bd80 con 0x7f9a1c190920 2021-02-10T11:10:55.238 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:55.238 7f9a0effd700 1 -- 172.21.2.102:0/2559764738 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a10000c90 con 0x7f9a1c190920 2021-02-10T11:10:55.238 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:55.238 7f9a247d0700 1 -- 172.21.2.102:0/2559764738 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a1c0c29c0 con 0x7f9a1c190920 2021-02-10T11:10:55.238 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:55.238 7f9a247d0700 1 -- 172.21.2.102:0/2559764738 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a1c0c2f50 con 0x7f9a1c190920 2021-02-10T11:10:55.238 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:55.238 7f9a0effd700 1 -- 172.21.2.102:0/2559764738 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9a100022b0 con 0x7f9a1c190920 2021-02-10T11:10:55.238 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:55.239 7f9a0effd700 1 -- 172.21.2.102:0/2559764738 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f9a1000a000 con 0x7f9a1c190920 2021-02-10T11:10:55.238 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:55.239 7f9a0effd700 1 --2- 172.21.2.102:0/2559764738 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f9a04018730 0x7f9a0401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:10:55.238 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:55.239 7f9a0effd700 1 -- 172.21.2.102:0/2559764738 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f9a100327c0 con 0x7f9a1c190920 2021-02-10T11:10:55.239 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:55.239 7f9a21deb700 1 --2- 172.21.2.102:0/2559764738 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f9a04018730 0x7f9a0401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:10:55.239 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:55.239 7f9a21deb700 1 --2- 172.21.2.102:0/2559764738 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f9a04018730 0x7f9a0401ac10 crc :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:10:55.266 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:55.270 7f9a247d0700 1 -- 172.21.2.102:0/2559764738 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a1c105ae0 con 0x7f9a1c190920 2021-02-10T11:10:55.268 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:55.272 7f9a0effd700 1 -- 172.21.2.102:0/2559764738 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f9a100009b0 con 0x7f9a1c190920 2021-02-10T11:10:55.395 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:55.398 7f9a247d0700 1 -- 172.21.2.102:0/2559764738 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344401", "target": ["mgr", ""]}) v1 -- 0x7f9a1c190d20 con 0x7f9a04018730 2021-02-10T11:10:55.421 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:55.425 7f9a0effd700 1 -- 172.21.2.102:0/2559764738 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f9a18000940 con 0x7f9a04018730 2021-02-10T11:10:55.428 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:55.432 7f9a247d0700 1 -- 172.21.2.102:0/2559764738 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f9a04018730 msgr2=0x7f9a0401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:10:55.428 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:55.432 7f9a247d0700 1 --2- 172.21.2.102:0/2559764738 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f9a04018730 0x7f9a0401ac10 crc :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:10:55.428 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:55.432 7f9a247d0700 1 -- 172.21.2.102:0/2559764738 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9a1c190920 msgr2=0x7f9a1c0b7be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:10:55.428 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:55.432 7f9a247d0700 1 --2- 172.21.2.102:0/2559764738 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9a1c190920 0x7f9a1c0b7be0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 rx=0x7f9a10006f40 tx=0x7f9a1000b990).stop 2021-02-10T11:10:55.428 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:55.432 7f9a247d0700 1 -- 172.21.2.102:0/2559764738 shutdown_connections 2021-02-10T11:10:55.428 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:55.432 7f9a247d0700 1 --2- 172.21.2.102:0/2559764738 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9a1c0c1d80 0x7f9a1c0c2180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:10:55.428 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:55.432 7f9a247d0700 1 --2- 172.21.2.102:0/2559764738 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f9a04018730 0x7f9a0401ac10 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:10:55.428 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:55.432 7f9a247d0700 1 --2- 172.21.2.102:0/2559764738 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f9a1c0b8200 0x7f9a1c0b8600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:10:55.429 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:55.432 7f9a247d0700 1 --2- 172.21.2.102:0/2559764738 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9a1c190920 0x7f9a1c0b7be0 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:10:55.429 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:55.432 7f9a247d0700 1 -- 172.21.2.102:0/2559764738 >> 172.21.2.102:0/2559764738 conn(0x7f9a1c094e10 msgr2=0x7f9a1c0a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:10:55.429 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:55.432 7f9a247d0700 1 -- 172.21.2.102:0/2559764738 shutdown_connections 2021-02-10T11:10:55.429 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:55.432 7f9a247d0700 1 -- 172.21.2.102:0/2559764738 wait complete. 2021-02-10T11:10:55.436 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000344401 2021-02-10T11:10:55.869 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:55.856 7f3a0bf28700 1 Processor -- start 2021-02-10T11:10:55.869 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:55.856 7f3a0bf28700 1 -- start start 2021-02-10T11:10:55.869 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:55.856 7f3a0bf28700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3a041b0870 0x7f3a040f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:10:55.869 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:55.856 7f3a0bf28700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f3a040efe00 0x7f3a040f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:10:55.869 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:55.856 7f3a0bf28700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a041b0c70 con 0x7f3a041b0470 2021-02-10T11:10:55.869 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:55.856 7f3a0bf28700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a040f0b50 con 0x7f3a040efe00 2021-02-10T11:10:55.869 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:55.856 7f3a0bf28700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a040f0820 con 0x7f3a041b0870 2021-02-10T11:10:55.869 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:55.856 7f3a09d44700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f3a041b0470 0x7f3a040ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:44206/0 (socket says 172.21.2.102:44206) 2021-02-10T11:10:55.869 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:55.856 7f3a09d44700 1 -- 172.21.2.102:0/766152881 learned_addr learned my addr 172.21.2.102:0/766152881 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:10:55.869 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:55.857 7f3a0a545700 1 --2- 172.21.2.102:0/766152881 >> v2:172.21.2.113:3300/0 conn(0x7f3a040efe00 0x7f3a040f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:10:55.870 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:55.857 7f3a09543700 1 --2- 172.21.2.102:0/766152881 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3a041b0870 0x7f3a040f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:10:55.870 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:55.857 7f3a08d42700 1 -- 172.21.2.102:0/766152881 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 506370867 0 0) 0x7f39f8000ce0 con 0x7f3a041b0470 2021-02-10T11:10:55.870 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:55.857 7f3a08d42700 1 -- 172.21.2.102:0/766152881 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39f0002fe0 con 0x7f3a041b0470 2021-02-10T11:10:55.870 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:55.857 7f3a09543700 1 -- 172.21.2.102:0/766152881 >> v2:172.21.2.113:3300/0 conn(0x7f3a040efe00 msgr2=0x7f3a040f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:10:55.870 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:55.857 7f3a09543700 1 --2- 172.21.2.102:0/766152881 >> v2:172.21.2.113:3300/0 conn(0x7f3a040efe00 0x7f3a040f0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:10:55.870 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:55.857 7f3a09543700 1 -- 172.21.2.102:0/766152881 >> v1:172.21.2.102:6789/0 conn(0x7f3a041b0470 legacy=0x7f3a040ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:10:55.870 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:55.857 7f3a09543700 1 -- 172.21.2.102:0/766152881 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a040e52e0 con 0x7f3a041b0870 2021-02-10T11:10:55.870 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:55.857 7f3a09543700 1 --2- 172.21.2.102:0/766152881 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3a041b0870 0x7f3a040f20e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7f3a00006650 tx=0x7f3a00009a80).ready entity=mon.2 client_cookie=fe81befe76414843 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:10:55.870 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:55.857 7f3a08d42700 1 -- 172.21.2.102:0/766152881 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3a0000c0c0 con 0x7f3a041b0870 2021-02-10T11:10:55.870 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:55.857 7f3a08d42700 1 -- 172.21.2.102:0/766152881 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a0000c340 con 0x7f3a041b0870 2021-02-10T11:10:55.870 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:55.857 7f3a08d42700 1 -- 172.21.2.102:0/766152881 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3a0000c860 con 0x7f3a041b0870 2021-02-10T11:10:55.870 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:55.858 7f3a0bf28700 1 -- 172.21.2.102:0/766152881 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3a041b0870 msgr2=0x7f3a040f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:10:55.870 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:55.858 7f3a0bf28700 1 --2- 172.21.2.102:0/766152881 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3a041b0870 0x7f3a040f20e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7f3a00006650 tx=0x7f3a00009a80).stop 2021-02-10T11:10:55.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:55.858 7f3a0bf28700 1 -- 172.21.2.102:0/766152881 shutdown_connections 2021-02-10T11:10:55.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:55.858 7f3a0bf28700 1 --2- 172.21.2.102:0/766152881 >> v2:172.21.2.113:3300/0 conn(0x7f3a040efe00 0x7f3a040f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:10:55.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:55.858 7f3a0bf28700 1 --2- 172.21.2.102:0/766152881 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3a041b0870 0x7f3a040f20e0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:10:55.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:55.858 7f3a0bf28700 1 -- 172.21.2.102:0/766152881 >> 172.21.2.102:0/766152881 conn(0x7f3a040ce3e0 msgr2=0x7f3a040e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:10:55.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:55.859 7f3a0bf28700 1 -- 172.21.2.102:0/766152881 shutdown_connections 2021-02-10T11:10:55.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:55.859 7f3a0bf28700 1 -- 172.21.2.102:0/766152881 wait complete. 2021-02-10T11:10:55.873 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:55.873 7f3a0bf28700 1 Processor -- start 2021-02-10T11:10:55.873 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:55.873 7f3a0bf28700 1 -- start start 2021-02-10T11:10:55.873 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:55.873 7f3a0bf28700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3a04190920 0x7f3a040b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:10:55.873 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:55.873 7f3a0bf28700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3a040b8200 0x7f3a040b8600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:10:55.873 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:55.874 7f3a0bf28700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3a040c1d80 0x7f3a040c2180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:10:55.873 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:55.874 7f3a0bf28700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a04190da0 con 0x7f3a040c1d80 2021-02-10T11:10:55.874 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:55.874 7f3a0bf28700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a040b8c20 con 0x7f3a04190920 2021-02-10T11:10:55.874 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:55.874 7f3a0bf28700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a040b8e10 con 0x7f3a040b8200 2021-02-10T11:10:55.874 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:55.874 7f3a0a545700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3a040c1d80 0x7f3a040c2180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:10:55.874 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:55.874 7f3a0a545700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3a040c1d80 0x7f3a040c2180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:51992/0 (socket says 172.21.2.102:51992) 2021-02-10T11:10:55.874 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:55.874 7f3a0a545700 1 -- 172.21.2.102:0/2597211177 learned_addr learned my addr 172.21.2.102:0/2597211177 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:10:55.874 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:55.874 7f3a09d44700 1 --2- 172.21.2.102:0/2597211177 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3a04190920 0x7f3a040b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:10:55.874 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:55.874 7f3a09543700 1 --2- 172.21.2.102:0/2597211177 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3a040b8200 0x7f3a040b8600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:10:55.874 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:55.874 7f3a0a545700 1 -- 172.21.2.102:0/2597211177 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3a040b8200 msgr2=0x7f3a040b8600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:10:55.874 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:55.874 7f3a0a545700 1 --2- 172.21.2.102:0/2597211177 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3a040b8200 0x7f3a040b8600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:10:55.874 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:55.874 7f3a0a545700 1 -- 172.21.2.102:0/2597211177 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3a04190920 msgr2=0x7f3a040b7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:10:55.874 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:55.874 7f3a0a545700 1 --2- 172.21.2.102:0/2597211177 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3a04190920 0x7f3a040b7be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:10:55.874 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:55.874 7f3a0a545700 1 -- 172.21.2.102:0/2597211177 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a040c27d0 con 0x7f3a040c1d80 2021-02-10T11:10:55.875 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:55.874 7f3a09d44700 1 --2- 172.21.2.102:0/2597211177 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3a04190920 0x7f3a040b7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:10:55.875 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:55.874 7f3a09543700 1 --2- 172.21.2.102:0/2597211177 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3a040b8200 0x7f3a040b8600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:10:55.875 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:55.874 7f3a0a545700 1 --2- 172.21.2.102:0/2597211177 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3a040c1d80 0x7f3a040c2180 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 rx=0x7f39fc00b970 tx=0x7f39fc00bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:10:55.875 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:55.875 7f39f67fc700 1 -- 172.21.2.102:0/2597211177 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f39fc000f90 con 0x7f3a040c1d80 2021-02-10T11:10:55.875 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:55.875 7f3a0bf28700 1 -- 172.21.2.102:0/2597211177 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a040c27d0 con 0x7f3a040c1d80 2021-02-10T11:10:55.875 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:55.875 7f39f67fc700 1 -- 172.21.2.102:0/2597211177 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39fc003030 con 0x7f3a040c1d80 2021-02-10T11:10:55.875 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:55.875 7f39f67fc700 1 -- 172.21.2.102:0/2597211177 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f39fc000f90 con 0x7f3a040c1d80 2021-02-10T11:10:55.875 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:55.875 7f3a0bf28700 1 -- 172.21.2.102:0/2597211177 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a040c27d0 con 0x7f3a040c1d80 2021-02-10T11:10:55.875 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:55.875 7f39f67fc700 1 -- 172.21.2.102:0/2597211177 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f39fc000e70 con 0x7f3a040c1d80 2021-02-10T11:10:55.875 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:55.875 7f39f67fc700 1 --2- 172.21.2.102:0/2597211177 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f39ec018730 0x7f39ec01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:10:55.875 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:55.875 7f39f67fc700 1 -- 172.21.2.102:0/2597211177 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f39fc032ba0 con 0x7f3a040c1d80 2021-02-10T11:10:55.875 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:55.876 7f3a09d44700 1 --2- 172.21.2.102:0/2597211177 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f39ec018730 0x7f39ec01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:10:55.875 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:55.876 7f3a09d44700 1 --2- 172.21.2.102:0/2597211177 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f39ec018730 0x7f39ec01ac10 crc :-1 s=READY pgs=247 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:10:55.902 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:55.906 7f3a0bf28700 1 -- 172.21.2.102:0/2597211177 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a04105ae0 con 0x7f3a040c1d80 2021-02-10T11:10:55.905 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:55.909 7f39f67fc700 1 -- 172.21.2.102:0/2597211177 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f39fc003010 con 0x7f3a040c1d80 2021-02-10T11:10:56.033 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:56.037 7f3a0bf28700 1 -- 172.21.2.102:0/2597211177 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344401", "target": ["mgr", ""]}) v1 -- 0x7f3a04105ae0 con 0x7f39ec018730 2021-02-10T11:10:56.037 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:56.041 7f39f67fc700 1 -- 172.21.2.102:0/2597211177 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f39f8006880 con 0x7f39ec018730 2021-02-10T11:10:56.041 INFO:teuthology.orchestra.run.gibba002.stdout:/volumes/_nogroup/subvolume_0000000000344401/6e41e8fa-bf57-48a1-b2a9-ab6597246231 2021-02-10T11:10:56.042 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:56.046 7f3a0bf28700 1 -- 172.21.2.102:0/2597211177 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f39ec018730 msgr2=0x7f39ec01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:10:56.043 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:56.046 7f3a0bf28700 1 --2- 172.21.2.102:0/2597211177 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f39ec018730 0x7f39ec01ac10 crc :-1 s=READY pgs=247 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:10:56.043 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:56.046 7f3a0bf28700 1 -- 172.21.2.102:0/2597211177 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3a040c1d80 msgr2=0x7f3a040c2180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:10:56.043 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:56.046 7f3a0bf28700 1 --2- 172.21.2.102:0/2597211177 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3a040c1d80 0x7f3a040c2180 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 rx=0x7f39fc00b970 tx=0x7f39fc00bd10).stop 2021-02-10T11:10:56.043 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:56.046 7f3a0bf28700 1 -- 172.21.2.102:0/2597211177 shutdown_connections 2021-02-10T11:10:56.043 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:56.046 7f3a0bf28700 1 --2- 172.21.2.102:0/2597211177 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3a040c1d80 0x7f3a040c2180 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:10:56.043 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:56.046 7f3a0bf28700 1 --2- 172.21.2.102:0/2597211177 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3a040b8200 0x7f3a040b8600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:10:56.043 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:56.046 7f3a0bf28700 1 --2- 172.21.2.102:0/2597211177 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f39ec018730 0x7f39ec01ac10 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:10:56.043 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:56.046 7f3a0bf28700 1 --2- 172.21.2.102:0/2597211177 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3a04190920 0x7f3a040b7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:10:56.043 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:56.046 7f3a0bf28700 1 -- 172.21.2.102:0/2597211177 >> 172.21.2.102:0/2597211177 conn(0x7f3a04094e10 msgr2=0x7f3a040a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:10:56.043 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:56.047 7f3a0bf28700 1 -- 172.21.2.102:0/2597211177 shutdown_connections 2021-02-10T11:10:56.044 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:10:56.047 7f3a0bf28700 1 -- 172.21.2.102:0/2597211177 wait complete. 2021-02-10T11:10:56.051 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000344401 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000344401/6e41e8fa-bf57-48a1-b2a9-ab6597246231 2021-02-10T11:10:56.051 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:10:56.051 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:10:56.051 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:10:56.387 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:10:56.387 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:10:56.388 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344401/6e41e8fa-bf57-48a1-b2a9-ab6597246231/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:10:56.768 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:10:56.768 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:10:56.768 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0231866 s, 45.2 MB/s 2021-02-10T11:10:56.769 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:10:56.769 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:10:56.770 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:10:57.104 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:10:57.105 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:10:57.105 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344401/6e41e8fa-bf57-48a1-b2a9-ab6597246231/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:10:57.480 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:10:57.480 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:10:57.481 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0235191 s, 44.6 MB/s 2021-02-10T11:10:57.482 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:10:57.482 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:10:57.482 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:10:57.819 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:10:57.819 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:10:57.820 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344401/6e41e8fa-bf57-48a1-b2a9-ab6597246231/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:10:58.196 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:10:58.196 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:10:58.196 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0238497 s, 44.0 MB/s 2021-02-10T11:10:58.197 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:10:58.197 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:10:58.197 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:10:58.532 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:10:58.532 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:10:58.532 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344401/6e41e8fa-bf57-48a1-b2a9-ab6597246231/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:10:58.906 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:10:58.906 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:10:58.906 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0236808 s, 44.3 MB/s 2021-02-10T11:10:58.907 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:10:58.907 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:10:58.907 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:10:59.244 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:10:59.244 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:10:59.244 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344401/6e41e8fa-bf57-48a1-b2a9-ab6597246231/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:10:59.621 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:10:59.621 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:10:59.621 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0246771 s, 42.5 MB/s 2021-02-10T11:10:59.622 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:10:59.622 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:10:59.622 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:10:59.957 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:10:59.957 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:10:59.958 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344401/6e41e8fa-bf57-48a1-b2a9-ab6597246231/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:11:00.333 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:11:00.333 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:11:00.333 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0231844 s, 45.2 MB/s 2021-02-10T11:11:00.334 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:11:00.334 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:11:00.334 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:11:00.669 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:11:00.670 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:11:00.670 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344401/6e41e8fa-bf57-48a1-b2a9-ab6597246231/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:11:01.045 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:11:01.045 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:11:01.045 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0243177 s, 43.1 MB/s 2021-02-10T11:11:01.046 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:11:01.046 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:11:01.046 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:11:01.381 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:11:01.381 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:11:01.381 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344401/6e41e8fa-bf57-48a1-b2a9-ab6597246231/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:11:01.758 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:11:01.758 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:11:01.758 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0241812 s, 43.4 MB/s 2021-02-10T11:11:01.759 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:11:01.759 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:11:01.759 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:11:02.094 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:11:02.094 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:11:02.094 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344401/6e41e8fa-bf57-48a1-b2a9-ab6597246231/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:11:02.470 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:11:02.470 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:11:02.470 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0245456 s, 42.7 MB/s 2021-02-10T11:11:02.471 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:11:02.471 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:11:02.471 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:11:02.807 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:11:02.808 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:11:02.808 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344401/6e41e8fa-bf57-48a1-b2a9-ab6597246231/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:11:03.183 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:11:03.183 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:11:03.183 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0238694 s, 43.9 MB/s 2021-02-10T11:11:03.184 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000344402 2021-02-10T11:11:03.614 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:03.602 7fccd0f4c700 1 Processor -- start 2021-02-10T11:11:03.615 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:03.602 7fccd0f4c700 1 -- start start 2021-02-10T11:11:03.615 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:03.602 7fccd0f4c700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fcccc1b0470 0x7fcccc1b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:11:03.615 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:03.602 7fccd0f4c700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fcccc0ef610 0x7fcccc0efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:11:03.615 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:03.602 7fccd0f4c700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcccc0f0030 0x7fcccc0f0430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:11:03.615 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:03.602 7fccd0f4c700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcccc0e5ea0 con 0x7fcccc0ef610 2021-02-10T11:11:03.615 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:03.602 7fccd0f4c700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcccc0f0b50 con 0x7fcccc1b0470 2021-02-10T11:11:03.615 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:03.602 7fccd0f4c700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcccc0e52e0 con 0x7fcccc0f0030 2021-02-10T11:11:03.615 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:03.603 7fccca312700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fcccc0ef610 0x7fcccc0efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:11:03.615 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:03.603 7fccca312700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fcccc0ef610 0x7fcccc0efa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:51998/0 (socket says 172.21.2.102:51998) 2021-02-10T11:11:03.615 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:03.603 7fccca312700 1 -- 172.21.2.102:0/4002529753 learned_addr learned my addr 172.21.2.102:0/4002529753 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:11:03.615 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:03.603 7fccca312700 1 -- 172.21.2.102:0/4002529753 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcccc0f0030 msgr2=0x7fcccc0f0430 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:11:03.615 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:03.603 7fccca312700 1 --2- 172.21.2.102:0/4002529753 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcccc0f0030 0x7fcccc0f0430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:11:03.615 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:03.603 7fccca312700 1 -- 172.21.2.102:0/4002529753 >> v2:172.21.2.113:3300/0 conn(0x7fcccc1b0470 msgr2=0x7fcccc1b0870 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:11:03.616 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:03.603 7fccca312700 1 --2- 172.21.2.102:0/4002529753 >> v2:172.21.2.113:3300/0 conn(0x7fcccc1b0470 0x7fcccc1b0870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:11:03.616 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:03.603 7fccca312700 1 -- 172.21.2.102:0/4002529753 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcccc0f10d0 con 0x7fcccc0ef610 2021-02-10T11:11:03.616 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:03.603 7fccca312700 1 --2- 172.21.2.102:0/4002529753 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcccc0ef610 0x7fcccc0efa10 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 rx=0x7fccb4006700 tx=0x7fccb4009b80).ready entity=mon.0 client_cookie=b85b12e56501b122 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:11:03.616 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:03.603 7fccc9b11700 1 -- 172.21.2.102:0/4002529753 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fccb400c0c0 con 0x7fcccc0ef610 2021-02-10T11:11:03.616 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:03.603 7fccc9b11700 1 -- 172.21.2.102:0/4002529753 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccb400c340 con 0x7fcccc0ef610 2021-02-10T11:11:03.616 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:03.603 7fccc9b11700 1 -- 172.21.2.102:0/4002529753 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fccb400c7a0 con 0x7fcccc0ef610 2021-02-10T11:11:03.616 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:03.603 7fccd0f4c700 1 -- 172.21.2.102:0/4002529753 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcccc0ef610 msgr2=0x7fcccc0efa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:11:03.616 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:03.603 7fccd0f4c700 1 --2- 172.21.2.102:0/4002529753 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcccc0ef610 0x7fcccc0efa10 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 rx=0x7fccb4006700 tx=0x7fccb4009b80).stop 2021-02-10T11:11:03.616 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:03.604 7fccd0f4c700 1 -- 172.21.2.102:0/4002529753 shutdown_connections 2021-02-10T11:11:03.616 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:03.604 7fccd0f4c700 1 --2- 172.21.2.102:0/4002529753 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcccc0f0030 0x7fcccc0f0430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:11:03.616 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:03.604 7fccd0f4c700 1 --2- 172.21.2.102:0/4002529753 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcccc0ef610 0x7fcccc0efa10 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:11:03.616 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:03.604 7fccd0f4c700 1 --2- 172.21.2.102:0/4002529753 >> v2:172.21.2.113:3300/0 conn(0x7fcccc1b0470 0x7fcccc1b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:11:03.616 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:03.604 7fccd0f4c700 1 -- 172.21.2.102:0/4002529753 >> 172.21.2.102:0/4002529753 conn(0x7fcccc0ce3e0 msgr2=0x7fcccc0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:11:03.617 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:03.604 7fccd0f4c700 1 -- 172.21.2.102:0/4002529753 shutdown_connections 2021-02-10T11:11:03.617 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:03.604 7fccd0f4c700 1 -- 172.21.2.102:0/4002529753 wait complete. 2021-02-10T11:11:03.618 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:03.619 7fccd0f4c700 1 Processor -- start 2021-02-10T11:11:03.618 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:03.619 7fccd0f4c700 1 -- start start 2021-02-10T11:11:03.618 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:03.619 7fccd0f4c700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcccc0b20b0 0x7fcccc0ac970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:11:03.618 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:03.619 7fccd0f4c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcccc0b24b0 0x7fcccc191dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:11:03.618 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:03.619 7fccd0f4c700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fcccc0acf90 0x7fcccc1923e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:11:03.618 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:03.619 7fccd0f4c700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcccc0b1b30 con 0x7fcccc0b24b0 2021-02-10T11:11:03.618 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:03.619 7fccd0f4c700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcccc1a04e0 con 0x7fcccc0acf90 2021-02-10T11:11:03.619 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:03.619 7fccd0f4c700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcccc1a06c0 con 0x7fcccc0b20b0 2021-02-10T11:11:03.619 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:03.619 7fccca312700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcccc0b24b0 0x7fcccc191dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:11:03.619 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:03.619 7fccca312700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcccc0b24b0 0x7fcccc191dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:52006/0 (socket says 172.21.2.102:52006) 2021-02-10T11:11:03.619 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:03.619 7fccca312700 1 -- 172.21.2.102:0/642340755 learned_addr learned my addr 172.21.2.102:0/642340755 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:11:03.619 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:03.620 7fccca312700 1 -- 172.21.2.102:0/642340755 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcccc0b20b0 msgr2=0x7fcccc0ac970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:11:03.619 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:03.620 7fccca312700 1 --2- 172.21.2.102:0/642340755 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcccc0b20b0 0x7fcccc0ac970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:11:03.619 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:03.620 7fccca312700 1 -- 172.21.2.102:0/642340755 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fcccc0acf90 msgr2=0x7fcccc1923e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:11:03.619 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:03.620 7fccca312700 1 --2- 172.21.2.102:0/642340755 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fcccc0acf90 0x7fcccc1923e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:11:03.619 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:03.620 7fccca312700 1 -- 172.21.2.102:0/642340755 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcccc192be0 con 0x7fcccc0b24b0 2021-02-10T11:11:03.619 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:03.620 7fccca312700 1 --2- 172.21.2.102:0/642340755 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcccc0b24b0 0x7fcccc191dc0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 rx=0x7fccb400b5e0 tx=0x7fccb400b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:11:03.619 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:03.620 7fccc37fe700 1 -- 172.21.2.102:0/642340755 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fccb400bd70 con 0x7fcccc0b24b0 2021-02-10T11:11:03.619 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:03.620 7fccc37fe700 1 -- 172.21.2.102:0/642340755 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccb4002030 con 0x7fcccc0b24b0 2021-02-10T11:11:03.619 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:03.620 7fccd0f4c700 1 -- 172.21.2.102:0/642340755 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcccc192be0 con 0x7fcccc0b24b0 2021-02-10T11:11:03.620 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:03.620 7fccd0f4c700 1 -- 172.21.2.102:0/642340755 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcccc193170 con 0x7fcccc0b24b0 2021-02-10T11:11:03.620 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:03.620 7fccc37fe700 1 -- 172.21.2.102:0/642340755 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fccb4002550 con 0x7fcccc0b24b0 2021-02-10T11:11:03.620 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:03.621 7fccc37fe700 1 -- 172.21.2.102:0/642340755 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fccb4000c90 con 0x7fcccc0b24b0 2021-02-10T11:11:03.620 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:03.621 7fccc37fe700 1 --2- 172.21.2.102:0/642340755 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fccb0018730 0x7fccb001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:11:03.620 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:03.621 7fccc37fe700 1 -- 172.21.2.102:0/642340755 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fccb4032690 con 0x7fcccc0b24b0 2021-02-10T11:11:03.620 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:03.621 7fcccab13700 1 --2- 172.21.2.102:0/642340755 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fccb0018730 0x7fccb001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:11:03.620 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:03.621 7fcccab13700 1 --2- 172.21.2.102:0/642340755 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fccb0018730 0x7fccb001ac10 crc :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:11:03.649 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:03.653 7fccd0f4c700 1 -- 172.21.2.102:0/642340755 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcccc105af0 con 0x7fcccc0b24b0 2021-02-10T11:11:03.651 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:03.655 7fccc37fe700 1 -- 172.21.2.102:0/642340755 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fccb4000940 con 0x7fcccc0b24b0 2021-02-10T11:11:03.777 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:03.781 7fccd0f4c700 1 -- 172.21.2.102:0/642340755 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344402", "target": ["mgr", ""]}) v1 -- 0x7fcccc0b1b30 con 0x7fccb0018730 2021-02-10T11:11:03.803 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:03.807 7fccc37fe700 1 -- 172.21.2.102:0/642340755 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fccbc006e40 con 0x7fccb0018730 2021-02-10T11:11:03.810 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:03.814 7fccd0f4c700 1 -- 172.21.2.102:0/642340755 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fccb0018730 msgr2=0x7fccb001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:11:03.810 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:03.814 7fccd0f4c700 1 --2- 172.21.2.102:0/642340755 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fccb0018730 0x7fccb001ac10 crc :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:11:03.810 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:03.814 7fccd0f4c700 1 -- 172.21.2.102:0/642340755 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcccc0b24b0 msgr2=0x7fcccc191dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:11:03.810 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:03.814 7fccd0f4c700 1 --2- 172.21.2.102:0/642340755 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcccc0b24b0 0x7fcccc191dc0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 rx=0x7fccb400b5e0 tx=0x7fccb400b980).stop 2021-02-10T11:11:03.811 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:03.814 7fccd0f4c700 1 -- 172.21.2.102:0/642340755 shutdown_connections 2021-02-10T11:11:03.811 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:03.814 7fccd0f4c700 1 --2- 172.21.2.102:0/642340755 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fccb0018730 0x7fccb001ac10 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:11:03.811 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:03.814 7fccd0f4c700 1 --2- 172.21.2.102:0/642340755 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fcccc0acf90 0x7fcccc1923e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:11:03.811 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:03.814 7fccd0f4c700 1 --2- 172.21.2.102:0/642340755 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcccc0b24b0 0x7fcccc191dc0 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:11:03.811 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:03.814 7fccd0f4c700 1 --2- 172.21.2.102:0/642340755 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcccc0b20b0 0x7fcccc0ac970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:11:03.811 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:03.814 7fccd0f4c700 1 -- 172.21.2.102:0/642340755 >> 172.21.2.102:0/642340755 conn(0x7fcccc0bf4e0 msgr2=0x7fcccc0be3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:11:03.811 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:03.814 7fccd0f4c700 1 -- 172.21.2.102:0/642340755 shutdown_connections 2021-02-10T11:11:03.811 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:03.814 7fccd0f4c700 1 -- 172.21.2.102:0/642340755 wait complete. 2021-02-10T11:11:03.819 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000344402 2021-02-10T11:11:04.251 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:04.237 7f9154746700 1 Processor -- start 2021-02-10T11:11:04.251 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:04.237 7f9154746700 1 -- start start 2021-02-10T11:11:04.251 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:04.238 7f9154746700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f914c1b0870 0x7f914c0f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:11:04.252 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:04.238 7f9154746700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f914c1b0c70 con 0x7f914c1b0470 2021-02-10T11:11:04.252 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:04.238 7f9154746700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f914c0f0b50 con 0x7f914c0efe00 2021-02-10T11:11:04.252 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:04.238 7f9154746700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f914c0e5ea0 con 0x7f914c1b0870 2021-02-10T11:11:04.252 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:04.238 7f9152562700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f914c1b0470 0x7f914c0ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:44234/0 (socket says 172.21.2.102:44234) 2021-02-10T11:11:04.252 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:04.238 7f9152562700 1 -- 172.21.2.102:0/1404596462 learned_addr learned my addr 172.21.2.102:0/1404596462 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:11:04.252 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:04.238 7f9151560700 1 -- 172.21.2.102:0/1404596462 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 668511319 0 0) 0x7f9140000ce0 con 0x7f914c1b0470 2021-02-10T11:11:04.252 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:04.238 7f9151560700 1 -- 172.21.2.102:0/1404596462 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9138002fe0 con 0x7f914c1b0470 2021-02-10T11:11:04.252 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:04.238 7f9151d61700 1 --2- 172.21.2.102:0/1404596462 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f914c1b0870 0x7f914c0f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:11:04.252 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:04.238 7f9151560700 1 -- 172.21.2.102:0/1404596462 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2801790606 0 0) 0x7f9140001180 con 0x7f914c1b0470 2021-02-10T11:11:04.252 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:04.238 7f9151560700 1 -- 172.21.2.102:0/1404596462 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f914c1b0870 msgr2=0x7f914c0f20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:11:04.253 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:04.238 7f9151560700 1 --2- 172.21.2.102:0/1404596462 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f914c1b0870 0x7f914c0f20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:11:04.253 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:04.238 7f9151560700 1 -- 172.21.2.102:0/1404596462 >> v1:172.21.2.113:6789/0 conn(0x7f914c0efe00 legacy=0x7f914c0f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:11:04.253 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:04.239 7f9151560700 1 -- 172.21.2.102:0/1404596462 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f914c0e52e0 con 0x7f914c1b0470 2021-02-10T11:11:04.253 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:04.239 7f9151560700 1 -- 172.21.2.102:0/1404596462 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7f9140001680 con 0x7f914c1b0470 2021-02-10T11:11:04.253 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:04.239 7f9151560700 1 -- 172.21.2.102:0/1404596462 >> v1:172.21.2.102:6789/0 conn(0x7f914c1b0470 legacy=0x7f914c0ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:11:04.253 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:04.239 7f9151560700 1 --2- 172.21.2.102:0/1404596462 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9138003a60 0x7f9138005eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:11:04.253 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:04.239 7f9151560700 1 --2- 172.21.2.102:0/1404596462 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9138006750 0x7f9138008c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:11:04.253 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:04.239 7f9151560700 1 --2- 172.21.2.102:0/1404596462 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f9138009510 0x7f913800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:11:04.253 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:04.239 7f9151560700 1 -- 172.21.2.102:0/1404596462 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f913800c2c0 con 0x7f9138006750 2021-02-10T11:11:04.253 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:04.239 7f9151560700 1 -- 172.21.2.102:0/1404596462 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f913800c4e0 con 0x7f9138009510 2021-02-10T11:11:04.253 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:04.239 7f9151560700 1 -- 172.21.2.102:0/1404596462 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f913800c700 con 0x7f9138003a60 2021-02-10T11:11:04.253 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:04.239 7f9151d61700 1 --2- 172.21.2.102:0/1404596462 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9138006750 0x7f9138008c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:11:04.253 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:04.239 7f9152562700 1 --2- 172.21.2.102:0/1404596462 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9138003a60 0x7f9138005eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:11:04.253 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:04.239 7f9152d63700 1 --2- 172.21.2.102:0/1404596462 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f9138009510 0x7f913800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:11:04.254 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:04.239 7f9151d61700 1 -- 172.21.2.102:0/1404596462 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9138003a60 msgr2=0x7f9138005eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:11:04.254 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:04.239 7f9151d61700 1 --2- 172.21.2.102:0/1404596462 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9138003a60 0x7f9138005eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:11:04.254 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:04.239 7f9151d61700 1 -- 172.21.2.102:0/1404596462 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f9138009510 msgr2=0x7f913800ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:11:04.254 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:04.239 7f9151d61700 1 --2- 172.21.2.102:0/1404596462 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f9138009510 0x7f913800ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:11:04.254 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:04.239 7f9151d61700 1 -- 172.21.2.102:0/1404596462 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f913800c9a0 con 0x7f9138006750 2021-02-10T11:11:04.254 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:04.239 7f9151d61700 1 --2- 172.21.2.102:0/1404596462 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9138006750 0x7f9138008c70 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 rx=0x7f914804da20 tx=0x7f914804ddc0).ready entity=mon.0 client_cookie=e56a98d35d07e373 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:11:04.254 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:04.239 7f9151560700 1 -- 172.21.2.102:0/1404596462 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9148006240 con 0x7f9138006750 2021-02-10T11:11:04.254 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:04.239 7f9151560700 1 -- 172.21.2.102:0/1404596462 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9148006540 con 0x7f9138006750 2021-02-10T11:11:04.254 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:04.239 7f9151560700 1 -- 172.21.2.102:0/1404596462 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f91480058e0 con 0x7f9138006750 2021-02-10T11:11:04.254 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:04.240 7f9154746700 1 -- 172.21.2.102:0/1404596462 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9138006750 msgr2=0x7f9138008c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:11:04.254 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:04.240 7f9154746700 1 --2- 172.21.2.102:0/1404596462 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9138006750 0x7f9138008c70 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 rx=0x7f914804da20 tx=0x7f914804ddc0).stop 2021-02-10T11:11:04.254 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:04.240 7f9152d63700 1 -- 172.21.2.102:0/1404596462 reap_dead start 2021-02-10T11:11:04.255 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:04.240 7f9154746700 1 -- 172.21.2.102:0/1404596462 shutdown_connections 2021-02-10T11:11:04.255 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:04.240 7f9154746700 1 -- 172.21.2.102:0/1404596462 >> 172.21.2.102:0/1404596462 conn(0x7f914c0ce3e0 msgr2=0x7f914c0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:11:04.255 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:04.241 7f9154746700 1 -- 172.21.2.102:0/1404596462 shutdown_connections 2021-02-10T11:11:04.255 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:04.241 7f9154746700 1 -- 172.21.2.102:0/1404596462 wait complete. 2021-02-10T11:11:04.255 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:04.255 7f9154746700 1 Processor -- start 2021-02-10T11:11:04.255 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:04.255 7f9154746700 1 -- start start 2021-02-10T11:11:04.256 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:04.256 7f9154746700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f914c09b930 0x7f914c0b7c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:11:04.256 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:04.256 7f9154746700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f914c0b8410 0x7f914c0b8810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:11:04.256 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:04.256 7f9154746700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f914c0c1f80 0x7f914c0c2380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:11:04.256 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:04.256 7f9154746700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f914c0c1a00 con 0x7f914c09b930 2021-02-10T11:11:04.257 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:04.256 7f9154746700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f914c0b8e30 con 0x7f914c0c1f80 2021-02-10T11:11:04.257 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:04.256 7f9154746700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f914c0b9050 con 0x7f914c0b8410 2021-02-10T11:11:04.257 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:04.256 7f9152562700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f914c09b930 0x7f914c0b7c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:11:04.257 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:04.256 7f9152562700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f914c09b930 0x7f914c0b7c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:52024/0 (socket says 172.21.2.102:52024) 2021-02-10T11:11:04.257 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:04.256 7f9152562700 1 -- 172.21.2.102:0/882561129 learned_addr learned my addr 172.21.2.102:0/882561129 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:11:04.257 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:04.256 7f9152562700 1 -- 172.21.2.102:0/882561129 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f914c0b8410 msgr2=0x7f914c0b8810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:11:04.257 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:04.256 7f9152562700 1 --2- 172.21.2.102:0/882561129 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f914c0b8410 0x7f914c0b8810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:11:04.257 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:04.256 7f9152562700 1 -- 172.21.2.102:0/882561129 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f914c0c1f80 msgr2=0x7f914c0c2380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:11:04.257 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:04.256 7f9152562700 1 --2- 172.21.2.102:0/882561129 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f914c0c1f80 0x7f914c0c2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:11:04.257 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:04.257 7f9152562700 1 -- 172.21.2.102:0/882561129 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f914c0c29a0 con 0x7f914c09b930 2021-02-10T11:11:04.257 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:04.257 7f9152562700 1 --2- 172.21.2.102:0/882561129 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f914c09b930 0x7f914c0b7c20 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 rx=0x7f9140009610 tx=0x7f91400099b0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:11:04.257 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:04.257 7f913effd700 1 -- 172.21.2.102:0/882561129 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f91400070c0 con 0x7f914c09b930 2021-02-10T11:11:04.257 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:04.257 7f913effd700 1 -- 172.21.2.102:0/882561129 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91400043a0 con 0x7f914c09b930 2021-02-10T11:11:04.258 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:04.257 7f913effd700 1 -- 172.21.2.102:0/882561129 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9140004830 con 0x7f914c09b930 2021-02-10T11:11:04.258 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:04.257 7f9154746700 1 -- 172.21.2.102:0/882561129 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f914c0c29a0 con 0x7f914c09b930 2021-02-10T11:11:04.258 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:04.257 7f9154746700 1 -- 172.21.2.102:0/882561129 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f914c0c2f30 con 0x7f914c09b930 2021-02-10T11:11:04.258 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:04.258 7f913effd700 1 -- 172.21.2.102:0/882561129 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f9140007a70 con 0x7f914c09b930 2021-02-10T11:11:04.258 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:04.258 7f913effd700 1 --2- 172.21.2.102:0/882561129 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f91340186c0 0x7f913401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:11:04.258 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:04.258 7f913effd700 1 -- 172.21.2.102:0/882561129 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f9140032570 con 0x7f914c09b930 2021-02-10T11:11:04.258 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:04.258 7f9151d61700 1 --2- 172.21.2.102:0/882561129 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f91340186c0 0x7f913401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:11:04.258 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:04.258 7f9151d61700 1 --2- 172.21.2.102:0/882561129 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f91340186c0 0x7f913401abc0 crc :-1 s=READY pgs=249 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:11:04.283 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:04.287 7f9154746700 1 -- 172.21.2.102:0/882561129 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f914c105b30 con 0x7f914c09b930 2021-02-10T11:11:04.286 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:04.290 7f913effd700 1 -- 172.21.2.102:0/882561129 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f9140030d20 con 0x7f914c09b930 2021-02-10T11:11:04.414 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:04.418 7f9154746700 1 -- 172.21.2.102:0/882561129 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344402", "target": ["mgr", ""]}) v1 -- 0x7f914c0f0b50 con 0x7f91340186c0 2021-02-10T11:11:04.416 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:04.421 7f913effd700 1 -- 172.21.2.102:0/882561129 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f9148006ce0 con 0x7f91340186c0 2021-02-10T11:11:04.417 INFO:teuthology.orchestra.run.gibba002.stdout:/volumes/_nogroup/subvolume_0000000000344402/b180f782-13c4-43a7-8abe-c26ece759760 2021-02-10T11:11:04.419 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:04.422 7f9154746700 1 -- 172.21.2.102:0/882561129 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f91340186c0 msgr2=0x7f913401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:11:04.419 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:04.422 7f9154746700 1 --2- 172.21.2.102:0/882561129 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f91340186c0 0x7f913401abc0 crc :-1 s=READY pgs=249 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:11:04.419 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:04.422 7f9154746700 1 -- 172.21.2.102:0/882561129 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f914c09b930 msgr2=0x7f914c0b7c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:11:04.419 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:04.422 7f9154746700 1 --2- 172.21.2.102:0/882561129 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f914c09b930 0x7f914c0b7c20 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 rx=0x7f9140009610 tx=0x7f91400099b0).stop 2021-02-10T11:11:04.419 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:04.422 7f9154746700 1 -- 172.21.2.102:0/882561129 shutdown_connections 2021-02-10T11:11:04.419 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:04.422 7f9154746700 1 --2- 172.21.2.102:0/882561129 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f91340186c0 0x7f913401abc0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:11:04.419 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:04.422 7f9154746700 1 --2- 172.21.2.102:0/882561129 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f914c0c1f80 0x7f914c0c2380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:11:04.419 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:04.422 7f9154746700 1 --2- 172.21.2.102:0/882561129 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f914c0b8410 0x7f914c0b8810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:11:04.419 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:04.422 7f9154746700 1 --2- 172.21.2.102:0/882561129 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f914c09b930 0x7f914c0b7c20 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:11:04.419 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:04.422 7f9154746700 1 -- 172.21.2.102:0/882561129 >> 172.21.2.102:0/882561129 conn(0x7f914c094e10 msgr2=0x7f914c0a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:11:04.420 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:04.423 7f9154746700 1 -- 172.21.2.102:0/882561129 shutdown_connections 2021-02-10T11:11:04.420 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:04.423 7f9154746700 1 -- 172.21.2.102:0/882561129 wait complete. 2021-02-10T11:11:04.427 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000344402 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000344402/b180f782-13c4-43a7-8abe-c26ece759760 2021-02-10T11:11:04.427 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:11:04.427 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:11:04.427 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:11:04.762 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:11:04.763 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:11:04.763 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344402/b180f782-13c4-43a7-8abe-c26ece759760/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:11:05.145 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:11:05.146 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:11:05.146 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0247692 s, 42.3 MB/s 2021-02-10T11:11:05.148 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:11:05.148 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:11:05.149 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:11:05.489 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:11:05.489 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:11:05.489 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344402/b180f782-13c4-43a7-8abe-c26ece759760/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:11:05.865 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:11:05.865 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:11:05.865 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0244704 s, 42.9 MB/s 2021-02-10T11:11:05.866 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:11:05.866 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:11:05.866 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:11:06.200 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:11:06.200 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:11:06.201 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344402/b180f782-13c4-43a7-8abe-c26ece759760/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:11:06.577 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:11:06.577 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:11:06.577 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0257526 s, 40.7 MB/s 2021-02-10T11:11:06.578 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:11:06.578 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:11:06.578 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:11:06.915 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:11:06.915 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:11:06.915 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344402/b180f782-13c4-43a7-8abe-c26ece759760/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:11:07.293 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:11:10.799 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:11:10.799 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0241662 s, 43.4 MB/s 2021-02-10T11:11:10.800 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:11:10.800 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:11:10.800 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:11:11.148 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:11:11.148 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:11:11.148 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344402/b180f782-13c4-43a7-8abe-c26ece759760/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:11:11.530 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:11:11.530 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:11:11.530 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0237773 s, 44.1 MB/s 2021-02-10T11:11:11.531 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:11:11.531 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:11:11.532 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:11:11.877 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:11:11.878 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:11:11.878 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344402/b180f782-13c4-43a7-8abe-c26ece759760/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:11:12.255 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:11:12.255 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:11:12.255 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0237569 s, 44.1 MB/s 2021-02-10T11:11:12.256 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:11:12.256 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:11:12.257 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:11:12.592 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:11:12.593 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:11:12.593 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344402/b180f782-13c4-43a7-8abe-c26ece759760/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:11:12.966 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:11:12.966 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:11:12.967 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0224012 s, 46.8 MB/s 2021-02-10T11:11:12.967 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:11:12.968 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:11:12.968 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:11:13.303 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:11:13.303 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:11:13.303 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344402/b180f782-13c4-43a7-8abe-c26ece759760/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:11:13.676 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:11:13.676 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:11:13.676 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0241288 s, 43.5 MB/s 2021-02-10T11:11:13.677 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:11:13.677 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:11:13.677 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:11:14.013 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:11:14.013 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:11:14.013 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344402/b180f782-13c4-43a7-8abe-c26ece759760/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:11:14.389 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:11:14.389 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:11:14.390 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0248712 s, 42.2 MB/s 2021-02-10T11:11:14.390 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:11:14.391 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:11:14.391 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:11:14.725 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:11:14.726 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:11:14.726 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344402/b180f782-13c4-43a7-8abe-c26ece759760/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:11:15.100 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:11:15.100 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:11:15.100 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0229003 s, 45.8 MB/s 2021-02-10T11:11:15.101 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000344403 2021-02-10T11:11:15.532 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:15.518 7fda856c2700 1 Processor -- start 2021-02-10T11:11:15.532 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:15.518 7fda856c2700 1 -- start start 2021-02-10T11:11:15.532 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:15.519 7fda856c2700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fda801b0870 0x7fda800f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:11:15.532 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:15.519 7fda856c2700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda801b0c70 con 0x7fda801b0470 2021-02-10T11:11:15.532 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:15.519 7fda856c2700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda800f0b50 con 0x7fda800efe00 2021-02-10T11:11:15.532 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:15.519 7fda856c2700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda800e5ea0 con 0x7fda801b0870 2021-02-10T11:11:15.532 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:15.519 7fda7f2bf700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fda801b0470 0x7fda800ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:44252/0 (socket says 172.21.2.102:44252) 2021-02-10T11:11:15.532 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:15.519 7fda7f2bf700 1 -- 172.21.2.102:0/3415789315 learned_addr learned my addr 172.21.2.102:0/3415789315 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:11:15.532 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:15.520 7fda7e2bd700 1 -- 172.21.2.102:0/3415789315 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1473864413 0 0) 0x7fda680010b0 con 0x7fda801b0470 2021-02-10T11:11:15.532 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:15.520 7fda7e2bd700 1 -- 172.21.2.102:0/3415789315 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda6c002fe0 con 0x7fda801b0470 2021-02-10T11:11:15.532 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:15.520 7fda7eabe700 1 --2- 172.21.2.102:0/3415789315 >> v2:172.21.2.113:3300/0 conn(0x7fda801b0870 0x7fda800f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:11:15.533 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:15.520 7fda7e2bd700 1 -- 172.21.2.102:0/3415789315 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3112051381 0 0) 0x7fda680010b0 con 0x7fda801b0470 2021-02-10T11:11:15.533 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:15.520 7fda7e2bd700 1 -- 172.21.2.102:0/3415789315 >> v2:172.21.2.113:3300/0 conn(0x7fda801b0870 msgr2=0x7fda800f20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:11:15.533 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:15.520 7fda7e2bd700 1 --2- 172.21.2.102:0/3415789315 >> v2:172.21.2.113:3300/0 conn(0x7fda801b0870 0x7fda800f20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:11:15.533 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:15.520 7fda7e2bd700 1 -- 172.21.2.102:0/3415789315 >> v1:172.21.2.113:6789/0 conn(0x7fda800efe00 legacy=0x7fda800f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:11:15.533 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:15.520 7fda7e2bd700 1 -- 172.21.2.102:0/3415789315 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda800e52e0 con 0x7fda801b0470 2021-02-10T11:11:15.533 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:15.520 7fda7e2bd700 1 -- 172.21.2.102:0/3415789315 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7fda680015b0 con 0x7fda801b0470 2021-02-10T11:11:15.533 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:15.520 7fda7e2bd700 1 -- 172.21.2.102:0/3415789315 >> v1:172.21.2.102:6789/0 conn(0x7fda801b0470 legacy=0x7fda800ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:11:15.533 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:15.520 7fda7e2bd700 1 --2- 172.21.2.102:0/3415789315 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fda6c003a70 0x7fda6c005f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:11:15.533 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:15.520 7fda7e2bd700 1 --2- 172.21.2.102:0/3415789315 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fda6c006780 0x7fda6c008ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:11:15.533 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:15.520 7fda7e2bd700 1 --2- 172.21.2.102:0/3415789315 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fda6c0095a0 0x7fda6c00bac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:11:15.533 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:15.520 7fda7e2bd700 1 -- 172.21.2.102:0/3415789315 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda6c00c310 con 0x7fda6c0095a0 2021-02-10T11:11:15.533 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:15.520 7fda7e2bd700 1 -- 172.21.2.102:0/3415789315 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda6c00c530 con 0x7fda6c003a70 2021-02-10T11:11:15.533 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:15.520 7fda7e2bd700 1 -- 172.21.2.102:0/3415789315 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda6c00c750 con 0x7fda6c006780 2021-02-10T11:11:15.534 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:15.520 7fda7fac0700 1 --2- 172.21.2.102:0/3415789315 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fda6c0095a0 0x7fda6c00bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:11:15.534 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:15.520 7fda7eabe700 1 --2- 172.21.2.102:0/3415789315 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fda6c006780 0x7fda6c008ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:11:15.534 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:15.520 7fda7f2bf700 1 --2- 172.21.2.102:0/3415789315 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fda6c003a70 0x7fda6c005f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:11:15.534 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:15.521 7fda7fac0700 1 -- 172.21.2.102:0/3415789315 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fda6c006780 msgr2=0x7fda6c008ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:11:15.534 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:15.521 7fda7fac0700 1 --2- 172.21.2.102:0/3415789315 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fda6c006780 0x7fda6c008ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:11:15.534 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:15.521 7fda7fac0700 1 -- 172.21.2.102:0/3415789315 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fda6c003a70 msgr2=0x7fda6c005f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:11:15.534 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:15.521 7fda7fac0700 1 --2- 172.21.2.102:0/3415789315 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fda6c003a70 0x7fda6c005f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:11:15.534 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:15.521 7fda7fac0700 1 -- 172.21.2.102:0/3415789315 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda6c00c9f0 con 0x7fda6c0095a0 2021-02-10T11:11:15.534 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:15.521 7fda7fac0700 1 -- 172.21.2.102:0/3415789315 reap_dead start 2021-02-10T11:11:15.534 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:15.521 7fda7fac0700 1 --2- 172.21.2.102:0/3415789315 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fda6c0095a0 0x7fda6c00bac0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 rx=0x7fda7804ae40 tx=0x7fda7804de20).ready entity=mon.0 client_cookie=5f90e182c34a2b7 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:11:15.534 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:15.521 7fda7e2bd700 1 -- 172.21.2.102:0/3415789315 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fda78001400 con 0x7fda6c0095a0 2021-02-10T11:11:15.534 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:15.521 7fda7e2bd700 1 -- 172.21.2.102:0/3415789315 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda78003230 con 0x7fda6c0095a0 2021-02-10T11:11:15.534 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:15.521 7fda7e2bd700 1 -- 172.21.2.102:0/3415789315 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fda78001920 con 0x7fda6c0095a0 2021-02-10T11:11:15.535 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:15.521 7fda856c2700 1 -- 172.21.2.102:0/3415789315 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fda6c0095a0 msgr2=0x7fda6c00bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:11:15.535 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:15.521 7fda856c2700 1 --2- 172.21.2.102:0/3415789315 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fda6c0095a0 0x7fda6c00bac0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 rx=0x7fda7804ae40 tx=0x7fda7804de20).stop 2021-02-10T11:11:15.535 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:15.521 7fda856c2700 1 -- 172.21.2.102:0/3415789315 shutdown_connections 2021-02-10T11:11:15.535 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:15.521 7fda856c2700 1 --2- 172.21.2.102:0/3415789315 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fda6c0095a0 0x7fda6c00bac0 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:11:15.535 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:15.521 7fda856c2700 1 -- 172.21.2.102:0/3415789315 >> 172.21.2.102:0/3415789315 conn(0x7fda800ce3e0 msgr2=0x7fda800e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:11:15.535 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:15.521 7fda856c2700 1 -- 172.21.2.102:0/3415789315 shutdown_connections 2021-02-10T11:11:15.535 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:15.522 7fda856c2700 1 -- 172.21.2.102:0/3415789315 wait complete. 2021-02-10T11:11:15.536 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:15.536 7fda856c2700 1 Processor -- start 2021-02-10T11:11:15.537 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:15.536 7fda856c2700 1 -- start start 2021-02-10T11:11:15.537 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:15.537 7fda856c2700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fda800a0910 0x7fda80105d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:11:15.537 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:15.537 7fda856c2700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fda801a7a10 0x7fda801a7e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:11:15.537 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:15.537 7fda856c2700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fda80093770 0x7fda80093b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:11:15.537 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:15.537 7fda856c2700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda80106390 con 0x7fda800a0910 2021-02-10T11:11:15.537 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:15.537 7fda856c2700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda80094190 con 0x7fda80093770 2021-02-10T11:11:15.537 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:15.537 7fda856c2700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda800943b0 con 0x7fda801a7a10 2021-02-10T11:11:15.537 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:15.537 7fda7f2bf700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fda800a0910 0x7fda80105d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:11:15.537 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:15.537 7fda7f2bf700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fda800a0910 0x7fda80105d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:52040/0 (socket says 172.21.2.102:52040) 2021-02-10T11:11:15.537 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:15.537 7fda7f2bf700 1 -- 172.21.2.102:0/3482551886 learned_addr learned my addr 172.21.2.102:0/3482551886 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:11:15.537 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:15.537 7fda7eabe700 1 --2- 172.21.2.102:0/3482551886 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fda801a7a10 0x7fda801a7e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:11:15.537 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:15.537 7fda7fac0700 1 --2- 172.21.2.102:0/3482551886 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fda80093770 0x7fda80093b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:11:15.537 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:15.537 7fda7f2bf700 1 -- 172.21.2.102:0/3482551886 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fda801a7a10 msgr2=0x7fda801a7e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:11:15.538 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:15.537 7fda7f2bf700 1 --2- 172.21.2.102:0/3482551886 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fda801a7a10 0x7fda801a7e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:11:15.538 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:15.537 7fda7f2bf700 1 -- 172.21.2.102:0/3482551886 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fda80093770 msgr2=0x7fda80093b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:11:15.538 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:15.537 7fda7f2bf700 1 --2- 172.21.2.102:0/3482551886 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fda80093770 0x7fda80093b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:11:15.538 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:15.537 7fda7f2bf700 1 -- 172.21.2.102:0/3482551886 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda80106c30 con 0x7fda800a0910 2021-02-10T11:11:15.538 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:15.537 7fda7f2bf700 1 --2- 172.21.2.102:0/3482551886 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fda800a0910 0x7fda80105d70 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 rx=0x7fda68007650 tx=0x7fda6800a900).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:11:15.538 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:15.538 7fda77fff700 1 -- 172.21.2.102:0/3482551886 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fda6800acf0 con 0x7fda800a0910 2021-02-10T11:11:15.538 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:15.538 7fda77fff700 1 -- 172.21.2.102:0/3482551886 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda68004350 con 0x7fda800a0910 2021-02-10T11:11:15.538 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:15.538 7fda77fff700 1 -- 172.21.2.102:0/3482551886 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fda68004870 con 0x7fda800a0910 2021-02-10T11:11:15.538 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:15.538 7fda856c2700 1 -- 172.21.2.102:0/3482551886 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda80106c30 con 0x7fda800a0910 2021-02-10T11:11:15.538 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:15.538 7fda856c2700 1 -- 172.21.2.102:0/3482551886 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda801071c0 con 0x7fda800a0910 2021-02-10T11:11:15.538 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:15.538 7fda77fff700 1 -- 172.21.2.102:0/3482551886 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fda68004530 con 0x7fda800a0910 2021-02-10T11:11:15.538 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:15.538 7fda77fff700 1 --2- 172.21.2.102:0/3482551886 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fda640186c0 0x7fda6401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:11:15.538 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:15.538 7fda77fff700 1 -- 172.21.2.102:0/3482551886 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fda68032510 con 0x7fda800a0910 2021-02-10T11:11:15.539 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:15.538 7fda7eabe700 1 --2- 172.21.2.102:0/3482551886 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fda640186c0 0x7fda6401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:11:15.539 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:15.539 7fda7eabe700 1 --2- 172.21.2.102:0/3482551886 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fda640186c0 0x7fda6401abc0 crc :-1 s=READY pgs=250 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:11:15.565 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:15.569 7fda856c2700 1 -- 172.21.2.102:0/3482551886 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda80105b40 con 0x7fda800a0910 2021-02-10T11:11:15.567 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:15.571 7fda77fff700 1 -- 172.21.2.102:0/3482551886 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fda6802dfd0 con 0x7fda800a0910 2021-02-10T11:11:15.693 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:15.697 7fda856c2700 1 -- 172.21.2.102:0/3482551886 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344403", "target": ["mgr", ""]}) v1 -- 0x7fda801071c0 con 0x7fda640186c0 2021-02-10T11:11:15.717 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:15.721 7fda77fff700 1 -- 172.21.2.102:0/3482551886 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fda700009b0 con 0x7fda640186c0 2021-02-10T11:11:15.726 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:15.730 7fda856c2700 1 -- 172.21.2.102:0/3482551886 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fda640186c0 msgr2=0x7fda6401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:11:15.726 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:15.730 7fda856c2700 1 --2- 172.21.2.102:0/3482551886 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fda640186c0 0x7fda6401abc0 crc :-1 s=READY pgs=250 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:11:15.726 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:15.730 7fda856c2700 1 -- 172.21.2.102:0/3482551886 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fda800a0910 msgr2=0x7fda80105d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:11:15.727 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:15.730 7fda856c2700 1 --2- 172.21.2.102:0/3482551886 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fda800a0910 0x7fda80105d70 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 rx=0x7fda68007650 tx=0x7fda6800a900).stop 2021-02-10T11:11:15.727 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:15.730 7fda856c2700 1 -- 172.21.2.102:0/3482551886 shutdown_connections 2021-02-10T11:11:15.727 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:15.730 7fda856c2700 1 --2- 172.21.2.102:0/3482551886 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fda640186c0 0x7fda6401abc0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:11:15.727 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:15.730 7fda856c2700 1 --2- 172.21.2.102:0/3482551886 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fda80093770 0x7fda80093b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:11:15.727 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:15.730 7fda856c2700 1 --2- 172.21.2.102:0/3482551886 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fda801a7a10 0x7fda801a7e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:11:15.727 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:15.730 7fda856c2700 1 --2- 172.21.2.102:0/3482551886 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fda800a0910 0x7fda80105d70 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:11:15.727 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:15.730 7fda856c2700 1 -- 172.21.2.102:0/3482551886 >> 172.21.2.102:0/3482551886 conn(0x7fda80095ef0 msgr2=0x7fda80094e40 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:11:15.727 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:15.731 7fda856c2700 1 -- 172.21.2.102:0/3482551886 shutdown_connections 2021-02-10T11:11:15.727 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:15.731 7fda856c2700 1 -- 172.21.2.102:0/3482551886 wait complete. 2021-02-10T11:11:15.734 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000344403 2021-02-10T11:11:16.165 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:16.152 7f9c1838e700 1 Processor -- start 2021-02-10T11:11:16.165 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:16.152 7f9c1838e700 1 -- start start 2021-02-10T11:11:16.165 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:16.153 7f9c1838e700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f9c101b0470 0x7f9c101b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:11:16.165 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:16.153 7f9c1838e700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9c100f20e0 0x7f9c100f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:11:16.165 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:16.153 7f9c1838e700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c100f0b50 con 0x7f9c100ef610 2021-02-10T11:11:16.165 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:16.153 7f9c1838e700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c100f0820 con 0x7f9c101b0470 2021-02-10T11:11:16.166 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:16.153 7f9c1838e700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c100f24e0 con 0x7f9c100f20e0 2021-02-10T11:11:16.166 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:16.153 7f9c159a9700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f9c100ef610 0x7f9c100efa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:44270/0 (socket says 172.21.2.102:44270) 2021-02-10T11:11:16.166 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:16.153 7f9c159a9700 1 -- 172.21.2.102:0/3527247866 learned_addr learned my addr 172.21.2.102:0/3527247866 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:11:16.166 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:16.153 7f9c161aa700 1 --2- 172.21.2.102:0/3527247866 >> v2:172.21.2.113:3300/0 conn(0x7f9c101b0470 0x7f9c101b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:11:16.166 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:16.153 7f9c169ab700 1 --2- 172.21.2.102:0/3527247866 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9c100f20e0 0x7f9c100f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:11:16.166 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:16.153 7f9c151a8700 1 -- 172.21.2.102:0/3527247866 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1285540375 0 0) 0x7f9c00000ce0 con 0x7f9c100ef610 2021-02-10T11:11:16.166 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:16.153 7f9c151a8700 1 -- 172.21.2.102:0/3527247866 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9bfc002fe0 con 0x7f9c100ef610 2021-02-10T11:11:16.166 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:16.153 7f9c151a8700 1 -- 172.21.2.102:0/3527247866 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2582804733 0 0) 0x7f9c000010e0 con 0x7f9c100ef610 2021-02-10T11:11:16.166 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:16.153 7f9c169ab700 1 -- 172.21.2.102:0/3527247866 >> v2:172.21.2.113:3300/0 conn(0x7f9c101b0470 msgr2=0x7f9c101b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:11:16.166 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:16.153 7f9c169ab700 1 --2- 172.21.2.102:0/3527247866 >> v2:172.21.2.113:3300/0 conn(0x7f9c101b0470 0x7f9c101b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:11:16.166 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:16.153 7f9c169ab700 1 -- 172.21.2.102:0/3527247866 >> v1:172.21.2.102:6789/0 conn(0x7f9c100ef610 legacy=0x7f9c100efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:11:16.166 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:16.154 7f9c169ab700 1 -- 172.21.2.102:0/3527247866 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c100e52e0 con 0x7f9c100f20e0 2021-02-10T11:11:16.167 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:16.154 7f9c161aa700 1 --2- 172.21.2.102:0/3527247866 >> v2:172.21.2.113:3300/0 conn(0x7f9c101b0470 0x7f9c101b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:11:16.167 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:16.154 7f9c169ab700 1 --2- 172.21.2.102:0/3527247866 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9c100f20e0 0x7f9c100f0200 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f9c08006ad0 tx=0x7f9c08009ad0).ready entity=mon.2 client_cookie=f4835d03e695bc1a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:11:16.167 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:16.154 7f9c151a8700 1 -- 172.21.2.102:0/3527247866 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9c080022e0 con 0x7f9c100f20e0 2021-02-10T11:11:16.167 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:16.154 7f9c151a8700 1 -- 172.21.2.102:0/3527247866 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c08006410 con 0x7f9c100f20e0 2021-02-10T11:11:16.167 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:16.154 7f9c151a8700 1 -- 172.21.2.102:0/3527247866 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9c08002620 con 0x7f9c100f20e0 2021-02-10T11:11:16.167 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:16.154 7f9c1838e700 1 -- 172.21.2.102:0/3527247866 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9c100f20e0 msgr2=0x7f9c100f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:11:16.167 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:16.154 7f9c1838e700 1 --2- 172.21.2.102:0/3527247866 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9c100f20e0 0x7f9c100f0200 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f9c08006ad0 tx=0x7f9c08009ad0).stop 2021-02-10T11:11:16.167 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:16.155 7f9c1838e700 1 -- 172.21.2.102:0/3527247866 shutdown_connections 2021-02-10T11:11:16.167 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:16.155 7f9c1838e700 1 --2- 172.21.2.102:0/3527247866 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9c100f20e0 0x7f9c100f0200 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:11:16.167 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:16.155 7f9c1838e700 1 --2- 172.21.2.102:0/3527247866 >> v2:172.21.2.113:3300/0 conn(0x7f9c101b0470 0x7f9c101b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:11:16.167 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:16.155 7f9c1838e700 1 -- 172.21.2.102:0/3527247866 >> 172.21.2.102:0/3527247866 conn(0x7f9c100ce3e0 msgr2=0x7f9c100e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:11:16.167 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:16.155 7f9c1838e700 1 -- 172.21.2.102:0/3527247866 shutdown_connections 2021-02-10T11:11:16.167 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:16.155 7f9c1838e700 1 -- 172.21.2.102:0/3527247866 wait complete. 2021-02-10T11:11:16.168 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:16.169 7f9c1838e700 1 Processor -- start 2021-02-10T11:11:16.168 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:16.169 7f9c1838e700 1 -- start start 2021-02-10T11:11:16.169 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:16.170 7f9c1838e700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9c100a0420 0x7f9c100a0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:11:16.169 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:16.170 7f9c1838e700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f9c10189d00 0x7f9c1018a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:11:16.169 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:16.170 7f9c1838e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9c1018a720 0x7f9c100b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:11:16.170 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:16.170 7f9c1838e700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c10189800 con 0x7f9c1018a720 2021-02-10T11:11:16.170 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:16.170 7f9c1838e700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c1018ab20 con 0x7f9c10189d00 2021-02-10T11:11:16.170 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:16.170 7f9c1838e700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c100a1140 con 0x7f9c100a0420 2021-02-10T11:11:16.170 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:16.170 7f9c169ab700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9c1018a720 0x7f9c100b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:11:16.170 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:16.170 7f9c169ab700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9c1018a720 0x7f9c100b7be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:52054/0 (socket says 172.21.2.102:52054) 2021-02-10T11:11:16.170 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:16.170 7f9c169ab700 1 -- 172.21.2.102:0/3504359188 learned_addr learned my addr 172.21.2.102:0/3504359188 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:11:16.170 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:16.170 7f9c159a9700 1 --2- 172.21.2.102:0/3504359188 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f9c10189d00 0x7f9c1018a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:11:16.170 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:16.170 7f9c161aa700 1 --2- 172.21.2.102:0/3504359188 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9c100a0420 0x7f9c100a0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:11:16.170 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:16.170 7f9c169ab700 1 -- 172.21.2.102:0/3504359188 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9c100a0420 msgr2=0x7f9c100a0820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:11:16.170 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:16.170 7f9c169ab700 1 --2- 172.21.2.102:0/3504359188 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9c100a0420 0x7f9c100a0820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:11:16.170 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:16.170 7f9c169ab700 1 -- 172.21.2.102:0/3504359188 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f9c10189d00 msgr2=0x7f9c1018a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:11:16.170 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:16.170 7f9c169ab700 1 --2- 172.21.2.102:0/3504359188 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f9c10189d00 0x7f9c1018a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:11:16.170 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:16.170 7f9c169ab700 1 -- 172.21.2.102:0/3504359188 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c100b8200 con 0x7f9c1018a720 2021-02-10T11:11:16.171 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:16.170 7f9c169ab700 1 --2- 172.21.2.102:0/3504359188 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9c1018a720 0x7f9c100b7be0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 rx=0x7f9c0800b610 tx=0x7f9c0800b9b0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:11:16.171 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:16.170 7f9c06ffd700 1 -- 172.21.2.102:0/3504359188 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9c08002030 con 0x7f9c1018a720 2021-02-10T11:11:16.171 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:16.171 7f9c06ffd700 1 -- 172.21.2.102:0/3504359188 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c08002370 con 0x7f9c1018a720 2021-02-10T11:11:16.171 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:16.171 7f9c06ffd700 1 -- 172.21.2.102:0/3504359188 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9c08002850 con 0x7f9c1018a720 2021-02-10T11:11:16.171 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:16.171 7f9c1838e700 1 -- 172.21.2.102:0/3504359188 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c100b8200 con 0x7f9c1018a720 2021-02-10T11:11:16.171 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:16.171 7f9c1838e700 1 -- 172.21.2.102:0/3504359188 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c100b8790 con 0x7f9c1018a720 2021-02-10T11:11:16.171 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:16.171 7f9c06ffd700 1 -- 172.21.2.102:0/3504359188 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f9c08000a70 con 0x7f9c1018a720 2021-02-10T11:11:16.171 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:16.171 7f9c06ffd700 1 --2- 172.21.2.102:0/3504359188 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f9bf8018730 0x7f9bf801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:11:16.171 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:16.171 7f9c06ffd700 1 -- 172.21.2.102:0/3504359188 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f9c08032690 con 0x7f9c1018a720 2021-02-10T11:11:16.171 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:16.171 7f9c161aa700 1 --2- 172.21.2.102:0/3504359188 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f9bf8018730 0x7f9bf801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:11:16.171 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:16.172 7f9c161aa700 1 --2- 172.21.2.102:0/3504359188 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f9bf8018730 0x7f9bf801abc0 crc :-1 s=READY pgs=251 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:11:16.198 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:16.202 7f9c1838e700 1 -- 172.21.2.102:0/3504359188 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c1018ab20 con 0x7f9c1018a720 2021-02-10T11:11:16.201 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:16.205 7f9c06ffd700 1 -- 172.21.2.102:0/3504359188 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f9c0801f550 con 0x7f9c1018a720 2021-02-10T11:11:16.330 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:16.334 7f9c1838e700 1 -- 172.21.2.102:0/3504359188 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344403", "target": ["mgr", ""]}) v1 -- 0x7f9c1018ab20 con 0x7f9bf8018730 2021-02-10T11:11:16.333 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:16.337 7f9c06ffd700 1 -- 172.21.2.102:0/3504359188 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f9c0c000ce0 con 0x7f9bf8018730 2021-02-10T11:11:16.337 INFO:teuthology.orchestra.run.gibba002.stdout:/volumes/_nogroup/subvolume_0000000000344403/d9231228-480f-4e9a-a1ca-842ac771f7de 2021-02-10T11:11:16.339 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:16.343 7f9c1838e700 1 -- 172.21.2.102:0/3504359188 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f9bf8018730 msgr2=0x7f9bf801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:11:16.339 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:16.343 7f9c1838e700 1 --2- 172.21.2.102:0/3504359188 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f9bf8018730 0x7f9bf801abc0 crc :-1 s=READY pgs=251 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:11:16.339 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:16.343 7f9c1838e700 1 -- 172.21.2.102:0/3504359188 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9c1018a720 msgr2=0x7f9c100b7be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:11:16.339 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:16.343 7f9c1838e700 1 --2- 172.21.2.102:0/3504359188 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9c1018a720 0x7f9c100b7be0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 rx=0x7f9c0800b610 tx=0x7f9c0800b9b0).stop 2021-02-10T11:11:16.339 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:16.343 7f9c1838e700 1 -- 172.21.2.102:0/3504359188 shutdown_connections 2021-02-10T11:11:16.339 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:16.343 7f9c1838e700 1 --2- 172.21.2.102:0/3504359188 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9c1018a720 0x7f9c100b7be0 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:11:16.339 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:16.343 7f9c1838e700 1 --2- 172.21.2.102:0/3504359188 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f9bf8018730 0x7f9bf801abc0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:11:16.339 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:16.343 7f9c1838e700 1 --2- 172.21.2.102:0/3504359188 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f9c10189d00 0x7f9c1018a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:11:16.339 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:16.343 7f9c1838e700 1 --2- 172.21.2.102:0/3504359188 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9c100a0420 0x7f9c100a0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:11:16.340 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:16.343 7f9c1838e700 1 -- 172.21.2.102:0/3504359188 >> 172.21.2.102:0/3504359188 conn(0x7f9c10094e10 msgr2=0x7f9c100a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:11:16.340 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:16.343 7f9c1838e700 1 -- 172.21.2.102:0/3504359188 shutdown_connections 2021-02-10T11:11:16.340 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:16.343 7f9c1838e700 1 -- 172.21.2.102:0/3504359188 wait complete. 2021-02-10T11:11:16.347 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000344403 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000344403/d9231228-480f-4e9a-a1ca-842ac771f7de 2021-02-10T11:11:16.347 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:11:16.347 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:11:16.347 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:11:16.684 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:11:16.684 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:11:16.684 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344403/d9231228-480f-4e9a-a1ca-842ac771f7de/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:11:17.065 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:11:17.065 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:11:17.065 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0229958 s, 45.6 MB/s 2021-02-10T11:11:17.066 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:11:17.066 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:11:17.066 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:11:17.333 DEBUG:teuthology.orchestra.run.gibba002:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:11:17.336 DEBUG:teuthology.orchestra.run.gibba013:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:11:17.400 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:11:17.401 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:11:17.401 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344403/d9231228-480f-4e9a-a1ca-842ac771f7de/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:11:17.709 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:11:17.709 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:11:17.709 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0249218 s, 42.1 MB/s 2021-02-10T11:11:17.710 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:11:17.710 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:11:17.710 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:11:18.045 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:11:18.046 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:11:18.046 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344403/d9231228-480f-4e9a-a1ca-842ac771f7de/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:11:18.418 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:11:18.418 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:11:18.418 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.024168 s, 43.4 MB/s 2021-02-10T11:11:18.419 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:11:18.419 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:11:18.419 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:11:18.756 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:11:18.756 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:11:18.756 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344403/d9231228-480f-4e9a-a1ca-842ac771f7de/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:11:19.131 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:11:19.131 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:11:19.131 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.023381 s, 44.8 MB/s 2021-02-10T11:11:19.132 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:11:19.132 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:11:19.132 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:11:19.467 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:11:19.468 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:11:19.468 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344403/d9231228-480f-4e9a-a1ca-842ac771f7de/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:11:19.842 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:11:19.843 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:11:19.843 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0237161 s, 44.2 MB/s 2021-02-10T11:11:19.844 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:11:19.844 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:11:19.844 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:11:20.180 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:11:20.180 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:11:20.180 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344403/d9231228-480f-4e9a-a1ca-842ac771f7de/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:11:20.553 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:11:20.554 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:11:20.554 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0228301 s, 45.9 MB/s 2021-02-10T11:11:20.555 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:11:20.555 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:11:20.555 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:11:20.891 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:11:20.891 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:11:20.891 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344403/d9231228-480f-4e9a-a1ca-842ac771f7de/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:11:21.266 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:11:21.266 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:11:21.266 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0239912 s, 43.7 MB/s 2021-02-10T11:11:21.267 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:11:21.267 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:11:21.268 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:11:21.603 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:11:21.603 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:11:21.603 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344403/d9231228-480f-4e9a-a1ca-842ac771f7de/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:11:21.977 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:11:21.977 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:11:21.978 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0239111 s, 43.9 MB/s 2021-02-10T11:11:21.978 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:11:21.979 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:11:21.979 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:11:22.314 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:11:22.315 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:11:22.315 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344403/d9231228-480f-4e9a-a1ca-842ac771f7de/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:11:22.689 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:11:22.689 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:11:22.689 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0231823 s, 45.2 MB/s 2021-02-10T11:11:22.690 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:11:22.690 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:11:22.691 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:11:23.025 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:11:23.025 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:11:23.026 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344403/d9231228-480f-4e9a-a1ca-842ac771f7de/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:11:23.399 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:11:23.399 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:11:23.399 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.02408 s, 43.5 MB/s 2021-02-10T11:11:23.400 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000344404 2021-02-10T11:11:23.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:23.818 7f48a39a2700 1 Processor -- start 2021-02-10T11:11:23.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:23.818 7f48a39a2700 1 -- start start 2021-02-10T11:11:23.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:23.819 7f48a39a2700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f489c1b0870 0x7f489c0f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:11:23.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:23.819 7f48a39a2700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f489c0efe00 0x7f489c0f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:11:23.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:23.819 7f48a39a2700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f489c1b0c70 con 0x7f489c1b0470 2021-02-10T11:11:23.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:23.819 7f48a39a2700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f489c0f0b50 con 0x7f489c1b0870 2021-02-10T11:11:23.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:23.819 7f48a39a2700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f489c0f0820 con 0x7f489c0efe00 2021-02-10T11:11:23.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:23.819 7f48a17be700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f489c1b0470 0x7f489c0ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:44282/0 (socket says 172.21.2.102:44282) 2021-02-10T11:11:23.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:23.819 7f48a17be700 1 -- 172.21.2.102:0/2195744722 learned_addr learned my addr 172.21.2.102:0/2195744722 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:11:23.832 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:23.819 7f48a0fbd700 1 --2- 172.21.2.102:0/2195744722 >> v2:172.21.2.102:3300/0 conn(0x7f489c1b0870 0x7f489c0f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:11:23.832 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:23.819 7f48a1fbf700 1 --2- 172.21.2.102:0/2195744722 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f489c0efe00 0x7f489c0f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:11:23.832 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:23.819 7f488ffff700 1 -- 172.21.2.102:0/2195744722 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1932347321 0 0) 0x7f48900009b0 con 0x7f489c1b0470 2021-02-10T11:11:23.832 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:23.819 7f488ffff700 1 -- 172.21.2.102:0/2195744722 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4888002fe0 con 0x7f489c1b0470 2021-02-10T11:11:23.832 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:23.820 7f48a0fbd700 1 -- 172.21.2.102:0/2195744722 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f489c0efe00 msgr2=0x7f489c0f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:11:23.832 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:23.820 7f48a0fbd700 1 --2- 172.21.2.102:0/2195744722 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f489c0efe00 0x7f489c0f0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:11:23.832 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:23.820 7f48a0fbd700 1 -- 172.21.2.102:0/2195744722 >> v1:172.21.2.102:6789/0 conn(0x7f489c1b0470 legacy=0x7f489c0ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:11:23.832 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:23.820 7f48a0fbd700 1 -- 172.21.2.102:0/2195744722 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f489c0e52e0 con 0x7f489c1b0870 2021-02-10T11:11:23.832 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:23.820 7f48a0fbd700 1 --2- 172.21.2.102:0/2195744722 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f489c1b0870 0x7f489c0f20e0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 rx=0x7f4898006650 tx=0x7f4898009a80).ready entity=mon.0 client_cookie=7b8430c9592efdb0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:11:23.832 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:23.820 7f488ffff700 1 -- 172.21.2.102:0/2195744722 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f48980009b0 con 0x7f489c1b0870 2021-02-10T11:11:23.832 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:23.820 7f488ffff700 1 -- 172.21.2.102:0/2195744722 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f489800c0c0 con 0x7f489c1b0870 2021-02-10T11:11:23.832 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:23.820 7f488ffff700 1 -- 172.21.2.102:0/2195744722 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f489800c5e0 con 0x7f489c1b0870 2021-02-10T11:11:23.832 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:23.820 7f48a39a2700 1 -- 172.21.2.102:0/2195744722 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f489c1b0870 msgr2=0x7f489c0f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:11:23.833 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:23.820 7f48a39a2700 1 --2- 172.21.2.102:0/2195744722 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f489c1b0870 0x7f489c0f20e0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 rx=0x7f4898006650 tx=0x7f4898009a80).stop 2021-02-10T11:11:23.833 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:23.820 7f48a39a2700 1 -- 172.21.2.102:0/2195744722 shutdown_connections 2021-02-10T11:11:23.833 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:23.820 7f48a39a2700 1 --2- 172.21.2.102:0/2195744722 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f489c1b0870 0x7f489c0f20e0 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:11:23.833 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:23.820 7f48a39a2700 1 --2- 172.21.2.102:0/2195744722 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f489c0efe00 0x7f489c0f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:11:23.833 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:23.820 7f48a39a2700 1 -- 172.21.2.102:0/2195744722 >> 172.21.2.102:0/2195744722 conn(0x7f489c0ce3e0 msgr2=0x7f489c0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:11:23.833 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:23.821 7f48a39a2700 1 -- 172.21.2.102:0/2195744722 shutdown_connections 2021-02-10T11:11:23.833 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:23.821 7f48a39a2700 1 -- 172.21.2.102:0/2195744722 wait complete. 2021-02-10T11:11:23.834 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:23.835 7f48a39a2700 1 Processor -- start 2021-02-10T11:11:23.834 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:23.835 7f48a39a2700 1 -- start start 2021-02-10T11:11:23.835 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:23.836 7f48a39a2700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f489c190920 0x7f489c0b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:11:23.835 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:23.836 7f48a39a2700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f489c0b8200 0x7f489c0b8600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:11:23.835 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:23.836 7f48a39a2700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f489c0c1d80 0x7f489c0c2180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:11:23.835 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:23.836 7f48a39a2700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f489c190d20 con 0x7f489c0b8200 2021-02-10T11:11:23.835 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:23.836 7f48a39a2700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f489c0b8c20 con 0x7f489c0c1d80 2021-02-10T11:11:23.835 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:23.836 7f48a39a2700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f489c0c27a0 con 0x7f489c190920 2021-02-10T11:11:23.835 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:23.836 7f48a0fbd700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f489c0b8200 0x7f489c0b8600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:11:23.835 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:23.836 7f48a0fbd700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f489c0b8200 0x7f489c0b8600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:52066/0 (socket says 172.21.2.102:52066) 2021-02-10T11:11:23.835 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:23.836 7f48a0fbd700 1 -- 172.21.2.102:0/3417122660 learned_addr learned my addr 172.21.2.102:0/3417122660 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:11:23.835 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:23.836 7f48a1fbf700 1 --2- 172.21.2.102:0/3417122660 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f489c0c1d80 0x7f489c0c2180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:11:23.835 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:23.836 7f48a17be700 1 --2- 172.21.2.102:0/3417122660 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f489c190920 0x7f489c0b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:11:23.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:23.836 7f48a0fbd700 1 -- 172.21.2.102:0/3417122660 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f489c190920 msgr2=0x7f489c0b7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:11:23.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:23.837 7f48a0fbd700 1 --2- 172.21.2.102:0/3417122660 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f489c190920 0x7f489c0b7be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:11:23.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:23.837 7f48a0fbd700 1 -- 172.21.2.102:0/3417122660 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f489c0c1d80 msgr2=0x7f489c0c2180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:11:23.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:23.837 7f48a0fbd700 1 --2- 172.21.2.102:0/3417122660 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f489c0c1d80 0x7f489c0c2180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:11:23.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:23.837 7f48a0fbd700 1 -- 172.21.2.102:0/3417122660 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f489c0c29c0 con 0x7f489c0b8200 2021-02-10T11:11:23.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:23.837 7f48a0fbd700 1 --2- 172.21.2.102:0/3417122660 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f489c0b8200 0x7f489c0b8600 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 rx=0x7f4898009720 tx=0x7f489800b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:11:23.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:23.837 7f488dffb700 1 -- 172.21.2.102:0/3417122660 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f48980009b0 con 0x7f489c0b8200 2021-02-10T11:11:23.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:23.837 7f488dffb700 1 -- 172.21.2.102:0/3417122660 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4898006030 con 0x7f489c0b8200 2021-02-10T11:11:23.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:23.837 7f488dffb700 1 -- 172.21.2.102:0/3417122660 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f48980052e0 con 0x7f489c0b8200 2021-02-10T11:11:23.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:23.837 7f48a39a2700 1 -- 172.21.2.102:0/3417122660 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f489c0c29c0 con 0x7f489c0b8200 2021-02-10T11:11:23.837 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:23.837 7f48a39a2700 1 -- 172.21.2.102:0/3417122660 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f489c0c29c0 con 0x7f489c0b8200 2021-02-10T11:11:23.837 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:23.838 7f488dffb700 1 -- 172.21.2.102:0/3417122660 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f48980052e0 con 0x7f489c0b8200 2021-02-10T11:11:23.837 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:23.838 7f488dffb700 1 --2- 172.21.2.102:0/3417122660 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f487c018730 0x7f487c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:11:23.837 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:23.838 7f488dffb700 1 -- 172.21.2.102:0/3417122660 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f48980327c0 con 0x7f489c0b8200 2021-02-10T11:11:23.837 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:23.838 7f48a17be700 1 --2- 172.21.2.102:0/3417122660 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f487c018730 0x7f487c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:11:23.837 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:23.838 7f48a17be700 1 --2- 172.21.2.102:0/3417122660 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f487c018730 0x7f487c01abc0 crc :-1 s=READY pgs=252 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:11:23.864 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:23.868 7f48a39a2700 1 -- 172.21.2.102:0/3417122660 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f489c105ae0 con 0x7f489c0b8200 2021-02-10T11:11:23.867 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:23.871 7f488dffb700 1 -- 172.21.2.102:0/3417122660 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f4898006030 con 0x7f489c0b8200 2021-02-10T11:11:23.991 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:23.995 7f48a39a2700 1 -- 172.21.2.102:0/3417122660 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344404", "target": ["mgr", ""]}) v1 -- 0x7f489c0b8c20 con 0x7f487c018730 2021-02-10T11:11:24.019 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:24.024 7f488dffb700 1 -- 172.21.2.102:0/3417122660 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f4890001970 con 0x7f487c018730 2021-02-10T11:11:24.024 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:24.028 7f48a39a2700 1 -- 172.21.2.102:0/3417122660 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f487c018730 msgr2=0x7f487c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:11:24.025 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:24.028 7f48a39a2700 1 --2- 172.21.2.102:0/3417122660 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f487c018730 0x7f487c01abc0 crc :-1 s=READY pgs=252 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:11:24.025 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:24.028 7f48a39a2700 1 -- 172.21.2.102:0/3417122660 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f489c0b8200 msgr2=0x7f489c0b8600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:11:24.025 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:24.028 7f48a39a2700 1 --2- 172.21.2.102:0/3417122660 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f489c0b8200 0x7f489c0b8600 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 rx=0x7f4898009720 tx=0x7f489800b970).stop 2021-02-10T11:11:24.025 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:24.028 7f48a39a2700 1 -- 172.21.2.102:0/3417122660 shutdown_connections 2021-02-10T11:11:24.025 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:24.028 7f48a39a2700 1 --2- 172.21.2.102:0/3417122660 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f487c018730 0x7f487c01abc0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:11:24.025 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:24.028 7f48a39a2700 1 --2- 172.21.2.102:0/3417122660 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f489c0c1d80 0x7f489c0c2180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:11:24.025 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:24.028 7f48a39a2700 1 --2- 172.21.2.102:0/3417122660 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f489c0b8200 0x7f489c0b8600 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:11:24.025 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:24.028 7f48a39a2700 1 --2- 172.21.2.102:0/3417122660 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f489c190920 0x7f489c0b7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:11:24.025 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:24.029 7f48a39a2700 1 -- 172.21.2.102:0/3417122660 >> 172.21.2.102:0/3417122660 conn(0x7f489c094e10 msgr2=0x7f489c0a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:11:24.025 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:24.029 7f48a39a2700 1 -- 172.21.2.102:0/3417122660 shutdown_connections 2021-02-10T11:11:24.025 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:24.029 7f48a39a2700 1 -- 172.21.2.102:0/3417122660 wait complete. 2021-02-10T11:11:24.032 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000344404 2021-02-10T11:11:24.465 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:24.452 7f3b21b54700 1 Processor -- start 2021-02-10T11:11:24.466 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:24.452 7f3b21b54700 1 -- start start 2021-02-10T11:11:24.466 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:24.453 7f3b21b54700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f3b1c0f05f0 0x7f3b1c0f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:11:24.466 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:24.453 7f3b21b54700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b1c1b0c70 con 0x7f3b1c1b0470 2021-02-10T11:11:24.466 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:24.453 7f3b21b54700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b1c0f0b50 con 0x7f3b1c1b0870 2021-02-10T11:11:24.466 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:24.453 7f3b21b54700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b1c0e5ea0 con 0x7f3b1c0f05f0 2021-02-10T11:11:24.466 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:24.453 7f3b1b7fe700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f3b1c1b0470 0x7f3b1c0ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:44296/0 (socket says 172.21.2.102:44296) 2021-02-10T11:11:24.466 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:24.453 7f3b1b7fe700 1 -- 172.21.2.102:0/4013177535 learned_addr learned my addr 172.21.2.102:0/4013177535 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:11:24.466 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:24.453 7f3b1bfff700 1 --2- 172.21.2.102:0/4013177535 >> v2:172.21.2.113:3300/0 conn(0x7f3b1c0f05f0 0x7f3b1c0f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:11:24.466 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:24.453 7f3b1a7fc700 1 -- 172.21.2.102:0/4013177535 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3227293182 0 0) 0x7f3b0c0010b0 con 0x7f3b1c1b0470 2021-02-10T11:11:24.466 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:24.453 7f3b1a7fc700 1 -- 172.21.2.102:0/4013177535 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b04002fe0 con 0x7f3b1c1b0470 2021-02-10T11:11:24.467 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:24.453 7f3b1bfff700 1 -- 172.21.2.102:0/4013177535 >> v1:172.21.2.113:6789/0 conn(0x7f3b1c1b0870 legacy=0x7f3b1c0efe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:11:24.467 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:24.453 7f3b1bfff700 1 -- 172.21.2.102:0/4013177535 >> v1:172.21.2.102:6789/0 conn(0x7f3b1c1b0470 legacy=0x7f3b1c0ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:11:24.467 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:24.453 7f3b1bfff700 1 -- 172.21.2.102:0/4013177535 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b1c0e52e0 con 0x7f3b1c0f05f0 2021-02-10T11:11:24.467 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:24.454 7f3b1bfff700 1 --2- 172.21.2.102:0/4013177535 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3b1c0f05f0 0x7f3b1c0f20e0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7f3b14006e50 tx=0x7f3b14009e70).ready entity=mon.1 client_cookie=31341a2de275d111 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:11:24.467 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:24.454 7f3b1a7fc700 1 -- 172.21.2.102:0/4013177535 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3b14006a20 con 0x7f3b1c0f05f0 2021-02-10T11:11:24.467 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:24.454 7f3b1a7fc700 1 -- 172.21.2.102:0/4013177535 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b14006790 con 0x7f3b1c0f05f0 2021-02-10T11:11:24.467 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:24.454 7f3b1a7fc700 1 -- 172.21.2.102:0/4013177535 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3b1400b700 con 0x7f3b1c0f05f0 2021-02-10T11:11:24.467 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:24.454 7f3b21b54700 1 -- 172.21.2.102:0/4013177535 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3b1c0f05f0 msgr2=0x7f3b1c0f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:11:24.467 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:24.454 7f3b21b54700 1 --2- 172.21.2.102:0/4013177535 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3b1c0f05f0 0x7f3b1c0f20e0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7f3b14006e50 tx=0x7f3b14009e70).stop 2021-02-10T11:11:24.467 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:24.455 7f3b21b54700 1 -- 172.21.2.102:0/4013177535 shutdown_connections 2021-02-10T11:11:24.467 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:24.455 7f3b21b54700 1 --2- 172.21.2.102:0/4013177535 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3b1c0f05f0 0x7f3b1c0f20e0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:11:24.468 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:24.455 7f3b21b54700 1 -- 172.21.2.102:0/4013177535 >> 172.21.2.102:0/4013177535 conn(0x7f3b1c0ce3e0 msgr2=0x7f3b1c0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:11:24.468 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:24.455 7f3b21b54700 1 -- 172.21.2.102:0/4013177535 shutdown_connections 2021-02-10T11:11:24.468 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:24.455 7f3b21b54700 1 -- 172.21.2.102:0/4013177535 wait complete. 2021-02-10T11:11:24.468 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:24.470 7f3b21b54700 1 Processor -- start 2021-02-10T11:11:24.470 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:24.470 7f3b21b54700 1 -- start start 2021-02-10T11:11:24.470 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:24.470 7f3b21b54700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3b1c0c36a0 0x7f3b1c0c2b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:11:24.470 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:24.470 7f3b21b54700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3b1c0c3190 0x7f3b1c0b1d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:11:24.470 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:24.470 7f3b21b54700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3b1c0b2370 0x7f3b1c0b2d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:11:24.470 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:24.470 7f3b21b54700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b1c0b27c0 con 0x7f3b1c0b2370 2021-02-10T11:11:24.470 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:24.470 7f3b21b54700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b1c0b3360 con 0x7f3b1c0c3190 2021-02-10T11:11:24.470 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:24.470 7f3b21b54700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b1c0b3540 con 0x7f3b1c0c36a0 2021-02-10T11:11:24.470 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:24.471 7f3b1bfff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3b1c0b2370 0x7f3b1c0b2d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:11:24.470 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:24.471 7f3b1affd700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3b1c0c3190 0x7f3b1c0b1d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:11:24.470 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:24.471 7f3b1bfff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3b1c0b2370 0x7f3b1c0b2d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:52080/0 (socket says 172.21.2.102:52080) 2021-02-10T11:11:24.471 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:24.471 7f3b1bfff700 1 -- 172.21.2.102:0/2202168047 learned_addr learned my addr 172.21.2.102:0/2202168047 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:11:24.471 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:24.471 7f3b1b7fe700 1 --2- 172.21.2.102:0/2202168047 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3b1c0c36a0 0x7f3b1c0c2b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:11:24.471 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:24.471 7f3b1affd700 1 -- 172.21.2.102:0/2202168047 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3b1c0c36a0 msgr2=0x7f3b1c0c2b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:11:24.471 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:24.471 7f3b1affd700 1 --2- 172.21.2.102:0/2202168047 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3b1c0c36a0 0x7f3b1c0c2b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:11:24.471 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:24.471 7f3b1affd700 1 -- 172.21.2.102:0/2202168047 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3b1c0b2370 msgr2=0x7f3b1c0b2d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:11:24.472 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:24.471 7f3b1affd700 1 --2- 172.21.2.102:0/2202168047 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3b1c0b2370 0x7f3b1c0b2d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:11:24.472 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:24.471 7f3b1affd700 1 -- 172.21.2.102:0/2202168047 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b1c0b3760 con 0x7f3b1c0c3190 2021-02-10T11:11:24.473 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:24.471 7f3b1bfff700 1 --2- 172.21.2.102:0/2202168047 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3b1c0b2370 0x7f3b1c0b2d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:11:24.473 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:24.471 7f3b1affd700 1 --2- 172.21.2.102:0/2202168047 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3b1c0c3190 0x7f3b1c0b1d50 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f3b10006e80 tx=0x7f3b1000b8b0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:11:24.473 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:24.472 7f3b03fff700 1 -- 172.21.2.102:0/2202168047 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3b10006030 con 0x7f3b1c0c3190 2021-02-10T11:11:24.473 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:24.472 7f3b03fff700 1 -- 172.21.2.102:0/2202168047 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b10006230 con 0x7f3b1c0c3190 2021-02-10T11:11:24.473 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:24.472 7f3b21b54700 1 -- 172.21.2.102:0/2202168047 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b1c0b3760 con 0x7f3b1c0c3190 2021-02-10T11:11:24.473 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:24.472 7f3b21b54700 1 -- 172.21.2.102:0/2202168047 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b1c0b3cf0 con 0x7f3b1c0c3190 2021-02-10T11:11:24.473 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:24.472 7f3b03fff700 1 -- 172.21.2.102:0/2202168047 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3b10005730 con 0x7f3b1c0c3190 2021-02-10T11:11:24.473 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:24.472 7f3b03fff700 1 -- 172.21.2.102:0/2202168047 <== mon.1 v2:172.21.2.113:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f3b100054b0 con 0x7f3b1c0c3190 2021-02-10T11:11:24.473 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:24.472 7f3b03fff700 1 --2- 172.21.2.102:0/2202168047 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3afc0186c0 0x7f3afc01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:11:24.473 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:24.472 7f3b03fff700 1 -- 172.21.2.102:0/2202168047 <== mon.1 v2:172.21.2.113:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f3b10032450 con 0x7f3b1c0c3190 2021-02-10T11:11:24.474 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:24.472 7f3b1b7fe700 1 --2- 172.21.2.102:0/2202168047 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3afc0186c0 0x7f3afc01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:11:24.474 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:24.473 7f3b1b7fe700 1 --2- 172.21.2.102:0/2202168047 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3afc0186c0 0x7f3afc01abc0 crc :-1 s=READY pgs=253 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:11:24.498 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:24.503 7f3b21b54700 1 -- 172.21.2.102:0/2202168047 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b1c0edd10 con 0x7f3b1c0c3190 2021-02-10T11:11:24.503 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:24.507 7f3b03fff700 1 -- 172.21.2.102:0/2202168047 <== mon.1 v2:172.21.2.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f3b10023f00 con 0x7f3b1c0c3190 2021-02-10T11:11:24.633 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:24.637 7f3b21b54700 1 -- 172.21.2.102:0/2202168047 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344404", "target": ["mgr", ""]}) v1 -- 0x7f3b1c0b3cf0 con 0x7f3afc0186c0 2021-02-10T11:11:24.636 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:24.640 7f3b03fff700 1 -- 172.21.2.102:0/2202168047 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f3b0c0014b0 con 0x7f3afc0186c0 2021-02-10T11:11:24.636 INFO:teuthology.orchestra.run.gibba002.stdout:/volumes/_nogroup/subvolume_0000000000344404/6520c41f-84ce-4716-b48f-81947f1a51c3 2021-02-10T11:11:24.637 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:24.641 7f3b21b54700 1 -- 172.21.2.102:0/2202168047 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3afc0186c0 msgr2=0x7f3afc01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:11:24.638 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:24.641 7f3b21b54700 1 --2- 172.21.2.102:0/2202168047 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3afc0186c0 0x7f3afc01abc0 crc :-1 s=READY pgs=253 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:11:24.638 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:24.641 7f3b21b54700 1 -- 172.21.2.102:0/2202168047 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3b1c0c3190 msgr2=0x7f3b1c0b1d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:11:24.638 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:24.641 7f3b21b54700 1 --2- 172.21.2.102:0/2202168047 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3b1c0c3190 0x7f3b1c0b1d50 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f3b10006e80 tx=0x7f3b1000b8b0).stop 2021-02-10T11:11:24.638 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:24.642 7f3b21b54700 1 -- 172.21.2.102:0/2202168047 shutdown_connections 2021-02-10T11:11:24.638 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:24.642 7f3b21b54700 1 --2- 172.21.2.102:0/2202168047 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3b1c0b2370 0x7f3b1c0b2d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:11:24.638 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:24.642 7f3b21b54700 1 --2- 172.21.2.102:0/2202168047 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3afc0186c0 0x7f3afc01abc0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:11:24.638 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:24.642 7f3b21b54700 1 --2- 172.21.2.102:0/2202168047 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3b1c0c3190 0x7f3b1c0b1d50 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:11:24.638 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:24.642 7f3b21b54700 1 --2- 172.21.2.102:0/2202168047 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3b1c0c36a0 0x7f3b1c0c2b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:11:24.638 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:24.642 7f3b21b54700 1 -- 172.21.2.102:0/2202168047 >> 172.21.2.102:0/2202168047 conn(0x7f3b1c09f630 msgr2=0x7f3b1c0a0130 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:11:24.638 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:24.642 7f3b21b54700 1 -- 172.21.2.102:0/2202168047 shutdown_connections 2021-02-10T11:11:24.638 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:24.642 7f3b21b54700 1 -- 172.21.2.102:0/2202168047 wait complete. 2021-02-10T11:11:24.646 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000344404 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000344404/6520c41f-84ce-4716-b48f-81947f1a51c3 2021-02-10T11:11:24.646 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:11:24.646 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:11:24.646 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:11:24.983 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:11:24.983 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:11:24.983 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344404/6520c41f-84ce-4716-b48f-81947f1a51c3/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:11:25.366 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:11:25.366 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:11:25.366 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0262699 s, 39.9 MB/s 2021-02-10T11:11:25.367 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:11:25.367 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:11:25.368 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:11:25.702 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:11:25.703 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:11:25.703 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344404/6520c41f-84ce-4716-b48f-81947f1a51c3/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:11:26.077 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:11:26.078 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:11:26.078 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.023934 s, 43.8 MB/s 2021-02-10T11:11:26.079 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:11:26.079 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:11:26.079 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:11:26.414 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:11:26.415 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:11:26.415 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344404/6520c41f-84ce-4716-b48f-81947f1a51c3/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:11:26.789 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:11:26.789 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:11:26.789 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0225774 s, 46.4 MB/s 2021-02-10T11:11:26.790 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:11:26.790 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:11:26.790 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:11:27.127 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:11:27.127 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:11:27.127 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344404/6520c41f-84ce-4716-b48f-81947f1a51c3/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:11:27.500 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:11:27.699 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:11:27.699 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.022587 s, 46.4 MB/s 2021-02-10T11:11:27.700 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:11:27.700 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:11:27.700 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:11:28.035 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:11:28.036 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:11:28.036 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344404/6520c41f-84ce-4716-b48f-81947f1a51c3/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:11:28.406 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:11:28.407 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:11:28.407 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0231104 s, 45.4 MB/s 2021-02-10T11:11:28.408 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:11:28.408 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:11:28.408 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:11:28.744 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:11:28.745 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:11:28.745 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344404/6520c41f-84ce-4716-b48f-81947f1a51c3/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:11:29.119 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:11:29.120 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:11:29.120 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.023404 s, 44.8 MB/s 2021-02-10T11:11:29.121 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:11:29.121 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:11:29.121 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:11:29.455 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:11:29.455 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:11:29.456 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344404/6520c41f-84ce-4716-b48f-81947f1a51c3/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:11:29.829 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:11:29.949 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:11:29.950 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0231913 s, 45.2 MB/s 2021-02-10T11:11:29.951 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:11:29.951 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:11:29.951 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:11:30.286 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:11:30.286 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:11:30.287 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344404/6520c41f-84ce-4716-b48f-81947f1a51c3/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:11:30.665 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:11:30.665 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:11:30.665 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0259021 s, 40.5 MB/s 2021-02-10T11:11:30.666 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:11:30.666 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:11:30.666 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:11:31.002 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:11:31.003 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:11:31.003 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344404/6520c41f-84ce-4716-b48f-81947f1a51c3/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:11:31.379 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:11:31.380 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:11:31.380 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0249948 s, 42.0 MB/s 2021-02-10T11:11:31.381 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:11:31.381 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:11:31.381 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:11:31.717 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:11:31.717 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:11:31.718 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344404/6520c41f-84ce-4716-b48f-81947f1a51c3/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:11:32.104 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:11:32.105 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:11:32.105 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0236716 s, 44.3 MB/s 2021-02-10T11:11:32.106 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000344405 2021-02-10T11:11:32.541 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:32.528 7f32e034f700 1 Processor -- start 2021-02-10T11:11:32.541 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:32.529 7f32e034f700 1 -- start start 2021-02-10T11:11:32.541 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:32.529 7f32e034f700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f32d81b0870 0x7f32d80f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:11:32.541 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:32.529 7f32e034f700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f32d80efe00 0x7f32d80f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:11:32.541 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:32.529 7f32e034f700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32d81b0c70 con 0x7f32d81b0470 2021-02-10T11:11:32.542 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:32.529 7f32e034f700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32d80f0b50 con 0x7f32d80efe00 2021-02-10T11:11:32.542 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:32.529 7f32e034f700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32d80f0820 con 0x7f32d81b0870 2021-02-10T11:11:32.542 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:32.529 7f32de96c700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f32d80efe00 0x7f32d80f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:11:32.542 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:32.529 7f32de16b700 1 --1- >> v1:172.21.2.113:6789/0 conn(0x7f32d81b0470 0x7f32d80ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.113:6789/0 says I am v1:172.21.2.102:51632/0 (socket says 172.21.2.102:51632) 2021-02-10T11:11:32.542 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:32.529 7f32de96c700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f32d80efe00 0x7f32d80f0200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:52086/0 (socket says 172.21.2.102:52086) 2021-02-10T11:11:32.542 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:32.529 7f32de16b700 1 -- 172.21.2.102:0/4035920323 learned_addr learned my addr 172.21.2.102:0/4035920323 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:11:32.542 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:32.530 7f32dd96a700 1 --2- 172.21.2.102:0/4035920323 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f32d81b0870 0x7f32d80f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:11:32.542 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:32.530 7f32de96c700 1 -- 172.21.2.102:0/4035920323 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f32d81b0870 msgr2=0x7f32d80f20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:11:32.542 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:32.530 7f32de96c700 1 --2- 172.21.2.102:0/4035920323 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f32d81b0870 0x7f32d80f20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:11:32.542 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:32.530 7f32de96c700 1 -- 172.21.2.102:0/4035920323 >> v1:172.21.2.113:6789/0 conn(0x7f32d81b0470 legacy=0x7f32d80ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:11:32.542 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:32.530 7f32de96c700 1 -- 172.21.2.102:0/4035920323 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32d80e52e0 con 0x7f32d80efe00 2021-02-10T11:11:32.542 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:32.530 7f32dd96a700 1 --2- 172.21.2.102:0/4035920323 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f32d81b0870 0x7f32d80f20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:11:32.543 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:32.530 7f32de96c700 1 --2- 172.21.2.102:0/4035920323 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f32d80efe00 0x7f32d80f0200 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 rx=0x7f32d0006e30 tx=0x7f32d0009e60).ready entity=mon.0 client_cookie=f44b53b1299e55e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:11:32.543 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:32.530 7f32dd169700 1 -- 172.21.2.102:0/4035920323 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f32d0006a50 con 0x7f32d80efe00 2021-02-10T11:11:32.543 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:32.530 7f32dd169700 1 -- 172.21.2.102:0/4035920323 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32d000b300 con 0x7f32d80efe00 2021-02-10T11:11:32.543 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:32.530 7f32dd169700 1 -- 172.21.2.102:0/4035920323 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f32d000b820 con 0x7f32d80efe00 2021-02-10T11:11:32.543 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:32.530 7f32e034f700 1 -- 172.21.2.102:0/4035920323 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f32d80efe00 msgr2=0x7f32d80f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:11:32.543 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:32.530 7f32e034f700 1 --2- 172.21.2.102:0/4035920323 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f32d80efe00 0x7f32d80f0200 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 rx=0x7f32d0006e30 tx=0x7f32d0009e60).stop 2021-02-10T11:11:32.543 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:32.531 7f32e034f700 1 -- 172.21.2.102:0/4035920323 shutdown_connections 2021-02-10T11:11:32.543 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:32.531 7f32e034f700 1 --2- 172.21.2.102:0/4035920323 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f32d81b0870 0x7f32d80f20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:11:32.543 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:32.531 7f32e034f700 1 --2- 172.21.2.102:0/4035920323 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f32d80efe00 0x7f32d80f0200 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:11:32.543 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:32.531 7f32e034f700 1 -- 172.21.2.102:0/4035920323 >> 172.21.2.102:0/4035920323 conn(0x7f32d80ce3e0 msgr2=0x7f32d80e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:11:32.543 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:32.531 7f32e034f700 1 -- 172.21.2.102:0/4035920323 shutdown_connections 2021-02-10T11:11:32.543 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:32.531 7f32e034f700 1 -- 172.21.2.102:0/4035920323 wait complete. 2021-02-10T11:11:32.544 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:32.546 7f32e034f700 1 Processor -- start 2021-02-10T11:11:32.546 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:32.546 7f32e034f700 1 -- start start 2021-02-10T11:11:32.546 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:32.546 7f32e034f700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f32d8105d10 0x7f32d809eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:11:32.546 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:32.546 7f32e034f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f32d8106110 0x7f32d809f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:11:32.547 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:32.546 7f32e034f700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f32d80b2a00 0x7f32d80b2e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:11:32.547 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:32.546 7f32e034f700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32d809e560 con 0x7f32d8106110 2021-02-10T11:11:32.547 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:32.546 7f32e034f700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32d809c090 con 0x7f32d80b2a00 2021-02-10T11:11:32.547 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:32.546 7f32e034f700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32d80b3420 con 0x7f32d8105d10 2021-02-10T11:11:32.547 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:32.546 7f32dd96a700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f32d8106110 0x7f32d809f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:11:32.547 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:32.546 7f32dd96a700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f32d8106110 0x7f32d809f100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:52092/0 (socket says 172.21.2.102:52092) 2021-02-10T11:11:32.547 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:32.546 7f32dd96a700 1 -- 172.21.2.102:0/4228513410 learned_addr learned my addr 172.21.2.102:0/4228513410 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:11:32.547 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:32.547 7f32dd96a700 1 -- 172.21.2.102:0/4228513410 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f32d8105d10 msgr2=0x7f32d809eae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:11:32.547 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:32.547 7f32dd96a700 1 --2- 172.21.2.102:0/4228513410 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f32d8105d10 0x7f32d809eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:11:32.547 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:32.547 7f32dd96a700 1 -- 172.21.2.102:0/4228513410 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f32d80b2a00 msgr2=0x7f32d80b2e00 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:11:32.547 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:32.547 7f32dd96a700 1 --2- 172.21.2.102:0/4228513410 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f32d80b2a00 0x7f32d80b2e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:11:32.547 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:32.547 7f32dd96a700 1 -- 172.21.2.102:0/4228513410 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32d80b3640 con 0x7f32d8106110 2021-02-10T11:11:32.547 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:32.547 7f32dd96a700 1 --2- 172.21.2.102:0/4228513410 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f32d8106110 0x7f32d809f100 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 rx=0x7f32cc006f20 tx=0x7f32cc00b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:11:32.548 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:32.547 7f32caffd700 1 -- 172.21.2.102:0/4228513410 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f32cc00bd80 con 0x7f32d8106110 2021-02-10T11:11:32.548 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:32.547 7f32caffd700 1 -- 172.21.2.102:0/4228513410 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32cc000c90 con 0x7f32d8106110 2021-02-10T11:11:32.548 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:32.547 7f32e034f700 1 -- 172.21.2.102:0/4228513410 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32d80b3640 con 0x7f32d8106110 2021-02-10T11:11:32.548 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:32.547 7f32e034f700 1 -- 172.21.2.102:0/4228513410 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32d80b3bd0 con 0x7f32d8106110 2021-02-10T11:11:32.548 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:32.547 7f32caffd700 1 -- 172.21.2.102:0/4228513410 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f32cc0022b0 con 0x7f32d8106110 2021-02-10T11:11:32.548 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:32.548 7f32caffd700 1 -- 172.21.2.102:0/4228513410 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f32cc00a000 con 0x7f32d8106110 2021-02-10T11:11:32.548 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:32.548 7f32caffd700 1 --2- 172.21.2.102:0/4228513410 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f32c0018730 0x7f32c001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:11:32.548 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:32.548 7f32caffd700 1 -- 172.21.2.102:0/4228513410 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f32cc032780 con 0x7f32d8106110 2021-02-10T11:11:32.548 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:32.548 7f32de16b700 1 --2- 172.21.2.102:0/4228513410 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f32c0018730 0x7f32c001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:11:32.548 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:32.548 7f32de16b700 1 --2- 172.21.2.102:0/4228513410 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f32c0018730 0x7f32c001abc0 crc :-1 s=READY pgs=254 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:11:32.574 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:32.579 7f32e034f700 1 -- 172.21.2.102:0/4228513410 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32d8105ae0 con 0x7f32d8106110 2021-02-10T11:11:32.577 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:32.581 7f32caffd700 1 -- 172.21.2.102:0/4228513410 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f32cc0009b0 con 0x7f32d8106110 2021-02-10T11:11:32.704 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:32.708 7f32e034f700 1 -- 172.21.2.102:0/4228513410 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344405", "target": ["mgr", ""]}) v1 -- 0x7f32d81a16c0 con 0x7f32c0018730 2021-02-10T11:11:32.732 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:32.736 7f32caffd700 1 -- 172.21.2.102:0/4228513410 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f32d4000c90 con 0x7f32c0018730 2021-02-10T11:11:32.737 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:32.741 7f32e034f700 1 -- 172.21.2.102:0/4228513410 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f32c0018730 msgr2=0x7f32c001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:11:32.738 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:32.741 7f32e034f700 1 --2- 172.21.2.102:0/4228513410 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f32c0018730 0x7f32c001abc0 crc :-1 s=READY pgs=254 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:11:32.738 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:32.741 7f32e034f700 1 -- 172.21.2.102:0/4228513410 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f32d8106110 msgr2=0x7f32d809f100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:11:32.738 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:32.741 7f32e034f700 1 --2- 172.21.2.102:0/4228513410 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f32d8106110 0x7f32d809f100 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 rx=0x7f32cc006f20 tx=0x7f32cc00b990).stop 2021-02-10T11:11:32.738 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:32.741 7f32e034f700 1 -- 172.21.2.102:0/4228513410 shutdown_connections 2021-02-10T11:11:32.738 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:32.741 7f32e034f700 1 --2- 172.21.2.102:0/4228513410 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f32c0018730 0x7f32c001abc0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:11:32.738 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:32.741 7f32e034f700 1 --2- 172.21.2.102:0/4228513410 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f32d80b2a00 0x7f32d80b2e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:11:32.739 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:32.741 7f32e034f700 1 --2- 172.21.2.102:0/4228513410 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f32d8106110 0x7f32d809f100 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:11:32.739 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:32.741 7f32e034f700 1 --2- 172.21.2.102:0/4228513410 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f32d8105d10 0x7f32d809eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:11:32.739 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:32.741 7f32e034f700 1 -- 172.21.2.102:0/4228513410 >> 172.21.2.102:0/4228513410 conn(0x7f32d80c36f0 msgr2=0x7f32d80bd640 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:11:32.739 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:32.742 7f32e034f700 1 -- 172.21.2.102:0/4228513410 shutdown_connections 2021-02-10T11:11:32.739 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:32.742 7f32e034f700 1 -- 172.21.2.102:0/4228513410 wait complete. 2021-02-10T11:11:32.746 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000344405 2021-02-10T11:11:33.183 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:33.169 7f85cb3cd700 1 Processor -- start 2021-02-10T11:11:33.184 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:33.169 7f85cb3cd700 1 -- start start 2021-02-10T11:11:33.184 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:33.169 7f85cb3cd700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f85c41b0470 0x7f85c41b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:11:33.184 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:33.169 7f85cb3cd700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85c40f0b50 con 0x7f85c40ef610 2021-02-10T11:11:33.184 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:33.169 7f85cb3cd700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85c40f24e0 con 0x7f85c40f20e0 2021-02-10T11:11:33.184 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:33.169 7f85cb3cd700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85c40e5ea0 con 0x7f85c41b0470 2021-02-10T11:11:33.184 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:33.170 7f85c91e9700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f85c41b0470 0x7f85c41b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:11:33.185 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:33.170 7f85c89e8700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f85c40ef610 0x7f85c40efa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:44320/0 (socket says 172.21.2.102:44320) 2021-02-10T11:11:33.185 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:33.170 7f85c89e8700 1 -- 172.21.2.102:0/3457876637 learned_addr learned my addr 172.21.2.102:0/3457876637 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:11:33.185 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:33.170 7f85c91e9700 1 -- 172.21.2.102:0/3457876637 >> v1:172.21.2.113:6789/0 conn(0x7f85c40f20e0 legacy=0x7f85c40f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:11:33.185 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:33.170 7f85c91e9700 1 -- 172.21.2.102:0/3457876637 >> v1:172.21.2.102:6789/0 conn(0x7f85c40ef610 legacy=0x7f85c40efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:11:33.185 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:33.170 7f85bbfff700 1 -- 172.21.2.102:0/3457876637 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2937552802 0 0) 0x7f85b40010b0 con 0x7f85c40ef610 2021-02-10T11:11:33.185 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:33.170 7f85c91e9700 1 -- 172.21.2.102:0/3457876637 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85c40e52e0 con 0x7f85c41b0470 2021-02-10T11:11:33.185 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:33.170 7f85c91e9700 1 --2- 172.21.2.102:0/3457876637 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f85c41b0470 0x7f85c41b0870 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 rx=0x7f85c0006650 tx=0x7f85c0009a00).ready entity=mon.0 client_cookie=4112ecc37ea1c45c server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:11:33.185 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:33.170 7f85bbfff700 1 -- 172.21.2.102:0/3457876637 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f85c00009b0 con 0x7f85c41b0470 2021-02-10T11:11:33.186 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:33.170 7f85bbfff700 1 -- 172.21.2.102:0/3457876637 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85c000c0c0 con 0x7f85c41b0470 2021-02-10T11:11:33.186 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:33.170 7f85bbfff700 1 -- 172.21.2.102:0/3457876637 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f85c000c520 con 0x7f85c41b0470 2021-02-10T11:11:33.186 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:33.171 7f85cb3cd700 1 -- 172.21.2.102:0/3457876637 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f85c41b0470 msgr2=0x7f85c41b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:11:33.186 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:33.171 7f85cb3cd700 1 --2- 172.21.2.102:0/3457876637 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f85c41b0470 0x7f85c41b0870 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 rx=0x7f85c0006650 tx=0x7f85c0009a00).stop 2021-02-10T11:11:33.187 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:33.171 7f85cb3cd700 1 -- 172.21.2.102:0/3457876637 shutdown_connections 2021-02-10T11:11:33.187 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:33.171 7f85cb3cd700 1 --2- 172.21.2.102:0/3457876637 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f85c41b0470 0x7f85c41b0870 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:11:33.187 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:33.171 7f85cb3cd700 1 -- 172.21.2.102:0/3457876637 >> 172.21.2.102:0/3457876637 conn(0x7f85c40ce3e0 msgr2=0x7f85c40e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:11:33.188 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:33.171 7f85cb3cd700 1 -- 172.21.2.102:0/3457876637 shutdown_connections 2021-02-10T11:11:33.188 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:33.171 7f85cb3cd700 1 -- 172.21.2.102:0/3457876637 wait complete. 2021-02-10T11:11:33.189 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:33.186 7f85cb3cd700 1 Processor -- start 2021-02-10T11:11:33.189 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:33.186 7f85cb3cd700 1 -- start start 2021-02-10T11:11:33.189 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:33.186 7f85cb3cd700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f85c40a1dd0 0x7f85c4190920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:11:33.189 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:33.186 7f85cb3cd700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f85c408fd30 0x7f85c40effc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:11:33.189 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:33.186 7f85cb3cd700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f85c40f0e90 0x7f85c40f1290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:11:33.189 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:33.186 7f85cb3cd700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85c40f06f0 con 0x7f85c40f0e90 2021-02-10T11:11:33.190 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:33.186 7f85cb3cd700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85c4090160 con 0x7f85c40a1dd0 2021-02-10T11:11:33.190 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:33.186 7f85cb3cd700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85c40f18b0 con 0x7f85c408fd30 2021-02-10T11:11:33.190 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:33.187 7f85c99ea700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f85c40f0e90 0x7f85c40f1290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:11:33.190 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:33.187 7f85c99ea700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f85c40f0e90 0x7f85c40f1290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:52104/0 (socket says 172.21.2.102:52104) 2021-02-10T11:11:33.190 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:33.187 7f85c99ea700 1 -- 172.21.2.102:0/461845849 learned_addr learned my addr 172.21.2.102:0/461845849 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:11:33.190 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:33.187 7f85c99ea700 1 -- 172.21.2.102:0/461845849 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f85c408fd30 msgr2=0x7f85c40effc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:11:33.191 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:33.187 7f85c99ea700 1 --2- 172.21.2.102:0/461845849 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f85c408fd30 0x7f85c40effc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:11:33.191 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:33.187 7f85c99ea700 1 -- 172.21.2.102:0/461845849 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f85c40a1dd0 msgr2=0x7f85c4190920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:11:33.191 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:33.187 7f85c99ea700 1 --2- 172.21.2.102:0/461845849 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f85c40a1dd0 0x7f85c4190920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:11:33.191 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:33.187 7f85c99ea700 1 -- 172.21.2.102:0/461845849 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85c40f1ad0 con 0x7f85c40f0e90 2021-02-10T11:11:33.191 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:33.187 7f85c99ea700 1 --2- 172.21.2.102:0/461845849 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f85c40f0e90 0x7f85c40f1290 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 rx=0x7f85bc009940 tx=0x7f85bc00bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:11:33.191 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:33.187 7f85b9ffb700 1 -- 172.21.2.102:0/461845849 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f85bc000f90 con 0x7f85c40f0e90 2021-02-10T11:11:33.191 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:33.187 7f85b9ffb700 1 -- 172.21.2.102:0/461845849 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85bc00c2f0 con 0x7f85c40f0e90 2021-02-10T11:11:33.191 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:33.187 7f85b9ffb700 1 -- 172.21.2.102:0/461845849 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f85bc00c750 con 0x7f85c40f0e90 2021-02-10T11:11:33.193 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:33.187 7f85cb3cd700 1 -- 172.21.2.102:0/461845849 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85c40f1ad0 con 0x7f85c40f0e90 2021-02-10T11:11:33.193 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:33.187 7f85cb3cd700 1 -- 172.21.2.102:0/461845849 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85c40f2060 con 0x7f85c40f0e90 2021-02-10T11:11:33.194 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:33.188 7f85b9ffb700 1 -- 172.21.2.102:0/461845849 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f85bc001220 con 0x7f85c40f0e90 2021-02-10T11:11:33.194 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:33.188 7f85b9ffb700 1 --2- 172.21.2.102:0/461845849 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f85a40186c0 0x7f85a401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:11:33.194 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:33.188 7f85b9ffb700 1 -- 172.21.2.102:0/461845849 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f85bc032960 con 0x7f85c40f0e90 2021-02-10T11:11:33.195 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:33.188 7f85c91e9700 1 --2- 172.21.2.102:0/461845849 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f85a40186c0 0x7f85a401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:11:33.195 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:33.189 7f85c91e9700 1 --2- 172.21.2.102:0/461845849 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f85a40186c0 0x7f85a401abc0 crc :-1 s=READY pgs=255 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:11:33.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:33.219 7f85cb3cd700 1 -- 172.21.2.102:0/461845849 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85c4090130 con 0x7f85c40f0e90 2021-02-10T11:11:33.218 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:33.222 7f85b9ffb700 1 -- 172.21.2.102:0/461845849 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f85bc002ef0 con 0x7f85c40f0e90 2021-02-10T11:11:33.349 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:33.354 7f85cb3cd700 1 -- 172.21.2.102:0/461845849 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344405", "target": ["mgr", ""]}) v1 -- 0x7f85c40f2060 con 0x7f85a40186c0 2021-02-10T11:11:33.352 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:33.357 7f85b9ffb700 1 -- 172.21.2.102:0/461845849 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f85c0006e30 con 0x7f85a40186c0 2021-02-10T11:11:33.357 INFO:teuthology.orchestra.run.gibba002.stdout:/volumes/_nogroup/subvolume_0000000000344405/e9f3e223-6455-4529-9e12-13876fcd6d2f 2021-02-10T11:11:33.358 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:33.362 7f85cb3cd700 1 -- 172.21.2.102:0/461845849 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f85a40186c0 msgr2=0x7f85a401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:11:33.359 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:33.362 7f85cb3cd700 1 --2- 172.21.2.102:0/461845849 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f85a40186c0 0x7f85a401abc0 crc :-1 s=READY pgs=255 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:11:33.359 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:33.362 7f85cb3cd700 1 -- 172.21.2.102:0/461845849 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f85c40f0e90 msgr2=0x7f85c40f1290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:11:33.359 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:33.362 7f85cb3cd700 1 --2- 172.21.2.102:0/461845849 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f85c40f0e90 0x7f85c40f1290 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 rx=0x7f85bc009940 tx=0x7f85bc00bcf0).stop 2021-02-10T11:11:33.359 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:33.362 7f85cb3cd700 1 -- 172.21.2.102:0/461845849 shutdown_connections 2021-02-10T11:11:33.359 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:33.362 7f85cb3cd700 1 --2- 172.21.2.102:0/461845849 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f85c40f0e90 0x7f85c40f1290 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:11:33.359 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:33.362 7f85cb3cd700 1 --2- 172.21.2.102:0/461845849 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f85c408fd30 0x7f85c40effc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:11:33.359 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:33.362 7f85cb3cd700 1 --2- 172.21.2.102:0/461845849 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f85a40186c0 0x7f85a401abc0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:11:33.359 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:33.362 7f85cb3cd700 1 --2- 172.21.2.102:0/461845849 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f85c40a1dd0 0x7f85c4190920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:11:33.359 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:33.362 7f85cb3cd700 1 -- 172.21.2.102:0/461845849 >> 172.21.2.102:0/461845849 conn(0x7f85c40958b0 msgr2=0x7f85c40bf040 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:11:33.359 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:33.363 7f85cb3cd700 1 -- 172.21.2.102:0/461845849 shutdown_connections 2021-02-10T11:11:33.359 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:33.363 7f85cb3cd700 1 -- 172.21.2.102:0/461845849 wait complete. 2021-02-10T11:11:33.366 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000344405 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000344405/e9f3e223-6455-4529-9e12-13876fcd6d2f 2021-02-10T11:11:33.367 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:11:33.367 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:11:33.367 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:11:33.702 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:11:33.702 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:11:33.702 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344405/e9f3e223-6455-4529-9e12-13876fcd6d2f/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:11:34.082 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:11:34.082 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:11:34.082 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0246748 s, 42.5 MB/s 2021-02-10T11:11:34.083 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:11:34.083 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:11:34.084 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:11:34.418 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:11:34.418 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:11:34.419 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344405/e9f3e223-6455-4529-9e12-13876fcd6d2f/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:11:34.795 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:11:34.795 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:11:34.795 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0241903 s, 43.3 MB/s 2021-02-10T11:11:34.796 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:11:34.796 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:11:34.796 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:11:35.131 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:11:35.132 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:11:35.132 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344405/e9f3e223-6455-4529-9e12-13876fcd6d2f/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:11:35.510 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:11:35.510 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:11:35.510 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0267353 s, 39.2 MB/s 2021-02-10T11:11:35.512 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:11:35.512 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:11:35.512 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:11:35.846 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:11:35.847 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:11:35.847 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344405/e9f3e223-6455-4529-9e12-13876fcd6d2f/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:11:36.223 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:11:36.223 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:11:36.223 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0239737 s, 43.7 MB/s 2021-02-10T11:11:36.224 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:11:36.224 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:11:36.224 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:11:36.559 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:11:36.560 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:11:36.560 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344405/e9f3e223-6455-4529-9e12-13876fcd6d2f/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:11:36.935 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:11:36.936 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:11:36.936 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0243049 s, 43.1 MB/s 2021-02-10T11:11:36.936 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:11:36.937 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:11:36.937 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:11:37.271 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:11:37.272 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:11:37.272 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344405/e9f3e223-6455-4529-9e12-13876fcd6d2f/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:11:37.645 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:11:37.646 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:11:37.646 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0231254 s, 45.3 MB/s 2021-02-10T11:11:37.647 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:11:37.647 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:11:37.647 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:11:37.982 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:11:37.982 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:11:37.983 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344405/e9f3e223-6455-4529-9e12-13876fcd6d2f/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:11:38.356 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:11:38.356 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:11:38.356 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0237909 s, 44.1 MB/s 2021-02-10T11:11:38.357 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:11:38.357 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:11:38.357 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:11:38.692 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:11:38.693 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:11:38.693 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344405/e9f3e223-6455-4529-9e12-13876fcd6d2f/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:11:39.067 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:11:39.068 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:11:39.068 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0235479 s, 44.5 MB/s 2021-02-10T11:11:39.069 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:11:39.069 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:11:39.069 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:11:39.403 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:11:39.404 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:11:39.404 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344405/e9f3e223-6455-4529-9e12-13876fcd6d2f/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:11:39.778 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:11:39.778 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:11:39.778 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0228159 s, 46.0 MB/s 2021-02-10T11:11:39.779 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:11:39.779 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:11:39.779 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:11:40.114 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:11:40.115 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:11:40.115 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344405/e9f3e223-6455-4529-9e12-13876fcd6d2f/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:11:40.492 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:11:40.492 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:11:40.493 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0251964 s, 41.6 MB/s 2021-02-10T11:11:40.494 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000344406 2021-02-10T11:11:40.925 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:40.913 7f71dd1c8700 1 Processor -- start 2021-02-10T11:11:40.926 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:40.913 7f71dd1c8700 1 -- start start 2021-02-10T11:11:40.926 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:40.913 7f71dd1c8700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f71d81b0470 0x7f71d81b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:11:40.926 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:40.913 7f71dd1c8700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f71d80f20e0 0x7f71d80f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:11:40.926 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:40.913 7f71dd1c8700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71d80f0b50 con 0x7f71d80ef610 2021-02-10T11:11:40.926 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:40.913 7f71dd1c8700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71d80f0820 con 0x7f71d81b0470 2021-02-10T11:11:40.926 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:40.913 7f71dd1c8700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71d80f24e0 con 0x7f71d80f20e0 2021-02-10T11:11:40.926 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:40.914 7f71d659c700 1 --1- >> v1:172.21.2.113:6789/0 conn(0x7f71d80ef610 0x7f71d80efa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.113:6789/0 says I am v1:172.21.2.102:51658/0 (socket says 172.21.2.102:51658) 2021-02-10T11:11:40.926 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:40.914 7f71d659c700 1 -- 172.21.2.102:0/559168981 learned_addr learned my addr 172.21.2.102:0/559168981 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:11:40.926 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:40.914 7f71d759e700 1 --2- 172.21.2.102:0/559168981 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f71d80f20e0 0x7f71d80f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:11:40.926 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:40.914 7f71d6d9d700 1 --2- 172.21.2.102:0/559168981 >> v2:172.21.2.113:3300/0 conn(0x7f71d81b0470 0x7f71d81b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:11:40.927 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:40.914 7f71d759e700 1 -- 172.21.2.102:0/559168981 >> v2:172.21.2.113:3300/0 conn(0x7f71d81b0470 msgr2=0x7f71d81b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:11:40.927 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:40.914 7f71d759e700 1 --2- 172.21.2.102:0/559168981 >> v2:172.21.2.113:3300/0 conn(0x7f71d81b0470 0x7f71d81b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:11:40.927 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:40.914 7f71d759e700 1 -- 172.21.2.102:0/559168981 >> v1:172.21.2.113:6789/0 conn(0x7f71d80ef610 legacy=0x7f71d80efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:11:40.927 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:40.914 7f71d759e700 1 -- 172.21.2.102:0/559168981 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71d80e52e0 con 0x7f71d80f20e0 2021-02-10T11:11:40.927 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:40.914 7f71d759e700 1 --2- 172.21.2.102:0/559168981 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f71d80f20e0 0x7f71d80f0200 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f71d0006e50 tx=0x7f71d0009e70).ready entity=mon.2 client_cookie=26266c3e307fae5a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:11:40.927 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:40.915 7f71d5d9b700 1 -- 172.21.2.102:0/559168981 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f71d000b510 con 0x7f71d80f20e0 2021-02-10T11:11:40.927 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:40.915 7f71d5d9b700 1 -- 172.21.2.102:0/559168981 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71d0006790 con 0x7f71d80f20e0 2021-02-10T11:11:40.927 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:40.915 7f71d5d9b700 1 -- 172.21.2.102:0/559168981 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f71d000b790 con 0x7f71d80f20e0 2021-02-10T11:11:40.927 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:40.915 7f71dd1c8700 1 -- 172.21.2.102:0/559168981 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f71d80f20e0 msgr2=0x7f71d80f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:11:40.927 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:40.915 7f71dd1c8700 1 --2- 172.21.2.102:0/559168981 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f71d80f20e0 0x7f71d80f0200 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f71d0006e50 tx=0x7f71d0009e70).stop 2021-02-10T11:11:40.927 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:40.916 7f71dd1c8700 1 -- 172.21.2.102:0/559168981 shutdown_connections 2021-02-10T11:11:40.927 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:40.916 7f71dd1c8700 1 --2- 172.21.2.102:0/559168981 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f71d80f20e0 0x7f71d80f0200 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:11:40.927 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:40.916 7f71dd1c8700 1 --2- 172.21.2.102:0/559168981 >> v2:172.21.2.113:3300/0 conn(0x7f71d81b0470 0x7f71d81b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:11:40.928 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:40.916 7f71dd1c8700 1 -- 172.21.2.102:0/559168981 >> 172.21.2.102:0/559168981 conn(0x7f71d80ce3e0 msgr2=0x7f71d80e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:11:40.928 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:40.916 7f71dd1c8700 1 -- 172.21.2.102:0/559168981 shutdown_connections 2021-02-10T11:11:40.928 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:40.916 7f71dd1c8700 1 -- 172.21.2.102:0/559168981 wait complete. 2021-02-10T11:11:40.929 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:40.930 7f71dd1c8700 1 Processor -- start 2021-02-10T11:11:40.929 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:40.930 7f71dd1c8700 1 -- start start 2021-02-10T11:11:40.929 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:40.931 7f71dd1c8700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f71d8105d10 0x7f71d809eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:11:40.930 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:40.931 7f71dd1c8700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f71d8106110 0x7f71d809f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:11:40.930 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:40.931 7f71dd1c8700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f71d80b2a00 0x7f71d80b2e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:11:40.930 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:40.931 7f71dd1c8700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71d809e560 con 0x7f71d8105d10 2021-02-10T11:11:40.930 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:40.931 7f71dd1c8700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71d809c090 con 0x7f71d8106110 2021-02-10T11:11:40.930 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:40.931 7f71dd1c8700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71d80b3420 con 0x7f71d80b2a00 2021-02-10T11:11:40.930 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:40.931 7f71d6d9d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f71d8105d10 0x7f71d809eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:11:40.930 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:40.931 7f71d6d9d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f71d8105d10 0x7f71d809eae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:52116/0 (socket says 172.21.2.102:52116) 2021-02-10T11:11:40.930 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:40.931 7f71d6d9d700 1 -- 172.21.2.102:0/1156984248 learned_addr learned my addr 172.21.2.102:0/1156984248 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:11:40.930 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:40.931 7f71d6d9d700 1 -- 172.21.2.102:0/1156984248 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f71d80b2a00 msgr2=0x7f71d80b2e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:11:40.930 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:40.931 7f71d6d9d700 1 --2- 172.21.2.102:0/1156984248 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f71d80b2a00 0x7f71d80b2e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:11:40.930 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:40.931 7f71d6d9d700 1 -- 172.21.2.102:0/1156984248 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f71d8106110 msgr2=0x7f71d809f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:11:40.931 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:40.931 7f71d6d9d700 1 --2- 172.21.2.102:0/1156984248 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f71d8106110 0x7f71d809f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:11:40.931 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:40.931 7f71d6d9d700 1 -- 172.21.2.102:0/1156984248 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71d80b3640 con 0x7f71d8105d10 2021-02-10T11:11:40.931 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:40.931 7f71d6d9d700 1 --2- 172.21.2.102:0/1156984248 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f71d8105d10 0x7f71d809eae0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 rx=0x7f71c0006f60 tx=0x7f71c000b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:11:40.931 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:40.932 7f71cf7fe700 1 -- 172.21.2.102:0/1156984248 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f71c000bd60 con 0x7f71d8105d10 2021-02-10T11:11:40.931 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:40.932 7f71cf7fe700 1 -- 172.21.2.102:0/1156984248 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71c0000c90 con 0x7f71d8105d10 2021-02-10T11:11:40.931 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:40.932 7f71cf7fe700 1 -- 172.21.2.102:0/1156984248 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f71c00022b0 con 0x7f71d8105d10 2021-02-10T11:11:40.931 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:40.932 7f71dd1c8700 1 -- 172.21.2.102:0/1156984248 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71d80b3640 con 0x7f71d8105d10 2021-02-10T11:11:40.931 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:40.932 7f71dd1c8700 1 -- 172.21.2.102:0/1156984248 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71d80b3bd0 con 0x7f71d8105d10 2021-02-10T11:11:40.931 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:40.932 7f71cf7fe700 1 -- 172.21.2.102:0/1156984248 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f71c0006b00 con 0x7f71d8105d10 2021-02-10T11:11:40.931 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:40.932 7f71cf7fe700 1 --2- 172.21.2.102:0/1156984248 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f71bc0186c0 0x7f71bc01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:11:40.931 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:40.932 7f71cf7fe700 1 -- 172.21.2.102:0/1156984248 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f71c0032550 con 0x7f71d8105d10 2021-02-10T11:11:40.931 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:40.932 7f71d659c700 1 --2- 172.21.2.102:0/1156984248 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f71bc0186c0 0x7f71bc01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:11:40.931 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:40.933 7f71d659c700 1 --2- 172.21.2.102:0/1156984248 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f71bc0186c0 0x7f71bc01abc0 crc :-1 s=READY pgs=256 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:11:40.959 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:40.963 7f71dd1c8700 1 -- 172.21.2.102:0/1156984248 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71d8105ae0 con 0x7f71d8105d10 2021-02-10T11:11:40.962 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:40.966 7f71cf7fe700 1 -- 172.21.2.102:0/1156984248 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f71c00192a0 con 0x7f71d8105d10 2021-02-10T11:11:41.088 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:41.093 7f71dd1c8700 1 -- 172.21.2.102:0/1156984248 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344406", "target": ["mgr", ""]}) v1 -- 0x7f71d81a16c0 con 0x7f71bc0186c0 2021-02-10T11:11:41.116 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:41.120 7f71cf7fe700 1 -- 172.21.2.102:0/1156984248 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f71c8001590 con 0x7f71bc0186c0 2021-02-10T11:11:41.122 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:41.126 7f71dd1c8700 1 -- 172.21.2.102:0/1156984248 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f71bc0186c0 msgr2=0x7f71bc01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:11:41.122 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:41.126 7f71dd1c8700 1 --2- 172.21.2.102:0/1156984248 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f71bc0186c0 0x7f71bc01abc0 crc :-1 s=READY pgs=256 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:11:41.122 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:41.126 7f71dd1c8700 1 -- 172.21.2.102:0/1156984248 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f71d8105d10 msgr2=0x7f71d809eae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:11:41.123 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:41.126 7f71dd1c8700 1 --2- 172.21.2.102:0/1156984248 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f71d8105d10 0x7f71d809eae0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 rx=0x7f71c0006f60 tx=0x7f71c000b970).stop 2021-02-10T11:11:41.123 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:41.126 7f71dd1c8700 1 -- 172.21.2.102:0/1156984248 shutdown_connections 2021-02-10T11:11:41.123 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:41.126 7f71dd1c8700 1 --2- 172.21.2.102:0/1156984248 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f71d80b2a00 0x7f71d80b2e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:11:41.123 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:41.126 7f71dd1c8700 1 --2- 172.21.2.102:0/1156984248 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f71bc0186c0 0x7f71bc01abc0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:11:41.123 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:41.126 7f71dd1c8700 1 --2- 172.21.2.102:0/1156984248 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f71d8106110 0x7f71d809f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:11:41.123 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:41.126 7f71dd1c8700 1 --2- 172.21.2.102:0/1156984248 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f71d8105d10 0x7f71d809eae0 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:11:41.123 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:41.126 7f71dd1c8700 1 -- 172.21.2.102:0/1156984248 >> 172.21.2.102:0/1156984248 conn(0x7f71d80c36f0 msgr2=0x7f71d80bd640 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:11:41.123 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:41.126 7f71dd1c8700 1 -- 172.21.2.102:0/1156984248 shutdown_connections 2021-02-10T11:11:41.123 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:41.126 7f71dd1c8700 1 -- 172.21.2.102:0/1156984248 wait complete. 2021-02-10T11:11:41.130 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000344406 2021-02-10T11:11:41.561 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:41.549 7f164a009700 1 Processor -- start 2021-02-10T11:11:41.561 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:41.549 7f164a009700 1 -- start start 2021-02-10T11:11:41.561 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:41.549 7f164a009700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f16441b0870 0x7f16440f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:11:41.561 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:41.549 7f164a009700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16441b0c70 con 0x7f16441b0470 2021-02-10T11:11:41.561 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:41.549 7f164a009700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16440f0b50 con 0x7f16440efe00 2021-02-10T11:11:41.561 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:41.549 7f164a009700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16440e5ea0 con 0x7f16441b0870 2021-02-10T11:11:41.561 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:41.550 7f1642ffd700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f16441b0870 0x7f16440f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:11:41.562 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:41.550 7f16437fe700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f16441b0470 0x7f16440ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:44346/0 (socket says 172.21.2.102:44346) 2021-02-10T11:11:41.562 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:41.550 7f16437fe700 1 -- 172.21.2.102:0/3706635495 learned_addr learned my addr 172.21.2.102:0/3706635495 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:11:41.562 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:41.550 7f1642ffd700 1 -- 172.21.2.102:0/3706635495 >> v1:172.21.2.113:6789/0 conn(0x7f16440efe00 legacy=0x7f16440f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:11:41.562 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:41.550 7f1642ffd700 1 -- 172.21.2.102:0/3706635495 >> v1:172.21.2.102:6789/0 conn(0x7f16441b0470 legacy=0x7f16440ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:11:41.562 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:41.550 7f1642ffd700 1 -- 172.21.2.102:0/3706635495 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16440e52e0 con 0x7f16441b0870 2021-02-10T11:11:41.562 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:41.550 7f1642ffd700 1 --2- 172.21.2.102:0/3706635495 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f16441b0870 0x7f16440f20e0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 rx=0x7f1634006650 tx=0x7f1634009a00).ready entity=mon.0 client_cookie=45e9e67f6690ec05 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:11:41.562 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:41.551 7f16427fc700 1 -- 172.21.2.102:0/3706635495 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1634009df0 con 0x7f16441b0870 2021-02-10T11:11:41.562 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:41.551 7f16427fc700 1 -- 172.21.2.102:0/3706635495 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16340009b0 con 0x7f16441b0870 2021-02-10T11:11:41.562 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:41.551 7f16427fc700 1 -- 172.21.2.102:0/3706635495 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f163400c2b0 con 0x7f16441b0870 2021-02-10T11:11:41.562 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:41.551 7f164a009700 1 -- 172.21.2.102:0/3706635495 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f16441b0870 msgr2=0x7f16440f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:11:41.562 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:41.551 7f164a009700 1 --2- 172.21.2.102:0/3706635495 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f16441b0870 0x7f16440f20e0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 rx=0x7f1634006650 tx=0x7f1634009a00).stop 2021-02-10T11:11:41.563 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:41.551 7f164a009700 1 -- 172.21.2.102:0/3706635495 shutdown_connections 2021-02-10T11:11:41.563 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:41.551 7f164a009700 1 --2- 172.21.2.102:0/3706635495 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f16441b0870 0x7f16440f20e0 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:11:41.563 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:41.551 7f164a009700 1 -- 172.21.2.102:0/3706635495 >> 172.21.2.102:0/3706635495 conn(0x7f16440ce3e0 msgr2=0x7f16440e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:11:41.563 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:41.551 7f164a009700 1 -- 172.21.2.102:0/3706635495 shutdown_connections 2021-02-10T11:11:41.563 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:41.551 7f164a009700 1 -- 172.21.2.102:0/3706635495 wait complete. 2021-02-10T11:11:41.564 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:41.566 7f164a009700 1 Processor -- start 2021-02-10T11:11:41.564 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:41.566 7f164a009700 1 -- start start 2021-02-10T11:11:41.565 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:41.566 7f164a009700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1644087510 0x7f16440aedc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:11:41.565 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:41.566 7f164a009700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f16441c68f0 0x7f16441b1f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:11:41.565 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:41.566 7f164a009700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f164408c4d0 0x7f16441b18a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:11:41.565 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:41.566 7f164a009700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16440a2d50 con 0x7f1644087510 2021-02-10T11:11:41.565 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:41.566 7f164a009700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16440a2f70 con 0x7f164408c4d0 2021-02-10T11:11:41.565 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:41.566 7f164a009700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16440a3190 con 0x7f16441c68f0 2021-02-10T11:11:41.565 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:41.566 7f16437fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1644087510 0x7f16440aedc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:11:41.565 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:41.567 7f16437fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1644087510 0x7f16440aedc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:52126/0 (socket says 172.21.2.102:52126) 2021-02-10T11:11:41.565 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:41.567 7f16437fe700 1 -- 172.21.2.102:0/4134742554 learned_addr learned my addr 172.21.2.102:0/4134742554 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:11:41.565 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:41.567 7f1643fff700 1 --2- 172.21.2.102:0/4134742554 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f164408c4d0 0x7f16441b18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:11:41.565 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:41.567 7f16437fe700 1 -- 172.21.2.102:0/4134742554 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f16441c68f0 msgr2=0x7f16441b1f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:11:41.565 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:41.567 7f16437fe700 1 --2- 172.21.2.102:0/4134742554 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f16441c68f0 0x7f16441b1f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:11:41.566 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:41.567 7f16437fe700 1 -- 172.21.2.102:0/4134742554 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f164408c4d0 msgr2=0x7f16441b18a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:11:41.566 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:41.567 7f16437fe700 1 --2- 172.21.2.102:0/4134742554 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f164408c4d0 0x7f16441b18a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:11:41.566 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:41.567 7f16437fe700 1 -- 172.21.2.102:0/4134742554 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16440a33b0 con 0x7f1644087510 2021-02-10T11:11:41.566 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:41.567 7f1643fff700 1 --2- 172.21.2.102:0/4134742554 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f164408c4d0 0x7f16441b18a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:11:41.566 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:41.567 7f16437fe700 1 --2- 172.21.2.102:0/4134742554 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1644087510 0x7f16440aedc0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 rx=0x7f162c001410 tx=0x7f162c00a950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:11:41.566 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:41.567 7f163bfff700 1 -- 172.21.2.102:0/4134742554 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f162c00ad40 con 0x7f1644087510 2021-02-10T11:11:41.566 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:41.567 7f163bfff700 1 -- 172.21.2.102:0/4134742554 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f162c0009b0 con 0x7f1644087510 2021-02-10T11:11:41.566 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:41.567 7f163bfff700 1 -- 172.21.2.102:0/4134742554 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f162c004590 con 0x7f1644087510 2021-02-10T11:11:41.566 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:41.568 7f164a009700 1 -- 172.21.2.102:0/4134742554 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16440a33b0 con 0x7f1644087510 2021-02-10T11:11:41.566 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:41.568 7f164a009700 1 -- 172.21.2.102:0/4134742554 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16440afca0 con 0x7f1644087510 2021-02-10T11:11:41.566 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:41.568 7f163bfff700 1 -- 172.21.2.102:0/4134742554 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f162c0043a0 con 0x7f1644087510 2021-02-10T11:11:41.566 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:41.568 7f163bfff700 1 --2- 172.21.2.102:0/4134742554 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f16280186c0 0x7f162801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:11:41.566 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:41.568 7f163bfff700 1 -- 172.21.2.102:0/4134742554 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f162c0325e0 con 0x7f1644087510 2021-02-10T11:11:41.567 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:41.568 7f1642ffd700 1 --2- 172.21.2.102:0/4134742554 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f16280186c0 0x7f162801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:11:41.567 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:41.569 7f1642ffd700 1 --2- 172.21.2.102:0/4134742554 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f16280186c0 0x7f162801abc0 crc :-1 s=READY pgs=257 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:11:41.595 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:41.599 7f164a009700 1 -- 172.21.2.102:0/4134742554 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f164407e6b0 con 0x7f1644087510 2021-02-10T11:11:41.598 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:41.602 7f163bfff700 1 -- 172.21.2.102:0/4134742554 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f162c02c3d0 con 0x7f1644087510 2021-02-10T11:11:41.724 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:41.729 7f164a009700 1 -- 172.21.2.102:0/4134742554 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344406", "target": ["mgr", ""]}) v1 -- 0x7f16440afca0 con 0x7f16280186c0 2021-02-10T11:11:41.728 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:41.733 7f163bfff700 1 -- 172.21.2.102:0/4134742554 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f1634006740 con 0x7f16280186c0 2021-02-10T11:11:41.732 INFO:teuthology.orchestra.run.gibba002.stdout:/volumes/_nogroup/subvolume_0000000000344406/3d6c35ea-4f2a-420d-826c-5802da4eb7d0 2021-02-10T11:11:41.733 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:41.737 7f164a009700 1 -- 172.21.2.102:0/4134742554 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f16280186c0 msgr2=0x7f162801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:11:41.733 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:41.737 7f164a009700 1 --2- 172.21.2.102:0/4134742554 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f16280186c0 0x7f162801abc0 crc :-1 s=READY pgs=257 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:11:41.733 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:41.737 7f164a009700 1 -- 172.21.2.102:0/4134742554 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1644087510 msgr2=0x7f16440aedc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:11:41.733 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:41.737 7f164a009700 1 --2- 172.21.2.102:0/4134742554 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1644087510 0x7f16440aedc0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 rx=0x7f162c001410 tx=0x7f162c00a950).stop 2021-02-10T11:11:41.734 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:41.738 7f164a009700 1 -- 172.21.2.102:0/4134742554 shutdown_connections 2021-02-10T11:11:41.734 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:41.738 7f164a009700 1 --2- 172.21.2.102:0/4134742554 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f16280186c0 0x7f162801abc0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:11:41.734 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:41.738 7f164a009700 1 --2- 172.21.2.102:0/4134742554 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f164408c4d0 0x7f16441b18a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:11:41.734 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:41.738 7f164a009700 1 --2- 172.21.2.102:0/4134742554 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f16441c68f0 0x7f16441b1f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:11:41.734 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:41.738 7f164a009700 1 --2- 172.21.2.102:0/4134742554 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1644087510 0x7f16440aedc0 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:11:41.734 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:41.738 7f164a009700 1 -- 172.21.2.102:0/4134742554 >> 172.21.2.102:0/4134742554 conn(0x7f16440886d0 msgr2=0x7f1644081c40 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:11:41.734 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:41.738 7f164a009700 1 -- 172.21.2.102:0/4134742554 shutdown_connections 2021-02-10T11:11:41.734 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:41.738 7f164a009700 1 -- 172.21.2.102:0/4134742554 wait complete. 2021-02-10T11:11:41.741 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000344406 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000344406/3d6c35ea-4f2a-420d-826c-5802da4eb7d0 2021-02-10T11:11:41.741 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:11:41.742 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:11:41.742 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:11:42.078 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:11:42.079 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:11:42.079 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344406/3d6c35ea-4f2a-420d-826c-5802da4eb7d0/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:11:42.461 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:11:42.461 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:11:42.461 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0245957 s, 42.6 MB/s 2021-02-10T11:11:42.462 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:11:42.462 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:11:42.462 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:11:42.798 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:11:42.798 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:11:42.799 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344406/3d6c35ea-4f2a-420d-826c-5802da4eb7d0/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:11:43.172 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:11:43.172 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:11:43.172 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0228578 s, 45.9 MB/s 2021-02-10T11:11:43.173 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:11:43.173 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:11:43.173 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:11:43.509 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:11:43.510 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:11:43.510 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344406/3d6c35ea-4f2a-420d-826c-5802da4eb7d0/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:11:43.881 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:11:43.881 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:11:43.881 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.022715 s, 46.2 MB/s 2021-02-10T11:11:43.882 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:11:43.882 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:11:43.883 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:11:44.218 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:11:44.219 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:11:44.219 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344406/3d6c35ea-4f2a-420d-826c-5802da4eb7d0/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:11:44.595 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:11:44.595 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:11:44.595 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0242823 s, 43.2 MB/s 2021-02-10T11:11:44.596 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:11:44.596 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:11:44.596 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:11:44.932 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:11:44.932 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:11:44.932 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344406/3d6c35ea-4f2a-420d-826c-5802da4eb7d0/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:11:45.310 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:11:45.310 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:11:45.310 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0249225 s, 42.1 MB/s 2021-02-10T11:11:45.311 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:11:45.311 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:11:45.312 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:11:45.648 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:11:45.648 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:11:45.649 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344406/3d6c35ea-4f2a-420d-826c-5802da4eb7d0/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:11:46.025 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:11:46.025 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:11:46.025 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0250048 s, 41.9 MB/s 2021-02-10T11:11:46.026 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:11:46.027 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:11:46.027 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:11:46.363 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:11:46.364 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:11:46.364 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344406/3d6c35ea-4f2a-420d-826c-5802da4eb7d0/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:11:46.740 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:11:46.740 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:11:46.740 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.025336 s, 41.4 MB/s 2021-02-10T11:11:46.741 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:11:46.741 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:11:46.741 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:11:47.076 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:11:47.077 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:11:47.077 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344406/3d6c35ea-4f2a-420d-826c-5802da4eb7d0/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:11:47.453 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:11:50.199 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:11:50.199 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0257927 s, 40.7 MB/s 2021-02-10T11:11:50.201 DEBUG:teuthology.orchestra.run.gibba002:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:11:50.202 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:11:50.202 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:11:50.202 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:11:50.205 DEBUG:teuthology.orchestra.run.gibba013:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:11:50.540 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:11:50.541 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:11:50.541 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344406/3d6c35ea-4f2a-420d-826c-5802da4eb7d0/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:11:50.916 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:11:51.031 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:11:51.031 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0262628 s, 39.9 MB/s 2021-02-10T11:11:51.032 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:11:51.032 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:11:51.033 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:11:51.368 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:11:51.368 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:11:51.368 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344406/3d6c35ea-4f2a-420d-826c-5802da4eb7d0/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:11:51.745 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:11:51.745 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:11:51.745 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0242265 s, 43.3 MB/s 2021-02-10T11:11:51.746 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000344407 2021-02-10T11:11:52.177 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:52.164 7f19d7343700 1 Processor -- start 2021-02-10T11:11:52.177 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:52.164 7f19d7343700 1 -- start start 2021-02-10T11:11:52.177 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:52.165 7f19d7343700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f19d01b0470 0x7f19d01b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:11:52.177 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:52.165 7f19d7343700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f19d00ef610 0x7f19d00efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:11:52.177 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:52.165 7f19d7343700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19d00f0b50 con 0x7f19d00f0030 2021-02-10T11:11:52.177 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:52.165 7f19d7343700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19d00e52e0 con 0x7f19d00ef610 2021-02-10T11:11:52.177 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:52.165 7f19d7343700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19d00e5490 con 0x7f19d01b0470 2021-02-10T11:11:52.177 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:52.165 7f19d495e700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f19d00ef610 0x7f19d00efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:11:52.177 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:52.165 7f19d495e700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f19d00ef610 0x7f19d00efa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:52132/0 (socket says 172.21.2.102:52132) 2021-02-10T11:11:52.178 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:52.165 7f19d495e700 1 -- 172.21.2.102:0/2966838269 learned_addr learned my addr 172.21.2.102:0/2966838269 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:11:52.178 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:52.165 7f19d515f700 1 --2- 172.21.2.102:0/2966838269 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f19d01b0470 0x7f19d01b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:11:52.178 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:52.166 7f19d495e700 1 -- 172.21.2.102:0/2966838269 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f19d01b0470 msgr2=0x7f19d01b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:11:52.178 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:52.166 7f19d495e700 1 --2- 172.21.2.102:0/2966838269 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f19d01b0470 0x7f19d01b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:11:52.178 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:52.166 7f19d495e700 1 -- 172.21.2.102:0/2966838269 >> v1:172.21.2.113:6789/0 conn(0x7f19d00f0030 legacy=0x7f19d00f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:11:52.178 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:52.166 7f19d495e700 1 -- 172.21.2.102:0/2966838269 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19d00f0430 con 0x7f19d00ef610 2021-02-10T11:11:52.178 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:52.166 7f19d495e700 1 --2- 172.21.2.102:0/2966838269 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f19d00ef610 0x7f19d00efa10 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 rx=0x7f19c0006700 tx=0x7f19c0009b80).ready entity=mon.0 client_cookie=bd0977c7fd0ac6e8 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:11:52.178 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:52.166 7f19c7fff700 1 -- 172.21.2.102:0/2966838269 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f19c000c0c0 con 0x7f19d00ef610 2021-02-10T11:11:52.178 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:52.166 7f19c7fff700 1 -- 172.21.2.102:0/2966838269 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19c000c440 con 0x7f19d00ef610 2021-02-10T11:11:52.178 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:52.166 7f19c7fff700 1 -- 172.21.2.102:0/2966838269 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f19c000c960 con 0x7f19d00ef610 2021-02-10T11:11:52.178 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:52.166 7f19d7343700 1 -- 172.21.2.102:0/2966838269 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f19d00ef610 msgr2=0x7f19d00efa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:11:52.178 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:52.166 7f19d7343700 1 --2- 172.21.2.102:0/2966838269 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f19d00ef610 0x7f19d00efa10 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 rx=0x7f19c0006700 tx=0x7f19c0009b80).stop 2021-02-10T11:11:52.178 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:52.167 7f19d7343700 1 -- 172.21.2.102:0/2966838269 shutdown_connections 2021-02-10T11:11:52.179 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:52.167 7f19d7343700 1 --2- 172.21.2.102:0/2966838269 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f19d00ef610 0x7f19d00efa10 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:11:52.179 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:52.167 7f19d7343700 1 --2- 172.21.2.102:0/2966838269 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f19d01b0470 0x7f19d01b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:11:52.179 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:52.167 7f19d7343700 1 -- 172.21.2.102:0/2966838269 >> 172.21.2.102:0/2966838269 conn(0x7f19d00ce3e0 msgr2=0x7f19d00e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:11:52.179 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:52.167 7f19d7343700 1 -- 172.21.2.102:0/2966838269 shutdown_connections 2021-02-10T11:11:52.179 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:52.167 7f19d7343700 1 -- 172.21.2.102:0/2966838269 wait complete. 2021-02-10T11:11:52.180 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:52.182 7f19d7343700 1 Processor -- start 2021-02-10T11:11:52.181 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:52.182 7f19d7343700 1 -- start start 2021-02-10T11:11:52.181 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:52.182 7f19d7343700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f19d009f170 0x7f19d00bb310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:11:52.181 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:52.182 7f19d7343700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f19d009e640 0x7f19d009ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:11:52.181 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:52.182 7f19d7343700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f19d00bc1d0 0x7f19d00bc5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:11:52.181 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:52.182 7f19d7343700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19d00bbcd0 con 0x7f19d009f170 2021-02-10T11:11:52.182 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:52.182 7f19d7343700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19d00bf930 con 0x7f19d009e640 2021-02-10T11:11:52.182 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:52.182 7f19d7343700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19d00bb930 con 0x7f19d00bc1d0 2021-02-10T11:11:52.182 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:52.182 7f19d515f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f19d009f170 0x7f19d00bb310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:11:52.182 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:52.182 7f19d515f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f19d009f170 0x7f19d00bb310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:52136/0 (socket says 172.21.2.102:52136) 2021-02-10T11:11:52.182 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:52.182 7f19d515f700 1 -- 172.21.2.102:0/1368138258 learned_addr learned my addr 172.21.2.102:0/1368138258 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:11:52.182 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:52.183 7f19d515f700 1 -- 172.21.2.102:0/1368138258 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f19d00bc1d0 msgr2=0x7f19d00bc5d0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:11:52.182 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:52.183 7f19d515f700 1 --2- 172.21.2.102:0/1368138258 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f19d00bc1d0 0x7f19d00bc5d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:11:52.182 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:52.183 7f19d515f700 1 -- 172.21.2.102:0/1368138258 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f19d009e640 msgr2=0x7f19d009ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:11:52.182 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:52.183 7f19d515f700 1 --2- 172.21.2.102:0/1368138258 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f19d009e640 0x7f19d009ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:11:52.182 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:52.183 7f19d515f700 1 -- 172.21.2.102:0/1368138258 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19d00bcbf0 con 0x7f19d009f170 2021-02-10T11:11:52.182 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:52.183 7f19d515f700 1 --2- 172.21.2.102:0/1368138258 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f19d009f170 0x7f19d00bb310 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 rx=0x7f19cc006f00 tx=0x7f19cc00b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:11:52.182 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:52.183 7f19c5ffb700 1 -- 172.21.2.102:0/1368138258 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f19cc00bd30 con 0x7f19d009f170 2021-02-10T11:11:52.182 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:52.183 7f19c5ffb700 1 -- 172.21.2.102:0/1368138258 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19cc002030 con 0x7f19d009f170 2021-02-10T11:11:52.183 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:52.183 7f19c5ffb700 1 -- 172.21.2.102:0/1368138258 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f19cc002550 con 0x7f19d009f170 2021-02-10T11:11:52.183 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:52.183 7f19d7343700 1 -- 172.21.2.102:0/1368138258 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19d00bcbf0 con 0x7f19d009f170 2021-02-10T11:11:52.183 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:52.183 7f19d7343700 1 -- 172.21.2.102:0/1368138258 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19d00bd180 con 0x7f19d009f170 2021-02-10T11:11:52.183 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:52.184 7f19c5ffb700 1 -- 172.21.2.102:0/1368138258 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f19cc000c90 con 0x7f19d009f170 2021-02-10T11:11:52.183 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:52.184 7f19c5ffb700 1 --2- 172.21.2.102:0/1368138258 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f19b00186c0 0x7f19b001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:11:52.183 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:52.184 7f19c5ffb700 1 -- 172.21.2.102:0/1368138258 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f19cc032570 con 0x7f19d009f170 2021-02-10T11:11:52.183 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:52.184 7f19d495e700 1 --2- 172.21.2.102:0/1368138258 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f19b00186c0 0x7f19b001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:11:52.183 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:52.184 7f19d495e700 1 --2- 172.21.2.102:0/1368138258 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f19b00186c0 0x7f19b001abc0 crc :-1 s=READY pgs=258 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:11:52.210 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:52.215 7f19d7343700 1 -- 172.21.2.102:0/1368138258 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19d0105ae0 con 0x7f19d009f170 2021-02-10T11:11:52.213 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:52.217 7f19c5ffb700 1 -- 172.21.2.102:0/1368138258 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f19cc0009b0 con 0x7f19d009f170 2021-02-10T11:11:52.340 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:52.344 7f19d7343700 1 -- 172.21.2.102:0/1368138258 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344407", "target": ["mgr", ""]}) v1 -- 0x7f19d00bd180 con 0x7f19b00186c0 2021-02-10T11:11:52.367 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:52.371 7f19c5ffb700 1 -- 172.21.2.102:0/1368138258 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f19c0000990 con 0x7f19b00186c0 2021-02-10T11:11:52.373 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:52.378 7f19d7343700 1 -- 172.21.2.102:0/1368138258 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f19b00186c0 msgr2=0x7f19b001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:11:52.374 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:52.378 7f19d7343700 1 --2- 172.21.2.102:0/1368138258 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f19b00186c0 0x7f19b001abc0 crc :-1 s=READY pgs=258 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:11:52.374 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:52.378 7f19d7343700 1 -- 172.21.2.102:0/1368138258 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f19d009f170 msgr2=0x7f19d00bb310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:11:52.374 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:52.378 7f19d7343700 1 --2- 172.21.2.102:0/1368138258 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f19d009f170 0x7f19d00bb310 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 rx=0x7f19cc006f00 tx=0x7f19cc00b940).stop 2021-02-10T11:11:52.375 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:52.378 7f19d7343700 1 -- 172.21.2.102:0/1368138258 shutdown_connections 2021-02-10T11:11:52.375 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:52.378 7f19d7343700 1 --2- 172.21.2.102:0/1368138258 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f19d00bc1d0 0x7f19d00bc5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:11:52.375 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:52.378 7f19d7343700 1 --2- 172.21.2.102:0/1368138258 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f19b00186c0 0x7f19b001abc0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:11:52.375 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:52.378 7f19d7343700 1 --2- 172.21.2.102:0/1368138258 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f19d009e640 0x7f19d009ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:11:52.375 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:52.378 7f19d7343700 1 --2- 172.21.2.102:0/1368138258 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f19d009f170 0x7f19d00bb310 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:11:52.375 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:52.378 7f19d7343700 1 -- 172.21.2.102:0/1368138258 >> 172.21.2.102:0/1368138258 conn(0x7f19d009fc10 msgr2=0x7f19d00b3250 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:11:52.375 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:52.378 7f19d7343700 1 -- 172.21.2.102:0/1368138258 shutdown_connections 2021-02-10T11:11:52.375 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:52.378 7f19d7343700 1 -- 172.21.2.102:0/1368138258 wait complete. 2021-02-10T11:11:52.382 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000344407 2021-02-10T11:11:52.815 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:52.801 7f873b524700 1 Processor -- start 2021-02-10T11:11:52.815 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:52.802 7f873b524700 1 -- start start 2021-02-10T11:11:52.815 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:52.802 7f873b524700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f87340f05f0 0x7f87340f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:11:52.815 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:52.802 7f873b524700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87341b0c70 con 0x7f87341b0470 2021-02-10T11:11:52.815 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:52.802 7f873b524700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87340f0b50 con 0x7f87341b0870 2021-02-10T11:11:52.815 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:52.802 7f873b524700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87340e5ea0 con 0x7f87340f05f0 2021-02-10T11:11:52.815 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:52.802 7f8739340700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f87341b0470 0x7f87340ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:44368/0 (socket says 172.21.2.102:44368) 2021-02-10T11:11:52.816 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:52.802 7f8739340700 1 -- 172.21.2.102:0/2025655005 learned_addr learned my addr 172.21.2.102:0/2025655005 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:11:52.816 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:52.803 7f8739b41700 1 --2- 172.21.2.102:0/2025655005 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f87340f05f0 0x7f87340f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:11:52.816 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:52.803 7f8727fff700 1 -- 172.21.2.102:0/2025655005 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2039788811 0 0) 0x7f8730001180 con 0x7f87341b0470 2021-02-10T11:11:52.816 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:52.803 7f8727fff700 1 -- 172.21.2.102:0/2025655005 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8720002fe0 con 0x7f87341b0470 2021-02-10T11:11:52.816 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:52.803 7f8727fff700 1 -- 172.21.2.102:0/2025655005 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 478493441 0 0) 0x7f8730001180 con 0x7f87341b0470 2021-02-10T11:11:52.816 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:52.803 7f8727fff700 1 -- 172.21.2.102:0/2025655005 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f87340f05f0 msgr2=0x7f87340f20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:11:52.816 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:52.803 7f8727fff700 1 --2- 172.21.2.102:0/2025655005 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f87340f05f0 0x7f87340f20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:11:52.816 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:52.803 7f8727fff700 1 -- 172.21.2.102:0/2025655005 >> v1:172.21.2.113:6789/0 conn(0x7f87341b0870 legacy=0x7f87340efe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:11:52.816 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:52.803 7f8727fff700 1 -- 172.21.2.102:0/2025655005 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87340e52e0 con 0x7f87341b0470 2021-02-10T11:11:52.816 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:52.803 7f8727fff700 1 -- 172.21.2.102:0/2025655005 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7f8730001680 con 0x7f87341b0470 2021-02-10T11:11:52.825 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:52.803 7f8727fff700 1 -- 172.21.2.102:0/2025655005 >> v1:172.21.2.102:6789/0 conn(0x7f87341b0470 legacy=0x7f87340ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:11:52.825 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:52.803 7f8727fff700 1 --2- 172.21.2.102:0/2025655005 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8720003b90 0x7f8720005fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:11:52.825 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:52.803 7f8727fff700 1 --2- 172.21.2.102:0/2025655005 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f87200067e0 0x7f8720008ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:11:52.826 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:52.803 7f8727fff700 1 --2- 172.21.2.102:0/2025655005 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f87200095c0 0x7f872000ba90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:11:52.826 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:52.803 7f8727fff700 1 -- 172.21.2.102:0/2025655005 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f872000c2e0 con 0x7f8720003b90 2021-02-10T11:11:52.826 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:52.803 7f8727fff700 1 -- 172.21.2.102:0/2025655005 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f872000c500 con 0x7f87200067e0 2021-02-10T11:11:52.826 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:52.803 7f8727fff700 1 -- 172.21.2.102:0/2025655005 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f872000c720 con 0x7f87200095c0 2021-02-10T11:11:52.826 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:52.803 7f8739340700 1 --2- 172.21.2.102:0/2025655005 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8720003b90 0x7f8720005fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:11:52.826 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:52.804 7f8738b3f700 1 --2- 172.21.2.102:0/2025655005 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f87200067e0 0x7f8720008ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:11:52.826 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:52.804 7f8739340700 1 -- 172.21.2.102:0/2025655005 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f87200095c0 msgr2=0x7f872000ba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:11:52.826 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:52.804 7f8739340700 1 --2- 172.21.2.102:0/2025655005 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f87200095c0 0x7f872000ba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:11:52.826 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:52.804 7f8739340700 1 -- 172.21.2.102:0/2025655005 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f87200067e0 msgr2=0x7f8720008ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:11:52.826 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:52.804 7f8739340700 1 --2- 172.21.2.102:0/2025655005 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f87200067e0 0x7f8720008ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:11:52.826 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:52.804 7f8739340700 1 -- 172.21.2.102:0/2025655005 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8720050990 con 0x7f8720003b90 2021-02-10T11:11:52.827 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:52.804 7f8739340700 1 --2- 172.21.2.102:0/2025655005 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8720003b90 0x7f8720005fc0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 rx=0x7f8730001520 tx=0x7f8730007d40).ready entity=mon.0 client_cookie=c0bd3378080a9ee1 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:11:52.829 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:52.804 7f8727fff700 1 -- 172.21.2.102:0/2025655005 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f87300070c0 con 0x7f8720003b90 2021-02-10T11:11:52.829 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:52.804 7f8727fff700 1 -- 172.21.2.102:0/2025655005 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f873000a520 con 0x7f8720003b90 2021-02-10T11:11:52.829 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:52.804 7f8727fff700 1 -- 172.21.2.102:0/2025655005 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f873000aa00 con 0x7f8720003b90 2021-02-10T11:11:52.830 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:52.804 7f873b524700 1 -- 172.21.2.102:0/2025655005 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8720003b90 msgr2=0x7f8720005fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:11:52.830 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:52.804 7f873b524700 1 --2- 172.21.2.102:0/2025655005 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8720003b90 0x7f8720005fc0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 rx=0x7f8730001520 tx=0x7f8730007d40).stop 2021-02-10T11:11:52.830 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:52.804 7f8739b41700 1 -- 172.21.2.102:0/2025655005 reap_dead start 2021-02-10T11:11:52.830 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:52.805 7f873b524700 1 -- 172.21.2.102:0/2025655005 shutdown_connections 2021-02-10T11:11:52.830 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:52.805 7f873b524700 1 -- 172.21.2.102:0/2025655005 >> 172.21.2.102:0/2025655005 conn(0x7f87340ce3e0 msgr2=0x7f87340e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:11:52.830 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:52.805 7f873b524700 1 -- 172.21.2.102:0/2025655005 shutdown_connections 2021-02-10T11:11:52.830 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:52.805 7f873b524700 1 -- 172.21.2.102:0/2025655005 wait complete. 2021-02-10T11:11:52.830 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:52.819 7f873b524700 1 Processor -- start 2021-02-10T11:11:52.830 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:52.819 7f873b524700 1 -- start start 2021-02-10T11:11:52.830 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:52.820 7f873b524700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f873409b930 0x7f87340b7c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:11:52.830 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:52.820 7f873b524700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f87340b8410 0x7f87340b8810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:11:52.830 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:52.820 7f873b524700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f87340c1f80 0x7f87340c2380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:11:52.830 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:52.820 7f873b524700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87340c1a00 con 0x7f87340c1f80 2021-02-10T11:11:52.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:52.820 7f873b524700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87340b8e30 con 0x7f87340b8410 2021-02-10T11:11:52.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:52.820 7f873b524700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87340b9050 con 0x7f873409b930 2021-02-10T11:11:52.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:52.820 7f8739b41700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f87340c1f80 0x7f87340c2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:11:52.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:52.820 7f8739b41700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f87340c1f80 0x7f87340c2380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:52156/0 (socket says 172.21.2.102:52156) 2021-02-10T11:11:52.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:52.820 7f8739b41700 1 -- 172.21.2.102:0/328464975 learned_addr learned my addr 172.21.2.102:0/328464975 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:11:52.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:52.820 7f8739b41700 1 -- 172.21.2.102:0/328464975 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f873409b930 msgr2=0x7f87340b7c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:11:52.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:52.820 7f8739b41700 1 --2- 172.21.2.102:0/328464975 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f873409b930 0x7f87340b7c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:11:52.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:52.820 7f8739b41700 1 -- 172.21.2.102:0/328464975 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f87340b8410 msgr2=0x7f87340b8810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:11:52.832 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:52.820 7f8739b41700 1 --2- 172.21.2.102:0/328464975 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f87340b8410 0x7f87340b8810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:11:52.832 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:52.820 7f8739b41700 1 -- 172.21.2.102:0/328464975 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87340c29a0 con 0x7f87340c1f80 2021-02-10T11:11:52.832 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:52.821 7f8739b41700 1 --2- 172.21.2.102:0/328464975 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f87340c1f80 0x7f87340c2380 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 rx=0x7f872c009960 tx=0x7f872c00bd40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:11:52.832 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:52.821 7f8725ffb700 1 -- 172.21.2.102:0/328464975 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f872c0011f0 con 0x7f87340c1f80 2021-02-10T11:11:52.832 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:52.821 7f8725ffb700 1 -- 172.21.2.102:0/328464975 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f872c003030 con 0x7f87340c1f80 2021-02-10T11:11:52.832 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:52.821 7f8725ffb700 1 -- 172.21.2.102:0/328464975 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f872c001770 con 0x7f87340c1f80 2021-02-10T11:11:52.832 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:52.821 7f873b524700 1 -- 172.21.2.102:0/328464975 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87340c29a0 con 0x7f87340c1f80 2021-02-10T11:11:52.832 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:52.821 7f873b524700 1 -- 172.21.2.102:0/328464975 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87340c29a0 con 0x7f87340c1f80 2021-02-10T11:11:52.832 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:52.821 7f8725ffb700 1 -- 172.21.2.102:0/328464975 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f872c0011f0 con 0x7f87340c1f80 2021-02-10T11:11:52.838 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:52.822 7f8725ffb700 1 --2- 172.21.2.102:0/328464975 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f87140186c0 0x7f871401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:11:52.838 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:52.822 7f8725ffb700 1 -- 172.21.2.102:0/328464975 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f872c032770 con 0x7f87340c1f80 2021-02-10T11:11:52.838 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:52.822 7f8739340700 1 --2- 172.21.2.102:0/328464975 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f87140186c0 0x7f871401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:11:52.839 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:52.822 7f8739340700 1 --2- 172.21.2.102:0/328464975 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f87140186c0 0x7f871401abc0 crc :-1 s=READY pgs=259 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:11:52.847 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:52.852 7f873b524700 1 -- 172.21.2.102:0/328464975 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8734105b30 con 0x7f87340c1f80 2021-02-10T11:11:52.850 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:52.854 7f8725ffb700 1 -- 172.21.2.102:0/328464975 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f872c000f10 con 0x7f87340c1f80 2021-02-10T11:11:52.980 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:52.984 7f873b524700 1 -- 172.21.2.102:0/328464975 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344407", "target": ["mgr", ""]}) v1 -- 0x7f8734105b30 con 0x7f87140186c0 2021-02-10T11:11:52.983 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:52.988 7f8725ffb700 1 -- 172.21.2.102:0/328464975 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f8730000c90 con 0x7f87140186c0 2021-02-10T11:11:52.987 INFO:teuthology.orchestra.run.gibba002.stdout:/volumes/_nogroup/subvolume_0000000000344407/73bf87d0-141d-4372-a60a-04afeaf0fe71 2021-02-10T11:11:52.988 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:52.993 7f873b524700 1 -- 172.21.2.102:0/328464975 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f87140186c0 msgr2=0x7f871401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:11:52.988 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:52.993 7f873b524700 1 --2- 172.21.2.102:0/328464975 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f87140186c0 0x7f871401abc0 crc :-1 s=READY pgs=259 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:11:52.988 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:52.993 7f873b524700 1 -- 172.21.2.102:0/328464975 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f87340c1f80 msgr2=0x7f87340c2380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:11:52.988 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:52.993 7f873b524700 1 --2- 172.21.2.102:0/328464975 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f87340c1f80 0x7f87340c2380 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 rx=0x7f872c009960 tx=0x7f872c00bd40).stop 2021-02-10T11:11:52.989 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:52.993 7f873b524700 1 -- 172.21.2.102:0/328464975 shutdown_connections 2021-02-10T11:11:52.989 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:52.993 7f873b524700 1 --2- 172.21.2.102:0/328464975 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f87340c1f80 0x7f87340c2380 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:11:52.989 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:52.993 7f873b524700 1 --2- 172.21.2.102:0/328464975 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f87140186c0 0x7f871401abc0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:11:52.989 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:52.993 7f873b524700 1 --2- 172.21.2.102:0/328464975 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f87340b8410 0x7f87340b8810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:11:52.989 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:52.993 7f873b524700 1 --2- 172.21.2.102:0/328464975 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f873409b930 0x7f87340b7c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:11:52.989 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:52.993 7f873b524700 1 -- 172.21.2.102:0/328464975 >> 172.21.2.102:0/328464975 conn(0x7f8734094e10 msgr2=0x7f87340a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:11:52.989 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:52.993 7f873b524700 1 -- 172.21.2.102:0/328464975 shutdown_connections 2021-02-10T11:11:52.989 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:11:52.993 7f873b524700 1 -- 172.21.2.102:0/328464975 wait complete. 2021-02-10T11:11:52.999 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000344407 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000344407/73bf87d0-141d-4372-a60a-04afeaf0fe71 2021-02-10T11:11:52.999 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:11:52.999 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:11:52.999 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:11:53.375 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:11:53.375 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:11:53.375 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344407/73bf87d0-141d-4372-a60a-04afeaf0fe71/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:11:53.757 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:11:53.757 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:11:53.757 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0240909 s, 43.5 MB/s 2021-02-10T11:11:53.758 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:11:53.758 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:11:53.758 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:11:54.104 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:11:54.105 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:11:54.105 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344407/73bf87d0-141d-4372-a60a-04afeaf0fe71/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:11:54.493 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:11:54.493 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:11:54.493 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.022746 s, 46.1 MB/s 2021-02-10T11:11:54.494 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:11:54.494 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:11:54.494 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:11:54.831 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:11:54.832 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:11:54.832 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344407/73bf87d0-141d-4372-a60a-04afeaf0fe71/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:11:55.206 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:11:55.206 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:11:55.206 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0226227 s, 46.4 MB/s 2021-02-10T11:11:55.207 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:11:55.207 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:11:55.207 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:11:55.542 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:11:55.543 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:11:55.543 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344407/73bf87d0-141d-4372-a60a-04afeaf0fe71/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:11:55.919 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:11:55.920 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:11:55.920 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0237374 s, 44.2 MB/s 2021-02-10T11:11:55.920 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:11:55.921 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:11:55.921 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:11:56.257 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:11:56.257 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:11:56.257 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344407/73bf87d0-141d-4372-a60a-04afeaf0fe71/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:11:56.631 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:11:56.631 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:11:56.631 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0232901 s, 45.0 MB/s 2021-02-10T11:11:56.632 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:11:56.632 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:11:56.632 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:11:56.968 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:11:56.968 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:11:56.969 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344407/73bf87d0-141d-4372-a60a-04afeaf0fe71/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:11:57.343 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:11:57.343 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:11:57.343 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0243529 s, 43.1 MB/s 2021-02-10T11:11:57.344 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:11:57.344 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:11:57.345 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:11:57.680 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:11:57.681 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:11:57.681 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344407/73bf87d0-141d-4372-a60a-04afeaf0fe71/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:11:58.055 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:11:58.055 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:11:58.055 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0241764 s, 43.4 MB/s 2021-02-10T11:11:58.056 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:11:58.056 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:11:58.057 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:11:58.392 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:11:58.392 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:11:58.392 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344407/73bf87d0-141d-4372-a60a-04afeaf0fe71/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:11:58.764 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:11:58.764 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:11:58.765 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0245553 s, 42.7 MB/s 2021-02-10T11:11:58.765 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:11:58.766 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:11:58.766 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:11:59.100 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:11:59.101 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:11:59.101 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344407/73bf87d0-141d-4372-a60a-04afeaf0fe71/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:11:59.475 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:11:59.475 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:11:59.475 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0228561 s, 45.9 MB/s 2021-02-10T11:11:59.476 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:11:59.476 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:11:59.476 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:11:59.811 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:11:59.811 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:11:59.811 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344407/73bf87d0-141d-4372-a60a-04afeaf0fe71/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:12:00.186 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:12:00.186 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:12:00.187 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0237337 s, 44.2 MB/s 2021-02-10T11:12:00.188 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000344408 2021-02-10T11:12:00.621 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:00.607 7f0638965700 1 Processor -- start 2021-02-10T11:12:00.621 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:00.607 7f0638965700 1 -- start start 2021-02-10T11:12:00.621 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:00.607 7f0638965700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f06341b0470 0x7f06341b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:12:00.621 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:00.607 7f0638965700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f06340f20e0 0x7f06340f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:12:00.621 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:00.607 7f0638965700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06340f0b50 con 0x7f06340ef610 2021-02-10T11:12:00.621 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:00.607 7f0638965700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06340f0820 con 0x7f06341b0470 2021-02-10T11:12:00.622 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:00.607 7f0638965700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06340f24e0 con 0x7f06340f20e0 2021-02-10T11:12:00.622 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:00.608 7f0631e1b700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f06340ef610 0x7f06340efa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:44388/0 (socket says 172.21.2.102:44388) 2021-02-10T11:12:00.622 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:00.608 7f0631e1b700 1 -- 172.21.2.102:0/2454892562 learned_addr learned my addr 172.21.2.102:0/2454892562 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:12:00.622 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:00.608 7f063161a700 1 -- 172.21.2.102:0/2454892562 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3167687583 0 0) 0x7f061c0010b0 con 0x7f06340ef610 2021-02-10T11:12:00.622 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:00.608 7f063161a700 1 -- 172.21.2.102:0/2454892562 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0620002fe0 con 0x7f06340ef610 2021-02-10T11:12:00.622 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:00.608 7f063261c700 1 --2- 172.21.2.102:0/2454892562 >> v2:172.21.2.113:3300/0 conn(0x7f06341b0470 0x7f06341b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:12:00.622 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:00.608 7f0632e1d700 1 --2- 172.21.2.102:0/2454892562 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f06340f20e0 0x7f06340f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:12:00.622 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:00.608 7f063161a700 1 -- 172.21.2.102:0/2454892562 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3977363356 0 0) 0x7f061c001010 con 0x7f06340ef610 2021-02-10T11:12:00.622 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:00.608 7f063161a700 1 -- 172.21.2.102:0/2454892562 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f06340f20e0 msgr2=0x7f06340f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:12:00.622 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:00.608 7f063161a700 1 --2- 172.21.2.102:0/2454892562 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f06340f20e0 0x7f06340f0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:12:00.622 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:00.608 7f063161a700 1 -- 172.21.2.102:0/2454892562 >> v2:172.21.2.113:3300/0 conn(0x7f06341b0470 msgr2=0x7f06341b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:12:00.622 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:00.608 7f063161a700 1 --2- 172.21.2.102:0/2454892562 >> v2:172.21.2.113:3300/0 conn(0x7f06341b0470 0x7f06341b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:12:00.623 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:00.608 7f063161a700 1 -- 172.21.2.102:0/2454892562 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06340e52e0 con 0x7f06340ef610 2021-02-10T11:12:00.623 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:00.608 7f063161a700 1 -- 172.21.2.102:0/2454892562 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7f061c001530 con 0x7f06340ef610 2021-02-10T11:12:00.623 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:00.608 7f063161a700 1 -- 172.21.2.102:0/2454892562 >> v1:172.21.2.102:6789/0 conn(0x7f06340ef610 legacy=0x7f06340efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:12:00.623 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:00.608 7f063161a700 1 --2- 172.21.2.102:0/2454892562 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0620003ac0 0x7f0620005ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:12:00.623 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:00.608 7f063161a700 1 --2- 172.21.2.102:0/2454892562 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f062004a7a0 0x7f062004cc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:12:00.623 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:00.608 7f063161a700 1 --2- 172.21.2.102:0/2454892562 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f062004d510 0x7f062004fa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:12:00.623 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:00.608 7f063161a700 1 -- 172.21.2.102:0/2454892562 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06200502e0 con 0x7f0620003ac0 2021-02-10T11:12:00.623 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:00.608 7f063161a700 1 -- 172.21.2.102:0/2454892562 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0620050500 con 0x7f062004d510 2021-02-10T11:12:00.623 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:00.608 7f063161a700 1 -- 172.21.2.102:0/2454892562 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0620050720 con 0x7f062004a7a0 2021-02-10T11:12:00.623 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:00.608 7f063261c700 1 --2- 172.21.2.102:0/2454892562 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0620003ac0 0x7f0620005ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:12:00.623 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:00.609 7f0632e1d700 1 --2- 172.21.2.102:0/2454892562 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f062004d510 0x7f062004fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:12:00.623 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:00.609 7f063261c700 1 -- 172.21.2.102:0/2454892562 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f062004a7a0 msgr2=0x7f062004cc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:12:00.623 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:00.609 7f0631e1b700 1 --2- 172.21.2.102:0/2454892562 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f062004a7a0 0x7f062004cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:12:00.623 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:00.609 7f063261c700 1 --2- 172.21.2.102:0/2454892562 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f062004a7a0 0x7f062004cc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:12:00.624 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:00.609 7f063261c700 1 -- 172.21.2.102:0/2454892562 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f062004d510 msgr2=0x7f062004fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:12:00.624 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:00.609 7f063261c700 1 --2- 172.21.2.102:0/2454892562 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f062004d510 0x7f062004fa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:12:00.624 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:00.609 7f063261c700 1 -- 172.21.2.102:0/2454892562 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06200509c0 con 0x7f0620003ac0 2021-02-10T11:12:00.624 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:00.609 7f0632e1d700 1 --2- 172.21.2.102:0/2454892562 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f062004d510 0x7f062004fa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:12:00.624 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:00.609 7f0631e1b700 1 --2- 172.21.2.102:0/2454892562 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f062004a7a0 0x7f062004cc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:12:00.624 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:00.609 7f063261c700 1 --2- 172.21.2.102:0/2454892562 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0620003ac0 0x7f0620005ef0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 rx=0x7f062400c8f0 tx=0x7f062400cc90).ready entity=mon.0 client_cookie=aa0d7ab88db49e74 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:12:00.624 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:00.609 7f063161a700 1 -- 172.21.2.102:0/2454892562 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f062400cdd0 con 0x7f0620003ac0 2021-02-10T11:12:00.624 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:00.609 7f063161a700 1 -- 172.21.2.102:0/2454892562 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0624009310 con 0x7f0620003ac0 2021-02-10T11:12:00.624 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:00.609 7f063161a700 1 -- 172.21.2.102:0/2454892562 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0624009830 con 0x7f0620003ac0 2021-02-10T11:12:00.624 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:00.609 7f0638965700 1 -- 172.21.2.102:0/2454892562 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0620003ac0 msgr2=0x7f0620005ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:12:00.624 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:00.609 7f0638965700 1 --2- 172.21.2.102:0/2454892562 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0620003ac0 0x7f0620005ef0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 rx=0x7f062400c8f0 tx=0x7f062400cc90).stop 2021-02-10T11:12:00.624 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:00.609 7f0632e1d700 1 -- 172.21.2.102:0/2454892562 reap_dead start 2021-02-10T11:12:00.624 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:00.610 7f0638965700 1 -- 172.21.2.102:0/2454892562 shutdown_connections 2021-02-10T11:12:00.625 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:00.610 7f0638965700 1 -- 172.21.2.102:0/2454892562 >> 172.21.2.102:0/2454892562 conn(0x7f06340ce3e0 msgr2=0x7f06340e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:12:00.625 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:00.610 7f0638965700 1 -- 172.21.2.102:0/2454892562 shutdown_connections 2021-02-10T11:12:00.625 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:00.610 7f0638965700 1 -- 172.21.2.102:0/2454892562 wait complete. 2021-02-10T11:12:00.625 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:00.625 7f0638965700 1 Processor -- start 2021-02-10T11:12:00.625 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:00.625 7f0638965700 1 -- start start 2021-02-10T11:12:00.625 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:00.625 7f0638965700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f063409b930 0x7f06340b7c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:12:00.625 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:00.625 7f0638965700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f06340b8410 0x7f06340b8810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:12:00.625 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:00.625 7f0638965700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f06340c1f80 0x7f06340c2380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:12:00.625 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:00.625 7f0638965700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06340c1a00 con 0x7f06340b8410 2021-02-10T11:12:00.625 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:00.625 7f0638965700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06340b8e30 con 0x7f06340c1f80 2021-02-10T11:12:00.625 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:00.625 7f0638965700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06340b9050 con 0x7f063409b930 2021-02-10T11:12:00.625 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:00.625 7f0631e1b700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f06340b8410 0x7f06340b8810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:12:00.625 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:00.625 7f0631e1b700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f06340b8410 0x7f06340b8810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:52176/0 (socket says 172.21.2.102:52176) 2021-02-10T11:12:00.625 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:00.625 7f0631e1b700 1 -- 172.21.2.102:0/3620069188 learned_addr learned my addr 172.21.2.102:0/3620069188 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:12:00.627 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:00.626 7f0631e1b700 1 -- 172.21.2.102:0/3620069188 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f063409b930 msgr2=0x7f06340b7c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:12:00.627 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:00.626 7f0631e1b700 1 --2- 172.21.2.102:0/3620069188 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f063409b930 0x7f06340b7c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:12:00.627 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:00.626 7f0631e1b700 1 -- 172.21.2.102:0/3620069188 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f06340c1f80 msgr2=0x7f06340c2380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:12:00.627 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:00.626 7f0631e1b700 1 --2- 172.21.2.102:0/3620069188 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f06340c1f80 0x7f06340c2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:12:00.627 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:00.626 7f0631e1b700 1 -- 172.21.2.102:0/3620069188 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06340c29a0 con 0x7f06340b8410 2021-02-10T11:12:00.627 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:00.626 7f0631e1b700 1 --2- 172.21.2.102:0/3620069188 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f06340b8410 0x7f06340b8810 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 rx=0x7f061c001c60 tx=0x7f061c00a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:12:00.627 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:00.626 7f062affd700 1 -- 172.21.2.102:0/3620069188 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f061c00acc0 con 0x7f06340b8410 2021-02-10T11:12:00.627 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:00.626 7f062affd700 1 -- 172.21.2.102:0/3620069188 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f061c004350 con 0x7f06340b8410 2021-02-10T11:12:00.627 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:00.626 7f062affd700 1 -- 172.21.2.102:0/3620069188 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f061c004870 con 0x7f06340b8410 2021-02-10T11:12:00.628 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:00.626 7f0638965700 1 -- 172.21.2.102:0/3620069188 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06340c29a0 con 0x7f06340b8410 2021-02-10T11:12:00.628 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:00.626 7f0638965700 1 -- 172.21.2.102:0/3620069188 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06340c2f30 con 0x7f06340b8410 2021-02-10T11:12:00.628 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:00.627 7f062affd700 1 -- 172.21.2.102:0/3620069188 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f061c004530 con 0x7f06340b8410 2021-02-10T11:12:00.628 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:00.627 7f062affd700 1 --2- 172.21.2.102:0/3620069188 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f06180186c0 0x7f061801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:12:00.628 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:00.627 7f062affd700 1 -- 172.21.2.102:0/3620069188 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f061c032570 con 0x7f06340b8410 2021-02-10T11:12:00.628 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:00.627 7f063261c700 1 --2- 172.21.2.102:0/3620069188 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f06180186c0 0x7f061801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:12:00.628 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:00.627 7f063261c700 1 --2- 172.21.2.102:0/3620069188 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f06180186c0 0x7f061801abc0 crc :-1 s=READY pgs=260 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:12:00.653 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:00.657 7f0638965700 1 -- 172.21.2.102:0/3620069188 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0634105b30 con 0x7f06340b8410 2021-02-10T11:12:00.655 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:00.660 7f062affd700 1 -- 172.21.2.102:0/3620069188 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f061c015430 con 0x7f06340b8410 2021-02-10T11:12:00.785 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:00.789 7f0638965700 1 -- 172.21.2.102:0/3620069188 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344408", "target": ["mgr", ""]}) v1 -- 0x7f06340c2f30 con 0x7f06180186c0 2021-02-10T11:12:00.812 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:00.817 7f062affd700 1 -- 172.21.2.102:0/3620069188 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f06240009b0 con 0x7f06180186c0 2021-02-10T11:12:00.818 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:00.822 7f0638965700 1 -- 172.21.2.102:0/3620069188 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f06180186c0 msgr2=0x7f061801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:12:00.818 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:00.822 7f0638965700 1 --2- 172.21.2.102:0/3620069188 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f06180186c0 0x7f061801abc0 crc :-1 s=READY pgs=260 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:12:00.818 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:00.822 7f0638965700 1 -- 172.21.2.102:0/3620069188 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f06340b8410 msgr2=0x7f06340b8810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:12:00.818 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:00.822 7f0638965700 1 --2- 172.21.2.102:0/3620069188 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f06340b8410 0x7f06340b8810 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 rx=0x7f061c001c60 tx=0x7f061c00a8d0).stop 2021-02-10T11:12:00.818 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:00.822 7f0638965700 1 -- 172.21.2.102:0/3620069188 shutdown_connections 2021-02-10T11:12:00.818 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:00.822 7f0638965700 1 --2- 172.21.2.102:0/3620069188 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f06180186c0 0x7f061801abc0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:12:00.818 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:00.822 7f0638965700 1 --2- 172.21.2.102:0/3620069188 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f06340c1f80 0x7f06340c2380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:12:00.819 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:00.822 7f0638965700 1 --2- 172.21.2.102:0/3620069188 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f06340b8410 0x7f06340b8810 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:12:00.819 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:00.822 7f0638965700 1 --2- 172.21.2.102:0/3620069188 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f063409b930 0x7f06340b7c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:12:00.819 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:00.822 7f0638965700 1 -- 172.21.2.102:0/3620069188 >> 172.21.2.102:0/3620069188 conn(0x7f0634094e10 msgr2=0x7f06340a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:12:00.819 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:00.822 7f0638965700 1 -- 172.21.2.102:0/3620069188 shutdown_connections 2021-02-10T11:12:00.819 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:00.822 7f0638965700 1 -- 172.21.2.102:0/3620069188 wait complete. 2021-02-10T11:12:00.826 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000344408 2021-02-10T11:12:01.258 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:01.245 7fe8f1b3b700 1 Processor -- start 2021-02-10T11:12:01.258 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:01.245 7fe8f1b3b700 1 -- start start 2021-02-10T11:12:01.258 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:01.245 7fe8f1b3b700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe8ec1b0470 0x7fe8ec1b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:12:01.259 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:01.245 7fe8f1b3b700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8ec0f0b50 con 0x7fe8ec0ef610 2021-02-10T11:12:01.259 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:01.245 7fe8f1b3b700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8ec0f24e0 con 0x7fe8ec0f20e0 2021-02-10T11:12:01.259 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:01.245 7fe8f1b3b700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8ec0e5ea0 con 0x7fe8ec1b0470 2021-02-10T11:12:01.259 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:01.246 7fe8eaffd700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fe8ec0ef610 0x7fe8ec0efa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:44406/0 (socket says 172.21.2.102:44406) 2021-02-10T11:12:01.259 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:01.246 7fe8eaffd700 1 -- 172.21.2.102:0/3628191010 learned_addr learned my addr 172.21.2.102:0/3628191010 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:12:01.259 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:01.246 7fe8ea7fc700 1 -- 172.21.2.102:0/3628191010 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1013384955 0 0) 0x7fe8dc0010b0 con 0x7fe8ec0ef610 2021-02-10T11:12:01.259 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:01.246 7fe8ea7fc700 1 -- 172.21.2.102:0/3628191010 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8d8002fe0 con 0x7fe8ec0ef610 2021-02-10T11:12:01.259 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:01.246 7fe8eb7fe700 1 --2- 172.21.2.102:0/3628191010 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe8ec1b0470 0x7fe8ec1b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:12:01.259 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:01.246 7fe8ea7fc700 1 -- 172.21.2.102:0/3628191010 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1413025436 0 0) 0x7fe8dc0010b0 con 0x7fe8ec0ef610 2021-02-10T11:12:01.259 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:01.246 7fe8ea7fc700 1 -- 172.21.2.102:0/3628191010 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe8ec1b0470 msgr2=0x7fe8ec1b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:12:01.260 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:01.246 7fe8ea7fc700 1 --2- 172.21.2.102:0/3628191010 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe8ec1b0470 0x7fe8ec1b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:12:01.260 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:01.246 7fe8ea7fc700 1 -- 172.21.2.102:0/3628191010 >> v1:172.21.2.113:6789/0 conn(0x7fe8ec0f20e0 legacy=0x7fe8ec0f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:12:01.260 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:01.246 7fe8ea7fc700 1 -- 172.21.2.102:0/3628191010 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8ec0e52e0 con 0x7fe8ec0ef610 2021-02-10T11:12:01.260 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:01.246 7fe8ea7fc700 1 -- 172.21.2.102:0/3628191010 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7fe8dc0015b0 con 0x7fe8ec0ef610 2021-02-10T11:12:01.260 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:01.246 7fe8ea7fc700 1 -- 172.21.2.102:0/3628191010 >> v1:172.21.2.102:6789/0 conn(0x7fe8ec0ef610 legacy=0x7fe8ec0efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:12:01.260 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:01.247 7fe8ea7fc700 1 --2- 172.21.2.102:0/3628191010 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe8d8003aa0 0x7fe8d8005f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:12:01.260 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:01.247 7fe8ea7fc700 1 --2- 172.21.2.102:0/3628191010 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe8d8006750 0x7fe8d8008c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:12:01.260 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:01.247 7fe8ea7fc700 1 --2- 172.21.2.102:0/3628191010 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe8d80094f0 0x7fe8d800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:12:01.260 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:01.247 7fe8ea7fc700 1 -- 172.21.2.102:0/3628191010 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8d800c2c0 con 0x7fe8d8006750 2021-02-10T11:12:01.260 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:01.247 7fe8ea7fc700 1 -- 172.21.2.102:0/3628191010 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8d800c4e0 con 0x7fe8d80094f0 2021-02-10T11:12:01.260 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:01.247 7fe8ea7fc700 1 -- 172.21.2.102:0/3628191010 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8d800c700 con 0x7fe8d8003aa0 2021-02-10T11:12:01.260 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:01.247 7fe8eb7fe700 1 --2- 172.21.2.102:0/3628191010 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe8d8003aa0 0x7fe8d8005f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:12:01.260 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:01.247 7fe8eaffd700 1 --2- 172.21.2.102:0/3628191010 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe8d8006750 0x7fe8d8008c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:12:01.261 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:01.247 7fe8ebfff700 1 --2- 172.21.2.102:0/3628191010 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe8d80094f0 0x7fe8d800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:12:01.261 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:01.247 7fe8eaffd700 1 -- 172.21.2.102:0/3628191010 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe8d8003aa0 msgr2=0x7fe8d8005f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:12:01.261 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:01.247 7fe8eaffd700 1 --2- 172.21.2.102:0/3628191010 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe8d8003aa0 0x7fe8d8005f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:12:01.261 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:01.247 7fe8eaffd700 1 -- 172.21.2.102:0/3628191010 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe8d80094f0 msgr2=0x7fe8d800ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:12:01.261 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:01.247 7fe8eaffd700 1 --2- 172.21.2.102:0/3628191010 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe8d80094f0 0x7fe8d800ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:12:01.261 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:01.247 7fe8eaffd700 1 -- 172.21.2.102:0/3628191010 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8d8050970 con 0x7fe8d8006750 2021-02-10T11:12:01.261 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:01.247 7fe8eaffd700 1 --2- 172.21.2.102:0/3628191010 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe8d8006750 0x7fe8d8008c70 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 rx=0x7fe8dc001370 tx=0x7fe8dc007d30).ready entity=mon.0 client_cookie=dbe1535bdccd9ea0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:12:01.261 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:01.247 7fe8ea7fc700 1 -- 172.21.2.102:0/3628191010 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe8dc0010b0 con 0x7fe8d8006750 2021-02-10T11:12:01.261 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:01.247 7fe8ea7fc700 1 -- 172.21.2.102:0/3628191010 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8dc00a520 con 0x7fe8d8006750 2021-02-10T11:12:01.261 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:01.247 7fe8ea7fc700 1 -- 172.21.2.102:0/3628191010 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe8dc00a970 con 0x7fe8d8006750 2021-02-10T11:12:01.261 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:01.247 7fe8f1b3b700 1 -- 172.21.2.102:0/3628191010 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe8d8006750 msgr2=0x7fe8d8008c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:12:01.261 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:01.247 7fe8f1b3b700 1 --2- 172.21.2.102:0/3628191010 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe8d8006750 0x7fe8d8008c70 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 rx=0x7fe8dc001370 tx=0x7fe8dc007d30).stop 2021-02-10T11:12:01.261 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:01.247 7fe8ebfff700 1 -- 172.21.2.102:0/3628191010 reap_dead start 2021-02-10T11:12:01.262 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:01.248 7fe8f1b3b700 1 -- 172.21.2.102:0/3628191010 shutdown_connections 2021-02-10T11:12:01.262 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:01.248 7fe8f1b3b700 1 -- 172.21.2.102:0/3628191010 >> 172.21.2.102:0/3628191010 conn(0x7fe8ec0ce3e0 msgr2=0x7fe8ec0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:12:01.262 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:01.248 7fe8f1b3b700 1 -- 172.21.2.102:0/3628191010 shutdown_connections 2021-02-10T11:12:01.262 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:01.248 7fe8f1b3b700 1 -- 172.21.2.102:0/3628191010 wait complete. 2021-02-10T11:12:01.262 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:01.263 7fe8f1b3b700 1 Processor -- start 2021-02-10T11:12:01.262 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:01.263 7fe8f1b3b700 1 -- start start 2021-02-10T11:12:01.263 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:01.263 7fe8f1b3b700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe8ec09b930 0x7fe8ec0b7c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:12:01.263 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:01.263 7fe8f1b3b700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe8ec0b8410 0x7fe8ec0b8810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:12:01.263 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:01.263 7fe8f1b3b700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe8ec0c1f80 0x7fe8ec0c2380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:12:01.263 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:01.263 7fe8f1b3b700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8ec0c1a00 con 0x7fe8ec0c1f80 2021-02-10T11:12:01.264 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:01.263 7fe8f1b3b700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8ec0b8e30 con 0x7fe8ec0b8410 2021-02-10T11:12:01.264 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:01.263 7fe8f1b3b700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8ec0b9050 con 0x7fe8ec09b930 2021-02-10T11:12:01.264 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:01.263 7fe8ebfff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe8ec0c1f80 0x7fe8ec0c2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:12:01.264 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:01.263 7fe8ebfff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe8ec0c1f80 0x7fe8ec0c2380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:52196/0 (socket says 172.21.2.102:52196) 2021-02-10T11:12:01.264 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:01.263 7fe8ebfff700 1 -- 172.21.2.102:0/2209565599 learned_addr learned my addr 172.21.2.102:0/2209565599 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:12:01.264 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:01.264 7fe8ebfff700 1 -- 172.21.2.102:0/2209565599 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe8ec09b930 msgr2=0x7fe8ec0b7c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:12:01.264 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:01.264 7fe8ebfff700 1 --2- 172.21.2.102:0/2209565599 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe8ec09b930 0x7fe8ec0b7c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:12:01.264 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:01.264 7fe8ebfff700 1 -- 172.21.2.102:0/2209565599 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe8ec0b8410 msgr2=0x7fe8ec0b8810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:12:01.264 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:01.264 7fe8ebfff700 1 --2- 172.21.2.102:0/2209565599 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe8ec0b8410 0x7fe8ec0b8810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:12:01.264 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:01.264 7fe8ebfff700 1 -- 172.21.2.102:0/2209565599 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8ec0c29a0 con 0x7fe8ec0c1f80 2021-02-10T11:12:01.264 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:01.264 7fe8ebfff700 1 --2- 172.21.2.102:0/2209565599 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe8ec0c1f80 0x7fe8ec0c2380 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 rx=0x7fe8e4006900 tx=0x7fe8e400bd20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:12:01.264 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:01.264 7fe8e3fff700 1 -- 172.21.2.102:0/2209565599 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe8e4006240 con 0x7fe8ec0c1f80 2021-02-10T11:12:01.265 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:01.264 7fe8e3fff700 1 -- 172.21.2.102:0/2209565599 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8e400be60 con 0x7fe8ec0c1f80 2021-02-10T11:12:01.265 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:01.264 7fe8f1b3b700 1 -- 172.21.2.102:0/2209565599 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8ec0c29a0 con 0x7fe8ec0c1f80 2021-02-10T11:12:01.265 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:01.264 7fe8e3fff700 1 -- 172.21.2.102:0/2209565599 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe8e40064c0 con 0x7fe8ec0c1f80 2021-02-10T11:12:01.265 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:01.264 7fe8f1b3b700 1 -- 172.21.2.102:0/2209565599 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8ec0c29a0 con 0x7fe8ec0c1f80 2021-02-10T11:12:01.265 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:01.265 7fe8e3fff700 1 -- 172.21.2.102:0/2209565599 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fe8e4006a30 con 0x7fe8ec0c1f80 2021-02-10T11:12:01.265 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:01.265 7fe8e3fff700 1 --2- 172.21.2.102:0/2209565599 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe8d0018730 0x7fe8d001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:12:01.265 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:01.265 7fe8e3fff700 1 -- 172.21.2.102:0/2209565599 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fe8e40329c0 con 0x7fe8ec0c1f80 2021-02-10T11:12:01.265 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:01.265 7fe8eb7fe700 1 --2- 172.21.2.102:0/2209565599 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe8d0018730 0x7fe8d001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:12:01.265 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:01.265 7fe8eb7fe700 1 --2- 172.21.2.102:0/2209565599 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe8d0018730 0x7fe8d001abc0 crc :-1 s=READY pgs=261 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:12:01.291 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:01.295 7fe8f1b3b700 1 -- 172.21.2.102:0/2209565599 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8ec105b30 con 0x7fe8ec0c1f80 2021-02-10T11:12:01.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:01.298 7fe8e3fff700 1 -- 172.21.2.102:0/2209565599 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fe8e4006030 con 0x7fe8ec0c1f80 2021-02-10T11:12:01.422 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:01.426 7fe8f1b3b700 1 -- 172.21.2.102:0/2209565599 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344408", "target": ["mgr", ""]}) v1 -- 0x7fe8ec105b30 con 0x7fe8d0018730 2021-02-10T11:12:01.424 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:01.429 7fe8e3fff700 1 -- 172.21.2.102:0/2209565599 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7fe8d4006e70 con 0x7fe8d0018730 2021-02-10T11:12:01.425 INFO:teuthology.orchestra.run.gibba002.stdout:/volumes/_nogroup/subvolume_0000000000344408/fa3095e3-83e2-4a85-baeb-0e1d795ad80d 2021-02-10T11:12:01.426 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:01.430 7fe8f1b3b700 1 -- 172.21.2.102:0/2209565599 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe8d0018730 msgr2=0x7fe8d001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:12:01.426 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:01.430 7fe8f1b3b700 1 --2- 172.21.2.102:0/2209565599 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe8d0018730 0x7fe8d001abc0 crc :-1 s=READY pgs=261 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:12:01.426 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:01.430 7fe8f1b3b700 1 -- 172.21.2.102:0/2209565599 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe8ec0c1f80 msgr2=0x7fe8ec0c2380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:12:01.427 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:01.430 7fe8f1b3b700 1 --2- 172.21.2.102:0/2209565599 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe8ec0c1f80 0x7fe8ec0c2380 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 rx=0x7fe8e4006900 tx=0x7fe8e400bd20).stop 2021-02-10T11:12:01.427 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:01.431 7fe8f1b3b700 1 -- 172.21.2.102:0/2209565599 shutdown_connections 2021-02-10T11:12:01.427 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:01.431 7fe8f1b3b700 1 --2- 172.21.2.102:0/2209565599 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe8ec0c1f80 0x7fe8ec0c2380 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:12:01.427 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:01.431 7fe8f1b3b700 1 --2- 172.21.2.102:0/2209565599 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe8d0018730 0x7fe8d001abc0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:12:01.427 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:01.431 7fe8f1b3b700 1 --2- 172.21.2.102:0/2209565599 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe8ec0b8410 0x7fe8ec0b8810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:12:01.427 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:01.431 7fe8f1b3b700 1 --2- 172.21.2.102:0/2209565599 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe8ec09b930 0x7fe8ec0b7c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:12:01.427 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:01.431 7fe8f1b3b700 1 -- 172.21.2.102:0/2209565599 >> 172.21.2.102:0/2209565599 conn(0x7fe8ec094e10 msgr2=0x7fe8ec0a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:12:01.427 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:01.431 7fe8f1b3b700 1 -- 172.21.2.102:0/2209565599 shutdown_connections 2021-02-10T11:12:01.427 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:01.431 7fe8f1b3b700 1 -- 172.21.2.102:0/2209565599 wait complete. 2021-02-10T11:12:01.434 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000344408 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000344408/fa3095e3-83e2-4a85-baeb-0e1d795ad80d 2021-02-10T11:12:01.434 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:12:01.434 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:12:01.434 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:12:01.771 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:12:01.772 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:12:01.772 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344408/fa3095e3-83e2-4a85-baeb-0e1d795ad80d/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:12:02.154 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:12:02.154 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:12:02.154 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0249915 s, 42.0 MB/s 2021-02-10T11:12:02.156 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:12:02.156 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:12:02.156 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:12:02.493 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:12:02.493 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:12:02.493 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344408/fa3095e3-83e2-4a85-baeb-0e1d795ad80d/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:12:02.870 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:12:02.870 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:12:02.870 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0247446 s, 42.4 MB/s 2021-02-10T11:12:02.871 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:12:02.871 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:12:02.871 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:12:03.206 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:12:03.206 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:12:03.206 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344408/fa3095e3-83e2-4a85-baeb-0e1d795ad80d/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:12:03.578 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:12:03.578 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:12:03.578 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0234765 s, 44.7 MB/s 2021-02-10T11:12:03.579 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:12:03.579 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:12:03.580 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:12:03.915 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:12:03.915 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:12:03.915 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344408/fa3095e3-83e2-4a85-baeb-0e1d795ad80d/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:12:04.289 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:12:04.289 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:12:04.289 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0231808 s, 45.2 MB/s 2021-02-10T11:12:04.290 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:12:04.290 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:12:04.290 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:12:04.626 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:12:04.626 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:12:04.627 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344408/fa3095e3-83e2-4a85-baeb-0e1d795ad80d/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:12:05.004 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:12:05.004 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:12:05.004 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0240058 s, 43.7 MB/s 2021-02-10T11:12:05.005 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:12:05.005 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:12:05.005 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:12:05.340 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:12:05.341 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:12:05.341 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344408/fa3095e3-83e2-4a85-baeb-0e1d795ad80d/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:12:05.716 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:12:05.717 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:12:05.717 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0246174 s, 42.6 MB/s 2021-02-10T11:12:05.718 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:12:05.718 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:12:05.718 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:12:06.053 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:12:06.054 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:12:06.054 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344408/fa3095e3-83e2-4a85-baeb-0e1d795ad80d/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:12:06.429 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:12:06.430 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:12:06.430 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0245885 s, 42.6 MB/s 2021-02-10T11:12:06.431 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:12:06.431 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:12:06.431 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:12:06.767 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:12:06.768 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:12:06.768 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344408/fa3095e3-83e2-4a85-baeb-0e1d795ad80d/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:12:07.143 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:12:07.143 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:12:07.143 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0243126 s, 43.1 MB/s 2021-02-10T11:12:07.144 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:12:07.144 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:12:07.144 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:12:07.480 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:12:07.480 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:12:07.481 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344408/fa3095e3-83e2-4a85-baeb-0e1d795ad80d/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:12:07.856 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:12:07.856 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:12:07.856 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0247489 s, 42.4 MB/s 2021-02-10T11:12:07.857 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:12:07.857 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:12:07.858 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:12:08.193 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:12:08.193 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:12:08.194 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344408/fa3095e3-83e2-4a85-baeb-0e1d795ad80d/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:12:08.564 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:12:10.682 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:12:10.682 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0229111 s, 45.8 MB/s 2021-02-10T11:12:10.683 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000344409 2021-02-10T11:12:11.116 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:11.103 7f632cbcc700 1 Processor -- start 2021-02-10T11:12:11.116 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:11.103 7f632cbcc700 1 -- start start 2021-02-10T11:12:11.116 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:11.104 7f632cbcc700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f63280f05f0 0x7f63280f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:12:11.117 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:11.104 7f632cbcc700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63281b0c70 con 0x7f63281b0870 2021-02-10T11:12:11.117 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:11.104 7f632cbcc700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63280f0b50 con 0x7f63281b0470 2021-02-10T11:12:11.117 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:11.104 7f632cbcc700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63280e5ea0 con 0x7f63280f05f0 2021-02-10T11:12:11.117 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:11.104 7f632605d700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f63281b0870 0x7f63280efe00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:44424/0 (socket says 172.21.2.102:44424) 2021-02-10T11:12:11.117 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:11.104 7f632605d700 1 -- 172.21.2.102:0/4145093792 learned_addr learned my addr 172.21.2.102:0/4145093792 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:12:11.117 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:11.104 7f632585c700 1 -- 172.21.2.102:0/4145093792 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3962758880 0 0) 0x7f63100009b0 con 0x7f63281b0870 2021-02-10T11:12:11.117 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:11.105 7f632585c700 1 -- 172.21.2.102:0/4145093792 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6314002fe0 con 0x7f63281b0870 2021-02-10T11:12:11.117 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:11.105 7f632705f700 1 --2- 172.21.2.102:0/4145093792 >> v2:172.21.2.113:3300/0 conn(0x7f63280f05f0 0x7f63280f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:12:11.118 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:11.105 7f632585c700 1 -- 172.21.2.102:0/4145093792 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3814791880 0 0) 0x7f6310001010 con 0x7f63281b0870 2021-02-10T11:12:11.118 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:11.105 7f632585c700 1 -- 172.21.2.102:0/4145093792 >> v2:172.21.2.113:3300/0 conn(0x7f63280f05f0 msgr2=0x7f63280f20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:12:11.118 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:11.105 7f632585c700 1 --2- 172.21.2.102:0/4145093792 >> v2:172.21.2.113:3300/0 conn(0x7f63280f05f0 0x7f63280f20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:12:11.118 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:11.105 7f632585c700 1 -- 172.21.2.102:0/4145093792 >> v1:172.21.2.113:6789/0 conn(0x7f63281b0470 legacy=0x7f63280ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:12:11.118 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:11.105 7f632585c700 1 -- 172.21.2.102:0/4145093792 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63280e52e0 con 0x7f63281b0870 2021-02-10T11:12:11.118 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:11.105 7f632585c700 1 -- 172.21.2.102:0/4145093792 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7f6310001590 con 0x7f63281b0870 2021-02-10T11:12:11.118 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:11.105 7f632585c700 1 -- 172.21.2.102:0/4145093792 >> v1:172.21.2.102:6789/0 conn(0x7f63281b0870 legacy=0x7f63280efe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:12:11.118 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:11.105 7f632585c700 1 --2- 172.21.2.102:0/4145093792 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6314003b10 0x7f6314006010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:12:11.118 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:11.105 7f632585c700 1 --2- 172.21.2.102:0/4145093792 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6314006830 0x7f6314008d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:12:11.118 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:11.105 7f632585c700 1 --2- 172.21.2.102:0/4145093792 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6314009600 0x7f631400bae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:12:11.118 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:11.105 7f632585c700 1 -- 172.21.2.102:0/4145093792 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f631400c310 con 0x7f6314006830 2021-02-10T11:12:11.118 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:11.105 7f632585c700 1 -- 172.21.2.102:0/4145093792 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f631400c530 con 0x7f6314003b10 2021-02-10T11:12:11.119 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:11.105 7f632585c700 1 -- 172.21.2.102:0/4145093792 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f631400c750 con 0x7f6314009600 2021-02-10T11:12:11.119 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:11.105 7f632605d700 1 --2- 172.21.2.102:0/4145093792 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6314006830 0x7f6314008d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:12:11.119 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:11.105 7f632685e700 1 --2- 172.21.2.102:0/4145093792 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6314003b10 0x7f6314006010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:12:11.119 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:11.105 7f632705f700 1 --2- 172.21.2.102:0/4145093792 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6314009600 0x7f631400bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:12:11.119 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:11.105 7f632605d700 1 -- 172.21.2.102:0/4145093792 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6314009600 msgr2=0x7f631400bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:12:11.119 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:11.105 7f632605d700 1 --2- 172.21.2.102:0/4145093792 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6314009600 0x7f631400bae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:12:11.119 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:11.105 7f632605d700 1 -- 172.21.2.102:0/4145093792 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6314003b10 msgr2=0x7f6314006010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:12:11.119 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:11.105 7f632605d700 1 --2- 172.21.2.102:0/4145093792 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6314003b10 0x7f6314006010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:12:11.119 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:11.105 7f632605d700 1 -- 172.21.2.102:0/4145093792 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6314050990 con 0x7f6314006830 2021-02-10T11:12:11.119 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:11.105 7f632705f700 1 -- 172.21.2.102:0/4145093792 reap_dead start 2021-02-10T11:12:11.119 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:11.105 7f632605d700 1 --2- 172.21.2.102:0/4145093792 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6314006830 0x7f6314008d50 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 rx=0x7f6310007560 tx=0x7f6310007c90).ready entity=mon.0 client_cookie=ec82030ee5295412 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:12:11.119 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:11.106 7f632585c700 1 -- 172.21.2.102:0/4145093792 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f63100009b0 con 0x7f6314006830 2021-02-10T11:12:11.119 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:11.106 7f632585c700 1 -- 172.21.2.102:0/4145093792 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63100013d0 con 0x7f6314006830 2021-02-10T11:12:11.119 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:11.106 7f632585c700 1 -- 172.21.2.102:0/4145093792 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f631000a710 con 0x7f6314006830 2021-02-10T11:12:11.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:11.106 7f632cbcc700 1 -- 172.21.2.102:0/4145093792 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6314006830 msgr2=0x7f6314008d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:12:11.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:11.106 7f632cbcc700 1 --2- 172.21.2.102:0/4145093792 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6314006830 0x7f6314008d50 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 rx=0x7f6310007560 tx=0x7f6310007c90).stop 2021-02-10T11:12:11.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:11.106 7f632cbcc700 1 -- 172.21.2.102:0/4145093792 shutdown_connections 2021-02-10T11:12:11.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:11.106 7f632cbcc700 1 --2- 172.21.2.102:0/4145093792 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6314006830 0x7f6314008d50 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:12:11.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:11.106 7f632cbcc700 1 -- 172.21.2.102:0/4145093792 >> 172.21.2.102:0/4145093792 conn(0x7f63280ce3e0 msgr2=0x7f63280e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:12:11.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:11.106 7f632cbcc700 1 -- 172.21.2.102:0/4145093792 shutdown_connections 2021-02-10T11:12:11.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:11.106 7f632cbcc700 1 -- 172.21.2.102:0/4145093792 wait complete. 2021-02-10T11:12:11.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:11.121 7f632cbcc700 1 Processor -- start 2021-02-10T11:12:11.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:11.121 7f632cbcc700 1 -- start start 2021-02-10T11:12:11.122 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:11.121 7f632cbcc700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f63280a0910 0x7f6328105d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:12:11.122 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:11.121 7f632cbcc700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f63281a7a10 0x7f63281a7e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:12:11.122 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:11.121 7f632cbcc700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6328093770 0x7f6328093b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:12:11.122 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:11.121 7f632cbcc700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6328106390 con 0x7f63281a7a10 2021-02-10T11:12:11.122 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:11.121 7f632cbcc700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6328094190 con 0x7f63280a0910 2021-02-10T11:12:11.122 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:11.121 7f632cbcc700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63280943b0 con 0x7f6328093770 2021-02-10T11:12:11.122 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:11.122 7f632605d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f63281a7a10 0x7f63281a7e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:12:11.122 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:11.122 7f632605d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f63281a7a10 0x7f63281a7e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:52210/0 (socket says 172.21.2.102:52210) 2021-02-10T11:12:11.122 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:11.122 7f632605d700 1 -- 172.21.2.102:0/4173752127 learned_addr learned my addr 172.21.2.102:0/4173752127 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:12:11.122 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:11.122 7f632605d700 1 -- 172.21.2.102:0/4173752127 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6328093770 msgr2=0x7f6328093b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:12:11.123 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:11.122 7f632605d700 1 --2- 172.21.2.102:0/4173752127 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6328093770 0x7f6328093b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:12:11.123 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:11.122 7f632605d700 1 -- 172.21.2.102:0/4173752127 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f63280a0910 msgr2=0x7f6328105d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:12:11.123 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:11.122 7f632685e700 1 --2- 172.21.2.102:0/4173752127 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f63280a0910 0x7f6328105d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:12:11.123 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:11.122 7f632605d700 1 --2- 172.21.2.102:0/4173752127 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f63280a0910 0x7f6328105d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:12:11.123 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:11.122 7f632605d700 1 -- 172.21.2.102:0/4173752127 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6328106c30 con 0x7f63281a7a10 2021-02-10T11:12:11.123 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:11.122 7f632685e700 1 --2- 172.21.2.102:0/4173752127 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f63280a0910 0x7f6328105d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:12:11.123 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:11.122 7f632605d700 1 --2- 172.21.2.102:0/4173752127 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f63281a7a10 0x7f63281a7e10 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 rx=0x7f631000a4f0 tx=0x7f631000a890).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:12:11.123 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:11.122 7f631f7fe700 1 -- 172.21.2.102:0/4173752127 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6310000d90 con 0x7f63281a7a10 2021-02-10T11:12:11.123 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:11.122 7f631f7fe700 1 -- 172.21.2.102:0/4173752127 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f631000ac80 con 0x7f63281a7a10 2021-02-10T11:12:11.123 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:11.122 7f631f7fe700 1 -- 172.21.2.102:0/4173752127 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f63100045d0 con 0x7f63281a7a10 2021-02-10T11:12:11.123 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:11.123 7f632cbcc700 1 -- 172.21.2.102:0/4173752127 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6328106c30 con 0x7f63281a7a10 2021-02-10T11:12:11.123 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:11.123 7f632cbcc700 1 -- 172.21.2.102:0/4173752127 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6328106c30 con 0x7f63281a7a10 2021-02-10T11:12:11.123 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:11.123 7f631f7fe700 1 -- 172.21.2.102:0/4173752127 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f6310004140 con 0x7f63281a7a10 2021-02-10T11:12:11.124 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:11.123 7f631f7fe700 1 --2- 172.21.2.102:0/4173752127 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f630c0186c0 0x7f630c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:12:11.124 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:11.123 7f631f7fe700 1 -- 172.21.2.102:0/4173752127 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f63100323f0 con 0x7f63281a7a10 2021-02-10T11:12:11.124 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:11.123 7f632685e700 1 --2- 172.21.2.102:0/4173752127 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f630c0186c0 0x7f630c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:12:11.124 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:11.124 7f632685e700 1 --2- 172.21.2.102:0/4173752127 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f630c0186c0 0x7f630c01abc0 crc :-1 s=READY pgs=262 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:12:11.149 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:11.154 7f632cbcc700 1 -- 172.21.2.102:0/4173752127 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6328105b40 con 0x7f63281a7a10 2021-02-10T11:12:11.152 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:11.157 7f631f7fe700 1 -- 172.21.2.102:0/4173752127 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f6310000d90 con 0x7f63281a7a10 2021-02-10T11:12:11.278 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:11.282 7f632cbcc700 1 -- 172.21.2.102:0/4173752127 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344409", "target": ["mgr", ""]}) v1 -- 0x7f63280f0b50 con 0x7f630c0186c0 2021-02-10T11:12:11.304 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:11.308 7f631f7fe700 1 -- 172.21.2.102:0/4173752127 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f63180065f0 con 0x7f630c0186c0 2021-02-10T11:12:11.311 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:11.315 7f632cbcc700 1 -- 172.21.2.102:0/4173752127 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f630c0186c0 msgr2=0x7f630c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:12:11.311 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:11.315 7f632cbcc700 1 --2- 172.21.2.102:0/4173752127 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f630c0186c0 0x7f630c01abc0 crc :-1 s=READY pgs=262 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:12:11.311 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:11.315 7f632cbcc700 1 -- 172.21.2.102:0/4173752127 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f63281a7a10 msgr2=0x7f63281a7e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:12:11.311 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:11.315 7f632cbcc700 1 --2- 172.21.2.102:0/4173752127 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f63281a7a10 0x7f63281a7e10 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 rx=0x7f631000a4f0 tx=0x7f631000a890).stop 2021-02-10T11:12:11.312 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:11.315 7f632cbcc700 1 -- 172.21.2.102:0/4173752127 shutdown_connections 2021-02-10T11:12:11.312 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:11.315 7f632cbcc700 1 --2- 172.21.2.102:0/4173752127 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6328093770 0x7f6328093b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:12:11.312 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:11.315 7f632cbcc700 1 --2- 172.21.2.102:0/4173752127 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f63281a7a10 0x7f63281a7e10 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:12:11.312 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:11.315 7f632cbcc700 1 --2- 172.21.2.102:0/4173752127 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f630c0186c0 0x7f630c01abc0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:12:11.312 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:11.315 7f632cbcc700 1 --2- 172.21.2.102:0/4173752127 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f63280a0910 0x7f6328105d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:12:11.312 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:11.315 7f632cbcc700 1 -- 172.21.2.102:0/4173752127 >> 172.21.2.102:0/4173752127 conn(0x7f6328095ef0 msgr2=0x7f6328094e40 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:12:11.312 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:11.316 7f632cbcc700 1 -- 172.21.2.102:0/4173752127 shutdown_connections 2021-02-10T11:12:11.312 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:11.316 7f632cbcc700 1 -- 172.21.2.102:0/4173752127 wait complete. 2021-02-10T11:12:11.319 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000344409 2021-02-10T11:12:11.750 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:11.738 7f6c367c9700 1 Processor -- start 2021-02-10T11:12:11.750 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:11.738 7f6c367c9700 1 -- start start 2021-02-10T11:12:11.750 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:11.738 7f6c367c9700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f6c300f05f0 0x7f6c300f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:12:11.750 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:11.738 7f6c367c9700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c301b0c70 con 0x7f6c301b0470 2021-02-10T11:12:11.750 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:11.738 7f6c367c9700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c300f0b50 con 0x7f6c301b0870 2021-02-10T11:12:11.750 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:11.738 7f6c367c9700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c300e5ea0 con 0x7f6c300f05f0 2021-02-10T11:12:11.751 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:11.739 7f6c34de6700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f6c300f05f0 0x7f6c300f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:12:11.751 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:11.739 7f6c2ffff700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f6c301b0470 0x7f6c300ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:44442/0 (socket says 172.21.2.102:44442) 2021-02-10T11:12:11.751 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:11.739 7f6c2ffff700 1 -- 172.21.2.102:0/3025950071 learned_addr learned my addr 172.21.2.102:0/3025950071 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:12:11.751 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:11.739 7f6c34de6700 1 -- 172.21.2.102:0/3025950071 >> v1:172.21.2.113:6789/0 conn(0x7f6c301b0870 legacy=0x7f6c300efe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:12:11.751 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:11.739 7f6c34de6700 1 -- 172.21.2.102:0/3025950071 >> v1:172.21.2.102:6789/0 conn(0x7f6c301b0470 legacy=0x7f6c300ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:12:11.751 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:11.739 7f6c2effd700 1 -- 172.21.2.102:0/3025950071 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2461551321 0 0) 0x7f6c20001180 con 0x7f6c301b0470 2021-02-10T11:12:11.751 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:11.739 7f6c34de6700 1 -- 172.21.2.102:0/3025950071 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c300e52e0 con 0x7f6c300f05f0 2021-02-10T11:12:11.751 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:11.739 7f6c34de6700 1 --2- 172.21.2.102:0/3025950071 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6c300f05f0 0x7f6c300f20e0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 rx=0x7f6c28006e30 tx=0x7f6c28009de0).ready entity=mon.0 client_cookie=6d961c46fdde3a2d server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:12:11.751 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:11.740 7f6c2effd700 1 -- 172.21.2.102:0/3025950071 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6c2800b300 con 0x7f6c300f05f0 2021-02-10T11:12:11.751 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:11.740 7f6c2effd700 1 -- 172.21.2.102:0/3025950071 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c2800b680 con 0x7f6c300f05f0 2021-02-10T11:12:11.751 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:11.740 7f6c2effd700 1 -- 172.21.2.102:0/3025950071 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6c2800bba0 con 0x7f6c300f05f0 2021-02-10T11:12:11.751 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:11.740 7f6c367c9700 1 -- 172.21.2.102:0/3025950071 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6c300f05f0 msgr2=0x7f6c300f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:12:11.751 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:11.740 7f6c367c9700 1 --2- 172.21.2.102:0/3025950071 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6c300f05f0 0x7f6c300f20e0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 rx=0x7f6c28006e30 tx=0x7f6c28009de0).stop 2021-02-10T11:12:11.752 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:11.740 7f6c367c9700 1 -- 172.21.2.102:0/3025950071 shutdown_connections 2021-02-10T11:12:11.752 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:11.740 7f6c367c9700 1 --2- 172.21.2.102:0/3025950071 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6c300f05f0 0x7f6c300f20e0 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:12:11.752 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:11.740 7f6c367c9700 1 -- 172.21.2.102:0/3025950071 >> 172.21.2.102:0/3025950071 conn(0x7f6c300ce3e0 msgr2=0x7f6c300e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:12:11.752 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:11.740 7f6c367c9700 1 -- 172.21.2.102:0/3025950071 shutdown_connections 2021-02-10T11:12:11.752 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:11.740 7f6c367c9700 1 -- 172.21.2.102:0/3025950071 wait complete. 2021-02-10T11:12:11.753 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:11.755 7f6c367c9700 1 Processor -- start 2021-02-10T11:12:11.753 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:11.755 7f6c367c9700 1 -- start start 2021-02-10T11:12:11.753 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:11.756 7f6c367c9700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6c300a1dd0 0x7f6c30190920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:12:11.754 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:11.756 7f6c367c9700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6c3008fd30 0x7f6c300effc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:12:11.754 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:11.756 7f6c367c9700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6c300f0e90 0x7f6c300f1290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:12:11.754 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:11.756 7f6c367c9700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c300f06f0 con 0x7f6c3008fd30 2021-02-10T11:12:11.754 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:11.756 7f6c367c9700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c30090160 con 0x7f6c300a1dd0 2021-02-10T11:12:11.754 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:11.756 7f6c367c9700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c300f18b0 con 0x7f6c300f0e90 2021-02-10T11:12:11.754 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:11.756 7f6c2f7fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6c3008fd30 0x7f6c300effc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:12:11.754 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:11.756 7f6c2f7fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6c3008fd30 0x7f6c300effc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:52224/0 (socket says 172.21.2.102:52224) 2021-02-10T11:12:11.754 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:11.756 7f6c2f7fe700 1 -- 172.21.2.102:0/2168870096 learned_addr learned my addr 172.21.2.102:0/2168870096 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:12:11.754 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:11.756 7f6c2f7fe700 1 -- 172.21.2.102:0/2168870096 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6c300f0e90 msgr2=0x7f6c300f1290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:12:11.754 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:11.756 7f6c2f7fe700 1 --2- 172.21.2.102:0/2168870096 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6c300f0e90 0x7f6c300f1290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:12:11.754 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:11.756 7f6c2f7fe700 1 -- 172.21.2.102:0/2168870096 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6c300a1dd0 msgr2=0x7f6c30190920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:12:11.754 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:11.756 7f6c2f7fe700 1 --2- 172.21.2.102:0/2168870096 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6c300a1dd0 0x7f6c30190920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:12:11.755 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:11.756 7f6c2f7fe700 1 -- 172.21.2.102:0/2168870096 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c300f1ad0 con 0x7f6c3008fd30 2021-02-10T11:12:11.755 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:11.756 7f6c2f7fe700 1 --2- 172.21.2.102:0/2168870096 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6c3008fd30 0x7f6c300effc0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 rx=0x7f6c2400b5b0 tx=0x7f6c2400b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:12:11.755 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:11.757 7f6c2cff9700 1 -- 172.21.2.102:0/2168870096 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6c2400bd40 con 0x7f6c3008fd30 2021-02-10T11:12:11.755 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:11.757 7f6c2cff9700 1 -- 172.21.2.102:0/2168870096 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c24000e20 con 0x7f6c3008fd30 2021-02-10T11:12:11.755 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:11.757 7f6c2cff9700 1 -- 172.21.2.102:0/2168870096 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6c24006210 con 0x7f6c3008fd30 2021-02-10T11:12:11.755 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:11.757 7f6c367c9700 1 -- 172.21.2.102:0/2168870096 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c300f1ad0 con 0x7f6c3008fd30 2021-02-10T11:12:11.755 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:11.757 7f6c367c9700 1 -- 172.21.2.102:0/2168870096 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c300f2060 con 0x7f6c3008fd30 2021-02-10T11:12:11.755 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:11.757 7f6c2cff9700 1 -- 172.21.2.102:0/2168870096 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f6c24006020 con 0x7f6c3008fd30 2021-02-10T11:12:11.755 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:11.757 7f6c2cff9700 1 --2- 172.21.2.102:0/2168870096 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6c100186c0 0x7f6c1001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:12:11.755 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:11.757 7f6c2cff9700 1 -- 172.21.2.102:0/2168870096 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f6c24032560 con 0x7f6c3008fd30 2021-02-10T11:12:11.755 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:11.758 7f6c2ffff700 1 --2- 172.21.2.102:0/2168870096 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6c100186c0 0x7f6c1001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:12:11.755 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:11.758 7f6c2ffff700 1 --2- 172.21.2.102:0/2168870096 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6c100186c0 0x7f6c1001abc0 crc :-1 s=READY pgs=263 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:12:11.784 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:11.788 7f6c367c9700 1 -- 172.21.2.102:0/2168870096 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c300e52e0 con 0x7f6c3008fd30 2021-02-10T11:12:11.786 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:11.791 7f6c2cff9700 1 -- 172.21.2.102:0/2168870096 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f6c24007720 con 0x7f6c3008fd30 2021-02-10T11:12:11.916 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:11.921 7f6c367c9700 1 -- 172.21.2.102:0/2168870096 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344409", "target": ["mgr", ""]}) v1 -- 0x7f6c30090160 con 0x7f6c100186c0 2021-02-10T11:12:11.920 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:11.924 7f6c2cff9700 1 -- 172.21.2.102:0/2168870096 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f6c20001140 con 0x7f6c100186c0 2021-02-10T11:12:11.924 INFO:teuthology.orchestra.run.gibba002.stdout:/volumes/_nogroup/subvolume_0000000000344409/9c95278d-8d5e-4bf6-9136-aec8a98b6f9b 2021-02-10T11:12:11.925 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:11.930 7f6c367c9700 1 -- 172.21.2.102:0/2168870096 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6c100186c0 msgr2=0x7f6c1001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:12:11.925 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:11.930 7f6c367c9700 1 --2- 172.21.2.102:0/2168870096 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6c100186c0 0x7f6c1001abc0 crc :-1 s=READY pgs=263 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:12:11.925 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:11.930 7f6c367c9700 1 -- 172.21.2.102:0/2168870096 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6c3008fd30 msgr2=0x7f6c300effc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:12:11.926 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:11.930 7f6c367c9700 1 --2- 172.21.2.102:0/2168870096 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6c3008fd30 0x7f6c300effc0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 rx=0x7f6c2400b5b0 tx=0x7f6c2400b950).stop 2021-02-10T11:12:11.926 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:11.930 7f6c367c9700 1 -- 172.21.2.102:0/2168870096 shutdown_connections 2021-02-10T11:12:11.926 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:11.930 7f6c367c9700 1 --2- 172.21.2.102:0/2168870096 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6c300f0e90 0x7f6c300f1290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:12:11.926 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:11.930 7f6c367c9700 1 --2- 172.21.2.102:0/2168870096 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6c3008fd30 0x7f6c300effc0 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:12:11.926 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:11.930 7f6c367c9700 1 --2- 172.21.2.102:0/2168870096 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6c100186c0 0x7f6c1001abc0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:12:11.926 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:11.930 7f6c367c9700 1 --2- 172.21.2.102:0/2168870096 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6c300a1dd0 0x7f6c30190920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:12:11.926 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:11.930 7f6c367c9700 1 -- 172.21.2.102:0/2168870096 >> 172.21.2.102:0/2168870096 conn(0x7f6c300958b0 msgr2=0x7f6c300bf040 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:12:11.926 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:11.930 7f6c367c9700 1 -- 172.21.2.102:0/2168870096 shutdown_connections 2021-02-10T11:12:11.926 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:11.930 7f6c367c9700 1 -- 172.21.2.102:0/2168870096 wait complete. 2021-02-10T11:12:11.933 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000344409 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000344409/9c95278d-8d5e-4bf6-9136-aec8a98b6f9b 2021-02-10T11:12:11.934 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:12:11.934 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:12:11.934 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:12:12.269 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:12:12.269 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:12:12.269 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344409/9c95278d-8d5e-4bf6-9136-aec8a98b6f9b/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:12:12.651 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:12:12.652 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:12:12.652 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0243621 s, 43.0 MB/s 2021-02-10T11:12:12.653 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:12:12.653 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:12:12.653 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:12:12.989 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:12:12.989 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:12:12.990 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344409/9c95278d-8d5e-4bf6-9136-aec8a98b6f9b/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:12:13.362 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:12:13.362 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:12:13.362 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0243592 s, 43.0 MB/s 2021-02-10T11:12:13.363 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:12:13.363 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:12:13.364 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:12:13.698 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:12:13.699 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:12:13.699 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344409/9c95278d-8d5e-4bf6-9136-aec8a98b6f9b/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:12:14.074 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:12:14.074 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:12:14.074 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0243366 s, 43.1 MB/s 2021-02-10T11:12:14.075 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:12:14.075 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:12:14.075 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:12:14.414 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:12:14.414 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:12:14.415 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344409/9c95278d-8d5e-4bf6-9136-aec8a98b6f9b/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:12:14.790 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:12:14.791 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:12:14.791 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0239046 s, 43.9 MB/s 2021-02-10T11:12:14.792 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:12:14.792 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:12:14.792 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:12:15.131 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:12:15.131 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:12:15.132 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344409/9c95278d-8d5e-4bf6-9136-aec8a98b6f9b/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:12:15.519 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:12:15.519 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:12:15.519 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0252973 s, 41.5 MB/s 2021-02-10T11:12:15.519 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:12:15.520 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:12:15.520 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:12:15.865 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:12:15.865 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:12:15.865 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344409/9c95278d-8d5e-4bf6-9136-aec8a98b6f9b/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:12:16.241 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:12:16.242 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:12:16.242 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0248868 s, 42.1 MB/s 2021-02-10T11:12:16.242 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:12:16.243 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:12:16.243 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:12:16.577 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:12:16.577 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:12:16.578 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344409/9c95278d-8d5e-4bf6-9136-aec8a98b6f9b/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:12:16.952 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:12:16.953 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:12:16.953 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0233999 s, 44.8 MB/s 2021-02-10T11:12:16.954 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:12:16.954 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:12:16.954 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:12:17.289 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:12:17.289 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:12:17.290 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344409/9c95278d-8d5e-4bf6-9136-aec8a98b6f9b/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:12:17.665 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:12:17.666 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:12:17.666 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0238694 s, 43.9 MB/s 2021-02-10T11:12:17.667 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:12:17.667 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:12:17.667 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:12:18.002 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:12:18.002 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:12:18.002 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344409/9c95278d-8d5e-4bf6-9136-aec8a98b6f9b/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:12:18.371 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:12:18.372 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:12:18.372 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0229654 s, 45.7 MB/s 2021-02-10T11:12:18.373 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:12:18.373 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:12:18.373 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:12:18.708 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:12:18.708 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:12:18.709 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344409/9c95278d-8d5e-4bf6-9136-aec8a98b6f9b/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:12:19.084 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:12:19.084 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:12:19.084 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0242627 s, 43.2 MB/s 2021-02-10T11:12:19.085 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000344410 2021-02-10T11:12:19.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:19.505 7fdbc071f700 1 Processor -- start 2021-02-10T11:12:19.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:19.505 7fdbc071f700 1 -- start start 2021-02-10T11:12:19.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:19.506 7fdbc071f700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fdbb81b0470 0x7fdbb81b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:12:19.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:19.506 7fdbc071f700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fdbb80ef610 0x7fdbb80efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:12:19.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:19.506 7fdbc071f700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbb80f0b50 con 0x7fdbb80f0030 2021-02-10T11:12:19.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:19.506 7fdbc071f700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdbb80e52e0 con 0x7fdbb81b0470 2021-02-10T11:12:19.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:19.506 7fdbc071f700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdbb80e5490 con 0x7fdbb80ef610 2021-02-10T11:12:19.518 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:19.506 7fdbbe53b700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fdbb81b0470 0x7fdbb81b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:12:19.518 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:19.506 7fdbbe53b700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fdbb81b0470 0x7fdbb81b0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:52232/0 (socket says 172.21.2.102:52232) 2021-02-10T11:12:19.518 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:19.506 7fdbbe53b700 1 -- 172.21.2.102:0/1667286528 learned_addr learned my addr 172.21.2.102:0/1667286528 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:12:19.518 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:19.506 7fdbbdd3a700 1 --2- 172.21.2.102:0/1667286528 >> v2:172.21.2.113:3300/0 conn(0x7fdbb80ef610 0x7fdbb80efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:12:19.518 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:19.506 7fdbbe53b700 1 -- 172.21.2.102:0/1667286528 >> v2:172.21.2.113:3300/0 conn(0x7fdbb80ef610 msgr2=0x7fdbb80efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:12:19.518 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:19.506 7fdbbe53b700 1 --2- 172.21.2.102:0/1667286528 >> v2:172.21.2.113:3300/0 conn(0x7fdbb80ef610 0x7fdbb80efa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:12:19.518 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:19.506 7fdbbe53b700 1 -- 172.21.2.102:0/1667286528 >> v1:172.21.2.113:6789/0 conn(0x7fdbb80f0030 legacy=0x7fdbb80f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:12:19.518 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:19.506 7fdbbe53b700 1 -- 172.21.2.102:0/1667286528 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbb80f0430 con 0x7fdbb81b0470 2021-02-10T11:12:19.518 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:19.507 7fdbbe53b700 1 --2- 172.21.2.102:0/1667286528 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdbb81b0470 0x7fdbb81b0870 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 rx=0x7fdbb4006700 tx=0x7fdbb4009b80).ready entity=mon.0 client_cookie=76e2d675f152224d server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:12:19.518 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:19.507 7fdbbd539700 1 -- 172.21.2.102:0/1667286528 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdbb400c0c0 con 0x7fdbb81b0470 2021-02-10T11:12:19.518 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:19.507 7fdbbd539700 1 -- 172.21.2.102:0/1667286528 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbb400c440 con 0x7fdbb81b0470 2021-02-10T11:12:19.518 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:19.507 7fdbbd539700 1 -- 172.21.2.102:0/1667286528 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdbb400c920 con 0x7fdbb81b0470 2021-02-10T11:12:19.519 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:19.507 7fdbc071f700 1 -- 172.21.2.102:0/1667286528 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdbb81b0470 msgr2=0x7fdbb81b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:12:19.519 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:19.507 7fdbc071f700 1 --2- 172.21.2.102:0/1667286528 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdbb81b0470 0x7fdbb81b0870 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 rx=0x7fdbb4006700 tx=0x7fdbb4009b80).stop 2021-02-10T11:12:19.519 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:19.507 7fdbc071f700 1 -- 172.21.2.102:0/1667286528 shutdown_connections 2021-02-10T11:12:19.519 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:19.507 7fdbc071f700 1 --2- 172.21.2.102:0/1667286528 >> v2:172.21.2.113:3300/0 conn(0x7fdbb80ef610 0x7fdbb80efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:12:19.519 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:19.507 7fdbc071f700 1 --2- 172.21.2.102:0/1667286528 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdbb81b0470 0x7fdbb81b0870 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:12:19.519 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:19.507 7fdbc071f700 1 -- 172.21.2.102:0/1667286528 >> 172.21.2.102:0/1667286528 conn(0x7fdbb80ce3e0 msgr2=0x7fdbb80e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:12:19.519 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:19.507 7fdbc071f700 1 -- 172.21.2.102:0/1667286528 shutdown_connections 2021-02-10T11:12:19.519 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:19.507 7fdbc071f700 1 -- 172.21.2.102:0/1667286528 wait complete. 2021-02-10T11:12:19.521 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:19.522 7fdbc071f700 1 Processor -- start 2021-02-10T11:12:19.521 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:19.522 7fdbc071f700 1 -- start start 2021-02-10T11:12:19.521 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:19.522 7fdbc071f700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdbb80a0420 0x7fdbb80a0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:12:19.521 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:19.522 7fdbc071f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdbb8189d00 0x7fdbb818a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:12:19.521 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:19.522 7fdbc071f700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fdbb818a720 0x7fdbb80b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:12:19.521 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:19.522 7fdbc071f700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbb8189800 con 0x7fdbb8189d00 2021-02-10T11:12:19.521 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:19.522 7fdbc071f700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbb818ab20 con 0x7fdbb818a720 2021-02-10T11:12:19.521 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:19.522 7fdbc071f700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbb80a1140 con 0x7fdbb80a0420 2021-02-10T11:12:19.522 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:19.523 7fdbbdd3a700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdbb8189d00 0x7fdbb818a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:12:19.522 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:19.523 7fdbbdd3a700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdbb8189d00 0x7fdbb818a100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:52238/0 (socket says 172.21.2.102:52238) 2021-02-10T11:12:19.522 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:19.523 7fdbbdd3a700 1 -- 172.21.2.102:0/4062268192 learned_addr learned my addr 172.21.2.102:0/4062268192 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:12:19.522 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:19.523 7fdbbe53b700 1 --2- 172.21.2.102:0/4062268192 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdbb80a0420 0x7fdbb80a0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:12:19.522 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:19.523 7fdbbed3c700 1 --2- 172.21.2.102:0/4062268192 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fdbb818a720 0x7fdbb80b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:12:19.522 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:19.523 7fdbbdd3a700 1 -- 172.21.2.102:0/4062268192 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdbb80a0420 msgr2=0x7fdbb80a0820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:12:19.522 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:19.523 7fdbbdd3a700 1 --2- 172.21.2.102:0/4062268192 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdbb80a0420 0x7fdbb80a0820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:12:19.522 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:19.523 7fdbbdd3a700 1 -- 172.21.2.102:0/4062268192 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fdbb818a720 msgr2=0x7fdbb80b7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:12:19.522 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:19.523 7fdbbdd3a700 1 --2- 172.21.2.102:0/4062268192 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fdbb818a720 0x7fdbb80b7be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:12:19.522 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:19.523 7fdbbdd3a700 1 -- 172.21.2.102:0/4062268192 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbb80b8200 con 0x7fdbb8189d00 2021-02-10T11:12:19.522 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:19.523 7fdbbe53b700 1 --2- 172.21.2.102:0/4062268192 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdbb80a0420 0x7fdbb80a0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:12:19.522 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:19.523 7fdbbed3c700 1 --2- 172.21.2.102:0/4062268192 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fdbb818a720 0x7fdbb80b7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:12:19.522 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:19.523 7fdbbdd3a700 1 --2- 172.21.2.102:0/4062268192 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdbb8189d00 0x7fdbb818a100 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 rx=0x7fdba8006f60 tx=0x7fdba800b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:12:19.523 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:19.523 7fdbaeffd700 1 -- 172.21.2.102:0/4062268192 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdba800bd60 con 0x7fdbb8189d00 2021-02-10T11:12:19.523 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:19.523 7fdbaeffd700 1 -- 172.21.2.102:0/4062268192 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdba8000c90 con 0x7fdbb8189d00 2021-02-10T11:12:19.523 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:19.523 7fdbc071f700 1 -- 172.21.2.102:0/4062268192 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbb80b8200 con 0x7fdbb8189d00 2021-02-10T11:12:19.523 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:19.523 7fdbaeffd700 1 -- 172.21.2.102:0/4062268192 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdba80022b0 con 0x7fdbb8189d00 2021-02-10T11:12:19.523 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:19.523 7fdbc071f700 1 -- 172.21.2.102:0/4062268192 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbb80b8200 con 0x7fdbb8189d00 2021-02-10T11:12:19.523 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:19.524 7fdbaeffd700 1 -- 172.21.2.102:0/4062268192 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fdba8006b00 con 0x7fdbb8189d00 2021-02-10T11:12:19.523 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:19.524 7fdbaeffd700 1 --2- 172.21.2.102:0/4062268192 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fdba0018730 0x7fdba001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:12:19.523 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:19.524 7fdbaeffd700 1 -- 172.21.2.102:0/4062268192 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fdba8032670 con 0x7fdbb8189d00 2021-02-10T11:12:19.523 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:19.524 7fdbbe53b700 1 --2- 172.21.2.102:0/4062268192 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fdba0018730 0x7fdba001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:12:19.523 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:19.524 7fdbbe53b700 1 --2- 172.21.2.102:0/4062268192 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fdba0018730 0x7fdba001ac10 crc :-1 s=READY pgs=264 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:12:19.551 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:19.555 7fdbc071f700 1 -- 172.21.2.102:0/4062268192 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbb818ab20 con 0x7fdbb8189d00 2021-02-10T11:12:19.553 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:19.558 7fdbaeffd700 1 -- 172.21.2.102:0/4062268192 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fdba80052e0 con 0x7fdbb8189d00 2021-02-10T11:12:19.682 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:19.686 7fdbc071f700 1 -- 172.21.2.102:0/4062268192 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344410", "target": ["mgr", ""]}) v1 -- 0x7fdbb818ab20 con 0x7fdba0018730 2021-02-10T11:12:19.707 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:19.712 7fdbaeffd700 1 -- 172.21.2.102:0/4062268192 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fdbb4006e80 con 0x7fdba0018730 2021-02-10T11:12:19.715 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:19.719 7fdbc071f700 1 -- 172.21.2.102:0/4062268192 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fdba0018730 msgr2=0x7fdba001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:12:19.715 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:19.719 7fdbc071f700 1 --2- 172.21.2.102:0/4062268192 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fdba0018730 0x7fdba001ac10 crc :-1 s=READY pgs=264 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:12:19.715 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:19.719 7fdbc071f700 1 -- 172.21.2.102:0/4062268192 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdbb8189d00 msgr2=0x7fdbb818a100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:12:19.715 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:19.719 7fdbc071f700 1 --2- 172.21.2.102:0/4062268192 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdbb8189d00 0x7fdbb818a100 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 rx=0x7fdba8006f60 tx=0x7fdba800b970).stop 2021-02-10T11:12:19.715 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:19.720 7fdbc071f700 1 -- 172.21.2.102:0/4062268192 shutdown_connections 2021-02-10T11:12:19.715 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:19.720 7fdbc071f700 1 --2- 172.21.2.102:0/4062268192 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fdba0018730 0x7fdba001ac10 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:12:19.715 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:19.720 7fdbc071f700 1 --2- 172.21.2.102:0/4062268192 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fdbb818a720 0x7fdbb80b7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:12:19.716 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:19.720 7fdbc071f700 1 --2- 172.21.2.102:0/4062268192 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdbb8189d00 0x7fdbb818a100 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:12:19.716 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:19.720 7fdbc071f700 1 --2- 172.21.2.102:0/4062268192 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdbb80a0420 0x7fdbb80a0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:12:19.716 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:19.720 7fdbc071f700 1 -- 172.21.2.102:0/4062268192 >> 172.21.2.102:0/4062268192 conn(0x7fdbb8094e10 msgr2=0x7fdbb80a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:12:19.716 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:19.720 7fdbc071f700 1 -- 172.21.2.102:0/4062268192 shutdown_connections 2021-02-10T11:12:19.716 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:19.720 7fdbc071f700 1 -- 172.21.2.102:0/4062268192 wait complete. 2021-02-10T11:12:19.723 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000344410 2021-02-10T11:12:20.154 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:20.142 7fd2f154b700 1 Processor -- start 2021-02-10T11:12:20.154 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:20.142 7fd2f154b700 1 -- start start 2021-02-10T11:12:20.154 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:20.142 7fd2f154b700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fd2ec1b0870 0x7fd2ec0f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:12:20.154 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:20.142 7fd2f154b700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2ec1b0c70 con 0x7fd2ec1b0470 2021-02-10T11:12:20.154 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:20.142 7fd2f154b700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2ec0f0b50 con 0x7fd2ec0efe00 2021-02-10T11:12:20.154 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:20.142 7fd2f154b700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2ec0e5ea0 con 0x7fd2ec1b0870 2021-02-10T11:12:20.154 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:20.142 7fd2eb2bf700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fd2ec1b0470 0x7fd2ec0ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:44470/0 (socket says 172.21.2.102:44470) 2021-02-10T11:12:20.154 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:20.142 7fd2eb2bf700 1 -- 172.21.2.102:0/2138942602 learned_addr learned my addr 172.21.2.102:0/2138942602 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:12:20.155 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:20.143 7fd2eaabe700 1 --2- 172.21.2.102:0/2138942602 >> v2:172.21.2.113:3300/0 conn(0x7fd2ec1b0870 0x7fd2ec0f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:12:20.155 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:20.143 7fd2ea2bd700 1 -- 172.21.2.102:0/2138942602 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 653500043 0 0) 0x7fd2d4000ce0 con 0x7fd2ec1b0470 2021-02-10T11:12:20.155 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:20.143 7fd2ea2bd700 1 -- 172.21.2.102:0/2138942602 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2d8002fe0 con 0x7fd2ec1b0470 2021-02-10T11:12:20.155 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:20.143 7fd2eaabe700 1 -- 172.21.2.102:0/2138942602 >> v1:172.21.2.113:6789/0 conn(0x7fd2ec0efe00 legacy=0x7fd2ec0f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:12:20.155 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:20.143 7fd2eaabe700 1 -- 172.21.2.102:0/2138942602 >> v1:172.21.2.102:6789/0 conn(0x7fd2ec1b0470 legacy=0x7fd2ec0ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:12:20.155 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:20.143 7fd2eaabe700 1 -- 172.21.2.102:0/2138942602 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2ec0e52e0 con 0x7fd2ec1b0870 2021-02-10T11:12:20.155 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:20.143 7fd2eaabe700 1 --2- 172.21.2.102:0/2138942602 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd2ec1b0870 0x7fd2ec0f20e0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7fd2dc006650 tx=0x7fd2dc009a00).ready entity=mon.1 client_cookie=b0377ab253d5910b server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:12:20.155 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:20.143 7fd2ea2bd700 1 -- 172.21.2.102:0/2138942602 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd2dc0009b0 con 0x7fd2ec1b0870 2021-02-10T11:12:20.155 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:20.144 7fd2ea2bd700 1 -- 172.21.2.102:0/2138942602 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2dc00c0c0 con 0x7fd2ec1b0870 2021-02-10T11:12:20.155 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:20.144 7fd2ea2bd700 1 -- 172.21.2.102:0/2138942602 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd2dc00c5a0 con 0x7fd2ec1b0870 2021-02-10T11:12:20.155 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:20.144 7fd2f154b700 1 -- 172.21.2.102:0/2138942602 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd2ec1b0870 msgr2=0x7fd2ec0f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:12:20.155 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:20.144 7fd2f154b700 1 --2- 172.21.2.102:0/2138942602 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd2ec1b0870 0x7fd2ec0f20e0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7fd2dc006650 tx=0x7fd2dc009a00).stop 2021-02-10T11:12:20.156 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:20.144 7fd2f154b700 1 -- 172.21.2.102:0/2138942602 shutdown_connections 2021-02-10T11:12:20.156 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:20.144 7fd2f154b700 1 --2- 172.21.2.102:0/2138942602 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd2ec1b0870 0x7fd2ec0f20e0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:12:20.156 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:20.144 7fd2f154b700 1 -- 172.21.2.102:0/2138942602 >> 172.21.2.102:0/2138942602 conn(0x7fd2ec0ce3e0 msgr2=0x7fd2ec0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:12:20.156 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:20.144 7fd2f154b700 1 -- 172.21.2.102:0/2138942602 shutdown_connections 2021-02-10T11:12:20.156 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:20.144 7fd2f154b700 1 -- 172.21.2.102:0/2138942602 wait complete. 2021-02-10T11:12:20.157 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:20.159 7fd2f154b700 1 Processor -- start 2021-02-10T11:12:20.157 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:20.159 7fd2f154b700 1 -- start start 2021-02-10T11:12:20.158 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:20.159 7fd2f154b700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd2ec0b2840 0x7fd2ec1a4300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:12:20.158 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:20.159 7fd2f154b700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd2ec0b30d0 0x7fd2ec0b34d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:12:20.158 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:20.159 7fd2f154b700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd2ec0b3af0 0x7fd2ec0f0580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:12:20.158 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:20.159 7fd2f154b700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2ec0b2c40 con 0x7fd2ec0b2840 2021-02-10T11:12:20.158 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:20.159 7fd2f154b700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2ec1a4920 con 0x7fd2ec0b30d0 2021-02-10T11:12:20.158 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:20.159 7fd2f154b700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2ec0f0ba0 con 0x7fd2ec0b3af0 2021-02-10T11:12:20.158 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:20.160 7fd2eb2bf700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd2ec0b2840 0x7fd2ec1a4300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:12:20.158 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:20.160 7fd2eb2bf700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd2ec0b2840 0x7fd2ec1a4300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:52252/0 (socket says 172.21.2.102:52252) 2021-02-10T11:12:20.158 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:20.160 7fd2eb2bf700 1 -- 172.21.2.102:0/1447568816 learned_addr learned my addr 172.21.2.102:0/1447568816 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:12:20.158 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:20.160 7fd2eaabe700 1 --2- 172.21.2.102:0/1447568816 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd2ec0b30d0 0x7fd2ec0b34d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:12:20.158 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:20.160 7fd2ebac0700 1 --2- 172.21.2.102:0/1447568816 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd2ec0b3af0 0x7fd2ec0f0580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:12:20.159 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:20.160 7fd2eb2bf700 1 -- 172.21.2.102:0/1447568816 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd2ec0b3af0 msgr2=0x7fd2ec0f0580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:12:20.159 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:20.160 7fd2eb2bf700 1 --2- 172.21.2.102:0/1447568816 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd2ec0b3af0 0x7fd2ec0f0580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:12:20.159 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:20.160 7fd2eb2bf700 1 -- 172.21.2.102:0/1447568816 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd2ec0b30d0 msgr2=0x7fd2ec0b34d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:12:20.159 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:20.160 7fd2eb2bf700 1 --2- 172.21.2.102:0/1447568816 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd2ec0b30d0 0x7fd2ec0b34d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:12:20.159 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:20.160 7fd2eb2bf700 1 -- 172.21.2.102:0/1447568816 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2ec0f0dc0 con 0x7fd2ec0b2840 2021-02-10T11:12:20.159 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:20.160 7fd2eb2bf700 1 --2- 172.21.2.102:0/1447568816 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd2ec0b2840 0x7fd2ec1a4300 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 rx=0x7fd2d4006f00 tx=0x7fd2d400b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:12:20.159 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:20.160 7fd2e3fff700 1 -- 172.21.2.102:0/1447568816 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd2d400bd30 con 0x7fd2ec0b2840 2021-02-10T11:12:20.159 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:20.160 7fd2e3fff700 1 -- 172.21.2.102:0/1447568816 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2d4006030 con 0x7fd2ec0b2840 2021-02-10T11:12:20.159 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:20.160 7fd2e3fff700 1 -- 172.21.2.102:0/1447568816 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd2d4004620 con 0x7fd2ec0b2840 2021-02-10T11:12:20.159 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:20.160 7fd2f154b700 1 -- 172.21.2.102:0/1447568816 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2ec0f0dc0 con 0x7fd2ec0b2840 2021-02-10T11:12:20.159 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:20.160 7fd2f154b700 1 -- 172.21.2.102:0/1447568816 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2ec0f1350 con 0x7fd2ec0b2840 2021-02-10T11:12:20.159 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:20.161 7fd2e3fff700 1 -- 172.21.2.102:0/1447568816 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd2d40045b0 con 0x7fd2ec0b2840 2021-02-10T11:12:20.159 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:20.161 7fd2e3fff700 1 --2- 172.21.2.102:0/1447568816 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd2d00186c0 0x7fd2d001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:12:20.160 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:20.161 7fd2e3fff700 1 -- 172.21.2.102:0/1447568816 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fd2d4032530 con 0x7fd2ec0b2840 2021-02-10T11:12:20.160 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:20.161 7fd2eaabe700 1 --2- 172.21.2.102:0/1447568816 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd2d00186c0 0x7fd2d001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:12:20.160 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:20.161 7fd2eaabe700 1 --2- 172.21.2.102:0/1447568816 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd2d00186c0 0x7fd2d001abc0 crc :-1 s=READY pgs=265 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:12:20.187 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:20.192 7fd2f154b700 1 -- 172.21.2.102:0/1447568816 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2ec0f18f0 con 0x7fd2ec0b2840 2021-02-10T11:12:20.189 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:20.194 7fd2e3fff700 1 -- 172.21.2.102:0/1447568816 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd2d40043a0 con 0x7fd2ec0b2840 2021-02-10T11:12:20.320 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:20.325 7fd2f154b700 1 -- 172.21.2.102:0/1447568816 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344410", "target": ["mgr", ""]}) v1 -- 0x7fd2ec0f1350 con 0x7fd2d00186c0 2021-02-10T11:12:20.323 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:20.328 7fd2e3fff700 1 -- 172.21.2.102:0/1447568816 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7fd2dc0009b0 con 0x7fd2d00186c0 2021-02-10T11:12:20.323 INFO:teuthology.orchestra.run.gibba002.stdout:/volumes/_nogroup/subvolume_0000000000344410/47bdc9b4-f853-40fd-8a6d-5fbf04521d22 2021-02-10T11:12:20.324 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:20.329 7fd2f154b700 1 -- 172.21.2.102:0/1447568816 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd2d00186c0 msgr2=0x7fd2d001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:12:20.324 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:20.329 7fd2f154b700 1 --2- 172.21.2.102:0/1447568816 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd2d00186c0 0x7fd2d001abc0 crc :-1 s=READY pgs=265 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:12:20.324 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:20.329 7fd2f154b700 1 -- 172.21.2.102:0/1447568816 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd2ec0b2840 msgr2=0x7fd2ec1a4300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:12:20.324 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:20.329 7fd2f154b700 1 --2- 172.21.2.102:0/1447568816 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd2ec0b2840 0x7fd2ec1a4300 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 rx=0x7fd2d4006f00 tx=0x7fd2d400b940).stop 2021-02-10T11:12:20.325 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:20.329 7fd2f154b700 1 -- 172.21.2.102:0/1447568816 shutdown_connections 2021-02-10T11:12:20.325 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:20.329 7fd2f154b700 1 --2- 172.21.2.102:0/1447568816 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd2ec0b3af0 0x7fd2ec0f0580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:12:20.325 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:20.329 7fd2f154b700 1 --2- 172.21.2.102:0/1447568816 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd2d00186c0 0x7fd2d001abc0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:12:20.325 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:20.329 7fd2f154b700 1 --2- 172.21.2.102:0/1447568816 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd2ec0b30d0 0x7fd2ec0b34d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:12:20.325 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:20.329 7fd2f154b700 1 --2- 172.21.2.102:0/1447568816 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd2ec0b2840 0x7fd2ec1a4300 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:12:20.325 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:20.329 7fd2f154b700 1 -- 172.21.2.102:0/1447568816 >> 172.21.2.102:0/1447568816 conn(0x7fd2ec0c36f0 msgr2=0x7fd2ec094c30 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:12:20.325 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:20.329 7fd2f154b700 1 -- 172.21.2.102:0/1447568816 shutdown_connections 2021-02-10T11:12:20.325 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:20.329 7fd2f154b700 1 -- 172.21.2.102:0/1447568816 wait complete. 2021-02-10T11:12:20.333 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000344410 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000344410/47bdc9b4-f853-40fd-8a6d-5fbf04521d22 2021-02-10T11:12:20.333 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:12:20.333 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:12:20.333 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:12:20.632 DEBUG:teuthology.orchestra.run.gibba002:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:12:20.636 DEBUG:teuthology.orchestra.run.gibba013:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:12:20.669 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:12:20.669 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:12:20.669 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344410/47bdc9b4-f853-40fd-8a6d-5fbf04521d22/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:12:21.013 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:12:21.013 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:12:21.014 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0231666 s, 45.3 MB/s 2021-02-10T11:12:21.014 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:12:21.015 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:12:21.015 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:12:21.392 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:12:21.393 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:12:21.393 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344410/47bdc9b4-f853-40fd-8a6d-5fbf04521d22/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:12:21.769 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:12:21.769 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:12:21.769 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0241558 s, 43.4 MB/s 2021-02-10T11:12:21.770 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:12:21.770 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:12:21.771 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:12:22.107 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:12:22.107 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:12:22.108 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344410/47bdc9b4-f853-40fd-8a6d-5fbf04521d22/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:12:22.483 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:12:22.483 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:12:22.483 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0249301 s, 42.1 MB/s 2021-02-10T11:12:22.484 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:12:22.484 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:12:22.484 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:12:22.820 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:12:22.820 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:12:22.820 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344410/47bdc9b4-f853-40fd-8a6d-5fbf04521d22/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:12:23.197 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:12:23.197 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:12:23.197 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0251422 s, 41.7 MB/s 2021-02-10T11:12:23.198 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:12:23.198 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:12:23.198 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:12:23.533 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:12:23.533 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:12:23.534 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344410/47bdc9b4-f853-40fd-8a6d-5fbf04521d22/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:12:23.908 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:12:23.908 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:12:23.908 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0249137 s, 42.1 MB/s 2021-02-10T11:12:23.909 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:12:23.910 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:12:23.910 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:12:24.244 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:12:24.244 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:12:24.244 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344410/47bdc9b4-f853-40fd-8a6d-5fbf04521d22/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:12:24.619 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:12:24.619 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:12:24.619 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0233533 s, 44.9 MB/s 2021-02-10T11:12:24.620 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:12:24.621 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:12:24.621 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:12:24.955 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:12:24.955 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:12:24.956 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344410/47bdc9b4-f853-40fd-8a6d-5fbf04521d22/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:12:25.332 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:12:25.332 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:12:25.332 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0251537 s, 41.7 MB/s 2021-02-10T11:12:25.333 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:12:25.333 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:12:25.333 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:12:25.669 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:12:25.669 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:12:25.669 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344410/47bdc9b4-f853-40fd-8a6d-5fbf04521d22/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:12:26.043 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:12:26.043 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:12:26.043 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0223881 s, 46.8 MB/s 2021-02-10T11:12:26.044 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:12:26.044 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:12:26.044 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:12:26.380 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:12:26.381 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:12:26.381 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344410/47bdc9b4-f853-40fd-8a6d-5fbf04521d22/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:12:26.756 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:12:26.756 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:12:26.757 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0242179 s, 43.3 MB/s 2021-02-10T11:12:26.757 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:12:26.758 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:12:26.758 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:12:27.093 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:12:27.093 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:12:27.093 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344410/47bdc9b4-f853-40fd-8a6d-5fbf04521d22/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:12:27.468 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:12:27.468 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:12:27.469 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0243796 s, 43.0 MB/s 2021-02-10T11:12:27.469 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000344411 2021-02-10T11:12:27.903 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:27.891 7efe691d5700 1 Processor -- start 2021-02-10T11:12:27.903 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:27.891 7efe691d5700 1 -- start start 2021-02-10T11:12:27.903 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:27.891 7efe691d5700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7efe641b0470 0x7efe641b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:12:27.903 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:27.891 7efe691d5700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7efe640f20e0 0x7efe640f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:12:27.903 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:27.891 7efe691d5700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe640f0b50 con 0x7efe640ef610 2021-02-10T11:12:27.903 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:27.891 7efe691d5700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe640f0820 con 0x7efe641b0470 2021-02-10T11:12:27.903 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:27.891 7efe691d5700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe640f24e0 con 0x7efe640f20e0 2021-02-10T11:12:27.903 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:27.891 7efe62d9d700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7efe641b0470 0x7efe641b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:12:27.904 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:27.891 7efe6259c700 1 --1- >> v1:172.21.2.113:6789/0 conn(0x7efe640ef610 0x7efe640efa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.113:6789/0 says I am v1:172.21.2.102:51810/0 (socket says 172.21.2.102:51810) 2021-02-10T11:12:27.904 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:27.891 7efe6259c700 1 -- 172.21.2.102:0/3061934112 learned_addr learned my addr 172.21.2.102:0/3061934112 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:12:27.904 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:27.891 7efe62d9d700 1 --2- 172.21.2.102:0/3061934112 >> v2:172.21.2.102:3300/0 conn(0x7efe641b0470 0x7efe641b0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:52260/0 (socket says 172.21.2.102:52260) 2021-02-10T11:12:27.904 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:27.892 7efe6359e700 1 --2- 172.21.2.102:0/3061934112 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7efe640f20e0 0x7efe640f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:12:27.904 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:27.892 7efe62d9d700 1 -- 172.21.2.102:0/3061934112 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7efe640f20e0 msgr2=0x7efe640f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:12:27.904 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:27.892 7efe62d9d700 1 --2- 172.21.2.102:0/3061934112 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7efe640f20e0 0x7efe640f0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:12:27.904 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:27.892 7efe62d9d700 1 -- 172.21.2.102:0/3061934112 >> v1:172.21.2.113:6789/0 conn(0x7efe640ef610 legacy=0x7efe640efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:12:27.904 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:27.892 7efe62d9d700 1 -- 172.21.2.102:0/3061934112 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe640e52e0 con 0x7efe641b0470 2021-02-10T11:12:27.904 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:27.892 7efe62d9d700 1 --2- 172.21.2.102:0/3061934112 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efe641b0470 0x7efe641b0870 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 rx=0x7efe54006650 tx=0x7efe54009a80).ready entity=mon.0 client_cookie=5353c042f1993870 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:12:27.904 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:27.892 7efe61d9b700 1 -- 172.21.2.102:0/3061934112 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efe540009b0 con 0x7efe641b0470 2021-02-10T11:12:27.904 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:27.892 7efe61d9b700 1 -- 172.21.2.102:0/3061934112 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe5400c0c0 con 0x7efe641b0470 2021-02-10T11:12:27.904 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:27.892 7efe61d9b700 1 -- 172.21.2.102:0/3061934112 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efe5400c520 con 0x7efe641b0470 2021-02-10T11:12:27.905 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:27.892 7efe691d5700 1 -- 172.21.2.102:0/3061934112 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efe641b0470 msgr2=0x7efe641b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:12:27.905 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:27.892 7efe691d5700 1 --2- 172.21.2.102:0/3061934112 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efe641b0470 0x7efe641b0870 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 rx=0x7efe54006650 tx=0x7efe54009a80).stop 2021-02-10T11:12:27.905 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:27.893 7efe691d5700 1 -- 172.21.2.102:0/3061934112 shutdown_connections 2021-02-10T11:12:27.905 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:27.893 7efe691d5700 1 --2- 172.21.2.102:0/3061934112 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7efe640f20e0 0x7efe640f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:12:27.905 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:27.893 7efe691d5700 1 --2- 172.21.2.102:0/3061934112 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efe641b0470 0x7efe641b0870 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:12:27.905 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:27.893 7efe691d5700 1 -- 172.21.2.102:0/3061934112 >> 172.21.2.102:0/3061934112 conn(0x7efe640ce3e0 msgr2=0x7efe640e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:12:27.905 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:27.893 7efe691d5700 1 -- 172.21.2.102:0/3061934112 shutdown_connections 2021-02-10T11:12:27.905 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:27.893 7efe691d5700 1 -- 172.21.2.102:0/3061934112 wait complete. 2021-02-10T11:12:27.907 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:27.908 7efe691d5700 1 Processor -- start 2021-02-10T11:12:27.907 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:27.908 7efe691d5700 1 -- start start 2021-02-10T11:12:27.907 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:27.908 7efe691d5700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7efe64190920 0x7efe640b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:12:27.907 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:27.908 7efe691d5700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7efe640b8200 0x7efe640b8600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:12:27.907 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:27.908 7efe691d5700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efe640c1d80 0x7efe640c2180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:12:27.907 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:27.908 7efe691d5700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe64190da0 con 0x7efe640c1d80 2021-02-10T11:12:27.907 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:27.908 7efe691d5700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe640b8c20 con 0x7efe64190920 2021-02-10T11:12:27.907 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:27.908 7efe691d5700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe640b8e10 con 0x7efe640b8200 2021-02-10T11:12:27.908 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:27.908 7efe6359e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efe640c1d80 0x7efe640c2180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:12:27.908 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:27.908 7efe6359e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efe640c1d80 0x7efe640c2180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:52268/0 (socket says 172.21.2.102:52268) 2021-02-10T11:12:27.908 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:27.908 7efe6359e700 1 -- 172.21.2.102:0/1278003364 learned_addr learned my addr 172.21.2.102:0/1278003364 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:12:27.908 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:27.909 7efe6259c700 1 --2- 172.21.2.102:0/1278003364 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7efe640b8200 0x7efe640b8600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:12:27.908 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:27.909 7efe62d9d700 1 --2- 172.21.2.102:0/1278003364 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7efe64190920 0x7efe640b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:12:27.908 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:27.909 7efe6359e700 1 -- 172.21.2.102:0/1278003364 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7efe640b8200 msgr2=0x7efe640b8600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:12:27.908 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:27.909 7efe6359e700 1 --2- 172.21.2.102:0/1278003364 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7efe640b8200 0x7efe640b8600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:12:27.908 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:27.909 7efe6359e700 1 -- 172.21.2.102:0/1278003364 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7efe64190920 msgr2=0x7efe640b7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:12:27.908 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:27.909 7efe6359e700 1 --2- 172.21.2.102:0/1278003364 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7efe64190920 0x7efe640b7be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:12:27.908 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:27.909 7efe6359e700 1 -- 172.21.2.102:0/1278003364 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe640c27d0 con 0x7efe640c1d80 2021-02-10T11:12:27.908 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:27.909 7efe6359e700 1 --2- 172.21.2.102:0/1278003364 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efe640c1d80 0x7efe640c2180 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 rx=0x7efe5c00b970 tx=0x7efe5c00bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:12:27.908 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:27.909 7efe5b7fe700 1 -- 172.21.2.102:0/1278003364 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efe5c000f90 con 0x7efe640c1d80 2021-02-10T11:12:27.908 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:27.909 7efe5b7fe700 1 -- 172.21.2.102:0/1278003364 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe5c006aa0 con 0x7efe640c1d80 2021-02-10T11:12:27.909 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:27.909 7efe691d5700 1 -- 172.21.2.102:0/1278003364 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe640c27d0 con 0x7efe640c1d80 2021-02-10T11:12:27.909 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:27.909 7efe691d5700 1 -- 172.21.2.102:0/1278003364 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe640c27d0 con 0x7efe640c1d80 2021-02-10T11:12:27.909 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:27.910 7efe5b7fe700 1 -- 172.21.2.102:0/1278003364 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efe5c001680 con 0x7efe640c1d80 2021-02-10T11:12:27.909 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:27.910 7efe5b7fe700 1 -- 172.21.2.102:0/1278003364 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7efe5c000df0 con 0x7efe640c1d80 2021-02-10T11:12:27.909 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:27.910 7efe5b7fe700 1 --2- 172.21.2.102:0/1278003364 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7efe48018730 0x7efe4801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:12:27.909 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:27.910 7efe5b7fe700 1 -- 172.21.2.102:0/1278003364 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7efe5c032b40 con 0x7efe640c1d80 2021-02-10T11:12:27.909 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:27.910 7efe62d9d700 1 --2- 172.21.2.102:0/1278003364 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7efe48018730 0x7efe4801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:12:27.909 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:27.910 7efe62d9d700 1 --2- 172.21.2.102:0/1278003364 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7efe48018730 0x7efe4801ac10 crc :-1 s=READY pgs=266 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:12:27.936 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:27.941 7efe691d5700 1 -- 172.21.2.102:0/1278003364 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe64105ae0 con 0x7efe640c1d80 2021-02-10T11:12:27.938 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:27.943 7efe5b7fe700 1 -- 172.21.2.102:0/1278003364 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7efe5c006aa0 con 0x7efe640c1d80 2021-02-10T11:12:28.064 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:28.069 7efe691d5700 1 -- 172.21.2.102:0/1278003364 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344411", "target": ["mgr", ""]}) v1 -- 0x7efe640f0b50 con 0x7efe48018730 2021-02-10T11:12:28.090 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:28.095 7efe5b7fe700 1 -- 172.21.2.102:0/1278003364 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7efe54000c90 con 0x7efe48018730 2021-02-10T11:12:28.097 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:28.102 7efe691d5700 1 -- 172.21.2.102:0/1278003364 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7efe48018730 msgr2=0x7efe4801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:12:28.098 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:28.102 7efe691d5700 1 --2- 172.21.2.102:0/1278003364 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7efe48018730 0x7efe4801ac10 crc :-1 s=READY pgs=266 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:12:28.098 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:28.102 7efe691d5700 1 -- 172.21.2.102:0/1278003364 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efe640c1d80 msgr2=0x7efe640c2180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:12:28.098 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:28.102 7efe691d5700 1 --2- 172.21.2.102:0/1278003364 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efe640c1d80 0x7efe640c2180 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 rx=0x7efe5c00b970 tx=0x7efe5c00bd10).stop 2021-02-10T11:12:28.098 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:28.102 7efe691d5700 1 -- 172.21.2.102:0/1278003364 shutdown_connections 2021-02-10T11:12:28.098 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:28.102 7efe691d5700 1 --2- 172.21.2.102:0/1278003364 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efe640c1d80 0x7efe640c2180 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:12:28.098 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:28.102 7efe691d5700 1 --2- 172.21.2.102:0/1278003364 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7efe640b8200 0x7efe640b8600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:12:28.098 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:28.102 7efe691d5700 1 --2- 172.21.2.102:0/1278003364 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7efe48018730 0x7efe4801ac10 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:12:28.098 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:28.102 7efe691d5700 1 --2- 172.21.2.102:0/1278003364 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7efe64190920 0x7efe640b7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:12:28.098 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:28.102 7efe691d5700 1 -- 172.21.2.102:0/1278003364 >> 172.21.2.102:0/1278003364 conn(0x7efe64094e10 msgr2=0x7efe640a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:12:28.098 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:28.102 7efe691d5700 1 -- 172.21.2.102:0/1278003364 shutdown_connections 2021-02-10T11:12:28.098 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:28.102 7efe691d5700 1 -- 172.21.2.102:0/1278003364 wait complete. 2021-02-10T11:12:28.105 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000344411 2021-02-10T11:12:28.536 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:28.525 7f1fb8638700 1 Processor -- start 2021-02-10T11:12:28.536 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:28.525 7f1fb8638700 1 -- start start 2021-02-10T11:12:28.537 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:28.525 7f1fb8638700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1fb00f05f0 0x7f1fb00f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:12:28.537 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:28.525 7f1fb8638700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fb01b0c70 con 0x7f1fb01b0470 2021-02-10T11:12:28.537 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:28.525 7f1fb8638700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fb00f0b50 con 0x7f1fb01b0870 2021-02-10T11:12:28.537 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:28.525 7f1fb8638700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fb00e5ea0 con 0x7f1fb00f05f0 2021-02-10T11:12:28.537 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:28.525 7f1fb5c53700 1 --1- >> v1:172.21.2.113:6789/0 conn(0x7f1fb01b0870 0x7f1fb00efe00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.113:6789/0 says I am v1:172.21.2.102:51826/0 (socket says 172.21.2.102:51826) 2021-02-10T11:12:28.537 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:28.525 7f1fb6454700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f1fb01b0470 0x7f1fb00ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:44498/0 (socket says 172.21.2.102:44498) 2021-02-10T11:12:28.537 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:28.525 7f1fb5c53700 1 -- 172.21.2.102:0/3148592347 learned_addr learned my addr 172.21.2.102:0/3148592347 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:12:28.537 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:28.525 7f1fb6c55700 1 --2- 172.21.2.102:0/3148592347 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1fb00f05f0 0x7f1fb00f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:12:28.537 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:28.526 7f1fb5452700 1 -- 172.21.2.102:0/3148592347 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2691557922 0 0) 0x7f1fac000ce0 con 0x7f1fb01b0470 2021-02-10T11:12:28.537 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:28.526 7f1fb5452700 1 -- 172.21.2.102:0/3148592347 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f9c002fe0 con 0x7f1fb01b0470 2021-02-10T11:12:28.537 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:28.526 7f1fb5452700 1 -- 172.21.2.102:0/3148592347 <== mon.1 v1:172.21.2.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 747397399 0 0) 0x7f1fa00009b0 con 0x7f1fb01b0870 2021-02-10T11:12:28.537 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:28.526 7f1fb5452700 1 -- 172.21.2.102:0/3148592347 --> v1:172.21.2.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f9c003210 con 0x7f1fb01b0870 2021-02-10T11:12:28.537 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:28.526 7f1fb5452700 1 -- 172.21.2.102:0/3148592347 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4157904524 0 0) 0x7f1fac0010e0 con 0x7f1fb01b0470 2021-02-10T11:12:28.538 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:28.526 7f1fb6c55700 1 -- 172.21.2.102:0/3148592347 >> v1:172.21.2.113:6789/0 conn(0x7f1fb01b0870 legacy=0x7f1fb00efe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:12:28.538 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:28.526 7f1fb6c55700 1 -- 172.21.2.102:0/3148592347 >> v1:172.21.2.102:6789/0 conn(0x7f1fb01b0470 legacy=0x7f1fb00ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:12:28.538 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:28.526 7f1fb6c55700 1 -- 172.21.2.102:0/3148592347 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fb00e52e0 con 0x7f1fb00f05f0 2021-02-10T11:12:28.538 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:28.526 7f1fb6c55700 1 --2- 172.21.2.102:0/3148592347 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1fb00f05f0 0x7f1fb00f20e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f1fa8006e50 tx=0x7f1fa8009e70).ready entity=mon.2 client_cookie=eeaaba685ac122a8 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:12:28.538 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:28.526 7f1fb5452700 1 -- 172.21.2.102:0/3148592347 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1fa8006a20 con 0x7f1fb00f05f0 2021-02-10T11:12:28.538 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:28.526 7f1fb5452700 1 -- 172.21.2.102:0/3148592347 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fa8006790 con 0x7f1fb00f05f0 2021-02-10T11:12:28.538 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:28.526 7f1fb5452700 1 -- 172.21.2.102:0/3148592347 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1fa800b700 con 0x7f1fb00f05f0 2021-02-10T11:12:28.538 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:28.526 7f1fb8638700 1 -- 172.21.2.102:0/3148592347 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1fb00f05f0 msgr2=0x7f1fb00f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:12:28.538 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:28.526 7f1fb8638700 1 --2- 172.21.2.102:0/3148592347 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1fb00f05f0 0x7f1fb00f20e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f1fa8006e50 tx=0x7f1fa8009e70).stop 2021-02-10T11:12:28.538 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:28.527 7f1fb8638700 1 -- 172.21.2.102:0/3148592347 shutdown_connections 2021-02-10T11:12:28.538 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:28.527 7f1fb8638700 1 --2- 172.21.2.102:0/3148592347 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1fb00f05f0 0x7f1fb00f20e0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:12:28.538 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:28.527 7f1fb8638700 1 -- 172.21.2.102:0/3148592347 >> 172.21.2.102:0/3148592347 conn(0x7f1fb00ce3e0 msgr2=0x7f1fb00e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:12:28.538 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:28.527 7f1fb8638700 1 -- 172.21.2.102:0/3148592347 shutdown_connections 2021-02-10T11:12:28.539 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:28.527 7f1fb8638700 1 -- 172.21.2.102:0/3148592347 wait complete. 2021-02-10T11:12:28.540 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:28.541 7f1fb8638700 1 Processor -- start 2021-02-10T11:12:28.540 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:28.542 7f1fb8638700 1 -- start start 2021-02-10T11:12:28.540 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:28.542 7f1fb8638700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1fb009f140 0x7f1fb00c1940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:12:28.540 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:28.542 7f1fb8638700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1fb00c1f60 0x7f1fb009e300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:12:28.541 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:28.542 7f1fb8638700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1fb009e920 0x7f1fb00c2960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:12:28.541 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:28.542 7f1fb8638700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fb00c23e0 con 0x7f1fb009f140 2021-02-10T11:12:28.541 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:28.542 7f1fb8638700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fb009ed20 con 0x7f1fb00c1f60 2021-02-10T11:12:28.541 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:28.542 7f1fb8638700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fb0105d10 con 0x7f1fb009e920 2021-02-10T11:12:28.541 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:28.542 7f1fb5c53700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1fb00c1f60 0x7f1fb009e300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:12:28.541 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:28.542 7f1fb6454700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1fb009f140 0x7f1fb00c1940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:12:28.541 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:28.542 7f1fb6454700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1fb009f140 0x7f1fb00c1940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:52282/0 (socket says 172.21.2.102:52282) 2021-02-10T11:12:28.541 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:28.542 7f1fb6454700 1 -- 172.21.2.102:0/3754847238 learned_addr learned my addr 172.21.2.102:0/3754847238 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:12:28.541 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:28.542 7f1fb6c55700 1 --2- 172.21.2.102:0/3754847238 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1fb009e920 0x7f1fb00c2960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:12:28.541 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:28.543 7f1fb5c53700 1 -- 172.21.2.102:0/3754847238 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1fb009e920 msgr2=0x7f1fb00c2960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:12:28.541 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:28.543 7f1fb5c53700 1 --2- 172.21.2.102:0/3754847238 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1fb009e920 0x7f1fb00c2960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:12:28.541 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:28.543 7f1fb5c53700 1 -- 172.21.2.102:0/3754847238 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1fb009f140 msgr2=0x7f1fb00c1940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:12:28.541 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:28.543 7f1fb5c53700 1 --2- 172.21.2.102:0/3754847238 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1fb009f140 0x7f1fb00c1940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:12:28.542 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:28.543 7f1fb5c53700 1 -- 172.21.2.102:0/3754847238 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fb01a4910 con 0x7f1fb00c1f60 2021-02-10T11:12:28.542 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:28.543 7f1fb6454700 1 --2- 172.21.2.102:0/3754847238 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1fb009f140 0x7f1fb00c1940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:12:28.542 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:28.543 7f1fb5c53700 1 --2- 172.21.2.102:0/3754847238 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1fb00c1f60 0x7f1fb009e300 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f1fa000a4a0 tx=0x7f1fa000a840).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:12:28.542 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:28.543 7f1fa6ffd700 1 -- 172.21.2.102:0/3754847238 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1fa000ac30 con 0x7f1fb00c1f60 2021-02-10T11:12:28.542 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:28.543 7f1fa6ffd700 1 -- 172.21.2.102:0/3754847238 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fa0000f10 con 0x7f1fb00c1f60 2021-02-10T11:12:28.542 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:28.543 7f1fa6ffd700 1 -- 172.21.2.102:0/3754847238 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1fa00045d0 con 0x7f1fb00c1f60 2021-02-10T11:12:28.542 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:28.543 7f1fb8638700 1 -- 172.21.2.102:0/3754847238 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fb01a4910 con 0x7f1fb00c1f60 2021-02-10T11:12:28.542 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:28.543 7f1fb8638700 1 -- 172.21.2.102:0/3754847238 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fb01a4910 con 0x7f1fb00c1f60 2021-02-10T11:12:28.542 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:28.544 7f1fa6ffd700 1 -- 172.21.2.102:0/3754847238 <== mon.1 v2:172.21.2.113:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1fa0004140 con 0x7f1fb00c1f60 2021-02-10T11:12:28.543 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:28.544 7f1fa6ffd700 1 --2- 172.21.2.102:0/3754847238 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1f980186c0 0x7f1f9801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:12:28.543 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:28.544 7f1fa6ffd700 1 -- 172.21.2.102:0/3754847238 <== mon.1 v2:172.21.2.113:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f1fa0032490 con 0x7f1fb00c1f60 2021-02-10T11:12:28.543 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:28.544 7f1fb6454700 1 --2- 172.21.2.102:0/3754847238 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1f980186c0 0x7f1f9801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:12:28.543 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:28.544 7f1fb6454700 1 --2- 172.21.2.102:0/3754847238 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1f980186c0 0x7f1f9801abc0 crc :-1 s=READY pgs=267 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:12:28.570 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:28.575 7f1fb8638700 1 -- 172.21.2.102:0/3754847238 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fb00f0b50 con 0x7f1fb00c1f60 2021-02-10T11:12:28.574 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:28.579 7f1fa6ffd700 1 -- 172.21.2.102:0/3754847238 <== mon.1 v2:172.21.2.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f1fa000ac30 con 0x7f1fb00c1f60 2021-02-10T11:12:28.705 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:28.710 7f1fb8638700 1 -- 172.21.2.102:0/3754847238 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344411", "target": ["mgr", ""]}) v1 -- 0x7f1fb00f0b50 con 0x7f1f980186c0 2021-02-10T11:12:28.708 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:28.713 7f1fa6ffd700 1 -- 172.21.2.102:0/3754847238 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f1fac000f70 con 0x7f1f980186c0 2021-02-10T11:12:28.709 INFO:teuthology.orchestra.run.gibba002.stdout:/volumes/_nogroup/subvolume_0000000000344411/d1cc9a2c-4482-4b56-a0f1-4634b22cd860 2021-02-10T11:12:28.710 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:28.715 7f1fb8638700 1 -- 172.21.2.102:0/3754847238 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1f980186c0 msgr2=0x7f1f9801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:12:28.710 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:28.715 7f1fb8638700 1 --2- 172.21.2.102:0/3754847238 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1f980186c0 0x7f1f9801abc0 crc :-1 s=READY pgs=267 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:12:28.710 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:28.715 7f1fb8638700 1 -- 172.21.2.102:0/3754847238 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1fb00c1f60 msgr2=0x7f1fb009e300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:12:28.710 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:28.715 7f1fb8638700 1 --2- 172.21.2.102:0/3754847238 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1fb00c1f60 0x7f1fb009e300 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f1fa000a4a0 tx=0x7f1fa000a840).stop 2021-02-10T11:12:28.711 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:28.715 7f1fb8638700 1 -- 172.21.2.102:0/3754847238 shutdown_connections 2021-02-10T11:12:28.711 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:28.715 7f1fb8638700 1 --2- 172.21.2.102:0/3754847238 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1fb009e920 0x7f1fb00c2960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:12:28.711 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:28.715 7f1fb8638700 1 --2- 172.21.2.102:0/3754847238 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1f980186c0 0x7f1f9801abc0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:12:28.711 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:28.715 7f1fb8638700 1 --2- 172.21.2.102:0/3754847238 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1fb00c1f60 0x7f1fb009e300 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:12:28.711 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:28.715 7f1fb8638700 1 --2- 172.21.2.102:0/3754847238 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1fb009f140 0x7f1fb00c1940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:12:28.711 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:28.715 7f1fb8638700 1 -- 172.21.2.102:0/3754847238 >> 172.21.2.102:0/3754847238 conn(0x7f1fb009fc10 msgr2=0x7f1fb00b3250 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:12:28.711 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:28.715 7f1fb8638700 1 -- 172.21.2.102:0/3754847238 shutdown_connections 2021-02-10T11:12:28.711 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:28.715 7f1fb8638700 1 -- 172.21.2.102:0/3754847238 wait complete. 2021-02-10T11:12:28.718 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000344411 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000344411/d1cc9a2c-4482-4b56-a0f1-4634b22cd860 2021-02-10T11:12:28.718 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:12:28.718 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:12:28.719 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:12:29.054 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:12:29.055 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:12:29.055 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344411/d1cc9a2c-4482-4b56-a0f1-4634b22cd860/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:12:29.435 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:12:29.435 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:12:29.436 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0260154 s, 40.3 MB/s 2021-02-10T11:12:29.436 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:12:29.437 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:12:29.437 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:12:29.773 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:12:29.774 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:12:29.774 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344411/d1cc9a2c-4482-4b56-a0f1-4634b22cd860/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:12:30.151 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:12:30.151 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:12:30.151 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.025195 s, 41.6 MB/s 2021-02-10T11:12:30.152 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:12:30.152 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:12:30.152 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:12:30.487 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:12:30.488 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:12:30.488 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344411/d1cc9a2c-4482-4b56-a0f1-4634b22cd860/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:12:30.860 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:12:30.860 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:12:30.860 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0240865 s, 43.5 MB/s 2021-02-10T11:12:30.861 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:12:30.861 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:12:30.861 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:12:31.196 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:12:31.197 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:12:31.197 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344411/d1cc9a2c-4482-4b56-a0f1-4634b22cd860/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:12:31.572 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:12:31.572 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:12:31.572 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0228238 s, 45.9 MB/s 2021-02-10T11:12:31.573 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:12:31.573 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:12:31.573 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:12:31.910 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:12:31.910 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:12:31.911 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344411/d1cc9a2c-4482-4b56-a0f1-4634b22cd860/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:12:32.286 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:12:32.286 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:12:32.286 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0227381 s, 46.1 MB/s 2021-02-10T11:12:32.287 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:12:32.287 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:12:32.287 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:12:32.624 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:12:32.624 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:12:32.624 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344411/d1cc9a2c-4482-4b56-a0f1-4634b22cd860/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:12:33.005 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:12:33.005 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:12:33.005 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0230646 s, 45.5 MB/s 2021-02-10T11:12:33.006 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:12:33.006 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:12:33.006 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:12:33.342 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:12:33.342 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:12:33.343 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344411/d1cc9a2c-4482-4b56-a0f1-4634b22cd860/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:12:33.718 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:12:33.718 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:12:33.718 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0255523 s, 41.0 MB/s 2021-02-10T11:12:33.719 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:12:33.719 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:12:33.719 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:12:34.055 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:12:34.055 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:12:34.055 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344411/d1cc9a2c-4482-4b56-a0f1-4634b22cd860/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:12:34.433 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:12:34.433 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:12:34.433 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0246718 s, 42.5 MB/s 2021-02-10T11:12:34.434 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:12:34.434 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:12:34.434 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:12:34.769 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:12:34.770 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:12:34.770 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344411/d1cc9a2c-4482-4b56-a0f1-4634b22cd860/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:12:35.146 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:12:35.146 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:12:35.146 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.024024 s, 43.6 MB/s 2021-02-10T11:12:35.147 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:12:35.147 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:12:35.147 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:12:35.484 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:12:35.484 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:12:35.484 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344411/d1cc9a2c-4482-4b56-a0f1-4634b22cd860/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:12:35.859 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:12:35.860 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:12:35.860 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0240103 s, 43.7 MB/s 2021-02-10T11:12:35.861 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000344412 2021-02-10T11:12:36.311 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:36.298 7f83907bb700 1 Processor -- start 2021-02-10T11:12:36.311 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:36.298 7f83907bb700 1 -- start start 2021-02-10T11:12:36.311 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:36.299 7f83907bb700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f83880f05f0 0x7f83880f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:12:36.311 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:36.299 7f83907bb700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83881b0c70 con 0x7f83881b0470 2021-02-10T11:12:36.311 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:36.299 7f83907bb700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83880f0b50 con 0x7f83881b0870 2021-02-10T11:12:36.311 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:36.299 7f83907bb700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83880e5ea0 con 0x7f83880f05f0 2021-02-10T11:12:36.311 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:36.299 7f838e5d7700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f83881b0470 0x7f83880ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:44512/0 (socket says 172.21.2.102:44512) 2021-02-10T11:12:36.311 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:36.299 7f838e5d7700 1 -- 172.21.2.102:0/972915618 learned_addr learned my addr 172.21.2.102:0/972915618 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:12:36.311 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:36.299 7f838d5d5700 1 -- 172.21.2.102:0/972915618 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2717289960 0 0) 0x7f83840010b0 con 0x7f83881b0470 2021-02-10T11:12:36.312 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:36.299 7f838d5d5700 1 -- 172.21.2.102:0/972915618 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8374002fe0 con 0x7f83881b0470 2021-02-10T11:12:36.312 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:36.299 7f838edd8700 1 --2- 172.21.2.102:0/972915618 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f83880f05f0 0x7f83880f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:12:36.312 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:36.300 7f838d5d5700 1 -- 172.21.2.102:0/972915618 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 462572733 0 0) 0x7f83840010b0 con 0x7f83881b0470 2021-02-10T11:12:36.312 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:36.300 7f838d5d5700 1 -- 172.21.2.102:0/972915618 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f83880f05f0 msgr2=0x7f83880f20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:12:36.312 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:36.300 7f838d5d5700 1 --2- 172.21.2.102:0/972915618 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f83880f05f0 0x7f83880f20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:12:36.312 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:36.300 7f838d5d5700 1 -- 172.21.2.102:0/972915618 >> v1:172.21.2.113:6789/0 conn(0x7f83881b0870 legacy=0x7f83880efe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:12:36.312 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:36.300 7f838d5d5700 1 -- 172.21.2.102:0/972915618 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83880e52e0 con 0x7f83881b0470 2021-02-10T11:12:36.312 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:36.300 7f838d5d5700 1 -- 172.21.2.102:0/972915618 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7f83840015b0 con 0x7f83881b0470 2021-02-10T11:12:36.312 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:36.300 7f838d5d5700 1 -- 172.21.2.102:0/972915618 >> v1:172.21.2.102:6789/0 conn(0x7f83881b0470 legacy=0x7f83880ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:12:36.312 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:36.300 7f838d5d5700 1 --2- 172.21.2.102:0/972915618 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8374003ae0 0x7f8374005f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:12:36.312 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:36.300 7f838d5d5700 1 --2- 172.21.2.102:0/972915618 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8374006830 0x7f8374008d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:12:36.312 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:36.300 7f838d5d5700 1 --2- 172.21.2.102:0/972915618 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f83740095a0 0x7f837400bac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:12:36.314 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:36.300 7f838d5d5700 1 -- 172.21.2.102:0/972915618 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f837400c370 con 0x7f8374003ae0 2021-02-10T11:12:36.314 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:36.300 7f838d5d5700 1 -- 172.21.2.102:0/972915618 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f837400c590 con 0x7f83740095a0 2021-02-10T11:12:36.314 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:36.300 7f838d5d5700 1 -- 172.21.2.102:0/972915618 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f837400c7b0 con 0x7f8374006830 2021-02-10T11:12:36.314 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:36.300 7f838edd8700 1 --2- 172.21.2.102:0/972915618 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f83880f05f0 0x7f83880f20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:12:36.314 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:36.300 7f838e5d7700 1 --2- 172.21.2.102:0/972915618 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8374003ae0 0x7f8374005f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:12:36.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:36.300 7f838e5d7700 1 -- 172.21.2.102:0/972915618 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8374006830 msgr2=0x7f8374008d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:12:36.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:36.300 7f838ddd6700 1 --2- 172.21.2.102:0/972915618 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8374006830 0x7f8374008d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:12:36.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:36.300 7f838e5d7700 1 --2- 172.21.2.102:0/972915618 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8374006830 0x7f8374008d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:12:36.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:36.300 7f838edd8700 1 --2- 172.21.2.102:0/972915618 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f83740095a0 0x7f837400bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:12:36.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:36.300 7f838e5d7700 1 -- 172.21.2.102:0/972915618 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f83740095a0 msgr2=0x7f837400bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:12:36.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:36.300 7f838e5d7700 1 --2- 172.21.2.102:0/972915618 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f83740095a0 0x7f837400bac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:12:36.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:36.300 7f838e5d7700 1 -- 172.21.2.102:0/972915618 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f837400ca50 con 0x7f8374003ae0 2021-02-10T11:12:36.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:36.300 7f838ddd6700 1 --2- 172.21.2.102:0/972915618 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8374006830 0x7f8374008d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:12:36.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:36.300 7f838e5d7700 1 --2- 172.21.2.102:0/972915618 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8374003ae0 0x7f8374005f90 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 rx=0x7f83840017d0 tx=0x7f8384007d30).ready entity=mon.0 client_cookie=e83b6d5f0a28facf server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:12:36.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:36.300 7f838d5d5700 1 -- 172.21.2.102:0/972915618 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f83840013a0 con 0x7f8374003ae0 2021-02-10T11:12:36.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:36.300 7f838d5d5700 1 -- 172.21.2.102:0/972915618 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f838400a310 con 0x7f8374003ae0 2021-02-10T11:12:36.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:36.301 7f838d5d5700 1 -- 172.21.2.102:0/972915618 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f838400a7a0 con 0x7f8374003ae0 2021-02-10T11:12:36.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:36.301 7f83907bb700 1 -- 172.21.2.102:0/972915618 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8374003ae0 msgr2=0x7f8374005f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:12:36.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:36.301 7f83907bb700 1 --2- 172.21.2.102:0/972915618 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8374003ae0 0x7f8374005f90 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 rx=0x7f83840017d0 tx=0x7f8384007d30).stop 2021-02-10T11:12:36.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:36.301 7f838edd8700 1 -- 172.21.2.102:0/972915618 reap_dead start 2021-02-10T11:12:36.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:36.301 7f83907bb700 1 -- 172.21.2.102:0/972915618 shutdown_connections 2021-02-10T11:12:36.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:36.301 7f83907bb700 1 -- 172.21.2.102:0/972915618 >> 172.21.2.102:0/972915618 conn(0x7f83880ce3e0 msgr2=0x7f83880e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:12:36.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:36.301 7f83907bb700 1 -- 172.21.2.102:0/972915618 shutdown_connections 2021-02-10T11:12:36.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:36.301 7f83907bb700 1 -- 172.21.2.102:0/972915618 wait complete. 2021-02-10T11:12:36.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:36.316 7f83907bb700 1 Processor -- start 2021-02-10T11:12:36.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:36.316 7f83907bb700 1 -- start start 2021-02-10T11:12:36.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:36.316 7f83907bb700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f83880b7db0 0x7f83880b8260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:12:36.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:36.316 7f83907bb700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f83880b8880 0x7f83881a7450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:12:36.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:36.316 7f83907bb700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f83880b8c80 0x7f83881a7a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:12:36.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:36.316 7f83907bb700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83880b79b0 con 0x7f83880b8c80 2021-02-10T11:12:36.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:36.316 7f83907bb700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83881a8090 con 0x7f83880b7db0 2021-02-10T11:12:36.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:36.316 7f83907bb700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83881a82b0 con 0x7f83880b8880 2021-02-10T11:12:36.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:36.317 7f838edd8700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f83880b8c80 0x7f83881a7a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:12:36.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:36.317 7f838edd8700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f83880b8c80 0x7f83881a7a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:52300/0 (socket says 172.21.2.102:52300) 2021-02-10T11:12:36.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:36.317 7f838edd8700 1 -- 172.21.2.102:0/2688358648 learned_addr learned my addr 172.21.2.102:0/2688358648 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:12:36.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:36.317 7f838e5d7700 1 --2- 172.21.2.102:0/2688358648 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f83880b7db0 0x7f83880b8260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:12:36.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:36.317 7f838ddd6700 1 --2- 172.21.2.102:0/2688358648 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f83880b8880 0x7f83881a7450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:12:36.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:36.317 7f838edd8700 1 -- 172.21.2.102:0/2688358648 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f83880b8880 msgr2=0x7f83881a7450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:12:36.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:36.317 7f838edd8700 1 --2- 172.21.2.102:0/2688358648 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f83880b8880 0x7f83881a7450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:12:36.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:36.317 7f838edd8700 1 -- 172.21.2.102:0/2688358648 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f83880b7db0 msgr2=0x7f83880b8260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:12:36.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:36.317 7f838edd8700 1 --2- 172.21.2.102:0/2688358648 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f83880b7db0 0x7f83880b8260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:12:36.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:36.317 7f838edd8700 1 -- 172.21.2.102:0/2688358648 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8388190c10 con 0x7f83880b8c80 2021-02-10T11:12:36.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:36.317 7f838ddd6700 1 --2- 172.21.2.102:0/2688358648 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f83880b8880 0x7f83881a7450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:12:36.318 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:36.317 7f838edd8700 1 --2- 172.21.2.102:0/2688358648 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f83880b8c80 0x7f83881a7a70 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 rx=0x7f8380050990 tx=0x7f838004ecc0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:12:36.318 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:36.317 7f837affd700 1 -- 172.21.2.102:0/2688358648 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8380006970 con 0x7f83880b8c80 2021-02-10T11:12:36.318 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:36.317 7f837affd700 1 -- 172.21.2.102:0/2688358648 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8380003030 con 0x7f83880b8c80 2021-02-10T11:12:36.318 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:36.317 7f837affd700 1 -- 172.21.2.102:0/2688358648 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8380009590 con 0x7f83880b8c80 2021-02-10T11:12:36.318 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:36.317 7f83907bb700 1 -- 172.21.2.102:0/2688358648 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8388190c10 con 0x7f83880b8c80 2021-02-10T11:12:36.318 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:36.318 7f83907bb700 1 -- 172.21.2.102:0/2688358648 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83880c1eb0 con 0x7f83880b8c80 2021-02-10T11:12:36.318 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:36.318 7f837affd700 1 -- 172.21.2.102:0/2688358648 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f8380006c90 con 0x7f83880b8c80 2021-02-10T11:12:36.318 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:36.318 7f837affd700 1 --2- 172.21.2.102:0/2688358648 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f83700186c0 0x7f837001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:12:36.318 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:36.318 7f837affd700 1 -- 172.21.2.102:0/2688358648 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f83800767b0 con 0x7f83880b8c80 2021-02-10T11:12:36.318 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:36.318 7f838e5d7700 1 --2- 172.21.2.102:0/2688358648 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f83700186c0 0x7f837001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:12:36.319 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:36.319 7f838e5d7700 1 --2- 172.21.2.102:0/2688358648 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f83700186c0 0x7f837001abc0 crc :-1 s=READY pgs=268 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:12:36.344 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:36.348 7f83907bb700 1 -- 172.21.2.102:0/2688358648 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8388105b30 con 0x7f83880b8c80 2021-02-10T11:12:36.346 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:36.351 7f837affd700 1 -- 172.21.2.102:0/2688358648 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f83800638b0 con 0x7f83880b8c80 2021-02-10T11:12:36.472 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:36.477 7f83907bb700 1 -- 172.21.2.102:0/2688358648 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344412", "target": ["mgr", ""]}) v1 -- 0x7f83880c1eb0 con 0x7f83700186c0 2021-02-10T11:12:36.497 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:36.502 7f837affd700 1 -- 172.21.2.102:0/2688358648 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f83840072d0 con 0x7f83700186c0 2021-02-10T11:12:36.505 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:36.510 7f83907bb700 1 -- 172.21.2.102:0/2688358648 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f83700186c0 msgr2=0x7f837001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:12:36.505 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:36.510 7f83907bb700 1 --2- 172.21.2.102:0/2688358648 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f83700186c0 0x7f837001abc0 crc :-1 s=READY pgs=268 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:12:36.505 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:36.510 7f83907bb700 1 -- 172.21.2.102:0/2688358648 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f83880b8c80 msgr2=0x7f83881a7a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:12:36.505 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:36.510 7f83907bb700 1 --2- 172.21.2.102:0/2688358648 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f83880b8c80 0x7f83881a7a70 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 rx=0x7f8380050990 tx=0x7f838004ecc0).stop 2021-02-10T11:12:36.505 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:36.510 7f83907bb700 1 -- 172.21.2.102:0/2688358648 shutdown_connections 2021-02-10T11:12:36.506 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:36.510 7f83907bb700 1 --2- 172.21.2.102:0/2688358648 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f83880b8c80 0x7f83881a7a70 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:12:36.506 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:36.510 7f83907bb700 1 --2- 172.21.2.102:0/2688358648 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f83880b8880 0x7f83881a7450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:12:36.506 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:36.510 7f83907bb700 1 --2- 172.21.2.102:0/2688358648 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f83700186c0 0x7f837001abc0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:12:36.506 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:36.510 7f83907bb700 1 --2- 172.21.2.102:0/2688358648 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f83880b7db0 0x7f83880b8260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:12:36.506 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:36.510 7f83907bb700 1 -- 172.21.2.102:0/2688358648 >> 172.21.2.102:0/2688358648 conn(0x7f8388094e10 msgr2=0x7f83880a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:12:36.506 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:36.510 7f83907bb700 1 -- 172.21.2.102:0/2688358648 shutdown_connections 2021-02-10T11:12:36.506 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:36.510 7f83907bb700 1 -- 172.21.2.102:0/2688358648 wait complete. 2021-02-10T11:12:36.513 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000344412 2021-02-10T11:12:36.951 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:36.939 7f40a856a700 1 Processor -- start 2021-02-10T11:12:36.951 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:36.939 7f40a856a700 1 -- start start 2021-02-10T11:12:36.951 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:36.939 7f40a856a700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f40a01b0870 0x7f40a00f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:12:36.951 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:36.939 7f40a856a700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40a01b0c70 con 0x7f40a01b0470 2021-02-10T11:12:36.951 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:36.939 7f40a856a700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40a00f0b50 con 0x7f40a00efe00 2021-02-10T11:12:36.951 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:36.939 7f40a856a700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40a00e5ea0 con 0x7f40a01b0870 2021-02-10T11:12:36.951 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:36.940 7f40a6386700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f40a01b0470 0x7f40a00ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:44532/0 (socket says 172.21.2.102:44532) 2021-02-10T11:12:36.951 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:36.940 7f40a6386700 1 -- 172.21.2.102:0/1587231826 learned_addr learned my addr 172.21.2.102:0/1587231826 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:12:36.951 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:36.940 7f40a5b85700 1 --2- 172.21.2.102:0/1587231826 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f40a01b0870 0x7f40a00f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:12:36.951 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:36.940 7f40a5384700 1 -- 172.21.2.102:0/1587231826 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 624734667 0 0) 0x7f40940010b0 con 0x7f40a01b0470 2021-02-10T11:12:36.951 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:36.940 7f40a5384700 1 -- 172.21.2.102:0/1587231826 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f408c002fe0 con 0x7f40a01b0470 2021-02-10T11:12:36.952 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:36.940 7f40a5384700 1 -- 172.21.2.102:0/1587231826 <== mon.1 v1:172.21.2.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 70534942 0 0) 0x7f4098001400 con 0x7f40a00efe00 2021-02-10T11:12:36.952 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:36.940 7f40a5b85700 1 -- 172.21.2.102:0/1587231826 >> v1:172.21.2.113:6789/0 conn(0x7f40a00efe00 legacy=0x7f40a00f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:12:36.952 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:36.940 7f40a5b85700 1 -- 172.21.2.102:0/1587231826 >> v1:172.21.2.102:6789/0 conn(0x7f40a01b0470 legacy=0x7f40a00ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:12:36.952 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:36.940 7f40a5b85700 1 -- 172.21.2.102:0/1587231826 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40a00e52e0 con 0x7f40a01b0870 2021-02-10T11:12:36.952 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:36.940 7f40a5b85700 1 --2- 172.21.2.102:0/1587231826 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f40a01b0870 0x7f40a00f20e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f409c006650 tx=0x7f409c009a80).ready entity=mon.2 client_cookie=6536ac3ebbf375a1 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:12:36.952 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:36.941 7f40a5384700 1 -- 172.21.2.102:0/1587231826 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f409c00c0c0 con 0x7f40a01b0870 2021-02-10T11:12:36.952 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:36.941 7f40a5384700 1 -- 172.21.2.102:0/1587231826 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f409c00c340 con 0x7f40a01b0870 2021-02-10T11:12:36.952 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:36.941 7f40a5384700 1 -- 172.21.2.102:0/1587231826 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f409c0009b0 con 0x7f40a01b0870 2021-02-10T11:12:36.952 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:36.941 7f40a856a700 1 -- 172.21.2.102:0/1587231826 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f40a01b0870 msgr2=0x7f40a00f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:12:36.952 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:36.941 7f40a856a700 1 --2- 172.21.2.102:0/1587231826 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f40a01b0870 0x7f40a00f20e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f409c006650 tx=0x7f409c009a80).stop 2021-02-10T11:12:36.952 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:36.941 7f40a856a700 1 -- 172.21.2.102:0/1587231826 shutdown_connections 2021-02-10T11:12:36.952 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:36.941 7f40a856a700 1 --2- 172.21.2.102:0/1587231826 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f40a01b0870 0x7f40a00f20e0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:12:36.953 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:36.941 7f40a856a700 1 -- 172.21.2.102:0/1587231826 >> 172.21.2.102:0/1587231826 conn(0x7f40a00ce3e0 msgr2=0x7f40a00e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:12:36.953 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:36.941 7f40a856a700 1 -- 172.21.2.102:0/1587231826 shutdown_connections 2021-02-10T11:12:36.953 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:36.941 7f40a856a700 1 -- 172.21.2.102:0/1587231826 wait complete. 2021-02-10T11:12:36.953 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:36.956 7f40a856a700 1 Processor -- start 2021-02-10T11:12:36.954 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:36.956 7f40a856a700 1 -- start start 2021-02-10T11:12:36.954 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:36.956 7f40a856a700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f40a0190920 0x7f40a008f600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:12:36.954 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:36.956 7f40a856a700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f40a0190d20 0x7f40a008fc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:12:36.954 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:36.956 7f40a856a700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f40a0193270 0x7f40a0193670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:12:36.954 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:36.956 7f40a856a700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40a0090470 con 0x7f40a0193270 2021-02-10T11:12:36.954 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:36.956 7f40a856a700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40a0193c90 con 0x7f40a0190920 2021-02-10T11:12:36.954 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:36.956 7f40a856a700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40a0193e70 con 0x7f40a0190d20 2021-02-10T11:12:36.954 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:36.956 7f40a6b87700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f40a0193270 0x7f40a0193670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:12:36.954 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:36.956 7f40a6b87700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f40a0193270 0x7f40a0193670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:52318/0 (socket says 172.21.2.102:52318) 2021-02-10T11:12:36.954 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:36.956 7f40a6386700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f40a0190920 0x7f40a008f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:12:36.954 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:36.956 7f40a6b87700 1 -- 172.21.2.102:0/1036145868 learned_addr learned my addr 172.21.2.102:0/1036145868 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:12:36.955 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:36.957 7f40a5b85700 1 --2- 172.21.2.102:0/1036145868 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f40a0190d20 0x7f40a008fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:12:36.956 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:36.957 7f40a6b87700 1 -- 172.21.2.102:0/1036145868 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f40a0190d20 msgr2=0x7f40a008fc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:12:36.956 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:36.957 7f40a6b87700 1 --2- 172.21.2.102:0/1036145868 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f40a0190d20 0x7f40a008fc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:12:36.956 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:36.957 7f40a6b87700 1 -- 172.21.2.102:0/1036145868 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f40a0190920 msgr2=0x7f40a008f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:12:36.956 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:36.957 7f40a6b87700 1 --2- 172.21.2.102:0/1036145868 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f40a0190920 0x7f40a008f600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:12:36.956 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:36.957 7f40a6b87700 1 -- 172.21.2.102:0/1036145868 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40a0194090 con 0x7f40a0193270 2021-02-10T11:12:36.956 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:36.957 7f40a6b87700 1 --2- 172.21.2.102:0/1036145868 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f40a0193270 0x7f40a0193670 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 rx=0x7f4098009940 tx=0x7f409800bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:12:36.956 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:36.957 7f4092ffd700 1 -- 172.21.2.102:0/1036145868 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4098000f40 con 0x7f40a0193270 2021-02-10T11:12:36.956 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:36.957 7f4092ffd700 1 -- 172.21.2.102:0/1036145868 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f409800c2f0 con 0x7f40a0193270 2021-02-10T11:12:36.956 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:36.957 7f4092ffd700 1 -- 172.21.2.102:0/1036145868 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f409800c810 con 0x7f40a0193270 2021-02-10T11:12:36.956 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:36.957 7f40a856a700 1 -- 172.21.2.102:0/1036145868 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40a0194090 con 0x7f40a0193270 2021-02-10T11:12:36.956 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:36.957 7f40a856a700 1 -- 172.21.2.102:0/1036145868 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40a01920b0 con 0x7f40a0193270 2021-02-10T11:12:36.956 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:36.958 7f4092ffd700 1 -- 172.21.2.102:0/1036145868 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4098001250 con 0x7f40a0193270 2021-02-10T11:12:36.957 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:36.958 7f4092ffd700 1 --2- 172.21.2.102:0/1036145868 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f40880186c0 0x7f408801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:12:36.957 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:36.958 7f4092ffd700 1 -- 172.21.2.102:0/1036145868 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f4098032950 con 0x7f40a0193270 2021-02-10T11:12:36.957 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:36.958 7f40a6386700 1 --2- 172.21.2.102:0/1036145868 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f40880186c0 0x7f408801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:12:36.957 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:36.958 7f40a6386700 1 --2- 172.21.2.102:0/1036145868 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f40880186c0 0x7f408801abc0 crc :-1 s=READY pgs=269 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:12:36.984 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:36.989 7f40a856a700 1 -- 172.21.2.102:0/1036145868 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40a0105af0 con 0x7f40a0193270 2021-02-10T11:12:36.987 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:36.992 7f4092ffd700 1 -- 172.21.2.102:0/1036145868 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f4098030480 con 0x7f40a0193270 2021-02-10T11:12:37.115 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:37.120 7f40a856a700 1 -- 172.21.2.102:0/1036145868 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344412", "target": ["mgr", ""]}) v1 -- 0x7f40a01920b0 con 0x7f40880186c0 2021-02-10T11:12:37.117 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:37.122 7f4092ffd700 1 -- 172.21.2.102:0/1036145868 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f4094006a20 con 0x7f40880186c0 2021-02-10T11:12:37.118 INFO:teuthology.orchestra.run.gibba002.stdout:/volumes/_nogroup/subvolume_0000000000344412/982cdf77-da7c-4f7e-85af-c65217b16930 2021-02-10T11:12:37.119 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:37.124 7f40a856a700 1 -- 172.21.2.102:0/1036145868 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f40880186c0 msgr2=0x7f408801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:12:37.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:37.124 7f40a856a700 1 --2- 172.21.2.102:0/1036145868 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f40880186c0 0x7f408801abc0 crc :-1 s=READY pgs=269 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:12:37.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:37.124 7f40a856a700 1 -- 172.21.2.102:0/1036145868 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f40a0193270 msgr2=0x7f40a0193670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:12:37.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:37.124 7f40a856a700 1 --2- 172.21.2.102:0/1036145868 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f40a0193270 0x7f40a0193670 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 rx=0x7f4098009940 tx=0x7f409800bcf0).stop 2021-02-10T11:12:37.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:37.124 7f40a856a700 1 -- 172.21.2.102:0/1036145868 shutdown_connections 2021-02-10T11:12:37.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:37.124 7f40a856a700 1 --2- 172.21.2.102:0/1036145868 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f40a0193270 0x7f40a0193670 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:12:37.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:37.124 7f40a856a700 1 --2- 172.21.2.102:0/1036145868 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f40a0190d20 0x7f40a008fc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:12:37.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:37.124 7f40a856a700 1 --2- 172.21.2.102:0/1036145868 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f40880186c0 0x7f408801abc0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:12:37.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:37.124 7f40a856a700 1 --2- 172.21.2.102:0/1036145868 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f40a0190920 0x7f40a008f600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:12:37.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:37.124 7f40a856a700 1 -- 172.21.2.102:0/1036145868 >> 172.21.2.102:0/1036145868 conn(0x7f40a00bf4e0 msgr2=0x7f40a00be3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:12:37.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:37.124 7f40a856a700 1 -- 172.21.2.102:0/1036145868 shutdown_connections 2021-02-10T11:12:37.121 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:37.124 7f40a856a700 1 -- 172.21.2.102:0/1036145868 wait complete. 2021-02-10T11:12:37.127 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000344412 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000344412/982cdf77-da7c-4f7e-85af-c65217b16930 2021-02-10T11:12:37.128 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:12:37.128 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:12:37.128 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:12:37.466 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:12:37.466 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:12:37.467 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344412/982cdf77-da7c-4f7e-85af-c65217b16930/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:12:37.856 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:12:37.856 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:12:37.856 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.024885 s, 42.1 MB/s 2021-02-10T11:12:37.857 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:12:37.857 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:12:37.857 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:12:38.193 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:12:38.193 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:12:38.193 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344412/982cdf77-da7c-4f7e-85af-c65217b16930/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:12:38.565 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:12:38.565 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:12:38.565 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0242849 s, 43.2 MB/s 2021-02-10T11:12:38.571 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:12:38.571 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:12:38.571 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:12:38.907 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:12:38.907 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:12:38.907 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344412/982cdf77-da7c-4f7e-85af-c65217b16930/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:12:39.283 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:12:39.283 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:12:39.283 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0230294 s, 45.5 MB/s 2021-02-10T11:12:39.284 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:12:39.284 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:12:39.284 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:12:39.621 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:12:39.621 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:12:39.621 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344412/982cdf77-da7c-4f7e-85af-c65217b16930/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:12:40.000 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:12:40.000 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:12:40.000 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0250245 s, 41.9 MB/s 2021-02-10T11:12:40.001 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:12:40.001 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:12:40.001 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:12:40.337 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:12:40.337 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:12:40.337 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344412/982cdf77-da7c-4f7e-85af-c65217b16930/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:12:40.713 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:12:40.713 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:12:40.713 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.023972 s, 43.7 MB/s 2021-02-10T11:12:40.714 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:12:40.714 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:12:40.714 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:12:41.049 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:12:41.049 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:12:41.050 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344412/982cdf77-da7c-4f7e-85af-c65217b16930/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:12:41.425 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:12:41.425 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:12:41.425 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0241008 s, 43.5 MB/s 2021-02-10T11:12:41.426 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:12:41.426 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:12:41.427 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:12:41.762 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:12:41.762 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:12:41.762 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344412/982cdf77-da7c-4f7e-85af-c65217b16930/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:12:42.139 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:12:42.139 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:12:42.139 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0249478 s, 42.0 MB/s 2021-02-10T11:12:42.140 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:12:42.141 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:12:42.141 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:12:42.476 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:12:42.476 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:12:42.476 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344412/982cdf77-da7c-4f7e-85af-c65217b16930/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:12:42.851 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:12:42.851 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:12:42.851 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0235457 s, 44.5 MB/s 2021-02-10T11:12:42.852 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:12:42.852 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:12:42.852 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:12:43.188 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:12:43.188 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:12:43.188 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344412/982cdf77-da7c-4f7e-85af-c65217b16930/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:12:43.561 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:12:43.561 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:12:43.561 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.023483 s, 44.7 MB/s 2021-02-10T11:12:43.562 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:12:43.562 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:12:43.562 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:12:43.898 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:12:43.898 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:12:43.898 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344412/982cdf77-da7c-4f7e-85af-c65217b16930/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:12:44.274 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:12:44.274 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:12:44.275 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0252668 s, 41.5 MB/s 2021-02-10T11:12:44.275 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000344413 2021-02-10T11:12:44.709 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:44.696 7fcbc7de5700 1 Processor -- start 2021-02-10T11:12:44.709 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:44.697 7fcbc7de5700 1 -- start start 2021-02-10T11:12:44.709 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:44.697 7fcbc7de5700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcbc01b0470 0x7fcbc01b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:12:44.709 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:44.697 7fcbc7de5700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fcbc00f20e0 0x7fcbc00f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:12:44.709 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:44.697 7fcbc7de5700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbc00f0b50 con 0x7fcbc00ef610 2021-02-10T11:12:44.709 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:44.697 7fcbc7de5700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcbc00f0820 con 0x7fcbc00f20e0 2021-02-10T11:12:44.709 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:44.697 7fcbc7de5700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbc00f24e0 con 0x7fcbc01b0470 2021-02-10T11:12:44.710 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:44.697 7fcbc6402700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fcbc00f20e0 0x7fcbc00f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:12:44.710 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:44.697 7fcbc6402700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fcbc00f20e0 0x7fcbc00f0200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:52326/0 (socket says 172.21.2.102:52326) 2021-02-10T11:12:44.710 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:44.697 7fcbc6402700 1 -- 172.21.2.102:0/1788175260 learned_addr learned my addr 172.21.2.102:0/1788175260 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:12:44.710 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:44.698 7fcbc6402700 1 -- 172.21.2.102:0/1788175260 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcbc01b0470 msgr2=0x7fcbc01b0870 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:12:44.710 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:44.698 7fcbc6402700 1 --2- 172.21.2.102:0/1788175260 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcbc01b0470 0x7fcbc01b0870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:12:44.710 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:44.698 7fcbc6402700 1 -- 172.21.2.102:0/1788175260 >> v1:172.21.2.113:6789/0 conn(0x7fcbc00ef610 legacy=0x7fcbc00efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:12:44.710 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:44.698 7fcbc6402700 1 -- 172.21.2.102:0/1788175260 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbc00e52e0 con 0x7fcbc00f20e0 2021-02-10T11:12:44.710 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:44.698 7fcbc6402700 1 --2- 172.21.2.102:0/1788175260 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcbc00f20e0 0x7fcbc00f0200 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 rx=0x7fcbb8006f00 tx=0x7fcbb8009f10).ready entity=mon.0 client_cookie=55157994156d3ea7 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:12:44.710 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:44.698 7fcbc4bff700 1 -- 172.21.2.102:0/1788175260 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcbb800b3b0 con 0x7fcbc00f20e0 2021-02-10T11:12:44.710 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:44.698 7fcbc4bff700 1 -- 172.21.2.102:0/1788175260 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbb800b630 con 0x7fcbc00f20e0 2021-02-10T11:12:44.710 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:44.698 7fcbc4bff700 1 -- 172.21.2.102:0/1788175260 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcbb800bb50 con 0x7fcbc00f20e0 2021-02-10T11:12:44.711 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:44.698 7fcbc7de5700 1 -- 172.21.2.102:0/1788175260 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcbc00f20e0 msgr2=0x7fcbc00f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:12:44.711 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:44.698 7fcbc7de5700 1 --2- 172.21.2.102:0/1788175260 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcbc00f20e0 0x7fcbc00f0200 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 rx=0x7fcbb8006f00 tx=0x7fcbb8009f10).stop 2021-02-10T11:12:44.711 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:44.699 7fcbc7de5700 1 -- 172.21.2.102:0/1788175260 shutdown_connections 2021-02-10T11:12:44.711 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:44.699 7fcbc7de5700 1 --2- 172.21.2.102:0/1788175260 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcbc00f20e0 0x7fcbc00f0200 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:12:44.711 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:44.699 7fcbc7de5700 1 --2- 172.21.2.102:0/1788175260 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcbc01b0470 0x7fcbc01b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:12:44.711 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:44.699 7fcbc7de5700 1 -- 172.21.2.102:0/1788175260 >> 172.21.2.102:0/1788175260 conn(0x7fcbc00ce3e0 msgr2=0x7fcbc00e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:12:44.711 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:44.699 7fcbc7de5700 1 -- 172.21.2.102:0/1788175260 shutdown_connections 2021-02-10T11:12:44.711 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:44.699 7fcbc7de5700 1 -- 172.21.2.102:0/1788175260 wait complete. 2021-02-10T11:12:44.711 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:44.714 7fcbc7de5700 1 Processor -- start 2021-02-10T11:12:44.711 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:44.714 7fcbc7de5700 1 -- start start 2021-02-10T11:12:44.713 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:44.714 7fcbc7de5700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcbc00c36a0 0x7fcbc00c2b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:12:44.713 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:44.714 7fcbc7de5700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fcbc00c3190 0x7fcbc00b1d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:12:44.713 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:44.714 7fcbc7de5700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcbc00b2370 0x7fcbc00b2d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:12:44.713 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:44.714 7fcbc7de5700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbc00b27c0 con 0x7fcbc00b2370 2021-02-10T11:12:44.713 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:44.714 7fcbc7de5700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbc00b3360 con 0x7fcbc00c3190 2021-02-10T11:12:44.713 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:44.714 7fcbc7de5700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbc00b3540 con 0x7fcbc00c36a0 2021-02-10T11:12:44.713 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:44.714 7fcbc6402700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcbc00b2370 0x7fcbc00b2d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:12:44.713 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:44.714 7fcbc6402700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcbc00b2370 0x7fcbc00b2d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:52330/0 (socket says 172.21.2.102:52330) 2021-02-10T11:12:44.713 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:44.714 7fcbc6402700 1 -- 172.21.2.102:0/1761465100 learned_addr learned my addr 172.21.2.102:0/1761465100 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:12:44.714 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:44.715 7fcbc5c01700 1 --2- 172.21.2.102:0/1761465100 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcbc00c36a0 0x7fcbc00c2b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:12:44.714 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:44.715 7fcbc5400700 1 --2- 172.21.2.102:0/1761465100 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fcbc00c3190 0x7fcbc00b1d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:12:44.714 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:44.715 7fcbc6402700 1 -- 172.21.2.102:0/1761465100 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcbc00c36a0 msgr2=0x7fcbc00c2b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:12:44.714 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:44.715 7fcbc6402700 1 --2- 172.21.2.102:0/1761465100 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcbc00c36a0 0x7fcbc00c2b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:12:44.714 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:44.715 7fcbc6402700 1 -- 172.21.2.102:0/1761465100 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fcbc00c3190 msgr2=0x7fcbc00b1d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:12:44.714 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:44.715 7fcbc6402700 1 --2- 172.21.2.102:0/1761465100 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fcbc00c3190 0x7fcbc00b1d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:12:44.714 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:44.715 7fcbc6402700 1 -- 172.21.2.102:0/1761465100 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbc00b3760 con 0x7fcbc00b2370 2021-02-10T11:12:44.714 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:44.715 7fcbc6402700 1 --2- 172.21.2.102:0/1761465100 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcbc00b2370 0x7fcbc00b2d40 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 rx=0x7fcbb8009390 tx=0x7fcbb800bd40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:12:44.715 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:44.715 7fcbb67fc700 1 -- 172.21.2.102:0/1761465100 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcbb8003030 con 0x7fcbc00b2370 2021-02-10T11:12:44.715 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:44.715 7fcbb67fc700 1 -- 172.21.2.102:0/1761465100 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbb8006c80 con 0x7fcbc00b2370 2021-02-10T11:12:44.715 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:44.715 7fcbb67fc700 1 -- 172.21.2.102:0/1761465100 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcbb8001480 con 0x7fcbc00b2370 2021-02-10T11:12:44.715 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:44.715 7fcbc7de5700 1 -- 172.21.2.102:0/1761465100 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbc00b3760 con 0x7fcbc00b2370 2021-02-10T11:12:44.715 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:44.715 7fcbc7de5700 1 -- 172.21.2.102:0/1761465100 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbc00b3760 con 0x7fcbc00b2370 2021-02-10T11:12:44.715 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:44.716 7fcbb67fc700 1 -- 172.21.2.102:0/1761465100 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fcbb8001270 con 0x7fcbc00b2370 2021-02-10T11:12:44.715 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:44.716 7fcbb67fc700 1 --2- 172.21.2.102:0/1761465100 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcba80186c0 0x7fcba801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:12:44.715 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:44.716 7fcbb67fc700 1 -- 172.21.2.102:0/1761465100 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fcbb8032760 con 0x7fcbc00b2370 2021-02-10T11:12:44.716 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:44.716 7fcbc5c01700 1 --2- 172.21.2.102:0/1761465100 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcba80186c0 0x7fcba801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:12:44.716 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:44.717 7fcbc5c01700 1 --2- 172.21.2.102:0/1761465100 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcba80186c0 0x7fcba801abc0 crc :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:12:44.744 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:44.747 7fcbc7de5700 1 -- 172.21.2.102:0/1761465100 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcbc00edd10 con 0x7fcbc00b2370 2021-02-10T11:12:44.744 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:44.749 7fcbb67fc700 1 -- 172.21.2.102:0/1761465100 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fcbb8003020 con 0x7fcbc00b2370 2021-02-10T11:12:44.869 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:44.874 7fcbc7de5700 1 -- 172.21.2.102:0/1761465100 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344413", "target": ["mgr", ""]}) v1 -- 0x7fcbc00edd10 con 0x7fcba80186c0 2021-02-10T11:12:44.897 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:44.902 7fcbb67fc700 1 -- 172.21.2.102:0/1761465100 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fcbb0006a30 con 0x7fcba80186c0 2021-02-10T11:12:44.903 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:44.908 7fcbc7de5700 1 -- 172.21.2.102:0/1761465100 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcba80186c0 msgr2=0x7fcba801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:12:44.903 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:44.908 7fcbc7de5700 1 --2- 172.21.2.102:0/1761465100 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcba80186c0 0x7fcba801abc0 crc :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:12:44.903 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:44.908 7fcbc7de5700 1 -- 172.21.2.102:0/1761465100 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcbc00b2370 msgr2=0x7fcbc00b2d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:12:44.903 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:44.908 7fcbc7de5700 1 --2- 172.21.2.102:0/1761465100 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcbc00b2370 0x7fcbc00b2d40 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 rx=0x7fcbb8009390 tx=0x7fcbb800bd40).stop 2021-02-10T11:12:44.903 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:44.908 7fcbc7de5700 1 -- 172.21.2.102:0/1761465100 shutdown_connections 2021-02-10T11:12:44.904 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:44.908 7fcbc7de5700 1 --2- 172.21.2.102:0/1761465100 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcbc00b2370 0x7fcbc00b2d40 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:12:44.904 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:44.908 7fcbc7de5700 1 --2- 172.21.2.102:0/1761465100 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcba80186c0 0x7fcba801abc0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:12:44.904 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:44.908 7fcbc7de5700 1 --2- 172.21.2.102:0/1761465100 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fcbc00c3190 0x7fcbc00b1d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:12:44.904 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:44.908 7fcbc7de5700 1 --2- 172.21.2.102:0/1761465100 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcbc00c36a0 0x7fcbc00c2b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:12:44.904 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:44.908 7fcbc7de5700 1 -- 172.21.2.102:0/1761465100 >> 172.21.2.102:0/1761465100 conn(0x7fcbc009f630 msgr2=0x7fcbc00a0130 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:12:44.904 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:44.908 7fcbc7de5700 1 -- 172.21.2.102:0/1761465100 shutdown_connections 2021-02-10T11:12:44.904 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:44.908 7fcbc7de5700 1 -- 172.21.2.102:0/1761465100 wait complete. 2021-02-10T11:12:44.911 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000344413 2021-02-10T11:12:45.344 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:45.331 7f00200be700 1 Processor -- start 2021-02-10T11:12:45.344 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:45.331 7f00200be700 1 -- start start 2021-02-10T11:12:45.344 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:45.332 7f00200be700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f00181b0870 0x7f00180f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:12:45.344 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:45.332 7f00200be700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f00180efe00 0x7f00180f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:12:45.345 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:45.332 7f00200be700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00181b0c70 con 0x7f00181b0470 2021-02-10T11:12:45.345 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:45.332 7f00200be700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00180f0b50 con 0x7f00180efe00 2021-02-10T11:12:45.345 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:45.332 7f00200be700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00180f0820 con 0x7f00181b0870 2021-02-10T11:12:45.345 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:45.333 7f001e6db700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f00180efe00 0x7f00180f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:12:45.345 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:45.333 7f001e6db700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f00180efe00 0x7f00180f0200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.113:3300/0 says I am v2:172.21.2.102:55770/0 (socket says 172.21.2.102:55770) 2021-02-10T11:12:45.345 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:45.333 7f001e6db700 1 -- 172.21.2.102:0/63118522 learned_addr learned my addr 172.21.2.102:0/63118522 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:12:45.345 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:45.333 7f001d6d9700 1 --2- 172.21.2.102:0/63118522 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f00181b0870 0x7f00180f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:12:45.345 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:45.333 7f001deda700 1 --1- 172.21.2.102:0/63118522 >> v1:172.21.2.113:6789/0 conn(0x7f00181b0470 0x7f00180ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.113:6789/0 says I am v1:172.21.2.102:51884/0 (socket says 172.21.2.102:51884) 2021-02-10T11:12:45.345 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:45.333 7f001e6db700 1 -- 172.21.2.102:0/63118522 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f00181b0870 msgr2=0x7f00180f20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:12:45.345 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:45.333 7f001e6db700 1 --2- 172.21.2.102:0/63118522 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f00181b0870 0x7f00180f20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:12:45.345 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:45.333 7f001e6db700 1 -- 172.21.2.102:0/63118522 >> v1:172.21.2.113:6789/0 conn(0x7f00181b0470 legacy=0x7f00180ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:12:45.345 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:45.333 7f001e6db700 1 -- 172.21.2.102:0/63118522 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00180e52e0 con 0x7f00180efe00 2021-02-10T11:12:45.345 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:45.334 7f001e6db700 1 --2- 172.21.2.102:0/63118522 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f00180efe00 0x7f00180f0200 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f0010006aa0 tx=0x7f0010009ea0).ready entity=mon.1 client_cookie=cc03ddc3f39cc86a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:12:45.346 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:45.334 7f001ced8700 1 -- 172.21.2.102:0/63118522 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f001000b510 con 0x7f00180efe00 2021-02-10T11:12:45.346 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:45.334 7f001ced8700 1 -- 172.21.2.102:0/63118522 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0010003030 con 0x7f00180efe00 2021-02-10T11:12:45.346 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:45.334 7f001ced8700 1 -- 172.21.2.102:0/63118522 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f001000ba10 con 0x7f00180efe00 2021-02-10T11:12:45.346 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:45.334 7f00200be700 1 -- 172.21.2.102:0/63118522 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f00180efe00 msgr2=0x7f00180f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:12:45.346 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:45.334 7f00200be700 1 --2- 172.21.2.102:0/63118522 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f00180efe00 0x7f00180f0200 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f0010006aa0 tx=0x7f0010009ea0).stop 2021-02-10T11:12:45.346 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:45.335 7f00200be700 1 -- 172.21.2.102:0/63118522 shutdown_connections 2021-02-10T11:12:45.346 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:45.335 7f00200be700 1 --2- 172.21.2.102:0/63118522 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f00181b0870 0x7f00180f20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:12:45.346 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:45.335 7f00200be700 1 --2- 172.21.2.102:0/63118522 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f00180efe00 0x7f00180f0200 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:12:45.346 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:45.335 7f00200be700 1 -- 172.21.2.102:0/63118522 >> 172.21.2.102:0/63118522 conn(0x7f00180ce3e0 msgr2=0x7f00180e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:12:45.346 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:45.335 7f00200be700 1 -- 172.21.2.102:0/63118522 shutdown_connections 2021-02-10T11:12:45.346 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:45.335 7f00200be700 1 -- 172.21.2.102:0/63118522 wait complete. 2021-02-10T11:12:45.348 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:45.349 7f00200be700 1 Processor -- start 2021-02-10T11:12:45.348 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:45.349 7f00200be700 1 -- start start 2021-02-10T11:12:45.348 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:45.350 7f00200be700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f00181a48b0 0x7f00180bf930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:12:45.348 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:45.350 7f00200be700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f00180a73c0 0x7f00180bff50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:12:45.348 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:45.350 7f00200be700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f00180a5260 0x7f00180a5660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:12:45.348 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:45.350 7f00200be700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00180a6ec0 con 0x7f00181a48b0 2021-02-10T11:12:45.349 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:45.350 7f00200be700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00180bc010 con 0x7f00180a73c0 2021-02-10T11:12:45.349 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:45.350 7f00200be700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00180c0570 con 0x7f00180a5260 2021-02-10T11:12:45.349 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:45.350 7f001deda700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f00181a48b0 0x7f00180bf930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:12:45.349 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:45.350 7f001deda700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f00181a48b0 0x7f00180bf930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:52344/0 (socket says 172.21.2.102:52344) 2021-02-10T11:12:45.349 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:45.350 7f001deda700 1 -- 172.21.2.102:0/3379730785 learned_addr learned my addr 172.21.2.102:0/3379730785 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:12:45.349 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:45.351 7f001deda700 1 -- 172.21.2.102:0/3379730785 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f00180a5260 msgr2=0x7f00180a5660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:12:45.349 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:45.351 7f001deda700 1 --2- 172.21.2.102:0/3379730785 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f00180a5260 0x7f00180a5660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:12:45.349 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:45.351 7f001deda700 1 -- 172.21.2.102:0/3379730785 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f00180a73c0 msgr2=0x7f00180bff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:12:45.349 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:45.351 7f001deda700 1 --2- 172.21.2.102:0/3379730785 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f00180a73c0 0x7f00180bff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:12:45.349 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:45.351 7f001deda700 1 -- 172.21.2.102:0/3379730785 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00180a5c80 con 0x7f00181a48b0 2021-02-10T11:12:45.349 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:45.351 7f001deda700 1 --2- 172.21.2.102:0/3379730785 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f00181a48b0 0x7f00180bf930 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 rx=0x7f0014007e80 tx=0x7f001400a900).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:12:45.349 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:45.351 7f000a7fc700 1 -- 172.21.2.102:0/3379730785 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f001400acf0 con 0x7f00181a48b0 2021-02-10T11:12:45.349 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:45.351 7f00200be700 1 -- 172.21.2.102:0/3379730785 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00180a5c80 con 0x7f00181a48b0 2021-02-10T11:12:45.350 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:45.351 7f000a7fc700 1 -- 172.21.2.102:0/3379730785 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f001400acf0 con 0x7f00181a48b0 2021-02-10T11:12:45.350 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:45.351 7f000a7fc700 1 -- 172.21.2.102:0/3379730785 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0014004320 con 0x7f00181a48b0 2021-02-10T11:12:45.350 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:45.351 7f00200be700 1 -- 172.21.2.102:0/3379730785 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00180a5c80 con 0x7f00181a48b0 2021-02-10T11:12:45.350 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:45.352 7f000a7fc700 1 -- 172.21.2.102:0/3379730785 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f0014004320 con 0x7f00181a48b0 2021-02-10T11:12:45.350 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:45.352 7f000a7fc700 1 --2- 172.21.2.102:0/3379730785 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0000018730 0x7f000001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:12:45.350 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:45.352 7f000a7fc700 1 -- 172.21.2.102:0/3379730785 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f00140327c0 con 0x7f00181a48b0 2021-02-10T11:12:45.350 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:45.352 7f001d6d9700 1 --2- 172.21.2.102:0/3379730785 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0000018730 0x7f000001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:12:45.350 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:45.352 7f001d6d9700 1 --2- 172.21.2.102:0/3379730785 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0000018730 0x7f000001ac10 crc :-1 s=READY pgs=271 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:12:45.377 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:45.381 7f00200be700 1 -- 172.21.2.102:0/3379730785 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0018105ae0 con 0x7f00181a48b0 2021-02-10T11:12:45.379 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:45.384 7f000a7fc700 1 -- 172.21.2.102:0/3379730785 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f0014000c10 con 0x7f00181a48b0 2021-02-10T11:12:45.508 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:45.513 7f00200be700 1 -- 172.21.2.102:0/3379730785 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344413", "target": ["mgr", ""]}) v1 -- 0x7f0018105ae0 con 0x7f0000018730 2021-02-10T11:12:45.511 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:45.516 7f000a7fc700 1 -- 172.21.2.102:0/3379730785 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f000c006e70 con 0x7f0000018730 2021-02-10T11:12:45.515 INFO:teuthology.orchestra.run.gibba002.stdout:/volumes/_nogroup/subvolume_0000000000344413/08145ec3-13ed-4a03-89e4-4c0d6ba9f31e 2021-02-10T11:12:45.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:45.521 7f00200be700 1 -- 172.21.2.102:0/3379730785 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0000018730 msgr2=0x7f000001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:12:45.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:45.521 7f00200be700 1 --2- 172.21.2.102:0/3379730785 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0000018730 0x7f000001ac10 crc :-1 s=READY pgs=271 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:12:45.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:45.522 7f00200be700 1 -- 172.21.2.102:0/3379730785 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f00181a48b0 msgr2=0x7f00180bf930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:12:45.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:45.522 7f00200be700 1 --2- 172.21.2.102:0/3379730785 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f00181a48b0 0x7f00180bf930 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 rx=0x7f0014007e80 tx=0x7f001400a900).stop 2021-02-10T11:12:45.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:45.522 7f00200be700 1 -- 172.21.2.102:0/3379730785 shutdown_connections 2021-02-10T11:12:45.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:45.522 7f00200be700 1 --2- 172.21.2.102:0/3379730785 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f00180a5260 0x7f00180a5660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:12:45.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:45.522 7f00200be700 1 --2- 172.21.2.102:0/3379730785 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0000018730 0x7f000001ac10 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:12:45.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:45.522 7f00200be700 1 --2- 172.21.2.102:0/3379730785 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f00180a73c0 0x7f00180bff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:12:45.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:45.522 7f00200be700 1 --2- 172.21.2.102:0/3379730785 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f00181a48b0 0x7f00180bf930 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:12:45.518 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:45.522 7f00200be700 1 -- 172.21.2.102:0/3379730785 >> 172.21.2.102:0/3379730785 conn(0x7f0018109e80 msgr2=0x7f00180af060 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:12:45.518 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:45.522 7f00200be700 1 -- 172.21.2.102:0/3379730785 shutdown_connections 2021-02-10T11:12:45.518 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:45.522 7f00200be700 1 -- 172.21.2.102:0/3379730785 wait complete. 2021-02-10T11:12:45.525 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000344413 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000344413/08145ec3-13ed-4a03-89e4-4c0d6ba9f31e 2021-02-10T11:12:45.525 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:12:45.525 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:12:45.525 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:12:45.862 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:12:45.862 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:12:45.862 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344413/08145ec3-13ed-4a03-89e4-4c0d6ba9f31e/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:12:46.243 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:12:46.243 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:12:46.244 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0228519 s, 45.9 MB/s 2021-02-10T11:12:46.244 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:12:46.244 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:12:46.245 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:12:46.580 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:12:46.580 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:12:46.581 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344413/08145ec3-13ed-4a03-89e4-4c0d6ba9f31e/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:12:46.956 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:12:46.956 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:12:46.956 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0240856 s, 43.5 MB/s 2021-02-10T11:12:46.957 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:12:46.957 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:12:46.957 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:12:47.292 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:12:47.292 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:12:47.293 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344413/08145ec3-13ed-4a03-89e4-4c0d6ba9f31e/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:12:47.670 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:12:47.670 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:12:47.670 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0241147 s, 43.5 MB/s 2021-02-10T11:12:47.671 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:12:47.672 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:12:47.672 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:12:48.006 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:12:48.007 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:12:48.007 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344413/08145ec3-13ed-4a03-89e4-4c0d6ba9f31e/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:12:48.379 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:12:48.379 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:12:48.379 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0239768 s, 43.7 MB/s 2021-02-10T11:12:48.380 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:12:48.380 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:12:48.380 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:12:48.716 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:12:48.716 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:12:48.716 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344413/08145ec3-13ed-4a03-89e4-4c0d6ba9f31e/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:12:49.091 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:12:49.091 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:12:49.091 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0232663 s, 45.1 MB/s 2021-02-10T11:12:49.092 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:12:49.092 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:12:49.092 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:12:49.428 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:12:49.428 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:12:49.429 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344413/08145ec3-13ed-4a03-89e4-4c0d6ba9f31e/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:12:49.804 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:12:49.804 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:12:49.804 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0244019 s, 43.0 MB/s 2021-02-10T11:12:49.805 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:12:49.805 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:12:49.805 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:12:50.141 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:12:50.141 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:12:50.141 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344413/08145ec3-13ed-4a03-89e4-4c0d6ba9f31e/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:12:50.514 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:12:50.515 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:12:50.515 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0218786 s, 47.9 MB/s 2021-02-10T11:12:50.516 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:12:50.516 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:12:50.516 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:12:50.851 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:12:50.852 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:12:50.852 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344413/08145ec3-13ed-4a03-89e4-4c0d6ba9f31e/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:12:51.004 DEBUG:teuthology.orchestra.run.gibba002:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:12:51.008 DEBUG:teuthology.orchestra.run.gibba013:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:12:51.229 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:12:51.229 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:12:51.229 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0247243 s, 42.4 MB/s 2021-02-10T11:12:51.230 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:12:51.230 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:12:51.230 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:12:51.566 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:12:51.566 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:12:51.566 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344413/08145ec3-13ed-4a03-89e4-4c0d6ba9f31e/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:12:51.941 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:12:51.941 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:12:51.942 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0236034 s, 44.4 MB/s 2021-02-10T11:12:51.943 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:12:51.943 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:12:51.943 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:12:52.279 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:12:52.279 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:12:52.279 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344413/08145ec3-13ed-4a03-89e4-4c0d6ba9f31e/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:12:52.654 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:12:52.655 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:12:52.655 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.023944 s, 43.8 MB/s 2021-02-10T11:12:52.656 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000344414 2021-02-10T11:12:53.087 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:53.075 7fa1eb362700 1 Processor -- start 2021-02-10T11:12:53.087 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:53.075 7fa1eb362700 1 -- start start 2021-02-10T11:12:53.087 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:53.075 7fa1eb362700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fa1e41b0470 0x7fa1e41b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:12:53.088 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:53.075 7fa1eb362700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa1e40f20e0 0x7fa1e40f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:12:53.088 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:53.075 7fa1eb362700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1e40f0b50 con 0x7fa1e40ef610 2021-02-10T11:12:53.088 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:53.075 7fa1eb362700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1e40f0820 con 0x7fa1e41b0470 2021-02-10T11:12:53.088 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:53.075 7fa1eb362700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1e40f24e0 con 0x7fa1e40f20e0 2021-02-10T11:12:53.088 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:53.076 7fa1e917e700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fa1e41b0470 0x7fa1e41b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:12:53.088 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:53.076 7fa1e897d700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fa1e40ef610 0x7fa1e40efa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:44574/0 (socket says 172.21.2.102:44574) 2021-02-10T11:12:53.088 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:53.076 7fa1e897d700 1 -- 172.21.2.102:0/3743583862 learned_addr learned my addr 172.21.2.102:0/3743583862 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:12:53.088 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:53.076 7fa1d7fff700 1 -- 172.21.2.102:0/3743583862 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 286735665 0 0) 0x7fa1d80010b0 con 0x7fa1e40ef610 2021-02-10T11:12:53.088 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:53.076 7fa1d7fff700 1 -- 172.21.2.102:0/3743583862 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1d0002fe0 con 0x7fa1e40ef610 2021-02-10T11:12:53.088 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:53.076 7fa1e917e700 1 -- 172.21.2.102:0/3743583862 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa1e40f20e0 msgr2=0x7fa1e40f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:12:53.088 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:53.076 7fa1e917e700 1 --2- 172.21.2.102:0/3743583862 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa1e40f20e0 0x7fa1e40f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:12:53.088 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:53.076 7fa1e917e700 1 -- 172.21.2.102:0/3743583862 >> v1:172.21.2.102:6789/0 conn(0x7fa1e40ef610 legacy=0x7fa1e40efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:12:53.089 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:53.076 7fa1e917e700 1 -- 172.21.2.102:0/3743583862 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1e40e52e0 con 0x7fa1e41b0470 2021-02-10T11:12:53.089 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:53.076 7fa1e917e700 1 --2- 172.21.2.102:0/3743583862 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa1e41b0470 0x7fa1e41b0870 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 rx=0x7fa1e0006650 tx=0x7fa1e0009a80).ready entity=mon.0 client_cookie=479dc3b8eec38e89 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:12:53.089 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:53.077 7fa1d7fff700 1 -- 172.21.2.102:0/3743583862 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa1e000c0c0 con 0x7fa1e41b0470 2021-02-10T11:12:53.089 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:53.077 7fa1d7fff700 1 -- 172.21.2.102:0/3743583862 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1e000c340 con 0x7fa1e41b0470 2021-02-10T11:12:53.089 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:53.077 7fa1d7fff700 1 -- 172.21.2.102:0/3743583862 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa1e000c860 con 0x7fa1e41b0470 2021-02-10T11:12:53.089 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:53.077 7fa1eb362700 1 -- 172.21.2.102:0/3743583862 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa1e41b0470 msgr2=0x7fa1e41b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:12:53.089 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:53.077 7fa1eb362700 1 --2- 172.21.2.102:0/3743583862 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa1e41b0470 0x7fa1e41b0870 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 rx=0x7fa1e0006650 tx=0x7fa1e0009a80).stop 2021-02-10T11:12:53.089 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:53.078 7fa1eb362700 1 -- 172.21.2.102:0/3743583862 shutdown_connections 2021-02-10T11:12:53.089 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:53.078 7fa1eb362700 1 --2- 172.21.2.102:0/3743583862 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa1e40f20e0 0x7fa1e40f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:12:53.089 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:53.078 7fa1eb362700 1 --2- 172.21.2.102:0/3743583862 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa1e41b0470 0x7fa1e41b0870 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:12:53.089 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:53.078 7fa1eb362700 1 -- 172.21.2.102:0/3743583862 >> 172.21.2.102:0/3743583862 conn(0x7fa1e40ce3e0 msgr2=0x7fa1e40e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:12:53.089 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:53.078 7fa1eb362700 1 -- 172.21.2.102:0/3743583862 shutdown_connections 2021-02-10T11:12:53.090 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:53.078 7fa1eb362700 1 -- 172.21.2.102:0/3743583862 wait complete. 2021-02-10T11:12:53.091 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:53.092 7fa1eb362700 1 Processor -- start 2021-02-10T11:12:53.091 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:53.093 7fa1eb362700 1 -- start start 2021-02-10T11:12:53.091 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:53.093 7fa1eb362700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa1e409f140 0x7fa1e40c1940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:12:53.091 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:53.093 7fa1eb362700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa1e40c1f60 0x7fa1e409e300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:12:53.091 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:53.093 7fa1eb362700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa1e409e920 0x7fa1e40c2960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:12:53.092 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:53.093 7fa1eb362700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1e40c23e0 con 0x7fa1e40c1f60 2021-02-10T11:12:53.092 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:53.093 7fa1eb362700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1e409ed20 con 0x7fa1e409f140 2021-02-10T11:12:53.092 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:53.093 7fa1eb362700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1e4105d10 con 0x7fa1e409e920 2021-02-10T11:12:53.092 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:53.093 7fa1e897d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa1e40c1f60 0x7fa1e409e300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:12:53.092 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:53.093 7fa1e897d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa1e40c1f60 0x7fa1e409e300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:52354/0 (socket says 172.21.2.102:52354) 2021-02-10T11:12:53.092 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:53.093 7fa1e897d700 1 -- 172.21.2.102:0/964539445 learned_addr learned my addr 172.21.2.102:0/964539445 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:12:53.092 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:53.093 7fa1e897d700 1 -- 172.21.2.102:0/964539445 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa1e409e920 msgr2=0x7fa1e40c2960 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:12:53.092 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:53.093 7fa1e897d700 1 --2- 172.21.2.102:0/964539445 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa1e409e920 0x7fa1e40c2960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:12:53.092 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:53.093 7fa1e897d700 1 -- 172.21.2.102:0/964539445 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa1e409f140 msgr2=0x7fa1e40c1940 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:12:53.092 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:53.093 7fa1e897d700 1 --2- 172.21.2.102:0/964539445 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa1e409f140 0x7fa1e40c1940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:12:53.092 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:53.094 7fa1e897d700 1 -- 172.21.2.102:0/964539445 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1e41a4910 con 0x7fa1e40c1f60 2021-02-10T11:12:53.092 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:53.094 7fa1e897d700 1 --2- 172.21.2.102:0/964539445 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa1e40c1f60 0x7fa1e409e300 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 rx=0x7fa1d8001c60 tx=0x7fa1d800a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:12:53.093 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:53.094 7fa1d5ffb700 1 -- 172.21.2.102:0/964539445 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa1d800acc0 con 0x7fa1e40c1f60 2021-02-10T11:12:53.093 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:53.094 7fa1d5ffb700 1 -- 172.21.2.102:0/964539445 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1d8004140 con 0x7fa1e40c1f60 2021-02-10T11:12:53.093 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:53.094 7fa1d5ffb700 1 -- 172.21.2.102:0/964539445 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa1d8004660 con 0x7fa1e40c1f60 2021-02-10T11:12:53.093 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:53.094 7fa1eb362700 1 -- 172.21.2.102:0/964539445 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1e41a4910 con 0x7fa1e40c1f60 2021-02-10T11:12:53.093 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:53.094 7fa1eb362700 1 -- 172.21.2.102:0/964539445 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1e40c3270 con 0x7fa1e40c1f60 2021-02-10T11:12:53.093 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:53.095 7fa1d5ffb700 1 -- 172.21.2.102:0/964539445 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa1d8004320 con 0x7fa1e40c1f60 2021-02-10T11:12:53.093 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:53.095 7fa1d5ffb700 1 --2- 172.21.2.102:0/964539445 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa1c40186c0 0x7fa1c401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:12:53.093 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:53.095 7fa1d5ffb700 1 -- 172.21.2.102:0/964539445 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fa1d8032530 con 0x7fa1e40c1f60 2021-02-10T11:12:53.093 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:53.095 7fa1e917e700 1 --2- 172.21.2.102:0/964539445 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa1c40186c0 0x7fa1c401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:12:53.093 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:53.095 7fa1e917e700 1 --2- 172.21.2.102:0/964539445 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa1c40186c0 0x7fa1c401abc0 crc :-1 s=READY pgs=272 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:12:53.121 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:53.125 7fa1eb362700 1 -- 172.21.2.102:0/964539445 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1e40f0b50 con 0x7fa1e40c1f60 2021-02-10T11:12:53.123 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:53.128 7fa1d5ffb700 1 -- 172.21.2.102:0/964539445 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fa1d8000960 con 0x7fa1e40c1f60 2021-02-10T11:12:53.249 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:53.254 7fa1eb362700 1 -- 172.21.2.102:0/964539445 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344414", "target": ["mgr", ""]}) v1 -- 0x7fa1e40c3270 con 0x7fa1c40186c0 2021-02-10T11:12:53.274 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:53.279 7fa1d5ffb700 1 -- 172.21.2.102:0/964539445 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fa1e0000d10 con 0x7fa1c40186c0 2021-02-10T11:12:53.282 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:53.287 7fa1eb362700 1 -- 172.21.2.102:0/964539445 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa1c40186c0 msgr2=0x7fa1c401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:12:53.283 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:53.287 7fa1eb362700 1 --2- 172.21.2.102:0/964539445 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa1c40186c0 0x7fa1c401abc0 crc :-1 s=READY pgs=272 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:12:53.283 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:53.287 7fa1eb362700 1 -- 172.21.2.102:0/964539445 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa1e40c1f60 msgr2=0x7fa1e409e300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:12:53.283 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:53.287 7fa1eb362700 1 --2- 172.21.2.102:0/964539445 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa1e40c1f60 0x7fa1e409e300 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 rx=0x7fa1d8001c60 tx=0x7fa1d800a8d0).stop 2021-02-10T11:12:53.283 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:53.287 7fa1eb362700 1 -- 172.21.2.102:0/964539445 shutdown_connections 2021-02-10T11:12:53.283 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:53.287 7fa1eb362700 1 --2- 172.21.2.102:0/964539445 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa1e409e920 0x7fa1e40c2960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:12:53.283 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:53.287 7fa1eb362700 1 --2- 172.21.2.102:0/964539445 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa1e40c1f60 0x7fa1e409e300 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:12:53.283 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:53.287 7fa1eb362700 1 --2- 172.21.2.102:0/964539445 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa1c40186c0 0x7fa1c401abc0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:12:53.283 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:53.287 7fa1eb362700 1 --2- 172.21.2.102:0/964539445 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa1e409f140 0x7fa1e40c1940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:12:53.283 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:53.287 7fa1eb362700 1 -- 172.21.2.102:0/964539445 >> 172.21.2.102:0/964539445 conn(0x7fa1e409fc10 msgr2=0x7fa1e40b3250 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:12:53.283 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:53.287 7fa1eb362700 1 -- 172.21.2.102:0/964539445 shutdown_connections 2021-02-10T11:12:53.283 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:53.288 7fa1eb362700 1 -- 172.21.2.102:0/964539445 wait complete. 2021-02-10T11:12:53.291 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000344414 2021-02-10T11:12:53.725 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:53.712 7f5e91e17700 1 Processor -- start 2021-02-10T11:12:53.725 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:53.712 7f5e91e17700 1 -- start start 2021-02-10T11:12:53.725 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:53.713 7f5e91e17700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f5e8c1b0470 0x7f5e8c1b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:12:53.725 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:53.713 7f5e91e17700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5e8c0ef610 0x7f5e8c0efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:12:53.725 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:53.713 7f5e91e17700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e8c0f0b50 con 0x7f5e8c0f0030 2021-02-10T11:12:53.725 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:53.713 7f5e91e17700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e8c0e52e0 con 0x7f5e8c1b0470 2021-02-10T11:12:53.725 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:53.713 7f5e91e17700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e8c0e54d0 con 0x7f5e8c0ef610 2021-02-10T11:12:53.725 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:53.714 7f5e8b7fe700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f5e8c1b0470 0x7f5e8c1b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:12:53.725 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:53.714 7f5e8b7fe700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f5e8c1b0470 0x7f5e8c1b0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.113:3300/0 says I am v2:172.21.2.102:55792/0 (socket says 172.21.2.102:55792) 2021-02-10T11:12:53.726 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:53.714 7f5e8b7fe700 1 -- 172.21.2.102:0/1330200689 learned_addr learned my addr 172.21.2.102:0/1330200689 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:12:53.726 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:53.714 7f5e8affd700 1 --2- 172.21.2.102:0/1330200689 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5e8c0ef610 0x7f5e8c0efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:12:53.726 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:53.714 7f5e8b7fe700 1 -- 172.21.2.102:0/1330200689 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5e8c0ef610 msgr2=0x7f5e8c0efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:12:53.726 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:53.714 7f5e8b7fe700 1 --2- 172.21.2.102:0/1330200689 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5e8c0ef610 0x7f5e8c0efa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:12:53.726 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:53.714 7f5e8b7fe700 1 -- 172.21.2.102:0/1330200689 >> v1:172.21.2.113:6789/0 conn(0x7f5e8c0f0030 legacy=0x7f5e8c0f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:12:53.726 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:53.714 7f5e8b7fe700 1 -- 172.21.2.102:0/1330200689 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e8c0f0430 con 0x7f5e8c1b0470 2021-02-10T11:12:53.726 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:53.714 7f5e8b7fe700 1 --2- 172.21.2.102:0/1330200689 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5e8c1b0470 0x7f5e8c1b0870 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f5e84006a80 tx=0x7f5e84009f00).ready entity=mon.1 client_cookie=4e6579d516ae8d95 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:12:53.726 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:53.715 7f5e8a7fc700 1 -- 172.21.2.102:0/1330200689 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5e8400b3b0 con 0x7f5e8c1b0470 2021-02-10T11:12:53.726 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:53.715 7f5e8a7fc700 1 -- 172.21.2.102:0/1330200689 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e8400b5b0 con 0x7f5e8c1b0470 2021-02-10T11:12:53.726 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:53.715 7f5e8a7fc700 1 -- 172.21.2.102:0/1330200689 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5e8400bad0 con 0x7f5e8c1b0470 2021-02-10T11:12:53.726 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:53.715 7f5e91e17700 1 -- 172.21.2.102:0/1330200689 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5e8c1b0470 msgr2=0x7f5e8c1b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:12:53.727 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:53.715 7f5e91e17700 1 --2- 172.21.2.102:0/1330200689 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5e8c1b0470 0x7f5e8c1b0870 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f5e84006a80 tx=0x7f5e84009f00).stop 2021-02-10T11:12:53.727 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:53.715 7f5e91e17700 1 -- 172.21.2.102:0/1330200689 shutdown_connections 2021-02-10T11:12:53.727 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:53.715 7f5e91e17700 1 --2- 172.21.2.102:0/1330200689 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5e8c0ef610 0x7f5e8c0efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:12:53.727 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:53.715 7f5e91e17700 1 --2- 172.21.2.102:0/1330200689 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5e8c1b0470 0x7f5e8c1b0870 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:12:53.727 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:53.715 7f5e91e17700 1 -- 172.21.2.102:0/1330200689 >> 172.21.2.102:0/1330200689 conn(0x7f5e8c0ce3e0 msgr2=0x7f5e8c0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:12:53.727 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:53.715 7f5e91e17700 1 -- 172.21.2.102:0/1330200689 shutdown_connections 2021-02-10T11:12:53.727 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:53.715 7f5e91e17700 1 -- 172.21.2.102:0/1330200689 wait complete. 2021-02-10T11:12:53.728 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:53.730 7f5e91e17700 1 Processor -- start 2021-02-10T11:12:53.729 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:53.730 7f5e91e17700 1 -- start start 2021-02-10T11:12:53.729 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:53.731 7f5e91e17700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5e8c105d10 0x7f5e8c09eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:12:53.729 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:53.731 7f5e91e17700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5e8c106110 0x7f5e8c09f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:12:53.729 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:53.731 7f5e91e17700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5e8c0b2a00 0x7f5e8c0b2e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:12:53.729 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:53.731 7f5e91e17700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e8c09e560 con 0x7f5e8c106110 2021-02-10T11:12:53.729 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:53.731 7f5e91e17700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e8c09c090 con 0x7f5e8c0b2a00 2021-02-10T11:12:53.729 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:53.731 7f5e91e17700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e8c0b3420 con 0x7f5e8c105d10 2021-02-10T11:12:53.729 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:53.731 7f5e8affd700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5e8c106110 0x7f5e8c09f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:12:53.729 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:53.731 7f5e8affd700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5e8c106110 0x7f5e8c09f100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:52366/0 (socket says 172.21.2.102:52366) 2021-02-10T11:12:53.729 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:53.731 7f5e8affd700 1 -- 172.21.2.102:0/1470359612 learned_addr learned my addr 172.21.2.102:0/1470359612 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:12:53.729 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:53.731 7f5e8bfff700 1 --2- 172.21.2.102:0/1470359612 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5e8c0b2a00 0x7f5e8c0b2e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:12:53.730 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:53.731 7f5e8affd700 1 -- 172.21.2.102:0/1470359612 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5e8c105d10 msgr2=0x7f5e8c09eae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:12:53.730 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:53.731 7f5e8affd700 1 --2- 172.21.2.102:0/1470359612 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5e8c105d10 0x7f5e8c09eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:12:53.730 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:53.731 7f5e8affd700 1 -- 172.21.2.102:0/1470359612 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5e8c0b2a00 msgr2=0x7f5e8c0b2e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:12:53.730 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:53.731 7f5e8affd700 1 --2- 172.21.2.102:0/1470359612 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5e8c0b2a00 0x7f5e8c0b2e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:12:53.730 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:53.731 7f5e8affd700 1 -- 172.21.2.102:0/1470359612 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e8c0b3640 con 0x7f5e8c106110 2021-02-10T11:12:53.730 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:53.731 7f5e8bfff700 1 --2- 172.21.2.102:0/1470359612 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5e8c0b2a00 0x7f5e8c0b2e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:12:53.730 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:53.731 7f5e8affd700 1 --2- 172.21.2.102:0/1470359612 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5e8c106110 0x7f5e8c09f100 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 rx=0x7f5e80006f60 tx=0x7f5e8000b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:12:53.730 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:53.731 7f5e73fff700 1 -- 172.21.2.102:0/1470359612 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5e8000bd60 con 0x7f5e8c106110 2021-02-10T11:12:53.730 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:53.732 7f5e73fff700 1 -- 172.21.2.102:0/1470359612 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e80000c90 con 0x7f5e8c106110 2021-02-10T11:12:53.730 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:53.732 7f5e73fff700 1 -- 172.21.2.102:0/1470359612 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5e80002220 con 0x7f5e8c106110 2021-02-10T11:12:53.730 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:53.732 7f5e91e17700 1 -- 172.21.2.102:0/1470359612 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e8c0b3640 con 0x7f5e8c106110 2021-02-10T11:12:53.730 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:53.732 7f5e91e17700 1 -- 172.21.2.102:0/1470359612 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e8c0b3640 con 0x7f5e8c106110 2021-02-10T11:12:53.731 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:53.732 7f5e73fff700 1 -- 172.21.2.102:0/1470359612 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f5e800052e0 con 0x7f5e8c106110 2021-02-10T11:12:53.731 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:53.732 7f5e73fff700 1 --2- 172.21.2.102:0/1470359612 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5e6c0186c0 0x7f5e6c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:12:53.731 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:53.732 7f5e73fff700 1 -- 172.21.2.102:0/1470359612 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f5e80032530 con 0x7f5e8c106110 2021-02-10T11:12:53.731 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:53.732 7f5e8b7fe700 1 --2- 172.21.2.102:0/1470359612 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5e6c0186c0 0x7f5e6c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:12:53.731 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:53.733 7f5e8b7fe700 1 --2- 172.21.2.102:0/1470359612 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5e6c0186c0 0x7f5e6c01abc0 crc :-1 s=READY pgs=273 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:12:53.758 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:53.763 7f5e91e17700 1 -- 172.21.2.102:0/1470359612 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e8c0b3ec0 con 0x7f5e8c106110 2021-02-10T11:12:53.760 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:53.765 7f5e73fff700 1 -- 172.21.2.102:0/1470359612 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f5e80006bb0 con 0x7f5e8c106110 2021-02-10T11:12:53.892 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:53.897 7f5e91e17700 1 -- 172.21.2.102:0/1470359612 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344414", "target": ["mgr", ""]}) v1 -- 0x7f5e8c1a16c0 con 0x7f5e6c0186c0 2021-02-10T11:12:53.895 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:53.900 7f5e73fff700 1 -- 172.21.2.102:0/1470359612 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f5e84000cc0 con 0x7f5e6c0186c0 2021-02-10T11:12:53.899 INFO:teuthology.orchestra.run.gibba002.stdout:/volumes/_nogroup/subvolume_0000000000344414/5615c559-60b4-4b99-ae07-1bcd2063d0c6 2021-02-10T11:12:53.901 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:53.906 7f5e91e17700 1 -- 172.21.2.102:0/1470359612 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5e6c0186c0 msgr2=0x7f5e6c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:12:53.901 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:53.906 7f5e91e17700 1 --2- 172.21.2.102:0/1470359612 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5e6c0186c0 0x7f5e6c01abc0 crc :-1 s=READY pgs=273 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:12:53.901 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:53.906 7f5e91e17700 1 -- 172.21.2.102:0/1470359612 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5e8c106110 msgr2=0x7f5e8c09f100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:12:53.901 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:53.906 7f5e91e17700 1 --2- 172.21.2.102:0/1470359612 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5e8c106110 0x7f5e8c09f100 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 rx=0x7f5e80006f60 tx=0x7f5e8000b970).stop 2021-02-10T11:12:53.901 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:53.906 7f5e91e17700 1 -- 172.21.2.102:0/1470359612 shutdown_connections 2021-02-10T11:12:53.901 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:53.906 7f5e91e17700 1 --2- 172.21.2.102:0/1470359612 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5e6c0186c0 0x7f5e6c01abc0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:12:53.901 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:53.906 7f5e91e17700 1 --2- 172.21.2.102:0/1470359612 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5e8c0b2a00 0x7f5e8c0b2e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:12:53.901 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:53.906 7f5e91e17700 1 --2- 172.21.2.102:0/1470359612 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5e8c106110 0x7f5e8c09f100 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:12:53.902 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:53.906 7f5e91e17700 1 --2- 172.21.2.102:0/1470359612 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5e8c105d10 0x7f5e8c09eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:12:53.902 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:53.906 7f5e91e17700 1 -- 172.21.2.102:0/1470359612 >> 172.21.2.102:0/1470359612 conn(0x7f5e8c0c36f0 msgr2=0x7f5e8c0bd640 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:12:53.902 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:53.906 7f5e91e17700 1 -- 172.21.2.102:0/1470359612 shutdown_connections 2021-02-10T11:12:53.902 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:12:53.906 7f5e91e17700 1 -- 172.21.2.102:0/1470359612 wait complete. 2021-02-10T11:12:53.909 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000344414 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000344414/5615c559-60b4-4b99-ae07-1bcd2063d0c6 2021-02-10T11:12:53.909 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:12:53.909 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:12:53.909 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:12:54.245 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:12:54.246 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:12:54.246 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344414/5615c559-60b4-4b99-ae07-1bcd2063d0c6/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:12:54.626 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:12:56.491 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:12:56.491 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0248604 s, 42.2 MB/s 2021-02-10T11:12:56.493 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:12:56.493 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:12:56.493 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:12:56.828 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:12:56.829 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:12:56.829 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344414/5615c559-60b4-4b99-ae07-1bcd2063d0c6/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:12:57.206 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:12:57.206 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:12:57.206 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0254479 s, 41.2 MB/s 2021-02-10T11:12:57.207 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:12:57.207 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:12:57.208 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:12:57.542 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:12:57.542 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:12:57.543 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344414/5615c559-60b4-4b99-ae07-1bcd2063d0c6/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:12:57.926 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:12:57.926 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:12:57.926 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.022754 s, 46.1 MB/s 2021-02-10T11:12:57.927 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:12:57.927 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:12:57.928 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:12:58.264 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:12:58.265 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:12:58.265 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344414/5615c559-60b4-4b99-ae07-1bcd2063d0c6/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:12:58.649 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:12:58.649 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:12:58.649 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0237339 s, 44.2 MB/s 2021-02-10T11:12:58.650 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:12:58.650 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:12:58.650 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:12:58.997 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:12:58.997 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:12:58.998 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344414/5615c559-60b4-4b99-ae07-1bcd2063d0c6/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:12:59.377 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:12:59.377 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:12:59.378 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0229612 s, 45.7 MB/s 2021-02-10T11:12:59.378 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:12:59.379 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:12:59.379 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:12:59.715 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:12:59.715 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:12:59.716 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344414/5615c559-60b4-4b99-ae07-1bcd2063d0c6/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:13:00.091 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:13:00.091 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:13:00.091 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0229654 s, 45.7 MB/s 2021-02-10T11:13:00.092 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:13:00.092 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:13:00.092 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:13:00.428 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:13:00.428 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:13:00.429 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344414/5615c559-60b4-4b99-ae07-1bcd2063d0c6/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:13:00.800 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:13:00.800 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:13:00.801 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0227531 s, 46.1 MB/s 2021-02-10T11:13:00.801 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:13:00.802 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:13:00.802 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:13:01.139 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:13:01.139 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:13:01.140 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344414/5615c559-60b4-4b99-ae07-1bcd2063d0c6/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:13:01.517 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:13:01.518 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:13:01.518 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0250754 s, 41.8 MB/s 2021-02-10T11:13:01.519 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:13:01.519 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:13:01.519 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:13:01.854 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:13:01.855 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:13:01.855 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344414/5615c559-60b4-4b99-ae07-1bcd2063d0c6/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:13:02.232 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:13:02.232 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:13:02.232 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0258874 s, 40.5 MB/s 2021-02-10T11:13:02.233 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:13:02.233 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:13:02.234 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:13:02.569 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:13:02.569 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:13:02.569 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344414/5615c559-60b4-4b99-ae07-1bcd2063d0c6/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:13:02.944 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:13:02.944 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:13:02.944 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0238207 s, 44.0 MB/s 2021-02-10T11:13:02.945 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000344415 2021-02-10T11:13:03.381 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:03.369 7f55c41a6700 1 Processor -- start 2021-02-10T11:13:03.382 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:03.369 7f55c41a6700 1 -- start start 2021-02-10T11:13:03.382 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:03.369 7f55c41a6700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f55bc0f05f0 0x7f55bc0f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:13:03.382 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:03.369 7f55c41a6700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55bc1b0c70 con 0x7f55bc1b0870 2021-02-10T11:13:03.382 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:03.369 7f55c41a6700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55bc0f0b50 con 0x7f55bc1b0470 2021-02-10T11:13:03.382 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:03.369 7f55c41a6700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55bc0e5ea0 con 0x7f55bc0f05f0 2021-02-10T11:13:03.382 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:03.370 7f55c17c1700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f55bc1b0870 0x7f55bc0efe00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:44598/0 (socket says 172.21.2.102:44598) 2021-02-10T11:13:03.382 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:03.370 7f55c17c1700 1 -- 172.21.2.102:0/698170978 learned_addr learned my addr 172.21.2.102:0/698170978 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:13:03.382 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:03.370 7f55c27c3700 1 --2- 172.21.2.102:0/698170978 >> v2:172.21.2.113:3300/0 conn(0x7f55bc0f05f0 0x7f55bc0f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:13:03.382 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:03.370 7f55c0fc0700 1 -- 172.21.2.102:0/698170978 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2044288844 0 0) 0x7f55ac0010b0 con 0x7f55bc1b0870 2021-02-10T11:13:03.383 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:03.370 7f55c0fc0700 1 -- 172.21.2.102:0/698170978 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f55a8002fe0 con 0x7f55bc1b0870 2021-02-10T11:13:03.383 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:03.370 7f55c27c3700 1 -- 172.21.2.102:0/698170978 >> v1:172.21.2.113:6789/0 conn(0x7f55bc1b0470 legacy=0x7f55bc0ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:13:03.383 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:03.370 7f55c27c3700 1 -- 172.21.2.102:0/698170978 >> v1:172.21.2.102:6789/0 conn(0x7f55bc1b0870 legacy=0x7f55bc0efe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:13:03.383 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:03.370 7f55c0fc0700 1 -- 172.21.2.102:0/698170978 <== mon.1 v1:172.21.2.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2349547096 0 0) 0x7f55b8000ff0 con 0x7f55bc1b0470 2021-02-10T11:13:03.383 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:03.370 7f55c27c3700 1 -- 172.21.2.102:0/698170978 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55bc0e52e0 con 0x7f55bc0f05f0 2021-02-10T11:13:03.383 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:03.371 7f55c27c3700 1 --2- 172.21.2.102:0/698170978 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f55bc0f05f0 0x7f55bc0f20e0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f55b4006e50 tx=0x7f55b4009e70).ready entity=mon.1 client_cookie=3fd93f32a0242289 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:13:03.383 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:03.371 7f55c0fc0700 1 -- 172.21.2.102:0/698170978 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f55b400b510 con 0x7f55bc0f05f0 2021-02-10T11:13:03.383 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:03.371 7f55c0fc0700 1 -- 172.21.2.102:0/698170978 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55b4006790 con 0x7f55bc0f05f0 2021-02-10T11:13:03.383 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:03.371 7f55c0fc0700 1 -- 172.21.2.102:0/698170978 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f55b400b940 con 0x7f55bc0f05f0 2021-02-10T11:13:03.383 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:03.371 7f55c41a6700 1 -- 172.21.2.102:0/698170978 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f55bc0f05f0 msgr2=0x7f55bc0f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:13:03.383 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:03.371 7f55c41a6700 1 --2- 172.21.2.102:0/698170978 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f55bc0f05f0 0x7f55bc0f20e0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f55b4006e50 tx=0x7f55b4009e70).stop 2021-02-10T11:13:03.383 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:03.372 7f55c41a6700 1 -- 172.21.2.102:0/698170978 shutdown_connections 2021-02-10T11:13:03.384 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:03.372 7f55c41a6700 1 --2- 172.21.2.102:0/698170978 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f55bc0f05f0 0x7f55bc0f20e0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:03.384 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:03.372 7f55c41a6700 1 -- 172.21.2.102:0/698170978 >> 172.21.2.102:0/698170978 conn(0x7f55bc0ce3e0 msgr2=0x7f55bc0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:13:03.384 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:03.372 7f55c41a6700 1 -- 172.21.2.102:0/698170978 shutdown_connections 2021-02-10T11:13:03.384 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:03.372 7f55c41a6700 1 -- 172.21.2.102:0/698170978 wait complete. 2021-02-10T11:13:03.385 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:03.387 7f55c41a6700 1 Processor -- start 2021-02-10T11:13:03.385 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:03.387 7f55c41a6700 1 -- start start 2021-02-10T11:13:03.385 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:03.387 7f55c41a6700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f55bc1a42f0 0x7f55bc1a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:13:03.385 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:03.387 7f55c41a6700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f55bc196170 0x7f55bc196570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:13:03.386 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:03.387 7f55c41a6700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f55bc08f5b0 0x7f55bc08f9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:13:03.386 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:03.387 7f55c41a6700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55bc195bf0 con 0x7f55bc196170 2021-02-10T11:13:03.386 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:03.387 7f55c41a6700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55bc0c4130 con 0x7f55bc08f5b0 2021-02-10T11:13:03.386 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:03.387 7f55c41a6700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55bc0a6e70 con 0x7f55bc1a42f0 2021-02-10T11:13:03.386 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:03.388 7f55c17c1700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f55bc196170 0x7f55bc196570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:13:03.386 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:03.388 7f55c17c1700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f55bc196170 0x7f55bc196570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:52378/0 (socket says 172.21.2.102:52378) 2021-02-10T11:13:03.386 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:03.388 7f55c17c1700 1 -- 172.21.2.102:0/1099716701 learned_addr learned my addr 172.21.2.102:0/1099716701 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:13:03.386 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:03.388 7f55c17c1700 1 -- 172.21.2.102:0/1099716701 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f55bc1a42f0 msgr2=0x7f55bc1a46f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:13:03.386 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:03.388 7f55c17c1700 1 --2- 172.21.2.102:0/1099716701 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f55bc1a42f0 0x7f55bc1a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:13:03.386 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:03.388 7f55c17c1700 1 -- 172.21.2.102:0/1099716701 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f55bc08f5b0 msgr2=0x7f55bc08f9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:13:03.386 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:03.388 7f55c17c1700 1 --2- 172.21.2.102:0/1099716701 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f55bc08f5b0 0x7f55bc08f9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:13:03.386 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:03.388 7f55c17c1700 1 -- 172.21.2.102:0/1099716701 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55bc0a6a80 con 0x7f55bc196170 2021-02-10T11:13:03.387 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:03.388 7f55c17c1700 1 --2- 172.21.2.102:0/1099716701 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f55bc196170 0x7f55bc196570 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 rx=0x7f55ac001c60 tx=0x7f55ac00a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:13:03.387 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:03.388 7f55b27fc700 1 -- 172.21.2.102:0/1099716701 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f55ac00acc0 con 0x7f55bc196170 2021-02-10T11:13:03.387 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:03.388 7f55b27fc700 1 -- 172.21.2.102:0/1099716701 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55ac004140 con 0x7f55bc196170 2021-02-10T11:13:03.387 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:03.388 7f55b27fc700 1 -- 172.21.2.102:0/1099716701 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f55ac004660 con 0x7f55bc196170 2021-02-10T11:13:03.387 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:03.388 7f55c41a6700 1 -- 172.21.2.102:0/1099716701 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55bc0a6a80 con 0x7f55bc196170 2021-02-10T11:13:03.387 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:03.388 7f55c41a6700 1 -- 172.21.2.102:0/1099716701 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55bc0a6a80 con 0x7f55bc196170 2021-02-10T11:13:03.387 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:03.389 7f55b27fc700 1 -- 172.21.2.102:0/1099716701 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f55ac004320 con 0x7f55bc196170 2021-02-10T11:13:03.387 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:03.389 7f55b27fc700 1 --2- 172.21.2.102:0/1099716701 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f55a40186d0 0x7f55a401abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:13:03.387 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:03.389 7f55b27fc700 1 -- 172.21.2.102:0/1099716701 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f55ac032630 con 0x7f55bc196170 2021-02-10T11:13:03.387 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:03.389 7f55c1fc2700 1 --2- 172.21.2.102:0/1099716701 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f55a40186d0 0x7f55a401abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:13:03.387 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:03.389 7f55c1fc2700 1 --2- 172.21.2.102:0/1099716701 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f55a40186d0 0x7f55a401abd0 crc :-1 s=READY pgs=274 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:13:03.414 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:03.419 7f55c41a6700 1 -- 172.21.2.102:0/1099716701 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55bc0f0b50 con 0x7f55bc196170 2021-02-10T11:13:03.416 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:03.422 7f55b27fc700 1 -- 172.21.2.102:0/1099716701 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f55ac000e20 con 0x7f55bc196170 2021-02-10T11:13:03.546 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:03.551 7f55c41a6700 1 -- 172.21.2.102:0/1099716701 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344415", "target": ["mgr", ""]}) v1 -- 0x7f55bc0f0b50 con 0x7f55a40186d0 2021-02-10T11:13:03.572 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:03.577 7f55b27fc700 1 -- 172.21.2.102:0/1099716701 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f55b8007d80 con 0x7f55a40186d0 2021-02-10T11:13:03.579 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:03.584 7f55c41a6700 1 -- 172.21.2.102:0/1099716701 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f55a40186d0 msgr2=0x7f55a401abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:13:03.579 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:03.584 7f55c41a6700 1 --2- 172.21.2.102:0/1099716701 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f55a40186d0 0x7f55a401abd0 crc :-1 s=READY pgs=274 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:03.579 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:03.584 7f55c41a6700 1 -- 172.21.2.102:0/1099716701 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f55bc196170 msgr2=0x7f55bc196570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:13:03.579 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:03.584 7f55c41a6700 1 --2- 172.21.2.102:0/1099716701 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f55bc196170 0x7f55bc196570 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 rx=0x7f55ac001c60 tx=0x7f55ac00a8d0).stop 2021-02-10T11:13:03.579 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:03.584 7f55c41a6700 1 -- 172.21.2.102:0/1099716701 shutdown_connections 2021-02-10T11:13:03.579 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:03.584 7f55c41a6700 1 --2- 172.21.2.102:0/1099716701 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f55a40186d0 0x7f55a401abd0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:03.579 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:03.584 7f55c41a6700 1 --2- 172.21.2.102:0/1099716701 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f55bc08f5b0 0x7f55bc08f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:13:03.579 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:03.584 7f55c41a6700 1 --2- 172.21.2.102:0/1099716701 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f55bc196170 0x7f55bc196570 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:03.579 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:03.584 7f55c41a6700 1 --2- 172.21.2.102:0/1099716701 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f55bc1a42f0 0x7f55bc1a46f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:13:03.580 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:03.584 7f55c41a6700 1 -- 172.21.2.102:0/1099716701 >> 172.21.2.102:0/1099716701 conn(0x7f55bc0bf4e0 msgr2=0x7f55bc0be160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:13:03.580 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:03.584 7f55c41a6700 1 -- 172.21.2.102:0/1099716701 shutdown_connections 2021-02-10T11:13:03.580 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:03.584 7f55c41a6700 1 -- 172.21.2.102:0/1099716701 wait complete. 2021-02-10T11:13:03.587 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000344415 2021-02-10T11:13:04.019 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:04.008 7f6e72a3d700 1 Processor -- start 2021-02-10T11:13:04.019 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:04.008 7f6e72a3d700 1 -- start start 2021-02-10T11:13:04.019 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:04.008 7f6e72a3d700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f6e6c1b0470 0x7f6e6c1b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:13:04.020 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:04.008 7f6e72a3d700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6e6c0f20e0 0x7f6e6c0f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:13:04.020 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:04.008 7f6e72a3d700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e6c0f0b50 con 0x7f6e6c0ef610 2021-02-10T11:13:04.020 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:04.008 7f6e72a3d700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e6c0f0820 con 0x7f6e6c1b0470 2021-02-10T11:13:04.020 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:04.008 7f6e72a3d700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e6c0f24e0 con 0x7f6e6c0f20e0 2021-02-10T11:13:04.020 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:04.009 7f6e70859700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f6e6c1b0470 0x7f6e6c1b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:13:04.020 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:04.009 7f6e70859700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f6e6c1b0470 0x7f6e6c1b0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:52386/0 (socket says 172.21.2.102:52386) 2021-02-10T11:13:04.020 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:04.009 7f6e70859700 1 -- 172.21.2.102:0/2921992501 learned_addr learned my addr 172.21.2.102:0/2921992501 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:13:04.020 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:04.009 7f6e7105a700 1 --2- 172.21.2.102:0/2921992501 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6e6c0f20e0 0x7f6e6c0f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:13:04.020 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:04.009 7f6e70859700 1 -- 172.21.2.102:0/2921992501 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6e6c0f20e0 msgr2=0x7f6e6c0f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:13:04.020 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:04.009 7f6e70859700 1 --2- 172.21.2.102:0/2921992501 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6e6c0f20e0 0x7f6e6c0f0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:04.020 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:04.009 7f6e70859700 1 -- 172.21.2.102:0/2921992501 >> v1:172.21.2.113:6789/0 conn(0x7f6e6c0ef610 legacy=0x7f6e6c0efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:13:04.021 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:04.009 7f6e70859700 1 -- 172.21.2.102:0/2921992501 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e6c0e52e0 con 0x7f6e6c1b0470 2021-02-10T11:13:04.021 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:04.009 7f6e70859700 1 --2- 172.21.2.102:0/2921992501 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6e6c1b0470 0x7f6e6c1b0870 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 rx=0x7f6e54006700 tx=0x7f6e54009b00).ready entity=mon.0 client_cookie=2af4932a9201f925 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:13:04.021 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:04.009 7f6e6b7fe700 1 -- 172.21.2.102:0/2921992501 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6e5400c0c0 con 0x7f6e6c1b0470 2021-02-10T11:13:04.021 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:04.009 7f6e6b7fe700 1 -- 172.21.2.102:0/2921992501 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e5400c340 con 0x7f6e6c1b0470 2021-02-10T11:13:04.021 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:04.009 7f6e6b7fe700 1 -- 172.21.2.102:0/2921992501 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6e5400c820 con 0x7f6e6c1b0470 2021-02-10T11:13:04.021 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:04.009 7f6e72a3d700 1 -- 172.21.2.102:0/2921992501 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6e6c1b0470 msgr2=0x7f6e6c1b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:13:04.021 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:04.009 7f6e72a3d700 1 --2- 172.21.2.102:0/2921992501 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6e6c1b0470 0x7f6e6c1b0870 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 rx=0x7f6e54006700 tx=0x7f6e54009b00).stop 2021-02-10T11:13:04.021 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:04.010 7f6e72a3d700 1 -- 172.21.2.102:0/2921992501 shutdown_connections 2021-02-10T11:13:04.021 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:04.010 7f6e72a3d700 1 --2- 172.21.2.102:0/2921992501 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6e6c0f20e0 0x7f6e6c0f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:04.021 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:04.010 7f6e72a3d700 1 --2- 172.21.2.102:0/2921992501 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6e6c1b0470 0x7f6e6c1b0870 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:04.021 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:04.010 7f6e72a3d700 1 -- 172.21.2.102:0/2921992501 >> 172.21.2.102:0/2921992501 conn(0x7f6e6c0ce3e0 msgr2=0x7f6e6c0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:13:04.021 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:04.010 7f6e72a3d700 1 -- 172.21.2.102:0/2921992501 shutdown_connections 2021-02-10T11:13:04.021 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:04.010 7f6e72a3d700 1 -- 172.21.2.102:0/2921992501 wait complete. 2021-02-10T11:13:04.023 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:04.025 7f6e72a3d700 1 Processor -- start 2021-02-10T11:13:04.023 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:04.025 7f6e72a3d700 1 -- start start 2021-02-10T11:13:04.023 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:04.025 7f6e72a3d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6e6c0a0420 0x7f6e6c0a0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:13:04.023 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:04.025 7f6e72a3d700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6e6c189d00 0x7f6e6c18a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:13:04.023 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:04.025 7f6e72a3d700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6e6c18a720 0x7f6e6c0b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:13:04.023 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:04.025 7f6e72a3d700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e6c189800 con 0x7f6e6c0a0420 2021-02-10T11:13:04.023 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:04.025 7f6e72a3d700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e6c18ab20 con 0x7f6e6c189d00 2021-02-10T11:13:04.023 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:04.025 7f6e72a3d700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e6c0a1140 con 0x7f6e6c18a720 2021-02-10T11:13:04.023 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:04.025 7f6e70859700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6e6c0a0420 0x7f6e6c0a0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:13:04.024 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:04.025 7f6e70859700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6e6c0a0420 0x7f6e6c0a0820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:52394/0 (socket says 172.21.2.102:52394) 2021-02-10T11:13:04.024 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:04.025 7f6e70859700 1 -- 172.21.2.102:0/3465013679 learned_addr learned my addr 172.21.2.102:0/3465013679 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:13:04.024 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:04.026 7f6e70859700 1 -- 172.21.2.102:0/3465013679 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6e6c18a720 msgr2=0x7f6e6c0b7be0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:13:04.024 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:04.026 7f6e70859700 1 --2- 172.21.2.102:0/3465013679 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6e6c18a720 0x7f6e6c0b7be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:13:04.024 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:04.026 7f6e70859700 1 -- 172.21.2.102:0/3465013679 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6e6c189d00 msgr2=0x7f6e6c18a100 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:13:04.024 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:04.026 7f6e70859700 1 --2- 172.21.2.102:0/3465013679 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6e6c189d00 0x7f6e6c18a100 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:13:04.024 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:04.026 7f6e70859700 1 -- 172.21.2.102:0/3465013679 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e6c0b8200 con 0x7f6e6c0a0420 2021-02-10T11:13:04.024 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:04.026 7f6e70859700 1 --2- 172.21.2.102:0/3465013679 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6e6c0a0420 0x7f6e6c0a0820 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 rx=0x7f6e54006f20 tx=0x7f6e5400b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:13:04.024 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:04.026 7f6e697fa700 1 -- 172.21.2.102:0/3465013679 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6e5400bd80 con 0x7f6e6c0a0420 2021-02-10T11:13:04.024 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:04.026 7f6e697fa700 1 -- 172.21.2.102:0/3465013679 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e54000c90 con 0x7f6e6c0a0420 2021-02-10T11:13:04.024 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:04.026 7f6e72a3d700 1 -- 172.21.2.102:0/3465013679 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e6c0b8200 con 0x7f6e6c0a0420 2021-02-10T11:13:04.024 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:04.026 7f6e72a3d700 1 -- 172.21.2.102:0/3465013679 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e6c0b8790 con 0x7f6e6c0a0420 2021-02-10T11:13:04.024 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:04.026 7f6e697fa700 1 -- 172.21.2.102:0/3465013679 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6e540022b0 con 0x7f6e6c0a0420 2021-02-10T11:13:04.025 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:04.027 7f6e697fa700 1 -- 172.21.2.102:0/3465013679 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f6e54007000 con 0x7f6e6c0a0420 2021-02-10T11:13:04.025 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:04.027 7f6e697fa700 1 --2- 172.21.2.102:0/3465013679 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6e500186c0 0x7f6e5001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:13:04.025 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:04.027 7f6e697fa700 1 -- 172.21.2.102:0/3465013679 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f6e54032550 con 0x7f6e6c0a0420 2021-02-10T11:13:04.025 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:04.027 7f6e6bfff700 1 --2- 172.21.2.102:0/3465013679 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6e500186c0 0x7f6e5001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:13:04.025 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:04.027 7f6e6bfff700 1 --2- 172.21.2.102:0/3465013679 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6e500186c0 0x7f6e5001abc0 crc :-1 s=READY pgs=275 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:13:04.053 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:04.058 7f6e72a3d700 1 -- 172.21.2.102:0/3465013679 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e6c105ae0 con 0x7f6e6c0a0420 2021-02-10T11:13:04.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:04.061 7f6e697fa700 1 -- 172.21.2.102:0/3465013679 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f6e54030d80 con 0x7f6e6c0a0420 2021-02-10T11:13:04.183 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:04.188 7f6e72a3d700 1 -- 172.21.2.102:0/3465013679 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344415", "target": ["mgr", ""]}) v1 -- 0x7f6e6c0b8790 con 0x7f6e500186c0 2021-02-10T11:13:04.186 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:04.191 7f6e697fa700 1 -- 172.21.2.102:0/3465013679 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f6e5c006b00 con 0x7f6e500186c0 2021-02-10T11:13:04.190 INFO:teuthology.orchestra.run.gibba002.stdout:/volumes/_nogroup/subvolume_0000000000344415/78faef71-5077-4eb5-90f8-80dbb64529f6 2021-02-10T11:13:04.191 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:04.196 7f6e72a3d700 1 -- 172.21.2.102:0/3465013679 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6e500186c0 msgr2=0x7f6e5001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:13:04.191 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:04.196 7f6e72a3d700 1 --2- 172.21.2.102:0/3465013679 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6e500186c0 0x7f6e5001abc0 crc :-1 s=READY pgs=275 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:04.191 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:04.196 7f6e72a3d700 1 -- 172.21.2.102:0/3465013679 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6e6c0a0420 msgr2=0x7f6e6c0a0820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:13:04.191 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:04.196 7f6e72a3d700 1 --2- 172.21.2.102:0/3465013679 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6e6c0a0420 0x7f6e6c0a0820 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 rx=0x7f6e54006f20 tx=0x7f6e5400b990).stop 2021-02-10T11:13:04.192 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:04.197 7f6e72a3d700 1 -- 172.21.2.102:0/3465013679 shutdown_connections 2021-02-10T11:13:04.192 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:04.197 7f6e72a3d700 1 --2- 172.21.2.102:0/3465013679 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6e6c18a720 0x7f6e6c0b7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:13:04.192 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:04.197 7f6e72a3d700 1 --2- 172.21.2.102:0/3465013679 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6e500186c0 0x7f6e5001abc0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:04.192 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:04.197 7f6e72a3d700 1 --2- 172.21.2.102:0/3465013679 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6e6c189d00 0x7f6e6c18a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:13:04.192 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:04.197 7f6e72a3d700 1 --2- 172.21.2.102:0/3465013679 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6e6c0a0420 0x7f6e6c0a0820 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:04.192 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:04.197 7f6e72a3d700 1 -- 172.21.2.102:0/3465013679 >> 172.21.2.102:0/3465013679 conn(0x7f6e6c094e10 msgr2=0x7f6e6c0a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:13:04.192 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:04.197 7f6e72a3d700 1 -- 172.21.2.102:0/3465013679 shutdown_connections 2021-02-10T11:13:04.192 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:04.197 7f6e72a3d700 1 -- 172.21.2.102:0/3465013679 wait complete. 2021-02-10T11:13:04.200 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000344415 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000344415/78faef71-5077-4eb5-90f8-80dbb64529f6 2021-02-10T11:13:04.200 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:13:04.200 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:13:04.200 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:13:04.538 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:13:04.538 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:13:04.538 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344415/78faef71-5077-4eb5-90f8-80dbb64529f6/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:13:04.921 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:13:04.921 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:13:04.921 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0245503 s, 42.7 MB/s 2021-02-10T11:13:04.922 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:13:04.922 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:13:04.922 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:13:05.257 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:13:05.258 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:13:05.258 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344415/78faef71-5077-4eb5-90f8-80dbb64529f6/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:13:05.633 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:13:05.633 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:13:05.634 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0240146 s, 43.7 MB/s 2021-02-10T11:13:05.635 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:13:05.635 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:13:05.635 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:13:05.970 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:13:05.971 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:13:05.971 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344415/78faef71-5077-4eb5-90f8-80dbb64529f6/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:13:06.345 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:13:06.346 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:13:06.346 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0242251 s, 43.3 MB/s 2021-02-10T11:13:06.347 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:13:06.347 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:13:06.347 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:13:06.685 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:13:06.685 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:13:06.685 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344415/78faef71-5077-4eb5-90f8-80dbb64529f6/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:13:07.062 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:13:07.062 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:13:07.062 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.024784 s, 42.3 MB/s 2021-02-10T11:13:07.063 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:13:07.064 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:13:07.064 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:13:07.401 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:13:07.401 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:13:07.402 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344415/78faef71-5077-4eb5-90f8-80dbb64529f6/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:13:07.777 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:13:07.777 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:13:07.777 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0237505 s, 44.1 MB/s 2021-02-10T11:13:07.778 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:13:07.778 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:13:07.778 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:13:08.113 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:13:08.114 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:13:08.114 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344415/78faef71-5077-4eb5-90f8-80dbb64529f6/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:13:08.486 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:13:08.486 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:13:08.486 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0253501 s, 41.4 MB/s 2021-02-10T11:13:08.487 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:13:08.487 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:13:08.487 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:13:08.823 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:13:08.824 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:13:08.824 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344415/78faef71-5077-4eb5-90f8-80dbb64529f6/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:13:09.198 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:13:09.199 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:13:09.199 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0230959 s, 45.4 MB/s 2021-02-10T11:13:09.200 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:13:09.200 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:13:09.200 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:13:09.535 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:13:09.535 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:13:09.536 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344415/78faef71-5077-4eb5-90f8-80dbb64529f6/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:13:09.913 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:13:09.913 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:13:09.913 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0248551 s, 42.2 MB/s 2021-02-10T11:13:09.914 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:13:09.914 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:13:09.914 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:13:10.250 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:13:10.250 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:13:10.250 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344415/78faef71-5077-4eb5-90f8-80dbb64529f6/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:13:10.626 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:13:10.626 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:13:10.626 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0230761 s, 45.4 MB/s 2021-02-10T11:13:10.627 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:13:10.627 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:13:10.627 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:13:10.962 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:13:10.963 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:13:10.963 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344415/78faef71-5077-4eb5-90f8-80dbb64529f6/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:13:11.338 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:13:11.338 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:13:11.338 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0234724 s, 44.7 MB/s 2021-02-10T11:13:11.339 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000344416 2021-02-10T11:13:11.771 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:11.759 7f4a8b7ff700 1 Processor -- start 2021-02-10T11:13:11.771 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:11.759 7f4a8b7ff700 1 -- start start 2021-02-10T11:13:11.771 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:11.760 7f4a8b7ff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f4a841b0870 0x7f4a840f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:13:11.772 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:11.760 7f4a8b7ff700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4a840efe00 0x7f4a840f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:13:11.772 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:11.760 7f4a8b7ff700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a841b0c70 con 0x7f4a841b0470 2021-02-10T11:13:11.772 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:11.760 7f4a8b7ff700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a840f0b50 con 0x7f4a841b0870 2021-02-10T11:13:11.772 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:11.760 7f4a8b7ff700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a840f0820 con 0x7f4a840efe00 2021-02-10T11:13:11.772 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:11.760 7f4a88e1a700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f4a841b0870 0x7f4a840f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:13:11.772 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:11.760 7f4a88e1a700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f4a841b0870 0x7f4a840f20e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:52400/0 (socket says 172.21.2.102:52400) 2021-02-10T11:13:11.772 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:11.760 7f4a88e1a700 1 -- 172.21.2.102:0/4081515428 learned_addr learned my addr 172.21.2.102:0/4081515428 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:13:11.772 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:11.760 7f4a89e1c700 1 --2- 172.21.2.102:0/4081515428 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4a840efe00 0x7f4a840f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:13:11.772 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:11.761 7f4a88e1a700 1 -- 172.21.2.102:0/4081515428 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4a840efe00 msgr2=0x7f4a840f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:13:11.772 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:11.761 7f4a88e1a700 1 --2- 172.21.2.102:0/4081515428 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4a840efe00 0x7f4a840f0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:11.772 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:11.761 7f4a88e1a700 1 -- 172.21.2.102:0/4081515428 >> v1:172.21.2.113:6789/0 conn(0x7f4a841b0470 legacy=0x7f4a840ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:13:11.772 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:11.761 7f4a88e1a700 1 -- 172.21.2.102:0/4081515428 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a840e52e0 con 0x7f4a841b0870 2021-02-10T11:13:11.772 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:11.761 7f4a88e1a700 1 --2- 172.21.2.102:0/4081515428 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4a841b0870 0x7f4a840f20e0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 rx=0x7f4a74006700 tx=0x7f4a74009b80).ready entity=mon.0 client_cookie=d4a6466d0e1d8d1f server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:13:11.773 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:11.761 7f4a7bfff700 1 -- 172.21.2.102:0/4081515428 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4a7400c0c0 con 0x7f4a841b0870 2021-02-10T11:13:11.773 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:11.761 7f4a7bfff700 1 -- 172.21.2.102:0/4081515428 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a7400c400 con 0x7f4a841b0870 2021-02-10T11:13:11.773 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:11.761 7f4a7bfff700 1 -- 172.21.2.102:0/4081515428 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4a7400c920 con 0x7f4a841b0870 2021-02-10T11:13:11.773 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:11.761 7f4a8b7ff700 1 -- 172.21.2.102:0/4081515428 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4a841b0870 msgr2=0x7f4a840f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:13:11.773 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:11.761 7f4a8b7ff700 1 --2- 172.21.2.102:0/4081515428 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4a841b0870 0x7f4a840f20e0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 rx=0x7f4a74006700 tx=0x7f4a74009b80).stop 2021-02-10T11:13:11.773 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:11.762 7f4a8b7ff700 1 -- 172.21.2.102:0/4081515428 shutdown_connections 2021-02-10T11:13:11.773 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:11.762 7f4a8b7ff700 1 --2- 172.21.2.102:0/4081515428 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4a840efe00 0x7f4a840f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:11.773 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:11.762 7f4a8b7ff700 1 --2- 172.21.2.102:0/4081515428 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4a841b0870 0x7f4a840f20e0 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:11.773 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:11.762 7f4a8b7ff700 1 -- 172.21.2.102:0/4081515428 >> 172.21.2.102:0/4081515428 conn(0x7f4a840ce3e0 msgr2=0x7f4a840e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:13:11.773 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:11.762 7f4a8b7ff700 1 -- 172.21.2.102:0/4081515428 shutdown_connections 2021-02-10T11:13:11.773 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:11.762 7f4a8b7ff700 1 -- 172.21.2.102:0/4081515428 wait complete. 2021-02-10T11:13:11.775 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:11.777 7f4a8b7ff700 1 Processor -- start 2021-02-10T11:13:11.775 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:11.777 7f4a8b7ff700 1 -- start start 2021-02-10T11:13:11.775 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:11.778 7f4a8b7ff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4a840bf930 0x7f4a840bfd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:13:11.775 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:11.778 7f4a8b7ff700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4a840906c0 0x7f4a84090ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:13:11.775 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:11.778 7f4a8b7ff700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4a840c0960 0x7f4a840c0d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:13:11.775 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:11.778 7f4a8b7ff700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a840915c0 con 0x7f4a840bf930 2021-02-10T11:13:11.775 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:11.778 7f4a8b7ff700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a840910e0 con 0x7f4a840906c0 2021-02-10T11:13:11.775 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:11.778 7f4a8b7ff700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a84091300 con 0x7f4a840c0960 2021-02-10T11:13:11.775 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:11.778 7f4a8961b700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4a840bf930 0x7f4a840bfd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:13:11.775 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:11.778 7f4a8961b700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4a840bf930 0x7f4a840bfd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:52406/0 (socket says 172.21.2.102:52406) 2021-02-10T11:13:11.776 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:11.778 7f4a8961b700 1 -- 172.21.2.102:0/1993409830 learned_addr learned my addr 172.21.2.102:0/1993409830 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:13:11.776 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:11.778 7f4a8961b700 1 -- 172.21.2.102:0/1993409830 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4a840c0960 msgr2=0x7f4a840c0d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:13:11.776 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:11.778 7f4a8961b700 1 --2- 172.21.2.102:0/1993409830 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4a840c0960 0x7f4a840c0d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:13:11.776 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:11.778 7f4a8961b700 1 -- 172.21.2.102:0/1993409830 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4a840906c0 msgr2=0x7f4a84090ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:13:11.776 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:11.778 7f4a8961b700 1 --2- 172.21.2.102:0/1993409830 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4a840906c0 0x7f4a84090ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:13:11.776 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:11.778 7f4a8961b700 1 -- 172.21.2.102:0/1993409830 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a8419c0c0 con 0x7f4a840bf930 2021-02-10T11:13:11.776 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:11.779 7f4a8961b700 1 --2- 172.21.2.102:0/1993409830 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4a840bf930 0x7f4a840bfd30 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 rx=0x7f4a80006ed0 tx=0x7f4a8000b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:13:11.776 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:11.779 7f4a79ffb700 1 -- 172.21.2.102:0/1993409830 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4a80006240 con 0x7f4a840bf930 2021-02-10T11:13:11.776 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:11.779 7f4a79ffb700 1 -- 172.21.2.102:0/1993409830 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a800054b0 con 0x7f4a840bf930 2021-02-10T11:13:11.776 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:11.779 7f4a8b7ff700 1 -- 172.21.2.102:0/1993409830 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a8419c0c0 con 0x7f4a840bf930 2021-02-10T11:13:11.776 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:11.779 7f4a8b7ff700 1 -- 172.21.2.102:0/1993409830 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a840c1670 con 0x7f4a840bf930 2021-02-10T11:13:11.776 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:11.779 7f4a79ffb700 1 -- 172.21.2.102:0/1993409830 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4a80005990 con 0x7f4a840bf930 2021-02-10T11:13:11.776 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:11.779 7f4a79ffb700 1 -- 172.21.2.102:0/1993409830 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4a80006030 con 0x7f4a840bf930 2021-02-10T11:13:11.777 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:11.780 7f4a79ffb700 1 --2- 172.21.2.102:0/1993409830 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4a640186c0 0x7f4a6401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:13:11.777 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:11.780 7f4a79ffb700 1 -- 172.21.2.102:0/1993409830 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f4a80032550 con 0x7f4a840bf930 2021-02-10T11:13:11.777 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:11.780 7f4a88e1a700 1 --2- 172.21.2.102:0/1993409830 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4a640186c0 0x7f4a6401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:13:11.777 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:11.780 7f4a88e1a700 1 --2- 172.21.2.102:0/1993409830 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4a640186c0 0x7f4a6401abc0 crc :-1 s=READY pgs=276 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:13:11.804 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:11.809 7f4a8b7ff700 1 -- 172.21.2.102:0/1993409830 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a84105ae0 con 0x7f4a840bf930 2021-02-10T11:13:11.807 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:11.812 7f4a79ffb700 1 -- 172.21.2.102:0/1993409830 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f4a80030d00 con 0x7f4a840bf930 2021-02-10T11:13:11.934 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:11.940 7f4a8b7ff700 1 -- 172.21.2.102:0/1993409830 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344416", "target": ["mgr", ""]}) v1 -- 0x7f4a840c1670 con 0x7f4a640186c0 2021-02-10T11:13:11.958 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:11.963 7f4a79ffb700 1 -- 172.21.2.102:0/1993409830 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f4a74000940 con 0x7f4a640186c0 2021-02-10T11:13:11.968 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:11.972 7f4a8b7ff700 1 -- 172.21.2.102:0/1993409830 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4a640186c0 msgr2=0x7f4a6401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:13:11.968 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:11.972 7f4a8b7ff700 1 --2- 172.21.2.102:0/1993409830 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4a640186c0 0x7f4a6401abc0 crc :-1 s=READY pgs=276 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:11.968 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:11.972 7f4a8b7ff700 1 -- 172.21.2.102:0/1993409830 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4a840bf930 msgr2=0x7f4a840bfd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:13:11.968 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:11.972 7f4a8b7ff700 1 --2- 172.21.2.102:0/1993409830 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4a840bf930 0x7f4a840bfd30 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 rx=0x7f4a80006ed0 tx=0x7f4a8000b960).stop 2021-02-10T11:13:11.968 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:11.973 7f4a8b7ff700 1 -- 172.21.2.102:0/1993409830 shutdown_connections 2021-02-10T11:13:11.968 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:11.973 7f4a8b7ff700 1 --2- 172.21.2.102:0/1993409830 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4a840c0960 0x7f4a840c0d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:13:11.968 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:11.973 7f4a8b7ff700 1 --2- 172.21.2.102:0/1993409830 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4a640186c0 0x7f4a6401abc0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:11.968 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:11.973 7f4a8b7ff700 1 --2- 172.21.2.102:0/1993409830 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4a840906c0 0x7f4a84090ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:13:11.968 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:11.973 7f4a8b7ff700 1 --2- 172.21.2.102:0/1993409830 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4a840bf930 0x7f4a840bfd30 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:11.968 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:11.973 7f4a8b7ff700 1 -- 172.21.2.102:0/1993409830 >> 172.21.2.102:0/1993409830 conn(0x7f4a84094e10 msgr2=0x7f4a840a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:13:11.969 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:11.973 7f4a8b7ff700 1 -- 172.21.2.102:0/1993409830 shutdown_connections 2021-02-10T11:13:11.969 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:11.973 7f4a8b7ff700 1 -- 172.21.2.102:0/1993409830 wait complete. 2021-02-10T11:13:11.975 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000344416 2021-02-10T11:13:12.408 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:12.396 7f8973756700 1 Processor -- start 2021-02-10T11:13:12.408 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:12.397 7f8973756700 1 -- start start 2021-02-10T11:13:12.408 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:12.397 7f8973756700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f896c1b0470 0x7f896c1b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:13:12.408 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:12.397 7f8973756700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f896c0ef610 0x7f896c0efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:13:12.408 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:12.397 7f8973756700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f896c0f0b50 con 0x7f896c0f0030 2021-02-10T11:13:12.408 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:12.397 7f8973756700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f896c0e52e0 con 0x7f896c0ef610 2021-02-10T11:13:12.409 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:12.397 7f8973756700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f896c0e5490 con 0x7f896c1b0470 2021-02-10T11:13:12.409 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:12.397 7f8971d73700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f896c0f0030 0x7f896c0f10d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:44636/0 (socket says 172.21.2.102:44636) 2021-02-10T11:13:12.409 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:12.397 7f8971d73700 1 -- 172.21.2.102:0/951424240 learned_addr learned my addr 172.21.2.102:0/951424240 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:13:12.409 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:12.397 7f8970d71700 1 --2- 172.21.2.102:0/951424240 >> v2:172.21.2.102:3300/0 conn(0x7f896c0ef610 0x7f896c0efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:13:12.409 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:12.397 7f8971572700 1 --2- 172.21.2.102:0/951424240 >> v2:172.21.2.113:3300/0 conn(0x7f896c1b0470 0x7f896c1b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:13:12.409 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:12.398 7f895ffff700 1 -- 172.21.2.102:0/951424240 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 260699541 0 0) 0x7f8964001500 con 0x7f896c0f0030 2021-02-10T11:13:12.409 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:12.398 7f8970d71700 1 -- 172.21.2.102:0/951424240 >> v2:172.21.2.113:3300/0 conn(0x7f896c1b0470 msgr2=0x7f896c1b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:13:12.409 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:12.398 7f8970d71700 1 --2- 172.21.2.102:0/951424240 >> v2:172.21.2.113:3300/0 conn(0x7f896c1b0470 0x7f896c1b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:12.409 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:12.398 7f8970d71700 1 -- 172.21.2.102:0/951424240 >> v1:172.21.2.102:6789/0 conn(0x7f896c0f0030 legacy=0x7f896c0f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:13:12.409 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:12.398 7f8970d71700 1 -- 172.21.2.102:0/951424240 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f896c0f0430 con 0x7f896c0ef610 2021-02-10T11:13:12.409 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:12.398 7f8970d71700 1 --2- 172.21.2.102:0/951424240 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f896c0ef610 0x7f896c0efa10 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 rx=0x7f8960006650 tx=0x7f8960009a80).ready entity=mon.0 client_cookie=4e6de6afe8d215ec server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:13:12.409 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:12.398 7f895ffff700 1 -- 172.21.2.102:0/951424240 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f89600009b0 con 0x7f896c0ef610 2021-02-10T11:13:12.410 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:12.398 7f895ffff700 1 -- 172.21.2.102:0/951424240 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f896000c0c0 con 0x7f896c0ef610 2021-02-10T11:13:12.410 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:12.398 7f895ffff700 1 -- 172.21.2.102:0/951424240 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f896000c510 con 0x7f896c0ef610 2021-02-10T11:13:12.410 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:12.398 7f8973756700 1 -- 172.21.2.102:0/951424240 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f896c0ef610 msgr2=0x7f896c0efa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:13:12.410 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:12.398 7f8973756700 1 --2- 172.21.2.102:0/951424240 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f896c0ef610 0x7f896c0efa10 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 rx=0x7f8960006650 tx=0x7f8960009a80).stop 2021-02-10T11:13:12.410 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:12.399 7f8973756700 1 -- 172.21.2.102:0/951424240 shutdown_connections 2021-02-10T11:13:12.410 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:12.399 7f8973756700 1 --2- 172.21.2.102:0/951424240 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f896c0ef610 0x7f896c0efa10 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:12.410 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:12.399 7f8973756700 1 --2- 172.21.2.102:0/951424240 >> v2:172.21.2.113:3300/0 conn(0x7f896c1b0470 0x7f896c1b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:12.410 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:12.399 7f8973756700 1 -- 172.21.2.102:0/951424240 >> 172.21.2.102:0/951424240 conn(0x7f896c0ce3e0 msgr2=0x7f896c0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:13:12.410 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:12.399 7f8973756700 1 -- 172.21.2.102:0/951424240 shutdown_connections 2021-02-10T11:13:12.410 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:12.399 7f8973756700 1 -- 172.21.2.102:0/951424240 wait complete. 2021-02-10T11:13:12.412 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:12.414 7f8973756700 1 Processor -- start 2021-02-10T11:13:12.412 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:12.414 7f8973756700 1 -- start start 2021-02-10T11:13:12.412 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:12.414 7f8973756700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f896c190920 0x7f896c0b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:13:12.412 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:12.414 7f8973756700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f896c0b8200 0x7f896c0b8600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:13:12.412 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:12.414 7f8973756700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f896c0c1d80 0x7f896c0c2180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:13:12.412 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:12.414 7f8973756700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f896c190da0 con 0x7f896c0b8200 2021-02-10T11:13:12.412 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:12.414 7f8973756700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f896c0b8c20 con 0x7f896c190920 2021-02-10T11:13:12.413 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:12.414 7f8973756700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f896c0b8e10 con 0x7f896c0c1d80 2021-02-10T11:13:12.413 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:12.414 7f8970d71700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f896c0b8200 0x7f896c0b8600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:13:12.413 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:12.414 7f8970d71700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f896c0b8200 0x7f896c0b8600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:52418/0 (socket says 172.21.2.102:52418) 2021-02-10T11:13:12.413 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:12.414 7f8970d71700 1 -- 172.21.2.102:0/2309156503 learned_addr learned my addr 172.21.2.102:0/2309156503 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:13:12.413 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:12.415 7f8971572700 1 --2- 172.21.2.102:0/2309156503 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f896c190920 0x7f896c0b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:13:12.413 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:12.415 7f8971d73700 1 --2- 172.21.2.102:0/2309156503 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f896c0c1d80 0x7f896c0c2180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:13:12.413 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:12.415 7f8970d71700 1 -- 172.21.2.102:0/2309156503 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f896c0c1d80 msgr2=0x7f896c0c2180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:13:12.413 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:12.415 7f8970d71700 1 --2- 172.21.2.102:0/2309156503 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f896c0c1d80 0x7f896c0c2180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:12.413 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:12.415 7f8970d71700 1 -- 172.21.2.102:0/2309156503 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f896c190920 msgr2=0x7f896c0b7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:13:12.413 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:12.415 7f8970d71700 1 --2- 172.21.2.102:0/2309156503 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f896c190920 0x7f896c0b7be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:12.413 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:12.415 7f8970d71700 1 -- 172.21.2.102:0/2309156503 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f896c0c27d0 con 0x7f896c0b8200 2021-02-10T11:13:12.413 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:12.415 7f8970d71700 1 --2- 172.21.2.102:0/2309156503 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f896c0b8200 0x7f896c0b8600 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 rx=0x7f8960009630 tx=0x7f896000b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:13:12.413 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:12.415 7f895dffb700 1 -- 172.21.2.102:0/2309156503 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f89600093a0 con 0x7f896c0b8200 2021-02-10T11:13:12.413 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:12.415 7f895dffb700 1 -- 172.21.2.102:0/2309156503 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f896000bd70 con 0x7f896c0b8200 2021-02-10T11:13:12.414 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:12.415 7f895dffb700 1 -- 172.21.2.102:0/2309156503 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f89600054d0 con 0x7f896c0b8200 2021-02-10T11:13:12.414 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:12.415 7f8973756700 1 -- 172.21.2.102:0/2309156503 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f896c0c27d0 con 0x7f896c0b8200 2021-02-10T11:13:12.414 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:12.415 7f8973756700 1 -- 172.21.2.102:0/2309156503 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f896c0c2d60 con 0x7f896c0b8200 2021-02-10T11:13:12.414 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:12.416 7f895dffb700 1 -- 172.21.2.102:0/2309156503 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f896000a000 con 0x7f896c0b8200 2021-02-10T11:13:12.414 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:12.416 7f895dffb700 1 --2- 172.21.2.102:0/2309156503 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f894c0186c0 0x7f894c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:13:12.414 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:12.416 7f895dffb700 1 -- 172.21.2.102:0/2309156503 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f8960032550 con 0x7f896c0b8200 2021-02-10T11:13:12.414 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:12.416 7f8971572700 1 --2- 172.21.2.102:0/2309156503 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f894c0186c0 0x7f894c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:13:12.414 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:12.417 7f8971572700 1 --2- 172.21.2.102:0/2309156503 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f894c0186c0 0x7f894c01abc0 crc :-1 s=READY pgs=277 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:13:12.442 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:12.447 7f8973756700 1 -- 172.21.2.102:0/2309156503 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f896c105ae0 con 0x7f896c0b8200 2021-02-10T11:13:12.444 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:12.450 7f895dffb700 1 -- 172.21.2.102:0/2309156503 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f8960030d00 con 0x7f896c0b8200 2021-02-10T11:13:12.574 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:12.579 7f8973756700 1 -- 172.21.2.102:0/2309156503 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344416", "target": ["mgr", ""]}) v1 -- 0x7f896c0f0b50 con 0x7f894c0186c0 2021-02-10T11:13:12.577 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:12.582 7f895dffb700 1 -- 172.21.2.102:0/2309156503 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f8968006a70 con 0x7f894c0186c0 2021-02-10T11:13:12.577 INFO:teuthology.orchestra.run.gibba002.stdout:/volumes/_nogroup/subvolume_0000000000344416/437131bb-c357-45b3-b525-73b240992eea 2021-02-10T11:13:12.578 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:12.584 7f8973756700 1 -- 172.21.2.102:0/2309156503 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f894c0186c0 msgr2=0x7f894c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:13:12.578 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:12.584 7f8973756700 1 --2- 172.21.2.102:0/2309156503 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f894c0186c0 0x7f894c01abc0 crc :-1 s=READY pgs=277 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:12.579 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:12.584 7f8973756700 1 -- 172.21.2.102:0/2309156503 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f896c0b8200 msgr2=0x7f896c0b8600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:13:12.579 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:12.584 7f8973756700 1 --2- 172.21.2.102:0/2309156503 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f896c0b8200 0x7f896c0b8600 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 rx=0x7f8960009630 tx=0x7f896000b980).stop 2021-02-10T11:13:12.579 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:12.584 7f8973756700 1 -- 172.21.2.102:0/2309156503 shutdown_connections 2021-02-10T11:13:12.579 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:12.584 7f8973756700 1 --2- 172.21.2.102:0/2309156503 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f896c0c1d80 0x7f896c0c2180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:12.579 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:12.584 7f8973756700 1 --2- 172.21.2.102:0/2309156503 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f896c0b8200 0x7f896c0b8600 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:12.579 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:12.584 7f8973756700 1 --2- 172.21.2.102:0/2309156503 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f894c0186c0 0x7f894c01abc0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:12.580 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:12.584 7f8973756700 1 --2- 172.21.2.102:0/2309156503 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f896c190920 0x7f896c0b7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:12.580 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:12.584 7f8973756700 1 -- 172.21.2.102:0/2309156503 >> 172.21.2.102:0/2309156503 conn(0x7f896c094e10 msgr2=0x7f896c0a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:13:12.580 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:12.584 7f8973756700 1 -- 172.21.2.102:0/2309156503 shutdown_connections 2021-02-10T11:13:12.580 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:12.584 7f8973756700 1 -- 172.21.2.102:0/2309156503 wait complete. 2021-02-10T11:13:12.587 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000344416 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000344416/437131bb-c357-45b3-b525-73b240992eea 2021-02-10T11:13:12.587 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:13:12.587 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:13:12.587 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:13:12.924 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:13:12.925 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:13:12.925 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344416/437131bb-c357-45b3-b525-73b240992eea/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:13:13.308 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:13:13.309 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:13:13.309 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0276289 s, 38.0 MB/s 2021-02-10T11:13:13.310 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:13:13.310 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:13:13.310 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:13:13.646 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:13:13.646 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:13:13.647 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344416/437131bb-c357-45b3-b525-73b240992eea/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:13:14.019 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:13:14.020 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:13:14.020 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0243238 s, 43.1 MB/s 2021-02-10T11:13:14.021 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:13:14.021 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:13:14.021 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:13:14.356 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:13:14.357 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:13:14.357 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344416/437131bb-c357-45b3-b525-73b240992eea/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:13:14.731 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:13:14.731 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:13:14.731 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.023116 s, 45.4 MB/s 2021-02-10T11:13:14.732 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:13:14.732 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:13:14.732 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:13:15.069 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:13:15.070 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:13:15.070 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344416/437131bb-c357-45b3-b525-73b240992eea/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:13:15.447 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:13:15.448 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:13:15.448 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0260688 s, 40.2 MB/s 2021-02-10T11:13:15.449 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:13:15.449 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:13:15.449 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:13:15.785 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:13:15.786 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:13:15.786 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344416/437131bb-c357-45b3-b525-73b240992eea/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:13:16.160 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:13:16.160 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:13:16.160 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0241136 s, 43.5 MB/s 2021-02-10T11:13:16.161 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:13:16.161 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:13:16.161 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:13:16.497 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:13:16.498 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:13:16.498 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344416/437131bb-c357-45b3-b525-73b240992eea/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:13:16.875 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:13:16.875 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:13:16.875 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0258045 s, 40.6 MB/s 2021-02-10T11:13:16.876 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:13:16.876 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:13:16.877 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:13:17.212 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:13:17.213 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:13:17.213 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344416/437131bb-c357-45b3-b525-73b240992eea/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:13:17.587 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:13:17.587 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:13:17.587 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0226056 s, 46.4 MB/s 2021-02-10T11:13:17.588 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:13:17.588 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:13:17.588 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:13:17.923 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:13:17.924 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:13:17.924 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344416/437131bb-c357-45b3-b525-73b240992eea/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:13:18.301 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:13:18.302 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:13:18.302 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0252385 s, 41.5 MB/s 2021-02-10T11:13:18.303 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:13:18.303 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:13:18.303 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:13:18.638 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:13:18.639 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:13:18.639 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344416/437131bb-c357-45b3-b525-73b240992eea/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:13:19.020 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:13:23.199 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:13:23.199 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0245509 s, 42.7 MB/s 2021-02-10T11:13:23.199 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:13:23.200 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:13:23.200 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:13:23.201 DEBUG:teuthology.orchestra.run.gibba002:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:13:23.206 DEBUG:teuthology.orchestra.run.gibba013:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:13:23.541 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:13:23.541 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:13:23.541 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344416/437131bb-c357-45b3-b525-73b240992eea/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:13:23.914 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:13:23.914 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:13:23.914 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0244004 s, 43.0 MB/s 2021-02-10T11:13:23.915 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000344417 2021-02-10T11:13:24.348 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:24.338 7fbd6a1c8700 1 Processor -- start 2021-02-10T11:13:24.349 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:24.338 7fbd6a1c8700 1 -- start start 2021-02-10T11:13:24.349 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:24.338 7fbd6a1c8700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fbd641b0870 0x7fbd640f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:13:24.349 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:24.338 7fbd6a1c8700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fbd640efe00 0x7fbd640f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:13:24.349 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:24.338 7fbd6a1c8700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd641b0c70 con 0x7fbd641b0470 2021-02-10T11:13:24.349 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:24.338 7fbd6a1c8700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd640e5ea0 con 0x7fbd640efe00 2021-02-10T11:13:24.349 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:24.338 7fbd6a1c8700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd640f0b50 con 0x7fbd641b0870 2021-02-10T11:13:24.349 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:24.338 7fbd637fe700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fbd641b0470 0x7fbd640ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:44648/0 (socket says 172.21.2.102:44648) 2021-02-10T11:13:24.349 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:24.338 7fbd637fe700 1 -- 172.21.2.102:0/2026066953 learned_addr learned my addr 172.21.2.102:0/2026066953 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:13:24.349 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:24.338 7fbd63fff700 1 --2- 172.21.2.102:0/2026066953 >> v2:172.21.2.102:3300/0 conn(0x7fbd640efe00 0x7fbd640f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:13:24.349 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:24.339 7fbd63fff700 1 -- 172.21.2.102:0/2026066953 >> v2:172.21.2.113:3300/0 conn(0x7fbd641b0870 msgr2=0x7fbd640f20e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:13:24.349 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:24.339 7fbd63fff700 1 --2- 172.21.2.102:0/2026066953 >> v2:172.21.2.113:3300/0 conn(0x7fbd641b0870 0x7fbd640f20e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:13:24.350 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:24.339 7fbd63fff700 1 -- 172.21.2.102:0/2026066953 >> v1:172.21.2.102:6789/0 conn(0x7fbd641b0470 legacy=0x7fbd640ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:13:24.350 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:24.339 7fbd63fff700 1 -- 172.21.2.102:0/2026066953 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd640f0820 con 0x7fbd640efe00 2021-02-10T11:13:24.350 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:24.339 7fbd63fff700 1 --2- 172.21.2.102:0/2026066953 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbd640efe00 0x7fbd640f0200 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 rx=0x7fbd5c006e30 tx=0x7fbd5c009e60).ready entity=mon.0 client_cookie=c12b5179813a1041 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:13:24.350 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:24.339 7fbd627fc700 1 -- 172.21.2.102:0/2026066953 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbd5c006a50 con 0x7fbd640efe00 2021-02-10T11:13:24.350 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:24.339 7fbd627fc700 1 -- 172.21.2.102:0/2026066953 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd5c00b300 con 0x7fbd640efe00 2021-02-10T11:13:24.350 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:24.339 7fbd627fc700 1 -- 172.21.2.102:0/2026066953 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbd5c00b820 con 0x7fbd640efe00 2021-02-10T11:13:24.350 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:24.339 7fbd6a1c8700 1 -- 172.21.2.102:0/2026066953 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbd640efe00 msgr2=0x7fbd640f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:13:24.350 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:24.339 7fbd6a1c8700 1 --2- 172.21.2.102:0/2026066953 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbd640efe00 0x7fbd640f0200 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 rx=0x7fbd5c006e30 tx=0x7fbd5c009e60).stop 2021-02-10T11:13:24.350 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:24.340 7fbd6a1c8700 1 -- 172.21.2.102:0/2026066953 shutdown_connections 2021-02-10T11:13:24.350 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:24.340 7fbd6a1c8700 1 --2- 172.21.2.102:0/2026066953 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbd640efe00 0x7fbd640f0200 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:24.350 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:24.340 7fbd6a1c8700 1 --2- 172.21.2.102:0/2026066953 >> v2:172.21.2.113:3300/0 conn(0x7fbd641b0870 0x7fbd640f20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:13:24.350 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:24.340 7fbd6a1c8700 1 -- 172.21.2.102:0/2026066953 >> 172.21.2.102:0/2026066953 conn(0x7fbd640ce3e0 msgr2=0x7fbd640e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:13:24.351 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:24.340 7fbd6a1c8700 1 -- 172.21.2.102:0/2026066953 shutdown_connections 2021-02-10T11:13:24.351 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:24.340 7fbd6a1c8700 1 -- 172.21.2.102:0/2026066953 wait complete. 2021-02-10T11:13:24.352 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:24.354 7fbd6a1c8700 1 Processor -- start 2021-02-10T11:13:24.352 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:24.354 7fbd6a1c8700 1 -- start start 2021-02-10T11:13:24.352 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:24.355 7fbd6a1c8700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbd64094fb0 0x7fbd6409e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:13:24.352 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:24.355 7fbd6a1c8700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbd640c2be0 0x7fbd640c2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:13:24.352 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:24.355 7fbd6a1c8700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fbd640c3600 0x7fbd640a9c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:13:24.352 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:24.355 7fbd6a1c8700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd6409ebf0 con 0x7fbd640c2be0 2021-02-10T11:13:24.352 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:24.355 7fbd6a1c8700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd640aa2a0 con 0x7fbd640c3600 2021-02-10T11:13:24.352 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:24.355 7fbd6a1c8700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd640aa4c0 con 0x7fbd64094fb0 2021-02-10T11:13:24.353 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:24.355 7fbd62ffd700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbd640c2be0 0x7fbd640c2fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:13:24.353 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:24.355 7fbd62ffd700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbd640c2be0 0x7fbd640c2fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:52434/0 (socket says 172.21.2.102:52434) 2021-02-10T11:13:24.353 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:24.355 7fbd62ffd700 1 -- 172.21.2.102:0/1118918410 learned_addr learned my addr 172.21.2.102:0/1118918410 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:13:24.353 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:24.355 7fbd62ffd700 1 -- 172.21.2.102:0/1118918410 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbd64094fb0 msgr2=0x7fbd6409e5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:13:24.353 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:24.355 7fbd62ffd700 1 --2- 172.21.2.102:0/1118918410 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbd64094fb0 0x7fbd6409e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:13:24.353 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:24.355 7fbd62ffd700 1 -- 172.21.2.102:0/1118918410 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fbd640c3600 msgr2=0x7fbd640a9c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:13:24.353 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:24.355 7fbd62ffd700 1 --2- 172.21.2.102:0/1118918410 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fbd640c3600 0x7fbd640a9c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:13:24.353 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:24.355 7fbd62ffd700 1 -- 172.21.2.102:0/1118918410 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd640aa6e0 con 0x7fbd640c2be0 2021-02-10T11:13:24.353 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:24.356 7fbd62ffd700 1 --2- 172.21.2.102:0/1118918410 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbd640c2be0 0x7fbd640c2fe0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 rx=0x7fbd5400b560 tx=0x7fbd5400b900).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:13:24.353 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:24.356 7fbd5bfff700 1 -- 172.21.2.102:0/1118918410 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbd5400bcf0 con 0x7fbd640c2be0 2021-02-10T11:13:24.353 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:24.356 7fbd5bfff700 1 -- 172.21.2.102:0/1118918410 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd54006030 con 0x7fbd640c2be0 2021-02-10T11:13:24.353 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:24.356 7fbd5bfff700 1 -- 172.21.2.102:0/1118918410 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbd540054d0 con 0x7fbd640c2be0 2021-02-10T11:13:24.354 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:24.356 7fbd6a1c8700 1 -- 172.21.2.102:0/1118918410 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd640aa6e0 con 0x7fbd640c2be0 2021-02-10T11:13:24.354 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:24.356 7fbd6a1c8700 1 -- 172.21.2.102:0/1118918410 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd640aac70 con 0x7fbd640c2be0 2021-02-10T11:13:24.354 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:24.356 7fbd5bfff700 1 -- 172.21.2.102:0/1118918410 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fbd54006210 con 0x7fbd640c2be0 2021-02-10T11:13:24.354 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:24.356 7fbd5bfff700 1 --2- 172.21.2.102:0/1118918410 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fbd480186d0 0x7fbd4801abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:13:24.354 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:24.356 7fbd5bfff700 1 -- 172.21.2.102:0/1118918410 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fbd540327a0 con 0x7fbd640c2be0 2021-02-10T11:13:24.354 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:24.357 7fbd637fe700 1 --2- 172.21.2.102:0/1118918410 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fbd480186d0 0x7fbd4801abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:13:24.354 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:24.357 7fbd637fe700 1 --2- 172.21.2.102:0/1118918410 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fbd480186d0 0x7fbd4801abd0 crc :-1 s=READY pgs=278 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:13:24.383 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:24.388 7fbd6a1c8700 1 -- 172.21.2.102:0/1118918410 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd640a9750 con 0x7fbd640c2be0 2021-02-10T11:13:24.385 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:24.391 7fbd5bfff700 1 -- 172.21.2.102:0/1118918410 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fbd54000940 con 0x7fbd640c2be0 2021-02-10T11:13:24.511 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:24.516 7fbd6a1c8700 1 -- 172.21.2.102:0/1118918410 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344417", "target": ["mgr", ""]}) v1 -- 0x7fbd640aa2a0 con 0x7fbd480186d0 2021-02-10T11:13:24.538 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:24.543 7fbd5bfff700 1 -- 172.21.2.102:0/1118918410 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fbd4c001590 con 0x7fbd480186d0 2021-02-10T11:13:24.544 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:24.549 7fbd6a1c8700 1 -- 172.21.2.102:0/1118918410 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fbd480186d0 msgr2=0x7fbd4801abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:13:24.544 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:24.549 7fbd6a1c8700 1 --2- 172.21.2.102:0/1118918410 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fbd480186d0 0x7fbd4801abd0 crc :-1 s=READY pgs=278 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:24.545 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:24.549 7fbd6a1c8700 1 -- 172.21.2.102:0/1118918410 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbd640c2be0 msgr2=0x7fbd640c2fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:13:24.545 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:24.549 7fbd6a1c8700 1 --2- 172.21.2.102:0/1118918410 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbd640c2be0 0x7fbd640c2fe0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 rx=0x7fbd5400b560 tx=0x7fbd5400b900).stop 2021-02-10T11:13:24.545 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:24.549 7fbd6a1c8700 1 -- 172.21.2.102:0/1118918410 shutdown_connections 2021-02-10T11:13:24.545 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:24.549 7fbd6a1c8700 1 --2- 172.21.2.102:0/1118918410 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fbd480186d0 0x7fbd4801abd0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:24.545 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:24.550 7fbd6a1c8700 1 --2- 172.21.2.102:0/1118918410 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fbd640c3600 0x7fbd640a9c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:13:24.545 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:24.550 7fbd6a1c8700 1 --2- 172.21.2.102:0/1118918410 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbd640c2be0 0x7fbd640c2fe0 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:24.545 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:24.550 7fbd6a1c8700 1 --2- 172.21.2.102:0/1118918410 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbd64094fb0 0x7fbd6409e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:13:24.545 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:24.550 7fbd6a1c8700 1 -- 172.21.2.102:0/1118918410 >> 172.21.2.102:0/1118918410 conn(0x7fbd6409f630 msgr2=0x7fbd640a0130 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:13:24.545 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:24.550 7fbd6a1c8700 1 -- 172.21.2.102:0/1118918410 shutdown_connections 2021-02-10T11:13:24.545 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:24.550 7fbd6a1c8700 1 -- 172.21.2.102:0/1118918410 wait complete. 2021-02-10T11:13:24.552 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000344417 2021-02-10T11:13:24.990 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:24.979 7f63d8856700 1 Processor -- start 2021-02-10T11:13:24.991 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:24.979 7f63d8856700 1 -- start start 2021-02-10T11:13:24.991 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:24.980 7f63d8856700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f63d41b0870 0x7f63d40f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:13:24.991 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:24.980 7f63d8856700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f63d40efe00 0x7f63d40f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:13:24.991 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:24.980 7f63d8856700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63d41b0c70 con 0x7f63d41b0470 2021-02-10T11:13:24.991 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:24.980 7f63d8856700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63d40f0b50 con 0x7f63d41b0870 2021-02-10T11:13:24.991 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:24.980 7f63d8856700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63d40f0820 con 0x7f63d40efe00 2021-02-10T11:13:24.991 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:24.980 7f63d1e1b700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f63d41b0870 0x7f63d40f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:13:24.991 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:24.980 7f63d1e1b700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f63d41b0870 0x7f63d40f20e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:52438/0 (socket says 172.21.2.102:52438) 2021-02-10T11:13:24.991 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:24.980 7f63d1e1b700 1 -- 172.21.2.102:0/502875045 learned_addr learned my addr 172.21.2.102:0/502875045 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:13:24.991 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:24.980 7f63d1e1b700 1 -- 172.21.2.102:0/502875045 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f63d40efe00 msgr2=0x7f63d40f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:13:24.991 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:24.980 7f63d1e1b700 1 --2- 172.21.2.102:0/502875045 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f63d40efe00 0x7f63d40f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:13:24.991 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:24.980 7f63d1e1b700 1 -- 172.21.2.102:0/502875045 >> v1:172.21.2.102:6789/0 conn(0x7f63d41b0470 legacy=0x7f63d40ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:13:24.991 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:24.980 7f63d1e1b700 1 -- 172.21.2.102:0/502875045 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63d40e52e0 con 0x7f63d41b0870 2021-02-10T11:13:24.992 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:24.980 7f63d1e1b700 1 --2- 172.21.2.102:0/502875045 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f63d41b0870 0x7f63d40f20e0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 rx=0x7f63c4006700 tx=0x7f63c4009b80).ready entity=mon.0 client_cookie=7b22bc42de627aeb server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:13:24.992 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:24.981 7f63d161a700 1 -- 172.21.2.102:0/502875045 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f63c400c0c0 con 0x7f63d41b0870 2021-02-10T11:13:24.992 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:24.981 7f63d161a700 1 -- 172.21.2.102:0/502875045 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63c400c340 con 0x7f63d41b0870 2021-02-10T11:13:24.992 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:24.981 7f63d161a700 1 -- 172.21.2.102:0/502875045 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f63c400c860 con 0x7f63d41b0870 2021-02-10T11:13:24.992 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:24.981 7f63d8856700 1 -- 172.21.2.102:0/502875045 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f63d41b0870 msgr2=0x7f63d40f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:13:24.992 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:24.981 7f63d8856700 1 --2- 172.21.2.102:0/502875045 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f63d41b0870 0x7f63d40f20e0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 rx=0x7f63c4006700 tx=0x7f63c4009b80).stop 2021-02-10T11:13:24.992 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:24.981 7f63d8856700 1 -- 172.21.2.102:0/502875045 shutdown_connections 2021-02-10T11:13:24.992 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:24.981 7f63d8856700 1 --2- 172.21.2.102:0/502875045 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f63d41b0870 0x7f63d40f20e0 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:24.992 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:24.981 7f63d8856700 1 --2- 172.21.2.102:0/502875045 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f63d40efe00 0x7f63d40f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:13:24.992 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:24.981 7f63d8856700 1 -- 172.21.2.102:0/502875045 >> 172.21.2.102:0/502875045 conn(0x7f63d40ce3e0 msgr2=0x7f63d40e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:13:24.992 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:24.982 7f63d8856700 1 -- 172.21.2.102:0/502875045 shutdown_connections 2021-02-10T11:13:24.992 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:24.982 7f63d8856700 1 -- 172.21.2.102:0/502875045 wait complete. 2021-02-10T11:13:24.993 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:24.996 7f63d8856700 1 Processor -- start 2021-02-10T11:13:24.994 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:24.996 7f63d8856700 1 -- start start 2021-02-10T11:13:24.994 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:24.997 7f63d8856700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f63d4094fb0 0x7f63d409e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:13:24.994 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:24.997 7f63d8856700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f63d40c2be0 0x7f63d40c2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:13:24.994 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:24.997 7f63d8856700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f63d40c3600 0x7f63d40a9c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:13:24.994 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:24.997 7f63d8856700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63d409ebf0 con 0x7f63d40c2be0 2021-02-10T11:13:24.994 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:24.997 7f63d8856700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63d40aa2a0 con 0x7f63d4094fb0 2021-02-10T11:13:24.994 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:24.997 7f63d8856700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63d40aa4c0 con 0x7f63d40c3600 2021-02-10T11:13:24.994 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:24.997 7f63d1e1b700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f63d40c2be0 0x7f63d40c2fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:13:24.994 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:24.997 7f63d1e1b700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f63d40c2be0 0x7f63d40c2fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:52444/0 (socket says 172.21.2.102:52444) 2021-02-10T11:13:24.994 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:24.997 7f63d1e1b700 1 -- 172.21.2.102:0/3962136565 learned_addr learned my addr 172.21.2.102:0/3962136565 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:13:24.994 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:24.997 7f63d1e1b700 1 -- 172.21.2.102:0/3962136565 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f63d40c3600 msgr2=0x7f63d40a9c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:13:24.994 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:24.997 7f63d1e1b700 1 --2- 172.21.2.102:0/3962136565 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f63d40c3600 0x7f63d40a9c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:13:24.995 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:24.997 7f63d1e1b700 1 -- 172.21.2.102:0/3962136565 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f63d4094fb0 msgr2=0x7f63d409e5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:13:24.995 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:24.997 7f63d1e1b700 1 --2- 172.21.2.102:0/3962136565 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f63d4094fb0 0x7f63d409e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:13:24.995 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:24.997 7f63d1e1b700 1 -- 172.21.2.102:0/3962136565 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63d40aa6e0 con 0x7f63d40c2be0 2021-02-10T11:13:24.995 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:24.997 7f63d1e1b700 1 --2- 172.21.2.102:0/3962136565 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f63d40c2be0 0x7f63d40c2fe0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 rx=0x7f63c400b5e0 tx=0x7f63c400b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:13:24.995 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:24.997 7f63caffd700 1 -- 172.21.2.102:0/3962136565 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f63c400bd70 con 0x7f63d40c2be0 2021-02-10T11:13:24.995 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:24.997 7f63caffd700 1 -- 172.21.2.102:0/3962136565 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63c4002030 con 0x7f63d40c2be0 2021-02-10T11:13:24.995 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:24.997 7f63caffd700 1 -- 172.21.2.102:0/3962136565 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f63c4002550 con 0x7f63d40c2be0 2021-02-10T11:13:24.995 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:24.997 7f63d8856700 1 -- 172.21.2.102:0/3962136565 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63d40aa6e0 con 0x7f63d40c2be0 2021-02-10T11:13:24.995 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:24.997 7f63d8856700 1 -- 172.21.2.102:0/3962136565 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63d40aa6e0 con 0x7f63d40c2be0 2021-02-10T11:13:24.995 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:24.998 7f63caffd700 1 -- 172.21.2.102:0/3962136565 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f63c4000c90 con 0x7f63d40c2be0 2021-02-10T11:13:24.996 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:24.998 7f63caffd700 1 --2- 172.21.2.102:0/3962136565 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f63b80186d0 0x7f63b801abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:13:24.996 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:24.998 7f63caffd700 1 -- 172.21.2.102:0/3962136565 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f63c40327d0 con 0x7f63d40c2be0 2021-02-10T11:13:24.996 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:24.998 7f63d261c700 1 --2- 172.21.2.102:0/3962136565 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f63b80186d0 0x7f63b801abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:13:24.996 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:24.999 7f63d261c700 1 --2- 172.21.2.102:0/3962136565 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f63b80186d0 0x7f63b801abd0 crc :-1 s=READY pgs=279 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:13:25.024 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:25.029 7f63d8856700 1 -- 172.21.2.102:0/3962136565 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63d40edd10 con 0x7f63d40c2be0 2021-02-10T11:13:25.026 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:25.032 7f63caffd700 1 -- 172.21.2.102:0/3962136565 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f63c401f510 con 0x7f63d40c2be0 2021-02-10T11:13:25.156 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:25.161 7f63d8856700 1 -- 172.21.2.102:0/3962136565 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344417", "target": ["mgr", ""]}) v1 -- 0x7f63d409ebf0 con 0x7f63b80186d0 2021-02-10T11:13:25.159 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:25.164 7f63caffd700 1 -- 172.21.2.102:0/3962136565 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f63bc006e50 con 0x7f63b80186d0 2021-02-10T11:13:25.163 INFO:teuthology.orchestra.run.gibba002.stdout:/volumes/_nogroup/subvolume_0000000000344417/74196635-cd87-4eb4-9002-33d34c7f8168 2021-02-10T11:13:25.164 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:25.169 7f63d8856700 1 -- 172.21.2.102:0/3962136565 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f63b80186d0 msgr2=0x7f63b801abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:13:25.164 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:25.169 7f63d8856700 1 --2- 172.21.2.102:0/3962136565 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f63b80186d0 0x7f63b801abd0 crc :-1 s=READY pgs=279 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:25.164 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:25.169 7f63d8856700 1 -- 172.21.2.102:0/3962136565 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f63d40c2be0 msgr2=0x7f63d40c2fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:13:25.165 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:25.169 7f63d8856700 1 --2- 172.21.2.102:0/3962136565 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f63d40c2be0 0x7f63d40c2fe0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 rx=0x7f63c400b5e0 tx=0x7f63c400b980).stop 2021-02-10T11:13:25.165 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:25.169 7f63d8856700 1 -- 172.21.2.102:0/3962136565 shutdown_connections 2021-02-10T11:13:25.165 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:25.169 7f63d8856700 1 --2- 172.21.2.102:0/3962136565 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f63d40c3600 0x7f63d40a9c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:13:25.165 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:25.169 7f63d8856700 1 --2- 172.21.2.102:0/3962136565 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f63d40c2be0 0x7f63d40c2fe0 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:25.165 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:25.169 7f63d8856700 1 --2- 172.21.2.102:0/3962136565 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f63b80186d0 0x7f63b801abd0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:25.165 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:25.169 7f63d8856700 1 --2- 172.21.2.102:0/3962136565 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f63d4094fb0 0x7f63d409e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:13:25.165 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:25.169 7f63d8856700 1 -- 172.21.2.102:0/3962136565 >> 172.21.2.102:0/3962136565 conn(0x7f63d409f630 msgr2=0x7f63d40a0130 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:13:25.165 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:25.170 7f63d8856700 1 -- 172.21.2.102:0/3962136565 shutdown_connections 2021-02-10T11:13:25.165 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:25.170 7f63d8856700 1 -- 172.21.2.102:0/3962136565 wait complete. 2021-02-10T11:13:25.172 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000344417 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000344417/74196635-cd87-4eb4-9002-33d34c7f8168 2021-02-10T11:13:25.172 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:13:25.172 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:13:25.173 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:13:25.519 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:13:25.520 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:13:25.520 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344417/74196635-cd87-4eb4-9002-33d34c7f8168/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:13:25.907 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:13:25.907 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:13:25.907 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0224801 s, 46.6 MB/s 2021-02-10T11:13:25.909 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:13:25.909 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:13:25.909 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:13:26.247 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:13:26.247 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:13:26.247 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344417/74196635-cd87-4eb4-9002-33d34c7f8168/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:13:26.624 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:13:26.624 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:13:26.624 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0255838 s, 41.0 MB/s 2021-02-10T11:13:26.625 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:13:26.625 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:13:26.625 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:13:26.961 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:13:26.961 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:13:26.962 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344417/74196635-cd87-4eb4-9002-33d34c7f8168/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:13:27.337 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:13:27.337 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:13:27.338 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0244979 s, 42.8 MB/s 2021-02-10T11:13:27.339 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:13:27.339 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:13:27.339 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:13:27.674 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:13:27.675 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:13:27.675 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344417/74196635-cd87-4eb4-9002-33d34c7f8168/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:13:28.052 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:13:28.052 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:13:28.052 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0248458 s, 42.2 MB/s 2021-02-10T11:13:28.053 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:13:28.053 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:13:28.053 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:13:28.389 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:13:28.389 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:13:28.389 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344417/74196635-cd87-4eb4-9002-33d34c7f8168/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:13:28.765 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:13:28.765 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:13:28.766 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0242022 s, 43.3 MB/s 2021-02-10T11:13:28.767 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:13:28.767 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:13:28.767 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:13:29.102 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:13:29.103 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:13:29.103 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344417/74196635-cd87-4eb4-9002-33d34c7f8168/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:13:29.482 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:13:29.482 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:13:29.482 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0262808 s, 39.9 MB/s 2021-02-10T11:13:29.483 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:13:29.483 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:13:29.483 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:13:29.818 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:13:29.818 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:13:29.819 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344417/74196635-cd87-4eb4-9002-33d34c7f8168/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:13:30.194 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:13:30.195 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:13:30.195 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0246658 s, 42.5 MB/s 2021-02-10T11:13:30.196 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:13:30.196 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:13:30.196 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:13:30.532 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:13:30.532 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:13:30.533 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344417/74196635-cd87-4eb4-9002-33d34c7f8168/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:13:30.910 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:13:30.911 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:13:30.911 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0249095 s, 42.1 MB/s 2021-02-10T11:13:30.912 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:13:30.912 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:13:30.912 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:13:31.246 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:13:31.247 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:13:31.247 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344417/74196635-cd87-4eb4-9002-33d34c7f8168/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:13:31.624 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:13:31.624 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:13:31.624 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0254204 s, 41.2 MB/s 2021-02-10T11:13:31.625 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:13:31.625 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:13:31.625 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:13:31.960 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:13:31.960 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:13:31.961 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344417/74196635-cd87-4eb4-9002-33d34c7f8168/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:13:32.338 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:13:32.338 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:13:32.338 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.024062 s, 43.6 MB/s 2021-02-10T11:13:32.339 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000344418 2021-02-10T11:13:32.772 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:32.760 7fbc9f7c3700 1 Processor -- start 2021-02-10T11:13:32.772 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:32.760 7fbc9f7c3700 1 -- start start 2021-02-10T11:13:32.772 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:32.761 7fbc9f7c3700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbc981b0470 0x7fbc981b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:13:32.772 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:32.761 7fbc9f7c3700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fbc980ef610 0x7fbc980efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:13:32.773 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:32.761 7fbc9f7c3700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc980f0b50 con 0x7fbc980f0030 2021-02-10T11:13:32.773 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:32.761 7fbc9f7c3700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc980e52e0 con 0x7fbc980ef610 2021-02-10T11:13:32.773 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:32.761 7fbc9f7c3700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc980e5490 con 0x7fbc981b0470 2021-02-10T11:13:32.773 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:32.761 7fbc9cdde700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fbc980ef610 0x7fbc980efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:13:32.773 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:32.761 7fbc9cdde700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fbc980ef610 0x7fbc980efa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:52452/0 (socket says 172.21.2.102:52452) 2021-02-10T11:13:32.773 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:32.761 7fbc9cdde700 1 -- 172.21.2.102:0/1996488170 learned_addr learned my addr 172.21.2.102:0/1996488170 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:13:32.773 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:32.761 7fbc9d5df700 1 --2- 172.21.2.102:0/1996488170 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbc981b0470 0x7fbc981b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:13:32.773 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:32.762 7fbc9cdde700 1 -- 172.21.2.102:0/1996488170 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbc981b0470 msgr2=0x7fbc981b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:13:32.773 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:32.762 7fbc9cdde700 1 --2- 172.21.2.102:0/1996488170 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbc981b0470 0x7fbc981b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:32.773 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:32.762 7fbc9cdde700 1 -- 172.21.2.102:0/1996488170 >> v1:172.21.2.113:6789/0 conn(0x7fbc980f0030 legacy=0x7fbc980f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:13:32.773 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:32.762 7fbc9cdde700 1 -- 172.21.2.102:0/1996488170 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc980f0430 con 0x7fbc980ef610 2021-02-10T11:13:32.773 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:32.762 7fbc9cdde700 1 --2- 172.21.2.102:0/1996488170 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbc980ef610 0x7fbc980efa10 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 rx=0x7fbc88006700 tx=0x7fbc88009b00).ready entity=mon.0 client_cookie=e1b02825e17a84f9 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:13:32.773 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:32.762 7fbc8ffff700 1 -- 172.21.2.102:0/1996488170 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbc8800c0c0 con 0x7fbc980ef610 2021-02-10T11:13:32.774 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:32.762 7fbc8ffff700 1 -- 172.21.2.102:0/1996488170 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc8800c400 con 0x7fbc980ef610 2021-02-10T11:13:32.774 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:32.762 7fbc8ffff700 1 -- 172.21.2.102:0/1996488170 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbc8800c920 con 0x7fbc980ef610 2021-02-10T11:13:32.774 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:32.762 7fbc9f7c3700 1 -- 172.21.2.102:0/1996488170 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbc980ef610 msgr2=0x7fbc980efa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:13:32.774 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:32.762 7fbc9f7c3700 1 --2- 172.21.2.102:0/1996488170 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbc980ef610 0x7fbc980efa10 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 rx=0x7fbc88006700 tx=0x7fbc88009b00).stop 2021-02-10T11:13:32.774 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:32.763 7fbc9f7c3700 1 -- 172.21.2.102:0/1996488170 shutdown_connections 2021-02-10T11:13:32.774 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:32.763 7fbc9f7c3700 1 --2- 172.21.2.102:0/1996488170 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbc980ef610 0x7fbc980efa10 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:32.774 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:32.763 7fbc9f7c3700 1 --2- 172.21.2.102:0/1996488170 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbc981b0470 0x7fbc981b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:32.774 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:32.763 7fbc9f7c3700 1 -- 172.21.2.102:0/1996488170 >> 172.21.2.102:0/1996488170 conn(0x7fbc980ce3e0 msgr2=0x7fbc980e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:13:32.774 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:32.763 7fbc9f7c3700 1 -- 172.21.2.102:0/1996488170 shutdown_connections 2021-02-10T11:13:32.774 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:32.763 7fbc9f7c3700 1 -- 172.21.2.102:0/1996488170 wait complete. 2021-02-10T11:13:32.776 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:32.778 7fbc9f7c3700 1 Processor -- start 2021-02-10T11:13:32.776 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:32.778 7fbc9f7c3700 1 -- start start 2021-02-10T11:13:32.776 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:32.778 7fbc9f7c3700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fbc98105d10 0x7fbc9809eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:13:32.776 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:32.778 7fbc9f7c3700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbc98106110 0x7fbc9809f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:13:32.776 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:32.778 7fbc9f7c3700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbc980b2a00 0x7fbc980b2e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:13:32.777 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:32.778 7fbc9f7c3700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc9809e560 con 0x7fbc98106110 2021-02-10T11:13:32.777 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:32.778 7fbc9f7c3700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc9809c090 con 0x7fbc98105d10 2021-02-10T11:13:32.777 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:32.778 7fbc9f7c3700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc980b3420 con 0x7fbc980b2a00 2021-02-10T11:13:32.777 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:32.779 7fbc9cdde700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbc98106110 0x7fbc9809f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:13:32.777 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:32.779 7fbc9cdde700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbc98106110 0x7fbc9809f100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:52460/0 (socket says 172.21.2.102:52460) 2021-02-10T11:13:32.777 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:32.779 7fbc9cdde700 1 -- 172.21.2.102:0/3617664494 learned_addr learned my addr 172.21.2.102:0/3617664494 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:13:32.777 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:32.779 7fbc9d5df700 1 --2- 172.21.2.102:0/3617664494 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fbc98105d10 0x7fbc9809eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:13:32.777 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:32.779 7fbc9dde0700 1 --2- 172.21.2.102:0/3617664494 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbc980b2a00 0x7fbc980b2e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:13:32.777 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:32.779 7fbc9cdde700 1 -- 172.21.2.102:0/3617664494 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbc980b2a00 msgr2=0x7fbc980b2e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:13:32.777 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:32.779 7fbc9cdde700 1 --2- 172.21.2.102:0/3617664494 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbc980b2a00 0x7fbc980b2e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:32.777 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:32.779 7fbc9cdde700 1 -- 172.21.2.102:0/3617664494 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fbc98105d10 msgr2=0x7fbc9809eae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:13:32.777 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:32.779 7fbc9cdde700 1 --2- 172.21.2.102:0/3617664494 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fbc98105d10 0x7fbc9809eae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:32.777 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:32.779 7fbc9cdde700 1 -- 172.21.2.102:0/3617664494 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc980b3640 con 0x7fbc98106110 2021-02-10T11:13:32.778 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:32.779 7fbc9cdde700 1 --2- 172.21.2.102:0/3617664494 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbc98106110 0x7fbc9809f100 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 rx=0x7fbc8800b660 tx=0x7fbc8800ba00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:13:32.778 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:32.780 7fbc8dffb700 1 -- 172.21.2.102:0/3617664494 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbc88009440 con 0x7fbc98106110 2021-02-10T11:13:32.778 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:32.780 7fbc8dffb700 1 -- 172.21.2.102:0/3617664494 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc88000c90 con 0x7fbc98106110 2021-02-10T11:13:32.778 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:32.780 7fbc8dffb700 1 -- 172.21.2.102:0/3617664494 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbc88002030 con 0x7fbc98106110 2021-02-10T11:13:32.778 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:32.780 7fbc9f7c3700 1 -- 172.21.2.102:0/3617664494 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc980b3640 con 0x7fbc98106110 2021-02-10T11:13:32.778 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:32.780 7fbc9f7c3700 1 -- 172.21.2.102:0/3617664494 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc980b3640 con 0x7fbc98106110 2021-02-10T11:13:32.778 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:32.780 7fbc8dffb700 1 -- 172.21.2.102:0/3617664494 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fbc880093e0 con 0x7fbc98106110 2021-02-10T11:13:32.778 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:32.780 7fbc8dffb700 1 --2- 172.21.2.102:0/3617664494 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fbc780186c0 0x7fbc7801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:13:32.778 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:32.780 7fbc8dffb700 1 -- 172.21.2.102:0/3617664494 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fbc88032830 con 0x7fbc98106110 2021-02-10T11:13:32.778 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:32.780 7fbc9d5df700 1 --2- 172.21.2.102:0/3617664494 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fbc780186c0 0x7fbc7801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:13:32.778 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:32.781 7fbc9d5df700 1 --2- 172.21.2.102:0/3617664494 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fbc780186c0 0x7fbc7801abc0 crc :-1 s=READY pgs=280 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:13:32.805 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:32.811 7fbc9f7c3700 1 -- 172.21.2.102:0/3617664494 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc980b3ec0 con 0x7fbc98106110 2021-02-10T11:13:32.808 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:32.813 7fbc8dffb700 1 -- 172.21.2.102:0/3617664494 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fbc880090c0 con 0x7fbc98106110 2021-02-10T11:13:32.933 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:32.938 7fbc9f7c3700 1 -- 172.21.2.102:0/3617664494 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344418", "target": ["mgr", ""]}) v1 -- 0x7fbc981a16c0 con 0x7fbc780186c0 2021-02-10T11:13:32.959 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:32.965 7fbc8dffb700 1 -- 172.21.2.102:0/3617664494 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fbc94006a70 con 0x7fbc780186c0 2021-02-10T11:13:32.966 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:32.971 7fbc9f7c3700 1 -- 172.21.2.102:0/3617664494 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fbc780186c0 msgr2=0x7fbc7801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:13:32.966 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:32.971 7fbc9f7c3700 1 --2- 172.21.2.102:0/3617664494 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fbc780186c0 0x7fbc7801abc0 crc :-1 s=READY pgs=280 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:32.966 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:32.971 7fbc9f7c3700 1 -- 172.21.2.102:0/3617664494 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbc98106110 msgr2=0x7fbc9809f100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:13:32.966 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:32.971 7fbc9f7c3700 1 --2- 172.21.2.102:0/3617664494 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbc98106110 0x7fbc9809f100 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 rx=0x7fbc8800b660 tx=0x7fbc8800ba00).stop 2021-02-10T11:13:32.967 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:32.971 7fbc9f7c3700 1 -- 172.21.2.102:0/3617664494 shutdown_connections 2021-02-10T11:13:32.967 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:32.971 7fbc9f7c3700 1 --2- 172.21.2.102:0/3617664494 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbc980b2a00 0x7fbc980b2e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:32.967 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:32.971 7fbc9f7c3700 1 --2- 172.21.2.102:0/3617664494 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbc98106110 0x7fbc9809f100 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:32.967 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:32.971 7fbc9f7c3700 1 --2- 172.21.2.102:0/3617664494 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fbc780186c0 0x7fbc7801abc0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:32.967 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:32.971 7fbc9f7c3700 1 --2- 172.21.2.102:0/3617664494 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fbc98105d10 0x7fbc9809eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:32.967 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:32.971 7fbc9f7c3700 1 -- 172.21.2.102:0/3617664494 >> 172.21.2.102:0/3617664494 conn(0x7fbc980c36f0 msgr2=0x7fbc980bd640 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:13:32.967 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:32.971 7fbc9f7c3700 1 -- 172.21.2.102:0/3617664494 shutdown_connections 2021-02-10T11:13:32.967 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:32.971 7fbc9f7c3700 1 -- 172.21.2.102:0/3617664494 wait complete. 2021-02-10T11:13:32.974 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000344418 2021-02-10T11:13:33.409 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:33.397 7fda2799c700 1 Processor -- start 2021-02-10T11:13:33.409 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:33.397 7fda2799c700 1 -- start start 2021-02-10T11:13:33.409 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:33.398 7fda2799c700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fda201b0870 0x7fda200f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:13:33.409 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:33.398 7fda2799c700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda201b0c70 con 0x7fda200efe00 2021-02-10T11:13:33.410 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:33.398 7fda2799c700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda200f0b50 con 0x7fda201b0470 2021-02-10T11:13:33.410 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:33.398 7fda2799c700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda200e5ea0 con 0x7fda201b0870 2021-02-10T11:13:33.410 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:33.398 7fda25fb9700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fda200efe00 0x7fda200f0200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:44688/0 (socket says 172.21.2.102:44688) 2021-02-10T11:13:33.410 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:33.398 7fda25fb9700 1 -- 172.21.2.102:0/1058061507 learned_addr learned my addr 172.21.2.102:0/1058061507 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:13:33.410 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:33.398 7fda24fb7700 1 --2- 172.21.2.102:0/1058061507 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fda201b0870 0x7fda200f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:13:33.410 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:33.398 7fda13fff700 1 -- 172.21.2.102:0/1058061507 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4085312385 0 0) 0x7fda18001500 con 0x7fda200efe00 2021-02-10T11:13:33.410 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:33.398 7fda13fff700 1 -- 172.21.2.102:0/1058061507 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda0c002fe0 con 0x7fda200efe00 2021-02-10T11:13:33.410 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:33.398 7fda13fff700 1 -- 172.21.2.102:0/1058061507 <== mon.1 v1:172.21.2.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1763301244 0 0) 0x7fda14000ff0 con 0x7fda201b0470 2021-02-10T11:13:33.410 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:33.398 7fda13fff700 1 -- 172.21.2.102:0/1058061507 --> v1:172.21.2.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda0c003210 con 0x7fda201b0470 2021-02-10T11:13:33.410 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:33.398 7fda24fb7700 1 -- 172.21.2.102:0/1058061507 >> v1:172.21.2.113:6789/0 conn(0x7fda201b0470 legacy=0x7fda200ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:13:33.410 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:33.398 7fda24fb7700 1 -- 172.21.2.102:0/1058061507 >> v1:172.21.2.102:6789/0 conn(0x7fda200efe00 legacy=0x7fda200f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:13:33.411 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:33.398 7fda24fb7700 1 -- 172.21.2.102:0/1058061507 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda200e52e0 con 0x7fda201b0870 2021-02-10T11:13:33.411 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:33.399 7fda24fb7700 1 --2- 172.21.2.102:0/1058061507 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fda201b0870 0x7fda200f20e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7fda1c006650 tx=0x7fda1c009a00).ready entity=mon.2 client_cookie=8ce2dd0250ae8648 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:13:33.411 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:33.399 7fda13fff700 1 -- 172.21.2.102:0/1058061507 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fda1c0009b0 con 0x7fda201b0870 2021-02-10T11:13:33.411 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:33.399 7fda13fff700 1 -- 172.21.2.102:0/1058061507 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda1c00c0c0 con 0x7fda201b0870 2021-02-10T11:13:33.411 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:33.399 7fda13fff700 1 -- 172.21.2.102:0/1058061507 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fda1c00c5a0 con 0x7fda201b0870 2021-02-10T11:13:33.411 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:33.399 7fda2799c700 1 -- 172.21.2.102:0/1058061507 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fda201b0870 msgr2=0x7fda200f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:13:33.411 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:33.399 7fda2799c700 1 --2- 172.21.2.102:0/1058061507 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fda201b0870 0x7fda200f20e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7fda1c006650 tx=0x7fda1c009a00).stop 2021-02-10T11:13:33.411 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:33.400 7fda2799c700 1 -- 172.21.2.102:0/1058061507 shutdown_connections 2021-02-10T11:13:33.411 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:33.400 7fda2799c700 1 --2- 172.21.2.102:0/1058061507 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fda201b0870 0x7fda200f20e0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:33.411 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:33.400 7fda2799c700 1 -- 172.21.2.102:0/1058061507 >> 172.21.2.102:0/1058061507 conn(0x7fda200ce3e0 msgr2=0x7fda200e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:13:33.411 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:33.400 7fda2799c700 1 -- 172.21.2.102:0/1058061507 shutdown_connections 2021-02-10T11:13:33.411 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:33.400 7fda2799c700 1 -- 172.21.2.102:0/1058061507 wait complete. 2021-02-10T11:13:33.413 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:33.415 7fda2799c700 1 Processor -- start 2021-02-10T11:13:33.413 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:33.415 7fda2799c700 1 -- start start 2021-02-10T11:13:33.413 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:33.415 7fda2799c700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fda20190920 0x7fda200b81a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:13:33.413 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:33.415 7fda2799c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fda200b87c0 0x7fda200b70c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:13:33.413 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:33.415 7fda2799c700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fda200b8bc0 0x7fda200b76e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:13:33.413 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:33.415 7fda2799c700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda20190d20 con 0x7fda200b87c0 2021-02-10T11:13:33.413 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:33.415 7fda2799c700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda200b5ad0 con 0x7fda20190920 2021-02-10T11:13:33.414 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:33.415 7fda2799c700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda200b7d00 con 0x7fda200b8bc0 2021-02-10T11:13:33.414 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:33.416 7fda24fb7700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fda200b87c0 0x7fda200b70c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:13:33.414 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:33.416 7fda24fb7700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fda200b87c0 0x7fda200b70c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:52468/0 (socket says 172.21.2.102:52468) 2021-02-10T11:13:33.414 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:33.416 7fda24fb7700 1 -- 172.21.2.102:0/22860078 learned_addr learned my addr 172.21.2.102:0/22860078 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:13:33.414 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:33.416 7fda24fb7700 1 -- 172.21.2.102:0/22860078 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fda200b8bc0 msgr2=0x7fda200b76e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:13:33.414 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:33.416 7fda24fb7700 1 --2- 172.21.2.102:0/22860078 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fda200b8bc0 0x7fda200b76e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:13:33.414 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:33.416 7fda24fb7700 1 -- 172.21.2.102:0/22860078 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fda20190920 msgr2=0x7fda200b81a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:13:33.414 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:33.416 7fda24fb7700 1 --2- 172.21.2.102:0/22860078 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fda20190920 0x7fda200b81a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:13:33.414 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:33.416 7fda24fb7700 1 -- 172.21.2.102:0/22860078 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda200a4590 con 0x7fda200b87c0 2021-02-10T11:13:33.414 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:33.416 7fda24fb7700 1 --2- 172.21.2.102:0/22860078 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fda200b87c0 0x7fda200b70c0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 rx=0x7fda1c00b640 tx=0x7fda1c00b9e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:13:33.414 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:33.416 7fda11ffb700 1 -- 172.21.2.102:0/22860078 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fda1c00bdd0 con 0x7fda200b87c0 2021-02-10T11:13:33.414 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:33.416 7fda11ffb700 1 -- 172.21.2.102:0/22860078 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda1c000c90 con 0x7fda200b87c0 2021-02-10T11:13:33.415 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:33.416 7fda2799c700 1 -- 172.21.2.102:0/22860078 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda200a4590 con 0x7fda200b87c0 2021-02-10T11:13:33.415 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:33.416 7fda2799c700 1 -- 172.21.2.102:0/22860078 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda200a4b20 con 0x7fda200b87c0 2021-02-10T11:13:33.415 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:33.416 7fda11ffb700 1 -- 172.21.2.102:0/22860078 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fda1c0022b0 con 0x7fda200b87c0 2021-02-10T11:13:33.415 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:33.417 7fda11ffb700 1 -- 172.21.2.102:0/22860078 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fda1c0052e0 con 0x7fda200b87c0 2021-02-10T11:13:33.415 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:33.417 7fda11ffb700 1 --2- 172.21.2.102:0/22860078 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fda00018730 0x7fda0001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:13:33.415 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:33.417 7fda11ffb700 1 -- 172.21.2.102:0/22860078 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fda1c032640 con 0x7fda200b87c0 2021-02-10T11:13:33.415 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:33.417 7fda257b8700 1 --2- 172.21.2.102:0/22860078 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fda00018730 0x7fda0001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:13:33.415 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:33.417 7fda257b8700 1 --2- 172.21.2.102:0/22860078 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fda00018730 0x7fda0001abc0 crc :-1 s=READY pgs=281 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:13:33.442 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:33.447 7fda2799c700 1 -- 172.21.2.102:0/22860078 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9f8002fd0 con 0x7fda200b87c0 2021-02-10T11:13:33.445 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:33.450 7fda11ffb700 1 -- 172.21.2.102:0/22860078 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fda1c0009b0 con 0x7fda200b87c0 2021-02-10T11:13:33.575 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:33.581 7fda2799c700 1 -- 172.21.2.102:0/22860078 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344418", "target": ["mgr", ""]}) v1 -- 0x7fd9f8002f80 con 0x7fda00018730 2021-02-10T11:13:33.578 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:33.584 7fda11ffb700 1 -- 172.21.2.102:0/22860078 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7fda14006e40 con 0x7fda00018730 2021-02-10T11:13:33.582 INFO:teuthology.orchestra.run.gibba002.stdout:/volumes/_nogroup/subvolume_0000000000344418/9dcbe800-4755-4b89-8e48-522213d2ce59 2021-02-10T11:13:33.584 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:33.589 7fda2799c700 1 -- 172.21.2.102:0/22860078 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fda00018730 msgr2=0x7fda0001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:13:33.584 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:33.589 7fda2799c700 1 --2- 172.21.2.102:0/22860078 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fda00018730 0x7fda0001abc0 crc :-1 s=READY pgs=281 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:33.584 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:33.589 7fda2799c700 1 -- 172.21.2.102:0/22860078 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fda200b87c0 msgr2=0x7fda200b70c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:13:33.584 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:33.589 7fda2799c700 1 --2- 172.21.2.102:0/22860078 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fda200b87c0 0x7fda200b70c0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 rx=0x7fda1c00b640 tx=0x7fda1c00b9e0).stop 2021-02-10T11:13:33.584 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:33.589 7fda2799c700 1 -- 172.21.2.102:0/22860078 shutdown_connections 2021-02-10T11:13:33.584 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:33.589 7fda2799c700 1 --2- 172.21.2.102:0/22860078 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fda200b8bc0 0x7fda200b76e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:13:33.584 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:33.589 7fda2799c700 1 --2- 172.21.2.102:0/22860078 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fda200b87c0 0x7fda200b70c0 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:33.585 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:33.589 7fda2799c700 1 --2- 172.21.2.102:0/22860078 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fda00018730 0x7fda0001abc0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:33.585 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:33.589 7fda2799c700 1 --2- 172.21.2.102:0/22860078 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fda20190920 0x7fda200b81a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:13:33.585 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:33.589 7fda2799c700 1 -- 172.21.2.102:0/22860078 >> 172.21.2.102:0/22860078 conn(0x7fda2008ffe0 msgr2=0x7fda200bf050 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:13:33.585 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:33.590 7fda2799c700 1 -- 172.21.2.102:0/22860078 shutdown_connections 2021-02-10T11:13:33.585 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:33.590 7fda2799c700 1 -- 172.21.2.102:0/22860078 wait complete. 2021-02-10T11:13:33.592 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000344418 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000344418/9dcbe800-4755-4b89-8e48-522213d2ce59 2021-02-10T11:13:33.592 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:13:33.592 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:13:33.593 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:13:33.929 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:13:33.930 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:13:33.930 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344418/9dcbe800-4755-4b89-8e48-522213d2ce59/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:13:34.310 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:13:34.311 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:13:34.311 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.024256 s, 43.2 MB/s 2021-02-10T11:13:34.312 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:13:34.312 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:13:34.313 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:13:34.648 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:13:34.649 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:13:34.649 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344418/9dcbe800-4755-4b89-8e48-522213d2ce59/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:13:35.025 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:13:35.025 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:13:35.026 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0252049 s, 41.6 MB/s 2021-02-10T11:13:35.026 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:13:35.027 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:13:35.027 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:13:35.363 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:13:35.364 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:13:35.364 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344418/9dcbe800-4755-4b89-8e48-522213d2ce59/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:13:35.739 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:13:35.739 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:13:35.739 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0235065 s, 44.6 MB/s 2021-02-10T11:13:35.740 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:13:35.740 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:13:35.740 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:13:36.075 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:13:36.076 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:13:36.076 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344418/9dcbe800-4755-4b89-8e48-522213d2ce59/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:13:36.451 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:13:36.451 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:13:36.451 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0235154 s, 44.6 MB/s 2021-02-10T11:13:36.453 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:13:36.453 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:13:36.453 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:13:36.789 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:13:36.789 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:13:36.789 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344418/9dcbe800-4755-4b89-8e48-522213d2ce59/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:13:37.165 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:13:37.166 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:13:37.166 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.024085 s, 43.5 MB/s 2021-02-10T11:13:37.167 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:13:37.167 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:13:37.167 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:13:37.502 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:13:37.503 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:13:37.503 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344418/9dcbe800-4755-4b89-8e48-522213d2ce59/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:13:37.877 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:13:37.877 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:13:37.877 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0228828 s, 45.8 MB/s 2021-02-10T11:13:37.878 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:13:37.878 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:13:37.878 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:13:38.213 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:13:38.214 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:13:38.214 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344418/9dcbe800-4755-4b89-8e48-522213d2ce59/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:13:38.588 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:13:38.588 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:13:38.588 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0251443 s, 41.7 MB/s 2021-02-10T11:13:38.589 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:13:38.589 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:13:38.589 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:13:38.925 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:13:38.925 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:13:38.925 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344418/9dcbe800-4755-4b89-8e48-522213d2ce59/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:13:39.304 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:13:39.304 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:13:39.304 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0250835 s, 41.8 MB/s 2021-02-10T11:13:39.305 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:13:39.306 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:13:39.306 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:13:39.642 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:13:39.642 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:13:39.642 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344418/9dcbe800-4755-4b89-8e48-522213d2ce59/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:13:40.019 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:13:40.019 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:13:40.019 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0255747 s, 41.0 MB/s 2021-02-10T11:13:40.020 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:13:40.020 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:13:40.021 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:13:40.357 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:13:40.358 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:13:40.358 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344418/9dcbe800-4755-4b89-8e48-522213d2ce59/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:13:40.736 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:13:40.736 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:13:40.737 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.026299 s, 39.9 MB/s 2021-02-10T11:13:40.738 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000344419 2021-02-10T11:13:41.170 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:41.158 7fc6623b2700 1 Processor -- start 2021-02-10T11:13:41.171 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:41.158 7fc6623b2700 1 -- start start 2021-02-10T11:13:41.171 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:41.159 7fc6623b2700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc65c1b0870 0x7fc65c0f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:13:41.171 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:41.159 7fc6623b2700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fc65c0efe00 0x7fc65c0f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:13:41.171 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:41.159 7fc6623b2700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc65c1b0c70 con 0x7fc65c1b0470 2021-02-10T11:13:41.171 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:41.159 7fc6623b2700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc65c0f0b50 con 0x7fc65c0efe00 2021-02-10T11:13:41.171 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:41.159 7fc6623b2700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc65c0f0820 con 0x7fc65c1b0870 2021-02-10T11:13:41.171 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:41.159 7fc65bfff700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fc65c1b0470 0x7fc65c0ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:44698/0 (socket says 172.21.2.102:44698) 2021-02-10T11:13:41.171 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:41.159 7fc65bfff700 1 -- 172.21.2.102:0/3953250429 learned_addr learned my addr 172.21.2.102:0/3953250429 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:13:41.172 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:41.159 7fc6609cf700 1 --2- 172.21.2.102:0/3953250429 >> v2:172.21.2.113:3300/0 conn(0x7fc65c0efe00 0x7fc65c0f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:13:41.172 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:41.159 7fc65b7fe700 1 --2- 172.21.2.102:0/3953250429 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc65c1b0870 0x7fc65c0f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:13:41.172 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:41.160 7fc65affd700 1 -- 172.21.2.102:0/3953250429 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2395154932 0 0) 0x7fc64c001180 con 0x7fc65c1b0470 2021-02-10T11:13:41.172 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:41.160 7fc65affd700 1 -- 172.21.2.102:0/3953250429 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc644002fe0 con 0x7fc65c1b0470 2021-02-10T11:13:41.172 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:41.160 7fc65affd700 1 -- 172.21.2.102:0/3953250429 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3738909620 0 0) 0x7fc64c000ed0 con 0x7fc65c1b0470 2021-02-10T11:13:41.172 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:41.160 7fc6609cf700 1 -- 172.21.2.102:0/3953250429 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc65c1b0870 msgr2=0x7fc65c0f20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:13:41.172 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:41.160 7fc6609cf700 1 --2- 172.21.2.102:0/3953250429 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc65c1b0870 0x7fc65c0f20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:41.172 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:41.160 7fc6609cf700 1 -- 172.21.2.102:0/3953250429 >> v1:172.21.2.102:6789/0 conn(0x7fc65c1b0470 legacy=0x7fc65c0ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:13:41.172 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:41.160 7fc6609cf700 1 -- 172.21.2.102:0/3953250429 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc65c0e52e0 con 0x7fc65c0efe00 2021-02-10T11:13:41.172 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:41.160 7fc65b7fe700 1 --2- 172.21.2.102:0/3953250429 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc65c1b0870 0x7fc65c0f20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:13:41.172 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:41.161 7fc6609cf700 1 --2- 172.21.2.102:0/3953250429 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc65c0efe00 0x7fc65c0f0200 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7fc654006e50 tx=0x7fc654009e70).ready entity=mon.1 client_cookie=b3076d1f8cff6e6f server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:13:41.173 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:41.161 7fc65affd700 1 -- 172.21.2.102:0/3953250429 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc654006a20 con 0x7fc65c0efe00 2021-02-10T11:13:41.173 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:41.161 7fc65affd700 1 -- 172.21.2.102:0/3953250429 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc654006790 con 0x7fc65c0efe00 2021-02-10T11:13:41.173 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:41.161 7fc65affd700 1 -- 172.21.2.102:0/3953250429 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc65400b700 con 0x7fc65c0efe00 2021-02-10T11:13:41.173 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:41.161 7fc6623b2700 1 -- 172.21.2.102:0/3953250429 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc65c0efe00 msgr2=0x7fc65c0f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:13:41.173 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:41.161 7fc6623b2700 1 --2- 172.21.2.102:0/3953250429 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc65c0efe00 0x7fc65c0f0200 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7fc654006e50 tx=0x7fc654009e70).stop 2021-02-10T11:13:41.173 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:41.162 7fc6623b2700 1 -- 172.21.2.102:0/3953250429 shutdown_connections 2021-02-10T11:13:41.173 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:41.162 7fc6623b2700 1 --2- 172.21.2.102:0/3953250429 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc65c0efe00 0x7fc65c0f0200 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:41.173 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:41.162 7fc6623b2700 1 --2- 172.21.2.102:0/3953250429 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc65c1b0870 0x7fc65c0f20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:41.174 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:41.162 7fc6623b2700 1 -- 172.21.2.102:0/3953250429 >> 172.21.2.102:0/3953250429 conn(0x7fc65c0ce3e0 msgr2=0x7fc65c0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:13:41.174 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:41.162 7fc6623b2700 1 -- 172.21.2.102:0/3953250429 shutdown_connections 2021-02-10T11:13:41.174 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:41.162 7fc6623b2700 1 -- 172.21.2.102:0/3953250429 wait complete. 2021-02-10T11:13:41.176 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:41.177 7fc6623b2700 1 Processor -- start 2021-02-10T11:13:41.176 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:41.177 7fc6623b2700 1 -- start start 2021-02-10T11:13:41.176 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:41.177 7fc6623b2700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc65c0aa470 0x7fc65c19bb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:13:41.176 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:41.177 7fc6623b2700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc65c0acd10 0x7fc65c0adbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:13:41.176 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:41.177 7fc6623b2700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc65c0ae1f0 0x7fc65c0aee20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:13:41.176 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:41.177 7fc6623b2700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc65c0aa870 con 0x7fc65c0aa470 2021-02-10T11:13:41.176 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:41.177 7fc6623b2700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc65c19c1a0 con 0x7fc65c0ae1f0 2021-02-10T11:13:41.176 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:41.177 7fc6623b2700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc65c0af440 con 0x7fc65c0acd10 2021-02-10T11:13:41.176 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:41.177 7fc65bfff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc65c0aa470 0x7fc65c19bb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:13:41.176 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:41.177 7fc65bfff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc65c0aa470 0x7fc65c19bb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:52480/0 (socket says 172.21.2.102:52480) 2021-02-10T11:13:41.176 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:41.178 7fc65bfff700 1 -- 172.21.2.102:0/2361753207 learned_addr learned my addr 172.21.2.102:0/2361753207 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:13:41.176 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:41.178 7fc6609cf700 1 --2- 172.21.2.102:0/2361753207 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc65c0ae1f0 0x7fc65c0aee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:13:41.177 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:41.178 7fc65bfff700 1 -- 172.21.2.102:0/2361753207 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc65c0acd10 msgr2=0x7fc65c0adbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:13:41.177 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:41.178 7fc65bfff700 1 --2- 172.21.2.102:0/2361753207 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc65c0acd10 0x7fc65c0adbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:13:41.177 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:41.178 7fc65bfff700 1 -- 172.21.2.102:0/2361753207 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc65c0ae1f0 msgr2=0x7fc65c0aee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:13:41.177 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:41.178 7fc65bfff700 1 --2- 172.21.2.102:0/2361753207 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc65c0ae1f0 0x7fc65c0aee20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:41.177 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:41.178 7fc65bfff700 1 -- 172.21.2.102:0/2361753207 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc65c0af660 con 0x7fc65c0aa470 2021-02-10T11:13:41.177 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:41.178 7fc65bfff700 1 --2- 172.21.2.102:0/2361753207 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc65c0aa470 0x7fc65c19bb80 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 rx=0x7fc64c001690 tx=0x7fc64c00a920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:13:41.177 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:41.178 7fc658ff9700 1 -- 172.21.2.102:0/2361753207 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc64c001030 con 0x7fc65c0aa470 2021-02-10T11:13:41.177 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:41.178 7fc658ff9700 1 -- 172.21.2.102:0/2361753207 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc64c00ad10 con 0x7fc65c0aa470 2021-02-10T11:13:41.177 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:41.178 7fc658ff9700 1 -- 172.21.2.102:0/2361753207 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc64c004590 con 0x7fc65c0aa470 2021-02-10T11:13:41.177 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:41.178 7fc6623b2700 1 -- 172.21.2.102:0/2361753207 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc65c0af660 con 0x7fc65c0aa470 2021-02-10T11:13:41.177 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:41.179 7fc6623b2700 1 -- 172.21.2.102:0/2361753207 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc65c0af660 con 0x7fc65c0aa470 2021-02-10T11:13:41.177 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:41.179 7fc658ff9700 1 -- 172.21.2.102:0/2361753207 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fc64c008000 con 0x7fc65c0aa470 2021-02-10T11:13:41.177 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:41.179 7fc658ff9700 1 --2- 172.21.2.102:0/2361753207 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc63c0186c0 0x7fc63c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:13:41.178 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:41.179 7fc658ff9700 1 -- 172.21.2.102:0/2361753207 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fc64c032520 con 0x7fc65c0aa470 2021-02-10T11:13:41.178 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:41.179 7fc65b7fe700 1 --2- 172.21.2.102:0/2361753207 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc63c0186c0 0x7fc63c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:13:41.178 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:41.180 7fc65b7fe700 1 --2- 172.21.2.102:0/2361753207 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc63c0186c0 0x7fc63c01abc0 crc :-1 s=READY pgs=282 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:13:41.203 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:41.209 7fc6623b2700 1 -- 172.21.2.102:0/2361753207 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc65c0aa870 con 0x7fc65c0aa470 2021-02-10T11:13:41.206 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:41.212 7fc658ff9700 1 -- 172.21.2.102:0/2361753207 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fc64c001030 con 0x7fc65c0aa470 2021-02-10T11:13:41.332 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:41.337 7fc6623b2700 1 -- 172.21.2.102:0/2361753207 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344419", "target": ["mgr", ""]}) v1 -- 0x7fc65c0aa870 con 0x7fc63c0186c0 2021-02-10T11:13:41.358 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:41.364 7fc658ff9700 1 -- 172.21.2.102:0/2361753207 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fc6500009b0 con 0x7fc63c0186c0 2021-02-10T11:13:41.365 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:41.370 7fc6623b2700 1 -- 172.21.2.102:0/2361753207 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc63c0186c0 msgr2=0x7fc63c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:13:41.365 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:41.370 7fc6623b2700 1 --2- 172.21.2.102:0/2361753207 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc63c0186c0 0x7fc63c01abc0 crc :-1 s=READY pgs=282 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:41.365 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:41.370 7fc6623b2700 1 -- 172.21.2.102:0/2361753207 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc65c0aa470 msgr2=0x7fc65c19bb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:13:41.365 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:41.370 7fc6623b2700 1 --2- 172.21.2.102:0/2361753207 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc65c0aa470 0x7fc65c19bb80 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 rx=0x7fc64c001690 tx=0x7fc64c00a920).stop 2021-02-10T11:13:41.366 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:41.371 7fc6623b2700 1 -- 172.21.2.102:0/2361753207 shutdown_connections 2021-02-10T11:13:41.366 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:41.371 7fc6623b2700 1 --2- 172.21.2.102:0/2361753207 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc63c0186c0 0x7fc63c01abc0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:41.366 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:41.371 7fc6623b2700 1 --2- 172.21.2.102:0/2361753207 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc65c0ae1f0 0x7fc65c0aee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:41.366 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:41.371 7fc6623b2700 1 --2- 172.21.2.102:0/2361753207 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc65c0acd10 0x7fc65c0adbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:13:41.366 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:41.371 7fc6623b2700 1 --2- 172.21.2.102:0/2361753207 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc65c0aa470 0x7fc65c19bb80 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:41.366 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:41.371 7fc6623b2700 1 -- 172.21.2.102:0/2361753207 >> 172.21.2.102:0/2361753207 conn(0x7fc65c095a70 msgr2=0x7fc65c0bf050 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:13:41.366 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:41.371 7fc6623b2700 1 -- 172.21.2.102:0/2361753207 shutdown_connections 2021-02-10T11:13:41.366 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:41.371 7fc6623b2700 1 -- 172.21.2.102:0/2361753207 wait complete. 2021-02-10T11:13:41.373 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000344419 2021-02-10T11:13:41.807 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:41.795 7f74e4137700 1 Processor -- start 2021-02-10T11:13:41.807 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:41.795 7f74e4137700 1 -- start start 2021-02-10T11:13:41.807 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:41.796 7f74e4137700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f74dc1b0470 0x7f74dc1b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:13:41.807 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:41.796 7f74e4137700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f74dc0ef610 0x7f74dc0efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:13:41.807 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:41.796 7f74e4137700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74dc0f0b50 con 0x7f74dc0f0030 2021-02-10T11:13:41.808 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:41.796 7f74e4137700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74dc0e52e0 con 0x7f74dc0ef610 2021-02-10T11:13:41.808 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:41.796 7f74e4137700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74dc0e5490 con 0x7f74dc1b0470 2021-02-10T11:13:41.808 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:41.796 7f74e1752700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f74dc0ef610 0x7f74dc0efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:13:41.808 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:41.796 7f74e2754700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f74dc0f0030 0x7f74dc0f10d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:44712/0 (socket says 172.21.2.102:44712) 2021-02-10T11:13:41.808 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:41.796 7f74e2754700 1 -- 172.21.2.102:0/2493819206 learned_addr learned my addr 172.21.2.102:0/2493819206 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:13:41.808 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:41.796 7f74e1752700 1 -- 172.21.2.102:0/2493819206 >> v2:172.21.2.113:3300/0 conn(0x7f74dc1b0470 msgr2=0x7f74dc1b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:13:41.808 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:41.796 7f74e1752700 1 --2- 172.21.2.102:0/2493819206 >> v2:172.21.2.113:3300/0 conn(0x7f74dc1b0470 0x7f74dc1b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:13:41.809 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:41.796 7f74e1752700 1 -- 172.21.2.102:0/2493819206 >> v1:172.21.2.102:6789/0 conn(0x7f74dc0f0030 legacy=0x7f74dc0f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:13:41.809 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:41.796 7f74e1752700 1 -- 172.21.2.102:0/2493819206 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74dc0f0430 con 0x7f74dc0ef610 2021-02-10T11:13:41.809 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:41.796 7f74e1752700 1 --2- 172.21.2.102:0/2493819206 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f74dc0ef610 0x7f74dc0efa10 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 rx=0x7f74d0006650 tx=0x7f74d0009a00).ready entity=mon.0 client_cookie=e733b6ba4b7800c server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:13:41.809 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:41.797 7f74e0f51700 1 -- 172.21.2.102:0/2493819206 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f74d00009b0 con 0x7f74dc0ef610 2021-02-10T11:13:41.809 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:41.797 7f74e0f51700 1 -- 172.21.2.102:0/2493819206 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74d000c0c0 con 0x7f74dc0ef610 2021-02-10T11:13:41.809 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:41.797 7f74e0f51700 1 -- 172.21.2.102:0/2493819206 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f74d000c5a0 con 0x7f74dc0ef610 2021-02-10T11:13:41.809 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:41.797 7f74e4137700 1 -- 172.21.2.102:0/2493819206 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f74dc0ef610 msgr2=0x7f74dc0efa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:13:41.809 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:41.797 7f74e4137700 1 --2- 172.21.2.102:0/2493819206 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f74dc0ef610 0x7f74dc0efa10 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 rx=0x7f74d0006650 tx=0x7f74d0009a00).stop 2021-02-10T11:13:41.809 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:41.797 7f74e4137700 1 -- 172.21.2.102:0/2493819206 shutdown_connections 2021-02-10T11:13:41.809 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:41.797 7f74e4137700 1 --2- 172.21.2.102:0/2493819206 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f74dc0ef610 0x7f74dc0efa10 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:41.809 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:41.797 7f74e4137700 1 --2- 172.21.2.102:0/2493819206 >> v2:172.21.2.113:3300/0 conn(0x7f74dc1b0470 0x7f74dc1b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:13:41.810 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:41.797 7f74e4137700 1 -- 172.21.2.102:0/2493819206 >> 172.21.2.102:0/2493819206 conn(0x7f74dc0ce3e0 msgr2=0x7f74dc0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:13:41.810 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:41.798 7f74e4137700 1 -- 172.21.2.102:0/2493819206 shutdown_connections 2021-02-10T11:13:41.810 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:41.798 7f74e4137700 1 -- 172.21.2.102:0/2493819206 wait complete. 2021-02-10T11:13:41.810 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:41.812 7f74e4137700 1 Processor -- start 2021-02-10T11:13:41.810 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:41.812 7f74e4137700 1 -- start start 2021-02-10T11:13:41.813 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:41.813 7f74e4137700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f74dc0acc50 0x7f74dc098de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:13:41.813 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:41.813 7f74e4137700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f74dc0eec20 0x7f74dc0ef020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:13:41.813 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:41.813 7f74e4137700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f74dc0ef640 0x7f74dc1a0c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:13:41.813 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:41.813 7f74e4137700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74dc090cf0 con 0x7f74dc0acc50 2021-02-10T11:13:41.813 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:41.813 7f74e4137700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74dc0ad050 con 0x7f74dc0ef640 2021-02-10T11:13:41.814 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:41.813 7f74e4137700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74dc1a1250 con 0x7f74dc0eec20 2021-02-10T11:13:41.814 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:41.813 7f74e1f53700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f74dc0acc50 0x7f74dc098de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:13:41.814 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:41.813 7f74e1f53700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f74dc0acc50 0x7f74dc098de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:52492/0 (socket says 172.21.2.102:52492) 2021-02-10T11:13:41.814 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:41.813 7f74e1f53700 1 -- 172.21.2.102:0/3279826056 learned_addr learned my addr 172.21.2.102:0/3279826056 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:13:41.814 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:41.813 7f74e1752700 1 --2- 172.21.2.102:0/3279826056 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f74dc0eec20 0x7f74dc0ef020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:13:41.814 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:41.813 7f74e2754700 1 --2- 172.21.2.102:0/3279826056 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f74dc0ef640 0x7f74dc1a0c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:13:41.815 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:41.813 7f74e1f53700 1 -- 172.21.2.102:0/3279826056 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f74dc0eec20 msgr2=0x7f74dc0ef020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:13:41.815 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:41.813 7f74e1f53700 1 --2- 172.21.2.102:0/3279826056 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f74dc0eec20 0x7f74dc0ef020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:41.815 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:41.813 7f74e1f53700 1 -- 172.21.2.102:0/3279826056 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f74dc0ef640 msgr2=0x7f74dc1a0c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:13:41.815 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:41.813 7f74e1f53700 1 --2- 172.21.2.102:0/3279826056 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f74dc0ef640 0x7f74dc1a0c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:41.815 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:41.813 7f74e1f53700 1 -- 172.21.2.102:0/3279826056 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74dc1a1470 con 0x7f74dc0acc50 2021-02-10T11:13:41.816 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:41.814 7f74e1752700 1 --2- 172.21.2.102:0/3279826056 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f74dc0eec20 0x7f74dc0ef020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:13:41.816 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:41.814 7f74e2754700 1 --2- 172.21.2.102:0/3279826056 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f74dc0ef640 0x7f74dc1a0c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:13:41.816 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:41.814 7f74e1f53700 1 --2- 172.21.2.102:0/3279826056 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f74dc0acc50 0x7f74dc098de0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 rx=0x7f74d8006f00 tx=0x7f74d800b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:13:41.816 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:41.814 7f74ce7fc700 1 -- 172.21.2.102:0/3279826056 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f74d80009b0 con 0x7f74dc0acc50 2021-02-10T11:13:41.816 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:41.814 7f74ce7fc700 1 -- 172.21.2.102:0/3279826056 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74d8006350 con 0x7f74dc0acc50 2021-02-10T11:13:41.816 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:41.814 7f74ce7fc700 1 -- 172.21.2.102:0/3279826056 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f74d8000c90 con 0x7f74dc0acc50 2021-02-10T11:13:41.816 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:41.814 7f74e4137700 1 -- 172.21.2.102:0/3279826056 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74dc1a1470 con 0x7f74dc0acc50 2021-02-10T11:13:41.816 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:41.814 7f74e4137700 1 -- 172.21.2.102:0/3279826056 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74dc1a1470 con 0x7f74dc0acc50 2021-02-10T11:13:41.816 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:41.815 7f74ce7fc700 1 -- 172.21.2.102:0/3279826056 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f74d8007000 con 0x7f74dc0acc50 2021-02-10T11:13:41.816 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:41.815 7f74ce7fc700 1 --2- 172.21.2.102:0/3279826056 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f74c4018730 0x7f74c401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:13:41.816 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:41.815 7f74ce7fc700 1 -- 172.21.2.102:0/3279826056 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f74d8032630 con 0x7f74dc0acc50 2021-02-10T11:13:41.817 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:41.815 7f74e1752700 1 --2- 172.21.2.102:0/3279826056 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f74c4018730 0x7f74c401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:13:41.817 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:41.815 7f74e1752700 1 --2- 172.21.2.102:0/3279826056 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f74c4018730 0x7f74c401abc0 crc :-1 s=READY pgs=283 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:13:41.840 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:41.845 7f74e4137700 1 -- 172.21.2.102:0/3279826056 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74dc0edd10 con 0x7f74dc0acc50 2021-02-10T11:13:41.842 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:41.848 7f74ce7fc700 1 -- 172.21.2.102:0/3279826056 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f74d8006b90 con 0x7f74dc0acc50 2021-02-10T11:13:41.974 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:41.979 7f74e4137700 1 -- 172.21.2.102:0/3279826056 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344419", "target": ["mgr", ""]}) v1 -- 0x7f74dc0e52e0 con 0x7f74c4018730 2021-02-10T11:13:41.976 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:41.982 7f74ce7fc700 1 -- 172.21.2.102:0/3279826056 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f74d0000970 con 0x7f74c4018730 2021-02-10T11:13:41.977 INFO:teuthology.orchestra.run.gibba002.stdout:/volumes/_nogroup/subvolume_0000000000344419/65e28cc8-f714-43bc-afb5-48712cd5bd14 2021-02-10T11:13:41.978 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:41.984 7f74e4137700 1 -- 172.21.2.102:0/3279826056 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f74c4018730 msgr2=0x7f74c401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:13:41.979 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:41.984 7f74e4137700 1 --2- 172.21.2.102:0/3279826056 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f74c4018730 0x7f74c401abc0 crc :-1 s=READY pgs=283 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:41.979 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:41.984 7f74e4137700 1 -- 172.21.2.102:0/3279826056 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f74dc0acc50 msgr2=0x7f74dc098de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:13:41.979 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:41.984 7f74e4137700 1 --2- 172.21.2.102:0/3279826056 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f74dc0acc50 0x7f74dc098de0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 rx=0x7f74d8006f00 tx=0x7f74d800b940).stop 2021-02-10T11:13:41.979 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:41.984 7f74e4137700 1 -- 172.21.2.102:0/3279826056 shutdown_connections 2021-02-10T11:13:41.979 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:41.984 7f74e4137700 1 --2- 172.21.2.102:0/3279826056 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f74c4018730 0x7f74c401abc0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:41.979 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:41.984 7f74e4137700 1 --2- 172.21.2.102:0/3279826056 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f74dc0ef640 0x7f74dc1a0c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:41.979 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:41.984 7f74e4137700 1 --2- 172.21.2.102:0/3279826056 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f74dc0eec20 0x7f74dc0ef020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:41.980 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:41.984 7f74e4137700 1 --2- 172.21.2.102:0/3279826056 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f74dc0acc50 0x7f74dc098de0 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:41.980 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:41.984 7f74e4137700 1 -- 172.21.2.102:0/3279826056 >> 172.21.2.102:0/3279826056 conn(0x7f74dc0c40e0 msgr2=0x7f74dc0f22c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:13:41.980 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:41.984 7f74e4137700 1 -- 172.21.2.102:0/3279826056 shutdown_connections 2021-02-10T11:13:41.980 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:41.984 7f74e4137700 1 -- 172.21.2.102:0/3279826056 wait complete. 2021-02-10T11:13:41.987 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000344419 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000344419/65e28cc8-f714-43bc-afb5-48712cd5bd14 2021-02-10T11:13:41.987 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:13:41.987 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:13:41.987 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:13:42.324 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:13:42.324 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:13:42.325 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344419/65e28cc8-f714-43bc-afb5-48712cd5bd14/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:13:42.705 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:13:42.705 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:13:42.706 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0231889 s, 45.2 MB/s 2021-02-10T11:13:42.706 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:13:42.707 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:13:42.707 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:13:43.042 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:13:43.042 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:13:43.042 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344419/65e28cc8-f714-43bc-afb5-48712cd5bd14/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:13:43.415 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:13:43.415 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:13:43.415 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0240262 s, 43.6 MB/s 2021-02-10T11:13:43.416 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:13:43.416 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:13:43.416 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:13:43.751 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:13:43.752 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:13:43.752 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344419/65e28cc8-f714-43bc-afb5-48712cd5bd14/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:13:44.127 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:13:44.127 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:13:44.127 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0234164 s, 44.8 MB/s 2021-02-10T11:13:44.128 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:13:44.128 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:13:44.129 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:13:44.464 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:13:44.465 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:13:44.465 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344419/65e28cc8-f714-43bc-afb5-48712cd5bd14/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:13:44.842 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:13:44.842 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:13:44.842 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0251636 s, 41.7 MB/s 2021-02-10T11:13:44.843 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:13:44.844 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:13:44.844 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:13:45.179 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:13:45.179 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:13:45.179 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344419/65e28cc8-f714-43bc-afb5-48712cd5bd14/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:13:45.554 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:13:45.555 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:13:45.555 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0232766 s, 45.0 MB/s 2021-02-10T11:13:45.556 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:13:45.556 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:13:45.556 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:13:45.892 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:13:45.893 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:13:45.893 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344419/65e28cc8-f714-43bc-afb5-48712cd5bd14/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:13:46.269 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:13:46.269 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:13:46.269 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.023559 s, 44.5 MB/s 2021-02-10T11:13:46.271 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:13:46.271 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:13:46.271 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:13:46.607 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:13:46.607 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:13:46.607 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344419/65e28cc8-f714-43bc-afb5-48712cd5bd14/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:13:46.981 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:13:46.982 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:13:46.982 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.02239 s, 46.8 MB/s 2021-02-10T11:13:46.982 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:13:46.983 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:13:46.983 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:13:47.318 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:13:47.319 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:13:47.319 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344419/65e28cc8-f714-43bc-afb5-48712cd5bd14/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:13:47.691 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:13:47.692 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:13:47.692 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0232241 s, 45.2 MB/s 2021-02-10T11:13:47.693 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:13:47.693 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:13:47.693 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:13:48.031 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:13:49.023 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:13:49.023 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344419/65e28cc8-f714-43bc-afb5-48712cd5bd14/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:13:49.397 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:13:49.398 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:13:49.398 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0247738 s, 42.3 MB/s 2021-02-10T11:13:49.399 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:13:49.399 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:13:49.399 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:13:49.735 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:13:49.735 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:13:49.736 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344419/65e28cc8-f714-43bc-afb5-48712cd5bd14/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:13:50.114 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:13:50.115 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:13:50.115 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0261713 s, 40.1 MB/s 2021-02-10T11:13:50.116 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000344420 2021-02-10T11:13:50.546 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:50.535 7f8bfa0c4700 1 Processor -- start 2021-02-10T11:13:50.546 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:50.535 7f8bfa0c4700 1 -- start start 2021-02-10T11:13:50.547 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:50.536 7f8bfa0c4700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f8bf40f05f0 0x7f8bf40f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:13:50.547 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:50.536 7f8bfa0c4700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bf41b0c70 con 0x7f8bf41b0870 2021-02-10T11:13:50.547 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:50.536 7f8bfa0c4700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bf40f0b50 con 0x7f8bf41b0470 2021-02-10T11:13:50.547 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:50.536 7f8bfa0c4700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8bf40e5ea0 con 0x7f8bf40f05f0 2021-02-10T11:13:50.547 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:50.536 7f8bf3fff700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f8bf40f05f0 0x7f8bf40f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:13:50.547 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:50.536 7f8bf3fff700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f8bf40f05f0 0x7f8bf40f20e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.113:3300/0 says I am v2:172.21.2.102:55932/0 (socket says 172.21.2.102:55932) 2021-02-10T11:13:50.547 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:50.536 7f8bf3fff700 1 -- 172.21.2.102:0/1427468664 learned_addr learned my addr 172.21.2.102:0/1427468664 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:13:50.547 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:50.536 7f8bf27fc700 1 -- 172.21.2.102:0/1427468664 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2562184872 0 0) 0x7f8bdc000ff0 con 0x7f8bf41b0870 2021-02-10T11:13:50.547 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:50.537 7f8bf3fff700 1 -- 172.21.2.102:0/1427468664 >> v1:172.21.2.113:6789/0 conn(0x7f8bf41b0470 legacy=0x7f8bf40ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:13:50.547 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:50.537 7f8bf3fff700 1 -- 172.21.2.102:0/1427468664 >> v1:172.21.2.102:6789/0 conn(0x7f8bf41b0870 legacy=0x7f8bf40efe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:13:50.547 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:50.537 7f8bf3fff700 1 -- 172.21.2.102:0/1427468664 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bf40e52e0 con 0x7f8bf40f05f0 2021-02-10T11:13:50.548 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:50.537 7f8bf3fff700 1 --2- 172.21.2.102:0/1427468664 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8bf40f05f0 0x7f8bf40f20e0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7f8bec006aa0 tx=0x7f8bec009ea0).ready entity=mon.1 client_cookie=abbc44a07b8cb53e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:13:50.548 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:50.537 7f8bf27fc700 1 -- 172.21.2.102:0/1427468664 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8bec00b510 con 0x7f8bf40f05f0 2021-02-10T11:13:50.548 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:50.537 7f8bf27fc700 1 -- 172.21.2.102:0/1427468664 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bec003030 con 0x7f8bf40f05f0 2021-02-10T11:13:50.548 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:50.537 7f8bf27fc700 1 -- 172.21.2.102:0/1427468664 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8bec00bad0 con 0x7f8bf40f05f0 2021-02-10T11:13:50.548 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:50.537 7f8bfa0c4700 1 -- 172.21.2.102:0/1427468664 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8bf40f05f0 msgr2=0x7f8bf40f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:13:50.548 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:50.537 7f8bfa0c4700 1 --2- 172.21.2.102:0/1427468664 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8bf40f05f0 0x7f8bf40f20e0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7f8bec006aa0 tx=0x7f8bec009ea0).stop 2021-02-10T11:13:50.548 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:50.537 7f8bfa0c4700 1 -- 172.21.2.102:0/1427468664 shutdown_connections 2021-02-10T11:13:50.548 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:50.538 7f8bfa0c4700 1 --2- 172.21.2.102:0/1427468664 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8bf40f05f0 0x7f8bf40f20e0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:50.548 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:50.538 7f8bfa0c4700 1 -- 172.21.2.102:0/1427468664 >> 172.21.2.102:0/1427468664 conn(0x7f8bf40ce3e0 msgr2=0x7f8bf40e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:13:50.548 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:50.538 7f8bfa0c4700 1 -- 172.21.2.102:0/1427468664 shutdown_connections 2021-02-10T11:13:50.548 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:50.538 7f8bfa0c4700 1 -- 172.21.2.102:0/1427468664 wait complete. 2021-02-10T11:13:50.550 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:50.552 7f8bfa0c4700 1 Processor -- start 2021-02-10T11:13:50.550 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:50.552 7f8bfa0c4700 1 -- start start 2021-02-10T11:13:50.551 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:50.553 7f8bfa0c4700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8bf418ca80 0x7f8bf418d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:13:50.551 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:50.553 7f8bfa0c4700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8bf418d6c0 0x7f8bf418dac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:13:50.551 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:50.553 7f8bfa0c4700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8bf418e6b0 0x7f8bf418eab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:13:50.551 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:50.553 7f8bfa0c4700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bf419d290 con 0x7f8bf418ca80 2021-02-10T11:13:50.551 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:50.553 7f8bfa0c4700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bf418e130 con 0x7f8bf418e6b0 2021-02-10T11:13:50.551 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:50.553 7f8bfa0c4700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8bf418c5e0 con 0x7f8bf418d6c0 2021-02-10T11:13:50.551 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:50.553 7f8bf37fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8bf418ca80 0x7f8bf418d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:13:50.551 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:50.553 7f8bf3fff700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8bf418e6b0 0x7f8bf418eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:13:50.551 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:50.553 7f8bf37fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8bf418ca80 0x7f8bf418d0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:52506/0 (socket says 172.21.2.102:52506) 2021-02-10T11:13:50.551 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:50.553 7f8bf37fe700 1 -- 172.21.2.102:0/2687364688 learned_addr learned my addr 172.21.2.102:0/2687364688 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:13:50.551 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:50.553 7f8bf2ffd700 1 --2- 172.21.2.102:0/2687364688 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8bf418d6c0 0x7f8bf418dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:13:50.551 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:50.553 7f8bf3fff700 1 -- 172.21.2.102:0/2687364688 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8bf418d6c0 msgr2=0x7f8bf418dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:13:50.552 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:50.553 7f8bf3fff700 1 --2- 172.21.2.102:0/2687364688 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8bf418d6c0 0x7f8bf418dac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:50.552 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:50.554 7f8bf3fff700 1 -- 172.21.2.102:0/2687364688 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8bf418ca80 msgr2=0x7f8bf418d0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:13:50.552 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:50.554 7f8bf3fff700 1 --2- 172.21.2.102:0/2687364688 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8bf418ca80 0x7f8bf418d0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:50.552 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:50.554 7f8bf3fff700 1 -- 172.21.2.102:0/2687364688 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bf418f0d0 con 0x7f8bf418e6b0 2021-02-10T11:13:50.552 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:50.554 7f8bf37fe700 1 --2- 172.21.2.102:0/2687364688 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8bf418ca80 0x7f8bf418d0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:13:50.552 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:50.554 7f8bf3fff700 1 --2- 172.21.2.102:0/2687364688 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8bf418e6b0 0x7f8bf418eab0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f8bec0096c0 tx=0x7f8bec00bc20).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:13:50.552 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:50.554 7f8bebfff700 1 -- 172.21.2.102:0/2687364688 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8bec008030 con 0x7f8bf418e6b0 2021-02-10T11:13:50.552 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:50.554 7f8bebfff700 1 -- 172.21.2.102:0/2687364688 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bec0083b0 con 0x7f8bf418e6b0 2021-02-10T11:13:50.552 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:50.554 7f8bebfff700 1 -- 172.21.2.102:0/2687364688 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8bec008890 con 0x7f8bf418e6b0 2021-02-10T11:13:50.552 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:50.554 7f8bfa0c4700 1 -- 172.21.2.102:0/2687364688 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8bf418f0d0 con 0x7f8bf418e6b0 2021-02-10T11:13:50.552 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:50.554 7f8bfa0c4700 1 -- 172.21.2.102:0/2687364688 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8bf418f0d0 con 0x7f8bf418e6b0 2021-02-10T11:13:50.552 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:50.554 7f8bebfff700 1 -- 172.21.2.102:0/2687364688 <== mon.1 v2:172.21.2.113:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f8bec009700 con 0x7f8bf418e6b0 2021-02-10T11:13:50.553 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:50.555 7f8bebfff700 1 --2- 172.21.2.102:0/2687364688 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8bd80186c0 0x7f8bd801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:13:50.553 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:50.555 7f8bebfff700 1 -- 172.21.2.102:0/2687364688 <== mon.1 v2:172.21.2.113:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f8bec032820 con 0x7f8bf418e6b0 2021-02-10T11:13:50.553 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:50.555 7f8bf37fe700 1 --2- 172.21.2.102:0/2687364688 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8bd80186c0 0x7f8bd801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:13:50.553 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:50.555 7f8bf37fe700 1 --2- 172.21.2.102:0/2687364688 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8bd80186c0 0x7f8bd801abc0 crc :-1 s=READY pgs=284 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:13:50.580 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:50.585 7f8bfa0c4700 1 -- 172.21.2.102:0/2687364688 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8bf419d290 con 0x7f8bf418e6b0 2021-02-10T11:13:50.585 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:50.590 7f8bebfff700 1 -- 172.21.2.102:0/2687364688 <== mon.1 v2:172.21.2.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f8bec00bd60 con 0x7f8bf418e6b0 2021-02-10T11:13:50.716 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:50.717 7f8bfa0c4700 1 -- 172.21.2.102:0/2687364688 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344420", "target": ["mgr", ""]}) v1 -- 0x7f8bf419d290 con 0x7f8bd80186c0 2021-02-10T11:13:50.740 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:50.745 7f8bebfff700 1 -- 172.21.2.102:0/2687364688 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f8be4006d70 con 0x7f8bd80186c0 2021-02-10T11:13:50.752 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:50.750 7f8bfa0c4700 1 -- 172.21.2.102:0/2687364688 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8bd80186c0 msgr2=0x7f8bd801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:13:50.752 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:50.750 7f8bfa0c4700 1 --2- 172.21.2.102:0/2687364688 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8bd80186c0 0x7f8bd801abc0 crc :-1 s=READY pgs=284 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:50.752 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:50.750 7f8bfa0c4700 1 -- 172.21.2.102:0/2687364688 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8bf418e6b0 msgr2=0x7f8bf418eab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:13:50.752 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:50.750 7f8bfa0c4700 1 --2- 172.21.2.102:0/2687364688 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8bf418e6b0 0x7f8bf418eab0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f8bec0096c0 tx=0x7f8bec00bc20).stop 2021-02-10T11:13:50.752 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:50.750 7f8bfa0c4700 1 -- 172.21.2.102:0/2687364688 shutdown_connections 2021-02-10T11:13:50.752 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:50.750 7f8bfa0c4700 1 --2- 172.21.2.102:0/2687364688 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8bd80186c0 0x7f8bd801abc0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:50.752 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:50.750 7f8bfa0c4700 1 --2- 172.21.2.102:0/2687364688 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8bf418e6b0 0x7f8bf418eab0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:50.753 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:50.750 7f8bfa0c4700 1 --2- 172.21.2.102:0/2687364688 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8bf418d6c0 0x7f8bf418dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:50.753 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:50.750 7f8bfa0c4700 1 --2- 172.21.2.102:0/2687364688 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8bf418ca80 0x7f8bf418d0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:50.753 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:50.750 7f8bfa0c4700 1 -- 172.21.2.102:0/2687364688 >> 172.21.2.102:0/2687364688 conn(0x7f8bf40bf4f0 msgr2=0x7f8bf40be2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:13:50.753 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:50.750 7f8bfa0c4700 1 -- 172.21.2.102:0/2687364688 shutdown_connections 2021-02-10T11:13:50.753 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:50.750 7f8bfa0c4700 1 -- 172.21.2.102:0/2687364688 wait complete. 2021-02-10T11:13:50.754 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000344420 2021-02-10T11:13:51.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:51.201 7f6749737700 1 Processor -- start 2021-02-10T11:13:51.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:51.201 7f6749737700 1 -- start start 2021-02-10T11:13:51.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:51.201 7f6749737700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f67441b0470 0x7f67441b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:13:51.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:51.201 7f6749737700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f67440ef610 0x7f67440efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:13:51.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:51.201 7f6749737700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67440f0b50 con 0x7f67440f0030 2021-02-10T11:13:51.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:51.201 7f6749737700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67440e52e0 con 0x7f67440ef610 2021-02-10T11:13:51.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:51.201 7f6749737700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67440e5490 con 0x7f67441b0470 2021-02-10T11:13:51.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:51.201 7f6742d48700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f67440ef610 0x7f67440efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:13:51.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:51.201 7f6742d48700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f67440ef610 0x7f67440efa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:52512/0 (socket says 172.21.2.102:52512) 2021-02-10T11:13:51.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:51.201 7f6742d48700 1 -- 172.21.2.102:0/3807763225 learned_addr learned my addr 172.21.2.102:0/3807763225 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:13:51.213 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:51.201 7f6743549700 1 --2- 172.21.2.102:0/3807763225 >> v2:172.21.2.113:3300/0 conn(0x7f67441b0470 0x7f67441b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:13:51.213 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:51.202 7f6742d48700 1 -- 172.21.2.102:0/3807763225 >> v2:172.21.2.113:3300/0 conn(0x7f67441b0470 msgr2=0x7f67441b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:13:51.213 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:51.202 7f6742d48700 1 --2- 172.21.2.102:0/3807763225 >> v2:172.21.2.113:3300/0 conn(0x7f67441b0470 0x7f67441b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:51.213 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:51.202 7f6742d48700 1 -- 172.21.2.102:0/3807763225 >> v1:172.21.2.113:6789/0 conn(0x7f67440f0030 legacy=0x7f67440f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:13:51.213 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:51.202 7f6742d48700 1 -- 172.21.2.102:0/3807763225 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67440f0430 con 0x7f67440ef610 2021-02-10T11:13:51.213 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:51.202 7f6742d48700 1 --2- 172.21.2.102:0/3807763225 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f67440ef610 0x7f67440efa10 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 rx=0x7f6734006700 tx=0x7f6734009b80).ready entity=mon.0 client_cookie=cd1193f7c5cfcee1 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:13:51.213 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:51.202 7f6742547700 1 -- 172.21.2.102:0/3807763225 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f673400c0c0 con 0x7f67440ef610 2021-02-10T11:13:51.213 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:51.202 7f6742547700 1 -- 172.21.2.102:0/3807763225 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f673400c340 con 0x7f67440ef610 2021-02-10T11:13:51.213 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:51.202 7f6742547700 1 -- 172.21.2.102:0/3807763225 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f673400c860 con 0x7f67440ef610 2021-02-10T11:13:51.213 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:51.202 7f6749737700 1 -- 172.21.2.102:0/3807763225 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f67440ef610 msgr2=0x7f67440efa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:13:51.213 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:51.202 7f6749737700 1 --2- 172.21.2.102:0/3807763225 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f67440ef610 0x7f67440efa10 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 rx=0x7f6734006700 tx=0x7f6734009b80).stop 2021-02-10T11:13:51.213 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:51.203 7f6749737700 1 -- 172.21.2.102:0/3807763225 shutdown_connections 2021-02-10T11:13:51.213 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:51.203 7f6749737700 1 --2- 172.21.2.102:0/3807763225 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f67440ef610 0x7f67440efa10 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:51.214 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:51.203 7f6749737700 1 --2- 172.21.2.102:0/3807763225 >> v2:172.21.2.113:3300/0 conn(0x7f67441b0470 0x7f67441b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:51.214 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:51.203 7f6749737700 1 -- 172.21.2.102:0/3807763225 >> 172.21.2.102:0/3807763225 conn(0x7f67440ce3e0 msgr2=0x7f67440e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:13:51.214 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:51.203 7f6749737700 1 -- 172.21.2.102:0/3807763225 shutdown_connections 2021-02-10T11:13:51.214 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:51.203 7f6749737700 1 -- 172.21.2.102:0/3807763225 wait complete. 2021-02-10T11:13:51.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:51.218 7f6749737700 1 Processor -- start 2021-02-10T11:13:51.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:51.218 7f6749737700 1 -- start start 2021-02-10T11:13:51.216 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:51.218 7f6749737700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f67441a42e0 0x7f674409ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:13:51.216 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:51.218 7f6749737700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f67441a46e0 0x7f674409f310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:13:51.216 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:51.218 7f6749737700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f674408ad30 0x7f674408b130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:13:51.216 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:51.218 7f6749737700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f674408a7b0 con 0x7f674408ad30 2021-02-10T11:13:51.216 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:51.218 7f6749737700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67440a65f0 con 0x7f67441a42e0 2021-02-10T11:13:51.217 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:51.218 7f6749737700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f674408a430 con 0x7f67441a46e0 2021-02-10T11:13:51.217 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:51.218 7f6743d4a700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f674408ad30 0x7f674408b130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:13:51.217 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:51.218 7f6743d4a700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f674408ad30 0x7f674408b130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:52518/0 (socket says 172.21.2.102:52518) 2021-02-10T11:13:51.217 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:51.218 7f6743d4a700 1 -- 172.21.2.102:0/433164287 learned_addr learned my addr 172.21.2.102:0/433164287 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:13:51.217 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:51.219 7f6743d4a700 1 -- 172.21.2.102:0/433164287 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f67441a46e0 msgr2=0x7f674409f310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:13:51.217 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:51.219 7f6743d4a700 1 --2- 172.21.2.102:0/433164287 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f67441a46e0 0x7f674409f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:13:51.217 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:51.219 7f6743d4a700 1 -- 172.21.2.102:0/433164287 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f67441a42e0 msgr2=0x7f674409ecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:13:51.217 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:51.219 7f6743d4a700 1 --2- 172.21.2.102:0/433164287 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f67441a42e0 0x7f674409ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:13:51.217 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:51.219 7f6743d4a700 1 -- 172.21.2.102:0/433164287 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f674409fa50 con 0x7f674408ad30 2021-02-10T11:13:51.217 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:51.219 7f6743d4a700 1 --2- 172.21.2.102:0/433164287 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f674408ad30 0x7f674408b130 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 rx=0x7f673c00b8e0 tx=0x7f673c00bc80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:13:51.217 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:51.219 7f673bfff700 1 -- 172.21.2.102:0/433164287 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f673c000ce0 con 0x7f674408ad30 2021-02-10T11:13:51.217 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:51.219 7f673bfff700 1 -- 172.21.2.102:0/433164287 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f673c00c2f0 con 0x7f674408ad30 2021-02-10T11:13:51.218 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:51.219 7f6749737700 1 -- 172.21.2.102:0/433164287 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f674409fa50 con 0x7f674408ad30 2021-02-10T11:13:51.218 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:51.219 7f6749737700 1 -- 172.21.2.102:0/433164287 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f674408ba40 con 0x7f674408ad30 2021-02-10T11:13:51.218 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:51.219 7f673bfff700 1 -- 172.21.2.102:0/433164287 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f673c00c810 con 0x7f674408ad30 2021-02-10T11:13:51.218 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:51.219 7f673bfff700 1 -- 172.21.2.102:0/433164287 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f673c001220 con 0x7f674408ad30 2021-02-10T11:13:51.218 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:51.220 7f673bfff700 1 --2- 172.21.2.102:0/433164287 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f67280186c0 0x7f672801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:13:51.218 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:51.220 7f673bfff700 1 -- 172.21.2.102:0/433164287 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f673c032950 con 0x7f674408ad30 2021-02-10T11:13:51.218 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:51.220 7f6743549700 1 --2- 172.21.2.102:0/433164287 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f67280186c0 0x7f672801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:13:51.220 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:51.220 7f6743549700 1 --2- 172.21.2.102:0/433164287 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f67280186c0 0x7f672801abc0 crc :-1 s=READY pgs=285 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:13:51.246 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:51.251 7f6749737700 1 -- 172.21.2.102:0/433164287 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f671c002fd0 con 0x7f674408ad30 2021-02-10T11:13:51.248 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:51.254 7f673bfff700 1 -- 172.21.2.102:0/433164287 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f673c000cc0 con 0x7f674408ad30 2021-02-10T11:13:51.377 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:51.383 7f6749737700 1 -- 172.21.2.102:0/433164287 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344420", "target": ["mgr", ""]}) v1 -- 0x7f671c002f80 con 0x7f67280186c0 2021-02-10T11:13:51.380 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:51.386 7f673bfff700 1 -- 172.21.2.102:0/433164287 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f672c006e70 con 0x7f67280186c0 2021-02-10T11:13:51.380 INFO:teuthology.orchestra.run.gibba002.stdout:/volumes/_nogroup/subvolume_0000000000344420/b37de3b7-24f4-4df0-8f67-e76c3e3acf14 2021-02-10T11:13:51.382 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:51.387 7f6749737700 1 -- 172.21.2.102:0/433164287 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f67280186c0 msgr2=0x7f672801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:13:51.382 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:51.387 7f6749737700 1 --2- 172.21.2.102:0/433164287 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f67280186c0 0x7f672801abc0 crc :-1 s=READY pgs=285 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:51.382 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:51.387 7f6749737700 1 -- 172.21.2.102:0/433164287 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f674408ad30 msgr2=0x7f674408b130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:13:51.382 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:51.387 7f6749737700 1 --2- 172.21.2.102:0/433164287 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f674408ad30 0x7f674408b130 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 rx=0x7f673c00b8e0 tx=0x7f673c00bc80).stop 2021-02-10T11:13:51.382 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:51.387 7f6749737700 1 -- 172.21.2.102:0/433164287 shutdown_connections 2021-02-10T11:13:51.382 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:51.387 7f6749737700 1 --2- 172.21.2.102:0/433164287 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f674408ad30 0x7f674408b130 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:51.382 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:51.387 7f6749737700 1 --2- 172.21.2.102:0/433164287 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f67441a46e0 0x7f674409f310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:13:51.383 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:51.387 7f6749737700 1 --2- 172.21.2.102:0/433164287 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f67280186c0 0x7f672801abc0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:51.383 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:51.387 7f6749737700 1 --2- 172.21.2.102:0/433164287 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f67441a42e0 0x7f674409ecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:13:51.383 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:51.387 7f6749737700 1 -- 172.21.2.102:0/433164287 >> 172.21.2.102:0/433164287 conn(0x7f6744095a70 msgr2=0x7f67440bf050 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:13:51.383 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:51.388 7f6749737700 1 -- 172.21.2.102:0/433164287 shutdown_connections 2021-02-10T11:13:51.383 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:51.388 7f6749737700 1 -- 172.21.2.102:0/433164287 wait complete. 2021-02-10T11:13:51.390 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000344420 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000344420/b37de3b7-24f4-4df0-8f67-e76c3e3acf14 2021-02-10T11:13:51.390 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:13:51.391 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:13:51.391 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:13:51.728 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:13:51.729 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:13:51.729 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344420/b37de3b7-24f4-4df0-8f67-e76c3e3acf14/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:13:52.112 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:13:52.112 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:13:52.112 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0237184 s, 44.2 MB/s 2021-02-10T11:13:52.113 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:13:52.114 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:13:52.114 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:13:52.452 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:13:52.452 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:13:52.453 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344420/b37de3b7-24f4-4df0-8f67-e76c3e3acf14/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:13:52.830 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:13:52.830 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:13:52.830 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0244249 s, 42.9 MB/s 2021-02-10T11:13:52.831 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:13:52.831 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:13:52.832 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:13:53.168 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:13:53.169 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:13:53.169 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344420/b37de3b7-24f4-4df0-8f67-e76c3e3acf14/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:13:53.543 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:13:53.543 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:13:53.543 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0250123 s, 41.9 MB/s 2021-02-10T11:13:53.545 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:13:53.545 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:13:53.546 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:13:53.577 DEBUG:teuthology.orchestra.run.gibba002:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:13:53.590 DEBUG:teuthology.orchestra.run.gibba013:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:13:53.881 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:13:53.882 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:13:53.882 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344420/b37de3b7-24f4-4df0-8f67-e76c3e3acf14/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:13:54.255 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:13:54.256 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:13:54.256 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0234349 s, 44.7 MB/s 2021-02-10T11:13:54.257 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:13:54.257 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:13:54.257 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:13:54.593 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:13:54.593 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:13:54.593 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344420/b37de3b7-24f4-4df0-8f67-e76c3e3acf14/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:13:54.972 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:13:54.972 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:13:54.972 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0260652 s, 40.2 MB/s 2021-02-10T11:13:54.973 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:13:54.973 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:13:54.973 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:13:55.311 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:13:55.311 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:13:55.311 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344420/b37de3b7-24f4-4df0-8f67-e76c3e3acf14/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:13:55.686 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:13:55.686 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:13:55.687 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0223918 s, 46.8 MB/s 2021-02-10T11:13:55.687 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:13:55.688 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:13:55.688 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:13:56.023 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:13:56.023 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:13:56.024 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344420/b37de3b7-24f4-4df0-8f67-e76c3e3acf14/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:13:56.397 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:13:56.398 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:13:56.398 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0236616 s, 44.3 MB/s 2021-02-10T11:13:56.398 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:13:56.399 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:13:56.399 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:13:56.734 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:13:56.734 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:13:56.735 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344420/b37de3b7-24f4-4df0-8f67-e76c3e3acf14/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:13:57.109 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:13:57.109 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:13:57.109 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0231624 s, 45.3 MB/s 2021-02-10T11:13:57.110 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:13:57.110 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:13:57.110 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:13:57.446 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:13:57.447 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:13:57.447 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344420/b37de3b7-24f4-4df0-8f67-e76c3e3acf14/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:13:57.824 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:13:57.825 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:13:57.825 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0248673 s, 42.2 MB/s 2021-02-10T11:13:57.826 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:13:57.826 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:13:57.826 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:13:58.160 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:13:58.161 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:13:58.161 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344420/b37de3b7-24f4-4df0-8f67-e76c3e3acf14/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:13:58.535 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:13:58.536 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:13:58.536 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0252753 s, 41.5 MB/s 2021-02-10T11:13:58.537 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000344421 2021-02-10T11:13:58.970 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:58.958 7f0be29d5700 1 Processor -- start 2021-02-10T11:13:58.970 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:58.958 7f0be29d5700 1 -- start start 2021-02-10T11:13:58.970 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:58.959 7f0be29d5700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f0bdc1b0870 0x7f0bdc0f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:13:58.970 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:58.959 7f0be29d5700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bdc1b0c70 con 0x7f0bdc0efe00 2021-02-10T11:13:58.970 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:58.959 7f0be29d5700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bdc0f0b50 con 0x7f0bdc1b0470 2021-02-10T11:13:58.970 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:58.959 7f0be29d5700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0bdc0e5ea0 con 0x7f0bdc1b0870 2021-02-10T11:13:58.971 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:58.959 7f0be0ff2700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f0bdc0efe00 0x7f0bdc0f0200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:44748/0 (socket says 172.21.2.102:44748) 2021-02-10T11:13:58.971 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:58.959 7f0be0ff2700 1 -- 172.21.2.102:0/1017243024 learned_addr learned my addr 172.21.2.102:0/1017243024 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:13:58.971 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:58.959 7f0bdaffd700 1 -- 172.21.2.102:0/1017243024 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3974411886 0 0) 0x7f0bd4001180 con 0x7f0bdc0efe00 2021-02-10T11:13:58.971 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:58.959 7f0bdaffd700 1 -- 172.21.2.102:0/1017243024 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0bc8002fe0 con 0x7f0bdc0efe00 2021-02-10T11:13:58.971 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:58.959 7f0bdb7fe700 1 --2- 172.21.2.102:0/1017243024 >> v2:172.21.2.113:3300/0 conn(0x7f0bdc1b0870 0x7f0bdc0f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:13:58.971 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:58.960 7f0bdaffd700 1 -- 172.21.2.102:0/1017243024 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 71997632 0 0) 0x7f0bd4001180 con 0x7f0bdc0efe00 2021-02-10T11:13:58.971 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:58.960 7f0bdaffd700 1 -- 172.21.2.102:0/1017243024 >> v2:172.21.2.113:3300/0 conn(0x7f0bdc1b0870 msgr2=0x7f0bdc0f20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:13:58.971 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:58.960 7f0bdaffd700 1 --2- 172.21.2.102:0/1017243024 >> v2:172.21.2.113:3300/0 conn(0x7f0bdc1b0870 0x7f0bdc0f20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:58.971 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:58.960 7f0bdaffd700 1 -- 172.21.2.102:0/1017243024 >> v1:172.21.2.113:6789/0 conn(0x7f0bdc1b0470 legacy=0x7f0bdc0ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:13:58.971 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:58.960 7f0bdaffd700 1 -- 172.21.2.102:0/1017243024 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bdc0e52e0 con 0x7f0bdc0efe00 2021-02-10T11:13:58.971 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:58.960 7f0bdaffd700 1 -- 172.21.2.102:0/1017243024 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7f0bd4001680 con 0x7f0bdc0efe00 2021-02-10T11:13:58.972 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:58.960 7f0bdaffd700 1 -- 172.21.2.102:0/1017243024 >> v1:172.21.2.102:6789/0 conn(0x7f0bdc0efe00 legacy=0x7f0bdc0f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:13:58.972 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:58.960 7f0bdaffd700 1 --2- 172.21.2.102:0/1017243024 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0bc8003aa0 0x7f0bc8005ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:13:58.972 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:58.960 7f0bdaffd700 1 --2- 172.21.2.102:0/1017243024 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0bc8006790 0x7f0bc8008c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:13:58.972 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:58.960 7f0bdaffd700 1 --2- 172.21.2.102:0/1017243024 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0bc8009590 0x7f0bc800bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:13:58.972 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:58.960 7f0bdaffd700 1 -- 172.21.2.102:0/1017243024 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bc800c300 con 0x7f0bc8003aa0 2021-02-10T11:13:58.972 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:58.960 7f0bdaffd700 1 -- 172.21.2.102:0/1017243024 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bc800c520 con 0x7f0bc8009590 2021-02-10T11:13:58.972 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:58.960 7f0bdaffd700 1 -- 172.21.2.102:0/1017243024 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bc800c740 con 0x7f0bc8006790 2021-02-10T11:13:58.972 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:58.960 7f0bdbfff700 1 --2- 172.21.2.102:0/1017243024 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0bc8003aa0 0x7f0bc8005ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:13:58.972 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:58.960 7f0be0ff2700 1 --2- 172.21.2.102:0/1017243024 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0bc8009590 0x7f0bc800bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:13:58.972 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:58.960 7f0bdb7fe700 1 --2- 172.21.2.102:0/1017243024 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0bc8006790 0x7f0bc8008c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:13:58.972 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:58.960 7f0bdbfff700 1 -- 172.21.2.102:0/1017243024 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0bc8006790 msgr2=0x7f0bc8008c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:13:58.972 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:58.960 7f0bdbfff700 1 --2- 172.21.2.102:0/1017243024 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0bc8006790 0x7f0bc8008c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:58.972 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:58.960 7f0bdbfff700 1 -- 172.21.2.102:0/1017243024 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0bc8009590 msgr2=0x7f0bc800bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:13:58.973 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:58.960 7f0bdbfff700 1 --2- 172.21.2.102:0/1017243024 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0bc8009590 0x7f0bc800bab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:58.973 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:58.960 7f0be0ff2700 1 -- 172.21.2.102:0/1017243024 reap_dead start 2021-02-10T11:13:58.973 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:58.960 7f0bdbfff700 1 -- 172.21.2.102:0/1017243024 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bc8050980 con 0x7f0bc8003aa0 2021-02-10T11:13:58.973 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:58.960 7f0bdbfff700 1 --2- 172.21.2.102:0/1017243024 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0bc8003aa0 0x7f0bc8005ef0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 rx=0x7f0bcc006e30 tx=0x7f0bcc009e30).ready entity=mon.0 client_cookie=5f68e070bf2c1c21 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:13:58.973 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:58.961 7f0bdaffd700 1 -- 172.21.2.102:0/1017243024 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0bcc003030 con 0x7f0bc8003aa0 2021-02-10T11:13:58.973 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:58.961 7f0bdaffd700 1 -- 172.21.2.102:0/1017243024 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bcc006a20 con 0x7f0bc8003aa0 2021-02-10T11:13:58.973 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:58.961 7f0bdaffd700 1 -- 172.21.2.102:0/1017243024 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0bcc00b790 con 0x7f0bc8003aa0 2021-02-10T11:13:58.973 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:58.961 7f0be29d5700 1 -- 172.21.2.102:0/1017243024 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0bc8003aa0 msgr2=0x7f0bc8005ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:13:58.973 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:58.961 7f0be29d5700 1 --2- 172.21.2.102:0/1017243024 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0bc8003aa0 0x7f0bc8005ef0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 rx=0x7f0bcc006e30 tx=0x7f0bcc009e30).stop 2021-02-10T11:13:58.973 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:58.961 7f0be29d5700 1 -- 172.21.2.102:0/1017243024 shutdown_connections 2021-02-10T11:13:58.973 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:58.961 7f0be29d5700 1 --2- 172.21.2.102:0/1017243024 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0bc8003aa0 0x7f0bc8005ef0 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:58.973 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:58.961 7f0be29d5700 1 -- 172.21.2.102:0/1017243024 >> 172.21.2.102:0/1017243024 conn(0x7f0bdc0ce3e0 msgr2=0x7f0bdc0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:13:58.973 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:58.961 7f0be29d5700 1 -- 172.21.2.102:0/1017243024 shutdown_connections 2021-02-10T11:13:58.974 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:58.961 7f0be29d5700 1 -- 172.21.2.102:0/1017243024 wait complete. 2021-02-10T11:13:58.975 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:58.976 7f0be29d5700 1 Processor -- start 2021-02-10T11:13:58.975 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:58.976 7f0be29d5700 1 -- start start 2021-02-10T11:13:58.975 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:58.976 7f0be29d5700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0bdc09b930 0x7f0bdc0b7c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:13:58.976 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:58.976 7f0be29d5700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0bdc0b8410 0x7f0bdc0b8810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:13:58.976 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:58.976 7f0be29d5700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0bdc0c1f80 0x7f0bdc0c2380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:13:58.976 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:58.976 7f0be29d5700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bdc0c1a00 con 0x7f0bdc0b8410 2021-02-10T11:13:58.976 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:58.976 7f0be29d5700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bdc0b8e30 con 0x7f0bdc09b930 2021-02-10T11:13:58.976 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:58.976 7f0be29d5700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bdc0b9050 con 0x7f0bdc0c1f80 2021-02-10T11:13:58.976 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:58.977 7f0bdb7fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0bdc0b8410 0x7f0bdc0b8810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:13:58.976 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:58.977 7f0bdb7fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0bdc0b8410 0x7f0bdc0b8810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:52536/0 (socket says 172.21.2.102:52536) 2021-02-10T11:13:58.976 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:58.977 7f0bdb7fe700 1 -- 172.21.2.102:0/166002054 learned_addr learned my addr 172.21.2.102:0/166002054 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:13:58.976 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:58.977 7f0bdb7fe700 1 -- 172.21.2.102:0/166002054 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0bdc0c1f80 msgr2=0x7f0bdc0c2380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:13:58.976 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:58.977 7f0bdb7fe700 1 --2- 172.21.2.102:0/166002054 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0bdc0c1f80 0x7f0bdc0c2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:13:58.976 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:58.977 7f0bdb7fe700 1 -- 172.21.2.102:0/166002054 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0bdc09b930 msgr2=0x7f0bdc0b7c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:13:58.976 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:58.977 7f0bdbfff700 1 --2- 172.21.2.102:0/166002054 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0bdc09b930 0x7f0bdc0b7c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:13:58.977 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:58.977 7f0bdb7fe700 1 --2- 172.21.2.102:0/166002054 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0bdc09b930 0x7f0bdc0b7c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:58.977 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:58.977 7f0bdb7fe700 1 -- 172.21.2.102:0/166002054 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bdc0c29a0 con 0x7f0bdc0b8410 2021-02-10T11:13:58.977 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:58.977 7f0bdbfff700 1 --2- 172.21.2.102:0/166002054 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0bdc09b930 0x7f0bdc0b7c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:13:58.977 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:58.977 7f0bdb7fe700 1 --2- 172.21.2.102:0/166002054 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0bdc0b8410 0x7f0bdc0b8810 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 rx=0x7f0bc400b560 tx=0x7f0bc400b900).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:13:58.977 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:58.978 7f0bd8ff9700 1 -- 172.21.2.102:0/166002054 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0bc40009b0 con 0x7f0bdc0b8410 2021-02-10T11:13:58.977 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:58.978 7f0bd8ff9700 1 -- 172.21.2.102:0/166002054 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bc4006bb0 con 0x7f0bdc0b8410 2021-02-10T11:13:58.977 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:58.978 7f0bd8ff9700 1 -- 172.21.2.102:0/166002054 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0bc4002030 con 0x7f0bdc0b8410 2021-02-10T11:13:58.977 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:58.978 7f0be29d5700 1 -- 172.21.2.102:0/166002054 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bdc0c29a0 con 0x7f0bdc0b8410 2021-02-10T11:13:58.977 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:58.978 7f0be29d5700 1 -- 172.21.2.102:0/166002054 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bdc0c29a0 con 0x7f0bdc0b8410 2021-02-10T11:13:58.977 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:58.978 7f0bd8ff9700 1 -- 172.21.2.102:0/166002054 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f0bc40052e0 con 0x7f0bdc0b8410 2021-02-10T11:13:58.977 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:58.979 7f0bd8ff9700 1 --2- 172.21.2.102:0/166002054 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0bc00186c0 0x7f0bc001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:13:58.977 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:58.979 7f0bd8ff9700 1 -- 172.21.2.102:0/166002054 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f0bc4032510 con 0x7f0bdc0b8410 2021-02-10T11:13:58.977 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:58.979 7f0bdbfff700 1 --2- 172.21.2.102:0/166002054 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0bc00186c0 0x7f0bc001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:13:58.978 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:58.979 7f0bdbfff700 1 --2- 172.21.2.102:0/166002054 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0bc00186c0 0x7f0bc001abc0 crc :-1 s=READY pgs=286 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:13:59.003 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:59.008 7f0be29d5700 1 -- 172.21.2.102:0/166002054 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bdc105b30 con 0x7f0bdc0b8410 2021-02-10T11:13:59.005 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:59.011 7f0bd8ff9700 1 -- 172.21.2.102:0/166002054 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f0bc402e030 con 0x7f0bdc0b8410 2021-02-10T11:13:59.131 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:59.137 7f0be29d5700 1 -- 172.21.2.102:0/166002054 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344421", "target": ["mgr", ""]}) v1 -- 0x7f0bdc105b30 con 0x7f0bc00186c0 2021-02-10T11:13:59.158 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:59.163 7f0bd8ff9700 1 -- 172.21.2.102:0/166002054 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f0bcc001020 con 0x7f0bc00186c0 2021-02-10T11:13:59.164 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:59.170 7f0be29d5700 1 -- 172.21.2.102:0/166002054 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0bc00186c0 msgr2=0x7f0bc001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:13:59.165 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:59.170 7f0be29d5700 1 --2- 172.21.2.102:0/166002054 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0bc00186c0 0x7f0bc001abc0 crc :-1 s=READY pgs=286 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:59.165 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:59.170 7f0be29d5700 1 -- 172.21.2.102:0/166002054 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0bdc0b8410 msgr2=0x7f0bdc0b8810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:13:59.165 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:59.170 7f0be29d5700 1 --2- 172.21.2.102:0/166002054 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0bdc0b8410 0x7f0bdc0b8810 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 rx=0x7f0bc400b560 tx=0x7f0bc400b900).stop 2021-02-10T11:13:59.165 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:59.170 7f0be29d5700 1 -- 172.21.2.102:0/166002054 shutdown_connections 2021-02-10T11:13:59.165 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:59.170 7f0be29d5700 1 --2- 172.21.2.102:0/166002054 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0bdc0c1f80 0x7f0bdc0c2380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:13:59.165 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:59.170 7f0be29d5700 1 --2- 172.21.2.102:0/166002054 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0bdc0b8410 0x7f0bdc0b8810 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:59.165 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:59.170 7f0be29d5700 1 --2- 172.21.2.102:0/166002054 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0bc00186c0 0x7f0bc001abc0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:59.165 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:59.170 7f0be29d5700 1 --2- 172.21.2.102:0/166002054 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0bdc09b930 0x7f0bdc0b7c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:59.165 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:59.170 7f0be29d5700 1 -- 172.21.2.102:0/166002054 >> 172.21.2.102:0/166002054 conn(0x7f0bdc094e10 msgr2=0x7f0bdc0a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:13:59.166 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:59.171 7f0be29d5700 1 -- 172.21.2.102:0/166002054 shutdown_connections 2021-02-10T11:13:59.166 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:59.171 7f0be29d5700 1 -- 172.21.2.102:0/166002054 wait complete. 2021-02-10T11:13:59.172 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000344421 2021-02-10T11:13:59.605 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:59.594 7f7c37817700 1 Processor -- start 2021-02-10T11:13:59.605 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:59.594 7f7c37817700 1 -- start start 2021-02-10T11:13:59.605 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:59.594 7f7c37817700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f7c301b0470 0x7f7c301b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:13:59.605 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:59.594 7f7c37817700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f7c300f20e0 0x7f7c300f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:13:59.605 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:59.594 7f7c37817700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c300f0b50 con 0x7f7c300ef610 2021-02-10T11:13:59.605 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:59.594 7f7c37817700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c300f0820 con 0x7f7c300f20e0 2021-02-10T11:13:59.605 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:59.594 7f7c37817700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c300f24e0 con 0x7f7c301b0470 2021-02-10T11:13:59.605 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:59.595 7f7c35e34700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f7c300f20e0 0x7f7c300f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:13:59.606 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:59.595 7f7c35e34700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f7c300f20e0 0x7f7c300f0200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:52540/0 (socket says 172.21.2.102:52540) 2021-02-10T11:13:59.606 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:59.595 7f7c35e34700 1 -- 172.21.2.102:0/1286069416 learned_addr learned my addr 172.21.2.102:0/1286069416 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:13:59.606 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:59.595 7f7c35633700 1 --2- 172.21.2.102:0/1286069416 >> v2:172.21.2.113:3300/0 conn(0x7f7c301b0470 0x7f7c301b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:13:59.606 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:59.595 7f7c35e34700 1 -- 172.21.2.102:0/1286069416 >> v2:172.21.2.113:3300/0 conn(0x7f7c301b0470 msgr2=0x7f7c301b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:13:59.606 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:59.595 7f7c35e34700 1 --2- 172.21.2.102:0/1286069416 >> v2:172.21.2.113:3300/0 conn(0x7f7c301b0470 0x7f7c301b0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:59.606 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:59.595 7f7c35e34700 1 -- 172.21.2.102:0/1286069416 >> v1:172.21.2.113:6789/0 conn(0x7f7c300ef610 legacy=0x7f7c300efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:13:59.606 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:59.595 7f7c35e34700 1 -- 172.21.2.102:0/1286069416 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c300e52e0 con 0x7f7c300f20e0 2021-02-10T11:13:59.606 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:59.595 7f7c35e34700 1 --2- 172.21.2.102:0/1286069416 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7c300f20e0 0x7f7c300f0200 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 rx=0x7f7c28006f00 tx=0x7f7c28009e90).ready entity=mon.0 client_cookie=9deb74d422daf075 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:13:59.606 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:59.595 7f7c23fff700 1 -- 172.21.2.102:0/1286069416 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7c2800b300 con 0x7f7c300f20e0 2021-02-10T11:13:59.606 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:59.595 7f7c23fff700 1 -- 172.21.2.102:0/1286069416 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c2800b580 con 0x7f7c300f20e0 2021-02-10T11:13:59.606 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:59.595 7f7c23fff700 1 -- 172.21.2.102:0/1286069416 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7c2800b9d0 con 0x7f7c300f20e0 2021-02-10T11:13:59.606 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:59.595 7f7c37817700 1 -- 172.21.2.102:0/1286069416 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7c300f20e0 msgr2=0x7f7c300f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:13:59.607 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:59.595 7f7c37817700 1 --2- 172.21.2.102:0/1286069416 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7c300f20e0 0x7f7c300f0200 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 rx=0x7f7c28006f00 tx=0x7f7c28009e90).stop 2021-02-10T11:13:59.607 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:59.596 7f7c37817700 1 -- 172.21.2.102:0/1286069416 shutdown_connections 2021-02-10T11:13:59.607 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:59.596 7f7c37817700 1 --2- 172.21.2.102:0/1286069416 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7c300f20e0 0x7f7c300f0200 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:59.607 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:59.596 7f7c37817700 1 --2- 172.21.2.102:0/1286069416 >> v2:172.21.2.113:3300/0 conn(0x7f7c301b0470 0x7f7c301b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:59.607 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:59.596 7f7c37817700 1 -- 172.21.2.102:0/1286069416 >> 172.21.2.102:0/1286069416 conn(0x7f7c300ce3e0 msgr2=0x7f7c300e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:13:59.607 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:59.596 7f7c37817700 1 -- 172.21.2.102:0/1286069416 shutdown_connections 2021-02-10T11:13:59.607 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:59.596 7f7c37817700 1 -- 172.21.2.102:0/1286069416 wait complete. 2021-02-10T11:13:59.608 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:59.611 7f7c37817700 1 Processor -- start 2021-02-10T11:13:59.608 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:59.611 7f7c37817700 1 -- start start 2021-02-10T11:13:59.608 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:59.611 7f7c37817700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7c300a78f0 0x7f7c30109110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:13:59.608 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:59.611 7f7c37817700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7c300a7cf0 0x7f7c300a8770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:13:59.609 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:59.611 7f7c37817700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7c300a8d90 0x7f7c300a9190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:13:59.609 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:59.611 7f7c37817700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c30109c00 con 0x7f7c300a78f0 2021-02-10T11:13:59.609 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:59.611 7f7c37817700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c300a81f0 con 0x7f7c300a8d90 2021-02-10T11:13:59.609 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:59.611 7f7c37817700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c30109730 con 0x7f7c300a7cf0 2021-02-10T11:13:59.609 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:59.611 7f7c35633700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7c300a78f0 0x7f7c30109110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:13:59.609 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:59.612 7f7c35633700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7c300a78f0 0x7f7c30109110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:52544/0 (socket says 172.21.2.102:52544) 2021-02-10T11:13:59.609 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:59.612 7f7c35633700 1 -- 172.21.2.102:0/1329268837 learned_addr learned my addr 172.21.2.102:0/1329268837 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:13:59.609 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:59.612 7f7c35633700 1 -- 172.21.2.102:0/1329268837 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7c300a7cf0 msgr2=0x7f7c300a8770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:13:59.609 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:59.612 7f7c35633700 1 --2- 172.21.2.102:0/1329268837 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7c300a7cf0 0x7f7c300a8770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:13:59.609 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:59.612 7f7c35633700 1 -- 172.21.2.102:0/1329268837 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7c300a8d90 msgr2=0x7f7c300a9190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:13:59.609 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:59.612 7f7c35633700 1 --2- 172.21.2.102:0/1329268837 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7c300a8d90 0x7f7c300a9190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:13:59.609 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:59.612 7f7c35633700 1 -- 172.21.2.102:0/1329268837 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c30193ed0 con 0x7f7c300a78f0 2021-02-10T11:13:59.609 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:59.612 7f7c35633700 1 --2- 172.21.2.102:0/1329268837 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7c300a78f0 0x7f7c30109110 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 rx=0x7f7c2c006f10 tx=0x7f7c2c00b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:13:59.609 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:59.612 7f7c21ffb700 1 -- 172.21.2.102:0/1329268837 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7c2c00bd80 con 0x7f7c300a78f0 2021-02-10T11:13:59.610 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:59.612 7f7c21ffb700 1 -- 172.21.2.102:0/1329268837 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c2c000c90 con 0x7f7c300a78f0 2021-02-10T11:13:59.610 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:59.612 7f7c21ffb700 1 -- 172.21.2.102:0/1329268837 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7c2c002220 con 0x7f7c300a78f0 2021-02-10T11:13:59.610 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:59.612 7f7c37817700 1 -- 172.21.2.102:0/1329268837 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c30193ed0 con 0x7f7c300a78f0 2021-02-10T11:13:59.610 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:59.612 7f7c37817700 1 -- 172.21.2.102:0/1329268837 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c300a9aa0 con 0x7f7c300a78f0 2021-02-10T11:13:59.610 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:59.613 7f7c21ffb700 1 -- 172.21.2.102:0/1329268837 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f7c2c0052e0 con 0x7f7c300a78f0 2021-02-10T11:13:59.610 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:59.613 7f7c21ffb700 1 --2- 172.21.2.102:0/1329268837 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7c100186c0 0x7f7c1001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:13:59.610 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:59.613 7f7c21ffb700 1 -- 172.21.2.102:0/1329268837 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f7c2c0323f0 con 0x7f7c300a78f0 2021-02-10T11:13:59.610 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:59.613 7f7c34e32700 1 --2- 172.21.2.102:0/1329268837 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7c100186c0 0x7f7c1001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:13:59.610 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:59.614 7f7c34e32700 1 --2- 172.21.2.102:0/1329268837 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7c100186c0 0x7f7c1001abc0 crc :-1 s=READY pgs=287 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:13:59.639 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:59.644 7f7c37817700 1 -- 172.21.2.102:0/1329268837 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c300f0b50 con 0x7f7c300a78f0 2021-02-10T11:13:59.641 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:59.647 7f7c21ffb700 1 -- 172.21.2.102:0/1329268837 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f7c2c000960 con 0x7f7c300a78f0 2021-02-10T11:13:59.770 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:59.775 7f7c37817700 1 -- 172.21.2.102:0/1329268837 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344421", "target": ["mgr", ""]}) v1 -- 0x7f7c300f0b50 con 0x7f7c100186c0 2021-02-10T11:13:59.774 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:59.779 7f7c21ffb700 1 -- 172.21.2.102:0/1329268837 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f7c24006e40 con 0x7f7c100186c0 2021-02-10T11:13:59.777 INFO:teuthology.orchestra.run.gibba002.stdout:/volumes/_nogroup/subvolume_0000000000344421/752701da-4249-4ad5-b850-2433b9830516 2021-02-10T11:13:59.778 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:59.784 7f7c37817700 1 -- 172.21.2.102:0/1329268837 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7c100186c0 msgr2=0x7f7c1001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:13:59.779 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:59.784 7f7c37817700 1 --2- 172.21.2.102:0/1329268837 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7c100186c0 0x7f7c1001abc0 crc :-1 s=READY pgs=287 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:59.779 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:59.784 7f7c37817700 1 -- 172.21.2.102:0/1329268837 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7c300a78f0 msgr2=0x7f7c30109110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:13:59.779 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:59.784 7f7c37817700 1 --2- 172.21.2.102:0/1329268837 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7c300a78f0 0x7f7c30109110 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 rx=0x7f7c2c006f10 tx=0x7f7c2c00b990).stop 2021-02-10T11:13:59.779 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:59.785 7f7c37817700 1 -- 172.21.2.102:0/1329268837 shutdown_connections 2021-02-10T11:13:59.779 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:59.785 7f7c37817700 1 --2- 172.21.2.102:0/1329268837 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7c100186c0 0x7f7c1001abc0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:59.779 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:59.785 7f7c37817700 1 --2- 172.21.2.102:0/1329268837 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7c300a8d90 0x7f7c300a9190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:13:59.780 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:59.785 7f7c37817700 1 --2- 172.21.2.102:0/1329268837 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7c300a7cf0 0x7f7c300a8770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:13:59.780 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:59.785 7f7c37817700 1 --2- 172.21.2.102:0/1329268837 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7c300a78f0 0x7f7c30109110 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:13:59.780 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:59.785 7f7c37817700 1 -- 172.21.2.102:0/1329268837 >> 172.21.2.102:0/1329268837 conn(0x7f7c300bf4e0 msgr2=0x7f7c300b7550 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:13:59.780 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:59.785 7f7c37817700 1 -- 172.21.2.102:0/1329268837 shutdown_connections 2021-02-10T11:13:59.780 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:13:59.785 7f7c37817700 1 -- 172.21.2.102:0/1329268837 wait complete. 2021-02-10T11:13:59.787 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000344421 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000344421/752701da-4249-4ad5-b850-2433b9830516 2021-02-10T11:13:59.787 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:13:59.787 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:13:59.787 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:14:00.124 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:14:00.125 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:14:00.125 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344421/752701da-4249-4ad5-b850-2433b9830516/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:14:00.508 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:14:00.508 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:14:00.508 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0229924 s, 45.6 MB/s 2021-02-10T11:14:00.509 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:14:00.509 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:14:00.509 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:14:00.845 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:14:00.845 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:14:00.846 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344421/752701da-4249-4ad5-b850-2433b9830516/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:14:01.220 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:14:01.220 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:14:01.220 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0227341 s, 46.1 MB/s 2021-02-10T11:14:01.221 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:14:01.221 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:14:01.221 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:14:01.556 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:14:01.557 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:14:01.557 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344421/752701da-4249-4ad5-b850-2433b9830516/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:14:01.932 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:14:01.933 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:14:01.933 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0236416 s, 44.4 MB/s 2021-02-10T11:14:01.934 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:14:01.934 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:14:01.934 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:14:02.270 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:14:02.270 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:14:02.271 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344421/752701da-4249-4ad5-b850-2433b9830516/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:14:02.649 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:14:02.649 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:14:02.649 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0253171 s, 41.4 MB/s 2021-02-10T11:14:02.650 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:14:02.650 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:14:02.650 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:14:02.985 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:14:02.985 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:14:02.985 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344421/752701da-4249-4ad5-b850-2433b9830516/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:14:03.360 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:14:03.360 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:14:03.360 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.024711 s, 42.4 MB/s 2021-02-10T11:14:03.361 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:14:03.361 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:14:03.361 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:14:03.697 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:14:03.698 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:14:03.698 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344421/752701da-4249-4ad5-b850-2433b9830516/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:14:04.072 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:14:04.072 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:14:04.072 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0232389 s, 45.1 MB/s 2021-02-10T11:14:04.073 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:14:04.073 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:14:04.073 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:14:04.410 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:14:04.410 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:14:04.411 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344421/752701da-4249-4ad5-b850-2433b9830516/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:14:04.788 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:14:04.789 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:14:04.789 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0249446 s, 42.0 MB/s 2021-02-10T11:14:04.790 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:14:04.790 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:14:04.790 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:14:05.127 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:14:05.128 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:14:05.128 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344421/752701da-4249-4ad5-b850-2433b9830516/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:14:05.505 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:14:08.167 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:14:08.167 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.025349 s, 41.4 MB/s 2021-02-10T11:14:08.168 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:14:08.169 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:14:08.169 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:14:08.509 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:14:08.510 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:14:08.510 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344421/752701da-4249-4ad5-b850-2433b9830516/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:14:08.889 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:14:08.889 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:14:08.889 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0253062 s, 41.4 MB/s 2021-02-10T11:14:08.890 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:14:08.890 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:14:08.890 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:14:09.227 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:14:09.227 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:14:09.227 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344421/752701da-4249-4ad5-b850-2433b9830516/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:14:09.610 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:14:09.610 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:14:09.610 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0265503 s, 39.5 MB/s 2021-02-10T11:14:09.611 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000344422 2021-02-10T11:14:10.044 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:10.033 7fdfd8c74700 1 Processor -- start 2021-02-10T11:14:10.044 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:10.033 7fdfd8c74700 1 -- start start 2021-02-10T11:14:10.044 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:10.033 7fdfd8c74700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdfd41b0470 0x7fdfd41b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:10.044 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:10.033 7fdfd8c74700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fdfd40f20e0 0x7fdfd40f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:10.044 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:10.033 7fdfd8c74700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfd40f0b50 con 0x7fdfd40ef610 2021-02-10T11:14:10.044 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:10.033 7fdfd8c74700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdfd40f0820 con 0x7fdfd40f20e0 2021-02-10T11:14:10.044 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:10.033 7fdfd8c74700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdfd40f24e0 con 0x7fdfd41b0470 2021-02-10T11:14:10.044 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:10.033 7fdfd305f700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fdfd40f20e0 0x7fdfd40f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:10.044 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:10.033 7fdfd205d700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fdfd40ef610 0x7fdfd40efa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:44778/0 (socket says 172.21.2.102:44778) 2021-02-10T11:14:10.045 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:10.034 7fdfd305f700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fdfd40f20e0 0x7fdfd40f0200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:52556/0 (socket says 172.21.2.102:52556) 2021-02-10T11:14:10.045 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:10.034 7fdfd205d700 1 -- 172.21.2.102:0/4091359905 learned_addr learned my addr 172.21.2.102:0/4091359905 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:14:10.045 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:10.034 7fdfd285e700 1 --2- 172.21.2.102:0/4091359905 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdfd41b0470 0x7fdfd41b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:10.045 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:10.034 7fdfd305f700 1 -- 172.21.2.102:0/4091359905 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdfd41b0470 msgr2=0x7fdfd41b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:14:10.045 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:10.034 7fdfd305f700 1 --2- 172.21.2.102:0/4091359905 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdfd41b0470 0x7fdfd41b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:10.045 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:10.034 7fdfd305f700 1 -- 172.21.2.102:0/4091359905 >> v1:172.21.2.102:6789/0 conn(0x7fdfd40ef610 legacy=0x7fdfd40efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:10.045 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:10.034 7fdfd305f700 1 -- 172.21.2.102:0/4091359905 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfd40e52e0 con 0x7fdfd40f20e0 2021-02-10T11:14:10.045 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:10.034 7fdfd305f700 1 --2- 172.21.2.102:0/4091359905 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdfd40f20e0 0x7fdfd40f0200 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 rx=0x7fdfcc006e30 tx=0x7fdfcc009de0).ready entity=mon.0 client_cookie=4e99caf76b52a944 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:10.045 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:10.034 7fdfd185c700 1 -- 172.21.2.102:0/4091359905 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdfcc00b300 con 0x7fdfd40f20e0 2021-02-10T11:14:10.045 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:10.034 7fdfd185c700 1 -- 172.21.2.102:0/4091359905 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfcc00b580 con 0x7fdfd40f20e0 2021-02-10T11:14:10.045 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:10.034 7fdfd185c700 1 -- 172.21.2.102:0/4091359905 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdfcc00baa0 con 0x7fdfd40f20e0 2021-02-10T11:14:10.045 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:10.034 7fdfd8c74700 1 -- 172.21.2.102:0/4091359905 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdfd40f20e0 msgr2=0x7fdfd40f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:10.045 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:10.034 7fdfd8c74700 1 --2- 172.21.2.102:0/4091359905 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdfd40f20e0 0x7fdfd40f0200 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 rx=0x7fdfcc006e30 tx=0x7fdfcc009de0).stop 2021-02-10T11:14:10.046 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:10.035 7fdfd8c74700 1 -- 172.21.2.102:0/4091359905 shutdown_connections 2021-02-10T11:14:10.046 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:10.035 7fdfd8c74700 1 --2- 172.21.2.102:0/4091359905 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdfd40f20e0 0x7fdfd40f0200 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:10.046 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:10.035 7fdfd8c74700 1 --2- 172.21.2.102:0/4091359905 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdfd41b0470 0x7fdfd41b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:10.046 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:10.035 7fdfd8c74700 1 -- 172.21.2.102:0/4091359905 >> 172.21.2.102:0/4091359905 conn(0x7fdfd40ce3e0 msgr2=0x7fdfd40e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:14:10.046 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:10.035 7fdfd8c74700 1 -- 172.21.2.102:0/4091359905 shutdown_connections 2021-02-10T11:14:10.046 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:10.035 7fdfd8c74700 1 -- 172.21.2.102:0/4091359905 wait complete. 2021-02-10T11:14:10.047 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:10.050 7fdfd8c74700 1 Processor -- start 2021-02-10T11:14:10.048 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:10.050 7fdfd8c74700 1 -- start start 2021-02-10T11:14:10.048 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:10.050 7fdfd8c74700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fdfd40b7c20 0x7fdfd40b8410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:10.048 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:10.050 7fdfd8c74700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdfd40b8a30 0x7fdfd40c1f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:10.048 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:10.050 7fdfd8c74700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdfd40c25a0 0x7fdfd40c29a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:10.048 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:10.050 7fdfd8c74700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfd40c1a80 con 0x7fdfd40c25a0 2021-02-10T11:14:10.048 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:10.050 7fdfd8c74700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfd40b8020 con 0x7fdfd40b7c20 2021-02-10T11:14:10.048 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:10.050 7fdfd8c74700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdfd409b9c0 con 0x7fdfd40b8a30 2021-02-10T11:14:10.048 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:10.050 7fdfd305f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdfd40c25a0 0x7fdfd40c29a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:10.048 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:10.051 7fdfd305f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdfd40c25a0 0x7fdfd40c29a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:52560/0 (socket says 172.21.2.102:52560) 2021-02-10T11:14:10.048 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:10.051 7fdfd305f700 1 -- 172.21.2.102:0/1534798318 learned_addr learned my addr 172.21.2.102:0/1534798318 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:14:10.048 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:10.051 7fdfd285e700 1 --2- 172.21.2.102:0/1534798318 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fdfd40b7c20 0x7fdfd40b8410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:10.048 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:10.051 7fdfd205d700 1 --2- 172.21.2.102:0/1534798318 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdfd40b8a30 0x7fdfd40c1f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:10.048 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:10.051 7fdfd305f700 1 -- 172.21.2.102:0/1534798318 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdfd40b8a30 msgr2=0x7fdfd40c1f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:10.049 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:10.051 7fdfd305f700 1 --2- 172.21.2.102:0/1534798318 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdfd40b8a30 0x7fdfd40c1f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:10.049 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:10.051 7fdfd305f700 1 -- 172.21.2.102:0/1534798318 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fdfd40b7c20 msgr2=0x7fdfd40b8410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:10.049 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:10.051 7fdfd305f700 1 --2- 172.21.2.102:0/1534798318 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fdfd40b7c20 0x7fdfd40b8410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:10.049 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:10.051 7fdfd305f700 1 -- 172.21.2.102:0/1534798318 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfd40b8e30 con 0x7fdfd40c25a0 2021-02-10T11:14:10.049 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:10.051 7fdfd305f700 1 --2- 172.21.2.102:0/1534798318 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdfd40c25a0 0x7fdfd40c29a0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 rx=0x7fdfcc00b970 tx=0x7fdfcc00bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:10.049 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:10.051 7fdfc37fe700 1 -- 172.21.2.102:0/1534798318 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdfcc009380 con 0x7fdfd40c25a0 2021-02-10T11:14:10.049 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:10.051 7fdfc37fe700 1 -- 172.21.2.102:0/1534798318 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfcc005820 con 0x7fdfd40c25a0 2021-02-10T11:14:10.049 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:10.051 7fdfd8c74700 1 -- 172.21.2.102:0/1534798318 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfd40b8e30 con 0x7fdfd40c25a0 2021-02-10T11:14:10.049 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:10.051 7fdfc37fe700 1 -- 172.21.2.102:0/1534798318 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdfcc005d40 con 0x7fdfd40c25a0 2021-02-10T11:14:10.049 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:10.051 7fdfd8c74700 1 -- 172.21.2.102:0/1534798318 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfd40c32b0 con 0x7fdfd40c25a0 2021-02-10T11:14:10.049 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:10.052 7fdfc37fe700 1 -- 172.21.2.102:0/1534798318 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fdfcc005cd0 con 0x7fdfd40c25a0 2021-02-10T11:14:10.049 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:10.052 7fdfc37fe700 1 --2- 172.21.2.102:0/1534798318 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fdfb80186c0 0x7fdfb801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:10.049 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:10.052 7fdfc37fe700 1 -- 172.21.2.102:0/1534798318 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fdfcc032940 con 0x7fdfd40c25a0 2021-02-10T11:14:10.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:10.052 7fdfd285e700 1 --2- 172.21.2.102:0/1534798318 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fdfb80186c0 0x7fdfb801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:10.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:10.053 7fdfd285e700 1 --2- 172.21.2.102:0/1534798318 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fdfb80186c0 0x7fdfb801abc0 crc :-1 s=READY pgs=288 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:10.077 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:10.083 7fdfd8c74700 1 -- 172.21.2.102:0/1534798318 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdfd4105b30 con 0x7fdfd40c25a0 2021-02-10T11:14:10.080 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:10.086 7fdfc37fe700 1 -- 172.21.2.102:0/1534798318 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fdfcc025f10 con 0x7fdfd40c25a0 2021-02-10T11:14:10.207 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:10.212 7fdfd8c74700 1 -- 172.21.2.102:0/1534798318 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344422", "target": ["mgr", ""]}) v1 -- 0x7fdfd40b8020 con 0x7fdfb80186c0 2021-02-10T11:14:10.233 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:10.238 7fdfc37fe700 1 -- 172.21.2.102:0/1534798318 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fdfc8006a70 con 0x7fdfb80186c0 2021-02-10T11:14:10.240 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:10.245 7fdfd8c74700 1 -- 172.21.2.102:0/1534798318 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fdfb80186c0 msgr2=0x7fdfb801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:10.240 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:10.245 7fdfd8c74700 1 --2- 172.21.2.102:0/1534798318 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fdfb80186c0 0x7fdfb801abc0 crc :-1 s=READY pgs=288 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:10.240 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:10.245 7fdfd8c74700 1 -- 172.21.2.102:0/1534798318 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdfd40c25a0 msgr2=0x7fdfd40c29a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:10.240 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:10.245 7fdfd8c74700 1 --2- 172.21.2.102:0/1534798318 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdfd40c25a0 0x7fdfd40c29a0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 rx=0x7fdfcc00b970 tx=0x7fdfcc00bd10).stop 2021-02-10T11:14:10.240 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:10.245 7fdfd8c74700 1 -- 172.21.2.102:0/1534798318 shutdown_connections 2021-02-10T11:14:10.240 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:10.245 7fdfd8c74700 1 --2- 172.21.2.102:0/1534798318 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdfd40c25a0 0x7fdfd40c29a0 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:10.240 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:10.245 7fdfd8c74700 1 --2- 172.21.2.102:0/1534798318 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdfd40b8a30 0x7fdfd40c1f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:10.240 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:10.245 7fdfd8c74700 1 --2- 172.21.2.102:0/1534798318 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fdfb80186c0 0x7fdfb801abc0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:10.241 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:10.245 7fdfd8c74700 1 --2- 172.21.2.102:0/1534798318 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fdfd40b7c20 0x7fdfd40b8410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:10.241 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:10.245 7fdfd8c74700 1 -- 172.21.2.102:0/1534798318 >> 172.21.2.102:0/1534798318 conn(0x7fdfd4094e10 msgr2=0x7fdfd40a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:14:10.241 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:10.246 7fdfd8c74700 1 -- 172.21.2.102:0/1534798318 shutdown_connections 2021-02-10T11:14:10.241 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:10.246 7fdfd8c74700 1 -- 172.21.2.102:0/1534798318 wait complete. 2021-02-10T11:14:10.248 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000344422 2021-02-10T11:14:10.682 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:10.671 7f5e4703d700 1 Processor -- start 2021-02-10T11:14:10.683 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:10.671 7f5e4703d700 1 -- start start 2021-02-10T11:14:10.683 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:10.672 7f5e4703d700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f5e401b0870 0x7f5e400f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:10.683 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:10.672 7f5e4703d700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e401b0c70 con 0x7f5e400efe00 2021-02-10T11:14:10.683 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:10.672 7f5e4703d700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e400f0b50 con 0x7f5e401b0470 2021-02-10T11:14:10.683 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:10.672 7f5e4703d700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e400e5ea0 con 0x7f5e401b0870 2021-02-10T11:14:10.683 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:10.672 7f5e3ffff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f5e401b0870 0x7f5e400f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:10.683 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:10.672 7f5e4565a700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f5e400efe00 0x7f5e400f0200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:44790/0 (socket says 172.21.2.102:44790) 2021-02-10T11:14:10.683 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:10.672 7f5e4565a700 1 -- 172.21.2.102:0/858131015 learned_addr learned my addr 172.21.2.102:0/858131015 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:14:10.683 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:10.673 7f5e3ffff700 1 -- 172.21.2.102:0/858131015 >> v1:172.21.2.113:6789/0 conn(0x7f5e401b0470 legacy=0x7f5e400ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:10.683 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:10.673 7f5e3ffff700 1 -- 172.21.2.102:0/858131015 >> v1:172.21.2.102:6789/0 conn(0x7f5e400efe00 legacy=0x7f5e400f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:10.683 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:10.673 7f5e3ffff700 1 -- 172.21.2.102:0/858131015 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e400e52e0 con 0x7f5e401b0870 2021-02-10T11:14:10.683 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:10.673 7f5e3f7fe700 1 -- 172.21.2.102:0/858131015 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3208615169 0 0) 0x7f5e38000c10 con 0x7f5e400efe00 2021-02-10T11:14:10.683 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:10.673 7f5e3ffff700 1 --2- 172.21.2.102:0/858131015 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5e401b0870 0x7f5e400f20e0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 rx=0x7f5e300069d0 tx=0x7f5e30009dc0).ready entity=mon.0 client_cookie=e29ef6f25a54f28b server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:10.684 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:10.673 7f5e3f7fe700 1 -- 172.21.2.102:0/858131015 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5e3000b510 con 0x7f5e401b0870 2021-02-10T11:14:10.684 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:10.673 7f5e3f7fe700 1 -- 172.21.2.102:0/858131015 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e30006ac0 con 0x7f5e401b0870 2021-02-10T11:14:10.684 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:10.673 7f5e3f7fe700 1 -- 172.21.2.102:0/858131015 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5e3000ba80 con 0x7f5e401b0870 2021-02-10T11:14:10.684 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:10.673 7f5e4703d700 1 -- 172.21.2.102:0/858131015 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5e401b0870 msgr2=0x7f5e400f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:10.684 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:10.673 7f5e4703d700 1 --2- 172.21.2.102:0/858131015 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5e401b0870 0x7f5e400f20e0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 rx=0x7f5e300069d0 tx=0x7f5e30009dc0).stop 2021-02-10T11:14:10.684 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:10.674 7f5e4703d700 1 -- 172.21.2.102:0/858131015 shutdown_connections 2021-02-10T11:14:10.684 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:10.674 7f5e4703d700 1 --2- 172.21.2.102:0/858131015 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5e401b0870 0x7f5e400f20e0 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:10.684 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:10.674 7f5e4703d700 1 -- 172.21.2.102:0/858131015 >> 172.21.2.102:0/858131015 conn(0x7f5e400ce3e0 msgr2=0x7f5e400e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:14:10.684 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:10.674 7f5e4703d700 1 -- 172.21.2.102:0/858131015 shutdown_connections 2021-02-10T11:14:10.684 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:10.674 7f5e4703d700 1 -- 172.21.2.102:0/858131015 wait complete. 2021-02-10T11:14:10.685 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:10.689 7f5e4703d700 1 Processor -- start 2021-02-10T11:14:10.685 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:10.689 7f5e4703d700 1 -- start start 2021-02-10T11:14:10.686 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:10.689 7f5e4703d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5e40097a10 0x7f5e40097e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:10.686 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:10.689 7f5e4703d700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5e4008a430 0x7f5e4008a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:10.686 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:10.689 7f5e4703d700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5e4008ae50 0x7f5e4008ba00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:10.686 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:10.689 7f5e4703d700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e4008b480 con 0x7f5e40097a10 2021-02-10T11:14:10.686 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:10.689 7f5e4703d700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e4008c020 con 0x7f5e4008a430 2021-02-10T11:14:10.686 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:10.689 7f5e4703d700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e4008c200 con 0x7f5e4008ae50 2021-02-10T11:14:10.686 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:10.689 7f5e44e59700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5e40097a10 0x7f5e40097e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:10.686 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:10.689 7f5e44e59700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5e40097a10 0x7f5e40097e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:52572/0 (socket says 172.21.2.102:52572) 2021-02-10T11:14:10.686 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:10.689 7f5e44e59700 1 -- 172.21.2.102:0/2832575108 learned_addr learned my addr 172.21.2.102:0/2832575108 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:14:10.686 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:10.690 7f5e44e59700 1 -- 172.21.2.102:0/2832575108 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5e4008ae50 msgr2=0x7f5e4008ba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:10.686 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:10.690 7f5e44e59700 1 --2- 172.21.2.102:0/2832575108 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5e4008ae50 0x7f5e4008ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:10.686 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:10.690 7f5e44e59700 1 -- 172.21.2.102:0/2832575108 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5e4008a430 msgr2=0x7f5e4008a830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:10.687 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:10.690 7f5e44e59700 1 --2- 172.21.2.102:0/2832575108 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5e4008a430 0x7f5e4008a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:10.687 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:10.690 7f5e44e59700 1 -- 172.21.2.102:0/2832575108 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e4008c420 con 0x7f5e40097a10 2021-02-10T11:14:10.687 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:10.690 7f5e44e59700 1 --2- 172.21.2.102:0/2832575108 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5e40097a10 0x7f5e40097e10 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 rx=0x7f5e2800b5b0 tx=0x7f5e2800b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:10.687 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:10.690 7f5e3d7fa700 1 -- 172.21.2.102:0/2832575108 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5e2800bd40 con 0x7f5e40097a10 2021-02-10T11:14:10.687 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:10.690 7f5e3d7fa700 1 -- 172.21.2.102:0/2832575108 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e28000c60 con 0x7f5e40097a10 2021-02-10T11:14:10.687 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:10.690 7f5e3d7fa700 1 -- 172.21.2.102:0/2832575108 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5e280062a0 con 0x7f5e40097a10 2021-02-10T11:14:10.687 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:10.690 7f5e4703d700 1 -- 172.21.2.102:0/2832575108 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e4008c420 con 0x7f5e40097a10 2021-02-10T11:14:10.687 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:10.690 7f5e4703d700 1 -- 172.21.2.102:0/2832575108 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e4008c9b0 con 0x7f5e40097a10 2021-02-10T11:14:10.687 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:10.691 7f5e3d7fa700 1 -- 172.21.2.102:0/2832575108 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f5e28006020 con 0x7f5e40097a10 2021-02-10T11:14:10.687 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:10.691 7f5e3d7fa700 1 --2- 172.21.2.102:0/2832575108 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5e240186c0 0x7f5e2401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:10.687 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:10.691 7f5e3d7fa700 1 -- 172.21.2.102:0/2832575108 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f5e280325d0 con 0x7f5e40097a10 2021-02-10T11:14:10.687 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:10.691 7f5e3ffff700 1 --2- 172.21.2.102:0/2832575108 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5e240186c0 0x7f5e2401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:10.687 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:10.691 7f5e3ffff700 1 --2- 172.21.2.102:0/2832575108 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5e240186c0 0x7f5e2401abc0 crc :-1 s=READY pgs=289 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:10.716 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:10.721 7f5e4703d700 1 -- 172.21.2.102:0/2832575108 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e4008b250 con 0x7f5e40097a10 2021-02-10T11:14:10.718 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:10.724 7f5e3d7fa700 1 -- 172.21.2.102:0/2832575108 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f5e28006230 con 0x7f5e40097a10 2021-02-10T11:14:10.846 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:10.852 7f5e4703d700 1 -- 172.21.2.102:0/2832575108 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344422", "target": ["mgr", ""]}) v1 -- 0x7f5e4008b480 con 0x7f5e240186c0 2021-02-10T11:14:10.849 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:10.855 7f5e3d7fa700 1 -- 172.21.2.102:0/2832575108 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f5e30006e70 con 0x7f5e240186c0 2021-02-10T11:14:10.850 INFO:teuthology.orchestra.run.gibba002.stdout:/volumes/_nogroup/subvolume_0000000000344422/1ecb05b9-f987-4872-84bb-b26a93407aa6 2021-02-10T11:14:10.850 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:10.856 7f5e4703d700 1 -- 172.21.2.102:0/2832575108 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5e240186c0 msgr2=0x7f5e2401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:10.851 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:10.856 7f5e4703d700 1 --2- 172.21.2.102:0/2832575108 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5e240186c0 0x7f5e2401abc0 crc :-1 s=READY pgs=289 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:10.851 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:10.856 7f5e4703d700 1 -- 172.21.2.102:0/2832575108 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5e40097a10 msgr2=0x7f5e40097e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:10.851 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:10.856 7f5e4703d700 1 --2- 172.21.2.102:0/2832575108 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5e40097a10 0x7f5e40097e10 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 rx=0x7f5e2800b5b0 tx=0x7f5e2800b950).stop 2021-02-10T11:14:10.851 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:10.857 7f5e4703d700 1 -- 172.21.2.102:0/2832575108 shutdown_connections 2021-02-10T11:14:10.852 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:10.857 7f5e4703d700 1 --2- 172.21.2.102:0/2832575108 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5e4008ae50 0x7f5e4008ba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:10.852 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:10.857 7f5e4703d700 1 --2- 172.21.2.102:0/2832575108 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5e240186c0 0x7f5e2401abc0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:10.852 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:10.857 7f5e4703d700 1 --2- 172.21.2.102:0/2832575108 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5e4008a430 0x7f5e4008a830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:10.852 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:10.857 7f5e4703d700 1 --2- 172.21.2.102:0/2832575108 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5e40097a10 0x7f5e40097e10 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:10.852 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:10.857 7f5e4703d700 1 -- 172.21.2.102:0/2832575108 >> 172.21.2.102:0/2832575108 conn(0x7f5e400a73f0 msgr2=0x7f5e4009a6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:14:10.852 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:10.857 7f5e4703d700 1 -- 172.21.2.102:0/2832575108 shutdown_connections 2021-02-10T11:14:10.852 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:10.857 7f5e4703d700 1 -- 172.21.2.102:0/2832575108 wait complete. 2021-02-10T11:14:10.859 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000344422 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000344422/1ecb05b9-f987-4872-84bb-b26a93407aa6 2021-02-10T11:14:10.859 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:14:10.859 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:14:10.859 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:14:11.196 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:14:11.196 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:14:11.197 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344422/1ecb05b9-f987-4872-84bb-b26a93407aa6/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:14:11.577 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:14:11.577 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:14:11.577 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0225592 s, 46.5 MB/s 2021-02-10T11:14:11.578 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:14:11.578 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:14:11.578 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:14:11.914 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:14:11.914 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:14:11.914 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344422/1ecb05b9-f987-4872-84bb-b26a93407aa6/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:14:12.288 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:14:12.289 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:14:12.289 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0222066 s, 47.2 MB/s 2021-02-10T11:14:12.290 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:14:12.290 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:14:12.290 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:14:12.625 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:14:12.625 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:14:12.626 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344422/1ecb05b9-f987-4872-84bb-b26a93407aa6/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:14:13.003 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:14:13.004 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:14:13.004 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0253037 s, 41.4 MB/s 2021-02-10T11:14:13.005 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:14:13.005 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:14:13.005 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:14:13.340 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:14:13.341 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:14:13.341 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344422/1ecb05b9-f987-4872-84bb-b26a93407aa6/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:14:13.715 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:14:13.715 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:14:13.715 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.025648 s, 40.9 MB/s 2021-02-10T11:14:13.716 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:14:13.716 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:14:13.717 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:14:14.051 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:14:14.052 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:14:14.052 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344422/1ecb05b9-f987-4872-84bb-b26a93407aa6/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:14:14.428 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:14:14.428 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:14:14.428 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0243792 s, 43.0 MB/s 2021-02-10T11:14:14.429 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:14:14.429 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:14:14.429 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:14:14.766 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:14:14.766 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:14:14.767 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344422/1ecb05b9-f987-4872-84bb-b26a93407aa6/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:14:15.143 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:14:15.143 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:14:15.143 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0238572 s, 44.0 MB/s 2021-02-10T11:14:15.144 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:14:15.144 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:14:15.144 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:14:15.479 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:14:15.479 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:14:15.480 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344422/1ecb05b9-f987-4872-84bb-b26a93407aa6/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:14:15.857 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:14:15.857 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:14:15.858 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0245881 s, 42.6 MB/s 2021-02-10T11:14:15.859 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:14:15.859 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:14:15.859 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:14:16.194 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:14:16.194 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:14:16.194 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344422/1ecb05b9-f987-4872-84bb-b26a93407aa6/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:14:16.571 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:14:16.571 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:14:16.571 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0243238 s, 43.1 MB/s 2021-02-10T11:14:16.572 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:14:16.572 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:14:16.572 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:14:16.907 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:14:16.907 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:14:16.907 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344422/1ecb05b9-f987-4872-84bb-b26a93407aa6/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:14:17.284 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:14:17.284 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:14:17.284 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0248849 s, 42.1 MB/s 2021-02-10T11:14:17.285 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:14:17.285 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:14:17.286 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:14:17.622 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:14:17.623 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:14:17.623 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344422/1ecb05b9-f987-4872-84bb-b26a93407aa6/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:14:17.998 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:14:17.998 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:14:17.998 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.021877 s, 47.9 MB/s 2021-02-10T11:14:17.999 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000344423 2021-02-10T11:14:18.432 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:18.421 7f717e319700 1 Processor -- start 2021-02-10T11:14:18.432 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:18.421 7f717e319700 1 -- start start 2021-02-10T11:14:18.433 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:18.421 7f717e319700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f71781b0470 0x7f71781b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:18.433 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:18.421 7f717e319700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f71780f20e0 0x7f71780f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:18.433 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:18.421 7f717e319700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71780f0b50 con 0x7f71780ef610 2021-02-10T11:14:18.433 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:18.421 7f717e319700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71780f0820 con 0x7f71780f20e0 2021-02-10T11:14:18.433 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:18.421 7f717e319700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71780f24e0 con 0x7f71781b0470 2021-02-10T11:14:18.433 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:18.422 7f717c936700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f71780f20e0 0x7f71780f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:18.433 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:18.422 7f7177fff700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f71781b0470 0x7f71781b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:18.433 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:18.422 7f71777fe700 1 --1- >> v1:172.21.2.113:6789/0 conn(0x7f71780ef610 0x7f71780efa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.113:6789/0 says I am v1:172.21.2.102:52130/0 (socket says 172.21.2.102:52130) 2021-02-10T11:14:18.433 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:18.422 7f71777fe700 1 -- 172.21.2.102:0/1977609782 learned_addr learned my addr 172.21.2.102:0/1977609782 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:14:18.433 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:18.422 7f717c936700 1 -- 172.21.2.102:0/1977609782 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f71781b0470 msgr2=0x7f71781b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:14:18.433 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:18.422 7f717c936700 1 --2- 172.21.2.102:0/1977609782 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f71781b0470 0x7f71781b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:18.433 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:18.422 7f7176ffd700 1 -- 172.21.2.102:0/1977609782 <== mon.1 v1:172.21.2.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3998298423 0 0) 0x7f716c0010b0 con 0x7f71780ef610 2021-02-10T11:14:18.433 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:18.422 7f717c936700 1 -- 172.21.2.102:0/1977609782 >> v1:172.21.2.113:6789/0 conn(0x7f71780ef610 legacy=0x7f71780efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:18.434 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:18.422 7f717c936700 1 -- 172.21.2.102:0/1977609782 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71780e52e0 con 0x7f71780f20e0 2021-02-10T11:14:18.434 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:18.422 7f717c936700 1 --2- 172.21.2.102:0/1977609782 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f71780f20e0 0x7f71780f0200 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f7170006e50 tx=0x7f7170009e70).ready entity=mon.1 client_cookie=9e21a12aeb34052d server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:18.434 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:18.423 7f7176ffd700 1 -- 172.21.2.102:0/1977609782 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f717000b510 con 0x7f71780f20e0 2021-02-10T11:14:18.434 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:18.423 7f7176ffd700 1 -- 172.21.2.102:0/1977609782 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7170006790 con 0x7f71780f20e0 2021-02-10T11:14:18.434 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:18.423 7f7176ffd700 1 -- 172.21.2.102:0/1977609782 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f717000b940 con 0x7f71780f20e0 2021-02-10T11:14:18.434 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:18.423 7f717e319700 1 -- 172.21.2.102:0/1977609782 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f71780f20e0 msgr2=0x7f71780f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:18.434 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:18.423 7f717e319700 1 --2- 172.21.2.102:0/1977609782 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f71780f20e0 0x7f71780f0200 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f7170006e50 tx=0x7f7170009e70).stop 2021-02-10T11:14:18.434 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:18.424 7f717e319700 1 -- 172.21.2.102:0/1977609782 shutdown_connections 2021-02-10T11:14:18.434 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:18.424 7f717e319700 1 --2- 172.21.2.102:0/1977609782 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f71780f20e0 0x7f71780f0200 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:18.434 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:18.424 7f717e319700 1 --2- 172.21.2.102:0/1977609782 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f71781b0470 0x7f71781b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:18.434 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:18.424 7f717e319700 1 -- 172.21.2.102:0/1977609782 >> 172.21.2.102:0/1977609782 conn(0x7f71780ce3e0 msgr2=0x7f71780e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:14:18.434 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:18.424 7f717e319700 1 -- 172.21.2.102:0/1977609782 shutdown_connections 2021-02-10T11:14:18.435 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:18.424 7f717e319700 1 -- 172.21.2.102:0/1977609782 wait complete. 2021-02-10T11:14:18.436 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:18.439 7f717e319700 1 Processor -- start 2021-02-10T11:14:18.436 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:18.439 7f717e319700 1 -- start start 2021-02-10T11:14:18.436 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:18.439 7f717e319700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f71780b7c20 0x7f71780b8410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:18.436 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:18.439 7f717e319700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f71780b8a30 0x7f71780c1f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:18.437 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:18.439 7f717e319700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f71780c25a0 0x7f71780c29a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:18.437 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:18.439 7f717e319700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71780c1a80 con 0x7f71780c25a0 2021-02-10T11:14:18.437 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:18.439 7f717e319700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71780b8020 con 0x7f71780b8a30 2021-02-10T11:14:18.437 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:18.439 7f717e319700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f717809b9c0 con 0x7f71780b7c20 2021-02-10T11:14:18.437 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:18.439 7f717c936700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f71780c25a0 0x7f71780c29a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:18.437 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:18.439 7f717c936700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f71780c25a0 0x7f71780c29a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:52586/0 (socket says 172.21.2.102:52586) 2021-02-10T11:14:18.437 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:18.439 7f717c936700 1 -- 172.21.2.102:0/4279824920 learned_addr learned my addr 172.21.2.102:0/4279824920 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:14:18.437 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:18.439 7f71777fe700 1 --2- 172.21.2.102:0/4279824920 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f71780b8a30 0x7f71780c1f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:18.437 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:18.440 7f7177fff700 1 --2- 172.21.2.102:0/4279824920 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f71780b7c20 0x7f71780b8410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:18.437 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:18.440 7f717c936700 1 -- 172.21.2.102:0/4279824920 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f71780b7c20 msgr2=0x7f71780b8410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:18.437 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:18.440 7f717c936700 1 --2- 172.21.2.102:0/4279824920 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f71780b7c20 0x7f71780b8410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:18.437 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:18.440 7f717c936700 1 -- 172.21.2.102:0/4279824920 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f71780b8a30 msgr2=0x7f71780c1f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:18.437 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:18.440 7f717c936700 1 --2- 172.21.2.102:0/4279824920 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f71780b8a30 0x7f71780c1f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:18.438 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:18.440 7f717c936700 1 -- 172.21.2.102:0/4279824920 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71780b8e30 con 0x7f71780c25a0 2021-02-10T11:14:18.438 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:18.440 7f7177fff700 1 --2- 172.21.2.102:0/4279824920 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f71780b7c20 0x7f71780b8410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:14:18.438 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:18.440 7f717c936700 1 --2- 172.21.2.102:0/4279824920 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f71780c25a0 0x7f71780c29a0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 rx=0x7f7170006c50 tx=0x7f717000bd20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:18.438 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:18.440 7f7174ff9700 1 -- 172.21.2.102:0/4279824920 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f71700093d0 con 0x7f71780c25a0 2021-02-10T11:14:18.438 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:18.440 7f7174ff9700 1 -- 172.21.2.102:0/4279824920 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f717000be60 con 0x7f71780c25a0 2021-02-10T11:14:18.438 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:18.440 7f7174ff9700 1 -- 172.21.2.102:0/4279824920 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f717000c570 con 0x7f71780c25a0 2021-02-10T11:14:18.438 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:18.440 7f717e319700 1 -- 172.21.2.102:0/4279824920 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71780b8e30 con 0x7f71780c25a0 2021-02-10T11:14:18.438 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:18.440 7f717e319700 1 -- 172.21.2.102:0/4279824920 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71780b8e30 con 0x7f71780c25a0 2021-02-10T11:14:18.438 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:18.441 7f7174ff9700 1 -- 172.21.2.102:0/4279824920 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f7170002ef0 con 0x7f71780c25a0 2021-02-10T11:14:18.438 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:18.441 7f7174ff9700 1 --2- 172.21.2.102:0/4279824920 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f71580186d0 0x7f715801abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:18.438 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:18.441 7f7174ff9700 1 -- 172.21.2.102:0/4279824920 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f71700329b0 con 0x7f71780c25a0 2021-02-10T11:14:18.438 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:18.441 7f7177fff700 1 --2- 172.21.2.102:0/4279824920 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f71580186d0 0x7f715801abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:18.438 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:18.441 7f7177fff700 1 --2- 172.21.2.102:0/4279824920 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f71580186d0 0x7f715801abd0 crc :-1 s=READY pgs=290 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:18.464 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:18.470 7f717e319700 1 -- 172.21.2.102:0/4279824920 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7178105b30 con 0x7f71780c25a0 2021-02-10T11:14:18.467 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:18.473 7f7174ff9700 1 -- 172.21.2.102:0/4279824920 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f7170000df0 con 0x7f71780c25a0 2021-02-10T11:14:18.594 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:18.600 7f717e319700 1 -- 172.21.2.102:0/4279824920 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344423", "target": ["mgr", ""]}) v1 -- 0x7f71780f0b50 con 0x7f71580186d0 2021-02-10T11:14:18.621 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:18.627 7f7174ff9700 1 -- 172.21.2.102:0/4279824920 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f7168006b30 con 0x7f71580186d0 2021-02-10T11:14:18.628 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:18.633 7f717e319700 1 -- 172.21.2.102:0/4279824920 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f71580186d0 msgr2=0x7f715801abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:18.628 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:18.633 7f717e319700 1 --2- 172.21.2.102:0/4279824920 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f71580186d0 0x7f715801abd0 crc :-1 s=READY pgs=290 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:18.628 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:18.633 7f717e319700 1 -- 172.21.2.102:0/4279824920 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f71780c25a0 msgr2=0x7f71780c29a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:18.628 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:18.633 7f717e319700 1 --2- 172.21.2.102:0/4279824920 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f71780c25a0 0x7f71780c29a0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 rx=0x7f7170006c50 tx=0x7f717000bd20).stop 2021-02-10T11:14:18.628 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:18.633 7f717e319700 1 -- 172.21.2.102:0/4279824920 shutdown_connections 2021-02-10T11:14:18.628 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:18.633 7f717e319700 1 --2- 172.21.2.102:0/4279824920 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f71780c25a0 0x7f71780c29a0 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:18.628 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:18.633 7f717e319700 1 --2- 172.21.2.102:0/4279824920 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f71580186d0 0x7f715801abd0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:18.629 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:18.633 7f717e319700 1 --2- 172.21.2.102:0/4279824920 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f71780b8a30 0x7f71780c1f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:18.629 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:18.633 7f717e319700 1 --2- 172.21.2.102:0/4279824920 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f71780b7c20 0x7f71780b8410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:18.629 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:18.633 7f717e319700 1 -- 172.21.2.102:0/4279824920 >> 172.21.2.102:0/4279824920 conn(0x7f7178094e10 msgr2=0x7f71780a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:14:18.629 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:18.634 7f717e319700 1 -- 172.21.2.102:0/4279824920 shutdown_connections 2021-02-10T11:14:18.629 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:18.634 7f717e319700 1 -- 172.21.2.102:0/4279824920 wait complete. 2021-02-10T11:14:18.636 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000344423 2021-02-10T11:14:19.068 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:19.057 7f7a89d81700 1 Processor -- start 2021-02-10T11:14:19.068 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:19.057 7f7a89d81700 1 -- start start 2021-02-10T11:14:19.068 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:19.058 7f7a89d81700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f7a841b0870 0x7f7a840f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:19.068 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:19.058 7f7a89d81700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f7a840efe00 0x7f7a840f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:19.068 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:19.058 7f7a89d81700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a841b0c70 con 0x7f7a841b0470 2021-02-10T11:14:19.069 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:19.058 7f7a89d81700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a840e5ea0 con 0x7f7a840efe00 2021-02-10T11:14:19.069 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:19.058 7f7a89d81700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a840f0b50 con 0x7f7a841b0870 2021-02-10T11:14:19.069 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:19.058 7f7a83fff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f7a840efe00 0x7f7a840f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:19.069 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:19.058 7f7a83fff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f7a840efe00 0x7f7a840f0200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:52592/0 (socket says 172.21.2.102:52592) 2021-02-10T11:14:19.069 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:19.058 7f7a83fff700 1 -- 172.21.2.102:0/2039861425 learned_addr learned my addr 172.21.2.102:0/2039861425 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:14:19.069 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:19.058 7f7a82ffd700 1 --2- 172.21.2.102:0/2039861425 >> v2:172.21.2.113:3300/0 conn(0x7f7a841b0870 0x7f7a840f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:19.069 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:19.059 7f7a83fff700 1 -- 172.21.2.102:0/2039861425 >> v2:172.21.2.113:3300/0 conn(0x7f7a841b0870 msgr2=0x7f7a840f20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:14:19.069 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:19.059 7f7a83fff700 1 --2- 172.21.2.102:0/2039861425 >> v2:172.21.2.113:3300/0 conn(0x7f7a841b0870 0x7f7a840f20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:19.069 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:19.059 7f7a83fff700 1 -- 172.21.2.102:0/2039861425 >> v1:172.21.2.113:6789/0 conn(0x7f7a841b0470 legacy=0x7f7a840ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:14:19.069 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:19.059 7f7a83fff700 1 -- 172.21.2.102:0/2039861425 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a840f0820 con 0x7f7a840efe00 2021-02-10T11:14:19.069 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:19.059 7f7a83fff700 1 --2- 172.21.2.102:0/2039861425 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7a840efe00 0x7f7a840f0200 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 rx=0x7f7a7c006f00 tx=0x7f7a7c009f10).ready entity=mon.0 client_cookie=c1691d893282fb7b server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:19.069 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:19.059 7f7a827fc700 1 -- 172.21.2.102:0/2039861425 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7a7c00b3b0 con 0x7f7a840efe00 2021-02-10T11:14:19.070 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:19.059 7f7a827fc700 1 -- 172.21.2.102:0/2039861425 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a7c00b670 con 0x7f7a840efe00 2021-02-10T11:14:19.070 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:19.059 7f7a827fc700 1 -- 172.21.2.102:0/2039861425 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7a7c00bb50 con 0x7f7a840efe00 2021-02-10T11:14:19.070 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:19.059 7f7a89d81700 1 -- 172.21.2.102:0/2039861425 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7a840efe00 msgr2=0x7f7a840f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:19.070 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:19.059 7f7a89d81700 1 --2- 172.21.2.102:0/2039861425 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7a840efe00 0x7f7a840f0200 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 rx=0x7f7a7c006f00 tx=0x7f7a7c009f10).stop 2021-02-10T11:14:19.070 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:19.060 7f7a89d81700 1 -- 172.21.2.102:0/2039861425 shutdown_connections 2021-02-10T11:14:19.070 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:19.060 7f7a89d81700 1 --2- 172.21.2.102:0/2039861425 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7a840efe00 0x7f7a840f0200 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:19.070 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:19.060 7f7a89d81700 1 --2- 172.21.2.102:0/2039861425 >> v2:172.21.2.113:3300/0 conn(0x7f7a841b0870 0x7f7a840f20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:19.070 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:19.060 7f7a89d81700 1 -- 172.21.2.102:0/2039861425 >> 172.21.2.102:0/2039861425 conn(0x7f7a840ce3e0 msgr2=0x7f7a840e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:14:19.070 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:19.060 7f7a89d81700 1 -- 172.21.2.102:0/2039861425 shutdown_connections 2021-02-10T11:14:19.070 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:19.060 7f7a89d81700 1 -- 172.21.2.102:0/2039861425 wait complete. 2021-02-10T11:14:19.072 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:19.074 7f7a89d81700 1 Processor -- start 2021-02-10T11:14:19.072 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:19.075 7f7a89d81700 1 -- start start 2021-02-10T11:14:19.072 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:19.075 7f7a89d81700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7a8408ece0 0x7f7a840a6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:19.072 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:19.075 7f7a89d81700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7a840a6860 0x7f7a840a6c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:19.072 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:19.075 7f7a89d81700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7a840a7280 0x7f7a840a4590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:19.072 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:19.075 7f7a89d81700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a840a5cc0 con 0x7f7a840a6860 2021-02-10T11:14:19.073 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:19.075 7f7a89d81700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a8408f0e0 con 0x7f7a8408ece0 2021-02-10T11:14:19.073 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:19.075 7f7a89d81700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a8408f2c0 con 0x7f7a840a7280 2021-02-10T11:14:19.073 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:19.075 7f7a82ffd700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7a840a6860 0x7f7a840a6c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:19.073 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:19.075 7f7a82ffd700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7a840a6860 0x7f7a840a6c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:52598/0 (socket says 172.21.2.102:52598) 2021-02-10T11:14:19.073 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:19.075 7f7a82ffd700 1 -- 172.21.2.102:0/2663870168 learned_addr learned my addr 172.21.2.102:0/2663870168 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:14:19.073 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:19.076 7f7a83fff700 1 --2- 172.21.2.102:0/2663870168 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7a840a7280 0x7f7a840a4590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:19.073 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:19.076 7f7a82ffd700 1 -- 172.21.2.102:0/2663870168 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7a840a7280 msgr2=0x7f7a840a4590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:19.073 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:19.076 7f7a82ffd700 1 --2- 172.21.2.102:0/2663870168 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7a840a7280 0x7f7a840a4590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:19.073 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:19.076 7f7a82ffd700 1 -- 172.21.2.102:0/2663870168 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7a8408ece0 msgr2=0x7f7a840a6240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:19.073 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:19.076 7f7a837fe700 1 --2- 172.21.2.102:0/2663870168 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7a8408ece0 0x7f7a840a6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:19.073 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:19.076 7f7a82ffd700 1 --2- 172.21.2.102:0/2663870168 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7a8408ece0 0x7f7a840a6240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:19.073 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:19.076 7f7a82ffd700 1 -- 172.21.2.102:0/2663870168 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a840a4bb0 con 0x7f7a840a6860 2021-02-10T11:14:19.074 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:19.076 7f7a83fff700 1 --2- 172.21.2.102:0/2663870168 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7a840a7280 0x7f7a840a4590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:14:19.074 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:19.076 7f7a837fe700 1 --2- 172.21.2.102:0/2663870168 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7a8408ece0 0x7f7a840a6240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:14:19.074 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:19.076 7f7a82ffd700 1 --2- 172.21.2.102:0/2663870168 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7a840a6860 0x7f7a840a6c60 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 rx=0x7f7a78006f00 tx=0x7f7a7800b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:19.074 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:19.076 7f7a6bfff700 1 -- 172.21.2.102:0/2663870168 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7a7800bd30 con 0x7f7a840a6860 2021-02-10T11:14:19.074 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:19.076 7f7a6bfff700 1 -- 172.21.2.102:0/2663870168 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a78002030 con 0x7f7a840a6860 2021-02-10T11:14:19.074 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:19.076 7f7a6bfff700 1 -- 172.21.2.102:0/2663870168 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7a78002550 con 0x7f7a840a6860 2021-02-10T11:14:19.074 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:19.077 7f7a89d81700 1 -- 172.21.2.102:0/2663870168 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a840a4bb0 con 0x7f7a840a6860 2021-02-10T11:14:19.074 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:19.077 7f7a89d81700 1 -- 172.21.2.102:0/2663870168 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a840a5140 con 0x7f7a840a6860 2021-02-10T11:14:19.074 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:19.077 7f7a6bfff700 1 -- 172.21.2.102:0/2663870168 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f7a78000c90 con 0x7f7a840a6860 2021-02-10T11:14:19.074 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:19.077 7f7a6bfff700 1 --2- 172.21.2.102:0/2663870168 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7a640186c0 0x7f7a6401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:19.074 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:19.077 7f7a6bfff700 1 -- 172.21.2.102:0/2663870168 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f7a78032590 con 0x7f7a840a6860 2021-02-10T11:14:19.074 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:19.078 7f7a837fe700 1 --2- 172.21.2.102:0/2663870168 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7a640186c0 0x7f7a6401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:19.075 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:19.078 7f7a837fe700 1 --2- 172.21.2.102:0/2663870168 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7a640186c0 0x7f7a6401abc0 crc :-1 s=READY pgs=291 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:19.102 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:19.108 7f7a89d81700 1 -- 172.21.2.102:0/2663870168 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a8419d1a0 con 0x7f7a840a6860 2021-02-10T11:14:19.104 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:19.110 7f7a6bfff700 1 -- 172.21.2.102:0/2663870168 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f7a780009b0 con 0x7f7a840a6860 2021-02-10T11:14:19.231 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:19.237 7f7a89d81700 1 -- 172.21.2.102:0/2663870168 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344423", "target": ["mgr", ""]}) v1 -- 0x7f7a8419d1a0 con 0x7f7a640186c0 2021-02-10T11:14:19.234 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:19.240 7f7a6bfff700 1 -- 172.21.2.102:0/2663870168 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f7a74000960 con 0x7f7a640186c0 2021-02-10T11:14:19.238 INFO:teuthology.orchestra.run.gibba002.stdout:/volumes/_nogroup/subvolume_0000000000344423/73aa4fbf-1e87-4bd5-9036-e4dd43ef9902 2021-02-10T11:14:19.240 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:19.246 7f7a89d81700 1 -- 172.21.2.102:0/2663870168 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7a640186c0 msgr2=0x7f7a6401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:19.240 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:19.246 7f7a89d81700 1 --2- 172.21.2.102:0/2663870168 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7a640186c0 0x7f7a6401abc0 crc :-1 s=READY pgs=291 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:19.240 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:19.246 7f7a89d81700 1 -- 172.21.2.102:0/2663870168 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7a840a6860 msgr2=0x7f7a840a6c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:19.240 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:19.246 7f7a89d81700 1 --2- 172.21.2.102:0/2663870168 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7a840a6860 0x7f7a840a6c60 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 rx=0x7f7a78006f00 tx=0x7f7a7800b940).stop 2021-02-10T11:14:19.240 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:19.246 7f7a89d81700 1 -- 172.21.2.102:0/2663870168 shutdown_connections 2021-02-10T11:14:19.240 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:19.246 7f7a89d81700 1 --2- 172.21.2.102:0/2663870168 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7a840a7280 0x7f7a840a4590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:19.240 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:19.246 7f7a89d81700 1 --2- 172.21.2.102:0/2663870168 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7a840a6860 0x7f7a840a6c60 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:19.240 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:19.246 7f7a89d81700 1 --2- 172.21.2.102:0/2663870168 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7a640186c0 0x7f7a6401abc0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:19.240 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:19.246 7f7a89d81700 1 --2- 172.21.2.102:0/2663870168 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7a8408ece0 0x7f7a840a6240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:19.240 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:19.246 7f7a89d81700 1 -- 172.21.2.102:0/2663870168 >> 172.21.2.102:0/2663870168 conn(0x7f7a8408ffe0 msgr2=0x7f7a840bf050 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:14:19.241 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:19.246 7f7a89d81700 1 -- 172.21.2.102:0/2663870168 shutdown_connections 2021-02-10T11:14:19.241 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:19.246 7f7a89d81700 1 -- 172.21.2.102:0/2663870168 wait complete. 2021-02-10T11:14:19.248 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000344423 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000344423/73aa4fbf-1e87-4bd5-9036-e4dd43ef9902 2021-02-10T11:14:19.249 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:14:19.249 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:14:19.249 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:14:19.586 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:14:19.587 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:14:19.587 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344423/73aa4fbf-1e87-4bd5-9036-e4dd43ef9902/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:14:19.969 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:14:19.969 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:14:19.969 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.023094 s, 45.4 MB/s 2021-02-10T11:14:19.970 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:14:19.970 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:14:19.970 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:14:20.312 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:14:20.312 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:14:20.313 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344423/73aa4fbf-1e87-4bd5-9036-e4dd43ef9902/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:14:20.689 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:14:20.690 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:14:20.690 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0251564 s, 41.7 MB/s 2021-02-10T11:14:20.691 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:14:20.691 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:14:20.691 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:14:21.027 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:14:21.027 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:14:21.027 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344423/73aa4fbf-1e87-4bd5-9036-e4dd43ef9902/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:14:21.404 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:14:21.404 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:14:21.404 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0251691 s, 41.7 MB/s 2021-02-10T11:14:21.405 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:14:21.405 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:14:21.405 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:14:21.741 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:14:21.741 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:14:21.741 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344423/73aa4fbf-1e87-4bd5-9036-e4dd43ef9902/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:14:22.119 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:14:22.119 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:14:22.119 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0251532 s, 41.7 MB/s 2021-02-10T11:14:22.120 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:14:22.120 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:14:22.120 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:14:22.455 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:14:22.455 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:14:22.455 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344423/73aa4fbf-1e87-4bd5-9036-e4dd43ef9902/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:14:22.832 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:14:22.832 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:14:22.832 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0236295 s, 44.4 MB/s 2021-02-10T11:14:22.833 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:14:22.833 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:14:22.833 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:14:23.170 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:14:23.170 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:14:23.170 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344423/73aa4fbf-1e87-4bd5-9036-e4dd43ef9902/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:14:23.544 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:14:23.544 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:14:23.544 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0249338 s, 42.1 MB/s 2021-02-10T11:14:23.545 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:14:23.545 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:14:23.545 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:14:23.880 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:14:23.880 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:14:23.880 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344423/73aa4fbf-1e87-4bd5-9036-e4dd43ef9902/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:14:23.982 DEBUG:teuthology.orchestra.run.gibba002:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:14:23.985 DEBUG:teuthology.orchestra.run.gibba013:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:14:24.255 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:14:26.499 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:14:26.499 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0239872 s, 43.7 MB/s 2021-02-10T11:14:26.501 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:14:26.501 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:14:26.501 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:14:26.839 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:14:26.839 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:14:26.839 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344423/73aa4fbf-1e87-4bd5-9036-e4dd43ef9902/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:14:27.217 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:14:29.672 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:14:29.673 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0237437 s, 44.2 MB/s 2021-02-10T11:14:29.675 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:14:29.675 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:14:29.675 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:14:30.022 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:14:30.023 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:14:30.023 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344423/73aa4fbf-1e87-4bd5-9036-e4dd43ef9902/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:14:30.399 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:14:30.399 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:14:30.399 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0248692 s, 42.2 MB/s 2021-02-10T11:14:30.400 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:14:30.400 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:14:30.400 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:14:30.735 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:14:30.736 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:14:30.736 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344423/73aa4fbf-1e87-4bd5-9036-e4dd43ef9902/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:14:31.112 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:14:31.112 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:14:31.112 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0226781 s, 46.2 MB/s 2021-02-10T11:14:31.113 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000344424 2021-02-10T11:14:31.545 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:31.534 7fa07a9ee700 1 Processor -- start 2021-02-10T11:14:31.545 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:31.534 7fa07a9ee700 1 -- start start 2021-02-10T11:14:31.545 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:31.535 7fa07a9ee700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fa0741b0470 0x7fa0741b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:31.545 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:31.535 7fa07a9ee700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fa0740ef610 0x7fa0740efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:31.545 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:31.535 7fa07a9ee700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0740f0b50 con 0x7fa0740f0030 2021-02-10T11:14:31.545 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:31.535 7fa07a9ee700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0740e52e0 con 0x7fa0740ef610 2021-02-10T11:14:31.545 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:31.535 7fa07a9ee700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0740e5490 con 0x7fa0741b0470 2021-02-10T11:14:31.545 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:31.535 7fa073fff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fa0740ef610 0x7fa0740efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:31.545 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:31.535 7fa07880a700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fa0741b0470 0x7fa0741b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:31.546 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:31.535 7fa07900b700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fa0740f0030 0x7fa0740f10d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:44830/0 (socket says 172.21.2.102:44830) 2021-02-10T11:14:31.546 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:31.535 7fa073fff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fa0740ef610 0x7fa0740efa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:52602/0 (socket says 172.21.2.102:52602) 2021-02-10T11:14:31.546 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:31.535 7fa07880a700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fa0741b0470 0x7fa0741b0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.113:3300/0 says I am v2:172.21.2.102:56034/0 (socket says 172.21.2.102:56034) 2021-02-10T11:14:31.546 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:31.535 7fa07900b700 1 -- 172.21.2.102:0/4058711414 learned_addr learned my addr 172.21.2.102:0/4058711414 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:14:31.546 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:31.535 7fa073fff700 1 -- 172.21.2.102:0/4058711414 >> v2:172.21.2.113:3300/0 conn(0x7fa0741b0470 msgr2=0x7fa0741b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:14:31.546 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:31.535 7fa073fff700 1 --2- 172.21.2.102:0/4058711414 >> v2:172.21.2.113:3300/0 conn(0x7fa0741b0470 0x7fa0741b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:31.546 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:31.535 7fa073fff700 1 -- 172.21.2.102:0/4058711414 >> v1:172.21.2.102:6789/0 conn(0x7fa0740f0030 legacy=0x7fa0740f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:31.546 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:31.535 7fa0737fe700 1 -- 172.21.2.102:0/4058711414 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 159178464 0 0) 0x7fa06c000f90 con 0x7fa0740f0030 2021-02-10T11:14:31.546 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:31.535 7fa073fff700 1 -- 172.21.2.102:0/4058711414 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0740f0430 con 0x7fa0740ef610 2021-02-10T11:14:31.546 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:31.535 7fa07880a700 1 --2- 172.21.2.102:0/4058711414 >> v2:172.21.2.113:3300/0 conn(0x7fa0741b0470 0x7fa0741b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:14:31.546 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:31.536 7fa073fff700 1 --2- 172.21.2.102:0/4058711414 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa0740ef610 0x7fa0740efa10 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 rx=0x7fa064006650 tx=0x7fa064009a80).ready entity=mon.0 client_cookie=ccbc8f902bfb921f server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:31.546 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:31.536 7fa0737fe700 1 -- 172.21.2.102:0/4058711414 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa06400c0c0 con 0x7fa0740ef610 2021-02-10T11:14:31.547 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:31.536 7fa0737fe700 1 -- 172.21.2.102:0/4058711414 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa06400c340 con 0x7fa0740ef610 2021-02-10T11:14:31.547 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:31.536 7fa0737fe700 1 -- 172.21.2.102:0/4058711414 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa06400c860 con 0x7fa0740ef610 2021-02-10T11:14:31.547 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:31.536 7fa07a9ee700 1 -- 172.21.2.102:0/4058711414 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa0740ef610 msgr2=0x7fa0740efa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:31.547 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:31.536 7fa07a9ee700 1 --2- 172.21.2.102:0/4058711414 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa0740ef610 0x7fa0740efa10 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 rx=0x7fa064006650 tx=0x7fa064009a80).stop 2021-02-10T11:14:31.547 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:31.536 7fa07a9ee700 1 -- 172.21.2.102:0/4058711414 shutdown_connections 2021-02-10T11:14:31.547 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:31.536 7fa07a9ee700 1 --2- 172.21.2.102:0/4058711414 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa0740ef610 0x7fa0740efa10 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:31.547 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:31.536 7fa07a9ee700 1 --2- 172.21.2.102:0/4058711414 >> v2:172.21.2.113:3300/0 conn(0x7fa0741b0470 0x7fa0741b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:31.547 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:31.536 7fa07a9ee700 1 -- 172.21.2.102:0/4058711414 >> 172.21.2.102:0/4058711414 conn(0x7fa0740ce3e0 msgr2=0x7fa0740e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:14:31.547 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:31.536 7fa07a9ee700 1 -- 172.21.2.102:0/4058711414 shutdown_connections 2021-02-10T11:14:31.547 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:31.536 7fa07a9ee700 1 -- 172.21.2.102:0/4058711414 wait complete. 2021-02-10T11:14:31.547 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:31.551 7fa07a9ee700 1 Processor -- start 2021-02-10T11:14:31.547 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:31.551 7fa07a9ee700 1 -- start start 2021-02-10T11:14:31.549 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:31.551 7fa07a9ee700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa0740a0420 0x7fa0740a0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:31.549 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:31.551 7fa07a9ee700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa074189d00 0x7fa07418a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:31.549 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:31.551 7fa07a9ee700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa07418a720 0x7fa0740b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:31.549 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:31.551 7fa07a9ee700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa074189800 con 0x7fa0740a0420 2021-02-10T11:14:31.549 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:31.551 7fa07a9ee700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa07418ab20 con 0x7fa07418a720 2021-02-10T11:14:31.549 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:31.551 7fa07a9ee700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0740a1140 con 0x7fa074189d00 2021-02-10T11:14:31.549 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:31.552 7fa07880a700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa0740a0420 0x7fa0740a0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:31.549 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:31.552 7fa073fff700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa074189d00 0x7fa07418a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:31.549 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:31.552 7fa07880a700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa0740a0420 0x7fa0740a0820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:52610/0 (socket says 172.21.2.102:52610) 2021-02-10T11:14:31.549 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:31.552 7fa07880a700 1 -- 172.21.2.102:0/3542720106 learned_addr learned my addr 172.21.2.102:0/3542720106 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:14:31.550 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:31.552 7fa07900b700 1 --2- 172.21.2.102:0/3542720106 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa07418a720 0x7fa0740b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:31.550 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:31.552 7fa07900b700 1 -- 172.21.2.102:0/3542720106 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa074189d00 msgr2=0x7fa07418a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:31.550 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:31.552 7fa07900b700 1 --2- 172.21.2.102:0/3542720106 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa074189d00 0x7fa07418a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:31.550 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:31.552 7fa07900b700 1 -- 172.21.2.102:0/3542720106 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa0740a0420 msgr2=0x7fa0740a0820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:31.550 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:31.552 7fa07900b700 1 --2- 172.21.2.102:0/3542720106 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa0740a0420 0x7fa0740a0820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:31.550 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:31.552 7fa07900b700 1 -- 172.21.2.102:0/3542720106 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0740b8200 con 0x7fa07418a720 2021-02-10T11:14:31.550 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:31.552 7fa07900b700 1 --2- 172.21.2.102:0/3542720106 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa07418a720 0x7fa0740b7be0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7fa06c007f90 tx=0x7fa06c00abb0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:31.550 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:31.552 7fa0717fa700 1 -- 172.21.2.102:0/3542720106 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa06c007030 con 0x7fa07418a720 2021-02-10T11:14:31.550 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:31.552 7fa0717fa700 1 -- 172.21.2.102:0/3542720106 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa06c0072b0 con 0x7fa07418a720 2021-02-10T11:14:31.550 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:31.552 7fa07a9ee700 1 -- 172.21.2.102:0/3542720106 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0740b8200 con 0x7fa07418a720 2021-02-10T11:14:31.550 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:31.552 7fa07a9ee700 1 -- 172.21.2.102:0/3542720106 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0740b8200 con 0x7fa07418a720 2021-02-10T11:14:31.551 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:31.552 7fa0717fa700 1 -- 172.21.2.102:0/3542720106 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa06c004720 con 0x7fa07418a720 2021-02-10T11:14:31.551 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:31.553 7fa0717fa700 1 -- 172.21.2.102:0/3542720106 <== mon.1 v2:172.21.2.113:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa06c000d10 con 0x7fa07418a720 2021-02-10T11:14:31.551 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:31.553 7fa0717fa700 1 --2- 172.21.2.102:0/3542720106 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa058018700 0x7fa05801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:31.551 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:31.553 7fa0717fa700 1 -- 172.21.2.102:0/3542720106 <== mon.1 v2:172.21.2.113:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fa06c032a90 con 0x7fa07418a720 2021-02-10T11:14:31.551 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:31.553 7fa07880a700 1 --2- 172.21.2.102:0/3542720106 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa058018700 0x7fa05801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:31.551 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:31.553 7fa07880a700 1 --2- 172.21.2.102:0/3542720106 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa058018700 0x7fa05801abc0 crc :-1 s=READY pgs=292 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:31.578 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:31.584 7fa07a9ee700 1 -- 172.21.2.102:0/3542720106 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa07418ab20 con 0x7fa07418a720 2021-02-10T11:14:31.583 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:31.589 7fa0717fa700 1 -- 172.21.2.102:0/3542720106 <== mon.1 v2:172.21.2.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fa06c007b10 con 0x7fa07418a720 2021-02-10T11:14:31.711 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:31.717 7fa07a9ee700 1 -- 172.21.2.102:0/3542720106 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344424", "target": ["mgr", ""]}) v1 -- 0x7fa07418ab20 con 0x7fa058018700 2021-02-10T11:14:31.740 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:31.745 7fa0717fa700 1 -- 172.21.2.102:0/3542720106 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fa05c0009b0 con 0x7fa058018700 2021-02-10T11:14:31.745 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:31.750 7fa07a9ee700 1 -- 172.21.2.102:0/3542720106 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa058018700 msgr2=0x7fa05801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:31.745 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:31.751 7fa07a9ee700 1 --2- 172.21.2.102:0/3542720106 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa058018700 0x7fa05801abc0 crc :-1 s=READY pgs=292 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:31.745 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:31.751 7fa07a9ee700 1 -- 172.21.2.102:0/3542720106 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa07418a720 msgr2=0x7fa0740b7be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:31.745 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:31.751 7fa07a9ee700 1 --2- 172.21.2.102:0/3542720106 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa07418a720 0x7fa0740b7be0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7fa06c007f90 tx=0x7fa06c00abb0).stop 2021-02-10T11:14:31.745 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:31.751 7fa07a9ee700 1 -- 172.21.2.102:0/3542720106 shutdown_connections 2021-02-10T11:14:31.746 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:31.751 7fa07a9ee700 1 --2- 172.21.2.102:0/3542720106 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa058018700 0x7fa05801abc0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:31.746 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:31.751 7fa07a9ee700 1 --2- 172.21.2.102:0/3542720106 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa07418a720 0x7fa0740b7be0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:31.746 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:31.751 7fa07a9ee700 1 --2- 172.21.2.102:0/3542720106 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa074189d00 0x7fa07418a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:31.746 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:31.751 7fa07a9ee700 1 --2- 172.21.2.102:0/3542720106 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa0740a0420 0x7fa0740a0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:31.746 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:31.751 7fa07a9ee700 1 -- 172.21.2.102:0/3542720106 >> 172.21.2.102:0/3542720106 conn(0x7fa074094e10 msgr2=0x7fa0740a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:14:31.746 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:31.751 7fa07a9ee700 1 -- 172.21.2.102:0/3542720106 shutdown_connections 2021-02-10T11:14:31.746 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:31.751 7fa07a9ee700 1 -- 172.21.2.102:0/3542720106 wait complete. 2021-02-10T11:14:31.753 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000344424 2021-02-10T11:14:32.187 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:32.174 7f47fe724700 1 Processor -- start 2021-02-10T11:14:32.187 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:32.174 7f47fe724700 1 -- start start 2021-02-10T11:14:32.187 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:32.175 7f47fe724700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f47f81b0870 0x7f47f80f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:32.187 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:32.175 7f47fe724700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f47f80efe00 0x7f47f80f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:32.187 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:32.175 7f47fe724700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47f81b0c70 con 0x7f47f81b0470 2021-02-10T11:14:32.187 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:32.175 7f47fe724700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47f80f0b50 con 0x7f47f80efe00 2021-02-10T11:14:32.187 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:32.175 7f47fe724700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47f80f0820 con 0x7f47f81b0870 2021-02-10T11:14:32.187 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:32.175 7f47f7fff700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f47f81b0470 0x7f47f80ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:44840/0 (socket says 172.21.2.102:44840) 2021-02-10T11:14:32.188 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:32.175 7f47f7fff700 1 -- 172.21.2.102:0/4150728115 learned_addr learned my addr 172.21.2.102:0/4150728115 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:14:32.188 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:32.176 7f47f77fe700 1 --2- 172.21.2.102:0/4150728115 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f47f81b0870 0x7f47f80f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:32.188 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:32.176 7f47fcd41700 1 --2- 172.21.2.102:0/4150728115 >> v2:172.21.2.113:3300/0 conn(0x7f47f80efe00 0x7f47f80f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:32.188 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:32.176 7f47f6ffd700 1 -- 172.21.2.102:0/4150728115 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2368960553 0 0) 0x7f47e80010b0 con 0x7f47f81b0470 2021-02-10T11:14:32.188 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:32.176 7f47f6ffd700 1 -- 172.21.2.102:0/4150728115 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f47e0002fe0 con 0x7f47f81b0470 2021-02-10T11:14:32.188 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:32.176 7f47f6ffd700 1 -- 172.21.2.102:0/4150728115 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 405294667 0 0) 0x7f47e80010b0 con 0x7f47f81b0470 2021-02-10T11:14:32.188 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:32.176 7f47f6ffd700 1 -- 172.21.2.102:0/4150728115 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f47f81b0870 msgr2=0x7f47f80f20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:14:32.188 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:32.176 7f47f6ffd700 1 --2- 172.21.2.102:0/4150728115 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f47f81b0870 0x7f47f80f20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:32.188 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:32.176 7f47f6ffd700 1 -- 172.21.2.102:0/4150728115 >> v2:172.21.2.113:3300/0 conn(0x7f47f80efe00 msgr2=0x7f47f80f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:14:32.188 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:32.176 7f47f6ffd700 1 --2- 172.21.2.102:0/4150728115 >> v2:172.21.2.113:3300/0 conn(0x7f47f80efe00 0x7f47f80f0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:32.188 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:32.176 7f47f6ffd700 1 -- 172.21.2.102:0/4150728115 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47f80e52e0 con 0x7f47f81b0470 2021-02-10T11:14:32.188 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:32.176 7f47f6ffd700 1 -- 172.21.2.102:0/4150728115 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7f47e80015b0 con 0x7f47f81b0470 2021-02-10T11:14:32.188 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:32.176 7f47f6ffd700 1 -- 172.21.2.102:0/4150728115 >> v1:172.21.2.102:6789/0 conn(0x7f47f81b0470 legacy=0x7f47f80ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:32.189 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:32.176 7f47f6ffd700 1 --2- 172.21.2.102:0/4150728115 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f47e0003a60 0x7f47e0005f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:32.189 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:32.177 7f47f6ffd700 1 --2- 172.21.2.102:0/4150728115 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f47e004a790 0x7f47e004cc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:32.189 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:32.177 7f47f6ffd700 1 --2- 172.21.2.102:0/4150728115 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f47e004d500 0x7f47e004fa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:32.189 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:32.177 7f47f6ffd700 1 -- 172.21.2.102:0/4150728115 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47e00502d0 con 0x7f47e004d500 2021-02-10T11:14:32.189 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:32.177 7f47f6ffd700 1 -- 172.21.2.102:0/4150728115 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47e00504f0 con 0x7f47e0003a60 2021-02-10T11:14:32.189 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:32.177 7f47f6ffd700 1 -- 172.21.2.102:0/4150728115 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47e0050710 con 0x7f47e004a790 2021-02-10T11:14:32.189 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:32.177 7f47f7fff700 1 --2- 172.21.2.102:0/4150728115 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f47e0003a60 0x7f47e0005f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:32.189 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:32.177 7f47fcd41700 1 --2- 172.21.2.102:0/4150728115 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f47e004d500 0x7f47e004fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:32.189 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:32.177 7f47f77fe700 1 --2- 172.21.2.102:0/4150728115 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f47e004a790 0x7f47e004cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:32.189 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:32.177 7f47fcd41700 1 -- 172.21.2.102:0/4150728115 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f47e004a790 msgr2=0x7f47e004cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:14:32.189 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:32.177 7f47fcd41700 1 --2- 172.21.2.102:0/4150728115 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f47e004a790 0x7f47e004cc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:32.189 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:32.177 7f47fcd41700 1 -- 172.21.2.102:0/4150728115 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f47e0003a60 msgr2=0x7f47e0005f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:14:32.189 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:32.177 7f47fcd41700 1 --2- 172.21.2.102:0/4150728115 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f47e0003a60 0x7f47e0005f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:32.189 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:32.177 7f47fcd41700 1 -- 172.21.2.102:0/4150728115 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47e00509b0 con 0x7f47e004d500 2021-02-10T11:14:32.190 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:32.177 7f47fcd41700 1 --2- 172.21.2.102:0/4150728115 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f47e004d500 0x7f47e004fa20 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 rx=0x7f47f0006a40 tx=0x7f47f000c0c0).ready entity=mon.0 client_cookie=4a6ac422253b6ac2 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:32.190 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:32.177 7f47f6ffd700 1 -- 172.21.2.102:0/4150728115 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f47f000c4b0 con 0x7f47e004d500 2021-02-10T11:14:32.190 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:32.177 7f47f6ffd700 1 -- 172.21.2.102:0/4150728115 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47f000c7b0 con 0x7f47e004d500 2021-02-10T11:14:32.190 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:32.177 7f47f6ffd700 1 -- 172.21.2.102:0/4150728115 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f47f000cc90 con 0x7f47e004d500 2021-02-10T11:14:32.190 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:32.178 7f47fe724700 1 -- 172.21.2.102:0/4150728115 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f47e004d500 msgr2=0x7f47e004fa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:32.190 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:32.178 7f47fe724700 1 --2- 172.21.2.102:0/4150728115 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f47e004d500 0x7f47e004fa20 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 rx=0x7f47f0006a40 tx=0x7f47f000c0c0).stop 2021-02-10T11:14:32.190 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:32.178 7f47fcd41700 1 -- 172.21.2.102:0/4150728115 reap_dead start 2021-02-10T11:14:32.190 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:32.178 7f47fe724700 1 -- 172.21.2.102:0/4150728115 shutdown_connections 2021-02-10T11:14:32.190 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:32.178 7f47fe724700 1 -- 172.21.2.102:0/4150728115 >> 172.21.2.102:0/4150728115 conn(0x7f47f80ce3e0 msgr2=0x7f47f80e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:14:32.190 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:32.178 7f47fe724700 1 -- 172.21.2.102:0/4150728115 shutdown_connections 2021-02-10T11:14:32.190 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:32.178 7f47fe724700 1 -- 172.21.2.102:0/4150728115 wait complete. 2021-02-10T11:14:32.190 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:32.193 7f47fe724700 1 Processor -- start 2021-02-10T11:14:32.191 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:32.193 7f47fe724700 1 -- start start 2021-02-10T11:14:32.192 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:32.193 7f47fe724700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f47f80b7db0 0x7f47f80b8260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:32.192 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:32.193 7f47fe724700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f47f80b8880 0x7f47f81a7450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:32.192 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:32.193 7f47fe724700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f47f80b8c80 0x7f47f81a7a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:32.192 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:32.193 7f47fe724700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47f80b79b0 con 0x7f47f80b8c80 2021-02-10T11:14:32.192 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:32.193 7f47fe724700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47f81a8090 con 0x7f47f80b7db0 2021-02-10T11:14:32.192 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:32.193 7f47fe724700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47f81a82b0 con 0x7f47f80b8880 2021-02-10T11:14:32.192 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:32.193 7f47fcd41700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f47f80b8c80 0x7f47f81a7a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:32.192 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:32.194 7f47f7fff700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f47f80b7db0 0x7f47f80b8260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:32.193 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:32.194 7f47fcd41700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f47f80b8c80 0x7f47f81a7a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:52630/0 (socket says 172.21.2.102:52630) 2021-02-10T11:14:32.193 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:32.194 7f47fcd41700 1 -- 172.21.2.102:0/1312744015 learned_addr learned my addr 172.21.2.102:0/1312744015 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:14:32.193 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:32.194 7f47f77fe700 1 --2- 172.21.2.102:0/1312744015 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f47f80b8880 0x7f47f81a7450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:32.193 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:32.194 7f47fcd41700 1 -- 172.21.2.102:0/1312744015 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f47f80b8880 msgr2=0x7f47f81a7450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:32.193 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:32.194 7f47fcd41700 1 --2- 172.21.2.102:0/1312744015 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f47f80b8880 0x7f47f81a7450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:32.193 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:32.194 7f47fcd41700 1 -- 172.21.2.102:0/1312744015 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f47f80b7db0 msgr2=0x7f47f80b8260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:32.193 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:32.194 7f47fcd41700 1 --2- 172.21.2.102:0/1312744015 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f47f80b7db0 0x7f47f80b8260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:32.193 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:32.194 7f47fcd41700 1 -- 172.21.2.102:0/1312744015 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47f8190c10 con 0x7f47f80b8c80 2021-02-10T11:14:32.193 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:32.194 7f47f7fff700 1 --2- 172.21.2.102:0/1312744015 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f47f80b7db0 0x7f47f80b8260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:14:32.193 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:32.194 7f47fcd41700 1 --2- 172.21.2.102:0/1312744015 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f47f80b8c80 0x7f47f81a7a70 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 rx=0x7f47f000b940 tx=0x7f47f000bce0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:32.193 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:32.194 7f47f4ff9700 1 -- 172.21.2.102:0/1312744015 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f47f0000df0 con 0x7f47f80b8c80 2021-02-10T11:14:32.193 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:32.194 7f47f4ff9700 1 -- 172.21.2.102:0/1312744015 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47f00011a0 con 0x7f47f80b8c80 2021-02-10T11:14:32.193 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:32.194 7f47f4ff9700 1 -- 172.21.2.102:0/1312744015 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f47f00016c0 con 0x7f47f80b8c80 2021-02-10T11:14:32.194 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:32.194 7f47fe724700 1 -- 172.21.2.102:0/1312744015 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47f8190c10 con 0x7f47f80b8c80 2021-02-10T11:14:32.194 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:32.194 7f47fe724700 1 -- 172.21.2.102:0/1312744015 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47f8190c10 con 0x7f47f80b8c80 2021-02-10T11:14:32.194 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:32.195 7f47f4ff9700 1 -- 172.21.2.102:0/1312744015 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f47f0000df0 con 0x7f47f80b8c80 2021-02-10T11:14:32.194 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:32.195 7f47f4ff9700 1 --2- 172.21.2.102:0/1312744015 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f47d80186c0 0x7f47d801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:32.194 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:32.195 7f47f4ff9700 1 -- 172.21.2.102:0/1312744015 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f47f00328d0 con 0x7f47f80b8c80 2021-02-10T11:14:32.194 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:32.195 7f47f7fff700 1 --2- 172.21.2.102:0/1312744015 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f47d80186c0 0x7f47d801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:32.194 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:32.195 7f47f7fff700 1 --2- 172.21.2.102:0/1312744015 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f47d80186c0 0x7f47d801abc0 crc :-1 s=READY pgs=293 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:32.219 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:32.225 7f47fe724700 1 -- 172.21.2.102:0/1312744015 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47f8105b30 con 0x7f47f80b8c80 2021-02-10T11:14:32.222 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:32.228 7f47f4ff9700 1 -- 172.21.2.102:0/1312744015 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f47f000be20 con 0x7f47f80b8c80 2021-02-10T11:14:32.351 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:32.357 7f47fe724700 1 -- 172.21.2.102:0/1312744015 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344424", "target": ["mgr", ""]}) v1 -- 0x7f47f80f0b50 con 0x7f47d80186c0 2021-02-10T11:14:32.354 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:32.360 7f47f4ff9700 1 -- 172.21.2.102:0/1312744015 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f47e8000dd0 con 0x7f47d80186c0 2021-02-10T11:14:32.358 INFO:teuthology.orchestra.run.gibba002.stdout:/volumes/_nogroup/subvolume_0000000000344424/f7bcf0bc-a345-4eac-acae-035028c46fa2 2021-02-10T11:14:32.360 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:32.365 7f47fe724700 1 -- 172.21.2.102:0/1312744015 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f47d80186c0 msgr2=0x7f47d801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:32.360 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:32.365 7f47fe724700 1 --2- 172.21.2.102:0/1312744015 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f47d80186c0 0x7f47d801abc0 crc :-1 s=READY pgs=293 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:32.360 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:32.365 7f47fe724700 1 -- 172.21.2.102:0/1312744015 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f47f80b8c80 msgr2=0x7f47f81a7a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:32.360 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:32.365 7f47fe724700 1 --2- 172.21.2.102:0/1312744015 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f47f80b8c80 0x7f47f81a7a70 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 rx=0x7f47f000b940 tx=0x7f47f000bce0).stop 2021-02-10T11:14:32.360 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:32.366 7f47fe724700 1 -- 172.21.2.102:0/1312744015 shutdown_connections 2021-02-10T11:14:32.360 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:32.366 7f47fe724700 1 --2- 172.21.2.102:0/1312744015 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f47f80b8c80 0x7f47f81a7a70 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:32.360 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:32.366 7f47fe724700 1 --2- 172.21.2.102:0/1312744015 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f47f80b8880 0x7f47f81a7450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:32.360 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:32.366 7f47fe724700 1 --2- 172.21.2.102:0/1312744015 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f47d80186c0 0x7f47d801abc0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:32.360 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:32.366 7f47fe724700 1 --2- 172.21.2.102:0/1312744015 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f47f80b7db0 0x7f47f80b8260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:32.361 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:32.366 7f47fe724700 1 -- 172.21.2.102:0/1312744015 >> 172.21.2.102:0/1312744015 conn(0x7f47f8094e10 msgr2=0x7f47f80a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:14:32.361 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:32.366 7f47fe724700 1 -- 172.21.2.102:0/1312744015 shutdown_connections 2021-02-10T11:14:32.361 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:32.366 7f47fe724700 1 -- 172.21.2.102:0/1312744015 wait complete. 2021-02-10T11:14:32.368 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000344424 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000344424/f7bcf0bc-a345-4eac-acae-035028c46fa2 2021-02-10T11:14:32.368 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:14:32.368 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:14:32.368 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:14:32.703 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:14:32.704 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:14:32.704 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344424/f7bcf0bc-a345-4eac-acae-035028c46fa2/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:14:33.084 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:14:33.084 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:14:33.084 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0241037 s, 43.5 MB/s 2021-02-10T11:14:33.086 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:14:33.086 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:14:33.086 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:14:33.422 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:14:33.423 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:14:33.423 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344424/f7bcf0bc-a345-4eac-acae-035028c46fa2/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:14:33.797 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:14:33.797 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:14:33.797 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0254451 s, 41.2 MB/s 2021-02-10T11:14:33.798 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:14:33.798 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:14:33.798 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:14:34.133 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:14:34.133 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:14:34.133 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344424/f7bcf0bc-a345-4eac-acae-035028c46fa2/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:14:34.510 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:14:34.510 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:14:34.510 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0249195 s, 42.1 MB/s 2021-02-10T11:14:34.511 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:14:34.511 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:14:34.511 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:14:34.847 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:14:34.847 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:14:34.848 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344424/f7bcf0bc-a345-4eac-acae-035028c46fa2/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:14:35.226 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:14:35.226 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:14:35.226 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0256269 s, 40.9 MB/s 2021-02-10T11:14:35.227 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:14:35.227 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:14:35.228 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:14:35.563 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:14:35.563 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:14:35.563 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344424/f7bcf0bc-a345-4eac-acae-035028c46fa2/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:14:35.938 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:14:35.938 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:14:35.938 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0229403 s, 45.7 MB/s 2021-02-10T11:14:35.939 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:14:35.940 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:14:35.940 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:14:36.275 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:14:36.275 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:14:36.275 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344424/f7bcf0bc-a345-4eac-acae-035028c46fa2/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:14:36.650 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:14:36.650 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:14:36.650 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0228704 s, 45.8 MB/s 2021-02-10T11:14:36.651 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:14:36.651 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:14:36.651 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:14:36.987 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:14:36.988 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:14:36.988 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344424/f7bcf0bc-a345-4eac-acae-035028c46fa2/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:14:37.364 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:14:37.364 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:14:37.364 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0239535 s, 43.8 MB/s 2021-02-10T11:14:37.365 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:14:37.365 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:14:37.365 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:14:37.702 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:14:37.702 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:14:37.702 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344424/f7bcf0bc-a345-4eac-acae-035028c46fa2/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:14:38.078 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:14:38.079 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:14:38.079 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0236317 s, 44.4 MB/s 2021-02-10T11:14:38.080 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:14:38.080 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:14:38.080 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:14:38.415 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:14:38.416 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:14:38.416 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344424/f7bcf0bc-a345-4eac-acae-035028c46fa2/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:14:38.788 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:14:38.788 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:14:38.789 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0244602 s, 42.9 MB/s 2021-02-10T11:14:38.789 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:14:38.790 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:14:38.790 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:14:39.125 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:14:39.125 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:14:39.125 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344424/f7bcf0bc-a345-4eac-acae-035028c46fa2/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:14:39.502 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:14:39.502 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:14:39.502 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.024301 s, 43.1 MB/s 2021-02-10T11:14:39.504 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000344425 2021-02-10T11:14:39.938 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:39.925 7f2274dd3700 1 Processor -- start 2021-02-10T11:14:39.938 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:39.925 7f2274dd3700 1 -- start start 2021-02-10T11:14:39.938 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:39.926 7f2274dd3700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f22701b0470 0x7f22701b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:39.939 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:39.926 7f2274dd3700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f22700f20e0 0x7f22700f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:39.939 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:39.926 7f2274dd3700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22700f0b50 con 0x7f22700ef610 2021-02-10T11:14:39.939 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:39.926 7f2274dd3700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22700f0820 con 0x7f22701b0470 2021-02-10T11:14:39.939 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:39.926 7f2274dd3700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22700f24e0 con 0x7f22700f20e0 2021-02-10T11:14:39.939 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:39.927 7f226eb13700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f22701b0470 0x7f22701b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:39.939 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:39.927 7f226eb13700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f22701b0470 0x7f22701b0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.113:3300/0 says I am v2:172.21.2.102:56066/0 (socket says 172.21.2.102:56066) 2021-02-10T11:14:39.940 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:39.927 7f226eb13700 1 -- 172.21.2.102:0/2725749215 learned_addr learned my addr 172.21.2.102:0/2725749215 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:14:39.940 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:39.927 7f226f314700 1 --2- 172.21.2.102:0/2725749215 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f22700f20e0 0x7f22700f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:39.940 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:39.927 7f226f314700 1 -- 172.21.2.102:0/2725749215 >> v2:172.21.2.113:3300/0 conn(0x7f22701b0470 msgr2=0x7f22701b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:14:39.940 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:39.927 7f226f314700 1 --2- 172.21.2.102:0/2725749215 >> v2:172.21.2.113:3300/0 conn(0x7f22701b0470 0x7f22701b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:39.940 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:39.927 7f226f314700 1 -- 172.21.2.102:0/2725749215 >> v1:172.21.2.113:6789/0 conn(0x7f22700ef610 legacy=0x7f22700efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:39.941 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:39.927 7f226f314700 1 -- 172.21.2.102:0/2725749215 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22700e52e0 con 0x7f22700f20e0 2021-02-10T11:14:39.941 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:39.927 7f226db11700 1 -- 172.21.2.102:0/2725749215 <== mon.1 v1:172.21.2.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 116152320 0 0) 0x7f2264000ff0 con 0x7f22700ef610 2021-02-10T11:14:39.941 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:39.927 7f226eb13700 1 --2- 172.21.2.102:0/2725749215 >> v2:172.21.2.113:3300/0 conn(0x7f22701b0470 0x7f22701b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:14:39.941 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:39.927 7f226f314700 1 --2- 172.21.2.102:0/2725749215 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f22700f20e0 0x7f22700f0200 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f2260006ad0 tx=0x7f2260009ad0).ready entity=mon.2 client_cookie=4064aa022019733a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:39.941 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:39.928 7f226db11700 1 -- 172.21.2.102:0/2725749215 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2260000c10 con 0x7f22700f20e0 2021-02-10T11:14:39.941 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:39.928 7f226db11700 1 -- 172.21.2.102:0/2725749215 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2260006410 con 0x7f22700f20e0 2021-02-10T11:14:39.942 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:39.928 7f226db11700 1 -- 172.21.2.102:0/2725749215 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f22600024d0 con 0x7f22700f20e0 2021-02-10T11:14:39.942 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:39.928 7f2274dd3700 1 -- 172.21.2.102:0/2725749215 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f22700f20e0 msgr2=0x7f22700f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:39.942 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:39.928 7f2274dd3700 1 --2- 172.21.2.102:0/2725749215 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f22700f20e0 0x7f22700f0200 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f2260006ad0 tx=0x7f2260009ad0).stop 2021-02-10T11:14:39.942 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:39.928 7f2274dd3700 1 -- 172.21.2.102:0/2725749215 shutdown_connections 2021-02-10T11:14:39.942 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:39.928 7f2274dd3700 1 --2- 172.21.2.102:0/2725749215 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f22700f20e0 0x7f22700f0200 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:39.943 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:39.928 7f2274dd3700 1 --2- 172.21.2.102:0/2725749215 >> v2:172.21.2.113:3300/0 conn(0x7f22701b0470 0x7f22701b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:39.943 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:39.928 7f2274dd3700 1 -- 172.21.2.102:0/2725749215 >> 172.21.2.102:0/2725749215 conn(0x7f22700ce3e0 msgr2=0x7f22700e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:14:39.943 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:39.929 7f2274dd3700 1 -- 172.21.2.102:0/2725749215 shutdown_connections 2021-02-10T11:14:39.943 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:39.929 7f2274dd3700 1 -- 172.21.2.102:0/2725749215 wait complete. 2021-02-10T11:14:39.943 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:39.943 7f2274dd3700 1 Processor -- start 2021-02-10T11:14:39.943 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:39.943 7f2274dd3700 1 -- start start 2021-02-10T11:14:39.944 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:39.943 7f2274dd3700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f227009f140 0x7f22700c1940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:39.944 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:39.943 7f2274dd3700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f22700c1f60 0x7f227009e300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:39.944 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:39.943 7f2274dd3700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f227009e920 0x7f22700c2960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:39.944 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:39.943 7f2274dd3700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22700c23e0 con 0x7f22700c1f60 2021-02-10T11:14:39.944 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:39.943 7f2274dd3700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f227009ed20 con 0x7f227009e920 2021-02-10T11:14:39.945 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:39.943 7f2274dd3700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2270105d10 con 0x7f227009f140 2021-02-10T11:14:39.945 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:39.944 7f226e312700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f22700c1f60 0x7f227009e300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:39.945 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:39.944 7f226e312700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f22700c1f60 0x7f227009e300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:52646/0 (socket says 172.21.2.102:52646) 2021-02-10T11:14:39.945 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:39.944 7f226e312700 1 -- 172.21.2.102:0/730691835 learned_addr learned my addr 172.21.2.102:0/730691835 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:14:39.948 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:39.944 7f226eb13700 1 --2- 172.21.2.102:0/730691835 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f227009f140 0x7f22700c1940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:39.948 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:39.944 7f226f314700 1 --2- 172.21.2.102:0/730691835 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f227009e920 0x7f22700c2960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:39.948 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:39.944 7f226e312700 1 -- 172.21.2.102:0/730691835 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f227009f140 msgr2=0x7f22700c1940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:39.948 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:39.944 7f226e312700 1 --2- 172.21.2.102:0/730691835 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f227009f140 0x7f22700c1940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:39.948 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:39.944 7f226e312700 1 -- 172.21.2.102:0/730691835 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f227009e920 msgr2=0x7f22700c2960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:39.948 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:39.944 7f226e312700 1 --2- 172.21.2.102:0/730691835 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f227009e920 0x7f22700c2960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:39.948 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:39.944 7f226e312700 1 -- 172.21.2.102:0/730691835 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22701a4910 con 0x7f22700c1f60 2021-02-10T11:14:39.948 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:39.944 7f226eb13700 1 --2- 172.21.2.102:0/730691835 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f227009f140 0x7f22700c1940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:14:39.948 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:39.944 7f226f314700 1 --2- 172.21.2.102:0/730691835 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f227009e920 0x7f22700c2960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:14:39.948 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:39.944 7f226e312700 1 --2- 172.21.2.102:0/730691835 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f22700c1f60 0x7f227009e300 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 rx=0x7f226400a570 tx=0x7f226400a910).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:39.949 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:39.945 7f225f7fe700 1 -- 172.21.2.102:0/730691835 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f226400c0d0 con 0x7f22700c1f60 2021-02-10T11:14:39.949 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:39.945 7f225f7fe700 1 -- 172.21.2.102:0/730691835 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f226400c3d0 con 0x7f22700c1f60 2021-02-10T11:14:39.949 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:39.945 7f2274dd3700 1 -- 172.21.2.102:0/730691835 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22701a4910 con 0x7f22700c1f60 2021-02-10T11:14:39.949 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:39.945 7f225f7fe700 1 -- 172.21.2.102:0/730691835 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f226400c820 con 0x7f22700c1f60 2021-02-10T11:14:39.949 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:39.945 7f2274dd3700 1 -- 172.21.2.102:0/730691835 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22700c3270 con 0x7f22700c1f60 2021-02-10T11:14:39.949 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:39.945 7f225f7fe700 1 -- 172.21.2.102:0/730691835 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f2264008030 con 0x7f22700c1f60 2021-02-10T11:14:39.949 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:39.945 7f225f7fe700 1 --2- 172.21.2.102:0/730691835 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f22540186c0 0x7f225401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:39.949 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:39.945 7f225f7fe700 1 -- 172.21.2.102:0/730691835 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f2264032410 con 0x7f22700c1f60 2021-02-10T11:14:39.949 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:39.945 7f226eb13700 1 --2- 172.21.2.102:0/730691835 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f22540186c0 0x7f225401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:39.950 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:39.946 7f226eb13700 1 --2- 172.21.2.102:0/730691835 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f22540186c0 0x7f225401abc0 crc :-1 s=READY pgs=294 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:39.969 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:39.976 7f2274dd3700 1 -- 172.21.2.102:0/730691835 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22700f0b50 con 0x7f22700c1f60 2021-02-10T11:14:39.972 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:39.978 7f225f7fe700 1 -- 172.21.2.102:0/730691835 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f226402ef10 con 0x7f22700c1f60 2021-02-10T11:14:40.098 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:40.104 7f2274dd3700 1 -- 172.21.2.102:0/730691835 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344425", "target": ["mgr", ""]}) v1 -- 0x7f22700c23e0 con 0x7f22540186c0 2021-02-10T11:14:40.125 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:40.130 7f225f7fe700 1 -- 172.21.2.102:0/730691835 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f2268000d30 con 0x7f22540186c0 2021-02-10T11:14:40.131 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:40.136 7f2274dd3700 1 -- 172.21.2.102:0/730691835 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f22540186c0 msgr2=0x7f225401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:40.131 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:40.136 7f2274dd3700 1 --2- 172.21.2.102:0/730691835 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f22540186c0 0x7f225401abc0 crc :-1 s=READY pgs=294 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:40.131 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:40.136 7f2274dd3700 1 -- 172.21.2.102:0/730691835 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f22700c1f60 msgr2=0x7f227009e300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:40.131 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:40.136 7f2274dd3700 1 --2- 172.21.2.102:0/730691835 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f22700c1f60 0x7f227009e300 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 rx=0x7f226400a570 tx=0x7f226400a910).stop 2021-02-10T11:14:40.131 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:40.137 7f2274dd3700 1 -- 172.21.2.102:0/730691835 shutdown_connections 2021-02-10T11:14:40.131 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:40.137 7f2274dd3700 1 --2- 172.21.2.102:0/730691835 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f22540186c0 0x7f225401abc0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:40.131 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:40.137 7f2274dd3700 1 --2- 172.21.2.102:0/730691835 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f227009e920 0x7f22700c2960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:40.131 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:40.137 7f2274dd3700 1 --2- 172.21.2.102:0/730691835 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f22700c1f60 0x7f227009e300 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:40.131 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:40.137 7f2274dd3700 1 --2- 172.21.2.102:0/730691835 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f227009f140 0x7f22700c1940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:40.132 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:40.137 7f2274dd3700 1 -- 172.21.2.102:0/730691835 >> 172.21.2.102:0/730691835 conn(0x7f227009fc10 msgr2=0x7f22700b3250 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:14:40.132 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:40.137 7f2274dd3700 1 -- 172.21.2.102:0/730691835 shutdown_connections 2021-02-10T11:14:40.132 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:40.137 7f2274dd3700 1 -- 172.21.2.102:0/730691835 wait complete. 2021-02-10T11:14:40.139 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000344425 2021-02-10T11:14:40.571 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:40.559 7f1ff591a700 1 Processor -- start 2021-02-10T11:14:40.571 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:40.559 7f1ff591a700 1 -- start start 2021-02-10T11:14:40.571 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:40.560 7f1ff591a700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f1ff01b0870 0x7f1ff00f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:40.571 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:40.560 7f1ff591a700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ff01b0c70 con 0x7f1ff00efe00 2021-02-10T11:14:40.571 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:40.560 7f1ff591a700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ff00f0b50 con 0x7f1ff01b0470 2021-02-10T11:14:40.571 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:40.560 7f1ff591a700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ff00e5ea0 con 0x7f1ff01b0870 2021-02-10T11:14:40.572 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:40.560 7f1fef574700 1 --1- >> v1:172.21.2.113:6789/0 conn(0x7f1ff01b0470 0x7f1ff00ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.113:6789/0 says I am v1:172.21.2.102:52198/0 (socket says 172.21.2.102:52198) 2021-02-10T11:14:40.572 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:40.560 7f1fef574700 1 -- 172.21.2.102:0/3491727476 learned_addr learned my addr 172.21.2.102:0/3491727476 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:14:40.572 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:40.560 7f1fefd75700 1 --1- 172.21.2.102:0/3491727476 >> v1:172.21.2.102:6789/0 conn(0x7f1ff00efe00 0x7f1ff00f0200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:44874/0 (socket says 172.21.2.102:44874) 2021-02-10T11:14:40.572 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:40.560 7f1feed73700 1 --2- 172.21.2.102:0/3491727476 >> v2:172.21.2.113:3300/0 conn(0x7f1ff01b0870 0x7f1ff00f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:40.572 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:40.561 7f1fee572700 1 -- 172.21.2.102:0/3491727476 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 647341666 0 0) 0x7f1fe8000f90 con 0x7f1ff00efe00 2021-02-10T11:14:40.572 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:40.561 7f1fee572700 1 -- 172.21.2.102:0/3491727476 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1fdc002fe0 con 0x7f1ff00efe00 2021-02-10T11:14:40.572 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:40.561 7f1fee572700 1 -- 172.21.2.102:0/3491727476 <== mon.1 v1:172.21.2.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1548451455 0 0) 0x7f1fd8000c60 con 0x7f1ff01b0470 2021-02-10T11:14:40.572 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:40.561 7f1fee572700 1 -- 172.21.2.102:0/3491727476 --> v1:172.21.2.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1fdc004370 con 0x7f1ff01b0470 2021-02-10T11:14:40.572 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:40.561 7f1feed73700 1 -- 172.21.2.102:0/3491727476 >> v1:172.21.2.113:6789/0 conn(0x7f1ff01b0470 legacy=0x7f1ff00ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:40.572 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:40.561 7f1feed73700 1 -- 172.21.2.102:0/3491727476 >> v1:172.21.2.102:6789/0 conn(0x7f1ff00efe00 legacy=0x7f1ff00f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:40.572 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:40.561 7f1feed73700 1 -- 172.21.2.102:0/3491727476 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ff00e52e0 con 0x7f1ff01b0870 2021-02-10T11:14:40.572 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:40.561 7f1feed73700 1 --2- 172.21.2.102:0/3491727476 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1ff01b0870 0x7f1ff00f20e0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7f1fe0006650 tx=0x7f1fe0009a80).ready entity=mon.1 client_cookie=a44964928bb0bf8e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:40.572 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:40.561 7f1fee572700 1 -- 172.21.2.102:0/3491727476 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1fe00009b0 con 0x7f1ff01b0870 2021-02-10T11:14:40.573 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:40.561 7f1fee572700 1 -- 172.21.2.102:0/3491727476 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fe000c0c0 con 0x7f1ff01b0870 2021-02-10T11:14:40.573 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:40.561 7f1fee572700 1 -- 172.21.2.102:0/3491727476 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1fe000c5e0 con 0x7f1ff01b0870 2021-02-10T11:14:40.573 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:40.561 7f1ff591a700 1 -- 172.21.2.102:0/3491727476 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1ff01b0870 msgr2=0x7f1ff00f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:40.573 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:40.561 7f1ff591a700 1 --2- 172.21.2.102:0/3491727476 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1ff01b0870 0x7f1ff00f20e0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7f1fe0006650 tx=0x7f1fe0009a80).stop 2021-02-10T11:14:40.573 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:40.562 7f1ff591a700 1 -- 172.21.2.102:0/3491727476 shutdown_connections 2021-02-10T11:14:40.573 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:40.562 7f1ff591a700 1 --2- 172.21.2.102:0/3491727476 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1ff01b0870 0x7f1ff00f20e0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:40.573 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:40.562 7f1ff591a700 1 -- 172.21.2.102:0/3491727476 >> 172.21.2.102:0/3491727476 conn(0x7f1ff00ce3e0 msgr2=0x7f1ff00e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:14:40.573 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:40.562 7f1ff591a700 1 -- 172.21.2.102:0/3491727476 shutdown_connections 2021-02-10T11:14:40.573 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:40.562 7f1ff591a700 1 -- 172.21.2.102:0/3491727476 wait complete. 2021-02-10T11:14:40.573 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:40.576 7f1ff591a700 1 Processor -- start 2021-02-10T11:14:40.573 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:40.577 7f1ff591a700 1 -- start start 2021-02-10T11:14:40.576 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:40.577 7f1ff591a700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1ff0105d10 0x7f1ff009f880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:40.576 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:40.577 7f1ff591a700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1ff00961f0 0x7f1ff00965f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:40.576 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:40.577 7f1ff591a700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1ff0106890 0x7f1ff0106c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:40.576 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:40.577 7f1ff591a700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ff0106110 con 0x7f1ff00961f0 2021-02-10T11:14:40.576 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:40.577 7f1ff591a700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ff0106380 con 0x7f1ff0106890 2021-02-10T11:14:40.577 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:40.577 7f1ff591a700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ff00a6620 con 0x7f1ff0105d10 2021-02-10T11:14:40.577 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:40.577 7f1feed73700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1ff00961f0 0x7f1ff00965f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:40.577 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:40.577 7f1feed73700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1ff00961f0 0x7f1ff00965f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:52656/0 (socket says 172.21.2.102:52656) 2021-02-10T11:14:40.577 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:40.577 7f1feed73700 1 -- 172.21.2.102:0/1109187205 learned_addr learned my addr 172.21.2.102:0/1109187205 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:14:40.577 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:40.578 7f1fefd75700 1 --2- 172.21.2.102:0/1109187205 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1ff0106890 0x7f1ff0106c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:40.577 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:40.578 7f1fef574700 1 --2- 172.21.2.102:0/1109187205 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1ff0105d10 0x7f1ff009f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:40.577 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:40.578 7f1feed73700 1 -- 172.21.2.102:0/1109187205 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1ff0105d10 msgr2=0x7f1ff009f880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:40.577 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:40.578 7f1feed73700 1 --2- 172.21.2.102:0/1109187205 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1ff0105d10 0x7f1ff009f880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:40.577 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:40.578 7f1feed73700 1 -- 172.21.2.102:0/1109187205 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1ff0106890 msgr2=0x7f1ff0106c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:40.577 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:40.578 7f1feed73700 1 --2- 172.21.2.102:0/1109187205 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1ff0106890 0x7f1ff0106c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:40.577 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:40.578 7f1feed73700 1 -- 172.21.2.102:0/1109187205 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ff01072b0 con 0x7f1ff00961f0 2021-02-10T11:14:40.578 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:40.578 7f1fefd75700 1 --2- 172.21.2.102:0/1109187205 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1ff0106890 0x7f1ff0106c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:14:40.578 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:40.578 7f1fef574700 1 --2- 172.21.2.102:0/1109187205 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1ff0105d10 0x7f1ff009f880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:14:40.578 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:40.578 7f1feed73700 1 --2- 172.21.2.102:0/1109187205 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1ff00961f0 0x7f1ff00965f0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 rx=0x7f1fe0009630 tx=0x7f1fe000b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:40.578 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:40.578 7f1fe7fff700 1 -- 172.21.2.102:0/1109187205 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1fe000bd70 con 0x7f1ff00961f0 2021-02-10T11:14:40.578 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:40.578 7f1fe7fff700 1 -- 172.21.2.102:0/1109187205 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fe0000960 con 0x7f1ff00961f0 2021-02-10T11:14:40.578 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:40.578 7f1fe7fff700 1 -- 172.21.2.102:0/1109187205 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1fe00054d0 con 0x7f1ff00961f0 2021-02-10T11:14:40.578 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:40.578 7f1ff591a700 1 -- 172.21.2.102:0/1109187205 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ff01072b0 con 0x7f1ff00961f0 2021-02-10T11:14:40.578 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:40.578 7f1ff591a700 1 -- 172.21.2.102:0/1109187205 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ff01072b0 con 0x7f1ff00961f0 2021-02-10T11:14:40.578 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:40.579 7f1fe7fff700 1 -- 172.21.2.102:0/1109187205 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1fe000a000 con 0x7f1ff00961f0 2021-02-10T11:14:40.578 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:40.579 7f1fe7fff700 1 --2- 172.21.2.102:0/1109187205 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1fd40186d0 0x7f1fd401abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:40.578 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:40.579 7f1fe7fff700 1 -- 172.21.2.102:0/1109187205 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f1fe0032790 con 0x7f1ff00961f0 2021-02-10T11:14:40.578 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:40.579 7f1fef574700 1 --2- 172.21.2.102:0/1109187205 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1fd40186d0 0x7f1fd401abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:40.579 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:40.579 7f1fef574700 1 --2- 172.21.2.102:0/1109187205 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1fd40186d0 0x7f1fd401abd0 crc :-1 s=READY pgs=295 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:40.604 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:40.610 7f1ff591a700 1 -- 172.21.2.102:0/1109187205 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ff0105ae0 con 0x7f1ff00961f0 2021-02-10T11:14:40.606 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:40.612 7f1fe7fff700 1 -- 172.21.2.102:0/1109187205 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f1fe0000940 con 0x7f1ff00961f0 2021-02-10T11:14:40.735 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:40.740 7f1ff591a700 1 -- 172.21.2.102:0/1109187205 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344425", "target": ["mgr", ""]}) v1 -- 0x7f1ff00f0b50 con 0x7f1fd40186d0 2021-02-10T11:14:40.738 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:40.744 7f1fe7fff700 1 -- 172.21.2.102:0/1109187205 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f1fd8000c10 con 0x7f1fd40186d0 2021-02-10T11:14:40.742 INFO:teuthology.orchestra.run.gibba002.stdout:/volumes/_nogroup/subvolume_0000000000344425/c090ecf1-bae3-4f92-8f43-66e632d85e67 2021-02-10T11:14:40.743 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:40.749 7f1ff591a700 1 -- 172.21.2.102:0/1109187205 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1fd40186d0 msgr2=0x7f1fd401abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:40.743 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:40.749 7f1ff591a700 1 --2- 172.21.2.102:0/1109187205 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1fd40186d0 0x7f1fd401abd0 crc :-1 s=READY pgs=295 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:40.744 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:40.749 7f1ff591a700 1 -- 172.21.2.102:0/1109187205 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1ff00961f0 msgr2=0x7f1ff00965f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:40.744 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:40.749 7f1ff591a700 1 --2- 172.21.2.102:0/1109187205 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1ff00961f0 0x7f1ff00965f0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 rx=0x7f1fe0009630 tx=0x7f1fe000b980).stop 2021-02-10T11:14:40.744 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:40.749 7f1ff591a700 1 -- 172.21.2.102:0/1109187205 shutdown_connections 2021-02-10T11:14:40.744 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:40.749 7f1ff591a700 1 --2- 172.21.2.102:0/1109187205 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1fd40186d0 0x7f1fd401abd0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:40.744 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:40.749 7f1ff591a700 1 --2- 172.21.2.102:0/1109187205 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1ff0106890 0x7f1ff0106c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:40.744 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:40.749 7f1ff591a700 1 --2- 172.21.2.102:0/1109187205 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1ff00961f0 0x7f1ff00965f0 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:40.745 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:40.749 7f1ff591a700 1 --2- 172.21.2.102:0/1109187205 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1ff0105d10 0x7f1ff009f880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:40.745 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:40.749 7f1ff591a700 1 -- 172.21.2.102:0/1109187205 >> 172.21.2.102:0/1109187205 conn(0x7f1ff0095a70 msgr2=0x7f1ff00bf050 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:14:40.745 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:40.749 7f1ff591a700 1 -- 172.21.2.102:0/1109187205 shutdown_connections 2021-02-10T11:14:40.745 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:40.749 7f1ff591a700 1 -- 172.21.2.102:0/1109187205 wait complete. 2021-02-10T11:14:40.751 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000344425 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000344425/c090ecf1-bae3-4f92-8f43-66e632d85e67 2021-02-10T11:14:40.751 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:14:40.751 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:14:40.751 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:14:41.088 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:14:41.088 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:14:41.089 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344425/c090ecf1-bae3-4f92-8f43-66e632d85e67/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:14:41.471 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:14:41.471 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:14:41.471 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0233526 s, 44.9 MB/s 2021-02-10T11:14:41.472 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:14:41.472 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:14:41.472 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:14:41.810 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:14:41.810 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:14:41.811 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344425/c090ecf1-bae3-4f92-8f43-66e632d85e67/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:14:42.185 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:14:42.186 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:14:42.186 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0242291 s, 43.3 MB/s 2021-02-10T11:14:42.187 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:14:42.187 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:14:42.187 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:14:42.524 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:14:42.524 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:14:42.524 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344425/c090ecf1-bae3-4f92-8f43-66e632d85e67/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:14:42.900 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:14:42.900 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:14:42.900 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0231281 s, 45.3 MB/s 2021-02-10T11:14:42.901 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:14:42.901 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:14:42.902 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:14:43.238 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:14:43.238 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:14:43.238 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344425/c090ecf1-bae3-4f92-8f43-66e632d85e67/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:14:43.610 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:14:43.610 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:14:43.611 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.023619 s, 44.4 MB/s 2021-02-10T11:14:43.611 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:14:43.612 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:14:43.612 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:14:43.946 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:14:43.946 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:14:43.947 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344425/c090ecf1-bae3-4f92-8f43-66e632d85e67/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:14:44.322 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:14:52.844 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:14:52.844 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0245913 s, 42.6 MB/s 2021-02-10T11:14:52.846 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:14:52.846 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:14:52.846 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:14:53.181 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:14:53.182 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:14:53.182 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344425/c090ecf1-bae3-4f92-8f43-66e632d85e67/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:14:53.553 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:14:53.553 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:14:53.553 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0228659 s, 45.9 MB/s 2021-02-10T11:14:53.554 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:14:53.554 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:14:53.555 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:14:53.890 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:14:53.890 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:14:53.891 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344425/c090ecf1-bae3-4f92-8f43-66e632d85e67/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:14:54.265 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:14:54.266 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:14:54.266 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0236326 s, 44.4 MB/s 2021-02-10T11:14:54.267 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:14:54.267 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:14:54.267 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:14:54.604 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:14:54.605 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:14:54.605 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344425/c090ecf1-bae3-4f92-8f43-66e632d85e67/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:14:54.983 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:14:54.983 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:14:54.983 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0248345 s, 42.2 MB/s 2021-02-10T11:14:54.984 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:14:54.985 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:14:54.985 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:14:55.322 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:14:55.322 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:14:55.322 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344425/c090ecf1-bae3-4f92-8f43-66e632d85e67/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:14:55.698 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:14:55.699 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:14:55.699 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0232318 s, 45.1 MB/s 2021-02-10T11:14:55.699 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:14:55.700 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:14:55.700 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:14:56.035 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:14:56.035 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:14:56.035 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344425/c090ecf1-bae3-4f92-8f43-66e632d85e67/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:14:56.409 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:14:56.410 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:14:56.410 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0234684 s, 44.7 MB/s 2021-02-10T11:14:56.412 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000344426 2021-02-10T11:14:56.503 DEBUG:teuthology.orchestra.run.gibba002:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:14:56.505 DEBUG:teuthology.orchestra.run.gibba013:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:14:56.845 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:56.834 7fb985872700 1 Processor -- start 2021-02-10T11:14:56.845 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:56.834 7fb985872700 1 -- start start 2021-02-10T11:14:56.845 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:56.835 7fb985872700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fb9801b0470 0x7fb9801b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:56.845 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:56.835 7fb985872700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9800f0b50 con 0x7fb9800ef610 2021-02-10T11:14:56.846 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:56.835 7fb985872700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9800f24e0 con 0x7fb9800f20e0 2021-02-10T11:14:56.846 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:56.835 7fb985872700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9800e5ea0 con 0x7fb9801b0470 2021-02-10T11:14:56.846 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:56.835 7fb97f574700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fb9801b0470 0x7fb9801b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:56.846 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:56.835 7fb97f574700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fb9801b0470 0x7fb9801b0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:52662/0 (socket says 172.21.2.102:52662) 2021-02-10T11:14:56.846 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:56.835 7fb97f574700 1 -- 172.21.2.102:0/393451617 learned_addr learned my addr 172.21.2.102:0/393451617 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:14:56.846 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:56.835 7fb976d73700 1 --1- 172.21.2.102:0/393451617 >> v1:172.21.2.102:6789/0 conn(0x7fb9800ef610 0x7fb9800efa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:44886/0 (socket says 172.21.2.102:44886) 2021-02-10T11:14:56.846 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:56.836 7fb97ed73700 1 -- 172.21.2.102:0/393451617 <== mon.1 v1:172.21.2.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 93135774 0 0) 0x7fb9780009b0 con 0x7fb9800f20e0 2021-02-10T11:14:56.846 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:56.836 7fb97f574700 1 -- 172.21.2.102:0/393451617 >> v1:172.21.2.113:6789/0 conn(0x7fb9800f20e0 legacy=0x7fb9800f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:56.846 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:56.836 7fb97f574700 1 -- 172.21.2.102:0/393451617 >> v1:172.21.2.102:6789/0 conn(0x7fb9800ef610 legacy=0x7fb9800efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:56.846 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:56.836 7fb97f574700 1 -- 172.21.2.102:0/393451617 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9800e52e0 con 0x7fb9801b0470 2021-02-10T11:14:56.846 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:56.836 7fb97f574700 1 --2- 172.21.2.102:0/393451617 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb9801b0470 0x7fb9801b0870 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 rx=0x7fb970006a80 tx=0x7fb970009f00).ready entity=mon.0 client_cookie=a9e9c2ba3c3a333e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:56.846 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:56.836 7fb97ed73700 1 -- 172.21.2.102:0/393451617 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb97000b5c0 con 0x7fb9801b0470 2021-02-10T11:14:56.847 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:56.836 7fb97ed73700 1 -- 172.21.2.102:0/393451617 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb97000b8c0 con 0x7fb9801b0470 2021-02-10T11:14:56.847 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:56.836 7fb97ed73700 1 -- 172.21.2.102:0/393451617 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb97000bde0 con 0x7fb9801b0470 2021-02-10T11:14:56.847 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:56.836 7fb985872700 1 -- 172.21.2.102:0/393451617 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb9801b0470 msgr2=0x7fb9801b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:56.847 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:56.836 7fb985872700 1 --2- 172.21.2.102:0/393451617 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb9801b0470 0x7fb9801b0870 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 rx=0x7fb970006a80 tx=0x7fb970009f00).stop 2021-02-10T11:14:56.847 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:56.837 7fb985872700 1 -- 172.21.2.102:0/393451617 shutdown_connections 2021-02-10T11:14:56.847 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:56.837 7fb985872700 1 --2- 172.21.2.102:0/393451617 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb9801b0470 0x7fb9801b0870 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:56.847 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:56.837 7fb985872700 1 -- 172.21.2.102:0/393451617 >> 172.21.2.102:0/393451617 conn(0x7fb9800ce3e0 msgr2=0x7fb9800e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:14:56.847 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:56.837 7fb985872700 1 -- 172.21.2.102:0/393451617 shutdown_connections 2021-02-10T11:14:56.847 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:56.837 7fb985872700 1 -- 172.21.2.102:0/393451617 wait complete. 2021-02-10T11:14:56.848 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:56.852 7fb985872700 1 Processor -- start 2021-02-10T11:14:56.848 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:56.852 7fb985872700 1 -- start start 2021-02-10T11:14:56.848 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:56.852 7fb985872700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb9801a0570 0x7fb9800a4590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:56.848 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:56.852 7fb985872700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb9800f1fd0 0x7fb9800f23d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:56.848 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:56.852 7fb985872700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb9800a5450 0x7fb9800a5850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:56.848 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:56.852 7fb985872700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9800a4bb0 con 0x7fb9801a0570 2021-02-10T11:14:56.848 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:56.852 7fb985872700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9800b35e0 con 0x7fb9800a5450 2021-02-10T11:14:56.848 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:56.852 7fb985872700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9800a1f00 con 0x7fb9800f1fd0 2021-02-10T11:14:56.849 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:56.852 7fb97f574700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb9801a0570 0x7fb9800a4590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:56.850 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:56.852 7fb97f574700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb9801a0570 0x7fb9800a4590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:52668/0 (socket says 172.21.2.102:52668) 2021-02-10T11:14:56.850 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:56.852 7fb97f574700 1 -- 172.21.2.102:0/2165893219 learned_addr learned my addr 172.21.2.102:0/2165893219 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:14:56.850 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:56.852 7fb976d73700 1 --2- 172.21.2.102:0/2165893219 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb9800f1fd0 0x7fb9800f23d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:56.850 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:56.852 7fb97fd75700 1 --2- 172.21.2.102:0/2165893219 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb9800a5450 0x7fb9800a5850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:56.850 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:56.853 7fb97f574700 1 -- 172.21.2.102:0/2165893219 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb9800f1fd0 msgr2=0x7fb9800f23d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:56.850 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:56.853 7fb97f574700 1 --2- 172.21.2.102:0/2165893219 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb9800f1fd0 0x7fb9800f23d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:56.850 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:56.853 7fb97f574700 1 -- 172.21.2.102:0/2165893219 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb9800a5450 msgr2=0x7fb9800a5850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:56.850 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:56.853 7fb97f574700 1 --2- 172.21.2.102:0/2165893219 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb9800a5450 0x7fb9800a5850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:56.850 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:56.853 7fb97f574700 1 -- 172.21.2.102:0/2165893219 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9800a5e70 con 0x7fb9801a0570 2021-02-10T11:14:56.850 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:56.853 7fb97f574700 1 --2- 172.21.2.102:0/2165893219 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb9801a0570 0x7fb9800a4590 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 rx=0x7fb970009450 tx=0x7fb97000bd20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:56.850 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:56.853 7fb97cd6f700 1 -- 172.21.2.102:0/2165893219 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb970006e40 con 0x7fb9801a0570 2021-02-10T11:14:56.850 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:56.853 7fb97cd6f700 1 -- 172.21.2.102:0/2165893219 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb97000be60 con 0x7fb9801a0570 2021-02-10T11:14:56.851 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:56.853 7fb97cd6f700 1 -- 172.21.2.102:0/2165893219 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb97000c4e0 con 0x7fb9801a0570 2021-02-10T11:14:56.851 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:56.853 7fb985872700 1 -- 172.21.2.102:0/2165893219 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9800a5e70 con 0x7fb9801a0570 2021-02-10T11:14:56.851 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:56.853 7fb985872700 1 -- 172.21.2.102:0/2165893219 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9800a5e70 con 0x7fb9801a0570 2021-02-10T11:14:56.851 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:56.854 7fb97cd6f700 1 -- 172.21.2.102:0/2165893219 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fb970005660 con 0x7fb9801a0570 2021-02-10T11:14:56.851 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:56.854 7fb97cd6f700 1 --2- 172.21.2.102:0/2165893219 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb9640186c0 0x7fb96401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:56.851 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:56.854 7fb97cd6f700 1 -- 172.21.2.102:0/2165893219 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fb9700328d0 con 0x7fb9801a0570 2021-02-10T11:14:56.851 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:56.854 7fb976d73700 1 --2- 172.21.2.102:0/2165893219 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb9640186c0 0x7fb96401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:56.851 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:56.854 7fb976d73700 1 --2- 172.21.2.102:0/2165893219 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb9640186c0 0x7fb96401abc0 crc :-1 s=READY pgs=296 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:56.878 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:56.885 7fb985872700 1 -- 172.21.2.102:0/2165893219 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9800a4bb0 con 0x7fb9801a0570 2021-02-10T11:14:56.881 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:56.887 7fb97cd6f700 1 -- 172.21.2.102:0/2165893219 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fb97001f860 con 0x7fb9801a0570 2021-02-10T11:14:57.010 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:57.016 7fb985872700 1 -- 172.21.2.102:0/2165893219 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344426", "target": ["mgr", ""]}) v1 -- 0x7fb9800a4bb0 con 0x7fb9640186c0 2021-02-10T11:14:57.034 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:57.041 7fb97cd6f700 1 -- 172.21.2.102:0/2165893219 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fb9680073f0 con 0x7fb9640186c0 2021-02-10T11:14:57.044 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:57.049 7fb985872700 1 -- 172.21.2.102:0/2165893219 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb9640186c0 msgr2=0x7fb96401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:57.044 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:57.049 7fb985872700 1 --2- 172.21.2.102:0/2165893219 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb9640186c0 0x7fb96401abc0 crc :-1 s=READY pgs=296 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:57.044 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:57.049 7fb985872700 1 -- 172.21.2.102:0/2165893219 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb9801a0570 msgr2=0x7fb9800a4590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:57.044 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:57.049 7fb985872700 1 --2- 172.21.2.102:0/2165893219 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb9801a0570 0x7fb9800a4590 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 rx=0x7fb970009450 tx=0x7fb97000bd20).stop 2021-02-10T11:14:57.044 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:57.049 7fb985872700 1 -- 172.21.2.102:0/2165893219 shutdown_connections 2021-02-10T11:14:57.045 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:57.049 7fb985872700 1 --2- 172.21.2.102:0/2165893219 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb9640186c0 0x7fb96401abc0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:57.045 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:57.049 7fb985872700 1 --2- 172.21.2.102:0/2165893219 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb9800a5450 0x7fb9800a5850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:57.045 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:57.049 7fb985872700 1 --2- 172.21.2.102:0/2165893219 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb9800f1fd0 0x7fb9800f23d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:57.045 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:57.049 7fb985872700 1 --2- 172.21.2.102:0/2165893219 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb9801a0570 0x7fb9800a4590 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:57.045 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:57.049 7fb985872700 1 -- 172.21.2.102:0/2165893219 >> 172.21.2.102:0/2165893219 conn(0x7fb9800958b0 msgr2=0x7fb9800bf040 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:14:57.045 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:57.049 7fb985872700 1 -- 172.21.2.102:0/2165893219 shutdown_connections 2021-02-10T11:14:57.046 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:57.049 7fb985872700 1 -- 172.21.2.102:0/2165893219 wait complete. 2021-02-10T11:14:57.051 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000344426 2021-02-10T11:14:57.522 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:57.512 7f8a73144700 1 Processor -- start 2021-02-10T11:14:57.522 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:57.512 7f8a73144700 1 -- start start 2021-02-10T11:14:57.522 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:57.512 7f8a73144700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f8a6c1b0870 0x7f8a6c0f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:57.522 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:57.512 7f8a73144700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f8a6c0efe00 0x7f8a6c0f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:57.522 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:57.512 7f8a73144700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a6c1b0c70 con 0x7f8a6c1b0470 2021-02-10T11:14:57.523 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:57.512 7f8a73144700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a6c0e5ea0 con 0x7f8a6c0efe00 2021-02-10T11:14:57.523 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:57.512 7f8a73144700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a6c0f0b50 con 0x7f8a6c1b0870 2021-02-10T11:14:57.523 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:57.513 7f8a71761700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f8a6c0efe00 0x7f8a6c0f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:57.523 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:57.513 7f8a71761700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f8a6c0efe00 0x7f8a6c0f0200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:52680/0 (socket says 172.21.2.102:52680) 2021-02-10T11:14:57.523 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:57.513 7f8a71761700 1 -- 172.21.2.102:0/770837792 learned_addr learned my addr 172.21.2.102:0/770837792 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:14:57.523 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:57.513 7f8a71761700 1 -- 172.21.2.102:0/770837792 >> v2:172.21.2.113:3300/0 conn(0x7f8a6c1b0870 msgr2=0x7f8a6c0f20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:14:57.523 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:57.513 7f8a71761700 1 --2- 172.21.2.102:0/770837792 >> v2:172.21.2.113:3300/0 conn(0x7f8a6c1b0870 0x7f8a6c0f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:57.523 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:57.513 7f8a71761700 1 -- 172.21.2.102:0/770837792 >> v1:172.21.2.113:6789/0 conn(0x7f8a6c1b0470 legacy=0x7f8a6c0ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:14:57.523 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:57.513 7f8a71761700 1 -- 172.21.2.102:0/770837792 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a6c0f0820 con 0x7f8a6c0efe00 2021-02-10T11:14:57.523 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:57.513 7f8a71761700 1 --2- 172.21.2.102:0/770837792 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8a6c0efe00 0x7f8a6c0f0200 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 rx=0x7f8a64006f00 tx=0x7f8a64009f10).ready entity=mon.0 client_cookie=bf8164a191e8ce75 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:57.523 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:57.513 7f8a6b7fe700 1 -- 172.21.2.102:0/770837792 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8a6400b3b0 con 0x7f8a6c0efe00 2021-02-10T11:14:57.523 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:57.513 7f8a6b7fe700 1 -- 172.21.2.102:0/770837792 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a6400b730 con 0x7f8a6c0efe00 2021-02-10T11:14:57.524 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:57.513 7f8a6b7fe700 1 -- 172.21.2.102:0/770837792 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8a6400bc50 con 0x7f8a6c0efe00 2021-02-10T11:14:57.524 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:57.513 7f8a73144700 1 -- 172.21.2.102:0/770837792 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8a6c0efe00 msgr2=0x7f8a6c0f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:57.524 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:57.513 7f8a73144700 1 --2- 172.21.2.102:0/770837792 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8a6c0efe00 0x7f8a6c0f0200 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 rx=0x7f8a64006f00 tx=0x7f8a64009f10).stop 2021-02-10T11:14:57.524 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:57.514 7f8a73144700 1 -- 172.21.2.102:0/770837792 shutdown_connections 2021-02-10T11:14:57.524 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:57.514 7f8a73144700 1 --2- 172.21.2.102:0/770837792 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8a6c0efe00 0x7f8a6c0f0200 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:57.524 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:57.514 7f8a73144700 1 --2- 172.21.2.102:0/770837792 >> v2:172.21.2.113:3300/0 conn(0x7f8a6c1b0870 0x7f8a6c0f20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:57.524 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:57.514 7f8a73144700 1 -- 172.21.2.102:0/770837792 >> 172.21.2.102:0/770837792 conn(0x7f8a6c0ce3e0 msgr2=0x7f8a6c0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:14:57.524 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:57.514 7f8a73144700 1 -- 172.21.2.102:0/770837792 shutdown_connections 2021-02-10T11:14:57.524 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:57.514 7f8a73144700 1 -- 172.21.2.102:0/770837792 wait complete. 2021-02-10T11:14:57.527 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:57.528 7f8a73144700 1 Processor -- start 2021-02-10T11:14:57.527 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:57.529 7f8a73144700 1 -- start start 2021-02-10T11:14:57.527 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:57.529 7f8a73144700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8a6c0b2840 0x7f8a6c1a4300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:57.527 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:57.529 7f8a73144700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8a6c0b30d0 0x7f8a6c0b34d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:57.527 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:57.529 7f8a73144700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8a6c0b3af0 0x7f8a6c0f0580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:57.527 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:57.529 7f8a73144700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a6c0b2c40 con 0x7f8a6c0b30d0 2021-02-10T11:14:57.527 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:57.529 7f8a73144700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a6c1a4920 con 0x7f8a6c0b3af0 2021-02-10T11:14:57.527 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:57.529 7f8a73144700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a6c0f0ba0 con 0x7f8a6c0b2840 2021-02-10T11:14:57.527 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:57.529 7f8a6bfff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8a6c0b30d0 0x7f8a6c0b34d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:57.527 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:57.529 7f8a6bfff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8a6c0b30d0 0x7f8a6c0b34d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:52682/0 (socket says 172.21.2.102:52682) 2021-02-10T11:14:57.528 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:57.529 7f8a6bfff700 1 -- 172.21.2.102:0/1349080977 learned_addr learned my addr 172.21.2.102:0/1349080977 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:14:57.528 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:57.529 7f8a6bfff700 1 -- 172.21.2.102:0/1349080977 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8a6c0b2840 msgr2=0x7f8a6c1a4300 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:14:57.528 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:57.529 7f8a6bfff700 1 --2- 172.21.2.102:0/1349080977 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8a6c0b2840 0x7f8a6c1a4300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:57.528 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:57.529 7f8a6bfff700 1 -- 172.21.2.102:0/1349080977 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8a6c0b3af0 msgr2=0x7f8a6c0f0580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:14:57.528 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:57.529 7f8a6bfff700 1 --2- 172.21.2.102:0/1349080977 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8a6c0b3af0 0x7f8a6c0f0580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:57.528 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:57.529 7f8a6bfff700 1 -- 172.21.2.102:0/1349080977 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a6c0f0dc0 con 0x7f8a6c0b30d0 2021-02-10T11:14:57.528 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:57.530 7f8a6bfff700 1 --2- 172.21.2.102:0/1349080977 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8a6c0b30d0 0x7f8a6c0b34d0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 rx=0x7f8a5c006f00 tx=0x7f8a5c00b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:57.528 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:57.530 7f8a697fa700 1 -- 172.21.2.102:0/1349080977 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8a5c00bd30 con 0x7f8a6c0b30d0 2021-02-10T11:14:57.528 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:57.530 7f8a697fa700 1 -- 172.21.2.102:0/1349080977 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a5c000c90 con 0x7f8a6c0b30d0 2021-02-10T11:14:57.528 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:57.530 7f8a697fa700 1 -- 172.21.2.102:0/1349080977 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8a5c0022a0 con 0x7f8a6c0b30d0 2021-02-10T11:14:57.528 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:57.530 7f8a73144700 1 -- 172.21.2.102:0/1349080977 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a6c0f0dc0 con 0x7f8a6c0b30d0 2021-02-10T11:14:57.528 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:57.530 7f8a73144700 1 -- 172.21.2.102:0/1349080977 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a6c0f1350 con 0x7f8a6c0b30d0 2021-02-10T11:14:57.528 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:57.531 7f8a697fa700 1 -- 172.21.2.102:0/1349080977 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f8a5c005310 con 0x7f8a6c0b30d0 2021-02-10T11:14:57.529 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:57.531 7f8a697fa700 1 --2- 172.21.2.102:0/1349080977 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8a500186c0 0x7f8a5001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:14:57.529 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:57.531 7f8a697fa700 1 -- 172.21.2.102:0/1349080977 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f8a5c032510 con 0x7f8a6c0b30d0 2021-02-10T11:14:57.529 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:57.531 7f8a70f60700 1 --2- 172.21.2.102:0/1349080977 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8a500186c0 0x7f8a5001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:14:57.529 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:57.531 7f8a70f60700 1 --2- 172.21.2.102:0/1349080977 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8a500186c0 0x7f8a5001abc0 crc :-1 s=READY pgs=297 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:14:57.556 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:57.562 7f8a73144700 1 -- 172.21.2.102:0/1349080977 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a6c0f18f0 con 0x7f8a6c0b30d0 2021-02-10T11:14:57.558 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:57.565 7f8a697fa700 1 -- 172.21.2.102:0/1349080977 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f8a5c000960 con 0x7f8a6c0b30d0 2021-02-10T11:14:57.688 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:57.694 7f8a73144700 1 -- 172.21.2.102:0/1349080977 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344426", "target": ["mgr", ""]}) v1 -- 0x7f8a6c0f1350 con 0x7f8a500186c0 2021-02-10T11:14:57.690 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:57.697 7f8a697fa700 1 -- 172.21.2.102:0/1349080977 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f8a54006970 con 0x7f8a500186c0 2021-02-10T11:14:57.691 INFO:teuthology.orchestra.run.gibba002.stdout:/volumes/_nogroup/subvolume_0000000000344426/f325a80d-4fb5-4f1e-a381-ab4cd97503a0 2021-02-10T11:14:57.693 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:57.698 7f8a73144700 1 -- 172.21.2.102:0/1349080977 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8a500186c0 msgr2=0x7f8a5001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:57.693 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:57.698 7f8a73144700 1 --2- 172.21.2.102:0/1349080977 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8a500186c0 0x7f8a5001abc0 crc :-1 s=READY pgs=297 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:57.693 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:57.698 7f8a73144700 1 -- 172.21.2.102:0/1349080977 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8a6c0b30d0 msgr2=0x7f8a6c0b34d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:14:57.693 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:57.698 7f8a73144700 1 --2- 172.21.2.102:0/1349080977 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8a6c0b30d0 0x7f8a6c0b34d0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 rx=0x7f8a5c006f00 tx=0x7f8a5c00b940).stop 2021-02-10T11:14:57.693 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:57.699 7f8a73144700 1 -- 172.21.2.102:0/1349080977 shutdown_connections 2021-02-10T11:14:57.694 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:57.699 7f8a73144700 1 --2- 172.21.2.102:0/1349080977 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8a500186c0 0x7f8a5001abc0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:57.694 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:57.699 7f8a73144700 1 --2- 172.21.2.102:0/1349080977 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8a6c0b3af0 0x7f8a6c0f0580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:57.694 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:57.699 7f8a73144700 1 --2- 172.21.2.102:0/1349080977 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8a6c0b30d0 0x7f8a6c0b34d0 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:14:57.694 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:57.699 7f8a73144700 1 --2- 172.21.2.102:0/1349080977 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8a6c0b2840 0x7f8a6c1a4300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:14:57.694 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:57.699 7f8a73144700 1 -- 172.21.2.102:0/1349080977 >> 172.21.2.102:0/1349080977 conn(0x7f8a6c0c36f0 msgr2=0x7f8a6c094c30 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:14:57.694 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:57.699 7f8a73144700 1 -- 172.21.2.102:0/1349080977 shutdown_connections 2021-02-10T11:14:57.694 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:14:57.699 7f8a73144700 1 -- 172.21.2.102:0/1349080977 wait complete. 2021-02-10T11:14:57.701 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000344426 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000344426/f325a80d-4fb5-4f1e-a381-ab4cd97503a0 2021-02-10T11:14:57.701 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:14:57.702 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:14:57.702 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:14:58.038 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:14:58.038 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:14:58.039 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344426/f325a80d-4fb5-4f1e-a381-ab4cd97503a0/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:14:58.415 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:14:58.416 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:14:58.416 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.024221 s, 43.3 MB/s 2021-02-10T11:14:58.416 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:14:58.417 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:14:58.417 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:14:58.751 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:14:58.752 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:14:58.752 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344426/f325a80d-4fb5-4f1e-a381-ab4cd97503a0/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:14:59.128 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:14:59.128 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:14:59.128 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0238343 s, 44.0 MB/s 2021-02-10T11:14:59.130 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:14:59.130 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:14:59.131 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:14:59.466 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:14:59.467 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:14:59.467 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344426/f325a80d-4fb5-4f1e-a381-ab4cd97503a0/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:14:59.843 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:14:59.844 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:14:59.844 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0240732 s, 43.6 MB/s 2021-02-10T11:14:59.845 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:14:59.845 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:14:59.845 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:15:00.181 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:15:00.181 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:15:00.181 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344426/f325a80d-4fb5-4f1e-a381-ab4cd97503a0/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:15:00.558 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:15:00.559 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:15:00.559 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0253293 s, 41.4 MB/s 2021-02-10T11:15:00.560 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:15:00.561 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:15:00.561 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:15:00.896 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:15:00.897 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:15:00.897 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344426/f325a80d-4fb5-4f1e-a381-ab4cd97503a0/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:15:01.274 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:15:01.275 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:15:01.275 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.025071 s, 41.8 MB/s 2021-02-10T11:15:01.275 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:15:01.275 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:15:01.275 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:15:01.611 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:15:01.612 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:15:01.612 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344426/f325a80d-4fb5-4f1e-a381-ab4cd97503a0/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:15:01.987 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:15:01.987 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:15:01.987 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0241058 s, 43.5 MB/s 2021-02-10T11:15:01.988 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:15:01.988 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:15:01.988 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:15:02.323 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:15:02.324 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:15:02.324 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344426/f325a80d-4fb5-4f1e-a381-ab4cd97503a0/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:15:02.699 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:15:04.962 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:15:04.962 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0246622 s, 42.5 MB/s 2021-02-10T11:15:04.964 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:15:04.964 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:15:04.964 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:15:05.300 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:15:05.301 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:15:05.301 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344426/f325a80d-4fb5-4f1e-a381-ab4cd97503a0/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:15:05.672 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:15:05.673 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:15:05.673 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0227923 s, 46.0 MB/s 2021-02-10T11:15:05.674 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:15:05.674 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:15:05.674 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:15:06.010 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:15:06.011 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:15:06.011 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344426/f325a80d-4fb5-4f1e-a381-ab4cd97503a0/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:15:06.386 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:15:06.386 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:15:06.386 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0211834 s, 49.5 MB/s 2021-02-10T11:15:06.387 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:15:06.388 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:15:06.388 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:15:06.722 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:15:06.723 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:15:06.723 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344426/f325a80d-4fb5-4f1e-a381-ab4cd97503a0/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:15:07.099 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:15:07.099 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:15:07.099 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0236502 s, 44.3 MB/s 2021-02-10T11:15:07.100 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000344427 2021-02-10T11:15:07.532 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:07.521 7f24caa11700 1 Processor -- start 2021-02-10T11:15:07.532 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:07.521 7f24caa11700 1 -- start start 2021-02-10T11:15:07.532 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:07.522 7f24caa11700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f24c41b0870 0x7f24c40f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:07.533 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:07.522 7f24caa11700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f24c40efe00 0x7f24c40f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:07.533 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:07.522 7f24caa11700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24c41b0c70 con 0x7f24c41b0470 2021-02-10T11:15:07.533 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:07.522 7f24caa11700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24c40f0b50 con 0x7f24c40efe00 2021-02-10T11:15:07.533 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:07.522 7f24caa11700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24c40f0820 con 0x7f24c41b0870 2021-02-10T11:15:07.533 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:07.522 7f24c902e700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f24c40efe00 0x7f24c40f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:07.534 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:07.522 7f24c902e700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f24c40efe00 0x7f24c40f0200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:52694/0 (socket says 172.21.2.102:52694) 2021-02-10T11:15:07.534 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:07.522 7f24c902e700 1 -- 172.21.2.102:0/3066729015 learned_addr learned my addr 172.21.2.102:0/3066729015 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:15:07.534 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:07.522 7f24c3fff700 1 --2- 172.21.2.102:0/3066729015 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f24c41b0870 0x7f24c40f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:07.534 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:07.522 7f24c902e700 1 -- 172.21.2.102:0/3066729015 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f24c41b0870 msgr2=0x7f24c40f20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:15:07.534 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:07.522 7f24c902e700 1 --2- 172.21.2.102:0/3066729015 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f24c41b0870 0x7f24c40f20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:07.534 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:07.522 7f24c902e700 1 -- 172.21.2.102:0/3066729015 >> v1:172.21.2.113:6789/0 conn(0x7f24c41b0470 legacy=0x7f24c40ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:07.535 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:07.522 7f24c902e700 1 -- 172.21.2.102:0/3066729015 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24c40e52e0 con 0x7f24c40efe00 2021-02-10T11:15:07.535 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:07.522 7f24c902e700 1 --2- 172.21.2.102:0/3066729015 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f24c40efe00 0x7f24c40f0200 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 rx=0x7f24bc006f00 tx=0x7f24bc009e90).ready entity=mon.0 client_cookie=848d24ee0ab4ca4e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:07.535 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:07.523 7f24c37fe700 1 -- 172.21.2.102:0/3066729015 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f24bc00b300 con 0x7f24c40efe00 2021-02-10T11:15:07.537 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:07.523 7f24c37fe700 1 -- 172.21.2.102:0/3066729015 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24bc00b580 con 0x7f24c40efe00 2021-02-10T11:15:07.538 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:07.523 7f24c37fe700 1 -- 172.21.2.102:0/3066729015 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f24bc00b9e0 con 0x7f24c40efe00 2021-02-10T11:15:07.538 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:07.523 7f24caa11700 1 -- 172.21.2.102:0/3066729015 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f24c40efe00 msgr2=0x7f24c40f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:07.538 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:07.523 7f24caa11700 1 --2- 172.21.2.102:0/3066729015 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f24c40efe00 0x7f24c40f0200 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 rx=0x7f24bc006f00 tx=0x7f24bc009e90).stop 2021-02-10T11:15:07.538 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:07.523 7f24caa11700 1 -- 172.21.2.102:0/3066729015 shutdown_connections 2021-02-10T11:15:07.538 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:07.523 7f24caa11700 1 --2- 172.21.2.102:0/3066729015 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f24c41b0870 0x7f24c40f20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:07.539 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:07.523 7f24caa11700 1 --2- 172.21.2.102:0/3066729015 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f24c40efe00 0x7f24c40f0200 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:07.539 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:07.523 7f24caa11700 1 -- 172.21.2.102:0/3066729015 >> 172.21.2.102:0/3066729015 conn(0x7f24c40ce3e0 msgr2=0x7f24c40e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:15:07.539 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:07.523 7f24caa11700 1 -- 172.21.2.102:0/3066729015 shutdown_connections 2021-02-10T11:15:07.539 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:07.523 7f24caa11700 1 -- 172.21.2.102:0/3066729015 wait complete. 2021-02-10T11:15:07.539 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:07.538 7f24caa11700 1 Processor -- start 2021-02-10T11:15:07.539 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:07.538 7f24caa11700 1 -- start start 2021-02-10T11:15:07.539 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:07.538 7f24caa11700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f24c4190920 0x7f24c40b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:07.540 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:07.538 7f24caa11700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f24c40b8200 0x7f24c40b8600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:07.540 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:07.538 7f24caa11700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f24c40c1d80 0x7f24c40c2180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:07.540 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:07.538 7f24caa11700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24c4190d20 con 0x7f24c4190920 2021-02-10T11:15:07.540 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:07.538 7f24caa11700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24c40b8c20 con 0x7f24c40c1d80 2021-02-10T11:15:07.540 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:07.538 7f24caa11700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24c40c27a0 con 0x7f24c40b8200 2021-02-10T11:15:07.540 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:07.539 7f24c882d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f24c4190920 0x7f24c40b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:07.540 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:07.539 7f24c882d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f24c4190920 0x7f24c40b7be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:52700/0 (socket says 172.21.2.102:52700) 2021-02-10T11:15:07.541 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:07.539 7f24c882d700 1 -- 172.21.2.102:0/2522637522 learned_addr learned my addr 172.21.2.102:0/2522637522 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:15:07.542 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:07.539 7f24c882d700 1 -- 172.21.2.102:0/2522637522 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f24c40b8200 msgr2=0x7f24c40b8600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:07.542 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:07.539 7f24c882d700 1 --2- 172.21.2.102:0/2522637522 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f24c40b8200 0x7f24c40b8600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:15:07.543 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:07.539 7f24c882d700 1 -- 172.21.2.102:0/2522637522 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f24c40c1d80 msgr2=0x7f24c40c2180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:07.543 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:07.539 7f24c882d700 1 --2- 172.21.2.102:0/2522637522 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f24c40c1d80 0x7f24c40c2180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:15:07.543 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:07.539 7f24c882d700 1 -- 172.21.2.102:0/2522637522 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24c40c29c0 con 0x7f24c4190920 2021-02-10T11:15:07.543 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:07.539 7f24c882d700 1 --2- 172.21.2.102:0/2522637522 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f24c4190920 0x7f24c40b7be0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 rx=0x7f24ac006ed0 tx=0x7f24ac00b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:07.543 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:07.539 7f24c17fa700 1 -- 172.21.2.102:0/2522637522 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f24ac006020 con 0x7f24c4190920 2021-02-10T11:15:07.543 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:07.539 7f24c17fa700 1 -- 172.21.2.102:0/2522637522 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24ac006360 con 0x7f24c4190920 2021-02-10T11:15:07.543 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:07.539 7f24caa11700 1 -- 172.21.2.102:0/2522637522 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24c40c29c0 con 0x7f24c4190920 2021-02-10T11:15:07.544 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:07.539 7f24caa11700 1 -- 172.21.2.102:0/2522637522 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24c40c29c0 con 0x7f24c4190920 2021-02-10T11:15:07.544 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:07.540 7f24c17fa700 1 -- 172.21.2.102:0/2522637522 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f24ac005730 con 0x7f24c4190920 2021-02-10T11:15:07.544 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:07.540 7f24c17fa700 1 -- 172.21.2.102:0/2522637522 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f24ac0054b0 con 0x7f24c4190920 2021-02-10T11:15:07.544 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:07.540 7f24c17fa700 1 --2- 172.21.2.102:0/2522637522 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f24a8018730 0x7f24a801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:07.544 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:07.540 7f24c17fa700 1 -- 172.21.2.102:0/2522637522 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f24ac032810 con 0x7f24c4190920 2021-02-10T11:15:07.544 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:07.540 7f24c3fff700 1 --2- 172.21.2.102:0/2522637522 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f24a8018730 0x7f24a801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:07.545 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:07.540 7f24c3fff700 1 --2- 172.21.2.102:0/2522637522 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f24a8018730 0x7f24a801abc0 crc :-1 s=READY pgs=298 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:07.565 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:07.571 7f24caa11700 1 -- 172.21.2.102:0/2522637522 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24c4105ae0 con 0x7f24c4190920 2021-02-10T11:15:07.567 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:07.574 7f24c17fa700 1 -- 172.21.2.102:0/2522637522 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f24ac006000 con 0x7f24c4190920 2021-02-10T11:15:07.693 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:07.699 7f24caa11700 1 -- 172.21.2.102:0/2522637522 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344427", "target": ["mgr", ""]}) v1 -- 0x7f24c40b8c20 con 0x7f24a8018730 2021-02-10T11:15:07.720 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:07.727 7f24c17fa700 1 -- 172.21.2.102:0/2522637522 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f24b40052e0 con 0x7f24a8018730 2021-02-10T11:15:07.728 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:07.732 7f24caa11700 1 -- 172.21.2.102:0/2522637522 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f24a8018730 msgr2=0x7f24a801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:07.728 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:07.732 7f24caa11700 1 --2- 172.21.2.102:0/2522637522 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f24a8018730 0x7f24a801abc0 crc :-1 s=READY pgs=298 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:07.728 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:07.732 7f24caa11700 1 -- 172.21.2.102:0/2522637522 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f24c4190920 msgr2=0x7f24c40b7be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:07.729 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:07.732 7f24caa11700 1 --2- 172.21.2.102:0/2522637522 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f24c4190920 0x7f24c40b7be0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 rx=0x7f24ac006ed0 tx=0x7f24ac00b960).stop 2021-02-10T11:15:07.729 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:07.732 7f24caa11700 1 -- 172.21.2.102:0/2522637522 shutdown_connections 2021-02-10T11:15:07.729 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:07.732 7f24caa11700 1 --2- 172.21.2.102:0/2522637522 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f24a8018730 0x7f24a801abc0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:07.729 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:07.732 7f24caa11700 1 --2- 172.21.2.102:0/2522637522 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f24c40c1d80 0x7f24c40c2180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:15:07.729 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:07.732 7f24caa11700 1 --2- 172.21.2.102:0/2522637522 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f24c40b8200 0x7f24c40b8600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:15:07.729 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:07.732 7f24caa11700 1 --2- 172.21.2.102:0/2522637522 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f24c4190920 0x7f24c40b7be0 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:07.730 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:07.732 7f24caa11700 1 -- 172.21.2.102:0/2522637522 >> 172.21.2.102:0/2522637522 conn(0x7f24c4094e10 msgr2=0x7f24c40a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:15:07.730 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:07.732 7f24caa11700 1 -- 172.21.2.102:0/2522637522 shutdown_connections 2021-02-10T11:15:07.730 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:07.732 7f24caa11700 1 -- 172.21.2.102:0/2522637522 wait complete. 2021-02-10T11:15:07.735 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000344427 2021-02-10T11:15:08.169 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:08.158 7feb65f85700 1 Processor -- start 2021-02-10T11:15:08.169 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:08.158 7feb65f85700 1 -- start start 2021-02-10T11:15:08.169 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:08.158 7feb65f85700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7feb601b0470 0x7feb601b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:08.169 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:08.158 7feb65f85700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7feb600ef610 0x7feb600efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:08.170 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:08.158 7feb65f85700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb600f0b50 con 0x7feb600f0030 2021-02-10T11:15:08.170 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:08.158 7feb65f85700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb600e52e0 con 0x7feb601b0470 2021-02-10T11:15:08.170 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:08.158 7feb65f85700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb600e54d0 con 0x7feb600ef610 2021-02-10T11:15:08.170 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:08.160 7feb5ffff700 1 --1- >> v1:172.21.2.113:6789/0 conn(0x7feb600f0030 0x7feb600f10d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.113:6789/0 says I am v1:172.21.2.102:52252/0 (socket says 172.21.2.102:52252) 2021-02-10T11:15:08.170 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:08.160 7feb5f7fe700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7feb601b0470 0x7feb601b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:08.170 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:08.160 7feb5ffff700 1 -- 172.21.2.102:0/3628083007 learned_addr learned my addr 172.21.2.102:0/3628083007 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:15:08.170 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:08.160 7feb5effd700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7feb600ef610 0x7feb600efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:08.170 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:08.160 7feb5effd700 1 -- 172.21.2.102:0/3628083007 >> v2:172.21.2.113:3300/0 conn(0x7feb601b0470 msgr2=0x7feb601b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:15:08.170 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:08.160 7feb5effd700 1 --2- 172.21.2.102:0/3628083007 >> v2:172.21.2.113:3300/0 conn(0x7feb601b0470 0x7feb601b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:08.170 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:08.160 7feb5effd700 1 -- 172.21.2.102:0/3628083007 >> v1:172.21.2.113:6789/0 conn(0x7feb600f0030 legacy=0x7feb600f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:08.170 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:08.160 7feb5e7fc700 1 -- 172.21.2.102:0/3628083007 <== mon.1 v1:172.21.2.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2801519621 0 0) 0x7feb58001510 con 0x7feb600f0030 2021-02-10T11:15:08.170 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:08.160 7feb5effd700 1 -- 172.21.2.102:0/3628083007 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb600f0430 con 0x7feb600ef610 2021-02-10T11:15:08.171 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:08.160 7feb5effd700 1 --2- 172.21.2.102:0/3628083007 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7feb600ef610 0x7feb600efa10 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7feb54006650 tx=0x7feb54009a80).ready entity=mon.2 client_cookie=a91a125bb8f0db98 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:08.171 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:08.161 7feb5e7fc700 1 -- 172.21.2.102:0/3628083007 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7feb5400c0c0 con 0x7feb600ef610 2021-02-10T11:15:08.171 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:08.161 7feb5e7fc700 1 -- 172.21.2.102:0/3628083007 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb5400c340 con 0x7feb600ef610 2021-02-10T11:15:08.171 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:08.161 7feb5e7fc700 1 -- 172.21.2.102:0/3628083007 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7feb5400c860 con 0x7feb600ef610 2021-02-10T11:15:08.171 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:08.161 7feb65f85700 1 -- 172.21.2.102:0/3628083007 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7feb600ef610 msgr2=0x7feb600efa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:08.171 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:08.161 7feb65f85700 1 --2- 172.21.2.102:0/3628083007 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7feb600ef610 0x7feb600efa10 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7feb54006650 tx=0x7feb54009a80).stop 2021-02-10T11:15:08.171 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:08.161 7feb65f85700 1 -- 172.21.2.102:0/3628083007 shutdown_connections 2021-02-10T11:15:08.171 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:08.161 7feb65f85700 1 --2- 172.21.2.102:0/3628083007 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7feb600ef610 0x7feb600efa10 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:08.171 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:08.161 7feb65f85700 1 --2- 172.21.2.102:0/3628083007 >> v2:172.21.2.113:3300/0 conn(0x7feb601b0470 0x7feb601b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:08.171 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:08.161 7feb65f85700 1 -- 172.21.2.102:0/3628083007 >> 172.21.2.102:0/3628083007 conn(0x7feb600ce3e0 msgr2=0x7feb600e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:15:08.171 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:08.162 7feb65f85700 1 -- 172.21.2.102:0/3628083007 shutdown_connections 2021-02-10T11:15:08.171 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:08.162 7feb65f85700 1 -- 172.21.2.102:0/3628083007 wait complete. 2021-02-10T11:15:08.173 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:08.176 7feb65f85700 1 Processor -- start 2021-02-10T11:15:08.173 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:08.176 7feb65f85700 1 -- start start 2021-02-10T11:15:08.173 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:08.176 7feb65f85700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7feb600a0420 0x7feb600a0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:08.173 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:08.176 7feb65f85700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7feb60189d00 0x7feb6018a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:08.173 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:08.176 7feb65f85700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7feb6018a720 0x7feb600b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:08.174 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:08.176 7feb65f85700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb60189780 con 0x7feb6018a720 2021-02-10T11:15:08.174 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:08.176 7feb65f85700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb6018ab20 con 0x7feb600a0420 2021-02-10T11:15:08.174 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:08.176 7feb65f85700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb600a1140 con 0x7feb60189d00 2021-02-10T11:15:08.174 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:08.177 7feb5ffff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7feb6018a720 0x7feb600b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:08.174 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:08.177 7feb5ffff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7feb6018a720 0x7feb600b7be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:52712/0 (socket says 172.21.2.102:52712) 2021-02-10T11:15:08.174 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:08.177 7feb5ffff700 1 -- 172.21.2.102:0/1610435030 learned_addr learned my addr 172.21.2.102:0/1610435030 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:15:08.174 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:08.177 7feb5effd700 1 --2- 172.21.2.102:0/1610435030 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7feb60189d00 0x7feb6018a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:08.174 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:08.177 7feb5f7fe700 1 --2- 172.21.2.102:0/1610435030 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7feb600a0420 0x7feb600a0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:08.174 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:08.177 7feb5ffff700 1 -- 172.21.2.102:0/1610435030 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7feb60189d00 msgr2=0x7feb6018a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:08.174 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:08.177 7feb5ffff700 1 --2- 172.21.2.102:0/1610435030 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7feb60189d00 0x7feb6018a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:08.174 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:08.177 7feb5ffff700 1 -- 172.21.2.102:0/1610435030 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7feb600a0420 msgr2=0x7feb600a0820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:08.174 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:08.177 7feb5ffff700 1 --2- 172.21.2.102:0/1610435030 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7feb600a0420 0x7feb600a0820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:08.175 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:08.177 7feb5ffff700 1 -- 172.21.2.102:0/1610435030 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb600b8200 con 0x7feb6018a720 2021-02-10T11:15:08.175 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:08.177 7feb5ffff700 1 --2- 172.21.2.102:0/1610435030 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7feb6018a720 0x7feb600b7be0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 rx=0x7feb5800a900 tx=0x7feb5800aca0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:08.175 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:08.178 7feb47fff700 1 -- 172.21.2.102:0/1610435030 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7feb58000f40 con 0x7feb6018a720 2021-02-10T11:15:08.175 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:08.178 7feb47fff700 1 -- 172.21.2.102:0/1610435030 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb58007020 con 0x7feb6018a720 2021-02-10T11:15:08.175 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:08.178 7feb47fff700 1 -- 172.21.2.102:0/1610435030 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7feb580046a0 con 0x7feb6018a720 2021-02-10T11:15:08.175 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:08.178 7feb65f85700 1 -- 172.21.2.102:0/1610435030 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb600b8200 con 0x7feb6018a720 2021-02-10T11:15:08.175 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:08.178 7feb65f85700 1 -- 172.21.2.102:0/1610435030 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb600b8200 con 0x7feb6018a720 2021-02-10T11:15:08.175 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:08.178 7feb47fff700 1 -- 172.21.2.102:0/1610435030 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7feb58007200 con 0x7feb6018a720 2021-02-10T11:15:08.175 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:08.179 7feb47fff700 1 --2- 172.21.2.102:0/1610435030 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7feb40018730 0x7feb4001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:08.175 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:08.179 7feb47fff700 1 -- 172.21.2.102:0/1610435030 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7feb58032b50 con 0x7feb6018a720 2021-02-10T11:15:08.175 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:08.179 7feb5f7fe700 1 --2- 172.21.2.102:0/1610435030 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7feb40018730 0x7feb4001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:08.175 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:08.179 7feb5f7fe700 1 --2- 172.21.2.102:0/1610435030 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7feb40018730 0x7feb4001abc0 crc :-1 s=READY pgs=299 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:08.202 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:08.208 7feb65f85700 1 -- 172.21.2.102:0/1610435030 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb60105ae0 con 0x7feb6018a720 2021-02-10T11:15:08.204 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:08.211 7feb47fff700 1 -- 172.21.2.102:0/1610435030 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7feb58007c10 con 0x7feb6018a720 2021-02-10T11:15:08.333 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:08.340 7feb65f85700 1 -- 172.21.2.102:0/1610435030 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344427", "target": ["mgr", ""]}) v1 -- 0x7feb600f0b50 con 0x7feb40018730 2021-02-10T11:15:08.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:08.343 7feb47fff700 1 -- 172.21.2.102:0/1610435030 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7feb50006a70 con 0x7feb40018730 2021-02-10T11:15:08.341 INFO:teuthology.orchestra.run.gibba002.stdout:/volumes/_nogroup/subvolume_0000000000344427/184bf491-5fdf-47db-96c9-fe944be58e6d 2021-02-10T11:15:08.342 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:08.348 7feb65f85700 1 -- 172.21.2.102:0/1610435030 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7feb40018730 msgr2=0x7feb4001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:08.343 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:08.348 7feb65f85700 1 --2- 172.21.2.102:0/1610435030 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7feb40018730 0x7feb4001abc0 crc :-1 s=READY pgs=299 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:08.343 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:08.348 7feb65f85700 1 -- 172.21.2.102:0/1610435030 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7feb6018a720 msgr2=0x7feb600b7be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:08.343 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:08.348 7feb65f85700 1 --2- 172.21.2.102:0/1610435030 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7feb6018a720 0x7feb600b7be0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 rx=0x7feb5800a900 tx=0x7feb5800aca0).stop 2021-02-10T11:15:08.343 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:08.349 7feb65f85700 1 -- 172.21.2.102:0/1610435030 shutdown_connections 2021-02-10T11:15:08.343 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:08.349 7feb65f85700 1 --2- 172.21.2.102:0/1610435030 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7feb6018a720 0x7feb600b7be0 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:08.343 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:08.349 7feb65f85700 1 --2- 172.21.2.102:0/1610435030 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7feb60189d00 0x7feb6018a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:08.343 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:08.349 7feb65f85700 1 --2- 172.21.2.102:0/1610435030 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7feb40018730 0x7feb4001abc0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:08.343 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:08.349 7feb65f85700 1 --2- 172.21.2.102:0/1610435030 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7feb600a0420 0x7feb600a0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:08.343 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:08.349 7feb65f85700 1 -- 172.21.2.102:0/1610435030 >> 172.21.2.102:0/1610435030 conn(0x7feb60094e10 msgr2=0x7feb600a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:15:08.344 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:08.349 7feb65f85700 1 -- 172.21.2.102:0/1610435030 shutdown_connections 2021-02-10T11:15:08.344 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:08.349 7feb65f85700 1 -- 172.21.2.102:0/1610435030 wait complete. 2021-02-10T11:15:08.351 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000344427 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000344427/184bf491-5fdf-47db-96c9-fe944be58e6d 2021-02-10T11:15:08.351 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:15:08.352 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:15:08.352 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:15:08.690 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:15:08.691 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:15:08.691 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344427/184bf491-5fdf-47db-96c9-fe944be58e6d/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:15:09.070 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:15:09.070 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:15:09.070 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0224933 s, 46.6 MB/s 2021-02-10T11:15:09.071 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:15:09.071 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:15:09.071 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:15:09.407 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:15:09.408 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:15:09.408 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344427/184bf491-5fdf-47db-96c9-fe944be58e6d/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:15:09.784 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:15:09.784 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:15:09.784 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0239509 s, 43.8 MB/s 2021-02-10T11:15:09.785 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:15:09.786 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:15:09.786 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:15:10.121 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:15:10.122 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:15:10.122 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344427/184bf491-5fdf-47db-96c9-fe944be58e6d/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:15:10.500 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:15:10.500 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:15:10.500 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0252001 s, 41.6 MB/s 2021-02-10T11:15:10.501 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:15:10.501 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:15:10.501 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:15:10.837 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:15:10.838 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:15:10.838 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344427/184bf491-5fdf-47db-96c9-fe944be58e6d/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:15:11.215 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:15:11.215 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:15:11.215 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0247576 s, 42.4 MB/s 2021-02-10T11:15:11.216 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:15:11.216 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:15:11.216 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:15:11.551 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:15:11.552 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:15:11.552 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344427/184bf491-5fdf-47db-96c9-fe944be58e6d/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:15:11.928 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:15:11.928 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:15:11.928 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.024405 s, 43.0 MB/s 2021-02-10T11:15:11.929 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:15:11.929 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:15:11.929 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:15:12.264 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:15:12.265 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:15:12.265 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344427/184bf491-5fdf-47db-96c9-fe944be58e6d/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:15:12.639 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:15:12.639 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:15:12.639 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0227894 s, 46.0 MB/s 2021-02-10T11:15:12.640 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:15:12.640 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:15:12.641 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:15:12.976 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:15:12.976 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:15:12.976 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344427/184bf491-5fdf-47db-96c9-fe944be58e6d/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:15:13.350 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:15:13.351 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:15:13.351 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0233477 s, 44.9 MB/s 2021-02-10T11:15:13.352 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:15:13.352 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:15:13.352 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:15:13.688 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:15:13.688 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:15:13.689 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344427/184bf491-5fdf-47db-96c9-fe944be58e6d/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:15:14.063 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:15:14.063 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:15:14.063 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0228942 s, 45.8 MB/s 2021-02-10T11:15:14.064 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:15:14.064 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:15:14.064 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:15:14.400 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:15:14.401 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:15:14.401 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344427/184bf491-5fdf-47db-96c9-fe944be58e6d/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:15:14.779 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:15:14.779 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:15:14.779 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0251991 s, 41.6 MB/s 2021-02-10T11:15:14.780 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:15:14.780 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:15:14.780 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:15:15.115 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:15:15.116 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:15:15.116 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344427/184bf491-5fdf-47db-96c9-fe944be58e6d/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:15:15.491 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:15:15.491 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:15:15.491 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0234526 s, 44.7 MB/s 2021-02-10T11:15:15.492 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000344428 2021-02-10T11:15:15.925 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:15.914 7f266e2cc700 1 Processor -- start 2021-02-10T11:15:15.926 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:15.914 7f266e2cc700 1 -- start start 2021-02-10T11:15:15.926 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:15.914 7f266e2cc700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f26681b0470 0x7f26681b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:15.926 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:15.914 7f266e2cc700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26680f0b50 con 0x7f26680ef610 2021-02-10T11:15:15.926 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:15.914 7f266e2cc700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26680f24e0 con 0x7f26680f20e0 2021-02-10T11:15:15.926 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:15.914 7f266e2cc700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26680e5ea0 con 0x7f26681b0470 2021-02-10T11:15:15.926 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:15.914 7f26677fe700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f26680ef610 0x7f26680efa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:44944/0 (socket says 172.21.2.102:44944) 2021-02-10T11:15:15.926 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:15.914 7f26677fe700 1 -- 172.21.2.102:0/847713602 learned_addr learned my addr 172.21.2.102:0/847713602 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:15:15.926 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:15.915 7f2666ffd700 1 -- 172.21.2.102:0/847713602 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1933884141 0 0) 0x7f265c0009b0 con 0x7f26680ef610 2021-02-10T11:15:15.926 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:15.915 7f2666ffd700 1 -- 172.21.2.102:0/847713602 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2650002fe0 con 0x7f26680ef610 2021-02-10T11:15:15.926 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:15.915 7f2667fff700 1 --2- 172.21.2.102:0/847713602 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f26681b0470 0x7f26681b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:15.926 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:15.915 7f2666ffd700 1 -- 172.21.2.102:0/847713602 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1487822997 0 0) 0x7f265c001010 con 0x7f26680ef610 2021-02-10T11:15:15.927 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:15.915 7f2666ffd700 1 -- 172.21.2.102:0/847713602 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f26681b0470 msgr2=0x7f26681b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:15:15.927 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:15.915 7f2666ffd700 1 --2- 172.21.2.102:0/847713602 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f26681b0470 0x7f26681b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:15.927 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:15.915 7f2666ffd700 1 -- 172.21.2.102:0/847713602 >> v1:172.21.2.113:6789/0 conn(0x7f26680f20e0 legacy=0x7f26680f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:15.927 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:15.915 7f2666ffd700 1 -- 172.21.2.102:0/847713602 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26680e52e0 con 0x7f26680ef610 2021-02-10T11:15:15.927 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:15.915 7f2666ffd700 1 -- 172.21.2.102:0/847713602 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7f265c001590 con 0x7f26680ef610 2021-02-10T11:15:15.927 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:15.915 7f2666ffd700 1 -- 172.21.2.102:0/847713602 >> v1:172.21.2.102:6789/0 conn(0x7f26680ef610 legacy=0x7f26680efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:15.927 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:15.915 7f2666ffd700 1 --2- 172.21.2.102:0/847713602 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2650003ac0 0x7f2650005ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:15.927 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:15.915 7f2666ffd700 1 --2- 172.21.2.102:0/847713602 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2650006770 0x7f2650008c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:15.927 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:15.915 7f2666ffd700 1 --2- 172.21.2.102:0/847713602 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2650009510 0x7f265000ba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:15.927 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:15.915 7f2666ffd700 1 -- 172.21.2.102:0/847713602 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f265000c2e0 con 0x7f2650006770 2021-02-10T11:15:15.927 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:15.915 7f2666ffd700 1 -- 172.21.2.102:0/847713602 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f265000c500 con 0x7f2650009510 2021-02-10T11:15:15.927 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:15.915 7f2666ffd700 1 -- 172.21.2.102:0/847713602 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f265000c720 con 0x7f2650003ac0 2021-02-10T11:15:15.927 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:15.915 7f266c8e9700 1 --2- 172.21.2.102:0/847713602 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2650009510 0x7f265000ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:15.928 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:15.915 7f2667fff700 1 --2- 172.21.2.102:0/847713602 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2650003ac0 0x7f2650005ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:15.928 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:15.915 7f26677fe700 1 --2- 172.21.2.102:0/847713602 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2650006770 0x7f2650008c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:15.928 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:15.916 7f266c8e9700 1 -- 172.21.2.102:0/847713602 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2650003ac0 msgr2=0x7f2650005ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:15:15.928 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:15.916 7f266c8e9700 1 --2- 172.21.2.102:0/847713602 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2650003ac0 0x7f2650005ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:15.928 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:15.916 7f266c8e9700 1 -- 172.21.2.102:0/847713602 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2650006770 msgr2=0x7f2650008c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:15:15.928 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:15.916 7f266c8e9700 1 --2- 172.21.2.102:0/847713602 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2650006770 0x7f2650008c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:15.928 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:15.916 7f266c8e9700 1 -- 172.21.2.102:0/847713602 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2650050990 con 0x7f2650009510 2021-02-10T11:15:15.928 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:15.916 7f26677fe700 1 --2- 172.21.2.102:0/847713602 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2650006770 0x7f2650008c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:15:15.928 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:15.916 7f266c8e9700 1 --2- 172.21.2.102:0/847713602 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2650009510 0x7f265000ba30 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7f2660006e40 tx=0x7f2660009e30).ready entity=mon.1 client_cookie=3c14013c1a6b11ea server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:15.928 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:15.916 7f2666ffd700 1 -- 172.21.2.102:0/847713602 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f26600066e0 con 0x7f2650009510 2021-02-10T11:15:15.928 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:15.916 7f2666ffd700 1 -- 172.21.2.102:0/847713602 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f266000b510 con 0x7f2650009510 2021-02-10T11:15:15.928 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:15.916 7f2666ffd700 1 -- 172.21.2.102:0/847713602 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f266000ba30 con 0x7f2650009510 2021-02-10T11:15:15.928 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:15.916 7f266e2cc700 1 -- 172.21.2.102:0/847713602 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2650009510 msgr2=0x7f265000ba30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:15.929 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:15.916 7f266e2cc700 1 --2- 172.21.2.102:0/847713602 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2650009510 0x7f265000ba30 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7f2660006e40 tx=0x7f2660009e30).stop 2021-02-10T11:15:15.929 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:15.916 7f266c8e9700 1 -- 172.21.2.102:0/847713602 reap_dead start 2021-02-10T11:15:15.929 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:15.917 7f266e2cc700 1 -- 172.21.2.102:0/847713602 shutdown_connections 2021-02-10T11:15:15.929 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:15.917 7f266e2cc700 1 -- 172.21.2.102:0/847713602 >> 172.21.2.102:0/847713602 conn(0x7f26680ce3e0 msgr2=0x7f26680e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:15:15.929 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:15.917 7f266e2cc700 1 -- 172.21.2.102:0/847713602 shutdown_connections 2021-02-10T11:15:15.929 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:15.917 7f266e2cc700 1 -- 172.21.2.102:0/847713602 wait complete. 2021-02-10T11:15:15.929 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:15.932 7f266e2cc700 1 Processor -- start 2021-02-10T11:15:15.929 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:15.932 7f266e2cc700 1 -- start start 2021-02-10T11:15:15.931 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:15.932 7f266e2cc700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f266809b930 0x7f26680b7c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:15.931 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:15.932 7f266e2cc700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f26680b8410 0x7f26680b8810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:15.931 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:15.932 7f266e2cc700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f26680c1f80 0x7f26680c2380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:15.931 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:15.932 7f266e2cc700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26680c1a00 con 0x7f26680c1f80 2021-02-10T11:15:15.931 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:15.932 7f266e2cc700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26680b8e30 con 0x7f26680b8410 2021-02-10T11:15:15.931 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:15.932 7f266e2cc700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26680b9050 con 0x7f266809b930 2021-02-10T11:15:15.931 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:15.932 7f266c8e9700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f26680c1f80 0x7f26680c2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:15.931 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:15.933 7f266c8e9700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f26680c1f80 0x7f26680c2380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:52732/0 (socket says 172.21.2.102:52732) 2021-02-10T11:15:15.932 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:15.933 7f266c8e9700 1 -- 172.21.2.102:0/3559788524 learned_addr learned my addr 172.21.2.102:0/3559788524 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:15:15.932 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:15.933 7f266c8e9700 1 -- 172.21.2.102:0/3559788524 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f266809b930 msgr2=0x7f26680b7c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:15.932 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:15.933 7f266c8e9700 1 --2- 172.21.2.102:0/3559788524 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f266809b930 0x7f26680b7c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:15:15.932 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:15.933 7f26677fe700 1 --2- 172.21.2.102:0/3559788524 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f26680b8410 0x7f26680b8810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:15.932 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:15.933 7f266c8e9700 1 -- 172.21.2.102:0/3559788524 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f26680b8410 msgr2=0x7f26680b8810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:15.932 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:15.933 7f266c8e9700 1 --2- 172.21.2.102:0/3559788524 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f26680b8410 0x7f26680b8810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:15.932 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:15.933 7f266c8e9700 1 -- 172.21.2.102:0/3559788524 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26680c29a0 con 0x7f26680c1f80 2021-02-10T11:15:15.932 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:15.933 7f26677fe700 1 --2- 172.21.2.102:0/3559788524 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f26680b8410 0x7f26680b8810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:15:15.932 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:15.933 7f266c8e9700 1 --2- 172.21.2.102:0/3559788524 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f26680c1f80 0x7f26680c2380 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 rx=0x7f2660006760 tx=0x7f266000bc90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:15.932 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:15.933 7f2664ff9700 1 -- 172.21.2.102:0/3559788524 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2660002030 con 0x7f26680c1f80 2021-02-10T11:15:15.932 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:15.933 7f2664ff9700 1 -- 172.21.2.102:0/3559788524 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2660002370 con 0x7f26680c1f80 2021-02-10T11:15:15.932 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:15.933 7f2664ff9700 1 -- 172.21.2.102:0/3559788524 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f26600027d0 con 0x7f26680c1f80 2021-02-10T11:15:15.932 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:15.933 7f266e2cc700 1 -- 172.21.2.102:0/3559788524 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26680c29a0 con 0x7f26680c1f80 2021-02-10T11:15:15.932 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:15.933 7f266e2cc700 1 -- 172.21.2.102:0/3559788524 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26680c29a0 con 0x7f26680c1f80 2021-02-10T11:15:15.933 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:15.934 7f2664ff9700 1 -- 172.21.2.102:0/3559788524 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f2660002550 con 0x7f26680c1f80 2021-02-10T11:15:15.933 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:15.934 7f2664ff9700 1 --2- 172.21.2.102:0/3559788524 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2648018730 0x7f264801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:15.933 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:15.934 7f2664ff9700 1 -- 172.21.2.102:0/3559788524 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f26600329b0 con 0x7f26680c1f80 2021-02-10T11:15:15.933 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:15.934 7f2667fff700 1 --2- 172.21.2.102:0/3559788524 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2648018730 0x7f264801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:15.933 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:15.934 7f2667fff700 1 --2- 172.21.2.102:0/3559788524 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2648018730 0x7f264801ac10 crc :-1 s=READY pgs=300 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:15.957 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:15.964 7f266e2cc700 1 -- 172.21.2.102:0/3559788524 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2668105b30 con 0x7f26680c1f80 2021-02-10T11:15:15.960 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:15.966 7f2664ff9700 1 -- 172.21.2.102:0/3559788524 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f266000bdd0 con 0x7f26680c1f80 2021-02-10T11:15:16.087 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:16.093 7f266e2cc700 1 -- 172.21.2.102:0/3559788524 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344428", "target": ["mgr", ""]}) v1 -- 0x7f26680b8e30 con 0x7f2648018730 2021-02-10T11:15:16.115 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:16.122 7f2664ff9700 1 -- 172.21.2.102:0/3559788524 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f2658006e70 con 0x7f2648018730 2021-02-10T11:15:16.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:16.126 7f266e2cc700 1 -- 172.21.2.102:0/3559788524 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2648018730 msgr2=0x7f264801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:16.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:16.126 7f266e2cc700 1 --2- 172.21.2.102:0/3559788524 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2648018730 0x7f264801ac10 crc :-1 s=READY pgs=300 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:16.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:16.126 7f266e2cc700 1 -- 172.21.2.102:0/3559788524 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f26680c1f80 msgr2=0x7f26680c2380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:16.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:16.126 7f266e2cc700 1 --2- 172.21.2.102:0/3559788524 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f26680c1f80 0x7f26680c2380 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 rx=0x7f2660006760 tx=0x7f266000bc90).stop 2021-02-10T11:15:16.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:16.126 7f266e2cc700 1 -- 172.21.2.102:0/3559788524 shutdown_connections 2021-02-10T11:15:16.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:16.126 7f266e2cc700 1 --2- 172.21.2.102:0/3559788524 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f26680c1f80 0x7f26680c2380 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:16.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:16.126 7f266e2cc700 1 --2- 172.21.2.102:0/3559788524 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2648018730 0x7f264801ac10 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:16.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:16.126 7f266e2cc700 1 --2- 172.21.2.102:0/3559788524 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f26680b8410 0x7f26680b8810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:16.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:16.126 7f266e2cc700 1 --2- 172.21.2.102:0/3559788524 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f266809b930 0x7f26680b7c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:15:16.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:16.126 7f266e2cc700 1 -- 172.21.2.102:0/3559788524 >> 172.21.2.102:0/3559788524 conn(0x7f2668094e10 msgr2=0x7f26680a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:15:16.121 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:16.126 7f266e2cc700 1 -- 172.21.2.102:0/3559788524 shutdown_connections 2021-02-10T11:15:16.121 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:16.126 7f266e2cc700 1 -- 172.21.2.102:0/3559788524 wait complete. 2021-02-10T11:15:16.128 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000344428 2021-02-10T11:15:16.560 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:16.549 7fc2ce347700 1 Processor -- start 2021-02-10T11:15:16.560 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:16.549 7fc2ce347700 1 -- start start 2021-02-10T11:15:16.560 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:16.550 7fc2ce347700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fc2c81b0470 0x7fc2c81b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:16.560 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:16.550 7fc2ce347700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc2c80ef610 0x7fc2c80efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:16.560 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:16.550 7fc2ce347700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2c80f0b50 con 0x7fc2c80f0030 2021-02-10T11:15:16.560 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:16.550 7fc2ce347700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2c80e52e0 con 0x7fc2c81b0470 2021-02-10T11:15:16.560 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:16.550 7fc2ce347700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2c80e54d0 con 0x7fc2c80ef610 2021-02-10T11:15:16.560 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:16.550 7fc2cc964700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fc2c80f0030 0x7fc2c80f10d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:44964/0 (socket says 172.21.2.102:44964) 2021-02-10T11:15:16.560 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:16.550 7fc2cc964700 1 -- 172.21.2.102:0/3017069860 learned_addr learned my addr 172.21.2.102:0/3017069860 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:15:16.561 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:16.550 7fc2c7fff700 1 --2- 172.21.2.102:0/3017069860 >> v2:172.21.2.113:3300/0 conn(0x7fc2c81b0470 0x7fc2c81b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:16.561 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:16.550 7fc2c77fe700 1 --2- 172.21.2.102:0/3017069860 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc2c80ef610 0x7fc2c80efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:16.561 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:16.550 7fc2c6ffd700 1 -- 172.21.2.102:0/3017069860 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3891341738 0 0) 0x7fc2b8001180 con 0x7fc2c80f0030 2021-02-10T11:15:16.561 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:16.551 7fc2c6ffd700 1 -- 172.21.2.102:0/3017069860 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2b0002fe0 con 0x7fc2c80f0030 2021-02-10T11:15:16.561 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:16.551 7fc2c77fe700 1 -- 172.21.2.102:0/3017069860 >> v2:172.21.2.113:3300/0 conn(0x7fc2c81b0470 msgr2=0x7fc2c81b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:15:16.561 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:16.551 7fc2c77fe700 1 --2- 172.21.2.102:0/3017069860 >> v2:172.21.2.113:3300/0 conn(0x7fc2c81b0470 0x7fc2c81b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:16.561 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:16.551 7fc2c77fe700 1 -- 172.21.2.102:0/3017069860 >> v1:172.21.2.102:6789/0 conn(0x7fc2c80f0030 legacy=0x7fc2c80f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:16.561 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:16.551 7fc2c77fe700 1 -- 172.21.2.102:0/3017069860 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2c80f0430 con 0x7fc2c80ef610 2021-02-10T11:15:16.561 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:16.551 7fc2c7fff700 1 --2- 172.21.2.102:0/3017069860 >> v2:172.21.2.113:3300/0 conn(0x7fc2c81b0470 0x7fc2c81b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:15:16.561 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:16.551 7fc2c77fe700 1 --2- 172.21.2.102:0/3017069860 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc2c80ef610 0x7fc2c80efa10 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7fc2bc006650 tx=0x7fc2bc009a80).ready entity=mon.2 client_cookie=b59df4c253eb0f35 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:16.561 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:16.551 7fc2c6ffd700 1 -- 172.21.2.102:0/3017069860 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc2bc00c0c0 con 0x7fc2c80ef610 2021-02-10T11:15:16.561 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:16.551 7fc2c6ffd700 1 -- 172.21.2.102:0/3017069860 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2bc00c340 con 0x7fc2c80ef610 2021-02-10T11:15:16.561 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:16.551 7fc2c6ffd700 1 -- 172.21.2.102:0/3017069860 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc2bc00c860 con 0x7fc2c80ef610 2021-02-10T11:15:16.561 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:16.551 7fc2ce347700 1 -- 172.21.2.102:0/3017069860 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc2c80ef610 msgr2=0x7fc2c80efa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:16.562 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:16.551 7fc2ce347700 1 --2- 172.21.2.102:0/3017069860 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc2c80ef610 0x7fc2c80efa10 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7fc2bc006650 tx=0x7fc2bc009a80).stop 2021-02-10T11:15:16.562 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:16.552 7fc2ce347700 1 -- 172.21.2.102:0/3017069860 shutdown_connections 2021-02-10T11:15:16.562 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:16.552 7fc2ce347700 1 --2- 172.21.2.102:0/3017069860 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc2c80ef610 0x7fc2c80efa10 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:16.562 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:16.552 7fc2ce347700 1 --2- 172.21.2.102:0/3017069860 >> v2:172.21.2.113:3300/0 conn(0x7fc2c81b0470 0x7fc2c81b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:16.562 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:16.552 7fc2ce347700 1 -- 172.21.2.102:0/3017069860 >> 172.21.2.102:0/3017069860 conn(0x7fc2c80ce3e0 msgr2=0x7fc2c80e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:15:16.562 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:16.552 7fc2ce347700 1 -- 172.21.2.102:0/3017069860 shutdown_connections 2021-02-10T11:15:16.562 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:16.552 7fc2ce347700 1 -- 172.21.2.102:0/3017069860 wait complete. 2021-02-10T11:15:16.563 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:16.567 7fc2ce347700 1 Processor -- start 2021-02-10T11:15:16.564 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:16.567 7fc2ce347700 1 -- start start 2021-02-10T11:15:16.564 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:16.567 7fc2ce347700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc2c80a0280 0x7fc2c80aeff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:16.564 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:16.567 7fc2ce347700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc2c81a4450 0x7fc2c81a4850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:16.564 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:16.567 7fc2ce347700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc2c808bf50 0x7fc2c808c350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:16.564 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:16.567 7fc2ce347700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2c80a0700 con 0x7fc2c81a4450 2021-02-10T11:15:16.564 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:16.567 7fc2ce347700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2c80af610 con 0x7fc2c80a0280 2021-02-10T11:15:16.564 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:16.567 7fc2ce347700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2c808c970 con 0x7fc2c808bf50 2021-02-10T11:15:16.564 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:16.567 7fc2c77fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc2c81a4450 0x7fc2c81a4850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:16.564 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:16.567 7fc2c77fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc2c81a4450 0x7fc2c81a4850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:52746/0 (socket says 172.21.2.102:52746) 2021-02-10T11:15:16.564 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:16.567 7fc2c77fe700 1 -- 172.21.2.102:0/278275780 learned_addr learned my addr 172.21.2.102:0/278275780 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:15:16.564 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:16.568 7fc2cc964700 1 --2- 172.21.2.102:0/278275780 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc2c808bf50 0x7fc2c808c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:16.564 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:16.568 7fc2c7fff700 1 --2- 172.21.2.102:0/278275780 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc2c80a0280 0x7fc2c80aeff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:16.565 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:16.568 7fc2c77fe700 1 -- 172.21.2.102:0/278275780 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc2c808bf50 msgr2=0x7fc2c808c350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:16.565 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:16.568 7fc2c77fe700 1 --2- 172.21.2.102:0/278275780 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc2c808bf50 0x7fc2c808c350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:16.565 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:16.568 7fc2c77fe700 1 -- 172.21.2.102:0/278275780 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc2c80a0280 msgr2=0x7fc2c80aeff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:16.565 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:16.568 7fc2c77fe700 1 --2- 172.21.2.102:0/278275780 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc2c80a0280 0x7fc2c80aeff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:16.565 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:16.568 7fc2c77fe700 1 -- 172.21.2.102:0/278275780 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2c808cb90 con 0x7fc2c81a4450 2021-02-10T11:15:16.565 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:16.568 7fc2c77fe700 1 --2- 172.21.2.102:0/278275780 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc2c81a4450 0x7fc2c81a4850 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 rx=0x7fc2bc009650 tx=0x7fc2bc00b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:16.565 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:16.568 7fc2c4ff9700 1 -- 172.21.2.102:0/278275780 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc2bc00bd50 con 0x7fc2c81a4450 2021-02-10T11:15:16.565 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:16.568 7fc2c4ff9700 1 -- 172.21.2.102:0/278275780 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2bc006030 con 0x7fc2c81a4450 2021-02-10T11:15:16.565 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:16.568 7fc2c4ff9700 1 -- 172.21.2.102:0/278275780 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc2bc005560 con 0x7fc2c81a4450 2021-02-10T11:15:16.565 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:16.568 7fc2ce347700 1 -- 172.21.2.102:0/278275780 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2c808cb90 con 0x7fc2c81a4450 2021-02-10T11:15:16.565 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:16.568 7fc2ce347700 1 -- 172.21.2.102:0/278275780 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2c80b0270 con 0x7fc2c81a4450 2021-02-10T11:15:16.565 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:16.569 7fc2c4ff9700 1 -- 172.21.2.102:0/278275780 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fc2bc006030 con 0x7fc2c81a4450 2021-02-10T11:15:16.565 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:16.569 7fc2c4ff9700 1 --2- 172.21.2.102:0/278275780 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc2a80186c0 0x7fc2a801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:16.566 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:16.569 7fc2c4ff9700 1 -- 172.21.2.102:0/278275780 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fc2bc0323f0 con 0x7fc2c81a4450 2021-02-10T11:15:16.566 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:16.569 7fc2c7fff700 1 --2- 172.21.2.102:0/278275780 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc2a80186c0 0x7fc2a801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:16.566 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:16.569 7fc2c7fff700 1 --2- 172.21.2.102:0/278275780 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc2a80186c0 0x7fc2a801abc0 crc :-1 s=READY pgs=301 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:16.594 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:16.600 7fc2ce347700 1 -- 172.21.2.102:0/278275780 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2c8105ae0 con 0x7fc2c81a4450 2021-02-10T11:15:16.596 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:16.602 7fc2c4ff9700 1 -- 172.21.2.102:0/278275780 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fc2bc009360 con 0x7fc2c81a4450 2021-02-10T11:15:16.724 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:16.730 7fc2ce347700 1 -- 172.21.2.102:0/278275780 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344428", "target": ["mgr", ""]}) v1 -- 0x7fc2c80b0270 con 0x7fc2a80186c0 2021-02-10T11:15:16.728 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:16.734 7fc2c4ff9700 1 -- 172.21.2.102:0/278275780 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7fc2c00068e0 con 0x7fc2a80186c0 2021-02-10T11:15:16.731 INFO:teuthology.orchestra.run.gibba002.stdout:/volumes/_nogroup/subvolume_0000000000344428/1df867f0-2f02-4922-9545-dc17794c3bfc 2021-02-10T11:15:16.733 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:16.739 7fc2ce347700 1 -- 172.21.2.102:0/278275780 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc2a80186c0 msgr2=0x7fc2a801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:16.733 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:16.739 7fc2ce347700 1 --2- 172.21.2.102:0/278275780 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc2a80186c0 0x7fc2a801abc0 crc :-1 s=READY pgs=301 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:16.733 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:16.739 7fc2ce347700 1 -- 172.21.2.102:0/278275780 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc2c81a4450 msgr2=0x7fc2c81a4850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:16.733 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:16.739 7fc2ce347700 1 --2- 172.21.2.102:0/278275780 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc2c81a4450 0x7fc2c81a4850 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 rx=0x7fc2bc009650 tx=0x7fc2bc00b960).stop 2021-02-10T11:15:16.733 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:16.739 7fc2ce347700 1 -- 172.21.2.102:0/278275780 shutdown_connections 2021-02-10T11:15:16.733 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:16.739 7fc2ce347700 1 --2- 172.21.2.102:0/278275780 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc2c808bf50 0x7fc2c808c350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:16.733 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:16.739 7fc2ce347700 1 --2- 172.21.2.102:0/278275780 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc2c81a4450 0x7fc2c81a4850 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:16.734 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:16.739 7fc2ce347700 1 --2- 172.21.2.102:0/278275780 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc2a80186c0 0x7fc2a801abc0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:16.734 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:16.739 7fc2ce347700 1 --2- 172.21.2.102:0/278275780 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc2c80a0280 0x7fc2c80aeff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:16.734 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:16.739 7fc2ce347700 1 -- 172.21.2.102:0/278275780 >> 172.21.2.102:0/278275780 conn(0x7fc2c80bbf60 msgr2=0x7fc2c8105d10 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:15:16.734 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:16.740 7fc2ce347700 1 -- 172.21.2.102:0/278275780 shutdown_connections 2021-02-10T11:15:16.734 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:16.740 7fc2ce347700 1 -- 172.21.2.102:0/278275780 wait complete. 2021-02-10T11:15:16.741 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000344428 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000344428/1df867f0-2f02-4922-9545-dc17794c3bfc 2021-02-10T11:15:16.741 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:15:16.742 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:15:16.742 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:15:17.078 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:15:17.079 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:15:17.079 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344428/1df867f0-2f02-4922-9545-dc17794c3bfc/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:15:17.461 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:15:17.461 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:15:17.461 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.025162 s, 41.7 MB/s 2021-02-10T11:15:17.462 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:15:17.462 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:15:17.462 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:15:17.797 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:15:17.798 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:15:17.798 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344428/1df867f0-2f02-4922-9545-dc17794c3bfc/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:15:18.173 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:15:18.173 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:15:18.173 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0238275 s, 44.0 MB/s 2021-02-10T11:15:18.175 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:15:18.175 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:15:18.175 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:15:18.512 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:15:18.512 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:15:18.513 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344428/1df867f0-2f02-4922-9545-dc17794c3bfc/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:15:18.886 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:15:18.886 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:15:18.886 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0232087 s, 45.2 MB/s 2021-02-10T11:15:18.887 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:15:18.887 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:15:18.887 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:15:19.222 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:15:19.223 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:15:19.223 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344428/1df867f0-2f02-4922-9545-dc17794c3bfc/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:15:19.598 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:15:19.598 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:15:19.599 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.023003 s, 45.6 MB/s 2021-02-10T11:15:19.599 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:15:19.599 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:15:19.600 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:15:19.936 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:15:19.937 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:15:19.937 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344428/1df867f0-2f02-4922-9545-dc17794c3bfc/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:15:20.314 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:15:20.314 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:15:20.314 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0240817 s, 43.5 MB/s 2021-02-10T11:15:20.315 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:15:20.315 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:15:20.315 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:15:20.650 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:15:20.651 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:15:20.651 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344428/1df867f0-2f02-4922-9545-dc17794c3bfc/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:15:21.026 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:15:21.026 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:15:21.026 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0230345 s, 45.5 MB/s 2021-02-10T11:15:21.027 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:15:21.027 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:15:21.027 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:15:21.362 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:15:21.363 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:15:21.363 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344428/1df867f0-2f02-4922-9545-dc17794c3bfc/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:15:21.739 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:15:21.739 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:15:21.739 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0244816 s, 42.8 MB/s 2021-02-10T11:15:21.740 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:15:21.740 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:15:21.740 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:15:22.075 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:15:22.076 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:15:22.076 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344428/1df867f0-2f02-4922-9545-dc17794c3bfc/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:15:22.453 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:15:22.453 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:15:22.453 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0232419 s, 45.1 MB/s 2021-02-10T11:15:22.454 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:15:22.455 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:15:22.455 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:15:22.789 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:15:22.789 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:15:22.789 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344428/1df867f0-2f02-4922-9545-dc17794c3bfc/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:15:23.164 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:15:23.164 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:15:23.164 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0232579 s, 45.1 MB/s 2021-02-10T11:15:23.165 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:15:23.166 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:15:23.166 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:15:23.501 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:15:23.501 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:15:23.502 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344428/1df867f0-2f02-4922-9545-dc17794c3bfc/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:15:23.874 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:15:23.874 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:15:23.874 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0243567 s, 43.1 MB/s 2021-02-10T11:15:23.875 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000344429 2021-02-10T11:15:24.309 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:24.298 7f4903bba700 1 Processor -- start 2021-02-10T11:15:24.309 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:24.298 7f4903bba700 1 -- start start 2021-02-10T11:15:24.309 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:24.299 7f4903bba700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f48fc1b0870 0x7f48fc0f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:24.310 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:24.299 7f4903bba700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48fc1b0c70 con 0x7f48fc1b0470 2021-02-10T11:15:24.310 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:24.299 7f4903bba700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48fc0f0b50 con 0x7f48fc0efe00 2021-02-10T11:15:24.310 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:24.299 7f4903bba700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48fc0e5ea0 con 0x7f48fc1b0870 2021-02-10T11:15:24.310 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:24.299 7f49019d6700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f48fc1b0470 0x7f48fc0ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:44974/0 (socket says 172.21.2.102:44974) 2021-02-10T11:15:24.310 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:24.299 7f49019d6700 1 -- 172.21.2.102:0/612333271 learned_addr learned my addr 172.21.2.102:0/612333271 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:15:24.310 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:24.299 7f49011d5700 1 --2- 172.21.2.102:0/612333271 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f48fc1b0870 0x7f48fc0f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:24.310 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:24.299 7f49009d4700 1 -- 172.21.2.102:0/612333271 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1288425962 0 0) 0x7f48f80010b0 con 0x7f48fc1b0470 2021-02-10T11:15:24.310 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:24.299 7f49009d4700 1 -- 172.21.2.102:0/612333271 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48e8002fe0 con 0x7f48fc1b0470 2021-02-10T11:15:24.310 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:24.299 7f49009d4700 1 -- 172.21.2.102:0/612333271 <== mon.1 v1:172.21.2.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1668836221 0 0) 0x7f48f4000f90 con 0x7f48fc0efe00 2021-02-10T11:15:24.310 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:24.299 7f49009d4700 1 -- 172.21.2.102:0/612333271 --> v1:172.21.2.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48e8003210 con 0x7f48fc0efe00 2021-02-10T11:15:24.310 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:24.299 7f49009d4700 1 -- 172.21.2.102:0/612333271 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1670021604 0 0) 0x7f48f80010b0 con 0x7f48fc1b0470 2021-02-10T11:15:24.310 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:24.299 7f49009d4700 1 -- 172.21.2.102:0/612333271 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f48fc1b0870 msgr2=0x7f48fc0f20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:15:24.311 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:24.299 7f49009d4700 1 --2- 172.21.2.102:0/612333271 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f48fc1b0870 0x7f48fc0f20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:24.311 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:24.299 7f49009d4700 1 -- 172.21.2.102:0/612333271 >> v1:172.21.2.113:6789/0 conn(0x7f48fc0efe00 legacy=0x7f48fc0f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:24.311 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:24.299 7f49009d4700 1 -- 172.21.2.102:0/612333271 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48fc0e52e0 con 0x7f48fc1b0470 2021-02-10T11:15:24.311 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:24.300 7f49009d4700 1 -- 172.21.2.102:0/612333271 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7f48f80015b0 con 0x7f48fc1b0470 2021-02-10T11:15:24.311 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:24.300 7f49009d4700 1 -- 172.21.2.102:0/612333271 >> v1:172.21.2.102:6789/0 conn(0x7f48fc1b0470 legacy=0x7f48fc0ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:24.311 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:24.300 7f49009d4700 1 --2- 172.21.2.102:0/612333271 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f48e8003b90 0x7f48e8005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:24.311 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:24.300 7f49009d4700 1 --2- 172.21.2.102:0/612333271 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f48e8006770 0x7f48e8008c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:24.311 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:24.300 7f49009d4700 1 --2- 172.21.2.102:0/612333271 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f48e8009510 0x7f48e800ba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:24.311 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:24.300 7f49009d4700 1 -- 172.21.2.102:0/612333271 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48e80502f0 con 0x7f48e8009510 2021-02-10T11:15:24.311 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:24.300 7f49009d4700 1 -- 172.21.2.102:0/612333271 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48e80504d0 con 0x7f48e8006770 2021-02-10T11:15:24.311 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:24.300 7f49009d4700 1 -- 172.21.2.102:0/612333271 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48e80506f0 con 0x7f48e8003b90 2021-02-10T11:15:24.311 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:24.300 7f49021d7700 1 --2- 172.21.2.102:0/612333271 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f48e8009510 0x7f48e800ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:24.311 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:24.300 7f49019d6700 1 --2- 172.21.2.102:0/612333271 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f48e8003b90 0x7f48e8005fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:24.311 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:24.300 7f49011d5700 1 --2- 172.21.2.102:0/612333271 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f48e8006770 0x7f48e8008c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:24.312 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:24.300 7f49021d7700 1 -- 172.21.2.102:0/612333271 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f48e8003b90 msgr2=0x7f48e8005fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:15:24.312 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:24.300 7f49021d7700 1 --2- 172.21.2.102:0/612333271 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f48e8003b90 0x7f48e8005fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:24.312 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:24.300 7f49021d7700 1 -- 172.21.2.102:0/612333271 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f48e8006770 msgr2=0x7f48e8008c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:15:24.312 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:24.300 7f49021d7700 1 --2- 172.21.2.102:0/612333271 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f48e8006770 0x7f48e8008c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:24.312 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:24.300 7f49021d7700 1 -- 172.21.2.102:0/612333271 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48e8050990 con 0x7f48e8009510 2021-02-10T11:15:24.312 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:24.300 7f49021d7700 1 --2- 172.21.2.102:0/612333271 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f48e8009510 0x7f48e800ba30 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 rx=0x7f48f40013c0 tx=0x7f48f4009e20).ready entity=mon.0 client_cookie=9c3dfc0061e18574 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:24.312 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:24.300 7f49009d4700 1 -- 172.21.2.102:0/612333271 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f48f4001400 con 0x7f48e8009510 2021-02-10T11:15:24.312 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:24.300 7f49009d4700 1 -- 172.21.2.102:0/612333271 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48f4006a60 con 0x7f48e8009510 2021-02-10T11:15:24.312 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:24.300 7f49009d4700 1 -- 172.21.2.102:0/612333271 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f48f4001870 con 0x7f48e8009510 2021-02-10T11:15:24.312 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:24.300 7f4903bba700 1 -- 172.21.2.102:0/612333271 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f48e8009510 msgr2=0x7f48e800ba30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:24.312 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:24.300 7f4903bba700 1 --2- 172.21.2.102:0/612333271 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f48e8009510 0x7f48e800ba30 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 rx=0x7f48f40013c0 tx=0x7f48f4009e20).stop 2021-02-10T11:15:24.312 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:24.300 7f49021d7700 1 -- 172.21.2.102:0/612333271 reap_dead start 2021-02-10T11:15:24.313 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:24.301 7f4903bba700 1 -- 172.21.2.102:0/612333271 shutdown_connections 2021-02-10T11:15:24.313 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:24.301 7f4903bba700 1 -- 172.21.2.102:0/612333271 >> 172.21.2.102:0/612333271 conn(0x7f48fc0ce3e0 msgr2=0x7f48fc0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:15:24.313 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:24.301 7f4903bba700 1 -- 172.21.2.102:0/612333271 shutdown_connections 2021-02-10T11:15:24.313 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:24.301 7f4903bba700 1 -- 172.21.2.102:0/612333271 wait complete. 2021-02-10T11:15:24.313 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:24.316 7f4903bba700 1 Processor -- start 2021-02-10T11:15:24.313 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:24.316 7f4903bba700 1 -- start start 2021-02-10T11:15:24.314 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:24.316 7f4903bba700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f48fc0b7db0 0x7f48fc0b8260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:24.314 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:24.316 7f4903bba700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f48fc0b8880 0x7f48fc1a7450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:24.314 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:24.316 7f4903bba700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f48fc0b8c80 0x7f48fc1a7a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:24.314 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:24.316 7f4903bba700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48fc0b79b0 con 0x7f48fc0b8880 2021-02-10T11:15:24.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:24.316 7f4903bba700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48fc1a8090 con 0x7f48fc0b8c80 2021-02-10T11:15:24.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:24.316 7f4903bba700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48fc1a82b0 con 0x7f48fc0b7db0 2021-02-10T11:15:24.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:24.316 7f49011d5700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f48fc0b8880 0x7f48fc1a7450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:24.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:24.316 7f49011d5700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f48fc0b8880 0x7f48fc1a7450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:52762/0 (socket says 172.21.2.102:52762) 2021-02-10T11:15:24.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:24.316 7f49011d5700 1 -- 172.21.2.102:0/409339948 learned_addr learned my addr 172.21.2.102:0/409339948 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:15:24.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:24.316 7f49011d5700 1 -- 172.21.2.102:0/409339948 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f48fc0b7db0 msgr2=0x7f48fc0b8260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:24.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:24.316 7f49011d5700 1 --2- 172.21.2.102:0/409339948 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f48fc0b7db0 0x7f48fc0b8260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:15:24.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:24.317 7f49011d5700 1 -- 172.21.2.102:0/409339948 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f48fc0b8c80 msgr2=0x7f48fc1a7a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:24.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:24.317 7f49011d5700 1 --2- 172.21.2.102:0/409339948 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f48fc0b8c80 0x7f48fc1a7a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:15:24.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:24.317 7f49011d5700 1 -- 172.21.2.102:0/409339948 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48fc190c10 con 0x7f48fc0b8880 2021-02-10T11:15:24.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:24.317 7f49011d5700 1 --2- 172.21.2.102:0/409339948 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f48fc0b8880 0x7f48fc1a7450 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 rx=0x7f48f0006f60 tx=0x7f48f000b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:24.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:24.317 7f48ee7fc700 1 -- 172.21.2.102:0/409339948 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f48f000bd60 con 0x7f48fc0b8880 2021-02-10T11:15:24.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:24.317 7f48ee7fc700 1 -- 172.21.2.102:0/409339948 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48f0000c90 con 0x7f48fc0b8880 2021-02-10T11:15:24.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:24.317 7f48ee7fc700 1 -- 172.21.2.102:0/409339948 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f48f0002300 con 0x7f48fc0b8880 2021-02-10T11:15:24.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:24.317 7f4903bba700 1 -- 172.21.2.102:0/409339948 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48fc190c10 con 0x7f48fc0b8880 2021-02-10T11:15:24.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:24.317 7f4903bba700 1 -- 172.21.2.102:0/409339948 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48fc190c10 con 0x7f48fc0b8880 2021-02-10T11:15:24.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:24.317 7f48ee7fc700 1 -- 172.21.2.102:0/409339948 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f48f0002020 con 0x7f48fc0b8880 2021-02-10T11:15:24.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:24.317 7f48ee7fc700 1 --2- 172.21.2.102:0/409339948 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f48e40186c0 0x7f48e401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:24.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:24.317 7f48ee7fc700 1 -- 172.21.2.102:0/409339948 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f48f00323d0 con 0x7f48fc0b8880 2021-02-10T11:15:24.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:24.318 7f49019d6700 1 --2- 172.21.2.102:0/409339948 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f48e40186c0 0x7f48e401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:24.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:24.318 7f49019d6700 1 --2- 172.21.2.102:0/409339948 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f48e40186c0 0x7f48e401abc0 crc :-1 s=READY pgs=302 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:24.342 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:24.348 7f4903bba700 1 -- 172.21.2.102:0/409339948 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48fc105b30 con 0x7f48fc0b8880 2021-02-10T11:15:24.344 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:24.351 7f48ee7fc700 1 -- 172.21.2.102:0/409339948 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f48f0030d00 con 0x7f48fc0b8880 2021-02-10T11:15:24.470 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:24.476 7f4903bba700 1 -- 172.21.2.102:0/409339948 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344429", "target": ["mgr", ""]}) v1 -- 0x7f48fc1a8090 con 0x7f48e40186c0 2021-02-10T11:15:24.497 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:24.503 7f48ee7fc700 1 -- 172.21.2.102:0/409339948 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f48f8000e20 con 0x7f48e40186c0 2021-02-10T11:15:24.503 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:24.509 7f4903bba700 1 -- 172.21.2.102:0/409339948 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f48e40186c0 msgr2=0x7f48e401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:24.503 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:24.509 7f4903bba700 1 --2- 172.21.2.102:0/409339948 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f48e40186c0 0x7f48e401abc0 crc :-1 s=READY pgs=302 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:24.503 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:24.509 7f4903bba700 1 -- 172.21.2.102:0/409339948 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f48fc0b8880 msgr2=0x7f48fc1a7450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:24.503 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:24.509 7f4903bba700 1 --2- 172.21.2.102:0/409339948 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f48fc0b8880 0x7f48fc1a7450 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 rx=0x7f48f0006f60 tx=0x7f48f000b970).stop 2021-02-10T11:15:24.503 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:24.510 7f4903bba700 1 -- 172.21.2.102:0/409339948 shutdown_connections 2021-02-10T11:15:24.504 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:24.510 7f4903bba700 1 --2- 172.21.2.102:0/409339948 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f48e40186c0 0x7f48e401abc0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:24.504 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:24.510 7f4903bba700 1 --2- 172.21.2.102:0/409339948 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f48fc0b8c80 0x7f48fc1a7a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:15:24.504 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:24.510 7f4903bba700 1 --2- 172.21.2.102:0/409339948 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f48fc0b8880 0x7f48fc1a7450 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:24.504 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:24.510 7f4903bba700 1 --2- 172.21.2.102:0/409339948 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f48fc0b7db0 0x7f48fc0b8260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:15:24.504 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:24.510 7f4903bba700 1 -- 172.21.2.102:0/409339948 >> 172.21.2.102:0/409339948 conn(0x7f48fc094e10 msgr2=0x7f48fc0a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:15:24.504 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:24.510 7f4903bba700 1 -- 172.21.2.102:0/409339948 shutdown_connections 2021-02-10T11:15:24.504 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:24.510 7f4903bba700 1 -- 172.21.2.102:0/409339948 wait complete. 2021-02-10T11:15:24.511 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000344429 2021-02-10T11:15:24.943 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:24.932 7fe15e516700 1 Processor -- start 2021-02-10T11:15:24.943 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:24.932 7fe15e516700 1 -- start start 2021-02-10T11:15:24.943 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:24.932 7fe15e516700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fe1581b0470 0x7fe1581b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:24.943 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:24.932 7fe15e516700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe1580ef610 0x7fe1580efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:24.943 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:24.932 7fe15e516700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1580f0b50 con 0x7fe1580f0030 2021-02-10T11:15:24.943 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:24.932 7fe15e516700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1580e52e0 con 0x7fe1581b0470 2021-02-10T11:15:24.944 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:24.932 7fe15e516700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1580e54d0 con 0x7fe1580ef610 2021-02-10T11:15:24.944 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:24.933 7fe15cb33700 1 --1- >> v1:172.21.2.113:6789/0 conn(0x7fe1580f0030 0x7fe1580f10d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.113:6789/0 says I am v1:172.21.2.102:52314/0 (socket says 172.21.2.102:52314) 2021-02-10T11:15:24.944 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:24.933 7fe1577fe700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe1580ef610 0x7fe1580efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:24.944 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:24.933 7fe15cb33700 1 -- 172.21.2.102:0/2819777876 learned_addr learned my addr 172.21.2.102:0/2819777876 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:15:24.944 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:24.933 7fe157fff700 1 --2- 172.21.2.102:0/2819777876 >> v2:172.21.2.113:3300/0 conn(0x7fe1581b0470 0x7fe1581b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:24.944 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:24.933 7fe1577fe700 1 -- 172.21.2.102:0/2819777876 >> v2:172.21.2.113:3300/0 conn(0x7fe1581b0470 msgr2=0x7fe1581b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:15:24.944 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:24.933 7fe1577fe700 1 --2- 172.21.2.102:0/2819777876 >> v2:172.21.2.113:3300/0 conn(0x7fe1581b0470 0x7fe1581b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:24.944 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:24.934 7fe1577fe700 1 -- 172.21.2.102:0/2819777876 >> v1:172.21.2.113:6789/0 conn(0x7fe1580f0030 legacy=0x7fe1580f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:24.945 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:24.934 7fe1577fe700 1 -- 172.21.2.102:0/2819777876 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1580f0430 con 0x7fe1580ef610 2021-02-10T11:15:24.945 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:24.934 7fe157fff700 1 --2- 172.21.2.102:0/2819777876 >> v2:172.21.2.113:3300/0 conn(0x7fe1581b0470 0x7fe1581b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:15:24.945 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:24.934 7fe1577fe700 1 --2- 172.21.2.102:0/2819777876 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe1580ef610 0x7fe1580efa10 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7fe14c006650 tx=0x7fe14c009a80).ready entity=mon.2 client_cookie=723a424c54c60d07 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:24.945 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:24.934 7fe156ffd700 1 -- 172.21.2.102:0/2819777876 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe14c00c0c0 con 0x7fe1580ef610 2021-02-10T11:15:24.945 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:24.934 7fe156ffd700 1 -- 172.21.2.102:0/2819777876 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe14c00c340 con 0x7fe1580ef610 2021-02-10T11:15:24.945 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:24.934 7fe156ffd700 1 -- 172.21.2.102:0/2819777876 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe14c00c860 con 0x7fe1580ef610 2021-02-10T11:15:24.945 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:24.934 7fe15e516700 1 -- 172.21.2.102:0/2819777876 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe1580ef610 msgr2=0x7fe1580efa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:24.945 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:24.934 7fe15e516700 1 --2- 172.21.2.102:0/2819777876 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe1580ef610 0x7fe1580efa10 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7fe14c006650 tx=0x7fe14c009a80).stop 2021-02-10T11:15:24.945 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:24.935 7fe15e516700 1 -- 172.21.2.102:0/2819777876 shutdown_connections 2021-02-10T11:15:24.945 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:24.935 7fe15e516700 1 --2- 172.21.2.102:0/2819777876 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe1580ef610 0x7fe1580efa10 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:24.946 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:24.935 7fe15e516700 1 --2- 172.21.2.102:0/2819777876 >> v2:172.21.2.113:3300/0 conn(0x7fe1581b0470 0x7fe1581b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:24.946 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:24.935 7fe15e516700 1 -- 172.21.2.102:0/2819777876 >> 172.21.2.102:0/2819777876 conn(0x7fe1580ce3e0 msgr2=0x7fe1580e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:15:24.946 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:24.935 7fe15e516700 1 -- 172.21.2.102:0/2819777876 shutdown_connections 2021-02-10T11:15:24.946 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:24.935 7fe15e516700 1 -- 172.21.2.102:0/2819777876 wait complete. 2021-02-10T11:15:24.948 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:24.950 7fe15e516700 1 Processor -- start 2021-02-10T11:15:24.948 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:24.950 7fe15e516700 1 -- start start 2021-02-10T11:15:24.948 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:24.950 7fe15e516700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe158105d10 0x7fe15809eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:24.948 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:24.950 7fe15e516700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe158106110 0x7fe15809f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:24.948 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:24.950 7fe15e516700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe1580b2a00 0x7fe1580b2e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:24.948 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:24.950 7fe15e516700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe15809e560 con 0x7fe158106110 2021-02-10T11:15:24.948 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:24.950 7fe15e516700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe15809c090 con 0x7fe1580b2a00 2021-02-10T11:15:24.949 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:24.950 7fe15e516700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1580b3420 con 0x7fe158105d10 2021-02-10T11:15:24.949 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:24.950 7fe1577fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe158106110 0x7fe15809f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:24.949 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:24.950 7fe1577fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe158106110 0x7fe15809f100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:52772/0 (socket says 172.21.2.102:52772) 2021-02-10T11:15:24.949 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:24.950 7fe1577fe700 1 -- 172.21.2.102:0/1104177151 learned_addr learned my addr 172.21.2.102:0/1104177151 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:15:24.949 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:24.951 7fe1577fe700 1 -- 172.21.2.102:0/1104177151 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe158105d10 msgr2=0x7fe15809eae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:24.949 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:24.951 7fe1577fe700 1 --2- 172.21.2.102:0/1104177151 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe158105d10 0x7fe15809eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:15:24.949 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:24.951 7fe1577fe700 1 -- 172.21.2.102:0/1104177151 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe1580b2a00 msgr2=0x7fe1580b2e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:24.949 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:24.951 7fe1577fe700 1 --2- 172.21.2.102:0/1104177151 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe1580b2a00 0x7fe1580b2e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:15:24.950 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:24.951 7fe1577fe700 1 -- 172.21.2.102:0/1104177151 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1580b3640 con 0x7fe158106110 2021-02-10T11:15:24.950 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:24.951 7fe1577fe700 1 --2- 172.21.2.102:0/1104177151 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe158106110 0x7fe15809f100 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 rx=0x7fe14c009630 tx=0x7fe14c00b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:24.950 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:24.951 7fe154ff9700 1 -- 172.21.2.102:0/1104177151 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe14c0093a0 con 0x7fe158106110 2021-02-10T11:15:24.950 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:24.951 7fe154ff9700 1 -- 172.21.2.102:0/1104177151 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe14c00bd70 con 0x7fe158106110 2021-02-10T11:15:24.950 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:24.951 7fe15e516700 1 -- 172.21.2.102:0/1104177151 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1580b3640 con 0x7fe158106110 2021-02-10T11:15:24.950 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:24.951 7fe15e516700 1 -- 172.21.2.102:0/1104177151 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1580b3bd0 con 0x7fe158106110 2021-02-10T11:15:24.950 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:24.951 7fe154ff9700 1 -- 172.21.2.102:0/1104177151 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe14c0054d0 con 0x7fe158106110 2021-02-10T11:15:24.950 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:24.952 7fe154ff9700 1 -- 172.21.2.102:0/1104177151 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fe14c00a000 con 0x7fe158106110 2021-02-10T11:15:24.950 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:24.952 7fe154ff9700 1 --2- 172.21.2.102:0/1104177151 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe1380186c0 0x7fe13801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:24.951 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:24.952 7fe154ff9700 1 -- 172.21.2.102:0/1104177151 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fe14c032800 con 0x7fe158106110 2021-02-10T11:15:24.951 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:24.952 7fe157fff700 1 --2- 172.21.2.102:0/1104177151 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe1380186c0 0x7fe13801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:24.951 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:24.952 7fe157fff700 1 --2- 172.21.2.102:0/1104177151 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe1380186c0 0x7fe13801abc0 crc :-1 s=READY pgs=303 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:24.976 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:24.982 7fe15e516700 1 -- 172.21.2.102:0/1104177151 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe158105ae0 con 0x7fe158106110 2021-02-10T11:15:24.978 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:24.984 7fe154ff9700 1 -- 172.21.2.102:0/1104177151 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fe14c009340 con 0x7fe158106110 2021-02-10T11:15:25.107 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:25.113 7fe15e516700 1 -- 172.21.2.102:0/1104177151 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344429", "target": ["mgr", ""]}) v1 -- 0x7fe1581a16c0 con 0x7fe1380186c0 2021-02-10T11:15:25.110 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:25.117 7fe154ff9700 1 -- 172.21.2.102:0/1104177151 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7fe148000960 con 0x7fe1380186c0 2021-02-10T11:15:25.114 INFO:teuthology.orchestra.run.gibba002.stdout:/volumes/_nogroup/subvolume_0000000000344429/c511e701-0ee9-4f33-86f7-31a12839a1bd 2021-02-10T11:15:25.115 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:25.122 7fe15e516700 1 -- 172.21.2.102:0/1104177151 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe1380186c0 msgr2=0x7fe13801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:25.115 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:25.122 7fe15e516700 1 --2- 172.21.2.102:0/1104177151 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe1380186c0 0x7fe13801abc0 crc :-1 s=READY pgs=303 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:25.115 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:25.122 7fe15e516700 1 -- 172.21.2.102:0/1104177151 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe158106110 msgr2=0x7fe15809f100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:25.116 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:25.122 7fe15e516700 1 --2- 172.21.2.102:0/1104177151 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe158106110 0x7fe15809f100 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 rx=0x7fe14c009630 tx=0x7fe14c00b980).stop 2021-02-10T11:15:25.116 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:25.122 7fe15e516700 1 -- 172.21.2.102:0/1104177151 shutdown_connections 2021-02-10T11:15:25.116 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:25.122 7fe15e516700 1 --2- 172.21.2.102:0/1104177151 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe1380186c0 0x7fe13801abc0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:25.116 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:25.122 7fe15e516700 1 --2- 172.21.2.102:0/1104177151 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe1580b2a00 0x7fe1580b2e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:15:25.116 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:25.122 7fe15e516700 1 --2- 172.21.2.102:0/1104177151 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe158106110 0x7fe15809f100 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:25.116 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:25.122 7fe15e516700 1 --2- 172.21.2.102:0/1104177151 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe158105d10 0x7fe15809eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:15:25.116 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:25.122 7fe15e516700 1 -- 172.21.2.102:0/1104177151 >> 172.21.2.102:0/1104177151 conn(0x7fe1580c36f0 msgr2=0x7fe1580bd640 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:15:25.117 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:25.122 7fe15e516700 1 -- 172.21.2.102:0/1104177151 shutdown_connections 2021-02-10T11:15:25.117 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:25.122 7fe15e516700 1 -- 172.21.2.102:0/1104177151 wait complete. 2021-02-10T11:15:25.123 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000344429 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000344429/c511e701-0ee9-4f33-86f7-31a12839a1bd 2021-02-10T11:15:25.124 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:15:25.124 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:15:25.124 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:15:25.460 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:15:25.461 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:15:25.461 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344429/c511e701-0ee9-4f33-86f7-31a12839a1bd/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:15:25.842 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:15:25.842 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:15:25.843 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0246597 s, 42.5 MB/s 2021-02-10T11:15:25.843 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:15:25.844 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:15:25.844 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:15:26.180 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:15:26.181 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:15:26.181 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344429/c511e701-0ee9-4f33-86f7-31a12839a1bd/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:15:26.556 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:15:26.556 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:15:26.556 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0233145 s, 45.0 MB/s 2021-02-10T11:15:26.557 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:15:26.557 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:15:26.557 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:15:26.878 DEBUG:teuthology.orchestra.run.gibba002:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:15:26.881 DEBUG:teuthology.orchestra.run.gibba013:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:15:26.892 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:15:26.892 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:15:26.893 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344429/c511e701-0ee9-4f33-86f7-31a12839a1bd/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:15:27.256 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:15:27.257 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:15:27.257 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0246978 s, 42.5 MB/s 2021-02-10T11:15:27.258 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:15:27.258 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:15:27.258 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:15:27.593 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:15:27.594 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:15:27.594 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344429/c511e701-0ee9-4f33-86f7-31a12839a1bd/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:15:27.968 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:15:27.968 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:15:27.969 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0239981 s, 43.7 MB/s 2021-02-10T11:15:27.969 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:15:27.970 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:15:27.970 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:15:28.305 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:15:28.305 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:15:28.306 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344429/c511e701-0ee9-4f33-86f7-31a12839a1bd/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:15:28.680 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:15:28.680 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:15:28.680 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0242915 s, 43.2 MB/s 2021-02-10T11:15:28.681 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:15:28.681 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:15:28.681 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:15:29.016 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:15:29.017 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:15:29.017 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344429/c511e701-0ee9-4f33-86f7-31a12839a1bd/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:15:29.392 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:15:29.392 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:15:29.392 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0230901 s, 45.4 MB/s 2021-02-10T11:15:29.393 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:15:29.393 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:15:29.393 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:15:29.731 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:15:29.731 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:15:29.732 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344429/c511e701-0ee9-4f33-86f7-31a12839a1bd/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:15:30.108 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:15:30.108 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:15:30.108 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0235739 s, 44.5 MB/s 2021-02-10T11:15:30.109 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:15:30.109 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:15:30.109 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:15:30.448 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:15:30.449 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:15:30.449 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344429/c511e701-0ee9-4f33-86f7-31a12839a1bd/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:15:30.828 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:15:30.829 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:15:30.829 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0246604 s, 42.5 MB/s 2021-02-10T11:15:30.829 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:15:30.830 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:15:30.830 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:15:31.166 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:15:31.166 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:15:31.167 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344429/c511e701-0ee9-4f33-86f7-31a12839a1bd/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:15:31.541 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:15:31.542 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:15:31.542 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0234954 s, 44.6 MB/s 2021-02-10T11:15:31.543 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:15:31.543 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:15:31.543 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:15:31.878 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:15:31.879 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:15:31.879 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344429/c511e701-0ee9-4f33-86f7-31a12839a1bd/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:15:32.256 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:15:32.256 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:15:32.256 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0249801 s, 42.0 MB/s 2021-02-10T11:15:32.257 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000344430 2021-02-10T11:15:32.689 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:32.679 7fef0c54f700 1 Processor -- start 2021-02-10T11:15:32.689 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:32.679 7fef0c54f700 1 -- start start 2021-02-10T11:15:32.689 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:32.679 7fef0c54f700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fef041b0470 0x7fef041b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:32.689 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:32.679 7fef0c54f700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fef040ef610 0x7fef040efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:32.689 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:32.679 7fef0c54f700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef040f0b50 con 0x7fef040f0030 2021-02-10T11:15:32.689 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:32.679 7fef0c54f700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef040e52e0 con 0x7fef040ef610 2021-02-10T11:15:32.689 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:32.679 7fef0c54f700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef040e5490 con 0x7fef041b0470 2021-02-10T11:15:32.689 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:32.680 7fef09b6a700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fef040ef610 0x7fef040efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:32.689 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:32.680 7fef09b6a700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fef040ef610 0x7fef040efa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:52778/0 (socket says 172.21.2.102:52778) 2021-02-10T11:15:32.689 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:32.680 7fef09b6a700 1 -- 172.21.2.102:0/3171699292 learned_addr learned my addr 172.21.2.102:0/3171699292 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:15:32.690 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:32.680 7fef09b6a700 1 -- 172.21.2.102:0/3171699292 >> v2:172.21.2.113:3300/0 conn(0x7fef041b0470 msgr2=0x7fef041b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:15:32.690 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:32.680 7fef09b6a700 1 --2- 172.21.2.102:0/3171699292 >> v2:172.21.2.113:3300/0 conn(0x7fef041b0470 0x7fef041b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:15:32.690 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:32.680 7fef09b6a700 1 -- 172.21.2.102:0/3171699292 >> v1:172.21.2.113:6789/0 conn(0x7fef040f0030 legacy=0x7fef040f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:15:32.690 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:32.680 7fef09b6a700 1 -- 172.21.2.102:0/3171699292 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef040f0430 con 0x7fef040ef610 2021-02-10T11:15:32.690 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:32.680 7fef09b6a700 1 --2- 172.21.2.102:0/3171699292 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fef040ef610 0x7fef040efa10 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 rx=0x7feef4006700 tx=0x7feef4009b80).ready entity=mon.0 client_cookie=f653126fb676732e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:32.690 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:32.680 7fef09369700 1 -- 172.21.2.102:0/3171699292 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7feef400c0c0 con 0x7fef040ef610 2021-02-10T11:15:32.690 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:32.680 7fef09369700 1 -- 172.21.2.102:0/3171699292 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feef400c340 con 0x7fef040ef610 2021-02-10T11:15:32.690 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:32.680 7fef09369700 1 -- 172.21.2.102:0/3171699292 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7feef400c860 con 0x7fef040ef610 2021-02-10T11:15:32.690 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:32.680 7fef0c54f700 1 -- 172.21.2.102:0/3171699292 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fef040ef610 msgr2=0x7fef040efa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:32.690 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:32.680 7fef0c54f700 1 --2- 172.21.2.102:0/3171699292 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fef040ef610 0x7fef040efa10 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 rx=0x7feef4006700 tx=0x7feef4009b80).stop 2021-02-10T11:15:32.690 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:32.681 7fef0c54f700 1 -- 172.21.2.102:0/3171699292 shutdown_connections 2021-02-10T11:15:32.690 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:32.681 7fef0c54f700 1 --2- 172.21.2.102:0/3171699292 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fef040ef610 0x7fef040efa10 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:32.691 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:32.681 7fef0c54f700 1 --2- 172.21.2.102:0/3171699292 >> v2:172.21.2.113:3300/0 conn(0x7fef041b0470 0x7fef041b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:15:32.691 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:32.681 7fef0c54f700 1 -- 172.21.2.102:0/3171699292 >> 172.21.2.102:0/3171699292 conn(0x7fef040ce3e0 msgr2=0x7fef040e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:15:32.691 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:32.681 7fef0c54f700 1 -- 172.21.2.102:0/3171699292 shutdown_connections 2021-02-10T11:15:32.691 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:32.681 7fef0c54f700 1 -- 172.21.2.102:0/3171699292 wait complete. 2021-02-10T11:15:32.691 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:32.696 7fef0c54f700 1 Processor -- start 2021-02-10T11:15:32.691 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:32.696 7fef0c54f700 1 -- start start 2021-02-10T11:15:32.692 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:32.696 7fef0c54f700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fef04094fb0 0x7fef0409e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:32.692 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:32.696 7fef0c54f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fef040c2be0 0x7fef040c2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:32.692 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:32.696 7fef0c54f700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fef040c3600 0x7fef040a9c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:32.692 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:32.696 7fef0c54f700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef0409ebf0 con 0x7fef040c2be0 2021-02-10T11:15:32.692 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:32.696 7fef0c54f700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef040aa2a0 con 0x7fef04094fb0 2021-02-10T11:15:32.692 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:32.696 7fef0c54f700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef040aa4c0 con 0x7fef040c3600 2021-02-10T11:15:32.693 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:32.696 7fef09b6a700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fef040c2be0 0x7fef040c2fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:32.693 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:32.696 7fef09b6a700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fef040c2be0 0x7fef040c2fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:52784/0 (socket says 172.21.2.102:52784) 2021-02-10T11:15:32.693 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:32.696 7fef09b6a700 1 -- 172.21.2.102:0/2894235910 learned_addr learned my addr 172.21.2.102:0/2894235910 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:15:32.693 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:32.697 7fef0a36b700 1 --2- 172.21.2.102:0/2894235910 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fef04094fb0 0x7fef0409e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:32.694 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:32.697 7fef0ab6c700 1 --2- 172.21.2.102:0/2894235910 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fef040c3600 0x7fef040a9c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:32.694 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:32.697 7fef09b6a700 1 -- 172.21.2.102:0/2894235910 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fef040c3600 msgr2=0x7fef040a9c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:32.694 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:32.697 7fef09b6a700 1 --2- 172.21.2.102:0/2894235910 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fef040c3600 0x7fef040a9c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:32.694 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:32.697 7fef09b6a700 1 -- 172.21.2.102:0/2894235910 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fef04094fb0 msgr2=0x7fef0409e5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:32.694 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:32.697 7fef09b6a700 1 --2- 172.21.2.102:0/2894235910 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fef04094fb0 0x7fef0409e5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:32.694 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:32.697 7fef09b6a700 1 -- 172.21.2.102:0/2894235910 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef040aa6e0 con 0x7fef040c2be0 2021-02-10T11:15:32.694 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:32.697 7fef09b6a700 1 --2- 172.21.2.102:0/2894235910 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fef040c2be0 0x7fef040c2fe0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 rx=0x7feef400b650 tx=0x7feef400b9f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:32.694 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:32.697 7fef0a36b700 1 --2- 172.21.2.102:0/2894235910 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fef04094fb0 0x7fef0409e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:15:32.694 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:32.697 7feefaffd700 1 -- 172.21.2.102:0/2894235910 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7feef400bde0 con 0x7fef040c2be0 2021-02-10T11:15:32.694 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:32.697 7feefaffd700 1 -- 172.21.2.102:0/2894235910 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feef4006500 con 0x7fef040c2be0 2021-02-10T11:15:32.694 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:32.697 7feefaffd700 1 -- 172.21.2.102:0/2894235910 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7feef4002220 con 0x7fef040c2be0 2021-02-10T11:15:32.694 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:32.697 7fef0c54f700 1 -- 172.21.2.102:0/2894235910 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef040aa6e0 con 0x7fef040c2be0 2021-02-10T11:15:32.694 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:32.697 7fef0c54f700 1 -- 172.21.2.102:0/2894235910 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef040aa6e0 con 0x7fef040c2be0 2021-02-10T11:15:32.695 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:32.698 7feefaffd700 1 -- 172.21.2.102:0/2894235910 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7feef4000c90 con 0x7fef040c2be0 2021-02-10T11:15:32.695 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:32.698 7feefaffd700 1 --2- 172.21.2.102:0/2894235910 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7feeec0186c0 0x7feeec01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:32.695 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:32.698 7feefaffd700 1 -- 172.21.2.102:0/2894235910 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7feef4032610 con 0x7fef040c2be0 2021-02-10T11:15:32.695 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:32.698 7fef0a36b700 1 --2- 172.21.2.102:0/2894235910 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7feeec0186c0 0x7feeec01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:32.695 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:32.698 7fef0a36b700 1 --2- 172.21.2.102:0/2894235910 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7feeec0186c0 0x7feeec01abc0 crc :-1 s=READY pgs=304 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:32.723 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:32.729 7fef0c54f700 1 -- 172.21.2.102:0/2894235910 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef040edd10 con 0x7fef040c2be0 2021-02-10T11:15:32.725 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:32.732 7feefaffd700 1 -- 172.21.2.102:0/2894235910 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7feef4030dc0 con 0x7fef040c2be0 2021-02-10T11:15:32.852 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:32.858 7fef0c54f700 1 -- 172.21.2.102:0/2894235910 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344430", "target": ["mgr", ""]}) v1 -- 0x7fef0409ebf0 con 0x7feeec0186c0 2021-02-10T11:15:32.878 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:32.884 7feefaffd700 1 -- 172.21.2.102:0/2894235910 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fef000009b0 con 0x7feeec0186c0 2021-02-10T11:15:32.885 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:32.891 7fef0c54f700 1 -- 172.21.2.102:0/2894235910 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7feeec0186c0 msgr2=0x7feeec01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:32.885 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:32.891 7fef0c54f700 1 --2- 172.21.2.102:0/2894235910 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7feeec0186c0 0x7feeec01abc0 crc :-1 s=READY pgs=304 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:32.885 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:32.891 7fef0c54f700 1 -- 172.21.2.102:0/2894235910 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fef040c2be0 msgr2=0x7fef040c2fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:32.885 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:32.891 7fef0c54f700 1 --2- 172.21.2.102:0/2894235910 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fef040c2be0 0x7fef040c2fe0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 rx=0x7feef400b650 tx=0x7feef400b9f0).stop 2021-02-10T11:15:32.886 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:32.892 7fef0c54f700 1 -- 172.21.2.102:0/2894235910 shutdown_connections 2021-02-10T11:15:32.886 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:32.892 7fef0c54f700 1 --2- 172.21.2.102:0/2894235910 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fef040c3600 0x7fef040a9c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:32.886 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:32.892 7fef0c54f700 1 --2- 172.21.2.102:0/2894235910 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fef040c2be0 0x7fef040c2fe0 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:32.886 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:32.892 7fef0c54f700 1 --2- 172.21.2.102:0/2894235910 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7feeec0186c0 0x7feeec01abc0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:32.886 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:32.892 7fef0c54f700 1 --2- 172.21.2.102:0/2894235910 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fef04094fb0 0x7fef0409e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:32.886 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:32.892 7fef0c54f700 1 -- 172.21.2.102:0/2894235910 >> 172.21.2.102:0/2894235910 conn(0x7fef0409f630 msgr2=0x7fef040a0130 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:15:32.886 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:32.892 7fef0c54f700 1 -- 172.21.2.102:0/2894235910 shutdown_connections 2021-02-10T11:15:32.886 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:32.892 7fef0c54f700 1 -- 172.21.2.102:0/2894235910 wait complete. 2021-02-10T11:15:32.893 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000344430 2021-02-10T11:15:33.327 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:33.315 7f0236fcf700 1 Processor -- start 2021-02-10T11:15:33.327 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:33.315 7f0236fcf700 1 -- start start 2021-02-10T11:15:33.327 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:33.316 7f0236fcf700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f02301b0470 0x7f02301b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:33.327 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:33.316 7f0236fcf700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f02300ef610 0x7f02300efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:33.327 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:33.316 7f0236fcf700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02300f0b50 con 0x7f02300f0030 2021-02-10T11:15:33.328 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:33.316 7f0236fcf700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02300e52e0 con 0x7f02300ef610 2021-02-10T11:15:33.328 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:33.316 7f0236fcf700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02300e5490 con 0x7f02301b0470 2021-02-10T11:15:33.328 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:33.316 7f022ffff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f02300ef610 0x7f02300efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:33.328 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:33.316 7f022ffff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f02300ef610 0x7f02300efa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:52792/0 (socket says 172.21.2.102:52792) 2021-02-10T11:15:33.328 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:33.316 7f022ffff700 1 -- 172.21.2.102:0/3646217087 learned_addr learned my addr 172.21.2.102:0/3646217087 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:15:33.328 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:33.316 7f022ffff700 1 -- 172.21.2.102:0/3646217087 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f02301b0470 msgr2=0x7f02301b0870 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:15:33.328 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:33.316 7f022ffff700 1 --2- 172.21.2.102:0/3646217087 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f02301b0470 0x7f02301b0870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:15:33.328 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:33.316 7f022ffff700 1 -- 172.21.2.102:0/3646217087 >> v1:172.21.2.113:6789/0 conn(0x7f02300f0030 legacy=0x7f02300f10d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:15:33.329 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:33.316 7f022ffff700 1 -- 172.21.2.102:0/3646217087 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02300f0430 con 0x7f02300ef610 2021-02-10T11:15:33.329 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:33.316 7f022ffff700 1 --2- 172.21.2.102:0/3646217087 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f02300ef610 0x7f02300efa10 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 rx=0x7f0220006700 tx=0x7f0220009b80).ready entity=mon.0 client_cookie=3868ec627f4b7922 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:33.329 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:33.316 7f022f7fe700 1 -- 172.21.2.102:0/3646217087 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f02200009b0 con 0x7f02300ef610 2021-02-10T11:15:33.329 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:33.316 7f022f7fe700 1 -- 172.21.2.102:0/3646217087 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f022000c0c0 con 0x7f02300ef610 2021-02-10T11:15:33.329 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:33.316 7f022f7fe700 1 -- 172.21.2.102:0/3646217087 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f022000c530 con 0x7f02300ef610 2021-02-10T11:15:33.329 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:33.316 7f0236fcf700 1 -- 172.21.2.102:0/3646217087 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f02300ef610 msgr2=0x7f02300efa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:33.329 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:33.316 7f0236fcf700 1 --2- 172.21.2.102:0/3646217087 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f02300ef610 0x7f02300efa10 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 rx=0x7f0220006700 tx=0x7f0220009b80).stop 2021-02-10T11:15:33.329 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:33.317 7f0236fcf700 1 -- 172.21.2.102:0/3646217087 shutdown_connections 2021-02-10T11:15:33.329 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:33.317 7f0236fcf700 1 --2- 172.21.2.102:0/3646217087 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f02300ef610 0x7f02300efa10 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:33.330 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:33.317 7f0236fcf700 1 --2- 172.21.2.102:0/3646217087 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f02301b0470 0x7f02301b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:15:33.330 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:33.317 7f0236fcf700 1 -- 172.21.2.102:0/3646217087 >> 172.21.2.102:0/3646217087 conn(0x7f02300ce3e0 msgr2=0x7f02300e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:15:33.330 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:33.317 7f0236fcf700 1 -- 172.21.2.102:0/3646217087 shutdown_connections 2021-02-10T11:15:33.330 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:33.317 7f0236fcf700 1 -- 172.21.2.102:0/3646217087 wait complete. 2021-02-10T11:15:33.332 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:33.334 7f0236fcf700 1 Processor -- start 2021-02-10T11:15:33.332 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:33.334 7f0236fcf700 1 -- start start 2021-02-10T11:15:33.332 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:33.335 7f0236fcf700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f02300acc50 0x7f0230098de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:33.332 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:33.335 7f0236fcf700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f02300eec20 0x7f02300ef020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:33.332 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:33.335 7f0236fcf700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f02300ef640 0x7f02301a0c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:33.332 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:33.335 7f0236fcf700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0230090cf0 con 0x7f02300ef640 2021-02-10T11:15:33.332 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:33.335 7f0236fcf700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02300ad050 con 0x7f02300acc50 2021-02-10T11:15:33.332 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:33.335 7f0236fcf700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02301a1250 con 0x7f02300eec20 2021-02-10T11:15:33.333 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:33.335 7f02355ec700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f02300ef640 0x7f02301a0c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:33.333 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:33.335 7f022ffff700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f02300eec20 0x7f02300ef020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:33.333 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:33.335 7f022ffff700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f02300eec20 0x7f02300ef020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.113:3301/0 says I am v2:172.21.2.102:58332/0 (socket says 172.21.2.102:58332) 2021-02-10T11:15:33.333 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:33.335 7f022ffff700 1 -- 172.21.2.102:0/557523864 learned_addr learned my addr 172.21.2.102:0/557523864 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:15:33.333 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:33.335 7f02355ec700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f02300ef640 0x7f02301a0c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:52798/0 (socket says 172.21.2.102:52798) 2021-02-10T11:15:33.333 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:33.335 7f0234deb700 1 --2- 172.21.2.102:0/557523864 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f02300acc50 0x7f0230098de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:33.333 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:33.335 7f022ffff700 1 -- 172.21.2.102:0/557523864 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f02300acc50 msgr2=0x7f0230098de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:33.333 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:33.335 7f022ffff700 1 --2- 172.21.2.102:0/557523864 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f02300acc50 0x7f0230098de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:33.333 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:33.335 7f022ffff700 1 -- 172.21.2.102:0/557523864 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f02300ef640 msgr2=0x7f02301a0c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:33.333 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:33.335 7f022ffff700 1 --2- 172.21.2.102:0/557523864 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f02300ef640 0x7f02301a0c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:33.333 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:33.335 7f022ffff700 1 -- 172.21.2.102:0/557523864 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02301a1400 con 0x7f02300eec20 2021-02-10T11:15:33.333 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:33.335 7f02355ec700 1 --2- 172.21.2.102:0/557523864 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f02300ef640 0x7f02301a0c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:15:33.333 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:33.335 7f022ffff700 1 --2- 172.21.2.102:0/557523864 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f02300eec20 0x7f02300ef020 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f02200093a0 tx=0x7f022000b990).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:33.334 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:33.335 7f022d7fa700 1 -- 172.21.2.102:0/557523864 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f022000bd80 con 0x7f02300eec20 2021-02-10T11:15:33.334 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:33.335 7f022d7fa700 1 -- 172.21.2.102:0/557523864 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02200052e0 con 0x7f02300eec20 2021-02-10T11:15:33.334 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:33.336 7f0236fcf700 1 -- 172.21.2.102:0/557523864 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02301a1250 con 0x7f02300eec20 2021-02-10T11:15:33.334 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:33.336 7f022d7fa700 1 -- 172.21.2.102:0/557523864 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f02200009b0 con 0x7f02300eec20 2021-02-10T11:15:33.334 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:33.336 7f0236fcf700 1 -- 172.21.2.102:0/557523864 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02301a1250 con 0x7f02300eec20 2021-02-10T11:15:33.334 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:33.336 7f022d7fa700 1 -- 172.21.2.102:0/557523864 <== mon.2 v2:172.21.2.113:3301/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f0220006030 con 0x7f02300eec20 2021-02-10T11:15:33.334 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:33.336 7f022d7fa700 1 --2- 172.21.2.102:0/557523864 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f02140186c0 0x7f021401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:33.334 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:33.336 7f022d7fa700 1 -- 172.21.2.102:0/557523864 <== mon.2 v2:172.21.2.113:3301/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f0220032660 con 0x7f02300eec20 2021-02-10T11:15:33.334 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:33.336 7f0234deb700 1 --2- 172.21.2.102:0/557523864 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f02140186c0 0x7f021401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:33.334 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:33.336 7f0234deb700 1 --2- 172.21.2.102:0/557523864 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f02140186c0 0x7f021401abc0 crc :-1 s=READY pgs=305 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:33.358 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:33.365 7f0236fcf700 1 -- 172.21.2.102:0/557523864 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02300edd10 con 0x7f02300eec20 2021-02-10T11:15:33.361 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:33.368 7f022d7fa700 1 -- 172.21.2.102:0/557523864 <== mon.2 v2:172.21.2.113:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f02200009b0 con 0x7f02300eec20 2021-02-10T11:15:33.491 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:33.497 7f0236fcf700 1 -- 172.21.2.102:0/557523864 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344430", "target": ["mgr", ""]}) v1 -- 0x7f02300e52e0 con 0x7f02140186c0 2021-02-10T11:15:33.493 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:33.500 7f022d7fa700 1 -- 172.21.2.102:0/557523864 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f0218006a30 con 0x7f02140186c0 2021-02-10T11:15:33.494 INFO:teuthology.orchestra.run.gibba002.stdout:/volumes/_nogroup/subvolume_0000000000344430/fcd03f6a-ded9-4c12-9d6b-9f7325c372ca 2021-02-10T11:15:33.495 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:33.501 7f0236fcf700 1 -- 172.21.2.102:0/557523864 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f02140186c0 msgr2=0x7f021401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:33.495 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:33.501 7f0236fcf700 1 --2- 172.21.2.102:0/557523864 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f02140186c0 0x7f021401abc0 crc :-1 s=READY pgs=305 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:33.495 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:33.501 7f0236fcf700 1 -- 172.21.2.102:0/557523864 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f02300eec20 msgr2=0x7f02300ef020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:33.495 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:33.501 7f0236fcf700 1 --2- 172.21.2.102:0/557523864 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f02300eec20 0x7f02300ef020 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f02200093a0 tx=0x7f022000b990).stop 2021-02-10T11:15:33.496 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:33.502 7f0236fcf700 1 -- 172.21.2.102:0/557523864 shutdown_connections 2021-02-10T11:15:33.496 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:33.502 7f0236fcf700 1 --2- 172.21.2.102:0/557523864 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f02300ef640 0x7f02301a0c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:33.496 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:33.502 7f0236fcf700 1 --2- 172.21.2.102:0/557523864 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f02300eec20 0x7f02300ef020 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:33.496 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:33.502 7f0236fcf700 1 --2- 172.21.2.102:0/557523864 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f02140186c0 0x7f021401abc0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:33.496 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:33.502 7f0236fcf700 1 --2- 172.21.2.102:0/557523864 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f02300acc50 0x7f0230098de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:33.496 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:33.502 7f0236fcf700 1 -- 172.21.2.102:0/557523864 >> 172.21.2.102:0/557523864 conn(0x7f02300c40e0 msgr2=0x7f02300f22c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:15:33.496 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:33.502 7f0236fcf700 1 -- 172.21.2.102:0/557523864 shutdown_connections 2021-02-10T11:15:33.496 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:33.502 7f0236fcf700 1 -- 172.21.2.102:0/557523864 wait complete. 2021-02-10T11:15:33.503 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000344430 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000344430/fcd03f6a-ded9-4c12-9d6b-9f7325c372ca 2021-02-10T11:15:33.503 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:15:33.504 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:15:33.504 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:15:33.839 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:15:33.839 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:15:33.840 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344430/fcd03f6a-ded9-4c12-9d6b-9f7325c372ca/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:15:34.219 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:15:34.219 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:15:34.219 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0247562 s, 42.4 MB/s 2021-02-10T11:15:34.220 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:15:34.220 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:15:34.221 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:15:34.555 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:15:34.555 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:15:34.556 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344430/fcd03f6a-ded9-4c12-9d6b-9f7325c372ca/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:15:34.933 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:15:34.934 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:15:34.934 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.025107 s, 41.8 MB/s 2021-02-10T11:15:34.935 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:15:34.935 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:15:34.935 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:15:35.271 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:15:35.271 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:15:35.272 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344430/fcd03f6a-ded9-4c12-9d6b-9f7325c372ca/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:15:35.649 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:15:35.649 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:15:35.649 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0266069 s, 39.4 MB/s 2021-02-10T11:15:35.650 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:15:35.650 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:15:35.650 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:15:35.985 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:15:35.985 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:15:35.986 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344430/fcd03f6a-ded9-4c12-9d6b-9f7325c372ca/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:15:36.359 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:15:36.360 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:15:36.360 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0232324 s, 45.1 MB/s 2021-02-10T11:15:36.361 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:15:36.362 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:15:36.362 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:15:36.697 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:15:36.697 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:15:36.698 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344430/fcd03f6a-ded9-4c12-9d6b-9f7325c372ca/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:15:37.073 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:15:37.073 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:15:37.073 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0239902 s, 43.7 MB/s 2021-02-10T11:15:37.074 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:15:37.074 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:15:37.075 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:15:37.410 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:15:37.410 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:15:37.411 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344430/fcd03f6a-ded9-4c12-9d6b-9f7325c372ca/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:15:37.790 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:15:37.790 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:15:37.790 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0262194 s, 40.0 MB/s 2021-02-10T11:15:37.791 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:15:37.791 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:15:37.791 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:15:38.126 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:15:38.127 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:15:38.127 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344430/fcd03f6a-ded9-4c12-9d6b-9f7325c372ca/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:15:38.497 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:15:38.498 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:15:38.498 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0232211 s, 45.2 MB/s 2021-02-10T11:15:38.500 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:15:38.500 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:15:38.500 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:15:38.835 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:15:38.835 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:15:38.836 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344430/fcd03f6a-ded9-4c12-9d6b-9f7325c372ca/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:15:39.212 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:15:39.212 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:15:39.212 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0245156 s, 42.8 MB/s 2021-02-10T11:15:39.213 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:15:39.213 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:15:39.213 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:15:39.549 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:15:39.549 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:15:39.549 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344430/fcd03f6a-ded9-4c12-9d6b-9f7325c372ca/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:15:39.926 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:15:39.927 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:15:39.927 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0256443 s, 40.9 MB/s 2021-02-10T11:15:39.928 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:15:39.928 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:15:39.928 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:15:40.264 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:15:40.264 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:15:40.265 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344430/fcd03f6a-ded9-4c12-9d6b-9f7325c372ca/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:15:40.641 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:15:40.641 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:15:40.641 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0249045 s, 42.1 MB/s 2021-02-10T11:15:40.642 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000344431 2021-02-10T11:15:41.075 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:41.064 7f90a7a91700 1 Processor -- start 2021-02-10T11:15:41.075 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:41.064 7f90a7a91700 1 -- start start 2021-02-10T11:15:41.075 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:41.065 7f90a7a91700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f90a00f05f0 0x7f90a00f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:41.075 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:41.065 7f90a7a91700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90a01b0c70 con 0x7f90a01b0470 2021-02-10T11:15:41.075 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:41.065 7f90a7a91700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90a00f0b50 con 0x7f90a01b0870 2021-02-10T11:15:41.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:41.065 7f90a7a91700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90a00e5ea0 con 0x7f90a00f05f0 2021-02-10T11:15:41.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:41.065 7f90a50ac700 1 --1- >> v1:172.21.2.113:6789/0 conn(0x7f90a01b0870 0x7f90a00efe00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.113:6789/0 says I am v1:172.21.2.102:52350/0 (socket says 172.21.2.102:52350) 2021-02-10T11:15:41.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:41.065 7f90a58ad700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f90a01b0470 0x7f90a00ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:45026/0 (socket says 172.21.2.102:45026) 2021-02-10T11:15:41.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:41.065 7f90a60ae700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f90a00f05f0 0x7f90a00f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:41.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:41.065 7f90a58ad700 1 -- 172.21.2.102:0/3817493311 learned_addr learned my addr 172.21.2.102:0/3817493311 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:15:41.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:41.065 7f90a48ab700 1 -- 172.21.2.102:0/3817493311 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3508218670 0 0) 0x7f9090000ce0 con 0x7f90a01b0470 2021-02-10T11:15:41.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:41.065 7f90a48ab700 1 -- 172.21.2.102:0/3817493311 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f908c002fe0 con 0x7f90a01b0470 2021-02-10T11:15:41.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:41.065 7f90a60ae700 1 -- 172.21.2.102:0/3817493311 >> v1:172.21.2.113:6789/0 conn(0x7f90a01b0870 legacy=0x7f90a00efe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:41.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:41.065 7f90a48ab700 1 -- 172.21.2.102:0/3817493311 <== mon.1 v1:172.21.2.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3807248268 0 0) 0x7f9098000fe0 con 0x7f90a01b0870 2021-02-10T11:15:41.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:41.066 7f90a60ae700 1 -- 172.21.2.102:0/3817493311 >> v1:172.21.2.102:6789/0 conn(0x7f90a01b0470 legacy=0x7f90a00ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:41.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:41.066 7f90a60ae700 1 -- 172.21.2.102:0/3817493311 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90a00e52e0 con 0x7f90a00f05f0 2021-02-10T11:15:41.077 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:41.066 7f90a60ae700 1 --2- 172.21.2.102:0/3817493311 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f90a00f05f0 0x7f90a00f20e0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 rx=0x7f909c006ab0 tx=0x7f909c009b40).ready entity=mon.0 client_cookie=d3deea4050536a4 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:41.077 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:41.066 7f90a48ab700 1 -- 172.21.2.102:0/3817493311 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f909c0022e0 con 0x7f90a00f05f0 2021-02-10T11:15:41.077 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:41.066 7f90a48ab700 1 -- 172.21.2.102:0/3817493311 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f909c002660 con 0x7f90a00f05f0 2021-02-10T11:15:41.077 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:41.066 7f90a48ab700 1 -- 172.21.2.102:0/3817493311 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f909c002a70 con 0x7f90a00f05f0 2021-02-10T11:15:41.077 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:41.066 7f90a7a91700 1 -- 172.21.2.102:0/3817493311 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f90a00f05f0 msgr2=0x7f90a00f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:41.077 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:41.066 7f90a7a91700 1 --2- 172.21.2.102:0/3817493311 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f90a00f05f0 0x7f90a00f20e0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 rx=0x7f909c006ab0 tx=0x7f909c009b40).stop 2021-02-10T11:15:41.077 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:41.067 7f90a7a91700 1 -- 172.21.2.102:0/3817493311 shutdown_connections 2021-02-10T11:15:41.077 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:41.067 7f90a7a91700 1 --2- 172.21.2.102:0/3817493311 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f90a00f05f0 0x7f90a00f20e0 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:41.077 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:41.067 7f90a7a91700 1 -- 172.21.2.102:0/3817493311 >> 172.21.2.102:0/3817493311 conn(0x7f90a00ce3e0 msgr2=0x7f90a00e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:15:41.077 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:41.067 7f90a7a91700 1 -- 172.21.2.102:0/3817493311 shutdown_connections 2021-02-10T11:15:41.077 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:41.067 7f90a7a91700 1 -- 172.21.2.102:0/3817493311 wait complete. 2021-02-10T11:15:41.077 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:41.082 7f90a7a91700 1 Processor -- start 2021-02-10T11:15:41.078 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:41.082 7f90a7a91700 1 -- start start 2021-02-10T11:15:41.079 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:41.082 7f90a7a91700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f90a009f140 0x7f90a00c1940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:41.079 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:41.082 7f90a7a91700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f90a00c1f60 0x7f90a009e300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:41.080 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:41.082 7f90a7a91700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f90a009e920 0x7f90a00c2960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:41.080 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:41.082 7f90a7a91700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90a00c23e0 con 0x7f90a009e920 2021-02-10T11:15:41.080 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:41.082 7f90a7a91700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90a009ed20 con 0x7f90a009f140 2021-02-10T11:15:41.080 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:41.082 7f90a7a91700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90a0105d10 con 0x7f90a00c1f60 2021-02-10T11:15:41.080 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:41.082 7f90a60ae700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f90a009e920 0x7f90a00c2960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:41.080 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:41.082 7f90a60ae700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f90a009e920 0x7f90a00c2960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:52812/0 (socket says 172.21.2.102:52812) 2021-02-10T11:15:41.080 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:41.082 7f90a60ae700 1 -- 172.21.2.102:0/409517845 learned_addr learned my addr 172.21.2.102:0/409517845 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:15:41.080 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:41.083 7f90a58ad700 1 --2- 172.21.2.102:0/409517845 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f90a009f140 0x7f90a00c1940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:41.080 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:41.083 7f90a60ae700 1 -- 172.21.2.102:0/409517845 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f90a00c1f60 msgr2=0x7f90a009e300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:41.080 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:41.083 7f90a60ae700 1 --2- 172.21.2.102:0/409517845 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f90a00c1f60 0x7f90a009e300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:15:41.080 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:41.083 7f90a60ae700 1 -- 172.21.2.102:0/409517845 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f90a009f140 msgr2=0x7f90a00c1940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:41.081 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:41.083 7f90a60ae700 1 --2- 172.21.2.102:0/409517845 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f90a009f140 0x7f90a00c1940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:41.081 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:41.083 7f90a60ae700 1 -- 172.21.2.102:0/409517845 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90a01a4910 con 0x7f90a009e920 2021-02-10T11:15:41.081 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:41.083 7f90a58ad700 1 --2- 172.21.2.102:0/409517845 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f90a009f140 0x7f90a00c1940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:15:41.081 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:41.083 7f90a60ae700 1 --2- 172.21.2.102:0/409517845 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f90a009e920 0x7f90a00c2960 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 rx=0x7f909c006f50 tx=0x7f909c00b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:41.081 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:41.083 7f90967fc700 1 -- 172.21.2.102:0/409517845 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f909c00bd50 con 0x7f90a009e920 2021-02-10T11:15:41.081 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:41.083 7f90967fc700 1 -- 172.21.2.102:0/409517845 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f909c002080 con 0x7f90a009e920 2021-02-10T11:15:41.081 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:41.083 7f90967fc700 1 -- 172.21.2.102:0/409517845 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f909c0025a0 con 0x7f90a009e920 2021-02-10T11:15:41.081 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:41.083 7f90a7a91700 1 -- 172.21.2.102:0/409517845 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90a01a4910 con 0x7f90a009e920 2021-02-10T11:15:41.081 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:41.083 7f90a7a91700 1 -- 172.21.2.102:0/409517845 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90a01a4910 con 0x7f90a009e920 2021-02-10T11:15:41.081 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:41.084 7f90967fc700 1 -- 172.21.2.102:0/409517845 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f909c000b40 con 0x7f90a009e920 2021-02-10T11:15:41.081 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:41.084 7f90967fc700 1 --2- 172.21.2.102:0/409517845 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f90880186c0 0x7f908801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:41.082 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:41.084 7f90967fc700 1 -- 172.21.2.102:0/409517845 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f909c032550 con 0x7f90a009e920 2021-02-10T11:15:41.082 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:41.084 7f90a58ad700 1 --2- 172.21.2.102:0/409517845 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f90880186c0 0x7f908801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:41.082 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:41.084 7f90a58ad700 1 --2- 172.21.2.102:0/409517845 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f90880186c0 0x7f908801abc0 crc :-1 s=READY pgs=306 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:41.109 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:41.115 7f90a7a91700 1 -- 172.21.2.102:0/409517845 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90a00f0b50 con 0x7f90a009e920 2021-02-10T11:15:41.111 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:41.118 7f90967fc700 1 -- 172.21.2.102:0/409517845 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f909c030d00 con 0x7f90a009e920 2021-02-10T11:15:41.238 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:41.245 7f90a7a91700 1 -- 172.21.2.102:0/409517845 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344431", "target": ["mgr", ""]}) v1 -- 0x7f90a00f0b50 con 0x7f90880186c0 2021-02-10T11:15:41.264 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:41.271 7f90967fc700 1 -- 172.21.2.102:0/409517845 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f9090006880 con 0x7f90880186c0 2021-02-10T11:15:41.273 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:41.278 7f90a7a91700 1 -- 172.21.2.102:0/409517845 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f90880186c0 msgr2=0x7f908801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:41.273 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:41.278 7f90a7a91700 1 --2- 172.21.2.102:0/409517845 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f90880186c0 0x7f908801abc0 crc :-1 s=READY pgs=306 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:41.273 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:41.278 7f90a7a91700 1 -- 172.21.2.102:0/409517845 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f90a009e920 msgr2=0x7f90a00c2960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:41.273 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:41.278 7f90a7a91700 1 --2- 172.21.2.102:0/409517845 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f90a009e920 0x7f90a00c2960 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 rx=0x7f909c006f50 tx=0x7f909c00b960).stop 2021-02-10T11:15:41.273 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:41.278 7f90a7a91700 1 -- 172.21.2.102:0/409517845 shutdown_connections 2021-02-10T11:15:41.273 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:41.278 7f90a7a91700 1 --2- 172.21.2.102:0/409517845 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f90a009e920 0x7f90a00c2960 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:41.273 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:41.278 7f90a7a91700 1 --2- 172.21.2.102:0/409517845 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f90a00c1f60 0x7f90a009e300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:15:41.273 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:41.278 7f90a7a91700 1 --2- 172.21.2.102:0/409517845 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f90880186c0 0x7f908801abc0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:41.273 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:41.278 7f90a7a91700 1 --2- 172.21.2.102:0/409517845 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f90a009f140 0x7f90a00c1940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:41.274 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:41.278 7f90a7a91700 1 -- 172.21.2.102:0/409517845 >> 172.21.2.102:0/409517845 conn(0x7f90a009fc10 msgr2=0x7f90a00b3250 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:15:41.274 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:41.278 7f90a7a91700 1 -- 172.21.2.102:0/409517845 shutdown_connections 2021-02-10T11:15:41.274 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:41.278 7f90a7a91700 1 -- 172.21.2.102:0/409517845 wait complete. 2021-02-10T11:15:41.279 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000344431 2021-02-10T11:15:41.711 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:41.701 7f53090be700 1 Processor -- start 2021-02-10T11:15:41.711 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:41.701 7f53090be700 1 -- start start 2021-02-10T11:15:41.711 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:41.702 7f53090be700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f53041b0470 0x7f53041b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:41.711 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:41.702 7f53090be700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53040f0b50 con 0x7f53040ef610 2021-02-10T11:15:41.711 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:41.702 7f53090be700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53040f24e0 con 0x7f53040f20e0 2021-02-10T11:15:41.712 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:41.702 7f53090be700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53040e5ea0 con 0x7f53041b0470 2021-02-10T11:15:41.712 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:41.702 7f5302d9d700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f53041b0470 0x7f53041b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:41.712 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:41.702 7f530259c700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f53040ef610 0x7f53040efa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:45040/0 (socket says 172.21.2.102:45040) 2021-02-10T11:15:41.712 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:41.702 7f530259c700 1 -- 172.21.2.102:0/326326882 learned_addr learned my addr 172.21.2.102:0/326326882 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:15:41.712 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:41.702 7f5302d9d700 1 -- 172.21.2.102:0/326326882 >> v1:172.21.2.113:6789/0 conn(0x7f53040f20e0 legacy=0x7f53040f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:15:41.712 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:41.702 7f5302d9d700 1 -- 172.21.2.102:0/326326882 >> v1:172.21.2.102:6789/0 conn(0x7f53040ef610 legacy=0x7f53040efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:41.712 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:41.702 7f5302d9d700 1 -- 172.21.2.102:0/326326882 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53040e52e0 con 0x7f53041b0470 2021-02-10T11:15:41.712 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:41.702 7f5302d9d700 1 --2- 172.21.2.102:0/326326882 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f53041b0470 0x7f53041b0870 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 rx=0x7f52ec006650 tx=0x7f52ec009a80).ready entity=mon.0 client_cookie=d32600bb7b5f91c6 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:41.712 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:41.702 7f5301d9b700 1 -- 172.21.2.102:0/326326882 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f52ec0009b0 con 0x7f53041b0470 2021-02-10T11:15:41.712 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:41.702 7f5301d9b700 1 -- 172.21.2.102:0/326326882 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52ec00c0c0 con 0x7f53041b0470 2021-02-10T11:15:41.712 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:41.702 7f5301d9b700 1 -- 172.21.2.102:0/326326882 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f52ec00c520 con 0x7f53041b0470 2021-02-10T11:15:41.712 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:41.703 7f53090be700 1 -- 172.21.2.102:0/326326882 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f53041b0470 msgr2=0x7f53041b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:41.712 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:41.703 7f53090be700 1 --2- 172.21.2.102:0/326326882 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f53041b0470 0x7f53041b0870 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 rx=0x7f52ec006650 tx=0x7f52ec009a80).stop 2021-02-10T11:15:41.713 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:41.703 7f53090be700 1 -- 172.21.2.102:0/326326882 shutdown_connections 2021-02-10T11:15:41.713 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:41.703 7f53090be700 1 --2- 172.21.2.102:0/326326882 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f53041b0470 0x7f53041b0870 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:41.713 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:41.703 7f53090be700 1 -- 172.21.2.102:0/326326882 >> 172.21.2.102:0/326326882 conn(0x7f53040ce3e0 msgr2=0x7f53040e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:15:41.713 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:41.703 7f53090be700 1 -- 172.21.2.102:0/326326882 shutdown_connections 2021-02-10T11:15:41.713 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:41.703 7f53090be700 1 -- 172.21.2.102:0/326326882 wait complete. 2021-02-10T11:15:41.714 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:41.718 7f53090be700 1 Processor -- start 2021-02-10T11:15:41.714 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:41.718 7f53090be700 1 -- start start 2021-02-10T11:15:41.715 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:41.718 7f53090be700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5304099d70 0x7f5304081e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:41.715 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:41.718 7f53090be700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f53040a9bf0 0x7f53040a9ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:41.715 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:41.718 7f53090be700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f53040aa610 0x7f53040c1830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:41.715 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:41.718 7f53090be700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5304082480 con 0x7f53040a9bf0 2021-02-10T11:15:41.715 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:41.718 7f53090be700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53040c1e80 con 0x7f53040aa610 2021-02-10T11:15:41.715 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:41.718 7f53090be700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53040c20a0 con 0x7f5304099d70 2021-02-10T11:15:41.715 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:41.719 7f530259c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f53040a9bf0 0x7f53040a9ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:41.715 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:41.719 7f530259c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f53040a9bf0 0x7f53040a9ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:52820/0 (socket says 172.21.2.102:52820) 2021-02-10T11:15:41.715 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:41.719 7f530259c700 1 -- 172.21.2.102:0/506883627 learned_addr learned my addr 172.21.2.102:0/506883627 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:15:41.715 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:41.719 7f530259c700 1 -- 172.21.2.102:0/506883627 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5304099d70 msgr2=0x7f5304081e60 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:15:41.715 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:41.719 7f530259c700 1 --2- 172.21.2.102:0/506883627 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5304099d70 0x7f5304081e60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:15:41.715 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:41.719 7f530259c700 1 -- 172.21.2.102:0/506883627 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f53040aa610 msgr2=0x7f53040c1830 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:15:41.716 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:41.719 7f530259c700 1 --2- 172.21.2.102:0/506883627 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f53040aa610 0x7f53040c1830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:15:41.716 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:41.719 7f530259c700 1 -- 172.21.2.102:0/506883627 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53040c22c0 con 0x7f53040a9bf0 2021-02-10T11:15:41.716 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:41.719 7f530259c700 1 --2- 172.21.2.102:0/506883627 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f53040a9bf0 0x7f53040a9ff0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 rx=0x7f52f4001c60 tx=0x7f52f400a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:41.716 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:41.719 7f52fb7fe700 1 -- 172.21.2.102:0/506883627 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f52f400acc0 con 0x7f53040a9bf0 2021-02-10T11:15:41.716 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:41.719 7f52fb7fe700 1 -- 172.21.2.102:0/506883627 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52f4004140 con 0x7f53040a9bf0 2021-02-10T11:15:41.716 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:41.719 7f53090be700 1 -- 172.21.2.102:0/506883627 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53040c22c0 con 0x7f53040a9bf0 2021-02-10T11:15:41.716 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:41.719 7f53090be700 1 -- 172.21.2.102:0/506883627 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53040c2850 con 0x7f53040a9bf0 2021-02-10T11:15:41.716 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:41.719 7f52fb7fe700 1 -- 172.21.2.102:0/506883627 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f52f4004620 con 0x7f53040a9bf0 2021-02-10T11:15:41.716 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:41.720 7f52fb7fe700 1 -- 172.21.2.102:0/506883627 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f52f4004320 con 0x7f53040a9bf0 2021-02-10T11:15:41.716 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:41.720 7f52fb7fe700 1 --2- 172.21.2.102:0/506883627 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f52e8018730 0x7f52e801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:41.716 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:41.720 7f52fb7fe700 1 -- 172.21.2.102:0/506883627 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f52f4032630 con 0x7f53040a9bf0 2021-02-10T11:15:41.716 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:41.720 7f5302d9d700 1 --2- 172.21.2.102:0/506883627 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f52e8018730 0x7f52e801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:41.717 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:41.721 7f5302d9d700 1 --2- 172.21.2.102:0/506883627 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f52e8018730 0x7f52e801ac10 crc :-1 s=READY pgs=307 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:41.745 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:41.751 7f53090be700 1 -- 172.21.2.102:0/506883627 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f530407e6b0 con 0x7f53040a9bf0 2021-02-10T11:15:41.747 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:41.754 7f52fb7fe700 1 -- 172.21.2.102:0/506883627 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f52f4000dd0 con 0x7f53040a9bf0 2021-02-10T11:15:41.875 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:41.882 7f53090be700 1 -- 172.21.2.102:0/506883627 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344431", "target": ["mgr", ""]}) v1 -- 0x7f5304082480 con 0x7f52e8018730 2021-02-10T11:15:41.878 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:41.885 7f52fb7fe700 1 -- 172.21.2.102:0/506883627 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f52ec000c90 con 0x7f52e8018730 2021-02-10T11:15:41.882 INFO:teuthology.orchestra.run.gibba002.stdout:/volumes/_nogroup/subvolume_0000000000344431/73afbb83-673e-440f-80b3-3ed9efb2b68d 2021-02-10T11:15:41.884 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:41.891 7f53090be700 1 -- 172.21.2.102:0/506883627 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f52e8018730 msgr2=0x7f52e801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:41.884 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:41.891 7f53090be700 1 --2- 172.21.2.102:0/506883627 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f52e8018730 0x7f52e801ac10 crc :-1 s=READY pgs=307 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:41.885 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:41.891 7f53090be700 1 -- 172.21.2.102:0/506883627 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f53040a9bf0 msgr2=0x7f53040a9ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:41.885 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:41.891 7f53090be700 1 --2- 172.21.2.102:0/506883627 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f53040a9bf0 0x7f53040a9ff0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 rx=0x7f52f4001c60 tx=0x7f52f400a8d0).stop 2021-02-10T11:15:41.885 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:41.891 7f53090be700 1 -- 172.21.2.102:0/506883627 shutdown_connections 2021-02-10T11:15:41.885 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:41.891 7f53090be700 1 --2- 172.21.2.102:0/506883627 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f52e8018730 0x7f52e801ac10 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:41.885 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:41.891 7f53090be700 1 --2- 172.21.2.102:0/506883627 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f53040aa610 0x7f53040c1830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:15:41.885 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:41.891 7f53090be700 1 --2- 172.21.2.102:0/506883627 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f53040a9bf0 0x7f53040a9ff0 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:41.885 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:41.891 7f53090be700 1 --2- 172.21.2.102:0/506883627 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5304099d70 0x7f5304081e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:15:41.885 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:41.891 7f53090be700 1 -- 172.21.2.102:0/506883627 >> 172.21.2.102:0/506883627 conn(0x7f5304088220 msgr2=0x7f53040a2d50 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:15:41.885 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:41.891 7f53090be700 1 -- 172.21.2.102:0/506883627 shutdown_connections 2021-02-10T11:15:41.885 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:41.891 7f53090be700 1 -- 172.21.2.102:0/506883627 wait complete. 2021-02-10T11:15:41.892 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000344431 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000344431/73afbb83-673e-440f-80b3-3ed9efb2b68d 2021-02-10T11:15:41.892 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:15:41.893 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:15:41.893 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:15:42.228 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:15:42.229 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:15:42.229 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344431/73afbb83-673e-440f-80b3-3ed9efb2b68d/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:15:42.610 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:15:42.610 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:15:42.611 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0244237 s, 42.9 MB/s 2021-02-10T11:15:42.612 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:15:42.612 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:15:42.612 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:15:42.948 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:15:42.948 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:15:42.949 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344431/73afbb83-673e-440f-80b3-3ed9efb2b68d/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:15:43.323 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:15:43.323 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:15:43.323 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0231562 s, 45.3 MB/s 2021-02-10T11:15:43.324 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:15:43.324 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:15:43.324 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:15:43.660 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:15:43.660 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:15:43.661 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344431/73afbb83-673e-440f-80b3-3ed9efb2b68d/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:15:44.033 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:15:44.033 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:15:44.034 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.023745 s, 44.2 MB/s 2021-02-10T11:15:44.035 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:15:44.035 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:15:44.035 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:15:44.372 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:15:44.372 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:15:44.372 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344431/73afbb83-673e-440f-80b3-3ed9efb2b68d/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:15:44.749 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:15:44.749 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:15:44.749 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0241516 s, 43.4 MB/s 2021-02-10T11:15:44.750 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:15:44.750 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:15:44.750 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:15:45.087 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:15:45.087 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:15:45.088 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344431/73afbb83-673e-440f-80b3-3ed9efb2b68d/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:15:45.464 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:15:45.464 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:15:45.464 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0243625 s, 43.0 MB/s 2021-02-10T11:15:45.465 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:15:45.465 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:15:45.466 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:15:45.800 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:15:45.801 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:15:45.801 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344431/73afbb83-673e-440f-80b3-3ed9efb2b68d/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:15:46.177 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:15:46.177 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:15:46.177 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0228686 s, 45.9 MB/s 2021-02-10T11:15:46.178 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:15:46.178 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:15:46.178 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:15:46.514 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:15:46.514 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:15:46.514 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344431/73afbb83-673e-440f-80b3-3ed9efb2b68d/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:15:46.890 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:15:46.890 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:15:46.890 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0234332 s, 44.7 MB/s 2021-02-10T11:15:46.891 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:15:46.891 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:15:46.891 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:15:47.226 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:15:47.227 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:15:47.227 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344431/73afbb83-673e-440f-80b3-3ed9efb2b68d/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:15:47.601 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:15:47.601 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:15:47.602 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0217944 s, 48.1 MB/s 2021-02-10T11:15:47.602 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:15:47.603 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:15:47.603 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:15:47.938 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:15:47.938 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:15:47.939 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344431/73afbb83-673e-440f-80b3-3ed9efb2b68d/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:15:48.313 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:15:48.313 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:15:48.313 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.02336 s, 44.9 MB/s 2021-02-10T11:15:48.314 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:15:48.314 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:15:48.314 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:15:48.651 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:15:48.651 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:15:48.651 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344431/73afbb83-673e-440f-80b3-3ed9efb2b68d/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:15:49.026 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:15:49.026 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:15:49.026 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0238463 s, 44.0 MB/s 2021-02-10T11:15:49.027 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000344432 2021-02-10T11:15:49.458 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:49.448 7f4257451700 1 Processor -- start 2021-02-10T11:15:49.459 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:49.449 7f4257451700 1 -- start start 2021-02-10T11:15:49.459 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:49.449 7f4257451700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f42501b0870 0x7f42500f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:49.459 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:49.449 7f4257451700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f42500efe00 0x7f42500f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:49.459 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:49.449 7f4257451700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42501b0c70 con 0x7f42501b0470 2021-02-10T11:15:49.459 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:49.449 7f4257451700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42500f0b50 con 0x7f42500efe00 2021-02-10T11:15:49.459 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:49.449 7f4257451700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42500f0820 con 0x7f42501b0870 2021-02-10T11:15:50.141 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:49.449 7f425526d700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f42501b0470 0x7f42500ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:45052/0 (socket says 172.21.2.102:45052) 2021-02-10T11:15:50.142 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:49.449 7f425526d700 1 -- 172.21.2.102:0/2331456907 learned_addr learned my addr 172.21.2.102:0/2331456907 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:15:50.142 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:49.449 7f4255a6e700 1 --2- 172.21.2.102:0/2331456907 >> v2:172.21.2.113:3300/0 conn(0x7f42500efe00 0x7f42500f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:50.142 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:49.449 7f4254a6c700 1 --2- 172.21.2.102:0/2331456907 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f42501b0870 0x7f42500f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:50.142 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:49.450 7f4243fff700 1 -- 172.21.2.102:0/2331456907 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3644255293 0 0) 0x7f42440009b0 con 0x7f42501b0470 2021-02-10T11:15:50.142 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:49.450 7f4243fff700 1 -- 172.21.2.102:0/2331456907 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f423c002fe0 con 0x7f42501b0470 2021-02-10T11:15:50.142 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:49.450 7f4254a6c700 1 -- 172.21.2.102:0/2331456907 >> v2:172.21.2.113:3300/0 conn(0x7f42500efe00 msgr2=0x7f42500f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:15:50.142 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:49.450 7f4254a6c700 1 --2- 172.21.2.102:0/2331456907 >> v2:172.21.2.113:3300/0 conn(0x7f42500efe00 0x7f42500f0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:50.143 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:49.450 7f4254a6c700 1 -- 172.21.2.102:0/2331456907 >> v1:172.21.2.102:6789/0 conn(0x7f42501b0470 legacy=0x7f42500ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:50.143 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:49.450 7f4254a6c700 1 -- 172.21.2.102:0/2331456907 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42500e52e0 con 0x7f42501b0870 2021-02-10T11:15:50.143 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:49.450 7f4255a6e700 1 --2- 172.21.2.102:0/2331456907 >> v2:172.21.2.113:3300/0 conn(0x7f42500efe00 0x7f42500f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:15:50.143 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:49.450 7f4254a6c700 1 --2- 172.21.2.102:0/2331456907 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f42501b0870 0x7f42500f20e0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f424c006650 tx=0x7f424c009a80).ready entity=mon.2 client_cookie=67b6704756327c26 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:50.143 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:49.450 7f4243fff700 1 -- 172.21.2.102:0/2331456907 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f424c00c0c0 con 0x7f42501b0870 2021-02-10T11:15:50.143 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:49.450 7f4243fff700 1 -- 172.21.2.102:0/2331456907 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f424c00c340 con 0x7f42501b0870 2021-02-10T11:15:50.143 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:49.450 7f4243fff700 1 -- 172.21.2.102:0/2331456907 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f424c00c790 con 0x7f42501b0870 2021-02-10T11:15:50.143 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:49.451 7f4257451700 1 -- 172.21.2.102:0/2331456907 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f42501b0870 msgr2=0x7f42500f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:50.143 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:49.451 7f4257451700 1 --2- 172.21.2.102:0/2331456907 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f42501b0870 0x7f42500f20e0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f424c006650 tx=0x7f424c009a80).stop 2021-02-10T11:15:50.143 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:49.451 7f4257451700 1 -- 172.21.2.102:0/2331456907 shutdown_connections 2021-02-10T11:15:50.143 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:49.451 7f4257451700 1 --2- 172.21.2.102:0/2331456907 >> v2:172.21.2.113:3300/0 conn(0x7f42500efe00 0x7f42500f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:50.143 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:49.451 7f4257451700 1 --2- 172.21.2.102:0/2331456907 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f42501b0870 0x7f42500f20e0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:50.144 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:49.451 7f4257451700 1 -- 172.21.2.102:0/2331456907 >> 172.21.2.102:0/2331456907 conn(0x7f42500ce3e0 msgr2=0x7f42500e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:15:50.144 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:49.451 7f4257451700 1 -- 172.21.2.102:0/2331456907 shutdown_connections 2021-02-10T11:15:50.144 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:49.451 7f4257451700 1 -- 172.21.2.102:0/2331456907 wait complete. 2021-02-10T11:15:50.147 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:49.466 7f4257451700 1 Processor -- start 2021-02-10T11:15:50.147 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:49.466 7f4257451700 1 -- start start 2021-02-10T11:15:50.147 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:49.466 7f4257451700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f42500a0420 0x7f42500a0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:50.147 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:49.466 7f4257451700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4250189d00 0x7f425018a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:50.147 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:49.466 7f4257451700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f425018a720 0x7f42500b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:50.147 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:49.466 7f4257451700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4250189800 con 0x7f42500a0420 2021-02-10T11:15:50.147 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:49.466 7f4257451700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f425018ab20 con 0x7f425018a720 2021-02-10T11:15:50.147 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:49.466 7f4257451700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42500a1140 con 0x7f4250189d00 2021-02-10T11:15:50.147 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:49.466 7f4255a6e700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f425018a720 0x7f42500b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:50.147 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:49.466 7f4255a6e700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f425018a720 0x7f42500b7be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.113:3300/0 says I am v2:172.21.2.102:56268/0 (socket says 172.21.2.102:56268) 2021-02-10T11:15:50.148 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:49.466 7f4255a6e700 1 -- 172.21.2.102:0/2428277139 learned_addr learned my addr 172.21.2.102:0/2428277139 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:15:50.148 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:49.466 7f425526d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f42500a0420 0x7f42500a0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:50.148 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:49.466 7f4254a6c700 1 --2- 172.21.2.102:0/2428277139 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4250189d00 0x7f425018a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:50.148 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:49.467 7f4255a6e700 1 -- 172.21.2.102:0/2428277139 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4250189d00 msgr2=0x7f425018a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:50.148 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:49.467 7f4255a6e700 1 --2- 172.21.2.102:0/2428277139 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4250189d00 0x7f425018a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:50.148 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:49.467 7f4255a6e700 1 -- 172.21.2.102:0/2428277139 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f42500a0420 msgr2=0x7f42500a0820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:50.148 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:49.467 7f4255a6e700 1 --2- 172.21.2.102:0/2428277139 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f42500a0420 0x7f42500a0820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:50.148 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:49.467 7f4255a6e700 1 -- 172.21.2.102:0/2428277139 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42500b8200 con 0x7f425018a720 2021-02-10T11:15:50.148 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:49.467 7f425526d700 1 --2- 172.21.2.102:0/2428277139 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f42500a0420 0x7f42500a0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:15:50.148 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:49.467 7f4254a6c700 1 --2- 172.21.2.102:0/2428277139 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4250189d00 0x7f425018a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:15:50.148 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:49.467 7f4255a6e700 1 --2- 172.21.2.102:0/2428277139 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f425018a720 0x7f42500b7be0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0x7f424800b970 tx=0x7f424800bd10).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:50.148 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:49.467 7f4241ffb700 1 -- 172.21.2.102:0/2428277139 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4248000f90 con 0x7f425018a720 2021-02-10T11:15:50.148 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:49.467 7f4241ffb700 1 -- 172.21.2.102:0/2428277139 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4248003030 con 0x7f425018a720 2021-02-10T11:15:50.148 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:49.467 7f4257451700 1 -- 172.21.2.102:0/2428277139 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42500b8200 con 0x7f425018a720 2021-02-10T11:15:50.149 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:49.467 7f4241ffb700 1 -- 172.21.2.102:0/2428277139 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4248001400 con 0x7f425018a720 2021-02-10T11:15:50.149 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:49.467 7f4257451700 1 -- 172.21.2.102:0/2428277139 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42500b8200 con 0x7f425018a720 2021-02-10T11:15:50.149 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:49.468 7f4241ffb700 1 -- 172.21.2.102:0/2428277139 <== mon.1 v2:172.21.2.113:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4248000df0 con 0x7f425018a720 2021-02-10T11:15:50.149 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:49.468 7f4241ffb700 1 --2- 172.21.2.102:0/2428277139 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f42300186c0 0x7f423001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:50.149 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:49.468 7f4241ffb700 1 -- 172.21.2.102:0/2428277139 <== mon.1 v2:172.21.2.113:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f42480328d0 con 0x7f425018a720 2021-02-10T11:15:50.149 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:49.468 7f425526d700 1 --2- 172.21.2.102:0/2428277139 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f42300186c0 0x7f423001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:50.149 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:49.469 7f425526d700 1 --2- 172.21.2.102:0/2428277139 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f42300186c0 0x7f423001abc0 crc :-1 s=READY pgs=308 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:50.149 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:49.499 7f4257451700 1 -- 172.21.2.102:0/2428277139 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f425018ab20 con 0x7f425018a720 2021-02-10T11:15:50.149 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:49.502 7f4241ffb700 1 -- 172.21.2.102:0/2428277139 <== mon.1 v2:172.21.2.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f424802afd0 con 0x7f425018a720 2021-02-10T11:15:50.149 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:49.630 7f4257451700 1 -- 172.21.2.102:0/2428277139 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344432", "target": ["mgr", ""]}) v1 -- 0x7f425018ab20 con 0x7f42300186c0 2021-02-10T11:15:50.149 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:49.658 7f4241ffb700 1 -- 172.21.2.102:0/2428277139 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f42440009b0 con 0x7f42300186c0 2021-02-10T11:15:50.149 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:49.663 7f4257451700 1 -- 172.21.2.102:0/2428277139 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f42300186c0 msgr2=0x7f423001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:50.149 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:49.663 7f4257451700 1 --2- 172.21.2.102:0/2428277139 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f42300186c0 0x7f423001abc0 crc :-1 s=READY pgs=308 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:50.149 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:49.663 7f4257451700 1 -- 172.21.2.102:0/2428277139 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f425018a720 msgr2=0x7f42500b7be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:50.150 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:49.663 7f4257451700 1 --2- 172.21.2.102:0/2428277139 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f425018a720 0x7f42500b7be0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0x7f424800b970 tx=0x7f424800bd10).stop 2021-02-10T11:15:50.150 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:49.664 7f4257451700 1 -- 172.21.2.102:0/2428277139 shutdown_connections 2021-02-10T11:15:50.150 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:49.664 7f4257451700 1 --2- 172.21.2.102:0/2428277139 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f42300186c0 0x7f423001abc0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:50.150 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:49.664 7f4257451700 1 --2- 172.21.2.102:0/2428277139 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f425018a720 0x7f42500b7be0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:50.150 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:49.664 7f4257451700 1 --2- 172.21.2.102:0/2428277139 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4250189d00 0x7f425018a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:50.150 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:49.664 7f4257451700 1 --2- 172.21.2.102:0/2428277139 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f42500a0420 0x7f42500a0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:50.150 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:49.664 7f4257451700 1 -- 172.21.2.102:0/2428277139 >> 172.21.2.102:0/2428277139 conn(0x7f4250094e10 msgr2=0x7f42500a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:15:50.150 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:49.664 7f4257451700 1 -- 172.21.2.102:0/2428277139 shutdown_connections 2021-02-10T11:15:50.150 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:49.664 7f4257451700 1 -- 172.21.2.102:0/2428277139 wait complete. 2021-02-10T11:15:50.150 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000344432 2021-02-10T11:15:50.585 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:50.574 7f8b42734700 1 Processor -- start 2021-02-10T11:15:50.585 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:50.574 7f8b42734700 1 -- start start 2021-02-10T11:15:50.585 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:50.574 7f8b42734700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f8b3c1b0870 0x7f8b3c0f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:50.585 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:50.574 7f8b42734700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8b3c0efe00 0x7f8b3c0f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:50.585 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:50.574 7f8b42734700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b3c1b0c70 con 0x7f8b3c1b0470 2021-02-10T11:15:50.585 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:50.574 7f8b42734700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b3c0f0b50 con 0x7f8b3c1b0870 2021-02-10T11:15:50.585 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:50.574 7f8b42734700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b3c0f0820 con 0x7f8b3c0efe00 2021-02-10T11:15:50.585 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:50.575 7f8b40d51700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8b3c0efe00 0x7f8b3c0f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:50.585 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:50.575 7f8b3b7fe700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f8b3c1b0870 0x7f8b3c0f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:50.586 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:50.576 7f8b40d51700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8b3c0efe00 0x7f8b3c0f0200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.113:3301/0 says I am v2:172.21.2.102:58382/0 (socket says 172.21.2.102:58382) 2021-02-10T11:15:50.586 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:50.576 7f8b3b7fe700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f8b3c1b0870 0x7f8b3c0f20e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.113:3300/0 says I am v2:172.21.2.102:56272/0 (socket says 172.21.2.102:56272) 2021-02-10T11:15:50.586 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:50.576 7f8b40d51700 1 -- 172.21.2.102:0/829158222 learned_addr learned my addr 172.21.2.102:0/829158222 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:15:50.586 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:50.576 7f8b3bfff700 1 --1- 172.21.2.102:0/829158222 >> v1:172.21.2.113:6789/0 conn(0x7f8b3c1b0470 0x7f8b3c0ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.113:6789/0 says I am v1:172.21.2.102:52390/0 (socket says 172.21.2.102:52390) 2021-02-10T11:15:50.586 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:50.576 7f8b40d51700 1 -- 172.21.2.102:0/829158222 >> v2:172.21.2.113:3300/0 conn(0x7f8b3c1b0870 msgr2=0x7f8b3c0f20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:15:50.586 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:50.576 7f8b40d51700 1 --2- 172.21.2.102:0/829158222 >> v2:172.21.2.113:3300/0 conn(0x7f8b3c1b0870 0x7f8b3c0f20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:50.586 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:50.576 7f8b40d51700 1 -- 172.21.2.102:0/829158222 >> v1:172.21.2.113:6789/0 conn(0x7f8b3c1b0470 legacy=0x7f8b3c0ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:50.586 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:50.576 7f8b40d51700 1 -- 172.21.2.102:0/829158222 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b3c0e52e0 con 0x7f8b3c0efe00 2021-02-10T11:15:50.586 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:50.576 7f8b40d51700 1 --2- 172.21.2.102:0/829158222 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8b3c0efe00 0x7f8b3c0f0200 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f8b34006aa0 tx=0x7f8b34009f20).ready entity=mon.2 client_cookie=dc0fdb9a39f8fc61 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:50.586 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:50.577 7f8b3affd700 1 -- 172.21.2.102:0/829158222 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8b340066f0 con 0x7f8b3c0efe00 2021-02-10T11:15:50.586 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:50.577 7f8b3affd700 1 -- 172.21.2.102:0/829158222 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b34003030 con 0x7f8b3c0efe00 2021-02-10T11:15:50.586 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:50.577 7f8b3affd700 1 -- 172.21.2.102:0/829158222 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8b3400b7b0 con 0x7f8b3c0efe00 2021-02-10T11:15:50.587 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:50.577 7f8b42734700 1 -- 172.21.2.102:0/829158222 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8b3c0efe00 msgr2=0x7f8b3c0f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:50.587 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:50.577 7f8b42734700 1 --2- 172.21.2.102:0/829158222 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8b3c0efe00 0x7f8b3c0f0200 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f8b34006aa0 tx=0x7f8b34009f20).stop 2021-02-10T11:15:50.587 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:50.577 7f8b42734700 1 -- 172.21.2.102:0/829158222 shutdown_connections 2021-02-10T11:15:50.587 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:50.577 7f8b42734700 1 --2- 172.21.2.102:0/829158222 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8b3c0efe00 0x7f8b3c0f0200 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:50.587 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:50.577 7f8b42734700 1 --2- 172.21.2.102:0/829158222 >> v2:172.21.2.113:3300/0 conn(0x7f8b3c1b0870 0x7f8b3c0f20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:50.587 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:50.577 7f8b42734700 1 -- 172.21.2.102:0/829158222 >> 172.21.2.102:0/829158222 conn(0x7f8b3c0ce3e0 msgr2=0x7f8b3c0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:15:50.587 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:50.577 7f8b42734700 1 -- 172.21.2.102:0/829158222 shutdown_connections 2021-02-10T11:15:50.587 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:50.577 7f8b42734700 1 -- 172.21.2.102:0/829158222 wait complete. 2021-02-10T11:15:50.589 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:50.592 7f8b42734700 1 Processor -- start 2021-02-10T11:15:50.589 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:50.592 7f8b42734700 1 -- start start 2021-02-10T11:15:50.589 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:50.592 7f8b42734700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8b3c105d10 0x7f8b3c09eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:50.589 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:50.592 7f8b42734700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8b3c106110 0x7f8b3c09f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:50.589 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:50.592 7f8b42734700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8b3c0b2a00 0x7f8b3c0b2e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:50.589 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:50.592 7f8b42734700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b3c09e560 con 0x7f8b3c105d10 2021-02-10T11:15:50.589 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:50.592 7f8b42734700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b3c09c090 con 0x7f8b3c0b2a00 2021-02-10T11:15:50.590 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:50.592 7f8b42734700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b3c0b3420 con 0x7f8b3c106110 2021-02-10T11:15:50.590 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:50.593 7f8b3bfff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8b3c105d10 0x7f8b3c09eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:50.590 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:50.593 7f8b3bfff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8b3c105d10 0x7f8b3c09eae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:52850/0 (socket says 172.21.2.102:52850) 2021-02-10T11:15:50.590 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:50.593 7f8b40d51700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8b3c0b2a00 0x7f8b3c0b2e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:50.590 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:50.593 7f8b3bfff700 1 -- 172.21.2.102:0/2091134201 learned_addr learned my addr 172.21.2.102:0/2091134201 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:15:50.590 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:50.593 7f8b3b7fe700 1 --2- 172.21.2.102:0/2091134201 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8b3c106110 0x7f8b3c09f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:50.590 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:50.593 7f8b40d51700 1 -- 172.21.2.102:0/2091134201 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8b3c106110 msgr2=0x7f8b3c09f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:50.590 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:50.593 7f8b40d51700 1 --2- 172.21.2.102:0/2091134201 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8b3c106110 0x7f8b3c09f100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:50.590 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:50.593 7f8b40d51700 1 -- 172.21.2.102:0/2091134201 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8b3c105d10 msgr2=0x7f8b3c09eae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:50.590 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:50.593 7f8b40d51700 1 --2- 172.21.2.102:0/2091134201 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8b3c105d10 0x7f8b3c09eae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:50.590 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:50.593 7f8b40d51700 1 -- 172.21.2.102:0/2091134201 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b3c0b3640 con 0x7f8b3c0b2a00 2021-02-10T11:15:50.590 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:50.593 7f8b3b7fe700 1 --2- 172.21.2.102:0/2091134201 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8b3c106110 0x7f8b3c09f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:15:50.591 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:50.593 7f8b3bfff700 1 --2- 172.21.2.102:0/2091134201 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8b3c105d10 0x7f8b3c09eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:15:50.591 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:50.593 7f8b40d51700 1 --2- 172.21.2.102:0/2091134201 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8b3c0b2a00 0x7f8b3c0b2e00 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0x7f8b34006e60 tx=0x7f8b3400bca0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:50.591 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:50.594 7f8b38ff9700 1 -- 172.21.2.102:0/2091134201 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8b34009450 con 0x7f8b3c0b2a00 2021-02-10T11:15:50.591 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:50.594 7f8b38ff9700 1 -- 172.21.2.102:0/2091134201 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b34008030 con 0x7f8b3c0b2a00 2021-02-10T11:15:50.591 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:50.594 7f8b38ff9700 1 -- 172.21.2.102:0/2091134201 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8b340084d0 con 0x7f8b3c0b2a00 2021-02-10T11:15:50.591 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:50.594 7f8b42734700 1 -- 172.21.2.102:0/2091134201 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b3c0b3640 con 0x7f8b3c0b2a00 2021-02-10T11:15:50.591 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:50.594 7f8b42734700 1 -- 172.21.2.102:0/2091134201 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b3c0b3bd0 con 0x7f8b3c0b2a00 2021-02-10T11:15:50.591 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:50.594 7f8b38ff9700 1 -- 172.21.2.102:0/2091134201 <== mon.1 v2:172.21.2.113:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f8b340056b0 con 0x7f8b3c0b2a00 2021-02-10T11:15:50.591 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:50.594 7f8b38ff9700 1 --2- 172.21.2.102:0/2091134201 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8b1c0186c0 0x7f8b1c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:50.591 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:50.594 7f8b38ff9700 1 -- 172.21.2.102:0/2091134201 <== mon.1 v2:172.21.2.113:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f8b34032640 con 0x7f8b3c0b2a00 2021-02-10T11:15:50.591 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:50.595 7f8b3bfff700 1 --2- 172.21.2.102:0/2091134201 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8b1c0186c0 0x7f8b1c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:50.591 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:50.595 7f8b3bfff700 1 --2- 172.21.2.102:0/2091134201 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8b1c0186c0 0x7f8b1c01abc0 crc :-1 s=READY pgs=309 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:50.618 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:50.625 7f8b42734700 1 -- 172.21.2.102:0/2091134201 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b3c105ae0 con 0x7f8b3c0b2a00 2021-02-10T11:15:50.621 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:50.628 7f8b38ff9700 1 -- 172.21.2.102:0/2091134201 <== mon.1 v2:172.21.2.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f8b34009450 con 0x7f8b3c0b2a00 2021-02-10T11:15:50.753 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:50.759 7f8b42734700 1 -- 172.21.2.102:0/2091134201 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344432", "target": ["mgr", ""]}) v1 -- 0x7f8b3c1a16c0 con 0x7f8b1c0186c0 2021-02-10T11:15:50.756 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:50.763 7f8b38ff9700 1 -- 172.21.2.102:0/2091134201 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f8b2c0077a0 con 0x7f8b1c0186c0 2021-02-10T11:15:50.756 INFO:teuthology.orchestra.run.gibba002.stdout:/volumes/_nogroup/subvolume_0000000000344432/1e8ef6b7-ce2a-47f9-ba54-7bd0c7e6170d 2021-02-10T11:15:50.757 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:50.764 7f8b42734700 1 -- 172.21.2.102:0/2091134201 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8b1c0186c0 msgr2=0x7f8b1c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:50.757 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:50.764 7f8b42734700 1 --2- 172.21.2.102:0/2091134201 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8b1c0186c0 0x7f8b1c01abc0 crc :-1 s=READY pgs=309 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:50.758 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:50.764 7f8b42734700 1 -- 172.21.2.102:0/2091134201 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8b3c0b2a00 msgr2=0x7f8b3c0b2e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:50.758 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:50.764 7f8b42734700 1 --2- 172.21.2.102:0/2091134201 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8b3c0b2a00 0x7f8b3c0b2e00 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0x7f8b34006e60 tx=0x7f8b3400bca0).stop 2021-02-10T11:15:50.758 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:50.764 7f8b42734700 1 -- 172.21.2.102:0/2091134201 shutdown_connections 2021-02-10T11:15:50.758 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:50.764 7f8b42734700 1 --2- 172.21.2.102:0/2091134201 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8b1c0186c0 0x7f8b1c01abc0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:50.758 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:50.764 7f8b42734700 1 --2- 172.21.2.102:0/2091134201 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8b3c0b2a00 0x7f8b3c0b2e00 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:50.758 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:50.764 7f8b42734700 1 --2- 172.21.2.102:0/2091134201 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8b3c106110 0x7f8b3c09f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:50.758 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:50.764 7f8b42734700 1 --2- 172.21.2.102:0/2091134201 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8b3c105d10 0x7f8b3c09eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:50.758 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:50.764 7f8b42734700 1 -- 172.21.2.102:0/2091134201 >> 172.21.2.102:0/2091134201 conn(0x7f8b3c0c36f0 msgr2=0x7f8b3c0bd640 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:15:50.758 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:50.764 7f8b42734700 1 -- 172.21.2.102:0/2091134201 shutdown_connections 2021-02-10T11:15:50.758 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:50.764 7f8b42734700 1 -- 172.21.2.102:0/2091134201 wait complete. 2021-02-10T11:15:50.765 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000344432 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000344432/1e8ef6b7-ce2a-47f9-ba54-7bd0c7e6170d 2021-02-10T11:15:50.765 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:15:50.766 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:15:50.766 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:15:51.101 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:15:51.102 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:15:51.102 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344432/1e8ef6b7-ce2a-47f9-ba54-7bd0c7e6170d/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:15:51.484 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:15:51.484 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:15:51.484 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0233509 s, 44.9 MB/s 2021-02-10T11:15:51.485 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:15:51.485 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:15:51.485 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:15:51.826 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:15:51.827 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:15:51.827 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344432/1e8ef6b7-ce2a-47f9-ba54-7bd0c7e6170d/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:15:52.205 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:15:52.205 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:15:52.205 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0239786 s, 43.7 MB/s 2021-02-10T11:15:52.206 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:15:52.206 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:15:52.207 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:15:52.545 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:15:52.545 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:15:52.545 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344432/1e8ef6b7-ce2a-47f9-ba54-7bd0c7e6170d/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:15:52.926 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:15:52.927 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:15:52.927 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0247824 s, 42.3 MB/s 2021-02-10T11:15:52.928 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:15:52.928 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:15:52.928 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:15:53.265 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:15:53.265 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:15:53.266 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344432/1e8ef6b7-ce2a-47f9-ba54-7bd0c7e6170d/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:15:53.642 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:15:53.642 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:15:53.642 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0252346 s, 41.6 MB/s 2021-02-10T11:15:53.643 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:15:53.643 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:15:53.644 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:15:53.979 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:15:53.979 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:15:53.979 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344432/1e8ef6b7-ce2a-47f9-ba54-7bd0c7e6170d/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:15:54.358 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:15:54.358 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:15:54.358 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0256897 s, 40.8 MB/s 2021-02-10T11:15:54.359 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:15:54.359 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:15:54.359 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:15:54.696 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:15:54.697 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:15:54.697 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344432/1e8ef6b7-ce2a-47f9-ba54-7bd0c7e6170d/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:15:55.073 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:15:55.074 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:15:55.074 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0232791 s, 45.0 MB/s 2021-02-10T11:15:55.075 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:15:55.075 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:15:55.075 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:15:55.410 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:15:55.411 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:15:55.411 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344432/1e8ef6b7-ce2a-47f9-ba54-7bd0c7e6170d/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:15:55.789 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:15:55.790 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:15:55.790 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0244202 s, 42.9 MB/s 2021-02-10T11:15:55.791 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:15:55.791 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:15:55.791 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:15:56.127 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:15:56.127 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:15:56.128 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344432/1e8ef6b7-ce2a-47f9-ba54-7bd0c7e6170d/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:15:56.504 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:15:56.504 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:15:56.504 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0254237 s, 41.2 MB/s 2021-02-10T11:15:56.505 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:15:56.505 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:15:56.506 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:15:56.841 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:15:56.842 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:15:56.842 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344432/1e8ef6b7-ce2a-47f9-ba54-7bd0c7e6170d/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:15:57.216 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:15:57.217 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:15:57.217 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0235467 s, 44.5 MB/s 2021-02-10T11:15:57.217 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:15:57.218 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:15:57.218 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:15:57.308 DEBUG:teuthology.orchestra.run.gibba002:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:15:57.311 DEBUG:teuthology.orchestra.run.gibba013:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:15:57.553 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:15:57.553 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:15:57.554 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344432/1e8ef6b7-ce2a-47f9-ba54-7bd0c7e6170d/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:15:57.931 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:15:57.931 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:15:57.931 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.025177 s, 41.6 MB/s 2021-02-10T11:15:57.932 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000344433 2021-02-10T11:15:58.364 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:58.354 7fb236fc5700 1 Processor -- start 2021-02-10T11:15:58.364 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:58.354 7fb236fc5700 1 -- start start 2021-02-10T11:15:58.364 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:58.354 7fb236fc5700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fb2301b0470 0x7fb2301b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:58.364 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:58.354 7fb236fc5700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb2300f20e0 0x7fb2300f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:58.364 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:58.354 7fb236fc5700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2300f0b50 con 0x7fb2300ef610 2021-02-10T11:15:58.364 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:58.354 7fb236fc5700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2300f0820 con 0x7fb2301b0470 2021-02-10T11:15:58.364 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:58.354 7fb236fc5700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2300f24e0 con 0x7fb2300f20e0 2021-02-10T11:15:58.365 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:58.355 7fb22ffff700 1 --1- >> v1:172.21.2.113:6789/0 conn(0x7fb2300ef610 0x7fb2300efa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.113:6789/0 says I am v1:172.21.2.102:52408/0 (socket says 172.21.2.102:52408) 2021-02-10T11:15:58.365 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:58.355 7fb234de1700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fb2301b0470 0x7fb2301b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:58.365 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:58.355 7fb22ffff700 1 -- 172.21.2.102:0/548842216 learned_addr learned my addr 172.21.2.102:0/548842216 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:15:58.365 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:58.355 7fb2355e2700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb2300f20e0 0x7fb2300f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:58.365 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:58.355 7fb234de1700 1 -- 172.21.2.102:0/548842216 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb2300f20e0 msgr2=0x7fb2300f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:15:58.365 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:58.355 7fb234de1700 1 --2- 172.21.2.102:0/548842216 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb2300f20e0 0x7fb2300f0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:58.365 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:58.355 7fb234de1700 1 -- 172.21.2.102:0/548842216 >> v1:172.21.2.113:6789/0 conn(0x7fb2300ef610 legacy=0x7fb2300efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:58.365 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:58.355 7fb234de1700 1 -- 172.21.2.102:0/548842216 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2300e52e0 con 0x7fb2301b0470 2021-02-10T11:15:58.365 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:58.356 7fb234de1700 1 --2- 172.21.2.102:0/548842216 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb2301b0470 0x7fb2301b0870 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7fb220006650 tx=0x7fb220009a80).ready entity=mon.1 client_cookie=798722034779646a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:58.365 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:58.356 7fb22f7fe700 1 -- 172.21.2.102:0/548842216 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb22000c0c0 con 0x7fb2301b0470 2021-02-10T11:15:58.365 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:58.356 7fb22f7fe700 1 -- 172.21.2.102:0/548842216 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb22000c340 con 0x7fb2301b0470 2021-02-10T11:15:58.365 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:58.356 7fb22f7fe700 1 -- 172.21.2.102:0/548842216 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb22000c860 con 0x7fb2301b0470 2021-02-10T11:15:58.365 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:58.356 7fb236fc5700 1 -- 172.21.2.102:0/548842216 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb2301b0470 msgr2=0x7fb2301b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:58.366 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:58.356 7fb236fc5700 1 --2- 172.21.2.102:0/548842216 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb2301b0470 0x7fb2301b0870 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7fb220006650 tx=0x7fb220009a80).stop 2021-02-10T11:15:58.366 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:58.356 7fb236fc5700 1 -- 172.21.2.102:0/548842216 shutdown_connections 2021-02-10T11:15:58.366 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:58.356 7fb236fc5700 1 --2- 172.21.2.102:0/548842216 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb2300f20e0 0x7fb2300f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:58.366 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:58.357 7fb236fc5700 1 --2- 172.21.2.102:0/548842216 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb2301b0470 0x7fb2301b0870 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:58.366 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:58.357 7fb236fc5700 1 -- 172.21.2.102:0/548842216 >> 172.21.2.102:0/548842216 conn(0x7fb2300ce3e0 msgr2=0x7fb2300e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:15:58.366 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:58.357 7fb236fc5700 1 -- 172.21.2.102:0/548842216 shutdown_connections 2021-02-10T11:15:58.366 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:58.357 7fb236fc5700 1 -- 172.21.2.102:0/548842216 wait complete. 2021-02-10T11:15:58.368 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:58.371 7fb236fc5700 1 Processor -- start 2021-02-10T11:15:58.368 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:58.371 7fb236fc5700 1 -- start start 2021-02-10T11:15:58.368 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:58.372 7fb236fc5700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb2300ba760 0x7fb2300bab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:58.368 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:58.372 7fb236fc5700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb2300bb180 0x7fb23008a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:58.368 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:58.372 7fb236fc5700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb23008aa50 0x7fb23008ae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:58.368 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:58.372 7fb236fc5700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb23008b570 con 0x7fb2300bb180 2021-02-10T11:15:58.369 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:58.372 7fb236fc5700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb23008bc00 con 0x7fb23008aa50 2021-02-10T11:15:58.369 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:58.372 7fb236fc5700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb23008be20 con 0x7fb2300ba760 2021-02-10T11:15:58.369 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:58.372 7fb22ffff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb2300bb180 0x7fb23008a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:58.369 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:58.372 7fb22ffff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb2300bb180 0x7fb23008a430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:52864/0 (socket says 172.21.2.102:52864) 2021-02-10T11:15:58.369 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:58.372 7fb22ffff700 1 -- 172.21.2.102:0/4125926167 learned_addr learned my addr 172.21.2.102:0/4125926167 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:15:58.369 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:58.372 7fb2355e2700 1 --2- 172.21.2.102:0/4125926167 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb23008aa50 0x7fb23008ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:58.369 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:58.372 7fb234de1700 1 --2- 172.21.2.102:0/4125926167 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb2300ba760 0x7fb2300bab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:58.369 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:58.372 7fb22ffff700 1 -- 172.21.2.102:0/4125926167 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb2300ba760 msgr2=0x7fb2300bab60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:58.369 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:58.372 7fb22ffff700 1 --2- 172.21.2.102:0/4125926167 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb2300ba760 0x7fb2300bab60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:58.369 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:58.372 7fb22ffff700 1 -- 172.21.2.102:0/4125926167 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb23008aa50 msgr2=0x7fb23008ae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:58.369 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:58.372 7fb22ffff700 1 --2- 172.21.2.102:0/4125926167 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb23008aa50 0x7fb23008ae50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:58.369 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:58.372 7fb22ffff700 1 -- 172.21.2.102:0/4125926167 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb23008c040 con 0x7fb2300bb180 2021-02-10T11:15:58.369 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:58.372 7fb2355e2700 1 --2- 172.21.2.102:0/4125926167 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb23008aa50 0x7fb23008ae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:15:58.370 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:58.373 7fb22ffff700 1 --2- 172.21.2.102:0/4125926167 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb2300bb180 0x7fb23008a430 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 rx=0x7fb224001c60 tx=0x7fb22400a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:58.370 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:58.373 7fb22d7fa700 1 -- 172.21.2.102:0/4125926167 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb22400acc0 con 0x7fb2300bb180 2021-02-10T11:15:58.370 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:58.373 7fb22d7fa700 1 -- 172.21.2.102:0/4125926167 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb224004140 con 0x7fb2300bb180 2021-02-10T11:15:58.370 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:58.373 7fb22d7fa700 1 -- 172.21.2.102:0/4125926167 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb2240045a0 con 0x7fb2300bb180 2021-02-10T11:15:58.370 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:58.373 7fb236fc5700 1 -- 172.21.2.102:0/4125926167 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb23008c040 con 0x7fb2300bb180 2021-02-10T11:15:58.370 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:58.373 7fb236fc5700 1 -- 172.21.2.102:0/4125926167 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb23008c5d0 con 0x7fb2300bb180 2021-02-10T11:15:58.370 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:58.373 7fb22d7fa700 1 -- 172.21.2.102:0/4125926167 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fb224004530 con 0x7fb2300bb180 2021-02-10T11:15:58.370 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:58.373 7fb22d7fa700 1 --2- 172.21.2.102:0/4125926167 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb2100186c0 0x7fb21001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:58.370 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:58.373 7fb22d7fa700 1 -- 172.21.2.102:0/4125926167 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fb2240325b0 con 0x7fb2300bb180 2021-02-10T11:15:58.370 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:58.374 7fb234de1700 1 --2- 172.21.2.102:0/4125926167 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb2100186c0 0x7fb21001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:58.370 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:58.374 7fb234de1700 1 --2- 172.21.2.102:0/4125926167 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb2100186c0 0x7fb21001abc0 crc :-1 s=READY pgs=310 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:58.397 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:58.404 7fb236fc5700 1 -- 172.21.2.102:0/4125926167 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb230105ae0 con 0x7fb2300bb180 2021-02-10T11:15:58.399 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:58.406 7fb22d7fa700 1 -- 172.21.2.102:0/4125926167 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fb22402fde0 con 0x7fb2300bb180 2021-02-10T11:15:58.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:58.532 7fb236fc5700 1 -- 172.21.2.102:0/4125926167 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344433", "target": ["mgr", ""]}) v1 -- 0x7fb23008c5d0 con 0x7fb2100186c0 2021-02-10T11:15:58.552 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:58.559 7fb22d7fa700 1 -- 172.21.2.102:0/4125926167 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fb220000c90 con 0x7fb2100186c0 2021-02-10T11:15:58.558 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:58.565 7fb236fc5700 1 -- 172.21.2.102:0/4125926167 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb2100186c0 msgr2=0x7fb21001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:58.558 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:58.565 7fb236fc5700 1 --2- 172.21.2.102:0/4125926167 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb2100186c0 0x7fb21001abc0 crc :-1 s=READY pgs=310 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:58.558 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:58.565 7fb236fc5700 1 -- 172.21.2.102:0/4125926167 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb2300bb180 msgr2=0x7fb23008a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:58.558 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:58.565 7fb236fc5700 1 --2- 172.21.2.102:0/4125926167 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb2300bb180 0x7fb23008a430 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 rx=0x7fb224001c60 tx=0x7fb22400a8d0).stop 2021-02-10T11:15:58.559 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:58.565 7fb236fc5700 1 -- 172.21.2.102:0/4125926167 shutdown_connections 2021-02-10T11:15:58.559 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:58.565 7fb236fc5700 1 --2- 172.21.2.102:0/4125926167 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb2100186c0 0x7fb21001abc0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:58.559 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:58.565 7fb236fc5700 1 --2- 172.21.2.102:0/4125926167 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb23008aa50 0x7fb23008ae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:58.559 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:58.565 7fb236fc5700 1 --2- 172.21.2.102:0/4125926167 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb2300bb180 0x7fb23008a430 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:58.559 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:58.565 7fb236fc5700 1 --2- 172.21.2.102:0/4125926167 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb2300ba760 0x7fb2300bab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:58.559 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:58.565 7fb236fc5700 1 -- 172.21.2.102:0/4125926167 >> 172.21.2.102:0/4125926167 conn(0x7fb2300bbf60 msgr2=0x7fb230105d10 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:15:58.559 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:58.565 7fb236fc5700 1 -- 172.21.2.102:0/4125926167 shutdown_connections 2021-02-10T11:15:58.559 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:58.565 7fb236fc5700 1 -- 172.21.2.102:0/4125926167 wait complete. 2021-02-10T11:15:58.567 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000344433 2021-02-10T11:15:59.000 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:58.990 7feb49af2700 1 Processor -- start 2021-02-10T11:15:59.000 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:58.990 7feb49af2700 1 -- start start 2021-02-10T11:15:59.000 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:58.990 7feb49af2700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7feb441b0470 0x7feb441b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:59.000 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:58.990 7feb49af2700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7feb440ef610 0x7feb440efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:59.000 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:58.990 7feb49af2700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7feb440f0030 0x7feb440f0430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:59.000 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:58.990 7feb49af2700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb440e5ea0 con 0x7feb440ef610 2021-02-10T11:15:59.000 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:58.990 7feb49af2700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb440f0b50 con 0x7feb441b0470 2021-02-10T11:15:59.000 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:58.990 7feb49af2700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb440e52e0 con 0x7feb440f0030 2021-02-10T11:15:59.000 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:58.991 7feb42ffd700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7feb440ef610 0x7feb440efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:59.001 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:58.991 7feb42ffd700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7feb440ef610 0x7feb440efa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:52870/0 (socket says 172.21.2.102:52870) 2021-02-10T11:15:59.001 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:58.991 7feb42ffd700 1 -- 172.21.2.102:0/4060768010 learned_addr learned my addr 172.21.2.102:0/4060768010 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:15:59.001 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:58.991 7feb42ffd700 1 -- 172.21.2.102:0/4060768010 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7feb440f0030 msgr2=0x7feb440f0430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:15:59.001 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:58.991 7feb42ffd700 1 --2- 172.21.2.102:0/4060768010 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7feb440f0030 0x7feb440f0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:15:59.001 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:58.991 7feb42ffd700 1 -- 172.21.2.102:0/4060768010 >> v2:172.21.2.113:3300/0 conn(0x7feb441b0470 msgr2=0x7feb441b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:15:59.001 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:58.991 7feb42ffd700 1 --2- 172.21.2.102:0/4060768010 >> v2:172.21.2.113:3300/0 conn(0x7feb441b0470 0x7feb441b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:15:59.001 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:58.991 7feb42ffd700 1 -- 172.21.2.102:0/4060768010 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb440f10d0 con 0x7feb440ef610 2021-02-10T11:15:59.001 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:58.991 7feb42ffd700 1 --2- 172.21.2.102:0/4060768010 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7feb440ef610 0x7feb440efa10 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 rx=0x7feb34006700 tx=0x7feb34009b80).ready entity=mon.0 client_cookie=6c51a3f0967f2c5f server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:59.001 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:58.991 7feb427fc700 1 -- 172.21.2.102:0/4060768010 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7feb3400c0c0 con 0x7feb440ef610 2021-02-10T11:15:59.001 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:58.991 7feb427fc700 1 -- 172.21.2.102:0/4060768010 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb3400c440 con 0x7feb440ef610 2021-02-10T11:15:59.001 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:58.991 7feb427fc700 1 -- 172.21.2.102:0/4060768010 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7feb3400c960 con 0x7feb440ef610 2021-02-10T11:15:59.001 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:58.991 7feb49af2700 1 -- 172.21.2.102:0/4060768010 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7feb440ef610 msgr2=0x7feb440efa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:59.001 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:58.991 7feb49af2700 1 --2- 172.21.2.102:0/4060768010 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7feb440ef610 0x7feb440efa10 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 rx=0x7feb34006700 tx=0x7feb34009b80).stop 2021-02-10T11:15:59.002 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:58.992 7feb49af2700 1 -- 172.21.2.102:0/4060768010 shutdown_connections 2021-02-10T11:15:59.002 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:58.992 7feb49af2700 1 --2- 172.21.2.102:0/4060768010 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7feb440f0030 0x7feb440f0430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:15:59.002 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:58.992 7feb49af2700 1 --2- 172.21.2.102:0/4060768010 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7feb440ef610 0x7feb440efa10 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:59.002 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:58.992 7feb49af2700 1 --2- 172.21.2.102:0/4060768010 >> v2:172.21.2.113:3300/0 conn(0x7feb441b0470 0x7feb441b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:15:59.002 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:58.992 7feb49af2700 1 -- 172.21.2.102:0/4060768010 >> 172.21.2.102:0/4060768010 conn(0x7feb440ce3e0 msgr2=0x7feb440e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:15:59.002 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:58.992 7feb49af2700 1 -- 172.21.2.102:0/4060768010 shutdown_connections 2021-02-10T11:15:59.002 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:58.992 7feb49af2700 1 -- 172.21.2.102:0/4060768010 wait complete. 2021-02-10T11:15:59.003 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:59.007 7feb49af2700 1 Processor -- start 2021-02-10T11:15:59.004 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:59.007 7feb49af2700 1 -- start start 2021-02-10T11:15:59.004 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:59.007 7feb49af2700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7feb440b20b0 0x7feb440ac970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:59.004 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:59.007 7feb49af2700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7feb440b24b0 0x7feb44191dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:59.004 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:59.007 7feb49af2700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7feb440acf90 0x7feb441923e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:59.004 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:59.008 7feb49af2700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb440b1b30 con 0x7feb440b24b0 2021-02-10T11:15:59.004 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:59.008 7feb49af2700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb441a04e0 con 0x7feb440b20b0 2021-02-10T11:15:59.004 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:59.008 7feb49af2700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb441a06c0 con 0x7feb440acf90 2021-02-10T11:15:59.004 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:59.008 7feb42ffd700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7feb440b24b0 0x7feb44191dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:59.004 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:59.008 7feb42ffd700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7feb440b24b0 0x7feb44191dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:52880/0 (socket says 172.21.2.102:52880) 2021-02-10T11:15:59.004 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:59.008 7feb42ffd700 1 -- 172.21.2.102:0/2993329107 learned_addr learned my addr 172.21.2.102:0/2993329107 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:15:59.004 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:59.008 7feb42ffd700 1 -- 172.21.2.102:0/2993329107 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7feb440acf90 msgr2=0x7feb441923e0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:15:59.004 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:59.008 7feb42ffd700 1 --2- 172.21.2.102:0/2993329107 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7feb440acf90 0x7feb441923e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:15:59.005 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:59.008 7feb42ffd700 1 -- 172.21.2.102:0/2993329107 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7feb440b20b0 msgr2=0x7feb440ac970 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:15:59.005 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:59.008 7feb42ffd700 1 --2- 172.21.2.102:0/2993329107 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7feb440b20b0 0x7feb440ac970 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:15:59.005 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:59.008 7feb42ffd700 1 -- 172.21.2.102:0/2993329107 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb44192be0 con 0x7feb440b24b0 2021-02-10T11:15:59.005 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:59.008 7feb42ffd700 1 --2- 172.21.2.102:0/2993329107 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7feb440b24b0 0x7feb44191dc0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 rx=0x7feb3400b5e0 tx=0x7feb3400b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:59.005 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:59.008 7feb3bfff700 1 -- 172.21.2.102:0/2993329107 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7feb3400bd70 con 0x7feb440b24b0 2021-02-10T11:15:59.005 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:59.008 7feb3bfff700 1 -- 172.21.2.102:0/2993329107 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb34002030 con 0x7feb440b24b0 2021-02-10T11:15:59.005 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:59.008 7feb49af2700 1 -- 172.21.2.102:0/2993329107 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb44192be0 con 0x7feb440b24b0 2021-02-10T11:15:59.005 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:59.008 7feb3bfff700 1 -- 172.21.2.102:0/2993329107 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7feb34002550 con 0x7feb440b24b0 2021-02-10T11:15:59.005 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:59.008 7feb49af2700 1 -- 172.21.2.102:0/2993329107 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb44192be0 con 0x7feb440b24b0 2021-02-10T11:15:59.005 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:59.009 7feb3bfff700 1 -- 172.21.2.102:0/2993329107 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7feb34002210 con 0x7feb440b24b0 2021-02-10T11:15:59.005 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:59.009 7feb3bfff700 1 --2- 172.21.2.102:0/2993329107 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7feb28018730 0x7feb2801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:15:59.005 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:59.009 7feb3bfff700 1 -- 172.21.2.102:0/2993329107 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7feb34032620 con 0x7feb440b24b0 2021-02-10T11:15:59.005 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:59.009 7feb437fe700 1 --2- 172.21.2.102:0/2993329107 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7feb28018730 0x7feb2801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:15:59.005 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:59.009 7feb437fe700 1 --2- 172.21.2.102:0/2993329107 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7feb28018730 0x7feb2801abc0 crc :-1 s=READY pgs=311 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:15:59.034 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:59.041 7feb49af2700 1 -- 172.21.2.102:0/2993329107 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb44105af0 con 0x7feb440b24b0 2021-02-10T11:15:59.036 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:59.043 7feb3bfff700 1 -- 172.21.2.102:0/2993329107 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7feb34002240 con 0x7feb440b24b0 2021-02-10T11:15:59.164 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:59.171 7feb49af2700 1 -- 172.21.2.102:0/2993329107 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344433", "target": ["mgr", ""]}) v1 -- 0x7feb441a04e0 con 0x7feb28018730 2021-02-10T11:15:59.167 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:59.174 7feb3bfff700 1 -- 172.21.2.102:0/2993329107 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7feb2c006960 con 0x7feb28018730 2021-02-10T11:15:59.171 INFO:teuthology.orchestra.run.gibba002.stdout:/volumes/_nogroup/subvolume_0000000000344433/fe3519f0-0935-48ae-9d17-f54a4eb9aea0 2021-02-10T11:15:59.173 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:59.179 7feb49af2700 1 -- 172.21.2.102:0/2993329107 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7feb28018730 msgr2=0x7feb2801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:59.173 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:59.179 7feb49af2700 1 --2- 172.21.2.102:0/2993329107 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7feb28018730 0x7feb2801abc0 crc :-1 s=READY pgs=311 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:59.173 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:59.179 7feb49af2700 1 -- 172.21.2.102:0/2993329107 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7feb440b24b0 msgr2=0x7feb44191dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:15:59.173 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:59.179 7feb49af2700 1 --2- 172.21.2.102:0/2993329107 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7feb440b24b0 0x7feb44191dc0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 rx=0x7feb3400b5e0 tx=0x7feb3400b980).stop 2021-02-10T11:15:59.173 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:59.180 7feb49af2700 1 -- 172.21.2.102:0/2993329107 shutdown_connections 2021-02-10T11:15:59.173 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:59.180 7feb49af2700 1 --2- 172.21.2.102:0/2993329107 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7feb440acf90 0x7feb441923e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:15:59.173 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:59.180 7feb49af2700 1 --2- 172.21.2.102:0/2993329107 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7feb440b24b0 0x7feb44191dc0 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:59.173 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:59.180 7feb49af2700 1 --2- 172.21.2.102:0/2993329107 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7feb28018730 0x7feb2801abc0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:15:59.173 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:59.180 7feb49af2700 1 --2- 172.21.2.102:0/2993329107 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7feb440b20b0 0x7feb440ac970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:15:59.173 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:59.180 7feb49af2700 1 -- 172.21.2.102:0/2993329107 >> 172.21.2.102:0/2993329107 conn(0x7feb440bf4e0 msgr2=0x7feb440be3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:15:59.174 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:59.180 7feb49af2700 1 -- 172.21.2.102:0/2993329107 shutdown_connections 2021-02-10T11:15:59.174 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:15:59.180 7feb49af2700 1 -- 172.21.2.102:0/2993329107 wait complete. 2021-02-10T11:15:59.181 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000344433 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000344433/fe3519f0-0935-48ae-9d17-f54a4eb9aea0 2021-02-10T11:15:59.182 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:15:59.182 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:15:59.182 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:15:59.519 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:15:59.519 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:15:59.519 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344433/fe3519f0-0935-48ae-9d17-f54a4eb9aea0/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:15:59.902 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:15:59.902 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:15:59.902 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0248662 s, 42.2 MB/s 2021-02-10T11:15:59.903 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:15:59.903 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:15:59.903 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:16:00.240 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:16:00.240 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:16:00.241 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344433/fe3519f0-0935-48ae-9d17-f54a4eb9aea0/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:16:00.617 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:16:00.617 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:16:00.617 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0243224 s, 43.1 MB/s 2021-02-10T11:16:00.618 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:16:00.619 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:16:00.619 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:16:00.954 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:16:00.954 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:16:00.954 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344433/fe3519f0-0935-48ae-9d17-f54a4eb9aea0/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:16:01.330 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:16:01.330 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:16:01.330 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0247851 s, 42.3 MB/s 2021-02-10T11:16:01.331 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:16:01.331 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:16:01.332 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:16:01.668 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:16:01.668 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:16:01.668 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344433/fe3519f0-0935-48ae-9d17-f54a4eb9aea0/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:16:02.043 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:16:02.043 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:16:02.043 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0226269 s, 46.3 MB/s 2021-02-10T11:16:02.044 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:16:02.044 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:16:02.044 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:16:02.380 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:16:02.381 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:16:02.381 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344433/fe3519f0-0935-48ae-9d17-f54a4eb9aea0/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:16:02.757 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:16:02.757 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:16:02.757 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0240368 s, 43.6 MB/s 2021-02-10T11:16:02.758 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:16:02.758 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:16:02.758 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:16:03.093 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:16:03.093 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:16:03.093 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344433/fe3519f0-0935-48ae-9d17-f54a4eb9aea0/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:16:03.467 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:16:03.467 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:16:03.467 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0251458 s, 41.7 MB/s 2021-02-10T11:16:03.469 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:16:03.469 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:16:03.469 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:16:03.804 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:16:03.804 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:16:03.804 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344433/fe3519f0-0935-48ae-9d17-f54a4eb9aea0/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:16:04.182 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:16:04.182 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:16:04.182 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.024523 s, 42.8 MB/s 2021-02-10T11:16:04.183 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:16:04.183 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:16:04.183 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:16:04.520 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:16:04.520 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:16:04.521 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344433/fe3519f0-0935-48ae-9d17-f54a4eb9aea0/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:16:04.896 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:16:04.896 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:16:04.896 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.023926 s, 43.8 MB/s 2021-02-10T11:16:04.897 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:16:04.897 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:16:04.897 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:16:05.234 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:16:05.234 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:16:05.235 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344433/fe3519f0-0935-48ae-9d17-f54a4eb9aea0/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:16:05.611 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:16:05.611 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:16:05.611 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0235881 s, 44.5 MB/s 2021-02-10T11:16:05.612 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:16:05.613 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:16:05.613 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:16:05.949 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:16:05.949 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:16:05.949 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344433/fe3519f0-0935-48ae-9d17-f54a4eb9aea0/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:16:06.325 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:16:06.325 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:16:06.325 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.024803 s, 42.3 MB/s 2021-02-10T11:16:06.326 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000344434 2021-02-10T11:16:06.759 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:06.748 7f193c330700 1 Processor -- start 2021-02-10T11:16:06.759 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:06.748 7f193c330700 1 -- start start 2021-02-10T11:16:06.759 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:06.749 7f193c330700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f19341b0470 0x7f19341b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:16:06.759 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:06.749 7f193c330700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f19340f20e0 0x7f19340f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:16:06.759 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:06.749 7f193c330700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19340f0b50 con 0x7f19340ef610 2021-02-10T11:16:06.759 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:06.749 7f193c330700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19340f0820 con 0x7f19340f20e0 2021-02-10T11:16:06.759 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:06.749 7f193c330700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19340f24e0 con 0x7f19341b0470 2021-02-10T11:16:06.759 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:06.749 7f193994b700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f19340ef610 0x7f19340efa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:45108/0 (socket says 172.21.2.102:45108) 2021-02-10T11:16:06.760 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:06.749 7f193994b700 1 -- 172.21.2.102:0/3135046811 learned_addr learned my addr 172.21.2.102:0/3135046811 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:16:06.760 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:06.749 7f193a94d700 1 --2- 172.21.2.102:0/3135046811 >> v2:172.21.2.113:3300/0 conn(0x7f19340f20e0 0x7f19340f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:16:06.760 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:06.749 7f193a14c700 1 --2- 172.21.2.102:0/3135046811 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f19341b0470 0x7f19341b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:16:06.760 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:06.750 7f193914a700 1 -- 172.21.2.102:0/3135046811 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3615857199 0 0) 0x7f19300009b0 con 0x7f19340ef610 2021-02-10T11:16:06.760 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:06.750 7f193914a700 1 -- 172.21.2.102:0/3135046811 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1920002fe0 con 0x7f19340ef610 2021-02-10T11:16:06.760 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:06.750 7f193a14c700 1 -- 172.21.2.102:0/3135046811 >> v2:172.21.2.113:3300/0 conn(0x7f19340f20e0 msgr2=0x7f19340f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:16:06.760 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:06.750 7f193a14c700 1 --2- 172.21.2.102:0/3135046811 >> v2:172.21.2.113:3300/0 conn(0x7f19340f20e0 0x7f19340f0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:06.760 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:06.750 7f193a14c700 1 -- 172.21.2.102:0/3135046811 >> v1:172.21.2.102:6789/0 conn(0x7f19340ef610 legacy=0x7f19340efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:16:06.760 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:06.750 7f193a14c700 1 -- 172.21.2.102:0/3135046811 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19340e52e0 con 0x7f19341b0470 2021-02-10T11:16:06.760 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:06.750 7f193a94d700 1 --2- 172.21.2.102:0/3135046811 >> v2:172.21.2.113:3300/0 conn(0x7f19340f20e0 0x7f19340f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:16:06.760 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:06.750 7f193a14c700 1 --2- 172.21.2.102:0/3135046811 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f19341b0470 0x7f19341b0870 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7f1928006650 tx=0x7f1928009a80).ready entity=mon.2 client_cookie=1b194923c5bceddb server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:16:06.760 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:06.750 7f193914a700 1 -- 172.21.2.102:0/3135046811 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f192800c0c0 con 0x7f19341b0470 2021-02-10T11:16:06.760 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:06.750 7f193914a700 1 -- 172.21.2.102:0/3135046811 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f192800c340 con 0x7f19341b0470 2021-02-10T11:16:06.760 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:06.750 7f193914a700 1 -- 172.21.2.102:0/3135046811 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f192800c860 con 0x7f19341b0470 2021-02-10T11:16:06.761 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:06.750 7f193c330700 1 -- 172.21.2.102:0/3135046811 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f19341b0470 msgr2=0x7f19341b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:16:06.761 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:06.750 7f193c330700 1 --2- 172.21.2.102:0/3135046811 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f19341b0470 0x7f19341b0870 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7f1928006650 tx=0x7f1928009a80).stop 2021-02-10T11:16:06.761 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:06.751 7f193c330700 1 -- 172.21.2.102:0/3135046811 shutdown_connections 2021-02-10T11:16:06.761 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:06.751 7f193c330700 1 --2- 172.21.2.102:0/3135046811 >> v2:172.21.2.113:3300/0 conn(0x7f19340f20e0 0x7f19340f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:06.761 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:06.751 7f193c330700 1 --2- 172.21.2.102:0/3135046811 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f19341b0470 0x7f19341b0870 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:06.761 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:06.751 7f193c330700 1 -- 172.21.2.102:0/3135046811 >> 172.21.2.102:0/3135046811 conn(0x7f19340ce3e0 msgr2=0x7f19340e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:16:06.761 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:06.751 7f193c330700 1 -- 172.21.2.102:0/3135046811 shutdown_connections 2021-02-10T11:16:06.761 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:06.751 7f193c330700 1 -- 172.21.2.102:0/3135046811 wait complete. 2021-02-10T11:16:06.763 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:06.766 7f193c330700 1 Processor -- start 2021-02-10T11:16:06.763 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:06.766 7f193c330700 1 -- start start 2021-02-10T11:16:06.763 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:06.766 7f193c330700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f193409e560 0x7f19340a0130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:16:06.763 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:06.766 7f193c330700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f193409e960 0x7f19340a0750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:16:06.763 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:06.766 7f193c330700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f19340a0d70 0x7f19340a1af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:16:06.763 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:06.766 7f193c330700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19340c38f0 con 0x7f193409e960 2021-02-10T11:16:06.764 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:06.766 7f193c330700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19340a1170 con 0x7f193409e560 2021-02-10T11:16:06.764 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:06.766 7f193c330700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19340a1390 con 0x7f19340a0d70 2021-02-10T11:16:06.764 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:06.767 7f193994b700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f193409e960 0x7f19340a0750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:16:06.764 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:06.767 7f193994b700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f193409e960 0x7f19340a0750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:52890/0 (socket says 172.21.2.102:52890) 2021-02-10T11:16:06.764 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:06.767 7f193994b700 1 -- 172.21.2.102:0/929708351 learned_addr learned my addr 172.21.2.102:0/929708351 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:16:06.764 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:06.767 7f193994b700 1 -- 172.21.2.102:0/929708351 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f19340a0d70 msgr2=0x7f19340a1af0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:16:06.764 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:06.767 7f193994b700 1 --2- 172.21.2.102:0/929708351 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f19340a0d70 0x7f19340a1af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:16:06.764 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:06.767 7f193994b700 1 -- 172.21.2.102:0/929708351 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f193409e560 msgr2=0x7f19340a0130 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:16:06.764 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:06.767 7f193994b700 1 --2- 172.21.2.102:0/929708351 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f193409e560 0x7f19340a0130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:16:06.764 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:06.767 7f193994b700 1 -- 172.21.2.102:0/929708351 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19340a2110 con 0x7f193409e960 2021-02-10T11:16:06.764 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:06.767 7f193994b700 1 --2- 172.21.2.102:0/929708351 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f193409e960 0x7f19340a0750 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 rx=0x7f1930001c60 tx=0x7f193000a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:16:06.764 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:06.767 7f1926ffd700 1 -- 172.21.2.102:0/929708351 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f193000acc0 con 0x7f193409e960 2021-02-10T11:16:06.765 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:06.767 7f1926ffd700 1 -- 172.21.2.102:0/929708351 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1930004140 con 0x7f193409e960 2021-02-10T11:16:06.765 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:06.767 7f1926ffd700 1 -- 172.21.2.102:0/929708351 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f19300045a0 con 0x7f193409e960 2021-02-10T11:16:06.765 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:06.767 7f193c330700 1 -- 172.21.2.102:0/929708351 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19340a2110 con 0x7f193409e960 2021-02-10T11:16:06.765 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:06.767 7f193c330700 1 -- 172.21.2.102:0/929708351 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19340a2110 con 0x7f193409e960 2021-02-10T11:16:06.765 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:06.768 7f1926ffd700 1 -- 172.21.2.102:0/929708351 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1930004320 con 0x7f193409e960 2021-02-10T11:16:06.765 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:06.768 7f1926ffd700 1 --2- 172.21.2.102:0/929708351 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f191c018730 0x7f191c01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:16:06.765 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:06.768 7f1926ffd700 1 -- 172.21.2.102:0/929708351 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f1930032780 con 0x7f193409e960 2021-02-10T11:16:06.765 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:06.768 7f193a14c700 1 --2- 172.21.2.102:0/929708351 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f191c018730 0x7f191c01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:16:06.765 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:06.768 7f193a14c700 1 --2- 172.21.2.102:0/929708351 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f191c018730 0x7f191c01ac10 crc :-1 s=READY pgs=312 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:16:06.792 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:06.799 7f193c330700 1 -- 172.21.2.102:0/929708351 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1934105b30 con 0x7f193409e960 2021-02-10T11:16:06.794 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:06.801 7f1926ffd700 1 -- 172.21.2.102:0/929708351 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f1930000e20 con 0x7f193409e960 2021-02-10T11:16:06.920 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:06.927 7f193c330700 1 -- 172.21.2.102:0/929708351 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344434", "target": ["mgr", ""]}) v1 -- 0x7f19340f0b50 con 0x7f191c018730 2021-02-10T11:16:06.947 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:06.954 7f1926ffd700 1 -- 172.21.2.102:0/929708351 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f1928000d10 con 0x7f191c018730 2021-02-10T11:16:06.954 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:06.960 7f193c330700 1 -- 172.21.2.102:0/929708351 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f191c018730 msgr2=0x7f191c01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:16:06.954 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:06.960 7f193c330700 1 --2- 172.21.2.102:0/929708351 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f191c018730 0x7f191c01ac10 crc :-1 s=READY pgs=312 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:06.954 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:06.960 7f193c330700 1 -- 172.21.2.102:0/929708351 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f193409e960 msgr2=0x7f19340a0750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:16:06.954 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:06.960 7f193c330700 1 --2- 172.21.2.102:0/929708351 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f193409e960 0x7f19340a0750 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 rx=0x7f1930001c60 tx=0x7f193000a8d0).stop 2021-02-10T11:16:06.954 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:06.960 7f193c330700 1 -- 172.21.2.102:0/929708351 shutdown_connections 2021-02-10T11:16:06.954 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:06.960 7f193c330700 1 --2- 172.21.2.102:0/929708351 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f19340a0d70 0x7f19340a1af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:16:06.954 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:06.960 7f193c330700 1 --2- 172.21.2.102:0/929708351 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f193409e960 0x7f19340a0750 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:06.955 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:06.960 7f193c330700 1 --2- 172.21.2.102:0/929708351 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f191c018730 0x7f191c01ac10 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:06.955 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:06.960 7f193c330700 1 --2- 172.21.2.102:0/929708351 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f193409e560 0x7f19340a0130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:16:06.955 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:06.960 7f193c330700 1 -- 172.21.2.102:0/929708351 >> 172.21.2.102:0/929708351 conn(0x7f193409fc10 msgr2=0x7f1934094ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:16:06.955 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:06.960 7f193c330700 1 -- 172.21.2.102:0/929708351 shutdown_connections 2021-02-10T11:16:06.955 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:06.960 7f193c330700 1 -- 172.21.2.102:0/929708351 wait complete. 2021-02-10T11:16:06.962 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000344434 2021-02-10T11:16:07.395 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:07.386 7f47af228700 1 Processor -- start 2021-02-10T11:16:07.396 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:07.386 7f47af228700 1 -- start start 2021-02-10T11:16:07.396 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:07.387 7f47af228700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f47a81b0470 0x7f47a81b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:16:07.396 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:07.387 7f47af228700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f47a80f20e0 0x7f47a80f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:16:07.396 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:07.387 7f47af228700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47a80f0b50 con 0x7f47a80ef610 2021-02-10T11:16:07.396 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:07.387 7f47af228700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47a80f0820 con 0x7f47a81b0470 2021-02-10T11:16:07.396 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:07.387 7f47af228700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47a80f24e0 con 0x7f47a80f20e0 2021-02-10T11:16:07.396 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:07.387 7f47ac843700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f47a80ef610 0x7f47a80efa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:45120/0 (socket says 172.21.2.102:45120) 2021-02-10T11:16:07.396 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:07.387 7f47ac843700 1 -- 172.21.2.102:0/728786562 learned_addr learned my addr 172.21.2.102:0/728786562 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:16:07.396 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:07.387 7f47ad044700 1 --2- 172.21.2.102:0/728786562 >> v2:172.21.2.102:3300/0 conn(0x7f47a81b0470 0x7f47a81b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:16:07.396 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:07.387 7f47ad845700 1 --2- 172.21.2.102:0/728786562 >> v2:172.21.2.113:3300/0 conn(0x7f47a80f20e0 0x7f47a80f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:16:07.396 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:07.387 7f479bfff700 1 -- 172.21.2.102:0/728786562 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 88556214 0 0) 0x7f47a40010b0 con 0x7f47a80ef610 2021-02-10T11:16:07.397 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:07.387 7f479bfff700 1 -- 172.21.2.102:0/728786562 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4794002fe0 con 0x7f47a80ef610 2021-02-10T11:16:07.397 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:07.387 7f47ad044700 1 -- 172.21.2.102:0/728786562 >> v2:172.21.2.113:3300/0 conn(0x7f47a80f20e0 msgr2=0x7f47a80f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:16:07.397 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:07.387 7f47ad044700 1 --2- 172.21.2.102:0/728786562 >> v2:172.21.2.113:3300/0 conn(0x7f47a80f20e0 0x7f47a80f0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:07.397 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:07.387 7f47ad044700 1 -- 172.21.2.102:0/728786562 >> v1:172.21.2.102:6789/0 conn(0x7f47a80ef610 legacy=0x7f47a80efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:16:07.397 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:07.387 7f47ad044700 1 -- 172.21.2.102:0/728786562 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47a80e52e0 con 0x7f47a81b0470 2021-02-10T11:16:07.397 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:07.387 7f47ad044700 1 --2- 172.21.2.102:0/728786562 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f47a81b0470 0x7f47a81b0870 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 rx=0x7f479c006650 tx=0x7f479c009a80).ready entity=mon.0 client_cookie=be48cd6d422d3a87 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:16:07.397 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:07.387 7f479bfff700 1 -- 172.21.2.102:0/728786562 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f479c0009b0 con 0x7f47a81b0470 2021-02-10T11:16:07.397 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:07.387 7f479bfff700 1 -- 172.21.2.102:0/728786562 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f479c00c0c0 con 0x7f47a81b0470 2021-02-10T11:16:07.397 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:07.387 7f479bfff700 1 -- 172.21.2.102:0/728786562 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f479c00c520 con 0x7f47a81b0470 2021-02-10T11:16:07.397 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:07.388 7f47af228700 1 -- 172.21.2.102:0/728786562 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f47a81b0470 msgr2=0x7f47a81b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:16:07.397 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:07.388 7f47af228700 1 --2- 172.21.2.102:0/728786562 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f47a81b0470 0x7f47a81b0870 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 rx=0x7f479c006650 tx=0x7f479c009a80).stop 2021-02-10T11:16:07.397 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:07.388 7f47af228700 1 -- 172.21.2.102:0/728786562 shutdown_connections 2021-02-10T11:16:07.397 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:07.388 7f47af228700 1 --2- 172.21.2.102:0/728786562 >> v2:172.21.2.113:3300/0 conn(0x7f47a80f20e0 0x7f47a80f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:07.398 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:07.388 7f47af228700 1 --2- 172.21.2.102:0/728786562 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f47a81b0470 0x7f47a81b0870 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:07.398 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:07.388 7f47af228700 1 -- 172.21.2.102:0/728786562 >> 172.21.2.102:0/728786562 conn(0x7f47a80ce3e0 msgr2=0x7f47a80e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:16:07.398 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:07.388 7f47af228700 1 -- 172.21.2.102:0/728786562 shutdown_connections 2021-02-10T11:16:07.398 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:07.388 7f47af228700 1 -- 172.21.2.102:0/728786562 wait complete. 2021-02-10T11:16:07.399 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:07.403 7f47af228700 1 Processor -- start 2021-02-10T11:16:07.399 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:07.403 7f47af228700 1 -- start start 2021-02-10T11:16:07.399 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:07.403 7f47af228700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f47a80a0280 0x7f47a80aeff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:16:07.399 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:07.403 7f47af228700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f47a81a4450 0x7f47a81a4850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:16:07.400 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:07.403 7f47af228700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f47a808bf50 0x7f47a808c350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:16:07.400 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:07.403 7f47af228700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47a80a0700 con 0x7f47a80a0280 2021-02-10T11:16:07.400 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:07.403 7f47af228700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47a80af610 con 0x7f47a808bf50 2021-02-10T11:16:07.400 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:07.403 7f47af228700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47a808c970 con 0x7f47a81a4450 2021-02-10T11:16:07.400 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:07.404 7f47ad044700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f47a80a0280 0x7f47a80aeff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:16:07.400 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:07.404 7f47ad044700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f47a80a0280 0x7f47a80aeff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:52904/0 (socket says 172.21.2.102:52904) 2021-02-10T11:16:07.400 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:07.404 7f47ad044700 1 -- 172.21.2.102:0/934944244 learned_addr learned my addr 172.21.2.102:0/934944244 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:16:07.400 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:07.404 7f47ad845700 1 --2- 172.21.2.102:0/934944244 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f47a808bf50 0x7f47a808c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:16:07.400 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:07.404 7f47ac843700 1 --2- 172.21.2.102:0/934944244 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f47a81a4450 0x7f47a81a4850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:16:07.400 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:07.404 7f47ad044700 1 -- 172.21.2.102:0/934944244 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f47a81a4450 msgr2=0x7f47a81a4850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:16:07.400 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:07.404 7f47ad044700 1 --2- 172.21.2.102:0/934944244 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f47a81a4450 0x7f47a81a4850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:07.400 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:07.404 7f47ad044700 1 -- 172.21.2.102:0/934944244 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f47a808bf50 msgr2=0x7f47a808c350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:16:07.400 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:07.404 7f47ad044700 1 --2- 172.21.2.102:0/934944244 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f47a808bf50 0x7f47a808c350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:07.401 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:07.404 7f47ad044700 1 -- 172.21.2.102:0/934944244 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47a808cb90 con 0x7f47a80a0280 2021-02-10T11:16:07.401 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:07.404 7f47ad044700 1 --2- 172.21.2.102:0/934944244 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f47a80a0280 0x7f47a80aeff0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 rx=0x7f479c009630 tx=0x7f479c00b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:16:07.401 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:07.404 7f4799ffb700 1 -- 172.21.2.102:0/934944244 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f479c0093a0 con 0x7f47a80a0280 2021-02-10T11:16:07.401 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:07.404 7f4799ffb700 1 -- 172.21.2.102:0/934944244 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f479c00bd70 con 0x7f47a80a0280 2021-02-10T11:16:07.401 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:07.404 7f47af228700 1 -- 172.21.2.102:0/934944244 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47a808cb90 con 0x7f47a80a0280 2021-02-10T11:16:07.401 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:07.404 7f47af228700 1 -- 172.21.2.102:0/934944244 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47a808cb90 con 0x7f47a80a0280 2021-02-10T11:16:07.401 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:07.405 7f4799ffb700 1 -- 172.21.2.102:0/934944244 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f479c005560 con 0x7f47a80a0280 2021-02-10T11:16:07.401 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:07.405 7f4799ffb700 1 -- 172.21.2.102:0/934944244 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f479c0052e0 con 0x7f47a80a0280 2021-02-10T11:16:07.401 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:07.405 7f4799ffb700 1 --2- 172.21.2.102:0/934944244 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f47880186c0 0x7f478801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:16:07.401 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:07.405 7f4799ffb700 1 -- 172.21.2.102:0/934944244 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f479c0325e0 con 0x7f47a80a0280 2021-02-10T11:16:07.401 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:07.405 7f47ac843700 1 --2- 172.21.2.102:0/934944244 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f47880186c0 0x7f478801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:16:07.401 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:07.406 7f47ac843700 1 --2- 172.21.2.102:0/934944244 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f47880186c0 0x7f478801abc0 crc :-1 s=READY pgs=313 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:16:07.430 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:07.436 7f47af228700 1 -- 172.21.2.102:0/934944244 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47a8105ae0 con 0x7f47a80a0280 2021-02-10T11:16:07.432 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:07.439 7f4799ffb700 1 -- 172.21.2.102:0/934944244 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f479c030d90 con 0x7f47a80a0280 2021-02-10T11:16:07.561 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:07.568 7f47af228700 1 -- 172.21.2.102:0/934944244 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344434", "target": ["mgr", ""]}) v1 -- 0x7f47a80af610 con 0x7f47880186c0 2021-02-10T11:16:07.564 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:07.571 7f4799ffb700 1 -- 172.21.2.102:0/934944244 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f47a40014b0 con 0x7f47880186c0 2021-02-10T11:16:07.564 INFO:teuthology.orchestra.run.gibba002.stdout:/volumes/_nogroup/subvolume_0000000000344434/00bc553c-3b9b-4e82-8120-410a7f996524 2021-02-10T11:16:07.565 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:07.572 7f47af228700 1 -- 172.21.2.102:0/934944244 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f47880186c0 msgr2=0x7f478801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:16:07.565 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:07.572 7f47af228700 1 --2- 172.21.2.102:0/934944244 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f47880186c0 0x7f478801abc0 crc :-1 s=READY pgs=313 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:07.565 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:07.572 7f47af228700 1 -- 172.21.2.102:0/934944244 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f47a80a0280 msgr2=0x7f47a80aeff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:16:07.565 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:07.572 7f47af228700 1 --2- 172.21.2.102:0/934944244 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f47a80a0280 0x7f47a80aeff0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 rx=0x7f479c009630 tx=0x7f479c00b980).stop 2021-02-10T11:16:07.566 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:07.572 7f47af228700 1 -- 172.21.2.102:0/934944244 shutdown_connections 2021-02-10T11:16:07.566 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:07.572 7f47af228700 1 --2- 172.21.2.102:0/934944244 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f47880186c0 0x7f478801abc0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:07.566 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:07.572 7f47af228700 1 --2- 172.21.2.102:0/934944244 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f47a808bf50 0x7f47a808c350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:07.566 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:07.572 7f47af228700 1 --2- 172.21.2.102:0/934944244 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f47a81a4450 0x7f47a81a4850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:07.566 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:07.572 7f47af228700 1 --2- 172.21.2.102:0/934944244 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f47a80a0280 0x7f47a80aeff0 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:07.566 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:07.572 7f47af228700 1 -- 172.21.2.102:0/934944244 >> 172.21.2.102:0/934944244 conn(0x7f47a80bbf60 msgr2=0x7f47a8105d10 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:16:07.566 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:07.572 7f47af228700 1 -- 172.21.2.102:0/934944244 shutdown_connections 2021-02-10T11:16:07.566 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:07.572 7f47af228700 1 -- 172.21.2.102:0/934944244 wait complete. 2021-02-10T11:16:07.573 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000344434 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000344434/00bc553c-3b9b-4e82-8120-410a7f996524 2021-02-10T11:16:07.573 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:16:07.573 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:16:07.574 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:16:07.910 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:16:07.910 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:16:07.911 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344434/00bc553c-3b9b-4e82-8120-410a7f996524/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:16:08.292 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:16:08.293 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:16:08.293 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0220198 s, 47.6 MB/s 2021-02-10T11:16:08.294 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:16:08.294 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:16:08.294 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:16:08.630 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:16:08.630 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:16:08.630 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344434/00bc553c-3b9b-4e82-8120-410a7f996524/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:16:09.002 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:16:09.003 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:16:09.003 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0230505 s, 45.5 MB/s 2021-02-10T11:16:09.004 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:16:09.004 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:16:09.004 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:16:09.340 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:16:09.340 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:16:09.340 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344434/00bc553c-3b9b-4e82-8120-410a7f996524/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:16:09.717 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:16:09.718 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:16:09.718 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0246014 s, 42.6 MB/s 2021-02-10T11:16:09.719 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:16:09.719 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:16:09.719 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:16:10.054 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:16:10.055 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:16:10.055 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344434/00bc553c-3b9b-4e82-8120-410a7f996524/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:16:10.431 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:16:10.432 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:16:10.432 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0240674 s, 43.6 MB/s 2021-02-10T11:16:10.433 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:16:10.434 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:16:10.434 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:16:10.771 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:16:10.771 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:16:10.771 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344434/00bc553c-3b9b-4e82-8120-410a7f996524/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:16:11.147 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:16:11.148 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:16:11.148 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0245984 s, 42.6 MB/s 2021-02-10T11:16:11.149 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:16:11.150 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:16:11.150 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:16:11.487 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:16:11.487 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:16:11.487 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344434/00bc553c-3b9b-4e82-8120-410a7f996524/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:16:11.864 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:16:14.012 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:16:14.012 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.02475 s, 42.4 MB/s 2021-02-10T11:16:14.013 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:16:14.013 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:16:14.014 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:16:14.354 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:16:14.354 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:16:14.354 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344434/00bc553c-3b9b-4e82-8120-410a7f996524/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:16:14.730 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:16:14.730 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:16:14.731 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0243551 s, 43.1 MB/s 2021-02-10T11:16:14.732 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:16:14.732 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:16:14.732 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:16:15.073 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:16:15.073 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:16:15.074 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344434/00bc553c-3b9b-4e82-8120-410a7f996524/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:16:15.450 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:16:15.451 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:16:15.451 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0237447 s, 44.2 MB/s 2021-02-10T11:16:15.452 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:16:15.452 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:16:15.452 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:16:15.788 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:16:15.788 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:16:15.788 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344434/00bc553c-3b9b-4e82-8120-410a7f996524/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:16:16.164 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:16:16.164 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:16:16.164 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0233797 s, 44.8 MB/s 2021-02-10T11:16:16.165 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:16:16.165 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:16:16.165 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:16:16.501 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:16:16.502 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:16:16.502 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344434/00bc553c-3b9b-4e82-8120-410a7f996524/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:16:16.878 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:16:16.878 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:16:16.878 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0247242 s, 42.4 MB/s 2021-02-10T11:16:16.879 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000344435 2021-02-10T11:16:17.313 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:17.301 7f4e39146700 1 Processor -- start 2021-02-10T11:16:17.313 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:17.301 7f4e39146700 1 -- start start 2021-02-10T11:16:17.313 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:17.302 7f4e39146700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f4e341b0870 0x7f4e340f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:16:17.313 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:17.302 7f4e39146700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e341b0c70 con 0x7f4e341b0470 2021-02-10T11:16:17.313 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:17.302 7f4e39146700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e340f0b50 con 0x7f4e340efe00 2021-02-10T11:16:17.314 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:17.302 7f4e39146700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e340e5ea0 con 0x7f4e341b0870 2021-02-10T11:16:17.314 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:17.302 7f4e32d9d700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f4e341b0470 0x7f4e340ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:45134/0 (socket says 172.21.2.102:45134) 2021-02-10T11:16:17.314 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:17.302 7f4e32d9d700 1 -- 172.21.2.102:0/2937778440 learned_addr learned my addr 172.21.2.102:0/2937778440 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:16:17.314 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:17.302 7f4e31d9b700 1 -- 172.21.2.102:0/2937778440 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4074666270 0 0) 0x7f4e1c001180 con 0x7f4e341b0470 2021-02-10T11:16:17.314 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:17.302 7f4e31d9b700 1 -- 172.21.2.102:0/2937778440 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e20002fe0 con 0x7f4e341b0470 2021-02-10T11:16:17.314 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:17.303 7f4e3259c700 1 --2- 172.21.2.102:0/2937778440 >> v2:172.21.2.113:3300/0 conn(0x7f4e341b0870 0x7f4e340f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:16:17.314 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:17.303 7f4e31d9b700 1 -- 172.21.2.102:0/2937778440 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3167812419 0 0) 0x7f4e1c001180 con 0x7f4e341b0470 2021-02-10T11:16:17.314 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:17.303 7f4e31d9b700 1 -- 172.21.2.102:0/2937778440 >> v2:172.21.2.113:3300/0 conn(0x7f4e341b0870 msgr2=0x7f4e340f20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:16:17.314 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:17.303 7f4e31d9b700 1 --2- 172.21.2.102:0/2937778440 >> v2:172.21.2.113:3300/0 conn(0x7f4e341b0870 0x7f4e340f20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:17.314 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:17.303 7f4e31d9b700 1 -- 172.21.2.102:0/2937778440 >> v1:172.21.2.113:6789/0 conn(0x7f4e340efe00 legacy=0x7f4e340f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:16:17.314 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:17.303 7f4e31d9b700 1 -- 172.21.2.102:0/2937778440 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e340e52e0 con 0x7f4e341b0470 2021-02-10T11:16:17.314 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:17.303 7f4e31d9b700 1 -- 172.21.2.102:0/2937778440 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7f4e1c001680 con 0x7f4e341b0470 2021-02-10T11:16:17.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:17.303 7f4e31d9b700 1 -- 172.21.2.102:0/2937778440 >> v1:172.21.2.102:6789/0 conn(0x7f4e341b0470 legacy=0x7f4e340ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:16:17.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:17.303 7f4e31d9b700 1 --2- 172.21.2.102:0/2937778440 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4e20003b10 0x7f4e20005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:16:17.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:17.303 7f4e31d9b700 1 --2- 172.21.2.102:0/2937778440 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4e200067c0 0x7f4e20008cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:16:17.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:17.303 7f4e31d9b700 1 --2- 172.21.2.102:0/2937778440 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4e200095c0 0x7f4e2000bac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:16:17.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:17.303 7f4e31d9b700 1 -- 172.21.2.102:0/2937778440 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e2000c310 con 0x7f4e200095c0 2021-02-10T11:16:17.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:17.303 7f4e31d9b700 1 -- 172.21.2.102:0/2937778440 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e2000c530 con 0x7f4e20003b10 2021-02-10T11:16:17.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:17.303 7f4e31d9b700 1 -- 172.21.2.102:0/2937778440 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e2000c750 con 0x7f4e200067c0 2021-02-10T11:16:17.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:17.303 7f4e32d9d700 1 --2- 172.21.2.102:0/2937778440 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4e20003b10 0x7f4e20005fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:16:17.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:17.303 7f4e3359e700 1 --2- 172.21.2.102:0/2937778440 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4e200095c0 0x7f4e2000bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:16:17.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:17.303 7f4e3259c700 1 --2- 172.21.2.102:0/2937778440 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4e200067c0 0x7f4e20008cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:16:17.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:17.303 7f4e3359e700 1 -- 172.21.2.102:0/2937778440 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4e200067c0 msgr2=0x7f4e20008cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:16:17.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:17.303 7f4e3359e700 1 --2- 172.21.2.102:0/2937778440 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4e200067c0 0x7f4e20008cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:17.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:17.303 7f4e3359e700 1 -- 172.21.2.102:0/2937778440 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4e20003b10 msgr2=0x7f4e20005fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:16:17.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:17.303 7f4e3359e700 1 --2- 172.21.2.102:0/2937778440 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4e20003b10 0x7f4e20005fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:17.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:17.303 7f4e3359e700 1 -- 172.21.2.102:0/2937778440 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e2000c9f0 con 0x7f4e200095c0 2021-02-10T11:16:17.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:17.304 7f4e3359e700 1 -- 172.21.2.102:0/2937778440 reap_dead start 2021-02-10T11:16:17.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:17.304 7f4e3359e700 1 --2- 172.21.2.102:0/2937778440 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4e200095c0 0x7f4e2000bac0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 rx=0x7f4e2c04ae40 tx=0x7f4e2c04de20).ready entity=mon.0 client_cookie=129dad4d1142d401 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:16:17.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:17.304 7f4e31d9b700 1 -- 172.21.2.102:0/2937778440 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4e2c003210 con 0x7f4e200095c0 2021-02-10T11:16:17.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:17.304 7f4e31d9b700 1 -- 172.21.2.102:0/2937778440 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e2c001400 con 0x7f4e200095c0 2021-02-10T11:16:17.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:17.304 7f4e31d9b700 1 -- 172.21.2.102:0/2937778440 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4e2c0018b0 con 0x7f4e200095c0 2021-02-10T11:16:17.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:17.304 7f4e39146700 1 -- 172.21.2.102:0/2937778440 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4e200095c0 msgr2=0x7f4e2000bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:16:17.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:17.304 7f4e39146700 1 --2- 172.21.2.102:0/2937778440 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4e200095c0 0x7f4e2000bac0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 rx=0x7f4e2c04ae40 tx=0x7f4e2c04de20).stop 2021-02-10T11:16:17.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:17.304 7f4e39146700 1 -- 172.21.2.102:0/2937778440 shutdown_connections 2021-02-10T11:16:17.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:17.304 7f4e39146700 1 --2- 172.21.2.102:0/2937778440 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4e200095c0 0x7f4e2000bac0 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:17.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:17.304 7f4e39146700 1 -- 172.21.2.102:0/2937778440 >> 172.21.2.102:0/2937778440 conn(0x7f4e340ce3e0 msgr2=0x7f4e340e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:16:17.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:17.305 7f4e39146700 1 -- 172.21.2.102:0/2937778440 shutdown_connections 2021-02-10T11:16:17.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:17.305 7f4e39146700 1 -- 172.21.2.102:0/2937778440 wait complete. 2021-02-10T11:16:17.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:17.319 7f4e39146700 1 Processor -- start 2021-02-10T11:16:17.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:17.320 7f4e39146700 1 -- start start 2021-02-10T11:16:17.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:17.320 7f4e39146700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4e34106d30 0x7f4e34107130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:16:17.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:17.320 7f4e39146700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4e340c29f0 0x7f4e340c2df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:16:17.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:17.320 7f4e39146700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4e340c3410 0x7f4e34096d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:16:17.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:17.320 7f4e39146700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e340967e0 con 0x7f4e340c3410 2021-02-10T11:16:17.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:17.320 7f4e39146700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e34107750 con 0x7f4e340c29f0 2021-02-10T11:16:17.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:17.320 7f4e39146700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e34107970 con 0x7f4e34106d30 2021-02-10T11:16:17.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:17.320 7f4e3359e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4e340c3410 0x7f4e34096d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:16:17.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:17.320 7f4e3359e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4e340c3410 0x7f4e34096d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:52926/0 (socket says 172.21.2.102:52926) 2021-02-10T11:16:17.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:17.320 7f4e3359e700 1 -- 172.21.2.102:0/3064472325 learned_addr learned my addr 172.21.2.102:0/3064472325 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:16:17.318 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:17.320 7f4e3259c700 1 --2- 172.21.2.102:0/3064472325 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4e340c29f0 0x7f4e340c2df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:16:17.319 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:17.320 7f4e32d9d700 1 --2- 172.21.2.102:0/3064472325 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4e34106d30 0x7f4e34107130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:16:17.319 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:17.321 7f4e3359e700 1 -- 172.21.2.102:0/3064472325 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4e34106d30 msgr2=0x7f4e34107130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:16:17.319 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:17.321 7f4e3359e700 1 --2- 172.21.2.102:0/3064472325 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4e34106d30 0x7f4e34107130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:17.319 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:17.321 7f4e3359e700 1 -- 172.21.2.102:0/3064472325 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4e340c29f0 msgr2=0x7f4e340c2df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:16:17.319 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:17.321 7f4e3359e700 1 --2- 172.21.2.102:0/3064472325 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4e340c29f0 0x7f4e340c2df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:17.319 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:17.321 7f4e3359e700 1 -- 172.21.2.102:0/3064472325 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e340c3810 con 0x7f4e340c3410 2021-02-10T11:16:17.319 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:17.321 7f4e3359e700 1 --2- 172.21.2.102:0/3064472325 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4e340c3410 0x7f4e34096d20 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 rx=0x7f4e2c04d950 tx=0x7f4e2c04fc70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:16:17.319 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:17.321 7f4e2b7fe700 1 -- 172.21.2.102:0/3064472325 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4e2c0011a0 con 0x7f4e340c3410 2021-02-10T11:16:17.319 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:17.321 7f4e2b7fe700 1 -- 172.21.2.102:0/3064472325 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e2c0014a0 con 0x7f4e340c3410 2021-02-10T11:16:17.319 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:17.321 7f4e2b7fe700 1 -- 172.21.2.102:0/3064472325 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4e2c001980 con 0x7f4e340c3410 2021-02-10T11:16:17.319 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:17.321 7f4e39146700 1 -- 172.21.2.102:0/3064472325 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e340c3810 con 0x7f4e340c3410 2021-02-10T11:16:17.319 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:17.321 7f4e39146700 1 -- 172.21.2.102:0/3064472325 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e340c3810 con 0x7f4e340c3410 2021-02-10T11:16:17.320 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:17.322 7f4e2b7fe700 1 -- 172.21.2.102:0/3064472325 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4e2c04d3d0 con 0x7f4e340c3410 2021-02-10T11:16:17.320 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:17.322 7f4e2b7fe700 1 --2- 172.21.2.102:0/3064472325 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4e180186c0 0x7f4e1801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:16:17.320 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:17.322 7f4e2b7fe700 1 -- 172.21.2.102:0/3064472325 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f4e2c07ad70 con 0x7f4e340c3410 2021-02-10T11:16:17.320 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:17.322 7f4e32d9d700 1 --2- 172.21.2.102:0/3064472325 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4e180186c0 0x7f4e1801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:16:17.320 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:17.322 7f4e32d9d700 1 --2- 172.21.2.102:0/3064472325 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4e180186c0 0x7f4e1801abc0 crc :-1 s=READY pgs=314 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:16:17.345 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:17.352 7f4e39146700 1 -- 172.21.2.102:0/3064472325 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e340967e0 con 0x7f4e340c3410 2021-02-10T11:16:17.347 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:17.354 7f4e2b7fe700 1 -- 172.21.2.102:0/3064472325 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f4e2c04fdb0 con 0x7f4e340c3410 2021-02-10T11:16:17.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:17.482 7f4e39146700 1 -- 172.21.2.102:0/3064472325 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344435", "target": ["mgr", ""]}) v1 -- 0x7f4e340967e0 con 0x7f4e180186c0 2021-02-10T11:16:17.501 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:17.507 7f4e2b7fe700 1 -- 172.21.2.102:0/3064472325 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f4e1c000960 con 0x7f4e180186c0 2021-02-10T11:16:17.508 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:17.515 7f4e39146700 1 -- 172.21.2.102:0/3064472325 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4e180186c0 msgr2=0x7f4e1801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:16:17.508 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:17.515 7f4e39146700 1 --2- 172.21.2.102:0/3064472325 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4e180186c0 0x7f4e1801abc0 crc :-1 s=READY pgs=314 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:17.508 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:17.515 7f4e39146700 1 -- 172.21.2.102:0/3064472325 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4e340c3410 msgr2=0x7f4e34096d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:16:17.509 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:17.515 7f4e39146700 1 --2- 172.21.2.102:0/3064472325 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4e340c3410 0x7f4e34096d20 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 rx=0x7f4e2c04d950 tx=0x7f4e2c04fc70).stop 2021-02-10T11:16:17.509 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:17.515 7f4e39146700 1 -- 172.21.2.102:0/3064472325 shutdown_connections 2021-02-10T11:16:17.509 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:17.515 7f4e39146700 1 --2- 172.21.2.102:0/3064472325 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4e340c3410 0x7f4e34096d20 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:17.509 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:17.515 7f4e39146700 1 --2- 172.21.2.102:0/3064472325 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4e180186c0 0x7f4e1801abc0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:17.509 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:17.515 7f4e39146700 1 --2- 172.21.2.102:0/3064472325 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4e340c29f0 0x7f4e340c2df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:17.509 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:17.515 7f4e39146700 1 --2- 172.21.2.102:0/3064472325 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4e34106d30 0x7f4e34107130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:17.509 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:17.515 7f4e39146700 1 -- 172.21.2.102:0/3064472325 >> 172.21.2.102:0/3064472325 conn(0x7f4e340c40e0 msgr2=0x7f4e340bf620 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:16:17.509 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:17.515 7f4e39146700 1 -- 172.21.2.102:0/3064472325 shutdown_connections 2021-02-10T11:16:17.509 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:17.515 7f4e39146700 1 -- 172.21.2.102:0/3064472325 wait complete. 2021-02-10T11:16:17.517 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000344435 2021-02-10T11:16:17.948 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:17.939 7fdbfe534700 1 Processor -- start 2021-02-10T11:16:17.948 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:17.939 7fdbfe534700 1 -- start start 2021-02-10T11:16:17.948 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:17.940 7fdbfe534700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdbf81b0470 0x7fdbf81b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:16:17.949 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:17.940 7fdbfe534700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fdbf80ef610 0x7fdbf80efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:16:17.949 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:17.940 7fdbfe534700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbf80f0b50 con 0x7fdbf80f0030 2021-02-10T11:16:17.949 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:17.940 7fdbfe534700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdbf80e52e0 con 0x7fdbf80ef610 2021-02-10T11:16:17.949 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:17.940 7fdbfe534700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbf80e5490 con 0x7fdbf81b0470 2021-02-10T11:16:17.949 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:17.940 7fdbf77fe700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fdbf80ef610 0x7fdbf80efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:16:17.949 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:17.940 7fdbf77fe700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fdbf80ef610 0x7fdbf80efa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:52934/0 (socket says 172.21.2.102:52934) 2021-02-10T11:16:17.949 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:17.940 7fdbf77fe700 1 -- 172.21.2.102:0/2250878218 learned_addr learned my addr 172.21.2.102:0/2250878218 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:16:17.949 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:17.940 7fdbf77fe700 1 -- 172.21.2.102:0/2250878218 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdbf81b0470 msgr2=0x7fdbf81b0870 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:16:17.949 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:17.940 7fdbf77fe700 1 --2- 172.21.2.102:0/2250878218 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdbf81b0470 0x7fdbf81b0870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:16:17.949 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:17.940 7fdbf77fe700 1 -- 172.21.2.102:0/2250878218 >> v1:172.21.2.113:6789/0 conn(0x7fdbf80f0030 legacy=0x7fdbf80f10d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:16:17.949 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:17.940 7fdbf77fe700 1 -- 172.21.2.102:0/2250878218 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbf80f0430 con 0x7fdbf80ef610 2021-02-10T11:16:17.950 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:17.940 7fdbf77fe700 1 --2- 172.21.2.102:0/2250878218 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdbf80ef610 0x7fdbf80efa10 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 rx=0x7fdbec006700 tx=0x7fdbec009b80).ready entity=mon.0 client_cookie=ae13594bb84c3212 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:16:17.950 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:17.941 7fdbf6ffd700 1 -- 172.21.2.102:0/2250878218 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdbec00c0c0 con 0x7fdbf80ef610 2021-02-10T11:16:17.950 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:17.941 7fdbf6ffd700 1 -- 172.21.2.102:0/2250878218 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbec00c340 con 0x7fdbf80ef610 2021-02-10T11:16:17.950 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:17.941 7fdbf6ffd700 1 -- 172.21.2.102:0/2250878218 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdbec00c860 con 0x7fdbf80ef610 2021-02-10T11:16:17.950 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:17.941 7fdbfe534700 1 -- 172.21.2.102:0/2250878218 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdbf80ef610 msgr2=0x7fdbf80efa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:16:17.950 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:17.941 7fdbfe534700 1 --2- 172.21.2.102:0/2250878218 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdbf80ef610 0x7fdbf80efa10 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 rx=0x7fdbec006700 tx=0x7fdbec009b80).stop 2021-02-10T11:16:17.950 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:17.941 7fdbfe534700 1 -- 172.21.2.102:0/2250878218 shutdown_connections 2021-02-10T11:16:17.950 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:17.941 7fdbfe534700 1 --2- 172.21.2.102:0/2250878218 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdbf80ef610 0x7fdbf80efa10 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:17.950 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:17.941 7fdbfe534700 1 --2- 172.21.2.102:0/2250878218 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdbf81b0470 0x7fdbf81b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:16:17.950 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:17.941 7fdbfe534700 1 -- 172.21.2.102:0/2250878218 >> 172.21.2.102:0/2250878218 conn(0x7fdbf80ce3e0 msgr2=0x7fdbf80e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:16:17.950 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:17.941 7fdbfe534700 1 -- 172.21.2.102:0/2250878218 shutdown_connections 2021-02-10T11:16:17.951 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:17.941 7fdbfe534700 1 -- 172.21.2.102:0/2250878218 wait complete. 2021-02-10T11:16:17.952 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:17.956 7fdbfe534700 1 Processor -- start 2021-02-10T11:16:17.952 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:17.956 7fdbfe534700 1 -- start start 2021-02-10T11:16:17.952 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:17.956 7fdbfe534700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdbf80a6c90 0x7fdbf81924c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:16:17.953 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:17.956 7fdbfe534700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdbf8192ae0 0x7fdbf8192ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:16:17.953 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:17.956 7fdbfe534700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fdbf80919e0 0x7fdbf8091de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:16:17.953 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:17.956 7fdbfe534700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbf819d2e0 con 0x7fdbf80a6c90 2021-02-10T11:16:17.953 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:17.956 7fdbfe534700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbf80914d0 con 0x7fdbf80919e0 2021-02-10T11:16:17.953 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:17.956 7fdbfe534700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbf80a6820 con 0x7fdbf8192ae0 2021-02-10T11:16:17.953 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:17.956 7fdbf7fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdbf80a6c90 0x7fdbf81924c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:16:17.953 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:17.956 7fdbf7fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdbf80a6c90 0x7fdbf81924c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:52936/0 (socket says 172.21.2.102:52936) 2021-02-10T11:16:17.953 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:17.956 7fdbf7fff700 1 -- 172.21.2.102:0/3912744630 learned_addr learned my addr 172.21.2.102:0/3912744630 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:16:17.953 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:17.957 7fdbf7fff700 1 -- 172.21.2.102:0/3912744630 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdbf8192ae0 msgr2=0x7fdbf8192ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:16:17.953 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:17.957 7fdbf7fff700 1 --2- 172.21.2.102:0/3912744630 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdbf8192ae0 0x7fdbf8192ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:16:17.953 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:17.957 7fdbf7fff700 1 -- 172.21.2.102:0/3912744630 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fdbf80919e0 msgr2=0x7fdbf8091de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:16:17.953 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:17.957 7fdbf7fff700 1 --2- 172.21.2.102:0/3912744630 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fdbf80919e0 0x7fdbf8091de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:16:17.953 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:17.957 7fdbf7fff700 1 -- 172.21.2.102:0/3912744630 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbf8092400 con 0x7fdbf80a6c90 2021-02-10T11:16:17.954 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:17.957 7fdbf7fff700 1 --2- 172.21.2.102:0/3912744630 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdbf80a6c90 0x7fdbf81924c0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 rx=0x7fdbe800b5c0 tx=0x7fdbe800b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:16:17.954 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:17.957 7fdbf4ff9700 1 -- 172.21.2.102:0/3912744630 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdbe8006b80 con 0x7fdbf80a6c90 2021-02-10T11:16:17.954 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:17.957 7fdbf4ff9700 1 -- 172.21.2.102:0/3912744630 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbe800bd50 con 0x7fdbf80a6c90 2021-02-10T11:16:17.954 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:17.957 7fdbf4ff9700 1 -- 172.21.2.102:0/3912744630 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdbe80054d0 con 0x7fdbf80a6c90 2021-02-10T11:16:17.954 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:17.957 7fdbfe534700 1 -- 172.21.2.102:0/3912744630 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbf8092400 con 0x7fdbf80a6c90 2021-02-10T11:16:17.954 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:17.957 7fdbfe534700 1 -- 172.21.2.102:0/3912744630 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbf8092990 con 0x7fdbf80a6c90 2021-02-10T11:16:17.954 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:17.958 7fdbf4ff9700 1 -- 172.21.2.102:0/3912744630 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fdbe8006030 con 0x7fdbf80a6c90 2021-02-10T11:16:17.954 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:17.958 7fdbf4ff9700 1 --2- 172.21.2.102:0/3912744630 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fdbd8018730 0x7fdbd801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:16:17.954 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:17.958 7fdbf4ff9700 1 -- 172.21.2.102:0/3912744630 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fdbe8032640 con 0x7fdbf80a6c90 2021-02-10T11:16:17.954 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:17.958 7fdbf77fe700 1 --2- 172.21.2.102:0/3912744630 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fdbd8018730 0x7fdbd801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:16:17.954 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:17.958 7fdbf77fe700 1 --2- 172.21.2.102:0/3912744630 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fdbd8018730 0x7fdbd801abc0 crc :-1 s=READY pgs=315 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:16:17.983 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:17.990 7fdbfe534700 1 -- 172.21.2.102:0/3912744630 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbf80e52e0 con 0x7fdbf80a6c90 2021-02-10T11:16:17.985 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:17.993 7fdbf4ff9700 1 -- 172.21.2.102:0/3912744630 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fdbe8000940 con 0x7fdbf80a6c90 2021-02-10T11:16:18.112 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:18.119 7fdbfe534700 1 -- 172.21.2.102:0/3912744630 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344435", "target": ["mgr", ""]}) v1 -- 0x7fdbf8092990 con 0x7fdbd8018730 2021-02-10T11:16:18.115 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:18.122 7fdbf4ff9700 1 -- 172.21.2.102:0/3912744630 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7fdbec000970 con 0x7fdbd8018730 2021-02-10T11:16:18.116 INFO:teuthology.orchestra.run.gibba002.stdout:/volumes/_nogroup/subvolume_0000000000344435/f8e578a1-064d-44f5-b82e-9bee9f6f19d1 2021-02-10T11:16:18.117 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:18.123 7fdbfe534700 1 -- 172.21.2.102:0/3912744630 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fdbd8018730 msgr2=0x7fdbd801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:16:18.117 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:18.123 7fdbfe534700 1 --2- 172.21.2.102:0/3912744630 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fdbd8018730 0x7fdbd801abc0 crc :-1 s=READY pgs=315 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:18.117 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:18.123 7fdbfe534700 1 -- 172.21.2.102:0/3912744630 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdbf80a6c90 msgr2=0x7fdbf81924c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:16:18.118 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:18.123 7fdbfe534700 1 --2- 172.21.2.102:0/3912744630 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdbf80a6c90 0x7fdbf81924c0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 rx=0x7fdbe800b5c0 tx=0x7fdbe800b960).stop 2021-02-10T11:16:18.118 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:18.124 7fdbfe534700 1 -- 172.21.2.102:0/3912744630 shutdown_connections 2021-02-10T11:16:18.118 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:18.124 7fdbfe534700 1 --2- 172.21.2.102:0/3912744630 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fdbd8018730 0x7fdbd801abc0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:18.118 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:18.124 7fdbfe534700 1 --2- 172.21.2.102:0/3912744630 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fdbf80919e0 0x7fdbf8091de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:16:18.118 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:18.124 7fdbfe534700 1 --2- 172.21.2.102:0/3912744630 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdbf8192ae0 0x7fdbf8192ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:16:18.118 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:18.124 7fdbfe534700 1 --2- 172.21.2.102:0/3912744630 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdbf80a6c90 0x7fdbf81924c0 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:18.118 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:18.124 7fdbfe534700 1 -- 172.21.2.102:0/3912744630 >> 172.21.2.102:0/3912744630 conn(0x7fdbf80ac550 msgr2=0x7fdbf80be1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:16:18.119 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:18.124 7fdbfe534700 1 -- 172.21.2.102:0/3912744630 shutdown_connections 2021-02-10T11:16:18.119 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:18.124 7fdbfe534700 1 -- 172.21.2.102:0/3912744630 wait complete. 2021-02-10T11:16:18.125 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000344435 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000344435/f8e578a1-064d-44f5-b82e-9bee9f6f19d1 2021-02-10T11:16:18.125 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:16:18.125 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:16:18.125 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:16:18.461 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:16:18.462 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:16:18.462 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344435/f8e578a1-064d-44f5-b82e-9bee9f6f19d1/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:16:18.839 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:16:18.839 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:16:18.839 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0228457 s, 45.9 MB/s 2021-02-10T11:16:18.840 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:16:18.840 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:16:18.840 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:16:19.176 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:16:19.176 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:16:19.176 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344435/f8e578a1-064d-44f5-b82e-9bee9f6f19d1/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:16:19.551 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:16:19.551 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:16:19.551 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.023768 s, 44.1 MB/s 2021-02-10T11:16:19.552 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:16:19.553 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:16:19.553 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:16:19.887 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:16:19.887 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:16:19.888 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344435/f8e578a1-064d-44f5-b82e-9bee9f6f19d1/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:16:20.265 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:16:20.265 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:16:20.265 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0238409 s, 44.0 MB/s 2021-02-10T11:16:20.266 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:16:20.266 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:16:20.266 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:16:20.601 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:16:20.602 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:16:20.602 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344435/f8e578a1-064d-44f5-b82e-9bee9f6f19d1/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:16:20.979 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:16:20.979 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:16:20.980 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0254021 s, 41.3 MB/s 2021-02-10T11:16:20.981 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:16:20.981 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:16:20.981 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:16:21.317 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:16:21.317 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:16:21.318 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344435/f8e578a1-064d-44f5-b82e-9bee9f6f19d1/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:16:21.693 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:16:21.693 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:16:21.694 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0229361 s, 45.7 MB/s 2021-02-10T11:16:21.694 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:16:21.695 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:16:21.695 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:16:22.029 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:16:22.030 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:16:22.030 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344435/f8e578a1-064d-44f5-b82e-9bee9f6f19d1/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:16:22.406 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:16:22.406 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:16:22.406 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0249222 s, 42.1 MB/s 2021-02-10T11:16:22.407 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:16:22.407 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:16:22.407 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:16:22.743 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:16:22.744 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:16:22.744 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344435/f8e578a1-064d-44f5-b82e-9bee9f6f19d1/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:16:23.120 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:16:23.120 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:16:23.120 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0251059 s, 41.8 MB/s 2021-02-10T11:16:23.121 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:16:23.122 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:16:23.122 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:16:23.457 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:16:23.457 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:16:23.458 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344435/f8e578a1-064d-44f5-b82e-9bee9f6f19d1/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:16:23.832 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:16:23.832 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:16:23.832 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0260097 s, 40.3 MB/s 2021-02-10T11:16:23.833 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:16:23.834 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:16:23.834 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:16:24.170 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:16:24.170 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:16:24.170 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344435/f8e578a1-064d-44f5-b82e-9bee9f6f19d1/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:16:24.545 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:16:24.545 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:16:24.545 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0236075 s, 44.4 MB/s 2021-02-10T11:16:24.547 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:16:24.547 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:16:24.547 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:16:24.885 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:16:24.885 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:16:24.885 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344435/f8e578a1-064d-44f5-b82e-9bee9f6f19d1/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:16:25.262 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:16:25.262 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:16:25.263 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0249594 s, 42.0 MB/s 2021-02-10T11:16:25.264 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000344436 2021-02-10T11:16:25.695 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:25.686 7fe2f55d2700 1 Processor -- start 2021-02-10T11:16:25.695 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:25.686 7fe2f55d2700 1 -- start start 2021-02-10T11:16:25.695 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:25.686 7fe2f55d2700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fe2f01b0470 0x7fe2f01b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:16:25.695 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:25.686 7fe2f55d2700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe2f00ef610 0x7fe2f00efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:16:25.695 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:25.686 7fe2f55d2700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2f00f0b50 con 0x7fe2f00f0030 2021-02-10T11:16:25.696 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:25.686 7fe2f55d2700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2f00e52e0 con 0x7fe2f01b0470 2021-02-10T11:16:25.696 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:25.686 7fe2f55d2700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2f00e54d0 con 0x7fe2f00ef610 2021-02-10T11:16:25.696 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:25.686 7fe2ef2bf700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fe2f01b0470 0x7fe2f01b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:16:25.696 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:25.686 7fe2ef2bf700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fe2f01b0470 0x7fe2f01b0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:52944/0 (socket says 172.21.2.102:52944) 2021-02-10T11:16:25.696 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:25.686 7fe2ef2bf700 1 -- 172.21.2.102:0/4132373928 learned_addr learned my addr 172.21.2.102:0/4132373928 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:16:25.696 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:25.686 7fe2efac0700 1 --1- 172.21.2.102:0/4132373928 >> v1:172.21.2.113:6789/0 conn(0x7fe2f00f0030 0x7fe2f00f10d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.113:6789/0 says I am v1:172.21.2.102:52496/0 (socket says 172.21.2.102:52496) 2021-02-10T11:16:25.696 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:25.686 7fe2eeabe700 1 --2- 172.21.2.102:0/4132373928 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe2f00ef610 0x7fe2f00efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:16:25.696 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:25.687 7fe2ef2bf700 1 -- 172.21.2.102:0/4132373928 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe2f00ef610 msgr2=0x7fe2f00efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:16:25.696 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:25.687 7fe2ef2bf700 1 --2- 172.21.2.102:0/4132373928 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe2f00ef610 0x7fe2f00efa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:25.696 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:25.687 7fe2ef2bf700 1 -- 172.21.2.102:0/4132373928 >> v1:172.21.2.113:6789/0 conn(0x7fe2f00f0030 legacy=0x7fe2f00f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:16:25.696 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:25.687 7fe2ef2bf700 1 -- 172.21.2.102:0/4132373928 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2f00f0430 con 0x7fe2f01b0470 2021-02-10T11:16:25.696 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:25.687 7fe2ef2bf700 1 --2- 172.21.2.102:0/4132373928 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe2f01b0470 0x7fe2f01b0870 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 rx=0x7fe2e0006700 tx=0x7fe2e0009b00).ready entity=mon.0 client_cookie=979189e4c9a47236 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:16:25.696 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:25.687 7fe2ee2bd700 1 -- 172.21.2.102:0/4132373928 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe2e000c0c0 con 0x7fe2f01b0470 2021-02-10T11:16:25.697 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:25.687 7fe2ee2bd700 1 -- 172.21.2.102:0/4132373928 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2e000c400 con 0x7fe2f01b0470 2021-02-10T11:16:25.697 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:25.687 7fe2ee2bd700 1 -- 172.21.2.102:0/4132373928 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe2e000c920 con 0x7fe2f01b0470 2021-02-10T11:16:25.697 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:25.687 7fe2f55d2700 1 -- 172.21.2.102:0/4132373928 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe2f01b0470 msgr2=0x7fe2f01b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:16:25.697 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:25.687 7fe2f55d2700 1 --2- 172.21.2.102:0/4132373928 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe2f01b0470 0x7fe2f01b0870 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 rx=0x7fe2e0006700 tx=0x7fe2e0009b00).stop 2021-02-10T11:16:25.697 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:25.688 7fe2f55d2700 1 -- 172.21.2.102:0/4132373928 shutdown_connections 2021-02-10T11:16:25.697 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:25.688 7fe2f55d2700 1 --2- 172.21.2.102:0/4132373928 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe2f00ef610 0x7fe2f00efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:25.697 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:25.688 7fe2f55d2700 1 --2- 172.21.2.102:0/4132373928 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe2f01b0470 0x7fe2f01b0870 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:25.697 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:25.688 7fe2f55d2700 1 -- 172.21.2.102:0/4132373928 >> 172.21.2.102:0/4132373928 conn(0x7fe2f00ce3e0 msgr2=0x7fe2f00e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:16:25.697 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:25.688 7fe2f55d2700 1 -- 172.21.2.102:0/4132373928 shutdown_connections 2021-02-10T11:16:25.697 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:25.688 7fe2f55d2700 1 -- 172.21.2.102:0/4132373928 wait complete. 2021-02-10T11:16:25.697 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:25.702 7fe2f55d2700 1 Processor -- start 2021-02-10T11:16:25.697 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:25.702 7fe2f55d2700 1 -- start start 2021-02-10T11:16:25.699 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:25.703 7fe2f55d2700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe2f00a0420 0x7fe2f00a0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:16:25.699 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:25.703 7fe2f55d2700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe2f0189d00 0x7fe2f018a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:16:25.699 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:25.703 7fe2f55d2700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe2f018a720 0x7fe2f00b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:16:25.699 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:25.703 7fe2f55d2700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2f0189800 con 0x7fe2f0189d00 2021-02-10T11:16:25.699 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:25.703 7fe2f55d2700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2f018ab20 con 0x7fe2f00a0420 2021-02-10T11:16:25.700 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:25.703 7fe2f55d2700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2f00a1140 con 0x7fe2f018a720 2021-02-10T11:16:25.700 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:25.703 7fe2eeabe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe2f0189d00 0x7fe2f018a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:16:25.700 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:25.703 7fe2ef2bf700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe2f00a0420 0x7fe2f00a0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:16:25.700 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:25.703 7fe2eeabe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe2f0189d00 0x7fe2f018a100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:52950/0 (socket says 172.21.2.102:52950) 2021-02-10T11:16:25.700 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:25.703 7fe2eeabe700 1 -- 172.21.2.102:0/4263048172 learned_addr learned my addr 172.21.2.102:0/4263048172 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:16:25.700 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:25.703 7fe2efac0700 1 --2- 172.21.2.102:0/4263048172 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe2f018a720 0x7fe2f00b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:16:25.700 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:25.703 7fe2eeabe700 1 -- 172.21.2.102:0/4263048172 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe2f018a720 msgr2=0x7fe2f00b7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:16:25.700 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:25.703 7fe2eeabe700 1 --2- 172.21.2.102:0/4263048172 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe2f018a720 0x7fe2f00b7be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:25.700 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:25.703 7fe2eeabe700 1 -- 172.21.2.102:0/4263048172 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe2f00a0420 msgr2=0x7fe2f00a0820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:16:25.700 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:25.703 7fe2eeabe700 1 --2- 172.21.2.102:0/4263048172 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe2f00a0420 0x7fe2f00a0820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:25.700 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:25.703 7fe2eeabe700 1 -- 172.21.2.102:0/4263048172 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2f00b8200 con 0x7fe2f0189d00 2021-02-10T11:16:25.700 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:25.703 7fe2eeabe700 1 --2- 172.21.2.102:0/4263048172 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe2f0189d00 0x7fe2f018a100 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 rx=0x7fe2e4006f60 tx=0x7fe2e400b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:16:25.701 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:25.704 7fe2d7fff700 1 -- 172.21.2.102:0/4263048172 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe2e400bd60 con 0x7fe2f0189d00 2021-02-10T11:16:25.701 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:25.704 7fe2d7fff700 1 -- 172.21.2.102:0/4263048172 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2e4000c90 con 0x7fe2f0189d00 2021-02-10T11:16:25.701 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:25.704 7fe2f55d2700 1 -- 172.21.2.102:0/4263048172 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2f00b8200 con 0x7fe2f0189d00 2021-02-10T11:16:25.701 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:25.704 7fe2d7fff700 1 -- 172.21.2.102:0/4263048172 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe2e4002220 con 0x7fe2f0189d00 2021-02-10T11:16:25.701 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:25.704 7fe2f55d2700 1 -- 172.21.2.102:0/4263048172 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2f00b8790 con 0x7fe2f0189d00 2021-02-10T11:16:25.701 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:25.704 7fe2d7fff700 1 -- 172.21.2.102:0/4263048172 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fe2e40052e0 con 0x7fe2f0189d00 2021-02-10T11:16:25.701 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:25.704 7fe2d7fff700 1 --2- 172.21.2.102:0/4263048172 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe2d00186c0 0x7fe2d001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:16:25.701 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:25.704 7fe2d7fff700 1 -- 172.21.2.102:0/4263048172 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fe2e40327d0 con 0x7fe2f0189d00 2021-02-10T11:16:25.701 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:25.704 7fe2ef2bf700 1 --2- 172.21.2.102:0/4263048172 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe2d00186c0 0x7fe2d001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:16:25.701 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:25.705 7fe2ef2bf700 1 --2- 172.21.2.102:0/4263048172 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe2d00186c0 0x7fe2d001abc0 crc :-1 s=READY pgs=316 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:16:25.729 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:25.736 7fe2f55d2700 1 -- 172.21.2.102:0/4263048172 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2f0105ae0 con 0x7fe2f0189d00 2021-02-10T11:16:27.092 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:25.738 7fe2d7fff700 1 -- 172.21.2.102:0/4263048172 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fe2e4000940 con 0x7fe2f0189d00 2021-02-10T11:16:27.092 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:25.869 7fe2f55d2700 1 -- 172.21.2.102:0/4263048172 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344436", "target": ["mgr", ""]}) v1 -- 0x7fe2f00b9020 con 0x7fe2d00186c0 2021-02-10T11:16:27.093 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:25.896 7fe2d7fff700 1 -- 172.21.2.102:0/4263048172 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fe2e0006410 con 0x7fe2d00186c0 2021-02-10T11:16:27.093 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:25.902 7fe2f55d2700 1 -- 172.21.2.102:0/4263048172 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe2d00186c0 msgr2=0x7fe2d001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:16:27.093 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:25.902 7fe2f55d2700 1 --2- 172.21.2.102:0/4263048172 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe2d00186c0 0x7fe2d001abc0 crc :-1 s=READY pgs=316 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:27.093 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:25.902 7fe2f55d2700 1 -- 172.21.2.102:0/4263048172 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe2f0189d00 msgr2=0x7fe2f018a100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:16:27.093 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:25.902 7fe2f55d2700 1 --2- 172.21.2.102:0/4263048172 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe2f0189d00 0x7fe2f018a100 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 rx=0x7fe2e4006f60 tx=0x7fe2e400b970).stop 2021-02-10T11:16:27.093 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:25.902 7fe2f55d2700 1 -- 172.21.2.102:0/4263048172 shutdown_connections 2021-02-10T11:16:27.093 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:25.902 7fe2f55d2700 1 --2- 172.21.2.102:0/4263048172 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe2f018a720 0x7fe2f00b7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:27.093 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:25.902 7fe2f55d2700 1 --2- 172.21.2.102:0/4263048172 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe2f0189d00 0x7fe2f018a100 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:27.093 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:25.902 7fe2f55d2700 1 --2- 172.21.2.102:0/4263048172 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe2d00186c0 0x7fe2d001abc0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:27.093 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:25.902 7fe2f55d2700 1 --2- 172.21.2.102:0/4263048172 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe2f00a0420 0x7fe2f00a0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:27.093 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:25.902 7fe2f55d2700 1 -- 172.21.2.102:0/4263048172 >> 172.21.2.102:0/4263048172 conn(0x7fe2f0094e10 msgr2=0x7fe2f00a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:16:27.093 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:25.902 7fe2f55d2700 1 -- 172.21.2.102:0/4263048172 shutdown_connections 2021-02-10T11:16:27.094 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:25.903 7fe2f55d2700 1 -- 172.21.2.102:0/4263048172 wait complete. 2021-02-10T11:16:27.094 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000344436 2021-02-10T11:16:27.527 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:27.518 7f95eb334700 1 Processor -- start 2021-02-10T11:16:27.527 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:27.518 7f95eb334700 1 -- start start 2021-02-10T11:16:27.527 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:27.518 7f95eb334700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f95e41b0470 0x7f95e41b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:16:27.527 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:27.518 7f95eb334700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f95e40ef610 0x7f95e40efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:16:27.527 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:27.518 7f95eb334700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f95e40f0030 0x7f95e40f0430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:16:27.528 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:27.518 7f95eb334700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95e40e5ea0 con 0x7f95e40ef610 2021-02-10T11:16:27.528 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:27.518 7f95eb334700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95e40f0b50 con 0x7f95e40f0030 2021-02-10T11:16:27.528 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:27.518 7f95eb334700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95e40e52e0 con 0x7f95e41b0470 2021-02-10T11:16:27.528 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:27.519 7f95e894f700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f95e40ef610 0x7f95e40efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:16:27.528 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:27.519 7f95e894f700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f95e40ef610 0x7f95e40efa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:52958/0 (socket says 172.21.2.102:52958) 2021-02-10T11:16:27.528 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:27.519 7f95e894f700 1 -- 172.21.2.102:0/2787669389 learned_addr learned my addr 172.21.2.102:0/2787669389 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:16:27.528 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:27.519 7f95e9951700 1 --2- 172.21.2.102:0/2787669389 >> v2:172.21.2.113:3300/0 conn(0x7f95e40f0030 0x7f95e40f0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:16:27.528 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:27.519 7f95e9150700 1 --2- 172.21.2.102:0/2787669389 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f95e41b0470 0x7f95e41b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:16:27.528 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:27.519 7f95e894f700 1 -- 172.21.2.102:0/2787669389 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f95e41b0470 msgr2=0x7f95e41b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:16:27.528 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:27.519 7f95e894f700 1 --2- 172.21.2.102:0/2787669389 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f95e41b0470 0x7f95e41b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:27.528 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:27.519 7f95e894f700 1 -- 172.21.2.102:0/2787669389 >> v2:172.21.2.113:3300/0 conn(0x7f95e40f0030 msgr2=0x7f95e40f0430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:16:27.528 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:27.519 7f95e894f700 1 --2- 172.21.2.102:0/2787669389 >> v2:172.21.2.113:3300/0 conn(0x7f95e40f0030 0x7f95e40f0430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:27.528 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:27.519 7f95e894f700 1 -- 172.21.2.102:0/2787669389 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95e40f10d0 con 0x7f95e40ef610 2021-02-10T11:16:27.529 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:27.519 7f95e894f700 1 --2- 172.21.2.102:0/2787669389 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f95e40ef610 0x7f95e40efa10 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 rx=0x7f95e0006700 tx=0x7f95e0009b80).ready entity=mon.0 client_cookie=64a2b2d4d88c1e65 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:16:27.529 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:27.519 7f95d7fff700 1 -- 172.21.2.102:0/2787669389 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f95e000c0c0 con 0x7f95e40ef610 2021-02-10T11:16:27.529 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:27.519 7f95d7fff700 1 -- 172.21.2.102:0/2787669389 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95e000c340 con 0x7f95e40ef610 2021-02-10T11:16:27.529 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:27.520 7f95eb334700 1 -- 172.21.2.102:0/2787669389 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f95e40ef610 msgr2=0x7f95e40efa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:16:27.529 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:27.520 7f95eb334700 1 --2- 172.21.2.102:0/2787669389 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f95e40ef610 0x7f95e40efa10 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 rx=0x7f95e0006700 tx=0x7f95e0009b80).stop 2021-02-10T11:16:27.529 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:27.520 7f95d7fff700 1 -- 172.21.2.102:0/2787669389 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f95e000c860 con 0x7f95e40ef610 2021-02-10T11:16:27.529 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:27.520 7f95eb334700 1 -- 172.21.2.102:0/2787669389 shutdown_connections 2021-02-10T11:16:27.529 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:27.520 7f95eb334700 1 --2- 172.21.2.102:0/2787669389 >> v2:172.21.2.113:3300/0 conn(0x7f95e40f0030 0x7f95e40f0430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:27.529 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:27.520 7f95eb334700 1 --2- 172.21.2.102:0/2787669389 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f95e40ef610 0x7f95e40efa10 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:27.529 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:27.520 7f95eb334700 1 --2- 172.21.2.102:0/2787669389 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f95e41b0470 0x7f95e41b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:27.529 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:27.520 7f95eb334700 1 -- 172.21.2.102:0/2787669389 >> 172.21.2.102:0/2787669389 conn(0x7f95e40ce3e0 msgr2=0x7f95e40e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:16:27.529 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:27.520 7f95eb334700 1 -- 172.21.2.102:0/2787669389 shutdown_connections 2021-02-10T11:16:27.529 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:27.520 7f95eb334700 1 -- 172.21.2.102:0/2787669389 wait complete. 2021-02-10T11:16:27.530 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:27.535 7f95eb334700 1 Processor -- start 2021-02-10T11:16:27.530 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:27.535 7f95eb334700 1 -- start start 2021-02-10T11:16:27.530 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:27.535 7f95eb334700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f95e409b930 0x7f95e40b7c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:16:27.530 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:27.535 7f95eb334700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f95e40b8410 0x7f95e40b8810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:16:27.530 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:27.535 7f95eb334700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f95e40c1f80 0x7f95e40c2380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:16:27.530 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:27.535 7f95eb334700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95e40c1a00 con 0x7f95e409b930 2021-02-10T11:16:27.530 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:27.535 7f95eb334700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95e40b8e30 con 0x7f95e40b8410 2021-02-10T11:16:27.531 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:27.535 7f95eb334700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95e40b9050 con 0x7f95e40c1f80 2021-02-10T11:16:27.532 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:27.535 7f95e9150700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f95e409b930 0x7f95e40b7c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:16:27.532 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:27.535 7f95e9150700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f95e409b930 0x7f95e40b7c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:52962/0 (socket says 172.21.2.102:52962) 2021-02-10T11:16:27.532 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:27.535 7f95e9150700 1 -- 172.21.2.102:0/1104873007 learned_addr learned my addr 172.21.2.102:0/1104873007 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:16:27.532 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:27.536 7f95e9150700 1 -- 172.21.2.102:0/1104873007 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f95e40c1f80 msgr2=0x7f95e40c2380 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:16:27.532 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:27.536 7f95e9150700 1 --2- 172.21.2.102:0/1104873007 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f95e40c1f80 0x7f95e40c2380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:16:27.532 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:27.536 7f95e9150700 1 -- 172.21.2.102:0/1104873007 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f95e40b8410 msgr2=0x7f95e40b8810 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:16:27.532 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:27.536 7f95e9150700 1 --2- 172.21.2.102:0/1104873007 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f95e40b8410 0x7f95e40b8810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:16:27.532 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:27.536 7f95e9150700 1 -- 172.21.2.102:0/1104873007 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95e40c29a0 con 0x7f95e409b930 2021-02-10T11:16:27.532 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:27.536 7f95e9150700 1 --2- 172.21.2.102:0/1104873007 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f95e409b930 0x7f95e40b7c20 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 rx=0x7f95d8006f00 tx=0x7f95d800b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:16:27.532 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:27.536 7f95d5ffb700 1 -- 172.21.2.102:0/1104873007 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f95d800bd30 con 0x7f95e409b930 2021-02-10T11:16:27.532 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:27.536 7f95d5ffb700 1 -- 172.21.2.102:0/1104873007 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95d8002030 con 0x7f95e409b930 2021-02-10T11:16:27.532 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:27.536 7f95d5ffb700 1 -- 172.21.2.102:0/1104873007 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f95d8002550 con 0x7f95e409b930 2021-02-10T11:16:27.533 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:27.536 7f95eb334700 1 -- 172.21.2.102:0/1104873007 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95e40c29a0 con 0x7f95e409b930 2021-02-10T11:16:27.533 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:27.536 7f95eb334700 1 -- 172.21.2.102:0/1104873007 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95e40c29a0 con 0x7f95e409b930 2021-02-10T11:16:27.533 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:27.537 7f95d5ffb700 1 -- 172.21.2.102:0/1104873007 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f95d8002210 con 0x7f95e409b930 2021-02-10T11:16:27.533 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:27.537 7f95d5ffb700 1 --2- 172.21.2.102:0/1104873007 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f95c4018730 0x7f95c401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:16:27.533 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:27.537 7f95d5ffb700 1 -- 172.21.2.102:0/1104873007 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f95d8032660 con 0x7f95e409b930 2021-02-10T11:16:27.533 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:27.537 7f95e894f700 1 --2- 172.21.2.102:0/1104873007 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f95c4018730 0x7f95c401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:16:27.533 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:27.537 7f95e894f700 1 --2- 172.21.2.102:0/1104873007 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f95c4018730 0x7f95c401ac10 crc :-1 s=READY pgs=317 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:16:27.561 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:27.568 7f95eb334700 1 -- 172.21.2.102:0/1104873007 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95e4105b30 con 0x7f95e409b930 2021-02-10T11:16:27.563 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:27.571 7f95d5ffb700 1 -- 172.21.2.102:0/1104873007 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f95d8002240 con 0x7f95e409b930 2021-02-10T11:16:27.681 DEBUG:teuthology.orchestra.run.gibba002:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:16:27.683 DEBUG:teuthology.orchestra.run.gibba013:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:16:27.693 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:27.700 7f95eb334700 1 -- 172.21.2.102:0/1104873007 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344436", "target": ["mgr", ""]}) v1 -- 0x7f95e40b8e30 con 0x7f95c4018730 2021-02-10T11:16:27.697 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:27.704 7f95d5ffb700 1 -- 172.21.2.102:0/1104873007 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f95e0009100 con 0x7f95c4018730 2021-02-10T11:16:27.700 INFO:teuthology.orchestra.run.gibba002.stdout:/volumes/_nogroup/subvolume_0000000000344436/babc6dae-ad84-4cbf-a88b-86af338ad391 2021-02-10T11:16:27.702 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:27.709 7f95eb334700 1 -- 172.21.2.102:0/1104873007 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f95c4018730 msgr2=0x7f95c401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:16:27.702 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:27.709 7f95eb334700 1 --2- 172.21.2.102:0/1104873007 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f95c4018730 0x7f95c401ac10 crc :-1 s=READY pgs=317 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:27.702 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:27.709 7f95eb334700 1 -- 172.21.2.102:0/1104873007 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f95e409b930 msgr2=0x7f95e40b7c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:16:27.703 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:27.709 7f95eb334700 1 --2- 172.21.2.102:0/1104873007 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f95e409b930 0x7f95e40b7c20 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 rx=0x7f95d8006f00 tx=0x7f95d800b940).stop 2021-02-10T11:16:27.703 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:27.710 7f95eb334700 1 -- 172.21.2.102:0/1104873007 shutdown_connections 2021-02-10T11:16:27.703 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:27.710 7f95eb334700 1 --2- 172.21.2.102:0/1104873007 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f95e40c1f80 0x7f95e40c2380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:16:27.704 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:27.710 7f95eb334700 1 --2- 172.21.2.102:0/1104873007 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f95c4018730 0x7f95c401ac10 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:27.704 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:27.710 7f95eb334700 1 --2- 172.21.2.102:0/1104873007 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f95e40b8410 0x7f95e40b8810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:16:27.704 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:27.710 7f95eb334700 1 --2- 172.21.2.102:0/1104873007 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f95e409b930 0x7f95e40b7c20 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:27.704 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:27.710 7f95eb334700 1 -- 172.21.2.102:0/1104873007 >> 172.21.2.102:0/1104873007 conn(0x7f95e4094e10 msgr2=0x7f95e40a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:16:27.704 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:27.710 7f95eb334700 1 -- 172.21.2.102:0/1104873007 shutdown_connections 2021-02-10T11:16:27.704 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:27.710 7f95eb334700 1 -- 172.21.2.102:0/1104873007 wait complete. 2021-02-10T11:16:27.711 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000344436 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000344436/babc6dae-ad84-4cbf-a88b-86af338ad391 2021-02-10T11:16:27.711 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:16:27.711 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:16:27.711 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:16:28.019 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:16:28.019 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:16:28.019 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344436/babc6dae-ad84-4cbf-a88b-86af338ad391/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:16:28.398 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:16:28.399 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:16:28.399 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0253372 s, 41.4 MB/s 2021-02-10T11:16:28.400 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:16:28.400 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:16:28.400 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:16:28.735 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:16:28.736 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:16:28.736 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344436/babc6dae-ad84-4cbf-a88b-86af338ad391/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:16:29.114 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:16:29.114 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:16:29.114 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0247543 s, 42.4 MB/s 2021-02-10T11:16:29.115 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:16:29.116 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:16:29.116 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:16:29.453 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:16:29.454 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:16:29.455 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344436/babc6dae-ad84-4cbf-a88b-86af338ad391/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:16:29.850 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:16:29.850 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:16:29.850 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0246095 s, 42.6 MB/s 2021-02-10T11:16:29.850 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:16:29.850 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:16:29.851 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:16:30.197 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:16:30.198 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:16:30.198 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344436/babc6dae-ad84-4cbf-a88b-86af338ad391/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:16:30.580 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:16:30.580 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:16:30.580 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0256415 s, 40.9 MB/s 2021-02-10T11:16:30.581 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:16:30.581 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:16:30.581 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:16:30.922 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:16:30.923 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:16:30.923 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344436/babc6dae-ad84-4cbf-a88b-86af338ad391/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:16:31.303 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:16:31.303 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:16:31.303 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0248469 s, 42.2 MB/s 2021-02-10T11:16:31.304 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:16:31.304 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:16:31.305 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:16:31.643 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:16:31.644 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:16:31.644 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344436/babc6dae-ad84-4cbf-a88b-86af338ad391/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:16:32.019 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:16:32.020 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:16:32.020 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0233452 s, 44.9 MB/s 2021-02-10T11:16:32.021 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:16:32.021 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:16:32.021 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:16:32.356 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:16:32.356 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:16:32.356 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344436/babc6dae-ad84-4cbf-a88b-86af338ad391/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:16:32.732 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:16:32.733 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:16:32.733 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0229809 s, 45.6 MB/s 2021-02-10T11:16:32.734 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:16:32.734 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:16:32.734 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:16:33.070 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:16:33.070 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:16:33.071 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344436/babc6dae-ad84-4cbf-a88b-86af338ad391/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:16:33.442 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:16:33.442 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:16:33.442 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.023794 s, 44.1 MB/s 2021-02-10T11:16:33.443 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:16:33.443 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:16:33.443 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:16:33.779 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:16:33.780 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:16:33.780 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344436/babc6dae-ad84-4cbf-a88b-86af338ad391/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:16:34.156 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:16:34.156 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:16:34.156 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0251013 s, 41.8 MB/s 2021-02-10T11:16:34.157 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:16:34.157 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:16:34.157 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:16:34.492 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:16:34.493 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:16:34.493 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344436/babc6dae-ad84-4cbf-a88b-86af338ad391/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:16:34.869 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:16:34.870 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:16:34.870 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0236367 s, 44.4 MB/s 2021-02-10T11:16:34.871 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000344437 2021-02-10T11:16:35.305 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:35.295 7f6f3ed74700 1 Processor -- start 2021-02-10T11:16:35.305 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:35.295 7f6f3ed74700 1 -- start start 2021-02-10T11:16:35.305 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:35.296 7f6f3ed74700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f6f381b0470 0x7f6f381b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:16:35.305 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:35.296 7f6f3ed74700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f6f380f20e0 0x7f6f380f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:16:35.305 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:35.296 7f6f3ed74700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f380f0b50 con 0x7f6f380ef610 2021-02-10T11:16:35.305 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:35.296 7f6f3ed74700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f380f0820 con 0x7f6f381b0470 2021-02-10T11:16:35.305 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:35.296 7f6f3ed74700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f380f24e0 con 0x7f6f380f20e0 2021-02-10T11:16:35.305 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:35.296 7f6f3cb90700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f6f381b0470 0x7f6f381b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:16:35.305 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:35.296 7f6f37fff700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f6f380ef610 0x7f6f380efa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:45194/0 (socket says 172.21.2.102:45194) 2021-02-10T11:16:35.306 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:35.296 7f6f37fff700 1 -- 172.21.2.102:0/189263964 learned_addr learned my addr 172.21.2.102:0/189263964 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:16:35.306 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:35.296 7f6f3d391700 1 --2- 172.21.2.102:0/189263964 >> v2:172.21.2.113:3300/0 conn(0x7f6f380f20e0 0x7f6f380f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:16:35.306 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:35.296 7f6f3cb90700 1 -- 172.21.2.102:0/189263964 >> v2:172.21.2.113:3300/0 conn(0x7f6f380f20e0 msgr2=0x7f6f380f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:16:35.306 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:35.296 7f6f3cb90700 1 --2- 172.21.2.102:0/189263964 >> v2:172.21.2.113:3300/0 conn(0x7f6f380f20e0 0x7f6f380f0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:35.306 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:35.296 7f6f3cb90700 1 -- 172.21.2.102:0/189263964 >> v1:172.21.2.102:6789/0 conn(0x7f6f380ef610 legacy=0x7f6f380efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:16:35.306 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:35.296 7f6f3cb90700 1 -- 172.21.2.102:0/189263964 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f380e52e0 con 0x7f6f381b0470 2021-02-10T11:16:35.306 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:35.296 7f6f3d391700 1 --2- 172.21.2.102:0/189263964 >> v2:172.21.2.113:3300/0 conn(0x7f6f380f20e0 0x7f6f380f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:16:35.306 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:35.297 7f6f3cb90700 1 --2- 172.21.2.102:0/189263964 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6f381b0470 0x7f6f381b0870 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 rx=0x7f6f20006650 tx=0x7f6f20009a80).ready entity=mon.0 client_cookie=59ad5ca6a2879fdc server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:16:35.306 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:35.297 7f6f377fe700 1 -- 172.21.2.102:0/189263964 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6f200009b0 con 0x7f6f381b0470 2021-02-10T11:16:35.306 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:35.297 7f6f377fe700 1 -- 172.21.2.102:0/189263964 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f2000c0c0 con 0x7f6f381b0470 2021-02-10T11:16:35.306 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:35.297 7f6f377fe700 1 -- 172.21.2.102:0/189263964 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6f2000c5e0 con 0x7f6f381b0470 2021-02-10T11:16:35.306 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:35.297 7f6f3ed74700 1 -- 172.21.2.102:0/189263964 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6f381b0470 msgr2=0x7f6f381b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:16:35.307 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:35.297 7f6f3ed74700 1 --2- 172.21.2.102:0/189263964 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6f381b0470 0x7f6f381b0870 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 rx=0x7f6f20006650 tx=0x7f6f20009a80).stop 2021-02-10T11:16:35.307 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:35.298 7f6f3ed74700 1 -- 172.21.2.102:0/189263964 shutdown_connections 2021-02-10T11:16:35.307 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:35.298 7f6f3ed74700 1 --2- 172.21.2.102:0/189263964 >> v2:172.21.2.113:3300/0 conn(0x7f6f380f20e0 0x7f6f380f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:35.307 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:35.298 7f6f3ed74700 1 --2- 172.21.2.102:0/189263964 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6f381b0470 0x7f6f381b0870 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:35.307 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:35.298 7f6f3ed74700 1 -- 172.21.2.102:0/189263964 >> 172.21.2.102:0/189263964 conn(0x7f6f380ce3e0 msgr2=0x7f6f380e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:16:35.307 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:35.298 7f6f3ed74700 1 -- 172.21.2.102:0/189263964 shutdown_connections 2021-02-10T11:16:35.307 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:35.298 7f6f3ed74700 1 -- 172.21.2.102:0/189263964 wait complete. 2021-02-10T11:16:35.307 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:35.312 7f6f3ed74700 1 Processor -- start 2021-02-10T11:16:35.307 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:35.312 7f6f3ed74700 1 -- start start 2021-02-10T11:16:35.308 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:35.313 7f6f3ed74700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6f38196260 0x7f6f381a7850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:16:35.308 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:35.313 7f6f3ed74700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6f381a7e70 0x7f6f380a7ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:16:35.308 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:35.313 7f6f3ed74700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6f38196660 0x7f6f380a84f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:16:35.308 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:35.313 7f6f3ed74700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f380a9410 con 0x7f6f38196660 2021-02-10T11:16:35.308 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:35.313 7f6f3ed74700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f380a8b10 con 0x7f6f38196260 2021-02-10T11:16:35.309 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:35.313 7f6f3ed74700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f380a8cf0 con 0x7f6f381a7e70 2021-02-10T11:16:35.309 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:35.313 7f6f3d391700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6f38196660 0x7f6f380a84f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:16:35.309 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:35.313 7f6f3d391700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6f38196660 0x7f6f380a84f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:52978/0 (socket says 172.21.2.102:52978) 2021-02-10T11:16:35.310 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:35.313 7f6f3d391700 1 -- 172.21.2.102:0/2023092698 learned_addr learned my addr 172.21.2.102:0/2023092698 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:16:35.310 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:35.313 7f6f3d391700 1 -- 172.21.2.102:0/2023092698 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6f381a7e70 msgr2=0x7f6f380a7ed0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:16:35.310 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:35.313 7f6f3d391700 1 --2- 172.21.2.102:0/2023092698 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6f381a7e70 0x7f6f380a7ed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:16:35.310 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:35.313 7f6f3d391700 1 -- 172.21.2.102:0/2023092698 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6f38196260 msgr2=0x7f6f381a7850 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:16:35.310 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:35.313 7f6f3d391700 1 --2- 172.21.2.102:0/2023092698 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6f38196260 0x7f6f381a7850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:16:35.310 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:35.313 7f6f3d391700 1 -- 172.21.2.102:0/2023092698 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f380a8f10 con 0x7f6f38196660 2021-02-10T11:16:35.310 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:35.313 7f6f3d391700 1 --2- 172.21.2.102:0/2023092698 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6f38196660 0x7f6f380a84f0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 rx=0x7f6f30009980 tx=0x7f6f3000bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:16:35.310 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:35.313 7f6f357fa700 1 -- 172.21.2.102:0/2023092698 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6f3000c2f0 con 0x7f6f38196660 2021-02-10T11:16:35.310 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:35.314 7f6f357fa700 1 -- 172.21.2.102:0/2023092698 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f30006790 con 0x7f6f38196660 2021-02-10T11:16:35.310 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:35.314 7f6f357fa700 1 -- 172.21.2.102:0/2023092698 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6f3000c7f0 con 0x7f6f38196660 2021-02-10T11:16:35.310 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:35.314 7f6f3ed74700 1 -- 172.21.2.102:0/2023092698 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f380a8f10 con 0x7f6f38196660 2021-02-10T11:16:35.310 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:35.314 7f6f3ed74700 1 -- 172.21.2.102:0/2023092698 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f380a8f10 con 0x7f6f38196660 2021-02-10T11:16:35.311 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:35.314 7f6f357fa700 1 -- 172.21.2.102:0/2023092698 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f6f300011f0 con 0x7f6f38196660 2021-02-10T11:16:35.311 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:35.314 7f6f357fa700 1 --2- 172.21.2.102:0/2023092698 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6f1c0186c0 0x7f6f1c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:16:35.311 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:35.314 7f6f357fa700 1 -- 172.21.2.102:0/2023092698 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f6f300327c0 con 0x7f6f38196660 2021-02-10T11:16:35.311 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:35.314 7f6f3cb90700 1 --2- 172.21.2.102:0/2023092698 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6f1c0186c0 0x7f6f1c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:16:35.311 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:35.315 7f6f3cb90700 1 --2- 172.21.2.102:0/2023092698 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6f1c0186c0 0x7f6f1c01abc0 crc :-1 s=READY pgs=318 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:16:35.338 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:35.345 7f6f3ed74700 1 -- 172.21.2.102:0/2023092698 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f380a91b0 con 0x7f6f38196660 2021-02-10T11:16:35.341 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:35.348 7f6f357fa700 1 -- 172.21.2.102:0/2023092698 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f6f3000be10 con 0x7f6f38196660 2021-02-10T11:16:35.466 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:35.473 7f6f3ed74700 1 -- 172.21.2.102:0/2023092698 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344437", "target": ["mgr", ""]}) v1 -- 0x7f6f380a91b0 con 0x7f6f1c0186c0 2021-02-10T11:16:35.494 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:35.501 7f6f357fa700 1 -- 172.21.2.102:0/2023092698 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f6f20000d10 con 0x7f6f1c0186c0 2021-02-10T11:16:35.499 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:35.506 7f6f3ed74700 1 -- 172.21.2.102:0/2023092698 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6f1c0186c0 msgr2=0x7f6f1c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:16:35.499 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:35.506 7f6f3ed74700 1 --2- 172.21.2.102:0/2023092698 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6f1c0186c0 0x7f6f1c01abc0 crc :-1 s=READY pgs=318 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:35.499 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:35.506 7f6f3ed74700 1 -- 172.21.2.102:0/2023092698 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6f38196660 msgr2=0x7f6f380a84f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:16:35.499 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:35.506 7f6f3ed74700 1 --2- 172.21.2.102:0/2023092698 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6f38196660 0x7f6f380a84f0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 rx=0x7f6f30009980 tx=0x7f6f3000bcd0).stop 2021-02-10T11:16:35.500 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:35.506 7f6f3ed74700 1 -- 172.21.2.102:0/2023092698 shutdown_connections 2021-02-10T11:16:35.500 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:35.506 7f6f3ed74700 1 --2- 172.21.2.102:0/2023092698 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6f38196660 0x7f6f380a84f0 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:35.500 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:35.506 7f6f3ed74700 1 --2- 172.21.2.102:0/2023092698 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6f381a7e70 0x7f6f380a7ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:16:35.500 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:35.506 7f6f3ed74700 1 --2- 172.21.2.102:0/2023092698 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6f1c0186c0 0x7f6f1c01abc0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:35.500 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:35.506 7f6f3ed74700 1 --2- 172.21.2.102:0/2023092698 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6f38196260 0x7f6f381a7850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:16:35.500 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:35.506 7f6f3ed74700 1 -- 172.21.2.102:0/2023092698 >> 172.21.2.102:0/2023092698 conn(0x7f6f380bf4e0 msgr2=0x7f6f380937f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:16:35.500 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:35.507 7f6f3ed74700 1 -- 172.21.2.102:0/2023092698 shutdown_connections 2021-02-10T11:16:35.500 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:35.507 7f6f3ed74700 1 -- 172.21.2.102:0/2023092698 wait complete. 2021-02-10T11:16:35.508 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000344437 2021-02-10T11:16:35.940 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:35.930 7f7444e79700 1 Processor -- start 2021-02-10T11:16:35.940 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:35.930 7f7444e79700 1 -- start start 2021-02-10T11:16:35.940 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:35.931 7f7444e79700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f74401b0870 0x7f74400f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:16:35.940 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:35.931 7f7444e79700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f74400efe00 0x7f74400f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:16:35.940 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:35.931 7f7444e79700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74401b0c70 con 0x7f74401b0470 2021-02-10T11:16:35.940 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:35.931 7f7444e79700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74400f0b50 con 0x7f74400efe00 2021-02-10T11:16:35.940 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:35.931 7f7444e79700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74400f0820 con 0x7f74401b0870 2021-02-10T11:16:35.940 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:35.931 7f743f314700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f74400efe00 0x7f74400f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:16:35.941 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:35.931 7f743f314700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f74400efe00 0x7f74400f0200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.113:3300/0 says I am v2:172.21.2.102:56418/0 (socket says 172.21.2.102:56418) 2021-02-10T11:16:35.941 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:35.931 7f743f314700 1 -- 172.21.2.102:0/764283935 learned_addr learned my addr 172.21.2.102:0/764283935 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:16:35.941 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:35.931 7f743e312700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f74401b0870 0x7f74400f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:16:35.941 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:35.932 7f743f314700 1 -- 172.21.2.102:0/764283935 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f74401b0870 msgr2=0x7f74400f20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:16:35.941 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:35.932 7f743f314700 1 --2- 172.21.2.102:0/764283935 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f74401b0870 0x7f74400f20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:35.941 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:35.932 7f743f314700 1 -- 172.21.2.102:0/764283935 >> v1:172.21.2.113:6789/0 conn(0x7f74401b0470 legacy=0x7f74400ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:16:35.941 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:35.932 7f743f314700 1 -- 172.21.2.102:0/764283935 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74400e52e0 con 0x7f74400efe00 2021-02-10T11:16:35.941 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:35.932 7f743e312700 1 --2- 172.21.2.102:0/764283935 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f74401b0870 0x7f74400f20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:16:35.941 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:35.932 7f743f314700 1 --2- 172.21.2.102:0/764283935 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f74400efe00 0x7f74400f0200 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7f7438006aa0 tx=0x7f7438009f20).ready entity=mon.1 client_cookie=4c77530aa9fdb591 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:16:35.941 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:35.932 7f743db11700 1 -- 172.21.2.102:0/764283935 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7438000f90 con 0x7f74400efe00 2021-02-10T11:16:35.941 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:35.932 7f743db11700 1 -- 172.21.2.102:0/764283935 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7438003030 con 0x7f74400efe00 2021-02-10T11:16:35.941 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:35.932 7f7444e79700 1 -- 172.21.2.102:0/764283935 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f74400efe00 msgr2=0x7f74400f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:16:35.941 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:35.932 7f7444e79700 1 --2- 172.21.2.102:0/764283935 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f74400efe00 0x7f74400f0200 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7f7438006aa0 tx=0x7f7438009f20).stop 2021-02-10T11:16:35.942 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:35.933 7f7444e79700 1 -- 172.21.2.102:0/764283935 shutdown_connections 2021-02-10T11:16:35.942 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:35.933 7f7444e79700 1 --2- 172.21.2.102:0/764283935 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f74401b0870 0x7f74400f20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:35.942 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:35.933 7f7444e79700 1 --2- 172.21.2.102:0/764283935 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f74400efe00 0x7f74400f0200 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:35.942 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:35.933 7f7444e79700 1 -- 172.21.2.102:0/764283935 >> 172.21.2.102:0/764283935 conn(0x7f74400ce3e0 msgr2=0x7f74400e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:16:35.942 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:35.933 7f7444e79700 1 -- 172.21.2.102:0/764283935 shutdown_connections 2021-02-10T11:16:35.942 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:35.933 7f7444e79700 1 -- 172.21.2.102:0/764283935 wait complete. 2021-02-10T11:16:35.944 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:35.947 7f7444e79700 1 Processor -- start 2021-02-10T11:16:35.944 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:35.948 7f7444e79700 1 -- start start 2021-02-10T11:16:35.944 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:35.948 7f7444e79700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f74400a0420 0x7f74400a0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:16:35.944 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:35.948 7f7444e79700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7440189d00 0x7f744018a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:16:35.944 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:35.948 7f7444e79700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f744018a720 0x7f74400b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:16:35.944 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:35.948 7f7444e79700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7440189800 con 0x7f74400a0420 2021-02-10T11:16:35.944 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:35.948 7f7444e79700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f744018ab20 con 0x7f744018a720 2021-02-10T11:16:35.944 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:35.948 7f7444e79700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74400a1140 con 0x7f7440189d00 2021-02-10T11:16:35.944 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:35.948 7f743eb13700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f74400a0420 0x7f74400a0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:16:35.944 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:35.948 7f743eb13700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f74400a0420 0x7f74400a0820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:52992/0 (socket says 172.21.2.102:52992) 2021-02-10T11:16:35.944 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:35.948 7f743eb13700 1 -- 172.21.2.102:0/3002417537 learned_addr learned my addr 172.21.2.102:0/3002417537 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:16:35.945 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:35.948 7f743e312700 1 --2- 172.21.2.102:0/3002417537 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7440189d00 0x7f744018a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:16:35.945 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:35.949 7f743f314700 1 --2- 172.21.2.102:0/3002417537 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f744018a720 0x7f74400b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:16:35.945 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:35.949 7f743eb13700 1 -- 172.21.2.102:0/3002417537 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7440189d00 msgr2=0x7f744018a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:16:35.945 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:35.949 7f743eb13700 1 --2- 172.21.2.102:0/3002417537 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7440189d00 0x7f744018a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:35.945 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:35.949 7f743eb13700 1 -- 172.21.2.102:0/3002417537 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f744018a720 msgr2=0x7f74400b7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:16:35.945 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:35.949 7f743eb13700 1 --2- 172.21.2.102:0/3002417537 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f744018a720 0x7f74400b7be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:35.945 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:35.949 7f743eb13700 1 -- 172.21.2.102:0/3002417537 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74400b8200 con 0x7f74400a0420 2021-02-10T11:16:35.945 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:35.949 7f743e312700 1 --2- 172.21.2.102:0/3002417537 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7440189d00 0x7f744018a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:16:35.945 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:35.949 7f743f314700 1 --2- 172.21.2.102:0/3002417537 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f744018a720 0x7f74400b7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:16:35.945 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:35.949 7f743eb13700 1 --2- 172.21.2.102:0/3002417537 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f74400a0420 0x7f74400a0820 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 rx=0x7f742800b5b0 tx=0x7f742800b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:16:35.945 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:35.949 7f74377fe700 1 -- 172.21.2.102:0/3002417537 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f742800bd40 con 0x7f74400a0420 2021-02-10T11:16:35.945 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:35.949 7f74377fe700 1 -- 172.21.2.102:0/3002417537 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7428006020 con 0x7f74400a0420 2021-02-10T11:16:35.945 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:35.949 7f74377fe700 1 -- 172.21.2.102:0/3002417537 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f74280054a0 con 0x7f74400a0420 2021-02-10T11:16:35.946 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:35.949 7f7444e79700 1 -- 172.21.2.102:0/3002417537 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74400b8200 con 0x7f74400a0420 2021-02-10T11:16:35.946 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:35.949 7f7444e79700 1 -- 172.21.2.102:0/3002417537 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74400b8790 con 0x7f74400a0420 2021-02-10T11:16:35.946 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:35.950 7f74377fe700 1 -- 172.21.2.102:0/3002417537 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f7428006200 con 0x7f74400a0420 2021-02-10T11:16:35.946 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:35.950 7f74377fe700 1 --2- 172.21.2.102:0/3002417537 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f74240186c0 0x7f742401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:16:35.946 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:35.950 7f74377fe700 1 -- 172.21.2.102:0/3002417537 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f7428032440 con 0x7f74400a0420 2021-02-10T11:16:35.946 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:35.950 7f743e312700 1 --2- 172.21.2.102:0/3002417537 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f74240186c0 0x7f742401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:16:35.946 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:35.950 7f743e312700 1 --2- 172.21.2.102:0/3002417537 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f74240186c0 0x7f742401abc0 crc :-1 s=READY pgs=319 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:16:35.973 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:35.980 7f7444e79700 1 -- 172.21.2.102:0/3002417537 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7440105ae0 con 0x7f74400a0420 2021-02-10T11:16:35.975 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:35.983 7f74377fe700 1 -- 172.21.2.102:0/3002417537 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f742800aae0 con 0x7f74400a0420 2021-02-10T11:16:36.103 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:36.110 7f7444e79700 1 -- 172.21.2.102:0/3002417537 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344437", "target": ["mgr", ""]}) v1 -- 0x7f74400b8790 con 0x7f74240186c0 2021-02-10T11:16:36.106 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:36.113 7f74377fe700 1 -- 172.21.2.102:0/3002417537 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f7430006de0 con 0x7f74240186c0 2021-02-10T11:16:36.106 INFO:teuthology.orchestra.run.gibba002.stdout:/volumes/_nogroup/subvolume_0000000000344437/bf7a17f0-9f06-43e1-9b5b-7e81ea1ebdd0 2021-02-10T11:16:36.107 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:36.115 7f7444e79700 1 -- 172.21.2.102:0/3002417537 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f74240186c0 msgr2=0x7f742401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:16:36.108 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:36.115 7f7444e79700 1 --2- 172.21.2.102:0/3002417537 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f74240186c0 0x7f742401abc0 crc :-1 s=READY pgs=319 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:36.108 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:36.115 7f7444e79700 1 -- 172.21.2.102:0/3002417537 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f74400a0420 msgr2=0x7f74400a0820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:16:36.108 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:36.115 7f7444e79700 1 --2- 172.21.2.102:0/3002417537 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f74400a0420 0x7f74400a0820 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 rx=0x7f742800b5b0 tx=0x7f742800b950).stop 2021-02-10T11:16:36.108 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:36.115 7f7444e79700 1 -- 172.21.2.102:0/3002417537 shutdown_connections 2021-02-10T11:16:36.108 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:36.115 7f7444e79700 1 --2- 172.21.2.102:0/3002417537 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f74240186c0 0x7f742401abc0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:36.108 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:36.115 7f7444e79700 1 --2- 172.21.2.102:0/3002417537 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f744018a720 0x7f74400b7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:36.109 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:36.115 7f7444e79700 1 --2- 172.21.2.102:0/3002417537 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7440189d00 0x7f744018a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:36.109 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:36.115 7f7444e79700 1 --2- 172.21.2.102:0/3002417537 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f74400a0420 0x7f74400a0820 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:36.109 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:36.115 7f7444e79700 1 -- 172.21.2.102:0/3002417537 >> 172.21.2.102:0/3002417537 conn(0x7f7440094e10 msgr2=0x7f74400a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:16:36.109 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:36.115 7f7444e79700 1 -- 172.21.2.102:0/3002417537 shutdown_connections 2021-02-10T11:16:36.109 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:36.115 7f7444e79700 1 -- 172.21.2.102:0/3002417537 wait complete. 2021-02-10T11:16:36.116 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000344437 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000344437/bf7a17f0-9f06-43e1-9b5b-7e81ea1ebdd0 2021-02-10T11:16:36.116 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:16:36.116 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:16:36.116 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:16:36.451 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:16:36.452 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:16:36.452 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344437/bf7a17f0-9f06-43e1-9b5b-7e81ea1ebdd0/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:16:36.832 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:16:36.833 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:16:36.833 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0235576 s, 44.5 MB/s 2021-02-10T11:16:36.833 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:16:36.834 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:16:36.834 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:16:37.170 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:16:37.171 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:16:37.171 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344437/bf7a17f0-9f06-43e1-9b5b-7e81ea1ebdd0/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:16:37.547 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:16:37.547 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:16:37.547 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0240676 s, 43.6 MB/s 2021-02-10T11:16:37.548 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:16:37.548 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:16:37.548 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:16:37.886 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:16:37.886 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:16:37.886 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344437/bf7a17f0-9f06-43e1-9b5b-7e81ea1ebdd0/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:16:38.261 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:16:38.262 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:16:38.262 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0244655 s, 42.9 MB/s 2021-02-10T11:16:38.263 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:16:38.263 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:16:38.263 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:16:38.598 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:16:38.603 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:16:38.604 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344437/bf7a17f0-9f06-43e1-9b5b-7e81ea1ebdd0/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:16:38.982 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:16:38.982 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:16:38.982 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0301654 s, 34.8 MB/s 2021-02-10T11:16:38.983 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:16:38.983 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:16:38.983 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:16:39.319 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:16:39.320 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:16:39.320 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344437/bf7a17f0-9f06-43e1-9b5b-7e81ea1ebdd0/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:16:39.696 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:16:39.696 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:16:39.696 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0231125 s, 45.4 MB/s 2021-02-10T11:16:39.697 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:16:39.697 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:16:39.697 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:16:40.032 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:16:40.033 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:16:40.033 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344437/bf7a17f0-9f06-43e1-9b5b-7e81ea1ebdd0/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:16:40.409 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:16:40.410 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:16:40.410 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0246742 s, 42.5 MB/s 2021-02-10T11:16:40.411 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:16:40.411 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:16:40.411 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:16:40.747 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:16:40.748 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:16:40.748 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344437/bf7a17f0-9f06-43e1-9b5b-7e81ea1ebdd0/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:16:41.125 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:16:41.125 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:16:41.125 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0250082 s, 41.9 MB/s 2021-02-10T11:16:41.126 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:16:41.126 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:16:41.126 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:16:41.463 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:16:41.463 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:16:41.463 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344437/bf7a17f0-9f06-43e1-9b5b-7e81ea1ebdd0/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:16:41.839 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:16:41.839 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:16:41.839 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0247326 s, 42.4 MB/s 2021-02-10T11:16:41.840 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:16:41.840 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:16:41.840 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:16:42.176 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:16:42.176 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:16:42.177 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344437/bf7a17f0-9f06-43e1-9b5b-7e81ea1ebdd0/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:16:42.554 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:16:42.554 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:16:42.554 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0249308 s, 42.1 MB/s 2021-02-10T11:16:42.555 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:16:42.555 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:16:42.555 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:16:42.891 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:16:42.892 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:16:42.892 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344437/bf7a17f0-9f06-43e1-9b5b-7e81ea1ebdd0/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:16:43.269 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:16:43.269 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:16:43.269 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0250262 s, 41.9 MB/s 2021-02-10T11:16:43.270 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000344438 2021-02-10T11:16:43.703 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:43.694 7f9881e32700 1 Processor -- start 2021-02-10T11:16:43.703 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:43.694 7f9881e32700 1 -- start start 2021-02-10T11:16:43.703 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:43.695 7f9881e32700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f987c1b0870 0x7f987c0f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:16:43.703 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:43.695 7f9881e32700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f987c0efe00 0x7f987c0f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:16:43.703 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:43.695 7f9881e32700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f987c1b0c70 con 0x7f987c1b0470 2021-02-10T11:16:43.703 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:43.695 7f9881e32700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f987c0f0b50 con 0x7f987c0efe00 2021-02-10T11:16:43.704 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:43.695 7f9881e32700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f987c0f0820 con 0x7f987c1b0870 2021-02-10T11:16:43.704 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:43.695 7f987bfff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f987c0efe00 0x7f987c0f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:16:43.704 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:43.695 7f987bfff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f987c0efe00 0x7f987c0f0200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:53002/0 (socket says 172.21.2.102:53002) 2021-02-10T11:16:43.704 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:43.695 7f987bfff700 1 -- 172.21.2.102:0/2511173085 learned_addr learned my addr 172.21.2.102:0/2511173085 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:16:43.704 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:43.695 7f987bfff700 1 -- 172.21.2.102:0/2511173085 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f987c1b0870 msgr2=0x7f987c0f20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:16:43.704 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:43.695 7f987bfff700 1 --2- 172.21.2.102:0/2511173085 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f987c1b0870 0x7f987c0f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:16:43.704 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:43.695 7f987bfff700 1 -- 172.21.2.102:0/2511173085 >> v1:172.21.2.113:6789/0 conn(0x7f987c1b0470 legacy=0x7f987c0ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:16:43.704 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:43.695 7f987bfff700 1 -- 172.21.2.102:0/2511173085 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f987c0e52e0 con 0x7f987c0efe00 2021-02-10T11:16:43.704 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:43.695 7f987bfff700 1 --2- 172.21.2.102:0/2511173085 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f987c0efe00 0x7f987c0f0200 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 rx=0x7f9874006f00 tx=0x7f9874009e90).ready entity=mon.0 client_cookie=dccbd60b217734cc server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:16:43.704 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:43.695 7f987a7fc700 1 -- 172.21.2.102:0/2511173085 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f987400b300 con 0x7f987c0efe00 2021-02-10T11:16:43.704 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:43.696 7f987a7fc700 1 -- 172.21.2.102:0/2511173085 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f987400b580 con 0x7f987c0efe00 2021-02-10T11:16:43.704 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:43.696 7f987a7fc700 1 -- 172.21.2.102:0/2511173085 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f987400b9d0 con 0x7f987c0efe00 2021-02-10T11:16:43.705 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:43.696 7f9881e32700 1 -- 172.21.2.102:0/2511173085 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f987c0efe00 msgr2=0x7f987c0f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:16:43.705 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:43.696 7f9881e32700 1 --2- 172.21.2.102:0/2511173085 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f987c0efe00 0x7f987c0f0200 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 rx=0x7f9874006f00 tx=0x7f9874009e90).stop 2021-02-10T11:16:43.705 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:43.696 7f9881e32700 1 -- 172.21.2.102:0/2511173085 shutdown_connections 2021-02-10T11:16:43.705 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:43.696 7f9881e32700 1 --2- 172.21.2.102:0/2511173085 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f987c1b0870 0x7f987c0f20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:16:43.705 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:43.696 7f9881e32700 1 --2- 172.21.2.102:0/2511173085 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f987c0efe00 0x7f987c0f0200 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:43.705 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:43.696 7f9881e32700 1 -- 172.21.2.102:0/2511173085 >> 172.21.2.102:0/2511173085 conn(0x7f987c0ce3e0 msgr2=0x7f987c0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:16:43.705 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:43.696 7f9881e32700 1 -- 172.21.2.102:0/2511173085 shutdown_connections 2021-02-10T11:16:43.705 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:43.696 7f9881e32700 1 -- 172.21.2.102:0/2511173085 wait complete. 2021-02-10T11:16:43.706 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:43.711 7f9881e32700 1 Processor -- start 2021-02-10T11:16:43.707 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:43.711 7f9881e32700 1 -- start start 2021-02-10T11:16:43.707 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:43.711 7f9881e32700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f987c0a0960 0x7f987c0a67f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:16:43.707 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:43.711 7f9881e32700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f987c1924c0 0x7f987c1928c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:16:43.707 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:43.711 7f9881e32700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f987c192ee0 0x7f987c0a29f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:16:43.707 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:43.711 7f9881e32700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f987c0a6e10 con 0x7f987c1924c0 2021-02-10T11:16:43.707 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:43.711 7f9881e32700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f987c190630 con 0x7f987c192ee0 2021-02-10T11:16:43.707 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:43.711 7f9881e32700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f987c1932e0 con 0x7f987c0a0960 2021-02-10T11:16:43.707 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:43.711 7f987affd700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f987c1924c0 0x7f987c1928c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:16:43.707 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:43.711 7f987affd700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f987c1924c0 0x7f987c1928c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:53004/0 (socket says 172.21.2.102:53004) 2021-02-10T11:16:43.707 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:43.711 7f987affd700 1 -- 172.21.2.102:0/3103733696 learned_addr learned my addr 172.21.2.102:0/3103733696 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:16:43.707 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:43.712 7f987affd700 1 -- 172.21.2.102:0/3103733696 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f987c0a0960 msgr2=0x7f987c0a67f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:16:43.707 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:43.712 7f987affd700 1 --2- 172.21.2.102:0/3103733696 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f987c0a0960 0x7f987c0a67f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:16:43.707 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:43.712 7f987affd700 1 -- 172.21.2.102:0/3103733696 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f987c192ee0 msgr2=0x7f987c0a29f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:16:43.708 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:43.712 7f987affd700 1 --2- 172.21.2.102:0/3103733696 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f987c192ee0 0x7f987c0a29f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:16:43.708 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:43.712 7f987affd700 1 -- 172.21.2.102:0/3103733696 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f987c193500 con 0x7f987c1924c0 2021-02-10T11:16:43.708 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:43.712 7f987affd700 1 --2- 172.21.2.102:0/3103733696 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f987c1924c0 0x7f987c1928c0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 rx=0x7f986c00b5b0 tx=0x7f986c00b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:16:43.708 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:43.712 7f9873fff700 1 -- 172.21.2.102:0/3103733696 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f986c00bd40 con 0x7f987c1924c0 2021-02-10T11:16:43.708 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:43.712 7f9873fff700 1 -- 172.21.2.102:0/3103733696 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f986c006030 con 0x7f987c1924c0 2021-02-10T11:16:43.708 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:43.712 7f9873fff700 1 -- 172.21.2.102:0/3103733696 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f986c0054a0 con 0x7f987c1924c0 2021-02-10T11:16:43.708 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:43.712 7f9881e32700 1 -- 172.21.2.102:0/3103733696 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f987c193500 con 0x7f987c1924c0 2021-02-10T11:16:43.708 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:43.712 7f9881e32700 1 -- 172.21.2.102:0/3103733696 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f987c0a3300 con 0x7f987c1924c0 2021-02-10T11:16:43.708 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:43.713 7f9873fff700 1 -- 172.21.2.102:0/3103733696 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f986c006210 con 0x7f987c1924c0 2021-02-10T11:16:43.708 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:43.713 7f9873fff700 1 --2- 172.21.2.102:0/3103733696 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f98600186c0 0x7f986001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:16:43.708 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:43.713 7f9873fff700 1 -- 172.21.2.102:0/3103733696 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f986c032540 con 0x7f987c1924c0 2021-02-10T11:16:43.708 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:43.713 7f987b7fe700 1 --2- 172.21.2.102:0/3103733696 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f98600186c0 0x7f986001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:16:43.708 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:43.713 7f987b7fe700 1 --2- 172.21.2.102:0/3103733696 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f98600186c0 0x7f986001abc0 crc :-1 s=READY pgs=320 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:16:43.737 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:43.744 7f9881e32700 1 -- 172.21.2.102:0/3103733696 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f987c0a24c0 con 0x7f987c1924c0 2021-02-10T11:16:43.739 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:43.747 7f9873fff700 1 -- 172.21.2.102:0/3103733696 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f986c002df0 con 0x7f987c1924c0 2021-02-10T11:16:43.866 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:43.874 7f9881e32700 1 -- 172.21.2.102:0/3103733696 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344438", "target": ["mgr", ""]}) v1 -- 0x7f987c0a6e10 con 0x7f98600186c0 2021-02-10T11:16:43.894 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:43.901 7f9873fff700 1 -- 172.21.2.102:0/3103733696 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f9864006e40 con 0x7f98600186c0 2021-02-10T11:16:43.899 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:43.907 7f9881e32700 1 -- 172.21.2.102:0/3103733696 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f98600186c0 msgr2=0x7f986001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:16:43.899 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:43.907 7f9881e32700 1 --2- 172.21.2.102:0/3103733696 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f98600186c0 0x7f986001abc0 crc :-1 s=READY pgs=320 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:43.900 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:43.907 7f9881e32700 1 -- 172.21.2.102:0/3103733696 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f987c1924c0 msgr2=0x7f987c1928c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:16:43.900 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:43.907 7f9881e32700 1 --2- 172.21.2.102:0/3103733696 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f987c1924c0 0x7f987c1928c0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 rx=0x7f986c00b5b0 tx=0x7f986c00b950).stop 2021-02-10T11:16:43.900 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:43.907 7f9881e32700 1 -- 172.21.2.102:0/3103733696 shutdown_connections 2021-02-10T11:16:43.900 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:43.907 7f9881e32700 1 --2- 172.21.2.102:0/3103733696 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f98600186c0 0x7f986001abc0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:43.900 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:43.907 7f9881e32700 1 --2- 172.21.2.102:0/3103733696 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f987c192ee0 0x7f987c0a29f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:16:43.900 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:43.907 7f9881e32700 1 --2- 172.21.2.102:0/3103733696 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f987c1924c0 0x7f987c1928c0 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:43.900 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:43.907 7f9881e32700 1 --2- 172.21.2.102:0/3103733696 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f987c0a0960 0x7f987c0a67f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:16:43.900 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:43.907 7f9881e32700 1 -- 172.21.2.102:0/3103733696 >> 172.21.2.102:0/3103733696 conn(0x7f987c0ac550 msgr2=0x7f987c0be1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:16:43.901 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:43.907 7f9881e32700 1 -- 172.21.2.102:0/3103733696 shutdown_connections 2021-02-10T11:16:43.901 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:43.907 7f9881e32700 1 -- 172.21.2.102:0/3103733696 wait complete. 2021-02-10T11:16:43.908 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000344438 2021-02-10T11:16:44.339 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:44.330 7f6ae29f5700 1 Processor -- start 2021-02-10T11:16:44.340 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:44.331 7f6ae29f5700 1 -- start start 2021-02-10T11:16:44.340 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:44.331 7f6ae29f5700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6adc1b0470 0x7f6adc1b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:16:44.340 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:44.331 7f6ae29f5700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f6adc0ef610 0x7f6adc0efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:16:44.340 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:44.331 7f6ae29f5700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f6adc0f0030 0x7f6adc0f0430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:16:44.340 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:44.331 7f6ae29f5700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6adc0e5ea0 con 0x7f6adc0ef610 2021-02-10T11:16:44.340 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:44.331 7f6ae29f5700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6adc0f0b50 con 0x7f6adc0f0030 2021-02-10T11:16:44.340 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:44.331 7f6ae29f5700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6adc0e52e0 con 0x7f6adc1b0470 2021-02-10T11:16:44.340 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:44.331 7f6adbfff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f6adc0ef610 0x7f6adc0efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:16:44.340 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:44.331 7f6adbfff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f6adc0ef610 0x7f6adc0efa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:53012/0 (socket says 172.21.2.102:53012) 2021-02-10T11:16:44.340 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:44.331 7f6adbfff700 1 -- 172.21.2.102:0/128773925 learned_addr learned my addr 172.21.2.102:0/128773925 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:16:44.340 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:44.331 7f6adbfff700 1 -- 172.21.2.102:0/128773925 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6adc1b0470 msgr2=0x7f6adc1b0870 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:16:44.340 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:44.331 7f6adbfff700 1 --2- 172.21.2.102:0/128773925 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6adc1b0470 0x7f6adc1b0870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:16:44.340 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:44.331 7f6adbfff700 1 -- 172.21.2.102:0/128773925 >> v2:172.21.2.113:3300/0 conn(0x7f6adc0f0030 msgr2=0x7f6adc0f0430 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:16:44.341 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:44.331 7f6adbfff700 1 --2- 172.21.2.102:0/128773925 >> v2:172.21.2.113:3300/0 conn(0x7f6adc0f0030 0x7f6adc0f0430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:16:44.341 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:44.332 7f6adbfff700 1 -- 172.21.2.102:0/128773925 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6adc0f10d0 con 0x7f6adc0ef610 2021-02-10T11:16:44.341 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:44.332 7f6adbfff700 1 --2- 172.21.2.102:0/128773925 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6adc0ef610 0x7f6adc0efa10 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 rx=0x7f6acc006700 tx=0x7f6acc009b80).ready entity=mon.0 client_cookie=89b051270680c30d server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:16:44.341 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:44.332 7f6adb7fe700 1 -- 172.21.2.102:0/128773925 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6acc00c0c0 con 0x7f6adc0ef610 2021-02-10T11:16:44.341 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:44.332 7f6adb7fe700 1 -- 172.21.2.102:0/128773925 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6acc00c440 con 0x7f6adc0ef610 2021-02-10T11:16:44.341 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:44.332 7f6adb7fe700 1 -- 172.21.2.102:0/128773925 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6acc00c920 con 0x7f6adc0ef610 2021-02-10T11:16:44.341 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:44.332 7f6ae29f5700 1 -- 172.21.2.102:0/128773925 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6adc0ef610 msgr2=0x7f6adc0efa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:16:44.341 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:44.332 7f6ae29f5700 1 --2- 172.21.2.102:0/128773925 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6adc0ef610 0x7f6adc0efa10 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 rx=0x7f6acc006700 tx=0x7f6acc009b80).stop 2021-02-10T11:16:44.341 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:44.333 7f6ae29f5700 1 -- 172.21.2.102:0/128773925 shutdown_connections 2021-02-10T11:16:44.341 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:44.333 7f6ae29f5700 1 --2- 172.21.2.102:0/128773925 >> v2:172.21.2.113:3300/0 conn(0x7f6adc0f0030 0x7f6adc0f0430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:16:44.341 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:44.333 7f6ae29f5700 1 --2- 172.21.2.102:0/128773925 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6adc0ef610 0x7f6adc0efa10 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:44.341 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:44.333 7f6ae29f5700 1 --2- 172.21.2.102:0/128773925 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6adc1b0470 0x7f6adc1b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:16:44.342 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:44.333 7f6ae29f5700 1 -- 172.21.2.102:0/128773925 >> 172.21.2.102:0/128773925 conn(0x7f6adc0ce3e0 msgr2=0x7f6adc0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:16:44.342 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:44.333 7f6ae29f5700 1 -- 172.21.2.102:0/128773925 shutdown_connections 2021-02-10T11:16:44.342 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:44.333 7f6ae29f5700 1 -- 172.21.2.102:0/128773925 wait complete. 2021-02-10T11:16:44.343 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:44.347 7f6ae29f5700 1 Processor -- start 2021-02-10T11:16:44.343 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:44.347 7f6ae29f5700 1 -- start start 2021-02-10T11:16:44.343 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:44.348 7f6ae29f5700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6adc190920 0x7f6adc08f600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:16:44.343 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:44.348 7f6ae29f5700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6adc190d20 0x7f6adc08fc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:16:44.343 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:44.348 7f6ae29f5700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6adc193270 0x7f6adc193670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:16:44.343 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:44.348 7f6ae29f5700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6adc090470 con 0x7f6adc190d20 2021-02-10T11:16:44.344 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:44.348 7f6ae29f5700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6adc193c90 con 0x7f6adc190920 2021-02-10T11:16:44.344 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:44.348 7f6ae29f5700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6adc193e70 con 0x7f6adc193270 2021-02-10T11:16:44.344 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:44.348 7f6adbfff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6adc190d20 0x7f6adc08fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:16:44.344 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:44.348 7f6adbfff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6adc190d20 0x7f6adc08fc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:53020/0 (socket says 172.21.2.102:53020) 2021-02-10T11:16:44.344 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:44.348 7f6adbfff700 1 -- 172.21.2.102:0/717986945 learned_addr learned my addr 172.21.2.102:0/717986945 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:16:44.344 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:44.348 7f6adbfff700 1 -- 172.21.2.102:0/717986945 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6adc193270 msgr2=0x7f6adc193670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:16:44.344 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:44.348 7f6adbfff700 1 --2- 172.21.2.102:0/717986945 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6adc193270 0x7f6adc193670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:16:44.344 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:44.348 7f6adbfff700 1 -- 172.21.2.102:0/717986945 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6adc190920 msgr2=0x7f6adc08f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:16:44.344 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:44.348 7f6adbfff700 1 --2- 172.21.2.102:0/717986945 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6adc190920 0x7f6adc08f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:16:44.344 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:44.348 7f6adbfff700 1 -- 172.21.2.102:0/717986945 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6adc194090 con 0x7f6adc190d20 2021-02-10T11:16:44.344 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:44.348 7f6adbfff700 1 --2- 172.21.2.102:0/717986945 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6adc190d20 0x7f6adc08fc20 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 rx=0x7f6acc00b5e0 tx=0x7f6acc00b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:16:44.344 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:44.349 7f6ad97fa700 1 -- 172.21.2.102:0/717986945 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6acc00bd70 con 0x7f6adc190d20 2021-02-10T11:16:44.344 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:44.349 7f6ad97fa700 1 -- 172.21.2.102:0/717986945 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6acc002030 con 0x7f6adc190d20 2021-02-10T11:16:44.345 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:44.349 7f6ad97fa700 1 -- 172.21.2.102:0/717986945 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6acc002550 con 0x7f6adc190d20 2021-02-10T11:16:44.345 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:44.349 7f6ae29f5700 1 -- 172.21.2.102:0/717986945 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6adc194090 con 0x7f6adc190d20 2021-02-10T11:16:44.345 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:44.349 7f6ae29f5700 1 -- 172.21.2.102:0/717986945 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6adc1920b0 con 0x7f6adc190d20 2021-02-10T11:16:44.345 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:44.349 7f6ad97fa700 1 -- 172.21.2.102:0/717986945 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f6acc000c90 con 0x7f6adc190d20 2021-02-10T11:16:44.345 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:44.349 7f6ad97fa700 1 --2- 172.21.2.102:0/717986945 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6ac00186c0 0x7f6ac001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:16:44.345 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:44.349 7f6ad97fa700 1 -- 172.21.2.102:0/717986945 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f6acc032590 con 0x7f6adc190d20 2021-02-10T11:16:44.345 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:44.350 7f6ae0811700 1 --2- 172.21.2.102:0/717986945 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6ac00186c0 0x7f6ac001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:16:44.345 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:44.350 7f6ae0811700 1 --2- 172.21.2.102:0/717986945 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6ac00186c0 0x7f6ac001abc0 crc :-1 s=READY pgs=321 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:16:44.373 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:44.381 7f6ae29f5700 1 -- 172.21.2.102:0/717986945 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6adc105af0 con 0x7f6adc190d20 2021-02-10T11:16:44.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:44.383 7f6ad97fa700 1 -- 172.21.2.102:0/717986945 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f6acc030d40 con 0x7f6adc190d20 2021-02-10T11:16:44.503 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:44.510 7f6ae29f5700 1 -- 172.21.2.102:0/717986945 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344438", "target": ["mgr", ""]}) v1 -- 0x7f6adc1920b0 con 0x7f6ac00186c0 2021-02-10T11:16:44.506 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:44.513 7f6ad97fa700 1 -- 172.21.2.102:0/717986945 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f6ac4006e40 con 0x7f6ac00186c0 2021-02-10T11:16:44.506 INFO:teuthology.orchestra.run.gibba002.stdout:/volumes/_nogroup/subvolume_0000000000344438/7641ea72-f37e-4520-83bd-93ec91a71020 2021-02-10T11:16:44.507 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:44.515 7f6ae29f5700 1 -- 172.21.2.102:0/717986945 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6ac00186c0 msgr2=0x7f6ac001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:16:44.507 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:44.515 7f6ae29f5700 1 --2- 172.21.2.102:0/717986945 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6ac00186c0 0x7f6ac001abc0 crc :-1 s=READY pgs=321 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:44.507 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:44.515 7f6ae29f5700 1 -- 172.21.2.102:0/717986945 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6adc190d20 msgr2=0x7f6adc08fc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:16:44.507 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:44.515 7f6ae29f5700 1 --2- 172.21.2.102:0/717986945 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6adc190d20 0x7f6adc08fc20 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 rx=0x7f6acc00b5e0 tx=0x7f6acc00b980).stop 2021-02-10T11:16:44.508 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:44.515 7f6ae29f5700 1 -- 172.21.2.102:0/717986945 shutdown_connections 2021-02-10T11:16:44.508 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:44.515 7f6ae29f5700 1 --2- 172.21.2.102:0/717986945 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6adc193270 0x7f6adc193670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:16:44.508 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:44.515 7f6ae29f5700 1 --2- 172.21.2.102:0/717986945 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6adc190d20 0x7f6adc08fc20 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:44.508 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:44.515 7f6ae29f5700 1 --2- 172.21.2.102:0/717986945 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6ac00186c0 0x7f6ac001abc0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:44.508 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:44.515 7f6ae29f5700 1 --2- 172.21.2.102:0/717986945 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6adc190920 0x7f6adc08f600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:16:44.508 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:44.515 7f6ae29f5700 1 -- 172.21.2.102:0/717986945 >> 172.21.2.102:0/717986945 conn(0x7f6adc0bf4e0 msgr2=0x7f6adc0be3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:16:44.508 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:44.515 7f6ae29f5700 1 -- 172.21.2.102:0/717986945 shutdown_connections 2021-02-10T11:16:44.509 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:44.515 7f6ae29f5700 1 -- 172.21.2.102:0/717986945 wait complete. 2021-02-10T11:16:44.516 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000344438 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000344438/7641ea72-f37e-4520-83bd-93ec91a71020 2021-02-10T11:16:44.516 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:16:44.516 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:16:44.516 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:16:44.852 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:16:44.853 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:16:44.853 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344438/7641ea72-f37e-4520-83bd-93ec91a71020/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:16:45.235 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:16:45.235 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:16:45.236 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0234337 s, 44.7 MB/s 2021-02-10T11:16:45.237 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:16:45.237 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:16:45.238 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:16:45.575 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:16:45.576 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:16:45.576 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344438/7641ea72-f37e-4520-83bd-93ec91a71020/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:16:45.952 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:16:45.952 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:16:45.952 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0231204 s, 45.4 MB/s 2021-02-10T11:16:45.954 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:16:45.954 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:16:45.954 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:16:46.290 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:16:46.290 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:16:46.291 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344438/7641ea72-f37e-4520-83bd-93ec91a71020/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:16:46.669 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:16:46.669 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:16:46.669 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0260221 s, 40.3 MB/s 2021-02-10T11:16:46.669 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:16:46.670 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:16:46.670 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:16:47.005 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:16:47.006 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:16:47.006 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344438/7641ea72-f37e-4520-83bd-93ec91a71020/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:16:47.384 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:16:47.384 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:16:47.384 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.024188 s, 43.4 MB/s 2021-02-10T11:16:47.385 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:16:47.385 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:16:47.385 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:16:47.721 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:16:47.722 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:16:47.722 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344438/7641ea72-f37e-4520-83bd-93ec91a71020/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:16:48.096 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:16:48.096 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:16:48.096 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0229487 s, 45.7 MB/s 2021-02-10T11:16:48.097 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:16:48.097 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:16:48.098 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:16:48.434 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:16:48.435 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:16:48.435 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344438/7641ea72-f37e-4520-83bd-93ec91a71020/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:16:48.807 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:16:48.807 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:16:48.807 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0237515 s, 44.1 MB/s 2021-02-10T11:16:48.808 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:16:48.808 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:16:48.808 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:16:49.143 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:16:49.143 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:16:49.144 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344438/7641ea72-f37e-4520-83bd-93ec91a71020/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:16:49.520 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:16:49.520 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:16:49.521 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0248003 s, 42.3 MB/s 2021-02-10T11:16:49.521 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:16:49.522 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:16:49.522 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:16:49.856 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:16:49.857 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:16:49.857 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344438/7641ea72-f37e-4520-83bd-93ec91a71020/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:16:50.233 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:16:50.459 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:16:50.459 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0250588 s, 41.8 MB/s 2021-02-10T11:16:50.460 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:16:50.460 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:16:50.460 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:16:50.797 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:16:50.798 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:16:50.798 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344438/7641ea72-f37e-4520-83bd-93ec91a71020/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:16:51.174 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:16:51.175 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:16:51.175 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0241889 s, 43.3 MB/s 2021-02-10T11:16:51.176 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:16:51.176 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:16:51.176 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:16:51.512 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:16:51.512 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:16:51.513 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344438/7641ea72-f37e-4520-83bd-93ec91a71020/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:16:51.888 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:16:51.888 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:16:51.888 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0251469 s, 41.7 MB/s 2021-02-10T11:16:51.889 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000344439 2021-02-10T11:16:52.322 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:52.311 7fef887c7700 1 Processor -- start 2021-02-10T11:16:52.322 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:52.311 7fef887c7700 1 -- start start 2021-02-10T11:16:52.322 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:52.312 7fef887c7700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fef801b0870 0x7fef800f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:16:52.322 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:52.312 7fef887c7700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fef800efe00 0x7fef800f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:16:52.322 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:52.312 7fef887c7700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef801b0c70 con 0x7fef801b0470 2021-02-10T11:16:52.323 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:52.312 7fef887c7700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef800f0b50 con 0x7fef801b0870 2021-02-10T11:16:52.323 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:52.312 7fef887c7700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef800f0820 con 0x7fef800efe00 2021-02-10T11:16:52.323 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:52.312 7fef865e3700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fef801b0470 0x7fef800ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:45250/0 (socket says 172.21.2.102:45250) 2021-02-10T11:16:52.323 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:52.312 7fef865e3700 1 -- 172.21.2.102:0/4266887212 learned_addr learned my addr 172.21.2.102:0/4266887212 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:16:52.323 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:52.313 7fef855e1700 1 -- 172.21.2.102:0/4266887212 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2917347714 0 0) 0x7fef740010b0 con 0x7fef801b0470 2021-02-10T11:16:52.323 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:52.313 7fef855e1700 1 -- 172.21.2.102:0/4266887212 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef6c002fe0 con 0x7fef801b0470 2021-02-10T11:16:52.323 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:52.313 7fef86de4700 1 --2- 172.21.2.102:0/4266887212 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fef800efe00 0x7fef800f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:16:52.323 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:52.313 7fef85de2700 1 --2- 172.21.2.102:0/4266887212 >> v2:172.21.2.113:3300/0 conn(0x7fef801b0870 0x7fef800f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:16:52.323 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:52.313 7fef855e1700 1 -- 172.21.2.102:0/4266887212 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1276005453 0 0) 0x7fef740010b0 con 0x7fef801b0470 2021-02-10T11:16:52.323 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:52.313 7fef855e1700 1 -- 172.21.2.102:0/4266887212 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fef800efe00 msgr2=0x7fef800f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:16:52.323 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:52.313 7fef855e1700 1 --2- 172.21.2.102:0/4266887212 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fef800efe00 0x7fef800f0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:52.323 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:52.313 7fef855e1700 1 -- 172.21.2.102:0/4266887212 >> v2:172.21.2.113:3300/0 conn(0x7fef801b0870 msgr2=0x7fef800f20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:16:52.323 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:52.313 7fef855e1700 1 --2- 172.21.2.102:0/4266887212 >> v2:172.21.2.113:3300/0 conn(0x7fef801b0870 0x7fef800f20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:52.324 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:52.313 7fef855e1700 1 -- 172.21.2.102:0/4266887212 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef800e52e0 con 0x7fef801b0470 2021-02-10T11:16:52.324 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:52.313 7fef855e1700 1 -- 172.21.2.102:0/4266887212 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7fef740015b0 con 0x7fef801b0470 2021-02-10T11:16:52.324 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:52.313 7fef855e1700 1 -- 172.21.2.102:0/4266887212 >> v1:172.21.2.102:6789/0 conn(0x7fef801b0470 legacy=0x7fef800ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:16:52.324 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:52.313 7fef855e1700 1 --2- 172.21.2.102:0/4266887212 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fef6c003ae0 0x7fef6c005ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:16:52.324 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:52.313 7fef855e1700 1 --2- 172.21.2.102:0/4266887212 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fef6c006770 0x7fef6c008c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:16:52.324 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:52.313 7fef855e1700 1 --2- 172.21.2.102:0/4266887212 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fef6c009500 0x7fef6c00ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:16:52.324 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:52.313 7fef855e1700 1 -- 172.21.2.102:0/4266887212 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef6c0502e0 con 0x7fef6c003ae0 2021-02-10T11:16:52.324 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:52.313 7fef855e1700 1 -- 172.21.2.102:0/4266887212 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef6c050500 con 0x7fef6c009500 2021-02-10T11:16:52.324 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:52.313 7fef855e1700 1 -- 172.21.2.102:0/4266887212 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef6c050720 con 0x7fef6c006770 2021-02-10T11:16:52.324 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:52.313 7fef865e3700 1 --2- 172.21.2.102:0/4266887212 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fef6c003ae0 0x7fef6c005ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:16:52.324 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:52.313 7fef85de2700 1 --2- 172.21.2.102:0/4266887212 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fef6c006770 0x7fef6c008c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:16:52.324 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:52.313 7fef86de4700 1 --2- 172.21.2.102:0/4266887212 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fef6c009500 0x7fef6c00ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:16:52.324 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:52.313 7fef865e3700 1 -- 172.21.2.102:0/4266887212 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fef6c006770 msgr2=0x7fef6c008c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:16:52.325 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:52.313 7fef865e3700 1 --2- 172.21.2.102:0/4266887212 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fef6c006770 0x7fef6c008c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:52.325 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:52.313 7fef865e3700 1 -- 172.21.2.102:0/4266887212 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fef6c009500 msgr2=0x7fef6c00ba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:16:52.325 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:52.313 7fef865e3700 1 --2- 172.21.2.102:0/4266887212 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fef6c009500 0x7fef6c00ba20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:52.325 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:52.313 7fef865e3700 1 -- 172.21.2.102:0/4266887212 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef6c0509c0 con 0x7fef6c003ae0 2021-02-10T11:16:52.325 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:52.314 7fef865e3700 1 --2- 172.21.2.102:0/4266887212 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fef6c003ae0 0x7fef6c005ef0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 rx=0x7fef74000b50 tx=0x7fef74007d60).ready entity=mon.0 client_cookie=550c41b45b2bb983 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:16:52.325 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:52.314 7fef855e1700 1 -- 172.21.2.102:0/4266887212 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fef7400a310 con 0x7fef6c003ae0 2021-02-10T11:16:52.325 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:52.314 7fef855e1700 1 -- 172.21.2.102:0/4266887212 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef7400a650 con 0x7fef6c003ae0 2021-02-10T11:16:52.325 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:52.314 7fef855e1700 1 -- 172.21.2.102:0/4266887212 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fef7400ab70 con 0x7fef6c003ae0 2021-02-10T11:16:52.325 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:52.314 7fef887c7700 1 -- 172.21.2.102:0/4266887212 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fef6c003ae0 msgr2=0x7fef6c005ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:16:52.325 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:52.314 7fef887c7700 1 --2- 172.21.2.102:0/4266887212 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fef6c003ae0 0x7fef6c005ef0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 rx=0x7fef74000b50 tx=0x7fef74007d60).stop 2021-02-10T11:16:52.325 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:52.314 7fef86de4700 1 -- 172.21.2.102:0/4266887212 reap_dead start 2021-02-10T11:16:52.325 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:52.315 7fef887c7700 1 -- 172.21.2.102:0/4266887212 shutdown_connections 2021-02-10T11:16:52.325 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:52.315 7fef887c7700 1 -- 172.21.2.102:0/4266887212 >> 172.21.2.102:0/4266887212 conn(0x7fef800ce3e0 msgr2=0x7fef800e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:16:52.326 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:52.315 7fef887c7700 1 -- 172.21.2.102:0/4266887212 shutdown_connections 2021-02-10T11:16:52.326 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:52.315 7fef887c7700 1 -- 172.21.2.102:0/4266887212 wait complete. 2021-02-10T11:16:52.326 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:52.330 7fef887c7700 1 Processor -- start 2021-02-10T11:16:52.326 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:52.330 7fef887c7700 1 -- start start 2021-02-10T11:16:52.327 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:52.330 7fef887c7700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fef8009b930 0x7fef800b7c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:16:52.327 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:52.330 7fef887c7700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fef800b8410 0x7fef800b8810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:16:52.327 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:52.330 7fef887c7700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fef800c1f80 0x7fef800c2380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:16:52.327 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:52.330 7fef887c7700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef800c1a00 con 0x7fef800c1f80 2021-02-10T11:16:52.327 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:52.330 7fef887c7700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef800b8e30 con 0x7fef800b8410 2021-02-10T11:16:52.327 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:52.330 7fef887c7700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef800b9050 con 0x7fef8009b930 2021-02-10T11:16:52.327 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:52.330 7fef86de4700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fef800c1f80 0x7fef800c2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:16:52.328 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:52.330 7fef86de4700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fef800c1f80 0x7fef800c2380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:53042/0 (socket says 172.21.2.102:53042) 2021-02-10T11:16:52.328 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:52.330 7fef86de4700 1 -- 172.21.2.102:0/2848231360 learned_addr learned my addr 172.21.2.102:0/2848231360 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:16:52.328 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:52.331 7fef86de4700 1 -- 172.21.2.102:0/2848231360 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fef8009b930 msgr2=0x7fef800b7c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:16:52.328 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:52.331 7fef86de4700 1 --2- 172.21.2.102:0/2848231360 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fef8009b930 0x7fef800b7c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:16:52.328 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:52.331 7fef86de4700 1 -- 172.21.2.102:0/2848231360 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fef800b8410 msgr2=0x7fef800b8810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:16:52.328 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:52.331 7fef86de4700 1 --2- 172.21.2.102:0/2848231360 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fef800b8410 0x7fef800b8810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:16:52.328 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:52.331 7fef86de4700 1 -- 172.21.2.102:0/2848231360 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef800c29a0 con 0x7fef800c1f80 2021-02-10T11:16:52.328 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:52.331 7fef86de4700 1 --2- 172.21.2.102:0/2848231360 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fef800c1f80 0x7fef800c2380 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 rx=0x7fef780099e0 tx=0x7fef7800bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:16:52.328 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:52.331 7fef72ffd700 1 -- 172.21.2.102:0/2848231360 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fef78006030 con 0x7fef800c1f80 2021-02-10T11:16:52.328 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:52.331 7fef72ffd700 1 -- 172.21.2.102:0/2848231360 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef78006330 con 0x7fef800c1f80 2021-02-10T11:16:52.328 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:52.331 7fef72ffd700 1 -- 172.21.2.102:0/2848231360 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fef78005930 con 0x7fef800c1f80 2021-02-10T11:16:52.328 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:52.331 7fef887c7700 1 -- 172.21.2.102:0/2848231360 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef800c29a0 con 0x7fef800c1f80 2021-02-10T11:16:52.328 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:52.331 7fef887c7700 1 -- 172.21.2.102:0/2848231360 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef800c29a0 con 0x7fef800c1f80 2021-02-10T11:16:52.329 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:52.332 7fef72ffd700 1 -- 172.21.2.102:0/2848231360 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fef78006030 con 0x7fef800c1f80 2021-02-10T11:16:52.329 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:52.332 7fef72ffd700 1 --2- 172.21.2.102:0/2848231360 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fef680186c0 0x7fef6801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:16:52.329 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:52.332 7fef72ffd700 1 -- 172.21.2.102:0/2848231360 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fef78032b30 con 0x7fef800c1f80 2021-02-10T11:16:52.329 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:52.332 7fef865e3700 1 --2- 172.21.2.102:0/2848231360 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fef680186c0 0x7fef6801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:16:52.329 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:52.332 7fef865e3700 1 --2- 172.21.2.102:0/2848231360 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fef680186c0 0x7fef6801abc0 crc :-1 s=READY pgs=322 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:16:52.354 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:52.362 7fef887c7700 1 -- 172.21.2.102:0/2848231360 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef80105b30 con 0x7fef800c1f80 2021-02-10T11:16:52.357 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:52.364 7fef72ffd700 1 -- 172.21.2.102:0/2848231360 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fef78006030 con 0x7fef800c1f80 2021-02-10T11:16:52.484 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:52.491 7fef887c7700 1 -- 172.21.2.102:0/2848231360 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344439", "target": ["mgr", ""]}) v1 -- 0x7fef80105b30 con 0x7fef680186c0 2021-02-10T11:16:52.509 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:52.516 7fef72ffd700 1 -- 172.21.2.102:0/2848231360 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fef740010b0 con 0x7fef680186c0 2021-02-10T11:16:52.516 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:52.524 7fef887c7700 1 -- 172.21.2.102:0/2848231360 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fef680186c0 msgr2=0x7fef6801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:16:52.516 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:52.524 7fef887c7700 1 --2- 172.21.2.102:0/2848231360 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fef680186c0 0x7fef6801abc0 crc :-1 s=READY pgs=322 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:52.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:52.524 7fef887c7700 1 -- 172.21.2.102:0/2848231360 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fef800c1f80 msgr2=0x7fef800c2380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:16:52.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:52.524 7fef887c7700 1 --2- 172.21.2.102:0/2848231360 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fef800c1f80 0x7fef800c2380 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 rx=0x7fef780099e0 tx=0x7fef7800bcf0).stop 2021-02-10T11:16:52.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:52.524 7fef887c7700 1 -- 172.21.2.102:0/2848231360 shutdown_connections 2021-02-10T11:16:52.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:52.524 7fef887c7700 1 --2- 172.21.2.102:0/2848231360 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fef800c1f80 0x7fef800c2380 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:52.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:52.524 7fef887c7700 1 --2- 172.21.2.102:0/2848231360 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fef680186c0 0x7fef6801abc0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:52.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:52.524 7fef887c7700 1 --2- 172.21.2.102:0/2848231360 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fef800b8410 0x7fef800b8810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:16:52.518 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:52.524 7fef887c7700 1 --2- 172.21.2.102:0/2848231360 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fef8009b930 0x7fef800b7c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:16:52.518 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:52.524 7fef887c7700 1 -- 172.21.2.102:0/2848231360 >> 172.21.2.102:0/2848231360 conn(0x7fef80094e10 msgr2=0x7fef800a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:16:52.518 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:52.524 7fef887c7700 1 -- 172.21.2.102:0/2848231360 shutdown_connections 2021-02-10T11:16:52.518 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:52.524 7fef887c7700 1 -- 172.21.2.102:0/2848231360 wait complete. 2021-02-10T11:16:52.525 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000344439 2021-02-10T11:16:52.958 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:52.948 7fec000a9700 1 Processor -- start 2021-02-10T11:16:52.958 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:52.948 7fec000a9700 1 -- start start 2021-02-10T11:16:52.958 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:52.949 7fec000a9700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7febf81b0470 0x7febf81b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:16:52.958 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:52.949 7fec000a9700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febf80f0b50 con 0x7febf80f20e0 2021-02-10T11:16:52.958 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:52.949 7fec000a9700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febf80f24e0 con 0x7febf80ef610 2021-02-10T11:16:52.958 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:52.949 7fec000a9700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febf80e5ea0 con 0x7febf81b0470 2021-02-10T11:16:52.958 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:52.949 7febfe6c6700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7febf80f20e0 0x7febf80f0200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:45270/0 (socket says 172.21.2.102:45270) 2021-02-10T11:16:52.958 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:52.949 7febfe6c6700 1 -- 172.21.2.102:0/585542339 learned_addr learned my addr 172.21.2.102:0/585542339 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:16:52.959 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:52.949 7febfcec3700 1 -- 172.21.2.102:0/585542339 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1522448000 0 0) 0x7febf0000c10 con 0x7febf80f20e0 2021-02-10T11:16:52.959 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:52.949 7febfcec3700 1 -- 172.21.2.102:0/585542339 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7febe4002fe0 con 0x7febf80f20e0 2021-02-10T11:16:52.959 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:52.949 7febfdec5700 1 --2- 172.21.2.102:0/585542339 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7febf81b0470 0x7febf81b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:16:52.959 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:52.949 7febfcec3700 1 -- 172.21.2.102:0/585542339 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2403921723 0 0) 0x7febf00010e0 con 0x7febf80f20e0 2021-02-10T11:16:52.959 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:52.949 7febfcec3700 1 -- 172.21.2.102:0/585542339 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7febf81b0470 msgr2=0x7febf81b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:16:52.959 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:52.949 7febfcec3700 1 --2- 172.21.2.102:0/585542339 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7febf81b0470 0x7febf81b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:52.959 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:52.949 7febfcec3700 1 -- 172.21.2.102:0/585542339 >> v1:172.21.2.113:6789/0 conn(0x7febf80ef610 legacy=0x7febf80efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:16:52.959 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:52.949 7febfcec3700 1 -- 172.21.2.102:0/585542339 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febf80e52e0 con 0x7febf80f20e0 2021-02-10T11:16:52.959 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:52.949 7febfcec3700 1 -- 172.21.2.102:0/585542339 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7febf0001600 con 0x7febf80f20e0 2021-02-10T11:16:52.959 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:52.949 7febfcec3700 1 -- 172.21.2.102:0/585542339 >> v1:172.21.2.102:6789/0 conn(0x7febf80f20e0 legacy=0x7febf80f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:16:52.959 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:52.949 7febfcec3700 1 --2- 172.21.2.102:0/585542339 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7febe4003af0 0x7febe4005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:16:52.959 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:52.949 7febfcec3700 1 --2- 172.21.2.102:0/585542339 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7febe4006840 0x7febe4008d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:16:52.959 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:52.949 7febfcec3700 1 --2- 172.21.2.102:0/585542339 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7febe4009640 0x7febe400bb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:16:52.960 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:52.949 7febfcec3700 1 -- 172.21.2.102:0/585542339 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febe400c380 con 0x7febe4006840 2021-02-10T11:16:52.960 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:52.949 7febfcec3700 1 -- 172.21.2.102:0/585542339 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febe400c5a0 con 0x7febe4003af0 2021-02-10T11:16:52.960 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:52.949 7febfcec3700 1 -- 172.21.2.102:0/585542339 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febe400c7c0 con 0x7febe4009640 2021-02-10T11:16:52.960 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:52.950 7febfdec5700 1 --2- 172.21.2.102:0/585542339 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7febf81b0470 0x7febf81b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:16:52.960 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:52.950 7febfd6c4700 1 --2- 172.21.2.102:0/585542339 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7febe4006840 0x7febe4008d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:16:52.960 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:52.950 7febfdec5700 1 --2- 172.21.2.102:0/585542339 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7febe4003af0 0x7febe4005fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:16:52.960 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:52.950 7febfe6c6700 1 --2- 172.21.2.102:0/585542339 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7febe4009640 0x7febe400bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:16:52.960 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:52.950 7febfd6c4700 1 -- 172.21.2.102:0/585542339 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7febe4009640 msgr2=0x7febe400bb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:16:52.960 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:52.950 7febfd6c4700 1 --2- 172.21.2.102:0/585542339 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7febe4009640 0x7febe400bb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:52.960 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:52.950 7febfd6c4700 1 -- 172.21.2.102:0/585542339 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7febe4003af0 msgr2=0x7febe4005fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:16:52.960 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:52.950 7febfd6c4700 1 --2- 172.21.2.102:0/585542339 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7febe4003af0 0x7febe4005fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:52.960 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:52.950 7febfd6c4700 1 -- 172.21.2.102:0/585542339 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febe400ca60 con 0x7febe4006840 2021-02-10T11:16:52.960 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:52.950 7febfd6c4700 1 --2- 172.21.2.102:0/585542339 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7febe4006840 0x7febe4008d40 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 rx=0x7febf4006ae0 tx=0x7febf404db00).ready entity=mon.0 client_cookie=b3c41377cca6ce46 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:16:52.961 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:52.950 7febfcec3700 1 -- 172.21.2.102:0/585542339 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7febf404b260 con 0x7febe4006840 2021-02-10T11:16:52.961 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:52.950 7febfcec3700 1 -- 172.21.2.102:0/585542339 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febf404b4a0 con 0x7febe4006840 2021-02-10T11:16:52.961 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:52.950 7febfcec3700 1 -- 172.21.2.102:0/585542339 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7febf404b9e0 con 0x7febe4006840 2021-02-10T11:16:52.961 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:52.950 7fec000a9700 1 -- 172.21.2.102:0/585542339 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7febe4006840 msgr2=0x7febe4008d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:16:52.961 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:52.950 7fec000a9700 1 --2- 172.21.2.102:0/585542339 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7febe4006840 0x7febe4008d40 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 rx=0x7febf4006ae0 tx=0x7febf404db00).stop 2021-02-10T11:16:52.961 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:52.950 7febfe6c6700 1 -- 172.21.2.102:0/585542339 reap_dead start 2021-02-10T11:16:52.961 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:52.951 7fec000a9700 1 -- 172.21.2.102:0/585542339 shutdown_connections 2021-02-10T11:16:52.961 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:52.951 7fec000a9700 1 -- 172.21.2.102:0/585542339 >> 172.21.2.102:0/585542339 conn(0x7febf80ce3e0 msgr2=0x7febf80e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:16:52.961 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:52.951 7fec000a9700 1 -- 172.21.2.102:0/585542339 shutdown_connections 2021-02-10T11:16:52.961 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:52.951 7fec000a9700 1 -- 172.21.2.102:0/585542339 wait complete. 2021-02-10T11:16:52.963 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:52.966 7fec000a9700 1 Processor -- start 2021-02-10T11:16:52.963 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:52.966 7fec000a9700 1 -- start start 2021-02-10T11:16:52.963 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:52.966 7fec000a9700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7febf80b7db0 0x7febf80b8260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:16:52.963 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:52.966 7fec000a9700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7febf80b8880 0x7febf81a7450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:16:52.963 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:52.966 7fec000a9700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7febf80b8c80 0x7febf81a7a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:16:52.963 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:52.966 7fec000a9700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febf80b79b0 con 0x7febf80b8c80 2021-02-10T11:16:52.963 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:52.966 7fec000a9700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febf81a8090 con 0x7febf80b7db0 2021-02-10T11:16:52.963 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:52.966 7fec000a9700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febf81a82b0 con 0x7febf80b8880 2021-02-10T11:16:52.963 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:52.966 7febfe6c6700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7febf80b8c80 0x7febf81a7a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:16:52.963 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:52.966 7febfdec5700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7febf80b7db0 0x7febf80b8260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:16:52.963 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:52.966 7febfe6c6700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7febf80b8c80 0x7febf81a7a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:53062/0 (socket says 172.21.2.102:53062) 2021-02-10T11:16:52.964 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:52.966 7febfe6c6700 1 -- 172.21.2.102:0/2545404726 learned_addr learned my addr 172.21.2.102:0/2545404726 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:16:52.964 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:52.967 7febfd6c4700 1 --2- 172.21.2.102:0/2545404726 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7febf80b8880 0x7febf81a7450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:16:52.964 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:52.967 7febfe6c6700 1 -- 172.21.2.102:0/2545404726 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7febf80b8880 msgr2=0x7febf81a7450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:16:52.964 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:52.967 7febfe6c6700 1 --2- 172.21.2.102:0/2545404726 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7febf80b8880 0x7febf81a7450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:52.964 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:52.967 7febfe6c6700 1 -- 172.21.2.102:0/2545404726 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7febf80b7db0 msgr2=0x7febf80b8260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:16:52.964 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:52.967 7febfe6c6700 1 --2- 172.21.2.102:0/2545404726 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7febf80b7db0 0x7febf80b8260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:52.964 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:52.967 7febfe6c6700 1 -- 172.21.2.102:0/2545404726 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febf8190c10 con 0x7febf80b8c80 2021-02-10T11:16:52.964 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:52.967 7febfd6c4700 1 --2- 172.21.2.102:0/2545404726 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7febf80b8880 0x7febf81a7450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:16:52.964 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:52.967 7febfe6c6700 1 --2- 172.21.2.102:0/2545404726 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7febf80b8c80 0x7febf81a7a70 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 rx=0x7febf0001060 tx=0x7febf000a930).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:16:52.964 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:52.967 7febea7fc700 1 -- 172.21.2.102:0/2545404726 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7febf000ad20 con 0x7febf80b8c80 2021-02-10T11:16:52.964 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:52.967 7febea7fc700 1 -- 172.21.2.102:0/2545404726 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febf00070d0 con 0x7febf80b8c80 2021-02-10T11:16:52.964 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:52.967 7fec000a9700 1 -- 172.21.2.102:0/2545404726 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febf8190c10 con 0x7febf80b8c80 2021-02-10T11:16:52.964 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:52.967 7febea7fc700 1 -- 172.21.2.102:0/2545404726 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7febf00043a0 con 0x7febf80b8c80 2021-02-10T11:16:52.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:52.968 7fec000a9700 1 -- 172.21.2.102:0/2545404726 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7febf8190c10 con 0x7febf80b8c80 2021-02-10T11:16:52.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:52.968 7febea7fc700 1 -- 172.21.2.102:0/2545404726 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7febf0001a00 con 0x7febf80b8c80 2021-02-10T11:16:52.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:52.968 7febea7fc700 1 --2- 172.21.2.102:0/2545404726 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7febe0018730 0x7febe001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:16:52.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:52.968 7febea7fc700 1 -- 172.21.2.102:0/2545404726 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7febf0032610 con 0x7febf80b8c80 2021-02-10T11:16:52.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:52.968 7febfdec5700 1 --2- 172.21.2.102:0/2545404726 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7febe0018730 0x7febe001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:16:52.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:52.969 7febfdec5700 1 --2- 172.21.2.102:0/2545404726 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7febe0018730 0x7febe001ac10 crc :-1 s=READY pgs=323 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:16:52.991 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:52.999 7fec000a9700 1 -- 172.21.2.102:0/2545404726 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febf8105b30 con 0x7febf80b8c80 2021-02-10T11:16:52.993 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:53.001 7febea7fc700 1 -- 172.21.2.102:0/2545404726 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7febf00010a0 con 0x7febf80b8c80 2021-02-10T11:16:53.121 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:53.128 7fec000a9700 1 -- 172.21.2.102:0/2545404726 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344439", "target": ["mgr", ""]}) v1 -- 0x7febf80f0b50 con 0x7febe0018730 2021-02-10T11:16:53.124 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:53.131 7febea7fc700 1 -- 172.21.2.102:0/2545404726 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7febec0067f0 con 0x7febe0018730 2021-02-10T11:16:53.128 INFO:teuthology.orchestra.run.gibba002.stdout:/volumes/_nogroup/subvolume_0000000000344439/800c5e75-544e-484b-bc8f-9e6e31b2f20b 2021-02-10T11:16:53.129 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:53.137 7fec000a9700 1 -- 172.21.2.102:0/2545404726 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7febe0018730 msgr2=0x7febe001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:16:53.129 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:53.137 7fec000a9700 1 --2- 172.21.2.102:0/2545404726 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7febe0018730 0x7febe001ac10 crc :-1 s=READY pgs=323 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:53.129 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:53.137 7fec000a9700 1 -- 172.21.2.102:0/2545404726 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7febf80b8c80 msgr2=0x7febf81a7a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:16:53.129 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:53.137 7fec000a9700 1 --2- 172.21.2.102:0/2545404726 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7febf80b8c80 0x7febf81a7a70 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 rx=0x7febf0001060 tx=0x7febf000a930).stop 2021-02-10T11:16:53.130 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:53.137 7fec000a9700 1 -- 172.21.2.102:0/2545404726 shutdown_connections 2021-02-10T11:16:53.130 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:53.137 7fec000a9700 1 --2- 172.21.2.102:0/2545404726 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7febf80b8c80 0x7febf81a7a70 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:53.130 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:53.137 7fec000a9700 1 --2- 172.21.2.102:0/2545404726 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7febf80b8880 0x7febf81a7450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:53.130 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:53.137 7fec000a9700 1 --2- 172.21.2.102:0/2545404726 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7febe0018730 0x7febe001ac10 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:53.130 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:53.137 7fec000a9700 1 --2- 172.21.2.102:0/2545404726 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7febf80b7db0 0x7febf80b8260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:16:53.130 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:53.137 7fec000a9700 1 -- 172.21.2.102:0/2545404726 >> 172.21.2.102:0/2545404726 conn(0x7febf8094e10 msgr2=0x7febf80a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:16:53.130 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:53.137 7fec000a9700 1 -- 172.21.2.102:0/2545404726 shutdown_connections 2021-02-10T11:16:53.131 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:16:53.137 7fec000a9700 1 -- 172.21.2.102:0/2545404726 wait complete. 2021-02-10T11:16:53.137 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000344439 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000344439/800c5e75-544e-484b-bc8f-9e6e31b2f20b 2021-02-10T11:16:53.137 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:16:53.137 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:16:53.137 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:16:53.474 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:16:53.474 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:16:53.475 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344439/800c5e75-544e-484b-bc8f-9e6e31b2f20b/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:16:53.854 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:16:53.855 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:16:53.855 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0242589 s, 43.2 MB/s 2021-02-10T11:16:53.856 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:16:53.856 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:16:53.856 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:16:54.192 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:16:54.193 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:16:54.193 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344439/800c5e75-544e-484b-bc8f-9e6e31b2f20b/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:16:54.570 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:16:54.570 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:16:54.570 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0250277 s, 41.9 MB/s 2021-02-10T11:16:54.571 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:16:54.571 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:16:54.571 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:16:54.907 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:16:54.908 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:16:54.908 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344439/800c5e75-544e-484b-bc8f-9e6e31b2f20b/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:16:55.285 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:16:55.285 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:16:55.285 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0235426 s, 44.5 MB/s 2021-02-10T11:16:55.286 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:16:55.286 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:16:55.286 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:16:55.621 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:16:55.622 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:16:55.622 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344439/800c5e75-544e-484b-bc8f-9e6e31b2f20b/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:16:56.012 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:16:56.014 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:16:56.015 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0242307 s, 43.3 MB/s 2021-02-10T11:16:56.016 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:16:56.016 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:16:56.016 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:16:56.355 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:16:56.355 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:16:56.356 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344439/800c5e75-544e-484b-bc8f-9e6e31b2f20b/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:16:56.738 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:16:56.739 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:16:56.739 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0239693 s, 43.7 MB/s 2021-02-10T11:16:56.740 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:16:56.741 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:16:56.741 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:16:57.089 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:16:57.090 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:16:57.090 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344439/800c5e75-544e-484b-bc8f-9e6e31b2f20b/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:16:57.471 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:16:57.471 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:16:57.471 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0247217 s, 42.4 MB/s 2021-02-10T11:16:57.473 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:16:57.473 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:16:57.474 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:16:57.810 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:16:57.810 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:16:57.811 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344439/800c5e75-544e-484b-bc8f-9e6e31b2f20b/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:16:58.119 DEBUG:teuthology.orchestra.run.gibba002:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:16:58.123 DEBUG:teuthology.orchestra.run.gibba013:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:16:58.190 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:16:58.190 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:16:58.190 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0253849 s, 41.3 MB/s 2021-02-10T11:16:58.190 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:16:58.191 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:16:58.191 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:16:58.457 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:16:58.458 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:16:58.458 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344439/800c5e75-544e-484b-bc8f-9e6e31b2f20b/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:16:58.832 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:16:58.832 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:16:58.832 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0260598 s, 40.2 MB/s 2021-02-10T11:16:58.833 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:16:58.833 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:16:58.834 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:16:59.172 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:16:59.172 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:16:59.173 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344439/800c5e75-544e-484b-bc8f-9e6e31b2f20b/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:16:59.548 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:16:59.548 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:16:59.548 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0245025 s, 42.8 MB/s 2021-02-10T11:16:59.549 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:16:59.549 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:16:59.549 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:16:59.885 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:16:59.885 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:16:59.886 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344439/800c5e75-544e-484b-bc8f-9e6e31b2f20b/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:17:00.262 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:17:00.263 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:17:00.263 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0242297 s, 43.3 MB/s 2021-02-10T11:17:00.264 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000344440 2021-02-10T11:17:00.697 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:00.687 7f294bf36700 1 Processor -- start 2021-02-10T11:17:00.697 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:00.688 7f294bf36700 1 -- start start 2021-02-10T11:17:00.698 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:00.688 7f294bf36700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f29441b0470 0x7f29441b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:00.698 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:00.688 7f294bf36700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f29440f20e0 0x7f29440f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:00.698 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:00.688 7f294bf36700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29440f0b50 con 0x7f29440ef610 2021-02-10T11:17:00.698 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:00.688 7f294bf36700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29440f0820 con 0x7f29440f20e0 2021-02-10T11:17:00.698 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:00.688 7f294bf36700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29440f24e0 con 0x7f29441b0470 2021-02-10T11:17:00.699 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:00.688 7f294a553700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f29440f20e0 0x7f29440f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:17:00.699 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:00.688 7f2949d52700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f29441b0470 0x7f29441b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:17:00.699 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:00.688 7f2949551700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f29440ef610 0x7f29440efa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:45290/0 (socket says 172.21.2.102:45290) 2021-02-10T11:17:00.699 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:00.688 7f294a553700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f29440f20e0 0x7f29440f0200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.113:3300/0 says I am v2:172.21.2.102:56496/0 (socket says 172.21.2.102:56496) 2021-02-10T11:17:00.699 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:00.688 7f2949551700 1 -- 172.21.2.102:0/1847189388 learned_addr learned my addr 172.21.2.102:0/1847189388 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:17:00.699 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:00.688 7f2949d52700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f29441b0470 0x7f29441b0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.113:3301/0 says I am v2:172.21.2.102:58602/0 (socket says 172.21.2.102:58602) 2021-02-10T11:17:00.699 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:00.689 7f2949d52700 1 -- 172.21.2.102:0/1847189388 >> v2:172.21.2.113:3300/0 conn(0x7f29440f20e0 msgr2=0x7f29440f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:17:00.699 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:00.689 7f2949d52700 1 --2- 172.21.2.102:0/1847189388 >> v2:172.21.2.113:3300/0 conn(0x7f29440f20e0 0x7f29440f0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:00.699 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:00.689 7f2949d52700 1 -- 172.21.2.102:0/1847189388 >> v1:172.21.2.102:6789/0 conn(0x7f29440ef610 legacy=0x7f29440efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:17:00.699 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:00.689 7f2949d52700 1 -- 172.21.2.102:0/1847189388 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29440e52e0 con 0x7f29441b0470 2021-02-10T11:17:00.699 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:00.689 7f2949d52700 1 --2- 172.21.2.102:0/1847189388 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f29441b0470 0x7f29441b0870 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7f2938006650 tx=0x7f2938009a80).ready entity=mon.2 client_cookie=78d5af6d468b8258 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:17:00.699 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:00.689 7f2948d50700 1 -- 172.21.2.102:0/1847189388 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f29380009b0 con 0x7f29441b0470 2021-02-10T11:17:00.700 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:00.689 7f2948d50700 1 -- 172.21.2.102:0/1847189388 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f293800c0c0 con 0x7f29441b0470 2021-02-10T11:17:00.700 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:00.689 7f2948d50700 1 -- 172.21.2.102:0/1847189388 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f293800c510 con 0x7f29441b0470 2021-02-10T11:17:00.700 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:00.689 7f294bf36700 1 -- 172.21.2.102:0/1847189388 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f29441b0470 msgr2=0x7f29441b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:17:00.700 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:00.689 7f294bf36700 1 --2- 172.21.2.102:0/1847189388 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f29441b0470 0x7f29441b0870 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7f2938006650 tx=0x7f2938009a80).stop 2021-02-10T11:17:00.700 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:00.690 7f294bf36700 1 -- 172.21.2.102:0/1847189388 shutdown_connections 2021-02-10T11:17:00.700 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:00.690 7f294bf36700 1 --2- 172.21.2.102:0/1847189388 >> v2:172.21.2.113:3300/0 conn(0x7f29440f20e0 0x7f29440f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:00.701 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:00.690 7f294bf36700 1 --2- 172.21.2.102:0/1847189388 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f29441b0470 0x7f29441b0870 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:00.701 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:00.690 7f294bf36700 1 -- 172.21.2.102:0/1847189388 >> 172.21.2.102:0/1847189388 conn(0x7f29440ce3e0 msgr2=0x7f29440e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:17:00.701 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:00.690 7f294bf36700 1 -- 172.21.2.102:0/1847189388 shutdown_connections 2021-02-10T11:17:00.701 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:00.690 7f294bf36700 1 -- 172.21.2.102:0/1847189388 wait complete. 2021-02-10T11:17:00.701 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:00.705 7f294bf36700 1 Processor -- start 2021-02-10T11:17:00.701 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:00.705 7f294bf36700 1 -- start start 2021-02-10T11:17:00.703 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:00.705 7f294bf36700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f29440b0a10 0x7f29440a8240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:00.703 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:00.705 7f294bf36700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f29440a8860 0x7f29440af760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:00.703 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:00.705 7f294bf36700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f29440afd80 0x7f29440b0180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:00.704 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:00.705 7f294bf36700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29440a9010 con 0x7f29440a8860 2021-02-10T11:17:00.704 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:00.705 7f294bf36700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29440a8c60 con 0x7f29440b0a10 2021-02-10T11:17:00.704 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:00.705 7f294bf36700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29440a9590 con 0x7f29440afd80 2021-02-10T11:17:00.704 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:00.705 7f2949551700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f29440a8860 0x7f29440af760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:17:00.704 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:00.705 7f2949551700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f29440a8860 0x7f29440af760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:53068/0 (socket says 172.21.2.102:53068) 2021-02-10T11:17:00.704 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:00.705 7f2949551700 1 -- 172.21.2.102:0/3024022649 learned_addr learned my addr 172.21.2.102:0/3024022649 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:17:00.704 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:00.706 7f2949d52700 1 --2- 172.21.2.102:0/3024022649 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f29440b0a10 0x7f29440a8240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:17:00.704 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:00.706 7f294a553700 1 --2- 172.21.2.102:0/3024022649 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f29440afd80 0x7f29440b0180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:17:00.705 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:00.706 7f2949551700 1 -- 172.21.2.102:0/3024022649 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f29440afd80 msgr2=0x7f29440b0180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:17:00.705 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:00.706 7f2949551700 1 --2- 172.21.2.102:0/3024022649 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f29440afd80 0x7f29440b0180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:00.705 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:00.706 7f2949551700 1 -- 172.21.2.102:0/3024022649 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f29440b0a10 msgr2=0x7f29440a8240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:17:00.705 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:00.706 7f2949551700 1 --2- 172.21.2.102:0/3024022649 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f29440b0a10 0x7f29440a8240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:00.705 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:00.706 7f2949551700 1 -- 172.21.2.102:0/3024022649 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29440a97b0 con 0x7f29440a8860 2021-02-10T11:17:00.705 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:00.706 7f2949551700 1 --2- 172.21.2.102:0/3024022649 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f29440a8860 0x7f29440af760 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 rx=0x7f2940001c60 tx=0x7f294000a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:17:00.705 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:00.706 7f29367fc700 1 -- 172.21.2.102:0/3024022649 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f294000acc0 con 0x7f29440a8860 2021-02-10T11:17:00.705 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:00.706 7f29367fc700 1 -- 172.21.2.102:0/3024022649 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2940004140 con 0x7f29440a8860 2021-02-10T11:17:00.705 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:00.706 7f29367fc700 1 -- 172.21.2.102:0/3024022649 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2940004620 con 0x7f29440a8860 2021-02-10T11:17:00.705 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:00.706 7f294bf36700 1 -- 172.21.2.102:0/3024022649 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29440a97b0 con 0x7f29440a8860 2021-02-10T11:17:00.705 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:00.707 7f294bf36700 1 -- 172.21.2.102:0/3024022649 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29440a97b0 con 0x7f29440a8860 2021-02-10T11:17:00.705 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:00.707 7f29367fc700 1 -- 172.21.2.102:0/3024022649 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f2940000ce0 con 0x7f29440a8860 2021-02-10T11:17:00.705 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:00.707 7f29367fc700 1 --2- 172.21.2.102:0/3024022649 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f292c0186c0 0x7f292c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:00.705 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:00.707 7f29367fc700 1 -- 172.21.2.102:0/3024022649 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f2940032560 con 0x7f29440a8860 2021-02-10T11:17:00.706 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:00.707 7f2949d52700 1 --2- 172.21.2.102:0/3024022649 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f292c0186c0 0x7f292c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:17:00.706 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:00.708 7f2949d52700 1 --2- 172.21.2.102:0/3024022649 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f292c0186c0 0x7f292c01abc0 crc :-1 s=READY pgs=324 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:17:00.730 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:00.738 7f294bf36700 1 -- 172.21.2.102:0/3024022649 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2944105b40 con 0x7f29440a8860 2021-02-10T11:17:00.733 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:00.740 7f29367fc700 1 -- 172.21.2.102:0/3024022649 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f29400009b0 con 0x7f29440a8860 2021-02-10T11:17:00.858 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:00.866 7f294bf36700 1 -- 172.21.2.102:0/3024022649 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344440", "target": ["mgr", ""]}) v1 -- 0x7f2944105b40 con 0x7f292c0186c0 2021-02-10T11:17:00.884 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:00.891 7f29367fc700 1 -- 172.21.2.102:0/3024022649 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f2938000c90 con 0x7f292c0186c0 2021-02-10T11:17:00.891 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:00.898 7f294bf36700 1 -- 172.21.2.102:0/3024022649 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f292c0186c0 msgr2=0x7f292c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:17:00.891 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:00.898 7f294bf36700 1 --2- 172.21.2.102:0/3024022649 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f292c0186c0 0x7f292c01abc0 crc :-1 s=READY pgs=324 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:00.892 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:00.898 7f294bf36700 1 -- 172.21.2.102:0/3024022649 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f29440a8860 msgr2=0x7f29440af760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:17:00.892 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:00.898 7f294bf36700 1 --2- 172.21.2.102:0/3024022649 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f29440a8860 0x7f29440af760 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 rx=0x7f2940001c60 tx=0x7f294000a8d0).stop 2021-02-10T11:17:00.892 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:00.899 7f294bf36700 1 -- 172.21.2.102:0/3024022649 shutdown_connections 2021-02-10T11:17:00.892 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:00.899 7f294bf36700 1 --2- 172.21.2.102:0/3024022649 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f29440afd80 0x7f29440b0180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:00.892 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:00.899 7f294bf36700 1 --2- 172.21.2.102:0/3024022649 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f29440a8860 0x7f29440af760 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:00.892 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:00.899 7f294bf36700 1 --2- 172.21.2.102:0/3024022649 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f292c0186c0 0x7f292c01abc0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:00.892 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:00.899 7f294bf36700 1 --2- 172.21.2.102:0/3024022649 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f29440b0a10 0x7f29440a8240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:00.892 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:00.899 7f294bf36700 1 -- 172.21.2.102:0/3024022649 >> 172.21.2.102:0/3024022649 conn(0x7f29440bf4f0 msgr2=0x7f29440945e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:17:00.892 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:00.899 7f294bf36700 1 -- 172.21.2.102:0/3024022649 shutdown_connections 2021-02-10T11:17:00.892 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:00.899 7f294bf36700 1 -- 172.21.2.102:0/3024022649 wait complete. 2021-02-10T11:17:00.900 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000344440 2021-02-10T11:17:01.333 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:01.324 7f447894f700 1 Processor -- start 2021-02-10T11:17:01.333 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:01.324 7f447894f700 1 -- start start 2021-02-10T11:17:01.333 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:01.324 7f447894f700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f44741b0470 0x7f44741b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:01.333 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:01.324 7f447894f700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f44740ef610 0x7f44740efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:01.333 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:01.324 7f447894f700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44740f0b50 con 0x7f44740f0030 2021-02-10T11:17:01.334 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:01.324 7f447894f700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44740e52e0 con 0x7f44740ef610 2021-02-10T11:17:01.334 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:01.324 7f447894f700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44740e5490 con 0x7f44741b0470 2021-02-10T11:17:01.334 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:01.325 7f4471e1b700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f44740ef610 0x7f44740efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:17:01.334 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:01.325 7f4471e1b700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f44740ef610 0x7f44740efa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:53076/0 (socket says 172.21.2.102:53076) 2021-02-10T11:17:01.334 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:01.325 7f4471e1b700 1 -- 172.21.2.102:0/3391599153 learned_addr learned my addr 172.21.2.102:0/3391599153 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:17:01.334 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:01.325 7f447261c700 1 --2- 172.21.2.102:0/3391599153 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f44741b0470 0x7f44741b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:17:01.334 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:01.325 7f4471e1b700 1 -- 172.21.2.102:0/3391599153 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f44741b0470 msgr2=0x7f44741b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:17:01.334 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:01.325 7f4471e1b700 1 --2- 172.21.2.102:0/3391599153 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f44741b0470 0x7f44741b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:01.334 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:01.325 7f4471e1b700 1 -- 172.21.2.102:0/3391599153 >> v1:172.21.2.113:6789/0 conn(0x7f44740f0030 legacy=0x7f44740f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:17:01.334 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:01.325 7f4471e1b700 1 -- 172.21.2.102:0/3391599153 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44740f0430 con 0x7f44740ef610 2021-02-10T11:17:01.334 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:01.325 7f447261c700 1 --2- 172.21.2.102:0/3391599153 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f44741b0470 0x7f44741b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:17:01.334 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:01.325 7f4471e1b700 1 --2- 172.21.2.102:0/3391599153 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f44740ef610 0x7f44740efa10 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 rx=0x7f4464006700 tx=0x7f4464009b80).ready entity=mon.0 client_cookie=2f2dee73a44cbb94 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:17:01.335 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:01.326 7f447161a700 1 -- 172.21.2.102:0/3391599153 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f446400c0c0 con 0x7f44740ef610 2021-02-10T11:17:01.335 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:01.326 7f447161a700 1 -- 172.21.2.102:0/3391599153 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f446400c340 con 0x7f44740ef610 2021-02-10T11:17:01.335 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:01.326 7f447161a700 1 -- 172.21.2.102:0/3391599153 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f446400c860 con 0x7f44740ef610 2021-02-10T11:17:01.335 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:01.326 7f447894f700 1 -- 172.21.2.102:0/3391599153 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f44740ef610 msgr2=0x7f44740efa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:17:01.335 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:01.326 7f447894f700 1 --2- 172.21.2.102:0/3391599153 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f44740ef610 0x7f44740efa10 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 rx=0x7f4464006700 tx=0x7f4464009b80).stop 2021-02-10T11:17:01.335 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:01.326 7f447894f700 1 -- 172.21.2.102:0/3391599153 shutdown_connections 2021-02-10T11:17:01.335 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:01.326 7f447894f700 1 --2- 172.21.2.102:0/3391599153 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f44740ef610 0x7f44740efa10 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:01.335 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:01.326 7f447894f700 1 --2- 172.21.2.102:0/3391599153 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f44741b0470 0x7f44741b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:01.335 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:01.326 7f447894f700 1 -- 172.21.2.102:0/3391599153 >> 172.21.2.102:0/3391599153 conn(0x7f44740ce3e0 msgr2=0x7f44740e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:17:01.335 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:01.326 7f447894f700 1 -- 172.21.2.102:0/3391599153 shutdown_connections 2021-02-10T11:17:01.335 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:01.326 7f447894f700 1 -- 172.21.2.102:0/3391599153 wait complete. 2021-02-10T11:17:01.336 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:01.341 7f447894f700 1 Processor -- start 2021-02-10T11:17:01.336 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:01.341 7f447894f700 1 -- start start 2021-02-10T11:17:01.336 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:01.341 7f447894f700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f44740b20b0 0x7f44740ac970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:01.336 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:01.341 7f447894f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f44740b24b0 0x7f4474191dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:01.336 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:01.341 7f447894f700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f44740acf90 0x7f44741923e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:01.336 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:01.341 7f447894f700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44740b1b30 con 0x7f44740b24b0 2021-02-10T11:17:01.336 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:01.341 7f447894f700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44741a04e0 con 0x7f44740b20b0 2021-02-10T11:17:01.336 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:01.341 7f447894f700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44741a06c0 con 0x7f44740acf90 2021-02-10T11:17:01.338 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:01.342 7f4471e1b700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f44740b24b0 0x7f4474191dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:17:01.338 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:01.342 7f4471e1b700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f44740b24b0 0x7f4474191dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:53080/0 (socket says 172.21.2.102:53080) 2021-02-10T11:17:01.338 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:01.342 7f4471e1b700 1 -- 172.21.2.102:0/1008736509 learned_addr learned my addr 172.21.2.102:0/1008736509 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:17:01.338 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:01.342 7f447261c700 1 --2- 172.21.2.102:0/1008736509 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f44740b20b0 0x7f44740ac970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:17:01.338 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:01.342 7f4471e1b700 1 -- 172.21.2.102:0/1008736509 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f44740acf90 msgr2=0x7f44741923e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:17:01.338 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:01.342 7f4472e1d700 1 --2- 172.21.2.102:0/1008736509 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f44740acf90 0x7f44741923e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:17:01.338 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:01.342 7f4471e1b700 1 --2- 172.21.2.102:0/1008736509 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f44740acf90 0x7f44741923e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:01.338 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:01.342 7f4471e1b700 1 -- 172.21.2.102:0/1008736509 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f44740b20b0 msgr2=0x7f44740ac970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:17:01.338 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:01.342 7f4471e1b700 1 --2- 172.21.2.102:0/1008736509 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f44740b20b0 0x7f44740ac970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:01.338 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:01.342 7f4471e1b700 1 -- 172.21.2.102:0/1008736509 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4474192be0 con 0x7f44740b24b0 2021-02-10T11:17:01.339 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:01.342 7f447261c700 1 --2- 172.21.2.102:0/1008736509 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f44740b20b0 0x7f44740ac970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:17:01.339 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:01.342 7f4472e1d700 1 --2- 172.21.2.102:0/1008736509 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f44740acf90 0x7f44741923e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:17:01.339 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:01.342 7f4471e1b700 1 --2- 172.21.2.102:0/1008736509 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f44740b24b0 0x7f4474191dc0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 rx=0x7f446400b5e0 tx=0x7f446400b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:17:01.339 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:01.343 7f446affd700 1 -- 172.21.2.102:0/1008736509 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f446400bd70 con 0x7f44740b24b0 2021-02-10T11:17:01.339 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:01.343 7f446affd700 1 -- 172.21.2.102:0/1008736509 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4464002030 con 0x7f44740b24b0 2021-02-10T11:17:01.339 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:01.343 7f447894f700 1 -- 172.21.2.102:0/1008736509 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4474192be0 con 0x7f44740b24b0 2021-02-10T11:17:01.339 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:01.343 7f447894f700 1 -- 172.21.2.102:0/1008736509 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4474193170 con 0x7f44740b24b0 2021-02-10T11:17:01.339 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:01.343 7f446affd700 1 -- 172.21.2.102:0/1008736509 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4464002550 con 0x7f44740b24b0 2021-02-10T11:17:01.339 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:01.343 7f446affd700 1 -- 172.21.2.102:0/1008736509 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4464000c90 con 0x7f44740b24b0 2021-02-10T11:17:01.339 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:01.343 7f446affd700 1 --2- 172.21.2.102:0/1008736509 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4458018730 0x7f445801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:01.339 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:01.343 7f446affd700 1 -- 172.21.2.102:0/1008736509 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f4464032810 con 0x7f44740b24b0 2021-02-10T11:17:01.339 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:01.344 7f447261c700 1 --2- 172.21.2.102:0/1008736509 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4458018730 0x7f445801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:17:01.339 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:01.344 7f447261c700 1 --2- 172.21.2.102:0/1008736509 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4458018730 0x7f445801ac10 crc :-1 s=READY pgs=325 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:17:01.367 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:01.374 7f447894f700 1 -- 172.21.2.102:0/1008736509 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4474105af0 con 0x7f44740b24b0 2021-02-10T11:17:01.369 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:01.377 7f446affd700 1 -- 172.21.2.102:0/1008736509 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f4464000940 con 0x7f44740b24b0 2021-02-10T11:17:01.497 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:01.505 7f447894f700 1 -- 172.21.2.102:0/1008736509 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344440", "target": ["mgr", ""]}) v1 -- 0x7f44740b1b30 con 0x7f4458018730 2021-02-10T11:17:01.501 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:01.508 7f446affd700 1 -- 172.21.2.102:0/1008736509 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f445c006ad0 con 0x7f4458018730 2021-02-10T11:17:01.504 INFO:teuthology.orchestra.run.gibba002.stdout:/volumes/_nogroup/subvolume_0000000000344440/f81f3ed6-0ef5-480e-b801-c4a82cc4540f 2021-02-10T11:17:01.506 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:01.513 7f447894f700 1 -- 172.21.2.102:0/1008736509 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4458018730 msgr2=0x7f445801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:17:01.506 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:01.513 7f447894f700 1 --2- 172.21.2.102:0/1008736509 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4458018730 0x7f445801ac10 crc :-1 s=READY pgs=325 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:01.506 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:01.513 7f447894f700 1 -- 172.21.2.102:0/1008736509 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f44740b24b0 msgr2=0x7f4474191dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:17:01.506 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:01.513 7f447894f700 1 --2- 172.21.2.102:0/1008736509 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f44740b24b0 0x7f4474191dc0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 rx=0x7f446400b5e0 tx=0x7f446400b980).stop 2021-02-10T11:17:01.506 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:01.513 7f447894f700 1 -- 172.21.2.102:0/1008736509 shutdown_connections 2021-02-10T11:17:01.506 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:01.513 7f447894f700 1 --2- 172.21.2.102:0/1008736509 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f44740acf90 0x7f44741923e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:01.506 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:01.513 7f447894f700 1 --2- 172.21.2.102:0/1008736509 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f44740b24b0 0x7f4474191dc0 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:01.507 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:01.513 7f447894f700 1 --2- 172.21.2.102:0/1008736509 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4458018730 0x7f445801ac10 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:01.507 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:01.513 7f447894f700 1 --2- 172.21.2.102:0/1008736509 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f44740b20b0 0x7f44740ac970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:01.507 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:01.513 7f447894f700 1 -- 172.21.2.102:0/1008736509 >> 172.21.2.102:0/1008736509 conn(0x7f44740bf4e0 msgr2=0x7f44740be3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:17:01.507 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:01.513 7f447894f700 1 -- 172.21.2.102:0/1008736509 shutdown_connections 2021-02-10T11:17:01.507 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:01.514 7f447894f700 1 -- 172.21.2.102:0/1008736509 wait complete. 2021-02-10T11:17:01.514 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000344440 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000344440/f81f3ed6-0ef5-480e-b801-c4a82cc4540f 2021-02-10T11:17:01.514 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:17:01.514 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:17:01.514 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:17:01.850 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:17:01.850 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:17:01.850 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344440/f81f3ed6-0ef5-480e-b801-c4a82cc4540f/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:17:02.231 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:17:02.231 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:17:02.231 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0234187 s, 44.8 MB/s 2021-02-10T11:17:02.232 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:17:02.232 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:17:02.232 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:17:02.569 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:17:02.569 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:17:02.570 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344440/f81f3ed6-0ef5-480e-b801-c4a82cc4540f/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:17:02.946 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:17:02.946 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:17:02.946 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0239766 s, 43.7 MB/s 2021-02-10T11:17:02.947 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:17:02.947 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:17:02.947 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:17:03.284 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:17:03.285 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:17:03.285 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344440/f81f3ed6-0ef5-480e-b801-c4a82cc4540f/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:17:03.657 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:17:03.657 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:17:03.657 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0241968 s, 43.3 MB/s 2021-02-10T11:17:03.658 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:17:03.659 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:17:03.659 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:17:03.994 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:17:03.994 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:17:03.995 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344440/f81f3ed6-0ef5-480e-b801-c4a82cc4540f/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:17:04.372 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:17:04.372 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:17:04.372 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0234085 s, 44.8 MB/s 2021-02-10T11:17:04.373 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:17:04.373 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:17:04.374 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:17:04.709 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:17:04.710 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:17:04.710 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344440/f81f3ed6-0ef5-480e-b801-c4a82cc4540f/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:17:05.088 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:17:05.088 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:17:05.088 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0251404 s, 41.7 MB/s 2021-02-10T11:17:05.089 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:17:05.089 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:17:05.089 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:17:05.424 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:17:05.424 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:17:05.425 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344440/f81f3ed6-0ef5-480e-b801-c4a82cc4540f/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:17:05.801 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:17:05.801 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:17:05.801 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0241331 s, 43.4 MB/s 2021-02-10T11:17:05.802 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:17:05.802 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:17:05.802 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:17:06.138 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:17:06.139 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:17:06.139 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344440/f81f3ed6-0ef5-480e-b801-c4a82cc4540f/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:17:06.514 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:17:06.514 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:17:06.514 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0233474 s, 44.9 MB/s 2021-02-10T11:17:06.515 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:17:06.515 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:17:06.515 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:17:06.851 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:17:06.851 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:17:06.851 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344440/f81f3ed6-0ef5-480e-b801-c4a82cc4540f/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:17:07.227 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:17:07.227 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:17:07.228 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0239549 s, 43.8 MB/s 2021-02-10T11:17:07.228 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:17:07.229 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:17:07.229 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:17:07.566 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:17:07.566 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:17:07.567 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344440/f81f3ed6-0ef5-480e-b801-c4a82cc4540f/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:17:07.945 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:17:07.945 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:17:07.945 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0247184 s, 42.4 MB/s 2021-02-10T11:17:07.946 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:17:07.946 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:17:07.946 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:17:08.281 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:17:08.281 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:17:08.282 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344440/f81f3ed6-0ef5-480e-b801-c4a82cc4540f/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:17:08.656 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:17:08.656 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:17:08.656 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0243368 s, 43.1 MB/s 2021-02-10T11:17:08.657 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000344441 2021-02-10T11:17:09.090 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:09.080 7f7eeb8fe700 1 Processor -- start 2021-02-10T11:17:09.090 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:09.080 7f7eeb8fe700 1 -- start start 2021-02-10T11:17:09.090 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:09.081 7f7eeb8fe700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f7ee41b0470 0x7f7ee41b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:09.090 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:09.081 7f7eeb8fe700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7ee40ef610 0x7f7ee40efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:09.090 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:09.081 7f7eeb8fe700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ee40f0b50 con 0x7f7ee40f0030 2021-02-10T11:17:09.091 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:09.081 7f7eeb8fe700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ee40e52e0 con 0x7f7ee41b0470 2021-02-10T11:17:09.091 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:09.081 7f7eeb8fe700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ee40e54d0 con 0x7f7ee40ef610 2021-02-10T11:17:09.091 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:09.082 7f7ee971a700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f7ee41b0470 0x7f7ee41b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:17:09.091 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:09.082 7f7ee8f19700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7ee40ef610 0x7f7ee40efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:17:09.091 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:09.082 7f7ee971a700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f7ee41b0470 0x7f7ee41b0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.113:3300/0 says I am v2:172.21.2.102:56518/0 (socket says 172.21.2.102:56518) 2021-02-10T11:17:09.091 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:09.082 7f7ee971a700 1 -- 172.21.2.102:0/50739306 learned_addr learned my addr 172.21.2.102:0/50739306 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:17:09.091 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:09.082 7f7ee8f19700 1 -- 172.21.2.102:0/50739306 >> v2:172.21.2.113:3300/0 conn(0x7f7ee41b0470 msgr2=0x7f7ee41b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:17:09.091 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:09.082 7f7ee8f19700 1 --2- 172.21.2.102:0/50739306 >> v2:172.21.2.113:3300/0 conn(0x7f7ee41b0470 0x7f7ee41b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:09.091 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:09.082 7f7ee8f19700 1 -- 172.21.2.102:0/50739306 >> v1:172.21.2.113:6789/0 conn(0x7f7ee40f0030 legacy=0x7f7ee40f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:17:09.091 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:09.082 7f7ee8f19700 1 -- 172.21.2.102:0/50739306 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ee40f0430 con 0x7f7ee40ef610 2021-02-10T11:17:09.091 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:09.082 7f7ee971a700 1 --2- 172.21.2.102:0/50739306 >> v2:172.21.2.113:3300/0 conn(0x7f7ee41b0470 0x7f7ee41b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:17:09.091 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:09.082 7f7ee8f19700 1 --2- 172.21.2.102:0/50739306 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7ee40ef610 0x7f7ee40efa10 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f7ed4006650 tx=0x7f7ed4009a80).ready entity=mon.2 client_cookie=1624a6aca7e30d60 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:17:09.091 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:09.083 7f7edbfff700 1 -- 172.21.2.102:0/50739306 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7ed400c0c0 con 0x7f7ee40ef610 2021-02-10T11:17:09.092 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:09.083 7f7edbfff700 1 -- 172.21.2.102:0/50739306 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ed400c340 con 0x7f7ee40ef610 2021-02-10T11:17:09.092 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:09.083 7f7edbfff700 1 -- 172.21.2.102:0/50739306 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7ed400c860 con 0x7f7ee40ef610 2021-02-10T11:17:09.092 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:09.083 7f7eeb8fe700 1 -- 172.21.2.102:0/50739306 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7ee40ef610 msgr2=0x7f7ee40efa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:17:09.092 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:09.083 7f7eeb8fe700 1 --2- 172.21.2.102:0/50739306 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7ee40ef610 0x7f7ee40efa10 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f7ed4006650 tx=0x7f7ed4009a80).stop 2021-02-10T11:17:09.092 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:09.083 7f7eeb8fe700 1 -- 172.21.2.102:0/50739306 shutdown_connections 2021-02-10T11:17:09.092 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:09.083 7f7eeb8fe700 1 --2- 172.21.2.102:0/50739306 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7ee40ef610 0x7f7ee40efa10 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:09.092 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:09.083 7f7eeb8fe700 1 --2- 172.21.2.102:0/50739306 >> v2:172.21.2.113:3300/0 conn(0x7f7ee41b0470 0x7f7ee41b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:09.092 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:09.084 7f7eeb8fe700 1 -- 172.21.2.102:0/50739306 >> 172.21.2.102:0/50739306 conn(0x7f7ee40ce3e0 msgr2=0x7f7ee40e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:17:09.092 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:09.084 7f7eeb8fe700 1 -- 172.21.2.102:0/50739306 shutdown_connections 2021-02-10T11:17:09.092 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:09.084 7f7eeb8fe700 1 -- 172.21.2.102:0/50739306 wait complete. 2021-02-10T11:17:09.094 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:09.098 7f7eeb8fe700 1 Processor -- start 2021-02-10T11:17:09.094 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:09.098 7f7eeb8fe700 1 -- start start 2021-02-10T11:17:09.094 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:09.099 7f7eeb8fe700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7ee4190920 0x7f7ee40b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:09.094 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:09.099 7f7eeb8fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7ee40b8200 0x7f7ee40b8600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:09.094 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:09.099 7f7eeb8fe700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7ee40c1d80 0x7f7ee40c2180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:09.094 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:09.099 7f7eeb8fe700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ee4190d20 con 0x7f7ee40b8200 2021-02-10T11:17:09.094 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:09.099 7f7eeb8fe700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ee40b8c20 con 0x7f7ee4190920 2021-02-10T11:17:09.094 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:09.099 7f7eeb8fe700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ee40c27a0 con 0x7f7ee40c1d80 2021-02-10T11:17:09.094 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:09.099 7f7ee8f19700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7ee40b8200 0x7f7ee40b8600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:17:09.094 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:09.099 7f7ee8f19700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7ee40b8200 0x7f7ee40b8600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:53092/0 (socket says 172.21.2.102:53092) 2021-02-10T11:17:09.094 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:09.099 7f7ee8f19700 1 -- 172.21.2.102:0/2905681671 learned_addr learned my addr 172.21.2.102:0/2905681671 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:17:09.095 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:09.099 7f7ee8f19700 1 -- 172.21.2.102:0/2905681671 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7ee40c1d80 msgr2=0x7f7ee40c2180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:17:09.095 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:09.099 7f7ee8f19700 1 --2- 172.21.2.102:0/2905681671 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7ee40c1d80 0x7f7ee40c2180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:17:09.095 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:09.099 7f7ee971a700 1 --2- 172.21.2.102:0/2905681671 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7ee4190920 0x7f7ee40b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:17:09.095 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:09.099 7f7ee8f19700 1 -- 172.21.2.102:0/2905681671 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7ee4190920 msgr2=0x7f7ee40b7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:17:09.095 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:09.099 7f7ee8f19700 1 --2- 172.21.2.102:0/2905681671 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7ee4190920 0x7f7ee40b7be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:09.095 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:09.099 7f7ee8f19700 1 -- 172.21.2.102:0/2905681671 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ee40c29c0 con 0x7f7ee40b8200 2021-02-10T11:17:09.095 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:09.099 7f7ee971a700 1 --2- 172.21.2.102:0/2905681671 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7ee4190920 0x7f7ee40b7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:17:09.095 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:09.099 7f7ee8f19700 1 --2- 172.21.2.102:0/2905681671 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7ee40b8200 0x7f7ee40b8600 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 rx=0x7f7ed4009630 tx=0x7f7ed400b900).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:17:09.095 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:09.100 7f7ed9ffb700 1 -- 172.21.2.102:0/2905681671 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7ed40093a0 con 0x7f7ee40b8200 2021-02-10T11:17:09.095 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:09.100 7f7ed9ffb700 1 -- 172.21.2.102:0/2905681671 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ed400bcf0 con 0x7f7ee40b8200 2021-02-10T11:17:09.095 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:09.100 7f7eeb8fe700 1 -- 172.21.2.102:0/2905681671 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ee40c29c0 con 0x7f7ee40b8200 2021-02-10T11:17:09.095 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:09.100 7f7eeb8fe700 1 -- 172.21.2.102:0/2905681671 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ee40c2f50 con 0x7f7ee40b8200 2021-02-10T11:17:09.095 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:09.100 7f7ed9ffb700 1 -- 172.21.2.102:0/2905681671 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7ed4005560 con 0x7f7ee40b8200 2021-02-10T11:17:09.096 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:09.100 7f7ed9ffb700 1 -- 172.21.2.102:0/2905681671 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f7ed400a000 con 0x7f7ee40b8200 2021-02-10T11:17:09.096 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:09.100 7f7ed9ffb700 1 --2- 172.21.2.102:0/2905681671 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7ec40186c0 0x7f7ec401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:09.096 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:09.100 7f7ed9ffb700 1 -- 172.21.2.102:0/2905681671 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f7ed40325e0 con 0x7f7ee40b8200 2021-02-10T11:17:09.096 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:09.100 7f7ee971a700 1 --2- 172.21.2.102:0/2905681671 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7ec40186c0 0x7f7ec401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:17:09.096 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:09.101 7f7ee971a700 1 --2- 172.21.2.102:0/2905681671 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7ec40186c0 0x7f7ec401abc0 crc :-1 s=READY pgs=326 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:17:09.123 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:09.131 7f7eeb8fe700 1 -- 172.21.2.102:0/2905681671 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ee4105ae0 con 0x7f7ee40b8200 2021-02-10T11:17:09.125 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:09.133 7f7ed9ffb700 1 -- 172.21.2.102:0/2905681671 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f7ed402cf60 con 0x7f7ee40b8200 2021-02-10T11:17:09.253 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:09.261 7f7eeb8fe700 1 -- 172.21.2.102:0/2905681671 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344441", "target": ["mgr", ""]}) v1 -- 0x7f7ee4190d20 con 0x7f7ec40186c0 2021-02-10T11:17:09.281 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:09.289 7f7ed9ffb700 1 -- 172.21.2.102:0/2905681671 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f7ee00009b0 con 0x7f7ec40186c0 2021-02-10T11:17:09.286 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:09.294 7f7eeb8fe700 1 -- 172.21.2.102:0/2905681671 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7ec40186c0 msgr2=0x7f7ec401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:17:09.286 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:09.294 7f7eeb8fe700 1 --2- 172.21.2.102:0/2905681671 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7ec40186c0 0x7f7ec401abc0 crc :-1 s=READY pgs=326 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:09.286 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:09.294 7f7eeb8fe700 1 -- 172.21.2.102:0/2905681671 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7ee40b8200 msgr2=0x7f7ee40b8600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:17:09.286 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:09.294 7f7eeb8fe700 1 --2- 172.21.2.102:0/2905681671 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7ee40b8200 0x7f7ee40b8600 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 rx=0x7f7ed4009630 tx=0x7f7ed400b900).stop 2021-02-10T11:17:09.287 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:09.294 7f7eeb8fe700 1 -- 172.21.2.102:0/2905681671 shutdown_connections 2021-02-10T11:17:09.287 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:09.294 7f7eeb8fe700 1 --2- 172.21.2.102:0/2905681671 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7ee40c1d80 0x7f7ee40c2180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:17:09.287 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:09.294 7f7eeb8fe700 1 --2- 172.21.2.102:0/2905681671 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7ee40b8200 0x7f7ee40b8600 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:09.287 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:09.294 7f7eeb8fe700 1 --2- 172.21.2.102:0/2905681671 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7ec40186c0 0x7f7ec401abc0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:09.287 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:09.294 7f7eeb8fe700 1 --2- 172.21.2.102:0/2905681671 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7ee4190920 0x7f7ee40b7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:09.287 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:09.294 7f7eeb8fe700 1 -- 172.21.2.102:0/2905681671 >> 172.21.2.102:0/2905681671 conn(0x7f7ee4094e10 msgr2=0x7f7ee40a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:17:09.287 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:09.294 7f7eeb8fe700 1 -- 172.21.2.102:0/2905681671 shutdown_connections 2021-02-10T11:17:09.287 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:09.294 7f7eeb8fe700 1 -- 172.21.2.102:0/2905681671 wait complete. 2021-02-10T11:17:09.295 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000344441 2021-02-10T11:17:09.727 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:09.717 7f5240bbb700 1 Processor -- start 2021-02-10T11:17:09.727 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:09.717 7f5240bbb700 1 -- start start 2021-02-10T11:17:09.727 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:09.718 7f5240bbb700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f523c1b0470 0x7f523c1b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:09.727 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:09.718 7f5240bbb700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f523c0f20e0 0x7f523c0f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:09.727 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:09.718 7f5240bbb700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f523c0f0b50 con 0x7f523c0ef610 2021-02-10T11:17:09.727 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:09.718 7f5240bbb700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f523c0f0820 con 0x7f523c1b0470 2021-02-10T11:17:09.727 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:09.718 7f5240bbb700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f523c0f24e0 con 0x7f523c0f20e0 2021-02-10T11:17:09.727 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:09.718 7f523a85e700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f523c1b0470 0x7f523c1b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:17:09.727 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:09.718 7f523a05d700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f523c0ef610 0x7f523c0efa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:45326/0 (socket says 172.21.2.102:45326) 2021-02-10T11:17:09.728 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:09.718 7f523a05d700 1 -- 172.21.2.102:0/2879560551 learned_addr learned my addr 172.21.2.102:0/2879560551 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:17:09.728 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:09.718 7f523a85e700 1 -- 172.21.2.102:0/2879560551 >> v2:172.21.2.113:3300/0 conn(0x7f523c0f20e0 msgr2=0x7f523c0f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:17:09.728 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:09.718 7f523a85e700 1 --2- 172.21.2.102:0/2879560551 >> v2:172.21.2.113:3300/0 conn(0x7f523c0f20e0 0x7f523c0f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:17:09.728 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:09.719 7f523a85e700 1 -- 172.21.2.102:0/2879560551 >> v1:172.21.2.102:6789/0 conn(0x7f523c0ef610 legacy=0x7f523c0efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:17:09.728 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:09.718 7f523985c700 1 -- 172.21.2.102:0/2879560551 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3986854967 0 0) 0x7f52240010b0 con 0x7f523c0ef610 2021-02-10T11:17:09.728 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:09.719 7f523a85e700 1 -- 172.21.2.102:0/2879560551 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f523c0e52e0 con 0x7f523c1b0470 2021-02-10T11:17:09.728 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:09.719 7f523a85e700 1 --2- 172.21.2.102:0/2879560551 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f523c1b0470 0x7f523c1b0870 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 rx=0x7f522c006650 tx=0x7f522c009a00).ready entity=mon.0 client_cookie=5b4a8f370ad7ef23 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:17:09.728 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:09.719 7f523985c700 1 -- 172.21.2.102:0/2879560551 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f522c009df0 con 0x7f523c1b0470 2021-02-10T11:17:09.728 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:09.719 7f523985c700 1 -- 172.21.2.102:0/2879560551 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f522c0009b0 con 0x7f523c1b0470 2021-02-10T11:17:09.728 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:09.719 7f523985c700 1 -- 172.21.2.102:0/2879560551 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f522c00c2b0 con 0x7f523c1b0470 2021-02-10T11:17:09.728 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:09.719 7f5240bbb700 1 -- 172.21.2.102:0/2879560551 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f523c1b0470 msgr2=0x7f523c1b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:17:09.728 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:09.719 7f5240bbb700 1 --2- 172.21.2.102:0/2879560551 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f523c1b0470 0x7f523c1b0870 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 rx=0x7f522c006650 tx=0x7f522c009a00).stop 2021-02-10T11:17:09.728 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:09.720 7f5240bbb700 1 -- 172.21.2.102:0/2879560551 shutdown_connections 2021-02-10T11:17:09.729 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:09.720 7f5240bbb700 1 --2- 172.21.2.102:0/2879560551 >> v2:172.21.2.113:3300/0 conn(0x7f523c0f20e0 0x7f523c0f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:17:09.729 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:09.720 7f5240bbb700 1 --2- 172.21.2.102:0/2879560551 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f523c1b0470 0x7f523c1b0870 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:09.729 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:09.720 7f5240bbb700 1 -- 172.21.2.102:0/2879560551 >> 172.21.2.102:0/2879560551 conn(0x7f523c0ce3e0 msgr2=0x7f523c0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:17:09.729 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:09.720 7f5240bbb700 1 -- 172.21.2.102:0/2879560551 shutdown_connections 2021-02-10T11:17:09.729 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:09.720 7f5240bbb700 1 -- 172.21.2.102:0/2879560551 wait complete. 2021-02-10T11:17:09.730 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:09.735 7f5240bbb700 1 Processor -- start 2021-02-10T11:17:09.730 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:09.735 7f5240bbb700 1 -- start start 2021-02-10T11:17:09.731 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:09.735 7f5240bbb700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f523c190ae0 0x7f523c1a7a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:09.731 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:09.735 7f5240bbb700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f523c1a80a0 0x7f523c08a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:09.731 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:09.735 7f5240bbb700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f523c0ab6a0 0x7f523c08aa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:09.731 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:09.735 7f5240bbb700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f523c190ee0 con 0x7f523c190ae0 2021-02-10T11:17:09.731 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:09.735 7f5240bbb700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f523c08b1d0 con 0x7f523c0ab6a0 2021-02-10T11:17:09.731 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:09.735 7f5240bbb700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f523c0bf490 con 0x7f523c1a80a0 2021-02-10T11:17:09.731 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:09.736 7f523a85e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f523c190ae0 0x7f523c1a7a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:17:09.731 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:09.736 7f523a85e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f523c190ae0 0x7f523c1a7a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:53110/0 (socket says 172.21.2.102:53110) 2021-02-10T11:17:09.731 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:09.736 7f523a85e700 1 -- 172.21.2.102:0/1351250739 learned_addr learned my addr 172.21.2.102:0/1351250739 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:17:09.731 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:09.736 7f523a05d700 1 --2- 172.21.2.102:0/1351250739 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f523c1a80a0 0x7f523c08a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:17:09.731 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:09.736 7f523b05f700 1 --2- 172.21.2.102:0/1351250739 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f523c0ab6a0 0x7f523c08aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:17:09.731 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:09.736 7f523a85e700 1 -- 172.21.2.102:0/1351250739 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f523c1a80a0 msgr2=0x7f523c08a430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:17:09.731 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:09.736 7f523a85e700 1 --2- 172.21.2.102:0/1351250739 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f523c1a80a0 0x7f523c08a430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:09.732 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:09.736 7f523a85e700 1 -- 172.21.2.102:0/1351250739 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f523c0ab6a0 msgr2=0x7f523c08aa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:17:09.732 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:09.736 7f523a85e700 1 --2- 172.21.2.102:0/1351250739 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f523c0ab6a0 0x7f523c08aa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:09.732 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:09.736 7f523a85e700 1 -- 172.21.2.102:0/1351250739 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f523c08b750 con 0x7f523c190ae0 2021-02-10T11:17:09.732 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:09.736 7f523a85e700 1 --2- 172.21.2.102:0/1351250739 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f523c190ae0 0x7f523c1a7a80 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 rx=0x7f522c0096a0 tx=0x7f522c00b9b0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:17:09.732 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:09.736 7f52337fe700 1 -- 172.21.2.102:0/1351250739 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f522c000940 con 0x7f523c190ae0 2021-02-10T11:17:09.732 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:09.736 7f52337fe700 1 -- 172.21.2.102:0/1351250739 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f522c0093a0 con 0x7f523c190ae0 2021-02-10T11:17:09.732 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:09.736 7f52337fe700 1 -- 172.21.2.102:0/1351250739 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f522c0022b0 con 0x7f523c190ae0 2021-02-10T11:17:09.732 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:09.736 7f5240bbb700 1 -- 172.21.2.102:0/1351250739 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f523c08b750 con 0x7f523c190ae0 2021-02-10T11:17:09.732 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:09.736 7f5240bbb700 1 -- 172.21.2.102:0/1351250739 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f523c08b750 con 0x7f523c190ae0 2021-02-10T11:17:09.732 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:09.737 7f52337fe700 1 -- 172.21.2.102:0/1351250739 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f522c0052e0 con 0x7f523c190ae0 2021-02-10T11:17:09.732 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:09.737 7f52337fe700 1 --2- 172.21.2.102:0/1351250739 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f52200186c0 0x7f522001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:09.732 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:09.737 7f52337fe700 1 -- 172.21.2.102:0/1351250739 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f522c0323b0 con 0x7f523c190ae0 2021-02-10T11:17:09.732 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:09.737 7f523a05d700 1 --2- 172.21.2.102:0/1351250739 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f52200186c0 0x7f522001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:17:09.733 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:09.738 7f523a05d700 1 --2- 172.21.2.102:0/1351250739 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f52200186c0 0x7f522001abc0 crc :-1 s=READY pgs=327 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:17:09.760 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:09.767 7f5240bbb700 1 -- 172.21.2.102:0/1351250739 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f523c105af0 con 0x7f523c190ae0 2021-02-10T11:17:09.762 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:09.770 7f52337fe700 1 -- 172.21.2.102:0/1351250739 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f522c000c90 con 0x7f523c190ae0 2021-02-10T11:17:09.890 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:09.898 7f5240bbb700 1 -- 172.21.2.102:0/1351250739 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344441", "target": ["mgr", ""]}) v1 -- 0x7f523c08b1d0 con 0x7f52200186c0 2021-02-10T11:17:09.893 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:09.900 7f52337fe700 1 -- 172.21.2.102:0/1351250739 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f5224000970 con 0x7f52200186c0 2021-02-10T11:17:09.893 INFO:teuthology.orchestra.run.gibba002.stdout:/volumes/_nogroup/subvolume_0000000000344441/45591a11-d715-4b07-9ce6-07ffd66b7c4d 2021-02-10T11:17:09.895 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:09.902 7f5240bbb700 1 -- 172.21.2.102:0/1351250739 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f52200186c0 msgr2=0x7f522001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:17:09.895 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:09.902 7f5240bbb700 1 --2- 172.21.2.102:0/1351250739 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f52200186c0 0x7f522001abc0 crc :-1 s=READY pgs=327 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:09.895 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:09.902 7f5240bbb700 1 -- 172.21.2.102:0/1351250739 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f523c190ae0 msgr2=0x7f523c1a7a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:17:09.895 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:09.902 7f5240bbb700 1 --2- 172.21.2.102:0/1351250739 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f523c190ae0 0x7f523c1a7a80 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 rx=0x7f522c0096a0 tx=0x7f522c00b9b0).stop 2021-02-10T11:17:09.895 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:09.902 7f5240bbb700 1 -- 172.21.2.102:0/1351250739 shutdown_connections 2021-02-10T11:17:09.895 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:09.902 7f5240bbb700 1 --2- 172.21.2.102:0/1351250739 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f52200186c0 0x7f522001abc0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:09.895 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:09.902 7f5240bbb700 1 --2- 172.21.2.102:0/1351250739 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f523c0ab6a0 0x7f523c08aa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:09.895 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:09.902 7f5240bbb700 1 --2- 172.21.2.102:0/1351250739 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f523c1a80a0 0x7f523c08a430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:09.895 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:09.902 7f5240bbb700 1 --2- 172.21.2.102:0/1351250739 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f523c190ae0 0x7f523c1a7a80 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:09.895 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:09.902 7f5240bbb700 1 -- 172.21.2.102:0/1351250739 >> 172.21.2.102:0/1351250739 conn(0x7f523c0c40c0 msgr2=0x7f523c0a2ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:17:09.895 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:09.902 7f5240bbb700 1 -- 172.21.2.102:0/1351250739 shutdown_connections 2021-02-10T11:17:09.895 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:09.902 7f5240bbb700 1 -- 172.21.2.102:0/1351250739 wait complete. 2021-02-10T11:17:09.903 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000344441 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000344441/45591a11-d715-4b07-9ce6-07ffd66b7c4d 2021-02-10T11:17:09.904 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:17:09.904 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:17:09.904 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:17:10.240 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:17:10.240 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:17:10.240 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344441/45591a11-d715-4b07-9ce6-07ffd66b7c4d/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:17:10.623 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:17:10.624 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:17:10.624 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0244681 s, 42.9 MB/s 2021-02-10T11:17:10.625 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:17:10.625 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:17:10.625 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:17:10.961 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:17:10.962 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:17:10.962 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344441/45591a11-d715-4b07-9ce6-07ffd66b7c4d/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:17:11.339 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:17:11.339 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:17:11.340 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0249687 s, 42.0 MB/s 2021-02-10T11:17:11.340 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:17:11.341 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:17:11.341 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:17:11.676 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:17:11.676 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:17:11.677 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344441/45591a11-d715-4b07-9ce6-07ffd66b7c4d/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:17:12.053 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:17:12.053 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:17:12.053 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0240413 s, 43.6 MB/s 2021-02-10T11:17:12.054 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:17:12.054 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:17:12.054 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:17:12.391 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:17:12.391 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:17:12.391 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344441/45591a11-d715-4b07-9ce6-07ffd66b7c4d/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:17:12.767 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:17:12.767 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:17:12.768 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0243839 s, 43.0 MB/s 2021-02-10T11:17:12.768 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:17:12.769 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:17:12.769 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:17:13.105 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:17:13.105 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:17:13.106 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344441/45591a11-d715-4b07-9ce6-07ffd66b7c4d/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:17:13.479 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:17:13.479 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:17:13.480 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0246713 s, 42.5 MB/s 2021-02-10T11:17:13.480 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:17:13.481 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:17:13.481 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:17:13.816 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:17:13.817 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:17:13.817 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344441/45591a11-d715-4b07-9ce6-07ffd66b7c4d/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:17:14.190 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:17:14.191 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:17:14.191 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0221738 s, 47.3 MB/s 2021-02-10T11:17:14.192 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:17:14.192 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:17:14.192 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:17:14.528 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:17:14.528 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:17:14.529 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344441/45591a11-d715-4b07-9ce6-07ffd66b7c4d/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:17:14.905 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:17:14.905 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:17:14.905 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0242641 s, 43.2 MB/s 2021-02-10T11:17:14.906 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:17:14.906 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:17:14.906 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:17:15.242 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:17:15.242 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:17:15.242 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344441/45591a11-d715-4b07-9ce6-07ffd66b7c4d/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:17:15.618 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:17:15.618 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:17:15.618 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0236271 s, 44.4 MB/s 2021-02-10T11:17:15.619 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:17:15.619 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:17:15.619 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:17:15.956 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:17:15.956 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:17:15.956 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344441/45591a11-d715-4b07-9ce6-07ffd66b7c4d/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:17:16.333 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:17:16.333 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:17:16.333 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0245713 s, 42.7 MB/s 2021-02-10T11:17:16.334 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:17:16.334 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:17:16.334 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:17:16.670 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:17:16.670 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:17:16.671 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344441/45591a11-d715-4b07-9ce6-07ffd66b7c4d/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:17:17.048 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:17:17.049 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:17:17.049 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0257244 s, 40.8 MB/s 2021-02-10T11:17:17.050 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000344442 2021-02-10T11:17:17.483 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:17.474 7ff27f10d700 1 Processor -- start 2021-02-10T11:17:17.483 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:17.474 7ff27f10d700 1 -- start start 2021-02-10T11:17:17.483 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:17.475 7ff27f10d700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7ff2781b0470 0x7ff2781b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:17.483 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:17.475 7ff27f10d700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff2780ef610 0x7ff2780efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:17.483 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:17.475 7ff27f10d700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7ff2780f0030 0x7ff2780f0430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:17.483 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:17.475 7ff27f10d700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2780e5ea0 con 0x7ff2780f0030 2021-02-10T11:17:17.483 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:17.475 7ff27f10d700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2780f0b50 con 0x7ff2781b0470 2021-02-10T11:17:17.484 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:17.475 7ff27f10d700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2780e52e0 con 0x7ff2780ef610 2021-02-10T11:17:17.484 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:17.475 7ff27d72a700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7ff2780f0030 0x7ff2780f0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:17:17.484 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:17.475 7ff27d72a700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7ff2780f0030 0x7ff2780f0430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:53114/0 (socket says 172.21.2.102:53114) 2021-02-10T11:17:17.484 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:17.475 7ff27d72a700 1 -- 172.21.2.102:0/2215410673 learned_addr learned my addr 172.21.2.102:0/2215410673 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:17:17.484 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:17.475 7ff27d72a700 1 -- 172.21.2.102:0/2215410673 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff2780ef610 msgr2=0x7ff2780efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:17:17.484 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:17.475 7ff27d72a700 1 --2- 172.21.2.102:0/2215410673 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff2780ef610 0x7ff2780efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:17:17.484 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:17.475 7ff27d72a700 1 -- 172.21.2.102:0/2215410673 >> v2:172.21.2.113:3300/0 conn(0x7ff2781b0470 msgr2=0x7ff2781b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:17:17.484 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:17.475 7ff27d72a700 1 --2- 172.21.2.102:0/2215410673 >> v2:172.21.2.113:3300/0 conn(0x7ff2781b0470 0x7ff2781b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:17:17.484 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:17.475 7ff27d72a700 1 -- 172.21.2.102:0/2215410673 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2780f10d0 con 0x7ff2780f0030 2021-02-10T11:17:17.484 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:17.475 7ff27d72a700 1 --2- 172.21.2.102:0/2215410673 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff2780f0030 0x7ff2780f0430 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 rx=0x7ff270006f00 tx=0x7ff270009f10).ready entity=mon.0 client_cookie=809df13d801c83c0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:17:17.484 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:17.476 7ff2777fe700 1 -- 172.21.2.102:0/2215410673 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff27000b3b0 con 0x7ff2780f0030 2021-02-10T11:17:17.484 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:17.476 7ff2777fe700 1 -- 172.21.2.102:0/2215410673 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff27000b730 con 0x7ff2780f0030 2021-02-10T11:17:17.484 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:17.476 7ff2777fe700 1 -- 172.21.2.102:0/2215410673 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff27000bc10 con 0x7ff2780f0030 2021-02-10T11:17:17.484 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:17.476 7ff27f10d700 1 -- 172.21.2.102:0/2215410673 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff2780f0030 msgr2=0x7ff2780f0430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:17:17.485 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:17.476 7ff27f10d700 1 --2- 172.21.2.102:0/2215410673 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff2780f0030 0x7ff2780f0430 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 rx=0x7ff270006f00 tx=0x7ff270009f10).stop 2021-02-10T11:17:17.485 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:17.476 7ff27f10d700 1 -- 172.21.2.102:0/2215410673 shutdown_connections 2021-02-10T11:17:17.485 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:17.476 7ff27f10d700 1 --2- 172.21.2.102:0/2215410673 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff2780ef610 0x7ff2780efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:17:17.485 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:17.476 7ff27f10d700 1 --2- 172.21.2.102:0/2215410673 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff2780f0030 0x7ff2780f0430 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:17.485 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:17.476 7ff27f10d700 1 --2- 172.21.2.102:0/2215410673 >> v2:172.21.2.113:3300/0 conn(0x7ff2781b0470 0x7ff2781b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:17:17.485 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:17.476 7ff27f10d700 1 -- 172.21.2.102:0/2215410673 >> 172.21.2.102:0/2215410673 conn(0x7ff2780ce3e0 msgr2=0x7ff2780e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:17:17.485 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:17.476 7ff27f10d700 1 -- 172.21.2.102:0/2215410673 shutdown_connections 2021-02-10T11:17:17.485 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:17.476 7ff27f10d700 1 -- 172.21.2.102:0/2215410673 wait complete. 2021-02-10T11:17:17.487 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:17.491 7ff27f10d700 1 Processor -- start 2021-02-10T11:17:17.487 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:17.491 7ff27f10d700 1 -- start start 2021-02-10T11:17:17.487 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:17.491 7ff27f10d700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff2781a42f0 0x7ff2781a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:17.487 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:17.491 7ff27f10d700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff278196170 0x7ff278196570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:17.487 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:17.491 7ff27f10d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff27808f5b0 0x7ff27808f9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:17.487 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:17.491 7ff27f10d700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff278195bf0 con 0x7ff27808f5b0 2021-02-10T11:17:17.487 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:17.491 7ff27f10d700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2780c4130 con 0x7ff2781a42f0 2021-02-10T11:17:17.487 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:17.491 7ff27f10d700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2780a6e70 con 0x7ff278196170 2021-02-10T11:17:17.487 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:17.492 7ff27d72a700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff27808f5b0 0x7ff27808f9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:17:17.487 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:17.492 7ff27d72a700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff27808f5b0 0x7ff27808f9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:53122/0 (socket says 172.21.2.102:53122) 2021-02-10T11:17:17.488 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:17.492 7ff27d72a700 1 -- 172.21.2.102:0/2385468701 learned_addr learned my addr 172.21.2.102:0/2385468701 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:17:17.488 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:17.492 7ff27cf29700 1 --2- 172.21.2.102:0/2385468701 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff2781a42f0 0x7ff2781a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:17:17.488 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:17.492 7ff277fff700 1 --2- 172.21.2.102:0/2385468701 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff278196170 0x7ff278196570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:17:17.488 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:17.492 7ff27d72a700 1 -- 172.21.2.102:0/2385468701 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff278196170 msgr2=0x7ff278196570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:17:17.488 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:17.492 7ff27d72a700 1 --2- 172.21.2.102:0/2385468701 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff278196170 0x7ff278196570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:17.488 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:17.492 7ff27d72a700 1 -- 172.21.2.102:0/2385468701 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff2781a42f0 msgr2=0x7ff2781a46f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:17:17.488 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:17.492 7ff27d72a700 1 --2- 172.21.2.102:0/2385468701 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff2781a42f0 0x7ff2781a46f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:17.488 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:17.492 7ff27d72a700 1 -- 172.21.2.102:0/2385468701 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2780a6a80 con 0x7ff27808f5b0 2021-02-10T11:17:17.488 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:17.492 7ff27d72a700 1 --2- 172.21.2.102:0/2385468701 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff27808f5b0 0x7ff27808f9b0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 rx=0x7ff2700099f0 tx=0x7ff27000bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:17:17.488 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:17.493 7ff2757fa700 1 -- 172.21.2.102:0/2385468701 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff27000be10 con 0x7ff27808f5b0 2021-02-10T11:17:17.488 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:17.493 7ff2757fa700 1 -- 172.21.2.102:0/2385468701 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff270003030 con 0x7ff27808f5b0 2021-02-10T11:17:17.488 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:17.493 7ff2757fa700 1 -- 172.21.2.102:0/2385468701 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff27000c570 con 0x7ff27808f5b0 2021-02-10T11:17:17.489 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:17.493 7ff27f10d700 1 -- 172.21.2.102:0/2385468701 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2780a6a80 con 0x7ff27808f5b0 2021-02-10T11:17:17.489 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:17.493 7ff27f10d700 1 -- 172.21.2.102:0/2385468701 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2781924c0 con 0x7ff27808f5b0 2021-02-10T11:17:17.489 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:17.493 7ff2757fa700 1 -- 172.21.2.102:0/2385468701 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ff270001270 con 0x7ff27808f5b0 2021-02-10T11:17:17.489 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:17.493 7ff2757fa700 1 --2- 172.21.2.102:0/2385468701 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff25c0186c0 0x7ff25c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:17.489 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:17.493 7ff2757fa700 1 -- 172.21.2.102:0/2385468701 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7ff270032910 con 0x7ff27808f5b0 2021-02-10T11:17:17.489 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:17.494 7ff27cf29700 1 --2- 172.21.2.102:0/2385468701 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff25c0186c0 0x7ff25c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:17:17.489 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:17.494 7ff27cf29700 1 --2- 172.21.2.102:0/2385468701 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff25c0186c0 0x7ff25c01abc0 crc :-1 s=READY pgs=328 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:17:17.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:17.525 7ff27f10d700 1 -- 172.21.2.102:0/2385468701 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2780f0b50 con 0x7ff27808f5b0 2021-02-10T11:17:17.519 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:17.527 7ff2757fa700 1 -- 172.21.2.102:0/2385468701 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ff270000f90 con 0x7ff27808f5b0 2021-02-10T11:17:17.646 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:17.653 7ff27f10d700 1 -- 172.21.2.102:0/2385468701 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344442", "target": ["mgr", ""]}) v1 -- 0x7ff278195bf0 con 0x7ff25c0186c0 2021-02-10T11:17:17.672 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:17.679 7ff2757fa700 1 -- 172.21.2.102:0/2385468701 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7ff260006550 con 0x7ff25c0186c0 2021-02-10T11:17:17.678 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:17.686 7ff27f10d700 1 -- 172.21.2.102:0/2385468701 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff25c0186c0 msgr2=0x7ff25c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:17:17.678 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:17.686 7ff27f10d700 1 --2- 172.21.2.102:0/2385468701 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff25c0186c0 0x7ff25c01abc0 crc :-1 s=READY pgs=328 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:17.679 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:17.686 7ff27f10d700 1 -- 172.21.2.102:0/2385468701 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff27808f5b0 msgr2=0x7ff27808f9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:17:17.679 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:17.686 7ff27f10d700 1 --2- 172.21.2.102:0/2385468701 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff27808f5b0 0x7ff27808f9b0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 rx=0x7ff2700099f0 tx=0x7ff27000bcd0).stop 2021-02-10T11:17:17.679 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:17.687 7ff27f10d700 1 -- 172.21.2.102:0/2385468701 shutdown_connections 2021-02-10T11:17:17.679 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:17.687 7ff27f10d700 1 --2- 172.21.2.102:0/2385468701 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff27808f5b0 0x7ff27808f9b0 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:17.679 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:17.687 7ff27f10d700 1 --2- 172.21.2.102:0/2385468701 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff278196170 0x7ff278196570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:17.680 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:17.687 7ff27f10d700 1 --2- 172.21.2.102:0/2385468701 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff25c0186c0 0x7ff25c01abc0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:17.680 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:17.687 7ff27f10d700 1 --2- 172.21.2.102:0/2385468701 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff2781a42f0 0x7ff2781a46f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:17.680 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:17.687 7ff27f10d700 1 -- 172.21.2.102:0/2385468701 >> 172.21.2.102:0/2385468701 conn(0x7ff2780bf4e0 msgr2=0x7ff2780be160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:17:17.680 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:17.687 7ff27f10d700 1 -- 172.21.2.102:0/2385468701 shutdown_connections 2021-02-10T11:17:17.680 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:17.687 7ff27f10d700 1 -- 172.21.2.102:0/2385468701 wait complete. 2021-02-10T11:17:17.687 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000344442 2021-02-10T11:17:18.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:18.112 7f8fd2f9e700 1 Processor -- start 2021-02-10T11:17:18.121 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:18.112 7f8fd2f9e700 1 -- start start 2021-02-10T11:17:18.121 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:18.112 7f8fd2f9e700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f8fcc1b0470 0x7f8fcc1b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:18.121 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:18.112 7f8fd2f9e700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f8fcc0f20e0 0x7f8fcc0f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:18.121 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:18.112 7f8fd2f9e700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fcc0f0b50 con 0x7f8fcc0ef610 2021-02-10T11:17:18.121 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:18.112 7f8fd2f9e700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8fcc0f0820 con 0x7f8fcc0f20e0 2021-02-10T11:17:18.121 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:18.112 7f8fd2f9e700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8fcc0f24e0 con 0x7f8fcc1b0470 2021-02-10T11:17:18.121 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:18.112 7f8fd15bb700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f8fcc0f20e0 0x7f8fcc0f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:17:18.121 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:18.112 7f8fd15bb700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f8fcc0f20e0 0x7f8fcc0f0200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:53130/0 (socket says 172.21.2.102:53130) 2021-02-10T11:17:18.121 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:18.112 7f8fd15bb700 1 -- 172.21.2.102:0/1839626693 learned_addr learned my addr 172.21.2.102:0/1839626693 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:17:18.121 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:18.113 7f8fd0dba700 1 --2- 172.21.2.102:0/1839626693 >> v2:172.21.2.113:3300/0 conn(0x7f8fcc1b0470 0x7f8fcc1b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:17:18.121 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:18.113 7f8fd15bb700 1 -- 172.21.2.102:0/1839626693 >> v2:172.21.2.113:3300/0 conn(0x7f8fcc1b0470 msgr2=0x7f8fcc1b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:17:18.121 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:18.113 7f8fd15bb700 1 --2- 172.21.2.102:0/1839626693 >> v2:172.21.2.113:3300/0 conn(0x7f8fcc1b0470 0x7f8fcc1b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:18.122 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:18.113 7f8fd15bb700 1 -- 172.21.2.102:0/1839626693 >> v1:172.21.2.102:6789/0 conn(0x7f8fcc0ef610 legacy=0x7f8fcc0efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:17:18.122 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:18.113 7f8fd15bb700 1 -- 172.21.2.102:0/1839626693 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fcc0e52e0 con 0x7f8fcc0f20e0 2021-02-10T11:17:18.122 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:18.113 7f8fd15bb700 1 --2- 172.21.2.102:0/1839626693 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8fcc0f20e0 0x7f8fcc0f0200 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 rx=0x7f8fc4006f00 tx=0x7f8fc4009f10).ready entity=mon.0 client_cookie=1448a2ff0867f12b server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:17:18.122 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:18.113 7f8fcb7fe700 1 -- 172.21.2.102:0/1839626693 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8fc400b3b0 con 0x7f8fcc0f20e0 2021-02-10T11:17:18.122 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:18.113 7f8fcb7fe700 1 -- 172.21.2.102:0/1839626693 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fc400b630 con 0x7f8fcc0f20e0 2021-02-10T11:17:18.122 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:18.113 7f8fcb7fe700 1 -- 172.21.2.102:0/1839626693 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8fc400bb50 con 0x7f8fcc0f20e0 2021-02-10T11:17:18.122 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:18.113 7f8fd2f9e700 1 -- 172.21.2.102:0/1839626693 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8fcc0f20e0 msgr2=0x7f8fcc0f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:17:18.122 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:18.113 7f8fd2f9e700 1 --2- 172.21.2.102:0/1839626693 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8fcc0f20e0 0x7f8fcc0f0200 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 rx=0x7f8fc4006f00 tx=0x7f8fc4009f10).stop 2021-02-10T11:17:18.122 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:18.114 7f8fd2f9e700 1 -- 172.21.2.102:0/1839626693 shutdown_connections 2021-02-10T11:17:18.122 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:18.114 7f8fd2f9e700 1 --2- 172.21.2.102:0/1839626693 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8fcc0f20e0 0x7f8fcc0f0200 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:18.122 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:18.114 7f8fd2f9e700 1 --2- 172.21.2.102:0/1839626693 >> v2:172.21.2.113:3300/0 conn(0x7f8fcc1b0470 0x7f8fcc1b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:18.122 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:18.114 7f8fd2f9e700 1 -- 172.21.2.102:0/1839626693 >> 172.21.2.102:0/1839626693 conn(0x7f8fcc0ce3e0 msgr2=0x7f8fcc0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:17:18.123 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:18.114 7f8fd2f9e700 1 -- 172.21.2.102:0/1839626693 shutdown_connections 2021-02-10T11:17:18.123 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:18.114 7f8fd2f9e700 1 -- 172.21.2.102:0/1839626693 wait complete. 2021-02-10T11:17:18.124 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:18.129 7f8fd2f9e700 1 Processor -- start 2021-02-10T11:17:18.124 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:18.129 7f8fd2f9e700 1 -- start start 2021-02-10T11:17:18.124 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:18.129 7f8fd2f9e700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8fcc0a0420 0x7f8fcc0a0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:18.124 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:18.129 7f8fd2f9e700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8fcc189d00 0x7f8fcc18a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:18.124 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:18.129 7f8fd2f9e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8fcc18a720 0x7f8fcc0b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:18.124 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:18.129 7f8fd2f9e700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fcc189800 con 0x7f8fcc18a720 2021-02-10T11:17:18.125 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:18.129 7f8fd2f9e700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fcc18ab20 con 0x7f8fcc189d00 2021-02-10T11:17:18.125 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:18.129 7f8fd2f9e700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fcc0a1140 con 0x7f8fcc0a0420 2021-02-10T11:17:18.125 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:18.129 7f8fd15bb700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8fcc18a720 0x7f8fcc0b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:17:18.125 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:18.129 7f8fd15bb700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8fcc18a720 0x7f8fcc0b7be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:53136/0 (socket says 172.21.2.102:53136) 2021-02-10T11:17:18.125 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:18.129 7f8fd15bb700 1 -- 172.21.2.102:0/1607968433 learned_addr learned my addr 172.21.2.102:0/1607968433 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:17:18.125 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:18.129 7f8fcbfff700 1 --2- 172.21.2.102:0/1607968433 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8fcc189d00 0x7f8fcc18a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:17:18.125 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:18.129 7f8fd0dba700 1 --2- 172.21.2.102:0/1607968433 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8fcc0a0420 0x7f8fcc0a0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:17:18.125 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:18.130 7f8fd15bb700 1 -- 172.21.2.102:0/1607968433 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8fcc0a0420 msgr2=0x7f8fcc0a0820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:17:18.125 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:18.130 7f8fd15bb700 1 --2- 172.21.2.102:0/1607968433 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8fcc0a0420 0x7f8fcc0a0820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:18.125 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:18.130 7f8fd15bb700 1 -- 172.21.2.102:0/1607968433 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8fcc189d00 msgr2=0x7f8fcc18a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:17:18.125 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:18.130 7f8fd15bb700 1 --2- 172.21.2.102:0/1607968433 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8fcc189d00 0x7f8fcc18a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:18.125 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:18.130 7f8fd15bb700 1 -- 172.21.2.102:0/1607968433 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fcc0b8200 con 0x7f8fcc18a720 2021-02-10T11:17:18.125 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:18.130 7f8fd15bb700 1 --2- 172.21.2.102:0/1607968433 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8fcc18a720 0x7f8fcc0b7be0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 rx=0x7f8fc400b9b0 tx=0x7f8fc400bd50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:17:18.125 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:18.130 7f8fc97fa700 1 -- 172.21.2.102:0/1607968433 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8fc4005820 con 0x7f8fcc18a720 2021-02-10T11:17:18.126 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:18.130 7f8fc97fa700 1 -- 172.21.2.102:0/1607968433 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fc4005aa0 con 0x7f8fcc18a720 2021-02-10T11:17:18.126 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:18.130 7f8fd2f9e700 1 -- 172.21.2.102:0/1607968433 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fcc0b8200 con 0x7f8fcc18a720 2021-02-10T11:17:18.126 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:18.130 7f8fc97fa700 1 -- 172.21.2.102:0/1607968433 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8fc4006030 con 0x7f8fcc18a720 2021-02-10T11:17:18.126 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:18.130 7f8fd2f9e700 1 -- 172.21.2.102:0/1607968433 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fcc0b8790 con 0x7f8fcc18a720 2021-02-10T11:17:18.126 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:18.131 7f8fc97fa700 1 -- 172.21.2.102:0/1607968433 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f8fc4006030 con 0x7f8fcc18a720 2021-02-10T11:17:18.126 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:18.131 7f8fc97fa700 1 --2- 172.21.2.102:0/1607968433 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8fb00186c0 0x7f8fb001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:18.126 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:18.131 7f8fc97fa700 1 -- 172.21.2.102:0/1607968433 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f8fc4032b70 con 0x7f8fcc18a720 2021-02-10T11:17:18.126 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:18.131 7f8fd0dba700 1 --2- 172.21.2.102:0/1607968433 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8fb00186c0 0x7f8fb001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:17:18.126 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:18.131 7f8fd0dba700 1 --2- 172.21.2.102:0/1607968433 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8fb00186c0 0x7f8fb001abc0 crc :-1 s=READY pgs=329 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:17:18.154 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:18.162 7f8fd2f9e700 1 -- 172.21.2.102:0/1607968433 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fcc0b8fe0 con 0x7f8fcc18a720 2021-02-10T11:17:18.156 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:18.164 7f8fc97fa700 1 -- 172.21.2.102:0/1607968433 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f8fc4005aa0 con 0x7f8fcc18a720 2021-02-10T11:17:18.284 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:18.292 7f8fd2f9e700 1 -- 172.21.2.102:0/1607968433 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344442", "target": ["mgr", ""]}) v1 -- 0x7f8fcc0b8790 con 0x7f8fb00186c0 2021-02-10T11:17:18.287 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:18.295 7f8fc97fa700 1 -- 172.21.2.102:0/1607968433 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f8fb40065b0 con 0x7f8fb00186c0 2021-02-10T11:17:18.291 INFO:teuthology.orchestra.run.gibba002.stdout:/volumes/_nogroup/subvolume_0000000000344442/a94577e3-44e3-4929-a3f5-bdf19adfb5af 2021-02-10T11:17:18.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:18.300 7f8fd2f9e700 1 -- 172.21.2.102:0/1607968433 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8fb00186c0 msgr2=0x7f8fb001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:17:18.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:18.300 7f8fd2f9e700 1 --2- 172.21.2.102:0/1607968433 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8fb00186c0 0x7f8fb001abc0 crc :-1 s=READY pgs=329 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:18.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:18.300 7f8fd2f9e700 1 -- 172.21.2.102:0/1607968433 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8fcc18a720 msgr2=0x7f8fcc0b7be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:17:18.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:18.300 7f8fd2f9e700 1 --2- 172.21.2.102:0/1607968433 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8fcc18a720 0x7f8fcc0b7be0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 rx=0x7f8fc400b9b0 tx=0x7f8fc400bd50).stop 2021-02-10T11:17:18.294 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:18.300 7f8fd2f9e700 1 -- 172.21.2.102:0/1607968433 shutdown_connections 2021-02-10T11:17:18.294 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:18.300 7f8fd2f9e700 1 --2- 172.21.2.102:0/1607968433 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8fcc18a720 0x7f8fcc0b7be0 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:18.294 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:18.300 7f8fd2f9e700 1 --2- 172.21.2.102:0/1607968433 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8fb00186c0 0x7f8fb001abc0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:18.294 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:18.300 7f8fd2f9e700 1 --2- 172.21.2.102:0/1607968433 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8fcc189d00 0x7f8fcc18a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:18.294 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:18.300 7f8fd2f9e700 1 --2- 172.21.2.102:0/1607968433 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8fcc0a0420 0x7f8fcc0a0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:18.295 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:18.300 7f8fd2f9e700 1 -- 172.21.2.102:0/1607968433 >> 172.21.2.102:0/1607968433 conn(0x7f8fcc094e10 msgr2=0x7f8fcc0a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:17:18.295 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:18.300 7f8fd2f9e700 1 -- 172.21.2.102:0/1607968433 shutdown_connections 2021-02-10T11:17:18.295 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:18.301 7f8fd2f9e700 1 -- 172.21.2.102:0/1607968433 wait complete. 2021-02-10T11:17:18.301 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000344442 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000344442/a94577e3-44e3-4929-a3f5-bdf19adfb5af 2021-02-10T11:17:18.302 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:17:18.302 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:17:18.302 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:17:18.639 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:17:18.639 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:17:18.640 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344442/a94577e3-44e3-4929-a3f5-bdf19adfb5af/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:17:19.019 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:17:19.019 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:17:19.019 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0238541 s, 44.0 MB/s 2021-02-10T11:17:19.020 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:17:19.020 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:17:19.020 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:17:19.356 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:17:19.356 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:17:19.357 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344442/a94577e3-44e3-4929-a3f5-bdf19adfb5af/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:17:19.735 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:17:19.735 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:17:19.735 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0252709 s, 41.5 MB/s 2021-02-10T11:17:19.736 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:17:19.736 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:17:19.736 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:17:20.072 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:17:20.072 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:17:20.073 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344442/a94577e3-44e3-4929-a3f5-bdf19adfb5af/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:17:20.449 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:17:20.449 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:17:20.449 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0246306 s, 42.6 MB/s 2021-02-10T11:17:20.450 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:17:20.451 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:17:20.451 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:17:20.786 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:17:20.786 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:17:20.786 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344442/a94577e3-44e3-4929-a3f5-bdf19adfb5af/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:17:21.162 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:17:21.163 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:17:21.163 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0238235 s, 44.0 MB/s 2021-02-10T11:17:21.164 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:17:21.164 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:17:21.164 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:17:21.499 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:17:21.500 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:17:21.500 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344442/a94577e3-44e3-4929-a3f5-bdf19adfb5af/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:17:21.877 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:17:21.878 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:17:21.878 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0246902 s, 42.5 MB/s 2021-02-10T11:17:21.879 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:17:21.879 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:17:21.879 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:17:22.215 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:17:22.215 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:17:22.216 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344442/a94577e3-44e3-4929-a3f5-bdf19adfb5af/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:17:22.594 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:17:22.595 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:17:22.595 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0256088 s, 40.9 MB/s 2021-02-10T11:17:22.596 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:17:22.596 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:17:22.596 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:17:22.932 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:17:22.932 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:17:22.932 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344442/a94577e3-44e3-4929-a3f5-bdf19adfb5af/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:17:23.307 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:17:23.308 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:17:23.308 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.023415 s, 44.8 MB/s 2021-02-10T11:17:23.309 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:17:23.309 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:17:23.309 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:17:23.646 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:17:23.647 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:17:23.647 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344442/a94577e3-44e3-4929-a3f5-bdf19adfb5af/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:17:24.020 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:17:24.020 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:17:24.021 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0245601 s, 42.7 MB/s 2021-02-10T11:17:24.022 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:17:24.022 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:17:24.022 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:17:24.356 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:17:24.357 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:17:24.357 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344442/a94577e3-44e3-4929-a3f5-bdf19adfb5af/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:17:24.734 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:17:24.734 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:17:24.734 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0244383 s, 42.9 MB/s 2021-02-10T11:17:24.735 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:17:24.735 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:17:24.735 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:17:25.071 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:17:25.071 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:17:25.072 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344442/a94577e3-44e3-4929-a3f5-bdf19adfb5af/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:17:25.446 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:17:26.637 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:17:26.637 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0239809 s, 43.7 MB/s 2021-02-10T11:17:26.638 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000344443 2021-02-10T11:17:27.101 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:27.089 7fc9d19f9700 1 Processor -- start 2021-02-10T11:17:27.102 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:27.089 7fc9d19f9700 1 -- start start 2021-02-10T11:17:27.102 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:27.090 7fc9d19f9700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc9cc1b0470 0x7fc9cc1b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:27.102 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:27.090 7fc9d19f9700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fc9cc0ef610 0x7fc9cc0efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:27.102 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:27.090 7fc9d19f9700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9cc0f0b50 con 0x7fc9cc0f0030 2021-02-10T11:17:27.103 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:27.090 7fc9d19f9700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9cc0e52e0 con 0x7fc9cc0ef610 2021-02-10T11:17:27.103 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:27.090 7fc9d19f9700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9cc0e5490 con 0x7fc9cc1b0470 2021-02-10T11:17:27.103 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:27.091 7fc9caffd700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fc9cc0ef610 0x7fc9cc0efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:17:27.103 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:27.091 7fc9cb7fe700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc9cc1b0470 0x7fc9cc1b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:17:27.104 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:27.091 7fc9caffd700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fc9cc0ef610 0x7fc9cc0efa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.113:3300/0 says I am v2:172.21.2.102:56570/0 (socket says 172.21.2.102:56570) 2021-02-10T11:17:27.104 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:27.091 7fc9cb7fe700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc9cc1b0470 0x7fc9cc1b0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.113:3301/0 says I am v2:172.21.2.102:58676/0 (socket says 172.21.2.102:58676) 2021-02-10T11:17:27.104 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:27.091 7fc9cb7fe700 1 -- 172.21.2.102:0/131478505 learned_addr learned my addr 172.21.2.102:0/131478505 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:17:27.104 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:27.091 7fc9caffd700 1 -- 172.21.2.102:0/131478505 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc9cc1b0470 msgr2=0x7fc9cc1b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:17:27.104 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:27.091 7fc9caffd700 1 --2- 172.21.2.102:0/131478505 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc9cc1b0470 0x7fc9cc1b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:27.104 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:27.091 7fc9caffd700 1 -- 172.21.2.102:0/131478505 >> v1:172.21.2.113:6789/0 conn(0x7fc9cc0f0030 legacy=0x7fc9cc0f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:17:27.104 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:27.091 7fc9caffd700 1 -- 172.21.2.102:0/131478505 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9cc0f0430 con 0x7fc9cc0ef610 2021-02-10T11:17:27.105 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:27.091 7fc9cb7fe700 1 --2- 172.21.2.102:0/131478505 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc9cc1b0470 0x7fc9cc1b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:17:27.105 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:27.091 7fc9caffd700 1 --2- 172.21.2.102:0/131478505 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc9cc0ef610 0x7fc9cc0efa10 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 rx=0x7fc9b4006650 tx=0x7fc9b4009a80).ready entity=mon.1 client_cookie=fe040b26fee404f7 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:17:27.105 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:27.092 7fc9ca7fc700 1 -- 172.21.2.102:0/131478505 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc9b40009b0 con 0x7fc9cc0ef610 2021-02-10T11:17:27.105 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:27.092 7fc9ca7fc700 1 -- 172.21.2.102:0/131478505 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9b400c0c0 con 0x7fc9cc0ef610 2021-02-10T11:17:27.105 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:27.092 7fc9ca7fc700 1 -- 172.21.2.102:0/131478505 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc9b400c5e0 con 0x7fc9cc0ef610 2021-02-10T11:17:27.105 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:27.092 7fc9d19f9700 1 -- 172.21.2.102:0/131478505 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc9cc0ef610 msgr2=0x7fc9cc0efa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:17:27.106 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:27.092 7fc9d19f9700 1 --2- 172.21.2.102:0/131478505 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc9cc0ef610 0x7fc9cc0efa10 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 rx=0x7fc9b4006650 tx=0x7fc9b4009a80).stop 2021-02-10T11:17:27.106 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:27.093 7fc9d19f9700 1 -- 172.21.2.102:0/131478505 shutdown_connections 2021-02-10T11:17:27.107 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:27.093 7fc9d19f9700 1 --2- 172.21.2.102:0/131478505 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc9cc0ef610 0x7fc9cc0efa10 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:27.107 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:27.093 7fc9d19f9700 1 --2- 172.21.2.102:0/131478505 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc9cc1b0470 0x7fc9cc1b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:27.108 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:27.093 7fc9d19f9700 1 -- 172.21.2.102:0/131478505 >> 172.21.2.102:0/131478505 conn(0x7fc9cc0ce3e0 msgr2=0x7fc9cc0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:17:27.108 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:27.093 7fc9d19f9700 1 -- 172.21.2.102:0/131478505 shutdown_connections 2021-02-10T11:17:27.108 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:27.093 7fc9d19f9700 1 -- 172.21.2.102:0/131478505 wait complete. 2021-02-10T11:17:27.108 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:27.107 7fc9d19f9700 1 Processor -- start 2021-02-10T11:17:27.108 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:27.107 7fc9d19f9700 1 -- start start 2021-02-10T11:17:27.109 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:27.108 7fc9d19f9700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc9cc0a0420 0x7fc9cc0a0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:27.109 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:27.108 7fc9d19f9700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc9cc189d00 0x7fc9cc18a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:27.109 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:27.108 7fc9d19f9700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc9cc18a720 0x7fc9cc0b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:27.109 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:27.108 7fc9d19f9700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9cc189780 con 0x7fc9cc189d00 2021-02-10T11:17:27.109 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:27.108 7fc9d19f9700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9cc18ab20 con 0x7fc9cc18a720 2021-02-10T11:17:27.109 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:27.108 7fc9d19f9700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9cc0a1140 con 0x7fc9cc0a0420 2021-02-10T11:17:27.110 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:27.108 7fc9cb7fe700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc9cc0a0420 0x7fc9cc0a0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:17:27.110 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:27.108 7fc9cb7fe700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc9cc0a0420 0x7fc9cc0a0820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.113:3301/0 says I am v2:172.21.2.102:58680/0 (socket says 172.21.2.102:58680) 2021-02-10T11:17:27.110 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:27.108 7fc9cb7fe700 1 -- 172.21.2.102:0/2908164619 learned_addr learned my addr 172.21.2.102:0/2908164619 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:17:27.110 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:27.108 7fc9cbfff700 1 --2- 172.21.2.102:0/2908164619 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc9cc18a720 0x7fc9cc0b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:17:27.110 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:27.108 7fc9caffd700 1 --2- 172.21.2.102:0/2908164619 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc9cc189d00 0x7fc9cc18a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:17:27.113 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:27.109 7fc9cb7fe700 1 -- 172.21.2.102:0/2908164619 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc9cc18a720 msgr2=0x7fc9cc0b7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:17:27.113 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:27.109 7fc9cb7fe700 1 --2- 172.21.2.102:0/2908164619 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc9cc18a720 0x7fc9cc0b7be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:27.113 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:27.109 7fc9cb7fe700 1 -- 172.21.2.102:0/2908164619 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc9cc189d00 msgr2=0x7fc9cc18a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:17:27.113 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:27.109 7fc9cb7fe700 1 --2- 172.21.2.102:0/2908164619 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc9cc189d00 0x7fc9cc18a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:27.113 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:27.109 7fc9cb7fe700 1 -- 172.21.2.102:0/2908164619 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9cc0b8200 con 0x7fc9cc0a0420 2021-02-10T11:17:27.113 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:27.109 7fc9caffd700 1 --2- 172.21.2.102:0/2908164619 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc9cc189d00 0x7fc9cc18a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:17:27.113 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:27.109 7fc9cb7fe700 1 --2- 172.21.2.102:0/2908164619 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc9cc0a0420 0x7fc9cc0a0820 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7fc9bc00b5e0 tx=0x7fc9bc00b980).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:17:27.113 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:27.109 7fc9c3fff700 1 -- 172.21.2.102:0/2908164619 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc9bc00bd70 con 0x7fc9cc0a0420 2021-02-10T11:17:27.113 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:27.109 7fc9c3fff700 1 -- 172.21.2.102:0/2908164619 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9bc006030 con 0x7fc9cc0a0420 2021-02-10T11:17:27.113 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:27.109 7fc9d19f9700 1 -- 172.21.2.102:0/2908164619 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9cc0b8200 con 0x7fc9cc0a0420 2021-02-10T11:17:27.113 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:27.109 7fc9c3fff700 1 -- 172.21.2.102:0/2908164619 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc9bc005560 con 0x7fc9cc0a0420 2021-02-10T11:17:27.114 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:27.109 7fc9d19f9700 1 -- 172.21.2.102:0/2908164619 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9cc0b8200 con 0x7fc9cc0a0420 2021-02-10T11:17:27.114 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:27.110 7fc9c3fff700 1 -- 172.21.2.102:0/2908164619 <== mon.2 v2:172.21.2.113:3301/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fc9bc0052e0 con 0x7fc9cc0a0420 2021-02-10T11:17:27.114 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:27.110 7fc9c3fff700 1 --2- 172.21.2.102:0/2908164619 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc9b00186c0 0x7fc9b001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:27.114 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:27.110 7fc9c3fff700 1 -- 172.21.2.102:0/2908164619 <== mon.2 v2:172.21.2.113:3301/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fc9bc0323f0 con 0x7fc9cc0a0420 2021-02-10T11:17:27.114 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:27.110 7fc9caffd700 1 --2- 172.21.2.102:0/2908164619 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc9b00186c0 0x7fc9b001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:17:27.114 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:27.110 7fc9caffd700 1 --2- 172.21.2.102:0/2908164619 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc9b00186c0 0x7fc9b001abc0 crc :-1 s=READY pgs=330 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:17:27.132 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:27.139 7fc9d19f9700 1 -- 172.21.2.102:0/2908164619 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9cc105ae0 con 0x7fc9cc0a0420 2021-02-10T11:17:27.134 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:27.142 7fc9c3fff700 1 -- 172.21.2.102:0/2908164619 <== mon.2 v2:172.21.2.113:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fc9bc006210 con 0x7fc9cc0a0420 2021-02-10T11:17:27.265 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:27.273 7fc9d19f9700 1 -- 172.21.2.102:0/2908164619 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344443", "target": ["mgr", ""]}) v1 -- 0x7fc9cc189780 con 0x7fc9b00186c0 2021-02-10T11:17:27.294 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:27.302 7fc9c3fff700 1 -- 172.21.2.102:0/2908164619 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fc9b4009270 con 0x7fc9b00186c0 2021-02-10T11:17:27.299 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:27.306 7fc9d19f9700 1 -- 172.21.2.102:0/2908164619 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc9b00186c0 msgr2=0x7fc9b001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:17:27.299 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:27.306 7fc9d19f9700 1 --2- 172.21.2.102:0/2908164619 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc9b00186c0 0x7fc9b001abc0 crc :-1 s=READY pgs=330 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:27.299 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:27.306 7fc9d19f9700 1 -- 172.21.2.102:0/2908164619 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc9cc0a0420 msgr2=0x7fc9cc0a0820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:17:27.299 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:27.306 7fc9d19f9700 1 --2- 172.21.2.102:0/2908164619 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc9cc0a0420 0x7fc9cc0a0820 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7fc9bc00b5e0 tx=0x7fc9bc00b980).stop 2021-02-10T11:17:27.299 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:27.306 7fc9d19f9700 1 -- 172.21.2.102:0/2908164619 shutdown_connections 2021-02-10T11:17:27.299 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:27.306 7fc9d19f9700 1 --2- 172.21.2.102:0/2908164619 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc9b00186c0 0x7fc9b001abc0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:27.299 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:27.306 7fc9d19f9700 1 --2- 172.21.2.102:0/2908164619 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc9cc18a720 0x7fc9cc0b7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:27.302 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:27.306 7fc9d19f9700 1 --2- 172.21.2.102:0/2908164619 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc9cc189d00 0x7fc9cc18a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:27.302 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:27.306 7fc9d19f9700 1 --2- 172.21.2.102:0/2908164619 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc9cc0a0420 0x7fc9cc0a0820 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:27.302 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:27.306 7fc9d19f9700 1 -- 172.21.2.102:0/2908164619 >> 172.21.2.102:0/2908164619 conn(0x7fc9cc094e10 msgr2=0x7fc9cc0a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:17:27.302 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:27.307 7fc9d19f9700 1 -- 172.21.2.102:0/2908164619 shutdown_connections 2021-02-10T11:17:27.303 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:27.307 7fc9d19f9700 1 -- 172.21.2.102:0/2908164619 wait complete. 2021-02-10T11:17:27.307 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000344443 2021-02-10T11:17:27.750 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:27.742 7fa25f2f9700 1 Processor -- start 2021-02-10T11:17:27.751 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:27.742 7fa25f2f9700 1 -- start start 2021-02-10T11:17:27.751 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:27.742 7fa25f2f9700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa2581b0870 0x7fa2580f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:27.751 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:27.742 7fa25f2f9700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2581b0c70 con 0x7fa2580efe00 2021-02-10T11:17:27.751 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:27.742 7fa25f2f9700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2580f0b50 con 0x7fa2581b0470 2021-02-10T11:17:27.751 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:27.742 7fa25f2f9700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2580e5ea0 con 0x7fa2581b0870 2021-02-10T11:17:27.751 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:27.742 7fa25d916700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fa2580efe00 0x7fa2580f0200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:45376/0 (socket says 172.21.2.102:45376) 2021-02-10T11:17:27.751 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:27.742 7fa25d916700 1 -- 172.21.2.102:0/2944767595 learned_addr learned my addr 172.21.2.102:0/2944767595 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:17:27.751 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:27.743 7fa25c914700 1 --2- 172.21.2.102:0/2944767595 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa2581b0870 0x7fa2580f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:17:27.751 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:27.743 7fa24bfff700 1 -- 172.21.2.102:0/2944767595 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2919123225 0 0) 0x7fa250001500 con 0x7fa2580efe00 2021-02-10T11:17:27.751 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:27.743 7fa24bfff700 1 -- 172.21.2.102:0/2944767595 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa244002fe0 con 0x7fa2580efe00 2021-02-10T11:17:27.751 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:27.743 7fa24bfff700 1 -- 172.21.2.102:0/2944767595 <== mon.1 v1:172.21.2.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 952368187 0 0) 0x7fa254000ff0 con 0x7fa2581b0470 2021-02-10T11:17:27.752 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:27.743 7fa24bfff700 1 -- 172.21.2.102:0/2944767595 --> v1:172.21.2.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa244003210 con 0x7fa2581b0470 2021-02-10T11:17:27.752 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:27.743 7fa25c914700 1 -- 172.21.2.102:0/2944767595 >> v1:172.21.2.113:6789/0 conn(0x7fa2581b0470 legacy=0x7fa2580ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:17:27.752 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:27.743 7fa25c914700 1 -- 172.21.2.102:0/2944767595 >> v1:172.21.2.102:6789/0 conn(0x7fa2580efe00 legacy=0x7fa2580f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:17:27.752 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:27.743 7fa25c914700 1 -- 172.21.2.102:0/2944767595 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2580e52e0 con 0x7fa2581b0870 2021-02-10T11:17:27.752 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:27.743 7fa25c914700 1 --2- 172.21.2.102:0/2944767595 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa2581b0870 0x7fa2580f20e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7fa24c006650 tx=0x7fa24c009a80).ready entity=mon.2 client_cookie=8bb44e30a623c65d server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:17:27.752 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:27.744 7fa24bfff700 1 -- 172.21.2.102:0/2944767595 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa24c0009b0 con 0x7fa2581b0870 2021-02-10T11:17:27.752 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:27.744 7fa24bfff700 1 -- 172.21.2.102:0/2944767595 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa24c00c0c0 con 0x7fa2581b0870 2021-02-10T11:17:27.754 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:27.744 7fa24bfff700 1 -- 172.21.2.102:0/2944767595 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa24c00c560 con 0x7fa2581b0870 2021-02-10T11:17:27.754 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:27.744 7fa25f2f9700 1 -- 172.21.2.102:0/2944767595 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa2581b0870 msgr2=0x7fa2580f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:17:27.754 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:27.744 7fa25f2f9700 1 --2- 172.21.2.102:0/2944767595 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa2581b0870 0x7fa2580f20e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7fa24c006650 tx=0x7fa24c009a80).stop 2021-02-10T11:17:27.754 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:27.744 7fa25f2f9700 1 -- 172.21.2.102:0/2944767595 shutdown_connections 2021-02-10T11:17:27.754 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:27.744 7fa25f2f9700 1 --2- 172.21.2.102:0/2944767595 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa2581b0870 0x7fa2580f20e0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:27.754 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:27.744 7fa25f2f9700 1 -- 172.21.2.102:0/2944767595 >> 172.21.2.102:0/2944767595 conn(0x7fa2580ce3e0 msgr2=0x7fa2580e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:17:27.754 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:27.744 7fa25f2f9700 1 -- 172.21.2.102:0/2944767595 shutdown_connections 2021-02-10T11:17:27.754 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:27.744 7fa25f2f9700 1 -- 172.21.2.102:0/2944767595 wait complete. 2021-02-10T11:17:27.755 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:27.759 7fa25f2f9700 1 Processor -- start 2021-02-10T11:17:27.755 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:27.759 7fa25f2f9700 1 -- start start 2021-02-10T11:17:27.755 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:27.759 7fa25f2f9700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa25808ece0 0x7fa2580a6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:27.755 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:27.759 7fa25f2f9700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa2580a6860 0x7fa2580a6c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:27.755 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:27.759 7fa25f2f9700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa2580a7280 0x7fa2580a4590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:27.755 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:27.759 7fa25f2f9700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2580a5cc0 con 0x7fa2580a6860 2021-02-10T11:17:27.755 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:27.759 7fa25f2f9700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa25808f0e0 con 0x7fa2580a7280 2021-02-10T11:17:27.755 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:27.759 7fa25f2f9700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa25808f2c0 con 0x7fa25808ece0 2021-02-10T11:17:27.755 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:27.760 7fa25c914700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa2580a6860 0x7fa2580a6c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:17:27.755 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:27.760 7fa25c914700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa2580a6860 0x7fa2580a6c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:53156/0 (socket says 172.21.2.102:53156) 2021-02-10T11:17:27.755 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:27.760 7fa25c914700 1 -- 172.21.2.102:0/1880843898 learned_addr learned my addr 172.21.2.102:0/1880843898 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:17:27.755 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:27.760 7fa25d115700 1 --2- 172.21.2.102:0/1880843898 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa25808ece0 0x7fa2580a6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:17:27.755 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:27.760 7fa25d916700 1 --2- 172.21.2.102:0/1880843898 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa2580a7280 0x7fa2580a4590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:17:27.758 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:27.760 7fa25c914700 1 -- 172.21.2.102:0/1880843898 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa25808ece0 msgr2=0x7fa2580a6240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:17:27.758 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:27.760 7fa25c914700 1 --2- 172.21.2.102:0/1880843898 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa25808ece0 0x7fa2580a6240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:27.758 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:27.760 7fa25c914700 1 -- 172.21.2.102:0/1880843898 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa2580a7280 msgr2=0x7fa2580a4590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:17:27.758 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:27.760 7fa25c914700 1 --2- 172.21.2.102:0/1880843898 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa2580a7280 0x7fa2580a4590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:27.758 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:27.760 7fa25c914700 1 -- 172.21.2.102:0/1880843898 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2580a4bb0 con 0x7fa2580a6860 2021-02-10T11:17:27.758 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:27.760 7fa25c914700 1 --2- 172.21.2.102:0/1880843898 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa2580a6860 0x7fa2580a6c60 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 rx=0x7fa24c0096a0 tx=0x7fa24c00b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:17:27.758 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:27.760 7fa249ffb700 1 -- 172.21.2.102:0/1880843898 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa24c00bd80 con 0x7fa2580a6860 2021-02-10T11:17:27.758 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:27.760 7fa249ffb700 1 -- 172.21.2.102:0/1880843898 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa24c006030 con 0x7fa2580a6860 2021-02-10T11:17:27.758 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:27.760 7fa249ffb700 1 -- 172.21.2.102:0/1880843898 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa24c005560 con 0x7fa2580a6860 2021-02-10T11:17:27.758 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:27.760 7fa25f2f9700 1 -- 172.21.2.102:0/1880843898 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2580a4bb0 con 0x7fa2580a6860 2021-02-10T11:17:27.759 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:27.760 7fa25f2f9700 1 -- 172.21.2.102:0/1880843898 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2580a4bb0 con 0x7fa2580a6860 2021-02-10T11:17:27.759 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:27.761 7fa249ffb700 1 -- 172.21.2.102:0/1880843898 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa24c0052e0 con 0x7fa2580a6860 2021-02-10T11:17:27.759 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:27.761 7fa249ffb700 1 --2- 172.21.2.102:0/1880843898 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa2380186c0 0x7fa23801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:27.759 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:27.761 7fa249ffb700 1 -- 172.21.2.102:0/1880843898 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fa24c0325d0 con 0x7fa2580a6860 2021-02-10T11:17:27.759 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:27.761 7fa25d115700 1 --2- 172.21.2.102:0/1880843898 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa2380186c0 0x7fa23801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:17:27.759 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:27.762 7fa25d115700 1 --2- 172.21.2.102:0/1880843898 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa2380186c0 0x7fa23801abc0 crc :-1 s=READY pgs=331 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:17:27.788 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:27.792 7fa25f2f9700 1 -- 172.21.2.102:0/1880843898 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa258105ae0 con 0x7fa2580a6860 2021-02-10T11:17:27.788 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:27.795 7fa249ffb700 1 -- 172.21.2.102:0/1880843898 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fa24c0009a0 con 0x7fa2580a6860 2021-02-10T11:17:27.915 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:27.922 7fa25f2f9700 1 -- 172.21.2.102:0/1880843898 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344443", "target": ["mgr", ""]}) v1 -- 0x7fa2580f0b50 con 0x7fa2380186c0 2021-02-10T11:17:27.917 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:27.925 7fa249ffb700 1 -- 172.21.2.102:0/1880843898 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7fa254006e60 con 0x7fa2380186c0 2021-02-10T11:17:27.918 INFO:teuthology.orchestra.run.gibba002.stdout:/volumes/_nogroup/subvolume_0000000000344443/c107f868-1729-4810-9855-6a93103bda9a 2021-02-10T11:17:27.919 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:27.927 7fa25f2f9700 1 -- 172.21.2.102:0/1880843898 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa2380186c0 msgr2=0x7fa23801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:17:27.919 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:27.927 7fa25f2f9700 1 --2- 172.21.2.102:0/1880843898 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa2380186c0 0x7fa23801abc0 crc :-1 s=READY pgs=331 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:27.919 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:27.927 7fa25f2f9700 1 -- 172.21.2.102:0/1880843898 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa2580a6860 msgr2=0x7fa2580a6c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:17:27.919 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:27.927 7fa25f2f9700 1 --2- 172.21.2.102:0/1880843898 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa2580a6860 0x7fa2580a6c60 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 rx=0x7fa24c0096a0 tx=0x7fa24c00b990).stop 2021-02-10T11:17:27.922 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:27.927 7fa25f2f9700 1 -- 172.21.2.102:0/1880843898 shutdown_connections 2021-02-10T11:17:27.922 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:27.927 7fa25f2f9700 1 --2- 172.21.2.102:0/1880843898 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa2380186c0 0x7fa23801abc0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:27.922 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:27.927 7fa25f2f9700 1 --2- 172.21.2.102:0/1880843898 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa2580a7280 0x7fa2580a4590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:27.922 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:27.927 7fa25f2f9700 1 --2- 172.21.2.102:0/1880843898 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa2580a6860 0x7fa2580a6c60 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:27.922 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:27.927 7fa25f2f9700 1 --2- 172.21.2.102:0/1880843898 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa25808ece0 0x7fa2580a6240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:27.922 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:27.927 7fa25f2f9700 1 -- 172.21.2.102:0/1880843898 >> 172.21.2.102:0/1880843898 conn(0x7fa25808ffe0 msgr2=0x7fa2580bf050 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:17:27.922 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:27.927 7fa25f2f9700 1 -- 172.21.2.102:0/1880843898 shutdown_connections 2021-02-10T11:17:27.922 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:27.927 7fa25f2f9700 1 -- 172.21.2.102:0/1880843898 wait complete. 2021-02-10T11:17:27.927 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000344443 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000344443/c107f868-1729-4810-9855-6a93103bda9a 2021-02-10T11:17:27.928 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:17:27.928 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:17:27.928 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:17:28.277 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:17:28.277 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:17:28.277 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344443/c107f868-1729-4810-9855-6a93103bda9a/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:17:28.501 DEBUG:teuthology.orchestra.run.gibba002:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:17:28.506 DEBUG:teuthology.orchestra.run.gibba013:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:17:28.659 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:17:28.659 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:17:28.660 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0238568 s, 44.0 MB/s 2021-02-10T11:17:28.661 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:17:28.661 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:17:28.661 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:17:29.001 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:17:29.001 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:17:29.002 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344443/c107f868-1729-4810-9855-6a93103bda9a/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:17:29.381 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:17:29.381 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:17:29.381 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0260411 s, 40.3 MB/s 2021-02-10T11:17:29.382 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:17:29.383 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:17:29.383 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:17:29.718 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:17:29.718 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:17:29.719 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344443/c107f868-1729-4810-9855-6a93103bda9a/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:17:30.096 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:17:30.096 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:17:30.096 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0240745 s, 43.6 MB/s 2021-02-10T11:17:30.097 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:17:30.098 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:17:30.098 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:17:30.433 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:17:30.433 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:17:30.433 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344443/c107f868-1729-4810-9855-6a93103bda9a/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:17:30.809 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:17:30.809 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:17:30.809 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0241761 s, 43.4 MB/s 2021-02-10T11:17:30.810 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:17:30.810 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:17:30.810 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:17:31.146 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:17:31.146 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:17:31.147 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344443/c107f868-1729-4810-9855-6a93103bda9a/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:17:31.522 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:17:31.523 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:17:31.523 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0235192 s, 44.6 MB/s 2021-02-10T11:17:31.524 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:17:31.524 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:17:31.524 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:17:31.858 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:17:31.859 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:17:31.859 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344443/c107f868-1729-4810-9855-6a93103bda9a/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:17:32.235 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:17:32.235 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:17:32.235 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0245365 s, 42.7 MB/s 2021-02-10T11:17:32.236 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:17:32.237 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:17:32.237 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:17:32.572 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:17:32.572 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:17:32.572 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344443/c107f868-1729-4810-9855-6a93103bda9a/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:17:32.949 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:17:32.949 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:17:32.949 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0260175 s, 40.3 MB/s 2021-02-10T11:17:32.950 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:17:32.950 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:17:32.950 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:17:33.286 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:17:33.286 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:17:33.286 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344443/c107f868-1729-4810-9855-6a93103bda9a/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:17:33.660 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:17:33.660 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:17:33.660 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0243343 s, 43.1 MB/s 2021-02-10T11:17:33.661 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:17:33.661 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:17:33.661 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:17:33.997 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:17:33.997 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:17:33.997 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344443/c107f868-1729-4810-9855-6a93103bda9a/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:17:34.373 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:17:34.373 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:17:34.373 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0247522 s, 42.4 MB/s 2021-02-10T11:17:34.374 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:17:34.375 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:17:34.375 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:17:34.712 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:17:34.712 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:17:34.712 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344443/c107f868-1729-4810-9855-6a93103bda9a/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:17:35.086 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:17:35.086 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:17:35.086 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0223492 s, 46.9 MB/s 2021-02-10T11:17:35.087 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000344444 2021-02-10T11:17:35.519 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:35.510 7f08cf7b6700 1 Processor -- start 2021-02-10T11:17:35.519 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:35.510 7f08cf7b6700 1 -- start start 2021-02-10T11:17:35.519 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:35.511 7f08cf7b6700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f08c80f05f0 0x7f08c80f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:35.520 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:35.511 7f08cf7b6700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08c81b0c70 con 0x7f08c81b0870 2021-02-10T11:17:35.520 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:35.511 7f08cf7b6700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08c80f0b50 con 0x7f08c81b0470 2021-02-10T11:17:35.520 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:35.511 7f08cf7b6700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08c80e5ea0 con 0x7f08c80f05f0 2021-02-10T11:17:35.520 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:35.511 7f08ccdd1700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f08c81b0870 0x7f08c80efe00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:45388/0 (socket says 172.21.2.102:45388) 2021-02-10T11:17:35.520 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:35.511 7f08ccdd1700 1 -- 172.21.2.102:0/3312575819 learned_addr learned my addr 172.21.2.102:0/3312575819 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:17:35.520 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:35.511 7f08cddd3700 1 --2- 172.21.2.102:0/3312575819 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f08c80f05f0 0x7f08c80f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:17:35.520 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:35.511 7f08bbfff700 1 -- 172.21.2.102:0/3312575819 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2147955660 0 0) 0x7f08c40010b0 con 0x7f08c81b0870 2021-02-10T11:17:35.520 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:35.511 7f08bbfff700 1 -- 172.21.2.102:0/3312575819 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08b4002fe0 con 0x7f08c81b0870 2021-02-10T11:17:35.520 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:35.511 7f08bbfff700 1 -- 172.21.2.102:0/3312575819 <== mon.1 v1:172.21.2.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 473722623 0 0) 0x7f08bc000ff0 con 0x7f08c81b0470 2021-02-10T11:17:35.520 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:35.511 7f08bbfff700 1 -- 172.21.2.102:0/3312575819 --> v1:172.21.2.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08b4003210 con 0x7f08c81b0470 2021-02-10T11:17:35.520 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:35.512 7f08cddd3700 1 -- 172.21.2.102:0/3312575819 >> v1:172.21.2.113:6789/0 conn(0x7f08c81b0470 legacy=0x7f08c80ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:17:35.520 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:35.512 7f08cddd3700 1 -- 172.21.2.102:0/3312575819 >> v1:172.21.2.102:6789/0 conn(0x7f08c81b0870 legacy=0x7f08c80efe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:17:35.520 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:35.512 7f08cddd3700 1 -- 172.21.2.102:0/3312575819 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08c80e52e0 con 0x7f08c80f05f0 2021-02-10T11:17:35.521 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:35.512 7f08cddd3700 1 --2- 172.21.2.102:0/3312575819 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f08c80f05f0 0x7f08c80f20e0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f08c0006e50 tx=0x7f08c0009e70).ready entity=mon.2 client_cookie=f5997dde09caca3b server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:17:35.521 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:35.512 7f08bbfff700 1 -- 172.21.2.102:0/3312575819 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f08c000b510 con 0x7f08c80f05f0 2021-02-10T11:17:35.521 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:35.512 7f08bbfff700 1 -- 172.21.2.102:0/3312575819 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08c0006790 con 0x7f08c80f05f0 2021-02-10T11:17:35.521 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:35.512 7f08bbfff700 1 -- 172.21.2.102:0/3312575819 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f08c000b940 con 0x7f08c80f05f0 2021-02-10T11:17:35.521 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:35.512 7f08cf7b6700 1 -- 172.21.2.102:0/3312575819 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f08c80f05f0 msgr2=0x7f08c80f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:17:35.521 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:35.512 7f08cf7b6700 1 --2- 172.21.2.102:0/3312575819 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f08c80f05f0 0x7f08c80f20e0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f08c0006e50 tx=0x7f08c0009e70).stop 2021-02-10T11:17:35.521 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:35.513 7f08cf7b6700 1 -- 172.21.2.102:0/3312575819 shutdown_connections 2021-02-10T11:17:35.521 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:35.513 7f08cf7b6700 1 --2- 172.21.2.102:0/3312575819 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f08c80f05f0 0x7f08c80f20e0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:35.521 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:35.513 7f08cf7b6700 1 -- 172.21.2.102:0/3312575819 >> 172.21.2.102:0/3312575819 conn(0x7f08c80ce3e0 msgr2=0x7f08c80e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:17:35.521 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:35.513 7f08cf7b6700 1 -- 172.21.2.102:0/3312575819 shutdown_connections 2021-02-10T11:17:35.521 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:35.513 7f08cf7b6700 1 -- 172.21.2.102:0/3312575819 wait complete. 2021-02-10T11:17:35.522 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:35.528 7f08cf7b6700 1 Processor -- start 2021-02-10T11:17:35.522 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:35.528 7f08cf7b6700 1 -- start start 2021-02-10T11:17:35.522 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:35.528 7f08cf7b6700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f08c80a0420 0x7f08c80a0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:35.522 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:35.528 7f08cf7b6700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f08c8189d00 0x7f08c818a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:35.522 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:35.528 7f08cf7b6700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f08c818a720 0x7f08c80b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:35.522 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:35.528 7f08cf7b6700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08c8189780 con 0x7f08c818a720 2021-02-10T11:17:35.522 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:35.528 7f08cf7b6700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08c818ab20 con 0x7f08c80a0420 2021-02-10T11:17:35.523 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:35.528 7f08cf7b6700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08c80a1140 con 0x7f08c8189d00 2021-02-10T11:17:35.524 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:35.528 7f08cddd3700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f08c818a720 0x7f08c80b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:17:35.524 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:35.528 7f08cddd3700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f08c818a720 0x7f08c80b7be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:53174/0 (socket says 172.21.2.102:53174) 2021-02-10T11:17:35.524 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:35.528 7f08cddd3700 1 -- 172.21.2.102:0/1453612865 learned_addr learned my addr 172.21.2.102:0/1453612865 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:17:35.524 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:35.529 7f08cddd3700 1 -- 172.21.2.102:0/1453612865 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f08c8189d00 msgr2=0x7f08c818a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:17:35.524 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:35.529 7f08cddd3700 1 --2- 172.21.2.102:0/1453612865 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f08c8189d00 0x7f08c818a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:17:35.524 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:35.529 7f08cddd3700 1 -- 172.21.2.102:0/1453612865 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f08c80a0420 msgr2=0x7f08c80a0820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:17:35.524 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:35.529 7f08cddd3700 1 --2- 172.21.2.102:0/1453612865 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f08c80a0420 0x7f08c80a0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:17:35.524 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:35.529 7f08cddd3700 1 -- 172.21.2.102:0/1453612865 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08c80b8200 con 0x7f08c818a720 2021-02-10T11:17:35.524 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:35.529 7f08cddd3700 1 --2- 172.21.2.102:0/1453612865 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f08c818a720 0x7f08c80b7be0 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 rx=0x7f08c0006c50 tx=0x7f08c000bd20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:17:35.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:35.529 7f08b9ffb700 1 -- 172.21.2.102:0/1453612865 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f08c00093d0 con 0x7f08c818a720 2021-02-10T11:17:35.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:35.529 7f08b9ffb700 1 -- 172.21.2.102:0/1453612865 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08c000be60 con 0x7f08c818a720 2021-02-10T11:17:35.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:35.529 7f08b9ffb700 1 -- 172.21.2.102:0/1453612865 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f08c000c570 con 0x7f08c818a720 2021-02-10T11:17:35.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:35.529 7f08cf7b6700 1 -- 172.21.2.102:0/1453612865 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08c80b8200 con 0x7f08c818a720 2021-02-10T11:17:35.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:35.529 7f08cf7b6700 1 -- 172.21.2.102:0/1453612865 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08c80b8790 con 0x7f08c818a720 2021-02-10T11:17:35.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:35.530 7f08b9ffb700 1 -- 172.21.2.102:0/1453612865 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f08c00093d0 con 0x7f08c818a720 2021-02-10T11:17:35.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:35.530 7f08b9ffb700 1 --2- 172.21.2.102:0/1453612865 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f08a80186c0 0x7f08a801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:35.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:35.530 7f08b9ffb700 1 -- 172.21.2.102:0/1453612865 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f08c0032940 con 0x7f08c818a720 2021-02-10T11:17:35.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:35.530 7f08cd5d2700 1 --2- 172.21.2.102:0/1453612865 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f08a80186c0 0x7f08a801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:17:35.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:35.530 7f08cd5d2700 1 --2- 172.21.2.102:0/1453612865 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f08a80186c0 0x7f08a801abc0 crc :-1 s=READY pgs=332 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:17:35.553 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:35.561 7f08cf7b6700 1 -- 172.21.2.102:0/1453612865 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08c8105ae0 con 0x7f08c818a720 2021-02-10T11:17:35.556 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:35.564 7f08b9ffb700 1 -- 172.21.2.102:0/1453612865 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f08c001f6e0 con 0x7f08c818a720 2021-02-10T11:17:35.681 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:35.688 7f08cf7b6700 1 -- 172.21.2.102:0/1453612865 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344444", "target": ["mgr", ""]}) v1 -- 0x7f08c80b9020 con 0x7f08a80186c0 2021-02-10T11:17:35.708 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:35.716 7f08b9ffb700 1 -- 172.21.2.102:0/1453612865 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f08bc006a20 con 0x7f08a80186c0 2021-02-10T11:17:35.713 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:35.721 7f08cf7b6700 1 -- 172.21.2.102:0/1453612865 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f08a80186c0 msgr2=0x7f08a801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:17:35.714 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:35.721 7f08cf7b6700 1 --2- 172.21.2.102:0/1453612865 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f08a80186c0 0x7f08a801abc0 crc :-1 s=READY pgs=332 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:35.714 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:35.722 7f08cf7b6700 1 -- 172.21.2.102:0/1453612865 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f08c818a720 msgr2=0x7f08c80b7be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:17:35.714 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:35.722 7f08cf7b6700 1 --2- 172.21.2.102:0/1453612865 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f08c818a720 0x7f08c80b7be0 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 rx=0x7f08c0006c50 tx=0x7f08c000bd20).stop 2021-02-10T11:17:35.714 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:35.722 7f08cf7b6700 1 -- 172.21.2.102:0/1453612865 shutdown_connections 2021-02-10T11:17:35.714 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:35.722 7f08cf7b6700 1 --2- 172.21.2.102:0/1453612865 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f08c818a720 0x7f08c80b7be0 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:35.714 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:35.722 7f08cf7b6700 1 --2- 172.21.2.102:0/1453612865 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f08c8189d00 0x7f08c818a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:17:35.715 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:35.722 7f08cf7b6700 1 --2- 172.21.2.102:0/1453612865 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f08a80186c0 0x7f08a801abc0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:35.715 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:35.722 7f08cf7b6700 1 --2- 172.21.2.102:0/1453612865 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f08c80a0420 0x7f08c80a0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:17:35.715 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:35.722 7f08cf7b6700 1 -- 172.21.2.102:0/1453612865 >> 172.21.2.102:0/1453612865 conn(0x7f08c8094e10 msgr2=0x7f08c80a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:17:35.715 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:35.722 7f08cf7b6700 1 -- 172.21.2.102:0/1453612865 shutdown_connections 2021-02-10T11:17:35.715 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:35.722 7f08cf7b6700 1 -- 172.21.2.102:0/1453612865 wait complete. 2021-02-10T11:17:35.722 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000344444 2021-02-10T11:17:36.154 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:36.145 7fc733002700 1 Processor -- start 2021-02-10T11:17:36.154 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:36.145 7fc733002700 1 -- start start 2021-02-10T11:17:36.154 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:36.145 7fc733002700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fc72c1b0470 0x7fc72c1b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:36.154 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:36.145 7fc733002700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fc72c0ef610 0x7fc72c0efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:36.155 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:36.145 7fc733002700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc72c0f0030 0x7fc72c0f0430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:36.155 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:36.145 7fc733002700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc72c0e5ea0 con 0x7fc72c1b0470 2021-02-10T11:17:36.155 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:36.145 7fc733002700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc72c0f0b50 con 0x7fc72c0ef610 2021-02-10T11:17:36.155 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:36.145 7fc733002700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc72c0e52e0 con 0x7fc72c0f0030 2021-02-10T11:17:36.155 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:36.146 7fc730e1e700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fc72c1b0470 0x7fc72c1b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:17:36.155 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:36.146 7fc730e1e700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fc72c1b0470 0x7fc72c1b0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:53178/0 (socket says 172.21.2.102:53178) 2021-02-10T11:17:36.155 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:36.146 7fc730e1e700 1 -- 172.21.2.102:0/20287973 learned_addr learned my addr 172.21.2.102:0/20287973 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:17:36.155 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:36.146 7fc730e1e700 1 -- 172.21.2.102:0/20287973 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc72c0f0030 msgr2=0x7fc72c0f0430 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:17:36.155 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:36.146 7fc730e1e700 1 --2- 172.21.2.102:0/20287973 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc72c0f0030 0x7fc72c0f0430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:17:36.155 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:36.146 7fc730e1e700 1 -- 172.21.2.102:0/20287973 >> v2:172.21.2.113:3300/0 conn(0x7fc72c0ef610 msgr2=0x7fc72c0efa10 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:17:36.155 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:36.146 7fc730e1e700 1 --2- 172.21.2.102:0/20287973 >> v2:172.21.2.113:3300/0 conn(0x7fc72c0ef610 0x7fc72c0efa10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:17:36.155 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:36.146 7fc730e1e700 1 -- 172.21.2.102:0/20287973 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc72c0f10d0 con 0x7fc72c1b0470 2021-02-10T11:17:36.155 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:36.146 7fc730e1e700 1 --2- 172.21.2.102:0/20287973 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc72c1b0470 0x7fc72c1b0870 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 rx=0x7fc714006700 tx=0x7fc714009b80).ready entity=mon.0 client_cookie=13e5efbfea26e390 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:17:36.156 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:36.146 7fc72b7fe700 1 -- 172.21.2.102:0/20287973 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc71400c0c0 con 0x7fc72c1b0470 2021-02-10T11:17:36.156 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:36.146 7fc72b7fe700 1 -- 172.21.2.102:0/20287973 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc71400c340 con 0x7fc72c1b0470 2021-02-10T11:17:36.156 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:36.146 7fc72b7fe700 1 -- 172.21.2.102:0/20287973 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc71400c7a0 con 0x7fc72c1b0470 2021-02-10T11:17:36.156 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:36.146 7fc733002700 1 -- 172.21.2.102:0/20287973 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc72c1b0470 msgr2=0x7fc72c1b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:17:36.156 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:36.146 7fc733002700 1 --2- 172.21.2.102:0/20287973 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc72c1b0470 0x7fc72c1b0870 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 rx=0x7fc714006700 tx=0x7fc714009b80).stop 2021-02-10T11:17:36.156 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:36.147 7fc733002700 1 -- 172.21.2.102:0/20287973 shutdown_connections 2021-02-10T11:17:36.156 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:36.147 7fc733002700 1 --2- 172.21.2.102:0/20287973 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc72c0f0030 0x7fc72c0f0430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:17:36.156 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:36.147 7fc733002700 1 --2- 172.21.2.102:0/20287973 >> v2:172.21.2.113:3300/0 conn(0x7fc72c0ef610 0x7fc72c0efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:17:36.156 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:36.147 7fc733002700 1 --2- 172.21.2.102:0/20287973 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc72c1b0470 0x7fc72c1b0870 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:36.156 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:36.147 7fc733002700 1 -- 172.21.2.102:0/20287973 >> 172.21.2.102:0/20287973 conn(0x7fc72c0ce3e0 msgr2=0x7fc72c0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:17:36.156 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:36.147 7fc733002700 1 -- 172.21.2.102:0/20287973 shutdown_connections 2021-02-10T11:17:36.156 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:36.147 7fc733002700 1 -- 172.21.2.102:0/20287973 wait complete. 2021-02-10T11:17:36.156 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:36.162 7fc733002700 1 Processor -- start 2021-02-10T11:17:36.157 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:36.162 7fc733002700 1 -- start start 2021-02-10T11:17:36.158 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:36.162 7fc733002700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc72c0b20b0 0x7fc72c0ac970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:36.158 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:36.162 7fc733002700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc72c0b24b0 0x7fc72c191dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:36.158 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:36.162 7fc733002700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc72c0acf90 0x7fc72c1923e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:36.158 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:36.162 7fc733002700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc72c0b1b30 con 0x7fc72c0b20b0 2021-02-10T11:17:36.159 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:36.162 7fc733002700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc72c1a04e0 con 0x7fc72c0b24b0 2021-02-10T11:17:36.159 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:36.162 7fc733002700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc72c1a06c0 con 0x7fc72c0acf90 2021-02-10T11:17:36.159 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:36.162 7fc730e1e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc72c0b20b0 0x7fc72c0ac970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:17:36.159 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:36.162 7fc730e1e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc72c0b20b0 0x7fc72c0ac970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:53186/0 (socket says 172.21.2.102:53186) 2021-02-10T11:17:36.159 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:36.162 7fc730e1e700 1 -- 172.21.2.102:0/1186441660 learned_addr learned my addr 172.21.2.102:0/1186441660 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:17:36.159 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:36.163 7fc72bfff700 1 --2- 172.21.2.102:0/1186441660 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc72c0b24b0 0x7fc72c191dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:17:36.159 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:36.163 7fc730e1e700 1 -- 172.21.2.102:0/1186441660 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc72c0acf90 msgr2=0x7fc72c1923e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:17:36.159 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:36.163 7fc73161f700 1 --2- 172.21.2.102:0/1186441660 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc72c0acf90 0x7fc72c1923e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:17:36.159 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:36.163 7fc730e1e700 1 --2- 172.21.2.102:0/1186441660 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc72c0acf90 0x7fc72c1923e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:36.159 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:36.163 7fc730e1e700 1 -- 172.21.2.102:0/1186441660 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc72c0b24b0 msgr2=0x7fc72c191dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:17:36.160 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:36.163 7fc730e1e700 1 --2- 172.21.2.102:0/1186441660 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc72c0b24b0 0x7fc72c191dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:36.160 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:36.163 7fc730e1e700 1 -- 172.21.2.102:0/1186441660 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc72c192be0 con 0x7fc72c0b20b0 2021-02-10T11:17:36.160 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:36.163 7fc72bfff700 1 --2- 172.21.2.102:0/1186441660 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc72c0b24b0 0x7fc72c191dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:17:36.160 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:36.163 7fc73161f700 1 --2- 172.21.2.102:0/1186441660 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc72c0acf90 0x7fc72c1923e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:17:36.160 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:36.163 7fc730e1e700 1 --2- 172.21.2.102:0/1186441660 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc72c0b20b0 0x7fc72c0ac970 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 rx=0x7fc71400b5e0 tx=0x7fc71400b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:17:36.160 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:36.163 7fc7297fa700 1 -- 172.21.2.102:0/1186441660 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc71400bd70 con 0x7fc72c0b20b0 2021-02-10T11:17:36.160 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:36.163 7fc7297fa700 1 -- 172.21.2.102:0/1186441660 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc714002030 con 0x7fc72c0b20b0 2021-02-10T11:17:36.160 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:36.163 7fc733002700 1 -- 172.21.2.102:0/1186441660 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc72c192be0 con 0x7fc72c0b20b0 2021-02-10T11:17:36.160 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:36.163 7fc7297fa700 1 -- 172.21.2.102:0/1186441660 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc714002550 con 0x7fc72c0b20b0 2021-02-10T11:17:36.160 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:36.163 7fc733002700 1 -- 172.21.2.102:0/1186441660 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc72c192be0 con 0x7fc72c0b20b0 2021-02-10T11:17:36.160 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:36.164 7fc7297fa700 1 -- 172.21.2.102:0/1186441660 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fc714000c90 con 0x7fc72c0b20b0 2021-02-10T11:17:36.160 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:36.164 7fc7297fa700 1 --2- 172.21.2.102:0/1186441660 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc710018730 0x7fc71001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:36.160 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:36.164 7fc7297fa700 1 -- 172.21.2.102:0/1186441660 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fc7140326e0 con 0x7fc72c0b20b0 2021-02-10T11:17:36.160 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:36.164 7fc72bfff700 1 --2- 172.21.2.102:0/1186441660 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc710018730 0x7fc71001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:17:36.161 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:36.164 7fc72bfff700 1 --2- 172.21.2.102:0/1186441660 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc710018730 0x7fc71001ac10 crc :-1 s=READY pgs=333 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:17:36.187 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:36.195 7fc733002700 1 -- 172.21.2.102:0/1186441660 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc72c105af0 con 0x7fc72c0b20b0 2021-02-10T11:17:36.190 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:36.198 7fc7297fa700 1 -- 172.21.2.102:0/1186441660 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fc71401f5a0 con 0x7fc72c0b20b0 2021-02-10T11:17:36.318 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:36.326 7fc733002700 1 -- 172.21.2.102:0/1186441660 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344444", "target": ["mgr", ""]}) v1 -- 0x7fc72c0f0b50 con 0x7fc710018730 2021-02-10T11:17:36.321 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:36.329 7fc7297fa700 1 -- 172.21.2.102:0/1186441660 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7fc71c006900 con 0x7fc710018730 2021-02-10T11:17:36.325 INFO:teuthology.orchestra.run.gibba002.stdout:/volumes/_nogroup/subvolume_0000000000344444/6b0c729d-3183-4046-b9fc-27dacf139f27 2021-02-10T11:17:36.326 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:36.334 7fc733002700 1 -- 172.21.2.102:0/1186441660 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc710018730 msgr2=0x7fc71001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:17:36.326 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:36.334 7fc733002700 1 --2- 172.21.2.102:0/1186441660 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc710018730 0x7fc71001ac10 crc :-1 s=READY pgs=333 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:36.327 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:36.334 7fc733002700 1 -- 172.21.2.102:0/1186441660 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc72c0b20b0 msgr2=0x7fc72c0ac970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:17:36.327 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:36.334 7fc733002700 1 --2- 172.21.2.102:0/1186441660 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc72c0b20b0 0x7fc72c0ac970 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 rx=0x7fc71400b5e0 tx=0x7fc71400b980).stop 2021-02-10T11:17:36.327 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:36.335 7fc733002700 1 -- 172.21.2.102:0/1186441660 shutdown_connections 2021-02-10T11:17:36.327 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:36.335 7fc733002700 1 --2- 172.21.2.102:0/1186441660 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc72c0acf90 0x7fc72c1923e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:36.327 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:36.335 7fc733002700 1 --2- 172.21.2.102:0/1186441660 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc710018730 0x7fc71001ac10 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:36.327 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:36.335 7fc733002700 1 --2- 172.21.2.102:0/1186441660 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc72c0b24b0 0x7fc72c191dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:36.328 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:36.335 7fc733002700 1 --2- 172.21.2.102:0/1186441660 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc72c0b20b0 0x7fc72c0ac970 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:36.328 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:36.335 7fc733002700 1 -- 172.21.2.102:0/1186441660 >> 172.21.2.102:0/1186441660 conn(0x7fc72c0bf4e0 msgr2=0x7fc72c0be3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:17:36.328 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:36.335 7fc733002700 1 -- 172.21.2.102:0/1186441660 shutdown_connections 2021-02-10T11:17:36.328 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:36.335 7fc733002700 1 -- 172.21.2.102:0/1186441660 wait complete. 2021-02-10T11:17:36.335 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000344444 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000344444/6b0c729d-3183-4046-b9fc-27dacf139f27 2021-02-10T11:17:36.335 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:17:36.335 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:17:36.336 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:17:36.671 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:17:36.672 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:17:36.672 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344444/6b0c729d-3183-4046-b9fc-27dacf139f27/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:17:37.055 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:17:37.055 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:17:37.055 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0248276 s, 42.2 MB/s 2021-02-10T11:17:37.056 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:17:37.056 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:17:37.056 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:17:37.392 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:17:37.392 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:17:37.392 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344444/6b0c729d-3183-4046-b9fc-27dacf139f27/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:17:37.771 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:17:37.772 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:17:37.772 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.025787 s, 40.7 MB/s 2021-02-10T11:17:37.772 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:17:37.773 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:17:37.773 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:17:38.107 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:17:38.108 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:17:38.108 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344444/6b0c729d-3183-4046-b9fc-27dacf139f27/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:17:38.479 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:17:38.479 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:17:38.479 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0231989 s, 45.2 MB/s 2021-02-10T11:17:38.480 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:17:38.480 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:17:38.480 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:17:38.817 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:17:38.817 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:17:38.817 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344444/6b0c729d-3183-4046-b9fc-27dacf139f27/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:17:39.195 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:17:39.195 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:17:39.195 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0246248 s, 42.6 MB/s 2021-02-10T11:17:39.196 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:17:39.196 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:17:39.196 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:17:39.532 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:17:39.533 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:17:39.533 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344444/6b0c729d-3183-4046-b9fc-27dacf139f27/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:17:39.908 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:17:39.908 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:17:39.909 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0232124 s, 45.2 MB/s 2021-02-10T11:17:39.909 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:17:39.910 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:17:39.910 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:17:40.246 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:17:40.246 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:17:40.246 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344444/6b0c729d-3183-4046-b9fc-27dacf139f27/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:17:40.624 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:17:40.625 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:17:40.625 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0259945 s, 40.3 MB/s 2021-02-10T11:17:40.626 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:17:40.626 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:17:40.626 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:17:40.962 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:17:40.962 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:17:40.962 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344444/6b0c729d-3183-4046-b9fc-27dacf139f27/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:17:41.340 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:17:41.340 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:17:41.340 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0247216 s, 42.4 MB/s 2021-02-10T11:17:41.341 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:17:41.341 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:17:41.341 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:17:41.677 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:17:41.677 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:17:41.677 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344444/6b0c729d-3183-4046-b9fc-27dacf139f27/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:17:42.054 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:17:42.055 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:17:42.055 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0250938 s, 41.8 MB/s 2021-02-10T11:17:42.056 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:17:42.057 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:17:42.057 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:17:42.393 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:17:42.393 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:17:42.394 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344444/6b0c729d-3183-4046-b9fc-27dacf139f27/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:17:42.772 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:17:42.772 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:17:42.772 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.025794 s, 40.7 MB/s 2021-02-10T11:17:42.773 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:17:42.773 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:17:42.773 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:17:43.112 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:17:43.112 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:17:43.112 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344444/6b0c729d-3183-4046-b9fc-27dacf139f27/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:17:43.484 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:17:43.484 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:17:43.484 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0239677 s, 43.7 MB/s 2021-02-10T11:17:43.485 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000344445 2021-02-10T11:17:43.919 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:43.909 7f540cd85700 1 Processor -- start 2021-02-10T11:17:43.919 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:43.909 7f540cd85700 1 -- start start 2021-02-10T11:17:43.919 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:43.909 7f540cd85700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f54081b0470 0x7f54081b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:43.919 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:43.909 7f540cd85700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54080f0b50 con 0x7f54080ef610 2021-02-10T11:17:43.919 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:43.909 7f540cd85700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54080f24e0 con 0x7f54080f20e0 2021-02-10T11:17:43.919 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:43.909 7f540cd85700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54080e5ea0 con 0x7f54081b0470 2021-02-10T11:17:43.919 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:43.909 7f5406312700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f54080ef610 0x7f54080efa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:45414/0 (socket says 172.21.2.102:45414) 2021-02-10T11:17:43.919 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:43.909 7f5406312700 1 -- 172.21.2.102:0/127273216 learned_addr learned my addr 172.21.2.102:0/127273216 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:17:43.919 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:43.910 7f5406b13700 1 --2- 172.21.2.102:0/127273216 >> v2:172.21.2.113:3300/0 conn(0x7f54081b0470 0x7f54081b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:17:43.919 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:43.910 7f5405b11700 1 -- 172.21.2.102:0/127273216 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1674641090 0 0) 0x7f53fc001180 con 0x7f54080ef610 2021-02-10T11:17:43.920 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:43.910 7f5405b11700 1 -- 172.21.2.102:0/127273216 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53f0002fe0 con 0x7f54080ef610 2021-02-10T11:17:43.920 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:43.910 7f5406b13700 1 -- 172.21.2.102:0/127273216 >> v1:172.21.2.113:6789/0 conn(0x7f54080f20e0 legacy=0x7f54080f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:17:43.920 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:43.910 7f5406b13700 1 -- 172.21.2.102:0/127273216 >> v1:172.21.2.102:6789/0 conn(0x7f54080ef610 legacy=0x7f54080efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:17:43.920 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:43.910 7f5406b13700 1 -- 172.21.2.102:0/127273216 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54080e52e0 con 0x7f54081b0470 2021-02-10T11:17:43.920 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:43.910 7f5406b13700 1 --2- 172.21.2.102:0/127273216 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f54081b0470 0x7f54081b0870 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 rx=0x7f54000069d0 tx=0x7f5400009e40).ready entity=mon.1 client_cookie=d9063e69edc1b7be server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:17:43.920 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:43.911 7f5405b11700 1 -- 172.21.2.102:0/127273216 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f540000b510 con 0x7f54081b0470 2021-02-10T11:17:43.920 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:43.911 7f5405b11700 1 -- 172.21.2.102:0/127273216 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f540000b890 con 0x7f54081b0470 2021-02-10T11:17:43.920 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:43.911 7f5405b11700 1 -- 172.21.2.102:0/127273216 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f540000bcf0 con 0x7f54081b0470 2021-02-10T11:17:43.920 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:43.911 7f540cd85700 1 -- 172.21.2.102:0/127273216 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f54081b0470 msgr2=0x7f54081b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:17:43.920 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:43.911 7f540cd85700 1 --2- 172.21.2.102:0/127273216 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f54081b0470 0x7f54081b0870 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 rx=0x7f54000069d0 tx=0x7f5400009e40).stop 2021-02-10T11:17:43.920 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:43.912 7f540cd85700 1 -- 172.21.2.102:0/127273216 shutdown_connections 2021-02-10T11:17:43.920 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:43.912 7f540cd85700 1 --2- 172.21.2.102:0/127273216 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f54081b0470 0x7f54081b0870 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:43.920 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:43.912 7f540cd85700 1 -- 172.21.2.102:0/127273216 >> 172.21.2.102:0/127273216 conn(0x7f54080ce3e0 msgr2=0x7f54080e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:17:43.920 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:43.912 7f540cd85700 1 -- 172.21.2.102:0/127273216 shutdown_connections 2021-02-10T11:17:43.921 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:43.912 7f540cd85700 1 -- 172.21.2.102:0/127273216 wait complete. 2021-02-10T11:17:43.921 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:43.927 7f540cd85700 1 Processor -- start 2021-02-10T11:17:43.921 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:43.927 7f540cd85700 1 -- start start 2021-02-10T11:17:43.922 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:43.927 7f540cd85700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f54080b2840 0x7f54081a4300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:43.922 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:43.927 7f540cd85700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f54080b30d0 0x7f54080b34d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:43.922 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:43.927 7f540cd85700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f54080b3af0 0x7f54080f0580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:43.923 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:43.927 7f540cd85700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54080b2c40 con 0x7f54080b2840 2021-02-10T11:17:43.923 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:43.927 7f540cd85700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54081a4920 con 0x7f54080b3af0 2021-02-10T11:17:43.923 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:43.927 7f540cd85700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54080f0ba0 con 0x7f54080b30d0 2021-02-10T11:17:43.923 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:43.927 7f5406b13700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f54080b2840 0x7f54081a4300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:17:43.923 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:43.927 7f5406b13700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f54080b2840 0x7f54081a4300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:53196/0 (socket says 172.21.2.102:53196) 2021-02-10T11:17:43.923 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:43.927 7f5406b13700 1 -- 172.21.2.102:0/3970366830 learned_addr learned my addr 172.21.2.102:0/3970366830 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:17:43.923 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:43.928 7f5406b13700 1 -- 172.21.2.102:0/3970366830 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f54080b30d0 msgr2=0x7f54080b34d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:17:43.923 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:43.928 7f5406b13700 1 --2- 172.21.2.102:0/3970366830 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f54080b30d0 0x7f54080b34d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:17:43.923 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:43.928 7f5406b13700 1 -- 172.21.2.102:0/3970366830 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f54080b3af0 msgr2=0x7f54080f0580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:17:43.923 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:43.928 7f5406b13700 1 --2- 172.21.2.102:0/3970366830 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f54080b3af0 0x7f54080f0580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:17:43.923 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:43.928 7f5406b13700 1 -- 172.21.2.102:0/3970366830 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54080f0dc0 con 0x7f54080b2840 2021-02-10T11:17:43.923 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:43.928 7f5406b13700 1 --2- 172.21.2.102:0/3970366830 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f54080b2840 0x7f54081a4300 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 rx=0x7f5400009f40 tx=0x7f540000bd50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:17:43.923 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:43.928 7f53f77fe700 1 -- 172.21.2.102:0/3970366830 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5400006240 con 0x7f54080b2840 2021-02-10T11:17:43.924 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:43.928 7f53f77fe700 1 -- 172.21.2.102:0/3970366830 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54000065c0 con 0x7f54080b2840 2021-02-10T11:17:43.924 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:43.928 7f53f77fe700 1 -- 172.21.2.102:0/3970366830 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f54000058e0 con 0x7f54080b2840 2021-02-10T11:17:43.924 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:43.928 7f540cd85700 1 -- 172.21.2.102:0/3970366830 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54080f0dc0 con 0x7f54080b2840 2021-02-10T11:17:43.924 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:43.928 7f540cd85700 1 -- 172.21.2.102:0/3970366830 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54080f1350 con 0x7f54080b2840 2021-02-10T11:17:43.924 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:43.929 7f53f77fe700 1 -- 172.21.2.102:0/3970366830 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f5400006ca0 con 0x7f54080b2840 2021-02-10T11:17:43.924 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:43.929 7f53f77fe700 1 --2- 172.21.2.102:0/3970366830 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f53ec018730 0x7f53ec01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:43.924 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:43.929 7f53f77fe700 1 -- 172.21.2.102:0/3970366830 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f54000329b0 con 0x7f54080b2840 2021-02-10T11:17:43.924 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:43.929 7f5406312700 1 --2- 172.21.2.102:0/3970366830 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f53ec018730 0x7f53ec01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:17:43.924 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:43.929 7f5406312700 1 --2- 172.21.2.102:0/3970366830 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f53ec018730 0x7f53ec01abc0 crc :-1 s=READY pgs=334 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:17:43.951 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:43.959 7f540cd85700 1 -- 172.21.2.102:0/3970366830 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54080f0050 con 0x7f54080b2840 2021-02-10T11:17:43.954 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:43.962 7f53f77fe700 1 -- 172.21.2.102:0/3970366830 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f5400000d30 con 0x7f54080b2840 2021-02-10T11:17:44.080 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:44.088 7f540cd85700 1 -- 172.21.2.102:0/3970366830 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344445", "target": ["mgr", ""]}) v1 -- 0x7f54080f1350 con 0x7f53ec018730 2021-02-10T11:17:44.110 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:44.118 7f53f77fe700 1 -- 172.21.2.102:0/3970366830 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f53fc000ce0 con 0x7f53ec018730 2021-02-10T11:17:44.114 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:44.121 7f540cd85700 1 -- 172.21.2.102:0/3970366830 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f53ec018730 msgr2=0x7f53ec01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:17:44.114 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:44.121 7f540cd85700 1 --2- 172.21.2.102:0/3970366830 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f53ec018730 0x7f53ec01abc0 crc :-1 s=READY pgs=334 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:44.115 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:44.121 7f540cd85700 1 -- 172.21.2.102:0/3970366830 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f54080b2840 msgr2=0x7f54081a4300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:17:44.115 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:44.121 7f540cd85700 1 --2- 172.21.2.102:0/3970366830 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f54080b2840 0x7f54081a4300 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 rx=0x7f5400009f40 tx=0x7f540000bd50).stop 2021-02-10T11:17:44.115 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:44.122 7f540cd85700 1 -- 172.21.2.102:0/3970366830 shutdown_connections 2021-02-10T11:17:44.115 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:44.122 7f540cd85700 1 --2- 172.21.2.102:0/3970366830 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f53ec018730 0x7f53ec01abc0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:44.115 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:44.122 7f540cd85700 1 --2- 172.21.2.102:0/3970366830 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f54080b3af0 0x7f54080f0580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:17:44.115 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:44.122 7f540cd85700 1 --2- 172.21.2.102:0/3970366830 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f54080b30d0 0x7f54080b34d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:17:44.115 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:44.122 7f540cd85700 1 --2- 172.21.2.102:0/3970366830 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f54080b2840 0x7f54081a4300 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:44.115 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:44.122 7f540cd85700 1 -- 172.21.2.102:0/3970366830 >> 172.21.2.102:0/3970366830 conn(0x7f54080c36f0 msgr2=0x7f5408094c30 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:17:44.115 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:44.122 7f540cd85700 1 -- 172.21.2.102:0/3970366830 shutdown_connections 2021-02-10T11:17:44.115 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:44.122 7f540cd85700 1 -- 172.21.2.102:0/3970366830 wait complete. 2021-02-10T11:17:44.122 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000344445 2021-02-10T11:17:44.556 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:44.547 7fa8f1a88700 1 Processor -- start 2021-02-10T11:17:44.556 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:44.547 7fa8f1a88700 1 -- start start 2021-02-10T11:17:44.556 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:44.548 7fa8f1a88700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa8ec1b0470 0x7fa8ec1b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:44.556 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:44.548 7fa8f1a88700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fa8ec0f20e0 0x7fa8ec0f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:44.557 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:44.548 7fa8f1a88700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8ec0f0b50 con 0x7fa8ec0ef610 2021-02-10T11:17:44.557 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:44.548 7fa8f1a88700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8ec0f0820 con 0x7fa8ec0f20e0 2021-02-10T11:17:44.557 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:44.548 7fa8f1a88700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8ec0f24e0 con 0x7fa8ec1b0470 2021-02-10T11:17:44.557 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:44.548 7fa8ebfff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fa8ec0f20e0 0x7fa8ec0f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:17:44.557 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:44.548 7fa8ebfff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fa8ec0f20e0 0x7fa8ec0f0200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:53204/0 (socket says 172.21.2.102:53204) 2021-02-10T11:17:44.557 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:44.548 7fa8ebfff700 1 -- 172.21.2.102:0/828929590 learned_addr learned my addr 172.21.2.102:0/828929590 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:17:44.557 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:44.548 7fa8eb7fe700 1 --2- 172.21.2.102:0/828929590 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa8ec1b0470 0x7fa8ec1b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:17:44.557 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:44.549 7fa8ebfff700 1 -- 172.21.2.102:0/828929590 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa8ec1b0470 msgr2=0x7fa8ec1b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:17:44.557 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:44.549 7fa8ebfff700 1 --2- 172.21.2.102:0/828929590 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa8ec1b0470 0x7fa8ec1b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:44.557 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:44.549 7fa8ebfff700 1 -- 172.21.2.102:0/828929590 >> v1:172.21.2.102:6789/0 conn(0x7fa8ec0ef610 legacy=0x7fa8ec0efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:17:44.557 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:44.549 7fa8ebfff700 1 -- 172.21.2.102:0/828929590 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8ec0e52e0 con 0x7fa8ec0f20e0 2021-02-10T11:17:44.557 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:44.549 7fa8ebfff700 1 --2- 172.21.2.102:0/828929590 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa8ec0f20e0 0x7fa8ec0f0200 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 rx=0x7fa8e4006f00 tx=0x7fa8e4009f10).ready entity=mon.0 client_cookie=dc9c9602660a1701 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:17:44.558 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:44.549 7fa8ea7fc700 1 -- 172.21.2.102:0/828929590 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa8e400b3b0 con 0x7fa8ec0f20e0 2021-02-10T11:17:44.558 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:44.549 7fa8ea7fc700 1 -- 172.21.2.102:0/828929590 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8e400b630 con 0x7fa8ec0f20e0 2021-02-10T11:17:44.558 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:44.549 7fa8ea7fc700 1 -- 172.21.2.102:0/828929590 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa8e400bb50 con 0x7fa8ec0f20e0 2021-02-10T11:17:44.558 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:44.549 7fa8f1a88700 1 -- 172.21.2.102:0/828929590 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa8ec0f20e0 msgr2=0x7fa8ec0f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:17:44.558 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:44.549 7fa8f1a88700 1 --2- 172.21.2.102:0/828929590 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa8ec0f20e0 0x7fa8ec0f0200 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 rx=0x7fa8e4006f00 tx=0x7fa8e4009f10).stop 2021-02-10T11:17:44.558 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:44.550 7fa8f1a88700 1 -- 172.21.2.102:0/828929590 shutdown_connections 2021-02-10T11:17:44.558 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:44.550 7fa8f1a88700 1 --2- 172.21.2.102:0/828929590 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa8ec0f20e0 0x7fa8ec0f0200 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:44.558 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:44.550 7fa8f1a88700 1 --2- 172.21.2.102:0/828929590 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa8ec1b0470 0x7fa8ec1b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:44.558 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:44.550 7fa8f1a88700 1 -- 172.21.2.102:0/828929590 >> 172.21.2.102:0/828929590 conn(0x7fa8ec0ce3e0 msgr2=0x7fa8ec0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:17:44.558 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:44.550 7fa8f1a88700 1 -- 172.21.2.102:0/828929590 shutdown_connections 2021-02-10T11:17:44.558 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:44.550 7fa8f1a88700 1 -- 172.21.2.102:0/828929590 wait complete. 2021-02-10T11:17:44.558 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:44.564 7fa8f1a88700 1 Processor -- start 2021-02-10T11:17:44.559 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:44.565 7fa8f1a88700 1 -- start start 2021-02-10T11:17:44.559 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:44.565 7fa8f1a88700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa8ec0b7de0 0x7fa8ec190920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:44.559 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:44.565 7fa8f1a88700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa8ec0b82f0 0x7fa8ec0b86f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:44.559 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:44.565 7fa8f1a88700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa8ec0b8d10 0x7fa8ec0bf9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:44.559 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:44.565 7fa8f1a88700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8ec0b7930 con 0x7fa8ec0b7de0 2021-02-10T11:17:44.559 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:44.565 7fa8f1a88700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8ec0b7b20 con 0x7fa8ec0b82f0 2021-02-10T11:17:44.560 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:44.565 7fa8f1a88700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8ec0bffd0 con 0x7fa8ec0b8d10 2021-02-10T11:17:44.561 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:44.565 7fa8eb7fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa8ec0b7de0 0x7fa8ec190920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:17:44.561 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:44.565 7fa8eb7fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa8ec0b7de0 0x7fa8ec190920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:53208/0 (socket says 172.21.2.102:53208) 2021-02-10T11:17:44.561 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:44.565 7fa8eb7fe700 1 -- 172.21.2.102:0/3569398785 learned_addr learned my addr 172.21.2.102:0/3569398785 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:17:44.561 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:44.565 7fa8eaffd700 1 --2- 172.21.2.102:0/3569398785 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa8ec0b82f0 0x7fa8ec0b86f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:17:44.561 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:44.565 7fa8ebfff700 1 --2- 172.21.2.102:0/3569398785 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa8ec0b8d10 0x7fa8ec0bf9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:17:44.561 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:44.566 7fa8eb7fe700 1 -- 172.21.2.102:0/3569398785 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa8ec0b8d10 msgr2=0x7fa8ec0bf9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:17:44.561 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:44.566 7fa8eb7fe700 1 --2- 172.21.2.102:0/3569398785 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa8ec0b8d10 0x7fa8ec0bf9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:44.561 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:44.566 7fa8eb7fe700 1 -- 172.21.2.102:0/3569398785 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa8ec0b82f0 msgr2=0x7fa8ec0b86f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:17:44.562 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:44.566 7fa8eb7fe700 1 --2- 172.21.2.102:0/3569398785 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa8ec0b82f0 0x7fa8ec0b86f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:44.562 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:44.566 7fa8eb7fe700 1 -- 172.21.2.102:0/3569398785 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8ec0c01c0 con 0x7fa8ec0b7de0 2021-02-10T11:17:44.562 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:44.566 7fa8eb7fe700 1 --2- 172.21.2.102:0/3569398785 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa8ec0b7de0 0x7fa8ec190920 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 rx=0x7fa8d4006f60 tx=0x7fa8d400b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:17:44.562 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:44.566 7fa8e3fff700 1 -- 172.21.2.102:0/3569398785 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa8d400bd60 con 0x7fa8ec0b7de0 2021-02-10T11:17:44.562 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:44.566 7fa8e3fff700 1 -- 172.21.2.102:0/3569398785 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8d4000c90 con 0x7fa8ec0b7de0 2021-02-10T11:17:44.562 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:44.566 7fa8e3fff700 1 -- 172.21.2.102:0/3569398785 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa8d4002220 con 0x7fa8ec0b7de0 2021-02-10T11:17:44.562 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:44.566 7fa8f1a88700 1 -- 172.21.2.102:0/3569398785 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8ec0c01c0 con 0x7fa8ec0b7de0 2021-02-10T11:17:44.562 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:44.566 7fa8f1a88700 1 -- 172.21.2.102:0/3569398785 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8ec0c0750 con 0x7fa8ec0b7de0 2021-02-10T11:17:44.562 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:44.567 7fa8e3fff700 1 -- 172.21.2.102:0/3569398785 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa8d40052e0 con 0x7fa8ec0b7de0 2021-02-10T11:17:44.562 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:44.567 7fa8e3fff700 1 --2- 172.21.2.102:0/3569398785 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa8d00186c0 0x7fa8d001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:44.562 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:44.567 7fa8e3fff700 1 -- 172.21.2.102:0/3569398785 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fa8d4032540 con 0x7fa8ec0b7de0 2021-02-10T11:17:44.562 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:44.567 7fa8eaffd700 1 --2- 172.21.2.102:0/3569398785 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa8d00186c0 0x7fa8d001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:17:44.563 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:44.567 7fa8eaffd700 1 --2- 172.21.2.102:0/3569398785 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa8d00186c0 0x7fa8d001abc0 crc :-1 s=READY pgs=335 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:17:44.590 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:44.598 7fa8f1a88700 1 -- 172.21.2.102:0/3569398785 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8ec105b30 con 0x7fa8ec0b7de0 2021-02-10T11:17:44.592 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:44.601 7fa8e3fff700 1 -- 172.21.2.102:0/3569398785 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fa8d4025f70 con 0x7fa8ec0b7de0 2021-02-10T11:17:44.720 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:44.728 7fa8f1a88700 1 -- 172.21.2.102:0/3569398785 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344445", "target": ["mgr", ""]}) v1 -- 0x7fa8ec0c0750 con 0x7fa8d00186c0 2021-02-10T11:17:44.723 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:44.731 7fa8e3fff700 1 -- 172.21.2.102:0/3569398785 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7fa8dc0054b0 con 0x7fa8d00186c0 2021-02-10T11:17:44.727 INFO:teuthology.orchestra.run.gibba002.stdout:/volumes/_nogroup/subvolume_0000000000344445/e7eba796-dab9-4e51-8099-4f3005ec0676 2021-02-10T11:17:44.728 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:44.736 7fa8f1a88700 1 -- 172.21.2.102:0/3569398785 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa8d00186c0 msgr2=0x7fa8d001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:17:44.728 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:44.736 7fa8f1a88700 1 --2- 172.21.2.102:0/3569398785 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa8d00186c0 0x7fa8d001abc0 crc :-1 s=READY pgs=335 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:44.729 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:44.736 7fa8f1a88700 1 -- 172.21.2.102:0/3569398785 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa8ec0b7de0 msgr2=0x7fa8ec190920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:17:44.729 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:44.736 7fa8f1a88700 1 --2- 172.21.2.102:0/3569398785 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa8ec0b7de0 0x7fa8ec190920 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 rx=0x7fa8d4006f60 tx=0x7fa8d400b970).stop 2021-02-10T11:17:44.729 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:44.737 7fa8f1a88700 1 -- 172.21.2.102:0/3569398785 shutdown_connections 2021-02-10T11:17:44.729 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:44.737 7fa8f1a88700 1 --2- 172.21.2.102:0/3569398785 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa8ec0b8d10 0x7fa8ec0bf9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:44.729 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:44.737 7fa8f1a88700 1 --2- 172.21.2.102:0/3569398785 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa8d00186c0 0x7fa8d001abc0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:44.729 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:44.737 7fa8f1a88700 1 --2- 172.21.2.102:0/3569398785 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa8ec0b82f0 0x7fa8ec0b86f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:44.729 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:44.737 7fa8f1a88700 1 --2- 172.21.2.102:0/3569398785 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa8ec0b7de0 0x7fa8ec190920 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:44.729 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:44.737 7fa8f1a88700 1 -- 172.21.2.102:0/3569398785 >> 172.21.2.102:0/3569398785 conn(0x7fa8ec094e10 msgr2=0x7fa8ec0a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:17:44.730 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:44.737 7fa8f1a88700 1 -- 172.21.2.102:0/3569398785 shutdown_connections 2021-02-10T11:17:44.730 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:44.737 7fa8f1a88700 1 -- 172.21.2.102:0/3569398785 wait complete. 2021-02-10T11:17:44.737 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000344445 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000344445/e7eba796-dab9-4e51-8099-4f3005ec0676 2021-02-10T11:17:44.737 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:17:44.737 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:17:44.737 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:17:45.073 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:17:45.073 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:17:45.074 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344445/e7eba796-dab9-4e51-8099-4f3005ec0676/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:17:45.456 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:17:45.456 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:17:45.456 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0233381 s, 44.9 MB/s 2021-02-10T11:17:45.458 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:17:45.458 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:17:45.458 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:17:45.792 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:17:45.793 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:17:45.793 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344445/e7eba796-dab9-4e51-8099-4f3005ec0676/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:17:46.169 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:17:46.170 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:17:46.170 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0240368 s, 43.6 MB/s 2021-02-10T11:17:46.171 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:17:46.171 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:17:46.171 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:17:46.507 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:17:46.507 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:17:46.508 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344445/e7eba796-dab9-4e51-8099-4f3005ec0676/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:17:46.886 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:17:46.886 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:17:46.886 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0247883 s, 42.3 MB/s 2021-02-10T11:17:46.887 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:17:46.887 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:17:46.887 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:17:47.223 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:17:47.224 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:17:47.224 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344445/e7eba796-dab9-4e51-8099-4f3005ec0676/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:17:47.601 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:17:47.601 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:17:47.601 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.024955 s, 42.0 MB/s 2021-02-10T11:17:47.602 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:17:47.602 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:17:47.602 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:17:47.939 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:17:47.939 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:17:47.940 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344445/e7eba796-dab9-4e51-8099-4f3005ec0676/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:17:48.315 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:17:48.315 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:17:48.315 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0247277 s, 42.4 MB/s 2021-02-10T11:17:48.316 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:17:48.317 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:17:48.317 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:17:48.653 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:17:48.654 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:17:48.654 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344445/e7eba796-dab9-4e51-8099-4f3005ec0676/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:17:49.026 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:17:49.026 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:17:49.026 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0237121 s, 44.2 MB/s 2021-02-10T11:17:49.027 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:17:49.027 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:17:49.027 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:17:49.364 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:17:49.364 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:17:49.364 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344445/e7eba796-dab9-4e51-8099-4f3005ec0676/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:17:49.747 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:17:49.747 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:17:49.747 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0241806 s, 43.4 MB/s 2021-02-10T11:17:49.748 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:17:49.748 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:17:49.748 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:17:50.085 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:17:50.085 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:17:50.085 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344445/e7eba796-dab9-4e51-8099-4f3005ec0676/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:17:50.459 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:17:50.459 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:17:50.459 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0238097 s, 44.0 MB/s 2021-02-10T11:17:50.460 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:17:50.461 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:17:50.461 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:17:50.797 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:17:50.797 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:17:50.797 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344445/e7eba796-dab9-4e51-8099-4f3005ec0676/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:17:51.178 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:17:51.178 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:17:51.178 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0235066 s, 44.6 MB/s 2021-02-10T11:17:51.179 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:17:51.179 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:17:51.179 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:17:51.515 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:17:51.515 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:17:51.516 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344445/e7eba796-dab9-4e51-8099-4f3005ec0676/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:17:51.891 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:17:51.891 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:17:51.891 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0244868 s, 42.8 MB/s 2021-02-10T11:17:51.892 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000344446 2021-02-10T11:17:52.325 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:52.316 7fd3f5525700 1 Processor -- start 2021-02-10T11:17:52.326 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:52.316 7fd3f5525700 1 -- start start 2021-02-10T11:17:52.326 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:52.317 7fd3f5525700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fd3f01b0470 0x7fd3f01b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:52.326 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:52.317 7fd3f5525700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd3f00ef610 0x7fd3f00efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:52.326 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:52.317 7fd3f5525700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3f00f0b50 con 0x7fd3f00f0030 2021-02-10T11:17:52.326 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:52.317 7fd3f5525700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3f00e52e0 con 0x7fd3f01b0470 2021-02-10T11:17:52.326 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:52.317 7fd3f5525700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3f00e54d0 con 0x7fd3f00ef610 2021-02-10T11:17:52.326 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:52.317 7fd3efac0700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fd3f00f0030 0x7fd3f00f10d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:45440/0 (socket says 172.21.2.102:45440) 2021-02-10T11:17:52.326 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:52.317 7fd3efac0700 1 -- 172.21.2.102:0/4139126101 learned_addr learned my addr 172.21.2.102:0/4139126101 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:17:52.326 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:52.317 7fd3ef2bf700 1 --2- 172.21.2.102:0/4139126101 >> v2:172.21.2.113:3300/0 conn(0x7fd3f01b0470 0x7fd3f01b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:17:52.326 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:52.317 7fd3eeabe700 1 --2- 172.21.2.102:0/4139126101 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd3f00ef610 0x7fd3f00efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:17:52.326 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:52.318 7fd3ee2bd700 1 -- 172.21.2.102:0/4139126101 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 935702907 0 0) 0x7fd3e8001500 con 0x7fd3f00f0030 2021-02-10T11:17:52.327 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:52.318 7fd3ee2bd700 1 -- 172.21.2.102:0/4139126101 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd3dc002fe0 con 0x7fd3f00f0030 2021-02-10T11:17:52.327 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:52.318 7fd3ef2bf700 1 -- 172.21.2.102:0/4139126101 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd3f00ef610 msgr2=0x7fd3f00efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:17:52.327 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:52.318 7fd3ef2bf700 1 --2- 172.21.2.102:0/4139126101 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd3f00ef610 0x7fd3f00efa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:52.327 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:52.318 7fd3ef2bf700 1 -- 172.21.2.102:0/4139126101 >> v1:172.21.2.102:6789/0 conn(0x7fd3f00f0030 legacy=0x7fd3f00f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:17:52.327 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:52.318 7fd3ef2bf700 1 -- 172.21.2.102:0/4139126101 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3f00f0430 con 0x7fd3f01b0470 2021-02-10T11:17:52.327 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:52.318 7fd3eeabe700 1 --2- 172.21.2.102:0/4139126101 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd3f00ef610 0x7fd3f00efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:17:52.327 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:52.318 7fd3ef2bf700 1 --2- 172.21.2.102:0/4139126101 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd3f01b0470 0x7fd3f01b0870 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7fd3e0006650 tx=0x7fd3e0009a80).ready entity=mon.1 client_cookie=c461391f8458fb88 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:17:52.327 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:52.318 7fd3ee2bd700 1 -- 172.21.2.102:0/4139126101 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd3e000c0c0 con 0x7fd3f01b0470 2021-02-10T11:17:52.327 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:52.318 7fd3ee2bd700 1 -- 172.21.2.102:0/4139126101 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3e000c340 con 0x7fd3f01b0470 2021-02-10T11:17:52.327 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:52.318 7fd3ee2bd700 1 -- 172.21.2.102:0/4139126101 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd3e000c860 con 0x7fd3f01b0470 2021-02-10T11:17:52.327 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:52.318 7fd3f5525700 1 -- 172.21.2.102:0/4139126101 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd3f01b0470 msgr2=0x7fd3f01b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:17:52.327 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:52.318 7fd3f5525700 1 --2- 172.21.2.102:0/4139126101 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd3f01b0470 0x7fd3f01b0870 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7fd3e0006650 tx=0x7fd3e0009a80).stop 2021-02-10T11:17:52.327 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:52.319 7fd3f5525700 1 -- 172.21.2.102:0/4139126101 shutdown_connections 2021-02-10T11:17:52.328 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:52.319 7fd3f5525700 1 --2- 172.21.2.102:0/4139126101 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd3f00ef610 0x7fd3f00efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:52.328 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:52.319 7fd3f5525700 1 --2- 172.21.2.102:0/4139126101 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd3f01b0470 0x7fd3f01b0870 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:52.328 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:52.319 7fd3f5525700 1 -- 172.21.2.102:0/4139126101 >> 172.21.2.102:0/4139126101 conn(0x7fd3f00ce3e0 msgr2=0x7fd3f00e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:17:52.328 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:52.319 7fd3f5525700 1 -- 172.21.2.102:0/4139126101 shutdown_connections 2021-02-10T11:17:52.328 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:52.319 7fd3f5525700 1 -- 172.21.2.102:0/4139126101 wait complete. 2021-02-10T11:17:52.328 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:52.334 7fd3f5525700 1 Processor -- start 2021-02-10T11:17:52.329 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:52.334 7fd3f5525700 1 -- start start 2021-02-10T11:17:52.329 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:52.334 7fd3f5525700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd3f0190920 0x7fd3f00b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:52.329 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:52.334 7fd3f5525700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd3f00b8200 0x7fd3f00b8600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:52.329 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:52.334 7fd3f5525700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd3f00c1d80 0x7fd3f00c2180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:52.330 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:52.334 7fd3f5525700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3f0190da0 con 0x7fd3f00b8200 2021-02-10T11:17:52.330 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:52.334 7fd3f5525700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3f00b8c20 con 0x7fd3f00c1d80 2021-02-10T11:17:52.330 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:52.334 7fd3f5525700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3f00b8e10 con 0x7fd3f0190920 2021-02-10T11:17:52.330 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:52.334 7fd3eeabe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd3f00b8200 0x7fd3f00b8600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:17:52.330 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:52.334 7fd3eeabe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd3f00b8200 0x7fd3f00b8600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:53222/0 (socket says 172.21.2.102:53222) 2021-02-10T11:17:52.330 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:52.334 7fd3eeabe700 1 -- 172.21.2.102:0/1852718734 learned_addr learned my addr 172.21.2.102:0/1852718734 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:17:52.330 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:52.335 7fd3eeabe700 1 -- 172.21.2.102:0/1852718734 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd3f0190920 msgr2=0x7fd3f00b7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:17:52.330 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:52.335 7fd3eeabe700 1 --2- 172.21.2.102:0/1852718734 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd3f0190920 0x7fd3f00b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:17:52.330 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:52.335 7fd3eeabe700 1 -- 172.21.2.102:0/1852718734 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd3f00c1d80 msgr2=0x7fd3f00c2180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:17:52.330 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:52.335 7fd3eeabe700 1 --2- 172.21.2.102:0/1852718734 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd3f00c1d80 0x7fd3f00c2180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:17:52.330 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:52.335 7fd3eeabe700 1 -- 172.21.2.102:0/1852718734 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3f00c27d0 con 0x7fd3f00b8200 2021-02-10T11:17:52.330 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:52.335 7fd3eeabe700 1 --2- 172.21.2.102:0/1852718734 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd3f00b8200 0x7fd3f00b8600 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 rx=0x7fd3d8006f40 tx=0x7fd3d800b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:17:52.331 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:52.335 7fd3e7fff700 1 -- 172.21.2.102:0/1852718734 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd3d8006bb0 con 0x7fd3f00b8200 2021-02-10T11:17:52.331 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:52.335 7fd3e7fff700 1 -- 172.21.2.102:0/1852718734 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3d800bd80 con 0x7fd3f00b8200 2021-02-10T11:17:52.331 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:52.335 7fd3e7fff700 1 -- 172.21.2.102:0/1852718734 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd3d80022b0 con 0x7fd3f00b8200 2021-02-10T11:17:52.331 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:52.335 7fd3f5525700 1 -- 172.21.2.102:0/1852718734 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3f00c27d0 con 0x7fd3f00b8200 2021-02-10T11:17:52.331 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:52.335 7fd3f5525700 1 -- 172.21.2.102:0/1852718734 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3f00c2d60 con 0x7fd3f00b8200 2021-02-10T11:17:52.331 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:52.336 7fd3e7fff700 1 -- 172.21.2.102:0/1852718734 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd3d8006b00 con 0x7fd3f00b8200 2021-02-10T11:17:52.331 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:52.336 7fd3e7fff700 1 --2- 172.21.2.102:0/1852718734 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd3d40186c0 0x7fd3d401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:52.331 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:52.336 7fd3e7fff700 1 -- 172.21.2.102:0/1852718734 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fd3d8032540 con 0x7fd3f00b8200 2021-02-10T11:17:52.331 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:52.336 7fd3ef2bf700 1 --2- 172.21.2.102:0/1852718734 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd3d40186c0 0x7fd3d401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:17:52.331 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:52.336 7fd3ef2bf700 1 --2- 172.21.2.102:0/1852718734 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd3d40186c0 0x7fd3d401abc0 crc :-1 s=READY pgs=336 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:17:52.358 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:52.367 7fd3f5525700 1 -- 172.21.2.102:0/1852718734 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3f0105ae0 con 0x7fd3f00b8200 2021-02-10T11:17:52.361 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:52.369 7fd3e7fff700 1 -- 172.21.2.102:0/1852718734 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd3d8000940 con 0x7fd3f00b8200 2021-02-10T11:17:52.488 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:52.496 7fd3f5525700 1 -- 172.21.2.102:0/1852718734 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344446", "target": ["mgr", ""]}) v1 -- 0x7fd3f00c2d60 con 0x7fd3d40186c0 2021-02-10T11:17:52.514 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:52.522 7fd3e7fff700 1 -- 172.21.2.102:0/1852718734 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fd3e0000940 con 0x7fd3d40186c0 2021-02-10T11:17:52.521 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:52.529 7fd3f5525700 1 -- 172.21.2.102:0/1852718734 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd3d40186c0 msgr2=0x7fd3d401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:17:52.521 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:52.529 7fd3f5525700 1 --2- 172.21.2.102:0/1852718734 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd3d40186c0 0x7fd3d401abc0 crc :-1 s=READY pgs=336 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:52.522 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:52.529 7fd3f5525700 1 -- 172.21.2.102:0/1852718734 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd3f00b8200 msgr2=0x7fd3f00b8600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:17:52.522 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:52.529 7fd3f5525700 1 --2- 172.21.2.102:0/1852718734 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd3f00b8200 0x7fd3f00b8600 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 rx=0x7fd3d8006f40 tx=0x7fd3d800b990).stop 2021-02-10T11:17:52.522 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:52.529 7fd3f5525700 1 -- 172.21.2.102:0/1852718734 shutdown_connections 2021-02-10T11:17:52.522 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:52.529 7fd3f5525700 1 --2- 172.21.2.102:0/1852718734 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd3d40186c0 0x7fd3d401abc0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:52.522 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:52.529 7fd3f5525700 1 --2- 172.21.2.102:0/1852718734 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd3f00c1d80 0x7fd3f00c2180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:17:52.522 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:52.529 7fd3f5525700 1 --2- 172.21.2.102:0/1852718734 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd3f00b8200 0x7fd3f00b8600 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:52.522 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:52.529 7fd3f5525700 1 --2- 172.21.2.102:0/1852718734 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd3f0190920 0x7fd3f00b7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:17:52.522 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:52.529 7fd3f5525700 1 -- 172.21.2.102:0/1852718734 >> 172.21.2.102:0/1852718734 conn(0x7fd3f0094e10 msgr2=0x7fd3f00a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:17:52.523 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:52.530 7fd3f5525700 1 -- 172.21.2.102:0/1852718734 shutdown_connections 2021-02-10T11:17:52.523 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:52.530 7fd3f5525700 1 -- 172.21.2.102:0/1852718734 wait complete. 2021-02-10T11:17:52.529 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000344446 2021-02-10T11:17:52.963 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:52.955 7f57dd036700 1 Processor -- start 2021-02-10T11:17:52.964 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:52.955 7f57dd036700 1 -- start start 2021-02-10T11:17:52.964 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:52.955 7f57dd036700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f57d81b0470 0x7f57d81b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:52.964 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:52.955 7f57dd036700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f57d80ef610 0x7f57d80efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:52.964 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:52.955 7f57dd036700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57d80f0b50 con 0x7f57d80f0030 2021-02-10T11:17:52.964 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:52.955 7f57dd036700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57d80e52e0 con 0x7f57d80ef610 2021-02-10T11:17:52.964 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:52.955 7f57dd036700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57d80e5490 con 0x7f57d81b0470 2021-02-10T11:17:52.964 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:52.956 7f57d659c700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f57d80ef610 0x7f57d80efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:17:52.964 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:52.956 7f57d659c700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f57d80ef610 0x7f57d80efa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:53226/0 (socket says 172.21.2.102:53226) 2021-02-10T11:17:52.964 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:52.956 7f57d659c700 1 -- 172.21.2.102:0/3650638505 learned_addr learned my addr 172.21.2.102:0/3650638505 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:17:52.964 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:52.956 7f57d659c700 1 -- 172.21.2.102:0/3650638505 >> v2:172.21.2.113:3300/0 conn(0x7f57d81b0470 msgr2=0x7f57d81b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:17:52.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:52.956 7f57d659c700 1 --2- 172.21.2.102:0/3650638505 >> v2:172.21.2.113:3300/0 conn(0x7f57d81b0470 0x7f57d81b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:17:52.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:52.956 7f57d659c700 1 -- 172.21.2.102:0/3650638505 >> v1:172.21.2.113:6789/0 conn(0x7f57d80f0030 legacy=0x7f57d80f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:17:52.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:52.956 7f57d659c700 1 -- 172.21.2.102:0/3650638505 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57d80f0430 con 0x7f57d80ef610 2021-02-10T11:17:52.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:52.956 7f57d659c700 1 --2- 172.21.2.102:0/3650638505 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f57d80ef610 0x7f57d80efa10 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 rx=0x7f57c8006700 tx=0x7f57c8009b80).ready entity=mon.0 client_cookie=36b38d5fb13fb929 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:17:52.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:52.956 7f57d5d9b700 1 -- 172.21.2.102:0/3650638505 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f57c800c0c0 con 0x7f57d80ef610 2021-02-10T11:17:52.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:52.956 7f57d5d9b700 1 -- 172.21.2.102:0/3650638505 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57c800c340 con 0x7f57d80ef610 2021-02-10T11:17:52.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:52.956 7f57d5d9b700 1 -- 172.21.2.102:0/3650638505 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f57c800c820 con 0x7f57d80ef610 2021-02-10T11:17:52.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:52.956 7f57dd036700 1 -- 172.21.2.102:0/3650638505 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f57d80ef610 msgr2=0x7f57d80efa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:17:52.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:52.956 7f57dd036700 1 --2- 172.21.2.102:0/3650638505 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f57d80ef610 0x7f57d80efa10 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 rx=0x7f57c8006700 tx=0x7f57c8009b80).stop 2021-02-10T11:17:52.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:52.957 7f57dd036700 1 -- 172.21.2.102:0/3650638505 shutdown_connections 2021-02-10T11:17:52.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:52.957 7f57dd036700 1 --2- 172.21.2.102:0/3650638505 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f57d80ef610 0x7f57d80efa10 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:52.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:52.957 7f57dd036700 1 --2- 172.21.2.102:0/3650638505 >> v2:172.21.2.113:3300/0 conn(0x7f57d81b0470 0x7f57d81b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:17:52.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:52.957 7f57dd036700 1 -- 172.21.2.102:0/3650638505 >> 172.21.2.102:0/3650638505 conn(0x7f57d80ce3e0 msgr2=0x7f57d80e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:17:52.966 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:52.957 7f57dd036700 1 -- 172.21.2.102:0/3650638505 shutdown_connections 2021-02-10T11:17:52.966 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:52.957 7f57dd036700 1 -- 172.21.2.102:0/3650638505 wait complete. 2021-02-10T11:17:52.966 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:52.972 7f57dd036700 1 Processor -- start 2021-02-10T11:17:52.966 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:52.972 7f57dd036700 1 -- start start 2021-02-10T11:17:52.967 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:52.972 7f57dd036700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f57d80c36a0 0x7f57d80c2b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:52.967 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:52.972 7f57dd036700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f57d80c3190 0x7f57d80b1d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:52.967 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:52.972 7f57dd036700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f57d80b2370 0x7f57d80b2d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:52.967 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:52.972 7f57dd036700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57d80b27c0 con 0x7f57d80c36a0 2021-02-10T11:17:52.967 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:52.972 7f57dd036700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57d80b3360 con 0x7f57d80c3190 2021-02-10T11:17:52.968 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:52.972 7f57dd036700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57d80b3540 con 0x7f57d80b2370 2021-02-10T11:17:52.968 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:52.972 7f57d6d9d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f57d80c36a0 0x7f57d80c2b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:17:52.968 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:52.972 7f57d6d9d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f57d80c36a0 0x7f57d80c2b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:53230/0 (socket says 172.21.2.102:53230) 2021-02-10T11:17:52.968 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:52.973 7f57d6d9d700 1 -- 172.21.2.102:0/2597692662 learned_addr learned my addr 172.21.2.102:0/2597692662 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:17:52.968 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:52.973 7f57d659c700 1 --2- 172.21.2.102:0/2597692662 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f57d80c3190 0x7f57d80b1d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:17:52.968 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:52.973 7f57d759e700 1 --2- 172.21.2.102:0/2597692662 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f57d80b2370 0x7f57d80b2d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:17:52.968 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:52.973 7f57d6d9d700 1 -- 172.21.2.102:0/2597692662 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f57d80b2370 msgr2=0x7f57d80b2d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:17:52.968 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:52.973 7f57d6d9d700 1 --2- 172.21.2.102:0/2597692662 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f57d80b2370 0x7f57d80b2d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:52.968 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:52.973 7f57d6d9d700 1 -- 172.21.2.102:0/2597692662 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f57d80c3190 msgr2=0x7f57d80b1d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:17:52.968 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:52.973 7f57d6d9d700 1 --2- 172.21.2.102:0/2597692662 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f57d80c3190 0x7f57d80b1d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:52.968 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:52.973 7f57d6d9d700 1 -- 172.21.2.102:0/2597692662 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57d80b3760 con 0x7f57d80c36a0 2021-02-10T11:17:52.968 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:52.973 7f57d759e700 1 --2- 172.21.2.102:0/2597692662 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f57d80b2370 0x7f57d80b2d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:17:52.968 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:52.973 7f57d6d9d700 1 --2- 172.21.2.102:0/2597692662 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f57d80c36a0 0x7f57d80c2b70 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 rx=0x7f57c0006f00 tx=0x7f57c000b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:17:52.969 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:52.973 7f57cf7fe700 1 -- 172.21.2.102:0/2597692662 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f57c000bd30 con 0x7f57d80c36a0 2021-02-10T11:17:52.969 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:52.973 7f57cf7fe700 1 -- 172.21.2.102:0/2597692662 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57c0000c90 con 0x7f57d80c36a0 2021-02-10T11:17:52.969 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:52.973 7f57cf7fe700 1 -- 172.21.2.102:0/2597692662 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f57c00022a0 con 0x7f57d80c36a0 2021-02-10T11:17:52.969 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:52.973 7f57dd036700 1 -- 172.21.2.102:0/2597692662 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57d80b3760 con 0x7f57d80c36a0 2021-02-10T11:17:52.969 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:52.974 7f57dd036700 1 -- 172.21.2.102:0/2597692662 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57d80b3760 con 0x7f57d80c36a0 2021-02-10T11:17:52.969 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:52.974 7f57cf7fe700 1 -- 172.21.2.102:0/2597692662 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f57c0005310 con 0x7f57d80c36a0 2021-02-10T11:17:52.969 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:52.974 7f57cf7fe700 1 --2- 172.21.2.102:0/2597692662 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f57bc0186c0 0x7f57bc01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:17:52.969 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:52.974 7f57cf7fe700 1 -- 172.21.2.102:0/2597692662 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f57c00323d0 con 0x7f57d80c36a0 2021-02-10T11:17:52.969 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:52.974 7f57d659c700 1 --2- 172.21.2.102:0/2597692662 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f57bc0186c0 0x7f57bc01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:17:52.969 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:52.975 7f57d659c700 1 --2- 172.21.2.102:0/2597692662 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f57bc0186c0 0x7f57bc01abc0 crc :-1 s=READY pgs=337 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:17:52.998 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:53.005 7f57dd036700 1 -- 172.21.2.102:0/2597692662 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57d80edd10 con 0x7f57d80c36a0 2021-02-10T11:17:53.000 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:53.008 7f57cf7fe700 1 -- 172.21.2.102:0/2597692662 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f57c0006b90 con 0x7f57d80c36a0 2021-02-10T11:17:53.128 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:53.137 7f57dd036700 1 -- 172.21.2.102:0/2597692662 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344446", "target": ["mgr", ""]}) v1 -- 0x7f57d80edd10 con 0x7f57bc0186c0 2021-02-10T11:17:53.132 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:53.140 7f57cf7fe700 1 -- 172.21.2.102:0/2597692662 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f57c80009b0 con 0x7f57bc0186c0 2021-02-10T11:17:53.135 INFO:teuthology.orchestra.run.gibba002.stdout:/volumes/_nogroup/subvolume_0000000000344446/0e56616c-44b5-4805-ae82-2b8984c552d1 2021-02-10T11:17:53.137 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:53.145 7f57dd036700 1 -- 172.21.2.102:0/2597692662 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f57bc0186c0 msgr2=0x7f57bc01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:17:53.137 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:53.145 7f57dd036700 1 --2- 172.21.2.102:0/2597692662 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f57bc0186c0 0x7f57bc01abc0 crc :-1 s=READY pgs=337 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:53.137 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:53.145 7f57dd036700 1 -- 172.21.2.102:0/2597692662 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f57d80c36a0 msgr2=0x7f57d80c2b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:17:53.137 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:53.145 7f57dd036700 1 --2- 172.21.2.102:0/2597692662 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f57d80c36a0 0x7f57d80c2b70 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 rx=0x7f57c0006f00 tx=0x7f57c000b940).stop 2021-02-10T11:17:53.137 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:53.145 7f57dd036700 1 -- 172.21.2.102:0/2597692662 shutdown_connections 2021-02-10T11:17:53.137 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:53.145 7f57dd036700 1 --2- 172.21.2.102:0/2597692662 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f57d80b2370 0x7f57d80b2d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:53.137 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:53.145 7f57dd036700 1 --2- 172.21.2.102:0/2597692662 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f57bc0186c0 0x7f57bc01abc0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:53.138 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:53.145 7f57dd036700 1 --2- 172.21.2.102:0/2597692662 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f57d80c3190 0x7f57d80b1d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:53.138 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:53.145 7f57dd036700 1 --2- 172.21.2.102:0/2597692662 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f57d80c36a0 0x7f57d80c2b70 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:17:53.138 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:53.145 7f57dd036700 1 -- 172.21.2.102:0/2597692662 >> 172.21.2.102:0/2597692662 conn(0x7f57d809f630 msgr2=0x7f57d80a0130 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:17:53.138 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:53.145 7f57dd036700 1 -- 172.21.2.102:0/2597692662 shutdown_connections 2021-02-10T11:17:53.138 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:17:53.145 7f57dd036700 1 -- 172.21.2.102:0/2597692662 wait complete. 2021-02-10T11:17:53.145 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000344446 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000344446/0e56616c-44b5-4805-ae82-2b8984c552d1 2021-02-10T11:17:53.145 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:17:53.145 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:17:53.146 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:17:53.481 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:17:53.482 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:17:53.482 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344446/0e56616c-44b5-4805-ae82-2b8984c552d1/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:17:53.899 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:17:53.899 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:17:53.899 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0255361 s, 41.1 MB/s 2021-02-10T11:17:53.900 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:17:53.900 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:17:53.900 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:17:54.238 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:17:54.238 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:17:54.239 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344446/0e56616c-44b5-4805-ae82-2b8984c552d1/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:17:54.630 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:17:54.631 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:17:54.631 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0244901 s, 42.8 MB/s 2021-02-10T11:17:54.635 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:17:54.635 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:17:54.635 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:17:54.975 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:17:54.976 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:17:54.976 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344446/0e56616c-44b5-4805-ae82-2b8984c552d1/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:17:55.357 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:17:55.357 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:17:55.357 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.024469 s, 42.9 MB/s 2021-02-10T11:17:55.358 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:17:55.358 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:17:55.358 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:17:55.703 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:17:55.704 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:17:55.704 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344446/0e56616c-44b5-4805-ae82-2b8984c552d1/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:17:56.079 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:17:56.080 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:17:56.080 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0232877 s, 45.0 MB/s 2021-02-10T11:17:56.081 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:17:56.081 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:17:56.081 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:17:56.417 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:17:56.417 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:17:56.418 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344446/0e56616c-44b5-4805-ae82-2b8984c552d1/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:17:56.794 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:17:56.794 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:17:56.794 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0224938 s, 46.6 MB/s 2021-02-10T11:17:56.795 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:17:56.796 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:17:56.796 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:17:57.130 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:17:57.131 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:17:57.131 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344446/0e56616c-44b5-4805-ae82-2b8984c552d1/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:17:57.506 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:17:57.506 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:17:57.506 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0232989 s, 45.0 MB/s 2021-02-10T11:17:57.507 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:17:57.507 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:17:57.507 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:17:57.844 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:17:57.845 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:17:57.845 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344446/0e56616c-44b5-4805-ae82-2b8984c552d1/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:17:58.221 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:17:58.221 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:17:58.221 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0243334 s, 43.1 MB/s 2021-02-10T11:17:58.222 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:17:58.222 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:17:58.222 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:17:58.558 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:17:58.559 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:17:58.559 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344446/0e56616c-44b5-4805-ae82-2b8984c552d1/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:17:58.928 DEBUG:teuthology.orchestra.run.gibba002:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:17:58.931 DEBUG:teuthology.orchestra.run.gibba013:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:17:58.932 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:17:58.932 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:17:58.932 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0237809 s, 44.1 MB/s 2021-02-10T11:17:58.933 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:17:58.933 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:17:58.934 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:17:59.266 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:17:59.267 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:17:59.267 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344446/0e56616c-44b5-4805-ae82-2b8984c552d1/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:17:59.642 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:17:59.642 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:17:59.642 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0236649 s, 44.3 MB/s 2021-02-10T11:17:59.643 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:17:59.643 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:17:59.643 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:17:59.978 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:17:59.979 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:17:59.979 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344446/0e56616c-44b5-4805-ae82-2b8984c552d1/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:18:00.357 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:18:00.357 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:18:00.357 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0251792 s, 41.6 MB/s 2021-02-10T11:18:00.358 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000344447 2021-02-10T11:18:00.793 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:00.783 7f8f845e3700 1 Processor -- start 2021-02-10T11:18:00.794 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:00.783 7f8f845e3700 1 -- start start 2021-02-10T11:18:00.794 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:00.784 7f8f845e3700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f8f7c1b0470 0x7f8f7c1b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:00.794 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:00.784 7f8f845e3700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f7c0f0b50 con 0x7f8f7c0f20e0 2021-02-10T11:18:00.794 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:00.784 7f8f845e3700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f7c0f24e0 con 0x7f8f7c0ef610 2021-02-10T11:18:00.794 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:00.784 7f8f845e3700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f7c0e5ea0 con 0x7f8f7c1b0470 2021-02-10T11:18:00.794 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:00.784 7f8f82c00700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f8f7c0f20e0 0x7f8f7c0f0200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:45462/0 (socket says 172.21.2.102:45462) 2021-02-10T11:18:00.794 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:00.784 7f8f82c00700 1 -- 172.21.2.102:0/3674734867 learned_addr learned my addr 172.21.2.102:0/3674734867 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:18:00.795 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:00.784 7f8f813fd700 1 -- 172.21.2.102:0/3674734867 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 868159729 0 0) 0x7f8f74001500 con 0x7f8f7c0f20e0 2021-02-10T11:18:00.795 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:00.784 7f8f813fd700 1 -- 172.21.2.102:0/3674734867 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f68002fe0 con 0x7f8f7c0f20e0 2021-02-10T11:18:00.795 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:00.785 7f8f823ff700 1 --2- 172.21.2.102:0/3674734867 >> v2:172.21.2.113:3300/0 conn(0x7f8f7c1b0470 0x7f8f7c1b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:18:00.795 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:00.785 7f8f813fd700 1 -- 172.21.2.102:0/3674734867 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 269619815 0 0) 0x7f8f74001460 con 0x7f8f7c0f20e0 2021-02-10T11:18:00.795 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:00.785 7f8f813fd700 1 -- 172.21.2.102:0/3674734867 >> v2:172.21.2.113:3300/0 conn(0x7f8f7c1b0470 msgr2=0x7f8f7c1b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:18:00.795 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:00.785 7f8f813fd700 1 --2- 172.21.2.102:0/3674734867 >> v2:172.21.2.113:3300/0 conn(0x7f8f7c1b0470 0x7f8f7c1b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:00.795 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:00.785 7f8f813fd700 1 -- 172.21.2.102:0/3674734867 >> v1:172.21.2.113:6789/0 conn(0x7f8f7c0ef610 legacy=0x7f8f7c0efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:18:00.795 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:00.785 7f8f813fd700 1 -- 172.21.2.102:0/3674734867 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f7c0e52e0 con 0x7f8f7c0f20e0 2021-02-10T11:18:00.795 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:00.785 7f8f813fd700 1 -- 172.21.2.102:0/3674734867 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7f8f74001980 con 0x7f8f7c0f20e0 2021-02-10T11:18:00.795 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:00.785 7f8f813fd700 1 -- 172.21.2.102:0/3674734867 >> v1:172.21.2.102:6789/0 conn(0x7f8f7c0f20e0 legacy=0x7f8f7c0f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:18:00.795 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:00.785 7f8f813fd700 1 --2- 172.21.2.102:0/3674734867 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8f68003a60 0x7f8f68005ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:00.796 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:00.785 7f8f813fd700 1 --2- 172.21.2.102:0/3674734867 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8f68006790 0x7f8f68008c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:00.796 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:00.785 7f8f813fd700 1 --2- 172.21.2.102:0/3674734867 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8f68009590 0x7f8f6800bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:00.796 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:00.785 7f8f813fd700 1 -- 172.21.2.102:0/3674734867 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f6800c300 con 0x7f8f68009590 2021-02-10T11:18:00.796 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:00.785 7f8f813fd700 1 -- 172.21.2.102:0/3674734867 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f6800c520 con 0x7f8f68006790 2021-02-10T11:18:00.796 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:00.785 7f8f813fd700 1 -- 172.21.2.102:0/3674734867 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f68050750 con 0x7f8f68003a60 2021-02-10T11:18:00.796 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:00.785 7f8f82c00700 1 --2- 172.21.2.102:0/3674734867 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8f68009590 0x7f8f6800bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:18:00.796 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:00.785 7f8f81bfe700 1 --2- 172.21.2.102:0/3674734867 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8f68006790 0x7f8f68008c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:18:00.796 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:00.785 7f8f823ff700 1 --2- 172.21.2.102:0/3674734867 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8f68003a60 0x7f8f68005ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:18:00.796 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:00.785 7f8f82c00700 1 -- 172.21.2.102:0/3674734867 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8f68003a60 msgr2=0x7f8f68005ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:18:00.796 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:00.785 7f8f82c00700 1 --2- 172.21.2.102:0/3674734867 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8f68003a60 0x7f8f68005ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:00.796 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:00.785 7f8f82c00700 1 -- 172.21.2.102:0/3674734867 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8f68006790 msgr2=0x7f8f68008c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:18:00.797 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:00.785 7f8f82c00700 1 --2- 172.21.2.102:0/3674734867 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8f68006790 0x7f8f68008c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:00.797 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:00.785 7f8f82c00700 1 -- 172.21.2.102:0/3674734867 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f68050980 con 0x7f8f68009590 2021-02-10T11:18:00.797 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:00.785 7f8f81bfe700 1 --2- 172.21.2.102:0/3674734867 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8f68006790 0x7f8f68008c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:18:00.797 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:00.785 7f8f82c00700 1 -- 172.21.2.102:0/3674734867 reap_dead start 2021-02-10T11:18:00.797 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:00.786 7f8f82c00700 1 --2- 172.21.2.102:0/3674734867 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8f68009590 0x7f8f6800bab0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 rx=0x7f8f74001160 tx=0x7f8f7400a060).ready entity=mon.0 client_cookie=be2391a2a9d15258 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:18:00.797 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:00.786 7f8f813fd700 1 -- 172.21.2.102:0/3674734867 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8f740070c0 con 0x7f8f68009590 2021-02-10T11:18:00.797 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:00.786 7f8f813fd700 1 -- 172.21.2.102:0/3674734867 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f740073c0 con 0x7f8f68009590 2021-02-10T11:18:00.797 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:00.786 7f8f813fd700 1 -- 172.21.2.102:0/3674734867 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8f74004910 con 0x7f8f68009590 2021-02-10T11:18:00.797 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:00.786 7f8f845e3700 1 -- 172.21.2.102:0/3674734867 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8f68009590 msgr2=0x7f8f6800bab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:18:00.797 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:00.786 7f8f845e3700 1 --2- 172.21.2.102:0/3674734867 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8f68009590 0x7f8f6800bab0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 rx=0x7f8f74001160 tx=0x7f8f7400a060).stop 2021-02-10T11:18:00.798 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:00.787 7f8f845e3700 1 -- 172.21.2.102:0/3674734867 shutdown_connections 2021-02-10T11:18:00.798 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:00.787 7f8f845e3700 1 --2- 172.21.2.102:0/3674734867 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8f68009590 0x7f8f6800bab0 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:00.798 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:00.787 7f8f845e3700 1 -- 172.21.2.102:0/3674734867 >> 172.21.2.102:0/3674734867 conn(0x7f8f7c0ce3e0 msgr2=0x7f8f7c0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:18:00.798 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:00.787 7f8f845e3700 1 -- 172.21.2.102:0/3674734867 shutdown_connections 2021-02-10T11:18:00.798 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:00.787 7f8f845e3700 1 -- 172.21.2.102:0/3674734867 wait complete. 2021-02-10T11:18:00.798 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:00.801 7f8f845e3700 1 Processor -- start 2021-02-10T11:18:00.798 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:00.801 7f8f845e3700 1 -- start start 2021-02-10T11:18:00.801 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:00.802 7f8f845e3700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8f7c0924b0 0x7f8f7c1a3590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:00.801 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:00.802 7f8f845e3700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8f7c1a3bb0 0x7f8f7c0bf920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:00.801 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:00.802 7f8f845e3700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8f7c0bff40 0x7f8f7c0c0340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:00.801 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:00.802 7f8f845e3700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f7c09a0a0 con 0x7f8f7c0924b0 2021-02-10T11:18:00.801 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:00.802 7f8f845e3700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f7c0c0960 con 0x7f8f7c0bff40 2021-02-10T11:18:00.801 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:00.802 7f8f845e3700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f7c098420 con 0x7f8f7c1a3bb0 2021-02-10T11:18:00.802 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:00.802 7f8f823ff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8f7c0924b0 0x7f8f7c1a3590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:18:00.802 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:00.802 7f8f823ff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8f7c0924b0 0x7f8f7c1a3590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:53248/0 (socket says 172.21.2.102:53248) 2021-02-10T11:18:00.802 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:00.802 7f8f823ff700 1 -- 172.21.2.102:0/2348474616 learned_addr learned my addr 172.21.2.102:0/2348474616 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:18:00.802 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:00.802 7f8f82c00700 1 --2- 172.21.2.102:0/2348474616 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8f7c0bff40 0x7f8f7c0c0340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:18:00.802 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:00.802 7f8f81bfe700 1 --2- 172.21.2.102:0/2348474616 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8f7c1a3bb0 0x7f8f7c0bf920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:18:00.802 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:00.802 7f8f823ff700 1 -- 172.21.2.102:0/2348474616 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8f7c1a3bb0 msgr2=0x7f8f7c0bf920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:18:00.802 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:00.802 7f8f823ff700 1 --2- 172.21.2.102:0/2348474616 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8f7c1a3bb0 0x7f8f7c0bf920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:00.802 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:00.802 7f8f823ff700 1 -- 172.21.2.102:0/2348474616 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8f7c0bff40 msgr2=0x7f8f7c0c0340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:18:00.802 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:00.802 7f8f823ff700 1 --2- 172.21.2.102:0/2348474616 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8f7c0bff40 0x7f8f7c0c0340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:00.802 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:00.803 7f8f823ff700 1 -- 172.21.2.102:0/2348474616 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f7c098640 con 0x7f8f7c0924b0 2021-02-10T11:18:00.802 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:00.803 7f8f81bfe700 1 --2- 172.21.2.102:0/2348474616 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8f7c1a3bb0 0x7f8f7c0bf920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:18:00.803 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:00.803 7f8f82c00700 1 --2- 172.21.2.102:0/2348474616 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8f7c0bff40 0x7f8f7c0c0340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:18:00.803 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:00.803 7f8f823ff700 1 --2- 172.21.2.102:0/2348474616 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8f7c0924b0 0x7f8f7c1a3590 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 rx=0x7f8f78006f60 tx=0x7f8f7800b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:18:00.803 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:00.803 7f8f72ffd700 1 -- 172.21.2.102:0/2348474616 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8f780009b0 con 0x7f8f7c0924b0 2021-02-10T11:18:00.803 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:00.803 7f8f72ffd700 1 -- 172.21.2.102:0/2348474616 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f7800bdf0 con 0x7f8f7c0924b0 2021-02-10T11:18:00.803 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:00.803 7f8f72ffd700 1 -- 172.21.2.102:0/2348474616 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8f78002030 con 0x7f8f7c0924b0 2021-02-10T11:18:00.803 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:00.803 7f8f845e3700 1 -- 172.21.2.102:0/2348474616 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f7c098640 con 0x7f8f7c0924b0 2021-02-10T11:18:00.803 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:00.803 7f8f845e3700 1 -- 172.21.2.102:0/2348474616 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f7c098bd0 con 0x7f8f7c0924b0 2021-02-10T11:18:00.803 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:00.804 7f8f72ffd700 1 -- 172.21.2.102:0/2348474616 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f8f78006b00 con 0x7f8f7c0924b0 2021-02-10T11:18:00.803 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:00.804 7f8f72ffd700 1 --2- 172.21.2.102:0/2348474616 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8f640186c0 0x7f8f6401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:00.803 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:00.804 7f8f72ffd700 1 -- 172.21.2.102:0/2348474616 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f8f78032590 con 0x7f8f7c0924b0 2021-02-10T11:18:00.803 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:00.804 7f8f81bfe700 1 --2- 172.21.2.102:0/2348474616 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8f640186c0 0x7f8f6401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:18:00.804 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:00.804 7f8f81bfe700 1 --2- 172.21.2.102:0/2348474616 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8f640186c0 0x7f8f6401abc0 crc :-1 s=READY pgs=338 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:18:00.825 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:00.833 7f8f845e3700 1 -- 172.21.2.102:0/2348474616 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f7c105b40 con 0x7f8f7c0924b0 2021-02-10T11:18:00.828 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:00.836 7f8f72ffd700 1 -- 172.21.2.102:0/2348474616 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f8f78025d30 con 0x7f8f7c0924b0 2021-02-10T11:18:00.956 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:00.963 7f8f845e3700 1 -- 172.21.2.102:0/2348474616 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344447", "target": ["mgr", ""]}) v1 -- 0x7f8f7c098bd0 con 0x7f8f640186c0 2021-02-10T11:18:00.980 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:00.989 7f8f72ffd700 1 -- 172.21.2.102:0/2348474616 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f8f6c006750 con 0x7f8f640186c0 2021-02-10T11:18:00.988 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:00.996 7f8f845e3700 1 -- 172.21.2.102:0/2348474616 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8f640186c0 msgr2=0x7f8f6401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:18:00.988 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:00.996 7f8f845e3700 1 --2- 172.21.2.102:0/2348474616 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8f640186c0 0x7f8f6401abc0 crc :-1 s=READY pgs=338 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:00.988 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:00.996 7f8f845e3700 1 -- 172.21.2.102:0/2348474616 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8f7c0924b0 msgr2=0x7f8f7c1a3590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:18:00.988 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:00.996 7f8f845e3700 1 --2- 172.21.2.102:0/2348474616 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8f7c0924b0 0x7f8f7c1a3590 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 rx=0x7f8f78006f60 tx=0x7f8f7800b970).stop 2021-02-10T11:18:00.988 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:00.996 7f8f845e3700 1 -- 172.21.2.102:0/2348474616 shutdown_connections 2021-02-10T11:18:00.989 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:00.996 7f8f845e3700 1 --2- 172.21.2.102:0/2348474616 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8f640186c0 0x7f8f6401abc0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:00.989 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:00.996 7f8f845e3700 1 --2- 172.21.2.102:0/2348474616 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8f7c0bff40 0x7f8f7c0c0340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:00.989 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:00.996 7f8f845e3700 1 --2- 172.21.2.102:0/2348474616 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8f7c1a3bb0 0x7f8f7c0bf920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:00.989 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:00.996 7f8f845e3700 1 --2- 172.21.2.102:0/2348474616 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8f7c0924b0 0x7f8f7c1a3590 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:00.989 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:00.996 7f8f845e3700 1 -- 172.21.2.102:0/2348474616 >> 172.21.2.102:0/2348474616 conn(0x7f8f7c0b50c0 msgr2=0x7f8f7c092d90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:18:00.989 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:00.996 7f8f845e3700 1 -- 172.21.2.102:0/2348474616 shutdown_connections 2021-02-10T11:18:00.989 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:00.996 7f8f845e3700 1 -- 172.21.2.102:0/2348474616 wait complete. 2021-02-10T11:18:00.996 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000344447 2021-02-10T11:18:01.430 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:01.421 7f214eb75700 1 Processor -- start 2021-02-10T11:18:01.431 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:01.421 7f214eb75700 1 -- start start 2021-02-10T11:18:01.431 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:01.422 7f214eb75700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f21481b0470 0x7f21481b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:01.431 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:01.422 7f214eb75700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f21480ef610 0x7f21480efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:01.431 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:01.422 7f214eb75700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f21480f0030 0x7f21480f0430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:01.431 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:01.422 7f214eb75700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21480e5ea0 con 0x7f21480ef610 2021-02-10T11:18:01.431 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:01.422 7f214eb75700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21480f0b50 con 0x7f21481b0470 2021-02-10T11:18:01.431 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:01.422 7f214eb75700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21480e52e0 con 0x7f21480f0030 2021-02-10T11:18:01.431 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:01.422 7f2147fff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f21480ef610 0x7f21480efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:18:01.431 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:01.422 7f2147fff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f21480ef610 0x7f21480efa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:53258/0 (socket says 172.21.2.102:53258) 2021-02-10T11:18:01.431 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:01.422 7f2147fff700 1 -- 172.21.2.102:0/661408908 learned_addr learned my addr 172.21.2.102:0/661408908 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:18:01.431 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:01.422 7f214c991700 1 --2- 172.21.2.102:0/661408908 >> v2:172.21.2.113:3300/0 conn(0x7f21481b0470 0x7f21481b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:18:01.432 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:01.422 7f214d192700 1 --2- 172.21.2.102:0/661408908 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f21480f0030 0x7f21480f0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:18:01.432 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:01.422 7f2147fff700 1 -- 172.21.2.102:0/661408908 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f21480f0030 msgr2=0x7f21480f0430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:18:01.432 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:01.422 7f2147fff700 1 --2- 172.21.2.102:0/661408908 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f21480f0030 0x7f21480f0430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:01.432 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:01.422 7f2147fff700 1 -- 172.21.2.102:0/661408908 >> v2:172.21.2.113:3300/0 conn(0x7f21481b0470 msgr2=0x7f21481b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:18:01.432 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:01.422 7f2147fff700 1 --2- 172.21.2.102:0/661408908 >> v2:172.21.2.113:3300/0 conn(0x7f21481b0470 0x7f21481b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:01.432 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:01.422 7f2147fff700 1 -- 172.21.2.102:0/661408908 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21480f10d0 con 0x7f21480ef610 2021-02-10T11:18:01.432 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:01.423 7f2147fff700 1 --2- 172.21.2.102:0/661408908 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f21480ef610 0x7f21480efa10 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 rx=0x7f2138006700 tx=0x7f2138009b80).ready entity=mon.0 client_cookie=3f6e9235eac8a9f9 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:18:01.432 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:01.423 7f21477fe700 1 -- 172.21.2.102:0/661408908 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f213800c0c0 con 0x7f21480ef610 2021-02-10T11:18:01.432 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:01.423 7f21477fe700 1 -- 172.21.2.102:0/661408908 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f213800c400 con 0x7f21480ef610 2021-02-10T11:18:01.432 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:01.423 7f21477fe700 1 -- 172.21.2.102:0/661408908 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f213800c920 con 0x7f21480ef610 2021-02-10T11:18:01.432 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:01.423 7f214eb75700 1 -- 172.21.2.102:0/661408908 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f21480ef610 msgr2=0x7f21480efa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:18:01.432 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:01.423 7f214eb75700 1 --2- 172.21.2.102:0/661408908 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f21480ef610 0x7f21480efa10 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 rx=0x7f2138006700 tx=0x7f2138009b80).stop 2021-02-10T11:18:01.432 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:01.424 7f214eb75700 1 -- 172.21.2.102:0/661408908 shutdown_connections 2021-02-10T11:18:01.432 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:01.424 7f214eb75700 1 --2- 172.21.2.102:0/661408908 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f21480f0030 0x7f21480f0430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:01.433 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:01.424 7f214eb75700 1 --2- 172.21.2.102:0/661408908 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f21480ef610 0x7f21480efa10 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:01.433 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:01.424 7f214eb75700 1 --2- 172.21.2.102:0/661408908 >> v2:172.21.2.113:3300/0 conn(0x7f21481b0470 0x7f21481b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:01.433 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:01.424 7f214eb75700 1 -- 172.21.2.102:0/661408908 >> 172.21.2.102:0/661408908 conn(0x7f21480ce3e0 msgr2=0x7f21480e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:18:01.433 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:01.424 7f214eb75700 1 -- 172.21.2.102:0/661408908 shutdown_connections 2021-02-10T11:18:01.433 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:01.424 7f214eb75700 1 -- 172.21.2.102:0/661408908 wait complete. 2021-02-10T11:18:01.434 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:01.439 7f214eb75700 1 Processor -- start 2021-02-10T11:18:01.435 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:01.439 7f214eb75700 1 -- start start 2021-02-10T11:18:01.435 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:01.440 7f214eb75700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f21480a6e70 0x7f21480965f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:01.435 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:01.440 7f214eb75700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2148096c10 0x7f2148097010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:01.435 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:01.440 7f214eb75700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f21480c0ef0 0x7f21480c12f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:01.435 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:01.440 7f214eb75700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21480a7270 con 0x7f21480a6e70 2021-02-10T11:18:01.435 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:01.440 7f214eb75700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2148190c50 con 0x7f21480c0ef0 2021-02-10T11:18:01.435 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:01.440 7f214eb75700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21480c1910 con 0x7f2148096c10 2021-02-10T11:18:01.435 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:01.440 7f214c991700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f21480a6e70 0x7f21480965f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:18:01.435 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:01.440 7f214c991700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f21480a6e70 0x7f21480965f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:53262/0 (socket says 172.21.2.102:53262) 2021-02-10T11:18:01.435 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:01.440 7f214c991700 1 -- 172.21.2.102:0/2741047634 learned_addr learned my addr 172.21.2.102:0/2741047634 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:18:01.435 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:01.440 7f2147fff700 1 --2- 172.21.2.102:0/2741047634 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2148096c10 0x7f2148097010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:18:01.435 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:01.440 7f214d192700 1 --2- 172.21.2.102:0/2741047634 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f21480c0ef0 0x7f21480c12f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:18:01.435 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:01.440 7f214c991700 1 -- 172.21.2.102:0/2741047634 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2148096c10 msgr2=0x7f2148097010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:18:01.435 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:01.440 7f214c991700 1 --2- 172.21.2.102:0/2741047634 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2148096c10 0x7f2148097010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:01.436 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:01.440 7f214c991700 1 -- 172.21.2.102:0/2741047634 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f21480c0ef0 msgr2=0x7f21480c12f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:18:01.436 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:01.440 7f214c991700 1 --2- 172.21.2.102:0/2741047634 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f21480c0ef0 0x7f21480c12f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:01.436 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:01.440 7f214c991700 1 -- 172.21.2.102:0/2741047634 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21480c1b30 con 0x7f21480a6e70 2021-02-10T11:18:01.436 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:01.440 7f214c991700 1 --2- 172.21.2.102:0/2741047634 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f21480a6e70 0x7f21480965f0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 rx=0x7f2130006f60 tx=0x7f213000b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:18:01.436 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:01.441 7f21457fa700 1 -- 172.21.2.102:0/2741047634 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f21300009b0 con 0x7f21480a6e70 2021-02-10T11:18:01.436 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:01.441 7f21457fa700 1 -- 172.21.2.102:0/2741047634 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f213000bd60 con 0x7f21480a6e70 2021-02-10T11:18:01.436 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:01.441 7f21457fa700 1 -- 172.21.2.102:0/2741047634 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2130002030 con 0x7f21480a6e70 2021-02-10T11:18:01.436 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:01.441 7f214eb75700 1 -- 172.21.2.102:0/2741047634 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21480c1b30 con 0x7f21480a6e70 2021-02-10T11:18:01.436 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:01.441 7f214eb75700 1 -- 172.21.2.102:0/2741047634 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21480c20c0 con 0x7f21480a6e70 2021-02-10T11:18:01.436 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:01.441 7f21457fa700 1 -- 172.21.2.102:0/2741047634 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f2130006b00 con 0x7f21480a6e70 2021-02-10T11:18:01.436 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:01.441 7f21457fa700 1 --2- 172.21.2.102:0/2741047634 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f212c0186c0 0x7f212c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:01.436 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:01.441 7f21457fa700 1 -- 172.21.2.102:0/2741047634 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f2130032400 con 0x7f21480a6e70 2021-02-10T11:18:01.437 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:01.441 7f2147fff700 1 --2- 172.21.2.102:0/2741047634 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f212c0186c0 0x7f212c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:18:01.437 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:01.442 7f2147fff700 1 --2- 172.21.2.102:0/2741047634 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f212c0186c0 0x7f212c01abc0 crc :-1 s=READY pgs=339 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:18:01.463 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:01.471 7f214eb75700 1 -- 172.21.2.102:0/2741047634 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2148105b30 con 0x7f21480a6e70 2021-02-10T11:18:01.465 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:01.474 7f21457fa700 1 -- 172.21.2.102:0/2741047634 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f21300009b0 con 0x7f21480a6e70 2021-02-10T11:18:01.596 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:01.605 7f214eb75700 1 -- 172.21.2.102:0/2741047634 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344447", "target": ["mgr", ""]}) v1 -- 0x7f21480c20c0 con 0x7f212c0186c0 2021-02-10T11:18:01.599 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:01.607 7f21457fa700 1 -- 172.21.2.102:0/2741047634 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f2138009380 con 0x7f212c0186c0 2021-02-10T11:18:01.599 INFO:teuthology.orchestra.run.gibba002.stdout:/volumes/_nogroup/subvolume_0000000000344447/6c1ede30-ed64-4e64-8b3a-9e71c20e2853 2021-02-10T11:18:01.601 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:01.609 7f214eb75700 1 -- 172.21.2.102:0/2741047634 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f212c0186c0 msgr2=0x7f212c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:18:01.601 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:01.609 7f214eb75700 1 --2- 172.21.2.102:0/2741047634 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f212c0186c0 0x7f212c01abc0 crc :-1 s=READY pgs=339 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:01.601 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:01.609 7f214eb75700 1 -- 172.21.2.102:0/2741047634 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f21480a6e70 msgr2=0x7f21480965f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:18:01.601 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:01.609 7f214eb75700 1 --2- 172.21.2.102:0/2741047634 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f21480a6e70 0x7f21480965f0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 rx=0x7f2130006f60 tx=0x7f213000b970).stop 2021-02-10T11:18:01.601 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:01.609 7f214eb75700 1 -- 172.21.2.102:0/2741047634 shutdown_connections 2021-02-10T11:18:01.602 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:01.609 7f214eb75700 1 --2- 172.21.2.102:0/2741047634 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f212c0186c0 0x7f212c01abc0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:01.602 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:01.609 7f214eb75700 1 --2- 172.21.2.102:0/2741047634 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f21480c0ef0 0x7f21480c12f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:01.602 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:01.609 7f214eb75700 1 --2- 172.21.2.102:0/2741047634 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2148096c10 0x7f2148097010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:01.602 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:01.609 7f214eb75700 1 --2- 172.21.2.102:0/2741047634 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f21480a6e70 0x7f21480965f0 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:01.602 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:01.609 7f214eb75700 1 -- 172.21.2.102:0/2741047634 >> 172.21.2.102:0/2741047634 conn(0x7f21480c40e0 msgr2=0x7f21480957c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:18:01.602 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:01.609 7f214eb75700 1 -- 172.21.2.102:0/2741047634 shutdown_connections 2021-02-10T11:18:01.602 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:01.610 7f214eb75700 1 -- 172.21.2.102:0/2741047634 wait complete. 2021-02-10T11:18:01.609 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000344447 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000344447/6c1ede30-ed64-4e64-8b3a-9e71c20e2853 2021-02-10T11:18:01.609 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:18:01.609 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:18:01.610 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:18:01.947 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:18:01.947 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:18:01.947 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344447/6c1ede30-ed64-4e64-8b3a-9e71c20e2853/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:18:02.331 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:18:02.331 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:18:02.331 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0239701 s, 43.7 MB/s 2021-02-10T11:18:02.333 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:18:02.333 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:18:02.333 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:18:02.669 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:18:02.669 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:18:02.669 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344447/6c1ede30-ed64-4e64-8b3a-9e71c20e2853/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:18:03.046 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:18:03.047 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:18:03.047 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0254706 s, 41.2 MB/s 2021-02-10T11:18:03.047 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:18:03.048 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:18:03.048 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:18:03.384 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:18:03.384 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:18:03.385 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344447/6c1ede30-ed64-4e64-8b3a-9e71c20e2853/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:18:03.757 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:18:03.757 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:18:03.757 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.023447 s, 44.7 MB/s 2021-02-10T11:18:03.758 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:18:03.758 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:18:03.759 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:18:04.094 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:18:04.094 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:18:04.095 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344447/6c1ede30-ed64-4e64-8b3a-9e71c20e2853/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:18:04.472 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:18:04.472 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:18:04.472 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0255283 s, 41.1 MB/s 2021-02-10T11:18:04.473 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:18:04.473 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:18:04.473 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:18:04.810 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:18:04.811 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:18:04.811 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344447/6c1ede30-ed64-4e64-8b3a-9e71c20e2853/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:18:05.186 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:18:05.186 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:18:05.186 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0226827 s, 46.2 MB/s 2021-02-10T11:18:05.188 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:18:05.188 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:18:05.188 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:18:05.524 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:18:05.525 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:18:05.525 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344447/6c1ede30-ed64-4e64-8b3a-9e71c20e2853/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:18:05.902 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:18:05.902 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:18:05.902 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0240733 s, 43.6 MB/s 2021-02-10T11:18:05.903 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:18:05.903 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:18:05.904 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:18:06.239 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:18:06.239 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:18:06.239 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344447/6c1ede30-ed64-4e64-8b3a-9e71c20e2853/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:18:06.615 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:18:06.615 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:18:06.615 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0244837 s, 42.8 MB/s 2021-02-10T11:18:06.616 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:18:06.616 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:18:06.616 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:18:06.952 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:18:06.952 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:18:06.953 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344447/6c1ede30-ed64-4e64-8b3a-9e71c20e2853/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:18:07.327 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:18:07.328 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:18:07.328 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0231278 s, 45.3 MB/s 2021-02-10T11:18:07.329 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:18:07.329 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:18:07.329 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:18:07.667 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:18:07.667 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:18:07.667 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344447/6c1ede30-ed64-4e64-8b3a-9e71c20e2853/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:18:08.045 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:18:08.045 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:18:08.045 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.025566 s, 41.0 MB/s 2021-02-10T11:18:08.046 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:18:08.046 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:18:08.046 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:18:08.381 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:18:08.382 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:18:08.382 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344447/6c1ede30-ed64-4e64-8b3a-9e71c20e2853/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:18:08.755 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:18:08.755 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:18:08.756 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0254332 s, 41.2 MB/s 2021-02-10T11:18:08.756 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000344448 2021-02-10T11:18:09.188 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:09.179 7f0826ebb700 1 Processor -- start 2021-02-10T11:18:09.188 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:09.179 7f0826ebb700 1 -- start start 2021-02-10T11:18:09.188 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:09.180 7f0826ebb700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f08201b0470 0x7f08201b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:09.188 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:09.180 7f0826ebb700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08200f0b50 con 0x7f08200ef610 2021-02-10T11:18:09.188 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:09.180 7f0826ebb700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08200f24e0 con 0x7f08200f20e0 2021-02-10T11:18:09.188 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:09.180 7f0826ebb700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08200e5ea0 con 0x7f08201b0470 2021-02-10T11:18:09.188 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:09.180 7f0824cd7700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f08201b0470 0x7f08201b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:18:09.189 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:09.180 7f081ffff700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f08200ef610 0x7f08200efa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:45500/0 (socket says 172.21.2.102:45500) 2021-02-10T11:18:09.189 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:09.180 7f0824cd7700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f08201b0470 0x7f08201b0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:53276/0 (socket says 172.21.2.102:53276) 2021-02-10T11:18:09.189 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:09.180 7f0824cd7700 1 -- 172.21.2.102:0/1726461884 learned_addr learned my addr 172.21.2.102:0/1726461884 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:18:09.189 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:09.181 7f0824cd7700 1 -- 172.21.2.102:0/1726461884 >> v1:172.21.2.113:6789/0 conn(0x7f08200f20e0 legacy=0x7f08200f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:18:09.189 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:09.181 7f0824cd7700 1 -- 172.21.2.102:0/1726461884 >> v1:172.21.2.102:6789/0 conn(0x7f08200ef610 legacy=0x7f08200efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:18:09.189 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:09.181 7f0824cd7700 1 -- 172.21.2.102:0/1726461884 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08200e52e0 con 0x7f08201b0470 2021-02-10T11:18:09.189 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:09.181 7f0824cd7700 1 --2- 172.21.2.102:0/1726461884 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f08201b0470 0x7f08201b0870 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 rx=0x7f0808006700 tx=0x7f0808009b80).ready entity=mon.0 client_cookie=68afa007a1a9cc4b server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:18:09.189 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:09.181 7f081f7fe700 1 -- 172.21.2.102:0/1726461884 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f080800c0c0 con 0x7f08201b0470 2021-02-10T11:18:09.189 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:09.181 7f081f7fe700 1 -- 172.21.2.102:0/1726461884 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f080800c440 con 0x7f08201b0470 2021-02-10T11:18:09.189 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:09.181 7f081f7fe700 1 -- 172.21.2.102:0/1726461884 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f080800c920 con 0x7f08201b0470 2021-02-10T11:18:09.189 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:09.181 7f0826ebb700 1 -- 172.21.2.102:0/1726461884 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f08201b0470 msgr2=0x7f08201b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:18:09.189 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:09.181 7f0826ebb700 1 --2- 172.21.2.102:0/1726461884 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f08201b0470 0x7f08201b0870 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 rx=0x7f0808006700 tx=0x7f0808009b80).stop 2021-02-10T11:18:09.190 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:09.182 7f0826ebb700 1 -- 172.21.2.102:0/1726461884 shutdown_connections 2021-02-10T11:18:09.190 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:09.182 7f0826ebb700 1 --2- 172.21.2.102:0/1726461884 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f08201b0470 0x7f08201b0870 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:09.190 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:09.182 7f0826ebb700 1 -- 172.21.2.102:0/1726461884 >> 172.21.2.102:0/1726461884 conn(0x7f08200ce3e0 msgr2=0x7f08200e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:18:09.190 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:09.182 7f0826ebb700 1 -- 172.21.2.102:0/1726461884 shutdown_connections 2021-02-10T11:18:09.190 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:09.182 7f0826ebb700 1 -- 172.21.2.102:0/1726461884 wait complete. 2021-02-10T11:18:09.191 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:09.196 7f0826ebb700 1 Processor -- start 2021-02-10T11:18:09.191 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:09.196 7f0826ebb700 1 -- start start 2021-02-10T11:18:09.191 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:09.197 7f0826ebb700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0820087510 0x7f08200aedc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:09.191 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:09.197 7f0826ebb700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f08201c68f0 0x7f08201b1f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:09.191 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:09.197 7f0826ebb700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f082008c4d0 0x7f08201b18a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:09.191 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:09.197 7f0826ebb700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08200a2d50 con 0x7f082008c4d0 2021-02-10T11:18:09.191 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:09.197 7f0826ebb700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08200a2f70 con 0x7f08201c68f0 2021-02-10T11:18:09.191 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:09.197 7f0826ebb700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08200a3190 con 0x7f0820087510 2021-02-10T11:18:09.193 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:09.197 7f08254d8700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f082008c4d0 0x7f08201b18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:18:09.193 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:09.197 7f08254d8700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f082008c4d0 0x7f08201b18a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:53282/0 (socket says 172.21.2.102:53282) 2021-02-10T11:18:09.193 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:09.197 7f08254d8700 1 -- 172.21.2.102:0/954729136 learned_addr learned my addr 172.21.2.102:0/954729136 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:18:09.194 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:09.197 7f081ffff700 1 --2- 172.21.2.102:0/954729136 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f08201c68f0 0x7f08201b1f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:18:09.194 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:09.197 7f0824cd7700 1 --2- 172.21.2.102:0/954729136 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0820087510 0x7f08200aedc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:18:09.194 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:09.198 7f08254d8700 1 -- 172.21.2.102:0/954729136 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0820087510 msgr2=0x7f08200aedc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:18:09.194 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:09.198 7f08254d8700 1 --2- 172.21.2.102:0/954729136 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0820087510 0x7f08200aedc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:09.194 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:09.198 7f08254d8700 1 -- 172.21.2.102:0/954729136 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f08201c68f0 msgr2=0x7f08201b1f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:18:09.194 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:09.198 7f08254d8700 1 --2- 172.21.2.102:0/954729136 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f08201c68f0 0x7f08201b1f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:09.194 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:09.198 7f08254d8700 1 -- 172.21.2.102:0/954729136 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08200a33b0 con 0x7f082008c4d0 2021-02-10T11:18:09.194 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:09.198 7f0824cd7700 1 --2- 172.21.2.102:0/954729136 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0820087510 0x7f08200aedc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:18:09.194 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:09.198 7f08254d8700 1 --2- 172.21.2.102:0/954729136 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f082008c4d0 0x7f08201b18a0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 rx=0x7f0818006ed0 tx=0x7f081800b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:18:09.194 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:09.198 7f081d7fa700 1 -- 172.21.2.102:0/954729136 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f081800bd50 con 0x7f082008c4d0 2021-02-10T11:18:09.194 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:09.198 7f081d7fa700 1 -- 172.21.2.102:0/954729136 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0818006020 con 0x7f082008c4d0 2021-02-10T11:18:09.194 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:09.198 7f081d7fa700 1 -- 172.21.2.102:0/954729136 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f08180054b0 con 0x7f082008c4d0 2021-02-10T11:18:09.194 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:09.198 7f0826ebb700 1 -- 172.21.2.102:0/954729136 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08200a33b0 con 0x7f082008c4d0 2021-02-10T11:18:09.195 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:09.198 7f0826ebb700 1 -- 172.21.2.102:0/954729136 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08200afca0 con 0x7f082008c4d0 2021-02-10T11:18:09.195 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:09.199 7f081d7fa700 1 -- 172.21.2.102:0/954729136 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f0818006200 con 0x7f082008c4d0 2021-02-10T11:18:09.195 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:09.199 7f081d7fa700 1 --2- 172.21.2.102:0/954729136 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f08040186c0 0x7f080401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:09.195 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:09.199 7f081d7fa700 1 -- 172.21.2.102:0/954729136 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f0818032560 con 0x7f082008c4d0 2021-02-10T11:18:09.195 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:09.199 7f0824cd7700 1 --2- 172.21.2.102:0/954729136 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f08040186c0 0x7f080401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:18:09.195 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:09.199 7f0824cd7700 1 --2- 172.21.2.102:0/954729136 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f08040186c0 0x7f080401abc0 crc :-1 s=READY pgs=345 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:18:09.222 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:09.230 7f0826ebb700 1 -- 172.21.2.102:0/954729136 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f082007e6b0 con 0x7f082008c4d0 2021-02-10T11:18:09.224 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:09.233 7f081d7fa700 1 -- 172.21.2.102:0/954729136 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f0818000bc0 con 0x7f082008c4d0 2021-02-10T11:18:09.350 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:09.358 7f0826ebb700 1 -- 172.21.2.102:0/954729136 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344448", "target": ["mgr", ""]}) v1 -- 0x7f08200a2d50 con 0x7f08040186c0 2021-02-10T11:18:09.378 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:09.387 7f081d7fa700 1 -- 172.21.2.102:0/954729136 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f08080009b0 con 0x7f08040186c0 2021-02-10T11:18:09.383 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:09.391 7f0826ebb700 1 -- 172.21.2.102:0/954729136 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f08040186c0 msgr2=0x7f080401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:18:09.383 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:09.391 7f0826ebb700 1 --2- 172.21.2.102:0/954729136 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f08040186c0 0x7f080401abc0 crc :-1 s=READY pgs=345 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:09.383 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:09.391 7f0826ebb700 1 -- 172.21.2.102:0/954729136 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f082008c4d0 msgr2=0x7f08201b18a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:18:09.383 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:09.391 7f0826ebb700 1 --2- 172.21.2.102:0/954729136 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f082008c4d0 0x7f08201b18a0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 rx=0x7f0818006ed0 tx=0x7f081800b960).stop 2021-02-10T11:18:09.384 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:09.392 7f0826ebb700 1 -- 172.21.2.102:0/954729136 shutdown_connections 2021-02-10T11:18:09.384 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:09.392 7f0826ebb700 1 --2- 172.21.2.102:0/954729136 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f082008c4d0 0x7f08201b18a0 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:09.384 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:09.392 7f0826ebb700 1 --2- 172.21.2.102:0/954729136 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f08040186c0 0x7f080401abc0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:09.384 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:09.392 7f0826ebb700 1 --2- 172.21.2.102:0/954729136 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f08201c68f0 0x7f08201b1f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:09.384 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:09.392 7f0826ebb700 1 --2- 172.21.2.102:0/954729136 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0820087510 0x7f08200aedc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:09.384 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:09.392 7f0826ebb700 1 -- 172.21.2.102:0/954729136 >> 172.21.2.102:0/954729136 conn(0x7f08200886d0 msgr2=0x7f0820081c40 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:18:09.384 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:09.392 7f0826ebb700 1 -- 172.21.2.102:0/954729136 shutdown_connections 2021-02-10T11:18:09.385 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:09.392 7f0826ebb700 1 -- 172.21.2.102:0/954729136 wait complete. 2021-02-10T11:18:09.392 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000344448 2021-02-10T11:18:09.827 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:09.817 7fac790ea700 1 Processor -- start 2021-02-10T11:18:09.827 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:09.817 7fac790ea700 1 -- start start 2021-02-10T11:18:09.827 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:09.818 7fac790ea700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fac740f05f0 0x7fac740f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:09.827 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:09.818 7fac790ea700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac741b0c70 con 0x7fac741b0870 2021-02-10T11:18:09.827 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:09.818 7fac790ea700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac740f0b50 con 0x7fac741b0470 2021-02-10T11:18:09.828 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:09.818 7fac790ea700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac740e5ea0 con 0x7fac740f05f0 2021-02-10T11:18:09.828 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:09.818 7fac7259c700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fac741b0870 0x7fac740efe00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:45510/0 (socket says 172.21.2.102:45510) 2021-02-10T11:18:09.828 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:09.818 7fac7259c700 1 -- 172.21.2.102:0/559885421 learned_addr learned my addr 172.21.2.102:0/559885421 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:18:09.828 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:09.818 7fac71d9b700 1 -- 172.21.2.102:0/559885421 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3510235181 0 0) 0x7fac640010b0 con 0x7fac741b0870 2021-02-10T11:18:09.828 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:09.818 7fac71d9b700 1 -- 172.21.2.102:0/559885421 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac60002fe0 con 0x7fac741b0870 2021-02-10T11:18:09.828 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:09.819 7fac7359e700 1 --2- 172.21.2.102:0/559885421 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fac740f05f0 0x7fac740f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:18:09.828 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:09.819 7fac71d9b700 1 -- 172.21.2.102:0/559885421 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 938337538 0 0) 0x7fac640010b0 con 0x7fac741b0870 2021-02-10T11:18:09.828 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:09.819 7fac71d9b700 1 -- 172.21.2.102:0/559885421 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fac740f05f0 msgr2=0x7fac740f20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:18:09.828 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:09.819 7fac71d9b700 1 --2- 172.21.2.102:0/559885421 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fac740f05f0 0x7fac740f20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:09.828 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:09.819 7fac71d9b700 1 -- 172.21.2.102:0/559885421 >> v1:172.21.2.113:6789/0 conn(0x7fac741b0470 legacy=0x7fac740ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:18:09.828 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:09.819 7fac71d9b700 1 -- 172.21.2.102:0/559885421 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac740e52e0 con 0x7fac741b0870 2021-02-10T11:18:09.828 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:09.819 7fac71d9b700 1 -- 172.21.2.102:0/559885421 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7fac640015b0 con 0x7fac741b0870 2021-02-10T11:18:09.828 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:09.819 7fac71d9b700 1 -- 172.21.2.102:0/559885421 >> v1:172.21.2.102:6789/0 conn(0x7fac741b0870 legacy=0x7fac740efe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:18:09.829 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:09.819 7fac71d9b700 1 --2- 172.21.2.102:0/559885421 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fac60003aa0 0x7fac60005eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:09.829 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:09.819 7fac71d9b700 1 --2- 172.21.2.102:0/559885421 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fac60006750 0x7fac60008c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:09.829 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:09.819 7fac71d9b700 1 --2- 172.21.2.102:0/559885421 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fac600094f0 0x7fac6000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:09.829 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:09.819 7fac71d9b700 1 -- 172.21.2.102:0/559885421 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac6000c2c0 con 0x7fac60006750 2021-02-10T11:18:09.829 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:09.819 7fac71d9b700 1 -- 172.21.2.102:0/559885421 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac6000c4e0 con 0x7fac600094f0 2021-02-10T11:18:09.829 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:09.819 7fac71d9b700 1 -- 172.21.2.102:0/559885421 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac6000c700 con 0x7fac60003aa0 2021-02-10T11:18:09.829 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:09.819 7fac7259c700 1 --2- 172.21.2.102:0/559885421 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fac60006750 0x7fac60008c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:18:09.829 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:09.819 7fac7359e700 1 --2- 172.21.2.102:0/559885421 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fac600094f0 0x7fac6000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:18:09.829 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:09.819 7fac72d9d700 1 --2- 172.21.2.102:0/559885421 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fac60003aa0 0x7fac60005eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:18:09.829 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:09.819 7fac7259c700 1 -- 172.21.2.102:0/559885421 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fac60003aa0 msgr2=0x7fac60005eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:18:09.829 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:09.819 7fac7259c700 1 --2- 172.21.2.102:0/559885421 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fac60003aa0 0x7fac60005eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:09.829 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:09.819 7fac7259c700 1 -- 172.21.2.102:0/559885421 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fac600094f0 msgr2=0x7fac6000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:18:09.829 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:09.819 7fac7259c700 1 --2- 172.21.2.102:0/559885421 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fac600094f0 0x7fac6000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:09.830 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:09.819 7fac7259c700 1 -- 172.21.2.102:0/559885421 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac6000c9a0 con 0x7fac60006750 2021-02-10T11:18:09.830 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:09.819 7fac7259c700 1 --2- 172.21.2.102:0/559885421 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fac60006750 0x7fac60008c50 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 rx=0x7fac640017d0 tx=0x7fac6404bcc0).ready entity=mon.0 client_cookie=fcd2a98ae1aaf8a6 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:18:09.830 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:09.821 7fac71d9b700 1 -- 172.21.2.102:0/559885421 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fac640009b0 con 0x7fac60006750 2021-02-10T11:18:09.830 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:09.821 7fac71d9b700 1 -- 172.21.2.102:0/559885421 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac6404e310 con 0x7fac60006750 2021-02-10T11:18:09.830 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:09.821 7fac71d9b700 1 -- 172.21.2.102:0/559885421 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fac6404e770 con 0x7fac60006750 2021-02-10T11:18:09.830 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:09.821 7fac790ea700 1 -- 172.21.2.102:0/559885421 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fac60006750 msgr2=0x7fac60008c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:18:09.830 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:09.821 7fac790ea700 1 --2- 172.21.2.102:0/559885421 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fac60006750 0x7fac60008c50 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 rx=0x7fac640017d0 tx=0x7fac6404bcc0).stop 2021-02-10T11:18:09.830 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:09.821 7fac7359e700 1 -- 172.21.2.102:0/559885421 reap_dead start 2021-02-10T11:18:09.830 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:09.821 7fac790ea700 1 -- 172.21.2.102:0/559885421 shutdown_connections 2021-02-10T11:18:09.830 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:09.821 7fac790ea700 1 -- 172.21.2.102:0/559885421 >> 172.21.2.102:0/559885421 conn(0x7fac740ce3e0 msgr2=0x7fac740e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:18:09.830 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:09.821 7fac790ea700 1 -- 172.21.2.102:0/559885421 shutdown_connections 2021-02-10T11:18:09.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:09.821 7fac790ea700 1 -- 172.21.2.102:0/559885421 wait complete. 2021-02-10T11:18:09.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:09.835 7fac790ea700 1 Processor -- start 2021-02-10T11:18:09.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:09.835 7fac790ea700 1 -- start start 2021-02-10T11:18:09.832 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:09.836 7fac790ea700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fac740924b0 0x7fac741a3590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:09.832 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:09.836 7fac790ea700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fac741a3bb0 0x7fac740bf920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:09.832 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:09.836 7fac790ea700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fac740bff40 0x7fac740c0340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:09.832 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:09.836 7fac790ea700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac7409a0a0 con 0x7fac741a3bb0 2021-02-10T11:18:09.832 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:09.836 7fac790ea700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac740c0960 con 0x7fac740bff40 2021-02-10T11:18:09.832 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:09.836 7fac790ea700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac74098420 con 0x7fac740924b0 2021-02-10T11:18:09.832 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:09.836 7fac7259c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fac741a3bb0 0x7fac740bf920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:18:09.832 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:09.836 7fac7259c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fac741a3bb0 0x7fac740bf920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:53298/0 (socket says 172.21.2.102:53298) 2021-02-10T11:18:09.832 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:09.836 7fac7259c700 1 -- 172.21.2.102:0/2371117528 learned_addr learned my addr 172.21.2.102:0/2371117528 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:18:09.832 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:09.836 7fac7259c700 1 -- 172.21.2.102:0/2371117528 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fac740924b0 msgr2=0x7fac741a3590 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:18:09.832 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:09.836 7fac7259c700 1 --2- 172.21.2.102:0/2371117528 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fac740924b0 0x7fac741a3590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:18:09.832 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:09.836 7fac7259c700 1 -- 172.21.2.102:0/2371117528 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fac740bff40 msgr2=0x7fac740c0340 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:18:09.833 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:09.836 7fac7259c700 1 --2- 172.21.2.102:0/2371117528 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fac740bff40 0x7fac740c0340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:18:09.833 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:09.836 7fac7259c700 1 -- 172.21.2.102:0/2371117528 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac74098640 con 0x7fac741a3bb0 2021-02-10T11:18:09.833 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:09.836 7fac7259c700 1 --2- 172.21.2.102:0/2371117528 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fac741a3bb0 0x7fac740bf920 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 rx=0x7fac6404bfc0 tx=0x7fac6404e950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:18:09.833 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:09.836 7fac6b7fe700 1 -- 172.21.2.102:0/2371117528 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fac6404ed40 con 0x7fac741a3bb0 2021-02-10T11:18:09.833 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:09.837 7fac6b7fe700 1 -- 172.21.2.102:0/2371117528 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac64000d00 con 0x7fac741a3bb0 2021-02-10T11:18:09.833 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:09.837 7fac790ea700 1 -- 172.21.2.102:0/2371117528 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac74098640 con 0x7fac741a3bb0 2021-02-10T11:18:09.833 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:09.837 7fac6b7fe700 1 -- 172.21.2.102:0/2371117528 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fac640045d0 con 0x7fac741a3bb0 2021-02-10T11:18:09.833 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:09.837 7fac790ea700 1 -- 172.21.2.102:0/2371117528 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac74098bd0 con 0x7fac741a3bb0 2021-02-10T11:18:09.833 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:09.837 7fac6b7fe700 1 -- 172.21.2.102:0/2371117528 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fac64004140 con 0x7fac741a3bb0 2021-02-10T11:18:09.833 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:09.837 7fac6b7fe700 1 --2- 172.21.2.102:0/2371117528 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fac580186d0 0x7fac5801abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:09.833 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:09.837 7fac6b7fe700 1 -- 172.21.2.102:0/2371117528 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fac64076660 con 0x7fac741a3bb0 2021-02-10T11:18:09.833 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:09.837 7fac72d9d700 1 --2- 172.21.2.102:0/2371117528 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fac580186d0 0x7fac5801abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:18:09.833 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:09.838 7fac72d9d700 1 --2- 172.21.2.102:0/2371117528 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fac580186d0 0x7fac5801abd0 crc :-1 s=READY pgs=346 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:18:09.859 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:09.868 7fac790ea700 1 -- 172.21.2.102:0/2371117528 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac74105b40 con 0x7fac741a3bb0 2021-02-10T11:18:09.862 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:09.870 7fac6b7fe700 1 -- 172.21.2.102:0/2371117528 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fac640009b0 con 0x7fac741a3bb0 2021-02-10T11:18:09.992 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:10.000 7fac790ea700 1 -- 172.21.2.102:0/2371117528 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344448", "target": ["mgr", ""]}) v1 -- 0x7fac74098bd0 con 0x7fac580186d0 2021-02-10T11:18:09.996 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:10.004 7fac6b7fe700 1 -- 172.21.2.102:0/2371117528 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7fac5c006950 con 0x7fac580186d0 2021-02-10T11:18:09.999 INFO:teuthology.orchestra.run.gibba002.stdout:/volumes/_nogroup/subvolume_0000000000344448/d40d48ee-0900-4466-9af5-c6271118f09a 2021-02-10T11:18:10.001 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:10.009 7fac790ea700 1 -- 172.21.2.102:0/2371117528 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fac580186d0 msgr2=0x7fac5801abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:18:10.001 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:10.009 7fac790ea700 1 --2- 172.21.2.102:0/2371117528 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fac580186d0 0x7fac5801abd0 crc :-1 s=READY pgs=346 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:10.002 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:10.009 7fac790ea700 1 -- 172.21.2.102:0/2371117528 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fac741a3bb0 msgr2=0x7fac740bf920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:18:10.002 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:10.009 7fac790ea700 1 --2- 172.21.2.102:0/2371117528 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fac741a3bb0 0x7fac740bf920 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 rx=0x7fac6404bfc0 tx=0x7fac6404e950).stop 2021-02-10T11:18:10.002 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:10.009 7fac790ea700 1 -- 172.21.2.102:0/2371117528 shutdown_connections 2021-02-10T11:18:10.002 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:10.009 7fac790ea700 1 --2- 172.21.2.102:0/2371117528 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fac580186d0 0x7fac5801abd0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:10.002 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:10.009 7fac790ea700 1 --2- 172.21.2.102:0/2371117528 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fac740bff40 0x7fac740c0340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:18:10.002 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:10.009 7fac790ea700 1 --2- 172.21.2.102:0/2371117528 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fac741a3bb0 0x7fac740bf920 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:10.003 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:10.009 7fac790ea700 1 --2- 172.21.2.102:0/2371117528 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fac740924b0 0x7fac741a3590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:18:10.003 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:10.009 7fac790ea700 1 -- 172.21.2.102:0/2371117528 >> 172.21.2.102:0/2371117528 conn(0x7fac740b50c0 msgr2=0x7fac74092d90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:18:10.003 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:10.009 7fac790ea700 1 -- 172.21.2.102:0/2371117528 shutdown_connections 2021-02-10T11:18:10.003 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:10.009 7fac790ea700 1 -- 172.21.2.102:0/2371117528 wait complete. 2021-02-10T11:18:10.009 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000344448 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000344448/d40d48ee-0900-4466-9af5-c6271118f09a 2021-02-10T11:18:10.010 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:18:10.010 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:18:10.010 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:18:10.347 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:18:10.348 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:18:10.348 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344448/d40d48ee-0900-4466-9af5-c6271118f09a/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:18:10.731 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:18:10.731 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:18:10.731 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0250867 s, 41.8 MB/s 2021-02-10T11:18:10.732 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:18:10.732 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:18:10.732 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:18:11.068 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:18:12.538 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:18:12.539 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344448/d40d48ee-0900-4466-9af5-c6271118f09a/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:18:12.915 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:18:12.916 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:18:12.916 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0241213 s, 43.5 MB/s 2021-02-10T11:18:12.917 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:18:12.917 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:18:12.917 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:18:13.254 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:18:14.303 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:18:14.303 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344448/d40d48ee-0900-4466-9af5-c6271118f09a/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:18:14.678 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:18:14.678 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:18:14.678 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0242551 s, 43.2 MB/s 2021-02-10T11:18:14.679 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:18:14.679 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:18:14.679 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:18:15.014 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:18:15.014 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:18:15.015 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344448/d40d48ee-0900-4466-9af5-c6271118f09a/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:18:15.397 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:18:15.398 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:18:15.398 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0237108 s, 44.2 MB/s 2021-02-10T11:18:15.399 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:18:15.399 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:18:15.399 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:18:15.739 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:18:15.740 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:18:15.740 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344448/d40d48ee-0900-4466-9af5-c6271118f09a/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:18:16.130 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:18:16.131 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:18:16.131 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0236165 s, 44.4 MB/s 2021-02-10T11:18:16.132 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:18:16.132 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:18:16.132 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:18:16.473 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:18:16.474 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:18:16.474 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344448/d40d48ee-0900-4466-9af5-c6271118f09a/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:18:16.858 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:18:16.858 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:18:16.858 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.02378 s, 44.1 MB/s 2021-02-10T11:18:16.859 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:18:16.859 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:18:16.859 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:18:17.200 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:18:17.200 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:18:17.200 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344448/d40d48ee-0900-4466-9af5-c6271118f09a/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:18:17.575 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:18:17.575 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:18:17.575 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.022522 s, 46.6 MB/s 2021-02-10T11:18:17.576 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:18:17.576 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:18:17.576 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:18:17.913 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:18:17.913 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:18:17.914 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344448/d40d48ee-0900-4466-9af5-c6271118f09a/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:18:18.293 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:18:18.293 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:18:18.293 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0258704 s, 40.5 MB/s 2021-02-10T11:18:18.294 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:18:18.294 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:18:18.294 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:18:18.628 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:18:18.629 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:18:18.629 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344448/d40d48ee-0900-4466-9af5-c6271118f09a/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:18:19.001 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:18:19.002 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:18:19.002 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0233673 s, 44.9 MB/s 2021-02-10T11:18:19.003 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:18:19.003 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:18:19.003 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:18:19.339 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:18:19.340 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:18:19.340 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344448/d40d48ee-0900-4466-9af5-c6271118f09a/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:18:19.717 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:18:19.717 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:18:19.717 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0236262 s, 44.4 MB/s 2021-02-10T11:18:19.718 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000344449 2021-02-10T11:18:20.150 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:20.143 7ff5b7ce7700 1 Processor -- start 2021-02-10T11:18:20.150 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:20.144 7ff5b7ce7700 1 -- start start 2021-02-10T11:18:20.151 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:20.144 7ff5b7ce7700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7ff5b01b0470 0x7ff5b01b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:20.151 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:20.144 7ff5b7ce7700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7ff5b00ef610 0x7ff5b00efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:20.151 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:20.144 7ff5b7ce7700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5b00f0b50 con 0x7ff5b00f0030 2021-02-10T11:18:20.151 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:20.144 7ff5b7ce7700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5b00e52e0 con 0x7ff5b01b0470 2021-02-10T11:18:20.151 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:20.144 7ff5b7ce7700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5b00e5490 con 0x7ff5b00ef610 2021-02-10T11:18:20.151 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:20.144 7ff5b5b03700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7ff5b01b0470 0x7ff5b01b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:18:20.151 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:20.144 7ff5b5b03700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7ff5b01b0470 0x7ff5b01b0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:53306/0 (socket says 172.21.2.102:53306) 2021-02-10T11:18:20.151 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:20.144 7ff5b5b03700 1 -- 172.21.2.102:0/1166020656 learned_addr learned my addr 172.21.2.102:0/1166020656 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:18:20.151 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:20.144 7ff5b5b03700 1 -- 172.21.2.102:0/1166020656 >> v2:172.21.2.113:3300/0 conn(0x7ff5b00ef610 msgr2=0x7ff5b00efa10 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:18:20.151 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:20.144 7ff5b5b03700 1 --2- 172.21.2.102:0/1166020656 >> v2:172.21.2.113:3300/0 conn(0x7ff5b00ef610 0x7ff5b00efa10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:18:20.151 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:20.144 7ff5b5b03700 1 -- 172.21.2.102:0/1166020656 >> v1:172.21.2.113:6789/0 conn(0x7ff5b00f0030 legacy=0x7ff5b00f10d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:18:20.151 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:20.144 7ff5b5b03700 1 -- 172.21.2.102:0/1166020656 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5b00f0430 con 0x7ff5b01b0470 2021-02-10T11:18:20.152 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:20.145 7ff5b5b03700 1 --2- 172.21.2.102:0/1166020656 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff5b01b0470 0x7ff5b01b0870 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 rx=0x7ff5a4006700 tx=0x7ff5a4009b00).ready entity=mon.0 client_cookie=2dc03d6118fb1344 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:18:20.152 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:20.145 7ff5b4b01700 1 -- 172.21.2.102:0/1166020656 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff5a400c0c0 con 0x7ff5b01b0470 2021-02-10T11:18:20.152 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:20.145 7ff5b4b01700 1 -- 172.21.2.102:0/1166020656 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5a400c340 con 0x7ff5b01b0470 2021-02-10T11:18:20.152 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:20.145 7ff5b4b01700 1 -- 172.21.2.102:0/1166020656 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff5a400c820 con 0x7ff5b01b0470 2021-02-10T11:18:20.152 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:20.145 7ff5b7ce7700 1 -- 172.21.2.102:0/1166020656 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff5b01b0470 msgr2=0x7ff5b01b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:18:20.152 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:20.145 7ff5b7ce7700 1 --2- 172.21.2.102:0/1166020656 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff5b01b0470 0x7ff5b01b0870 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 rx=0x7ff5a4006700 tx=0x7ff5a4009b00).stop 2021-02-10T11:18:20.152 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:20.146 7ff5b7ce7700 1 -- 172.21.2.102:0/1166020656 shutdown_connections 2021-02-10T11:18:20.152 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:20.146 7ff5b7ce7700 1 --2- 172.21.2.102:0/1166020656 >> v2:172.21.2.113:3300/0 conn(0x7ff5b00ef610 0x7ff5b00efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:18:20.152 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:20.146 7ff5b7ce7700 1 --2- 172.21.2.102:0/1166020656 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff5b01b0470 0x7ff5b01b0870 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:20.152 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:20.146 7ff5b7ce7700 1 -- 172.21.2.102:0/1166020656 >> 172.21.2.102:0/1166020656 conn(0x7ff5b00ce3e0 msgr2=0x7ff5b00e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:18:20.152 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:20.146 7ff5b7ce7700 1 -- 172.21.2.102:0/1166020656 shutdown_connections 2021-02-10T11:18:20.152 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:20.146 7ff5b7ce7700 1 -- 172.21.2.102:0/1166020656 wait complete. 2021-02-10T11:18:20.153 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:20.160 7ff5b7ce7700 1 Processor -- start 2021-02-10T11:18:20.153 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:20.160 7ff5b7ce7700 1 -- start start 2021-02-10T11:18:20.153 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:20.161 7ff5b7ce7700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff5b00c36a0 0x7ff5b00c2b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:20.153 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:20.161 7ff5b7ce7700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff5b00c3190 0x7ff5b00b1d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:20.153 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:20.161 7ff5b7ce7700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff5b00b2370 0x7ff5b00b2d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:20.153 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:20.161 7ff5b7ce7700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5b00b27c0 con 0x7ff5b00b2370 2021-02-10T11:18:20.153 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:20.161 7ff5b7ce7700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5b00b3360 con 0x7ff5b00c36a0 2021-02-10T11:18:20.154 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:20.161 7ff5b7ce7700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5b00b3540 con 0x7ff5b00c3190 2021-02-10T11:18:20.155 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:20.161 7ff5b6304700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff5b00b2370 0x7ff5b00b2d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:18:20.155 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:20.161 7ff5b6304700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff5b00b2370 0x7ff5b00b2d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:53314/0 (socket says 172.21.2.102:53314) 2021-02-10T11:18:20.155 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:20.161 7ff5b6304700 1 -- 172.21.2.102:0/3039622 learned_addr learned my addr 172.21.2.102:0/3039622 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:18:20.155 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:20.161 7ff5b6304700 1 -- 172.21.2.102:0/3039622 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff5b00c3190 msgr2=0x7ff5b00b1d50 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:18:20.155 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:20.161 7ff5b6304700 1 --2- 172.21.2.102:0/3039622 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff5b00c3190 0x7ff5b00b1d50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:18:20.155 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:20.161 7ff5b6304700 1 -- 172.21.2.102:0/3039622 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff5b00c36a0 msgr2=0x7ff5b00c2b70 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:18:20.155 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:20.161 7ff5b6304700 1 --2- 172.21.2.102:0/3039622 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff5b00c36a0 0x7ff5b00c2b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:18:20.156 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:20.161 7ff5b6304700 1 -- 172.21.2.102:0/3039622 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5b00b3760 con 0x7ff5b00b2370 2021-02-10T11:18:20.156 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:20.161 7ff5b6304700 1 --2- 172.21.2.102:0/3039622 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff5b00b2370 0x7ff5b00b2d40 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 rx=0x7ff5a8009940 tx=0x7ff5a800bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:18:20.156 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:20.161 7ff5a27fc700 1 -- 172.21.2.102:0/3039622 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff5a8003030 con 0x7ff5b00b2370 2021-02-10T11:18:20.156 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:20.161 7ff5a27fc700 1 -- 172.21.2.102:0/3039622 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5a8001270 con 0x7ff5b00b2370 2021-02-10T11:18:20.156 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:20.161 7ff5b7ce7700 1 -- 172.21.2.102:0/3039622 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5b00b3760 con 0x7ff5b00b2370 2021-02-10T11:18:20.156 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:20.162 7ff5a27fc700 1 -- 172.21.2.102:0/3039622 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff5a8001750 con 0x7ff5b00b2370 2021-02-10T11:18:20.156 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:20.162 7ff5b7ce7700 1 -- 172.21.2.102:0/3039622 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5b00b3760 con 0x7ff5b00b2370 2021-02-10T11:18:20.156 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:20.162 7ff5a27fc700 1 -- 172.21.2.102:0/3039622 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ff5a8001450 con 0x7ff5b00b2370 2021-02-10T11:18:20.156 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:20.162 7ff5a27fc700 1 --2- 172.21.2.102:0/3039622 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff598018730 0x7ff59801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:20.156 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:20.162 7ff5a27fc700 1 -- 172.21.2.102:0/3039622 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7ff5a8032b60 con 0x7ff5b00b2370 2021-02-10T11:18:20.156 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:20.162 7ff5b5b03700 1 --2- 172.21.2.102:0/3039622 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff598018730 0x7ff59801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:18:20.156 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:20.162 7ff5b5b03700 1 --2- 172.21.2.102:0/3039622 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff598018730 0x7ff59801abc0 crc :-1 s=READY pgs=347 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:18:20.185 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:20.194 7ff5b7ce7700 1 -- 172.21.2.102:0/3039622 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5b00edd10 con 0x7ff5b00b2370 2021-02-10T11:18:20.187 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:20.197 7ff5a27fc700 1 -- 172.21.2.102:0/3039622 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ff5a800be30 con 0x7ff5b00b2370 2021-02-10T11:18:20.312 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:20.321 7ff5b7ce7700 1 -- 172.21.2.102:0/3039622 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344449", "target": ["mgr", ""]}) v1 -- 0x7ff5b00edd10 con 0x7ff598018730 2021-02-10T11:18:20.341 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:20.350 7ff5a27fc700 1 -- 172.21.2.102:0/3039622 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7ff5a4009390 con 0x7ff598018730 2021-02-10T11:18:20.345 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:20.354 7ff5b7ce7700 1 -- 172.21.2.102:0/3039622 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff598018730 msgr2=0x7ff59801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:18:20.345 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:20.354 7ff5b7ce7700 1 --2- 172.21.2.102:0/3039622 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff598018730 0x7ff59801abc0 crc :-1 s=READY pgs=347 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:20.345 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:20.354 7ff5b7ce7700 1 -- 172.21.2.102:0/3039622 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff5b00b2370 msgr2=0x7ff5b00b2d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:18:20.346 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:20.354 7ff5b7ce7700 1 --2- 172.21.2.102:0/3039622 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff5b00b2370 0x7ff5b00b2d40 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 rx=0x7ff5a8009940 tx=0x7ff5a800bcf0).stop 2021-02-10T11:18:20.346 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:20.354 7ff5b7ce7700 1 -- 172.21.2.102:0/3039622 shutdown_connections 2021-02-10T11:18:20.346 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:20.354 7ff5b7ce7700 1 --2- 172.21.2.102:0/3039622 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff5b00b2370 0x7ff5b00b2d40 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:20.346 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:20.354 7ff5b7ce7700 1 --2- 172.21.2.102:0/3039622 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff5b00c3190 0x7ff5b00b1d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:18:20.346 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:20.354 7ff5b7ce7700 1 --2- 172.21.2.102:0/3039622 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff598018730 0x7ff59801abc0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:20.346 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:20.354 7ff5b7ce7700 1 --2- 172.21.2.102:0/3039622 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff5b00c36a0 0x7ff5b00c2b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:18:20.346 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:20.354 7ff5b7ce7700 1 -- 172.21.2.102:0/3039622 >> 172.21.2.102:0/3039622 conn(0x7ff5b009f630 msgr2=0x7ff5b00a0130 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:18:20.346 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:20.354 7ff5b7ce7700 1 -- 172.21.2.102:0/3039622 shutdown_connections 2021-02-10T11:18:20.346 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:20.354 7ff5b7ce7700 1 -- 172.21.2.102:0/3039622 wait complete. 2021-02-10T11:18:20.353 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000344449 2021-02-10T11:18:20.786 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:20.779 7fdd62ff3700 1 Processor -- start 2021-02-10T11:18:20.787 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:20.779 7fdd62ff3700 1 -- start start 2021-02-10T11:18:20.787 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:20.780 7fdd62ff3700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdd5c1b0470 0x7fdd5c1b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:20.787 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:20.780 7fdd62ff3700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fdd5c0f20e0 0x7fdd5c0f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:20.787 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:20.780 7fdd62ff3700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd5c0f0b50 con 0x7fdd5c0ef610 2021-02-10T11:18:20.787 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:20.780 7fdd62ff3700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd5c0f0820 con 0x7fdd5c0f20e0 2021-02-10T11:18:20.787 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:20.780 7fdd62ff3700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd5c0f24e0 con 0x7fdd5c1b0470 2021-02-10T11:18:20.787 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:20.780 7fdd61610700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fdd5c0f20e0 0x7fdd5c0f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:18:20.787 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:20.780 7fdd61610700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fdd5c0f20e0 0x7fdd5c0f0200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:53322/0 (socket says 172.21.2.102:53322) 2021-02-10T11:18:20.787 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:20.780 7fdd61610700 1 -- 172.21.2.102:0/3840894003 learned_addr learned my addr 172.21.2.102:0/3840894003 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:18:20.787 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:20.780 7fdd61610700 1 -- 172.21.2.102:0/3840894003 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdd5c1b0470 msgr2=0x7fdd5c1b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:18:20.788 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:20.780 7fdd61610700 1 --2- 172.21.2.102:0/3840894003 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdd5c1b0470 0x7fdd5c1b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:18:20.788 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:20.780 7fdd61610700 1 -- 172.21.2.102:0/3840894003 >> v1:172.21.2.113:6789/0 conn(0x7fdd5c0ef610 legacy=0x7fdd5c0efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:18:20.788 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:20.780 7fdd61610700 1 -- 172.21.2.102:0/3840894003 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd5c0e52e0 con 0x7fdd5c0f20e0 2021-02-10T11:18:20.788 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:20.781 7fdd61610700 1 --2- 172.21.2.102:0/3840894003 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdd5c0f20e0 0x7fdd5c0f0200 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 rx=0x7fdd54006b80 tx=0x7fdd54009b90).ready entity=mon.0 client_cookie=b1282e31c543b35d server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:18:20.788 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:20.781 7fdd5b7fe700 1 -- 172.21.2.102:0/3840894003 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdd540022e0 con 0x7fdd5c0f20e0 2021-02-10T11:18:20.788 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:20.781 7fdd5b7fe700 1 -- 172.21.2.102:0/3840894003 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd54002560 con 0x7fdd5c0f20e0 2021-02-10T11:18:20.788 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:20.781 7fdd5b7fe700 1 -- 172.21.2.102:0/3840894003 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdd54002a80 con 0x7fdd5c0f20e0 2021-02-10T11:18:20.788 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:20.781 7fdd62ff3700 1 -- 172.21.2.102:0/3840894003 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdd5c0f20e0 msgr2=0x7fdd5c0f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:18:20.788 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:20.781 7fdd62ff3700 1 --2- 172.21.2.102:0/3840894003 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdd5c0f20e0 0x7fdd5c0f0200 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 rx=0x7fdd54006b80 tx=0x7fdd54009b90).stop 2021-02-10T11:18:20.788 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:20.781 7fdd62ff3700 1 -- 172.21.2.102:0/3840894003 shutdown_connections 2021-02-10T11:18:20.788 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:20.781 7fdd62ff3700 1 --2- 172.21.2.102:0/3840894003 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdd5c0f20e0 0x7fdd5c0f0200 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:20.789 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:20.781 7fdd62ff3700 1 --2- 172.21.2.102:0/3840894003 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdd5c1b0470 0x7fdd5c1b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:18:20.789 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:20.781 7fdd62ff3700 1 -- 172.21.2.102:0/3840894003 >> 172.21.2.102:0/3840894003 conn(0x7fdd5c0ce3e0 msgr2=0x7fdd5c0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:18:20.789 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:20.781 7fdd62ff3700 1 -- 172.21.2.102:0/3840894003 shutdown_connections 2021-02-10T11:18:20.789 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:20.781 7fdd62ff3700 1 -- 172.21.2.102:0/3840894003 wait complete. 2021-02-10T11:18:20.789 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:20.796 7fdd62ff3700 1 Processor -- start 2021-02-10T11:18:20.789 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:20.796 7fdd62ff3700 1 -- start start 2021-02-10T11:18:20.791 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:20.796 7fdd62ff3700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdd5c0b2840 0x7fdd5c1a4300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:20.791 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:20.796 7fdd62ff3700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdd5c0b30d0 0x7fdd5c0b34d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:20.791 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:20.796 7fdd62ff3700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fdd5c0b3af0 0x7fdd5c0f0580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:20.791 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:20.796 7fdd62ff3700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd5c0b2c40 con 0x7fdd5c0b30d0 2021-02-10T11:18:20.791 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:20.796 7fdd62ff3700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd5c1a4920 con 0x7fdd5c0b3af0 2021-02-10T11:18:20.791 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:20.796 7fdd62ff3700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd5c0f0ba0 con 0x7fdd5c0b2840 2021-02-10T11:18:20.791 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:20.796 7fdd5bfff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdd5c0b30d0 0x7fdd5c0b34d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:18:20.791 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:20.797 7fdd5bfff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdd5c0b30d0 0x7fdd5c0b34d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:53326/0 (socket says 172.21.2.102:53326) 2021-02-10T11:18:20.791 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:20.797 7fdd5bfff700 1 -- 172.21.2.102:0/3503328876 learned_addr learned my addr 172.21.2.102:0/3503328876 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:18:20.792 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:20.797 7fdd5bfff700 1 -- 172.21.2.102:0/3503328876 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdd5c0b2840 msgr2=0x7fdd5c1a4300 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:18:20.792 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:20.797 7fdd5bfff700 1 --2- 172.21.2.102:0/3503328876 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdd5c0b2840 0x7fdd5c1a4300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:18:20.792 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:20.797 7fdd5bfff700 1 -- 172.21.2.102:0/3503328876 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fdd5c0b3af0 msgr2=0x7fdd5c0f0580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:18:20.792 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:20.797 7fdd5bfff700 1 --2- 172.21.2.102:0/3503328876 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fdd5c0b3af0 0x7fdd5c0f0580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:18:20.792 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:20.797 7fdd5bfff700 1 -- 172.21.2.102:0/3503328876 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd5c0f0dc0 con 0x7fdd5c0b30d0 2021-02-10T11:18:20.792 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:20.797 7fdd5bfff700 1 --2- 172.21.2.102:0/3503328876 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdd5c0b30d0 0x7fdd5c0b34d0 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 rx=0x7fdd4c009940 tx=0x7fdd4c00bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:18:20.792 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:20.797 7fdd597fa700 1 -- 172.21.2.102:0/3503328876 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdd4c000fe0 con 0x7fdd5c0b30d0 2021-02-10T11:18:20.792 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:20.797 7fdd597fa700 1 -- 172.21.2.102:0/3503328876 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd4c000d30 con 0x7fdd5c0b30d0 2021-02-10T11:18:20.792 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:20.797 7fdd62ff3700 1 -- 172.21.2.102:0/3503328876 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd5c0f0dc0 con 0x7fdd5c0b30d0 2021-02-10T11:18:20.792 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:20.797 7fdd597fa700 1 -- 172.21.2.102:0/3503328876 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdd4c00c570 con 0x7fdd5c0b30d0 2021-02-10T11:18:20.792 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:20.797 7fdd62ff3700 1 -- 172.21.2.102:0/3503328876 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd5c0f1350 con 0x7fdd5c0b30d0 2021-02-10T11:18:20.792 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:20.798 7fdd597fa700 1 -- 172.21.2.102:0/3503328876 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fdd4c001230 con 0x7fdd5c0b30d0 2021-02-10T11:18:20.792 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:20.798 7fdd597fa700 1 --2- 172.21.2.102:0/3503328876 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fdd400186c0 0x7fdd4001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:20.793 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:20.798 7fdd597fa700 1 -- 172.21.2.102:0/3503328876 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fdd4c032920 con 0x7fdd5c0b30d0 2021-02-10T11:18:20.793 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:20.798 7fdd60e0f700 1 --2- 172.21.2.102:0/3503328876 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fdd400186c0 0x7fdd4001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:18:20.793 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:20.798 7fdd60e0f700 1 --2- 172.21.2.102:0/3503328876 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fdd400186c0 0x7fdd4001abc0 crc :-1 s=READY pgs=348 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:18:20.821 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:20.830 7fdd62ff3700 1 -- 172.21.2.102:0/3503328876 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd5c0f18f0 con 0x7fdd5c0b30d0 2021-02-10T11:18:20.823 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:20.832 7fdd597fa700 1 -- 172.21.2.102:0/3503328876 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fdd4c000f90 con 0x7fdd5c0b30d0 2021-02-10T11:18:20.950 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:20.960 7fdd62ff3700 1 -- 172.21.2.102:0/3503328876 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344449", "target": ["mgr", ""]}) v1 -- 0x7fdd5c0f1350 con 0x7fdd400186c0 2021-02-10T11:18:20.953 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:20.963 7fdd597fa700 1 -- 172.21.2.102:0/3503328876 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7fdd44006960 con 0x7fdd400186c0 2021-02-10T11:18:20.954 INFO:teuthology.orchestra.run.gibba002.stdout:/volumes/_nogroup/subvolume_0000000000344449/f5b6bfc0-20f6-44ac-a781-85bf37a69b3e 2021-02-10T11:18:20.955 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:20.964 7fdd62ff3700 1 -- 172.21.2.102:0/3503328876 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fdd400186c0 msgr2=0x7fdd4001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:18:20.955 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:20.964 7fdd62ff3700 1 --2- 172.21.2.102:0/3503328876 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fdd400186c0 0x7fdd4001abc0 crc :-1 s=READY pgs=348 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:20.955 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:20.964 7fdd62ff3700 1 -- 172.21.2.102:0/3503328876 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdd5c0b30d0 msgr2=0x7fdd5c0b34d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:18:20.955 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:20.964 7fdd62ff3700 1 --2- 172.21.2.102:0/3503328876 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdd5c0b30d0 0x7fdd5c0b34d0 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 rx=0x7fdd4c009940 tx=0x7fdd4c00bcf0).stop 2021-02-10T11:18:20.955 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:20.964 7fdd62ff3700 1 -- 172.21.2.102:0/3503328876 shutdown_connections 2021-02-10T11:18:20.955 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:20.964 7fdd62ff3700 1 --2- 172.21.2.102:0/3503328876 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fdd400186c0 0x7fdd4001abc0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:20.955 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:20.964 7fdd62ff3700 1 --2- 172.21.2.102:0/3503328876 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fdd5c0b3af0 0x7fdd5c0f0580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:18:20.956 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:20.964 7fdd62ff3700 1 --2- 172.21.2.102:0/3503328876 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdd5c0b30d0 0x7fdd5c0b34d0 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:20.956 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:20.964 7fdd62ff3700 1 --2- 172.21.2.102:0/3503328876 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdd5c0b2840 0x7fdd5c1a4300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:18:20.956 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:20.964 7fdd62ff3700 1 -- 172.21.2.102:0/3503328876 >> 172.21.2.102:0/3503328876 conn(0x7fdd5c0c36f0 msgr2=0x7fdd5c094c30 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:18:20.956 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:20.964 7fdd62ff3700 1 -- 172.21.2.102:0/3503328876 shutdown_connections 2021-02-10T11:18:20.956 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:20.964 7fdd62ff3700 1 -- 172.21.2.102:0/3503328876 wait complete. 2021-02-10T11:18:20.964 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000344449 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000344449/f5b6bfc0-20f6-44ac-a781-85bf37a69b3e 2021-02-10T11:18:20.964 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:18:20.965 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:18:20.965 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:18:21.301 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:18:21.302 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:18:21.302 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344449/f5b6bfc0-20f6-44ac-a781-85bf37a69b3e/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:18:21.681 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:18:21.682 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:18:21.682 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0221578 s, 47.3 MB/s 2021-02-10T11:18:21.683 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:18:21.683 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:18:21.683 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:18:22.019 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:18:22.020 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:18:22.020 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344449/f5b6bfc0-20f6-44ac-a781-85bf37a69b3e/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:18:22.397 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:18:22.398 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:18:22.398 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0253761 s, 41.3 MB/s 2021-02-10T11:18:22.399 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:18:22.399 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:18:22.399 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:18:22.734 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:18:22.735 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:18:22.735 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344449/f5b6bfc0-20f6-44ac-a781-85bf37a69b3e/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:18:23.109 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:18:23.109 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:18:23.109 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0229748 s, 45.6 MB/s 2021-02-10T11:18:23.110 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:18:23.110 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:18:23.110 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:18:23.445 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:18:23.445 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:18:23.446 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344449/f5b6bfc0-20f6-44ac-a781-85bf37a69b3e/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:18:23.821 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:18:23.821 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:18:23.821 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0248933 s, 42.1 MB/s 2021-02-10T11:18:23.822 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:18:23.822 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:18:23.822 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:18:24.160 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:18:24.160 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:18:24.160 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344449/f5b6bfc0-20f6-44ac-a781-85bf37a69b3e/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:18:24.535 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:18:24.535 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:18:24.535 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0237581 s, 44.1 MB/s 2021-02-10T11:18:24.536 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:18:24.536 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:18:24.536 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:18:24.872 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:18:24.873 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:18:24.873 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344449/f5b6bfc0-20f6-44ac-a781-85bf37a69b3e/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:18:25.249 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:18:25.249 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:18:25.249 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0253304 s, 41.4 MB/s 2021-02-10T11:18:25.250 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:18:25.250 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:18:25.250 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:18:25.585 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:18:25.585 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:18:25.586 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344449/f5b6bfc0-20f6-44ac-a781-85bf37a69b3e/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:18:25.961 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:18:25.962 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:18:25.962 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.023235 s, 45.1 MB/s 2021-02-10T11:18:25.963 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:18:25.963 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:18:25.963 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:18:26.299 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:18:26.300 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:18:26.300 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344449/f5b6bfc0-20f6-44ac-a781-85bf37a69b3e/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:18:26.677 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:18:26.677 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:18:26.677 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0234017 s, 44.8 MB/s 2021-02-10T11:18:26.678 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:18:26.678 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:18:26.678 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:18:27.013 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:18:27.014 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:18:27.014 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344449/f5b6bfc0-20f6-44ac-a781-85bf37a69b3e/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:18:27.391 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:18:27.391 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:18:27.391 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0249581 s, 42.0 MB/s 2021-02-10T11:18:27.392 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:18:27.393 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:18:27.393 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:18:27.729 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:18:27.729 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:18:27.730 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344449/f5b6bfc0-20f6-44ac-a781-85bf37a69b3e/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:18:28.109 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:18:28.109 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:18:28.109 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.026594 s, 39.4 MB/s 2021-02-10T11:18:28.110 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000344450 2021-02-10T11:18:28.541 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:28.534 7f5170ff5700 1 Processor -- start 2021-02-10T11:18:28.541 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:28.534 7f5170ff5700 1 -- start start 2021-02-10T11:18:28.541 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:28.534 7f5170ff5700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f516c1b0470 0x7f516c1b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:28.541 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:28.534 7f5170ff5700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f516c0ef610 0x7f516c0efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:28.541 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:28.534 7f5170ff5700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f516c0f0b50 con 0x7f516c0f0030 2021-02-10T11:18:28.542 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:28.534 7f5170ff5700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f516c0e52e0 con 0x7f516c0ef610 2021-02-10T11:18:28.542 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:28.534 7f5170ff5700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f516c0e5490 con 0x7f516c1b0470 2021-02-10T11:18:28.542 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:28.534 7f516a59c700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f516c0ef610 0x7f516c0efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:18:28.542 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:28.534 7f516a59c700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f516c0ef610 0x7f516c0efa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:53334/0 (socket says 172.21.2.102:53334) 2021-02-10T11:18:28.542 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:28.534 7f516a59c700 1 -- 172.21.2.102:0/1482781222 learned_addr learned my addr 172.21.2.102:0/1482781222 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:18:28.542 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:28.534 7f516ad9d700 1 --2- 172.21.2.102:0/1482781222 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f516c1b0470 0x7f516c1b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:18:28.542 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:28.535 7f516a59c700 1 -- 172.21.2.102:0/1482781222 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f516c1b0470 msgr2=0x7f516c1b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:18:28.542 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:28.535 7f516a59c700 1 --2- 172.21.2.102:0/1482781222 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f516c1b0470 0x7f516c1b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:28.542 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:28.535 7f516a59c700 1 -- 172.21.2.102:0/1482781222 >> v1:172.21.2.102:6789/0 conn(0x7f516c0f0030 legacy=0x7f516c0f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:18:28.542 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:28.535 7f516a59c700 1 -- 172.21.2.102:0/1482781222 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f516c0f0430 con 0x7f516c0ef610 2021-02-10T11:18:28.542 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:28.535 7f516ad9d700 1 --2- 172.21.2.102:0/1482781222 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f516c1b0470 0x7f516c1b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:18:28.542 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:28.535 7f516a59c700 1 --2- 172.21.2.102:0/1482781222 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f516c0ef610 0x7f516c0efa10 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 rx=0x7f5164006a80 tx=0x7f5164009f00).ready entity=mon.0 client_cookie=119d0bc5746bcb4f server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:18:28.543 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:28.535 7f5169d9b700 1 -- 172.21.2.102:0/1482781222 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f516400b5c0 con 0x7f516c0ef610 2021-02-10T11:18:28.543 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:28.535 7f5169d9b700 1 -- 172.21.2.102:0/1482781222 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f516400b840 con 0x7f516c0ef610 2021-02-10T11:18:28.543 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:28.535 7f5169d9b700 1 -- 172.21.2.102:0/1482781222 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f516400bd60 con 0x7f516c0ef610 2021-02-10T11:18:28.543 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:28.535 7f5170ff5700 1 -- 172.21.2.102:0/1482781222 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f516c0ef610 msgr2=0x7f516c0efa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:18:28.543 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:28.535 7f5170ff5700 1 --2- 172.21.2.102:0/1482781222 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f516c0ef610 0x7f516c0efa10 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 rx=0x7f5164006a80 tx=0x7f5164009f00).stop 2021-02-10T11:18:28.543 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:28.536 7f5170ff5700 1 -- 172.21.2.102:0/1482781222 shutdown_connections 2021-02-10T11:18:28.543 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:28.536 7f5170ff5700 1 --2- 172.21.2.102:0/1482781222 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f516c0ef610 0x7f516c0efa10 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:28.543 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:28.536 7f5170ff5700 1 --2- 172.21.2.102:0/1482781222 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f516c1b0470 0x7f516c1b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:28.543 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:28.536 7f5170ff5700 1 -- 172.21.2.102:0/1482781222 >> 172.21.2.102:0/1482781222 conn(0x7f516c0ce3e0 msgr2=0x7f516c0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:18:28.543 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:28.536 7f5170ff5700 1 -- 172.21.2.102:0/1482781222 shutdown_connections 2021-02-10T11:18:28.543 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:28.536 7f5170ff5700 1 -- 172.21.2.102:0/1482781222 wait complete. 2021-02-10T11:18:28.545 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:28.551 7f5170ff5700 1 Processor -- start 2021-02-10T11:18:28.545 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:28.551 7f5170ff5700 1 -- start start 2021-02-10T11:18:28.545 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:28.551 7f5170ff5700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f516c0a0420 0x7f516c0a0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:28.545 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:28.551 7f5170ff5700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f516c189d00 0x7f516c18a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:28.545 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:28.551 7f5170ff5700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f516c18a720 0x7f516c0b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:28.545 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:28.551 7f5170ff5700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f516c189780 con 0x7f516c18a720 2021-02-10T11:18:28.545 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:28.551 7f5170ff5700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f516c18ab20 con 0x7f516c189d00 2021-02-10T11:18:28.545 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:28.551 7f5170ff5700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f516c0a1140 con 0x7f516c0a0420 2021-02-10T11:18:28.545 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:28.551 7f516b59e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f516c18a720 0x7f516c0b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:18:28.545 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:28.551 7f516b59e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f516c18a720 0x7f516c0b7be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:53342/0 (socket says 172.21.2.102:53342) 2021-02-10T11:18:28.546 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:28.551 7f516b59e700 1 -- 172.21.2.102:0/3709217778 learned_addr learned my addr 172.21.2.102:0/3709217778 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:18:28.546 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:28.551 7f516a59c700 1 --2- 172.21.2.102:0/3709217778 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f516c189d00 0x7f516c18a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:18:28.546 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:28.552 7f516ad9d700 1 --2- 172.21.2.102:0/3709217778 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f516c0a0420 0x7f516c0a0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:18:28.546 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:28.552 7f516b59e700 1 -- 172.21.2.102:0/3709217778 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f516c0a0420 msgr2=0x7f516c0a0820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:18:28.546 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:28.552 7f516b59e700 1 --2- 172.21.2.102:0/3709217778 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f516c0a0420 0x7f516c0a0820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:28.546 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:28.552 7f516b59e700 1 -- 172.21.2.102:0/3709217778 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f516c189d00 msgr2=0x7f516c18a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:18:28.546 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:28.552 7f516b59e700 1 --2- 172.21.2.102:0/3709217778 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f516c189d00 0x7f516c18a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:28.546 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:28.552 7f516b59e700 1 -- 172.21.2.102:0/3709217778 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f516c0b8200 con 0x7f516c18a720 2021-02-10T11:18:28.546 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:28.552 7f516b59e700 1 --2- 172.21.2.102:0/3709217778 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f516c18a720 0x7f516c0b7be0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 rx=0x7f515c006f00 tx=0x7f515c00b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:18:28.546 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:28.552 7f51637fe700 1 -- 172.21.2.102:0/3709217778 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f515c006030 con 0x7f516c18a720 2021-02-10T11:18:28.546 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:28.552 7f51637fe700 1 -- 172.21.2.102:0/3709217778 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f515c0062f0 con 0x7f516c18a720 2021-02-10T11:18:28.546 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:28.552 7f5170ff5700 1 -- 172.21.2.102:0/3709217778 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f516c0b8200 con 0x7f516c18a720 2021-02-10T11:18:28.547 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:28.552 7f51637fe700 1 -- 172.21.2.102:0/3709217778 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f515c005770 con 0x7f516c18a720 2021-02-10T11:18:28.547 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:28.552 7f5170ff5700 1 -- 172.21.2.102:0/3709217778 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f516c0b8790 con 0x7f516c18a720 2021-02-10T11:18:28.547 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:28.553 7f51637fe700 1 -- 172.21.2.102:0/3709217778 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f515c0054f0 con 0x7f516c18a720 2021-02-10T11:18:28.547 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:28.553 7f51637fe700 1 --2- 172.21.2.102:0/3709217778 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f51500186c0 0x7f515001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:28.547 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:28.553 7f51637fe700 1 -- 172.21.2.102:0/3709217778 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f515c032560 con 0x7f516c18a720 2021-02-10T11:18:28.547 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:28.553 7f516ad9d700 1 --2- 172.21.2.102:0/3709217778 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f51500186c0 0x7f515001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:18:28.547 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:28.553 7f516ad9d700 1 --2- 172.21.2.102:0/3709217778 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f51500186c0 0x7f515001abc0 crc :-1 s=READY pgs=349 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:18:28.575 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:28.584 7f5170ff5700 1 -- 172.21.2.102:0/3709217778 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f516c105ae0 con 0x7f516c18a720 2021-02-10T11:18:28.577 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:28.587 7f51637fe700 1 -- 172.21.2.102:0/3709217778 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f515c030d10 con 0x7f516c18a720 2021-02-10T11:18:28.702 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:28.711 7f5170ff5700 1 -- 172.21.2.102:0/3709217778 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344450", "target": ["mgr", ""]}) v1 -- 0x7f516c0b9020 con 0x7f51500186c0 2021-02-10T11:18:28.730 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:28.739 7f51637fe700 1 -- 172.21.2.102:0/3709217778 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f5154006a30 con 0x7f51500186c0 2021-02-10T11:18:28.735 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:28.744 7f5170ff5700 1 -- 172.21.2.102:0/3709217778 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f51500186c0 msgr2=0x7f515001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:18:28.735 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:28.744 7f5170ff5700 1 --2- 172.21.2.102:0/3709217778 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f51500186c0 0x7f515001abc0 crc :-1 s=READY pgs=349 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:28.735 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:28.744 7f5170ff5700 1 -- 172.21.2.102:0/3709217778 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f516c18a720 msgr2=0x7f516c0b7be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:18:28.735 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:28.744 7f5170ff5700 1 --2- 172.21.2.102:0/3709217778 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f516c18a720 0x7f516c0b7be0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 rx=0x7f515c006f00 tx=0x7f515c00b940).stop 2021-02-10T11:18:28.735 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:28.744 7f5170ff5700 1 -- 172.21.2.102:0/3709217778 shutdown_connections 2021-02-10T11:18:28.735 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:28.744 7f5170ff5700 1 --2- 172.21.2.102:0/3709217778 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f516c18a720 0x7f516c0b7be0 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:28.735 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:28.744 7f5170ff5700 1 --2- 172.21.2.102:0/3709217778 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f51500186c0 0x7f515001abc0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:28.735 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:28.744 7f5170ff5700 1 --2- 172.21.2.102:0/3709217778 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f516c189d00 0x7f516c18a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:28.735 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:28.744 7f5170ff5700 1 --2- 172.21.2.102:0/3709217778 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f516c0a0420 0x7f516c0a0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:28.736 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:28.744 7f5170ff5700 1 -- 172.21.2.102:0/3709217778 >> 172.21.2.102:0/3709217778 conn(0x7f516c094e10 msgr2=0x7f516c0a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:18:28.736 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:28.744 7f5170ff5700 1 -- 172.21.2.102:0/3709217778 shutdown_connections 2021-02-10T11:18:28.736 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:28.744 7f5170ff5700 1 -- 172.21.2.102:0/3709217778 wait complete. 2021-02-10T11:18:28.743 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000344450 2021-02-10T11:18:29.175 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:29.167 7f14bb505700 1 Processor -- start 2021-02-10T11:18:29.176 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:29.167 7f14bb505700 1 -- start start 2021-02-10T11:18:29.176 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:29.167 7f14bb505700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f14b41b0470 0x7f14b41b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:29.176 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:29.167 7f14bb505700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14b40f0b50 con 0x7f14b40f20e0 2021-02-10T11:18:29.176 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:29.167 7f14bb505700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14b40f24e0 con 0x7f14b40ef610 2021-02-10T11:18:29.176 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:29.167 7f14bb505700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14b40e5ea0 con 0x7f14b41b0470 2021-02-10T11:18:29.176 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:29.168 7f14b9b22700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f14b40f20e0 0x7f14b40f0200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:45570/0 (socket says 172.21.2.102:45570) 2021-02-10T11:18:29.176 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:29.168 7f14b9b22700 1 -- 172.21.2.102:0/3026859009 learned_addr learned my addr 172.21.2.102:0/3026859009 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:18:29.176 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:29.168 7f14b9321700 1 --2- 172.21.2.102:0/3026859009 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f14b41b0470 0x7f14b41b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:18:29.176 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:29.168 7f14a7fff700 1 -- 172.21.2.102:0/3026859009 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2591849376 0 0) 0x7f14ac000f90 con 0x7f14b40f20e0 2021-02-10T11:18:29.176 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:29.168 7f14a7fff700 1 -- 172.21.2.102:0/3026859009 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14a0002fe0 con 0x7f14b40f20e0 2021-02-10T11:18:29.177 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:29.168 7f14a7fff700 1 -- 172.21.2.102:0/3026859009 <== mon.1 v1:172.21.2.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2630753136 0 0) 0x7f14a8001080 con 0x7f14b40ef610 2021-02-10T11:18:29.177 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:29.168 7f14a7fff700 1 -- 172.21.2.102:0/3026859009 --> v1:172.21.2.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14a0003210 con 0x7f14b40ef610 2021-02-10T11:18:29.177 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:29.169 7f14a7fff700 1 -- 172.21.2.102:0/3026859009 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2216813868 0 0) 0x7f14ac001500 con 0x7f14b40f20e0 2021-02-10T11:18:29.177 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:29.169 7f14a7fff700 1 -- 172.21.2.102:0/3026859009 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f14b41b0470 msgr2=0x7f14b41b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:18:29.177 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:29.169 7f14a7fff700 1 --2- 172.21.2.102:0/3026859009 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f14b41b0470 0x7f14b41b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:29.177 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:29.169 7f14a7fff700 1 -- 172.21.2.102:0/3026859009 >> v1:172.21.2.113:6789/0 conn(0x7f14b40ef610 legacy=0x7f14b40efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:18:29.177 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:29.169 7f14a7fff700 1 -- 172.21.2.102:0/3026859009 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14b40e52e0 con 0x7f14b40f20e0 2021-02-10T11:18:29.177 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:29.169 7f14a7fff700 1 -- 172.21.2.102:0/3026859009 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7f14ac001a00 con 0x7f14b40f20e0 2021-02-10T11:18:29.177 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:29.169 7f14a7fff700 1 -- 172.21.2.102:0/3026859009 >> v1:172.21.2.102:6789/0 conn(0x7f14b40f20e0 legacy=0x7f14b40f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:18:29.177 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:29.169 7f14a7fff700 1 --2- 172.21.2.102:0/3026859009 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f14a0003b90 0x7f14a0005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:29.177 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:29.169 7f14a7fff700 1 --2- 172.21.2.102:0/3026859009 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f14a00067a0 0x7f14a0008c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:29.177 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:29.169 7f14a7fff700 1 --2- 172.21.2.102:0/3026859009 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f14a0009510 0x7f14a000ba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:29.177 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:29.169 7f14a7fff700 1 -- 172.21.2.102:0/3026859009 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14a00502f0 con 0x7f14a0003b90 2021-02-10T11:18:29.178 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:29.169 7f14a7fff700 1 -- 172.21.2.102:0/3026859009 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14a00504d0 con 0x7f14a0009510 2021-02-10T11:18:29.178 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:29.169 7f14a7fff700 1 -- 172.21.2.102:0/3026859009 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14a00506f0 con 0x7f14a00067a0 2021-02-10T11:18:29.178 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:29.169 7f14b9321700 1 --2- 172.21.2.102:0/3026859009 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f14a0003b90 0x7f14a0005fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:18:29.178 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:29.169 7f14b9b22700 1 --2- 172.21.2.102:0/3026859009 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f14a0009510 0x7f14a000ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:18:29.178 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:29.169 7f14b8b20700 1 --2- 172.21.2.102:0/3026859009 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f14a00067a0 0x7f14a0008c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:18:29.178 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:29.169 7f14b9321700 1 -- 172.21.2.102:0/3026859009 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f14a00067a0 msgr2=0x7f14a0008c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:18:29.178 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:29.169 7f14b9321700 1 --2- 172.21.2.102:0/3026859009 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f14a00067a0 0x7f14a0008c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:29.178 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:29.169 7f14b9321700 1 -- 172.21.2.102:0/3026859009 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f14a0009510 msgr2=0x7f14a000ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:18:29.178 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:29.169 7f14b9321700 1 --2- 172.21.2.102:0/3026859009 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f14a0009510 0x7f14a000ba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:29.178 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:29.169 7f14b9321700 1 -- 172.21.2.102:0/3026859009 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14a0050990 con 0x7f14a0003b90 2021-02-10T11:18:29.178 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:29.169 7f14b9b22700 1 --2- 172.21.2.102:0/3026859009 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f14a0009510 0x7f14a000ba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:18:29.178 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:29.169 7f14b9321700 1 --2- 172.21.2.102:0/3026859009 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f14a0003b90 0x7f14a0005fa0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 rx=0x7f14b0009310 tx=0x7f14b0009a90).ready entity=mon.0 client_cookie=661f2fe57a87f9d6 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:18:29.179 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:29.169 7f14a7fff700 1 -- 172.21.2.102:0/3026859009 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f14b000c0c0 con 0x7f14a0003b90 2021-02-10T11:18:29.179 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:29.169 7f14a7fff700 1 -- 172.21.2.102:0/3026859009 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14b0006400 con 0x7f14a0003b90 2021-02-10T11:18:29.179 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:29.169 7f14a7fff700 1 -- 172.21.2.102:0/3026859009 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f14b000c0c0 con 0x7f14a0003b90 2021-02-10T11:18:29.179 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:29.170 7f14bb505700 1 -- 172.21.2.102:0/3026859009 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f14a0003b90 msgr2=0x7f14a0005fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:18:29.179 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:29.170 7f14bb505700 1 --2- 172.21.2.102:0/3026859009 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f14a0003b90 0x7f14a0005fa0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 rx=0x7f14b0009310 tx=0x7f14b0009a90).stop 2021-02-10T11:18:29.179 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:29.170 7f14b9b22700 1 -- 172.21.2.102:0/3026859009 reap_dead start 2021-02-10T11:18:29.179 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:29.170 7f14bb505700 1 -- 172.21.2.102:0/3026859009 shutdown_connections 2021-02-10T11:18:29.179 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:29.170 7f14bb505700 1 -- 172.21.2.102:0/3026859009 >> 172.21.2.102:0/3026859009 conn(0x7f14b40ce3e0 msgr2=0x7f14b40e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:18:29.179 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:29.170 7f14bb505700 1 -- 172.21.2.102:0/3026859009 shutdown_connections 2021-02-10T11:18:29.179 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:29.170 7f14bb505700 1 -- 172.21.2.102:0/3026859009 wait complete. 2021-02-10T11:18:29.179 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:29.185 7f14bb505700 1 Processor -- start 2021-02-10T11:18:29.179 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:29.185 7f14bb505700 1 -- start start 2021-02-10T11:18:29.181 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:29.185 7f14bb505700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f14b409e560 0x7f14b40a0130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:29.181 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:29.185 7f14bb505700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f14b409e960 0x7f14b40a0750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:29.181 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:29.185 7f14bb505700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f14b40a0d70 0x7f14b40a1af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:29.181 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:29.185 7f14bb505700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14b40c38f0 con 0x7f14b409e960 2021-02-10T11:18:29.181 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:29.185 7f14bb505700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14b40a1170 con 0x7f14b409e560 2021-02-10T11:18:29.181 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:29.185 7f14bb505700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14b40a1390 con 0x7f14b40a0d70 2021-02-10T11:18:29.181 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:29.186 7f14b8b20700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f14b409e960 0x7f14b40a0750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:18:29.181 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:29.186 7f14b8b20700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f14b409e960 0x7f14b40a0750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:53356/0 (socket says 172.21.2.102:53356) 2021-02-10T11:18:29.182 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:29.186 7f14b8b20700 1 -- 172.21.2.102:0/955945318 learned_addr learned my addr 172.21.2.102:0/955945318 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:18:29.182 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:29.186 7f14b9321700 1 --2- 172.21.2.102:0/955945318 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f14b409e560 0x7f14b40a0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:18:29.182 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:29.186 7f14b9b22700 1 --2- 172.21.2.102:0/955945318 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f14b40a0d70 0x7f14b40a1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:18:29.182 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:29.186 7f14b8b20700 1 -- 172.21.2.102:0/955945318 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f14b40a0d70 msgr2=0x7f14b40a1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:18:29.182 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:29.186 7f14b8b20700 1 --2- 172.21.2.102:0/955945318 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f14b40a0d70 0x7f14b40a1af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:29.182 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:29.186 7f14b8b20700 1 -- 172.21.2.102:0/955945318 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f14b409e560 msgr2=0x7f14b40a0130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:18:29.182 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:29.186 7f14b8b20700 1 --2- 172.21.2.102:0/955945318 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f14b409e560 0x7f14b40a0130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:29.182 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:29.186 7f14b8b20700 1 -- 172.21.2.102:0/955945318 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14b40a2110 con 0x7f14b409e960 2021-02-10T11:18:29.182 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:29.186 7f14b8b20700 1 --2- 172.21.2.102:0/955945318 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f14b409e960 0x7f14b40a0750 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 rx=0x7f14a8006940 tx=0x7f14a800b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:18:29.182 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:29.186 7f14a5ffb700 1 -- 172.21.2.102:0/955945318 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f14a800bd80 con 0x7f14b409e960 2021-02-10T11:18:29.182 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:29.186 7f14a5ffb700 1 -- 172.21.2.102:0/955945318 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14a8000c10 con 0x7f14b409e960 2021-02-10T11:18:29.182 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:29.186 7f14bb505700 1 -- 172.21.2.102:0/955945318 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14b40a2110 con 0x7f14b409e960 2021-02-10T11:18:29.183 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:29.186 7f14a5ffb700 1 -- 172.21.2.102:0/955945318 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f14a80022a0 con 0x7f14b409e960 2021-02-10T11:18:29.183 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:29.187 7f14bb505700 1 -- 172.21.2.102:0/955945318 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14b40a26a0 con 0x7f14b409e960 2021-02-10T11:18:29.183 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:29.187 7f14a5ffb700 1 -- 172.21.2.102:0/955945318 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f14a80052e0 con 0x7f14b409e960 2021-02-10T11:18:29.183 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:29.187 7f14a5ffb700 1 --2- 172.21.2.102:0/955945318 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f14940186c0 0x7f149401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:29.183 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:29.187 7f14a5ffb700 1 -- 172.21.2.102:0/955945318 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f14a8032430 con 0x7f14b409e960 2021-02-10T11:18:29.183 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:29.187 7f14b9321700 1 --2- 172.21.2.102:0/955945318 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f14940186c0 0x7f149401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:18:29.183 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:29.188 7f14b9321700 1 --2- 172.21.2.102:0/955945318 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f14940186c0 0x7f149401abc0 crc :-1 s=READY pgs=350 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:18:29.208 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:29.217 7f14bb505700 1 -- 172.21.2.102:0/955945318 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14b4105b30 con 0x7f14b409e960 2021-02-10T11:18:29.210 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:29.220 7f14a5ffb700 1 -- 172.21.2.102:0/955945318 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f14a8006b40 con 0x7f14b409e960 2021-02-10T11:18:29.332 DEBUG:teuthology.orchestra.run.gibba002:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:18:29.335 DEBUG:teuthology.orchestra.run.gibba013:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:18:29.341 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:29.350 7f14bb505700 1 -- 172.21.2.102:0/955945318 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344450", "target": ["mgr", ""]}) v1 -- 0x7f14b40a26a0 con 0x7f14940186c0 2021-02-10T11:18:29.344 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:29.353 7f14a5ffb700 1 -- 172.21.2.102:0/955945318 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f14b00063a0 con 0x7f14940186c0 2021-02-10T11:18:29.344 INFO:teuthology.orchestra.run.gibba002.stdout:/volumes/_nogroup/subvolume_0000000000344450/edfaba3e-a276-4e83-85e2-3ec937307d02 2021-02-10T11:18:29.346 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:29.355 7f14bb505700 1 -- 172.21.2.102:0/955945318 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f14940186c0 msgr2=0x7f149401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:18:29.346 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:29.355 7f14bb505700 1 --2- 172.21.2.102:0/955945318 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f14940186c0 0x7f149401abc0 crc :-1 s=READY pgs=350 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:29.346 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:29.355 7f14bb505700 1 -- 172.21.2.102:0/955945318 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f14b409e960 msgr2=0x7f14b40a0750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:18:29.346 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:29.355 7f14bb505700 1 --2- 172.21.2.102:0/955945318 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f14b409e960 0x7f14b40a0750 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 rx=0x7f14a8006940 tx=0x7f14a800b990).stop 2021-02-10T11:18:29.346 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:29.355 7f14bb505700 1 -- 172.21.2.102:0/955945318 shutdown_connections 2021-02-10T11:18:29.346 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:29.355 7f14bb505700 1 --2- 172.21.2.102:0/955945318 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f14b40a0d70 0x7f14b40a1af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:29.347 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:29.355 7f14bb505700 1 --2- 172.21.2.102:0/955945318 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f14b409e960 0x7f14b40a0750 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:29.347 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:29.355 7f14bb505700 1 --2- 172.21.2.102:0/955945318 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f14940186c0 0x7f149401abc0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:29.347 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:29.355 7f14bb505700 1 --2- 172.21.2.102:0/955945318 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f14b409e560 0x7f14b40a0130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:29.347 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:29.355 7f14bb505700 1 -- 172.21.2.102:0/955945318 >> 172.21.2.102:0/955945318 conn(0x7f14b409fc10 msgr2=0x7f14b4094ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:18:29.347 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:29.355 7f14bb505700 1 -- 172.21.2.102:0/955945318 shutdown_connections 2021-02-10T11:18:29.347 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:29.356 7f14bb505700 1 -- 172.21.2.102:0/955945318 wait complete. 2021-02-10T11:18:29.354 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000344450 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000344450/edfaba3e-a276-4e83-85e2-3ec937307d02 2021-02-10T11:18:29.354 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:18:29.354 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:18:29.355 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:18:29.669 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:18:29.669 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:18:29.669 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344450/edfaba3e-a276-4e83-85e2-3ec937307d02/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:18:30.050 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:18:30.051 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:18:30.051 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0244257 s, 42.9 MB/s 2021-02-10T11:18:30.052 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:18:30.052 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:18:30.052 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:18:30.389 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:18:30.389 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:18:30.389 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344450/edfaba3e-a276-4e83-85e2-3ec937307d02/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:18:30.766 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:18:30.767 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:18:30.767 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0247123 s, 42.4 MB/s 2021-02-10T11:18:30.768 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:18:30.768 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:18:30.768 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:18:31.106 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:18:31.106 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:18:31.106 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344450/edfaba3e-a276-4e83-85e2-3ec937307d02/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:18:31.483 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:18:31.483 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:18:31.483 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0240642 s, 43.6 MB/s 2021-02-10T11:18:31.485 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:18:31.485 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:18:31.485 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:18:31.822 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:18:31.823 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:18:31.823 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344450/edfaba3e-a276-4e83-85e2-3ec937307d02/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:18:32.199 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:18:33.836 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:18:33.836 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0243541 s, 43.1 MB/s 2021-02-10T11:18:33.837 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:18:33.838 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:18:33.838 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:18:34.173 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:18:34.174 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:18:34.174 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344450/edfaba3e-a276-4e83-85e2-3ec937307d02/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:18:34.547 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:18:34.547 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:18:34.548 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0240837 s, 43.5 MB/s 2021-02-10T11:18:34.548 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:18:34.549 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:18:34.549 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:18:34.885 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:18:34.886 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:18:34.886 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344450/edfaba3e-a276-4e83-85e2-3ec937307d02/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:18:35.264 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:18:35.264 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:18:35.264 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0255712 s, 41.0 MB/s 2021-02-10T11:18:35.265 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:18:35.265 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:18:35.265 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:18:35.601 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:18:35.602 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:18:35.602 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344450/edfaba3e-a276-4e83-85e2-3ec937307d02/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:18:35.979 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:18:35.979 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:18:35.979 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0247541 s, 42.4 MB/s 2021-02-10T11:18:35.980 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:18:35.980 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:18:35.980 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:18:36.316 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:18:36.316 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:18:36.317 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344450/edfaba3e-a276-4e83-85e2-3ec937307d02/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:18:36.692 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:18:36.692 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:18:36.692 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0236993 s, 44.2 MB/s 2021-02-10T11:18:36.693 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:18:36.693 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:18:36.693 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:18:37.028 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:18:37.028 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:18:37.029 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344450/edfaba3e-a276-4e83-85e2-3ec937307d02/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:18:37.404 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:18:37.405 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:18:37.405 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.024174 s, 43.4 MB/s 2021-02-10T11:18:37.406 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:18:37.407 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:18:37.407 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:18:37.749 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:18:37.749 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:18:37.750 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344450/edfaba3e-a276-4e83-85e2-3ec937307d02/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:18:38.127 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:18:38.128 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:18:38.128 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0241224 s, 43.5 MB/s 2021-02-10T11:18:38.129 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000344451 2021-02-10T11:18:38.577 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:38.569 7ffa9d978700 1 Processor -- start 2021-02-10T11:18:38.577 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:38.569 7ffa9d978700 1 -- start start 2021-02-10T11:18:38.577 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:38.570 7ffa9d978700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7ffa981b0470 0x7ffa981b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:38.577 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:38.570 7ffa9d978700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7ffa980ef610 0x7ffa980efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:38.578 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:38.570 7ffa9d978700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa980f0b50 con 0x7ffa980f0030 2021-02-10T11:18:38.578 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:38.570 7ffa9d978700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa980e52e0 con 0x7ffa980ef610 2021-02-10T11:18:38.578 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:38.570 7ffa9d978700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa980e5490 con 0x7ffa981b0470 2021-02-10T11:18:38.578 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:38.570 7ffa96d73700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7ffa980ef610 0x7ffa980efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:18:38.578 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:38.570 7ffa96d73700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7ffa980ef610 0x7ffa980efa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:53372/0 (socket says 172.21.2.102:53372) 2021-02-10T11:18:38.578 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:38.570 7ffa96d73700 1 -- 172.21.2.102:0/1058376286 learned_addr learned my addr 172.21.2.102:0/1058376286 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:18:38.578 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:38.571 7ffa96d73700 1 -- 172.21.2.102:0/1058376286 >> v2:172.21.2.113:3300/0 conn(0x7ffa981b0470 msgr2=0x7ffa981b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:18:38.578 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:38.571 7ffa96d73700 1 --2- 172.21.2.102:0/1058376286 >> v2:172.21.2.113:3300/0 conn(0x7ffa981b0470 0x7ffa981b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:18:38.578 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:38.571 7ffa96d73700 1 -- 172.21.2.102:0/1058376286 >> v1:172.21.2.102:6789/0 conn(0x7ffa980f0030 legacy=0x7ffa980f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:18:38.579 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:38.571 7ffa96d73700 1 -- 172.21.2.102:0/1058376286 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa980f0430 con 0x7ffa980ef610 2021-02-10T11:18:38.579 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:38.571 7ffa96d73700 1 --2- 172.21.2.102:0/1058376286 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ffa980ef610 0x7ffa980efa10 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 rx=0x7ffa8c006700 tx=0x7ffa8c009b00).ready entity=mon.0 client_cookie=a45c357fb20a6b23 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:18:38.579 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:38.571 7ffa96572700 1 -- 172.21.2.102:0/1058376286 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ffa8c00c0c0 con 0x7ffa980ef610 2021-02-10T11:18:38.579 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:38.571 7ffa96572700 1 -- 172.21.2.102:0/1058376286 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa8c00c400 con 0x7ffa980ef610 2021-02-10T11:18:38.579 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:38.571 7ffa96572700 1 -- 172.21.2.102:0/1058376286 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ffa8c00c860 con 0x7ffa980ef610 2021-02-10T11:18:38.580 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:38.571 7ffa9d978700 1 -- 172.21.2.102:0/1058376286 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ffa980ef610 msgr2=0x7ffa980efa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:18:38.580 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:38.571 7ffa9d978700 1 --2- 172.21.2.102:0/1058376286 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ffa980ef610 0x7ffa980efa10 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 rx=0x7ffa8c006700 tx=0x7ffa8c009b00).stop 2021-02-10T11:18:38.580 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:38.572 7ffa9d978700 1 -- 172.21.2.102:0/1058376286 shutdown_connections 2021-02-10T11:18:38.580 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:38.572 7ffa9d978700 1 --2- 172.21.2.102:0/1058376286 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ffa980ef610 0x7ffa980efa10 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:38.580 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:38.572 7ffa9d978700 1 --2- 172.21.2.102:0/1058376286 >> v2:172.21.2.113:3300/0 conn(0x7ffa981b0470 0x7ffa981b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:18:38.580 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:38.572 7ffa9d978700 1 -- 172.21.2.102:0/1058376286 >> 172.21.2.102:0/1058376286 conn(0x7ffa980ce3e0 msgr2=0x7ffa980e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:18:38.580 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:38.572 7ffa9d978700 1 -- 172.21.2.102:0/1058376286 shutdown_connections 2021-02-10T11:18:38.580 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:38.572 7ffa9d978700 1 -- 172.21.2.102:0/1058376286 wait complete. 2021-02-10T11:18:38.580 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:38.586 7ffa9d978700 1 Processor -- start 2021-02-10T11:18:38.580 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:38.586 7ffa9d978700 1 -- start start 2021-02-10T11:18:38.580 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:38.587 7ffa9d978700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ffa980c36a0 0x7ffa980c2b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:38.580 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:38.587 7ffa9d978700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ffa980c3190 0x7ffa980b1d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:38.581 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:38.587 7ffa9d978700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ffa980b2370 0x7ffa980b2d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:38.581 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:38.587 7ffa9d978700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa980b27c0 con 0x7ffa980c36a0 2021-02-10T11:18:38.581 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:38.587 7ffa9d978700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa980b3360 con 0x7ffa980b2370 2021-02-10T11:18:38.581 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:38.587 7ffa9d978700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa980b3540 con 0x7ffa980c3190 2021-02-10T11:18:38.581 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:38.587 7ffa97574700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ffa980c36a0 0x7ffa980c2b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:18:38.581 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:38.587 7ffa97574700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ffa980c36a0 0x7ffa980c2b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:53376/0 (socket says 172.21.2.102:53376) 2021-02-10T11:18:38.581 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:38.587 7ffa97574700 1 -- 172.21.2.102:0/686758117 learned_addr learned my addr 172.21.2.102:0/686758117 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:18:38.582 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:38.587 7ffa97574700 1 -- 172.21.2.102:0/686758117 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ffa980c3190 msgr2=0x7ffa980b1d50 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:18:38.582 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:38.587 7ffa97574700 1 --2- 172.21.2.102:0/686758117 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ffa980c3190 0x7ffa980b1d50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:18:38.582 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:38.587 7ffa97574700 1 -- 172.21.2.102:0/686758117 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ffa980b2370 msgr2=0x7ffa980b2d40 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:18:38.583 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:38.587 7ffa97574700 1 --2- 172.21.2.102:0/686758117 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ffa980b2370 0x7ffa980b2d40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:18:38.583 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:38.587 7ffa97574700 1 -- 172.21.2.102:0/686758117 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa980b3760 con 0x7ffa980c36a0 2021-02-10T11:18:38.583 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:38.588 7ffa97574700 1 --2- 172.21.2.102:0/686758117 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ffa980c36a0 0x7ffa980c2b70 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 rx=0x7ffa88006f00 tx=0x7ffa8800b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:18:38.584 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:38.588 7ffa7ffff700 1 -- 172.21.2.102:0/686758117 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ffa8800bd30 con 0x7ffa980c36a0 2021-02-10T11:18:38.584 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:38.588 7ffa7ffff700 1 -- 172.21.2.102:0/686758117 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa88000c90 con 0x7ffa980c36a0 2021-02-10T11:18:38.584 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:38.588 7ffa7ffff700 1 -- 172.21.2.102:0/686758117 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ffa880022a0 con 0x7ffa980c36a0 2021-02-10T11:18:38.584 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:38.588 7ffa9d978700 1 -- 172.21.2.102:0/686758117 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa980b3760 con 0x7ffa980c36a0 2021-02-10T11:18:38.584 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:38.588 7ffa9d978700 1 -- 172.21.2.102:0/686758117 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa980b3cf0 con 0x7ffa980c36a0 2021-02-10T11:18:38.584 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:38.588 7ffa7ffff700 1 -- 172.21.2.102:0/686758117 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ffa88005310 con 0x7ffa980c36a0 2021-02-10T11:18:38.584 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:38.589 7ffa7ffff700 1 --2- 172.21.2.102:0/686758117 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ffa780186c0 0x7ffa7801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:38.584 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:38.589 7ffa7ffff700 1 -- 172.21.2.102:0/686758117 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7ffa88032560 con 0x7ffa980c36a0 2021-02-10T11:18:38.584 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:38.589 7ffa96d73700 1 --2- 172.21.2.102:0/686758117 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ffa780186c0 0x7ffa7801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:18:38.584 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:38.589 7ffa96d73700 1 --2- 172.21.2.102:0/686758117 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ffa780186c0 0x7ffa7801abc0 crc :-1 s=READY pgs=359 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:18:38.611 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:38.620 7ffa9d978700 1 -- 172.21.2.102:0/686758117 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa980edd10 con 0x7ffa980c36a0 2021-02-10T11:18:38.613 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:38.622 7ffa7ffff700 1 -- 172.21.2.102:0/686758117 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ffa88000960 con 0x7ffa980c36a0 2021-02-10T11:18:38.741 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:38.750 7ffa9d978700 1 -- 172.21.2.102:0/686758117 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344451", "target": ["mgr", ""]}) v1 -- 0x7ffa980b27c0 con 0x7ffa780186c0 2021-02-10T11:18:38.768 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:38.777 7ffa7ffff700 1 -- 172.21.2.102:0/686758117 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7ffa8c009440 con 0x7ffa780186c0 2021-02-10T11:18:38.774 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:38.783 7ffa9d978700 1 -- 172.21.2.102:0/686758117 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ffa780186c0 msgr2=0x7ffa7801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:18:38.775 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:38.783 7ffa9d978700 1 --2- 172.21.2.102:0/686758117 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ffa780186c0 0x7ffa7801abc0 crc :-1 s=READY pgs=359 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:38.775 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:38.783 7ffa9d978700 1 -- 172.21.2.102:0/686758117 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ffa980c36a0 msgr2=0x7ffa980c2b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:18:38.775 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:38.783 7ffa9d978700 1 --2- 172.21.2.102:0/686758117 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ffa980c36a0 0x7ffa980c2b70 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 rx=0x7ffa88006f00 tx=0x7ffa8800b940).stop 2021-02-10T11:18:38.775 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:38.783 7ffa9d978700 1 -- 172.21.2.102:0/686758117 shutdown_connections 2021-02-10T11:18:38.775 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:38.783 7ffa9d978700 1 --2- 172.21.2.102:0/686758117 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ffa780186c0 0x7ffa7801abc0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:38.775 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:38.783 7ffa9d978700 1 --2- 172.21.2.102:0/686758117 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ffa980b2370 0x7ffa980b2d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:18:38.775 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:38.783 7ffa9d978700 1 --2- 172.21.2.102:0/686758117 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ffa980c3190 0x7ffa980b1d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:18:38.775 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:38.783 7ffa9d978700 1 --2- 172.21.2.102:0/686758117 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ffa980c36a0 0x7ffa980c2b70 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:38.775 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:38.783 7ffa9d978700 1 -- 172.21.2.102:0/686758117 >> 172.21.2.102:0/686758117 conn(0x7ffa9809f630 msgr2=0x7ffa980a0130 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:18:38.775 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:38.784 7ffa9d978700 1 -- 172.21.2.102:0/686758117 shutdown_connections 2021-02-10T11:18:38.776 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:38.784 7ffa9d978700 1 -- 172.21.2.102:0/686758117 wait complete. 2021-02-10T11:18:38.786 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000344451 2021-02-10T11:18:39.227 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:39.215 7f16bbaa8700 1 Processor -- start 2021-02-10T11:18:39.227 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:39.216 7f16bbaa8700 1 -- start start 2021-02-10T11:18:39.227 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:39.216 7f16bbaa8700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f16b41b0470 0x7f16b41b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:39.227 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:39.216 7f16bbaa8700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f16b40ef610 0x7f16b40efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:39.227 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:39.216 7f16bbaa8700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16b40f0b50 con 0x7f16b40f0030 2021-02-10T11:18:39.227 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:39.216 7f16bbaa8700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16b40e52e0 con 0x7f16b40ef610 2021-02-10T11:18:39.227 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:39.216 7f16bbaa8700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16b40e5490 con 0x7f16b41b0470 2021-02-10T11:18:39.228 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:39.217 7f16ba0c5700 1 --1- >> v1:172.21.2.113:6789/0 conn(0x7f16b40f0030 0x7f16b40f10d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.113:6789/0 says I am v1:172.21.2.102:52932/0 (socket says 172.21.2.102:52932) 2021-02-10T11:18:39.228 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:39.217 7f16ba0c5700 1 -- 172.21.2.102:0/1751403187 learned_addr learned my addr 172.21.2.102:0/1751403187 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:18:39.228 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:39.217 7f16b90c3700 1 --2- 172.21.2.102:0/1751403187 >> v2:172.21.2.113:3300/0 conn(0x7f16b40ef610 0x7f16b40efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:18:39.228 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:39.217 7f16b98c4700 1 --2- 172.21.2.102:0/1751403187 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f16b41b0470 0x7f16b41b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:18:39.228 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:39.218 7f16b98c4700 1 -- 172.21.2.102:0/1751403187 >> v2:172.21.2.113:3300/0 conn(0x7f16b40ef610 msgr2=0x7f16b40efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:18:39.228 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:39.218 7f16b98c4700 1 --2- 172.21.2.102:0/1751403187 >> v2:172.21.2.113:3300/0 conn(0x7f16b40ef610 0x7f16b40efa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:39.228 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:39.218 7f16b98c4700 1 -- 172.21.2.102:0/1751403187 >> v1:172.21.2.113:6789/0 conn(0x7f16b40f0030 legacy=0x7f16b40f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:18:39.228 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:39.218 7f16b98c4700 1 -- 172.21.2.102:0/1751403187 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16b40f0430 con 0x7f16b41b0470 2021-02-10T11:18:39.228 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:39.218 7f16b90c3700 1 --2- 172.21.2.102:0/1751403187 >> v2:172.21.2.113:3300/0 conn(0x7f16b40ef610 0x7f16b40efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:18:39.229 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:39.218 7f16b98c4700 1 --2- 172.21.2.102:0/1751403187 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f16b41b0470 0x7f16b41b0870 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7f16a8006650 tx=0x7f16a8009a80).ready entity=mon.2 client_cookie=3afb7479394a976c server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:18:39.229 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:39.218 7f16b88c2700 1 -- 172.21.2.102:0/1751403187 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f16a800c0c0 con 0x7f16b41b0470 2021-02-10T11:18:39.229 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:39.218 7f16b88c2700 1 -- 172.21.2.102:0/1751403187 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16a800c340 con 0x7f16b41b0470 2021-02-10T11:18:39.229 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:39.218 7f16b88c2700 1 -- 172.21.2.102:0/1751403187 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f16a800c860 con 0x7f16b41b0470 2021-02-10T11:18:39.229 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:39.218 7f16bbaa8700 1 -- 172.21.2.102:0/1751403187 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f16b41b0470 msgr2=0x7f16b41b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:18:39.229 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:39.218 7f16bbaa8700 1 --2- 172.21.2.102:0/1751403187 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f16b41b0470 0x7f16b41b0870 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7f16a8006650 tx=0x7f16a8009a80).stop 2021-02-10T11:18:39.229 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:39.219 7f16bbaa8700 1 -- 172.21.2.102:0/1751403187 shutdown_connections 2021-02-10T11:18:39.229 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:39.219 7f16bbaa8700 1 --2- 172.21.2.102:0/1751403187 >> v2:172.21.2.113:3300/0 conn(0x7f16b40ef610 0x7f16b40efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:39.229 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:39.219 7f16bbaa8700 1 --2- 172.21.2.102:0/1751403187 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f16b41b0470 0x7f16b41b0870 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:39.229 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:39.219 7f16bbaa8700 1 -- 172.21.2.102:0/1751403187 >> 172.21.2.102:0/1751403187 conn(0x7f16b40ce3e0 msgr2=0x7f16b40e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:18:39.230 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:39.219 7f16bbaa8700 1 -- 172.21.2.102:0/1751403187 shutdown_connections 2021-02-10T11:18:39.230 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:39.219 7f16bbaa8700 1 -- 172.21.2.102:0/1751403187 wait complete. 2021-02-10T11:18:39.230 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:39.234 7f16bbaa8700 1 Processor -- start 2021-02-10T11:18:39.230 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:39.234 7f16bbaa8700 1 -- start start 2021-02-10T11:18:39.230 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:39.234 7f16bbaa8700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f16b40a0420 0x7f16b40a0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:39.230 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:39.234 7f16bbaa8700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f16b4189d00 0x7f16b418a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:39.230 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:39.234 7f16bbaa8700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f16b418a720 0x7f16b40b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:39.230 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:39.234 7f16bbaa8700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16b4189780 con 0x7f16b40a0420 2021-02-10T11:18:39.230 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:39.234 7f16bbaa8700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16b418ab20 con 0x7f16b418a720 2021-02-10T11:18:39.231 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:39.234 7f16bbaa8700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16b40a1140 con 0x7f16b4189d00 2021-02-10T11:18:39.231 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:39.234 7f16b98c4700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f16b40a0420 0x7f16b40a0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:18:39.231 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:39.234 7f16b98c4700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f16b40a0420 0x7f16b40a0820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:53388/0 (socket says 172.21.2.102:53388) 2021-02-10T11:18:39.231 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:39.235 7f16b98c4700 1 -- 172.21.2.102:0/2447545237 learned_addr learned my addr 172.21.2.102:0/2447545237 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:18:39.231 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:39.235 7f16b98c4700 1 -- 172.21.2.102:0/2447545237 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f16b4189d00 msgr2=0x7f16b418a100 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:18:39.231 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:39.235 7f16b98c4700 1 --2- 172.21.2.102:0/2447545237 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f16b4189d00 0x7f16b418a100 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:18:39.231 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:39.235 7f16b98c4700 1 -- 172.21.2.102:0/2447545237 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f16b418a720 msgr2=0x7f16b40b7be0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:18:39.232 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:39.235 7f16b98c4700 1 --2- 172.21.2.102:0/2447545237 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f16b418a720 0x7f16b40b7be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:18:39.232 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:39.235 7f16b98c4700 1 -- 172.21.2.102:0/2447545237 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16b40b8200 con 0x7f16b40a0420 2021-02-10T11:18:39.232 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:39.235 7f16b98c4700 1 --2- 172.21.2.102:0/2447545237 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f16b40a0420 0x7f16b40a0820 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 rx=0x7f16a8006e90 tx=0x7f16a800b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:18:39.232 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:39.235 7f16a67fc700 1 -- 172.21.2.102:0/2447545237 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f16a80009b0 con 0x7f16b40a0420 2021-02-10T11:18:39.232 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:39.235 7f16a67fc700 1 -- 172.21.2.102:0/2447545237 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16a800bd80 con 0x7f16b40a0420 2021-02-10T11:18:39.232 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:39.235 7f16bbaa8700 1 -- 172.21.2.102:0/2447545237 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16b40b8200 con 0x7f16b40a0420 2021-02-10T11:18:39.232 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:39.235 7f16a67fc700 1 -- 172.21.2.102:0/2447545237 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f16a80054d0 con 0x7f16b40a0420 2021-02-10T11:18:39.232 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:39.235 7f16bbaa8700 1 -- 172.21.2.102:0/2447545237 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16b40b8200 con 0x7f16b40a0420 2021-02-10T11:18:39.233 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:39.236 7f16a67fc700 1 -- 172.21.2.102:0/2447545237 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f16a80052e0 con 0x7f16b40a0420 2021-02-10T11:18:39.233 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:39.236 7f16a67fc700 1 --2- 172.21.2.102:0/2447545237 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f169c018730 0x7f169c01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:39.233 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:39.236 7f16a67fc700 1 -- 172.21.2.102:0/2447545237 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f16a8032810 con 0x7f16b40a0420 2021-02-10T11:18:39.233 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:39.236 7f16b90c3700 1 --2- 172.21.2.102:0/2447545237 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f169c018730 0x7f169c01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:18:39.233 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:39.236 7f16b90c3700 1 --2- 172.21.2.102:0/2447545237 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f169c018730 0x7f169c01ac10 crc :-1 s=READY pgs=360 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:18:39.257 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:39.266 7f16bbaa8700 1 -- 172.21.2.102:0/2447545237 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16b4105ae0 con 0x7f16b40a0420 2021-02-10T11:18:39.260 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:39.268 7f16a67fc700 1 -- 172.21.2.102:0/2447545237 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f16a8006030 con 0x7f16b40a0420 2021-02-10T11:18:39.388 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:39.397 7f16bbaa8700 1 -- 172.21.2.102:0/2447545237 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344451", "target": ["mgr", ""]}) v1 -- 0x7f16b4105ae0 con 0x7f169c018730 2021-02-10T11:18:39.391 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:39.400 7f16a67fc700 1 -- 172.21.2.102:0/2447545237 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f16b0000940 con 0x7f169c018730 2021-02-10T11:18:39.395 INFO:teuthology.orchestra.run.gibba002.stdout:/volumes/_nogroup/subvolume_0000000000344451/3fa8af02-b907-4d74-be57-c83f4d3fa667 2021-02-10T11:18:39.396 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:39.406 7f16bbaa8700 1 -- 172.21.2.102:0/2447545237 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f169c018730 msgr2=0x7f169c01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:18:39.396 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:39.406 7f16bbaa8700 1 --2- 172.21.2.102:0/2447545237 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f169c018730 0x7f169c01ac10 crc :-1 s=READY pgs=360 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:39.396 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:39.406 7f16bbaa8700 1 -- 172.21.2.102:0/2447545237 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f16b40a0420 msgr2=0x7f16b40a0820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:18:39.396 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:39.406 7f16bbaa8700 1 --2- 172.21.2.102:0/2447545237 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f16b40a0420 0x7f16b40a0820 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 rx=0x7f16a8006e90 tx=0x7f16a800b990).stop 2021-02-10T11:18:39.397 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:39.406 7f16bbaa8700 1 -- 172.21.2.102:0/2447545237 shutdown_connections 2021-02-10T11:18:39.397 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:39.406 7f16bbaa8700 1 --2- 172.21.2.102:0/2447545237 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f169c018730 0x7f169c01ac10 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:39.397 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:39.406 7f16bbaa8700 1 --2- 172.21.2.102:0/2447545237 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f16b418a720 0x7f16b40b7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:18:39.397 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:39.406 7f16bbaa8700 1 --2- 172.21.2.102:0/2447545237 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f16b4189d00 0x7f16b418a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:18:39.397 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:39.406 7f16bbaa8700 1 --2- 172.21.2.102:0/2447545237 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f16b40a0420 0x7f16b40a0820 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:39.397 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:39.406 7f16bbaa8700 1 -- 172.21.2.102:0/2447545237 >> 172.21.2.102:0/2447545237 conn(0x7f16b4094e10 msgr2=0x7f16b40a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:18:39.397 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:39.406 7f16bbaa8700 1 -- 172.21.2.102:0/2447545237 shutdown_connections 2021-02-10T11:18:39.397 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:39.406 7f16bbaa8700 1 -- 172.21.2.102:0/2447545237 wait complete. 2021-02-10T11:18:39.405 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000344451 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000344451/3fa8af02-b907-4d74-be57-c83f4d3fa667 2021-02-10T11:18:39.405 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:18:39.405 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:18:39.405 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:18:39.742 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:18:39.743 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:18:39.743 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344451/3fa8af02-b907-4d74-be57-c83f4d3fa667/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:18:40.126 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:18:40.126 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:18:40.126 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0235183 s, 44.6 MB/s 2021-02-10T11:18:40.127 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:18:40.127 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:18:40.128 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:18:40.463 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:18:40.463 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:18:40.464 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344451/3fa8af02-b907-4d74-be57-c83f4d3fa667/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:18:40.841 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:18:40.842 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:18:40.842 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0250392 s, 41.9 MB/s 2021-02-10T11:18:40.843 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:18:40.843 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:18:40.843 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:18:41.178 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:18:41.178 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:18:41.179 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344451/3fa8af02-b907-4d74-be57-c83f4d3fa667/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:18:41.553 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:18:41.553 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:18:41.553 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0230276 s, 45.5 MB/s 2021-02-10T11:18:41.554 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:18:41.554 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:18:41.554 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:18:41.890 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:18:41.890 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:18:41.891 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344451/3fa8af02-b907-4d74-be57-c83f4d3fa667/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:18:42.267 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:18:42.267 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:18:42.267 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0231101 s, 45.4 MB/s 2021-02-10T11:18:42.268 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:18:42.268 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:18:42.268 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:18:42.604 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:18:42.604 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:18:42.605 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344451/3fa8af02-b907-4d74-be57-c83f4d3fa667/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:18:42.980 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:18:42.980 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:18:42.980 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0237684 s, 44.1 MB/s 2021-02-10T11:18:42.981 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:18:42.981 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:18:42.981 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:18:43.317 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:18:43.317 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:18:43.317 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344451/3fa8af02-b907-4d74-be57-c83f4d3fa667/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:18:43.693 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:18:43.693 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:18:43.693 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0235873 s, 44.5 MB/s 2021-02-10T11:18:43.694 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:18:43.694 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:18:43.694 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:18:44.029 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:18:44.030 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:18:44.031 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344451/3fa8af02-b907-4d74-be57-c83f4d3fa667/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:18:44.407 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:18:44.407 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:18:44.408 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0239882 s, 43.7 MB/s 2021-02-10T11:18:44.409 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:18:44.409 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:18:44.409 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:18:44.746 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:18:44.747 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:18:44.747 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344451/3fa8af02-b907-4d74-be57-c83f4d3fa667/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:18:45.129 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:18:45.130 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:18:45.130 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0296403 s, 35.4 MB/s 2021-02-10T11:18:45.131 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:18:45.131 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:18:45.131 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:18:45.467 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:18:45.468 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:18:45.468 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344451/3fa8af02-b907-4d74-be57-c83f4d3fa667/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:18:45.843 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:18:45.843 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:18:45.843 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0233423 s, 44.9 MB/s 2021-02-10T11:18:45.844 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:18:45.844 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:18:45.844 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:18:46.181 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:18:46.182 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:18:46.182 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344451/3fa8af02-b907-4d74-be57-c83f4d3fa667/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:18:46.560 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:18:46.560 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:18:46.560 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0254627 s, 41.2 MB/s 2021-02-10T11:18:46.561 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000344452 2021-02-10T11:18:46.996 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:46.987 7f6d89965700 1 Processor -- start 2021-02-10T11:18:46.996 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:46.987 7f6d89965700 1 -- start start 2021-02-10T11:18:46.996 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:46.988 7f6d89965700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6d841b0470 0x7f6d841b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:46.996 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:46.988 7f6d89965700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f6d840ef610 0x7f6d840efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:46.996 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:46.988 7f6d89965700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d840f0b50 con 0x7f6d840f0030 2021-02-10T11:18:46.996 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:46.988 7f6d89965700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d840e52e0 con 0x7f6d840ef610 2021-02-10T11:18:46.997 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:46.988 7f6d89965700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d840e5490 con 0x7f6d841b0470 2021-02-10T11:18:46.997 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:46.988 7f6d83d75700 1 --1- >> v1:172.21.2.113:6789/0 conn(0x7f6d840f0030 0x7f6d840f10d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.113:6789/0 says I am v1:172.21.2.102:52944/0 (socket says 172.21.2.102:52944) 2021-02-10T11:18:46.997 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:46.988 7f6d83d75700 1 -- 172.21.2.102:0/3242942363 learned_addr learned my addr 172.21.2.102:0/3242942363 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:18:46.997 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:46.988 7f6d83574700 1 --2- 172.21.2.102:0/3242942363 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6d841b0470 0x7f6d841b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:18:46.997 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:46.988 7f6d82d73700 1 --2- 172.21.2.102:0/3242942363 >> v2:172.21.2.113:3300/0 conn(0x7f6d840ef610 0x7f6d840efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:18:46.997 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:46.988 7f6d82d73700 1 -- 172.21.2.102:0/3242942363 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6d841b0470 msgr2=0x7f6d841b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:18:46.997 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:46.988 7f6d82d73700 1 --2- 172.21.2.102:0/3242942363 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6d841b0470 0x7f6d841b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:46.997 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:46.988 7f6d82d73700 1 -- 172.21.2.102:0/3242942363 >> v1:172.21.2.113:6789/0 conn(0x7f6d840f0030 legacy=0x7f6d840f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:18:46.997 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:46.988 7f6d82d73700 1 -- 172.21.2.102:0/3242942363 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d840f0430 con 0x7f6d840ef610 2021-02-10T11:18:46.997 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:46.988 7f6d83574700 1 --2- 172.21.2.102:0/3242942363 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6d841b0470 0x7f6d841b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:18:46.997 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:46.989 7f6d82d73700 1 --2- 172.21.2.102:0/3242942363 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6d840ef610 0x7f6d840efa10 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7f6d6c006650 tx=0x7f6d6c009a80).ready entity=mon.1 client_cookie=f53790ad511c6852 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:18:46.997 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:46.989 7f6d82572700 1 -- 172.21.2.102:0/3242942363 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6d6c0009b0 con 0x7f6d840ef610 2021-02-10T11:18:46.998 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:46.989 7f6d82572700 1 -- 172.21.2.102:0/3242942363 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d6c00c0c0 con 0x7f6d840ef610 2021-02-10T11:18:46.998 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:46.989 7f6d82572700 1 -- 172.21.2.102:0/3242942363 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6d6c00c520 con 0x7f6d840ef610 2021-02-10T11:18:46.998 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:46.989 7f6d89965700 1 -- 172.21.2.102:0/3242942363 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6d840ef610 msgr2=0x7f6d840efa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:18:46.998 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:46.989 7f6d89965700 1 --2- 172.21.2.102:0/3242942363 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6d840ef610 0x7f6d840efa10 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7f6d6c006650 tx=0x7f6d6c009a80).stop 2021-02-10T11:18:46.998 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:46.990 7f6d89965700 1 -- 172.21.2.102:0/3242942363 shutdown_connections 2021-02-10T11:18:46.998 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:46.990 7f6d89965700 1 --2- 172.21.2.102:0/3242942363 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6d840ef610 0x7f6d840efa10 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:46.998 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:46.990 7f6d89965700 1 --2- 172.21.2.102:0/3242942363 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6d841b0470 0x7f6d841b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:46.998 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:46.990 7f6d89965700 1 -- 172.21.2.102:0/3242942363 >> 172.21.2.102:0/3242942363 conn(0x7f6d840ce3e0 msgr2=0x7f6d840e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:18:46.998 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:46.990 7f6d89965700 1 -- 172.21.2.102:0/3242942363 shutdown_connections 2021-02-10T11:18:46.998 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:46.990 7f6d89965700 1 -- 172.21.2.102:0/3242942363 wait complete. 2021-02-10T11:18:47.000 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:47.005 7f6d89965700 1 Processor -- start 2021-02-10T11:18:47.000 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:47.005 7f6d89965700 1 -- start start 2021-02-10T11:18:47.000 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:47.005 7f6d89965700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6d84190920 0x7f6d840b81a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:47.000 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:47.005 7f6d89965700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6d840b87c0 0x7f6d840b70c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:47.001 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:47.005 7f6d89965700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6d840b8bc0 0x7f6d840b76e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:47.001 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:47.005 7f6d89965700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d84190d20 con 0x7f6d84190920 2021-02-10T11:18:47.001 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:47.005 7f6d89965700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d840b5ad0 con 0x7f6d840b87c0 2021-02-10T11:18:47.001 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:47.005 7f6d89965700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d840b7d00 con 0x7f6d840b8bc0 2021-02-10T11:18:47.001 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:47.005 7f6d83574700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6d84190920 0x7f6d840b81a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:18:47.001 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:47.005 7f6d83574700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6d84190920 0x7f6d840b81a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:53400/0 (socket says 172.21.2.102:53400) 2021-02-10T11:18:47.001 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:47.005 7f6d83574700 1 -- 172.21.2.102:0/1595320202 learned_addr learned my addr 172.21.2.102:0/1595320202 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:18:47.001 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:47.005 7f6d82d73700 1 --2- 172.21.2.102:0/1595320202 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6d840b87c0 0x7f6d840b70c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:18:47.001 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:47.005 7f6d83d75700 1 --2- 172.21.2.102:0/1595320202 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6d840b8bc0 0x7f6d840b76e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:18:47.001 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:47.006 7f6d82d73700 1 -- 172.21.2.102:0/1595320202 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6d840b8bc0 msgr2=0x7f6d840b76e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:18:47.001 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:47.006 7f6d82d73700 1 --2- 172.21.2.102:0/1595320202 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6d840b8bc0 0x7f6d840b76e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:47.002 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:47.006 7f6d82d73700 1 -- 172.21.2.102:0/1595320202 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6d84190920 msgr2=0x7f6d840b81a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:18:47.002 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:47.006 7f6d82d73700 1 --2- 172.21.2.102:0/1595320202 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6d84190920 0x7f6d840b81a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:47.002 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:47.006 7f6d82d73700 1 -- 172.21.2.102:0/1595320202 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d840a4590 con 0x7f6d840b87c0 2021-02-10T11:18:47.002 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:47.006 7f6d82d73700 1 --2- 172.21.2.102:0/1595320202 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6d840b87c0 0x7f6d840b70c0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 rx=0x7f6d6c00b530 tx=0x7f6d6c00b8d0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:18:47.002 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:47.006 7f6d7bfff700 1 -- 172.21.2.102:0/1595320202 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6d6c000a10 con 0x7f6d840b87c0 2021-02-10T11:18:47.002 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:47.006 7f6d7bfff700 1 -- 172.21.2.102:0/1595320202 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d6c00bcc0 con 0x7f6d840b87c0 2021-02-10T11:18:47.002 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:47.006 7f6d89965700 1 -- 172.21.2.102:0/1595320202 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d840a4590 con 0x7f6d840b87c0 2021-02-10T11:18:47.002 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:47.006 7f6d89965700 1 -- 172.21.2.102:0/1595320202 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d840a4590 con 0x7f6d840b87c0 2021-02-10T11:18:47.002 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:47.006 7f6d7bfff700 1 -- 172.21.2.102:0/1595320202 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6d6c002030 con 0x7f6d840b87c0 2021-02-10T11:18:47.002 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:47.007 7f6d7bfff700 1 -- 172.21.2.102:0/1595320202 <== mon.1 v2:172.21.2.113:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f6d6c009280 con 0x7f6d840b87c0 2021-02-10T11:18:47.002 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:47.007 7f6d7bfff700 1 --2- 172.21.2.102:0/1595320202 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6d68018700 0x7f6d6801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:47.003 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:47.007 7f6d7bfff700 1 -- 172.21.2.102:0/1595320202 <== mon.1 v2:172.21.2.113:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f6d6c0326e0 con 0x7f6d840b87c0 2021-02-10T11:18:47.003 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:47.007 7f6d83574700 1 --2- 172.21.2.102:0/1595320202 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6d68018700 0x7f6d6801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:18:47.003 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:47.007 7f6d83574700 1 --2- 172.21.2.102:0/1595320202 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6d68018700 0x7f6d6801abc0 crc :-1 s=READY pgs=361 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:18:47.029 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:47.037 7f6d89965700 1 -- 172.21.2.102:0/1595320202 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d84105ae0 con 0x7f6d840b87c0 2021-02-10T11:18:47.032 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:47.040 7f6d7bfff700 1 -- 172.21.2.102:0/1595320202 <== mon.1 v2:172.21.2.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f6d6c00bcc0 con 0x7f6d840b87c0 2021-02-10T11:18:47.158 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:47.166 7f6d89965700 1 -- 172.21.2.102:0/1595320202 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344452", "target": ["mgr", ""]}) v1 -- 0x7f6d84105ae0 con 0x7f6d68018700 2021-02-10T11:18:47.186 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:47.194 7f6d7bfff700 1 -- 172.21.2.102:0/1595320202 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f6d740009b0 con 0x7f6d68018700 2021-02-10T11:18:47.191 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:47.199 7f6d89965700 1 -- 172.21.2.102:0/1595320202 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6d68018700 msgr2=0x7f6d6801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:18:47.192 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:47.199 7f6d89965700 1 --2- 172.21.2.102:0/1595320202 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6d68018700 0x7f6d6801abc0 crc :-1 s=READY pgs=361 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:47.192 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:47.199 7f6d89965700 1 -- 172.21.2.102:0/1595320202 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6d840b87c0 msgr2=0x7f6d840b70c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:18:47.192 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:47.199 7f6d89965700 1 --2- 172.21.2.102:0/1595320202 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6d840b87c0 0x7f6d840b70c0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 rx=0x7f6d6c00b530 tx=0x7f6d6c00b8d0).stop 2021-02-10T11:18:47.192 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:47.200 7f6d89965700 1 -- 172.21.2.102:0/1595320202 shutdown_connections 2021-02-10T11:18:47.192 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:47.200 7f6d89965700 1 --2- 172.21.2.102:0/1595320202 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6d840b8bc0 0x7f6d840b76e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:47.192 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:47.200 7f6d89965700 1 --2- 172.21.2.102:0/1595320202 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6d68018700 0x7f6d6801abc0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:47.192 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:47.200 7f6d89965700 1 --2- 172.21.2.102:0/1595320202 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6d840b87c0 0x7f6d840b70c0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:47.192 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:47.200 7f6d89965700 1 --2- 172.21.2.102:0/1595320202 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6d84190920 0x7f6d840b81a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:47.192 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:47.200 7f6d89965700 1 -- 172.21.2.102:0/1595320202 >> 172.21.2.102:0/1595320202 conn(0x7f6d8408ffe0 msgr2=0x7f6d840bf050 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:18:47.192 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:47.200 7f6d89965700 1 -- 172.21.2.102:0/1595320202 shutdown_connections 2021-02-10T11:18:47.192 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:47.200 7f6d89965700 1 -- 172.21.2.102:0/1595320202 wait complete. 2021-02-10T11:18:47.199 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000344452 2021-02-10T11:18:47.633 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:47.625 7fc7d39d2700 1 Processor -- start 2021-02-10T11:18:47.633 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:47.625 7fc7d39d2700 1 -- start start 2021-02-10T11:18:47.633 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:47.625 7fc7d39d2700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fc7cc1b0470 0x7fc7cc1b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:47.634 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:47.625 7fc7d39d2700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fc7cc0ef610 0x7fc7cc0efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:47.634 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:47.625 7fc7d39d2700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc7cc0f0030 0x7fc7cc0f0430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:47.634 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:47.625 7fc7d39d2700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7cc0e5ea0 con 0x7fc7cc1b0470 2021-02-10T11:18:47.634 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:47.625 7fc7d39d2700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7cc0f0b50 con 0x7fc7cc0ef610 2021-02-10T11:18:47.634 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:47.625 7fc7d39d2700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7cc0e52e0 con 0x7fc7cc0f0030 2021-02-10T11:18:47.634 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:47.626 7fc7d17ee700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fc7cc1b0470 0x7fc7cc1b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:18:47.634 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:47.626 7fc7d17ee700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fc7cc1b0470 0x7fc7cc1b0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:53408/0 (socket says 172.21.2.102:53408) 2021-02-10T11:18:47.634 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:47.626 7fc7d17ee700 1 -- 172.21.2.102:0/1299138225 learned_addr learned my addr 172.21.2.102:0/1299138225 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:18:47.634 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:47.626 7fc7d17ee700 1 -- 172.21.2.102:0/1299138225 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc7cc0f0030 msgr2=0x7fc7cc0f0430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:18:47.634 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:47.626 7fc7d17ee700 1 --2- 172.21.2.102:0/1299138225 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc7cc0f0030 0x7fc7cc0f0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:18:47.635 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:47.626 7fc7d17ee700 1 -- 172.21.2.102:0/1299138225 >> v2:172.21.2.113:3300/0 conn(0x7fc7cc0ef610 msgr2=0x7fc7cc0efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:18:47.635 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:47.626 7fc7d17ee700 1 --2- 172.21.2.102:0/1299138225 >> v2:172.21.2.113:3300/0 conn(0x7fc7cc0ef610 0x7fc7cc0efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:18:47.635 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:47.626 7fc7d17ee700 1 -- 172.21.2.102:0/1299138225 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7cc0f10d0 con 0x7fc7cc1b0470 2021-02-10T11:18:47.635 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:47.626 7fc7d17ee700 1 --2- 172.21.2.102:0/1299138225 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc7cc1b0470 0x7fc7cc1b0870 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 rx=0x7fc7c0006700 tx=0x7fc7c0009b80).ready entity=mon.0 client_cookie=d537231ed8ec6f28 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:18:47.635 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:47.626 7fc7bffff700 1 -- 172.21.2.102:0/1299138225 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc7c000c0c0 con 0x7fc7cc1b0470 2021-02-10T11:18:47.635 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:47.626 7fc7bffff700 1 -- 172.21.2.102:0/1299138225 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7c000c3c0 con 0x7fc7cc1b0470 2021-02-10T11:18:47.635 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:47.626 7fc7bffff700 1 -- 172.21.2.102:0/1299138225 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc7c000c8e0 con 0x7fc7cc1b0470 2021-02-10T11:18:47.635 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:47.626 7fc7d39d2700 1 -- 172.21.2.102:0/1299138225 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc7cc1b0470 msgr2=0x7fc7cc1b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:18:47.635 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:47.626 7fc7d39d2700 1 --2- 172.21.2.102:0/1299138225 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc7cc1b0470 0x7fc7cc1b0870 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 rx=0x7fc7c0006700 tx=0x7fc7c0009b80).stop 2021-02-10T11:18:47.635 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:47.627 7fc7d39d2700 1 -- 172.21.2.102:0/1299138225 shutdown_connections 2021-02-10T11:18:47.635 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:47.627 7fc7d39d2700 1 --2- 172.21.2.102:0/1299138225 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc7cc0f0030 0x7fc7cc0f0430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:18:47.636 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:47.627 7fc7d39d2700 1 --2- 172.21.2.102:0/1299138225 >> v2:172.21.2.113:3300/0 conn(0x7fc7cc0ef610 0x7fc7cc0efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:18:47.636 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:47.627 7fc7d39d2700 1 --2- 172.21.2.102:0/1299138225 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc7cc1b0470 0x7fc7cc1b0870 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:47.636 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:47.627 7fc7d39d2700 1 -- 172.21.2.102:0/1299138225 >> 172.21.2.102:0/1299138225 conn(0x7fc7cc0ce3e0 msgr2=0x7fc7cc0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:18:47.636 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:47.627 7fc7d39d2700 1 -- 172.21.2.102:0/1299138225 shutdown_connections 2021-02-10T11:18:47.636 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:47.627 7fc7d39d2700 1 -- 172.21.2.102:0/1299138225 wait complete. 2021-02-10T11:18:47.638 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:47.642 7fc7d39d2700 1 Processor -- start 2021-02-10T11:18:47.638 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:47.642 7fc7d39d2700 1 -- start start 2021-02-10T11:18:47.638 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:47.642 7fc7d39d2700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc7cc196260 0x7fc7cc1a7850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:47.638 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:47.642 7fc7d39d2700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc7cc1a7e70 0x7fc7cc0a7ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:47.638 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:47.642 7fc7d39d2700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc7cc196660 0x7fc7cc0a84f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:47.638 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:47.642 7fc7d39d2700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7cc0a9410 con 0x7fc7cc196660 2021-02-10T11:18:47.638 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:47.642 7fc7d39d2700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7cc0a8b10 con 0x7fc7cc1a7e70 2021-02-10T11:18:47.638 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:47.642 7fc7d39d2700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7cc0a8cf0 con 0x7fc7cc196260 2021-02-10T11:18:47.638 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:47.643 7fc7d1fef700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc7cc196660 0x7fc7cc0a84f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:18:47.638 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:47.643 7fc7d1fef700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc7cc196660 0x7fc7cc0a84f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:53418/0 (socket says 172.21.2.102:53418) 2021-02-10T11:18:47.639 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:47.643 7fc7d1fef700 1 -- 172.21.2.102:0/690208225 learned_addr learned my addr 172.21.2.102:0/690208225 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:18:47.639 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:47.643 7fc7d1fef700 1 -- 172.21.2.102:0/690208225 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc7cc196260 msgr2=0x7fc7cc1a7850 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:18:47.639 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:47.643 7fc7d1fef700 1 --2- 172.21.2.102:0/690208225 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc7cc196260 0x7fc7cc1a7850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:18:47.639 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:47.643 7fc7d1fef700 1 -- 172.21.2.102:0/690208225 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc7cc1a7e70 msgr2=0x7fc7cc0a7ed0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:18:47.639 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:47.643 7fc7d1fef700 1 --2- 172.21.2.102:0/690208225 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc7cc1a7e70 0x7fc7cc0a7ed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:18:47.639 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:47.643 7fc7d1fef700 1 -- 172.21.2.102:0/690208225 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7cc0a8f10 con 0x7fc7cc196660 2021-02-10T11:18:47.639 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:47.643 7fc7d1fef700 1 --2- 172.21.2.102:0/690208225 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc7cc196660 0x7fc7cc0a84f0 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 rx=0x7fc7c4009980 tx=0x7fc7c400bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:18:47.639 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:47.643 7fc7bdffb700 1 -- 172.21.2.102:0/690208225 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc7c4005af0 con 0x7fc7cc196660 2021-02-10T11:18:47.639 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:47.643 7fc7bdffb700 1 -- 172.21.2.102:0/690208225 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7c4005d70 con 0x7fc7cc196660 2021-02-10T11:18:47.639 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:47.643 7fc7bdffb700 1 -- 172.21.2.102:0/690208225 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc7c40062b0 con 0x7fc7cc196660 2021-02-10T11:18:47.639 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:47.643 7fc7d39d2700 1 -- 172.21.2.102:0/690208225 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7cc0a8f10 con 0x7fc7cc196660 2021-02-10T11:18:47.640 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:47.643 7fc7d39d2700 1 -- 172.21.2.102:0/690208225 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7cc0a9c80 con 0x7fc7cc196660 2021-02-10T11:18:47.640 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:47.644 7fc7bdffb700 1 -- 172.21.2.102:0/690208225 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fc7c4006030 con 0x7fc7cc196660 2021-02-10T11:18:47.640 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:47.644 7fc7bdffb700 1 --2- 172.21.2.102:0/690208225 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc7ac0186c0 0x7fc7ac01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:47.640 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:47.644 7fc7bdffb700 1 -- 172.21.2.102:0/690208225 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fc7c4032900 con 0x7fc7cc196660 2021-02-10T11:18:47.640 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:47.644 7fc7d17ee700 1 --2- 172.21.2.102:0/690208225 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc7ac0186c0 0x7fc7ac01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:18:47.640 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:47.644 7fc7d17ee700 1 --2- 172.21.2.102:0/690208225 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc7ac0186c0 0x7fc7ac01abc0 crc :-1 s=READY pgs=362 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:18:47.667 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:47.675 7fc7d39d2700 1 -- 172.21.2.102:0/690208225 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7cc0a91b0 con 0x7fc7cc196660 2021-02-10T11:18:47.670 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:47.678 7fc7bdffb700 1 -- 172.21.2.102:0/690208225 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fc7c4006570 con 0x7fc7cc196660 2021-02-10T11:18:47.800 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:47.808 7fc7d39d2700 1 -- 172.21.2.102:0/690208225 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344452", "target": ["mgr", ""]}) v1 -- 0x7fc7cc0a9c80 con 0x7fc7ac0186c0 2021-02-10T11:18:47.803 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:47.812 7fc7bdffb700 1 -- 172.21.2.102:0/690208225 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7fc7c0006310 con 0x7fc7ac0186c0 2021-02-10T11:18:47.807 INFO:teuthology.orchestra.run.gibba002.stdout:/volumes/_nogroup/subvolume_0000000000344452/c5efc030-4f36-439f-90ab-21702f30f743 2021-02-10T11:18:47.808 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:47.817 7fc7d39d2700 1 -- 172.21.2.102:0/690208225 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc7ac0186c0 msgr2=0x7fc7ac01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:18:47.808 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:47.817 7fc7d39d2700 1 --2- 172.21.2.102:0/690208225 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc7ac0186c0 0x7fc7ac01abc0 crc :-1 s=READY pgs=362 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:47.809 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:47.817 7fc7d39d2700 1 -- 172.21.2.102:0/690208225 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc7cc196660 msgr2=0x7fc7cc0a84f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:18:47.809 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:47.817 7fc7d39d2700 1 --2- 172.21.2.102:0/690208225 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc7cc196660 0x7fc7cc0a84f0 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 rx=0x7fc7c4009980 tx=0x7fc7c400bcd0).stop 2021-02-10T11:18:47.809 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:47.817 7fc7d39d2700 1 -- 172.21.2.102:0/690208225 shutdown_connections 2021-02-10T11:18:47.809 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:47.817 7fc7d39d2700 1 --2- 172.21.2.102:0/690208225 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc7cc196660 0x7fc7cc0a84f0 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:47.809 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:47.817 7fc7d39d2700 1 --2- 172.21.2.102:0/690208225 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc7ac0186c0 0x7fc7ac01abc0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:47.810 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:47.817 7fc7d39d2700 1 --2- 172.21.2.102:0/690208225 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc7cc1a7e70 0x7fc7cc0a7ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:18:47.810 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:47.817 7fc7d39d2700 1 --2- 172.21.2.102:0/690208225 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc7cc196260 0x7fc7cc1a7850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:18:47.810 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:47.817 7fc7d39d2700 1 -- 172.21.2.102:0/690208225 >> 172.21.2.102:0/690208225 conn(0x7fc7cc0bf4e0 msgr2=0x7fc7cc0937f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:18:47.810 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:47.817 7fc7d39d2700 1 -- 172.21.2.102:0/690208225 shutdown_connections 2021-02-10T11:18:47.810 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:47.817 7fc7d39d2700 1 -- 172.21.2.102:0/690208225 wait complete. 2021-02-10T11:18:47.817 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000344452 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000344452/c5efc030-4f36-439f-90ab-21702f30f743 2021-02-10T11:18:47.817 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:18:47.817 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:18:47.817 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:18:48.153 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:18:48.154 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:18:48.154 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344452/c5efc030-4f36-439f-90ab-21702f30f743/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:18:48.532 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:18:48.532 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:18:48.532 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0242847 s, 43.2 MB/s 2021-02-10T11:18:48.533 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:18:48.534 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:18:48.534 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:18:48.869 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:18:48.869 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:18:48.869 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344452/c5efc030-4f36-439f-90ab-21702f30f743/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:18:49.246 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:18:49.247 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:18:49.247 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0237798 s, 44.1 MB/s 2021-02-10T11:18:49.248 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:18:49.248 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:18:49.248 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:18:49.583 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:18:49.584 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:18:49.584 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344452/c5efc030-4f36-439f-90ab-21702f30f743/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:18:49.961 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:18:49.961 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:18:49.962 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0257857 s, 40.7 MB/s 2021-02-10T11:18:49.963 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:18:49.963 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:18:49.963 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:18:50.298 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:18:50.299 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:18:50.299 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344452/c5efc030-4f36-439f-90ab-21702f30f743/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:18:50.677 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:18:50.677 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:18:50.677 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.025758 s, 40.7 MB/s 2021-02-10T11:18:50.678 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:18:50.678 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:18:50.678 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:18:51.014 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:18:51.015 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:18:51.015 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344452/c5efc030-4f36-439f-90ab-21702f30f743/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:18:51.391 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:18:51.391 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:18:51.391 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0235788 s, 44.5 MB/s 2021-02-10T11:18:51.392 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:18:51.392 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:18:51.392 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:18:51.727 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:18:51.728 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:18:51.728 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344452/c5efc030-4f36-439f-90ab-21702f30f743/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:18:52.104 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:18:52.104 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:18:52.104 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0225581 s, 46.5 MB/s 2021-02-10T11:18:52.105 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:18:52.105 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:18:52.105 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:18:52.441 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:18:52.442 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:18:52.442 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344452/c5efc030-4f36-439f-90ab-21702f30f743/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:18:52.821 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:18:52.821 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:18:52.821 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0252586 s, 41.5 MB/s 2021-02-10T11:18:52.822 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:18:52.822 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:18:52.822 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:18:53.158 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:18:53.158 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:18:53.158 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344452/c5efc030-4f36-439f-90ab-21702f30f743/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:18:53.532 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:18:53.533 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:18:53.533 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0250487 s, 41.9 MB/s 2021-02-10T11:18:53.534 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:18:53.534 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:18:53.535 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:18:53.870 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:18:53.871 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:18:53.871 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344452/c5efc030-4f36-439f-90ab-21702f30f743/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:18:54.249 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:18:54.249 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:18:54.249 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0251084 s, 41.8 MB/s 2021-02-10T11:18:54.250 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:18:54.250 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:18:54.250 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:18:54.586 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:18:54.587 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:18:54.587 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344452/c5efc030-4f36-439f-90ab-21702f30f743/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:18:54.963 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:18:54.963 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:18:54.963 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0233412 s, 44.9 MB/s 2021-02-10T11:18:54.964 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000344453 2021-02-10T11:18:55.397 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:55.389 7ff4e2e6a700 1 Processor -- start 2021-02-10T11:18:55.397 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:55.389 7ff4e2e6a700 1 -- start start 2021-02-10T11:18:55.397 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:55.390 7ff4e2e6a700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7ff4dc1b0870 0x7ff4dc0f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:55.397 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:55.390 7ff4e2e6a700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4dc1b0c70 con 0x7ff4dc0efe00 2021-02-10T11:18:55.397 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:55.390 7ff4e2e6a700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4dc0f0b50 con 0x7ff4dc1b0470 2021-02-10T11:18:55.398 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:55.390 7ff4e2e6a700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4dc0e5ea0 con 0x7ff4dc1b0870 2021-02-10T11:18:55.398 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:55.390 7ff4dbfff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7ff4dc1b0870 0x7ff4dc0f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:18:55.398 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:55.390 7ff4e1487700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7ff4dc0efe00 0x7ff4dc0f0200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:45646/0 (socket says 172.21.2.102:45646) 2021-02-10T11:18:55.398 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:55.390 7ff4e1487700 1 -- 172.21.2.102:0/3488315085 learned_addr learned my addr 172.21.2.102:0/3488315085 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:18:55.398 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:55.390 7ff4dbfff700 1 -- 172.21.2.102:0/3488315085 >> v1:172.21.2.113:6789/0 conn(0x7ff4dc1b0470 legacy=0x7ff4dc0ef610 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:18:55.398 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:55.390 7ff4dbfff700 1 -- 172.21.2.102:0/3488315085 >> v1:172.21.2.102:6789/0 conn(0x7ff4dc0efe00 legacy=0x7ff4dc0f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:18:55.398 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:55.390 7ff4dbfff700 1 -- 172.21.2.102:0/3488315085 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4dc0e52e0 con 0x7ff4dc1b0870 2021-02-10T11:18:55.398 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:55.390 7ff4dbfff700 1 --2- 172.21.2.102:0/3488315085 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff4dc1b0870 0x7ff4dc0f20e0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 rx=0x7ff4cc0069d0 tx=0x7ff4cc009e40).ready entity=mon.0 client_cookie=b8f0db3135cc7bb4 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:18:55.398 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:55.391 7ff4db7fe700 1 -- 172.21.2.102:0/3488315085 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff4cc00b510 con 0x7ff4dc1b0870 2021-02-10T11:18:55.398 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:55.391 7ff4db7fe700 1 -- 172.21.2.102:0/3488315085 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4cc00b890 con 0x7ff4dc1b0870 2021-02-10T11:18:55.398 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:55.391 7ff4db7fe700 1 -- 172.21.2.102:0/3488315085 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff4cc00bd70 con 0x7ff4dc1b0870 2021-02-10T11:18:55.398 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:55.391 7ff4e2e6a700 1 -- 172.21.2.102:0/3488315085 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff4dc1b0870 msgr2=0x7ff4dc0f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:18:55.399 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:55.391 7ff4e2e6a700 1 --2- 172.21.2.102:0/3488315085 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff4dc1b0870 0x7ff4dc0f20e0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 rx=0x7ff4cc0069d0 tx=0x7ff4cc009e40).stop 2021-02-10T11:18:55.399 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:55.391 7ff4e2e6a700 1 -- 172.21.2.102:0/3488315085 shutdown_connections 2021-02-10T11:18:55.399 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:55.391 7ff4e2e6a700 1 --2- 172.21.2.102:0/3488315085 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff4dc1b0870 0x7ff4dc0f20e0 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:55.399 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:55.391 7ff4e2e6a700 1 -- 172.21.2.102:0/3488315085 >> 172.21.2.102:0/3488315085 conn(0x7ff4dc0ce3e0 msgr2=0x7ff4dc0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:18:55.399 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:55.391 7ff4e2e6a700 1 -- 172.21.2.102:0/3488315085 shutdown_connections 2021-02-10T11:18:55.399 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:55.391 7ff4e2e6a700 1 -- 172.21.2.102:0/3488315085 wait complete. 2021-02-10T11:18:55.400 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:55.406 7ff4e2e6a700 1 Processor -- start 2021-02-10T11:18:55.400 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:55.406 7ff4e2e6a700 1 -- start start 2021-02-10T11:18:55.401 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:55.406 7ff4e2e6a700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff4dc0a3b50 0x7ff4dc1b3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:55.401 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:55.406 7ff4e2e6a700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff4dc1c25b0 0x7ff4dc1b3d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:55.401 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:55.406 7ff4e2e6a700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff4dc1b4350 0x7ff4dc1b18a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:55.401 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:55.406 7ff4e2e6a700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4dc1b3190 con 0x7ff4dc1c25b0 2021-02-10T11:18:55.401 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:55.406 7ff4e2e6a700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4dc087750 con 0x7ff4dc0a3b50 2021-02-10T11:18:55.401 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:55.406 7ff4e2e6a700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4dc0a6bf0 con 0x7ff4dc1b4350 2021-02-10T11:18:55.401 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:55.407 7ff4dbfff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff4dc1c25b0 0x7ff4dc1b3d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:18:55.401 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:55.407 7ff4dbfff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff4dc1c25b0 0x7ff4dc1b3d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:53430/0 (socket says 172.21.2.102:53430) 2021-02-10T11:18:55.401 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:55.407 7ff4dbfff700 1 -- 172.21.2.102:0/2594179738 learned_addr learned my addr 172.21.2.102:0/2594179738 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:18:55.401 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:55.407 7ff4dbfff700 1 -- 172.21.2.102:0/2594179738 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff4dc1b4350 msgr2=0x7ff4dc1b18a0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:18:55.401 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:55.407 7ff4dbfff700 1 --2- 172.21.2.102:0/2594179738 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff4dc1b4350 0x7ff4dc1b18a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:18:55.401 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:55.407 7ff4dbfff700 1 -- 172.21.2.102:0/2594179738 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff4dc0a3b50 msgr2=0x7ff4dc1b3710 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:18:55.401 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:55.407 7ff4dbfff700 1 --2- 172.21.2.102:0/2594179738 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff4dc0a3b50 0x7ff4dc1b3710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:18:55.402 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:55.407 7ff4dbfff700 1 -- 172.21.2.102:0/2594179738 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4dc1b1ec0 con 0x7ff4dc1c25b0 2021-02-10T11:18:55.402 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:55.407 7ff4dbfff700 1 --2- 172.21.2.102:0/2594179738 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff4dc1c25b0 0x7ff4dc1b3d30 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 rx=0x7ff4cc009f40 tx=0x7ff4cc00bd50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:18:55.402 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:55.407 7ff4d97fa700 1 -- 172.21.2.102:0/2594179738 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff4cc006240 con 0x7ff4dc1c25b0 2021-02-10T11:18:55.402 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:55.407 7ff4d97fa700 1 -- 172.21.2.102:0/2594179738 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4cc006480 con 0x7ff4dc1c25b0 2021-02-10T11:18:55.402 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:55.407 7ff4d97fa700 1 -- 172.21.2.102:0/2594179738 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff4cc0058e0 con 0x7ff4dc1c25b0 2021-02-10T11:18:55.402 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:55.407 7ff4e2e6a700 1 -- 172.21.2.102:0/2594179738 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4dc1b1ec0 con 0x7ff4dc1c25b0 2021-02-10T11:18:55.402 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:55.407 7ff4e2e6a700 1 -- 172.21.2.102:0/2594179738 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4dc1b2450 con 0x7ff4dc1c25b0 2021-02-10T11:18:55.402 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:55.408 7ff4d97fa700 1 -- 172.21.2.102:0/2594179738 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ff4cc006ca0 con 0x7ff4dc1c25b0 2021-02-10T11:18:55.402 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:55.408 7ff4d97fa700 1 --2- 172.21.2.102:0/2594179738 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff4c00186c0 0x7ff4c001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:55.402 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:55.408 7ff4d97fa700 1 -- 172.21.2.102:0/2594179738 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7ff4cc0328f0 con 0x7ff4dc1c25b0 2021-02-10T11:18:55.402 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:55.408 7ff4e0c86700 1 --2- 172.21.2.102:0/2594179738 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff4c00186c0 0x7ff4c001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:18:55.402 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:55.408 7ff4e0c86700 1 --2- 172.21.2.102:0/2594179738 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff4c00186c0 0x7ff4c001abc0 crc :-1 s=READY pgs=363 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:18:55.431 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:55.440 7ff4e2e6a700 1 -- 172.21.2.102:0/2594179738 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4dc07e6b0 con 0x7ff4dc1c25b0 2021-02-10T11:18:55.434 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:55.442 7ff4d97fa700 1 -- 172.21.2.102:0/2594179738 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ff4cc01f6c0 con 0x7ff4dc1c25b0 2021-02-10T11:18:55.562 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:55.570 7ff4e2e6a700 1 -- 172.21.2.102:0/2594179738 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344453", "target": ["mgr", ""]}) v1 -- 0x7ff4dc1b2450 con 0x7ff4c00186c0 2021-02-10T11:18:55.590 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:55.598 7ff4d97fa700 1 -- 172.21.2.102:0/2594179738 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7ff4c4000c60 con 0x7ff4c00186c0 2021-02-10T11:18:55.595 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:55.603 7ff4e2e6a700 1 -- 172.21.2.102:0/2594179738 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff4c00186c0 msgr2=0x7ff4c001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:18:55.595 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:55.603 7ff4e2e6a700 1 --2- 172.21.2.102:0/2594179738 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff4c00186c0 0x7ff4c001abc0 crc :-1 s=READY pgs=363 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:55.595 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:55.603 7ff4e2e6a700 1 -- 172.21.2.102:0/2594179738 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff4dc1c25b0 msgr2=0x7ff4dc1b3d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:18:55.595 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:55.603 7ff4e2e6a700 1 --2- 172.21.2.102:0/2594179738 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff4dc1c25b0 0x7ff4dc1b3d30 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 rx=0x7ff4cc009f40 tx=0x7ff4cc00bd50).stop 2021-02-10T11:18:55.596 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:55.604 7ff4e2e6a700 1 -- 172.21.2.102:0/2594179738 shutdown_connections 2021-02-10T11:18:55.596 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:55.604 7ff4e2e6a700 1 --2- 172.21.2.102:0/2594179738 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff4dc1b4350 0x7ff4dc1b18a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:18:55.596 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:55.604 7ff4e2e6a700 1 --2- 172.21.2.102:0/2594179738 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff4dc1c25b0 0x7ff4dc1b3d30 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:55.596 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:55.604 7ff4e2e6a700 1 --2- 172.21.2.102:0/2594179738 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff4c00186c0 0x7ff4c001abc0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:55.596 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:55.604 7ff4e2e6a700 1 --2- 172.21.2.102:0/2594179738 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff4dc0a3b50 0x7ff4dc1b3710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:18:55.596 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:55.604 7ff4e2e6a700 1 -- 172.21.2.102:0/2594179738 >> 172.21.2.102:0/2594179738 conn(0x7ff4dc0886d0 msgr2=0x7ff4dc081c40 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:18:55.596 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:55.604 7ff4e2e6a700 1 -- 172.21.2.102:0/2594179738 shutdown_connections 2021-02-10T11:18:55.596 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:55.604 7ff4e2e6a700 1 -- 172.21.2.102:0/2594179738 wait complete. 2021-02-10T11:18:55.603 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000344453 2021-02-10T11:18:56.037 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:56.029 7f3e6808e700 1 Processor -- start 2021-02-10T11:18:56.037 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:56.029 7f3e6808e700 1 -- start start 2021-02-10T11:18:56.038 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:56.029 7f3e6808e700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3e601b0470 0x7f3e601b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:56.038 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:56.029 7f3e6808e700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f3e600ef610 0x7f3e600efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:56.038 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:56.029 7f3e6808e700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e600f0b50 con 0x7f3e600f0030 2021-02-10T11:18:56.038 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:56.029 7f3e6808e700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e600e52e0 con 0x7f3e600ef610 2021-02-10T11:18:56.038 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:56.029 7f3e6808e700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e600e5490 con 0x7f3e601b0470 2021-02-10T11:18:56.038 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:56.030 7f3e656a9700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f3e600ef610 0x7f3e600efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:18:56.038 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:56.030 7f3e656a9700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f3e600ef610 0x7f3e600efa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:53434/0 (socket says 172.21.2.102:53434) 2021-02-10T11:18:56.038 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:56.030 7f3e656a9700 1 -- 172.21.2.102:0/3695892261 learned_addr learned my addr 172.21.2.102:0/3695892261 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:18:56.038 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:56.030 7f3e656a9700 1 -- 172.21.2.102:0/3695892261 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3e601b0470 msgr2=0x7f3e601b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:18:56.039 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:56.030 7f3e656a9700 1 --2- 172.21.2.102:0/3695892261 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3e601b0470 0x7f3e601b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:18:56.039 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:56.030 7f3e656a9700 1 -- 172.21.2.102:0/3695892261 >> v1:172.21.2.113:6789/0 conn(0x7f3e600f0030 legacy=0x7f3e600f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:18:56.039 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:56.030 7f3e656a9700 1 -- 172.21.2.102:0/3695892261 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e600f0430 con 0x7f3e600ef610 2021-02-10T11:18:56.039 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:56.030 7f3e656a9700 1 --2- 172.21.2.102:0/3695892261 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3e600ef610 0x7f3e600efa10 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 rx=0x7f3e5c006700 tx=0x7f3e5c009b80).ready entity=mon.0 client_cookie=7dd965ae9fe1baec server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:18:56.039 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:56.030 7f3e64ea8700 1 -- 172.21.2.102:0/3695892261 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3e5c000c90 con 0x7f3e600ef610 2021-02-10T11:18:56.039 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:56.030 7f3e64ea8700 1 -- 172.21.2.102:0/3695892261 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e5c00c0c0 con 0x7f3e600ef610 2021-02-10T11:18:56.039 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:56.030 7f3e64ea8700 1 -- 172.21.2.102:0/3695892261 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3e5c00c5e0 con 0x7f3e600ef610 2021-02-10T11:18:56.039 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:56.030 7f3e6808e700 1 -- 172.21.2.102:0/3695892261 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3e600ef610 msgr2=0x7f3e600efa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:18:56.039 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:56.030 7f3e6808e700 1 --2- 172.21.2.102:0/3695892261 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3e600ef610 0x7f3e600efa10 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 rx=0x7f3e5c006700 tx=0x7f3e5c009b80).stop 2021-02-10T11:18:56.039 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:56.031 7f3e6808e700 1 -- 172.21.2.102:0/3695892261 shutdown_connections 2021-02-10T11:18:56.040 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:56.031 7f3e6808e700 1 --2- 172.21.2.102:0/3695892261 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3e600ef610 0x7f3e600efa10 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:56.040 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:56.031 7f3e6808e700 1 --2- 172.21.2.102:0/3695892261 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3e601b0470 0x7f3e601b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:18:56.040 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:56.031 7f3e6808e700 1 -- 172.21.2.102:0/3695892261 >> 172.21.2.102:0/3695892261 conn(0x7f3e600ce3e0 msgr2=0x7f3e600e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:18:56.040 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:56.031 7f3e6808e700 1 -- 172.21.2.102:0/3695892261 shutdown_connections 2021-02-10T11:18:56.040 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:56.031 7f3e6808e700 1 -- 172.21.2.102:0/3695892261 wait complete. 2021-02-10T11:18:56.042 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:56.046 7f3e6808e700 1 Processor -- start 2021-02-10T11:18:56.042 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:56.046 7f3e6808e700 1 -- start start 2021-02-10T11:18:56.042 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:56.046 7f3e6808e700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3e600c36a0 0x7f3e600c2b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:56.042 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:56.046 7f3e6808e700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3e600c3190 0x7f3e600b1d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:56.042 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:56.046 7f3e6808e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3e600b2370 0x7f3e600b2d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:56.042 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:56.046 7f3e6808e700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e600b27c0 con 0x7f3e600b2370 2021-02-10T11:18:56.042 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:56.046 7f3e6808e700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e600b3360 con 0x7f3e600c36a0 2021-02-10T11:18:56.042 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:56.046 7f3e6808e700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e600b3540 con 0x7f3e600c3190 2021-02-10T11:18:56.043 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:56.047 7f3e666ab700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3e600b2370 0x7f3e600b2d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:18:56.043 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:56.047 7f3e666ab700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3e600b2370 0x7f3e600b2d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:53440/0 (socket says 172.21.2.102:53440) 2021-02-10T11:18:56.043 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:56.047 7f3e666ab700 1 -- 172.21.2.102:0/1448429035 learned_addr learned my addr 172.21.2.102:0/1448429035 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:18:56.043 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:56.047 7f3e666ab700 1 -- 172.21.2.102:0/1448429035 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3e600c3190 msgr2=0x7f3e600b1d50 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:18:56.043 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:56.047 7f3e666ab700 1 --2- 172.21.2.102:0/1448429035 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3e600c3190 0x7f3e600b1d50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:18:56.043 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:56.047 7f3e666ab700 1 -- 172.21.2.102:0/1448429035 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3e600c36a0 msgr2=0x7f3e600c2b70 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:18:56.043 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:56.047 7f3e666ab700 1 --2- 172.21.2.102:0/1448429035 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3e600c36a0 0x7f3e600c2b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:18:56.043 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:56.047 7f3e666ab700 1 -- 172.21.2.102:0/1448429035 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e600b3760 con 0x7f3e600b2370 2021-02-10T11:18:56.043 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:56.047 7f3e666ab700 1 --2- 172.21.2.102:0/1448429035 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3e600b2370 0x7f3e600b2d40 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 rx=0x7f3e58009940 tx=0x7f3e5800bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:18:56.043 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:56.047 7f3e527fc700 1 -- 172.21.2.102:0/1448429035 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3e58000f90 con 0x7f3e600b2370 2021-02-10T11:18:56.043 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:56.047 7f3e527fc700 1 -- 172.21.2.102:0/1448429035 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e5800c2f0 con 0x7f3e600b2370 2021-02-10T11:18:56.043 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:56.047 7f3e527fc700 1 -- 172.21.2.102:0/1448429035 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3e5800c810 con 0x7f3e600b2370 2021-02-10T11:18:56.043 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:56.047 7f3e6808e700 1 -- 172.21.2.102:0/1448429035 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e600b3760 con 0x7f3e600b2370 2021-02-10T11:18:56.044 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:56.047 7f3e6808e700 1 -- 172.21.2.102:0/1448429035 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e600b3760 con 0x7f3e600b2370 2021-02-10T11:18:56.044 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:56.048 7f3e527fc700 1 -- 172.21.2.102:0/1448429035 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f3e58001220 con 0x7f3e600b2370 2021-02-10T11:18:56.044 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:56.048 7f3e527fc700 1 --2- 172.21.2.102:0/1448429035 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3e480186d0 0x7f3e4801abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:18:56.044 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:56.048 7f3e527fc700 1 -- 172.21.2.102:0/1448429035 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f3e58032b80 con 0x7f3e600b2370 2021-02-10T11:18:56.044 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:56.048 7f3e65eaa700 1 --2- 172.21.2.102:0/1448429035 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3e480186d0 0x7f3e4801abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:18:56.044 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:56.048 7f3e65eaa700 1 --2- 172.21.2.102:0/1448429035 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3e480186d0 0x7f3e4801abd0 crc :-1 s=READY pgs=364 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:18:56.071 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:56.079 7f3e6808e700 1 -- 172.21.2.102:0/1448429035 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e600edd10 con 0x7f3e600b2370 2021-02-10T11:18:56.073 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:56.082 7f3e527fc700 1 -- 172.21.2.102:0/1448429035 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f3e5800be30 con 0x7f3e600b2370 2021-02-10T11:18:56.203 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:56.211 7f3e6808e700 1 -- 172.21.2.102:0/1448429035 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344453", "target": ["mgr", ""]}) v1 -- 0x7f3e600edd10 con 0x7f3e480186d0 2021-02-10T11:18:56.205 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:56.214 7f3e527fc700 1 -- 172.21.2.102:0/1448429035 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f3e54006960 con 0x7f3e480186d0 2021-02-10T11:18:56.206 INFO:teuthology.orchestra.run.gibba002.stdout:/volumes/_nogroup/subvolume_0000000000344453/51f7c27f-f078-4c25-914f-b32c670eb6ae 2021-02-10T11:18:56.208 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:56.216 7f3e6808e700 1 -- 172.21.2.102:0/1448429035 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3e480186d0 msgr2=0x7f3e4801abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:18:56.208 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:56.216 7f3e6808e700 1 --2- 172.21.2.102:0/1448429035 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3e480186d0 0x7f3e4801abd0 crc :-1 s=READY pgs=364 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:56.209 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:56.216 7f3e6808e700 1 -- 172.21.2.102:0/1448429035 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3e600b2370 msgr2=0x7f3e600b2d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:18:56.209 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:56.216 7f3e6808e700 1 --2- 172.21.2.102:0/1448429035 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3e600b2370 0x7f3e600b2d40 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 rx=0x7f3e58009940 tx=0x7f3e5800bcf0).stop 2021-02-10T11:18:56.209 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:56.216 7f3e6808e700 1 -- 172.21.2.102:0/1448429035 shutdown_connections 2021-02-10T11:18:56.209 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:56.216 7f3e6808e700 1 --2- 172.21.2.102:0/1448429035 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3e600b2370 0x7f3e600b2d40 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:56.209 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:56.216 7f3e6808e700 1 --2- 172.21.2.102:0/1448429035 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3e600c3190 0x7f3e600b1d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:18:56.209 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:56.216 7f3e6808e700 1 --2- 172.21.2.102:0/1448429035 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3e480186d0 0x7f3e4801abd0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:18:56.209 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:56.216 7f3e6808e700 1 --2- 172.21.2.102:0/1448429035 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3e600c36a0 0x7f3e600c2b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:18:56.209 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:56.216 7f3e6808e700 1 -- 172.21.2.102:0/1448429035 >> 172.21.2.102:0/1448429035 conn(0x7f3e6009f630 msgr2=0x7f3e600a0130 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:18:56.209 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:56.216 7f3e6808e700 1 -- 172.21.2.102:0/1448429035 shutdown_connections 2021-02-10T11:18:56.209 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:18:56.216 7f3e6808e700 1 -- 172.21.2.102:0/1448429035 wait complete. 2021-02-10T11:18:56.216 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000344453 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000344453/51f7c27f-f078-4c25-914f-b32c670eb6ae 2021-02-10T11:18:56.216 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:18:56.216 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:18:56.216 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:18:56.553 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:18:56.554 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:18:56.554 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344453/51f7c27f-f078-4c25-914f-b32c670eb6ae/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:18:56.934 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:18:56.935 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:18:56.935 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0235282 s, 44.6 MB/s 2021-02-10T11:18:56.936 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:18:56.936 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:18:56.936 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:18:57.272 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:18:57.272 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:18:57.273 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344453/51f7c27f-f078-4c25-914f-b32c670eb6ae/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:18:57.648 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:18:57.649 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:18:57.649 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0234895 s, 44.6 MB/s 2021-02-10T11:18:57.650 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:18:57.650 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:18:57.650 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:18:57.986 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:18:57.986 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:18:57.987 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344453/51f7c27f-f078-4c25-914f-b32c670eb6ae/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:18:58.362 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:18:58.362 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:18:58.362 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.023619 s, 44.4 MB/s 2021-02-10T11:18:58.364 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:18:58.364 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:18:58.364 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:18:58.701 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:18:58.701 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:18:58.702 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344453/51f7c27f-f078-4c25-914f-b32c670eb6ae/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:18:59.077 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:18:59.077 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:18:59.077 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0231203 s, 45.4 MB/s 2021-02-10T11:18:59.078 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:18:59.079 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:18:59.079 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:18:59.416 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:18:59.417 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:18:59.417 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344453/51f7c27f-f078-4c25-914f-b32c670eb6ae/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:18:59.698 DEBUG:teuthology.orchestra.run.gibba002:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:18:59.701 DEBUG:teuthology.orchestra.run.gibba013:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:18:59.795 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:18:59.795 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:18:59.795 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0246828 s, 42.5 MB/s 2021-02-10T11:18:59.796 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:18:59.796 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:18:59.796 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:19:00.039 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:19:00.039 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:19:00.039 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344453/51f7c27f-f078-4c25-914f-b32c670eb6ae/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:19:00.418 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:19:00.418 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:19:00.418 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0262545 s, 39.9 MB/s 2021-02-10T11:19:00.419 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:19:00.419 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:19:00.419 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:19:00.755 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:19:00.756 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:19:00.756 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344453/51f7c27f-f078-4c25-914f-b32c670eb6ae/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:19:01.133 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:19:01.133 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:19:01.133 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0254732 s, 41.2 MB/s 2021-02-10T11:19:01.134 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:19:01.134 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:19:01.134 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:19:01.470 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:19:01.470 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:19:01.471 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344453/51f7c27f-f078-4c25-914f-b32c670eb6ae/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:19:01.846 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:19:01.846 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:19:01.847 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0240512 s, 43.6 MB/s 2021-02-10T11:19:01.847 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:19:01.848 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:19:01.848 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:19:02.185 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:19:02.186 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:19:02.186 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344453/51f7c27f-f078-4c25-914f-b32c670eb6ae/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:19:02.562 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:19:02.562 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:19:02.562 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0247816 s, 42.3 MB/s 2021-02-10T11:19:02.563 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:19:02.563 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:19:02.564 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:19:02.900 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:19:02.900 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:19:02.900 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344453/51f7c27f-f078-4c25-914f-b32c670eb6ae/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:19:03.276 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:19:03.276 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:19:03.277 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0233629 s, 44.9 MB/s 2021-02-10T11:19:03.277 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000344454 2021-02-10T11:19:03.710 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:03.702 7efcb32d3700 1 Processor -- start 2021-02-10T11:19:04.560 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:03.702 7efcb32d3700 1 -- start start 2021-02-10T11:19:04.560 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:03.703 7efcb32d3700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7efcac1b0870 0x7efcac0f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:04.560 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:03.703 7efcb32d3700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcac1b0c70 con 0x7efcac0efe00 2021-02-10T11:19:04.560 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:03.703 7efcb32d3700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcac0f0b50 con 0x7efcac1b0470 2021-02-10T11:19:04.561 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:03.703 7efcb32d3700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcac0e5ea0 con 0x7efcac1b0870 2021-02-10T11:19:04.561 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:03.703 7efcb18f0700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7efcac0efe00 0x7efcac0f0200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:45670/0 (socket says 172.21.2.102:45670) 2021-02-10T11:19:04.561 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:03.703 7efcb18f0700 1 -- 172.21.2.102:0/3778440491 learned_addr learned my addr 172.21.2.102:0/3778440491 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:19:04.561 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:03.703 7efcb08ee700 1 --2- 172.21.2.102:0/3778440491 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7efcac1b0870 0x7efcac0f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:19:04.561 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:03.703 7efc9ffff700 1 -- 172.21.2.102:0/3778440491 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3351534813 0 0) 0x7efca4001500 con 0x7efcac0efe00 2021-02-10T11:19:04.561 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:03.703 7efc9ffff700 1 -- 172.21.2.102:0/3778440491 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc98002fe0 con 0x7efcac0efe00 2021-02-10T11:19:04.561 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:03.703 7efc9ffff700 1 -- 172.21.2.102:0/3778440491 <== mon.1 v1:172.21.2.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3983072530 0 0) 0x7efca8000ff0 con 0x7efcac1b0470 2021-02-10T11:19:04.561 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:03.704 7efcb08ee700 1 -- 172.21.2.102:0/3778440491 >> v1:172.21.2.113:6789/0 conn(0x7efcac1b0470 legacy=0x7efcac0ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:19:04.561 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:03.704 7efcb08ee700 1 -- 172.21.2.102:0/3778440491 >> v1:172.21.2.102:6789/0 conn(0x7efcac0efe00 legacy=0x7efcac0f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:19:04.561 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:03.704 7efcb08ee700 1 -- 172.21.2.102:0/3778440491 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcac0e52e0 con 0x7efcac1b0870 2021-02-10T11:19:04.561 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:03.704 7efcb08ee700 1 --2- 172.21.2.102:0/3778440491 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7efcac1b0870 0x7efcac0f20e0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7efca0006650 tx=0x7efca0009a80).ready entity=mon.2 client_cookie=55ed9346690b049e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:19:04.561 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:03.704 7efc9ffff700 1 -- 172.21.2.102:0/3778440491 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efca00009b0 con 0x7efcac1b0870 2021-02-10T11:19:04.562 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:03.704 7efc9ffff700 1 -- 172.21.2.102:0/3778440491 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efca000c0c0 con 0x7efcac1b0870 2021-02-10T11:19:04.562 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:03.704 7efcb32d3700 1 -- 172.21.2.102:0/3778440491 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7efcac1b0870 msgr2=0x7efcac0f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:19:04.562 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:03.704 7efcb32d3700 1 --2- 172.21.2.102:0/3778440491 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7efcac1b0870 0x7efcac0f20e0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7efca0006650 tx=0x7efca0009a80).stop 2021-02-10T11:19:04.562 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:03.704 7efcb32d3700 1 -- 172.21.2.102:0/3778440491 shutdown_connections 2021-02-10T11:19:04.562 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:03.704 7efcb32d3700 1 --2- 172.21.2.102:0/3778440491 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7efcac1b0870 0x7efcac0f20e0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:04.562 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:03.704 7efcb32d3700 1 -- 172.21.2.102:0/3778440491 >> 172.21.2.102:0/3778440491 conn(0x7efcac0ce3e0 msgr2=0x7efcac0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:19:04.562 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:03.705 7efcb32d3700 1 -- 172.21.2.102:0/3778440491 shutdown_connections 2021-02-10T11:19:04.562 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:03.705 7efcb32d3700 1 -- 172.21.2.102:0/3778440491 wait complete. 2021-02-10T11:19:04.565 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:03.719 7efcb32d3700 1 Processor -- start 2021-02-10T11:19:04.565 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:03.719 7efcb32d3700 1 -- start start 2021-02-10T11:19:04.565 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:03.719 7efcb32d3700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efcac1a0490 0x7efcac0ada30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:04.565 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:03.719 7efcb32d3700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7efcac0ae050 0x7efcac0b6320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:04.565 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:03.719 7efcb32d3700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7efcac0b6940 0x7efcac0eebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:04.565 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:03.719 7efcb32d3700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcac0ae450 con 0x7efcac1a0490 2021-02-10T11:19:04.565 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:03.719 7efcb32d3700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcac0b6d40 con 0x7efcac0ae050 2021-02-10T11:19:04.565 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:03.719 7efcb32d3700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcac0b6f60 con 0x7efcac0b6940 2021-02-10T11:19:04.565 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:03.720 7efcb08ee700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7efcac0ae050 0x7efcac0b6320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:19:04.565 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:03.720 7efcb08ee700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7efcac0ae050 0x7efcac0b6320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.113:3300/0 says I am v2:172.21.2.102:56882/0 (socket says 172.21.2.102:56882) 2021-02-10T11:19:04.566 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:03.720 7efcb08ee700 1 -- 172.21.2.102:0/3380983781 learned_addr learned my addr 172.21.2.102:0/3380983781 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:19:04.566 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:03.720 7efcb10ef700 1 --2- 172.21.2.102:0/3380983781 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efcac1a0490 0x7efcac0ada30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:19:04.566 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:03.720 7efcb18f0700 1 --2- 172.21.2.102:0/3380983781 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7efcac0b6940 0x7efcac0eebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:19:04.566 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:03.720 7efcb08ee700 1 -- 172.21.2.102:0/3380983781 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7efcac0b6940 msgr2=0x7efcac0eebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:19:04.566 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:03.720 7efcb08ee700 1 --2- 172.21.2.102:0/3380983781 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7efcac0b6940 0x7efcac0eebd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:04.566 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:03.720 7efcb08ee700 1 -- 172.21.2.102:0/3380983781 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efcac1a0490 msgr2=0x7efcac0ada30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:19:04.566 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:03.720 7efcb08ee700 1 --2- 172.21.2.102:0/3380983781 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efcac1a0490 0x7efcac0ada30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:04.566 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:03.720 7efcb08ee700 1 -- 172.21.2.102:0/3380983781 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcac0ef1f0 con 0x7efcac0ae050 2021-02-10T11:19:04.566 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:03.720 7efcb18f0700 1 --2- 172.21.2.102:0/3380983781 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7efcac0b6940 0x7efcac0eebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:19:04.566 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:03.720 7efcb08ee700 1 --2- 172.21.2.102:0/3380983781 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7efcac0ae050 0x7efcac0b6320 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 rx=0x7efca0006fc0 tx=0x7efca000b960).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:19:04.566 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:03.721 7efc9dffb700 1 -- 172.21.2.102:0/3380983781 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efca0002030 con 0x7efcac0ae050 2021-02-10T11:19:04.566 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:03.721 7efc9dffb700 1 -- 172.21.2.102:0/3380983781 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efca0000a10 con 0x7efcac0ae050 2021-02-10T11:19:04.566 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:03.721 7efc9dffb700 1 -- 172.21.2.102:0/3380983781 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efca0002630 con 0x7efcac0ae050 2021-02-10T11:19:04.567 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:03.721 7efcb32d3700 1 -- 172.21.2.102:0/3380983781 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efcac0ef1f0 con 0x7efcac0ae050 2021-02-10T11:19:04.567 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:03.721 7efcb32d3700 1 -- 172.21.2.102:0/3380983781 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efcac0ef750 con 0x7efcac0ae050 2021-02-10T11:19:04.567 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:03.721 7efc9dffb700 1 -- 172.21.2.102:0/3380983781 <== mon.1 v2:172.21.2.113:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7efca0000c90 con 0x7efcac0ae050 2021-02-10T11:19:04.567 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:03.721 7efc9dffb700 1 --2- 172.21.2.102:0/3380983781 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7efc8c0186c0 0x7efc8c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:04.567 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:03.721 7efc9dffb700 1 -- 172.21.2.102:0/3380983781 <== mon.1 v2:172.21.2.113:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7efca00323f0 con 0x7efcac0ae050 2021-02-10T11:19:04.567 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:03.721 7efcb10ef700 1 --2- 172.21.2.102:0/3380983781 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7efc8c0186c0 0x7efc8c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:19:04.567 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:03.722 7efcb10ef700 1 --2- 172.21.2.102:0/3380983781 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7efc8c0186c0 0x7efc8c01abc0 crc :-1 s=READY pgs=365 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:19:04.567 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:03.752 7efcb32d3700 1 -- 172.21.2.102:0/3380983781 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efcac0efa00 con 0x7efcac0ae050 2021-02-10T11:19:04.567 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:03.755 7efc9dffb700 1 -- 172.21.2.102:0/3380983781 <== mon.1 v2:172.21.2.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7efca0009360 con 0x7efcac0ae050 2021-02-10T11:19:04.567 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:03.882 7efcb32d3700 1 -- 172.21.2.102:0/3380983781 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344454", "target": ["mgr", ""]}) v1 -- 0x7efcac0ef750 con 0x7efc8c0186c0 2021-02-10T11:19:04.567 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:03.910 7efc9dffb700 1 -- 172.21.2.102:0/3380983781 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7efca8007880 con 0x7efc8c0186c0 2021-02-10T11:19:04.567 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:03.915 7efcb32d3700 1 -- 172.21.2.102:0/3380983781 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7efc8c0186c0 msgr2=0x7efc8c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:19:04.567 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:03.915 7efcb32d3700 1 --2- 172.21.2.102:0/3380983781 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7efc8c0186c0 0x7efc8c01abc0 crc :-1 s=READY pgs=365 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:04.568 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:03.915 7efcb32d3700 1 -- 172.21.2.102:0/3380983781 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7efcac0ae050 msgr2=0x7efcac0b6320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:19:04.568 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:03.915 7efcb32d3700 1 --2- 172.21.2.102:0/3380983781 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7efcac0ae050 0x7efcac0b6320 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 rx=0x7efca0006fc0 tx=0x7efca000b960).stop 2021-02-10T11:19:04.568 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:03.916 7efcb32d3700 1 -- 172.21.2.102:0/3380983781 shutdown_connections 2021-02-10T11:19:04.568 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:03.916 7efcb32d3700 1 --2- 172.21.2.102:0/3380983781 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7efcac0b6940 0x7efcac0eebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:04.568 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:03.916 7efcb32d3700 1 --2- 172.21.2.102:0/3380983781 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7efc8c0186c0 0x7efc8c01abc0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:04.568 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:03.916 7efcb32d3700 1 --2- 172.21.2.102:0/3380983781 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7efcac0ae050 0x7efcac0b6320 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:04.568 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:03.916 7efcb32d3700 1 --2- 172.21.2.102:0/3380983781 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efcac1a0490 0x7efcac0ada30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:04.568 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:03.916 7efcb32d3700 1 -- 172.21.2.102:0/3380983781 >> 172.21.2.102:0/3380983781 conn(0x7efcac0bbf60 msgr2=0x7efcac0edef0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:19:04.568 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:03.916 7efcb32d3700 1 -- 172.21.2.102:0/3380983781 shutdown_connections 2021-02-10T11:19:04.568 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:03.916 7efcb32d3700 1 -- 172.21.2.102:0/3380983781 wait complete. 2021-02-10T11:19:04.569 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000344454 2021-02-10T11:19:05.002 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:04.994 7f4037d16700 1 Processor -- start 2021-02-10T11:19:05.002 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:04.994 7f4037d16700 1 -- start start 2021-02-10T11:19:05.002 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:04.994 7f4037d16700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f40301b0470 0x7f40301b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:05.002 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:04.994 7f4037d16700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40300f0b50 con 0x7f40300ef610 2021-02-10T11:19:05.002 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:04.994 7f4037d16700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40300f24e0 con 0x7f40300f20e0 2021-02-10T11:19:05.002 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:04.994 7f4037d16700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40300e5ea0 con 0x7f40301b0470 2021-02-10T11:19:05.003 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:04.995 7f4035b32700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f40301b0470 0x7f40301b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:19:05.003 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:04.995 7f4035331700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f40300ef610 0x7f40300efa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:45682/0 (socket says 172.21.2.102:45682) 2021-02-10T11:19:05.003 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:04.995 7f4035331700 1 -- 172.21.2.102:0/3356699453 learned_addr learned my addr 172.21.2.102:0/3356699453 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:19:05.003 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:04.995 7f4035b32700 1 -- 172.21.2.102:0/3356699453 >> v1:172.21.2.113:6789/0 conn(0x7f40300f20e0 legacy=0x7f40300f0200 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:19:05.003 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:04.995 7f4035b32700 1 -- 172.21.2.102:0/3356699453 >> v1:172.21.2.102:6789/0 conn(0x7f40300ef610 legacy=0x7f40300efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:19:05.003 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:04.995 7f4035b32700 1 -- 172.21.2.102:0/3356699453 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40300e52e0 con 0x7f40301b0470 2021-02-10T11:19:05.003 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:04.995 7f4035b32700 1 --2- 172.21.2.102:0/3356699453 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f40301b0470 0x7f40301b0870 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 rx=0x7f402c006650 tx=0x7f402c009a80).ready entity=mon.0 client_cookie=6e9131a7426288ee server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:19:05.003 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:04.995 7f4034b30700 1 -- 172.21.2.102:0/3356699453 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f402c00c0c0 con 0x7f40301b0470 2021-02-10T11:19:05.003 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:04.995 7f4034b30700 1 -- 172.21.2.102:0/3356699453 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f402c00c340 con 0x7f40301b0470 2021-02-10T11:19:05.003 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:04.995 7f4034b30700 1 -- 172.21.2.102:0/3356699453 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f402c00c790 con 0x7f40301b0470 2021-02-10T11:19:05.004 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:04.995 7f4037d16700 1 -- 172.21.2.102:0/3356699453 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f40301b0470 msgr2=0x7f40301b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:19:05.004 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:04.995 7f4037d16700 1 --2- 172.21.2.102:0/3356699453 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f40301b0470 0x7f40301b0870 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 rx=0x7f402c006650 tx=0x7f402c009a80).stop 2021-02-10T11:19:05.004 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:04.996 7f4037d16700 1 -- 172.21.2.102:0/3356699453 shutdown_connections 2021-02-10T11:19:05.004 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:04.996 7f4037d16700 1 --2- 172.21.2.102:0/3356699453 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f40301b0470 0x7f40301b0870 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:05.004 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:04.996 7f4037d16700 1 -- 172.21.2.102:0/3356699453 >> 172.21.2.102:0/3356699453 conn(0x7f40300ce3e0 msgr2=0x7f40300e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:19:05.004 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:04.996 7f4037d16700 1 -- 172.21.2.102:0/3356699453 shutdown_connections 2021-02-10T11:19:05.004 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:04.996 7f4037d16700 1 -- 172.21.2.102:0/3356699453 wait complete. 2021-02-10T11:19:05.005 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:05.011 7f4037d16700 1 Processor -- start 2021-02-10T11:19:05.005 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:05.011 7f4037d16700 1 -- start start 2021-02-10T11:19:05.005 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:05.011 7f4037d16700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4030087510 0x7f40300aedc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:05.005 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:05.011 7f4037d16700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f40301c68f0 0x7f40301b1f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:05.005 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:05.011 7f4037d16700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f403008c4d0 0x7f40301b18a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:05.005 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:05.011 7f4037d16700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40300a2d50 con 0x7f403008c4d0 2021-02-10T11:19:05.005 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:05.011 7f4037d16700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40300a2f70 con 0x7f40301c68f0 2021-02-10T11:19:05.005 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:05.011 7f4037d16700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40300a3190 con 0x7f4030087510 2021-02-10T11:19:05.007 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:05.011 7f4036333700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f403008c4d0 0x7f40301b18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:19:05.007 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:05.012 7f4036333700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f403008c4d0 0x7f40301b18a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:53464/0 (socket says 172.21.2.102:53464) 2021-02-10T11:19:05.007 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:05.012 7f4036333700 1 -- 172.21.2.102:0/1875770628 learned_addr learned my addr 172.21.2.102:0/1875770628 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:19:05.007 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:05.012 7f4036333700 1 -- 172.21.2.102:0/1875770628 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4030087510 msgr2=0x7f40300aedc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:19:05.007 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:05.012 7f4036333700 1 --2- 172.21.2.102:0/1875770628 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4030087510 0x7f40300aedc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:19:05.007 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:05.012 7f4036333700 1 -- 172.21.2.102:0/1875770628 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f40301c68f0 msgr2=0x7f40301b1f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:19:05.007 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:05.012 7f4036333700 1 --2- 172.21.2.102:0/1875770628 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f40301c68f0 0x7f40301b1f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:19:05.008 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:05.012 7f4036333700 1 -- 172.21.2.102:0/1875770628 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40300a33b0 con 0x7f403008c4d0 2021-02-10T11:19:05.008 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:05.012 7f4036333700 1 --2- 172.21.2.102:0/1875770628 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f403008c4d0 0x7f40301b18a0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 rx=0x7f4028009940 tx=0x7f402800bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:19:05.008 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:05.012 7f40227fc700 1 -- 172.21.2.102:0/1875770628 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4028003030 con 0x7f403008c4d0 2021-02-10T11:19:05.008 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:05.012 7f40227fc700 1 -- 172.21.2.102:0/1875770628 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4028001270 con 0x7f403008c4d0 2021-02-10T11:19:05.008 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:05.012 7f4037d16700 1 -- 172.21.2.102:0/1875770628 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40300a33b0 con 0x7f403008c4d0 2021-02-10T11:19:05.008 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:05.012 7f40227fc700 1 -- 172.21.2.102:0/1875770628 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4028001790 con 0x7f403008c4d0 2021-02-10T11:19:05.008 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:05.013 7f4037d16700 1 -- 172.21.2.102:0/1875770628 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40300a33b0 con 0x7f403008c4d0 2021-02-10T11:19:05.008 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:05.013 7f40227fc700 1 -- 172.21.2.102:0/1875770628 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4028001450 con 0x7f403008c4d0 2021-02-10T11:19:05.008 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:05.013 7f40227fc700 1 --2- 172.21.2.102:0/1875770628 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f40180186d0 0x7f401801abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:05.008 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:05.013 7f40227fc700 1 -- 172.21.2.102:0/1875770628 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f4028032b60 con 0x7f403008c4d0 2021-02-10T11:19:05.008 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:05.013 7f4035b32700 1 --2- 172.21.2.102:0/1875770628 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f40180186d0 0x7f401801abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:19:05.008 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:05.013 7f4035b32700 1 --2- 172.21.2.102:0/1875770628 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f40180186d0 0x7f401801abd0 crc :-1 s=READY pgs=366 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:19:05.036 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:05.044 7f4037d16700 1 -- 172.21.2.102:0/1875770628 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f403007e6b0 con 0x7f403008c4d0 2021-02-10T11:19:05.038 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:05.047 7f40227fc700 1 -- 172.21.2.102:0/1875770628 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f402800be30 con 0x7f403008c4d0 2021-02-10T11:19:05.169 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:05.177 7f4037d16700 1 -- 172.21.2.102:0/1875770628 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344454", "target": ["mgr", ""]}) v1 -- 0x7f403007e6b0 con 0x7f40180186d0 2021-02-10T11:19:05.172 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:05.180 7f40227fc700 1 -- 172.21.2.102:0/1875770628 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f402c006370 con 0x7f40180186d0 2021-02-10T11:19:05.176 INFO:teuthology.orchestra.run.gibba002.stdout:/volumes/_nogroup/subvolume_0000000000344454/194341e5-ef4a-419f-997e-ab39b7397b34 2021-02-10T11:19:05.177 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:05.186 7f4037d16700 1 -- 172.21.2.102:0/1875770628 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f40180186d0 msgr2=0x7f401801abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:19:05.177 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:05.186 7f4037d16700 1 --2- 172.21.2.102:0/1875770628 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f40180186d0 0x7f401801abd0 crc :-1 s=READY pgs=366 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:05.177 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:05.186 7f4037d16700 1 -- 172.21.2.102:0/1875770628 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f403008c4d0 msgr2=0x7f40301b18a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:19:05.177 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:05.186 7f4037d16700 1 --2- 172.21.2.102:0/1875770628 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f403008c4d0 0x7f40301b18a0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 rx=0x7f4028009940 tx=0x7f402800bcf0).stop 2021-02-10T11:19:05.178 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:05.186 7f4037d16700 1 -- 172.21.2.102:0/1875770628 shutdown_connections 2021-02-10T11:19:05.178 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:05.186 7f4037d16700 1 --2- 172.21.2.102:0/1875770628 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f403008c4d0 0x7f40301b18a0 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:05.178 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:05.186 7f4037d16700 1 --2- 172.21.2.102:0/1875770628 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f40180186d0 0x7f401801abd0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:05.178 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:05.186 7f4037d16700 1 --2- 172.21.2.102:0/1875770628 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f40301c68f0 0x7f40301b1f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:19:05.179 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:05.186 7f4037d16700 1 --2- 172.21.2.102:0/1875770628 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4030087510 0x7f40300aedc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:19:05.179 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:05.186 7f4037d16700 1 -- 172.21.2.102:0/1875770628 >> 172.21.2.102:0/1875770628 conn(0x7f40300886d0 msgr2=0x7f4030081c40 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:19:05.179 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:05.186 7f4037d16700 1 -- 172.21.2.102:0/1875770628 shutdown_connections 2021-02-10T11:19:05.179 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:05.186 7f4037d16700 1 -- 172.21.2.102:0/1875770628 wait complete. 2021-02-10T11:19:05.186 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000344454 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000344454/194341e5-ef4a-419f-997e-ab39b7397b34 2021-02-10T11:19:05.186 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:19:05.186 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:19:05.186 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:19:05.522 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:19:05.523 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:19:05.523 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344454/194341e5-ef4a-419f-997e-ab39b7397b34/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:19:05.905 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:19:05.905 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:19:05.905 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0229626 s, 45.7 MB/s 2021-02-10T11:19:05.906 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:19:05.906 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:19:05.906 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:19:06.243 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:19:06.244 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:19:06.244 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344454/194341e5-ef4a-419f-997e-ab39b7397b34/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:19:06.620 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:19:06.621 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:19:06.621 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0240303 s, 43.6 MB/s 2021-02-10T11:19:06.622 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:19:06.622 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:19:06.622 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:19:06.957 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:19:06.958 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:19:06.958 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344454/194341e5-ef4a-419f-997e-ab39b7397b34/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:19:07.335 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:19:07.335 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:19:07.335 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.024618 s, 42.6 MB/s 2021-02-10T11:19:07.336 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:19:07.337 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:19:07.337 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:19:07.673 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:19:07.674 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:19:07.674 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344454/194341e5-ef4a-419f-997e-ab39b7397b34/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:19:08.051 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:19:08.051 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:19:08.051 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0245009 s, 42.8 MB/s 2021-02-10T11:19:08.052 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:19:08.052 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:19:08.052 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:19:08.388 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:19:08.389 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:19:08.389 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344454/194341e5-ef4a-419f-997e-ab39b7397b34/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:19:08.763 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:19:08.763 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:19:08.763 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0252774 s, 41.5 MB/s 2021-02-10T11:19:08.764 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:19:08.764 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:19:08.764 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:19:09.103 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:19:09.104 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:19:09.104 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344454/194341e5-ef4a-419f-997e-ab39b7397b34/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:19:09.484 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:19:09.484 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:19:09.484 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0236825 s, 44.3 MB/s 2021-02-10T11:19:09.486 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:19:09.486 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:19:09.486 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:19:09.823 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:19:09.824 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:19:09.824 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344454/194341e5-ef4a-419f-997e-ab39b7397b34/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:19:10.200 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:19:10.200 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:19:10.201 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0241029 s, 43.5 MB/s 2021-02-10T11:19:10.201 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:19:10.202 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:19:10.202 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:19:10.537 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:19:10.538 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:19:10.538 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344454/194341e5-ef4a-419f-997e-ab39b7397b34/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:19:10.913 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:19:10.913 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:19:10.913 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0231214 s, 45.4 MB/s 2021-02-10T11:19:10.914 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:19:10.914 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:19:10.914 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:19:11.249 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:19:11.250 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:19:11.250 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344454/194341e5-ef4a-419f-997e-ab39b7397b34/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:19:11.629 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:19:11.629 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:19:11.629 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0253775 s, 41.3 MB/s 2021-02-10T11:19:11.630 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:19:11.631 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:19:11.631 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:19:11.967 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:19:11.967 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:19:11.967 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344454/194341e5-ef4a-419f-997e-ab39b7397b34/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:19:12.343 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:19:12.343 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:19:12.343 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0228526 s, 45.9 MB/s 2021-02-10T11:19:12.344 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000344455 2021-02-10T11:19:12.779 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:12.770 7fbfaee65700 1 Processor -- start 2021-02-10T11:19:12.779 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:12.770 7fbfaee65700 1 -- start start 2021-02-10T11:19:12.779 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:12.770 7fbfaee65700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fbfa81b0470 0x7fbfa81b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:12.779 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:12.770 7fbfaee65700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fbfa80ef610 0x7fbfa80efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:12.779 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:12.770 7fbfaee65700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfa80f0b50 con 0x7fbfa80f0030 2021-02-10T11:19:12.779 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:12.770 7fbfaee65700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbfa80e52e0 con 0x7fbfa80ef610 2021-02-10T11:19:12.780 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:12.770 7fbfaee65700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbfa80e5490 con 0x7fbfa81b0470 2021-02-10T11:19:12.780 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:12.771 7fbfa7fff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fbfa80ef610 0x7fbfa80efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:19:12.780 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:12.771 7fbfa7fff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fbfa80ef610 0x7fbfa80efa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:53470/0 (socket says 172.21.2.102:53470) 2021-02-10T11:19:12.780 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:12.771 7fbfa7fff700 1 -- 172.21.2.102:0/106516514 learned_addr learned my addr 172.21.2.102:0/106516514 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:19:12.780 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:12.771 7fbfacc81700 1 --2- 172.21.2.102:0/106516514 >> v2:172.21.2.113:3300/0 conn(0x7fbfa81b0470 0x7fbfa81b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:19:12.780 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:12.771 7fbfa7fff700 1 -- 172.21.2.102:0/106516514 >> v2:172.21.2.113:3300/0 conn(0x7fbfa81b0470 msgr2=0x7fbfa81b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:19:12.780 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:12.771 7fbfa7fff700 1 --2- 172.21.2.102:0/106516514 >> v2:172.21.2.113:3300/0 conn(0x7fbfa81b0470 0x7fbfa81b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:12.780 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:12.771 7fbfa7fff700 1 -- 172.21.2.102:0/106516514 >> v1:172.21.2.113:6789/0 conn(0x7fbfa80f0030 legacy=0x7fbfa80f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:19:12.780 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:12.771 7fbfa7fff700 1 -- 172.21.2.102:0/106516514 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfa80f0430 con 0x7fbfa80ef610 2021-02-10T11:19:12.780 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:12.771 7fbfacc81700 1 --2- 172.21.2.102:0/106516514 >> v2:172.21.2.113:3300/0 conn(0x7fbfa81b0470 0x7fbfa81b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:19:12.780 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:12.772 7fbfa7fff700 1 --2- 172.21.2.102:0/106516514 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbfa80ef610 0x7fbfa80efa10 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 rx=0x7fbf98006a80 tx=0x7fbf98009f00).ready entity=mon.0 client_cookie=c27ba5267d43c89c server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:19:12.780 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:12.772 7fbfa77fe700 1 -- 172.21.2.102:0/106516514 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbf9800b5c0 con 0x7fbfa80ef610 2021-02-10T11:19:12.780 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:12.772 7fbfa77fe700 1 -- 172.21.2.102:0/106516514 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf9800b900 con 0x7fbfa80ef610 2021-02-10T11:19:12.781 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:12.772 7fbfa77fe700 1 -- 172.21.2.102:0/106516514 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbf9800bd60 con 0x7fbfa80ef610 2021-02-10T11:19:12.781 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:12.772 7fbfaee65700 1 -- 172.21.2.102:0/106516514 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbfa80ef610 msgr2=0x7fbfa80efa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:19:12.781 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:12.772 7fbfaee65700 1 --2- 172.21.2.102:0/106516514 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbfa80ef610 0x7fbfa80efa10 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 rx=0x7fbf98006a80 tx=0x7fbf98009f00).stop 2021-02-10T11:19:12.781 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:12.773 7fbfaee65700 1 -- 172.21.2.102:0/106516514 shutdown_connections 2021-02-10T11:19:12.781 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:12.773 7fbfaee65700 1 --2- 172.21.2.102:0/106516514 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbfa80ef610 0x7fbfa80efa10 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:12.781 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:12.773 7fbfaee65700 1 --2- 172.21.2.102:0/106516514 >> v2:172.21.2.113:3300/0 conn(0x7fbfa81b0470 0x7fbfa81b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:12.781 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:12.773 7fbfaee65700 1 -- 172.21.2.102:0/106516514 >> 172.21.2.102:0/106516514 conn(0x7fbfa80ce3e0 msgr2=0x7fbfa80e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:19:12.781 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:12.773 7fbfaee65700 1 -- 172.21.2.102:0/106516514 shutdown_connections 2021-02-10T11:19:12.781 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:12.773 7fbfaee65700 1 -- 172.21.2.102:0/106516514 wait complete. 2021-02-10T11:19:12.783 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:12.788 7fbfaee65700 1 Processor -- start 2021-02-10T11:19:12.783 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:12.788 7fbfaee65700 1 -- start start 2021-02-10T11:19:12.783 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:12.788 7fbfaee65700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbfa80a0420 0x7fbfa80a0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:12.783 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:12.788 7fbfaee65700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbfa8189d00 0x7fbfa818a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:12.783 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:12.788 7fbfaee65700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fbfa818a720 0x7fbfa80b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:12.783 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:12.788 7fbfaee65700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfa8189780 con 0x7fbfa80a0420 2021-02-10T11:19:12.783 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:12.788 7fbfaee65700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfa818ab20 con 0x7fbfa818a720 2021-02-10T11:19:12.783 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:12.788 7fbfaee65700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfa80a1140 con 0x7fbfa8189d00 2021-02-10T11:19:12.783 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:12.788 7fbfacc81700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbfa80a0420 0x7fbfa80a0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:19:12.783 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:12.788 7fbfacc81700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbfa80a0420 0x7fbfa80a0820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:53476/0 (socket says 172.21.2.102:53476) 2021-02-10T11:19:12.783 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:12.788 7fbfacc81700 1 -- 172.21.2.102:0/792101770 learned_addr learned my addr 172.21.2.102:0/792101770 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:19:12.783 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:12.788 7fbfa7fff700 1 --2- 172.21.2.102:0/792101770 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbfa8189d00 0x7fbfa818a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:19:12.783 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:12.788 7fbfad482700 1 --2- 172.21.2.102:0/792101770 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fbfa818a720 0x7fbfa80b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:19:12.784 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:12.789 7fbfacc81700 1 -- 172.21.2.102:0/792101770 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbfa8189d00 msgr2=0x7fbfa818a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:19:12.784 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:12.789 7fbfacc81700 1 --2- 172.21.2.102:0/792101770 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbfa8189d00 0x7fbfa818a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:12.784 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:12.789 7fbfacc81700 1 -- 172.21.2.102:0/792101770 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fbfa818a720 msgr2=0x7fbfa80b7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:19:12.784 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:12.789 7fbfacc81700 1 --2- 172.21.2.102:0/792101770 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fbfa818a720 0x7fbfa80b7be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:12.784 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:12.789 7fbfacc81700 1 -- 172.21.2.102:0/792101770 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfa80b8200 con 0x7fbfa80a0420 2021-02-10T11:19:12.784 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:12.789 7fbfacc81700 1 --2- 172.21.2.102:0/792101770 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbfa80a0420 0x7fbfa80a0820 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 rx=0x7fbf9000b580 tx=0x7fbf9000b920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:19:12.784 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:12.789 7fbfa57fa700 1 -- 172.21.2.102:0/792101770 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbf9000bd10 con 0x7fbfa80a0420 2021-02-10T11:19:12.784 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:12.789 7fbfa57fa700 1 -- 172.21.2.102:0/792101770 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf90000c90 con 0x7fbfa80a0420 2021-02-10T11:19:12.784 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:12.789 7fbfaee65700 1 -- 172.21.2.102:0/792101770 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfa80b8200 con 0x7fbfa80a0420 2021-02-10T11:19:12.784 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:12.789 7fbfa57fa700 1 -- 172.21.2.102:0/792101770 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbf90002220 con 0x7fbfa80a0420 2021-02-10T11:19:12.784 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:12.789 7fbfaee65700 1 -- 172.21.2.102:0/792101770 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfa80b8790 con 0x7fbfa80a0420 2021-02-10T11:19:12.784 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:12.789 7fbfa57fa700 1 -- 172.21.2.102:0/792101770 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fbf9000a000 con 0x7fbfa80a0420 2021-02-10T11:19:12.784 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:12.789 7fbfa57fa700 1 --2- 172.21.2.102:0/792101770 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fbf8c0186d0 0x7fbf8c01abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:12.785 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:12.790 7fbfa57fa700 1 -- 172.21.2.102:0/792101770 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fbf90032820 con 0x7fbfa80a0420 2021-02-10T11:19:12.785 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:12.790 7fbfa7fff700 1 --2- 172.21.2.102:0/792101770 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fbf8c0186d0 0x7fbf8c01abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:19:12.785 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:12.790 7fbfa7fff700 1 --2- 172.21.2.102:0/792101770 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fbf8c0186d0 0x7fbf8c01abd0 crc :-1 s=READY pgs=367 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:19:12.812 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:12.820 7fbfaee65700 1 -- 172.21.2.102:0/792101770 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbfa80a1360 con 0x7fbfa80a0420 2021-02-10T11:19:12.814 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:12.822 7fbfa57fa700 1 -- 172.21.2.102:0/792101770 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fbf90000940 con 0x7fbfa80a0420 2021-02-10T11:19:12.940 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:12.948 7fbfaee65700 1 -- 172.21.2.102:0/792101770 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344455", "target": ["mgr", ""]}) v1 -- 0x7fbfa80b9020 con 0x7fbf8c0186d0 2021-02-10T11:19:12.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:12.974 7fbfa57fa700 1 -- 172.21.2.102:0/792101770 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fbf98000d30 con 0x7fbf8c0186d0 2021-02-10T11:19:12.973 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:12.981 7fbfaee65700 1 -- 172.21.2.102:0/792101770 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fbf8c0186d0 msgr2=0x7fbf8c01abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:19:12.974 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:12.981 7fbfaee65700 1 --2- 172.21.2.102:0/792101770 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fbf8c0186d0 0x7fbf8c01abd0 crc :-1 s=READY pgs=367 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:12.974 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:12.981 7fbfaee65700 1 -- 172.21.2.102:0/792101770 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbfa80a0420 msgr2=0x7fbfa80a0820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:19:12.974 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:12.981 7fbfaee65700 1 --2- 172.21.2.102:0/792101770 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbfa80a0420 0x7fbfa80a0820 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 rx=0x7fbf9000b580 tx=0x7fbf9000b920).stop 2021-02-10T11:19:12.974 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:12.982 7fbfaee65700 1 -- 172.21.2.102:0/792101770 shutdown_connections 2021-02-10T11:19:12.974 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:12.982 7fbfaee65700 1 --2- 172.21.2.102:0/792101770 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fbf8c0186d0 0x7fbf8c01abd0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:12.974 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:12.982 7fbfaee65700 1 --2- 172.21.2.102:0/792101770 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fbfa818a720 0x7fbfa80b7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:12.974 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:12.982 7fbfaee65700 1 --2- 172.21.2.102:0/792101770 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbfa8189d00 0x7fbfa818a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:12.974 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:12.982 7fbfaee65700 1 --2- 172.21.2.102:0/792101770 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbfa80a0420 0x7fbfa80a0820 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:12.974 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:12.982 7fbfaee65700 1 -- 172.21.2.102:0/792101770 >> 172.21.2.102:0/792101770 conn(0x7fbfa8094e10 msgr2=0x7fbfa80a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:19:12.974 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:12.982 7fbfaee65700 1 -- 172.21.2.102:0/792101770 shutdown_connections 2021-02-10T11:19:12.974 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:12.982 7fbfaee65700 1 -- 172.21.2.102:0/792101770 wait complete. 2021-02-10T11:19:12.982 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000344455 2021-02-10T11:19:13.416 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:13.409 7f46d15b1700 1 Processor -- start 2021-02-10T11:19:13.416 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:13.409 7f46d15b1700 1 -- start start 2021-02-10T11:19:13.416 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:13.409 7f46d15b1700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f46cc1b0470 0x7f46cc1b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:13.417 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:13.409 7f46d15b1700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f46cc0f20e0 0x7f46cc0f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:13.417 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:13.409 7f46d15b1700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46cc0f0b50 con 0x7f46cc0ef610 2021-02-10T11:19:13.417 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:13.409 7f46d15b1700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46cc0f0820 con 0x7f46cc0f20e0 2021-02-10T11:19:13.417 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:13.409 7f46d15b1700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46cc0f24e0 con 0x7f46cc1b0470 2021-02-10T11:19:13.417 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:13.409 7f46cbac0700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f46cc0f20e0 0x7f46cc0f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:19:13.417 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:13.409 7f46cbac0700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f46cc0f20e0 0x7f46cc0f0200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:53480/0 (socket says 172.21.2.102:53480) 2021-02-10T11:19:13.417 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:13.409 7f46cbac0700 1 -- 172.21.2.102:0/4200353381 learned_addr learned my addr 172.21.2.102:0/4200353381 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:19:13.417 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:13.409 7f46caabe700 1 --1- 172.21.2.102:0/4200353381 >> v1:172.21.2.102:6789/0 conn(0x7f46cc0ef610 0x7f46cc0efa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:45704/0 (socket says 172.21.2.102:45704) 2021-02-10T11:19:13.417 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:13.410 7f46cbac0700 1 -- 172.21.2.102:0/4200353381 >> v2:172.21.2.113:3300/0 conn(0x7f46cc1b0470 msgr2=0x7f46cc1b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:19:13.417 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:13.410 7f46cbac0700 1 --2- 172.21.2.102:0/4200353381 >> v2:172.21.2.113:3300/0 conn(0x7f46cc1b0470 0x7f46cc1b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:19:13.417 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:13.410 7f46cbac0700 1 -- 172.21.2.102:0/4200353381 >> v1:172.21.2.102:6789/0 conn(0x7f46cc0ef610 legacy=0x7f46cc0efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:19:13.417 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:13.410 7f46cbac0700 1 -- 172.21.2.102:0/4200353381 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46cc0e52e0 con 0x7f46cc0f20e0 2021-02-10T11:19:13.417 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:13.410 7f46cbac0700 1 --2- 172.21.2.102:0/4200353381 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f46cc0f20e0 0x7f46cc0f0200 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 rx=0x7f46c4006f00 tx=0x7f46c4009f10).ready entity=mon.0 client_cookie=7305f75dcd505967 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:19:13.418 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:13.410 7f46ca2bd700 1 -- 172.21.2.102:0/4200353381 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f46c400b3b0 con 0x7f46cc0f20e0 2021-02-10T11:19:13.418 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:13.410 7f46ca2bd700 1 -- 172.21.2.102:0/4200353381 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46c400b630 con 0x7f46cc0f20e0 2021-02-10T11:19:13.418 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:13.410 7f46ca2bd700 1 -- 172.21.2.102:0/4200353381 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f46c400bb50 con 0x7f46cc0f20e0 2021-02-10T11:19:13.418 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:13.410 7f46d15b1700 1 -- 172.21.2.102:0/4200353381 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f46cc0f20e0 msgr2=0x7f46cc0f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:19:13.418 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:13.410 7f46d15b1700 1 --2- 172.21.2.102:0/4200353381 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f46cc0f20e0 0x7f46cc0f0200 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 rx=0x7f46c4006f00 tx=0x7f46c4009f10).stop 2021-02-10T11:19:13.418 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:13.410 7f46d15b1700 1 -- 172.21.2.102:0/4200353381 shutdown_connections 2021-02-10T11:19:13.418 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:13.410 7f46d15b1700 1 --2- 172.21.2.102:0/4200353381 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f46cc0f20e0 0x7f46cc0f0200 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:13.418 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:13.410 7f46d15b1700 1 --2- 172.21.2.102:0/4200353381 >> v2:172.21.2.113:3300/0 conn(0x7f46cc1b0470 0x7f46cc1b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:19:13.418 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:13.410 7f46d15b1700 1 -- 172.21.2.102:0/4200353381 >> 172.21.2.102:0/4200353381 conn(0x7f46cc0ce3e0 msgr2=0x7f46cc0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:19:13.418 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:13.410 7f46d15b1700 1 -- 172.21.2.102:0/4200353381 shutdown_connections 2021-02-10T11:19:13.418 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:13.410 7f46d15b1700 1 -- 172.21.2.102:0/4200353381 wait complete. 2021-02-10T11:19:13.420 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:13.425 7f46d15b1700 1 Processor -- start 2021-02-10T11:19:13.420 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:13.425 7f46d15b1700 1 -- start start 2021-02-10T11:19:13.420 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:13.425 7f46d15b1700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f46cc0c0b90 0x7f46cc0c0f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:13.420 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:13.425 7f46d15b1700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f46cc0c15b0 0x7f46cc0c19b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:13.420 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:13.425 7f46d15b1700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f46cc0aa4e0 0x7f46cc0aa8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:13.420 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:13.425 7f46d15b1700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46cc199f40 con 0x7f46cc0c0b90 2021-02-10T11:19:13.420 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:13.425 7f46d15b1700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46cc094bb0 con 0x7f46cc0aa4e0 2021-02-10T11:19:13.420 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:13.425 7f46d15b1700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46cc0aaf00 con 0x7f46cc0c15b0 2021-02-10T11:19:13.420 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:13.425 7f46cb2bf700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f46cc0c0b90 0x7f46cc0c0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:19:13.420 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:13.425 7f46cb2bf700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f46cc0c0b90 0x7f46cc0c0f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:53488/0 (socket says 172.21.2.102:53488) 2021-02-10T11:19:13.420 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:13.425 7f46cb2bf700 1 -- 172.21.2.102:0/3781892378 learned_addr learned my addr 172.21.2.102:0/3781892378 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:19:13.421 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:13.426 7f46cbac0700 1 --2- 172.21.2.102:0/3781892378 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f46cc0aa4e0 0x7f46cc0aa8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:19:13.421 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:13.426 7f46cb2bf700 1 -- 172.21.2.102:0/3781892378 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f46cc0c15b0 msgr2=0x7f46cc0c19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:19:13.421 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:13.426 7f46cb2bf700 1 --2- 172.21.2.102:0/3781892378 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f46cc0c15b0 0x7f46cc0c19b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:19:13.421 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:13.426 7f46cb2bf700 1 -- 172.21.2.102:0/3781892378 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f46cc0aa4e0 msgr2=0x7f46cc0aa8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:19:13.421 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:13.426 7f46cb2bf700 1 --2- 172.21.2.102:0/3781892378 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f46cc0aa4e0 0x7f46cc0aa8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:13.421 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:13.426 7f46cb2bf700 1 -- 172.21.2.102:0/3781892378 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46cc0ab120 con 0x7f46cc0c0b90 2021-02-10T11:19:13.421 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:13.426 7f46cbac0700 1 --2- 172.21.2.102:0/3781892378 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f46cc0aa4e0 0x7f46cc0aa8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:19:13.421 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:13.426 7f46cb2bf700 1 --2- 172.21.2.102:0/3781892378 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f46cc0c0b90 0x7f46cc0c0f90 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 rx=0x7f46bc00d760 tx=0x7f46bc00db00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:19:13.421 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:13.426 7f46c3fff700 1 -- 172.21.2.102:0/3781892378 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f46bc00b030 con 0x7f46cc0c0b90 2021-02-10T11:19:13.421 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:13.426 7f46c3fff700 1 -- 172.21.2.102:0/3781892378 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46bc000c90 con 0x7f46cc0c0b90 2021-02-10T11:19:13.421 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:13.426 7f46c3fff700 1 -- 172.21.2.102:0/3781892378 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f46bc002210 con 0x7f46cc0c0b90 2021-02-10T11:19:13.421 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:13.426 7f46d15b1700 1 -- 172.21.2.102:0/3781892378 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46cc0ab120 con 0x7f46cc0c0b90 2021-02-10T11:19:13.421 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:13.426 7f46d15b1700 1 -- 172.21.2.102:0/3781892378 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46cc0ab0f0 con 0x7f46cc0c0b90 2021-02-10T11:19:13.421 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:13.427 7f46c3fff700 1 -- 172.21.2.102:0/3781892378 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f46bc005310 con 0x7f46cc0c0b90 2021-02-10T11:19:13.422 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:13.427 7f46c3fff700 1 --2- 172.21.2.102:0/3781892378 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f46b00186b0 0x7f46b001abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:13.422 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:13.427 7f46c3fff700 1 -- 172.21.2.102:0/3781892378 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f46bc032440 con 0x7f46cc0c0b90 2021-02-10T11:19:13.422 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:13.427 7f46caabe700 1 --2- 172.21.2.102:0/3781892378 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f46b00186b0 0x7f46b001abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:19:13.422 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:13.427 7f46caabe700 1 --2- 172.21.2.102:0/3781892378 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f46b00186b0 0x7f46b001abd0 crc :-1 s=READY pgs=368 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:19:13.450 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:13.459 7f46d15b1700 1 -- 172.21.2.102:0/3781892378 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46cc105af0 con 0x7f46cc0c0b90 2021-02-10T11:19:13.453 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:13.461 7f46c3fff700 1 -- 172.21.2.102:0/3781892378 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f46bc02df20 con 0x7f46cc0c0b90 2021-02-10T11:19:13.580 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:13.588 7f46d15b1700 1 -- 172.21.2.102:0/3781892378 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344455", "target": ["mgr", ""]}) v1 -- 0x7f46cc0ab9a0 con 0x7f46b00186b0 2021-02-10T11:19:13.583 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:13.591 7f46c3fff700 1 -- 172.21.2.102:0/3781892378 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f46b4001590 con 0x7f46b00186b0 2021-02-10T11:19:13.587 INFO:teuthology.orchestra.run.gibba002.stdout:/volumes/_nogroup/subvolume_0000000000344455/4a28d807-2be4-4438-b58e-93cdf2a439ee 2021-02-10T11:19:13.589 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:13.596 7f46d15b1700 1 -- 172.21.2.102:0/3781892378 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f46b00186b0 msgr2=0x7f46b001abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:19:13.589 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:13.596 7f46d15b1700 1 --2- 172.21.2.102:0/3781892378 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f46b00186b0 0x7f46b001abd0 crc :-1 s=READY pgs=368 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:13.589 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:13.596 7f46d15b1700 1 -- 172.21.2.102:0/3781892378 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f46cc0c0b90 msgr2=0x7f46cc0c0f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:19:13.589 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:13.596 7f46d15b1700 1 --2- 172.21.2.102:0/3781892378 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f46cc0c0b90 0x7f46cc0c0f90 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 rx=0x7f46bc00d760 tx=0x7f46bc00db00).stop 2021-02-10T11:19:13.589 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:13.597 7f46d15b1700 1 -- 172.21.2.102:0/3781892378 shutdown_connections 2021-02-10T11:19:13.589 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:13.597 7f46d15b1700 1 --2- 172.21.2.102:0/3781892378 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f46b00186b0 0x7f46b001abd0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:13.589 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:13.597 7f46d15b1700 1 --2- 172.21.2.102:0/3781892378 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f46cc0aa4e0 0x7f46cc0aa8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:13.590 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:13.597 7f46d15b1700 1 --2- 172.21.2.102:0/3781892378 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f46cc0c15b0 0x7f46cc0c19b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:19:13.590 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:13.597 7f46d15b1700 1 --2- 172.21.2.102:0/3781892378 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f46cc0c0b90 0x7f46cc0c0f90 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:13.590 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:13.597 7f46d15b1700 1 -- 172.21.2.102:0/3781892378 >> 172.21.2.102:0/3781892378 conn(0x7f46cc0c40c0 msgr2=0x7f46cc0a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:19:13.590 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:13.597 7f46d15b1700 1 -- 172.21.2.102:0/3781892378 shutdown_connections 2021-02-10T11:19:13.590 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:13.597 7f46d15b1700 1 -- 172.21.2.102:0/3781892378 wait complete. 2021-02-10T11:19:13.597 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000344455 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000344455/4a28d807-2be4-4438-b58e-93cdf2a439ee 2021-02-10T11:19:13.597 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:19:13.597 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:19:13.597 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:19:13.933 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:19:13.934 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:19:13.934 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344455/4a28d807-2be4-4438-b58e-93cdf2a439ee/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:19:14.313 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:19:14.313 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:19:14.313 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0238357 s, 44.0 MB/s 2021-02-10T11:19:14.314 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:19:14.314 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:19:14.314 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:19:14.650 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:19:14.651 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:19:14.651 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344455/4a28d807-2be4-4438-b58e-93cdf2a439ee/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:19:15.027 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:19:15.027 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:19:15.027 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0242371 s, 43.3 MB/s 2021-02-10T11:19:15.028 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:19:15.028 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:19:15.029 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:19:15.365 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:19:15.365 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:19:15.365 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344455/4a28d807-2be4-4438-b58e-93cdf2a439ee/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:19:15.742 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:19:15.742 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:19:15.742 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0249474 s, 42.0 MB/s 2021-02-10T11:19:15.743 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:19:15.743 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:19:15.743 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:19:16.079 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:19:16.080 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:19:16.080 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344455/4a28d807-2be4-4438-b58e-93cdf2a439ee/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:19:16.457 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:19:16.457 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:19:16.457 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0248338 s, 42.2 MB/s 2021-02-10T11:19:16.458 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:19:16.458 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:19:16.459 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:19:16.794 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:19:16.795 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:19:16.795 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344455/4a28d807-2be4-4438-b58e-93cdf2a439ee/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:19:17.169 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:19:17.169 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:19:17.169 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0237323 s, 44.2 MB/s 2021-02-10T11:19:17.170 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:19:17.171 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:19:17.171 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:19:17.507 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:19:17.507 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:19:17.507 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344455/4a28d807-2be4-4438-b58e-93cdf2a439ee/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:19:17.884 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:19:17.884 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:19:17.884 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0243621 s, 43.0 MB/s 2021-02-10T11:19:17.886 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:19:17.887 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:19:17.887 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:19:18.224 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:19:18.224 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:19:18.224 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344455/4a28d807-2be4-4438-b58e-93cdf2a439ee/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:19:18.597 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:19:18.597 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:19:18.597 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0234494 s, 44.7 MB/s 2021-02-10T11:19:18.598 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:19:18.598 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:19:18.598 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:19:18.935 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:19:18.935 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:19:18.936 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344455/4a28d807-2be4-4438-b58e-93cdf2a439ee/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:19:19.313 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:19:19.313 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:19:19.313 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0235844 s, 44.5 MB/s 2021-02-10T11:19:19.314 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:19:19.314 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:19:19.314 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:19:19.649 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:19:19.650 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:19:19.650 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344455/4a28d807-2be4-4438-b58e-93cdf2a439ee/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:19:20.027 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:19:20.027 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:19:20.027 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0248055 s, 42.3 MB/s 2021-02-10T11:19:20.028 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:19:20.029 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:19:20.029 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:19:20.366 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:19:20.366 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:19:20.367 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344455/4a28d807-2be4-4438-b58e-93cdf2a439ee/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:19:20.744 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:19:20.744 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:19:20.744 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0251311 s, 41.7 MB/s 2021-02-10T11:19:20.745 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000344456 2021-02-10T11:19:21.180 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:21.171 7fc8cc17a700 1 Processor -- start 2021-02-10T11:19:21.180 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:21.172 7fc8cc17a700 1 -- start start 2021-02-10T11:19:21.180 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:21.172 7fc8cc17a700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc8c41b0470 0x7fc8c41b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:21.180 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:21.172 7fc8cc17a700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8c40f0b50 con 0x7fc8c40ef610 2021-02-10T11:19:21.180 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:21.172 7fc8cc17a700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8c40f24e0 con 0x7fc8c40f20e0 2021-02-10T11:19:21.180 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:21.172 7fc8cc17a700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8c40e5ea0 con 0x7fc8c41b0470 2021-02-10T11:19:21.180 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:21.172 7fc8c9795700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fc8c40ef610 0x7fc8c40efa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:45716/0 (socket says 172.21.2.102:45716) 2021-02-10T11:19:21.180 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:21.172 7fc8c9795700 1 -- 172.21.2.102:0/2671239836 learned_addr learned my addr 172.21.2.102:0/2671239836 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:19:21.181 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:21.172 7fc8c9f96700 1 --2- 172.21.2.102:0/2671239836 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc8c41b0470 0x7fc8c41b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:19:21.181 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:21.173 7fc8c8f94700 1 -- 172.21.2.102:0/2671239836 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1002752476 0 0) 0x7fc8b40010b0 con 0x7fc8c40ef610 2021-02-10T11:19:21.181 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:21.173 7fc8c8f94700 1 -- 172.21.2.102:0/2671239836 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8b0002fe0 con 0x7fc8c40ef610 2021-02-10T11:19:21.181 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:21.173 7fc8c8f94700 1 -- 172.21.2.102:0/2671239836 <== mon.1 v1:172.21.2.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2297783862 0 0) 0x7fc8bc001400 con 0x7fc8c40f20e0 2021-02-10T11:19:21.181 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:21.173 7fc8c9f96700 1 -- 172.21.2.102:0/2671239836 >> v1:172.21.2.113:6789/0 conn(0x7fc8c40f20e0 legacy=0x7fc8c40f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:19:21.181 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:21.173 7fc8c9f96700 1 -- 172.21.2.102:0/2671239836 >> v1:172.21.2.102:6789/0 conn(0x7fc8c40ef610 legacy=0x7fc8c40efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:19:21.181 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:21.173 7fc8c9f96700 1 -- 172.21.2.102:0/2671239836 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8c40e52e0 con 0x7fc8c41b0470 2021-02-10T11:19:21.181 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:21.173 7fc8c9f96700 1 --2- 172.21.2.102:0/2671239836 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc8c41b0470 0x7fc8c41b0870 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7fc8c0006650 tx=0x7fc8c0009a80).ready entity=mon.2 client_cookie=81f1b948c1b1f94a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:19:21.181 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:21.174 7fc8c8f94700 1 -- 172.21.2.102:0/2671239836 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc8c000c0c0 con 0x7fc8c41b0470 2021-02-10T11:19:21.181 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:21.174 7fc8c8f94700 1 -- 172.21.2.102:0/2671239836 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8c000c340 con 0x7fc8c41b0470 2021-02-10T11:19:21.181 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:21.174 7fc8c8f94700 1 -- 172.21.2.102:0/2671239836 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc8c000c860 con 0x7fc8c41b0470 2021-02-10T11:19:21.181 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:21.174 7fc8cc17a700 1 -- 172.21.2.102:0/2671239836 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc8c41b0470 msgr2=0x7fc8c41b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:19:21.181 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:21.174 7fc8cc17a700 1 --2- 172.21.2.102:0/2671239836 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc8c41b0470 0x7fc8c41b0870 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7fc8c0006650 tx=0x7fc8c0009a80).stop 2021-02-10T11:19:21.182 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:21.174 7fc8cc17a700 1 -- 172.21.2.102:0/2671239836 shutdown_connections 2021-02-10T11:19:21.182 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:21.174 7fc8cc17a700 1 --2- 172.21.2.102:0/2671239836 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc8c41b0470 0x7fc8c41b0870 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:21.182 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:21.174 7fc8cc17a700 1 -- 172.21.2.102:0/2671239836 >> 172.21.2.102:0/2671239836 conn(0x7fc8c40ce3e0 msgr2=0x7fc8c40e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:19:21.182 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:21.175 7fc8cc17a700 1 -- 172.21.2.102:0/2671239836 shutdown_connections 2021-02-10T11:19:21.182 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:21.175 7fc8cc17a700 1 -- 172.21.2.102:0/2671239836 wait complete. 2021-02-10T11:19:21.183 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:21.189 7fc8cc17a700 1 Processor -- start 2021-02-10T11:19:21.183 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:21.189 7fc8cc17a700 1 -- start start 2021-02-10T11:19:21.183 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:21.189 7fc8cc17a700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc8c41a42f0 0x7fc8c41a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:21.183 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:21.190 7fc8cc17a700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc8c4196170 0x7fc8c4196570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:21.184 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:21.190 7fc8cc17a700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc8c408f5b0 0x7fc8c408f9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:21.184 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:21.190 7fc8cc17a700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8c4195bf0 con 0x7fc8c408f5b0 2021-02-10T11:19:21.184 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:21.190 7fc8cc17a700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8c40c4130 con 0x7fc8c41a42f0 2021-02-10T11:19:21.184 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:21.190 7fc8cc17a700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8c40a6e70 con 0x7fc8c4196170 2021-02-10T11:19:21.184 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:21.190 7fc8ca797700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc8c408f5b0 0x7fc8c408f9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:19:21.184 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:21.190 7fc8ca797700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc8c408f5b0 0x7fc8c408f9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:53498/0 (socket says 172.21.2.102:53498) 2021-02-10T11:19:21.184 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:21.190 7fc8ca797700 1 -- 172.21.2.102:0/1303324773 learned_addr learned my addr 172.21.2.102:0/1303324773 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:19:21.184 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:21.190 7fc8ca797700 1 -- 172.21.2.102:0/1303324773 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc8c4196170 msgr2=0x7fc8c4196570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:19:21.184 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:21.190 7fc8ca797700 1 --2- 172.21.2.102:0/1303324773 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc8c4196170 0x7fc8c4196570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:19:21.184 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:21.190 7fc8ca797700 1 -- 172.21.2.102:0/1303324773 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc8c41a42f0 msgr2=0x7fc8c41a46f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:19:21.184 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:21.190 7fc8ca797700 1 --2- 172.21.2.102:0/1303324773 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc8c41a42f0 0x7fc8c41a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:19:21.184 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:21.190 7fc8ca797700 1 -- 172.21.2.102:0/1303324773 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8c40a6a80 con 0x7fc8c408f5b0 2021-02-10T11:19:21.185 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:21.190 7fc8ca797700 1 --2- 172.21.2.102:0/1303324773 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc8c408f5b0 0x7fc8c408f9b0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 rx=0x7fc8bc009980 tx=0x7fc8bc00bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:19:21.185 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:21.190 7fc8ba7fc700 1 -- 172.21.2.102:0/1303324773 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc8bc003020 con 0x7fc8c408f5b0 2021-02-10T11:19:21.185 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:21.190 7fc8ba7fc700 1 -- 172.21.2.102:0/1303324773 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8bc0058a0 con 0x7fc8c408f5b0 2021-02-10T11:19:21.185 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:21.190 7fc8cc17a700 1 -- 172.21.2.102:0/1303324773 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8c40a6a80 con 0x7fc8c408f5b0 2021-02-10T11:19:21.185 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:21.190 7fc8ba7fc700 1 -- 172.21.2.102:0/1303324773 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc8bc005d00 con 0x7fc8c408f5b0 2021-02-10T11:19:21.185 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:21.191 7fc8cc17a700 1 -- 172.21.2.102:0/1303324773 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8c40a6a80 con 0x7fc8c408f5b0 2021-02-10T11:19:21.185 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:21.191 7fc8ba7fc700 1 -- 172.21.2.102:0/1303324773 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fc8bc003020 con 0x7fc8c408f5b0 2021-02-10T11:19:21.185 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:21.191 7fc8ba7fc700 1 --2- 172.21.2.102:0/1303324773 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc8ac018730 0x7fc8ac01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:21.185 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:21.191 7fc8ba7fc700 1 -- 172.21.2.102:0/1303324773 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fc8bc0329a0 con 0x7fc8c408f5b0 2021-02-10T11:19:21.185 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:21.191 7fc8c9f96700 1 --2- 172.21.2.102:0/1303324773 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc8ac018730 0x7fc8ac01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:19:21.185 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:21.192 7fc8c9f96700 1 --2- 172.21.2.102:0/1303324773 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc8ac018730 0x7fc8ac01ac10 crc :-1 s=READY pgs=369 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:19:21.213 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:21.222 7fc8cc17a700 1 -- 172.21.2.102:0/1303324773 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8c40f0b50 con 0x7fc8c408f5b0 2021-02-10T11:19:21.216 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:21.225 7fc8ba7fc700 1 -- 172.21.2.102:0/1303324773 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fc8bc00be10 con 0x7fc8c408f5b0 2021-02-10T11:19:21.343 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:21.351 7fc8cc17a700 1 -- 172.21.2.102:0/1303324773 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344456", "target": ["mgr", ""]}) v1 -- 0x7fc8c40f0b50 con 0x7fc8ac018730 2021-02-10T11:19:21.369 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:21.378 7fc8ba7fc700 1 -- 172.21.2.102:0/1303324773 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fc8c0000940 con 0x7fc8ac018730 2021-02-10T11:19:21.375 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:21.384 7fc8cc17a700 1 -- 172.21.2.102:0/1303324773 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc8ac018730 msgr2=0x7fc8ac01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:19:21.375 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:21.384 7fc8cc17a700 1 --2- 172.21.2.102:0/1303324773 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc8ac018730 0x7fc8ac01ac10 crc :-1 s=READY pgs=369 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:21.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:21.384 7fc8cc17a700 1 -- 172.21.2.102:0/1303324773 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc8c408f5b0 msgr2=0x7fc8c408f9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:19:21.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:21.384 7fc8cc17a700 1 --2- 172.21.2.102:0/1303324773 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc8c408f5b0 0x7fc8c408f9b0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 rx=0x7fc8bc009980 tx=0x7fc8bc00bcd0).stop 2021-02-10T11:19:21.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:21.385 7fc8cc17a700 1 -- 172.21.2.102:0/1303324773 shutdown_connections 2021-02-10T11:19:21.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:21.385 7fc8cc17a700 1 --2- 172.21.2.102:0/1303324773 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc8c408f5b0 0x7fc8c408f9b0 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:21.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:21.385 7fc8cc17a700 1 --2- 172.21.2.102:0/1303324773 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc8c4196170 0x7fc8c4196570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:19:21.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:21.385 7fc8cc17a700 1 --2- 172.21.2.102:0/1303324773 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc8ac018730 0x7fc8ac01ac10 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:21.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:21.385 7fc8cc17a700 1 --2- 172.21.2.102:0/1303324773 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc8c41a42f0 0x7fc8c41a46f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:19:21.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:21.385 7fc8cc17a700 1 -- 172.21.2.102:0/1303324773 >> 172.21.2.102:0/1303324773 conn(0x7fc8c40bf4e0 msgr2=0x7fc8c40be160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:19:21.377 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:21.385 7fc8cc17a700 1 -- 172.21.2.102:0/1303324773 shutdown_connections 2021-02-10T11:19:21.377 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:21.385 7fc8cc17a700 1 -- 172.21.2.102:0/1303324773 wait complete. 2021-02-10T11:19:21.384 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000344456 2021-02-10T11:19:21.816 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:21.809 7fe70e445700 1 Processor -- start 2021-02-10T11:19:21.816 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:21.809 7fe70e445700 1 -- start start 2021-02-10T11:19:21.816 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:21.810 7fe70e445700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fe7081b0470 0x7fe7081b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:21.816 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:21.810 7fe70e445700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe7080ef610 0x7fe7080efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:21.816 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:21.810 7fe70e445700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fe7080f0030 0x7fe7080f0430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:21.817 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:21.810 7fe70e445700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7080e5ea0 con 0x7fe7080f0030 2021-02-10T11:19:21.817 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:21.810 7fe70e445700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7080f0b50 con 0x7fe7081b0470 2021-02-10T11:19:21.817 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:21.810 7fe70e445700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7080e52e0 con 0x7fe7080ef610 2021-02-10T11:19:21.817 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:21.810 7fe70ca62700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fe7080f0030 0x7fe7080f0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:19:21.817 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:21.810 7fe70ca62700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fe7080f0030 0x7fe7080f0430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:53508/0 (socket says 172.21.2.102:53508) 2021-02-10T11:19:21.817 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:21.810 7fe70ca62700 1 -- 172.21.2.102:0/307839902 learned_addr learned my addr 172.21.2.102:0/307839902 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:19:21.817 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:21.810 7fe707fff700 1 --2- 172.21.2.102:0/307839902 >> v2:172.21.2.113:3300/0 conn(0x7fe7081b0470 0x7fe7081b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:19:21.817 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:21.810 7fe7077fe700 1 --2- 172.21.2.102:0/307839902 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe7080ef610 0x7fe7080efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:19:21.817 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:21.810 7fe70ca62700 1 -- 172.21.2.102:0/307839902 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe7080ef610 msgr2=0x7fe7080efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:19:21.817 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:21.810 7fe70ca62700 1 --2- 172.21.2.102:0/307839902 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe7080ef610 0x7fe7080efa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:21.817 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:21.810 7fe70ca62700 1 -- 172.21.2.102:0/307839902 >> v2:172.21.2.113:3300/0 conn(0x7fe7081b0470 msgr2=0x7fe7081b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:19:21.817 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:21.810 7fe70ca62700 1 --2- 172.21.2.102:0/307839902 >> v2:172.21.2.113:3300/0 conn(0x7fe7081b0470 0x7fe7081b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:21.818 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:21.810 7fe70ca62700 1 -- 172.21.2.102:0/307839902 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7080f10d0 con 0x7fe7080f0030 2021-02-10T11:19:21.818 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:21.810 7fe707fff700 1 --2- 172.21.2.102:0/307839902 >> v2:172.21.2.113:3300/0 conn(0x7fe7081b0470 0x7fe7081b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:19:21.818 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:21.810 7fe70ca62700 1 --2- 172.21.2.102:0/307839902 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe7080f0030 0x7fe7080f0430 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 rx=0x7fe700006f00 tx=0x7fe700009e90).ready entity=mon.0 client_cookie=6fb86e0c2ba1ae4f server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:19:21.818 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:21.810 7fe706ffd700 1 -- 172.21.2.102:0/307839902 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe70000b300 con 0x7fe7080f0030 2021-02-10T11:19:21.818 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:21.811 7fe706ffd700 1 -- 172.21.2.102:0/307839902 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe70000b580 con 0x7fe7080f0030 2021-02-10T11:19:21.818 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:21.811 7fe70e445700 1 -- 172.21.2.102:0/307839902 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe7080f0030 msgr2=0x7fe7080f0430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:19:21.818 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:21.811 7fe70e445700 1 --2- 172.21.2.102:0/307839902 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe7080f0030 0x7fe7080f0430 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 rx=0x7fe700006f00 tx=0x7fe700009e90).stop 2021-02-10T11:19:21.818 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:21.811 7fe706ffd700 1 -- 172.21.2.102:0/307839902 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe700000f90 con 0x7fe7080f0030 2021-02-10T11:19:21.818 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:21.811 7fe70e445700 1 -- 172.21.2.102:0/307839902 shutdown_connections 2021-02-10T11:19:21.819 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:21.811 7fe70e445700 1 --2- 172.21.2.102:0/307839902 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe7080ef610 0x7fe7080efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:21.819 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:21.811 7fe70e445700 1 --2- 172.21.2.102:0/307839902 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe7080f0030 0x7fe7080f0430 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:21.819 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:21.811 7fe70e445700 1 --2- 172.21.2.102:0/307839902 >> v2:172.21.2.113:3300/0 conn(0x7fe7081b0470 0x7fe7081b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:21.819 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:21.811 7fe70e445700 1 -- 172.21.2.102:0/307839902 >> 172.21.2.102:0/307839902 conn(0x7fe7080ce3e0 msgr2=0x7fe7080e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:19:21.819 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:21.811 7fe70e445700 1 -- 172.21.2.102:0/307839902 shutdown_connections 2021-02-10T11:19:21.819 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:21.811 7fe70e445700 1 -- 172.21.2.102:0/307839902 wait complete. 2021-02-10T11:19:21.822 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:21.825 7fe70e445700 1 Processor -- start 2021-02-10T11:19:21.822 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:21.826 7fe70e445700 1 -- start start 2021-02-10T11:19:21.822 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:21.826 7fe70e445700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe7081a0470 0x7fe708098600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:21.822 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:21.826 7fe70e445700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe708098c20 0x7fe708106a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:21.823 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:21.826 7fe70e445700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe708107060 0x7fe7080c0900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:21.823 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:21.826 7fe70e445700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe708107860 con 0x7fe708098c20 2021-02-10T11:19:21.823 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:21.826 7fe70e445700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe708107460 con 0x7fe7081a0470 2021-02-10T11:19:21.823 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:21.826 7fe70e445700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7080c0f20 con 0x7fe708107060 2021-02-10T11:19:21.823 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:21.826 7fe7077fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe708098c20 0x7fe708106a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:19:21.823 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:21.826 7fe7077fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe708098c20 0x7fe708106a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:53512/0 (socket says 172.21.2.102:53512) 2021-02-10T11:19:21.823 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:21.826 7fe7077fe700 1 -- 172.21.2.102:0/2977422499 learned_addr learned my addr 172.21.2.102:0/2977422499 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:19:21.823 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:21.826 7fe7077fe700 1 -- 172.21.2.102:0/2977422499 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe708107060 msgr2=0x7fe7080c0900 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:19:21.823 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:21.826 7fe7077fe700 1 --2- 172.21.2.102:0/2977422499 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe708107060 0x7fe7080c0900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:19:21.823 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:21.826 7fe7077fe700 1 -- 172.21.2.102:0/2977422499 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe7081a0470 msgr2=0x7fe708098600 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:19:21.823 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:21.826 7fe7077fe700 1 --2- 172.21.2.102:0/2977422499 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe7081a0470 0x7fe708098600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:19:21.823 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:21.826 7fe7077fe700 1 -- 172.21.2.102:0/2977422499 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7080c1140 con 0x7fe708098c20 2021-02-10T11:19:21.823 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:21.826 7fe7077fe700 1 --2- 172.21.2.102:0/2977422499 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe708098c20 0x7fe708106a40 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 rx=0x7fe6fc0067e0 tx=0x7fe6fc00b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:19:21.824 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:21.826 7fe704ff9700 1 -- 172.21.2.102:0/2977422499 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe6fc00bd80 con 0x7fe708098c20 2021-02-10T11:19:21.824 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:21.827 7fe704ff9700 1 -- 172.21.2.102:0/2977422499 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6fc006900 con 0x7fe708098c20 2021-02-10T11:19:21.824 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:21.827 7fe70e445700 1 -- 172.21.2.102:0/2977422499 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7080c1140 con 0x7fe708098c20 2021-02-10T11:19:21.824 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:21.827 7fe704ff9700 1 -- 172.21.2.102:0/2977422499 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe6fc002080 con 0x7fe708098c20 2021-02-10T11:19:21.824 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:21.827 7fe70e445700 1 -- 172.21.2.102:0/2977422499 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7080c1140 con 0x7fe708098c20 2021-02-10T11:19:21.824 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:21.827 7fe704ff9700 1 -- 172.21.2.102:0/2977422499 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fe6fc000c90 con 0x7fe708098c20 2021-02-10T11:19:21.824 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:21.827 7fe704ff9700 1 --2- 172.21.2.102:0/2977422499 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe6e8018730 0x7fe6e801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:21.824 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:21.827 7fe704ff9700 1 -- 172.21.2.102:0/2977422499 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fe6fc032670 con 0x7fe708098c20 2021-02-10T11:19:21.824 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:21.827 7fe707fff700 1 --2- 172.21.2.102:0/2977422499 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe6e8018730 0x7fe6e801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:19:21.824 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:21.828 7fe707fff700 1 --2- 172.21.2.102:0/2977422499 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe6e8018730 0x7fe6e801ac10 crc :-1 s=READY pgs=370 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:19:21.850 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:21.859 7fe70e445700 1 -- 172.21.2.102:0/2977422499 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe708105b30 con 0x7fe708098c20 2021-02-10T11:19:21.853 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:21.862 7fe704ff9700 1 -- 172.21.2.102:0/2977422499 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fe6fc0068e0 con 0x7fe708098c20 2021-02-10T11:19:21.981 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:21.990 7fe70e445700 1 -- 172.21.2.102:0/2977422499 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344456", "target": ["mgr", ""]}) v1 -- 0x7fe708107460 con 0x7fe6e8018730 2021-02-10T11:19:21.985 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:21.994 7fe704ff9700 1 -- 172.21.2.102:0/2977422499 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7fe6f8000940 con 0x7fe6e8018730 2021-02-10T11:19:21.988 INFO:teuthology.orchestra.run.gibba002.stdout:/volumes/_nogroup/subvolume_0000000000344456/ff91af16-7a12-45cb-b808-2980c069dd8b 2021-02-10T11:19:21.990 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:21.999 7fe70e445700 1 -- 172.21.2.102:0/2977422499 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe6e8018730 msgr2=0x7fe6e801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:19:21.990 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:21.999 7fe70e445700 1 --2- 172.21.2.102:0/2977422499 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe6e8018730 0x7fe6e801ac10 crc :-1 s=READY pgs=370 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:21.990 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:21.999 7fe70e445700 1 -- 172.21.2.102:0/2977422499 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe708098c20 msgr2=0x7fe708106a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:19:21.990 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:21.999 7fe70e445700 1 --2- 172.21.2.102:0/2977422499 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe708098c20 0x7fe708106a40 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 rx=0x7fe6fc0067e0 tx=0x7fe6fc00b990).stop 2021-02-10T11:19:21.991 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:21.999 7fe70e445700 1 -- 172.21.2.102:0/2977422499 shutdown_connections 2021-02-10T11:19:21.991 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:21.999 7fe70e445700 1 --2- 172.21.2.102:0/2977422499 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe708107060 0x7fe7080c0900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:19:21.991 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:21.999 7fe70e445700 1 --2- 172.21.2.102:0/2977422499 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe708098c20 0x7fe708106a40 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:21.991 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:21.999 7fe70e445700 1 --2- 172.21.2.102:0/2977422499 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe6e8018730 0x7fe6e801ac10 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:21.991 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:21.999 7fe70e445700 1 --2- 172.21.2.102:0/2977422499 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe7081a0470 0x7fe708098600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:19:21.991 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:21.999 7fe70e445700 1 -- 172.21.2.102:0/2977422499 >> 172.21.2.102:0/2977422499 conn(0x7fe7080c40e0 msgr2=0x7fe7080b0130 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:19:21.991 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:21.999 7fe70e445700 1 -- 172.21.2.102:0/2977422499 shutdown_connections 2021-02-10T11:19:21.991 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:21.999 7fe70e445700 1 -- 172.21.2.102:0/2977422499 wait complete. 2021-02-10T11:19:21.998 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000344456 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000344456/ff91af16-7a12-45cb-b808-2980c069dd8b 2021-02-10T11:19:21.998 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:19:21.998 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:19:21.998 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:19:22.335 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:19:22.335 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:19:22.335 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344456/ff91af16-7a12-45cb-b808-2980c069dd8b/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:19:22.719 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:19:22.719 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:19:22.720 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.025034 s, 41.9 MB/s 2021-02-10T11:19:22.720 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:19:22.721 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:19:22.721 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:19:23.058 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:19:23.058 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:19:23.058 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344456/ff91af16-7a12-45cb-b808-2980c069dd8b/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:19:23.431 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:19:23.431 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:19:23.431 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0241395 s, 43.4 MB/s 2021-02-10T11:19:23.433 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:19:23.433 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:19:23.433 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:19:23.769 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:19:23.769 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:19:23.769 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344456/ff91af16-7a12-45cb-b808-2980c069dd8b/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:19:24.145 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:19:24.145 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:19:24.145 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0244019 s, 43.0 MB/s 2021-02-10T11:19:24.146 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:19:24.146 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:19:24.147 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:19:24.483 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:19:24.483 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:19:24.484 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344456/ff91af16-7a12-45cb-b808-2980c069dd8b/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:19:24.859 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:19:24.859 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:19:24.860 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0234726 s, 44.7 MB/s 2021-02-10T11:19:24.860 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:19:24.861 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:19:24.861 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:19:25.196 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:19:25.196 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:19:25.196 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344456/ff91af16-7a12-45cb-b808-2980c069dd8b/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:19:25.573 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:19:25.574 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:19:25.574 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0244872 s, 42.8 MB/s 2021-02-10T11:19:25.575 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:19:25.575 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:19:25.575 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:19:25.910 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:19:25.910 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:19:25.911 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344456/ff91af16-7a12-45cb-b808-2980c069dd8b/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:19:26.286 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:19:26.287 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:19:26.287 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.023431 s, 44.8 MB/s 2021-02-10T11:19:26.287 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:19:26.288 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:19:26.288 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:19:26.624 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:19:26.624 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:19:26.625 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344456/ff91af16-7a12-45cb-b808-2980c069dd8b/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:19:27.000 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:19:27.000 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:19:27.000 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0240576 s, 43.6 MB/s 2021-02-10T11:19:27.001 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:19:27.001 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:19:27.001 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:19:27.336 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:19:27.336 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:19:27.336 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344456/ff91af16-7a12-45cb-b808-2980c069dd8b/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:19:27.714 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:19:27.714 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:19:27.714 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0261438 s, 40.1 MB/s 2021-02-10T11:19:27.715 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:19:27.715 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:19:27.715 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:19:28.051 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:19:28.052 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:19:28.052 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344456/ff91af16-7a12-45cb-b808-2980c069dd8b/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:19:28.426 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:19:28.426 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:19:28.426 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0252138 s, 41.6 MB/s 2021-02-10T11:19:28.427 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:19:28.427 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:19:28.427 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:19:28.762 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:19:28.762 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:19:28.762 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344456/ff91af16-7a12-45cb-b808-2980c069dd8b/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:19:29.141 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:19:29.141 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:19:29.141 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0252165 s, 41.6 MB/s 2021-02-10T11:19:29.142 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000344457 2021-02-10T11:19:29.575 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:29.564 7f751ee7b700 1 Processor -- start 2021-02-10T11:19:29.575 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:29.565 7f751ee7b700 1 -- start start 2021-02-10T11:19:29.576 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:29.565 7f751ee7b700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f75181b0870 0x7f75180f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:29.576 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:29.565 7f751ee7b700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f75180efe00 0x7f75180f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:29.576 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:29.565 7f751ee7b700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75181b0c70 con 0x7f75181b0470 2021-02-10T11:19:29.576 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:29.565 7f751ee7b700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75180f0b50 con 0x7f75180efe00 2021-02-10T11:19:29.577 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:29.565 7f751ee7b700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75180f0820 con 0x7f75181b0870 2021-02-10T11:19:29.577 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:29.565 7f751d498700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f75180efe00 0x7f75180f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:19:29.577 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:29.565 7f751d498700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f75180efe00 0x7f75180f0200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:53518/0 (socket says 172.21.2.102:53518) 2021-02-10T11:19:29.577 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:29.565 7f751d498700 1 -- 172.21.2.102:0/144588313 learned_addr learned my addr 172.21.2.102:0/144588313 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:19:29.578 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:29.565 7f7517fff700 1 --2- 172.21.2.102:0/144588313 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f75181b0870 0x7f75180f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:19:29.578 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:29.566 7f751d498700 1 -- 172.21.2.102:0/144588313 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f75181b0870 msgr2=0x7f75180f20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:19:29.578 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:29.566 7f751d498700 1 --2- 172.21.2.102:0/144588313 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f75181b0870 0x7f75180f20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:29.578 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:29.566 7f751d498700 1 -- 172.21.2.102:0/144588313 >> v1:172.21.2.102:6789/0 conn(0x7f75181b0470 legacy=0x7f75180ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:19:29.579 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:29.566 7f751d498700 1 -- 172.21.2.102:0/144588313 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75180e52e0 con 0x7f75180efe00 2021-02-10T11:19:29.579 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:29.566 7f751d498700 1 --2- 172.21.2.102:0/144588313 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f75180efe00 0x7f75180f0200 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 rx=0x7f7510006b80 tx=0x7f7510009b90).ready entity=mon.0 client_cookie=ef5e1c6559bfed82 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:19:29.579 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:29.566 7f75177fe700 1 -- 172.21.2.102:0/144588313 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f75100022e0 con 0x7f75180efe00 2021-02-10T11:19:29.580 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:29.566 7f75177fe700 1 -- 172.21.2.102:0/144588313 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7510002560 con 0x7f75180efe00 2021-02-10T11:19:29.580 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:29.566 7f75177fe700 1 -- 172.21.2.102:0/144588313 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7510002a80 con 0x7f75180efe00 2021-02-10T11:19:29.580 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:29.566 7f751ee7b700 1 -- 172.21.2.102:0/144588313 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f75180efe00 msgr2=0x7f75180f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:19:29.580 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:29.566 7f751ee7b700 1 --2- 172.21.2.102:0/144588313 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f75180efe00 0x7f75180f0200 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 rx=0x7f7510006b80 tx=0x7f7510009b90).stop 2021-02-10T11:19:29.581 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:29.567 7f751ee7b700 1 -- 172.21.2.102:0/144588313 shutdown_connections 2021-02-10T11:19:29.581 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:29.567 7f751ee7b700 1 --2- 172.21.2.102:0/144588313 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f75180efe00 0x7f75180f0200 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:29.582 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:29.567 7f751ee7b700 1 --2- 172.21.2.102:0/144588313 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f75181b0870 0x7f75180f20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:29.582 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:29.567 7f751ee7b700 1 -- 172.21.2.102:0/144588313 >> 172.21.2.102:0/144588313 conn(0x7f75180ce3e0 msgr2=0x7f75180e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:19:29.582 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:29.567 7f751ee7b700 1 -- 172.21.2.102:0/144588313 shutdown_connections 2021-02-10T11:19:29.582 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:29.567 7f751ee7b700 1 -- 172.21.2.102:0/144588313 wait complete. 2021-02-10T11:19:29.587 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:29.582 7f751ee7b700 1 Processor -- start 2021-02-10T11:19:29.587 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:29.582 7f751ee7b700 1 -- start start 2021-02-10T11:19:29.587 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:29.582 7f751ee7b700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f75180a0420 0x7f75180a0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:29.587 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:29.582 7f751ee7b700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7518189d00 0x7f751818a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:29.587 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:29.582 7f751ee7b700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f751818a720 0x7f75180b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:29.587 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:29.582 7f751ee7b700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7518189780 con 0x7f75180a0420 2021-02-10T11:19:29.588 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:29.582 7f751ee7b700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f751818ab20 con 0x7f7518189d00 2021-02-10T11:19:29.588 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:29.582 7f751ee7b700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75180a1140 con 0x7f751818a720 2021-02-10T11:19:29.588 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:29.583 7f751d498700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f751818a720 0x7f75180b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:19:29.588 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:29.583 7f751cc97700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f75180a0420 0x7f75180a0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:19:29.588 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:29.583 7f7517fff700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7518189d00 0x7f751818a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:19:29.588 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:29.583 7f751d498700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f751818a720 0x7f75180b7be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.113:3301/0 says I am v2:172.21.2.102:59058/0 (socket says 172.21.2.102:59058) 2021-02-10T11:19:29.589 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:29.583 7f751d498700 1 -- 172.21.2.102:0/111614828 learned_addr learned my addr 172.21.2.102:0/111614828 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:19:29.589 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:29.583 7f751cc97700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f75180a0420 0x7f75180a0820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:53524/0 (socket says 172.21.2.102:53524) 2021-02-10T11:19:29.589 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:29.583 7f751d498700 1 -- 172.21.2.102:0/111614828 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7518189d00 msgr2=0x7f751818a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:19:29.589 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:29.583 7f751d498700 1 --2- 172.21.2.102:0/111614828 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7518189d00 0x7f751818a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:29.589 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:29.583 7f751d498700 1 -- 172.21.2.102:0/111614828 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f75180a0420 msgr2=0x7f75180a0820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:19:29.589 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:29.583 7f751d498700 1 --2- 172.21.2.102:0/111614828 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f75180a0420 0x7f75180a0820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:29.589 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:29.583 7f751d498700 1 -- 172.21.2.102:0/111614828 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75180b8200 con 0x7f751818a720 2021-02-10T11:19:29.589 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:29.583 7f751cc97700 1 --2- 172.21.2.102:0/111614828 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f75180a0420 0x7f75180a0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:19:29.590 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:29.583 7f751d498700 1 --2- 172.21.2.102:0/111614828 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f751818a720 0x7f75180b7be0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7f75100095c0 tx=0x7f751000b990).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:19:29.590 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:29.583 7f75157fa700 1 -- 172.21.2.102:0/111614828 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f751000bd80 con 0x7f751818a720 2021-02-10T11:19:29.591 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:29.584 7f75157fa700 1 -- 172.21.2.102:0/111614828 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7510006e20 con 0x7f751818a720 2021-02-10T11:19:29.591 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:29.584 7f75157fa700 1 -- 172.21.2.102:0/111614828 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f75100064b0 con 0x7f751818a720 2021-02-10T11:19:29.591 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:29.584 7f751ee7b700 1 -- 172.21.2.102:0/111614828 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75180b8200 con 0x7f751818a720 2021-02-10T11:19:29.592 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:29.584 7f751ee7b700 1 -- 172.21.2.102:0/111614828 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75180b8200 con 0x7f751818a720 2021-02-10T11:19:29.592 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:29.584 7f75157fa700 1 -- 172.21.2.102:0/111614828 <== mon.2 v2:172.21.2.113:3301/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f7510006240 con 0x7f751818a720 2021-02-10T11:19:29.593 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:29.584 7f75157fa700 1 --2- 172.21.2.102:0/111614828 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f74fc0186c0 0x7f74fc01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:29.593 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:29.584 7f75157fa700 1 -- 172.21.2.102:0/111614828 <== mon.2 v2:172.21.2.113:3301/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f75100325b0 con 0x7f751818a720 2021-02-10T11:19:29.593 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:29.584 7f751cc97700 1 --2- 172.21.2.102:0/111614828 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f74fc0186c0 0x7f74fc01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:19:29.593 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:29.585 7f751cc97700 1 --2- 172.21.2.102:0/111614828 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f74fc0186c0 0x7f74fc01abc0 crc :-1 s=READY pgs=371 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:19:29.607 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:29.615 7f751ee7b700 1 -- 172.21.2.102:0/111614828 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7518105ae0 con 0x7f751818a720 2021-02-10T11:19:29.611 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:29.619 7f75157fa700 1 -- 172.21.2.102:0/111614828 <== mon.2 v2:172.21.2.113:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f7510006030 con 0x7f751818a720 2021-02-10T11:19:29.739 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:29.747 7f751ee7b700 1 -- 172.21.2.102:0/111614828 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344457", "target": ["mgr", ""]}) v1 -- 0x7f75180f0b50 con 0x7f74fc0186c0 2021-02-10T11:19:29.766 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:29.773 7f75157fa700 1 -- 172.21.2.102:0/111614828 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f7500006a40 con 0x7f74fc0186c0 2021-02-10T11:19:29.772 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:29.780 7f751ee7b700 1 -- 172.21.2.102:0/111614828 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f74fc0186c0 msgr2=0x7f74fc01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:19:29.772 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:29.780 7f751ee7b700 1 --2- 172.21.2.102:0/111614828 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f74fc0186c0 0x7f74fc01abc0 crc :-1 s=READY pgs=371 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:29.773 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:29.780 7f751ee7b700 1 -- 172.21.2.102:0/111614828 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f751818a720 msgr2=0x7f75180b7be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:19:29.773 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:29.780 7f751ee7b700 1 --2- 172.21.2.102:0/111614828 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f751818a720 0x7f75180b7be0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7f75100095c0 tx=0x7f751000b990).stop 2021-02-10T11:19:29.773 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:29.780 7f751ee7b700 1 -- 172.21.2.102:0/111614828 shutdown_connections 2021-02-10T11:19:29.773 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:29.780 7f751ee7b700 1 --2- 172.21.2.102:0/111614828 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f751818a720 0x7f75180b7be0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:29.773 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:29.780 7f751ee7b700 1 --2- 172.21.2.102:0/111614828 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f74fc0186c0 0x7f74fc01abc0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:29.773 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:29.780 7f751ee7b700 1 --2- 172.21.2.102:0/111614828 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7518189d00 0x7f751818a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:29.773 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:29.780 7f751ee7b700 1 --2- 172.21.2.102:0/111614828 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f75180a0420 0x7f75180a0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:29.773 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:29.780 7f751ee7b700 1 -- 172.21.2.102:0/111614828 >> 172.21.2.102:0/111614828 conn(0x7f7518094e10 msgr2=0x7f75180a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:19:29.773 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:29.780 7f751ee7b700 1 -- 172.21.2.102:0/111614828 shutdown_connections 2021-02-10T11:19:29.774 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:29.780 7f751ee7b700 1 -- 172.21.2.102:0/111614828 wait complete. 2021-02-10T11:19:29.781 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000344457 2021-02-10T11:19:30.074 DEBUG:teuthology.orchestra.run.gibba002:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:19:30.078 DEBUG:teuthology.orchestra.run.gibba013:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:19:30.216 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:30.207 7f38fdc3e700 1 Processor -- start 2021-02-10T11:19:30.217 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:30.207 7f38fdc3e700 1 -- start start 2021-02-10T11:19:30.217 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:30.208 7f38fdc3e700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f38f80f05f0 0x7f38f80f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:30.217 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:30.208 7f38fdc3e700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38f81b0c70 con 0x7f38f81b0870 2021-02-10T11:19:30.217 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:30.208 7f38fdc3e700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38f80f0b50 con 0x7f38f81b0470 2021-02-10T11:19:30.217 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:30.208 7f38fdc3e700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38f80e5ea0 con 0x7f38f80f05f0 2021-02-10T11:19:30.217 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:30.208 7f38f6ffd700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f38f81b0870 0x7f38f80efe00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:45754/0 (socket says 172.21.2.102:45754) 2021-02-10T11:19:30.217 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:30.208 7f38f6ffd700 1 -- 172.21.2.102:0/2547422588 learned_addr learned my addr 172.21.2.102:0/2547422588 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:19:30.217 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:30.208 7f38f7fff700 1 --2- 172.21.2.102:0/2547422588 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f38f80f05f0 0x7f38f80f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:19:30.217 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:30.209 7f38f67fc700 1 -- 172.21.2.102:0/2547422588 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 945585789 0 0) 0x7f38e0000ce0 con 0x7f38f81b0870 2021-02-10T11:19:30.217 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:30.209 7f38f67fc700 1 -- 172.21.2.102:0/2547422588 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38e4002fe0 con 0x7f38f81b0870 2021-02-10T11:19:30.217 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:30.209 7f38f67fc700 1 -- 172.21.2.102:0/2547422588 <== mon.1 v1:172.21.2.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1434384287 0 0) 0x7f38e8000c60 con 0x7f38f81b0470 2021-02-10T11:19:30.218 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:30.209 7f38f67fc700 1 -- 172.21.2.102:0/2547422588 --> v1:172.21.2.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38e4003020 con 0x7f38f81b0470 2021-02-10T11:19:30.218 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:30.209 7f38f7fff700 1 -- 172.21.2.102:0/2547422588 >> v1:172.21.2.113:6789/0 conn(0x7f38f81b0470 legacy=0x7f38f80ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:19:30.218 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:30.209 7f38f7fff700 1 -- 172.21.2.102:0/2547422588 >> v1:172.21.2.102:6789/0 conn(0x7f38f81b0870 legacy=0x7f38f80efe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:19:30.218 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:30.209 7f38f7fff700 1 -- 172.21.2.102:0/2547422588 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38f80d2fb0 con 0x7f38f80f05f0 2021-02-10T11:19:30.218 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:30.209 7f38f7fff700 1 --2- 172.21.2.102:0/2547422588 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f38f80f05f0 0x7f38f80f20e0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7f38f0006e50 tx=0x7f38f0009e70).ready entity=mon.2 client_cookie=ffd2f5e3e6d29874 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:19:30.218 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:30.209 7f38f67fc700 1 -- 172.21.2.102:0/2547422588 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f38f000b510 con 0x7f38f80f05f0 2021-02-10T11:19:30.218 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:30.209 7f38f67fc700 1 -- 172.21.2.102:0/2547422588 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38f0006790 con 0x7f38f80f05f0 2021-02-10T11:19:30.218 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:30.209 7f38fdc3e700 1 -- 172.21.2.102:0/2547422588 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f38f80f05f0 msgr2=0x7f38f80f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:19:30.218 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:30.209 7f38fdc3e700 1 --2- 172.21.2.102:0/2547422588 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f38f80f05f0 0x7f38f80f20e0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7f38f0006e50 tx=0x7f38f0009e70).stop 2021-02-10T11:19:30.218 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:30.210 7f38fdc3e700 1 -- 172.21.2.102:0/2547422588 shutdown_connections 2021-02-10T11:19:30.218 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:30.210 7f38fdc3e700 1 --2- 172.21.2.102:0/2547422588 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f38f80f05f0 0x7f38f80f20e0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:30.219 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:30.210 7f38fdc3e700 1 -- 172.21.2.102:0/2547422588 >> 172.21.2.102:0/2547422588 conn(0x7f38f80ce3e0 msgr2=0x7f38f80e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:19:30.219 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:30.210 7f38fdc3e700 1 -- 172.21.2.102:0/2547422588 shutdown_connections 2021-02-10T11:19:30.219 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:30.210 7f38fdc3e700 1 -- 172.21.2.102:0/2547422588 wait complete. 2021-02-10T11:19:30.221 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:30.225 7f38fdc3e700 1 Processor -- start 2021-02-10T11:19:30.221 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:30.225 7f38fdc3e700 1 -- start start 2021-02-10T11:19:30.221 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:30.225 7f38fdc3e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f38f80a6c90 0x7f38f81924c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:30.222 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:30.225 7f38fdc3e700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f38f8192ae0 0x7f38f8192ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:30.222 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:30.225 7f38fdc3e700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f38f80919e0 0x7f38f8091de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:30.222 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:30.225 7f38fdc3e700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38f819d2e0 con 0x7f38f80a6c90 2021-02-10T11:19:30.222 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:30.225 7f38fdc3e700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38f80914d0 con 0x7f38f80919e0 2021-02-10T11:19:30.222 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:30.225 7f38fdc3e700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38f80a6820 con 0x7f38f8192ae0 2021-02-10T11:19:30.223 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:30.225 7f38f77fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f38f80a6c90 0x7f38f81924c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:19:30.223 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:30.225 7f38f77fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f38f80a6c90 0x7f38f81924c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:53538/0 (socket says 172.21.2.102:53538) 2021-02-10T11:19:30.223 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:30.225 7f38f77fe700 1 -- 172.21.2.102:0/1753618325 learned_addr learned my addr 172.21.2.102:0/1753618325 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:19:30.223 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:30.226 7f38f6ffd700 1 --2- 172.21.2.102:0/1753618325 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f38f8192ae0 0x7f38f8192ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:19:30.223 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:30.226 7f38f77fe700 1 -- 172.21.2.102:0/1753618325 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f38f8192ae0 msgr2=0x7f38f8192ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:19:30.223 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:30.226 7f38f77fe700 1 --2- 172.21.2.102:0/1753618325 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f38f8192ae0 0x7f38f8192ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:30.224 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:30.226 7f38f77fe700 1 -- 172.21.2.102:0/1753618325 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f38f80919e0 msgr2=0x7f38f8091de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:19:30.224 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:30.226 7f38f7fff700 1 --2- 172.21.2.102:0/1753618325 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f38f80919e0 0x7f38f8091de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:19:30.224 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:30.226 7f38f77fe700 1 --2- 172.21.2.102:0/1753618325 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f38f80919e0 0x7f38f8091de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:30.224 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:30.226 7f38f77fe700 1 -- 172.21.2.102:0/1753618325 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38f8092400 con 0x7f38f80a6c90 2021-02-10T11:19:30.224 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:30.226 7f38f7fff700 1 --2- 172.21.2.102:0/1753618325 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f38f80919e0 0x7f38f8091de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:19:30.224 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:30.226 7f38f77fe700 1 --2- 172.21.2.102:0/1753618325 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f38f80a6c90 0x7f38f81924c0 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 rx=0x7f38e800b580 tx=0x7f38e800b920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:19:30.224 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:30.226 7f38effff700 1 -- 172.21.2.102:0/1753618325 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f38e800bd10 con 0x7f38f80a6c90 2021-02-10T11:19:30.224 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:30.226 7f38effff700 1 -- 172.21.2.102:0/1753618325 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38e8000e20 con 0x7f38f80a6c90 2021-02-10T11:19:30.225 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:30.226 7f38effff700 1 -- 172.21.2.102:0/1753618325 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f38e80054f0 con 0x7f38f80a6c90 2021-02-10T11:19:30.225 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:30.226 7f38fdc3e700 1 -- 172.21.2.102:0/1753618325 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38f8092400 con 0x7f38f80a6c90 2021-02-10T11:19:30.225 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:30.226 7f38fdc3e700 1 -- 172.21.2.102:0/1753618325 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38f8092400 con 0x7f38f80a6c90 2021-02-10T11:19:30.225 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:30.227 7f38effff700 1 -- 172.21.2.102:0/1753618325 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f38e8009fe0 con 0x7f38f80a6c90 2021-02-10T11:19:30.225 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:30.227 7f38effff700 1 --2- 172.21.2.102:0/1753618325 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f38dc0186c0 0x7f38dc01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:30.225 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:30.227 7f38effff700 1 -- 172.21.2.102:0/1753618325 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f38e80327a0 con 0x7f38f80a6c90 2021-02-10T11:19:30.225 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:30.227 7f38f6ffd700 1 --2- 172.21.2.102:0/1753618325 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f38dc0186c0 0x7f38dc01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:19:30.225 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:30.227 7f38f6ffd700 1 --2- 172.21.2.102:0/1753618325 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f38dc0186c0 0x7f38dc01abc0 crc :-1 s=READY pgs=372 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:19:30.250 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:30.257 7f38fdc3e700 1 -- 172.21.2.102:0/1753618325 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38f80e52e0 con 0x7f38f80a6c90 2021-02-10T11:19:30.252 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:30.260 7f38effff700 1 -- 172.21.2.102:0/1753618325 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f38e801f4e0 con 0x7f38f80a6c90 2021-02-10T11:19:30.383 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:30.391 7f38fdc3e700 1 -- 172.21.2.102:0/1753618325 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344457", "target": ["mgr", ""]}) v1 -- 0x7f38f80e52e0 con 0x7f38dc0186c0 2021-02-10T11:19:30.387 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:30.394 7f38effff700 1 -- 172.21.2.102:0/1753618325 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f38e0008880 con 0x7f38dc0186c0 2021-02-10T11:19:30.391 INFO:teuthology.orchestra.run.gibba002.stdout:/volumes/_nogroup/subvolume_0000000000344457/92900233-3ca0-4be1-9612-c794fb829a4e 2021-02-10T11:19:30.392 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:30.400 7f38fdc3e700 1 -- 172.21.2.102:0/1753618325 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f38dc0186c0 msgr2=0x7f38dc01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:19:30.392 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:30.400 7f38fdc3e700 1 --2- 172.21.2.102:0/1753618325 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f38dc0186c0 0x7f38dc01abc0 crc :-1 s=READY pgs=372 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:30.392 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:30.400 7f38fdc3e700 1 -- 172.21.2.102:0/1753618325 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f38f80a6c90 msgr2=0x7f38f81924c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:19:30.392 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:30.400 7f38fdc3e700 1 --2- 172.21.2.102:0/1753618325 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f38f80a6c90 0x7f38f81924c0 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 rx=0x7f38e800b580 tx=0x7f38e800b920).stop 2021-02-10T11:19:30.393 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:30.400 7f38fdc3e700 1 -- 172.21.2.102:0/1753618325 shutdown_connections 2021-02-10T11:19:30.393 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:30.400 7f38fdc3e700 1 --2- 172.21.2.102:0/1753618325 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f38dc0186c0 0x7f38dc01abc0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:30.393 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:30.400 7f38fdc3e700 1 --2- 172.21.2.102:0/1753618325 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f38f80919e0 0x7f38f8091de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:30.393 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:30.400 7f38fdc3e700 1 --2- 172.21.2.102:0/1753618325 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f38f8192ae0 0x7f38f8192ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:30.393 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:30.400 7f38fdc3e700 1 --2- 172.21.2.102:0/1753618325 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f38f80a6c90 0x7f38f81924c0 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:30.393 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:30.400 7f38fdc3e700 1 -- 172.21.2.102:0/1753618325 >> 172.21.2.102:0/1753618325 conn(0x7f38f80ac550 msgr2=0x7f38f80be1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:19:30.393 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:30.400 7f38fdc3e700 1 -- 172.21.2.102:0/1753618325 shutdown_connections 2021-02-10T11:19:30.393 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:30.400 7f38fdc3e700 1 -- 172.21.2.102:0/1753618325 wait complete. 2021-02-10T11:19:30.400 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000344457 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000344457/92900233-3ca0-4be1-9612-c794fb829a4e 2021-02-10T11:19:30.401 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:19:30.401 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:19:30.401 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:19:30.739 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:19:30.739 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:19:30.740 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344457/92900233-3ca0-4be1-9612-c794fb829a4e/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:19:31.121 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:19:31.121 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:19:31.121 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0228261 s, 45.9 MB/s 2021-02-10T11:19:31.122 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:19:31.122 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:19:31.122 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:19:31.458 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:19:31.459 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:19:31.459 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344457/92900233-3ca0-4be1-9612-c794fb829a4e/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:19:31.836 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:19:31.836 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:19:31.836 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0243153 s, 43.1 MB/s 2021-02-10T11:19:31.837 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:19:31.837 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:19:31.837 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:19:32.175 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:19:32.176 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:19:32.176 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344457/92900233-3ca0-4be1-9612-c794fb829a4e/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:19:32.552 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:19:32.552 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:19:32.553 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0233593 s, 44.9 MB/s 2021-02-10T11:19:32.553 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:19:32.554 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:19:32.554 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:19:32.890 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:19:32.890 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:19:32.891 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344457/92900233-3ca0-4be1-9612-c794fb829a4e/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:19:33.268 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:19:33.269 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:19:33.269 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0246403 s, 42.6 MB/s 2021-02-10T11:19:33.270 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:19:33.270 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:19:33.270 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:19:33.605 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:19:33.605 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:19:33.606 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344457/92900233-3ca0-4be1-9612-c794fb829a4e/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:19:33.980 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:19:33.980 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:19:33.980 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0249208 s, 42.1 MB/s 2021-02-10T11:19:33.981 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:19:33.981 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:19:33.981 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:19:34.317 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:19:34.318 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:19:34.318 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344457/92900233-3ca0-4be1-9612-c794fb829a4e/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:19:34.696 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:19:37.185 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:19:37.185 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0245347 s, 42.7 MB/s 2021-02-10T11:19:37.187 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:19:37.187 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:19:37.187 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:19:37.523 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:19:37.523 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:19:37.524 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344457/92900233-3ca0-4be1-9612-c794fb829a4e/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:19:37.900 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:19:37.900 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:19:37.900 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0233845 s, 44.8 MB/s 2021-02-10T11:19:37.901 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:19:37.901 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:19:37.901 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:19:38.237 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:19:38.237 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:19:38.238 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344457/92900233-3ca0-4be1-9612-c794fb829a4e/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:19:38.611 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:19:38.611 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:19:38.611 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0241145 s, 43.5 MB/s 2021-02-10T11:19:38.612 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:19:38.612 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:19:38.612 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:19:38.949 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:19:38.950 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:19:38.950 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344457/92900233-3ca0-4be1-9612-c794fb829a4e/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:19:39.328 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:19:39.328 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:19:39.328 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0244829 s, 42.8 MB/s 2021-02-10T11:19:39.329 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:19:39.329 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:19:39.330 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:19:39.665 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:19:39.666 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:19:39.666 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344457/92900233-3ca0-4be1-9612-c794fb829a4e/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:19:40.044 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:19:40.044 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:19:40.044 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0246748 s, 42.5 MB/s 2021-02-10T11:19:40.045 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000344458 2021-02-10T11:19:40.480 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:40.470 7f7f3d35f700 1 Processor -- start 2021-02-10T11:19:40.480 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:40.470 7f7f3d35f700 1 -- start start 2021-02-10T11:19:40.480 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:40.471 7f7f3d35f700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f7f380f05f0 0x7f7f380f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:40.480 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:40.471 7f7f3d35f700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f381b0c70 con 0x7f7f381b0470 2021-02-10T11:19:40.481 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:40.471 7f7f3d35f700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f380f0b50 con 0x7f7f381b0870 2021-02-10T11:19:40.481 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:40.471 7f7f3d35f700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f380e5ea0 con 0x7f7f380f05f0 2021-02-10T11:19:40.481 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:40.471 7f7f3707d700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f7f381b0470 0x7f7f380ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:45768/0 (socket says 172.21.2.102:45768) 2021-02-10T11:19:40.481 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:40.471 7f7f3707d700 1 -- 172.21.2.102:0/1814265069 learned_addr learned my addr 172.21.2.102:0/1814265069 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:19:40.481 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:40.471 7f7f3607b700 1 -- 172.21.2.102:0/1814265069 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3569671294 0 0) 0x7f7f28001180 con 0x7f7f381b0470 2021-02-10T11:19:40.481 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:40.471 7f7f3787e700 1 --2- 172.21.2.102:0/1814265069 >> v2:172.21.2.113:3300/0 conn(0x7f7f380f05f0 0x7f7f380f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:19:40.481 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:40.471 7f7f3607b700 1 -- 172.21.2.102:0/1814265069 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f20002fe0 con 0x7f7f381b0470 2021-02-10T11:19:40.481 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:40.471 7f7f3607b700 1 -- 172.21.2.102:0/1814265069 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2626107913 0 0) 0x7f7f28001180 con 0x7f7f381b0470 2021-02-10T11:19:40.481 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:40.471 7f7f3607b700 1 -- 172.21.2.102:0/1814265069 >> v2:172.21.2.113:3300/0 conn(0x7f7f380f05f0 msgr2=0x7f7f380f20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:19:40.481 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:40.471 7f7f3607b700 1 --2- 172.21.2.102:0/1814265069 >> v2:172.21.2.113:3300/0 conn(0x7f7f380f05f0 0x7f7f380f20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:40.481 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:40.471 7f7f3607b700 1 -- 172.21.2.102:0/1814265069 >> v1:172.21.2.113:6789/0 conn(0x7f7f381b0870 legacy=0x7f7f380efe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:19:40.481 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:40.471 7f7f3607b700 1 -- 172.21.2.102:0/1814265069 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f380e52e0 con 0x7f7f381b0470 2021-02-10T11:19:40.482 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:40.471 7f7f3607b700 1 -- 172.21.2.102:0/1814265069 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7f7f28001680 con 0x7f7f381b0470 2021-02-10T11:19:40.482 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:40.472 7f7f3607b700 1 -- 172.21.2.102:0/1814265069 >> v1:172.21.2.102:6789/0 conn(0x7f7f381b0470 legacy=0x7f7f380ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:19:40.482 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:40.472 7f7f3607b700 1 --2- 172.21.2.102:0/1814265069 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7f20003a60 0x7f7f20005ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:40.482 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:40.472 7f7f3607b700 1 --2- 172.21.2.102:0/1814265069 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7f20006790 0x7f7f20008c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:40.482 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:40.472 7f7f3607b700 1 --2- 172.21.2.102:0/1814265069 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7f20009590 0x7f7f2000bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:40.482 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:40.472 7f7f3607b700 1 -- 172.21.2.102:0/1814265069 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f2000c300 con 0x7f7f20009590 2021-02-10T11:19:40.482 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:40.472 7f7f3607b700 1 -- 172.21.2.102:0/1814265069 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f2000c520 con 0x7f7f20006790 2021-02-10T11:19:40.482 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:40.472 7f7f3607b700 1 -- 172.21.2.102:0/1814265069 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f20050750 con 0x7f7f20003a60 2021-02-10T11:19:40.482 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:40.472 7f7f3787e700 1 --2- 172.21.2.102:0/1814265069 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7f20009590 0x7f7f2000bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:19:40.482 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:40.472 7f7f3687c700 1 --2- 172.21.2.102:0/1814265069 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7f20006790 0x7f7f20008c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:19:40.482 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:40.472 7f7f3707d700 1 --2- 172.21.2.102:0/1814265069 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7f20003a60 0x7f7f20005ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:19:40.482 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:40.472 7f7f3787e700 1 -- 172.21.2.102:0/1814265069 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7f20003a60 msgr2=0x7f7f20005ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:19:40.482 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:40.472 7f7f3787e700 1 --2- 172.21.2.102:0/1814265069 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7f20003a60 0x7f7f20005ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:40.483 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:40.472 7f7f3787e700 1 -- 172.21.2.102:0/1814265069 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7f20006790 msgr2=0x7f7f20008c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:19:40.483 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:40.472 7f7f3787e700 1 --2- 172.21.2.102:0/1814265069 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7f20006790 0x7f7f20008c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:40.483 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:40.472 7f7f3787e700 1 -- 172.21.2.102:0/1814265069 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f200509b0 con 0x7f7f20009590 2021-02-10T11:19:40.483 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:40.472 7f7f3787e700 1 -- 172.21.2.102:0/1814265069 reap_dead start 2021-02-10T11:19:40.483 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:40.472 7f7f3787e700 1 --2- 172.21.2.102:0/1814265069 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7f20009590 0x7f7f2000bab0 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 rx=0x7f7f30006a40 tx=0x7f7f3000c0c0).ready entity=mon.0 client_cookie=6bdbc44c22fb58d9 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:19:40.483 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:40.472 7f7f3607b700 1 -- 172.21.2.102:0/1814265069 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7f30006ab0 con 0x7f7f20009590 2021-02-10T11:19:40.483 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:40.472 7f7f3607b700 1 -- 172.21.2.102:0/1814265069 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f30007070 con 0x7f7f20009590 2021-02-10T11:19:40.483 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:40.472 7f7f3607b700 1 -- 172.21.2.102:0/1814265069 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7f300074c0 con 0x7f7f20009590 2021-02-10T11:19:40.483 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:40.472 7f7f3d35f700 1 -- 172.21.2.102:0/1814265069 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7f20009590 msgr2=0x7f7f2000bab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:19:40.483 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:40.472 7f7f3d35f700 1 --2- 172.21.2.102:0/1814265069 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7f20009590 0x7f7f2000bab0 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 rx=0x7f7f30006a40 tx=0x7f7f3000c0c0).stop 2021-02-10T11:19:40.483 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:40.473 7f7f3d35f700 1 -- 172.21.2.102:0/1814265069 shutdown_connections 2021-02-10T11:19:40.483 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:40.473 7f7f3d35f700 1 --2- 172.21.2.102:0/1814265069 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7f20009590 0x7f7f2000bab0 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:40.484 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:40.473 7f7f3d35f700 1 -- 172.21.2.102:0/1814265069 >> 172.21.2.102:0/1814265069 conn(0x7f7f380ce3e0 msgr2=0x7f7f380e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:19:40.484 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:40.473 7f7f3d35f700 1 -- 172.21.2.102:0/1814265069 shutdown_connections 2021-02-10T11:19:40.484 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:40.473 7f7f3d35f700 1 -- 172.21.2.102:0/1814265069 wait complete. 2021-02-10T11:19:40.484 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:40.488 7f7f3d35f700 1 Processor -- start 2021-02-10T11:19:40.485 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:40.488 7f7f3d35f700 1 -- start start 2021-02-10T11:19:40.485 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:40.488 7f7f3d35f700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7f3809e560 0x7f7f380a0130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:40.485 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:40.488 7f7f3d35f700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7f3809e960 0x7f7f380a0750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:40.485 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:40.488 7f7f3d35f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7f380a0d70 0x7f7f380a1af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:40.485 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:40.488 7f7f3d35f700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f380c38f0 con 0x7f7f380a0d70 2021-02-10T11:19:40.485 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:40.488 7f7f3d35f700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f380a1170 con 0x7f7f3809e960 2021-02-10T11:19:40.486 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:40.488 7f7f3d35f700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f380a1390 con 0x7f7f3809e560 2021-02-10T11:19:40.486 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:40.488 7f7f3787e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7f380a0d70 0x7f7f380a1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:19:40.486 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:40.488 7f7f3787e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7f380a0d70 0x7f7f380a1af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:53558/0 (socket says 172.21.2.102:53558) 2021-02-10T11:19:40.486 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:40.488 7f7f3787e700 1 -- 172.21.2.102:0/1449249434 learned_addr learned my addr 172.21.2.102:0/1449249434 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:19:40.486 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:40.489 7f7f3707d700 1 --2- 172.21.2.102:0/1449249434 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7f3809e560 0x7f7f380a0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:19:40.486 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:40.489 7f7f3687c700 1 --2- 172.21.2.102:0/1449249434 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7f3809e960 0x7f7f380a0750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:19:40.486 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:40.489 7f7f3787e700 1 -- 172.21.2.102:0/1449249434 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7f3809e560 msgr2=0x7f7f380a0130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:19:40.486 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:40.489 7f7f3787e700 1 --2- 172.21.2.102:0/1449249434 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7f3809e560 0x7f7f380a0130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:40.486 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:40.489 7f7f3787e700 1 -- 172.21.2.102:0/1449249434 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7f3809e960 msgr2=0x7f7f380a0750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:19:40.486 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:40.489 7f7f3787e700 1 --2- 172.21.2.102:0/1449249434 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7f3809e960 0x7f7f380a0750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:40.486 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:40.489 7f7f3787e700 1 -- 172.21.2.102:0/1449249434 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f380a2110 con 0x7f7f380a0d70 2021-02-10T11:19:40.486 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:40.489 7f7f3707d700 1 --2- 172.21.2.102:0/1449249434 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7f3809e560 0x7f7f380a0130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:19:40.486 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:40.489 7f7f3787e700 1 --2- 172.21.2.102:0/1449249434 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7f380a0d70 0x7f7f380a1af0 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 rx=0x7f7f300096f0 tx=0x7f7f3000bd20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:19:40.487 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:40.489 7f7f1ffff700 1 -- 172.21.2.102:0/1449249434 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7f30006d40 con 0x7f7f380a0d70 2021-02-10T11:19:40.487 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:40.489 7f7f1ffff700 1 -- 172.21.2.102:0/1449249434 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f3000be60 con 0x7f7f380a0d70 2021-02-10T11:19:40.487 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:40.489 7f7f1ffff700 1 -- 172.21.2.102:0/1449249434 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7f3000c560 con 0x7f7f380a0d70 2021-02-10T11:19:40.487 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:40.489 7f7f3d35f700 1 -- 172.21.2.102:0/1449249434 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f380a2110 con 0x7f7f380a0d70 2021-02-10T11:19:40.487 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:40.489 7f7f3d35f700 1 -- 172.21.2.102:0/1449249434 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f380a2110 con 0x7f7f380a0d70 2021-02-10T11:19:40.487 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:40.490 7f7f1ffff700 1 -- 172.21.2.102:0/1449249434 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f7f300056b0 con 0x7f7f380a0d70 2021-02-10T11:19:40.487 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:40.490 7f7f1ffff700 1 --2- 172.21.2.102:0/1449249434 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7f180186c0 0x7f7f1801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:40.487 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:40.490 7f7f1ffff700 1 -- 172.21.2.102:0/1449249434 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f7f300328e0 con 0x7f7f380a0d70 2021-02-10T11:19:40.487 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:40.490 7f7f3707d700 1 --2- 172.21.2.102:0/1449249434 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7f180186c0 0x7f7f1801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:19:40.487 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:40.490 7f7f3707d700 1 --2- 172.21.2.102:0/1449249434 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7f180186c0 0x7f7f1801abc0 crc :-1 s=READY pgs=373 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:19:40.513 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:40.521 7f7f3d35f700 1 -- 172.21.2.102:0/1449249434 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f38105b30 con 0x7f7f380a0d70 2021-02-10T11:19:40.516 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:40.523 7f7f1ffff700 1 -- 172.21.2.102:0/1449249434 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f7f3002d450 con 0x7f7f380a0d70 2021-02-10T11:19:40.642 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:40.649 7f7f3d35f700 1 -- 172.21.2.102:0/1449249434 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344458", "target": ["mgr", ""]}) v1 -- 0x7f7f380f0b50 con 0x7f7f180186c0 2021-02-10T11:19:40.670 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:40.678 7f7f1ffff700 1 -- 172.21.2.102:0/1449249434 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f7f28007660 con 0x7f7f180186c0 2021-02-10T11:19:40.675 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:40.682 7f7f3d35f700 1 -- 172.21.2.102:0/1449249434 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7f180186c0 msgr2=0x7f7f1801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:19:40.675 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:40.682 7f7f3d35f700 1 --2- 172.21.2.102:0/1449249434 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7f180186c0 0x7f7f1801abc0 crc :-1 s=READY pgs=373 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:40.675 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:40.682 7f7f3d35f700 1 -- 172.21.2.102:0/1449249434 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7f380a0d70 msgr2=0x7f7f380a1af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:19:40.676 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:40.682 7f7f3d35f700 1 --2- 172.21.2.102:0/1449249434 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7f380a0d70 0x7f7f380a1af0 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 rx=0x7f7f300096f0 tx=0x7f7f3000bd20).stop 2021-02-10T11:19:40.676 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:40.682 7f7f3d35f700 1 -- 172.21.2.102:0/1449249434 shutdown_connections 2021-02-10T11:19:40.676 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:40.682 7f7f3d35f700 1 --2- 172.21.2.102:0/1449249434 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7f380a0d70 0x7f7f380a1af0 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:40.676 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:40.682 7f7f3d35f700 1 --2- 172.21.2.102:0/1449249434 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7f180186c0 0x7f7f1801abc0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:40.676 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:40.682 7f7f3d35f700 1 --2- 172.21.2.102:0/1449249434 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7f3809e960 0x7f7f380a0750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:40.676 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:40.682 7f7f3d35f700 1 --2- 172.21.2.102:0/1449249434 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7f3809e560 0x7f7f380a0130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:40.676 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:40.682 7f7f3d35f700 1 -- 172.21.2.102:0/1449249434 >> 172.21.2.102:0/1449249434 conn(0x7f7f3809fc10 msgr2=0x7f7f38094ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:19:40.676 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:40.683 7f7f3d35f700 1 -- 172.21.2.102:0/1449249434 shutdown_connections 2021-02-10T11:19:40.676 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:40.683 7f7f3d35f700 1 -- 172.21.2.102:0/1449249434 wait complete. 2021-02-10T11:19:40.683 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000344458 2021-02-10T11:19:41.116 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:41.107 7f9de6705700 1 Processor -- start 2021-02-10T11:19:41.116 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:41.107 7f9de6705700 1 -- start start 2021-02-10T11:19:41.116 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:41.107 7f9de6705700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f9de01b0470 0x7f9de01b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:41.116 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:41.107 7f9de6705700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9de00f20e0 0x7f9de00f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:41.116 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:41.107 7f9de6705700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9de00f0b50 con 0x7f9de00ef610 2021-02-10T11:19:41.116 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:41.107 7f9de6705700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9de00f0820 con 0x7f9de01b0470 2021-02-10T11:19:41.116 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:41.107 7f9de6705700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9de00f24e0 con 0x7f9de00f20e0 2021-02-10T11:19:41.116 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:41.107 7f9ddffff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f9de01b0470 0x7f9de01b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:19:41.117 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:41.107 7f9ddffff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f9de01b0470 0x7f9de01b0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:53564/0 (socket says 172.21.2.102:53564) 2021-02-10T11:19:41.117 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:41.107 7f9ddffff700 1 -- 172.21.2.102:0/1643072226 learned_addr learned my addr 172.21.2.102:0/1643072226 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:19:41.117 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:41.107 7f9de4d22700 1 --2- 172.21.2.102:0/1643072226 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9de00f20e0 0x7f9de00f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:19:41.117 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:41.108 7f9ddffff700 1 -- 172.21.2.102:0/1643072226 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9de00f20e0 msgr2=0x7f9de00f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:19:41.117 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:41.108 7f9ddffff700 1 --2- 172.21.2.102:0/1643072226 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9de00f20e0 0x7f9de00f0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:41.117 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:41.108 7f9ddffff700 1 -- 172.21.2.102:0/1643072226 >> v1:172.21.2.113:6789/0 conn(0x7f9de00ef610 legacy=0x7f9de00efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:19:41.117 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:41.108 7f9ddffff700 1 -- 172.21.2.102:0/1643072226 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9de00e52e0 con 0x7f9de01b0470 2021-02-10T11:19:41.117 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:41.108 7f9ddffff700 1 --2- 172.21.2.102:0/1643072226 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9de01b0470 0x7f9de01b0870 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 rx=0x7f9dd0006700 tx=0x7f9dd0009b80).ready entity=mon.0 client_cookie=8cb4f9d9c82ce13e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:19:41.117 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:41.108 7f9ddeffd700 1 -- 172.21.2.102:0/1643072226 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9dd000c0c0 con 0x7f9de01b0470 2021-02-10T11:19:41.117 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:41.108 7f9ddeffd700 1 -- 172.21.2.102:0/1643072226 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9dd000c340 con 0x7f9de01b0470 2021-02-10T11:19:41.117 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:41.108 7f9ddeffd700 1 -- 172.21.2.102:0/1643072226 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9dd000c860 con 0x7f9de01b0470 2021-02-10T11:19:41.117 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:41.108 7f9de6705700 1 -- 172.21.2.102:0/1643072226 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9de01b0470 msgr2=0x7f9de01b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:19:41.117 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:41.108 7f9de6705700 1 --2- 172.21.2.102:0/1643072226 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9de01b0470 0x7f9de01b0870 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 rx=0x7f9dd0006700 tx=0x7f9dd0009b80).stop 2021-02-10T11:19:41.118 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:41.109 7f9de6705700 1 -- 172.21.2.102:0/1643072226 shutdown_connections 2021-02-10T11:19:41.118 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:41.109 7f9de6705700 1 --2- 172.21.2.102:0/1643072226 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9de00f20e0 0x7f9de00f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:41.118 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:41.109 7f9de6705700 1 --2- 172.21.2.102:0/1643072226 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9de01b0470 0x7f9de01b0870 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:41.118 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:41.109 7f9de6705700 1 -- 172.21.2.102:0/1643072226 >> 172.21.2.102:0/1643072226 conn(0x7f9de00ce3e0 msgr2=0x7f9de00e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:19:41.118 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:41.109 7f9de6705700 1 -- 172.21.2.102:0/1643072226 shutdown_connections 2021-02-10T11:19:41.118 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:41.109 7f9de6705700 1 -- 172.21.2.102:0/1643072226 wait complete. 2021-02-10T11:19:41.119 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:41.123 7f9de6705700 1 Processor -- start 2021-02-10T11:19:41.119 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:41.124 7f9de6705700 1 -- start start 2021-02-10T11:19:41.119 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:41.124 7f9de6705700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9de009f140 0x7f9de00c1940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:41.119 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:41.124 7f9de6705700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9de00c1f60 0x7f9de009e300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:41.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:41.124 7f9de6705700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f9de009e920 0x7f9de00c2960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:41.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:41.124 7f9de6705700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9de00c23e0 con 0x7f9de00c1f60 2021-02-10T11:19:41.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:41.124 7f9de6705700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9de009ed20 con 0x7f9de009e920 2021-02-10T11:19:41.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:41.124 7f9de6705700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9de0105d10 con 0x7f9de009f140 2021-02-10T11:19:41.121 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:41.124 7f9ddf7fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9de00c1f60 0x7f9de009e300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:19:41.121 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:41.124 7f9ddf7fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9de00c1f60 0x7f9de009e300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:53572/0 (socket says 172.21.2.102:53572) 2021-02-10T11:19:41.121 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:41.124 7f9ddf7fe700 1 -- 172.21.2.102:0/2968596824 learned_addr learned my addr 172.21.2.102:0/2968596824 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:19:41.121 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:41.124 7f9de4d22700 1 --2- 172.21.2.102:0/2968596824 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f9de009e920 0x7f9de00c2960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:19:41.121 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:41.124 7f9ddffff700 1 --2- 172.21.2.102:0/2968596824 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9de009f140 0x7f9de00c1940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:19:41.121 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:41.125 7f9ddf7fe700 1 -- 172.21.2.102:0/2968596824 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9de009f140 msgr2=0x7f9de00c1940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:19:41.121 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:41.125 7f9ddf7fe700 1 --2- 172.21.2.102:0/2968596824 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9de009f140 0x7f9de00c1940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:41.122 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:41.125 7f9ddf7fe700 1 -- 172.21.2.102:0/2968596824 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f9de009e920 msgr2=0x7f9de00c2960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:19:41.122 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:41.125 7f9ddf7fe700 1 --2- 172.21.2.102:0/2968596824 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f9de009e920 0x7f9de00c2960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:41.122 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:41.125 7f9ddf7fe700 1 -- 172.21.2.102:0/2968596824 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9de01a4910 con 0x7f9de00c1f60 2021-02-10T11:19:41.122 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:41.125 7f9ddf7fe700 1 --2- 172.21.2.102:0/2968596824 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9de00c1f60 0x7f9de009e300 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 rx=0x7f9dd4006ed0 tx=0x7f9dd400b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:19:41.122 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:41.125 7f9ddcff9700 1 -- 172.21.2.102:0/2968596824 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9dd4006240 con 0x7f9de00c1f60 2021-02-10T11:19:41.122 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:41.125 7f9ddcff9700 1 -- 172.21.2.102:0/2968596824 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9dd40054b0 con 0x7f9de00c1f60 2021-02-10T11:19:41.122 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:41.125 7f9ddcff9700 1 -- 172.21.2.102:0/2968596824 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9dd40059d0 con 0x7f9de00c1f60 2021-02-10T11:19:41.122 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:41.125 7f9de6705700 1 -- 172.21.2.102:0/2968596824 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9de01a4910 con 0x7f9de00c1f60 2021-02-10T11:19:41.122 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:41.125 7f9de6705700 1 -- 172.21.2.102:0/2968596824 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9de00c3270 con 0x7f9de00c1f60 2021-02-10T11:19:41.122 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:41.126 7f9ddcff9700 1 -- 172.21.2.102:0/2968596824 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f9dd4006030 con 0x7f9de00c1f60 2021-02-10T11:19:41.122 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:41.126 7f9ddcff9700 1 --2- 172.21.2.102:0/2968596824 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f9dc00186c0 0x7f9dc001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:41.122 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:41.126 7f9ddcff9700 1 -- 172.21.2.102:0/2968596824 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f9dd4032650 con 0x7f9de00c1f60 2021-02-10T11:19:41.123 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:41.126 7f9ddffff700 1 --2- 172.21.2.102:0/2968596824 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f9dc00186c0 0x7f9dc001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:19:41.123 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:41.126 7f9ddffff700 1 --2- 172.21.2.102:0/2968596824 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f9dc00186c0 0x7f9dc001abc0 crc :-1 s=READY pgs=374 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:19:41.150 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:41.157 7f9de6705700 1 -- 172.21.2.102:0/2968596824 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9de00f0b50 con 0x7f9de00c1f60 2021-02-10T11:19:41.152 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:41.160 7f9ddcff9700 1 -- 172.21.2.102:0/2968596824 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f9dd4000de0 con 0x7f9de00c1f60 2021-02-10T11:19:41.279 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:41.286 7f9de6705700 1 -- 172.21.2.102:0/2968596824 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344458", "target": ["mgr", ""]}) v1 -- 0x7f9de00c3270 con 0x7f9dc00186c0 2021-02-10T11:19:41.282 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:41.290 7f9ddcff9700 1 -- 172.21.2.102:0/2968596824 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f9dd0006430 con 0x7f9dc00186c0 2021-02-10T11:19:41.286 INFO:teuthology.orchestra.run.gibba002.stdout:/volumes/_nogroup/subvolume_0000000000344458/6b22ff27-60bf-4dae-a1d4-656857896d2b 2021-02-10T11:19:41.287 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:41.295 7f9de6705700 1 -- 172.21.2.102:0/2968596824 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f9dc00186c0 msgr2=0x7f9dc001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:19:41.287 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:41.295 7f9de6705700 1 --2- 172.21.2.102:0/2968596824 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f9dc00186c0 0x7f9dc001abc0 crc :-1 s=READY pgs=374 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:41.287 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:41.295 7f9de6705700 1 -- 172.21.2.102:0/2968596824 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9de00c1f60 msgr2=0x7f9de009e300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:19:41.287 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:41.295 7f9de6705700 1 --2- 172.21.2.102:0/2968596824 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9de00c1f60 0x7f9de009e300 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 rx=0x7f9dd4006ed0 tx=0x7f9dd400b960).stop 2021-02-10T11:19:41.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:41.295 7f9de6705700 1 -- 172.21.2.102:0/2968596824 shutdown_connections 2021-02-10T11:19:41.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:41.295 7f9de6705700 1 --2- 172.21.2.102:0/2968596824 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f9dc00186c0 0x7f9dc001abc0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:41.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:41.295 7f9de6705700 1 --2- 172.21.2.102:0/2968596824 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f9de009e920 0x7f9de00c2960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:41.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:41.295 7f9de6705700 1 --2- 172.21.2.102:0/2968596824 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9de00c1f60 0x7f9de009e300 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:41.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:41.295 7f9de6705700 1 --2- 172.21.2.102:0/2968596824 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9de009f140 0x7f9de00c1940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:41.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:41.295 7f9de6705700 1 -- 172.21.2.102:0/2968596824 >> 172.21.2.102:0/2968596824 conn(0x7f9de009fc10 msgr2=0x7f9de00b3250 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:19:41.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:41.295 7f9de6705700 1 -- 172.21.2.102:0/2968596824 shutdown_connections 2021-02-10T11:19:41.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:41.295 7f9de6705700 1 -- 172.21.2.102:0/2968596824 wait complete. 2021-02-10T11:19:41.296 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000344458 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000344458/6b22ff27-60bf-4dae-a1d4-656857896d2b 2021-02-10T11:19:41.296 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:19:41.296 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:19:41.296 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:19:41.633 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:19:41.634 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:19:41.634 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344458/6b22ff27-60bf-4dae-a1d4-656857896d2b/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:19:42.016 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:19:42.017 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:19:42.017 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0243196 s, 43.1 MB/s 2021-02-10T11:19:42.018 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:19:42.018 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:19:42.019 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:19:42.354 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:19:42.355 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:19:42.355 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344458/6b22ff27-60bf-4dae-a1d4-656857896d2b/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:19:42.730 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:19:42.730 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:19:42.730 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.022962 s, 45.7 MB/s 2021-02-10T11:19:42.731 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:19:42.731 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:19:42.732 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:19:43.069 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:19:43.069 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:19:43.069 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344458/6b22ff27-60bf-4dae-a1d4-656857896d2b/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:19:43.442 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:19:43.443 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:19:43.443 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0240935 s, 43.5 MB/s 2021-02-10T11:19:43.444 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:19:43.444 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:19:43.444 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:19:43.781 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:19:43.782 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:19:43.782 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344458/6b22ff27-60bf-4dae-a1d4-656857896d2b/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:19:44.159 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:19:44.159 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:19:44.159 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0249153 s, 42.1 MB/s 2021-02-10T11:19:44.160 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:19:44.161 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:19:44.161 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:19:44.496 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:19:44.497 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:19:44.497 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344458/6b22ff27-60bf-4dae-a1d4-656857896d2b/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:19:44.873 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:19:44.873 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:19:44.873 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.023589 s, 44.5 MB/s 2021-02-10T11:19:44.874 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:19:44.874 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:19:44.874 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:19:45.210 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:19:45.210 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:19:45.210 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344458/6b22ff27-60bf-4dae-a1d4-656857896d2b/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:19:45.583 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:19:45.583 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:19:45.583 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0225343 s, 46.5 MB/s 2021-02-10T11:19:45.584 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:19:45.584 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:19:45.584 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:19:45.920 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:19:45.921 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:19:45.921 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344458/6b22ff27-60bf-4dae-a1d4-656857896d2b/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:19:46.296 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:19:46.297 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:19:46.297 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0239826 s, 43.7 MB/s 2021-02-10T11:19:46.298 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:19:46.298 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:19:46.298 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:19:46.634 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:19:46.634 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:19:46.634 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344458/6b22ff27-60bf-4dae-a1d4-656857896d2b/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:19:47.008 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:19:47.009 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:19:47.009 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0230284 s, 45.5 MB/s 2021-02-10T11:19:47.010 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:19:47.010 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:19:47.010 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:19:47.345 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:19:47.346 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:19:47.346 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344458/6b22ff27-60bf-4dae-a1d4-656857896d2b/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:19:47.722 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:19:47.722 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:19:47.722 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0241152 s, 43.5 MB/s 2021-02-10T11:19:47.723 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:19:47.723 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:19:47.723 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:19:48.059 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:19:48.060 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:19:48.060 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344458/6b22ff27-60bf-4dae-a1d4-656857896d2b/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:19:48.432 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:19:48.432 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:19:48.432 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0237458 s, 44.2 MB/s 2021-02-10T11:19:48.433 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000344459 2021-02-10T11:19:48.865 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:48.857 7fe272951700 1 Processor -- start 2021-02-10T11:19:48.865 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:48.857 7fe272951700 1 -- start start 2021-02-10T11:19:48.865 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:48.858 7fe272951700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fe26c1b0870 0x7fe26c0f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:48.865 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:48.858 7fe272951700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe26c1b0c70 con 0x7fe26c0efe00 2021-02-10T11:19:48.865 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:48.858 7fe272951700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe26c0f0b50 con 0x7fe26c1b0470 2021-02-10T11:19:48.865 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:48.858 7fe272951700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe26c0e5ea0 con 0x7fe26c1b0870 2021-02-10T11:19:48.866 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:48.858 7fe26b7fe700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fe26c1b0870 0x7fe26c0f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:19:48.866 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:48.858 7fe270f6e700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fe26c0efe00 0x7fe26c0f0200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:45806/0 (socket says 172.21.2.102:45806) 2021-02-10T11:19:48.866 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:48.858 7fe270f6e700 1 -- 172.21.2.102:0/3840384000 learned_addr learned my addr 172.21.2.102:0/3840384000 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:19:48.866 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:48.858 7fe26b7fe700 1 --2- 172.21.2.102:0/3840384000 >> v2:172.21.2.102:3300/0 conn(0x7fe26c1b0870 0x7fe26c0f20e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:53580/0 (socket says 172.21.2.102:53580) 2021-02-10T11:19:48.866 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:48.858 7fe26b7fe700 1 -- 172.21.2.102:0/3840384000 >> v1:172.21.2.113:6789/0 conn(0x7fe26c1b0470 legacy=0x7fe26c0ef610 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:19:48.866 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:48.858 7fe26b7fe700 1 -- 172.21.2.102:0/3840384000 >> v1:172.21.2.102:6789/0 conn(0x7fe26c0efe00 legacy=0x7fe26c0f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:19:48.866 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:48.858 7fe26b7fe700 1 -- 172.21.2.102:0/3840384000 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe26c0e52e0 con 0x7fe26c1b0870 2021-02-10T11:19:48.866 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:48.858 7fe26b7fe700 1 --2- 172.21.2.102:0/3840384000 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe26c1b0870 0x7fe26c0f20e0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 rx=0x7fe254006650 tx=0x7fe254009a80).ready entity=mon.0 client_cookie=f7ffdd351e2be0d7 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:19:48.866 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:48.858 7fe26affd700 1 -- 172.21.2.102:0/3840384000 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe2540009b0 con 0x7fe26c1b0870 2021-02-10T11:19:48.866 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:48.858 7fe26affd700 1 -- 172.21.2.102:0/3840384000 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe25400c0c0 con 0x7fe26c1b0870 2021-02-10T11:19:48.866 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:48.858 7fe26affd700 1 -- 172.21.2.102:0/3840384000 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe25400c5e0 con 0x7fe26c1b0870 2021-02-10T11:19:48.866 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:48.858 7fe272951700 1 -- 172.21.2.102:0/3840384000 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe26c1b0870 msgr2=0x7fe26c0f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:19:48.866 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:48.858 7fe272951700 1 --2- 172.21.2.102:0/3840384000 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe26c1b0870 0x7fe26c0f20e0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 rx=0x7fe254006650 tx=0x7fe254009a80).stop 2021-02-10T11:19:48.867 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:48.859 7fe272951700 1 -- 172.21.2.102:0/3840384000 shutdown_connections 2021-02-10T11:19:48.867 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:48.859 7fe272951700 1 --2- 172.21.2.102:0/3840384000 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe26c1b0870 0x7fe26c0f20e0 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:48.867 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:48.859 7fe272951700 1 -- 172.21.2.102:0/3840384000 >> 172.21.2.102:0/3840384000 conn(0x7fe26c0ce3e0 msgr2=0x7fe26c0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:19:48.867 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:48.859 7fe272951700 1 -- 172.21.2.102:0/3840384000 shutdown_connections 2021-02-10T11:19:48.867 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:48.859 7fe272951700 1 -- 172.21.2.102:0/3840384000 wait complete. 2021-02-10T11:19:48.868 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:48.874 7fe272951700 1 Processor -- start 2021-02-10T11:19:48.868 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:48.874 7fe272951700 1 -- start start 2021-02-10T11:19:48.868 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:48.874 7fe272951700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe26c087510 0x7fe26c0aedc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:48.868 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:48.874 7fe272951700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe26c1c68f0 0x7fe26c1b1f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:48.868 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:48.874 7fe272951700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe26c08c4d0 0x7fe26c1b18a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:48.868 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:48.874 7fe272951700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe26c0a2d50 con 0x7fe26c08c4d0 2021-02-10T11:19:48.869 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:48.874 7fe272951700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe26c0a2f70 con 0x7fe26c1c68f0 2021-02-10T11:19:48.869 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:48.874 7fe272951700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe26c0a3190 con 0x7fe26c087510 2021-02-10T11:19:48.869 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:48.874 7fe270f6e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe26c08c4d0 0x7fe26c1b18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:19:48.869 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:48.874 7fe270f6e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe26c08c4d0 0x7fe26c1b18a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:53586/0 (socket says 172.21.2.102:53586) 2021-02-10T11:19:48.869 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:48.874 7fe270f6e700 1 -- 172.21.2.102:0/3447541517 learned_addr learned my addr 172.21.2.102:0/3447541517 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:19:48.869 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:48.875 7fe270f6e700 1 -- 172.21.2.102:0/3447541517 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe26c087510 msgr2=0x7fe26c0aedc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:19:48.869 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:48.875 7fe270f6e700 1 --2- 172.21.2.102:0/3447541517 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe26c087510 0x7fe26c0aedc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:19:48.869 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:48.875 7fe270f6e700 1 -- 172.21.2.102:0/3447541517 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe26c1c68f0 msgr2=0x7fe26c1b1f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:19:48.869 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:48.875 7fe270f6e700 1 --2- 172.21.2.102:0/3447541517 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe26c1c68f0 0x7fe26c1b1f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:19:48.869 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:48.875 7fe270f6e700 1 -- 172.21.2.102:0/3447541517 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe26c0a33b0 con 0x7fe26c08c4d0 2021-02-10T11:19:48.869 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:48.875 7fe270f6e700 1 --2- 172.21.2.102:0/3447541517 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe26c08c4d0 0x7fe26c1b18a0 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 rx=0x7fe26400a940 tx=0x7fe26400ace0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:19:48.869 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:48.875 7fe268ff9700 1 -- 172.21.2.102:0/3447541517 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe264001270 con 0x7fe26c08c4d0 2021-02-10T11:19:48.869 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:48.875 7fe268ff9700 1 -- 172.21.2.102:0/3447541517 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2640070b0 con 0x7fe26c08c4d0 2021-02-10T11:19:48.870 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:48.875 7fe272951700 1 -- 172.21.2.102:0/3447541517 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe26c0a33b0 con 0x7fe26c08c4d0 2021-02-10T11:19:48.870 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:48.875 7fe268ff9700 1 -- 172.21.2.102:0/3447541517 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe264007540 con 0x7fe26c08c4d0 2021-02-10T11:19:48.870 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:48.875 7fe272951700 1 -- 172.21.2.102:0/3447541517 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe26c0afca0 con 0x7fe26c08c4d0 2021-02-10T11:19:48.870 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:48.876 7fe268ff9700 1 -- 172.21.2.102:0/3447541517 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fe264007290 con 0x7fe26c08c4d0 2021-02-10T11:19:48.870 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:48.876 7fe268ff9700 1 --2- 172.21.2.102:0/3447541517 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe2500186c0 0x7fe25001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:48.870 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:48.876 7fe268ff9700 1 -- 172.21.2.102:0/3447541517 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fe264032730 con 0x7fe26c08c4d0 2021-02-10T11:19:48.870 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:48.876 7fe26bfff700 1 --2- 172.21.2.102:0/3447541517 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe2500186c0 0x7fe25001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:19:48.870 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:48.876 7fe26bfff700 1 --2- 172.21.2.102:0/3447541517 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe2500186c0 0x7fe25001abc0 crc :-1 s=READY pgs=375 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:19:48.899 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:48.908 7fe272951700 1 -- 172.21.2.102:0/3447541517 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe26c07e6b0 con 0x7fe26c08c4d0 2021-02-10T11:19:48.902 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:48.910 7fe268ff9700 1 -- 172.21.2.102:0/3447541517 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fe2640279c0 con 0x7fe26c08c4d0 2021-02-10T11:19:49.029 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:49.037 7fe272951700 1 -- 172.21.2.102:0/3447541517 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344459", "target": ["mgr", ""]}) v1 -- 0x7fe26c0a2d50 con 0x7fe2500186c0 2021-02-10T11:19:49.056 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:49.064 7fe268ff9700 1 -- 172.21.2.102:0/3447541517 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fe25c006e40 con 0x7fe2500186c0 2021-02-10T11:19:49.062 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:49.070 7fe272951700 1 -- 172.21.2.102:0/3447541517 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe2500186c0 msgr2=0x7fe25001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:19:49.062 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:49.070 7fe272951700 1 --2- 172.21.2.102:0/3447541517 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe2500186c0 0x7fe25001abc0 crc :-1 s=READY pgs=375 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:49.062 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:49.070 7fe272951700 1 -- 172.21.2.102:0/3447541517 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe26c08c4d0 msgr2=0x7fe26c1b18a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:19:49.062 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:49.070 7fe272951700 1 --2- 172.21.2.102:0/3447541517 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe26c08c4d0 0x7fe26c1b18a0 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 rx=0x7fe26400a940 tx=0x7fe26400ace0).stop 2021-02-10T11:19:49.062 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:49.070 7fe272951700 1 -- 172.21.2.102:0/3447541517 shutdown_connections 2021-02-10T11:19:49.062 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:49.070 7fe272951700 1 --2- 172.21.2.102:0/3447541517 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe26c08c4d0 0x7fe26c1b18a0 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:49.062 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:49.070 7fe272951700 1 --2- 172.21.2.102:0/3447541517 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe2500186c0 0x7fe25001abc0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:49.062 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:49.070 7fe272951700 1 --2- 172.21.2.102:0/3447541517 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe26c1c68f0 0x7fe26c1b1f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:19:49.063 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:49.070 7fe272951700 1 --2- 172.21.2.102:0/3447541517 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe26c087510 0x7fe26c0aedc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:19:49.063 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:49.070 7fe272951700 1 -- 172.21.2.102:0/3447541517 >> 172.21.2.102:0/3447541517 conn(0x7fe26c0886d0 msgr2=0x7fe26c081c40 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:19:49.063 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:49.070 7fe272951700 1 -- 172.21.2.102:0/3447541517 shutdown_connections 2021-02-10T11:19:49.063 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:49.070 7fe272951700 1 -- 172.21.2.102:0/3447541517 wait complete. 2021-02-10T11:19:49.070 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000344459 2021-02-10T11:19:49.504 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:49.496 7f839d4c8700 1 Processor -- start 2021-02-10T11:19:49.504 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:49.496 7f839d4c8700 1 -- start start 2021-02-10T11:19:49.504 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:49.497 7f839d4c8700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f83981b0870 0x7f83980f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:49.504 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:49.497 7f839d4c8700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f83980efe00 0x7f83980f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:49.504 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:49.497 7f839d4c8700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83981b0c70 con 0x7f83981b0470 2021-02-10T11:19:49.504 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:49.497 7f839d4c8700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83980e5ea0 con 0x7f83981b0870 2021-02-10T11:19:49.504 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:49.497 7f839d4c8700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83980f0b50 con 0x7f83980efe00 2021-02-10T11:19:49.504 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:49.497 7f8396abe700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f83981b0870 0x7f83980f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:19:49.504 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:49.497 7f8396abe700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f83981b0870 0x7f83980f20e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:53590/0 (socket says 172.21.2.102:53590) 2021-02-10T11:19:49.505 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:49.497 7f8396abe700 1 -- 172.21.2.102:0/1152212219 learned_addr learned my addr 172.21.2.102:0/1152212219 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:19:49.505 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:49.497 7f8396abe700 1 -- 172.21.2.102:0/1152212219 >> v2:172.21.2.113:3300/0 conn(0x7f83980efe00 msgr2=0x7f83980f0200 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:19:49.505 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:49.497 7f8396abe700 1 --2- 172.21.2.102:0/1152212219 >> v2:172.21.2.113:3300/0 conn(0x7f83980efe00 0x7f83980f0200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:19:49.505 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:49.497 7f8396abe700 1 -- 172.21.2.102:0/1152212219 >> v1:172.21.2.113:6789/0 conn(0x7f83981b0470 legacy=0x7f83980ef610 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:19:49.505 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:49.497 7f8396abe700 1 -- 172.21.2.102:0/1152212219 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83980f0820 con 0x7f83981b0870 2021-02-10T11:19:49.505 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:49.497 7f8396abe700 1 --2- 172.21.2.102:0/1152212219 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f83981b0870 0x7f83980f20e0 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 rx=0x7f8380006700 tx=0x7f8380009b80).ready entity=mon.0 client_cookie=ccbe28b77dd10e0c server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:19:49.505 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:49.497 7f83962bd700 1 -- 172.21.2.102:0/1152212219 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f838000c0c0 con 0x7f83981b0870 2021-02-10T11:19:49.505 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:49.497 7f83962bd700 1 -- 172.21.2.102:0/1152212219 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f838000c340 con 0x7f83981b0870 2021-02-10T11:19:49.505 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:49.497 7f83962bd700 1 -- 172.21.2.102:0/1152212219 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f838000c820 con 0x7f83981b0870 2021-02-10T11:19:49.505 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:49.497 7f839d4c8700 1 -- 172.21.2.102:0/1152212219 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f83981b0870 msgr2=0x7f83980f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:19:49.505 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:49.497 7f839d4c8700 1 --2- 172.21.2.102:0/1152212219 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f83981b0870 0x7f83980f20e0 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 rx=0x7f8380006700 tx=0x7f8380009b80).stop 2021-02-10T11:19:49.505 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:49.498 7f839d4c8700 1 -- 172.21.2.102:0/1152212219 shutdown_connections 2021-02-10T11:19:49.505 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:49.498 7f839d4c8700 1 --2- 172.21.2.102:0/1152212219 >> v2:172.21.2.113:3300/0 conn(0x7f83980efe00 0x7f83980f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:19:49.505 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:49.498 7f839d4c8700 1 --2- 172.21.2.102:0/1152212219 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f83981b0870 0x7f83980f20e0 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:49.506 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:49.498 7f839d4c8700 1 -- 172.21.2.102:0/1152212219 >> 172.21.2.102:0/1152212219 conn(0x7f83980ce3e0 msgr2=0x7f83980e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:19:49.506 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:49.498 7f839d4c8700 1 -- 172.21.2.102:0/1152212219 shutdown_connections 2021-02-10T11:19:49.506 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:49.498 7f839d4c8700 1 -- 172.21.2.102:0/1152212219 wait complete. 2021-02-10T11:19:49.507 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:49.512 7f839d4c8700 1 Processor -- start 2021-02-10T11:19:49.507 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:49.512 7f839d4c8700 1 -- start start 2021-02-10T11:19:49.507 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:49.513 7f839d4c8700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f83980c36a0 0x7f83980c2b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:49.507 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:49.513 7f839d4c8700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f83980c3190 0x7f83980b1d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:49.507 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:49.513 7f839d4c8700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f83980b2370 0x7f83980b2d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:49.508 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:49.513 7f839d4c8700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83980b27c0 con 0x7f83980c3190 2021-02-10T11:19:49.508 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:49.513 7f839d4c8700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83980b3360 con 0x7f83980c36a0 2021-02-10T11:19:49.508 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:49.513 7f839d4c8700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83980b3540 con 0x7f83980b2370 2021-02-10T11:19:49.508 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:49.513 7f8396abe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f83980c3190 0x7f83980b1d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:19:49.508 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:49.513 7f8396abe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f83980c3190 0x7f83980b1d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:53598/0 (socket says 172.21.2.102:53598) 2021-02-10T11:19:49.508 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:49.513 7f8396abe700 1 -- 172.21.2.102:0/1471696169 learned_addr learned my addr 172.21.2.102:0/1471696169 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:19:49.508 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:49.513 7f8396abe700 1 -- 172.21.2.102:0/1471696169 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f83980b2370 msgr2=0x7f83980b2d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:19:49.508 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:49.513 7f8396abe700 1 --2- 172.21.2.102:0/1471696169 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f83980b2370 0x7f83980b2d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:19:49.508 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:49.513 7f8396abe700 1 -- 172.21.2.102:0/1471696169 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f83980c36a0 msgr2=0x7f83980c2b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:19:49.508 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:49.513 7f8396abe700 1 --2- 172.21.2.102:0/1471696169 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f83980c36a0 0x7f83980c2b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:19:49.508 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:49.513 7f8396abe700 1 -- 172.21.2.102:0/1471696169 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83980b3760 con 0x7f83980c3190 2021-02-10T11:19:49.508 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:49.514 7f8396abe700 1 --2- 172.21.2.102:0/1471696169 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f83980c3190 0x7f83980b1d50 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 rx=0x7f838000b650 tx=0x7f838000b9f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:19:49.508 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:49.514 7f838ffff700 1 -- 172.21.2.102:0/1471696169 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f838000bde0 con 0x7f83980c3190 2021-02-10T11:19:49.509 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:49.514 7f838ffff700 1 -- 172.21.2.102:0/1471696169 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8380000c90 con 0x7f83980c3190 2021-02-10T11:19:49.509 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:49.514 7f838ffff700 1 -- 172.21.2.102:0/1471696169 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8380002220 con 0x7f83980c3190 2021-02-10T11:19:49.509 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:49.514 7f839d4c8700 1 -- 172.21.2.102:0/1471696169 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83980b3760 con 0x7f83980c3190 2021-02-10T11:19:49.509 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:49.514 7f839d4c8700 1 -- 172.21.2.102:0/1471696169 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83980b3cf0 con 0x7f83980c3190 2021-02-10T11:19:49.509 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:49.514 7f838ffff700 1 -- 172.21.2.102:0/1471696169 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f8380009f90 con 0x7f83980c3190 2021-02-10T11:19:49.509 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:49.514 7f838ffff700 1 --2- 172.21.2.102:0/1471696169 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f837c0186c0 0x7f837c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:49.509 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:49.514 7f838ffff700 1 -- 172.21.2.102:0/1471696169 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f83800325a0 con 0x7f83980c3190 2021-02-10T11:19:49.509 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:49.514 7f83972bf700 1 --2- 172.21.2.102:0/1471696169 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f837c0186c0 0x7f837c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:19:49.509 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:49.515 7f83972bf700 1 --2- 172.21.2.102:0/1471696169 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f837c0186c0 0x7f837c01abc0 crc :-1 s=READY pgs=376 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:19:49.538 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:49.546 7f839d4c8700 1 -- 172.21.2.102:0/1471696169 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83980edd10 con 0x7f83980c3190 2021-02-10T11:19:49.541 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:49.549 7f838ffff700 1 -- 172.21.2.102:0/1471696169 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f83800009b0 con 0x7f83980c3190 2021-02-10T11:19:49.670 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:49.678 7f839d4c8700 1 -- 172.21.2.102:0/1471696169 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344459", "target": ["mgr", ""]}) v1 -- 0x7f83980b27c0 con 0x7f837c0186c0 2021-02-10T11:19:49.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:49.681 7f838ffff700 1 -- 172.21.2.102:0/1471696169 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f8388006e40 con 0x7f837c0186c0 2021-02-10T11:19:49.673 INFO:teuthology.orchestra.run.gibba002.stdout:/volumes/_nogroup/subvolume_0000000000344459/004e24e5-8c7e-49f5-b9a6-1dc4e9579b21 2021-02-10T11:19:49.674 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:49.683 7f839d4c8700 1 -- 172.21.2.102:0/1471696169 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f837c0186c0 msgr2=0x7f837c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:19:49.675 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:49.683 7f839d4c8700 1 --2- 172.21.2.102:0/1471696169 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f837c0186c0 0x7f837c01abc0 crc :-1 s=READY pgs=376 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:49.675 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:49.683 7f839d4c8700 1 -- 172.21.2.102:0/1471696169 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f83980c3190 msgr2=0x7f83980b1d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:19:49.675 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:49.683 7f839d4c8700 1 --2- 172.21.2.102:0/1471696169 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f83980c3190 0x7f83980b1d50 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 rx=0x7f838000b650 tx=0x7f838000b9f0).stop 2021-02-10T11:19:49.675 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:49.683 7f839d4c8700 1 -- 172.21.2.102:0/1471696169 shutdown_connections 2021-02-10T11:19:49.675 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:49.683 7f839d4c8700 1 --2- 172.21.2.102:0/1471696169 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f83980b2370 0x7f83980b2d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:19:49.675 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:49.683 7f839d4c8700 1 --2- 172.21.2.102:0/1471696169 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f83980c3190 0x7f83980b1d50 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:49.675 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:49.683 7f839d4c8700 1 --2- 172.21.2.102:0/1471696169 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f837c0186c0 0x7f837c01abc0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:49.675 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:49.683 7f839d4c8700 1 --2- 172.21.2.102:0/1471696169 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f83980c36a0 0x7f83980c2b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:19:49.675 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:49.683 7f839d4c8700 1 -- 172.21.2.102:0/1471696169 >> 172.21.2.102:0/1471696169 conn(0x7f839809f630 msgr2=0x7f83980a0130 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:19:49.676 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:49.683 7f839d4c8700 1 -- 172.21.2.102:0/1471696169 shutdown_connections 2021-02-10T11:19:49.676 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:49.683 7f839d4c8700 1 -- 172.21.2.102:0/1471696169 wait complete. 2021-02-10T11:19:49.683 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000344459 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000344459/004e24e5-8c7e-49f5-b9a6-1dc4e9579b21 2021-02-10T11:19:49.683 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:19:49.683 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:19:49.683 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:19:50.020 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:19:50.021 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:19:50.021 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344459/004e24e5-8c7e-49f5-b9a6-1dc4e9579b21/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:19:50.405 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:19:50.406 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:19:50.406 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.024316 s, 43.1 MB/s 2021-02-10T11:19:50.406 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:19:50.407 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:19:50.407 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:19:50.743 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:19:50.743 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:19:50.743 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344459/004e24e5-8c7e-49f5-b9a6-1dc4e9579b21/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:19:51.118 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:19:51.119 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:19:51.119 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0246331 s, 42.6 MB/s 2021-02-10T11:19:51.120 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:19:51.120 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:19:51.120 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:19:51.457 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:19:51.457 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:19:51.457 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344459/004e24e5-8c7e-49f5-b9a6-1dc4e9579b21/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:19:51.833 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:19:51.833 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:19:51.833 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.023013 s, 45.6 MB/s 2021-02-10T11:19:51.834 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:19:51.834 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:19:51.834 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:19:52.171 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:19:52.171 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:19:52.171 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344459/004e24e5-8c7e-49f5-b9a6-1dc4e9579b21/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:19:52.547 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:19:52.548 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:19:52.548 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0249308 s, 42.1 MB/s 2021-02-10T11:19:52.549 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:19:52.549 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:19:52.549 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:19:52.884 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:19:52.884 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:19:52.884 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344459/004e24e5-8c7e-49f5-b9a6-1dc4e9579b21/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:19:53.261 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:19:53.261 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:19:53.261 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.02304 s, 45.5 MB/s 2021-02-10T11:19:53.262 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:19:53.262 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:19:53.262 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:19:53.597 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:19:53.598 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:19:53.598 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344459/004e24e5-8c7e-49f5-b9a6-1dc4e9579b21/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:19:53.972 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:19:53.972 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:19:53.972 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0238628 s, 43.9 MB/s 2021-02-10T11:19:53.973 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:19:53.973 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:19:53.973 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:19:54.310 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:19:54.311 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:19:54.311 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344459/004e24e5-8c7e-49f5-b9a6-1dc4e9579b21/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:19:54.687 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:19:54.688 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:19:54.688 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0230757 s, 45.4 MB/s 2021-02-10T11:19:54.689 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:19:54.689 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:19:54.689 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:19:55.025 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:19:55.025 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:19:55.025 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344459/004e24e5-8c7e-49f5-b9a6-1dc4e9579b21/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:19:55.405 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:19:55.405 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:19:55.405 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0248183 s, 42.3 MB/s 2021-02-10T11:19:55.406 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:19:55.406 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:19:55.406 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:19:55.743 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:19:55.743 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:19:55.743 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344459/004e24e5-8c7e-49f5-b9a6-1dc4e9579b21/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:19:56.122 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:19:56.123 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:19:56.123 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0244262 s, 42.9 MB/s 2021-02-10T11:19:56.123 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:19:56.124 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:19:56.124 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:19:56.459 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:19:56.460 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:19:56.460 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344459/004e24e5-8c7e-49f5-b9a6-1dc4e9579b21/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:19:56.838 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:19:56.839 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:19:56.839 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0252405 s, 41.5 MB/s 2021-02-10T11:19:56.840 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000344460 2021-02-10T11:19:57.274 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:57.264 7f8151cf8700 1 Processor -- start 2021-02-10T11:19:57.274 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:57.264 7f8151cf8700 1 -- start start 2021-02-10T11:19:57.274 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:57.265 7f8151cf8700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f814c1b0470 0x7f814c1b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:57.274 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:57.265 7f8151cf8700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f814c0ef610 0x7f814c0efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:57.274 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:57.265 7f8151cf8700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f814c0f0030 0x7f814c0f0430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:57.275 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:57.265 7f8151cf8700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f814c0e5ea0 con 0x7f814c0f0030 2021-02-10T11:19:57.275 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:57.265 7f8151cf8700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f814c0f0b50 con 0x7f814c0ef610 2021-02-10T11:19:57.275 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:57.265 7f8151cf8700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f814c0e52e0 con 0x7f814c1b0470 2021-02-10T11:19:57.275 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:57.265 7f814bfff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f814c0f0030 0x7f814c0f0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:19:57.275 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:57.265 7f814bfff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f814c0f0030 0x7f814c0f0430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:53604/0 (socket says 172.21.2.102:53604) 2021-02-10T11:19:57.275 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:57.265 7f814bfff700 1 -- 172.21.2.102:0/3838787178 learned_addr learned my addr 172.21.2.102:0/3838787178 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:19:57.275 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:57.265 7f814affd700 1 --2- 172.21.2.102:0/3838787178 >> v2:172.21.2.113:3300/0 conn(0x7f814c0ef610 0x7f814c0efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:19:57.275 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:57.265 7f814b7fe700 1 --2- 172.21.2.102:0/3838787178 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f814c1b0470 0x7f814c1b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:19:57.275 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:57.265 7f814bfff700 1 -- 172.21.2.102:0/3838787178 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f814c1b0470 msgr2=0x7f814c1b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:19:57.275 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:57.265 7f814bfff700 1 --2- 172.21.2.102:0/3838787178 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f814c1b0470 0x7f814c1b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:57.275 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:57.265 7f814bfff700 1 -- 172.21.2.102:0/3838787178 >> v2:172.21.2.113:3300/0 conn(0x7f814c0ef610 msgr2=0x7f814c0efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:19:57.275 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:57.265 7f814bfff700 1 --2- 172.21.2.102:0/3838787178 >> v2:172.21.2.113:3300/0 conn(0x7f814c0ef610 0x7f814c0efa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:57.275 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:57.265 7f814bfff700 1 -- 172.21.2.102:0/3838787178 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f814c0f10d0 con 0x7f814c0f0030 2021-02-10T11:19:57.275 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:57.265 7f814b7fe700 1 --2- 172.21.2.102:0/3838787178 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f814c1b0470 0x7f814c1b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:19:57.276 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:57.266 7f814bfff700 1 --2- 172.21.2.102:0/3838787178 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f814c0f0030 0x7f814c0f0430 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 rx=0x7f8144006f00 tx=0x7f8144009f10).ready entity=mon.0 client_cookie=c44b037320763214 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:19:57.276 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:57.266 7f814a7fc700 1 -- 172.21.2.102:0/3838787178 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f814400b3b0 con 0x7f814c0f0030 2021-02-10T11:19:57.276 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:57.266 7f814a7fc700 1 -- 172.21.2.102:0/3838787178 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f814400b670 con 0x7f814c0f0030 2021-02-10T11:19:57.276 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:57.266 7f814a7fc700 1 -- 172.21.2.102:0/3838787178 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f814400bb50 con 0x7f814c0f0030 2021-02-10T11:19:57.276 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:57.266 7f8151cf8700 1 -- 172.21.2.102:0/3838787178 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f814c0f0030 msgr2=0x7f814c0f0430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:19:57.276 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:57.266 7f8151cf8700 1 --2- 172.21.2.102:0/3838787178 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f814c0f0030 0x7f814c0f0430 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 rx=0x7f8144006f00 tx=0x7f8144009f10).stop 2021-02-10T11:19:57.276 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:57.267 7f8151cf8700 1 -- 172.21.2.102:0/3838787178 shutdown_connections 2021-02-10T11:19:57.276 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:57.267 7f8151cf8700 1 --2- 172.21.2.102:0/3838787178 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f814c0f0030 0x7f814c0f0430 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:57.276 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:57.267 7f8151cf8700 1 --2- 172.21.2.102:0/3838787178 >> v2:172.21.2.113:3300/0 conn(0x7f814c0ef610 0x7f814c0efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:57.276 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:57.267 7f8151cf8700 1 --2- 172.21.2.102:0/3838787178 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f814c1b0470 0x7f814c1b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:57.276 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:57.267 7f8151cf8700 1 -- 172.21.2.102:0/3838787178 >> 172.21.2.102:0/3838787178 conn(0x7f814c0ce3e0 msgr2=0x7f814c0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:19:57.276 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:57.267 7f8151cf8700 1 -- 172.21.2.102:0/3838787178 shutdown_connections 2021-02-10T11:19:57.276 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:57.267 7f8151cf8700 1 -- 172.21.2.102:0/3838787178 wait complete. 2021-02-10T11:19:57.278 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:57.281 7f8151cf8700 1 Processor -- start 2021-02-10T11:19:57.278 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:57.281 7f8151cf8700 1 -- start start 2021-02-10T11:19:57.278 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:57.282 7f8151cf8700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f814c09b930 0x7f814c0b7c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:57.278 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:57.282 7f8151cf8700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f814c0b8410 0x7f814c0b8810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:57.278 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:57.282 7f8151cf8700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f814c0c1f80 0x7f814c0c2380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:57.278 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:57.282 7f8151cf8700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f814c0c1a00 con 0x7f814c0b8410 2021-02-10T11:19:57.278 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:57.282 7f8151cf8700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f814c0b8e30 con 0x7f814c09b930 2021-02-10T11:19:57.278 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:57.282 7f8151cf8700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f814c0b9050 con 0x7f814c0c1f80 2021-02-10T11:19:57.278 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:57.282 7f814affd700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f814c0b8410 0x7f814c0b8810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:19:57.279 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:57.282 7f814affd700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f814c0b8410 0x7f814c0b8810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:53608/0 (socket says 172.21.2.102:53608) 2021-02-10T11:19:57.279 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:57.282 7f814affd700 1 -- 172.21.2.102:0/1916366869 learned_addr learned my addr 172.21.2.102:0/1916366869 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:19:57.279 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:57.282 7f814affd700 1 -- 172.21.2.102:0/1916366869 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f814c0c1f80 msgr2=0x7f814c0c2380 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:19:57.279 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:57.282 7f814affd700 1 --2- 172.21.2.102:0/1916366869 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f814c0c1f80 0x7f814c0c2380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:19:57.279 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:57.282 7f814affd700 1 -- 172.21.2.102:0/1916366869 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f814c09b930 msgr2=0x7f814c0b7c20 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:19:57.279 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:57.282 7f814affd700 1 --2- 172.21.2.102:0/1916366869 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f814c09b930 0x7f814c0b7c20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:19:57.279 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:57.282 7f814affd700 1 -- 172.21.2.102:0/1916366869 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f814c0c29a0 con 0x7f814c0b8410 2021-02-10T11:19:57.279 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:57.282 7f814affd700 1 --2- 172.21.2.102:0/1916366869 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f814c0b8410 0x7f814c0b8810 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 rx=0x7f813c006f60 tx=0x7f813c00b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:19:57.279 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:57.283 7f8143fff700 1 -- 172.21.2.102:0/1916366869 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f813c00bd60 con 0x7f814c0b8410 2021-02-10T11:19:57.279 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:57.283 7f8143fff700 1 -- 172.21.2.102:0/1916366869 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f813c000c90 con 0x7f814c0b8410 2021-02-10T11:19:57.279 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:57.283 7f8151cf8700 1 -- 172.21.2.102:0/1916366869 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f814c0c29a0 con 0x7f814c0b8410 2021-02-10T11:19:57.279 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:57.283 7f8151cf8700 1 -- 172.21.2.102:0/1916366869 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f814c0c2f30 con 0x7f814c0b8410 2021-02-10T11:19:57.279 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:57.283 7f8143fff700 1 -- 172.21.2.102:0/1916366869 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f813c0022b0 con 0x7f814c0b8410 2021-02-10T11:19:57.280 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:57.283 7f8143fff700 1 -- 172.21.2.102:0/1916366869 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f813c006b00 con 0x7f814c0b8410 2021-02-10T11:19:57.280 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:57.283 7f8143fff700 1 --2- 172.21.2.102:0/1916366869 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f81300186c0 0x7f813001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:57.280 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:57.283 7f8143fff700 1 -- 172.21.2.102:0/1916366869 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f813c032540 con 0x7f814c0b8410 2021-02-10T11:19:57.280 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:57.283 7f814b7fe700 1 --2- 172.21.2.102:0/1916366869 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f81300186c0 0x7f813001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:19:57.280 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:57.283 7f814b7fe700 1 --2- 172.21.2.102:0/1916366869 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f81300186c0 0x7f813001abc0 crc :-1 s=READY pgs=377 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:19:57.307 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:57.314 7f8151cf8700 1 -- 172.21.2.102:0/1916366869 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f814c105b30 con 0x7f814c0b8410 2021-02-10T11:19:57.310 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:57.317 7f8143fff700 1 -- 172.21.2.102:0/1916366869 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f813c000940 con 0x7f814c0b8410 2021-02-10T11:19:57.436 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:57.443 7f8151cf8700 1 -- 172.21.2.102:0/1916366869 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344460", "target": ["mgr", ""]}) v1 -- 0x7f814c0f0b50 con 0x7f81300186c0 2021-02-10T11:19:57.462 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:57.469 7f8143fff700 1 -- 172.21.2.102:0/1916366869 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f81340009b0 con 0x7f81300186c0 2021-02-10T11:19:57.469 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:57.476 7f8151cf8700 1 -- 172.21.2.102:0/1916366869 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f81300186c0 msgr2=0x7f813001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:19:57.469 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:57.476 7f8151cf8700 1 --2- 172.21.2.102:0/1916366869 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f81300186c0 0x7f813001abc0 crc :-1 s=READY pgs=377 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:57.469 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:57.476 7f8151cf8700 1 -- 172.21.2.102:0/1916366869 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f814c0b8410 msgr2=0x7f814c0b8810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:19:57.469 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:57.476 7f8151cf8700 1 --2- 172.21.2.102:0/1916366869 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f814c0b8410 0x7f814c0b8810 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 rx=0x7f813c006f60 tx=0x7f813c00b970).stop 2021-02-10T11:19:57.469 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:57.476 7f8151cf8700 1 -- 172.21.2.102:0/1916366869 shutdown_connections 2021-02-10T11:19:57.469 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:57.476 7f8151cf8700 1 --2- 172.21.2.102:0/1916366869 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f814c0c1f80 0x7f814c0c2380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:19:57.469 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:57.476 7f8151cf8700 1 --2- 172.21.2.102:0/1916366869 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f814c0b8410 0x7f814c0b8810 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:57.469 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:57.476 7f8151cf8700 1 --2- 172.21.2.102:0/1916366869 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f81300186c0 0x7f813001abc0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:57.469 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:57.476 7f8151cf8700 1 --2- 172.21.2.102:0/1916366869 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f814c09b930 0x7f814c0b7c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:19:57.469 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:57.476 7f8151cf8700 1 -- 172.21.2.102:0/1916366869 >> 172.21.2.102:0/1916366869 conn(0x7f814c094e10 msgr2=0x7f814c0a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:19:57.470 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:57.476 7f8151cf8700 1 -- 172.21.2.102:0/1916366869 shutdown_connections 2021-02-10T11:19:57.470 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:57.476 7f8151cf8700 1 -- 172.21.2.102:0/1916366869 wait complete. 2021-02-10T11:19:57.477 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000344460 2021-02-10T11:19:57.910 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:57.901 7f78a48d5700 1 Processor -- start 2021-02-10T11:19:57.910 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:57.901 7f78a48d5700 1 -- start start 2021-02-10T11:19:57.910 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:57.902 7f78a48d5700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f78a01b0470 0x7f78a01b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:57.910 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:57.902 7f78a48d5700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f78a00f20e0 0x7f78a00f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:57.910 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:57.902 7f78a48d5700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78a00f0b50 con 0x7f78a00ef610 2021-02-10T11:19:57.911 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:57.902 7f78a48d5700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78a00f0820 con 0x7f78a00f20e0 2021-02-10T11:19:57.911 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:57.902 7f78a48d5700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78a00f24e0 con 0x7f78a01b0470 2021-02-10T11:19:57.911 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:57.902 7f789ee1d700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f78a00f20e0 0x7f78a00f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:19:57.911 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:57.902 7f789ee1d700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f78a00f20e0 0x7f78a00f0200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:53614/0 (socket says 172.21.2.102:53614) 2021-02-10T11:19:57.911 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:57.902 7f789ee1d700 1 -- 172.21.2.102:0/3619102027 learned_addr learned my addr 172.21.2.102:0/3619102027 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:19:57.911 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:57.902 7f789de1b700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f78a00ef610 0x7f78a00efa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:45838/0 (socket says 172.21.2.102:45838) 2021-02-10T11:19:57.911 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:57.902 7f789ee1d700 1 -- 172.21.2.102:0/3619102027 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f78a01b0470 msgr2=0x7f78a01b0870 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:19:57.911 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:57.902 7f789ee1d700 1 --2- 172.21.2.102:0/3619102027 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f78a01b0470 0x7f78a01b0870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:19:57.911 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:57.902 7f789ee1d700 1 -- 172.21.2.102:0/3619102027 >> v1:172.21.2.102:6789/0 conn(0x7f78a00ef610 legacy=0x7f78a00efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:19:57.911 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:57.902 7f789ee1d700 1 -- 172.21.2.102:0/3619102027 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78a00e52e0 con 0x7f78a00f20e0 2021-02-10T11:19:57.911 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:57.902 7f789ee1d700 1 --2- 172.21.2.102:0/3619102027 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f78a00f20e0 0x7f78a00f0200 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 rx=0x7f7898006f00 tx=0x7f7898009f10).ready entity=mon.0 client_cookie=137bc92c79ba4200 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:19:57.911 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:57.902 7f789d61a700 1 -- 172.21.2.102:0/3619102027 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f789800b3b0 con 0x7f78a00f20e0 2021-02-10T11:19:57.912 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:57.902 7f789d61a700 1 -- 172.21.2.102:0/3619102027 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f789800b730 con 0x7f78a00f20e0 2021-02-10T11:19:57.912 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:57.902 7f789d61a700 1 -- 172.21.2.102:0/3619102027 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f789800bc50 con 0x7f78a00f20e0 2021-02-10T11:19:57.912 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:57.903 7f78a48d5700 1 -- 172.21.2.102:0/3619102027 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f78a00f20e0 msgr2=0x7f78a00f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:19:57.912 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:57.903 7f78a48d5700 1 --2- 172.21.2.102:0/3619102027 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f78a00f20e0 0x7f78a00f0200 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 rx=0x7f7898006f00 tx=0x7f7898009f10).stop 2021-02-10T11:19:57.912 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:57.903 7f78a48d5700 1 -- 172.21.2.102:0/3619102027 shutdown_connections 2021-02-10T11:19:57.912 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:57.903 7f78a48d5700 1 --2- 172.21.2.102:0/3619102027 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f78a00f20e0 0x7f78a00f0200 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:57.912 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:57.903 7f78a48d5700 1 --2- 172.21.2.102:0/3619102027 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f78a01b0470 0x7f78a01b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:19:57.912 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:57.903 7f78a48d5700 1 -- 172.21.2.102:0/3619102027 >> 172.21.2.102:0/3619102027 conn(0x7f78a00ce3e0 msgr2=0x7f78a00e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:19:57.912 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:57.903 7f78a48d5700 1 -- 172.21.2.102:0/3619102027 shutdown_connections 2021-02-10T11:19:57.912 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:57.903 7f78a48d5700 1 -- 172.21.2.102:0/3619102027 wait complete. 2021-02-10T11:19:57.914 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:57.917 7f78a48d5700 1 Processor -- start 2021-02-10T11:19:57.914 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:57.917 7f78a48d5700 1 -- start start 2021-02-10T11:19:57.914 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:57.918 7f78a48d5700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f78a00aefc0 0x7f78a00ac4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:57.914 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:57.918 7f78a48d5700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f78a008c430 0x7f78a008c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:57.914 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:57.918 7f78a48d5700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f78a008ce50 0x7f78a01a74c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:57.914 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:57.918 7f78a48d5700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78a008d250 con 0x7f78a008c430 2021-02-10T11:19:57.914 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:57.918 7f78a48d5700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78a008beb0 con 0x7f78a008ce50 2021-02-10T11:19:57.914 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:57.918 7f78a48d5700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78a01a7ae0 con 0x7f78a00aefc0 2021-02-10T11:19:57.914 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:57.918 7f789de1b700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f78a008c430 0x7f78a008c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:19:57.915 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:57.918 7f789de1b700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f78a008c430 0x7f78a008c830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:53620/0 (socket says 172.21.2.102:53620) 2021-02-10T11:19:57.915 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:57.918 7f789de1b700 1 -- 172.21.2.102:0/3805421318 learned_addr learned my addr 172.21.2.102:0/3805421318 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:19:57.915 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:57.918 7f789de1b700 1 -- 172.21.2.102:0/3805421318 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f78a00aefc0 msgr2=0x7f78a00ac4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:19:57.915 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:57.918 7f789de1b700 1 --2- 172.21.2.102:0/3805421318 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f78a00aefc0 0x7f78a00ac4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:19:57.915 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:57.918 7f789de1b700 1 -- 172.21.2.102:0/3805421318 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f78a008ce50 msgr2=0x7f78a01a74c0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:19:57.915 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:57.918 7f789de1b700 1 --2- 172.21.2.102:0/3805421318 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f78a008ce50 0x7f78a01a74c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:19:57.915 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:57.918 7f789de1b700 1 -- 172.21.2.102:0/3805421318 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78a01a7cc0 con 0x7f78a008c430 2021-02-10T11:19:57.915 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:57.918 7f789de1b700 1 --2- 172.21.2.102:0/3805421318 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f78a008c430 0x7f78a008c830 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 rx=0x7f7890001c60 tx=0x7f789000a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:19:57.915 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:57.918 7f7896ffd700 1 -- 172.21.2.102:0/3805421318 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f789000acc0 con 0x7f78a008c430 2021-02-10T11:19:57.915 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:57.918 7f7896ffd700 1 -- 172.21.2.102:0/3805421318 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7890004140 con 0x7f78a008c430 2021-02-10T11:19:57.915 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:57.918 7f78a48d5700 1 -- 172.21.2.102:0/3805421318 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78a01a7cc0 con 0x7f78a008c430 2021-02-10T11:19:57.915 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:57.918 7f7896ffd700 1 -- 172.21.2.102:0/3805421318 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7890004620 con 0x7f78a008c430 2021-02-10T11:19:57.915 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:57.919 7f78a48d5700 1 -- 172.21.2.102:0/3805421318 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78a01a7cc0 con 0x7f78a008c430 2021-02-10T11:19:57.916 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:57.919 7f7896ffd700 1 -- 172.21.2.102:0/3805421318 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f7890000ce0 con 0x7f78a008c430 2021-02-10T11:19:57.916 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:57.919 7f7896ffd700 1 --2- 172.21.2.102:0/3805421318 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f78840186d0 0x7f788401abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:19:57.916 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:57.919 7f7896ffd700 1 -- 172.21.2.102:0/3805421318 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f7890032620 con 0x7f78a008c430 2021-02-10T11:19:57.916 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:57.919 7f789e61c700 1 --2- 172.21.2.102:0/3805421318 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f78840186d0 0x7f788401abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:19:57.916 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:57.919 7f789e61c700 1 --2- 172.21.2.102:0/3805421318 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f78840186d0 0x7f788401abd0 crc :-1 s=READY pgs=378 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:19:57.916 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:57.920 7f78a48d5700 1 -- 172.21.2.102:0/3805421318 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78a00edd40 con 0x7f78a008c430 2021-02-10T11:19:57.916 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:57.922 7f7896ffd700 1 -- 172.21.2.102:0/3805421318 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f78900009b0 con 0x7f78a008c430 2021-02-10T11:19:58.046 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:58.052 7f78a48d5700 1 -- 172.21.2.102:0/3805421318 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344460", "target": ["mgr", ""]}) v1 -- 0x7f78a008d250 con 0x7f78840186d0 2021-02-10T11:19:58.049 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:58.056 7f7896ffd700 1 -- 172.21.2.102:0/3805421318 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f7888006970 con 0x7f78840186d0 2021-02-10T11:19:58.053 INFO:teuthology.orchestra.run.gibba002.stdout:/volumes/_nogroup/subvolume_0000000000344460/29d95c22-88b1-40f2-8b55-00181dd92fb2 2021-02-10T11:19:58.054 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:58.061 7f78a48d5700 1 -- 172.21.2.102:0/3805421318 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f78840186d0 msgr2=0x7f788401abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:19:58.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:58.061 7f78a48d5700 1 --2- 172.21.2.102:0/3805421318 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f78840186d0 0x7f788401abd0 crc :-1 s=READY pgs=378 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:58.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:58.061 7f78a48d5700 1 -- 172.21.2.102:0/3805421318 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f78a008c430 msgr2=0x7f78a008c830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:19:58.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:58.061 7f78a48d5700 1 --2- 172.21.2.102:0/3805421318 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f78a008c430 0x7f78a008c830 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 rx=0x7f7890001c60 tx=0x7f789000a8d0).stop 2021-02-10T11:19:58.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:58.061 7f78a48d5700 1 -- 172.21.2.102:0/3805421318 shutdown_connections 2021-02-10T11:19:58.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:58.061 7f78a48d5700 1 --2- 172.21.2.102:0/3805421318 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f78840186d0 0x7f788401abd0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:58.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:58.061 7f78a48d5700 1 --2- 172.21.2.102:0/3805421318 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f78a008ce50 0x7f78a01a74c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:19:58.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:58.061 7f78a48d5700 1 --2- 172.21.2.102:0/3805421318 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f78a008c430 0x7f78a008c830 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:19:58.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:58.061 7f78a48d5700 1 --2- 172.21.2.102:0/3805421318 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f78a00aefc0 0x7f78a00ac4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:19:58.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:58.061 7f78a48d5700 1 -- 172.21.2.102:0/3805421318 >> 172.21.2.102:0/3805421318 conn(0x7f78a00c40d0 msgr2=0x7f78a00a2ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:19:58.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:58.061 7f78a48d5700 1 -- 172.21.2.102:0/3805421318 shutdown_connections 2021-02-10T11:19:58.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:19:58.061 7f78a48d5700 1 -- 172.21.2.102:0/3805421318 wait complete. 2021-02-10T11:19:58.062 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000344460 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000344460/29d95c22-88b1-40f2-8b55-00181dd92fb2 2021-02-10T11:19:58.062 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:19:58.062 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:19:58.063 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:19:58.397 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:19:58.398 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:19:58.398 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344460/29d95c22-88b1-40f2-8b55-00181dd92fb2/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:19:58.780 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:19:58.780 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:19:58.780 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0256231 s, 40.9 MB/s 2021-02-10T11:19:58.781 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:19:58.781 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:19:58.781 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:19:59.117 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:19:59.117 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:19:59.117 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344460/29d95c22-88b1-40f2-8b55-00181dd92fb2/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:19:59.492 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:19:59.492 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:19:59.492 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0235927 s, 44.4 MB/s 2021-02-10T11:19:59.493 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:19:59.493 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:19:59.494 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:19:59.830 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:19:59.831 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:19:59.831 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344460/29d95c22-88b1-40f2-8b55-00181dd92fb2/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:20:00.207 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:20:00.208 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:20:00.208 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0251483 s, 41.7 MB/s 2021-02-10T11:20:00.209 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:20:00.209 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:20:00.209 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:20:00.444 DEBUG:teuthology.orchestra.run.gibba002:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:20:00.447 DEBUG:teuthology.orchestra.run.gibba013:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:20:00.546 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:20:00.546 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:20:00.546 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344460/29d95c22-88b1-40f2-8b55-00181dd92fb2/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:20:00.937 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:20:00.937 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:20:00.937 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0228839 s, 45.8 MB/s 2021-02-10T11:20:00.938 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:20:00.938 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:20:00.938 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:20:01.274 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:20:01.274 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:20:01.275 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344460/29d95c22-88b1-40f2-8b55-00181dd92fb2/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:20:01.651 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:20:01.651 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:20:01.651 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0235048 s, 44.6 MB/s 2021-02-10T11:20:01.652 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:20:01.652 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:20:01.652 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:20:01.914 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:20:01.914 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:20:01.914 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344460/29d95c22-88b1-40f2-8b55-00181dd92fb2/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:20:02.290 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:20:02.290 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:20:02.290 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0218909 s, 47.9 MB/s 2021-02-10T11:20:02.291 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:20:02.291 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:20:02.291 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:20:02.626 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:20:02.627 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:20:02.627 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344460/29d95c22-88b1-40f2-8b55-00181dd92fb2/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:20:03.006 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:20:03.006 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:20:03.006 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0243928 s, 43.0 MB/s 2021-02-10T11:20:03.007 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:20:03.007 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:20:03.007 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:20:03.345 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:20:03.345 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:20:03.345 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344460/29d95c22-88b1-40f2-8b55-00181dd92fb2/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:20:03.720 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:20:03.720 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:20:03.720 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0269904 s, 38.8 MB/s 2021-02-10T11:20:03.721 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:20:03.721 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:20:03.721 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:20:04.057 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:20:04.057 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:20:04.058 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344460/29d95c22-88b1-40f2-8b55-00181dd92fb2/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:20:04.433 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:20:04.433 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:20:04.433 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0232423 s, 45.1 MB/s 2021-02-10T11:20:04.434 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:20:04.434 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:20:04.434 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:20:04.770 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:20:04.770 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:20:04.771 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344460/29d95c22-88b1-40f2-8b55-00181dd92fb2/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:20:05.146 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:20:05.146 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:20:05.146 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0228764 s, 45.8 MB/s 2021-02-10T11:20:05.147 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000344461 2021-02-10T11:20:05.580 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:05.570 7fb384d7e700 1 Processor -- start 2021-02-10T11:20:08.631 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:05.570 7fb384d7e700 1 -- start start 2021-02-10T11:20:08.631 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:05.570 7fb384d7e700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb3800f05f0 0x7fb3800f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:08.631 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:05.570 7fb384d7e700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3801b0c70 con 0x7fb3801b0470 2021-02-10T11:20:08.632 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:05.570 7fb384d7e700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3800f0b50 con 0x7fb3801b0870 2021-02-10T11:20:08.632 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:05.570 7fb384d7e700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3800e5ea0 con 0x7fb3800f05f0 2021-02-10T11:20:08.632 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:05.571 7fb37eb13700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fb3801b0470 0x7fb3800ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:45850/0 (socket says 172.21.2.102:45850) 2021-02-10T11:20:08.632 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:05.571 7fb37eb13700 1 -- 172.21.2.102:0/3139507194 learned_addr learned my addr 172.21.2.102:0/3139507194 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:20:08.632 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:05.571 7fb37f314700 1 --2- 172.21.2.102:0/3139507194 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb3800f05f0 0x7fb3800f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:08.632 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:05.571 7fb37db11700 1 -- 172.21.2.102:0/3139507194 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3737226131 0 0) 0x7fb370000ce0 con 0x7fb3801b0470 2021-02-10T11:20:08.632 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:05.571 7fb37db11700 1 -- 172.21.2.102:0/3139507194 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb368002fe0 con 0x7fb3801b0470 2021-02-10T11:20:08.632 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:05.571 7fb37db11700 1 -- 172.21.2.102:0/3139507194 <== mon.1 v1:172.21.2.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2513052019 0 0) 0x7fb374001080 con 0x7fb3801b0870 2021-02-10T11:20:08.632 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:05.571 7fb37db11700 1 -- 172.21.2.102:0/3139507194 --> v1:172.21.2.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb368004370 con 0x7fb3801b0870 2021-02-10T11:20:08.632 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:05.571 7fb37f314700 1 -- 172.21.2.102:0/3139507194 >> v1:172.21.2.113:6789/0 conn(0x7fb3801b0870 legacy=0x7fb3800efe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:08.632 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:05.571 7fb37f314700 1 -- 172.21.2.102:0/3139507194 >> v1:172.21.2.102:6789/0 conn(0x7fb3801b0470 legacy=0x7fb3800ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:08.632 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:05.571 7fb37f314700 1 -- 172.21.2.102:0/3139507194 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3800e52e0 con 0x7fb3800f05f0 2021-02-10T11:20:08.632 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:05.572 7fb37f314700 1 --2- 172.21.2.102:0/3139507194 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb3800f05f0 0x7fb3800f20e0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7fb378006e50 tx=0x7fb378009e70).ready entity=mon.2 client_cookie=9779542edf28c981 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:20:08.633 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:05.572 7fb37db11700 1 -- 172.21.2.102:0/3139507194 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb378006a20 con 0x7fb3800f05f0 2021-02-10T11:20:08.633 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:05.572 7fb37db11700 1 -- 172.21.2.102:0/3139507194 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb378006790 con 0x7fb3800f05f0 2021-02-10T11:20:08.633 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:05.572 7fb37db11700 1 -- 172.21.2.102:0/3139507194 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb378000f90 con 0x7fb3800f05f0 2021-02-10T11:20:08.633 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:05.572 7fb384d7e700 1 -- 172.21.2.102:0/3139507194 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb3800f05f0 msgr2=0x7fb3800f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:08.633 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:05.572 7fb384d7e700 1 --2- 172.21.2.102:0/3139507194 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb3800f05f0 0x7fb3800f20e0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7fb378006e50 tx=0x7fb378009e70).stop 2021-02-10T11:20:08.633 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:05.573 7fb384d7e700 1 -- 172.21.2.102:0/3139507194 shutdown_connections 2021-02-10T11:20:08.633 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:05.573 7fb384d7e700 1 --2- 172.21.2.102:0/3139507194 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb3800f05f0 0x7fb3800f20e0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:08.633 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:05.573 7fb384d7e700 1 -- 172.21.2.102:0/3139507194 >> 172.21.2.102:0/3139507194 conn(0x7fb3800ce3e0 msgr2=0x7fb3800e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:20:08.633 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:05.573 7fb384d7e700 1 -- 172.21.2.102:0/3139507194 shutdown_connections 2021-02-10T11:20:08.633 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:05.573 7fb384d7e700 1 -- 172.21.2.102:0/3139507194 wait complete. 2021-02-10T11:20:08.636 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:05.588 7fb384d7e700 1 Processor -- start 2021-02-10T11:20:08.636 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:05.588 7fb384d7e700 1 -- start start 2021-02-10T11:20:08.636 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:05.588 7fb384d7e700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb380190920 0x7fb3800b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:08.636 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:05.588 7fb384d7e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb3800b8200 0x7fb3800b8600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:08.636 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:05.588 7fb384d7e700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb3800c1d80 0x7fb3800c2180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:08.637 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:05.588 7fb384d7e700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb380190d20 con 0x7fb3800b8200 2021-02-10T11:20:08.637 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:05.588 7fb384d7e700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3800b8c20 con 0x7fb380190920 2021-02-10T11:20:08.637 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:05.588 7fb384d7e700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3800c27a0 con 0x7fb3800c1d80 2021-02-10T11:20:08.637 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:05.588 7fb37e312700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb3800b8200 0x7fb3800b8600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:08.637 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:05.588 7fb37e312700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb3800b8200 0x7fb3800b8600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:53632/0 (socket says 172.21.2.102:53632) 2021-02-10T11:20:08.637 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:05.588 7fb37e312700 1 -- 172.21.2.102:0/644119407 learned_addr learned my addr 172.21.2.102:0/644119407 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:20:08.638 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:05.589 7fb37e312700 1 -- 172.21.2.102:0/644119407 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb3800c1d80 msgr2=0x7fb3800c2180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:08.638 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:05.589 7fb37e312700 1 --2- 172.21.2.102:0/644119407 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb3800c1d80 0x7fb3800c2180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:20:08.638 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:05.589 7fb37e312700 1 -- 172.21.2.102:0/644119407 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb380190920 msgr2=0x7fb3800b7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:08.638 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:05.589 7fb37e312700 1 --2- 172.21.2.102:0/644119407 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb380190920 0x7fb3800b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:20:08.638 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:05.589 7fb37e312700 1 -- 172.21.2.102:0/644119407 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3800c29c0 con 0x7fb3800b8200 2021-02-10T11:20:08.638 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:05.589 7fb37e312700 1 --2- 172.21.2.102:0/644119407 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb3800b8200 0x7fb3800b8600 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 rx=0x7fb374006f00 tx=0x7fb37400b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:20:08.638 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:05.589 7fb36f7fe700 1 -- 172.21.2.102:0/644119407 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb37400bd30 con 0x7fb3800b8200 2021-02-10T11:20:08.638 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:05.589 7fb384d7e700 1 -- 172.21.2.102:0/644119407 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3800c29c0 con 0x7fb3800b8200 2021-02-10T11:20:08.638 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:05.589 7fb36f7fe700 1 -- 172.21.2.102:0/644119407 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb374000e20 con 0x7fb3800b8200 2021-02-10T11:20:08.638 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:05.589 7fb36f7fe700 1 -- 172.21.2.102:0/644119407 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb374000bc0 con 0x7fb3800b8200 2021-02-10T11:20:08.639 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:05.589 7fb384d7e700 1 -- 172.21.2.102:0/644119407 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3800c29c0 con 0x7fb3800b8200 2021-02-10T11:20:08.639 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:05.590 7fb36f7fe700 1 -- 172.21.2.102:0/644119407 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fb374006030 con 0x7fb3800b8200 2021-02-10T11:20:08.639 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:05.590 7fb36f7fe700 1 --2- 172.21.2.102:0/644119407 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb364018730 0x7fb36401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:08.639 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:05.590 7fb36f7fe700 1 -- 172.21.2.102:0/644119407 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fb3740327d0 con 0x7fb3800b8200 2021-02-10T11:20:08.639 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:05.590 7fb37eb13700 1 --2- 172.21.2.102:0/644119407 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb364018730 0x7fb36401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:08.639 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:05.590 7fb37eb13700 1 --2- 172.21.2.102:0/644119407 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb364018730 0x7fb36401ac10 crc :-1 s=READY pgs=379 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:20:08.639 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:05.620 7fb384d7e700 1 -- 172.21.2.102:0/644119407 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb380105ae0 con 0x7fb3800b8200 2021-02-10T11:20:08.639 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:05.623 7fb36f7fe700 1 -- 172.21.2.102:0/644119407 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fb374000bc0 con 0x7fb3800b8200 2021-02-10T11:20:08.639 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:05.747 7fb384d7e700 1 -- 172.21.2.102:0/644119407 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344461", "target": ["mgr", ""]}) v1 -- 0x7fb380105ae0 con 0x7fb364018730 2021-02-10T11:20:08.639 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:05.776 7fb36f7fe700 1 -- 172.21.2.102:0/644119407 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fb370000ce0 con 0x7fb364018730 2021-02-10T11:20:08.639 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:05.780 7fb384d7e700 1 -- 172.21.2.102:0/644119407 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb364018730 msgr2=0x7fb36401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:08.639 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:05.780 7fb384d7e700 1 --2- 172.21.2.102:0/644119407 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb364018730 0x7fb36401ac10 crc :-1 s=READY pgs=379 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:08.639 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:05.780 7fb384d7e700 1 -- 172.21.2.102:0/644119407 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb3800b8200 msgr2=0x7fb3800b8600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:08.640 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:05.780 7fb384d7e700 1 --2- 172.21.2.102:0/644119407 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb3800b8200 0x7fb3800b8600 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 rx=0x7fb374006f00 tx=0x7fb37400b940).stop 2021-02-10T11:20:08.640 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:05.780 7fb384d7e700 1 -- 172.21.2.102:0/644119407 shutdown_connections 2021-02-10T11:20:08.640 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:05.780 7fb384d7e700 1 --2- 172.21.2.102:0/644119407 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb3800c1d80 0x7fb3800c2180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:20:08.640 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:05.780 7fb384d7e700 1 --2- 172.21.2.102:0/644119407 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb3800b8200 0x7fb3800b8600 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:08.640 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:05.780 7fb384d7e700 1 --2- 172.21.2.102:0/644119407 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb364018730 0x7fb36401ac10 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:08.640 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:05.780 7fb384d7e700 1 --2- 172.21.2.102:0/644119407 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb380190920 0x7fb3800b7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:20:08.640 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:05.780 7fb384d7e700 1 -- 172.21.2.102:0/644119407 >> 172.21.2.102:0/644119407 conn(0x7fb380094e10 msgr2=0x7fb3800a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:20:08.640 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:05.780 7fb384d7e700 1 -- 172.21.2.102:0/644119407 shutdown_connections 2021-02-10T11:20:08.641 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:05.780 7fb384d7e700 1 -- 172.21.2.102:0/644119407 wait complete. 2021-02-10T11:20:08.641 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000344461 2021-02-10T11:20:09.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:09.066 7f6d5780e700 1 Processor -- start 2021-02-10T11:20:09.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:09.066 7f6d5780e700 1 -- start start 2021-02-10T11:20:09.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:09.067 7f6d5780e700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f6d501b0470 0x7f6d501b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:09.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:09.067 7f6d5780e700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f6d500ef610 0x7f6d500efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:09.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:09.067 7f6d5780e700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d500f0b50 con 0x7f6d500f0030 2021-02-10T11:20:09.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:09.067 7f6d5780e700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d500e52e0 con 0x7f6d501b0470 2021-02-10T11:20:09.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:09.067 7f6d5780e700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d500e5490 con 0x7f6d500ef610 2021-02-10T11:20:09.077 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:09.067 7f6d5562a700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f6d501b0470 0x7f6d501b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:09.077 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:09.067 7f6d5562a700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f6d501b0470 0x7f6d501b0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:53640/0 (socket says 172.21.2.102:53640) 2021-02-10T11:20:09.077 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:09.067 7f6d5562a700 1 -- 172.21.2.102:0/2357692413 learned_addr learned my addr 172.21.2.102:0/2357692413 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:20:09.077 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:09.067 7f6d55e2b700 1 --1- 172.21.2.102:0/2357692413 >> v1:172.21.2.102:6789/0 conn(0x7f6d500f0030 0x7f6d500f10d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:45864/0 (socket says 172.21.2.102:45864) 2021-02-10T11:20:09.077 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:09.067 7f6d47fff700 1 -- 172.21.2.102:0/2357692413 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2587171685 0 0) 0x7f6d48001480 con 0x7f6d500f0030 2021-02-10T11:20:09.077 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:09.067 7f6d5562a700 1 -- 172.21.2.102:0/2357692413 >> v2:172.21.2.113:3300/0 conn(0x7f6d500ef610 msgr2=0x7f6d500efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:20:09.077 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:09.067 7f6d5562a700 1 --2- 172.21.2.102:0/2357692413 >> v2:172.21.2.113:3300/0 conn(0x7f6d500ef610 0x7f6d500efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:20:09.077 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:09.067 7f6d5562a700 1 -- 172.21.2.102:0/2357692413 >> v1:172.21.2.102:6789/0 conn(0x7f6d500f0030 legacy=0x7f6d500f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:09.077 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:09.067 7f6d5562a700 1 -- 172.21.2.102:0/2357692413 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d500f0430 con 0x7f6d501b0470 2021-02-10T11:20:09.077 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:09.068 7f6d5562a700 1 --2- 172.21.2.102:0/2357692413 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6d501b0470 0x7f6d501b0870 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 rx=0x7f6d4c006700 tx=0x7f6d4c009b80).ready entity=mon.0 client_cookie=1b86515d3076bb6a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:20:09.077 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:09.068 7f6d47fff700 1 -- 172.21.2.102:0/2357692413 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6d4c00c0c0 con 0x7f6d501b0470 2021-02-10T11:20:09.077 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:09.068 7f6d47fff700 1 -- 172.21.2.102:0/2357692413 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d4c00c440 con 0x7f6d501b0470 2021-02-10T11:20:09.078 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:09.068 7f6d47fff700 1 -- 172.21.2.102:0/2357692413 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6d4c00c960 con 0x7f6d501b0470 2021-02-10T11:20:09.078 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:09.068 7f6d5780e700 1 -- 172.21.2.102:0/2357692413 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6d501b0470 msgr2=0x7f6d501b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:09.078 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:09.068 7f6d5780e700 1 --2- 172.21.2.102:0/2357692413 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6d501b0470 0x7f6d501b0870 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 rx=0x7f6d4c006700 tx=0x7f6d4c009b80).stop 2021-02-10T11:20:09.078 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:09.068 7f6d5780e700 1 -- 172.21.2.102:0/2357692413 shutdown_connections 2021-02-10T11:20:09.078 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:09.068 7f6d5780e700 1 --2- 172.21.2.102:0/2357692413 >> v2:172.21.2.113:3300/0 conn(0x7f6d500ef610 0x7f6d500efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:20:09.078 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:09.068 7f6d5780e700 1 --2- 172.21.2.102:0/2357692413 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6d501b0470 0x7f6d501b0870 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:09.078 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:09.068 7f6d5780e700 1 -- 172.21.2.102:0/2357692413 >> 172.21.2.102:0/2357692413 conn(0x7f6d500ce3e0 msgr2=0x7f6d500e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:20:09.078 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:09.069 7f6d5780e700 1 -- 172.21.2.102:0/2357692413 shutdown_connections 2021-02-10T11:20:09.078 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:09.069 7f6d5780e700 1 -- 172.21.2.102:0/2357692413 wait complete. 2021-02-10T11:20:09.080 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:09.083 7f6d5780e700 1 Processor -- start 2021-02-10T11:20:09.080 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:09.083 7f6d5780e700 1 -- start start 2021-02-10T11:20:09.080 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:09.083 7f6d5780e700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6d500b20b0 0x7f6d500ac970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:09.080 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:09.083 7f6d5780e700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6d500b24b0 0x7f6d50191dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:09.080 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:09.083 7f6d5780e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6d500acf90 0x7f6d501923e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:09.080 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:09.083 7f6d5780e700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d500b1b30 con 0x7f6d500acf90 2021-02-10T11:20:09.080 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:09.083 7f6d5780e700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d501a04e0 con 0x7f6d500b24b0 2021-02-10T11:20:09.080 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:09.083 7f6d5780e700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d501a06c0 con 0x7f6d500b20b0 2021-02-10T11:20:09.080 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:09.084 7f6d55e2b700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6d500acf90 0x7f6d501923e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:09.080 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:09.084 7f6d55e2b700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6d500acf90 0x7f6d501923e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:53648/0 (socket says 172.21.2.102:53648) 2021-02-10T11:20:09.080 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:09.084 7f6d55e2b700 1 -- 172.21.2.102:0/1681857314 learned_addr learned my addr 172.21.2.102:0/1681857314 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:20:09.080 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:09.084 7f6d55e2b700 1 -- 172.21.2.102:0/1681857314 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6d500b20b0 msgr2=0x7f6d500ac970 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:20:09.081 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:09.084 7f6d55e2b700 1 --2- 172.21.2.102:0/1681857314 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6d500b20b0 0x7f6d500ac970 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:20:09.081 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:09.084 7f6d55e2b700 1 -- 172.21.2.102:0/1681857314 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6d500b24b0 msgr2=0x7f6d50191dc0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:20:09.081 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:09.084 7f6d55e2b700 1 --2- 172.21.2.102:0/1681857314 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6d500b24b0 0x7f6d50191dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:20:09.081 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:09.084 7f6d55e2b700 1 -- 172.21.2.102:0/1681857314 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d50192be0 con 0x7f6d500acf90 2021-02-10T11:20:09.081 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:09.084 7f6d55e2b700 1 --2- 172.21.2.102:0/1681857314 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6d500acf90 0x7f6d501923e0 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 rx=0x7f6d4800a8f0 tx=0x7f6d4800ac90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:20:09.081 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:09.084 7f6d45ffb700 1 -- 172.21.2.102:0/1681857314 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6d480070d0 con 0x7f6d500acf90 2021-02-10T11:20:09.081 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:09.084 7f6d45ffb700 1 -- 172.21.2.102:0/1681857314 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d48007450 con 0x7f6d500acf90 2021-02-10T11:20:09.081 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:09.084 7f6d45ffb700 1 -- 172.21.2.102:0/1681857314 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6d480049a0 con 0x7f6d500acf90 2021-02-10T11:20:09.081 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:09.084 7f6d5780e700 1 -- 172.21.2.102:0/1681857314 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d50192be0 con 0x7f6d500acf90 2021-02-10T11:20:09.081 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:09.085 7f6d5780e700 1 -- 172.21.2.102:0/1681857314 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d50192be0 con 0x7f6d500acf90 2021-02-10T11:20:09.081 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:09.085 7f6d45ffb700 1 -- 172.21.2.102:0/1681857314 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f6d4800add0 con 0x7f6d500acf90 2021-02-10T11:20:09.081 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:09.085 7f6d45ffb700 1 --2- 172.21.2.102:0/1681857314 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6d300186c0 0x7f6d3001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:09.081 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:09.085 7f6d45ffb700 1 -- 172.21.2.102:0/1681857314 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f6d48032750 con 0x7f6d500acf90 2021-02-10T11:20:09.081 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:09.085 7f6d5562a700 1 --2- 172.21.2.102:0/1681857314 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6d300186c0 0x7f6d3001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:09.082 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:09.085 7f6d5562a700 1 --2- 172.21.2.102:0/1681857314 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6d300186c0 0x7f6d3001abc0 crc :-1 s=READY pgs=382 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:20:09.110 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:09.117 7f6d5780e700 1 -- 172.21.2.102:0/1681857314 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d50105af0 con 0x7f6d500acf90 2021-02-10T11:20:09.112 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:09.119 7f6d45ffb700 1 -- 172.21.2.102:0/1681857314 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f6d48007d50 con 0x7f6d500acf90 2021-02-10T11:20:09.241 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:09.248 7f6d5780e700 1 -- 172.21.2.102:0/1681857314 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344461", "target": ["mgr", ""]}) v1 -- 0x7f6d50105af0 con 0x7f6d300186c0 2021-02-10T11:20:09.245 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:09.251 7f6d45ffb700 1 -- 172.21.2.102:0/1681857314 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f6d4c009100 con 0x7f6d300186c0 2021-02-10T11:20:09.249 INFO:teuthology.orchestra.run.gibba002.stdout:/volumes/_nogroup/subvolume_0000000000344461/59e934ba-0173-4f6b-affa-89c7ab27733a 2021-02-10T11:20:09.250 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:09.256 7f6d5780e700 1 -- 172.21.2.102:0/1681857314 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6d300186c0 msgr2=0x7f6d3001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:09.250 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:09.256 7f6d5780e700 1 --2- 172.21.2.102:0/1681857314 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6d300186c0 0x7f6d3001abc0 crc :-1 s=READY pgs=382 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:09.250 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:09.256 7f6d5780e700 1 -- 172.21.2.102:0/1681857314 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6d500acf90 msgr2=0x7f6d501923e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:09.250 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:09.256 7f6d5780e700 1 --2- 172.21.2.102:0/1681857314 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6d500acf90 0x7f6d501923e0 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 rx=0x7f6d4800a8f0 tx=0x7f6d4800ac90).stop 2021-02-10T11:20:09.251 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:09.257 7f6d5780e700 1 -- 172.21.2.102:0/1681857314 shutdown_connections 2021-02-10T11:20:09.251 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:09.257 7f6d5780e700 1 --2- 172.21.2.102:0/1681857314 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6d500acf90 0x7f6d501923e0 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:09.251 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:09.257 7f6d5780e700 1 --2- 172.21.2.102:0/1681857314 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6d300186c0 0x7f6d3001abc0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:09.251 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:09.257 7f6d5780e700 1 --2- 172.21.2.102:0/1681857314 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6d500b24b0 0x7f6d50191dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:20:09.251 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:09.257 7f6d5780e700 1 --2- 172.21.2.102:0/1681857314 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6d500b20b0 0x7f6d500ac970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:20:09.251 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:09.257 7f6d5780e700 1 -- 172.21.2.102:0/1681857314 >> 172.21.2.102:0/1681857314 conn(0x7f6d500bf4e0 msgr2=0x7f6d500be3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:20:09.251 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:09.257 7f6d5780e700 1 -- 172.21.2.102:0/1681857314 shutdown_connections 2021-02-10T11:20:09.251 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:09.257 7f6d5780e700 1 -- 172.21.2.102:0/1681857314 wait complete. 2021-02-10T11:20:09.259 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000344461 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000344461/59e934ba-0173-4f6b-affa-89c7ab27733a 2021-02-10T11:20:09.259 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:20:09.259 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:20:09.259 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:20:09.596 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:20:09.596 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:20:09.597 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344461/59e934ba-0173-4f6b-affa-89c7ab27733a/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:20:09.977 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:20:09.977 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:20:09.977 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0244731 s, 42.8 MB/s 2021-02-10T11:20:09.978 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:20:09.979 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:20:09.979 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:20:10.313 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:20:10.314 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:20:10.314 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344461/59e934ba-0173-4f6b-affa-89c7ab27733a/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:20:10.690 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:20:10.690 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:20:10.690 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0242341 s, 43.3 MB/s 2021-02-10T11:20:10.691 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:20:10.691 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:20:10.691 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:20:11.026 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:20:11.026 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:20:11.027 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344461/59e934ba-0173-4f6b-affa-89c7ab27733a/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:20:11.402 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:20:11.403 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:20:11.403 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0243716 s, 43.0 MB/s 2021-02-10T11:20:11.403 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:20:11.404 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:20:11.404 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:20:11.738 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:20:11.738 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:20:11.738 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344461/59e934ba-0173-4f6b-affa-89c7ab27733a/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:20:12.123 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:20:12.123 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:20:12.123 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0250673 s, 41.8 MB/s 2021-02-10T11:20:12.124 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:20:12.124 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:20:12.124 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:20:12.464 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:20:12.464 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:20:12.465 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344461/59e934ba-0173-4f6b-affa-89c7ab27733a/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:20:12.843 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:20:12.843 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:20:12.843 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0237023 s, 44.2 MB/s 2021-02-10T11:20:12.844 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:20:12.844 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:20:12.844 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:20:13.182 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:20:13.182 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:20:13.183 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344461/59e934ba-0173-4f6b-affa-89c7ab27733a/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:20:13.556 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:20:13.556 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:20:13.556 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0250084 s, 41.9 MB/s 2021-02-10T11:20:13.557 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:20:13.557 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:20:13.557 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:20:13.892 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:20:13.893 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:20:13.893 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344461/59e934ba-0173-4f6b-affa-89c7ab27733a/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:20:14.269 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:20:14.269 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:20:14.269 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.024836 s, 42.2 MB/s 2021-02-10T11:20:14.270 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:20:14.270 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:20:14.270 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:20:14.605 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:20:14.606 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:20:14.606 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344461/59e934ba-0173-4f6b-affa-89c7ab27733a/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:20:14.981 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:20:14.981 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:20:14.982 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0247631 s, 42.3 MB/s 2021-02-10T11:20:14.982 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:20:14.983 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:20:14.983 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:20:15.317 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:20:15.317 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:20:15.317 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344461/59e934ba-0173-4f6b-affa-89c7ab27733a/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:20:15.690 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:20:15.690 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:20:15.690 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0226918 s, 46.2 MB/s 2021-02-10T11:20:15.691 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:20:15.691 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:20:15.691 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:20:16.025 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:20:16.025 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:20:16.026 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344461/59e934ba-0173-4f6b-affa-89c7ab27733a/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:20:16.398 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:20:16.398 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:20:16.398 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0234446 s, 44.7 MB/s 2021-02-10T11:20:16.399 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000344462 2021-02-10T11:20:16.827 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:16.817 7fd041c3d700 1 Processor -- start 2021-02-10T11:20:16.827 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:16.817 7fd041c3d700 1 -- start start 2021-02-10T11:20:16.827 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:16.817 7fd041c3d700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fd03c1b0870 0x7fd03c0f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:16.827 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:16.817 7fd041c3d700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd03c0efe00 0x7fd03c0f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:16.827 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:16.817 7fd041c3d700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd03c1b0c70 con 0x7fd03c1b0470 2021-02-10T11:20:16.827 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:16.817 7fd041c3d700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd03c0f0b50 con 0x7fd03c1b0870 2021-02-10T11:20:16.827 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:16.817 7fd041c3d700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd03c0f0820 con 0x7fd03c0efe00 2021-02-10T11:20:16.827 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:16.818 7fd03affd700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fd03c1b0870 0x7fd03c0f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:16.827 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:16.818 7fd03affd700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fd03c1b0870 0x7fd03c0f20e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:53652/0 (socket says 172.21.2.102:53652) 2021-02-10T11:20:16.828 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:16.818 7fd03affd700 1 -- 172.21.2.102:0/3107716804 learned_addr learned my addr 172.21.2.102:0/3107716804 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:20:16.828 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:16.818 7fd03affd700 1 -- 172.21.2.102:0/3107716804 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd03c0efe00 msgr2=0x7fd03c0f0200 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:20:16.828 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:16.818 7fd03affd700 1 --2- 172.21.2.102:0/3107716804 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd03c0efe00 0x7fd03c0f0200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:20:16.828 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:16.818 7fd03affd700 1 -- 172.21.2.102:0/3107716804 >> v1:172.21.2.113:6789/0 conn(0x7fd03c1b0470 legacy=0x7fd03c0ef610 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:20:16.828 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:16.818 7fd03affd700 1 -- 172.21.2.102:0/3107716804 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd03c0e52e0 con 0x7fd03c1b0870 2021-02-10T11:20:16.828 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:16.818 7fd03affd700 1 --2- 172.21.2.102:0/3107716804 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd03c1b0870 0x7fd03c0f20e0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 rx=0x7fd024006700 tx=0x7fd024009b80).ready entity=mon.0 client_cookie=9bf76852afe90c85 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:20:16.828 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:16.818 7fd03a7fc700 1 -- 172.21.2.102:0/3107716804 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd02400c0c0 con 0x7fd03c1b0870 2021-02-10T11:20:16.828 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:16.818 7fd03a7fc700 1 -- 172.21.2.102:0/3107716804 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd02400c440 con 0x7fd03c1b0870 2021-02-10T11:20:16.828 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:16.818 7fd03a7fc700 1 -- 172.21.2.102:0/3107716804 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd02400c960 con 0x7fd03c1b0870 2021-02-10T11:20:16.828 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:16.818 7fd041c3d700 1 -- 172.21.2.102:0/3107716804 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd03c1b0870 msgr2=0x7fd03c0f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:16.828 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:16.818 7fd041c3d700 1 --2- 172.21.2.102:0/3107716804 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd03c1b0870 0x7fd03c0f20e0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 rx=0x7fd024006700 tx=0x7fd024009b80).stop 2021-02-10T11:20:16.828 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:16.819 7fd041c3d700 1 -- 172.21.2.102:0/3107716804 shutdown_connections 2021-02-10T11:20:16.828 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:16.819 7fd041c3d700 1 --2- 172.21.2.102:0/3107716804 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd03c0efe00 0x7fd03c0f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:20:16.829 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:16.819 7fd041c3d700 1 --2- 172.21.2.102:0/3107716804 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd03c1b0870 0x7fd03c0f20e0 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:16.829 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:16.819 7fd041c3d700 1 -- 172.21.2.102:0/3107716804 >> 172.21.2.102:0/3107716804 conn(0x7fd03c0ce3e0 msgr2=0x7fd03c0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:20:16.829 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:16.819 7fd041c3d700 1 -- 172.21.2.102:0/3107716804 shutdown_connections 2021-02-10T11:20:16.829 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:16.819 7fd041c3d700 1 -- 172.21.2.102:0/3107716804 wait complete. 2021-02-10T11:20:16.830 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:16.834 7fd041c3d700 1 Processor -- start 2021-02-10T11:20:16.830 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:16.834 7fd041c3d700 1 -- start start 2021-02-10T11:20:16.830 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:16.834 7fd041c3d700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd03c0c36a0 0x7fd03c0c2b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:16.830 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:16.834 7fd041c3d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd03c0c3190 0x7fd03c0b1d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:16.830 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:16.834 7fd041c3d700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd03c0b2370 0x7fd03c0b2d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:16.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:16.834 7fd041c3d700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd03c0b27c0 con 0x7fd03c0c3190 2021-02-10T11:20:16.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:16.834 7fd041c3d700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd03c0b3360 con 0x7fd03c0c36a0 2021-02-10T11:20:16.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:16.834 7fd041c3d700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd03c0b3540 con 0x7fd03c0b2370 2021-02-10T11:20:16.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:16.834 7fd03affd700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd03c0c3190 0x7fd03c0b1d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:16.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:16.834 7fd03affd700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd03c0c3190 0x7fd03c0b1d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:53660/0 (socket says 172.21.2.102:53660) 2021-02-10T11:20:16.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:16.834 7fd03affd700 1 -- 172.21.2.102:0/3625170742 learned_addr learned my addr 172.21.2.102:0/3625170742 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:20:16.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:16.835 7fd03affd700 1 -- 172.21.2.102:0/3625170742 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd03c0b2370 msgr2=0x7fd03c0b2d40 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:20:16.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:16.835 7fd03affd700 1 --2- 172.21.2.102:0/3625170742 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd03c0b2370 0x7fd03c0b2d40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:20:16.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:16.835 7fd03affd700 1 -- 172.21.2.102:0/3625170742 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd03c0c36a0 msgr2=0x7fd03c0c2b70 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:20:16.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:16.835 7fd03affd700 1 --2- 172.21.2.102:0/3625170742 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd03c0c36a0 0x7fd03c0c2b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:20:16.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:16.835 7fd03affd700 1 -- 172.21.2.102:0/3625170742 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd03c0b3760 con 0x7fd03c0c3190 2021-02-10T11:20:16.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:16.835 7fd03affd700 1 --2- 172.21.2.102:0/3625170742 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd03c0c3190 0x7fd03c0b1d50 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 rx=0x7fd02400b5e0 tx=0x7fd02400b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:20:16.832 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:16.835 7fd033fff700 1 -- 172.21.2.102:0/3625170742 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd02400bd70 con 0x7fd03c0c3190 2021-02-10T11:20:16.832 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:16.835 7fd033fff700 1 -- 172.21.2.102:0/3625170742 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd024002030 con 0x7fd03c0c3190 2021-02-10T11:20:16.832 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:16.835 7fd033fff700 1 -- 172.21.2.102:0/3625170742 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd024002550 con 0x7fd03c0c3190 2021-02-10T11:20:16.832 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:16.835 7fd041c3d700 1 -- 172.21.2.102:0/3625170742 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd03c0b3760 con 0x7fd03c0c3190 2021-02-10T11:20:16.832 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:16.835 7fd041c3d700 1 -- 172.21.2.102:0/3625170742 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd03c0b3cf0 con 0x7fd03c0c3190 2021-02-10T11:20:16.832 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:16.836 7fd033fff700 1 -- 172.21.2.102:0/3625170742 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd024000c90 con 0x7fd03c0c3190 2021-02-10T11:20:16.832 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:16.836 7fd033fff700 1 --2- 172.21.2.102:0/3625170742 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd0200186c0 0x7fd02001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:16.832 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:16.836 7fd033fff700 1 -- 172.21.2.102:0/3625170742 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fd0240325e0 con 0x7fd03c0c3190 2021-02-10T11:20:16.832 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:16.836 7fd03b7fe700 1 --2- 172.21.2.102:0/3625170742 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd0200186c0 0x7fd02001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:16.832 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:16.836 7fd03b7fe700 1 --2- 172.21.2.102:0/3625170742 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd0200186c0 0x7fd02001abc0 crc :-1 s=READY pgs=383 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:20:16.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:16.867 7fd041c3d700 1 -- 172.21.2.102:0/3625170742 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd03c0edd10 con 0x7fd03c0c3190 2021-02-10T11:20:16.863 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:16.870 7fd033fff700 1 -- 172.21.2.102:0/3625170742 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd0240009b0 con 0x7fd03c0c3190 2021-02-10T11:20:16.991 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:16.997 7fd041c3d700 1 -- 172.21.2.102:0/3625170742 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344462", "target": ["mgr", ""]}) v1 -- 0x7fd03c0b3cf0 con 0x7fd0200186c0 2021-02-10T11:20:17.020 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:17.026 7fd033fff700 1 -- 172.21.2.102:0/3625170742 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fd02c0009b0 con 0x7fd0200186c0 2021-02-10T11:20:17.024 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:17.030 7fd041c3d700 1 -- 172.21.2.102:0/3625170742 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd0200186c0 msgr2=0x7fd02001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:17.024 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:17.030 7fd041c3d700 1 --2- 172.21.2.102:0/3625170742 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd0200186c0 0x7fd02001abc0 crc :-1 s=READY pgs=383 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:17.024 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:17.030 7fd041c3d700 1 -- 172.21.2.102:0/3625170742 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd03c0c3190 msgr2=0x7fd03c0b1d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:17.024 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:17.030 7fd041c3d700 1 --2- 172.21.2.102:0/3625170742 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd03c0c3190 0x7fd03c0b1d50 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 rx=0x7fd02400b5e0 tx=0x7fd02400b980).stop 2021-02-10T11:20:17.024 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:17.030 7fd041c3d700 1 -- 172.21.2.102:0/3625170742 shutdown_connections 2021-02-10T11:20:17.024 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:17.030 7fd041c3d700 1 --2- 172.21.2.102:0/3625170742 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd03c0b2370 0x7fd03c0b2d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:20:17.024 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:17.030 7fd041c3d700 1 --2- 172.21.2.102:0/3625170742 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd03c0c3190 0x7fd03c0b1d50 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:17.024 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:17.030 7fd041c3d700 1 --2- 172.21.2.102:0/3625170742 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd0200186c0 0x7fd02001abc0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:17.025 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:17.030 7fd041c3d700 1 --2- 172.21.2.102:0/3625170742 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd03c0c36a0 0x7fd03c0c2b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:20:17.025 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:17.030 7fd041c3d700 1 -- 172.21.2.102:0/3625170742 >> 172.21.2.102:0/3625170742 conn(0x7fd03c09f630 msgr2=0x7fd03c0a0130 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:20:17.025 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:17.030 7fd041c3d700 1 -- 172.21.2.102:0/3625170742 shutdown_connections 2021-02-10T11:20:17.025 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:17.030 7fd041c3d700 1 -- 172.21.2.102:0/3625170742 wait complete. 2021-02-10T11:20:17.032 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000344462 2021-02-10T11:20:17.462 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:17.451 7f921b694700 1 Processor -- start 2021-02-10T11:20:17.462 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:17.451 7f921b694700 1 -- start start 2021-02-10T11:20:17.462 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:17.452 7f921b694700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f92141b0870 0x7f92140f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:17.462 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:17.452 7f921b694700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f92140efe00 0x7f92140f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:17.462 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:17.452 7f921b694700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92141b0c70 con 0x7f92141b0470 2021-02-10T11:20:17.462 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:17.452 7f921b694700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92140e5ea0 con 0x7f92141b0870 2021-02-10T11:20:17.462 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:17.452 7f921b694700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92140f0b50 con 0x7f92140efe00 2021-02-10T11:20:17.462 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:17.452 7f9218caf700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f92141b0870 0x7f92140f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:17.462 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:17.452 7f92194b0700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f92141b0470 0x7f92140ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:45888/0 (socket says 172.21.2.102:45888) 2021-02-10T11:20:17.462 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:17.452 7f92194b0700 1 -- 172.21.2.102:0/2127472159 learned_addr learned my addr 172.21.2.102:0/2127472159 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:20:17.463 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:17.452 7f9218caf700 1 -- 172.21.2.102:0/2127472159 >> v2:172.21.2.113:3300/0 conn(0x7f92140efe00 msgr2=0x7f92140f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:20:17.463 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:17.452 7f9218caf700 1 --2- 172.21.2.102:0/2127472159 >> v2:172.21.2.113:3300/0 conn(0x7f92140efe00 0x7f92140f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:20:17.463 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:17.452 7f9218caf700 1 -- 172.21.2.102:0/2127472159 >> v1:172.21.2.102:6789/0 conn(0x7f92141b0470 legacy=0x7f92140ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:17.463 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:17.452 7f9218caf700 1 -- 172.21.2.102:0/2127472159 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92140f0820 con 0x7f92141b0870 2021-02-10T11:20:17.463 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:17.453 7f9218caf700 1 --2- 172.21.2.102:0/2127472159 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f92141b0870 0x7f92140f20e0 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 rx=0x7f9204006650 tx=0x7f9204009a80).ready entity=mon.0 client_cookie=6ef50f44f9a87301 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:20:17.463 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:17.453 7f920bfff700 1 -- 172.21.2.102:0/2127472159 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f92040009b0 con 0x7f92141b0870 2021-02-10T11:20:17.463 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:17.453 7f920bfff700 1 -- 172.21.2.102:0/2127472159 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f920400c0c0 con 0x7f92141b0870 2021-02-10T11:20:17.463 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:17.453 7f920bfff700 1 -- 172.21.2.102:0/2127472159 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f920400c5e0 con 0x7f92141b0870 2021-02-10T11:20:17.463 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:17.453 7f921b694700 1 -- 172.21.2.102:0/2127472159 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f92141b0870 msgr2=0x7f92140f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:17.463 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:17.453 7f921b694700 1 --2- 172.21.2.102:0/2127472159 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f92141b0870 0x7f92140f20e0 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 rx=0x7f9204006650 tx=0x7f9204009a80).stop 2021-02-10T11:20:17.463 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:17.454 7f921b694700 1 -- 172.21.2.102:0/2127472159 shutdown_connections 2021-02-10T11:20:17.463 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:17.454 7f921b694700 1 --2- 172.21.2.102:0/2127472159 >> v2:172.21.2.113:3300/0 conn(0x7f92140efe00 0x7f92140f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:20:17.463 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:17.454 7f921b694700 1 --2- 172.21.2.102:0/2127472159 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f92141b0870 0x7f92140f20e0 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:17.464 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:17.454 7f921b694700 1 -- 172.21.2.102:0/2127472159 >> 172.21.2.102:0/2127472159 conn(0x7f92140ce3e0 msgr2=0x7f92140e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:20:17.464 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:17.454 7f921b694700 1 -- 172.21.2.102:0/2127472159 shutdown_connections 2021-02-10T11:20:17.464 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:17.454 7f921b694700 1 -- 172.21.2.102:0/2127472159 wait complete. 2021-02-10T11:20:17.464 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:17.469 7f921b694700 1 Processor -- start 2021-02-10T11:20:17.464 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:17.469 7f921b694700 1 -- start start 2021-02-10T11:20:17.464 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:17.469 7f921b694700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f92140c36a0 0x7f92140c2b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:17.465 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:17.469 7f921b694700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f92140c3190 0x7f92140b1d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:17.465 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:17.469 7f921b694700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f92140b2370 0x7f92140b2d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:17.465 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:17.469 7f921b694700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92140b27c0 con 0x7f92140b2370 2021-02-10T11:20:17.465 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:17.469 7f921b694700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92140b3360 con 0x7f92140c3190 2021-02-10T11:20:17.465 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:17.469 7f921b694700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92140b3540 con 0x7f92140c36a0 2021-02-10T11:20:17.466 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:17.469 7f9219cb1700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f92140b2370 0x7f92140b2d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:17.466 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:17.469 7f9219cb1700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f92140b2370 0x7f92140b2d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:53672/0 (socket says 172.21.2.102:53672) 2021-02-10T11:20:17.466 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:17.469 7f9219cb1700 1 -- 172.21.2.102:0/1874582943 learned_addr learned my addr 172.21.2.102:0/1874582943 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:20:17.466 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:17.470 7f9219cb1700 1 -- 172.21.2.102:0/1874582943 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f92140c36a0 msgr2=0x7f92140c2b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:17.466 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:17.470 7f9219cb1700 1 --2- 172.21.2.102:0/1874582943 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f92140c36a0 0x7f92140c2b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:20:17.466 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:17.470 7f9219cb1700 1 -- 172.21.2.102:0/1874582943 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f92140c3190 msgr2=0x7f92140b1d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:17.466 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:17.470 7f9219cb1700 1 --2- 172.21.2.102:0/1874582943 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f92140c3190 0x7f92140b1d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:20:17.467 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:17.470 7f9219cb1700 1 -- 172.21.2.102:0/1874582943 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92140b3760 con 0x7f92140b2370 2021-02-10T11:20:17.467 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:17.470 7f9219cb1700 1 --2- 172.21.2.102:0/1874582943 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f92140b2370 0x7f92140b2d40 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 rx=0x7f920c009980 tx=0x7f920c00bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:20:17.467 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:17.470 7f9209ffb700 1 -- 172.21.2.102:0/1874582943 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f920c005870 con 0x7f92140b2370 2021-02-10T11:20:17.467 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:17.470 7f9209ffb700 1 -- 172.21.2.102:0/1874582943 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f920c005b30 con 0x7f92140b2370 2021-02-10T11:20:17.467 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:17.470 7f9209ffb700 1 -- 172.21.2.102:0/1874582943 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f920c006030 con 0x7f92140b2370 2021-02-10T11:20:17.467 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:17.470 7f921b694700 1 -- 172.21.2.102:0/1874582943 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92140b3760 con 0x7f92140b2370 2021-02-10T11:20:17.467 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:17.470 7f921b694700 1 -- 172.21.2.102:0/1874582943 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92140b3cf0 con 0x7f92140b2370 2021-02-10T11:20:17.467 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:17.471 7f9209ffb700 1 -- 172.21.2.102:0/1874582943 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f920c005d10 con 0x7f92140b2370 2021-02-10T11:20:17.467 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:17.471 7f9209ffb700 1 --2- 172.21.2.102:0/1874582943 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f91f40186c0 0x7f91f401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:17.467 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:17.471 7f9209ffb700 1 -- 172.21.2.102:0/1874582943 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f920c032940 con 0x7f92140b2370 2021-02-10T11:20:17.467 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:17.471 7f92194b0700 1 --2- 172.21.2.102:0/1874582943 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f91f40186c0 0x7f91f401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:17.467 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:17.472 7f92194b0700 1 --2- 172.21.2.102:0/1874582943 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f91f40186c0 0x7f91f401abc0 crc :-1 s=READY pgs=384 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:20:17.496 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:17.502 7f921b694700 1 -- 172.21.2.102:0/1874582943 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92140edd10 con 0x7f92140b2370 2021-02-10T11:20:17.498 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:17.505 7f9209ffb700 1 -- 172.21.2.102:0/1874582943 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f920c025f10 con 0x7f92140b2370 2021-02-10T11:20:17.626 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:17.632 7f921b694700 1 -- 172.21.2.102:0/1874582943 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344462", "target": ["mgr", ""]}) v1 -- 0x7f92140b3cf0 con 0x7f91f40186c0 2021-02-10T11:20:17.629 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:17.635 7f9209ffb700 1 -- 172.21.2.102:0/1874582943 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f9210000e90 con 0x7f91f40186c0 2021-02-10T11:20:17.633 INFO:teuthology.orchestra.run.gibba002.stdout:/volumes/_nogroup/subvolume_0000000000344462/8fe576d8-518c-49aa-b0d4-f631e6cf09b6 2021-02-10T11:20:17.634 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:17.640 7f921b694700 1 -- 172.21.2.102:0/1874582943 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f91f40186c0 msgr2=0x7f91f401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:17.634 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:17.640 7f921b694700 1 --2- 172.21.2.102:0/1874582943 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f91f40186c0 0x7f91f401abc0 crc :-1 s=READY pgs=384 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:17.634 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:17.640 7f921b694700 1 -- 172.21.2.102:0/1874582943 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f92140b2370 msgr2=0x7f92140b2d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:17.634 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:17.640 7f921b694700 1 --2- 172.21.2.102:0/1874582943 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f92140b2370 0x7f92140b2d40 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 rx=0x7f920c009980 tx=0x7f920c00bcd0).stop 2021-02-10T11:20:17.635 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:17.641 7f921b694700 1 -- 172.21.2.102:0/1874582943 shutdown_connections 2021-02-10T11:20:17.635 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:17.641 7f921b694700 1 --2- 172.21.2.102:0/1874582943 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f92140b2370 0x7f92140b2d40 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:17.635 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:17.641 7f921b694700 1 --2- 172.21.2.102:0/1874582943 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f91f40186c0 0x7f91f401abc0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:17.635 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:17.641 7f921b694700 1 --2- 172.21.2.102:0/1874582943 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f92140c3190 0x7f92140b1d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:20:17.635 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:17.641 7f921b694700 1 --2- 172.21.2.102:0/1874582943 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f92140c36a0 0x7f92140c2b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:20:17.635 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:17.641 7f921b694700 1 -- 172.21.2.102:0/1874582943 >> 172.21.2.102:0/1874582943 conn(0x7f921409f630 msgr2=0x7f92140a0130 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:20:17.635 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:17.641 7f921b694700 1 -- 172.21.2.102:0/1874582943 shutdown_connections 2021-02-10T11:20:17.635 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:17.641 7f921b694700 1 -- 172.21.2.102:0/1874582943 wait complete. 2021-02-10T11:20:17.643 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000344462 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000344462/8fe576d8-518c-49aa-b0d4-f631e6cf09b6 2021-02-10T11:20:17.643 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:20:17.643 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:20:17.643 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:20:17.977 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:20:17.977 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:20:17.978 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344462/8fe576d8-518c-49aa-b0d4-f631e6cf09b6/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:20:18.361 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:20:18.362 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:20:18.362 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0254886 s, 41.1 MB/s 2021-02-10T11:20:18.363 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:20:18.363 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:20:18.363 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:20:18.698 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:20:18.698 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:20:18.698 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344462/8fe576d8-518c-49aa-b0d4-f631e6cf09b6/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:20:19.070 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:20:19.071 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:20:19.071 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0260768 s, 40.2 MB/s 2021-02-10T11:20:19.071 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:20:19.072 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:20:19.072 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:20:19.407 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:20:19.407 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:20:19.408 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344462/8fe576d8-518c-49aa-b0d4-f631e6cf09b6/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:20:19.782 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:20:19.782 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:20:19.782 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.022959 s, 45.7 MB/s 2021-02-10T11:20:19.783 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:20:19.783 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:20:19.783 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:20:20.117 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:20:20.117 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:20:20.118 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344462/8fe576d8-518c-49aa-b0d4-f631e6cf09b6/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:20:20.492 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:20:20.492 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:20:20.492 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0229694 s, 45.7 MB/s 2021-02-10T11:20:20.493 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:20:20.493 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:20:20.493 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:20:20.829 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:20:20.829 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:20:20.829 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344462/8fe576d8-518c-49aa-b0d4-f631e6cf09b6/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:20:21.204 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:20:21.205 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:20:21.205 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.024002 s, 43.7 MB/s 2021-02-10T11:20:21.205 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:20:21.206 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:20:21.206 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:20:21.540 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:20:21.541 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:20:21.541 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344462/8fe576d8-518c-49aa-b0d4-f631e6cf09b6/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:20:21.916 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:20:21.916 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:20:21.916 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0230854 s, 45.4 MB/s 2021-02-10T11:20:21.917 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:20:21.917 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:20:21.917 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:20:22.252 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:20:22.253 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:20:22.253 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344462/8fe576d8-518c-49aa-b0d4-f631e6cf09b6/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:20:22.626 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:20:22.626 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:20:22.627 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0229901 s, 45.6 MB/s 2021-02-10T11:20:22.627 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:20:22.628 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:20:22.628 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:20:22.962 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:20:22.962 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:20:22.962 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344462/8fe576d8-518c-49aa-b0d4-f631e6cf09b6/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:20:23.336 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:20:23.336 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:20:23.336 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0235005 s, 44.6 MB/s 2021-02-10T11:20:23.337 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:20:23.337 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:20:23.338 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:20:23.674 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:20:23.674 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:20:23.674 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344462/8fe576d8-518c-49aa-b0d4-f631e6cf09b6/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:20:24.046 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:20:24.046 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:20:24.046 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.023971 s, 43.7 MB/s 2021-02-10T11:20:24.047 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:20:24.047 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:20:24.047 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:20:24.383 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:20:24.383 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:20:24.384 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344462/8fe576d8-518c-49aa-b0d4-f631e6cf09b6/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:20:24.759 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:20:24.759 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:20:24.759 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0234952 s, 44.6 MB/s 2021-02-10T11:20:24.760 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000344463 2021-02-10T11:20:25.190 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:25.179 7fb70dcf9700 1 Processor -- start 2021-02-10T11:20:25.190 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:25.179 7fb70dcf9700 1 -- start start 2021-02-10T11:20:25.190 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:25.179 7fb70dcf9700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fb7081b0870 0x7fb7080f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:25.190 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:25.179 7fb70dcf9700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7081b0c70 con 0x7fb7080efe00 2021-02-10T11:20:25.190 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:25.179 7fb70dcf9700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7080f0b50 con 0x7fb7081b0470 2021-02-10T11:20:25.190 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:25.179 7fb70dcf9700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7080e5ea0 con 0x7fb7081b0870 2021-02-10T11:20:25.190 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:25.179 7fb707fff700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fb7080efe00 0x7fb7080f0200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:45904/0 (socket says 172.21.2.102:45904) 2021-02-10T11:20:25.190 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:25.179 7fb707fff700 1 -- 172.21.2.102:0/4283998222 learned_addr learned my addr 172.21.2.102:0/4283998222 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:20:25.190 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:25.180 7fb7067fc700 1 -- 172.21.2.102:0/4283998222 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 653403044 0 0) 0x7fb700001500 con 0x7fb7080efe00 2021-02-10T11:20:25.191 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:25.180 7fb7067fc700 1 -- 172.21.2.102:0/4283998222 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6f0002fe0 con 0x7fb7080efe00 2021-02-10T11:20:25.191 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:25.180 7fb7067fc700 1 -- 172.21.2.102:0/4283998222 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2665621009 0 0) 0x7fb700001460 con 0x7fb7080efe00 2021-02-10T11:20:25.191 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:25.180 7fb7067fc700 1 -- 172.21.2.102:0/4283998222 >> v2:172.21.2.113:3300/0 conn(0x7fb7081b0870 msgr2=0x7fb7080f20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:20:25.191 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:25.180 7fb7067fc700 1 --2- 172.21.2.102:0/4283998222 >> v2:172.21.2.113:3300/0 conn(0x7fb7081b0870 0x7fb7080f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:20:25.191 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:25.180 7fb7067fc700 1 -- 172.21.2.102:0/4283998222 >> v1:172.21.2.113:6789/0 conn(0x7fb7081b0470 legacy=0x7fb7080ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:20:25.191 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:25.180 7fb7067fc700 1 -- 172.21.2.102:0/4283998222 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7080e52e0 con 0x7fb7080efe00 2021-02-10T11:20:25.191 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:25.180 7fb7067fc700 1 -- 172.21.2.102:0/4283998222 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7fb700001980 con 0x7fb7080efe00 2021-02-10T11:20:25.191 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:25.180 7fb7067fc700 1 -- 172.21.2.102:0/4283998222 >> v1:172.21.2.102:6789/0 conn(0x7fb7080efe00 legacy=0x7fb7080f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:25.191 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:25.180 7fb7067fc700 1 --2- 172.21.2.102:0/4283998222 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb6f0003a70 0x7fb6f0005f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:25.191 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:25.180 7fb7067fc700 1 --2- 172.21.2.102:0/4283998222 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb6f0006790 0x7fb6f0008cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:25.191 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:25.180 7fb7067fc700 1 --2- 172.21.2.102:0/4283998222 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb6f0009590 0x7fb6f000bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:25.191 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:25.180 7fb7067fc700 1 -- 172.21.2.102:0/4283998222 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6f000c300 con 0x7fb6f0006790 2021-02-10T11:20:25.191 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:25.180 7fb7067fc700 1 -- 172.21.2.102:0/4283998222 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6f000c520 con 0x7fb6f0009590 2021-02-10T11:20:25.191 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:25.180 7fb7067fc700 1 -- 172.21.2.102:0/4283998222 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6f000c740 con 0x7fb6f0003a70 2021-02-10T11:20:25.192 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:25.180 7fb706ffd700 1 --2- 172.21.2.102:0/4283998222 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb6f0006790 0x7fb6f0008cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:25.192 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:25.180 7fb707fff700 1 --2- 172.21.2.102:0/4283998222 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb6f0009590 0x7fb6f000bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:25.192 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:25.180 7fb7077fe700 1 --2- 172.21.2.102:0/4283998222 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb6f0003a70 0x7fb6f0005f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:25.192 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:25.180 7fb706ffd700 1 -- 172.21.2.102:0/4283998222 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb6f0003a70 msgr2=0x7fb6f0005f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:20:25.192 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:25.180 7fb706ffd700 1 --2- 172.21.2.102:0/4283998222 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb6f0003a70 0x7fb6f0005f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:25.192 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:25.180 7fb706ffd700 1 -- 172.21.2.102:0/4283998222 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb6f0009590 msgr2=0x7fb6f000bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:20:25.192 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:25.180 7fb706ffd700 1 --2- 172.21.2.102:0/4283998222 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb6f0009590 0x7fb6f000bab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:25.192 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:25.180 7fb706ffd700 1 -- 172.21.2.102:0/4283998222 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6f000c9e0 con 0x7fb6f0006790 2021-02-10T11:20:25.192 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:25.180 7fb707fff700 1 -- 172.21.2.102:0/4283998222 reap_dead start 2021-02-10T11:20:25.192 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:25.180 7fb7077fe700 1 --2- 172.21.2.102:0/4283998222 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb6f0003a70 0x7fb6f0005f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:20:25.192 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:25.180 7fb706ffd700 1 --2- 172.21.2.102:0/4283998222 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb6f0006790 0x7fb6f0008cb0 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 rx=0x7fb6fc006bf0 tx=0x7fb6fc04fcd0).ready entity=mon.0 client_cookie=8d69ecb894c3336e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:20:25.192 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:25.180 7fb7067fc700 1 -- 172.21.2.102:0/4283998222 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb6fc0009b0 con 0x7fb6f0006790 2021-02-10T11:20:25.192 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:25.181 7fb7067fc700 1 -- 172.21.2.102:0/4283998222 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6fc04fe10 con 0x7fb6f0006790 2021-02-10T11:20:25.192 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:25.181 7fb7067fc700 1 -- 172.21.2.102:0/4283998222 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb6fc002030 con 0x7fb6f0006790 2021-02-10T11:20:25.193 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:25.181 7fb70dcf9700 1 -- 172.21.2.102:0/4283998222 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb6f0006790 msgr2=0x7fb6f0008cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:25.193 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:25.181 7fb70dcf9700 1 --2- 172.21.2.102:0/4283998222 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb6f0006790 0x7fb6f0008cb0 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 rx=0x7fb6fc006bf0 tx=0x7fb6fc04fcd0).stop 2021-02-10T11:20:25.193 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:25.181 7fb70dcf9700 1 -- 172.21.2.102:0/4283998222 shutdown_connections 2021-02-10T11:20:25.193 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:25.181 7fb70dcf9700 1 --2- 172.21.2.102:0/4283998222 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb6f0006790 0x7fb6f0008cb0 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:25.193 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:25.181 7fb70dcf9700 1 -- 172.21.2.102:0/4283998222 >> 172.21.2.102:0/4283998222 conn(0x7fb7080ce3e0 msgr2=0x7fb7080e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:20:25.193 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:25.181 7fb70dcf9700 1 -- 172.21.2.102:0/4283998222 shutdown_connections 2021-02-10T11:20:25.193 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:25.181 7fb70dcf9700 1 -- 172.21.2.102:0/4283998222 wait complete. 2021-02-10T11:20:25.193 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:25.196 7fb70dcf9700 1 Processor -- start 2021-02-10T11:20:25.193 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:25.196 7fb70dcf9700 1 -- start start 2021-02-10T11:20:25.195 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:25.196 7fb70dcf9700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb7080b7db0 0x7fb7080b8260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:25.195 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:25.196 7fb70dcf9700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb7080b8880 0x7fb7081a7450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:25.195 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:25.196 7fb70dcf9700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb7080b8c80 0x7fb7081a7a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:25.195 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:25.196 7fb70dcf9700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7080b79b0 con 0x7fb7080b7db0 2021-02-10T11:20:25.195 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:25.196 7fb70dcf9700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7081a8090 con 0x7fb7080b8880 2021-02-10T11:20:25.195 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:25.196 7fb70dcf9700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7081a82b0 con 0x7fb7080b8c80 2021-02-10T11:20:25.195 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:25.196 7fb7077fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb7080b7db0 0x7fb7080b8260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:25.195 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:25.196 7fb7077fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb7080b7db0 0x7fb7080b8260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:53688/0 (socket says 172.21.2.102:53688) 2021-02-10T11:20:25.196 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:25.197 7fb7077fe700 1 -- 172.21.2.102:0/3758086975 learned_addr learned my addr 172.21.2.102:0/3758086975 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:20:25.196 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:25.197 7fb7077fe700 1 -- 172.21.2.102:0/3758086975 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb7080b8c80 msgr2=0x7fb7081a7a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:25.196 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:25.197 7fb7077fe700 1 --2- 172.21.2.102:0/3758086975 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb7080b8c80 0x7fb7081a7a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:20:25.196 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:25.197 7fb7077fe700 1 -- 172.21.2.102:0/3758086975 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb7080b8880 msgr2=0x7fb7081a7450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:25.196 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:25.197 7fb7077fe700 1 --2- 172.21.2.102:0/3758086975 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb7080b8880 0x7fb7081a7450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:20:25.196 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:25.197 7fb7077fe700 1 -- 172.21.2.102:0/3758086975 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb708190c10 con 0x7fb7080b7db0 2021-02-10T11:20:25.196 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:25.197 7fb7077fe700 1 --2- 172.21.2.102:0/3758086975 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb7080b7db0 0x7fb7080b8260 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 rx=0x7fb6f8006f00 tx=0x7fb6f800b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:20:25.196 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:25.197 7fb6effff700 1 -- 172.21.2.102:0/3758086975 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb6f800bd30 con 0x7fb7080b7db0 2021-02-10T11:20:25.196 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:25.197 7fb6effff700 1 -- 172.21.2.102:0/3758086975 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6f8006030 con 0x7fb7080b7db0 2021-02-10T11:20:25.197 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:25.197 7fb6effff700 1 -- 172.21.2.102:0/3758086975 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb6f80052e0 con 0x7fb7080b7db0 2021-02-10T11:20:25.197 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:25.197 7fb70dcf9700 1 -- 172.21.2.102:0/3758086975 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb708190c10 con 0x7fb7080b7db0 2021-02-10T11:20:25.197 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:25.197 7fb70dcf9700 1 -- 172.21.2.102:0/3758086975 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7080c1eb0 con 0x7fb7080b7db0 2021-02-10T11:20:25.197 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:25.198 7fb6effff700 1 -- 172.21.2.102:0/3758086975 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fb6f8006210 con 0x7fb7080b7db0 2021-02-10T11:20:25.197 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:25.198 7fb6effff700 1 --2- 172.21.2.102:0/3758086975 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb6e80186c0 0x7fb6e801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:25.197 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:25.198 7fb6effff700 1 -- 172.21.2.102:0/3758086975 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fb6f8032530 con 0x7fb7080b7db0 2021-02-10T11:20:25.197 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:25.198 7fb706ffd700 1 --2- 172.21.2.102:0/3758086975 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb6e80186c0 0x7fb6e801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:25.197 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:25.198 7fb706ffd700 1 --2- 172.21.2.102:0/3758086975 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb6e80186c0 0x7fb6e801abc0 crc :-1 s=READY pgs=385 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:20:25.223 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:25.229 7fb70dcf9700 1 -- 172.21.2.102:0/3758086975 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb708105b30 con 0x7fb7080b7db0 2021-02-10T11:20:25.225 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:25.232 7fb6effff700 1 -- 172.21.2.102:0/3758086975 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fb6f802dff0 con 0x7fb7080b7db0 2021-02-10T11:20:25.351 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:25.357 7fb70dcf9700 1 -- 172.21.2.102:0/3758086975 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344463", "target": ["mgr", ""]}) v1 -- 0x7fb7080c1eb0 con 0x7fb6e80186c0 2021-02-10T11:20:25.378 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:25.385 7fb6effff700 1 -- 172.21.2.102:0/3758086975 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fb6fc000940 con 0x7fb6e80186c0 2021-02-10T11:20:25.384 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:25.390 7fb70dcf9700 1 -- 172.21.2.102:0/3758086975 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb6e80186c0 msgr2=0x7fb6e801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:25.384 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:25.390 7fb70dcf9700 1 --2- 172.21.2.102:0/3758086975 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb6e80186c0 0x7fb6e801abc0 crc :-1 s=READY pgs=385 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:25.385 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:25.390 7fb70dcf9700 1 -- 172.21.2.102:0/3758086975 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb7080b7db0 msgr2=0x7fb7080b8260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:25.385 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:25.390 7fb70dcf9700 1 --2- 172.21.2.102:0/3758086975 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb7080b7db0 0x7fb7080b8260 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 rx=0x7fb6f8006f00 tx=0x7fb6f800b940).stop 2021-02-10T11:20:25.385 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:25.390 7fb70dcf9700 1 -- 172.21.2.102:0/3758086975 shutdown_connections 2021-02-10T11:20:25.385 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:25.390 7fb70dcf9700 1 --2- 172.21.2.102:0/3758086975 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb7080b8c80 0x7fb7081a7a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:20:25.385 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:25.390 7fb70dcf9700 1 --2- 172.21.2.102:0/3758086975 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb6e80186c0 0x7fb6e801abc0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:25.385 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:25.390 7fb70dcf9700 1 --2- 172.21.2.102:0/3758086975 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb7080b8880 0x7fb7081a7450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:20:25.385 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:25.390 7fb70dcf9700 1 --2- 172.21.2.102:0/3758086975 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb7080b7db0 0x7fb7080b8260 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:25.385 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:25.390 7fb70dcf9700 1 -- 172.21.2.102:0/3758086975 >> 172.21.2.102:0/3758086975 conn(0x7fb708094e10 msgr2=0x7fb7080a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:20:25.385 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:25.391 7fb70dcf9700 1 -- 172.21.2.102:0/3758086975 shutdown_connections 2021-02-10T11:20:25.385 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:25.391 7fb70dcf9700 1 -- 172.21.2.102:0/3758086975 wait complete. 2021-02-10T11:20:25.393 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000344463 2021-02-10T11:20:25.821 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:25.812 7fc5830fa700 1 Processor -- start 2021-02-10T11:20:25.821 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:25.812 7fc5830fa700 1 -- start start 2021-02-10T11:20:25.822 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:25.812 7fc5830fa700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fc57c1b0470 0x7fc57c1b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:25.822 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:25.812 7fc5830fa700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc57c0f0b50 con 0x7fc57c0ef610 2021-02-10T11:20:25.822 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:25.812 7fc5830fa700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc57c0f24e0 con 0x7fc57c0f20e0 2021-02-10T11:20:25.822 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:25.812 7fc5830fa700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc57c0e5ea0 con 0x7fc57c1b0470 2021-02-10T11:20:25.822 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:25.812 7fc580f16700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fc57c1b0470 0x7fc57c1b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:25.822 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:25.812 7fc580f16700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fc57c1b0470 0x7fc57c1b0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:53698/0 (socket says 172.21.2.102:53698) 2021-02-10T11:20:25.822 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:25.812 7fc580f16700 1 -- 172.21.2.102:0/2722094567 learned_addr learned my addr 172.21.2.102:0/2722094567 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:20:25.822 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:25.813 7fc580f16700 1 -- 172.21.2.102:0/2722094567 >> v1:172.21.2.113:6789/0 conn(0x7fc57c0f20e0 legacy=0x7fc57c0f0200 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:20:25.822 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:25.813 7fc580f16700 1 -- 172.21.2.102:0/2722094567 >> v1:172.21.2.102:6789/0 conn(0x7fc57c0ef610 legacy=0x7fc57c0efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:25.822 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:25.813 7fc580f16700 1 -- 172.21.2.102:0/2722094567 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc57c0e52e0 con 0x7fc57c1b0470 2021-02-10T11:20:25.823 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:25.813 7fc580f16700 1 --2- 172.21.2.102:0/2722094567 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc57c1b0470 0x7fc57c1b0870 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 rx=0x7fc564006700 tx=0x7fc564009b80).ready entity=mon.0 client_cookie=1927c930217454fa server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:20:25.823 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:25.813 7fc57b7fe700 1 -- 172.21.2.102:0/2722094567 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc56400c0c0 con 0x7fc57c1b0470 2021-02-10T11:20:25.823 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:25.813 7fc57b7fe700 1 -- 172.21.2.102:0/2722094567 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc56400c340 con 0x7fc57c1b0470 2021-02-10T11:20:25.823 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:25.813 7fc57b7fe700 1 -- 172.21.2.102:0/2722094567 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc56400c860 con 0x7fc57c1b0470 2021-02-10T11:20:25.823 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:25.813 7fc5830fa700 1 -- 172.21.2.102:0/2722094567 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc57c1b0470 msgr2=0x7fc57c1b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:25.823 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:25.813 7fc5830fa700 1 --2- 172.21.2.102:0/2722094567 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc57c1b0470 0x7fc57c1b0870 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 rx=0x7fc564006700 tx=0x7fc564009b80).stop 2021-02-10T11:20:25.823 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:25.813 7fc5830fa700 1 -- 172.21.2.102:0/2722094567 shutdown_connections 2021-02-10T11:20:25.823 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:25.813 7fc5830fa700 1 --2- 172.21.2.102:0/2722094567 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc57c1b0470 0x7fc57c1b0870 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:25.823 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:25.813 7fc5830fa700 1 -- 172.21.2.102:0/2722094567 >> 172.21.2.102:0/2722094567 conn(0x7fc57c0ce3e0 msgr2=0x7fc57c0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:20:25.823 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:25.814 7fc5830fa700 1 -- 172.21.2.102:0/2722094567 shutdown_connections 2021-02-10T11:20:25.823 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:25.814 7fc5830fa700 1 -- 172.21.2.102:0/2722094567 wait complete. 2021-02-10T11:20:25.825 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:25.828 7fc5830fa700 1 Processor -- start 2021-02-10T11:20:25.825 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:25.828 7fc5830fa700 1 -- start start 2021-02-10T11:20:25.825 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:25.828 7fc5830fa700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc57c087510 0x7fc57c0aedc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:25.825 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:25.828 7fc5830fa700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc57c1c68f0 0x7fc57c1b1f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:25.825 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:25.828 7fc5830fa700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc57c08c4d0 0x7fc57c1b18a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:25.825 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:25.828 7fc5830fa700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc57c0a2d50 con 0x7fc57c08c4d0 2021-02-10T11:20:25.826 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:25.828 7fc5830fa700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc57c0a2f70 con 0x7fc57c087510 2021-02-10T11:20:25.826 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:25.828 7fc5830fa700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc57c0a3190 con 0x7fc57c1c68f0 2021-02-10T11:20:25.826 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:25.829 7fc581717700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc57c08c4d0 0x7fc57c1b18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:25.826 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:25.829 7fc581717700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc57c08c4d0 0x7fc57c1b18a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:53706/0 (socket says 172.21.2.102:53706) 2021-02-10T11:20:25.826 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:25.829 7fc581717700 1 -- 172.21.2.102:0/4127002068 learned_addr learned my addr 172.21.2.102:0/4127002068 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:20:25.826 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:25.829 7fc581717700 1 -- 172.21.2.102:0/4127002068 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc57c1c68f0 msgr2=0x7fc57c1b1f60 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:20:25.826 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:25.829 7fc581717700 1 --2- 172.21.2.102:0/4127002068 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc57c1c68f0 0x7fc57c1b1f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:20:25.826 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:25.829 7fc581717700 1 -- 172.21.2.102:0/4127002068 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc57c087510 msgr2=0x7fc57c0aedc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:25.826 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:25.829 7fc581717700 1 --2- 172.21.2.102:0/4127002068 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc57c087510 0x7fc57c0aedc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:20:25.826 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:25.829 7fc581717700 1 -- 172.21.2.102:0/4127002068 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc57c0a33b0 con 0x7fc57c08c4d0 2021-02-10T11:20:25.826 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:25.829 7fc581717700 1 --2- 172.21.2.102:0/4127002068 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc57c08c4d0 0x7fc57c1b18a0 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 rx=0x7fc57400b8e0 tx=0x7fc57400bc80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:20:25.826 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:25.829 7fc5797fa700 1 -- 172.21.2.102:0/4127002068 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc574003030 con 0x7fc57c08c4d0 2021-02-10T11:20:25.826 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:25.829 7fc5797fa700 1 -- 172.21.2.102:0/4127002068 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc574000d30 con 0x7fc57c08c4d0 2021-02-10T11:20:25.827 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:25.829 7fc5797fa700 1 -- 172.21.2.102:0/4127002068 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc5740015b0 con 0x7fc57c08c4d0 2021-02-10T11:20:25.827 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:25.829 7fc5830fa700 1 -- 172.21.2.102:0/4127002068 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc57c0a33b0 con 0x7fc57c08c4d0 2021-02-10T11:20:25.827 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:25.829 7fc5830fa700 1 -- 172.21.2.102:0/4127002068 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc57c0afca0 con 0x7fc57c08c4d0 2021-02-10T11:20:25.827 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:25.830 7fc5797fa700 1 -- 172.21.2.102:0/4127002068 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fc57400a000 con 0x7fc57c08c4d0 2021-02-10T11:20:25.827 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:25.830 7fc5797fa700 1 --2- 172.21.2.102:0/4127002068 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc5600186c0 0x7fc56001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:25.827 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:25.830 7fc5797fa700 1 -- 172.21.2.102:0/4127002068 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fc5740328f0 con 0x7fc57c08c4d0 2021-02-10T11:20:25.827 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:25.830 7fc580f16700 1 --2- 172.21.2.102:0/4127002068 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc5600186c0 0x7fc56001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:25.827 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:25.830 7fc580f16700 1 --2- 172.21.2.102:0/4127002068 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc5600186c0 0x7fc56001abc0 crc :-1 s=READY pgs=386 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:20:25.856 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:25.862 7fc5830fa700 1 -- 172.21.2.102:0/4127002068 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc57c07e6b0 con 0x7fc57c08c4d0 2021-02-10T11:20:25.858 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:25.864 7fc5797fa700 1 -- 172.21.2.102:0/4127002068 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fc574000f90 con 0x7fc57c08c4d0 2021-02-10T11:20:25.988 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:25.994 7fc5830fa700 1 -- 172.21.2.102:0/4127002068 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344463", "target": ["mgr", ""]}) v1 -- 0x7fc57c0a2d50 con 0x7fc5600186c0 2021-02-10T11:20:25.990 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:25.996 7fc5797fa700 1 -- 172.21.2.102:0/4127002068 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7fc564000970 con 0x7fc5600186c0 2021-02-10T11:20:25.990 INFO:teuthology.orchestra.run.gibba002.stdout:/volumes/_nogroup/subvolume_0000000000344463/30f6f968-9c9f-4690-baac-45b5fbc94baa 2021-02-10T11:20:25.992 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:25.998 7fc5830fa700 1 -- 172.21.2.102:0/4127002068 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc5600186c0 msgr2=0x7fc56001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:25.992 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:25.998 7fc5830fa700 1 --2- 172.21.2.102:0/4127002068 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc5600186c0 0x7fc56001abc0 crc :-1 s=READY pgs=386 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:25.992 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:25.998 7fc5830fa700 1 -- 172.21.2.102:0/4127002068 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc57c08c4d0 msgr2=0x7fc57c1b18a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:25.992 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:25.998 7fc5830fa700 1 --2- 172.21.2.102:0/4127002068 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc57c08c4d0 0x7fc57c1b18a0 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 rx=0x7fc57400b8e0 tx=0x7fc57400bc80).stop 2021-02-10T11:20:25.993 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:25.999 7fc5830fa700 1 -- 172.21.2.102:0/4127002068 shutdown_connections 2021-02-10T11:20:25.993 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:25.999 7fc5830fa700 1 --2- 172.21.2.102:0/4127002068 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc57c08c4d0 0x7fc57c1b18a0 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:25.993 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:25.999 7fc5830fa700 1 --2- 172.21.2.102:0/4127002068 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc57c1c68f0 0x7fc57c1b1f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:20:25.993 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:25.999 7fc5830fa700 1 --2- 172.21.2.102:0/4127002068 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc5600186c0 0x7fc56001abc0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:25.993 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:25.999 7fc5830fa700 1 --2- 172.21.2.102:0/4127002068 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc57c087510 0x7fc57c0aedc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:20:25.993 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:25.999 7fc5830fa700 1 -- 172.21.2.102:0/4127002068 >> 172.21.2.102:0/4127002068 conn(0x7fc57c0886d0 msgr2=0x7fc57c081c40 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:20:25.993 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:25.999 7fc5830fa700 1 -- 172.21.2.102:0/4127002068 shutdown_connections 2021-02-10T11:20:25.994 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:25.999 7fc5830fa700 1 -- 172.21.2.102:0/4127002068 wait complete. 2021-02-10T11:20:26.000 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000344463 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000344463/30f6f968-9c9f-4690-baac-45b5fbc94baa 2021-02-10T11:20:26.001 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:20:26.001 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:20:26.001 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:20:26.337 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:20:26.337 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:20:26.337 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344463/30f6f968-9c9f-4690-baac-45b5fbc94baa/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:20:26.719 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:20:26.719 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:20:26.719 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0236688 s, 44.3 MB/s 2021-02-10T11:20:26.720 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:20:26.720 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:20:26.720 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:20:27.055 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:20:27.055 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:20:27.056 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344463/30f6f968-9c9f-4690-baac-45b5fbc94baa/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:20:27.429 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:20:27.429 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:20:27.429 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0229734 s, 45.6 MB/s 2021-02-10T11:20:27.430 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:20:27.430 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:20:27.430 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:20:27.765 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:20:27.766 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:20:27.766 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344463/30f6f968-9c9f-4690-baac-45b5fbc94baa/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:20:28.142 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:20:28.143 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:20:28.143 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0259095 s, 40.5 MB/s 2021-02-10T11:20:28.143 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:20:28.144 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:20:28.144 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:20:28.478 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:20:28.478 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:20:28.478 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344463/30f6f968-9c9f-4690-baac-45b5fbc94baa/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:20:28.849 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:20:28.849 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:20:28.850 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0245453 s, 42.7 MB/s 2021-02-10T11:20:28.850 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:20:28.851 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:20:28.851 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:20:29.184 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:20:29.185 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:20:29.185 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344463/30f6f968-9c9f-4690-baac-45b5fbc94baa/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:20:29.560 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:20:29.560 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:20:29.560 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0243901 s, 43.0 MB/s 2021-02-10T11:20:29.561 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:20:29.561 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:20:29.561 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:20:29.897 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:20:29.897 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:20:29.898 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344463/30f6f968-9c9f-4690-baac-45b5fbc94baa/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:20:30.273 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:20:30.274 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:20:30.274 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0247284 s, 42.4 MB/s 2021-02-10T11:20:30.275 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:20:30.275 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:20:30.275 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:20:30.608 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:20:30.609 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:20:30.609 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344463/30f6f968-9c9f-4690-baac-45b5fbc94baa/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:20:30.820 DEBUG:teuthology.orchestra.run.gibba002:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:20:30.823 DEBUG:teuthology.orchestra.run.gibba013:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:20:30.980 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:20:30.980 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:20:30.981 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0232162 s, 45.2 MB/s 2021-02-10T11:20:30.981 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:20:30.982 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:20:30.982 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:20:31.317 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:20:31.317 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:20:31.317 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344463/30f6f968-9c9f-4690-baac-45b5fbc94baa/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:20:31.695 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:20:31.695 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:20:31.695 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0252777 s, 41.5 MB/s 2021-02-10T11:20:31.696 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:20:31.696 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:20:31.696 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:20:32.029 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:20:32.030 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:20:32.030 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344463/30f6f968-9c9f-4690-baac-45b5fbc94baa/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:20:32.404 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:20:32.404 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:20:32.404 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0234554 s, 44.7 MB/s 2021-02-10T11:20:32.405 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:20:32.405 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:20:32.405 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:20:32.738 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:20:32.739 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:20:32.739 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000344463/30f6f968-9c9f-4690-baac-45b5fbc94baa/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:20:33.116 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:20:33.116 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:20:33.117 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0252617 s, 41.5 MB/s 2021-02-10T11:20:33.118 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:20:33.118 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:20:33.118 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:20:33.451 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:20:33.452 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:20:33.452 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@gibba002.front.sepia.ceph.com... 2021-02-10T11:20:33.452 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-02-10T11:20:33.452 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:20:33.862 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba002.stderr:ceph-fuse[44913]: fuse finished with error 0 and tester_r 0 2021-02-10T11:20:33.864 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:20:33.864 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:20:33.864 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:20:34.201 INFO:teuthology.orchestra.run.gibba002.stdout:ext2/ext3 2021-02-10T11:20:34.202 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2021-02-10T11:20:34.202 INFO:teuthology.orchestra.run:waiting for 900 2021-02-10T11:20:40.304 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:20:40.304 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:20:40.642 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000344364 2021-02-10T11:20:41.072 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:41.060 7fcacba2e700 1 Processor -- start 2021-02-10T11:20:41.073 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:41.061 7fcacba2e700 1 -- start start 2021-02-10T11:20:41.073 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:41.061 7fcacba2e700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fcac41b0470 0x7fcac41b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:41.073 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:41.061 7fcacba2e700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcac40ef610 0x7fcac40efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:41.073 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:41.061 7fcacba2e700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcac40f0b50 con 0x7fcac40f0030 2021-02-10T11:20:41.073 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:41.061 7fcacba2e700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcac40e52e0 con 0x7fcac41b0470 2021-02-10T11:20:41.073 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:41.061 7fcacba2e700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcac40e54d0 con 0x7fcac40ef610 2021-02-10T11:20:41.073 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:41.062 7fcac984a700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fcac41b0470 0x7fcac41b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:41.073 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:41.062 7fcac9049700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcac40ef610 0x7fcac40efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:41.073 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:41.062 7fcaca04b700 1 --1- >> v1:172.21.2.113:6789/0 conn(0x7fcac40f0030 0x7fcac40f10d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.113:6789/0 says I am v1:172.21.2.102:53258/0 (socket says 172.21.2.102:53258) 2021-02-10T11:20:41.073 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:41.062 7fcaca04b700 1 -- 172.21.2.102:0/2952357922 learned_addr learned my addr 172.21.2.102:0/2952357922 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:20:41.073 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:41.063 7fcac9049700 1 -- 172.21.2.102:0/2952357922 >> v2:172.21.2.113:3300/0 conn(0x7fcac41b0470 msgr2=0x7fcac41b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:20:41.073 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:41.063 7fcac9049700 1 --2- 172.21.2.102:0/2952357922 >> v2:172.21.2.113:3300/0 conn(0x7fcac41b0470 0x7fcac41b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:41.074 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:41.063 7fcac9049700 1 -- 172.21.2.102:0/2952357922 >> v1:172.21.2.113:6789/0 conn(0x7fcac40f0030 legacy=0x7fcac40f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:41.074 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:41.063 7fcac9049700 1 -- 172.21.2.102:0/2952357922 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcac40f0430 con 0x7fcac40ef610 2021-02-10T11:20:41.074 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:41.063 7fcac984a700 1 --2- 172.21.2.102:0/2952357922 >> v2:172.21.2.113:3300/0 conn(0x7fcac41b0470 0x7fcac41b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:20:41.074 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:41.063 7fcac9049700 1 --2- 172.21.2.102:0/2952357922 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcac40ef610 0x7fcac40efa10 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7fcab4006650 tx=0x7fcab4009a80).ready entity=mon.2 client_cookie=5bb8f44aa87a635d server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:20:41.074 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:41.063 7fcac8848700 1 -- 172.21.2.102:0/2952357922 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcab400c0c0 con 0x7fcac40ef610 2021-02-10T11:20:41.074 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:41.063 7fcac8848700 1 -- 172.21.2.102:0/2952357922 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcab400c340 con 0x7fcac40ef610 2021-02-10T11:20:41.074 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:41.063 7fcac8848700 1 -- 172.21.2.102:0/2952357922 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcab400c860 con 0x7fcac40ef610 2021-02-10T11:20:41.074 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:41.063 7fcacba2e700 1 -- 172.21.2.102:0/2952357922 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcac40ef610 msgr2=0x7fcac40efa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:41.074 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:41.063 7fcacba2e700 1 --2- 172.21.2.102:0/2952357922 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcac40ef610 0x7fcac40efa10 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7fcab4006650 tx=0x7fcab4009a80).stop 2021-02-10T11:20:41.074 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:41.064 7fcacba2e700 1 -- 172.21.2.102:0/2952357922 shutdown_connections 2021-02-10T11:20:41.074 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:41.064 7fcacba2e700 1 --2- 172.21.2.102:0/2952357922 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcac40ef610 0x7fcac40efa10 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:41.074 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:41.064 7fcacba2e700 1 --2- 172.21.2.102:0/2952357922 >> v2:172.21.2.113:3300/0 conn(0x7fcac41b0470 0x7fcac41b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:41.075 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:41.064 7fcacba2e700 1 -- 172.21.2.102:0/2952357922 >> 172.21.2.102:0/2952357922 conn(0x7fcac40ce3e0 msgr2=0x7fcac40e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:20:41.075 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:41.064 7fcacba2e700 1 -- 172.21.2.102:0/2952357922 shutdown_connections 2021-02-10T11:20:41.075 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:41.064 7fcacba2e700 1 -- 172.21.2.102:0/2952357922 wait complete. 2021-02-10T11:20:41.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:41.079 7fcacba2e700 1 Processor -- start 2021-02-10T11:20:41.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:41.079 7fcacba2e700 1 -- start start 2021-02-10T11:20:41.077 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:41.079 7fcacba2e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcac40a0420 0x7fcac40a0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:41.077 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:41.079 7fcacba2e700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcac4189d00 0x7fcac418a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:41.077 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:41.079 7fcacba2e700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fcac418a720 0x7fcac40b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:41.077 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:41.079 7fcacba2e700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcac4189800 con 0x7fcac40a0420 2021-02-10T11:20:41.077 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:41.079 7fcacba2e700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcac418ab20 con 0x7fcac418a720 2021-02-10T11:20:41.077 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:41.079 7fcacba2e700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcac40a1140 con 0x7fcac4189d00 2021-02-10T11:20:41.077 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:41.079 7fcac984a700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcac40a0420 0x7fcac40a0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:41.077 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:41.079 7fcac984a700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcac40a0420 0x7fcac40a0820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:53714/0 (socket says 172.21.2.102:53714) 2021-02-10T11:20:41.077 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:41.079 7fcac984a700 1 -- 172.21.2.102:0/1237402068 learned_addr learned my addr 172.21.2.102:0/1237402068 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:20:41.077 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:41.080 7fcac984a700 1 -- 172.21.2.102:0/1237402068 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcac4189d00 msgr2=0x7fcac418a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:41.077 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:41.080 7fcac984a700 1 --2- 172.21.2.102:0/1237402068 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcac4189d00 0x7fcac418a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:20:41.077 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:41.080 7fcac984a700 1 -- 172.21.2.102:0/1237402068 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fcac418a720 msgr2=0x7fcac40b7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:41.077 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:41.080 7fcac984a700 1 --2- 172.21.2.102:0/1237402068 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fcac418a720 0x7fcac40b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:20:41.078 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:41.080 7fcac984a700 1 -- 172.21.2.102:0/1237402068 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcac40b8200 con 0x7fcac40a0420 2021-02-10T11:20:41.078 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:41.080 7fcac984a700 1 --2- 172.21.2.102:0/1237402068 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcac40a0420 0x7fcac40a0820 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 rx=0x7fcac0006f40 tx=0x7fcac000b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:20:41.078 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:41.080 7fcaba7fc700 1 -- 172.21.2.102:0/1237402068 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcac000bd80 con 0x7fcac40a0420 2021-02-10T11:20:41.078 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:41.080 7fcaba7fc700 1 -- 172.21.2.102:0/1237402068 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcac0000c90 con 0x7fcac40a0420 2021-02-10T11:20:41.078 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:41.080 7fcacba2e700 1 -- 172.21.2.102:0/1237402068 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcac40b8200 con 0x7fcac40a0420 2021-02-10T11:20:41.078 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:41.080 7fcaba7fc700 1 -- 172.21.2.102:0/1237402068 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcac0002220 con 0x7fcac40a0420 2021-02-10T11:20:41.078 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:41.080 7fcacba2e700 1 -- 172.21.2.102:0/1237402068 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcac40b8200 con 0x7fcac40a0420 2021-02-10T11:20:41.078 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:41.081 7fcaba7fc700 1 -- 172.21.2.102:0/1237402068 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fcac000a000 con 0x7fcac40a0420 2021-02-10T11:20:41.078 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:41.081 7fcaba7fc700 1 --2- 172.21.2.102:0/1237402068 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcaac018730 0x7fcaac01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:41.078 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:41.081 7fcaba7fc700 1 -- 172.21.2.102:0/1237402068 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fcac0032610 con 0x7fcac40a0420 2021-02-10T11:20:41.078 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:41.081 7fcac9049700 1 --2- 172.21.2.102:0/1237402068 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcaac018730 0x7fcaac01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:41.078 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:41.081 7fcac9049700 1 --2- 172.21.2.102:0/1237402068 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcaac018730 0x7fcaac01ac10 crc :-1 s=READY pgs=387 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:20:41.105 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:41.111 7fcacba2e700 1 -- 172.21.2.102:0/1237402068 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcac4105ae0 con 0x7fcac40a0420 2021-02-10T11:20:41.108 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:41.114 7fcaba7fc700 1 -- 172.21.2.102:0/1237402068 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fcac0000c90 con 0x7fcac40a0420 2021-02-10T11:20:41.233 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:41.239 7fcacba2e700 1 -- 172.21.2.102:0/1237402068 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344364", "target": ["mgr", ""]}) v1 -- 0x7fcac418ab20 con 0x7fcaac018730 2021-02-10T11:20:41.247 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:41.253 7fcaba7fc700 1 -- 172.21.2.102:0/1237402068 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fcab4000940 con 0x7fcaac018730 2021-02-10T11:20:41.250 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:41.256 7fcacba2e700 1 -- 172.21.2.102:0/1237402068 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcaac018730 msgr2=0x7fcaac01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:41.250 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:41.256 7fcacba2e700 1 --2- 172.21.2.102:0/1237402068 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcaac018730 0x7fcaac01ac10 crc :-1 s=READY pgs=387 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:41.250 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:41.256 7fcacba2e700 1 -- 172.21.2.102:0/1237402068 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcac40a0420 msgr2=0x7fcac40a0820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:41.250 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:41.256 7fcacba2e700 1 --2- 172.21.2.102:0/1237402068 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcac40a0420 0x7fcac40a0820 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 rx=0x7fcac0006f40 tx=0x7fcac000b990).stop 2021-02-10T11:20:41.250 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:41.256 7fcacba2e700 1 -- 172.21.2.102:0/1237402068 shutdown_connections 2021-02-10T11:20:41.251 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:41.256 7fcacba2e700 1 --2- 172.21.2.102:0/1237402068 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcaac018730 0x7fcaac01ac10 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:41.251 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:41.256 7fcacba2e700 1 --2- 172.21.2.102:0/1237402068 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fcac418a720 0x7fcac40b7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:20:41.251 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:41.256 7fcacba2e700 1 --2- 172.21.2.102:0/1237402068 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcac4189d00 0x7fcac418a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:20:41.251 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:41.256 7fcacba2e700 1 --2- 172.21.2.102:0/1237402068 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcac40a0420 0x7fcac40a0820 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:41.251 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:41.256 7fcacba2e700 1 -- 172.21.2.102:0/1237402068 >> 172.21.2.102:0/1237402068 conn(0x7fcac4094e10 msgr2=0x7fcac40a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:20:41.251 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:41.256 7fcacba2e700 1 -- 172.21.2.102:0/1237402068 shutdown_connections 2021-02-10T11:20:41.251 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:41.256 7fcacba2e700 1 -- 172.21.2.102:0/1237402068 wait complete. 2021-02-10T11:20:41.260 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000344365 2021-02-10T11:20:41.690 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:41.680 7f20cbb09700 1 Processor -- start 2021-02-10T11:20:41.691 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:41.680 7f20cbb09700 1 -- start start 2021-02-10T11:20:41.691 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:41.680 7f20cbb09700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f20c41b0870 0x7f20c40f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:41.691 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:41.680 7f20cbb09700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f20c40efe00 0x7f20c40f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:41.691 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:41.680 7f20cbb09700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20c41b0c70 con 0x7f20c41b0470 2021-02-10T11:20:41.691 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:41.680 7f20cbb09700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20c40e5ea0 con 0x7f20c41b0870 2021-02-10T11:20:41.691 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:41.680 7f20cbb09700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20c40f0b50 con 0x7f20c40efe00 2021-02-10T11:20:41.691 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:41.681 7f20c9124700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f20c41b0870 0x7f20c40f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:41.691 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:41.681 7f20c9925700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f20c41b0470 0x7f20c40ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:45946/0 (socket says 172.21.2.102:45946) 2021-02-10T11:20:41.691 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:41.681 7f20c9925700 1 -- 172.21.2.102:0/3106667964 learned_addr learned my addr 172.21.2.102:0/3106667964 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:20:41.691 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:41.681 7f20ca126700 1 --2- 172.21.2.102:0/3106667964 >> v2:172.21.2.113:3300/0 conn(0x7f20c40efe00 0x7f20c40f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:41.691 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:41.681 7f20c9124700 1 -- 172.21.2.102:0/3106667964 >> v2:172.21.2.113:3300/0 conn(0x7f20c40efe00 msgr2=0x7f20c40f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:20:41.692 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:41.681 7f20c9124700 1 --2- 172.21.2.102:0/3106667964 >> v2:172.21.2.113:3300/0 conn(0x7f20c40efe00 0x7f20c40f0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:41.692 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:41.681 7f20c9124700 1 -- 172.21.2.102:0/3106667964 >> v1:172.21.2.102:6789/0 conn(0x7f20c41b0470 legacy=0x7f20c40ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:41.692 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:41.681 7f20c9124700 1 -- 172.21.2.102:0/3106667964 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20c40f0820 con 0x7f20c41b0870 2021-02-10T11:20:41.692 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:41.681 7f20ca126700 1 --2- 172.21.2.102:0/3106667964 >> v2:172.21.2.113:3300/0 conn(0x7f20c40efe00 0x7f20c40f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:20:41.692 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:41.681 7f20c9124700 1 --2- 172.21.2.102:0/3106667964 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f20c41b0870 0x7f20c40f20e0 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 rx=0x7f20b8006650 tx=0x7f20b8009a80).ready entity=mon.0 client_cookie=ce638da31a44be4f server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:20:41.692 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:41.681 7f20c8923700 1 -- 172.21.2.102:0/3106667964 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f20b80009b0 con 0x7f20c41b0870 2021-02-10T11:20:41.692 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:41.681 7f20c8923700 1 -- 172.21.2.102:0/3106667964 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20b800c0c0 con 0x7f20c41b0870 2021-02-10T11:20:41.692 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:41.681 7f20c8923700 1 -- 172.21.2.102:0/3106667964 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f20b800c520 con 0x7f20c41b0870 2021-02-10T11:20:41.692 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:41.681 7f20cbb09700 1 -- 172.21.2.102:0/3106667964 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f20c41b0870 msgr2=0x7f20c40f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:41.692 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:41.681 7f20cbb09700 1 --2- 172.21.2.102:0/3106667964 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f20c41b0870 0x7f20c40f20e0 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 rx=0x7f20b8006650 tx=0x7f20b8009a80).stop 2021-02-10T11:20:41.692 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:41.682 7f20cbb09700 1 -- 172.21.2.102:0/3106667964 shutdown_connections 2021-02-10T11:20:41.692 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:41.682 7f20cbb09700 1 --2- 172.21.2.102:0/3106667964 >> v2:172.21.2.113:3300/0 conn(0x7f20c40efe00 0x7f20c40f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:41.692 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:41.682 7f20cbb09700 1 --2- 172.21.2.102:0/3106667964 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f20c41b0870 0x7f20c40f20e0 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:41.693 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:41.682 7f20cbb09700 1 -- 172.21.2.102:0/3106667964 >> 172.21.2.102:0/3106667964 conn(0x7f20c40ce3e0 msgr2=0x7f20c40e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:20:41.693 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:41.682 7f20cbb09700 1 -- 172.21.2.102:0/3106667964 shutdown_connections 2021-02-10T11:20:41.693 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:41.682 7f20cbb09700 1 -- 172.21.2.102:0/3106667964 wait complete. 2021-02-10T11:20:41.694 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:41.697 7f20cbb09700 1 Processor -- start 2021-02-10T11:20:41.694 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:41.697 7f20cbb09700 1 -- start start 2021-02-10T11:20:41.694 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:41.697 7f20cbb09700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f20c40a0420 0x7f20c40a0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:41.694 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:41.697 7f20cbb09700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f20c4189d00 0x7f20c418a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:41.694 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:41.697 7f20cbb09700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f20c418a720 0x7f20c40b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:41.695 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:41.697 7f20cbb09700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20c4189780 con 0x7f20c40a0420 2021-02-10T11:20:41.695 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:41.697 7f20cbb09700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20c418ab20 con 0x7f20c418a720 2021-02-10T11:20:41.695 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:41.697 7f20cbb09700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20c40a1140 con 0x7f20c4189d00 2021-02-10T11:20:41.695 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:41.697 7f20c9925700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f20c40a0420 0x7f20c40a0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:41.695 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:41.697 7f20c9925700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f20c40a0420 0x7f20c40a0820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:53726/0 (socket says 172.21.2.102:53726) 2021-02-10T11:20:41.695 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:41.697 7f20c9925700 1 -- 172.21.2.102:0/671589826 learned_addr learned my addr 172.21.2.102:0/671589826 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:20:41.695 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:41.698 7f20c9925700 1 -- 172.21.2.102:0/671589826 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f20c4189d00 msgr2=0x7f20c418a100 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:20:41.695 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:41.698 7f20c9925700 1 --2- 172.21.2.102:0/671589826 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f20c4189d00 0x7f20c418a100 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:20:41.695 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:41.698 7f20c9925700 1 -- 172.21.2.102:0/671589826 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f20c418a720 msgr2=0x7f20c40b7be0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:20:41.695 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:41.698 7f20c9925700 1 --2- 172.21.2.102:0/671589826 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f20c418a720 0x7f20c40b7be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:20:41.695 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:41.698 7f20c9925700 1 -- 172.21.2.102:0/671589826 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20c40b8200 con 0x7f20c40a0420 2021-02-10T11:20:41.695 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:41.698 7f20c9925700 1 --2- 172.21.2.102:0/671589826 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f20c40a0420 0x7f20c40a0820 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 rx=0x7f20c0001c60 tx=0x7f20c000a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:20:41.695 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:41.698 7f20b67fc700 1 -- 172.21.2.102:0/671589826 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f20c000acc0 con 0x7f20c40a0420 2021-02-10T11:20:41.695 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:41.698 7f20b67fc700 1 -- 172.21.2.102:0/671589826 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20c0004140 con 0x7f20c40a0420 2021-02-10T11:20:41.696 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:41.698 7f20b67fc700 1 -- 172.21.2.102:0/671589826 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f20c00045a0 con 0x7f20c40a0420 2021-02-10T11:20:41.696 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:41.698 7f20cbb09700 1 -- 172.21.2.102:0/671589826 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20c40b8200 con 0x7f20c40a0420 2021-02-10T11:20:41.696 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:41.698 7f20cbb09700 1 -- 172.21.2.102:0/671589826 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20c40b8200 con 0x7f20c40a0420 2021-02-10T11:20:41.696 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:41.699 7f20b67fc700 1 -- 172.21.2.102:0/671589826 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f20c0000ce0 con 0x7f20c40a0420 2021-02-10T11:20:41.696 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:41.699 7f20b67fc700 1 --2- 172.21.2.102:0/671589826 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f20ac0186c0 0x7f20ac01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:41.696 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:41.699 7f20b67fc700 1 -- 172.21.2.102:0/671589826 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f20c00327c0 con 0x7f20c40a0420 2021-02-10T11:20:41.696 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:41.699 7f20c9124700 1 --2- 172.21.2.102:0/671589826 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f20ac0186c0 0x7f20ac01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:41.696 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:41.699 7f20c9124700 1 --2- 172.21.2.102:0/671589826 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f20ac0186c0 0x7f20ac01abc0 crc :-1 s=READY pgs=388 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:20:41.724 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:41.730 7f20cbb09700 1 -- 172.21.2.102:0/671589826 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20c4105ae0 con 0x7f20c40a0420 2021-02-10T11:20:41.727 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:41.733 7f20b67fc700 1 -- 172.21.2.102:0/671589826 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f20c00009b0 con 0x7f20c40a0420 2021-02-10T11:20:41.853 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:41.859 7f20cbb09700 1 -- 172.21.2.102:0/671589826 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344365", "target": ["mgr", ""]}) v1 -- 0x7f20c4105ae0 con 0x7f20ac0186c0 2021-02-10T11:20:41.865 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:41.871 7f20b67fc700 1 -- 172.21.2.102:0/671589826 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f20b8000d10 con 0x7f20ac0186c0 2021-02-10T11:20:41.870 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:41.876 7f20cbb09700 1 -- 172.21.2.102:0/671589826 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f20ac0186c0 msgr2=0x7f20ac01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:41.870 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:41.876 7f20cbb09700 1 --2- 172.21.2.102:0/671589826 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f20ac0186c0 0x7f20ac01abc0 crc :-1 s=READY pgs=388 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:41.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:41.876 7f20cbb09700 1 -- 172.21.2.102:0/671589826 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f20c40a0420 msgr2=0x7f20c40a0820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:41.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:41.876 7f20cbb09700 1 --2- 172.21.2.102:0/671589826 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f20c40a0420 0x7f20c40a0820 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 rx=0x7f20c0001c60 tx=0x7f20c000a8d0).stop 2021-02-10T11:20:41.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:41.876 7f20cbb09700 1 -- 172.21.2.102:0/671589826 shutdown_connections 2021-02-10T11:20:41.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:41.876 7f20cbb09700 1 --2- 172.21.2.102:0/671589826 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f20ac0186c0 0x7f20ac01abc0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:41.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:41.876 7f20cbb09700 1 --2- 172.21.2.102:0/671589826 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f20c418a720 0x7f20c40b7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:20:41.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:41.876 7f20cbb09700 1 --2- 172.21.2.102:0/671589826 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f20c4189d00 0x7f20c418a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:20:41.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:41.876 7f20cbb09700 1 --2- 172.21.2.102:0/671589826 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f20c40a0420 0x7f20c40a0820 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:41.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:41.876 7f20cbb09700 1 -- 172.21.2.102:0/671589826 >> 172.21.2.102:0/671589826 conn(0x7f20c4094e10 msgr2=0x7f20c40a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:20:41.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:41.876 7f20cbb09700 1 -- 172.21.2.102:0/671589826 shutdown_connections 2021-02-10T11:20:41.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:41.876 7f20cbb09700 1 -- 172.21.2.102:0/671589826 wait complete. 2021-02-10T11:20:41.880 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000344366 2021-02-10T11:20:42.311 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:42.299 7f80340f0700 1 Processor -- start 2021-02-10T11:20:42.311 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:42.299 7f80340f0700 1 -- start start 2021-02-10T11:20:42.311 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:42.300 7f80340f0700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f802c1b0470 0x7f802c1b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:42.311 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:42.300 7f80340f0700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f802c0ef610 0x7f802c0efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:42.311 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:42.300 7f80340f0700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f802c0f0b50 con 0x7f802c0f0030 2021-02-10T11:20:42.311 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:42.300 7f80340f0700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f802c0e52e0 con 0x7f802c1b0470 2021-02-10T11:20:42.311 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:42.300 7f80340f0700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f802c0e5490 con 0x7f802c0ef610 2021-02-10T11:20:42.311 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:42.300 7f8031f0c700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f802c1b0470 0x7f802c1b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:42.311 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:42.300 7f8031f0c700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f802c1b0470 0x7f802c1b0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:53734/0 (socket says 172.21.2.102:53734) 2021-02-10T11:20:42.311 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:42.300 7f8031f0c700 1 -- 172.21.2.102:0/466523061 learned_addr learned my addr 172.21.2.102:0/466523061 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:20:42.312 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:42.300 7f8031f0c700 1 -- 172.21.2.102:0/466523061 >> v2:172.21.2.113:3300/0 conn(0x7f802c0ef610 msgr2=0x7f802c0efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:20:42.312 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:42.300 7f8031f0c700 1 --2- 172.21.2.102:0/466523061 >> v2:172.21.2.113:3300/0 conn(0x7f802c0ef610 0x7f802c0efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:20:42.312 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:42.300 7f8031f0c700 1 -- 172.21.2.102:0/466523061 >> v1:172.21.2.113:6789/0 conn(0x7f802c0f0030 legacy=0x7f802c0f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:20:42.312 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:42.300 7f8031f0c700 1 -- 172.21.2.102:0/466523061 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f802c0f0430 con 0x7f802c1b0470 2021-02-10T11:20:42.312 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:42.301 7f8031f0c700 1 --2- 172.21.2.102:0/466523061 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f802c1b0470 0x7f802c1b0870 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 rx=0x7f8028006700 tx=0x7f8028009b80).ready entity=mon.0 client_cookie=7f3ad4e2e1f37d5e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:20:42.312 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:42.301 7f8030f0a700 1 -- 172.21.2.102:0/466523061 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f802800c0c0 con 0x7f802c1b0470 2021-02-10T11:20:42.312 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:42.301 7f8030f0a700 1 -- 172.21.2.102:0/466523061 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f802800c400 con 0x7f802c1b0470 2021-02-10T11:20:42.312 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:42.301 7f8030f0a700 1 -- 172.21.2.102:0/466523061 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f802800c860 con 0x7f802c1b0470 2021-02-10T11:20:42.312 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:42.301 7f80340f0700 1 -- 172.21.2.102:0/466523061 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f802c1b0470 msgr2=0x7f802c1b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:42.312 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:42.301 7f80340f0700 1 --2- 172.21.2.102:0/466523061 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f802c1b0470 0x7f802c1b0870 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 rx=0x7f8028006700 tx=0x7f8028009b80).stop 2021-02-10T11:20:42.312 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:42.302 7f80340f0700 1 -- 172.21.2.102:0/466523061 shutdown_connections 2021-02-10T11:20:42.312 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:42.302 7f80340f0700 1 --2- 172.21.2.102:0/466523061 >> v2:172.21.2.113:3300/0 conn(0x7f802c0ef610 0x7f802c0efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:20:42.312 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:42.302 7f80340f0700 1 --2- 172.21.2.102:0/466523061 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f802c1b0470 0x7f802c1b0870 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:42.313 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:42.302 7f80340f0700 1 -- 172.21.2.102:0/466523061 >> 172.21.2.102:0/466523061 conn(0x7f802c0ce3e0 msgr2=0x7f802c0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:20:42.313 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:42.302 7f80340f0700 1 -- 172.21.2.102:0/466523061 shutdown_connections 2021-02-10T11:20:42.313 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:42.302 7f80340f0700 1 -- 172.21.2.102:0/466523061 wait complete. 2021-02-10T11:20:42.314 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:42.317 7f80340f0700 1 Processor -- start 2021-02-10T11:20:42.314 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:42.317 7f80340f0700 1 -- start start 2021-02-10T11:20:42.314 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:42.317 7f80340f0700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f802c0ee230 0x7f802c0bfc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:42.314 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:42.317 7f80340f0700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f802c0ee630 0x7f802c0eea30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:42.314 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:42.317 7f80340f0700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f802c0c09e0 0x7f802c0c0de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:42.314 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:42.317 7f80340f0700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f802c0c1f60 con 0x7f802c0ee630 2021-02-10T11:20:42.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:42.317 7f80340f0700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f802c18bdf0 con 0x7f802c0ee230 2021-02-10T11:20:42.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:42.317 7f80340f0700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f802c0c0460 con 0x7f802c0c09e0 2021-02-10T11:20:42.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:42.317 7f803170b700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f802c0ee630 0x7f802c0eea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:42.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:42.317 7f803170b700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f802c0ee630 0x7f802c0eea30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:53740/0 (socket says 172.21.2.102:53740) 2021-02-10T11:20:42.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:42.317 7f803170b700 1 -- 172.21.2.102:0/1136192553 learned_addr learned my addr 172.21.2.102:0/1136192553 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:20:42.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:42.317 7f8031f0c700 1 --2- 172.21.2.102:0/1136192553 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f802c0ee230 0x7f802c0bfc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:42.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:42.317 7f803270d700 1 --2- 172.21.2.102:0/1136192553 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f802c0c09e0 0x7f802c0c0de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:42.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:42.318 7f803170b700 1 -- 172.21.2.102:0/1136192553 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f802c0c09e0 msgr2=0x7f802c0c0de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:42.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:42.318 7f803170b700 1 --2- 172.21.2.102:0/1136192553 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f802c0c09e0 0x7f802c0c0de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:42.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:42.318 7f803170b700 1 -- 172.21.2.102:0/1136192553 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f802c0ee230 msgr2=0x7f802c0bfc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:42.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:42.318 7f803170b700 1 --2- 172.21.2.102:0/1136192553 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f802c0ee230 0x7f802c0bfc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:42.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:42.318 7f803170b700 1 -- 172.21.2.102:0/1136192553 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f802c0c1400 con 0x7f802c0ee630 2021-02-10T11:20:42.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:42.318 7f803170b700 1 --2- 172.21.2.102:0/1136192553 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f802c0ee630 0x7f802c0eea30 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 rx=0x7f8020006f00 tx=0x7f802000b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:20:42.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:42.318 7f801e7fc700 1 -- 172.21.2.102:0/1136192553 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f802000bd30 con 0x7f802c0ee630 2021-02-10T11:20:42.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:42.318 7f801e7fc700 1 -- 172.21.2.102:0/1136192553 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8020000c90 con 0x7f802c0ee630 2021-02-10T11:20:42.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:42.318 7f801e7fc700 1 -- 172.21.2.102:0/1136192553 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f80200022a0 con 0x7f802c0ee630 2021-02-10T11:20:42.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:42.318 7f80340f0700 1 -- 172.21.2.102:0/1136192553 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f802c0c1400 con 0x7f802c0ee630 2021-02-10T11:20:42.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:42.318 7f80340f0700 1 -- 172.21.2.102:0/1136192553 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f802c0c1400 con 0x7f802c0ee630 2021-02-10T11:20:42.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:42.319 7f801e7fc700 1 -- 172.21.2.102:0/1136192553 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f8020005310 con 0x7f802c0ee630 2021-02-10T11:20:42.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:42.319 7f801e7fc700 1 --2- 172.21.2.102:0/1136192553 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8014018730 0x7f801401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:42.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:42.319 7f801e7fc700 1 -- 172.21.2.102:0/1136192553 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f8020032620 con 0x7f802c0ee630 2021-02-10T11:20:42.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:42.319 7f8031f0c700 1 --2- 172.21.2.102:0/1136192553 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8014018730 0x7f801401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:42.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:42.319 7f8031f0c700 1 --2- 172.21.2.102:0/1136192553 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8014018730 0x7f801401ac10 crc :-1 s=READY pgs=389 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:20:42.344 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:42.350 7f80340f0700 1 -- 172.21.2.102:0/1136192553 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f802c0eda20 con 0x7f802c0ee630 2021-02-10T11:20:42.347 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:42.352 7f801e7fc700 1 -- 172.21.2.102:0/1136192553 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f8020006b90 con 0x7f802c0ee630 2021-02-10T11:20:42.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:42.481 7f80340f0700 1 -- 172.21.2.102:0/1136192553 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344366", "target": ["mgr", ""]}) v1 -- 0x7f802c0eda20 con 0x7f8014018730 2021-02-10T11:20:42.488 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:42.494 7f801e7fc700 1 -- 172.21.2.102:0/1136192553 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f8028009380 con 0x7f8014018730 2021-02-10T11:20:42.493 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:42.498 7f80340f0700 1 -- 172.21.2.102:0/1136192553 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8014018730 msgr2=0x7f801401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:42.493 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:42.498 7f80340f0700 1 --2- 172.21.2.102:0/1136192553 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8014018730 0x7f801401ac10 crc :-1 s=READY pgs=389 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:42.493 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:42.499 7f80340f0700 1 -- 172.21.2.102:0/1136192553 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f802c0ee630 msgr2=0x7f802c0eea30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:42.493 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:42.499 7f80340f0700 1 --2- 172.21.2.102:0/1136192553 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f802c0ee630 0x7f802c0eea30 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 rx=0x7f8020006f00 tx=0x7f802000b940).stop 2021-02-10T11:20:42.494 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:42.499 7f80340f0700 1 -- 172.21.2.102:0/1136192553 shutdown_connections 2021-02-10T11:20:42.494 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:42.499 7f80340f0700 1 --2- 172.21.2.102:0/1136192553 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f802c0c09e0 0x7f802c0c0de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:42.494 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:42.499 7f80340f0700 1 --2- 172.21.2.102:0/1136192553 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f802c0ee630 0x7f802c0eea30 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:42.494 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:42.499 7f80340f0700 1 --2- 172.21.2.102:0/1136192553 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8014018730 0x7f801401ac10 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:42.494 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:42.499 7f80340f0700 1 --2- 172.21.2.102:0/1136192553 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f802c0ee230 0x7f802c0bfc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:42.494 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:42.499 7f80340f0700 1 -- 172.21.2.102:0/1136192553 >> 172.21.2.102:0/1136192553 conn(0x7f802c09dfd0 msgr2=0x7f802c09abf0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:20:42.494 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:42.499 7f80340f0700 1 -- 172.21.2.102:0/1136192553 shutdown_connections 2021-02-10T11:20:42.494 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:42.499 7f80340f0700 1 -- 172.21.2.102:0/1136192553 wait complete. 2021-02-10T11:20:42.502 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000344367 2021-02-10T11:20:42.937 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:42.924 7f011096e700 1 Processor -- start 2021-02-10T11:20:42.937 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:42.924 7f011096e700 1 -- start start 2021-02-10T11:20:42.937 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:42.925 7f011096e700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f010c1b0470 0x7f010c1b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:42.937 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:42.925 7f011096e700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f010c0ef610 0x7f010c0efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:42.937 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:42.925 7f011096e700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f010c0f0b50 con 0x7f010c0f0030 2021-02-10T11:20:42.937 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:42.925 7f011096e700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f010c0e52e0 con 0x7f010c0ef610 2021-02-10T11:20:42.937 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:42.925 7f011096e700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f010c0e5490 con 0x7f010c1b0470 2021-02-10T11:20:42.937 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:42.925 7f010ae1d700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f010c0f0030 0x7f010c0f10d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:45970/0 (socket says 172.21.2.102:45970) 2021-02-10T11:20:42.938 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:42.925 7f010ae1d700 1 -- 172.21.2.102:0/2779410563 learned_addr learned my addr 172.21.2.102:0/2779410563 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:20:42.938 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:42.925 7f0109e1b700 1 --2- 172.21.2.102:0/2779410563 >> v2:172.21.2.113:3300/0 conn(0x7f010c0ef610 0x7f010c0efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:42.938 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:42.925 7f010a61c700 1 --2- 172.21.2.102:0/2779410563 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f010c1b0470 0x7f010c1b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:42.938 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:42.925 7f010961a700 1 -- 172.21.2.102:0/2779410563 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3222427558 0 0) 0x7f0104001500 con 0x7f010c0f0030 2021-02-10T11:20:42.938 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:42.926 7f010961a700 1 -- 172.21.2.102:0/2779410563 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f00f4002fe0 con 0x7f010c0f0030 2021-02-10T11:20:42.938 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:42.926 7f010961a700 1 -- 172.21.2.102:0/2779410563 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3043094314 0 0) 0x7f0104001270 con 0x7f010c0f0030 2021-02-10T11:20:42.938 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:42.926 7f010961a700 1 -- 172.21.2.102:0/2779410563 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f010c1b0470 msgr2=0x7f010c1b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:20:42.938 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:42.926 7f010961a700 1 --2- 172.21.2.102:0/2779410563 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f010c1b0470 0x7f010c1b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:42.938 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:42.926 7f010961a700 1 -- 172.21.2.102:0/2779410563 >> v2:172.21.2.113:3300/0 conn(0x7f010c0ef610 msgr2=0x7f010c0efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:20:42.938 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:42.926 7f010961a700 1 --2- 172.21.2.102:0/2779410563 >> v2:172.21.2.113:3300/0 conn(0x7f010c0ef610 0x7f010c0efa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:42.938 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:42.926 7f010961a700 1 -- 172.21.2.102:0/2779410563 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f010c0f0430 con 0x7f010c0f0030 2021-02-10T11:20:42.938 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:42.926 7f010961a700 1 -- 172.21.2.102:0/2779410563 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7f0104001960 con 0x7f010c0f0030 2021-02-10T11:20:42.939 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:42.926 7f010961a700 1 -- 172.21.2.102:0/2779410563 >> v1:172.21.2.102:6789/0 conn(0x7f010c0f0030 legacy=0x7f010c0f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:42.939 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:42.926 7f010961a700 1 --2- 172.21.2.102:0/2779410563 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f00f4003b90 0x7f00f4006040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:42.939 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:42.926 7f010961a700 1 --2- 172.21.2.102:0/2779410563 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f00f404a8f0 0x7f00f404cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:42.939 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:42.926 7f010961a700 1 --2- 172.21.2.102:0/2779410563 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f00f404d6f0 0x7f00f404fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:42.939 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:42.926 7f010961a700 1 -- 172.21.2.102:0/2779410563 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00f4050430 con 0x7f00f404a8f0 2021-02-10T11:20:42.939 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:42.926 7f010961a700 1 -- 172.21.2.102:0/2779410563 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00f4050650 con 0x7f00f4003b90 2021-02-10T11:20:42.939 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:42.926 7f010961a700 1 -- 172.21.2.102:0/2779410563 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00f4050870 con 0x7f00f404d6f0 2021-02-10T11:20:42.939 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:42.926 7f010a61c700 1 --2- 172.21.2.102:0/2779410563 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f010c1b0470 0x7f010c1b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:20:42.939 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:42.926 7f0109e1b700 1 --2- 172.21.2.102:0/2779410563 >> v2:172.21.2.113:3300/0 conn(0x7f010c0ef610 0x7f010c0efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:20:42.939 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:42.926 7f010ae1d700 1 --2- 172.21.2.102:0/2779410563 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f00f404d6f0 0x7f00f404fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:42.939 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:42.926 7f010a61c700 1 --2- 172.21.2.102:0/2779410563 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f00f4003b90 0x7f00f4006040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:42.939 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:42.926 7f0109e1b700 1 --2- 172.21.2.102:0/2779410563 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f00f404a8f0 0x7f00f404cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:42.939 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:42.926 7f0109e1b700 1 -- 172.21.2.102:0/2779410563 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f00f404d6f0 msgr2=0x7f00f404fbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:20:42.939 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:42.926 7f0109e1b700 1 --2- 172.21.2.102:0/2779410563 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f00f404d6f0 0x7f00f404fbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:42.940 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:42.926 7f0109e1b700 1 -- 172.21.2.102:0/2779410563 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f00f4003b90 msgr2=0x7f00f4006040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:20:42.940 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:42.926 7f0109e1b700 1 --2- 172.21.2.102:0/2779410563 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f00f4003b90 0x7f00f4006040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:42.940 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:42.927 7f0109e1b700 1 -- 172.21.2.102:0/2779410563 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00f4050b10 con 0x7f00f404a8f0 2021-02-10T11:20:42.940 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:42.927 7f010a61c700 1 --2- 172.21.2.102:0/2779410563 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f00f4003b90 0x7f00f4006040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:20:42.940 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:42.927 7f010ae1d700 1 --2- 172.21.2.102:0/2779410563 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f00f404d6f0 0x7f00f404fbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:20:42.940 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:42.927 7f0109e1b700 1 --2- 172.21.2.102:0/2779410563 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f00f404a8f0 0x7f00f404cdf0 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 rx=0x7f0100000b80 tx=0x7f0100009e80).ready entity=mon.0 client_cookie=92007f1775302499 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:20:42.940 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:42.927 7f010961a700 1 -- 172.21.2.102:0/2779410563 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f010000c560 con 0x7f00f404a8f0 2021-02-10T11:20:42.940 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:42.927 7f010961a700 1 -- 172.21.2.102:0/2779410563 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f010000c7a0 con 0x7f00f404a8f0 2021-02-10T11:20:42.940 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:42.927 7f010961a700 1 -- 172.21.2.102:0/2779410563 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f010000cbf0 con 0x7f00f404a8f0 2021-02-10T11:20:42.940 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:42.927 7f011096e700 1 -- 172.21.2.102:0/2779410563 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f00f404a8f0 msgr2=0x7f00f404cdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:42.940 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:42.927 7f011096e700 1 --2- 172.21.2.102:0/2779410563 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f00f404a8f0 0x7f00f404cdf0 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 rx=0x7f0100000b80 tx=0x7f0100009e80).stop 2021-02-10T11:20:42.940 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:42.927 7f010ae1d700 1 -- 172.21.2.102:0/2779410563 reap_dead start 2021-02-10T11:20:42.940 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:42.928 7f011096e700 1 -- 172.21.2.102:0/2779410563 shutdown_connections 2021-02-10T11:20:42.941 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:42.928 7f011096e700 1 -- 172.21.2.102:0/2779410563 >> 172.21.2.102:0/2779410563 conn(0x7f010c0ce3e0 msgr2=0x7f010c0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:20:42.941 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:42.928 7f011096e700 1 -- 172.21.2.102:0/2779410563 shutdown_connections 2021-02-10T11:20:42.941 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:42.928 7f011096e700 1 -- 172.21.2.102:0/2779410563 wait complete. 2021-02-10T11:20:42.941 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:42.942 7f011096e700 1 Processor -- start 2021-02-10T11:20:42.941 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:42.942 7f011096e700 1 -- start start 2021-02-10T11:20:42.943 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:42.943 7f011096e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f010c0b7db0 0x7f010c0b8260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:42.943 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:42.943 7f011096e700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f010c0b8880 0x7f010c1a7450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:42.943 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:42.943 7f011096e700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f010c0b8c80 0x7f010c1a7a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:42.943 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:42.943 7f011096e700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f010c0b79b0 con 0x7f010c0b7db0 2021-02-10T11:20:42.943 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:42.943 7f011096e700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f010c1a8090 con 0x7f010c0b8880 2021-02-10T11:20:42.943 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:42.943 7f011096e700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f010c1a82b0 con 0x7f010c0b8c80 2021-02-10T11:20:42.943 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:42.943 7f010a61c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f010c0b7db0 0x7f010c0b8260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:42.943 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:42.943 7f0109e1b700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f010c0b8880 0x7f010c1a7450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:42.943 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:42.943 7f0109e1b700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f010c0b8880 0x7f010c1a7450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.113:3300/0 says I am v2:172.21.2.102:57190/0 (socket says 172.21.2.102:57190) 2021-02-10T11:20:42.943 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:42.943 7f0109e1b700 1 -- 172.21.2.102:0/4289063175 learned_addr learned my addr 172.21.2.102:0/4289063175 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:20:42.944 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:42.943 7f010ae1d700 1 --2- 172.21.2.102:0/4289063175 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f010c0b8c80 0x7f010c1a7a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:42.944 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:42.944 7f0109e1b700 1 -- 172.21.2.102:0/4289063175 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f010c0b8c80 msgr2=0x7f010c1a7a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:42.944 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:42.944 7f0109e1b700 1 --2- 172.21.2.102:0/4289063175 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f010c0b8c80 0x7f010c1a7a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:42.944 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:42.944 7f0109e1b700 1 -- 172.21.2.102:0/4289063175 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f010c0b7db0 msgr2=0x7f010c0b8260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:42.944 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:42.944 7f0109e1b700 1 --2- 172.21.2.102:0/4289063175 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f010c0b7db0 0x7f010c0b8260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:42.944 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:42.944 7f0109e1b700 1 -- 172.21.2.102:0/4289063175 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f010c190c10 con 0x7f010c0b8880 2021-02-10T11:20:42.944 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:42.944 7f010a61c700 1 --2- 172.21.2.102:0/4289063175 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f010c0b7db0 0x7f010c0b8260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:20:42.944 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:42.944 7f0109e1b700 1 --2- 172.21.2.102:0/4289063175 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f010c0b8880 0x7f010c1a7450 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 rx=0x7f01000095b0 tx=0x7f010000b990).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:20:42.944 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:42.944 7f00faffd700 1 -- 172.21.2.102:0/4289063175 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0100006910 con 0x7f010c0b8880 2021-02-10T11:20:42.944 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:42.944 7f00faffd700 1 -- 172.21.2.102:0/4289063175 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0100006030 con 0x7f010c0b8880 2021-02-10T11:20:42.944 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:42.944 7f011096e700 1 -- 172.21.2.102:0/4289063175 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f010c190c10 con 0x7f010c0b8880 2021-02-10T11:20:42.944 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:42.944 7f011096e700 1 -- 172.21.2.102:0/4289063175 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f010c0c1eb0 con 0x7f010c0b8880 2021-02-10T11:20:42.945 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:42.944 7f00faffd700 1 -- 172.21.2.102:0/4289063175 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f01000052e0 con 0x7f010c0b8880 2021-02-10T11:20:42.945 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:42.944 7f00faffd700 1 -- 172.21.2.102:0/4289063175 <== mon.1 v2:172.21.2.113:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f0100006210 con 0x7f010c0b8880 2021-02-10T11:20:42.945 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:42.944 7f00faffd700 1 --2- 172.21.2.102:0/4289063175 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f00f0018730 0x7f00f001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:42.945 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:42.945 7f00faffd700 1 -- 172.21.2.102:0/4289063175 <== mon.1 v2:172.21.2.113:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f0100032660 con 0x7f010c0b8880 2021-02-10T11:20:42.945 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:42.945 7f010a61c700 1 --2- 172.21.2.102:0/4289063175 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f00f0018730 0x7f00f001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:42.945 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:42.945 7f010a61c700 1 --2- 172.21.2.102:0/4289063175 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f00f0018730 0x7f00f001ac10 crc :-1 s=READY pgs=390 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:20:42.968 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:42.974 7f011096e700 1 -- 172.21.2.102:0/4289063175 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f010c105b30 con 0x7f010c0b8880 2021-02-10T11:20:42.971 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:42.977 7f00faffd700 1 -- 172.21.2.102:0/4289063175 <== mon.1 v2:172.21.2.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f0100006240 con 0x7f010c0b8880 2021-02-10T11:20:43.101 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:43.106 7f011096e700 1 -- 172.21.2.102:0/4289063175 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344367", "target": ["mgr", ""]}) v1 -- 0x7f010c0c1eb0 con 0x7f00f0018730 2021-02-10T11:20:43.112 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:43.118 7f00faffd700 1 -- 172.21.2.102:0/4289063175 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f00fc006a30 con 0x7f00f0018730 2021-02-10T11:20:43.118 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:43.123 7f011096e700 1 -- 172.21.2.102:0/4289063175 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f00f0018730 msgr2=0x7f00f001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:43.118 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:43.123 7f011096e700 1 --2- 172.21.2.102:0/4289063175 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f00f0018730 0x7f00f001ac10 crc :-1 s=READY pgs=390 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:43.118 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:43.123 7f011096e700 1 -- 172.21.2.102:0/4289063175 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f010c0b8880 msgr2=0x7f010c1a7450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:43.118 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:43.123 7f011096e700 1 --2- 172.21.2.102:0/4289063175 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f010c0b8880 0x7f010c1a7450 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 rx=0x7f01000095b0 tx=0x7f010000b990).stop 2021-02-10T11:20:43.118 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:43.123 7f011096e700 1 -- 172.21.2.102:0/4289063175 shutdown_connections 2021-02-10T11:20:43.118 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:43.123 7f011096e700 1 --2- 172.21.2.102:0/4289063175 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f010c0b8c80 0x7f010c1a7a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:43.118 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:43.123 7f011096e700 1 --2- 172.21.2.102:0/4289063175 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f00f0018730 0x7f00f001ac10 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:43.118 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:43.123 7f011096e700 1 --2- 172.21.2.102:0/4289063175 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f010c0b8880 0x7f010c1a7450 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:43.118 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:43.123 7f011096e700 1 --2- 172.21.2.102:0/4289063175 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f010c0b7db0 0x7f010c0b8260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:43.119 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:43.123 7f011096e700 1 -- 172.21.2.102:0/4289063175 >> 172.21.2.102:0/4289063175 conn(0x7f010c094e10 msgr2=0x7f010c0a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:20:43.119 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:43.123 7f011096e700 1 -- 172.21.2.102:0/4289063175 shutdown_connections 2021-02-10T11:20:43.119 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:43.123 7f011096e700 1 -- 172.21.2.102:0/4289063175 wait complete. 2021-02-10T11:20:43.127 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000344368 2021-02-10T11:20:43.559 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:43.547 7f1113185700 1 Processor -- start 2021-02-10T11:20:43.559 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:43.547 7f1113185700 1 -- start start 2021-02-10T11:20:43.559 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:43.548 7f1113185700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f110c1b0470 0x7f110c1b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:43.559 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:43.548 7f1113185700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f110c0f20e0 0x7f110c0f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:43.559 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:43.548 7f1113185700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f110c0f0b50 con 0x7f110c0ef610 2021-02-10T11:20:43.559 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:43.548 7f1113185700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f110c0f0820 con 0x7f110c0f20e0 2021-02-10T11:20:43.559 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:43.548 7f1113185700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f110c0f24e0 con 0x7f110c1b0470 2021-02-10T11:20:43.559 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:43.548 7f11117a2700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f110c0f20e0 0x7f110c0f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:43.559 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:43.548 7f11117a2700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f110c0f20e0 0x7f110c0f0200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:53766/0 (socket says 172.21.2.102:53766) 2021-02-10T11:20:43.560 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:43.548 7f11117a2700 1 -- 172.21.2.102:0/1811482530 learned_addr learned my addr 172.21.2.102:0/1811482530 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:20:43.560 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:43.548 7f1110fa1700 1 --2- 172.21.2.102:0/1811482530 >> v2:172.21.2.113:3300/0 conn(0x7f110c1b0470 0x7f110c1b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:43.560 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:43.549 7f1110fa1700 1 -- 172.21.2.102:0/1811482530 >> v2:172.21.2.102:3300/0 conn(0x7f110c0f20e0 msgr2=0x7f110c0f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:20:43.560 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:43.549 7f1110fa1700 1 --2- 172.21.2.102:0/1811482530 >> v2:172.21.2.102:3300/0 conn(0x7f110c0f20e0 0x7f110c0f0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:43.560 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:43.549 7f1110fa1700 1 -- 172.21.2.102:0/1811482530 >> v1:172.21.2.113:6789/0 conn(0x7f110c0ef610 legacy=0x7f110c0efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:43.560 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:43.549 7f1110fa1700 1 -- 172.21.2.102:0/1811482530 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f110c0e52e0 con 0x7f110c1b0470 2021-02-10T11:20:43.560 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:43.549 7f11117a2700 1 --2- 172.21.2.102:0/1811482530 >> v2:172.21.2.102:3300/0 conn(0x7f110c0f20e0 0x7f110c0f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:20:43.560 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:43.549 7f1110fa1700 1 --2- 172.21.2.102:0/1811482530 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f110c1b0470 0x7f110c1b0870 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 rx=0x7f10f4006650 tx=0x7f10f4009a80).ready entity=mon.1 client_cookie=12dfa5826657f43c server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:20:43.560 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:43.549 7f110b7fe700 1 -- 172.21.2.102:0/1811482530 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f10f400c0c0 con 0x7f110c1b0470 2021-02-10T11:20:43.560 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:43.549 7f110b7fe700 1 -- 172.21.2.102:0/1811482530 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10f400c340 con 0x7f110c1b0470 2021-02-10T11:20:43.560 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:43.549 7f110b7fe700 1 -- 172.21.2.102:0/1811482530 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f10f400c820 con 0x7f110c1b0470 2021-02-10T11:20:43.560 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:43.549 7f1113185700 1 -- 172.21.2.102:0/1811482530 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f110c1b0470 msgr2=0x7f110c1b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:43.561 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:43.549 7f1113185700 1 --2- 172.21.2.102:0/1811482530 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f110c1b0470 0x7f110c1b0870 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 rx=0x7f10f4006650 tx=0x7f10f4009a80).stop 2021-02-10T11:20:43.561 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:43.550 7f1113185700 1 -- 172.21.2.102:0/1811482530 shutdown_connections 2021-02-10T11:20:43.561 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:43.550 7f1113185700 1 --2- 172.21.2.102:0/1811482530 >> v2:172.21.2.102:3300/0 conn(0x7f110c0f20e0 0x7f110c0f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:43.561 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:43.550 7f1113185700 1 --2- 172.21.2.102:0/1811482530 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f110c1b0470 0x7f110c1b0870 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:43.561 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:43.550 7f1113185700 1 -- 172.21.2.102:0/1811482530 >> 172.21.2.102:0/1811482530 conn(0x7f110c0ce3e0 msgr2=0x7f110c0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:20:43.561 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:43.550 7f1113185700 1 -- 172.21.2.102:0/1811482530 shutdown_connections 2021-02-10T11:20:43.561 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:43.550 7f1113185700 1 -- 172.21.2.102:0/1811482530 wait complete. 2021-02-10T11:20:43.563 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:43.565 7f1113185700 1 Processor -- start 2021-02-10T11:20:43.563 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:43.565 7f1113185700 1 -- start start 2021-02-10T11:20:43.564 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:43.565 7f1113185700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f110c190920 0x7f110c0b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:43.564 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:43.565 7f1113185700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f110c0b8200 0x7f110c0b8600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:43.564 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:43.565 7f1113185700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f110c0c1d80 0x7f110c0c2180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:43.564 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:43.565 7f1113185700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f110c190da0 con 0x7f110c0b8200 2021-02-10T11:20:43.564 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:43.565 7f1113185700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f110c0b8c20 con 0x7f110c190920 2021-02-10T11:20:43.564 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:43.565 7f1113185700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f110c0b8e10 con 0x7f110c0c1d80 2021-02-10T11:20:43.564 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:43.565 7f110bfff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f110c0b8200 0x7f110c0b8600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:43.564 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:43.565 7f110bfff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f110c0b8200 0x7f110c0b8600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:53770/0 (socket says 172.21.2.102:53770) 2021-02-10T11:20:43.564 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:43.566 7f110bfff700 1 -- 172.21.2.102:0/2523874671 learned_addr learned my addr 172.21.2.102:0/2523874671 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:20:43.564 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:43.566 7f1110fa1700 1 --2- 172.21.2.102:0/2523874671 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f110c190920 0x7f110c0b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:43.564 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:43.566 7f11117a2700 1 --2- 172.21.2.102:0/2523874671 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f110c0c1d80 0x7f110c0c2180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:43.564 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:43.566 7f110bfff700 1 -- 172.21.2.102:0/2523874671 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f110c0c1d80 msgr2=0x7f110c0c2180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:43.565 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:43.566 7f110bfff700 1 --2- 172.21.2.102:0/2523874671 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f110c0c1d80 0x7f110c0c2180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:43.565 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:43.566 7f110bfff700 1 -- 172.21.2.102:0/2523874671 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f110c190920 msgr2=0x7f110c0b7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:43.565 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:43.566 7f110bfff700 1 --2- 172.21.2.102:0/2523874671 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f110c190920 0x7f110c0b7be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:43.565 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:43.566 7f110bfff700 1 -- 172.21.2.102:0/2523874671 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f110c0c27d0 con 0x7f110c0b8200 2021-02-10T11:20:43.565 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:43.566 7f110bfff700 1 --2- 172.21.2.102:0/2523874671 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f110c0b8200 0x7f110c0b8600 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 rx=0x7f10fc009940 tx=0x7f10fc00bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:20:43.565 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:43.566 7f11097fa700 1 -- 172.21.2.102:0/2523874671 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f10fc005ab0 con 0x7f110c0b8200 2021-02-10T11:20:43.565 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:43.566 7f11097fa700 1 -- 172.21.2.102:0/2523874671 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10fc005d30 con 0x7f110c0b8200 2021-02-10T11:20:43.565 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:43.566 7f1113185700 1 -- 172.21.2.102:0/2523874671 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f110c0c27d0 con 0x7f110c0b8200 2021-02-10T11:20:43.565 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:43.566 7f1113185700 1 -- 172.21.2.102:0/2523874671 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f110c0c2d60 con 0x7f110c0b8200 2021-02-10T11:20:43.565 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:43.566 7f11097fa700 1 -- 172.21.2.102:0/2523874671 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f10fc0062b0 con 0x7f110c0b8200 2021-02-10T11:20:43.565 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:43.567 7f11097fa700 1 -- 172.21.2.102:0/2523874671 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f10fc006030 con 0x7f110c0b8200 2021-02-10T11:20:43.565 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:43.567 7f11097fa700 1 --2- 172.21.2.102:0/2523874671 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f10f00186c0 0x7f10f001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:43.565 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:43.567 7f11097fa700 1 -- 172.21.2.102:0/2523874671 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f10fc032b80 con 0x7f110c0b8200 2021-02-10T11:20:43.566 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:43.567 7f1110fa1700 1 --2- 172.21.2.102:0/2523874671 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f10f00186c0 0x7f10f001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:43.566 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:43.567 7f1110fa1700 1 --2- 172.21.2.102:0/2523874671 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f10f00186c0 0x7f10f001abc0 crc :-1 s=READY pgs=391 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:20:43.592 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:43.598 7f1113185700 1 -- 172.21.2.102:0/2523874671 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f110c105ae0 con 0x7f110c0b8200 2021-02-10T11:20:43.594 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:43.600 7f11097fa700 1 -- 172.21.2.102:0/2523874671 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f10fc006570 con 0x7f110c0b8200 2021-02-10T11:20:43.728 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:43.734 7f1113185700 1 -- 172.21.2.102:0/2523874671 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344368", "target": ["mgr", ""]}) v1 -- 0x7f110c0c2d60 con 0x7f10f00186c0 2021-02-10T11:20:43.739 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:43.744 7f11097fa700 1 -- 172.21.2.102:0/2523874671 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f10f4009180 con 0x7f10f00186c0 2021-02-10T11:20:43.745 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:43.750 7f1113185700 1 -- 172.21.2.102:0/2523874671 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f10f00186c0 msgr2=0x7f10f001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:43.745 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:43.750 7f1113185700 1 --2- 172.21.2.102:0/2523874671 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f10f00186c0 0x7f10f001abc0 crc :-1 s=READY pgs=391 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:43.745 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:43.750 7f1113185700 1 -- 172.21.2.102:0/2523874671 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f110c0b8200 msgr2=0x7f110c0b8600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:43.745 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:43.750 7f1113185700 1 --2- 172.21.2.102:0/2523874671 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f110c0b8200 0x7f110c0b8600 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 rx=0x7f10fc009940 tx=0x7f10fc00bcf0).stop 2021-02-10T11:20:43.745 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:43.751 7f1113185700 1 -- 172.21.2.102:0/2523874671 shutdown_connections 2021-02-10T11:20:43.745 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:43.751 7f1113185700 1 --2- 172.21.2.102:0/2523874671 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f110c0c1d80 0x7f110c0c2180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:43.746 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:43.751 7f1113185700 1 --2- 172.21.2.102:0/2523874671 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f110c0b8200 0x7f110c0b8600 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:43.746 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:43.751 7f1113185700 1 --2- 172.21.2.102:0/2523874671 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f10f00186c0 0x7f10f001abc0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:43.746 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:43.751 7f1113185700 1 --2- 172.21.2.102:0/2523874671 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f110c190920 0x7f110c0b7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:43.746 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:43.751 7f1113185700 1 -- 172.21.2.102:0/2523874671 >> 172.21.2.102:0/2523874671 conn(0x7f110c094e10 msgr2=0x7f110c0a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:20:43.746 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:43.751 7f1113185700 1 -- 172.21.2.102:0/2523874671 shutdown_connections 2021-02-10T11:20:43.746 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:43.751 7f1113185700 1 -- 172.21.2.102:0/2523874671 wait complete. 2021-02-10T11:20:43.754 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000344369 2021-02-10T11:20:44.184 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:44.172 7fb9f539b700 1 Processor -- start 2021-02-10T11:20:44.184 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:44.172 7fb9f539b700 1 -- start start 2021-02-10T11:20:44.184 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:44.172 7fb9f539b700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb9f01b0470 0x7fb9f01b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:44.185 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:44.172 7fb9f539b700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fb9f00ef610 0x7fb9f00efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:44.185 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:44.172 7fb9f539b700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fb9f00f0030 0x7fb9f00f0430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:44.185 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:44.172 7fb9f539b700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9f00e5ea0 con 0x7fb9f00ef610 2021-02-10T11:20:44.185 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:44.173 7fb9f539b700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9f00f0b50 con 0x7fb9f00f0030 2021-02-10T11:20:44.185 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:44.173 7fb9f539b700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9f00e52e0 con 0x7fb9f01b0470 2021-02-10T11:20:44.185 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:44.173 7fb9ee87c700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fb9f00ef610 0x7fb9f00efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:44.185 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:44.173 7fb9ee87c700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fb9f00ef610 0x7fb9f00efa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:53776/0 (socket says 172.21.2.102:53776) 2021-02-10T11:20:44.185 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:44.173 7fb9ee87c700 1 -- 172.21.2.102:0/959489891 learned_addr learned my addr 172.21.2.102:0/959489891 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:20:44.185 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:44.173 7fb9ef87e700 1 --2- 172.21.2.102:0/959489891 >> v2:172.21.2.113:3300/0 conn(0x7fb9f00f0030 0x7fb9f00f0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:44.185 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:44.173 7fb9ef07d700 1 --2- 172.21.2.102:0/959489891 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb9f01b0470 0x7fb9f01b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:44.185 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:44.173 7fb9ee87c700 1 -- 172.21.2.102:0/959489891 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb9f01b0470 msgr2=0x7fb9f01b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:20:44.185 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:44.173 7fb9ee87c700 1 --2- 172.21.2.102:0/959489891 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb9f01b0470 0x7fb9f01b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:44.186 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:44.173 7fb9ee87c700 1 -- 172.21.2.102:0/959489891 >> v2:172.21.2.113:3300/0 conn(0x7fb9f00f0030 msgr2=0x7fb9f00f0430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:20:44.186 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:44.173 7fb9ee87c700 1 --2- 172.21.2.102:0/959489891 >> v2:172.21.2.113:3300/0 conn(0x7fb9f00f0030 0x7fb9f00f0430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:44.186 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:44.173 7fb9ee87c700 1 -- 172.21.2.102:0/959489891 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9f00f10d0 con 0x7fb9f00ef610 2021-02-10T11:20:44.186 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:44.173 7fb9ef07d700 1 --2- 172.21.2.102:0/959489891 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb9f01b0470 0x7fb9f01b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:20:44.186 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:44.173 7fb9ee87c700 1 --2- 172.21.2.102:0/959489891 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb9f00ef610 0x7fb9f00efa10 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 rx=0x7fb9e4006700 tx=0x7fb9e4009b80).ready entity=mon.0 client_cookie=6fc1d2a87cd4d916 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:20:44.186 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:44.174 7fb9ee07b700 1 -- 172.21.2.102:0/959489891 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb9e4006340 con 0x7fb9f00ef610 2021-02-10T11:20:44.186 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:44.174 7fb9ee07b700 1 -- 172.21.2.102:0/959489891 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9e400c0c0 con 0x7fb9f00ef610 2021-02-10T11:20:44.186 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:44.174 7fb9ee07b700 1 -- 172.21.2.102:0/959489891 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb9e400c5e0 con 0x7fb9f00ef610 2021-02-10T11:20:44.186 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:44.174 7fb9f539b700 1 -- 172.21.2.102:0/959489891 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb9f00ef610 msgr2=0x7fb9f00efa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:44.186 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:44.174 7fb9f539b700 1 --2- 172.21.2.102:0/959489891 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb9f00ef610 0x7fb9f00efa10 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 rx=0x7fb9e4006700 tx=0x7fb9e4009b80).stop 2021-02-10T11:20:44.186 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:44.175 7fb9f539b700 1 -- 172.21.2.102:0/959489891 shutdown_connections 2021-02-10T11:20:44.186 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:44.175 7fb9f539b700 1 --2- 172.21.2.102:0/959489891 >> v2:172.21.2.113:3300/0 conn(0x7fb9f00f0030 0x7fb9f00f0430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:44.187 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:44.175 7fb9f539b700 1 --2- 172.21.2.102:0/959489891 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb9f00ef610 0x7fb9f00efa10 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:44.187 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:44.175 7fb9f539b700 1 --2- 172.21.2.102:0/959489891 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb9f01b0470 0x7fb9f01b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:44.187 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:44.175 7fb9f539b700 1 -- 172.21.2.102:0/959489891 >> 172.21.2.102:0/959489891 conn(0x7fb9f00ce3e0 msgr2=0x7fb9f00e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:20:44.187 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:44.175 7fb9f539b700 1 -- 172.21.2.102:0/959489891 shutdown_connections 2021-02-10T11:20:44.187 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:44.175 7fb9f539b700 1 -- 172.21.2.102:0/959489891 wait complete. 2021-02-10T11:20:44.189 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:44.190 7fb9f539b700 1 Processor -- start 2021-02-10T11:20:44.189 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:44.190 7fb9f539b700 1 -- start start 2021-02-10T11:20:44.189 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:44.191 7fb9f539b700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb9f00b7db0 0x7fb9f00b8260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:44.189 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:44.191 7fb9f539b700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb9f00b8880 0x7fb9f01a7450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:44.189 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:44.191 7fb9f539b700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb9f00b8c80 0x7fb9f01a7a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:44.189 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:44.191 7fb9f539b700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9f00b79b0 con 0x7fb9f00b8880 2021-02-10T11:20:44.190 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:44.191 7fb9f539b700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9f01a8090 con 0x7fb9f00b7db0 2021-02-10T11:20:44.190 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:44.191 7fb9f539b700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9f01a82b0 con 0x7fb9f00b8c80 2021-02-10T11:20:44.190 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:44.191 7fb9ee87c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb9f00b8880 0x7fb9f01a7450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:44.190 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:44.191 7fb9ee87c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb9f00b8880 0x7fb9f01a7450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:53778/0 (socket says 172.21.2.102:53778) 2021-02-10T11:20:44.190 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:44.191 7fb9ee87c700 1 -- 172.21.2.102:0/974755077 learned_addr learned my addr 172.21.2.102:0/974755077 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:20:44.190 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:44.191 7fb9ef07d700 1 --2- 172.21.2.102:0/974755077 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb9f00b7db0 0x7fb9f00b8260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:44.190 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:44.191 7fb9ef87e700 1 --2- 172.21.2.102:0/974755077 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb9f00b8c80 0x7fb9f01a7a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:44.190 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:44.191 7fb9ee87c700 1 -- 172.21.2.102:0/974755077 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb9f00b8c80 msgr2=0x7fb9f01a7a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:44.190 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:44.191 7fb9ee87c700 1 --2- 172.21.2.102:0/974755077 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb9f00b8c80 0x7fb9f01a7a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:44.190 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:44.191 7fb9ee87c700 1 -- 172.21.2.102:0/974755077 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb9f00b7db0 msgr2=0x7fb9f00b8260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:44.190 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:44.191 7fb9ee87c700 1 --2- 172.21.2.102:0/974755077 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb9f00b7db0 0x7fb9f00b8260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:44.191 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:44.191 7fb9ee87c700 1 -- 172.21.2.102:0/974755077 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9f0190c10 con 0x7fb9f00b8880 2021-02-10T11:20:44.191 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:44.191 7fb9ee87c700 1 --2- 172.21.2.102:0/974755077 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb9f00b8880 0x7fb9f01a7450 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 rx=0x7fb9e400b600 tx=0x7fb9e400b9a0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:20:44.191 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:44.192 7fb9d7fff700 1 -- 172.21.2.102:0/974755077 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb9e4002030 con 0x7fb9f00b8880 2021-02-10T11:20:44.191 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:44.192 7fb9d7fff700 1 -- 172.21.2.102:0/974755077 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9e4002230 con 0x7fb9f00b8880 2021-02-10T11:20:44.191 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:44.192 7fb9d7fff700 1 -- 172.21.2.102:0/974755077 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb9e4002690 con 0x7fb9f00b8880 2021-02-10T11:20:44.191 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:44.192 7fb9f539b700 1 -- 172.21.2.102:0/974755077 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9f0190c10 con 0x7fb9f00b8880 2021-02-10T11:20:44.191 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:44.192 7fb9f539b700 1 -- 172.21.2.102:0/974755077 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9f00c1eb0 con 0x7fb9f00b8880 2021-02-10T11:20:44.191 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:44.192 7fb9d7fff700 1 -- 172.21.2.102:0/974755077 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fb9e4000c90 con 0x7fb9f00b8880 2021-02-10T11:20:44.191 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:44.192 7fb9d7fff700 1 --2- 172.21.2.102:0/974755077 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb9d00186c0 0x7fb9d001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:44.191 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:44.192 7fb9d7fff700 1 -- 172.21.2.102:0/974755077 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fb9e4032580 con 0x7fb9f00b8880 2021-02-10T11:20:44.192 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:44.192 7fb9ef07d700 1 --2- 172.21.2.102:0/974755077 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb9d00186c0 0x7fb9d001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:44.192 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:44.193 7fb9ef07d700 1 --2- 172.21.2.102:0/974755077 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb9d00186c0 0x7fb9d001abc0 crc :-1 s=READY pgs=392 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:20:44.217 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:44.223 7fb9f539b700 1 -- 172.21.2.102:0/974755077 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9f0105b30 con 0x7fb9f00b8880 2021-02-10T11:20:44.220 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:44.225 7fb9d7fff700 1 -- 172.21.2.102:0/974755077 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fb9e4000940 con 0x7fb9f00b8880 2021-02-10T11:20:44.349 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:44.354 7fb9f539b700 1 -- 172.21.2.102:0/974755077 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344369", "target": ["mgr", ""]}) v1 -- 0x7fb9f00c1eb0 con 0x7fb9d00186c0 2021-02-10T11:20:44.361 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:44.367 7fb9d7fff700 1 -- 172.21.2.102:0/974755077 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fb9e0006540 con 0x7fb9d00186c0 2021-02-10T11:20:44.366 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:44.371 7fb9f539b700 1 -- 172.21.2.102:0/974755077 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb9d00186c0 msgr2=0x7fb9d001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:44.366 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:44.371 7fb9f539b700 1 --2- 172.21.2.102:0/974755077 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb9d00186c0 0x7fb9d001abc0 crc :-1 s=READY pgs=392 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:44.366 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:44.371 7fb9f539b700 1 -- 172.21.2.102:0/974755077 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb9f00b8880 msgr2=0x7fb9f01a7450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:44.366 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:44.372 7fb9f539b700 1 --2- 172.21.2.102:0/974755077 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb9f00b8880 0x7fb9f01a7450 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 rx=0x7fb9e400b600 tx=0x7fb9e400b9a0).stop 2021-02-10T11:20:44.367 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:44.372 7fb9f539b700 1 -- 172.21.2.102:0/974755077 shutdown_connections 2021-02-10T11:20:44.367 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:44.372 7fb9f539b700 1 --2- 172.21.2.102:0/974755077 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb9f00b8c80 0x7fb9f01a7a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:44.367 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:44.372 7fb9f539b700 1 --2- 172.21.2.102:0/974755077 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb9f00b8880 0x7fb9f01a7450 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:44.367 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:44.372 7fb9f539b700 1 --2- 172.21.2.102:0/974755077 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb9d00186c0 0x7fb9d001abc0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:44.367 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:44.372 7fb9f539b700 1 --2- 172.21.2.102:0/974755077 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb9f00b7db0 0x7fb9f00b8260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:44.367 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:44.372 7fb9f539b700 1 -- 172.21.2.102:0/974755077 >> 172.21.2.102:0/974755077 conn(0x7fb9f0094e10 msgr2=0x7fb9f00a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:20:44.367 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:44.372 7fb9f539b700 1 -- 172.21.2.102:0/974755077 shutdown_connections 2021-02-10T11:20:44.367 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:44.372 7fb9f539b700 1 -- 172.21.2.102:0/974755077 wait complete. 2021-02-10T11:20:44.375 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000344370 2021-02-10T11:20:44.804 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:44.793 7fb306387700 1 Processor -- start 2021-02-10T11:20:44.804 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:44.793 7fb306387700 1 -- start start 2021-02-10T11:20:44.804 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:44.793 7fb306387700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fb3001b0470 0x7fb3001b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:44.804 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:44.793 7fb306387700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fb3000ef610 0x7fb3000efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:44.804 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:44.793 7fb306387700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3000f0b50 con 0x7fb3000f0030 2021-02-10T11:20:44.804 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:44.793 7fb306387700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3000e52e0 con 0x7fb3001b0470 2021-02-10T11:20:44.804 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:44.793 7fb306387700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3000e5490 con 0x7fb3000ef610 2021-02-10T11:20:44.805 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:44.793 7fb2fffff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fb3001b0470 0x7fb3001b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:44.805 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:44.794 7fb2fffff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fb3001b0470 0x7fb3001b0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:53786/0 (socket says 172.21.2.102:53786) 2021-02-10T11:20:44.805 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:44.794 7fb2fffff700 1 -- 172.21.2.102:0/422763373 learned_addr learned my addr 172.21.2.102:0/422763373 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:20:44.805 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:44.794 7fb2ff7fe700 1 --2- 172.21.2.102:0/422763373 >> v2:172.21.2.113:3300/0 conn(0x7fb3000ef610 0x7fb3000efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:44.805 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:44.794 7fb2fffff700 1 -- 172.21.2.102:0/422763373 >> v2:172.21.2.113:3300/0 conn(0x7fb3000ef610 msgr2=0x7fb3000efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:20:44.805 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:44.794 7fb2fffff700 1 --2- 172.21.2.102:0/422763373 >> v2:172.21.2.113:3300/0 conn(0x7fb3000ef610 0x7fb3000efa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:44.805 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:44.794 7fb2fffff700 1 -- 172.21.2.102:0/422763373 >> v1:172.21.2.113:6789/0 conn(0x7fb3000f0030 legacy=0x7fb3000f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:44.805 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:44.794 7fb2fffff700 1 -- 172.21.2.102:0/422763373 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3000f0430 con 0x7fb3001b0470 2021-02-10T11:20:44.805 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:44.794 7fb2fffff700 1 --2- 172.21.2.102:0/422763373 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb3001b0470 0x7fb3001b0870 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 rx=0x7fb2f0006700 tx=0x7fb2f0009b00).ready entity=mon.0 client_cookie=626b2d5a4b31d736 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:20:44.805 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:44.794 7fb2feffd700 1 -- 172.21.2.102:0/422763373 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb2f000c0c0 con 0x7fb3001b0470 2021-02-10T11:20:44.805 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:44.794 7fb2feffd700 1 -- 172.21.2.102:0/422763373 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2f000c340 con 0x7fb3001b0470 2021-02-10T11:20:44.805 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:44.794 7fb2feffd700 1 -- 172.21.2.102:0/422763373 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb2f000c820 con 0x7fb3001b0470 2021-02-10T11:20:44.805 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:44.794 7fb306387700 1 -- 172.21.2.102:0/422763373 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb3001b0470 msgr2=0x7fb3001b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:44.806 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:44.794 7fb306387700 1 --2- 172.21.2.102:0/422763373 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb3001b0470 0x7fb3001b0870 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 rx=0x7fb2f0006700 tx=0x7fb2f0009b00).stop 2021-02-10T11:20:44.806 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:44.795 7fb306387700 1 -- 172.21.2.102:0/422763373 shutdown_connections 2021-02-10T11:20:44.806 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:44.795 7fb306387700 1 --2- 172.21.2.102:0/422763373 >> v2:172.21.2.113:3300/0 conn(0x7fb3000ef610 0x7fb3000efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:44.806 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:44.795 7fb306387700 1 --2- 172.21.2.102:0/422763373 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb3001b0470 0x7fb3001b0870 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:44.806 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:44.795 7fb306387700 1 -- 172.21.2.102:0/422763373 >> 172.21.2.102:0/422763373 conn(0x7fb3000ce3e0 msgr2=0x7fb3000e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:20:44.806 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:44.795 7fb306387700 1 -- 172.21.2.102:0/422763373 shutdown_connections 2021-02-10T11:20:44.806 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:44.795 7fb306387700 1 -- 172.21.2.102:0/422763373 wait complete. 2021-02-10T11:20:44.807 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:44.810 7fb306387700 1 Processor -- start 2021-02-10T11:20:44.808 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:44.810 7fb306387700 1 -- start start 2021-02-10T11:20:44.808 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:44.810 7fb306387700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb3000a0420 0x7fb3000a0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:44.808 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:44.810 7fb306387700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb300189d00 0x7fb30018a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:44.808 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:44.810 7fb306387700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb30018a720 0x7fb3000b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:44.808 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:44.810 7fb306387700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb300189800 con 0x7fb3000a0420 2021-02-10T11:20:44.808 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:44.810 7fb306387700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb30018ab20 con 0x7fb300189d00 2021-02-10T11:20:44.808 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:44.810 7fb306387700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3000a1140 con 0x7fb30018a720 2021-02-10T11:20:44.808 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:44.811 7fb2fffff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb3000a0420 0x7fb3000a0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:44.808 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:44.811 7fb2fffff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb3000a0420 0x7fb3000a0820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:53792/0 (socket says 172.21.2.102:53792) 2021-02-10T11:20:44.808 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:44.811 7fb2fffff700 1 -- 172.21.2.102:0/3258495731 learned_addr learned my addr 172.21.2.102:0/3258495731 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:20:44.808 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:44.811 7fb2ff7fe700 1 --2- 172.21.2.102:0/3258495731 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb300189d00 0x7fb30018a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:44.808 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:44.811 7fb3049a4700 1 --2- 172.21.2.102:0/3258495731 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb30018a720 0x7fb3000b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:44.809 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:44.811 7fb2fffff700 1 -- 172.21.2.102:0/3258495731 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb30018a720 msgr2=0x7fb3000b7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:44.809 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:44.811 7fb2fffff700 1 --2- 172.21.2.102:0/3258495731 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb30018a720 0x7fb3000b7be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:44.809 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:44.811 7fb2fffff700 1 -- 172.21.2.102:0/3258495731 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb300189d00 msgr2=0x7fb30018a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:44.809 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:44.811 7fb2fffff700 1 --2- 172.21.2.102:0/3258495731 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb300189d00 0x7fb30018a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:44.809 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:44.811 7fb2fffff700 1 -- 172.21.2.102:0/3258495731 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3000b8200 con 0x7fb3000a0420 2021-02-10T11:20:44.809 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:44.811 7fb3049a4700 1 --2- 172.21.2.102:0/3258495731 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb30018a720 0x7fb3000b7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:20:44.809 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:44.811 7fb2fffff700 1 --2- 172.21.2.102:0/3258495731 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb3000a0420 0x7fb3000a0820 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 rx=0x7fb2f0006f20 tx=0x7fb2f000b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:20:44.809 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:44.811 7fb2fcff9700 1 -- 172.21.2.102:0/3258495731 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb2f000bd80 con 0x7fb3000a0420 2021-02-10T11:20:44.809 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:44.811 7fb2fcff9700 1 -- 172.21.2.102:0/3258495731 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2f0009100 con 0x7fb3000a0420 2021-02-10T11:20:44.810 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:44.811 7fb306387700 1 -- 172.21.2.102:0/3258495731 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3000b8200 con 0x7fb3000a0420 2021-02-10T11:20:44.810 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:44.811 7fb306387700 1 -- 172.21.2.102:0/3258495731 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3000b8200 con 0x7fb3000a0420 2021-02-10T11:20:44.810 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:44.811 7fb2fcff9700 1 -- 172.21.2.102:0/3258495731 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb2f0005580 con 0x7fb3000a0420 2021-02-10T11:20:44.810 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:44.811 7fb2fcff9700 1 -- 172.21.2.102:0/3258495731 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fb2f000a000 con 0x7fb3000a0420 2021-02-10T11:20:44.810 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:44.812 7fb2fcff9700 1 --2- 172.21.2.102:0/3258495731 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb2e0018730 0x7fb2e001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:44.810 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:44.812 7fb2fcff9700 1 -- 172.21.2.102:0/3258495731 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fb2f00325f0 con 0x7fb3000a0420 2021-02-10T11:20:44.810 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:44.812 7fb2ff7fe700 1 --2- 172.21.2.102:0/3258495731 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb2e0018730 0x7fb2e001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:44.810 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:44.812 7fb2ff7fe700 1 --2- 172.21.2.102:0/3258495731 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb2e0018730 0x7fb2e001ac10 crc :-1 s=READY pgs=393 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:20:44.837 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:44.843 7fb306387700 1 -- 172.21.2.102:0/3258495731 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb300105ae0 con 0x7fb3000a0420 2021-02-10T11:20:44.840 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:44.846 7fb2fcff9700 1 -- 172.21.2.102:0/3258495731 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fb2f001f4b0 con 0x7fb3000a0420 2021-02-10T11:20:44.967 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:44.973 7fb306387700 1 -- 172.21.2.102:0/3258495731 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344370", "target": ["mgr", ""]}) v1 -- 0x7fb30018ab20 con 0x7fb2e0018730 2021-02-10T11:20:44.979 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:44.985 7fb2fcff9700 1 -- 172.21.2.102:0/3258495731 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fb2f4006a70 con 0x7fb2e0018730 2021-02-10T11:20:44.984 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:44.989 7fb306387700 1 -- 172.21.2.102:0/3258495731 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb2e0018730 msgr2=0x7fb2e001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:44.984 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:44.989 7fb306387700 1 --2- 172.21.2.102:0/3258495731 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb2e0018730 0x7fb2e001ac10 crc :-1 s=READY pgs=393 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:44.984 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:44.989 7fb306387700 1 -- 172.21.2.102:0/3258495731 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb3000a0420 msgr2=0x7fb3000a0820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:44.984 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:44.989 7fb306387700 1 --2- 172.21.2.102:0/3258495731 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb3000a0420 0x7fb3000a0820 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 rx=0x7fb2f0006f20 tx=0x7fb2f000b990).stop 2021-02-10T11:20:44.984 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:44.990 7fb306387700 1 -- 172.21.2.102:0/3258495731 shutdown_connections 2021-02-10T11:20:44.985 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:44.990 7fb306387700 1 --2- 172.21.2.102:0/3258495731 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb30018a720 0x7fb3000b7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:44.985 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:44.990 7fb306387700 1 --2- 172.21.2.102:0/3258495731 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb2e0018730 0x7fb2e001ac10 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:44.985 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:44.990 7fb306387700 1 --2- 172.21.2.102:0/3258495731 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb300189d00 0x7fb30018a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:44.985 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:44.990 7fb306387700 1 --2- 172.21.2.102:0/3258495731 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb3000a0420 0x7fb3000a0820 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:44.985 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:44.990 7fb306387700 1 -- 172.21.2.102:0/3258495731 >> 172.21.2.102:0/3258495731 conn(0x7fb300094e10 msgr2=0x7fb3000a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:20:44.985 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:44.990 7fb306387700 1 -- 172.21.2.102:0/3258495731 shutdown_connections 2021-02-10T11:20:44.985 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:44.990 7fb306387700 1 -- 172.21.2.102:0/3258495731 wait complete. 2021-02-10T11:20:44.994 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000344371 2021-02-10T11:20:45.427 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:45.415 7f280a199700 1 Processor -- start 2021-02-10T11:20:45.427 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:45.415 7f280a199700 1 -- start start 2021-02-10T11:20:45.427 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:45.416 7f280a199700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f28041b0470 0x7f28041b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:45.427 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:45.416 7f280a199700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f28040ef610 0x7f28040efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:45.427 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:45.416 7f280a199700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28040f0b50 con 0x7f28040f0030 2021-02-10T11:20:45.427 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:45.416 7f280a199700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28040e52e0 con 0x7f28040ef610 2021-02-10T11:20:45.427 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:45.416 7f280a199700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28040e5490 con 0x7f28041b0470 2021-02-10T11:20:45.428 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:45.416 7f2802ffd700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f28040ef610 0x7f28040efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:45.428 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:45.416 7f2802ffd700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f28040ef610 0x7f28040efa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:53800/0 (socket says 172.21.2.102:53800) 2021-02-10T11:20:45.428 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:45.416 7f2802ffd700 1 -- 172.21.2.102:0/407435205 learned_addr learned my addr 172.21.2.102:0/407435205 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:20:45.428 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:45.416 7f28037fe700 1 --2- 172.21.2.102:0/407435205 >> v2:172.21.2.113:3300/0 conn(0x7f28041b0470 0x7f28041b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:45.428 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:45.416 7f2802ffd700 1 -- 172.21.2.102:0/407435205 >> v2:172.21.2.113:3300/0 conn(0x7f28041b0470 msgr2=0x7f28041b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:20:45.428 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:45.416 7f2802ffd700 1 --2- 172.21.2.102:0/407435205 >> v2:172.21.2.113:3300/0 conn(0x7f28041b0470 0x7f28041b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:45.428 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:45.416 7f2802ffd700 1 -- 172.21.2.102:0/407435205 >> v1:172.21.2.113:6789/0 conn(0x7f28040f0030 legacy=0x7f28040f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:45.428 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:45.416 7f2802ffd700 1 -- 172.21.2.102:0/407435205 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28040f0430 con 0x7f28040ef610 2021-02-10T11:20:45.428 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:45.416 7f2802ffd700 1 --2- 172.21.2.102:0/407435205 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f28040ef610 0x7f28040efa10 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 rx=0x7f27ec006700 tx=0x7f27ec009b80).ready entity=mon.0 client_cookie=15d3a6181d2ca2fa server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:20:45.428 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:45.417 7f28027fc700 1 -- 172.21.2.102:0/407435205 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f27ec0009b0 con 0x7f28040ef610 2021-02-10T11:20:45.428 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:45.417 7f28027fc700 1 -- 172.21.2.102:0/407435205 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27ec00c0c0 con 0x7f28040ef610 2021-02-10T11:20:45.428 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:45.417 7f28027fc700 1 -- 172.21.2.102:0/407435205 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f27ec00c5e0 con 0x7f28040ef610 2021-02-10T11:20:45.428 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:45.417 7f280a199700 1 -- 172.21.2.102:0/407435205 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f28040ef610 msgr2=0x7f28040efa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:45.429 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:45.417 7f280a199700 1 --2- 172.21.2.102:0/407435205 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f28040ef610 0x7f28040efa10 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 rx=0x7f27ec006700 tx=0x7f27ec009b80).stop 2021-02-10T11:20:45.429 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:45.418 7f280a199700 1 -- 172.21.2.102:0/407435205 shutdown_connections 2021-02-10T11:20:45.429 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:45.418 7f280a199700 1 --2- 172.21.2.102:0/407435205 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f28040ef610 0x7f28040efa10 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:45.429 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:45.418 7f280a199700 1 --2- 172.21.2.102:0/407435205 >> v2:172.21.2.113:3300/0 conn(0x7f28041b0470 0x7f28041b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:45.429 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:45.418 7f280a199700 1 -- 172.21.2.102:0/407435205 >> 172.21.2.102:0/407435205 conn(0x7f28040ce3e0 msgr2=0x7f28040e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:20:45.429 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:45.418 7f280a199700 1 -- 172.21.2.102:0/407435205 shutdown_connections 2021-02-10T11:20:45.429 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:45.418 7f280a199700 1 -- 172.21.2.102:0/407435205 wait complete. 2021-02-10T11:20:45.431 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:45.433 7f280a199700 1 Processor -- start 2021-02-10T11:20:45.431 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:45.433 7f280a199700 1 -- start start 2021-02-10T11:20:45.431 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:45.433 7f280a199700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2804190920 0x7f28040b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:45.431 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:45.433 7f280a199700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f28040b8200 0x7f28040b8600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:45.431 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:45.433 7f280a199700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f28040c1d80 0x7f28040c2180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:45.431 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:45.433 7f280a199700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2804190d20 con 0x7f28040c1d80 2021-02-10T11:20:45.431 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:45.433 7f280a199700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28040b8c20 con 0x7f28040b8200 2021-02-10T11:20:45.431 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:45.433 7f280a199700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28040c27a0 con 0x7f2804190920 2021-02-10T11:20:45.431 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:45.434 7f2803fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f28040c1d80 0x7f28040c2180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:45.431 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:45.434 7f2803fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f28040c1d80 0x7f28040c2180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:53808/0 (socket says 172.21.2.102:53808) 2021-02-10T11:20:45.431 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:45.434 7f2803fff700 1 -- 172.21.2.102:0/3895519188 learned_addr learned my addr 172.21.2.102:0/3895519188 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:20:45.431 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:45.434 7f28037fe700 1 --2- 172.21.2.102:0/3895519188 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2804190920 0x7f28040b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:45.432 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:45.434 7f2802ffd700 1 --2- 172.21.2.102:0/3895519188 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f28040b8200 0x7f28040b8600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:45.432 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:45.434 7f2803fff700 1 -- 172.21.2.102:0/3895519188 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2804190920 msgr2=0x7f28040b7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:45.432 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:45.434 7f2803fff700 1 --2- 172.21.2.102:0/3895519188 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2804190920 0x7f28040b7be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:45.432 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:45.434 7f2803fff700 1 -- 172.21.2.102:0/3895519188 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f28040b8200 msgr2=0x7f28040b8600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:45.432 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:45.434 7f2803fff700 1 --2- 172.21.2.102:0/3895519188 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f28040b8200 0x7f28040b8600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:45.432 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:45.434 7f2803fff700 1 -- 172.21.2.102:0/3895519188 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28040c29c0 con 0x7f28040c1d80 2021-02-10T11:20:45.432 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:45.434 7f2803fff700 1 --2- 172.21.2.102:0/3895519188 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f28040c1d80 0x7f28040c2180 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 rx=0x7f27fc009940 tx=0x7f27fc00bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:20:45.432 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:45.434 7f27fbfff700 1 -- 172.21.2.102:0/3895519188 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f27fc000ce0 con 0x7f28040c1d80 2021-02-10T11:20:45.432 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:45.435 7f27fbfff700 1 -- 172.21.2.102:0/3895519188 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27fc00c2f0 con 0x7f28040c1d80 2021-02-10T11:20:45.432 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:45.435 7f27fbfff700 1 -- 172.21.2.102:0/3895519188 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f27fc00c750 con 0x7f28040c1d80 2021-02-10T11:20:45.432 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:45.435 7f280a199700 1 -- 172.21.2.102:0/3895519188 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28040c29c0 con 0x7f28040c1d80 2021-02-10T11:20:45.432 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:45.435 7f280a199700 1 -- 172.21.2.102:0/3895519188 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28040c29c0 con 0x7f28040c1d80 2021-02-10T11:20:45.432 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:45.435 7f27fbfff700 1 -- 172.21.2.102:0/3895519188 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f27fc001220 con 0x7f28040c1d80 2021-02-10T11:20:45.433 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:45.435 7f27fbfff700 1 --2- 172.21.2.102:0/3895519188 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f27e8018730 0x7f27e801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:45.433 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:45.435 7f27fbfff700 1 -- 172.21.2.102:0/3895519188 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f27fc0329e0 con 0x7f28040c1d80 2021-02-10T11:20:45.433 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:45.435 7f28037fe700 1 --2- 172.21.2.102:0/3895519188 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f27e8018730 0x7f27e801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:45.433 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:45.436 7f28037fe700 1 --2- 172.21.2.102:0/3895519188 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f27e8018730 0x7f27e801ac10 crc :-1 s=READY pgs=394 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:20:45.459 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:45.465 7f280a199700 1 -- 172.21.2.102:0/3895519188 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2804105ae0 con 0x7f28040c1d80 2021-02-10T11:20:45.462 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:45.468 7f27fbfff700 1 -- 172.21.2.102:0/3895519188 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f27fc000f90 con 0x7f28040c1d80 2021-02-10T11:20:45.589 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:45.595 7f280a199700 1 -- 172.21.2.102:0/3895519188 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344371", "target": ["mgr", ""]}) v1 -- 0x7f2804105ae0 con 0x7f27e8018730 2021-02-10T11:20:45.602 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:45.608 7f27fbfff700 1 -- 172.21.2.102:0/3895519188 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f27f4006a70 con 0x7f27e8018730 2021-02-10T11:20:45.606 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:45.611 7f280a199700 1 -- 172.21.2.102:0/3895519188 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f27e8018730 msgr2=0x7f27e801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:45.606 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:45.611 7f280a199700 1 --2- 172.21.2.102:0/3895519188 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f27e8018730 0x7f27e801ac10 crc :-1 s=READY pgs=394 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:45.607 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:45.611 7f280a199700 1 -- 172.21.2.102:0/3895519188 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f28040c1d80 msgr2=0x7f28040c2180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:45.607 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:45.611 7f280a199700 1 --2- 172.21.2.102:0/3895519188 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f28040c1d80 0x7f28040c2180 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 rx=0x7f27fc009940 tx=0x7f27fc00bcf0).stop 2021-02-10T11:20:45.607 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:45.612 7f280a199700 1 -- 172.21.2.102:0/3895519188 shutdown_connections 2021-02-10T11:20:45.607 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:45.612 7f280a199700 1 --2- 172.21.2.102:0/3895519188 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f28040c1d80 0x7f28040c2180 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:45.607 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:45.612 7f280a199700 1 --2- 172.21.2.102:0/3895519188 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f27e8018730 0x7f27e801ac10 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:45.607 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:45.612 7f280a199700 1 --2- 172.21.2.102:0/3895519188 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f28040b8200 0x7f28040b8600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:45.607 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:45.612 7f280a199700 1 --2- 172.21.2.102:0/3895519188 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2804190920 0x7f28040b7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:45.607 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:45.612 7f280a199700 1 -- 172.21.2.102:0/3895519188 >> 172.21.2.102:0/3895519188 conn(0x7f2804094e10 msgr2=0x7f28040a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:20:45.607 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:45.612 7f280a199700 1 -- 172.21.2.102:0/3895519188 shutdown_connections 2021-02-10T11:20:45.607 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:45.612 7f280a199700 1 -- 172.21.2.102:0/3895519188 wait complete. 2021-02-10T11:20:45.616 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000344372 2021-02-10T11:20:46.049 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:46.038 7f163906c700 1 Processor -- start 2021-02-10T11:20:46.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:46.038 7f163906c700 1 -- start start 2021-02-10T11:20:46.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:46.038 7f163906c700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f16341b0470 0x7f16341b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:46.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:46.038 7f163906c700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f16340ef610 0x7f16340efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:46.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:46.038 7f163906c700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f16340f0030 0x7f16340f0430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:46.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:46.038 7f163906c700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16340e5ea0 con 0x7f16340ef610 2021-02-10T11:20:46.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:46.038 7f163906c700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16340f0b50 con 0x7f16341b0470 2021-02-10T11:20:46.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:46.038 7f163906c700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16340e52e0 con 0x7f16340f0030 2021-02-10T11:20:46.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:46.039 7f163259c700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f16340ef610 0x7f16340efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:46.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:46.039 7f163259c700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f16340ef610 0x7f16340efa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:53816/0 (socket says 172.21.2.102:53816) 2021-02-10T11:20:46.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:46.039 7f163259c700 1 -- 172.21.2.102:0/3683516621 learned_addr learned my addr 172.21.2.102:0/3683516621 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:20:46.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:46.039 7f1632d9d700 1 --2- 172.21.2.102:0/3683516621 >> v2:172.21.2.113:3300/0 conn(0x7f16341b0470 0x7f16341b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:46.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:46.039 7f163359e700 1 --2- 172.21.2.102:0/3683516621 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f16340f0030 0x7f16340f0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:46.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:46.039 7f163259c700 1 -- 172.21.2.102:0/3683516621 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f16340f0030 msgr2=0x7f16340f0430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:20:46.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:46.039 7f163259c700 1 --2- 172.21.2.102:0/3683516621 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f16340f0030 0x7f16340f0430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:46.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:46.039 7f163259c700 1 -- 172.21.2.102:0/3683516621 >> v2:172.21.2.113:3300/0 conn(0x7f16341b0470 msgr2=0x7f16341b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:20:46.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:46.039 7f163259c700 1 --2- 172.21.2.102:0/3683516621 >> v2:172.21.2.113:3300/0 conn(0x7f16341b0470 0x7f16341b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:46.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:46.039 7f163259c700 1 -- 172.21.2.102:0/3683516621 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16340f10d0 con 0x7f16340ef610 2021-02-10T11:20:46.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:46.039 7f163259c700 1 --2- 172.21.2.102:0/3683516621 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f16340ef610 0x7f16340efa10 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 rx=0x7f1628006700 tx=0x7f1628009b80).ready entity=mon.0 client_cookie=2eb513fd2992f157 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:20:46.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:46.039 7f1631d9b700 1 -- 172.21.2.102:0/3683516621 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f162800c0c0 con 0x7f16340ef610 2021-02-10T11:20:46.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:46.039 7f1631d9b700 1 -- 172.21.2.102:0/3683516621 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f162800c340 con 0x7f16340ef610 2021-02-10T11:20:46.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:46.039 7f1631d9b700 1 -- 172.21.2.102:0/3683516621 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f162800c820 con 0x7f16340ef610 2021-02-10T11:20:46.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:46.039 7f163906c700 1 -- 172.21.2.102:0/3683516621 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f16340ef610 msgr2=0x7f16340efa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:46.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:46.039 7f163906c700 1 --2- 172.21.2.102:0/3683516621 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f16340ef610 0x7f16340efa10 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 rx=0x7f1628006700 tx=0x7f1628009b80).stop 2021-02-10T11:20:46.052 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:46.040 7f163906c700 1 -- 172.21.2.102:0/3683516621 shutdown_connections 2021-02-10T11:20:46.052 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:46.040 7f163906c700 1 --2- 172.21.2.102:0/3683516621 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f16340f0030 0x7f16340f0430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:46.052 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:46.040 7f163906c700 1 --2- 172.21.2.102:0/3683516621 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f16340ef610 0x7f16340efa10 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:46.052 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:46.040 7f163906c700 1 --2- 172.21.2.102:0/3683516621 >> v2:172.21.2.113:3300/0 conn(0x7f16341b0470 0x7f16341b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:46.052 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:46.040 7f163906c700 1 -- 172.21.2.102:0/3683516621 >> 172.21.2.102:0/3683516621 conn(0x7f16340ce3e0 msgr2=0x7f16340e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:20:46.052 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:46.040 7f163906c700 1 -- 172.21.2.102:0/3683516621 shutdown_connections 2021-02-10T11:20:46.052 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:46.040 7f163906c700 1 -- 172.21.2.102:0/3683516621 wait complete. 2021-02-10T11:20:46.054 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:46.056 7f163906c700 1 Processor -- start 2021-02-10T11:20:46.054 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:46.056 7f163906c700 1 -- start start 2021-02-10T11:20:46.054 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:46.056 7f163906c700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f16340a6e70 0x7f16340965f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:46.054 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:46.056 7f163906c700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1634096c10 0x7f1634097010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:46.054 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:46.056 7f163906c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f16340c0ef0 0x7f16340c12f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:46.054 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:46.056 7f163906c700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16340a7270 con 0x7f16340c0ef0 2021-02-10T11:20:46.054 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:46.056 7f163906c700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1634190c50 con 0x7f16340a6e70 2021-02-10T11:20:46.054 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:46.056 7f163906c700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16340c1910 con 0x7f1634096c10 2021-02-10T11:20:46.054 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:46.056 7f163359e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f16340c0ef0 0x7f16340c12f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:46.054 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:46.056 7f163359e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f16340c0ef0 0x7f16340c12f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:53824/0 (socket says 172.21.2.102:53824) 2021-02-10T11:20:46.054 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:46.056 7f163359e700 1 -- 172.21.2.102:0/2730608661 learned_addr learned my addr 172.21.2.102:0/2730608661 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:20:46.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:46.056 7f1632d9d700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f16340a6e70 0x7f16340965f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:46.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:46.056 7f163259c700 1 --2- 172.21.2.102:0/2730608661 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1634096c10 0x7f1634097010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:46.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:46.057 7f163359e700 1 -- 172.21.2.102:0/2730608661 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1634096c10 msgr2=0x7f1634097010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:46.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:46.057 7f163359e700 1 --2- 172.21.2.102:0/2730608661 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1634096c10 0x7f1634097010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:46.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:46.057 7f163359e700 1 -- 172.21.2.102:0/2730608661 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f16340a6e70 msgr2=0x7f16340965f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:46.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:46.057 7f163359e700 1 --2- 172.21.2.102:0/2730608661 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f16340a6e70 0x7f16340965f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:46.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:46.057 7f163359e700 1 -- 172.21.2.102:0/2730608661 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16340c1b30 con 0x7f16340c0ef0 2021-02-10T11:20:46.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:46.057 7f163359e700 1 --2- 172.21.2.102:0/2730608661 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f16340c0ef0 0x7f16340c12f0 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 rx=0x7f162c00b970 tx=0x7f162c00bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:20:46.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:46.057 7f16237fe700 1 -- 172.21.2.102:0/2730608661 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f162c000f90 con 0x7f16340c0ef0 2021-02-10T11:20:46.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:46.057 7f16237fe700 1 -- 172.21.2.102:0/2730608661 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f162c006aa0 con 0x7f16340c0ef0 2021-02-10T11:20:46.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:46.057 7f163906c700 1 -- 172.21.2.102:0/2730608661 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16340c1b30 con 0x7f16340c0ef0 2021-02-10T11:20:46.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:46.057 7f16237fe700 1 -- 172.21.2.102:0/2730608661 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f162c0015f0 con 0x7f16340c0ef0 2021-02-10T11:20:46.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:46.057 7f163906c700 1 -- 172.21.2.102:0/2730608661 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16340c20c0 con 0x7f16340c0ef0 2021-02-10T11:20:46.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:46.057 7f16237fe700 1 -- 172.21.2.102:0/2730608661 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f162c0011f0 con 0x7f16340c0ef0 2021-02-10T11:20:46.056 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:46.057 7f16237fe700 1 --2- 172.21.2.102:0/2730608661 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1618018730 0x7f161801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:46.056 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:46.058 7f16237fe700 1 -- 172.21.2.102:0/2730608661 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f162c0329b0 con 0x7f16340c0ef0 2021-02-10T11:20:46.056 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:46.058 7f1632d9d700 1 --2- 172.21.2.102:0/2730608661 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1618018730 0x7f161801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:46.056 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:46.058 7f1632d9d700 1 --2- 172.21.2.102:0/2730608661 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1618018730 0x7f161801ac10 crc :-1 s=READY pgs=395 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:20:46.082 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:46.088 7f163906c700 1 -- 172.21.2.102:0/2730608661 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1634105b30 con 0x7f16340c0ef0 2021-02-10T11:20:46.085 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:46.090 7f16237fe700 1 -- 172.21.2.102:0/2730608661 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f162c000f40 con 0x7f16340c0ef0 2021-02-10T11:20:46.211 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:46.216 7f163906c700 1 -- 172.21.2.102:0/2730608661 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344372", "target": ["mgr", ""]}) v1 -- 0x7f16340c20c0 con 0x7f1618018730 2021-02-10T11:20:46.224 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:46.230 7f16237fe700 1 -- 172.21.2.102:0/2730608661 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f1624006a70 con 0x7f1618018730 2021-02-10T11:20:46.228 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:46.234 7f163906c700 1 -- 172.21.2.102:0/2730608661 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1618018730 msgr2=0x7f161801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:46.228 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:46.234 7f163906c700 1 --2- 172.21.2.102:0/2730608661 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1618018730 0x7f161801ac10 crc :-1 s=READY pgs=395 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:46.228 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:46.234 7f163906c700 1 -- 172.21.2.102:0/2730608661 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f16340c0ef0 msgr2=0x7f16340c12f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:46.229 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:46.234 7f163906c700 1 --2- 172.21.2.102:0/2730608661 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f16340c0ef0 0x7f16340c12f0 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 rx=0x7f162c00b970 tx=0x7f162c00bd10).stop 2021-02-10T11:20:46.229 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:46.234 7f163906c700 1 -- 172.21.2.102:0/2730608661 shutdown_connections 2021-02-10T11:20:46.229 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:46.234 7f163906c700 1 --2- 172.21.2.102:0/2730608661 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f16340c0ef0 0x7f16340c12f0 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:46.229 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:46.234 7f163906c700 1 --2- 172.21.2.102:0/2730608661 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1634096c10 0x7f1634097010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:46.229 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:46.234 7f163906c700 1 --2- 172.21.2.102:0/2730608661 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1618018730 0x7f161801ac10 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:46.229 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:46.234 7f163906c700 1 --2- 172.21.2.102:0/2730608661 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f16340a6e70 0x7f16340965f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:46.229 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:46.234 7f163906c700 1 -- 172.21.2.102:0/2730608661 >> 172.21.2.102:0/2730608661 conn(0x7f16340c40e0 msgr2=0x7f16340957c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:20:46.229 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:46.234 7f163906c700 1 -- 172.21.2.102:0/2730608661 shutdown_connections 2021-02-10T11:20:46.229 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:46.234 7f163906c700 1 -- 172.21.2.102:0/2730608661 wait complete. 2021-02-10T11:20:46.238 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000344373 2021-02-10T11:20:46.672 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:46.659 7f2eb9cab700 1 Processor -- start 2021-02-10T11:20:46.672 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:46.659 7f2eb9cab700 1 -- start start 2021-02-10T11:20:46.672 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:46.660 7f2eb9cab700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f2eb41b0870 0x7f2eb40f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:46.672 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:46.660 7f2eb9cab700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2eb41b0c70 con 0x7f2eb41b0470 2021-02-10T11:20:46.672 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:46.660 7f2eb9cab700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2eb40f0b50 con 0x7f2eb40efe00 2021-02-10T11:20:46.672 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:46.660 7f2eb9cab700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2eb40e5ea0 con 0x7f2eb41b0870 2021-02-10T11:20:46.672 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:46.660 7f2eb37fe700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f2eb41b0470 0x7f2eb40ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:46052/0 (socket says 172.21.2.102:46052) 2021-02-10T11:20:46.672 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:46.660 7f2eb37fe700 1 -- 172.21.2.102:0/1038590825 learned_addr learned my addr 172.21.2.102:0/1038590825 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:20:46.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:46.660 7f2eb2ffd700 1 --2- 172.21.2.102:0/1038590825 >> v2:172.21.2.113:3300/0 conn(0x7f2eb41b0870 0x7f2eb40f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:46.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:46.660 7f2eb27fc700 1 -- 172.21.2.102:0/1038590825 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4149273866 0 0) 0x7f2ea4001180 con 0x7f2eb41b0470 2021-02-10T11:20:46.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:46.660 7f2eb27fc700 1 -- 172.21.2.102:0/1038590825 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e9c002fe0 con 0x7f2eb41b0470 2021-02-10T11:20:46.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:46.661 7f2eb2ffd700 1 -- 172.21.2.102:0/1038590825 >> v1:172.21.2.113:6789/0 conn(0x7f2eb40efe00 legacy=0x7f2eb40f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:46.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:46.661 7f2eb2ffd700 1 -- 172.21.2.102:0/1038590825 >> v1:172.21.2.102:6789/0 conn(0x7f2eb41b0470 legacy=0x7f2eb40ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:46.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:46.661 7f2eb2ffd700 1 -- 172.21.2.102:0/1038590825 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2eb40e52e0 con 0x7f2eb41b0870 2021-02-10T11:20:46.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:46.661 7f2eb2ffd700 1 --2- 172.21.2.102:0/1038590825 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2eb41b0870 0x7f2eb40f20e0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 rx=0x7f2ea8006650 tx=0x7f2ea8009a80).ready entity=mon.1 client_cookie=9dfc6fd7d8364a09 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:20:46.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:46.661 7f2eb27fc700 1 -- 172.21.2.102:0/1038590825 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2ea800c0c0 con 0x7f2eb41b0870 2021-02-10T11:20:46.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:46.661 7f2eb27fc700 1 -- 172.21.2.102:0/1038590825 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ea800c340 con 0x7f2eb41b0870 2021-02-10T11:20:46.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:46.661 7f2eb27fc700 1 -- 172.21.2.102:0/1038590825 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2ea800c860 con 0x7f2eb41b0870 2021-02-10T11:20:46.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:46.661 7f2eb9cab700 1 -- 172.21.2.102:0/1038590825 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2eb41b0870 msgr2=0x7f2eb40f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:46.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:46.661 7f2eb9cab700 1 --2- 172.21.2.102:0/1038590825 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2eb41b0870 0x7f2eb40f20e0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 rx=0x7f2ea8006650 tx=0x7f2ea8009a80).stop 2021-02-10T11:20:46.674 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:46.662 7f2eb9cab700 1 -- 172.21.2.102:0/1038590825 shutdown_connections 2021-02-10T11:20:46.674 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:46.662 7f2eb9cab700 1 --2- 172.21.2.102:0/1038590825 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2eb41b0870 0x7f2eb40f20e0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:46.674 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:46.662 7f2eb9cab700 1 -- 172.21.2.102:0/1038590825 >> 172.21.2.102:0/1038590825 conn(0x7f2eb40ce3e0 msgr2=0x7f2eb40e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:20:46.674 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:46.662 7f2eb9cab700 1 -- 172.21.2.102:0/1038590825 shutdown_connections 2021-02-10T11:20:46.674 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:46.662 7f2eb9cab700 1 -- 172.21.2.102:0/1038590825 wait complete. 2021-02-10T11:20:46.676 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:46.678 7f2eb9cab700 1 Processor -- start 2021-02-10T11:20:46.676 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:46.678 7f2eb9cab700 1 -- start start 2021-02-10T11:20:46.676 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:46.678 7f2eb9cab700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2eb40c36a0 0x7f2eb40c2b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:46.676 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:46.678 7f2eb9cab700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2eb40c3190 0x7f2eb40b1d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:46.676 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:46.678 7f2eb9cab700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2eb40b2370 0x7f2eb40b2d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:46.676 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:46.678 7f2eb9cab700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2eb40b27c0 con 0x7f2eb40c3190 2021-02-10T11:20:46.676 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:46.678 7f2eb9cab700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2eb40b3360 con 0x7f2eb40b2370 2021-02-10T11:20:46.676 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:46.678 7f2eb9cab700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2eb40b3540 con 0x7f2eb40c36a0 2021-02-10T11:20:46.676 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:46.678 7f2eb2ffd700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2eb40c3190 0x7f2eb40b1d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:46.676 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:46.679 7f2eb2ffd700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2eb40c3190 0x7f2eb40b1d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:53836/0 (socket says 172.21.2.102:53836) 2021-02-10T11:20:46.676 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:46.679 7f2eb2ffd700 1 -- 172.21.2.102:0/3533399027 learned_addr learned my addr 172.21.2.102:0/3533399027 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:20:46.677 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:46.679 7f2eb3fff700 1 --2- 172.21.2.102:0/3533399027 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2eb40b2370 0x7f2eb40b2d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:46.677 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:46.679 7f2eb37fe700 1 --2- 172.21.2.102:0/3533399027 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2eb40c36a0 0x7f2eb40c2b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:46.677 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:46.679 7f2eb3fff700 1 -- 172.21.2.102:0/3533399027 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2eb40c36a0 msgr2=0x7f2eb40c2b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:46.677 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:46.679 7f2eb3fff700 1 --2- 172.21.2.102:0/3533399027 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2eb40c36a0 0x7f2eb40c2b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:46.677 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:46.679 7f2eb3fff700 1 -- 172.21.2.102:0/3533399027 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2eb40c3190 msgr2=0x7f2eb40b1d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:46.677 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:46.679 7f2eb3fff700 1 --2- 172.21.2.102:0/3533399027 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2eb40c3190 0x7f2eb40b1d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:46.677 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:46.679 7f2eb3fff700 1 -- 172.21.2.102:0/3533399027 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2eb40b3760 con 0x7f2eb40b2370 2021-02-10T11:20:46.677 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:46.679 7f2eb2ffd700 1 --2- 172.21.2.102:0/3533399027 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2eb40c3190 0x7f2eb40b1d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:20:46.677 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:46.679 7f2eb3fff700 1 --2- 172.21.2.102:0/3533399027 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2eb40b2370 0x7f2eb40b2d40 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7f2eac00b890 tx=0x7f2eac00bc30).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:20:46.677 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:46.679 7f2e9bfff700 1 -- 172.21.2.102:0/3533399027 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2eac0066e0 con 0x7f2eb40b2370 2021-02-10T11:20:46.677 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:46.679 7f2e9bfff700 1 -- 172.21.2.102:0/3533399027 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2eac0011a0 con 0x7f2eb40b2370 2021-02-10T11:20:46.677 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:46.679 7f2eb9cab700 1 -- 172.21.2.102:0/3533399027 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2eb40b3760 con 0x7f2eb40b2370 2021-02-10T11:20:46.678 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:46.679 7f2e9bfff700 1 -- 172.21.2.102:0/3533399027 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2eac001570 con 0x7f2eb40b2370 2021-02-10T11:20:46.678 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:46.679 7f2eb9cab700 1 -- 172.21.2.102:0/3533399027 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2eb40b3760 con 0x7f2eb40b2370 2021-02-10T11:20:46.678 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:46.680 7f2e9bfff700 1 -- 172.21.2.102:0/3533399027 <== mon.1 v2:172.21.2.113:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f2eac001380 con 0x7f2eb40b2370 2021-02-10T11:20:46.678 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:46.680 7f2e9bfff700 1 --2- 172.21.2.102:0/3533399027 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2e940186c0 0x7f2e9401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:46.678 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:46.680 7f2e9bfff700 1 -- 172.21.2.102:0/3533399027 <== mon.1 v2:172.21.2.113:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f2eac032710 con 0x7f2eb40b2370 2021-02-10T11:20:46.678 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:46.680 7f2eb37fe700 1 --2- 172.21.2.102:0/3533399027 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2e940186c0 0x7f2e9401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:46.678 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:46.680 7f2eb37fe700 1 --2- 172.21.2.102:0/3533399027 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2e940186c0 0x7f2e9401abc0 crc :-1 s=READY pgs=396 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:20:46.704 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:46.710 7f2eb9cab700 1 -- 172.21.2.102:0/3533399027 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2eb40edd10 con 0x7f2eb40b2370 2021-02-10T11:20:46.706 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:46.712 7f2e9bfff700 1 -- 172.21.2.102:0/3533399027 <== mon.1 v2:172.21.2.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f2eac00bd70 con 0x7f2eb40b2370 2021-02-10T11:20:46.835 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:46.840 7f2eb9cab700 1 -- 172.21.2.102:0/3533399027 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344373", "target": ["mgr", ""]}) v1 -- 0x7f2eb40edd10 con 0x7f2e940186c0 2021-02-10T11:20:46.847 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:46.852 7f2e9bfff700 1 -- 172.21.2.102:0/3533399027 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f2ea4000ce0 con 0x7f2e940186c0 2021-02-10T11:20:46.851 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:46.857 7f2eb9cab700 1 -- 172.21.2.102:0/3533399027 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2e940186c0 msgr2=0x7f2e9401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:46.852 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:46.857 7f2eb9cab700 1 --2- 172.21.2.102:0/3533399027 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2e940186c0 0x7f2e9401abc0 crc :-1 s=READY pgs=396 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:46.852 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:46.857 7f2eb9cab700 1 -- 172.21.2.102:0/3533399027 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2eb40b2370 msgr2=0x7f2eb40b2d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:46.852 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:46.857 7f2eb9cab700 1 --2- 172.21.2.102:0/3533399027 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2eb40b2370 0x7f2eb40b2d40 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7f2eac00b890 tx=0x7f2eac00bc30).stop 2021-02-10T11:20:46.852 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:46.857 7f2eb9cab700 1 -- 172.21.2.102:0/3533399027 shutdown_connections 2021-02-10T11:20:46.852 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:46.857 7f2eb9cab700 1 --2- 172.21.2.102:0/3533399027 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2e940186c0 0x7f2e9401abc0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:46.852 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:46.857 7f2eb9cab700 1 --2- 172.21.2.102:0/3533399027 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2eb40b2370 0x7f2eb40b2d40 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:46.852 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:46.857 7f2eb9cab700 1 --2- 172.21.2.102:0/3533399027 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2eb40c3190 0x7f2eb40b1d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:46.852 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:46.857 7f2eb9cab700 1 --2- 172.21.2.102:0/3533399027 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2eb40c36a0 0x7f2eb40c2b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:46.852 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:46.857 7f2eb9cab700 1 -- 172.21.2.102:0/3533399027 >> 172.21.2.102:0/3533399027 conn(0x7f2eb409f630 msgr2=0x7f2eb40a0130 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:20:46.853 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:46.858 7f2eb9cab700 1 -- 172.21.2.102:0/3533399027 shutdown_connections 2021-02-10T11:20:46.853 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:46.858 7f2eb9cab700 1 -- 172.21.2.102:0/3533399027 wait complete. 2021-02-10T11:20:46.861 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000344374 2021-02-10T11:20:47.292 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:47.279 7fc53677a700 1 Processor -- start 2021-02-10T11:20:47.292 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:47.279 7fc53677a700 1 -- start start 2021-02-10T11:20:47.292 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:47.280 7fc53677a700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc5301b0470 0x7fc5301b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:47.292 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:47.280 7fc53677a700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fc5300f20e0 0x7fc5300f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:47.292 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:47.280 7fc53677a700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5300f0b50 con 0x7fc5300ef610 2021-02-10T11:20:47.292 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:47.280 7fc53677a700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5300f0820 con 0x7fc5300f20e0 2021-02-10T11:20:47.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:47.280 7fc53677a700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5300f24e0 con 0x7fc5301b0470 2021-02-10T11:20:47.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:47.280 7fc52f7fe700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fc5300ef610 0x7fc5300efa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:46068/0 (socket says 172.21.2.102:46068) 2021-02-10T11:20:47.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:47.280 7fc52f7fe700 1 -- 172.21.2.102:0/1180788851 learned_addr learned my addr 172.21.2.102:0/1180788851 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:20:47.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:47.280 7fc52effd700 1 -- 172.21.2.102:0/1180788851 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 336761306 0 0) 0x7fc5240009b0 con 0x7fc5300ef610 2021-02-10T11:20:47.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:47.280 7fc52effd700 1 -- 172.21.2.102:0/1180788851 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc518002fe0 con 0x7fc5300ef610 2021-02-10T11:20:47.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:47.280 7fc534d97700 1 --2- 172.21.2.102:0/1180788851 >> v2:172.21.2.113:3300/0 conn(0x7fc5300f20e0 0x7fc5300f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:47.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:47.280 7fc52ffff700 1 --2- 172.21.2.102:0/1180788851 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc5301b0470 0x7fc5301b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:47.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:47.281 7fc52effd700 1 -- 172.21.2.102:0/1180788851 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3567566751 0 0) 0x7fc524001010 con 0x7fc5300ef610 2021-02-10T11:20:47.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:47.281 7fc52effd700 1 -- 172.21.2.102:0/1180788851 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc5301b0470 msgr2=0x7fc5301b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:20:47.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:47.281 7fc52effd700 1 --2- 172.21.2.102:0/1180788851 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc5301b0470 0x7fc5301b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:47.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:47.281 7fc52effd700 1 -- 172.21.2.102:0/1180788851 >> v2:172.21.2.113:3300/0 conn(0x7fc5300f20e0 msgr2=0x7fc5300f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:20:47.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:47.281 7fc52effd700 1 --2- 172.21.2.102:0/1180788851 >> v2:172.21.2.113:3300/0 conn(0x7fc5300f20e0 0x7fc5300f0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:47.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:47.281 7fc52effd700 1 -- 172.21.2.102:0/1180788851 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5300e52e0 con 0x7fc5300ef610 2021-02-10T11:20:47.294 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:47.281 7fc52effd700 1 -- 172.21.2.102:0/1180788851 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7fc524001590 con 0x7fc5300ef610 2021-02-10T11:20:47.294 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:47.281 7fc52ffff700 1 --2- 172.21.2.102:0/1180788851 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc5301b0470 0x7fc5301b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:20:47.294 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:47.281 7fc52effd700 1 -- 172.21.2.102:0/1180788851 >> v1:172.21.2.102:6789/0 conn(0x7fc5300ef610 legacy=0x7fc5300efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:47.294 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:47.281 7fc52effd700 1 --2- 172.21.2.102:0/1180788851 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc518003a60 0x7fc518005ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:47.294 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:47.281 7fc52effd700 1 --2- 172.21.2.102:0/1180788851 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc518006770 0x7fc518008c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:47.294 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:47.281 7fc52effd700 1 --2- 172.21.2.102:0/1180788851 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc51804d5a0 0x7fc51804fa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:47.294 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:47.281 7fc52effd700 1 -- 172.21.2.102:0/1180788851 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5180502e0 con 0x7fc518006770 2021-02-10T11:20:47.294 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:47.281 7fc52effd700 1 -- 172.21.2.102:0/1180788851 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc518050500 con 0x7fc518003a60 2021-02-10T11:20:47.294 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:47.281 7fc52effd700 1 -- 172.21.2.102:0/1180788851 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc518050720 con 0x7fc51804d5a0 2021-02-10T11:20:47.294 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:47.281 7fc52ffff700 1 --2- 172.21.2.102:0/1180788851 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc518003a60 0x7fc518005ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:47.294 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:47.281 7fc534d97700 1 --2- 172.21.2.102:0/1180788851 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc51804d5a0 0x7fc51804fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:47.294 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:47.281 7fc52f7fe700 1 --2- 172.21.2.102:0/1180788851 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc518006770 0x7fc518008c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:47.294 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:47.281 7fc52ffff700 1 -- 172.21.2.102:0/1180788851 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc51804d5a0 msgr2=0x7fc51804fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:20:47.295 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:47.281 7fc52ffff700 1 --2- 172.21.2.102:0/1180788851 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc51804d5a0 0x7fc51804fa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:47.295 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:47.281 7fc52ffff700 1 -- 172.21.2.102:0/1180788851 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc518006770 msgr2=0x7fc518008c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:20:47.295 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:47.281 7fc52ffff700 1 --2- 172.21.2.102:0/1180788851 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc518006770 0x7fc518008c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:47.295 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:47.281 7fc52ffff700 1 -- 172.21.2.102:0/1180788851 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5180509c0 con 0x7fc518003a60 2021-02-10T11:20:47.295 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:47.281 7fc52f7fe700 1 --2- 172.21.2.102:0/1180788851 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc518006770 0x7fc518008c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:20:47.295 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:47.282 7fc52ffff700 1 --2- 172.21.2.102:0/1180788851 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc518003a60 0x7fc518005ef0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 rx=0x7fc520002dc0 tx=0x7fc520009a80).ready entity=mon.1 client_cookie=eee5246c4fe6e7e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:20:47.295 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:47.282 7fc52effd700 1 -- 172.21.2.102:0/1180788851 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc5200009b0 con 0x7fc518003a60 2021-02-10T11:20:47.295 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:47.282 7fc52effd700 1 -- 172.21.2.102:0/1180788851 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc520006360 con 0x7fc518003a60 2021-02-10T11:20:47.295 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:47.282 7fc52effd700 1 -- 172.21.2.102:0/1180788851 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc520006b40 con 0x7fc518003a60 2021-02-10T11:20:47.295 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:47.282 7fc53677a700 1 -- 172.21.2.102:0/1180788851 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc518003a60 msgr2=0x7fc518005ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:47.295 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:47.282 7fc53677a700 1 --2- 172.21.2.102:0/1180788851 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc518003a60 0x7fc518005ef0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 rx=0x7fc520002dc0 tx=0x7fc520009a80).stop 2021-02-10T11:20:47.295 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:47.282 7fc534d97700 1 -- 172.21.2.102:0/1180788851 reap_dead start 2021-02-10T11:20:47.295 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:47.282 7fc53677a700 1 -- 172.21.2.102:0/1180788851 shutdown_connections 2021-02-10T11:20:47.296 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:47.282 7fc53677a700 1 -- 172.21.2.102:0/1180788851 >> 172.21.2.102:0/1180788851 conn(0x7fc5300ce3e0 msgr2=0x7fc5300e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:20:47.296 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:47.283 7fc53677a700 1 -- 172.21.2.102:0/1180788851 shutdown_connections 2021-02-10T11:20:47.296 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:47.283 7fc53677a700 1 -- 172.21.2.102:0/1180788851 wait complete. 2021-02-10T11:20:47.296 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:47.297 7fc53677a700 1 Processor -- start 2021-02-10T11:20:47.296 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:47.298 7fc53677a700 1 -- start start 2021-02-10T11:20:47.297 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:47.298 7fc53677a700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc53009b930 0x7fc5300b7c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:47.297 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:47.298 7fc53677a700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc5300b8410 0x7fc5300b8810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:47.297 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:47.298 7fc53677a700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc5300c1f80 0x7fc5300c2380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:47.297 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:47.298 7fc53677a700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5300c1a00 con 0x7fc5300b8410 2021-02-10T11:20:47.298 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:47.298 7fc53677a700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5300b8e30 con 0x7fc5300c1f80 2021-02-10T11:20:47.298 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:47.298 7fc53677a700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5300b9050 con 0x7fc53009b930 2021-02-10T11:20:47.298 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:47.298 7fc52f7fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc5300b8410 0x7fc5300b8810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:47.298 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:47.298 7fc534d97700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc5300c1f80 0x7fc5300c2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:47.298 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:47.298 7fc52f7fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc5300b8410 0x7fc5300b8810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:53854/0 (socket says 172.21.2.102:53854) 2021-02-10T11:20:47.298 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:47.298 7fc52f7fe700 1 -- 172.21.2.102:0/1488093296 learned_addr learned my addr 172.21.2.102:0/1488093296 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:20:47.298 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:47.298 7fc52ffff700 1 --2- 172.21.2.102:0/1488093296 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc53009b930 0x7fc5300b7c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:47.298 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:47.299 7fc52f7fe700 1 -- 172.21.2.102:0/1488093296 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc53009b930 msgr2=0x7fc5300b7c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:47.298 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:47.299 7fc52f7fe700 1 --2- 172.21.2.102:0/1488093296 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc53009b930 0x7fc5300b7c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:47.298 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:47.299 7fc52f7fe700 1 -- 172.21.2.102:0/1488093296 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc5300c1f80 msgr2=0x7fc5300c2380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:47.298 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:47.299 7fc52f7fe700 1 --2- 172.21.2.102:0/1488093296 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc5300c1f80 0x7fc5300c2380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:47.298 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:47.299 7fc52f7fe700 1 -- 172.21.2.102:0/1488093296 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5300c29a0 con 0x7fc5300b8410 2021-02-10T11:20:47.298 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:47.299 7fc534d97700 1 --2- 172.21.2.102:0/1488093296 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc5300c1f80 0x7fc5300c2380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:20:47.299 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:47.299 7fc52f7fe700 1 --2- 172.21.2.102:0/1488093296 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc5300b8410 0x7fc5300b8810 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 rx=0x7fc524001c40 tx=0x7fc52400a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:20:47.299 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:47.299 7fc52cff9700 1 -- 172.21.2.102:0/1488093296 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc52400acc0 con 0x7fc5300b8410 2021-02-10T11:20:47.299 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:47.299 7fc52cff9700 1 -- 172.21.2.102:0/1488093296 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5240012f0 con 0x7fc5300b8410 2021-02-10T11:20:47.299 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:47.299 7fc52cff9700 1 -- 172.21.2.102:0/1488093296 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc5240043c0 con 0x7fc5300b8410 2021-02-10T11:20:47.299 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:47.299 7fc53677a700 1 -- 172.21.2.102:0/1488093296 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5300c29a0 con 0x7fc5300b8410 2021-02-10T11:20:47.299 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:47.299 7fc53677a700 1 -- 172.21.2.102:0/1488093296 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5300c29a0 con 0x7fc5300b8410 2021-02-10T11:20:47.299 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:47.300 7fc52cff9700 1 -- 172.21.2.102:0/1488093296 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fc524004140 con 0x7fc5300b8410 2021-02-10T11:20:47.299 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:47.300 7fc52cff9700 1 --2- 172.21.2.102:0/1488093296 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc510018730 0x7fc51001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:47.299 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:47.300 7fc52cff9700 1 -- 172.21.2.102:0/1488093296 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fc524032610 con 0x7fc5300b8410 2021-02-10T11:20:47.299 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:47.300 7fc52ffff700 1 --2- 172.21.2.102:0/1488093296 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc510018730 0x7fc51001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:47.299 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:47.300 7fc52ffff700 1 --2- 172.21.2.102:0/1488093296 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc510018730 0x7fc51001ac10 crc :-1 s=READY pgs=397 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:20:47.324 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:47.330 7fc53677a700 1 -- 172.21.2.102:0/1488093296 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc530105b30 con 0x7fc5300b8410 2021-02-10T11:20:47.327 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:47.332 7fc52cff9700 1 -- 172.21.2.102:0/1488093296 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fc524000940 con 0x7fc5300b8410 2021-02-10T11:20:47.455 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:47.460 7fc53677a700 1 -- 172.21.2.102:0/1488093296 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344374", "target": ["mgr", ""]}) v1 -- 0x7fc530105b30 con 0x7fc510018730 2021-02-10T11:20:47.467 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:47.473 7fc52cff9700 1 -- 172.21.2.102:0/1488093296 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fc520006580 con 0x7fc510018730 2021-02-10T11:20:47.472 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:47.477 7fc53677a700 1 -- 172.21.2.102:0/1488093296 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc510018730 msgr2=0x7fc51001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:47.472 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:47.477 7fc53677a700 1 --2- 172.21.2.102:0/1488093296 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc510018730 0x7fc51001ac10 crc :-1 s=READY pgs=397 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:47.472 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:47.477 7fc53677a700 1 -- 172.21.2.102:0/1488093296 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc5300b8410 msgr2=0x7fc5300b8810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:47.473 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:47.477 7fc53677a700 1 --2- 172.21.2.102:0/1488093296 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc5300b8410 0x7fc5300b8810 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 rx=0x7fc524001c40 tx=0x7fc52400a8d0).stop 2021-02-10T11:20:47.473 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:47.477 7fc53677a700 1 -- 172.21.2.102:0/1488093296 shutdown_connections 2021-02-10T11:20:47.473 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:47.477 7fc53677a700 1 --2- 172.21.2.102:0/1488093296 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc510018730 0x7fc51001ac10 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:47.473 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:47.477 7fc53677a700 1 --2- 172.21.2.102:0/1488093296 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc5300c1f80 0x7fc5300c2380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:47.473 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:47.477 7fc53677a700 1 --2- 172.21.2.102:0/1488093296 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc5300b8410 0x7fc5300b8810 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:47.474 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:47.477 7fc53677a700 1 --2- 172.21.2.102:0/1488093296 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc53009b930 0x7fc5300b7c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:47.474 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:47.477 7fc53677a700 1 -- 172.21.2.102:0/1488093296 >> 172.21.2.102:0/1488093296 conn(0x7fc530094e10 msgr2=0x7fc5300a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:20:47.474 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:47.477 7fc53677a700 1 -- 172.21.2.102:0/1488093296 shutdown_connections 2021-02-10T11:20:47.474 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:47.477 7fc53677a700 1 -- 172.21.2.102:0/1488093296 wait complete. 2021-02-10T11:20:47.481 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000344375 2021-02-10T11:20:47.914 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:47.901 7fba68951700 1 Processor -- start 2021-02-10T11:20:47.914 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:47.901 7fba68951700 1 -- start start 2021-02-10T11:20:47.915 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:47.902 7fba68951700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fba641b0870 0x7fba640f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:47.915 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:47.902 7fba68951700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba641b0c70 con 0x7fba640efe00 2021-02-10T11:20:47.915 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:47.902 7fba68951700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba640f0b50 con 0x7fba641b0470 2021-02-10T11:20:47.915 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:47.902 7fba68951700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba640e5ea0 con 0x7fba641b0870 2021-02-10T11:20:47.915 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:47.902 7fba62e1d700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fba640efe00 0x7fba640f0200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:46088/0 (socket says 172.21.2.102:46088) 2021-02-10T11:20:47.916 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:47.902 7fba62e1d700 1 -- 172.21.2.102:0/1480002132 learned_addr learned my addr 172.21.2.102:0/1480002132 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:20:47.916 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:47.902 7fba61e1b700 1 --2- 172.21.2.102:0/1480002132 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fba641b0870 0x7fba640f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:47.916 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:47.902 7fba6161a700 1 -- 172.21.2.102:0/1480002132 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 973272790 0 0) 0x7fba5c000f90 con 0x7fba640efe00 2021-02-10T11:20:47.916 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:47.902 7fba6161a700 1 -- 172.21.2.102:0/1480002132 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba50002fe0 con 0x7fba640efe00 2021-02-10T11:20:47.916 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:47.903 7fba6161a700 1 -- 172.21.2.102:0/1480002132 <== mon.1 v1:172.21.2.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 635423334 0 0) 0x7fba54001080 con 0x7fba641b0470 2021-02-10T11:20:47.917 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:47.903 7fba6161a700 1 -- 172.21.2.102:0/1480002132 --> v1:172.21.2.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba50003210 con 0x7fba641b0470 2021-02-10T11:20:47.917 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:47.903 7fba6161a700 1 -- 172.21.2.102:0/1480002132 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3865494104 0 0) 0x7fba5c001460 con 0x7fba640efe00 2021-02-10T11:20:47.917 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:47.903 7fba61e1b700 1 -- 172.21.2.102:0/1480002132 >> v1:172.21.2.113:6789/0 conn(0x7fba641b0470 legacy=0x7fba640ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:47.917 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:47.903 7fba61e1b700 1 -- 172.21.2.102:0/1480002132 >> v1:172.21.2.102:6789/0 conn(0x7fba640efe00 legacy=0x7fba640f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:47.917 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:47.903 7fba61e1b700 1 -- 172.21.2.102:0/1480002132 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba640e52e0 con 0x7fba641b0870 2021-02-10T11:20:47.918 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:47.903 7fba61e1b700 1 --2- 172.21.2.102:0/1480002132 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fba641b0870 0x7fba640f20e0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7fba4c006650 tx=0x7fba4c009a00).ready entity=mon.2 client_cookie=21d44353e5cfd6a2 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:20:47.918 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:47.903 7fba6161a700 1 -- 172.21.2.102:0/1480002132 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fba4c0009b0 con 0x7fba641b0870 2021-02-10T11:20:47.918 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:47.903 7fba6161a700 1 -- 172.21.2.102:0/1480002132 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba4c00c0c0 con 0x7fba641b0870 2021-02-10T11:20:47.918 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:47.903 7fba6161a700 1 -- 172.21.2.102:0/1480002132 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fba4c00c510 con 0x7fba641b0870 2021-02-10T11:20:47.918 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:47.903 7fba68951700 1 -- 172.21.2.102:0/1480002132 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fba641b0870 msgr2=0x7fba640f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:47.918 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:47.903 7fba68951700 1 --2- 172.21.2.102:0/1480002132 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fba641b0870 0x7fba640f20e0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7fba4c006650 tx=0x7fba4c009a00).stop 2021-02-10T11:20:47.919 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:47.904 7fba68951700 1 -- 172.21.2.102:0/1480002132 shutdown_connections 2021-02-10T11:20:47.919 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:47.904 7fba68951700 1 --2- 172.21.2.102:0/1480002132 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fba641b0870 0x7fba640f20e0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:47.919 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:47.904 7fba68951700 1 -- 172.21.2.102:0/1480002132 >> 172.21.2.102:0/1480002132 conn(0x7fba640ce3e0 msgr2=0x7fba640e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:20:47.919 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:47.904 7fba68951700 1 -- 172.21.2.102:0/1480002132 shutdown_connections 2021-02-10T11:20:47.919 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:47.904 7fba68951700 1 -- 172.21.2.102:0/1480002132 wait complete. 2021-02-10T11:20:47.920 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:47.919 7fba68951700 1 Processor -- start 2021-02-10T11:20:47.920 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:47.919 7fba68951700 1 -- start start 2021-02-10T11:20:47.923 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:47.920 7fba68951700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fba640a0420 0x7fba640a0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:47.923 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:47.920 7fba68951700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fba64189d00 0x7fba6418a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:47.924 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:47.920 7fba68951700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fba6418a720 0x7fba640b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:47.924 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:47.920 7fba68951700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba64189780 con 0x7fba640a0420 2021-02-10T11:20:47.924 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:47.920 7fba68951700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba6418ab20 con 0x7fba64189d00 2021-02-10T11:20:47.924 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:47.920 7fba68951700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba640a1140 con 0x7fba6418a720 2021-02-10T11:20:47.924 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:47.920 7fba6261c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fba640a0420 0x7fba640a0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:47.924 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:47.920 7fba6261c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fba640a0420 0x7fba640a0820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:53868/0 (socket says 172.21.2.102:53868) 2021-02-10T11:20:47.925 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:47.920 7fba6261c700 1 -- 172.21.2.102:0/1937219748 learned_addr learned my addr 172.21.2.102:0/1937219748 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:20:47.925 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:47.920 7fba61e1b700 1 --2- 172.21.2.102:0/1937219748 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fba64189d00 0x7fba6418a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:47.925 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:47.920 7fba62e1d700 1 --2- 172.21.2.102:0/1937219748 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fba6418a720 0x7fba640b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:47.925 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:47.920 7fba6261c700 1 -- 172.21.2.102:0/1937219748 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fba6418a720 msgr2=0x7fba640b7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:47.925 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:47.920 7fba6261c700 1 --2- 172.21.2.102:0/1937219748 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fba6418a720 0x7fba640b7be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:47.926 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:47.920 7fba6261c700 1 -- 172.21.2.102:0/1937219748 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fba64189d00 msgr2=0x7fba6418a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:47.926 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:47.920 7fba6261c700 1 --2- 172.21.2.102:0/1937219748 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fba64189d00 0x7fba6418a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:47.926 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:47.920 7fba6261c700 1 -- 172.21.2.102:0/1937219748 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba640b8200 con 0x7fba640a0420 2021-02-10T11:20:47.926 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:47.920 7fba6261c700 1 --2- 172.21.2.102:0/1937219748 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fba640a0420 0x7fba640a0820 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 rx=0x7fba5400b5b0 tx=0x7fba5400b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:20:47.926 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:47.921 7fba5affd700 1 -- 172.21.2.102:0/1937219748 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fba5400bd40 con 0x7fba640a0420 2021-02-10T11:20:47.927 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:47.921 7fba5affd700 1 -- 172.21.2.102:0/1937219748 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba54006020 con 0x7fba640a0420 2021-02-10T11:20:47.927 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:47.921 7fba5affd700 1 -- 172.21.2.102:0/1937219748 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fba540054a0 con 0x7fba640a0420 2021-02-10T11:20:47.927 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:47.921 7fba68951700 1 -- 172.21.2.102:0/1937219748 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba640b8200 con 0x7fba640a0420 2021-02-10T11:20:47.927 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:47.921 7fba68951700 1 -- 172.21.2.102:0/1937219748 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba640b8790 con 0x7fba640a0420 2021-02-10T11:20:47.927 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:47.922 7fba5affd700 1 -- 172.21.2.102:0/1937219748 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fba54006200 con 0x7fba640a0420 2021-02-10T11:20:47.928 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:47.922 7fba5affd700 1 --2- 172.21.2.102:0/1937219748 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fba480186c0 0x7fba4801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:47.928 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:47.922 7fba5affd700 1 -- 172.21.2.102:0/1937219748 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fba54032410 con 0x7fba640a0420 2021-02-10T11:20:47.928 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:47.922 7fba61e1b700 1 --2- 172.21.2.102:0/1937219748 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fba480186c0 0x7fba4801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:47.928 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:47.922 7fba61e1b700 1 --2- 172.21.2.102:0/1937219748 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fba480186c0 0x7fba4801abc0 crc :-1 s=READY pgs=398 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:20:47.946 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:47.952 7fba68951700 1 -- 172.21.2.102:0/1937219748 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba64105ae0 con 0x7fba640a0420 2021-02-10T11:20:47.949 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:47.954 7fba5affd700 1 -- 172.21.2.102:0/1937219748 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fba54000dd0 con 0x7fba640a0420 2021-02-10T11:20:48.078 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:48.083 7fba68951700 1 -- 172.21.2.102:0/1937219748 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344375", "target": ["mgr", ""]}) v1 -- 0x7fba640b8790 con 0x7fba480186c0 2021-02-10T11:20:48.089 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:48.095 7fba5affd700 1 -- 172.21.2.102:0/1937219748 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fba4c0009b0 con 0x7fba480186c0 2021-02-10T11:20:48.095 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:48.100 7fba68951700 1 -- 172.21.2.102:0/1937219748 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fba480186c0 msgr2=0x7fba4801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:48.095 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:48.100 7fba68951700 1 --2- 172.21.2.102:0/1937219748 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fba480186c0 0x7fba4801abc0 crc :-1 s=READY pgs=398 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:48.095 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:48.100 7fba68951700 1 -- 172.21.2.102:0/1937219748 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fba640a0420 msgr2=0x7fba640a0820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:48.096 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:48.100 7fba68951700 1 --2- 172.21.2.102:0/1937219748 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fba640a0420 0x7fba640a0820 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 rx=0x7fba5400b5b0 tx=0x7fba5400b950).stop 2021-02-10T11:20:48.096 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:48.100 7fba68951700 1 -- 172.21.2.102:0/1937219748 shutdown_connections 2021-02-10T11:20:48.096 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:48.100 7fba68951700 1 --2- 172.21.2.102:0/1937219748 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fba6418a720 0x7fba640b7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:48.096 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:48.100 7fba68951700 1 --2- 172.21.2.102:0/1937219748 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fba480186c0 0x7fba4801abc0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:48.096 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:48.100 7fba68951700 1 --2- 172.21.2.102:0/1937219748 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fba64189d00 0x7fba6418a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:48.096 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:48.100 7fba68951700 1 --2- 172.21.2.102:0/1937219748 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fba640a0420 0x7fba640a0820 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:48.096 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:48.100 7fba68951700 1 -- 172.21.2.102:0/1937219748 >> 172.21.2.102:0/1937219748 conn(0x7fba64094e10 msgr2=0x7fba640a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:20:48.096 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:48.100 7fba68951700 1 -- 172.21.2.102:0/1937219748 shutdown_connections 2021-02-10T11:20:48.097 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:48.100 7fba68951700 1 -- 172.21.2.102:0/1937219748 wait complete. 2021-02-10T11:20:48.105 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000344376 2021-02-10T11:20:48.537 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:48.526 7f680b305700 1 Processor -- start 2021-02-10T11:20:48.538 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:48.526 7f680b305700 1 -- start start 2021-02-10T11:20:48.538 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:48.527 7f680b305700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f68041b0870 0x7f68040f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:48.538 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:48.527 7f680b305700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f68040efe00 0x7f68040f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:48.538 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:48.527 7f680b305700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68041b0c70 con 0x7f68041b0470 2021-02-10T11:20:48.538 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:48.527 7f680b305700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68040e5ea0 con 0x7f68040efe00 2021-02-10T11:20:48.538 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:48.527 7f680b305700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68040f0b50 con 0x7f68041b0870 2021-02-10T11:20:48.538 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:48.527 7f6809922700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f68040efe00 0x7f68040f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:48.538 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:48.527 7f6809121700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f68041b0470 0x7f68040ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:46098/0 (socket says 172.21.2.102:46098) 2021-02-10T11:20:48.538 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:48.527 7f6809121700 1 -- 172.21.2.102:0/981449768 learned_addr learned my addr 172.21.2.102:0/981449768 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:20:48.538 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:48.527 7f6809922700 1 --2- 172.21.2.102:0/981449768 >> v2:172.21.2.102:3300/0 conn(0x7f68040efe00 0x7f68040f0200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:53878/0 (socket says 172.21.2.102:53878) 2021-02-10T11:20:48.538 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:48.527 7f6808920700 1 --2- 172.21.2.102:0/981449768 >> v2:172.21.2.113:3300/0 conn(0x7f68041b0870 0x7f68040f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:48.539 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:48.527 7f6809922700 1 -- 172.21.2.102:0/981449768 >> v2:172.21.2.113:3300/0 conn(0x7f68041b0870 msgr2=0x7f68040f20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:20:48.539 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:48.527 7f6809922700 1 --2- 172.21.2.102:0/981449768 >> v2:172.21.2.113:3300/0 conn(0x7f68041b0870 0x7f68040f20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:48.539 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:48.527 7f6809922700 1 -- 172.21.2.102:0/981449768 >> v1:172.21.2.102:6789/0 conn(0x7f68041b0470 legacy=0x7f68040ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:48.539 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:48.527 7f6809922700 1 -- 172.21.2.102:0/981449768 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68040f0820 con 0x7f68040efe00 2021-02-10T11:20:48.539 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:48.528 7f6809922700 1 --2- 172.21.2.102:0/981449768 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f68040efe00 0x7f68040f0200 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 rx=0x7f67fc006e30 tx=0x7f67fc009e60).ready entity=mon.0 client_cookie=28c7d9ab0e28198f server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:20:48.539 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:48.528 7f67fbfff700 1 -- 172.21.2.102:0/981449768 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f67fc006a50 con 0x7f68040efe00 2021-02-10T11:20:48.539 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:48.528 7f67fbfff700 1 -- 172.21.2.102:0/981449768 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67fc00b300 con 0x7f68040efe00 2021-02-10T11:20:48.539 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:48.528 7f67fbfff700 1 -- 172.21.2.102:0/981449768 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f67fc00b7e0 con 0x7f68040efe00 2021-02-10T11:20:48.539 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:48.528 7f680b305700 1 -- 172.21.2.102:0/981449768 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f68040efe00 msgr2=0x7f68040f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:48.539 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:48.528 7f680b305700 1 --2- 172.21.2.102:0/981449768 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f68040efe00 0x7f68040f0200 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 rx=0x7f67fc006e30 tx=0x7f67fc009e60).stop 2021-02-10T11:20:48.539 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:48.529 7f680b305700 1 -- 172.21.2.102:0/981449768 shutdown_connections 2021-02-10T11:20:48.539 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:48.529 7f680b305700 1 --2- 172.21.2.102:0/981449768 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f68040efe00 0x7f68040f0200 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:48.539 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:48.529 7f680b305700 1 --2- 172.21.2.102:0/981449768 >> v2:172.21.2.113:3300/0 conn(0x7f68041b0870 0x7f68040f20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:48.540 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:48.529 7f680b305700 1 -- 172.21.2.102:0/981449768 >> 172.21.2.102:0/981449768 conn(0x7f68040ce3e0 msgr2=0x7f68040e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:20:48.540 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:48.529 7f680b305700 1 -- 172.21.2.102:0/981449768 shutdown_connections 2021-02-10T11:20:48.540 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:48.529 7f680b305700 1 -- 172.21.2.102:0/981449768 wait complete. 2021-02-10T11:20:48.541 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:48.543 7f680b305700 1 Processor -- start 2021-02-10T11:20:48.541 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:48.543 7f680b305700 1 -- start start 2021-02-10T11:20:48.541 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:48.544 7f680b305700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f68040a0420 0x7f68040a0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:48.541 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:48.544 7f680b305700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6804189d00 0x7f680418a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:48.541 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:48.544 7f680b305700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f680418a720 0x7f68040b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:48.541 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:48.544 7f680b305700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6804189780 con 0x7f680418a720 2021-02-10T11:20:48.541 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:48.544 7f680b305700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f680418ab20 con 0x7f68040a0420 2021-02-10T11:20:48.542 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:48.544 7f680b305700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68040a1140 con 0x7f6804189d00 2021-02-10T11:20:48.542 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:48.544 7f6808920700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6804189d00 0x7f680418a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:48.542 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:48.544 7f6809922700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f680418a720 0x7f68040b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:48.542 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:48.544 7f6809121700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f68040a0420 0x7f68040a0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:48.542 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:48.544 7f6809922700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f680418a720 0x7f68040b7be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:53884/0 (socket says 172.21.2.102:53884) 2021-02-10T11:20:48.542 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:48.544 7f6809922700 1 -- 172.21.2.102:0/1265739976 learned_addr learned my addr 172.21.2.102:0/1265739976 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:20:48.542 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:48.545 7f6809922700 1 -- 172.21.2.102:0/1265739976 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6804189d00 msgr2=0x7f680418a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:48.542 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:48.545 7f6809922700 1 --2- 172.21.2.102:0/1265739976 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6804189d00 0x7f680418a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:48.542 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:48.545 7f6809922700 1 -- 172.21.2.102:0/1265739976 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f68040a0420 msgr2=0x7f68040a0820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:48.542 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:48.545 7f6809922700 1 --2- 172.21.2.102:0/1265739976 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f68040a0420 0x7f68040a0820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:48.542 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:48.545 7f6809922700 1 -- 172.21.2.102:0/1265739976 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68040b8200 con 0x7f680418a720 2021-02-10T11:20:48.542 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:48.545 7f6809922700 1 --2- 172.21.2.102:0/1265739976 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f680418a720 0x7f68040b7be0 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 rx=0x7f67fc00b970 tx=0x7f67fc00bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:20:48.542 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:48.545 7f67f9ffb700 1 -- 172.21.2.102:0/1265739976 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f67fc009380 con 0x7f680418a720 2021-02-10T11:20:48.543 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:48.545 7f67f9ffb700 1 -- 172.21.2.102:0/1265739976 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67fc005820 con 0x7f680418a720 2021-02-10T11:20:48.543 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:48.545 7f680b305700 1 -- 172.21.2.102:0/1265739976 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68040b8200 con 0x7f680418a720 2021-02-10T11:20:48.543 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:48.545 7f680b305700 1 -- 172.21.2.102:0/1265739976 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68040b8790 con 0x7f680418a720 2021-02-10T11:20:48.543 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:48.545 7f67f9ffb700 1 -- 172.21.2.102:0/1265739976 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f67fc005d40 con 0x7f680418a720 2021-02-10T11:20:48.543 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:48.546 7f67f9ffb700 1 -- 172.21.2.102:0/1265739976 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f67fc005a00 con 0x7f680418a720 2021-02-10T11:20:48.543 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:48.546 7f67f9ffb700 1 --2- 172.21.2.102:0/1265739976 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f67e4018730 0x7f67e401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:48.543 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:48.546 7f67f9ffb700 1 -- 172.21.2.102:0/1265739976 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f67fc032b30 con 0x7f680418a720 2021-02-10T11:20:48.543 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:48.546 7f6809121700 1 --2- 172.21.2.102:0/1265739976 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f67e4018730 0x7f67e401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:48.543 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:48.546 7f6809121700 1 --2- 172.21.2.102:0/1265739976 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f67e4018730 0x7f67e401abc0 crc :-1 s=READY pgs=399 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:20:48.571 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:48.576 7f680b305700 1 -- 172.21.2.102:0/1265739976 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6804105ae0 con 0x7f680418a720 2021-02-10T11:20:48.573 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:48.579 7f67f9ffb700 1 -- 172.21.2.102:0/1265739976 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f67fc032d40 con 0x7f680418a720 2021-02-10T11:20:48.707 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:48.712 7f680b305700 1 -- 172.21.2.102:0/1265739976 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344376", "target": ["mgr", ""]}) v1 -- 0x7f68040b9020 con 0x7f67e4018730 2021-02-10T11:20:48.717 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:48.722 7f67f9ffb700 1 -- 172.21.2.102:0/1265739976 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f6800001a00 con 0x7f67e4018730 2021-02-10T11:20:48.725 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:48.730 7f680b305700 1 -- 172.21.2.102:0/1265739976 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f67e4018730 msgr2=0x7f67e401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:48.725 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:48.730 7f680b305700 1 --2- 172.21.2.102:0/1265739976 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f67e4018730 0x7f67e401abc0 crc :-1 s=READY pgs=399 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:48.725 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:48.730 7f680b305700 1 -- 172.21.2.102:0/1265739976 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f680418a720 msgr2=0x7f68040b7be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:48.725 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:48.730 7f680b305700 1 --2- 172.21.2.102:0/1265739976 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f680418a720 0x7f68040b7be0 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 rx=0x7f67fc00b970 tx=0x7f67fc00bd10).stop 2021-02-10T11:20:48.725 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:48.730 7f680b305700 1 -- 172.21.2.102:0/1265739976 shutdown_connections 2021-02-10T11:20:48.725 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:48.730 7f680b305700 1 --2- 172.21.2.102:0/1265739976 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f680418a720 0x7f68040b7be0 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:48.725 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:48.730 7f680b305700 1 --2- 172.21.2.102:0/1265739976 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6804189d00 0x7f680418a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:48.725 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:48.730 7f680b305700 1 --2- 172.21.2.102:0/1265739976 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f67e4018730 0x7f67e401abc0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:48.725 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:48.730 7f680b305700 1 --2- 172.21.2.102:0/1265739976 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f68040a0420 0x7f68040a0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:48.726 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:48.730 7f680b305700 1 -- 172.21.2.102:0/1265739976 >> 172.21.2.102:0/1265739976 conn(0x7f6804094e10 msgr2=0x7f68040a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:20:48.726 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:48.730 7f680b305700 1 -- 172.21.2.102:0/1265739976 shutdown_connections 2021-02-10T11:20:48.726 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:48.730 7f680b305700 1 -- 172.21.2.102:0/1265739976 wait complete. 2021-02-10T11:20:48.734 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000344377 2021-02-10T11:20:49.166 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:49.155 7f5c39cab700 1 Processor -- start 2021-02-10T11:20:49.167 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:49.155 7f5c39cab700 1 -- start start 2021-02-10T11:20:49.167 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:49.156 7f5c39cab700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f5c341b0870 0x7f5c340f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:49.167 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:49.156 7f5c39cab700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c341b0c70 con 0x7f5c341b0470 2021-02-10T11:20:49.167 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:49.156 7f5c39cab700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c340f0b50 con 0x7f5c340efe00 2021-02-10T11:20:49.167 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:49.156 7f5c39cab700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c340e5ea0 con 0x7f5c341b0870 2021-02-10T11:20:49.168 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:49.156 7f5c32ffd700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f5c341b0870 0x7f5c340f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:49.168 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:49.156 7f5c32ffd700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f5c341b0870 0x7f5c340f20e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:53888/0 (socket says 172.21.2.102:53888) 2021-02-10T11:20:49.168 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:49.156 7f5c32ffd700 1 -- 172.21.2.102:0/3762136109 learned_addr learned my addr 172.21.2.102:0/3762136109 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:20:49.168 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:49.156 7f5c32ffd700 1 -- 172.21.2.102:0/3762136109 >> v1:172.21.2.113:6789/0 conn(0x7f5c340efe00 legacy=0x7f5c340f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:49.168 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:49.156 7f5c32ffd700 1 -- 172.21.2.102:0/3762136109 >> v1:172.21.2.102:6789/0 conn(0x7f5c341b0470 legacy=0x7f5c340ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:49.168 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:49.156 7f5c32ffd700 1 -- 172.21.2.102:0/3762136109 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c340e52e0 con 0x7f5c341b0870 2021-02-10T11:20:49.169 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:49.157 7f5c32ffd700 1 --2- 172.21.2.102:0/3762136109 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5c341b0870 0x7f5c340f20e0 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 rx=0x7f5c24006700 tx=0x7f5c24009b80).ready entity=mon.0 client_cookie=e7228198784e3ece server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:20:49.169 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:49.157 7f5c327fc700 1 -- 172.21.2.102:0/3762136109 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5c2400c0c0 con 0x7f5c341b0870 2021-02-10T11:20:49.169 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:49.157 7f5c327fc700 1 -- 172.21.2.102:0/3762136109 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c2400c440 con 0x7f5c341b0870 2021-02-10T11:20:49.169 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:49.157 7f5c327fc700 1 -- 172.21.2.102:0/3762136109 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5c2400c8a0 con 0x7f5c341b0870 2021-02-10T11:20:49.169 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:49.157 7f5c39cab700 1 -- 172.21.2.102:0/3762136109 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5c341b0870 msgr2=0x7f5c340f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:49.169 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:49.157 7f5c39cab700 1 --2- 172.21.2.102:0/3762136109 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5c341b0870 0x7f5c340f20e0 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 rx=0x7f5c24006700 tx=0x7f5c24009b80).stop 2021-02-10T11:20:49.170 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:49.157 7f5c39cab700 1 -- 172.21.2.102:0/3762136109 shutdown_connections 2021-02-10T11:20:49.170 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:49.157 7f5c39cab700 1 --2- 172.21.2.102:0/3762136109 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5c341b0870 0x7f5c340f20e0 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:49.170 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:49.157 7f5c39cab700 1 -- 172.21.2.102:0/3762136109 >> 172.21.2.102:0/3762136109 conn(0x7f5c340ce3e0 msgr2=0x7f5c340e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:20:49.170 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:49.157 7f5c39cab700 1 -- 172.21.2.102:0/3762136109 shutdown_connections 2021-02-10T11:20:49.170 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:49.157 7f5c39cab700 1 -- 172.21.2.102:0/3762136109 wait complete. 2021-02-10T11:20:49.170 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:49.172 7f5c39cab700 1 Processor -- start 2021-02-10T11:20:49.173 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:49.172 7f5c39cab700 1 -- start start 2021-02-10T11:20:49.173 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:49.172 7f5c39cab700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5c34087510 0x7f5c340aedc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:49.173 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:49.172 7f5c39cab700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5c341c68f0 0x7f5c341b1f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:49.173 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:49.172 7f5c39cab700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5c3408c4d0 0x7f5c341b18a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:49.173 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:49.172 7f5c39cab700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c340a2d50 con 0x7f5c34087510 2021-02-10T11:20:49.173 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:49.172 7f5c39cab700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c340a2f70 con 0x7f5c341c68f0 2021-02-10T11:20:49.173 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:49.172 7f5c39cab700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c340a3190 con 0x7f5c3408c4d0 2021-02-10T11:20:49.173 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:49.173 7f5c337fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5c34087510 0x7f5c340aedc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:49.174 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:49.173 7f5c337fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5c34087510 0x7f5c340aedc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:53892/0 (socket says 172.21.2.102:53892) 2021-02-10T11:20:49.174 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:49.173 7f5c337fe700 1 -- 172.21.2.102:0/1011153642 learned_addr learned my addr 172.21.2.102:0/1011153642 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:20:49.174 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:49.173 7f5c32ffd700 1 --2- 172.21.2.102:0/1011153642 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5c341c68f0 0x7f5c341b1f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:49.174 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:49.173 7f5c33fff700 1 --2- 172.21.2.102:0/1011153642 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5c3408c4d0 0x7f5c341b18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:49.174 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:49.173 7f5c337fe700 1 -- 172.21.2.102:0/1011153642 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5c3408c4d0 msgr2=0x7f5c341b18a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:49.174 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:49.173 7f5c337fe700 1 --2- 172.21.2.102:0/1011153642 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5c3408c4d0 0x7f5c341b18a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:49.174 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:49.173 7f5c337fe700 1 -- 172.21.2.102:0/1011153642 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5c341c68f0 msgr2=0x7f5c341b1f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:49.174 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:49.173 7f5c337fe700 1 --2- 172.21.2.102:0/1011153642 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5c341c68f0 0x7f5c341b1f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:49.174 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:49.173 7f5c337fe700 1 -- 172.21.2.102:0/1011153642 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c340a33b0 con 0x7f5c34087510 2021-02-10T11:20:49.174 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:49.173 7f5c32ffd700 1 --2- 172.21.2.102:0/1011153642 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5c341c68f0 0x7f5c341b1f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:20:49.175 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:49.173 7f5c337fe700 1 --2- 172.21.2.102:0/1011153642 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5c34087510 0x7f5c340aedc0 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 rx=0x7f5c1c006ed0 tx=0x7f5c1c00b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:20:49.175 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:49.173 7f5c2bfff700 1 -- 172.21.2.102:0/1011153642 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5c1c000d20 con 0x7f5c34087510 2021-02-10T11:20:49.175 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:49.174 7f5c2bfff700 1 -- 172.21.2.102:0/1011153642 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c1c0022e0 con 0x7f5c34087510 2021-02-10T11:20:49.175 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:49.174 7f5c39cab700 1 -- 172.21.2.102:0/1011153642 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c340a33b0 con 0x7f5c34087510 2021-02-10T11:20:49.175 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:49.174 7f5c2bfff700 1 -- 172.21.2.102:0/1011153642 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5c1c002800 con 0x7f5c34087510 2021-02-10T11:20:49.175 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:49.174 7f5c39cab700 1 -- 172.21.2.102:0/1011153642 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c340afca0 con 0x7f5c34087510 2021-02-10T11:20:49.175 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:49.174 7f5c2bfff700 1 -- 172.21.2.102:0/1011153642 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f5c1c0024c0 con 0x7f5c34087510 2021-02-10T11:20:49.175 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:49.174 7f5c2bfff700 1 --2- 172.21.2.102:0/1011153642 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5c18018730 0x7f5c1801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:49.175 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:49.174 7f5c2bfff700 1 -- 172.21.2.102:0/1011153642 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f5c1c032620 con 0x7f5c34087510 2021-02-10T11:20:49.176 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:49.174 7f5c32ffd700 1 --2- 172.21.2.102:0/1011153642 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5c18018730 0x7f5c1801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:49.176 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:49.175 7f5c32ffd700 1 --2- 172.21.2.102:0/1011153642 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5c18018730 0x7f5c1801ac10 crc :-1 s=READY pgs=400 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:20:49.200 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:49.205 7f5c39cab700 1 -- 172.21.2.102:0/1011153642 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c3407e6b0 con 0x7f5c34087510 2021-02-10T11:20:49.202 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:49.208 7f5c2bfff700 1 -- 172.21.2.102:0/1011153642 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f5c1c000f20 con 0x7f5c34087510 2021-02-10T11:20:49.330 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:49.336 7f5c39cab700 1 -- 172.21.2.102:0/1011153642 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344377", "target": ["mgr", ""]}) v1 -- 0x7f5c340a2d50 con 0x7f5c18018730 2021-02-10T11:20:49.341 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:49.347 7f5c2bfff700 1 -- 172.21.2.102:0/1011153642 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f5c240009b0 con 0x7f5c18018730 2021-02-10T11:20:49.347 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:49.353 7f5c39cab700 1 -- 172.21.2.102:0/1011153642 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5c18018730 msgr2=0x7f5c1801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:49.348 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:49.353 7f5c39cab700 1 --2- 172.21.2.102:0/1011153642 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5c18018730 0x7f5c1801ac10 crc :-1 s=READY pgs=400 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:49.348 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:49.353 7f5c39cab700 1 -- 172.21.2.102:0/1011153642 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5c34087510 msgr2=0x7f5c340aedc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:49.348 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:49.353 7f5c39cab700 1 --2- 172.21.2.102:0/1011153642 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5c34087510 0x7f5c340aedc0 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 rx=0x7f5c1c006ed0 tx=0x7f5c1c00b960).stop 2021-02-10T11:20:49.348 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:49.353 7f5c39cab700 1 -- 172.21.2.102:0/1011153642 shutdown_connections 2021-02-10T11:20:49.348 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:49.353 7f5c39cab700 1 --2- 172.21.2.102:0/1011153642 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5c3408c4d0 0x7f5c341b18a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:49.348 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:49.353 7f5c39cab700 1 --2- 172.21.2.102:0/1011153642 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5c18018730 0x7f5c1801ac10 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:49.348 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:49.353 7f5c39cab700 1 --2- 172.21.2.102:0/1011153642 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5c341c68f0 0x7f5c341b1f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:49.348 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:49.353 7f5c39cab700 1 --2- 172.21.2.102:0/1011153642 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5c34087510 0x7f5c340aedc0 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:49.348 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:49.353 7f5c39cab700 1 -- 172.21.2.102:0/1011153642 >> 172.21.2.102:0/1011153642 conn(0x7f5c340886d0 msgr2=0x7f5c34081c40 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:20:49.349 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:49.353 7f5c39cab700 1 -- 172.21.2.102:0/1011153642 shutdown_connections 2021-02-10T11:20:49.349 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:49.353 7f5c39cab700 1 -- 172.21.2.102:0/1011153642 wait complete. 2021-02-10T11:20:49.357 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000344378 2021-02-10T11:20:49.791 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:49.776 7facb1afd700 1 Processor -- start 2021-02-10T11:20:49.791 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:49.776 7facb1afd700 1 -- start start 2021-02-10T11:20:49.792 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:49.777 7facb1afd700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7facac1b0470 0x7facac1b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:49.792 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:49.777 7facb1afd700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7facac0f20e0 0x7facac0f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:49.792 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:49.777 7facb1afd700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facac0f0b50 con 0x7facac0ef610 2021-02-10T11:20:49.792 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:49.777 7facb1afd700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7facac0f0820 con 0x7facac1b0470 2021-02-10T11:20:49.792 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:49.777 7facb1afd700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facac0f24e0 con 0x7facac0f20e0 2021-02-10T11:20:49.792 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:49.777 7facab7fe700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7facac1b0470 0x7facac1b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:49.793 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:49.777 7facab7fe700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7facac1b0470 0x7facac1b0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:53900/0 (socket says 172.21.2.102:53900) 2021-02-10T11:20:49.793 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:49.777 7facab7fe700 1 -- 172.21.2.102:0/3643492901 learned_addr learned my addr 172.21.2.102:0/3643492901 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:20:49.793 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:49.777 7facaaffd700 1 --1- 172.21.2.102:0/3643492901 >> v1:172.21.2.102:6789/0 conn(0x7facac0ef610 0x7facac0efa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:46124/0 (socket says 172.21.2.102:46124) 2021-02-10T11:20:49.793 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:49.777 7facabfff700 1 --2- 172.21.2.102:0/3643492901 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7facac0f20e0 0x7facac0f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:49.793 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:49.778 7facab7fe700 1 -- 172.21.2.102:0/3643492901 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7facac0f20e0 msgr2=0x7facac0f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:20:49.793 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:49.778 7facab7fe700 1 --2- 172.21.2.102:0/3643492901 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7facac0f20e0 0x7facac0f0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:49.794 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:49.778 7facab7fe700 1 -- 172.21.2.102:0/3643492901 >> v1:172.21.2.102:6789/0 conn(0x7facac0ef610 legacy=0x7facac0efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:49.794 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:49.778 7facab7fe700 1 -- 172.21.2.102:0/3643492901 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facac0e52e0 con 0x7facac1b0470 2021-02-10T11:20:49.794 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:49.778 7facab7fe700 1 --2- 172.21.2.102:0/3643492901 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7facac1b0470 0x7facac1b0870 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 rx=0x7fac94006700 tx=0x7fac94009b80).ready entity=mon.0 client_cookie=77a887efd448de5c server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:20:49.794 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:49.778 7facaa7fc700 1 -- 172.21.2.102:0/3643492901 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fac9400c0c0 con 0x7facac1b0470 2021-02-10T11:20:49.794 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:49.778 7facaa7fc700 1 -- 172.21.2.102:0/3643492901 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac9400c440 con 0x7facac1b0470 2021-02-10T11:20:49.794 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:49.778 7facaa7fc700 1 -- 172.21.2.102:0/3643492901 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fac9400c960 con 0x7facac1b0470 2021-02-10T11:20:49.795 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:49.778 7facb1afd700 1 -- 172.21.2.102:0/3643492901 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7facac1b0470 msgr2=0x7facac1b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:49.795 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:49.778 7facb1afd700 1 --2- 172.21.2.102:0/3643492901 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7facac1b0470 0x7facac1b0870 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 rx=0x7fac94006700 tx=0x7fac94009b80).stop 2021-02-10T11:20:49.795 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:49.779 7facb1afd700 1 -- 172.21.2.102:0/3643492901 shutdown_connections 2021-02-10T11:20:49.795 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:49.779 7facb1afd700 1 --2- 172.21.2.102:0/3643492901 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7facac0f20e0 0x7facac0f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:49.795 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:49.779 7facb1afd700 1 --2- 172.21.2.102:0/3643492901 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7facac1b0470 0x7facac1b0870 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:49.795 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:49.779 7facb1afd700 1 -- 172.21.2.102:0/3643492901 >> 172.21.2.102:0/3643492901 conn(0x7facac0ce3e0 msgr2=0x7facac0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:20:49.795 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:49.779 7facb1afd700 1 -- 172.21.2.102:0/3643492901 shutdown_connections 2021-02-10T11:20:49.795 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:49.779 7facb1afd700 1 -- 172.21.2.102:0/3643492901 wait complete. 2021-02-10T11:20:49.795 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:49.794 7facb1afd700 1 Processor -- start 2021-02-10T11:20:49.795 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:49.794 7facb1afd700 1 -- start start 2021-02-10T11:20:49.795 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:49.794 7facb1afd700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7facac0a0420 0x7facac0a0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:49.795 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:49.794 7facb1afd700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7facac189d00 0x7facac18a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:49.795 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:49.794 7facb1afd700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7facac18a720 0x7facac0b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:49.796 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:49.794 7facb1afd700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facac189780 con 0x7facac0a0420 2021-02-10T11:20:49.796 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:49.794 7facb1afd700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facac18ab20 con 0x7facac189d00 2021-02-10T11:20:49.796 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:49.794 7facb1afd700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facac0a1140 con 0x7facac18a720 2021-02-10T11:20:49.796 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:49.795 7facab7fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7facac0a0420 0x7facac0a0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:49.796 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:49.795 7facab7fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7facac0a0420 0x7facac0a0820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:53904/0 (socket says 172.21.2.102:53904) 2021-02-10T11:20:49.796 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:49.795 7facab7fe700 1 -- 172.21.2.102:0/2857396056 learned_addr learned my addr 172.21.2.102:0/2857396056 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:20:49.796 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:49.795 7facaaffd700 1 --2- 172.21.2.102:0/2857396056 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7facac189d00 0x7facac18a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:49.797 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:49.795 7facabfff700 1 --2- 172.21.2.102:0/2857396056 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7facac18a720 0x7facac0b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:49.797 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:49.795 7facab7fe700 1 -- 172.21.2.102:0/2857396056 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7facac18a720 msgr2=0x7facac0b7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:49.797 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:49.795 7facab7fe700 1 --2- 172.21.2.102:0/2857396056 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7facac18a720 0x7facac0b7be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:49.797 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:49.795 7facab7fe700 1 -- 172.21.2.102:0/2857396056 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7facac189d00 msgr2=0x7facac18a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:49.797 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:49.795 7facab7fe700 1 --2- 172.21.2.102:0/2857396056 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7facac189d00 0x7facac18a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:49.798 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:49.795 7facab7fe700 1 -- 172.21.2.102:0/2857396056 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facac0b8200 con 0x7facac0a0420 2021-02-10T11:20:49.798 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:49.795 7facab7fe700 1 --2- 172.21.2.102:0/2857396056 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7facac0a0420 0x7facac0a0820 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 rx=0x7fac9400b5e0 tx=0x7fac9400b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:20:49.798 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:49.795 7faca3fff700 1 -- 172.21.2.102:0/2857396056 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fac9400bd70 con 0x7facac0a0420 2021-02-10T11:20:49.798 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:49.795 7faca3fff700 1 -- 172.21.2.102:0/2857396056 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac94002030 con 0x7facac0a0420 2021-02-10T11:20:49.798 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:49.795 7facb1afd700 1 -- 172.21.2.102:0/2857396056 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7facac0b8200 con 0x7facac0a0420 2021-02-10T11:20:49.798 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:49.795 7faca3fff700 1 -- 172.21.2.102:0/2857396056 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fac940024d0 con 0x7facac0a0420 2021-02-10T11:20:49.799 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:49.795 7facb1afd700 1 -- 172.21.2.102:0/2857396056 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7facac0b8790 con 0x7facac0a0420 2021-02-10T11:20:49.799 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:49.796 7faca3fff700 1 -- 172.21.2.102:0/2857396056 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fac94000c90 con 0x7facac0a0420 2021-02-10T11:20:49.799 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:49.796 7faca3fff700 1 --2- 172.21.2.102:0/2857396056 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fac900186c0 0x7fac9001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:49.799 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:49.796 7faca3fff700 1 -- 172.21.2.102:0/2857396056 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fac940325d0 con 0x7facac0a0420 2021-02-10T11:20:49.799 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:49.796 7facaaffd700 1 --2- 172.21.2.102:0/2857396056 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fac900186c0 0x7fac9001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:49.799 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:49.796 7facaaffd700 1 --2- 172.21.2.102:0/2857396056 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fac900186c0 0x7fac9001abc0 crc :-1 s=READY pgs=401 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:20:49.820 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:49.826 7facb1afd700 1 -- 172.21.2.102:0/2857396056 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7facac0a1360 con 0x7facac0a0420 2021-02-10T11:20:49.823 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:49.828 7faca3fff700 1 -- 172.21.2.102:0/2857396056 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fac94030d80 con 0x7facac0a0420 2021-02-10T11:20:49.952 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:49.958 7facb1afd700 1 -- 172.21.2.102:0/2857396056 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344378", "target": ["mgr", ""]}) v1 -- 0x7facac0b9020 con 0x7fac900186c0 2021-02-10T11:20:49.963 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:49.968 7faca3fff700 1 -- 172.21.2.102:0/2857396056 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fac9c001a00 con 0x7fac900186c0 2021-02-10T11:20:49.969 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:49.974 7facb1afd700 1 -- 172.21.2.102:0/2857396056 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fac900186c0 msgr2=0x7fac9001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:49.970 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:49.974 7facb1afd700 1 --2- 172.21.2.102:0/2857396056 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fac900186c0 0x7fac9001abc0 crc :-1 s=READY pgs=401 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:49.970 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:49.974 7facb1afd700 1 -- 172.21.2.102:0/2857396056 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7facac0a0420 msgr2=0x7facac0a0820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:49.970 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:49.974 7facb1afd700 1 --2- 172.21.2.102:0/2857396056 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7facac0a0420 0x7facac0a0820 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 rx=0x7fac9400b5e0 tx=0x7fac9400b980).stop 2021-02-10T11:20:49.970 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:49.975 7facb1afd700 1 -- 172.21.2.102:0/2857396056 shutdown_connections 2021-02-10T11:20:49.970 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:49.975 7facb1afd700 1 --2- 172.21.2.102:0/2857396056 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7facac18a720 0x7facac0b7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:49.970 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:49.975 7facb1afd700 1 --2- 172.21.2.102:0/2857396056 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fac900186c0 0x7fac9001abc0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:49.970 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:49.975 7facb1afd700 1 --2- 172.21.2.102:0/2857396056 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7facac189d00 0x7facac18a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:49.970 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:49.975 7facb1afd700 1 --2- 172.21.2.102:0/2857396056 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7facac0a0420 0x7facac0a0820 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:49.970 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:49.975 7facb1afd700 1 -- 172.21.2.102:0/2857396056 >> 172.21.2.102:0/2857396056 conn(0x7facac094e10 msgr2=0x7facac0a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:20:49.970 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:49.975 7facb1afd700 1 -- 172.21.2.102:0/2857396056 shutdown_connections 2021-02-10T11:20:49.970 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:49.975 7facb1afd700 1 -- 172.21.2.102:0/2857396056 wait complete. 2021-02-10T11:20:49.978 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000344379 2021-02-10T11:20:50.407 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:50.397 7f37fe9ef700 1 Processor -- start 2021-02-10T11:20:50.407 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:50.397 7f37fe9ef700 1 -- start start 2021-02-10T11:20:50.407 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:50.397 7f37fe9ef700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f37f81b0870 0x7f37f80f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:50.407 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:50.397 7f37fe9ef700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f37f80efe00 0x7f37f80f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:50.407 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:50.397 7f37fe9ef700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37f81b0c70 con 0x7f37f81b0470 2021-02-10T11:20:50.407 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:50.397 7f37fe9ef700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37f80e5ea0 con 0x7f37f80efe00 2021-02-10T11:20:50.407 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:50.397 7f37fe9ef700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37f80f0b50 con 0x7f37f81b0870 2021-02-10T11:20:50.408 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:50.397 7f37fd00c700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f37f80efe00 0x7f37f80f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:50.408 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:50.397 7f37fd00c700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f37f80efe00 0x7f37f80f0200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:53914/0 (socket says 172.21.2.102:53914) 2021-02-10T11:20:50.408 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:50.397 7f37fd00c700 1 -- 172.21.2.102:0/2338091854 learned_addr learned my addr 172.21.2.102:0/2338091854 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:20:50.408 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:50.397 7f37effff700 1 --2- 172.21.2.102:0/2338091854 >> v2:172.21.2.113:3300/0 conn(0x7f37f81b0870 0x7f37f80f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:50.408 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:50.397 7f37fd00c700 1 -- 172.21.2.102:0/2338091854 >> v2:172.21.2.113:3300/0 conn(0x7f37f81b0870 msgr2=0x7f37f80f20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:20:50.408 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:50.397 7f37fd00c700 1 --2- 172.21.2.102:0/2338091854 >> v2:172.21.2.113:3300/0 conn(0x7f37f81b0870 0x7f37f80f20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:50.408 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:50.397 7f37fd00c700 1 -- 172.21.2.102:0/2338091854 >> v1:172.21.2.113:6789/0 conn(0x7f37f81b0470 legacy=0x7f37f80ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:20:50.408 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:50.397 7f37fd00c700 1 -- 172.21.2.102:0/2338091854 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37f80f0820 con 0x7f37f80efe00 2021-02-10T11:20:50.408 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:50.397 7f37effff700 1 --2- 172.21.2.102:0/2338091854 >> v2:172.21.2.113:3300/0 conn(0x7f37f81b0870 0x7f37f80f20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:20:50.408 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:50.398 7f37fd00c700 1 --2- 172.21.2.102:0/2338091854 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f37f80efe00 0x7f37f80f0200 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 rx=0x7f37f0006f00 tx=0x7f37f0009f10).ready entity=mon.0 client_cookie=70c6db91dbd8ea21 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:20:50.408 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:50.398 7f37ef7fe700 1 -- 172.21.2.102:0/2338091854 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f37f000b3b0 con 0x7f37f80efe00 2021-02-10T11:20:50.408 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:50.398 7f37ef7fe700 1 -- 172.21.2.102:0/2338091854 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37f000b630 con 0x7f37f80efe00 2021-02-10T11:20:50.408 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:50.398 7f37ef7fe700 1 -- 172.21.2.102:0/2338091854 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f37f000bb50 con 0x7f37f80efe00 2021-02-10T11:20:50.408 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:50.398 7f37fe9ef700 1 -- 172.21.2.102:0/2338091854 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f37f80efe00 msgr2=0x7f37f80f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:50.409 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:50.398 7f37fe9ef700 1 --2- 172.21.2.102:0/2338091854 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f37f80efe00 0x7f37f80f0200 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 rx=0x7f37f0006f00 tx=0x7f37f0009f10).stop 2021-02-10T11:20:50.409 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:50.398 7f37fe9ef700 1 -- 172.21.2.102:0/2338091854 shutdown_connections 2021-02-10T11:20:50.409 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:50.398 7f37fe9ef700 1 --2- 172.21.2.102:0/2338091854 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f37f80efe00 0x7f37f80f0200 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:50.409 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:50.398 7f37fe9ef700 1 --2- 172.21.2.102:0/2338091854 >> v2:172.21.2.113:3300/0 conn(0x7f37f81b0870 0x7f37f80f20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:50.409 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:50.398 7f37fe9ef700 1 -- 172.21.2.102:0/2338091854 >> 172.21.2.102:0/2338091854 conn(0x7f37f80ce3e0 msgr2=0x7f37f80e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:20:50.409 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:50.398 7f37fe9ef700 1 -- 172.21.2.102:0/2338091854 shutdown_connections 2021-02-10T11:20:50.409 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:50.398 7f37fe9ef700 1 -- 172.21.2.102:0/2338091854 wait complete. 2021-02-10T11:20:50.410 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:50.413 7f37fe9ef700 1 Processor -- start 2021-02-10T11:20:50.411 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:50.413 7f37fe9ef700 1 -- start start 2021-02-10T11:20:50.411 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:50.413 7f37fe9ef700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f37f8190920 0x7f37f808f600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:50.411 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:50.413 7f37fe9ef700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f37f8190d20 0x7f37f808fc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:50.411 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:50.413 7f37fe9ef700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f37f8193270 0x7f37f8193670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:50.411 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:50.413 7f37fe9ef700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37f8090470 con 0x7f37f8190d20 2021-02-10T11:20:50.411 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:50.413 7f37fe9ef700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37f8193c90 con 0x7f37f8193270 2021-02-10T11:20:50.411 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:50.413 7f37fe9ef700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37f8193e70 con 0x7f37f8190920 2021-02-10T11:20:50.411 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:50.413 7f37effff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f37f8190d20 0x7f37f808fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:50.411 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:50.413 7f37effff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f37f8190d20 0x7f37f808fc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:53920/0 (socket says 172.21.2.102:53920) 2021-02-10T11:20:50.411 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:50.413 7f37effff700 1 -- 172.21.2.102:0/706893114 learned_addr learned my addr 172.21.2.102:0/706893114 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:20:50.411 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:50.414 7f37effff700 1 -- 172.21.2.102:0/706893114 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f37f8190920 msgr2=0x7f37f808f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:50.411 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:50.414 7f37effff700 1 --2- 172.21.2.102:0/706893114 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f37f8190920 0x7f37f808f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:20:50.412 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:50.414 7f37fd00c700 1 --2- 172.21.2.102:0/706893114 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f37f8193270 0x7f37f8193670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:50.412 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:50.414 7f37effff700 1 -- 172.21.2.102:0/706893114 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f37f8193270 msgr2=0x7f37f8193670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:50.412 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:50.414 7f37effff700 1 --2- 172.21.2.102:0/706893114 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f37f8193270 0x7f37f8193670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:50.412 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:50.414 7f37effff700 1 -- 172.21.2.102:0/706893114 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37f8194090 con 0x7f37f8190d20 2021-02-10T11:20:50.412 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:50.414 7f37effff700 1 --2- 172.21.2.102:0/706893114 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f37f8190d20 0x7f37f808fc20 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 rx=0x7f37f400b580 tx=0x7f37f400b920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:20:50.412 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:50.414 7f37ed7fa700 1 -- 172.21.2.102:0/706893114 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f37f400bd10 con 0x7f37f8190d20 2021-02-10T11:20:50.412 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:50.414 7f37ed7fa700 1 -- 172.21.2.102:0/706893114 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37f4000c90 con 0x7f37f8190d20 2021-02-10T11:20:50.412 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:50.414 7f37ed7fa700 1 -- 172.21.2.102:0/706893114 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f37f40022b0 con 0x7f37f8190d20 2021-02-10T11:20:50.412 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:50.414 7f37fe9ef700 1 -- 172.21.2.102:0/706893114 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37f8194090 con 0x7f37f8190d20 2021-02-10T11:20:50.412 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:50.414 7f37fe9ef700 1 -- 172.21.2.102:0/706893114 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37f8194090 con 0x7f37f8190d20 2021-02-10T11:20:50.412 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:50.415 7f37ed7fa700 1 -- 172.21.2.102:0/706893114 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f37f400a000 con 0x7f37f8190d20 2021-02-10T11:20:50.412 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:50.415 7f37ed7fa700 1 --2- 172.21.2.102:0/706893114 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f37d80186d0 0x7f37d801abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:50.412 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:50.415 7f37ed7fa700 1 -- 172.21.2.102:0/706893114 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f37f40327b0 con 0x7f37f8190d20 2021-02-10T11:20:50.412 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:50.415 7f37fc80b700 1 --2- 172.21.2.102:0/706893114 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f37d80186d0 0x7f37d801abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:50.413 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:50.415 7f37fc80b700 1 --2- 172.21.2.102:0/706893114 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f37d80186d0 0x7f37d801abd0 crc :-1 s=READY pgs=402 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:20:50.441 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:50.446 7f37fe9ef700 1 -- 172.21.2.102:0/706893114 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37f8105af0 con 0x7f37f8190d20 2021-02-10T11:20:50.443 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:50.449 7f37ed7fa700 1 -- 172.21.2.102:0/706893114 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f37f4006bb0 con 0x7f37f8190d20 2021-02-10T11:20:50.571 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:50.576 7f37fe9ef700 1 -- 172.21.2.102:0/706893114 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344379", "target": ["mgr", ""]}) v1 -- 0x7f37f8105af0 con 0x7f37d80186d0 2021-02-10T11:20:50.583 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:50.588 7f37ed7fa700 1 -- 172.21.2.102:0/706893114 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f37e8006e40 con 0x7f37d80186d0 2021-02-10T11:20:50.588 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:50.593 7f37fe9ef700 1 -- 172.21.2.102:0/706893114 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f37d80186d0 msgr2=0x7f37d801abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:50.588 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:50.593 7f37fe9ef700 1 --2- 172.21.2.102:0/706893114 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f37d80186d0 0x7f37d801abd0 crc :-1 s=READY pgs=402 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:50.588 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:50.593 7f37fe9ef700 1 -- 172.21.2.102:0/706893114 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f37f8190d20 msgr2=0x7f37f808fc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:50.588 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:50.593 7f37fe9ef700 1 --2- 172.21.2.102:0/706893114 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f37f8190d20 0x7f37f808fc20 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 rx=0x7f37f400b580 tx=0x7f37f400b920).stop 2021-02-10T11:20:50.589 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:50.594 7f37fe9ef700 1 -- 172.21.2.102:0/706893114 shutdown_connections 2021-02-10T11:20:50.589 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:50.594 7f37fe9ef700 1 --2- 172.21.2.102:0/706893114 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f37d80186d0 0x7f37d801abd0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:50.589 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:50.594 7f37fe9ef700 1 --2- 172.21.2.102:0/706893114 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f37f8193270 0x7f37f8193670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:50.589 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:50.594 7f37fe9ef700 1 --2- 172.21.2.102:0/706893114 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f37f8190d20 0x7f37f808fc20 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:50.589 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:50.594 7f37fe9ef700 1 --2- 172.21.2.102:0/706893114 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f37f8190920 0x7f37f808f600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:20:50.589 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:50.594 7f37fe9ef700 1 -- 172.21.2.102:0/706893114 >> 172.21.2.102:0/706893114 conn(0x7f37f80bf4e0 msgr2=0x7f37f80be3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:20:50.589 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:50.594 7f37fe9ef700 1 -- 172.21.2.102:0/706893114 shutdown_connections 2021-02-10T11:20:50.589 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:50.594 7f37fe9ef700 1 -- 172.21.2.102:0/706893114 wait complete. 2021-02-10T11:20:50.600 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000344380 2021-02-10T11:20:51.030 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:51.019 7f5f18c14700 1 Processor -- start 2021-02-10T11:20:51.030 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:51.019 7f5f18c14700 1 -- start start 2021-02-10T11:20:51.030 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:51.019 7f5f18c14700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f5f141b0470 0x7f5f141b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:51.030 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:51.019 7f5f18c14700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5f140ef610 0x7f5f140efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:51.031 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:51.019 7f5f18c14700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f140f0b50 con 0x7f5f140f0030 2021-02-10T11:20:51.031 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:51.019 7f5f18c14700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f140e52e0 con 0x7f5f141b0470 2021-02-10T11:20:51.031 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:51.019 7f5f18c14700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f140e54d0 con 0x7f5f140ef610 2021-02-10T11:20:51.031 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:51.019 7f5f1285e700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f5f141b0470 0x7f5f141b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:51.031 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:51.020 7f5f1285e700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f5f141b0470 0x7f5f141b0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:53926/0 (socket says 172.21.2.102:53926) 2021-02-10T11:20:51.032 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:51.020 7f5f1285e700 1 -- 172.21.2.102:0/479426032 learned_addr learned my addr 172.21.2.102:0/479426032 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:20:51.032 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:51.020 7f5f1305f700 1 --1- 172.21.2.102:0/479426032 >> v1:172.21.2.102:6789/0 conn(0x7f5f140f0030 0x7f5f140f10d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:46150/0 (socket says 172.21.2.102:46150) 2021-02-10T11:20:51.032 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:51.020 7f5f1205d700 1 --2- 172.21.2.102:0/479426032 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5f140ef610 0x7f5f140efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:51.032 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:51.020 7f5f1285e700 1 -- 172.21.2.102:0/479426032 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5f140ef610 msgr2=0x7f5f140efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:20:51.032 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:51.020 7f5f1285e700 1 --2- 172.21.2.102:0/479426032 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5f140ef610 0x7f5f140efa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:51.033 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:51.020 7f5f1285e700 1 -- 172.21.2.102:0/479426032 >> v1:172.21.2.102:6789/0 conn(0x7f5f140f0030 legacy=0x7f5f140f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:51.033 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:51.020 7f5f1285e700 1 -- 172.21.2.102:0/479426032 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f140f0430 con 0x7f5f141b0470 2021-02-10T11:20:51.033 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:51.020 7f5f1205d700 1 --2- 172.21.2.102:0/479426032 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5f140ef610 0x7f5f140efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:20:51.033 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:51.020 7f5f1285e700 1 --2- 172.21.2.102:0/479426032 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5f141b0470 0x7f5f141b0870 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 rx=0x7f5efc006700 tx=0x7f5efc009b80).ready entity=mon.0 client_cookie=8e226871b166a2a5 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:20:51.033 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:51.020 7f5f1185c700 1 -- 172.21.2.102:0/479426032 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5efc00c0c0 con 0x7f5f141b0470 2021-02-10T11:20:51.033 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:51.020 7f5f1185c700 1 -- 172.21.2.102:0/479426032 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5efc00c440 con 0x7f5f141b0470 2021-02-10T11:20:51.034 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:51.020 7f5f1185c700 1 -- 172.21.2.102:0/479426032 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5efc00c960 con 0x7f5f141b0470 2021-02-10T11:20:51.034 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:51.020 7f5f18c14700 1 -- 172.21.2.102:0/479426032 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5f141b0470 msgr2=0x7f5f141b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:51.034 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:51.020 7f5f18c14700 1 --2- 172.21.2.102:0/479426032 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5f141b0470 0x7f5f141b0870 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 rx=0x7f5efc006700 tx=0x7f5efc009b80).stop 2021-02-10T11:20:51.034 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:51.021 7f5f18c14700 1 -- 172.21.2.102:0/479426032 shutdown_connections 2021-02-10T11:20:51.034 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:51.021 7f5f18c14700 1 --2- 172.21.2.102:0/479426032 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5f140ef610 0x7f5f140efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:51.035 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:51.021 7f5f18c14700 1 --2- 172.21.2.102:0/479426032 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5f141b0470 0x7f5f141b0870 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:51.035 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:51.021 7f5f18c14700 1 -- 172.21.2.102:0/479426032 >> 172.21.2.102:0/479426032 conn(0x7f5f140ce3e0 msgr2=0x7f5f140e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:20:51.035 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:51.021 7f5f18c14700 1 -- 172.21.2.102:0/479426032 shutdown_connections 2021-02-10T11:20:51.035 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:51.021 7f5f18c14700 1 -- 172.21.2.102:0/479426032 wait complete. 2021-02-10T11:20:51.036 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:51.035 7f5f18c14700 1 Processor -- start 2021-02-10T11:20:51.036 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:51.036 7f5f18c14700 1 -- start start 2021-02-10T11:20:51.037 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:51.036 7f5f18c14700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5f14105d10 0x7f5f1409eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:51.037 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:51.036 7f5f18c14700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5f14106110 0x7f5f1409f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:51.037 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:51.036 7f5f18c14700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5f140b2a00 0x7f5f140b2e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:51.037 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:51.036 7f5f18c14700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f1409e560 con 0x7f5f14105d10 2021-02-10T11:20:51.037 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:51.036 7f5f18c14700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f1409c090 con 0x7f5f140b2a00 2021-02-10T11:20:51.037 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:51.036 7f5f18c14700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f140b3420 con 0x7f5f14106110 2021-02-10T11:20:51.038 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:51.036 7f5f1285e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5f14105d10 0x7f5f1409eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:51.038 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:51.036 7f5f1305f700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5f140b2a00 0x7f5f140b2e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:51.038 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:51.036 7f5f1205d700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5f14106110 0x7f5f1409f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:51.038 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:51.036 7f5f1305f700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5f140b2a00 0x7f5f140b2e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.113:3300/0 says I am v2:172.21.2.102:57366/0 (socket says 172.21.2.102:57366) 2021-02-10T11:20:51.038 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:51.036 7f5f1205d700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5f14106110 0x7f5f1409f100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.113:3301/0 says I am v2:172.21.2.102:59468/0 (socket says 172.21.2.102:59468) 2021-02-10T11:20:51.038 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:51.036 7f5f1285e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5f14105d10 0x7f5f1409eae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:53934/0 (socket says 172.21.2.102:53934) 2021-02-10T11:20:51.038 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:51.036 7f5f1305f700 1 -- 172.21.2.102:0/3912700369 learned_addr learned my addr 172.21.2.102:0/3912700369 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:20:51.038 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:51.037 7f5f1285e700 1 -- 172.21.2.102:0/3912700369 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5f14106110 msgr2=0x7f5f1409f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:51.038 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:51.037 7f5f1285e700 1 --2- 172.21.2.102:0/3912700369 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5f14106110 0x7f5f1409f100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:51.038 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:51.037 7f5f1285e700 1 -- 172.21.2.102:0/3912700369 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5f140b2a00 msgr2=0x7f5f140b2e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:51.038 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:51.037 7f5f1285e700 1 --2- 172.21.2.102:0/3912700369 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5f140b2a00 0x7f5f140b2e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:51.038 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:51.037 7f5f1285e700 1 -- 172.21.2.102:0/3912700369 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f140b3640 con 0x7f5f14105d10 2021-02-10T11:20:51.039 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:51.037 7f5f1285e700 1 --2- 172.21.2.102:0/3912700369 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5f14105d10 0x7f5f1409eae0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 rx=0x7f5efc0096d0 tx=0x7f5efc00b840).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:20:51.039 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:51.037 7f5f0b7fe700 1 -- 172.21.2.102:0/3912700369 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5efc00bc30 con 0x7f5f14105d10 2021-02-10T11:20:51.039 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:51.037 7f5f0b7fe700 1 -- 172.21.2.102:0/3912700369 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5efc002030 con 0x7f5f14105d10 2021-02-10T11:20:51.039 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:51.037 7f5f0b7fe700 1 -- 172.21.2.102:0/3912700369 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5efc002550 con 0x7f5f14105d10 2021-02-10T11:20:51.039 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:51.037 7f5f18c14700 1 -- 172.21.2.102:0/3912700369 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f140b3640 con 0x7f5f14105d10 2021-02-10T11:20:51.039 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:51.037 7f5f18c14700 1 -- 172.21.2.102:0/3912700369 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f140b3640 con 0x7f5f14105d10 2021-02-10T11:20:51.039 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:51.038 7f5f0b7fe700 1 -- 172.21.2.102:0/3912700369 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f5efc002210 con 0x7f5f14105d10 2021-02-10T11:20:51.039 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:51.038 7f5f0b7fe700 1 --2- 172.21.2.102:0/3912700369 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5ef80186c0 0x7f5ef801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:51.039 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:51.038 7f5f0b7fe700 1 -- 172.21.2.102:0/3912700369 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f5efc0326a0 con 0x7f5f14105d10 2021-02-10T11:20:51.039 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:51.038 7f5f1205d700 1 --2- 172.21.2.102:0/3912700369 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5ef80186c0 0x7f5ef801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:51.039 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:51.038 7f5f1205d700 1 --2- 172.21.2.102:0/3912700369 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5ef80186c0 0x7f5ef801abc0 crc :-1 s=READY pgs=403 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:20:51.063 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:51.069 7f5f18c14700 1 -- 172.21.2.102:0/3912700369 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f140b3ec0 con 0x7f5f14105d10 2021-02-10T11:20:51.066 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:51.071 7f5f0b7fe700 1 -- 172.21.2.102:0/3912700369 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f5efc000c90 con 0x7f5f14105d10 2021-02-10T11:20:51.194 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:51.199 7f5f18c14700 1 -- 172.21.2.102:0/3912700369 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344380", "target": ["mgr", ""]}) v1 -- 0x7f5f141a16c0 con 0x7f5ef80186c0 2021-02-10T11:20:51.205 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:51.211 7f5f0b7fe700 1 -- 172.21.2.102:0/3912700369 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f5f04006540 con 0x7f5ef80186c0 2021-02-10T11:20:51.211 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:51.216 7f5f18c14700 1 -- 172.21.2.102:0/3912700369 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5ef80186c0 msgr2=0x7f5ef801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:51.211 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:51.216 7f5f18c14700 1 --2- 172.21.2.102:0/3912700369 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5ef80186c0 0x7f5ef801abc0 crc :-1 s=READY pgs=403 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:51.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:51.216 7f5f18c14700 1 -- 172.21.2.102:0/3912700369 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5f14105d10 msgr2=0x7f5f1409eae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:51.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:51.216 7f5f18c14700 1 --2- 172.21.2.102:0/3912700369 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5f14105d10 0x7f5f1409eae0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 rx=0x7f5efc0096d0 tx=0x7f5efc00b840).stop 2021-02-10T11:20:51.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:51.216 7f5f18c14700 1 -- 172.21.2.102:0/3912700369 shutdown_connections 2021-02-10T11:20:51.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:51.216 7f5f18c14700 1 --2- 172.21.2.102:0/3912700369 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5ef80186c0 0x7f5ef801abc0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:51.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:51.216 7f5f18c14700 1 --2- 172.21.2.102:0/3912700369 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5f140b2a00 0x7f5f140b2e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:51.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:51.216 7f5f18c14700 1 --2- 172.21.2.102:0/3912700369 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5f14106110 0x7f5f1409f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:51.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:51.216 7f5f18c14700 1 --2- 172.21.2.102:0/3912700369 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5f14105d10 0x7f5f1409eae0 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:51.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:51.216 7f5f18c14700 1 -- 172.21.2.102:0/3912700369 >> 172.21.2.102:0/3912700369 conn(0x7f5f140c36f0 msgr2=0x7f5f140bd640 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:20:51.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:51.216 7f5f18c14700 1 -- 172.21.2.102:0/3912700369 shutdown_connections 2021-02-10T11:20:51.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:51.216 7f5f18c14700 1 -- 172.21.2.102:0/3912700369 wait complete. 2021-02-10T11:20:51.220 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000344381 2021-02-10T11:20:51.653 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:51.641 7f0d8151a700 1 Processor -- start 2021-02-10T11:20:51.653 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:51.641 7f0d8151a700 1 -- start start 2021-02-10T11:20:51.653 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:51.642 7f0d8151a700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0d7c1b0870 0x7f0d7c0f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:51.653 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:51.642 7f0d8151a700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f0d7c0efe00 0x7f0d7c0f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:51.653 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:51.642 7f0d8151a700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d7c1b0c70 con 0x7f0d7c1b0470 2021-02-10T11:20:51.653 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:51.642 7f0d8151a700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d7c0f0b50 con 0x7f0d7c0efe00 2021-02-10T11:20:51.653 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:51.642 7f0d8151a700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d7c0f0820 con 0x7f0d7c1b0870 2021-02-10T11:20:51.654 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:51.642 7f0d7b2bf700 1 --1- >> v1:172.21.2.113:6789/0 conn(0x7f0d7c1b0470 0x7f0d7c0ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.113:6789/0 says I am v1:172.21.2.102:53488/0 (socket says 172.21.2.102:53488) 2021-02-10T11:20:51.654 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:51.642 7f0d7b2bf700 1 -- 172.21.2.102:0/3428558962 learned_addr learned my addr 172.21.2.102:0/3428558962 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:20:51.654 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:51.642 7f0d7aabe700 1 --2- 172.21.2.102:0/3428558962 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0d7c1b0870 0x7f0d7c0f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:51.654 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:51.642 7f0d7bac0700 1 --2- 172.21.2.102:0/3428558962 >> v2:172.21.2.113:3300/0 conn(0x7f0d7c0efe00 0x7f0d7c0f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:51.654 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:51.642 7f0d7a2bd700 1 -- 172.21.2.102:0/3428558962 <== mon.1 v1:172.21.2.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2411340124 0 0) 0x7f0d6c000c60 con 0x7f0d7c1b0470 2021-02-10T11:20:51.654 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:51.642 7f0d7a2bd700 1 -- 172.21.2.102:0/3428558962 --> v1:172.21.2.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d68002fe0 con 0x7f0d7c1b0470 2021-02-10T11:20:51.654 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:51.642 7f0d7aabe700 1 -- 172.21.2.102:0/3428558962 >> v2:172.21.2.113:3300/0 conn(0x7f0d7c0efe00 msgr2=0x7f0d7c0f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:20:51.654 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:51.642 7f0d7aabe700 1 --2- 172.21.2.102:0/3428558962 >> v2:172.21.2.113:3300/0 conn(0x7f0d7c0efe00 0x7f0d7c0f0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:51.654 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:51.642 7f0d7aabe700 1 -- 172.21.2.102:0/3428558962 >> v1:172.21.2.113:6789/0 conn(0x7f0d7c1b0470 legacy=0x7f0d7c0ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:51.654 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:51.642 7f0d7aabe700 1 -- 172.21.2.102:0/3428558962 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d7c0d2fb0 con 0x7f0d7c1b0870 2021-02-10T11:20:51.654 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:51.643 7f0d7aabe700 1 --2- 172.21.2.102:0/3428558962 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0d7c1b0870 0x7f0d7c0f20e0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f0d64006650 tx=0x7f0d64009a80).ready entity=mon.2 client_cookie=aa0e05ccd5080e3c server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:20:51.654 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:51.643 7f0d7a2bd700 1 -- 172.21.2.102:0/3428558962 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0d6400c0c0 con 0x7f0d7c1b0870 2021-02-10T11:20:51.655 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:51.643 7f0d7a2bd700 1 -- 172.21.2.102:0/3428558962 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d6400c340 con 0x7f0d7c1b0870 2021-02-10T11:20:51.655 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:51.643 7f0d7a2bd700 1 -- 172.21.2.102:0/3428558962 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0d6400c0c0 con 0x7f0d7c1b0870 2021-02-10T11:20:51.655 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:51.643 7f0d8151a700 1 -- 172.21.2.102:0/3428558962 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0d7c1b0870 msgr2=0x7f0d7c0f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:51.655 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:51.643 7f0d8151a700 1 --2- 172.21.2.102:0/3428558962 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0d7c1b0870 0x7f0d7c0f20e0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f0d64006650 tx=0x7f0d64009a80).stop 2021-02-10T11:20:51.655 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:51.644 7f0d8151a700 1 -- 172.21.2.102:0/3428558962 shutdown_connections 2021-02-10T11:20:51.655 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:51.644 7f0d8151a700 1 --2- 172.21.2.102:0/3428558962 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0d7c1b0870 0x7f0d7c0f20e0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:51.655 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:51.644 7f0d8151a700 1 --2- 172.21.2.102:0/3428558962 >> v2:172.21.2.113:3300/0 conn(0x7f0d7c0efe00 0x7f0d7c0f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:51.655 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:51.644 7f0d8151a700 1 -- 172.21.2.102:0/3428558962 >> 172.21.2.102:0/3428558962 conn(0x7f0d7c0ce3e0 msgr2=0x7f0d7c0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:20:51.655 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:51.644 7f0d8151a700 1 -- 172.21.2.102:0/3428558962 shutdown_connections 2021-02-10T11:20:51.655 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:51.644 7f0d8151a700 1 -- 172.21.2.102:0/3428558962 wait complete. 2021-02-10T11:20:51.657 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:51.659 7f0d8151a700 1 Processor -- start 2021-02-10T11:20:51.657 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:51.659 7f0d8151a700 1 -- start start 2021-02-10T11:20:51.657 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:51.659 7f0d8151a700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0d7c190920 0x7f0d7c0b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:51.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:51.659 7f0d8151a700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0d7c0b8200 0x7f0d7c0b8600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:51.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:51.659 7f0d8151a700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0d7c0c1d80 0x7f0d7c0c2180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:51.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:51.659 7f0d8151a700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d7c190da0 con 0x7f0d7c190920 2021-02-10T11:20:51.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:51.659 7f0d8151a700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d7c0b8c20 con 0x7f0d7c0b8200 2021-02-10T11:20:51.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:51.659 7f0d8151a700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d7c0b8e10 con 0x7f0d7c0c1d80 2021-02-10T11:20:51.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:51.660 7f0d7b2bf700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0d7c190920 0x7f0d7c0b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:51.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:51.660 7f0d7aabe700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0d7c0b8200 0x7f0d7c0b8600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:51.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:51.660 7f0d7b2bf700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0d7c190920 0x7f0d7c0b7be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:53946/0 (socket says 172.21.2.102:53946) 2021-02-10T11:20:51.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:51.660 7f0d7bac0700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0d7c0c1d80 0x7f0d7c0c2180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:51.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:51.660 7f0d7b2bf700 1 -- 172.21.2.102:0/3940675751 learned_addr learned my addr 172.21.2.102:0/3940675751 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:20:51.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:51.660 7f0d7bac0700 1 -- 172.21.2.102:0/3940675751 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0d7c0b8200 msgr2=0x7f0d7c0b8600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:51.659 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:51.660 7f0d7bac0700 1 --2- 172.21.2.102:0/3940675751 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0d7c0b8200 0x7f0d7c0b8600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:51.659 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:51.660 7f0d7bac0700 1 -- 172.21.2.102:0/3940675751 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0d7c190920 msgr2=0x7f0d7c0b7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:51.659 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:51.660 7f0d7bac0700 1 --2- 172.21.2.102:0/3940675751 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0d7c190920 0x7f0d7c0b7be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:51.659 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:51.660 7f0d7bac0700 1 -- 172.21.2.102:0/3940675751 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d7c0c27d0 con 0x7f0d7c0c1d80 2021-02-10T11:20:51.659 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:51.660 7f0d7b2bf700 1 --2- 172.21.2.102:0/3940675751 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0d7c190920 0x7f0d7c0b7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:20:51.659 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:51.660 7f0d7bac0700 1 --2- 172.21.2.102:0/3940675751 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0d7c0c1d80 0x7f0d7c0c2180 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f0d7400b8a0 tx=0x7f0d7400bc40).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:20:51.659 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:51.660 7f0d73fff700 1 -- 172.21.2.102:0/3940675751 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0d74000f90 con 0x7f0d7c0c1d80 2021-02-10T11:20:51.659 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:51.660 7f0d73fff700 1 -- 172.21.2.102:0/3940675751 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d74006b10 con 0x7f0d7c0c1d80 2021-02-10T11:20:51.660 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:51.660 7f0d8151a700 1 -- 172.21.2.102:0/3940675751 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d7c0c27d0 con 0x7f0d7c0c1d80 2021-02-10T11:20:51.660 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:51.660 7f0d73fff700 1 -- 172.21.2.102:0/3940675751 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0d7400bd80 con 0x7f0d7c0c1d80 2021-02-10T11:20:51.660 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:51.661 7f0d8151a700 1 -- 172.21.2.102:0/3940675751 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d7c0c27d0 con 0x7f0d7c0c1d80 2021-02-10T11:20:51.660 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:51.661 7f0d73fff700 1 -- 172.21.2.102:0/3940675751 <== mon.2 v2:172.21.2.113:3301/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f0d740011f0 con 0x7f0d7c0c1d80 2021-02-10T11:20:51.660 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:51.661 7f0d73fff700 1 --2- 172.21.2.102:0/3940675751 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0d600186c0 0x7f0d6001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:51.660 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:51.661 7f0d73fff700 1 -- 172.21.2.102:0/3940675751 <== mon.2 v2:172.21.2.113:3301/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f0d74032950 con 0x7f0d7c0c1d80 2021-02-10T11:20:51.660 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:51.661 7f0d7b2bf700 1 --2- 172.21.2.102:0/3940675751 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0d600186c0 0x7f0d6001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:51.660 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:51.661 7f0d7b2bf700 1 --2- 172.21.2.102:0/3940675751 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0d600186c0 0x7f0d6001abc0 crc :-1 s=READY pgs=404 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:20:51.685 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:51.691 7f0d8151a700 1 -- 172.21.2.102:0/3940675751 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d7c105ae0 con 0x7f0d7c0c1d80 2021-02-10T11:20:51.688 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:51.694 7f0d73fff700 1 -- 172.21.2.102:0/3940675751 <== mon.2 v2:172.21.2.113:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f0d74006710 con 0x7f0d7c0c1d80 2021-02-10T11:20:51.819 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:51.824 7f0d8151a700 1 -- 172.21.2.102:0/3940675751 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344381", "target": ["mgr", ""]}) v1 -- 0x7f0d7c0f0b50 con 0x7f0d600186c0 2021-02-10T11:20:51.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:51.836 7f0d73fff700 1 -- 172.21.2.102:0/3940675751 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f0d6c0079d0 con 0x7f0d600186c0 2021-02-10T11:20:51.835 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:51.840 7f0d8151a700 1 -- 172.21.2.102:0/3940675751 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0d600186c0 msgr2=0x7f0d6001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:51.835 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:51.840 7f0d8151a700 1 --2- 172.21.2.102:0/3940675751 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0d600186c0 0x7f0d6001abc0 crc :-1 s=READY pgs=404 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:51.835 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:51.840 7f0d8151a700 1 -- 172.21.2.102:0/3940675751 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0d7c0c1d80 msgr2=0x7f0d7c0c2180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:51.835 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:51.840 7f0d8151a700 1 --2- 172.21.2.102:0/3940675751 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0d7c0c1d80 0x7f0d7c0c2180 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f0d7400b8a0 tx=0x7f0d7400bc40).stop 2021-02-10T11:20:51.835 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:51.840 7f0d8151a700 1 -- 172.21.2.102:0/3940675751 shutdown_connections 2021-02-10T11:20:51.835 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:51.840 7f0d8151a700 1 --2- 172.21.2.102:0/3940675751 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0d7c0c1d80 0x7f0d7c0c2180 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:51.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:51.840 7f0d8151a700 1 --2- 172.21.2.102:0/3940675751 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0d600186c0 0x7f0d6001abc0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:51.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:51.840 7f0d8151a700 1 --2- 172.21.2.102:0/3940675751 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0d7c0b8200 0x7f0d7c0b8600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:51.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:51.840 7f0d8151a700 1 --2- 172.21.2.102:0/3940675751 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0d7c190920 0x7f0d7c0b7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:51.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:51.840 7f0d8151a700 1 -- 172.21.2.102:0/3940675751 >> 172.21.2.102:0/3940675751 conn(0x7f0d7c094e10 msgr2=0x7f0d7c0a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:20:51.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:51.840 7f0d8151a700 1 -- 172.21.2.102:0/3940675751 shutdown_connections 2021-02-10T11:20:51.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:51.841 7f0d8151a700 1 -- 172.21.2.102:0/3940675751 wait complete. 2021-02-10T11:20:51.844 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000344382 2021-02-10T11:20:52.275 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:52.263 7f59e5b9d700 1 Processor -- start 2021-02-10T11:20:52.275 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:52.263 7f59e5b9d700 1 -- start start 2021-02-10T11:20:52.275 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:52.264 7f59e5b9d700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f59e00f05f0 0x7f59e00f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:52.275 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:52.264 7f59e5b9d700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59e01b0c70 con 0x7f59e01b0470 2021-02-10T11:20:52.276 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:52.264 7f59e5b9d700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59e00f0b50 con 0x7f59e01b0870 2021-02-10T11:20:52.276 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:52.264 7f59e5b9d700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59e00e5ea0 con 0x7f59e00f05f0 2021-02-10T11:20:52.276 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:52.264 7f59df7fe700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f59e01b0470 0x7f59e00ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:46178/0 (socket says 172.21.2.102:46178) 2021-02-10T11:20:52.276 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:52.264 7f59df7fe700 1 -- 172.21.2.102:0/1388089135 learned_addr learned my addr 172.21.2.102:0/1388089135 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:20:52.276 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:52.264 7f59dffff700 1 --2- 172.21.2.102:0/1388089135 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f59e00f05f0 0x7f59e00f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:52.276 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:52.264 7f59de7fc700 1 -- 172.21.2.102:0/1388089135 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2102918979 0 0) 0x7f59d0000ce0 con 0x7f59e01b0470 2021-02-10T11:20:52.276 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:52.264 7f59de7fc700 1 -- 172.21.2.102:0/1388089135 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59c8002fe0 con 0x7f59e01b0470 2021-02-10T11:20:52.276 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:52.265 7f59de7fc700 1 -- 172.21.2.102:0/1388089135 <== mon.1 v1:172.21.2.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3025463022 0 0) 0x7f59d4001080 con 0x7f59e01b0870 2021-02-10T11:20:52.276 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:52.265 7f59dffff700 1 -- 172.21.2.102:0/1388089135 >> v1:172.21.2.113:6789/0 conn(0x7f59e01b0870 legacy=0x7f59e00efe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:52.276 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:52.265 7f59dffff700 1 -- 172.21.2.102:0/1388089135 >> v1:172.21.2.102:6789/0 conn(0x7f59e01b0470 legacy=0x7f59e00ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:52.276 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:52.265 7f59dffff700 1 -- 172.21.2.102:0/1388089135 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59e00e52e0 con 0x7f59e00f05f0 2021-02-10T11:20:52.276 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:52.265 7f59dffff700 1 --2- 172.21.2.102:0/1388089135 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f59e00f05f0 0x7f59e00f20e0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f59d8006e50 tx=0x7f59d8009e70).ready entity=mon.2 client_cookie=f5d49b2f5ae1f887 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:20:52.277 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:52.265 7f59de7fc700 1 -- 172.21.2.102:0/1388089135 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f59d80066f0 con 0x7f59e00f05f0 2021-02-10T11:20:52.277 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:52.265 7f59de7fc700 1 -- 172.21.2.102:0/1388089135 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59d8003030 con 0x7f59e00f05f0 2021-02-10T11:20:52.277 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:52.265 7f59de7fc700 1 -- 172.21.2.102:0/1388089135 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f59d800b700 con 0x7f59e00f05f0 2021-02-10T11:20:52.277 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:52.265 7f59e5b9d700 1 -- 172.21.2.102:0/1388089135 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f59e00f05f0 msgr2=0x7f59e00f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:52.277 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:52.265 7f59e5b9d700 1 --2- 172.21.2.102:0/1388089135 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f59e00f05f0 0x7f59e00f20e0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f59d8006e50 tx=0x7f59d8009e70).stop 2021-02-10T11:20:52.277 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:52.266 7f59e5b9d700 1 -- 172.21.2.102:0/1388089135 shutdown_connections 2021-02-10T11:20:52.277 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:52.266 7f59e5b9d700 1 --2- 172.21.2.102:0/1388089135 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f59e00f05f0 0x7f59e00f20e0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:52.277 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:52.266 7f59e5b9d700 1 -- 172.21.2.102:0/1388089135 >> 172.21.2.102:0/1388089135 conn(0x7f59e00ce3e0 msgr2=0x7f59e00e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:20:52.277 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:52.266 7f59e5b9d700 1 -- 172.21.2.102:0/1388089135 shutdown_connections 2021-02-10T11:20:52.277 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:52.266 7f59e5b9d700 1 -- 172.21.2.102:0/1388089135 wait complete. 2021-02-10T11:20:52.279 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:52.281 7f59e5b9d700 1 Processor -- start 2021-02-10T11:20:52.279 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:52.281 7f59e5b9d700 1 -- start start 2021-02-10T11:20:52.280 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:52.281 7f59e5b9d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f59e00b4c60 0x7f59e0106fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:52.280 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:52.281 7f59e5b9d700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f59e0190630 0x7f59e0099f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:52.280 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:52.281 7f59e5b9d700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f59e0190a30 0x7f59e009b130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:52.280 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:52.281 7f59e5b9d700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59e0107920 con 0x7f59e00b4c60 2021-02-10T11:20:52.280 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:52.281 7f59e5b9d700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59e009abb0 con 0x7f59e0190a30 2021-02-10T11:20:52.280 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:52.281 7f59e5b9d700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59e00b5060 con 0x7f59e0190630 2021-02-10T11:20:52.280 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:52.282 7f59df7fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f59e00b4c60 0x7f59e0106fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:52.280 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:52.282 7f59df7fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f59e00b4c60 0x7f59e0106fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:53960/0 (socket says 172.21.2.102:53960) 2021-02-10T11:20:52.280 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:52.282 7f59df7fe700 1 -- 172.21.2.102:0/1879816997 learned_addr learned my addr 172.21.2.102:0/1879816997 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:20:52.280 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:52.282 7f59deffd700 1 --2- 172.21.2.102:0/1879816997 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f59e0190630 0x7f59e0099f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:52.280 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:52.282 7f59dffff700 1 --2- 172.21.2.102:0/1879816997 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f59e0190a30 0x7f59e009b130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:52.280 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:52.282 7f59df7fe700 1 -- 172.21.2.102:0/1879816997 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f59e0190630 msgr2=0x7f59e0099f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:52.280 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:52.282 7f59df7fe700 1 --2- 172.21.2.102:0/1879816997 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f59e0190630 0x7f59e0099f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:52.281 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:52.282 7f59df7fe700 1 -- 172.21.2.102:0/1879816997 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f59e0190a30 msgr2=0x7f59e009b130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:52.281 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:52.282 7f59df7fe700 1 --2- 172.21.2.102:0/1879816997 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f59e0190a30 0x7f59e009b130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:52.281 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:52.282 7f59df7fe700 1 -- 172.21.2.102:0/1879816997 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59e009a5b0 con 0x7f59e00b4c60 2021-02-10T11:20:52.281 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:52.282 7f59dffff700 1 --2- 172.21.2.102:0/1879816997 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f59e0190a30 0x7f59e009b130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:20:52.281 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:52.282 7f59df7fe700 1 --2- 172.21.2.102:0/1879816997 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f59e00b4c60 0x7f59e0106fe0 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 rx=0x7f59d0006f00 tx=0x7f59d000b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:20:52.281 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:52.282 7f59c7fff700 1 -- 172.21.2.102:0/1879816997 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f59d000bd30 con 0x7f59e00b4c60 2021-02-10T11:20:52.281 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:52.282 7f59c7fff700 1 -- 172.21.2.102:0/1879816997 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59d0006030 con 0x7f59e00b4c60 2021-02-10T11:20:52.281 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:52.282 7f59c7fff700 1 -- 172.21.2.102:0/1879816997 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f59d0004590 con 0x7f59e00b4c60 2021-02-10T11:20:52.281 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:52.282 7f59e5b9d700 1 -- 172.21.2.102:0/1879816997 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59e009a5b0 con 0x7f59e00b4c60 2021-02-10T11:20:52.281 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:52.282 7f59e5b9d700 1 -- 172.21.2.102:0/1879816997 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59e009a5b0 con 0x7f59e00b4c60 2021-02-10T11:20:52.281 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:52.283 7f59c7fff700 1 -- 172.21.2.102:0/1879816997 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f59d00043a0 con 0x7f59e00b4c60 2021-02-10T11:20:52.281 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:52.283 7f59c7fff700 1 --2- 172.21.2.102:0/1879816997 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f59c00186c0 0x7f59c001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:52.281 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:52.283 7f59c7fff700 1 -- 172.21.2.102:0/1879816997 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f59d0032750 con 0x7f59e00b4c60 2021-02-10T11:20:52.282 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:52.283 7f59deffd700 1 --2- 172.21.2.102:0/1879816997 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f59c00186c0 0x7f59c001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:52.282 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:52.283 7f59deffd700 1 --2- 172.21.2.102:0/1879816997 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f59c00186c0 0x7f59c001abc0 crc :-1 s=READY pgs=405 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:20:52.307 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:52.313 7f59e5b9d700 1 -- 172.21.2.102:0/1879816997 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59e0105af0 con 0x7f59e00b4c60 2021-02-10T11:20:52.310 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:52.315 7f59c7fff700 1 -- 172.21.2.102:0/1879816997 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f59d0006b90 con 0x7f59e00b4c60 2021-02-10T11:20:52.439 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:52.444 7f59e5b9d700 1 -- 172.21.2.102:0/1879816997 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344382", "target": ["mgr", ""]}) v1 -- 0x7f59e009abb0 con 0x7f59c00186c0 2021-02-10T11:20:52.452 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:52.457 7f59c7fff700 1 -- 172.21.2.102:0/1879816997 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f59d4006550 con 0x7f59c00186c0 2021-02-10T11:20:52.456 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:52.461 7f59e5b9d700 1 -- 172.21.2.102:0/1879816997 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f59c00186c0 msgr2=0x7f59c001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:52.456 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:52.461 7f59e5b9d700 1 --2- 172.21.2.102:0/1879816997 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f59c00186c0 0x7f59c001abc0 crc :-1 s=READY pgs=405 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:52.456 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:52.461 7f59e5b9d700 1 -- 172.21.2.102:0/1879816997 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f59e00b4c60 msgr2=0x7f59e0106fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:52.456 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:52.461 7f59e5b9d700 1 --2- 172.21.2.102:0/1879816997 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f59e00b4c60 0x7f59e0106fe0 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 rx=0x7f59d0006f00 tx=0x7f59d000b940).stop 2021-02-10T11:20:52.456 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:52.461 7f59e5b9d700 1 -- 172.21.2.102:0/1879816997 shutdown_connections 2021-02-10T11:20:52.456 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:52.461 7f59e5b9d700 1 --2- 172.21.2.102:0/1879816997 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f59c00186c0 0x7f59c001abc0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:52.456 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:52.461 7f59e5b9d700 1 --2- 172.21.2.102:0/1879816997 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f59e0190a30 0x7f59e009b130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:52.456 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:52.461 7f59e5b9d700 1 --2- 172.21.2.102:0/1879816997 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f59e0190630 0x7f59e0099f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:52.456 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:52.461 7f59e5b9d700 1 --2- 172.21.2.102:0/1879816997 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f59e00b4c60 0x7f59e0106fe0 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:52.456 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:52.461 7f59e5b9d700 1 -- 172.21.2.102:0/1879816997 >> 172.21.2.102:0/1879816997 conn(0x7f59e00c40c0 msgr2=0x7f59e00a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:20:52.457 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:52.461 7f59e5b9d700 1 -- 172.21.2.102:0/1879816997 shutdown_connections 2021-02-10T11:20:52.457 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:52.461 7f59e5b9d700 1 -- 172.21.2.102:0/1879816997 wait complete. 2021-02-10T11:20:52.465 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000344383 2021-02-10T11:20:52.894 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:52.882 7f0d61ed5700 1 Processor -- start 2021-02-10T11:20:52.894 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:52.882 7f0d61ed5700 1 -- start start 2021-02-10T11:20:52.894 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:52.883 7f0d61ed5700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f0d5c1b0470 0x7f0d5c1b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:52.894 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:52.883 7f0d61ed5700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0d5c0ef610 0x7f0d5c0efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:52.894 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:52.883 7f0d61ed5700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d5c0f0b50 con 0x7f0d5c0f0030 2021-02-10T11:20:52.894 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:52.883 7f0d61ed5700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d5c0e52e0 con 0x7f0d5c1b0470 2021-02-10T11:20:52.894 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:52.883 7f0d61ed5700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d5c0e54d0 con 0x7f0d5c0ef610 2021-02-10T11:20:52.894 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:52.883 7f0d5b7fe700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f0d5c1b0470 0x7f0d5c1b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:52.894 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:52.883 7f0d5b7fe700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f0d5c1b0470 0x7f0d5c1b0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:53968/0 (socket says 172.21.2.102:53968) 2021-02-10T11:20:52.895 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:52.883 7f0d5b7fe700 1 -- 172.21.2.102:0/571944804 learned_addr learned my addr 172.21.2.102:0/571944804 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:20:52.895 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:52.883 7f0d5affd700 1 --2- 172.21.2.102:0/571944804 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0d5c0ef610 0x7f0d5c0efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:52.895 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:52.883 7f0d5b7fe700 1 -- 172.21.2.102:0/571944804 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0d5c0ef610 msgr2=0x7f0d5c0efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:20:52.895 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:52.883 7f0d5b7fe700 1 --2- 172.21.2.102:0/571944804 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0d5c0ef610 0x7f0d5c0efa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:52.895 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:52.883 7f0d5b7fe700 1 -- 172.21.2.102:0/571944804 >> v1:172.21.2.113:6789/0 conn(0x7f0d5c0f0030 legacy=0x7f0d5c0f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:52.895 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:52.883 7f0d5a7fc700 1 -- 172.21.2.102:0/571944804 <== mon.1 v1:172.21.2.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2541774323 0 0) 0x7f0d54001400 con 0x7f0d5c0f0030 2021-02-10T11:20:52.895 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:52.883 7f0d5b7fe700 1 -- 172.21.2.102:0/571944804 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d5c0f0430 con 0x7f0d5c1b0470 2021-02-10T11:20:52.895 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:52.883 7f0d5affd700 1 --2- 172.21.2.102:0/571944804 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0d5c0ef610 0x7f0d5c0efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:20:52.895 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:52.883 7f0d5b7fe700 1 --2- 172.21.2.102:0/571944804 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0d5c1b0470 0x7f0d5c1b0870 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 rx=0x7f0d4c006700 tx=0x7f0d4c009b80).ready entity=mon.0 client_cookie=eecea3f0c5a37dc2 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:20:52.895 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:52.884 7f0d5a7fc700 1 -- 172.21.2.102:0/571944804 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0d4c00c0c0 con 0x7f0d5c1b0470 2021-02-10T11:20:52.895 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:52.884 7f0d5a7fc700 1 -- 172.21.2.102:0/571944804 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d4c00c440 con 0x7f0d5c1b0470 2021-02-10T11:20:52.896 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:52.884 7f0d5a7fc700 1 -- 172.21.2.102:0/571944804 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0d4c00c960 con 0x7f0d5c1b0470 2021-02-10T11:20:52.896 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:52.884 7f0d61ed5700 1 -- 172.21.2.102:0/571944804 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0d5c1b0470 msgr2=0x7f0d5c1b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:52.896 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:52.884 7f0d61ed5700 1 --2- 172.21.2.102:0/571944804 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0d5c1b0470 0x7f0d5c1b0870 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 rx=0x7f0d4c006700 tx=0x7f0d4c009b80).stop 2021-02-10T11:20:52.896 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:52.884 7f0d61ed5700 1 -- 172.21.2.102:0/571944804 shutdown_connections 2021-02-10T11:20:52.896 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:52.884 7f0d61ed5700 1 --2- 172.21.2.102:0/571944804 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0d5c0ef610 0x7f0d5c0efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:52.896 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:52.884 7f0d61ed5700 1 --2- 172.21.2.102:0/571944804 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0d5c1b0470 0x7f0d5c1b0870 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:52.897 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:52.884 7f0d61ed5700 1 -- 172.21.2.102:0/571944804 >> 172.21.2.102:0/571944804 conn(0x7f0d5c0ce3e0 msgr2=0x7f0d5c0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:20:52.897 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:52.884 7f0d61ed5700 1 -- 172.21.2.102:0/571944804 shutdown_connections 2021-02-10T11:20:52.897 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:52.884 7f0d61ed5700 1 -- 172.21.2.102:0/571944804 wait complete. 2021-02-10T11:20:52.897 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:52.899 7f0d61ed5700 1 Processor -- start 2021-02-10T11:20:52.897 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:52.899 7f0d61ed5700 1 -- start start 2021-02-10T11:20:52.898 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:52.899 7f0d61ed5700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0d5c09f170 0x7f0d5c0bb310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:52.898 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:52.899 7f0d61ed5700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0d5c09e640 0x7f0d5c09ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:52.899 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:52.899 7f0d61ed5700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0d5c0bc1d0 0x7f0d5c0bc5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:52.899 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:52.899 7f0d61ed5700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d5c0bbcd0 con 0x7f0d5c09e640 2021-02-10T11:20:52.899 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:52.899 7f0d61ed5700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d5c0bf930 con 0x7f0d5c0bc1d0 2021-02-10T11:20:52.899 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:52.899 7f0d61ed5700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d5c0bb930 con 0x7f0d5c09f170 2021-02-10T11:20:52.899 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:52.900 7f0d5affd700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0d5c09e640 0x7f0d5c09ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:52.899 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:52.900 7f0d5affd700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0d5c09e640 0x7f0d5c09ea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:53972/0 (socket says 172.21.2.102:53972) 2021-02-10T11:20:52.899 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:52.900 7f0d5affd700 1 -- 172.21.2.102:0/2322129777 learned_addr learned my addr 172.21.2.102:0/2322129777 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:20:52.899 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:52.900 7f0d5bfff700 1 --2- 172.21.2.102:0/2322129777 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0d5c0bc1d0 0x7f0d5c0bc5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:52.899 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:52.900 7f0d5b7fe700 1 --2- 172.21.2.102:0/2322129777 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0d5c09f170 0x7f0d5c0bb310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:52.899 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:52.900 7f0d5affd700 1 -- 172.21.2.102:0/2322129777 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0d5c09f170 msgr2=0x7f0d5c0bb310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:52.899 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:52.900 7f0d5affd700 1 --2- 172.21.2.102:0/2322129777 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0d5c09f170 0x7f0d5c0bb310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:52.899 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:52.900 7f0d5affd700 1 -- 172.21.2.102:0/2322129777 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0d5c0bc1d0 msgr2=0x7f0d5c0bc5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:52.900 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:52.900 7f0d5affd700 1 --2- 172.21.2.102:0/2322129777 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0d5c0bc1d0 0x7f0d5c0bc5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:52.900 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:52.900 7f0d5affd700 1 -- 172.21.2.102:0/2322129777 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d5c0bcbf0 con 0x7f0d5c09e640 2021-02-10T11:20:52.900 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:52.900 7f0d5affd700 1 --2- 172.21.2.102:0/2322129777 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0d5c09e640 0x7f0d5c09ea40 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 rx=0x7f0d44006f20 tx=0x7f0d4400b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:20:52.900 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:52.901 7f0d53fff700 1 -- 172.21.2.102:0/2322129777 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0d4400bd80 con 0x7f0d5c09e640 2021-02-10T11:20:52.900 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:52.901 7f0d53fff700 1 -- 172.21.2.102:0/2322129777 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d44000c90 con 0x7f0d5c09e640 2021-02-10T11:20:52.900 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:52.901 7f0d53fff700 1 -- 172.21.2.102:0/2322129777 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0d440022b0 con 0x7f0d5c09e640 2021-02-10T11:20:52.900 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:52.901 7f0d61ed5700 1 -- 172.21.2.102:0/2322129777 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d5c0bcbf0 con 0x7f0d5c09e640 2021-02-10T11:20:52.900 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:52.901 7f0d61ed5700 1 -- 172.21.2.102:0/2322129777 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d5c0bd180 con 0x7f0d5c09e640 2021-02-10T11:20:52.900 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:52.901 7f0d53fff700 1 -- 172.21.2.102:0/2322129777 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f0d4400a000 con 0x7f0d5c09e640 2021-02-10T11:20:52.900 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:52.901 7f0d53fff700 1 --2- 172.21.2.102:0/2322129777 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0d40018730 0x7f0d4001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:52.900 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:52.901 7f0d53fff700 1 -- 172.21.2.102:0/2322129777 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f0d44032780 con 0x7f0d5c09e640 2021-02-10T11:20:52.900 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:52.901 7f0d5b7fe700 1 --2- 172.21.2.102:0/2322129777 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0d40018730 0x7f0d4001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:52.901 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:52.902 7f0d5b7fe700 1 --2- 172.21.2.102:0/2322129777 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0d40018730 0x7f0d4001abc0 crc :-1 s=READY pgs=406 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:20:52.927 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:52.932 7f0d61ed5700 1 -- 172.21.2.102:0/2322129777 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d5c105ae0 con 0x7f0d5c09e640 2021-02-10T11:20:52.929 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:52.935 7f0d53fff700 1 -- 172.21.2.102:0/2322129777 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f0d440009b0 con 0x7f0d5c09e640 2021-02-10T11:20:53.058 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:53.063 7f0d61ed5700 1 -- 172.21.2.102:0/2322129777 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344383", "target": ["mgr", ""]}) v1 -- 0x7f0d5c0bd180 con 0x7f0d40018730 2021-02-10T11:20:53.071 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:53.076 7f0d53fff700 1 -- 172.21.2.102:0/2322129777 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f0d4c006430 con 0x7f0d40018730 2021-02-10T11:20:53.075 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:53.080 7f0d61ed5700 1 -- 172.21.2.102:0/2322129777 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0d40018730 msgr2=0x7f0d4001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:53.075 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:53.080 7f0d61ed5700 1 --2- 172.21.2.102:0/2322129777 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0d40018730 0x7f0d4001abc0 crc :-1 s=READY pgs=406 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:53.075 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:53.080 7f0d61ed5700 1 -- 172.21.2.102:0/2322129777 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0d5c09e640 msgr2=0x7f0d5c09ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:53.075 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:53.080 7f0d61ed5700 1 --2- 172.21.2.102:0/2322129777 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0d5c09e640 0x7f0d5c09ea40 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 rx=0x7f0d44006f20 tx=0x7f0d4400b990).stop 2021-02-10T11:20:53.075 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:53.080 7f0d61ed5700 1 -- 172.21.2.102:0/2322129777 shutdown_connections 2021-02-10T11:20:53.075 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:53.080 7f0d61ed5700 1 --2- 172.21.2.102:0/2322129777 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0d40018730 0x7f0d4001abc0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:53.075 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:53.080 7f0d61ed5700 1 --2- 172.21.2.102:0/2322129777 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0d5c0bc1d0 0x7f0d5c0bc5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:53.075 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:53.080 7f0d61ed5700 1 --2- 172.21.2.102:0/2322129777 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0d5c09e640 0x7f0d5c09ea40 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:53.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:53.080 7f0d61ed5700 1 --2- 172.21.2.102:0/2322129777 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0d5c09f170 0x7f0d5c0bb310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:53.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:53.080 7f0d61ed5700 1 -- 172.21.2.102:0/2322129777 >> 172.21.2.102:0/2322129777 conn(0x7f0d5c09fc10 msgr2=0x7f0d5c0b3250 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:20:53.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:53.081 7f0d61ed5700 1 -- 172.21.2.102:0/2322129777 shutdown_connections 2021-02-10T11:20:53.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:53.081 7f0d61ed5700 1 -- 172.21.2.102:0/2322129777 wait complete. 2021-02-10T11:20:53.084 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000344384 2021-02-10T11:20:53.515 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:53.503 7f7952449700 1 Processor -- start 2021-02-10T11:20:53.515 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:53.503 7f7952449700 1 -- start start 2021-02-10T11:20:53.515 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:53.504 7f7952449700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f794c1b0470 0x7f794c1b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:53.515 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:53.504 7f7952449700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f794c0f20e0 0x7f794c0f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:53.515 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:53.504 7f7952449700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f794c0f0b50 con 0x7f794c0ef610 2021-02-10T11:20:53.515 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:53.504 7f7952449700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f794c0f0820 con 0x7f794c0f20e0 2021-02-10T11:20:53.515 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:53.504 7f7952449700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f794c0f24e0 con 0x7f794c1b0470 2021-02-10T11:20:53.516 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:53.504 7f794b7fe700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f794c0ef610 0x7f794c0efa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:46206/0 (socket says 172.21.2.102:46206) 2021-02-10T11:20:53.516 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:53.504 7f794b7fe700 1 -- 172.21.2.102:0/2526559685 learned_addr learned my addr 172.21.2.102:0/2526559685 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:20:53.516 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:53.504 7f7950a66700 1 --2- 172.21.2.102:0/2526559685 >> v2:172.21.2.113:3300/0 conn(0x7f794c0f20e0 0x7f794c0f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:53.516 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:53.504 7f794bfff700 1 --2- 172.21.2.102:0/2526559685 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f794c1b0470 0x7f794c1b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:53.516 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:53.505 7f794bfff700 1 -- 172.21.2.102:0/2526559685 >> v2:172.21.2.113:3300/0 conn(0x7f794c0f20e0 msgr2=0x7f794c0f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:20:53.516 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:53.504 7f794affd700 1 -- 172.21.2.102:0/2526559685 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2403411507 0 0) 0x7f79400010b0 con 0x7f794c0ef610 2021-02-10T11:20:53.516 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:53.505 7f794bfff700 1 --2- 172.21.2.102:0/2526559685 >> v2:172.21.2.113:3300/0 conn(0x7f794c0f20e0 0x7f794c0f0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:53.516 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:53.505 7f794bfff700 1 -- 172.21.2.102:0/2526559685 >> v1:172.21.2.102:6789/0 conn(0x7f794c0ef610 legacy=0x7f794c0efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:53.516 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:53.505 7f794bfff700 1 -- 172.21.2.102:0/2526559685 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f794c0e52e0 con 0x7f794c1b0470 2021-02-10T11:20:53.516 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:53.505 7f7950a66700 1 --2- 172.21.2.102:0/2526559685 >> v2:172.21.2.113:3300/0 conn(0x7f794c0f20e0 0x7f794c0f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:20:53.516 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:53.505 7f794bfff700 1 --2- 172.21.2.102:0/2526559685 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f794c1b0470 0x7f794c1b0870 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7f793c006650 tx=0x7f793c009a80).ready entity=mon.2 client_cookie=33603db7196733e5 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:20:53.516 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:53.505 7f794affd700 1 -- 172.21.2.102:0/2526559685 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f793c00c0c0 con 0x7f794c1b0470 2021-02-10T11:20:53.516 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:53.505 7f794affd700 1 -- 172.21.2.102:0/2526559685 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f793c00c0c0 con 0x7f794c1b0470 2021-02-10T11:20:53.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:53.505 7f794affd700 1 -- 172.21.2.102:0/2526559685 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f793c00c340 con 0x7f794c1b0470 2021-02-10T11:20:53.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:53.505 7f7952449700 1 -- 172.21.2.102:0/2526559685 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f794c1b0470 msgr2=0x7f794c1b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:53.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:53.505 7f7952449700 1 --2- 172.21.2.102:0/2526559685 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f794c1b0470 0x7f794c1b0870 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7f793c006650 tx=0x7f793c009a80).stop 2021-02-10T11:20:53.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:53.506 7f7952449700 1 -- 172.21.2.102:0/2526559685 shutdown_connections 2021-02-10T11:20:53.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:53.506 7f7952449700 1 --2- 172.21.2.102:0/2526559685 >> v2:172.21.2.113:3300/0 conn(0x7f794c0f20e0 0x7f794c0f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:53.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:53.506 7f7952449700 1 --2- 172.21.2.102:0/2526559685 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f794c1b0470 0x7f794c1b0870 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:53.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:53.506 7f7952449700 1 -- 172.21.2.102:0/2526559685 >> 172.21.2.102:0/2526559685 conn(0x7f794c0ce3e0 msgr2=0x7f794c0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:20:53.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:53.506 7f7952449700 1 -- 172.21.2.102:0/2526559685 shutdown_connections 2021-02-10T11:20:53.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:53.506 7f7952449700 1 -- 172.21.2.102:0/2526559685 wait complete. 2021-02-10T11:20:53.519 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:53.521 7f7952449700 1 Processor -- start 2021-02-10T11:20:53.519 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:53.521 7f7952449700 1 -- start start 2021-02-10T11:20:53.519 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:53.521 7f7952449700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f794c09b930 0x7f794c0b7c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:53.519 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:53.521 7f7952449700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f794c0b8410 0x7f794c0b8810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:53.519 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:53.521 7f7952449700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f794c0c1f80 0x7f794c0c2380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:53.519 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:53.521 7f7952449700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f794c0c1a00 con 0x7f794c09b930 2021-02-10T11:20:53.519 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:53.521 7f7952449700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f794c0b8e30 con 0x7f794c0c1f80 2021-02-10T11:20:53.519 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:53.521 7f7952449700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f794c0b9050 con 0x7f794c0b8410 2021-02-10T11:20:53.519 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:53.521 7f794bfff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f794c09b930 0x7f794c0b7c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:53.519 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:53.521 7f794bfff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f794c09b930 0x7f794c0b7c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:53986/0 (socket says 172.21.2.102:53986) 2021-02-10T11:20:53.519 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:53.521 7f794bfff700 1 -- 172.21.2.102:0/1063706261 learned_addr learned my addr 172.21.2.102:0/1063706261 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:20:53.520 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:53.521 7f7950a66700 1 --2- 172.21.2.102:0/1063706261 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f794c0c1f80 0x7f794c0c2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:53.520 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:53.522 7f794b7fe700 1 --2- 172.21.2.102:0/1063706261 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f794c0b8410 0x7f794c0b8810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:53.520 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:53.522 7f794bfff700 1 -- 172.21.2.102:0/1063706261 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f794c0b8410 msgr2=0x7f794c0b8810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:53.520 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:53.522 7f794bfff700 1 --2- 172.21.2.102:0/1063706261 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f794c0b8410 0x7f794c0b8810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:53.520 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:53.522 7f794bfff700 1 -- 172.21.2.102:0/1063706261 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f794c0c1f80 msgr2=0x7f794c0c2380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:53.520 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:53.522 7f794bfff700 1 --2- 172.21.2.102:0/1063706261 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f794c0c1f80 0x7f794c0c2380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:53.520 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:53.522 7f794bfff700 1 -- 172.21.2.102:0/1063706261 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f794c0c29a0 con 0x7f794c09b930 2021-02-10T11:20:53.520 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:53.522 7f794bfff700 1 --2- 172.21.2.102:0/1063706261 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f794c09b930 0x7f794c0b7c20 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 rx=0x7f793c00b5e0 tx=0x7f793c00b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:20:53.520 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:53.522 7f7948ff9700 1 -- 172.21.2.102:0/1063706261 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f793c000c90 con 0x7f794c09b930 2021-02-10T11:20:53.520 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:53.522 7f7948ff9700 1 -- 172.21.2.102:0/1063706261 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f793c002030 con 0x7f794c09b930 2021-02-10T11:20:53.520 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:53.522 7f7948ff9700 1 -- 172.21.2.102:0/1063706261 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f793c002550 con 0x7f794c09b930 2021-02-10T11:20:53.520 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:53.522 7f7952449700 1 -- 172.21.2.102:0/1063706261 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f794c0c29a0 con 0x7f794c09b930 2021-02-10T11:20:53.521 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:53.522 7f7952449700 1 -- 172.21.2.102:0/1063706261 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f794c0c2f30 con 0x7f794c09b930 2021-02-10T11:20:53.521 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:53.523 7f7948ff9700 1 -- 172.21.2.102:0/1063706261 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f793c000c90 con 0x7f794c09b930 2021-02-10T11:20:53.521 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:53.523 7f7948ff9700 1 --2- 172.21.2.102:0/1063706261 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f792c0186c0 0x7f792c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:53.521 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:53.523 7f7948ff9700 1 -- 172.21.2.102:0/1063706261 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f793c0327b0 con 0x7f794c09b930 2021-02-10T11:20:53.521 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:53.523 7f794b7fe700 1 --2- 172.21.2.102:0/1063706261 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f792c0186c0 0x7f792c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:53.521 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:53.523 7f794b7fe700 1 --2- 172.21.2.102:0/1063706261 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f792c0186c0 0x7f792c01abc0 crc :-1 s=READY pgs=407 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:20:53.548 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:53.553 7f7952449700 1 -- 172.21.2.102:0/1063706261 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f794c105b30 con 0x7f794c09b930 2021-02-10T11:20:53.550 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:53.556 7f7948ff9700 1 -- 172.21.2.102:0/1063706261 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f793c0093a0 con 0x7f794c09b930 2021-02-10T11:20:53.678 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:53.684 7f7952449700 1 -- 172.21.2.102:0/1063706261 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344384", "target": ["mgr", ""]}) v1 -- 0x7f794c0f0b50 con 0x7f792c0186c0 2021-02-10T11:20:53.687 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:53.693 7f7948ff9700 1 -- 172.21.2.102:0/1063706261 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f79400009b0 con 0x7f792c0186c0 2021-02-10T11:20:53.696 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:53.701 7f7952449700 1 -- 172.21.2.102:0/1063706261 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f792c0186c0 msgr2=0x7f792c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:53.696 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:53.701 7f7952449700 1 --2- 172.21.2.102:0/1063706261 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f792c0186c0 0x7f792c01abc0 crc :-1 s=READY pgs=407 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:53.696 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:53.701 7f7952449700 1 -- 172.21.2.102:0/1063706261 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f794c09b930 msgr2=0x7f794c0b7c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:53.697 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:53.701 7f7952449700 1 --2- 172.21.2.102:0/1063706261 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f794c09b930 0x7f794c0b7c20 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 rx=0x7f793c00b5e0 tx=0x7f793c00b980).stop 2021-02-10T11:20:53.697 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:53.701 7f7952449700 1 -- 172.21.2.102:0/1063706261 shutdown_connections 2021-02-10T11:20:53.697 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:53.701 7f7952449700 1 --2- 172.21.2.102:0/1063706261 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f792c0186c0 0x7f792c01abc0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:53.697 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:53.701 7f7952449700 1 --2- 172.21.2.102:0/1063706261 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f794c0c1f80 0x7f794c0c2380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:53.697 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:53.701 7f7952449700 1 --2- 172.21.2.102:0/1063706261 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f794c0b8410 0x7f794c0b8810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:53.697 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:53.701 7f7952449700 1 --2- 172.21.2.102:0/1063706261 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f794c09b930 0x7f794c0b7c20 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:53.697 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:53.701 7f7952449700 1 -- 172.21.2.102:0/1063706261 >> 172.21.2.102:0/1063706261 conn(0x7f794c094e10 msgr2=0x7f794c0a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:20:53.697 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:53.702 7f7952449700 1 -- 172.21.2.102:0/1063706261 shutdown_connections 2021-02-10T11:20:53.697 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:53.702 7f7952449700 1 -- 172.21.2.102:0/1063706261 wait complete. 2021-02-10T11:20:53.705 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000344385 2021-02-10T11:20:54.134 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:54.122 7f2076b90700 1 Processor -- start 2021-02-10T11:20:54.135 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:54.122 7f2076b90700 1 -- start start 2021-02-10T11:20:54.135 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:54.122 7f2076b90700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f20701b0470 0x7f20701b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:54.135 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:54.122 7f2076b90700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f20700ef610 0x7f20700efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:54.135 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:54.122 7f2076b90700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20700f0b50 con 0x7f20700f0030 2021-02-10T11:20:54.135 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:54.122 7f2076b90700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20700e52e0 con 0x7f20701b0470 2021-02-10T11:20:54.135 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:54.122 7f2076b90700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20700e5490 con 0x7f20700ef610 2021-02-10T11:20:54.135 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:54.123 7f20749ac700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f20701b0470 0x7f20701b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:54.135 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:54.123 7f20749ac700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f20701b0470 0x7f20701b0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:53994/0 (socket says 172.21.2.102:53994) 2021-02-10T11:20:54.135 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:54.123 7f20749ac700 1 -- 172.21.2.102:0/3026630463 learned_addr learned my addr 172.21.2.102:0/3026630463 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:20:54.135 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:54.123 7f20751ad700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f20700f0030 0x7f20700f10d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:46222/0 (socket says 172.21.2.102:46222) 2021-02-10T11:20:54.135 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:54.123 7f206ffff700 1 --2- 172.21.2.102:0/3026630463 >> v2:172.21.2.113:3300/0 conn(0x7f20700ef610 0x7f20700efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:54.136 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:54.123 7f20749ac700 1 -- 172.21.2.102:0/3026630463 >> v2:172.21.2.113:3300/0 conn(0x7f20700ef610 msgr2=0x7f20700efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:20:54.136 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:54.123 7f20749ac700 1 --2- 172.21.2.102:0/3026630463 >> v2:172.21.2.113:3300/0 conn(0x7f20700ef610 0x7f20700efa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:54.136 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:54.123 7f20749ac700 1 -- 172.21.2.102:0/3026630463 >> v1:172.21.2.102:6789/0 conn(0x7f20700f0030 legacy=0x7f20700f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:54.136 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:54.123 7f20749ac700 1 -- 172.21.2.102:0/3026630463 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20700f0430 con 0x7f20701b0470 2021-02-10T11:20:54.136 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:54.123 7f206ffff700 1 --2- 172.21.2.102:0/3026630463 >> v2:172.21.2.113:3300/0 conn(0x7f20700ef610 0x7f20700efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:20:54.136 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:54.123 7f20749ac700 1 --2- 172.21.2.102:0/3026630463 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f20701b0470 0x7f20701b0870 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 rx=0x7f2058006700 tx=0x7f2058009b80).ready entity=mon.0 client_cookie=e6a92706bbe15a1d server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:20:54.136 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:54.124 7f206f7fe700 1 -- 172.21.2.102:0/3026630463 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f205800c0c0 con 0x7f20701b0470 2021-02-10T11:20:54.136 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:54.124 7f206f7fe700 1 -- 172.21.2.102:0/3026630463 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f205800c340 con 0x7f20701b0470 2021-02-10T11:20:54.136 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:54.124 7f206f7fe700 1 -- 172.21.2.102:0/3026630463 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f205800c860 con 0x7f20701b0470 2021-02-10T11:20:54.136 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:54.124 7f2076b90700 1 -- 172.21.2.102:0/3026630463 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f20701b0470 msgr2=0x7f20701b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:54.136 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:54.124 7f2076b90700 1 --2- 172.21.2.102:0/3026630463 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f20701b0470 0x7f20701b0870 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 rx=0x7f2058006700 tx=0x7f2058009b80).stop 2021-02-10T11:20:54.136 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:54.124 7f2076b90700 1 -- 172.21.2.102:0/3026630463 shutdown_connections 2021-02-10T11:20:54.136 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:54.124 7f2076b90700 1 --2- 172.21.2.102:0/3026630463 >> v2:172.21.2.113:3300/0 conn(0x7f20700ef610 0x7f20700efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:54.137 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:54.124 7f2076b90700 1 --2- 172.21.2.102:0/3026630463 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f20701b0470 0x7f20701b0870 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:54.137 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:54.124 7f2076b90700 1 -- 172.21.2.102:0/3026630463 >> 172.21.2.102:0/3026630463 conn(0x7f20700ce3e0 msgr2=0x7f20700e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:20:54.137 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:54.124 7f2076b90700 1 -- 172.21.2.102:0/3026630463 shutdown_connections 2021-02-10T11:20:54.137 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:54.124 7f2076b90700 1 -- 172.21.2.102:0/3026630463 wait complete. 2021-02-10T11:20:54.138 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:54.140 7f2076b90700 1 Processor -- start 2021-02-10T11:20:54.138 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:54.140 7f2076b90700 1 -- start start 2021-02-10T11:20:54.138 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:54.141 7f2076b90700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2070105d10 0x7f207009eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:54.138 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:54.141 7f2076b90700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2070106110 0x7f207009f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:54.139 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:54.141 7f2076b90700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f20700b2a00 0x7f20700b2e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:54.139 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:54.141 7f2076b90700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f207009e560 con 0x7f2070106110 2021-02-10T11:20:54.139 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:54.141 7f2076b90700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f207009c090 con 0x7f20700b2a00 2021-02-10T11:20:54.139 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:54.141 7f2076b90700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20700b3420 con 0x7f2070105d10 2021-02-10T11:20:54.139 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:54.141 7f206ffff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2070106110 0x7f207009f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:54.139 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:54.141 7f206ffff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2070106110 0x7f207009f100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:54004/0 (socket says 172.21.2.102:54004) 2021-02-10T11:20:54.139 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:54.141 7f206ffff700 1 -- 172.21.2.102:0/1918994884 learned_addr learned my addr 172.21.2.102:0/1918994884 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:20:54.139 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:54.141 7f20751ad700 1 --2- 172.21.2.102:0/1918994884 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f20700b2a00 0x7f20700b2e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:54.139 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:54.141 7f20749ac700 1 --2- 172.21.2.102:0/1918994884 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2070105d10 0x7f207009eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:54.139 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:54.141 7f20751ad700 1 -- 172.21.2.102:0/1918994884 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2070105d10 msgr2=0x7f207009eae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:54.139 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:54.141 7f20751ad700 1 --2- 172.21.2.102:0/1918994884 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2070105d10 0x7f207009eae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:54.139 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:54.141 7f20751ad700 1 -- 172.21.2.102:0/1918994884 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2070106110 msgr2=0x7f207009f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:54.139 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:54.141 7f20751ad700 1 --2- 172.21.2.102:0/1918994884 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2070106110 0x7f207009f100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:54.140 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:54.141 7f20751ad700 1 -- 172.21.2.102:0/1918994884 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20700b3640 con 0x7f20700b2a00 2021-02-10T11:20:54.140 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:54.142 7f20749ac700 1 --2- 172.21.2.102:0/1918994884 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2070105d10 0x7f207009eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:20:54.140 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:54.142 7f206ffff700 1 --2- 172.21.2.102:0/1918994884 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2070106110 0x7f207009f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:20:54.140 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:54.142 7f20751ad700 1 --2- 172.21.2.102:0/1918994884 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f20700b2a00 0x7f20700b2e00 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 rx=0x7f206800a4d0 tx=0x7f206800a870).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:20:54.140 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:54.142 7f206d7fa700 1 -- 172.21.2.102:0/1918994884 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2068000af0 con 0x7f20700b2a00 2021-02-10T11:20:54.140 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:54.142 7f206d7fa700 1 -- 172.21.2.102:0/1918994884 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f206800ac60 con 0x7f20700b2a00 2021-02-10T11:20:54.140 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:54.142 7f206d7fa700 1 -- 172.21.2.102:0/1918994884 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2068004620 con 0x7f20700b2a00 2021-02-10T11:20:54.140 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:54.142 7f2076b90700 1 -- 172.21.2.102:0/1918994884 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20700b3640 con 0x7f20700b2a00 2021-02-10T11:20:54.140 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:54.142 7f2076b90700 1 -- 172.21.2.102:0/1918994884 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20700b3640 con 0x7f20700b2a00 2021-02-10T11:20:54.140 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:54.143 7f206d7fa700 1 -- 172.21.2.102:0/1918994884 <== mon.1 v2:172.21.2.113:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f20680043a0 con 0x7f20700b2a00 2021-02-10T11:20:54.140 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:54.143 7f206d7fa700 1 --2- 172.21.2.102:0/1918994884 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f20540186d0 0x7f205401abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:54.140 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:54.143 7f206d7fa700 1 -- 172.21.2.102:0/1918994884 <== mon.1 v2:172.21.2.113:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f2068032570 con 0x7f20700b2a00 2021-02-10T11:20:54.140 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:54.143 7f20749ac700 1 --2- 172.21.2.102:0/1918994884 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f20540186d0 0x7f205401abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:54.141 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:54.143 7f20749ac700 1 --2- 172.21.2.102:0/1918994884 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f20540186d0 0x7f205401abd0 crc :-1 s=READY pgs=408 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:20:54.167 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:54.172 7f2076b90700 1 -- 172.21.2.102:0/1918994884 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20700b3ec0 con 0x7f20700b2a00 2021-02-10T11:20:54.171 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:54.176 7f206d7fa700 1 -- 172.21.2.102:0/1918994884 <== mon.1 v2:172.21.2.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f20680009b0 con 0x7f20700b2a00 2021-02-10T11:20:54.299 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:54.304 7f2076b90700 1 -- 172.21.2.102:0/1918994884 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344385", "target": ["mgr", ""]}) v1 -- 0x7f20701a16c0 con 0x7f20540186d0 2021-02-10T11:20:54.309 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:54.314 7f206d7fa700 1 -- 172.21.2.102:0/1918994884 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f20580090b0 con 0x7f20540186d0 2021-02-10T11:20:54.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:54.321 7f2076b90700 1 -- 172.21.2.102:0/1918994884 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f20540186d0 msgr2=0x7f205401abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:54.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:54.321 7f2076b90700 1 --2- 172.21.2.102:0/1918994884 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f20540186d0 0x7f205401abd0 crc :-1 s=READY pgs=408 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:54.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:54.321 7f2076b90700 1 -- 172.21.2.102:0/1918994884 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f20700b2a00 msgr2=0x7f20700b2e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:54.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:54.321 7f2076b90700 1 --2- 172.21.2.102:0/1918994884 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f20700b2a00 0x7f20700b2e00 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 rx=0x7f206800a4d0 tx=0x7f206800a870).stop 2021-02-10T11:20:54.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:54.321 7f2076b90700 1 -- 172.21.2.102:0/1918994884 shutdown_connections 2021-02-10T11:20:54.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:54.321 7f2076b90700 1 --2- 172.21.2.102:0/1918994884 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f20540186d0 0x7f205401abd0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:54.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:54.321 7f2076b90700 1 --2- 172.21.2.102:0/1918994884 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f20700b2a00 0x7f20700b2e00 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:54.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:54.321 7f2076b90700 1 --2- 172.21.2.102:0/1918994884 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2070106110 0x7f207009f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:54.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:54.321 7f2076b90700 1 --2- 172.21.2.102:0/1918994884 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2070105d10 0x7f207009eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:54.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:54.321 7f2076b90700 1 -- 172.21.2.102:0/1918994884 >> 172.21.2.102:0/1918994884 conn(0x7f20700c36f0 msgr2=0x7f20700bd640 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:20:54.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:54.321 7f2076b90700 1 -- 172.21.2.102:0/1918994884 shutdown_connections 2021-02-10T11:20:54.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:54.321 7f2076b90700 1 -- 172.21.2.102:0/1918994884 wait complete. 2021-02-10T11:20:54.326 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000344386 2021-02-10T11:20:54.760 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:54.749 7f888daeb700 1 Processor -- start 2021-02-10T11:20:54.760 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:54.749 7f888daeb700 1 -- start start 2021-02-10T11:20:54.761 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:54.749 7f888daeb700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f88881b0870 0x7f88880f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:54.761 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:54.749 7f888daeb700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f88880efe00 0x7f88880f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:54.761 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:54.749 7f888daeb700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88881b0c70 con 0x7f88881b0470 2021-02-10T11:20:54.761 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:54.749 7f888daeb700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88880f0b50 con 0x7f88880efe00 2021-02-10T11:20:54.761 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:54.749 7f888daeb700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88880f0820 con 0x7f88881b0870 2021-02-10T11:20:54.761 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:54.750 7f8887fff700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f88880efe00 0x7f88880f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:54.761 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:54.750 7f8887fff700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f88880efe00 0x7f88880f0200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.113:3300/0 says I am v2:172.21.2.102:57442/0 (socket says 172.21.2.102:57442) 2021-02-10T11:20:54.761 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:54.750 7f8887fff700 1 -- 172.21.2.102:0/2838083594 learned_addr learned my addr 172.21.2.102:0/2838083594 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:20:54.761 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:54.750 7f8886ffd700 1 --2- 172.21.2.102:0/2838083594 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f88881b0870 0x7f88880f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:54.761 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:54.750 7f8887fff700 1 -- 172.21.2.102:0/2838083594 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f88881b0870 msgr2=0x7f88880f20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:20:54.761 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:54.750 7f8887fff700 1 --2- 172.21.2.102:0/2838083594 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f88881b0870 0x7f88880f20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:54.761 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:54.750 7f8887fff700 1 -- 172.21.2.102:0/2838083594 >> v1:172.21.2.113:6789/0 conn(0x7f88881b0470 legacy=0x7f88880ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:54.761 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:54.750 7f8887fff700 1 -- 172.21.2.102:0/2838083594 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88880e52e0 con 0x7f88880efe00 2021-02-10T11:20:54.762 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:54.750 7f8887fff700 1 --2- 172.21.2.102:0/2838083594 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f88880efe00 0x7f88880f0200 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 rx=0x7f8880006aa0 tx=0x7f8880009f20).ready entity=mon.1 client_cookie=326b481678e71105 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:20:54.762 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:54.750 7f88867fc700 1 -- 172.21.2.102:0/2838083594 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f888000b5c0 con 0x7f88880efe00 2021-02-10T11:20:54.762 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:54.750 7f88867fc700 1 -- 172.21.2.102:0/2838083594 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8880003030 con 0x7f88880efe00 2021-02-10T11:20:54.762 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:54.750 7f88867fc700 1 -- 172.21.2.102:0/2838083594 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f888000b9f0 con 0x7f88880efe00 2021-02-10T11:20:54.762 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:54.750 7f888daeb700 1 -- 172.21.2.102:0/2838083594 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f88880efe00 msgr2=0x7f88880f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:54.762 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:54.750 7f888daeb700 1 --2- 172.21.2.102:0/2838083594 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f88880efe00 0x7f88880f0200 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 rx=0x7f8880006aa0 tx=0x7f8880009f20).stop 2021-02-10T11:20:54.762 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:54.751 7f888daeb700 1 -- 172.21.2.102:0/2838083594 shutdown_connections 2021-02-10T11:20:54.762 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:54.751 7f888daeb700 1 --2- 172.21.2.102:0/2838083594 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f88881b0870 0x7f88880f20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:54.762 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:54.751 7f888daeb700 1 --2- 172.21.2.102:0/2838083594 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f88880efe00 0x7f88880f0200 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:54.762 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:54.751 7f888daeb700 1 -- 172.21.2.102:0/2838083594 >> 172.21.2.102:0/2838083594 conn(0x7f88880ce3e0 msgr2=0x7f88880e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:20:54.762 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:54.751 7f888daeb700 1 -- 172.21.2.102:0/2838083594 shutdown_connections 2021-02-10T11:20:54.762 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:54.751 7f888daeb700 1 -- 172.21.2.102:0/2838083594 wait complete. 2021-02-10T11:20:54.762 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:54.766 7f888daeb700 1 Processor -- start 2021-02-10T11:20:54.763 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:54.766 7f888daeb700 1 -- start start 2021-02-10T11:20:54.764 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:54.766 7f888daeb700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8888190920 0x7f88880b81a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:54.764 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:54.766 7f888daeb700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f88880b87c0 0x7f88880b70c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:54.764 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:54.766 7f888daeb700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f88880b8bc0 0x7f88880b76e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:54.764 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:54.766 7f888daeb700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8888190d20 con 0x7f88880b87c0 2021-02-10T11:20:54.764 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:54.766 7f888daeb700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88880b5ad0 con 0x7f88880b8bc0 2021-02-10T11:20:54.765 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:54.766 7f888daeb700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88880b7d00 con 0x7f8888190920 2021-02-10T11:20:54.765 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:54.766 7f8886ffd700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f88880b87c0 0x7f88880b70c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:54.765 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:54.766 7f8887fff700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f88880b8bc0 0x7f88880b76e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:54.765 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:54.766 7f8886ffd700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f88880b87c0 0x7f88880b70c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:54014/0 (socket says 172.21.2.102:54014) 2021-02-10T11:20:54.765 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:54.766 7f8886ffd700 1 -- 172.21.2.102:0/2419630041 learned_addr learned my addr 172.21.2.102:0/2419630041 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:20:54.765 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:54.766 7f88877fe700 1 --2- 172.21.2.102:0/2419630041 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8888190920 0x7f88880b81a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:54.765 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:54.767 7f8887fff700 1 -- 172.21.2.102:0/2419630041 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8888190920 msgr2=0x7f88880b81a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:54.765 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:54.767 7f8887fff700 1 --2- 172.21.2.102:0/2419630041 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8888190920 0x7f88880b81a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:54.765 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:54.767 7f8887fff700 1 -- 172.21.2.102:0/2419630041 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f88880b87c0 msgr2=0x7f88880b70c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:54.765 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:54.767 7f8887fff700 1 --2- 172.21.2.102:0/2419630041 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f88880b87c0 0x7f88880b70c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:54.765 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:54.767 7f8887fff700 1 -- 172.21.2.102:0/2419630041 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88880a4590 con 0x7f88880b8bc0 2021-02-10T11:20:54.765 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:54.767 7f8887fff700 1 --2- 172.21.2.102:0/2419630041 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f88880b8bc0 0x7f88880b76e0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 rx=0x7f888000b890 tx=0x7f888000bc30).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:20:54.766 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:54.767 7f887ffff700 1 -- 172.21.2.102:0/2419630041 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f888000bd70 con 0x7f88880b8bc0 2021-02-10T11:20:54.766 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:54.767 7f887ffff700 1 -- 172.21.2.102:0/2419630041 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8880006bb0 con 0x7f88880b8bc0 2021-02-10T11:20:54.766 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:54.767 7f887ffff700 1 -- 172.21.2.102:0/2419630041 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8880002220 con 0x7f88880b8bc0 2021-02-10T11:20:54.766 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:54.767 7f888daeb700 1 -- 172.21.2.102:0/2419630041 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88880a4590 con 0x7f88880b8bc0 2021-02-10T11:20:54.766 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:54.767 7f888daeb700 1 -- 172.21.2.102:0/2419630041 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88880a4590 con 0x7f88880b8bc0 2021-02-10T11:20:54.766 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:54.768 7f887ffff700 1 -- 172.21.2.102:0/2419630041 <== mon.1 v2:172.21.2.113:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f888000bd70 con 0x7f88880b8bc0 2021-02-10T11:20:54.766 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:54.768 7f887ffff700 1 --2- 172.21.2.102:0/2419630041 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f886c0186c0 0x7f886c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:54.766 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:54.768 7f887ffff700 1 -- 172.21.2.102:0/2419630041 <== mon.1 v2:172.21.2.113:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f88800327e0 con 0x7f88880b8bc0 2021-02-10T11:20:54.766 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:54.768 7f88877fe700 1 --2- 172.21.2.102:0/2419630041 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f886c0186c0 0x7f886c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:54.766 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:54.768 7f88877fe700 1 --2- 172.21.2.102:0/2419630041 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f886c0186c0 0x7f886c01abc0 crc :-1 s=READY pgs=409 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:20:54.793 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:54.799 7f888daeb700 1 -- 172.21.2.102:0/2419630041 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8888105ae0 con 0x7f88880b8bc0 2021-02-10T11:20:54.797 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:54.802 7f887ffff700 1 -- 172.21.2.102:0/2419630041 <== mon.1 v2:172.21.2.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f8880025f50 con 0x7f88880b8bc0 2021-02-10T11:20:54.929 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:54.934 7f888daeb700 1 -- 172.21.2.102:0/2419630041 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344386", "target": ["mgr", ""]}) v1 -- 0x7f88880f0b50 con 0x7f886c0186c0 2021-02-10T11:20:54.941 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:54.946 7f887ffff700 1 -- 172.21.2.102:0/2419630041 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f8870006a20 con 0x7f886c0186c0 2021-02-10T11:20:54.946 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:54.951 7f888daeb700 1 -- 172.21.2.102:0/2419630041 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f886c0186c0 msgr2=0x7f886c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:54.946 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:54.951 7f888daeb700 1 --2- 172.21.2.102:0/2419630041 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f886c0186c0 0x7f886c01abc0 crc :-1 s=READY pgs=409 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:54.946 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:54.951 7f888daeb700 1 -- 172.21.2.102:0/2419630041 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f88880b8bc0 msgr2=0x7f88880b76e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:54.947 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:54.951 7f888daeb700 1 --2- 172.21.2.102:0/2419630041 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f88880b8bc0 0x7f88880b76e0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 rx=0x7f888000b890 tx=0x7f888000bc30).stop 2021-02-10T11:20:54.947 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:54.951 7f888daeb700 1 -- 172.21.2.102:0/2419630041 shutdown_connections 2021-02-10T11:20:54.947 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:54.951 7f888daeb700 1 --2- 172.21.2.102:0/2419630041 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f886c0186c0 0x7f886c01abc0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:54.947 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:54.951 7f888daeb700 1 --2- 172.21.2.102:0/2419630041 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f88880b8bc0 0x7f88880b76e0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:54.947 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:54.951 7f888daeb700 1 --2- 172.21.2.102:0/2419630041 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f88880b87c0 0x7f88880b70c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:54.947 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:54.951 7f888daeb700 1 --2- 172.21.2.102:0/2419630041 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8888190920 0x7f88880b81a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:54.947 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:54.951 7f888daeb700 1 -- 172.21.2.102:0/2419630041 >> 172.21.2.102:0/2419630041 conn(0x7f888808ffe0 msgr2=0x7f88880bf050 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:20:54.947 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:54.952 7f888daeb700 1 -- 172.21.2.102:0/2419630041 shutdown_connections 2021-02-10T11:20:54.947 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:54.952 7f888daeb700 1 -- 172.21.2.102:0/2419630041 wait complete. 2021-02-10T11:20:54.956 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000344387 2021-02-10T11:20:55.388 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:55.377 7f0ad18b2700 1 Processor -- start 2021-02-10T11:20:55.388 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:55.377 7f0ad18b2700 1 -- start start 2021-02-10T11:20:55.388 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:55.377 7f0ad18b2700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f0acc1b0470 0x7f0acc1b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:55.388 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:55.377 7f0ad18b2700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0acc0ef610 0x7f0acc0efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:55.388 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:55.377 7f0ad18b2700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0acc0f0b50 con 0x7f0acc0f0030 2021-02-10T11:20:55.388 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:55.377 7f0ad18b2700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0acc0e52e0 con 0x7f0acc1b0470 2021-02-10T11:20:55.389 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:55.377 7f0ad18b2700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0acc0e54d0 con 0x7f0acc0ef610 2021-02-10T11:20:55.389 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:55.377 7f0acb574700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f0acc1b0470 0x7f0acc1b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:55.389 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:55.377 7f0acbd75700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f0acc0f0030 0x7f0acc0f10d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:46246/0 (socket says 172.21.2.102:46246) 2021-02-10T11:20:55.389 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:55.377 7f0acbd75700 1 -- 172.21.2.102:0/3811127505 learned_addr learned my addr 172.21.2.102:0/3811127505 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:20:55.389 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:55.378 7f0acad73700 1 --2- 172.21.2.102:0/3811127505 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0acc0ef610 0x7f0acc0efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:55.389 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:55.378 7f0acb574700 1 -- 172.21.2.102:0/3811127505 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0acc0ef610 msgr2=0x7f0acc0efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:20:55.389 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:55.378 7f0acb574700 1 --2- 172.21.2.102:0/3811127505 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0acc0ef610 0x7f0acc0efa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:55.389 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:55.378 7f0acb574700 1 -- 172.21.2.102:0/3811127505 >> v1:172.21.2.102:6789/0 conn(0x7f0acc0f0030 legacy=0x7f0acc0f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:55.389 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:55.378 7f0acb574700 1 -- 172.21.2.102:0/3811127505 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0acc0f0430 con 0x7f0acc1b0470 2021-02-10T11:20:55.389 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:55.378 7f0acb574700 1 --2- 172.21.2.102:0/3811127505 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0acc1b0470 0x7f0acc1b0870 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 rx=0x7f0ab40069d0 tx=0x7f0ab4009e40).ready entity=mon.0 client_cookie=1643884e7e45eaae server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:20:55.389 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:55.378 7f0aca572700 1 -- 172.21.2.102:0/3811127505 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0ab400b510 con 0x7f0acc1b0470 2021-02-10T11:20:55.389 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:55.378 7f0aca572700 1 -- 172.21.2.102:0/3811127505 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ab400b890 con 0x7f0acc1b0470 2021-02-10T11:20:55.389 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:55.378 7f0aca572700 1 -- 172.21.2.102:0/3811127505 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0ab400bcf0 con 0x7f0acc1b0470 2021-02-10T11:20:55.390 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:55.378 7f0ad18b2700 1 -- 172.21.2.102:0/3811127505 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0acc1b0470 msgr2=0x7f0acc1b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:55.390 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:55.378 7f0ad18b2700 1 --2- 172.21.2.102:0/3811127505 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0acc1b0470 0x7f0acc1b0870 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 rx=0x7f0ab40069d0 tx=0x7f0ab4009e40).stop 2021-02-10T11:20:55.390 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:55.379 7f0ad18b2700 1 -- 172.21.2.102:0/3811127505 shutdown_connections 2021-02-10T11:20:55.390 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:55.379 7f0ad18b2700 1 --2- 172.21.2.102:0/3811127505 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0acc0ef610 0x7f0acc0efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:55.390 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:55.379 7f0ad18b2700 1 --2- 172.21.2.102:0/3811127505 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0acc1b0470 0x7f0acc1b0870 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:55.390 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:55.379 7f0ad18b2700 1 -- 172.21.2.102:0/3811127505 >> 172.21.2.102:0/3811127505 conn(0x7f0acc0ce3e0 msgr2=0x7f0acc0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:20:55.390 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:55.379 7f0ad18b2700 1 -- 172.21.2.102:0/3811127505 shutdown_connections 2021-02-10T11:20:55.390 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:55.379 7f0ad18b2700 1 -- 172.21.2.102:0/3811127505 wait complete. 2021-02-10T11:20:55.392 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:55.394 7f0ad18b2700 1 Processor -- start 2021-02-10T11:20:55.392 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:55.394 7f0ad18b2700 1 -- start start 2021-02-10T11:20:55.392 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:55.394 7f0ad18b2700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0acc105d10 0x7f0acc09eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:55.392 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:55.394 7f0ad18b2700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0acc106110 0x7f0acc09f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:55.392 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:55.394 7f0ad18b2700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0acc0b2a00 0x7f0acc0b2e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:55.392 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:55.394 7f0ad18b2700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0acc09e560 con 0x7f0acc0b2a00 2021-02-10T11:20:55.392 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:55.394 7f0ad18b2700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0acc09c090 con 0x7f0acc105d10 2021-02-10T11:20:55.392 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:55.394 7f0ad18b2700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0acc0b3420 con 0x7f0acc106110 2021-02-10T11:20:55.392 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:55.394 7f0acbd75700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0acc0b2a00 0x7f0acc0b2e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:55.392 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:55.394 7f0acbd75700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0acc0b2a00 0x7f0acc0b2e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:54026/0 (socket says 172.21.2.102:54026) 2021-02-10T11:20:55.392 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:55.394 7f0acbd75700 1 -- 172.21.2.102:0/686028303 learned_addr learned my addr 172.21.2.102:0/686028303 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:20:55.393 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:55.395 7f0acb574700 1 --2- 172.21.2.102:0/686028303 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0acc105d10 0x7f0acc09eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:55.393 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:55.395 7f0acad73700 1 --2- 172.21.2.102:0/686028303 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0acc106110 0x7f0acc09f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:55.393 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:55.395 7f0acbd75700 1 -- 172.21.2.102:0/686028303 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0acc106110 msgr2=0x7f0acc09f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:55.393 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:55.395 7f0acbd75700 1 --2- 172.21.2.102:0/686028303 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0acc106110 0x7f0acc09f100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:55.393 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:55.395 7f0acbd75700 1 -- 172.21.2.102:0/686028303 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0acc105d10 msgr2=0x7f0acc09eae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:55.393 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:55.395 7f0acbd75700 1 --2- 172.21.2.102:0/686028303 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0acc105d10 0x7f0acc09eae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:55.393 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:55.395 7f0acbd75700 1 -- 172.21.2.102:0/686028303 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0acc0b3640 con 0x7f0acc0b2a00 2021-02-10T11:20:55.393 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:55.395 7f0acbd75700 1 --2- 172.21.2.102:0/686028303 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0acc0b2a00 0x7f0acc0b2e00 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 rx=0x7f0ac4001410 tx=0x7f0ac400a950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:20:55.393 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:55.395 7f0ac3fff700 1 -- 172.21.2.102:0/686028303 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0ac400ad40 con 0x7f0acc0b2a00 2021-02-10T11:20:55.393 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:55.395 7f0ac3fff700 1 -- 172.21.2.102:0/686028303 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ac40070d0 con 0x7f0acc0b2a00 2021-02-10T11:20:55.393 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:55.395 7f0ac3fff700 1 -- 172.21.2.102:0/686028303 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0ac4004620 con 0x7f0acc0b2a00 2021-02-10T11:20:55.393 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:55.395 7f0ad18b2700 1 -- 172.21.2.102:0/686028303 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0acc0b3640 con 0x7f0acc0b2a00 2021-02-10T11:20:55.393 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:55.395 7f0ad18b2700 1 -- 172.21.2.102:0/686028303 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0acc0b3640 con 0x7f0acc0b2a00 2021-02-10T11:20:55.394 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:55.396 7f0ac3fff700 1 -- 172.21.2.102:0/686028303 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f0ac40009b0 con 0x7f0acc0b2a00 2021-02-10T11:20:55.394 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:55.396 7f0ac3fff700 1 --2- 172.21.2.102:0/686028303 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0ab00186c0 0x7f0ab001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:55.394 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:55.396 7f0ac3fff700 1 -- 172.21.2.102:0/686028303 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f0ac4032580 con 0x7f0acc0b2a00 2021-02-10T11:20:55.394 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:55.396 7f0acb574700 1 --2- 172.21.2.102:0/686028303 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0ab00186c0 0x7f0ab001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:55.394 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:55.396 7f0acb574700 1 --2- 172.21.2.102:0/686028303 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0ab00186c0 0x7f0ab001abc0 crc :-1 s=READY pgs=410 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:20:55.421 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:55.426 7f0ad18b2700 1 -- 172.21.2.102:0/686028303 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0acc0b3ec0 con 0x7f0acc0b2a00 2021-02-10T11:20:55.423 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:55.429 7f0ac3fff700 1 -- 172.21.2.102:0/686028303 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f0ac4030d30 con 0x7f0acc0b2a00 2021-02-10T11:20:55.551 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:55.556 7f0ad18b2700 1 -- 172.21.2.102:0/686028303 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344387", "target": ["mgr", ""]}) v1 -- 0x7f0acc1a16c0 con 0x7f0ab00186c0 2021-02-10T11:20:55.562 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:55.568 7f0ac3fff700 1 -- 172.21.2.102:0/686028303 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f0ab4006e60 con 0x7f0ab00186c0 2021-02-10T11:20:55.568 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:55.573 7f0ad18b2700 1 -- 172.21.2.102:0/686028303 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0ab00186c0 msgr2=0x7f0ab001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:55.568 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:55.573 7f0ad18b2700 1 --2- 172.21.2.102:0/686028303 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0ab00186c0 0x7f0ab001abc0 crc :-1 s=READY pgs=410 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:55.568 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:55.573 7f0ad18b2700 1 -- 172.21.2.102:0/686028303 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0acc0b2a00 msgr2=0x7f0acc0b2e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:55.568 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:55.573 7f0ad18b2700 1 --2- 172.21.2.102:0/686028303 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0acc0b2a00 0x7f0acc0b2e00 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 rx=0x7f0ac4001410 tx=0x7f0ac400a950).stop 2021-02-10T11:20:55.569 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:55.573 7f0ad18b2700 1 -- 172.21.2.102:0/686028303 shutdown_connections 2021-02-10T11:20:55.569 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:55.574 7f0ad18b2700 1 --2- 172.21.2.102:0/686028303 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0acc0b2a00 0x7f0acc0b2e00 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:55.569 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:55.574 7f0ad18b2700 1 --2- 172.21.2.102:0/686028303 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0acc106110 0x7f0acc09f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:55.569 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:55.574 7f0ad18b2700 1 --2- 172.21.2.102:0/686028303 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0ab00186c0 0x7f0ab001abc0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:55.569 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:55.574 7f0ad18b2700 1 --2- 172.21.2.102:0/686028303 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0acc105d10 0x7f0acc09eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:55.569 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:55.574 7f0ad18b2700 1 -- 172.21.2.102:0/686028303 >> 172.21.2.102:0/686028303 conn(0x7f0acc0c36f0 msgr2=0x7f0acc0bd640 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:20:55.569 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:55.574 7f0ad18b2700 1 -- 172.21.2.102:0/686028303 shutdown_connections 2021-02-10T11:20:55.569 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:55.574 7f0ad18b2700 1 -- 172.21.2.102:0/686028303 wait complete. 2021-02-10T11:20:55.577 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000344388 2021-02-10T11:20:56.009 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:55.997 7fd565e49700 1 Processor -- start 2021-02-10T11:20:56.009 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:55.997 7fd565e49700 1 -- start start 2021-02-10T11:20:56.009 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:55.998 7fd565e49700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fd5601b0870 0x7fd5600f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:56.009 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:55.998 7fd565e49700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5601b0c70 con 0x7fd5601b0470 2021-02-10T11:20:56.009 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:55.998 7fd565e49700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5600f0b50 con 0x7fd5600efe00 2021-02-10T11:20:56.009 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:55.998 7fd565e49700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5600e5ea0 con 0x7fd5601b0870 2021-02-10T11:20:56.009 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:55.998 7fd55f7fe700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fd5601b0470 0x7fd5600ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:46258/0 (socket says 172.21.2.102:46258) 2021-02-10T11:20:56.009 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:55.998 7fd55f7fe700 1 -- 172.21.2.102:0/3964886058 learned_addr learned my addr 172.21.2.102:0/3964886058 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:20:56.009 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:55.998 7fd55effd700 1 --2- 172.21.2.102:0/3964886058 >> v2:172.21.2.113:3300/0 conn(0x7fd5601b0870 0x7fd5600f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:56.010 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:55.999 7fd55e7fc700 1 -- 172.21.2.102:0/3964886058 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2412195481 0 0) 0x7fd5500009b0 con 0x7fd5601b0470 2021-02-10T11:20:56.010 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:55.999 7fd55e7fc700 1 -- 172.21.2.102:0/3964886058 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd544002fe0 con 0x7fd5601b0470 2021-02-10T11:20:56.010 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:55.999 7fd55e7fc700 1 -- 172.21.2.102:0/3964886058 <== mon.1 v1:172.21.2.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1134437152 0 0) 0x7fd558000d30 con 0x7fd5600efe00 2021-02-10T11:20:56.010 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:55.999 7fd55e7fc700 1 -- 172.21.2.102:0/3964886058 --> v1:172.21.2.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd544004370 con 0x7fd5600efe00 2021-02-10T11:20:56.010 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:55.999 7fd55effd700 1 -- 172.21.2.102:0/3964886058 >> v1:172.21.2.113:6789/0 conn(0x7fd5600efe00 legacy=0x7fd5600f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:56.010 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:55.999 7fd55effd700 1 -- 172.21.2.102:0/3964886058 >> v1:172.21.2.102:6789/0 conn(0x7fd5601b0470 legacy=0x7fd5600ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:56.010 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:55.999 7fd55effd700 1 -- 172.21.2.102:0/3964886058 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5600e52e0 con 0x7fd5601b0870 2021-02-10T11:20:56.010 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:55.999 7fd55effd700 1 --2- 172.21.2.102:0/3964886058 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd5601b0870 0x7fd5600f20e0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 rx=0x7fd54c006650 tx=0x7fd54c009a80).ready entity=mon.1 client_cookie=8ee26a31a9cc64ea server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:20:56.010 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:55.999 7fd55e7fc700 1 -- 172.21.2.102:0/3964886058 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd54c0009b0 con 0x7fd5601b0870 2021-02-10T11:20:56.010 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:55.999 7fd55e7fc700 1 -- 172.21.2.102:0/3964886058 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd54c00c0c0 con 0x7fd5601b0870 2021-02-10T11:20:56.011 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:55.999 7fd55e7fc700 1 -- 172.21.2.102:0/3964886058 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd54c00c520 con 0x7fd5601b0870 2021-02-10T11:20:56.011 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:55.999 7fd565e49700 1 -- 172.21.2.102:0/3964886058 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd5601b0870 msgr2=0x7fd5600f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:56.011 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:55.999 7fd565e49700 1 --2- 172.21.2.102:0/3964886058 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd5601b0870 0x7fd5600f20e0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 rx=0x7fd54c006650 tx=0x7fd54c009a80).stop 2021-02-10T11:20:56.011 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:56.000 7fd565e49700 1 -- 172.21.2.102:0/3964886058 shutdown_connections 2021-02-10T11:20:56.011 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:56.000 7fd565e49700 1 --2- 172.21.2.102:0/3964886058 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd5601b0870 0x7fd5600f20e0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:56.011 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:56.000 7fd565e49700 1 -- 172.21.2.102:0/3964886058 >> 172.21.2.102:0/3964886058 conn(0x7fd5600ce3e0 msgr2=0x7fd5600e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:20:56.011 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:56.000 7fd565e49700 1 -- 172.21.2.102:0/3964886058 shutdown_connections 2021-02-10T11:20:56.011 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:56.000 7fd565e49700 1 -- 172.21.2.102:0/3964886058 wait complete. 2021-02-10T11:20:56.013 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:56.014 7fd565e49700 1 Processor -- start 2021-02-10T11:20:56.013 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:56.014 7fd565e49700 1 -- start start 2021-02-10T11:20:56.013 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:56.015 7fd565e49700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd56008ece0 0x7fd5600a6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:56.013 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:56.015 7fd565e49700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd5600a6860 0x7fd5600a6c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:56.013 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:56.015 7fd565e49700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd5600a7280 0x7fd5600a4590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:56.013 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:56.015 7fd565e49700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5600a5cc0 con 0x7fd56008ece0 2021-02-10T11:20:56.013 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:56.015 7fd565e49700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd56008f0e0 con 0x7fd5600a7280 2021-02-10T11:20:56.013 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:56.015 7fd565e49700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd56008f2c0 con 0x7fd5600a6860 2021-02-10T11:20:56.013 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:56.015 7fd55f7fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd56008ece0 0x7fd5600a6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:56.014 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:56.015 7fd55f7fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd56008ece0 0x7fd5600a6240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:54040/0 (socket says 172.21.2.102:54040) 2021-02-10T11:20:56.014 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:56.015 7fd55f7fe700 1 -- 172.21.2.102:0/1596032815 learned_addr learned my addr 172.21.2.102:0/1596032815 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:20:56.014 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:56.015 7fd55ffff700 1 --2- 172.21.2.102:0/1596032815 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd5600a7280 0x7fd5600a4590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:56.014 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:56.015 7fd55effd700 1 --2- 172.21.2.102:0/1596032815 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd5600a6860 0x7fd5600a6c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:56.014 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:56.015 7fd55f7fe700 1 -- 172.21.2.102:0/1596032815 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd5600a6860 msgr2=0x7fd5600a6c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:56.014 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:56.015 7fd55f7fe700 1 --2- 172.21.2.102:0/1596032815 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd5600a6860 0x7fd5600a6c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:56.014 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:56.015 7fd55f7fe700 1 -- 172.21.2.102:0/1596032815 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd5600a7280 msgr2=0x7fd5600a4590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:56.014 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:56.015 7fd55f7fe700 1 --2- 172.21.2.102:0/1596032815 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd5600a7280 0x7fd5600a4590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:56.014 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:56.015 7fd55f7fe700 1 -- 172.21.2.102:0/1596032815 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5600a4bb0 con 0x7fd56008ece0 2021-02-10T11:20:56.014 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:56.016 7fd55f7fe700 1 --2- 172.21.2.102:0/1596032815 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd56008ece0 0x7fd5600a6240 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 rx=0x7fd550001c60 tx=0x7fd55000a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:20:56.014 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:56.016 7fd557fff700 1 -- 172.21.2.102:0/1596032815 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd55000acc0 con 0x7fd56008ece0 2021-02-10T11:20:56.014 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:56.016 7fd557fff700 1 -- 172.21.2.102:0/1596032815 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd550004140 con 0x7fd56008ece0 2021-02-10T11:20:56.015 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:56.016 7fd557fff700 1 -- 172.21.2.102:0/1596032815 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd550004620 con 0x7fd56008ece0 2021-02-10T11:20:56.015 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:56.016 7fd565e49700 1 -- 172.21.2.102:0/1596032815 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5600a4bb0 con 0x7fd56008ece0 2021-02-10T11:20:56.015 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:56.016 7fd565e49700 1 -- 172.21.2.102:0/1596032815 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5600a5140 con 0x7fd56008ece0 2021-02-10T11:20:56.015 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:56.017 7fd557fff700 1 -- 172.21.2.102:0/1596032815 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd550007f30 con 0x7fd56008ece0 2021-02-10T11:20:56.015 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:56.017 7fd557fff700 1 --2- 172.21.2.102:0/1596032815 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd53c0186c0 0x7fd53c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:56.015 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:56.017 7fd557fff700 1 -- 172.21.2.102:0/1596032815 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fd550032510 con 0x7fd56008ece0 2021-02-10T11:20:56.015 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:56.017 7fd55effd700 1 --2- 172.21.2.102:0/1596032815 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd53c0186c0 0x7fd53c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:56.015 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:56.017 7fd55effd700 1 --2- 172.21.2.102:0/1596032815 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd53c0186c0 0x7fd53c01abc0 crc :-1 s=READY pgs=411 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:20:56.042 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:56.047 7fd565e49700 1 -- 172.21.2.102:0/1596032815 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd56019d1a0 con 0x7fd56008ece0 2021-02-10T11:20:56.045 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:56.050 7fd557fff700 1 -- 172.21.2.102:0/1596032815 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd550000940 con 0x7fd56008ece0 2021-02-10T11:20:56.172 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:56.177 7fd565e49700 1 -- 172.21.2.102:0/1596032815 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344388", "target": ["mgr", ""]}) v1 -- 0x7fd5600a5cc0 con 0x7fd53c0186c0 2021-02-10T11:20:56.183 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:56.189 7fd557fff700 1 -- 172.21.2.102:0/1596032815 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fd54c009080 con 0x7fd53c0186c0 2021-02-10T11:20:56.189 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:56.194 7fd565e49700 1 -- 172.21.2.102:0/1596032815 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd53c0186c0 msgr2=0x7fd53c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:56.189 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:56.194 7fd565e49700 1 --2- 172.21.2.102:0/1596032815 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd53c0186c0 0x7fd53c01abc0 crc :-1 s=READY pgs=411 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:56.189 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:56.194 7fd565e49700 1 -- 172.21.2.102:0/1596032815 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd56008ece0 msgr2=0x7fd5600a6240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:56.189 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:56.194 7fd565e49700 1 --2- 172.21.2.102:0/1596032815 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd56008ece0 0x7fd5600a6240 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 rx=0x7fd550001c60 tx=0x7fd55000a8d0).stop 2021-02-10T11:20:56.189 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:56.194 7fd565e49700 1 -- 172.21.2.102:0/1596032815 shutdown_connections 2021-02-10T11:20:56.189 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:56.194 7fd565e49700 1 --2- 172.21.2.102:0/1596032815 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd53c0186c0 0x7fd53c01abc0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:56.190 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:56.194 7fd565e49700 1 --2- 172.21.2.102:0/1596032815 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd5600a7280 0x7fd5600a4590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:56.190 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:56.194 7fd565e49700 1 --2- 172.21.2.102:0/1596032815 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd5600a6860 0x7fd5600a6c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:56.190 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:56.194 7fd565e49700 1 --2- 172.21.2.102:0/1596032815 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd56008ece0 0x7fd5600a6240 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:56.190 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:56.194 7fd565e49700 1 -- 172.21.2.102:0/1596032815 >> 172.21.2.102:0/1596032815 conn(0x7fd56008ffe0 msgr2=0x7fd5600bf050 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:20:56.190 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:56.194 7fd565e49700 1 -- 172.21.2.102:0/1596032815 shutdown_connections 2021-02-10T11:20:56.190 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:56.194 7fd565e49700 1 -- 172.21.2.102:0/1596032815 wait complete. 2021-02-10T11:20:56.198 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000344389 2021-02-10T11:20:56.630 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:56.618 7f8fd10aa700 1 Processor -- start 2021-02-10T11:20:56.630 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:56.618 7f8fd10aa700 1 -- start start 2021-02-10T11:20:56.630 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:56.618 7f8fd10aa700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f8fcc0f05f0 0x7f8fcc0f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:56.630 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:56.618 7f8fd10aa700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fcc1b0c70 con 0x7f8fcc1b0470 2021-02-10T11:20:56.630 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:56.618 7f8fd10aa700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fcc0f0b50 con 0x7f8fcc1b0870 2021-02-10T11:20:56.630 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:56.618 7f8fd10aa700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8fcc0e5ea0 con 0x7f8fcc0f05f0 2021-02-10T11:20:56.630 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:56.619 7f8fcad9d700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f8fcc1b0470 0x7f8fcc0ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:46270/0 (socket says 172.21.2.102:46270) 2021-02-10T11:20:56.630 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:56.619 7f8fcad9d700 1 -- 172.21.2.102:0/1761920582 learned_addr learned my addr 172.21.2.102:0/1761920582 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:20:56.630 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:56.619 7f8fcb59e700 1 --2- 172.21.2.102:0/1761920582 >> v2:172.21.2.113:3300/0 conn(0x7f8fcc0f05f0 0x7f8fcc0f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:56.630 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:56.619 7f8fc9d9b700 1 -- 172.21.2.102:0/1761920582 <== mon.1 v1:172.21.2.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3071130357 0 0) 0x7f8fc0000c60 con 0x7f8fcc1b0870 2021-02-10T11:20:56.630 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:56.619 7f8fc9d9b700 1 -- 172.21.2.102:0/1761920582 --> v1:172.21.2.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8fb4002fe0 con 0x7f8fcc1b0870 2021-02-10T11:20:56.631 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:56.619 7f8fc9d9b700 1 -- 172.21.2.102:0/1761920582 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3964920661 0 0) 0x7f8fbc001180 con 0x7f8fcc1b0470 2021-02-10T11:20:56.631 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:56.619 7f8fc9d9b700 1 -- 172.21.2.102:0/1761920582 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8fb4002fe0 con 0x7f8fcc1b0470 2021-02-10T11:20:56.631 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:56.619 7f8fcb59e700 1 -- 172.21.2.102:0/1761920582 >> v1:172.21.2.113:6789/0 conn(0x7f8fcc1b0870 legacy=0x7f8fcc0efe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:56.631 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:56.619 7f8fcb59e700 1 -- 172.21.2.102:0/1761920582 >> v1:172.21.2.102:6789/0 conn(0x7f8fcc1b0470 legacy=0x7f8fcc0ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:56.631 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:56.619 7f8fcb59e700 1 -- 172.21.2.102:0/1761920582 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fcc0e52e0 con 0x7f8fcc0f05f0 2021-02-10T11:20:56.631 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:56.619 7f8fcb59e700 1 --2- 172.21.2.102:0/1761920582 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8fcc0f05f0 0x7f8fcc0f20e0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 rx=0x7f8fc4006e50 tx=0x7f8fc4009df0).ready entity=mon.1 client_cookie=cb0e1a7769a22dd1 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:20:56.631 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:56.620 7f8fc9d9b700 1 -- 172.21.2.102:0/1761920582 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8fc400b510 con 0x7f8fcc0f05f0 2021-02-10T11:20:56.631 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:56.620 7f8fc9d9b700 1 -- 172.21.2.102:0/1761920582 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fc4003030 con 0x7f8fcc0f05f0 2021-02-10T11:20:56.631 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:56.620 7f8fc9d9b700 1 -- 172.21.2.102:0/1761920582 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8fc400b7d0 con 0x7f8fcc0f05f0 2021-02-10T11:20:56.631 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:56.620 7f8fd10aa700 1 -- 172.21.2.102:0/1761920582 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8fcc0f05f0 msgr2=0x7f8fcc0f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:56.631 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:56.620 7f8fd10aa700 1 --2- 172.21.2.102:0/1761920582 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8fcc0f05f0 0x7f8fcc0f20e0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 rx=0x7f8fc4006e50 tx=0x7f8fc4009df0).stop 2021-02-10T11:20:56.631 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:56.621 7f8fd10aa700 1 -- 172.21.2.102:0/1761920582 shutdown_connections 2021-02-10T11:20:56.631 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:56.621 7f8fd10aa700 1 --2- 172.21.2.102:0/1761920582 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8fcc0f05f0 0x7f8fcc0f20e0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:56.632 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:56.621 7f8fd10aa700 1 -- 172.21.2.102:0/1761920582 >> 172.21.2.102:0/1761920582 conn(0x7f8fcc0ce3e0 msgr2=0x7f8fcc0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:20:56.632 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:56.621 7f8fd10aa700 1 -- 172.21.2.102:0/1761920582 shutdown_connections 2021-02-10T11:20:56.632 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:56.621 7f8fd10aa700 1 -- 172.21.2.102:0/1761920582 wait complete. 2021-02-10T11:20:56.633 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:56.635 7f8fd10aa700 1 Processor -- start 2021-02-10T11:20:56.633 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:56.635 7f8fd10aa700 1 -- start start 2021-02-10T11:20:56.633 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:56.636 7f8fd10aa700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8fcc105d10 0x7f8fcc09eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:56.633 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:56.636 7f8fd10aa700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8fcc106110 0x7f8fcc09f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:56.634 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:56.636 7f8fd10aa700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8fcc0b2a00 0x7f8fcc0b2e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:56.634 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:56.636 7f8fd10aa700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fcc09e560 con 0x7f8fcc0b2a00 2021-02-10T11:20:56.634 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:56.636 7f8fd10aa700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fcc09c090 con 0x7f8fcc106110 2021-02-10T11:20:56.634 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:56.636 7f8fd10aa700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fcc0b3420 con 0x7f8fcc105d10 2021-02-10T11:20:56.634 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:56.636 7f8fcb59e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8fcc0b2a00 0x7f8fcc0b2e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:56.634 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:56.636 7f8fcb59e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8fcc0b2a00 0x7f8fcc0b2e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:54052/0 (socket says 172.21.2.102:54052) 2021-02-10T11:20:56.634 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:56.636 7f8fcb59e700 1 -- 172.21.2.102:0/3292006153 learned_addr learned my addr 172.21.2.102:0/3292006153 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:20:56.634 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:56.636 7f8fca59c700 1 --2- 172.21.2.102:0/3292006153 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8fcc106110 0x7f8fcc09f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:56.634 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:56.636 7f8fcad9d700 1 --2- 172.21.2.102:0/3292006153 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8fcc105d10 0x7f8fcc09eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:56.634 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:56.636 7f8fcb59e700 1 -- 172.21.2.102:0/3292006153 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8fcc105d10 msgr2=0x7f8fcc09eae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:56.634 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:56.636 7f8fcb59e700 1 --2- 172.21.2.102:0/3292006153 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8fcc105d10 0x7f8fcc09eae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:56.634 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:56.636 7f8fcb59e700 1 -- 172.21.2.102:0/3292006153 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8fcc106110 msgr2=0x7f8fcc09f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:56.635 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:56.636 7f8fcb59e700 1 --2- 172.21.2.102:0/3292006153 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8fcc106110 0x7f8fcc09f100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:56.635 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:56.636 7f8fcb59e700 1 -- 172.21.2.102:0/3292006153 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fcc0b3640 con 0x7f8fcc0b2a00 2021-02-10T11:20:56.635 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:56.636 7f8fca59c700 1 --2- 172.21.2.102:0/3292006153 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8fcc106110 0x7f8fcc09f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:20:56.635 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:56.636 7f8fcb59e700 1 --2- 172.21.2.102:0/3292006153 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8fcc0b2a00 0x7f8fcc0b2e00 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 rx=0x7f8fc4009f80 tx=0x7f8fc400bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:20:56.635 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:56.636 7f8fbb7fe700 1 -- 172.21.2.102:0/3292006153 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8fc40093d0 con 0x7f8fcc0b2a00 2021-02-10T11:20:56.635 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:56.637 7f8fbb7fe700 1 -- 172.21.2.102:0/3292006153 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fc4001240 con 0x7f8fcc0b2a00 2021-02-10T11:20:56.635 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:56.637 7f8fd10aa700 1 -- 172.21.2.102:0/3292006153 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fcc0b3640 con 0x7f8fcc0b2a00 2021-02-10T11:20:56.635 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:56.637 7f8fd10aa700 1 -- 172.21.2.102:0/3292006153 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fcc0b3640 con 0x7f8fcc0b2a00 2021-02-10T11:20:56.635 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:56.637 7f8fbb7fe700 1 -- 172.21.2.102:0/3292006153 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8fc4001690 con 0x7f8fcc0b2a00 2021-02-10T11:20:56.635 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:56.637 7f8fbb7fe700 1 -- 172.21.2.102:0/3292006153 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f8fc40097c0 con 0x7f8fcc0b2a00 2021-02-10T11:20:56.636 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:56.637 7f8fbb7fe700 1 --2- 172.21.2.102:0/3292006153 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8fb0018730 0x7f8fb001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:56.636 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:56.637 7f8fbb7fe700 1 -- 172.21.2.102:0/3292006153 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f8fc4032ba0 con 0x7f8fcc0b2a00 2021-02-10T11:20:56.636 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:56.637 7f8fcad9d700 1 --2- 172.21.2.102:0/3292006153 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8fb0018730 0x7f8fb001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:56.636 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:56.638 7f8fcad9d700 1 --2- 172.21.2.102:0/3292006153 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8fb0018730 0x7f8fb001ac10 crc :-1 s=READY pgs=412 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:20:56.663 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:56.668 7f8fd10aa700 1 -- 172.21.2.102:0/3292006153 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fcc0b3ec0 con 0x7f8fcc0b2a00 2021-02-10T11:20:56.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:56.671 7f8fbb7fe700 1 -- 172.21.2.102:0/3292006153 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f8fc401f8e0 con 0x7f8fcc0b2a00 2021-02-10T11:20:56.797 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:56.802 7f8fd10aa700 1 -- 172.21.2.102:0/3292006153 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344389", "target": ["mgr", ""]}) v1 -- 0x7f8fcc1a16c0 con 0x7f8fb0018730 2021-02-10T11:20:56.808 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:56.813 7f8fbb7fe700 1 -- 172.21.2.102:0/3292006153 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f8fbc000ce0 con 0x7f8fb0018730 2021-02-10T11:20:56.814 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:56.819 7f8fd10aa700 1 -- 172.21.2.102:0/3292006153 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8fb0018730 msgr2=0x7f8fb001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:56.814 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:56.819 7f8fd10aa700 1 --2- 172.21.2.102:0/3292006153 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8fb0018730 0x7f8fb001ac10 crc :-1 s=READY pgs=412 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:56.814 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:56.819 7f8fd10aa700 1 -- 172.21.2.102:0/3292006153 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8fcc0b2a00 msgr2=0x7f8fcc0b2e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:56.814 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:56.819 7f8fd10aa700 1 --2- 172.21.2.102:0/3292006153 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8fcc0b2a00 0x7f8fcc0b2e00 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 rx=0x7f8fc4009f80 tx=0x7f8fc400bd10).stop 2021-02-10T11:20:56.814 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:56.819 7f8fd10aa700 1 -- 172.21.2.102:0/3292006153 shutdown_connections 2021-02-10T11:20:56.814 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:56.819 7f8fd10aa700 1 --2- 172.21.2.102:0/3292006153 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8fcc0b2a00 0x7f8fcc0b2e00 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:56.814 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:56.819 7f8fd10aa700 1 --2- 172.21.2.102:0/3292006153 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8fb0018730 0x7f8fb001ac10 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:56.814 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:56.819 7f8fd10aa700 1 --2- 172.21.2.102:0/3292006153 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8fcc106110 0x7f8fcc09f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:56.814 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:56.819 7f8fd10aa700 1 --2- 172.21.2.102:0/3292006153 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8fcc105d10 0x7f8fcc09eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:56.814 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:56.819 7f8fd10aa700 1 -- 172.21.2.102:0/3292006153 >> 172.21.2.102:0/3292006153 conn(0x7f8fcc0c36f0 msgr2=0x7f8fcc0bd640 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:20:56.815 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:56.819 7f8fd10aa700 1 -- 172.21.2.102:0/3292006153 shutdown_connections 2021-02-10T11:20:56.815 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:56.819 7f8fd10aa700 1 -- 172.21.2.102:0/3292006153 wait complete. 2021-02-10T11:20:56.822 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000344390 2021-02-10T11:20:57.254 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:57.239 7f1298b58700 1 Processor -- start 2021-02-10T11:20:57.255 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:57.239 7f1298b58700 1 -- start start 2021-02-10T11:20:57.255 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:57.240 7f1298b58700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f12941b0470 0x7f12941b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:57.255 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:57.240 7f1298b58700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f12940f20e0 0x7f12940f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:57.255 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:57.240 7f1298b58700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12940f0b50 con 0x7f12940ef610 2021-02-10T11:20:57.256 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:57.240 7f1298b58700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12940f0820 con 0x7f12940f20e0 2021-02-10T11:20:57.256 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:57.240 7f1298b58700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12940f24e0 con 0x7f12941b0470 2021-02-10T11:20:57.256 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:57.240 7f129305f700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f12940f20e0 0x7f12940f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:57.256 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:57.240 7f129305f700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f12940f20e0 0x7f12940f0200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:54058/0 (socket says 172.21.2.102:54058) 2021-02-10T11:20:57.256 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:57.240 7f129305f700 1 -- 172.21.2.102:0/3921854361 learned_addr learned my addr 172.21.2.102:0/3921854361 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:20:57.256 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:57.240 7f129285e700 1 --2- 172.21.2.102:0/3921854361 >> v2:172.21.2.113:3300/0 conn(0x7f12941b0470 0x7f12941b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:57.257 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:57.240 7f129305f700 1 -- 172.21.2.102:0/3921854361 >> v2:172.21.2.113:3300/0 conn(0x7f12941b0470 msgr2=0x7f12941b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:20:57.257 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:57.240 7f129305f700 1 --2- 172.21.2.102:0/3921854361 >> v2:172.21.2.113:3300/0 conn(0x7f12941b0470 0x7f12941b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:57.257 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:57.240 7f129305f700 1 -- 172.21.2.102:0/3921854361 >> v1:172.21.2.113:6789/0 conn(0x7f12940ef610 legacy=0x7f12940efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:57.257 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:57.240 7f129305f700 1 -- 172.21.2.102:0/3921854361 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12940e52e0 con 0x7f12940f20e0 2021-02-10T11:20:57.258 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:57.240 7f129305f700 1 --2- 172.21.2.102:0/3921854361 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f12940f20e0 0x7f12940f0200 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 rx=0x7f128c006f00 tx=0x7f128c009f10).ready entity=mon.0 client_cookie=37cefc270cdd94e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:20:57.258 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:57.240 7f129185c700 1 -- 172.21.2.102:0/3921854361 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f128c00b3b0 con 0x7f12940f20e0 2021-02-10T11:20:57.258 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:57.240 7f129185c700 1 -- 172.21.2.102:0/3921854361 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f128c00b630 con 0x7f12940f20e0 2021-02-10T11:20:57.258 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:57.240 7f129185c700 1 -- 172.21.2.102:0/3921854361 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f128c00bb50 con 0x7f12940f20e0 2021-02-10T11:20:57.258 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:57.241 7f1298b58700 1 -- 172.21.2.102:0/3921854361 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f12940f20e0 msgr2=0x7f12940f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:57.259 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:57.241 7f1298b58700 1 --2- 172.21.2.102:0/3921854361 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f12940f20e0 0x7f12940f0200 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 rx=0x7f128c006f00 tx=0x7f128c009f10).stop 2021-02-10T11:20:57.259 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:57.241 7f1298b58700 1 -- 172.21.2.102:0/3921854361 shutdown_connections 2021-02-10T11:20:57.259 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:57.241 7f1298b58700 1 --2- 172.21.2.102:0/3921854361 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f12940f20e0 0x7f12940f0200 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:57.260 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:57.241 7f1298b58700 1 --2- 172.21.2.102:0/3921854361 >> v2:172.21.2.113:3300/0 conn(0x7f12941b0470 0x7f12941b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:57.260 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:57.241 7f1298b58700 1 -- 172.21.2.102:0/3921854361 >> 172.21.2.102:0/3921854361 conn(0x7f12940ce3e0 msgr2=0x7f12940e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:20:57.260 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:57.241 7f1298b58700 1 -- 172.21.2.102:0/3921854361 shutdown_connections 2021-02-10T11:20:57.260 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:57.241 7f1298b58700 1 -- 172.21.2.102:0/3921854361 wait complete. 2021-02-10T11:20:57.260 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:57.256 7f1298b58700 1 Processor -- start 2021-02-10T11:20:57.260 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:57.256 7f1298b58700 1 -- start start 2021-02-10T11:20:57.261 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:57.256 7f1298b58700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f12940b20b0 0x7f12940ac970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:57.261 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:57.256 7f1298b58700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f12940b24b0 0x7f1294191dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:57.261 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:57.256 7f1298b58700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f12940acf90 0x7f12941923e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:57.261 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:57.256 7f1298b58700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12940b1b30 con 0x7f12940b20b0 2021-02-10T11:20:57.261 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:57.256 7f1298b58700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12941a04e0 con 0x7f12940b24b0 2021-02-10T11:20:57.262 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:57.256 7f1298b58700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12941a06c0 con 0x7f12940acf90 2021-02-10T11:20:57.262 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:57.257 7f129285e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f12940b20b0 0x7f12940ac970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:57.262 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:57.257 7f129285e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f12940b20b0 0x7f12940ac970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:54066/0 (socket says 172.21.2.102:54066) 2021-02-10T11:20:57.262 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:57.257 7f129285e700 1 -- 172.21.2.102:0/3771333262 learned_addr learned my addr 172.21.2.102:0/3771333262 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:20:57.262 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:57.257 7f129205d700 1 --2- 172.21.2.102:0/3771333262 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f12940b24b0 0x7f1294191dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:57.262 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:57.257 7f129305f700 1 --2- 172.21.2.102:0/3771333262 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f12940acf90 0x7f12941923e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:57.263 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:57.257 7f129285e700 1 -- 172.21.2.102:0/3771333262 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f12940acf90 msgr2=0x7f12941923e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:57.263 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:57.257 7f129285e700 1 --2- 172.21.2.102:0/3771333262 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f12940acf90 0x7f12941923e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:57.263 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:57.257 7f129285e700 1 -- 172.21.2.102:0/3771333262 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f12940b24b0 msgr2=0x7f1294191dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:57.263 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:57.257 7f129285e700 1 --2- 172.21.2.102:0/3771333262 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f12940b24b0 0x7f1294191dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:57.263 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:57.257 7f129285e700 1 -- 172.21.2.102:0/3771333262 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1294192be0 con 0x7f12940b20b0 2021-02-10T11:20:57.264 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:57.257 7f129285e700 1 --2- 172.21.2.102:0/3771333262 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f12940b20b0 0x7f12940ac970 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 rx=0x7f1284006f00 tx=0x7f128400b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:20:57.264 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:57.257 7f12837fe700 1 -- 172.21.2.102:0/3771333262 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f128400bd30 con 0x7f12940b20b0 2021-02-10T11:20:57.264 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:57.257 7f12837fe700 1 -- 172.21.2.102:0/3771333262 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1284002030 con 0x7f12940b20b0 2021-02-10T11:20:57.264 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:57.257 7f1298b58700 1 -- 172.21.2.102:0/3771333262 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1294192be0 con 0x7f12940b20b0 2021-02-10T11:20:57.264 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:57.257 7f12837fe700 1 -- 172.21.2.102:0/3771333262 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1284002480 con 0x7f12940b20b0 2021-02-10T11:20:57.265 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:57.258 7f1298b58700 1 -- 172.21.2.102:0/3771333262 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1294193170 con 0x7f12940b20b0 2021-02-10T11:20:57.265 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:57.258 7f12837fe700 1 -- 172.21.2.102:0/3771333262 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1284000c90 con 0x7f12940b20b0 2021-02-10T11:20:57.265 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:57.258 7f12837fe700 1 --2- 172.21.2.102:0/3771333262 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f12780186c0 0x7f127801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:57.265 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:57.258 7f12837fe700 1 -- 172.21.2.102:0/3771333262 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f12840325a0 con 0x7f12940b20b0 2021-02-10T11:20:57.265 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:57.258 7f129205d700 1 --2- 172.21.2.102:0/3771333262 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f12780186c0 0x7f127801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:57.265 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:57.259 7f129205d700 1 --2- 172.21.2.102:0/3771333262 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f12780186c0 0x7f127801abc0 crc :-1 s=READY pgs=413 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:20:57.284 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:57.289 7f1298b58700 1 -- 172.21.2.102:0/3771333262 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1294105af0 con 0x7f12940b20b0 2021-02-10T11:20:57.286 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:57.292 7f12837fe700 1 -- 172.21.2.102:0/3771333262 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f12840009b0 con 0x7f12940b20b0 2021-02-10T11:20:57.411 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:57.416 7f1298b58700 1 -- 172.21.2.102:0/3771333262 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344390", "target": ["mgr", ""]}) v1 -- 0x7f12940f0b50 con 0x7f12780186c0 2021-02-10T11:20:57.423 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:57.428 7f12837fe700 1 -- 172.21.2.102:0/3771333262 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f1288006a90 con 0x7f12780186c0 2021-02-10T11:20:57.429 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:57.434 7f1298b58700 1 -- 172.21.2.102:0/3771333262 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f12780186c0 msgr2=0x7f127801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:57.429 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:57.434 7f1298b58700 1 --2- 172.21.2.102:0/3771333262 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f12780186c0 0x7f127801abc0 crc :-1 s=READY pgs=413 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:57.429 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:57.434 7f1298b58700 1 -- 172.21.2.102:0/3771333262 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f12940b20b0 msgr2=0x7f12940ac970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:57.430 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:57.434 7f1298b58700 1 --2- 172.21.2.102:0/3771333262 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f12940b20b0 0x7f12940ac970 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 rx=0x7f1284006f00 tx=0x7f128400b940).stop 2021-02-10T11:20:57.430 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:57.434 7f1298b58700 1 -- 172.21.2.102:0/3771333262 shutdown_connections 2021-02-10T11:20:57.430 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:57.434 7f1298b58700 1 --2- 172.21.2.102:0/3771333262 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f12940acf90 0x7f12941923e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:57.430 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:57.434 7f1298b58700 1 --2- 172.21.2.102:0/3771333262 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f12780186c0 0x7f127801abc0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:57.430 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:57.434 7f1298b58700 1 --2- 172.21.2.102:0/3771333262 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f12940b24b0 0x7f1294191dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:57.431 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:57.434 7f1298b58700 1 --2- 172.21.2.102:0/3771333262 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f12940b20b0 0x7f12940ac970 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:57.431 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:57.434 7f1298b58700 1 -- 172.21.2.102:0/3771333262 >> 172.21.2.102:0/3771333262 conn(0x7f12940bf4e0 msgr2=0x7f12940be3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:20:57.431 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:57.434 7f1298b58700 1 -- 172.21.2.102:0/3771333262 shutdown_connections 2021-02-10T11:20:57.433 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:57.434 7f1298b58700 1 -- 172.21.2.102:0/3771333262 wait complete. 2021-02-10T11:20:57.438 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000344391 2021-02-10T11:20:57.870 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:57.858 7fc60be61700 1 Processor -- start 2021-02-10T11:20:57.870 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:57.858 7fc60be61700 1 -- start start 2021-02-10T11:20:57.870 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:57.858 7fc60be61700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fc6041b0470 0x7fc6041b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:57.870 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:57.858 7fc60be61700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc6040ef610 0x7fc6040efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:57.870 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:57.858 7fc60be61700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6040f0b50 con 0x7fc6040f0030 2021-02-10T11:20:57.870 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:57.858 7fc60be61700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6040e52e0 con 0x7fc6041b0470 2021-02-10T11:20:57.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:57.858 7fc60be61700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6040e54d0 con 0x7fc6040ef610 2021-02-10T11:20:57.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:57.859 7fc609c7d700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fc6041b0470 0x7fc6041b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:57.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:57.859 7fc60a47e700 1 --1- >> v1:172.21.2.113:6789/0 conn(0x7fc6040f0030 0x7fc6040f10d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.113:6789/0 says I am v1:172.21.2.102:53622/0 (socket says 172.21.2.102:53622) 2021-02-10T11:20:57.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:57.859 7fc60a47e700 1 -- 172.21.2.102:0/1827419834 learned_addr learned my addr 172.21.2.102:0/1827419834 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:20:57.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:57.859 7fc60947c700 1 --2- 172.21.2.102:0/1827419834 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc6040ef610 0x7fc6040efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:57.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:57.859 7fc608c7b700 1 -- 172.21.2.102:0/1827419834 <== mon.1 v1:172.21.2.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3143141397 0 0) 0x7fc5fc001510 con 0x7fc6040f0030 2021-02-10T11:20:57.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:57.859 7fc60947c700 1 -- 172.21.2.102:0/1827419834 >> v2:172.21.2.113:3300/0 conn(0x7fc6041b0470 msgr2=0x7fc6041b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:20:57.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:57.859 7fc60947c700 1 --2- 172.21.2.102:0/1827419834 >> v2:172.21.2.113:3300/0 conn(0x7fc6041b0470 0x7fc6041b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:57.872 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:57.859 7fc60947c700 1 -- 172.21.2.102:0/1827419834 >> v1:172.21.2.113:6789/0 conn(0x7fc6040f0030 legacy=0x7fc6040f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:57.872 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:57.859 7fc60947c700 1 -- 172.21.2.102:0/1827419834 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6040f0430 con 0x7fc6040ef610 2021-02-10T11:20:57.872 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:57.859 7fc60947c700 1 --2- 172.21.2.102:0/1827419834 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc6040ef610 0x7fc6040efa10 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7fc600006650 tx=0x7fc600009a80).ready entity=mon.2 client_cookie=a68f0410546e14a9 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:20:57.872 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:57.860 7fc608c7b700 1 -- 172.21.2.102:0/1827419834 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc60000c0c0 con 0x7fc6040ef610 2021-02-10T11:20:57.872 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:57.860 7fc608c7b700 1 -- 172.21.2.102:0/1827419834 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc60000c340 con 0x7fc6040ef610 2021-02-10T11:20:57.872 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:57.860 7fc608c7b700 1 -- 172.21.2.102:0/1827419834 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc60000c860 con 0x7fc6040ef610 2021-02-10T11:20:57.872 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:57.860 7fc60be61700 1 -- 172.21.2.102:0/1827419834 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc6040ef610 msgr2=0x7fc6040efa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:57.872 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:57.860 7fc60be61700 1 --2- 172.21.2.102:0/1827419834 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc6040ef610 0x7fc6040efa10 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7fc600006650 tx=0x7fc600009a80).stop 2021-02-10T11:20:57.872 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:57.860 7fc60be61700 1 -- 172.21.2.102:0/1827419834 shutdown_connections 2021-02-10T11:20:57.872 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:57.860 7fc60be61700 1 --2- 172.21.2.102:0/1827419834 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc6040ef610 0x7fc6040efa10 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:57.872 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:57.860 7fc60be61700 1 --2- 172.21.2.102:0/1827419834 >> v2:172.21.2.113:3300/0 conn(0x7fc6041b0470 0x7fc6041b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:57.872 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:57.860 7fc60be61700 1 -- 172.21.2.102:0/1827419834 >> 172.21.2.102:0/1827419834 conn(0x7fc6040ce3e0 msgr2=0x7fc6040e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:20:57.873 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:57.861 7fc60be61700 1 -- 172.21.2.102:0/1827419834 shutdown_connections 2021-02-10T11:20:57.873 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:57.861 7fc60be61700 1 -- 172.21.2.102:0/1827419834 wait complete. 2021-02-10T11:20:57.873 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:57.875 7fc60be61700 1 Processor -- start 2021-02-10T11:20:57.873 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:57.875 7fc60be61700 1 -- start start 2021-02-10T11:20:57.874 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:57.876 7fc60be61700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc604190920 0x7fc6040b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:57.874 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:57.876 7fc60be61700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc6040b8200 0x7fc6040b8600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:57.875 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:57.876 7fc60be61700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc6040c1d80 0x7fc6040c2180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:57.875 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:57.876 7fc60be61700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc604190da0 con 0x7fc604190920 2021-02-10T11:20:57.875 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:57.876 7fc60be61700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6040b8c20 con 0x7fc6040b8200 2021-02-10T11:20:57.875 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:57.876 7fc60be61700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6040b8e10 con 0x7fc6040c1d80 2021-02-10T11:20:57.875 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:57.876 7fc609c7d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc604190920 0x7fc6040b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:57.875 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:57.876 7fc609c7d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc604190920 0x7fc6040b7be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:54076/0 (socket says 172.21.2.102:54076) 2021-02-10T11:20:57.875 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:57.876 7fc609c7d700 1 -- 172.21.2.102:0/1516607350 learned_addr learned my addr 172.21.2.102:0/1516607350 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:20:57.875 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:57.876 7fc60947c700 1 --2- 172.21.2.102:0/1516607350 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc6040b8200 0x7fc6040b8600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:57.875 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:57.876 7fc60a47e700 1 --2- 172.21.2.102:0/1516607350 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc6040c1d80 0x7fc6040c2180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:57.875 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:57.876 7fc609c7d700 1 -- 172.21.2.102:0/1516607350 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc6040c1d80 msgr2=0x7fc6040c2180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:57.875 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:57.876 7fc609c7d700 1 --2- 172.21.2.102:0/1516607350 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc6040c1d80 0x7fc6040c2180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:57.875 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:57.876 7fc609c7d700 1 -- 172.21.2.102:0/1516607350 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc6040b8200 msgr2=0x7fc6040b8600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:57.875 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:57.876 7fc609c7d700 1 --2- 172.21.2.102:0/1516607350 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc6040b8200 0x7fc6040b8600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:57.876 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:57.876 7fc609c7d700 1 -- 172.21.2.102:0/1516607350 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6040c27d0 con 0x7fc604190920 2021-02-10T11:20:57.876 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:57.876 7fc609c7d700 1 --2- 172.21.2.102:0/1516607350 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc604190920 0x7fc6040b7be0 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 rx=0x7fc5f8006f60 tx=0x7fc5f800b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:20:57.876 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:57.877 7fc5f67fc700 1 -- 172.21.2.102:0/1516607350 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc5f80009b0 con 0x7fc604190920 2021-02-10T11:20:57.876 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:57.877 7fc5f67fc700 1 -- 172.21.2.102:0/1516607350 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5f800bd60 con 0x7fc604190920 2021-02-10T11:20:57.876 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:57.877 7fc5f67fc700 1 -- 172.21.2.102:0/1516607350 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc5f8002030 con 0x7fc604190920 2021-02-10T11:20:57.876 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:57.877 7fc60be61700 1 -- 172.21.2.102:0/1516607350 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6040c27d0 con 0x7fc604190920 2021-02-10T11:20:57.876 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:57.877 7fc60be61700 1 -- 172.21.2.102:0/1516607350 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6040c27d0 con 0x7fc604190920 2021-02-10T11:20:57.876 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:57.877 7fc5f67fc700 1 -- 172.21.2.102:0/1516607350 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fc5f8006b00 con 0x7fc604190920 2021-02-10T11:20:57.876 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:57.877 7fc5f67fc700 1 --2- 172.21.2.102:0/1516607350 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc5ec018730 0x7fc5ec01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:57.876 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:57.877 7fc5f67fc700 1 -- 172.21.2.102:0/1516607350 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fc5f8032a30 con 0x7fc604190920 2021-02-10T11:20:57.876 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:57.877 7fc60947c700 1 --2- 172.21.2.102:0/1516607350 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc5ec018730 0x7fc5ec01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:57.876 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:57.877 7fc60947c700 1 --2- 172.21.2.102:0/1516607350 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc5ec018730 0x7fc5ec01ac10 crc :-1 s=READY pgs=414 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:20:57.903 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:57.908 7fc60be61700 1 -- 172.21.2.102:0/1516607350 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc604105ae0 con 0x7fc604190920 2021-02-10T11:20:57.905 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:57.910 7fc5f67fc700 1 -- 172.21.2.102:0/1516607350 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fc5f80009b0 con 0x7fc604190920 2021-02-10T11:20:58.031 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:58.037 7fc60be61700 1 -- 172.21.2.102:0/1516607350 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344391", "target": ["mgr", ""]}) v1 -- 0x7fc604105ae0 con 0x7fc5ec018730 2021-02-10T11:20:58.042 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:58.047 7fc5f67fc700 1 -- 172.21.2.102:0/1516607350 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fc600000c90 con 0x7fc5ec018730 2021-02-10T11:20:58.049 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:58.054 7fc60be61700 1 -- 172.21.2.102:0/1516607350 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc5ec018730 msgr2=0x7fc5ec01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:58.049 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:58.054 7fc60be61700 1 --2- 172.21.2.102:0/1516607350 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc5ec018730 0x7fc5ec01ac10 crc :-1 s=READY pgs=414 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:58.049 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:58.054 7fc60be61700 1 -- 172.21.2.102:0/1516607350 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc604190920 msgr2=0x7fc6040b7be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:58.049 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:58.054 7fc60be61700 1 --2- 172.21.2.102:0/1516607350 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc604190920 0x7fc6040b7be0 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 rx=0x7fc5f8006f60 tx=0x7fc5f800b970).stop 2021-02-10T11:20:58.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:58.054 7fc60be61700 1 -- 172.21.2.102:0/1516607350 shutdown_connections 2021-02-10T11:20:58.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:58.054 7fc60be61700 1 --2- 172.21.2.102:0/1516607350 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc6040c1d80 0x7fc6040c2180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:58.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:58.054 7fc60be61700 1 --2- 172.21.2.102:0/1516607350 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc5ec018730 0x7fc5ec01ac10 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:58.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:58.054 7fc60be61700 1 --2- 172.21.2.102:0/1516607350 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc6040b8200 0x7fc6040b8600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:58.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:58.054 7fc60be61700 1 --2- 172.21.2.102:0/1516607350 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc604190920 0x7fc6040b7be0 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:58.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:58.054 7fc60be61700 1 -- 172.21.2.102:0/1516607350 >> 172.21.2.102:0/1516607350 conn(0x7fc604094e10 msgr2=0x7fc6040a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:20:58.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:58.054 7fc60be61700 1 -- 172.21.2.102:0/1516607350 shutdown_connections 2021-02-10T11:20:58.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:58.054 7fc60be61700 1 -- 172.21.2.102:0/1516607350 wait complete. 2021-02-10T11:20:58.058 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000344392 2021-02-10T11:20:58.489 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:58.477 7f0501528700 1 Processor -- start 2021-02-10T11:20:58.489 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:58.477 7f0501528700 1 -- start start 2021-02-10T11:20:58.489 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:58.477 7f0501528700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f04fc1b0470 0x7f04fc1b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:58.489 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:58.477 7f0501528700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f04fc0ef610 0x7f04fc0efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:58.489 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:58.477 7f0501528700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04fc0f0b50 con 0x7f04fc0f0030 2021-02-10T11:20:58.489 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:58.477 7f0501528700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04fc0e52e0 con 0x7f04fc0ef610 2021-02-10T11:20:58.490 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:58.478 7f0501528700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04fc0e5490 con 0x7f04fc1b0470 2021-02-10T11:20:58.490 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:58.478 7f04faabe700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f04fc0ef610 0x7f04fc0efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:58.490 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:58.478 7f04faabe700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f04fc0ef610 0x7f04fc0efa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:54084/0 (socket says 172.21.2.102:54084) 2021-02-10T11:20:58.490 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:58.478 7f04faabe700 1 -- 172.21.2.102:0/1613298767 learned_addr learned my addr 172.21.2.102:0/1613298767 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:20:58.490 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:58.478 7f04fb2bf700 1 --2- 172.21.2.102:0/1613298767 >> v2:172.21.2.113:3300/0 conn(0x7f04fc1b0470 0x7f04fc1b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:58.490 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:58.478 7f04faabe700 1 -- 172.21.2.102:0/1613298767 >> v2:172.21.2.113:3300/0 conn(0x7f04fc1b0470 msgr2=0x7f04fc1b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:20:58.490 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:58.478 7f04faabe700 1 --2- 172.21.2.102:0/1613298767 >> v2:172.21.2.113:3300/0 conn(0x7f04fc1b0470 0x7f04fc1b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:58.490 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:58.478 7f04faabe700 1 -- 172.21.2.102:0/1613298767 >> v1:172.21.2.113:6789/0 conn(0x7f04fc0f0030 legacy=0x7f04fc0f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:58.490 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:58.478 7f04faabe700 1 -- 172.21.2.102:0/1613298767 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04fc0f0430 con 0x7f04fc0ef610 2021-02-10T11:20:58.490 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:58.478 7f04faabe700 1 --2- 172.21.2.102:0/1613298767 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f04fc0ef610 0x7f04fc0efa10 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 rx=0x7f04e4006700 tx=0x7f04e4009b80).ready entity=mon.0 client_cookie=4ace8d4051fde7d4 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:20:58.490 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:58.478 7f04fa2bd700 1 -- 172.21.2.102:0/1613298767 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f04e400c0c0 con 0x7f04fc0ef610 2021-02-10T11:20:58.491 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:58.479 7f04fa2bd700 1 -- 172.21.2.102:0/1613298767 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04e400c440 con 0x7f04fc0ef610 2021-02-10T11:20:58.491 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:58.479 7f04fa2bd700 1 -- 172.21.2.102:0/1613298767 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f04e400c8a0 con 0x7f04fc0ef610 2021-02-10T11:20:58.491 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:58.479 7f0501528700 1 -- 172.21.2.102:0/1613298767 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f04fc0ef610 msgr2=0x7f04fc0efa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:58.491 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:58.479 7f0501528700 1 --2- 172.21.2.102:0/1613298767 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f04fc0ef610 0x7f04fc0efa10 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 rx=0x7f04e4006700 tx=0x7f04e4009b80).stop 2021-02-10T11:20:58.491 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:58.479 7f0501528700 1 -- 172.21.2.102:0/1613298767 shutdown_connections 2021-02-10T11:20:58.491 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:58.479 7f0501528700 1 --2- 172.21.2.102:0/1613298767 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f04fc0ef610 0x7f04fc0efa10 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:58.491 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:58.479 7f0501528700 1 --2- 172.21.2.102:0/1613298767 >> v2:172.21.2.113:3300/0 conn(0x7f04fc1b0470 0x7f04fc1b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:58.491 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:58.479 7f0501528700 1 -- 172.21.2.102:0/1613298767 >> 172.21.2.102:0/1613298767 conn(0x7f04fc0ce3e0 msgr2=0x7f04fc0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:20:58.491 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:58.479 7f0501528700 1 -- 172.21.2.102:0/1613298767 shutdown_connections 2021-02-10T11:20:58.491 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:58.479 7f0501528700 1 -- 172.21.2.102:0/1613298767 wait complete. 2021-02-10T11:20:58.491 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:58.494 7f0501528700 1 Processor -- start 2021-02-10T11:20:58.491 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:58.494 7f0501528700 1 -- start start 2021-02-10T11:20:58.493 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:58.494 7f0501528700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f04fc0a0420 0x7f04fc0a0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:58.493 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:58.494 7f0501528700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f04fc189d00 0x7f04fc18a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:58.493 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:58.494 7f0501528700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f04fc18a720 0x7f04fc0b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:58.493 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:58.494 7f0501528700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04fc189780 con 0x7f04fc0a0420 2021-02-10T11:20:58.493 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:58.494 7f0501528700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04fc18ab20 con 0x7f04fc189d00 2021-02-10T11:20:58.493 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:58.494 7f0501528700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04fc0a1140 con 0x7f04fc18a720 2021-02-10T11:20:58.493 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:58.495 7f04fb2bf700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f04fc0a0420 0x7f04fc0a0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:58.494 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:58.495 7f04fb2bf700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f04fc0a0420 0x7f04fc0a0820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:54088/0 (socket says 172.21.2.102:54088) 2021-02-10T11:20:58.494 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:58.495 7f04fbac0700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f04fc18a720 0x7f04fc0b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:58.494 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:58.495 7f04fb2bf700 1 -- 172.21.2.102:0/2891903675 learned_addr learned my addr 172.21.2.102:0/2891903675 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:20:58.494 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:58.495 7f04faabe700 1 --2- 172.21.2.102:0/2891903675 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f04fc189d00 0x7f04fc18a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:58.494 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:58.495 7f04faabe700 1 -- 172.21.2.102:0/2891903675 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f04fc18a720 msgr2=0x7f04fc0b7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:58.494 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:58.495 7f04faabe700 1 --2- 172.21.2.102:0/2891903675 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f04fc18a720 0x7f04fc0b7be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:58.494 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:58.495 7f04faabe700 1 -- 172.21.2.102:0/2891903675 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f04fc0a0420 msgr2=0x7f04fc0a0820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:58.494 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:58.495 7f04faabe700 1 --2- 172.21.2.102:0/2891903675 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f04fc0a0420 0x7f04fc0a0820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:58.494 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:58.495 7f04faabe700 1 -- 172.21.2.102:0/2891903675 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04fc0b8200 con 0x7f04fc189d00 2021-02-10T11:20:58.494 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:58.495 7f04fb2bf700 1 --2- 172.21.2.102:0/2891903675 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f04fc0a0420 0x7f04fc0a0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:20:58.494 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:58.495 7f04faabe700 1 --2- 172.21.2.102:0/2891903675 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f04fc189d00 0x7f04fc18a100 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 rx=0x7f04e40096d0 tx=0x7f04e400b8c0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:20:58.495 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:58.495 7f04f3fff700 1 -- 172.21.2.102:0/2891903675 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f04e40009b0 con 0x7f04fc189d00 2021-02-10T11:20:58.495 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:58.495 7f04f3fff700 1 -- 172.21.2.102:0/2891903675 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04e400bcb0 con 0x7f04fc189d00 2021-02-10T11:20:58.495 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:58.495 7f04f3fff700 1 -- 172.21.2.102:0/2891903675 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f04e40022b0 con 0x7f04fc189d00 2021-02-10T11:20:58.495 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:58.495 7f0501528700 1 -- 172.21.2.102:0/2891903675 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04fc0b8200 con 0x7f04fc189d00 2021-02-10T11:20:58.495 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:58.495 7f0501528700 1 -- 172.21.2.102:0/2891903675 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04fc0b8200 con 0x7f04fc189d00 2021-02-10T11:20:58.495 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:58.496 7f04f3fff700 1 -- 172.21.2.102:0/2891903675 <== mon.1 v2:172.21.2.113:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f04e4000c90 con 0x7f04fc189d00 2021-02-10T11:20:58.495 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:58.496 7f04f3fff700 1 --2- 172.21.2.102:0/2891903675 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f04e0018730 0x7f04e001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:58.495 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:58.496 7f04f3fff700 1 -- 172.21.2.102:0/2891903675 <== mon.1 v2:172.21.2.113:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f04e40325c0 con 0x7f04fc189d00 2021-02-10T11:20:58.495 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:58.496 7f04fb2bf700 1 --2- 172.21.2.102:0/2891903675 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f04e0018730 0x7f04e001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:58.495 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:58.496 7f04fb2bf700 1 --2- 172.21.2.102:0/2891903675 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f04e0018730 0x7f04e001ac10 crc :-1 s=READY pgs=415 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:20:58.522 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:58.527 7f0501528700 1 -- 172.21.2.102:0/2891903675 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04fc105ae0 con 0x7f04fc189d00 2021-02-10T11:20:58.524 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:58.529 7f04f3fff700 1 -- 172.21.2.102:0/2891903675 <== mon.1 v2:172.21.2.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f04e40064f0 con 0x7f04fc189d00 2021-02-10T11:20:58.650 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:58.656 7f0501528700 1 -- 172.21.2.102:0/2891903675 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344392", "target": ["mgr", ""]}) v1 -- 0x7f04fc105ae0 con 0x7f04e0018730 2021-02-10T11:20:58.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:58.666 7f04f3fff700 1 -- 172.21.2.102:0/2891903675 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f04ec006400 con 0x7f04e0018730 2021-02-10T11:20:58.668 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:58.672 7f0501528700 1 -- 172.21.2.102:0/2891903675 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f04e0018730 msgr2=0x7f04e001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:58.668 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:58.672 7f0501528700 1 --2- 172.21.2.102:0/2891903675 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f04e0018730 0x7f04e001ac10 crc :-1 s=READY pgs=415 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:58.668 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:58.672 7f0501528700 1 -- 172.21.2.102:0/2891903675 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f04fc189d00 msgr2=0x7f04fc18a100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:58.668 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:58.672 7f0501528700 1 --2- 172.21.2.102:0/2891903675 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f04fc189d00 0x7f04fc18a100 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 rx=0x7f04e40096d0 tx=0x7f04e400b8c0).stop 2021-02-10T11:20:58.668 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:58.672 7f0501528700 1 -- 172.21.2.102:0/2891903675 shutdown_connections 2021-02-10T11:20:58.668 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:58.672 7f0501528700 1 --2- 172.21.2.102:0/2891903675 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f04fc18a720 0x7f04fc0b7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:58.668 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:58.672 7f0501528700 1 --2- 172.21.2.102:0/2891903675 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f04e0018730 0x7f04e001ac10 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:58.668 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:58.672 7f0501528700 1 --2- 172.21.2.102:0/2891903675 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f04fc189d00 0x7f04fc18a100 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:58.668 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:58.672 7f0501528700 1 --2- 172.21.2.102:0/2891903675 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f04fc0a0420 0x7f04fc0a0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:58.668 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:58.672 7f0501528700 1 -- 172.21.2.102:0/2891903675 >> 172.21.2.102:0/2891903675 conn(0x7f04fc094e10 msgr2=0x7f04fc0a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:20:58.668 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:58.673 7f0501528700 1 -- 172.21.2.102:0/2891903675 shutdown_connections 2021-02-10T11:20:58.669 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:58.673 7f0501528700 1 -- 172.21.2.102:0/2891903675 wait complete. 2021-02-10T11:20:58.677 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000344393 2021-02-10T11:20:59.108 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:59.097 7fe50aceb700 1 Processor -- start 2021-02-10T11:20:59.108 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:59.097 7fe50aceb700 1 -- start start 2021-02-10T11:20:59.108 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:59.097 7fe50aceb700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe5041b0470 0x7fe5041b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:59.108 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:59.097 7fe50aceb700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fe5040ef610 0x7fe5040efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:59.108 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:59.097 7fe50aceb700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5040f0b50 con 0x7fe5040f0030 2021-02-10T11:20:59.108 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:59.097 7fe50aceb700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5040e52e0 con 0x7fe5040ef610 2021-02-10T11:20:59.108 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:59.097 7fe50aceb700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5040e5490 con 0x7fe5041b0470 2021-02-10T11:20:59.108 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:59.097 7fe503fff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fe5040ef610 0x7fe5040efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:59.108 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:59.097 7fe509308700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fe5040f0030 0x7fe5040f10d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:46322/0 (socket says 172.21.2.102:46322) 2021-02-10T11:20:59.109 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:59.097 7fe509308700 1 -- 172.21.2.102:0/694311990 learned_addr learned my addr 172.21.2.102:0/694311990 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:20:59.109 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:59.097 7fe508b07700 1 --2- 172.21.2.102:0/694311990 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe5041b0470 0x7fe5041b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:59.109 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:59.097 7fe503fff700 1 -- 172.21.2.102:0/694311990 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe5041b0470 msgr2=0x7fe5041b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:20:59.109 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:59.097 7fe503fff700 1 --2- 172.21.2.102:0/694311990 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe5041b0470 0x7fe5041b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:59.109 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:59.097 7fe503fff700 1 -- 172.21.2.102:0/694311990 >> v1:172.21.2.102:6789/0 conn(0x7fe5040f0030 legacy=0x7fe5040f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:59.109 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:59.097 7fe5037fe700 1 -- 172.21.2.102:0/694311990 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2477582814 0 0) 0x7fe4fc001500 con 0x7fe5040f0030 2021-02-10T11:20:59.109 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:59.098 7fe503fff700 1 -- 172.21.2.102:0/694311990 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5040f0430 con 0x7fe5040ef610 2021-02-10T11:20:59.109 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:59.098 7fe508b07700 1 --2- 172.21.2.102:0/694311990 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe5041b0470 0x7fe5041b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:20:59.109 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:59.098 7fe503fff700 1 --2- 172.21.2.102:0/694311990 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe5040ef610 0x7fe5040efa10 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 rx=0x7fe4f4006650 tx=0x7fe4f4009a80).ready entity=mon.0 client_cookie=4fd463cec2791f52 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:20:59.109 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:59.098 7fe5037fe700 1 -- 172.21.2.102:0/694311990 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe4f400c0c0 con 0x7fe5040ef610 2021-02-10T11:20:59.109 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:59.098 7fe5037fe700 1 -- 172.21.2.102:0/694311990 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4f400c340 con 0x7fe5040ef610 2021-02-10T11:20:59.109 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:59.098 7fe5037fe700 1 -- 172.21.2.102:0/694311990 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe4f400c860 con 0x7fe5040ef610 2021-02-10T11:20:59.109 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:59.098 7fe50aceb700 1 -- 172.21.2.102:0/694311990 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe5040ef610 msgr2=0x7fe5040efa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:59.110 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:59.098 7fe50aceb700 1 --2- 172.21.2.102:0/694311990 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe5040ef610 0x7fe5040efa10 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 rx=0x7fe4f4006650 tx=0x7fe4f4009a80).stop 2021-02-10T11:20:59.110 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:59.100 7fe50aceb700 1 -- 172.21.2.102:0/694311990 shutdown_connections 2021-02-10T11:20:59.110 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:59.100 7fe50aceb700 1 --2- 172.21.2.102:0/694311990 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe5040ef610 0x7fe5040efa10 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:59.110 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:59.100 7fe50aceb700 1 --2- 172.21.2.102:0/694311990 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe5041b0470 0x7fe5041b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:59.110 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:59.100 7fe50aceb700 1 -- 172.21.2.102:0/694311990 >> 172.21.2.102:0/694311990 conn(0x7fe5040ce3e0 msgr2=0x7fe5040e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:20:59.110 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:59.100 7fe50aceb700 1 -- 172.21.2.102:0/694311990 shutdown_connections 2021-02-10T11:20:59.110 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:59.100 7fe50aceb700 1 -- 172.21.2.102:0/694311990 wait complete. 2021-02-10T11:20:59.112 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:59.113 7fe50aceb700 1 Processor -- start 2021-02-10T11:20:59.112 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:59.113 7fe50aceb700 1 -- start start 2021-02-10T11:20:59.112 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:59.114 7fe50aceb700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe5040a0420 0x7fe5040a0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:59.112 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:59.114 7fe50aceb700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe504189d00 0x7fe50418a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:59.112 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:59.114 7fe50aceb700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe50418a720 0x7fe5040b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:59.112 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:59.114 7fe50aceb700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe504189800 con 0x7fe50418a720 2021-02-10T11:20:59.112 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:59.114 7fe50aceb700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe50418ab20 con 0x7fe5040a0420 2021-02-10T11:20:59.112 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:59.114 7fe50aceb700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5040a1140 con 0x7fe504189d00 2021-02-10T11:20:59.112 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:59.114 7fe509308700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe50418a720 0x7fe5040b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:59.112 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:59.114 7fe508b07700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe5040a0420 0x7fe5040a0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:59.112 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:59.114 7fe509308700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe50418a720 0x7fe5040b7be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:54104/0 (socket says 172.21.2.102:54104) 2021-02-10T11:20:59.112 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:59.114 7fe509308700 1 -- 172.21.2.102:0/3379591349 learned_addr learned my addr 172.21.2.102:0/3379591349 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:20:59.113 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:59.114 7fe503fff700 1 --2- 172.21.2.102:0/3379591349 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe504189d00 0x7fe50418a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:59.113 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:59.116 7fe509308700 1 -- 172.21.2.102:0/3379591349 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe504189d00 msgr2=0x7fe50418a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:59.113 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:59.116 7fe509308700 1 --2- 172.21.2.102:0/3379591349 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe504189d00 0x7fe50418a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:59.113 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:59.116 7fe509308700 1 -- 172.21.2.102:0/3379591349 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe5040a0420 msgr2=0x7fe5040a0820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:59.113 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:59.116 7fe509308700 1 --2- 172.21.2.102:0/3379591349 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe5040a0420 0x7fe5040a0820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:59.113 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:59.116 7fe509308700 1 -- 172.21.2.102:0/3379591349 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5040b8200 con 0x7fe50418a720 2021-02-10T11:20:59.113 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:59.116 7fe508b07700 1 --2- 172.21.2.102:0/3379591349 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe5040a0420 0x7fe5040a0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:20:59.113 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:59.116 7fe509308700 1 --2- 172.21.2.102:0/3379591349 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe50418a720 0x7fe5040b7be0 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 rx=0x7fe4fc001c10 tx=0x7fe4fc00ac60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:20:59.113 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:59.116 7fe5017fa700 1 -- 172.21.2.102:0/3379591349 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe4fc000f90 con 0x7fe50418a720 2021-02-10T11:20:59.113 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:59.116 7fe5017fa700 1 -- 172.21.2.102:0/3379591349 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4fc0070d0 con 0x7fe50418a720 2021-02-10T11:20:59.113 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:59.116 7fe50aceb700 1 -- 172.21.2.102:0/3379591349 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5040b8200 con 0x7fe50418a720 2021-02-10T11:20:59.113 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:59.116 7fe5017fa700 1 -- 172.21.2.102:0/3379591349 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe4fc0074a0 con 0x7fe50418a720 2021-02-10T11:20:59.113 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:59.116 7fe50aceb700 1 -- 172.21.2.102:0/3379591349 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5040b8200 con 0x7fe50418a720 2021-02-10T11:20:59.114 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:59.116 7fe5017fa700 1 -- 172.21.2.102:0/3379591349 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fe4fc0072b0 con 0x7fe50418a720 2021-02-10T11:20:59.114 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:59.116 7fe5017fa700 1 --2- 172.21.2.102:0/3379591349 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe4e8018730 0x7fe4e801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:59.114 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:59.116 7fe5017fa700 1 -- 172.21.2.102:0/3379591349 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fe4fc032b50 con 0x7fe50418a720 2021-02-10T11:20:59.114 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:59.116 7fe508b07700 1 --2- 172.21.2.102:0/3379591349 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe4e8018730 0x7fe4e801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:59.114 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:59.116 7fe508b07700 1 --2- 172.21.2.102:0/3379591349 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe4e8018730 0x7fe4e801ac10 crc :-1 s=READY pgs=416 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:20:59.140 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:59.146 7fe50aceb700 1 -- 172.21.2.102:0/3379591349 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe50418ab20 con 0x7fe50418a720 2021-02-10T11:20:59.143 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:59.149 7fe5017fa700 1 -- 172.21.2.102:0/3379591349 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fe4fc000cc0 con 0x7fe50418a720 2021-02-10T11:20:59.272 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:59.277 7fe50aceb700 1 -- 172.21.2.102:0/3379591349 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344393", "target": ["mgr", ""]}) v1 -- 0x7fe50418ab20 con 0x7fe4e8018730 2021-02-10T11:20:59.284 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:59.290 7fe5017fa700 1 -- 172.21.2.102:0/3379591349 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fe4ec0009b0 con 0x7fe4e8018730 2021-02-10T11:20:59.289 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:59.296 7fe50aceb700 1 -- 172.21.2.102:0/3379591349 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe4e8018730 msgr2=0x7fe4e801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:59.290 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:59.296 7fe50aceb700 1 --2- 172.21.2.102:0/3379591349 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe4e8018730 0x7fe4e801ac10 crc :-1 s=READY pgs=416 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:59.290 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:59.296 7fe50aceb700 1 -- 172.21.2.102:0/3379591349 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe50418a720 msgr2=0x7fe5040b7be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:59.290 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:59.296 7fe50aceb700 1 --2- 172.21.2.102:0/3379591349 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe50418a720 0x7fe5040b7be0 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 rx=0x7fe4fc001c10 tx=0x7fe4fc00ac60).stop 2021-02-10T11:20:59.290 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:59.296 7fe50aceb700 1 -- 172.21.2.102:0/3379591349 shutdown_connections 2021-02-10T11:20:59.290 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:59.296 7fe50aceb700 1 --2- 172.21.2.102:0/3379591349 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe50418a720 0x7fe5040b7be0 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:59.290 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:59.296 7fe50aceb700 1 --2- 172.21.2.102:0/3379591349 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe504189d00 0x7fe50418a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:59.291 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:59.296 7fe50aceb700 1 --2- 172.21.2.102:0/3379591349 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe4e8018730 0x7fe4e801ac10 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:59.291 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:59.296 7fe50aceb700 1 --2- 172.21.2.102:0/3379591349 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe5040a0420 0x7fe5040a0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:59.291 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:59.296 7fe50aceb700 1 -- 172.21.2.102:0/3379591349 >> 172.21.2.102:0/3379591349 conn(0x7fe504094e10 msgr2=0x7fe5040a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:20:59.291 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:59.296 7fe50aceb700 1 -- 172.21.2.102:0/3379591349 shutdown_connections 2021-02-10T11:20:59.291 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:59.296 7fe50aceb700 1 -- 172.21.2.102:0/3379591349 wait complete. 2021-02-10T11:20:59.300 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000344394 2021-02-10T11:20:59.733 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:59.722 7f156067d700 1 Processor -- start 2021-02-10T11:20:59.733 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:59.723 7f156067d700 1 -- start start 2021-02-10T11:20:59.733 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:59.723 7f156067d700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f15581b0870 0x7f15580f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:59.733 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:59.723 7f156067d700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f15580efe00 0x7f15580f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:59.733 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:59.723 7f156067d700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15581b0c70 con 0x7f15581b0470 2021-02-10T11:20:59.733 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:59.723 7f156067d700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15580e5ea0 con 0x7f15580efe00 2021-02-10T11:20:59.733 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:59.723 7f156067d700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15580f0b50 con 0x7f15581b0870 2021-02-10T11:20:59.733 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:59.723 7f155ec9a700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f15580efe00 0x7f15580f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:59.733 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:59.723 7f155ec9a700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f15580efe00 0x7f15580f0200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:54110/0 (socket says 172.21.2.102:54110) 2021-02-10T11:20:59.734 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:59.723 7f155ec9a700 1 -- 172.21.2.102:0/2993792289 learned_addr learned my addr 172.21.2.102:0/2993792289 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:20:59.734 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:59.723 7f155dc98700 1 --2- 172.21.2.102:0/2993792289 >> v2:172.21.2.113:3300/0 conn(0x7f15581b0870 0x7f15580f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:59.734 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:59.724 7f155ec9a700 1 -- 172.21.2.102:0/2993792289 >> v2:172.21.2.113:3300/0 conn(0x7f15581b0870 msgr2=0x7f15580f20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:20:59.734 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:59.724 7f155ec9a700 1 --2- 172.21.2.102:0/2993792289 >> v2:172.21.2.113:3300/0 conn(0x7f15581b0870 0x7f15580f20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:59.734 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:59.724 7f155ec9a700 1 -- 172.21.2.102:0/2993792289 >> v1:172.21.2.102:6789/0 conn(0x7f15581b0470 legacy=0x7f15580ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:59.734 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:59.724 7f155ec9a700 1 -- 172.21.2.102:0/2993792289 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15580f0820 con 0x7f15580efe00 2021-02-10T11:20:59.734 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:59.724 7f155ec9a700 1 --2- 172.21.2.102:0/2993792289 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f15580efe00 0x7f15580f0200 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 rx=0x7f1550006b80 tx=0x7f1550009b90).ready entity=mon.0 client_cookie=813d812ed4e4730d server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:20:59.734 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:59.724 7f155d497700 1 -- 172.21.2.102:0/2993792289 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f15500022e0 con 0x7f15580efe00 2021-02-10T11:20:59.734 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:59.724 7f155d497700 1 -- 172.21.2.102:0/2993792289 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1550002620 con 0x7f15580efe00 2021-02-10T11:20:59.734 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:59.724 7f155d497700 1 -- 172.21.2.102:0/2993792289 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1550002b00 con 0x7f15580efe00 2021-02-10T11:20:59.734 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:59.724 7f156067d700 1 -- 172.21.2.102:0/2993792289 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f15580efe00 msgr2=0x7f15580f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:59.734 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:59.724 7f156067d700 1 --2- 172.21.2.102:0/2993792289 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f15580efe00 0x7f15580f0200 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 rx=0x7f1550006b80 tx=0x7f1550009b90).stop 2021-02-10T11:20:59.735 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:59.725 7f156067d700 1 -- 172.21.2.102:0/2993792289 shutdown_connections 2021-02-10T11:20:59.735 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:59.725 7f156067d700 1 --2- 172.21.2.102:0/2993792289 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f15580efe00 0x7f15580f0200 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:59.735 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:59.725 7f156067d700 1 --2- 172.21.2.102:0/2993792289 >> v2:172.21.2.113:3300/0 conn(0x7f15581b0870 0x7f15580f20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:59.735 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:59.725 7f156067d700 1 -- 172.21.2.102:0/2993792289 >> 172.21.2.102:0/2993792289 conn(0x7f15580ce3e0 msgr2=0x7f15580e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:20:59.735 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:59.725 7f156067d700 1 -- 172.21.2.102:0/2993792289 shutdown_connections 2021-02-10T11:20:59.735 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:59.725 7f156067d700 1 -- 172.21.2.102:0/2993792289 wait complete. 2021-02-10T11:20:59.736 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:59.739 7f156067d700 1 Processor -- start 2021-02-10T11:20:59.736 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:59.739 7f156067d700 1 -- start start 2021-02-10T11:20:59.736 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:59.740 7f156067d700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f15580a0420 0x7f15580a0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:59.737 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:59.740 7f156067d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1558189d00 0x7f155818a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:59.737 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:59.740 7f156067d700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f155818a720 0x7f15580b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:59.737 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:59.740 7f156067d700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1558189780 con 0x7f1558189d00 2021-02-10T11:20:59.737 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:59.740 7f156067d700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f155818ab20 con 0x7f155818a720 2021-02-10T11:20:59.737 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:59.740 7f156067d700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15580a1140 con 0x7f15580a0420 2021-02-10T11:20:59.737 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:59.740 7f155dc98700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1558189d00 0x7f155818a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:59.737 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:59.740 7f155ec9a700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f155818a720 0x7f15580b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:59.737 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:59.740 7f155dc98700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1558189d00 0x7f155818a100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:54116/0 (socket says 172.21.2.102:54116) 2021-02-10T11:20:59.737 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:59.740 7f155dc98700 1 -- 172.21.2.102:0/3341964938 learned_addr learned my addr 172.21.2.102:0/3341964938 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:20:59.737 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:59.740 7f155e499700 1 --2- 172.21.2.102:0/3341964938 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f15580a0420 0x7f15580a0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:59.737 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:59.740 7f155ec9a700 1 -- 172.21.2.102:0/3341964938 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f15580a0420 msgr2=0x7f15580a0820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:59.737 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:59.740 7f155ec9a700 1 --2- 172.21.2.102:0/3341964938 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f15580a0420 0x7f15580a0820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:59.738 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:59.740 7f155ec9a700 1 -- 172.21.2.102:0/3341964938 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1558189d00 msgr2=0x7f155818a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:59.738 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:59.740 7f155ec9a700 1 --2- 172.21.2.102:0/3341964938 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1558189d00 0x7f155818a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:59.738 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:59.740 7f155ec9a700 1 -- 172.21.2.102:0/3341964938 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15580b8200 con 0x7f155818a720 2021-02-10T11:20:59.738 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:59.741 7f155ec9a700 1 --2- 172.21.2.102:0/3341964938 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f155818a720 0x7f15580b7be0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 rx=0x7f15500095c0 tx=0x7f155000b8c0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:20:59.738 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:59.741 7f154effd700 1 -- 172.21.2.102:0/3341964938 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f15500090b0 con 0x7f155818a720 2021-02-10T11:20:59.738 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:59.741 7f154effd700 1 -- 172.21.2.102:0/3341964938 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f155000bdc0 con 0x7f155818a720 2021-02-10T11:20:59.738 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:59.741 7f154effd700 1 -- 172.21.2.102:0/3341964938 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f15500064b0 con 0x7f155818a720 2021-02-10T11:20:59.738 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:59.741 7f156067d700 1 -- 172.21.2.102:0/3341964938 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15580b8200 con 0x7f155818a720 2021-02-10T11:20:59.738 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:59.741 7f156067d700 1 -- 172.21.2.102:0/3341964938 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15580b8200 con 0x7f155818a720 2021-02-10T11:20:59.738 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:59.741 7f154effd700 1 -- 172.21.2.102:0/3341964938 <== mon.1 v2:172.21.2.113:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1550009fd0 con 0x7f155818a720 2021-02-10T11:20:59.738 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:59.742 7f154effd700 1 --2- 172.21.2.102:0/3341964938 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f15400186c0 0x7f154001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:20:59.738 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:59.742 7f154effd700 1 -- 172.21.2.102:0/3341964938 <== mon.1 v2:172.21.2.113:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f15500324a0 con 0x7f155818a720 2021-02-10T11:20:59.739 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:59.742 7f155e499700 1 --2- 172.21.2.102:0/3341964938 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f15400186c0 0x7f154001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:20:59.739 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:59.742 7f155e499700 1 --2- 172.21.2.102:0/3341964938 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f15400186c0 0x7f154001abc0 crc :-1 s=READY pgs=417 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:20:59.766 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:59.772 7f156067d700 1 -- 172.21.2.102:0/3341964938 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1558105ae0 con 0x7f155818a720 2021-02-10T11:20:59.769 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:59.775 7f154effd700 1 -- 172.21.2.102:0/3341964938 <== mon.1 v2:172.21.2.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f1550006a00 con 0x7f155818a720 2021-02-10T11:20:59.898 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:59.904 7f156067d700 1 -- 172.21.2.102:0/3341964938 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344394", "target": ["mgr", ""]}) v1 -- 0x7f1558189780 con 0x7f15400186c0 2021-02-10T11:20:59.909 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:59.915 7f154effd700 1 -- 172.21.2.102:0/3341964938 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f1554005830 con 0x7f15400186c0 2021-02-10T11:20:59.916 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:59.921 7f156067d700 1 -- 172.21.2.102:0/3341964938 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f15400186c0 msgr2=0x7f154001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:59.916 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:59.921 7f156067d700 1 --2- 172.21.2.102:0/3341964938 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f15400186c0 0x7f154001abc0 crc :-1 s=READY pgs=417 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:59.916 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:59.921 7f156067d700 1 -- 172.21.2.102:0/3341964938 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f155818a720 msgr2=0x7f15580b7be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:20:59.916 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:59.921 7f156067d700 1 --2- 172.21.2.102:0/3341964938 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f155818a720 0x7f15580b7be0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 rx=0x7f15500095c0 tx=0x7f155000b8c0).stop 2021-02-10T11:20:59.916 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:59.921 7f156067d700 1 -- 172.21.2.102:0/3341964938 shutdown_connections 2021-02-10T11:20:59.916 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:59.921 7f156067d700 1 --2- 172.21.2.102:0/3341964938 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f15400186c0 0x7f154001abc0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:59.916 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:59.921 7f156067d700 1 --2- 172.21.2.102:0/3341964938 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f155818a720 0x7f15580b7be0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:59.916 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:59.921 7f156067d700 1 --2- 172.21.2.102:0/3341964938 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1558189d00 0x7f155818a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:59.916 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:59.921 7f156067d700 1 --2- 172.21.2.102:0/3341964938 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f15580a0420 0x7f15580a0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:20:59.916 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:59.921 7f156067d700 1 -- 172.21.2.102:0/3341964938 >> 172.21.2.102:0/3341964938 conn(0x7f1558094e10 msgr2=0x7f15580a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:20:59.916 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:59.922 7f156067d700 1 -- 172.21.2.102:0/3341964938 shutdown_connections 2021-02-10T11:20:59.917 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:20:59.922 7f156067d700 1 -- 172.21.2.102:0/3341964938 wait complete. 2021-02-10T11:20:59.924 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000344395 2021-02-10T11:21:00.356 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:00.345 7faa4bb07700 1 Processor -- start 2021-02-10T11:21:00.356 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:00.345 7faa4bb07700 1 -- start start 2021-02-10T11:21:00.356 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:00.346 7faa4bb07700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7faa441b0470 0x7faa441b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:00.357 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:00.346 7faa4bb07700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7faa440ef610 0x7faa440efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:00.357 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:00.346 7faa4bb07700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7faa440f0030 0x7faa440f0430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:00.357 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:00.346 7faa4bb07700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa440e5ea0 con 0x7faa440ef610 2021-02-10T11:21:00.357 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:00.346 7faa4bb07700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa440f0b50 con 0x7faa440f0030 2021-02-10T11:21:00.357 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:00.346 7faa4bb07700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa440e52e0 con 0x7faa441b0470 2021-02-10T11:21:00.357 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:00.346 7faa49122700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7faa440ef610 0x7faa440efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:00.357 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:00.346 7faa49122700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7faa440ef610 0x7faa440efa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:54128/0 (socket says 172.21.2.102:54128) 2021-02-10T11:21:00.357 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:00.346 7faa49122700 1 -- 172.21.2.102:0/2443383506 learned_addr learned my addr 172.21.2.102:0/2443383506 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:21:00.357 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:00.346 7faa4a124700 1 --2- 172.21.2.102:0/2443383506 >> v2:172.21.2.113:3300/0 conn(0x7faa440f0030 0x7faa440f0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:00.357 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:00.346 7faa49923700 1 --2- 172.21.2.102:0/2443383506 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7faa441b0470 0x7faa441b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:00.357 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:00.347 7faa49122700 1 -- 172.21.2.102:0/2443383506 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7faa441b0470 msgr2=0x7faa441b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:21:00.357 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:00.347 7faa49122700 1 --2- 172.21.2.102:0/2443383506 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7faa441b0470 0x7faa441b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:00.357 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:00.347 7faa49122700 1 -- 172.21.2.102:0/2443383506 >> v2:172.21.2.113:3300/0 conn(0x7faa440f0030 msgr2=0x7faa440f0430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:21:00.358 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:00.347 7faa49122700 1 --2- 172.21.2.102:0/2443383506 >> v2:172.21.2.113:3300/0 conn(0x7faa440f0030 0x7faa440f0430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:00.358 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:00.347 7faa49122700 1 -- 172.21.2.102:0/2443383506 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa440f10d0 con 0x7faa440ef610 2021-02-10T11:21:00.358 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:00.347 7faa4a124700 1 --2- 172.21.2.102:0/2443383506 >> v2:172.21.2.113:3300/0 conn(0x7faa440f0030 0x7faa440f0430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:21:00.358 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:00.347 7faa49122700 1 --2- 172.21.2.102:0/2443383506 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7faa440ef610 0x7faa440efa10 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 rx=0x7faa34006700 tx=0x7faa34009b80).ready entity=mon.0 client_cookie=a390d257e38248c2 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:00.358 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:00.347 7faa48921700 1 -- 172.21.2.102:0/2443383506 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faa3400c0c0 con 0x7faa440ef610 2021-02-10T11:21:00.358 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:00.347 7faa48921700 1 -- 172.21.2.102:0/2443383506 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa3400c440 con 0x7faa440ef610 2021-02-10T11:21:00.358 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:00.347 7faa48921700 1 -- 172.21.2.102:0/2443383506 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faa3400c960 con 0x7faa440ef610 2021-02-10T11:21:00.358 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:00.347 7faa4bb07700 1 -- 172.21.2.102:0/2443383506 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7faa440ef610 msgr2=0x7faa440efa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:00.358 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:00.347 7faa4bb07700 1 --2- 172.21.2.102:0/2443383506 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7faa440ef610 0x7faa440efa10 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 rx=0x7faa34006700 tx=0x7faa34009b80).stop 2021-02-10T11:21:00.358 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:00.348 7faa4bb07700 1 -- 172.21.2.102:0/2443383506 shutdown_connections 2021-02-10T11:21:00.358 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:00.348 7faa4bb07700 1 --2- 172.21.2.102:0/2443383506 >> v2:172.21.2.113:3300/0 conn(0x7faa440f0030 0x7faa440f0430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:00.358 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:00.348 7faa4bb07700 1 --2- 172.21.2.102:0/2443383506 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7faa440ef610 0x7faa440efa10 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:00.359 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:00.348 7faa4bb07700 1 --2- 172.21.2.102:0/2443383506 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7faa441b0470 0x7faa441b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:00.359 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:00.348 7faa4bb07700 1 -- 172.21.2.102:0/2443383506 >> 172.21.2.102:0/2443383506 conn(0x7faa440ce3e0 msgr2=0x7faa440e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:00.359 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:00.348 7faa4bb07700 1 -- 172.21.2.102:0/2443383506 shutdown_connections 2021-02-10T11:21:00.359 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:00.348 7faa4bb07700 1 -- 172.21.2.102:0/2443383506 wait complete. 2021-02-10T11:21:00.360 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:00.363 7faa4bb07700 1 Processor -- start 2021-02-10T11:21:00.360 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:00.363 7faa4bb07700 1 -- start start 2021-02-10T11:21:00.360 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:00.363 7faa4bb07700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7faa440b7de0 0x7faa44190920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:00.361 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:00.363 7faa4bb07700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7faa440b82f0 0x7faa440b86f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:00.361 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:00.363 7faa4bb07700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7faa440b8d10 0x7faa440bf9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:00.361 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:00.363 7faa4bb07700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa440b7930 con 0x7faa440b82f0 2021-02-10T11:21:00.361 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:00.363 7faa4bb07700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa440b7b20 con 0x7faa440b8d10 2021-02-10T11:21:00.361 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:00.363 7faa4bb07700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa440bffd0 con 0x7faa440b7de0 2021-02-10T11:21:00.361 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:00.363 7faa49122700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7faa440b82f0 0x7faa440b86f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:00.361 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:00.363 7faa49923700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7faa440b7de0 0x7faa44190920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:00.361 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:00.363 7faa49122700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7faa440b82f0 0x7faa440b86f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:54134/0 (socket says 172.21.2.102:54134) 2021-02-10T11:21:00.361 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:00.363 7faa49122700 1 -- 172.21.2.102:0/3690848781 learned_addr learned my addr 172.21.2.102:0/3690848781 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:21:00.361 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:00.363 7faa4a124700 1 --2- 172.21.2.102:0/3690848781 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7faa440b8d10 0x7faa440bf9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:00.361 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:00.364 7faa49923700 1 -- 172.21.2.102:0/3690848781 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7faa440b8d10 msgr2=0x7faa440bf9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:00.361 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:00.364 7faa49923700 1 --2- 172.21.2.102:0/3690848781 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7faa440b8d10 0x7faa440bf9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:00.361 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:00.364 7faa49923700 1 -- 172.21.2.102:0/3690848781 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7faa440b82f0 msgr2=0x7faa440b86f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:00.362 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:00.364 7faa49923700 1 --2- 172.21.2.102:0/3690848781 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7faa440b82f0 0x7faa440b86f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:00.362 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:00.364 7faa49923700 1 -- 172.21.2.102:0/3690848781 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa440c01c0 con 0x7faa440b7de0 2021-02-10T11:21:00.362 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:00.364 7faa49122700 1 --2- 172.21.2.102:0/3690848781 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7faa440b82f0 0x7faa440b86f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:21:00.362 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:00.364 7faa49923700 1 --2- 172.21.2.102:0/3690848781 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7faa440b7de0 0x7faa44190920 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7faa40009540 tx=0x7faa4000b8e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:00.362 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:00.364 7faa3a7fc700 1 -- 172.21.2.102:0/3690848781 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faa400009b0 con 0x7faa440b7de0 2021-02-10T11:21:00.362 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:00.364 7faa3a7fc700 1 -- 172.21.2.102:0/3690848781 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa40006340 con 0x7faa440b7de0 2021-02-10T11:21:00.362 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:00.364 7faa3a7fc700 1 -- 172.21.2.102:0/3690848781 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faa40002030 con 0x7faa440b7de0 2021-02-10T11:21:00.362 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:00.364 7faa4bb07700 1 -- 172.21.2.102:0/3690848781 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa440c01c0 con 0x7faa440b7de0 2021-02-10T11:21:00.362 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:00.364 7faa4bb07700 1 -- 172.21.2.102:0/3690848781 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa440c01c0 con 0x7faa440b7de0 2021-02-10T11:21:00.362 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:00.365 7faa3a7fc700 1 -- 172.21.2.102:0/3690848781 <== mon.2 v2:172.21.2.113:3301/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7faa40000c90 con 0x7faa440b7de0 2021-02-10T11:21:00.362 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:00.365 7faa3a7fc700 1 --2- 172.21.2.102:0/3690848781 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7faa2c018730 0x7faa2c01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:00.362 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:00.365 7faa3a7fc700 1 -- 172.21.2.102:0/3690848781 <== mon.2 v2:172.21.2.113:3301/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7faa40032740 con 0x7faa440b7de0 2021-02-10T11:21:00.363 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:00.365 7faa49122700 1 --2- 172.21.2.102:0/3690848781 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7faa2c018730 0x7faa2c01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:00.363 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:00.365 7faa49122700 1 --2- 172.21.2.102:0/3690848781 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7faa2c018730 0x7faa2c01ac10 crc :-1 s=READY pgs=418 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:00.389 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:00.396 7faa4bb07700 1 -- 172.21.2.102:0/3690848781 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa44105b30 con 0x7faa440b7de0 2021-02-10T11:21:00.393 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:00.399 7faa3a7fc700 1 -- 172.21.2.102:0/3690848781 <== mon.2 v2:172.21.2.113:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7faa40006ae0 con 0x7faa440b7de0 2021-02-10T11:21:00.522 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:00.529 7faa4bb07700 1 -- 172.21.2.102:0/3690848781 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344395", "target": ["mgr", ""]}) v1 -- 0x7faa44105b30 con 0x7faa2c018730 2021-02-10T11:21:00.534 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:00.540 7faa3a7fc700 1 -- 172.21.2.102:0/3690848781 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7faa34006c70 con 0x7faa2c018730 2021-02-10T11:21:00.540 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:00.546 7faa4bb07700 1 -- 172.21.2.102:0/3690848781 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7faa2c018730 msgr2=0x7faa2c01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:00.540 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:00.546 7faa4bb07700 1 --2- 172.21.2.102:0/3690848781 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7faa2c018730 0x7faa2c01ac10 crc :-1 s=READY pgs=418 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:00.540 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:00.546 7faa4bb07700 1 -- 172.21.2.102:0/3690848781 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7faa440b7de0 msgr2=0x7faa44190920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:00.540 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:00.546 7faa4bb07700 1 --2- 172.21.2.102:0/3690848781 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7faa440b7de0 0x7faa44190920 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7faa40009540 tx=0x7faa4000b8e0).stop 2021-02-10T11:21:00.540 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:00.546 7faa4bb07700 1 -- 172.21.2.102:0/3690848781 shutdown_connections 2021-02-10T11:21:00.541 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:00.546 7faa4bb07700 1 --2- 172.21.2.102:0/3690848781 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7faa2c018730 0x7faa2c01ac10 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:00.541 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:00.546 7faa4bb07700 1 --2- 172.21.2.102:0/3690848781 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7faa440b8d10 0x7faa440bf9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:00.541 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:00.546 7faa4bb07700 1 --2- 172.21.2.102:0/3690848781 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7faa440b82f0 0x7faa440b86f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:00.541 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:00.546 7faa4bb07700 1 --2- 172.21.2.102:0/3690848781 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7faa440b7de0 0x7faa44190920 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:00.541 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:00.546 7faa4bb07700 1 -- 172.21.2.102:0/3690848781 >> 172.21.2.102:0/3690848781 conn(0x7faa44094e10 msgr2=0x7faa440a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:00.541 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:00.546 7faa4bb07700 1 -- 172.21.2.102:0/3690848781 shutdown_connections 2021-02-10T11:21:00.541 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:00.546 7faa4bb07700 1 -- 172.21.2.102:0/3690848781 wait complete. 2021-02-10T11:21:00.550 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000344396 2021-02-10T11:21:00.982 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:00.971 7fc229d8f700 1 Processor -- start 2021-02-10T11:21:00.983 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:00.971 7fc229d8f700 1 -- start start 2021-02-10T11:21:00.983 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:00.972 7fc229d8f700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fc2241b0870 0x7fc2240f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:00.983 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:00.972 7fc229d8f700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fc2240efe00 0x7fc2240f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:00.983 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:00.972 7fc229d8f700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2241b0c70 con 0x7fc2241b0470 2021-02-10T11:21:00.983 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:00.972 7fc229d8f700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2240e5ea0 con 0x7fc2241b0870 2021-02-10T11:21:00.983 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:00.972 7fc229d8f700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2240f0b50 con 0x7fc2240efe00 2021-02-10T11:21:00.983 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:00.972 7fc222ffd700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fc2241b0870 0x7fc2240f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:00.983 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:00.972 7fc2237fe700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fc2241b0470 0x7fc2240ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:46362/0 (socket says 172.21.2.102:46362) 2021-02-10T11:21:00.983 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:00.972 7fc2237fe700 1 -- 172.21.2.102:0/3233618638 learned_addr learned my addr 172.21.2.102:0/3233618638 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:21:00.984 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:00.972 7fc223fff700 1 --2- 172.21.2.102:0/3233618638 >> v2:172.21.2.113:3300/0 conn(0x7fc2240efe00 0x7fc2240f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:00.984 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:00.972 7fc222ffd700 1 -- 172.21.2.102:0/3233618638 >> v2:172.21.2.113:3300/0 conn(0x7fc2240efe00 msgr2=0x7fc2240f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:21:00.984 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:00.973 7fc222ffd700 1 --2- 172.21.2.102:0/3233618638 >> v2:172.21.2.113:3300/0 conn(0x7fc2240efe00 0x7fc2240f0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:00.984 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:00.973 7fc222ffd700 1 -- 172.21.2.102:0/3233618638 >> v1:172.21.2.102:6789/0 conn(0x7fc2241b0470 legacy=0x7fc2240ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:00.984 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:00.973 7fc2227fc700 1 -- 172.21.2.102:0/3233618638 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3948678550 0 0) 0x7fc214001180 con 0x7fc2241b0470 2021-02-10T11:21:00.984 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:00.973 7fc222ffd700 1 -- 172.21.2.102:0/3233618638 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2240f0820 con 0x7fc2241b0870 2021-02-10T11:21:00.984 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:00.973 7fc223fff700 1 --2- 172.21.2.102:0/3233618638 >> v2:172.21.2.113:3300/0 conn(0x7fc2240efe00 0x7fc2240f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:21:00.985 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:00.973 7fc222ffd700 1 --2- 172.21.2.102:0/3233618638 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc2241b0870 0x7fc2240f20e0 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 rx=0x7fc20c006650 tx=0x7fc20c009a80).ready entity=mon.0 client_cookie=ca2ee63e7163e972 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:00.985 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:00.973 7fc2227fc700 1 -- 172.21.2.102:0/3233618638 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc20c0009b0 con 0x7fc2241b0870 2021-02-10T11:21:00.985 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:00.973 7fc2227fc700 1 -- 172.21.2.102:0/3233618638 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc20c00c0c0 con 0x7fc2241b0870 2021-02-10T11:21:00.985 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:00.973 7fc2227fc700 1 -- 172.21.2.102:0/3233618638 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc20c00c5e0 con 0x7fc2241b0870 2021-02-10T11:21:00.985 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:00.973 7fc229d8f700 1 -- 172.21.2.102:0/3233618638 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc2241b0870 msgr2=0x7fc2240f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:00.985 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:00.973 7fc229d8f700 1 --2- 172.21.2.102:0/3233618638 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc2241b0870 0x7fc2240f20e0 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 rx=0x7fc20c006650 tx=0x7fc20c009a80).stop 2021-02-10T11:21:00.985 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:00.974 7fc229d8f700 1 -- 172.21.2.102:0/3233618638 shutdown_connections 2021-02-10T11:21:00.985 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:00.974 7fc229d8f700 1 --2- 172.21.2.102:0/3233618638 >> v2:172.21.2.113:3300/0 conn(0x7fc2240efe00 0x7fc2240f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:00.985 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:00.974 7fc229d8f700 1 --2- 172.21.2.102:0/3233618638 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc2241b0870 0x7fc2240f20e0 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:00.985 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:00.974 7fc229d8f700 1 -- 172.21.2.102:0/3233618638 >> 172.21.2.102:0/3233618638 conn(0x7fc2240ce3e0 msgr2=0x7fc2240e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:00.985 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:00.974 7fc229d8f700 1 -- 172.21.2.102:0/3233618638 shutdown_connections 2021-02-10T11:21:00.985 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:00.974 7fc229d8f700 1 -- 172.21.2.102:0/3233618638 wait complete. 2021-02-10T11:21:00.985 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:00.989 7fc229d8f700 1 Processor -- start 2021-02-10T11:21:00.986 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:00.989 7fc229d8f700 1 -- start start 2021-02-10T11:21:00.988 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:00.989 7fc229d8f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc2240a0420 0x7fc2240a0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:00.988 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:00.989 7fc229d8f700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc224189d00 0x7fc22418a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:00.988 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:00.989 7fc229d8f700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc22418a720 0x7fc2240b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:00.988 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:00.989 7fc229d8f700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc224189800 con 0x7fc2240a0420 2021-02-10T11:21:00.988 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:00.989 7fc229d8f700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc22418ab20 con 0x7fc22418a720 2021-02-10T11:21:00.988 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:00.989 7fc229d8f700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2240a1140 con 0x7fc224189d00 2021-02-10T11:21:00.988 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:00.989 7fc2237fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc2240a0420 0x7fc2240a0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:00.988 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:00.989 7fc2237fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc2240a0420 0x7fc2240a0820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:54144/0 (socket says 172.21.2.102:54144) 2021-02-10T11:21:00.989 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:00.989 7fc222ffd700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc224189d00 0x7fc22418a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:00.989 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:00.989 7fc2237fe700 1 -- 172.21.2.102:0/3719805389 learned_addr learned my addr 172.21.2.102:0/3719805389 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:21:00.989 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:00.989 7fc223fff700 1 --2- 172.21.2.102:0/3719805389 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc22418a720 0x7fc2240b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:00.989 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:00.990 7fc2237fe700 1 -- 172.21.2.102:0/3719805389 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc224189d00 msgr2=0x7fc22418a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:00.989 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:00.990 7fc2237fe700 1 --2- 172.21.2.102:0/3719805389 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc224189d00 0x7fc22418a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:00.989 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:00.990 7fc2237fe700 1 -- 172.21.2.102:0/3719805389 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc22418a720 msgr2=0x7fc2240b7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:00.989 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:00.990 7fc2237fe700 1 --2- 172.21.2.102:0/3719805389 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc22418a720 0x7fc2240b7be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:00.989 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:00.990 7fc2237fe700 1 -- 172.21.2.102:0/3719805389 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2240b8200 con 0x7fc2240a0420 2021-02-10T11:21:00.989 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:00.990 7fc2237fe700 1 --2- 172.21.2.102:0/3719805389 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc2240a0420 0x7fc2240a0820 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 rx=0x7fc21400a570 tx=0x7fc21400a910).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:00.989 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:00.990 7fc21bfff700 1 -- 172.21.2.102:0/3719805389 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc21400ad00 con 0x7fc2240a0420 2021-02-10T11:21:00.989 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:00.990 7fc21bfff700 1 -- 172.21.2.102:0/3719805389 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc214001180 con 0x7fc2240a0420 2021-02-10T11:21:00.989 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:00.990 7fc229d8f700 1 -- 172.21.2.102:0/3719805389 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2240b8200 con 0x7fc2240a0420 2021-02-10T11:21:00.989 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:00.990 7fc21bfff700 1 -- 172.21.2.102:0/3719805389 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc214004590 con 0x7fc2240a0420 2021-02-10T11:21:00.990 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:00.990 7fc229d8f700 1 -- 172.21.2.102:0/3719805389 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2240b8200 con 0x7fc2240a0420 2021-02-10T11:21:00.990 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:00.991 7fc21bfff700 1 -- 172.21.2.102:0/3719805389 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fc2140043a0 con 0x7fc2240a0420 2021-02-10T11:21:00.990 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:00.991 7fc21bfff700 1 --2- 172.21.2.102:0/3719805389 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc208018730 0x7fc20801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:00.990 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:00.991 7fc21bfff700 1 -- 172.21.2.102:0/3719805389 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fc214032600 con 0x7fc2240a0420 2021-02-10T11:21:00.990 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:00.991 7fc222ffd700 1 --2- 172.21.2.102:0/3719805389 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc208018730 0x7fc20801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:00.990 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:00.991 7fc222ffd700 1 --2- 172.21.2.102:0/3719805389 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc208018730 0x7fc20801ac10 crc :-1 s=READY pgs=419 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:01.015 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:01.021 7fc229d8f700 1 -- 172.21.2.102:0/3719805389 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc224105ae0 con 0x7fc2240a0420 2021-02-10T11:21:01.017 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:01.024 7fc21bfff700 1 -- 172.21.2.102:0/3719805389 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fc214000c90 con 0x7fc2240a0420 2021-02-10T11:21:01.144 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:01.150 7fc229d8f700 1 -- 172.21.2.102:0/3719805389 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344396", "target": ["mgr", ""]}) v1 -- 0x7fc22418ab20 con 0x7fc208018730 2021-02-10T11:21:01.155 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:01.161 7fc21bfff700 1 -- 172.21.2.102:0/3719805389 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fc20c009180 con 0x7fc208018730 2021-02-10T11:21:01.161 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:01.167 7fc229d8f700 1 -- 172.21.2.102:0/3719805389 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc208018730 msgr2=0x7fc20801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:01.161 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:01.167 7fc229d8f700 1 --2- 172.21.2.102:0/3719805389 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc208018730 0x7fc20801ac10 crc :-1 s=READY pgs=419 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:01.161 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:01.167 7fc229d8f700 1 -- 172.21.2.102:0/3719805389 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc2240a0420 msgr2=0x7fc2240a0820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:01.161 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:01.167 7fc229d8f700 1 --2- 172.21.2.102:0/3719805389 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc2240a0420 0x7fc2240a0820 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 rx=0x7fc21400a570 tx=0x7fc21400a910).stop 2021-02-10T11:21:01.161 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:01.167 7fc229d8f700 1 -- 172.21.2.102:0/3719805389 shutdown_connections 2021-02-10T11:21:01.161 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:01.167 7fc229d8f700 1 --2- 172.21.2.102:0/3719805389 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc208018730 0x7fc20801ac10 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:01.162 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:01.167 7fc229d8f700 1 --2- 172.21.2.102:0/3719805389 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc22418a720 0x7fc2240b7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:01.162 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:01.167 7fc229d8f700 1 --2- 172.21.2.102:0/3719805389 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc224189d00 0x7fc22418a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:01.162 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:01.167 7fc229d8f700 1 --2- 172.21.2.102:0/3719805389 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc2240a0420 0x7fc2240a0820 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:01.162 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:01.167 7fc229d8f700 1 -- 172.21.2.102:0/3719805389 >> 172.21.2.102:0/3719805389 conn(0x7fc224094e10 msgr2=0x7fc2240a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:01.162 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:01.167 7fc229d8f700 1 -- 172.21.2.102:0/3719805389 shutdown_connections 2021-02-10T11:21:01.162 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:01.167 7fc229d8f700 1 -- 172.21.2.102:0/3719805389 wait complete. 2021-02-10T11:21:01.170 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000344397 2021-02-10T11:21:01.198 DEBUG:teuthology.orchestra.run.gibba002:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:21:01.214 DEBUG:teuthology.orchestra.run.gibba013:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:21:01.603 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:01.592 7fd411e65700 1 Processor -- start 2021-02-10T11:21:07.124 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:01.592 7fd411e65700 1 -- start start 2021-02-10T11:21:07.124 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:01.593 7fd411e65700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd40c1b0470 0x7fd40c1b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:07.124 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:01.593 7fd411e65700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fd40c0ef610 0x7fd40c0efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:07.124 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:01.593 7fd411e65700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd40c0f0b50 con 0x7fd40c0f0030 2021-02-10T11:21:07.124 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:01.593 7fd411e65700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd40c0e52e0 con 0x7fd40c0ef610 2021-02-10T11:21:07.125 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:01.593 7fd411e65700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd40c0e5490 con 0x7fd40c1b0470 2021-02-10T11:21:07.125 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:01.593 7fd40bfff700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fd40c0f0030 0x7fd40c0f10d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:46378/0 (socket says 172.21.2.102:46378) 2021-02-10T11:21:07.125 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:01.593 7fd40bfff700 1 -- 172.21.2.102:0/2192149397 learned_addr learned my addr 172.21.2.102:0/2192149397 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:21:07.125 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:01.593 7fd40affd700 1 --2- 172.21.2.102:0/2192149397 >> v2:172.21.2.113:3300/0 conn(0x7fd40c0ef610 0x7fd40c0efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:07.125 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:01.593 7fd40b7fe700 1 --2- 172.21.2.102:0/2192149397 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd40c1b0470 0x7fd40c1b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:07.125 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:01.593 7fd40a7fc700 1 -- 172.21.2.102:0/2192149397 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 292481517 0 0) 0x7fd404000d30 con 0x7fd40c0f0030 2021-02-10T11:21:07.125 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:01.593 7fd40a7fc700 1 -- 172.21.2.102:0/2192149397 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd3f4002fe0 con 0x7fd40c0f0030 2021-02-10T11:21:07.125 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:01.594 7fd40b7fe700 1 -- 172.21.2.102:0/2192149397 >> v2:172.21.2.113:3300/0 conn(0x7fd40c0ef610 msgr2=0x7fd40c0efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:21:07.125 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:01.594 7fd40a7fc700 1 -- 172.21.2.102:0/2192149397 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 876888222 0 0) 0x7fd404001500 con 0x7fd40c0f0030 2021-02-10T11:21:07.125 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:01.594 7fd40b7fe700 1 --2- 172.21.2.102:0/2192149397 >> v2:172.21.2.113:3300/0 conn(0x7fd40c0ef610 0x7fd40c0efa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:07.125 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:01.594 7fd40b7fe700 1 -- 172.21.2.102:0/2192149397 >> v1:172.21.2.102:6789/0 conn(0x7fd40c0f0030 legacy=0x7fd40c0f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:07.125 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:01.594 7fd40b7fe700 1 -- 172.21.2.102:0/2192149397 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd40c0f0430 con 0x7fd40c1b0470 2021-02-10T11:21:07.125 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:01.594 7fd40b7fe700 1 --2- 172.21.2.102:0/2192149397 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd40c1b0470 0x7fd40c1b0870 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7fd3fc006650 tx=0x7fd3fc009a80).ready entity=mon.2 client_cookie=672dbc6d3aa5f096 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:07.126 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:01.594 7fd40a7fc700 1 -- 172.21.2.102:0/2192149397 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd3fc00c0c0 con 0x7fd40c1b0470 2021-02-10T11:21:07.126 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:01.594 7fd40a7fc700 1 -- 172.21.2.102:0/2192149397 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3fc00c340 con 0x7fd40c1b0470 2021-02-10T11:21:07.126 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:01.594 7fd40a7fc700 1 -- 172.21.2.102:0/2192149397 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd3fc00c860 con 0x7fd40c1b0470 2021-02-10T11:21:07.126 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:01.594 7fd411e65700 1 -- 172.21.2.102:0/2192149397 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd40c1b0470 msgr2=0x7fd40c1b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:07.126 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:01.594 7fd411e65700 1 --2- 172.21.2.102:0/2192149397 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd40c1b0470 0x7fd40c1b0870 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7fd3fc006650 tx=0x7fd3fc009a80).stop 2021-02-10T11:21:07.126 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:01.595 7fd411e65700 1 -- 172.21.2.102:0/2192149397 shutdown_connections 2021-02-10T11:21:07.126 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:01.595 7fd411e65700 1 --2- 172.21.2.102:0/2192149397 >> v2:172.21.2.113:3300/0 conn(0x7fd40c0ef610 0x7fd40c0efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:07.126 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:01.595 7fd411e65700 1 --2- 172.21.2.102:0/2192149397 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd40c1b0470 0x7fd40c1b0870 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:07.126 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:01.595 7fd411e65700 1 -- 172.21.2.102:0/2192149397 >> 172.21.2.102:0/2192149397 conn(0x7fd40c0ce3e0 msgr2=0x7fd40c0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:07.126 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:01.595 7fd411e65700 1 -- 172.21.2.102:0/2192149397 shutdown_connections 2021-02-10T11:21:07.126 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:01.595 7fd411e65700 1 -- 172.21.2.102:0/2192149397 wait complete. 2021-02-10T11:21:07.126 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:01.610 7fd411e65700 1 Processor -- start 2021-02-10T11:21:07.129 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:01.610 7fd411e65700 1 -- start start 2021-02-10T11:21:07.129 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:01.610 7fd411e65700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd40c0a0420 0x7fd40c0a0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:07.129 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:01.610 7fd411e65700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd40c189d00 0x7fd40c18a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:07.129 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:01.610 7fd411e65700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd40c18a720 0x7fd40c0b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:07.129 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:01.610 7fd411e65700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd40c189800 con 0x7fd40c0a0420 2021-02-10T11:21:07.129 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:01.610 7fd411e65700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd40c18ab20 con 0x7fd40c189d00 2021-02-10T11:21:07.130 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:01.610 7fd411e65700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd40c0a1140 con 0x7fd40c18a720 2021-02-10T11:21:07.130 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:01.610 7fd40b7fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd40c0a0420 0x7fd40c0a0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:07.130 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:01.610 7fd40b7fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd40c0a0420 0x7fd40c0a0820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:54160/0 (socket says 172.21.2.102:54160) 2021-02-10T11:21:07.130 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:01.610 7fd40b7fe700 1 -- 172.21.2.102:0/1084633733 learned_addr learned my addr 172.21.2.102:0/1084633733 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:21:07.130 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:01.611 7fd40b7fe700 1 -- 172.21.2.102:0/1084633733 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd40c18a720 msgr2=0x7fd40c0b7be0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:21:07.130 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:01.611 7fd40b7fe700 1 --2- 172.21.2.102:0/1084633733 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd40c18a720 0x7fd40c0b7be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:21:07.130 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:01.611 7fd40b7fe700 1 -- 172.21.2.102:0/1084633733 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd40c189d00 msgr2=0x7fd40c18a100 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:21:07.130 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:01.611 7fd40b7fe700 1 --2- 172.21.2.102:0/1084633733 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd40c189d00 0x7fd40c18a100 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:21:07.130 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:01.611 7fd40b7fe700 1 -- 172.21.2.102:0/1084633733 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd40c0b8200 con 0x7fd40c0a0420 2021-02-10T11:21:07.130 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:01.611 7fd40b7fe700 1 --2- 172.21.2.102:0/1084633733 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd40c0a0420 0x7fd40c0a0820 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 rx=0x7fd3fc009630 tx=0x7fd3fc00b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:07.130 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:01.611 7fd3f3fff700 1 -- 172.21.2.102:0/1084633733 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd3fc0093a0 con 0x7fd40c0a0420 2021-02-10T11:21:07.130 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:01.611 7fd3f3fff700 1 -- 172.21.2.102:0/1084633733 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3fc00bd70 con 0x7fd40c0a0420 2021-02-10T11:21:07.130 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:01.611 7fd3f3fff700 1 -- 172.21.2.102:0/1084633733 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd3fc0054d0 con 0x7fd40c0a0420 2021-02-10T11:21:07.131 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:01.611 7fd411e65700 1 -- 172.21.2.102:0/1084633733 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd40c0b8200 con 0x7fd40c0a0420 2021-02-10T11:21:07.131 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:01.611 7fd411e65700 1 -- 172.21.2.102:0/1084633733 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd40c0b8200 con 0x7fd40c0a0420 2021-02-10T11:21:07.131 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:01.611 7fd3f3fff700 1 -- 172.21.2.102:0/1084633733 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd3fc00a000 con 0x7fd40c0a0420 2021-02-10T11:21:07.131 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:01.612 7fd3f3fff700 1 --2- 172.21.2.102:0/1084633733 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd3ec018730 0x7fd3ec01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:07.131 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:01.612 7fd3f3fff700 1 -- 172.21.2.102:0/1084633733 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fd3fc032810 con 0x7fd40c0a0420 2021-02-10T11:21:07.131 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:01.612 7fd40affd700 1 --2- 172.21.2.102:0/1084633733 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd3ec018730 0x7fd3ec01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:07.131 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:01.612 7fd40affd700 1 --2- 172.21.2.102:0/1084633733 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd3ec018730 0x7fd3ec01abc0 crc :-1 s=READY pgs=420 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:07.131 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:01.642 7fd411e65700 1 -- 172.21.2.102:0/1084633733 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd40c105ae0 con 0x7fd40c0a0420 2021-02-10T11:21:07.131 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:01.645 7fd3f3fff700 1 -- 172.21.2.102:0/1084633733 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd3fc01f550 con 0x7fd40c0a0420 2021-02-10T11:21:07.131 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:01.775 7fd411e65700 1 -- 172.21.2.102:0/1084633733 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344397", "target": ["mgr", ""]}) v1 -- 0x7fd40c18ab20 con 0x7fd3ec018730 2021-02-10T11:21:07.131 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:01.786 7fd3f3fff700 1 -- 172.21.2.102:0/1084633733 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fd4000009b0 con 0x7fd3ec018730 2021-02-10T11:21:07.131 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:01.792 7fd411e65700 1 -- 172.21.2.102:0/1084633733 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd3ec018730 msgr2=0x7fd3ec01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:07.131 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:01.792 7fd411e65700 1 --2- 172.21.2.102:0/1084633733 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd3ec018730 0x7fd3ec01abc0 crc :-1 s=READY pgs=420 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:07.131 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:01.792 7fd411e65700 1 -- 172.21.2.102:0/1084633733 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd40c0a0420 msgr2=0x7fd40c0a0820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:07.132 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:01.792 7fd411e65700 1 --2- 172.21.2.102:0/1084633733 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd40c0a0420 0x7fd40c0a0820 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 rx=0x7fd3fc009630 tx=0x7fd3fc00b980).stop 2021-02-10T11:21:07.132 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:01.792 7fd411e65700 1 -- 172.21.2.102:0/1084633733 shutdown_connections 2021-02-10T11:21:07.132 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:01.792 7fd411e65700 1 --2- 172.21.2.102:0/1084633733 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd40c18a720 0x7fd40c0b7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:21:07.132 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:01.792 7fd411e65700 1 --2- 172.21.2.102:0/1084633733 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd3ec018730 0x7fd3ec01abc0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:07.132 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:01.792 7fd411e65700 1 --2- 172.21.2.102:0/1084633733 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd40c189d00 0x7fd40c18a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:21:07.132 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:01.792 7fd411e65700 1 --2- 172.21.2.102:0/1084633733 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd40c0a0420 0x7fd40c0a0820 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:07.132 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:01.792 7fd411e65700 1 -- 172.21.2.102:0/1084633733 >> 172.21.2.102:0/1084633733 conn(0x7fd40c094e10 msgr2=0x7fd40c0a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:07.132 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:01.792 7fd411e65700 1 -- 172.21.2.102:0/1084633733 shutdown_connections 2021-02-10T11:21:07.132 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:01.792 7fd411e65700 1 -- 172.21.2.102:0/1084633733 wait complete. 2021-02-10T11:21:07.133 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000344398 2021-02-10T11:21:07.563 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:07.552 7fcb7d3d0700 1 Processor -- start 2021-02-10T11:21:07.564 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:07.552 7fcb7d3d0700 1 -- start start 2021-02-10T11:21:07.564 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:07.553 7fcb7d3d0700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fcb781b0470 0x7fcb781b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:07.564 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:07.553 7fcb7d3d0700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fcb780f20e0 0x7fcb780f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:07.564 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:07.553 7fcb7d3d0700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb780f0b50 con 0x7fcb780ef610 2021-02-10T11:21:07.564 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:07.553 7fcb7d3d0700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb780f0820 con 0x7fcb781b0470 2021-02-10T11:21:07.564 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:07.553 7fcb7d3d0700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb780f24e0 con 0x7fcb780f20e0 2021-02-10T11:21:07.564 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:07.553 7fcb7707d700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fcb781b0470 0x7fcb781b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:07.564 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:07.553 7fcb7707d700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fcb781b0470 0x7fcb781b0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:54166/0 (socket says 172.21.2.102:54166) 2021-02-10T11:21:07.564 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:07.553 7fcb7707d700 1 -- 172.21.2.102:0/3607542473 learned_addr learned my addr 172.21.2.102:0/3607542473 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:21:07.564 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:07.553 7fcb7787e700 1 --2- 172.21.2.102:0/3607542473 >> v2:172.21.2.113:3300/0 conn(0x7fcb780f20e0 0x7fcb780f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:07.564 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:07.553 7fcb7707d700 1 -- 172.21.2.102:0/3607542473 >> v2:172.21.2.113:3300/0 conn(0x7fcb780f20e0 msgr2=0x7fcb780f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:21:07.564 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:07.553 7fcb7707d700 1 --2- 172.21.2.102:0/3607542473 >> v2:172.21.2.113:3300/0 conn(0x7fcb780f20e0 0x7fcb780f0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:07.565 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:07.553 7fcb7707d700 1 -- 172.21.2.102:0/3607542473 >> v1:172.21.2.113:6789/0 conn(0x7fcb780ef610 legacy=0x7fcb780efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:21:07.565 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:07.553 7fcb7707d700 1 -- 172.21.2.102:0/3607542473 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb780e52e0 con 0x7fcb781b0470 2021-02-10T11:21:07.565 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:07.553 7fcb7707d700 1 --2- 172.21.2.102:0/3607542473 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcb781b0470 0x7fcb781b0870 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 rx=0x7fcb68006700 tx=0x7fcb68009b80).ready entity=mon.0 client_cookie=8f59493f204cc7cb server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:07.565 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:07.554 7fcb7607b700 1 -- 172.21.2.102:0/3607542473 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcb6800c0c0 con 0x7fcb781b0470 2021-02-10T11:21:07.565 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:07.554 7fcb7607b700 1 -- 172.21.2.102:0/3607542473 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb6800c440 con 0x7fcb781b0470 2021-02-10T11:21:07.565 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:07.554 7fcb7607b700 1 -- 172.21.2.102:0/3607542473 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcb6800c960 con 0x7fcb781b0470 2021-02-10T11:21:07.565 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:07.554 7fcb7d3d0700 1 -- 172.21.2.102:0/3607542473 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcb781b0470 msgr2=0x7fcb781b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:07.565 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:07.554 7fcb7d3d0700 1 --2- 172.21.2.102:0/3607542473 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcb781b0470 0x7fcb781b0870 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 rx=0x7fcb68006700 tx=0x7fcb68009b80).stop 2021-02-10T11:21:07.565 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:07.554 7fcb7d3d0700 1 -- 172.21.2.102:0/3607542473 shutdown_connections 2021-02-10T11:21:07.565 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:07.554 7fcb7d3d0700 1 --2- 172.21.2.102:0/3607542473 >> v2:172.21.2.113:3300/0 conn(0x7fcb780f20e0 0x7fcb780f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:07.565 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:07.554 7fcb7d3d0700 1 --2- 172.21.2.102:0/3607542473 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcb781b0470 0x7fcb781b0870 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:07.565 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:07.554 7fcb7d3d0700 1 -- 172.21.2.102:0/3607542473 >> 172.21.2.102:0/3607542473 conn(0x7fcb780ce3e0 msgr2=0x7fcb780e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:07.565 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:07.554 7fcb7d3d0700 1 -- 172.21.2.102:0/3607542473 shutdown_connections 2021-02-10T11:21:07.565 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:07.554 7fcb7d3d0700 1 -- 172.21.2.102:0/3607542473 wait complete. 2021-02-10T11:21:07.567 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:07.569 7fcb7d3d0700 1 Processor -- start 2021-02-10T11:21:07.567 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:07.569 7fcb7d3d0700 1 -- start start 2021-02-10T11:21:07.567 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:07.570 7fcb7d3d0700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcb780a0420 0x7fcb780a0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:07.568 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:07.570 7fcb7d3d0700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcb78189d00 0x7fcb7818a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:07.568 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:07.570 7fcb7d3d0700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fcb7818a720 0x7fcb780b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:07.568 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:07.570 7fcb7d3d0700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb78189800 con 0x7fcb78189d00 2021-02-10T11:21:07.568 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:07.570 7fcb7d3d0700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb7818ab20 con 0x7fcb7818a720 2021-02-10T11:21:07.568 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:07.570 7fcb7d3d0700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb780a1140 con 0x7fcb780a0420 2021-02-10T11:21:07.568 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:07.570 7fcb7687c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcb78189d00 0x7fcb7818a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:07.568 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:07.570 7fcb7687c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcb78189d00 0x7fcb7818a100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:54176/0 (socket says 172.21.2.102:54176) 2021-02-10T11:21:07.568 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:07.570 7fcb7687c700 1 -- 172.21.2.102:0/2859961640 learned_addr learned my addr 172.21.2.102:0/2859961640 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:21:07.568 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:07.570 7fcb7787e700 1 --2- 172.21.2.102:0/2859961640 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fcb7818a720 0x7fcb780b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:07.568 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:07.570 7fcb7707d700 1 --2- 172.21.2.102:0/2859961640 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcb780a0420 0x7fcb780a0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:07.568 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:07.570 7fcb7687c700 1 -- 172.21.2.102:0/2859961640 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcb780a0420 msgr2=0x7fcb780a0820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:07.568 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:07.570 7fcb7687c700 1 --2- 172.21.2.102:0/2859961640 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcb780a0420 0x7fcb780a0820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:07.568 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:07.570 7fcb7687c700 1 -- 172.21.2.102:0/2859961640 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fcb7818a720 msgr2=0x7fcb780b7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:07.569 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:07.570 7fcb7687c700 1 --2- 172.21.2.102:0/2859961640 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fcb7818a720 0x7fcb780b7be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:07.569 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:07.570 7fcb7687c700 1 -- 172.21.2.102:0/2859961640 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb780b8200 con 0x7fcb78189d00 2021-02-10T11:21:07.569 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:07.570 7fcb7787e700 1 --2- 172.21.2.102:0/2859961640 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fcb7818a720 0x7fcb780b7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:21:07.569 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:07.570 7fcb7687c700 1 --2- 172.21.2.102:0/2859961640 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcb78189d00 0x7fcb7818a100 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 rx=0x7fcb6c00b530 tx=0x7fcb6c00b8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:07.569 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:07.571 7fcb5ffff700 1 -- 172.21.2.102:0/2859961640 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcb6c00bcc0 con 0x7fcb78189d00 2021-02-10T11:21:07.569 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:07.571 7fcb5ffff700 1 -- 172.21.2.102:0/2859961640 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb6c006020 con 0x7fcb78189d00 2021-02-10T11:21:07.569 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:07.571 7fcb5ffff700 1 -- 172.21.2.102:0/2859961640 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcb6c0054a0 con 0x7fcb78189d00 2021-02-10T11:21:07.569 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:07.571 7fcb7d3d0700 1 -- 172.21.2.102:0/2859961640 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb780b8200 con 0x7fcb78189d00 2021-02-10T11:21:07.569 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:07.571 7fcb7d3d0700 1 -- 172.21.2.102:0/2859961640 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb780b8790 con 0x7fcb78189d00 2021-02-10T11:21:07.569 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:07.571 7fcb5ffff700 1 -- 172.21.2.102:0/2859961640 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fcb6c006200 con 0x7fcb78189d00 2021-02-10T11:21:07.570 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:07.571 7fcb5ffff700 1 --2- 172.21.2.102:0/2859961640 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcb580186c0 0x7fcb5801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:07.570 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:07.571 7fcb5ffff700 1 -- 172.21.2.102:0/2859961640 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fcb6c032400 con 0x7fcb78189d00 2021-02-10T11:21:07.570 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:07.571 7fcb7707d700 1 --2- 172.21.2.102:0/2859961640 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcb580186c0 0x7fcb5801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:07.570 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:07.572 7fcb7707d700 1 --2- 172.21.2.102:0/2859961640 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcb580186c0 0x7fcb5801abc0 crc :-1 s=READY pgs=421 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:07.597 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:07.602 7fcb7d3d0700 1 -- 172.21.2.102:0/2859961640 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb78105ae0 con 0x7fcb78189d00 2021-02-10T11:21:07.601 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:07.605 7fcb5ffff700 1 -- 172.21.2.102:0/2859961640 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fcb6c02cf00 con 0x7fcb78189d00 2021-02-10T11:21:07.725 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:07.730 7fcb7d3d0700 1 -- 172.21.2.102:0/2859961640 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344398", "target": ["mgr", ""]}) v1 -- 0x7fcb780b9020 con 0x7fcb580186c0 2021-02-10T11:21:07.737 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:07.742 7fcb5ffff700 1 -- 172.21.2.102:0/2859961640 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fcb680009b0 con 0x7fcb580186c0 2021-02-10T11:21:07.743 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:07.747 7fcb7d3d0700 1 -- 172.21.2.102:0/2859961640 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcb580186c0 msgr2=0x7fcb5801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:07.743 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:07.747 7fcb7d3d0700 1 --2- 172.21.2.102:0/2859961640 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcb580186c0 0x7fcb5801abc0 crc :-1 s=READY pgs=421 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:07.743 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:07.747 7fcb7d3d0700 1 -- 172.21.2.102:0/2859961640 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcb78189d00 msgr2=0x7fcb7818a100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:07.743 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:07.747 7fcb7d3d0700 1 --2- 172.21.2.102:0/2859961640 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcb78189d00 0x7fcb7818a100 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 rx=0x7fcb6c00b530 tx=0x7fcb6c00b8d0).stop 2021-02-10T11:21:07.743 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:07.748 7fcb7d3d0700 1 -- 172.21.2.102:0/2859961640 shutdown_connections 2021-02-10T11:21:07.743 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:07.748 7fcb7d3d0700 1 --2- 172.21.2.102:0/2859961640 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcb580186c0 0x7fcb5801abc0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:07.743 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:07.748 7fcb7d3d0700 1 --2- 172.21.2.102:0/2859961640 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fcb7818a720 0x7fcb780b7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:07.743 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:07.748 7fcb7d3d0700 1 --2- 172.21.2.102:0/2859961640 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcb78189d00 0x7fcb7818a100 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:07.743 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:07.748 7fcb7d3d0700 1 --2- 172.21.2.102:0/2859961640 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcb780a0420 0x7fcb780a0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:07.743 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:07.748 7fcb7d3d0700 1 -- 172.21.2.102:0/2859961640 >> 172.21.2.102:0/2859961640 conn(0x7fcb78094e10 msgr2=0x7fcb780a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:07.743 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:07.748 7fcb7d3d0700 1 -- 172.21.2.102:0/2859961640 shutdown_connections 2021-02-10T11:21:07.744 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:07.748 7fcb7d3d0700 1 -- 172.21.2.102:0/2859961640 wait complete. 2021-02-10T11:21:07.752 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000344399 2021-02-10T11:21:08.183 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:08.172 7f77c9586700 1 Processor -- start 2021-02-10T11:21:08.183 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:08.172 7f77c9586700 1 -- start start 2021-02-10T11:21:08.184 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:08.172 7f77c9586700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f77c41b0470 0x7f77c41b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:08.184 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:08.172 7f77c9586700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77c40f0b50 con 0x7f77c40f20e0 2021-02-10T11:21:08.184 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:08.172 7f77c9586700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77c40f24e0 con 0x7f77c40ef610 2021-02-10T11:21:08.184 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:08.172 7f77c9586700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77c40e5ea0 con 0x7f77c41b0470 2021-02-10T11:21:08.184 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:08.173 7f77c32bf700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f77c41b0470 0x7f77c41b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:08.184 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:08.173 7f77c32bf700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f77c41b0470 0x7f77c41b0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:54180/0 (socket says 172.21.2.102:54180) 2021-02-10T11:21:08.184 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:08.173 7f77c32bf700 1 -- 172.21.2.102:0/2823693753 learned_addr learned my addr 172.21.2.102:0/2823693753 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:21:08.184 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:08.173 7f77c32bf700 1 -- 172.21.2.102:0/2823693753 >> v1:172.21.2.113:6789/0 conn(0x7f77c40ef610 legacy=0x7f77c40efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:08.184 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:08.173 7f77c32bf700 1 -- 172.21.2.102:0/2823693753 >> v1:172.21.2.102:6789/0 conn(0x7f77c40f20e0 legacy=0x7f77c40f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:08.184 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:08.173 7f77c22bd700 1 -- 172.21.2.102:0/2823693753 <== mon.1 v1:172.21.2.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2591843452 0 0) 0x7f77b8000ff0 con 0x7f77c40ef610 2021-02-10T11:21:08.184 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:08.173 7f77c32bf700 1 -- 172.21.2.102:0/2823693753 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77c40e52e0 con 0x7f77c41b0470 2021-02-10T11:21:08.184 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:08.173 7f77c32bf700 1 --2- 172.21.2.102:0/2823693753 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f77c41b0470 0x7f77c41b0870 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 rx=0x7f77b4006700 tx=0x7f77b4009b80).ready entity=mon.0 client_cookie=699389c704285d10 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:08.185 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:08.173 7f77c22bd700 1 -- 172.21.2.102:0/2823693753 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f77b400c0c0 con 0x7f77c41b0470 2021-02-10T11:21:08.185 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:08.174 7f77c22bd700 1 -- 172.21.2.102:0/2823693753 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77b400c440 con 0x7f77c41b0470 2021-02-10T11:21:08.185 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:08.174 7f77c22bd700 1 -- 172.21.2.102:0/2823693753 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f77b400c960 con 0x7f77c41b0470 2021-02-10T11:21:08.185 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:08.174 7f77c9586700 1 -- 172.21.2.102:0/2823693753 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f77c41b0470 msgr2=0x7f77c41b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:08.185 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:08.174 7f77c9586700 1 --2- 172.21.2.102:0/2823693753 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f77c41b0470 0x7f77c41b0870 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 rx=0x7f77b4006700 tx=0x7f77b4009b80).stop 2021-02-10T11:21:08.185 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:08.174 7f77c9586700 1 -- 172.21.2.102:0/2823693753 shutdown_connections 2021-02-10T11:21:08.185 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:08.174 7f77c9586700 1 --2- 172.21.2.102:0/2823693753 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f77c41b0470 0x7f77c41b0870 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:08.185 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:08.174 7f77c9586700 1 -- 172.21.2.102:0/2823693753 >> 172.21.2.102:0/2823693753 conn(0x7f77c40ce3e0 msgr2=0x7f77c40e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:08.185 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:08.174 7f77c9586700 1 -- 172.21.2.102:0/2823693753 shutdown_connections 2021-02-10T11:21:08.185 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:08.174 7f77c9586700 1 -- 172.21.2.102:0/2823693753 wait complete. 2021-02-10T11:21:08.187 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:08.189 7f77c9586700 1 Processor -- start 2021-02-10T11:21:08.187 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:08.189 7f77c9586700 1 -- start start 2021-02-10T11:21:08.187 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:08.189 7f77c9586700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f77c41a0570 0x7f77c40a4590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:08.187 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:08.189 7f77c9586700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f77c40f1fd0 0x7f77c40f23d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:08.187 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:08.189 7f77c9586700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f77c40a5450 0x7f77c40a5850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:08.187 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:08.189 7f77c9586700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77c40a4bb0 con 0x7f77c41a0570 2021-02-10T11:21:08.187 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:08.189 7f77c9586700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77c40b35e0 con 0x7f77c40a5450 2021-02-10T11:21:08.187 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:08.189 7f77c9586700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77c40a1f00 con 0x7f77c40f1fd0 2021-02-10T11:21:08.187 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:08.189 7f77c32bf700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f77c41a0570 0x7f77c40a4590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:08.187 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:08.189 7f77c32bf700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f77c41a0570 0x7f77c40a4590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:54186/0 (socket says 172.21.2.102:54186) 2021-02-10T11:21:08.188 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:08.189 7f77c32bf700 1 -- 172.21.2.102:0/3755630233 learned_addr learned my addr 172.21.2.102:0/3755630233 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:21:08.188 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:08.190 7f77c32bf700 1 -- 172.21.2.102:0/3755630233 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f77c40f1fd0 msgr2=0x7f77c40f23d0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:21:08.188 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:08.190 7f77c32bf700 1 --2- 172.21.2.102:0/3755630233 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f77c40f1fd0 0x7f77c40f23d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:21:08.188 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:08.190 7f77c32bf700 1 -- 172.21.2.102:0/3755630233 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f77c40a5450 msgr2=0x7f77c40a5850 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:21:08.188 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:08.190 7f77c32bf700 1 --2- 172.21.2.102:0/3755630233 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f77c40a5450 0x7f77c40a5850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:21:08.188 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:08.190 7f77c32bf700 1 -- 172.21.2.102:0/3755630233 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77c40a5e70 con 0x7f77c41a0570 2021-02-10T11:21:08.188 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:08.190 7f77c32bf700 1 --2- 172.21.2.102:0/3755630233 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f77c41a0570 0x7f77c40a4590 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 rx=0x7f77b400b5e0 tx=0x7f77b400b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:08.188 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:08.190 7f77abfff700 1 -- 172.21.2.102:0/3755630233 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f77b400bd70 con 0x7f77c41a0570 2021-02-10T11:21:08.188 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:08.190 7f77abfff700 1 -- 172.21.2.102:0/3755630233 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77b4002030 con 0x7f77c41a0570 2021-02-10T11:21:08.188 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:08.190 7f77c9586700 1 -- 172.21.2.102:0/3755630233 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77c40a5e70 con 0x7f77c41a0570 2021-02-10T11:21:08.188 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:08.190 7f77c9586700 1 -- 172.21.2.102:0/3755630233 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77c40a6400 con 0x7f77c41a0570 2021-02-10T11:21:08.188 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:08.190 7f77abfff700 1 -- 172.21.2.102:0/3755630233 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f77b4002550 con 0x7f77c41a0570 2021-02-10T11:21:08.188 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:08.191 7f77abfff700 1 -- 172.21.2.102:0/3755630233 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f77b4002210 con 0x7f77c41a0570 2021-02-10T11:21:08.189 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:08.191 7f77abfff700 1 --2- 172.21.2.102:0/3755630233 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f77a40186c0 0x7f77a401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:08.189 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:08.191 7f77abfff700 1 -- 172.21.2.102:0/3755630233 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f77b4032560 con 0x7f77c41a0570 2021-02-10T11:21:08.189 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:08.191 7f77c2abe700 1 --2- 172.21.2.102:0/3755630233 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f77a40186c0 0x7f77a401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:08.189 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:08.191 7f77c2abe700 1 --2- 172.21.2.102:0/3755630233 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f77a40186c0 0x7f77a401abc0 crc :-1 s=READY pgs=422 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:08.218 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:08.223 7f77c9586700 1 -- 172.21.2.102:0/3755630233 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f779c002fd0 con 0x7f77c41a0570 2021-02-10T11:21:08.220 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:08.225 7f77abfff700 1 -- 172.21.2.102:0/3755630233 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f77b4000940 con 0x7f77c41a0570 2021-02-10T11:21:08.352 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:08.357 7f77c9586700 1 -- 172.21.2.102:0/3755630233 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344399", "target": ["mgr", ""]}) v1 -- 0x7f779c002f80 con 0x7f77a40186c0 2021-02-10T11:21:08.362 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:08.367 7f77abfff700 1 -- 172.21.2.102:0/3755630233 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f77b8006b80 con 0x7f77a40186c0 2021-02-10T11:21:08.370 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:08.374 7f77c9586700 1 -- 172.21.2.102:0/3755630233 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f77a40186c0 msgr2=0x7f77a401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:08.370 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:08.374 7f77c9586700 1 --2- 172.21.2.102:0/3755630233 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f77a40186c0 0x7f77a401abc0 crc :-1 s=READY pgs=422 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:08.370 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:08.374 7f77c9586700 1 -- 172.21.2.102:0/3755630233 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f77c41a0570 msgr2=0x7f77c40a4590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:08.370 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:08.374 7f77c9586700 1 --2- 172.21.2.102:0/3755630233 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f77c41a0570 0x7f77c40a4590 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 rx=0x7f77b400b5e0 tx=0x7f77b400b980).stop 2021-02-10T11:21:08.370 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:08.374 7f77c9586700 1 -- 172.21.2.102:0/3755630233 shutdown_connections 2021-02-10T11:21:08.370 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:08.374 7f77c9586700 1 --2- 172.21.2.102:0/3755630233 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f77a40186c0 0x7f77a401abc0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:08.370 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:08.374 7f77c9586700 1 --2- 172.21.2.102:0/3755630233 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f77c40a5450 0x7f77c40a5850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:21:08.370 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:08.374 7f77c9586700 1 --2- 172.21.2.102:0/3755630233 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f77c40f1fd0 0x7f77c40f23d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:21:08.370 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:08.375 7f77c9586700 1 --2- 172.21.2.102:0/3755630233 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f77c41a0570 0x7f77c40a4590 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:08.371 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:08.375 7f77c9586700 1 -- 172.21.2.102:0/3755630233 >> 172.21.2.102:0/3755630233 conn(0x7f77c40958b0 msgr2=0x7f77c40bf040 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:08.371 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:08.375 7f77c9586700 1 -- 172.21.2.102:0/3755630233 shutdown_connections 2021-02-10T11:21:08.371 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:08.375 7f77c9586700 1 -- 172.21.2.102:0/3755630233 wait complete. 2021-02-10T11:21:08.378 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000344400 2021-02-10T11:21:08.807 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:08.795 7f5fd108f700 1 Processor -- start 2021-02-10T11:21:08.807 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:08.795 7f5fd108f700 1 -- start start 2021-02-10T11:21:08.808 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:08.796 7f5fd108f700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f5fcc0f05f0 0x7f5fcc0f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:08.808 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:08.796 7f5fd108f700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fcc1b0c70 con 0x7f5fcc1b0470 2021-02-10T11:21:08.808 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:08.796 7f5fd108f700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fcc0f0b50 con 0x7f5fcc1b0870 2021-02-10T11:21:08.808 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:08.796 7f5fd108f700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5fcc0e5ea0 con 0x7f5fcc0f05f0 2021-02-10T11:21:08.808 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:08.796 7f5fcad9d700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f5fcc1b0470 0x7f5fcc0ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:46416/0 (socket says 172.21.2.102:46416) 2021-02-10T11:21:08.808 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:08.796 7f5fcad9d700 1 -- 172.21.2.102:0/1863578518 learned_addr learned my addr 172.21.2.102:0/1863578518 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:21:08.808 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:08.796 7f5fcb59e700 1 --2- 172.21.2.102:0/1863578518 >> v2:172.21.2.113:3300/0 conn(0x7f5fcc0f05f0 0x7f5fcc0f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:08.808 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:08.796 7f5fc9d9b700 1 -- 172.21.2.102:0/1863578518 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 275776586 0 0) 0x7f5fb4000ce0 con 0x7f5fcc1b0470 2021-02-10T11:21:08.808 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:08.796 7f5fc9d9b700 1 -- 172.21.2.102:0/1863578518 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5fb8002fe0 con 0x7f5fcc1b0470 2021-02-10T11:21:08.808 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:08.796 7f5fcb59e700 1 -- 172.21.2.102:0/1863578518 >> v1:172.21.2.113:6789/0 conn(0x7f5fcc1b0870 legacy=0x7f5fcc0efe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:08.808 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:08.796 7f5fcb59e700 1 -- 172.21.2.102:0/1863578518 >> v1:172.21.2.102:6789/0 conn(0x7f5fcc1b0470 legacy=0x7f5fcc0ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:08.808 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:08.796 7f5fcb59e700 1 -- 172.21.2.102:0/1863578518 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fcc0e52e0 con 0x7f5fcc0f05f0 2021-02-10T11:21:08.809 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:08.797 7f5fcb59e700 1 --2- 172.21.2.102:0/1863578518 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5fcc0f05f0 0x7f5fcc0f20e0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 rx=0x7f5fc4006e50 tx=0x7f5fc4009e70).ready entity=mon.1 client_cookie=3cf6ca89e8fbebb9 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:08.809 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:08.797 7f5fc9d9b700 1 -- 172.21.2.102:0/1863578518 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5fc4006a20 con 0x7f5fcc0f05f0 2021-02-10T11:21:08.809 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:08.797 7f5fc9d9b700 1 -- 172.21.2.102:0/1863578518 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fc4006790 con 0x7f5fcc0f05f0 2021-02-10T11:21:08.809 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:08.797 7f5fc9d9b700 1 -- 172.21.2.102:0/1863578518 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5fc400b700 con 0x7f5fcc0f05f0 2021-02-10T11:21:08.809 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:08.797 7f5fd108f700 1 -- 172.21.2.102:0/1863578518 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5fcc0f05f0 msgr2=0x7f5fcc0f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:08.809 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:08.797 7f5fd108f700 1 --2- 172.21.2.102:0/1863578518 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5fcc0f05f0 0x7f5fcc0f20e0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 rx=0x7f5fc4006e50 tx=0x7f5fc4009e70).stop 2021-02-10T11:21:08.809 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:08.798 7f5fd108f700 1 -- 172.21.2.102:0/1863578518 shutdown_connections 2021-02-10T11:21:08.809 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:08.798 7f5fd108f700 1 --2- 172.21.2.102:0/1863578518 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5fcc0f05f0 0x7f5fcc0f20e0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:08.809 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:08.798 7f5fd108f700 1 -- 172.21.2.102:0/1863578518 >> 172.21.2.102:0/1863578518 conn(0x7f5fcc0ce3e0 msgr2=0x7f5fcc0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:08.809 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:08.798 7f5fd108f700 1 -- 172.21.2.102:0/1863578518 shutdown_connections 2021-02-10T11:21:08.809 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:08.798 7f5fd108f700 1 -- 172.21.2.102:0/1863578518 wait complete. 2021-02-10T11:21:08.809 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:08.812 7f5fd108f700 1 Processor -- start 2021-02-10T11:21:08.809 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:08.813 7f5fd108f700 1 -- start start 2021-02-10T11:21:08.811 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:08.813 7f5fd108f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5fcc0ae2c0 0x7f5fcc1906b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:08.811 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:08.813 7f5fd108f700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5fcc0f22e0 0x7f5fcc0a96a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:08.811 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:08.813 7f5fd108f700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5fcc0f26e0 0x7f5fcc0a9cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:08.811 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:08.813 7f5fd108f700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fcc0f1d60 con 0x7f5fcc0ae2c0 2021-02-10T11:21:08.811 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:08.813 7f5fd108f700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fcc190cd0 con 0x7f5fcc0f26e0 2021-02-10T11:21:08.812 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:08.813 7f5fd108f700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fcc0f0f00 con 0x7f5fcc0f22e0 2021-02-10T11:21:08.812 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:08.813 7f5fcad9d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5fcc0ae2c0 0x7f5fcc1906b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:08.812 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:08.813 7f5fcad9d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5fcc0ae2c0 0x7f5fcc1906b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:54196/0 (socket says 172.21.2.102:54196) 2021-02-10T11:21:08.812 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:08.813 7f5fcad9d700 1 -- 172.21.2.102:0/4285304908 learned_addr learned my addr 172.21.2.102:0/4285304908 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:21:08.812 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:08.813 7f5fcb59e700 1 --2- 172.21.2.102:0/4285304908 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5fcc0f26e0 0x7f5fcc0a9cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:08.812 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:08.813 7f5fca59c700 1 --2- 172.21.2.102:0/4285304908 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5fcc0f22e0 0x7f5fcc0a96a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:08.812 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:08.813 7f5fcad9d700 1 -- 172.21.2.102:0/4285304908 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5fcc0f22e0 msgr2=0x7f5fcc0a96a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:08.812 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:08.813 7f5fcad9d700 1 --2- 172.21.2.102:0/4285304908 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5fcc0f22e0 0x7f5fcc0a96a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:08.812 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:08.813 7f5fcad9d700 1 -- 172.21.2.102:0/4285304908 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5fcc0f26e0 msgr2=0x7f5fcc0a9cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:08.812 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:08.813 7f5fcad9d700 1 --2- 172.21.2.102:0/4285304908 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5fcc0f26e0 0x7f5fcc0a9cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:08.812 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:08.813 7f5fcad9d700 1 -- 172.21.2.102:0/4285304908 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fcc0f1120 con 0x7f5fcc0ae2c0 2021-02-10T11:21:08.812 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:08.813 7f5fca59c700 1 --2- 172.21.2.102:0/4285304908 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5fcc0f22e0 0x7f5fcc0a96a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:21:08.813 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:08.813 7f5fcb59e700 1 --2- 172.21.2.102:0/4285304908 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5fcc0f26e0 0x7f5fcc0a9cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:21:08.813 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:08.813 7f5fcad9d700 1 --2- 172.21.2.102:0/4285304908 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5fcc0ae2c0 0x7f5fcc1906b0 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 rx=0x7f5fb400b5c0 tx=0x7f5fb400b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:08.813 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:08.814 7f5fc37fe700 1 -- 172.21.2.102:0/4285304908 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5fb400bd50 con 0x7f5fcc0ae2c0 2021-02-10T11:21:08.813 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:08.814 7f5fc37fe700 1 -- 172.21.2.102:0/4285304908 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fb4006280 con 0x7f5fcc0ae2c0 2021-02-10T11:21:08.813 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:08.814 7f5fd108f700 1 -- 172.21.2.102:0/4285304908 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fcc0f1120 con 0x7f5fcc0ae2c0 2021-02-10T11:21:08.813 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:08.814 7f5fc37fe700 1 -- 172.21.2.102:0/4285304908 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5fb4004620 con 0x7f5fcc0ae2c0 2021-02-10T11:21:08.813 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:08.814 7f5fd108f700 1 -- 172.21.2.102:0/4285304908 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fcc0f16b0 con 0x7f5fcc0ae2c0 2021-02-10T11:21:08.813 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:08.814 7f5fc37fe700 1 -- 172.21.2.102:0/4285304908 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f5fb40043a0 con 0x7f5fcc0ae2c0 2021-02-10T11:21:08.813 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:08.814 7f5fc37fe700 1 --2- 172.21.2.102:0/4285304908 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5fb00186c0 0x7f5fb001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:08.813 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:08.814 7f5fc37fe700 1 -- 172.21.2.102:0/4285304908 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f5fb40323e0 con 0x7f5fcc0ae2c0 2021-02-10T11:21:08.813 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:08.814 7f5fca59c700 1 --2- 172.21.2.102:0/4285304908 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5fb00186c0 0x7f5fb001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:08.813 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:08.814 7f5fca59c700 1 --2- 172.21.2.102:0/4285304908 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5fb00186c0 0x7f5fb001abc0 crc :-1 s=READY pgs=423 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:08.840 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:08.845 7f5fd108f700 1 -- 172.21.2.102:0/4285304908 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fcc0edd40 con 0x7f5fcc0ae2c0 2021-02-10T11:21:08.843 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:08.848 7f5fc37fe700 1 -- 172.21.2.102:0/4285304908 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f5fb4030d10 con 0x7f5fcc0ae2c0 2021-02-10T11:21:08.969 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:08.974 7f5fd108f700 1 -- 172.21.2.102:0/4285304908 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344400", "target": ["mgr", ""]}) v1 -- 0x7f5fcc0f16b0 con 0x7f5fb00186c0 2021-02-10T11:21:08.981 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:08.987 7f5fc37fe700 1 -- 172.21.2.102:0/4285304908 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f5fbc0068e0 con 0x7f5fb00186c0 2021-02-10T11:21:08.987 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:08.991 7f5fd108f700 1 -- 172.21.2.102:0/4285304908 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5fb00186c0 msgr2=0x7f5fb001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:08.987 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:08.992 7f5fd108f700 1 --2- 172.21.2.102:0/4285304908 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5fb00186c0 0x7f5fb001abc0 crc :-1 s=READY pgs=423 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:08.987 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:08.992 7f5fd108f700 1 -- 172.21.2.102:0/4285304908 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5fcc0ae2c0 msgr2=0x7f5fcc1906b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:08.987 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:08.992 7f5fd108f700 1 --2- 172.21.2.102:0/4285304908 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5fcc0ae2c0 0x7f5fcc1906b0 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 rx=0x7f5fb400b5c0 tx=0x7f5fb400b960).stop 2021-02-10T11:21:08.987 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:08.992 7f5fd108f700 1 -- 172.21.2.102:0/4285304908 shutdown_connections 2021-02-10T11:21:08.987 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:08.992 7f5fd108f700 1 --2- 172.21.2.102:0/4285304908 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5fb00186c0 0x7f5fb001abc0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:08.987 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:08.992 7f5fd108f700 1 --2- 172.21.2.102:0/4285304908 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5fcc0f26e0 0x7f5fcc0a9cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:08.987 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:08.992 7f5fd108f700 1 --2- 172.21.2.102:0/4285304908 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5fcc0f22e0 0x7f5fcc0a96a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:08.988 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:08.992 7f5fd108f700 1 --2- 172.21.2.102:0/4285304908 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5fcc0ae2c0 0x7f5fcc1906b0 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:08.988 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:08.992 7f5fd108f700 1 -- 172.21.2.102:0/4285304908 >> 172.21.2.102:0/4285304908 conn(0x7f5fcc0c40d0 msgr2=0x7f5fcc0a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:08.988 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:08.992 7f5fd108f700 1 -- 172.21.2.102:0/4285304908 shutdown_connections 2021-02-10T11:21:08.988 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:08.992 7f5fd108f700 1 -- 172.21.2.102:0/4285304908 wait complete. 2021-02-10T11:21:08.996 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000344401 2021-02-10T11:21:09.426 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:09.414 7f1fb9e81700 1 Processor -- start 2021-02-10T11:21:09.426 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:09.414 7f1fb9e81700 1 -- start start 2021-02-10T11:21:09.426 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:09.415 7f1fb9e81700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1fb41b0870 0x7f1fb40f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:09.426 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:09.415 7f1fb9e81700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f1fb40efe00 0x7f1fb40f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:09.426 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:09.415 7f1fb9e81700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fb41b0c70 con 0x7f1fb41b0470 2021-02-10T11:21:09.427 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:09.415 7f1fb9e81700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1fb40f0b50 con 0x7f1fb40efe00 2021-02-10T11:21:09.427 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:09.415 7f1fb9e81700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fb40f0820 con 0x7f1fb41b0870 2021-02-10T11:21:09.427 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:09.415 7f1fb37fe700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f1fb41b0470 0x7f1fb40ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:46428/0 (socket says 172.21.2.102:46428) 2021-02-10T11:21:09.427 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:09.415 7f1fb37fe700 1 -- 172.21.2.102:0/537534710 learned_addr learned my addr 172.21.2.102:0/537534710 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:21:09.427 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:09.415 7f1fb2ffd700 1 --2- 172.21.2.102:0/537534710 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1fb41b0870 0x7f1fb40f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:09.427 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:09.415 7f1fb3fff700 1 --2- 172.21.2.102:0/537534710 >> v2:172.21.2.113:3300/0 conn(0x7f1fb40efe00 0x7f1fb40f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:09.427 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:09.416 7f1fb2ffd700 1 -- 172.21.2.102:0/537534710 >> v2:172.21.2.113:3300/0 conn(0x7f1fb40efe00 msgr2=0x7f1fb40f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:21:09.427 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:09.416 7f1fb2ffd700 1 --2- 172.21.2.102:0/537534710 >> v2:172.21.2.113:3300/0 conn(0x7f1fb40efe00 0x7f1fb40f0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:09.427 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:09.416 7f1fb2ffd700 1 -- 172.21.2.102:0/537534710 >> v1:172.21.2.102:6789/0 conn(0x7f1fb41b0470 legacy=0x7f1fb40ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:09.427 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:09.416 7f1fb2ffd700 1 -- 172.21.2.102:0/537534710 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fb40e52e0 con 0x7f1fb41b0870 2021-02-10T11:21:09.427 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:09.416 7f1fb27fc700 1 -- 172.21.2.102:0/537534710 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4056809148 0 0) 0x7f1fa4001500 con 0x7f1fb41b0470 2021-02-10T11:21:09.427 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:09.416 7f1fb2ffd700 1 --2- 172.21.2.102:0/537534710 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1fb41b0870 0x7f1fb40f20e0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7f1f9c006650 tx=0x7f1f9c009a00).ready entity=mon.2 client_cookie=905efd2ed7069c38 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:09.427 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:09.416 7f1fb27fc700 1 -- 172.21.2.102:0/537534710 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1f9c009df0 con 0x7f1fb41b0870 2021-02-10T11:21:09.428 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:09.416 7f1fb27fc700 1 -- 172.21.2.102:0/537534710 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f9c0009b0 con 0x7f1fb41b0870 2021-02-10T11:21:09.428 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:09.416 7f1fb27fc700 1 -- 172.21.2.102:0/537534710 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1f9c00c2b0 con 0x7f1fb41b0870 2021-02-10T11:21:09.428 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:09.416 7f1fb9e81700 1 -- 172.21.2.102:0/537534710 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1fb41b0870 msgr2=0x7f1fb40f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:09.428 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:09.416 7f1fb9e81700 1 --2- 172.21.2.102:0/537534710 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1fb41b0870 0x7f1fb40f20e0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7f1f9c006650 tx=0x7f1f9c009a00).stop 2021-02-10T11:21:09.428 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:09.417 7f1fb9e81700 1 -- 172.21.2.102:0/537534710 shutdown_connections 2021-02-10T11:21:09.428 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:09.417 7f1fb9e81700 1 --2- 172.21.2.102:0/537534710 >> v2:172.21.2.113:3300/0 conn(0x7f1fb40efe00 0x7f1fb40f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:09.428 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:09.417 7f1fb9e81700 1 --2- 172.21.2.102:0/537534710 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1fb41b0870 0x7f1fb40f20e0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:09.428 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:09.417 7f1fb9e81700 1 -- 172.21.2.102:0/537534710 >> 172.21.2.102:0/537534710 conn(0x7f1fb40ce3e0 msgr2=0x7f1fb40e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:09.428 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:09.417 7f1fb9e81700 1 -- 172.21.2.102:0/537534710 shutdown_connections 2021-02-10T11:21:09.428 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:09.417 7f1fb9e81700 1 -- 172.21.2.102:0/537534710 wait complete. 2021-02-10T11:21:09.430 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:09.431 7f1fb9e81700 1 Processor -- start 2021-02-10T11:21:09.430 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:09.431 7f1fb9e81700 1 -- start start 2021-02-10T11:21:09.430 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:09.432 7f1fb9e81700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1fb40a0420 0x7f1fb40a0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:09.430 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:09.432 7f1fb9e81700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1fb4189d00 0x7f1fb418a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:09.430 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:09.432 7f1fb9e81700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1fb418a720 0x7f1fb40b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:09.430 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:09.432 7f1fb9e81700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fb4189780 con 0x7f1fb418a720 2021-02-10T11:21:09.430 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:09.432 7f1fb9e81700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fb418ab20 con 0x7f1fb40a0420 2021-02-10T11:21:09.430 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:09.432 7f1fb9e81700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fb40a1140 con 0x7f1fb4189d00 2021-02-10T11:21:09.430 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:09.432 7f1fb3fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1fb418a720 0x7f1fb40b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:09.431 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:09.432 7f1fb3fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1fb418a720 0x7f1fb40b7be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:54212/0 (socket says 172.21.2.102:54212) 2021-02-10T11:21:09.431 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:09.432 7f1fb3fff700 1 -- 172.21.2.102:0/396233354 learned_addr learned my addr 172.21.2.102:0/396233354 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:21:09.431 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:09.432 7f1fb37fe700 1 --2- 172.21.2.102:0/396233354 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1fb40a0420 0x7f1fb40a0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:09.431 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:09.432 7f1fb2ffd700 1 --2- 172.21.2.102:0/396233354 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1fb4189d00 0x7f1fb418a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:09.431 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:09.432 7f1fb3fff700 1 -- 172.21.2.102:0/396233354 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1fb4189d00 msgr2=0x7f1fb418a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:09.431 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:09.432 7f1fb3fff700 1 --2- 172.21.2.102:0/396233354 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1fb4189d00 0x7f1fb418a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:09.431 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:09.432 7f1fb3fff700 1 -- 172.21.2.102:0/396233354 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1fb40a0420 msgr2=0x7f1fb40a0820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:09.431 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:09.432 7f1fb3fff700 1 --2- 172.21.2.102:0/396233354 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1fb40a0420 0x7f1fb40a0820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:09.431 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:09.433 7f1fb3fff700 1 -- 172.21.2.102:0/396233354 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fb40b8200 con 0x7f1fb418a720 2021-02-10T11:21:09.431 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:09.433 7f1fb2ffd700 1 --2- 172.21.2.102:0/396233354 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1fb4189d00 0x7f1fb418a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:21:09.431 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:09.433 7f1fb3fff700 1 --2- 172.21.2.102:0/396233354 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1fb418a720 0x7f1fb40b7be0 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 rx=0x7f1fac00b5e0 tx=0x7f1fac00b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:09.431 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:09.433 7f1fabfff700 1 -- 172.21.2.102:0/396233354 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1fac006370 con 0x7f1fb418a720 2021-02-10T11:21:09.431 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:09.433 7f1fabfff700 1 -- 172.21.2.102:0/396233354 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fac00bd70 con 0x7f1fb418a720 2021-02-10T11:21:09.432 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:09.433 7f1fb9e81700 1 -- 172.21.2.102:0/396233354 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fb40b8200 con 0x7f1fb418a720 2021-02-10T11:21:09.432 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:09.433 7f1fb9e81700 1 -- 172.21.2.102:0/396233354 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fb40b8200 con 0x7f1fb418a720 2021-02-10T11:21:09.432 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:09.433 7f1fabfff700 1 -- 172.21.2.102:0/396233354 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1fac002560 con 0x7f1fb418a720 2021-02-10T11:21:09.432 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:09.433 7f1fabfff700 1 -- 172.21.2.102:0/396233354 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1fac0022e0 con 0x7f1fb418a720 2021-02-10T11:21:09.432 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:09.433 7f1fabfff700 1 --2- 172.21.2.102:0/396233354 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1f98018730 0x7f1f9801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:09.432 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:09.434 7f1fabfff700 1 -- 172.21.2.102:0/396233354 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f1fac0327b0 con 0x7f1fb418a720 2021-02-10T11:21:09.432 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:09.434 7f1fb37fe700 1 --2- 172.21.2.102:0/396233354 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1f98018730 0x7f1f9801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:09.432 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:09.434 7f1fb37fe700 1 --2- 172.21.2.102:0/396233354 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1f98018730 0x7f1f9801ac10 crc :-1 s=READY pgs=424 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:09.459 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:09.464 7f1fb9e81700 1 -- 172.21.2.102:0/396233354 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fb4105ae0 con 0x7f1fb418a720 2021-02-10T11:21:09.462 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:09.467 7f1fabfff700 1 -- 172.21.2.102:0/396233354 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f1fac0020d0 con 0x7f1fb418a720 2021-02-10T11:21:09.588 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:09.593 7f1fb9e81700 1 -- 172.21.2.102:0/396233354 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344401", "target": ["mgr", ""]}) v1 -- 0x7f1fb4189780 con 0x7f1f98018730 2021-02-10T11:21:09.600 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:09.605 7f1fabfff700 1 -- 172.21.2.102:0/396233354 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f1fa4007dc0 con 0x7f1f98018730 2021-02-10T11:21:09.605 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:09.610 7f1fb9e81700 1 -- 172.21.2.102:0/396233354 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1f98018730 msgr2=0x7f1f9801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:09.605 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:09.610 7f1fb9e81700 1 --2- 172.21.2.102:0/396233354 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1f98018730 0x7f1f9801ac10 crc :-1 s=READY pgs=424 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:09.605 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:09.610 7f1fb9e81700 1 -- 172.21.2.102:0/396233354 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1fb418a720 msgr2=0x7f1fb40b7be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:09.605 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:09.610 7f1fb9e81700 1 --2- 172.21.2.102:0/396233354 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1fb418a720 0x7f1fb40b7be0 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 rx=0x7f1fac00b5e0 tx=0x7f1fac00b980).stop 2021-02-10T11:21:09.605 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:09.610 7f1fb9e81700 1 -- 172.21.2.102:0/396233354 shutdown_connections 2021-02-10T11:21:09.606 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:09.610 7f1fb9e81700 1 --2- 172.21.2.102:0/396233354 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1fb418a720 0x7f1fb40b7be0 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:09.606 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:09.610 7f1fb9e81700 1 --2- 172.21.2.102:0/396233354 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1fb4189d00 0x7f1fb418a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:09.606 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:09.610 7f1fb9e81700 1 --2- 172.21.2.102:0/396233354 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1f98018730 0x7f1f9801ac10 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:09.606 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:09.610 7f1fb9e81700 1 --2- 172.21.2.102:0/396233354 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1fb40a0420 0x7f1fb40a0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:09.606 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:09.610 7f1fb9e81700 1 -- 172.21.2.102:0/396233354 >> 172.21.2.102:0/396233354 conn(0x7f1fb4094e10 msgr2=0x7f1fb40a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:09.606 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:09.610 7f1fb9e81700 1 -- 172.21.2.102:0/396233354 shutdown_connections 2021-02-10T11:21:09.606 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:09.610 7f1fb9e81700 1 -- 172.21.2.102:0/396233354 wait complete. 2021-02-10T11:21:09.614 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000344402 2021-02-10T11:21:10.047 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:10.035 7fbaaf29e700 1 Processor -- start 2021-02-10T11:21:10.048 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:10.035 7fbaaf29e700 1 -- start start 2021-02-10T11:21:10.048 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:10.035 7fbaaf29e700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbaa81b0870 0x7fbaa80f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:10.048 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:10.036 7fbaaf29e700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbaa81b0c70 con 0x7fbaa81b0470 2021-02-10T11:21:10.048 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:10.036 7fbaaf29e700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbaa80f0b50 con 0x7fbaa80efe00 2021-02-10T11:21:10.048 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:10.036 7fbaaf29e700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbaa80e5ea0 con 0x7fbaa81b0870 2021-02-10T11:21:10.048 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:10.036 7fbaad0ba700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fbaa81b0470 0x7fbaa80ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:46442/0 (socket says 172.21.2.102:46442) 2021-02-10T11:21:10.048 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:10.036 7fbaad0ba700 1 -- 172.21.2.102:0/2410116486 learned_addr learned my addr 172.21.2.102:0/2410116486 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:21:10.048 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:10.036 7fbaac8b9700 1 --2- 172.21.2.102:0/2410116486 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbaa81b0870 0x7fbaa80f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:10.048 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:10.036 7fba9ffff700 1 -- 172.21.2.102:0/2410116486 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3255092825 0 0) 0x7fba980010b0 con 0x7fbaa81b0470 2021-02-10T11:21:10.048 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:10.036 7fba9ffff700 1 -- 172.21.2.102:0/2410116486 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba94002fe0 con 0x7fbaa81b0470 2021-02-10T11:21:10.048 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:10.036 7fba9ffff700 1 -- 172.21.2.102:0/2410116486 <== mon.1 v1:172.21.2.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 469614241 0 0) 0x7fbaa0001080 con 0x7fbaa80efe00 2021-02-10T11:21:10.049 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:10.036 7fba9ffff700 1 -- 172.21.2.102:0/2410116486 --> v1:172.21.2.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba94003210 con 0x7fbaa80efe00 2021-02-10T11:21:10.049 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:10.037 7fba9ffff700 1 -- 172.21.2.102:0/2410116486 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 116410533 0 0) 0x7fba980010b0 con 0x7fbaa81b0470 2021-02-10T11:21:10.049 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:10.037 7fbaac8b9700 1 -- 172.21.2.102:0/2410116486 >> v1:172.21.2.113:6789/0 conn(0x7fbaa80efe00 legacy=0x7fbaa80f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:10.049 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:10.037 7fbaac8b9700 1 -- 172.21.2.102:0/2410116486 >> v1:172.21.2.102:6789/0 conn(0x7fbaa81b0470 legacy=0x7fbaa80ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:10.049 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:10.037 7fbaac8b9700 1 -- 172.21.2.102:0/2410116486 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbaa80e52e0 con 0x7fbaa81b0870 2021-02-10T11:21:10.049 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:10.037 7fbaac8b9700 1 --2- 172.21.2.102:0/2410116486 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbaa81b0870 0x7fbaa80f20e0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7fbaa40069d0 tx=0x7fbaa4009e40).ready entity=mon.2 client_cookie=4a4075eef2fcb6d9 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:10.049 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:10.037 7fba9ffff700 1 -- 172.21.2.102:0/2410116486 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbaa400b510 con 0x7fbaa81b0870 2021-02-10T11:21:10.049 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:10.037 7fba9ffff700 1 -- 172.21.2.102:0/2410116486 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbaa400b890 con 0x7fbaa81b0870 2021-02-10T11:21:10.049 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:10.037 7fba9ffff700 1 -- 172.21.2.102:0/2410116486 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbaa400bcf0 con 0x7fbaa81b0870 2021-02-10T11:21:10.049 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:10.037 7fbaaf29e700 1 -- 172.21.2.102:0/2410116486 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbaa81b0870 msgr2=0x7fbaa80f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:10.049 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:10.037 7fbaaf29e700 1 --2- 172.21.2.102:0/2410116486 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbaa81b0870 0x7fbaa80f20e0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7fbaa40069d0 tx=0x7fbaa4009e40).stop 2021-02-10T11:21:10.049 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:10.038 7fbaaf29e700 1 -- 172.21.2.102:0/2410116486 shutdown_connections 2021-02-10T11:21:10.049 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:10.038 7fbaaf29e700 1 --2- 172.21.2.102:0/2410116486 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbaa81b0870 0x7fbaa80f20e0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:10.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:10.038 7fbaaf29e700 1 -- 172.21.2.102:0/2410116486 >> 172.21.2.102:0/2410116486 conn(0x7fbaa80ce3e0 msgr2=0x7fbaa80e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:10.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:10.038 7fbaaf29e700 1 -- 172.21.2.102:0/2410116486 shutdown_connections 2021-02-10T11:21:10.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:10.038 7fbaaf29e700 1 -- 172.21.2.102:0/2410116486 wait complete. 2021-02-10T11:21:10.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:10.053 7fbaaf29e700 1 Processor -- start 2021-02-10T11:21:10.052 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:10.053 7fbaaf29e700 1 -- start start 2021-02-10T11:21:10.052 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:10.053 7fbaaf29e700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbaa81a48b0 0x7fbaa80bf930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:10.052 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:10.053 7fbaaf29e700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fbaa80a73c0 0x7fbaa80bff50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:10.052 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:10.053 7fbaaf29e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbaa80a5260 0x7fbaa80a5660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:10.052 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:10.053 7fbaaf29e700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbaa80a6ec0 con 0x7fbaa80a5260 2021-02-10T11:21:10.052 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:10.053 7fbaaf29e700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbaa80bc010 con 0x7fbaa80a73c0 2021-02-10T11:21:10.052 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:10.053 7fbaaf29e700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbaa80c0570 con 0x7fbaa81a48b0 2021-02-10T11:21:10.052 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:10.053 7fbaad8bb700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbaa80a5260 0x7fbaa80a5660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:10.052 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:10.054 7fbaad8bb700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbaa80a5260 0x7fbaa80a5660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:54226/0 (socket says 172.21.2.102:54226) 2021-02-10T11:21:10.052 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:10.054 7fbaad8bb700 1 -- 172.21.2.102:0/1379444787 learned_addr learned my addr 172.21.2.102:0/1379444787 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:21:10.052 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:10.054 7fbaac8b9700 1 --2- 172.21.2.102:0/1379444787 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fbaa80a73c0 0x7fbaa80bff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:10.052 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:10.054 7fbaad0ba700 1 --2- 172.21.2.102:0/1379444787 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbaa81a48b0 0x7fbaa80bf930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:10.052 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:10.054 7fbaad8bb700 1 -- 172.21.2.102:0/1379444787 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbaa81a48b0 msgr2=0x7fbaa80bf930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:10.053 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:10.054 7fbaad8bb700 1 --2- 172.21.2.102:0/1379444787 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbaa81a48b0 0x7fbaa80bf930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:10.053 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:10.054 7fbaad8bb700 1 -- 172.21.2.102:0/1379444787 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fbaa80a73c0 msgr2=0x7fbaa80bff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:10.053 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:10.054 7fbaad8bb700 1 --2- 172.21.2.102:0/1379444787 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fbaa80a73c0 0x7fbaa80bff50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:10.053 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:10.054 7fbaad8bb700 1 -- 172.21.2.102:0/1379444787 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbaa80a5c80 con 0x7fbaa80a5260 2021-02-10T11:21:10.053 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:10.054 7fbaad8bb700 1 --2- 172.21.2.102:0/1379444787 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbaa80a5260 0x7fbaa80a5660 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 rx=0x7fbaa000b5b0 tx=0x7fbaa000b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:10.053 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:10.054 7fba9dffb700 1 -- 172.21.2.102:0/1379444787 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbaa0000bc0 con 0x7fbaa80a5260 2021-02-10T11:21:10.053 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:10.054 7fba9dffb700 1 -- 172.21.2.102:0/1379444787 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbaa0006240 con 0x7fbaa80a5260 2021-02-10T11:21:10.053 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:10.054 7fbaaf29e700 1 -- 172.21.2.102:0/1379444787 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbaa80a5c80 con 0x7fbaa80a5260 2021-02-10T11:21:10.053 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:10.054 7fba9dffb700 1 -- 172.21.2.102:0/1379444787 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbaa0005720 con 0x7fbaa80a5260 2021-02-10T11:21:10.053 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:10.054 7fbaaf29e700 1 -- 172.21.2.102:0/1379444787 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbaa80a5c80 con 0x7fbaa80a5260 2021-02-10T11:21:10.053 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:10.055 7fba9dffb700 1 -- 172.21.2.102:0/1379444787 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fbaa00054a0 con 0x7fbaa80a5260 2021-02-10T11:21:10.053 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:10.055 7fba9dffb700 1 --2- 172.21.2.102:0/1379444787 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fba88018730 0x7fba8801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:10.053 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:10.055 7fba9dffb700 1 -- 172.21.2.102:0/1379444787 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fbaa0032670 con 0x7fbaa80a5260 2021-02-10T11:21:10.053 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:10.055 7fbaad0ba700 1 --2- 172.21.2.102:0/1379444787 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fba88018730 0x7fba8801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:10.054 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:10.055 7fbaad0ba700 1 --2- 172.21.2.102:0/1379444787 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fba88018730 0x7fba8801ac10 crc :-1 s=READY pgs=425 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:10.080 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:10.085 7fbaaf29e700 1 -- 172.21.2.102:0/1379444787 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbaa8105ae0 con 0x7fbaa80a5260 2021-02-10T11:21:10.082 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:10.087 7fba9dffb700 1 -- 172.21.2.102:0/1379444787 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fbaa0006030 con 0x7fbaa80a5260 2021-02-10T11:21:10.210 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:10.215 7fbaaf29e700 1 -- 172.21.2.102:0/1379444787 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344402", "target": ["mgr", ""]}) v1 -- 0x7fbaa8105ae0 con 0x7fba88018730 2021-02-10T11:21:10.222 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:10.227 7fba9dffb700 1 -- 172.21.2.102:0/1379444787 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fba980014a0 con 0x7fba88018730 2021-02-10T11:21:10.227 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:10.232 7fbaaf29e700 1 -- 172.21.2.102:0/1379444787 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fba88018730 msgr2=0x7fba8801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:10.227 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:10.232 7fbaaf29e700 1 --2- 172.21.2.102:0/1379444787 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fba88018730 0x7fba8801ac10 crc :-1 s=READY pgs=425 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:10.227 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:10.232 7fbaaf29e700 1 -- 172.21.2.102:0/1379444787 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbaa80a5260 msgr2=0x7fbaa80a5660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:10.227 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:10.232 7fbaaf29e700 1 --2- 172.21.2.102:0/1379444787 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbaa80a5260 0x7fbaa80a5660 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 rx=0x7fbaa000b5b0 tx=0x7fbaa000b950).stop 2021-02-10T11:21:10.227 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:10.232 7fbaaf29e700 1 -- 172.21.2.102:0/1379444787 shutdown_connections 2021-02-10T11:21:10.228 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:10.232 7fbaaf29e700 1 --2- 172.21.2.102:0/1379444787 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbaa80a5260 0x7fbaa80a5660 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:10.228 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:10.232 7fbaaf29e700 1 --2- 172.21.2.102:0/1379444787 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fba88018730 0x7fba8801ac10 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:10.228 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:10.232 7fbaaf29e700 1 --2- 172.21.2.102:0/1379444787 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fbaa80a73c0 0x7fbaa80bff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:10.228 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:10.232 7fbaaf29e700 1 --2- 172.21.2.102:0/1379444787 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbaa81a48b0 0x7fbaa80bf930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:10.228 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:10.232 7fbaaf29e700 1 -- 172.21.2.102:0/1379444787 >> 172.21.2.102:0/1379444787 conn(0x7fbaa8109e80 msgr2=0x7fbaa80af060 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:10.228 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:10.232 7fbaaf29e700 1 -- 172.21.2.102:0/1379444787 shutdown_connections 2021-02-10T11:21:10.228 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:10.232 7fbaaf29e700 1 -- 172.21.2.102:0/1379444787 wait complete. 2021-02-10T11:21:10.236 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000344403 2021-02-10T11:21:10.668 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:10.656 7f2092998700 1 Processor -- start 2021-02-10T11:21:10.669 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:10.657 7f2092998700 1 -- start start 2021-02-10T11:21:10.669 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:10.657 7f2092998700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f208c0f05f0 0x7f208c0f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:10.669 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:10.657 7f2092998700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f208c1b0c70 con 0x7f208c1b0870 2021-02-10T11:21:10.669 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:10.657 7f2092998700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f208c0f0b50 con 0x7f208c1b0470 2021-02-10T11:21:10.669 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:10.657 7f2092998700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f208c0e5ea0 con 0x7f208c0f05f0 2021-02-10T11:21:10.669 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:10.657 7f208b7fe700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f208c1b0870 0x7f208c0efe00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:46456/0 (socket says 172.21.2.102:46456) 2021-02-10T11:21:10.669 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:10.657 7f208b7fe700 1 -- 172.21.2.102:0/4168796087 learned_addr learned my addr 172.21.2.102:0/4168796087 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:21:10.669 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:10.657 7f2090fb5700 1 --2- 172.21.2.102:0/4168796087 >> v2:172.21.2.102:3300/0 conn(0x7f208c0f05f0 0x7f208c0f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:10.669 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:10.658 7f2090fb5700 1 -- 172.21.2.102:0/4168796087 >> v1:172.21.2.113:6789/0 conn(0x7f208c1b0470 legacy=0x7f208c0ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:10.669 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:10.658 7f208affd700 1 -- 172.21.2.102:0/4168796087 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3022521989 0 0) 0x7f2074000ce0 con 0x7f208c1b0870 2021-02-10T11:21:10.669 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:10.658 7f2090fb5700 1 -- 172.21.2.102:0/4168796087 >> v1:172.21.2.102:6789/0 conn(0x7f208c1b0870 legacy=0x7f208c0efe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:10.670 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:10.658 7f2090fb5700 1 -- 172.21.2.102:0/4168796087 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f208c0e52e0 con 0x7f208c0f05f0 2021-02-10T11:21:10.670 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:10.658 7f2090fb5700 1 --2- 172.21.2.102:0/4168796087 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f208c0f05f0 0x7f208c0f20e0 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 rx=0x7f2084006ab0 tx=0x7f2084009b40).ready entity=mon.0 client_cookie=8b17afc953cf46f7 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:10.670 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:10.658 7f208affd700 1 -- 172.21.2.102:0/4168796087 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f20840022e0 con 0x7f208c0f05f0 2021-02-10T11:21:10.670 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:10.658 7f208affd700 1 -- 172.21.2.102:0/4168796087 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2084002660 con 0x7f208c0f05f0 2021-02-10T11:21:10.670 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:10.658 7f208affd700 1 -- 172.21.2.102:0/4168796087 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2084002a70 con 0x7f208c0f05f0 2021-02-10T11:21:10.670 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:10.658 7f2092998700 1 -- 172.21.2.102:0/4168796087 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f208c0f05f0 msgr2=0x7f208c0f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:10.670 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:10.658 7f2092998700 1 --2- 172.21.2.102:0/4168796087 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f208c0f05f0 0x7f208c0f20e0 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 rx=0x7f2084006ab0 tx=0x7f2084009b40).stop 2021-02-10T11:21:10.670 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:10.659 7f2092998700 1 -- 172.21.2.102:0/4168796087 shutdown_connections 2021-02-10T11:21:10.670 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:10.659 7f2092998700 1 --2- 172.21.2.102:0/4168796087 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f208c0f05f0 0x7f208c0f20e0 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:10.670 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:10.659 7f2092998700 1 -- 172.21.2.102:0/4168796087 >> 172.21.2.102:0/4168796087 conn(0x7f208c0ce3e0 msgr2=0x7f208c0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:10.670 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:10.659 7f2092998700 1 -- 172.21.2.102:0/4168796087 shutdown_connections 2021-02-10T11:21:10.670 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:10.659 7f2092998700 1 -- 172.21.2.102:0/4168796087 wait complete. 2021-02-10T11:21:10.672 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:10.674 7f2092998700 1 Processor -- start 2021-02-10T11:21:10.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:10.674 7f2092998700 1 -- start start 2021-02-10T11:21:10.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:10.674 7f2092998700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f208c0a1dd0 0x7f208c190920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:10.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:10.674 7f2092998700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f208c08fd30 0x7f208c0effc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:10.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:10.674 7f2092998700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f208c0f0e90 0x7f208c0f1290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:10.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:10.674 7f2092998700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f208c0f06f0 con 0x7f208c08fd30 2021-02-10T11:21:10.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:10.674 7f2092998700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f208c090160 con 0x7f208c0a1dd0 2021-02-10T11:21:10.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:10.674 7f2092998700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f208c0f18b0 con 0x7f208c0f0e90 2021-02-10T11:21:10.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:10.675 7f208b7fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f208c08fd30 0x7f208c0effc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:10.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:10.675 7f208b7fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f208c08fd30 0x7f208c0effc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:54238/0 (socket says 172.21.2.102:54238) 2021-02-10T11:21:10.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:10.675 7f208b7fe700 1 -- 172.21.2.102:0/3136630680 learned_addr learned my addr 172.21.2.102:0/3136630680 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:21:10.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:10.675 7f2090fb5700 1 --2- 172.21.2.102:0/3136630680 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f208c0f0e90 0x7f208c0f1290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:10.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:10.675 7f208bfff700 1 --2- 172.21.2.102:0/3136630680 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f208c0a1dd0 0x7f208c190920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:10.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:10.675 7f208b7fe700 1 -- 172.21.2.102:0/3136630680 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f208c0f0e90 msgr2=0x7f208c0f1290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:10.674 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:10.675 7f208b7fe700 1 --2- 172.21.2.102:0/3136630680 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f208c0f0e90 0x7f208c0f1290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:10.674 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:10.675 7f208b7fe700 1 -- 172.21.2.102:0/3136630680 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f208c0a1dd0 msgr2=0x7f208c190920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:10.674 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:10.675 7f208b7fe700 1 --2- 172.21.2.102:0/3136630680 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f208c0a1dd0 0x7f208c190920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:10.674 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:10.675 7f208b7fe700 1 -- 172.21.2.102:0/3136630680 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f208c0f1ad0 con 0x7f208c08fd30 2021-02-10T11:21:10.674 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:10.675 7f208b7fe700 1 --2- 172.21.2.102:0/3136630680 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f208c08fd30 0x7f208c0effc0 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 rx=0x7f207400b580 tx=0x7f207400b920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:10.674 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:10.675 7f2088ff9700 1 -- 172.21.2.102:0/3136630680 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2074006020 con 0x7f208c08fd30 2021-02-10T11:21:10.674 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:10.675 7f2088ff9700 1 -- 172.21.2.102:0/3136630680 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2074006360 con 0x7f208c08fd30 2021-02-10T11:21:10.674 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:10.675 7f2092998700 1 -- 172.21.2.102:0/3136630680 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f208c0f1ad0 con 0x7f208c08fd30 2021-02-10T11:21:10.674 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:10.675 7f2088ff9700 1 -- 172.21.2.102:0/3136630680 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2074004620 con 0x7f208c08fd30 2021-02-10T11:21:10.674 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:10.675 7f2092998700 1 -- 172.21.2.102:0/3136630680 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f208c0f1ad0 con 0x7f208c08fd30 2021-02-10T11:21:10.674 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:10.676 7f2088ff9700 1 -- 172.21.2.102:0/3136630680 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f2074009fe0 con 0x7f208c08fd30 2021-02-10T11:21:10.674 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:10.676 7f2088ff9700 1 --2- 172.21.2.102:0/3136630680 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2070018730 0x7f207001abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:10.674 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:10.676 7f2088ff9700 1 -- 172.21.2.102:0/3136630680 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f20740327c0 con 0x7f208c08fd30 2021-02-10T11:21:10.675 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:10.676 7f208bfff700 1 --2- 172.21.2.102:0/3136630680 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2070018730 0x7f207001abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:10.675 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:10.676 7f208bfff700 1 --2- 172.21.2.102:0/3136630680 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2070018730 0x7f207001abf0 crc :-1 s=READY pgs=426 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:10.701 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:10.706 7f2092998700 1 -- 172.21.2.102:0/3136630680 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f208c0e52e0 con 0x7f208c08fd30 2021-02-10T11:21:10.703 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:10.708 7f2088ff9700 1 -- 172.21.2.102:0/3136630680 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f2074006ba0 con 0x7f208c08fd30 2021-02-10T11:21:10.835 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:10.840 7f2092998700 1 -- 172.21.2.102:0/3136630680 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344403", "target": ["mgr", ""]}) v1 -- 0x7f208c0e52e0 con 0x7f2070018730 2021-02-10T11:21:10.846 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:10.851 7f2088ff9700 1 -- 172.21.2.102:0/3136630680 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f207c0091e0 con 0x7f2070018730 2021-02-10T11:21:10.852 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:10.856 7f2092998700 1 -- 172.21.2.102:0/3136630680 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2070018730 msgr2=0x7f207001abf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:10.852 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:10.856 7f2092998700 1 --2- 172.21.2.102:0/3136630680 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2070018730 0x7f207001abf0 crc :-1 s=READY pgs=426 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:10.852 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:10.856 7f2092998700 1 -- 172.21.2.102:0/3136630680 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f208c08fd30 msgr2=0x7f208c0effc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:10.852 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:10.856 7f2092998700 1 --2- 172.21.2.102:0/3136630680 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f208c08fd30 0x7f208c0effc0 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 rx=0x7f207400b580 tx=0x7f207400b920).stop 2021-02-10T11:21:10.852 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:10.857 7f2092998700 1 -- 172.21.2.102:0/3136630680 shutdown_connections 2021-02-10T11:21:10.852 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:10.857 7f2092998700 1 --2- 172.21.2.102:0/3136630680 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f208c0f0e90 0x7f208c0f1290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:10.852 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:10.857 7f2092998700 1 --2- 172.21.2.102:0/3136630680 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f208c08fd30 0x7f208c0effc0 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:10.852 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:10.857 7f2092998700 1 --2- 172.21.2.102:0/3136630680 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2070018730 0x7f207001abf0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:10.853 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:10.857 7f2092998700 1 --2- 172.21.2.102:0/3136630680 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f208c0a1dd0 0x7f208c190920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:10.853 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:10.857 7f2092998700 1 -- 172.21.2.102:0/3136630680 >> 172.21.2.102:0/3136630680 conn(0x7f208c0958b0 msgr2=0x7f208c0bf040 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:10.853 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:10.857 7f2092998700 1 -- 172.21.2.102:0/3136630680 shutdown_connections 2021-02-10T11:21:10.853 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:10.857 7f2092998700 1 -- 172.21.2.102:0/3136630680 wait complete. 2021-02-10T11:21:10.860 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000344404 2021-02-10T11:21:11.290 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:11.279 7fac64289700 1 Processor -- start 2021-02-10T11:21:11.290 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:11.279 7fac64289700 1 -- start start 2021-02-10T11:21:11.291 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:11.279 7fac64289700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fac5c1b0870 0x7fac5c0f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:11.291 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:11.280 7fac64289700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fac5c0efe00 0x7fac5c0f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:11.291 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:11.280 7fac64289700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac5c1b0c70 con 0x7fac5c1b0470 2021-02-10T11:21:11.291 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:11.280 7fac64289700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac5c0e5ea0 con 0x7fac5c1b0870 2021-02-10T11:21:11.291 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:11.280 7fac64289700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac5c0f0b50 con 0x7fac5c0efe00 2021-02-10T11:21:11.291 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:11.280 7fac618a4700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fac5c1b0870 0x7fac5c0f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:11.291 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:11.280 7fac618a4700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fac5c1b0870 0x7fac5c0f20e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:54244/0 (socket says 172.21.2.102:54244) 2021-02-10T11:21:11.291 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:11.280 7fac618a4700 1 -- 172.21.2.102:0/3096915876 learned_addr learned my addr 172.21.2.102:0/3096915876 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:21:11.291 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:11.280 7fac628a6700 1 --2- 172.21.2.102:0/3096915876 >> v2:172.21.2.113:3300/0 conn(0x7fac5c0efe00 0x7fac5c0f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:11.291 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:11.280 7fac618a4700 1 -- 172.21.2.102:0/3096915876 >> v2:172.21.2.113:3300/0 conn(0x7fac5c0efe00 msgr2=0x7fac5c0f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:21:11.291 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:11.280 7fac618a4700 1 --2- 172.21.2.102:0/3096915876 >> v2:172.21.2.113:3300/0 conn(0x7fac5c0efe00 0x7fac5c0f0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:11.292 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:11.280 7fac618a4700 1 -- 172.21.2.102:0/3096915876 >> v1:172.21.2.113:6789/0 conn(0x7fac5c1b0470 legacy=0x7fac5c0ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:21:11.292 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:11.280 7fac618a4700 1 -- 172.21.2.102:0/3096915876 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac5c0f0820 con 0x7fac5c1b0870 2021-02-10T11:21:11.292 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:11.280 7fac628a6700 1 --2- 172.21.2.102:0/3096915876 >> v2:172.21.2.113:3300/0 conn(0x7fac5c0efe00 0x7fac5c0f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:21:11.292 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:11.280 7fac618a4700 1 --2- 172.21.2.102:0/3096915876 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fac5c1b0870 0x7fac5c0f20e0 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 rx=0x7fac50006700 tx=0x7fac50009b80).ready entity=mon.0 client_cookie=9b8c9fd3ec721a61 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:11.292 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:11.280 7fac610a3700 1 -- 172.21.2.102:0/3096915876 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fac50006340 con 0x7fac5c1b0870 2021-02-10T11:21:11.292 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:11.280 7fac610a3700 1 -- 172.21.2.102:0/3096915876 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac5000c0c0 con 0x7fac5c1b0870 2021-02-10T11:21:11.292 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:11.280 7fac610a3700 1 -- 172.21.2.102:0/3096915876 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fac5000c5a0 con 0x7fac5c1b0870 2021-02-10T11:21:11.292 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:11.280 7fac64289700 1 -- 172.21.2.102:0/3096915876 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fac5c1b0870 msgr2=0x7fac5c0f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:11.292 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:11.280 7fac64289700 1 --2- 172.21.2.102:0/3096915876 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fac5c1b0870 0x7fac5c0f20e0 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 rx=0x7fac50006700 tx=0x7fac50009b80).stop 2021-02-10T11:21:11.292 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:11.281 7fac64289700 1 -- 172.21.2.102:0/3096915876 shutdown_connections 2021-02-10T11:21:11.292 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:11.281 7fac64289700 1 --2- 172.21.2.102:0/3096915876 >> v2:172.21.2.113:3300/0 conn(0x7fac5c0efe00 0x7fac5c0f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:11.292 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:11.281 7fac64289700 1 --2- 172.21.2.102:0/3096915876 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fac5c1b0870 0x7fac5c0f20e0 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:11.292 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:11.281 7fac64289700 1 -- 172.21.2.102:0/3096915876 >> 172.21.2.102:0/3096915876 conn(0x7fac5c0ce3e0 msgr2=0x7fac5c0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:11.292 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:11.281 7fac64289700 1 -- 172.21.2.102:0/3096915876 shutdown_connections 2021-02-10T11:21:11.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:11.281 7fac64289700 1 -- 172.21.2.102:0/3096915876 wait complete. 2021-02-10T11:21:11.294 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:11.296 7fac64289700 1 Processor -- start 2021-02-10T11:21:11.294 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:11.296 7fac64289700 1 -- start start 2021-02-10T11:21:11.295 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:11.296 7fac64289700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fac5c193230 0x7fac5c0a6a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:11.295 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:11.296 7fac64289700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fac5c08f5b0 0x7fac5c08f9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:11.295 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:11.296 7fac64289700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fac5c08e6a0 0x7fac5c08eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:11.295 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:11.296 7fac64289700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac5c08ffd0 con 0x7fac5c193230 2021-02-10T11:21:11.295 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:11.296 7fac64289700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac5c090240 con 0x7fac5c08f5b0 2021-02-10T11:21:11.295 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:11.296 7fac64289700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac5c08f0c0 con 0x7fac5c08e6a0 2021-02-10T11:21:11.295 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:11.296 7fac620a5700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fac5c193230 0x7fac5c0a6a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:11.295 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:11.296 7fac620a5700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fac5c193230 0x7fac5c0a6a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:54254/0 (socket says 172.21.2.102:54254) 2021-02-10T11:21:11.295 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:11.296 7fac620a5700 1 -- 172.21.2.102:0/3785797763 learned_addr learned my addr 172.21.2.102:0/3785797763 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:21:11.295 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:11.296 7fac618a4700 1 --2- 172.21.2.102:0/3785797763 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fac5c08f5b0 0x7fac5c08f9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:11.295 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:11.296 7fac628a6700 1 --2- 172.21.2.102:0/3785797763 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fac5c08e6a0 0x7fac5c08eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:11.295 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:11.297 7fac620a5700 1 -- 172.21.2.102:0/3785797763 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fac5c08e6a0 msgr2=0x7fac5c08eaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:11.295 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:11.297 7fac620a5700 1 --2- 172.21.2.102:0/3785797763 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fac5c08e6a0 0x7fac5c08eaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:11.296 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:11.297 7fac620a5700 1 -- 172.21.2.102:0/3785797763 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fac5c08f5b0 msgr2=0x7fac5c08f9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:11.296 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:11.297 7fac620a5700 1 --2- 172.21.2.102:0/3785797763 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fac5c08f5b0 0x7fac5c08f9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:11.296 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:11.297 7fac620a5700 1 -- 172.21.2.102:0/3785797763 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac5c08f2a0 con 0x7fac5c193230 2021-02-10T11:21:11.296 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:11.297 7fac620a5700 1 --2- 172.21.2.102:0/3785797763 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fac5c193230 0x7fac5c0a6a80 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 rx=0x7fac5800b5b0 tx=0x7fac5800b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:11.296 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:11.297 7fac4effd700 1 -- 172.21.2.102:0/3785797763 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fac5800bd40 con 0x7fac5c193230 2021-02-10T11:21:11.296 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:11.297 7fac4effd700 1 -- 172.21.2.102:0/3785797763 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac58006240 con 0x7fac5c193230 2021-02-10T11:21:11.296 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:11.297 7fac64289700 1 -- 172.21.2.102:0/3785797763 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac5c08f2a0 con 0x7fac5c193230 2021-02-10T11:21:11.296 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:11.297 7fac4effd700 1 -- 172.21.2.102:0/3785797763 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fac58005720 con 0x7fac5c193230 2021-02-10T11:21:11.296 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:11.297 7fac64289700 1 -- 172.21.2.102:0/3785797763 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac5c1924f0 con 0x7fac5c193230 2021-02-10T11:21:11.296 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:11.298 7fac4effd700 1 -- 172.21.2.102:0/3785797763 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fac580054a0 con 0x7fac5c193230 2021-02-10T11:21:11.297 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:11.298 7fac4effd700 1 --2- 172.21.2.102:0/3785797763 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fac44018730 0x7fac4401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:11.297 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:11.298 7fac4effd700 1 -- 172.21.2.102:0/3785797763 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fac58032670 con 0x7fac5c193230 2021-02-10T11:21:11.297 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:11.298 7fac618a4700 1 --2- 172.21.2.102:0/3785797763 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fac44018730 0x7fac4401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:11.297 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:11.298 7fac618a4700 1 --2- 172.21.2.102:0/3785797763 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fac44018730 0x7fac4401ac10 crc :-1 s=READY pgs=427 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:11.323 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:11.328 7fac64289700 1 -- 172.21.2.102:0/3785797763 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac5c0f0b50 con 0x7fac5c193230 2021-02-10T11:21:11.326 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:11.331 7fac4effd700 1 -- 172.21.2.102:0/3785797763 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fac58000c60 con 0x7fac5c193230 2021-02-10T11:21:11.452 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:11.457 7fac64289700 1 -- 172.21.2.102:0/3785797763 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344404", "target": ["mgr", ""]}) v1 -- 0x7fac5c08ffd0 con 0x7fac44018730 2021-02-10T11:21:11.464 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:11.469 7fac4effd700 1 -- 172.21.2.102:0/3785797763 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fac500009b0 con 0x7fac44018730 2021-02-10T11:21:11.469 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:11.474 7fac64289700 1 -- 172.21.2.102:0/3785797763 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fac44018730 msgr2=0x7fac4401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:11.469 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:11.474 7fac64289700 1 --2- 172.21.2.102:0/3785797763 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fac44018730 0x7fac4401ac10 crc :-1 s=READY pgs=427 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:11.470 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:11.474 7fac64289700 1 -- 172.21.2.102:0/3785797763 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fac5c193230 msgr2=0x7fac5c0a6a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:11.470 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:11.474 7fac64289700 1 --2- 172.21.2.102:0/3785797763 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fac5c193230 0x7fac5c0a6a80 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 rx=0x7fac5800b5b0 tx=0x7fac5800b950).stop 2021-02-10T11:21:11.470 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:11.474 7fac64289700 1 -- 172.21.2.102:0/3785797763 shutdown_connections 2021-02-10T11:21:11.470 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:11.474 7fac64289700 1 --2- 172.21.2.102:0/3785797763 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fac5c08e6a0 0x7fac5c08eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:11.470 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:11.474 7fac64289700 1 --2- 172.21.2.102:0/3785797763 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fac44018730 0x7fac4401ac10 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:11.470 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:11.474 7fac64289700 1 --2- 172.21.2.102:0/3785797763 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fac5c08f5b0 0x7fac5c08f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:11.470 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:11.474 7fac64289700 1 --2- 172.21.2.102:0/3785797763 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fac5c193230 0x7fac5c0a6a80 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:11.470 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:11.474 7fac64289700 1 -- 172.21.2.102:0/3785797763 >> 172.21.2.102:0/3785797763 conn(0x7fac5c0bf4e0 msgr2=0x7fac5c0be160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:11.470 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:11.474 7fac64289700 1 -- 172.21.2.102:0/3785797763 shutdown_connections 2021-02-10T11:21:11.470 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:11.474 7fac64289700 1 -- 172.21.2.102:0/3785797763 wait complete. 2021-02-10T11:21:11.479 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000344405 2021-02-10T11:21:12.045 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:11.898 7f3352c3c700 1 Processor -- start 2021-02-10T11:21:12.046 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:11.898 7f3352c3c700 1 -- start start 2021-02-10T11:21:12.046 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:11.898 7f3352c3c700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f334c1b0470 0x7f334c1b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:12.046 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:11.898 7f3352c3c700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f334c0ef610 0x7f334c0efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:12.046 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:11.898 7f3352c3c700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f334c0f0b50 con 0x7f334c0f0030 2021-02-10T11:21:12.046 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:11.898 7f3352c3c700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f334c0e52e0 con 0x7f334c0ef610 2021-02-10T11:21:12.046 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:11.898 7f3352c3c700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f334c0e5490 con 0x7f334c1b0470 2021-02-10T11:21:12.046 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:11.899 7f3351259700 1 --1- >> v1:172.21.2.113:6789/0 conn(0x7f334c0f0030 0x7f334c0f10d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.113:6789/0 says I am v1:172.21.2.102:53808/0 (socket says 172.21.2.102:53808) 2021-02-10T11:21:12.046 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:11.899 7f334bfff700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f334c0ef610 0x7f334c0efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:12.046 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:11.899 7f3351259700 1 -- 172.21.2.102:0/738943359 learned_addr learned my addr 172.21.2.102:0/738943359 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:21:12.046 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:11.899 7f3350a58700 1 --2- 172.21.2.102:0/738943359 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f334c1b0470 0x7f334c1b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:12.046 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:11.899 7f334bfff700 1 -- 172.21.2.102:0/738943359 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f334c1b0470 msgr2=0x7f334c1b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:21:12.046 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:11.899 7f334bfff700 1 --2- 172.21.2.102:0/738943359 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f334c1b0470 0x7f334c1b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:12.047 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:11.899 7f334bfff700 1 -- 172.21.2.102:0/738943359 >> v1:172.21.2.113:6789/0 conn(0x7f334c0f0030 legacy=0x7f334c0f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:12.047 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:11.899 7f334bfff700 1 -- 172.21.2.102:0/738943359 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f334c0f0430 con 0x7f334c0ef610 2021-02-10T11:21:12.047 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:11.900 7f334bfff700 1 --2- 172.21.2.102:0/738943359 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f334c0ef610 0x7f334c0efa10 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 rx=0x7f333c006650 tx=0x7f333c009a00).ready entity=mon.1 client_cookie=320b2881acf4b090 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:12.047 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:11.900 7f334b7fe700 1 -- 172.21.2.102:0/738943359 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f333c009df0 con 0x7f334c0ef610 2021-02-10T11:21:12.047 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:11.900 7f334b7fe700 1 -- 172.21.2.102:0/738943359 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f333c0009b0 con 0x7f334c0ef610 2021-02-10T11:21:12.047 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:11.900 7f334b7fe700 1 -- 172.21.2.102:0/738943359 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f333c00c2b0 con 0x7f334c0ef610 2021-02-10T11:21:12.047 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:11.900 7f3352c3c700 1 -- 172.21.2.102:0/738943359 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f334c0ef610 msgr2=0x7f334c0efa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:12.047 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:11.900 7f3352c3c700 1 --2- 172.21.2.102:0/738943359 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f334c0ef610 0x7f334c0efa10 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 rx=0x7f333c006650 tx=0x7f333c009a00).stop 2021-02-10T11:21:12.047 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:11.901 7f3352c3c700 1 -- 172.21.2.102:0/738943359 shutdown_connections 2021-02-10T11:21:12.047 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:11.901 7f3352c3c700 1 --2- 172.21.2.102:0/738943359 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f334c0ef610 0x7f334c0efa10 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:12.047 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:11.901 7f3352c3c700 1 --2- 172.21.2.102:0/738943359 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f334c1b0470 0x7f334c1b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:12.047 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:11.901 7f3352c3c700 1 -- 172.21.2.102:0/738943359 >> 172.21.2.102:0/738943359 conn(0x7f334c0ce3e0 msgr2=0x7f334c0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:12.047 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:11.901 7f3352c3c700 1 -- 172.21.2.102:0/738943359 shutdown_connections 2021-02-10T11:21:12.047 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:11.901 7f3352c3c700 1 -- 172.21.2.102:0/738943359 wait complete. 2021-02-10T11:21:12.049 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:12.051 7f3352c3c700 1 Processor -- start 2021-02-10T11:21:12.049 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:12.051 7f3352c3c700 1 -- start start 2021-02-10T11:21:12.049 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:12.051 7f3352c3c700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f334c09f170 0x7f334c0bb310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:12.049 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:12.051 7f3352c3c700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f334c09e640 0x7f334c09ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:12.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:12.051 7f3352c3c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f334c0bc1d0 0x7f334c0bc5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:12.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:12.051 7f3352c3c700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f334c0bbcd0 con 0x7f334c0bc1d0 2021-02-10T11:21:12.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:12.051 7f3352c3c700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f334c0bf930 con 0x7f334c09f170 2021-02-10T11:21:12.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:12.051 7f3352c3c700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f334c0bb930 con 0x7f334c09e640 2021-02-10T11:21:12.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:12.051 7f3351259700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f334c0bc1d0 0x7f334c0bc5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:12.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:12.052 7f3351259700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f334c0bc1d0 0x7f334c0bc5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:54264/0 (socket says 172.21.2.102:54264) 2021-02-10T11:21:12.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:12.052 7f3351259700 1 -- 172.21.2.102:0/751328223 learned_addr learned my addr 172.21.2.102:0/751328223 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:21:12.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:12.052 7f3350a58700 1 --2- 172.21.2.102:0/751328223 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f334c09f170 0x7f334c0bb310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:12.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:12.052 7f334bfff700 1 --2- 172.21.2.102:0/751328223 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f334c09e640 0x7f334c09ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:12.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:12.052 7f3351259700 1 -- 172.21.2.102:0/751328223 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f334c09e640 msgr2=0x7f334c09ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:12.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:12.052 7f3351259700 1 --2- 172.21.2.102:0/751328223 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f334c09e640 0x7f334c09ea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:12.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:12.052 7f3351259700 1 -- 172.21.2.102:0/751328223 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f334c09f170 msgr2=0x7f334c0bb310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:12.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:12.052 7f3351259700 1 --2- 172.21.2.102:0/751328223 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f334c09f170 0x7f334c0bb310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:12.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:12.052 7f3351259700 1 -- 172.21.2.102:0/751328223 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f334c0bcbf0 con 0x7f334c0bc1d0 2021-02-10T11:21:12.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:12.052 7f3350a58700 1 --2- 172.21.2.102:0/751328223 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f334c09f170 0x7f334c0bb310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:21:12.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:12.052 7f3351259700 1 --2- 172.21.2.102:0/751328223 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f334c0bc1d0 0x7f334c0bc5d0 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 rx=0x7f334400a8f0 tx=0x7f334400ac90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:12.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:12.052 7f33497fa700 1 -- 172.21.2.102:0/751328223 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f33440048b0 con 0x7f334c0bc1d0 2021-02-10T11:21:12.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:12.052 7f33497fa700 1 -- 172.21.2.102:0/751328223 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3344004c30 con 0x7f334c0bc1d0 2021-02-10T11:21:12.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:12.052 7f33497fa700 1 -- 172.21.2.102:0/751328223 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3344007350 con 0x7f334c0bc1d0 2021-02-10T11:21:12.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:12.052 7f3352c3c700 1 -- 172.21.2.102:0/751328223 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f334c0bcbf0 con 0x7f334c0bc1d0 2021-02-10T11:21:12.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:12.052 7f3352c3c700 1 -- 172.21.2.102:0/751328223 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f334c0bcbf0 con 0x7f334c0bc1d0 2021-02-10T11:21:12.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:12.053 7f33497fa700 1 -- 172.21.2.102:0/751328223 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f334400add0 con 0x7f334c0bc1d0 2021-02-10T11:21:12.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:12.053 7f33497fa700 1 --2- 172.21.2.102:0/751328223 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3330018730 0x7f333001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:12.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:12.053 7f33497fa700 1 -- 172.21.2.102:0/751328223 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f3344032b30 con 0x7f334c0bc1d0 2021-02-10T11:21:12.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:12.053 7f3350a58700 1 --2- 172.21.2.102:0/751328223 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3330018730 0x7f333001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:12.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:12.053 7f3350a58700 1 --2- 172.21.2.102:0/751328223 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3330018730 0x7f333001ac10 crc :-1 s=READY pgs=428 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:12.093 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:12.098 7f3352c3c700 1 -- 172.21.2.102:0/751328223 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f334c0bb930 con 0x7f334c0bc1d0 2021-02-10T11:21:12.096 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:12.100 7f33497fa700 1 -- 172.21.2.102:0/751328223 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f3344004c10 con 0x7f334c0bc1d0 2021-02-10T11:21:12.223 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:12.227 7f3352c3c700 1 -- 172.21.2.102:0/751328223 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344405", "target": ["mgr", ""]}) v1 -- 0x7f334c0bf930 con 0x7f3330018730 2021-02-10T11:21:12.233 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:12.238 7f33497fa700 1 -- 172.21.2.102:0/751328223 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f33340009b0 con 0x7f3330018730 2021-02-10T11:21:12.240 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:12.245 7f3352c3c700 1 -- 172.21.2.102:0/751328223 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3330018730 msgr2=0x7f333001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:12.240 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:12.245 7f3352c3c700 1 --2- 172.21.2.102:0/751328223 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3330018730 0x7f333001ac10 crc :-1 s=READY pgs=428 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:12.240 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:12.245 7f3352c3c700 1 -- 172.21.2.102:0/751328223 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f334c0bc1d0 msgr2=0x7f334c0bc5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:12.240 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:12.245 7f3352c3c700 1 --2- 172.21.2.102:0/751328223 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f334c0bc1d0 0x7f334c0bc5d0 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 rx=0x7f334400a8f0 tx=0x7f334400ac90).stop 2021-02-10T11:21:12.240 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:12.245 7f3352c3c700 1 -- 172.21.2.102:0/751328223 shutdown_connections 2021-02-10T11:21:12.240 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:12.245 7f3352c3c700 1 --2- 172.21.2.102:0/751328223 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f334c0bc1d0 0x7f334c0bc5d0 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:12.241 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:12.245 7f3352c3c700 1 --2- 172.21.2.102:0/751328223 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f334c09e640 0x7f334c09ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:12.241 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:12.245 7f3352c3c700 1 --2- 172.21.2.102:0/751328223 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3330018730 0x7f333001ac10 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:12.241 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:12.245 7f3352c3c700 1 --2- 172.21.2.102:0/751328223 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f334c09f170 0x7f334c0bb310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:12.241 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:12.245 7f3352c3c700 1 -- 172.21.2.102:0/751328223 >> 172.21.2.102:0/751328223 conn(0x7f334c09fc10 msgr2=0x7f334c0b3250 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:12.241 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:12.245 7f3352c3c700 1 -- 172.21.2.102:0/751328223 shutdown_connections 2021-02-10T11:21:12.241 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:12.245 7f3352c3c700 1 -- 172.21.2.102:0/751328223 wait complete. 2021-02-10T11:21:12.249 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000344406 2021-02-10T11:21:12.682 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:12.669 7f3fd69d5700 1 Processor -- start 2021-02-10T11:21:12.682 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:12.669 7f3fd69d5700 1 -- start start 2021-02-10T11:21:12.682 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:12.670 7f3fd69d5700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f3fd01b0470 0x7f3fd01b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:12.682 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:12.670 7f3fd69d5700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f3fd00f20e0 0x7f3fd00f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:12.682 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:12.670 7f3fd69d5700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fd00f0b50 con 0x7f3fd00ef610 2021-02-10T11:21:12.683 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:12.670 7f3fd69d5700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3fd00f0820 con 0x7f3fd01b0470 2021-02-10T11:21:12.683 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:12.670 7f3fd69d5700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3fd00f24e0 con 0x7f3fd00f20e0 2021-02-10T11:21:12.683 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:12.670 7f3fcffff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f3fd01b0470 0x7f3fd01b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:12.683 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:12.670 7f3fcffff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f3fd01b0470 0x7f3fd01b0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:54270/0 (socket says 172.21.2.102:54270) 2021-02-10T11:21:12.683 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:12.670 7f3fcffff700 1 -- 172.21.2.102:0/2624613684 learned_addr learned my addr 172.21.2.102:0/2624613684 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:21:12.683 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:12.670 7f3fcffff700 1 -- 172.21.2.102:0/2624613684 >> v2:172.21.2.113:3300/0 conn(0x7f3fd00f20e0 msgr2=0x7f3fd00f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:21:12.683 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:12.670 7f3fcffff700 1 --2- 172.21.2.102:0/2624613684 >> v2:172.21.2.113:3300/0 conn(0x7f3fd00f20e0 0x7f3fd00f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:21:12.683 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:12.670 7f3fcffff700 1 -- 172.21.2.102:0/2624613684 >> v1:172.21.2.102:6789/0 conn(0x7f3fd00ef610 legacy=0x7f3fd00efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:12.683 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:12.670 7f3fcffff700 1 -- 172.21.2.102:0/2624613684 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fd00e52e0 con 0x7f3fd01b0470 2021-02-10T11:21:12.683 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:12.670 7f3fcffff700 1 --2- 172.21.2.102:0/2624613684 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3fd01b0470 0x7f3fd01b0870 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 rx=0x7f3fc0006700 tx=0x7f3fc0009b80).ready entity=mon.0 client_cookie=996915d51e8e75e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:12.683 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:12.670 7f3fceffd700 1 -- 172.21.2.102:0/2624613684 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3fc000c0c0 con 0x7f3fd01b0470 2021-02-10T11:21:12.683 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:12.671 7f3fceffd700 1 -- 172.21.2.102:0/2624613684 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fc000c440 con 0x7f3fd01b0470 2021-02-10T11:21:12.683 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:12.671 7f3fceffd700 1 -- 172.21.2.102:0/2624613684 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3fc000c8a0 con 0x7f3fd01b0470 2021-02-10T11:21:12.684 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:12.671 7f3fd69d5700 1 -- 172.21.2.102:0/2624613684 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3fd01b0470 msgr2=0x7f3fd01b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:12.684 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:12.671 7f3fd69d5700 1 --2- 172.21.2.102:0/2624613684 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3fd01b0470 0x7f3fd01b0870 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 rx=0x7f3fc0006700 tx=0x7f3fc0009b80).stop 2021-02-10T11:21:12.684 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:12.671 7f3fd69d5700 1 -- 172.21.2.102:0/2624613684 shutdown_connections 2021-02-10T11:21:12.684 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:12.671 7f3fd69d5700 1 --2- 172.21.2.102:0/2624613684 >> v2:172.21.2.113:3300/0 conn(0x7f3fd00f20e0 0x7f3fd00f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:21:12.684 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:12.671 7f3fd69d5700 1 --2- 172.21.2.102:0/2624613684 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3fd01b0470 0x7f3fd01b0870 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:12.684 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:12.671 7f3fd69d5700 1 -- 172.21.2.102:0/2624613684 >> 172.21.2.102:0/2624613684 conn(0x7f3fd00ce3e0 msgr2=0x7f3fd00e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:12.684 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:12.671 7f3fd69d5700 1 -- 172.21.2.102:0/2624613684 shutdown_connections 2021-02-10T11:21:12.684 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:12.671 7f3fd69d5700 1 -- 172.21.2.102:0/2624613684 wait complete. 2021-02-10T11:21:12.684 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:12.687 7f3fd69d5700 1 Processor -- start 2021-02-10T11:21:12.687 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:12.687 7f3fd69d5700 1 -- start start 2021-02-10T11:21:12.687 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:12.688 7f3fd69d5700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3fd00ad2e0 0x7f3fd00aca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:12.688 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:12.688 7f3fd69d5700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3fd019dcd0 0x7f3fd00ad7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:12.688 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:12.688 7f3fd69d5700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3fd00addd0 0x7f3fd00c2d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:12.688 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:12.688 7f3fd69d5700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fd00a37f0 con 0x7f3fd019dcd0 2021-02-10T11:21:12.688 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:12.688 7f3fd69d5700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fd00c2810 con 0x7f3fd00ad2e0 2021-02-10T11:21:12.688 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:12.688 7f3fd69d5700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fd019e0d0 con 0x7f3fd00addd0 2021-02-10T11:21:12.688 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:12.688 7f3fcf7fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3fd019dcd0 0x7f3fd00ad7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:12.688 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:12.688 7f3fcffff700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3fd00ad2e0 0x7f3fd00aca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:12.688 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:12.688 7f3fd4ff2700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3fd00addd0 0x7f3fd00c2d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:12.689 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:12.688 7f3fcf7fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3fd019dcd0 0x7f3fd00ad7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:54280/0 (socket says 172.21.2.102:54280) 2021-02-10T11:21:12.689 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:12.688 7f3fcf7fe700 1 -- 172.21.2.102:0/2363422527 learned_addr learned my addr 172.21.2.102:0/2363422527 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:21:12.689 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:12.688 7f3fcf7fe700 1 -- 172.21.2.102:0/2363422527 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3fd00addd0 msgr2=0x7f3fd00c2d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:12.689 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:12.688 7f3fcf7fe700 1 --2- 172.21.2.102:0/2363422527 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3fd00addd0 0x7f3fd00c2d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:12.689 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:12.688 7f3fcf7fe700 1 -- 172.21.2.102:0/2363422527 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3fd00ad2e0 msgr2=0x7f3fd00aca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:12.689 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:12.688 7f3fcf7fe700 1 --2- 172.21.2.102:0/2363422527 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3fd00ad2e0 0x7f3fd00aca50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:12.689 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:12.688 7f3fcf7fe700 1 -- 172.21.2.102:0/2363422527 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fd019e2b0 con 0x7f3fd019dcd0 2021-02-10T11:21:12.689 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:12.688 7f3fcf7fe700 1 --2- 172.21.2.102:0/2363422527 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3fd019dcd0 0x7f3fd00ad7b0 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 rx=0x7f3fc4006ed0 tx=0x7f3fc400b8e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:12.689 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:12.688 7f3fccff9700 1 -- 172.21.2.102:0/2363422527 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3fc4006020 con 0x7f3fd019dcd0 2021-02-10T11:21:12.689 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:12.688 7f3fccff9700 1 -- 172.21.2.102:0/2363422527 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fc4006360 con 0x7f3fd019dcd0 2021-02-10T11:21:12.689 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:12.688 7f3fccff9700 1 -- 172.21.2.102:0/2363422527 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3fc4005730 con 0x7f3fd019dcd0 2021-02-10T11:21:12.689 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:12.688 7f3fd69d5700 1 -- 172.21.2.102:0/2363422527 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fd019e2b0 con 0x7f3fd019dcd0 2021-02-10T11:21:12.689 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:12.688 7f3fd69d5700 1 -- 172.21.2.102:0/2363422527 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fd019e2b0 con 0x7f3fd019dcd0 2021-02-10T11:21:12.690 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:12.689 7f3fccff9700 1 -- 172.21.2.102:0/2363422527 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f3fc40054b0 con 0x7f3fd019dcd0 2021-02-10T11:21:12.690 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:12.689 7f3fccff9700 1 --2- 172.21.2.102:0/2363422527 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3fb0018730 0x7f3fb001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:12.690 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:12.689 7f3fccff9700 1 -- 172.21.2.102:0/2363422527 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f3fc4032690 con 0x7f3fd019dcd0 2021-02-10T11:21:12.690 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:12.689 7f3fcffff700 1 --2- 172.21.2.102:0/2363422527 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3fb0018730 0x7f3fb001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:12.690 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:12.689 7f3fcffff700 1 --2- 172.21.2.102:0/2363422527 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3fb0018730 0x7f3fb001ac10 crc :-1 s=READY pgs=429 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:12.715 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:12.720 7f3fd69d5700 1 -- 172.21.2.102:0/2363422527 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fd00edd10 con 0x7f3fd019dcd0 2021-02-10T11:21:12.717 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:12.722 7f3fccff9700 1 -- 172.21.2.102:0/2363422527 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f3fc40054b0 con 0x7f3fd019dcd0 2021-02-10T11:21:12.846 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:12.851 7f3fd69d5700 1 -- 172.21.2.102:0/2363422527 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344406", "target": ["mgr", ""]}) v1 -- 0x7f3fd00e52e0 con 0x7f3fb0018730 2021-02-10T11:21:12.857 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:12.862 7f3fccff9700 1 -- 172.21.2.102:0/2363422527 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f3fc0006430 con 0x7f3fb0018730 2021-02-10T11:21:12.863 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:12.868 7f3fd69d5700 1 -- 172.21.2.102:0/2363422527 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3fb0018730 msgr2=0x7f3fb001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:12.864 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:12.868 7f3fd69d5700 1 --2- 172.21.2.102:0/2363422527 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3fb0018730 0x7f3fb001ac10 crc :-1 s=READY pgs=429 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:12.864 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:12.868 7f3fd69d5700 1 -- 172.21.2.102:0/2363422527 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3fd019dcd0 msgr2=0x7f3fd00ad7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:12.864 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:12.868 7f3fd69d5700 1 --2- 172.21.2.102:0/2363422527 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3fd019dcd0 0x7f3fd00ad7b0 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 rx=0x7f3fc4006ed0 tx=0x7f3fc400b8e0).stop 2021-02-10T11:21:12.864 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:12.868 7f3fd69d5700 1 -- 172.21.2.102:0/2363422527 shutdown_connections 2021-02-10T11:21:12.864 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:12.868 7f3fd69d5700 1 --2- 172.21.2.102:0/2363422527 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3fd00addd0 0x7f3fd00c2d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:12.864 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:12.868 7f3fd69d5700 1 --2- 172.21.2.102:0/2363422527 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3fd019dcd0 0x7f3fd00ad7b0 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:12.864 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:12.868 7f3fd69d5700 1 --2- 172.21.2.102:0/2363422527 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3fb0018730 0x7f3fb001ac10 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:12.864 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:12.868 7f3fd69d5700 1 --2- 172.21.2.102:0/2363422527 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3fd00ad2e0 0x7f3fd00aca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:12.864 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:12.868 7f3fd69d5700 1 -- 172.21.2.102:0/2363422527 >> 172.21.2.102:0/2363422527 conn(0x7f3fd00c40e0 msgr2=0x7f3fd00f05a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:12.864 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:12.868 7f3fd69d5700 1 -- 172.21.2.102:0/2363422527 shutdown_connections 2021-02-10T11:21:12.864 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:12.868 7f3fd69d5700 1 -- 172.21.2.102:0/2363422527 wait complete. 2021-02-10T11:21:12.872 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000344407 2021-02-10T11:21:13.301 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:13.289 7f0f5dd01700 1 Processor -- start 2021-02-10T11:21:13.301 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:13.290 7f0f5dd01700 1 -- start start 2021-02-10T11:21:13.301 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:13.290 7f0f5dd01700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0f581b0870 0x7f0f580f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:13.301 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:13.290 7f0f5dd01700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f0f580efe00 0x7f0f580f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:13.301 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:13.290 7f0f5dd01700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f581b0c70 con 0x7f0f581b0470 2021-02-10T11:21:13.302 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:13.290 7f0f5dd01700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f580f0b50 con 0x7f0f580efe00 2021-02-10T11:21:13.302 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:13.290 7f0f5dd01700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f580f0820 con 0x7f0f581b0870 2021-02-10T11:21:13.302 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:13.290 7f0f577fe700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f0f581b0470 0x7f0f580ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:46508/0 (socket says 172.21.2.102:46508) 2021-02-10T11:21:13.302 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:13.290 7f0f577fe700 1 -- 172.21.2.102:0/3572326196 learned_addr learned my addr 172.21.2.102:0/3572326196 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:21:13.302 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:13.290 7f0f57fff700 1 --2- 172.21.2.102:0/3572326196 >> v2:172.21.2.113:3300/0 conn(0x7f0f580efe00 0x7f0f580f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:13.302 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:13.290 7f0f56ffd700 1 --2- 172.21.2.102:0/3572326196 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0f581b0870 0x7f0f580f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:13.302 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:13.291 7f0f57fff700 1 -- 172.21.2.102:0/3572326196 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0f581b0870 msgr2=0x7f0f580f20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:21:13.302 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:13.291 7f0f57fff700 1 --2- 172.21.2.102:0/3572326196 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0f581b0870 0x7f0f580f20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:13.302 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:13.291 7f0f57fff700 1 -- 172.21.2.102:0/3572326196 >> v1:172.21.2.102:6789/0 conn(0x7f0f581b0470 legacy=0x7f0f580ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:13.302 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:13.291 7f0f567fc700 1 -- 172.21.2.102:0/3572326196 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1275181535 0 0) 0x7f0f48000ce0 con 0x7f0f581b0470 2021-02-10T11:21:13.302 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:13.291 7f0f57fff700 1 -- 172.21.2.102:0/3572326196 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f580e52e0 con 0x7f0f580efe00 2021-02-10T11:21:13.302 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:13.291 7f0f57fff700 1 --2- 172.21.2.102:0/3572326196 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0f580efe00 0x7f0f580f0200 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 rx=0x7f0f50006e50 tx=0x7f0f50009e70).ready entity=mon.1 client_cookie=a3102ec3ea9e4376 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:13.302 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:13.291 7f0f567fc700 1 -- 172.21.2.102:0/3572326196 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0f50006a20 con 0x7f0f580efe00 2021-02-10T11:21:13.303 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:13.291 7f0f567fc700 1 -- 172.21.2.102:0/3572326196 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f50006790 con 0x7f0f580efe00 2021-02-10T11:21:13.303 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:13.291 7f0f567fc700 1 -- 172.21.2.102:0/3572326196 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0f50000f90 con 0x7f0f580efe00 2021-02-10T11:21:13.303 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:13.291 7f0f5dd01700 1 -- 172.21.2.102:0/3572326196 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0f580efe00 msgr2=0x7f0f580f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:13.303 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:13.291 7f0f5dd01700 1 --2- 172.21.2.102:0/3572326196 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0f580efe00 0x7f0f580f0200 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 rx=0x7f0f50006e50 tx=0x7f0f50009e70).stop 2021-02-10T11:21:13.303 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:13.292 7f0f5dd01700 1 -- 172.21.2.102:0/3572326196 shutdown_connections 2021-02-10T11:21:13.303 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:13.292 7f0f5dd01700 1 --2- 172.21.2.102:0/3572326196 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0f580efe00 0x7f0f580f0200 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:13.303 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:13.292 7f0f5dd01700 1 --2- 172.21.2.102:0/3572326196 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0f581b0870 0x7f0f580f20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:13.303 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:13.292 7f0f5dd01700 1 -- 172.21.2.102:0/3572326196 >> 172.21.2.102:0/3572326196 conn(0x7f0f580ce3e0 msgr2=0x7f0f580e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:13.303 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:13.292 7f0f5dd01700 1 -- 172.21.2.102:0/3572326196 shutdown_connections 2021-02-10T11:21:13.303 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:13.292 7f0f5dd01700 1 -- 172.21.2.102:0/3572326196 wait complete. 2021-02-10T11:21:13.305 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:13.306 7f0f5dd01700 1 Processor -- start 2021-02-10T11:21:13.305 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:13.306 7f0f5dd01700 1 -- start start 2021-02-10T11:21:13.305 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:13.307 7f0f5dd01700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0f580a0420 0x7f0f580a0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:13.305 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:13.307 7f0f5dd01700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0f58189d00 0x7f0f5818a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:13.305 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:13.307 7f0f5dd01700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0f5818a720 0x7f0f580b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:13.305 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:13.307 7f0f5dd01700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f58189780 con 0x7f0f580a0420 2021-02-10T11:21:13.305 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:13.307 7f0f5dd01700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f5818ab20 con 0x7f0f58189d00 2021-02-10T11:21:13.305 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:13.307 7f0f5dd01700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f580a1140 con 0x7f0f5818a720 2021-02-10T11:21:13.305 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:13.307 7f0f577fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0f580a0420 0x7f0f580a0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:13.305 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:13.307 7f0f577fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0f580a0420 0x7f0f580a0820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:54292/0 (socket says 172.21.2.102:54292) 2021-02-10T11:21:13.305 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:13.307 7f0f577fe700 1 -- 172.21.2.102:0/1259573882 learned_addr learned my addr 172.21.2.102:0/1259573882 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:21:13.306 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:13.307 7f0f577fe700 1 -- 172.21.2.102:0/1259573882 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0f5818a720 msgr2=0x7f0f580b7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:13.306 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:13.307 7f0f577fe700 1 --2- 172.21.2.102:0/1259573882 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0f5818a720 0x7f0f580b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:21:13.306 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:13.307 7f0f577fe700 1 -- 172.21.2.102:0/1259573882 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0f58189d00 msgr2=0x7f0f5818a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:13.306 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:13.307 7f0f577fe700 1 --2- 172.21.2.102:0/1259573882 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0f58189d00 0x7f0f5818a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:21:13.306 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:13.307 7f0f577fe700 1 -- 172.21.2.102:0/1259573882 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f580b8200 con 0x7f0f580a0420 2021-02-10T11:21:13.306 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:13.308 7f0f577fe700 1 --2- 172.21.2.102:0/1259573882 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0f580a0420 0x7f0f580a0820 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 rx=0x7f0f48006f00 tx=0x7f0f4800b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:13.306 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:13.308 7f0f3ffff700 1 -- 172.21.2.102:0/1259573882 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0f48006020 con 0x7f0f580a0420 2021-02-10T11:21:13.306 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:13.308 7f0f3ffff700 1 -- 172.21.2.102:0/1259573882 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f480063a0 con 0x7f0f580a0420 2021-02-10T11:21:13.306 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:13.308 7f0f3ffff700 1 -- 172.21.2.102:0/1259573882 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0f48004620 con 0x7f0f580a0420 2021-02-10T11:21:13.306 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:13.308 7f0f5dd01700 1 -- 172.21.2.102:0/1259573882 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f580b8200 con 0x7f0f580a0420 2021-02-10T11:21:13.306 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:13.308 7f0f5dd01700 1 -- 172.21.2.102:0/1259573882 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f580b8200 con 0x7f0f580a0420 2021-02-10T11:21:13.306 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:13.308 7f0f3ffff700 1 -- 172.21.2.102:0/1259573882 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f0f4800a000 con 0x7f0f580a0420 2021-02-10T11:21:13.306 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:13.308 7f0f3ffff700 1 --2- 172.21.2.102:0/1259573882 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0f38018730 0x7f0f3801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:13.306 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:13.308 7f0f3ffff700 1 -- 172.21.2.102:0/1259573882 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f0f48032660 con 0x7f0f580a0420 2021-02-10T11:21:13.307 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:13.308 7f0f56ffd700 1 --2- 172.21.2.102:0/1259573882 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0f38018730 0x7f0f3801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:13.307 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:13.309 7f0f56ffd700 1 --2- 172.21.2.102:0/1259573882 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0f38018730 0x7f0f3801abc0 crc :-1 s=READY pgs=430 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:13.335 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:13.340 7f0f5dd01700 1 -- 172.21.2.102:0/1259573882 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f58105ae0 con 0x7f0f580a0420 2021-02-10T11:21:13.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:13.342 7f0f3ffff700 1 -- 172.21.2.102:0/1259573882 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f0f48006380 con 0x7f0f580a0420 2021-02-10T11:21:13.468 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:13.473 7f0f5dd01700 1 -- 172.21.2.102:0/1259573882 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344407", "target": ["mgr", ""]}) v1 -- 0x7f0f580f0b50 con 0x7f0f38018730 2021-02-10T11:21:13.479 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:13.484 7f0f3ffff700 1 -- 172.21.2.102:0/1259573882 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f0f4c006e70 con 0x7f0f38018730 2021-02-10T11:21:13.485 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:13.490 7f0f5dd01700 1 -- 172.21.2.102:0/1259573882 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0f38018730 msgr2=0x7f0f3801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:13.485 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:13.490 7f0f5dd01700 1 --2- 172.21.2.102:0/1259573882 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0f38018730 0x7f0f3801abc0 crc :-1 s=READY pgs=430 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:13.485 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:13.490 7f0f5dd01700 1 -- 172.21.2.102:0/1259573882 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0f580a0420 msgr2=0x7f0f580a0820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:13.485 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:13.490 7f0f5dd01700 1 --2- 172.21.2.102:0/1259573882 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0f580a0420 0x7f0f580a0820 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 rx=0x7f0f48006f00 tx=0x7f0f4800b940).stop 2021-02-10T11:21:13.485 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:13.490 7f0f5dd01700 1 -- 172.21.2.102:0/1259573882 shutdown_connections 2021-02-10T11:21:13.486 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:13.490 7f0f5dd01700 1 --2- 172.21.2.102:0/1259573882 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0f5818a720 0x7f0f580b7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:21:13.486 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:13.490 7f0f5dd01700 1 --2- 172.21.2.102:0/1259573882 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0f38018730 0x7f0f3801abc0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:13.486 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:13.490 7f0f5dd01700 1 --2- 172.21.2.102:0/1259573882 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0f58189d00 0x7f0f5818a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:21:13.486 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:13.490 7f0f5dd01700 1 --2- 172.21.2.102:0/1259573882 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0f580a0420 0x7f0f580a0820 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:13.486 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:13.490 7f0f5dd01700 1 -- 172.21.2.102:0/1259573882 >> 172.21.2.102:0/1259573882 conn(0x7f0f58094e10 msgr2=0x7f0f580a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:13.486 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:13.490 7f0f5dd01700 1 -- 172.21.2.102:0/1259573882 shutdown_connections 2021-02-10T11:21:13.486 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:13.490 7f0f5dd01700 1 -- 172.21.2.102:0/1259573882 wait complete. 2021-02-10T11:21:13.494 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000344408 2021-02-10T11:21:13.926 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:13.913 7f6a829fc700 1 Processor -- start 2021-02-10T11:21:13.926 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:13.913 7f6a829fc700 1 -- start start 2021-02-10T11:21:13.926 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:13.914 7f6a829fc700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6a7c1b0470 0x7f6a7c1b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:13.926 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:13.914 7f6a829fc700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a7c0f0b50 con 0x7f6a7c0ef610 2021-02-10T11:21:13.926 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:13.914 7f6a829fc700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a7c0f24e0 con 0x7f6a7c0f20e0 2021-02-10T11:21:13.927 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:13.914 7f6a829fc700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a7c0e5ea0 con 0x7f6a7c1b0470 2021-02-10T11:21:13.927 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:13.914 7f6a7bfff700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f6a7c0ef610 0x7f6a7c0efa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:46524/0 (socket says 172.21.2.102:46524) 2021-02-10T11:21:13.927 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:13.914 7f6a7bfff700 1 -- 172.21.2.102:0/3152250183 learned_addr learned my addr 172.21.2.102:0/3152250183 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:21:13.927 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:13.914 7f6a80818700 1 --2- 172.21.2.102:0/3152250183 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6a7c1b0470 0x7f6a7c1b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:13.927 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:13.914 7f6a7b7fe700 1 -- 172.21.2.102:0/3152250183 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 393640660 0 0) 0x7f6a700009b0 con 0x7f6a7c0ef610 2021-02-10T11:21:13.927 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:13.915 7f6a7b7fe700 1 -- 172.21.2.102:0/3152250183 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a64002fe0 con 0x7f6a7c0ef610 2021-02-10T11:21:13.927 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:13.915 7f6a7b7fe700 1 -- 172.21.2.102:0/3152250183 <== mon.1 v1:172.21.2.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3815676669 0 0) 0x7f6a6c001080 con 0x7f6a7c0f20e0 2021-02-10T11:21:13.927 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:13.915 7f6a7b7fe700 1 -- 172.21.2.102:0/3152250183 --> v1:172.21.2.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a64004370 con 0x7f6a7c0f20e0 2021-02-10T11:21:13.927 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:13.915 7f6a80818700 1 -- 172.21.2.102:0/3152250183 >> v1:172.21.2.113:6789/0 conn(0x7f6a7c0f20e0 legacy=0x7f6a7c0f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:13.927 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:13.915 7f6a80818700 1 -- 172.21.2.102:0/3152250183 >> v1:172.21.2.102:6789/0 conn(0x7f6a7c0ef610 legacy=0x7f6a7c0efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:13.927 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:13.915 7f6a7b7fe700 1 -- 172.21.2.102:0/3152250183 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2706232413 0 0) 0x7f6a700010b0 con 0x7f6a7c0ef610 2021-02-10T11:21:13.928 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:13.915 7f6a80818700 1 -- 172.21.2.102:0/3152250183 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a7c0e52e0 con 0x7f6a7c1b0470 2021-02-10T11:21:13.928 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:13.915 7f6a80818700 1 --2- 172.21.2.102:0/3152250183 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6a7c1b0470 0x7f6a7c1b0870 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7f6a740069d0 tx=0x7f6a74009e40).ready entity=mon.2 client_cookie=f145bd60ee152aa9 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:13.928 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:13.915 7f6a7b7fe700 1 -- 172.21.2.102:0/3152250183 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6a7400b510 con 0x7f6a7c1b0470 2021-02-10T11:21:13.928 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:13.915 7f6a7b7fe700 1 -- 172.21.2.102:0/3152250183 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a7400b890 con 0x7f6a7c1b0470 2021-02-10T11:21:13.928 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:13.915 7f6a7b7fe700 1 -- 172.21.2.102:0/3152250183 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6a7400bcf0 con 0x7f6a7c1b0470 2021-02-10T11:21:13.928 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:13.915 7f6a829fc700 1 -- 172.21.2.102:0/3152250183 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6a7c1b0470 msgr2=0x7f6a7c1b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:13.928 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:13.915 7f6a829fc700 1 --2- 172.21.2.102:0/3152250183 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6a7c1b0470 0x7f6a7c1b0870 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7f6a740069d0 tx=0x7f6a74009e40).stop 2021-02-10T11:21:13.928 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:13.916 7f6a829fc700 1 -- 172.21.2.102:0/3152250183 shutdown_connections 2021-02-10T11:21:13.928 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:13.916 7f6a829fc700 1 --2- 172.21.2.102:0/3152250183 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6a7c1b0470 0x7f6a7c1b0870 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:13.928 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:13.916 7f6a829fc700 1 -- 172.21.2.102:0/3152250183 >> 172.21.2.102:0/3152250183 conn(0x7f6a7c0ce3e0 msgr2=0x7f6a7c0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:13.928 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:13.916 7f6a829fc700 1 -- 172.21.2.102:0/3152250183 shutdown_connections 2021-02-10T11:21:13.929 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:13.916 7f6a829fc700 1 -- 172.21.2.102:0/3152250183 wait complete. 2021-02-10T11:21:13.929 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:13.931 7f6a829fc700 1 Processor -- start 2021-02-10T11:21:13.929 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:13.931 7f6a829fc700 1 -- start start 2021-02-10T11:21:13.930 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:13.931 7f6a829fc700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6a7c08ece0 0x7f6a7c0a6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:13.931 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:13.931 7f6a829fc700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6a7c0a6860 0x7f6a7c0a6c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:13.931 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:13.931 7f6a829fc700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6a7c0a7280 0x7f6a7c0a4590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:13.931 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:13.931 7f6a829fc700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a7c0a5cc0 con 0x7f6a7c0a7280 2021-02-10T11:21:13.931 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:13.931 7f6a829fc700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a7c08f0e0 con 0x7f6a7c0a6860 2021-02-10T11:21:13.931 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:13.931 7f6a829fc700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a7c08f2c0 con 0x7f6a7c08ece0 2021-02-10T11:21:13.931 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:13.931 7f6a81019700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6a7c0a7280 0x7f6a7c0a4590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:13.931 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:13.931 7f6a81019700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6a7c0a7280 0x7f6a7c0a4590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:54308/0 (socket says 172.21.2.102:54308) 2021-02-10T11:21:13.931 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:13.931 7f6a81019700 1 -- 172.21.2.102:0/1832681684 learned_addr learned my addr 172.21.2.102:0/1832681684 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:21:13.931 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:13.932 7f6a80818700 1 --2- 172.21.2.102:0/1832681684 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6a7c08ece0 0x7f6a7c0a6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:13.931 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:13.932 7f6a7bfff700 1 --2- 172.21.2.102:0/1832681684 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6a7c0a6860 0x7f6a7c0a6c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:13.932 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:13.932 7f6a81019700 1 -- 172.21.2.102:0/1832681684 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6a7c08ece0 msgr2=0x7f6a7c0a6240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:13.932 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:13.932 7f6a81019700 1 --2- 172.21.2.102:0/1832681684 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6a7c08ece0 0x7f6a7c0a6240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:13.932 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:13.932 7f6a81019700 1 -- 172.21.2.102:0/1832681684 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6a7c0a6860 msgr2=0x7f6a7c0a6c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:13.932 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:13.932 7f6a81019700 1 --2- 172.21.2.102:0/1832681684 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6a7c0a6860 0x7f6a7c0a6c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:13.932 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:13.932 7f6a81019700 1 -- 172.21.2.102:0/1832681684 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a7c0a4bb0 con 0x7f6a7c0a7280 2021-02-10T11:21:13.932 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:13.932 7f6a81019700 1 --2- 172.21.2.102:0/1832681684 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6a7c0a7280 0x7f6a7c0a4590 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 rx=0x7f6a6c00b5b0 tx=0x7f6a6c00b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:13.932 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:13.932 7f6a797fa700 1 -- 172.21.2.102:0/1832681684 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6a6c000bc0 con 0x7f6a7c0a7280 2021-02-10T11:21:13.932 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:13.932 7f6a797fa700 1 -- 172.21.2.102:0/1832681684 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a6c006240 con 0x7f6a7c0a7280 2021-02-10T11:21:13.932 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:13.932 7f6a797fa700 1 -- 172.21.2.102:0/1832681684 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6a6c005690 con 0x7f6a7c0a7280 2021-02-10T11:21:13.932 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:13.932 7f6a829fc700 1 -- 172.21.2.102:0/1832681684 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a7c0a4bb0 con 0x7f6a7c0a7280 2021-02-10T11:21:13.932 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:13.932 7f6a829fc700 1 -- 172.21.2.102:0/1832681684 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a7c0a5140 con 0x7f6a7c0a7280 2021-02-10T11:21:13.933 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:13.933 7f6a797fa700 1 -- 172.21.2.102:0/1832681684 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f6a6c0054a0 con 0x7f6a7c0a7280 2021-02-10T11:21:13.933 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:13.933 7f6a797fa700 1 --2- 172.21.2.102:0/1832681684 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6a5c0186c0 0x7f6a5c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:13.933 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:13.933 7f6a797fa700 1 -- 172.21.2.102:0/1832681684 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f6a6c0327d0 con 0x7f6a7c0a7280 2021-02-10T11:21:13.933 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:13.933 7f6a80818700 1 --2- 172.21.2.102:0/1832681684 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6a5c0186c0 0x7f6a5c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:13.933 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:13.933 7f6a80818700 1 --2- 172.21.2.102:0/1832681684 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6a5c0186c0 0x7f6a5c01abc0 crc :-1 s=READY pgs=431 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:13.959 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:13.963 7f6a829fc700 1 -- 172.21.2.102:0/1832681684 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a7c19d1a0 con 0x7f6a7c0a7280 2021-02-10T11:21:13.961 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:13.966 7f6a797fa700 1 -- 172.21.2.102:0/1832681684 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f6a6c005800 con 0x7f6a7c0a7280 2021-02-10T11:21:14.088 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:14.093 7f6a829fc700 1 -- 172.21.2.102:0/1832681684 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344408", "target": ["mgr", ""]}) v1 -- 0x7f6a7c0a5cc0 con 0x7f6a5c0186c0 2021-02-10T11:21:14.099 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:14.104 7f6a797fa700 1 -- 172.21.2.102:0/1832681684 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f6a74006e60 con 0x7f6a5c0186c0 2021-02-10T11:21:14.110 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:14.110 7f6a829fc700 1 -- 172.21.2.102:0/1832681684 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6a5c0186c0 msgr2=0x7f6a5c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:14.110 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:14.110 7f6a829fc700 1 --2- 172.21.2.102:0/1832681684 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6a5c0186c0 0x7f6a5c01abc0 crc :-1 s=READY pgs=431 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:14.110 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:14.110 7f6a829fc700 1 -- 172.21.2.102:0/1832681684 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6a7c0a7280 msgr2=0x7f6a7c0a4590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:14.110 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:14.110 7f6a829fc700 1 --2- 172.21.2.102:0/1832681684 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6a7c0a7280 0x7f6a7c0a4590 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 rx=0x7f6a6c00b5b0 tx=0x7f6a6c00b950).stop 2021-02-10T11:21:14.110 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:14.110 7f6a829fc700 1 -- 172.21.2.102:0/1832681684 shutdown_connections 2021-02-10T11:21:14.110 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:14.110 7f6a829fc700 1 --2- 172.21.2.102:0/1832681684 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6a7c0a7280 0x7f6a7c0a4590 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:14.110 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:14.110 7f6a829fc700 1 --2- 172.21.2.102:0/1832681684 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6a5c0186c0 0x7f6a5c01abc0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:14.110 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:14.110 7f6a829fc700 1 --2- 172.21.2.102:0/1832681684 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6a7c0a6860 0x7f6a7c0a6c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:14.110 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:14.110 7f6a829fc700 1 --2- 172.21.2.102:0/1832681684 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6a7c08ece0 0x7f6a7c0a6240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:14.111 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:14.110 7f6a829fc700 1 -- 172.21.2.102:0/1832681684 >> 172.21.2.102:0/1832681684 conn(0x7f6a7c08ffe0 msgr2=0x7f6a7c0bf050 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:14.111 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:14.110 7f6a829fc700 1 -- 172.21.2.102:0/1832681684 shutdown_connections 2021-02-10T11:21:14.111 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:14.110 7f6a829fc700 1 -- 172.21.2.102:0/1832681684 wait complete. 2021-02-10T11:21:14.115 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000344409 2021-02-10T11:21:14.549 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:14.536 7ff434561700 1 Processor -- start 2021-02-10T11:21:14.549 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:14.536 7ff434561700 1 -- start start 2021-02-10T11:21:14.549 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:14.537 7ff434561700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff42c0f05f0 0x7ff42c0f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:14.549 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:14.537 7ff434561700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff42c1b0c70 con 0x7ff42c1b0870 2021-02-10T11:21:14.549 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:14.537 7ff434561700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff42c0f0b50 con 0x7ff42c1b0470 2021-02-10T11:21:14.549 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:14.537 7ff434561700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff42c0e5ea0 con 0x7ff42c0f05f0 2021-02-10T11:21:14.549 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:14.537 7ff431b7c700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7ff42c1b0870 0x7ff42c0efe00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:46540/0 (socket says 172.21.2.102:46540) 2021-02-10T11:21:14.550 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:14.537 7ff431b7c700 1 -- 172.21.2.102:0/3683305231 learned_addr learned my addr 172.21.2.102:0/3683305231 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:21:14.550 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:14.537 7ff432b7e700 1 --2- 172.21.2.102:0/3683305231 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff42c0f05f0 0x7ff42c0f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:14.550 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:14.537 7ff43137b700 1 -- 172.21.2.102:0/3683305231 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2257882913 0 0) 0x7ff41c0009b0 con 0x7ff42c1b0870 2021-02-10T11:21:14.550 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:14.537 7ff43137b700 1 -- 172.21.2.102:0/3683305231 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff418002fe0 con 0x7ff42c1b0870 2021-02-10T11:21:14.550 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:14.538 7ff43137b700 1 -- 172.21.2.102:0/3683305231 <== mon.1 v1:172.21.2.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2852949860 0 0) 0x7ff428000c60 con 0x7ff42c1b0470 2021-02-10T11:21:14.550 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:14.538 7ff43137b700 1 -- 172.21.2.102:0/3683305231 --> v1:172.21.2.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff418003210 con 0x7ff42c1b0470 2021-02-10T11:21:14.550 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:14.538 7ff432b7e700 1 -- 172.21.2.102:0/3683305231 >> v1:172.21.2.113:6789/0 conn(0x7ff42c1b0470 legacy=0x7ff42c0ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:14.550 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:14.538 7ff432b7e700 1 -- 172.21.2.102:0/3683305231 >> v1:172.21.2.102:6789/0 conn(0x7ff42c1b0870 legacy=0x7ff42c0efe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:14.550 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:14.538 7ff432b7e700 1 -- 172.21.2.102:0/3683305231 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff42c0e52e0 con 0x7ff42c0f05f0 2021-02-10T11:21:14.550 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:14.538 7ff432b7e700 1 --2- 172.21.2.102:0/3683305231 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff42c0f05f0 0x7ff42c0f20e0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7ff424006e50 tx=0x7ff424009e70).ready entity=mon.2 client_cookie=d1271213f38edccf server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:14.551 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:14.538 7ff43137b700 1 -- 172.21.2.102:0/3683305231 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff424006a20 con 0x7ff42c0f05f0 2021-02-10T11:21:14.551 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:14.538 7ff43137b700 1 -- 172.21.2.102:0/3683305231 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff424006790 con 0x7ff42c0f05f0 2021-02-10T11:21:14.551 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:14.538 7ff43137b700 1 -- 172.21.2.102:0/3683305231 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff42400b700 con 0x7ff42c0f05f0 2021-02-10T11:21:14.551 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:14.538 7ff434561700 1 -- 172.21.2.102:0/3683305231 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff42c0f05f0 msgr2=0x7ff42c0f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:14.551 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:14.538 7ff434561700 1 --2- 172.21.2.102:0/3683305231 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff42c0f05f0 0x7ff42c0f20e0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7ff424006e50 tx=0x7ff424009e70).stop 2021-02-10T11:21:14.551 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:14.539 7ff434561700 1 -- 172.21.2.102:0/3683305231 shutdown_connections 2021-02-10T11:21:14.551 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:14.539 7ff434561700 1 --2- 172.21.2.102:0/3683305231 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff42c0f05f0 0x7ff42c0f20e0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:14.551 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:14.539 7ff434561700 1 -- 172.21.2.102:0/3683305231 >> 172.21.2.102:0/3683305231 conn(0x7ff42c0ce3e0 msgr2=0x7ff42c0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:14.551 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:14.539 7ff434561700 1 -- 172.21.2.102:0/3683305231 shutdown_connections 2021-02-10T11:21:14.551 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:14.539 7ff434561700 1 -- 172.21.2.102:0/3683305231 wait complete. 2021-02-10T11:21:14.553 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:14.554 7ff434561700 1 Processor -- start 2021-02-10T11:21:14.553 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:14.554 7ff434561700 1 -- start start 2021-02-10T11:21:14.554 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:14.554 7ff434561700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff42c0a0420 0x7ff42c0a0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:14.554 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:14.554 7ff434561700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff42c189d00 0x7ff42c18a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:14.554 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:14.554 7ff434561700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff42c18a720 0x7ff42c0b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:14.554 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:14.554 7ff434561700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff42c189780 con 0x7ff42c18a720 2021-02-10T11:21:14.554 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:14.554 7ff434561700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff42c18ab20 con 0x7ff42c0a0420 2021-02-10T11:21:14.554 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:14.554 7ff434561700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff42c0a1140 con 0x7ff42c189d00 2021-02-10T11:21:14.554 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:14.554 7ff432b7e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff42c18a720 0x7ff42c0b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:14.554 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:14.554 7ff432b7e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff42c18a720 0x7ff42c0b7be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:54320/0 (socket says 172.21.2.102:54320) 2021-02-10T11:21:14.554 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:14.554 7ff432b7e700 1 -- 172.21.2.102:0/1123496951 learned_addr learned my addr 172.21.2.102:0/1123496951 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:21:14.554 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:14.555 7ff432b7e700 1 -- 172.21.2.102:0/1123496951 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff42c189d00 msgr2=0x7ff42c18a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:14.555 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:14.555 7ff432b7e700 1 --2- 172.21.2.102:0/1123496951 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff42c189d00 0x7ff42c18a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:21:14.555 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:14.555 7ff432b7e700 1 -- 172.21.2.102:0/1123496951 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff42c0a0420 msgr2=0x7ff42c0a0820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:14.555 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:14.555 7ff432b7e700 1 --2- 172.21.2.102:0/1123496951 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff42c0a0420 0x7ff42c0a0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:21:14.555 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:14.555 7ff432b7e700 1 -- 172.21.2.102:0/1123496951 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff42c0b8200 con 0x7ff42c18a720 2021-02-10T11:21:14.555 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:14.555 7ff432b7e700 1 --2- 172.21.2.102:0/1123496951 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff42c18a720 0x7ff42c0b7be0 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 rx=0x7ff424006e60 tx=0x7ff42400bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:14.555 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:14.555 7ff422ffd700 1 -- 172.21.2.102:0/1123496951 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff424009130 con 0x7ff42c18a720 2021-02-10T11:21:14.555 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:14.555 7ff422ffd700 1 -- 172.21.2.102:0/1123496951 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff424000e80 con 0x7ff42c18a720 2021-02-10T11:21:14.555 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:14.555 7ff422ffd700 1 -- 172.21.2.102:0/1123496951 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff424001390 con 0x7ff42c18a720 2021-02-10T11:21:14.555 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:14.555 7ff434561700 1 -- 172.21.2.102:0/1123496951 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff42c0b8200 con 0x7ff42c18a720 2021-02-10T11:21:14.555 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:14.555 7ff434561700 1 -- 172.21.2.102:0/1123496951 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff42c0b8790 con 0x7ff42c18a720 2021-02-10T11:21:14.555 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:14.556 7ff422ffd700 1 -- 172.21.2.102:0/1123496951 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ff4240011a0 con 0x7ff42c18a720 2021-02-10T11:21:14.556 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:14.556 7ff422ffd700 1 --2- 172.21.2.102:0/1123496951 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff4140186c0 0x7ff41401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:14.556 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:14.556 7ff422ffd700 1 -- 172.21.2.102:0/1123496951 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7ff4240329c0 con 0x7ff42c18a720 2021-02-10T11:21:14.556 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:14.556 7ff43237d700 1 --2- 172.21.2.102:0/1123496951 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff4140186c0 0x7ff41401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:14.556 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:14.556 7ff43237d700 1 --2- 172.21.2.102:0/1123496951 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff4140186c0 0x7ff41401abc0 crc :-1 s=READY pgs=432 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:14.582 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:14.586 7ff434561700 1 -- 172.21.2.102:0/1123496951 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff42c105ae0 con 0x7ff42c18a720 2021-02-10T11:21:14.584 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:14.589 7ff422ffd700 1 -- 172.21.2.102:0/1123496951 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ff4240093d0 con 0x7ff42c18a720 2021-02-10T11:21:14.715 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:14.720 7ff434561700 1 -- 172.21.2.102:0/1123496951 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344409", "target": ["mgr", ""]}) v1 -- 0x7ff42c0b9020 con 0x7ff4140186c0 2021-02-10T11:21:14.724 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:14.729 7ff422ffd700 1 -- 172.21.2.102:0/1123496951 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7ff428006a20 con 0x7ff4140186c0 2021-02-10T11:21:14.732 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:14.737 7ff434561700 1 -- 172.21.2.102:0/1123496951 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff4140186c0 msgr2=0x7ff41401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:14.733 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:14.737 7ff434561700 1 --2- 172.21.2.102:0/1123496951 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff4140186c0 0x7ff41401abc0 crc :-1 s=READY pgs=432 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:14.733 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:14.737 7ff434561700 1 -- 172.21.2.102:0/1123496951 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff42c18a720 msgr2=0x7ff42c0b7be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:14.733 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:14.737 7ff434561700 1 --2- 172.21.2.102:0/1123496951 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff42c18a720 0x7ff42c0b7be0 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 rx=0x7ff424006e60 tx=0x7ff42400bcf0).stop 2021-02-10T11:21:14.733 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:14.737 7ff434561700 1 -- 172.21.2.102:0/1123496951 shutdown_connections 2021-02-10T11:21:14.733 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:14.737 7ff434561700 1 --2- 172.21.2.102:0/1123496951 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff42c18a720 0x7ff42c0b7be0 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:14.733 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:14.737 7ff434561700 1 --2- 172.21.2.102:0/1123496951 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff42c189d00 0x7ff42c18a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:21:14.733 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:14.737 7ff434561700 1 --2- 172.21.2.102:0/1123496951 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff4140186c0 0x7ff41401abc0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:14.733 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:14.737 7ff434561700 1 --2- 172.21.2.102:0/1123496951 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff42c0a0420 0x7ff42c0a0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:21:14.733 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:14.737 7ff434561700 1 -- 172.21.2.102:0/1123496951 >> 172.21.2.102:0/1123496951 conn(0x7ff42c094e10 msgr2=0x7ff42c0a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:14.733 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:14.737 7ff434561700 1 -- 172.21.2.102:0/1123496951 shutdown_connections 2021-02-10T11:21:14.733 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:14.737 7ff434561700 1 -- 172.21.2.102:0/1123496951 wait complete. 2021-02-10T11:21:14.742 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000344410 2021-02-10T11:21:15.175 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:15.163 7f49327ca700 1 Processor -- start 2021-02-10T11:21:15.175 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:15.163 7f49327ca700 1 -- start start 2021-02-10T11:21:15.175 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:15.163 7f49327ca700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f492c1b0870 0x7f492c0f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:15.175 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:15.163 7f49327ca700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f492c0efe00 0x7f492c0f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:15.175 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:15.163 7f49327ca700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f492c1b0c70 con 0x7f492c1b0470 2021-02-10T11:21:15.175 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:15.163 7f49327ca700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f492c0f0b50 con 0x7f492c0efe00 2021-02-10T11:21:15.175 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:15.163 7f49327ca700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f492c0f0820 con 0x7f492c1b0870 2021-02-10T11:21:15.175 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:15.164 7f492bfff700 1 --1- >> v1:172.21.2.113:6789/0 conn(0x7f492c1b0470 0x7f492c0ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.113:6789/0 says I am v1:172.21.2.102:53874/0 (socket says 172.21.2.102:53874) 2021-02-10T11:21:15.176 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:15.164 7f4930de7700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f492c0efe00 0x7f492c0f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:15.176 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:15.164 7f492bfff700 1 -- 172.21.2.102:0/3269511519 learned_addr learned my addr 172.21.2.102:0/3269511519 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:21:15.176 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:15.164 7f492b7fe700 1 --2- 172.21.2.102:0/3269511519 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f492c1b0870 0x7f492c0f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:15.176 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:15.164 7f4930de7700 1 -- 172.21.2.102:0/3269511519 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f492c1b0870 msgr2=0x7f492c0f20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:21:15.176 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:15.164 7f4930de7700 1 --2- 172.21.2.102:0/3269511519 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f492c1b0870 0x7f492c0f20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:15.176 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:15.164 7f4930de7700 1 -- 172.21.2.102:0/3269511519 >> v1:172.21.2.113:6789/0 conn(0x7f492c1b0470 legacy=0x7f492c0ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:15.176 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:15.164 7f4930de7700 1 -- 172.21.2.102:0/3269511519 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f492c0d2fb0 con 0x7f492c0efe00 2021-02-10T11:21:15.176 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:15.164 7f4930de7700 1 --2- 172.21.2.102:0/3269511519 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f492c0efe00 0x7f492c0f0200 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 rx=0x7f4924006e50 tx=0x7f4924009e70).ready entity=mon.1 client_cookie=53765f4db2fdd2c2 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:15.176 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:15.165 7f492affd700 1 -- 172.21.2.102:0/3269511519 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4924006a20 con 0x7f492c0efe00 2021-02-10T11:21:15.176 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:15.165 7f492affd700 1 -- 172.21.2.102:0/3269511519 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4924006790 con 0x7f492c0efe00 2021-02-10T11:21:15.176 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:15.165 7f492affd700 1 -- 172.21.2.102:0/3269511519 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f492400b700 con 0x7f492c0efe00 2021-02-10T11:21:15.176 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:15.165 7f49327ca700 1 -- 172.21.2.102:0/3269511519 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f492c0efe00 msgr2=0x7f492c0f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:15.177 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:15.165 7f49327ca700 1 --2- 172.21.2.102:0/3269511519 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f492c0efe00 0x7f492c0f0200 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 rx=0x7f4924006e50 tx=0x7f4924009e70).stop 2021-02-10T11:21:15.177 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:15.165 7f49327ca700 1 -- 172.21.2.102:0/3269511519 shutdown_connections 2021-02-10T11:21:15.177 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:15.165 7f49327ca700 1 --2- 172.21.2.102:0/3269511519 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f492c1b0870 0x7f492c0f20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:15.177 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:15.165 7f49327ca700 1 --2- 172.21.2.102:0/3269511519 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f492c0efe00 0x7f492c0f0200 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:15.177 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:15.165 7f49327ca700 1 -- 172.21.2.102:0/3269511519 >> 172.21.2.102:0/3269511519 conn(0x7f492c0ce3e0 msgr2=0x7f492c0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:15.177 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:15.165 7f49327ca700 1 -- 172.21.2.102:0/3269511519 shutdown_connections 2021-02-10T11:21:15.177 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:15.165 7f49327ca700 1 -- 172.21.2.102:0/3269511519 wait complete. 2021-02-10T11:21:15.179 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:15.180 7f49327ca700 1 Processor -- start 2021-02-10T11:21:15.179 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:15.180 7f49327ca700 1 -- start start 2021-02-10T11:21:15.179 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:15.180 7f49327ca700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f492c190920 0x7f492c0b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:15.179 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:15.180 7f49327ca700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f492c0b8200 0x7f492c0b8600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:15.180 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:15.180 7f49327ca700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f492c0c1d80 0x7f492c0c2180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:15.180 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:15.180 7f49327ca700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f492c190d20 con 0x7f492c0c1d80 2021-02-10T11:21:15.180 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:15.180 7f49327ca700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f492c0b8c20 con 0x7f492c0b8200 2021-02-10T11:21:15.180 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:15.180 7f49327ca700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f492c0c27a0 con 0x7f492c190920 2021-02-10T11:21:15.180 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:15.180 7f4930de7700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f492c0c1d80 0x7f492c0c2180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:15.180 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:15.180 7f4930de7700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f492c0c1d80 0x7f492c0c2180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:54332/0 (socket says 172.21.2.102:54332) 2021-02-10T11:21:15.180 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:15.180 7f4930de7700 1 -- 172.21.2.102:0/907760574 learned_addr learned my addr 172.21.2.102:0/907760574 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:21:15.180 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:15.181 7f492bfff700 1 --2- 172.21.2.102:0/907760574 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f492c190920 0x7f492c0b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:15.180 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:15.181 7f492b7fe700 1 --2- 172.21.2.102:0/907760574 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f492c0b8200 0x7f492c0b8600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:15.180 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:15.181 7f4930de7700 1 -- 172.21.2.102:0/907760574 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f492c190920 msgr2=0x7f492c0b7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:15.180 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:15.181 7f4930de7700 1 --2- 172.21.2.102:0/907760574 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f492c190920 0x7f492c0b7be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:15.181 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:15.181 7f4930de7700 1 -- 172.21.2.102:0/907760574 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f492c0b8200 msgr2=0x7f492c0b8600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:15.181 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:15.181 7f4930de7700 1 --2- 172.21.2.102:0/907760574 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f492c0b8200 0x7f492c0b8600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:15.181 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:15.181 7f4930de7700 1 -- 172.21.2.102:0/907760574 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f492c0c29c0 con 0x7f492c0c1d80 2021-02-10T11:21:15.181 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:15.181 7f492b7fe700 1 --2- 172.21.2.102:0/907760574 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f492c0b8200 0x7f492c0b8600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:21:15.181 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:15.181 7f4930de7700 1 --2- 172.21.2.102:0/907760574 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f492c0c1d80 0x7f492c0c2180 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 rx=0x7f4924006e60 tx=0x7f492400bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:15.181 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:15.181 7f4928ff9700 1 -- 172.21.2.102:0/907760574 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4924009130 con 0x7f492c0c1d80 2021-02-10T11:21:15.181 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:15.181 7f4928ff9700 1 -- 172.21.2.102:0/907760574 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4924000df0 con 0x7f492c0c1d80 2021-02-10T11:21:15.181 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:15.181 7f4928ff9700 1 -- 172.21.2.102:0/907760574 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4924001390 con 0x7f492c0c1d80 2021-02-10T11:21:15.181 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:15.181 7f49327ca700 1 -- 172.21.2.102:0/907760574 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f492c0c29c0 con 0x7f492c0c1d80 2021-02-10T11:21:15.181 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:15.181 7f49327ca700 1 -- 172.21.2.102:0/907760574 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f492c0c29c0 con 0x7f492c0c1d80 2021-02-10T11:21:15.181 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:15.182 7f4928ff9700 1 -- 172.21.2.102:0/907760574 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4924009130 con 0x7f492c0c1d80 2021-02-10T11:21:15.181 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:15.182 7f4928ff9700 1 --2- 172.21.2.102:0/907760574 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f490c0186c0 0x7f490c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:15.182 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:15.182 7f4928ff9700 1 -- 172.21.2.102:0/907760574 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f4924032990 con 0x7f492c0c1d80 2021-02-10T11:21:15.182 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:15.182 7f492bfff700 1 --2- 172.21.2.102:0/907760574 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f490c0186c0 0x7f490c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:15.182 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:15.182 7f492bfff700 1 --2- 172.21.2.102:0/907760574 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f490c0186c0 0x7f490c01abc0 crc :-1 s=READY pgs=433 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:15.206 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:15.211 7f49327ca700 1 -- 172.21.2.102:0/907760574 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f492c105ae0 con 0x7f492c0c1d80 2021-02-10T11:21:15.209 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:15.214 7f4928ff9700 1 -- 172.21.2.102:0/907760574 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f492400be30 con 0x7f492c0c1d80 2021-02-10T11:21:15.338 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:15.342 7f49327ca700 1 -- 172.21.2.102:0/907760574 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344410", "target": ["mgr", ""]}) v1 -- 0x7f492c105ae0 con 0x7f490c0186c0 2021-02-10T11:21:15.348 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:15.353 7f4928ff9700 1 -- 172.21.2.102:0/907760574 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f491c007ce0 con 0x7f490c0186c0 2021-02-10T11:21:15.354 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:15.359 7f49327ca700 1 -- 172.21.2.102:0/907760574 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f490c0186c0 msgr2=0x7f490c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:15.354 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:15.359 7f49327ca700 1 --2- 172.21.2.102:0/907760574 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f490c0186c0 0x7f490c01abc0 crc :-1 s=READY pgs=433 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:15.355 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:15.359 7f49327ca700 1 -- 172.21.2.102:0/907760574 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f492c0c1d80 msgr2=0x7f492c0c2180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:15.355 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:15.359 7f49327ca700 1 --2- 172.21.2.102:0/907760574 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f492c0c1d80 0x7f492c0c2180 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 rx=0x7f4924006e60 tx=0x7f492400bcf0).stop 2021-02-10T11:21:15.355 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:15.359 7f49327ca700 1 -- 172.21.2.102:0/907760574 shutdown_connections 2021-02-10T11:21:15.355 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:15.359 7f49327ca700 1 --2- 172.21.2.102:0/907760574 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f492c0c1d80 0x7f492c0c2180 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:15.355 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:15.359 7f49327ca700 1 --2- 172.21.2.102:0/907760574 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f490c0186c0 0x7f490c01abc0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:15.355 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:15.359 7f49327ca700 1 --2- 172.21.2.102:0/907760574 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f492c0b8200 0x7f492c0b8600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:15.355 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:15.359 7f49327ca700 1 --2- 172.21.2.102:0/907760574 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f492c190920 0x7f492c0b7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:15.355 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:15.359 7f49327ca700 1 -- 172.21.2.102:0/907760574 >> 172.21.2.102:0/907760574 conn(0x7f492c094e10 msgr2=0x7f492c0a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:15.356 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:15.360 7f49327ca700 1 -- 172.21.2.102:0/907760574 shutdown_connections 2021-02-10T11:21:15.356 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:15.360 7f49327ca700 1 -- 172.21.2.102:0/907760574 wait complete. 2021-02-10T11:21:15.363 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000344411 2021-02-10T11:21:15.794 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:15.782 7f9530083700 1 Processor -- start 2021-02-10T11:21:15.794 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:15.782 7f9530083700 1 -- start start 2021-02-10T11:21:15.794 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:15.783 7f9530083700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f95281b0870 0x7f95280f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:15.795 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:15.783 7f9530083700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f95280efe00 0x7f95280f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:15.795 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:15.783 7f9530083700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95281b0c70 con 0x7f95281b0470 2021-02-10T11:21:15.795 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:15.783 7f9530083700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95280e5ea0 con 0x7f95281b0870 2021-02-10T11:21:15.795 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:15.783 7f9530083700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95280f0b50 con 0x7f95280efe00 2021-02-10T11:21:15.795 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:15.783 7f952d69e700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f95281b0870 0x7f95280f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:15.795 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:15.783 7f952d69e700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f95281b0870 0x7f95280f20e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:54340/0 (socket says 172.21.2.102:54340) 2021-02-10T11:21:15.795 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:15.783 7f952d69e700 1 -- 172.21.2.102:0/308548706 learned_addr learned my addr 172.21.2.102:0/308548706 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:21:15.795 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:15.783 7f952e6a0700 1 --2- 172.21.2.102:0/308548706 >> v2:172.21.2.113:3300/0 conn(0x7f95280efe00 0x7f95280f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:15.795 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:15.783 7f952d69e700 1 -- 172.21.2.102:0/308548706 >> v2:172.21.2.113:3300/0 conn(0x7f95280efe00 msgr2=0x7f95280f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:21:15.795 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:15.783 7f952d69e700 1 --2- 172.21.2.102:0/308548706 >> v2:172.21.2.113:3300/0 conn(0x7f95280efe00 0x7f95280f0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:15.795 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:15.783 7f952d69e700 1 -- 172.21.2.102:0/308548706 >> v1:172.21.2.113:6789/0 conn(0x7f95281b0470 legacy=0x7f95280ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:15.796 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:15.783 7f952d69e700 1 -- 172.21.2.102:0/308548706 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95280f0820 con 0x7f95281b0870 2021-02-10T11:21:15.796 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:15.783 7f952e6a0700 1 --2- 172.21.2.102:0/308548706 >> v2:172.21.2.113:3300/0 conn(0x7f95280efe00 0x7f95280f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:21:15.796 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:15.783 7f952d69e700 1 --2- 172.21.2.102:0/308548706 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f95281b0870 0x7f95280f20e0 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 rx=0x7f9524006700 tx=0x7f9524009b80).ready entity=mon.0 client_cookie=2f894cb0fff2861 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:15.796 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:15.783 7f952ce9d700 1 -- 172.21.2.102:0/308548706 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f952400c0c0 con 0x7f95281b0870 2021-02-10T11:21:15.796 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:15.783 7f952ce9d700 1 -- 172.21.2.102:0/308548706 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f952400c340 con 0x7f95281b0870 2021-02-10T11:21:15.796 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:15.783 7f952ce9d700 1 -- 172.21.2.102:0/308548706 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f952400c860 con 0x7f95281b0870 2021-02-10T11:21:15.796 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:15.784 7f9530083700 1 -- 172.21.2.102:0/308548706 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f95281b0870 msgr2=0x7f95280f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:15.796 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:15.784 7f9530083700 1 --2- 172.21.2.102:0/308548706 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f95281b0870 0x7f95280f20e0 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 rx=0x7f9524006700 tx=0x7f9524009b80).stop 2021-02-10T11:21:15.796 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:15.784 7f9530083700 1 -- 172.21.2.102:0/308548706 shutdown_connections 2021-02-10T11:21:15.796 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:15.784 7f9530083700 1 --2- 172.21.2.102:0/308548706 >> v2:172.21.2.113:3300/0 conn(0x7f95280efe00 0x7f95280f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:15.796 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:15.784 7f9530083700 1 --2- 172.21.2.102:0/308548706 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f95281b0870 0x7f95280f20e0 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:15.796 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:15.784 7f9530083700 1 -- 172.21.2.102:0/308548706 >> 172.21.2.102:0/308548706 conn(0x7f95280ce3e0 msgr2=0x7f95280e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:15.797 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:15.784 7f9530083700 1 -- 172.21.2.102:0/308548706 shutdown_connections 2021-02-10T11:21:15.797 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:15.785 7f9530083700 1 -- 172.21.2.102:0/308548706 wait complete. 2021-02-10T11:21:15.798 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:15.799 7f9530083700 1 Processor -- start 2021-02-10T11:21:15.798 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:15.799 7f9530083700 1 -- start start 2021-02-10T11:21:15.798 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:15.800 7f9530083700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f95280a0420 0x7f95280a0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:15.799 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:15.800 7f9530083700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9528189d00 0x7f952818a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:15.799 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:15.800 7f9530083700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f952818a720 0x7f95280b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:15.799 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:15.800 7f9530083700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9528189800 con 0x7f9528189d00 2021-02-10T11:21:15.799 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:15.800 7f9530083700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f952818ab20 con 0x7f95280a0420 2021-02-10T11:21:15.799 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:15.800 7f9530083700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95280a1140 con 0x7f952818a720 2021-02-10T11:21:15.799 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:15.800 7f952d69e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9528189d00 0x7f952818a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:15.799 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:15.800 7f952d69e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9528189d00 0x7f952818a100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:54346/0 (socket says 172.21.2.102:54346) 2021-02-10T11:21:15.799 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:15.800 7f952d69e700 1 -- 172.21.2.102:0/4077598873 learned_addr learned my addr 172.21.2.102:0/4077598873 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:21:15.799 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:15.800 7f952de9f700 1 --2- 172.21.2.102:0/4077598873 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f95280a0420 0x7f95280a0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:15.799 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:15.800 7f952e6a0700 1 --2- 172.21.2.102:0/4077598873 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f952818a720 0x7f95280b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:15.799 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:15.800 7f952d69e700 1 -- 172.21.2.102:0/4077598873 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f952818a720 msgr2=0x7f95280b7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:15.799 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:15.800 7f952d69e700 1 --2- 172.21.2.102:0/4077598873 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f952818a720 0x7f95280b7be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:15.800 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:15.800 7f952d69e700 1 -- 172.21.2.102:0/4077598873 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f95280a0420 msgr2=0x7f95280a0820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:15.800 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:15.800 7f952d69e700 1 --2- 172.21.2.102:0/4077598873 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f95280a0420 0x7f95280a0820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:15.800 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:15.800 7f952d69e700 1 -- 172.21.2.102:0/4077598873 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95280b8200 con 0x7f9528189d00 2021-02-10T11:21:15.800 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:15.800 7f952d69e700 1 --2- 172.21.2.102:0/4077598873 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9528189d00 0x7f952818a100 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 rx=0x7f952400b650 tx=0x7f952400b9f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:15.800 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:15.800 7f951a7fc700 1 -- 172.21.2.102:0/4077598873 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f95240090b0 con 0x7f9528189d00 2021-02-10T11:21:15.800 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:15.801 7f951a7fc700 1 -- 172.21.2.102:0/4077598873 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f952400bde0 con 0x7f9528189d00 2021-02-10T11:21:15.800 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:15.801 7f951a7fc700 1 -- 172.21.2.102:0/4077598873 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f95240022b0 con 0x7f9528189d00 2021-02-10T11:21:15.800 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:15.801 7f9530083700 1 -- 172.21.2.102:0/4077598873 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95280b8200 con 0x7f9528189d00 2021-02-10T11:21:15.800 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:15.801 7f9530083700 1 -- 172.21.2.102:0/4077598873 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95280b8200 con 0x7f9528189d00 2021-02-10T11:21:15.800 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:15.801 7f951a7fc700 1 -- 172.21.2.102:0/4077598873 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f95240052e0 con 0x7f9528189d00 2021-02-10T11:21:15.800 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:15.801 7f951a7fc700 1 --2- 172.21.2.102:0/4077598873 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f95100186c0 0x7f951001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:15.800 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:15.801 7f951a7fc700 1 -- 172.21.2.102:0/4077598873 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f95240325d0 con 0x7f9528189d00 2021-02-10T11:21:15.801 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:15.801 7f952de9f700 1 --2- 172.21.2.102:0/4077598873 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f95100186c0 0x7f951001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:15.801 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:15.802 7f952de9f700 1 --2- 172.21.2.102:0/4077598873 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f95100186c0 0x7f951001abc0 crc :-1 s=READY pgs=434 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:15.827 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:15.832 7f9530083700 1 -- 172.21.2.102:0/4077598873 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9528105ae0 con 0x7f9528189d00 2021-02-10T11:21:15.829 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:15.834 7f951a7fc700 1 -- 172.21.2.102:0/4077598873 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f9524025720 con 0x7f9528189d00 2021-02-10T11:21:15.958 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:15.963 7f9530083700 1 -- 172.21.2.102:0/4077598873 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344411", "target": ["mgr", ""]}) v1 -- 0x7f95280f0b50 con 0x7f95100186c0 2021-02-10T11:21:15.969 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:15.973 7f951a7fc700 1 -- 172.21.2.102:0/4077598873 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f951c006b60 con 0x7f95100186c0 2021-02-10T11:21:15.976 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:15.980 7f9530083700 1 -- 172.21.2.102:0/4077598873 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f95100186c0 msgr2=0x7f951001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:15.976 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:15.980 7f9530083700 1 --2- 172.21.2.102:0/4077598873 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f95100186c0 0x7f951001abc0 crc :-1 s=READY pgs=434 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:15.976 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:15.980 7f9530083700 1 -- 172.21.2.102:0/4077598873 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9528189d00 msgr2=0x7f952818a100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:15.976 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:15.980 7f9530083700 1 --2- 172.21.2.102:0/4077598873 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9528189d00 0x7f952818a100 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 rx=0x7f952400b650 tx=0x7f952400b9f0).stop 2021-02-10T11:21:15.976 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:15.980 7f9530083700 1 -- 172.21.2.102:0/4077598873 shutdown_connections 2021-02-10T11:21:15.976 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:15.980 7f9530083700 1 --2- 172.21.2.102:0/4077598873 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f952818a720 0x7f95280b7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:15.977 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:15.980 7f9530083700 1 --2- 172.21.2.102:0/4077598873 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9528189d00 0x7f952818a100 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:15.977 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:15.980 7f9530083700 1 --2- 172.21.2.102:0/4077598873 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f95100186c0 0x7f951001abc0 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:15.977 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:15.980 7f9530083700 1 --2- 172.21.2.102:0/4077598873 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f95280a0420 0x7f95280a0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:15.977 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:15.980 7f9530083700 1 -- 172.21.2.102:0/4077598873 >> 172.21.2.102:0/4077598873 conn(0x7f9528094e10 msgr2=0x7f95280a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:15.977 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:15.980 7f9530083700 1 -- 172.21.2.102:0/4077598873 shutdown_connections 2021-02-10T11:21:15.977 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:15.981 7f9530083700 1 -- 172.21.2.102:0/4077598873 wait complete. 2021-02-10T11:21:15.984 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000344412 2021-02-10T11:21:16.414 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:16.402 7f2ae7e28700 1 Processor -- start 2021-02-10T11:21:16.414 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:16.402 7f2ae7e28700 1 -- start start 2021-02-10T11:21:16.414 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:16.403 7f2ae7e28700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f2ae01b0470 0x7f2ae01b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:16.415 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:16.403 7f2ae7e28700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2ae00ef610 0x7f2ae00efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:16.415 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:16.403 7f2ae7e28700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f2ae00f0030 0x7f2ae00f0430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:16.415 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:16.403 7f2ae7e28700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ae00e5ea0 con 0x7f2ae01b0470 2021-02-10T11:21:16.415 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:16.403 7f2ae7e28700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ae00f0b50 con 0x7f2ae00f0030 2021-02-10T11:21:16.415 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:16.403 7f2ae7e28700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ae00e52e0 con 0x7f2ae00ef610 2021-02-10T11:21:16.415 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:16.403 7f2ae5c44700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f2ae01b0470 0x7f2ae01b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:16.415 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:16.403 7f2ae5c44700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f2ae01b0470 0x7f2ae01b0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:54354/0 (socket says 172.21.2.102:54354) 2021-02-10T11:21:16.415 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:16.403 7f2ae5c44700 1 -- 172.21.2.102:0/766064091 learned_addr learned my addr 172.21.2.102:0/766064091 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:21:16.415 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:16.403 7f2ae5443700 1 --2- 172.21.2.102:0/766064091 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2ae00ef610 0x7f2ae00efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:16.415 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:16.403 7f2ae6445700 1 --2- 172.21.2.102:0/766064091 >> v2:172.21.2.113:3300/0 conn(0x7f2ae00f0030 0x7f2ae00f0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:16.415 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:16.403 7f2ae5c44700 1 -- 172.21.2.102:0/766064091 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2ae00ef610 msgr2=0x7f2ae00efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:21:16.415 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:16.403 7f2ae5c44700 1 --2- 172.21.2.102:0/766064091 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2ae00ef610 0x7f2ae00efa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:16.415 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:16.403 7f2ae5c44700 1 -- 172.21.2.102:0/766064091 >> v2:172.21.2.113:3300/0 conn(0x7f2ae00f0030 msgr2=0x7f2ae00f0430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:21:16.416 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:16.403 7f2ae5c44700 1 --2- 172.21.2.102:0/766064091 >> v2:172.21.2.113:3300/0 conn(0x7f2ae00f0030 0x7f2ae00f0430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:16.416 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:16.403 7f2ae5c44700 1 -- 172.21.2.102:0/766064091 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ae00f10d0 con 0x7f2ae01b0470 2021-02-10T11:21:16.416 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:16.404 7f2ae5c44700 1 --2- 172.21.2.102:0/766064091 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2ae01b0470 0x7f2ae01b0870 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 rx=0x7f2ad0006700 tx=0x7f2ad0009b80).ready entity=mon.0 client_cookie=f9be4916906b7650 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:16.416 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:16.404 7f2ae4c42700 1 -- 172.21.2.102:0/766064091 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2ad000c0c0 con 0x7f2ae01b0470 2021-02-10T11:21:16.416 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:16.404 7f2ae4c42700 1 -- 172.21.2.102:0/766064091 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ad000c340 con 0x7f2ae01b0470 2021-02-10T11:21:16.416 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:16.404 7f2ae4c42700 1 -- 172.21.2.102:0/766064091 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2ad000c860 con 0x7f2ae01b0470 2021-02-10T11:21:16.416 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:16.404 7f2ae7e28700 1 -- 172.21.2.102:0/766064091 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2ae01b0470 msgr2=0x7f2ae01b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:16.416 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:16.404 7f2ae7e28700 1 --2- 172.21.2.102:0/766064091 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2ae01b0470 0x7f2ae01b0870 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 rx=0x7f2ad0006700 tx=0x7f2ad0009b80).stop 2021-02-10T11:21:16.416 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:16.404 7f2ae7e28700 1 -- 172.21.2.102:0/766064091 shutdown_connections 2021-02-10T11:21:16.416 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:16.404 7f2ae7e28700 1 --2- 172.21.2.102:0/766064091 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2ae00ef610 0x7f2ae00efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:16.416 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:16.405 7f2ae7e28700 1 --2- 172.21.2.102:0/766064091 >> v2:172.21.2.113:3300/0 conn(0x7f2ae00f0030 0x7f2ae00f0430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:16.416 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:16.405 7f2ae7e28700 1 --2- 172.21.2.102:0/766064091 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2ae01b0470 0x7f2ae01b0870 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:16.416 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:16.405 7f2ae7e28700 1 -- 172.21.2.102:0/766064091 >> 172.21.2.102:0/766064091 conn(0x7f2ae00ce3e0 msgr2=0x7f2ae00e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:16.417 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:16.405 7f2ae7e28700 1 -- 172.21.2.102:0/766064091 shutdown_connections 2021-02-10T11:21:16.417 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:16.405 7f2ae7e28700 1 -- 172.21.2.102:0/766064091 wait complete. 2021-02-10T11:21:16.418 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:16.419 7f2ae7e28700 1 Processor -- start 2021-02-10T11:21:16.418 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:16.419 7f2ae7e28700 1 -- start start 2021-02-10T11:21:16.418 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:16.420 7f2ae7e28700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2ae00924b0 0x7f2ae01a3590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:16.418 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:16.420 7f2ae7e28700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2ae01a3bb0 0x7f2ae00bf920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:16.418 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:16.420 7f2ae7e28700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2ae00bff40 0x7f2ae00c0340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:16.418 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:16.420 7f2ae7e28700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ae009a0a0 con 0x7f2ae01a3bb0 2021-02-10T11:21:16.419 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:16.420 7f2ae7e28700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ae00c0960 con 0x7f2ae00924b0 2021-02-10T11:21:16.419 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:16.420 7f2ae7e28700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ae0098420 con 0x7f2ae00bff40 2021-02-10T11:21:16.419 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:16.420 7f2ae5443700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2ae01a3bb0 0x7f2ae00bf920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:16.419 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:16.420 7f2ae5443700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2ae01a3bb0 0x7f2ae00bf920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:54362/0 (socket says 172.21.2.102:54362) 2021-02-10T11:21:16.419 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:16.420 7f2ae5443700 1 -- 172.21.2.102:0/1650273154 learned_addr learned my addr 172.21.2.102:0/1650273154 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:21:16.419 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:16.420 7f2ae6445700 1 --2- 172.21.2.102:0/1650273154 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2ae00bff40 0x7f2ae00c0340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:16.419 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:16.420 7f2ae5c44700 1 --2- 172.21.2.102:0/1650273154 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2ae00924b0 0x7f2ae01a3590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:16.419 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:16.420 7f2ae5443700 1 -- 172.21.2.102:0/1650273154 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2ae00bff40 msgr2=0x7f2ae00c0340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:16.419 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:16.420 7f2ae5443700 1 --2- 172.21.2.102:0/1650273154 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2ae00bff40 0x7f2ae00c0340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:16.419 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:16.420 7f2ae5443700 1 -- 172.21.2.102:0/1650273154 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2ae00924b0 msgr2=0x7f2ae01a3590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:16.419 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:16.420 7f2ae5443700 1 --2- 172.21.2.102:0/1650273154 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2ae00924b0 0x7f2ae01a3590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:16.419 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:16.420 7f2ae5443700 1 -- 172.21.2.102:0/1650273154 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ae0098640 con 0x7f2ae01a3bb0 2021-02-10T11:21:16.419 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:16.421 7f2ae5443700 1 --2- 172.21.2.102:0/1650273154 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2ae01a3bb0 0x7f2ae00bf920 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 rx=0x7f2ad80099e0 tx=0x7f2ad800bd40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:16.420 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:16.421 7f2ad67fc700 1 -- 172.21.2.102:0/1650273154 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2ad800c2f0 con 0x7f2ae01a3bb0 2021-02-10T11:21:16.420 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:16.421 7f2ad67fc700 1 -- 172.21.2.102:0/1650273154 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ad800c5f0 con 0x7f2ae01a3bb0 2021-02-10T11:21:16.420 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:16.421 7f2ad67fc700 1 -- 172.21.2.102:0/1650273154 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2ad800ca50 con 0x7f2ae01a3bb0 2021-02-10T11:21:16.420 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:16.421 7f2ae7e28700 1 -- 172.21.2.102:0/1650273154 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ae0098640 con 0x7f2ae01a3bb0 2021-02-10T11:21:16.420 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:16.421 7f2ae7e28700 1 -- 172.21.2.102:0/1650273154 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ae0098640 con 0x7f2ae01a3bb0 2021-02-10T11:21:16.420 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:16.422 7f2ad67fc700 1 -- 172.21.2.102:0/1650273154 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f2ad800a020 con 0x7f2ae01a3bb0 2021-02-10T11:21:16.420 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:16.422 7f2ad67fc700 1 --2- 172.21.2.102:0/1650273154 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2ac80186c0 0x7f2ac801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:16.420 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:16.422 7f2ad67fc700 1 -- 172.21.2.102:0/1650273154 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f2ad80328f0 con 0x7f2ae01a3bb0 2021-02-10T11:21:16.420 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:16.422 7f2ae5c44700 1 --2- 172.21.2.102:0/1650273154 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2ac80186c0 0x7f2ac801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:16.421 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:16.422 7f2ae5c44700 1 --2- 172.21.2.102:0/1650273154 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2ac80186c0 0x7f2ac801abc0 crc :-1 s=READY pgs=435 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:16.448 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:16.452 7f2ae7e28700 1 -- 172.21.2.102:0/1650273154 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ae0105b40 con 0x7f2ae01a3bb0 2021-02-10T11:21:16.450 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:16.455 7f2ad67fc700 1 -- 172.21.2.102:0/1650273154 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f2ad800c2f0 con 0x7f2ae01a3bb0 2021-02-10T11:21:16.578 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:16.582 7f2ae7e28700 1 -- 172.21.2.102:0/1650273154 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344412", "target": ["mgr", ""]}) v1 -- 0x7f2ae00c0960 con 0x7f2ac80186c0 2021-02-10T11:21:16.589 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:16.594 7f2ad67fc700 1 -- 172.21.2.102:0/1650273154 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f2ad00090b0 con 0x7f2ac80186c0 2021-02-10T11:21:16.594 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:16.599 7f2ae7e28700 1 -- 172.21.2.102:0/1650273154 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2ac80186c0 msgr2=0x7f2ac801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:16.594 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:16.599 7f2ae7e28700 1 --2- 172.21.2.102:0/1650273154 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2ac80186c0 0x7f2ac801abc0 crc :-1 s=READY pgs=435 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:16.594 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:16.599 7f2ae7e28700 1 -- 172.21.2.102:0/1650273154 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2ae01a3bb0 msgr2=0x7f2ae00bf920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:16.595 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:16.599 7f2ae7e28700 1 --2- 172.21.2.102:0/1650273154 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2ae01a3bb0 0x7f2ae00bf920 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 rx=0x7f2ad80099e0 tx=0x7f2ad800bd40).stop 2021-02-10T11:21:16.595 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:16.599 7f2ae7e28700 1 -- 172.21.2.102:0/1650273154 shutdown_connections 2021-02-10T11:21:16.595 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:16.599 7f2ae7e28700 1 --2- 172.21.2.102:0/1650273154 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2ae00bff40 0x7f2ae00c0340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:16.595 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:16.599 7f2ae7e28700 1 --2- 172.21.2.102:0/1650273154 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2ae01a3bb0 0x7f2ae00bf920 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:16.595 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:16.599 7f2ae7e28700 1 --2- 172.21.2.102:0/1650273154 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2ac80186c0 0x7f2ac801abc0 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:16.595 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:16.599 7f2ae7e28700 1 --2- 172.21.2.102:0/1650273154 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2ae00924b0 0x7f2ae01a3590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:16.595 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:16.599 7f2ae7e28700 1 -- 172.21.2.102:0/1650273154 >> 172.21.2.102:0/1650273154 conn(0x7f2ae00b50c0 msgr2=0x7f2ae0092d90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:16.595 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:16.599 7f2ae7e28700 1 -- 172.21.2.102:0/1650273154 shutdown_connections 2021-02-10T11:21:16.595 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:16.599 7f2ae7e28700 1 -- 172.21.2.102:0/1650273154 wait complete. 2021-02-10T11:21:16.604 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000344413 2021-02-10T11:21:17.035 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:17.024 7f3eadf7a700 1 Processor -- start 2021-02-10T11:21:17.036 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:17.024 7f3eadf7a700 1 -- start start 2021-02-10T11:21:17.036 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:17.024 7f3eadf7a700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f3ea81b0870 0x7f3ea80f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:17.036 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:17.024 7f3eadf7a700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3ea80efe00 0x7f3ea80f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:17.036 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:17.024 7f3eadf7a700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ea81b0c70 con 0x7f3ea81b0470 2021-02-10T11:21:17.036 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:17.024 7f3eadf7a700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ea80f0b50 con 0x7f3ea81b0870 2021-02-10T11:21:17.036 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:17.024 7f3eadf7a700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ea80f0820 con 0x7f3ea80efe00 2021-02-10T11:21:17.036 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:17.024 7f3e9effd700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f3ea81b0870 0x7f3ea80f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:17.036 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:17.024 7f3e9effd700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f3ea81b0870 0x7f3ea80f20e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:54368/0 (socket says 172.21.2.102:54368) 2021-02-10T11:21:17.037 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:17.024 7f3e9effd700 1 -- 172.21.2.102:0/3754952464 learned_addr learned my addr 172.21.2.102:0/3754952464 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:21:17.037 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:17.024 7f3ea7fff700 1 --2- 172.21.2.102:0/3754952464 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3ea80efe00 0x7f3ea80f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:17.037 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:17.025 7f3e9effd700 1 -- 172.21.2.102:0/3754952464 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3ea80efe00 msgr2=0x7f3ea80f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:21:17.037 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:17.025 7f3e9effd700 1 --2- 172.21.2.102:0/3754952464 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3ea80efe00 0x7f3ea80f0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:17.037 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:17.025 7f3e9effd700 1 -- 172.21.2.102:0/3754952464 >> v1:172.21.2.113:6789/0 conn(0x7f3ea81b0470 legacy=0x7f3ea80ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:17.037 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:17.025 7f3e9effd700 1 -- 172.21.2.102:0/3754952464 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ea80e52e0 con 0x7f3ea81b0870 2021-02-10T11:21:17.037 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:17.025 7f3e9effd700 1 --2- 172.21.2.102:0/3754952464 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3ea81b0870 0x7f3ea80f20e0 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 rx=0x7f3e90006700 tx=0x7f3e90009b80).ready entity=mon.0 client_cookie=32a7eec293bb91e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:17.037 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:17.025 7f3ea6ffd700 1 -- 172.21.2.102:0/3754952464 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3e9000c0c0 con 0x7f3ea81b0870 2021-02-10T11:21:17.037 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:17.025 7f3ea6ffd700 1 -- 172.21.2.102:0/3754952464 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e9000c440 con 0x7f3ea81b0870 2021-02-10T11:21:17.037 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:17.025 7f3ea6ffd700 1 -- 172.21.2.102:0/3754952464 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3e9000c8a0 con 0x7f3ea81b0870 2021-02-10T11:21:17.037 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:17.025 7f3eadf7a700 1 -- 172.21.2.102:0/3754952464 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3ea81b0870 msgr2=0x7f3ea80f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:17.037 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:17.025 7f3eadf7a700 1 --2- 172.21.2.102:0/3754952464 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3ea81b0870 0x7f3ea80f20e0 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 rx=0x7f3e90006700 tx=0x7f3e90009b80).stop 2021-02-10T11:21:17.038 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:17.026 7f3eadf7a700 1 -- 172.21.2.102:0/3754952464 shutdown_connections 2021-02-10T11:21:17.038 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:17.026 7f3eadf7a700 1 --2- 172.21.2.102:0/3754952464 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3ea80efe00 0x7f3ea80f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:17.038 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:17.026 7f3eadf7a700 1 --2- 172.21.2.102:0/3754952464 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3ea81b0870 0x7f3ea80f20e0 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:17.038 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:17.026 7f3eadf7a700 1 -- 172.21.2.102:0/3754952464 >> 172.21.2.102:0/3754952464 conn(0x7f3ea80ce3e0 msgr2=0x7f3ea80e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:17.038 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:17.026 7f3eadf7a700 1 -- 172.21.2.102:0/3754952464 shutdown_connections 2021-02-10T11:21:17.038 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:17.026 7f3eadf7a700 1 -- 172.21.2.102:0/3754952464 wait complete. 2021-02-10T11:21:17.038 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:17.040 7f3eadf7a700 1 Processor -- start 2021-02-10T11:21:17.040 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:17.040 7f3eadf7a700 1 -- start start 2021-02-10T11:21:17.040 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:17.041 7f3eadf7a700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3ea80a0420 0x7f3ea80a0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:17.040 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:17.041 7f3eadf7a700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3ea8189d00 0x7f3ea818a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:17.041 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:17.041 7f3eadf7a700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3ea818a720 0x7f3ea80b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:17.041 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:17.041 7f3eadf7a700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ea8189800 con 0x7f3ea80a0420 2021-02-10T11:21:17.041 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:17.041 7f3eadf7a700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ea818ab20 con 0x7f3ea8189d00 2021-02-10T11:21:17.041 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:17.041 7f3eadf7a700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ea80a1140 con 0x7f3ea818a720 2021-02-10T11:21:17.041 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:17.041 7f3ea77fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3ea80a0420 0x7f3ea80a0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:17.041 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:17.041 7f3ea77fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3ea80a0420 0x7f3ea80a0820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:54374/0 (socket says 172.21.2.102:54374) 2021-02-10T11:21:17.041 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:17.041 7f3ea77fe700 1 -- 172.21.2.102:0/211562907 learned_addr learned my addr 172.21.2.102:0/211562907 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:21:17.041 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:17.041 7f3ea77fe700 1 -- 172.21.2.102:0/211562907 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3ea818a720 msgr2=0x7f3ea80b7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:17.041 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:17.041 7f3ea77fe700 1 --2- 172.21.2.102:0/211562907 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3ea818a720 0x7f3ea80b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:21:17.041 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:17.041 7f3ea77fe700 1 -- 172.21.2.102:0/211562907 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3ea8189d00 msgr2=0x7f3ea818a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:17.041 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:17.041 7f3e9effd700 1 --2- 172.21.2.102:0/211562907 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3ea8189d00 0x7f3ea818a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:17.042 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:17.041 7f3ea77fe700 1 --2- 172.21.2.102:0/211562907 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3ea8189d00 0x7f3ea818a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:17.042 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:17.041 7f3ea77fe700 1 -- 172.21.2.102:0/211562907 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ea80b8200 con 0x7f3ea80a0420 2021-02-10T11:21:17.042 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:17.041 7f3e9effd700 1 --2- 172.21.2.102:0/211562907 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3ea8189d00 0x7f3ea818a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:21:17.042 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:17.042 7f3ea77fe700 1 --2- 172.21.2.102:0/211562907 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3ea80a0420 0x7f3ea80a0820 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 rx=0x7f3e98006ed0 tx=0x7f3e9800b8e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:17.042 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:17.042 7f3ea4ff9700 1 -- 172.21.2.102:0/211562907 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3e98006240 con 0x7f3ea80a0420 2021-02-10T11:21:17.042 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:17.042 7f3ea4ff9700 1 -- 172.21.2.102:0/211562907 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e980054b0 con 0x7f3ea80a0420 2021-02-10T11:21:17.042 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:17.042 7f3ea4ff9700 1 -- 172.21.2.102:0/211562907 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3e98005910 con 0x7f3ea80a0420 2021-02-10T11:21:17.042 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:17.042 7f3eadf7a700 1 -- 172.21.2.102:0/211562907 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ea80b8200 con 0x7f3ea80a0420 2021-02-10T11:21:17.042 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:17.042 7f3eadf7a700 1 -- 172.21.2.102:0/211562907 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ea80b8790 con 0x7f3ea80a0420 2021-02-10T11:21:17.042 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:17.042 7f3ea4ff9700 1 -- 172.21.2.102:0/211562907 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f3e98006030 con 0x7f3ea80a0420 2021-02-10T11:21:17.043 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:17.043 7f3ea4ff9700 1 --2- 172.21.2.102:0/211562907 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3e8c0186c0 0x7f3e8c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:17.043 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:17.043 7f3ea4ff9700 1 -- 172.21.2.102:0/211562907 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f3e980327f0 con 0x7f3ea80a0420 2021-02-10T11:21:17.043 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:17.043 7f3e9effd700 1 --2- 172.21.2.102:0/211562907 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3e8c0186c0 0x7f3e8c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:17.043 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:17.043 7f3e9effd700 1 --2- 172.21.2.102:0/211562907 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3e8c0186c0 0x7f3e8c01abc0 crc :-1 s=READY pgs=436 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:17.068 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:17.073 7f3eadf7a700 1 -- 172.21.2.102:0/211562907 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ea8105ae0 con 0x7f3ea80a0420 2021-02-10T11:21:17.071 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:17.076 7f3ea4ff9700 1 -- 172.21.2.102:0/211562907 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f3e98000dc0 con 0x7f3ea80a0420 2021-02-10T11:21:17.197 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:17.201 7f3eadf7a700 1 -- 172.21.2.102:0/211562907 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344413", "target": ["mgr", ""]}) v1 -- 0x7f3ea80b8790 con 0x7f3e8c0186c0 2021-02-10T11:21:17.209 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:17.214 7f3ea4ff9700 1 -- 172.21.2.102:0/211562907 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f3e900009b0 con 0x7f3e8c0186c0 2021-02-10T11:21:17.214 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:17.218 7f3eadf7a700 1 -- 172.21.2.102:0/211562907 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3e8c0186c0 msgr2=0x7f3e8c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:17.214 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:17.218 7f3eadf7a700 1 --2- 172.21.2.102:0/211562907 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3e8c0186c0 0x7f3e8c01abc0 crc :-1 s=READY pgs=436 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:17.214 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:17.218 7f3eadf7a700 1 -- 172.21.2.102:0/211562907 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3ea80a0420 msgr2=0x7f3ea80a0820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:17.214 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:17.218 7f3eadf7a700 1 --2- 172.21.2.102:0/211562907 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3ea80a0420 0x7f3ea80a0820 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 rx=0x7f3e98006ed0 tx=0x7f3e9800b8e0).stop 2021-02-10T11:21:17.214 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:17.218 7f3eadf7a700 1 -- 172.21.2.102:0/211562907 shutdown_connections 2021-02-10T11:21:17.214 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:17.218 7f3eadf7a700 1 --2- 172.21.2.102:0/211562907 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3ea818a720 0x7f3ea80b7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:21:17.214 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:17.218 7f3eadf7a700 1 --2- 172.21.2.102:0/211562907 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3e8c0186c0 0x7f3e8c01abc0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:17.214 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:17.218 7f3eadf7a700 1 --2- 172.21.2.102:0/211562907 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3ea8189d00 0x7f3ea818a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:17.214 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:17.218 7f3eadf7a700 1 --2- 172.21.2.102:0/211562907 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3ea80a0420 0x7f3ea80a0820 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:17.214 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:17.218 7f3eadf7a700 1 -- 172.21.2.102:0/211562907 >> 172.21.2.102:0/211562907 conn(0x7f3ea8094e10 msgr2=0x7f3ea80a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:17.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:17.218 7f3eadf7a700 1 -- 172.21.2.102:0/211562907 shutdown_connections 2021-02-10T11:21:17.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:17.218 7f3eadf7a700 1 -- 172.21.2.102:0/211562907 wait complete. 2021-02-10T11:21:17.223 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000344414 2021-02-10T11:21:17.656 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:17.644 7fbaec118700 1 Processor -- start 2021-02-10T11:21:17.656 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:17.644 7fbaec118700 1 -- start start 2021-02-10T11:21:17.656 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:17.644 7fbaec118700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fbae41b0470 0x7fbae41b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:17.656 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:17.644 7fbaec118700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbae40f0b50 con 0x7fbae40f20e0 2021-02-10T11:21:17.656 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:17.644 7fbaec118700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbae40f24e0 con 0x7fbae40ef610 2021-02-10T11:21:17.656 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:17.644 7fbaec118700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbae40e5ea0 con 0x7fbae41b0470 2021-02-10T11:21:17.657 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:17.645 7fbaea735700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fbae40f20e0 0x7fbae40f0200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:46606/0 (socket says 172.21.2.102:46606) 2021-02-10T11:21:17.657 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:17.645 7fbaea735700 1 -- 172.21.2.102:0/1225658772 learned_addr learned my addr 172.21.2.102:0/1225658772 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:21:17.657 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:17.645 7fbae9f34700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fbae41b0470 0x7fbae41b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:17.657 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:17.645 7fbae8f32700 1 -- 172.21.2.102:0/1225658772 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1625514701 0 0) 0x7fbadc000c10 con 0x7fbae40f20e0 2021-02-10T11:21:17.657 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:17.645 7fbae8f32700 1 -- 172.21.2.102:0/1225658772 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbad0002fe0 con 0x7fbae40f20e0 2021-02-10T11:21:17.657 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:17.645 7fbae9f34700 1 -- 172.21.2.102:0/1225658772 >> v1:172.21.2.113:6789/0 conn(0x7fbae40ef610 legacy=0x7fbae40efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:17.657 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:17.645 7fbae9f34700 1 -- 172.21.2.102:0/1225658772 >> v1:172.21.2.102:6789/0 conn(0x7fbae40f20e0 legacy=0x7fbae40f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:17.657 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:17.645 7fbae9f34700 1 -- 172.21.2.102:0/1225658772 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbae40e52e0 con 0x7fbae41b0470 2021-02-10T11:21:17.657 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:17.645 7fbae9f34700 1 --2- 172.21.2.102:0/1225658772 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fbae41b0470 0x7fbae41b0870 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 rx=0x7fbad80069d0 tx=0x7fbad8009e40).ready entity=mon.1 client_cookie=1f7d18421d936cad server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:17.657 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:17.646 7fbae8f32700 1 -- 172.21.2.102:0/1225658772 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbad800b510 con 0x7fbae41b0470 2021-02-10T11:21:17.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:17.646 7fbae8f32700 1 -- 172.21.2.102:0/1225658772 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbad800b810 con 0x7fbae41b0470 2021-02-10T11:21:17.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:17.646 7fbae8f32700 1 -- 172.21.2.102:0/1225658772 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbad800bca0 con 0x7fbae41b0470 2021-02-10T11:21:17.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:17.646 7fbaec118700 1 -- 172.21.2.102:0/1225658772 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fbae41b0470 msgr2=0x7fbae41b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:17.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:17.646 7fbaec118700 1 --2- 172.21.2.102:0/1225658772 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fbae41b0470 0x7fbae41b0870 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 rx=0x7fbad80069d0 tx=0x7fbad8009e40).stop 2021-02-10T11:21:17.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:17.646 7fbaec118700 1 -- 172.21.2.102:0/1225658772 shutdown_connections 2021-02-10T11:21:17.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:17.646 7fbaec118700 1 --2- 172.21.2.102:0/1225658772 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fbae41b0470 0x7fbae41b0870 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:17.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:17.646 7fbaec118700 1 -- 172.21.2.102:0/1225658772 >> 172.21.2.102:0/1225658772 conn(0x7fbae40ce3e0 msgr2=0x7fbae40e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:17.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:17.646 7fbaec118700 1 -- 172.21.2.102:0/1225658772 shutdown_connections 2021-02-10T11:21:17.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:17.646 7fbaec118700 1 -- 172.21.2.102:0/1225658772 wait complete. 2021-02-10T11:21:17.660 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:17.661 7fbaec118700 1 Processor -- start 2021-02-10T11:21:17.660 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:17.661 7fbaec118700 1 -- start start 2021-02-10T11:21:17.660 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:17.661 7fbaec118700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbae40b2500 0x7fbae40b2900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:17.661 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:17.661 7fbaec118700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fbae40b3080 0x7fbae40b3480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:17.661 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:17.661 7fbaec118700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbae40b3aa0 0x7fbae40f0530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:17.661 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:17.661 7fbaec118700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbae40effb0 con 0x7fbae40b3aa0 2021-02-10T11:21:17.661 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:17.661 7fbaec118700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbae40f0b50 con 0x7fbae40b3080 2021-02-10T11:21:17.661 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:17.661 7fbaec118700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbae40f0d30 con 0x7fbae40b2500 2021-02-10T11:21:17.661 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:17.662 7fbaea735700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbae40b3aa0 0x7fbae40f0530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:17.661 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:17.662 7fbaea735700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbae40b3aa0 0x7fbae40f0530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:54386/0 (socket says 172.21.2.102:54386) 2021-02-10T11:21:17.661 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:17.662 7fbaea735700 1 -- 172.21.2.102:0/1160075304 learned_addr learned my addr 172.21.2.102:0/1160075304 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:21:17.661 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:17.662 7fbae9733700 1 --2- 172.21.2.102:0/1160075304 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fbae40b3080 0x7fbae40b3480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:17.661 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:17.662 7fbae9f34700 1 --2- 172.21.2.102:0/1160075304 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbae40b2500 0x7fbae40b2900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:17.661 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:17.662 7fbaea735700 1 -- 172.21.2.102:0/1160075304 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbae40b2500 msgr2=0x7fbae40b2900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:17.661 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:17.662 7fbaea735700 1 --2- 172.21.2.102:0/1160075304 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbae40b2500 0x7fbae40b2900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:17.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:17.662 7fbaea735700 1 -- 172.21.2.102:0/1160075304 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fbae40b3080 msgr2=0x7fbae40b3480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:17.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:17.662 7fbaea735700 1 --2- 172.21.2.102:0/1160075304 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fbae40b3080 0x7fbae40b3480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:17.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:17.662 7fbaea735700 1 -- 172.21.2.102:0/1160075304 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbae40f0f50 con 0x7fbae40b3aa0 2021-02-10T11:21:17.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:17.662 7fbaea735700 1 --2- 172.21.2.102:0/1160075304 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbae40b3aa0 0x7fbae40f0530 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 rx=0x7fbadc00a5d0 tx=0x7fbadc00a970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:17.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:17.662 7fbad67fc700 1 -- 172.21.2.102:0/1160075304 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbadc00ad60 con 0x7fbae40b3aa0 2021-02-10T11:21:17.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:17.662 7fbad67fc700 1 -- 172.21.2.102:0/1160075304 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbadc0043a0 con 0x7fbae40b3aa0 2021-02-10T11:21:17.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:17.662 7fbaec118700 1 -- 172.21.2.102:0/1160075304 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbae40f0f50 con 0x7fbae40b3aa0 2021-02-10T11:21:17.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:17.663 7fbad67fc700 1 -- 172.21.2.102:0/1160075304 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbadc0048c0 con 0x7fbae40b3aa0 2021-02-10T11:21:17.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:17.663 7fbaec118700 1 -- 172.21.2.102:0/1160075304 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbae40f0f50 con 0x7fbae40b3aa0 2021-02-10T11:21:17.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:17.663 7fbad67fc700 1 -- 172.21.2.102:0/1160075304 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fbadc000b10 con 0x7fbae40b3aa0 2021-02-10T11:21:17.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:17.663 7fbad67fc700 1 --2- 172.21.2.102:0/1160075304 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fbacc018730 0x7fbacc01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:17.663 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:17.663 7fbad67fc700 1 -- 172.21.2.102:0/1160075304 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fbadc032780 con 0x7fbae40b3aa0 2021-02-10T11:21:17.663 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:17.663 7fbae9f34700 1 --2- 172.21.2.102:0/1160075304 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fbacc018730 0x7fbacc01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:17.663 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:17.664 7fbae9f34700 1 --2- 172.21.2.102:0/1160075304 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fbacc018730 0x7fbacc01ac10 crc :-1 s=READY pgs=437 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:17.688 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:17.693 7fbaec118700 1 -- 172.21.2.102:0/1160075304 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbae40b3ef0 con 0x7fbae40b3aa0 2021-02-10T11:21:17.691 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:17.696 7fbad67fc700 1 -- 172.21.2.102:0/1160075304 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fbadc0043a0 con 0x7fbae40b3aa0 2021-02-10T11:21:17.823 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:17.828 7fbaec118700 1 -- 172.21.2.102:0/1160075304 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344414", "target": ["mgr", ""]}) v1 -- 0x7fbae40b3ef0 con 0x7fbacc018730 2021-02-10T11:21:17.832 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:17.837 7fbad67fc700 1 -- 172.21.2.102:0/1160075304 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fbad8000d30 con 0x7fbacc018730 2021-02-10T11:21:17.840 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:17.844 7fbaec118700 1 -- 172.21.2.102:0/1160075304 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fbacc018730 msgr2=0x7fbacc01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:17.840 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:17.844 7fbaec118700 1 --2- 172.21.2.102:0/1160075304 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fbacc018730 0x7fbacc01ac10 crc :-1 s=READY pgs=437 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:17.840 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:17.844 7fbaec118700 1 -- 172.21.2.102:0/1160075304 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbae40b3aa0 msgr2=0x7fbae40f0530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:17.840 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:17.844 7fbaec118700 1 --2- 172.21.2.102:0/1160075304 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbae40b3aa0 0x7fbae40f0530 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 rx=0x7fbadc00a5d0 tx=0x7fbadc00a970).stop 2021-02-10T11:21:17.840 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:17.844 7fbaec118700 1 -- 172.21.2.102:0/1160075304 shutdown_connections 2021-02-10T11:21:17.840 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:17.844 7fbaec118700 1 --2- 172.21.2.102:0/1160075304 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbae40b3aa0 0x7fbae40f0530 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:17.840 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:17.844 7fbaec118700 1 --2- 172.21.2.102:0/1160075304 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fbacc018730 0x7fbacc01ac10 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:17.841 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:17.844 7fbaec118700 1 --2- 172.21.2.102:0/1160075304 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fbae40b3080 0x7fbae40b3480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:17.841 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:17.844 7fbaec118700 1 --2- 172.21.2.102:0/1160075304 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbae40b2500 0x7fbae40b2900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:17.841 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:17.844 7fbaec118700 1 -- 172.21.2.102:0/1160075304 >> 172.21.2.102:0/1160075304 conn(0x7fbae40c3c70 msgr2=0x7fbae4094c30 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:17.841 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:17.845 7fbaec118700 1 -- 172.21.2.102:0/1160075304 shutdown_connections 2021-02-10T11:21:17.841 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:17.845 7fbaec118700 1 -- 172.21.2.102:0/1160075304 wait complete. 2021-02-10T11:21:17.849 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000344415 2021-02-10T11:21:18.279 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:18.267 7f5066ea0700 1 Processor -- start 2021-02-10T11:21:18.279 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:18.267 7f5066ea0700 1 -- start start 2021-02-10T11:21:18.279 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:18.267 7f5066ea0700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f50600f05f0 0x7f50600f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:18.280 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:18.267 7f5066ea0700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50601b0c70 con 0x7f50601b0870 2021-02-10T11:21:18.280 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:18.267 7f5066ea0700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50600f0b50 con 0x7f50601b0470 2021-02-10T11:21:18.280 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:18.267 7f5066ea0700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50600e5ea0 con 0x7f50600f05f0 2021-02-10T11:21:18.280 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:18.268 7f505ffff700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f50601b0870 0x7f50600efe00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:46618/0 (socket says 172.21.2.102:46618) 2021-02-10T11:21:18.280 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:18.268 7f505ffff700 1 -- 172.21.2.102:0/3437673836 learned_addr learned my addr 172.21.2.102:0/3437673836 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:21:18.280 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:18.268 7f50654bd700 1 --2- 172.21.2.102:0/3437673836 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f50600f05f0 0x7f50600f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:18.280 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:18.268 7f505f7fe700 1 -- 172.21.2.102:0/3437673836 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3262866381 0 0) 0x7f5050000d30 con 0x7f50601b0870 2021-02-10T11:21:18.280 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:18.268 7f505f7fe700 1 -- 172.21.2.102:0/3437673836 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f504c002fe0 con 0x7f50601b0870 2021-02-10T11:21:18.280 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:18.268 7f505f7fe700 1 -- 172.21.2.102:0/3437673836 <== mon.1 v1:172.21.2.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 448328306 0 0) 0x7f5048000c60 con 0x7f50601b0470 2021-02-10T11:21:18.281 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:18.268 7f505f7fe700 1 -- 172.21.2.102:0/3437673836 --> v1:172.21.2.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f504c004370 con 0x7f50601b0470 2021-02-10T11:21:18.281 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:18.268 7f50654bd700 1 -- 172.21.2.102:0/3437673836 >> v1:172.21.2.113:6789/0 conn(0x7f50601b0470 legacy=0x7f50600ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:18.281 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:18.268 7f50654bd700 1 -- 172.21.2.102:0/3437673836 >> v1:172.21.2.102:6789/0 conn(0x7f50601b0870 legacy=0x7f50600efe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:18.281 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:18.268 7f50654bd700 1 -- 172.21.2.102:0/3437673836 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50600e52e0 con 0x7f50600f05f0 2021-02-10T11:21:18.281 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:18.268 7f50654bd700 1 --2- 172.21.2.102:0/3437673836 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f50600f05f0 0x7f50600f20e0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7f5058006ad0 tx=0x7f5058009ad0).ready entity=mon.2 client_cookie=583f5c7beea590f6 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:18.281 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:18.269 7f505f7fe700 1 -- 172.21.2.102:0/3437673836 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f50580022e0 con 0x7f50600f05f0 2021-02-10T11:21:18.281 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:18.269 7f505f7fe700 1 -- 172.21.2.102:0/3437673836 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5058006410 con 0x7f50600f05f0 2021-02-10T11:21:18.281 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:18.269 7f505f7fe700 1 -- 172.21.2.102:0/3437673836 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5058002620 con 0x7f50600f05f0 2021-02-10T11:21:18.281 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:18.269 7f5066ea0700 1 -- 172.21.2.102:0/3437673836 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f50600f05f0 msgr2=0x7f50600f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:18.281 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:18.269 7f5066ea0700 1 --2- 172.21.2.102:0/3437673836 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f50600f05f0 0x7f50600f20e0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7f5058006ad0 tx=0x7f5058009ad0).stop 2021-02-10T11:21:18.281 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:18.270 7f5066ea0700 1 -- 172.21.2.102:0/3437673836 shutdown_connections 2021-02-10T11:21:18.281 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:18.270 7f5066ea0700 1 --2- 172.21.2.102:0/3437673836 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f50600f05f0 0x7f50600f20e0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:18.281 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:18.270 7f5066ea0700 1 -- 172.21.2.102:0/3437673836 >> 172.21.2.102:0/3437673836 conn(0x7f50600ce3e0 msgr2=0x7f50600e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:18.282 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:18.270 7f5066ea0700 1 -- 172.21.2.102:0/3437673836 shutdown_connections 2021-02-10T11:21:18.282 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:18.270 7f5066ea0700 1 -- 172.21.2.102:0/3437673836 wait complete. 2021-02-10T11:21:18.283 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:18.284 7f5066ea0700 1 Processor -- start 2021-02-10T11:21:18.283 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:18.284 7f5066ea0700 1 -- start start 2021-02-10T11:21:18.283 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:18.285 7f5066ea0700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f50600a0420 0x7f50600a0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:18.284 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:18.285 7f5066ea0700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5060189d00 0x7f506018a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:18.284 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:18.285 7f5066ea0700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f506018a720 0x7f50600b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:18.284 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:18.285 7f5066ea0700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5060189780 con 0x7f50600a0420 2021-02-10T11:21:18.284 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:18.285 7f5066ea0700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f506018ab20 con 0x7f506018a720 2021-02-10T11:21:18.284 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:18.285 7f5066ea0700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50600a1140 con 0x7f5060189d00 2021-02-10T11:21:18.284 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:18.285 7f5064cbc700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f50600a0420 0x7f50600a0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:18.284 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:18.285 7f5064cbc700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f50600a0420 0x7f50600a0820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:54400/0 (socket says 172.21.2.102:54400) 2021-02-10T11:21:18.284 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:18.285 7f5064cbc700 1 -- 172.21.2.102:0/826678020 learned_addr learned my addr 172.21.2.102:0/826678020 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:21:18.284 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:18.285 7f50654bd700 1 --2- 172.21.2.102:0/826678020 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f506018a720 0x7f50600b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:18.284 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:18.285 7f505ffff700 1 --2- 172.21.2.102:0/826678020 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5060189d00 0x7f506018a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:18.284 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:18.285 7f5064cbc700 1 -- 172.21.2.102:0/826678020 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5060189d00 msgr2=0x7f506018a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:18.284 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:18.285 7f5064cbc700 1 --2- 172.21.2.102:0/826678020 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5060189d00 0x7f506018a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:18.285 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:18.285 7f5064cbc700 1 -- 172.21.2.102:0/826678020 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f506018a720 msgr2=0x7f50600b7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:18.285 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:18.285 7f5064cbc700 1 --2- 172.21.2.102:0/826678020 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f506018a720 0x7f50600b7be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:18.285 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:18.285 7f5064cbc700 1 -- 172.21.2.102:0/826678020 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50600b8200 con 0x7f50600a0420 2021-02-10T11:21:18.285 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:18.286 7f505ffff700 1 --2- 172.21.2.102:0/826678020 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5060189d00 0x7f506018a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:21:18.285 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:18.286 7f5064cbc700 1 --2- 172.21.2.102:0/826678020 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f50600a0420 0x7f50600a0820 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 rx=0x7f5048006f10 tx=0x7f504800b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:18.285 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:18.286 7f505d7fa700 1 -- 172.21.2.102:0/826678020 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f504800bd80 con 0x7f50600a0420 2021-02-10T11:21:18.285 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:18.286 7f505d7fa700 1 -- 172.21.2.102:0/826678020 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5048000e20 con 0x7f50600a0420 2021-02-10T11:21:18.285 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:18.286 7f505d7fa700 1 -- 172.21.2.102:0/826678020 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f50480054f0 con 0x7f50600a0420 2021-02-10T11:21:18.285 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:18.286 7f5066ea0700 1 -- 172.21.2.102:0/826678020 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50600b8200 con 0x7f50600a0420 2021-02-10T11:21:18.285 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:18.286 7f5066ea0700 1 -- 172.21.2.102:0/826678020 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50600b8200 con 0x7f50600a0420 2021-02-10T11:21:18.285 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:18.286 7f505d7fa700 1 -- 172.21.2.102:0/826678020 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f5048009fe0 con 0x7f50600a0420 2021-02-10T11:21:18.285 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:18.286 7f505d7fa700 1 --2- 172.21.2.102:0/826678020 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f50440186c0 0x7f504401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:18.285 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:18.286 7f505d7fa700 1 -- 172.21.2.102:0/826678020 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f5048032660 con 0x7f50600a0420 2021-02-10T11:21:18.286 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:18.286 7f505ffff700 1 --2- 172.21.2.102:0/826678020 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f50440186c0 0x7f504401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:18.286 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:18.287 7f505ffff700 1 --2- 172.21.2.102:0/826678020 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f50440186c0 0x7f504401abc0 crc :-1 s=READY pgs=438 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:18.311 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:18.316 7f5066ea0700 1 -- 172.21.2.102:0/826678020 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5060105ae0 con 0x7f50600a0420 2021-02-10T11:21:18.313 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:18.318 7f505d7fa700 1 -- 172.21.2.102:0/826678020 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f5048000e20 con 0x7f50600a0420 2021-02-10T11:21:18.443 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:18.448 7f5066ea0700 1 -- 172.21.2.102:0/826678020 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344415", "target": ["mgr", ""]}) v1 -- 0x7f5060189780 con 0x7f50440186c0 2021-02-10T11:21:18.453 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:18.458 7f505d7fa700 1 -- 172.21.2.102:0/826678020 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f5050000d30 con 0x7f50440186c0 2021-02-10T11:21:18.460 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:18.464 7f5066ea0700 1 -- 172.21.2.102:0/826678020 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f50440186c0 msgr2=0x7f504401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:18.461 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:18.464 7f5066ea0700 1 --2- 172.21.2.102:0/826678020 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f50440186c0 0x7f504401abc0 crc :-1 s=READY pgs=438 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:18.461 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:18.464 7f5066ea0700 1 -- 172.21.2.102:0/826678020 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f50600a0420 msgr2=0x7f50600a0820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:18.461 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:18.464 7f5066ea0700 1 --2- 172.21.2.102:0/826678020 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f50600a0420 0x7f50600a0820 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 rx=0x7f5048006f10 tx=0x7f504800b990).stop 2021-02-10T11:21:18.461 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:18.465 7f5066ea0700 1 -- 172.21.2.102:0/826678020 shutdown_connections 2021-02-10T11:21:18.461 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:18.465 7f5066ea0700 1 --2- 172.21.2.102:0/826678020 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f50440186c0 0x7f504401abc0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:18.461 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:18.465 7f5066ea0700 1 --2- 172.21.2.102:0/826678020 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f506018a720 0x7f50600b7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:18.461 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:18.465 7f5066ea0700 1 --2- 172.21.2.102:0/826678020 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5060189d00 0x7f506018a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:18.461 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:18.465 7f5066ea0700 1 --2- 172.21.2.102:0/826678020 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f50600a0420 0x7f50600a0820 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:18.461 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:18.465 7f5066ea0700 1 -- 172.21.2.102:0/826678020 >> 172.21.2.102:0/826678020 conn(0x7f5060094e10 msgr2=0x7f50600a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:18.461 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:18.465 7f5066ea0700 1 -- 172.21.2.102:0/826678020 shutdown_connections 2021-02-10T11:21:18.461 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:18.465 7f5066ea0700 1 -- 172.21.2.102:0/826678020 wait complete. 2021-02-10T11:21:18.468 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000344416 2021-02-10T11:21:18.898 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:18.886 7fbbccf3b700 1 Processor -- start 2021-02-10T11:21:18.898 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:18.886 7fbbccf3b700 1 -- start start 2021-02-10T11:21:18.898 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:18.886 7fbbccf3b700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbbc81b0470 0x7fbbc81b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:18.898 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:18.886 7fbbccf3b700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbc80f0b50 con 0x7fbbc80ef610 2021-02-10T11:21:18.898 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:18.886 7fbbccf3b700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbc80f24e0 con 0x7fbbc80f20e0 2021-02-10T11:21:18.898 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:18.886 7fbbccf3b700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbbc80e5ea0 con 0x7fbbc81b0470 2021-02-10T11:21:18.898 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:18.886 7fbbc7314700 1 --1- >> v1:172.21.2.113:6789/0 conn(0x7fbbc80f20e0 0x7fbbc80f0200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.113:6789/0 says I am v1:172.21.2.102:53956/0 (socket says 172.21.2.102:53956) 2021-02-10T11:21:18.898 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:18.886 7fbbbe312700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fbbc80ef610 0x7fbbc80efa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:46634/0 (socket says 172.21.2.102:46634) 2021-02-10T11:21:18.898 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:18.886 7fbbc7314700 1 -- 172.21.2.102:0/1522495329 learned_addr learned my addr 172.21.2.102:0/1522495329 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:21:18.898 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:18.887 7fbbc6b13700 1 --2- 172.21.2.102:0/1522495329 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbbc81b0470 0x7fbbc81b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:18.899 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:18.887 7fbbc6312700 1 -- 172.21.2.102:0/1522495329 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2730977034 0 0) 0x7fbbb00009b0 con 0x7fbbc80ef610 2021-02-10T11:21:18.899 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:18.887 7fbbc6312700 1 -- 172.21.2.102:0/1522495329 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbbb4002fe0 con 0x7fbbc80ef610 2021-02-10T11:21:18.899 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:18.887 7fbbc6312700 1 -- 172.21.2.102:0/1522495329 <== mon.1 v1:172.21.2.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 89768851 0 0) 0x7fbbc0001080 con 0x7fbbc80f20e0 2021-02-10T11:21:18.899 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:18.887 7fbbc6312700 1 -- 172.21.2.102:0/1522495329 --> v1:172.21.2.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbbb4003210 con 0x7fbbc80f20e0 2021-02-10T11:21:18.899 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:18.887 7fbbc6b13700 1 -- 172.21.2.102:0/1522495329 >> v1:172.21.2.113:6789/0 conn(0x7fbbc80f20e0 legacy=0x7fbbc80f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:18.899 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:18.887 7fbbc6b13700 1 -- 172.21.2.102:0/1522495329 >> v1:172.21.2.102:6789/0 conn(0x7fbbc80ef610 legacy=0x7fbbc80efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:18.899 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:18.887 7fbbc6b13700 1 -- 172.21.2.102:0/1522495329 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbc80e52e0 con 0x7fbbc81b0470 2021-02-10T11:21:18.899 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:18.887 7fbbc6b13700 1 --2- 172.21.2.102:0/1522495329 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbbc81b0470 0x7fbbc81b0870 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7fbbb80069d0 tx=0x7fbbb8009e40).ready entity=mon.2 client_cookie=edbf0a03fb5cb729 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:18.899 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:18.887 7fbbc6312700 1 -- 172.21.2.102:0/1522495329 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbbb800b510 con 0x7fbbc81b0470 2021-02-10T11:21:18.899 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:18.887 7fbbc6312700 1 -- 172.21.2.102:0/1522495329 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbb800b890 con 0x7fbbc81b0470 2021-02-10T11:21:18.899 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:18.888 7fbbc6312700 1 -- 172.21.2.102:0/1522495329 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbbb800bd70 con 0x7fbbc81b0470 2021-02-10T11:21:18.899 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:18.888 7fbbccf3b700 1 -- 172.21.2.102:0/1522495329 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbbc81b0470 msgr2=0x7fbbc81b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:18.899 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:18.888 7fbbccf3b700 1 --2- 172.21.2.102:0/1522495329 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbbc81b0470 0x7fbbc81b0870 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7fbbb80069d0 tx=0x7fbbb8009e40).stop 2021-02-10T11:21:18.899 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:18.888 7fbbccf3b700 1 -- 172.21.2.102:0/1522495329 shutdown_connections 2021-02-10T11:21:18.900 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:18.888 7fbbccf3b700 1 --2- 172.21.2.102:0/1522495329 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbbc81b0470 0x7fbbc81b0870 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:18.900 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:18.888 7fbbccf3b700 1 -- 172.21.2.102:0/1522495329 >> 172.21.2.102:0/1522495329 conn(0x7fbbc80ce3e0 msgr2=0x7fbbc80e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:18.900 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:18.888 7fbbccf3b700 1 -- 172.21.2.102:0/1522495329 shutdown_connections 2021-02-10T11:21:18.900 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:18.888 7fbbccf3b700 1 -- 172.21.2.102:0/1522495329 wait complete. 2021-02-10T11:21:18.902 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:18.903 7fbbccf3b700 1 Processor -- start 2021-02-10T11:21:18.902 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:18.903 7fbbccf3b700 1 -- start start 2021-02-10T11:21:18.902 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:18.903 7fbbccf3b700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbbc80a0420 0x7fbbc80a0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:18.902 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:18.903 7fbbccf3b700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbbc8189d00 0x7fbbc818a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:18.902 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:18.903 7fbbccf3b700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fbbc818a720 0x7fbbc80b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:18.902 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:18.903 7fbbccf3b700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbc8189780 con 0x7fbbc80a0420 2021-02-10T11:21:18.902 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:18.903 7fbbccf3b700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbc818ab20 con 0x7fbbc818a720 2021-02-10T11:21:18.902 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:18.903 7fbbccf3b700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbbc80a1140 con 0x7fbbc8189d00 2021-02-10T11:21:18.902 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:18.903 7fbbc6b13700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbbc80a0420 0x7fbbc80a0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:18.902 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:18.903 7fbbc6b13700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbbc80a0420 0x7fbbc80a0820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:54412/0 (socket says 172.21.2.102:54412) 2021-02-10T11:21:18.902 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:18.903 7fbbc6b13700 1 -- 172.21.2.102:0/1785528519 learned_addr learned my addr 172.21.2.102:0/1785528519 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:21:18.903 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:18.903 7fbbc7314700 1 --2- 172.21.2.102:0/1785528519 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fbbc818a720 0x7fbbc80b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:18.903 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:18.903 7fbbbe312700 1 --2- 172.21.2.102:0/1785528519 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbbc8189d00 0x7fbbc818a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:18.903 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:18.904 7fbbc6b13700 1 -- 172.21.2.102:0/1785528519 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbbc8189d00 msgr2=0x7fbbc818a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:18.903 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:18.904 7fbbc6b13700 1 --2- 172.21.2.102:0/1785528519 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbbc8189d00 0x7fbbc818a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:18.903 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:18.904 7fbbc6b13700 1 -- 172.21.2.102:0/1785528519 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fbbc818a720 msgr2=0x7fbbc80b7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:18.903 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:18.904 7fbbc6b13700 1 --2- 172.21.2.102:0/1785528519 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fbbc818a720 0x7fbbc80b7be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:18.903 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:18.904 7fbbc6b13700 1 -- 172.21.2.102:0/1785528519 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbc80b8200 con 0x7fbbc80a0420 2021-02-10T11:21:18.903 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:18.904 7fbbbe312700 1 --2- 172.21.2.102:0/1785528519 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbbc8189d00 0x7fbbc818a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:21:18.903 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:18.904 7fbbc6b13700 1 --2- 172.21.2.102:0/1785528519 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbbc80a0420 0x7fbbc80a0820 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 rx=0x7fbbb8009f40 tx=0x7fbbb800bd50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:18.903 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:18.904 7fbbbffff700 1 -- 172.21.2.102:0/1785528519 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbbb8006240 con 0x7fbbc80a0420 2021-02-10T11:21:18.903 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:18.904 7fbbbffff700 1 -- 172.21.2.102:0/1785528519 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbb80065c0 con 0x7fbbc80a0420 2021-02-10T11:21:18.903 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:18.904 7fbbbffff700 1 -- 172.21.2.102:0/1785528519 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbbb80058e0 con 0x7fbbc80a0420 2021-02-10T11:21:18.903 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:18.904 7fbbccf3b700 1 -- 172.21.2.102:0/1785528519 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbbc80b8200 con 0x7fbbc80a0420 2021-02-10T11:21:18.903 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:18.904 7fbbccf3b700 1 -- 172.21.2.102:0/1785528519 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbc80b8790 con 0x7fbbc80a0420 2021-02-10T11:21:18.904 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:18.905 7fbbbffff700 1 -- 172.21.2.102:0/1785528519 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fbbb8005660 con 0x7fbbc80a0420 2021-02-10T11:21:18.904 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:18.905 7fbbbffff700 1 --2- 172.21.2.102:0/1785528519 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fbbac0186c0 0x7fbbac01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:18.904 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:18.905 7fbbbffff700 1 -- 172.21.2.102:0/1785528519 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fbbb80328a0 con 0x7fbbc80a0420 2021-02-10T11:21:18.904 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:18.905 7fbbbe312700 1 --2- 172.21.2.102:0/1785528519 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fbbac0186c0 0x7fbbac01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:18.904 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:18.905 7fbbbe312700 1 --2- 172.21.2.102:0/1785528519 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fbbac0186c0 0x7fbbac01abc0 crc :-1 s=READY pgs=439 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:18.931 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:18.936 7fbbccf3b700 1 -- 172.21.2.102:0/1785528519 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbbc8105ae0 con 0x7fbbc80a0420 2021-02-10T11:21:18.934 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:18.939 7fbbbffff700 1 -- 172.21.2.102:0/1785528519 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fbbb802ef00 con 0x7fbbc80a0420 2021-02-10T11:21:19.060 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:19.064 7fbbccf3b700 1 -- 172.21.2.102:0/1785528519 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344416", "target": ["mgr", ""]}) v1 -- 0x7fbbc80b8790 con 0x7fbbac0186c0 2021-02-10T11:21:19.071 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:19.075 7fbbbffff700 1 -- 172.21.2.102:0/1785528519 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fbbb00009b0 con 0x7fbbac0186c0 2021-02-10T11:21:19.077 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:19.082 7fbbccf3b700 1 -- 172.21.2.102:0/1785528519 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fbbac0186c0 msgr2=0x7fbbac01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:19.077 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:19.082 7fbbccf3b700 1 --2- 172.21.2.102:0/1785528519 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fbbac0186c0 0x7fbbac01abc0 crc :-1 s=READY pgs=439 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:19.077 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:19.082 7fbbccf3b700 1 -- 172.21.2.102:0/1785528519 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbbc80a0420 msgr2=0x7fbbc80a0820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:19.077 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:19.082 7fbbccf3b700 1 --2- 172.21.2.102:0/1785528519 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbbc80a0420 0x7fbbc80a0820 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 rx=0x7fbbb8009f40 tx=0x7fbbb800bd50).stop 2021-02-10T11:21:19.078 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:19.082 7fbbccf3b700 1 -- 172.21.2.102:0/1785528519 shutdown_connections 2021-02-10T11:21:19.078 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:19.082 7fbbccf3b700 1 --2- 172.21.2.102:0/1785528519 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fbbac0186c0 0x7fbbac01abc0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:19.078 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:19.082 7fbbccf3b700 1 --2- 172.21.2.102:0/1785528519 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fbbc818a720 0x7fbbc80b7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:19.078 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:19.082 7fbbccf3b700 1 --2- 172.21.2.102:0/1785528519 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbbc8189d00 0x7fbbc818a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:19.078 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:19.082 7fbbccf3b700 1 --2- 172.21.2.102:0/1785528519 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbbc80a0420 0x7fbbc80a0820 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:19.078 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:19.082 7fbbccf3b700 1 -- 172.21.2.102:0/1785528519 >> 172.21.2.102:0/1785528519 conn(0x7fbbc8094e10 msgr2=0x7fbbc80a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:19.078 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:19.082 7fbbccf3b700 1 -- 172.21.2.102:0/1785528519 shutdown_connections 2021-02-10T11:21:19.078 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:19.082 7fbbccf3b700 1 -- 172.21.2.102:0/1785528519 wait complete. 2021-02-10T11:21:19.087 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000344417 2021-02-10T11:21:19.518 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:19.507 7f5dd295b700 1 Processor -- start 2021-02-10T11:21:19.518 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:19.507 7f5dd295b700 1 -- start start 2021-02-10T11:21:19.518 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:19.507 7f5dd295b700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f5dcc1b0470 0x7f5dcc1b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:19.519 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:19.507 7f5dd295b700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5dcc0ef610 0x7f5dcc0efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:19.519 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:19.507 7f5dd295b700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f5dcc0f0030 0x7f5dcc0f0430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:19.519 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:19.507 7f5dd295b700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5dcc0e5ea0 con 0x7f5dcc0f0030 2021-02-10T11:21:19.519 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:19.507 7f5dd295b700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5dcc0f0b50 con 0x7f5dcc1b0470 2021-02-10T11:21:19.519 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:19.507 7f5dd295b700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5dcc0e52e0 con 0x7f5dcc0ef610 2021-02-10T11:21:19.519 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:19.507 7f5dd0f78700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f5dcc0f0030 0x7f5dcc0f0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:19.519 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:19.507 7f5dd0f78700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f5dcc0f0030 0x7f5dcc0f0430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:54424/0 (socket says 172.21.2.102:54424) 2021-02-10T11:21:19.519 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:19.507 7f5dd0f78700 1 -- 172.21.2.102:0/87102100 learned_addr learned my addr 172.21.2.102:0/87102100 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:21:19.519 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:19.507 7f5dcb7fe700 1 --2- 172.21.2.102:0/87102100 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5dcc0ef610 0x7f5dcc0efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:19.519 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:19.507 7f5dcbfff700 1 --2- 172.21.2.102:0/87102100 >> v2:172.21.2.113:3300/0 conn(0x7f5dcc1b0470 0x7f5dcc1b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:19.519 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:19.508 7f5dd0f78700 1 -- 172.21.2.102:0/87102100 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5dcc0ef610 msgr2=0x7f5dcc0efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:21:19.519 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:19.508 7f5dd0f78700 1 --2- 172.21.2.102:0/87102100 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5dcc0ef610 0x7f5dcc0efa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:19.520 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:19.508 7f5dd0f78700 1 -- 172.21.2.102:0/87102100 >> v2:172.21.2.113:3300/0 conn(0x7f5dcc1b0470 msgr2=0x7f5dcc1b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:21:19.520 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:19.508 7f5dd0f78700 1 --2- 172.21.2.102:0/87102100 >> v2:172.21.2.113:3300/0 conn(0x7f5dcc1b0470 0x7f5dcc1b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:19.520 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:19.508 7f5dd0f78700 1 -- 172.21.2.102:0/87102100 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dcc0f10d0 con 0x7f5dcc0f0030 2021-02-10T11:21:19.520 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:19.508 7f5dcbfff700 1 --2- 172.21.2.102:0/87102100 >> v2:172.21.2.113:3300/0 conn(0x7f5dcc1b0470 0x7f5dcc1b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:21:19.520 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:19.508 7f5dcb7fe700 1 --2- 172.21.2.102:0/87102100 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5dcc0ef610 0x7f5dcc0efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:21:19.520 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:19.508 7f5dd0f78700 1 --2- 172.21.2.102:0/87102100 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5dcc0f0030 0x7f5dcc0f0430 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 rx=0x7f5dc4006f00 tx=0x7f5dc4009f10).ready entity=mon.0 client_cookie=fd0c4b8470367780 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:19.520 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:19.508 7f5dcaffd700 1 -- 172.21.2.102:0/87102100 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5dc400b3b0 con 0x7f5dcc0f0030 2021-02-10T11:21:19.520 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:19.508 7f5dcaffd700 1 -- 172.21.2.102:0/87102100 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5dc400b630 con 0x7f5dcc0f0030 2021-02-10T11:21:19.520 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:19.508 7f5dcaffd700 1 -- 172.21.2.102:0/87102100 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5dc400bb50 con 0x7f5dcc0f0030 2021-02-10T11:21:19.520 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:19.508 7f5dd295b700 1 -- 172.21.2.102:0/87102100 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5dcc0f0030 msgr2=0x7f5dcc0f0430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:19.520 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:19.508 7f5dd295b700 1 --2- 172.21.2.102:0/87102100 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5dcc0f0030 0x7f5dcc0f0430 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 rx=0x7f5dc4006f00 tx=0x7f5dc4009f10).stop 2021-02-10T11:21:19.520 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:19.509 7f5dd295b700 1 -- 172.21.2.102:0/87102100 shutdown_connections 2021-02-10T11:21:19.520 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:19.509 7f5dd295b700 1 --2- 172.21.2.102:0/87102100 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5dcc0ef610 0x7f5dcc0efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:19.520 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:19.509 7f5dd295b700 1 --2- 172.21.2.102:0/87102100 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5dcc0f0030 0x7f5dcc0f0430 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:19.521 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:19.509 7f5dd295b700 1 --2- 172.21.2.102:0/87102100 >> v2:172.21.2.113:3300/0 conn(0x7f5dcc1b0470 0x7f5dcc1b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:19.521 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:19.509 7f5dd295b700 1 -- 172.21.2.102:0/87102100 >> 172.21.2.102:0/87102100 conn(0x7f5dcc0ce3e0 msgr2=0x7f5dcc0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:19.521 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:19.509 7f5dd295b700 1 -- 172.21.2.102:0/87102100 shutdown_connections 2021-02-10T11:21:19.521 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:19.509 7f5dd295b700 1 -- 172.21.2.102:0/87102100 wait complete. 2021-02-10T11:21:19.524 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:19.523 7f5dd295b700 1 Processor -- start 2021-02-10T11:21:19.524 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:19.523 7f5dd295b700 1 -- start start 2021-02-10T11:21:19.524 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:19.524 7f5dd295b700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5dcc0b7db0 0x7f5dcc0b8260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:19.524 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:19.524 7f5dd295b700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5dcc0b8880 0x7f5dcc1a7450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:19.524 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:19.524 7f5dd295b700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5dcc0b8c80 0x7f5dcc1a7a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:19.524 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:19.524 7f5dd295b700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dcc0b79b0 con 0x7f5dcc0b8c80 2021-02-10T11:21:19.524 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:19.524 7f5dd295b700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dcc1a8090 con 0x7f5dcc0b7db0 2021-02-10T11:21:19.524 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:19.524 7f5dd295b700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5dcc1a82b0 con 0x7f5dcc0b8880 2021-02-10T11:21:19.524 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:19.524 7f5dd0f78700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5dcc0b8c80 0x7f5dcc1a7a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:19.524 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:19.524 7f5dd0f78700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5dcc0b8c80 0x7f5dcc1a7a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:54426/0 (socket says 172.21.2.102:54426) 2021-02-10T11:21:19.524 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:19.524 7f5dd0f78700 1 -- 172.21.2.102:0/3818893338 learned_addr learned my addr 172.21.2.102:0/3818893338 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:21:19.524 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:19.524 7f5dcb7fe700 1 --2- 172.21.2.102:0/3818893338 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5dcc0b8880 0x7f5dcc1a7450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:19.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:19.524 7f5dcbfff700 1 --2- 172.21.2.102:0/3818893338 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5dcc0b7db0 0x7f5dcc0b8260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:19.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:19.524 7f5dd0f78700 1 -- 172.21.2.102:0/3818893338 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5dcc0b8880 msgr2=0x7f5dcc1a7450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:19.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:19.524 7f5dd0f78700 1 --2- 172.21.2.102:0/3818893338 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5dcc0b8880 0x7f5dcc1a7450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:19.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:19.524 7f5dd0f78700 1 -- 172.21.2.102:0/3818893338 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5dcc0b7db0 msgr2=0x7f5dcc0b8260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:19.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:19.524 7f5dd0f78700 1 --2- 172.21.2.102:0/3818893338 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5dcc0b7db0 0x7f5dcc0b8260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:19.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:19.524 7f5dd0f78700 1 -- 172.21.2.102:0/3818893338 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dcc190c10 con 0x7f5dcc0b8c80 2021-02-10T11:21:19.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:19.524 7f5dcbfff700 1 --2- 172.21.2.102:0/3818893338 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5dcc0b7db0 0x7f5dcc0b8260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:21:19.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:19.524 7f5dd0f78700 1 --2- 172.21.2.102:0/3818893338 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5dcc0b8c80 0x7f5dcc1a7a70 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 rx=0x7f5dc400b9b0 tx=0x7f5dc400bd50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:19.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:19.525 7f5dc8ff9700 1 -- 172.21.2.102:0/3818893338 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5dc4005820 con 0x7f5dcc0b8c80 2021-02-10T11:21:19.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:19.525 7f5dc8ff9700 1 -- 172.21.2.102:0/3818893338 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5dc4005aa0 con 0x7f5dcc0b8c80 2021-02-10T11:21:19.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:19.525 7f5dc8ff9700 1 -- 172.21.2.102:0/3818893338 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5dc4006030 con 0x7f5dcc0b8c80 2021-02-10T11:21:19.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:19.525 7f5dd295b700 1 -- 172.21.2.102:0/3818893338 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5dcc190c10 con 0x7f5dcc0b8c80 2021-02-10T11:21:19.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:19.525 7f5dd295b700 1 -- 172.21.2.102:0/3818893338 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5dcc190c10 con 0x7f5dcc0b8c80 2021-02-10T11:21:19.526 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:19.525 7f5dc8ff9700 1 -- 172.21.2.102:0/3818893338 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f5dc4006030 con 0x7f5dcc0b8c80 2021-02-10T11:21:19.526 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:19.525 7f5dc8ff9700 1 --2- 172.21.2.102:0/3818893338 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5dac018730 0x7f5dac01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:19.526 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:19.525 7f5dc8ff9700 1 -- 172.21.2.102:0/3818893338 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f5dc40329c0 con 0x7f5dcc0b8c80 2021-02-10T11:21:19.526 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:19.525 7f5dcbfff700 1 --2- 172.21.2.102:0/3818893338 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5dac018730 0x7f5dac01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:19.526 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:19.526 7f5dcbfff700 1 --2- 172.21.2.102:0/3818893338 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5dac018730 0x7f5dac01ac10 crc :-1 s=READY pgs=440 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:19.553 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:19.557 7f5dd295b700 1 -- 172.21.2.102:0/3818893338 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5dcc105b30 con 0x7f5dcc0b8c80 2021-02-10T11:21:19.555 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:19.560 7f5dc8ff9700 1 -- 172.21.2.102:0/3818893338 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f5dc4000f90 con 0x7f5dcc0b8c80 2021-02-10T11:21:19.680 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:19.685 7f5dd295b700 1 -- 172.21.2.102:0/3818893338 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344417", "target": ["mgr", ""]}) v1 -- 0x7f5dcc0f0b50 con 0x7f5dac018730 2021-02-10T11:21:19.690 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:19.695 7f5dc8ff9700 1 -- 172.21.2.102:0/3818893338 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f5dbc006ad0 con 0x7f5dac018730 2021-02-10T11:21:19.697 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:19.702 7f5dd295b700 1 -- 172.21.2.102:0/3818893338 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5dac018730 msgr2=0x7f5dac01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:19.697 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:19.702 7f5dd295b700 1 --2- 172.21.2.102:0/3818893338 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5dac018730 0x7f5dac01ac10 crc :-1 s=READY pgs=440 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:19.697 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:19.702 7f5dd295b700 1 -- 172.21.2.102:0/3818893338 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5dcc0b8c80 msgr2=0x7f5dcc1a7a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:19.697 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:19.702 7f5dd295b700 1 --2- 172.21.2.102:0/3818893338 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5dcc0b8c80 0x7f5dcc1a7a70 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 rx=0x7f5dc400b9b0 tx=0x7f5dc400bd50).stop 2021-02-10T11:21:19.698 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:19.702 7f5dd295b700 1 -- 172.21.2.102:0/3818893338 shutdown_connections 2021-02-10T11:21:19.698 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:19.702 7f5dd295b700 1 --2- 172.21.2.102:0/3818893338 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5dcc0b8c80 0x7f5dcc1a7a70 unknown :-1 s=CLOSED pgs=902 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:19.698 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:19.702 7f5dd295b700 1 --2- 172.21.2.102:0/3818893338 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5dcc0b8880 0x7f5dcc1a7450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:19.698 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:19.702 7f5dd295b700 1 --2- 172.21.2.102:0/3818893338 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5dac018730 0x7f5dac01ac10 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:19.698 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:19.702 7f5dd295b700 1 --2- 172.21.2.102:0/3818893338 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5dcc0b7db0 0x7f5dcc0b8260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:19.698 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:19.702 7f5dd295b700 1 -- 172.21.2.102:0/3818893338 >> 172.21.2.102:0/3818893338 conn(0x7f5dcc094e10 msgr2=0x7f5dcc0a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:19.698 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:19.702 7f5dd295b700 1 -- 172.21.2.102:0/3818893338 shutdown_connections 2021-02-10T11:21:19.698 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:19.702 7f5dd295b700 1 -- 172.21.2.102:0/3818893338 wait complete. 2021-02-10T11:21:19.707 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000344418 2021-02-10T11:21:20.138 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:20.126 7fb01a4ba700 1 Processor -- start 2021-02-10T11:21:20.138 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:20.126 7fb01a4ba700 1 -- start start 2021-02-10T11:21:20.138 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:20.126 7fb01a4ba700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb0141b0470 0x7fb0141b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:20.138 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:20.127 7fb01a4ba700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fb0140f20e0 0x7fb0140f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:20.138 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:20.127 7fb01a4ba700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0140f0b50 con 0x7fb0140ef610 2021-02-10T11:21:20.138 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:20.127 7fb01a4ba700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0140f0820 con 0x7fb0140f20e0 2021-02-10T11:21:20.138 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:20.127 7fb01a4ba700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0140f24e0 con 0x7fb0141b0470 2021-02-10T11:21:20.138 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:20.127 7fb0137fe700 1 --1- >> v1:172.21.2.113:6789/0 conn(0x7fb0140ef610 0x7fb0140efa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.113:6789/0 says I am v1:172.21.2.102:53982/0 (socket says 172.21.2.102:53982) 2021-02-10T11:21:20.138 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:20.127 7fb013fff700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb0141b0470 0x7fb0141b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:20.139 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:20.127 7fb0137fe700 1 -- 172.21.2.102:0/2530052654 learned_addr learned my addr 172.21.2.102:0/2530052654 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:21:20.139 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:20.127 7fb018ad7700 1 --2- 172.21.2.102:0/2530052654 >> v2:172.21.2.113:3300/0 conn(0x7fb0140f20e0 0x7fb0140f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:20.139 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:20.127 7fb012ffd700 1 -- 172.21.2.102:0/2530052654 <== mon.1 v1:172.21.2.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1592377222 0 0) 0x7faffc001190 con 0x7fb0140ef610 2021-02-10T11:21:20.139 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:20.127 7fb012ffd700 1 -- 172.21.2.102:0/2530052654 --> v1:172.21.2.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb000002fe0 con 0x7fb0140ef610 2021-02-10T11:21:20.139 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:20.127 7fb013fff700 1 -- 172.21.2.102:0/2530052654 >> v2:172.21.2.113:3300/0 conn(0x7fb0140f20e0 msgr2=0x7fb0140f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:21:20.139 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:20.127 7fb013fff700 1 --2- 172.21.2.102:0/2530052654 >> v2:172.21.2.113:3300/0 conn(0x7fb0140f20e0 0x7fb0140f0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:20.139 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:20.127 7fb013fff700 1 -- 172.21.2.102:0/2530052654 >> v1:172.21.2.113:6789/0 conn(0x7fb0140ef610 legacy=0x7fb0140efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:20.139 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:20.127 7fb013fff700 1 -- 172.21.2.102:0/2530052654 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0140e52e0 con 0x7fb0141b0470 2021-02-10T11:21:20.139 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:20.127 7fb013fff700 1 --2- 172.21.2.102:0/2530052654 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb0141b0470 0x7fb0141b0870 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7fb004006650 tx=0x7fb004009a80).ready entity=mon.2 client_cookie=34fd036f49e5c662 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:20.139 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:20.128 7fb012ffd700 1 -- 172.21.2.102:0/2530052654 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb0040009b0 con 0x7fb0141b0470 2021-02-10T11:21:20.139 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:20.128 7fb012ffd700 1 -- 172.21.2.102:0/2530052654 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb00400c0c0 con 0x7fb0141b0470 2021-02-10T11:21:20.139 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:20.128 7fb012ffd700 1 -- 172.21.2.102:0/2530052654 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb00400c5e0 con 0x7fb0141b0470 2021-02-10T11:21:20.139 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:20.128 7fb01a4ba700 1 -- 172.21.2.102:0/2530052654 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb0141b0470 msgr2=0x7fb0141b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:20.140 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:20.128 7fb01a4ba700 1 --2- 172.21.2.102:0/2530052654 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb0141b0470 0x7fb0141b0870 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7fb004006650 tx=0x7fb004009a80).stop 2021-02-10T11:21:20.140 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:20.128 7fb01a4ba700 1 -- 172.21.2.102:0/2530052654 shutdown_connections 2021-02-10T11:21:20.140 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:20.128 7fb01a4ba700 1 --2- 172.21.2.102:0/2530052654 >> v2:172.21.2.113:3300/0 conn(0x7fb0140f20e0 0x7fb0140f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:20.140 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:20.128 7fb01a4ba700 1 --2- 172.21.2.102:0/2530052654 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb0141b0470 0x7fb0141b0870 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:20.140 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:20.128 7fb01a4ba700 1 -- 172.21.2.102:0/2530052654 >> 172.21.2.102:0/2530052654 conn(0x7fb0140ce3e0 msgr2=0x7fb0140e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:20.140 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:20.128 7fb01a4ba700 1 -- 172.21.2.102:0/2530052654 shutdown_connections 2021-02-10T11:21:20.140 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:20.128 7fb01a4ba700 1 -- 172.21.2.102:0/2530052654 wait complete. 2021-02-10T11:21:20.142 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:20.143 7fb01a4ba700 1 Processor -- start 2021-02-10T11:21:20.142 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:20.143 7fb01a4ba700 1 -- start start 2021-02-10T11:21:20.142 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:20.143 7fb01a4ba700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb0140a0910 0x7fb014105d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:20.142 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:20.143 7fb01a4ba700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb0141a7a10 0x7fb0141a7e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:20.142 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:20.143 7fb01a4ba700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb014093770 0x7fb014093b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:20.142 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:20.143 7fb01a4ba700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb014106390 con 0x7fb014093770 2021-02-10T11:21:20.142 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:20.143 7fb01a4ba700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb014094190 con 0x7fb0140a0910 2021-02-10T11:21:20.142 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:20.143 7fb01a4ba700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0140943b0 con 0x7fb0141a7a10 2021-02-10T11:21:20.143 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:20.143 7fb018ad7700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb014093770 0x7fb014093b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:20.143 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:20.144 7fb018ad7700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb014093770 0x7fb014093b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:54444/0 (socket says 172.21.2.102:54444) 2021-02-10T11:21:20.143 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:20.144 7fb018ad7700 1 -- 172.21.2.102:0/3545998998 learned_addr learned my addr 172.21.2.102:0/3545998998 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:21:20.143 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:20.144 7fb013fff700 1 --2- 172.21.2.102:0/3545998998 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb0140a0910 0x7fb014105d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:20.143 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:20.144 7fb0137fe700 1 --2- 172.21.2.102:0/3545998998 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb0141a7a10 0x7fb0141a7e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:20.143 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:20.144 7fb018ad7700 1 -- 172.21.2.102:0/3545998998 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb0141a7a10 msgr2=0x7fb0141a7e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:20.143 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:20.144 7fb018ad7700 1 --2- 172.21.2.102:0/3545998998 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb0141a7a10 0x7fb0141a7e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:20.143 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:20.144 7fb018ad7700 1 -- 172.21.2.102:0/3545998998 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb0140a0910 msgr2=0x7fb014105d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:20.143 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:20.144 7fb018ad7700 1 --2- 172.21.2.102:0/3545998998 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb0140a0910 0x7fb014105d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:20.143 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:20.144 7fb018ad7700 1 -- 172.21.2.102:0/3545998998 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb014106c30 con 0x7fb014093770 2021-02-10T11:21:20.143 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:20.144 7fb0137fe700 1 --2- 172.21.2.102:0/3545998998 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb0141a7a10 0x7fb0141a7e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:21:20.143 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:20.144 7fb018ad7700 1 --2- 172.21.2.102:0/3545998998 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb014093770 0x7fb014093b70 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 rx=0x7fb00c00b970 tx=0x7fb00c00bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:20.143 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:20.144 7fb010ff9700 1 -- 172.21.2.102:0/3545998998 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb00c000f90 con 0x7fb014093770 2021-02-10T11:21:20.143 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:20.144 7fb010ff9700 1 -- 172.21.2.102:0/3545998998 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb00c003030 con 0x7fb014093770 2021-02-10T11:21:20.144 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:20.144 7fb01a4ba700 1 -- 172.21.2.102:0/3545998998 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb014106c30 con 0x7fb014093770 2021-02-10T11:21:20.144 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:20.144 7fb01a4ba700 1 -- 172.21.2.102:0/3545998998 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0141071c0 con 0x7fb014093770 2021-02-10T11:21:20.144 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:20.144 7fb010ff9700 1 -- 172.21.2.102:0/3545998998 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb00c001400 con 0x7fb014093770 2021-02-10T11:21:20.144 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:20.145 7fb010ff9700 1 -- 172.21.2.102:0/3545998998 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fb00c0011f0 con 0x7fb014093770 2021-02-10T11:21:20.144 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:20.145 7fb010ff9700 1 --2- 172.21.2.102:0/3545998998 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7faff80186c0 0x7faff801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:20.144 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:20.145 7fb010ff9700 1 -- 172.21.2.102:0/3545998998 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fb00c032930 con 0x7fb014093770 2021-02-10T11:21:20.144 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:20.145 7fb013fff700 1 --2- 172.21.2.102:0/3545998998 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7faff80186c0 0x7faff801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:20.144 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:20.145 7fb013fff700 1 --2- 172.21.2.102:0/3545998998 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7faff80186c0 0x7faff801abc0 crc :-1 s=READY pgs=441 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:20.171 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:20.176 7fb01a4ba700 1 -- 172.21.2.102:0/3545998998 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb014105b40 con 0x7fb014093770 2021-02-10T11:21:20.173 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:20.178 7fb010ff9700 1 -- 172.21.2.102:0/3545998998 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fb00c000f40 con 0x7fb014093770 2021-02-10T11:21:20.299 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:20.303 7fb01a4ba700 1 -- 172.21.2.102:0/3545998998 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344418", "target": ["mgr", ""]}) v1 -- 0x7fb014106390 con 0x7faff80186c0 2021-02-10T11:21:20.310 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:20.315 7fb010ff9700 1 -- 172.21.2.102:0/3545998998 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fb004009180 con 0x7faff80186c0 2021-02-10T11:21:20.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:20.321 7fb01a4ba700 1 -- 172.21.2.102:0/3545998998 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7faff80186c0 msgr2=0x7faff801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:20.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:20.321 7fb01a4ba700 1 --2- 172.21.2.102:0/3545998998 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7faff80186c0 0x7faff801abc0 crc :-1 s=READY pgs=441 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:20.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:20.321 7fb01a4ba700 1 -- 172.21.2.102:0/3545998998 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb014093770 msgr2=0x7fb014093b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:20.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:20.321 7fb01a4ba700 1 --2- 172.21.2.102:0/3545998998 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb014093770 0x7fb014093b70 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 rx=0x7fb00c00b970 tx=0x7fb00c00bd10).stop 2021-02-10T11:21:20.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:20.321 7fb01a4ba700 1 -- 172.21.2.102:0/3545998998 shutdown_connections 2021-02-10T11:21:20.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:20.321 7fb01a4ba700 1 --2- 172.21.2.102:0/3545998998 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb014093770 0x7fb014093b70 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:20.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:20.321 7fb01a4ba700 1 --2- 172.21.2.102:0/3545998998 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb0141a7a10 0x7fb0141a7e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:20.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:20.321 7fb01a4ba700 1 --2- 172.21.2.102:0/3545998998 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7faff80186c0 0x7faff801abc0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:20.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:20.321 7fb01a4ba700 1 --2- 172.21.2.102:0/3545998998 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb0140a0910 0x7fb014105d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:20.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:20.321 7fb01a4ba700 1 -- 172.21.2.102:0/3545998998 >> 172.21.2.102:0/3545998998 conn(0x7fb014095ef0 msgr2=0x7fb014094e40 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:20.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:20.321 7fb01a4ba700 1 -- 172.21.2.102:0/3545998998 shutdown_connections 2021-02-10T11:21:20.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:20.321 7fb01a4ba700 1 -- 172.21.2.102:0/3545998998 wait complete. 2021-02-10T11:21:20.325 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000344419 2021-02-10T11:21:20.758 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:20.746 7f2821025700 1 Processor -- start 2021-02-10T11:21:20.759 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:20.746 7f2821025700 1 -- start start 2021-02-10T11:21:20.759 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:20.747 7f2821025700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f281c1b0470 0x7f281c1b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:20.759 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:20.747 7f2821025700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f281c0f0b50 con 0x7f281c0f20e0 2021-02-10T11:21:20.759 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:20.747 7f2821025700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f281c0f24e0 con 0x7f281c0ef610 2021-02-10T11:21:20.759 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:20.747 7f2821025700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f281c0e5ea0 con 0x7f281c1b0470 2021-02-10T11:21:20.759 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:20.747 7f281b59e700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f281c0f20e0 0x7f281c0f0200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:46676/0 (socket says 172.21.2.102:46676) 2021-02-10T11:21:20.759 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:20.747 7f281b59e700 1 -- 172.21.2.102:0/878561435 learned_addr learned my addr 172.21.2.102:0/878561435 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:21:20.759 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:20.747 7f281ad9d700 1 --2- 172.21.2.102:0/878561435 >> v2:172.21.2.113:3300/0 conn(0x7f281c1b0470 0x7f281c1b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:20.759 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:20.747 7f2819d9b700 1 -- 172.21.2.102:0/878561435 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 269651330 0 0) 0x7f2814000f90 con 0x7f281c0f20e0 2021-02-10T11:21:20.759 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:20.747 7f2819d9b700 1 -- 172.21.2.102:0/878561435 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2804002fe0 con 0x7f281c0f20e0 2021-02-10T11:21:20.759 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:20.747 7f2819d9b700 1 -- 172.21.2.102:0/878561435 <== mon.1 v1:172.21.2.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3697885244 0 0) 0x7f2810001080 con 0x7f281c0ef610 2021-02-10T11:21:20.760 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:20.747 7f2819d9b700 1 -- 172.21.2.102:0/878561435 --> v1:172.21.2.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f28040042c0 con 0x7f281c0ef610 2021-02-10T11:21:20.760 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:20.747 7f281ad9d700 1 -- 172.21.2.102:0/878561435 >> v1:172.21.2.113:6789/0 conn(0x7f281c0ef610 legacy=0x7f281c0efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:20.760 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:20.747 7f281ad9d700 1 -- 172.21.2.102:0/878561435 >> v1:172.21.2.102:6789/0 conn(0x7f281c0f20e0 legacy=0x7f281c0f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:20.760 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:20.747 7f281ad9d700 1 -- 172.21.2.102:0/878561435 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f281c0e52e0 con 0x7f281c1b0470 2021-02-10T11:21:20.760 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:20.748 7f281ad9d700 1 --2- 172.21.2.102:0/878561435 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f281c1b0470 0x7f281c1b0870 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 rx=0x7f280c006650 tx=0x7f280c009a80).ready entity=mon.1 client_cookie=3bbeeb0377737a18 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:20.760 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:20.748 7f2819d9b700 1 -- 172.21.2.102:0/878561435 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f280c00c0c0 con 0x7f281c1b0470 2021-02-10T11:21:20.760 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:20.748 7f2819d9b700 1 -- 172.21.2.102:0/878561435 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f280c00c340 con 0x7f281c1b0470 2021-02-10T11:21:20.760 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:20.748 7f2819d9b700 1 -- 172.21.2.102:0/878561435 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f280c00c860 con 0x7f281c1b0470 2021-02-10T11:21:20.760 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:20.748 7f2821025700 1 -- 172.21.2.102:0/878561435 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f281c1b0470 msgr2=0x7f281c1b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:20.760 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:20.748 7f2821025700 1 --2- 172.21.2.102:0/878561435 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f281c1b0470 0x7f281c1b0870 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 rx=0x7f280c006650 tx=0x7f280c009a80).stop 2021-02-10T11:21:20.760 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:20.749 7f2821025700 1 -- 172.21.2.102:0/878561435 shutdown_connections 2021-02-10T11:21:20.761 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:20.749 7f2821025700 1 --2- 172.21.2.102:0/878561435 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f281c1b0470 0x7f281c1b0870 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:20.761 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:20.749 7f2821025700 1 -- 172.21.2.102:0/878561435 >> 172.21.2.102:0/878561435 conn(0x7f281c0ce3e0 msgr2=0x7f281c0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:20.761 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:20.749 7f2821025700 1 -- 172.21.2.102:0/878561435 shutdown_connections 2021-02-10T11:21:20.761 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:20.749 7f2821025700 1 -- 172.21.2.102:0/878561435 wait complete. 2021-02-10T11:21:20.763 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:20.763 7f2821025700 1 Processor -- start 2021-02-10T11:21:20.763 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:20.763 7f2821025700 1 -- start start 2021-02-10T11:21:20.763 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:20.764 7f2821025700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f281c08ccf0 0x7f281c0bd480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:20.763 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:20.764 7f2821025700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f281c0bdaa0 0x7f281c0b14c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:20.763 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:20.764 7f2821025700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f281c08d0f0 0x7f281c0b1ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:20.763 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:20.764 7f2821025700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f281c0b2100 con 0x7f281c08ccf0 2021-02-10T11:21:20.763 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:20.764 7f2821025700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f281c08c7a0 con 0x7f281c08d0f0 2021-02-10T11:21:20.763 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:20.764 7f2821025700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f281c08c440 con 0x7f281c0bdaa0 2021-02-10T11:21:20.764 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:20.764 7f281ad9d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f281c08ccf0 0x7f281c0bd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:20.764 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:20.764 7f281ad9d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f281c08ccf0 0x7f281c0bd480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:54454/0 (socket says 172.21.2.102:54454) 2021-02-10T11:21:20.764 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:20.764 7f281ad9d700 1 -- 172.21.2.102:0/3543552245 learned_addr learned my addr 172.21.2.102:0/3543552245 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:21:20.764 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:20.764 7f281b59e700 1 --2- 172.21.2.102:0/3543552245 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f281c08d0f0 0x7f281c0b1ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:20.764 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:20.764 7f281a59c700 1 --2- 172.21.2.102:0/3543552245 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f281c0bdaa0 0x7f281c0b14c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:20.764 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:20.765 7f281ad9d700 1 -- 172.21.2.102:0/3543552245 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f281c0bdaa0 msgr2=0x7f281c0b14c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:20.764 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:20.765 7f281ad9d700 1 --2- 172.21.2.102:0/3543552245 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f281c0bdaa0 0x7f281c0b14c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:20.764 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:20.765 7f281ad9d700 1 -- 172.21.2.102:0/3543552245 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f281c08d0f0 msgr2=0x7f281c0b1ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:20.764 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:20.765 7f281ad9d700 1 --2- 172.21.2.102:0/3543552245 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f281c08d0f0 0x7f281c0b1ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:20.764 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:20.765 7f281ad9d700 1 -- 172.21.2.102:0/3543552245 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f281c18e820 con 0x7f281c08ccf0 2021-02-10T11:21:20.764 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:20.765 7f281ad9d700 1 --2- 172.21.2.102:0/3543552245 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f281c08ccf0 0x7f281c0bd480 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 rx=0x7f280c006fc0 tx=0x7f280c00b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:20.765 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:20.765 7f280b7fe700 1 -- 172.21.2.102:0/3543552245 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f280c00bd50 con 0x7f281c08ccf0 2021-02-10T11:21:20.765 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:20.765 7f280b7fe700 1 -- 172.21.2.102:0/3543552245 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f280c000c90 con 0x7f281c08ccf0 2021-02-10T11:21:20.765 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:20.765 7f2821025700 1 -- 172.21.2.102:0/3543552245 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f281c18e820 con 0x7f281c08ccf0 2021-02-10T11:21:20.765 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:20.765 7f280b7fe700 1 -- 172.21.2.102:0/3543552245 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f280c002300 con 0x7f281c08ccf0 2021-02-10T11:21:20.765 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:20.765 7f2821025700 1 -- 172.21.2.102:0/3543552245 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f281c18e820 con 0x7f281c08ccf0 2021-02-10T11:21:20.765 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:20.766 7f280b7fe700 1 -- 172.21.2.102:0/3543552245 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f280c0052e0 con 0x7f281c08ccf0 2021-02-10T11:21:20.765 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:20.766 7f280b7fe700 1 --2- 172.21.2.102:0/3543552245 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f28000186d0 0x7f280001abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:20.765 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:20.766 7f280b7fe700 1 -- 172.21.2.102:0/3543552245 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f280c032640 con 0x7f281c08ccf0 2021-02-10T11:21:20.765 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:20.766 7f281a59c700 1 --2- 172.21.2.102:0/3543552245 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f28000186d0 0x7f280001abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:20.765 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:20.766 7f281a59c700 1 --2- 172.21.2.102:0/3543552245 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f28000186d0 0x7f280001abd0 crc :-1 s=READY pgs=442 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:20.791 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:20.795 7f2821025700 1 -- 172.21.2.102:0/3543552245 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f281c0b2100 con 0x7f281c08ccf0 2021-02-10T11:21:20.793 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:20.798 7f280b7fe700 1 -- 172.21.2.102:0/3543552245 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f280c000940 con 0x7f281c08ccf0 2021-02-10T11:21:20.921 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:20.926 7f2821025700 1 -- 172.21.2.102:0/3543552245 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344419", "target": ["mgr", ""]}) v1 -- 0x7f281c0b2100 con 0x7f28000186d0 2021-02-10T11:21:20.931 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:20.936 7f280b7fe700 1 -- 172.21.2.102:0/3543552245 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f2810006550 con 0x7f28000186d0 2021-02-10T11:21:20.939 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:20.943 7f2821025700 1 -- 172.21.2.102:0/3543552245 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f28000186d0 msgr2=0x7f280001abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:20.939 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:20.943 7f2821025700 1 --2- 172.21.2.102:0/3543552245 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f28000186d0 0x7f280001abd0 crc :-1 s=READY pgs=442 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:20.939 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:20.943 7f2821025700 1 -- 172.21.2.102:0/3543552245 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f281c08ccf0 msgr2=0x7f281c0bd480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:20.939 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:20.943 7f2821025700 1 --2- 172.21.2.102:0/3543552245 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f281c08ccf0 0x7f281c0bd480 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 rx=0x7f280c006fc0 tx=0x7f280c00b960).stop 2021-02-10T11:21:20.939 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:20.943 7f2821025700 1 -- 172.21.2.102:0/3543552245 shutdown_connections 2021-02-10T11:21:20.939 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:20.943 7f2821025700 1 --2- 172.21.2.102:0/3543552245 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f28000186d0 0x7f280001abd0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:20.939 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:20.943 7f2821025700 1 --2- 172.21.2.102:0/3543552245 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f281c08d0f0 0x7f281c0b1ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:20.939 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:20.943 7f2821025700 1 --2- 172.21.2.102:0/3543552245 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f281c0bdaa0 0x7f281c0b14c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:20.939 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:20.943 7f2821025700 1 --2- 172.21.2.102:0/3543552245 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f281c08ccf0 0x7f281c0bd480 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:20.939 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:20.943 7f2821025700 1 -- 172.21.2.102:0/3543552245 >> 172.21.2.102:0/3543552245 conn(0x7f281c0bd060 msgr2=0x7f281c09a020 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:20.940 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:20.943 7f2821025700 1 -- 172.21.2.102:0/3543552245 shutdown_connections 2021-02-10T11:21:20.940 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:20.943 7f2821025700 1 -- 172.21.2.102:0/3543552245 wait complete. 2021-02-10T11:21:20.948 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000344420 2021-02-10T11:21:21.381 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:21.369 7f35865f9700 1 Processor -- start 2021-02-10T11:21:21.382 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:21.369 7f35865f9700 1 -- start start 2021-02-10T11:21:21.382 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:21.370 7f35865f9700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f35800f05f0 0x7f35800f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:21.382 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:21.370 7f35865f9700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35801b0c70 con 0x7f35801b0870 2021-02-10T11:21:21.382 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:21.370 7f35865f9700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35800f0b50 con 0x7f35801b0470 2021-02-10T11:21:21.382 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:21.370 7f35865f9700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35800e5ea0 con 0x7f35800f05f0 2021-02-10T11:21:21.382 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:21.370 7f357f7fe700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f35801b0870 0x7f35800efe00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:46686/0 (socket says 172.21.2.102:46686) 2021-02-10T11:21:21.382 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:21.370 7f357f7fe700 1 -- 172.21.2.102:0/270618054 learned_addr learned my addr 172.21.2.102:0/270618054 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:21:21.382 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:21.370 7f3584c16700 1 --2- 172.21.2.102:0/270618054 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f35800f05f0 0x7f35800f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:21.382 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:21.371 7f357effd700 1 -- 172.21.2.102:0/270618054 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1754033235 0 0) 0x7f35680010b0 con 0x7f35801b0870 2021-02-10T11:21:21.383 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:21.371 7f357effd700 1 -- 172.21.2.102:0/270618054 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f356c002fe0 con 0x7f35801b0870 2021-02-10T11:21:21.383 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:21.371 7f357effd700 1 -- 172.21.2.102:0/270618054 <== mon.1 v1:172.21.2.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2396585355 0 0) 0x7f3570000fe0 con 0x7f35801b0470 2021-02-10T11:21:21.383 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:21.371 7f357effd700 1 -- 172.21.2.102:0/270618054 --> v1:172.21.2.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f356c004370 con 0x7f35801b0470 2021-02-10T11:21:21.383 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:21.371 7f3584c16700 1 -- 172.21.2.102:0/270618054 >> v1:172.21.2.113:6789/0 conn(0x7f35801b0470 legacy=0x7f35800ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:21.383 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:21.371 7f3584c16700 1 -- 172.21.2.102:0/270618054 >> v1:172.21.2.102:6789/0 conn(0x7f35801b0870 legacy=0x7f35800efe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:21.383 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:21.371 7f3584c16700 1 -- 172.21.2.102:0/270618054 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35800e52e0 con 0x7f35800f05f0 2021-02-10T11:21:21.383 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:21.371 7f3584c16700 1 --2- 172.21.2.102:0/270618054 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f35800f05f0 0x7f35800f20e0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f3578006ad0 tx=0x7f3578009ad0).ready entity=mon.2 client_cookie=fb44a912c4616c5e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:21.383 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:21.371 7f357effd700 1 -- 172.21.2.102:0/270618054 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f35780022e0 con 0x7f35800f05f0 2021-02-10T11:21:21.383 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:21.371 7f357effd700 1 -- 172.21.2.102:0/270618054 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3578006410 con 0x7f35800f05f0 2021-02-10T11:21:21.383 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:21.371 7f357effd700 1 -- 172.21.2.102:0/270618054 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3578002620 con 0x7f35800f05f0 2021-02-10T11:21:21.383 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:21.371 7f35865f9700 1 -- 172.21.2.102:0/270618054 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f35800f05f0 msgr2=0x7f35800f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:21.384 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:21.371 7f35865f9700 1 --2- 172.21.2.102:0/270618054 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f35800f05f0 0x7f35800f20e0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f3578006ad0 tx=0x7f3578009ad0).stop 2021-02-10T11:21:21.384 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:21.372 7f35865f9700 1 -- 172.21.2.102:0/270618054 shutdown_connections 2021-02-10T11:21:21.384 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:21.372 7f35865f9700 1 --2- 172.21.2.102:0/270618054 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f35800f05f0 0x7f35800f20e0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:21.384 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:21.372 7f35865f9700 1 -- 172.21.2.102:0/270618054 >> 172.21.2.102:0/270618054 conn(0x7f35800ce3e0 msgr2=0x7f35800e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:21.384 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:21.372 7f35865f9700 1 -- 172.21.2.102:0/270618054 shutdown_connections 2021-02-10T11:21:21.384 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:21.372 7f35865f9700 1 -- 172.21.2.102:0/270618054 wait complete. 2021-02-10T11:21:21.386 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:21.387 7f35865f9700 1 Processor -- start 2021-02-10T11:21:21.386 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:21.387 7f35865f9700 1 -- start start 2021-02-10T11:21:21.386 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:21.387 7f35865f9700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3580193230 0x7f35800a6a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:21.386 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:21.387 7f35865f9700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f358008f5b0 0x7f358008f9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:21.386 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:21.387 7f35865f9700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f358008e6a0 0x7f358008eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:21.386 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:21.387 7f35865f9700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f358008ffd0 con 0x7f3580193230 2021-02-10T11:21:21.386 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:21.387 7f35865f9700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3580090240 con 0x7f358008f5b0 2021-02-10T11:21:21.387 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:21.387 7f35865f9700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f358008f0c0 con 0x7f358008e6a0 2021-02-10T11:21:21.387 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:21.387 7f357ffff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3580193230 0x7f35800a6a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:21.387 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:21.387 7f357ffff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3580193230 0x7f35800a6a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:54468/0 (socket says 172.21.2.102:54468) 2021-02-10T11:21:21.387 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:21.387 7f357ffff700 1 -- 172.21.2.102:0/3201189448 learned_addr learned my addr 172.21.2.102:0/3201189448 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:21:21.387 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:21.387 7f357f7fe700 1 --2- 172.21.2.102:0/3201189448 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f358008f5b0 0x7f358008f9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:21.387 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:21.387 7f3584c16700 1 --2- 172.21.2.102:0/3201189448 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f358008e6a0 0x7f358008eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:21.387 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:21.388 7f357ffff700 1 -- 172.21.2.102:0/3201189448 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f358008e6a0 msgr2=0x7f358008eaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:21.387 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:21.388 7f357ffff700 1 --2- 172.21.2.102:0/3201189448 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f358008e6a0 0x7f358008eaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:21.387 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:21.388 7f357ffff700 1 -- 172.21.2.102:0/3201189448 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f358008f5b0 msgr2=0x7f358008f9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:21.387 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:21.388 7f357ffff700 1 --2- 172.21.2.102:0/3201189448 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f358008f5b0 0x7f358008f9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:21.387 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:21.388 7f357ffff700 1 -- 172.21.2.102:0/3201189448 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f358008f2a0 con 0x7f3580193230 2021-02-10T11:21:21.388 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:21.388 7f3584c16700 1 --2- 172.21.2.102:0/3201189448 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f358008e6a0 0x7f358008eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:21:21.388 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:21.388 7f357ffff700 1 --2- 172.21.2.102:0/3201189448 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3580193230 0x7f35800a6a80 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 rx=0x7f3570009980 tx=0x7f357000bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:21.388 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:21.388 7f357cff9700 1 -- 172.21.2.102:0/3201189448 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f357000be10 con 0x7f3580193230 2021-02-10T11:21:21.388 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:21.388 7f357cff9700 1 -- 172.21.2.102:0/3201189448 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f357000c2f0 con 0x7f3580193230 2021-02-10T11:21:21.388 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:21.388 7f35865f9700 1 -- 172.21.2.102:0/3201189448 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f358008f2a0 con 0x7f3580193230 2021-02-10T11:21:21.388 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:21.388 7f357cff9700 1 -- 172.21.2.102:0/3201189448 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f357000c810 con 0x7f3580193230 2021-02-10T11:21:21.388 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:21.388 7f35865f9700 1 -- 172.21.2.102:0/3201189448 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35801924f0 con 0x7f3580193230 2021-02-10T11:21:21.388 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:21.389 7f357cff9700 1 -- 172.21.2.102:0/3201189448 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f3570003070 con 0x7f3580193230 2021-02-10T11:21:21.388 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:21.389 7f357cff9700 1 --2- 172.21.2.102:0/3201189448 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3564018730 0x7f356401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:21.388 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:21.389 7f357cff9700 1 -- 172.21.2.102:0/3201189448 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f3570032a40 con 0x7f3580193230 2021-02-10T11:21:21.388 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:21.389 7f357f7fe700 1 --2- 172.21.2.102:0/3201189448 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3564018730 0x7f356401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:21.389 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:21.389 7f357f7fe700 1 --2- 172.21.2.102:0/3201189448 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3564018730 0x7f356401ac10 crc :-1 s=READY pgs=443 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:21.414 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:21.418 7f35865f9700 1 -- 172.21.2.102:0/3201189448 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35800f0b50 con 0x7f3580193230 2021-02-10T11:21:21.416 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:21.421 7f357cff9700 1 -- 172.21.2.102:0/3201189448 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f3570000f00 con 0x7f3580193230 2021-02-10T11:21:21.545 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:21.549 7f35865f9700 1 -- 172.21.2.102:0/3201189448 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344420", "target": ["mgr", ""]}) v1 -- 0x7f3580090240 con 0x7f3564018730 2021-02-10T11:21:21.556 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:21.560 7f357cff9700 1 -- 172.21.2.102:0/3201189448 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f35680014b0 con 0x7f3564018730 2021-02-10T11:21:21.562 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:21.566 7f35865f9700 1 -- 172.21.2.102:0/3201189448 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3564018730 msgr2=0x7f356401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:21.562 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:21.566 7f35865f9700 1 --2- 172.21.2.102:0/3201189448 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3564018730 0x7f356401ac10 crc :-1 s=READY pgs=443 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:21.562 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:21.566 7f35865f9700 1 -- 172.21.2.102:0/3201189448 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3580193230 msgr2=0x7f35800a6a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:21.562 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:21.566 7f35865f9700 1 --2- 172.21.2.102:0/3201189448 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3580193230 0x7f35800a6a80 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 rx=0x7f3570009980 tx=0x7f357000bcd0).stop 2021-02-10T11:21:21.562 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:21.566 7f35865f9700 1 -- 172.21.2.102:0/3201189448 shutdown_connections 2021-02-10T11:21:21.562 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:21.566 7f35865f9700 1 --2- 172.21.2.102:0/3201189448 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f358008e6a0 0x7f358008eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:21.562 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:21.566 7f35865f9700 1 --2- 172.21.2.102:0/3201189448 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3564018730 0x7f356401ac10 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:21.562 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:21.566 7f35865f9700 1 --2- 172.21.2.102:0/3201189448 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f358008f5b0 0x7f358008f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:21.562 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:21.566 7f35865f9700 1 --2- 172.21.2.102:0/3201189448 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3580193230 0x7f35800a6a80 unknown :-1 s=CLOSED pgs=907 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:21.563 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:21.566 7f35865f9700 1 -- 172.21.2.102:0/3201189448 >> 172.21.2.102:0/3201189448 conn(0x7f35800bf4e0 msgr2=0x7f35800be160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:21.563 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:21.566 7f35865f9700 1 -- 172.21.2.102:0/3201189448 shutdown_connections 2021-02-10T11:21:21.563 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:21.566 7f35865f9700 1 -- 172.21.2.102:0/3201189448 wait complete. 2021-02-10T11:21:21.571 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000344421 2021-02-10T11:21:22.002 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:21.989 7fc1d1a32700 1 Processor -- start 2021-02-10T11:21:22.003 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:21.990 7fc1d1a32700 1 -- start start 2021-02-10T11:21:22.003 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:21.990 7fc1d1a32700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc1cc1b0870 0x7fc1cc0f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:22.003 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:21.990 7fc1d1a32700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fc1cc0efe00 0x7fc1cc0f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:22.003 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:21.990 7fc1d1a32700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1cc1b0c70 con 0x7fc1cc1b0470 2021-02-10T11:21:22.003 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:21.990 7fc1d1a32700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1cc0f0b50 con 0x7fc1cc0efe00 2021-02-10T11:21:22.004 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:21.990 7fc1d1a32700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1cc0f0820 con 0x7fc1cc1b0870 2021-02-10T11:21:22.004 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:21.990 7fc1cbfff700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fc1cc0efe00 0x7fc1cc0f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:22.004 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:21.990 7fc1cbfff700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fc1cc0efe00 0x7fc1cc0f0200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.113:3300/0 says I am v2:172.21.2.102:57906/0 (socket says 172.21.2.102:57906) 2021-02-10T11:21:22.004 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:21.990 7fc1cbfff700 1 -- 172.21.2.102:0/3140350824 learned_addr learned my addr 172.21.2.102:0/3140350824 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:21:22.004 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:21.990 7fc1caffd700 1 --2- 172.21.2.102:0/3140350824 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc1cc1b0870 0x7fc1cc0f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:22.004 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:21.991 7fc1cbfff700 1 -- 172.21.2.102:0/3140350824 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc1cc1b0870 msgr2=0x7fc1cc0f20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:21:22.005 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:21.991 7fc1cbfff700 1 --2- 172.21.2.102:0/3140350824 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc1cc1b0870 0x7fc1cc0f20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:22.005 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:21.991 7fc1cbfff700 1 -- 172.21.2.102:0/3140350824 >> v1:172.21.2.113:6789/0 conn(0x7fc1cc1b0470 legacy=0x7fc1cc0ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:22.005 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:21.991 7fc1cbfff700 1 -- 172.21.2.102:0/3140350824 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1cc0e52e0 con 0x7fc1cc0efe00 2021-02-10T11:21:22.005 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:21.991 7fc1cbfff700 1 --2- 172.21.2.102:0/3140350824 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc1cc0efe00 0x7fc1cc0f0200 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 rx=0x7fc1c4006aa0 tx=0x7fc1c4009f20).ready entity=mon.1 client_cookie=39643a76e0e489a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:22.005 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:21.991 7fc1ca7fc700 1 -- 172.21.2.102:0/3140350824 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc1c400b5c0 con 0x7fc1cc0efe00 2021-02-10T11:21:22.005 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:21.991 7fc1ca7fc700 1 -- 172.21.2.102:0/3140350824 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1c4003030 con 0x7fc1cc0efe00 2021-02-10T11:21:22.005 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:21.991 7fc1ca7fc700 1 -- 172.21.2.102:0/3140350824 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc1c400b9f0 con 0x7fc1cc0efe00 2021-02-10T11:21:22.005 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:21.991 7fc1d1a32700 1 -- 172.21.2.102:0/3140350824 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc1cc0efe00 msgr2=0x7fc1cc0f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:22.005 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:21.991 7fc1d1a32700 1 --2- 172.21.2.102:0/3140350824 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc1cc0efe00 0x7fc1cc0f0200 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 rx=0x7fc1c4006aa0 tx=0x7fc1c4009f20).stop 2021-02-10T11:21:22.005 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:21.992 7fc1d1a32700 1 -- 172.21.2.102:0/3140350824 shutdown_connections 2021-02-10T11:21:22.005 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:21.992 7fc1d1a32700 1 --2- 172.21.2.102:0/3140350824 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc1cc1b0870 0x7fc1cc0f20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:22.005 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:21.992 7fc1d1a32700 1 --2- 172.21.2.102:0/3140350824 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc1cc0efe00 0x7fc1cc0f0200 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:22.006 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:21.992 7fc1d1a32700 1 -- 172.21.2.102:0/3140350824 >> 172.21.2.102:0/3140350824 conn(0x7fc1cc0ce3e0 msgr2=0x7fc1cc0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:22.006 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:21.992 7fc1d1a32700 1 -- 172.21.2.102:0/3140350824 shutdown_connections 2021-02-10T11:21:22.006 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:21.992 7fc1d1a32700 1 -- 172.21.2.102:0/3140350824 wait complete. 2021-02-10T11:21:22.006 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:22.007 7fc1d1a32700 1 Processor -- start 2021-02-10T11:21:22.006 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:22.007 7fc1d1a32700 1 -- start start 2021-02-10T11:21:22.008 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:22.007 7fc1d1a32700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc1cc190920 0x7fc1cc0b81a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:22.008 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:22.007 7fc1d1a32700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc1cc0b87c0 0x7fc1cc0b70c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:22.008 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:22.007 7fc1d1a32700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc1cc0b8bc0 0x7fc1cc0b76e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:22.008 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:22.007 7fc1d1a32700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1cc190d20 con 0x7fc1cc190920 2021-02-10T11:21:22.008 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:22.007 7fc1d1a32700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1cc0b5ad0 con 0x7fc1cc0b8bc0 2021-02-10T11:21:22.008 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:22.007 7fc1d1a32700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1cc0b7d00 con 0x7fc1cc0b87c0 2021-02-10T11:21:22.009 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:22.008 7fc1cbfff700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc1cc0b8bc0 0x7fc1cc0b76e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:22.009 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:22.008 7fc1cb7fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc1cc190920 0x7fc1cc0b81a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:22.009 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:22.008 7fc1cb7fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc1cc190920 0x7fc1cc0b81a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:54478/0 (socket says 172.21.2.102:54478) 2021-02-10T11:21:22.009 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:22.008 7fc1cb7fe700 1 -- 172.21.2.102:0/1851774167 learned_addr learned my addr 172.21.2.102:0/1851774167 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:21:22.009 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:22.008 7fc1caffd700 1 --2- 172.21.2.102:0/1851774167 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc1cc0b87c0 0x7fc1cc0b70c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:22.009 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:22.008 7fc1cb7fe700 1 -- 172.21.2.102:0/1851774167 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc1cc0b87c0 msgr2=0x7fc1cc0b70c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:22.009 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:22.008 7fc1cb7fe700 1 --2- 172.21.2.102:0/1851774167 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc1cc0b87c0 0x7fc1cc0b70c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:22.009 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:22.008 7fc1cb7fe700 1 -- 172.21.2.102:0/1851774167 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc1cc0b8bc0 msgr2=0x7fc1cc0b76e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:22.009 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:22.008 7fc1cb7fe700 1 --2- 172.21.2.102:0/1851774167 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc1cc0b8bc0 0x7fc1cc0b76e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:22.009 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:22.008 7fc1cb7fe700 1 -- 172.21.2.102:0/1851774167 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1cc0a4590 con 0x7fc1cc190920 2021-02-10T11:21:22.009 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:22.008 7fc1cb7fe700 1 --2- 172.21.2.102:0/1851774167 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc1cc190920 0x7fc1cc0b81a0 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 rx=0x7fc1b4006ed0 tx=0x7fc1b400b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:22.009 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:22.008 7fc1c3fff700 1 -- 172.21.2.102:0/1851774167 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc1b4006240 con 0x7fc1cc190920 2021-02-10T11:21:22.009 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:22.008 7fc1c3fff700 1 -- 172.21.2.102:0/1851774167 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1b40054b0 con 0x7fc1cc190920 2021-02-10T11:21:22.010 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:22.008 7fc1c3fff700 1 -- 172.21.2.102:0/1851774167 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc1b4005900 con 0x7fc1cc190920 2021-02-10T11:21:22.010 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:22.008 7fc1d1a32700 1 -- 172.21.2.102:0/1851774167 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1cc0a4590 con 0x7fc1cc190920 2021-02-10T11:21:22.010 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:22.009 7fc1d1a32700 1 -- 172.21.2.102:0/1851774167 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1cc0a4590 con 0x7fc1cc190920 2021-02-10T11:21:22.010 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:22.009 7fc1c3fff700 1 -- 172.21.2.102:0/1851774167 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fc1b4006030 con 0x7fc1cc190920 2021-02-10T11:21:22.010 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:22.009 7fc1c3fff700 1 --2- 172.21.2.102:0/1851774167 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc1b00186c0 0x7fc1b001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:22.010 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:22.009 7fc1c3fff700 1 -- 172.21.2.102:0/1851774167 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fc1b40327f0 con 0x7fc1cc190920 2021-02-10T11:21:22.010 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:22.009 7fc1caffd700 1 --2- 172.21.2.102:0/1851774167 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc1b00186c0 0x7fc1b001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:22.010 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:22.009 7fc1caffd700 1 --2- 172.21.2.102:0/1851774167 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc1b00186c0 0x7fc1b001abc0 crc :-1 s=READY pgs=444 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:22.035 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:22.040 7fc1d1a32700 1 -- 172.21.2.102:0/1851774167 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1cc105ae0 con 0x7fc1cc190920 2021-02-10T11:21:22.038 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:22.042 7fc1c3fff700 1 -- 172.21.2.102:0/1851774167 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fc1b4000de0 con 0x7fc1cc190920 2021-02-10T11:21:22.164 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:22.169 7fc1d1a32700 1 -- 172.21.2.102:0/1851774167 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344421", "target": ["mgr", ""]}) v1 -- 0x7fc1cc0f0b50 con 0x7fc1b00186c0 2021-02-10T11:21:22.174 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:22.179 7fc1c3fff700 1 -- 172.21.2.102:0/1851774167 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fc1bc006a70 con 0x7fc1b00186c0 2021-02-10T11:21:22.181 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:22.186 7fc1d1a32700 1 -- 172.21.2.102:0/1851774167 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc1b00186c0 msgr2=0x7fc1b001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:22.181 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:22.186 7fc1d1a32700 1 --2- 172.21.2.102:0/1851774167 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc1b00186c0 0x7fc1b001abc0 crc :-1 s=READY pgs=444 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:22.181 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:22.186 7fc1d1a32700 1 -- 172.21.2.102:0/1851774167 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc1cc190920 msgr2=0x7fc1cc0b81a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:22.181 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:22.186 7fc1d1a32700 1 --2- 172.21.2.102:0/1851774167 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc1cc190920 0x7fc1cc0b81a0 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 rx=0x7fc1b4006ed0 tx=0x7fc1b400b960).stop 2021-02-10T11:21:22.182 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:22.186 7fc1d1a32700 1 -- 172.21.2.102:0/1851774167 shutdown_connections 2021-02-10T11:21:22.182 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:22.186 7fc1d1a32700 1 --2- 172.21.2.102:0/1851774167 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc1b00186c0 0x7fc1b001abc0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:22.182 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:22.186 7fc1d1a32700 1 --2- 172.21.2.102:0/1851774167 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc1cc0b8bc0 0x7fc1cc0b76e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:22.182 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:22.186 7fc1d1a32700 1 --2- 172.21.2.102:0/1851774167 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc1cc0b87c0 0x7fc1cc0b70c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:22.182 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:22.186 7fc1d1a32700 1 --2- 172.21.2.102:0/1851774167 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc1cc190920 0x7fc1cc0b81a0 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:22.182 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:22.186 7fc1d1a32700 1 -- 172.21.2.102:0/1851774167 >> 172.21.2.102:0/1851774167 conn(0x7fc1cc08ffe0 msgr2=0x7fc1cc0bf050 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:22.182 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:22.186 7fc1d1a32700 1 -- 172.21.2.102:0/1851774167 shutdown_connections 2021-02-10T11:21:22.182 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:22.186 7fc1d1a32700 1 -- 172.21.2.102:0/1851774167 wait complete. 2021-02-10T11:21:22.190 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000344422 2021-02-10T11:21:22.621 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:22.609 7f5cea711700 1 Processor -- start 2021-02-10T11:21:22.621 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:22.609 7f5cea711700 1 -- start start 2021-02-10T11:21:22.621 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:22.609 7f5cea711700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f5ce41b0470 0x7f5ce41b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:22.621 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:22.609 7f5cea711700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f5ce40f20e0 0x7f5ce40f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:22.621 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:22.609 7f5cea711700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ce40f0b50 con 0x7f5ce40ef610 2021-02-10T11:21:22.621 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:22.609 7f5cea711700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ce40f0820 con 0x7f5ce41b0470 2021-02-10T11:21:22.621 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:22.609 7f5cea711700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ce40f24e0 con 0x7f5ce40f20e0 2021-02-10T11:21:22.622 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:22.610 7f5ce3fff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f5ce41b0470 0x7f5ce41b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:22.622 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:22.610 7f5ce37fe700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f5ce40ef610 0x7f5ce40efa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:46710/0 (socket says 172.21.2.102:46710) 2021-02-10T11:21:22.622 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:22.610 7f5ce37fe700 1 -- 172.21.2.102:0/2829397442 learned_addr learned my addr 172.21.2.102:0/2829397442 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:21:22.622 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:22.610 7f5ce8d2e700 1 --2- 172.21.2.102:0/2829397442 >> v2:172.21.2.113:3300/0 conn(0x7f5ce40f20e0 0x7f5ce40f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:22.622 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:22.610 7f5ce3fff700 1 -- 172.21.2.102:0/2829397442 >> v2:172.21.2.113:3300/0 conn(0x7f5ce40f20e0 msgr2=0x7f5ce40f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:21:22.622 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:22.610 7f5ce3fff700 1 --2- 172.21.2.102:0/2829397442 >> v2:172.21.2.113:3300/0 conn(0x7f5ce40f20e0 0x7f5ce40f0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:22.622 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:22.610 7f5ce3fff700 1 -- 172.21.2.102:0/2829397442 >> v1:172.21.2.102:6789/0 conn(0x7f5ce40ef610 legacy=0x7f5ce40efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:22.622 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:22.610 7f5ce3fff700 1 -- 172.21.2.102:0/2829397442 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ce40e52e0 con 0x7f5ce41b0470 2021-02-10T11:21:22.622 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:22.610 7f5ce3fff700 1 --2- 172.21.2.102:0/2829397442 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5ce41b0470 0x7f5ce41b0870 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 rx=0x7f5cd4006650 tx=0x7f5cd4009a80).ready entity=mon.0 client_cookie=a4afd4231463f13c server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:22.622 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:22.611 7f5ce2ffd700 1 -- 172.21.2.102:0/2829397442 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5cd400c0c0 con 0x7f5ce41b0470 2021-02-10T11:21:22.622 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:22.611 7f5ce2ffd700 1 -- 172.21.2.102:0/2829397442 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cd400c340 con 0x7f5ce41b0470 2021-02-10T11:21:22.622 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:22.611 7f5ce2ffd700 1 -- 172.21.2.102:0/2829397442 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5cd400c7a0 con 0x7f5ce41b0470 2021-02-10T11:21:22.623 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:22.611 7f5cea711700 1 -- 172.21.2.102:0/2829397442 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5ce41b0470 msgr2=0x7f5ce41b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:22.623 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:22.611 7f5cea711700 1 --2- 172.21.2.102:0/2829397442 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5ce41b0470 0x7f5ce41b0870 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 rx=0x7f5cd4006650 tx=0x7f5cd4009a80).stop 2021-02-10T11:21:22.623 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:22.611 7f5cea711700 1 -- 172.21.2.102:0/2829397442 shutdown_connections 2021-02-10T11:21:22.623 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:22.611 7f5cea711700 1 --2- 172.21.2.102:0/2829397442 >> v2:172.21.2.113:3300/0 conn(0x7f5ce40f20e0 0x7f5ce40f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:22.623 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:22.611 7f5cea711700 1 --2- 172.21.2.102:0/2829397442 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5ce41b0470 0x7f5ce41b0870 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:22.623 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:22.611 7f5cea711700 1 -- 172.21.2.102:0/2829397442 >> 172.21.2.102:0/2829397442 conn(0x7f5ce40ce3e0 msgr2=0x7f5ce40e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:22.623 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:22.611 7f5cea711700 1 -- 172.21.2.102:0/2829397442 shutdown_connections 2021-02-10T11:21:22.623 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:22.611 7f5cea711700 1 -- 172.21.2.102:0/2829397442 wait complete. 2021-02-10T11:21:22.625 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:22.626 7f5cea711700 1 Processor -- start 2021-02-10T11:21:22.625 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:22.626 7f5cea711700 1 -- start start 2021-02-10T11:21:22.625 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:22.626 7f5cea711700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5ce4105d10 0x7f5ce409f880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:22.625 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:22.626 7f5cea711700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5ce40961f0 0x7f5ce40965f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:22.625 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:22.626 7f5cea711700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5ce4106890 0x7f5ce4106c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:22.625 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:22.626 7f5cea711700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ce4106110 con 0x7f5ce40961f0 2021-02-10T11:21:22.625 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:22.626 7f5cea711700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ce4106380 con 0x7f5ce4106890 2021-02-10T11:21:22.625 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:22.626 7f5cea711700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ce40a6620 con 0x7f5ce4105d10 2021-02-10T11:21:22.625 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:22.627 7f5ce37fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5ce40961f0 0x7f5ce40965f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:22.625 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:22.627 7f5ce37fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5ce40961f0 0x7f5ce40965f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:54492/0 (socket says 172.21.2.102:54492) 2021-02-10T11:21:22.626 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:22.627 7f5ce37fe700 1 -- 172.21.2.102:0/4264961862 learned_addr learned my addr 172.21.2.102:0/4264961862 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:21:22.626 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:22.627 7f5ce8d2e700 1 --2- 172.21.2.102:0/4264961862 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5ce4106890 0x7f5ce4106c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:22.626 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:22.627 7f5ce3fff700 1 --2- 172.21.2.102:0/4264961862 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5ce4105d10 0x7f5ce409f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:22.626 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:22.627 7f5ce37fe700 1 -- 172.21.2.102:0/4264961862 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5ce4105d10 msgr2=0x7f5ce409f880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:22.626 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:22.627 7f5ce37fe700 1 --2- 172.21.2.102:0/4264961862 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5ce4105d10 0x7f5ce409f880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:22.626 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:22.627 7f5ce37fe700 1 -- 172.21.2.102:0/4264961862 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5ce4106890 msgr2=0x7f5ce4106c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:22.626 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:22.627 7f5ce37fe700 1 --2- 172.21.2.102:0/4264961862 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5ce4106890 0x7f5ce4106c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:22.626 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:22.627 7f5ce37fe700 1 -- 172.21.2.102:0/4264961862 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ce41072b0 con 0x7f5ce40961f0 2021-02-10T11:21:22.626 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:22.627 7f5ce37fe700 1 --2- 172.21.2.102:0/4264961862 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5ce40961f0 0x7f5ce40965f0 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 rx=0x7f5cd800a4f0 tx=0x7f5cd800a890).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:22.626 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:22.627 7f5ce0ff9700 1 -- 172.21.2.102:0/4264961862 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5cd800ac80 con 0x7f5ce40961f0 2021-02-10T11:21:22.626 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:22.627 7f5ce0ff9700 1 -- 172.21.2.102:0/4264961862 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cd80009b0 con 0x7f5ce40961f0 2021-02-10T11:21:22.626 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:22.627 7f5cea711700 1 -- 172.21.2.102:0/4264961862 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ce41072b0 con 0x7f5ce40961f0 2021-02-10T11:21:22.626 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:22.627 7f5cea711700 1 -- 172.21.2.102:0/4264961862 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ce41072b0 con 0x7f5ce40961f0 2021-02-10T11:21:22.627 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:22.627 7f5ce0ff9700 1 -- 172.21.2.102:0/4264961862 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5cd8004590 con 0x7f5ce40961f0 2021-02-10T11:21:22.627 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:22.628 7f5ce0ff9700 1 -- 172.21.2.102:0/4264961862 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f5cd80043a0 con 0x7f5ce40961f0 2021-02-10T11:21:22.627 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:22.628 7f5ce0ff9700 1 --2- 172.21.2.102:0/4264961862 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5cc4018730 0x7f5cc401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:22.627 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:22.628 7f5ce0ff9700 1 -- 172.21.2.102:0/4264961862 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f5cd8032830 con 0x7f5ce40961f0 2021-02-10T11:21:22.627 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:22.628 7f5ce3fff700 1 --2- 172.21.2.102:0/4264961862 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5cc4018730 0x7f5cc401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:22.627 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:22.628 7f5ce3fff700 1 --2- 172.21.2.102:0/4264961862 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5cc4018730 0x7f5cc401ac10 crc :-1 s=READY pgs=445 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:22.655 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:22.659 7f5cea711700 1 -- 172.21.2.102:0/4264961862 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ce4105ae0 con 0x7f5ce40961f0 2021-02-10T11:21:22.657 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:22.662 7f5ce0ff9700 1 -- 172.21.2.102:0/4264961862 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f5cd8007070 con 0x7f5ce40961f0 2021-02-10T11:21:22.786 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:22.791 7f5cea711700 1 -- 172.21.2.102:0/4264961862 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344422", "target": ["mgr", ""]}) v1 -- 0x7f5ce40f0b50 con 0x7f5cc4018730 2021-02-10T11:21:22.797 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:22.801 7f5ce0ff9700 1 -- 172.21.2.102:0/4264961862 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f5cd4000c90 con 0x7f5cc4018730 2021-02-10T11:21:22.803 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:22.807 7f5cea711700 1 -- 172.21.2.102:0/4264961862 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5cc4018730 msgr2=0x7f5cc401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:22.804 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:22.807 7f5cea711700 1 --2- 172.21.2.102:0/4264961862 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5cc4018730 0x7f5cc401ac10 crc :-1 s=READY pgs=445 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:22.804 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:22.807 7f5cea711700 1 -- 172.21.2.102:0/4264961862 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5ce40961f0 msgr2=0x7f5ce40965f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:22.804 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:22.807 7f5cea711700 1 --2- 172.21.2.102:0/4264961862 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5ce40961f0 0x7f5ce40965f0 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 rx=0x7f5cd800a4f0 tx=0x7f5cd800a890).stop 2021-02-10T11:21:22.804 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:22.808 7f5cea711700 1 -- 172.21.2.102:0/4264961862 shutdown_connections 2021-02-10T11:21:22.804 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:22.808 7f5cea711700 1 --2- 172.21.2.102:0/4264961862 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5cc4018730 0x7f5cc401ac10 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:22.804 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:22.808 7f5cea711700 1 --2- 172.21.2.102:0/4264961862 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5ce4106890 0x7f5ce4106c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:22.804 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:22.808 7f5cea711700 1 --2- 172.21.2.102:0/4264961862 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5ce40961f0 0x7f5ce40965f0 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:22.804 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:22.808 7f5cea711700 1 --2- 172.21.2.102:0/4264961862 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5ce4105d10 0x7f5ce409f880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:22.804 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:22.808 7f5cea711700 1 -- 172.21.2.102:0/4264961862 >> 172.21.2.102:0/4264961862 conn(0x7f5ce4095a70 msgr2=0x7f5ce40bf050 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:22.804 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:22.808 7f5cea711700 1 -- 172.21.2.102:0/4264961862 shutdown_connections 2021-02-10T11:21:22.805 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:22.808 7f5cea711700 1 -- 172.21.2.102:0/4264961862 wait complete. 2021-02-10T11:21:22.812 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000344423 2021-02-10T11:21:23.239 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:23.227 7f1e42f70700 1 Processor -- start 2021-02-10T11:21:23.239 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:23.227 7f1e42f70700 1 -- start start 2021-02-10T11:21:23.239 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:23.228 7f1e42f70700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f1e3c0f05f0 0x7f1e3c0f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:23.239 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:23.228 7f1e42f70700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e3c1b0c70 con 0x7f1e3c1b0470 2021-02-10T11:21:23.239 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:23.228 7f1e42f70700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e3c0f0b50 con 0x7f1e3c1b0870 2021-02-10T11:21:23.239 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:23.228 7f1e42f70700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e3c0e5ea0 con 0x7f1e3c0f05f0 2021-02-10T11:21:23.239 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:23.228 7f1e4158d700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f1e3c0f05f0 0x7f1e3c0f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:23.239 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:23.228 7f1e4158d700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f1e3c0f05f0 0x7f1e3c0f20e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:54504/0 (socket says 172.21.2.102:54504) 2021-02-10T11:21:23.239 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:23.228 7f1e4158d700 1 -- 172.21.2.102:0/3764874961 learned_addr learned my addr 172.21.2.102:0/3764874961 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:21:23.239 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:23.228 7f1e40d8c700 1 --1- 172.21.2.102:0/3764874961 >> v1:172.21.2.102:6789/0 conn(0x7f1e3c1b0470 0x7f1e3c0ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:46724/0 (socket says 172.21.2.102:46724) 2021-02-10T11:21:23.239 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:23.228 7f1e3b7fe700 1 -- 172.21.2.102:0/3764874961 <== mon.1 v1:172.21.2.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 307434546 0 0) 0x7f1e28000ff0 con 0x7f1e3c1b0870 2021-02-10T11:21:23.240 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:23.228 7f1e3b7fe700 1 -- 172.21.2.102:0/3764874961 --> v1:172.21.2.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e20002fe0 con 0x7f1e3c1b0870 2021-02-10T11:21:23.240 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:23.228 7f1e3b7fe700 1 -- 172.21.2.102:0/3764874961 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2075422182 0 0) 0x7f1e2c000ce0 con 0x7f1e3c1b0470 2021-02-10T11:21:23.240 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:23.228 7f1e4158d700 1 -- 172.21.2.102:0/3764874961 >> v1:172.21.2.113:6789/0 conn(0x7f1e3c1b0870 legacy=0x7f1e3c0efe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:23.240 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:23.228 7f1e4158d700 1 -- 172.21.2.102:0/3764874961 >> v1:172.21.2.102:6789/0 conn(0x7f1e3c1b0470 legacy=0x7f1e3c0ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:23.240 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:23.228 7f1e4158d700 1 -- 172.21.2.102:0/3764874961 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e3c0e52e0 con 0x7f1e3c0f05f0 2021-02-10T11:21:23.240 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:23.228 7f1e4158d700 1 --2- 172.21.2.102:0/3764874961 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1e3c0f05f0 0x7f1e3c0f20e0 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 rx=0x7f1e34006f00 tx=0x7f1e34009e90).ready entity=mon.0 client_cookie=400d1d21b9b3d87b server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:23.240 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:23.229 7f1e3b7fe700 1 -- 172.21.2.102:0/3764874961 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1e3400b300 con 0x7f1e3c0f05f0 2021-02-10T11:21:23.240 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:23.229 7f1e3b7fe700 1 -- 172.21.2.102:0/3764874961 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e3400b640 con 0x7f1e3c0f05f0 2021-02-10T11:21:23.240 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:23.229 7f1e3b7fe700 1 -- 172.21.2.102:0/3764874961 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1e3400ba90 con 0x7f1e3c0f05f0 2021-02-10T11:21:23.240 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:23.229 7f1e42f70700 1 -- 172.21.2.102:0/3764874961 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1e3c0f05f0 msgr2=0x7f1e3c0f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:23.240 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:23.229 7f1e42f70700 1 --2- 172.21.2.102:0/3764874961 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1e3c0f05f0 0x7f1e3c0f20e0 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 rx=0x7f1e34006f00 tx=0x7f1e34009e90).stop 2021-02-10T11:21:23.240 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:23.229 7f1e42f70700 1 -- 172.21.2.102:0/3764874961 shutdown_connections 2021-02-10T11:21:23.240 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:23.229 7f1e42f70700 1 --2- 172.21.2.102:0/3764874961 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1e3c0f05f0 0x7f1e3c0f20e0 unknown :-1 s=CLOSED pgs=913 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:23.240 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:23.229 7f1e42f70700 1 -- 172.21.2.102:0/3764874961 >> 172.21.2.102:0/3764874961 conn(0x7f1e3c0ce3e0 msgr2=0x7f1e3c0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:23.241 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:23.229 7f1e42f70700 1 -- 172.21.2.102:0/3764874961 shutdown_connections 2021-02-10T11:21:23.241 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:23.229 7f1e42f70700 1 -- 172.21.2.102:0/3764874961 wait complete. 2021-02-10T11:21:23.242 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:23.244 7f1e42f70700 1 Processor -- start 2021-02-10T11:21:23.242 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:23.244 7f1e42f70700 1 -- start start 2021-02-10T11:21:23.243 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:23.244 7f1e42f70700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1e3c0a0420 0x7f1e3c0a0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:23.243 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:23.244 7f1e42f70700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1e3c189d00 0x7f1e3c18a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:23.243 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:23.244 7f1e42f70700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1e3c18a720 0x7f1e3c0b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:23.243 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:23.244 7f1e42f70700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e3c189780 con 0x7f1e3c189d00 2021-02-10T11:21:23.243 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:23.244 7f1e42f70700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e3c18ab20 con 0x7f1e3c0a0420 2021-02-10T11:21:23.243 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:23.244 7f1e42f70700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e3c0a1140 con 0x7f1e3c18a720 2021-02-10T11:21:23.243 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:23.244 7f1e3bfff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1e3c189d00 0x7f1e3c18a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:23.243 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:23.244 7f1e3bfff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1e3c189d00 0x7f1e3c18a100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:54506/0 (socket says 172.21.2.102:54506) 2021-02-10T11:21:23.243 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:23.244 7f1e3bfff700 1 -- 172.21.2.102:0/3524409160 learned_addr learned my addr 172.21.2.102:0/3524409160 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:21:23.243 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:23.244 7f1e40d8c700 1 --2- 172.21.2.102:0/3524409160 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1e3c0a0420 0x7f1e3c0a0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:23.243 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:23.245 7f1e4158d700 1 --2- 172.21.2.102:0/3524409160 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1e3c18a720 0x7f1e3c0b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:23.243 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:23.245 7f1e3bfff700 1 -- 172.21.2.102:0/3524409160 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1e3c18a720 msgr2=0x7f1e3c0b7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:23.244 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:23.245 7f1e3bfff700 1 --2- 172.21.2.102:0/3524409160 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1e3c18a720 0x7f1e3c0b7be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:23.244 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:23.245 7f1e3bfff700 1 -- 172.21.2.102:0/3524409160 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1e3c0a0420 msgr2=0x7f1e3c0a0820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:23.244 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:23.245 7f1e3bfff700 1 --2- 172.21.2.102:0/3524409160 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1e3c0a0420 0x7f1e3c0a0820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:23.244 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:23.245 7f1e3bfff700 1 -- 172.21.2.102:0/3524409160 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e3c0b8200 con 0x7f1e3c189d00 2021-02-10T11:21:23.244 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:23.245 7f1e3bfff700 1 --2- 172.21.2.102:0/3524409160 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1e3c189d00 0x7f1e3c18a100 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 rx=0x7f1e28006ed0 tx=0x7f1e2800b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:23.244 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:23.245 7f1e397fa700 1 -- 172.21.2.102:0/3524409160 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1e2800bd50 con 0x7f1e3c189d00 2021-02-10T11:21:23.244 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:23.245 7f1e397fa700 1 -- 172.21.2.102:0/3524409160 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e28006240 con 0x7f1e3c189d00 2021-02-10T11:21:23.244 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:23.245 7f1e42f70700 1 -- 172.21.2.102:0/3524409160 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e3c0b8200 con 0x7f1e3c189d00 2021-02-10T11:21:23.244 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:23.245 7f1e397fa700 1 -- 172.21.2.102:0/3524409160 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1e28005730 con 0x7f1e3c189d00 2021-02-10T11:21:23.244 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:23.245 7f1e42f70700 1 -- 172.21.2.102:0/3524409160 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e3c0b8200 con 0x7f1e3c189d00 2021-02-10T11:21:23.244 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:23.246 7f1e397fa700 1 -- 172.21.2.102:0/3524409160 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1e280054b0 con 0x7f1e3c189d00 2021-02-10T11:21:23.244 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:23.246 7f1e397fa700 1 --2- 172.21.2.102:0/3524409160 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1e18018730 0x7f1e1801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:23.244 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:23.246 7f1e397fa700 1 -- 172.21.2.102:0/3524409160 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f1e28032660 con 0x7f1e3c189d00 2021-02-10T11:21:23.244 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:23.246 7f1e40d8c700 1 --2- 172.21.2.102:0/3524409160 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1e18018730 0x7f1e1801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:23.245 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:23.246 7f1e40d8c700 1 --2- 172.21.2.102:0/3524409160 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1e18018730 0x7f1e1801ac10 crc :-1 s=READY pgs=446 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:23.273 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:23.277 7f1e42f70700 1 -- 172.21.2.102:0/3524409160 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e3c105ae0 con 0x7f1e3c189d00 2021-02-10T11:21:23.275 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:23.280 7f1e397fa700 1 -- 172.21.2.102:0/3524409160 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f1e28006030 con 0x7f1e3c189d00 2021-02-10T11:21:23.405 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:23.410 7f1e42f70700 1 -- 172.21.2.102:0/3524409160 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344423", "target": ["mgr", ""]}) v1 -- 0x7f1e3c0f0b50 con 0x7f1e18018730 2021-02-10T11:21:23.417 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:23.421 7f1e397fa700 1 -- 172.21.2.102:0/3524409160 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f1e2c006820 con 0x7f1e18018730 2021-02-10T11:21:23.422 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:23.426 7f1e42f70700 1 -- 172.21.2.102:0/3524409160 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1e18018730 msgr2=0x7f1e1801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:23.422 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:23.426 7f1e42f70700 1 --2- 172.21.2.102:0/3524409160 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1e18018730 0x7f1e1801ac10 crc :-1 s=READY pgs=446 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:23.422 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:23.426 7f1e42f70700 1 -- 172.21.2.102:0/3524409160 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1e3c189d00 msgr2=0x7f1e3c18a100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:23.422 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:23.426 7f1e42f70700 1 --2- 172.21.2.102:0/3524409160 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1e3c189d00 0x7f1e3c18a100 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 rx=0x7f1e28006ed0 tx=0x7f1e2800b960).stop 2021-02-10T11:21:23.422 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:23.427 7f1e42f70700 1 -- 172.21.2.102:0/3524409160 shutdown_connections 2021-02-10T11:21:23.422 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:23.427 7f1e42f70700 1 --2- 172.21.2.102:0/3524409160 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1e3c18a720 0x7f1e3c0b7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:23.422 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:23.427 7f1e42f70700 1 --2- 172.21.2.102:0/3524409160 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1e3c189d00 0x7f1e3c18a100 unknown :-1 s=CLOSED pgs=914 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:23.423 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:23.427 7f1e42f70700 1 --2- 172.21.2.102:0/3524409160 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1e18018730 0x7f1e1801ac10 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:23.423 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:23.427 7f1e42f70700 1 --2- 172.21.2.102:0/3524409160 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1e3c0a0420 0x7f1e3c0a0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:23.423 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:23.427 7f1e42f70700 1 -- 172.21.2.102:0/3524409160 >> 172.21.2.102:0/3524409160 conn(0x7f1e3c094e10 msgr2=0x7f1e3c0a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:23.423 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:23.427 7f1e42f70700 1 -- 172.21.2.102:0/3524409160 shutdown_connections 2021-02-10T11:21:23.423 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:23.427 7f1e42f70700 1 -- 172.21.2.102:0/3524409160 wait complete. 2021-02-10T11:21:23.431 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000344424 2021-02-10T11:21:23.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:23.850 7f28f97b8700 1 Processor -- start 2021-02-10T11:21:23.862 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:23.850 7f28f97b8700 1 -- start start 2021-02-10T11:21:23.862 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:23.850 7f28f97b8700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f28f41b0470 0x7f28f41b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:23.862 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:23.850 7f28f97b8700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f28f40f20e0 0x7f28f40f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:23.862 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:23.850 7f28f97b8700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28f40f0b50 con 0x7f28f40ef610 2021-02-10T11:21:23.862 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:23.850 7f28f97b8700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28f40f0820 con 0x7f28f41b0470 2021-02-10T11:21:23.862 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:23.850 7f28f97b8700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28f40f24e0 con 0x7f28f40f20e0 2021-02-10T11:21:23.862 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:23.850 7f28f3574700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f28f41b0470 0x7f28f41b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:23.862 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:23.850 7f28f3574700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f28f41b0470 0x7f28f41b0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:54514/0 (socket says 172.21.2.102:54514) 2021-02-10T11:21:23.862 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:23.850 7f28f3574700 1 -- 172.21.2.102:0/401447498 learned_addr learned my addr 172.21.2.102:0/401447498 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:21:23.862 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:23.850 7f28f3d75700 1 --2- 172.21.2.102:0/401447498 >> v2:172.21.2.113:3300/0 conn(0x7f28f40f20e0 0x7f28f40f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:23.863 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:23.850 7f28f3574700 1 -- 172.21.2.102:0/401447498 >> v2:172.21.2.113:3300/0 conn(0x7f28f40f20e0 msgr2=0x7f28f40f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:21:23.863 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:23.850 7f28f3574700 1 --2- 172.21.2.102:0/401447498 >> v2:172.21.2.113:3300/0 conn(0x7f28f40f20e0 0x7f28f40f0200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:23.863 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:23.850 7f28f3574700 1 -- 172.21.2.102:0/401447498 >> v1:172.21.2.102:6789/0 conn(0x7f28f40ef610 legacy=0x7f28f40efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:23.863 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:23.850 7f28f3574700 1 -- 172.21.2.102:0/401447498 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28f40e52e0 con 0x7f28f41b0470 2021-02-10T11:21:23.863 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:23.851 7f28f3574700 1 --2- 172.21.2.102:0/401447498 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f28f41b0470 0x7f28f41b0870 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 rx=0x7f28ec006a80 tx=0x7f28ec009f00).ready entity=mon.0 client_cookie=1c6ca799d12c9f53 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:23.863 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:23.851 7f28f2572700 1 -- 172.21.2.102:0/401447498 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f28ec00b5c0 con 0x7f28f41b0470 2021-02-10T11:21:23.863 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:23.851 7f28f2572700 1 -- 172.21.2.102:0/401447498 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28ec00b800 con 0x7f28f41b0470 2021-02-10T11:21:23.863 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:23.851 7f28f2572700 1 -- 172.21.2.102:0/401447498 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f28ec00bce0 con 0x7f28f41b0470 2021-02-10T11:21:23.863 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:23.851 7f28f97b8700 1 -- 172.21.2.102:0/401447498 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f28f41b0470 msgr2=0x7f28f41b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:23.863 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:23.851 7f28f97b8700 1 --2- 172.21.2.102:0/401447498 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f28f41b0470 0x7f28f41b0870 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 rx=0x7f28ec006a80 tx=0x7f28ec009f00).stop 2021-02-10T11:21:23.863 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:23.851 7f28f97b8700 1 -- 172.21.2.102:0/401447498 shutdown_connections 2021-02-10T11:21:23.864 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:23.851 7f28f97b8700 1 --2- 172.21.2.102:0/401447498 >> v2:172.21.2.113:3300/0 conn(0x7f28f40f20e0 0x7f28f40f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:23.864 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:23.851 7f28f97b8700 1 --2- 172.21.2.102:0/401447498 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f28f41b0470 0x7f28f41b0870 unknown :-1 s=CLOSED pgs=916 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:23.864 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:23.851 7f28f97b8700 1 -- 172.21.2.102:0/401447498 >> 172.21.2.102:0/401447498 conn(0x7f28f40ce3e0 msgr2=0x7f28f40e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:23.864 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:23.851 7f28f97b8700 1 -- 172.21.2.102:0/401447498 shutdown_connections 2021-02-10T11:21:23.864 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:23.851 7f28f97b8700 1 -- 172.21.2.102:0/401447498 wait complete. 2021-02-10T11:21:23.864 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:23.866 7f28f97b8700 1 Processor -- start 2021-02-10T11:21:23.866 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:23.866 7f28f97b8700 1 -- start start 2021-02-10T11:21:23.866 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:23.867 7f28f97b8700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f28f40ab760 0x7f28f41a74c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:23.866 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:23.867 7f28f97b8700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f28f40a2d80 0x7f28f41a7ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:23.866 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:23.867 7f28f97b8700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f28f40ba720 0x7f28f40bab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:23.866 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:23.867 7f28f97b8700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28f40ab260 con 0x7f28f40ba720 2021-02-10T11:21:23.866 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:23.867 7f28f97b8700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28f410a710 con 0x7f28f40ab760 2021-02-10T11:21:23.866 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:23.867 7f28f97b8700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28f41a8100 con 0x7f28f40a2d80 2021-02-10T11:21:23.866 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:23.867 7f28f3d75700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f28f40ba720 0x7f28f40bab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:23.867 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:23.867 7f28f3d75700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f28f40ba720 0x7f28f40bab20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:54524/0 (socket says 172.21.2.102:54524) 2021-02-10T11:21:23.867 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:23.867 7f28f3d75700 1 -- 172.21.2.102:0/3027850710 learned_addr learned my addr 172.21.2.102:0/3027850710 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:21:23.867 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:23.867 7f28f3574700 1 --2- 172.21.2.102:0/3027850710 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f28f40ab760 0x7f28f41a74c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:23.867 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:23.867 7f28f2d73700 1 --2- 172.21.2.102:0/3027850710 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f28f40a2d80 0x7f28f41a7ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:23.867 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:23.867 7f28f3d75700 1 -- 172.21.2.102:0/3027850710 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f28f40a2d80 msgr2=0x7f28f41a7ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:23.867 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:23.867 7f28f3d75700 1 --2- 172.21.2.102:0/3027850710 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f28f40a2d80 0x7f28f41a7ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:23.867 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:23.867 7f28f3d75700 1 -- 172.21.2.102:0/3027850710 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f28f40ab760 msgr2=0x7f28f41a74c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:23.867 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:23.867 7f28f3d75700 1 --2- 172.21.2.102:0/3027850710 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f28f40ab760 0x7f28f41a74c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:23.867 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:23.867 7f28f3d75700 1 -- 172.21.2.102:0/3027850710 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28f40bb140 con 0x7f28f40ba720 2021-02-10T11:21:23.867 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:23.867 7f28f3d75700 1 --2- 172.21.2.102:0/3027850710 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f28f40ba720 0x7f28f40bab20 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 rx=0x7f28e4006f10 tx=0x7f28e400b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:23.868 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:23.867 7f28dbfff700 1 -- 172.21.2.102:0/3027850710 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f28e400bd80 con 0x7f28f40ba720 2021-02-10T11:21:23.868 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:23.867 7f28dbfff700 1 -- 172.21.2.102:0/3027850710 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28e4002280 con 0x7f28f40ba720 2021-02-10T11:21:23.868 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:23.867 7f28dbfff700 1 -- 172.21.2.102:0/3027850710 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f28e4002760 con 0x7f28f40ba720 2021-02-10T11:21:23.868 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:23.867 7f28f97b8700 1 -- 172.21.2.102:0/3027850710 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28f40bb140 con 0x7f28f40ba720 2021-02-10T11:21:23.868 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:23.868 7f28f97b8700 1 -- 172.21.2.102:0/3027850710 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28f4109290 con 0x7f28f40ba720 2021-02-10T11:21:23.868 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:23.868 7f28dbfff700 1 -- 172.21.2.102:0/3027850710 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f28e4002460 con 0x7f28f40ba720 2021-02-10T11:21:23.868 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:23.868 7f28dbfff700 1 --2- 172.21.2.102:0/3027850710 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f28d40186d0 0x7f28d401abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:23.868 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:23.868 7f28dbfff700 1 -- 172.21.2.102:0/3027850710 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f28e4032650 con 0x7f28f40ba720 2021-02-10T11:21:23.868 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:23.868 7f28f3574700 1 --2- 172.21.2.102:0/3027850710 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f28d40186d0 0x7f28d401abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:23.868 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:23.868 7f28f3574700 1 --2- 172.21.2.102:0/3027850710 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f28d40186d0 0x7f28d401abd0 crc :-1 s=READY pgs=447 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:23.895 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:23.899 7f28f97b8700 1 -- 172.21.2.102:0/3027850710 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28f4105af0 con 0x7f28f40ba720 2021-02-10T11:21:23.897 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:23.902 7f28dbfff700 1 -- 172.21.2.102:0/3027850710 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f28e4002490 con 0x7f28f40ba720 2021-02-10T11:21:24.027 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:24.032 7f28f97b8700 1 -- 172.21.2.102:0/3027850710 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344424", "target": ["mgr", ""]}) v1 -- 0x7f28f4109290 con 0x7f28d40186d0 2021-02-10T11:21:24.039 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:24.043 7f28dbfff700 1 -- 172.21.2.102:0/3027850710 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f28ec000d30 con 0x7f28d40186d0 2021-02-10T11:21:24.045 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:24.049 7f28f97b8700 1 -- 172.21.2.102:0/3027850710 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f28d40186d0 msgr2=0x7f28d401abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:24.045 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:24.049 7f28f97b8700 1 --2- 172.21.2.102:0/3027850710 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f28d40186d0 0x7f28d401abd0 crc :-1 s=READY pgs=447 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:24.045 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:24.049 7f28f97b8700 1 -- 172.21.2.102:0/3027850710 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f28f40ba720 msgr2=0x7f28f40bab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:24.045 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:24.049 7f28f97b8700 1 --2- 172.21.2.102:0/3027850710 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f28f40ba720 0x7f28f40bab20 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 rx=0x7f28e4006f10 tx=0x7f28e400b990).stop 2021-02-10T11:21:24.045 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:24.049 7f28f97b8700 1 -- 172.21.2.102:0/3027850710 shutdown_connections 2021-02-10T11:21:24.045 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:24.049 7f28f97b8700 1 --2- 172.21.2.102:0/3027850710 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f28f40ba720 0x7f28f40bab20 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:24.045 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:24.049 7f28f97b8700 1 --2- 172.21.2.102:0/3027850710 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f28f40a2d80 0x7f28f41a7ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:24.046 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:24.049 7f28f97b8700 1 --2- 172.21.2.102:0/3027850710 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f28d40186d0 0x7f28d401abd0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:24.046 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:24.049 7f28f97b8700 1 --2- 172.21.2.102:0/3027850710 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f28f40ab760 0x7f28f41a74c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:24.046 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:24.049 7f28f97b8700 1 -- 172.21.2.102:0/3027850710 >> 172.21.2.102:0/3027850710 conn(0x7f28f40c40c0 msgr2=0x7f28f40a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:24.046 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:24.049 7f28f97b8700 1 -- 172.21.2.102:0/3027850710 shutdown_connections 2021-02-10T11:21:24.046 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:24.049 7f28f97b8700 1 -- 172.21.2.102:0/3027850710 wait complete. 2021-02-10T11:21:24.054 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000344425 2021-02-10T11:21:24.486 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:24.473 7f84ebf7d700 1 Processor -- start 2021-02-10T11:21:24.486 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:24.473 7f84ebf7d700 1 -- start start 2021-02-10T11:21:24.486 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:24.474 7f84ebf7d700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f84e41b0470 0x7f84e41b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:24.486 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:24.474 7f84ebf7d700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f84e40f20e0 0x7f84e40f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:24.487 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:24.474 7f84ebf7d700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84e40f0b50 con 0x7f84e40ef610 2021-02-10T11:21:24.487 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:24.474 7f84ebf7d700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84e40f0820 con 0x7f84e41b0470 2021-02-10T11:21:24.487 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:24.474 7f84ebf7d700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84e40f24e0 con 0x7f84e40f20e0 2021-02-10T11:21:24.487 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:24.474 7f84e9d99700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f84e41b0470 0x7f84e41b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:24.487 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:24.474 7f84e9d99700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f84e41b0470 0x7f84e41b0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:54528/0 (socket says 172.21.2.102:54528) 2021-02-10T11:21:24.487 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:24.474 7f84e9d99700 1 -- 172.21.2.102:0/739814468 learned_addr learned my addr 172.21.2.102:0/739814468 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:21:24.487 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:24.474 7f84ea59a700 1 --2- 172.21.2.102:0/739814468 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f84e40f20e0 0x7f84e40f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:24.487 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:24.475 7f84e9d99700 1 -- 172.21.2.102:0/739814468 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f84e40f20e0 msgr2=0x7f84e40f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:21:24.487 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:24.475 7f84e9d99700 1 --2- 172.21.2.102:0/739814468 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f84e40f20e0 0x7f84e40f0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:24.487 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:24.475 7f84e9d99700 1 -- 172.21.2.102:0/739814468 >> v1:172.21.2.113:6789/0 conn(0x7f84e40ef610 legacy=0x7f84e40efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:24.487 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:24.475 7f84e9d99700 1 -- 172.21.2.102:0/739814468 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84e40e52e0 con 0x7f84e41b0470 2021-02-10T11:21:24.487 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:24.475 7f84e9d99700 1 --2- 172.21.2.102:0/739814468 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f84e41b0470 0x7f84e41b0870 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 rx=0x7f84e0006700 tx=0x7f84e0009b80).ready entity=mon.0 client_cookie=f9950359e5cdd829 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:24.487 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:24.475 7f84e8d97700 1 -- 172.21.2.102:0/739814468 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f84e000c0c0 con 0x7f84e41b0470 2021-02-10T11:21:24.488 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:24.475 7f84e8d97700 1 -- 172.21.2.102:0/739814468 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84e000c3c0 con 0x7f84e41b0470 2021-02-10T11:21:24.488 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:24.475 7f84e8d97700 1 -- 172.21.2.102:0/739814468 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f84e000c8e0 con 0x7f84e41b0470 2021-02-10T11:21:24.488 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:24.475 7f84ebf7d700 1 -- 172.21.2.102:0/739814468 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f84e41b0470 msgr2=0x7f84e41b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:24.488 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:24.475 7f84ebf7d700 1 --2- 172.21.2.102:0/739814468 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f84e41b0470 0x7f84e41b0870 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 rx=0x7f84e0006700 tx=0x7f84e0009b80).stop 2021-02-10T11:21:24.488 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:24.476 7f84ebf7d700 1 -- 172.21.2.102:0/739814468 shutdown_connections 2021-02-10T11:21:24.488 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:24.476 7f84ebf7d700 1 --2- 172.21.2.102:0/739814468 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f84e40f20e0 0x7f84e40f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:24.488 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:24.476 7f84ebf7d700 1 --2- 172.21.2.102:0/739814468 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f84e41b0470 0x7f84e41b0870 unknown :-1 s=CLOSED pgs=918 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:24.488 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:24.476 7f84ebf7d700 1 -- 172.21.2.102:0/739814468 >> 172.21.2.102:0/739814468 conn(0x7f84e40ce3e0 msgr2=0x7f84e40e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:24.488 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:24.476 7f84ebf7d700 1 -- 172.21.2.102:0/739814468 shutdown_connections 2021-02-10T11:21:24.489 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:24.476 7f84ebf7d700 1 -- 172.21.2.102:0/739814468 wait complete. 2021-02-10T11:21:24.490 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:24.491 7f84ebf7d700 1 Processor -- start 2021-02-10T11:21:24.491 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:24.491 7f84ebf7d700 1 -- start start 2021-02-10T11:21:24.491 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:24.491 7f84ebf7d700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f84e40a0420 0x7f84e40a0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:24.491 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:24.491 7f84ebf7d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f84e4189d00 0x7f84e418a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:24.491 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:24.491 7f84ebf7d700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f84e418a720 0x7f84e40b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:24.491 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:24.491 7f84ebf7d700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84e4189780 con 0x7f84e4189d00 2021-02-10T11:21:24.491 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:24.491 7f84ebf7d700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84e418ab20 con 0x7f84e40a0420 2021-02-10T11:21:24.491 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:24.491 7f84ebf7d700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84e40a1140 con 0x7f84e418a720 2021-02-10T11:21:24.491 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:24.491 7f84e9598700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f84e4189d00 0x7f84e418a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:24.491 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:24.492 7f84e9598700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f84e4189d00 0x7f84e418a100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:54534/0 (socket says 172.21.2.102:54534) 2021-02-10T11:21:24.491 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:24.492 7f84e9598700 1 -- 172.21.2.102:0/833093919 learned_addr learned my addr 172.21.2.102:0/833093919 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:21:24.491 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:24.492 7f84e9d99700 1 --2- 172.21.2.102:0/833093919 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f84e40a0420 0x7f84e40a0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:24.492 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:24.492 7f84ea59a700 1 --2- 172.21.2.102:0/833093919 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f84e418a720 0x7f84e40b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:24.492 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:24.492 7f84e9598700 1 -- 172.21.2.102:0/833093919 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f84e418a720 msgr2=0x7f84e40b7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:24.492 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:24.492 7f84e9598700 1 --2- 172.21.2.102:0/833093919 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f84e418a720 0x7f84e40b7be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:24.492 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:24.492 7f84e9598700 1 -- 172.21.2.102:0/833093919 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f84e40a0420 msgr2=0x7f84e40a0820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:24.492 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:24.492 7f84e9598700 1 --2- 172.21.2.102:0/833093919 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f84e40a0420 0x7f84e40a0820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:24.492 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:24.492 7f84e9598700 1 -- 172.21.2.102:0/833093919 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84e40b8200 con 0x7f84e4189d00 2021-02-10T11:21:24.492 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:24.492 7f84e9598700 1 --2- 172.21.2.102:0/833093919 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f84e4189d00 0x7f84e418a100 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 rx=0x7f84d8006ed0 tx=0x7f84d800b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:24.492 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:24.492 7f84d67fc700 1 -- 172.21.2.102:0/833093919 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f84d8006240 con 0x7f84e4189d00 2021-02-10T11:21:24.492 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:24.492 7f84d67fc700 1 -- 172.21.2.102:0/833093919 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84d80054b0 con 0x7f84e4189d00 2021-02-10T11:21:24.492 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:24.492 7f84d67fc700 1 -- 172.21.2.102:0/833093919 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f84d8005900 con 0x7f84e4189d00 2021-02-10T11:21:24.492 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:24.492 7f84ebf7d700 1 -- 172.21.2.102:0/833093919 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84e40b8200 con 0x7f84e4189d00 2021-02-10T11:21:24.492 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:24.492 7f84ebf7d700 1 -- 172.21.2.102:0/833093919 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84e40b8200 con 0x7f84e4189d00 2021-02-10T11:21:24.492 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:24.493 7f84d67fc700 1 -- 172.21.2.102:0/833093919 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f84d8006030 con 0x7f84e4189d00 2021-02-10T11:21:24.493 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:24.493 7f84d67fc700 1 --2- 172.21.2.102:0/833093919 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f84cc018730 0x7f84cc01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:24.493 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:24.493 7f84d67fc700 1 -- 172.21.2.102:0/833093919 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f84d8032670 con 0x7f84e4189d00 2021-02-10T11:21:24.493 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:24.493 7f84e9d99700 1 --2- 172.21.2.102:0/833093919 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f84cc018730 0x7f84cc01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:24.493 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:24.494 7f84e9d99700 1 --2- 172.21.2.102:0/833093919 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f84cc018730 0x7f84cc01ac10 crc :-1 s=READY pgs=448 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:24.518 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:24.523 7f84ebf7d700 1 -- 172.21.2.102:0/833093919 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84e4105ae0 con 0x7f84e4189d00 2021-02-10T11:21:24.521 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:24.525 7f84d67fc700 1 -- 172.21.2.102:0/833093919 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f84d8000de0 con 0x7f84e4189d00 2021-02-10T11:21:24.647 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:24.651 7f84ebf7d700 1 -- 172.21.2.102:0/833093919 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344425", "target": ["mgr", ""]}) v1 -- 0x7f84e4105ae0 con 0x7f84cc018730 2021-02-10T11:21:24.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:24.662 7f84d67fc700 1 -- 172.21.2.102:0/833093919 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f84e0006430 con 0x7f84cc018730 2021-02-10T11:21:24.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:24.669 7f84ebf7d700 1 -- 172.21.2.102:0/833093919 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f84cc018730 msgr2=0x7f84cc01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:24.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:24.669 7f84ebf7d700 1 --2- 172.21.2.102:0/833093919 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f84cc018730 0x7f84cc01ac10 crc :-1 s=READY pgs=448 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:24.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:24.669 7f84ebf7d700 1 -- 172.21.2.102:0/833093919 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f84e4189d00 msgr2=0x7f84e418a100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:24.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:24.669 7f84ebf7d700 1 --2- 172.21.2.102:0/833093919 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f84e4189d00 0x7f84e418a100 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 rx=0x7f84d8006ed0 tx=0x7f84d800b960).stop 2021-02-10T11:21:24.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:24.669 7f84ebf7d700 1 -- 172.21.2.102:0/833093919 shutdown_connections 2021-02-10T11:21:24.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:24.669 7f84ebf7d700 1 --2- 172.21.2.102:0/833093919 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f84e418a720 0x7f84e40b7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:24.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:24.669 7f84ebf7d700 1 --2- 172.21.2.102:0/833093919 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f84e4189d00 0x7f84e418a100 unknown :-1 s=CLOSED pgs=919 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:24.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:24.669 7f84ebf7d700 1 --2- 172.21.2.102:0/833093919 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f84cc018730 0x7f84cc01ac10 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:24.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:24.669 7f84ebf7d700 1 --2- 172.21.2.102:0/833093919 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f84e40a0420 0x7f84e40a0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:24.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:24.669 7f84ebf7d700 1 -- 172.21.2.102:0/833093919 >> 172.21.2.102:0/833093919 conn(0x7f84e4094e10 msgr2=0x7f84e40a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:24.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:24.669 7f84ebf7d700 1 -- 172.21.2.102:0/833093919 shutdown_connections 2021-02-10T11:21:24.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:24.669 7f84ebf7d700 1 -- 172.21.2.102:0/833093919 wait complete. 2021-02-10T11:21:24.674 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000344426 2021-02-10T11:21:25.104 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:25.092 7f286081b700 1 Processor -- start 2021-02-10T11:21:25.104 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:25.092 7f286081b700 1 -- start start 2021-02-10T11:21:25.104 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:25.092 7f286081b700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f285c1b0470 0x7f285c1b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:25.104 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:25.092 7f286081b700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f285c0f20e0 0x7f285c0f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:25.104 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:25.092 7f286081b700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f285c0f0b50 con 0x7f285c0ef610 2021-02-10T11:21:25.104 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:25.092 7f286081b700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f285c0f0820 con 0x7f285c1b0470 2021-02-10T11:21:25.105 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:25.092 7f286081b700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f285c0f24e0 con 0x7f285c0f20e0 2021-02-10T11:21:25.105 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:25.093 7f2859e1b700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f285c0ef610 0x7f285c0efa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:46770/0 (socket says 172.21.2.102:46770) 2021-02-10T11:21:25.105 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:25.093 7f2859e1b700 1 -- 172.21.2.102:0/2348511407 learned_addr learned my addr 172.21.2.102:0/2348511407 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:21:25.105 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:25.093 7f285a61c700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f285c1b0470 0x7f285c1b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:25.105 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:25.093 7f285ae1d700 1 --2- 172.21.2.102:0/2348511407 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f285c0f20e0 0x7f285c0f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:25.105 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:25.093 7f285ae1d700 1 -- 172.21.2.102:0/2348511407 >> v2:172.21.2.113:3300/0 conn(0x7f285c1b0470 msgr2=0x7f285c1b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:21:25.105 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:25.093 7f285ae1d700 1 --2- 172.21.2.102:0/2348511407 >> v2:172.21.2.113:3300/0 conn(0x7f285c1b0470 0x7f285c1b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:25.105 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:25.093 7f285ae1d700 1 -- 172.21.2.102:0/2348511407 >> v1:172.21.2.102:6789/0 conn(0x7f285c0ef610 legacy=0x7f285c0efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:25.105 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:25.093 7f285ae1d700 1 -- 172.21.2.102:0/2348511407 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f285c0e52e0 con 0x7f285c0f20e0 2021-02-10T11:21:25.105 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:25.093 7f285ae1d700 1 --2- 172.21.2.102:0/2348511407 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f285c0f20e0 0x7f285c0f0200 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7f2854006e50 tx=0x7f2854009e70).ready entity=mon.2 client_cookie=83b0fea5237a5d52 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:25.105 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:25.093 7f285961a700 1 -- 172.21.2.102:0/2348511407 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2854006a20 con 0x7f285c0f20e0 2021-02-10T11:21:25.105 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:25.093 7f285961a700 1 -- 172.21.2.102:0/2348511407 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2854006790 con 0x7f285c0f20e0 2021-02-10T11:21:25.105 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:25.093 7f285961a700 1 -- 172.21.2.102:0/2348511407 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f285400b700 con 0x7f285c0f20e0 2021-02-10T11:21:25.106 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:25.094 7f286081b700 1 -- 172.21.2.102:0/2348511407 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f285c0f20e0 msgr2=0x7f285c0f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:25.106 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:25.094 7f286081b700 1 --2- 172.21.2.102:0/2348511407 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f285c0f20e0 0x7f285c0f0200 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7f2854006e50 tx=0x7f2854009e70).stop 2021-02-10T11:21:25.106 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:25.094 7f286081b700 1 -- 172.21.2.102:0/2348511407 shutdown_connections 2021-02-10T11:21:25.106 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:25.094 7f286081b700 1 --2- 172.21.2.102:0/2348511407 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f285c0f20e0 0x7f285c0f0200 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:25.106 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:25.094 7f286081b700 1 --2- 172.21.2.102:0/2348511407 >> v2:172.21.2.113:3300/0 conn(0x7f285c1b0470 0x7f285c1b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:25.106 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:25.094 7f286081b700 1 -- 172.21.2.102:0/2348511407 >> 172.21.2.102:0/2348511407 conn(0x7f285c0ce3e0 msgr2=0x7f285c0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:25.106 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:25.094 7f286081b700 1 -- 172.21.2.102:0/2348511407 shutdown_connections 2021-02-10T11:21:25.106 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:25.094 7f286081b700 1 -- 172.21.2.102:0/2348511407 wait complete. 2021-02-10T11:21:25.107 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:25.109 7f286081b700 1 Processor -- start 2021-02-10T11:21:25.107 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:25.109 7f286081b700 1 -- start start 2021-02-10T11:21:25.107 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:25.109 7f286081b700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f285c105d10 0x7f285c09f880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:25.107 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:25.109 7f286081b700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f285c0961f0 0x7f285c0965f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:25.107 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:25.109 7f286081b700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f285c106890 0x7f285c106c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:25.107 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:25.109 7f286081b700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f285c106110 con 0x7f285c106890 2021-02-10T11:21:25.108 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:25.109 7f286081b700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f285c106380 con 0x7f285c105d10 2021-02-10T11:21:25.108 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:25.109 7f286081b700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f285c0a6620 con 0x7f285c0961f0 2021-02-10T11:21:25.108 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:25.109 7f285ae1d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f285c106890 0x7f285c106c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:25.108 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:25.109 7f285ae1d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f285c106890 0x7f285c106c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:54552/0 (socket says 172.21.2.102:54552) 2021-02-10T11:21:25.108 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:25.109 7f285ae1d700 1 -- 172.21.2.102:0/114433543 learned_addr learned my addr 172.21.2.102:0/114433543 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:21:25.108 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:25.109 7f2859e1b700 1 --2- 172.21.2.102:0/114433543 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f285c0961f0 0x7f285c0965f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:25.109 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:25.110 7f285a61c700 1 --2- 172.21.2.102:0/114433543 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f285c105d10 0x7f285c09f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:25.109 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:25.110 7f285ae1d700 1 -- 172.21.2.102:0/114433543 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f285c0961f0 msgr2=0x7f285c0965f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:25.109 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:25.110 7f285ae1d700 1 --2- 172.21.2.102:0/114433543 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f285c0961f0 0x7f285c0965f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:25.109 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:25.110 7f285ae1d700 1 -- 172.21.2.102:0/114433543 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f285c105d10 msgr2=0x7f285c09f880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:25.110 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:25.110 7f285ae1d700 1 --2- 172.21.2.102:0/114433543 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f285c105d10 0x7f285c09f880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:25.110 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:25.110 7f285ae1d700 1 -- 172.21.2.102:0/114433543 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f285c1072b0 con 0x7f285c106890 2021-02-10T11:21:25.110 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:25.110 7f285ae1d700 1 --2- 172.21.2.102:0/114433543 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f285c106890 0x7f285c106c90 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 rx=0x7f28540099f0 tx=0x7f285400bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:25.110 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:25.110 7f284affd700 1 -- 172.21.2.102:0/114433543 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f28540093d0 con 0x7f285c106890 2021-02-10T11:21:25.110 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:25.110 7f284affd700 1 -- 172.21.2.102:0/114433543 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28540090c0 con 0x7f285c106890 2021-02-10T11:21:25.110 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:25.110 7f284affd700 1 -- 172.21.2.102:0/114433543 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2854001420 con 0x7f285c106890 2021-02-10T11:21:25.110 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:25.110 7f286081b700 1 -- 172.21.2.102:0/114433543 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f285c1072b0 con 0x7f285c106890 2021-02-10T11:21:25.110 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:25.110 7f286081b700 1 -- 172.21.2.102:0/114433543 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f285c107840 con 0x7f285c106890 2021-02-10T11:21:25.110 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:25.111 7f284affd700 1 -- 172.21.2.102:0/114433543 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f28540056b0 con 0x7f285c106890 2021-02-10T11:21:25.110 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:25.111 7f284affd700 1 --2- 172.21.2.102:0/114433543 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2840018730 0x7f284001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:25.110 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:25.111 7f284affd700 1 -- 172.21.2.102:0/114433543 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f2854032b20 con 0x7f285c106890 2021-02-10T11:21:25.110 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:25.111 7f285a61c700 1 --2- 172.21.2.102:0/114433543 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2840018730 0x7f284001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:25.110 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:25.111 7f285a61c700 1 --2- 172.21.2.102:0/114433543 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2840018730 0x7f284001ac10 crc :-1 s=READY pgs=449 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:25.138 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:25.142 7f286081b700 1 -- 172.21.2.102:0/114433543 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f285c105ae0 con 0x7f285c106890 2021-02-10T11:21:25.140 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:25.145 7f284affd700 1 -- 172.21.2.102:0/114433543 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f28540056b0 con 0x7f285c106890 2021-02-10T11:21:25.266 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:25.271 7f286081b700 1 -- 172.21.2.102:0/114433543 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344426", "target": ["mgr", ""]}) v1 -- 0x7f285c0f0b50 con 0x7f2840018730 2021-02-10T11:21:25.276 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:25.281 7f284affd700 1 -- 172.21.2.102:0/114433543 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f284c006a70 con 0x7f2840018730 2021-02-10T11:21:25.283 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:25.287 7f286081b700 1 -- 172.21.2.102:0/114433543 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2840018730 msgr2=0x7f284001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:25.283 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:25.287 7f286081b700 1 --2- 172.21.2.102:0/114433543 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2840018730 0x7f284001ac10 crc :-1 s=READY pgs=449 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:25.283 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:25.287 7f286081b700 1 -- 172.21.2.102:0/114433543 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f285c106890 msgr2=0x7f285c106c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:25.283 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:25.287 7f286081b700 1 --2- 172.21.2.102:0/114433543 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f285c106890 0x7f285c106c90 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 rx=0x7f28540099f0 tx=0x7f285400bd10).stop 2021-02-10T11:21:25.284 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:25.287 7f286081b700 1 -- 172.21.2.102:0/114433543 shutdown_connections 2021-02-10T11:21:25.284 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:25.287 7f286081b700 1 --2- 172.21.2.102:0/114433543 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f285c106890 0x7f285c106c90 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:25.284 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:25.287 7f286081b700 1 --2- 172.21.2.102:0/114433543 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f285c0961f0 0x7f285c0965f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:25.284 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:25.287 7f286081b700 1 --2- 172.21.2.102:0/114433543 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2840018730 0x7f284001ac10 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:25.284 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:25.287 7f286081b700 1 --2- 172.21.2.102:0/114433543 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f285c105d10 0x7f285c09f880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:25.284 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:25.287 7f286081b700 1 -- 172.21.2.102:0/114433543 >> 172.21.2.102:0/114433543 conn(0x7f285c095a70 msgr2=0x7f285c0bf050 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:25.284 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:25.288 7f286081b700 1 -- 172.21.2.102:0/114433543 shutdown_connections 2021-02-10T11:21:25.284 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:25.288 7f286081b700 1 -- 172.21.2.102:0/114433543 wait complete. 2021-02-10T11:21:25.292 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000344427 2021-02-10T11:21:25.725 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:25.713 7fbb28c95700 1 Processor -- start 2021-02-10T11:21:25.726 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:25.713 7fbb28c95700 1 -- start start 2021-02-10T11:21:25.726 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:25.714 7fbb28c95700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fbb241b0470 0x7fbb241b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:25.726 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:25.714 7fbb28c95700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb240f0b50 con 0x7fbb240f20e0 2021-02-10T11:21:25.726 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:25.714 7fbb28c95700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb240f24e0 con 0x7fbb240ef610 2021-02-10T11:21:25.726 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:25.714 7fbb28c95700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb240e5ea0 con 0x7fbb241b0470 2021-02-10T11:21:25.726 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:25.714 7fbb2285e700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fbb241b0470 0x7fbb241b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:25.726 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:25.714 7fbb2305f700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fbb240f20e0 0x7fbb240f0200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:46784/0 (socket says 172.21.2.102:46784) 2021-02-10T11:21:25.727 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:25.714 7fbb2305f700 1 -- 172.21.2.102:0/2282489330 learned_addr learned my addr 172.21.2.102:0/2282489330 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:21:25.727 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:25.714 7fbb2285e700 1 -- 172.21.2.102:0/2282489330 >> v1:172.21.2.113:6789/0 conn(0x7fbb240ef610 legacy=0x7fbb240efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:25.727 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:25.714 7fbb2185c700 1 -- 172.21.2.102:0/2282489330 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3453940327 0 0) 0x7fbb1c001500 con 0x7fbb240f20e0 2021-02-10T11:21:25.727 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:25.714 7fbb2285e700 1 -- 172.21.2.102:0/2282489330 >> v1:172.21.2.102:6789/0 conn(0x7fbb240f20e0 legacy=0x7fbb240f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:25.727 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:25.714 7fbb2285e700 1 -- 172.21.2.102:0/2282489330 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb240e52e0 con 0x7fbb241b0470 2021-02-10T11:21:25.728 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:25.714 7fbb2285e700 1 --2- 172.21.2.102:0/2282489330 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fbb241b0470 0x7fbb241b0870 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 rx=0x7fbb14006650 tx=0x7fbb14009a80).ready entity=mon.1 client_cookie=f330458eff9013ec server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:25.728 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:25.715 7fbb2185c700 1 -- 172.21.2.102:0/2282489330 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbb140009b0 con 0x7fbb241b0470 2021-02-10T11:21:25.728 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:25.715 7fbb2185c700 1 -- 172.21.2.102:0/2282489330 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb1400c0c0 con 0x7fbb241b0470 2021-02-10T11:21:25.728 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:25.715 7fbb2185c700 1 -- 172.21.2.102:0/2282489330 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbb1400c5e0 con 0x7fbb241b0470 2021-02-10T11:21:25.728 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:25.715 7fbb28c95700 1 -- 172.21.2.102:0/2282489330 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fbb241b0470 msgr2=0x7fbb241b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:25.728 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:25.715 7fbb28c95700 1 --2- 172.21.2.102:0/2282489330 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fbb241b0470 0x7fbb241b0870 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 rx=0x7fbb14006650 tx=0x7fbb14009a80).stop 2021-02-10T11:21:25.728 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:25.715 7fbb28c95700 1 -- 172.21.2.102:0/2282489330 shutdown_connections 2021-02-10T11:21:25.728 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:25.715 7fbb28c95700 1 --2- 172.21.2.102:0/2282489330 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fbb241b0470 0x7fbb241b0870 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:25.728 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:25.715 7fbb28c95700 1 -- 172.21.2.102:0/2282489330 >> 172.21.2.102:0/2282489330 conn(0x7fbb240ce3e0 msgr2=0x7fbb240e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:25.728 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:25.715 7fbb28c95700 1 -- 172.21.2.102:0/2282489330 shutdown_connections 2021-02-10T11:21:25.729 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:25.715 7fbb28c95700 1 -- 172.21.2.102:0/2282489330 wait complete. 2021-02-10T11:21:25.729 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:25.730 7fbb28c95700 1 Processor -- start 2021-02-10T11:21:25.729 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:25.730 7fbb28c95700 1 -- start start 2021-02-10T11:21:25.730 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:25.730 7fbb28c95700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbb241a0570 0x7fbb240a4590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:25.730 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:25.730 7fbb28c95700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbb240f1fd0 0x7fbb240f23d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:25.731 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:25.730 7fbb28c95700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fbb240a5450 0x7fbb240a5850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:25.731 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:25.730 7fbb28c95700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb240a4bb0 con 0x7fbb241a0570 2021-02-10T11:21:25.731 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:25.730 7fbb28c95700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb240b35e0 con 0x7fbb240a5450 2021-02-10T11:21:25.731 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:25.730 7fbb28c95700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb240a1f00 con 0x7fbb240f1fd0 2021-02-10T11:21:25.731 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:25.731 7fbb2305f700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fbb240a5450 0x7fbb240a5850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:25.731 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:25.731 7fbb2285e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbb241a0570 0x7fbb240a4590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:25.731 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:25.731 7fbb2305f700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fbb240a5450 0x7fbb240a5850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.113:3300/0 says I am v2:172.21.2.102:57994/0 (socket says 172.21.2.102:57994) 2021-02-10T11:21:25.732 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:25.731 7fbb2285e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbb241a0570 0x7fbb240a4590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:54566/0 (socket says 172.21.2.102:54566) 2021-02-10T11:21:25.732 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:25.731 7fbb2305f700 1 -- 172.21.2.102:0/2292961902 learned_addr learned my addr 172.21.2.102:0/2292961902 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:21:25.732 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:25.731 7fbb2205d700 1 --2- 172.21.2.102:0/2292961902 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbb240f1fd0 0x7fbb240f23d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:25.732 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:25.731 7fbb2305f700 1 -- 172.21.2.102:0/2292961902 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbb240f1fd0 msgr2=0x7fbb240f23d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:25.732 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:25.731 7fbb2305f700 1 --2- 172.21.2.102:0/2292961902 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbb240f1fd0 0x7fbb240f23d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:25.732 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:25.731 7fbb2305f700 1 -- 172.21.2.102:0/2292961902 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbb241a0570 msgr2=0x7fbb240a4590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:25.732 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:25.731 7fbb2305f700 1 --2- 172.21.2.102:0/2292961902 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbb241a0570 0x7fbb240a4590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:25.733 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:25.731 7fbb2305f700 1 -- 172.21.2.102:0/2292961902 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb240a5e70 con 0x7fbb240a5450 2021-02-10T11:21:25.733 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:25.731 7fbb2305f700 1 --2- 172.21.2.102:0/2292961902 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fbb240a5450 0x7fbb240a5850 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 rx=0x7fbb1c001c10 tx=0x7fbb1c00ac60).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:25.733 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:25.731 7fbb1b7fe700 1 -- 172.21.2.102:0/2292961902 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbb1c0070d0 con 0x7fbb240a5450 2021-02-10T11:21:25.733 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:25.731 7fbb1b7fe700 1 -- 172.21.2.102:0/2292961902 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb1c0070d0 con 0x7fbb240a5450 2021-02-10T11:21:25.733 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:25.731 7fbb1b7fe700 1 -- 172.21.2.102:0/2292961902 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbb1c0073d0 con 0x7fbb240a5450 2021-02-10T11:21:25.733 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:25.731 7fbb28c95700 1 -- 172.21.2.102:0/2292961902 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb240a5e70 con 0x7fbb240a5450 2021-02-10T11:21:25.733 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:25.731 7fbb28c95700 1 -- 172.21.2.102:0/2292961902 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb240a5e70 con 0x7fbb240a5450 2021-02-10T11:21:25.733 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:25.732 7fbb1b7fe700 1 -- 172.21.2.102:0/2292961902 <== mon.1 v2:172.21.2.113:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fbb1c004720 con 0x7fbb240a5450 2021-02-10T11:21:25.733 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:25.732 7fbb1b7fe700 1 --2- 172.21.2.102:0/2292961902 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fbb08018730 0x7fbb0801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:25.733 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:25.732 7fbb1b7fe700 1 -- 172.21.2.102:0/2292961902 <== mon.1 v2:172.21.2.113:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fbb1c032b40 con 0x7fbb240a5450 2021-02-10T11:21:25.734 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:25.732 7fbb2285e700 1 --2- 172.21.2.102:0/2292961902 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fbb08018730 0x7fbb0801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:25.734 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:25.732 7fbb2285e700 1 --2- 172.21.2.102:0/2292961902 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fbb08018730 0x7fbb0801ac10 crc :-1 s=READY pgs=450 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:25.758 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:25.762 7fbb28c95700 1 -- 172.21.2.102:0/2292961902 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb240a4bb0 con 0x7fbb240a5450 2021-02-10T11:21:25.761 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:25.766 7fbb1b7fe700 1 -- 172.21.2.102:0/2292961902 <== mon.1 v2:172.21.2.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fbb1c0070b0 con 0x7fbb240a5450 2021-02-10T11:21:25.893 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:25.898 7fbb28c95700 1 -- 172.21.2.102:0/2292961902 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344427", "target": ["mgr", ""]}) v1 -- 0x7fbb240a4bb0 con 0x7fbb08018730 2021-02-10T11:21:25.904 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:25.909 7fbb1b7fe700 1 -- 172.21.2.102:0/2292961902 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fbb14006540 con 0x7fbb08018730 2021-02-10T11:21:25.911 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:25.915 7fbb28c95700 1 -- 172.21.2.102:0/2292961902 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fbb08018730 msgr2=0x7fbb0801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:25.911 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:25.915 7fbb28c95700 1 --2- 172.21.2.102:0/2292961902 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fbb08018730 0x7fbb0801ac10 crc :-1 s=READY pgs=450 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:25.911 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:25.915 7fbb28c95700 1 -- 172.21.2.102:0/2292961902 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fbb240a5450 msgr2=0x7fbb240a5850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:25.911 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:25.915 7fbb28c95700 1 --2- 172.21.2.102:0/2292961902 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fbb240a5450 0x7fbb240a5850 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 rx=0x7fbb1c001c10 tx=0x7fbb1c00ac60).stop 2021-02-10T11:21:25.911 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:25.915 7fbb28c95700 1 -- 172.21.2.102:0/2292961902 shutdown_connections 2021-02-10T11:21:25.911 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:25.915 7fbb28c95700 1 --2- 172.21.2.102:0/2292961902 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fbb08018730 0x7fbb0801ac10 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:25.911 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:25.915 7fbb28c95700 1 --2- 172.21.2.102:0/2292961902 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fbb240a5450 0x7fbb240a5850 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:25.911 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:25.915 7fbb28c95700 1 --2- 172.21.2.102:0/2292961902 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbb240f1fd0 0x7fbb240f23d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:25.911 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:25.915 7fbb28c95700 1 --2- 172.21.2.102:0/2292961902 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbb241a0570 0x7fbb240a4590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:25.911 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:25.915 7fbb28c95700 1 -- 172.21.2.102:0/2292961902 >> 172.21.2.102:0/2292961902 conn(0x7fbb240958b0 msgr2=0x7fbb240bf040 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:25.911 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:25.915 7fbb28c95700 1 -- 172.21.2.102:0/2292961902 shutdown_connections 2021-02-10T11:21:25.911 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:25.915 7fbb28c95700 1 -- 172.21.2.102:0/2292961902 wait complete. 2021-02-10T11:21:25.919 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000344428 2021-02-10T11:21:26.350 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:26.337 7f4dc8fe1700 1 Processor -- start 2021-02-10T11:21:26.350 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:26.338 7f4dc8fe1700 1 -- start start 2021-02-10T11:21:26.350 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:26.338 7f4dc8fe1700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f4dc41b0470 0x7f4dc41b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:26.350 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:26.338 7f4dc8fe1700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4dc40f20e0 0x7f4dc40f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:26.350 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:26.338 7f4dc8fe1700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4dc40f0b50 con 0x7f4dc40ef610 2021-02-10T11:21:26.350 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:26.338 7f4dc8fe1700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4dc40f0820 con 0x7f4dc41b0470 2021-02-10T11:21:26.351 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:26.338 7f4dc8fe1700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4dc40f24e0 con 0x7f4dc40f20e0 2021-02-10T11:21:26.351 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:26.338 7f4dc2d9d700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f4dc41b0470 0x7f4dc41b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:26.351 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:26.338 7f4dc259c700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f4dc40ef610 0x7f4dc40efa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:46796/0 (socket says 172.21.2.102:46796) 2021-02-10T11:21:26.351 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:26.338 7f4dc259c700 1 -- 172.21.2.102:0/1170304392 learned_addr learned my addr 172.21.2.102:0/1170304392 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:21:26.351 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:26.338 7f4dc359e700 1 --2- 172.21.2.102:0/1170304392 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4dc40f20e0 0x7f4dc40f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:26.351 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:26.340 7f4dc2d9d700 1 -- 172.21.2.102:0/1170304392 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4dc40f20e0 msgr2=0x7f4dc40f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:21:26.351 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:26.340 7f4dc2d9d700 1 --2- 172.21.2.102:0/1170304392 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4dc40f20e0 0x7f4dc40f0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:26.351 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:26.340 7f4dc2d9d700 1 -- 172.21.2.102:0/1170304392 >> v1:172.21.2.102:6789/0 conn(0x7f4dc40ef610 legacy=0x7f4dc40efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:26.351 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:26.340 7f4dc2d9d700 1 -- 172.21.2.102:0/1170304392 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dc40e52e0 con 0x7f4dc41b0470 2021-02-10T11:21:26.351 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:26.340 7f4dc2d9d700 1 --2- 172.21.2.102:0/1170304392 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4dc41b0470 0x7f4dc41b0870 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 rx=0x7f4dac006650 tx=0x7f4dac009a00).ready entity=mon.0 client_cookie=a27c5ff09d0e80e1 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:26.351 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:26.340 7f4dc1d9b700 1 -- 172.21.2.102:0/1170304392 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4dac0009b0 con 0x7f4dc41b0470 2021-02-10T11:21:26.351 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:26.340 7f4dc1d9b700 1 -- 172.21.2.102:0/1170304392 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4dac00c0c0 con 0x7f4dc41b0470 2021-02-10T11:21:26.352 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:26.340 7f4dc1d9b700 1 -- 172.21.2.102:0/1170304392 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4dac00c5a0 con 0x7f4dc41b0470 2021-02-10T11:21:26.352 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:26.340 7f4dc8fe1700 1 -- 172.21.2.102:0/1170304392 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4dc41b0470 msgr2=0x7f4dc41b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:26.352 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:26.340 7f4dc8fe1700 1 --2- 172.21.2.102:0/1170304392 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4dc41b0470 0x7f4dc41b0870 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 rx=0x7f4dac006650 tx=0x7f4dac009a00).stop 2021-02-10T11:21:26.352 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:26.340 7f4dc8fe1700 1 -- 172.21.2.102:0/1170304392 shutdown_connections 2021-02-10T11:21:26.352 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:26.340 7f4dc8fe1700 1 --2- 172.21.2.102:0/1170304392 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4dc40f20e0 0x7f4dc40f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:26.352 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:26.340 7f4dc8fe1700 1 --2- 172.21.2.102:0/1170304392 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4dc41b0470 0x7f4dc41b0870 unknown :-1 s=CLOSED pgs=924 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:26.352 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:26.340 7f4dc8fe1700 1 -- 172.21.2.102:0/1170304392 >> 172.21.2.102:0/1170304392 conn(0x7f4dc40ce3e0 msgr2=0x7f4dc40e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:26.352 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:26.340 7f4dc8fe1700 1 -- 172.21.2.102:0/1170304392 shutdown_connections 2021-02-10T11:21:26.352 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:26.340 7f4dc8fe1700 1 -- 172.21.2.102:0/1170304392 wait complete. 2021-02-10T11:21:26.355 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:26.355 7f4dc8fe1700 1 Processor -- start 2021-02-10T11:21:26.355 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:26.355 7f4dc8fe1700 1 -- start start 2021-02-10T11:21:26.355 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:26.355 7f4dc8fe1700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4dc4105d10 0x7f4dc409eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:26.355 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:26.355 7f4dc8fe1700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4dc4106110 0x7f4dc409f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:26.355 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:26.355 7f4dc8fe1700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4dc40b2a00 0x7f4dc40b2e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:26.355 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:26.355 7f4dc8fe1700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dc409e560 con 0x7f4dc4105d10 2021-02-10T11:21:26.356 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:26.355 7f4dc8fe1700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dc409c090 con 0x7f4dc40b2a00 2021-02-10T11:21:26.356 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:26.355 7f4dc8fe1700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4dc40b3420 con 0x7f4dc4106110 2021-02-10T11:21:26.356 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:26.356 7f4dc2d9d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4dc4105d10 0x7f4dc409eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:26.356 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:26.356 7f4dc2d9d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4dc4105d10 0x7f4dc409eae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:54580/0 (socket says 172.21.2.102:54580) 2021-02-10T11:21:26.356 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:26.356 7f4dc2d9d700 1 -- 172.21.2.102:0/1443563896 learned_addr learned my addr 172.21.2.102:0/1443563896 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:21:26.356 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:26.356 7f4dc359e700 1 --2- 172.21.2.102:0/1443563896 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4dc40b2a00 0x7f4dc40b2e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:26.356 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:26.356 7f4dc259c700 1 --2- 172.21.2.102:0/1443563896 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4dc4106110 0x7f4dc409f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:26.356 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:26.356 7f4dc359e700 1 -- 172.21.2.102:0/1443563896 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4dc4106110 msgr2=0x7f4dc409f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:26.356 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:26.356 7f4dc359e700 1 --2- 172.21.2.102:0/1443563896 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4dc4106110 0x7f4dc409f100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:26.356 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:26.356 7f4dc359e700 1 -- 172.21.2.102:0/1443563896 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4dc4105d10 msgr2=0x7f4dc409eae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:26.356 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:26.356 7f4dc359e700 1 --2- 172.21.2.102:0/1443563896 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4dc4105d10 0x7f4dc409eae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:26.356 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:26.356 7f4dc359e700 1 -- 172.21.2.102:0/1443563896 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dc40b3640 con 0x7f4dc40b2a00 2021-02-10T11:21:26.356 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:26.356 7f4dc2d9d700 1 --2- 172.21.2.102:0/1443563896 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4dc4105d10 0x7f4dc409eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:21:26.357 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:26.356 7f4dc359e700 1 --2- 172.21.2.102:0/1443563896 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4dc40b2a00 0x7f4dc40b2e00 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 rx=0x7f4dbc00b8a0 tx=0x7f4dbc00bc40).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:26.357 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:26.356 7f4dbb7fe700 1 -- 172.21.2.102:0/1443563896 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4dbc000f90 con 0x7f4dc40b2a00 2021-02-10T11:21:26.357 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:26.357 7f4dbb7fe700 1 -- 172.21.2.102:0/1443563896 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4dbc00bd80 con 0x7f4dc40b2a00 2021-02-10T11:21:26.357 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:26.357 7f4dc8fe1700 1 -- 172.21.2.102:0/1443563896 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4dc40b3640 con 0x7f4dc40b2a00 2021-02-10T11:21:26.357 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:26.357 7f4dc8fe1700 1 -- 172.21.2.102:0/1443563896 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4dc40b3bd0 con 0x7f4dc40b2a00 2021-02-10T11:21:26.357 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:26.357 7f4dbb7fe700 1 -- 172.21.2.102:0/1443563896 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4dbc000f90 con 0x7f4dc40b2a00 2021-02-10T11:21:26.357 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:26.357 7f4dbb7fe700 1 -- 172.21.2.102:0/1443563896 <== mon.1 v2:172.21.2.113:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4dbc0056b0 con 0x7f4dc40b2a00 2021-02-10T11:21:26.357 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:26.357 7f4dbb7fe700 1 --2- 172.21.2.102:0/1443563896 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4da8018730 0x7f4da801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:26.357 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:26.357 7f4dbb7fe700 1 -- 172.21.2.102:0/1443563896 <== mon.1 v2:172.21.2.113:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f4dbc032920 con 0x7f4dc40b2a00 2021-02-10T11:21:26.357 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:26.357 7f4dc2d9d700 1 --2- 172.21.2.102:0/1443563896 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4da8018730 0x7f4da801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:26.357 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:26.359 7f4dc2d9d700 1 --2- 172.21.2.102:0/1443563896 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4da8018730 0x7f4da801ac10 crc :-1 s=READY pgs=451 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:26.383 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:26.388 7f4dc8fe1700 1 -- 172.21.2.102:0/1443563896 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4dc4105ae0 con 0x7f4dc40b2a00 2021-02-10T11:21:26.385 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:26.391 7f4dbb7fe700 1 -- 172.21.2.102:0/1443563896 <== mon.1 v2:172.21.2.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f4dbc0013e0 con 0x7f4dc40b2a00 2021-02-10T11:21:26.513 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:26.518 7f4dc8fe1700 1 -- 172.21.2.102:0/1443563896 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344428", "target": ["mgr", ""]}) v1 -- 0x7f4dc41a16c0 con 0x7f4da8018730 2021-02-10T11:21:26.524 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:26.530 7f4dbb7fe700 1 -- 172.21.2.102:0/1443563896 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f4dac006560 con 0x7f4da8018730 2021-02-10T11:21:26.531 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:26.535 7f4dc8fe1700 1 -- 172.21.2.102:0/1443563896 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4da8018730 msgr2=0x7f4da801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:26.531 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:26.535 7f4dc8fe1700 1 --2- 172.21.2.102:0/1443563896 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4da8018730 0x7f4da801ac10 crc :-1 s=READY pgs=451 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:26.531 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:26.535 7f4dc8fe1700 1 -- 172.21.2.102:0/1443563896 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4dc40b2a00 msgr2=0x7f4dc40b2e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:26.531 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:26.535 7f4dc8fe1700 1 --2- 172.21.2.102:0/1443563896 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4dc40b2a00 0x7f4dc40b2e00 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 rx=0x7f4dbc00b8a0 tx=0x7f4dbc00bc40).stop 2021-02-10T11:21:26.531 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:26.535 7f4dc8fe1700 1 -- 172.21.2.102:0/1443563896 shutdown_connections 2021-02-10T11:21:26.531 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:26.535 7f4dc8fe1700 1 --2- 172.21.2.102:0/1443563896 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4da8018730 0x7f4da801ac10 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:26.531 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:26.535 7f4dc8fe1700 1 --2- 172.21.2.102:0/1443563896 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4dc40b2a00 0x7f4dc40b2e00 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:26.531 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:26.535 7f4dc8fe1700 1 --2- 172.21.2.102:0/1443563896 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4dc4106110 0x7f4dc409f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:26.531 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:26.535 7f4dc8fe1700 1 --2- 172.21.2.102:0/1443563896 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4dc4105d10 0x7f4dc409eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:26.531 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:26.535 7f4dc8fe1700 1 -- 172.21.2.102:0/1443563896 >> 172.21.2.102:0/1443563896 conn(0x7f4dc40c36f0 msgr2=0x7f4dc40bd640 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:26.531 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:26.535 7f4dc8fe1700 1 -- 172.21.2.102:0/1443563896 shutdown_connections 2021-02-10T11:21:26.532 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:26.535 7f4dc8fe1700 1 -- 172.21.2.102:0/1443563896 wait complete. 2021-02-10T11:21:26.539 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000344429 2021-02-10T11:21:26.975 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:26.963 7fe51abb5700 1 Processor -- start 2021-02-10T11:21:26.976 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:26.963 7fe51abb5700 1 -- start start 2021-02-10T11:21:26.976 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:26.964 7fe51abb5700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe5141b0870 0x7fe5140f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:26.976 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:26.964 7fe51abb5700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fe5140efe00 0x7fe5140f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:26.976 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:26.964 7fe51abb5700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5141b0c70 con 0x7fe5141b0470 2021-02-10T11:21:26.976 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:26.964 7fe51abb5700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5140f0b50 con 0x7fe5140efe00 2021-02-10T11:21:26.976 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:26.964 7fe51abb5700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5140f0820 con 0x7fe5141b0870 2021-02-10T11:21:26.976 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:26.964 7fe5189d1700 1 --1- >> v1:172.21.2.113:6789/0 conn(0x7fe5141b0470 0x7fe5140ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.113:6789/0 says I am v1:172.21.2.102:54132/0 (socket says 172.21.2.102:54132) 2021-02-10T11:21:26.976 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:26.964 7fe5189d1700 1 -- 172.21.2.102:0/4154159904 learned_addr learned my addr 172.21.2.102:0/4154159904 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:21:26.976 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:26.964 7fe513fff700 1 --2- 172.21.2.102:0/4154159904 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe5141b0870 0x7fe5140f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:26.976 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:26.964 7fe5191d2700 1 --2- 172.21.2.102:0/4154159904 >> v2:172.21.2.113:3300/0 conn(0x7fe5140efe00 0x7fe5140f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:26.976 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:26.964 7fe5137fe700 1 -- 172.21.2.102:0/4154159904 <== mon.1 v1:172.21.2.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2814250713 0 0) 0x7fe50c000fe0 con 0x7fe5141b0470 2021-02-10T11:21:26.977 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:26.964 7fe5137fe700 1 -- 172.21.2.102:0/4154159904 --> v1:172.21.2.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4f8002fe0 con 0x7fe5141b0470 2021-02-10T11:21:26.977 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:26.965 7fe513fff700 1 -- 172.21.2.102:0/4154159904 >> v2:172.21.2.113:3300/0 conn(0x7fe5140efe00 msgr2=0x7fe5140f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:21:26.977 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:26.965 7fe513fff700 1 --2- 172.21.2.102:0/4154159904 >> v2:172.21.2.113:3300/0 conn(0x7fe5140efe00 0x7fe5140f0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:26.977 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:26.965 7fe513fff700 1 -- 172.21.2.102:0/4154159904 >> v1:172.21.2.113:6789/0 conn(0x7fe5141b0470 legacy=0x7fe5140ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:26.977 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:26.965 7fe513fff700 1 -- 172.21.2.102:0/4154159904 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5140d2fb0 con 0x7fe5141b0870 2021-02-10T11:21:26.977 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:26.965 7fe513fff700 1 --2- 172.21.2.102:0/4154159904 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe5141b0870 0x7fe5140f20e0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7fe500006650 tx=0x7fe500009a80).ready entity=mon.2 client_cookie=ae6ebcfa96b99006 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:26.977 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:26.965 7fe5137fe700 1 -- 172.21.2.102:0/4154159904 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe50000c0c0 con 0x7fe5141b0870 2021-02-10T11:21:26.977 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:26.965 7fe5137fe700 1 -- 172.21.2.102:0/4154159904 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe50000c340 con 0x7fe5141b0870 2021-02-10T11:21:26.977 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:26.965 7fe51abb5700 1 -- 172.21.2.102:0/4154159904 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe5141b0870 msgr2=0x7fe5140f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:26.977 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:26.965 7fe51abb5700 1 --2- 172.21.2.102:0/4154159904 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe5141b0870 0x7fe5140f20e0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7fe500006650 tx=0x7fe500009a80).stop 2021-02-10T11:21:26.977 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:26.965 7fe5137fe700 1 -- 172.21.2.102:0/4154159904 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe5000009b0 con 0x7fe5141b0870 2021-02-10T11:21:26.977 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:26.966 7fe51abb5700 1 -- 172.21.2.102:0/4154159904 shutdown_connections 2021-02-10T11:21:26.977 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:26.966 7fe51abb5700 1 --2- 172.21.2.102:0/4154159904 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe5141b0870 0x7fe5140f20e0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:26.978 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:26.966 7fe51abb5700 1 --2- 172.21.2.102:0/4154159904 >> v2:172.21.2.113:3300/0 conn(0x7fe5140efe00 0x7fe5140f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:26.978 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:26.966 7fe51abb5700 1 -- 172.21.2.102:0/4154159904 >> 172.21.2.102:0/4154159904 conn(0x7fe5140ce3e0 msgr2=0x7fe5140e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:26.978 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:26.966 7fe51abb5700 1 -- 172.21.2.102:0/4154159904 shutdown_connections 2021-02-10T11:21:26.978 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:26.966 7fe51abb5700 1 -- 172.21.2.102:0/4154159904 wait complete. 2021-02-10T11:21:26.979 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:26.981 7fe51abb5700 1 Processor -- start 2021-02-10T11:21:26.979 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:26.981 7fe51abb5700 1 -- start start 2021-02-10T11:21:26.979 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:26.982 7fe51abb5700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe5140a0420 0x7fe5140a0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:26.979 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:26.982 7fe51abb5700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe514189d00 0x7fe51418a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:26.980 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:26.982 7fe51abb5700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe51418a720 0x7fe5140b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:26.980 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:26.982 7fe51abb5700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe514189780 con 0x7fe51418a720 2021-02-10T11:21:26.980 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:26.982 7fe51abb5700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe51418ab20 con 0x7fe5140a0420 2021-02-10T11:21:26.980 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:26.982 7fe51abb5700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5140a1140 con 0x7fe514189d00 2021-02-10T11:21:26.980 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:26.982 7fe5191d2700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe51418a720 0x7fe5140b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:26.980 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:26.982 7fe5189d1700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe5140a0420 0x7fe5140a0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:26.980 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:26.982 7fe5191d2700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe51418a720 0x7fe5140b7be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:54592/0 (socket says 172.21.2.102:54592) 2021-02-10T11:21:26.980 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:26.982 7fe5189d1700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe5140a0420 0x7fe5140a0820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.113:3300/0 says I am v2:172.21.2.102:58020/0 (socket says 172.21.2.102:58020) 2021-02-10T11:21:26.981 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:26.982 7fe5191d2700 1 -- 172.21.2.102:0/199955288 learned_addr learned my addr 172.21.2.102:0/199955288 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:21:26.981 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:26.982 7fe513fff700 1 --2- 172.21.2.102:0/199955288 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe514189d00 0x7fe51418a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:26.981 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:26.983 7fe5191d2700 1 -- 172.21.2.102:0/199955288 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe514189d00 msgr2=0x7fe51418a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:26.981 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:26.983 7fe5191d2700 1 --2- 172.21.2.102:0/199955288 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe514189d00 0x7fe51418a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:26.981 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:26.983 7fe5191d2700 1 -- 172.21.2.102:0/199955288 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe5140a0420 msgr2=0x7fe5140a0820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:26.981 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:26.983 7fe5191d2700 1 --2- 172.21.2.102:0/199955288 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe5140a0420 0x7fe5140a0820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:26.981 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:26.983 7fe5191d2700 1 -- 172.21.2.102:0/199955288 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5140b8200 con 0x7fe51418a720 2021-02-10T11:21:26.981 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:26.983 7fe5189d1700 1 --2- 172.21.2.102:0/199955288 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe5140a0420 0x7fe5140a0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:21:26.981 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:26.983 7fe5191d2700 1 --2- 172.21.2.102:0/199955288 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe51418a720 0x7fe5140b7be0 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 rx=0x7fe50400b5e0 tx=0x7fe50400b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:26.981 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:26.983 7fe5117fa700 1 -- 172.21.2.102:0/199955288 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe504006370 con 0x7fe51418a720 2021-02-10T11:21:26.981 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:26.983 7fe5117fa700 1 -- 172.21.2.102:0/199955288 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe50400bd70 con 0x7fe51418a720 2021-02-10T11:21:26.981 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:26.983 7fe5117fa700 1 -- 172.21.2.102:0/199955288 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe504002560 con 0x7fe51418a720 2021-02-10T11:21:26.982 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:26.983 7fe51abb5700 1 -- 172.21.2.102:0/199955288 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5140b8200 con 0x7fe51418a720 2021-02-10T11:21:26.982 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:26.983 7fe51abb5700 1 -- 172.21.2.102:0/199955288 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5140b8200 con 0x7fe51418a720 2021-02-10T11:21:26.982 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:26.984 7fe5117fa700 1 -- 172.21.2.102:0/199955288 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fe504005330 con 0x7fe51418a720 2021-02-10T11:21:26.982 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:26.984 7fe5117fa700 1 --2- 172.21.2.102:0/199955288 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe4f00186c0 0x7fe4f001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:26.982 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:26.984 7fe5117fa700 1 -- 172.21.2.102:0/199955288 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fe504032410 con 0x7fe51418a720 2021-02-10T11:21:26.982 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:26.984 7fe5189d1700 1 --2- 172.21.2.102:0/199955288 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe4f00186c0 0x7fe4f001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:26.982 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:26.984 7fe5189d1700 1 --2- 172.21.2.102:0/199955288 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe4f00186c0 0x7fe4f001abc0 crc :-1 s=READY pgs=452 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:27.006 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:27.012 7fe51abb5700 1 -- 172.21.2.102:0/199955288 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe514105ae0 con 0x7fe51418a720 2021-02-10T11:21:27.009 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:27.014 7fe5117fa700 1 -- 172.21.2.102:0/199955288 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fe50402e230 con 0x7fe51418a720 2021-02-10T11:21:27.136 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:27.141 7fe51abb5700 1 -- 172.21.2.102:0/199955288 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344429", "target": ["mgr", ""]}) v1 -- 0x7fe5140f0b50 con 0x7fe4f00186c0 2021-02-10T11:21:27.147 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:27.152 7fe5117fa700 1 -- 172.21.2.102:0/199955288 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fe50c0067f0 con 0x7fe4f00186c0 2021-02-10T11:21:27.153 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:27.158 7fe51abb5700 1 -- 172.21.2.102:0/199955288 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe4f00186c0 msgr2=0x7fe4f001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:27.153 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:27.158 7fe51abb5700 1 --2- 172.21.2.102:0/199955288 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe4f00186c0 0x7fe4f001abc0 crc :-1 s=READY pgs=452 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:27.153 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:27.158 7fe51abb5700 1 -- 172.21.2.102:0/199955288 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe51418a720 msgr2=0x7fe5140b7be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:27.153 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:27.158 7fe51abb5700 1 --2- 172.21.2.102:0/199955288 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe51418a720 0x7fe5140b7be0 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 rx=0x7fe50400b5e0 tx=0x7fe50400b980).stop 2021-02-10T11:21:27.154 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:27.159 7fe51abb5700 1 -- 172.21.2.102:0/199955288 shutdown_connections 2021-02-10T11:21:27.154 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:27.159 7fe51abb5700 1 --2- 172.21.2.102:0/199955288 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe51418a720 0x7fe5140b7be0 unknown :-1 s=CLOSED pgs=925 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:27.154 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:27.159 7fe51abb5700 1 --2- 172.21.2.102:0/199955288 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe514189d00 0x7fe51418a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:27.154 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:27.159 7fe51abb5700 1 --2- 172.21.2.102:0/199955288 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe4f00186c0 0x7fe4f001abc0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:27.154 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:27.159 7fe51abb5700 1 --2- 172.21.2.102:0/199955288 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe5140a0420 0x7fe5140a0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:27.154 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:27.159 7fe51abb5700 1 -- 172.21.2.102:0/199955288 >> 172.21.2.102:0/199955288 conn(0x7fe514094e10 msgr2=0x7fe5140a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:27.154 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:27.159 7fe51abb5700 1 -- 172.21.2.102:0/199955288 shutdown_connections 2021-02-10T11:21:27.154 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:27.159 7fe51abb5700 1 -- 172.21.2.102:0/199955288 wait complete. 2021-02-10T11:21:27.162 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000344430 2021-02-10T11:21:27.593 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:27.582 7f8f3f878700 1 Processor -- start 2021-02-10T11:21:27.593 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:27.582 7f8f3f878700 1 -- start start 2021-02-10T11:21:27.594 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:27.583 7f8f3f878700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8f380f05f0 0x7f8f380f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:27.594 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:27.583 7f8f3f878700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f381b0c70 con 0x7f8f381b0870 2021-02-10T11:21:27.594 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:27.583 7f8f3f878700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f380f0b50 con 0x7f8f381b0470 2021-02-10T11:21:27.594 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:27.583 7f8f3f878700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f380e5ea0 con 0x7f8f380f05f0 2021-02-10T11:21:27.594 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:27.583 7f8f3d694700 1 --1- >> v1:172.21.2.113:6789/0 conn(0x7f8f381b0470 0x7f8f380ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.113:6789/0 says I am v1:172.21.2.102:54146/0 (socket says 172.21.2.102:54146) 2021-02-10T11:21:27.594 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:27.583 7f8f3d694700 1 -- 172.21.2.102:0/995789428 learned_addr learned my addr 172.21.2.102:0/995789428 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:21:27.594 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:27.583 7f8f3ce93700 1 --1- 172.21.2.102:0/995789428 >> v1:172.21.2.102:6789/0 conn(0x7f8f381b0870 0x7f8f380efe00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:46824/0 (socket says 172.21.2.102:46824) 2021-02-10T11:21:27.594 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:27.583 7f8f3de95700 1 --2- 172.21.2.102:0/995789428 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8f380f05f0 0x7f8f380f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:27.594 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:27.583 7f8f2ffff700 1 -- 172.21.2.102:0/995789428 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3678952964 0 0) 0x7f8f280009b0 con 0x7f8f381b0870 2021-02-10T11:21:27.594 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:27.583 7f8f3de95700 1 -- 172.21.2.102:0/995789428 >> v1:172.21.2.113:6789/0 conn(0x7f8f381b0470 legacy=0x7f8f380ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:27.594 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:27.583 7f8f3de95700 1 -- 172.21.2.102:0/995789428 >> v1:172.21.2.102:6789/0 conn(0x7f8f381b0870 legacy=0x7f8f380efe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:27.594 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:27.584 7f8f3de95700 1 -- 172.21.2.102:0/995789428 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f380d2fb0 con 0x7f8f380f05f0 2021-02-10T11:21:27.595 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:27.584 7f8f3de95700 1 --2- 172.21.2.102:0/995789428 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8f380f05f0 0x7f8f380f20e0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7f8f34006e50 tx=0x7f8f34009df0).ready entity=mon.2 client_cookie=fb97edd9f8a52339 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:27.595 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:27.584 7f8f2ffff700 1 -- 172.21.2.102:0/995789428 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8f3400b510 con 0x7f8f380f05f0 2021-02-10T11:21:27.595 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:27.584 7f8f2ffff700 1 -- 172.21.2.102:0/995789428 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f34003030 con 0x7f8f380f05f0 2021-02-10T11:21:27.595 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:27.584 7f8f2ffff700 1 -- 172.21.2.102:0/995789428 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8f34000f90 con 0x7f8f380f05f0 2021-02-10T11:21:27.595 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:27.584 7f8f3f878700 1 -- 172.21.2.102:0/995789428 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8f380f05f0 msgr2=0x7f8f380f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:27.595 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:27.584 7f8f3f878700 1 --2- 172.21.2.102:0/995789428 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8f380f05f0 0x7f8f380f20e0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7f8f34006e50 tx=0x7f8f34009df0).stop 2021-02-10T11:21:27.595 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:27.584 7f8f3f878700 1 -- 172.21.2.102:0/995789428 shutdown_connections 2021-02-10T11:21:27.595 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:27.584 7f8f3f878700 1 --2- 172.21.2.102:0/995789428 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8f380f05f0 0x7f8f380f20e0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:27.595 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:27.584 7f8f3f878700 1 -- 172.21.2.102:0/995789428 >> 172.21.2.102:0/995789428 conn(0x7f8f380ce3e0 msgr2=0x7f8f380e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:27.595 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:27.585 7f8f3f878700 1 -- 172.21.2.102:0/995789428 shutdown_connections 2021-02-10T11:21:27.595 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:27.585 7f8f3f878700 1 -- 172.21.2.102:0/995789428 wait complete. 2021-02-10T11:21:27.597 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:27.599 7f8f3f878700 1 Processor -- start 2021-02-10T11:21:27.597 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:27.599 7f8f3f878700 1 -- start start 2021-02-10T11:21:27.597 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:27.600 7f8f3f878700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8f3808b0b0 0x7f8f3807e910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:27.597 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:27.600 7f8f3f878700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8f3807ef30 0x7f8f381b5420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:27.597 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:27.600 7f8f3f878700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8f381b5a40 0x7f8f38092e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:27.597 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:27.600 7f8f3f878700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f380928e0 con 0x7f8f381b5a40 2021-02-10T11:21:27.598 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:27.600 7f8f3f878700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f3807f330 con 0x7f8f3808b0b0 2021-02-10T11:21:27.598 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:27.600 7f8f3f878700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f3808b4b0 con 0x7f8f3807ef30 2021-02-10T11:21:27.598 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:27.600 7f8f3de95700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8f381b5a40 0x7f8f38092e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:27.598 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:27.600 7f8f3de95700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8f381b5a40 0x7f8f38092e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:54606/0 (socket says 172.21.2.102:54606) 2021-02-10T11:21:27.598 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:27.600 7f8f3de95700 1 -- 172.21.2.102:0/3294819136 learned_addr learned my addr 172.21.2.102:0/3294819136 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:21:27.598 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:27.600 7f8f3ce93700 1 --2- 172.21.2.102:0/3294819136 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8f3807ef30 0x7f8f381b5420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:27.598 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:27.600 7f8f3d694700 1 --2- 172.21.2.102:0/3294819136 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8f3808b0b0 0x7f8f3807e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:27.598 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:27.601 7f8f3de95700 1 -- 172.21.2.102:0/3294819136 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8f3807ef30 msgr2=0x7f8f381b5420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:27.598 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:27.601 7f8f3de95700 1 --2- 172.21.2.102:0/3294819136 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8f3807ef30 0x7f8f381b5420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:27.598 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:27.601 7f8f3de95700 1 -- 172.21.2.102:0/3294819136 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8f3808b0b0 msgr2=0x7f8f3807e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:27.598 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:27.601 7f8f3de95700 1 --2- 172.21.2.102:0/3294819136 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8f3808b0b0 0x7f8f3807e910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:27.598 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:27.601 7f8f3de95700 1 -- 172.21.2.102:0/3294819136 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f381be1d0 con 0x7f8f381b5a40 2021-02-10T11:21:27.598 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:27.601 7f8f3ce93700 1 --2- 172.21.2.102:0/3294819136 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8f3807ef30 0x7f8f381b5420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:21:27.599 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:27.601 7f8f3de95700 1 --2- 172.21.2.102:0/3294819136 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8f381b5a40 0x7f8f38092e60 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 rx=0x7f8f34009fc0 tx=0x7f8f3400bd40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:27.599 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:27.601 7f8f2dffb700 1 -- 172.21.2.102:0/3294819136 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8f340013b0 con 0x7f8f381b5a40 2021-02-10T11:21:27.599 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:27.601 7f8f2dffb700 1 -- 172.21.2.102:0/3294819136 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f340015b0 con 0x7f8f381b5a40 2021-02-10T11:21:27.599 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:27.601 7f8f2dffb700 1 -- 172.21.2.102:0/3294819136 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8f34001a10 con 0x7f8f381b5a40 2021-02-10T11:21:27.599 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:27.601 7f8f3f878700 1 -- 172.21.2.102:0/3294819136 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f381be1d0 con 0x7f8f381b5a40 2021-02-10T11:21:27.599 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:27.601 7f8f3f878700 1 -- 172.21.2.102:0/3294819136 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f381b6130 con 0x7f8f381b5a40 2021-02-10T11:21:27.599 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:27.602 7f8f2dffb700 1 -- 172.21.2.102:0/3294819136 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f8f34001790 con 0x7f8f381b5a40 2021-02-10T11:21:27.599 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:27.602 7f8f2dffb700 1 --2- 172.21.2.102:0/3294819136 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8f180186c0 0x7f8f1801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:27.599 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:27.602 7f8f2dffb700 1 -- 172.21.2.102:0/3294819136 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f8f34032b70 con 0x7f8f381b5a40 2021-02-10T11:21:27.599 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:27.602 7f8f3d694700 1 --2- 172.21.2.102:0/3294819136 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8f180186c0 0x7f8f1801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:27.599 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:27.602 7f8f3d694700 1 --2- 172.21.2.102:0/3294819136 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8f180186c0 0x7f8f1801abc0 crc :-1 s=READY pgs=453 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:27.626 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:27.631 7f8f3f878700 1 -- 172.21.2.102:0/3294819136 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f3807e6e0 con 0x7f8f381b5a40 2021-02-10T11:21:27.629 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:27.634 7f8f2dffb700 1 -- 172.21.2.102:0/3294819136 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f8f340017c0 con 0x7f8f381b5a40 2021-02-10T11:21:27.758 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:27.764 7f8f3f878700 1 -- 172.21.2.102:0/3294819136 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344430", "target": ["mgr", ""]}) v1 -- 0x7f8f381b6130 con 0x7f8f180186c0 2021-02-10T11:21:27.768 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:27.774 7f8f2dffb700 1 -- 172.21.2.102:0/3294819136 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f8f30006a30 con 0x7f8f180186c0 2021-02-10T11:21:27.775 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:27.780 7f8f3f878700 1 -- 172.21.2.102:0/3294819136 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8f180186c0 msgr2=0x7f8f1801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:27.775 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:27.780 7f8f3f878700 1 --2- 172.21.2.102:0/3294819136 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8f180186c0 0x7f8f1801abc0 crc :-1 s=READY pgs=453 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:27.775 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:27.781 7f8f3f878700 1 -- 172.21.2.102:0/3294819136 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8f381b5a40 msgr2=0x7f8f38092e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:27.775 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:27.781 7f8f3f878700 1 --2- 172.21.2.102:0/3294819136 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8f381b5a40 0x7f8f38092e60 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 rx=0x7f8f34009fc0 tx=0x7f8f3400bd40).stop 2021-02-10T11:21:27.776 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:27.781 7f8f3f878700 1 -- 172.21.2.102:0/3294819136 shutdown_connections 2021-02-10T11:21:27.776 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:27.781 7f8f3f878700 1 --2- 172.21.2.102:0/3294819136 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8f381b5a40 0x7f8f38092e60 unknown :-1 s=CLOSED pgs=927 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:27.776 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:27.781 7f8f3f878700 1 --2- 172.21.2.102:0/3294819136 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8f3807ef30 0x7f8f381b5420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:27.776 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:27.781 7f8f3f878700 1 --2- 172.21.2.102:0/3294819136 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8f180186c0 0x7f8f1801abc0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:27.776 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:27.781 7f8f3f878700 1 --2- 172.21.2.102:0/3294819136 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8f3808b0b0 0x7f8f3807e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:27.776 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:27.781 7f8f3f878700 1 -- 172.21.2.102:0/3294819136 >> 172.21.2.102:0/3294819136 conn(0x7f8f38088700 msgr2=0x7f8f38095780 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:27.776 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:27.781 7f8f3f878700 1 -- 172.21.2.102:0/3294819136 shutdown_connections 2021-02-10T11:21:27.776 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:27.781 7f8f3f878700 1 -- 172.21.2.102:0/3294819136 wait complete. 2021-02-10T11:21:27.784 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000344431 2021-02-10T11:21:28.216 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:28.202 7f5b87a00700 1 Processor -- start 2021-02-10T11:21:28.216 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:28.202 7f5b87a00700 1 -- start start 2021-02-10T11:21:28.216 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:28.202 7f5b87a00700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f5b800f05f0 0x7f5b800f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:28.216 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:28.202 7f5b87a00700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b801b0c70 con 0x7f5b801b0470 2021-02-10T11:21:28.216 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:28.202 7f5b87a00700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b800f0b50 con 0x7f5b801b0870 2021-02-10T11:21:28.216 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:28.202 7f5b87a00700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b800e5ea0 con 0x7f5b800f05f0 2021-02-10T11:21:28.217 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:28.203 7f5b8581c700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f5b801b0470 0x7f5b800ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:46834/0 (socket says 172.21.2.102:46834) 2021-02-10T11:21:28.217 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:28.203 7f5b8581c700 1 -- 172.21.2.102:0/1335290859 learned_addr learned my addr 172.21.2.102:0/1335290859 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:21:28.217 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:28.203 7f5b8601d700 1 --2- 172.21.2.102:0/1335290859 >> v2:172.21.2.113:3300/0 conn(0x7f5b800f05f0 0x7f5b800f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:28.217 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:28.203 7f5b8481a700 1 -- 172.21.2.102:0/1335290859 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2798692181 0 0) 0x7f5b70001180 con 0x7f5b801b0470 2021-02-10T11:21:28.217 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:28.203 7f5b8481a700 1 -- 172.21.2.102:0/1335290859 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b68002fe0 con 0x7f5b801b0470 2021-02-10T11:21:28.217 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:28.203 7f5b8481a700 1 -- 172.21.2.102:0/1335290859 <== mon.1 v1:172.21.2.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2626752402 0 0) 0x7f5b74000fe0 con 0x7f5b801b0870 2021-02-10T11:21:28.217 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:28.203 7f5b8481a700 1 -- 172.21.2.102:0/1335290859 --> v1:172.21.2.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b68003210 con 0x7f5b801b0870 2021-02-10T11:21:28.217 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:28.203 7f5b8481a700 1 -- 172.21.2.102:0/1335290859 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2529348749 0 0) 0x7f5b700010e0 con 0x7f5b801b0470 2021-02-10T11:21:28.217 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:28.203 7f5b8481a700 1 -- 172.21.2.102:0/1335290859 >> v2:172.21.2.113:3300/0 conn(0x7f5b800f05f0 msgr2=0x7f5b800f20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:21:28.217 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:28.203 7f5b8481a700 1 --2- 172.21.2.102:0/1335290859 >> v2:172.21.2.113:3300/0 conn(0x7f5b800f05f0 0x7f5b800f20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:28.218 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:28.203 7f5b8481a700 1 -- 172.21.2.102:0/1335290859 >> v1:172.21.2.113:6789/0 conn(0x7f5b801b0870 legacy=0x7f5b800efe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:28.218 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:28.203 7f5b8481a700 1 -- 172.21.2.102:0/1335290859 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b800d3330 con 0x7f5b801b0470 2021-02-10T11:21:28.218 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:28.203 7f5b8481a700 1 -- 172.21.2.102:0/1335290859 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7f5b70001600 con 0x7f5b801b0470 2021-02-10T11:21:28.218 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:28.203 7f5b8481a700 1 -- 172.21.2.102:0/1335290859 >> v1:172.21.2.102:6789/0 conn(0x7f5b801b0470 legacy=0x7f5b800ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:28.218 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:28.203 7f5b8481a700 1 --2- 172.21.2.102:0/1335290859 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5b68003b90 0x7f5b68005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:28.218 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:28.204 7f5b8481a700 1 --2- 172.21.2.102:0/1335290859 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5b680067a0 0x7f5b68008ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:28.218 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:28.204 7f5b8481a700 1 --2- 172.21.2.102:0/1335290859 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5b680095a0 0x7f5b6800bac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:28.218 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:28.204 7f5b8481a700 1 -- 172.21.2.102:0/1335290859 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b68050320 con 0x7f5b680095a0 2021-02-10T11:21:28.218 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:28.204 7f5b8481a700 1 -- 172.21.2.102:0/1335290859 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b68050500 con 0x7f5b680067a0 2021-02-10T11:21:28.219 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:28.204 7f5b8481a700 1 -- 172.21.2.102:0/1335290859 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b68050720 con 0x7f5b68003b90 2021-02-10T11:21:28.219 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:28.204 7f5b8601d700 1 --2- 172.21.2.102:0/1335290859 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5b680095a0 0x7f5b6800bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:28.219 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:28.204 7f5b8501b700 1 --2- 172.21.2.102:0/1335290859 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5b680067a0 0x7f5b68008ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:28.219 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:28.204 7f5b8581c700 1 --2- 172.21.2.102:0/1335290859 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5b68003b90 0x7f5b68005fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:28.219 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:28.204 7f5b8601d700 1 -- 172.21.2.102:0/1335290859 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5b68003b90 msgr2=0x7f5b68005fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:21:28.219 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:28.204 7f5b8601d700 1 --2- 172.21.2.102:0/1335290859 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5b68003b90 0x7f5b68005fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:28.219 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:28.204 7f5b8601d700 1 -- 172.21.2.102:0/1335290859 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5b680067a0 msgr2=0x7f5b68008ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:21:28.219 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:28.204 7f5b8601d700 1 --2- 172.21.2.102:0/1335290859 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5b680067a0 0x7f5b68008ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:28.219 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:28.204 7f5b8601d700 1 -- 172.21.2.102:0/1335290859 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b680509c0 con 0x7f5b680095a0 2021-02-10T11:21:28.219 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:28.204 7f5b8581c700 1 --2- 172.21.2.102:0/1335290859 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5b68003b90 0x7f5b68005fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:21:28.220 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:28.204 7f5b8601d700 1 -- 172.21.2.102:0/1335290859 reap_dead start 2021-02-10T11:21:28.220 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:28.204 7f5b8601d700 1 --2- 172.21.2.102:0/1335290859 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5b680095a0 0x7f5b6800bac0 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 rx=0x7f5b7800c940 tx=0x7f5b7800cce0).ready entity=mon.0 client_cookie=9345ad5539d93dd3 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:28.220 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:28.204 7f5b8481a700 1 -- 172.21.2.102:0/1335290859 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5b780067d0 con 0x7f5b680095a0 2021-02-10T11:21:28.220 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:28.204 7f5b8481a700 1 -- 172.21.2.102:0/1335290859 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b7800ce20 con 0x7f5b680095a0 2021-02-10T11:21:28.220 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:28.204 7f5b8481a700 1 -- 172.21.2.102:0/1335290859 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5b78009500 con 0x7f5b680095a0 2021-02-10T11:21:28.220 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:28.204 7f5b87a00700 1 -- 172.21.2.102:0/1335290859 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5b680095a0 msgr2=0x7f5b6800bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:28.220 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:28.204 7f5b87a00700 1 --2- 172.21.2.102:0/1335290859 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5b680095a0 0x7f5b6800bac0 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 rx=0x7f5b7800c940 tx=0x7f5b7800cce0).stop 2021-02-10T11:21:28.220 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:28.205 7f5b87a00700 1 -- 172.21.2.102:0/1335290859 shutdown_connections 2021-02-10T11:21:28.220 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:28.205 7f5b87a00700 1 --2- 172.21.2.102:0/1335290859 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5b680095a0 0x7f5b6800bac0 unknown :-1 s=CLOSED pgs=929 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:28.220 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:28.205 7f5b87a00700 1 -- 172.21.2.102:0/1335290859 >> 172.21.2.102:0/1335290859 conn(0x7f5b800ce3e0 msgr2=0x7f5b800e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:28.221 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:28.205 7f5b87a00700 1 -- 172.21.2.102:0/1335290859 shutdown_connections 2021-02-10T11:21:28.221 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:28.205 7f5b87a00700 1 -- 172.21.2.102:0/1335290859 wait complete. 2021-02-10T11:21:28.221 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:28.220 7f5b87a00700 1 Processor -- start 2021-02-10T11:21:28.221 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:28.220 7f5b87a00700 1 -- start start 2021-02-10T11:21:28.221 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:28.220 7f5b87a00700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5b8009b930 0x7f5b800b7c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:28.221 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:28.220 7f5b87a00700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5b800b8410 0x7f5b800b8810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:28.221 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:28.220 7f5b87a00700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5b800c1f80 0x7f5b800c2380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:28.221 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:28.220 7f5b87a00700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b800c1a00 con 0x7f5b800c1f80 2021-02-10T11:21:28.221 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:28.220 7f5b87a00700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b800b8e30 con 0x7f5b800b8410 2021-02-10T11:21:28.222 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:28.220 7f5b87a00700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b800b9050 con 0x7f5b8009b930 2021-02-10T11:21:28.222 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:28.221 7f5b8581c700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5b8009b930 0x7f5b800b7c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:28.222 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:28.221 7f5b8601d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5b800c1f80 0x7f5b800c2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:28.222 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:28.221 7f5b8501b700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5b800b8410 0x7f5b800b8810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:28.222 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:28.221 7f5b8581c700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5b8009b930 0x7f5b800b7c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.113:3301/0 says I am v2:172.21.2.102:60156/0 (socket says 172.21.2.102:60156) 2021-02-10T11:21:28.222 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:28.221 7f5b8581c700 1 -- 172.21.2.102:0/965380725 learned_addr learned my addr 172.21.2.102:0/965380725 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:21:28.222 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:28.221 7f5b8581c700 1 -- 172.21.2.102:0/965380725 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5b800b8410 msgr2=0x7f5b800b8810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:28.222 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:28.221 7f5b8581c700 1 --2- 172.21.2.102:0/965380725 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5b800b8410 0x7f5b800b8810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:28.223 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:28.221 7f5b8581c700 1 -- 172.21.2.102:0/965380725 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5b800c1f80 msgr2=0x7f5b800c2380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:28.223 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:28.221 7f5b8581c700 1 --2- 172.21.2.102:0/965380725 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5b800c1f80 0x7f5b800c2380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:28.223 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:28.221 7f5b8581c700 1 -- 172.21.2.102:0/965380725 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b800c29a0 con 0x7f5b8009b930 2021-02-10T11:21:28.223 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:28.221 7f5b8581c700 1 --2- 172.21.2.102:0/965380725 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5b8009b930 0x7f5b800b7c20 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f5b7000a590 tx=0x7f5b7000a930).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:28.223 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:28.221 7f5b7e7fc700 1 -- 172.21.2.102:0/965380725 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5b70000ef0 con 0x7f5b8009b930 2021-02-10T11:21:28.223 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:28.221 7f5b7e7fc700 1 -- 172.21.2.102:0/965380725 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b7000ad20 con 0x7f5b8009b930 2021-02-10T11:21:28.223 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:28.221 7f5b7e7fc700 1 -- 172.21.2.102:0/965380725 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5b70004620 con 0x7f5b8009b930 2021-02-10T11:21:28.224 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:28.221 7f5b87a00700 1 -- 172.21.2.102:0/965380725 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b800c29a0 con 0x7f5b8009b930 2021-02-10T11:21:28.225 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:28.222 7f5b87a00700 1 -- 172.21.2.102:0/965380725 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b800c29a0 con 0x7f5b8009b930 2021-02-10T11:21:28.225 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:28.222 7f5b7e7fc700 1 -- 172.21.2.102:0/965380725 <== mon.2 v2:172.21.2.113:3301/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f5b700043a0 con 0x7f5b8009b930 2021-02-10T11:21:28.225 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:28.222 7f5b7e7fc700 1 --2- 172.21.2.102:0/965380725 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5b60018730 0x7f5b6001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:28.225 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:28.222 7f5b7e7fc700 1 -- 172.21.2.102:0/965380725 <== mon.2 v2:172.21.2.113:3301/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f5b700327f0 con 0x7f5b8009b930 2021-02-10T11:21:28.225 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:28.222 7f5b8501b700 1 --2- 172.21.2.102:0/965380725 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5b60018730 0x7f5b6001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:28.225 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:28.223 7f5b8501b700 1 --2- 172.21.2.102:0/965380725 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5b60018730 0x7f5b6001ac10 crc :-1 s=READY pgs=454 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:28.245 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:28.251 7f5b87a00700 1 -- 172.21.2.102:0/965380725 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b80105b30 con 0x7f5b8009b930 2021-02-10T11:21:28.248 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:28.254 7f5b7e7fc700 1 -- 172.21.2.102:0/965380725 <== mon.2 v2:172.21.2.113:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f5b70007070 con 0x7f5b8009b930 2021-02-10T11:21:28.381 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:28.387 7f5b87a00700 1 -- 172.21.2.102:0/965380725 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344431", "target": ["mgr", ""]}) v1 -- 0x7f5b80105b30 con 0x7f5b60018730 2021-02-10T11:21:28.391 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:28.396 7f5b7e7fc700 1 -- 172.21.2.102:0/965380725 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f5b74006e60 con 0x7f5b60018730 2021-02-10T11:21:28.398 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:28.403 7f5b87a00700 1 -- 172.21.2.102:0/965380725 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5b60018730 msgr2=0x7f5b6001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:28.398 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:28.403 7f5b87a00700 1 --2- 172.21.2.102:0/965380725 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5b60018730 0x7f5b6001ac10 crc :-1 s=READY pgs=454 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:28.399 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:28.403 7f5b87a00700 1 -- 172.21.2.102:0/965380725 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5b8009b930 msgr2=0x7f5b800b7c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:28.399 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:28.403 7f5b87a00700 1 --2- 172.21.2.102:0/965380725 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5b8009b930 0x7f5b800b7c20 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f5b7000a590 tx=0x7f5b7000a930).stop 2021-02-10T11:21:28.399 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:28.403 7f5b87a00700 1 -- 172.21.2.102:0/965380725 shutdown_connections 2021-02-10T11:21:28.399 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:28.403 7f5b87a00700 1 --2- 172.21.2.102:0/965380725 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5b800c1f80 0x7f5b800c2380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:28.399 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:28.403 7f5b87a00700 1 --2- 172.21.2.102:0/965380725 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5b60018730 0x7f5b6001ac10 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:28.399 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:28.403 7f5b87a00700 1 --2- 172.21.2.102:0/965380725 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5b800b8410 0x7f5b800b8810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:28.399 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:28.404 7f5b87a00700 1 --2- 172.21.2.102:0/965380725 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5b8009b930 0x7f5b800b7c20 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:28.399 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:28.404 7f5b87a00700 1 -- 172.21.2.102:0/965380725 >> 172.21.2.102:0/965380725 conn(0x7f5b80094e10 msgr2=0x7f5b800a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:28.399 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:28.404 7f5b87a00700 1 -- 172.21.2.102:0/965380725 shutdown_connections 2021-02-10T11:21:28.399 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:28.404 7f5b87a00700 1 -- 172.21.2.102:0/965380725 wait complete. 2021-02-10T11:21:28.407 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000344432 2021-02-10T11:21:28.838 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:28.826 7f4898cdb700 1 Processor -- start 2021-02-10T11:21:28.838 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:28.826 7f4898cdb700 1 -- start start 2021-02-10T11:21:28.838 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:28.827 7f4898cdb700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f48940f05f0 0x7f48940f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:28.838 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:28.827 7f4898cdb700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48941b0c70 con 0x7f48941b0470 2021-02-10T11:21:28.838 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:28.827 7f4898cdb700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48940f0b50 con 0x7f48941b0870 2021-02-10T11:21:28.838 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:28.827 7f4898cdb700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48940e5ea0 con 0x7f48940f05f0 2021-02-10T11:21:28.838 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:28.827 7f4892ae8700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f48941b0470 0x7f48940ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:46852/0 (socket says 172.21.2.102:46852) 2021-02-10T11:21:28.839 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:28.827 7f4892ae8700 1 -- 172.21.2.102:0/1359807606 learned_addr learned my addr 172.21.2.102:0/1359807606 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:21:28.839 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:28.827 7f48932e9700 1 --2- 172.21.2.102:0/1359807606 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f48940f05f0 0x7f48940f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:28.839 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:28.827 7f48922e7700 1 -- 172.21.2.102:0/1359807606 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3600041227 0 0) 0x7f4884000ce0 con 0x7f48941b0470 2021-02-10T11:21:28.839 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:28.827 7f48922e7700 1 -- 172.21.2.102:0/1359807606 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4880002fe0 con 0x7f48941b0470 2021-02-10T11:21:28.839 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:28.828 7f48932e9700 1 -- 172.21.2.102:0/1359807606 >> v1:172.21.2.113:6789/0 conn(0x7f48941b0870 legacy=0x7f48940efe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:28.839 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:28.828 7f48932e9700 1 -- 172.21.2.102:0/1359807606 >> v1:172.21.2.102:6789/0 conn(0x7f48941b0470 legacy=0x7f48940ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:28.839 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:28.828 7f48932e9700 1 -- 172.21.2.102:0/1359807606 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48940e52e0 con 0x7f48940f05f0 2021-02-10T11:21:28.839 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:28.828 7f48932e9700 1 --2- 172.21.2.102:0/1359807606 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f48940f05f0 0x7f48940f20e0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7f488c006e50 tx=0x7f488c009e70).ready entity=mon.2 client_cookie=fc7c4f4f1888709b server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:28.839 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:28.828 7f48922e7700 1 -- 172.21.2.102:0/1359807606 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f488c006a20 con 0x7f48940f05f0 2021-02-10T11:21:28.839 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:28.828 7f48922e7700 1 -- 172.21.2.102:0/1359807606 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f488c006790 con 0x7f48940f05f0 2021-02-10T11:21:28.839 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:28.828 7f48922e7700 1 -- 172.21.2.102:0/1359807606 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f488c00b700 con 0x7f48940f05f0 2021-02-10T11:21:28.839 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:28.828 7f4898cdb700 1 -- 172.21.2.102:0/1359807606 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f48940f05f0 msgr2=0x7f48940f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:28.839 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:28.828 7f4898cdb700 1 --2- 172.21.2.102:0/1359807606 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f48940f05f0 0x7f48940f20e0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7f488c006e50 tx=0x7f488c009e70).stop 2021-02-10T11:21:28.840 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:28.829 7f4898cdb700 1 -- 172.21.2.102:0/1359807606 shutdown_connections 2021-02-10T11:21:28.840 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:28.829 7f4898cdb700 1 --2- 172.21.2.102:0/1359807606 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f48940f05f0 0x7f48940f20e0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:28.840 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:28.829 7f4898cdb700 1 -- 172.21.2.102:0/1359807606 >> 172.21.2.102:0/1359807606 conn(0x7f48940ce3e0 msgr2=0x7f48940e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:28.840 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:28.829 7f4898cdb700 1 -- 172.21.2.102:0/1359807606 shutdown_connections 2021-02-10T11:21:28.840 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:28.829 7f4898cdb700 1 -- 172.21.2.102:0/1359807606 wait complete. 2021-02-10T11:21:28.841 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:28.844 7f4898cdb700 1 Processor -- start 2021-02-10T11:21:28.841 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:28.844 7f4898cdb700 1 -- start start 2021-02-10T11:21:28.842 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:28.844 7f4898cdb700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f48941a48a0 0x7f489409ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:28.842 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:28.844 7f4898cdb700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4894092530 0x7f48940a9690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:28.842 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:28.844 7f4898cdb700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f48940f26f0 0x7f48940a9cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:28.842 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:28.844 7f4898cdb700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4894091fb0 con 0x7f48941a48a0 2021-02-10T11:21:28.842 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:28.844 7f4898cdb700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48940aa2d0 con 0x7f48940f26f0 2021-02-10T11:21:28.842 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:28.844 7f4898cdb700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48940aa4f0 con 0x7f4894092530 2021-02-10T11:21:28.842 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:28.844 7f4892ae8700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f48941a48a0 0x7f489409ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:28.842 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:28.844 7f4892ae8700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f48941a48a0 0x7f489409ee00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:54636/0 (socket says 172.21.2.102:54636) 2021-02-10T11:21:28.842 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:28.844 7f4892ae8700 1 -- 172.21.2.102:0/1977417902 learned_addr learned my addr 172.21.2.102:0/1977417902 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:21:28.842 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:28.845 7f48932e9700 1 --2- 172.21.2.102:0/1977417902 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f48940f26f0 0x7f48940a9cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:28.842 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:28.845 7f488a2e7700 1 --2- 172.21.2.102:0/1977417902 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4894092530 0x7f48940a9690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:28.843 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:28.845 7f4892ae8700 1 -- 172.21.2.102:0/1977417902 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4894092530 msgr2=0x7f48940a9690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:28.843 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:28.845 7f4892ae8700 1 --2- 172.21.2.102:0/1977417902 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4894092530 0x7f48940a9690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:28.843 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:28.845 7f4892ae8700 1 -- 172.21.2.102:0/1977417902 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f48940f26f0 msgr2=0x7f48940a9cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:28.843 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:28.845 7f4892ae8700 1 --2- 172.21.2.102:0/1977417902 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f48940f26f0 0x7f48940a9cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:28.843 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:28.845 7f4892ae8700 1 -- 172.21.2.102:0/1977417902 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f489409f420 con 0x7f48941a48a0 2021-02-10T11:21:28.843 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:28.845 7f4892ae8700 1 --2- 172.21.2.102:0/1977417902 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f48941a48a0 0x7f489409ee00 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 rx=0x7f488400b5c0 tx=0x7f488400b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:28.843 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:28.845 7f488bfff700 1 -- 172.21.2.102:0/1977417902 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4884006030 con 0x7f48941a48a0 2021-02-10T11:21:28.843 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:28.845 7f488bfff700 1 -- 172.21.2.102:0/1977417902 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48840043a0 con 0x7f48941a48a0 2021-02-10T11:21:28.843 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:28.845 7f488bfff700 1 -- 172.21.2.102:0/1977417902 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4884004880 con 0x7f48941a48a0 2021-02-10T11:21:28.843 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:28.845 7f4898cdb700 1 -- 172.21.2.102:0/1977417902 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f489409f420 con 0x7f48941a48a0 2021-02-10T11:21:28.843 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:28.845 7f4898cdb700 1 -- 172.21.2.102:0/1977417902 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f489409f420 con 0x7f48941a48a0 2021-02-10T11:21:28.843 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:28.846 7f488bfff700 1 -- 172.21.2.102:0/1977417902 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4884004580 con 0x7f48941a48a0 2021-02-10T11:21:28.843 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:28.846 7f488bfff700 1 --2- 172.21.2.102:0/1977417902 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f48780186c0 0x7f487801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:28.843 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:28.846 7f488bfff700 1 -- 172.21.2.102:0/1977417902 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f48840325b0 con 0x7f48941a48a0 2021-02-10T11:21:28.844 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:28.846 7f488a2e7700 1 --2- 172.21.2.102:0/1977417902 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f48780186c0 0x7f487801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:28.844 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:28.846 7f488a2e7700 1 --2- 172.21.2.102:0/1977417902 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f48780186c0 0x7f487801abc0 crc :-1 s=READY pgs=455 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:28.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:28.876 7f4898cdb700 1 -- 172.21.2.102:0/1977417902 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48940edd10 con 0x7f48941a48a0 2021-02-10T11:21:28.873 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:28.879 7f488bfff700 1 -- 172.21.2.102:0/1977417902 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f4884006b80 con 0x7f48941a48a0 2021-02-10T11:21:28.999 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:29.004 7f4898cdb700 1 -- 172.21.2.102:0/1977417902 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344432", "target": ["mgr", ""]}) v1 -- 0x7f48940e52e0 con 0x7f48780186c0 2021-02-10T11:21:29.010 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:29.015 7f488bfff700 1 -- 172.21.2.102:0/1977417902 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f487c006b60 con 0x7f48780186c0 2021-02-10T11:21:29.017 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:29.022 7f4898cdb700 1 -- 172.21.2.102:0/1977417902 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f48780186c0 msgr2=0x7f487801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:29.017 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:29.022 7f4898cdb700 1 --2- 172.21.2.102:0/1977417902 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f48780186c0 0x7f487801abc0 crc :-1 s=READY pgs=455 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:29.017 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:29.022 7f4898cdb700 1 -- 172.21.2.102:0/1977417902 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f48941a48a0 msgr2=0x7f489409ee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:29.017 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:29.022 7f4898cdb700 1 --2- 172.21.2.102:0/1977417902 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f48941a48a0 0x7f489409ee00 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 rx=0x7f488400b5c0 tx=0x7f488400b960).stop 2021-02-10T11:21:29.018 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:29.022 7f4898cdb700 1 -- 172.21.2.102:0/1977417902 shutdown_connections 2021-02-10T11:21:29.018 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:29.022 7f4898cdb700 1 --2- 172.21.2.102:0/1977417902 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f48780186c0 0x7f487801abc0 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:29.018 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:29.022 7f4898cdb700 1 --2- 172.21.2.102:0/1977417902 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f48940f26f0 0x7f48940a9cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:29.018 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:29.022 7f4898cdb700 1 --2- 172.21.2.102:0/1977417902 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4894092530 0x7f48940a9690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:29.018 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:29.022 7f4898cdb700 1 --2- 172.21.2.102:0/1977417902 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f48941a48a0 0x7f489409ee00 unknown :-1 s=CLOSED pgs=931 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:29.018 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:29.022 7f4898cdb700 1 -- 172.21.2.102:0/1977417902 >> 172.21.2.102:0/1977417902 conn(0x7f48940c40e0 msgr2=0x7f4894094eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:29.018 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:29.022 7f4898cdb700 1 -- 172.21.2.102:0/1977417902 shutdown_connections 2021-02-10T11:21:29.018 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:29.022 7f4898cdb700 1 -- 172.21.2.102:0/1977417902 wait complete. 2021-02-10T11:21:29.029 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000344433 2021-02-10T11:21:29.463 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:29.451 7f9ce3c27700 1 Processor -- start 2021-02-10T11:21:29.463 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:29.451 7f9ce3c27700 1 -- start start 2021-02-10T11:21:29.463 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:29.452 7f9ce3c27700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f9cdc1b0470 0x7f9cdc1b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:29.463 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:29.452 7f9ce3c27700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f9cdc0ef610 0x7f9cdc0efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:29.463 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:29.452 7f9ce3c27700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9cdc0f0b50 con 0x7f9cdc0f0030 2021-02-10T11:21:29.463 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:29.452 7f9ce3c27700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9cdc0e52e0 con 0x7f9cdc0ef610 2021-02-10T11:21:29.463 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:29.452 7f9ce3c27700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9cdc0e5490 con 0x7f9cdc1b0470 2021-02-10T11:21:29.463 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:29.452 7f9ce1242700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f9cdc0ef610 0x7f9cdc0efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:29.464 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:29.452 7f9ce1242700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f9cdc0ef610 0x7f9cdc0efa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:54640/0 (socket says 172.21.2.102:54640) 2021-02-10T11:21:29.464 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:29.452 7f9ce2244700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f9cdc0f0030 0x7f9cdc0f10d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:46864/0 (socket says 172.21.2.102:46864) 2021-02-10T11:21:29.464 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:29.452 7f9ce1242700 1 -- 172.21.2.102:0/1298563996 learned_addr learned my addr 172.21.2.102:0/1298563996 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:21:29.464 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:29.453 7f9ce1a43700 1 --2- 172.21.2.102:0/1298563996 >> v2:172.21.2.113:3300/0 conn(0x7f9cdc1b0470 0x7f9cdc1b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:29.464 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:29.453 7f9ce1242700 1 -- 172.21.2.102:0/1298563996 >> v2:172.21.2.113:3300/0 conn(0x7f9cdc1b0470 msgr2=0x7f9cdc1b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:21:29.464 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:29.453 7f9ce1242700 1 --2- 172.21.2.102:0/1298563996 >> v2:172.21.2.113:3300/0 conn(0x7f9cdc1b0470 0x7f9cdc1b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:29.464 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:29.453 7f9ce1242700 1 -- 172.21.2.102:0/1298563996 >> v1:172.21.2.102:6789/0 conn(0x7f9cdc0f0030 legacy=0x7f9cdc0f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:29.464 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:29.453 7f9ce1242700 1 -- 172.21.2.102:0/1298563996 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cdc0f0430 con 0x7f9cdc0ef610 2021-02-10T11:21:29.464 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:29.453 7f9ce1a43700 1 --2- 172.21.2.102:0/1298563996 >> v2:172.21.2.113:3300/0 conn(0x7f9cdc1b0470 0x7f9cdc1b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:21:29.464 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:29.453 7f9ce1242700 1 --2- 172.21.2.102:0/1298563996 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9cdc0ef610 0x7f9cdc0efa10 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 rx=0x7f9cd8006700 tx=0x7f9cd8009b80).ready entity=mon.0 client_cookie=958181acfccd67e8 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:29.464 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:29.453 7f9ce0a41700 1 -- 172.21.2.102:0/1298563996 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9cd800c0c0 con 0x7f9cdc0ef610 2021-02-10T11:21:29.464 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:29.453 7f9ce0a41700 1 -- 172.21.2.102:0/1298563996 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cd800c440 con 0x7f9cdc0ef610 2021-02-10T11:21:29.465 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:29.453 7f9ce0a41700 1 -- 172.21.2.102:0/1298563996 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9cd800c8a0 con 0x7f9cdc0ef610 2021-02-10T11:21:29.465 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:29.453 7f9ce3c27700 1 -- 172.21.2.102:0/1298563996 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9cdc0ef610 msgr2=0x7f9cdc0efa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:29.465 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:29.453 7f9ce3c27700 1 --2- 172.21.2.102:0/1298563996 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9cdc0ef610 0x7f9cdc0efa10 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 rx=0x7f9cd8006700 tx=0x7f9cd8009b80).stop 2021-02-10T11:21:29.465 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:29.454 7f9ce3c27700 1 -- 172.21.2.102:0/1298563996 shutdown_connections 2021-02-10T11:21:29.465 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:29.454 7f9ce3c27700 1 --2- 172.21.2.102:0/1298563996 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9cdc0ef610 0x7f9cdc0efa10 unknown :-1 s=CLOSED pgs=933 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:29.465 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:29.454 7f9ce3c27700 1 --2- 172.21.2.102:0/1298563996 >> v2:172.21.2.113:3300/0 conn(0x7f9cdc1b0470 0x7f9cdc1b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:29.465 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:29.454 7f9ce3c27700 1 -- 172.21.2.102:0/1298563996 >> 172.21.2.102:0/1298563996 conn(0x7f9cdc0ce3e0 msgr2=0x7f9cdc0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:29.465 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:29.454 7f9ce3c27700 1 -- 172.21.2.102:0/1298563996 shutdown_connections 2021-02-10T11:21:29.465 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:29.454 7f9ce3c27700 1 -- 172.21.2.102:0/1298563996 wait complete. 2021-02-10T11:21:29.472 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:29.468 7f9ce3c27700 1 Processor -- start 2021-02-10T11:21:29.472 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:29.468 7f9ce3c27700 1 -- start start 2021-02-10T11:21:29.472 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:29.469 7f9ce3c27700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9cdc0a0420 0x7f9cdc0a0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:29.472 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:29.469 7f9ce3c27700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f9cdc189d00 0x7f9cdc18a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:29.472 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:29.469 7f9ce3c27700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9cdc18a720 0x7f9cdc0b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:29.472 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:29.469 7f9ce3c27700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cdc189780 con 0x7f9cdc0a0420 2021-02-10T11:21:29.473 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:29.469 7f9ce3c27700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cdc18ab20 con 0x7f9cdc189d00 2021-02-10T11:21:29.473 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:29.469 7f9ce3c27700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9cdc0a1140 con 0x7f9cdc18a720 2021-02-10T11:21:29.474 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:29.469 7f9ce1a43700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9cdc0a0420 0x7f9cdc0a0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:29.474 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:29.469 7f9ce1a43700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9cdc0a0420 0x7f9cdc0a0820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:54644/0 (socket says 172.21.2.102:54644) 2021-02-10T11:21:29.474 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:29.469 7f9ce1a43700 1 -- 172.21.2.102:0/69571214 learned_addr learned my addr 172.21.2.102:0/69571214 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:21:29.474 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:29.469 7f9ce2244700 1 --2- 172.21.2.102:0/69571214 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9cdc18a720 0x7f9cdc0b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:29.474 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:29.469 7f9ce1242700 1 --2- 172.21.2.102:0/69571214 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f9cdc189d00 0x7f9cdc18a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:29.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:29.469 7f9ce1a43700 1 -- 172.21.2.102:0/69571214 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9cdc18a720 msgr2=0x7f9cdc0b7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:29.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:29.469 7f9ce1a43700 1 --2- 172.21.2.102:0/69571214 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9cdc18a720 0x7f9cdc0b7be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:29.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:29.469 7f9ce1a43700 1 -- 172.21.2.102:0/69571214 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f9cdc189d00 msgr2=0x7f9cdc18a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:29.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:29.469 7f9ce1a43700 1 --2- 172.21.2.102:0/69571214 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f9cdc189d00 0x7f9cdc18a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:29.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:29.469 7f9ce1a43700 1 -- 172.21.2.102:0/69571214 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cdc0b8200 con 0x7f9cdc0a0420 2021-02-10T11:21:29.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:29.470 7f9ce1a43700 1 --2- 172.21.2.102:0/69571214 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9cdc0a0420 0x7f9cdc0a0820 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 rx=0x7f9ccc006f20 tx=0x7f9ccc00b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:29.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:29.470 7f9cd27fc700 1 -- 172.21.2.102:0/69571214 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9ccc00bd80 con 0x7f9cdc0a0420 2021-02-10T11:21:29.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:29.470 7f9cd27fc700 1 -- 172.21.2.102:0/69571214 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ccc000c90 con 0x7f9cdc0a0420 2021-02-10T11:21:29.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:29.470 7f9cd27fc700 1 -- 172.21.2.102:0/69571214 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9ccc002220 con 0x7f9cdc0a0420 2021-02-10T11:21:29.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:29.470 7f9ce3c27700 1 -- 172.21.2.102:0/69571214 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9cdc0b8200 con 0x7f9cdc0a0420 2021-02-10T11:21:29.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:29.470 7f9ce3c27700 1 -- 172.21.2.102:0/69571214 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9cdc0b8200 con 0x7f9cdc0a0420 2021-02-10T11:21:29.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:29.470 7f9cd27fc700 1 -- 172.21.2.102:0/69571214 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f9ccc00a000 con 0x7f9cdc0a0420 2021-02-10T11:21:29.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:29.470 7f9cd27fc700 1 --2- 172.21.2.102:0/69571214 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f9cc4018730 0x7f9cc401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:29.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:29.470 7f9cd27fc700 1 -- 172.21.2.102:0/69571214 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f9ccc032630 con 0x7f9cdc0a0420 2021-02-10T11:21:29.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:29.470 7f9ce1242700 1 --2- 172.21.2.102:0/69571214 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f9cc4018730 0x7f9cc401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:29.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:29.471 7f9ce1242700 1 --2- 172.21.2.102:0/69571214 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f9cc4018730 0x7f9cc401abc0 crc :-1 s=READY pgs=456 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:29.494 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:29.500 7f9ce3c27700 1 -- 172.21.2.102:0/69571214 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9cdc105ae0 con 0x7f9cdc0a0420 2021-02-10T11:21:29.497 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:29.502 7f9cd27fc700 1 -- 172.21.2.102:0/69571214 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f9ccc006bb0 con 0x7f9cdc0a0420 2021-02-10T11:21:29.624 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:29.629 7f9ce3c27700 1 -- 172.21.2.102:0/69571214 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344433", "target": ["mgr", ""]}) v1 -- 0x7f9cdc105ae0 con 0x7f9cc4018730 2021-02-10T11:21:29.635 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:29.641 7f9cd27fc700 1 -- 172.21.2.102:0/69571214 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f9cd8006430 con 0x7f9cc4018730 2021-02-10T11:21:29.641 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:29.646 7f9ce3c27700 1 -- 172.21.2.102:0/69571214 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f9cc4018730 msgr2=0x7f9cc401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:29.641 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:29.646 7f9ce3c27700 1 --2- 172.21.2.102:0/69571214 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f9cc4018730 0x7f9cc401abc0 crc :-1 s=READY pgs=456 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:29.641 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:29.646 7f9ce3c27700 1 -- 172.21.2.102:0/69571214 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9cdc0a0420 msgr2=0x7f9cdc0a0820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:29.641 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:29.646 7f9ce3c27700 1 --2- 172.21.2.102:0/69571214 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9cdc0a0420 0x7f9cdc0a0820 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 rx=0x7f9ccc006f20 tx=0x7f9ccc00b990).stop 2021-02-10T11:21:29.641 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:29.646 7f9ce3c27700 1 -- 172.21.2.102:0/69571214 shutdown_connections 2021-02-10T11:21:29.642 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:29.646 7f9ce3c27700 1 --2- 172.21.2.102:0/69571214 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9cdc18a720 0x7f9cdc0b7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:29.642 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:29.646 7f9ce3c27700 1 --2- 172.21.2.102:0/69571214 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f9cc4018730 0x7f9cc401abc0 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:29.642 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:29.646 7f9ce3c27700 1 --2- 172.21.2.102:0/69571214 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f9cdc189d00 0x7f9cdc18a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:29.642 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:29.646 7f9ce3c27700 1 --2- 172.21.2.102:0/69571214 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9cdc0a0420 0x7f9cdc0a0820 unknown :-1 s=CLOSED pgs=934 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:29.642 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:29.646 7f9ce3c27700 1 -- 172.21.2.102:0/69571214 >> 172.21.2.102:0/69571214 conn(0x7f9cdc094e10 msgr2=0x7f9cdc0a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:29.642 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:29.646 7f9ce3c27700 1 -- 172.21.2.102:0/69571214 shutdown_connections 2021-02-10T11:21:29.642 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:29.646 7f9ce3c27700 1 -- 172.21.2.102:0/69571214 wait complete. 2021-02-10T11:21:29.650 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000344434 2021-02-10T11:21:30.089 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:30.077 7fc5d6c65700 1 Processor -- start 2021-02-10T11:21:30.089 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:30.078 7fc5d6c65700 1 -- start start 2021-02-10T11:21:30.089 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:30.078 7fc5d6c65700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fc5d00f05f0 0x7fc5d00f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:30.089 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:30.078 7fc5d6c65700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5d01b0c70 con 0x7fc5d01b0870 2021-02-10T11:21:30.090 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:30.078 7fc5d6c65700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5d00f0b50 con 0x7fc5d01b0470 2021-02-10T11:21:30.090 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:30.078 7fc5d6c65700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5d00e5ea0 con 0x7fc5d00f05f0 2021-02-10T11:21:30.090 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:30.078 7fc5cffff700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fc5d01b0870 0x7fc5d00efe00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:46876/0 (socket says 172.21.2.102:46876) 2021-02-10T11:21:30.090 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:30.078 7fc5cffff700 1 -- 172.21.2.102:0/50292558 learned_addr learned my addr 172.21.2.102:0/50292558 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:21:30.090 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:30.078 7fc5d5282700 1 --2- 172.21.2.102:0/50292558 >> v2:172.21.2.113:3300/0 conn(0x7fc5d00f05f0 0x7fc5d00f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:30.090 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:30.079 7fc5cf7fe700 1 -- 172.21.2.102:0/50292558 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 29111649 0 0) 0x7fc5c80009b0 con 0x7fc5d01b0870 2021-02-10T11:21:30.090 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:30.079 7fc5cf7fe700 1 -- 172.21.2.102:0/50292558 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5bc002fe0 con 0x7fc5d01b0870 2021-02-10T11:21:30.090 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:30.079 7fc5cf7fe700 1 -- 172.21.2.102:0/50292558 <== mon.1 v1:172.21.2.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1558076574 0 0) 0x7fc5b8001080 con 0x7fc5d01b0470 2021-02-10T11:21:30.090 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:30.079 7fc5cf7fe700 1 -- 172.21.2.102:0/50292558 --> v1:172.21.2.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5bc003020 con 0x7fc5d01b0470 2021-02-10T11:21:30.090 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:30.079 7fc5d5282700 1 -- 172.21.2.102:0/50292558 >> v1:172.21.2.113:6789/0 conn(0x7fc5d01b0470 legacy=0x7fc5d00ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:30.090 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:30.079 7fc5d5282700 1 -- 172.21.2.102:0/50292558 >> v1:172.21.2.102:6789/0 conn(0x7fc5d01b0870 legacy=0x7fc5d00efe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:30.090 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:30.079 7fc5d5282700 1 -- 172.21.2.102:0/50292558 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5d00e52e0 con 0x7fc5d00f05f0 2021-02-10T11:21:30.091 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:30.079 7fc5d5282700 1 --2- 172.21.2.102:0/50292558 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc5d00f05f0 0x7fc5d00f20e0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 rx=0x7fc5c0006e50 tx=0x7fc5c0009df0).ready entity=mon.1 client_cookie=b6b933c417090b46 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:30.091 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:30.079 7fc5cf7fe700 1 -- 172.21.2.102:0/50292558 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc5c000b510 con 0x7fc5d00f05f0 2021-02-10T11:21:30.091 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:30.079 7fc5cf7fe700 1 -- 172.21.2.102:0/50292558 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5c0006790 con 0x7fc5d00f05f0 2021-02-10T11:21:30.091 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:30.079 7fc5cf7fe700 1 -- 172.21.2.102:0/50292558 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc5c0000f90 con 0x7fc5d00f05f0 2021-02-10T11:21:30.091 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:30.080 7fc5d6c65700 1 -- 172.21.2.102:0/50292558 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc5d00f05f0 msgr2=0x7fc5d00f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:30.091 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:30.080 7fc5d6c65700 1 --2- 172.21.2.102:0/50292558 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc5d00f05f0 0x7fc5d00f20e0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 rx=0x7fc5c0006e50 tx=0x7fc5c0009df0).stop 2021-02-10T11:21:30.091 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:30.080 7fc5d6c65700 1 -- 172.21.2.102:0/50292558 shutdown_connections 2021-02-10T11:21:30.091 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:30.080 7fc5d6c65700 1 --2- 172.21.2.102:0/50292558 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc5d00f05f0 0x7fc5d00f20e0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:30.091 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:30.080 7fc5d6c65700 1 -- 172.21.2.102:0/50292558 >> 172.21.2.102:0/50292558 conn(0x7fc5d00ce3e0 msgr2=0x7fc5d00e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:30.091 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:30.080 7fc5d6c65700 1 -- 172.21.2.102:0/50292558 shutdown_connections 2021-02-10T11:21:30.091 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:30.080 7fc5d6c65700 1 -- 172.21.2.102:0/50292558 wait complete. 2021-02-10T11:21:30.093 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:30.095 7fc5d6c65700 1 Processor -- start 2021-02-10T11:21:30.093 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:30.095 7fc5d6c65700 1 -- start start 2021-02-10T11:21:30.093 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:30.095 7fc5d6c65700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc5d0106fd0 0x7fc5d01a7500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:30.093 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:30.095 7fc5d6c65700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc5d01a7b20 0x7fc5d00c25e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:30.093 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:30.095 7fc5d6c65700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc5d01a7f20 0x7fc5d00c2c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:30.093 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:30.095 7fc5d6c65700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5d008c920 con 0x7fc5d01a7b20 2021-02-10T11:21:30.093 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:30.095 7fc5d6c65700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5d008c680 con 0x7fc5d0106fd0 2021-02-10T11:21:30.093 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:30.095 7fc5d6c65700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5d0107810 con 0x7fc5d01a7f20 2021-02-10T11:21:30.093 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:30.096 7fc5cffff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc5d01a7b20 0x7fc5d00c25e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:30.093 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:30.096 7fc5d4a81700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc5d0106fd0 0x7fc5d01a7500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:30.093 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:30.096 7fc5cffff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc5d01a7b20 0x7fc5d00c25e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:54658/0 (socket says 172.21.2.102:54658) 2021-02-10T11:21:30.093 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:30.096 7fc5cffff700 1 -- 172.21.2.102:0/1436093634 learned_addr learned my addr 172.21.2.102:0/1436093634 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:21:30.093 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:30.096 7fc5d5282700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc5d01a7f20 0x7fc5d00c2c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:30.094 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:30.096 7fc5cffff700 1 -- 172.21.2.102:0/1436093634 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc5d01a7f20 msgr2=0x7fc5d00c2c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:30.094 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:30.096 7fc5cffff700 1 --2- 172.21.2.102:0/1436093634 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc5d01a7f20 0x7fc5d00c2c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:30.094 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:30.096 7fc5cffff700 1 -- 172.21.2.102:0/1436093634 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc5d0106fd0 msgr2=0x7fc5d01a7500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:30.094 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:30.096 7fc5cffff700 1 --2- 172.21.2.102:0/1436093634 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc5d0106fd0 0x7fc5d01a7500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:30.094 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:30.096 7fc5cffff700 1 -- 172.21.2.102:0/1436093634 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5d01073d0 con 0x7fc5d01a7b20 2021-02-10T11:21:30.094 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:30.096 7fc5cffff700 1 --2- 172.21.2.102:0/1436093634 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc5d01a7b20 0x7fc5d00c25e0 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 rx=0x7fc5c8001c60 tx=0x7fc5c800a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:30.094 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:30.097 7fc5cd7fa700 1 -- 172.21.2.102:0/1436093634 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc5c800acc0 con 0x7fc5d01a7b20 2021-02-10T11:21:30.094 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:30.097 7fc5d6c65700 1 -- 172.21.2.102:0/1436093634 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5d01073d0 con 0x7fc5d01a7b20 2021-02-10T11:21:30.094 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:30.097 7fc5d6c65700 1 -- 172.21.2.102:0/1436093634 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5d01073d0 con 0x7fc5d01a7b20 2021-02-10T11:21:30.094 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:30.097 7fc5cd7fa700 1 -- 172.21.2.102:0/1436093634 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5c800acc0 con 0x7fc5d01a7b20 2021-02-10T11:21:30.094 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:30.097 7fc5cd7fa700 1 -- 172.21.2.102:0/1436093634 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc5c8004140 con 0x7fc5d01a7b20 2021-02-10T11:21:30.094 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:30.097 7fc5cd7fa700 1 -- 172.21.2.102:0/1436093634 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fc5c8004140 con 0x7fc5d01a7b20 2021-02-10T11:21:30.095 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:30.097 7fc5cd7fa700 1 --2- 172.21.2.102:0/1436093634 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc5b40186c0 0x7fc5b401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:30.095 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:30.097 7fc5cd7fa700 1 -- 172.21.2.102:0/1436093634 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fc5c80327b0 con 0x7fc5d01a7b20 2021-02-10T11:21:30.095 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:30.097 7fc5d4a81700 1 --2- 172.21.2.102:0/1436093634 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc5b40186c0 0x7fc5b401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:30.095 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:30.098 7fc5d4a81700 1 --2- 172.21.2.102:0/1436093634 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc5b40186c0 0x7fc5b401abc0 crc :-1 s=READY pgs=457 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:30.122 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:30.127 7fc5d6c65700 1 -- 172.21.2.102:0/1436093634 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5d008c680 con 0x7fc5d01a7b20 2021-02-10T11:21:30.124 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:30.130 7fc5cd7fa700 1 -- 172.21.2.102:0/1436093634 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fc5c80009b0 con 0x7fc5d01a7b20 2021-02-10T11:21:30.252 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:30.257 7fc5d6c65700 1 -- 172.21.2.102:0/1436093634 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344434", "target": ["mgr", ""]}) v1 -- 0x7fc5d008c920 con 0x7fc5b40186c0 2021-02-10T11:21:30.263 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:30.268 7fc5cd7fa700 1 -- 172.21.2.102:0/1436093634 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fc5b8006550 con 0x7fc5b40186c0 2021-02-10T11:21:30.270 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:30.275 7fc5d6c65700 1 -- 172.21.2.102:0/1436093634 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc5b40186c0 msgr2=0x7fc5b401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:30.270 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:30.275 7fc5d6c65700 1 --2- 172.21.2.102:0/1436093634 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc5b40186c0 0x7fc5b401abc0 crc :-1 s=READY pgs=457 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:30.270 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:30.275 7fc5d6c65700 1 -- 172.21.2.102:0/1436093634 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc5d01a7b20 msgr2=0x7fc5d00c25e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:30.270 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:30.275 7fc5d6c65700 1 --2- 172.21.2.102:0/1436093634 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc5d01a7b20 0x7fc5d00c25e0 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 rx=0x7fc5c8001c60 tx=0x7fc5c800a8d0).stop 2021-02-10T11:21:30.270 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:30.275 7fc5d6c65700 1 -- 172.21.2.102:0/1436093634 shutdown_connections 2021-02-10T11:21:30.270 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:30.275 7fc5d6c65700 1 --2- 172.21.2.102:0/1436093634 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc5d01a7f20 0x7fc5d00c2c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:30.270 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:30.275 7fc5d6c65700 1 --2- 172.21.2.102:0/1436093634 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc5d01a7b20 0x7fc5d00c25e0 unknown :-1 s=CLOSED pgs=936 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:30.270 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:30.275 7fc5d6c65700 1 --2- 172.21.2.102:0/1436093634 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc5b40186c0 0x7fc5b401abc0 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:30.270 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:30.275 7fc5d6c65700 1 --2- 172.21.2.102:0/1436093634 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc5d0106fd0 0x7fc5d01a7500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:30.271 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:30.275 7fc5d6c65700 1 -- 172.21.2.102:0/1436093634 >> 172.21.2.102:0/1436093634 conn(0x7fc5d00bbf60 msgr2=0x7fc5d0105d10 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:30.271 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:30.275 7fc5d6c65700 1 -- 172.21.2.102:0/1436093634 shutdown_connections 2021-02-10T11:21:30.271 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:30.275 7fc5d6c65700 1 -- 172.21.2.102:0/1436093634 wait complete. 2021-02-10T11:21:30.278 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000344435 2021-02-10T11:21:30.714 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:30.703 7fb5e7759700 1 Processor -- start 2021-02-10T11:21:30.715 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:30.703 7fb5e7759700 1 -- start start 2021-02-10T11:21:30.715 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:30.703 7fb5e7759700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fb5e01b0470 0x7fb5e01b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:30.715 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:30.703 7fb5e7759700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb5e00ef610 0x7fb5e00efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:30.715 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:30.703 7fb5e7759700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5e00f0b50 con 0x7fb5e00f0030 2021-02-10T11:21:30.715 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:30.703 7fb5e7759700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5e00e52e0 con 0x7fb5e01b0470 2021-02-10T11:21:30.715 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:30.703 7fb5e7759700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5e00e54d0 con 0x7fb5e00ef610 2021-02-10T11:21:30.715 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:30.704 7fb5e5575700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fb5e01b0470 0x7fb5e01b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:30.715 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:30.704 7fb5e4d74700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb5e00ef610 0x7fb5e00efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:30.715 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:30.704 7fb5e5d76700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fb5e00f0030 0x7fb5e00f10d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:46888/0 (socket says 172.21.2.102:46888) 2021-02-10T11:21:30.715 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:30.704 7fb5e5d76700 1 -- 172.21.2.102:0/1386040536 learned_addr learned my addr 172.21.2.102:0/1386040536 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:21:30.715 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:30.704 7fb5e4d74700 1 -- 172.21.2.102:0/1386040536 >> v2:172.21.2.102:3300/0 conn(0x7fb5e01b0470 msgr2=0x7fb5e01b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:21:30.715 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:30.704 7fb5e4d74700 1 --2- 172.21.2.102:0/1386040536 >> v2:172.21.2.102:3300/0 conn(0x7fb5e01b0470 0x7fb5e01b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:30.716 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:30.704 7fb5e4d74700 1 -- 172.21.2.102:0/1386040536 >> v1:172.21.2.102:6789/0 conn(0x7fb5e00f0030 legacy=0x7fb5e00f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:30.716 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:30.704 7fb5e4d74700 1 -- 172.21.2.102:0/1386040536 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5e00f0430 con 0x7fb5e00ef610 2021-02-10T11:21:30.716 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:30.704 7fb5e5575700 1 --2- 172.21.2.102:0/1386040536 >> v2:172.21.2.102:3300/0 conn(0x7fb5e01b0470 0x7fb5e01b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:21:30.716 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:30.704 7fb5e4d74700 1 --2- 172.21.2.102:0/1386040536 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb5e00ef610 0x7fb5e00efa10 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7fb5d0006650 tx=0x7fb5d0009a80).ready entity=mon.2 client_cookie=47c1eb21f961ae08 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:30.716 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:30.704 7fb5d7fff700 1 -- 172.21.2.102:0/1386040536 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb5d00009b0 con 0x7fb5e00ef610 2021-02-10T11:21:30.716 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:30.704 7fb5d7fff700 1 -- 172.21.2.102:0/1386040536 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5d000c0c0 con 0x7fb5e00ef610 2021-02-10T11:21:30.716 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:30.704 7fb5d7fff700 1 -- 172.21.2.102:0/1386040536 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb5d000c510 con 0x7fb5e00ef610 2021-02-10T11:21:30.716 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:30.705 7fb5e7759700 1 -- 172.21.2.102:0/1386040536 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb5e00ef610 msgr2=0x7fb5e00efa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:30.716 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:30.705 7fb5e7759700 1 --2- 172.21.2.102:0/1386040536 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb5e00ef610 0x7fb5e00efa10 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7fb5d0006650 tx=0x7fb5d0009a80).stop 2021-02-10T11:21:30.716 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:30.705 7fb5e7759700 1 -- 172.21.2.102:0/1386040536 shutdown_connections 2021-02-10T11:21:30.716 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:30.705 7fb5e7759700 1 --2- 172.21.2.102:0/1386040536 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb5e00ef610 0x7fb5e00efa10 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:30.716 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:30.705 7fb5e7759700 1 --2- 172.21.2.102:0/1386040536 >> v2:172.21.2.102:3300/0 conn(0x7fb5e01b0470 0x7fb5e01b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:30.716 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:30.705 7fb5e7759700 1 -- 172.21.2.102:0/1386040536 >> 172.21.2.102:0/1386040536 conn(0x7fb5e00ce3e0 msgr2=0x7fb5e00e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:30.717 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:30.705 7fb5e7759700 1 -- 172.21.2.102:0/1386040536 shutdown_connections 2021-02-10T11:21:30.717 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:30.705 7fb5e7759700 1 -- 172.21.2.102:0/1386040536 wait complete. 2021-02-10T11:21:30.718 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:30.720 7fb5e7759700 1 Processor -- start 2021-02-10T11:21:30.718 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:30.720 7fb5e7759700 1 -- start start 2021-02-10T11:21:30.718 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:30.721 7fb5e7759700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb5e00a0420 0x7fb5e00a0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:30.718 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:30.721 7fb5e7759700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb5e0189d00 0x7fb5e018a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:30.718 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:30.721 7fb5e7759700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb5e018a720 0x7fb5e00b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:30.718 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:30.721 7fb5e7759700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5e0189780 con 0x7fb5e0189d00 2021-02-10T11:21:30.718 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:30.721 7fb5e7759700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5e018ab20 con 0x7fb5e00a0420 2021-02-10T11:21:30.718 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:30.721 7fb5e7759700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5e00a1140 con 0x7fb5e018a720 2021-02-10T11:21:30.719 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:30.721 7fb5e4d74700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb5e0189d00 0x7fb5e018a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:30.719 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:30.721 7fb5e4d74700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb5e0189d00 0x7fb5e018a100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:54672/0 (socket says 172.21.2.102:54672) 2021-02-10T11:21:30.719 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:30.721 7fb5e4d74700 1 -- 172.21.2.102:0/2382415914 learned_addr learned my addr 172.21.2.102:0/2382415914 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:21:30.719 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:30.721 7fb5e5575700 1 --2- 172.21.2.102:0/2382415914 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb5e00a0420 0x7fb5e00a0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:30.719 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:30.721 7fb5e5d76700 1 --2- 172.21.2.102:0/2382415914 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb5e018a720 0x7fb5e00b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:30.719 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:30.721 7fb5e4d74700 1 -- 172.21.2.102:0/2382415914 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb5e018a720 msgr2=0x7fb5e00b7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:30.719 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:30.721 7fb5e4d74700 1 --2- 172.21.2.102:0/2382415914 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb5e018a720 0x7fb5e00b7be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:30.719 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:30.721 7fb5e4d74700 1 -- 172.21.2.102:0/2382415914 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb5e00a0420 msgr2=0x7fb5e00a0820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:30.719 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:30.721 7fb5e4d74700 1 --2- 172.21.2.102:0/2382415914 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb5e00a0420 0x7fb5e00a0820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:30.719 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:30.721 7fb5e4d74700 1 -- 172.21.2.102:0/2382415914 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5e00b8200 con 0x7fb5e0189d00 2021-02-10T11:21:30.719 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:30.721 7fb5e5575700 1 --2- 172.21.2.102:0/2382415914 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb5e00a0420 0x7fb5e00a0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:21:30.719 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:30.722 7fb5e4d74700 1 --2- 172.21.2.102:0/2382415914 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb5e0189d00 0x7fb5e018a100 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 rx=0x7fb5d0009650 tx=0x7fb5d000b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:30.719 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:30.722 7fb5d5ffb700 1 -- 172.21.2.102:0/2382415914 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb5d00093a0 con 0x7fb5e0189d00 2021-02-10T11:21:30.719 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:30.722 7fb5d5ffb700 1 -- 172.21.2.102:0/2382415914 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5d000bd50 con 0x7fb5e0189d00 2021-02-10T11:21:30.720 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:30.722 7fb5d5ffb700 1 -- 172.21.2.102:0/2382415914 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb5d00052e0 con 0x7fb5e0189d00 2021-02-10T11:21:30.720 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:30.722 7fb5e7759700 1 -- 172.21.2.102:0/2382415914 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5e00b8200 con 0x7fb5e0189d00 2021-02-10T11:21:30.720 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:30.722 7fb5e7759700 1 -- 172.21.2.102:0/2382415914 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5e00b8200 con 0x7fb5e0189d00 2021-02-10T11:21:30.720 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:30.722 7fb5d5ffb700 1 -- 172.21.2.102:0/2382415914 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fb5d0006030 con 0x7fb5e0189d00 2021-02-10T11:21:30.720 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:30.722 7fb5d5ffb700 1 --2- 172.21.2.102:0/2382415914 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb5c0018730 0x7fb5c001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:30.720 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:30.722 7fb5d5ffb700 1 -- 172.21.2.102:0/2382415914 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fb5d0032630 con 0x7fb5e0189d00 2021-02-10T11:21:30.720 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:30.722 7fb5e5575700 1 --2- 172.21.2.102:0/2382415914 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb5c0018730 0x7fb5c001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:30.720 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:30.723 7fb5e5575700 1 --2- 172.21.2.102:0/2382415914 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb5c0018730 0x7fb5c001ac10 crc :-1 s=READY pgs=458 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:30.746 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:30.751 7fb5e7759700 1 -- 172.21.2.102:0/2382415914 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5e0105ae0 con 0x7fb5e0189d00 2021-02-10T11:21:30.748 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:30.754 7fb5d5ffb700 1 -- 172.21.2.102:0/2382415914 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fb5d0000940 con 0x7fb5e0189d00 2021-02-10T11:21:30.880 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:30.884 7fb5e7759700 1 -- 172.21.2.102:0/2382415914 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344435", "target": ["mgr", ""]}) v1 -- 0x7fb5e00f0b50 con 0x7fb5c0018730 2021-02-10T11:21:30.891 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:30.895 7fb5d5ffb700 1 -- 172.21.2.102:0/2382415914 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fb5dc0009b0 con 0x7fb5c0018730 2021-02-10T11:21:30.895 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:30.900 7fb5e7759700 1 -- 172.21.2.102:0/2382415914 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb5c0018730 msgr2=0x7fb5c001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:30.896 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:30.900 7fb5e7759700 1 --2- 172.21.2.102:0/2382415914 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb5c0018730 0x7fb5c001ac10 crc :-1 s=READY pgs=458 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:30.896 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:30.900 7fb5e7759700 1 -- 172.21.2.102:0/2382415914 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb5e0189d00 msgr2=0x7fb5e018a100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:30.896 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:30.900 7fb5e7759700 1 --2- 172.21.2.102:0/2382415914 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb5e0189d00 0x7fb5e018a100 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 rx=0x7fb5d0009650 tx=0x7fb5d000b960).stop 2021-02-10T11:21:30.896 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:30.901 7fb5e7759700 1 -- 172.21.2.102:0/2382415914 shutdown_connections 2021-02-10T11:21:30.896 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:30.901 7fb5e7759700 1 --2- 172.21.2.102:0/2382415914 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb5e018a720 0x7fb5e00b7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:30.896 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:30.901 7fb5e7759700 1 --2- 172.21.2.102:0/2382415914 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb5e0189d00 0x7fb5e018a100 unknown :-1 s=CLOSED pgs=938 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:30.896 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:30.901 7fb5e7759700 1 --2- 172.21.2.102:0/2382415914 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb5c0018730 0x7fb5c001ac10 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:30.896 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:30.901 7fb5e7759700 1 --2- 172.21.2.102:0/2382415914 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb5e00a0420 0x7fb5e00a0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:30.896 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:30.901 7fb5e7759700 1 -- 172.21.2.102:0/2382415914 >> 172.21.2.102:0/2382415914 conn(0x7fb5e0094e10 msgr2=0x7fb5e00a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:30.896 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:30.901 7fb5e7759700 1 -- 172.21.2.102:0/2382415914 shutdown_connections 2021-02-10T11:21:30.896 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:30.901 7fb5e7759700 1 -- 172.21.2.102:0/2382415914 wait complete. 2021-02-10T11:21:30.904 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000344436 2021-02-10T11:21:31.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:31.325 7fe6376c8700 1 Processor -- start 2021-02-10T11:21:31.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:31.325 7fe6376c8700 1 -- start start 2021-02-10T11:21:31.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:31.326 7fe6376c8700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fe6301b0470 0x7fe6301b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:31.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:31.326 7fe6376c8700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fe6300ef610 0x7fe6300efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:31.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:31.326 7fe6376c8700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe6300f0030 0x7fe6300f0430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:31.338 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:31.326 7fe6376c8700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6300e5ea0 con 0x7fe6300ef610 2021-02-10T11:21:31.338 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:31.326 7fe6376c8700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6300f0b50 con 0x7fe6301b0470 2021-02-10T11:21:31.338 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:31.326 7fe6376c8700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6300e52e0 con 0x7fe6300f0030 2021-02-10T11:21:31.338 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:31.326 7fe634ce3700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fe6300ef610 0x7fe6300efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:31.338 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:31.326 7fe634ce3700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fe6300ef610 0x7fe6300efa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:54678/0 (socket says 172.21.2.102:54678) 2021-02-10T11:21:31.338 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:31.326 7fe634ce3700 1 -- 172.21.2.102:0/2712799913 learned_addr learned my addr 172.21.2.102:0/2712799913 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:21:31.338 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:31.326 7fe6354e4700 1 --2- 172.21.2.102:0/2712799913 >> v2:172.21.2.113:3300/0 conn(0x7fe6301b0470 0x7fe6301b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:31.338 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:31.326 7fe635ce5700 1 --2- 172.21.2.102:0/2712799913 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe6300f0030 0x7fe6300f0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:31.338 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:31.326 7fe634ce3700 1 -- 172.21.2.102:0/2712799913 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe6300f0030 msgr2=0x7fe6300f0430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:21:31.338 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:31.326 7fe634ce3700 1 --2- 172.21.2.102:0/2712799913 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe6300f0030 0x7fe6300f0430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:31.338 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:31.326 7fe634ce3700 1 -- 172.21.2.102:0/2712799913 >> v2:172.21.2.113:3300/0 conn(0x7fe6301b0470 msgr2=0x7fe6301b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:21:31.338 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:31.326 7fe634ce3700 1 --2- 172.21.2.102:0/2712799913 >> v2:172.21.2.113:3300/0 conn(0x7fe6301b0470 0x7fe6301b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:31.339 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:31.326 7fe634ce3700 1 -- 172.21.2.102:0/2712799913 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6300f10d0 con 0x7fe6300ef610 2021-02-10T11:21:31.339 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:31.327 7fe634ce3700 1 --2- 172.21.2.102:0/2712799913 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe6300ef610 0x7fe6300efa10 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 rx=0x7fe628006a80 tx=0x7fe628009f00).ready entity=mon.0 client_cookie=c1e807bc490ac7bb server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:31.339 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:31.327 7fe627fff700 1 -- 172.21.2.102:0/2712799913 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe62800b5c0 con 0x7fe6300ef610 2021-02-10T11:21:31.339 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:31.327 7fe627fff700 1 -- 172.21.2.102:0/2712799913 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe62800b840 con 0x7fe6300ef610 2021-02-10T11:21:31.339 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:31.327 7fe627fff700 1 -- 172.21.2.102:0/2712799913 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe62800bd60 con 0x7fe6300ef610 2021-02-10T11:21:31.339 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:31.327 7fe6376c8700 1 -- 172.21.2.102:0/2712799913 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe6300ef610 msgr2=0x7fe6300efa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:31.339 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:31.327 7fe6376c8700 1 --2- 172.21.2.102:0/2712799913 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe6300ef610 0x7fe6300efa10 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 rx=0x7fe628006a80 tx=0x7fe628009f00).stop 2021-02-10T11:21:31.339 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:31.327 7fe6376c8700 1 -- 172.21.2.102:0/2712799913 shutdown_connections 2021-02-10T11:21:31.339 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:31.328 7fe6376c8700 1 --2- 172.21.2.102:0/2712799913 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe6300f0030 0x7fe6300f0430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:31.339 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:31.328 7fe6376c8700 1 --2- 172.21.2.102:0/2712799913 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe6300ef610 0x7fe6300efa10 unknown :-1 s=CLOSED pgs=939 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:31.339 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:31.328 7fe6376c8700 1 --2- 172.21.2.102:0/2712799913 >> v2:172.21.2.113:3300/0 conn(0x7fe6301b0470 0x7fe6301b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:31.340 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:31.328 7fe6376c8700 1 -- 172.21.2.102:0/2712799913 >> 172.21.2.102:0/2712799913 conn(0x7fe6300ce3e0 msgr2=0x7fe6300e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:31.340 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:31.328 7fe6376c8700 1 -- 172.21.2.102:0/2712799913 shutdown_connections 2021-02-10T11:21:31.340 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:31.328 7fe6376c8700 1 -- 172.21.2.102:0/2712799913 wait complete. 2021-02-10T11:21:31.340 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:31.342 7fe6376c8700 1 Processor -- start 2021-02-10T11:21:31.340 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:31.342 7fe6376c8700 1 -- start start 2021-02-10T11:21:31.341 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:31.343 7fe6376c8700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe6300924b0 0x7fe6301a3590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:31.341 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:31.343 7fe6376c8700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe6301a3bb0 0x7fe6300bf920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:31.341 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:31.343 7fe6376c8700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe6300bff40 0x7fe6300c0340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:31.342 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:31.343 7fe6376c8700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe63009a0a0 con 0x7fe6300924b0 2021-02-10T11:21:31.342 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:31.343 7fe6376c8700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6300c0960 con 0x7fe6301a3bb0 2021-02-10T11:21:31.342 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:31.343 7fe6376c8700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe630098420 con 0x7fe6300bff40 2021-02-10T11:21:31.342 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:31.343 7fe6354e4700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe6300924b0 0x7fe6301a3590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:31.342 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:31.343 7fe6354e4700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe6300924b0 0x7fe6301a3590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:54682/0 (socket says 172.21.2.102:54682) 2021-02-10T11:21:31.342 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:31.343 7fe6354e4700 1 -- 172.21.2.102:0/624062665 learned_addr learned my addr 172.21.2.102:0/624062665 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:21:31.342 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:31.343 7fe635ce5700 1 --2- 172.21.2.102:0/624062665 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe6300bff40 0x7fe6300c0340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:31.342 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:31.343 7fe634ce3700 1 --2- 172.21.2.102:0/624062665 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe6301a3bb0 0x7fe6300bf920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:31.342 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:31.343 7fe6354e4700 1 -- 172.21.2.102:0/624062665 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe6300bff40 msgr2=0x7fe6300c0340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:31.342 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:31.343 7fe6354e4700 1 --2- 172.21.2.102:0/624062665 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe6300bff40 0x7fe6300c0340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:31.342 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:31.343 7fe6354e4700 1 -- 172.21.2.102:0/624062665 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe6301a3bb0 msgr2=0x7fe6300bf920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:31.342 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:31.343 7fe6354e4700 1 --2- 172.21.2.102:0/624062665 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe6301a3bb0 0x7fe6300bf920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:31.343 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:31.343 7fe6354e4700 1 -- 172.21.2.102:0/624062665 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe630098640 con 0x7fe6300924b0 2021-02-10T11:21:31.343 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:31.344 7fe6354e4700 1 --2- 172.21.2.102:0/624062665 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe6300924b0 0x7fe6301a3590 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 rx=0x7fe620006f60 tx=0x7fe62000b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:31.343 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:31.344 7fe625ffb700 1 -- 172.21.2.102:0/624062665 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe6200009b0 con 0x7fe6300924b0 2021-02-10T11:21:31.343 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:31.344 7fe625ffb700 1 -- 172.21.2.102:0/624062665 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe62000bdf0 con 0x7fe6300924b0 2021-02-10T11:21:31.343 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:31.344 7fe625ffb700 1 -- 172.21.2.102:0/624062665 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe620002030 con 0x7fe6300924b0 2021-02-10T11:21:31.343 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:31.344 7fe6376c8700 1 -- 172.21.2.102:0/624062665 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe630098640 con 0x7fe6300924b0 2021-02-10T11:21:31.343 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:31.344 7fe6376c8700 1 -- 172.21.2.102:0/624062665 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe630098bd0 con 0x7fe6300924b0 2021-02-10T11:21:31.343 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:31.344 7fe625ffb700 1 -- 172.21.2.102:0/624062665 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fe620000c90 con 0x7fe6300924b0 2021-02-10T11:21:31.343 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:31.344 7fe625ffb700 1 --2- 172.21.2.102:0/624062665 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe6100186c0 0x7fe61001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:31.343 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:31.344 7fe625ffb700 1 -- 172.21.2.102:0/624062665 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fe620032400 con 0x7fe6300924b0 2021-02-10T11:21:31.343 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:31.345 7fe634ce3700 1 --2- 172.21.2.102:0/624062665 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe6100186c0 0x7fe61001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:31.343 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:31.345 7fe634ce3700 1 --2- 172.21.2.102:0/624062665 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe6100186c0 0x7fe61001abc0 crc :-1 s=READY pgs=459 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:31.370 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:31.376 7fe6376c8700 1 -- 172.21.2.102:0/624062665 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe630105b40 con 0x7fe6300924b0 2021-02-10T11:21:31.372 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:31.378 7fe625ffb700 1 -- 172.21.2.102:0/624062665 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fe620000960 con 0x7fe6300924b0 2021-02-10T11:21:31.500 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:31.505 7fe6376c8700 1 -- 172.21.2.102:0/624062665 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344436", "target": ["mgr", ""]}) v1 -- 0x7fe630098bd0 con 0x7fe6100186c0 2021-02-10T11:21:31.512 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:31.517 7fe625ffb700 1 -- 172.21.2.102:0/624062665 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fe628006900 con 0x7fe6100186c0 2021-02-10T11:21:31.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:31.522 7fe6376c8700 1 -- 172.21.2.102:0/624062665 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe6100186c0 msgr2=0x7fe61001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:31.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:31.522 7fe6376c8700 1 --2- 172.21.2.102:0/624062665 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe6100186c0 0x7fe61001abc0 crc :-1 s=READY pgs=459 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:31.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:31.522 7fe6376c8700 1 -- 172.21.2.102:0/624062665 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe6300924b0 msgr2=0x7fe6301a3590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:31.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:31.522 7fe6376c8700 1 --2- 172.21.2.102:0/624062665 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe6300924b0 0x7fe6301a3590 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 rx=0x7fe620006f60 tx=0x7fe62000b970).stop 2021-02-10T11:21:31.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:31.522 7fe6376c8700 1 -- 172.21.2.102:0/624062665 shutdown_connections 2021-02-10T11:21:31.518 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:31.522 7fe6376c8700 1 --2- 172.21.2.102:0/624062665 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe6300bff40 0x7fe6300c0340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:31.518 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:31.522 7fe6376c8700 1 --2- 172.21.2.102:0/624062665 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe6100186c0 0x7fe61001abc0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:31.518 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:31.522 7fe6376c8700 1 --2- 172.21.2.102:0/624062665 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe6301a3bb0 0x7fe6300bf920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:31.518 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:31.522 7fe6376c8700 1 --2- 172.21.2.102:0/624062665 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe6300924b0 0x7fe6301a3590 unknown :-1 s=CLOSED pgs=940 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:31.518 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:31.522 7fe6376c8700 1 -- 172.21.2.102:0/624062665 >> 172.21.2.102:0/624062665 conn(0x7fe6300b50c0 msgr2=0x7fe630092d90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:31.518 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:31.523 7fe6376c8700 1 -- 172.21.2.102:0/624062665 shutdown_connections 2021-02-10T11:21:31.518 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:31.523 7fe6376c8700 1 -- 172.21.2.102:0/624062665 wait complete. 2021-02-10T11:21:31.526 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000344437 2021-02-10T11:21:31.581 DEBUG:teuthology.orchestra.run.gibba002:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:21:31.585 DEBUG:teuthology.orchestra.run.gibba013:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:21:31.960 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:31.948 7f48563be700 1 Processor -- start 2021-02-10T11:21:31.960 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:31.948 7f48563be700 1 -- start start 2021-02-10T11:21:31.960 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:31.949 7f48563be700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f48501b0470 0x7f48501b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:31.960 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:31.949 7f48563be700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f48500f20e0 0x7f48500f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:31.960 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:31.949 7f48563be700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48500f0b50 con 0x7f48500ef610 2021-02-10T11:21:31.961 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:31.949 7f48563be700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48500f0820 con 0x7f48501b0470 2021-02-10T11:21:31.961 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:31.949 7f48563be700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48500f24e0 con 0x7f48500f20e0 2021-02-10T11:21:31.961 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:31.949 7f48549db700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f48500f20e0 0x7f48500f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:31.961 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:31.949 7f48549db700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f48500f20e0 0x7f48500f0200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.113:3301/0 says I am v2:172.21.2.102:60228/0 (socket says 172.21.2.102:60228) 2021-02-10T11:21:31.961 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:31.949 7f4847fff700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f48500ef610 0x7f48500efa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:46918/0 (socket says 172.21.2.102:46918) 2021-02-10T11:21:31.961 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:31.949 7f48549db700 1 -- 172.21.2.102:0/2572296765 learned_addr learned my addr 172.21.2.102:0/2572296765 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:21:31.961 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:31.949 7f484ffff700 1 --2- 172.21.2.102:0/2572296765 >> v2:172.21.2.113:3300/0 conn(0x7f48501b0470 0x7f48501b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:31.961 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:31.950 7f48549db700 1 -- 172.21.2.102:0/2572296765 >> v2:172.21.2.113:3300/0 conn(0x7f48501b0470 msgr2=0x7f48501b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:21:31.961 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:31.950 7f48549db700 1 --2- 172.21.2.102:0/2572296765 >> v2:172.21.2.113:3300/0 conn(0x7f48501b0470 0x7f48501b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:31.961 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:31.950 7f48549db700 1 -- 172.21.2.102:0/2572296765 >> v1:172.21.2.102:6789/0 conn(0x7f48500ef610 legacy=0x7f48500efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:31.961 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:31.950 7f484f7fe700 1 -- 172.21.2.102:0/2572296765 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 203487635 0 0) 0x7f48400009b0 con 0x7f48500ef610 2021-02-10T11:21:31.961 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:31.950 7f48549db700 1 -- 172.21.2.102:0/2572296765 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48500e52e0 con 0x7f48500f20e0 2021-02-10T11:21:31.961 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:31.950 7f48549db700 1 --2- 172.21.2.102:0/2572296765 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f48500f20e0 0x7f48500f0200 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7f4848006aa0 tx=0x7f4848009ea0).ready entity=mon.2 client_cookie=1b33fbf0f6f38438 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:31.962 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:31.950 7f484f7fe700 1 -- 172.21.2.102:0/2572296765 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f484800b510 con 0x7f48500f20e0 2021-02-10T11:21:31.962 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:31.950 7f484f7fe700 1 -- 172.21.2.102:0/2572296765 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4848003030 con 0x7f48500f20e0 2021-02-10T11:21:31.962 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:31.950 7f484f7fe700 1 -- 172.21.2.102:0/2572296765 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f484800bad0 con 0x7f48500f20e0 2021-02-10T11:21:31.962 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:31.950 7f48563be700 1 -- 172.21.2.102:0/2572296765 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f48500f20e0 msgr2=0x7f48500f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:31.962 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:31.950 7f48563be700 1 --2- 172.21.2.102:0/2572296765 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f48500f20e0 0x7f48500f0200 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7f4848006aa0 tx=0x7f4848009ea0).stop 2021-02-10T11:21:31.962 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:31.951 7f48563be700 1 -- 172.21.2.102:0/2572296765 shutdown_connections 2021-02-10T11:21:31.962 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:31.951 7f48563be700 1 --2- 172.21.2.102:0/2572296765 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f48500f20e0 0x7f48500f0200 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:31.962 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:31.951 7f48563be700 1 --2- 172.21.2.102:0/2572296765 >> v2:172.21.2.113:3300/0 conn(0x7f48501b0470 0x7f48501b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:31.962 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:31.951 7f48563be700 1 -- 172.21.2.102:0/2572296765 >> 172.21.2.102:0/2572296765 conn(0x7f48500ce3e0 msgr2=0x7f48500e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:31.962 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:31.951 7f48563be700 1 -- 172.21.2.102:0/2572296765 shutdown_connections 2021-02-10T11:21:31.962 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:31.951 7f48563be700 1 -- 172.21.2.102:0/2572296765 wait complete. 2021-02-10T11:21:31.964 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:31.966 7f48563be700 1 Processor -- start 2021-02-10T11:21:31.964 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:31.966 7f48563be700 1 -- start start 2021-02-10T11:21:31.964 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:31.966 7f48563be700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f48501a42e0 0x7f485009ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:31.964 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:31.966 7f48563be700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f48501a46e0 0x7f485009f310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:31.964 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:31.966 7f48563be700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f485008ad30 0x7f485008b130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:31.964 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:31.966 7f48563be700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f485008a7b0 con 0x7f48501a42e0 2021-02-10T11:21:31.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:31.966 7f48563be700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48500a65f0 con 0x7f48501a46e0 2021-02-10T11:21:31.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:31.966 7f48563be700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f485008a430 con 0x7f485008ad30 2021-02-10T11:21:31.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:31.967 7f484ffff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f48501a42e0 0x7f485009ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:31.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:31.967 7f484ffff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f48501a42e0 0x7f485009ecf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:54696/0 (socket says 172.21.2.102:54696) 2021-02-10T11:21:31.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:31.967 7f484ffff700 1 -- 172.21.2.102:0/832914477 learned_addr learned my addr 172.21.2.102:0/832914477 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:21:31.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:31.967 7f4847fff700 1 --2- 172.21.2.102:0/832914477 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f48501a46e0 0x7f485009f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:31.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:31.967 7f48549db700 1 --2- 172.21.2.102:0/832914477 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f485008ad30 0x7f485008b130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:31.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:31.967 7f484ffff700 1 -- 172.21.2.102:0/832914477 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f485008ad30 msgr2=0x7f485008b130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:31.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:31.967 7f484ffff700 1 --2- 172.21.2.102:0/832914477 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f485008ad30 0x7f485008b130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:31.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:31.967 7f484ffff700 1 -- 172.21.2.102:0/832914477 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f48501a46e0 msgr2=0x7f485009f310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:31.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:31.967 7f484ffff700 1 --2- 172.21.2.102:0/832914477 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f48501a46e0 0x7f485009f310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:31.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:31.967 7f484ffff700 1 -- 172.21.2.102:0/832914477 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f485009fa50 con 0x7f48501a42e0 2021-02-10T11:21:31.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:31.967 7f484ffff700 1 --2- 172.21.2.102:0/832914477 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f48501a42e0 0x7f485009ecf0 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 rx=0x7f4838006f60 tx=0x7f483800b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:31.966 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:31.967 7f484d7fa700 1 -- 172.21.2.102:0/832914477 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f483800bd60 con 0x7f48501a42e0 2021-02-10T11:21:31.966 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:31.967 7f484d7fa700 1 -- 172.21.2.102:0/832914477 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4838000c90 con 0x7f48501a42e0 2021-02-10T11:21:31.966 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:31.967 7f484d7fa700 1 -- 172.21.2.102:0/832914477 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f48380022b0 con 0x7f48501a42e0 2021-02-10T11:21:31.966 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:31.967 7f48563be700 1 -- 172.21.2.102:0/832914477 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f485009fa50 con 0x7f48501a42e0 2021-02-10T11:21:31.966 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:31.967 7f48563be700 1 -- 172.21.2.102:0/832914477 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f485009fa50 con 0x7f48501a42e0 2021-02-10T11:21:31.966 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:31.968 7f484d7fa700 1 -- 172.21.2.102:0/832914477 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4838006b00 con 0x7f48501a42e0 2021-02-10T11:21:31.966 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:31.968 7f484d7fa700 1 --2- 172.21.2.102:0/832914477 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4834018730 0x7f483401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:31.967 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:31.968 7f484d7fa700 1 -- 172.21.2.102:0/832914477 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f4838032620 con 0x7f48501a42e0 2021-02-10T11:21:31.967 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:31.968 7f4847fff700 1 --2- 172.21.2.102:0/832914477 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4834018730 0x7f483401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:31.967 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:31.968 7f4847fff700 1 --2- 172.21.2.102:0/832914477 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4834018730 0x7f483401ac10 crc :-1 s=READY pgs=460 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:31.993 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:31.998 7f48563be700 1 -- 172.21.2.102:0/832914477 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4850105ae0 con 0x7f48501a42e0 2021-02-10T11:21:31.995 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:32.001 7f484d7fa700 1 -- 172.21.2.102:0/832914477 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f4838006340 con 0x7f48501a42e0 2021-02-10T11:21:32.123 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:32.129 7f48563be700 1 -- 172.21.2.102:0/832914477 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344437", "target": ["mgr", ""]}) v1 -- 0x7f48500f0b50 con 0x7f4834018730 2021-02-10T11:21:32.134 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:32.140 7f484d7fa700 1 -- 172.21.2.102:0/832914477 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f4840001970 con 0x7f4834018730 2021-02-10T11:21:32.140 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:32.145 7f48563be700 1 -- 172.21.2.102:0/832914477 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4834018730 msgr2=0x7f483401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:32.141 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:32.145 7f48563be700 1 --2- 172.21.2.102:0/832914477 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4834018730 0x7f483401ac10 crc :-1 s=READY pgs=460 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:32.141 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:32.145 7f48563be700 1 -- 172.21.2.102:0/832914477 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f48501a42e0 msgr2=0x7f485009ecf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:32.141 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:32.145 7f48563be700 1 --2- 172.21.2.102:0/832914477 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f48501a42e0 0x7f485009ecf0 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 rx=0x7f4838006f60 tx=0x7f483800b970).stop 2021-02-10T11:21:32.141 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:32.146 7f48563be700 1 -- 172.21.2.102:0/832914477 shutdown_connections 2021-02-10T11:21:32.141 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:32.146 7f48563be700 1 --2- 172.21.2.102:0/832914477 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f485008ad30 0x7f485008b130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:32.141 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:32.146 7f48563be700 1 --2- 172.21.2.102:0/832914477 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4834018730 0x7f483401ac10 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:32.141 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:32.146 7f48563be700 1 --2- 172.21.2.102:0/832914477 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f48501a46e0 0x7f485009f310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:32.141 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:32.146 7f48563be700 1 --2- 172.21.2.102:0/832914477 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f48501a42e0 0x7f485009ecf0 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:32.141 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:32.146 7f48563be700 1 -- 172.21.2.102:0/832914477 >> 172.21.2.102:0/832914477 conn(0x7f4850095a70 msgr2=0x7f48500bf050 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:32.141 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:32.146 7f48563be700 1 -- 172.21.2.102:0/832914477 shutdown_connections 2021-02-10T11:21:32.141 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:32.146 7f48563be700 1 -- 172.21.2.102:0/832914477 wait complete. 2021-02-10T11:21:32.150 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000344438 2021-02-10T11:21:32.584 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:32.572 7f792465d700 1 Processor -- start 2021-02-10T11:21:32.584 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:32.572 7f792465d700 1 -- start start 2021-02-10T11:21:32.584 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:32.573 7f792465d700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f791c1b0870 0x7f791c0f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:32.584 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:32.573 7f792465d700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f791c0efe00 0x7f791c0f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:32.584 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:32.573 7f792465d700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f791c1b0c70 con 0x7f791c1b0470 2021-02-10T11:21:32.585 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:32.573 7f792465d700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f791c0f0b50 con 0x7f791c0efe00 2021-02-10T11:21:32.585 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:32.573 7f792465d700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f791c0f0820 con 0x7f791c1b0870 2021-02-10T11:21:32.585 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:32.573 7f7922c7a700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f791c0efe00 0x7f791c0f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:32.585 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:32.573 7f7922c7a700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f791c0efe00 0x7f791c0f0200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:54708/0 (socket says 172.21.2.102:54708) 2021-02-10T11:21:32.585 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:32.573 7f7922c7a700 1 -- 172.21.2.102:0/907309405 learned_addr learned my addr 172.21.2.102:0/907309405 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:21:32.585 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:32.573 7f7921c78700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f791c1b0870 0x7f791c0f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:32.585 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:32.573 7f7921c78700 1 -- 172.21.2.102:0/907309405 >> v2:172.21.2.102:3300/0 conn(0x7f791c0efe00 msgr2=0x7f791c0f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:21:32.585 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:32.573 7f7921c78700 1 --2- 172.21.2.102:0/907309405 >> v2:172.21.2.102:3300/0 conn(0x7f791c0efe00 0x7f791c0f0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:32.585 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:32.573 7f7921c78700 1 -- 172.21.2.102:0/907309405 >> v1:172.21.2.113:6789/0 conn(0x7f791c1b0470 legacy=0x7f791c0ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:32.585 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:32.573 7f7921c78700 1 -- 172.21.2.102:0/907309405 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f791c0e52e0 con 0x7f791c1b0870 2021-02-10T11:21:32.585 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:32.574 7f7921c78700 1 --2- 172.21.2.102:0/907309405 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f791c1b0870 0x7f791c0f20e0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f79180069d0 tx=0x7f7918009e40).ready entity=mon.2 client_cookie=d339330953aa0666 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:32.585 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:32.574 7f7921477700 1 -- 172.21.2.102:0/907309405 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f791800b510 con 0x7f791c1b0870 2021-02-10T11:21:32.585 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:32.574 7f7921477700 1 -- 172.21.2.102:0/907309405 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f791800b790 con 0x7f791c1b0870 2021-02-10T11:21:32.586 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:32.574 7f7921477700 1 -- 172.21.2.102:0/907309405 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f791800bcb0 con 0x7f791c1b0870 2021-02-10T11:21:32.586 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:32.574 7f792465d700 1 -- 172.21.2.102:0/907309405 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f791c1b0870 msgr2=0x7f791c0f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:32.586 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:32.574 7f792465d700 1 --2- 172.21.2.102:0/907309405 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f791c1b0870 0x7f791c0f20e0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f79180069d0 tx=0x7f7918009e40).stop 2021-02-10T11:21:32.586 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:32.575 7f792465d700 1 -- 172.21.2.102:0/907309405 shutdown_connections 2021-02-10T11:21:32.586 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:32.575 7f792465d700 1 --2- 172.21.2.102:0/907309405 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f791c1b0870 0x7f791c0f20e0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:32.586 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:32.575 7f792465d700 1 --2- 172.21.2.102:0/907309405 >> v2:172.21.2.102:3300/0 conn(0x7f791c0efe00 0x7f791c0f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:32.586 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:32.575 7f792465d700 1 -- 172.21.2.102:0/907309405 >> 172.21.2.102:0/907309405 conn(0x7f791c0ce3e0 msgr2=0x7f791c0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:32.586 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:32.575 7f792465d700 1 -- 172.21.2.102:0/907309405 shutdown_connections 2021-02-10T11:21:32.586 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:32.575 7f792465d700 1 -- 172.21.2.102:0/907309405 wait complete. 2021-02-10T11:21:32.588 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:32.590 7f792465d700 1 Processor -- start 2021-02-10T11:21:32.588 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:32.590 7f792465d700 1 -- start start 2021-02-10T11:21:32.588 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:32.590 7f792465d700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f791c0a0420 0x7f791c0a0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:32.588 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:32.590 7f792465d700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f791c189d00 0x7f791c18a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:32.588 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:32.590 7f792465d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f791c18a720 0x7f791c0b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:32.589 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:32.590 7f792465d700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f791c189800 con 0x7f791c18a720 2021-02-10T11:21:32.589 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:32.590 7f792465d700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f791c18ab20 con 0x7f791c0a0420 2021-02-10T11:21:32.589 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:32.590 7f792465d700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f791c0a1140 con 0x7f791c189d00 2021-02-10T11:21:32.589 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:32.591 7f7922c7a700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f791c18a720 0x7f791c0b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:32.589 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:32.591 7f7922479700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f791c0a0420 0x7f791c0a0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:32.589 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:32.591 7f7922c7a700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f791c18a720 0x7f791c0b7be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:54714/0 (socket says 172.21.2.102:54714) 2021-02-10T11:21:32.589 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:32.591 7f7922c7a700 1 -- 172.21.2.102:0/1588183712 learned_addr learned my addr 172.21.2.102:0/1588183712 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:21:32.589 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:32.591 7f7921c78700 1 --2- 172.21.2.102:0/1588183712 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f791c189d00 0x7f791c18a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:32.589 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:32.591 7f7922c7a700 1 -- 172.21.2.102:0/1588183712 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f791c189d00 msgr2=0x7f791c18a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:32.589 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:32.591 7f7922c7a700 1 --2- 172.21.2.102:0/1588183712 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f791c189d00 0x7f791c18a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:32.589 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:32.591 7f7922c7a700 1 -- 172.21.2.102:0/1588183712 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f791c0a0420 msgr2=0x7f791c0a0820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:32.589 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:32.591 7f7922c7a700 1 --2- 172.21.2.102:0/1588183712 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f791c0a0420 0x7f791c0a0820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:32.589 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:32.591 7f7922c7a700 1 -- 172.21.2.102:0/1588183712 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f791c0b8200 con 0x7f791c18a720 2021-02-10T11:21:32.590 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:32.591 7f7922c7a700 1 --2- 172.21.2.102:0/1588183712 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f791c18a720 0x7f791c0b7be0 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 rx=0x7f7914006ed0 tx=0x7f791400b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:32.590 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:32.591 7f7912ffd700 1 -- 172.21.2.102:0/1588183712 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7914000c10 con 0x7f791c18a720 2021-02-10T11:21:32.590 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:32.591 7f7912ffd700 1 -- 172.21.2.102:0/1588183712 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f791400bd50 con 0x7f791c18a720 2021-02-10T11:21:32.590 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:32.591 7f792465d700 1 -- 172.21.2.102:0/1588183712 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f791c0b8200 con 0x7f791c18a720 2021-02-10T11:21:32.590 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:32.591 7f7912ffd700 1 -- 172.21.2.102:0/1588183712 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7914002220 con 0x7f791c18a720 2021-02-10T11:21:32.590 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:32.591 7f792465d700 1 -- 172.21.2.102:0/1588183712 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f791c0b8790 con 0x7f791c18a720 2021-02-10T11:21:32.590 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:32.592 7f7912ffd700 1 -- 172.21.2.102:0/1588183712 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f7914000ab0 con 0x7f791c18a720 2021-02-10T11:21:32.590 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:32.592 7f7912ffd700 1 --2- 172.21.2.102:0/1588183712 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7904018730 0x7f790401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:32.590 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:32.592 7f7912ffd700 1 -- 172.21.2.102:0/1588183712 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f7914032610 con 0x7f791c18a720 2021-02-10T11:21:32.590 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:32.592 7f7922479700 1 --2- 172.21.2.102:0/1588183712 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7904018730 0x7f790401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:32.590 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:32.592 7f7922479700 1 --2- 172.21.2.102:0/1588183712 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7904018730 0x7f790401ac10 crc :-1 s=READY pgs=461 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:32.616 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:32.621 7f792465d700 1 -- 172.21.2.102:0/1588183712 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f791c18ab20 con 0x7f791c18a720 2021-02-10T11:21:32.618 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:32.624 7f7912ffd700 1 -- 172.21.2.102:0/1588183712 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f79140065e0 con 0x7f791c18a720 2021-02-10T11:21:32.748 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:32.753 7f792465d700 1 -- 172.21.2.102:0/1588183712 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344438", "target": ["mgr", ""]}) v1 -- 0x7f791c18ab20 con 0x7f7904018730 2021-02-10T11:21:32.758 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:32.763 7f7912ffd700 1 -- 172.21.2.102:0/1588183712 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f790c006550 con 0x7f7904018730 2021-02-10T11:21:32.765 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:32.771 7f792465d700 1 -- 172.21.2.102:0/1588183712 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7904018730 msgr2=0x7f790401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:32.765 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:32.771 7f792465d700 1 --2- 172.21.2.102:0/1588183712 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7904018730 0x7f790401ac10 crc :-1 s=READY pgs=461 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:32.765 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:32.771 7f792465d700 1 -- 172.21.2.102:0/1588183712 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f791c18a720 msgr2=0x7f791c0b7be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:32.766 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:32.771 7f792465d700 1 --2- 172.21.2.102:0/1588183712 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f791c18a720 0x7f791c0b7be0 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 rx=0x7f7914006ed0 tx=0x7f791400b960).stop 2021-02-10T11:21:32.766 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:32.771 7f792465d700 1 -- 172.21.2.102:0/1588183712 shutdown_connections 2021-02-10T11:21:32.766 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:32.771 7f792465d700 1 --2- 172.21.2.102:0/1588183712 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f791c18a720 0x7f791c0b7be0 unknown :-1 s=CLOSED pgs=943 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:32.766 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:32.771 7f792465d700 1 --2- 172.21.2.102:0/1588183712 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f791c189d00 0x7f791c18a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:32.766 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:32.771 7f792465d700 1 --2- 172.21.2.102:0/1588183712 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7904018730 0x7f790401ac10 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:32.766 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:32.771 7f792465d700 1 --2- 172.21.2.102:0/1588183712 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f791c0a0420 0x7f791c0a0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:32.766 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:32.771 7f792465d700 1 -- 172.21.2.102:0/1588183712 >> 172.21.2.102:0/1588183712 conn(0x7f791c094e10 msgr2=0x7f791c0a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:32.766 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:32.771 7f792465d700 1 -- 172.21.2.102:0/1588183712 shutdown_connections 2021-02-10T11:21:32.766 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:32.771 7f792465d700 1 -- 172.21.2.102:0/1588183712 wait complete. 2021-02-10T11:21:32.774 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000344439 2021-02-10T11:21:33.203 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:33.191 7efc5c8f3700 1 Processor -- start 2021-02-10T11:21:33.203 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:33.191 7efc5c8f3700 1 -- start start 2021-02-10T11:21:33.203 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:33.191 7efc5c8f3700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7efc581b0870 0x7efc580f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:33.203 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:33.191 7efc5c8f3700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7efc580efe00 0x7efc580f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:33.204 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:33.191 7efc5c8f3700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc581b0c70 con 0x7efc581b0470 2021-02-10T11:21:33.204 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:33.191 7efc5c8f3700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc580f0b50 con 0x7efc580efe00 2021-02-10T11:21:33.204 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:33.191 7efc5c8f3700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc580f0820 con 0x7efc581b0870 2021-02-10T11:21:33.204 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:33.191 7efc5661c700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7efc581b0470 0x7efc580ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:46942/0 (socket says 172.21.2.102:46942) 2021-02-10T11:21:33.204 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:33.191 7efc5661c700 1 -- 172.21.2.102:0/2288696131 learned_addr learned my addr 172.21.2.102:0/2288696131 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:21:33.204 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:33.191 7efc56e1d700 1 --2- 172.21.2.102:0/2288696131 >> v2:172.21.2.113:3300/0 conn(0x7efc580efe00 0x7efc580f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:33.204 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:33.192 7efc4de1b700 1 --2- 172.21.2.102:0/2288696131 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7efc581b0870 0x7efc580f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:33.204 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:33.192 7efc55e1b700 1 -- 172.21.2.102:0/2288696131 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 832060659 0 0) 0x7efc48001180 con 0x7efc581b0470 2021-02-10T11:21:33.204 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:33.192 7efc55e1b700 1 -- 172.21.2.102:0/2288696131 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc44002fe0 con 0x7efc581b0470 2021-02-10T11:21:33.204 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:33.192 7efc55e1b700 1 -- 172.21.2.102:0/2288696131 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1162476533 0 0) 0x7efc48000ed0 con 0x7efc581b0470 2021-02-10T11:21:33.204 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:33.192 7efc56e1d700 1 -- 172.21.2.102:0/2288696131 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7efc581b0870 msgr2=0x7efc580f20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:21:33.204 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:33.192 7efc56e1d700 1 --2- 172.21.2.102:0/2288696131 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7efc581b0870 0x7efc580f20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:33.204 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:33.192 7efc56e1d700 1 -- 172.21.2.102:0/2288696131 >> v1:172.21.2.102:6789/0 conn(0x7efc581b0470 legacy=0x7efc580ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:33.205 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:33.192 7efc56e1d700 1 -- 172.21.2.102:0/2288696131 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc580e52e0 con 0x7efc580efe00 2021-02-10T11:21:33.205 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:33.192 7efc56e1d700 1 --2- 172.21.2.102:0/2288696131 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7efc580efe00 0x7efc580f0200 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 rx=0x7efc50006e50 tx=0x7efc50009e70).ready entity=mon.1 client_cookie=6c6a940e6093ef25 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:33.205 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:33.192 7efc55e1b700 1 -- 172.21.2.102:0/2288696131 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efc5000b510 con 0x7efc580efe00 2021-02-10T11:21:33.205 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:33.192 7efc55e1b700 1 -- 172.21.2.102:0/2288696131 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc50006790 con 0x7efc580efe00 2021-02-10T11:21:33.205 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:33.192 7efc55e1b700 1 -- 172.21.2.102:0/2288696131 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efc5000b940 con 0x7efc580efe00 2021-02-10T11:21:33.205 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:33.192 7efc5c8f3700 1 -- 172.21.2.102:0/2288696131 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7efc580efe00 msgr2=0x7efc580f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:33.205 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:33.192 7efc5c8f3700 1 --2- 172.21.2.102:0/2288696131 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7efc580efe00 0x7efc580f0200 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 rx=0x7efc50006e50 tx=0x7efc50009e70).stop 2021-02-10T11:21:33.205 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:33.193 7efc5c8f3700 1 -- 172.21.2.102:0/2288696131 shutdown_connections 2021-02-10T11:21:33.205 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:33.193 7efc5c8f3700 1 --2- 172.21.2.102:0/2288696131 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7efc580efe00 0x7efc580f0200 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:33.205 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:33.193 7efc5c8f3700 1 --2- 172.21.2.102:0/2288696131 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7efc581b0870 0x7efc580f20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:33.205 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:33.193 7efc5c8f3700 1 -- 172.21.2.102:0/2288696131 >> 172.21.2.102:0/2288696131 conn(0x7efc580ce3e0 msgr2=0x7efc580e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:33.205 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:33.193 7efc5c8f3700 1 -- 172.21.2.102:0/2288696131 shutdown_connections 2021-02-10T11:21:33.205 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:33.193 7efc5c8f3700 1 -- 172.21.2.102:0/2288696131 wait complete. 2021-02-10T11:21:33.207 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:33.208 7efc5c8f3700 1 Processor -- start 2021-02-10T11:21:33.207 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:33.208 7efc5c8f3700 1 -- start start 2021-02-10T11:21:33.207 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:33.208 7efc5c8f3700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7efc5809f140 0x7efc580c1940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:33.207 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:33.208 7efc5c8f3700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efc580c1f60 0x7efc5809e300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:33.207 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:33.208 7efc5c8f3700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7efc5809e920 0x7efc580c2960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:33.207 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:33.208 7efc5c8f3700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc580c23e0 con 0x7efc580c1f60 2021-02-10T11:21:33.208 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:33.208 7efc5c8f3700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc5809ed20 con 0x7efc5809f140 2021-02-10T11:21:33.208 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:33.208 7efc5c8f3700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc58105d10 con 0x7efc5809e920 2021-02-10T11:21:33.208 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:33.208 7efc4de1b700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efc580c1f60 0x7efc5809e300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:33.208 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:33.208 7efc4de1b700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efc580c1f60 0x7efc5809e300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:54724/0 (socket says 172.21.2.102:54724) 2021-02-10T11:21:33.208 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:33.208 7efc4de1b700 1 -- 172.21.2.102:0/866546397 learned_addr learned my addr 172.21.2.102:0/866546397 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:21:33.208 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:33.208 7efc56e1d700 1 --2- 172.21.2.102:0/866546397 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7efc5809e920 0x7efc580c2960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:33.208 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:33.209 7efc5661c700 1 --2- 172.21.2.102:0/866546397 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7efc5809f140 0x7efc580c1940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:33.208 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:33.209 7efc4de1b700 1 -- 172.21.2.102:0/866546397 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7efc5809e920 msgr2=0x7efc580c2960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:33.208 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:33.209 7efc4de1b700 1 --2- 172.21.2.102:0/866546397 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7efc5809e920 0x7efc580c2960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:33.208 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:33.209 7efc4de1b700 1 -- 172.21.2.102:0/866546397 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7efc5809f140 msgr2=0x7efc580c1940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:33.208 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:33.209 7efc4de1b700 1 --2- 172.21.2.102:0/866546397 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7efc5809f140 0x7efc580c1940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:33.208 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:33.209 7efc4de1b700 1 -- 172.21.2.102:0/866546397 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc581a4910 con 0x7efc580c1f60 2021-02-10T11:21:33.209 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:33.209 7efc4de1b700 1 --2- 172.21.2.102:0/866546397 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efc580c1f60 0x7efc5809e300 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 rx=0x7efc40006f60 tx=0x7efc4000b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:33.209 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:33.209 7efc4f7fe700 1 -- 172.21.2.102:0/866546397 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efc4000bd60 con 0x7efc580c1f60 2021-02-10T11:21:33.209 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:33.209 7efc4f7fe700 1 -- 172.21.2.102:0/866546397 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc40000c90 con 0x7efc580c1f60 2021-02-10T11:21:33.209 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:33.209 7efc4f7fe700 1 -- 172.21.2.102:0/866546397 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efc40002220 con 0x7efc580c1f60 2021-02-10T11:21:33.209 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:33.209 7efc5c8f3700 1 -- 172.21.2.102:0/866546397 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc581a4910 con 0x7efc580c1f60 2021-02-10T11:21:33.209 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:33.209 7efc5c8f3700 1 -- 172.21.2.102:0/866546397 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc581a4910 con 0x7efc580c1f60 2021-02-10T11:21:33.209 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:33.210 7efc4f7fe700 1 -- 172.21.2.102:0/866546397 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7efc400052e0 con 0x7efc580c1f60 2021-02-10T11:21:33.209 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:33.210 7efc4f7fe700 1 --2- 172.21.2.102:0/866546397 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7efc3c0186c0 0x7efc3c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:33.209 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:33.210 7efc4f7fe700 1 -- 172.21.2.102:0/866546397 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7efc40032540 con 0x7efc580c1f60 2021-02-10T11:21:33.209 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:33.210 7efc5661c700 1 --2- 172.21.2.102:0/866546397 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7efc3c0186c0 0x7efc3c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:33.209 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:33.210 7efc5661c700 1 --2- 172.21.2.102:0/866546397 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7efc3c0186c0 0x7efc3c01abc0 crc :-1 s=READY pgs=462 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:33.236 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:33.241 7efc5c8f3700 1 -- 172.21.2.102:0/866546397 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc580f0b50 con 0x7efc580c1f60 2021-02-10T11:21:33.239 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:33.243 7efc4f7fe700 1 -- 172.21.2.102:0/866546397 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7efc40006b00 con 0x7efc580c1f60 2021-02-10T11:21:33.368 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:33.372 7efc5c8f3700 1 -- 172.21.2.102:0/866546397 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344439", "target": ["mgr", ""]}) v1 -- 0x7efc580f0b50 con 0x7efc3c0186c0 2021-02-10T11:21:33.378 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:33.382 7efc4f7fe700 1 -- 172.21.2.102:0/866546397 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7efc480013f0 con 0x7efc3c0186c0 2021-02-10T11:21:33.384 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:33.388 7efc5c8f3700 1 -- 172.21.2.102:0/866546397 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7efc3c0186c0 msgr2=0x7efc3c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:33.384 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:33.388 7efc5c8f3700 1 --2- 172.21.2.102:0/866546397 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7efc3c0186c0 0x7efc3c01abc0 crc :-1 s=READY pgs=462 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:33.384 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:33.388 7efc5c8f3700 1 -- 172.21.2.102:0/866546397 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efc580c1f60 msgr2=0x7efc5809e300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:33.384 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:33.388 7efc5c8f3700 1 --2- 172.21.2.102:0/866546397 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efc580c1f60 0x7efc5809e300 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 rx=0x7efc40006f60 tx=0x7efc4000b970).stop 2021-02-10T11:21:33.385 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:33.388 7efc5c8f3700 1 -- 172.21.2.102:0/866546397 shutdown_connections 2021-02-10T11:21:33.385 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:33.388 7efc5c8f3700 1 --2- 172.21.2.102:0/866546397 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7efc5809e920 0x7efc580c2960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:33.385 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:33.388 7efc5c8f3700 1 --2- 172.21.2.102:0/866546397 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efc580c1f60 0x7efc5809e300 unknown :-1 s=CLOSED pgs=945 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:33.385 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:33.388 7efc5c8f3700 1 --2- 172.21.2.102:0/866546397 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7efc3c0186c0 0x7efc3c01abc0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:33.385 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:33.388 7efc5c8f3700 1 --2- 172.21.2.102:0/866546397 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7efc5809f140 0x7efc580c1940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:33.385 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:33.388 7efc5c8f3700 1 -- 172.21.2.102:0/866546397 >> 172.21.2.102:0/866546397 conn(0x7efc5809fc10 msgr2=0x7efc580b3250 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:33.385 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:33.389 7efc5c8f3700 1 -- 172.21.2.102:0/866546397 shutdown_connections 2021-02-10T11:21:33.385 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:33.389 7efc5c8f3700 1 -- 172.21.2.102:0/866546397 wait complete. 2021-02-10T11:21:33.393 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000344440 2021-02-10T11:21:33.823 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:33.811 7fa2d7f05700 1 Processor -- start 2021-02-10T11:21:33.823 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:33.811 7fa2d7f05700 1 -- start start 2021-02-10T11:21:33.823 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:33.811 7fa2d7f05700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa2d01b0470 0x7fa2d01b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:33.823 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:33.811 7fa2d7f05700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fa2d00ef610 0x7fa2d00efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:33.823 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:33.811 7fa2d7f05700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fa2d00f0030 0x7fa2d00f0430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:33.823 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:33.811 7fa2d7f05700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2d00e5ea0 con 0x7fa2d00f0030 2021-02-10T11:21:33.823 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:33.811 7fa2d7f05700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2d00f0b50 con 0x7fa2d00ef610 2021-02-10T11:21:33.824 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:33.811 7fa2d7f05700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2d00e52e0 con 0x7fa2d01b0470 2021-02-10T11:21:33.824 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:33.811 7fa2d6522700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fa2d00f0030 0x7fa2d00f0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:33.824 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:33.811 7fa2d6522700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fa2d00f0030 0x7fa2d00f0430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:54734/0 (socket says 172.21.2.102:54734) 2021-02-10T11:21:33.824 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:33.811 7fa2d6522700 1 -- 172.21.2.102:0/625587427 learned_addr learned my addr 172.21.2.102:0/625587427 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:21:33.824 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:33.811 7fa2d5d21700 1 --2- 172.21.2.102:0/625587427 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa2d01b0470 0x7fa2d01b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:33.824 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:33.812 7fa2d6522700 1 -- 172.21.2.102:0/625587427 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa2d01b0470 msgr2=0x7fa2d01b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:21:33.824 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:33.812 7fa2d6522700 1 --2- 172.21.2.102:0/625587427 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa2d01b0470 0x7fa2d01b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:33.824 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:33.812 7fa2d5520700 1 --2- 172.21.2.102:0/625587427 >> v2:172.21.2.113:3300/0 conn(0x7fa2d00ef610 0x7fa2d00efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:33.824 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:33.812 7fa2d6522700 1 -- 172.21.2.102:0/625587427 >> v2:172.21.2.113:3300/0 conn(0x7fa2d00ef610 msgr2=0x7fa2d00efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:21:33.824 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:33.812 7fa2d6522700 1 --2- 172.21.2.102:0/625587427 >> v2:172.21.2.113:3300/0 conn(0x7fa2d00ef610 0x7fa2d00efa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:33.824 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:33.812 7fa2d6522700 1 -- 172.21.2.102:0/625587427 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2d00f10d0 con 0x7fa2d00f0030 2021-02-10T11:21:33.825 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:33.812 7fa2d5520700 1 --2- 172.21.2.102:0/625587427 >> v2:172.21.2.113:3300/0 conn(0x7fa2d00ef610 0x7fa2d00efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:21:33.825 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:33.812 7fa2d6522700 1 --2- 172.21.2.102:0/625587427 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa2d00f0030 0x7fa2d00f0430 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 rx=0x7fa2c8006f00 tx=0x7fa2c8009e90).ready entity=mon.0 client_cookie=b26f8f602aa055f8 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:33.825 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:33.812 7fa2d4d1f700 1 -- 172.21.2.102:0/625587427 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa2c800b300 con 0x7fa2d00f0030 2021-02-10T11:21:33.825 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:33.812 7fa2d4d1f700 1 -- 172.21.2.102:0/625587427 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2c800b580 con 0x7fa2d00f0030 2021-02-10T11:21:33.825 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:33.812 7fa2d4d1f700 1 -- 172.21.2.102:0/625587427 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa2c800baa0 con 0x7fa2d00f0030 2021-02-10T11:21:33.825 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:33.812 7fa2d7f05700 1 -- 172.21.2.102:0/625587427 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa2d00f0030 msgr2=0x7fa2d00f0430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:33.825 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:33.812 7fa2d7f05700 1 --2- 172.21.2.102:0/625587427 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa2d00f0030 0x7fa2d00f0430 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 rx=0x7fa2c8006f00 tx=0x7fa2c8009e90).stop 2021-02-10T11:21:33.825 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:33.812 7fa2d7f05700 1 -- 172.21.2.102:0/625587427 shutdown_connections 2021-02-10T11:21:33.825 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:33.812 7fa2d7f05700 1 --2- 172.21.2.102:0/625587427 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa2d00f0030 0x7fa2d00f0430 unknown :-1 s=CLOSED pgs=946 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:33.825 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:33.812 7fa2d7f05700 1 --2- 172.21.2.102:0/625587427 >> v2:172.21.2.113:3300/0 conn(0x7fa2d00ef610 0x7fa2d00efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:33.825 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:33.812 7fa2d7f05700 1 --2- 172.21.2.102:0/625587427 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa2d01b0470 0x7fa2d01b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:33.825 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:33.812 7fa2d7f05700 1 -- 172.21.2.102:0/625587427 >> 172.21.2.102:0/625587427 conn(0x7fa2d00ce3e0 msgr2=0x7fa2d00e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:33.825 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:33.813 7fa2d7f05700 1 -- 172.21.2.102:0/625587427 shutdown_connections 2021-02-10T11:21:33.826 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:33.813 7fa2d7f05700 1 -- 172.21.2.102:0/625587427 wait complete. 2021-02-10T11:21:33.827 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:33.827 7fa2d7f05700 1 Processor -- start 2021-02-10T11:21:33.827 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:33.827 7fa2d7f05700 1 -- start start 2021-02-10T11:21:33.827 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:33.828 7fa2d7f05700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa2d009b930 0x7fa2d00b7c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:33.828 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:33.828 7fa2d7f05700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa2d00b8410 0x7fa2d00b8810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:33.828 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:33.828 7fa2d7f05700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa2d00c1f80 0x7fa2d00c2380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:33.828 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:33.828 7fa2d7f05700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2d00c1a00 con 0x7fa2d00b8410 2021-02-10T11:21:33.828 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:33.828 7fa2d7f05700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2d00b8e30 con 0x7fa2d009b930 2021-02-10T11:21:33.828 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:33.828 7fa2d7f05700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2d00b9050 con 0x7fa2d00c1f80 2021-02-10T11:21:33.828 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:33.828 7fa2d5520700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa2d00b8410 0x7fa2d00b8810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:33.828 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:33.828 7fa2d5520700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa2d00b8410 0x7fa2d00b8810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:54740/0 (socket says 172.21.2.102:54740) 2021-02-10T11:21:33.828 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:33.828 7fa2d5520700 1 -- 172.21.2.102:0/1683274481 learned_addr learned my addr 172.21.2.102:0/1683274481 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:21:33.828 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:33.828 7fa2d5d21700 1 --2- 172.21.2.102:0/1683274481 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa2d009b930 0x7fa2d00b7c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:33.828 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:33.828 7fa2d6522700 1 --2- 172.21.2.102:0/1683274481 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa2d00c1f80 0x7fa2d00c2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:33.828 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:33.828 7fa2d5520700 1 -- 172.21.2.102:0/1683274481 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa2d00c1f80 msgr2=0x7fa2d00c2380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:33.829 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:33.828 7fa2d5520700 1 --2- 172.21.2.102:0/1683274481 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa2d00c1f80 0x7fa2d00c2380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:33.829 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:33.828 7fa2d5520700 1 -- 172.21.2.102:0/1683274481 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa2d009b930 msgr2=0x7fa2d00b7c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:33.829 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:33.828 7fa2d5520700 1 --2- 172.21.2.102:0/1683274481 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa2d009b930 0x7fa2d00b7c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:33.829 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:33.828 7fa2d5520700 1 -- 172.21.2.102:0/1683274481 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2d00c29a0 con 0x7fa2d00b8410 2021-02-10T11:21:33.829 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:33.828 7fa2d5520700 1 --2- 172.21.2.102:0/1683274481 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa2d00b8410 0x7fa2d00b8810 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 rx=0x7fa2cc006f20 tx=0x7fa2cc00b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:33.829 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:33.829 7fa2c27fc700 1 -- 172.21.2.102:0/1683274481 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa2cc00bd80 con 0x7fa2d00b8410 2021-02-10T11:21:33.829 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:33.829 7fa2c27fc700 1 -- 172.21.2.102:0/1683274481 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2cc000c90 con 0x7fa2d00b8410 2021-02-10T11:21:33.829 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:33.829 7fa2d7f05700 1 -- 172.21.2.102:0/1683274481 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2d00c29a0 con 0x7fa2d00b8410 2021-02-10T11:21:33.829 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:33.829 7fa2c27fc700 1 -- 172.21.2.102:0/1683274481 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa2cc0022b0 con 0x7fa2d00b8410 2021-02-10T11:21:33.829 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:33.829 7fa2d7f05700 1 -- 172.21.2.102:0/1683274481 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2d00c29a0 con 0x7fa2d00b8410 2021-02-10T11:21:33.829 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:33.829 7fa2c27fc700 1 -- 172.21.2.102:0/1683274481 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa2cc00a000 con 0x7fa2d00b8410 2021-02-10T11:21:33.829 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:33.829 7fa2c27fc700 1 --2- 172.21.2.102:0/1683274481 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa2b8018730 0x7fa2b801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:33.830 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:33.829 7fa2c27fc700 1 -- 172.21.2.102:0/1683274481 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fa2cc032610 con 0x7fa2d00b8410 2021-02-10T11:21:33.830 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:33.829 7fa2d5d21700 1 --2- 172.21.2.102:0/1683274481 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa2b8018730 0x7fa2b801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:33.830 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:33.830 7fa2d5d21700 1 --2- 172.21.2.102:0/1683274481 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa2b8018730 0x7fa2b801ac10 crc :-1 s=READY pgs=463 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:33.856 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:33.860 7fa2d7f05700 1 -- 172.21.2.102:0/1683274481 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2d0105b30 con 0x7fa2d00b8410 2021-02-10T11:21:33.858 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:33.863 7fa2c27fc700 1 -- 172.21.2.102:0/1683274481 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fa2cc006bb0 con 0x7fa2d00b8410 2021-02-10T11:21:33.987 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:33.991 7fa2d7f05700 1 -- 172.21.2.102:0/1683274481 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344440", "target": ["mgr", ""]}) v1 -- 0x7fa2d0105b30 con 0x7fa2b8018730 2021-02-10T11:21:33.998 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:34.003 7fa2c27fc700 1 -- 172.21.2.102:0/1683274481 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fa2c40009b0 con 0x7fa2b8018730 2021-02-10T11:21:34.004 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:34.008 7fa2d7f05700 1 -- 172.21.2.102:0/1683274481 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa2b8018730 msgr2=0x7fa2b801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:34.005 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:34.008 7fa2d7f05700 1 --2- 172.21.2.102:0/1683274481 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa2b8018730 0x7fa2b801ac10 crc :-1 s=READY pgs=463 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:34.005 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:34.008 7fa2d7f05700 1 -- 172.21.2.102:0/1683274481 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa2d00b8410 msgr2=0x7fa2d00b8810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:34.005 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:34.008 7fa2d7f05700 1 --2- 172.21.2.102:0/1683274481 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa2d00b8410 0x7fa2d00b8810 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 rx=0x7fa2cc006f20 tx=0x7fa2cc00b990).stop 2021-02-10T11:21:34.005 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:34.009 7fa2d7f05700 1 -- 172.21.2.102:0/1683274481 shutdown_connections 2021-02-10T11:21:34.005 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:34.009 7fa2d7f05700 1 --2- 172.21.2.102:0/1683274481 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa2d00c1f80 0x7fa2d00c2380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:34.005 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:34.009 7fa2d7f05700 1 --2- 172.21.2.102:0/1683274481 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa2d00b8410 0x7fa2d00b8810 unknown :-1 s=CLOSED pgs=947 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:34.005 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:34.009 7fa2d7f05700 1 --2- 172.21.2.102:0/1683274481 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa2b8018730 0x7fa2b801ac10 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:34.005 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:34.009 7fa2d7f05700 1 --2- 172.21.2.102:0/1683274481 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa2d009b930 0x7fa2d00b7c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:34.005 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:34.009 7fa2d7f05700 1 -- 172.21.2.102:0/1683274481 >> 172.21.2.102:0/1683274481 conn(0x7fa2d0094e10 msgr2=0x7fa2d00a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:34.005 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:34.009 7fa2d7f05700 1 -- 172.21.2.102:0/1683274481 shutdown_connections 2021-02-10T11:21:34.005 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:34.009 7fa2d7f05700 1 -- 172.21.2.102:0/1683274481 wait complete. 2021-02-10T11:21:34.014 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000344441 2021-02-10T11:21:34.444 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:34.432 7f630cd73700 1 Processor -- start 2021-02-10T11:21:34.444 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:34.432 7f630cd73700 1 -- start start 2021-02-10T11:21:34.444 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:34.432 7f630cd73700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f63081b0470 0x7f63081b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:34.444 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:34.432 7f630cd73700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f63080ef610 0x7f63080efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:34.444 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:34.432 7f630cd73700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63080f0b50 con 0x7f63080f0030 2021-02-10T11:21:34.444 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:34.432 7f630cd73700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63080e52e0 con 0x7f63081b0470 2021-02-10T11:21:34.444 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:34.432 7f630cd73700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63080e54d0 con 0x7f63080ef610 2021-02-10T11:21:34.444 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:34.432 7f6306b13700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f63081b0470 0x7f63081b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:34.444 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:34.432 7f6306b13700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f63081b0470 0x7f63081b0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:54746/0 (socket says 172.21.2.102:54746) 2021-02-10T11:21:34.445 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:34.432 7f6306b13700 1 -- 172.21.2.102:0/2988153252 learned_addr learned my addr 172.21.2.102:0/2988153252 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:21:34.445 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:34.433 7f6306312700 1 --2- 172.21.2.102:0/2988153252 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f63080ef610 0x7f63080efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:34.445 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:34.433 7f6306b13700 1 -- 172.21.2.102:0/2988153252 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f63080ef610 msgr2=0x7f63080efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:21:34.445 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:34.433 7f6306b13700 1 --2- 172.21.2.102:0/2988153252 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f63080ef610 0x7f63080efa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:34.445 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:34.433 7f6306b13700 1 -- 172.21.2.102:0/2988153252 >> v1:172.21.2.113:6789/0 conn(0x7f63080f0030 legacy=0x7f63080f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:34.445 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:34.433 7f6306b13700 1 -- 172.21.2.102:0/2988153252 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63080f0430 con 0x7f63081b0470 2021-02-10T11:21:34.445 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:34.433 7f6306b13700 1 --2- 172.21.2.102:0/2988153252 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f63081b0470 0x7f63081b0870 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 rx=0x7f62fc006700 tx=0x7f62fc009b80).ready entity=mon.0 client_cookie=29a2b478cb628256 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:34.445 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:34.433 7f6305b11700 1 -- 172.21.2.102:0/2988153252 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f62fc00c0c0 con 0x7f63081b0470 2021-02-10T11:21:34.445 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:34.433 7f6305b11700 1 -- 172.21.2.102:0/2988153252 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62fc00c340 con 0x7f63081b0470 2021-02-10T11:21:34.445 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:34.433 7f6305b11700 1 -- 172.21.2.102:0/2988153252 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f62fc00c860 con 0x7f63081b0470 2021-02-10T11:21:34.445 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:34.433 7f630cd73700 1 -- 172.21.2.102:0/2988153252 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f63081b0470 msgr2=0x7f63081b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:34.445 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:34.433 7f630cd73700 1 --2- 172.21.2.102:0/2988153252 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f63081b0470 0x7f63081b0870 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 rx=0x7f62fc006700 tx=0x7f62fc009b80).stop 2021-02-10T11:21:34.446 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:34.434 7f630cd73700 1 -- 172.21.2.102:0/2988153252 shutdown_connections 2021-02-10T11:21:34.446 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:34.434 7f630cd73700 1 --2- 172.21.2.102:0/2988153252 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f63080ef610 0x7f63080efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:34.446 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:34.434 7f630cd73700 1 --2- 172.21.2.102:0/2988153252 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f63081b0470 0x7f63081b0870 unknown :-1 s=CLOSED pgs=948 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:34.446 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:34.434 7f630cd73700 1 -- 172.21.2.102:0/2988153252 >> 172.21.2.102:0/2988153252 conn(0x7f63080ce3e0 msgr2=0x7f63080e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:34.446 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:34.434 7f630cd73700 1 -- 172.21.2.102:0/2988153252 shutdown_connections 2021-02-10T11:21:34.446 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:34.434 7f630cd73700 1 -- 172.21.2.102:0/2988153252 wait complete. 2021-02-10T11:21:34.447 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:34.448 7f630cd73700 1 Processor -- start 2021-02-10T11:21:34.447 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:34.448 7f630cd73700 1 -- start start 2021-02-10T11:21:34.448 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:34.449 7f630cd73700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6308190920 0x7f63080b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:34.448 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:34.449 7f630cd73700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f63080b8200 0x7f63080b8600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:34.448 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:34.449 7f630cd73700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f63080c1d80 0x7f63080c2180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:34.448 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:34.449 7f630cd73700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6308190d20 con 0x7f6308190920 2021-02-10T11:21:34.448 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:34.449 7f630cd73700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63080b8c20 con 0x7f63080b8200 2021-02-10T11:21:34.448 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:34.449 7f630cd73700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63080c27a0 con 0x7f63080c1d80 2021-02-10T11:21:34.448 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:34.449 7f6306b13700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6308190920 0x7f63080b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:34.448 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:34.449 7f6306b13700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6308190920 0x7f63080b7be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:54752/0 (socket says 172.21.2.102:54752) 2021-02-10T11:21:34.448 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:34.449 7f6306b13700 1 -- 172.21.2.102:0/2075961805 learned_addr learned my addr 172.21.2.102:0/2075961805 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:21:34.448 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:34.449 7f6306b13700 1 -- 172.21.2.102:0/2075961805 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f63080c1d80 msgr2=0x7f63080c2180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:34.448 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:34.449 7f6306b13700 1 --2- 172.21.2.102:0/2075961805 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f63080c1d80 0x7f63080c2180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:21:34.448 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:34.449 7f6306b13700 1 -- 172.21.2.102:0/2075961805 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f63080b8200 msgr2=0x7f63080b8600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:34.449 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:34.449 7f6306b13700 1 --2- 172.21.2.102:0/2075961805 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f63080b8200 0x7f63080b8600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:21:34.449 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:34.449 7f6306b13700 1 -- 172.21.2.102:0/2075961805 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63080c29c0 con 0x7f6308190920 2021-02-10T11:21:34.449 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:34.449 7f6306b13700 1 --2- 172.21.2.102:0/2075961805 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6308190920 0x7f63080b7be0 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 rx=0x7f62fc00b580 tx=0x7f62fc00b920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:34.449 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:34.450 7f62f77fe700 1 -- 172.21.2.102:0/2075961805 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f62fc00bd10 con 0x7f6308190920 2021-02-10T11:21:34.449 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:34.450 7f62f77fe700 1 -- 172.21.2.102:0/2075961805 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62fc002030 con 0x7f6308190920 2021-02-10T11:21:34.449 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:34.450 7f62f77fe700 1 -- 172.21.2.102:0/2075961805 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f62fc002480 con 0x7f6308190920 2021-02-10T11:21:34.449 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:34.450 7f630cd73700 1 -- 172.21.2.102:0/2075961805 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63080c29c0 con 0x7f6308190920 2021-02-10T11:21:34.449 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:34.450 7f630cd73700 1 -- 172.21.2.102:0/2075961805 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63080c2f50 con 0x7f6308190920 2021-02-10T11:21:34.449 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:34.450 7f62f77fe700 1 -- 172.21.2.102:0/2075961805 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f62fc000c90 con 0x7f6308190920 2021-02-10T11:21:34.449 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:34.450 7f62f77fe700 1 --2- 172.21.2.102:0/2075961805 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f62ec0186c0 0x7f62ec01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:34.449 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:34.450 7f62f77fe700 1 -- 172.21.2.102:0/2075961805 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f62fc0327b0 con 0x7f6308190920 2021-02-10T11:21:34.449 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:34.450 7f6306312700 1 --2- 172.21.2.102:0/2075961805 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f62ec0186c0 0x7f62ec01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:34.449 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:34.451 7f6306312700 1 --2- 172.21.2.102:0/2075961805 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f62ec0186c0 0x7f62ec01abc0 crc :-1 s=READY pgs=464 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:34.478 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:34.482 7f630cd73700 1 -- 172.21.2.102:0/2075961805 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6308105ae0 con 0x7f6308190920 2021-02-10T11:21:34.480 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:34.485 7f62f77fe700 1 -- 172.21.2.102:0/2075961805 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f62fc0009b0 con 0x7f6308190920 2021-02-10T11:21:34.606 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:34.610 7f630cd73700 1 -- 172.21.2.102:0/2075961805 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344441", "target": ["mgr", ""]}) v1 -- 0x7f63080c27a0 con 0x7f62ec0186c0 2021-02-10T11:21:34.617 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:34.622 7f62f77fe700 1 -- 172.21.2.102:0/2075961805 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f62f8006e70 con 0x7f62ec0186c0 2021-02-10T11:21:34.623 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:34.627 7f630cd73700 1 -- 172.21.2.102:0/2075961805 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f62ec0186c0 msgr2=0x7f62ec01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:34.623 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:34.627 7f630cd73700 1 --2- 172.21.2.102:0/2075961805 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f62ec0186c0 0x7f62ec01abc0 crc :-1 s=READY pgs=464 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:34.624 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:34.627 7f630cd73700 1 -- 172.21.2.102:0/2075961805 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6308190920 msgr2=0x7f63080b7be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:34.624 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:34.627 7f630cd73700 1 --2- 172.21.2.102:0/2075961805 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6308190920 0x7f63080b7be0 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 rx=0x7f62fc00b580 tx=0x7f62fc00b920).stop 2021-02-10T11:21:34.624 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:34.627 7f630cd73700 1 -- 172.21.2.102:0/2075961805 shutdown_connections 2021-02-10T11:21:34.624 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:34.627 7f630cd73700 1 --2- 172.21.2.102:0/2075961805 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f63080c1d80 0x7f63080c2180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:21:34.624 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:34.627 7f630cd73700 1 --2- 172.21.2.102:0/2075961805 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f62ec0186c0 0x7f62ec01abc0 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:34.624 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:34.627 7f630cd73700 1 --2- 172.21.2.102:0/2075961805 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f63080b8200 0x7f63080b8600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:21:34.624 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:34.627 7f630cd73700 1 --2- 172.21.2.102:0/2075961805 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6308190920 0x7f63080b7be0 unknown :-1 s=CLOSED pgs=949 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:34.624 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:34.627 7f630cd73700 1 -- 172.21.2.102:0/2075961805 >> 172.21.2.102:0/2075961805 conn(0x7f6308094e10 msgr2=0x7f63080a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:34.624 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:34.628 7f630cd73700 1 -- 172.21.2.102:0/2075961805 shutdown_connections 2021-02-10T11:21:34.624 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:34.628 7f630cd73700 1 -- 172.21.2.102:0/2075961805 wait complete. 2021-02-10T11:21:34.632 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000344442 2021-02-10T11:21:35.063 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:35.051 7f6894839700 1 Processor -- start 2021-02-10T11:21:35.063 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:35.051 7f6894839700 1 -- start start 2021-02-10T11:21:35.063 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:35.051 7f6894839700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f68901b0470 0x7f68901b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:35.063 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:35.051 7f6894839700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f68900ef610 0x7f68900efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:35.063 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:35.051 7f6894839700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68900f0b50 con 0x7f68900f0030 2021-02-10T11:21:35.064 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:35.051 7f6894839700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68900e52e0 con 0x7f68901b0470 2021-02-10T11:21:35.064 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:35.051 7f6894839700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68900e54d0 con 0x7f68900ef610 2021-02-10T11:21:35.064 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:35.052 7f688e61c700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f68901b0470 0x7f68901b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:35.064 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:35.052 7f688e61c700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f68901b0470 0x7f68901b0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:54760/0 (socket says 172.21.2.102:54760) 2021-02-10T11:21:35.064 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:35.052 7f688e61c700 1 -- 172.21.2.102:0/2535582973 learned_addr learned my addr 172.21.2.102:0/2535582973 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:21:35.064 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:35.052 7f688de1b700 1 --2- 172.21.2.102:0/2535582973 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f68900ef610 0x7f68900efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:35.064 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:35.052 7f688e61c700 1 -- 172.21.2.102:0/2535582973 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f68900ef610 msgr2=0x7f68900efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:21:35.064 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:35.052 7f688e61c700 1 --2- 172.21.2.102:0/2535582973 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f68900ef610 0x7f68900efa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:35.064 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:35.052 7f688e61c700 1 -- 172.21.2.102:0/2535582973 >> v1:172.21.2.113:6789/0 conn(0x7f68900f0030 legacy=0x7f68900f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:35.064 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:35.052 7f688e61c700 1 -- 172.21.2.102:0/2535582973 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68900f0430 con 0x7f68901b0470 2021-02-10T11:21:35.064 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:35.052 7f688e61c700 1 --2- 172.21.2.102:0/2535582973 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f68901b0470 0x7f68901b0870 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 rx=0x7f6880006700 tx=0x7f6880009b00).ready entity=mon.0 client_cookie=2aba8c6c4fc88a58 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:35.064 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:35.052 7f688d61a700 1 -- 172.21.2.102:0/2535582973 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f688000c0c0 con 0x7f68901b0470 2021-02-10T11:21:35.064 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:35.052 7f688d61a700 1 -- 172.21.2.102:0/2535582973 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f688000c340 con 0x7f68901b0470 2021-02-10T11:21:35.064 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:35.052 7f688d61a700 1 -- 172.21.2.102:0/2535582973 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f688000c820 con 0x7f68901b0470 2021-02-10T11:21:35.065 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:35.052 7f6894839700 1 -- 172.21.2.102:0/2535582973 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f68901b0470 msgr2=0x7f68901b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:35.065 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:35.052 7f6894839700 1 --2- 172.21.2.102:0/2535582973 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f68901b0470 0x7f68901b0870 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 rx=0x7f6880006700 tx=0x7f6880009b00).stop 2021-02-10T11:21:35.065 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:35.053 7f6894839700 1 -- 172.21.2.102:0/2535582973 shutdown_connections 2021-02-10T11:21:35.065 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:35.053 7f6894839700 1 --2- 172.21.2.102:0/2535582973 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f68900ef610 0x7f68900efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:35.065 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:35.053 7f6894839700 1 --2- 172.21.2.102:0/2535582973 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f68901b0470 0x7f68901b0870 unknown :-1 s=CLOSED pgs=950 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:35.065 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:35.053 7f6894839700 1 -- 172.21.2.102:0/2535582973 >> 172.21.2.102:0/2535582973 conn(0x7f68900ce3e0 msgr2=0x7f68900e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:35.065 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:35.053 7f6894839700 1 -- 172.21.2.102:0/2535582973 shutdown_connections 2021-02-10T11:21:35.065 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:35.053 7f6894839700 1 -- 172.21.2.102:0/2535582973 wait complete. 2021-02-10T11:21:35.067 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:35.068 7f6894839700 1 Processor -- start 2021-02-10T11:21:35.067 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:35.068 7f6894839700 1 -- start start 2021-02-10T11:21:35.067 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:35.068 7f6894839700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6890190920 0x7f68900b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:35.068 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:35.068 7f6894839700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f68900b8200 0x7f68900b8600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:35.068 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:35.068 7f6894839700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f68900c1d80 0x7f68900c2180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:35.068 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:35.068 7f6894839700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6890190d20 con 0x7f68900c1d80 2021-02-10T11:21:35.068 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:35.068 7f6894839700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68900b8c20 con 0x7f6890190920 2021-02-10T11:21:35.068 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:35.068 7f6894839700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68900c27a0 con 0x7f68900b8200 2021-02-10T11:21:35.068 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:35.068 7f688ee1d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f68900c1d80 0x7f68900c2180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:35.068 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:35.068 7f688e61c700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6890190920 0x7f68900b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:35.068 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:35.068 7f688e61c700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6890190920 0x7f68900b7be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.113:3300/0 says I am v2:172.21.2.102:58196/0 (socket says 172.21.2.102:58196) 2021-02-10T11:21:35.068 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:35.068 7f688e61c700 1 -- 172.21.2.102:0/1524395605 learned_addr learned my addr 172.21.2.102:0/1524395605 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:21:35.068 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:35.068 7f688ee1d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f68900c1d80 0x7f68900c2180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:54770/0 (socket says 172.21.2.102:54770) 2021-02-10T11:21:35.068 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:35.069 7f688de1b700 1 --2- 172.21.2.102:0/1524395605 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f68900b8200 0x7f68900b8600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:35.068 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:35.069 7f688ee1d700 1 -- 172.21.2.102:0/1524395605 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f68900b8200 msgr2=0x7f68900b8600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:35.068 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:35.069 7f688ee1d700 1 --2- 172.21.2.102:0/1524395605 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f68900b8200 0x7f68900b8600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:35.069 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:35.069 7f688ee1d700 1 -- 172.21.2.102:0/1524395605 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6890190920 msgr2=0x7f68900b7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:35.069 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:35.069 7f688ee1d700 1 --2- 172.21.2.102:0/1524395605 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6890190920 0x7f68900b7be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:35.069 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:35.069 7f688ee1d700 1 -- 172.21.2.102:0/1524395605 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68900c29c0 con 0x7f68900c1d80 2021-02-10T11:21:35.069 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:35.069 7f688de1b700 1 --2- 172.21.2.102:0/1524395605 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f68900b8200 0x7f68900b8600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:21:35.069 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:35.069 7f688ee1d700 1 --2- 172.21.2.102:0/1524395605 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f68900c1d80 0x7f68900c2180 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 rx=0x7f6888009fc0 tx=0x7f688800bc90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:35.069 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:35.069 7f687effd700 1 -- 172.21.2.102:0/1524395605 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6888003080 con 0x7f68900c1d80 2021-02-10T11:21:35.069 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:35.069 7f687effd700 1 -- 172.21.2.102:0/1524395605 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6888000f90 con 0x7f68900c1d80 2021-02-10T11:21:35.069 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:35.069 7f6894839700 1 -- 172.21.2.102:0/1524395605 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68900c29c0 con 0x7f68900c1d80 2021-02-10T11:21:35.069 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:35.069 7f687effd700 1 -- 172.21.2.102:0/1524395605 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f688800c4e0 con 0x7f68900c1d80 2021-02-10T11:21:35.069 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:35.069 7f6894839700 1 -- 172.21.2.102:0/1524395605 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68900c29c0 con 0x7f68900c1d80 2021-02-10T11:21:35.069 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:35.070 7f687effd700 1 -- 172.21.2.102:0/1524395605 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f6888001220 con 0x7f68900c1d80 2021-02-10T11:21:35.069 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:35.070 7f687effd700 1 --2- 172.21.2.102:0/1524395605 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6874018730 0x7f687401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:35.069 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:35.070 7f687effd700 1 -- 172.21.2.102:0/1524395605 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f6888032930 con 0x7f68900c1d80 2021-02-10T11:21:35.069 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:35.070 7f688e61c700 1 --2- 172.21.2.102:0/1524395605 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6874018730 0x7f687401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:35.070 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:35.070 7f688e61c700 1 --2- 172.21.2.102:0/1524395605 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6874018730 0x7f687401ac10 crc :-1 s=READY pgs=465 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:35.097 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:35.101 7f6894839700 1 -- 172.21.2.102:0/1524395605 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6890105ae0 con 0x7f68900c1d80 2021-02-10T11:21:35.099 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:35.103 7f687effd700 1 -- 172.21.2.102:0/1524395605 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f688800bdd0 con 0x7f68900c1d80 2021-02-10T11:21:35.227 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:35.232 7f6894839700 1 -- 172.21.2.102:0/1524395605 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344442", "target": ["mgr", ""]}) v1 -- 0x7f6890105ae0 con 0x7f6874018730 2021-02-10T11:21:35.239 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:35.243 7f687effd700 1 -- 172.21.2.102:0/1524395605 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f68800064f0 con 0x7f6874018730 2021-02-10T11:21:35.245 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:35.249 7f6894839700 1 -- 172.21.2.102:0/1524395605 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6874018730 msgr2=0x7f687401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:35.245 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:35.249 7f6894839700 1 --2- 172.21.2.102:0/1524395605 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6874018730 0x7f687401ac10 crc :-1 s=READY pgs=465 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:35.245 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:35.249 7f6894839700 1 -- 172.21.2.102:0/1524395605 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f68900c1d80 msgr2=0x7f68900c2180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:35.245 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:35.249 7f6894839700 1 --2- 172.21.2.102:0/1524395605 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f68900c1d80 0x7f68900c2180 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 rx=0x7f6888009fc0 tx=0x7f688800bc90).stop 2021-02-10T11:21:35.246 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:35.249 7f6894839700 1 -- 172.21.2.102:0/1524395605 shutdown_connections 2021-02-10T11:21:35.246 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:35.249 7f6894839700 1 --2- 172.21.2.102:0/1524395605 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f68900c1d80 0x7f68900c2180 unknown :-1 s=CLOSED pgs=951 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:35.246 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:35.249 7f6894839700 1 --2- 172.21.2.102:0/1524395605 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f68900b8200 0x7f68900b8600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:35.246 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:35.249 7f6894839700 1 --2- 172.21.2.102:0/1524395605 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6874018730 0x7f687401ac10 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:35.246 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:35.249 7f6894839700 1 --2- 172.21.2.102:0/1524395605 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6890190920 0x7f68900b7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:35.246 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:35.249 7f6894839700 1 -- 172.21.2.102:0/1524395605 >> 172.21.2.102:0/1524395605 conn(0x7f6890094e10 msgr2=0x7f68900a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:35.246 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:35.249 7f6894839700 1 -- 172.21.2.102:0/1524395605 shutdown_connections 2021-02-10T11:21:35.246 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:35.249 7f6894839700 1 -- 172.21.2.102:0/1524395605 wait complete. 2021-02-10T11:21:35.254 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000344443 2021-02-10T11:21:35.689 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:35.674 7ff6c418a700 1 Processor -- start 2021-02-10T11:21:35.689 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:35.674 7ff6c418a700 1 -- start start 2021-02-10T11:21:35.689 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:35.674 7ff6c418a700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7ff6bc1b0870 0x7ff6bc0f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:35.689 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:35.674 7ff6c418a700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7ff6bc0efe00 0x7ff6bc0f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:35.689 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:35.674 7ff6c418a700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6bc1b0c70 con 0x7ff6bc1b0470 2021-02-10T11:21:35.689 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:35.674 7ff6c418a700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6bc0e5ea0 con 0x7ff6bc0efe00 2021-02-10T11:21:35.689 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:35.674 7ff6c418a700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6bc0f0b50 con 0x7ff6bc1b0870 2021-02-10T11:21:35.689 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:35.675 7ff6c27a7700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7ff6bc0efe00 0x7ff6bc0f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:35.690 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:35.675 7ff6c27a7700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7ff6bc0efe00 0x7ff6bc0f0200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:54778/0 (socket says 172.21.2.102:54778) 2021-02-10T11:21:35.690 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:35.675 7ff6c27a7700 1 -- 172.21.2.102:0/1201943538 learned_addr learned my addr 172.21.2.102:0/1201943538 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:21:35.690 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:35.675 7ff6c1fa6700 1 --1- 172.21.2.102:0/1201943538 >> v1:172.21.2.102:6789/0 conn(0x7ff6bc1b0470 0x7ff6bc0ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:46998/0 (socket says 172.21.2.102:46998) 2021-02-10T11:21:35.690 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:35.675 7ff6c17a5700 1 --2- 172.21.2.102:0/1201943538 >> v2:172.21.2.113:3300/0 conn(0x7ff6bc1b0870 0x7ff6bc0f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:35.690 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:35.675 7ff6c27a7700 1 -- 172.21.2.102:0/1201943538 >> v2:172.21.2.113:3300/0 conn(0x7ff6bc1b0870 msgr2=0x7ff6bc0f20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:21:35.690 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:35.675 7ff6c27a7700 1 --2- 172.21.2.102:0/1201943538 >> v2:172.21.2.113:3300/0 conn(0x7ff6bc1b0870 0x7ff6bc0f20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:35.690 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:35.675 7ff6c27a7700 1 -- 172.21.2.102:0/1201943538 >> v1:172.21.2.102:6789/0 conn(0x7ff6bc1b0470 legacy=0x7ff6bc0ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:35.690 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:35.675 7ff6c27a7700 1 -- 172.21.2.102:0/1201943538 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6bc0f0820 con 0x7ff6bc0efe00 2021-02-10T11:21:35.690 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:35.675 7ff6c27a7700 1 --2- 172.21.2.102:0/1201943538 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff6bc0efe00 0x7ff6bc0f0200 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 rx=0x7ff6b4006f00 tx=0x7ff6b4009e90).ready entity=mon.0 client_cookie=6a4ce121116eabd2 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:35.690 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:35.675 7ff6c0fa4700 1 -- 172.21.2.102:0/1201943538 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff6b400b300 con 0x7ff6bc0efe00 2021-02-10T11:21:35.690 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:35.675 7ff6c0fa4700 1 -- 172.21.2.102:0/1201943538 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6b400b580 con 0x7ff6bc0efe00 2021-02-10T11:21:35.690 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:35.675 7ff6c0fa4700 1 -- 172.21.2.102:0/1201943538 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff6b400b9d0 con 0x7ff6bc0efe00 2021-02-10T11:21:35.691 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:35.675 7ff6c418a700 1 -- 172.21.2.102:0/1201943538 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff6bc0efe00 msgr2=0x7ff6bc0f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:35.691 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:35.675 7ff6c418a700 1 --2- 172.21.2.102:0/1201943538 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff6bc0efe00 0x7ff6bc0f0200 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 rx=0x7ff6b4006f00 tx=0x7ff6b4009e90).stop 2021-02-10T11:21:35.691 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:35.676 7ff6c418a700 1 -- 172.21.2.102:0/1201943538 shutdown_connections 2021-02-10T11:21:35.691 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:35.676 7ff6c418a700 1 --2- 172.21.2.102:0/1201943538 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff6bc0efe00 0x7ff6bc0f0200 unknown :-1 s=CLOSED pgs=953 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:35.691 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:35.676 7ff6c418a700 1 --2- 172.21.2.102:0/1201943538 >> v2:172.21.2.113:3300/0 conn(0x7ff6bc1b0870 0x7ff6bc0f20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:35.691 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:35.676 7ff6c418a700 1 -- 172.21.2.102:0/1201943538 >> 172.21.2.102:0/1201943538 conn(0x7ff6bc0ce3e0 msgr2=0x7ff6bc0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:35.691 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:35.676 7ff6c418a700 1 -- 172.21.2.102:0/1201943538 shutdown_connections 2021-02-10T11:21:35.691 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:35.676 7ff6c418a700 1 -- 172.21.2.102:0/1201943538 wait complete. 2021-02-10T11:21:35.693 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:35.694 7ff6c418a700 1 Processor -- start 2021-02-10T11:21:35.693 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:35.694 7ff6c418a700 1 -- start start 2021-02-10T11:21:35.694 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:35.694 7ff6c418a700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff6bc0a6d10 0x7ff6bc092c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:35.694 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:35.694 7ff6c418a700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff6bc105d10 0x7ff6bc106110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:35.694 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:35.694 7ff6c418a700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff6bc106730 0x7ff6bc1073e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:35.694 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:35.694 7ff6c418a700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6bc093260 con 0x7ff6bc0a6d10 2021-02-10T11:21:35.694 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:35.694 7ff6c418a700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6bc0a7110 con 0x7ff6bc105d10 2021-02-10T11:21:35.694 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:35.694 7ff6c418a700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6bc106b30 con 0x7ff6bc106730 2021-02-10T11:21:35.694 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:35.694 7ff6c1fa6700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff6bc0a6d10 0x7ff6bc092c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:35.694 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:35.694 7ff6c1fa6700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff6bc0a6d10 0x7ff6bc092c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:54780/0 (socket says 172.21.2.102:54780) 2021-02-10T11:21:35.694 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:35.694 7ff6c1fa6700 1 -- 172.21.2.102:0/738846634 learned_addr learned my addr 172.21.2.102:0/738846634 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:21:35.694 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:35.694 7ff6c27a7700 1 --2- 172.21.2.102:0/738846634 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff6bc106730 0x7ff6bc1073e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:35.694 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:35.694 7ff6c17a5700 1 --2- 172.21.2.102:0/738846634 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff6bc105d10 0x7ff6bc106110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:35.694 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:35.695 7ff6c1fa6700 1 -- 172.21.2.102:0/738846634 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff6bc106730 msgr2=0x7ff6bc1073e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:35.695 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:35.695 7ff6c1fa6700 1 --2- 172.21.2.102:0/738846634 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff6bc106730 0x7ff6bc1073e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:35.695 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:35.695 7ff6c1fa6700 1 -- 172.21.2.102:0/738846634 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff6bc105d10 msgr2=0x7ff6bc106110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:35.695 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:35.695 7ff6c1fa6700 1 --2- 172.21.2.102:0/738846634 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff6bc105d10 0x7ff6bc106110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:35.695 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:35.695 7ff6c1fa6700 1 -- 172.21.2.102:0/738846634 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6bc107a00 con 0x7ff6bc0a6d10 2021-02-10T11:21:35.695 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:35.695 7ff6c27a7700 1 --2- 172.21.2.102:0/738846634 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff6bc106730 0x7ff6bc1073e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:21:35.695 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:35.695 7ff6c1fa6700 1 --2- 172.21.2.102:0/738846634 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff6bc0a6d10 0x7ff6bc092c40 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 rx=0x7ff6b800a590 tx=0x7ff6b800a930).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:35.695 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:35.695 7ff6ae7fc700 1 -- 172.21.2.102:0/738846634 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff6b8000ef0 con 0x7ff6bc0a6d10 2021-02-10T11:21:35.695 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:35.695 7ff6ae7fc700 1 -- 172.21.2.102:0/738846634 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6b800ad20 con 0x7ff6bc0a6d10 2021-02-10T11:21:35.695 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:35.695 7ff6ae7fc700 1 -- 172.21.2.102:0/738846634 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff6b8004590 con 0x7ff6bc0a6d10 2021-02-10T11:21:35.695 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:35.695 7ff6c418a700 1 -- 172.21.2.102:0/738846634 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6bc107a00 con 0x7ff6bc0a6d10 2021-02-10T11:21:35.695 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:35.695 7ff6c418a700 1 -- 172.21.2.102:0/738846634 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6bc107a00 con 0x7ff6bc0a6d10 2021-02-10T11:21:35.695 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:35.696 7ff6ae7fc700 1 -- 172.21.2.102:0/738846634 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ff6b80043a0 con 0x7ff6bc0a6d10 2021-02-10T11:21:35.695 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:35.696 7ff6ae7fc700 1 --2- 172.21.2.102:0/738846634 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff6a4018730 0x7ff6a401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:35.696 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:35.696 7ff6ae7fc700 1 -- 172.21.2.102:0/738846634 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7ff6b8032640 con 0x7ff6bc0a6d10 2021-02-10T11:21:35.696 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:35.696 7ff6c17a5700 1 --2- 172.21.2.102:0/738846634 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff6a4018730 0x7ff6a401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:35.696 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:35.696 7ff6c17a5700 1 --2- 172.21.2.102:0/738846634 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff6a4018730 0x7ff6a401ac10 crc :-1 s=READY pgs=466 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:35.719 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:35.723 7ff6c418a700 1 -- 172.21.2.102:0/738846634 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6bc105ae0 con 0x7ff6bc0a6d10 2021-02-10T11:21:35.721 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:35.726 7ff6ae7fc700 1 -- 172.21.2.102:0/738846634 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ff6b8006f70 con 0x7ff6bc0a6d10 2021-02-10T11:21:35.853 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:35.857 7ff6c418a700 1 -- 172.21.2.102:0/738846634 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344443", "target": ["mgr", ""]}) v1 -- 0x7ff6bc105ae0 con 0x7ff6a4018730 2021-02-10T11:21:35.866 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:35.870 7ff6ae7fc700 1 -- 172.21.2.102:0/738846634 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7ff6b0006a70 con 0x7ff6a4018730 2021-02-10T11:21:35.869 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:35.873 7ff6c418a700 1 -- 172.21.2.102:0/738846634 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff6a4018730 msgr2=0x7ff6a401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:35.869 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:35.873 7ff6c418a700 1 --2- 172.21.2.102:0/738846634 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff6a4018730 0x7ff6a401ac10 crc :-1 s=READY pgs=466 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:35.870 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:35.873 7ff6c418a700 1 -- 172.21.2.102:0/738846634 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff6bc0a6d10 msgr2=0x7ff6bc092c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:35.870 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:35.873 7ff6c418a700 1 --2- 172.21.2.102:0/738846634 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff6bc0a6d10 0x7ff6bc092c40 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 rx=0x7ff6b800a590 tx=0x7ff6b800a930).stop 2021-02-10T11:21:35.870 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:35.873 7ff6c418a700 1 -- 172.21.2.102:0/738846634 shutdown_connections 2021-02-10T11:21:35.870 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:35.873 7ff6c418a700 1 --2- 172.21.2.102:0/738846634 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff6bc106730 0x7ff6bc1073e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:35.870 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:35.873 7ff6c418a700 1 --2- 172.21.2.102:0/738846634 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff6a4018730 0x7ff6a401ac10 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:35.870 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:35.873 7ff6c418a700 1 --2- 172.21.2.102:0/738846634 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff6bc105d10 0x7ff6bc106110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:35.870 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:35.873 7ff6c418a700 1 --2- 172.21.2.102:0/738846634 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff6bc0a6d10 0x7ff6bc092c40 unknown :-1 s=CLOSED pgs=954 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:35.870 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:35.873 7ff6c418a700 1 -- 172.21.2.102:0/738846634 >> 172.21.2.102:0/738846634 conn(0x7ff6bc09a9d0 msgr2=0x7ff6bc0b4b90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:35.870 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:35.873 7ff6c418a700 1 -- 172.21.2.102:0/738846634 shutdown_connections 2021-02-10T11:21:35.870 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:35.873 7ff6c418a700 1 -- 172.21.2.102:0/738846634 wait complete. 2021-02-10T11:21:35.879 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000344444 2021-02-10T11:21:36.309 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:36.296 7f86f8ce4700 1 Processor -- start 2021-02-10T11:21:36.309 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:36.296 7f86f8ce4700 1 -- start start 2021-02-10T11:21:36.309 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:36.296 7f86f8ce4700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f86f41b0870 0x7f86f40f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:36.309 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:36.296 7f86f8ce4700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f86f40efe00 0x7f86f40f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:36.309 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:36.296 7f86f8ce4700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86f41b0c70 con 0x7f86f41b0470 2021-02-10T11:21:36.309 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:36.296 7f86f8ce4700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86f40f0b50 con 0x7f86f40efe00 2021-02-10T11:21:36.310 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:36.296 7f86f8ce4700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86f40f0820 con 0x7f86f41b0870 2021-02-10T11:21:36.310 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:36.297 7f86f32e9700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f86f40efe00 0x7f86f40f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:36.310 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:36.297 7f86f32e9700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f86f40efe00 0x7f86f40f0200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:54792/0 (socket says 172.21.2.102:54792) 2021-02-10T11:21:36.310 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:36.297 7f86f32e9700 1 -- 172.21.2.102:0/3959679125 learned_addr learned my addr 172.21.2.102:0/3959679125 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:21:36.310 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:36.297 7f86f22e7700 1 --2- 172.21.2.102:0/3959679125 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f86f41b0870 0x7f86f40f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:36.310 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:36.297 7f86f22e7700 1 -- 172.21.2.102:0/3959679125 >> v2:172.21.2.102:3300/0 conn(0x7f86f40efe00 msgr2=0x7f86f40f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:21:36.310 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:36.297 7f86f22e7700 1 --2- 172.21.2.102:0/3959679125 >> v2:172.21.2.102:3300/0 conn(0x7f86f40efe00 0x7f86f40f0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:36.310 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:36.297 7f86f22e7700 1 -- 172.21.2.102:0/3959679125 >> v1:172.21.2.113:6789/0 conn(0x7f86f41b0470 legacy=0x7f86f40ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:36.310 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:36.297 7f86f22e7700 1 -- 172.21.2.102:0/3959679125 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86f40e52e0 con 0x7f86f41b0870 2021-02-10T11:21:36.310 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:36.297 7f86f32e9700 1 --2- 172.21.2.102:0/3959679125 >> v2:172.21.2.102:3300/0 conn(0x7f86f40efe00 0x7f86f40f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:21:36.310 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:36.297 7f86f22e7700 1 --2- 172.21.2.102:0/3959679125 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f86f41b0870 0x7f86f40f20e0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7f86ec0069d0 tx=0x7f86ec009e40).ready entity=mon.2 client_cookie=b778cffe5b5e2474 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:36.310 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:36.298 7f86f1ae6700 1 -- 172.21.2.102:0/3959679125 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f86ec0066c0 con 0x7f86f41b0870 2021-02-10T11:21:36.310 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:36.298 7f86f1ae6700 1 -- 172.21.2.102:0/3959679125 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86ec00b510 con 0x7f86f41b0870 2021-02-10T11:21:36.311 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:36.298 7f86f1ae6700 1 -- 172.21.2.102:0/3959679125 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f86ec00ba30 con 0x7f86f41b0870 2021-02-10T11:21:36.311 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:36.298 7f86f8ce4700 1 -- 172.21.2.102:0/3959679125 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f86f41b0870 msgr2=0x7f86f40f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:36.311 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:36.298 7f86f8ce4700 1 --2- 172.21.2.102:0/3959679125 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f86f41b0870 0x7f86f40f20e0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7f86ec0069d0 tx=0x7f86ec009e40).stop 2021-02-10T11:21:36.311 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:36.299 7f86f8ce4700 1 -- 172.21.2.102:0/3959679125 shutdown_connections 2021-02-10T11:21:36.311 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:36.299 7f86f8ce4700 1 --2- 172.21.2.102:0/3959679125 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f86f41b0870 0x7f86f40f20e0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:36.311 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:36.299 7f86f8ce4700 1 --2- 172.21.2.102:0/3959679125 >> v2:172.21.2.102:3300/0 conn(0x7f86f40efe00 0x7f86f40f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:36.311 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:36.299 7f86f8ce4700 1 -- 172.21.2.102:0/3959679125 >> 172.21.2.102:0/3959679125 conn(0x7f86f40ce3e0 msgr2=0x7f86f40e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:36.311 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:36.299 7f86f8ce4700 1 -- 172.21.2.102:0/3959679125 shutdown_connections 2021-02-10T11:21:36.311 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:36.299 7f86f8ce4700 1 -- 172.21.2.102:0/3959679125 wait complete. 2021-02-10T11:21:36.311 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:36.313 7f86f8ce4700 1 Processor -- start 2021-02-10T11:21:36.311 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:36.313 7f86f8ce4700 1 -- start start 2021-02-10T11:21:36.312 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:36.314 7f86f8ce4700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f86f40a0420 0x7f86f40a0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:36.313 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:36.314 7f86f8ce4700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f86f4189d00 0x7f86f418a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:36.313 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:36.314 7f86f8ce4700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f86f418a720 0x7f86f40b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:36.313 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:36.314 7f86f8ce4700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86f4189780 con 0x7f86f40a0420 2021-02-10T11:21:36.313 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:36.314 7f86f8ce4700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86f418ab20 con 0x7f86f418a720 2021-02-10T11:21:36.313 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:36.314 7f86f8ce4700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86f40a1140 con 0x7f86f4189d00 2021-02-10T11:21:36.313 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:36.314 7f86f2ae8700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f86f40a0420 0x7f86f40a0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:36.313 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:36.314 7f86f2ae8700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f86f40a0420 0x7f86f40a0820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:54794/0 (socket says 172.21.2.102:54794) 2021-02-10T11:21:36.313 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:36.314 7f86f2ae8700 1 -- 172.21.2.102:0/3651779669 learned_addr learned my addr 172.21.2.102:0/3651779669 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:21:36.313 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:36.314 7f86f2ae8700 1 -- 172.21.2.102:0/3651779669 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f86f4189d00 msgr2=0x7f86f418a100 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:21:36.313 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:36.314 7f86f2ae8700 1 --2- 172.21.2.102:0/3651779669 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f86f4189d00 0x7f86f418a100 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:21:36.313 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:36.314 7f86f2ae8700 1 -- 172.21.2.102:0/3651779669 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f86f418a720 msgr2=0x7f86f40b7be0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:21:36.313 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:36.314 7f86f2ae8700 1 --2- 172.21.2.102:0/3651779669 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f86f418a720 0x7f86f40b7be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:21:36.313 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:36.314 7f86f2ae8700 1 -- 172.21.2.102:0/3651779669 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86f40b8200 con 0x7f86f40a0420 2021-02-10T11:21:36.314 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:36.314 7f86f2ae8700 1 --2- 172.21.2.102:0/3651779669 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f86f40a0420 0x7f86f40a0820 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 rx=0x7f86e800b5b0 tx=0x7f86e800b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:36.314 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:36.315 7f86e37fe700 1 -- 172.21.2.102:0/3651779669 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f86e800bd40 con 0x7f86f40a0420 2021-02-10T11:21:36.314 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:36.315 7f86e37fe700 1 -- 172.21.2.102:0/3651779669 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86e8006240 con 0x7f86f40a0420 2021-02-10T11:21:36.314 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:36.315 7f86f8ce4700 1 -- 172.21.2.102:0/3651779669 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86f40b8200 con 0x7f86f40a0420 2021-02-10T11:21:36.314 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:36.315 7f86f8ce4700 1 -- 172.21.2.102:0/3651779669 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86f40b8790 con 0x7f86f40a0420 2021-02-10T11:21:36.314 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:36.315 7f86e37fe700 1 -- 172.21.2.102:0/3651779669 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f86e8005720 con 0x7f86f40a0420 2021-02-10T11:21:36.314 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:36.315 7f86e37fe700 1 -- 172.21.2.102:0/3651779669 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f86e80054a0 con 0x7f86f40a0420 2021-02-10T11:21:36.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:36.315 7f86e37fe700 1 --2- 172.21.2.102:0/3651779669 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f86d8018730 0x7f86d801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:36.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:36.315 7f86e37fe700 1 -- 172.21.2.102:0/3651779669 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f86e8032820 con 0x7f86f40a0420 2021-02-10T11:21:36.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:36.315 7f86f22e7700 1 --2- 172.21.2.102:0/3651779669 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f86d8018730 0x7f86d801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:36.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:36.316 7f86f22e7700 1 --2- 172.21.2.102:0/3651779669 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f86d8018730 0x7f86d801abc0 crc :-1 s=READY pgs=467 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:36.342 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:36.346 7f86f8ce4700 1 -- 172.21.2.102:0/3651779669 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86f40a1360 con 0x7f86f40a0420 2021-02-10T11:21:36.344 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:36.348 7f86e37fe700 1 -- 172.21.2.102:0/3651779669 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f86e8000940 con 0x7f86f40a0420 2021-02-10T11:21:36.471 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:36.475 7f86f8ce4700 1 -- 172.21.2.102:0/3651779669 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344444", "target": ["mgr", ""]}) v1 -- 0x7f86f40b9020 con 0x7f86d8018730 2021-02-10T11:21:36.482 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:36.487 7f86e37fe700 1 -- 172.21.2.102:0/3651779669 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f86ec006e60 con 0x7f86d8018730 2021-02-10T11:21:36.489 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:36.493 7f86f8ce4700 1 -- 172.21.2.102:0/3651779669 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f86d8018730 msgr2=0x7f86d801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:36.489 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:36.493 7f86f8ce4700 1 --2- 172.21.2.102:0/3651779669 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f86d8018730 0x7f86d801abc0 crc :-1 s=READY pgs=467 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:36.489 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:36.493 7f86f8ce4700 1 -- 172.21.2.102:0/3651779669 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f86f40a0420 msgr2=0x7f86f40a0820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:36.489 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:36.493 7f86f8ce4700 1 --2- 172.21.2.102:0/3651779669 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f86f40a0420 0x7f86f40a0820 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 rx=0x7f86e800b5b0 tx=0x7f86e800b950).stop 2021-02-10T11:21:36.489 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:36.493 7f86f8ce4700 1 -- 172.21.2.102:0/3651779669 shutdown_connections 2021-02-10T11:21:36.489 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:36.493 7f86f8ce4700 1 --2- 172.21.2.102:0/3651779669 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f86d8018730 0x7f86d801abc0 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:36.489 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:36.493 7f86f8ce4700 1 --2- 172.21.2.102:0/3651779669 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f86f418a720 0x7f86f40b7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:21:36.490 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:36.493 7f86f8ce4700 1 --2- 172.21.2.102:0/3651779669 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f86f4189d00 0x7f86f418a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:21:36.490 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:36.493 7f86f8ce4700 1 --2- 172.21.2.102:0/3651779669 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f86f40a0420 0x7f86f40a0820 unknown :-1 s=CLOSED pgs=955 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:36.490 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:36.493 7f86f8ce4700 1 -- 172.21.2.102:0/3651779669 >> 172.21.2.102:0/3651779669 conn(0x7f86f4094e10 msgr2=0x7f86f40a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:36.490 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:36.493 7f86f8ce4700 1 -- 172.21.2.102:0/3651779669 shutdown_connections 2021-02-10T11:21:36.490 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:36.493 7f86f8ce4700 1 -- 172.21.2.102:0/3651779669 wait complete. 2021-02-10T11:21:36.498 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000344445 2021-02-10T11:21:36.930 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:36.916 7f2bbd0be700 1 Processor -- start 2021-02-10T11:21:36.930 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:36.916 7f2bbd0be700 1 -- start start 2021-02-10T11:21:36.930 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:36.916 7f2bbd0be700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f2bb81b0470 0x7f2bb81b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:36.930 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:36.916 7f2bbd0be700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bb80f0b50 con 0x7f2bb80ef610 2021-02-10T11:21:36.930 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:36.916 7f2bbd0be700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bb80f24e0 con 0x7f2bb80f20e0 2021-02-10T11:21:36.930 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:36.916 7f2bbd0be700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2bb80e5ea0 con 0x7f2bb81b0470 2021-02-10T11:21:36.931 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:36.917 7f2bb6d9d700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f2bb81b0470 0x7f2bb81b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:36.931 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:36.917 7f2bb6d9d700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f2bb81b0470 0x7f2bb81b0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:54802/0 (socket says 172.21.2.102:54802) 2021-02-10T11:21:36.931 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:36.917 7f2bb6d9d700 1 -- 172.21.2.102:0/3585033414 learned_addr learned my addr 172.21.2.102:0/3585033414 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:21:36.931 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:36.917 7f2bb6d9d700 1 -- 172.21.2.102:0/3585033414 >> v1:172.21.2.113:6789/0 conn(0x7f2bb80f20e0 legacy=0x7f2bb80f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:21:36.931 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:36.917 7f2bb6d9d700 1 -- 172.21.2.102:0/3585033414 >> v1:172.21.2.102:6789/0 conn(0x7f2bb80ef610 legacy=0x7f2bb80efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:36.931 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:36.917 7f2bb6d9d700 1 -- 172.21.2.102:0/3585033414 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bb80e52e0 con 0x7f2bb81b0470 2021-02-10T11:21:36.931 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:36.917 7f2bb6d9d700 1 --2- 172.21.2.102:0/3585033414 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2bb81b0470 0x7f2bb81b0870 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 rx=0x7f2ba8006700 tx=0x7f2ba8009b80).ready entity=mon.0 client_cookie=6baf060fb0b365ae server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:36.931 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:36.917 7f2bb5d9b700 1 -- 172.21.2.102:0/3585033414 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2ba800c0c0 con 0x7f2bb81b0470 2021-02-10T11:21:36.931 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:36.917 7f2bb5d9b700 1 -- 172.21.2.102:0/3585033414 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ba800c400 con 0x7f2bb81b0470 2021-02-10T11:21:36.931 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:36.917 7f2bb5d9b700 1 -- 172.21.2.102:0/3585033414 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2ba800c8e0 con 0x7f2bb81b0470 2021-02-10T11:21:36.931 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:36.917 7f2bbd0be700 1 -- 172.21.2.102:0/3585033414 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2bb81b0470 msgr2=0x7f2bb81b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:36.931 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:36.917 7f2bbd0be700 1 --2- 172.21.2.102:0/3585033414 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2bb81b0470 0x7f2bb81b0870 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 rx=0x7f2ba8006700 tx=0x7f2ba8009b80).stop 2021-02-10T11:21:36.931 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:36.918 7f2bbd0be700 1 -- 172.21.2.102:0/3585033414 shutdown_connections 2021-02-10T11:21:36.932 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:36.918 7f2bbd0be700 1 --2- 172.21.2.102:0/3585033414 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2bb81b0470 0x7f2bb81b0870 unknown :-1 s=CLOSED pgs=957 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:36.932 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:36.918 7f2bbd0be700 1 -- 172.21.2.102:0/3585033414 >> 172.21.2.102:0/3585033414 conn(0x7f2bb80ce3e0 msgr2=0x7f2bb80e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:36.932 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:36.918 7f2bbd0be700 1 -- 172.21.2.102:0/3585033414 shutdown_connections 2021-02-10T11:21:36.932 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:36.918 7f2bbd0be700 1 -- 172.21.2.102:0/3585033414 wait complete. 2021-02-10T11:21:36.932 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:36.932 7f2bbd0be700 1 Processor -- start 2021-02-10T11:21:36.932 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:36.933 7f2bbd0be700 1 -- start start 2021-02-10T11:21:36.932 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:36.933 7f2bbd0be700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2bb80a3b50 0x7f2bb81b3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:36.932 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:36.933 7f2bbd0be700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2bb81c25b0 0x7f2bb81b3d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:36.932 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:36.933 7f2bbd0be700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2bb81b4350 0x7f2bb81b18a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:36.932 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:36.933 7f2bbd0be700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bb81b3190 con 0x7f2bb81b4350 2021-02-10T11:21:36.932 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:36.933 7f2bbd0be700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bb8087750 con 0x7f2bb80a3b50 2021-02-10T11:21:36.932 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:36.933 7f2bbd0be700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2bb80a6bf0 con 0x7f2bb81c25b0 2021-02-10T11:21:36.932 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:36.933 7f2bb759e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2bb81b4350 0x7f2bb81b18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:36.932 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:36.933 7f2bb759e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2bb81b4350 0x7f2bb81b18a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:54808/0 (socket says 172.21.2.102:54808) 2021-02-10T11:21:36.933 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:36.933 7f2bb759e700 1 -- 172.21.2.102:0/1665899255 learned_addr learned my addr 172.21.2.102:0/1665899255 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:21:36.933 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:36.933 7f2bb6d9d700 1 --2- 172.21.2.102:0/1665899255 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2bb80a3b50 0x7f2bb81b3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:36.933 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:36.933 7f2bb659c700 1 --2- 172.21.2.102:0/1665899255 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2bb81c25b0 0x7f2bb81b3d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:36.933 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:36.934 7f2bb759e700 1 -- 172.21.2.102:0/1665899255 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2bb81c25b0 msgr2=0x7f2bb81b3d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:36.933 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:36.934 7f2bb759e700 1 --2- 172.21.2.102:0/1665899255 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2bb81c25b0 0x7f2bb81b3d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:36.933 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:36.934 7f2bb759e700 1 -- 172.21.2.102:0/1665899255 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2bb80a3b50 msgr2=0x7f2bb81b3710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:36.933 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:36.934 7f2bb759e700 1 --2- 172.21.2.102:0/1665899255 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2bb80a3b50 0x7f2bb81b3710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:36.933 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:36.934 7f2bb759e700 1 -- 172.21.2.102:0/1665899255 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bb81b1ec0 con 0x7f2bb81b4350 2021-02-10T11:21:36.933 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:36.934 7f2bb6d9d700 1 --2- 172.21.2.102:0/1665899255 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2bb80a3b50 0x7f2bb81b3710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:21:36.933 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:36.934 7f2bb759e700 1 --2- 172.21.2.102:0/1665899255 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2bb81b4350 0x7f2bb81b18a0 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 rx=0x7f2bb000b8e0 tx=0x7f2bb000bc80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:36.933 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:36.934 7f2baf7fe700 1 -- 172.21.2.102:0/1665899255 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2bb0000f90 con 0x7f2bb81b4350 2021-02-10T11:21:36.933 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:36.934 7f2baf7fe700 1 -- 172.21.2.102:0/1665899255 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bb000c2f0 con 0x7f2bb81b4350 2021-02-10T11:21:36.934 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:36.934 7f2baf7fe700 1 -- 172.21.2.102:0/1665899255 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2bb000c7d0 con 0x7f2bb81b4350 2021-02-10T11:21:36.934 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:36.934 7f2bbd0be700 1 -- 172.21.2.102:0/1665899255 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2bb81b1ec0 con 0x7f2bb81b4350 2021-02-10T11:21:36.934 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:36.934 7f2bbd0be700 1 -- 172.21.2.102:0/1665899255 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2bb81b1ec0 con 0x7f2bb81b4350 2021-02-10T11:21:36.935 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:36.935 7f2baf7fe700 1 -- 172.21.2.102:0/1665899255 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f2bb0001220 con 0x7f2bb81b4350 2021-02-10T11:21:36.935 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:36.935 7f2baf7fe700 1 --2- 172.21.2.102:0/1665899255 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2b9c0186d0 0x7f2b9c01abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:36.935 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:36.935 7f2baf7fe700 1 -- 172.21.2.102:0/1665899255 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f2bb0032a00 con 0x7f2bb81b4350 2021-02-10T11:21:36.935 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:36.935 7f2bb6d9d700 1 --2- 172.21.2.102:0/1665899255 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2b9c0186d0 0x7f2b9c01abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:36.935 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:36.935 7f2bb6d9d700 1 --2- 172.21.2.102:0/1665899255 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2b9c0186d0 0x7f2b9c01abd0 crc :-1 s=READY pgs=468 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:36.962 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:36.966 7f2bbd0be700 1 -- 172.21.2.102:0/1665899255 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2bb807e6b0 con 0x7f2bb81b4350 2021-02-10T11:21:36.969 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:36.969 7f2baf7fe700 1 -- 172.21.2.102:0/1665899255 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f2bb000bdc0 con 0x7f2bb81b4350 2021-02-10T11:21:37.092 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:37.096 7f2bbd0be700 1 -- 172.21.2.102:0/1665899255 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344445", "target": ["mgr", ""]}) v1 -- 0x7f2bb80ce3e0 con 0x7f2b9c0186d0 2021-02-10T11:21:37.104 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:37.108 7f2baf7fe700 1 -- 172.21.2.102:0/1665899255 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f2ba8006c70 con 0x7f2b9c0186d0 2021-02-10T11:21:37.109 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:37.113 7f2bbd0be700 1 -- 172.21.2.102:0/1665899255 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2b9c0186d0 msgr2=0x7f2b9c01abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:37.109 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:37.113 7f2bbd0be700 1 --2- 172.21.2.102:0/1665899255 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2b9c0186d0 0x7f2b9c01abd0 crc :-1 s=READY pgs=468 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:37.109 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:37.113 7f2bbd0be700 1 -- 172.21.2.102:0/1665899255 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2bb81b4350 msgr2=0x7f2bb81b18a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:37.109 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:37.113 7f2bbd0be700 1 --2- 172.21.2.102:0/1665899255 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2bb81b4350 0x7f2bb81b18a0 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 rx=0x7f2bb000b8e0 tx=0x7f2bb000bc80).stop 2021-02-10T11:21:37.109 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:37.113 7f2bbd0be700 1 -- 172.21.2.102:0/1665899255 shutdown_connections 2021-02-10T11:21:37.109 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:37.113 7f2bbd0be700 1 --2- 172.21.2.102:0/1665899255 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2bb81b4350 0x7f2bb81b18a0 unknown :-1 s=CLOSED pgs=958 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:37.109 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:37.113 7f2bbd0be700 1 --2- 172.21.2.102:0/1665899255 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2bb81c25b0 0x7f2bb81b3d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:37.110 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:37.113 7f2bbd0be700 1 --2- 172.21.2.102:0/1665899255 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2b9c0186d0 0x7f2b9c01abd0 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:37.110 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:37.113 7f2bbd0be700 1 --2- 172.21.2.102:0/1665899255 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2bb80a3b50 0x7f2bb81b3710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:37.110 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:37.113 7f2bbd0be700 1 -- 172.21.2.102:0/1665899255 >> 172.21.2.102:0/1665899255 conn(0x7f2bb80886d0 msgr2=0x7f2bb8081c40 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:37.110 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:37.113 7f2bbd0be700 1 -- 172.21.2.102:0/1665899255 shutdown_connections 2021-02-10T11:21:37.110 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:37.113 7f2bbd0be700 1 -- 172.21.2.102:0/1665899255 wait complete. 2021-02-10T11:21:37.119 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000344446 2021-02-10T11:21:37.669 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:37.537 7fb8a109a700 1 Processor -- start 2021-02-10T11:21:37.670 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:37.537 7fb8a109a700 1 -- start start 2021-02-10T11:21:37.670 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:37.538 7fb8a109a700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fb89c1b0470 0x7fb89c1b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:37.670 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:37.538 7fb8a109a700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb89c0f0b50 con 0x7fb89c0ef610 2021-02-10T11:21:37.670 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:37.538 7fb8a109a700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb89c0f24e0 con 0x7fb89c0f20e0 2021-02-10T11:21:37.670 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:37.538 7fb8a109a700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb89c0e5ea0 con 0x7fb89c1b0470 2021-02-10T11:21:37.670 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:37.538 7fb89ad9d700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fb89c1b0470 0x7fb89c1b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:37.670 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:37.538 7fb89ad9d700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fb89c1b0470 0x7fb89c1b0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:54816/0 (socket says 172.21.2.102:54816) 2021-02-10T11:21:37.670 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:37.538 7fb89ad9d700 1 -- 172.21.2.102:0/4274568694 learned_addr learned my addr 172.21.2.102:0/4274568694 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:21:37.670 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:37.538 7fb89a59c700 1 --1- 172.21.2.102:0/4274568694 >> v1:172.21.2.102:6789/0 conn(0x7fb89c0ef610 0x7fb89c0efa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:47040/0 (socket says 172.21.2.102:47040) 2021-02-10T11:21:37.670 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:37.539 7fb89ad9d700 1 -- 172.21.2.102:0/4274568694 >> v1:172.21.2.113:6789/0 conn(0x7fb89c0f20e0 legacy=0x7fb89c0f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:37.671 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:37.539 7fb89ad9d700 1 -- 172.21.2.102:0/4274568694 >> v1:172.21.2.102:6789/0 conn(0x7fb89c0ef610 legacy=0x7fb89c0efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:37.671 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:37.539 7fb89ad9d700 1 -- 172.21.2.102:0/4274568694 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb89c0e52e0 con 0x7fb89c1b0470 2021-02-10T11:21:37.671 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:37.539 7fb89ad9d700 1 --2- 172.21.2.102:0/4274568694 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb89c1b0470 0x7fb89c1b0870 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 rx=0x7fb894006a80 tx=0x7fb894009f00).ready entity=mon.0 client_cookie=80e6129ffe0b9203 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:37.671 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:37.539 7fb899d9b700 1 -- 172.21.2.102:0/4274568694 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb89400b5c0 con 0x7fb89c1b0470 2021-02-10T11:21:37.671 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:37.539 7fb899d9b700 1 -- 172.21.2.102:0/4274568694 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb89400b8c0 con 0x7fb89c1b0470 2021-02-10T11:21:37.671 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:37.539 7fb899d9b700 1 -- 172.21.2.102:0/4274568694 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb89400bd20 con 0x7fb89c1b0470 2021-02-10T11:21:37.671 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:37.539 7fb8a109a700 1 -- 172.21.2.102:0/4274568694 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb89c1b0470 msgr2=0x7fb89c1b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:37.671 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:37.539 7fb8a109a700 1 --2- 172.21.2.102:0/4274568694 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb89c1b0470 0x7fb89c1b0870 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 rx=0x7fb894006a80 tx=0x7fb894009f00).stop 2021-02-10T11:21:37.671 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:37.540 7fb8a109a700 1 -- 172.21.2.102:0/4274568694 shutdown_connections 2021-02-10T11:21:37.671 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:37.540 7fb8a109a700 1 --2- 172.21.2.102:0/4274568694 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb89c1b0470 0x7fb89c1b0870 unknown :-1 s=CLOSED pgs=960 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:37.671 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:37.540 7fb8a109a700 1 -- 172.21.2.102:0/4274568694 >> 172.21.2.102:0/4274568694 conn(0x7fb89c0ce3e0 msgr2=0x7fb89c0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:37.671 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:37.540 7fb8a109a700 1 -- 172.21.2.102:0/4274568694 shutdown_connections 2021-02-10T11:21:37.671 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:37.540 7fb8a109a700 1 -- 172.21.2.102:0/4274568694 wait complete. 2021-02-10T11:21:37.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:37.674 7fb8a109a700 1 Processor -- start 2021-02-10T11:21:37.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:37.674 7fb8a109a700 1 -- start start 2021-02-10T11:21:37.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:37.675 7fb8a109a700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb89c0a3b50 0x7fb89c1b3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:37.674 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:37.675 7fb8a109a700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb89c1c25b0 0x7fb89c1b3d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:37.674 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:37.675 7fb8a109a700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb89c1b4350 0x7fb89c1b18a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:37.674 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:37.675 7fb8a109a700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb89c1b3190 con 0x7fb89c0a3b50 2021-02-10T11:21:37.674 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:37.675 7fb8a109a700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb89c087750 con 0x7fb89c1c25b0 2021-02-10T11:21:37.674 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:37.675 7fb8a109a700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb89c0a6bf0 con 0x7fb89c1b4350 2021-02-10T11:21:37.674 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:37.675 7fb89ad9d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb89c0a3b50 0x7fb89c1b3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:37.674 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:37.675 7fb89ad9d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb89c0a3b50 0x7fb89c1b3710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:54822/0 (socket says 172.21.2.102:54822) 2021-02-10T11:21:37.674 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:37.675 7fb89ad9d700 1 -- 172.21.2.102:0/2019907161 learned_addr learned my addr 172.21.2.102:0/2019907161 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:21:37.674 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:37.675 7fb89a59c700 1 --2- 172.21.2.102:0/2019907161 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb89c1c25b0 0x7fb89c1b3d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:37.674 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:37.675 7fb89b59e700 1 --2- 172.21.2.102:0/2019907161 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb89c1b4350 0x7fb89c1b18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:37.674 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:37.675 7fb89ad9d700 1 -- 172.21.2.102:0/2019907161 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb89c1b4350 msgr2=0x7fb89c1b18a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:37.674 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:37.675 7fb89ad9d700 1 --2- 172.21.2.102:0/2019907161 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb89c1b4350 0x7fb89c1b18a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:37.674 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:37.675 7fb89ad9d700 1 -- 172.21.2.102:0/2019907161 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb89c1c25b0 msgr2=0x7fb89c1b3d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:37.675 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:37.675 7fb89ad9d700 1 --2- 172.21.2.102:0/2019907161 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb89c1c25b0 0x7fb89c1b3d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:37.675 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:37.676 7fb89ad9d700 1 -- 172.21.2.102:0/2019907161 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb89c1b1ec0 con 0x7fb89c0a3b50 2021-02-10T11:21:37.675 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:37.676 7fb89b59e700 1 --2- 172.21.2.102:0/2019907161 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb89c1b4350 0x7fb89c1b18a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:21:37.675 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:37.676 7fb89ad9d700 1 --2- 172.21.2.102:0/2019907161 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb89c0a3b50 0x7fb89c1b3710 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 rx=0x7fb894009450 tx=0x7fb89400bd20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:37.675 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:37.676 7fb88b7fe700 1 -- 172.21.2.102:0/2019907161 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb89400c2f0 con 0x7fb89c0a3b50 2021-02-10T11:21:37.675 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:37.676 7fb88b7fe700 1 -- 172.21.2.102:0/2019907161 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb89400be60 con 0x7fb89c0a3b50 2021-02-10T11:21:37.675 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:37.676 7fb88b7fe700 1 -- 172.21.2.102:0/2019907161 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb89400c6e0 con 0x7fb89c0a3b50 2021-02-10T11:21:37.675 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:37.676 7fb8a109a700 1 -- 172.21.2.102:0/2019907161 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb89c1b1ec0 con 0x7fb89c0a3b50 2021-02-10T11:21:37.675 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:37.676 7fb8a109a700 1 -- 172.21.2.102:0/2019907161 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb89c1b2450 con 0x7fb89c0a3b50 2021-02-10T11:21:37.675 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:37.677 7fb88b7fe700 1 -- 172.21.2.102:0/2019907161 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fb894008020 con 0x7fb89c0a3b50 2021-02-10T11:21:37.675 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:37.677 7fb88b7fe700 1 --2- 172.21.2.102:0/2019907161 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb8800186c0 0x7fb88001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:37.675 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:37.677 7fb88b7fe700 1 -- 172.21.2.102:0/2019907161 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fb894032b70 con 0x7fb89c0a3b50 2021-02-10T11:21:37.675 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:37.677 7fb89a59c700 1 --2- 172.21.2.102:0/2019907161 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb8800186c0 0x7fb88001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:37.676 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:37.677 7fb89a59c700 1 --2- 172.21.2.102:0/2019907161 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb8800186c0 0x7fb88001abc0 crc :-1 s=READY pgs=469 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:37.683 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:37.687 7fb8a109a700 1 -- 172.21.2.102:0/2019907161 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb89c07e6b0 con 0x7fb89c0a3b50 2021-02-10T11:21:37.686 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:37.690 7fb88b7fe700 1 -- 172.21.2.102:0/2019907161 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fb894000d30 con 0x7fb89c0a3b50 2021-02-10T11:21:37.817 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:37.821 7fb8a109a700 1 -- 172.21.2.102:0/2019907161 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344446", "target": ["mgr", ""]}) v1 -- 0x7fb89c0ce3e0 con 0x7fb8800186c0 2021-02-10T11:21:37.828 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:37.832 7fb88b7fe700 1 -- 172.21.2.102:0/2019907161 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fb890000e20 con 0x7fb8800186c0 2021-02-10T11:21:37.834 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:37.838 7fb8a109a700 1 -- 172.21.2.102:0/2019907161 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb8800186c0 msgr2=0x7fb88001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:37.834 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:37.838 7fb8a109a700 1 --2- 172.21.2.102:0/2019907161 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb8800186c0 0x7fb88001abc0 crc :-1 s=READY pgs=469 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:37.834 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:37.838 7fb8a109a700 1 -- 172.21.2.102:0/2019907161 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb89c0a3b50 msgr2=0x7fb89c1b3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:37.834 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:37.838 7fb8a109a700 1 --2- 172.21.2.102:0/2019907161 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb89c0a3b50 0x7fb89c1b3710 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 rx=0x7fb894009450 tx=0x7fb89400bd20).stop 2021-02-10T11:21:37.835 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:37.838 7fb8a109a700 1 -- 172.21.2.102:0/2019907161 shutdown_connections 2021-02-10T11:21:37.835 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:37.838 7fb8a109a700 1 --2- 172.21.2.102:0/2019907161 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb89c1b4350 0x7fb89c1b18a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:37.835 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:37.838 7fb8a109a700 1 --2- 172.21.2.102:0/2019907161 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb8800186c0 0x7fb88001abc0 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:37.835 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:37.838 7fb8a109a700 1 --2- 172.21.2.102:0/2019907161 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb89c1c25b0 0x7fb89c1b3d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:37.835 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:37.838 7fb8a109a700 1 --2- 172.21.2.102:0/2019907161 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb89c0a3b50 0x7fb89c1b3710 unknown :-1 s=CLOSED pgs=961 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:37.835 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:37.838 7fb8a109a700 1 -- 172.21.2.102:0/2019907161 >> 172.21.2.102:0/2019907161 conn(0x7fb89c0886d0 msgr2=0x7fb89c081c40 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:37.835 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:37.838 7fb8a109a700 1 -- 172.21.2.102:0/2019907161 shutdown_connections 2021-02-10T11:21:37.835 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:37.838 7fb8a109a700 1 -- 172.21.2.102:0/2019907161 wait complete. 2021-02-10T11:21:37.843 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000344447 2021-02-10T11:21:38.273 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:38.261 7fa749872700 1 Processor -- start 2021-02-10T11:21:38.273 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:38.261 7fa749872700 1 -- start start 2021-02-10T11:21:38.273 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:38.261 7fa749872700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fa7440f05f0 0x7fa7440f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:38.273 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:38.261 7fa749872700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7441b0c70 con 0x7fa7441b0870 2021-02-10T11:21:38.273 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:38.261 7fa749872700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7440f0b50 con 0x7fa7441b0470 2021-02-10T11:21:38.273 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:38.261 7fa749872700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7440e5ea0 con 0x7fa7440f05f0 2021-02-10T11:21:38.274 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:38.262 7fa743d75700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fa7440f05f0 0x7fa7440f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:38.274 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:38.262 7fa742d73700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fa7441b0870 0x7fa7440efe00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:47052/0 (socket says 172.21.2.102:47052) 2021-02-10T11:21:38.274 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:38.262 7fa742d73700 1 -- 172.21.2.102:0/3393496933 learned_addr learned my addr 172.21.2.102:0/3393496933 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:21:38.274 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:38.262 7fa743d75700 1 -- 172.21.2.102:0/3393496933 >> v1:172.21.2.113:6789/0 conn(0x7fa7441b0470 legacy=0x7fa7440ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:21:38.274 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:38.262 7fa743d75700 1 -- 172.21.2.102:0/3393496933 >> v1:172.21.2.102:6789/0 conn(0x7fa7441b0870 legacy=0x7fa7440efe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:38.274 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:38.262 7fa743d75700 1 -- 172.21.2.102:0/3393496933 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7440e52e0 con 0x7fa7440f05f0 2021-02-10T11:21:38.274 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:38.262 7fa743d75700 1 --2- 172.21.2.102:0/3393496933 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa7440f05f0 0x7fa7440f20e0 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 rx=0x7fa73c006e30 tx=0x7fa73c009e60).ready entity=mon.0 client_cookie=7b262d82a254f76f server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:38.274 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:38.263 7fa742572700 1 -- 172.21.2.102:0/3393496933 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa73c006a50 con 0x7fa7440f05f0 2021-02-10T11:21:38.274 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:38.263 7fa742572700 1 -- 172.21.2.102:0/3393496933 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa73c00b300 con 0x7fa7440f05f0 2021-02-10T11:21:38.274 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:38.263 7fa742572700 1 -- 172.21.2.102:0/3393496933 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa73c00b7e0 con 0x7fa7440f05f0 2021-02-10T11:21:38.274 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:38.263 7fa749872700 1 -- 172.21.2.102:0/3393496933 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa7440f05f0 msgr2=0x7fa7440f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:38.274 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:38.263 7fa749872700 1 --2- 172.21.2.102:0/3393496933 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa7440f05f0 0x7fa7440f20e0 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 rx=0x7fa73c006e30 tx=0x7fa73c009e60).stop 2021-02-10T11:21:38.274 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:38.263 7fa749872700 1 -- 172.21.2.102:0/3393496933 shutdown_connections 2021-02-10T11:21:38.275 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:38.263 7fa749872700 1 --2- 172.21.2.102:0/3393496933 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa7440f05f0 0x7fa7440f20e0 unknown :-1 s=CLOSED pgs=963 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:38.275 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:38.263 7fa749872700 1 -- 172.21.2.102:0/3393496933 >> 172.21.2.102:0/3393496933 conn(0x7fa7440ce3e0 msgr2=0x7fa7440e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:38.275 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:38.263 7fa749872700 1 -- 172.21.2.102:0/3393496933 shutdown_connections 2021-02-10T11:21:38.275 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:38.263 7fa749872700 1 -- 172.21.2.102:0/3393496933 wait complete. 2021-02-10T11:21:38.276 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:38.278 7fa749872700 1 Processor -- start 2021-02-10T11:21:38.276 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:38.278 7fa749872700 1 -- start start 2021-02-10T11:21:38.276 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:38.278 7fa749872700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa744087510 0x7fa7440aedc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:38.276 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:38.278 7fa749872700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa7441c68f0 0x7fa7441b1f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:38.276 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:38.278 7fa749872700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa74408c4d0 0x7fa7441b18a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:38.277 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:38.278 7fa749872700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7440a2d50 con 0x7fa7441c68f0 2021-02-10T11:21:38.277 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:38.278 7fa749872700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7440a2f70 con 0x7fa74408c4d0 2021-02-10T11:21:38.277 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:38.278 7fa749872700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7440a3190 con 0x7fa744087510 2021-02-10T11:21:38.277 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:38.278 7fa742d73700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa7441c68f0 0x7fa7441b1f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:38.277 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:38.278 7fa742d73700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa7441c68f0 0x7fa7441b1f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:54834/0 (socket says 172.21.2.102:54834) 2021-02-10T11:21:38.277 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:38.278 7fa742d73700 1 -- 172.21.2.102:0/3796476772 learned_addr learned my addr 172.21.2.102:0/3796476772 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:21:38.277 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:38.279 7fa742d73700 1 -- 172.21.2.102:0/3796476772 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa744087510 msgr2=0x7fa7440aedc0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:21:38.277 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:38.279 7fa742d73700 1 --2- 172.21.2.102:0/3796476772 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa744087510 0x7fa7440aedc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:21:38.277 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:38.279 7fa742d73700 1 -- 172.21.2.102:0/3796476772 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa74408c4d0 msgr2=0x7fa7441b18a0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:21:38.277 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:38.279 7fa742d73700 1 --2- 172.21.2.102:0/3796476772 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa74408c4d0 0x7fa7441b18a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:21:38.277 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:38.279 7fa742d73700 1 -- 172.21.2.102:0/3796476772 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7440a33b0 con 0x7fa7441c68f0 2021-02-10T11:21:38.277 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:38.279 7fa742d73700 1 --2- 172.21.2.102:0/3796476772 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa7441c68f0 0x7fa7441b1f60 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 rx=0x7fa734001410 tx=0x7fa73400a950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:38.277 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:38.279 7fa73bfff700 1 -- 172.21.2.102:0/3796476772 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa73400ad40 con 0x7fa7441c68f0 2021-02-10T11:21:38.277 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:38.279 7fa73bfff700 1 -- 172.21.2.102:0/3796476772 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa734000c90 con 0x7fa7441c68f0 2021-02-10T11:21:38.278 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:38.279 7fa73bfff700 1 -- 172.21.2.102:0/3796476772 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa734004590 con 0x7fa7441c68f0 2021-02-10T11:21:38.278 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:38.279 7fa749872700 1 -- 172.21.2.102:0/3796476772 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7440a33b0 con 0x7fa7441c68f0 2021-02-10T11:21:38.278 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:38.279 7fa749872700 1 -- 172.21.2.102:0/3796476772 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7440afca0 con 0x7fa7441c68f0 2021-02-10T11:21:38.278 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:38.280 7fa73bfff700 1 -- 172.21.2.102:0/3796476772 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa7340043a0 con 0x7fa7441c68f0 2021-02-10T11:21:38.278 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:38.280 7fa73bfff700 1 --2- 172.21.2.102:0/3796476772 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa7280186c0 0x7fa72801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:38.278 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:38.280 7fa73bfff700 1 -- 172.21.2.102:0/3796476772 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fa7340325d0 con 0x7fa7441c68f0 2021-02-10T11:21:38.278 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:38.280 7fa743574700 1 --2- 172.21.2.102:0/3796476772 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa7280186c0 0x7fa72801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:38.278 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:38.280 7fa743574700 1 --2- 172.21.2.102:0/3796476772 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa7280186c0 0x7fa72801abc0 crc :-1 s=READY pgs=470 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:38.307 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:38.311 7fa749872700 1 -- 172.21.2.102:0/3796476772 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa74407e6b0 con 0x7fa7441c68f0 2021-02-10T11:21:38.310 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:38.314 7fa73bfff700 1 -- 172.21.2.102:0/3796476772 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fa734000ef0 con 0x7fa7441c68f0 2021-02-10T11:21:38.442 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:38.447 7fa749872700 1 -- 172.21.2.102:0/3796476772 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344447", "target": ["mgr", ""]}) v1 -- 0x7fa7440afca0 con 0x7fa7280186c0 2021-02-10T11:21:38.453 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:38.457 7fa73bfff700 1 -- 172.21.2.102:0/3796476772 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fa72c000c60 con 0x7fa7280186c0 2021-02-10T11:21:38.459 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:38.463 7fa749872700 1 -- 172.21.2.102:0/3796476772 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa7280186c0 msgr2=0x7fa72801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:38.460 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:38.463 7fa749872700 1 --2- 172.21.2.102:0/3796476772 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa7280186c0 0x7fa72801abc0 crc :-1 s=READY pgs=470 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:38.460 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:38.463 7fa749872700 1 -- 172.21.2.102:0/3796476772 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa7441c68f0 msgr2=0x7fa7441b1f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:38.460 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:38.463 7fa749872700 1 --2- 172.21.2.102:0/3796476772 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa7441c68f0 0x7fa7441b1f60 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 rx=0x7fa734001410 tx=0x7fa73400a950).stop 2021-02-10T11:21:38.460 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:38.463 7fa749872700 1 -- 172.21.2.102:0/3796476772 shutdown_connections 2021-02-10T11:21:38.460 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:38.463 7fa749872700 1 --2- 172.21.2.102:0/3796476772 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa7280186c0 0x7fa72801abc0 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:38.460 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:38.463 7fa749872700 1 --2- 172.21.2.102:0/3796476772 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa74408c4d0 0x7fa7441b18a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:21:38.460 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:38.463 7fa749872700 1 --2- 172.21.2.102:0/3796476772 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa7441c68f0 0x7fa7441b1f60 unknown :-1 s=CLOSED pgs=964 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:38.460 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:38.463 7fa749872700 1 --2- 172.21.2.102:0/3796476772 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa744087510 0x7fa7440aedc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:21:38.460 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:38.463 7fa749872700 1 -- 172.21.2.102:0/3796476772 >> 172.21.2.102:0/3796476772 conn(0x7fa7440886d0 msgr2=0x7fa744081c40 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:38.460 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:38.464 7fa749872700 1 -- 172.21.2.102:0/3796476772 shutdown_connections 2021-02-10T11:21:38.460 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:38.464 7fa749872700 1 -- 172.21.2.102:0/3796476772 wait complete. 2021-02-10T11:21:38.468 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000344448 2021-02-10T11:21:38.900 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:38.887 7f16c2c9a700 1 Processor -- start 2021-02-10T11:21:38.900 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:38.887 7f16c2c9a700 1 -- start start 2021-02-10T11:21:38.901 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:38.887 7f16c2c9a700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f16bc0f05f0 0x7f16bc0f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:38.901 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:38.887 7f16c2c9a700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16bc1b0c70 con 0x7f16bc1b0870 2021-02-10T11:21:38.901 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:38.887 7f16c2c9a700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16bc0f0b50 con 0x7f16bc1b0470 2021-02-10T11:21:38.901 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:38.887 7f16c2c9a700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16bc0e5ea0 con 0x7f16bc0f05f0 2021-02-10T11:21:38.901 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:38.888 7f16bbfff700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f16bc1b0870 0x7f16bc0efe00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:47068/0 (socket says 172.21.2.102:47068) 2021-02-10T11:21:38.901 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:38.888 7f16bbfff700 1 -- 172.21.2.102:0/1345825014 learned_addr learned my addr 172.21.2.102:0/1345825014 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:21:38.901 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:38.888 7f16c12b7700 1 --2- 172.21.2.102:0/1345825014 >> v2:172.21.2.113:3300/0 conn(0x7f16bc0f05f0 0x7f16bc0f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:38.901 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:38.888 7f16bb7fe700 1 -- 172.21.2.102:0/1345825014 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2201816433 0 0) 0x7f16b00009b0 con 0x7f16bc1b0870 2021-02-10T11:21:38.901 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:38.888 7f16bb7fe700 1 -- 172.21.2.102:0/1345825014 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16a4002fe0 con 0x7f16bc1b0870 2021-02-10T11:21:38.901 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:38.888 7f16bb7fe700 1 -- 172.21.2.102:0/1345825014 <== mon.1 v1:172.21.2.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 265787294 0 0) 0x7f16ac000c60 con 0x7f16bc1b0470 2021-02-10T11:21:38.901 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:38.888 7f16bb7fe700 1 -- 172.21.2.102:0/1345825014 --> v1:172.21.2.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16a4003210 con 0x7f16bc1b0470 2021-02-10T11:21:38.901 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:38.888 7f16bb7fe700 1 -- 172.21.2.102:0/1345825014 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4045490106 0 0) 0x7f16b0001010 con 0x7f16bc1b0870 2021-02-10T11:21:38.901 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:38.888 7f16bb7fe700 1 -- 172.21.2.102:0/1345825014 >> v2:172.21.2.113:3300/0 conn(0x7f16bc0f05f0 msgr2=0x7f16bc0f20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:21:38.902 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:38.888 7f16bb7fe700 1 --2- 172.21.2.102:0/1345825014 >> v2:172.21.2.113:3300/0 conn(0x7f16bc0f05f0 0x7f16bc0f20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:38.902 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:38.888 7f16bb7fe700 1 -- 172.21.2.102:0/1345825014 >> v1:172.21.2.113:6789/0 conn(0x7f16bc1b0470 legacy=0x7f16bc0ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:38.902 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:38.888 7f16bb7fe700 1 -- 172.21.2.102:0/1345825014 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16bc0e52e0 con 0x7f16bc1b0870 2021-02-10T11:21:38.902 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:38.888 7f16bb7fe700 1 -- 172.21.2.102:0/1345825014 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7f16b0001530 con 0x7f16bc1b0870 2021-02-10T11:21:38.902 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:38.888 7f16bb7fe700 1 -- 172.21.2.102:0/1345825014 >> v1:172.21.2.102:6789/0 conn(0x7f16bc1b0870 legacy=0x7f16bc0efe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:38.902 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:38.888 7f16bb7fe700 1 --2- 172.21.2.102:0/1345825014 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f16a4003b90 0x7f16a4005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:38.902 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:38.888 7f16bb7fe700 1 --2- 172.21.2.102:0/1345825014 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f16a4006840 0x7f16a4008d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:38.902 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:38.888 7f16bb7fe700 1 --2- 172.21.2.102:0/1345825014 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f16a4009640 0x7f16a400bb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:38.902 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:38.888 7f16bb7fe700 1 -- 172.21.2.102:0/1345825014 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16a400c3b0 con 0x7f16a4009640 2021-02-10T11:21:38.902 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:38.888 7f16bb7fe700 1 -- 172.21.2.102:0/1345825014 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16a40505a0 con 0x7f16a4006840 2021-02-10T11:21:38.902 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:38.888 7f16bb7fe700 1 -- 172.21.2.102:0/1345825014 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16a40507c0 con 0x7f16a4003b90 2021-02-10T11:21:38.902 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:38.888 7f16c12b7700 1 --2- 172.21.2.102:0/1345825014 >> v2:172.21.2.113:3300/0 conn(0x7f16bc0f05f0 0x7f16bc0f20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:21:38.902 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:38.889 7f16c12b7700 1 --2- 172.21.2.102:0/1345825014 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f16a4009640 0x7f16a400bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:38.903 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:38.889 7f16c0ab6700 1 --2- 172.21.2.102:0/1345825014 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f16a4003b90 0x7f16a4005fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:38.903 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:38.889 7f16bbfff700 1 --2- 172.21.2.102:0/1345825014 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f16a4006840 0x7f16a4008d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:38.903 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:38.889 7f16c12b7700 1 -- 172.21.2.102:0/1345825014 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f16a4003b90 msgr2=0x7f16a4005fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:21:38.903 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:38.889 7f16c12b7700 1 --2- 172.21.2.102:0/1345825014 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f16a4003b90 0x7f16a4005fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:38.903 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:38.889 7f16c12b7700 1 -- 172.21.2.102:0/1345825014 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f16a4006840 msgr2=0x7f16a4008d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:21:38.903 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:38.889 7f16c12b7700 1 --2- 172.21.2.102:0/1345825014 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f16a4006840 0x7f16a4008d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:38.903 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:38.889 7f16c12b7700 1 -- 172.21.2.102:0/1345825014 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16a4050a60 con 0x7f16a4009640 2021-02-10T11:21:38.903 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:38.889 7f16c12b7700 1 -- 172.21.2.102:0/1345825014 reap_dead start 2021-02-10T11:21:38.903 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:38.889 7f16c12b7700 1 --2- 172.21.2.102:0/1345825014 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f16a4009640 0x7f16a400bb60 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 rx=0x7f16b4006cd0 tx=0x7f16b400c370).ready entity=mon.0 client_cookie=a0441984acc0df62 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:38.903 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:38.889 7f16bb7fe700 1 -- 172.21.2.102:0/1345825014 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f16b4000f40 con 0x7f16a4009640 2021-02-10T11:21:38.903 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:38.889 7f16bb7fe700 1 -- 172.21.2.102:0/1345825014 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16b4006790 con 0x7f16a4009640 2021-02-10T11:21:38.903 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:38.889 7f16bb7fe700 1 -- 172.21.2.102:0/1345825014 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f16b40072e0 con 0x7f16a4009640 2021-02-10T11:21:38.903 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:38.889 7f16c2c9a700 1 -- 172.21.2.102:0/1345825014 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f16a4009640 msgr2=0x7f16a400bb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:38.903 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:38.889 7f16c2c9a700 1 --2- 172.21.2.102:0/1345825014 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f16a4009640 0x7f16a400bb60 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 rx=0x7f16b4006cd0 tx=0x7f16b400c370).stop 2021-02-10T11:21:38.904 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:38.890 7f16c2c9a700 1 -- 172.21.2.102:0/1345825014 shutdown_connections 2021-02-10T11:21:38.904 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:38.890 7f16c2c9a700 1 --2- 172.21.2.102:0/1345825014 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f16a4009640 0x7f16a400bb60 unknown :-1 s=CLOSED pgs=966 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:38.904 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:38.890 7f16c2c9a700 1 -- 172.21.2.102:0/1345825014 >> 172.21.2.102:0/1345825014 conn(0x7f16bc0ce3e0 msgr2=0x7f16bc0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:38.904 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:38.890 7f16c2c9a700 1 -- 172.21.2.102:0/1345825014 shutdown_connections 2021-02-10T11:21:38.904 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:38.890 7f16c2c9a700 1 -- 172.21.2.102:0/1345825014 wait complete. 2021-02-10T11:21:38.906 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:38.905 7f16c2c9a700 1 Processor -- start 2021-02-10T11:21:38.906 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:38.905 7f16c2c9a700 1 -- start start 2021-02-10T11:21:38.906 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:38.905 7f16c2c9a700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f16bc09eb80 0x7f16bc190630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:38.906 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:38.905 7f16c2c9a700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f16bc09c2e0 0x7f16bc1a42b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:38.906 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:38.905 7f16c2c9a700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f16bc107030 0x7f16bc107430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:38.906 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:38.905 7f16c2c9a700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16bc190c50 con 0x7f16bc09c2e0 2021-02-10T11:21:38.906 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:38.905 7f16c2c9a700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16bc0ab800 con 0x7f16bc107030 2021-02-10T11:21:38.906 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:38.905 7f16c2c9a700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16bc1a48d0 con 0x7f16bc09eb80 2021-02-10T11:21:38.906 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:38.906 7f16bbfff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f16bc09c2e0 0x7f16bc1a42b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:38.906 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:38.906 7f16bbfff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f16bc09c2e0 0x7f16bc1a42b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:54854/0 (socket says 172.21.2.102:54854) 2021-02-10T11:21:38.906 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:38.906 7f16bbfff700 1 -- 172.21.2.102:0/3615158779 learned_addr learned my addr 172.21.2.102:0/3615158779 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:21:38.907 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:38.906 7f16c12b7700 1 --2- 172.21.2.102:0/3615158779 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f16bc107030 0x7f16bc107430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:38.907 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:38.906 7f16c0ab6700 1 --2- 172.21.2.102:0/3615158779 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f16bc09eb80 0x7f16bc190630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:38.907 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:38.906 7f16c12b7700 1 -- 172.21.2.102:0/3615158779 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f16bc09eb80 msgr2=0x7f16bc190630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:38.907 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:38.906 7f16c12b7700 1 --2- 172.21.2.102:0/3615158779 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f16bc09eb80 0x7f16bc190630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:38.907 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:38.906 7f16c12b7700 1 -- 172.21.2.102:0/3615158779 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f16bc09c2e0 msgr2=0x7f16bc1a42b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:38.907 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:38.906 7f16c12b7700 1 --2- 172.21.2.102:0/3615158779 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f16bc09c2e0 0x7f16bc1a42b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:38.907 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:38.906 7f16c12b7700 1 -- 172.21.2.102:0/3615158779 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16bc099f90 con 0x7f16bc107030 2021-02-10T11:21:38.907 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:38.906 7f16bbfff700 1 --2- 172.21.2.102:0/3615158779 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f16bc09c2e0 0x7f16bc1a42b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:21:38.907 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:38.906 7f16c12b7700 1 --2- 172.21.2.102:0/3615158779 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f16bc107030 0x7f16bc107430 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 rx=0x7f16b400b8c0 tx=0x7f16b400bc60).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:38.907 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:38.906 7f16b97fa700 1 -- 172.21.2.102:0/3615158779 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f16b400bda0 con 0x7f16bc107030 2021-02-10T11:21:38.907 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:38.906 7f16b97fa700 1 -- 172.21.2.102:0/3615158779 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16b40069c0 con 0x7f16bc107030 2021-02-10T11:21:38.907 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:38.906 7f16c2c9a700 1 -- 172.21.2.102:0/3615158779 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16bc099f90 con 0x7f16bc107030 2021-02-10T11:21:38.907 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:38.906 7f16c2c9a700 1 -- 172.21.2.102:0/3615158779 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16bc099f90 con 0x7f16bc107030 2021-02-10T11:21:38.907 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:38.907 7f16b97fa700 1 -- 172.21.2.102:0/3615158779 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f16b40022b0 con 0x7f16bc107030 2021-02-10T11:21:38.908 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:38.907 7f16b97fa700 1 -- 172.21.2.102:0/3615158779 <== mon.1 v2:172.21.2.113:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f16b40056b0 con 0x7f16bc107030 2021-02-10T11:21:38.908 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:38.907 7f16b97fa700 1 --2- 172.21.2.102:0/3615158779 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f169c0186d0 0x7f169c01abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:38.908 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:38.907 7f16b97fa700 1 -- 172.21.2.102:0/3615158779 <== mon.1 v2:172.21.2.113:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f16b4032a80 con 0x7f16bc107030 2021-02-10T11:21:38.908 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:38.907 7f16c0ab6700 1 --2- 172.21.2.102:0/3615158779 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f169c0186d0 0x7f169c01abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:38.908 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:38.907 7f16c0ab6700 1 --2- 172.21.2.102:0/3615158779 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f169c0186d0 0x7f169c01abd0 crc :-1 s=READY pgs=471 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:38.932 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:38.936 7f16c2c9a700 1 -- 172.21.2.102:0/3615158779 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16bc190c50 con 0x7f16bc107030 2021-02-10T11:21:38.936 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:38.940 7f16b97fa700 1 -- 172.21.2.102:0/3615158779 <== mon.1 v2:172.21.2.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f16b40056b0 con 0x7f16bc107030 2021-02-10T11:21:39.067 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:39.071 7f16c2c9a700 1 -- 172.21.2.102:0/3615158779 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344448", "target": ["mgr", ""]}) v1 -- 0x7f16bc190c50 con 0x7f169c0186d0 2021-02-10T11:21:39.079 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:39.083 7f16b97fa700 1 -- 172.21.2.102:0/3615158779 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f16ac007c70 con 0x7f169c0186d0 2021-02-10T11:21:39.084 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:39.088 7f16c2c9a700 1 -- 172.21.2.102:0/3615158779 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f169c0186d0 msgr2=0x7f169c01abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:39.085 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:39.088 7f16c2c9a700 1 --2- 172.21.2.102:0/3615158779 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f169c0186d0 0x7f169c01abd0 crc :-1 s=READY pgs=471 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:39.085 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:39.088 7f16c2c9a700 1 -- 172.21.2.102:0/3615158779 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f16bc107030 msgr2=0x7f16bc107430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:39.085 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:39.088 7f16c2c9a700 1 --2- 172.21.2.102:0/3615158779 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f16bc107030 0x7f16bc107430 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 rx=0x7f16b400b8c0 tx=0x7f16b400bc60).stop 2021-02-10T11:21:39.085 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:39.089 7f16c2c9a700 1 -- 172.21.2.102:0/3615158779 shutdown_connections 2021-02-10T11:21:39.085 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:39.089 7f16c2c9a700 1 --2- 172.21.2.102:0/3615158779 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f169c0186d0 0x7f169c01abd0 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:39.085 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:39.089 7f16c2c9a700 1 --2- 172.21.2.102:0/3615158779 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f16bc107030 0x7f16bc107430 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:39.085 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:39.089 7f16c2c9a700 1 --2- 172.21.2.102:0/3615158779 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f16bc09c2e0 0x7f16bc1a42b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:39.085 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:39.089 7f16c2c9a700 1 --2- 172.21.2.102:0/3615158779 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f16bc09eb80 0x7f16bc190630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:39.085 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:39.089 7f16c2c9a700 1 -- 172.21.2.102:0/3615158779 >> 172.21.2.102:0/3615158779 conn(0x7f16bc0c40d0 msgr2=0x7f16bc0bf750 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:39.085 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:39.089 7f16c2c9a700 1 -- 172.21.2.102:0/3615158779 shutdown_connections 2021-02-10T11:21:39.086 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:39.089 7f16c2c9a700 1 -- 172.21.2.102:0/3615158779 wait complete. 2021-02-10T11:21:39.094 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000344449 2021-02-10T11:21:39.527 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:39.514 7fcc5593f700 1 Processor -- start 2021-02-10T11:21:39.527 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:39.514 7fcc5593f700 1 -- start start 2021-02-10T11:21:39.527 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:39.515 7fcc5593f700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcc501b0870 0x7fcc500f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:39.527 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:39.515 7fcc5593f700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fcc500efe00 0x7fcc500f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:39.527 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:39.515 7fcc5593f700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc501b0c70 con 0x7fcc501b0470 2021-02-10T11:21:39.527 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:39.515 7fcc5593f700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc500f0b50 con 0x7fcc500efe00 2021-02-10T11:21:39.527 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:39.515 7fcc5593f700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc500f0820 con 0x7fcc501b0870 2021-02-10T11:21:39.527 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:39.515 7fcc4fd75700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fcc500efe00 0x7fcc500f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:39.527 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:39.515 7fcc4f574700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fcc501b0470 0x7fcc500ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:47084/0 (socket says 172.21.2.102:47084) 2021-02-10T11:21:39.528 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:39.515 7fcc4f574700 1 -- 172.21.2.102:0/1038491795 learned_addr learned my addr 172.21.2.102:0/1038491795 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:21:39.528 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:39.515 7fcc4ed73700 1 --2- 172.21.2.102:0/1038491795 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcc501b0870 0x7fcc500f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:39.528 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:39.515 7fcc4fd75700 1 -- 172.21.2.102:0/1038491795 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcc501b0870 msgr2=0x7fcc500f20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:21:39.528 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:39.515 7fcc4fd75700 1 --2- 172.21.2.102:0/1038491795 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcc501b0870 0x7fcc500f20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:39.528 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:39.515 7fcc4fd75700 1 -- 172.21.2.102:0/1038491795 >> v1:172.21.2.102:6789/0 conn(0x7fcc501b0470 legacy=0x7fcc500ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:39.528 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:39.515 7fcc4fd75700 1 -- 172.21.2.102:0/1038491795 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc500e52e0 con 0x7fcc500efe00 2021-02-10T11:21:39.528 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:39.515 7fcc4fd75700 1 --2- 172.21.2.102:0/1038491795 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcc500efe00 0x7fcc500f0200 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 rx=0x7fcc48006e30 tx=0x7fcc48009e60).ready entity=mon.0 client_cookie=b32d06a3d56c9c63 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:39.528 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:39.516 7fcc4e572700 1 -- 172.21.2.102:0/1038491795 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcc48006a50 con 0x7fcc500efe00 2021-02-10T11:21:39.528 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:39.516 7fcc4e572700 1 -- 172.21.2.102:0/1038491795 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc4800b300 con 0x7fcc500efe00 2021-02-10T11:21:39.528 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:39.516 7fcc4e572700 1 -- 172.21.2.102:0/1038491795 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcc4800b7a0 con 0x7fcc500efe00 2021-02-10T11:21:39.528 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:39.516 7fcc5593f700 1 -- 172.21.2.102:0/1038491795 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcc500efe00 msgr2=0x7fcc500f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:39.528 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:39.516 7fcc5593f700 1 --2- 172.21.2.102:0/1038491795 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcc500efe00 0x7fcc500f0200 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 rx=0x7fcc48006e30 tx=0x7fcc48009e60).stop 2021-02-10T11:21:39.528 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:39.516 7fcc5593f700 1 -- 172.21.2.102:0/1038491795 shutdown_connections 2021-02-10T11:21:39.529 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:39.516 7fcc5593f700 1 --2- 172.21.2.102:0/1038491795 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcc500efe00 0x7fcc500f0200 unknown :-1 s=CLOSED pgs=968 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:39.529 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:39.516 7fcc5593f700 1 --2- 172.21.2.102:0/1038491795 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcc501b0870 0x7fcc500f20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:39.529 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:39.516 7fcc5593f700 1 -- 172.21.2.102:0/1038491795 >> 172.21.2.102:0/1038491795 conn(0x7fcc500ce3e0 msgr2=0x7fcc500e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:39.529 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:39.516 7fcc5593f700 1 -- 172.21.2.102:0/1038491795 shutdown_connections 2021-02-10T11:21:39.529 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:39.516 7fcc5593f700 1 -- 172.21.2.102:0/1038491795 wait complete. 2021-02-10T11:21:39.529 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:39.531 7fcc5593f700 1 Processor -- start 2021-02-10T11:21:39.529 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:39.531 7fcc5593f700 1 -- start start 2021-02-10T11:21:39.530 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:39.531 7fcc5593f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcc500a0420 0x7fcc500a0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:39.531 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:39.531 7fcc5593f700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcc50189d00 0x7fcc5018a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:39.531 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:39.531 7fcc5593f700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fcc5018a720 0x7fcc500b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:39.531 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:39.531 7fcc5593f700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc50189780 con 0x7fcc500a0420 2021-02-10T11:21:39.531 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:39.531 7fcc5593f700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc5018ab20 con 0x7fcc5018a720 2021-02-10T11:21:39.531 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:39.531 7fcc5593f700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc500a1140 con 0x7fcc50189d00 2021-02-10T11:21:39.531 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:39.532 7fcc4f574700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcc500a0420 0x7fcc500a0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:39.531 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:39.532 7fcc4f574700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcc500a0420 0x7fcc500a0820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:54864/0 (socket says 172.21.2.102:54864) 2021-02-10T11:21:39.531 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:39.532 7fcc4f574700 1 -- 172.21.2.102:0/1721895718 learned_addr learned my addr 172.21.2.102:0/1721895718 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:21:39.531 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:39.532 7fcc4fd75700 1 --2- 172.21.2.102:0/1721895718 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fcc5018a720 0x7fcc500b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:39.531 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:39.532 7fcc4ed73700 1 --2- 172.21.2.102:0/1721895718 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcc50189d00 0x7fcc5018a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:39.531 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:39.532 7fcc4f574700 1 -- 172.21.2.102:0/1721895718 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcc50189d00 msgr2=0x7fcc5018a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:39.532 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:39.532 7fcc4f574700 1 --2- 172.21.2.102:0/1721895718 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcc50189d00 0x7fcc5018a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:39.532 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:39.532 7fcc4f574700 1 -- 172.21.2.102:0/1721895718 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fcc5018a720 msgr2=0x7fcc500b7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:39.532 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:39.532 7fcc4f574700 1 --2- 172.21.2.102:0/1721895718 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fcc5018a720 0x7fcc500b7be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:39.532 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:39.532 7fcc4f574700 1 -- 172.21.2.102:0/1721895718 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc500b8200 con 0x7fcc500a0420 2021-02-10T11:21:39.532 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:39.532 7fcc4f574700 1 --2- 172.21.2.102:0/1721895718 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcc500a0420 0x7fcc500a0820 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 rx=0x7fcc40001c60 tx=0x7fcc4000a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:39.532 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:39.532 7fcc47fff700 1 -- 172.21.2.102:0/1721895718 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcc4000acc0 con 0x7fcc500a0420 2021-02-10T11:21:39.532 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:39.532 7fcc47fff700 1 -- 172.21.2.102:0/1721895718 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc40004140 con 0x7fcc500a0420 2021-02-10T11:21:39.532 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:39.532 7fcc47fff700 1 -- 172.21.2.102:0/1721895718 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcc400045d0 con 0x7fcc500a0420 2021-02-10T11:21:39.532 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:39.532 7fcc5593f700 1 -- 172.21.2.102:0/1721895718 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc500b8200 con 0x7fcc500a0420 2021-02-10T11:21:39.533 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:39.532 7fcc5593f700 1 -- 172.21.2.102:0/1721895718 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc500b8790 con 0x7fcc500a0420 2021-02-10T11:21:39.533 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:39.533 7fcc47fff700 1 -- 172.21.2.102:0/1721895718 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fcc40004320 con 0x7fcc500a0420 2021-02-10T11:21:39.533 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:39.533 7fcc47fff700 1 --2- 172.21.2.102:0/1721895718 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcc340186c0 0x7fcc3401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:39.533 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:39.533 7fcc47fff700 1 -- 172.21.2.102:0/1721895718 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fcc40032550 con 0x7fcc500a0420 2021-02-10T11:21:39.533 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:39.533 7fcc4ed73700 1 --2- 172.21.2.102:0/1721895718 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcc340186c0 0x7fcc3401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:39.533 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:39.534 7fcc4ed73700 1 --2- 172.21.2.102:0/1721895718 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcc340186c0 0x7fcc3401abc0 crc :-1 s=READY pgs=472 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:39.560 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:39.564 7fcc5593f700 1 -- 172.21.2.102:0/1721895718 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc500a1360 con 0x7fcc500a0420 2021-02-10T11:21:39.562 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:39.566 7fcc47fff700 1 -- 172.21.2.102:0/1721895718 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fcc4002dee0 con 0x7fcc500a0420 2021-02-10T11:21:39.693 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:39.697 7fcc5593f700 1 -- 172.21.2.102:0/1721895718 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344449", "target": ["mgr", ""]}) v1 -- 0x7fcc500b9020 con 0x7fcc340186c0 2021-02-10T11:21:39.703 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:39.707 7fcc47fff700 1 -- 172.21.2.102:0/1721895718 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fcc380009b0 con 0x7fcc340186c0 2021-02-10T11:21:39.710 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:39.714 7fcc5593f700 1 -- 172.21.2.102:0/1721895718 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcc340186c0 msgr2=0x7fcc3401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:39.710 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:39.714 7fcc5593f700 1 --2- 172.21.2.102:0/1721895718 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcc340186c0 0x7fcc3401abc0 crc :-1 s=READY pgs=472 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:39.710 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:39.714 7fcc5593f700 1 -- 172.21.2.102:0/1721895718 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcc500a0420 msgr2=0x7fcc500a0820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:39.711 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:39.714 7fcc5593f700 1 --2- 172.21.2.102:0/1721895718 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcc500a0420 0x7fcc500a0820 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 rx=0x7fcc40001c60 tx=0x7fcc4000a8d0).stop 2021-02-10T11:21:39.711 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:39.714 7fcc5593f700 1 -- 172.21.2.102:0/1721895718 shutdown_connections 2021-02-10T11:21:39.711 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:39.714 7fcc5593f700 1 --2- 172.21.2.102:0/1721895718 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcc340186c0 0x7fcc3401abc0 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:39.711 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:39.714 7fcc5593f700 1 --2- 172.21.2.102:0/1721895718 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fcc5018a720 0x7fcc500b7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:39.711 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:39.714 7fcc5593f700 1 --2- 172.21.2.102:0/1721895718 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcc50189d00 0x7fcc5018a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:39.711 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:39.714 7fcc5593f700 1 --2- 172.21.2.102:0/1721895718 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcc500a0420 0x7fcc500a0820 unknown :-1 s=CLOSED pgs=969 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:39.711 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:39.714 7fcc5593f700 1 -- 172.21.2.102:0/1721895718 >> 172.21.2.102:0/1721895718 conn(0x7fcc50094e10 msgr2=0x7fcc500a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:39.711 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:39.715 7fcc5593f700 1 -- 172.21.2.102:0/1721895718 shutdown_connections 2021-02-10T11:21:39.711 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:39.715 7fcc5593f700 1 -- 172.21.2.102:0/1721895718 wait complete. 2021-02-10T11:21:39.720 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000344450 2021-02-10T11:21:40.150 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:40.138 7ff9b1fc4700 1 Processor -- start 2021-02-10T11:21:40.151 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:40.138 7ff9b1fc4700 1 -- start start 2021-02-10T11:21:40.151 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:40.139 7ff9b1fc4700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff9ac1b0470 0x7ff9ac1b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:40.151 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:40.139 7ff9b1fc4700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7ff9ac0ef610 0x7ff9ac0efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:40.151 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:40.139 7ff9b1fc4700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7ff9ac0f0030 0x7ff9ac0f0430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:40.151 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:40.139 7ff9b1fc4700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9ac0e5ea0 con 0x7ff9ac0ef610 2021-02-10T11:21:40.151 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:40.139 7ff9b1fc4700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9ac0f0b50 con 0x7ff9ac0f0030 2021-02-10T11:21:40.151 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:40.139 7ff9b1fc4700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9ac0e52e0 con 0x7ff9ac1b0470 2021-02-10T11:21:40.151 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:40.139 7ff9aaffd700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7ff9ac0ef610 0x7ff9ac0efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:40.151 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:40.139 7ff9aaffd700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7ff9ac0ef610 0x7ff9ac0efa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:54872/0 (socket says 172.21.2.102:54872) 2021-02-10T11:21:40.151 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:40.139 7ff9aaffd700 1 -- 172.21.2.102:0/1504098470 learned_addr learned my addr 172.21.2.102:0/1504098470 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:21:40.151 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:40.139 7ff9abfff700 1 --2- 172.21.2.102:0/1504098470 >> v2:172.21.2.113:3300/0 conn(0x7ff9ac0f0030 0x7ff9ac0f0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:40.152 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:40.139 7ff9ab7fe700 1 --2- 172.21.2.102:0/1504098470 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff9ac1b0470 0x7ff9ac1b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:40.152 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:40.139 7ff9aaffd700 1 -- 172.21.2.102:0/1504098470 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff9ac1b0470 msgr2=0x7ff9ac1b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:21:40.152 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:40.139 7ff9aaffd700 1 --2- 172.21.2.102:0/1504098470 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff9ac1b0470 0x7ff9ac1b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:40.152 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:40.139 7ff9aaffd700 1 -- 172.21.2.102:0/1504098470 >> v2:172.21.2.113:3300/0 conn(0x7ff9ac0f0030 msgr2=0x7ff9ac0f0430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:21:40.152 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:40.139 7ff9aaffd700 1 --2- 172.21.2.102:0/1504098470 >> v2:172.21.2.113:3300/0 conn(0x7ff9ac0f0030 0x7ff9ac0f0430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:40.152 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:40.139 7ff9aaffd700 1 -- 172.21.2.102:0/1504098470 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9ac0f10d0 con 0x7ff9ac0ef610 2021-02-10T11:21:40.152 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:40.139 7ff9aaffd700 1 --2- 172.21.2.102:0/1504098470 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff9ac0ef610 0x7ff9ac0efa10 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 rx=0x7ff9a0006700 tx=0x7ff9a0009b00).ready entity=mon.0 client_cookie=d90981c81a2c4aed server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:40.152 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:40.140 7ff9aa7fc700 1 -- 172.21.2.102:0/1504098470 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff9a000c0c0 con 0x7ff9ac0ef610 2021-02-10T11:21:40.152 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:40.140 7ff9aa7fc700 1 -- 172.21.2.102:0/1504098470 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9a000c400 con 0x7ff9ac0ef610 2021-02-10T11:21:40.152 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:40.140 7ff9aa7fc700 1 -- 172.21.2.102:0/1504098470 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff9a00009b0 con 0x7ff9ac0ef610 2021-02-10T11:21:40.152 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:40.140 7ff9b1fc4700 1 -- 172.21.2.102:0/1504098470 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff9ac0ef610 msgr2=0x7ff9ac0efa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:40.152 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:40.140 7ff9b1fc4700 1 --2- 172.21.2.102:0/1504098470 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff9ac0ef610 0x7ff9ac0efa10 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 rx=0x7ff9a0006700 tx=0x7ff9a0009b00).stop 2021-02-10T11:21:40.152 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:40.140 7ff9b1fc4700 1 -- 172.21.2.102:0/1504098470 shutdown_connections 2021-02-10T11:21:40.153 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:40.140 7ff9b1fc4700 1 --2- 172.21.2.102:0/1504098470 >> v2:172.21.2.113:3300/0 conn(0x7ff9ac0f0030 0x7ff9ac0f0430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:40.153 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:40.140 7ff9b1fc4700 1 --2- 172.21.2.102:0/1504098470 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff9ac0ef610 0x7ff9ac0efa10 unknown :-1 s=CLOSED pgs=970 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:40.153 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:40.140 7ff9b1fc4700 1 --2- 172.21.2.102:0/1504098470 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff9ac1b0470 0x7ff9ac1b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:40.153 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:40.140 7ff9b1fc4700 1 -- 172.21.2.102:0/1504098470 >> 172.21.2.102:0/1504098470 conn(0x7ff9ac0ce3e0 msgr2=0x7ff9ac0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:40.153 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:40.140 7ff9b1fc4700 1 -- 172.21.2.102:0/1504098470 shutdown_connections 2021-02-10T11:21:40.153 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:40.140 7ff9b1fc4700 1 -- 172.21.2.102:0/1504098470 wait complete. 2021-02-10T11:21:40.153 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:40.155 7ff9b1fc4700 1 Processor -- start 2021-02-10T11:21:40.154 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:40.155 7ff9b1fc4700 1 -- start start 2021-02-10T11:21:40.155 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:40.155 7ff9b1fc4700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff9ac09b930 0x7ff9ac0b7c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:40.155 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:40.155 7ff9b1fc4700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff9ac0b8410 0x7ff9ac0b8810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:40.155 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:40.155 7ff9b1fc4700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff9ac0c1f80 0x7ff9ac0c2380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:40.155 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:40.155 7ff9b1fc4700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9ac0c1a00 con 0x7ff9ac0c1f80 2021-02-10T11:21:40.155 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:40.155 7ff9b1fc4700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9ac0b8e30 con 0x7ff9ac0b8410 2021-02-10T11:21:40.155 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:40.155 7ff9b1fc4700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9ac0b9050 con 0x7ff9ac09b930 2021-02-10T11:21:40.155 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:40.156 7ff9abfff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff9ac0c1f80 0x7ff9ac0c2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:40.155 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:40.156 7ff9abfff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff9ac0c1f80 0x7ff9ac0c2380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:54876/0 (socket says 172.21.2.102:54876) 2021-02-10T11:21:40.155 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:40.156 7ff9abfff700 1 -- 172.21.2.102:0/1510382570 learned_addr learned my addr 172.21.2.102:0/1510382570 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:21:40.155 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:40.156 7ff9ab7fe700 1 --2- 172.21.2.102:0/1510382570 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff9ac09b930 0x7ff9ac0b7c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:40.155 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:40.156 7ff9aaffd700 1 --2- 172.21.2.102:0/1510382570 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff9ac0b8410 0x7ff9ac0b8810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:40.155 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:40.156 7ff9abfff700 1 -- 172.21.2.102:0/1510382570 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff9ac09b930 msgr2=0x7ff9ac0b7c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:40.156 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:40.156 7ff9abfff700 1 --2- 172.21.2.102:0/1510382570 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff9ac09b930 0x7ff9ac0b7c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:40.156 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:40.156 7ff9abfff700 1 -- 172.21.2.102:0/1510382570 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff9ac0b8410 msgr2=0x7ff9ac0b8810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:40.156 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:40.156 7ff9abfff700 1 --2- 172.21.2.102:0/1510382570 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff9ac0b8410 0x7ff9ac0b8810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:40.156 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:40.156 7ff9abfff700 1 -- 172.21.2.102:0/1510382570 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9ac0c29a0 con 0x7ff9ac0c1f80 2021-02-10T11:21:40.156 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:40.156 7ff9ab7fe700 1 --2- 172.21.2.102:0/1510382570 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff9ac09b930 0x7ff9ac0b7c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:21:40.156 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:40.156 7ff9abfff700 1 --2- 172.21.2.102:0/1510382570 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff9ac0c1f80 0x7ff9ac0c2380 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 rx=0x7ff99c006f20 tx=0x7ff99c00b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:40.156 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:40.156 7ff993fff700 1 -- 172.21.2.102:0/1510382570 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff99c006720 con 0x7ff9ac0c1f80 2021-02-10T11:21:40.156 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:40.156 7ff993fff700 1 -- 172.21.2.102:0/1510382570 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99c00be10 con 0x7ff9ac0c1f80 2021-02-10T11:21:40.156 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:40.156 7ff9b1fc4700 1 -- 172.21.2.102:0/1510382570 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9ac0c29a0 con 0x7ff9ac0c1f80 2021-02-10T11:21:40.156 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:40.157 7ff9b1fc4700 1 -- 172.21.2.102:0/1510382570 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9ac0c2f30 con 0x7ff9ac0c1f80 2021-02-10T11:21:40.156 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:40.157 7ff993fff700 1 -- 172.21.2.102:0/1510382570 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff99c005330 con 0x7ff9ac0c1f80 2021-02-10T11:21:40.156 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:40.157 7ff993fff700 1 -- 172.21.2.102:0/1510382570 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ff99c006640 con 0x7ff9ac0c1f80 2021-02-10T11:21:40.156 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:40.157 7ff993fff700 1 --2- 172.21.2.102:0/1510382570 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff98c018730 0x7ff98c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:40.156 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:40.157 7ff993fff700 1 -- 172.21.2.102:0/1510382570 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7ff99c0327b0 con 0x7ff9ac0c1f80 2021-02-10T11:21:40.157 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:40.157 7ff9ab7fe700 1 --2- 172.21.2.102:0/1510382570 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff98c018730 0x7ff98c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:40.157 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:40.158 7ff9ab7fe700 1 --2- 172.21.2.102:0/1510382570 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff98c018730 0x7ff98c01abc0 crc :-1 s=READY pgs=473 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:40.183 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:40.187 7ff9b1fc4700 1 -- 172.21.2.102:0/1510382570 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9ac105b30 con 0x7ff9ac0c1f80 2021-02-10T11:21:40.186 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:40.190 7ff993fff700 1 -- 172.21.2.102:0/1510382570 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ff99c01f4f0 con 0x7ff9ac0c1f80 2021-02-10T11:21:40.311 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:40.315 7ff9b1fc4700 1 -- 172.21.2.102:0/1510382570 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344450", "target": ["mgr", ""]}) v1 -- 0x7ff9ac0c2f30 con 0x7ff98c018730 2021-02-10T11:21:40.323 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:40.327 7ff993fff700 1 -- 172.21.2.102:0/1510382570 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7ff9a4000d30 con 0x7ff98c018730 2021-02-10T11:21:40.328 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:40.332 7ff9b1fc4700 1 -- 172.21.2.102:0/1510382570 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff98c018730 msgr2=0x7ff98c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:40.328 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:40.332 7ff9b1fc4700 1 --2- 172.21.2.102:0/1510382570 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff98c018730 0x7ff98c01abc0 crc :-1 s=READY pgs=473 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:40.328 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:40.332 7ff9b1fc4700 1 -- 172.21.2.102:0/1510382570 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff9ac0c1f80 msgr2=0x7ff9ac0c2380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:40.328 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:40.332 7ff9b1fc4700 1 --2- 172.21.2.102:0/1510382570 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff9ac0c1f80 0x7ff9ac0c2380 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 rx=0x7ff99c006f20 tx=0x7ff99c00b990).stop 2021-02-10T11:21:40.328 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:40.332 7ff9b1fc4700 1 -- 172.21.2.102:0/1510382570 shutdown_connections 2021-02-10T11:21:40.328 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:40.332 7ff9b1fc4700 1 --2- 172.21.2.102:0/1510382570 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff9ac0c1f80 0x7ff9ac0c2380 unknown :-1 s=CLOSED pgs=971 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:40.328 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:40.332 7ff9b1fc4700 1 --2- 172.21.2.102:0/1510382570 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff98c018730 0x7ff98c01abc0 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:40.329 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:40.332 7ff9b1fc4700 1 --2- 172.21.2.102:0/1510382570 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff9ac0b8410 0x7ff9ac0b8810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:40.329 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:40.332 7ff9b1fc4700 1 --2- 172.21.2.102:0/1510382570 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff9ac09b930 0x7ff9ac0b7c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:40.329 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:40.332 7ff9b1fc4700 1 -- 172.21.2.102:0/1510382570 >> 172.21.2.102:0/1510382570 conn(0x7ff9ac094e10 msgr2=0x7ff9ac0a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:40.329 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:40.332 7ff9b1fc4700 1 -- 172.21.2.102:0/1510382570 shutdown_connections 2021-02-10T11:21:40.329 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:40.332 7ff9b1fc4700 1 -- 172.21.2.102:0/1510382570 wait complete. 2021-02-10T11:21:40.337 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000344451 2021-02-10T11:21:40.771 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:40.759 7f9267a9c700 1 Processor -- start 2021-02-10T11:21:40.772 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:40.759 7f9267a9c700 1 -- start start 2021-02-10T11:21:40.772 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:40.759 7f9267a9c700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f92601b0470 0x7f92601b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:40.772 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:40.759 7f9267a9c700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f92600ef610 0x7f92600efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:40.772 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:40.760 7f9267a9c700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92600f0b50 con 0x7f92600f0030 2021-02-10T11:21:40.772 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:40.760 7f9267a9c700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92600e52e0 con 0x7f92601b0470 2021-02-10T11:21:40.772 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:40.760 7f9267a9c700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92600e5490 con 0x7f92600ef610 2021-02-10T11:21:40.772 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:40.760 7f92658b8700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f92601b0470 0x7f92601b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:40.772 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:40.760 7f92660b9700 1 --1- >> v1:172.21.2.113:6789/0 conn(0x7f92600f0030 0x7f92600f10d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.113:6789/0 says I am v1:172.21.2.102:54430/0 (socket says 172.21.2.102:54430) 2021-02-10T11:21:40.772 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:40.760 7f92660b9700 1 -- 172.21.2.102:0/2447938753 learned_addr learned my addr 172.21.2.102:0/2447938753 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:21:40.772 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:40.760 7f92650b7700 1 --2- 172.21.2.102:0/2447938753 >> v2:172.21.2.113:3300/0 conn(0x7f92600ef610 0x7f92600efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:40.772 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:40.760 7f92658b8700 1 -- 172.21.2.102:0/2447938753 >> v2:172.21.2.113:3300/0 conn(0x7f92600ef610 msgr2=0x7f92600efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:21:40.773 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:40.760 7f92658b8700 1 --2- 172.21.2.102:0/2447938753 >> v2:172.21.2.113:3300/0 conn(0x7f92600ef610 0x7f92600efa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:40.773 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:40.760 7f92658b8700 1 -- 172.21.2.102:0/2447938753 >> v1:172.21.2.113:6789/0 conn(0x7f92600f0030 legacy=0x7f92600f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:40.773 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:40.760 7f92658b8700 1 -- 172.21.2.102:0/2447938753 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92600f0430 con 0x7f92601b0470 2021-02-10T11:21:40.773 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:40.760 7f92658b8700 1 --2- 172.21.2.102:0/2447938753 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f92601b0470 0x7f92601b0870 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 rx=0x7f9254006650 tx=0x7f9254009a80).ready entity=mon.0 client_cookie=38feed21ed09af97 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:40.773 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:40.760 7f92648b6700 1 -- 172.21.2.102:0/2447938753 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f92540009b0 con 0x7f92601b0470 2021-02-10T11:21:40.773 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:40.760 7f92648b6700 1 -- 172.21.2.102:0/2447938753 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f925400c0c0 con 0x7f92601b0470 2021-02-10T11:21:40.773 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:40.760 7f92648b6700 1 -- 172.21.2.102:0/2447938753 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f92540009b0 con 0x7f92601b0470 2021-02-10T11:21:40.773 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:40.760 7f9267a9c700 1 -- 172.21.2.102:0/2447938753 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f92601b0470 msgr2=0x7f92601b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:40.773 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:40.760 7f9267a9c700 1 --2- 172.21.2.102:0/2447938753 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f92601b0470 0x7f92601b0870 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 rx=0x7f9254006650 tx=0x7f9254009a80).stop 2021-02-10T11:21:40.773 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:40.761 7f9267a9c700 1 -- 172.21.2.102:0/2447938753 shutdown_connections 2021-02-10T11:21:40.773 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:40.761 7f9267a9c700 1 --2- 172.21.2.102:0/2447938753 >> v2:172.21.2.113:3300/0 conn(0x7f92600ef610 0x7f92600efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:40.773 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:40.761 7f9267a9c700 1 --2- 172.21.2.102:0/2447938753 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f92601b0470 0x7f92601b0870 unknown :-1 s=CLOSED pgs=972 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:40.773 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:40.761 7f9267a9c700 1 -- 172.21.2.102:0/2447938753 >> 172.21.2.102:0/2447938753 conn(0x7f92600ce3e0 msgr2=0x7f92600e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:40.774 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:40.761 7f9267a9c700 1 -- 172.21.2.102:0/2447938753 shutdown_connections 2021-02-10T11:21:40.774 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:40.761 7f9267a9c700 1 -- 172.21.2.102:0/2447938753 wait complete. 2021-02-10T11:21:40.775 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:40.776 7f9267a9c700 1 Processor -- start 2021-02-10T11:21:40.775 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:40.776 7f9267a9c700 1 -- start start 2021-02-10T11:21:40.775 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:40.776 7f9267a9c700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f92600a0420 0x7f92600a0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:40.775 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:40.776 7f9267a9c700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9260189d00 0x7f926018a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:40.776 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:40.776 7f9267a9c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f926018a720 0x7f92600b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:40.776 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:40.776 7f9267a9c700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9260189800 con 0x7f926018a720 2021-02-10T11:21:40.776 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:40.776 7f9267a9c700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f926018ab20 con 0x7f92600a0420 2021-02-10T11:21:40.776 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:40.777 7f9267a9c700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92600a1140 con 0x7f9260189d00 2021-02-10T11:21:40.776 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:40.777 7f92660b9700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f926018a720 0x7f92600b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:40.776 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:40.777 7f92660b9700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f926018a720 0x7f92600b7be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:54886/0 (socket says 172.21.2.102:54886) 2021-02-10T11:21:40.776 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:40.777 7f92660b9700 1 -- 172.21.2.102:0/3523249566 learned_addr learned my addr 172.21.2.102:0/3523249566 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:21:40.776 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:40.777 7f92658b8700 1 --2- 172.21.2.102:0/3523249566 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f92600a0420 0x7f92600a0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:40.776 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:40.777 7f92650b7700 1 --2- 172.21.2.102:0/3523249566 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9260189d00 0x7f926018a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:40.776 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:40.777 7f92660b9700 1 -- 172.21.2.102:0/3523249566 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9260189d00 msgr2=0x7f926018a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:40.776 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:40.777 7f92660b9700 1 --2- 172.21.2.102:0/3523249566 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9260189d00 0x7f926018a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:40.776 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:40.777 7f92660b9700 1 -- 172.21.2.102:0/3523249566 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f92600a0420 msgr2=0x7f92600a0820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:40.777 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:40.777 7f92660b9700 1 --2- 172.21.2.102:0/3523249566 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f92600a0420 0x7f92600a0820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:40.777 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:40.777 7f92660b9700 1 -- 172.21.2.102:0/3523249566 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92600b8200 con 0x7f926018a720 2021-02-10T11:21:40.777 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:40.777 7f92658b8700 1 --2- 172.21.2.102:0/3523249566 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f92600a0420 0x7f92600a0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:21:40.777 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:40.777 7f92660b9700 1 --2- 172.21.2.102:0/3523249566 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f926018a720 0x7f92600b7be0 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 rx=0x7f925800a900 tx=0x7f925800aca0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:40.777 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:40.777 7f92527fc700 1 -- 172.21.2.102:0/3523249566 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f92580070d0 con 0x7f926018a720 2021-02-10T11:21:40.777 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:40.777 7f92527fc700 1 -- 172.21.2.102:0/3523249566 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92580073d0 con 0x7f926018a720 2021-02-10T11:21:40.777 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:40.777 7f9267a9c700 1 -- 172.21.2.102:0/3523249566 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92600b8200 con 0x7f926018a720 2021-02-10T11:21:40.777 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:40.777 7f92527fc700 1 -- 172.21.2.102:0/3523249566 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9258004890 con 0x7f926018a720 2021-02-10T11:21:40.777 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:40.777 7f9267a9c700 1 -- 172.21.2.102:0/3523249566 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92600b8200 con 0x7f926018a720 2021-02-10T11:21:40.777 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:40.778 7f92527fc700 1 -- 172.21.2.102:0/3523249566 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f92580046a0 con 0x7f926018a720 2021-02-10T11:21:40.778 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:40.778 7f92527fc700 1 --2- 172.21.2.102:0/3523249566 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f9248018730 0x7f924801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:40.778 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:40.778 7f92527fc700 1 -- 172.21.2.102:0/3523249566 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f9258032a30 con 0x7f926018a720 2021-02-10T11:21:40.778 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:40.778 7f92658b8700 1 --2- 172.21.2.102:0/3523249566 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f9248018730 0x7f924801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:40.778 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:40.778 7f92658b8700 1 --2- 172.21.2.102:0/3523249566 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f9248018730 0x7f924801ac10 crc :-1 s=READY pgs=474 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:40.805 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:40.809 7f9267a9c700 1 -- 172.21.2.102:0/3523249566 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9260105ae0 con 0x7f926018a720 2021-02-10T11:21:40.807 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:40.811 7f92527fc700 1 -- 172.21.2.102:0/3523249566 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f9258000f40 con 0x7f926018a720 2021-02-10T11:21:40.938 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:40.942 7f9267a9c700 1 -- 172.21.2.102:0/3523249566 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344451", "target": ["mgr", ""]}) v1 -- 0x7f9260105ae0 con 0x7f9248018730 2021-02-10T11:21:40.949 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:40.954 7f92527fc700 1 -- 172.21.2.102:0/3523249566 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f9254000c90 con 0x7f9248018730 2021-02-10T11:21:40.955 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:40.959 7f9267a9c700 1 -- 172.21.2.102:0/3523249566 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f9248018730 msgr2=0x7f924801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:40.955 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:40.959 7f9267a9c700 1 --2- 172.21.2.102:0/3523249566 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f9248018730 0x7f924801ac10 crc :-1 s=READY pgs=474 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:40.955 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:40.959 7f9267a9c700 1 -- 172.21.2.102:0/3523249566 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f926018a720 msgr2=0x7f92600b7be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:40.955 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:40.959 7f9267a9c700 1 --2- 172.21.2.102:0/3523249566 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f926018a720 0x7f92600b7be0 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 rx=0x7f925800a900 tx=0x7f925800aca0).stop 2021-02-10T11:21:40.955 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:40.959 7f9267a9c700 1 -- 172.21.2.102:0/3523249566 shutdown_connections 2021-02-10T11:21:40.956 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:40.959 7f9267a9c700 1 --2- 172.21.2.102:0/3523249566 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f926018a720 0x7f92600b7be0 unknown :-1 s=CLOSED pgs=973 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:40.956 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:40.959 7f9267a9c700 1 --2- 172.21.2.102:0/3523249566 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9260189d00 0x7f926018a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:40.956 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:40.959 7f9267a9c700 1 --2- 172.21.2.102:0/3523249566 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f9248018730 0x7f924801ac10 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:40.956 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:40.959 7f9267a9c700 1 --2- 172.21.2.102:0/3523249566 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f92600a0420 0x7f92600a0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:40.956 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:40.959 7f9267a9c700 1 -- 172.21.2.102:0/3523249566 >> 172.21.2.102:0/3523249566 conn(0x7f9260094e10 msgr2=0x7f92600a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:40.956 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:40.959 7f9267a9c700 1 -- 172.21.2.102:0/3523249566 shutdown_connections 2021-02-10T11:21:40.956 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:40.959 7f9267a9c700 1 -- 172.21.2.102:0/3523249566 wait complete. 2021-02-10T11:21:40.964 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000344452 2021-02-10T11:21:41.395 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:41.382 7f1ecb261700 1 Processor -- start 2021-02-10T11:21:41.395 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:41.382 7f1ecb261700 1 -- start start 2021-02-10T11:21:41.395 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:41.383 7f1ecb261700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f1ec41b0470 0x7f1ec41b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:41.395 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:41.383 7f1ecb261700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f1ec40f20e0 0x7f1ec40f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:41.395 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:41.383 7f1ecb261700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ec40f0b50 con 0x7f1ec40ef610 2021-02-10T11:21:41.395 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:41.383 7f1ecb261700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ec40f0820 con 0x7f1ec40f20e0 2021-02-10T11:21:41.395 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:41.383 7f1ecb261700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ec40f24e0 con 0x7f1ec41b0470 2021-02-10T11:21:41.395 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:41.383 7f1ec987e700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f1ec40f20e0 0x7f1ec40f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:41.395 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:41.383 7f1ec987e700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f1ec40f20e0 0x7f1ec40f0200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:54898/0 (socket says 172.21.2.102:54898) 2021-02-10T11:21:41.396 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:41.383 7f1ec907d700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f1ec41b0470 0x7f1ec41b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:41.396 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:41.383 7f1ec987e700 1 -- 172.21.2.102:0/762771802 learned_addr learned my addr 172.21.2.102:0/762771802 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:21:41.396 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:41.383 7f1ec987e700 1 -- 172.21.2.102:0/762771802 >> v2:172.21.2.113:3300/0 conn(0x7f1ec41b0470 msgr2=0x7f1ec41b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:21:41.396 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:41.383 7f1ec987e700 1 --2- 172.21.2.102:0/762771802 >> v2:172.21.2.113:3300/0 conn(0x7f1ec41b0470 0x7f1ec41b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:41.396 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:41.383 7f1ec987e700 1 -- 172.21.2.102:0/762771802 >> v1:172.21.2.113:6789/0 conn(0x7f1ec40ef610 legacy=0x7f1ec40efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:41.396 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:41.383 7f1ec987e700 1 -- 172.21.2.102:0/762771802 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ec40e52e0 con 0x7f1ec40f20e0 2021-02-10T11:21:41.396 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:41.384 7f1ec987e700 1 --2- 172.21.2.102:0/762771802 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1ec40f20e0 0x7f1ec40f0200 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 rx=0x7f1ec0006b80 tx=0x7f1ec0009b90).ready entity=mon.0 client_cookie=240fbdaffa9e8d2a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:41.396 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:41.384 7f1ebbfff700 1 -- 172.21.2.102:0/762771802 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1ec00022e0 con 0x7f1ec40f20e0 2021-02-10T11:21:41.396 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:41.384 7f1ebbfff700 1 -- 172.21.2.102:0/762771802 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ec0002560 con 0x7f1ec40f20e0 2021-02-10T11:21:41.396 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:41.384 7f1ebbfff700 1 -- 172.21.2.102:0/762771802 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1ec0002a80 con 0x7f1ec40f20e0 2021-02-10T11:21:41.396 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:41.384 7f1ecb261700 1 -- 172.21.2.102:0/762771802 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1ec40f20e0 msgr2=0x7f1ec40f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:41.396 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:41.384 7f1ecb261700 1 --2- 172.21.2.102:0/762771802 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1ec40f20e0 0x7f1ec40f0200 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 rx=0x7f1ec0006b80 tx=0x7f1ec0009b90).stop 2021-02-10T11:21:41.396 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:41.384 7f1ecb261700 1 -- 172.21.2.102:0/762771802 shutdown_connections 2021-02-10T11:21:41.397 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:41.384 7f1ecb261700 1 --2- 172.21.2.102:0/762771802 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1ec40f20e0 0x7f1ec40f0200 unknown :-1 s=CLOSED pgs=974 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:41.397 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:41.384 7f1ecb261700 1 --2- 172.21.2.102:0/762771802 >> v2:172.21.2.113:3300/0 conn(0x7f1ec41b0470 0x7f1ec41b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:41.397 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:41.384 7f1ecb261700 1 -- 172.21.2.102:0/762771802 >> 172.21.2.102:0/762771802 conn(0x7f1ec40ce3e0 msgr2=0x7f1ec40e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:41.397 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:41.385 7f1ecb261700 1 -- 172.21.2.102:0/762771802 shutdown_connections 2021-02-10T11:21:41.397 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:41.385 7f1ecb261700 1 -- 172.21.2.102:0/762771802 wait complete. 2021-02-10T11:21:41.397 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:41.399 7f1ecb261700 1 Processor -- start 2021-02-10T11:21:41.397 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:41.399 7f1ecb261700 1 -- start start 2021-02-10T11:21:41.398 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:41.400 7f1ecb261700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1ec4190920 0x7f1ec40b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:41.399 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:41.400 7f1ecb261700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1ec40b8200 0x7f1ec40b8600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:41.399 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:41.400 7f1ecb261700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1ec40c1d80 0x7f1ec40c2180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:41.399 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:41.400 7f1ecb261700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ec4190d20 con 0x7f1ec40c1d80 2021-02-10T11:21:41.399 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:41.400 7f1ecb261700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ec40b8c20 con 0x7f1ec4190920 2021-02-10T11:21:41.399 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:41.400 7f1ecb261700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ec40c27a0 con 0x7f1ec40b8200 2021-02-10T11:21:41.399 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:41.400 7f1ec987e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1ec40c1d80 0x7f1ec40c2180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:41.399 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:41.400 7f1ec987e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1ec40c1d80 0x7f1ec40c2180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:54902/0 (socket says 172.21.2.102:54902) 2021-02-10T11:21:41.399 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:41.400 7f1ec987e700 1 -- 172.21.2.102:0/882719270 learned_addr learned my addr 172.21.2.102:0/882719270 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:21:41.399 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:41.400 7f1ec887c700 1 --2- 172.21.2.102:0/882719270 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1ec40b8200 0x7f1ec40b8600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:41.399 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:41.400 7f1ec907d700 1 --2- 172.21.2.102:0/882719270 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1ec4190920 0x7f1ec40b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:41.399 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:41.400 7f1ec987e700 1 -- 172.21.2.102:0/882719270 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1ec40b8200 msgr2=0x7f1ec40b8600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:41.399 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:41.400 7f1ec987e700 1 --2- 172.21.2.102:0/882719270 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1ec40b8200 0x7f1ec40b8600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:41.400 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:41.400 7f1ec987e700 1 -- 172.21.2.102:0/882719270 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1ec4190920 msgr2=0x7f1ec40b7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:41.400 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:41.400 7f1ec987e700 1 --2- 172.21.2.102:0/882719270 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1ec4190920 0x7f1ec40b7be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:41.400 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:41.400 7f1ec987e700 1 -- 172.21.2.102:0/882719270 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ec40c29c0 con 0x7f1ec40c1d80 2021-02-10T11:21:41.400 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:41.400 7f1ec987e700 1 --2- 172.21.2.102:0/882719270 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1ec40c1d80 0x7f1ec40c2180 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 rx=0x7f1ec0009640 tx=0x7f1ec000b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:41.400 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:41.401 7f1eb9ffb700 1 -- 172.21.2.102:0/882719270 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1ec000bd50 con 0x7f1ec40c1d80 2021-02-10T11:21:41.400 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:41.401 7f1eb9ffb700 1 -- 172.21.2.102:0/882719270 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ec0006240 con 0x7f1ec40c1d80 2021-02-10T11:21:41.400 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:41.401 7f1eb9ffb700 1 -- 172.21.2.102:0/882719270 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1ec00054a0 con 0x7f1ec40c1d80 2021-02-10T11:21:41.400 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:41.401 7f1ecb261700 1 -- 172.21.2.102:0/882719270 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ec40c29c0 con 0x7f1ec40c1d80 2021-02-10T11:21:41.400 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:41.401 7f1ecb261700 1 -- 172.21.2.102:0/882719270 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ec40c29c0 con 0x7f1ec40c1d80 2021-02-10T11:21:41.400 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:41.401 7f1eb9ffb700 1 -- 172.21.2.102:0/882719270 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1ec0006420 con 0x7f1ec40c1d80 2021-02-10T11:21:41.401 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:41.401 7f1eb9ffb700 1 --2- 172.21.2.102:0/882719270 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1ea40186c0 0x7f1ea401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:41.401 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:41.401 7f1eb9ffb700 1 -- 172.21.2.102:0/882719270 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f1ec0032440 con 0x7f1ec40c1d80 2021-02-10T11:21:41.401 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:41.401 7f1ec907d700 1 --2- 172.21.2.102:0/882719270 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1ea40186c0 0x7f1ea401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:41.401 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:41.401 7f1ec907d700 1 --2- 172.21.2.102:0/882719270 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1ea40186c0 0x7f1ea401abc0 crc :-1 s=READY pgs=475 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:41.427 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:41.431 7f1ecb261700 1 -- 172.21.2.102:0/882719270 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ec4105ae0 con 0x7f1ec40c1d80 2021-02-10T11:21:41.430 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:41.434 7f1eb9ffb700 1 -- 172.21.2.102:0/882719270 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f1ec0006030 con 0x7f1ec40c1d80 2021-02-10T11:21:41.558 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:41.562 7f1ecb261700 1 -- 172.21.2.102:0/882719270 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344452", "target": ["mgr", ""]}) v1 -- 0x7f1ec4105ae0 con 0x7f1ea40186c0 2021-02-10T11:21:41.570 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:41.574 7f1eb9ffb700 1 -- 172.21.2.102:0/882719270 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f1eb4006a70 con 0x7f1ea40186c0 2021-02-10T11:21:41.575 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:41.579 7f1ecb261700 1 -- 172.21.2.102:0/882719270 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1ea40186c0 msgr2=0x7f1ea401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:41.575 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:41.579 7f1ecb261700 1 --2- 172.21.2.102:0/882719270 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1ea40186c0 0x7f1ea401abc0 crc :-1 s=READY pgs=475 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:41.576 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:41.579 7f1ecb261700 1 -- 172.21.2.102:0/882719270 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1ec40c1d80 msgr2=0x7f1ec40c2180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:41.576 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:41.579 7f1ecb261700 1 --2- 172.21.2.102:0/882719270 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1ec40c1d80 0x7f1ec40c2180 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 rx=0x7f1ec0009640 tx=0x7f1ec000b960).stop 2021-02-10T11:21:41.576 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:41.579 7f1ecb261700 1 -- 172.21.2.102:0/882719270 shutdown_connections 2021-02-10T11:21:41.576 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:41.579 7f1ecb261700 1 --2- 172.21.2.102:0/882719270 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1ec40c1d80 0x7f1ec40c2180 unknown :-1 s=CLOSED pgs=975 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:41.576 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:41.579 7f1ecb261700 1 --2- 172.21.2.102:0/882719270 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1ec40b8200 0x7f1ec40b8600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:41.576 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:41.579 7f1ecb261700 1 --2- 172.21.2.102:0/882719270 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1ea40186c0 0x7f1ea401abc0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:41.576 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:41.579 7f1ecb261700 1 --2- 172.21.2.102:0/882719270 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1ec4190920 0x7f1ec40b7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:41.576 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:41.579 7f1ecb261700 1 -- 172.21.2.102:0/882719270 >> 172.21.2.102:0/882719270 conn(0x7f1ec4094e10 msgr2=0x7f1ec40a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:41.576 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:41.579 7f1ecb261700 1 -- 172.21.2.102:0/882719270 shutdown_connections 2021-02-10T11:21:41.577 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:41.579 7f1ecb261700 1 -- 172.21.2.102:0/882719270 wait complete. 2021-02-10T11:21:41.585 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000344453 2021-02-10T11:21:42.016 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:42.004 7ff841858700 1 Processor -- start 2021-02-10T11:21:42.016 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:42.004 7ff841858700 1 -- start start 2021-02-10T11:21:42.016 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:42.004 7ff841858700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff83c1b0470 0x7ff83c1b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:42.016 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:42.004 7ff841858700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7ff83c0ef610 0x7ff83c0efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:42.016 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:42.004 7ff841858700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7ff83c0f0030 0x7ff83c0f0430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:42.016 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:42.004 7ff841858700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff83c0e5ea0 con 0x7ff83c0ef610 2021-02-10T11:21:42.016 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:42.004 7ff841858700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff83c0f0b50 con 0x7ff83c0f0030 2021-02-10T11:21:42.016 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:42.004 7ff841858700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff83c0e52e0 con 0x7ff83c1b0470 2021-02-10T11:21:42.016 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:42.004 7ff832d73700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7ff83c0ef610 0x7ff83c0efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:42.017 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:42.004 7ff832d73700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7ff83c0ef610 0x7ff83c0efa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:54908/0 (socket says 172.21.2.102:54908) 2021-02-10T11:21:42.017 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:42.004 7ff832d73700 1 -- 172.21.2.102:0/1200508370 learned_addr learned my addr 172.21.2.102:0/1200508370 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:21:42.017 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:42.004 7ff83bd75700 1 --2- 172.21.2.102:0/1200508370 >> v2:172.21.2.113:3300/0 conn(0x7ff83c0f0030 0x7ff83c0f0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:42.017 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:42.004 7ff83b574700 1 --2- 172.21.2.102:0/1200508370 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff83c1b0470 0x7ff83c1b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:42.017 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:42.004 7ff832d73700 1 -- 172.21.2.102:0/1200508370 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff83c1b0470 msgr2=0x7ff83c1b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:21:42.017 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:42.005 7ff832d73700 1 --2- 172.21.2.102:0/1200508370 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff83c1b0470 0x7ff83c1b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:42.017 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:42.005 7ff832d73700 1 -- 172.21.2.102:0/1200508370 >> v2:172.21.2.113:3300/0 conn(0x7ff83c0f0030 msgr2=0x7ff83c0f0430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:21:42.017 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:42.005 7ff832d73700 1 --2- 172.21.2.102:0/1200508370 >> v2:172.21.2.113:3300/0 conn(0x7ff83c0f0030 0x7ff83c0f0430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:42.017 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:42.005 7ff832d73700 1 -- 172.21.2.102:0/1200508370 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff83c0f10d0 con 0x7ff83c0ef610 2021-02-10T11:21:42.017 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:42.005 7ff83bd75700 1 --2- 172.21.2.102:0/1200508370 >> v2:172.21.2.113:3300/0 conn(0x7ff83c0f0030 0x7ff83c0f0430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:21:42.017 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:42.005 7ff83b574700 1 --2- 172.21.2.102:0/1200508370 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff83c1b0470 0x7ff83c1b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:21:42.017 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:42.005 7ff832d73700 1 --2- 172.21.2.102:0/1200508370 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff83c0ef610 0x7ff83c0efa10 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 rx=0x7ff824006700 tx=0x7ff824009b80).ready entity=mon.0 client_cookie=d0c634c1c64decb8 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:42.018 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:42.005 7ff83ad73700 1 -- 172.21.2.102:0/1200508370 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff824006340 con 0x7ff83c0ef610 2021-02-10T11:21:42.018 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:42.005 7ff83ad73700 1 -- 172.21.2.102:0/1200508370 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff82400c0c0 con 0x7ff83c0ef610 2021-02-10T11:21:42.018 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:42.005 7ff83ad73700 1 -- 172.21.2.102:0/1200508370 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff82400c510 con 0x7ff83c0ef610 2021-02-10T11:21:42.018 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:42.005 7ff841858700 1 -- 172.21.2.102:0/1200508370 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff83c0ef610 msgr2=0x7ff83c0efa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:42.018 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:42.005 7ff841858700 1 --2- 172.21.2.102:0/1200508370 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff83c0ef610 0x7ff83c0efa10 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 rx=0x7ff824006700 tx=0x7ff824009b80).stop 2021-02-10T11:21:42.018 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:42.005 7ff841858700 1 -- 172.21.2.102:0/1200508370 shutdown_connections 2021-02-10T11:21:42.018 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:42.005 7ff841858700 1 --2- 172.21.2.102:0/1200508370 >> v2:172.21.2.113:3300/0 conn(0x7ff83c0f0030 0x7ff83c0f0430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:42.018 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:42.005 7ff841858700 1 --2- 172.21.2.102:0/1200508370 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff83c0ef610 0x7ff83c0efa10 unknown :-1 s=CLOSED pgs=976 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:42.018 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:42.005 7ff841858700 1 --2- 172.21.2.102:0/1200508370 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff83c1b0470 0x7ff83c1b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:42.018 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:42.005 7ff841858700 1 -- 172.21.2.102:0/1200508370 >> 172.21.2.102:0/1200508370 conn(0x7ff83c0ce3e0 msgr2=0x7ff83c0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:42.018 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:42.006 7ff841858700 1 -- 172.21.2.102:0/1200508370 shutdown_connections 2021-02-10T11:21:42.018 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:42.006 7ff841858700 1 -- 172.21.2.102:0/1200508370 wait complete. 2021-02-10T11:21:42.020 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:42.020 7ff841858700 1 Processor -- start 2021-02-10T11:21:42.020 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:42.020 7ff841858700 1 -- start start 2021-02-10T11:21:42.020 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:42.021 7ff841858700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff83c092c40 0x7ff83c0b3160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:42.020 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:42.021 7ff841858700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff83c093040 0x7ff83c106a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:42.020 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:42.021 7ff841858700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff83c107060 0x7ff83c107460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:42.020 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:42.021 7ff841858700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff83c0c2120 con 0x7ff83c093040 2021-02-10T11:21:42.020 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:42.021 7ff841858700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff83c0b3c00 con 0x7ff83c092c40 2021-02-10T11:21:42.021 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:42.021 7ff841858700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff83c0b3780 con 0x7ff83c107060 2021-02-10T11:21:42.021 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:42.021 7ff832d73700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff83c093040 0x7ff83c106a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:42.021 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:42.021 7ff83b574700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff83c092c40 0x7ff83c0b3160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:42.021 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:42.021 7ff832d73700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff83c093040 0x7ff83c106a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:54918/0 (socket says 172.21.2.102:54918) 2021-02-10T11:21:42.021 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:42.021 7ff832d73700 1 -- 172.21.2.102:0/2529847910 learned_addr learned my addr 172.21.2.102:0/2529847910 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:21:42.021 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:42.021 7ff83b574700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff83c092c40 0x7ff83c0b3160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.113:3300/0 says I am v2:172.21.2.102:58344/0 (socket says 172.21.2.102:58344) 2021-02-10T11:21:42.021 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:42.021 7ff83bd75700 1 --2- 172.21.2.102:0/2529847910 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff83c107060 0x7ff83c107460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:42.021 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:42.021 7ff832d73700 1 -- 172.21.2.102:0/2529847910 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff83c107060 msgr2=0x7ff83c107460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:42.021 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:42.022 7ff832d73700 1 --2- 172.21.2.102:0/2529847910 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff83c107060 0x7ff83c107460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:42.021 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:42.022 7ff832d73700 1 -- 172.21.2.102:0/2529847910 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff83c092c40 msgr2=0x7ff83c0b3160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:42.021 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:42.022 7ff832d73700 1 --2- 172.21.2.102:0/2529847910 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff83c092c40 0x7ff83c0b3160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:42.021 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:42.022 7ff832d73700 1 -- 172.21.2.102:0/2529847910 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff83c0c0ba0 con 0x7ff83c093040 2021-02-10T11:21:42.022 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:42.022 7ff83b574700 1 --2- 172.21.2.102:0/2529847910 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff83c092c40 0x7ff83c0b3160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:21:42.022 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:42.022 7ff832d73700 1 --2- 172.21.2.102:0/2529847910 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff83c093040 0x7ff83c106a40 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 rx=0x7ff82400b5d0 tx=0x7ff82400b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:42.022 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:42.022 7ff838d6f700 1 -- 172.21.2.102:0/2529847910 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff82400bd60 con 0x7ff83c093040 2021-02-10T11:21:42.022 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:42.022 7ff838d6f700 1 -- 172.21.2.102:0/2529847910 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff824006590 con 0x7ff83c093040 2021-02-10T11:21:42.022 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:42.022 7ff838d6f700 1 -- 172.21.2.102:0/2529847910 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff824002220 con 0x7ff83c093040 2021-02-10T11:21:42.022 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:42.022 7ff841858700 1 -- 172.21.2.102:0/2529847910 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff83c0c0ba0 con 0x7ff83c093040 2021-02-10T11:21:42.022 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:42.022 7ff841858700 1 -- 172.21.2.102:0/2529847910 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff83c0c1130 con 0x7ff83c093040 2021-02-10T11:21:42.022 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:42.023 7ff838d6f700 1 -- 172.21.2.102:0/2529847910 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ff824000c90 con 0x7ff83c093040 2021-02-10T11:21:42.022 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:42.023 7ff838d6f700 1 --2- 172.21.2.102:0/2529847910 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff8200186c0 0x7ff82001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:42.022 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:42.023 7ff838d6f700 1 -- 172.21.2.102:0/2529847910 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7ff824032480 con 0x7ff83c093040 2021-02-10T11:21:42.022 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:42.023 7ff83b574700 1 --2- 172.21.2.102:0/2529847910 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff8200186c0 0x7ff82001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:42.023 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:42.023 7ff83b574700 1 --2- 172.21.2.102:0/2529847910 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff8200186c0 0x7ff82001abc0 crc :-1 s=READY pgs=476 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:42.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:42.054 7ff841858700 1 -- 172.21.2.102:0/2529847910 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff83c105b30 con 0x7ff83c093040 2021-02-10T11:21:42.052 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:42.056 7ff838d6f700 1 -- 172.21.2.102:0/2529847910 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ff8240009b0 con 0x7ff83c093040 2021-02-10T11:21:42.181 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:42.185 7ff841858700 1 -- 172.21.2.102:0/2529847910 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344453", "target": ["mgr", ""]}) v1 -- 0x7ff83c0c1130 con 0x7ff8200186c0 2021-02-10T11:21:42.192 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:42.197 7ff838d6f700 1 -- 172.21.2.102:0/2529847910 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7ff82c000d30 con 0x7ff8200186c0 2021-02-10T11:21:42.198 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:42.202 7ff841858700 1 -- 172.21.2.102:0/2529847910 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff8200186c0 msgr2=0x7ff82001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:42.199 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:42.202 7ff841858700 1 --2- 172.21.2.102:0/2529847910 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff8200186c0 0x7ff82001abc0 crc :-1 s=READY pgs=476 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:42.199 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:42.202 7ff841858700 1 -- 172.21.2.102:0/2529847910 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff83c093040 msgr2=0x7ff83c106a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:42.199 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:42.202 7ff841858700 1 --2- 172.21.2.102:0/2529847910 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff83c093040 0x7ff83c106a40 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 rx=0x7ff82400b5d0 tx=0x7ff82400b970).stop 2021-02-10T11:21:42.199 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:42.202 7ff841858700 1 -- 172.21.2.102:0/2529847910 shutdown_connections 2021-02-10T11:21:42.199 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:42.202 7ff841858700 1 --2- 172.21.2.102:0/2529847910 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff83c107060 0x7ff83c107460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:42.199 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:42.202 7ff841858700 1 --2- 172.21.2.102:0/2529847910 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff83c093040 0x7ff83c106a40 unknown :-1 s=CLOSED pgs=977 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:42.199 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:42.202 7ff841858700 1 --2- 172.21.2.102:0/2529847910 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff8200186c0 0x7ff82001abc0 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:42.199 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:42.202 7ff841858700 1 --2- 172.21.2.102:0/2529847910 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff83c092c40 0x7ff83c0b3160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:42.199 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:42.202 7ff841858700 1 -- 172.21.2.102:0/2529847910 >> 172.21.2.102:0/2529847910 conn(0x7ff83c0c40e0 msgr2=0x7ff83c0bf620 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:42.199 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:42.203 7ff841858700 1 -- 172.21.2.102:0/2529847910 shutdown_connections 2021-02-10T11:21:42.199 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:42.203 7ff841858700 1 -- 172.21.2.102:0/2529847910 wait complete. 2021-02-10T11:21:42.207 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000344454 2021-02-10T11:21:42.643 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:42.629 7f6528c3a700 1 Processor -- start 2021-02-10T11:21:42.643 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:42.629 7f6528c3a700 1 -- start start 2021-02-10T11:21:42.643 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:42.630 7f6528c3a700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f65241b0870 0x7f65240f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:42.643 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:42.630 7f6528c3a700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f65240efe00 0x7f65240f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:42.643 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:42.630 7f6528c3a700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65241b0c70 con 0x7f65241b0470 2021-02-10T11:21:42.643 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:42.630 7f6528c3a700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65240f0b50 con 0x7f65240efe00 2021-02-10T11:21:42.643 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:42.630 7f6528c3a700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65240f0820 con 0x7f65241b0870 2021-02-10T11:21:42.643 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:42.630 7f652285e700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f65241b0470 0x7f65240ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:47146/0 (socket says 172.21.2.102:47146) 2021-02-10T11:21:42.644 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:42.630 7f652285e700 1 -- 172.21.2.102:0/1327719935 learned_addr learned my addr 172.21.2.102:0/1327719935 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:21:42.644 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:42.630 7f652205d700 1 --2- 172.21.2.102:0/1327719935 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f65241b0870 0x7f65240f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:42.644 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:42.630 7f652305f700 1 --2- 172.21.2.102:0/1327719935 >> v2:172.21.2.113:3300/0 conn(0x7f65240efe00 0x7f65240f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:42.644 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:42.630 7f652185c700 1 -- 172.21.2.102:0/1327719935 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 978380677 0 0) 0x7f6518001180 con 0x7f65241b0470 2021-02-10T11:21:42.644 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:42.630 7f652185c700 1 -- 172.21.2.102:0/1327719935 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f650c002fe0 con 0x7f65241b0470 2021-02-10T11:21:42.644 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:42.630 7f652205d700 1 -- 172.21.2.102:0/1327719935 >> v2:172.21.2.113:3300/0 conn(0x7f65240efe00 msgr2=0x7f65240f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:21:42.644 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:42.630 7f652205d700 1 --2- 172.21.2.102:0/1327719935 >> v2:172.21.2.113:3300/0 conn(0x7f65240efe00 0x7f65240f0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:42.644 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:42.630 7f652205d700 1 -- 172.21.2.102:0/1327719935 >> v1:172.21.2.102:6789/0 conn(0x7f65241b0470 legacy=0x7f65240ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:42.644 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:42.631 7f652205d700 1 -- 172.21.2.102:0/1327719935 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65240e52e0 con 0x7f65241b0870 2021-02-10T11:21:42.644 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:42.631 7f652305f700 1 --2- 172.21.2.102:0/1327719935 >> v2:172.21.2.113:3300/0 conn(0x7f65240efe00 0x7f65240f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:21:42.645 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:42.631 7f652205d700 1 --2- 172.21.2.102:0/1327719935 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f65241b0870 0x7f65240f20e0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7f651c0069d0 tx=0x7f651c009e40).ready entity=mon.2 client_cookie=3374116f4479d0cf server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:42.645 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:42.631 7f652185c700 1 -- 172.21.2.102:0/1327719935 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f651c00b510 con 0x7f65241b0870 2021-02-10T11:21:42.645 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:42.631 7f652185c700 1 -- 172.21.2.102:0/1327719935 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f651c00b890 con 0x7f65241b0870 2021-02-10T11:21:42.645 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:42.631 7f652185c700 1 -- 172.21.2.102:0/1327719935 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f651c00bdb0 con 0x7f65241b0870 2021-02-10T11:21:42.645 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:42.631 7f6528c3a700 1 -- 172.21.2.102:0/1327719935 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f65241b0870 msgr2=0x7f65240f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:42.645 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:42.631 7f6528c3a700 1 --2- 172.21.2.102:0/1327719935 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f65241b0870 0x7f65240f20e0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7f651c0069d0 tx=0x7f651c009e40).stop 2021-02-10T11:21:42.645 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:42.632 7f6528c3a700 1 -- 172.21.2.102:0/1327719935 shutdown_connections 2021-02-10T11:21:42.645 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:42.632 7f6528c3a700 1 --2- 172.21.2.102:0/1327719935 >> v2:172.21.2.113:3300/0 conn(0x7f65240efe00 0x7f65240f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:42.645 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:42.632 7f6528c3a700 1 --2- 172.21.2.102:0/1327719935 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f65241b0870 0x7f65240f20e0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:42.645 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:42.632 7f6528c3a700 1 -- 172.21.2.102:0/1327719935 >> 172.21.2.102:0/1327719935 conn(0x7f65240ce3e0 msgr2=0x7f65240e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:42.645 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:42.632 7f6528c3a700 1 -- 172.21.2.102:0/1327719935 shutdown_connections 2021-02-10T11:21:42.646 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:42.632 7f6528c3a700 1 -- 172.21.2.102:0/1327719935 wait complete. 2021-02-10T11:21:42.646 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:42.646 7f6528c3a700 1 Processor -- start 2021-02-10T11:21:42.646 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:42.646 7f6528c3a700 1 -- start start 2021-02-10T11:21:42.646 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:42.647 7f6528c3a700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6524190920 0x7f65240b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:42.646 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:42.647 7f6528c3a700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f65240b8200 0x7f65240b8600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:42.646 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:42.647 7f6528c3a700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f65240c1d80 0x7f65240c2180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:42.646 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:42.647 7f6528c3a700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6524190da0 con 0x7f6524190920 2021-02-10T11:21:42.646 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:42.647 7f6528c3a700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65240b8c20 con 0x7f65240b8200 2021-02-10T11:21:42.646 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:42.647 7f6528c3a700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65240b8e10 con 0x7f65240c1d80 2021-02-10T11:21:42.649 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:42.647 7f652285e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6524190920 0x7f65240b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:42.649 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:42.647 7f652285e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6524190920 0x7f65240b7be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:54928/0 (socket says 172.21.2.102:54928) 2021-02-10T11:21:42.649 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:42.647 7f652285e700 1 -- 172.21.2.102:0/3681625248 learned_addr learned my addr 172.21.2.102:0/3681625248 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:21:42.649 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:42.647 7f652205d700 1 --2- 172.21.2.102:0/3681625248 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f65240b8200 0x7f65240b8600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:42.649 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:42.647 7f652305f700 1 --2- 172.21.2.102:0/3681625248 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f65240c1d80 0x7f65240c2180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:42.649 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:42.648 7f652285e700 1 -- 172.21.2.102:0/3681625248 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f65240c1d80 msgr2=0x7f65240c2180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:42.649 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:42.648 7f652285e700 1 --2- 172.21.2.102:0/3681625248 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f65240c1d80 0x7f65240c2180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:42.649 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:42.648 7f652285e700 1 -- 172.21.2.102:0/3681625248 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f65240b8200 msgr2=0x7f65240b8600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:42.650 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:42.648 7f652285e700 1 --2- 172.21.2.102:0/3681625248 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f65240b8200 0x7f65240b8600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:42.650 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:42.648 7f652285e700 1 -- 172.21.2.102:0/3681625248 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65240c27d0 con 0x7f6524190920 2021-02-10T11:21:42.650 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:42.648 7f652285e700 1 --2- 172.21.2.102:0/3681625248 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6524190920 0x7f65240b7be0 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 rx=0x7f651800a5d0 tx=0x7f651800a970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:42.650 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:42.648 7f65137fe700 1 -- 172.21.2.102:0/3681625248 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6518000ce0 con 0x7f6524190920 2021-02-10T11:21:42.651 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:42.648 7f65137fe700 1 -- 172.21.2.102:0/3681625248 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65180043a0 con 0x7f6524190920 2021-02-10T11:21:42.651 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:42.648 7f65137fe700 1 -- 172.21.2.102:0/3681625248 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6518004800 con 0x7f6524190920 2021-02-10T11:21:42.651 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:42.648 7f6528c3a700 1 -- 172.21.2.102:0/3681625248 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65240c27d0 con 0x7f6524190920 2021-02-10T11:21:42.651 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:42.648 7f6528c3a700 1 -- 172.21.2.102:0/3681625248 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65240c2d60 con 0x7f6524190920 2021-02-10T11:21:42.651 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:42.649 7f65137fe700 1 -- 172.21.2.102:0/3681625248 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f6518004580 con 0x7f6524190920 2021-02-10T11:21:42.651 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:42.649 7f65137fe700 1 --2- 172.21.2.102:0/3681625248 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f65080186c0 0x7f650801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:42.651 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:42.649 7f65137fe700 1 -- 172.21.2.102:0/3681625248 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f6518032540 con 0x7f6524190920 2021-02-10T11:21:42.651 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:42.649 7f652205d700 1 --2- 172.21.2.102:0/3681625248 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f65080186c0 0x7f650801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:42.651 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:42.649 7f652205d700 1 --2- 172.21.2.102:0/3681625248 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f65080186c0 0x7f650801abc0 crc :-1 s=READY pgs=477 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:42.675 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:42.679 7f6528c3a700 1 -- 172.21.2.102:0/3681625248 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6524105ae0 con 0x7f6524190920 2021-02-10T11:21:42.678 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:42.683 7f65137fe700 1 -- 172.21.2.102:0/3681625248 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f651802ac70 con 0x7f6524190920 2021-02-10T11:21:42.811 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:42.815 7f6528c3a700 1 -- 172.21.2.102:0/3681625248 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344454", "target": ["mgr", ""]}) v1 -- 0x7f65240f0b50 con 0x7f65080186c0 2021-02-10T11:21:42.821 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:42.825 7f65137fe700 1 -- 172.21.2.102:0/3681625248 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f651c006e60 con 0x7f65080186c0 2021-02-10T11:21:42.828 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:42.832 7f6528c3a700 1 -- 172.21.2.102:0/3681625248 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f65080186c0 msgr2=0x7f650801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:42.828 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:42.832 7f6528c3a700 1 --2- 172.21.2.102:0/3681625248 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f65080186c0 0x7f650801abc0 crc :-1 s=READY pgs=477 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:42.828 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:42.832 7f6528c3a700 1 -- 172.21.2.102:0/3681625248 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6524190920 msgr2=0x7f65240b7be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:42.828 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:42.832 7f6528c3a700 1 --2- 172.21.2.102:0/3681625248 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6524190920 0x7f65240b7be0 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 rx=0x7f651800a5d0 tx=0x7f651800a970).stop 2021-02-10T11:21:42.828 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:42.832 7f6528c3a700 1 -- 172.21.2.102:0/3681625248 shutdown_connections 2021-02-10T11:21:42.829 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:42.832 7f6528c3a700 1 --2- 172.21.2.102:0/3681625248 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f65240c1d80 0x7f65240c2180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:42.829 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:42.832 7f6528c3a700 1 --2- 172.21.2.102:0/3681625248 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f65080186c0 0x7f650801abc0 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:42.829 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:42.832 7f6528c3a700 1 --2- 172.21.2.102:0/3681625248 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f65240b8200 0x7f65240b8600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:42.829 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:42.832 7f6528c3a700 1 --2- 172.21.2.102:0/3681625248 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6524190920 0x7f65240b7be0 unknown :-1 s=CLOSED pgs=979 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:42.829 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:42.832 7f6528c3a700 1 -- 172.21.2.102:0/3681625248 >> 172.21.2.102:0/3681625248 conn(0x7f6524094e10 msgr2=0x7f65240a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:42.829 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:42.832 7f6528c3a700 1 -- 172.21.2.102:0/3681625248 shutdown_connections 2021-02-10T11:21:42.829 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:42.832 7f6528c3a700 1 -- 172.21.2.102:0/3681625248 wait complete. 2021-02-10T11:21:42.836 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000344455 2021-02-10T11:21:43.267 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:43.255 7fb6ddbff700 1 Processor -- start 2021-02-10T11:21:43.268 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:43.255 7fb6ddbff700 1 -- start start 2021-02-10T11:21:43.268 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:43.255 7fb6ddbff700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fb6d81b0870 0x7fb6d80f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:43.268 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:43.255 7fb6ddbff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fb6d80efe00 0x7fb6d80f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:43.268 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:43.255 7fb6ddbff700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6d81b0c70 con 0x7fb6d81b0470 2021-02-10T11:21:43.268 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:43.255 7fb6ddbff700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6d80e5ea0 con 0x7fb6d80efe00 2021-02-10T11:21:43.268 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:43.255 7fb6ddbff700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6d80f0b50 con 0x7fb6d81b0870 2021-02-10T11:21:43.268 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:43.256 7fb6d7fff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fb6d80efe00 0x7fb6d80f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:43.268 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:43.256 7fb6d7fff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fb6d80efe00 0x7fb6d80f0200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:54940/0 (socket says 172.21.2.102:54940) 2021-02-10T11:21:43.268 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:43.256 7fb6d6ffd700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fb6d81b0870 0x7fb6d80f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:43.268 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:43.256 7fb6d7fff700 1 -- 172.21.2.102:0/2199426784 learned_addr learned my addr 172.21.2.102:0/2199426784 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:21:43.268 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:43.256 7fb6d7fff700 1 -- 172.21.2.102:0/2199426784 >> v2:172.21.2.113:3300/0 conn(0x7fb6d81b0870 msgr2=0x7fb6d80f20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:21:43.269 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:43.256 7fb6d7fff700 1 --2- 172.21.2.102:0/2199426784 >> v2:172.21.2.113:3300/0 conn(0x7fb6d81b0870 0x7fb6d80f20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:43.269 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:43.256 7fb6d7fff700 1 -- 172.21.2.102:0/2199426784 >> v1:172.21.2.113:6789/0 conn(0x7fb6d81b0470 legacy=0x7fb6d80ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:43.269 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:43.256 7fb6d7fff700 1 -- 172.21.2.102:0/2199426784 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6d80f0820 con 0x7fb6d80efe00 2021-02-10T11:21:43.269 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:43.256 7fb6d7fff700 1 --2- 172.21.2.102:0/2199426784 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb6d80efe00 0x7fb6d80f0200 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 rx=0x7fb6d0006f00 tx=0x7fb6d0009f10).ready entity=mon.0 client_cookie=fcf1f3a8eda4847 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:43.269 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:43.256 7fb6d67fc700 1 -- 172.21.2.102:0/2199426784 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb6d000b3b0 con 0x7fb6d80efe00 2021-02-10T11:21:43.269 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:43.257 7fb6d67fc700 1 -- 172.21.2.102:0/2199426784 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6d000b630 con 0x7fb6d80efe00 2021-02-10T11:21:43.269 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:43.257 7fb6d67fc700 1 -- 172.21.2.102:0/2199426784 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb6d000bb50 con 0x7fb6d80efe00 2021-02-10T11:21:43.269 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:43.257 7fb6ddbff700 1 -- 172.21.2.102:0/2199426784 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb6d80efe00 msgr2=0x7fb6d80f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:43.269 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:43.257 7fb6ddbff700 1 --2- 172.21.2.102:0/2199426784 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb6d80efe00 0x7fb6d80f0200 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 rx=0x7fb6d0006f00 tx=0x7fb6d0009f10).stop 2021-02-10T11:21:43.269 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:43.257 7fb6ddbff700 1 -- 172.21.2.102:0/2199426784 shutdown_connections 2021-02-10T11:21:43.269 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:43.257 7fb6ddbff700 1 --2- 172.21.2.102:0/2199426784 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb6d80efe00 0x7fb6d80f0200 unknown :-1 s=CLOSED pgs=980 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:43.269 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:43.257 7fb6ddbff700 1 --2- 172.21.2.102:0/2199426784 >> v2:172.21.2.113:3300/0 conn(0x7fb6d81b0870 0x7fb6d80f20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:43.269 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:43.257 7fb6ddbff700 1 -- 172.21.2.102:0/2199426784 >> 172.21.2.102:0/2199426784 conn(0x7fb6d80ce3e0 msgr2=0x7fb6d80e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:43.270 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:43.257 7fb6ddbff700 1 -- 172.21.2.102:0/2199426784 shutdown_connections 2021-02-10T11:21:43.270 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:43.257 7fb6ddbff700 1 -- 172.21.2.102:0/2199426784 wait complete. 2021-02-10T11:21:43.271 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:43.272 7fb6ddbff700 1 Processor -- start 2021-02-10T11:21:43.271 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:43.272 7fb6ddbff700 1 -- start start 2021-02-10T11:21:43.271 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:43.273 7fb6ddbff700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb6d80a0420 0x7fb6d80a0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:43.271 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:43.273 7fb6ddbff700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb6d8189d00 0x7fb6d818a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:43.272 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:43.273 7fb6ddbff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb6d818a720 0x7fb6d80b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:43.272 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:43.273 7fb6ddbff700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6d8189780 con 0x7fb6d818a720 2021-02-10T11:21:43.272 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:43.273 7fb6ddbff700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6d818ab20 con 0x7fb6d8189d00 2021-02-10T11:21:43.272 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:43.273 7fb6ddbff700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6d80a1140 con 0x7fb6d80a0420 2021-02-10T11:21:43.272 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:43.273 7fb6d7fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb6d818a720 0x7fb6d80b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:43.272 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:43.273 7fb6d7fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb6d818a720 0x7fb6d80b7be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:54944/0 (socket says 172.21.2.102:54944) 2021-02-10T11:21:43.272 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:43.273 7fb6d7fff700 1 -- 172.21.2.102:0/3177134410 learned_addr learned my addr 172.21.2.102:0/3177134410 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:21:43.272 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:43.273 7fb6d6ffd700 1 --2- 172.21.2.102:0/3177134410 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb6d8189d00 0x7fb6d818a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:43.272 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:43.273 7fb6d77fe700 1 --2- 172.21.2.102:0/3177134410 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb6d80a0420 0x7fb6d80a0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:43.272 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:43.273 7fb6d7fff700 1 -- 172.21.2.102:0/3177134410 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb6d80a0420 msgr2=0x7fb6d80a0820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:43.272 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:43.273 7fb6d7fff700 1 --2- 172.21.2.102:0/3177134410 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb6d80a0420 0x7fb6d80a0820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:43.272 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:43.273 7fb6d7fff700 1 -- 172.21.2.102:0/3177134410 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb6d8189d00 msgr2=0x7fb6d818a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:43.272 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:43.273 7fb6d7fff700 1 --2- 172.21.2.102:0/3177134410 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb6d8189d00 0x7fb6d818a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:43.273 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:43.273 7fb6d7fff700 1 -- 172.21.2.102:0/3177134410 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6d80b8200 con 0x7fb6d818a720 2021-02-10T11:21:43.273 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:43.273 7fb6d6ffd700 1 --2- 172.21.2.102:0/3177134410 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb6d8189d00 0x7fb6d818a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:21:43.273 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:43.273 7fb6d77fe700 1 --2- 172.21.2.102:0/3177134410 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb6d80a0420 0x7fb6d80a0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:21:43.273 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:43.273 7fb6d7fff700 1 --2- 172.21.2.102:0/3177134410 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb6d818a720 0x7fb6d80b7be0 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 rx=0x7fb6d000b9b0 tx=0x7fb6d000bd50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:43.273 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:43.273 7fb6bffff700 1 -- 172.21.2.102:0/3177134410 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb6d0003030 con 0x7fb6d818a720 2021-02-10T11:21:43.273 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:43.274 7fb6bffff700 1 -- 172.21.2.102:0/3177134410 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6d000c2f0 con 0x7fb6d818a720 2021-02-10T11:21:43.273 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:43.274 7fb6bffff700 1 -- 172.21.2.102:0/3177134410 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb6d000c810 con 0x7fb6d818a720 2021-02-10T11:21:43.273 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:43.274 7fb6ddbff700 1 -- 172.21.2.102:0/3177134410 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6d80b8200 con 0x7fb6d818a720 2021-02-10T11:21:43.273 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:43.274 7fb6ddbff700 1 -- 172.21.2.102:0/3177134410 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6d80b8200 con 0x7fb6d818a720 2021-02-10T11:21:43.273 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:43.274 7fb6bffff700 1 -- 172.21.2.102:0/3177134410 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fb6d0009450 con 0x7fb6d818a720 2021-02-10T11:21:43.273 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:43.274 7fb6bffff700 1 --2- 172.21.2.102:0/3177134410 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb6b80186c0 0x7fb6b801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:43.273 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:43.274 7fb6bffff700 1 -- 172.21.2.102:0/3177134410 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fb6d0032970 con 0x7fb6d818a720 2021-02-10T11:21:43.273 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:43.274 7fb6d77fe700 1 --2- 172.21.2.102:0/3177134410 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb6b80186c0 0x7fb6b801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:43.274 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:43.275 7fb6d77fe700 1 --2- 172.21.2.102:0/3177134410 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb6b80186c0 0x7fb6b801abc0 crc :-1 s=READY pgs=478 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:43.300 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:43.304 7fb6ddbff700 1 -- 172.21.2.102:0/3177134410 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6d8105ae0 con 0x7fb6d818a720 2021-02-10T11:21:43.303 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:43.307 7fb6bffff700 1 -- 172.21.2.102:0/3177134410 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fb6d0000f90 con 0x7fb6d818a720 2021-02-10T11:21:43.435 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:43.439 7fb6ddbff700 1 -- 172.21.2.102:0/3177134410 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344455", "target": ["mgr", ""]}) v1 -- 0x7fb6d80f0b50 con 0x7fb6b80186c0 2021-02-10T11:21:43.445 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:43.449 7fb6bffff700 1 -- 172.21.2.102:0/3177134410 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fb6c8000960 con 0x7fb6b80186c0 2021-02-10T11:21:43.452 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:43.456 7fb6ddbff700 1 -- 172.21.2.102:0/3177134410 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb6b80186c0 msgr2=0x7fb6b801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:43.452 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:43.456 7fb6ddbff700 1 --2- 172.21.2.102:0/3177134410 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb6b80186c0 0x7fb6b801abc0 crc :-1 s=READY pgs=478 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:43.453 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:43.456 7fb6ddbff700 1 -- 172.21.2.102:0/3177134410 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb6d818a720 msgr2=0x7fb6d80b7be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:43.453 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:43.456 7fb6ddbff700 1 --2- 172.21.2.102:0/3177134410 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb6d818a720 0x7fb6d80b7be0 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 rx=0x7fb6d000b9b0 tx=0x7fb6d000bd50).stop 2021-02-10T11:21:43.453 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:43.456 7fb6ddbff700 1 -- 172.21.2.102:0/3177134410 shutdown_connections 2021-02-10T11:21:43.453 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:43.456 7fb6ddbff700 1 --2- 172.21.2.102:0/3177134410 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb6d818a720 0x7fb6d80b7be0 unknown :-1 s=CLOSED pgs=981 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:43.453 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:43.456 7fb6ddbff700 1 --2- 172.21.2.102:0/3177134410 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb6b80186c0 0x7fb6b801abc0 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:43.453 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:43.456 7fb6ddbff700 1 --2- 172.21.2.102:0/3177134410 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb6d8189d00 0x7fb6d818a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:43.453 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:43.456 7fb6ddbff700 1 --2- 172.21.2.102:0/3177134410 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb6d80a0420 0x7fb6d80a0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:43.453 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:43.456 7fb6ddbff700 1 -- 172.21.2.102:0/3177134410 >> 172.21.2.102:0/3177134410 conn(0x7fb6d8094e10 msgr2=0x7fb6d80a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:43.453 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:43.456 7fb6ddbff700 1 -- 172.21.2.102:0/3177134410 shutdown_connections 2021-02-10T11:21:43.453 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:43.456 7fb6ddbff700 1 -- 172.21.2.102:0/3177134410 wait complete. 2021-02-10T11:21:43.461 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000344456 2021-02-10T11:21:43.891 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:43.879 7fb05a38c700 1 Processor -- start 2021-02-10T11:21:43.891 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:43.879 7fb05a38c700 1 -- start start 2021-02-10T11:21:43.892 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:43.879 7fb05a38c700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fb0541b0870 0x7fb0540f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:43.892 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:43.879 7fb05a38c700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb0540efe00 0x7fb0540f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:43.892 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:43.879 7fb05a38c700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0541b0c70 con 0x7fb0541b0470 2021-02-10T11:21:43.892 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:43.880 7fb05a38c700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0540f0b50 con 0x7fb0541b0870 2021-02-10T11:21:43.892 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:43.880 7fb05a38c700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0540f0820 con 0x7fb0540efe00 2021-02-10T11:21:43.892 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:43.880 7fb053fff700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fb0541b0470 0x7fb0540ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:47174/0 (socket says 172.21.2.102:47174) 2021-02-10T11:21:43.892 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:43.880 7fb053fff700 1 -- 172.21.2.102:0/449848773 learned_addr learned my addr 172.21.2.102:0/449848773 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:21:43.892 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:43.880 7fb0537fe700 1 --2- 172.21.2.102:0/449848773 >> v2:172.21.2.102:3300/0 conn(0x7fb0541b0870 0x7fb0540f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:43.892 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:43.880 7fb0589a9700 1 --2- 172.21.2.102:0/449848773 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb0540efe00 0x7fb0540f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:43.892 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:43.880 7fb0537fe700 1 -- 172.21.2.102:0/449848773 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb0540efe00 msgr2=0x7fb0540f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:21:43.892 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:43.880 7fb0537fe700 1 --2- 172.21.2.102:0/449848773 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb0540efe00 0x7fb0540f0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:43.893 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:43.880 7fb0537fe700 1 -- 172.21.2.102:0/449848773 >> v1:172.21.2.102:6789/0 conn(0x7fb0541b0470 legacy=0x7fb0540ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:43.893 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:43.880 7fb052ffd700 1 -- 172.21.2.102:0/449848773 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 925507511 0 0) 0x7fb04c001500 con 0x7fb0541b0470 2021-02-10T11:21:43.893 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:43.880 7fb0537fe700 1 -- 172.21.2.102:0/449848773 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0540e52e0 con 0x7fb0541b0870 2021-02-10T11:21:43.893 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:43.880 7fb0537fe700 1 --2- 172.21.2.102:0/449848773 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb0541b0870 0x7fb0540f20e0 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 rx=0x7fb048006650 tx=0x7fb048009a80).ready entity=mon.0 client_cookie=22632d03b0b47415 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:43.893 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:43.880 7fb052ffd700 1 -- 172.21.2.102:0/449848773 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb0480009b0 con 0x7fb0541b0870 2021-02-10T11:21:43.893 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:43.880 7fb052ffd700 1 -- 172.21.2.102:0/449848773 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb04800c0c0 con 0x7fb0541b0870 2021-02-10T11:21:43.893 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:43.880 7fb052ffd700 1 -- 172.21.2.102:0/449848773 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb0480009b0 con 0x7fb0541b0870 2021-02-10T11:21:43.893 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:43.881 7fb05a38c700 1 -- 172.21.2.102:0/449848773 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb0541b0870 msgr2=0x7fb0540f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:43.893 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:43.881 7fb05a38c700 1 --2- 172.21.2.102:0/449848773 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb0541b0870 0x7fb0540f20e0 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 rx=0x7fb048006650 tx=0x7fb048009a80).stop 2021-02-10T11:21:43.893 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:43.881 7fb05a38c700 1 -- 172.21.2.102:0/449848773 shutdown_connections 2021-02-10T11:21:43.893 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:43.881 7fb05a38c700 1 --2- 172.21.2.102:0/449848773 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb0541b0870 0x7fb0540f20e0 unknown :-1 s=CLOSED pgs=983 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:43.893 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:43.881 7fb05a38c700 1 --2- 172.21.2.102:0/449848773 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb0540efe00 0x7fb0540f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:43.893 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:43.881 7fb05a38c700 1 -- 172.21.2.102:0/449848773 >> 172.21.2.102:0/449848773 conn(0x7fb0540ce3e0 msgr2=0x7fb0540e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:43.894 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:43.881 7fb05a38c700 1 -- 172.21.2.102:0/449848773 shutdown_connections 2021-02-10T11:21:43.894 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:43.881 7fb05a38c700 1 -- 172.21.2.102:0/449848773 wait complete. 2021-02-10T11:21:43.895 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:43.896 7fb05a38c700 1 Processor -- start 2021-02-10T11:21:43.895 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:43.896 7fb05a38c700 1 -- start start 2021-02-10T11:21:43.896 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:43.896 7fb05a38c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb0540a0420 0x7fb0540a0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:43.896 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:43.896 7fb05a38c700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb054189d00 0x7fb05418a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:43.896 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:43.896 7fb05a38c700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb05418a720 0x7fb0540b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:43.896 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:43.896 7fb05a38c700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb054189780 con 0x7fb0540a0420 2021-02-10T11:21:43.896 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:43.896 7fb05a38c700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb05418ab20 con 0x7fb054189d00 2021-02-10T11:21:43.896 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:43.896 7fb05a38c700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0540a1140 con 0x7fb05418a720 2021-02-10T11:21:43.896 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:43.896 7fb053fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb0540a0420 0x7fb0540a0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:43.896 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:43.897 7fb053fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb0540a0420 0x7fb0540a0820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:54956/0 (socket says 172.21.2.102:54956) 2021-02-10T11:21:43.896 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:43.897 7fb053fff700 1 -- 172.21.2.102:0/3405313571 learned_addr learned my addr 172.21.2.102:0/3405313571 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:21:43.896 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:43.897 7fb0537fe700 1 --2- 172.21.2.102:0/3405313571 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb054189d00 0x7fb05418a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:43.896 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:43.897 7fb0589a9700 1 --2- 172.21.2.102:0/3405313571 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb05418a720 0x7fb0540b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:43.896 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:43.897 7fb053fff700 1 -- 172.21.2.102:0/3405313571 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb05418a720 msgr2=0x7fb0540b7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:43.896 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:43.897 7fb053fff700 1 --2- 172.21.2.102:0/3405313571 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb05418a720 0x7fb0540b7be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:43.897 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:43.897 7fb053fff700 1 -- 172.21.2.102:0/3405313571 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb054189d00 msgr2=0x7fb05418a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:43.897 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:43.897 7fb053fff700 1 --2- 172.21.2.102:0/3405313571 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb054189d00 0x7fb05418a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:43.897 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:43.897 7fb053fff700 1 -- 172.21.2.102:0/3405313571 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0540b8200 con 0x7fb0540a0420 2021-02-10T11:21:43.897 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:43.897 7fb053fff700 1 --2- 172.21.2.102:0/3405313571 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb0540a0420 0x7fb0540a0820 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 rx=0x7fb04c00a940 tx=0x7fb04c00ace0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:43.897 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:43.897 7fb050ff9700 1 -- 172.21.2.102:0/3405313571 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb04c001010 con 0x7fb0540a0420 2021-02-10T11:21:43.897 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:43.897 7fb050ff9700 1 -- 172.21.2.102:0/3405313571 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb04c007220 con 0x7fb0540a0420 2021-02-10T11:21:43.897 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:43.897 7fb050ff9700 1 -- 172.21.2.102:0/3405313571 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb04c0075f0 con 0x7fb0540a0420 2021-02-10T11:21:43.897 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:43.897 7fb05a38c700 1 -- 172.21.2.102:0/3405313571 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0540b8200 con 0x7fb0540a0420 2021-02-10T11:21:43.897 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:43.897 7fb05a38c700 1 -- 172.21.2.102:0/3405313571 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0540b8200 con 0x7fb0540a0420 2021-02-10T11:21:43.897 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:43.898 7fb050ff9700 1 -- 172.21.2.102:0/3405313571 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fb04c007400 con 0x7fb0540a0420 2021-02-10T11:21:43.897 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:43.898 7fb050ff9700 1 --2- 172.21.2.102:0/3405313571 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb034018730 0x7fb03401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:43.897 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:43.898 7fb050ff9700 1 -- 172.21.2.102:0/3405313571 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fb04c0329a0 con 0x7fb0540a0420 2021-02-10T11:21:43.897 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:43.898 7fb0537fe700 1 --2- 172.21.2.102:0/3405313571 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb034018730 0x7fb03401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:43.898 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:43.898 7fb0537fe700 1 --2- 172.21.2.102:0/3405313571 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb034018730 0x7fb03401ac10 crc :-1 s=READY pgs=479 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:43.924 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:43.929 7fb05a38c700 1 -- 172.21.2.102:0/3405313571 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb054105ae0 con 0x7fb0540a0420 2021-02-10T11:21:43.927 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:43.931 7fb050ff9700 1 -- 172.21.2.102:0/3405313571 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fb04c007a90 con 0x7fb0540a0420 2021-02-10T11:21:44.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:44.059 7fb05a38c700 1 -- 172.21.2.102:0/3405313571 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344456", "target": ["mgr", ""]}) v1 -- 0x7fb0540f0b50 con 0x7fb034018730 2021-02-10T11:21:44.066 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:44.071 7fb050ff9700 1 -- 172.21.2.102:0/3405313571 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fb048000c90 con 0x7fb034018730 2021-02-10T11:21:44.073 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:44.076 7fb05a38c700 1 -- 172.21.2.102:0/3405313571 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb034018730 msgr2=0x7fb03401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:44.073 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:44.076 7fb05a38c700 1 --2- 172.21.2.102:0/3405313571 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb034018730 0x7fb03401ac10 crc :-1 s=READY pgs=479 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:44.073 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:44.076 7fb05a38c700 1 -- 172.21.2.102:0/3405313571 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb0540a0420 msgr2=0x7fb0540a0820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:44.073 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:44.076 7fb05a38c700 1 --2- 172.21.2.102:0/3405313571 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb0540a0420 0x7fb0540a0820 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 rx=0x7fb04c00a940 tx=0x7fb04c00ace0).stop 2021-02-10T11:21:44.073 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:44.076 7fb05a38c700 1 -- 172.21.2.102:0/3405313571 shutdown_connections 2021-02-10T11:21:44.073 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:44.076 7fb05a38c700 1 --2- 172.21.2.102:0/3405313571 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb05418a720 0x7fb0540b7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:44.073 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:44.076 7fb05a38c700 1 --2- 172.21.2.102:0/3405313571 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb034018730 0x7fb03401ac10 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:44.073 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:44.076 7fb05a38c700 1 --2- 172.21.2.102:0/3405313571 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb054189d00 0x7fb05418a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:44.073 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:44.076 7fb05a38c700 1 --2- 172.21.2.102:0/3405313571 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb0540a0420 0x7fb0540a0820 unknown :-1 s=CLOSED pgs=984 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:44.073 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:44.076 7fb05a38c700 1 -- 172.21.2.102:0/3405313571 >> 172.21.2.102:0/3405313571 conn(0x7fb054094e10 msgr2=0x7fb0540a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:44.073 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:44.077 7fb05a38c700 1 -- 172.21.2.102:0/3405313571 shutdown_connections 2021-02-10T11:21:44.074 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:44.077 7fb05a38c700 1 -- 172.21.2.102:0/3405313571 wait complete. 2021-02-10T11:21:44.081 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000344457 2021-02-10T11:21:44.512 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:44.499 7f3f1f84f700 1 Processor -- start 2021-02-10T11:21:44.512 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:44.500 7f3f1f84f700 1 -- start start 2021-02-10T11:21:44.512 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:44.500 7f3f1f84f700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f3f181b0470 0x7f3f181b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:44.513 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:44.500 7f3f1f84f700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f3f180ef610 0x7f3f180efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:44.513 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:44.500 7f3f1f84f700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3f180f0030 0x7f3f180f0430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:44.513 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:44.500 7f3f1f84f700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f180e5ea0 con 0x7f3f181b0470 2021-02-10T11:21:44.513 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:44.500 7f3f1f84f700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f180f0b50 con 0x7f3f180ef610 2021-02-10T11:21:44.513 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:44.500 7f3f1f84f700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f180e52e0 con 0x7f3f180f0030 2021-02-10T11:21:44.513 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:44.500 7f3f1d66b700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f3f181b0470 0x7f3f181b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:44.513 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:44.500 7f3f1d66b700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f3f181b0470 0x7f3f181b0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:54964/0 (socket says 172.21.2.102:54964) 2021-02-10T11:21:44.513 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:44.500 7f3f1d66b700 1 -- 172.21.2.102:0/3401382381 learned_addr learned my addr 172.21.2.102:0/3401382381 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:21:44.513 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:44.500 7f3f1ce6a700 1 --2- 172.21.2.102:0/3401382381 >> v2:172.21.2.113:3300/0 conn(0x7f3f180ef610 0x7f3f180efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:44.513 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:44.500 7f3f1de6c700 1 --2- 172.21.2.102:0/3401382381 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3f180f0030 0x7f3f180f0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:44.513 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:44.500 7f3f1d66b700 1 -- 172.21.2.102:0/3401382381 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3f180f0030 msgr2=0x7f3f180f0430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:21:44.514 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:44.500 7f3f1d66b700 1 --2- 172.21.2.102:0/3401382381 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3f180f0030 0x7f3f180f0430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:44.514 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:44.500 7f3f1d66b700 1 -- 172.21.2.102:0/3401382381 >> v2:172.21.2.113:3300/0 conn(0x7f3f180ef610 msgr2=0x7f3f180efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:21:44.514 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:44.500 7f3f1d66b700 1 --2- 172.21.2.102:0/3401382381 >> v2:172.21.2.113:3300/0 conn(0x7f3f180ef610 0x7f3f180efa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:44.514 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:44.500 7f3f1d66b700 1 -- 172.21.2.102:0/3401382381 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f180f10d0 con 0x7f3f181b0470 2021-02-10T11:21:44.514 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:44.501 7f3f1de6c700 1 --2- 172.21.2.102:0/3401382381 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3f180f0030 0x7f3f180f0430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:21:44.514 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:44.501 7f3f1d66b700 1 --2- 172.21.2.102:0/3401382381 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3f181b0470 0x7f3f181b0870 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 rx=0x7f3f0c006700 tx=0x7f3f0c009b80).ready entity=mon.0 client_cookie=840ce55f451c3d74 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:44.514 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:44.501 7f3f0bfff700 1 -- 172.21.2.102:0/3401382381 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3f0c00c0c0 con 0x7f3f181b0470 2021-02-10T11:21:44.514 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:44.501 7f3f0bfff700 1 -- 172.21.2.102:0/3401382381 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f0c00c3c0 con 0x7f3f181b0470 2021-02-10T11:21:44.514 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:44.501 7f3f0bfff700 1 -- 172.21.2.102:0/3401382381 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3f0c00c8e0 con 0x7f3f181b0470 2021-02-10T11:21:44.514 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:44.501 7f3f1f84f700 1 -- 172.21.2.102:0/3401382381 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3f181b0470 msgr2=0x7f3f181b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:44.514 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:44.501 7f3f1f84f700 1 --2- 172.21.2.102:0/3401382381 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3f181b0470 0x7f3f181b0870 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 rx=0x7f3f0c006700 tx=0x7f3f0c009b80).stop 2021-02-10T11:21:44.514 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:44.502 7f3f1f84f700 1 -- 172.21.2.102:0/3401382381 shutdown_connections 2021-02-10T11:21:44.515 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:44.502 7f3f1f84f700 1 --2- 172.21.2.102:0/3401382381 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3f180f0030 0x7f3f180f0430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:44.515 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:44.502 7f3f1f84f700 1 --2- 172.21.2.102:0/3401382381 >> v2:172.21.2.113:3300/0 conn(0x7f3f180ef610 0x7f3f180efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:44.515 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:44.502 7f3f1f84f700 1 --2- 172.21.2.102:0/3401382381 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3f181b0470 0x7f3f181b0870 unknown :-1 s=CLOSED pgs=985 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:44.515 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:44.502 7f3f1f84f700 1 -- 172.21.2.102:0/3401382381 >> 172.21.2.102:0/3401382381 conn(0x7f3f180ce3e0 msgr2=0x7f3f180e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:44.515 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:44.502 7f3f1f84f700 1 -- 172.21.2.102:0/3401382381 shutdown_connections 2021-02-10T11:21:44.515 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:44.502 7f3f1f84f700 1 -- 172.21.2.102:0/3401382381 wait complete. 2021-02-10T11:21:44.516 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:44.517 7f3f1f84f700 1 Processor -- start 2021-02-10T11:21:44.516 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:44.517 7f3f1f84f700 1 -- start start 2021-02-10T11:21:44.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:44.517 7f3f1f84f700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3f180924b0 0x7f3f181a3590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:44.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:44.517 7f3f1f84f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3f181a3bb0 0x7f3f180bf920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:44.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:44.517 7f3f1f84f700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3f180bff40 0x7f3f180c0340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:44.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:44.517 7f3f1f84f700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f1809a0a0 con 0x7f3f181a3bb0 2021-02-10T11:21:44.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:44.517 7f3f1f84f700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f180c0960 con 0x7f3f180bff40 2021-02-10T11:21:44.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:44.517 7f3f1f84f700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f18098420 con 0x7f3f180924b0 2021-02-10T11:21:44.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:44.517 7f3f1ce6a700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3f181a3bb0 0x7f3f180bf920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:44.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:44.517 7f3f1ce6a700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3f181a3bb0 0x7f3f180bf920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:54970/0 (socket says 172.21.2.102:54970) 2021-02-10T11:21:44.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:44.517 7f3f1ce6a700 1 -- 172.21.2.102:0/3896242105 learned_addr learned my addr 172.21.2.102:0/3896242105 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:21:44.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:44.518 7f3f1ce6a700 1 -- 172.21.2.102:0/3896242105 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3f180924b0 msgr2=0x7f3f181a3590 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:21:44.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:44.518 7f3f1ce6a700 1 --2- 172.21.2.102:0/3896242105 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3f180924b0 0x7f3f181a3590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:21:44.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:44.518 7f3f1ce6a700 1 -- 172.21.2.102:0/3896242105 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3f180bff40 msgr2=0x7f3f180c0340 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:21:44.518 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:44.518 7f3f1ce6a700 1 --2- 172.21.2.102:0/3896242105 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3f180bff40 0x7f3f180c0340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:21:44.518 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:44.518 7f3f1ce6a700 1 -- 172.21.2.102:0/3896242105 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f18098640 con 0x7f3f181a3bb0 2021-02-10T11:21:44.518 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:44.518 7f3f1ce6a700 1 --2- 172.21.2.102:0/3896242105 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3f181a3bb0 0x7f3f180bf920 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 rx=0x7f3f14006f60 tx=0x7f3f1400b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:44.518 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:44.518 7f3f09ffb700 1 -- 172.21.2.102:0/3896242105 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3f140009b0 con 0x7f3f181a3bb0 2021-02-10T11:21:44.518 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:44.518 7f3f09ffb700 1 -- 172.21.2.102:0/3896242105 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f1400bd60 con 0x7f3f181a3bb0 2021-02-10T11:21:44.518 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:44.518 7f3f1f84f700 1 -- 172.21.2.102:0/3896242105 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f18098640 con 0x7f3f181a3bb0 2021-02-10T11:21:44.518 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:44.518 7f3f09ffb700 1 -- 172.21.2.102:0/3896242105 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3f14002030 con 0x7f3f181a3bb0 2021-02-10T11:21:44.518 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:44.518 7f3f1f84f700 1 -- 172.21.2.102:0/3896242105 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f18098bd0 con 0x7f3f181a3bb0 2021-02-10T11:21:44.518 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:44.518 7f3f09ffb700 1 -- 172.21.2.102:0/3896242105 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f3f14000c90 con 0x7f3f181a3bb0 2021-02-10T11:21:44.518 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:44.519 7f3f09ffb700 1 --2- 172.21.2.102:0/3896242105 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3ef80186c0 0x7f3ef801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:44.518 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:44.519 7f3f09ffb700 1 -- 172.21.2.102:0/3896242105 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f3f140327c0 con 0x7f3f181a3bb0 2021-02-10T11:21:44.518 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:44.519 7f3f1d66b700 1 --2- 172.21.2.102:0/3896242105 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3ef80186c0 0x7f3ef801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:44.519 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:44.519 7f3f1d66b700 1 --2- 172.21.2.102:0/3896242105 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3ef80186c0 0x7f3ef801abc0 crc :-1 s=READY pgs=480 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:44.546 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:44.549 7f3f1f84f700 1 -- 172.21.2.102:0/3896242105 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f18105b40 con 0x7f3f181a3bb0 2021-02-10T11:21:44.548 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:44.552 7f3f09ffb700 1 -- 172.21.2.102:0/3896242105 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f3f140009b0 con 0x7f3f181a3bb0 2021-02-10T11:21:44.674 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:44.678 7f3f1f84f700 1 -- 172.21.2.102:0/3896242105 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344457", "target": ["mgr", ""]}) v1 -- 0x7f3f18098bd0 con 0x7f3ef80186c0 2021-02-10T11:21:44.685 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:44.690 7f3f09ffb700 1 -- 172.21.2.102:0/3896242105 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f3f0c009390 con 0x7f3ef80186c0 2021-02-10T11:21:44.692 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:44.695 7f3f1f84f700 1 -- 172.21.2.102:0/3896242105 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3ef80186c0 msgr2=0x7f3ef801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:44.692 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:44.695 7f3f1f84f700 1 --2- 172.21.2.102:0/3896242105 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3ef80186c0 0x7f3ef801abc0 crc :-1 s=READY pgs=480 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:44.692 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:44.695 7f3f1f84f700 1 -- 172.21.2.102:0/3896242105 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3f181a3bb0 msgr2=0x7f3f180bf920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:44.692 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:44.695 7f3f1f84f700 1 --2- 172.21.2.102:0/3896242105 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3f181a3bb0 0x7f3f180bf920 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 rx=0x7f3f14006f60 tx=0x7f3f1400b970).stop 2021-02-10T11:21:44.692 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:44.696 7f3f1f84f700 1 -- 172.21.2.102:0/3896242105 shutdown_connections 2021-02-10T11:21:44.692 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:44.696 7f3f1f84f700 1 --2- 172.21.2.102:0/3896242105 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3ef80186c0 0x7f3ef801abc0 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:44.692 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:44.696 7f3f1f84f700 1 --2- 172.21.2.102:0/3896242105 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3f180bff40 0x7f3f180c0340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:21:44.692 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:44.696 7f3f1f84f700 1 --2- 172.21.2.102:0/3896242105 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3f181a3bb0 0x7f3f180bf920 unknown :-1 s=CLOSED pgs=986 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:44.692 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:44.696 7f3f1f84f700 1 --2- 172.21.2.102:0/3896242105 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3f180924b0 0x7f3f181a3590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:21:44.692 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:44.696 7f3f1f84f700 1 -- 172.21.2.102:0/3896242105 >> 172.21.2.102:0/3896242105 conn(0x7f3f180b50c0 msgr2=0x7f3f18092d90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:44.692 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:44.696 7f3f1f84f700 1 -- 172.21.2.102:0/3896242105 shutdown_connections 2021-02-10T11:21:44.692 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:44.696 7f3f1f84f700 1 -- 172.21.2.102:0/3896242105 wait complete. 2021-02-10T11:21:44.701 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000344458 2021-02-10T11:21:45.132 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:45.120 7f9296919700 1 Processor -- start 2021-02-10T11:21:45.132 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:45.121 7f9296919700 1 -- start start 2021-02-10T11:21:45.133 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:45.121 7f9296919700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f92901b0870 0x7f92900f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:45.133 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:45.121 7f9296919700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92901b0c70 con 0x7f92900efe00 2021-02-10T11:21:45.133 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:45.121 7f9296919700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92900f0b50 con 0x7f92901b0470 2021-02-10T11:21:45.133 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:45.121 7f9296919700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92900e5ea0 con 0x7f92901b0870 2021-02-10T11:21:45.133 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:45.121 7f928f7fe700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f92901b0870 0x7f92900f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:45.133 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:45.121 7f9294f36700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f92900efe00 0x7f92900f0200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:47206/0 (socket says 172.21.2.102:47206) 2021-02-10T11:21:45.133 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:45.121 7f9294f36700 1 -- 172.21.2.102:0/2786292228 learned_addr learned my addr 172.21.2.102:0/2786292228 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:21:45.133 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:45.121 7f928f7fe700 1 -- 172.21.2.102:0/2786292228 >> v1:172.21.2.113:6789/0 conn(0x7f92901b0470 legacy=0x7f92900ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:45.133 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:45.121 7f928f7fe700 1 -- 172.21.2.102:0/2786292228 >> v1:172.21.2.102:6789/0 conn(0x7f92900efe00 legacy=0x7f92900f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:45.133 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:45.121 7f928f7fe700 1 -- 172.21.2.102:0/2786292228 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92900d2fb0 con 0x7f92901b0870 2021-02-10T11:21:45.133 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:45.122 7f928f7fe700 1 --2- 172.21.2.102:0/2786292228 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f92901b0870 0x7f92900f20e0 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 rx=0x7f9284006650 tx=0x7f9284009a80).ready entity=mon.0 client_cookie=84d442a183763afa server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:45.133 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:45.122 7f928effd700 1 -- 172.21.2.102:0/2786292228 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f92840009b0 con 0x7f92901b0870 2021-02-10T11:21:45.134 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:45.122 7f928effd700 1 -- 172.21.2.102:0/2786292228 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f928400c0c0 con 0x7f92901b0870 2021-02-10T11:21:45.134 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:45.122 7f928effd700 1 -- 172.21.2.102:0/2786292228 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f928400c520 con 0x7f92901b0870 2021-02-10T11:21:45.134 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:45.122 7f9296919700 1 -- 172.21.2.102:0/2786292228 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f92901b0870 msgr2=0x7f92900f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:45.134 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:45.122 7f9296919700 1 --2- 172.21.2.102:0/2786292228 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f92901b0870 0x7f92900f20e0 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 rx=0x7f9284006650 tx=0x7f9284009a80).stop 2021-02-10T11:21:45.134 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:45.122 7f9296919700 1 -- 172.21.2.102:0/2786292228 shutdown_connections 2021-02-10T11:21:45.134 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:45.122 7f9296919700 1 --2- 172.21.2.102:0/2786292228 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f92901b0870 0x7f92900f20e0 unknown :-1 s=CLOSED pgs=988 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:45.134 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:45.122 7f9296919700 1 -- 172.21.2.102:0/2786292228 >> 172.21.2.102:0/2786292228 conn(0x7f92900ce3e0 msgr2=0x7f92900e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:45.134 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:45.122 7f9296919700 1 -- 172.21.2.102:0/2786292228 shutdown_connections 2021-02-10T11:21:45.134 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:45.122 7f9296919700 1 -- 172.21.2.102:0/2786292228 wait complete. 2021-02-10T11:21:45.136 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:45.137 7f9296919700 1 Processor -- start 2021-02-10T11:21:45.136 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:45.137 7f9296919700 1 -- start start 2021-02-10T11:21:45.136 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:45.137 7f9296919700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f92901c2580 0x7f92900c2e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:45.136 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:45.137 7f9296919700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f92900c34a0 0x7f92900a9800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:45.136 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:45.137 7f9296919700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f92900881a0 0x7f92900a9e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:45.136 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:45.137 7f9296919700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92900a9280 con 0x7f92901c2580 2021-02-10T11:21:45.136 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:45.137 7f9296919700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92900913f0 con 0x7f92900881a0 2021-02-10T11:21:45.136 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:45.137 7f9296919700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92900aa440 con 0x7f92900c34a0 2021-02-10T11:21:45.136 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:45.137 7f928ffff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f92901c2580 0x7f92900c2e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:45.136 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:45.137 7f928ffff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f92901c2580 0x7f92900c2e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:54986/0 (socket says 172.21.2.102:54986) 2021-02-10T11:21:45.137 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:45.137 7f928ffff700 1 -- 172.21.2.102:0/2338572051 learned_addr learned my addr 172.21.2.102:0/2338572051 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:21:45.137 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:45.138 7f9294f36700 1 --2- 172.21.2.102:0/2338572051 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f92900881a0 0x7f92900a9e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:45.137 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:45.138 7f928f7fe700 1 --2- 172.21.2.102:0/2338572051 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f92900c34a0 0x7f92900a9800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:45.137 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:45.138 7f928ffff700 1 -- 172.21.2.102:0/2338572051 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f92900c34a0 msgr2=0x7f92900a9800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:45.137 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:45.138 7f928ffff700 1 --2- 172.21.2.102:0/2338572051 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f92900c34a0 0x7f92900a9800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:45.137 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:45.138 7f928ffff700 1 -- 172.21.2.102:0/2338572051 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f92900881a0 msgr2=0x7f92900a9e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:45.137 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:45.138 7f928ffff700 1 --2- 172.21.2.102:0/2338572051 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f92900881a0 0x7f92900a9e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:45.137 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:45.138 7f928ffff700 1 -- 172.21.2.102:0/2338572051 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92900aa660 con 0x7f92901c2580 2021-02-10T11:21:45.137 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:45.138 7f928f7fe700 1 --2- 172.21.2.102:0/2338572051 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f92900c34a0 0x7f92900a9800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:21:45.137 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:45.138 7f928ffff700 1 --2- 172.21.2.102:0/2338572051 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f92901c2580 0x7f92900c2e80 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 rx=0x7f9288009940 tx=0x7f928800bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:45.137 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:45.138 7f928cff9700 1 -- 172.21.2.102:0/2338572051 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9288000ce0 con 0x7f92901c2580 2021-02-10T11:21:45.137 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:45.138 7f928cff9700 1 -- 172.21.2.102:0/2338572051 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9288000fe0 con 0x7f92901c2580 2021-02-10T11:21:45.138 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:45.138 7f9296919700 1 -- 172.21.2.102:0/2338572051 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92900aa660 con 0x7f92901c2580 2021-02-10T11:21:45.138 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:45.138 7f9296919700 1 -- 172.21.2.102:0/2338572051 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92900aa660 con 0x7f92901c2580 2021-02-10T11:21:45.138 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:45.138 7f928cff9700 1 -- 172.21.2.102:0/2338572051 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f928800c570 con 0x7f92901c2580 2021-02-10T11:21:45.138 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:45.139 7f928cff9700 1 -- 172.21.2.102:0/2338572051 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f9288001230 con 0x7f92901c2580 2021-02-10T11:21:45.138 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:45.139 7f928cff9700 1 --2- 172.21.2.102:0/2338572051 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f9270018730 0x7f927001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:45.138 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:45.139 7f928cff9700 1 -- 172.21.2.102:0/2338572051 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f9288032b40 con 0x7f92901c2580 2021-02-10T11:21:45.138 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:45.139 7f928f7fe700 1 --2- 172.21.2.102:0/2338572051 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f9270018730 0x7f927001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:45.138 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:45.139 7f928f7fe700 1 --2- 172.21.2.102:0/2338572051 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f9270018730 0x7f927001abc0 crc :-1 s=READY pgs=481 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:45.166 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:45.170 7f9296919700 1 -- 172.21.2.102:0/2338572051 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f929007e710 con 0x7f92901c2580 2021-02-10T11:21:45.169 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:45.173 7f928cff9700 1 -- 172.21.2.102:0/2338572051 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f928800be30 con 0x7f92901c2580 2021-02-10T11:21:45.297 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:45.301 7f9296919700 1 -- 172.21.2.102:0/2338572051 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344458", "target": ["mgr", ""]}) v1 -- 0x7f92900913f0 con 0x7f9270018730 2021-02-10T11:21:45.308 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:45.312 7f928cff9700 1 -- 172.21.2.102:0/2338572051 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f9284000940 con 0x7f9270018730 2021-02-10T11:21:45.314 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:45.318 7f9296919700 1 -- 172.21.2.102:0/2338572051 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f9270018730 msgr2=0x7f927001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:45.314 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:45.318 7f9296919700 1 --2- 172.21.2.102:0/2338572051 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f9270018730 0x7f927001abc0 crc :-1 s=READY pgs=481 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:45.314 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:45.318 7f9296919700 1 -- 172.21.2.102:0/2338572051 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f92901c2580 msgr2=0x7f92900c2e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:45.314 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:45.318 7f9296919700 1 --2- 172.21.2.102:0/2338572051 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f92901c2580 0x7f92900c2e80 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 rx=0x7f9288009940 tx=0x7f928800bcf0).stop 2021-02-10T11:21:45.314 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:45.318 7f9296919700 1 -- 172.21.2.102:0/2338572051 shutdown_connections 2021-02-10T11:21:45.314 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:45.318 7f9296919700 1 --2- 172.21.2.102:0/2338572051 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f9270018730 0x7f927001abc0 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:45.314 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:45.318 7f9296919700 1 --2- 172.21.2.102:0/2338572051 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f92900881a0 0x7f92900a9e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:45.314 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:45.318 7f9296919700 1 --2- 172.21.2.102:0/2338572051 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f92900c34a0 0x7f92900a9800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:45.314 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:45.318 7f9296919700 1 --2- 172.21.2.102:0/2338572051 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f92901c2580 0x7f92900c2e80 unknown :-1 s=CLOSED pgs=989 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:45.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:45.318 7f9296919700 1 -- 172.21.2.102:0/2338572051 >> 172.21.2.102:0/2338572051 conn(0x7f9290088730 msgr2=0x7f92900c1860 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:45.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:45.318 7f9296919700 1 -- 172.21.2.102:0/2338572051 shutdown_connections 2021-02-10T11:21:45.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:45.318 7f9296919700 1 -- 172.21.2.102:0/2338572051 wait complete. 2021-02-10T11:21:45.323 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000344459 2021-02-10T11:21:45.757 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:45.745 7f2d00c45700 1 Processor -- start 2021-02-10T11:21:45.757 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:45.745 7f2d00c45700 1 -- start start 2021-02-10T11:21:45.758 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:45.745 7f2d00c45700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f2cfc1b0870 0x7f2cfc0f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:45.758 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:45.745 7f2d00c45700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cfc1b0c70 con 0x7f2cfc1b0470 2021-02-10T11:21:45.758 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:45.745 7f2d00c45700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cfc0f0b50 con 0x7f2cfc0efe00 2021-02-10T11:21:45.758 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:45.745 7f2d00c45700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2cfc0e5ea0 con 0x7f2cfc1b0870 2021-02-10T11:21:45.758 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:45.746 7f2cfa85e700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f2cfc1b0470 0x7f2cfc0ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:47216/0 (socket says 172.21.2.102:47216) 2021-02-10T11:21:45.758 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:45.746 7f2cfa85e700 1 -- 172.21.2.102:0/3887209314 learned_addr learned my addr 172.21.2.102:0/3887209314 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:21:45.758 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:45.746 7f2cfa05d700 1 --2- 172.21.2.102:0/3887209314 >> v2:172.21.2.113:3300/0 conn(0x7f2cfc1b0870 0x7f2cfc0f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:45.758 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:45.746 7f2cf985c700 1 -- 172.21.2.102:0/3887209314 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4199455073 0 0) 0x7f2cec001180 con 0x7f2cfc1b0470 2021-02-10T11:21:45.758 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:45.746 7f2cf985c700 1 -- 172.21.2.102:0/3887209314 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ce8002fe0 con 0x7f2cfc1b0470 2021-02-10T11:21:45.758 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:45.746 7f2cfa05d700 1 -- 172.21.2.102:0/3887209314 >> v1:172.21.2.113:6789/0 conn(0x7f2cfc0efe00 legacy=0x7f2cfc0f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:45.758 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:45.746 7f2cf985c700 1 -- 172.21.2.102:0/3887209314 <== mon.1 v1:172.21.2.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 860759684 0 0) 0x7f2cf4000f90 con 0x7f2cfc0efe00 2021-02-10T11:21:45.758 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:45.746 7f2cfa05d700 1 -- 172.21.2.102:0/3887209314 >> v1:172.21.2.102:6789/0 conn(0x7f2cfc1b0470 legacy=0x7f2cfc0ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:45.759 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:45.746 7f2cfa05d700 1 -- 172.21.2.102:0/3887209314 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cfc0e52e0 con 0x7f2cfc1b0870 2021-02-10T11:21:45.759 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:45.746 7f2cfa05d700 1 --2- 172.21.2.102:0/3887209314 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2cfc1b0870 0x7f2cfc0f20e0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 rx=0x7f2ce4006650 tx=0x7f2ce4009a00).ready entity=mon.1 client_cookie=59af47ae9e5aa82e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:45.759 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:45.747 7f2cf985c700 1 -- 172.21.2.102:0/3887209314 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2ce4009df0 con 0x7f2cfc1b0870 2021-02-10T11:21:45.759 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:45.747 7f2cf985c700 1 -- 172.21.2.102:0/3887209314 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ce40009b0 con 0x7f2cfc1b0870 2021-02-10T11:21:45.759 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:45.747 7f2cf985c700 1 -- 172.21.2.102:0/3887209314 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2ce400c2b0 con 0x7f2cfc1b0870 2021-02-10T11:21:45.759 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:45.747 7f2d00c45700 1 -- 172.21.2.102:0/3887209314 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2cfc1b0870 msgr2=0x7f2cfc0f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:45.759 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:45.747 7f2d00c45700 1 --2- 172.21.2.102:0/3887209314 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2cfc1b0870 0x7f2cfc0f20e0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 rx=0x7f2ce4006650 tx=0x7f2ce4009a00).stop 2021-02-10T11:21:45.759 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:45.747 7f2d00c45700 1 -- 172.21.2.102:0/3887209314 shutdown_connections 2021-02-10T11:21:45.759 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:45.747 7f2d00c45700 1 --2- 172.21.2.102:0/3887209314 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2cfc1b0870 0x7f2cfc0f20e0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:45.759 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:45.747 7f2d00c45700 1 -- 172.21.2.102:0/3887209314 >> 172.21.2.102:0/3887209314 conn(0x7f2cfc0ce3e0 msgr2=0x7f2cfc0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:45.759 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:45.747 7f2d00c45700 1 -- 172.21.2.102:0/3887209314 shutdown_connections 2021-02-10T11:21:45.759 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:45.748 7f2d00c45700 1 -- 172.21.2.102:0/3887209314 wait complete. 2021-02-10T11:21:45.761 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:45.762 7f2d00c45700 1 Processor -- start 2021-02-10T11:21:45.761 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:45.762 7f2d00c45700 1 -- start start 2021-02-10T11:21:45.761 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:45.762 7f2d00c45700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2cfc0b20b0 0x7f2cfc0ac970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:45.761 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:45.762 7f2d00c45700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2cfc0b24b0 0x7f2cfc191dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:45.761 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:45.762 7f2d00c45700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2cfc0acf90 0x7f2cfc1923e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:45.761 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:45.762 7f2d00c45700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cfc0b1b30 con 0x7f2cfc0b20b0 2021-02-10T11:21:45.761 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:45.762 7f2d00c45700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cfc1a04e0 con 0x7f2cfc0b24b0 2021-02-10T11:21:45.762 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:45.762 7f2d00c45700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cfc1a06c0 con 0x7f2cfc0acf90 2021-02-10T11:21:45.762 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:45.762 7f2cfa85e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2cfc0b20b0 0x7f2cfc0ac970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:45.762 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:45.762 7f2cfa85e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2cfc0b20b0 0x7f2cfc0ac970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:54998/0 (socket says 172.21.2.102:54998) 2021-02-10T11:21:45.762 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:45.762 7f2cfa85e700 1 -- 172.21.2.102:0/1884820891 learned_addr learned my addr 172.21.2.102:0/1884820891 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:21:45.762 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:45.762 7f2cfa05d700 1 --2- 172.21.2.102:0/1884820891 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2cfc0b24b0 0x7f2cfc191dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:45.762 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:45.762 7f2cfb05f700 1 --2- 172.21.2.102:0/1884820891 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2cfc0acf90 0x7f2cfc1923e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:45.762 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:45.763 7f2cfa05d700 1 -- 172.21.2.102:0/1884820891 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2cfc0acf90 msgr2=0x7f2cfc1923e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:45.762 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:45.763 7f2cfa05d700 1 --2- 172.21.2.102:0/1884820891 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2cfc0acf90 0x7f2cfc1923e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:45.762 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:45.763 7f2cfa05d700 1 -- 172.21.2.102:0/1884820891 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2cfc0b20b0 msgr2=0x7f2cfc0ac970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:45.762 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:45.763 7f2cfa05d700 1 --2- 172.21.2.102:0/1884820891 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2cfc0b20b0 0x7f2cfc0ac970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:45.762 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:45.763 7f2cfa05d700 1 -- 172.21.2.102:0/1884820891 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cfc192be0 con 0x7f2cfc0b24b0 2021-02-10T11:21:45.762 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:45.763 7f2cfb05f700 1 --2- 172.21.2.102:0/1884820891 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2cfc0acf90 0x7f2cfc1923e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:21:45.762 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:45.763 7f2cfa05d700 1 --2- 172.21.2.102:0/1884820891 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2cfc0b24b0 0x7f2cfc191dc0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 rx=0x7f2ce4006640 tx=0x7f2ce400b910).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:45.763 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:45.763 7f2cf37fe700 1 -- 172.21.2.102:0/1884820891 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2ce40009c0 con 0x7f2cfc0b24b0 2021-02-10T11:21:45.763 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:45.763 7f2cf37fe700 1 -- 172.21.2.102:0/1884820891 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ce40067f0 con 0x7f2cfc0b24b0 2021-02-10T11:21:45.763 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:45.763 7f2cf37fe700 1 -- 172.21.2.102:0/1884820891 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2ce4002030 con 0x7f2cfc0b24b0 2021-02-10T11:21:45.763 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:45.763 7f2d00c45700 1 -- 172.21.2.102:0/1884820891 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2cfc192be0 con 0x7f2cfc0b24b0 2021-02-10T11:21:45.763 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:45.763 7f2d00c45700 1 -- 172.21.2.102:0/1884820891 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2cfc192be0 con 0x7f2cfc0b24b0 2021-02-10T11:21:45.763 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:45.764 7f2cf37fe700 1 -- 172.21.2.102:0/1884820891 <== mon.1 v2:172.21.2.113:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f2ce4000c90 con 0x7f2cfc0b24b0 2021-02-10T11:21:45.763 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:45.764 7f2cf37fe700 1 --2- 172.21.2.102:0/1884820891 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2ce0018730 0x7f2ce001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:45.763 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:45.764 7f2cf37fe700 1 -- 172.21.2.102:0/1884820891 <== mon.1 v2:172.21.2.113:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f2ce4032720 con 0x7f2cfc0b24b0 2021-02-10T11:21:45.763 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:45.764 7f2cfa85e700 1 --2- 172.21.2.102:0/1884820891 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2ce0018730 0x7f2ce001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:45.763 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:45.764 7f2cfa85e700 1 --2- 172.21.2.102:0/1884820891 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2ce0018730 0x7f2ce001ac10 crc :-1 s=READY pgs=482 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:45.790 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:45.794 7f2d00c45700 1 -- 172.21.2.102:0/1884820891 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2cfc105af0 con 0x7f2cfc0b24b0 2021-02-10T11:21:45.793 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:45.797 7f2cf37fe700 1 -- 172.21.2.102:0/1884820891 <== mon.1 v2:172.21.2.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f2ce4000940 con 0x7f2cfc0b24b0 2021-02-10T11:21:45.923 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:45.927 7f2d00c45700 1 -- 172.21.2.102:0/1884820891 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344459", "target": ["mgr", ""]}) v1 -- 0x7f2cfc1a04e0 con 0x7f2ce0018730 2021-02-10T11:21:45.934 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:45.938 7f2cf37fe700 1 -- 172.21.2.102:0/1884820891 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f2cec001ab0 con 0x7f2ce0018730 2021-02-10T11:21:45.940 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:45.944 7f2d00c45700 1 -- 172.21.2.102:0/1884820891 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2ce0018730 msgr2=0x7f2ce001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:45.940 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:45.944 7f2d00c45700 1 --2- 172.21.2.102:0/1884820891 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2ce0018730 0x7f2ce001ac10 crc :-1 s=READY pgs=482 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:45.940 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:45.944 7f2d00c45700 1 -- 172.21.2.102:0/1884820891 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2cfc0b24b0 msgr2=0x7f2cfc191dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:45.940 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:45.944 7f2d00c45700 1 --2- 172.21.2.102:0/1884820891 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2cfc0b24b0 0x7f2cfc191dc0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 rx=0x7f2ce4006640 tx=0x7f2ce400b910).stop 2021-02-10T11:21:45.940 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:45.944 7f2d00c45700 1 -- 172.21.2.102:0/1884820891 shutdown_connections 2021-02-10T11:21:45.940 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:45.944 7f2d00c45700 1 --2- 172.21.2.102:0/1884820891 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2cfc0acf90 0x7f2cfc1923e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:45.940 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:45.944 7f2d00c45700 1 --2- 172.21.2.102:0/1884820891 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2ce0018730 0x7f2ce001ac10 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:45.940 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:45.944 7f2d00c45700 1 --2- 172.21.2.102:0/1884820891 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2cfc0b24b0 0x7f2cfc191dc0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:45.940 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:45.944 7f2d00c45700 1 --2- 172.21.2.102:0/1884820891 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2cfc0b20b0 0x7f2cfc0ac970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:45.941 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:45.944 7f2d00c45700 1 -- 172.21.2.102:0/1884820891 >> 172.21.2.102:0/1884820891 conn(0x7f2cfc0bf4e0 msgr2=0x7f2cfc0be3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:45.941 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:45.944 7f2d00c45700 1 -- 172.21.2.102:0/1884820891 shutdown_connections 2021-02-10T11:21:45.941 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:45.944 7f2d00c45700 1 -- 172.21.2.102:0/1884820891 wait complete. 2021-02-10T11:21:45.949 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000344460 2021-02-10T11:21:46.382 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:46.369 7f5426601700 1 Processor -- start 2021-02-10T11:21:46.382 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:46.369 7f5426601700 1 -- start start 2021-02-10T11:21:46.382 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:46.370 7f5426601700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f54201b0470 0x7f54201b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:46.382 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:46.370 7f5426601700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54200f0b50 con 0x7f54200ef610 2021-02-10T11:21:46.382 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:46.370 7f5426601700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54200f24e0 con 0x7f54200f20e0 2021-02-10T11:21:46.382 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:46.370 7f5426601700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54200e5ea0 con 0x7f54201b0470 2021-02-10T11:21:46.382 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:46.370 7f541f7fe700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f54200ef610 0x7f54200efa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:47230/0 (socket says 172.21.2.102:47230) 2021-02-10T11:21:46.382 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:46.370 7f541f7fe700 1 -- 172.21.2.102:0/2836087758 learned_addr learned my addr 172.21.2.102:0/2836087758 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:21:46.382 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:46.370 7f541ffff700 1 --2- 172.21.2.102:0/2836087758 >> v2:172.21.2.113:3300/0 conn(0x7f54201b0470 0x7f54201b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:46.382 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:46.370 7f541effd700 1 -- 172.21.2.102:0/2836087758 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3898619944 0 0) 0x7f54140010b0 con 0x7f54200ef610 2021-02-10T11:21:46.383 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:46.370 7f541effd700 1 -- 172.21.2.102:0/2836087758 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5408002fe0 con 0x7f54200ef610 2021-02-10T11:21:46.383 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:46.370 7f541ffff700 1 -- 172.21.2.102:0/2836087758 >> v1:172.21.2.113:6789/0 conn(0x7f54200f20e0 legacy=0x7f54200f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:46.383 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:46.370 7f541ffff700 1 -- 172.21.2.102:0/2836087758 >> v1:172.21.2.102:6789/0 conn(0x7f54200ef610 legacy=0x7f54200efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:46.383 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:46.370 7f541ffff700 1 -- 172.21.2.102:0/2836087758 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54200e52e0 con 0x7f54201b0470 2021-02-10T11:21:46.383 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:46.371 7f541ffff700 1 --2- 172.21.2.102:0/2836087758 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f54201b0470 0x7f54201b0870 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 rx=0x7f54180069d0 tx=0x7f5418009e40).ready entity=mon.1 client_cookie=9325217a88dfbf63 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:46.383 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:46.371 7f541effd700 1 -- 172.21.2.102:0/2836087758 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f541800b510 con 0x7f54201b0470 2021-02-10T11:21:46.383 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:46.371 7f541effd700 1 -- 172.21.2.102:0/2836087758 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f541800b890 con 0x7f54201b0470 2021-02-10T11:21:46.383 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:46.371 7f541effd700 1 -- 172.21.2.102:0/2836087758 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f541800bcf0 con 0x7f54201b0470 2021-02-10T11:21:46.383 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:46.371 7f5426601700 1 -- 172.21.2.102:0/2836087758 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f54201b0470 msgr2=0x7f54201b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:46.383 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:46.371 7f5426601700 1 --2- 172.21.2.102:0/2836087758 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f54201b0470 0x7f54201b0870 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 rx=0x7f54180069d0 tx=0x7f5418009e40).stop 2021-02-10T11:21:46.383 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:46.371 7f5426601700 1 -- 172.21.2.102:0/2836087758 shutdown_connections 2021-02-10T11:21:46.383 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:46.371 7f5426601700 1 --2- 172.21.2.102:0/2836087758 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f54201b0470 0x7f54201b0870 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:46.383 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:46.371 7f5426601700 1 -- 172.21.2.102:0/2836087758 >> 172.21.2.102:0/2836087758 conn(0x7f54200ce3e0 msgr2=0x7f54200e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:46.384 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:46.372 7f5426601700 1 -- 172.21.2.102:0/2836087758 shutdown_connections 2021-02-10T11:21:46.384 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:46.372 7f5426601700 1 -- 172.21.2.102:0/2836087758 wait complete. 2021-02-10T11:21:46.385 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:46.386 7f5426601700 1 Processor -- start 2021-02-10T11:21:46.385 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:46.386 7f5426601700 1 -- start start 2021-02-10T11:21:46.385 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:46.387 7f5426601700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f54200c36a0 0x7f54200c2b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:46.385 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:46.387 7f5426601700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f54200c3190 0x7f54200b1d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:46.385 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:46.387 7f5426601700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f54200b2370 0x7f54200b2d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:46.385 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:46.387 7f5426601700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54200b27c0 con 0x7f54200c3190 2021-02-10T11:21:46.385 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:46.387 7f5426601700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54200b3360 con 0x7f54200b2370 2021-02-10T11:21:46.385 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:46.387 7f5426601700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54200b3540 con 0x7f54200c36a0 2021-02-10T11:21:46.386 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:46.387 7f541f7fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f54200c3190 0x7f54200b1d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:46.386 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:46.387 7f541f7fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f54200c3190 0x7f54200b1d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:55014/0 (socket says 172.21.2.102:55014) 2021-02-10T11:21:46.386 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:46.387 7f541f7fe700 1 -- 172.21.2.102:0/192662707 learned_addr learned my addr 172.21.2.102:0/192662707 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:21:46.386 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:46.387 7f541f7fe700 1 -- 172.21.2.102:0/192662707 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f54200c36a0 msgr2=0x7f54200c2b70 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:21:46.386 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:46.387 7f541f7fe700 1 --2- 172.21.2.102:0/192662707 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f54200c36a0 0x7f54200c2b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:21:46.386 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:46.387 7f541f7fe700 1 -- 172.21.2.102:0/192662707 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f54200b2370 msgr2=0x7f54200b2d40 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:21:46.386 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:46.387 7f541f7fe700 1 --2- 172.21.2.102:0/192662707 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f54200b2370 0x7f54200b2d40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:21:46.386 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:46.387 7f541f7fe700 1 -- 172.21.2.102:0/192662707 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54200b3760 con 0x7f54200c3190 2021-02-10T11:21:46.386 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:46.387 7f541f7fe700 1 --2- 172.21.2.102:0/192662707 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f54200c3190 0x7f54200b1d50 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 rx=0x7f5414001c60 tx=0x7f541400a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:46.386 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:46.387 7f541cff9700 1 -- 172.21.2.102:0/192662707 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f541400acc0 con 0x7f54200c3190 2021-02-10T11:21:46.386 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:46.387 7f541cff9700 1 -- 172.21.2.102:0/192662707 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5414004350 con 0x7f54200c3190 2021-02-10T11:21:46.386 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:46.387 7f5426601700 1 -- 172.21.2.102:0/192662707 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54200b3760 con 0x7f54200c3190 2021-02-10T11:21:46.386 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:46.387 7f541cff9700 1 -- 172.21.2.102:0/192662707 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5414004830 con 0x7f54200c3190 2021-02-10T11:21:46.387 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:46.387 7f5426601700 1 -- 172.21.2.102:0/192662707 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54200b3cf0 con 0x7f54200c3190 2021-02-10T11:21:46.387 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:46.388 7f541cff9700 1 -- 172.21.2.102:0/192662707 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f5414004530 con 0x7f54200c3190 2021-02-10T11:21:46.387 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:46.388 7f541cff9700 1 --2- 172.21.2.102:0/192662707 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5400018730 0x7f540001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:46.387 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:46.388 7f541cff9700 1 -- 172.21.2.102:0/192662707 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f5414032630 con 0x7f54200c3190 2021-02-10T11:21:46.387 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:46.388 7f541ffff700 1 --2- 172.21.2.102:0/192662707 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5400018730 0x7f540001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:46.387 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:46.388 7f541ffff700 1 --2- 172.21.2.102:0/192662707 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5400018730 0x7f540001ac10 crc :-1 s=READY pgs=483 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:46.414 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:46.418 7f5426601700 1 -- 172.21.2.102:0/192662707 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54200edd10 con 0x7f54200c3190 2021-02-10T11:21:46.417 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:46.421 7f541cff9700 1 -- 172.21.2.102:0/192662707 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f5414000940 con 0x7f54200c3190 2021-02-10T11:21:46.544 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:46.548 7f5426601700 1 -- 172.21.2.102:0/192662707 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344460", "target": ["mgr", ""]}) v1 -- 0x7f54200b27c0 con 0x7f5400018730 2021-02-10T11:21:46.555 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:46.559 7f541cff9700 1 -- 172.21.2.102:0/192662707 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f5418006e60 con 0x7f5400018730 2021-02-10T11:21:46.562 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:46.566 7f5426601700 1 -- 172.21.2.102:0/192662707 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5400018730 msgr2=0x7f540001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:46.562 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:46.566 7f5426601700 1 --2- 172.21.2.102:0/192662707 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5400018730 0x7f540001ac10 crc :-1 s=READY pgs=483 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:46.562 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:46.566 7f5426601700 1 -- 172.21.2.102:0/192662707 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f54200c3190 msgr2=0x7f54200b1d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:46.562 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:46.566 7f5426601700 1 --2- 172.21.2.102:0/192662707 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f54200c3190 0x7f54200b1d50 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 rx=0x7f5414001c60 tx=0x7f541400a8d0).stop 2021-02-10T11:21:46.562 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:46.566 7f5426601700 1 -- 172.21.2.102:0/192662707 shutdown_connections 2021-02-10T11:21:46.562 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:46.566 7f5426601700 1 --2- 172.21.2.102:0/192662707 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5400018730 0x7f540001ac10 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:46.562 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:46.566 7f5426601700 1 --2- 172.21.2.102:0/192662707 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f54200b2370 0x7f54200b2d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:21:46.562 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:46.566 7f5426601700 1 --2- 172.21.2.102:0/192662707 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f54200c3190 0x7f54200b1d50 unknown :-1 s=CLOSED pgs=992 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:46.562 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:46.566 7f5426601700 1 --2- 172.21.2.102:0/192662707 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f54200c36a0 0x7f54200c2b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:21:46.562 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:46.566 7f5426601700 1 -- 172.21.2.102:0/192662707 >> 172.21.2.102:0/192662707 conn(0x7f542009f630 msgr2=0x7f54200a0130 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:46.563 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:46.566 7f5426601700 1 -- 172.21.2.102:0/192662707 shutdown_connections 2021-02-10T11:21:46.563 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:46.566 7f5426601700 1 -- 172.21.2.102:0/192662707 wait complete. 2021-02-10T11:21:46.571 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000344461 2021-02-10T11:21:46.999 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:46.987 7fc99333f700 1 Processor -- start 2021-02-10T11:21:46.999 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:46.987 7fc99333f700 1 -- start start 2021-02-10T11:21:46.999 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:46.988 7fc99333f700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fc98c1b0870 0x7fc98c0f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:47.000 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:46.988 7fc99333f700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fc98c0efe00 0x7fc98c0f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:47.000 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:46.988 7fc99333f700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc98c1b0c70 con 0x7fc98c1b0470 2021-02-10T11:21:47.000 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:46.988 7fc99333f700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc98c0e5ea0 con 0x7fc98c1b0870 2021-02-10T11:21:47.000 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:46.988 7fc99333f700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc98c0f0b50 con 0x7fc98c0efe00 2021-02-10T11:21:47.000 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:46.988 7fc99115b700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fc98c1b0470 0x7fc98c0ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:47244/0 (socket says 172.21.2.102:47244) 2021-02-10T11:21:47.000 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:46.988 7fc99115b700 1 -- 172.21.2.102:0/600704011 learned_addr learned my addr 172.21.2.102:0/600704011 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:21:47.000 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:46.988 7fc99095a700 1 --2- 172.21.2.102:0/600704011 >> v2:172.21.2.102:3300/0 conn(0x7fc98c1b0870 0x7fc98c0f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:47.000 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:46.988 7fc99195c700 1 --2- 172.21.2.102:0/600704011 >> v2:172.21.2.113:3300/0 conn(0x7fc98c0efe00 0x7fc98c0f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:47.000 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:46.988 7fc99095a700 1 -- 172.21.2.102:0/600704011 >> v2:172.21.2.113:3300/0 conn(0x7fc98c0efe00 msgr2=0x7fc98c0f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:21:47.000 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:46.988 7fc99095a700 1 --2- 172.21.2.102:0/600704011 >> v2:172.21.2.113:3300/0 conn(0x7fc98c0efe00 0x7fc98c0f0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:47.000 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:46.988 7fc99095a700 1 -- 172.21.2.102:0/600704011 >> v1:172.21.2.102:6789/0 conn(0x7fc98c1b0470 legacy=0x7fc98c0ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:47.000 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:46.988 7fc983fff700 1 -- 172.21.2.102:0/600704011 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1431871540 0 0) 0x7fc988000ce0 con 0x7fc98c1b0470 2021-02-10T11:21:47.000 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:46.988 7fc99095a700 1 -- 172.21.2.102:0/600704011 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc98c0f0820 con 0x7fc98c1b0870 2021-02-10T11:21:47.001 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:46.988 7fc99095a700 1 --2- 172.21.2.102:0/600704011 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc98c1b0870 0x7fc98c0f20e0 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 rx=0x7fc97c006650 tx=0x7fc97c009a00).ready entity=mon.0 client_cookie=be31f7c98c40cead server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:47.001 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:46.989 7fc983fff700 1 -- 172.21.2.102:0/600704011 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc97c0009b0 con 0x7fc98c1b0870 2021-02-10T11:21:47.001 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:46.989 7fc983fff700 1 -- 172.21.2.102:0/600704011 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc97c00c0c0 con 0x7fc98c1b0870 2021-02-10T11:21:47.001 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:46.989 7fc983fff700 1 -- 172.21.2.102:0/600704011 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc97c00c5a0 con 0x7fc98c1b0870 2021-02-10T11:21:47.001 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:46.989 7fc99333f700 1 -- 172.21.2.102:0/600704011 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc98c1b0870 msgr2=0x7fc98c0f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:47.001 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:46.989 7fc99333f700 1 --2- 172.21.2.102:0/600704011 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc98c1b0870 0x7fc98c0f20e0 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 rx=0x7fc97c006650 tx=0x7fc97c009a00).stop 2021-02-10T11:21:47.001 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:46.989 7fc99333f700 1 -- 172.21.2.102:0/600704011 shutdown_connections 2021-02-10T11:21:47.001 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:46.989 7fc99333f700 1 --2- 172.21.2.102:0/600704011 >> v2:172.21.2.113:3300/0 conn(0x7fc98c0efe00 0x7fc98c0f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:47.001 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:46.989 7fc99333f700 1 --2- 172.21.2.102:0/600704011 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc98c1b0870 0x7fc98c0f20e0 unknown :-1 s=CLOSED pgs=994 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:47.001 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:46.989 7fc99333f700 1 -- 172.21.2.102:0/600704011 >> 172.21.2.102:0/600704011 conn(0x7fc98c0ce3e0 msgr2=0x7fc98c0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:47.001 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:46.989 7fc99333f700 1 -- 172.21.2.102:0/600704011 shutdown_connections 2021-02-10T11:21:47.001 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:46.989 7fc99333f700 1 -- 172.21.2.102:0/600704011 wait complete. 2021-02-10T11:21:47.003 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:47.004 7fc99333f700 1 Processor -- start 2021-02-10T11:21:47.003 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:47.004 7fc99333f700 1 -- start start 2021-02-10T11:21:47.004 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:47.004 7fc99333f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc98c095db0 0x7fc98c10a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:47.004 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:47.004 7fc99333f700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc98c19e280 0x7fc98c0ab870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:47.004 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:47.004 7fc99333f700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc98c09f970 0x7fc98c0ac940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:47.004 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:47.004 7fc99333f700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc98c096330 con 0x7fc98c095db0 2021-02-10T11:21:47.004 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:47.004 7fc99333f700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc98c0ac3c0 con 0x7fc98c19e280 2021-02-10T11:21:47.004 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:47.004 7fc99333f700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc98c0abe90 con 0x7fc98c09f970 2021-02-10T11:21:47.004 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:47.004 7fc99115b700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc98c095db0 0x7fc98c10a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:47.005 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:47.004 7fc99115b700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc98c095db0 0x7fc98c10a2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:55026/0 (socket says 172.21.2.102:55026) 2021-02-10T11:21:47.005 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:47.004 7fc99115b700 1 -- 172.21.2.102:0/528873440 learned_addr learned my addr 172.21.2.102:0/528873440 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:21:47.005 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:47.004 7fc99095a700 1 --2- 172.21.2.102:0/528873440 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc98c19e280 0x7fc98c0ab870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:47.005 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:47.004 7fc99195c700 1 --2- 172.21.2.102:0/528873440 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc98c09f970 0x7fc98c0ac940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:47.005 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:47.005 7fc99115b700 1 -- 172.21.2.102:0/528873440 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc98c09f970 msgr2=0x7fc98c0ac940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:47.005 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:47.005 7fc99115b700 1 --2- 172.21.2.102:0/528873440 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc98c09f970 0x7fc98c0ac940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:47.005 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:47.005 7fc99115b700 1 -- 172.21.2.102:0/528873440 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc98c19e280 msgr2=0x7fc98c0ab870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:47.005 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:47.005 7fc99115b700 1 --2- 172.21.2.102:0/528873440 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc98c19e280 0x7fc98c0ab870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:47.005 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:47.005 7fc99115b700 1 -- 172.21.2.102:0/528873440 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc98c0ac070 con 0x7fc98c095db0 2021-02-10T11:21:47.005 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:47.005 7fc99195c700 1 --2- 172.21.2.102:0/528873440 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc98c09f970 0x7fc98c0ac940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:21:47.005 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:47.005 7fc99115b700 1 --2- 172.21.2.102:0/528873440 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc98c095db0 0x7fc98c10a2b0 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 rx=0x7fc988006f00 tx=0x7fc98800b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:47.005 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:47.005 7fc981ffb700 1 -- 172.21.2.102:0/528873440 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc988006020 con 0x7fc98c095db0 2021-02-10T11:21:47.006 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:47.005 7fc981ffb700 1 -- 172.21.2.102:0/528873440 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9880062a0 con 0x7fc98c095db0 2021-02-10T11:21:47.006 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:47.005 7fc99333f700 1 -- 172.21.2.102:0/528873440 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc98c0ac070 con 0x7fc98c095db0 2021-02-10T11:21:47.006 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:47.005 7fc99333f700 1 -- 172.21.2.102:0/528873440 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc98c0ad250 con 0x7fc98c095db0 2021-02-10T11:21:47.006 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:47.005 7fc981ffb700 1 -- 172.21.2.102:0/528873440 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc988004620 con 0x7fc98c095db0 2021-02-10T11:21:47.006 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:47.006 7fc981ffb700 1 -- 172.21.2.102:0/528873440 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fc9880043a0 con 0x7fc98c095db0 2021-02-10T11:21:47.006 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:47.006 7fc981ffb700 1 --2- 172.21.2.102:0/528873440 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc96c0186c0 0x7fc96c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:47.006 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:47.006 7fc981ffb700 1 -- 172.21.2.102:0/528873440 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fc9880327b0 con 0x7fc98c095db0 2021-02-10T11:21:47.006 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:47.006 7fc99095a700 1 --2- 172.21.2.102:0/528873440 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc96c0186c0 0x7fc96c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:47.006 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:47.006 7fc99095a700 1 --2- 172.21.2.102:0/528873440 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc96c0186c0 0x7fc96c01abc0 crc :-1 s=READY pgs=484 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:47.033 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:47.037 7fc99333f700 1 -- 172.21.2.102:0/528873440 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc98c0962c0 con 0x7fc98c095db0 2021-02-10T11:21:47.035 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:47.040 7fc981ffb700 1 -- 172.21.2.102:0/528873440 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fc988004880 con 0x7fc98c095db0 2021-02-10T11:21:47.163 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:47.167 7fc99333f700 1 -- 172.21.2.102:0/528873440 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344461", "target": ["mgr", ""]}) v1 -- 0x7fc98c0ac3c0 con 0x7fc96c0186c0 2021-02-10T11:21:47.174 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:47.178 7fc981ffb700 1 -- 172.21.2.102:0/528873440 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fc97c006450 con 0x7fc96c0186c0 2021-02-10T11:21:47.180 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:47.184 7fc99333f700 1 -- 172.21.2.102:0/528873440 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc96c0186c0 msgr2=0x7fc96c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:47.180 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:47.184 7fc99333f700 1 --2- 172.21.2.102:0/528873440 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc96c0186c0 0x7fc96c01abc0 crc :-1 s=READY pgs=484 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:47.180 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:47.184 7fc99333f700 1 -- 172.21.2.102:0/528873440 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc98c095db0 msgr2=0x7fc98c10a2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:47.180 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:47.184 7fc99333f700 1 --2- 172.21.2.102:0/528873440 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc98c095db0 0x7fc98c10a2b0 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 rx=0x7fc988006f00 tx=0x7fc98800b940).stop 2021-02-10T11:21:47.180 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:47.184 7fc99333f700 1 -- 172.21.2.102:0/528873440 shutdown_connections 2021-02-10T11:21:47.181 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:47.184 7fc99333f700 1 --2- 172.21.2.102:0/528873440 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc98c09f970 0x7fc98c0ac940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:47.181 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:47.184 7fc99333f700 1 --2- 172.21.2.102:0/528873440 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc96c0186c0 0x7fc96c01abc0 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:47.181 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:47.184 7fc99333f700 1 --2- 172.21.2.102:0/528873440 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc98c19e280 0x7fc98c0ab870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:47.181 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:47.184 7fc99333f700 1 --2- 172.21.2.102:0/528873440 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc98c095db0 0x7fc98c10a2b0 unknown :-1 s=CLOSED pgs=995 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:47.181 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:47.184 7fc99333f700 1 -- 172.21.2.102:0/528873440 >> 172.21.2.102:0/528873440 conn(0x7fc98c0bbf60 msgr2=0x7fc98c09cf60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:47.181 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:47.184 7fc99333f700 1 -- 172.21.2.102:0/528873440 shutdown_connections 2021-02-10T11:21:47.181 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:47.184 7fc99333f700 1 -- 172.21.2.102:0/528873440 wait complete. 2021-02-10T11:21:47.189 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000344462 2021-02-10T11:21:47.622 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:47.610 7fd66a586700 1 Processor -- start 2021-02-10T11:21:47.623 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:47.610 7fd66a586700 1 -- start start 2021-02-10T11:21:47.623 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:47.611 7fd66a586700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fd6641b0470 0x7fd6641b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:47.623 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:47.611 7fd66a586700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fd6640ef610 0x7fd6640efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:47.623 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:47.611 7fd66a586700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd6640f0030 0x7fd6640f0430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:47.623 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:47.611 7fd66a586700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6640e5ea0 con 0x7fd6641b0470 2021-02-10T11:21:47.623 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:47.611 7fd66a586700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6640f0b50 con 0x7fd6640ef610 2021-02-10T11:21:47.623 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:47.611 7fd66a586700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6640e52e0 con 0x7fd6640f0030 2021-02-10T11:21:47.623 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:47.611 7fd663fff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fd6641b0470 0x7fd6641b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:47.623 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:47.611 7fd663fff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fd6641b0470 0x7fd6641b0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:55034/0 (socket says 172.21.2.102:55034) 2021-02-10T11:21:47.623 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:47.611 7fd663fff700 1 -- 172.21.2.102:0/3475233498 learned_addr learned my addr 172.21.2.102:0/3475233498 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:21:47.623 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:47.611 7fd65bfff700 1 --2- 172.21.2.102:0/3475233498 >> v2:172.21.2.113:3300/0 conn(0x7fd6640ef610 0x7fd6640efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:47.623 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:47.611 7fd668ba3700 1 --2- 172.21.2.102:0/3475233498 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd6640f0030 0x7fd6640f0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:47.623 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:47.611 7fd663fff700 1 -- 172.21.2.102:0/3475233498 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd6640f0030 msgr2=0x7fd6640f0430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:21:47.624 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:47.611 7fd663fff700 1 --2- 172.21.2.102:0/3475233498 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd6640f0030 0x7fd6640f0430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:47.624 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:47.611 7fd663fff700 1 -- 172.21.2.102:0/3475233498 >> v2:172.21.2.113:3300/0 conn(0x7fd6640ef610 msgr2=0x7fd6640efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:21:47.624 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:47.611 7fd663fff700 1 --2- 172.21.2.102:0/3475233498 >> v2:172.21.2.113:3300/0 conn(0x7fd6640ef610 0x7fd6640efa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:47.624 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:47.611 7fd663fff700 1 -- 172.21.2.102:0/3475233498 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6640f10d0 con 0x7fd6641b0470 2021-02-10T11:21:47.624 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:47.611 7fd663fff700 1 --2- 172.21.2.102:0/3475233498 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd6641b0470 0x7fd6641b0870 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 rx=0x7fd64c006700 tx=0x7fd64c009b80).ready entity=mon.0 client_cookie=86d2392c8c08a24 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:47.624 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:47.612 7fd6637fe700 1 -- 172.21.2.102:0/3475233498 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd64c00c0c0 con 0x7fd6641b0470 2021-02-10T11:21:47.624 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:47.612 7fd6637fe700 1 -- 172.21.2.102:0/3475233498 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd64c00c400 con 0x7fd6641b0470 2021-02-10T11:21:47.624 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:47.612 7fd6637fe700 1 -- 172.21.2.102:0/3475233498 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd64c00c8e0 con 0x7fd6641b0470 2021-02-10T11:21:47.624 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:47.612 7fd66a586700 1 -- 172.21.2.102:0/3475233498 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd6641b0470 msgr2=0x7fd6641b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:47.624 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:47.612 7fd66a586700 1 --2- 172.21.2.102:0/3475233498 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd6641b0470 0x7fd6641b0870 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 rx=0x7fd64c006700 tx=0x7fd64c009b80).stop 2021-02-10T11:21:47.624 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:47.612 7fd66a586700 1 -- 172.21.2.102:0/3475233498 shutdown_connections 2021-02-10T11:21:47.624 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:47.612 7fd66a586700 1 --2- 172.21.2.102:0/3475233498 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd6640f0030 0x7fd6640f0430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:47.624 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:47.612 7fd66a586700 1 --2- 172.21.2.102:0/3475233498 >> v2:172.21.2.113:3300/0 conn(0x7fd6640ef610 0x7fd6640efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:47.625 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:47.612 7fd66a586700 1 --2- 172.21.2.102:0/3475233498 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd6641b0470 0x7fd6641b0870 unknown :-1 s=CLOSED pgs=996 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:47.625 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:47.612 7fd66a586700 1 -- 172.21.2.102:0/3475233498 >> 172.21.2.102:0/3475233498 conn(0x7fd6640ce3e0 msgr2=0x7fd6640e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:47.625 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:47.612 7fd66a586700 1 -- 172.21.2.102:0/3475233498 shutdown_connections 2021-02-10T11:21:47.625 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:47.612 7fd66a586700 1 -- 172.21.2.102:0/3475233498 wait complete. 2021-02-10T11:21:47.626 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:47.627 7fd66a586700 1 Processor -- start 2021-02-10T11:21:47.627 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:47.627 7fd66a586700 1 -- start start 2021-02-10T11:21:47.627 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:47.627 7fd66a586700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd66409b930 0x7fd6640b7c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:47.627 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:47.627 7fd66a586700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd6640b8410 0x7fd6640b8810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:47.627 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:47.627 7fd66a586700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd6640c1f80 0x7fd6640c2380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:47.627 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:47.627 7fd66a586700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6640c1a00 con 0x7fd6640b8410 2021-02-10T11:21:47.627 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:47.627 7fd66a586700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6640b8e30 con 0x7fd66409b930 2021-02-10T11:21:47.627 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:47.627 7fd66a586700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6640b9050 con 0x7fd6640c1f80 2021-02-10T11:21:47.627 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:47.627 7fd65bfff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd6640b8410 0x7fd6640b8810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:47.627 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:47.627 7fd65bfff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd6640b8410 0x7fd6640b8810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:55044/0 (socket says 172.21.2.102:55044) 2021-02-10T11:21:47.627 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:47.627 7fd65bfff700 1 -- 172.21.2.102:0/1867402831 learned_addr learned my addr 172.21.2.102:0/1867402831 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:21:47.627 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:47.627 7fd668ba3700 1 --2- 172.21.2.102:0/1867402831 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd6640c1f80 0x7fd6640c2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:47.627 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:47.627 7fd663fff700 1 --2- 172.21.2.102:0/1867402831 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd66409b930 0x7fd6640b7c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:47.627 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:47.628 7fd668ba3700 1 -- 172.21.2.102:0/1867402831 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd66409b930 msgr2=0x7fd6640b7c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:47.628 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:47.628 7fd668ba3700 1 --2- 172.21.2.102:0/1867402831 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd66409b930 0x7fd6640b7c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:47.628 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:47.628 7fd668ba3700 1 -- 172.21.2.102:0/1867402831 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd6640b8410 msgr2=0x7fd6640b8810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:47.628 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:47.628 7fd668ba3700 1 --2- 172.21.2.102:0/1867402831 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd6640b8410 0x7fd6640b8810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:47.628 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:47.628 7fd668ba3700 1 -- 172.21.2.102:0/1867402831 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6640c29a0 con 0x7fd6640c1f80 2021-02-10T11:21:47.628 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:47.628 7fd65bfff700 1 --2- 172.21.2.102:0/1867402831 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd6640b8410 0x7fd6640b8810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:21:47.628 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:47.628 7fd668ba3700 1 --2- 172.21.2.102:0/1867402831 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd6640c1f80 0x7fd6640c2380 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7fd65c00b8a0 tx=0x7fd65c00bc40).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:47.628 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:47.628 7fd6617fa700 1 -- 172.21.2.102:0/1867402831 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd65c00bd80 con 0x7fd6640c1f80 2021-02-10T11:21:47.628 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:47.628 7fd6617fa700 1 -- 172.21.2.102:0/1867402831 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd65c003030 con 0x7fd6640c1f80 2021-02-10T11:21:47.628 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:47.628 7fd66a586700 1 -- 172.21.2.102:0/1867402831 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6640c29a0 con 0x7fd6640c1f80 2021-02-10T11:21:47.628 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:47.628 7fd6617fa700 1 -- 172.21.2.102:0/1867402831 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd65c001470 con 0x7fd6640c1f80 2021-02-10T11:21:47.628 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:47.628 7fd66a586700 1 -- 172.21.2.102:0/1867402831 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6640c29a0 con 0x7fd6640c1f80 2021-02-10T11:21:47.628 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:47.629 7fd6617fa700 1 -- 172.21.2.102:0/1867402831 <== mon.2 v2:172.21.2.113:3301/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd65c0011f0 con 0x7fd6640c1f80 2021-02-10T11:21:47.628 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:47.629 7fd6617fa700 1 --2- 172.21.2.102:0/1867402831 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd648018730 0x7fd64801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:47.628 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:47.629 7fd6617fa700 1 -- 172.21.2.102:0/1867402831 <== mon.2 v2:172.21.2.113:3301/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fd65c032aa0 con 0x7fd6640c1f80 2021-02-10T11:21:47.629 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:47.629 7fd663fff700 1 --2- 172.21.2.102:0/1867402831 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd648018730 0x7fd64801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:47.629 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:47.629 7fd663fff700 1 --2- 172.21.2.102:0/1867402831 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd648018730 0x7fd64801ac10 crc :-1 s=READY pgs=485 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:47.656 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:47.660 7fd66a586700 1 -- 172.21.2.102:0/1867402831 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd664105b30 con 0x7fd6640c1f80 2021-02-10T11:21:47.659 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:47.664 7fd6617fa700 1 -- 172.21.2.102:0/1867402831 <== mon.2 v2:172.21.2.113:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd65c003010 con 0x7fd6640c1f80 2021-02-10T11:21:47.796 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:47.800 7fd66a586700 1 -- 172.21.2.102:0/1867402831 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344462", "target": ["mgr", ""]}) v1 -- 0x7fd664105b30 con 0x7fd648018730 2021-02-10T11:21:47.807 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:47.811 7fd6617fa700 1 -- 172.21.2.102:0/1867402831 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fd64c009380 con 0x7fd648018730 2021-02-10T11:21:47.813 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:47.817 7fd66a586700 1 -- 172.21.2.102:0/1867402831 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd648018730 msgr2=0x7fd64801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:47.813 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:47.817 7fd66a586700 1 --2- 172.21.2.102:0/1867402831 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd648018730 0x7fd64801ac10 crc :-1 s=READY pgs=485 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:47.813 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:47.817 7fd66a586700 1 -- 172.21.2.102:0/1867402831 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd6640c1f80 msgr2=0x7fd6640c2380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:47.813 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:47.817 7fd66a586700 1 --2- 172.21.2.102:0/1867402831 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd6640c1f80 0x7fd6640c2380 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7fd65c00b8a0 tx=0x7fd65c00bc40).stop 2021-02-10T11:21:47.814 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:47.817 7fd66a586700 1 -- 172.21.2.102:0/1867402831 shutdown_connections 2021-02-10T11:21:47.814 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:47.817 7fd66a586700 1 --2- 172.21.2.102:0/1867402831 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd6640c1f80 0x7fd6640c2380 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:47.814 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:47.817 7fd66a586700 1 --2- 172.21.2.102:0/1867402831 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd6640b8410 0x7fd6640b8810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:47.814 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:47.817 7fd66a586700 1 --2- 172.21.2.102:0/1867402831 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd648018730 0x7fd64801ac10 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:47.814 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:47.817 7fd66a586700 1 --2- 172.21.2.102:0/1867402831 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd66409b930 0x7fd6640b7c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:47.814 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:47.817 7fd66a586700 1 -- 172.21.2.102:0/1867402831 >> 172.21.2.102:0/1867402831 conn(0x7fd664094e10 msgr2=0x7fd6640a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:47.814 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:47.817 7fd66a586700 1 -- 172.21.2.102:0/1867402831 shutdown_connections 2021-02-10T11:21:47.814 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:47.817 7fd66a586700 1 -- 172.21.2.102:0/1867402831 wait complete. 2021-02-10T11:21:47.822 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000344463 2021-02-10T11:21:48.250 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:48.238 7f8729670700 1 Processor -- start 2021-02-10T11:21:48.250 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:48.238 7f8729670700 1 -- start start 2021-02-10T11:21:48.250 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:48.239 7f8729670700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f87241b0470 0x7f87241b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:48.250 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:48.239 7f8729670700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f87240ef610 0x7f87240efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:48.250 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:48.239 7f8729670700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f87240f0030 0x7f87240f0430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:48.250 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:48.239 7f8729670700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87240e5ea0 con 0x7f87240ef610 2021-02-10T11:21:48.251 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:48.239 7f8729670700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87240f0b50 con 0x7f87240f0030 2021-02-10T11:21:48.251 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:48.239 7f8729670700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87240e52e0 con 0x7f87241b0470 2021-02-10T11:21:48.251 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:48.239 7f871bfff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f87240ef610 0x7f87240efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:48.251 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:48.239 7f871bfff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f87240ef610 0x7f87240efa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:55048/0 (socket says 172.21.2.102:55048) 2021-02-10T11:21:48.251 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:48.239 7f871bfff700 1 -- 172.21.2.102:0/1655511416 learned_addr learned my addr 172.21.2.102:0/1655511416 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:21:48.251 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:48.239 7f8723ac0700 1 --2- 172.21.2.102:0/1655511416 >> v2:172.21.2.113:3300/0 conn(0x7f87240f0030 0x7f87240f0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:48.251 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:48.239 7f871bfff700 1 -- 172.21.2.102:0/1655511416 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f87241b0470 msgr2=0x7f87241b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:21:48.251 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:48.239 7f871bfff700 1 --2- 172.21.2.102:0/1655511416 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f87241b0470 0x7f87241b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:21:48.251 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:48.239 7f871bfff700 1 -- 172.21.2.102:0/1655511416 >> v2:172.21.2.113:3300/0 conn(0x7f87240f0030 msgr2=0x7f87240f0430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:21:48.251 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:48.239 7f871bfff700 1 --2- 172.21.2.102:0/1655511416 >> v2:172.21.2.113:3300/0 conn(0x7f87240f0030 0x7f87240f0430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:48.251 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:48.239 7f871bfff700 1 -- 172.21.2.102:0/1655511416 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87240f10d0 con 0x7f87240ef610 2021-02-10T11:21:48.251 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:48.239 7f871bfff700 1 --2- 172.21.2.102:0/1655511416 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f87240ef610 0x7f87240efa10 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 rx=0x7f870c006700 tx=0x7f870c009b80).ready entity=mon.0 client_cookie=e858508adac1e7ff server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:48.251 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:48.239 7f8722abe700 1 -- 172.21.2.102:0/1655511416 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f870c006340 con 0x7f87240ef610 2021-02-10T11:21:48.252 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:48.239 7f8722abe700 1 -- 172.21.2.102:0/1655511416 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f870c00c0c0 con 0x7f87240ef610 2021-02-10T11:21:48.252 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:48.239 7f8722abe700 1 -- 172.21.2.102:0/1655511416 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f870c00c510 con 0x7f87240ef610 2021-02-10T11:21:48.252 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:48.239 7f8729670700 1 -- 172.21.2.102:0/1655511416 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f87240ef610 msgr2=0x7f87240efa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:48.252 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:48.239 7f8729670700 1 --2- 172.21.2.102:0/1655511416 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f87240ef610 0x7f87240efa10 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 rx=0x7f870c006700 tx=0x7f870c009b80).stop 2021-02-10T11:21:48.252 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:48.240 7f8729670700 1 -- 172.21.2.102:0/1655511416 shutdown_connections 2021-02-10T11:21:48.252 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:48.240 7f8729670700 1 --2- 172.21.2.102:0/1655511416 >> v2:172.21.2.113:3300/0 conn(0x7f87240f0030 0x7f87240f0430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:48.252 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:48.240 7f8729670700 1 --2- 172.21.2.102:0/1655511416 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f87240ef610 0x7f87240efa10 unknown :-1 s=CLOSED pgs=997 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:48.252 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:48.240 7f8729670700 1 --2- 172.21.2.102:0/1655511416 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f87241b0470 0x7f87241b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:21:48.252 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:48.240 7f8729670700 1 -- 172.21.2.102:0/1655511416 >> 172.21.2.102:0/1655511416 conn(0x7f87240ce3e0 msgr2=0x7f87240e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:48.252 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:48.240 7f8729670700 1 -- 172.21.2.102:0/1655511416 shutdown_connections 2021-02-10T11:21:48.252 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:48.240 7f8729670700 1 -- 172.21.2.102:0/1655511416 wait complete. 2021-02-10T11:21:48.254 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:48.254 7f8729670700 1 Processor -- start 2021-02-10T11:21:48.254 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:48.254 7f8729670700 1 -- start start 2021-02-10T11:21:48.254 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:48.255 7f8729670700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f87240b7db0 0x7f87240b8260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:48.254 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:48.255 7f8729670700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f87240b8880 0x7f87241a7450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:48.254 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:48.255 7f8729670700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f87240b8c80 0x7f87241a7a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:48.254 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:48.255 7f8729670700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87240b79b0 con 0x7f87240b8880 2021-02-10T11:21:48.255 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:48.255 7f8729670700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87241a8090 con 0x7f87240b7db0 2021-02-10T11:21:48.255 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:48.255 7f8729670700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87241a82b0 con 0x7f87240b8c80 2021-02-10T11:21:48.255 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:48.255 7f871bfff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f87240b8880 0x7f87241a7450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:48.255 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:48.255 7f871bfff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f87240b8880 0x7f87241a7450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:55056/0 (socket says 172.21.2.102:55056) 2021-02-10T11:21:48.255 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:48.255 7f871bfff700 1 -- 172.21.2.102:0/1229600059 learned_addr learned my addr 172.21.2.102:0/1229600059 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:21:48.255 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:48.255 7f8723ac0700 1 --2- 172.21.2.102:0/1229600059 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f87240b8c80 0x7f87241a7a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:48.255 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:48.255 7f87232bf700 1 --2- 172.21.2.102:0/1229600059 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f87240b7db0 0x7f87240b8260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:48.255 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:48.256 7f871bfff700 1 -- 172.21.2.102:0/1229600059 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f87240b8c80 msgr2=0x7f87241a7a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:48.255 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:48.256 7f871bfff700 1 --2- 172.21.2.102:0/1229600059 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f87240b8c80 0x7f87241a7a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:48.255 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:48.256 7f871bfff700 1 -- 172.21.2.102:0/1229600059 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f87240b7db0 msgr2=0x7f87240b8260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:48.255 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:48.256 7f871bfff700 1 --2- 172.21.2.102:0/1229600059 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f87240b7db0 0x7f87240b8260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:48.255 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:48.256 7f871bfff700 1 -- 172.21.2.102:0/1229600059 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8724190c10 con 0x7f87240b8880 2021-02-10T11:21:48.256 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:48.256 7f87232bf700 1 --2- 172.21.2.102:0/1229600059 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f87240b7db0 0x7f87240b8260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:21:48.256 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:48.256 7f871bfff700 1 --2- 172.21.2.102:0/1229600059 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f87240b8880 0x7f87241a7450 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 rx=0x7f870c00b600 tx=0x7f870c00b9a0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:48.256 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:48.256 7f8720aba700 1 -- 172.21.2.102:0/1229600059 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f870c002030 con 0x7f87240b8880 2021-02-10T11:21:48.256 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:48.256 7f8720aba700 1 -- 172.21.2.102:0/1229600059 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f870c002330 con 0x7f87240b8880 2021-02-10T11:21:48.256 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:48.256 7f8720aba700 1 -- 172.21.2.102:0/1229600059 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f870c002850 con 0x7f87240b8880 2021-02-10T11:21:48.256 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:48.256 7f8729670700 1 -- 172.21.2.102:0/1229600059 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8724190c10 con 0x7f87240b8880 2021-02-10T11:21:48.256 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:48.256 7f8729670700 1 -- 172.21.2.102:0/1229600059 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87240c1eb0 con 0x7f87240b8880 2021-02-10T11:21:48.256 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:48.257 7f8720aba700 1 -- 172.21.2.102:0/1229600059 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f870c002020 con 0x7f87240b8880 2021-02-10T11:21:48.256 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:48.257 7f8720aba700 1 --2- 172.21.2.102:0/1229600059 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f87080186c0 0x7f870801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:48.256 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:48.257 7f8720aba700 1 -- 172.21.2.102:0/1229600059 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f870c0323e0 con 0x7f87240b8880 2021-02-10T11:21:48.256 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:48.257 7f87232bf700 1 --2- 172.21.2.102:0/1229600059 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f87080186c0 0x7f870801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:48.256 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:48.257 7f87232bf700 1 --2- 172.21.2.102:0/1229600059 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f87080186c0 0x7f870801abc0 crc :-1 s=READY pgs=486 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:48.283 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:48.287 7f8729670700 1 -- 172.21.2.102:0/1229600059 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8724105b30 con 0x7f87240b8880 2021-02-10T11:21:48.286 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:48.290 7f8720aba700 1 -- 172.21.2.102:0/1229600059 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f870c0009b0 con 0x7f87240b8880 2021-02-10T11:21:48.416 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:48.420 7f8729670700 1 -- 172.21.2.102:0/1229600059 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000344463", "target": ["mgr", ""]}) v1 -- 0x7f87240c1eb0 con 0x7f87080186c0 2021-02-10T11:21:48.428 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:48.431 7f8720aba700 1 -- 172.21.2.102:0/1229600059 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f8714006930 con 0x7f87080186c0 2021-02-10T11:21:48.433 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:48.436 7f8729670700 1 -- 172.21.2.102:0/1229600059 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f87080186c0 msgr2=0x7f870801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:48.433 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:48.436 7f8729670700 1 --2- 172.21.2.102:0/1229600059 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f87080186c0 0x7f870801abc0 crc :-1 s=READY pgs=486 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:48.433 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:48.436 7f8729670700 1 -- 172.21.2.102:0/1229600059 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f87240b8880 msgr2=0x7f87241a7450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:48.433 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:48.436 7f8729670700 1 --2- 172.21.2.102:0/1229600059 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f87240b8880 0x7f87241a7450 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 rx=0x7f870c00b600 tx=0x7f870c00b9a0).stop 2021-02-10T11:21:48.433 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:48.437 7f8729670700 1 -- 172.21.2.102:0/1229600059 shutdown_connections 2021-02-10T11:21:48.433 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:48.437 7f8729670700 1 --2- 172.21.2.102:0/1229600059 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f87240b8c80 0x7f87241a7a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:48.433 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:48.437 7f8729670700 1 --2- 172.21.2.102:0/1229600059 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f87240b8880 0x7f87241a7450 unknown :-1 s=CLOSED pgs=998 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:48.433 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:48.437 7f8729670700 1 --2- 172.21.2.102:0/1229600059 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f87080186c0 0x7f870801abc0 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:48.433 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:48.437 7f8729670700 1 --2- 172.21.2.102:0/1229600059 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f87240b7db0 0x7f87240b8260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:48.434 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:48.437 7f8729670700 1 -- 172.21.2.102:0/1229600059 >> 172.21.2.102:0/1229600059 conn(0x7f8724094e10 msgr2=0x7f87240a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:48.434 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:48.437 7f8729670700 1 -- 172.21.2.102:0/1229600059 shutdown_connections 2021-02-10T11:21:48.434 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:48.437 7f8729670700 1 -- 172.21.2.102:0/1229600059 wait complete. 2021-02-10T11:21:48.442 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:21:48.870 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:48.857 7fd36f30f700 1 Processor -- start 2021-02-10T11:21:48.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:48.857 7fd36f30f700 1 -- start start 2021-02-10T11:21:48.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:48.857 7fd36f30f700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fd36807e6a0 0x7fd3680d1790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:48.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:48.857 7fd36f30f700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd36807ed80 con 0x7fd3680e5070 2021-02-10T11:21:48.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:48.857 7fd36f30f700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd36807eaa0 con 0x7fd3680e5470 2021-02-10T11:21:48.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:48.857 7fd36f30f700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3681b4080 con 0x7fd36807e6a0 2021-02-10T11:21:48.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:48.857 7fd36d12b700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fd3680e5070 0x7fd3681b3820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:47284/0 (socket says 172.21.2.102:47284) 2021-02-10T11:21:48.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:48.857 7fd36d12b700 1 -- 172.21.2.102:0/439164221 learned_addr learned my addr 172.21.2.102:0/439164221 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:21:48.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:48.858 7fd36d92c700 1 --2- 172.21.2.102:0/439164221 >> v2:172.21.2.113:3300/0 conn(0x7fd36807e6a0 0x7fd3680d1790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:48.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:48.858 7fd35bfff700 1 -- 172.21.2.102:0/439164221 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3701870566 0 0) 0x7fd35c001180 con 0x7fd3680e5070 2021-02-10T11:21:48.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:48.858 7fd35bfff700 1 -- 172.21.2.102:0/439164221 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd354002fe0 con 0x7fd3680e5070 2021-02-10T11:21:48.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:48.858 7fd35bfff700 1 -- 172.21.2.102:0/439164221 <== mon.1 v1:172.21.2.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1919838297 0 0) 0x7fd364001080 con 0x7fd3680e5470 2021-02-10T11:21:48.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:48.858 7fd35bfff700 1 -- 172.21.2.102:0/439164221 --> v1:172.21.2.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd354002fe0 con 0x7fd3680e5470 2021-02-10T11:21:48.872 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:48.858 7fd36d92c700 1 -- 172.21.2.102:0/439164221 >> v1:172.21.2.113:6789/0 conn(0x7fd3680e5470 legacy=0x7fd36807f300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:48.872 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:48.858 7fd36d92c700 1 -- 172.21.2.102:0/439164221 >> v1:172.21.2.102:6789/0 conn(0x7fd3680e5070 legacy=0x7fd3681b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:48.872 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:48.858 7fd36d92c700 1 -- 172.21.2.102:0/439164221 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3680d1db0 con 0x7fd36807e6a0 2021-02-10T11:21:48.872 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:48.858 7fd36d92c700 1 --2- 172.21.2.102:0/439164221 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd36807e6a0 0x7fd3680d1790 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 rx=0x7fd360006e50 tx=0x7fd360009e70).ready entity=mon.1 client_cookie=63e9394239b901ce server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:48.872 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:48.859 7fd35bfff700 1 -- 172.21.2.102:0/439164221 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd360006a20 con 0x7fd36807e6a0 2021-02-10T11:21:48.872 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:48.859 7fd35bfff700 1 -- 172.21.2.102:0/439164221 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd360006790 con 0x7fd36807e6a0 2021-02-10T11:21:48.872 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:48.859 7fd35bfff700 1 -- 172.21.2.102:0/439164221 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd36000b700 con 0x7fd36807e6a0 2021-02-10T11:21:48.872 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:48.859 7fd36f30f700 1 -- 172.21.2.102:0/439164221 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd36807e6a0 msgr2=0x7fd3680d1790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:48.872 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:48.859 7fd36f30f700 1 --2- 172.21.2.102:0/439164221 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd36807e6a0 0x7fd3680d1790 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 rx=0x7fd360006e50 tx=0x7fd360009e70).stop 2021-02-10T11:21:48.872 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:48.859 7fd36f30f700 1 -- 172.21.2.102:0/439164221 shutdown_connections 2021-02-10T11:21:48.872 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:48.859 7fd36f30f700 1 --2- 172.21.2.102:0/439164221 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd36807e6a0 0x7fd3680d1790 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:48.872 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:48.859 7fd36f30f700 1 -- 172.21.2.102:0/439164221 >> 172.21.2.102:0/439164221 conn(0x7fd3680ce690 msgr2=0x7fd3680e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:48.873 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:48.860 7fd36f30f700 1 -- 172.21.2.102:0/439164221 shutdown_connections 2021-02-10T11:21:48.873 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:48.860 7fd36f30f700 1 -- 172.21.2.102:0/439164221 wait complete. 2021-02-10T11:21:48.874 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:48.875 7fd36f30f700 1 Processor -- start 2021-02-10T11:21:48.874 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:48.875 7fd36f30f700 1 -- start start 2021-02-10T11:21:48.874 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:48.875 7fd36f30f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd368097250 0x7fd3681a1b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:48.874 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:48.875 7fd36f30f700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd36808d5a0 0x7fd3680ae3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:48.875 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:48.875 7fd36f30f700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd3680ae9c0 0x7fd3681a2870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:48.875 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:48.875 7fd36f30f700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3681a22f0 con 0x7fd368097250 2021-02-10T11:21:48.875 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:48.875 7fd36f30f700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3681b2d40 con 0x7fd36808d5a0 2021-02-10T11:21:48.875 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:48.875 7fd36f30f700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3681b2f20 con 0x7fd3680ae9c0 2021-02-10T11:21:48.875 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:48.875 7fd36d12b700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd368097250 0x7fd3681a1b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:48.875 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:48.875 7fd36d12b700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd368097250 0x7fd3681a1b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:55064/0 (socket says 172.21.2.102:55064) 2021-02-10T11:21:48.876 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:48.875 7fd36d12b700 1 -- 172.21.2.102:0/416665801 learned_addr learned my addr 172.21.2.102:0/416665801 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:21:48.876 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:48.875 7fd36c92a700 1 --2- 172.21.2.102:0/416665801 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd36808d5a0 0x7fd3680ae3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:48.876 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:48.875 7fd36d92c700 1 --2- 172.21.2.102:0/416665801 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd3680ae9c0 0x7fd3681a2870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:48.876 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:48.875 7fd36d12b700 1 -- 172.21.2.102:0/416665801 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd3680ae9c0 msgr2=0x7fd3681a2870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:48.876 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:48.875 7fd36d12b700 1 --2- 172.21.2.102:0/416665801 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd3680ae9c0 0x7fd3681a2870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:48.876 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:48.875 7fd36d12b700 1 -- 172.21.2.102:0/416665801 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd36808d5a0 msgr2=0x7fd3680ae3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:48.876 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:48.875 7fd36d12b700 1 --2- 172.21.2.102:0/416665801 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd36808d5a0 0x7fd3680ae3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:48.876 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:48.875 7fd36d12b700 1 -- 172.21.2.102:0/416665801 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd36808d9d0 con 0x7fd368097250 2021-02-10T11:21:48.876 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:48.876 7fd36d12b700 1 --2- 172.21.2.102:0/416665801 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd368097250 0x7fd3681a1b90 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 rx=0x7fd35c00a5a0 tx=0x7fd35c00a940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:48.876 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:48.876 7fd359ffb700 1 -- 172.21.2.102:0/416665801 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd35c00ad30 con 0x7fd368097250 2021-02-10T11:21:48.876 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:48.876 7fd359ffb700 1 -- 172.21.2.102:0/416665801 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd35c0070d0 con 0x7fd368097250 2021-02-10T11:21:48.876 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:48.876 7fd359ffb700 1 -- 172.21.2.102:0/416665801 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd35c004620 con 0x7fd368097250 2021-02-10T11:21:48.876 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:48.876 7fd36f30f700 1 -- 172.21.2.102:0/416665801 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd36808d9d0 con 0x7fd368097250 2021-02-10T11:21:48.877 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:48.876 7fd36f30f700 1 -- 172.21.2.102:0/416665801 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3681a3180 con 0x7fd368097250 2021-02-10T11:21:48.877 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:48.877 7fd359ffb700 1 -- 172.21.2.102:0/416665801 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd35c0043a0 con 0x7fd368097250 2021-02-10T11:21:48.877 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:48.877 7fd359ffb700 1 --2- 172.21.2.102:0/416665801 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd3480186c0 0x7fd34801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:48.877 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:48.877 7fd359ffb700 1 -- 172.21.2.102:0/416665801 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fd35c032550 con 0x7fd368097250 2021-02-10T11:21:48.877 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:48.877 7fd36c92a700 1 --2- 172.21.2.102:0/416665801 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd3480186c0 0x7fd34801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:48.877 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:48.877 7fd36c92a700 1 --2- 172.21.2.102:0/416665801 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd3480186c0 0x7fd34801abc0 crc :-1 s=READY pgs=487 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:48.902 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:48.906 7fd36f30f700 1 -- 172.21.2.102:0/416665801 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd368107170 con 0x7fd368097250 2021-02-10T11:21:48.905 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:48.909 7fd359ffb700 1 -- 172.21.2.102:0/416665801 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd35c000ef0 con 0x7fd368097250 2021-02-10T11:21:49.031 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:49.034 7fd36f30f700 1 -- 172.21.2.102:0/416665801 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd36807ed80 con 0x7fd368097250 2021-02-10T11:21:49.031 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:49.035 7fd359ffb700 1 -- 172.21.2.102:0/416665801 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v30) v1 ==== 74+0+13993 (secure 0 0 0) 0x7fd35c0043a0 con 0x7fd368097250 2021-02-10T11:21:49.032 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:21:49.032 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":30,"fsid":"68db30ea-dbce-4aef-a6df-f01c6fea0513","created":"2021-02-10 11:03:06.331557","modified":"2021-02-10 11:05:13.569263","last_up_change":"2021-02-10 11:03:32.690562","last_in_change":"2021-02-10 11:03:32.690562","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:03:37.322519","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:05:10.538786","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":5,"pool_name":"cephfs_data","create_time":"2021-02-10 11:05:11.559128","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"30","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"be691b4d-a039-408a-af80-9d53f02b6f1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6818","nonce":36088},{"type":"v1","addr":"172.21.2.102:6820","nonce":36088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6822","nonce":36088},{"type":"v1","addr":"172.21.2.102:6824","nonce":36088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6830","nonce":36088},{"type":"v1","addr":"172.21.2.102:6832","nonce":36088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6826","nonce":36088},{"type":"v1","addr":"172.21.2.102:6828","nonce":36088}]},"public_addr":"172.21.2.102:6820/36088","cluster_addr":"172.21.2.102:6824/36088","heartbeat_back_addr":"172.21.2.102:6832/36088","heartbeat_front_addr":"172.21.2.102:6828/36088","state":["exists","up"]},{"osd":1,"uuid":"64e3ccc8-9c69-4d7c-865e-af8e83822635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6819","nonce":36087},{"type":"v1","addr":"172.21.2.102:6821","nonce":36087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6823","nonce":36087},{"type":"v1","addr":"172.21.2.102:6825","nonce":36087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6831","nonce":36087},{"type":"v1","addr":"172.21.2.102:6833","nonce":36087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6827","nonce":36087},{"type":"v1","addr":"172.21.2.102:6829","nonce":36087}]},"public_addr":"172.21.2.102:6821/36087","cluster_addr":"172.21.2.102:6825/36087","heartbeat_back_addr":"172.21.2.102:6833/36087","heartbeat_front_addr":"172.21.2.102:6829/36087","state":["exists","up"]},{"osd":2,"uuid":"153700af-747f-481f-bdf4-d6072d3a842f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6810","nonce":36089},{"type":"v1","addr":"172.21.2.102:6811","nonce":36089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6812","nonce":36089},{"type":"v1","addr":"172.21.2.102:6813","nonce":36089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6816","nonce":36089},{"type":"v1","addr":"172.21.2.102:6817","nonce":36089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6814","nonce":36089},{"type":"v1","addr":"172.21.2.102:6815","nonce":36089}]},"public_addr":"172.21.2.102:6811/36089","cluster_addr":"172.21.2.102:6813/36089","heartbeat_back_addr":"172.21.2.102:6817/36089","heartbeat_front_addr":"172.21.2.102:6815/36089","state":["exists","up"]},{"osd":3,"uuid":"cbc678f4-83a9-45e6-8830-2f7c232b1ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6802","nonce":36090},{"type":"v1","addr":"172.21.2.102:6803","nonce":36090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6804","nonce":36090},{"type":"v1","addr":"172.21.2.102:6805","nonce":36090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6808","nonce":36090},{"type":"v1","addr":"172.21.2.102:6809","nonce":36090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6806","nonce":36090},{"type":"v1","addr":"172.21.2.102:6807","nonce":36090}]},"public_addr":"172.21.2.102:6803/36090","cluster_addr":"172.21.2.102:6805/36090","heartbeat_back_addr":"172.21.2.102:6809/36090","heartbeat_front_addr":"172.21.2.102:6807/36090","state":["exists","up"]},{"osd":4,"uuid":"c01e4008-b258-4f10-afd2-9e7f37214351","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6808","nonce":36368},{"type":"v1","addr":"172.21.2.113:6811","nonce":36368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6814","nonce":36368},{"type":"v1","addr":"172.21.2.113:6818","nonce":36368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6826","nonce":36368},{"type":"v1","addr":"172.21.2.113:6828","nonce":36368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6820","nonce":36368},{"type":"v1","addr":"172.21.2.113:6824","nonce":36368}]},"public_addr":"172.21.2.113:6811/36368","cluster_addr":"172.21.2.113:6818/36368","heartbeat_back_addr":"172.21.2.113:6828/36368","heartbeat_front_addr":"172.21.2.113:6824/36368","state":["exists","up"]},{"osd":5,"uuid":"551f6007-daf8-4a98-8ee0-c2b16bcccb5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6802","nonce":36366},{"type":"v1","addr":"172.21.2.113:6805","nonce":36366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6807","nonce":36366},{"type":"v1","addr":"172.21.2.113:6810","nonce":36366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6819","nonce":36366},{"type":"v1","addr":"172.21.2.113:6822","nonce":36366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6812","nonce":36366},{"type":"v1","addr":"172.21.2.113:6815","nonce":36366}]},"public_addr":"172.21.2.113:6805/36366","cluster_addr":"172.21.2.113:6810/36366","heartbeat_back_addr":"172.21.2.113:6822/36366","heartbeat_front_addr":"172.21.2.113:6815/36366","state":["exists","up"]},{"osd":6,"uuid":"bf67c040-0e50-48be-b102-c3943bc9ebef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6800","nonce":36367},{"type":"v1","addr":"172.21.2.113:6801","nonce":36367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6803","nonce":36367},{"type":"v1","addr":"172.21.2.113:6804","nonce":36367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6813","nonce":36367},{"type":"v1","addr":"172.21.2.113:6816","nonce":36367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6806","nonce":36367},{"type":"v1","addr":"172.21.2.113:6809","nonce":36367}]},"public_addr":"172.21.2.113:6801/36367","cluster_addr":"172.21.2.113:6804/36367","heartbeat_back_addr":"172.21.2.113:6816/36367","heartbeat_front_addr":"172.21.2.113:6809/36367","state":["exists","up"]},{"osd":7,"uuid":"f5461f80-018d-428f-be89-2eb2bcbe7bb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6817","nonce":36365},{"type":"v1","addr":"172.21.2.113:6821","nonce":36365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6823","nonce":36365},{"type":"v1","addr":"172.21.2.113:6825","nonce":36365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6830","nonce":36365},{"type":"v1","addr":"172.21.2.113:6831","nonce":36365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6827","nonce":36365},{"type":"v1","addr":"172.21.2.113:6829","nonce":36365}]},"public_addr":"172.21.2.113:6821/36365","cluster_addr":"172.21.2.113:6825/36365","heartbeat_back_addr":"172.21.2.113:6831/36365","heartbeat_front_addr":"172.21.2.113:6829/36365","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:21:49.033 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:49.037 7fd36f30f700 1 -- 172.21.2.102:0/416665801 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd3480186c0 msgr2=0x7fd34801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:49.033 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:49.037 7fd36f30f700 1 --2- 172.21.2.102:0/416665801 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd3480186c0 0x7fd34801abc0 crc :-1 s=READY pgs=487 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:49.033 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:49.037 7fd36f30f700 1 -- 172.21.2.102:0/416665801 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd368097250 msgr2=0x7fd3681a1b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:49.033 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:49.037 7fd36f30f700 1 --2- 172.21.2.102:0/416665801 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd368097250 0x7fd3681a1b90 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 rx=0x7fd35c00a5a0 tx=0x7fd35c00a940).stop 2021-02-10T11:21:49.034 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:49.037 7fd36f30f700 1 -- 172.21.2.102:0/416665801 shutdown_connections 2021-02-10T11:21:49.034 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:49.037 7fd36f30f700 1 --2- 172.21.2.102:0/416665801 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd3680ae9c0 0x7fd3681a2870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:49.034 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:49.037 7fd36f30f700 1 --2- 172.21.2.102:0/416665801 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd3480186c0 0x7fd34801abc0 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:49.034 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:49.037 7fd36f30f700 1 --2- 172.21.2.102:0/416665801 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd36808d5a0 0x7fd3680ae3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:49.034 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:49.037 7fd36f30f700 1 --2- 172.21.2.102:0/416665801 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd368097250 0x7fd3681a1b90 unknown :-1 s=CLOSED pgs=1000 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:49.034 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:49.037 7fd36f30f700 1 -- 172.21.2.102:0/416665801 >> 172.21.2.102:0/416665801 conn(0x7fd3681ac620 msgr2=0x7fd3680896e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:49.034 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:49.037 7fd36f30f700 1 -- 172.21.2.102:0/416665801 shutdown_connections 2021-02-10T11:21:49.034 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:49.037 7fd36f30f700 1 -- 172.21.2.102:0/416665801 wait complete. 2021-02-10T11:21:49.042 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:03:37.322519', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 4, 'pool_name': 'cephfs_metadata', 'create_time': '2021-02-10 11:05:10.538786', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '29', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 5, 'pool_name': 'cephfs_data', 'create_time': '2021-02-10 11:05:11.559128', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '30', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-02-10T11:21:49.043 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:21:49.471 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:49.459 7f8b41aee700 1 Processor -- start 2021-02-10T11:21:49.472 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:49.459 7f8b41aee700 1 -- start start 2021-02-10T11:21:49.472 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:49.460 7f8b41aee700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f8b3c1b0470 0x7f8b3c1b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:49.472 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:49.460 7f8b41aee700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f8b3c0f20e0 0x7f8b3c0f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:49.472 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:49.460 7f8b41aee700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b3c0f0b50 con 0x7f8b3c0ef610 2021-02-10T11:21:49.472 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:49.460 7f8b41aee700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b3c0f0820 con 0x7f8b3c0f20e0 2021-02-10T11:21:49.472 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:49.460 7f8b41aee700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b3c0f24e0 con 0x7f8b3c1b0470 2021-02-10T11:21:49.472 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:49.460 7f8b3bfff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f8b3c0f20e0 0x7f8b3c0f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:49.472 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:49.460 7f8b3bfff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f8b3c0f20e0 0x7f8b3c0f0200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:55072/0 (socket says 172.21.2.102:55072) 2021-02-10T11:21:49.472 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:49.460 7f8b3bfff700 1 -- 172.21.2.102:0/3185737775 learned_addr learned my addr 172.21.2.102:0/3185737775 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:21:49.472 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:49.460 7f8b3bfff700 1 -- 172.21.2.102:0/3185737775 >> v2:172.21.2.113:3300/0 conn(0x7f8b3c1b0470 msgr2=0x7f8b3c1b0870 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:21:49.472 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:49.460 7f8b3bfff700 1 --2- 172.21.2.102:0/3185737775 >> v2:172.21.2.113:3300/0 conn(0x7f8b3c1b0470 0x7f8b3c1b0870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:21:49.473 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:49.460 7f8b3bfff700 1 -- 172.21.2.102:0/3185737775 >> v1:172.21.2.113:6789/0 conn(0x7f8b3c0ef610 legacy=0x7f8b3c0efa10 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:21:49.473 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:49.460 7f8b3bfff700 1 -- 172.21.2.102:0/3185737775 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b3c0e52e0 con 0x7f8b3c0f20e0 2021-02-10T11:21:49.473 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:49.460 7f8b3bfff700 1 --2- 172.21.2.102:0/3185737775 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8b3c0f20e0 0x7f8b3c0f0200 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 rx=0x7f8b34006f00 tx=0x7f8b34009f10).ready entity=mon.0 client_cookie=f983fe76ab6412 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:49.473 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:49.461 7f8b3a7fc700 1 -- 172.21.2.102:0/3185737775 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8b3400b3b0 con 0x7f8b3c0f20e0 2021-02-10T11:21:49.473 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:49.461 7f8b3a7fc700 1 -- 172.21.2.102:0/3185737775 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b3400b630 con 0x7f8b3c0f20e0 2021-02-10T11:21:49.473 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:49.461 7f8b3a7fc700 1 -- 172.21.2.102:0/3185737775 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8b3400bb10 con 0x7f8b3c0f20e0 2021-02-10T11:21:49.473 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:49.461 7f8b41aee700 1 -- 172.21.2.102:0/3185737775 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8b3c0f20e0 msgr2=0x7f8b3c0f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:49.473 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:49.461 7f8b41aee700 1 --2- 172.21.2.102:0/3185737775 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8b3c0f20e0 0x7f8b3c0f0200 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 rx=0x7f8b34006f00 tx=0x7f8b34009f10).stop 2021-02-10T11:21:49.473 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:49.461 7f8b41aee700 1 -- 172.21.2.102:0/3185737775 shutdown_connections 2021-02-10T11:21:49.473 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:49.461 7f8b41aee700 1 --2- 172.21.2.102:0/3185737775 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8b3c0f20e0 0x7f8b3c0f0200 unknown :-1 s=CLOSED pgs=1001 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:49.473 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:49.461 7f8b41aee700 1 --2- 172.21.2.102:0/3185737775 >> v2:172.21.2.113:3300/0 conn(0x7f8b3c1b0470 0x7f8b3c1b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:21:49.474 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:49.461 7f8b41aee700 1 -- 172.21.2.102:0/3185737775 >> 172.21.2.102:0/3185737775 conn(0x7f8b3c0ce3e0 msgr2=0x7f8b3c0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:49.474 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:49.461 7f8b41aee700 1 -- 172.21.2.102:0/3185737775 shutdown_connections 2021-02-10T11:21:49.474 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:49.461 7f8b41aee700 1 -- 172.21.2.102:0/3185737775 wait complete. 2021-02-10T11:21:49.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:49.476 7f8b41aee700 1 Processor -- start 2021-02-10T11:21:49.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:49.476 7f8b41aee700 1 -- start start 2021-02-10T11:21:49.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:49.476 7f8b41aee700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8b3c0c36a0 0x7f8b3c0c2b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:49.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:49.476 7f8b41aee700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8b3c0c3190 0x7f8b3c0b1d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:49.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:49.476 7f8b41aee700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8b3c0b2370 0x7f8b3c0b2d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:49.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:49.476 7f8b41aee700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b3c0b27c0 con 0x7f8b3c0b2370 2021-02-10T11:21:49.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:49.476 7f8b41aee700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b3c0b3360 con 0x7f8b3c0c36a0 2021-02-10T11:21:49.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:49.476 7f8b41aee700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b3c0b3540 con 0x7f8b3c0c3190 2021-02-10T11:21:49.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:49.477 7f8b3bfff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8b3c0b2370 0x7f8b3c0b2d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:49.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:49.477 7f8b3bfff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8b3c0b2370 0x7f8b3c0b2d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:55076/0 (socket says 172.21.2.102:55076) 2021-02-10T11:21:49.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:49.477 7f8b3bfff700 1 -- 172.21.2.102:0/3523907513 learned_addr learned my addr 172.21.2.102:0/3523907513 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:21:49.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:49.477 7f8b3bfff700 1 -- 172.21.2.102:0/3523907513 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8b3c0c3190 msgr2=0x7f8b3c0b1d50 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:21:49.477 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:49.477 7f8b3bfff700 1 --2- 172.21.2.102:0/3523907513 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8b3c0c3190 0x7f8b3c0b1d50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:21:49.477 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:49.477 7f8b3bfff700 1 -- 172.21.2.102:0/3523907513 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8b3c0c36a0 msgr2=0x7f8b3c0c2b70 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:21:49.477 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:49.477 7f8b3bfff700 1 --2- 172.21.2.102:0/3523907513 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8b3c0c36a0 0x7f8b3c0c2b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:21:49.477 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:49.477 7f8b3bfff700 1 -- 172.21.2.102:0/3523907513 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b3c0b3760 con 0x7f8b3c0b2370 2021-02-10T11:21:49.477 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:49.477 7f8b3bfff700 1 --2- 172.21.2.102:0/3523907513 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8b3c0b2370 0x7f8b3c0b2d40 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 rx=0x7f8b3400b9b0 tx=0x7f8b3400bd50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:49.477 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:49.477 7f8b23fff700 1 -- 172.21.2.102:0/3523907513 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8b34005820 con 0x7f8b3c0b2370 2021-02-10T11:21:49.477 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:49.477 7f8b23fff700 1 -- 172.21.2.102:0/3523907513 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b34005aa0 con 0x7f8b3c0b2370 2021-02-10T11:21:49.477 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:49.477 7f8b23fff700 1 -- 172.21.2.102:0/3523907513 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8b34006030 con 0x7f8b3c0b2370 2021-02-10T11:21:49.477 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:49.477 7f8b41aee700 1 -- 172.21.2.102:0/3523907513 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b3c0b3760 con 0x7f8b3c0b2370 2021-02-10T11:21:49.477 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:49.477 7f8b41aee700 1 -- 172.21.2.102:0/3523907513 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b3c0b3cf0 con 0x7f8b3c0b2370 2021-02-10T11:21:49.477 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:49.478 7f8b23fff700 1 -- 172.21.2.102:0/3523907513 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f8b34009450 con 0x7f8b3c0b2370 2021-02-10T11:21:49.477 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:49.478 7f8b23fff700 1 --2- 172.21.2.102:0/3523907513 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8b1c0186c0 0x7f8b1c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:49.478 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:49.478 7f8b23fff700 1 -- 172.21.2.102:0/3523907513 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f8b34032950 con 0x7f8b3c0b2370 2021-02-10T11:21:49.478 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:49.478 7f8b3b7fe700 1 --2- 172.21.2.102:0/3523907513 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8b1c0186c0 0x7f8b1c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:49.478 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:49.478 7f8b3b7fe700 1 --2- 172.21.2.102:0/3523907513 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8b1c0186c0 0x7f8b1c01abc0 crc :-1 s=READY pgs=488 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:49.506 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:49.510 7f8b41aee700 1 -- 172.21.2.102:0/3523907513 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b3c0edd10 con 0x7f8b3c0b2370 2021-02-10T11:21:49.509 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:49.513 7f8b23fff700 1 -- 172.21.2.102:0/3523907513 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f8b340061f0 con 0x7f8b3c0b2370 2021-02-10T11:21:49.635 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:49.639 7f8b41aee700 1 -- 172.21.2.102:0/3523907513 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f8b3c0e52e0 con 0x7f8b3c0b2370 2021-02-10T11:21:49.636 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:49.640 7f8b23fff700 1 -- 172.21.2.102:0/3523907513 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v30) v1 ==== 92+0+10 (secure 0 0 0) 0x7f8b34006400 con 0x7f8b3c0b2370 2021-02-10T11:21:49.636 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:21:49.638 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:49.641 7f8b41aee700 1 -- 172.21.2.102:0/3523907513 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8b1c0186c0 msgr2=0x7f8b1c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:49.638 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:49.641 7f8b41aee700 1 --2- 172.21.2.102:0/3523907513 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8b1c0186c0 0x7f8b1c01abc0 crc :-1 s=READY pgs=488 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:49.638 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:49.641 7f8b41aee700 1 -- 172.21.2.102:0/3523907513 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8b3c0b2370 msgr2=0x7f8b3c0b2d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:49.638 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:49.641 7f8b41aee700 1 --2- 172.21.2.102:0/3523907513 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8b3c0b2370 0x7f8b3c0b2d40 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 rx=0x7f8b3400b9b0 tx=0x7f8b3400bd50).stop 2021-02-10T11:21:49.638 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:49.642 7f8b41aee700 1 -- 172.21.2.102:0/3523907513 shutdown_connections 2021-02-10T11:21:49.638 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:49.642 7f8b41aee700 1 --2- 172.21.2.102:0/3523907513 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8b3c0b2370 0x7f8b3c0b2d40 unknown :-1 s=CLOSED pgs=1002 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:49.638 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:49.642 7f8b41aee700 1 --2- 172.21.2.102:0/3523907513 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8b3c0c3190 0x7f8b3c0b1d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:21:49.638 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:49.642 7f8b41aee700 1 --2- 172.21.2.102:0/3523907513 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8b1c0186c0 0x7f8b1c01abc0 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:49.638 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:49.642 7f8b41aee700 1 --2- 172.21.2.102:0/3523907513 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8b3c0c36a0 0x7f8b3c0c2b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:21:49.638 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:49.642 7f8b41aee700 1 -- 172.21.2.102:0/3523907513 >> 172.21.2.102:0/3523907513 conn(0x7f8b3c09f630 msgr2=0x7f8b3c0a0130 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:49.639 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:49.642 7f8b41aee700 1 -- 172.21.2.102:0/3523907513 shutdown_connections 2021-02-10T11:21:49.639 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:49.642 7f8b41aee700 1 -- 172.21.2.102:0/3523907513 wait complete. 2021-02-10T11:21:49.646 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-02-10T11:21:50.075 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:50.062 7f60c2161700 1 Processor -- start 2021-02-10T11:21:50.075 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:50.062 7f60c2161700 1 -- start start 2021-02-10T11:21:50.075 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:50.063 7f60c2161700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f60bc1b0470 0x7f60bc1b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:50.075 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:50.063 7f60c2161700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f60bc0ef610 0x7f60bc0efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:50.075 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:50.063 7f60c2161700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f60bc0f0030 0x7f60bc0f0430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:50.075 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:50.063 7f60c2161700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60bc0e5ea0 con 0x7f60bc1b0470 2021-02-10T11:21:50.075 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:50.063 7f60c2161700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60bc0f0b50 con 0x7f60bc0ef610 2021-02-10T11:21:50.075 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:50.063 7f60c2161700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60bc0e52e0 con 0x7f60bc0f0030 2021-02-10T11:21:50.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:50.063 7f60bb7fe700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f60bc1b0470 0x7f60bc1b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:50.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:50.063 7f60bb7fe700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f60bc1b0470 0x7f60bc1b0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:55082/0 (socket says 172.21.2.102:55082) 2021-02-10T11:21:50.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:50.063 7f60bb7fe700 1 -- 172.21.2.102:0/177191974 learned_addr learned my addr 172.21.2.102:0/177191974 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:21:50.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:50.063 7f60bb7fe700 1 -- 172.21.2.102:0/177191974 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f60bc0f0030 msgr2=0x7f60bc0f0430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:21:50.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:50.063 7f60bb7fe700 1 --2- 172.21.2.102:0/177191974 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f60bc0f0030 0x7f60bc0f0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:21:50.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:50.063 7f60bb7fe700 1 -- 172.21.2.102:0/177191974 >> v2:172.21.2.113:3300/0 conn(0x7f60bc0ef610 msgr2=0x7f60bc0efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:21:50.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:50.063 7f60baffd700 1 --2- 172.21.2.102:0/177191974 >> v2:172.21.2.113:3300/0 conn(0x7f60bc0ef610 0x7f60bc0efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:50.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:50.063 7f60bb7fe700 1 --2- 172.21.2.102:0/177191974 >> v2:172.21.2.113:3300/0 conn(0x7f60bc0ef610 0x7f60bc0efa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:50.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:50.063 7f60bb7fe700 1 -- 172.21.2.102:0/177191974 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60bc0f10d0 con 0x7f60bc1b0470 2021-02-10T11:21:50.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:50.063 7f60baffd700 1 --2- 172.21.2.102:0/177191974 >> v2:172.21.2.113:3300/0 conn(0x7f60bc0ef610 0x7f60bc0efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:21:50.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:50.064 7f60bb7fe700 1 --2- 172.21.2.102:0/177191974 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f60bc1b0470 0x7f60bc1b0870 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 rx=0x7f60a4006700 tx=0x7f60a4009b00).ready entity=mon.0 client_cookie=205f052afb1ce249 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:50.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:50.064 7f60ba7fc700 1 -- 172.21.2.102:0/177191974 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f60a400c0c0 con 0x7f60bc1b0470 2021-02-10T11:21:50.077 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:50.064 7f60ba7fc700 1 -- 172.21.2.102:0/177191974 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60a400c400 con 0x7f60bc1b0470 2021-02-10T11:21:50.077 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:50.064 7f60ba7fc700 1 -- 172.21.2.102:0/177191974 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f60a400c920 con 0x7f60bc1b0470 2021-02-10T11:21:50.077 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:50.064 7f60c2161700 1 -- 172.21.2.102:0/177191974 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f60bc1b0470 msgr2=0x7f60bc1b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:50.077 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:50.064 7f60c2161700 1 --2- 172.21.2.102:0/177191974 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f60bc1b0470 0x7f60bc1b0870 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 rx=0x7f60a4006700 tx=0x7f60a4009b00).stop 2021-02-10T11:21:50.077 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:50.065 7f60c2161700 1 -- 172.21.2.102:0/177191974 shutdown_connections 2021-02-10T11:21:50.077 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:50.065 7f60c2161700 1 --2- 172.21.2.102:0/177191974 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f60bc0f0030 0x7f60bc0f0430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:21:50.077 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:50.065 7f60c2161700 1 --2- 172.21.2.102:0/177191974 >> v2:172.21.2.113:3300/0 conn(0x7f60bc0ef610 0x7f60bc0efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:50.077 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:50.065 7f60c2161700 1 --2- 172.21.2.102:0/177191974 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f60bc1b0470 0x7f60bc1b0870 unknown :-1 s=CLOSED pgs=1003 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:50.077 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:50.065 7f60c2161700 1 -- 172.21.2.102:0/177191974 >> 172.21.2.102:0/177191974 conn(0x7f60bc0ce3e0 msgr2=0x7f60bc0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:50.077 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:50.065 7f60c2161700 1 -- 172.21.2.102:0/177191974 shutdown_connections 2021-02-10T11:21:50.077 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:50.065 7f60c2161700 1 -- 172.21.2.102:0/177191974 wait complete. 2021-02-10T11:21:50.079 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:50.079 7f60c2161700 1 Processor -- start 2021-02-10T11:21:50.079 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:50.079 7f60c2161700 1 -- start start 2021-02-10T11:21:50.079 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:50.080 7f60c2161700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f60bc0b7c20 0x7f60bc0b8410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:50.079 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:50.080 7f60c2161700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f60bc0b8a30 0x7f60bc0c1f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:50.079 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:50.080 7f60c2161700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f60bc0c25a0 0x7f60bc0c29a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:50.079 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:50.080 7f60c2161700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60bc0c1a80 con 0x7f60bc0b7c20 2021-02-10T11:21:50.079 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:50.080 7f60c2161700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60bc0b8020 con 0x7f60bc0c25a0 2021-02-10T11:21:50.079 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:50.080 7f60c2161700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60bc09b9c0 con 0x7f60bc0b8a30 2021-02-10T11:21:50.079 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:50.080 7f60bb7fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f60bc0b7c20 0x7f60bc0b8410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:50.079 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:50.080 7f60bb7fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f60bc0b7c20 0x7f60bc0b8410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:55088/0 (socket says 172.21.2.102:55088) 2021-02-10T11:21:50.080 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:50.080 7f60bb7fe700 1 -- 172.21.2.102:0/3617830881 learned_addr learned my addr 172.21.2.102:0/3617830881 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:21:50.080 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:50.080 7f60bb7fe700 1 -- 172.21.2.102:0/3617830881 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f60bc0b8a30 msgr2=0x7f60bc0c1f80 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:21:50.080 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:50.080 7f60bb7fe700 1 --2- 172.21.2.102:0/3617830881 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f60bc0b8a30 0x7f60bc0c1f80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:21:50.080 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:50.080 7f60bb7fe700 1 -- 172.21.2.102:0/3617830881 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f60bc0c25a0 msgr2=0x7f60bc0c29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:50.080 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:50.080 7f60bb7fe700 1 --2- 172.21.2.102:0/3617830881 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f60bc0c25a0 0x7f60bc0c29a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:21:50.080 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:50.080 7f60bb7fe700 1 -- 172.21.2.102:0/3617830881 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60bc0b8e30 con 0x7f60bc0b7c20 2021-02-10T11:21:50.080 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:50.080 7f60bb7fe700 1 --2- 172.21.2.102:0/3617830881 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f60bc0b7c20 0x7f60bc0b8410 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 rx=0x7f60a4006f20 tx=0x7f60a400b9e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:50.080 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:50.081 7f60b3fff700 1 -- 172.21.2.102:0/3617830881 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f60a400bdd0 con 0x7f60bc0b7c20 2021-02-10T11:21:50.080 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:50.081 7f60b3fff700 1 -- 172.21.2.102:0/3617830881 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60a4006440 con 0x7f60bc0b7c20 2021-02-10T11:21:50.080 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:50.081 7f60b3fff700 1 -- 172.21.2.102:0/3617830881 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f60a40052e0 con 0x7f60bc0b7c20 2021-02-10T11:21:50.080 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:50.081 7f60c2161700 1 -- 172.21.2.102:0/3617830881 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60bc0b8e30 con 0x7f60bc0b7c20 2021-02-10T11:21:50.080 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:50.081 7f60c2161700 1 -- 172.21.2.102:0/3617830881 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60bc0c32b0 con 0x7f60bc0b7c20 2021-02-10T11:21:50.080 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:50.081 7f60b3fff700 1 -- 172.21.2.102:0/3617830881 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f60a40052e0 con 0x7f60bc0b7c20 2021-02-10T11:21:50.081 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:50.081 7f60b3fff700 1 --2- 172.21.2.102:0/3617830881 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f60a00186c0 0x7f60a001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:50.081 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:50.081 7f60b3fff700 1 -- 172.21.2.102:0/3617830881 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f60a4032570 con 0x7f60bc0b7c20 2021-02-10T11:21:50.081 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:50.081 7f60baffd700 1 --2- 172.21.2.102:0/3617830881 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f60a00186c0 0x7f60a001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:50.081 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:50.081 7f60baffd700 1 --2- 172.21.2.102:0/3617830881 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f60a00186c0 0x7f60a001abc0 crc :-1 s=READY pgs=489 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:50.109 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:50.113 7f60c2161700 1 -- 172.21.2.102:0/3617830881 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60bc105b30 con 0x7f60bc0b7c20 2021-02-10T11:21:50.111 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:50.115 7f60b3fff700 1 -- 172.21.2.102:0/3617830881 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f60a402cf20 con 0x7f60bc0b7c20 2021-02-10T11:21:50.241 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:50.245 7f60c2161700 1 -- 172.21.2.102:0/3617830881 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7f60bc105b30 con 0x7f60bc0b7c20 2021-02-10T11:21:50.242 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:50.246 7f60b3fff700 1 -- 172.21.2.102:0/3617830881 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v30) v1 ==== 104+0+10 (secure 0 0 0) 0x7f60a402cf20 con 0x7f60bc0b7c20 2021-02-10T11:21:50.242 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:21:50.243 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:50.248 7f60c2161700 1 -- 172.21.2.102:0/3617830881 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f60a00186c0 msgr2=0x7f60a001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:50.244 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:50.248 7f60c2161700 1 --2- 172.21.2.102:0/3617830881 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f60a00186c0 0x7f60a001abc0 crc :-1 s=READY pgs=489 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:50.244 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:50.248 7f60c2161700 1 -- 172.21.2.102:0/3617830881 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f60bc0b7c20 msgr2=0x7f60bc0b8410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:50.244 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:50.248 7f60c2161700 1 --2- 172.21.2.102:0/3617830881 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f60bc0b7c20 0x7f60bc0b8410 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 rx=0x7f60a4006f20 tx=0x7f60a400b9e0).stop 2021-02-10T11:21:50.244 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:50.248 7f60c2161700 1 -- 172.21.2.102:0/3617830881 shutdown_connections 2021-02-10T11:21:50.244 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:50.248 7f60c2161700 1 --2- 172.21.2.102:0/3617830881 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f60a00186c0 0x7f60a001abc0 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:50.245 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:50.248 7f60c2161700 1 --2- 172.21.2.102:0/3617830881 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f60bc0c25a0 0x7f60bc0c29a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:21:50.245 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:50.248 7f60c2161700 1 --2- 172.21.2.102:0/3617830881 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f60bc0b8a30 0x7f60bc0c1f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:21:50.245 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:50.248 7f60c2161700 1 --2- 172.21.2.102:0/3617830881 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f60bc0b7c20 0x7f60bc0b8410 unknown :-1 s=CLOSED pgs=1004 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:50.245 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:50.248 7f60c2161700 1 -- 172.21.2.102:0/3617830881 >> 172.21.2.102:0/3617830881 conn(0x7f60bc094e10 msgr2=0x7f60bc0a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:50.245 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:50.248 7f60c2161700 1 -- 172.21.2.102:0/3617830881 shutdown_connections 2021-02-10T11:21:50.245 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:50.248 7f60c2161700 1 -- 172.21.2.102:0/3617830881 wait complete. 2021-02-10T11:21:50.252 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-02-10T11:21:50.680 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:50.668 7f7bb600f700 1 Processor -- start 2021-02-10T11:21:50.681 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:50.668 7f7bb600f700 1 -- start start 2021-02-10T11:21:50.681 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:50.669 7f7bb600f700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f7bb01b0470 0x7f7bb01b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:50.681 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:50.669 7f7bb600f700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f7bb00f20e0 0x7f7bb00f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:50.681 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:50.669 7f7bb600f700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bb00f0b50 con 0x7f7bb00ef610 2021-02-10T11:21:50.681 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:50.669 7f7bb600f700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7bb00f0820 con 0x7f7bb00f20e0 2021-02-10T11:21:50.681 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:50.669 7f7bb600f700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7bb00f24e0 con 0x7f7bb01b0470 2021-02-10T11:21:50.681 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:50.669 7f7baffff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f7bb00f20e0 0x7f7bb00f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:50.681 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:50.669 7f7baffff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f7bb00f20e0 0x7f7bb00f0200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:55096/0 (socket says 172.21.2.102:55096) 2021-02-10T11:21:50.681 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:50.669 7f7baffff700 1 -- 172.21.2.102:0/2489633871 learned_addr learned my addr 172.21.2.102:0/2489633871 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:21:50.681 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:50.669 7f7baffff700 1 -- 172.21.2.102:0/2489633871 >> v2:172.21.2.113:3300/0 conn(0x7f7bb01b0470 msgr2=0x7f7bb01b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:21:50.681 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:50.669 7f7baffff700 1 --2- 172.21.2.102:0/2489633871 >> v2:172.21.2.113:3300/0 conn(0x7f7bb01b0470 0x7f7bb01b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:21:50.682 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:50.669 7f7baffff700 1 -- 172.21.2.102:0/2489633871 >> v1:172.21.2.113:6789/0 conn(0x7f7bb00ef610 legacy=0x7f7bb00efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:21:50.682 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:50.669 7f7baffff700 1 -- 172.21.2.102:0/2489633871 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bb00e52e0 con 0x7f7bb00f20e0 2021-02-10T11:21:50.682 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:50.670 7f7baffff700 1 --2- 172.21.2.102:0/2489633871 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7bb00f20e0 0x7f7bb00f0200 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 rx=0x7f7ba8006f00 tx=0x7f7ba8009f10).ready entity=mon.0 client_cookie=3a77b40aa7bc6b74 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:50.682 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:50.670 7f7bae7fc700 1 -- 172.21.2.102:0/2489633871 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7ba800b3b0 con 0x7f7bb00f20e0 2021-02-10T11:21:50.682 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:50.670 7f7bae7fc700 1 -- 172.21.2.102:0/2489633871 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ba800b730 con 0x7f7bb00f20e0 2021-02-10T11:21:50.682 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:50.670 7f7bae7fc700 1 -- 172.21.2.102:0/2489633871 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7ba800bc10 con 0x7f7bb00f20e0 2021-02-10T11:21:50.682 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:50.670 7f7bb600f700 1 -- 172.21.2.102:0/2489633871 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7bb00f20e0 msgr2=0x7f7bb00f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:50.682 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:50.670 7f7bb600f700 1 --2- 172.21.2.102:0/2489633871 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7bb00f20e0 0x7f7bb00f0200 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 rx=0x7f7ba8006f00 tx=0x7f7ba8009f10).stop 2021-02-10T11:21:50.682 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:50.670 7f7bb600f700 1 -- 172.21.2.102:0/2489633871 shutdown_connections 2021-02-10T11:21:50.682 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:50.670 7f7bb600f700 1 --2- 172.21.2.102:0/2489633871 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7bb00f20e0 0x7f7bb00f0200 unknown :-1 s=CLOSED pgs=1005 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:50.682 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:50.670 7f7bb600f700 1 --2- 172.21.2.102:0/2489633871 >> v2:172.21.2.113:3300/0 conn(0x7f7bb01b0470 0x7f7bb01b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:21:50.683 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:50.670 7f7bb600f700 1 -- 172.21.2.102:0/2489633871 >> 172.21.2.102:0/2489633871 conn(0x7f7bb00ce3e0 msgr2=0x7f7bb00e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:50.683 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:50.671 7f7bb600f700 1 -- 172.21.2.102:0/2489633871 shutdown_connections 2021-02-10T11:21:50.683 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:50.671 7f7bb600f700 1 -- 172.21.2.102:0/2489633871 wait complete. 2021-02-10T11:21:50.684 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:50.685 7f7bb600f700 1 Processor -- start 2021-02-10T11:21:50.684 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:50.685 7f7bb600f700 1 -- start start 2021-02-10T11:21:50.684 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:50.685 7f7bb600f700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7bb00a0960 0x7f7bb00a67f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:50.684 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:50.685 7f7bb600f700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7bb01924c0 0x7f7bb01928c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:50.685 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:50.685 7f7bb600f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7bb0192ee0 0x7f7bb00a29f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:50.685 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:50.685 7f7bb600f700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bb00a6e10 con 0x7f7bb0192ee0 2021-02-10T11:21:50.685 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:50.685 7f7bb600f700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bb0190630 con 0x7f7bb01924c0 2021-02-10T11:21:50.685 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:50.685 7f7bb600f700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7bb01932e0 con 0x7f7bb00a0960 2021-02-10T11:21:50.685 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:50.686 7f7baffff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7bb0192ee0 0x7f7bb00a29f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:50.685 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:50.686 7f7baffff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7bb0192ee0 0x7f7bb00a29f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:55106/0 (socket says 172.21.2.102:55106) 2021-02-10T11:21:50.685 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:50.686 7f7baffff700 1 -- 172.21.2.102:0/1478691890 learned_addr learned my addr 172.21.2.102:0/1478691890 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:21:50.685 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:50.686 7f7baffff700 1 -- 172.21.2.102:0/1478691890 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7bb00a0960 msgr2=0x7f7bb00a67f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:50.685 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:50.686 7f7baffff700 1 --2- 172.21.2.102:0/1478691890 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7bb00a0960 0x7f7bb00a67f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:21:50.685 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:50.686 7f7baffff700 1 -- 172.21.2.102:0/1478691890 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7bb01924c0 msgr2=0x7f7bb01928c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:50.685 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:50.686 7f7baffff700 1 --2- 172.21.2.102:0/1478691890 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7bb01924c0 0x7f7bb01928c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:21:50.685 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:50.686 7f7baffff700 1 -- 172.21.2.102:0/1478691890 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bb0193500 con 0x7f7bb0192ee0 2021-02-10T11:21:50.685 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:50.686 7f7baffff700 1 --2- 172.21.2.102:0/1478691890 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7bb0192ee0 0x7f7bb00a29f0 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 rx=0x7f7ba800b9b0 tx=0x7f7ba800bd50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:50.686 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:50.686 7f7b97fff700 1 -- 172.21.2.102:0/1478691890 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7ba8009450 con 0x7f7bb0192ee0 2021-02-10T11:21:50.686 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:50.686 7f7b97fff700 1 -- 172.21.2.102:0/1478691890 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ba800c2f0 con 0x7f7bb0192ee0 2021-02-10T11:21:50.686 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:50.686 7f7bb600f700 1 -- 172.21.2.102:0/1478691890 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7bb0193500 con 0x7f7bb0192ee0 2021-02-10T11:21:50.686 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:50.686 7f7bb600f700 1 -- 172.21.2.102:0/1478691890 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7bb00a3300 con 0x7f7bb0192ee0 2021-02-10T11:21:50.686 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:50.686 7f7b97fff700 1 -- 172.21.2.102:0/1478691890 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7ba800c780 con 0x7f7bb0192ee0 2021-02-10T11:21:50.686 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:50.687 7f7b97fff700 1 -- 172.21.2.102:0/1478691890 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f7ba8001270 con 0x7f7bb0192ee0 2021-02-10T11:21:50.686 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:50.687 7f7b97fff700 1 --2- 172.21.2.102:0/1478691890 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7b900186c0 0x7f7b9001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:50.686 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:50.687 7f7b97fff700 1 -- 172.21.2.102:0/1478691890 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f7ba8032b80 con 0x7f7bb0192ee0 2021-02-10T11:21:50.686 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:50.687 7f7baf7fe700 1 --2- 172.21.2.102:0/1478691890 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7b900186c0 0x7f7b9001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:50.686 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:50.687 7f7baf7fe700 1 --2- 172.21.2.102:0/1478691890 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7b900186c0 0x7f7b9001abc0 crc :-1 s=READY pgs=490 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:50.715 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:50.719 7f7bb600f700 1 -- 172.21.2.102:0/1478691890 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7bb00e52e0 con 0x7f7bb0192ee0 2021-02-10T11:21:50.718 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:50.721 7f7b97fff700 1 -- 172.21.2.102:0/1478691890 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f7ba8003030 con 0x7f7bb0192ee0 2021-02-10T11:21:50.846 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:50.850 7f7bb600f700 1 -- 172.21.2.102:0/1478691890 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7f7bb00e52e0 con 0x7f7bb0192ee0 2021-02-10T11:21:50.846 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:50.850 7f7b97fff700 1 -- 172.21.2.102:0/1478691890 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v30) v1 ==== 100+0+10 (secure 0 0 0) 0x7f7ba8001270 con 0x7f7bb0192ee0 2021-02-10T11:21:50.847 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:21:50.848 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:50.852 7f7bb600f700 1 -- 172.21.2.102:0/1478691890 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7b900186c0 msgr2=0x7f7b9001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:50.848 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:50.852 7f7bb600f700 1 --2- 172.21.2.102:0/1478691890 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7b900186c0 0x7f7b9001abc0 crc :-1 s=READY pgs=490 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:50.848 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:50.852 7f7bb600f700 1 -- 172.21.2.102:0/1478691890 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7bb0192ee0 msgr2=0x7f7bb00a29f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:50.848 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:50.852 7f7bb600f700 1 --2- 172.21.2.102:0/1478691890 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7bb0192ee0 0x7f7bb00a29f0 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 rx=0x7f7ba800b9b0 tx=0x7f7ba800bd50).stop 2021-02-10T11:21:50.849 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:50.852 7f7bb600f700 1 -- 172.21.2.102:0/1478691890 shutdown_connections 2021-02-10T11:21:50.849 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:50.852 7f7bb600f700 1 --2- 172.21.2.102:0/1478691890 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7bb0192ee0 0x7f7bb00a29f0 unknown :-1 s=CLOSED pgs=1006 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:50.849 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:50.852 7f7bb600f700 1 --2- 172.21.2.102:0/1478691890 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7b900186c0 0x7f7b9001abc0 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:50.849 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:50.852 7f7bb600f700 1 --2- 172.21.2.102:0/1478691890 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7bb01924c0 0x7f7bb01928c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:21:50.849 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:50.852 7f7bb600f700 1 --2- 172.21.2.102:0/1478691890 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7bb00a0960 0x7f7bb00a67f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:21:50.849 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:50.852 7f7bb600f700 1 -- 172.21.2.102:0/1478691890 >> 172.21.2.102:0/1478691890 conn(0x7f7bb00ac550 msgr2=0x7f7bb00be1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:50.849 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:50.853 7f7bb600f700 1 -- 172.21.2.102:0/1478691890 shutdown_connections 2021-02-10T11:21:50.849 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:50.853 7f7bb600f700 1 -- 172.21.2.102:0/1478691890 wait complete. 2021-02-10T11:21:50.857 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-02-10T11:21:50.858 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:21:51.283 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:51.271 7f20a9c1a700 1 Processor -- start 2021-02-10T11:21:51.283 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:51.271 7f20a9c1a700 1 -- start start 2021-02-10T11:21:51.283 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:51.272 7f20a9c1a700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f20a40cff00 0x7f20a41b39d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:51.283 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:51.272 7f20a9c1a700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f20a40d1bf0 0x7f20a40ef620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:51.283 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:51.272 7f20a9c1a700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20a40f0b20 con 0x7f20a40d17f0 2021-02-10T11:21:51.284 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:51.272 7f20a9c1a700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20a41b3ff0 con 0x7f20a40d1bf0 2021-02-10T11:21:51.284 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:51.272 7f20a9c1a700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20a40f20b0 con 0x7f20a40cff00 2021-02-10T11:21:51.284 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:51.272 7f20a3fff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f20a40d1bf0 0x7f20a40ef620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:51.284 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:51.272 7f20a3fff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f20a40d1bf0 0x7f20a40ef620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:55114/0 (socket says 172.21.2.102:55114) 2021-02-10T11:21:51.284 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:51.272 7f20a3fff700 1 -- 172.21.2.102:0/400905912 learned_addr learned my addr 172.21.2.102:0/400905912 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:21:51.284 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:51.272 7f20a3fff700 1 -- 172.21.2.102:0/400905912 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f20a40cff00 msgr2=0x7f20a41b39d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:21:51.284 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:51.272 7f20a3fff700 1 --2- 172.21.2.102:0/400905912 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f20a40cff00 0x7f20a41b39d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:21:51.284 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:51.272 7f20a3fff700 1 -- 172.21.2.102:0/400905912 >> v1:172.21.2.113:6789/0 conn(0x7f20a40d17f0 legacy=0x7f20a41b31e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:21:51.284 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:51.272 7f20a3fff700 1 -- 172.21.2.102:0/400905912 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20a40f22a0 con 0x7f20a40d1bf0 2021-02-10T11:21:51.284 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:51.272 7f20a3fff700 1 --2- 172.21.2.102:0/400905912 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f20a40d1bf0 0x7f20a40ef620 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 rx=0x7f209c006b80 tx=0x7f209c009b90).ready entity=mon.0 client_cookie=15f4765f713d6e47 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:51.284 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:51.272 7f20a27fc700 1 -- 172.21.2.102:0/400905912 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f209c0022e0 con 0x7f20a40d1bf0 2021-02-10T11:21:51.285 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:51.272 7f20a27fc700 1 -- 172.21.2.102:0/400905912 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f209c002620 con 0x7f20a40d1bf0 2021-02-10T11:21:51.285 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:51.272 7f20a27fc700 1 -- 172.21.2.102:0/400905912 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f209c002b00 con 0x7f20a40d1bf0 2021-02-10T11:21:51.285 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:51.272 7f20a9c1a700 1 -- 172.21.2.102:0/400905912 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f20a40d1bf0 msgr2=0x7f20a40ef620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:51.285 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:51.272 7f20a9c1a700 1 --2- 172.21.2.102:0/400905912 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f20a40d1bf0 0x7f20a40ef620 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 rx=0x7f209c006b80 tx=0x7f209c009b90).stop 2021-02-10T11:21:51.285 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:51.273 7f20a9c1a700 1 -- 172.21.2.102:0/400905912 shutdown_connections 2021-02-10T11:21:51.285 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:51.273 7f20a9c1a700 1 --2- 172.21.2.102:0/400905912 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f20a40cff00 0x7f20a41b39d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:21:51.285 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:51.273 7f20a9c1a700 1 --2- 172.21.2.102:0/400905912 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f20a40d1bf0 0x7f20a40ef620 unknown :-1 s=CLOSED pgs=1007 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:51.285 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:51.273 7f20a9c1a700 1 -- 172.21.2.102:0/400905912 >> 172.21.2.102:0/400905912 conn(0x7f20a40fde40 msgr2=0x7f20a40ce390 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:51.285 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:51.273 7f20a9c1a700 1 -- 172.21.2.102:0/400905912 shutdown_connections 2021-02-10T11:21:51.285 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:51.273 7f20a9c1a700 1 -- 172.21.2.102:0/400905912 wait complete. 2021-02-10T11:21:51.287 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:51.287 7f20a9c1a700 1 Processor -- start 2021-02-10T11:21:51.287 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:51.287 7f20a9c1a700 1 -- start start 2021-02-10T11:21:51.287 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:51.288 7f20a9c1a700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f20a419ed10 0x7f20a41917b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:51.287 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:51.288 7f20a9c1a700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f20a4191dd0 0x7f20a409aef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:51.287 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:51.288 7f20a9c1a700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f20a41921d0 0x7f20a409b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:51.287 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:51.288 7f20a9c1a700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20a4191230 con 0x7f20a4191dd0 2021-02-10T11:21:51.287 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:51.288 7f20a9c1a700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20a419f110 con 0x7f20a419ed10 2021-02-10T11:21:51.287 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:51.288 7f20a9c1a700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20a409bb30 con 0x7f20a41921d0 2021-02-10T11:21:51.287 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:51.288 7f20a2ffd700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f20a4191dd0 0x7f20a409aef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:51.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:51.288 7f20a2ffd700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f20a4191dd0 0x7f20a409aef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:55120/0 (socket says 172.21.2.102:55120) 2021-02-10T11:21:51.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:51.288 7f20a2ffd700 1 -- 172.21.2.102:0/2781942658 learned_addr learned my addr 172.21.2.102:0/2781942658 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:21:51.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:51.288 7f20a2ffd700 1 -- 172.21.2.102:0/2781942658 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f20a41921d0 msgr2=0x7f20a409b510 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:21:51.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:51.288 7f20a2ffd700 1 --2- 172.21.2.102:0/2781942658 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f20a41921d0 0x7f20a409b510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:21:51.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:51.288 7f20a2ffd700 1 -- 172.21.2.102:0/2781942658 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f20a419ed10 msgr2=0x7f20a41917b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:21:51.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:51.288 7f20a2ffd700 1 --2- 172.21.2.102:0/2781942658 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f20a419ed10 0x7f20a41917b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:21:51.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:51.288 7f20a2ffd700 1 -- 172.21.2.102:0/2781942658 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20a409bd50 con 0x7f20a4191dd0 2021-02-10T11:21:51.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:51.288 7f20a2ffd700 1 --2- 172.21.2.102:0/2781942658 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f20a4191dd0 0x7f20a409aef0 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 rx=0x7f208c009960 tx=0x7f208c00bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:51.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:51.289 7f209bfff700 1 -- 172.21.2.102:0/2781942658 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f208c000fc0 con 0x7f20a4191dd0 2021-02-10T11:21:51.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:51.289 7f209bfff700 1 -- 172.21.2.102:0/2781942658 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f208c0030c0 con 0x7f20a4191dd0 2021-02-10T11:21:51.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:51.289 7f20a9c1a700 1 -- 172.21.2.102:0/2781942658 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20a409bd50 con 0x7f20a4191dd0 2021-02-10T11:21:51.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:51.289 7f20a9c1a700 1 -- 172.21.2.102:0/2781942658 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20a409c2e0 con 0x7f20a4191dd0 2021-02-10T11:21:51.289 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:51.289 7f209bfff700 1 -- 172.21.2.102:0/2781942658 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f208c001500 con 0x7f20a4191dd0 2021-02-10T11:21:51.289 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:51.289 7f209bfff700 1 -- 172.21.2.102:0/2781942658 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f208c001310 con 0x7f20a4191dd0 2021-02-10T11:21:51.289 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:51.289 7f209bfff700 1 --2- 172.21.2.102:0/2781942658 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f20880186c0 0x7f208801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:51.289 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:51.289 7f209bfff700 1 -- 172.21.2.102:0/2781942658 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f208c0329b0 con 0x7f20a4191dd0 2021-02-10T11:21:51.289 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:51.290 7f20a37fe700 1 --2- 172.21.2.102:0/2781942658 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f20880186c0 0x7f208801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:51.289 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:51.290 7f20a37fe700 1 --2- 172.21.2.102:0/2781942658 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f20880186c0 0x7f208801abc0 crc :-1 s=READY pgs=491 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:51.318 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:51.322 7f20a9c1a700 1 -- 172.21.2.102:0/2781942658 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20a41036a0 con 0x7f20a4191dd0 2021-02-10T11:21:51.320 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:51.324 7f209bfff700 1 -- 172.21.2.102:0/2781942658 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f208c000cc0 con 0x7f20a4191dd0 2021-02-10T11:21:51.450 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:51.454 7f20a9c1a700 1 -- 172.21.2.102:0/2781942658 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f20a40f0b20 con 0x7f20a4191dd0 2021-02-10T11:21:51.518 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:51.454 7f209bfff700 1 -- 172.21.2.102:0/2781942658 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+2099 (secure 0 0 0) 0x7f208c001310 con 0x7f20a4191dd0 2021-02-10T11:21:51.519 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:21:51.520 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":15,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4845,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.102:6835/3747080008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6834","nonce":3747080008},{"type":"v1","addr":"172.21.2.102:6835","nonce":3747080008}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":10}],"filesystems":[{"mdsmap":{"epoch":14,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:05:12.567145","modified":"2021-02-10 11:05:14.922542","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4510},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4510":{"gid":4510,"name":"b","rank":0,"incarnation":11,"state":"up:active","state_seq":4,"addr":"172.21.2.113:6833/3898434901","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6832","nonce":3898434901},{"type":"v1","addr":"172.21.2.113:6833","nonce":3898434901}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":2}]} 2021-02-10T11:21:51.520 INFO:teuthology.orchestra.run.gibba002.stderr:dumped fsmap epoch 15 2021-02-10T11:21:51.520 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:51.456 7f20a9c1a700 1 -- 172.21.2.102:0/2781942658 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f20880186c0 msgr2=0x7f208801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:51.520 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:51.456 7f20a9c1a700 1 --2- 172.21.2.102:0/2781942658 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f20880186c0 0x7f208801abc0 crc :-1 s=READY pgs=491 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:51.520 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:51.456 7f20a9c1a700 1 -- 172.21.2.102:0/2781942658 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f20a4191dd0 msgr2=0x7f20a409aef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:51.520 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:51.456 7f20a9c1a700 1 --2- 172.21.2.102:0/2781942658 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f20a4191dd0 0x7f20a409aef0 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 rx=0x7f208c009960 tx=0x7f208c00bcf0).stop 2021-02-10T11:21:51.520 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:51.456 7f20a9c1a700 1 -- 172.21.2.102:0/2781942658 shutdown_connections 2021-02-10T11:21:51.520 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:51.456 7f20a9c1a700 1 --2- 172.21.2.102:0/2781942658 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f20a41921d0 0x7f20a409b510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:21:51.520 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:51.456 7f20a9c1a700 1 --2- 172.21.2.102:0/2781942658 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f20a4191dd0 0x7f20a409aef0 unknown :-1 s=CLOSED pgs=1008 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:51.520 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:51.456 7f20a9c1a700 1 --2- 172.21.2.102:0/2781942658 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f20880186c0 0x7f208801abc0 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:51.521 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:51.456 7f20a9c1a700 1 --2- 172.21.2.102:0/2781942658 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f20a419ed10 0x7f20a41917b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:21:51.521 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:51.456 7f20a9c1a700 1 -- 172.21.2.102:0/2781942658 >> 172.21.2.102:0/2781942658 conn(0x7f20a41a3780 msgr2=0x7f20a4092c70 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:51.521 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:51.457 7f20a9c1a700 1 -- 172.21.2.102:0/2781942658 shutdown_connections 2021-02-10T11:21:51.521 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:21:51.457 7f20a9c1a700 1 -- 172.21.2.102:0/2781942658 wait complete. 2021-02-10T11:21:51.722 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 14, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:05:12.567145', 'modified': '2021-02-10 11:05:14.922542', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4510}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4510': {'gid': 4510, 'name': 'b', 'rank': 0, 'incarnation': 11, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.2.113:6833/3898434901', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.113:6832', 'nonce': 3898434901}, {'type': 'v1', 'addr': '172.21.2.113:6833', 'nonce': 3898434901}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [5], 'metadata_pool': 4, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:21:51.723 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:21:51.723 DEBUG:teuthology.orchestra.run.gibba013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:21:52.125 INFO:teuthology.orchestra.run.gibba013.stdout:{ 2021-02-10T11:21:52.125 INFO:teuthology.orchestra.run.gibba013.stdout: "cluster_fsid": "68db30ea-dbce-4aef-a6df-f01c6fea0513", 2021-02-10T11:21:52.125 INFO:teuthology.orchestra.run.gibba013.stdout: "whoami": 0, 2021-02-10T11:21:52.125 INFO:teuthology.orchestra.run.gibba013.stdout: "id": 4510, 2021-02-10T11:21:52.125 INFO:teuthology.orchestra.run.gibba013.stdout: "want_state": "up:active", 2021-02-10T11:21:52.125 INFO:teuthology.orchestra.run.gibba013.stdout: "state": "up:active", 2021-02-10T11:21:52.125 INFO:teuthology.orchestra.run.gibba013.stdout: "rank_uptime": 999.55785326299997, 2021-02-10T11:21:52.125 INFO:teuthology.orchestra.run.gibba013.stdout: "mdsmap_epoch": 14, 2021-02-10T11:21:52.125 INFO:teuthology.orchestra.run.gibba013.stdout: "osdmap_epoch": 30, 2021-02-10T11:21:52.126 INFO:teuthology.orchestra.run.gibba013.stdout: "osdmap_epoch_barrier": 30, 2021-02-10T11:21:52.126 INFO:teuthology.orchestra.run.gibba013.stdout: "uptime": 1008.841636969 2021-02-10T11:21:52.126 INFO:teuthology.orchestra.run.gibba013.stdout:} 2021-02-10T11:21:52.129 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "68db30ea-dbce-4aef-a6df-f01c6fea0513", "whoami": 0, "id": 4510, "want_state": "up:active", "state": "up:active", "rank_uptime": 999.55785326299997, "mdsmap_epoch": 14, "osdmap_epoch": 30, "osdmap_epoch_barrier": 30, "uptime": 1008.841636969 } 2021-02-10T11:21:52.129 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2021-02-10T11:21:52.129 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2021-02-10T11:21:52.129 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@gibba002.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2021-02-10T11:21:52.130 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:21:52.130 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec mkdir -p /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:21:52.466 DEBUG:teuthology.orchestra.run.gibba002:> sudo modprobe fuse 2021-02-10T11:21:52.810 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:21:52.810 DEBUG:teuthology.orchestra.run.gibba002:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:21:53.156 INFO:teuthology.orchestra.run.gibba002.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:21:53.156 INFO:teuthology.orchestra.run.gibba002.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:21:53.156 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:21:53.157 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:21:53.157 DEBUG:teuthology.orchestra.run.gibba002:> ls /sys/fs/fuse/connections 2021-02-10T11:21:53.490 INFO:teuthology.orchestra.run.gibba002.stdout:44 2021-02-10T11:21:53.490 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [44] 2021-02-10T11:21:53.491 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:21:53.493 DEBUG:teuthology.orchestra.run.gibba002:> sudo modprobe fuse 2021-02-10T11:21:53.835 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:21:53.836 DEBUG:teuthology.orchestra.run.gibba002:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:21:53.903 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba002.stderr:ceph-fuse[142615]: starting ceph client 2021-02-10T11:21:53.903 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba002.stderr:2021-02-10 11:21:53.907 7f5565f3cf80 -1 init, newargv = 0x5585802b0900 newargc=9 2021-02-10T11:21:53.908 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba002.stderr:ceph-fuse[142615]: starting fuse 2021-02-10T11:21:54.178 INFO:teuthology.orchestra.run.gibba002.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:21:54.178 INFO:teuthology.orchestra.run.gibba002.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:21:54.179 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:21:54.179 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:21:54.179 DEBUG:teuthology.orchestra.run.gibba002:> ls /sys/fs/fuse/connections 2021-02-10T11:21:54.515 INFO:teuthology.orchestra.run.gibba002.stdout:43 2021-02-10T11:21:54.515 INFO:teuthology.orchestra.run.gibba002.stdout:44 2021-02-10T11:21:54.516 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43, 44] 2021-02-10T11:21:54.516 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:21:54.516 DEBUG:teuthology.orchestra.run.gibba002:> sudo python3 -c ' 2021-02-10T11:21:54.516 DEBUG:teuthology.orchestra.run.gibba002:> import glob 2021-02-10T11:21:54.516 DEBUG:teuthology.orchestra.run.gibba002:> import re 2021-02-10T11:21:54.516 DEBUG:teuthology.orchestra.run.gibba002:> import os 2021-02-10T11:21:54.516 DEBUG:teuthology.orchestra.run.gibba002:> import subprocess 2021-02-10T11:21:54.516 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:21:54.516 DEBUG:teuthology.orchestra.run.gibba002:> def find_socket(client_name): 2021-02-10T11:21:54.516 DEBUG:teuthology.orchestra.run.gibba002:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2021-02-10T11:21:54.517 DEBUG:teuthology.orchestra.run.gibba002:> files = glob.glob(asok_path) 2021-02-10T11:21:54.517 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:21:54.517 DEBUG:teuthology.orchestra.run.gibba002:> # Given a non-glob path, it better be there 2021-02-10T11:21:54.517 DEBUG:teuthology.orchestra.run.gibba002:> if "*" not in asok_path: 2021-02-10T11:21:54.517 DEBUG:teuthology.orchestra.run.gibba002:> assert(len(files) == 1) 2021-02-10T11:21:54.517 DEBUG:teuthology.orchestra.run.gibba002:> return files[0] 2021-02-10T11:21:54.517 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:21:54.517 DEBUG:teuthology.orchestra.run.gibba002:> for f in files: 2021-02-10T11:21:54.517 DEBUG:teuthology.orchestra.run.gibba002:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2021-02-10T11:21:54.517 DEBUG:teuthology.orchestra.run.gibba002:> if os.path.exists("/proc/{0}".format(pid)): 2021-02-10T11:21:54.517 DEBUG:teuthology.orchestra.run.gibba002:> return f 2021-02-10T11:21:54.517 DEBUG:teuthology.orchestra.run.gibba002:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2021-02-10T11:21:54.517 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:21:54.518 DEBUG:teuthology.orchestra.run.gibba002:> print(find_socket("client.0")) 2021-02-10T11:21:54.518 DEBUG:teuthology.orchestra.run.gibba002:> ' 2021-02-10T11:21:54.878 INFO:teuthology.orchestra.run.gibba002.stdout:/var/run/ceph/ceph-client.0.142615.asok 2021-02-10T11:21:54.881 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.142615.asok 2021-02-10T11:21:54.882 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:21:54.882 DEBUG:teuthology.orchestra.run.gibba002:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.142615.asok status 2021-02-10T11:21:55.273 INFO:teuthology.orchestra.run.gibba002.stdout:{ 2021-02-10T11:22:01.789 INFO:teuthology.orchestra.run.gibba002.stdout: "metadata": { 2021-02-10T11:22:01.790 INFO:teuthology.orchestra.run.gibba002.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:22:01.790 INFO:teuthology.orchestra.run.gibba002.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:22:01.790 INFO:teuthology.orchestra.run.gibba002.stdout: "entity_id": "0", 2021-02-10T11:22:01.790 INFO:teuthology.orchestra.run.gibba002.stdout: "hostname": "gibba002", 2021-02-10T11:22:01.790 INFO:teuthology.orchestra.run.gibba002.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2021-02-10T11:22:01.790 INFO:teuthology.orchestra.run.gibba002.stdout: "pid": "142615", 2021-02-10T11:22:01.790 INFO:teuthology.orchestra.run.gibba002.stdout: "root": "/" 2021-02-10T11:22:01.790 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:22:01.790 INFO:teuthology.orchestra.run.gibba002.stdout: "dentry_count": 0, 2021-02-10T11:22:01.790 INFO:teuthology.orchestra.run.gibba002.stdout: "dentry_pinned_count": 0, 2021-02-10T11:22:01.791 INFO:teuthology.orchestra.run.gibba002.stdout: "id": 7104, 2021-02-10T11:22:01.791 INFO:teuthology.orchestra.run.gibba002.stdout: "inst": { 2021-02-10T11:22:01.791 INFO:teuthology.orchestra.run.gibba002.stdout: "name": { 2021-02-10T11:22:01.791 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "client", 2021-02-10T11:22:01.791 INFO:teuthology.orchestra.run.gibba002.stdout: "num": 7104 2021-02-10T11:22:01.791 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:22:01.791 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": { 2021-02-10T11:22:01.791 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:22:01.791 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:0", 2021-02-10T11:22:01.791 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 359949303 2021-02-10T11:22:01.791 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:22:01.791 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:22:01.792 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": { 2021-02-10T11:22:01.792 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:22:01.792 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:0", 2021-02-10T11:22:01.792 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 359949303 2021-02-10T11:22:01.792 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:22:01.792 INFO:teuthology.orchestra.run.gibba002.stdout: "inst_str": "client.7104 172.21.2.102:0/359949303", 2021-02-10T11:22:01.792 INFO:teuthology.orchestra.run.gibba002.stdout: "addr_str": "172.21.2.102:0/359949303", 2021-02-10T11:22:01.792 INFO:teuthology.orchestra.run.gibba002.stdout: "inode_count": 1, 2021-02-10T11:22:01.792 INFO:teuthology.orchestra.run.gibba002.stdout: "mds_epoch": 14, 2021-02-10T11:22:01.792 INFO:teuthology.orchestra.run.gibba002.stdout: "osd_epoch": 30, 2021-02-10T11:22:01.792 INFO:teuthology.orchestra.run.gibba002.stdout: "osd_epoch_barrier": 0, 2021-02-10T11:22:01.793 INFO:teuthology.orchestra.run.gibba002.stdout: "blacklisted": false 2021-02-10T11:22:01.793 INFO:teuthology.orchestra.run.gibba002.stdout:} 2021-02-10T11:22:01.794 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo getfattr --only-values -n ceph.dir.entries /home/ubuntu/cephtest/mnt.0/./volumes/_deleting 2021-02-10T11:22:01.941 DEBUG:teuthology.orchestra.run.gibba002:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:22:01.944 DEBUG:teuthology.orchestra.run.gibba013:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:22:02.144 INFO:teuthology.orchestra.run.gibba002.stderr:getfattr: Removing leading '/' from absolute path names 2021-02-10T11:22:02.145 INFO:teuthology.orchestra.run.gibba002.stdout:0 2021-02-10T11:22:02.145 DEBUG:tasks.cephfs.mount:Directory ./volumes/_deleting seen empty from 0 after 0s 2021-02-10T11:22:02.145 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2021-02-10T11:22:02.146 DEBUG:teuthology.orchestra.run.gibba002:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:22:02.492 DEBUG:teuthology.orchestra.run.gibba013:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:22:02.836 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2021-02-10T11:22:02.837 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:22:02.837 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:22:02.837 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:22:03.176 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:22:03.176 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:22:03.176 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@gibba002.front.sepia.ceph.com... 2021-02-10T11:22:03.176 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-02-10T11:22:03.177 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:22:03.531 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba002.stderr:ceph-fuse[142615]: fuse finished with error 0 and tester_r 0 2021-02-10T11:22:03.532 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:22:03.532 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:22:03.533 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:22:03.867 INFO:teuthology.orchestra.run.gibba002.stdout:ext2/ext3 2021-02-10T11:22:03.868 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2021-02-10T11:22:03.868 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:22:03.868 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:22:04.204 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:22:04.204 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:22:04.204 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:22:04.539 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:22:04.539 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2021-02-10T11:22:04.540 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@gibba002.front.sepia.ceph.com... 2021-02-10T11:22:04.540 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-02-10T11:22:04.540 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:22:04.895 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba002.stderr:ceph-fuse[45554]: fuse finished with error 0 and tester_r 0 2021-02-10T11:22:04.896 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:22:04.896 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:22:04.896 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:22:05.229 INFO:teuthology.orchestra.run.gibba002.stdout:ext2/ext3 2021-02-10T11:22:05.229 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2021-02-10T11:22:05.230 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:22:05.230 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:22:05.564 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:22:05.565 DEBUG:teuthology.orchestra.run.gibba013:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:22:05.565 DEBUG:teuthology.orchestra.run.gibba013:> ' -- /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:22:05.900 INFO:teuthology.orchestra.run.gibba013.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.2’: No such file or directory 2021-02-10T11:22:05.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:22:05.901 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2021-02-10T11:22:05.901 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:22:05.901 DEBUG:teuthology.orchestra.run.gibba013:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:22:06.236 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:22:06.236 DEBUG:teuthology.orchestra.run.gibba013:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:22:06.236 DEBUG:teuthology.orchestra.run.gibba013:> ' -- /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:22:06.572 INFO:teuthology.orchestra.run.gibba013.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.3’: No such file or directory 2021-02-10T11:22:06.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:22:06.572 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.3 2021-02-10T11:22:06.572 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:22:06.573 DEBUG:teuthology.orchestra.run.gibba013:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:22:06.906 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config rm mon mon_allow_pool_delete 2021-02-10T11:22:07.336 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:07.323 7f43584c5700 1 Processor -- start 2021-02-10T11:22:07.336 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:07.323 7f43584c5700 1 -- start start 2021-02-10T11:22:07.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:07.324 7f43584c5700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f43500d17f0 0x7f43501b31e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:07.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:07.324 7f43584c5700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43500f0b20 con 0x7f43500cff00 2021-02-10T11:22:07.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:07.324 7f43584c5700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43501b3ff0 con 0x7f43500d1bf0 2021-02-10T11:22:07.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:07.324 7f43584c5700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43500f20b0 con 0x7f43500d17f0 2021-02-10T11:22:07.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:07.324 7f4356ae2700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f43500cff00 0x7f43500ef620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:47360/0 (socket says 172.21.2.102:47360) 2021-02-10T11:22:07.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:07.324 7f4356ae2700 1 -- 172.21.2.102:0/3185525900 learned_addr learned my addr 172.21.2.102:0/3185525900 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:22:07.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:07.324 7f43562e1700 1 --2- 172.21.2.102:0/3185525900 >> v2:172.21.2.113:3300/0 conn(0x7f43500d17f0 0x7f43501b31e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:07.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:07.325 7f43552df700 1 -- 172.21.2.102:0/3185525900 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3303718925 0 0) 0x7f4348001500 con 0x7f43500cff00 2021-02-10T11:22:07.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:07.325 7f43552df700 1 -- 172.21.2.102:0/3185525900 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f433c002fe0 con 0x7f43500cff00 2021-02-10T11:22:07.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:07.325 7f43562e1700 1 -- 172.21.2.102:0/3185525900 >> v1:172.21.2.113:6789/0 conn(0x7f43500d1bf0 legacy=0x7f43501b3800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:07.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:07.325 7f43562e1700 1 -- 172.21.2.102:0/3185525900 >> v1:172.21.2.102:6789/0 conn(0x7f43500cff00 legacy=0x7f43500ef620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:07.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:07.325 7f43562e1700 1 -- 172.21.2.102:0/3185525900 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43500f22d0 con 0x7f43500d17f0 2021-02-10T11:22:07.338 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:07.325 7f43562e1700 1 --2- 172.21.2.102:0/3185525900 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f43500d17f0 0x7f43501b31e0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 rx=0x7f4344006650 tx=0x7f4344009a80).ready entity=mon.1 client_cookie=2091f7d09c01f87a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:07.338 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:07.325 7f43552df700 1 -- 172.21.2.102:0/3185525900 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f434400c0c0 con 0x7f43500d17f0 2021-02-10T11:22:07.338 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:07.325 7f43552df700 1 -- 172.21.2.102:0/3185525900 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f434400c340 con 0x7f43500d17f0 2021-02-10T11:22:07.338 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:07.325 7f43552df700 1 -- 172.21.2.102:0/3185525900 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f434400c860 con 0x7f43500d17f0 2021-02-10T11:22:07.338 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:07.325 7f43584c5700 1 -- 172.21.2.102:0/3185525900 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f43500d17f0 msgr2=0x7f43501b31e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:07.338 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:07.325 7f43584c5700 1 --2- 172.21.2.102:0/3185525900 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f43500d17f0 0x7f43501b31e0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 rx=0x7f4344006650 tx=0x7f4344009a80).stop 2021-02-10T11:22:07.338 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:07.326 7f43584c5700 1 -- 172.21.2.102:0/3185525900 shutdown_connections 2021-02-10T11:22:07.338 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:07.326 7f43584c5700 1 --2- 172.21.2.102:0/3185525900 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f43500d17f0 0x7f43501b31e0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:07.338 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:07.326 7f43584c5700 1 -- 172.21.2.102:0/3185525900 >> 172.21.2.102:0/3185525900 conn(0x7f43500fde40 msgr2=0x7f43500ce390 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:07.338 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:07.326 7f43584c5700 1 -- 172.21.2.102:0/3185525900 shutdown_connections 2021-02-10T11:22:07.338 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:07.326 7f43584c5700 1 -- 172.21.2.102:0/3185525900 wait complete. 2021-02-10T11:22:07.339 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:07.340 7f43584c5700 1 Processor -- start 2021-02-10T11:22:07.339 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:07.340 7f43584c5700 1 -- start start 2021-02-10T11:22:07.340 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:07.341 7f43584c5700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f43501960f0 0x7f435019b6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:07.341 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:07.341 7f43584c5700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f435018ecc0 0x7f4350195740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:07.341 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:07.341 7f43584c5700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f435019c280 0x7f435019c680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:07.341 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:07.341 7f43584c5700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f435018f0c0 con 0x7f435019c280 2021-02-10T11:22:07.341 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:07.341 7f43584c5700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4350195d60 con 0x7f435018ecc0 2021-02-10T11:22:07.341 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:07.341 7f43584c5700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f435019cca0 con 0x7f43501960f0 2021-02-10T11:22:07.341 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:07.341 7f4356ae2700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f435019c280 0x7f435019c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:07.341 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:07.341 7f4356ae2700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f435019c280 0x7f435019c680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:55142/0 (socket says 172.21.2.102:55142) 2021-02-10T11:22:07.341 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:07.341 7f4356ae2700 1 -- 172.21.2.102:0/3261839874 learned_addr learned my addr 172.21.2.102:0/3261839874 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:22:07.341 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:07.341 7f4355ae0700 1 --2- 172.21.2.102:0/3261839874 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f435018ecc0 0x7f4350195740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:07.341 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:07.341 7f43562e1700 1 --2- 172.21.2.102:0/3261839874 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f43501960f0 0x7f435019b6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:07.341 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:07.341 7f4356ae2700 1 -- 172.21.2.102:0/3261839874 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f43501960f0 msgr2=0x7f435019b6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:07.341 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:07.341 7f4356ae2700 1 --2- 172.21.2.102:0/3261839874 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f43501960f0 0x7f435019b6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:07.341 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:07.341 7f4356ae2700 1 -- 172.21.2.102:0/3261839874 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f435018ecc0 msgr2=0x7f4350195740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:07.342 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:07.341 7f4356ae2700 1 --2- 172.21.2.102:0/3261839874 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f435018ecc0 0x7f4350195740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:07.342 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:07.341 7f4356ae2700 1 -- 172.21.2.102:0/3261839874 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f435019cec0 con 0x7f435019c280 2021-02-10T11:22:07.342 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:07.341 7f4356ae2700 1 --2- 172.21.2.102:0/3261839874 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f435019c280 0x7f435019c680 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 rx=0x7f4348009960 tx=0x7f434800bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:07.342 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:07.342 7f4342ffd700 1 -- 172.21.2.102:0/3261839874 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4348006030 con 0x7f435019c280 2021-02-10T11:22:07.342 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:07.342 7f4342ffd700 1 -- 172.21.2.102:0/3261839874 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4348006330 con 0x7f435019c280 2021-02-10T11:22:07.342 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:07.342 7f4342ffd700 1 -- 172.21.2.102:0/3261839874 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4348004720 con 0x7f435019c280 2021-02-10T11:22:07.342 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:07.342 7f43584c5700 1 -- 172.21.2.102:0/3261839874 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f435019cec0 con 0x7f435019c280 2021-02-10T11:22:07.342 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:07.342 7f43584c5700 1 -- 172.21.2.102:0/3261839874 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f435019d450 con 0x7f435019c280 2021-02-10T11:22:07.342 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:07.342 7f4342ffd700 1 -- 172.21.2.102:0/3261839874 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4348006510 con 0x7f435019c280 2021-02-10T11:22:07.342 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:07.342 7f4342ffd700 1 --2- 172.21.2.102:0/3261839874 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f43380186c0 0x7f433801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:07.342 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:07.342 7f4342ffd700 1 -- 172.21.2.102:0/3261839874 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f4348032910 con 0x7f435019c280 2021-02-10T11:22:07.342 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:07.342 7f43562e1700 1 --2- 172.21.2.102:0/3261839874 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f43380186c0 0x7f433801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:07.342 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:07.343 7f43562e1700 1 --2- 172.21.2.102:0/3261839874 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f43380186c0 0x7f433801abc0 crc :-1 s=READY pgs=492 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:07.370 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:07.374 7f43584c5700 1 -- 172.21.2.102:0/3261839874 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f435019bd50 con 0x7f435019c280 2021-02-10T11:22:07.373 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:07.376 7f4342ffd700 1 -- 172.21.2.102:0/3261839874 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f4348000f40 con 0x7f435019c280 2021-02-10T11:22:07.521 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:07.525 7f43584c5700 1 -- 172.21.2.102:0/3261839874 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f435019bd50 con 0x7f435019c280 2021-02-10T11:22:07.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:07.529 7f4342ffd700 1 -- 172.21.2.102:0/3261839874 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v3) v1 ==== 104+0+0 (secure 0 0 0) 0x7f4348006510 con 0x7f435019c280 2021-02-10T11:22:07.530 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:07.534 7f43584c5700 1 -- 172.21.2.102:0/3261839874 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f43380186c0 msgr2=0x7f433801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:07.530 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:07.534 7f43584c5700 1 --2- 172.21.2.102:0/3261839874 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f43380186c0 0x7f433801abc0 crc :-1 s=READY pgs=492 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:07.531 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:07.534 7f43584c5700 1 -- 172.21.2.102:0/3261839874 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f435019c280 msgr2=0x7f435019c680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:07.531 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:07.534 7f43584c5700 1 --2- 172.21.2.102:0/3261839874 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f435019c280 0x7f435019c680 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 rx=0x7f4348009960 tx=0x7f434800bcf0).stop 2021-02-10T11:22:07.531 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:07.534 7f43584c5700 1 -- 172.21.2.102:0/3261839874 shutdown_connections 2021-02-10T11:22:07.531 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:07.534 7f43584c5700 1 --2- 172.21.2.102:0/3261839874 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f435019c280 0x7f435019c680 unknown :-1 s=CLOSED pgs=1011 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:07.531 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:07.534 7f43584c5700 1 --2- 172.21.2.102:0/3261839874 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f43380186c0 0x7f433801abc0 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:07.531 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:07.534 7f43584c5700 1 --2- 172.21.2.102:0/3261839874 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f435018ecc0 0x7f4350195740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:07.531 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:07.534 7f43584c5700 1 --2- 172.21.2.102:0/3261839874 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f43501960f0 0x7f435019b6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:07.531 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:07.534 7f43584c5700 1 -- 172.21.2.102:0/3261839874 >> 172.21.2.102:0/3261839874 conn(0x7f435019f7e0 msgr2=0x7f435018c240 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:07.531 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:07.534 7f43584c5700 1 -- 172.21.2.102:0/3261839874 shutdown_connections 2021-02-10T11:22:07.531 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:07.534 7f43584c5700 1 -- 172.21.2.102:0/3261839874 wait complete. 2021-02-10T11:22:07.539 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestVolumes.test_async_subvolume_rm' 2021-02-10T11:22:07.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:07.951 7f6fba26a700 1 Processor -- start 2021-02-10T11:22:07.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:07.951 7f6fba26a700 1 -- start start 2021-02-10T11:22:07.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:07.952 7f6fba26a700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f6fb40e5070 0x7f6fb41b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:07.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:07.952 7f6fba26a700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6fb41b3e40 0x7f6fb407d690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:07.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:07.952 7f6fba26a700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fb407ed80 con 0x7f6fb40e5470 2021-02-10T11:22:07.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:07.952 7f6fba26a700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6fb4080310 con 0x7f6fb40e5070 2021-02-10T11:22:07.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:07.952 7f6fba26a700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fb4080500 con 0x7f6fb41b3e40 2021-02-10T11:22:07.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:07.952 7f6fb8887700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f6fb40e5470 0x7f6fb407dcb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:47370/0 (socket says 172.21.2.102:47370) 2021-02-10T11:22:07.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:07.952 7f6fb8887700 1 -- 172.21.2.102:0/3643447668 learned_addr learned my addr 172.21.2.102:0/3643447668 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:22:07.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:07.952 7f6fb3fff700 1 --2- 172.21.2.102:0/3643447668 >> v2:172.21.2.113:3300/0 conn(0x7f6fb40e5070 0x7f6fb41b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:07.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:07.952 7f6fb37fe700 1 --2- 172.21.2.102:0/3643447668 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6fb41b3e40 0x7f6fb407d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:07.966 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:07.953 7f6fb2ffd700 1 -- 172.21.2.102:0/3643447668 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1462418406 0 0) 0x7f6fac000f90 con 0x7f6fb40e5470 2021-02-10T11:22:07.966 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:07.953 7f6fb2ffd700 1 -- 172.21.2.102:0/3643447668 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f9c002fe0 con 0x7f6fb40e5470 2021-02-10T11:22:07.966 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:07.953 7f6fb3fff700 1 -- 172.21.2.102:0/3643447668 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6fb41b3e40 msgr2=0x7f6fb407d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:07.966 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:07.953 7f6fb3fff700 1 --2- 172.21.2.102:0/3643447668 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6fb41b3e40 0x7f6fb407d690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:07.966 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:07.953 7f6fb3fff700 1 -- 172.21.2.102:0/3643447668 >> v1:172.21.2.102:6789/0 conn(0x7f6fb40e5470 legacy=0x7f6fb407dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:07.966 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:07.953 7f6fb3fff700 1 -- 172.21.2.102:0/3643447668 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fb40806f0 con 0x7f6fb40e5070 2021-02-10T11:22:07.966 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:07.953 7f6fb3fff700 1 --2- 172.21.2.102:0/3643447668 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6fb40e5070 0x7f6fb41b3820 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 rx=0x7f6fa4006650 tx=0x7f6fa4009a80).ready entity=mon.1 client_cookie=deadbab8c4c81961 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:07.966 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:07.953 7f6fb2ffd700 1 -- 172.21.2.102:0/3643447668 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6fa40009b0 con 0x7f6fb40e5070 2021-02-10T11:22:07.966 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:07.953 7f6fb2ffd700 1 -- 172.21.2.102:0/3643447668 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fa400c0c0 con 0x7f6fb40e5070 2021-02-10T11:22:07.966 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:07.953 7f6fb2ffd700 1 -- 172.21.2.102:0/3643447668 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6fa400c5e0 con 0x7f6fb40e5070 2021-02-10T11:22:07.966 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:07.953 7f6fba26a700 1 -- 172.21.2.102:0/3643447668 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6fb40e5070 msgr2=0x7f6fb41b3820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:07.966 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:07.953 7f6fba26a700 1 --2- 172.21.2.102:0/3643447668 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6fb40e5070 0x7f6fb41b3820 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 rx=0x7f6fa4006650 tx=0x7f6fa4009a80).stop 2021-02-10T11:22:07.967 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:07.954 7f6fba26a700 1 -- 172.21.2.102:0/3643447668 shutdown_connections 2021-02-10T11:22:07.967 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:07.954 7f6fba26a700 1 --2- 172.21.2.102:0/3643447668 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6fb41b3e40 0x7f6fb407d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:07.967 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:07.954 7f6fba26a700 1 --2- 172.21.2.102:0/3643447668 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6fb40e5070 0x7f6fb41b3820 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:07.967 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:07.954 7f6fba26a700 1 -- 172.21.2.102:0/3643447668 >> 172.21.2.102:0/3643447668 conn(0x7f6fb40ce690 msgr2=0x7f6fb40e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:07.967 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:07.954 7f6fba26a700 1 -- 172.21.2.102:0/3643447668 shutdown_connections 2021-02-10T11:22:07.967 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:07.954 7f6fba26a700 1 -- 172.21.2.102:0/3643447668 wait complete. 2021-02-10T11:22:07.968 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:07.969 7f6fba26a700 1 Processor -- start 2021-02-10T11:22:07.969 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:07.969 7f6fba26a700 1 -- start start 2021-02-10T11:22:07.969 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:07.969 7f6fba26a700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6fb4191bb0 0x7f6fb40a2380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:07.969 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:07.969 7f6fba26a700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6fb40a29a0 0x7f6fb40a2da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:07.969 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:07.969 7f6fba26a700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6fb40a33c0 0x7f6fb40a03d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:07.969 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:07.969 7f6fba26a700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fb40adf90 con 0x7f6fb40a33c0 2021-02-10T11:22:07.969 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:07.969 7f6fba26a700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fb40a1e00 con 0x7f6fb4191bb0 2021-02-10T11:22:07.969 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:07.969 7f6fba26a700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fb40a09f0 con 0x7f6fb40a29a0 2021-02-10T11:22:07.969 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:07.969 7f6fb8887700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6fb40a33c0 0x7f6fb40a03d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:07.969 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:07.970 7f6fb8887700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6fb40a33c0 0x7f6fb40a03d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:55150/0 (socket says 172.21.2.102:55150) 2021-02-10T11:22:07.969 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:07.970 7f6fb8887700 1 -- 172.21.2.102:0/2222731217 learned_addr learned my addr 172.21.2.102:0/2222731217 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:22:07.969 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:07.970 7f6fb3fff700 1 --2- 172.21.2.102:0/2222731217 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6fb4191bb0 0x7f6fb40a2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:07.969 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:07.970 7f6fb37fe700 1 --2- 172.21.2.102:0/2222731217 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6fb40a29a0 0x7f6fb40a2da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:07.970 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:07.970 7f6fb8887700 1 -- 172.21.2.102:0/2222731217 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6fb40a29a0 msgr2=0x7f6fb40a2da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:07.970 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:07.970 7f6fb8887700 1 --2- 172.21.2.102:0/2222731217 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6fb40a29a0 0x7f6fb40a2da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:07.970 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:07.970 7f6fb8887700 1 -- 172.21.2.102:0/2222731217 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6fb4191bb0 msgr2=0x7f6fb40a2380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:07.970 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:07.970 7f6fb8887700 1 --2- 172.21.2.102:0/2222731217 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6fb4191bb0 0x7f6fb40a2380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:07.970 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:07.970 7f6fb8887700 1 -- 172.21.2.102:0/2222731217 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fb40a0bd0 con 0x7f6fb40a33c0 2021-02-10T11:22:07.970 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:07.970 7f6fb3fff700 1 --2- 172.21.2.102:0/2222731217 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6fb4191bb0 0x7f6fb40a2380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:22:07.970 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:07.970 7f6fb8887700 1 --2- 172.21.2.102:0/2222731217 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6fb40a33c0 0x7f6fb40a03d0 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 rx=0x7f6fac009980 tx=0x7f6fac00bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:07.970 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:07.970 7f6fb0ff9700 1 -- 172.21.2.102:0/2222731217 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6fac00be10 con 0x7f6fb40a33c0 2021-02-10T11:22:07.970 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:07.970 7f6fb0ff9700 1 -- 172.21.2.102:0/2222731217 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fac006080 con 0x7f6fb40a33c0 2021-02-10T11:22:07.970 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:07.970 7f6fb0ff9700 1 -- 172.21.2.102:0/2222731217 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6fac0064d0 con 0x7f6fb40a33c0 2021-02-10T11:22:07.970 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:07.970 7f6fba26a700 1 -- 172.21.2.102:0/2222731217 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fb40a0bd0 con 0x7f6fb40a33c0 2021-02-10T11:22:07.970 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:07.970 7f6fba26a700 1 -- 172.21.2.102:0/2222731217 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fb40a1160 con 0x7f6fb40a33c0 2021-02-10T11:22:07.970 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:07.971 7f6fb0ff9700 1 -- 172.21.2.102:0/2222731217 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f6fac006260 con 0x7f6fb40a33c0 2021-02-10T11:22:07.971 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:07.971 7f6fb0ff9700 1 --2- 172.21.2.102:0/2222731217 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6f940186c0 0x7f6f9401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:07.971 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:07.971 7f6fb0ff9700 1 -- 172.21.2.102:0/2222731217 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f6fac032770 con 0x7f6fb40a33c0 2021-02-10T11:22:07.971 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:07.971 7f6fb3fff700 1 --2- 172.21.2.102:0/2222731217 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6f940186c0 0x7f6f9401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:07.971 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:07.972 7f6fb3fff700 1 --2- 172.21.2.102:0/2222731217 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6f940186c0 0x7f6f9401abc0 crc :-1 s=READY pgs=493 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:07.998 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:08.002 7f6fba26a700 1 -- 172.21.2.102:0/2222731217 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fb407ed80 con 0x7f6fb40a33c0 2021-02-10T11:22:08.001 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:08.004 7f6fb0ff9700 1 -- 172.21.2.102:0/2222731217 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f6fac000f40 con 0x7f6fb40a33c0 2021-02-10T11:22:08.132 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:08.135 7f6fba26a700 1 -- 172.21.2.102:0/2222731217 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestVolumes.test_async_subvolume_rm"]} v 0) v1 -- 0x7f6fb40adf90 con 0x7f6fb40a33c0 2021-02-10T11:22:08.583 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:08.586 7f6fb0ff9700 1 -- 172.21.2.102:0/2222731217 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestVolumes.test_async_subvolume_rm"]}]=0 v211) v1 ==== 140+0+0 (secure 0 0 0) 0x7f6fac006260 con 0x7f6fb40a33c0 2021-02-10T11:22:08.601 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:08.605 7f6fba26a700 1 -- 172.21.2.102:0/2222731217 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6f940186c0 msgr2=0x7f6f9401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:08.601 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:08.605 7f6fba26a700 1 --2- 172.21.2.102:0/2222731217 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6f940186c0 0x7f6f9401abc0 crc :-1 s=READY pgs=493 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:08.601 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:08.605 7f6fba26a700 1 -- 172.21.2.102:0/2222731217 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6fb40a33c0 msgr2=0x7f6fb40a03d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:08.602 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:08.605 7f6fba26a700 1 --2- 172.21.2.102:0/2222731217 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6fb40a33c0 0x7f6fb40a03d0 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 rx=0x7f6fac009980 tx=0x7f6fac00bcd0).stop 2021-02-10T11:22:08.602 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:08.605 7f6fba26a700 1 -- 172.21.2.102:0/2222731217 shutdown_connections 2021-02-10T11:22:08.602 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:08.605 7f6fba26a700 1 --2- 172.21.2.102:0/2222731217 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6fb40a33c0 0x7f6fb40a03d0 unknown :-1 s=CLOSED pgs=1013 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:08.602 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:08.605 7f6fba26a700 1 --2- 172.21.2.102:0/2222731217 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6fb40a29a0 0x7f6fb40a2da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:08.602 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:08.605 7f6fba26a700 1 --2- 172.21.2.102:0/2222731217 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6f940186c0 0x7f6f9401abc0 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:08.602 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:08.605 7f6fba26a700 1 --2- 172.21.2.102:0/2222731217 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6fb4191bb0 0x7f6fb40a2380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:08.602 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:08.605 7f6fba26a700 1 -- 172.21.2.102:0/2222731217 >> 172.21.2.102:0/2222731217 conn(0x7f6fb41a31d0 msgr2=0x7f6fb408b830 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:08.602 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:08.605 7f6fba26a700 1 -- 172.21.2.102:0/2222731217 shutdown_connections 2021-02-10T11:22:08.603 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:08.605 7f6fba26a700 1 -- 172.21.2.102:0/2222731217 wait complete. 2021-02-10T11:22:08.610 INFO:tasks.cephfs_test_runner:test_async_subvolume_rm (tasks.cephfs.test_volumes.TestVolumes) ... ok 2021-02-10T11:22:08.610 INFO:tasks.cephfs_test_runner:Starting test: test_authorize_allow_existing_id_option (tasks.cephfs.test_volumes.TestVolumes) 2021-02-10T11:22:08.610 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestVolumes.test_authorize_allow_existing_id_option' 2021-02-10T11:22:09.034 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:09.022 7fbb75e4b700 1 Processor -- start 2021-02-10T11:22:09.034 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:09.022 7fbb75e4b700 1 -- start start 2021-02-10T11:22:09.035 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:09.022 7fbb75e4b700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fbb700e5070 0x7fbb701b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:09.035 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:09.022 7fbb75e4b700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbb700e5470 0x7fbb70080310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:09.035 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:09.022 7fbb75e4b700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb7007ed80 con 0x7fbb701b3e40 2021-02-10T11:22:09.035 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:09.022 7fbb75e4b700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb700d1790 con 0x7fbb700e5070 2021-02-10T11:22:09.035 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:09.022 7fbb75e4b700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb700d19b0 con 0x7fbb700e5470 2021-02-10T11:22:09.035 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:09.023 7fbb6f7fe700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fbb700e5070 0x7fbb701b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:09.035 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:09.023 7fbb6effd700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fbb701b3e40 0x7fbb7007d690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:47382/0 (socket says 172.21.2.102:47382) 2021-02-10T11:22:09.035 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:09.023 7fbb6effd700 1 -- 172.21.2.102:0/2160506867 learned_addr learned my addr 172.21.2.102:0/2160506867 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:22:09.035 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:09.023 7fbb6e7fc700 1 -- 172.21.2.102:0/2160506867 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1232530134 0 0) 0x7fbb600009b0 con 0x7fbb701b3e40 2021-02-10T11:22:09.035 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:09.023 7fbb6e7fc700 1 -- 172.21.2.102:0/2160506867 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb5c002fe0 con 0x7fbb701b3e40 2021-02-10T11:22:09.035 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:09.023 7fbb6f7fe700 1 -- 172.21.2.102:0/2160506867 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbb700e5470 msgr2=0x7fbb70080310 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:22:09.035 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:09.023 7fbb6f7fe700 1 --2- 172.21.2.102:0/2160506867 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbb700e5470 0x7fbb70080310 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:09.036 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:09.023 7fbb6f7fe700 1 -- 172.21.2.102:0/2160506867 >> v1:172.21.2.102:6789/0 conn(0x7fbb701b3e40 legacy=0x7fbb7007d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:09.036 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:09.023 7fbb6f7fe700 1 -- 172.21.2.102:0/2160506867 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb700d1bd0 con 0x7fbb700e5070 2021-02-10T11:22:09.036 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:09.023 7fbb6f7fe700 1 --2- 172.21.2.102:0/2160506867 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbb700e5070 0x7fbb701b3820 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 rx=0x7fbb58006650 tx=0x7fbb58009a80).ready entity=mon.0 client_cookie=33d7782fbeca065a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:09.036 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:09.023 7fbb6e7fc700 1 -- 172.21.2.102:0/2160506867 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbb5800c0c0 con 0x7fbb700e5070 2021-02-10T11:22:09.036 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:09.023 7fbb6e7fc700 1 -- 172.21.2.102:0/2160506867 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb5800c440 con 0x7fbb700e5070 2021-02-10T11:22:09.036 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:09.023 7fbb6e7fc700 1 -- 172.21.2.102:0/2160506867 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbb5800c8a0 con 0x7fbb700e5070 2021-02-10T11:22:09.036 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:09.023 7fbb75e4b700 1 -- 172.21.2.102:0/2160506867 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbb700e5070 msgr2=0x7fbb701b3820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:09.036 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:09.023 7fbb75e4b700 1 --2- 172.21.2.102:0/2160506867 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbb700e5070 0x7fbb701b3820 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 rx=0x7fbb58006650 tx=0x7fbb58009a80).stop 2021-02-10T11:22:09.036 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:09.024 7fbb75e4b700 1 -- 172.21.2.102:0/2160506867 shutdown_connections 2021-02-10T11:22:09.036 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:09.024 7fbb75e4b700 1 --2- 172.21.2.102:0/2160506867 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbb700e5470 0x7fbb70080310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:09.036 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:09.024 7fbb75e4b700 1 --2- 172.21.2.102:0/2160506867 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbb700e5070 0x7fbb701b3820 unknown :-1 s=CLOSED pgs=1015 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:09.036 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:09.024 7fbb75e4b700 1 -- 172.21.2.102:0/2160506867 >> 172.21.2.102:0/2160506867 conn(0x7fbb700ce690 msgr2=0x7fbb700e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:09.036 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:09.024 7fbb75e4b700 1 -- 172.21.2.102:0/2160506867 shutdown_connections 2021-02-10T11:22:09.037 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:09.024 7fbb75e4b700 1 -- 172.21.2.102:0/2160506867 wait complete. 2021-02-10T11:22:09.038 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:09.038 7fbb75e4b700 1 Processor -- start 2021-02-10T11:22:09.038 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:09.039 7fbb75e4b700 1 -- start start 2021-02-10T11:22:09.038 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:09.039 7fbb75e4b700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fbb7008d400 0x7fbb70081230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:09.038 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:09.039 7fbb75e4b700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbb70081850 0x7fbb700acf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:09.038 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:09.039 7fbb75e4b700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbb70081c50 0x7fbb700ad5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:09.039 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:09.039 7fbb75e4b700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb70082110 con 0x7fbb70081850 2021-02-10T11:22:09.039 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:09.039 7fbb75e4b700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb7008d800 con 0x7fbb7008d400 2021-02-10T11:22:09.039 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:09.039 7fbb75e4b700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb700adbc0 con 0x7fbb70081c50 2021-02-10T11:22:09.039 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:09.039 7fbb6effd700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbb70081850 0x7fbb700acf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:09.039 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:09.039 7fbb6effd700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbb70081850 0x7fbb700acf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:55164/0 (socket says 172.21.2.102:55164) 2021-02-10T11:22:09.039 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:09.039 7fbb6effd700 1 -- 172.21.2.102:0/3936136013 learned_addr learned my addr 172.21.2.102:0/3936136013 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:22:09.039 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:09.039 7fbb6effd700 1 -- 172.21.2.102:0/3936136013 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbb70081c50 msgr2=0x7fbb700ad5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:09.039 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:09.039 7fbb6effd700 1 --2- 172.21.2.102:0/3936136013 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbb70081c50 0x7fbb700ad5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:09.039 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:09.039 7fbb6effd700 1 -- 172.21.2.102:0/3936136013 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fbb7008d400 msgr2=0x7fbb70081230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:09.039 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:09.039 7fbb6effd700 1 --2- 172.21.2.102:0/3936136013 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fbb7008d400 0x7fbb70081230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:09.039 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:09.039 7fbb6effd700 1 -- 172.21.2.102:0/3936136013 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb70187fc0 con 0x7fbb70081850 2021-02-10T11:22:09.040 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:09.040 7fbb6effd700 1 --2- 172.21.2.102:0/3936136013 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbb70081850 0x7fbb700acf80 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 rx=0x7fbb60001c60 tx=0x7fbb6000a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:09.040 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:09.040 7fbb67fff700 1 -- 172.21.2.102:0/3936136013 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbb6000acc0 con 0x7fbb70081850 2021-02-10T11:22:09.040 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:09.040 7fbb67fff700 1 -- 172.21.2.102:0/3936136013 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb60004140 con 0x7fbb70081850 2021-02-10T11:22:09.040 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:09.040 7fbb75e4b700 1 -- 172.21.2.102:0/3936136013 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb70187fc0 con 0x7fbb70081850 2021-02-10T11:22:09.040 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:09.040 7fbb67fff700 1 -- 172.21.2.102:0/3936136013 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbb600045a0 con 0x7fbb70081850 2021-02-10T11:22:09.040 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:09.040 7fbb75e4b700 1 -- 172.21.2.102:0/3936136013 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb70187fc0 con 0x7fbb70081850 2021-02-10T11:22:09.040 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:09.041 7fbb67fff700 1 -- 172.21.2.102:0/3936136013 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fbb60007f30 con 0x7fbb70081850 2021-02-10T11:22:09.040 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:09.041 7fbb67fff700 1 --2- 172.21.2.102:0/3936136013 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fbb540186c0 0x7fbb5401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:09.040 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:09.041 7fbb67fff700 1 -- 172.21.2.102:0/3936136013 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fbb600327b0 con 0x7fbb70081850 2021-02-10T11:22:09.040 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:09.041 7fbb6f7fe700 1 --2- 172.21.2.102:0/3936136013 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fbb540186c0 0x7fbb5401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:09.041 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:09.041 7fbb6f7fe700 1 --2- 172.21.2.102:0/3936136013 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fbb540186c0 0x7fbb5401abc0 crc :-1 s=READY pgs=494 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:09.069 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:09.073 7fbb75e4b700 1 -- 172.21.2.102:0/3936136013 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb7008d800 con 0x7fbb70081850 2021-02-10T11:22:09.071 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:09.075 7fbb67fff700 1 -- 172.21.2.102:0/3936136013 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fbb60000e20 con 0x7fbb70081850 2021-02-10T11:22:09.203 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:09.206 7fbb75e4b700 1 -- 172.21.2.102:0/3936136013 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestVolumes.test_authorize_allow_existing_id_option"]} v 0) v1 -- 0x7fbb7008d800 con 0x7fbb70081850 2021-02-10T11:22:09.586 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:09.590 7fbb67fff700 1 -- 172.21.2.102:0/3936136013 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestVolumes.test_authorize_allow_existing_id_option"]}]=0 v212) v1 ==== 159+0+0 (secure 0 0 0) 0x7fbb60007f30 con 0x7fbb70081850 2021-02-10T11:22:09.621 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:09.624 7fbb75e4b700 1 -- 172.21.2.102:0/3936136013 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fbb540186c0 msgr2=0x7fbb5401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:09.621 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:09.624 7fbb75e4b700 1 --2- 172.21.2.102:0/3936136013 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fbb540186c0 0x7fbb5401abc0 crc :-1 s=READY pgs=494 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:09.621 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:09.624 7fbb75e4b700 1 -- 172.21.2.102:0/3936136013 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbb70081850 msgr2=0x7fbb700acf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:09.621 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:09.624 7fbb75e4b700 1 --2- 172.21.2.102:0/3936136013 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbb70081850 0x7fbb700acf80 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 rx=0x7fbb60001c60 tx=0x7fbb6000a8d0).stop 2021-02-10T11:22:09.621 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:09.624 7fbb75e4b700 1 -- 172.21.2.102:0/3936136013 shutdown_connections 2021-02-10T11:22:09.621 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:09.624 7fbb75e4b700 1 --2- 172.21.2.102:0/3936136013 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbb70081c50 0x7fbb700ad5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:09.622 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:09.624 7fbb75e4b700 1 --2- 172.21.2.102:0/3936136013 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbb70081850 0x7fbb700acf80 unknown :-1 s=CLOSED pgs=1016 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:09.622 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:09.624 7fbb75e4b700 1 --2- 172.21.2.102:0/3936136013 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fbb540186c0 0x7fbb5401abc0 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:09.622 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:09.624 7fbb75e4b700 1 --2- 172.21.2.102:0/3936136013 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fbb7008d400 0x7fbb70081230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:09.622 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:09.624 7fbb75e4b700 1 -- 172.21.2.102:0/3936136013 >> 172.21.2.102:0/3936136013 conn(0x7fbb701ac620 msgr2=0x7fbb700896e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:09.622 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:09.625 7fbb75e4b700 1 -- 172.21.2.102:0/3936136013 shutdown_connections 2021-02-10T11:22:09.622 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:09.625 7fbb75e4b700 1 -- 172.21.2.102:0/3936136013 wait complete. 2021-02-10T11:22:09.630 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config set mon mon_allow_pool_delete True 2021-02-10T11:22:10.062 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:10.047 7f058cef9700 1 Processor -- start 2021-02-10T11:22:10.062 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:10.047 7f058cef9700 1 -- start start 2021-02-10T11:22:10.062 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:10.047 7f058cef9700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f05881b0870 0x7f05880f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:10.062 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:10.047 7f058cef9700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05881b0c70 con 0x7f05880efe00 2021-02-10T11:22:10.063 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:10.047 7f058cef9700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05880f0b50 con 0x7f05881b0470 2021-02-10T11:22:10.063 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:10.047 7f058cef9700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05880e5ea0 con 0x7f05881b0870 2021-02-10T11:22:10.063 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:10.048 7f0587314700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f05880efe00 0x7f05880f0200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:47398/0 (socket says 172.21.2.102:47398) 2021-02-10T11:22:10.063 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:10.048 7f0587314700 1 -- 172.21.2.102:0/1138954720 learned_addr learned my addr 172.21.2.102:0/1138954720 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:22:10.063 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:10.048 7f0585b11700 1 -- 172.21.2.102:0/1138954720 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3255671482 0 0) 0x7f0580001500 con 0x7f05880efe00 2021-02-10T11:22:10.063 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:10.048 7f0585b11700 1 -- 172.21.2.102:0/1138954720 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0574002fe0 con 0x7f05880efe00 2021-02-10T11:22:10.063 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:10.048 7f0586312700 1 --2- 172.21.2.102:0/1138954720 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f05881b0870 0x7f05880f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:10.063 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:10.048 7f0585b11700 1 -- 172.21.2.102:0/1138954720 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 10689084 0 0) 0x7f0580001270 con 0x7f05880efe00 2021-02-10T11:22:10.063 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:10.048 7f0585b11700 1 -- 172.21.2.102:0/1138954720 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f05881b0870 msgr2=0x7f05880f20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:10.063 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:10.048 7f0585b11700 1 --2- 172.21.2.102:0/1138954720 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f05881b0870 0x7f05880f20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:10.063 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:10.048 7f0585b11700 1 -- 172.21.2.102:0/1138954720 >> v1:172.21.2.113:6789/0 conn(0x7f05881b0470 legacy=0x7f05880ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:10.064 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:10.048 7f0585b11700 1 -- 172.21.2.102:0/1138954720 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05880e52e0 con 0x7f05880efe00 2021-02-10T11:22:10.064 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:10.048 7f0585b11700 1 -- 172.21.2.102:0/1138954720 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7f0580001b50 con 0x7f05880efe00 2021-02-10T11:22:10.064 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:10.048 7f0585b11700 1 -- 172.21.2.102:0/1138954720 >> v1:172.21.2.102:6789/0 conn(0x7f05880efe00 legacy=0x7f05880f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:10.064 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:10.048 7f0585b11700 1 --2- 172.21.2.102:0/1138954720 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0574003a60 0x7f0574005f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:10.064 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:10.049 7f0585b11700 1 --2- 172.21.2.102:0/1138954720 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0574006780 0x7f0574008c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:10.064 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:10.049 7f0585b11700 1 --2- 172.21.2.102:0/1138954720 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0574009580 0x7f057400ba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:10.064 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:10.049 7f0585b11700 1 -- 172.21.2.102:0/1138954720 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f057400c2c0 con 0x7f0574006780 2021-02-10T11:22:10.064 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:10.049 7f0585b11700 1 -- 172.21.2.102:0/1138954720 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f057400c4e0 con 0x7f0574003a60 2021-02-10T11:22:10.064 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:10.049 7f0585b11700 1 -- 172.21.2.102:0/1138954720 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f057400c700 con 0x7f0574009580 2021-02-10T11:22:10.064 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:10.049 7f0586312700 1 --2- 172.21.2.102:0/1138954720 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0574006780 0x7f0574008c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:10.064 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:10.049 7f0587314700 1 --2- 172.21.2.102:0/1138954720 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0574009580 0x7f057400ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:10.064 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:10.049 7f0586b13700 1 --2- 172.21.2.102:0/1138954720 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0574003a60 0x7f0574005f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:10.064 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:10.049 7f0586312700 1 -- 172.21.2.102:0/1138954720 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0574009580 msgr2=0x7f057400ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:10.065 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:10.049 7f0586312700 1 --2- 172.21.2.102:0/1138954720 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0574009580 0x7f057400ba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:10.065 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:10.049 7f0586312700 1 -- 172.21.2.102:0/1138954720 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0574003a60 msgr2=0x7f0574005f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:10.065 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:10.049 7f0586312700 1 --2- 172.21.2.102:0/1138954720 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0574003a60 0x7f0574005f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:10.065 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:10.049 7f0586312700 1 -- 172.21.2.102:0/1138954720 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0574050970 con 0x7f0574006780 2021-02-10T11:22:10.065 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:10.049 7f0587314700 1 --2- 172.21.2.102:0/1138954720 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0574009580 0x7f057400ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:22:10.065 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:10.049 7f0586312700 1 --2- 172.21.2.102:0/1138954720 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0574006780 0x7f0574008c80 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 rx=0x7f0570009310 tx=0x7f0570009a90).ready entity=mon.0 client_cookie=b2084e803694f4ce server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:10.065 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:10.049 7f0585b11700 1 -- 172.21.2.102:0/1138954720 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f057000c0c0 con 0x7f0574006780 2021-02-10T11:22:10.065 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:10.049 7f0585b11700 1 -- 172.21.2.102:0/1138954720 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0570006400 con 0x7f0574006780 2021-02-10T11:22:10.065 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:10.049 7f0585b11700 1 -- 172.21.2.102:0/1138954720 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f057000c340 con 0x7f0574006780 2021-02-10T11:22:10.065 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:10.049 7f058cef9700 1 -- 172.21.2.102:0/1138954720 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0574006780 msgr2=0x7f0574008c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:10.065 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:10.049 7f058cef9700 1 --2- 172.21.2.102:0/1138954720 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0574006780 0x7f0574008c80 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 rx=0x7f0570009310 tx=0x7f0570009a90).stop 2021-02-10T11:22:10.065 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:10.049 7f0587314700 1 -- 172.21.2.102:0/1138954720 reap_dead start 2021-02-10T11:22:10.065 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:10.050 7f058cef9700 1 -- 172.21.2.102:0/1138954720 shutdown_connections 2021-02-10T11:22:10.065 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:10.050 7f058cef9700 1 -- 172.21.2.102:0/1138954720 >> 172.21.2.102:0/1138954720 conn(0x7f05880ce3e0 msgr2=0x7f05880e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:10.066 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:10.050 7f058cef9700 1 -- 172.21.2.102:0/1138954720 shutdown_connections 2021-02-10T11:22:10.066 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:10.050 7f058cef9700 1 -- 172.21.2.102:0/1138954720 wait complete. 2021-02-10T11:22:10.066 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:10.065 7f058cef9700 1 Processor -- start 2021-02-10T11:22:10.066 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:10.065 7f058cef9700 1 -- start start 2021-02-10T11:22:10.066 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:10.065 7f058cef9700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f058809b930 0x7f05880b7c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:10.066 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:10.065 7f058cef9700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f05880b8410 0x7f05880b8810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:10.066 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:10.065 7f058cef9700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f05880c1f80 0x7f05880c2380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:10.066 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:10.065 7f058cef9700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05880c1a00 con 0x7f05880b8410 2021-02-10T11:22:10.066 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:10.065 7f058cef9700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05880b8e30 con 0x7f05880c1f80 2021-02-10T11:22:10.066 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:10.065 7f058cef9700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05880b9050 con 0x7f058809b930 2021-02-10T11:22:10.066 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:10.066 7f0586312700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f05880b8410 0x7f05880b8810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:10.066 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:10.066 7f0586312700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f05880b8410 0x7f05880b8810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:55184/0 (socket says 172.21.2.102:55184) 2021-02-10T11:22:10.066 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:10.066 7f0586312700 1 -- 172.21.2.102:0/1320918557 learned_addr learned my addr 172.21.2.102:0/1320918557 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:22:10.067 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:10.066 7f0587314700 1 --2- 172.21.2.102:0/1320918557 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f05880c1f80 0x7f05880c2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:10.067 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:10.066 7f0586b13700 1 --2- 172.21.2.102:0/1320918557 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f058809b930 0x7f05880b7c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:10.068 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:10.066 7f0586312700 1 -- 172.21.2.102:0/1320918557 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f058809b930 msgr2=0x7f05880b7c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:10.068 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:10.066 7f0586312700 1 --2- 172.21.2.102:0/1320918557 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f058809b930 0x7f05880b7c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:10.068 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:10.066 7f0586312700 1 -- 172.21.2.102:0/1320918557 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f05880c1f80 msgr2=0x7f05880c2380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:10.068 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:10.066 7f0586312700 1 --2- 172.21.2.102:0/1320918557 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f05880c1f80 0x7f05880c2380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:10.068 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:10.066 7f0586312700 1 -- 172.21.2.102:0/1320918557 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05880c29a0 con 0x7f05880b8410 2021-02-10T11:22:10.068 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:10.066 7f0587314700 1 --2- 172.21.2.102:0/1320918557 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f05880c1f80 0x7f05880c2380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:22:10.068 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:10.066 7f0586b13700 1 --2- 172.21.2.102:0/1320918557 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f058809b930 0x7f05880b7c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:22:10.068 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:10.066 7f0586312700 1 --2- 172.21.2.102:0/1320918557 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f05880b8410 0x7f05880b8810 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 rx=0x7f0570009670 tx=0x7f057000b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:10.068 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:10.066 7f057f7fe700 1 -- 172.21.2.102:0/1320918557 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0570000940 con 0x7f05880b8410 2021-02-10T11:22:10.068 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:10.066 7f057f7fe700 1 -- 172.21.2.102:0/1320918557 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05700093e0 con 0x7f05880b8410 2021-02-10T11:22:10.069 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:10.066 7f057f7fe700 1 -- 172.21.2.102:0/1320918557 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0570002080 con 0x7f05880b8410 2021-02-10T11:22:10.069 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:10.066 7f058cef9700 1 -- 172.21.2.102:0/1320918557 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05880c29a0 con 0x7f05880b8410 2021-02-10T11:22:10.069 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:10.066 7f058cef9700 1 -- 172.21.2.102:0/1320918557 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05880c29a0 con 0x7f05880b8410 2021-02-10T11:22:10.069 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:10.067 7f057f7fe700 1 -- 172.21.2.102:0/1320918557 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f0570000c90 con 0x7f05880b8410 2021-02-10T11:22:10.069 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:10.067 7f057f7fe700 1 --2- 172.21.2.102:0/1320918557 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f056c0186c0 0x7f056c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:10.069 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:10.067 7f057f7fe700 1 -- 172.21.2.102:0/1320918557 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f05700327e0 con 0x7f05880b8410 2021-02-10T11:22:10.069 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:10.067 7f0586b13700 1 --2- 172.21.2.102:0/1320918557 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f056c0186c0 0x7f056c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:10.069 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:10.067 7f0586b13700 1 --2- 172.21.2.102:0/1320918557 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f056c0186c0 0x7f056c01abc0 crc :-1 s=READY pgs=495 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:10.094 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:10.098 7f058cef9700 1 -- 172.21.2.102:0/1320918557 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0588105b30 con 0x7f05880b8410 2021-02-10T11:22:10.096 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:10.100 7f057f7fe700 1 -- 172.21.2.102:0/1320918557 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f05700093a0 con 0x7f05880b8410 2021-02-10T11:22:10.244 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:10.248 7f058cef9700 1 -- 172.21.2.102:0/1320918557 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f0588105b30 con 0x7f05880b8410 2021-02-10T11:22:10.249 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:10.252 7f057f7fe700 1 -- 172.21.2.102:0/1320918557 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v4)=0 v4) v1 ==== 122+0+0 (secure 0 0 0) 0x7f0570000c90 con 0x7f05880b8410 2021-02-10T11:22:10.253 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:10.257 7f058cef9700 1 -- 172.21.2.102:0/1320918557 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f056c0186c0 msgr2=0x7f056c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:10.253 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:10.257 7f058cef9700 1 --2- 172.21.2.102:0/1320918557 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f056c0186c0 0x7f056c01abc0 crc :-1 s=READY pgs=495 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:10.253 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:10.257 7f058cef9700 1 -- 172.21.2.102:0/1320918557 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f05880b8410 msgr2=0x7f05880b8810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:10.253 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:10.257 7f058cef9700 1 --2- 172.21.2.102:0/1320918557 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f05880b8410 0x7f05880b8810 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 rx=0x7f0570009670 tx=0x7f057000b960).stop 2021-02-10T11:22:10.254 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:10.257 7f058cef9700 1 -- 172.21.2.102:0/1320918557 shutdown_connections 2021-02-10T11:22:10.254 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:10.257 7f058cef9700 1 --2- 172.21.2.102:0/1320918557 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f056c0186c0 0x7f056c01abc0 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:10.254 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:10.257 7f058cef9700 1 --2- 172.21.2.102:0/1320918557 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f05880c1f80 0x7f05880c2380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:10.254 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:10.257 7f058cef9700 1 --2- 172.21.2.102:0/1320918557 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f05880b8410 0x7f05880b8810 unknown :-1 s=CLOSED pgs=1019 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:10.254 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:10.257 7f058cef9700 1 --2- 172.21.2.102:0/1320918557 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f058809b930 0x7f05880b7c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:10.254 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:10.257 7f058cef9700 1 -- 172.21.2.102:0/1320918557 >> 172.21.2.102:0/1320918557 conn(0x7f0588094e10 msgr2=0x7f05880a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:10.255 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:10.257 7f058cef9700 1 -- 172.21.2.102:0/1320918557 shutdown_connections 2021-02-10T11:22:10.255 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:10.257 7f058cef9700 1 -- 172.21.2.102:0/1320918557 wait complete. 2021-02-10T11:22:10.262 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2021-02-10T11:22:10.262 DEBUG:teuthology.orchestra.run.gibba002:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:22:10.608 DEBUG:teuthology.orchestra.run.gibba013:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:22:10.952 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2021-02-10T11:22:10.952 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:22:10.952 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:22:10.952 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:22:11.287 INFO:teuthology.orchestra.run.gibba002.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.0’: No such file or directory 2021-02-10T11:22:11.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:22:11.287 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.0 2021-02-10T11:22:11.287 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:22:11.288 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:22:11.288 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:22:11.622 INFO:teuthology.orchestra.run.gibba002.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.1’: No such file or directory 2021-02-10T11:22:11.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:22:11.622 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2021-02-10T11:22:11.622 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:22:11.623 DEBUG:teuthology.orchestra.run.gibba013:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:22:11.623 DEBUG:teuthology.orchestra.run.gibba013:> ' -- /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:22:11.956 INFO:teuthology.orchestra.run.gibba013.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.2’: No such file or directory 2021-02-10T11:22:11.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:22:11.957 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2021-02-10T11:22:11.957 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:22:11.957 DEBUG:teuthology.orchestra.run.gibba013:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:22:11.957 DEBUG:teuthology.orchestra.run.gibba013:> ' -- /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:22:12.291 INFO:teuthology.orchestra.run.gibba013.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.3’: No such file or directory 2021-02-10T11:22:12.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:22:12.292 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.3 2021-02-10T11:22:12.292 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json-pretty 2021-02-10T11:22:12.720 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:12.706 7f0a9d2b7700 1 Processor -- start 2021-02-10T11:22:12.720 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:12.706 7f0a9d2b7700 1 -- start start 2021-02-10T11:22:12.720 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:12.707 7f0a9d2b7700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0a980e5070 0x7f0a981b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:12.721 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:12.707 7f0a9d2b7700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f0a981b3e40 0x7f0a98080310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:12.721 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:12.707 7f0a9d2b7700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a9807ed80 con 0x7f0a980e5470 2021-02-10T11:22:12.721 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:12.707 7f0a9d2b7700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a980d1790 con 0x7f0a981b3e40 2021-02-10T11:22:12.721 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:12.707 7f0a9d2b7700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a980d1980 con 0x7f0a980e5070 2021-02-10T11:22:12.721 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:12.707 7f0a9787e700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f0a981b3e40 0x7f0a98080310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:12.721 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:12.707 7f0a9787e700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f0a981b3e40 0x7f0a98080310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.113:3300/0 says I am v2:172.21.2.102:58620/0 (socket says 172.21.2.102:58620) 2021-02-10T11:22:12.722 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:12.707 7f0a9787e700 1 -- 172.21.2.102:0/1469111355 learned_addr learned my addr 172.21.2.102:0/1469111355 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:22:12.722 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:12.707 7f0a9707d700 1 --2- 172.21.2.102:0/1469111355 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0a980e5070 0x7f0a981b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:12.722 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:12.707 7f0a9687c700 1 --1- 172.21.2.102:0/1469111355 >> v1:172.21.2.113:6789/0 conn(0x7f0a980e5470 0x7f0a9807d690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.113:6789/0 says I am v1:172.21.2.102:54738/0 (socket says 172.21.2.102:54738) 2021-02-10T11:22:12.722 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:12.708 7f0a9707d700 1 -- 172.21.2.102:0/1469111355 >> v2:172.21.2.113:3300/0 conn(0x7f0a981b3e40 msgr2=0x7f0a98080310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:12.722 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:12.708 7f0a9707d700 1 --2- 172.21.2.102:0/1469111355 >> v2:172.21.2.113:3300/0 conn(0x7f0a981b3e40 0x7f0a98080310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:12.722 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:12.708 7f0a9707d700 1 -- 172.21.2.102:0/1469111355 >> v1:172.21.2.113:6789/0 conn(0x7f0a980e5470 legacy=0x7f0a9807d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:12.723 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:12.708 7f0a9707d700 1 -- 172.21.2.102:0/1469111355 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a980d1ba0 con 0x7f0a980e5070 2021-02-10T11:22:12.723 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:12.708 7f0a9787e700 1 --2- 172.21.2.102:0/1469111355 >> v2:172.21.2.113:3300/0 conn(0x7f0a981b3e40 0x7f0a98080310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:22:12.723 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:12.708 7f0a9707d700 1 --2- 172.21.2.102:0/1469111355 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0a980e5070 0x7f0a981b3820 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7f0a80006650 tx=0x7f0a80009a80).ready entity=mon.2 client_cookie=f9a0ed0b81d53a04 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:12.723 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:12.708 7f0a9607b700 1 -- 172.21.2.102:0/1469111355 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0a8000c0c0 con 0x7f0a980e5070 2021-02-10T11:22:12.723 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:12.708 7f0a9607b700 1 -- 172.21.2.102:0/1469111355 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a8000c340 con 0x7f0a980e5070 2021-02-10T11:22:12.723 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:12.708 7f0a9607b700 1 -- 172.21.2.102:0/1469111355 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0a8000c860 con 0x7f0a980e5070 2021-02-10T11:22:12.724 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:12.708 7f0a9d2b7700 1 -- 172.21.2.102:0/1469111355 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0a980e5070 msgr2=0x7f0a981b3820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:12.724 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:12.708 7f0a9d2b7700 1 --2- 172.21.2.102:0/1469111355 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0a980e5070 0x7f0a981b3820 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7f0a80006650 tx=0x7f0a80009a80).stop 2021-02-10T11:22:12.724 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:12.709 7f0a9d2b7700 1 -- 172.21.2.102:0/1469111355 shutdown_connections 2021-02-10T11:22:12.724 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:12.709 7f0a9d2b7700 1 --2- 172.21.2.102:0/1469111355 >> v2:172.21.2.113:3300/0 conn(0x7f0a981b3e40 0x7f0a98080310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:12.724 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:12.709 7f0a9d2b7700 1 --2- 172.21.2.102:0/1469111355 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0a980e5070 0x7f0a981b3820 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:12.724 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:12.709 7f0a9d2b7700 1 -- 172.21.2.102:0/1469111355 >> 172.21.2.102:0/1469111355 conn(0x7f0a980ce690 msgr2=0x7f0a980e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:12.724 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:12.709 7f0a9d2b7700 1 -- 172.21.2.102:0/1469111355 shutdown_connections 2021-02-10T11:22:12.724 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:12.709 7f0a9d2b7700 1 -- 172.21.2.102:0/1469111355 wait complete. 2021-02-10T11:22:12.725 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:12.723 7f0a9d2b7700 1 Processor -- start 2021-02-10T11:22:12.725 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:12.723 7f0a9d2b7700 1 -- start start 2021-02-10T11:22:12.726 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:12.724 7f0a9d2b7700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0a98097250 0x7f0a981a1b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:12.726 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:12.724 7f0a9d2b7700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0a9808d5a0 0x7f0a980ae3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:12.726 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:12.724 7f0a9d2b7700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0a980ae9c0 0x7f0a981a2870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:12.726 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:12.724 7f0a9d2b7700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a981a22f0 con 0x7f0a980ae9c0 2021-02-10T11:22:12.726 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:12.724 7f0a9d2b7700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a981b2d40 con 0x7f0a9808d5a0 2021-02-10T11:22:12.726 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:12.724 7f0a9d2b7700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a981b2f20 con 0x7f0a98097250 2021-02-10T11:22:12.726 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:12.724 7f0a9787e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0a980ae9c0 0x7f0a981a2870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:12.727 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:12.724 7f0a9787e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0a980ae9c0 0x7f0a981a2870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:55194/0 (socket says 172.21.2.102:55194) 2021-02-10T11:22:12.727 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:12.724 7f0a9787e700 1 -- 172.21.2.102:0/861306313 learned_addr learned my addr 172.21.2.102:0/861306313 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:22:12.727 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:12.724 7f0a9707d700 1 --2- 172.21.2.102:0/861306313 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0a98097250 0x7f0a981a1b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:12.727 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:12.724 7f0a9687c700 1 --2- 172.21.2.102:0/861306313 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0a9808d5a0 0x7f0a980ae3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:12.727 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:12.724 7f0a9787e700 1 -- 172.21.2.102:0/861306313 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0a98097250 msgr2=0x7f0a981a1b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:12.727 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:12.724 7f0a9787e700 1 --2- 172.21.2.102:0/861306313 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0a98097250 0x7f0a981a1b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:12.727 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:12.724 7f0a9787e700 1 -- 172.21.2.102:0/861306313 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0a9808d5a0 msgr2=0x7f0a980ae3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:12.727 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:12.724 7f0a9787e700 1 --2- 172.21.2.102:0/861306313 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0a9808d5a0 0x7f0a980ae3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:12.727 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:12.724 7f0a9787e700 1 -- 172.21.2.102:0/861306313 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a9808d9d0 con 0x7f0a980ae9c0 2021-02-10T11:22:12.727 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:12.724 7f0a9787e700 1 --2- 172.21.2.102:0/861306313 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0a980ae9c0 0x7f0a981a2870 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 rx=0x7f0a90009940 tx=0x7f0a9000bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:12.727 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:12.725 7f0a8ffff700 1 -- 172.21.2.102:0/861306313 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0a9000c2f0 con 0x7f0a980ae9c0 2021-02-10T11:22:12.727 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:12.725 7f0a8ffff700 1 -- 172.21.2.102:0/861306313 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a90006790 con 0x7f0a980ae9c0 2021-02-10T11:22:12.727 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:12.725 7f0a8ffff700 1 -- 172.21.2.102:0/861306313 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0a9000c7f0 con 0x7f0a980ae9c0 2021-02-10T11:22:12.728 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:12.725 7f0a9d2b7700 1 -- 172.21.2.102:0/861306313 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a9808d9d0 con 0x7f0a980ae9c0 2021-02-10T11:22:12.728 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:12.725 7f0a9d2b7700 1 -- 172.21.2.102:0/861306313 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a981a3180 con 0x7f0a980ae9c0 2021-02-10T11:22:12.728 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:12.725 7f0a8ffff700 1 -- 172.21.2.102:0/861306313 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f0a900011f0 con 0x7f0a980ae9c0 2021-02-10T11:22:12.728 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:12.725 7f0a8ffff700 1 --2- 172.21.2.102:0/861306313 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0a7c0186c0 0x7f0a7c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:12.728 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:12.725 7f0a8ffff700 1 -- 172.21.2.102:0/861306313 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f0a90032910 con 0x7f0a980ae9c0 2021-02-10T11:22:12.728 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:12.726 7f0a9707d700 1 --2- 172.21.2.102:0/861306313 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0a7c0186c0 0x7f0a7c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:12.728 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:12.726 7f0a9707d700 1 --2- 172.21.2.102:0/861306313 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0a7c0186c0 0x7f0a7c01abc0 crc :-1 s=READY pgs=496 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:12.753 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:12.757 7f0a9d2b7700 1 -- 172.21.2.102:0/861306313 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a98107170 con 0x7f0a980ae9c0 2021-02-10T11:22:12.756 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:12.760 7f0a8ffff700 1 -- 172.21.2.102:0/861306313 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f0a90020ee0 con 0x7f0a980ae9c0 2021-02-10T11:22:12.879 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:12.883 7f0a9d2b7700 1 -- 172.21.2.102:0/861306313 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7f0a9807ed80 con 0x7f0a980ae9c0 2021-02-10T11:22:12.880 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:12.884 7f0a8ffff700 1 -- 172.21.2.102:0/861306313 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v30) v1 ==== 81+0+30360 (secure 0 0 0) 0x7f0a90020ee0 con 0x7f0a980ae9c0 2021-02-10T11:22:12.881 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:22:12.881 INFO:teuthology.orchestra.run.gibba002.stdout:{ 2021-02-10T11:22:12.881 INFO:teuthology.orchestra.run.gibba002.stdout: "epoch": 30, 2021-02-10T11:22:12.881 INFO:teuthology.orchestra.run.gibba002.stdout: "fsid": "68db30ea-dbce-4aef-a6df-f01c6fea0513", 2021-02-10T11:22:12.881 INFO:teuthology.orchestra.run.gibba002.stdout: "created": "2021-02-10 11:03:06.331557", 2021-02-10T11:22:12.881 INFO:teuthology.orchestra.run.gibba002.stdout: "modified": "2021-02-10 11:05:13.569263", 2021-02-10T11:22:12.881 INFO:teuthology.orchestra.run.gibba002.stdout: "last_up_change": "2021-02-10 11:03:32.690562", 2021-02-10T11:22:12.881 INFO:teuthology.orchestra.run.gibba002.stdout: "last_in_change": "2021-02-10 11:03:32.690562", 2021-02-10T11:22:12.881 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2021-02-10T11:22:12.881 INFO:teuthology.orchestra.run.gibba002.stdout: "flags_num": 5799936, 2021-02-10T11:22:12.882 INFO:teuthology.orchestra.run.gibba002.stdout: "flags_set": [ 2021-02-10T11:22:12.882 INFO:teuthology.orchestra.run.gibba002.stdout: "pglog_hardlimit", 2021-02-10T11:22:12.882 INFO:teuthology.orchestra.run.gibba002.stdout: "purged_snapdirs", 2021-02-10T11:22:12.882 INFO:teuthology.orchestra.run.gibba002.stdout: "recovery_deletes", 2021-02-10T11:22:12.882 INFO:teuthology.orchestra.run.gibba002.stdout: "sortbitwise" 2021-02-10T11:22:12.882 INFO:teuthology.orchestra.run.gibba002.stdout: ], 2021-02-10T11:22:12.882 INFO:teuthology.orchestra.run.gibba002.stdout: "crush_version": 4, 2021-02-10T11:22:12.882 INFO:teuthology.orchestra.run.gibba002.stdout: "full_ratio": 0.94999998807907104, 2021-02-10T11:22:12.882 INFO:teuthology.orchestra.run.gibba002.stdout: "backfillfull_ratio": 0.89999997615814209, 2021-02-10T11:22:12.882 INFO:teuthology.orchestra.run.gibba002.stdout: "nearfull_ratio": 0.85000002384185791, 2021-02-10T11:22:12.882 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_snapshot": "", 2021-02-10T11:22:12.882 INFO:teuthology.orchestra.run.gibba002.stdout: "pool_max": 5, 2021-02-10T11:22:12.883 INFO:teuthology.orchestra.run.gibba002.stdout: "max_osd": 8, 2021-02-10T11:22:12.883 INFO:teuthology.orchestra.run.gibba002.stdout: "require_min_compat_client": "jewel", 2021-02-10T11:22:12.883 INFO:teuthology.orchestra.run.gibba002.stdout: "min_compat_client": "jewel", 2021-02-10T11:22:12.883 INFO:teuthology.orchestra.run.gibba002.stdout: "require_osd_release": "nautilus", 2021-02-10T11:22:12.883 INFO:teuthology.orchestra.run.gibba002.stdout: "pools": [ 2021-02-10T11:22:12.883 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:22:12.883 INFO:teuthology.orchestra.run.gibba002.stdout: "pool": 1, 2021-02-10T11:22:12.883 INFO:teuthology.orchestra.run.gibba002.stdout: "pool_name": "rbd", 2021-02-10T11:22:12.883 INFO:teuthology.orchestra.run.gibba002.stdout: "create_time": "2021-02-10 11:03:37.322519", 2021-02-10T11:22:12.883 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 1, 2021-02-10T11:22:12.883 INFO:teuthology.orchestra.run.gibba002.stdout: "flags_names": "hashpspool", 2021-02-10T11:22:12.883 INFO:teuthology.orchestra.run.gibba002.stdout: "type": 1, 2021-02-10T11:22:12.883 INFO:teuthology.orchestra.run.gibba002.stdout: "size": 2, 2021-02-10T11:22:12.884 INFO:teuthology.orchestra.run.gibba002.stdout: "min_size": 1, 2021-02-10T11:22:12.884 INFO:teuthology.orchestra.run.gibba002.stdout: "crush_rule": 0, 2021-02-10T11:22:12.884 INFO:teuthology.orchestra.run.gibba002.stdout: "object_hash": 2, 2021-02-10T11:22:12.884 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:22:12.884 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num": 8, 2021-02-10T11:22:12.884 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_placement_num": 8, 2021-02-10T11:22:12.884 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_placement_num_target": 8, 2021-02-10T11:22:12.884 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num_target": 8, 2021-02-10T11:22:12.884 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num_pending": 8, 2021-02-10T11:22:12.884 INFO:teuthology.orchestra.run.gibba002.stdout: "last_pg_merge_meta": { 2021-02-10T11:22:12.884 INFO:teuthology.orchestra.run.gibba002.stdout: "source_pgid": "0.0", 2021-02-10T11:22:12.884 INFO:teuthology.orchestra.run.gibba002.stdout: "ready_epoch": 0, 2021-02-10T11:22:12.884 INFO:teuthology.orchestra.run.gibba002.stdout: "last_epoch_started": 0, 2021-02-10T11:22:12.885 INFO:teuthology.orchestra.run.gibba002.stdout: "last_epoch_clean": 0, 2021-02-10T11:22:12.885 INFO:teuthology.orchestra.run.gibba002.stdout: "source_version": "0'0", 2021-02-10T11:22:12.885 INFO:teuthology.orchestra.run.gibba002.stdout: "target_version": "0'0" 2021-02-10T11:22:12.885 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:22:12.885 INFO:teuthology.orchestra.run.gibba002.stdout: "last_change": "16", 2021-02-10T11:22:12.885 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend": "0", 2021-02-10T11:22:12.885 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:22:12.885 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:22:12.885 INFO:teuthology.orchestra.run.gibba002.stdout: "auid": 0, 2021-02-10T11:22:12.885 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:22:12.885 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_seq": 0, 2021-02-10T11:22:12.885 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_epoch": 0, 2021-02-10T11:22:12.885 INFO:teuthology.orchestra.run.gibba002.stdout: "pool_snaps": [], 2021-02-10T11:22:12.886 INFO:teuthology.orchestra.run.gibba002.stdout: "removed_snaps": "[]", 2021-02-10T11:22:12.886 INFO:teuthology.orchestra.run.gibba002.stdout: "quota_max_bytes": 0, 2021-02-10T11:22:12.886 INFO:teuthology.orchestra.run.gibba002.stdout: "quota_max_objects": 0, 2021-02-10T11:22:12.886 INFO:teuthology.orchestra.run.gibba002.stdout: "tiers": [], 2021-02-10T11:22:12.886 INFO:teuthology.orchestra.run.gibba002.stdout: "tier_of": -1, 2021-02-10T11:22:12.886 INFO:teuthology.orchestra.run.gibba002.stdout: "read_tier": -1, 2021-02-10T11:22:12.886 INFO:teuthology.orchestra.run.gibba002.stdout: "write_tier": -1, 2021-02-10T11:22:12.886 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_mode": "none", 2021-02-10T11:22:12.886 INFO:teuthology.orchestra.run.gibba002.stdout: "target_max_bytes": 0, 2021-02-10T11:22:12.886 INFO:teuthology.orchestra.run.gibba002.stdout: "target_max_objects": 0, 2021-02-10T11:22:12.886 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:22:12.886 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:22:12.886 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:22:12.887 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_min_flush_age": 0, 2021-02-10T11:22:12.887 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_min_evict_age": 0, 2021-02-10T11:22:12.887 INFO:teuthology.orchestra.run.gibba002.stdout: "erasure_code_profile": "", 2021-02-10T11:22:12.887 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_params": { 2021-02-10T11:22:12.887 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "none" 2021-02-10T11:22:12.887 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:22:12.887 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_period": 0, 2021-02-10T11:22:12.887 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_count": 0, 2021-02-10T11:22:12.887 INFO:teuthology.orchestra.run.gibba002.stdout: "use_gmt_hitset": true, 2021-02-10T11:22:12.887 INFO:teuthology.orchestra.run.gibba002.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:22:12.887 INFO:teuthology.orchestra.run.gibba002.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:22:12.887 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:22:12.887 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:22:12.888 INFO:teuthology.orchestra.run.gibba002.stdout: "grade_table": [], 2021-02-10T11:22:12.888 INFO:teuthology.orchestra.run.gibba002.stdout: "stripe_width": 0, 2021-02-10T11:22:12.888 INFO:teuthology.orchestra.run.gibba002.stdout: "expected_num_objects": 0, 2021-02-10T11:22:12.888 INFO:teuthology.orchestra.run.gibba002.stdout: "fast_read": false, 2021-02-10T11:22:12.888 INFO:teuthology.orchestra.run.gibba002.stdout: "options": {}, 2021-02-10T11:22:12.888 INFO:teuthology.orchestra.run.gibba002.stdout: "application_metadata": { 2021-02-10T11:22:12.888 INFO:teuthology.orchestra.run.gibba002.stdout: "rbd": {} 2021-02-10T11:22:12.888 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:22:12.888 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:22:12.888 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:22:12.888 INFO:teuthology.orchestra.run.gibba002.stdout: "pool": 4, 2021-02-10T11:22:12.888 INFO:teuthology.orchestra.run.gibba002.stdout: "pool_name": "cephfs_metadata", 2021-02-10T11:22:12.888 INFO:teuthology.orchestra.run.gibba002.stdout: "create_time": "2021-02-10 11:05:10.538786", 2021-02-10T11:22:12.889 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 1, 2021-02-10T11:22:12.889 INFO:teuthology.orchestra.run.gibba002.stdout: "flags_names": "hashpspool", 2021-02-10T11:22:12.889 INFO:teuthology.orchestra.run.gibba002.stdout: "type": 1, 2021-02-10T11:22:12.889 INFO:teuthology.orchestra.run.gibba002.stdout: "size": 2, 2021-02-10T11:22:12.889 INFO:teuthology.orchestra.run.gibba002.stdout: "min_size": 1, 2021-02-10T11:22:12.889 INFO:teuthology.orchestra.run.gibba002.stdout: "crush_rule": 0, 2021-02-10T11:22:12.889 INFO:teuthology.orchestra.run.gibba002.stdout: "object_hash": 2, 2021-02-10T11:22:12.889 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:22:12.889 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num": 8, 2021-02-10T11:22:12.889 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_placement_num": 8, 2021-02-10T11:22:12.889 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_placement_num_target": 8, 2021-02-10T11:22:12.889 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num_target": 8, 2021-02-10T11:22:12.889 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num_pending": 8, 2021-02-10T11:22:12.890 INFO:teuthology.orchestra.run.gibba002.stdout: "last_pg_merge_meta": { 2021-02-10T11:22:12.890 INFO:teuthology.orchestra.run.gibba002.stdout: "source_pgid": "0.0", 2021-02-10T11:22:12.890 INFO:teuthology.orchestra.run.gibba002.stdout: "ready_epoch": 0, 2021-02-10T11:22:12.890 INFO:teuthology.orchestra.run.gibba002.stdout: "last_epoch_started": 0, 2021-02-10T11:22:12.890 INFO:teuthology.orchestra.run.gibba002.stdout: "last_epoch_clean": 0, 2021-02-10T11:22:12.890 INFO:teuthology.orchestra.run.gibba002.stdout: "source_version": "0'0", 2021-02-10T11:22:12.890 INFO:teuthology.orchestra.run.gibba002.stdout: "target_version": "0'0" 2021-02-10T11:22:12.890 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:22:12.890 INFO:teuthology.orchestra.run.gibba002.stdout: "last_change": "29", 2021-02-10T11:22:12.890 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend": "0", 2021-02-10T11:22:12.890 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:22:12.890 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:22:12.890 INFO:teuthology.orchestra.run.gibba002.stdout: "auid": 0, 2021-02-10T11:22:12.891 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:22:12.891 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_seq": 0, 2021-02-10T11:22:12.891 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_epoch": 0, 2021-02-10T11:22:12.891 INFO:teuthology.orchestra.run.gibba002.stdout: "pool_snaps": [], 2021-02-10T11:22:12.891 INFO:teuthology.orchestra.run.gibba002.stdout: "removed_snaps": "[]", 2021-02-10T11:22:12.891 INFO:teuthology.orchestra.run.gibba002.stdout: "quota_max_bytes": 0, 2021-02-10T11:22:12.891 INFO:teuthology.orchestra.run.gibba002.stdout: "quota_max_objects": 0, 2021-02-10T11:22:12.891 INFO:teuthology.orchestra.run.gibba002.stdout: "tiers": [], 2021-02-10T11:22:12.891 INFO:teuthology.orchestra.run.gibba002.stdout: "tier_of": -1, 2021-02-10T11:22:12.891 INFO:teuthology.orchestra.run.gibba002.stdout: "read_tier": -1, 2021-02-10T11:22:12.891 INFO:teuthology.orchestra.run.gibba002.stdout: "write_tier": -1, 2021-02-10T11:22:12.891 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_mode": "none", 2021-02-10T11:22:12.891 INFO:teuthology.orchestra.run.gibba002.stdout: "target_max_bytes": 0, 2021-02-10T11:22:12.891 INFO:teuthology.orchestra.run.gibba002.stdout: "target_max_objects": 0, 2021-02-10T11:22:12.892 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:22:12.892 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:22:12.892 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:22:12.892 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_min_flush_age": 0, 2021-02-10T11:22:12.892 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_min_evict_age": 0, 2021-02-10T11:22:12.892 INFO:teuthology.orchestra.run.gibba002.stdout: "erasure_code_profile": "", 2021-02-10T11:22:12.892 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_params": { 2021-02-10T11:22:12.892 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "none" 2021-02-10T11:22:12.892 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:22:12.892 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_period": 0, 2021-02-10T11:22:12.892 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_count": 0, 2021-02-10T11:22:12.892 INFO:teuthology.orchestra.run.gibba002.stdout: "use_gmt_hitset": true, 2021-02-10T11:22:12.892 INFO:teuthology.orchestra.run.gibba002.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:22:12.892 INFO:teuthology.orchestra.run.gibba002.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:22:12.893 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:22:12.893 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:22:12.893 INFO:teuthology.orchestra.run.gibba002.stdout: "grade_table": [], 2021-02-10T11:22:12.893 INFO:teuthology.orchestra.run.gibba002.stdout: "stripe_width": 0, 2021-02-10T11:22:12.893 INFO:teuthology.orchestra.run.gibba002.stdout: "expected_num_objects": 0, 2021-02-10T11:22:12.893 INFO:teuthology.orchestra.run.gibba002.stdout: "fast_read": false, 2021-02-10T11:22:12.893 INFO:teuthology.orchestra.run.gibba002.stdout: "options": { 2021-02-10T11:22:12.893 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_autoscale_bias": 4, 2021-02-10T11:22:12.893 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num_min": 16, 2021-02-10T11:22:12.893 INFO:teuthology.orchestra.run.gibba002.stdout: "recovery_priority": 5 2021-02-10T11:22:12.893 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:22:12.893 INFO:teuthology.orchestra.run.gibba002.stdout: "application_metadata": { 2021-02-10T11:22:12.893 INFO:teuthology.orchestra.run.gibba002.stdout: "cephfs": { 2021-02-10T11:22:12.894 INFO:teuthology.orchestra.run.gibba002.stdout: "metadata": "cephfs" 2021-02-10T11:22:12.894 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:22:12.894 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:22:12.894 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:22:12.894 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:22:12.894 INFO:teuthology.orchestra.run.gibba002.stdout: "pool": 5, 2021-02-10T11:22:12.894 INFO:teuthology.orchestra.run.gibba002.stdout: "pool_name": "cephfs_data", 2021-02-10T11:22:12.894 INFO:teuthology.orchestra.run.gibba002.stdout: "create_time": "2021-02-10 11:05:11.559128", 2021-02-10T11:22:12.894 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 1, 2021-02-10T11:22:12.894 INFO:teuthology.orchestra.run.gibba002.stdout: "flags_names": "hashpspool", 2021-02-10T11:22:12.894 INFO:teuthology.orchestra.run.gibba002.stdout: "type": 1, 2021-02-10T11:22:12.894 INFO:teuthology.orchestra.run.gibba002.stdout: "size": 2, 2021-02-10T11:22:12.894 INFO:teuthology.orchestra.run.gibba002.stdout: "min_size": 1, 2021-02-10T11:22:12.894 INFO:teuthology.orchestra.run.gibba002.stdout: "crush_rule": 0, 2021-02-10T11:22:12.895 INFO:teuthology.orchestra.run.gibba002.stdout: "object_hash": 2, 2021-02-10T11:22:12.895 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:22:12.895 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num": 8, 2021-02-10T11:22:12.895 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_placement_num": 8, 2021-02-10T11:22:12.895 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_placement_num_target": 8, 2021-02-10T11:22:12.895 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num_target": 8, 2021-02-10T11:22:12.895 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num_pending": 8, 2021-02-10T11:22:12.895 INFO:teuthology.orchestra.run.gibba002.stdout: "last_pg_merge_meta": { 2021-02-10T11:22:12.895 INFO:teuthology.orchestra.run.gibba002.stdout: "source_pgid": "0.0", 2021-02-10T11:22:12.895 INFO:teuthology.orchestra.run.gibba002.stdout: "ready_epoch": 0, 2021-02-10T11:22:12.895 INFO:teuthology.orchestra.run.gibba002.stdout: "last_epoch_started": 0, 2021-02-10T11:22:12.895 INFO:teuthology.orchestra.run.gibba002.stdout: "last_epoch_clean": 0, 2021-02-10T11:22:12.895 INFO:teuthology.orchestra.run.gibba002.stdout: "source_version": "0'0", 2021-02-10T11:22:12.896 INFO:teuthology.orchestra.run.gibba002.stdout: "target_version": "0'0" 2021-02-10T11:22:12.896 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:22:12.896 INFO:teuthology.orchestra.run.gibba002.stdout: "last_change": "30", 2021-02-10T11:22:12.896 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend": "0", 2021-02-10T11:22:12.896 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:22:12.896 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:22:12.896 INFO:teuthology.orchestra.run.gibba002.stdout: "auid": 0, 2021-02-10T11:22:12.896 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:22:12.896 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_seq": 0, 2021-02-10T11:22:12.896 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_epoch": 0, 2021-02-10T11:22:12.896 INFO:teuthology.orchestra.run.gibba002.stdout: "pool_snaps": [], 2021-02-10T11:22:12.896 INFO:teuthology.orchestra.run.gibba002.stdout: "removed_snaps": "[]", 2021-02-10T11:22:12.896 INFO:teuthology.orchestra.run.gibba002.stdout: "quota_max_bytes": 0, 2021-02-10T11:22:12.896 INFO:teuthology.orchestra.run.gibba002.stdout: "quota_max_objects": 0, 2021-02-10T11:22:12.897 INFO:teuthology.orchestra.run.gibba002.stdout: "tiers": [], 2021-02-10T11:22:12.897 INFO:teuthology.orchestra.run.gibba002.stdout: "tier_of": -1, 2021-02-10T11:22:12.897 INFO:teuthology.orchestra.run.gibba002.stdout: "read_tier": -1, 2021-02-10T11:22:12.897 INFO:teuthology.orchestra.run.gibba002.stdout: "write_tier": -1, 2021-02-10T11:22:12.897 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_mode": "none", 2021-02-10T11:22:12.897 INFO:teuthology.orchestra.run.gibba002.stdout: "target_max_bytes": 0, 2021-02-10T11:22:12.897 INFO:teuthology.orchestra.run.gibba002.stdout: "target_max_objects": 0, 2021-02-10T11:22:12.897 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:22:12.897 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:22:12.897 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:22:12.897 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_min_flush_age": 0, 2021-02-10T11:22:12.897 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_min_evict_age": 0, 2021-02-10T11:22:12.897 INFO:teuthology.orchestra.run.gibba002.stdout: "erasure_code_profile": "", 2021-02-10T11:22:12.897 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_params": { 2021-02-10T11:22:12.898 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "none" 2021-02-10T11:22:12.898 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:22:12.898 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_period": 0, 2021-02-10T11:22:12.898 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_count": 0, 2021-02-10T11:22:12.898 INFO:teuthology.orchestra.run.gibba002.stdout: "use_gmt_hitset": true, 2021-02-10T11:22:12.898 INFO:teuthology.orchestra.run.gibba002.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:22:12.898 INFO:teuthology.orchestra.run.gibba002.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:22:12.898 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:22:12.898 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:22:12.898 INFO:teuthology.orchestra.run.gibba002.stdout: "grade_table": [], 2021-02-10T11:22:12.898 INFO:teuthology.orchestra.run.gibba002.stdout: "stripe_width": 0, 2021-02-10T11:22:12.898 INFO:teuthology.orchestra.run.gibba002.stdout: "expected_num_objects": 0, 2021-02-10T11:22:12.898 INFO:teuthology.orchestra.run.gibba002.stdout: "fast_read": false, 2021-02-10T11:22:12.899 INFO:teuthology.orchestra.run.gibba002.stdout: "options": {}, 2021-02-10T11:22:12.899 INFO:teuthology.orchestra.run.gibba002.stdout: "application_metadata": { 2021-02-10T11:22:12.899 INFO:teuthology.orchestra.run.gibba002.stdout: "cephfs": { 2021-02-10T11:22:12.899 INFO:teuthology.orchestra.run.gibba002.stdout: "data": "cephfs" 2021-02-10T11:22:12.899 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:22:12.899 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:22:12.899 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:22:12.899 INFO:teuthology.orchestra.run.gibba002.stdout: ], 2021-02-10T11:22:12.899 INFO:teuthology.orchestra.run.gibba002.stdout: "osds": [ 2021-02-10T11:22:12.899 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:22:12.899 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 0, 2021-02-10T11:22:12.899 INFO:teuthology.orchestra.run.gibba002.stdout: "uuid": "be691b4d-a039-408a-af80-9d53f02b6f1f", 2021-02-10T11:22:12.899 INFO:teuthology.orchestra.run.gibba002.stdout: "up": 1, 2021-02-10T11:22:12.900 INFO:teuthology.orchestra.run.gibba002.stdout: "in": 1, 2021-02-10T11:22:12.900 INFO:teuthology.orchestra.run.gibba002.stdout: "weight": 1, 2021-02-10T11:22:12.900 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_affinity": 1, 2021-02-10T11:22:12.900 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_begin": 0, 2021-02-10T11:22:12.900 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_end": 0, 2021-02-10T11:22:12.900 INFO:teuthology.orchestra.run.gibba002.stdout: "up_from": 13, 2021-02-10T11:22:12.900 INFO:teuthology.orchestra.run.gibba002.stdout: "up_thru": 28, 2021-02-10T11:22:12.900 INFO:teuthology.orchestra.run.gibba002.stdout: "down_at": 0, 2021-02-10T11:22:12.900 INFO:teuthology.orchestra.run.gibba002.stdout: "lost_at": 0, 2021-02-10T11:22:12.900 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addrs": { 2021-02-10T11:22:12.900 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:22:12.900 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:22:12.900 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:22:12.901 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6818", 2021-02-10T11:22:12.901 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36088 2021-02-10T11:22:12.901 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:22:12.901 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:22:12.901 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:22:12.901 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6820", 2021-02-10T11:22:12.901 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36088 2021-02-10T11:22:12.901 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:22:12.901 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:22:12.901 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:22:12.901 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addrs": { 2021-02-10T11:22:12.901 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:22:12.901 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:22:12.902 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:22:12.902 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6822", 2021-02-10T11:22:12.902 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36088 2021-02-10T11:22:12.902 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:22:12.902 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:22:12.902 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:22:12.902 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6824", 2021-02-10T11:22:12.902 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36088 2021-02-10T11:22:12.902 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:22:12.902 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:22:12.902 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:22:12.902 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addrs": { 2021-02-10T11:22:12.902 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:22:12.903 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:22:12.903 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:22:12.903 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6830", 2021-02-10T11:22:12.903 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36088 2021-02-10T11:22:12.903 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:22:12.903 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:22:12.903 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:22:12.903 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6832", 2021-02-10T11:22:12.903 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36088 2021-02-10T11:22:12.903 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:22:12.903 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:22:12.903 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:22:12.903 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addrs": { 2021-02-10T11:22:12.904 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:22:12.904 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:22:12.904 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:22:12.904 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6826", 2021-02-10T11:22:12.904 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36088 2021-02-10T11:22:12.904 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:22:12.904 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:22:12.904 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:22:12.904 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6828", 2021-02-10T11:22:12.904 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36088 2021-02-10T11:22:12.904 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:22:12.904 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:22:12.904 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:22:12.904 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addr": "172.21.2.102:6820/36088", 2021-02-10T11:22:12.905 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addr": "172.21.2.102:6824/36088", 2021-02-10T11:22:12.905 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addr": "172.21.2.102:6832/36088", 2021-02-10T11:22:12.905 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addr": "172.21.2.102:6828/36088", 2021-02-10T11:22:12.905 INFO:teuthology.orchestra.run.gibba002.stdout: "state": [ 2021-02-10T11:22:12.905 INFO:teuthology.orchestra.run.gibba002.stdout: "exists", 2021-02-10T11:22:12.905 INFO:teuthology.orchestra.run.gibba002.stdout: "up" 2021-02-10T11:22:12.905 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:22:12.905 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:22:12.905 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:22:12.905 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 1, 2021-02-10T11:22:12.905 INFO:teuthology.orchestra.run.gibba002.stdout: "uuid": "64e3ccc8-9c69-4d7c-865e-af8e83822635", 2021-02-10T11:22:12.905 INFO:teuthology.orchestra.run.gibba002.stdout: "up": 1, 2021-02-10T11:22:12.905 INFO:teuthology.orchestra.run.gibba002.stdout: "in": 1, 2021-02-10T11:22:12.906 INFO:teuthology.orchestra.run.gibba002.stdout: "weight": 1, 2021-02-10T11:22:12.906 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_affinity": 1, 2021-02-10T11:22:12.906 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_begin": 0, 2021-02-10T11:22:12.906 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_end": 0, 2021-02-10T11:22:12.906 INFO:teuthology.orchestra.run.gibba002.stdout: "up_from": 13, 2021-02-10T11:22:12.906 INFO:teuthology.orchestra.run.gibba002.stdout: "up_thru": 27, 2021-02-10T11:22:12.906 INFO:teuthology.orchestra.run.gibba002.stdout: "down_at": 0, 2021-02-10T11:22:12.906 INFO:teuthology.orchestra.run.gibba002.stdout: "lost_at": 0, 2021-02-10T11:22:12.906 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addrs": { 2021-02-10T11:22:12.906 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:22:12.906 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:22:12.906 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:22:12.906 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6819", 2021-02-10T11:22:12.907 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36087 2021-02-10T11:22:12.907 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:22:12.907 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:22:12.907 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:22:12.907 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6821", 2021-02-10T11:22:12.907 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36087 2021-02-10T11:22:12.907 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:22:12.907 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:22:12.907 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:22:12.907 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addrs": { 2021-02-10T11:22:12.907 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:22:12.907 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:22:12.907 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:22:12.907 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6823", 2021-02-10T11:22:12.908 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36087 2021-02-10T11:22:12.908 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:22:12.908 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:22:12.908 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:22:12.908 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6825", 2021-02-10T11:22:12.908 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36087 2021-02-10T11:22:12.908 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:22:12.908 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:22:12.908 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:22:12.908 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addrs": { 2021-02-10T11:22:12.908 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:22:12.908 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:22:12.908 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:22:12.908 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6831", 2021-02-10T11:22:12.909 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36087 2021-02-10T11:22:12.909 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:22:12.909 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:22:12.909 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:22:12.909 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6833", 2021-02-10T11:22:12.909 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36087 2021-02-10T11:22:12.909 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:22:12.909 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:22:12.909 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:22:12.909 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addrs": { 2021-02-10T11:22:12.909 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:22:12.909 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:22:12.909 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:22:12.909 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6827", 2021-02-10T11:22:12.910 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36087 2021-02-10T11:22:12.910 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:22:12.910 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:22:12.910 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:22:12.910 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6829", 2021-02-10T11:22:12.910 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36087 2021-02-10T11:22:12.910 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:22:12.910 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:22:12.910 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:22:12.910 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addr": "172.21.2.102:6821/36087", 2021-02-10T11:22:12.910 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addr": "172.21.2.102:6825/36087", 2021-02-10T11:22:12.910 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addr": "172.21.2.102:6833/36087", 2021-02-10T11:22:12.910 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addr": "172.21.2.102:6829/36087", 2021-02-10T11:22:12.911 INFO:teuthology.orchestra.run.gibba002.stdout: "state": [ 2021-02-10T11:22:12.911 INFO:teuthology.orchestra.run.gibba002.stdout: "exists", 2021-02-10T11:22:12.911 INFO:teuthology.orchestra.run.gibba002.stdout: "up" 2021-02-10T11:22:12.911 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:22:12.911 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:22:12.911 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:22:12.911 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 2, 2021-02-10T11:22:12.911 INFO:teuthology.orchestra.run.gibba002.stdout: "uuid": "153700af-747f-481f-bdf4-d6072d3a842f", 2021-02-10T11:22:12.911 INFO:teuthology.orchestra.run.gibba002.stdout: "up": 1, 2021-02-10T11:22:12.911 INFO:teuthology.orchestra.run.gibba002.stdout: "in": 1, 2021-02-10T11:22:12.911 INFO:teuthology.orchestra.run.gibba002.stdout: "weight": 1, 2021-02-10T11:22:12.911 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_affinity": 1, 2021-02-10T11:22:12.912 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_begin": 0, 2021-02-10T11:22:12.912 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_end": 0, 2021-02-10T11:22:12.912 INFO:teuthology.orchestra.run.gibba002.stdout: "up_from": 13, 2021-02-10T11:22:12.912 INFO:teuthology.orchestra.run.gibba002.stdout: "up_thru": 28, 2021-02-10T11:22:12.912 INFO:teuthology.orchestra.run.gibba002.stdout: "down_at": 0, 2021-02-10T11:22:12.912 INFO:teuthology.orchestra.run.gibba002.stdout: "lost_at": 0, 2021-02-10T11:22:12.912 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addrs": { 2021-02-10T11:22:12.912 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:22:12.912 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:22:12.912 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:22:12.912 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6810", 2021-02-10T11:22:12.912 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36089 2021-02-10T11:22:12.912 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:22:12.912 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:22:12.913 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:22:12.913 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6811", 2021-02-10T11:22:12.913 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36089 2021-02-10T11:22:12.913 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:22:12.913 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:22:12.913 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:22:12.913 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addrs": { 2021-02-10T11:22:12.913 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:22:12.913 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:22:12.913 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:22:12.913 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6812", 2021-02-10T11:22:12.913 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36089 2021-02-10T11:22:12.913 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:22:12.914 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:22:12.914 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:22:12.914 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6813", 2021-02-10T11:22:12.914 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36089 2021-02-10T11:22:12.914 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:22:12.914 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:22:12.914 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:22:12.914 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addrs": { 2021-02-10T11:22:12.914 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:22:12.914 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:22:12.914 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:22:12.914 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6816", 2021-02-10T11:22:12.914 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36089 2021-02-10T11:22:12.915 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:22:12.915 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:22:12.915 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:22:12.915 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6817", 2021-02-10T11:22:12.915 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36089 2021-02-10T11:22:12.915 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:22:12.915 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:22:12.915 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:22:12.915 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addrs": { 2021-02-10T11:22:12.915 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:22:12.915 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:22:12.915 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:22:12.915 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6814", 2021-02-10T11:22:12.915 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36089 2021-02-10T11:22:12.916 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:22:12.916 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:22:12.916 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:22:12.916 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6815", 2021-02-10T11:22:12.916 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36089 2021-02-10T11:22:12.916 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:22:12.916 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:22:12.916 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:22:12.916 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addr": "172.21.2.102:6811/36089", 2021-02-10T11:22:12.916 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addr": "172.21.2.102:6813/36089", 2021-02-10T11:22:12.916 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addr": "172.21.2.102:6817/36089", 2021-02-10T11:22:12.916 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addr": "172.21.2.102:6815/36089", 2021-02-10T11:22:12.916 INFO:teuthology.orchestra.run.gibba002.stdout: "state": [ 2021-02-10T11:22:12.917 INFO:teuthology.orchestra.run.gibba002.stdout: "exists", 2021-02-10T11:22:12.917 INFO:teuthology.orchestra.run.gibba002.stdout: "up" 2021-02-10T11:22:12.917 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:22:12.917 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:22:12.917 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:22:12.917 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 3, 2021-02-10T11:22:12.917 INFO:teuthology.orchestra.run.gibba002.stdout: "uuid": "cbc678f4-83a9-45e6-8830-2f7c232b1ea7", 2021-02-10T11:22:12.917 INFO:teuthology.orchestra.run.gibba002.stdout: "up": 1, 2021-02-10T11:22:12.917 INFO:teuthology.orchestra.run.gibba002.stdout: "in": 1, 2021-02-10T11:22:12.917 INFO:teuthology.orchestra.run.gibba002.stdout: "weight": 1, 2021-02-10T11:22:12.917 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_affinity": 1, 2021-02-10T11:22:12.917 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_begin": 0, 2021-02-10T11:22:12.917 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_end": 0, 2021-02-10T11:22:12.918 INFO:teuthology.orchestra.run.gibba002.stdout: "up_from": 13, 2021-02-10T11:22:12.918 INFO:teuthology.orchestra.run.gibba002.stdout: "up_thru": 28, 2021-02-10T11:22:12.918 INFO:teuthology.orchestra.run.gibba002.stdout: "down_at": 0, 2021-02-10T11:22:12.918 INFO:teuthology.orchestra.run.gibba002.stdout: "lost_at": 0, 2021-02-10T11:22:12.918 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addrs": { 2021-02-10T11:22:12.918 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:22:12.918 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:22:12.918 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:22:12.918 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6802", 2021-02-10T11:22:12.918 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36090 2021-02-10T11:22:12.918 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:22:12.918 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:22:12.918 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:22:12.918 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6803", 2021-02-10T11:22:12.919 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36090 2021-02-10T11:22:12.920 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:22:12.920 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:22:12.920 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:22:12.920 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addrs": { 2021-02-10T11:22:12.920 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:22:12.920 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:22:12.921 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:22:12.921 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6804", 2021-02-10T11:22:12.921 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36090 2021-02-10T11:22:12.921 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:22:12.921 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:22:12.921 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:22:12.921 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6805", 2021-02-10T11:22:12.921 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36090 2021-02-10T11:22:12.921 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:22:12.921 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:22:12.921 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:22:12.921 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addrs": { 2021-02-10T11:22:12.922 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:22:12.922 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:22:12.922 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:22:12.922 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6808", 2021-02-10T11:22:12.922 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36090 2021-02-10T11:22:12.922 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:22:12.922 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:22:12.922 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:22:12.922 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6809", 2021-02-10T11:22:12.922 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36090 2021-02-10T11:22:12.922 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:22:12.922 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:22:12.922 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:22:12.922 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addrs": { 2021-02-10T11:22:12.923 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:22:12.923 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:22:12.923 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:22:12.923 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6806", 2021-02-10T11:22:12.923 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36090 2021-02-10T11:22:12.923 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:22:12.923 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:22:12.923 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:22:12.923 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6807", 2021-02-10T11:22:12.923 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36090 2021-02-10T11:22:12.923 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:22:12.923 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:22:12.923 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:22:12.923 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addr": "172.21.2.102:6803/36090", 2021-02-10T11:22:12.924 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addr": "172.21.2.102:6805/36090", 2021-02-10T11:22:12.924 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addr": "172.21.2.102:6809/36090", 2021-02-10T11:22:12.924 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addr": "172.21.2.102:6807/36090", 2021-02-10T11:22:12.924 INFO:teuthology.orchestra.run.gibba002.stdout: "state": [ 2021-02-10T11:22:12.924 INFO:teuthology.orchestra.run.gibba002.stdout: "exists", 2021-02-10T11:22:12.924 INFO:teuthology.orchestra.run.gibba002.stdout: "up" 2021-02-10T11:22:12.924 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:22:12.924 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:22:12.924 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:22:12.924 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 4, 2021-02-10T11:22:12.924 INFO:teuthology.orchestra.run.gibba002.stdout: "uuid": "c01e4008-b258-4f10-afd2-9e7f37214351", 2021-02-10T11:22:12.924 INFO:teuthology.orchestra.run.gibba002.stdout: "up": 1, 2021-02-10T11:22:12.924 INFO:teuthology.orchestra.run.gibba002.stdout: "in": 1, 2021-02-10T11:22:12.925 INFO:teuthology.orchestra.run.gibba002.stdout: "weight": 1, 2021-02-10T11:22:12.925 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_affinity": 1, 2021-02-10T11:22:12.925 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_begin": 0, 2021-02-10T11:22:12.925 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_end": 0, 2021-02-10T11:22:12.925 INFO:teuthology.orchestra.run.gibba002.stdout: "up_from": 13, 2021-02-10T11:22:12.925 INFO:teuthology.orchestra.run.gibba002.stdout: "up_thru": 28, 2021-02-10T11:22:12.925 INFO:teuthology.orchestra.run.gibba002.stdout: "down_at": 0, 2021-02-10T11:22:12.925 INFO:teuthology.orchestra.run.gibba002.stdout: "lost_at": 0, 2021-02-10T11:22:12.925 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addrs": { 2021-02-10T11:22:12.925 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:22:12.925 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:22:12.925 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:22:12.925 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6808", 2021-02-10T11:22:12.926 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36368 2021-02-10T11:22:12.926 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:22:12.926 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:22:12.926 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:22:12.926 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6811", 2021-02-10T11:22:12.926 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36368 2021-02-10T11:22:12.926 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:22:12.926 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:22:12.926 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:22:12.926 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addrs": { 2021-02-10T11:22:12.926 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:22:12.926 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:22:12.926 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:22:12.926 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6814", 2021-02-10T11:22:12.927 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36368 2021-02-10T11:22:12.927 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:22:12.927 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:22:12.927 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:22:12.927 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6818", 2021-02-10T11:22:12.927 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36368 2021-02-10T11:22:12.927 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:22:12.927 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:22:12.927 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:22:12.927 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addrs": { 2021-02-10T11:22:12.927 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:22:12.927 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:22:12.927 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:22:12.928 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6826", 2021-02-10T11:22:12.928 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36368 2021-02-10T11:22:12.928 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:22:12.928 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:22:12.928 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:22:12.928 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6828", 2021-02-10T11:22:12.928 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36368 2021-02-10T11:22:12.928 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:22:12.928 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:22:12.928 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:22:12.928 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addrs": { 2021-02-10T11:22:12.928 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:22:12.928 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:22:12.928 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:22:12.929 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6820", 2021-02-10T11:22:12.929 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36368 2021-02-10T11:22:12.929 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:22:12.929 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:22:12.929 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:22:12.929 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6824", 2021-02-10T11:22:12.929 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36368 2021-02-10T11:22:12.929 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:22:12.929 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:22:12.929 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:22:12.929 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addr": "172.21.2.113:6811/36368", 2021-02-10T11:22:12.929 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addr": "172.21.2.113:6818/36368", 2021-02-10T11:22:12.929 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addr": "172.21.2.113:6828/36368", 2021-02-10T11:22:12.929 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addr": "172.21.2.113:6824/36368", 2021-02-10T11:22:12.930 INFO:teuthology.orchestra.run.gibba002.stdout: "state": [ 2021-02-10T11:22:12.930 INFO:teuthology.orchestra.run.gibba002.stdout: "exists", 2021-02-10T11:22:12.930 INFO:teuthology.orchestra.run.gibba002.stdout: "up" 2021-02-10T11:22:12.930 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:22:12.930 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:22:12.930 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:22:12.930 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 5, 2021-02-10T11:22:12.930 INFO:teuthology.orchestra.run.gibba002.stdout: "uuid": "551f6007-daf8-4a98-8ee0-c2b16bcccb5e", 2021-02-10T11:22:12.930 INFO:teuthology.orchestra.run.gibba002.stdout: "up": 1, 2021-02-10T11:22:12.930 INFO:teuthology.orchestra.run.gibba002.stdout: "in": 1, 2021-02-10T11:22:12.930 INFO:teuthology.orchestra.run.gibba002.stdout: "weight": 1, 2021-02-10T11:22:12.930 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_affinity": 1, 2021-02-10T11:22:12.930 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_begin": 0, 2021-02-10T11:22:12.931 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_end": 0, 2021-02-10T11:22:12.931 INFO:teuthology.orchestra.run.gibba002.stdout: "up_from": 13, 2021-02-10T11:22:12.931 INFO:teuthology.orchestra.run.gibba002.stdout: "up_thru": 28, 2021-02-10T11:22:12.931 INFO:teuthology.orchestra.run.gibba002.stdout: "down_at": 0, 2021-02-10T11:22:12.931 INFO:teuthology.orchestra.run.gibba002.stdout: "lost_at": 0, 2021-02-10T11:22:12.931 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addrs": { 2021-02-10T11:22:12.931 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:22:12.931 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:22:12.931 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:22:12.931 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6802", 2021-02-10T11:22:12.931 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36366 2021-02-10T11:22:12.931 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:22:12.931 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:22:12.932 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:22:12.932 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6805", 2021-02-10T11:22:12.932 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36366 2021-02-10T11:22:12.932 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:22:12.932 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:22:12.932 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:22:12.932 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addrs": { 2021-02-10T11:22:12.932 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:22:12.932 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:22:12.932 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:22:12.932 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6807", 2021-02-10T11:22:12.932 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36366 2021-02-10T11:22:12.932 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:22:12.933 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:22:12.933 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:22:12.933 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6810", 2021-02-10T11:22:12.933 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36366 2021-02-10T11:22:12.933 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:22:12.933 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:22:12.933 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:22:12.933 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addrs": { 2021-02-10T11:22:12.933 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:22:12.933 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:22:12.933 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:22:12.933 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6819", 2021-02-10T11:22:12.933 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36366 2021-02-10T11:22:12.933 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:22:12.934 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:22:12.934 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:22:12.934 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6822", 2021-02-10T11:22:12.934 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36366 2021-02-10T11:22:12.934 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:22:12.934 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:22:12.934 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:22:12.934 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addrs": { 2021-02-10T11:22:12.934 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:22:12.934 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:22:12.934 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:22:12.934 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6812", 2021-02-10T11:22:12.935 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36366 2021-02-10T11:22:12.935 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:22:12.935 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:22:12.935 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:22:12.935 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6815", 2021-02-10T11:22:12.935 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36366 2021-02-10T11:22:12.935 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:22:12.935 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:22:12.935 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:22:12.935 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addr": "172.21.2.113:6805/36366", 2021-02-10T11:22:12.935 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addr": "172.21.2.113:6810/36366", 2021-02-10T11:22:12.935 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addr": "172.21.2.113:6822/36366", 2021-02-10T11:22:12.935 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addr": "172.21.2.113:6815/36366", 2021-02-10T11:22:12.935 INFO:teuthology.orchestra.run.gibba002.stdout: "state": [ 2021-02-10T11:22:12.935 INFO:teuthology.orchestra.run.gibba002.stdout: "exists", 2021-02-10T11:22:12.936 INFO:teuthology.orchestra.run.gibba002.stdout: "up" 2021-02-10T11:22:12.936 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:22:12.936 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:22:12.936 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:22:12.936 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 6, 2021-02-10T11:22:12.936 INFO:teuthology.orchestra.run.gibba002.stdout: "uuid": "bf67c040-0e50-48be-b102-c3943bc9ebef", 2021-02-10T11:22:12.936 INFO:teuthology.orchestra.run.gibba002.stdout: "up": 1, 2021-02-10T11:22:12.936 INFO:teuthology.orchestra.run.gibba002.stdout: "in": 1, 2021-02-10T11:22:12.936 INFO:teuthology.orchestra.run.gibba002.stdout: "weight": 1, 2021-02-10T11:22:12.936 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_affinity": 1, 2021-02-10T11:22:12.936 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_begin": 0, 2021-02-10T11:22:12.936 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_end": 0, 2021-02-10T11:22:12.936 INFO:teuthology.orchestra.run.gibba002.stdout: "up_from": 13, 2021-02-10T11:22:12.937 INFO:teuthology.orchestra.run.gibba002.stdout: "up_thru": 28, 2021-02-10T11:22:12.937 INFO:teuthology.orchestra.run.gibba002.stdout: "down_at": 0, 2021-02-10T11:22:12.937 INFO:teuthology.orchestra.run.gibba002.stdout: "lost_at": 0, 2021-02-10T11:22:12.937 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addrs": { 2021-02-10T11:22:12.937 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:22:12.937 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:22:12.937 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:22:12.937 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6800", 2021-02-10T11:22:12.937 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36367 2021-02-10T11:22:12.937 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:22:12.937 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:22:12.937 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:22:12.937 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6801", 2021-02-10T11:22:12.938 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36367 2021-02-10T11:22:12.938 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:22:12.938 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:22:12.938 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:22:12.938 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addrs": { 2021-02-10T11:22:12.938 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:22:12.938 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:22:12.938 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:22:12.938 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6803", 2021-02-10T11:22:12.938 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36367 2021-02-10T11:22:12.938 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:22:12.938 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:22:12.938 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:22:12.938 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6804", 2021-02-10T11:22:12.939 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36367 2021-02-10T11:22:12.939 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:22:12.939 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:22:12.939 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:22:12.939 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addrs": { 2021-02-10T11:22:12.939 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:22:12.939 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:22:12.939 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:22:12.939 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6813", 2021-02-10T11:22:12.939 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36367 2021-02-10T11:22:12.939 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:22:12.939 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:22:12.939 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:22:12.939 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6816", 2021-02-10T11:22:12.940 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36367 2021-02-10T11:22:12.940 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:22:12.940 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:22:12.940 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:22:12.940 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addrs": { 2021-02-10T11:22:12.940 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:22:12.940 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:22:12.940 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:22:12.940 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6806", 2021-02-10T11:22:12.940 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36367 2021-02-10T11:22:12.940 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:22:12.940 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:22:12.940 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:22:12.940 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6809", 2021-02-10T11:22:12.941 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36367 2021-02-10T11:22:12.941 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:22:12.941 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:22:12.941 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:22:12.941 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addr": "172.21.2.113:6801/36367", 2021-02-10T11:22:12.941 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addr": "172.21.2.113:6804/36367", 2021-02-10T11:22:12.941 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addr": "172.21.2.113:6816/36367", 2021-02-10T11:22:12.941 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addr": "172.21.2.113:6809/36367", 2021-02-10T11:22:12.941 INFO:teuthology.orchestra.run.gibba002.stdout: "state": [ 2021-02-10T11:22:12.941 INFO:teuthology.orchestra.run.gibba002.stdout: "exists", 2021-02-10T11:22:12.941 INFO:teuthology.orchestra.run.gibba002.stdout: "up" 2021-02-10T11:22:12.941 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:22:12.941 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:22:12.941 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:22:12.942 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 7, 2021-02-10T11:22:12.942 INFO:teuthology.orchestra.run.gibba002.stdout: "uuid": "f5461f80-018d-428f-be89-2eb2bcbe7bb6", 2021-02-10T11:22:12.942 INFO:teuthology.orchestra.run.gibba002.stdout: "up": 1, 2021-02-10T11:22:12.942 INFO:teuthology.orchestra.run.gibba002.stdout: "in": 1, 2021-02-10T11:22:12.942 INFO:teuthology.orchestra.run.gibba002.stdout: "weight": 1, 2021-02-10T11:22:12.942 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_affinity": 1, 2021-02-10T11:22:12.942 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_begin": 0, 2021-02-10T11:22:12.942 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_end": 0, 2021-02-10T11:22:12.942 INFO:teuthology.orchestra.run.gibba002.stdout: "up_from": 13, 2021-02-10T11:22:12.942 INFO:teuthology.orchestra.run.gibba002.stdout: "up_thru": 28, 2021-02-10T11:22:12.942 INFO:teuthology.orchestra.run.gibba002.stdout: "down_at": 0, 2021-02-10T11:22:12.942 INFO:teuthology.orchestra.run.gibba002.stdout: "lost_at": 0, 2021-02-10T11:22:12.943 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addrs": { 2021-02-10T11:22:12.943 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:22:12.943 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:22:12.943 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:22:12.943 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6817", 2021-02-10T11:22:12.943 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36365 2021-02-10T11:22:12.943 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:22:12.943 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:22:12.943 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:22:12.943 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6821", 2021-02-10T11:22:12.943 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36365 2021-02-10T11:22:12.943 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:22:12.943 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:22:12.943 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:22:12.944 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addrs": { 2021-02-10T11:22:12.944 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:22:12.944 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:22:12.944 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:22:12.944 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6823", 2021-02-10T11:22:12.944 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36365 2021-02-10T11:22:12.944 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:22:12.944 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:22:12.944 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:22:12.944 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6825", 2021-02-10T11:22:12.944 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36365 2021-02-10T11:22:12.944 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:22:12.944 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:22:12.944 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:22:12.945 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addrs": { 2021-02-10T11:22:12.945 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:22:12.945 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:22:12.945 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:22:12.945 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6830", 2021-02-10T11:22:12.945 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36365 2021-02-10T11:22:12.945 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:22:12.945 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:22:12.945 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:22:12.945 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6831", 2021-02-10T11:22:12.945 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36365 2021-02-10T11:22:12.945 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:22:12.945 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:22:12.945 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:22:12.946 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addrs": { 2021-02-10T11:22:12.946 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:22:12.946 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:22:12.946 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:22:12.946 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6827", 2021-02-10T11:22:12.946 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36365 2021-02-10T11:22:12.946 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:22:12.946 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:22:12.946 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:22:12.946 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6829", 2021-02-10T11:22:12.946 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36365 2021-02-10T11:22:12.946 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:22:12.947 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:22:12.947 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:22:12.947 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addr": "172.21.2.113:6821/36365", 2021-02-10T11:22:12.947 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addr": "172.21.2.113:6825/36365", 2021-02-10T11:22:12.947 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addr": "172.21.2.113:6831/36365", 2021-02-10T11:22:12.947 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addr": "172.21.2.113:6829/36365", 2021-02-10T11:22:12.947 INFO:teuthology.orchestra.run.gibba002.stdout: "state": [ 2021-02-10T11:22:12.947 INFO:teuthology.orchestra.run.gibba002.stdout: "exists", 2021-02-10T11:22:12.947 INFO:teuthology.orchestra.run.gibba002.stdout: "up" 2021-02-10T11:22:12.947 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:22:12.947 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:22:12.947 INFO:teuthology.orchestra.run.gibba002.stdout: ], 2021-02-10T11:22:12.947 INFO:teuthology.orchestra.run.gibba002.stdout: "osd_xinfo": [ 2021-02-10T11:22:12.947 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:22:12.948 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 0, 2021-02-10T11:22:12.948 INFO:teuthology.orchestra.run.gibba002.stdout: "down_stamp": "0.000000", 2021-02-10T11:22:12.948 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_probability": 0, 2021-02-10T11:22:12.948 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_interval": 0, 2021-02-10T11:22:12.948 INFO:teuthology.orchestra.run.gibba002.stdout: "features": 4611087854035861503, 2021-02-10T11:22:12.948 INFO:teuthology.orchestra.run.gibba002.stdout: "old_weight": 0 2021-02-10T11:22:12.948 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:22:12.948 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:22:12.948 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 1, 2021-02-10T11:22:12.948 INFO:teuthology.orchestra.run.gibba002.stdout: "down_stamp": "0.000000", 2021-02-10T11:22:12.948 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_probability": 0, 2021-02-10T11:22:12.948 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_interval": 0, 2021-02-10T11:22:12.948 INFO:teuthology.orchestra.run.gibba002.stdout: "features": 4611087854035861503, 2021-02-10T11:22:12.949 INFO:teuthology.orchestra.run.gibba002.stdout: "old_weight": 0 2021-02-10T11:22:12.949 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:22:12.949 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:22:12.949 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 2, 2021-02-10T11:22:12.949 INFO:teuthology.orchestra.run.gibba002.stdout: "down_stamp": "0.000000", 2021-02-10T11:22:12.949 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_probability": 0, 2021-02-10T11:22:12.949 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_interval": 0, 2021-02-10T11:22:12.949 INFO:teuthology.orchestra.run.gibba002.stdout: "features": 4611087854035861503, 2021-02-10T11:22:12.949 INFO:teuthology.orchestra.run.gibba002.stdout: "old_weight": 0 2021-02-10T11:22:12.949 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:22:12.949 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:22:12.949 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 3, 2021-02-10T11:22:12.949 INFO:teuthology.orchestra.run.gibba002.stdout: "down_stamp": "0.000000", 2021-02-10T11:22:12.949 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_probability": 0, 2021-02-10T11:22:12.949 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_interval": 0, 2021-02-10T11:22:12.950 INFO:teuthology.orchestra.run.gibba002.stdout: "features": 4611087854035861503, 2021-02-10T11:22:12.950 INFO:teuthology.orchestra.run.gibba002.stdout: "old_weight": 0 2021-02-10T11:22:12.950 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:22:12.950 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:22:12.950 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 4, 2021-02-10T11:22:12.950 INFO:teuthology.orchestra.run.gibba002.stdout: "down_stamp": "0.000000", 2021-02-10T11:22:12.950 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_probability": 0, 2021-02-10T11:22:12.950 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_interval": 0, 2021-02-10T11:22:12.950 INFO:teuthology.orchestra.run.gibba002.stdout: "features": 4611087854035861503, 2021-02-10T11:22:12.950 INFO:teuthology.orchestra.run.gibba002.stdout: "old_weight": 0 2021-02-10T11:22:12.950 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:22:12.950 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:22:12.950 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 5, 2021-02-10T11:22:12.951 INFO:teuthology.orchestra.run.gibba002.stdout: "down_stamp": "0.000000", 2021-02-10T11:22:12.951 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_probability": 0, 2021-02-10T11:22:12.951 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_interval": 0, 2021-02-10T11:22:12.951 INFO:teuthology.orchestra.run.gibba002.stdout: "features": 4611087854035861503, 2021-02-10T11:22:12.951 INFO:teuthology.orchestra.run.gibba002.stdout: "old_weight": 0 2021-02-10T11:22:12.951 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:22:12.951 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:22:12.951 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 6, 2021-02-10T11:22:12.951 INFO:teuthology.orchestra.run.gibba002.stdout: "down_stamp": "0.000000", 2021-02-10T11:22:12.951 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_probability": 0, 2021-02-10T11:22:12.951 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_interval": 0, 2021-02-10T11:22:12.951 INFO:teuthology.orchestra.run.gibba002.stdout: "features": 4611087854035861503, 2021-02-10T11:22:12.951 INFO:teuthology.orchestra.run.gibba002.stdout: "old_weight": 0 2021-02-10T11:22:12.952 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:22:12.952 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:22:12.952 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 7, 2021-02-10T11:22:12.952 INFO:teuthology.orchestra.run.gibba002.stdout: "down_stamp": "0.000000", 2021-02-10T11:22:12.952 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_probability": 0, 2021-02-10T11:22:12.952 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_interval": 0, 2021-02-10T11:22:12.952 INFO:teuthology.orchestra.run.gibba002.stdout: "features": 4611087854035861503, 2021-02-10T11:22:12.952 INFO:teuthology.orchestra.run.gibba002.stdout: "old_weight": 0 2021-02-10T11:22:12.952 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:22:12.952 INFO:teuthology.orchestra.run.gibba002.stdout: ], 2021-02-10T11:22:12.952 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_upmap": [], 2021-02-10T11:22:12.952 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_upmap_items": [], 2021-02-10T11:22:12.952 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_temp": [], 2021-02-10T11:22:12.953 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_temp": [], 2021-02-10T11:22:12.953 INFO:teuthology.orchestra.run.gibba002.stdout: "blacklist": {}, 2021-02-10T11:22:12.953 INFO:teuthology.orchestra.run.gibba002.stdout: "erasure_code_profiles": { 2021-02-10T11:22:12.953 INFO:teuthology.orchestra.run.gibba002.stdout: "default": { 2021-02-10T11:22:12.953 INFO:teuthology.orchestra.run.gibba002.stdout: "crush-failure-domain": "osd", 2021-02-10T11:22:12.953 INFO:teuthology.orchestra.run.gibba002.stdout: "k": "2", 2021-02-10T11:22:12.953 INFO:teuthology.orchestra.run.gibba002.stdout: "m": "1", 2021-02-10T11:22:12.953 INFO:teuthology.orchestra.run.gibba002.stdout: "plugin": "jerasure", 2021-02-10T11:22:12.953 INFO:teuthology.orchestra.run.gibba002.stdout: "ruleset-failure-domain": "osd", 2021-02-10T11:22:12.953 INFO:teuthology.orchestra.run.gibba002.stdout: "technique": "reed_sol_van" 2021-02-10T11:22:12.953 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:22:12.953 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:22:12.953 INFO:teuthology.orchestra.run.gibba002.stdout: "removed_snaps_queue": [], 2021-02-10T11:22:12.953 INFO:teuthology.orchestra.run.gibba002.stdout: "new_removed_snaps": [], 2021-02-10T11:22:12.954 INFO:teuthology.orchestra.run.gibba002.stdout: "new_purged_snaps": [], 2021-02-10T11:22:12.954 INFO:teuthology.orchestra.run.gibba002.stdout: "crush_node_flags": {}, 2021-02-10T11:22:12.954 INFO:teuthology.orchestra.run.gibba002.stdout: "device_class_flags": {} 2021-02-10T11:22:12.954 INFO:teuthology.orchestra.run.gibba002.stdout:} 2021-02-10T11:22:12.954 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:12.885 7f0a9d2b7700 1 -- 172.21.2.102:0/861306313 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0a7c0186c0 msgr2=0x7f0a7c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:12.954 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:12.885 7f0a9d2b7700 1 --2- 172.21.2.102:0/861306313 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0a7c0186c0 0x7f0a7c01abc0 crc :-1 s=READY pgs=496 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:12.954 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:12.885 7f0a9d2b7700 1 -- 172.21.2.102:0/861306313 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0a980ae9c0 msgr2=0x7f0a981a2870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:12.954 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:12.885 7f0a9d2b7700 1 --2- 172.21.2.102:0/861306313 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0a980ae9c0 0x7f0a981a2870 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 rx=0x7f0a90009940 tx=0x7f0a9000bcf0).stop 2021-02-10T11:22:12.954 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:12.885 7f0a9d2b7700 1 -- 172.21.2.102:0/861306313 shutdown_connections 2021-02-10T11:22:12.954 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:12.885 7f0a9d2b7700 1 --2- 172.21.2.102:0/861306313 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0a980ae9c0 0x7f0a981a2870 unknown :-1 s=CLOSED pgs=1020 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:12.954 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:12.885 7f0a9d2b7700 1 --2- 172.21.2.102:0/861306313 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0a7c0186c0 0x7f0a7c01abc0 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:12.955 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:12.885 7f0a9d2b7700 1 --2- 172.21.2.102:0/861306313 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0a9808d5a0 0x7f0a980ae3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:12.955 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:12.886 7f0a9d2b7700 1 --2- 172.21.2.102:0/861306313 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0a98097250 0x7f0a981a1b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:12.955 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:12.886 7f0a9d2b7700 1 -- 172.21.2.102:0/861306313 >> 172.21.2.102:0/861306313 conn(0x7f0a981ac620 msgr2=0x7f0a980896e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:12.955 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:12.886 7f0a9d2b7700 1 -- 172.21.2.102:0/861306313 shutdown_connections 2021-02-10T11:22:12.955 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:12.886 7f0a9d2b7700 1 -- 172.21.2.102:0/861306313 wait complete. 2021-02-10T11:22:12.955 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:22:13.385 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:13.370 7fb76bbc7700 1 Processor -- start 2021-02-10T11:22:13.385 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:13.370 7fb76bbc7700 1 -- start start 2021-02-10T11:22:13.386 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:13.371 7fb76bbc7700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fb76407e6a0 0x7fb7640d1790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:13.386 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:13.371 7fb76bbc7700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb76407ed80 con 0x7fb7640e5470 2021-02-10T11:22:13.386 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:13.371 7fb76bbc7700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb76407eaa0 con 0x7fb7640e5070 2021-02-10T11:22:13.386 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:13.371 7fb76bbc7700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7641b4080 con 0x7fb76407e6a0 2021-02-10T11:22:13.386 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:13.371 7fb7691e2700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fb7640e5470 0x7fb76407f300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:47426/0 (socket says 172.21.2.102:47426) 2021-02-10T11:22:13.386 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:13.371 7fb7691e2700 1 -- 172.21.2.102:0/1495765939 learned_addr learned my addr 172.21.2.102:0/1495765939 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:22:13.386 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:13.371 7fb7689e1700 1 -- 172.21.2.102:0/1495765939 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3673222256 0 0) 0x7fb754000ce0 con 0x7fb7640e5470 2021-02-10T11:22:13.386 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:13.371 7fb76a1e4700 1 --2- 172.21.2.102:0/1495765939 >> v2:172.21.2.113:3300/0 conn(0x7fb76407e6a0 0x7fb7640d1790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:13.386 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:13.371 7fb7689e1700 1 -- 172.21.2.102:0/1495765939 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb750002fe0 con 0x7fb7640e5470 2021-02-10T11:22:13.386 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:13.372 7fb7689e1700 1 -- 172.21.2.102:0/1495765939 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3562099713 0 0) 0x7fb7540010e0 con 0x7fb7640e5470 2021-02-10T11:22:13.386 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:13.372 7fb7689e1700 1 -- 172.21.2.102:0/1495765939 >> v2:172.21.2.113:3300/0 conn(0x7fb76407e6a0 msgr2=0x7fb7640d1790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:13.387 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:13.372 7fb7689e1700 1 --2- 172.21.2.102:0/1495765939 >> v2:172.21.2.113:3300/0 conn(0x7fb76407e6a0 0x7fb7640d1790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:13.387 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:13.372 7fb7689e1700 1 -- 172.21.2.102:0/1495765939 >> v1:172.21.2.113:6789/0 conn(0x7fb7640e5070 legacy=0x7fb7641b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:13.387 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:13.372 7fb7689e1700 1 -- 172.21.2.102:0/1495765939 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7640d1db0 con 0x7fb7640e5470 2021-02-10T11:22:13.387 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:13.372 7fb7689e1700 1 -- 172.21.2.102:0/1495765939 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7fb754001660 con 0x7fb7640e5470 2021-02-10T11:22:13.387 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:13.372 7fb76a1e4700 1 --2- 172.21.2.102:0/1495765939 >> v2:172.21.2.113:3300/0 conn(0x7fb76407e6a0 0x7fb7640d1790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:22:13.387 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:13.372 7fb7689e1700 1 -- 172.21.2.102:0/1495765939 >> v1:172.21.2.102:6789/0 conn(0x7fb7640e5470 legacy=0x7fb76407f300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:13.387 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:13.372 7fb7689e1700 1 --2- 172.21.2.102:0/1495765939 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb750003a60 0x7fb750005ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:13.387 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:13.372 7fb7689e1700 1 --2- 172.21.2.102:0/1495765939 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb750006770 0x7fb750008c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:13.387 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:13.372 7fb7689e1700 1 --2- 172.21.2.102:0/1495765939 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb750009590 0x7fb75000bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:13.387 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:13.372 7fb7689e1700 1 -- 172.21.2.102:0/1495765939 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb75000c300 con 0x7fb750009590 2021-02-10T11:22:13.387 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:13.372 7fb7689e1700 1 -- 172.21.2.102:0/1495765939 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb75000c520 con 0x7fb750006770 2021-02-10T11:22:13.387 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:13.372 7fb7689e1700 1 -- 172.21.2.102:0/1495765939 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb750050750 con 0x7fb750003a60 2021-02-10T11:22:13.387 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:13.372 7fb76a1e4700 1 --2- 172.21.2.102:0/1495765939 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb750009590 0x7fb75000bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:13.388 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:13.372 7fb7691e2700 1 --2- 172.21.2.102:0/1495765939 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb750006770 0x7fb750008c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:13.388 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:13.372 7fb7699e3700 1 --2- 172.21.2.102:0/1495765939 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb750003a60 0x7fb750005ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:13.388 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:13.372 7fb76a1e4700 1 -- 172.21.2.102:0/1495765939 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb750003a60 msgr2=0x7fb750005ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:13.388 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:13.372 7fb76a1e4700 1 --2- 172.21.2.102:0/1495765939 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb750003a60 0x7fb750005ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:13.388 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:13.372 7fb76a1e4700 1 -- 172.21.2.102:0/1495765939 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb750006770 msgr2=0x7fb750008c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:13.388 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:13.372 7fb76a1e4700 1 --2- 172.21.2.102:0/1495765939 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb750006770 0x7fb750008c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:13.388 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:13.372 7fb76a1e4700 1 -- 172.21.2.102:0/1495765939 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb750050980 con 0x7fb750009590 2021-02-10T11:22:13.388 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:13.372 7fb7691e2700 1 --2- 172.21.2.102:0/1495765939 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb750006770 0x7fb750008c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:22:13.388 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:13.372 7fb76a1e4700 1 -- 172.21.2.102:0/1495765939 reap_dead start 2021-02-10T11:22:13.388 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:13.372 7fb76a1e4700 1 --2- 172.21.2.102:0/1495765939 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb750009590 0x7fb75000bab0 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 rx=0x7fb75c009ec0 tx=0x7fb75c00c370).ready entity=mon.0 client_cookie=2e9bcab231c2608f server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:13.388 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:13.373 7fb7689e1700 1 -- 172.21.2.102:0/1495765939 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb75c007070 con 0x7fb750009590 2021-02-10T11:22:13.388 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:13.373 7fb7689e1700 1 -- 172.21.2.102:0/1495765939 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb75c006b00 con 0x7fb750009590 2021-02-10T11:22:13.389 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:13.373 7fb7689e1700 1 -- 172.21.2.102:0/1495765939 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb75c007670 con 0x7fb750009590 2021-02-10T11:22:13.389 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:13.373 7fb76bbc7700 1 -- 172.21.2.102:0/1495765939 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb750009590 msgr2=0x7fb75000bab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:13.389 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:13.373 7fb76bbc7700 1 --2- 172.21.2.102:0/1495765939 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb750009590 0x7fb75000bab0 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 rx=0x7fb75c009ec0 tx=0x7fb75c00c370).stop 2021-02-10T11:22:13.389 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:13.373 7fb76bbc7700 1 -- 172.21.2.102:0/1495765939 shutdown_connections 2021-02-10T11:22:13.389 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:13.373 7fb76bbc7700 1 --2- 172.21.2.102:0/1495765939 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb750009590 0x7fb75000bab0 unknown :-1 s=CLOSED pgs=1022 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:13.389 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:13.373 7fb76bbc7700 1 -- 172.21.2.102:0/1495765939 >> 172.21.2.102:0/1495765939 conn(0x7fb7640ce690 msgr2=0x7fb7640e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:13.389 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:13.373 7fb76bbc7700 1 -- 172.21.2.102:0/1495765939 shutdown_connections 2021-02-10T11:22:13.389 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:13.373 7fb76bbc7700 1 -- 172.21.2.102:0/1495765939 wait complete. 2021-02-10T11:22:13.389 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:13.388 7fb76bbc7700 1 Processor -- start 2021-02-10T11:22:13.389 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:13.388 7fb76bbc7700 1 -- start start 2021-02-10T11:22:13.389 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:13.388 7fb76bbc7700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb7641a6c30 0x7fb7641a7030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:13.389 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:13.388 7fb76bbc7700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb7641a7650 0x7fb76408c6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:13.389 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:13.388 7fb76bbc7700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb76408ccd0 0x7fb76408d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:13.390 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:13.388 7fb76bbc7700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb76408d830 con 0x7fb7641a6c30 2021-02-10T11:22:13.390 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:13.388 7fb76bbc7700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb764197830 con 0x7fb76408ccd0 2021-02-10T11:22:13.390 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:13.388 7fb76bbc7700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb76408ddb0 con 0x7fb7641a7650 2021-02-10T11:22:13.390 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:13.389 7fb7699e3700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb7641a6c30 0x7fb7641a7030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:13.390 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:13.389 7fb7699e3700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb7641a6c30 0x7fb7641a7030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:55214/0 (socket says 172.21.2.102:55214) 2021-02-10T11:22:13.390 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:13.389 7fb7699e3700 1 -- 172.21.2.102:0/1477419156 learned_addr learned my addr 172.21.2.102:0/1477419156 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:22:13.390 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:13.389 7fb7699e3700 1 -- 172.21.2.102:0/1477419156 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb7641a7650 msgr2=0x7fb76408c6b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:22:13.390 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:13.389 7fb7699e3700 1 --2- 172.21.2.102:0/1477419156 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb7641a7650 0x7fb76408c6b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:13.390 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:13.389 7fb7699e3700 1 -- 172.21.2.102:0/1477419156 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb76408ccd0 msgr2=0x7fb76408d0d0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:22:13.390 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:13.389 7fb7699e3700 1 --2- 172.21.2.102:0/1477419156 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb76408ccd0 0x7fb76408d0d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:13.390 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:13.389 7fb7699e3700 1 -- 172.21.2.102:0/1477419156 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb76408dfd0 con 0x7fb7641a6c30 2021-02-10T11:22:13.390 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:13.389 7fb7699e3700 1 --2- 172.21.2.102:0/1477419156 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb7641a6c30 0x7fb7641a7030 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 rx=0x7fb760006f10 tx=0x7fb76000b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:13.391 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:13.389 7fb75a7fc700 1 -- 172.21.2.102:0/1477419156 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb760006ba0 con 0x7fb7641a6c30 2021-02-10T11:22:13.392 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:13.389 7fb75a7fc700 1 -- 172.21.2.102:0/1477419156 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb76000be10 con 0x7fb7641a6c30 2021-02-10T11:22:13.392 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:13.389 7fb76bbc7700 1 -- 172.21.2.102:0/1477419156 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb76408dfd0 con 0x7fb7641a6c30 2021-02-10T11:22:13.392 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:13.389 7fb75a7fc700 1 -- 172.21.2.102:0/1477419156 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb760000ca0 con 0x7fb7641a6c30 2021-02-10T11:22:13.392 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:13.389 7fb76bbc7700 1 -- 172.21.2.102:0/1477419156 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb76408dfd0 con 0x7fb7641a6c30 2021-02-10T11:22:13.392 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:13.390 7fb75a7fc700 1 -- 172.21.2.102:0/1477419156 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fb760006240 con 0x7fb7641a6c30 2021-02-10T11:22:13.392 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:13.390 7fb75a7fc700 1 --2- 172.21.2.102:0/1477419156 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb74c018730 0x7fb74c01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:13.392 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:13.390 7fb75a7fc700 1 -- 172.21.2.102:0/1477419156 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fb7600327b0 con 0x7fb7641a6c30 2021-02-10T11:22:13.392 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:13.390 7fb7691e2700 1 --2- 172.21.2.102:0/1477419156 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb74c018730 0x7fb74c01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:13.392 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:13.390 7fb7691e2700 1 --2- 172.21.2.102:0/1477419156 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb74c018730 0x7fb74c01ac10 crc :-1 s=READY pgs=497 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:13.417 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:13.421 7fb76bbc7700 1 -- 172.21.2.102:0/1477419156 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb764084830 con 0x7fb7641a6c30 2021-02-10T11:22:13.420 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:13.423 7fb75a7fc700 1 -- 172.21.2.102:0/1477419156 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fb760006030 con 0x7fb7641a6c30 2021-02-10T11:22:13.548 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:13.552 7fb76bbc7700 1 -- 172.21.2.102:0/1477419156 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb76407ed80 con 0x7fb7641a6c30 2021-02-10T11:22:13.549 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:13.552 7fb75a7fc700 1 -- 172.21.2.102:0/1477419156 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+2099 (secure 0 0 0) 0x7fb760006240 con 0x7fb7641a6c30 2021-02-10T11:22:13.549 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:22:13.549 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":15,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4845,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.102:6835/3747080008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6834","nonce":3747080008},{"type":"v1","addr":"172.21.2.102:6835","nonce":3747080008}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":10}],"filesystems":[{"mdsmap":{"epoch":14,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:05:12.567145","modified":"2021-02-10 11:05:14.922542","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4510},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4510":{"gid":4510,"name":"b","rank":0,"incarnation":11,"state":"up:active","state_seq":4,"addr":"172.21.2.113:6833/3898434901","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6832","nonce":3898434901},{"type":"v1","addr":"172.21.2.113:6833","nonce":3898434901}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":2}]} 2021-02-10T11:22:13.550 INFO:teuthology.orchestra.run.gibba002.stderr:dumped fsmap epoch 15 2021-02-10T11:22:13.551 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:13.554 7fb76bbc7700 1 -- 172.21.2.102:0/1477419156 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb74c018730 msgr2=0x7fb74c01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:13.551 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:13.554 7fb76bbc7700 1 --2- 172.21.2.102:0/1477419156 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb74c018730 0x7fb74c01ac10 crc :-1 s=READY pgs=497 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:13.551 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:13.554 7fb76bbc7700 1 -- 172.21.2.102:0/1477419156 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb7641a6c30 msgr2=0x7fb7641a7030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:13.551 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:13.554 7fb76bbc7700 1 --2- 172.21.2.102:0/1477419156 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb7641a6c30 0x7fb7641a7030 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 rx=0x7fb760006f10 tx=0x7fb76000b990).stop 2021-02-10T11:22:13.551 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:13.554 7fb76bbc7700 1 -- 172.21.2.102:0/1477419156 shutdown_connections 2021-02-10T11:22:13.551 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:13.554 7fb76bbc7700 1 --2- 172.21.2.102:0/1477419156 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb74c018730 0x7fb74c01ac10 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:13.551 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:13.554 7fb76bbc7700 1 --2- 172.21.2.102:0/1477419156 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb76408ccd0 0x7fb76408d0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:13.551 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:13.554 7fb76bbc7700 1 --2- 172.21.2.102:0/1477419156 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb7641a7650 0x7fb76408c6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:13.551 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:13.554 7fb76bbc7700 1 --2- 172.21.2.102:0/1477419156 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb7641a6c30 0x7fb7641a7030 unknown :-1 s=CLOSED pgs=1023 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:13.552 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:13.554 7fb76bbc7700 1 -- 172.21.2.102:0/1477419156 >> 172.21.2.102:0/1477419156 conn(0x7fb7641b2930 msgr2=0x7fb764084e00 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:13.552 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:13.554 7fb76bbc7700 1 -- 172.21.2.102:0/1477419156 shutdown_connections 2021-02-10T11:22:13.552 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:13.554 7fb76bbc7700 1 -- 172.21.2.102:0/1477419156 wait complete. 2021-02-10T11:22:13.559 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs fail cephfs 2021-02-10T11:22:13.986 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:13.972 7fc1da443700 1 Processor -- start 2021-02-10T11:22:13.986 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:13.972 7fc1da443700 1 -- start start 2021-02-10T11:22:13.986 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:13.973 7fc1da443700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fc1d40e5070 0x7fc1d41b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:13.986 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:13.973 7fc1da443700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1d407ed80 con 0x7fc1d41b3e40 2021-02-10T11:22:13.986 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:13.973 7fc1da443700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1d407e670 con 0x7fc1d40e5470 2021-02-10T11:22:13.986 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:13.973 7fc1da443700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1d407e890 con 0x7fc1d40e5070 2021-02-10T11:22:13.987 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:13.973 7fc1d37fe700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fc1d41b3e40 0x7fc1d407d690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:47446/0 (socket says 172.21.2.102:47446) 2021-02-10T11:22:13.987 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:13.973 7fc1d37fe700 1 -- 172.21.2.102:0/1592012849 learned_addr learned my addr 172.21.2.102:0/1592012849 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:22:13.987 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:13.973 7fc1d3fff700 1 --2- 172.21.2.102:0/1592012849 >> v2:172.21.2.113:3300/0 conn(0x7fc1d40e5070 0x7fc1d41b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:13.987 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:13.973 7fc1d3fff700 1 -- 172.21.2.102:0/1592012849 >> v1:172.21.2.113:6789/0 conn(0x7fc1d40e5470 legacy=0x7fc1d407de80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:13.987 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:13.973 7fc1d3fff700 1 -- 172.21.2.102:0/1592012849 >> v1:172.21.2.102:6789/0 conn(0x7fc1d41b3e40 legacy=0x7fc1d407d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:13.987 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:13.973 7fc1d3fff700 1 -- 172.21.2.102:0/1592012849 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1d407eab0 con 0x7fc1d40e5070 2021-02-10T11:22:13.987 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:13.974 7fc1d3fff700 1 --2- 172.21.2.102:0/1592012849 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc1d40e5070 0x7fc1d41b3820 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 rx=0x7fc1bc006650 tx=0x7fc1bc009a80).ready entity=mon.1 client_cookie=ce7b8ca368b1b60d server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:13.987 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:13.974 7fc1d2ffd700 1 -- 172.21.2.102:0/1592012849 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc1bc00c0c0 con 0x7fc1d40e5070 2021-02-10T11:22:13.987 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:13.974 7fc1d2ffd700 1 -- 172.21.2.102:0/1592012849 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1bc00c340 con 0x7fc1d40e5070 2021-02-10T11:22:13.987 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:13.974 7fc1d2ffd700 1 -- 172.21.2.102:0/1592012849 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc1bc00c860 con 0x7fc1d40e5070 2021-02-10T11:22:13.987 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:13.974 7fc1da443700 1 -- 172.21.2.102:0/1592012849 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc1d40e5070 msgr2=0x7fc1d41b3820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:13.987 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:13.974 7fc1da443700 1 --2- 172.21.2.102:0/1592012849 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc1d40e5070 0x7fc1d41b3820 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 rx=0x7fc1bc006650 tx=0x7fc1bc009a80).stop 2021-02-10T11:22:13.987 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:13.975 7fc1da443700 1 -- 172.21.2.102:0/1592012849 shutdown_connections 2021-02-10T11:22:13.988 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:13.975 7fc1da443700 1 --2- 172.21.2.102:0/1592012849 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc1d40e5070 0x7fc1d41b3820 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:13.988 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:13.975 7fc1da443700 1 -- 172.21.2.102:0/1592012849 >> 172.21.2.102:0/1592012849 conn(0x7fc1d40ce690 msgr2=0x7fc1d40e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:13.988 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:13.975 7fc1da443700 1 -- 172.21.2.102:0/1592012849 shutdown_connections 2021-02-10T11:22:13.988 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:13.975 7fc1da443700 1 -- 172.21.2.102:0/1592012849 wait complete. 2021-02-10T11:22:13.990 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:13.990 7fc1da443700 1 Processor -- start 2021-02-10T11:22:13.990 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:13.990 7fc1da443700 1 -- start start 2021-02-10T11:22:13.990 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:13.990 7fc1da443700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc1d4095680 0x7fc1d40b93e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:13.990 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:13.990 7fc1da443700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc1d4095a80 0x7fc1d40941d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:13.990 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:13.990 7fc1da443700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc1d40b9a00 0x7fc1d40947f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:13.990 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:13.990 7fc1da443700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1d4090b00 con 0x7fc1d4095a80 2021-02-10T11:22:13.990 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:13.990 7fc1da443700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1d41c25c0 con 0x7fc1d4095680 2021-02-10T11:22:13.990 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:13.990 7fc1da443700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1d41c27a0 con 0x7fc1d40b9a00 2021-02-10T11:22:13.990 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:13.991 7fc1d37fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc1d4095a80 0x7fc1d40941d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:13.990 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:13.991 7fc1d37fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc1d4095a80 0x7fc1d40941d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:55232/0 (socket says 172.21.2.102:55232) 2021-02-10T11:22:13.991 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:13.991 7fc1d37fe700 1 -- 172.21.2.102:0/849585850 learned_addr learned my addr 172.21.2.102:0/849585850 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:22:13.991 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:13.991 7fc1d3fff700 1 --2- 172.21.2.102:0/849585850 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc1d4095680 0x7fc1d40b93e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:13.991 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:13.991 7fc1d8a60700 1 --2- 172.21.2.102:0/849585850 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc1d40b9a00 0x7fc1d40947f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:13.991 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:13.991 7fc1d37fe700 1 -- 172.21.2.102:0/849585850 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc1d40b9a00 msgr2=0x7fc1d40947f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:13.991 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:13.991 7fc1d37fe700 1 --2- 172.21.2.102:0/849585850 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc1d40b9a00 0x7fc1d40947f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:13.991 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:13.991 7fc1d37fe700 1 -- 172.21.2.102:0/849585850 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc1d4095680 msgr2=0x7fc1d40b93e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:13.991 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:13.991 7fc1d37fe700 1 --2- 172.21.2.102:0/849585850 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc1d4095680 0x7fc1d40b93e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:13.991 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:13.991 7fc1d37fe700 1 -- 172.21.2.102:0/849585850 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1d4094e10 con 0x7fc1d4095a80 2021-02-10T11:22:13.991 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:13.991 7fc1d8a60700 1 --2- 172.21.2.102:0/849585850 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc1d40b9a00 0x7fc1d40947f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:22:13.991 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:13.991 7fc1d37fe700 1 --2- 172.21.2.102:0/849585850 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc1d4095a80 0x7fc1d40941d0 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 rx=0x7fc1c4001c60 tx=0x7fc1c400a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:13.991 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:13.991 7fc1d0ff9700 1 -- 172.21.2.102:0/849585850 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc1c400acc0 con 0x7fc1d4095a80 2021-02-10T11:22:13.991 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:13.991 7fc1d0ff9700 1 -- 172.21.2.102:0/849585850 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1c4004140 con 0x7fc1d4095a80 2021-02-10T11:22:13.991 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:13.991 7fc1d0ff9700 1 -- 172.21.2.102:0/849585850 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc1c4004620 con 0x7fc1d4095a80 2021-02-10T11:22:13.991 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:13.991 7fc1da443700 1 -- 172.21.2.102:0/849585850 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1d4094e10 con 0x7fc1d4095a80 2021-02-10T11:22:13.992 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:13.991 7fc1da443700 1 -- 172.21.2.102:0/849585850 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1d4094e10 con 0x7fc1d4095a80 2021-02-10T11:22:13.992 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:13.992 7fc1d0ff9700 1 -- 172.21.2.102:0/849585850 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fc1c4000d90 con 0x7fc1d4095a80 2021-02-10T11:22:13.992 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:13.992 7fc1d0ff9700 1 --2- 172.21.2.102:0/849585850 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc1b8018730 0x7fc1b801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:13.992 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:13.992 7fc1d0ff9700 1 -- 172.21.2.102:0/849585850 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fc1c40327a0 con 0x7fc1d4095a80 2021-02-10T11:22:13.992 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:13.992 7fc1d3fff700 1 --2- 172.21.2.102:0/849585850 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc1b8018730 0x7fc1b801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:13.992 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:13.992 7fc1d3fff700 1 --2- 172.21.2.102:0/849585850 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc1b8018730 0x7fc1b801ac10 crc :-1 s=READY pgs=498 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:14.018 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:14.022 7fc1da443700 1 -- 172.21.2.102:0/849585850 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1d4089810 con 0x7fc1d4095a80 2021-02-10T11:22:14.021 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:14.024 7fc1d0ff9700 1 -- 172.21.2.102:0/849585850 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fc1c40009b0 con 0x7fc1d4095a80 2021-02-10T11:22:14.154 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:14.155 7fc1da443700 1 -- 172.21.2.102:0/849585850 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fc1d40ce690 con 0x7fc1d4095a80 2021-02-10T11:22:14.972 INFO:tasks.ceph.mon.a.gibba002.stderr:2021-02-10 11:22:14.975 7f8dd2755700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-02-10T11:22:14.977 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:14.980 7fc1d0ff9700 1 -- 172.21.2.102:0/849585850 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v16) v1 ==== 161+0+0 (secure 0 0 0) 0x7fc1c4000d90 con 0x7fc1d4095a80 2021-02-10T11:22:15.021 INFO:teuthology.orchestra.run.gibba002.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2021-02-10T11:22:15.023 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:15.026 7fc1da443700 1 -- 172.21.2.102:0/849585850 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc1b8018730 msgr2=0x7fc1b801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:15.023 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:15.026 7fc1da443700 1 --2- 172.21.2.102:0/849585850 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc1b8018730 0x7fc1b801ac10 crc :-1 s=READY pgs=498 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:15.023 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:15.026 7fc1da443700 1 -- 172.21.2.102:0/849585850 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc1d4095a80 msgr2=0x7fc1d40941d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:15.023 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:15.026 7fc1da443700 1 --2- 172.21.2.102:0/849585850 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc1d4095a80 0x7fc1d40941d0 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 rx=0x7fc1c4001c60 tx=0x7fc1c400a8d0).stop 2021-02-10T11:22:15.023 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:15.026 7fc1da443700 1 -- 172.21.2.102:0/849585850 shutdown_connections 2021-02-10T11:22:15.024 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:15.026 7fc1da443700 1 --2- 172.21.2.102:0/849585850 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc1d40b9a00 0x7fc1d40947f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:15.024 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:15.026 7fc1da443700 1 --2- 172.21.2.102:0/849585850 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc1d4095a80 0x7fc1d40941d0 unknown :-1 s=CLOSED pgs=1025 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:15.024 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:15.026 7fc1da443700 1 --2- 172.21.2.102:0/849585850 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc1b8018730 0x7fc1b801ac10 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:15.024 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:15.026 7fc1da443700 1 --2- 172.21.2.102:0/849585850 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc1d4095680 0x7fc1d40b93e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:15.024 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:15.026 7fc1da443700 1 -- 172.21.2.102:0/849585850 >> 172.21.2.102:0/849585850 conn(0x7fc1d40ba050 msgr2=0x7fc1d40afd70 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:15.024 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:15.027 7fc1da443700 1 -- 172.21.2.102:0/849585850 shutdown_connections 2021-02-10T11:22:15.024 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:15.027 7fc1da443700 1 -- 172.21.2.102:0/849585850 wait complete. 2021-02-10T11:22:15.029 INFO:tasks.ceph.mds.b.gibba013.stdout:starting mds.b at 2021-02-10T11:22:15.032 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:22:15.459 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:15.446 7f990074b700 1 Processor -- start 2021-02-10T11:22:15.459 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:15.446 7f990074b700 1 -- start start 2021-02-10T11:22:15.459 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:15.446 7f990074b700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f98f80e5470 0x7f98f807e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:15.459 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:15.446 7f990074b700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f98f807f300 0x7f98f807f700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:15.459 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:15.446 7f990074b700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98f807ed80 con 0x7f98f80e5070 2021-02-10T11:22:15.459 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:15.446 7f990074b700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98f81b4010 con 0x7f98f80e5470 2021-02-10T11:22:15.460 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:15.446 7f990074b700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98f80d1790 con 0x7f98f807f300 2021-02-10T11:22:15.460 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:15.447 7f98fe567700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f98f80e5070 0x7f98f81b3820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:47460/0 (socket says 172.21.2.102:47460) 2021-02-10T11:22:15.460 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:15.447 7f98fe567700 1 -- 172.21.2.102:0/1025382048 learned_addr learned my addr 172.21.2.102:0/1025382048 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:22:15.460 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:15.447 7f98fdd66700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f98f80e5470 0x7f98f807e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:15.460 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:15.447 7f98fd565700 1 -- 172.21.2.102:0/1025382048 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1082047540 0 0) 0x7f98f4001180 con 0x7f98f80e5070 2021-02-10T11:22:15.460 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:15.447 7f98fdd66700 1 -- 172.21.2.102:0/1025382048 >> v2:172.21.2.113:3300/0 conn(0x7f98f807f300 msgr2=0x7f98f807f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:15.460 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:15.447 7f98fdd66700 1 --2- 172.21.2.102:0/1025382048 >> v2:172.21.2.113:3300/0 conn(0x7f98f807f300 0x7f98f807f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:15.460 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:15.447 7f98fdd66700 1 -- 172.21.2.102:0/1025382048 >> v1:172.21.2.102:6789/0 conn(0x7f98f80e5070 legacy=0x7f98f81b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:15.460 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:15.447 7f98fdd66700 1 -- 172.21.2.102:0/1025382048 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98f80d19b0 con 0x7f98f80e5470 2021-02-10T11:22:15.460 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:15.447 7f98fdd66700 1 --2- 172.21.2.102:0/1025382048 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f98f80e5470 0x7f98f807e6a0 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 rx=0x7f98ec006650 tx=0x7f98ec009a80).ready entity=mon.0 client_cookie=5524896ecee4e505 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:15.460 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:15.447 7f98fd565700 1 -- 172.21.2.102:0/1025382048 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f98ec0009b0 con 0x7f98f80e5470 2021-02-10T11:22:15.460 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:15.447 7f98fd565700 1 -- 172.21.2.102:0/1025382048 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98ec00c0c0 con 0x7f98f80e5470 2021-02-10T11:22:15.461 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:15.447 7f98fd565700 1 -- 172.21.2.102:0/1025382048 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f98ec00c5e0 con 0x7f98f80e5470 2021-02-10T11:22:15.461 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:15.447 7f990074b700 1 -- 172.21.2.102:0/1025382048 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f98f80e5470 msgr2=0x7f98f807e6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:15.461 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:15.448 7f990074b700 1 --2- 172.21.2.102:0/1025382048 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f98f80e5470 0x7f98f807e6a0 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 rx=0x7f98ec006650 tx=0x7f98ec009a80).stop 2021-02-10T11:22:15.461 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:15.448 7f990074b700 1 -- 172.21.2.102:0/1025382048 shutdown_connections 2021-02-10T11:22:15.461 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:15.448 7f990074b700 1 --2- 172.21.2.102:0/1025382048 >> v2:172.21.2.113:3300/0 conn(0x7f98f807f300 0x7f98f807f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:15.461 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:15.448 7f990074b700 1 --2- 172.21.2.102:0/1025382048 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f98f80e5470 0x7f98f807e6a0 unknown :-1 s=CLOSED pgs=1029 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:15.461 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:15.448 7f990074b700 1 -- 172.21.2.102:0/1025382048 >> 172.21.2.102:0/1025382048 conn(0x7f98f80ce690 msgr2=0x7f98f80e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:15.461 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:15.448 7f990074b700 1 -- 172.21.2.102:0/1025382048 shutdown_connections 2021-02-10T11:22:15.461 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:15.448 7f990074b700 1 -- 172.21.2.102:0/1025382048 wait complete. 2021-02-10T11:22:15.463 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:15.463 7f990074b700 1 Processor -- start 2021-02-10T11:22:15.463 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:15.463 7f990074b700 1 -- start start 2021-02-10T11:22:15.463 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:15.463 7f990074b700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f98f808d1c0 0x7f98f8081230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:15.463 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:15.463 7f990074b700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f98f8081850 0x7f98f80ad0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:15.463 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:15.463 7f990074b700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f98f80ad6f0 0x7f98f81a1d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:15.463 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:15.463 7f990074b700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98f8087740 con 0x7f98f80ad6f0 2021-02-10T11:22:15.463 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:15.463 7f990074b700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98f8081db0 con 0x7f98f808d1c0 2021-02-10T11:22:15.463 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:15.463 7f990074b700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98f808d5c0 con 0x7f98f8081850 2021-02-10T11:22:15.463 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:15.463 7f98fed68700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f98f80ad6f0 0x7f98f81a1d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:15.463 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:15.464 7f98fed68700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f98f80ad6f0 0x7f98f81a1d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:55244/0 (socket says 172.21.2.102:55244) 2021-02-10T11:22:15.463 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:15.464 7f98fed68700 1 -- 172.21.2.102:0/945718475 learned_addr learned my addr 172.21.2.102:0/945718475 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:22:15.463 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:15.464 7f98fed68700 1 -- 172.21.2.102:0/945718475 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f98f8081850 msgr2=0x7f98f80ad0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:15.464 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:15.464 7f98fed68700 1 --2- 172.21.2.102:0/945718475 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f98f8081850 0x7f98f80ad0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:15.464 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:15.464 7f98fed68700 1 -- 172.21.2.102:0/945718475 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f98f808d1c0 msgr2=0x7f98f8081230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:15.464 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:15.464 7f98fed68700 1 --2- 172.21.2.102:0/945718475 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f98f808d1c0 0x7f98f8081230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:15.464 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:15.464 7f98fed68700 1 -- 172.21.2.102:0/945718475 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98f808da00 con 0x7f98f80ad6f0 2021-02-10T11:22:15.464 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:15.464 7f98fed68700 1 --2- 172.21.2.102:0/945718475 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f98f80ad6f0 0x7f98f81a1d30 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 rx=0x7f98f0009980 tx=0x7f98f000bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:15.464 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:15.464 7f98eaffd700 1 -- 172.21.2.102:0/945718475 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f98f0005870 con 0x7f98f80ad6f0 2021-02-10T11:22:15.464 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:15.464 7f98eaffd700 1 -- 172.21.2.102:0/945718475 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98f0005b30 con 0x7f98f80ad6f0 2021-02-10T11:22:15.464 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:15.464 7f98eaffd700 1 -- 172.21.2.102:0/945718475 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f98f0006030 con 0x7f98f80ad6f0 2021-02-10T11:22:15.464 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:15.464 7f990074b700 1 -- 172.21.2.102:0/945718475 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98f808da00 con 0x7f98f80ad6f0 2021-02-10T11:22:15.464 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:15.464 7f990074b700 1 -- 172.21.2.102:0/945718475 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98f80adde0 con 0x7f98f80ad6f0 2021-02-10T11:22:15.464 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:15.465 7f98eaffd700 1 -- 172.21.2.102:0/945718475 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f98f0005d10 con 0x7f98f80ad6f0 2021-02-10T11:22:15.464 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:15.465 7f98eaffd700 1 --2- 172.21.2.102:0/945718475 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f98e00186c0 0x7f98e001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:15.464 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:15.465 7f98eaffd700 1 -- 172.21.2.102:0/945718475 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5659+0+0 (secure 0 0 0) 0x7f98f00329d0 con 0x7f98f80ad6f0 2021-02-10T11:22:15.465 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:15.465 7f98fe567700 1 --2- 172.21.2.102:0/945718475 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f98e00186c0 0x7f98e001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:15.465 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:15.466 7f98fe567700 1 --2- 172.21.2.102:0/945718475 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f98e00186c0 0x7f98e001abc0 crc :-1 s=READY pgs=499 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:15.493 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:15.496 7f990074b700 1 -- 172.21.2.102:0/945718475 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98f8087740 con 0x7f98f80ad6f0 2021-02-10T11:22:15.495 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:15.499 7f98eaffd700 1 -- 172.21.2.102:0/945718475 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f98f0006290 con 0x7f98f80ad6f0 2021-02-10T11:22:15.625 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:15.629 7f990074b700 1 -- 172.21.2.102:0/945718475 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f98f8087740 con 0x7f98f80ad6f0 2021-02-10T11:22:15.626 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:15.629 7f98eaffd700 1 -- 172.21.2.102:0/945718475 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+1754 (secure 0 0 0) 0x7f98f0006f20 con 0x7f98f80ad6f0 2021-02-10T11:22:15.626 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:22:15.626 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4845,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.102:6835/3747080008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6834","nonce":3747080008},{"type":"v1","addr":"172.21.2.102:6835","nonce":3747080008}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":10}],"filesystems":[{"mdsmap":{"epoch":16,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:05:12.567145","modified":"2021-02-10 11:22:14.976725","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":2}]} 2021-02-10T11:22:15.627 INFO:teuthology.orchestra.run.gibba002.stderr:dumped fsmap epoch 16 2021-02-10T11:22:15.628 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:15.631 7f990074b700 1 -- 172.21.2.102:0/945718475 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f98e00186c0 msgr2=0x7f98e001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:15.628 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:15.631 7f990074b700 1 --2- 172.21.2.102:0/945718475 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f98e00186c0 0x7f98e001abc0 crc :-1 s=READY pgs=499 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:15.628 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:15.631 7f990074b700 1 -- 172.21.2.102:0/945718475 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f98f80ad6f0 msgr2=0x7f98f81a1d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:15.628 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:15.631 7f990074b700 1 --2- 172.21.2.102:0/945718475 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f98f80ad6f0 0x7f98f81a1d30 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 rx=0x7f98f0009980 tx=0x7f98f000bcd0).stop 2021-02-10T11:22:15.628 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:15.631 7f990074b700 1 -- 172.21.2.102:0/945718475 shutdown_connections 2021-02-10T11:22:15.629 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:15.631 7f990074b700 1 --2- 172.21.2.102:0/945718475 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f98f80ad6f0 0x7f98f81a1d30 unknown :-1 s=CLOSED pgs=1030 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:15.629 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:15.631 7f990074b700 1 --2- 172.21.2.102:0/945718475 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f98f8081850 0x7f98f80ad0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:15.629 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:15.631 7f990074b700 1 --2- 172.21.2.102:0/945718475 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f98e00186c0 0x7f98e001abc0 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:15.629 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:15.631 7f990074b700 1 --2- 172.21.2.102:0/945718475 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f98f808d1c0 0x7f98f8081230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:15.629 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:15.631 7f990074b700 1 -- 172.21.2.102:0/945718475 >> 172.21.2.102:0/945718475 conn(0x7f98f819b930 msgr2=0x7f98f81a0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:15.629 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:15.632 7f990074b700 1 -- 172.21.2.102:0/945718475 shutdown_connections 2021-02-10T11:22:15.629 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:15.632 7f990074b700 1 -- 172.21.2.102:0/945718475 wait complete. 2021-02-10T11:22:15.637 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2021-02-10T11:22:16.069 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:16.056 7fb765b6f700 1 Processor -- start 2021-02-10T11:22:16.069 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:16.056 7fb765b6f700 1 -- start start 2021-02-10T11:22:16.070 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:16.056 7fb765b6f700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fb7600d17f0 0x7fb7601b31e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:16.070 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:16.056 7fb765b6f700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb7600d1bf0 0x7fb7601b3800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:16.070 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:16.056 7fb765b6f700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7600f0b20 con 0x7fb7601b3e20 2021-02-10T11:22:16.070 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:16.056 7fb765b6f700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7600e5eb0 con 0x7fb7600d17f0 2021-02-10T11:22:16.070 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:16.056 7fb765b6f700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7600f0630 con 0x7fb7600d1bf0 2021-02-10T11:22:16.070 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:16.057 7fb75f7fe700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fb7600d17f0 0x7fb7601b31e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:16.071 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:16.057 7fb75f7fe700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fb7600d17f0 0x7fb7601b31e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:55248/0 (socket says 172.21.2.102:55248) 2021-02-10T11:22:16.071 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:16.057 7fb75f7fe700 1 -- 172.21.2.102:0/2671750578 learned_addr learned my addr 172.21.2.102:0/2671750578 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:22:16.071 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:16.057 7fb75f7fe700 1 -- 172.21.2.102:0/2671750578 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb7600d1bf0 msgr2=0x7fb7601b3800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:16.071 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:16.057 7fb75f7fe700 1 --2- 172.21.2.102:0/2671750578 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb7600d1bf0 0x7fb7601b3800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:16.071 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:16.057 7fb75f7fe700 1 -- 172.21.2.102:0/2671750578 >> v1:172.21.2.113:6789/0 conn(0x7fb7601b3e20 legacy=0x7fb7600ceef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:16.071 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:16.057 7fb75f7fe700 1 -- 172.21.2.102:0/2671750578 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7600f0820 con 0x7fb7600d17f0 2021-02-10T11:22:16.071 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:16.057 7fb75f7fe700 1 --2- 172.21.2.102:0/2671750578 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb7600d17f0 0x7fb7601b31e0 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 rx=0x7fb750006700 tx=0x7fb750009b00).ready entity=mon.0 client_cookie=64d572165f15ec08 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:16.071 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:16.057 7fb75e7fc700 1 -- 172.21.2.102:0/2671750578 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb75000c0c0 con 0x7fb7600d17f0 2021-02-10T11:22:16.071 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:16.057 7fb75e7fc700 1 -- 172.21.2.102:0/2671750578 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb75000c340 con 0x7fb7600d17f0 2021-02-10T11:22:16.071 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:16.057 7fb75e7fc700 1 -- 172.21.2.102:0/2671750578 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb75000c820 con 0x7fb7600d17f0 2021-02-10T11:22:16.071 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:16.057 7fb765b6f700 1 -- 172.21.2.102:0/2671750578 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb7600d17f0 msgr2=0x7fb7601b31e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:16.071 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:16.057 7fb765b6f700 1 --2- 172.21.2.102:0/2671750578 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb7600d17f0 0x7fb7601b31e0 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 rx=0x7fb750006700 tx=0x7fb750009b00).stop 2021-02-10T11:22:16.071 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:16.058 7fb765b6f700 1 -- 172.21.2.102:0/2671750578 shutdown_connections 2021-02-10T11:22:16.072 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:16.058 7fb765b6f700 1 --2- 172.21.2.102:0/2671750578 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb7600d1bf0 0x7fb7601b3800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:16.072 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:16.058 7fb765b6f700 1 --2- 172.21.2.102:0/2671750578 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb7600d17f0 0x7fb7601b31e0 unknown :-1 s=CLOSED pgs=1031 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:16.072 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:16.058 7fb765b6f700 1 -- 172.21.2.102:0/2671750578 >> 172.21.2.102:0/2671750578 conn(0x7fb7600fde40 msgr2=0x7fb7600ce390 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:16.072 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:16.058 7fb765b6f700 1 -- 172.21.2.102:0/2671750578 shutdown_connections 2021-02-10T11:22:16.072 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:16.058 7fb765b6f700 1 -- 172.21.2.102:0/2671750578 wait complete. 2021-02-10T11:22:16.072 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:16.072 7fb765b6f700 1 Processor -- start 2021-02-10T11:22:16.072 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:16.073 7fb765b6f700 1 -- start start 2021-02-10T11:22:16.073 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:16.073 7fb765b6f700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb7600ac220 0x7fb76008d660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:16.073 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:16.073 7fb765b6f700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb76009d490 0x7fb76009d890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:16.073 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:16.073 7fb765b6f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb76009deb0 0x7fb76018f400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:16.073 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:16.073 7fb765b6f700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb76009cf10 con 0x7fb76009deb0 2021-02-10T11:22:16.074 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:16.073 7fb765b6f700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb76008dc80 con 0x7fb7600ac220 2021-02-10T11:22:16.074 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:16.073 7fb765b6f700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb76018fa20 con 0x7fb76009d490 2021-02-10T11:22:16.074 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:16.073 7fb75ffff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb76009deb0 0x7fb76018f400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:16.074 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:16.073 7fb75ffff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb76009deb0 0x7fb76018f400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:55256/0 (socket says 172.21.2.102:55256) 2021-02-10T11:22:16.074 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:16.073 7fb75ffff700 1 -- 172.21.2.102:0/1580508767 learned_addr learned my addr 172.21.2.102:0/1580508767 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:22:16.074 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:16.074 7fb75ffff700 1 -- 172.21.2.102:0/1580508767 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb76009d490 msgr2=0x7fb76009d890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:16.074 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:16.074 7fb75ffff700 1 --2- 172.21.2.102:0/1580508767 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb76009d490 0x7fb76009d890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:16.074 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:16.074 7fb75ffff700 1 -- 172.21.2.102:0/1580508767 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb7600ac220 msgr2=0x7fb76008d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:16.074 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:16.074 7fb75ffff700 1 --2- 172.21.2.102:0/1580508767 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb7600ac220 0x7fb76008d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:16.074 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:16.074 7fb75ffff700 1 -- 172.21.2.102:0/1580508767 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb76018fc40 con 0x7fb76009deb0 2021-02-10T11:22:16.074 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:16.074 7fb75ffff700 1 --2- 172.21.2.102:0/1580508767 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb76009deb0 0x7fb76018f400 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 rx=0x7fb758009940 tx=0x7fb75800bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:16.074 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:16.074 7fb747fff700 1 -- 172.21.2.102:0/1580508767 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb758000f90 con 0x7fb76009deb0 2021-02-10T11:22:16.074 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:16.074 7fb747fff700 1 -- 172.21.2.102:0/1580508767 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb75800c2f0 con 0x7fb76009deb0 2021-02-10T11:22:16.075 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:16.074 7fb747fff700 1 -- 172.21.2.102:0/1580508767 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb75800c810 con 0x7fb76009deb0 2021-02-10T11:22:16.075 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:16.074 7fb765b6f700 1 -- 172.21.2.102:0/1580508767 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb76018fc40 con 0x7fb76009deb0 2021-02-10T11:22:16.075 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:16.074 7fb765b6f700 1 -- 172.21.2.102:0/1580508767 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7601901d0 con 0x7fb76009deb0 2021-02-10T11:22:16.075 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:16.075 7fb747fff700 1 -- 172.21.2.102:0/1580508767 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fb758001220 con 0x7fb76009deb0 2021-02-10T11:22:16.075 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:16.075 7fb747fff700 1 --2- 172.21.2.102:0/1580508767 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb7400186d0 0x7fb74001abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:16.075 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:16.075 7fb747fff700 1 -- 172.21.2.102:0/1580508767 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5659+0+0 (secure 0 0 0) 0x7fb7580329f0 con 0x7fb76009deb0 2021-02-10T11:22:16.075 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:16.075 7fb75f7fe700 1 --2- 172.21.2.102:0/1580508767 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb7400186d0 0x7fb74001abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:16.075 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:16.075 7fb75f7fe700 1 --2- 172.21.2.102:0/1580508767 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb7400186d0 0x7fb74001abd0 crc :-1 s=READY pgs=500 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:16.103 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:16.106 7fb765b6f700 1 -- 172.21.2.102:0/1580508767 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7601036f0 con 0x7fb76009deb0 2021-02-10T11:22:16.105 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:16.109 7fb747fff700 1 -- 172.21.2.102:0/1580508767 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fb758000f40 con 0x7fb76009deb0 2021-02-10T11:22:16.234 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:16.238 7fb765b6f700 1 -- 172.21.2.102:0/1580508767 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fb7600f0b20 con 0x7fb76009deb0 2021-02-10T11:22:16.985 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:16.989 7fb747fff700 1 -- 172.21.2.102:0/1580508767 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v17) v1 ==== 104+0+0 (secure 0 0 0) 0x7fb758001220 con 0x7fb76009deb0 2021-02-10T11:22:17.005 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:17.008 7fb765b6f700 1 -- 172.21.2.102:0/1580508767 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb7400186d0 msgr2=0x7fb74001abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:17.005 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:17.008 7fb765b6f700 1 --2- 172.21.2.102:0/1580508767 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb7400186d0 0x7fb74001abd0 crc :-1 s=READY pgs=500 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:17.006 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:17.009 7fb765b6f700 1 -- 172.21.2.102:0/1580508767 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb76009deb0 msgr2=0x7fb76018f400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:17.006 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:17.009 7fb765b6f700 1 --2- 172.21.2.102:0/1580508767 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb76009deb0 0x7fb76018f400 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 rx=0x7fb758009940 tx=0x7fb75800bcf0).stop 2021-02-10T11:22:17.006 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:17.009 7fb765b6f700 1 -- 172.21.2.102:0/1580508767 shutdown_connections 2021-02-10T11:22:17.006 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:17.009 7fb765b6f700 1 --2- 172.21.2.102:0/1580508767 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb76009deb0 0x7fb76018f400 unknown :-1 s=CLOSED pgs=1032 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:17.006 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:17.009 7fb765b6f700 1 --2- 172.21.2.102:0/1580508767 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb76009d490 0x7fb76009d890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:17.006 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:17.009 7fb765b6f700 1 --2- 172.21.2.102:0/1580508767 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb7400186d0 0x7fb74001abd0 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:17.006 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:17.009 7fb765b6f700 1 --2- 172.21.2.102:0/1580508767 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb7600ac220 0x7fb76008d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:17.006 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:17.009 7fb765b6f700 1 -- 172.21.2.102:0/1580508767 >> 172.21.2.102:0/1580508767 conn(0x7fb760105300 msgr2=0x7fb760103920 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:17.007 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:17.009 7fb765b6f700 1 -- 172.21.2.102:0/1580508767 shutdown_connections 2021-02-10T11:22:17.007 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:17.009 7fb765b6f700 1 -- 172.21.2.102:0/1580508767 wait complete. 2021-02-10T11:22:17.014 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool delete cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2021-02-10T11:22:17.445 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:17.431 7fdd1bdcd700 1 Processor -- start 2021-02-10T11:22:17.446 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:17.431 7fdd1bdcd700 1 -- start start 2021-02-10T11:22:17.446 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:17.432 7fdd1bdcd700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fdd1410aff0 0x7fdd140cf750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:17.446 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:17.432 7fdd1bdcd700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd140fe1c0 con 0x7fdd140d1c70 2021-02-10T11:22:17.446 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:17.432 7fdd1bdcd700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd141b0cb0 con 0x7fdd140d1870 2021-02-10T11:22:17.446 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:17.432 7fdd1bdcd700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd14109a60 con 0x7fdd1410aff0 2021-02-10T11:22:17.446 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:17.432 7fdd193e8700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fdd140d1c70 0x7fdd140cef60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:47484/0 (socket says 172.21.2.102:47484) 2021-02-10T11:22:17.446 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:17.432 7fdd193e8700 1 -- 172.21.2.102:0/1873798379 learned_addr learned my addr 172.21.2.102:0/1873798379 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:22:17.446 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:17.432 7fdd18be7700 1 -- 172.21.2.102:0/1873798379 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1069633823 0 0) 0x7fdd08001180 con 0x7fdd140d1c70 2021-02-10T11:22:17.446 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:17.432 7fdd1a3ea700 1 --2- 172.21.2.102:0/1873798379 >> v2:172.21.2.113:3300/0 conn(0x7fdd1410aff0 0x7fdd140cf750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:17.446 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:17.432 7fdd18be7700 1 -- 172.21.2.102:0/1873798379 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd00002fe0 con 0x7fdd140d1c70 2021-02-10T11:22:17.446 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:17.433 7fdd18be7700 1 -- 172.21.2.102:0/1873798379 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1595679025 0 0) 0x7fdd08001180 con 0x7fdd140d1c70 2021-02-10T11:22:17.446 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:17.433 7fdd18be7700 1 -- 172.21.2.102:0/1873798379 >> v2:172.21.2.113:3300/0 conn(0x7fdd1410aff0 msgr2=0x7fdd140cf750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:17.447 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:17.433 7fdd18be7700 1 --2- 172.21.2.102:0/1873798379 >> v2:172.21.2.113:3300/0 conn(0x7fdd1410aff0 0x7fdd140cf750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:17.447 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:17.433 7fdd18be7700 1 -- 172.21.2.102:0/1873798379 >> v1:172.21.2.113:6789/0 conn(0x7fdd140d1870 legacy=0x7fdd141b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:17.447 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:17.433 7fdd18be7700 1 -- 172.21.2.102:0/1873798379 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd1410b3f0 con 0x7fdd140d1c70 2021-02-10T11:22:17.447 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:17.433 7fdd18be7700 1 -- 172.21.2.102:0/1873798379 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7fdd08001680 con 0x7fdd140d1c70 2021-02-10T11:22:17.447 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:17.433 7fdd18be7700 1 -- 172.21.2.102:0/1873798379 >> v1:172.21.2.102:6789/0 conn(0x7fdd140d1c70 legacy=0x7fdd140cef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:17.447 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:17.433 7fdd18be7700 1 --2- 172.21.2.102:0/1873798379 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fdd00003ac0 0x7fdd00005f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:17.447 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:17.433 7fdd18be7700 1 --2- 172.21.2.102:0/1873798379 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdd00006800 0x7fdd00008d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:17.447 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:17.433 7fdd18be7700 1 --2- 172.21.2.102:0/1873798379 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdd00009620 0x7fdd0000bb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:17.447 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:17.433 7fdd18be7700 1 -- 172.21.2.102:0/1873798379 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd0000c310 con 0x7fdd00009620 2021-02-10T11:22:17.447 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:17.433 7fdd18be7700 1 -- 172.21.2.102:0/1873798379 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd0000c530 con 0x7fdd00003ac0 2021-02-10T11:22:17.447 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:17.433 7fdd18be7700 1 -- 172.21.2.102:0/1873798379 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd0000c750 con 0x7fdd00006800 2021-02-10T11:22:17.447 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:17.433 7fdd1a3ea700 1 --2- 172.21.2.102:0/1873798379 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdd00009620 0x7fdd0000bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:17.447 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:17.433 7fdd193e8700 1 --2- 172.21.2.102:0/1873798379 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdd00006800 0x7fdd00008d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:17.448 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:17.433 7fdd19be9700 1 --2- 172.21.2.102:0/1873798379 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fdd00003ac0 0x7fdd00005f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:17.448 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:17.433 7fdd1a3ea700 1 -- 172.21.2.102:0/1873798379 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdd00006800 msgr2=0x7fdd00008d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:17.448 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:17.433 7fdd1a3ea700 1 --2- 172.21.2.102:0/1873798379 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdd00006800 0x7fdd00008d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:17.448 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:17.433 7fdd1a3ea700 1 -- 172.21.2.102:0/1873798379 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fdd00003ac0 msgr2=0x7fdd00005f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:17.448 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:17.433 7fdd1a3ea700 1 --2- 172.21.2.102:0/1873798379 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fdd00003ac0 0x7fdd00005f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:17.448 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:17.433 7fdd1a3ea700 1 -- 172.21.2.102:0/1873798379 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd00050990 con 0x7fdd00009620 2021-02-10T11:22:17.448 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:17.433 7fdd1a3ea700 1 -- 172.21.2.102:0/1873798379 reap_dead start 2021-02-10T11:22:17.448 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:17.433 7fdd1a3ea700 1 --2- 172.21.2.102:0/1873798379 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdd00009620 0x7fdd0000bb40 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 rx=0x7fdd0c006a40 tx=0x7fdd0c00c0c0).ready entity=mon.0 client_cookie=e3d910daf0626a82 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:17.448 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:17.433 7fdd18be7700 1 -- 172.21.2.102:0/1873798379 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdd0c006a80 con 0x7fdd00009620 2021-02-10T11:22:17.448 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:17.433 7fdd18be7700 1 -- 172.21.2.102:0/1873798379 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd0c007070 con 0x7fdd00009620 2021-02-10T11:22:17.448 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:17.433 7fdd18be7700 1 -- 172.21.2.102:0/1873798379 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdd0c0074c0 con 0x7fdd00009620 2021-02-10T11:22:17.448 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:17.434 7fdd1bdcd700 1 -- 172.21.2.102:0/1873798379 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdd00009620 msgr2=0x7fdd0000bb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:17.449 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:17.434 7fdd1bdcd700 1 --2- 172.21.2.102:0/1873798379 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdd00009620 0x7fdd0000bb40 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 rx=0x7fdd0c006a40 tx=0x7fdd0c00c0c0).stop 2021-02-10T11:22:17.449 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:17.434 7fdd1bdcd700 1 -- 172.21.2.102:0/1873798379 shutdown_connections 2021-02-10T11:22:17.449 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:17.434 7fdd1bdcd700 1 --2- 172.21.2.102:0/1873798379 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdd00009620 0x7fdd0000bb40 unknown :-1 s=CLOSED pgs=1034 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:17.449 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:17.434 7fdd1bdcd700 1 -- 172.21.2.102:0/1873798379 >> 172.21.2.102:0/1873798379 conn(0x7fdd140ce420 msgr2=0x7fdd140d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:17.449 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:17.434 7fdd1bdcd700 1 -- 172.21.2.102:0/1873798379 shutdown_connections 2021-02-10T11:22:17.449 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:17.434 7fdd1bdcd700 1 -- 172.21.2.102:0/1873798379 wait complete. 2021-02-10T11:22:17.449 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:17.449 7fdd1bdcd700 1 Processor -- start 2021-02-10T11:22:17.449 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:17.449 7fdd1bdcd700 1 -- start start 2021-02-10T11:22:17.451 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:17.449 7fdd1bdcd700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdd140b8070 0x7fdd140b77c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:17.451 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:17.449 7fdd1bdcd700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdd140ba4a0 0x7fdd140ba8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:17.451 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:17.449 7fdd1bdcd700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fdd141a2fd0 0x7fdd141a33d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:17.451 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:17.449 7fdd1bdcd700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd140baec0 con 0x7fdd140ba4a0 2021-02-10T11:22:17.451 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:17.449 7fdd1bdcd700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd141a39f0 con 0x7fdd141a2fd0 2021-02-10T11:22:17.451 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:17.449 7fdd1bdcd700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd141a3c10 con 0x7fdd140b8070 2021-02-10T11:22:17.451 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:17.449 7fdd193e8700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdd140ba4a0 0x7fdd140ba8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:17.452 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:17.449 7fdd193e8700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdd140ba4a0 0x7fdd140ba8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:55276/0 (socket says 172.21.2.102:55276) 2021-02-10T11:22:17.452 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:17.449 7fdd193e8700 1 -- 172.21.2.102:0/1344346286 learned_addr learned my addr 172.21.2.102:0/1344346286 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:22:17.452 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:17.450 7fdd19be9700 1 --2- 172.21.2.102:0/1344346286 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdd140b8070 0x7fdd140b77c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:17.452 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:17.450 7fdd1a3ea700 1 --2- 172.21.2.102:0/1344346286 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fdd141a2fd0 0x7fdd141a33d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:17.452 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:17.450 7fdd193e8700 1 -- 172.21.2.102:0/1344346286 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdd140b8070 msgr2=0x7fdd140b77c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:17.452 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:17.450 7fdd193e8700 1 --2- 172.21.2.102:0/1344346286 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdd140b8070 0x7fdd140b77c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:17.452 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:17.450 7fdd193e8700 1 -- 172.21.2.102:0/1344346286 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fdd141a2fd0 msgr2=0x7fdd141a33d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:17.452 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:17.450 7fdd193e8700 1 --2- 172.21.2.102:0/1344346286 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fdd141a2fd0 0x7fdd141a33d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:17.452 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:17.450 7fdd193e8700 1 -- 172.21.2.102:0/1344346286 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd140ad700 con 0x7fdd140ba4a0 2021-02-10T11:22:17.452 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:17.450 7fdd193e8700 1 --2- 172.21.2.102:0/1344346286 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdd140ba4a0 0x7fdd140ba8a0 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 rx=0x7fdd08007fc0 tx=0x7fdd0800a900).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:17.452 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:17.450 7fdd067fc700 1 -- 172.21.2.102:0/1344346286 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdd08001180 con 0x7fdd140ba4a0 2021-02-10T11:22:17.452 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:17.450 7fdd067fc700 1 -- 172.21.2.102:0/1344346286 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd080043a0 con 0x7fdd140ba4a0 2021-02-10T11:22:17.453 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:17.450 7fdd067fc700 1 -- 172.21.2.102:0/1344346286 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdd08004800 con 0x7fdd140ba4a0 2021-02-10T11:22:17.453 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:17.450 7fdd1bdcd700 1 -- 172.21.2.102:0/1344346286 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd140ad700 con 0x7fdd140ba4a0 2021-02-10T11:22:17.453 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:17.451 7fdd1bdcd700 1 -- 172.21.2.102:0/1344346286 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd140ad700 con 0x7fdd140ba4a0 2021-02-10T11:22:17.453 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:17.451 7fdd067fc700 1 -- 172.21.2.102:0/1344346286 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fdd08001160 con 0x7fdd140ba4a0 2021-02-10T11:22:17.453 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:17.451 7fdd067fc700 1 --2- 172.21.2.102:0/1344346286 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fdcfc0186c0 0x7fdcfc01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:17.453 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:17.451 7fdd067fc700 1 -- 172.21.2.102:0/1344346286 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5659+0+0 (secure 0 0 0) 0x7fdd08032640 con 0x7fdd140ba4a0 2021-02-10T11:22:17.453 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:17.451 7fdd19be9700 1 --2- 172.21.2.102:0/1344346286 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fdcfc0186c0 0x7fdcfc01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:17.453 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:17.452 7fdd19be9700 1 --2- 172.21.2.102:0/1344346286 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fdcfc0186c0 0x7fdcfc01abc0 crc :-1 s=READY pgs=501 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:17.478 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:17.481 7fdd1bdcd700 1 -- 172.21.2.102:0/1344346286 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd140828e0 con 0x7fdd140ba4a0 2021-02-10T11:22:17.480 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:17.484 7fdd067fc700 1 -- 172.21.2.102:0/1344346286 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fdd08007070 con 0x7fdd140ba4a0 2021-02-10T11:22:17.607 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:17.610 7fdd1bdcd700 1 -- 172.21.2.102:0/1344346286 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"pool2": "cephfs_metadata", "prefix": "osd pool delete", "pool": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fdd140828e0 con 0x7fdd140ba4a0 2021-02-10T11:22:17.990 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:17.993 7fdd067fc700 1 -- 172.21.2.102:0/1344346286 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"pool2": "cephfs_metadata", "prefix": "osd pool delete", "pool": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v32) v1 ==== 185+0+0 (secure 0 0 0) 0x7fdd08001160 con 0x7fdd140ba4a0 2021-02-10T11:22:18.024 INFO:teuthology.orchestra.run.gibba002.stderr:pool 'cephfs_metadata' removed 2021-02-10T11:22:18.026 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:18.029 7fdd1bdcd700 1 -- 172.21.2.102:0/1344346286 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fdcfc0186c0 msgr2=0x7fdcfc01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:18.026 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:18.029 7fdd1bdcd700 1 --2- 172.21.2.102:0/1344346286 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fdcfc0186c0 0x7fdcfc01abc0 crc :-1 s=READY pgs=501 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:18.026 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:18.029 7fdd1bdcd700 1 -- 172.21.2.102:0/1344346286 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdd140ba4a0 msgr2=0x7fdd140ba8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:18.026 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:18.030 7fdd1bdcd700 1 --2- 172.21.2.102:0/1344346286 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdd140ba4a0 0x7fdd140ba8a0 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 rx=0x7fdd08007fc0 tx=0x7fdd0800a900).stop 2021-02-10T11:22:18.027 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:18.030 7fdd1bdcd700 1 -- 172.21.2.102:0/1344346286 shutdown_connections 2021-02-10T11:22:18.027 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:18.030 7fdd1bdcd700 1 --2- 172.21.2.102:0/1344346286 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fdcfc0186c0 0x7fdcfc01abc0 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:18.027 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:18.030 7fdd1bdcd700 1 --2- 172.21.2.102:0/1344346286 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fdd141a2fd0 0x7fdd141a33d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:18.028 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:18.030 7fdd1bdcd700 1 --2- 172.21.2.102:0/1344346286 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdd140ba4a0 0x7fdd140ba8a0 unknown :-1 s=CLOSED pgs=1035 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:18.028 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:18.030 7fdd1bdcd700 1 --2- 172.21.2.102:0/1344346286 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdd140b8070 0x7fdd140b77c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:18.028 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:18.030 7fdd1bdcd700 1 -- 172.21.2.102:0/1344346286 >> 172.21.2.102:0/1344346286 conn(0x7fdd140b58d0 msgr2=0x7fdd140b5010 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:18.028 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:18.030 7fdd1bdcd700 1 -- 172.21.2.102:0/1344346286 shutdown_connections 2021-02-10T11:22:18.028 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:18.030 7fdd1bdcd700 1 -- 172.21.2.102:0/1344346286 wait complete. 2021-02-10T11:22:18.035 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool delete cephfs_data cephfs_data --yes-i-really-really-mean-it 2021-02-10T11:22:18.464 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:18.451 7f70cba76700 1 Processor -- start 2021-02-10T11:22:18.465 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:18.451 7f70cba76700 1 -- start start 2021-02-10T11:22:18.465 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:18.451 7f70cba76700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f70c40d1870 0x7f70c41b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:18.465 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:18.451 7f70cba76700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f70c40cff70 0x7f70c4109130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:18.465 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:18.451 7f70cba76700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70c40fe1c0 con 0x7f70c41b0ae0 2021-02-10T11:22:18.465 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:18.451 7f70cba76700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70c4109ad0 con 0x7f70c40d1870 2021-02-10T11:22:18.465 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:18.451 7f70cba76700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70c4109750 con 0x7f70c40cff70 2021-02-10T11:22:18.465 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:18.452 7f70c9892700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f70c40d1870 0x7f70c41b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:18.465 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:18.452 7f70c9892700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f70c40d1870 0x7f70c41b04c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:55281/0 (socket says 172.21.2.102:55281) 2021-02-10T11:22:18.465 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:18.452 7f70c9892700 1 -- 172.21.2.102:0/985887536 learned_addr learned my addr 172.21.2.102:0/985887536 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:22:18.465 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:18.452 7f70c9892700 1 -- 172.21.2.102:0/985887536 >> v2:172.21.2.113:3300/0 conn(0x7f70c40cff70 msgr2=0x7f70c4109130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:18.465 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:18.452 7f70c9892700 1 --2- 172.21.2.102:0/985887536 >> v2:172.21.2.113:3300/0 conn(0x7f70c40cff70 0x7f70c4109130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:18.466 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:18.452 7f70c9892700 1 -- 172.21.2.102:0/985887536 >> v1:172.21.2.113:6789/0 conn(0x7f70c41b0ae0 legacy=0x7f70c4108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:18.466 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:18.452 7f70c9892700 1 -- 172.21.2.102:0/985887536 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70c4088020 con 0x7f70c40d1870 2021-02-10T11:22:18.466 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:18.452 7f70c9892700 1 --2- 172.21.2.102:0/985887536 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f70c40d1870 0x7f70c41b04c0 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 rx=0x7f70c0006700 tx=0x7f70c0009b80).ready entity=mon.0 client_cookie=b262dd3ea7cdb3bc server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:18.466 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:18.453 7f70c8890700 1 -- 172.21.2.102:0/985887536 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f70c000c0c0 con 0x7f70c40d1870 2021-02-10T11:22:18.466 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:18.453 7f70c8890700 1 -- 172.21.2.102:0/985887536 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70c000c440 con 0x7f70c40d1870 2021-02-10T11:22:18.466 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:18.453 7f70c8890700 1 -- 172.21.2.102:0/985887536 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f70c000c960 con 0x7f70c40d1870 2021-02-10T11:22:18.466 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:18.453 7f70cba76700 1 -- 172.21.2.102:0/985887536 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f70c40d1870 msgr2=0x7f70c41b04c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:18.466 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:18.453 7f70cba76700 1 --2- 172.21.2.102:0/985887536 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f70c40d1870 0x7f70c41b04c0 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 rx=0x7f70c0006700 tx=0x7f70c0009b80).stop 2021-02-10T11:22:18.466 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:18.453 7f70cba76700 1 -- 172.21.2.102:0/985887536 shutdown_connections 2021-02-10T11:22:18.466 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:18.453 7f70cba76700 1 --2- 172.21.2.102:0/985887536 >> v2:172.21.2.113:3300/0 conn(0x7f70c40cff70 0x7f70c4109130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:18.466 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:18.453 7f70cba76700 1 --2- 172.21.2.102:0/985887536 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f70c40d1870 0x7f70c41b04c0 unknown :-1 s=CLOSED pgs=1036 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:18.466 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:18.453 7f70cba76700 1 -- 172.21.2.102:0/985887536 >> 172.21.2.102:0/985887536 conn(0x7f70c40ce420 msgr2=0x7f70c40d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:18.466 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:18.454 7f70cba76700 1 -- 172.21.2.102:0/985887536 shutdown_connections 2021-02-10T11:22:18.467 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:18.454 7f70cba76700 1 -- 172.21.2.102:0/985887536 wait complete. 2021-02-10T11:22:18.468 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:18.468 7f70cba76700 1 Processor -- start 2021-02-10T11:22:18.468 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:18.468 7f70cba76700 1 -- start start 2021-02-10T11:22:18.468 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:18.469 7f70cba76700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f70c41a9930 0x7f70c41a9080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:18.468 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:18.469 7f70cba76700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f70c41abd60 0x7f70c4192a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:18.468 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:18.469 7f70cba76700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f70c40b03e0 0x7f70c40b07e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:18.468 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:18.469 7f70cba76700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70c41932c0 con 0x7f70c41abd60 2021-02-10T11:22:18.469 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:18.469 7f70cba76700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70c4193840 con 0x7f70c40b03e0 2021-02-10T11:22:18.469 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:18.469 7f70cba76700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70c4193a20 con 0x7f70c41a9930 2021-02-10T11:22:18.469 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:18.469 7f70c9091700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f70c41abd60 0x7f70c4192a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:18.469 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:18.469 7f70c9091700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f70c41abd60 0x7f70c4192a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:55286/0 (socket says 172.21.2.102:55286) 2021-02-10T11:22:18.469 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:18.469 7f70c9091700 1 -- 172.21.2.102:0/3669776642 learned_addr learned my addr 172.21.2.102:0/3669776642 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:22:18.469 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:18.469 7f70ca093700 1 --2- 172.21.2.102:0/3669776642 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f70c40b03e0 0x7f70c40b07e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:18.469 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:18.469 7f70c9892700 1 --2- 172.21.2.102:0/3669776642 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f70c41a9930 0x7f70c41a9080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:18.469 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:18.469 7f70c9091700 1 -- 172.21.2.102:0/3669776642 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f70c41a9930 msgr2=0x7f70c41a9080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:18.469 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:18.469 7f70c9091700 1 --2- 172.21.2.102:0/3669776642 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f70c41a9930 0x7f70c41a9080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:18.469 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:18.469 7f70c9091700 1 -- 172.21.2.102:0/3669776642 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f70c40b03e0 msgr2=0x7f70c40b07e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:18.469 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:18.469 7f70c9091700 1 --2- 172.21.2.102:0/3669776642 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f70c40b03e0 0x7f70c40b07e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:18.469 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:18.470 7f70c9091700 1 -- 172.21.2.102:0/3669776642 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70c4193c40 con 0x7f70c41abd60 2021-02-10T11:22:18.469 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:18.470 7f70c9091700 1 --2- 172.21.2.102:0/3669776642 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f70c41abd60 0x7f70c4192a60 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 rx=0x7f70b400b5b0 tx=0x7f70b400b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:18.469 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:18.470 7f70ba7fc700 1 -- 172.21.2.102:0/3669776642 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f70b400bd40 con 0x7f70c41abd60 2021-02-10T11:22:18.470 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:18.470 7f70ba7fc700 1 -- 172.21.2.102:0/3669776642 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70b4000e20 con 0x7f70c41abd60 2021-02-10T11:22:18.470 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:18.470 7f70ba7fc700 1 -- 172.21.2.102:0/3669776642 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f70b40054a0 con 0x7f70c41abd60 2021-02-10T11:22:18.470 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:18.470 7f70cba76700 1 -- 172.21.2.102:0/3669776642 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70c4193c40 con 0x7f70c41abd60 2021-02-10T11:22:18.470 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:18.470 7f70cba76700 1 -- 172.21.2.102:0/3669776642 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70c4193c40 con 0x7f70c41abd60 2021-02-10T11:22:18.470 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:18.471 7f70ba7fc700 1 -- 172.21.2.102:0/3669776642 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f70b4006030 con 0x7f70c41abd60 2021-02-10T11:22:18.470 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:18.471 7f70ba7fc700 1 --2- 172.21.2.102:0/3669776642 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f70ac018730 0x7f70ac01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:18.470 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:18.471 7f70ba7fc700 1 -- 172.21.2.102:0/3669776642 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 5221+0+0 (secure 0 0 0) 0x7f70b4032640 con 0x7f70c41abd60 2021-02-10T11:22:18.470 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:18.471 7f70c9892700 1 --2- 172.21.2.102:0/3669776642 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f70ac018730 0x7f70ac01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:18.470 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:18.471 7f70c9892700 1 --2- 172.21.2.102:0/3669776642 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f70ac018730 0x7f70ac01ac10 crc :-1 s=READY pgs=502 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:18.498 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:18.501 7f70cba76700 1 -- 172.21.2.102:0/3669776642 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70c41937e0 con 0x7f70c41abd60 2021-02-10T11:22:18.501 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:18.504 7f70ba7fc700 1 -- 172.21.2.102:0/3669776642 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f70b4000ce0 con 0x7f70c41abd60 2021-02-10T11:22:18.629 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:18.632 7f70cba76700 1 -- 172.21.2.102:0/3669776642 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"pool2": "cephfs_data", "prefix": "osd pool delete", "pool": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f70c41937e0 con 0x7f70c41abd60 2021-02-10T11:22:18.991 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:18.995 7f70ba7fc700 1 -- 172.21.2.102:0/3669776642 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"pool2": "cephfs_data", "prefix": "osd pool delete", "pool": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v33) v1 ==== 173+0+0 (secure 0 0 0) 0x7f70b4000ef0 con 0x7f70c41abd60 2021-02-10T11:22:18.993 INFO:teuthology.orchestra.run.gibba002.stderr:pool 'cephfs_data' removed 2021-02-10T11:22:18.995 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:18.999 7f70cba76700 1 -- 172.21.2.102:0/3669776642 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f70ac018730 msgr2=0x7f70ac01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:18.995 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:18.999 7f70cba76700 1 --2- 172.21.2.102:0/3669776642 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f70ac018730 0x7f70ac01ac10 crc :-1 s=READY pgs=502 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:18.995 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:18.999 7f70cba76700 1 -- 172.21.2.102:0/3669776642 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f70c41abd60 msgr2=0x7f70c4192a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:18.995 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:18.999 7f70cba76700 1 --2- 172.21.2.102:0/3669776642 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f70c41abd60 0x7f70c4192a60 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 rx=0x7f70b400b5b0 tx=0x7f70b400b950).stop 2021-02-10T11:22:18.996 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:18.999 7f70cba76700 1 -- 172.21.2.102:0/3669776642 shutdown_connections 2021-02-10T11:22:18.996 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:18.999 7f70cba76700 1 --2- 172.21.2.102:0/3669776642 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f70ac018730 0x7f70ac01ac10 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:18.996 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:18.999 7f70cba76700 1 --2- 172.21.2.102:0/3669776642 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f70c40b03e0 0x7f70c40b07e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:18.996 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:18.999 7f70cba76700 1 --2- 172.21.2.102:0/3669776642 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f70c41abd60 0x7f70c4192a60 unknown :-1 s=CLOSED pgs=1037 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:18.996 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:18.999 7f70cba76700 1 --2- 172.21.2.102:0/3669776642 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f70c41a9930 0x7f70c41a9080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:18.996 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:18.999 7f70cba76700 1 -- 172.21.2.102:0/3669776642 >> 172.21.2.102:0/3669776642 conn(0x7f70c419f820 msgr2=0x7f70c4082b10 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:18.996 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:18.999 7f70cba76700 1 -- 172.21.2.102:0/3669776642 shutdown_connections 2021-02-10T11:22:18.997 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:18.999 7f70cba76700 1 -- 172.21.2.102:0/3669776642 wait complete. 2021-02-10T11:22:19.004 INFO:tasks.ceph.mds.a:Restarting daemon 2021-02-10T11:22:19.004 INFO:tasks.ceph.mds.a:Stopping old one... 2021-02-10T11:22:19.004 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2021-02-10T11:22:19.004 INFO:teuthology.orchestra.run:waiting for 300 2021-02-10T11:22:19.005 INFO:tasks.ceph.mds.b:Restarting daemon 2021-02-10T11:22:19.005 INFO:tasks.ceph.mds.b:Stopping old one... 2021-02-10T11:22:19.005 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2021-02-10T11:22:19.005 INFO:teuthology.orchestra.run:waiting for 300 2021-02-10T11:22:19.010 INFO:tasks.ceph.mds.a:Stopped 2021-02-10T11:22:19.011 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2021-02-10T11:22:19.012 INFO:tasks.ceph.mds.b:Stopped 2021-02-10T11:22:19.012 DEBUG:teuthology.orchestra.run.gibba013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2021-02-10T11:22:19.053 INFO:tasks.ceph.mds.a:Started 2021-02-10T11:22:19.053 DEBUG:teuthology.parallel:result is None 2021-02-10T11:22:19.054 INFO:tasks.ceph.mds.b:Started 2021-02-10T11:22:19.055 DEBUG:teuthology.parallel:result is None 2021-02-10T11:22:19.055 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd blacklist clear 2021-02-10T11:22:19.459 INFO:tasks.ceph.mds.b.gibba013.stdout:starting mds.b at 2021-02-10T11:22:19.460 INFO:tasks.ceph.mds.a.gibba002.stdout:starting mds.a at 2021-02-10T11:22:19.484 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:19.470 7f1774772700 1 Processor -- start 2021-02-10T11:22:19.484 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:19.470 7f1774772700 1 -- start start 2021-02-10T11:22:19.484 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:19.471 7f1774772700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f176c0e5470 0x7f176c07e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:19.484 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:19.471 7f1774772700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f176c07f300 0x7f176c07f700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:19.484 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:19.471 7f1774772700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f176c07ed80 con 0x7f176c0e5070 2021-02-10T11:22:19.484 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:19.471 7f1774772700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f176c1b4010 con 0x7f176c07f300 2021-02-10T11:22:19.484 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:19.471 7f1774772700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f176c0d1790 con 0x7f176c0e5470 2021-02-10T11:22:19.484 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:19.471 7f177258e700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f176c0e5070 0x7f176c1b3820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:47530/0 (socket says 172.21.2.102:47530) 2021-02-10T11:22:19.484 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:19.471 7f177258e700 1 -- 172.21.2.102:0/2651749561 learned_addr learned my addr 172.21.2.102:0/2651749561 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:22:19.485 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:19.471 7f1772d8f700 1 --2- 172.21.2.102:0/2651749561 >> v2:172.21.2.102:3300/0 conn(0x7f176c07f300 0x7f176c07f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:19.485 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:19.471 7f1771d8d700 1 --2- 172.21.2.102:0/2651749561 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f176c0e5470 0x7f176c07e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:19.485 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:19.471 7f177158c700 1 -- 172.21.2.102:0/2651749561 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2418036667 0 0) 0x7f1764001500 con 0x7f176c0e5070 2021-02-10T11:22:19.485 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:19.471 7f177158c700 1 -- 172.21.2.102:0/2651749561 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1758002fe0 con 0x7f176c0e5070 2021-02-10T11:22:19.485 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:19.471 7f1772d8f700 1 -- 172.21.2.102:0/2651749561 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f176c0e5470 msgr2=0x7f176c07e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:19.485 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:19.471 7f1772d8f700 1 --2- 172.21.2.102:0/2651749561 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f176c0e5470 0x7f176c07e6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:19.485 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:19.471 7f1772d8f700 1 -- 172.21.2.102:0/2651749561 >> v1:172.21.2.102:6789/0 conn(0x7f176c0e5070 legacy=0x7f176c1b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:19.485 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:19.471 7f1772d8f700 1 -- 172.21.2.102:0/2651749561 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f176c0d1980 con 0x7f176c07f300 2021-02-10T11:22:19.485 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:19.471 7f1771d8d700 1 --2- 172.21.2.102:0/2651749561 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f176c0e5470 0x7f176c07e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:22:19.485 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:19.471 7f1772d8f700 1 --2- 172.21.2.102:0/2651749561 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f176c07f300 0x7f176c07f700 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 rx=0x7f17680068b0 tx=0x7f1768009ac0).ready entity=mon.0 client_cookie=44662add8d790949 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:19.485 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:19.472 7f177158c700 1 -- 172.21.2.102:0/2651749561 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f176800c0c0 con 0x7f176c07f300 2021-02-10T11:22:19.485 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:19.472 7f177158c700 1 -- 172.21.2.102:0/2651749561 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f176800c400 con 0x7f176c07f300 2021-02-10T11:22:19.485 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:19.472 7f177158c700 1 -- 172.21.2.102:0/2651749561 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f176800c920 con 0x7f176c07f300 2021-02-10T11:22:19.486 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:19.472 7f1774772700 1 -- 172.21.2.102:0/2651749561 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f176c07f300 msgr2=0x7f176c07f700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:19.486 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:19.472 7f1774772700 1 --2- 172.21.2.102:0/2651749561 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f176c07f300 0x7f176c07f700 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 rx=0x7f17680068b0 tx=0x7f1768009ac0).stop 2021-02-10T11:22:19.486 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:19.472 7f1774772700 1 -- 172.21.2.102:0/2651749561 shutdown_connections 2021-02-10T11:22:19.486 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:19.472 7f1774772700 1 --2- 172.21.2.102:0/2651749561 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f176c07f300 0x7f176c07f700 unknown :-1 s=CLOSED pgs=1044 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:19.486 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:19.472 7f1774772700 1 --2- 172.21.2.102:0/2651749561 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f176c0e5470 0x7f176c07e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:19.486 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:19.472 7f1774772700 1 -- 172.21.2.102:0/2651749561 >> 172.21.2.102:0/2651749561 conn(0x7f176c0ce690 msgr2=0x7f176c0e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:19.486 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:19.472 7f1774772700 1 -- 172.21.2.102:0/2651749561 shutdown_connections 2021-02-10T11:22:19.486 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:19.472 7f1774772700 1 -- 172.21.2.102:0/2651749561 wait complete. 2021-02-10T11:22:19.488 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:19.487 7f1774772700 1 Processor -- start 2021-02-10T11:22:19.488 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:19.487 7f1774772700 1 -- start start 2021-02-10T11:22:19.488 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:19.488 7f1774772700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f176c0be120 0x7f176c0be720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:19.488 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:19.488 7f1774772700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f176c0896d0 0x7f176c198b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:19.488 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:19.488 7f1774772700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f176c199150 0x7f176c0ba680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:19.488 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:19.488 7f1774772700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f176c0baca0 con 0x7f176c199150 2021-02-10T11:22:19.488 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:19.488 7f1774772700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f176c0baec0 con 0x7f176c0be120 2021-02-10T11:22:19.488 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:19.488 7f1774772700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f176c089ad0 con 0x7f176c0896d0 2021-02-10T11:22:19.488 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:19.488 7f1772d8f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f176c199150 0x7f176c0ba680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:19.489 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:19.488 7f1772d8f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f176c199150 0x7f176c0ba680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:55314/0 (socket says 172.21.2.102:55314) 2021-02-10T11:22:19.489 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:19.488 7f1772d8f700 1 -- 172.21.2.102:0/2387807197 learned_addr learned my addr 172.21.2.102:0/2387807197 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:22:19.489 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:19.488 7f1771d8d700 1 --2- 172.21.2.102:0/2387807197 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f176c0896d0 0x7f176c198b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:19.489 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:19.488 7f177258e700 1 --2- 172.21.2.102:0/2387807197 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f176c0be120 0x7f176c0be720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:19.489 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:19.488 7f1772d8f700 1 -- 172.21.2.102:0/2387807197 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f176c0896d0 msgr2=0x7f176c198b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:19.489 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:19.488 7f1772d8f700 1 --2- 172.21.2.102:0/2387807197 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f176c0896d0 0x7f176c198b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:19.489 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:19.488 7f1772d8f700 1 -- 172.21.2.102:0/2387807197 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f176c0be120 msgr2=0x7f176c0be720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:19.489 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:19.488 7f1772d8f700 1 --2- 172.21.2.102:0/2387807197 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f176c0be120 0x7f176c0be720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:19.489 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:19.488 7f1772d8f700 1 -- 172.21.2.102:0/2387807197 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f176c089cf0 con 0x7f176c199150 2021-02-10T11:22:19.489 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:19.488 7f1772d8f700 1 --2- 172.21.2.102:0/2387807197 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f176c199150 0x7f176c0ba680 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 rx=0x7f1768009690 tx=0x7f176800b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:19.489 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:19.489 7f1762ffd700 1 -- 172.21.2.102:0/2387807197 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f176800bd50 con 0x7f176c199150 2021-02-10T11:22:19.489 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:19.489 7f1762ffd700 1 -- 172.21.2.102:0/2387807197 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1768000c50 con 0x7f176c199150 2021-02-10T11:22:19.489 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:19.489 7f1762ffd700 1 -- 172.21.2.102:0/2387807197 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1768005540 con 0x7f176c199150 2021-02-10T11:22:19.490 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:19.489 7f1774772700 1 -- 172.21.2.102:0/2387807197 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f176c089cf0 con 0x7f176c199150 2021-02-10T11:22:19.490 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:19.489 7f1774772700 1 -- 172.21.2.102:0/2387807197 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f176c089cf0 con 0x7f176c199150 2021-02-10T11:22:19.490 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:19.489 7f1762ffd700 1 -- 172.21.2.102:0/2387807197 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1768009320 con 0x7f176c199150 2021-02-10T11:22:19.490 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:19.489 7f1762ffd700 1 --2- 172.21.2.102:0/2387807197 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f17540186c0 0x7f175401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:19.490 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:19.489 7f1762ffd700 1 -- 172.21.2.102:0/2387807197 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 4839+0+0 (secure 0 0 0) 0x7f1768032140 con 0x7f176c199150 2021-02-10T11:22:19.490 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:19.489 7f177258e700 1 --2- 172.21.2.102:0/2387807197 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f17540186c0 0x7f175401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:19.490 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:19.490 7f177258e700 1 --2- 172.21.2.102:0/2387807197 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f17540186c0 0x7f175401abc0 crc :-1 s=READY pgs=503 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:19.516 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:19.519 7f1774772700 1 -- 172.21.2.102:0/2387807197 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f176c107180 con 0x7f176c199150 2021-02-10T11:22:19.519 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:19.522 7f1762ffd700 1 -- 172.21.2.102:0/2387807197 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f1768020f10 con 0x7f176c199150 2021-02-10T11:22:19.657 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:19.661 7f1774772700 1 -- 172.21.2.102:0/2387807197 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd blacklist clear"} v 0) v1 -- 0x7f176c07ed80 con 0x7f176c199150 2021-02-10T11:22:20.000 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:20.003 7f1762ffd700 1 -- 172.21.2.102:0/2387807197 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd blacklist clear"}]=0 removed all blacklist entries v34) v1 ==== 97+0+0 (secure 0 0 0) 0x7f1768020f10 con 0x7f176c199150 2021-02-10T11:22:20.024 INFO:teuthology.orchestra.run.gibba002.stderr: removed all blacklist entries 2021-02-10T11:22:20.026 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:20.029 7f1774772700 1 -- 172.21.2.102:0/2387807197 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f17540186c0 msgr2=0x7f175401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:20.026 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:20.029 7f1774772700 1 --2- 172.21.2.102:0/2387807197 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f17540186c0 0x7f175401abc0 crc :-1 s=READY pgs=503 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:20.026 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:20.029 7f1774772700 1 -- 172.21.2.102:0/2387807197 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f176c199150 msgr2=0x7f176c0ba680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:20.026 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:20.029 7f1774772700 1 --2- 172.21.2.102:0/2387807197 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f176c199150 0x7f176c0ba680 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 rx=0x7f1768009690 tx=0x7f176800b960).stop 2021-02-10T11:22:20.027 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:20.029 7f1774772700 1 -- 172.21.2.102:0/2387807197 shutdown_connections 2021-02-10T11:22:20.027 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:20.029 7f1774772700 1 --2- 172.21.2.102:0/2387807197 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f176c199150 0x7f176c0ba680 unknown :-1 s=CLOSED pgs=1045 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:20.027 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:20.029 7f1774772700 1 --2- 172.21.2.102:0/2387807197 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f176c0896d0 0x7f176c198b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:20.027 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:20.029 7f1774772700 1 --2- 172.21.2.102:0/2387807197 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f17540186c0 0x7f175401abc0 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:20.027 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:20.029 7f1774772700 1 --2- 172.21.2.102:0/2387807197 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f176c0be120 0x7f176c0be720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:20.027 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:20.029 7f1774772700 1 -- 172.21.2.102:0/2387807197 >> 172.21.2.102:0/2387807197 conn(0x7f176c098740 msgr2=0x7f176c0c3590 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:20.027 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:20.030 7f1774772700 1 -- 172.21.2.102:0/2387807197 shutdown_connections 2021-02-10T11:22:20.027 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:20.030 7f1774772700 1 -- 172.21.2.102:0/2387807197 wait complete. 2021-02-10T11:22:20.035 INFO:tasks.cephfs.cephfs_test_case:['0', '1', '2', '3'] 2021-02-10T11:22:20.035 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth ls --format=json-pretty 2021-02-10T11:22:20.462 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:20.447 7f8dcdd85700 1 Processor -- start 2021-02-10T11:22:21.135 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:20.447 7f8dcdd85700 1 -- start start 2021-02-10T11:22:21.135 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:20.448 7f8dcdd85700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8dc80e5470 0x7f8dc807e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:21.135 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:20.448 7f8dcdd85700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8dc807ed80 con 0x7f8dc807f300 2021-02-10T11:22:21.135 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:20.448 7f8dcdd85700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8dc81b4010 con 0x7f8dc80e5070 2021-02-10T11:22:21.135 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:20.448 7f8dcdd85700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8dc80d1790 con 0x7f8dc80e5470 2021-02-10T11:22:21.136 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:20.448 7f8dc7fff700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f8dc807f300 0x7f8dc807f700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:47546/0 (socket says 172.21.2.102:47546) 2021-02-10T11:22:21.136 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:20.448 7f8dc7fff700 1 -- 172.21.2.102:0/2673541776 learned_addr learned my addr 172.21.2.102:0/2673541776 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:22:21.136 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:20.449 7f8dc67fc700 1 -- 172.21.2.102:0/2673541776 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 207930925 0 0) 0x7f8dc0000f90 con 0x7f8dc807f300 2021-02-10T11:22:21.136 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:20.449 7f8dc6ffd700 1 --2- 172.21.2.102:0/2673541776 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8dc80e5470 0x7f8dc807e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:21.136 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:20.449 7f8dc67fc700 1 -- 172.21.2.102:0/2673541776 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8db0002fe0 con 0x7f8dc807f300 2021-02-10T11:22:21.136 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:20.449 7f8dc67fc700 1 -- 172.21.2.102:0/2673541776 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1956458408 0 0) 0x7f8dc0001460 con 0x7f8dc807f300 2021-02-10T11:22:21.136 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:20.449 7f8dc67fc700 1 -- 172.21.2.102:0/2673541776 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8dc80e5470 msgr2=0x7f8dc807e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:21.136 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:20.449 7f8dc67fc700 1 --2- 172.21.2.102:0/2673541776 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8dc80e5470 0x7f8dc807e6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:21.136 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:20.449 7f8dc67fc700 1 -- 172.21.2.102:0/2673541776 >> v1:172.21.2.113:6789/0 conn(0x7f8dc80e5070 legacy=0x7f8dc81b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:21.136 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:20.449 7f8dc67fc700 1 -- 172.21.2.102:0/2673541776 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8dc80d1980 con 0x7f8dc807f300 2021-02-10T11:22:21.136 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:20.449 7f8dc67fc700 1 -- 172.21.2.102:0/2673541776 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7f8dc0001980 con 0x7f8dc807f300 2021-02-10T11:22:21.136 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:20.449 7f8dc67fc700 1 -- 172.21.2.102:0/2673541776 >> v1:172.21.2.102:6789/0 conn(0x7f8dc807f300 legacy=0x7f8dc807f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:21.136 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:20.449 7f8dc67fc700 1 --2- 172.21.2.102:0/2673541776 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8db0003aa0 0x7f8db0005eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:21.137 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:20.449 7f8dc67fc700 1 --2- 172.21.2.102:0/2673541776 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8db0006730 0x7f8db0008c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:21.137 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:20.449 7f8dc67fc700 1 --2- 172.21.2.102:0/2673541776 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8db00094f0 0x7f8db000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:21.137 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:20.449 7f8dc67fc700 1 -- 172.21.2.102:0/2673541776 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8db000c2c0 con 0x7f8db0006730 2021-02-10T11:22:21.137 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:20.449 7f8dc67fc700 1 -- 172.21.2.102:0/2673541776 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8db000c4e0 con 0x7f8db00094f0 2021-02-10T11:22:21.137 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:20.449 7f8dc67fc700 1 -- 172.21.2.102:0/2673541776 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8db000c700 con 0x7f8db0003aa0 2021-02-10T11:22:21.137 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:20.449 7f8dc6ffd700 1 --2- 172.21.2.102:0/2673541776 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8db0006730 0x7f8db0008c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:21.137 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:20.450 7f8dc6ffd700 1 -- 172.21.2.102:0/2673541776 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8db0003aa0 msgr2=0x7f8db0005eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:21.137 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:20.450 7f8dc6ffd700 1 --2- 172.21.2.102:0/2673541776 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8db0003aa0 0x7f8db0005eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:21.137 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:20.450 7f8dc6ffd700 1 -- 172.21.2.102:0/2673541776 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8db00094f0 msgr2=0x7f8db000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:21.137 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:20.450 7f8dc6ffd700 1 --2- 172.21.2.102:0/2673541776 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8db00094f0 0x7f8db000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:21.137 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:20.450 7f8dc6ffd700 1 -- 172.21.2.102:0/2673541776 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8db0050940 con 0x7f8db0006730 2021-02-10T11:22:21.137 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:20.450 7f8dc6ffd700 1 --2- 172.21.2.102:0/2673541776 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8db0006730 0x7f8db0008c50 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 rx=0x7f8dbc009310 tx=0x7f8dbc009a90).ready entity=mon.0 client_cookie=c97b0c5eb4e94df9 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:21.137 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:20.450 7f8dc67fc700 1 -- 172.21.2.102:0/2673541776 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8dbc00c0c0 con 0x7f8db0006730 2021-02-10T11:22:21.138 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:20.450 7f8dc67fc700 1 -- 172.21.2.102:0/2673541776 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8dbc006400 con 0x7f8db0006730 2021-02-10T11:22:21.138 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:20.450 7f8dc67fc700 1 -- 172.21.2.102:0/2673541776 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8dbc00c340 con 0x7f8db0006730 2021-02-10T11:22:21.138 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:20.450 7f8dcdd85700 1 -- 172.21.2.102:0/2673541776 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8db0006730 msgr2=0x7f8db0008c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:21.138 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:20.450 7f8dcdd85700 1 --2- 172.21.2.102:0/2673541776 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8db0006730 0x7f8db0008c50 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 rx=0x7f8dbc009310 tx=0x7f8dbc009a90).stop 2021-02-10T11:22:21.138 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:20.450 7f8dc7fff700 1 -- 172.21.2.102:0/2673541776 reap_dead start 2021-02-10T11:22:21.138 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:20.451 7f8dcdd85700 1 -- 172.21.2.102:0/2673541776 shutdown_connections 2021-02-10T11:22:21.138 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:20.451 7f8dcdd85700 1 -- 172.21.2.102:0/2673541776 >> 172.21.2.102:0/2673541776 conn(0x7f8dc80ce690 msgr2=0x7f8dc80e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:21.138 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:20.451 7f8dcdd85700 1 -- 172.21.2.102:0/2673541776 shutdown_connections 2021-02-10T11:22:21.138 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:20.451 7f8dcdd85700 1 -- 172.21.2.102:0/2673541776 wait complete. 2021-02-10T11:22:21.141 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:20.466 7f8dcdd85700 1 Processor -- start 2021-02-10T11:22:21.141 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:20.466 7f8dcdd85700 1 -- start start 2021-02-10T11:22:21.141 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:20.466 7f8dcdd85700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8dc8194000 0x7f8dc80b0d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:21.141 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:20.466 7f8dcdd85700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8dc80b1330 0x7f8dc8087050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:21.141 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:20.466 7f8dcdd85700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8dc8087670 0x7f8dc8087a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:21.141 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:20.466 7f8dcdd85700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dc809c0e0 con 0x7f8dc80b1330 2021-02-10T11:22:21.141 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:20.466 7f8dcdd85700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dc8194440 con 0x7f8dc8194000 2021-02-10T11:22:21.142 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:20.466 7f8dcdd85700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8dc809c5e0 con 0x7f8dc8087670 2021-02-10T11:22:21.142 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:20.466 7f8dc6ffd700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8dc80b1330 0x7f8dc8087050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:21.142 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:20.467 7f8dc6ffd700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8dc80b1330 0x7f8dc8087050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:55330/0 (socket says 172.21.2.102:55330) 2021-02-10T11:22:21.142 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:20.467 7f8dc6ffd700 1 -- 172.21.2.102:0/1122552894 learned_addr learned my addr 172.21.2.102:0/1122552894 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:22:21.142 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:20.467 7f8dc6ffd700 1 -- 172.21.2.102:0/1122552894 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8dc8087670 msgr2=0x7f8dc8087a70 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:22:21.142 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:20.467 7f8dc6ffd700 1 --2- 172.21.2.102:0/1122552894 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8dc8087670 0x7f8dc8087a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:21.142 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:20.467 7f8dc6ffd700 1 -- 172.21.2.102:0/1122552894 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8dc8194000 msgr2=0x7f8dc80b0d10 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:22:21.142 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:20.467 7f8dc6ffd700 1 --2- 172.21.2.102:0/1122552894 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8dc8194000 0x7f8dc80b0d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:21.142 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:20.467 7f8dc6ffd700 1 -- 172.21.2.102:0/1122552894 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8dc809c7d0 con 0x7f8dc80b1330 2021-02-10T11:22:21.142 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:20.467 7f8dc6ffd700 1 --2- 172.21.2.102:0/1122552894 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8dc80b1330 0x7f8dc8087050 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 rx=0x7f8dbc009670 tx=0x7f8dbc00b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:21.142 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:20.467 7f8daffff700 1 -- 172.21.2.102:0/1122552894 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8dbc0093e0 con 0x7f8dc80b1330 2021-02-10T11:22:21.142 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:20.467 7f8daffff700 1 -- 172.21.2.102:0/1122552894 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8dbc00bd50 con 0x7f8dc80b1330 2021-02-10T11:22:21.142 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:20.467 7f8dcdd85700 1 -- 172.21.2.102:0/1122552894 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8dc809c7d0 con 0x7f8dc80b1330 2021-02-10T11:22:21.143 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:20.467 7f8dcdd85700 1 -- 172.21.2.102:0/1122552894 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8dc809c7d0 con 0x7f8dc80b1330 2021-02-10T11:22:21.143 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:20.467 7f8daffff700 1 -- 172.21.2.102:0/1122552894 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8dbc002270 con 0x7f8dc80b1330 2021-02-10T11:22:21.143 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:20.468 7f8daffff700 1 -- 172.21.2.102:0/1122552894 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f8dbc002020 con 0x7f8dc80b1330 2021-02-10T11:22:21.143 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:20.468 7f8daffff700 1 --2- 172.21.2.102:0/1122552894 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8da80186c0 0x7f8da801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:21.143 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:20.468 7f8daffff700 1 -- 172.21.2.102:0/1122552894 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 4753+0+0 (secure 0 0 0) 0x7f8dbc032280 con 0x7f8dc80b1330 2021-02-10T11:22:21.143 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:20.468 7f8dc77fe700 1 --2- 172.21.2.102:0/1122552894 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8da80186c0 0x7f8da801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:21.143 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:20.469 7f8dc77fe700 1 --2- 172.21.2.102:0/1122552894 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8da80186c0 0x7f8da801abc0 crc :-1 s=READY pgs=504 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:21.143 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:20.498 7f8dcdd85700 1 -- 172.21.2.102:0/1122552894 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8dc8084b20 con 0x7f8dc80b1330 2021-02-10T11:22:21.143 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:20.501 7f8daffff700 1 -- 172.21.2.102:0/1122552894 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f8dbc000c90 con 0x7f8dc80b1330 2021-02-10T11:22:21.143 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:20.634 7f8dcdd85700 1 -- 172.21.2.102:0/1122552894 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f8dc8084b20 con 0x7f8dc80b1330 2021-02-10T11:22:21.143 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:20.634 7f8daffff700 1 -- 172.21.2.102:0/1122552894 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v14) v1 ==== 80+0+4513 (secure 0 0 0) 0x7f8dbc002000 con 0x7f8dc80b1330 2021-02-10T11:22:21.143 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:20.636 7f8dcdd85700 1 -- 172.21.2.102:0/1122552894 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8da80186c0 msgr2=0x7f8da801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:21.143 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:20.636 7f8dcdd85700 1 --2- 172.21.2.102:0/1122552894 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8da80186c0 0x7f8da801abc0 crc :-1 s=READY pgs=504 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:21.144 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:20.636 7f8dcdd85700 1 -- 172.21.2.102:0/1122552894 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8dc80b1330 msgr2=0x7f8dc8087050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:21.144 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:20.636 7f8dcdd85700 1 --2- 172.21.2.102:0/1122552894 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8dc80b1330 0x7f8dc8087050 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 rx=0x7f8dbc009670 tx=0x7f8dbc00b960).stop 2021-02-10T11:22:21.144 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:20.636 7f8dcdd85700 1 -- 172.21.2.102:0/1122552894 shutdown_connections 2021-02-10T11:22:21.144 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:20.636 7f8dcdd85700 1 --2- 172.21.2.102:0/1122552894 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8dc8087670 0x7f8dc8087a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:21.144 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:20.636 7f8dcdd85700 1 --2- 172.21.2.102:0/1122552894 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8dc80b1330 0x7f8dc8087050 unknown :-1 s=CLOSED pgs=1048 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:21.144 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:20.636 7f8dcdd85700 1 --2- 172.21.2.102:0/1122552894 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8da80186c0 0x7f8da801abc0 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:21.144 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:20.636 7f8dcdd85700 1 --2- 172.21.2.102:0/1122552894 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8dc8194000 0x7f8dc80b0d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:21.144 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:20.636 7f8dcdd85700 1 -- 172.21.2.102:0/1122552894 >> 172.21.2.102:0/1122552894 conn(0x7f8dc8092c20 msgr2=0x7f8dc8195070 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:21.144 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:20.637 7f8dcdd85700 1 -- 172.21.2.102:0/1122552894 shutdown_connections 2021-02-10T11:22:21.144 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:20.637 7f8dcdd85700 1 -- 172.21.2.102:0/1122552894 wait complete. 2021-02-10T11:22:21.144 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:22:21.144 INFO:teuthology.orchestra.run.gibba002.stdout:{ 2021-02-10T11:22:21.145 INFO:teuthology.orchestra.run.gibba002.stdout: "auth_dump": [ 2021-02-10T11:22:21.145 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:22:21.145 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "mds.a", 2021-02-10T11:22:21.145 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAevSNgsgyEARAAvr0cUjjCoEHLxaZSZ2HtVA==", 2021-02-10T11:22:21.145 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:22:21.145 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow", 2021-02-10T11:22:21.145 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:22:21.145 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:22:21.145 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:22:21.145 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:22:21.145 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:22:21.145 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:22:21.145 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "mds.b", 2021-02-10T11:22:21.146 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAevSNgwOFMMRAALfb1fbX56ao1TU8ClnTW1A==", 2021-02-10T11:22:21.146 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:22:21.146 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow", 2021-02-10T11:22:21.146 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:22:21.146 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:22:21.146 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:22:21.146 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:22:21.146 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:22:21.146 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:22:21.146 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.0", 2021-02-10T11:22:21.146 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAmvSNgj+bMMBAAD3HUD6fLiMAg+++iNzLJKQ==", 2021-02-10T11:22:21.146 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:22:21.146 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:22:21.147 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:22:21.147 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:22:21.147 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:22:21.147 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:22:21.147 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:22:21.147 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.1", 2021-02-10T11:22:21.147 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQApvSNgnvc+MRAAAOxqlNwiyLHqNITOrYeFKQ==", 2021-02-10T11:22:21.147 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:22:21.147 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:22:21.147 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:22:21.147 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:22:21.147 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:22:21.147 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:22:21.148 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:22:21.148 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.2", 2021-02-10T11:22:21.148 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAsvSNgZ4SbMRAA76IOYWfbiAuBoM/4k81GxA==", 2021-02-10T11:22:21.148 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:22:21.148 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:22:21.148 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:22:21.148 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:22:21.148 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:22:21.148 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:22:21.148 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:22:21.148 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.3", 2021-02-10T11:22:21.148 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAvvSNgFk3rMRAAWCmKKyw0LDQ+UlAHZAITOw==", 2021-02-10T11:22:21.148 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:22:21.149 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:22:21.149 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:22:21.149 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:22:21.149 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:22:21.149 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:22:21.149 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:22:21.149 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.4", 2021-02-10T11:22:21.149 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQA4vSNgpKcNJBAApJMmvj5YxVHPDOmvmQONOg==", 2021-02-10T11:22:21.149 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:22:21.149 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:22:21.149 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:22:21.149 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:22:21.149 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:22:21.149 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:22:21.150 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:22:21.150 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.5", 2021-02-10T11:22:21.150 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQA7vSNgf2ujJBAAfkuYpyb3WTzc8V5kw+Wktg==", 2021-02-10T11:22:21.150 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:22:21.150 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:22:21.150 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:22:21.150 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:22:21.150 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:22:21.150 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:22:21.150 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:22:21.150 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.6", 2021-02-10T11:22:21.150 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQA+vSNgqw9uJRAAFMU0R+UxU9C3o/9byb3e0g==", 2021-02-10T11:22:21.150 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:22:21.150 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:22:21.151 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:22:21.151 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:22:21.151 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:22:21.151 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:22:21.151 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:22:21.151 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.7", 2021-02-10T11:22:21.151 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQBBvSNgvWSrJRAA2KaCHmLpzovRgYzLvyofTQ==", 2021-02-10T11:22:21.151 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:22:21.151 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:22:21.151 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:22:21.151 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:22:21.151 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:22:21.151 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:22:21.152 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:22:21.152 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "client.0", 2021-02-10T11:22:21.152 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAfvSNgZd9XIBAAvyrvl8fdKuHnJZfczfRoFw==", 2021-02-10T11:22:21.152 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:22:21.152 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow", 2021-02-10T11:22:21.152 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow r", 2021-02-10T11:22:21.152 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:22:21.152 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:22:21.152 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:22:21.152 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:22:21.152 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "client.1", 2021-02-10T11:22:21.152 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAfvSNgSyfXNhAAIduPnF7Vd/6HSkJ0MlcMSA==", 2021-02-10T11:22:21.152 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:22:21.152 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow", 2021-02-10T11:22:21.153 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow r", 2021-02-10T11:22:21.153 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:22:21.153 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:22:21.153 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:22:21.153 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:22:21.153 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "client.2", 2021-02-10T11:22:21.153 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAgvSNgiIG/ERAA9ZTamaTI7ZsPSnxBqjplCg==", 2021-02-10T11:22:21.153 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:22:21.153 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow", 2021-02-10T11:22:21.153 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow r", 2021-02-10T11:22:21.153 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:22:21.153 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:22:21.153 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:22:21.153 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:22:21.154 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "client.3", 2021-02-10T11:22:21.154 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAgvSNgOWpOKBAAlXsEB04Q/GvtL06PqRMLeA==", 2021-02-10T11:22:21.154 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:22:21.154 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow", 2021-02-10T11:22:21.154 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow r", 2021-02-10T11:22:21.154 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:22:21.154 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:22:21.154 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:22:21.154 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:22:21.154 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "client.admin", 2021-02-10T11:22:21.154 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAavSNgO3GQBhAAL+NVnYskQRKVWLgyDPY/xw==", 2021-02-10T11:22:21.154 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:22:21.154 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow *", 2021-02-10T11:22:21.154 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:22:21.155 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:22:21.155 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:22:21.155 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:22:21.155 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:22:21.155 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:22:21.155 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "mgr.x", 2021-02-10T11:22:21.155 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAdvSNgXZZgJhAA7dtBSGzH+DsBwnddZpOBJA==", 2021-02-10T11:22:21.155 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:22:21.155 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow *", 2021-02-10T11:22:21.155 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow profile mgr", 2021-02-10T11:22:21.155 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:22:21.155 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:22:21.155 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:22:21.156 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:22:21.156 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "mgr.y", 2021-02-10T11:22:21.156 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAdvSNgVqQFDRAAwo6YEdWZCAr5GdOWVL/Qvg==", 2021-02-10T11:22:21.156 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:22:21.156 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow *", 2021-02-10T11:22:21.156 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow profile mgr", 2021-02-10T11:22:21.156 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:22:21.156 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:22:21.156 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:22:21.156 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:22:21.156 INFO:teuthology.orchestra.run.gibba002.stdout:} 2021-02-10T11:22:21.157 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:22:21.584 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:21.570 7faa8705b700 1 Processor -- start 2021-02-10T11:22:21.584 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:21.570 7faa8705b700 1 -- start start 2021-02-10T11:22:21.584 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:21.571 7faa8705b700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7faa800e5470 0x7faa8007e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:21.584 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:21.571 7faa8705b700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7faa8007f300 0x7faa8007f700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:21.584 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:21.571 7faa8705b700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa8007ed80 con 0x7faa800e5070 2021-02-10T11:22:21.584 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:21.571 7faa8705b700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa801b4010 con 0x7faa800e5470 2021-02-10T11:22:21.585 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:21.571 7faa8705b700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa800d1790 con 0x7faa8007f300 2021-02-10T11:22:21.585 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:21.571 7faa7ffff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7faa800e5470 0x7faa8007e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:21.585 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:21.571 7faa84e77700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7faa800e5070 0x7faa801b3820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:47562/0 (socket says 172.21.2.102:47562) 2021-02-10T11:22:21.585 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:21.571 7faa84e77700 1 -- 172.21.2.102:0/2661297789 learned_addr learned my addr 172.21.2.102:0/2661297789 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:22:21.585 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:21.571 7faa85678700 1 --2- 172.21.2.102:0/2661297789 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7faa8007f300 0x7faa8007f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:21.585 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:21.571 7faa7ffff700 1 -- 172.21.2.102:0/2661297789 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7faa8007f300 msgr2=0x7faa8007f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:21.585 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:21.571 7faa7ffff700 1 --2- 172.21.2.102:0/2661297789 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7faa8007f300 0x7faa8007f700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:21.585 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:21.571 7faa7ffff700 1 -- 172.21.2.102:0/2661297789 >> v1:172.21.2.102:6789/0 conn(0x7faa800e5070 legacy=0x7faa801b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:21.585 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:21.571 7faa7ffff700 1 -- 172.21.2.102:0/2661297789 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa800d19b0 con 0x7faa800e5470 2021-02-10T11:22:21.585 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:21.571 7faa85678700 1 --2- 172.21.2.102:0/2661297789 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7faa8007f300 0x7faa8007f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:22:21.585 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:21.572 7faa7ffff700 1 --2- 172.21.2.102:0/2661297789 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7faa800e5470 0x7faa8007e6a0 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 rx=0x7faa700069d0 tx=0x7faa70009e40).ready entity=mon.0 client_cookie=4cbcab0e3a12f15b server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:21.585 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:21.572 7faa7f7fe700 1 -- 172.21.2.102:0/2661297789 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faa7000b510 con 0x7faa800e5470 2021-02-10T11:22:21.585 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:21.572 7faa7f7fe700 1 -- 172.21.2.102:0/2661297789 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa7000b810 con 0x7faa800e5470 2021-02-10T11:22:21.586 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:21.572 7faa7f7fe700 1 -- 172.21.2.102:0/2661297789 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faa7000bca0 con 0x7faa800e5470 2021-02-10T11:22:21.586 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:21.572 7faa8705b700 1 -- 172.21.2.102:0/2661297789 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7faa800e5470 msgr2=0x7faa8007e6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:21.586 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:21.572 7faa8705b700 1 --2- 172.21.2.102:0/2661297789 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7faa800e5470 0x7faa8007e6a0 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 rx=0x7faa700069d0 tx=0x7faa70009e40).stop 2021-02-10T11:22:21.586 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:21.573 7faa8705b700 1 -- 172.21.2.102:0/2661297789 shutdown_connections 2021-02-10T11:22:21.586 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:21.573 7faa8705b700 1 --2- 172.21.2.102:0/2661297789 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7faa800e5470 0x7faa8007e6a0 unknown :-1 s=CLOSED pgs=1050 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:21.586 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:21.573 7faa8705b700 1 --2- 172.21.2.102:0/2661297789 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7faa8007f300 0x7faa8007f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:21.586 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:21.573 7faa8705b700 1 -- 172.21.2.102:0/2661297789 >> 172.21.2.102:0/2661297789 conn(0x7faa800ce690 msgr2=0x7faa800e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:21.586 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:21.573 7faa8705b700 1 -- 172.21.2.102:0/2661297789 shutdown_connections 2021-02-10T11:22:21.586 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:21.573 7faa8705b700 1 -- 172.21.2.102:0/2661297789 wait complete. 2021-02-10T11:22:21.587 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:21.588 7faa8705b700 1 Processor -- start 2021-02-10T11:22:21.588 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:21.588 7faa8705b700 1 -- start start 2021-02-10T11:22:21.588 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:21.588 7faa8705b700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7faa8008d400 0x7faa80096c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:21.588 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:21.588 7faa8705b700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7faa800b1450 0x7faa800b1850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:21.588 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:21.588 7faa8705b700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7faa800b1e70 0x7faa801b1f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:21.588 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:21.588 7faa8705b700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa80097310 con 0x7faa800b1450 2021-02-10T11:22:21.588 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:21.588 7faa8705b700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa8008d800 con 0x7faa8008d400 2021-02-10T11:22:21.588 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:21.588 7faa8705b700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa8008d9f0 con 0x7faa800b1e70 2021-02-10T11:22:21.588 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:21.588 7faa7ffff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7faa800b1450 0x7faa800b1850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:21.588 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:21.588 7faa7ffff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7faa800b1450 0x7faa800b1850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:55344/0 (socket says 172.21.2.102:55344) 2021-02-10T11:22:21.588 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:21.588 7faa7ffff700 1 -- 172.21.2.102:0/2201250840 learned_addr learned my addr 172.21.2.102:0/2201250840 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:22:21.588 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:21.589 7faa7ffff700 1 -- 172.21.2.102:0/2201250840 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7faa800b1e70 msgr2=0x7faa801b1f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:21.588 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:21.589 7faa7ffff700 1 --2- 172.21.2.102:0/2201250840 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7faa800b1e70 0x7faa801b1f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:21.589 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:21.589 7faa7ffff700 1 -- 172.21.2.102:0/2201250840 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7faa8008d400 msgr2=0x7faa80096c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:21.589 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:21.589 7faa7ffff700 1 --2- 172.21.2.102:0/2201250840 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7faa8008d400 0x7faa80096c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:21.589 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:21.589 7faa7ffff700 1 -- 172.21.2.102:0/2201250840 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa801b25f0 con 0x7faa800b1450 2021-02-10T11:22:21.589 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:21.589 7faa7ffff700 1 --2- 172.21.2.102:0/2201250840 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7faa800b1450 0x7faa800b1850 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 rx=0x7faa70009f40 tx=0x7faa7000bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:21.589 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:21.589 7faa7d7fa700 1 -- 172.21.2.102:0/2201250840 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faa7000c2f0 con 0x7faa800b1450 2021-02-10T11:22:21.589 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:21.589 7faa7d7fa700 1 -- 172.21.2.102:0/2201250840 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa7000c5f0 con 0x7faa800b1450 2021-02-10T11:22:21.589 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:21.589 7faa7d7fa700 1 -- 172.21.2.102:0/2201250840 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faa7000cb10 con 0x7faa800b1450 2021-02-10T11:22:21.589 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:21.589 7faa8705b700 1 -- 172.21.2.102:0/2201250840 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa801b25f0 con 0x7faa800b1450 2021-02-10T11:22:21.589 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:21.589 7faa8705b700 1 -- 172.21.2.102:0/2201250840 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa801b2b80 con 0x7faa800b1450 2021-02-10T11:22:21.589 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:21.590 7faa7d7fa700 1 -- 172.21.2.102:0/2201250840 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7faa7000c2f0 con 0x7faa800b1450 2021-02-10T11:22:21.589 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:21.590 7faa7d7fa700 1 --2- 172.21.2.102:0/2201250840 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7faa640186c0 0x7faa6401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:21.589 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:21.590 7faa7d7fa700 1 -- 172.21.2.102:0/2201250840 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 4753+0+0 (secure 0 0 0) 0x7faa700325c0 con 0x7faa800b1450 2021-02-10T11:22:21.589 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:21.590 7faa84e77700 1 --2- 172.21.2.102:0/2201250840 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7faa640186c0 0x7faa6401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:21.590 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:21.590 7faa84e77700 1 --2- 172.21.2.102:0/2201250840 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7faa640186c0 0x7faa6401abc0 crc :-1 s=READY pgs=505 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:21.617 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:21.620 7faa8705b700 1 -- 172.21.2.102:0/2201250840 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa8007ed80 con 0x7faa800b1450 2021-02-10T11:22:21.620 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:21.623 7faa7d7fa700 1 -- 172.21.2.102:0/2201250840 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7faa7002aff0 con 0x7faa800b1450 2021-02-10T11:22:21.743 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:21.747 7faa8705b700 1 -- 172.21.2.102:0/2201250840 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7faa8007ed80 con 0x7faa800b1450 2021-02-10T11:22:21.744 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:21.747 7faa7d7fa700 1 -- 172.21.2.102:0/2201250840 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v34) v1 ==== 74+0+11144 (secure 0 0 0) 0x7faa7002aff0 con 0x7faa800b1450 2021-02-10T11:22:21.744 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:22:21.745 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":34,"fsid":"68db30ea-dbce-4aef-a6df-f01c6fea0513","created":"2021-02-10 11:03:06.331557","modified":"2021-02-10 11:22:19.998506","last_up_change":"2021-02-10 11:03:32.690562","last_in_change":"2021-02-10 11:03:32.690562","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:03:37.322519","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"be691b4d-a039-408a-af80-9d53f02b6f1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6818","nonce":36088},{"type":"v1","addr":"172.21.2.102:6820","nonce":36088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6822","nonce":36088},{"type":"v1","addr":"172.21.2.102:6824","nonce":36088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6830","nonce":36088},{"type":"v1","addr":"172.21.2.102:6832","nonce":36088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6826","nonce":36088},{"type":"v1","addr":"172.21.2.102:6828","nonce":36088}]},"public_addr":"172.21.2.102:6820/36088","cluster_addr":"172.21.2.102:6824/36088","heartbeat_back_addr":"172.21.2.102:6832/36088","heartbeat_front_addr":"172.21.2.102:6828/36088","state":["exists","up"]},{"osd":1,"uuid":"64e3ccc8-9c69-4d7c-865e-af8e83822635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6819","nonce":36087},{"type":"v1","addr":"172.21.2.102:6821","nonce":36087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6823","nonce":36087},{"type":"v1","addr":"172.21.2.102:6825","nonce":36087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6831","nonce":36087},{"type":"v1","addr":"172.21.2.102:6833","nonce":36087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6827","nonce":36087},{"type":"v1","addr":"172.21.2.102:6829","nonce":36087}]},"public_addr":"172.21.2.102:6821/36087","cluster_addr":"172.21.2.102:6825/36087","heartbeat_back_addr":"172.21.2.102:6833/36087","heartbeat_front_addr":"172.21.2.102:6829/36087","state":["exists","up"]},{"osd":2,"uuid":"153700af-747f-481f-bdf4-d6072d3a842f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6810","nonce":36089},{"type":"v1","addr":"172.21.2.102:6811","nonce":36089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6812","nonce":36089},{"type":"v1","addr":"172.21.2.102:6813","nonce":36089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6816","nonce":36089},{"type":"v1","addr":"172.21.2.102:6817","nonce":36089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6814","nonce":36089},{"type":"v1","addr":"172.21.2.102:6815","nonce":36089}]},"public_addr":"172.21.2.102:6811/36089","cluster_addr":"172.21.2.102:6813/36089","heartbeat_back_addr":"172.21.2.102:6817/36089","heartbeat_front_addr":"172.21.2.102:6815/36089","state":["exists","up"]},{"osd":3,"uuid":"cbc678f4-83a9-45e6-8830-2f7c232b1ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6802","nonce":36090},{"type":"v1","addr":"172.21.2.102:6803","nonce":36090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6804","nonce":36090},{"type":"v1","addr":"172.21.2.102:6805","nonce":36090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6808","nonce":36090},{"type":"v1","addr":"172.21.2.102:6809","nonce":36090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6806","nonce":36090},{"type":"v1","addr":"172.21.2.102:6807","nonce":36090}]},"public_addr":"172.21.2.102:6803/36090","cluster_addr":"172.21.2.102:6805/36090","heartbeat_back_addr":"172.21.2.102:6809/36090","heartbeat_front_addr":"172.21.2.102:6807/36090","state":["exists","up"]},{"osd":4,"uuid":"c01e4008-b258-4f10-afd2-9e7f37214351","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6808","nonce":36368},{"type":"v1","addr":"172.21.2.113:6811","nonce":36368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6814","nonce":36368},{"type":"v1","addr":"172.21.2.113:6818","nonce":36368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6826","nonce":36368},{"type":"v1","addr":"172.21.2.113:6828","nonce":36368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6820","nonce":36368},{"type":"v1","addr":"172.21.2.113:6824","nonce":36368}]},"public_addr":"172.21.2.113:6811/36368","cluster_addr":"172.21.2.113:6818/36368","heartbeat_back_addr":"172.21.2.113:6828/36368","heartbeat_front_addr":"172.21.2.113:6824/36368","state":["exists","up"]},{"osd":5,"uuid":"551f6007-daf8-4a98-8ee0-c2b16bcccb5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6802","nonce":36366},{"type":"v1","addr":"172.21.2.113:6805","nonce":36366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6807","nonce":36366},{"type":"v1","addr":"172.21.2.113:6810","nonce":36366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6819","nonce":36366},{"type":"v1","addr":"172.21.2.113:6822","nonce":36366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6812","nonce":36366},{"type":"v1","addr":"172.21.2.113:6815","nonce":36366}]},"public_addr":"172.21.2.113:6805/36366","cluster_addr":"172.21.2.113:6810/36366","heartbeat_back_addr":"172.21.2.113:6822/36366","heartbeat_front_addr":"172.21.2.113:6815/36366","state":["exists","up"]},{"osd":6,"uuid":"bf67c040-0e50-48be-b102-c3943bc9ebef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6800","nonce":36367},{"type":"v1","addr":"172.21.2.113:6801","nonce":36367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6803","nonce":36367},{"type":"v1","addr":"172.21.2.113:6804","nonce":36367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6813","nonce":36367},{"type":"v1","addr":"172.21.2.113:6816","nonce":36367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6806","nonce":36367},{"type":"v1","addr":"172.21.2.113:6809","nonce":36367}]},"public_addr":"172.21.2.113:6801/36367","cluster_addr":"172.21.2.113:6804/36367","heartbeat_back_addr":"172.21.2.113:6816/36367","heartbeat_front_addr":"172.21.2.113:6809/36367","state":["exists","up"]},{"osd":7,"uuid":"f5461f80-018d-428f-be89-2eb2bcbe7bb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6817","nonce":36365},{"type":"v1","addr":"172.21.2.113:6821","nonce":36365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6823","nonce":36365},{"type":"v1","addr":"172.21.2.113:6825","nonce":36365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6830","nonce":36365},{"type":"v1","addr":"172.21.2.113:6831","nonce":36365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6827","nonce":36365},{"type":"v1","addr":"172.21.2.113:6829","nonce":36365}]},"public_addr":"172.21.2.113:6821/36365","cluster_addr":"172.21.2.113:6825/36365","heartbeat_back_addr":"172.21.2.113:6831/36365","heartbeat_front_addr":"172.21.2.113:6829/36365","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:22:21.746 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:21.749 7faa8705b700 1 -- 172.21.2.102:0/2201250840 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7faa640186c0 msgr2=0x7faa6401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:21.746 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:21.749 7faa8705b700 1 --2- 172.21.2.102:0/2201250840 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7faa640186c0 0x7faa6401abc0 crc :-1 s=READY pgs=505 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:21.746 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:21.749 7faa8705b700 1 -- 172.21.2.102:0/2201250840 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7faa800b1450 msgr2=0x7faa800b1850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:21.746 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:21.749 7faa8705b700 1 --2- 172.21.2.102:0/2201250840 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7faa800b1450 0x7faa800b1850 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 rx=0x7faa70009f40 tx=0x7faa7000bcd0).stop 2021-02-10T11:22:21.746 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:21.749 7faa8705b700 1 -- 172.21.2.102:0/2201250840 shutdown_connections 2021-02-10T11:22:21.747 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:21.749 7faa8705b700 1 --2- 172.21.2.102:0/2201250840 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7faa800b1e70 0x7faa801b1f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:21.747 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:21.749 7faa8705b700 1 --2- 172.21.2.102:0/2201250840 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7faa800b1450 0x7faa800b1850 unknown :-1 s=CLOSED pgs=1051 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:21.747 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:21.749 7faa8705b700 1 --2- 172.21.2.102:0/2201250840 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7faa640186c0 0x7faa6401abc0 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:21.747 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:21.749 7faa8705b700 1 --2- 172.21.2.102:0/2201250840 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7faa8008d400 0x7faa80096c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:21.747 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:21.749 7faa8705b700 1 -- 172.21.2.102:0/2201250840 >> 172.21.2.102:0/2201250840 conn(0x7faa801ac620 msgr2=0x7faa800896e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:21.747 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:21.749 7faa8705b700 1 -- 172.21.2.102:0/2201250840 shutdown_connections 2021-02-10T11:22:21.747 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:21.749 7faa8705b700 1 -- 172.21.2.102:0/2201250840 wait complete. 2021-02-10T11:22:21.754 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:03:37.322519', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2021-02-10T11:22:21.755 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:22:22.186 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:22.171 7ff89d2be700 1 Processor -- start 2021-02-10T11:22:22.186 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:22.172 7ff89d2be700 1 -- start start 2021-02-10T11:22:22.186 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:22.172 7ff89d2be700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7ff8981b0470 0x7ff8981b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:22.186 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:22.172 7ff89d2be700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff8980ef610 0x7ff8980efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:22.187 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:22.172 7ff89d2be700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8980f0b50 con 0x7ff8980f0030 2021-02-10T11:22:22.187 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:22.172 7ff89d2be700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8980e52e0 con 0x7ff8981b0470 2021-02-10T11:22:22.187 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:22.172 7ff89d2be700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8980e54d0 con 0x7ff8980ef610 2021-02-10T11:22:22.187 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:22.174 7ff89787e700 1 --1- >> v1:172.21.2.113:6789/0 conn(0x7ff8980f0030 0x7ff8980f10d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.113:6789/0 says I am v1:172.21.2.102:54898/0 (socket says 172.21.2.102:54898) 2021-02-10T11:22:22.187 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:22.174 7ff89787e700 1 -- 172.21.2.102:0/1294535570 learned_addr learned my addr 172.21.2.102:0/1294535570 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:22:22.187 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:22.174 7ff89707d700 1 --2- 172.21.2.102:0/1294535570 >> v2:172.21.2.113:3300/0 conn(0x7ff8981b0470 0x7ff8981b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:22.187 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:22.174 7ff89687c700 1 --2- 172.21.2.102:0/1294535570 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff8980ef610 0x7ff8980efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:22.187 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:22.174 7ff89607b700 1 -- 172.21.2.102:0/1294535570 <== mon.1 v1:172.21.2.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 146838323 0 0) 0x7ff890001440 con 0x7ff8980f0030 2021-02-10T11:22:22.187 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:22.174 7ff89607b700 1 -- 172.21.2.102:0/1294535570 --> v1:172.21.2.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff884002fe0 con 0x7ff8980f0030 2021-02-10T11:22:22.187 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:22.174 7ff89707d700 1 -- 172.21.2.102:0/1294535570 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff8980ef610 msgr2=0x7ff8980efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:22.187 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:22.174 7ff89707d700 1 --2- 172.21.2.102:0/1294535570 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff8980ef610 0x7ff8980efa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:22.187 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:22.174 7ff89707d700 1 -- 172.21.2.102:0/1294535570 >> v1:172.21.2.113:6789/0 conn(0x7ff8980f0030 legacy=0x7ff8980f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:22.187 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:22.174 7ff89707d700 1 -- 172.21.2.102:0/1294535570 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8980f0430 con 0x7ff8981b0470 2021-02-10T11:22:22.188 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:22.174 7ff89687c700 1 --2- 172.21.2.102:0/1294535570 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff8980ef610 0x7ff8980efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:22:22.188 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:22.174 7ff89707d700 1 --2- 172.21.2.102:0/1294535570 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff8981b0470 0x7ff8981b0870 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 rx=0x7ff880006650 tx=0x7ff880009a80).ready entity=mon.1 client_cookie=c29598374c257cfb server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:22.188 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:22.174 7ff89607b700 1 -- 172.21.2.102:0/1294535570 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff8800009b0 con 0x7ff8981b0470 2021-02-10T11:22:22.188 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:22.174 7ff89607b700 1 -- 172.21.2.102:0/1294535570 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff88000c0c0 con 0x7ff8981b0470 2021-02-10T11:22:22.188 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:22.174 7ff89607b700 1 -- 172.21.2.102:0/1294535570 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff88000c510 con 0x7ff8981b0470 2021-02-10T11:22:22.188 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:22.175 7ff89d2be700 1 -- 172.21.2.102:0/1294535570 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff8981b0470 msgr2=0x7ff8981b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:22.188 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:22.175 7ff89d2be700 1 --2- 172.21.2.102:0/1294535570 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff8981b0470 0x7ff8981b0870 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 rx=0x7ff880006650 tx=0x7ff880009a80).stop 2021-02-10T11:22:22.188 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:22.175 7ff89d2be700 1 -- 172.21.2.102:0/1294535570 shutdown_connections 2021-02-10T11:22:22.188 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:22.175 7ff89d2be700 1 --2- 172.21.2.102:0/1294535570 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff8980ef610 0x7ff8980efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:22.188 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:22.175 7ff89d2be700 1 --2- 172.21.2.102:0/1294535570 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff8981b0470 0x7ff8981b0870 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:22.188 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:22.175 7ff89d2be700 1 -- 172.21.2.102:0/1294535570 >> 172.21.2.102:0/1294535570 conn(0x7ff8980ce3e0 msgr2=0x7ff8980e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:22.188 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:22.175 7ff89d2be700 1 -- 172.21.2.102:0/1294535570 shutdown_connections 2021-02-10T11:22:22.188 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:22.175 7ff89d2be700 1 -- 172.21.2.102:0/1294535570 wait complete. 2021-02-10T11:22:22.188 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:22.190 7ff89d2be700 1 Processor -- start 2021-02-10T11:22:22.190 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:22.190 7ff89d2be700 1 -- start start 2021-02-10T11:22:22.190 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:22.190 7ff89d2be700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff89809f140 0x7ff8980c1940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:22.190 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:22.190 7ff89d2be700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff8980c1f60 0x7ff89809e300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:22.190 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:22.190 7ff89d2be700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff89809e920 0x7ff8980c2960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:22.190 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:22.190 7ff89d2be700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8980c23e0 con 0x7ff89809e920 2021-02-10T11:22:22.190 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:22.190 7ff89d2be700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff89809ed20 con 0x7ff89809f140 2021-02-10T11:22:22.190 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:22.190 7ff89d2be700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff898105d10 con 0x7ff8980c1f60 2021-02-10T11:22:22.190 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:22.190 7ff89787e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff89809e920 0x7ff8980c2960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:22.191 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:22.190 7ff89787e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff89809e920 0x7ff8980c2960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:55354/0 (socket says 172.21.2.102:55354) 2021-02-10T11:22:22.191 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:22.190 7ff89787e700 1 -- 172.21.2.102:0/1470498837 learned_addr learned my addr 172.21.2.102:0/1470498837 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:22:22.191 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:22.190 7ff89687c700 1 --2- 172.21.2.102:0/1470498837 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff8980c1f60 0x7ff89809e300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:22.191 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:22.190 7ff89707d700 1 --2- 172.21.2.102:0/1470498837 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff89809f140 0x7ff8980c1940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:22.191 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:22.191 7ff89787e700 1 -- 172.21.2.102:0/1470498837 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff8980c1f60 msgr2=0x7ff89809e300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:22.191 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:22.191 7ff89787e700 1 --2- 172.21.2.102:0/1470498837 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff8980c1f60 0x7ff89809e300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:22.191 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:22.191 7ff89787e700 1 -- 172.21.2.102:0/1470498837 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff89809f140 msgr2=0x7ff8980c1940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:22.191 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:22.191 7ff89787e700 1 --2- 172.21.2.102:0/1470498837 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff89809f140 0x7ff8980c1940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:22.191 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:22.191 7ff89787e700 1 -- 172.21.2.102:0/1470498837 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8981a4910 con 0x7ff89809e920 2021-02-10T11:22:22.191 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:22.191 7ff89787e700 1 --2- 172.21.2.102:0/1470498837 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff89809e920 0x7ff8980c2960 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 rx=0x7ff89000a940 tx=0x7ff89000ace0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:22.191 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:22.191 7ff88ffff700 1 -- 172.21.2.102:0/1470498837 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff890000f90 con 0x7ff89809e920 2021-02-10T11:22:22.191 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:22.191 7ff88ffff700 1 -- 172.21.2.102:0/1470498837 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8900048b0 con 0x7ff89809e920 2021-02-10T11:22:22.191 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:22.191 7ff88ffff700 1 -- 172.21.2.102:0/1470498837 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff890004d00 con 0x7ff89809e920 2021-02-10T11:22:22.191 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:22.191 7ff89d2be700 1 -- 172.21.2.102:0/1470498837 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8981a4910 con 0x7ff89809e920 2021-02-10T11:22:22.192 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:22.191 7ff89d2be700 1 -- 172.21.2.102:0/1470498837 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8981a4910 con 0x7ff89809e920 2021-02-10T11:22:22.192 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:22.192 7ff88ffff700 1 -- 172.21.2.102:0/1470498837 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ff890004a90 con 0x7ff89809e920 2021-02-10T11:22:22.192 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:22.192 7ff88ffff700 1 --2- 172.21.2.102:0/1470498837 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff87c0186c0 0x7ff87c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:22.192 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:22.192 7ff88ffff700 1 -- 172.21.2.102:0/1470498837 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 4753+0+0 (secure 0 0 0) 0x7ff890032410 con 0x7ff89809e920 2021-02-10T11:22:22.192 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:22.192 7ff89707d700 1 --2- 172.21.2.102:0/1470498837 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff87c0186c0 0x7ff87c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:22.192 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:22.192 7ff89707d700 1 --2- 172.21.2.102:0/1470498837 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff87c0186c0 0x7ff87c01abc0 crc :-1 s=READY pgs=506 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:22.219 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:22.222 7ff89d2be700 1 -- 172.21.2.102:0/1470498837 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8980f0b50 con 0x7ff89809e920 2021-02-10T11:22:22.221 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:22.225 7ff88ffff700 1 -- 172.21.2.102:0/1470498837 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ff890000f90 con 0x7ff89809e920 2021-02-10T11:22:22.350 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:22.353 7ff89d2be700 1 -- 172.21.2.102:0/1470498837 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7ff8980f0b50 con 0x7ff89809e920 2021-02-10T11:22:22.351 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:22.354 7ff88ffff700 1 -- 172.21.2.102:0/1470498837 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v34) v1 ==== 92+0+10 (secure 0 0 0) 0x7ff890004a50 con 0x7ff89809e920 2021-02-10T11:22:22.351 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:22:22.353 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:22.356 7ff89d2be700 1 -- 172.21.2.102:0/1470498837 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff87c0186c0 msgr2=0x7ff87c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:22.353 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:22.356 7ff89d2be700 1 --2- 172.21.2.102:0/1470498837 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff87c0186c0 0x7ff87c01abc0 crc :-1 s=READY pgs=506 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:22.353 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:22.356 7ff89d2be700 1 -- 172.21.2.102:0/1470498837 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff89809e920 msgr2=0x7ff8980c2960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:22.353 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:22.356 7ff89d2be700 1 --2- 172.21.2.102:0/1470498837 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff89809e920 0x7ff8980c2960 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 rx=0x7ff89000a940 tx=0x7ff89000ace0).stop 2021-02-10T11:22:22.353 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:22.356 7ff89d2be700 1 -- 172.21.2.102:0/1470498837 shutdown_connections 2021-02-10T11:22:22.353 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:22.356 7ff89d2be700 1 --2- 172.21.2.102:0/1470498837 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff89809e920 0x7ff8980c2960 unknown :-1 s=CLOSED pgs=1052 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:22.353 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:22.356 7ff89d2be700 1 --2- 172.21.2.102:0/1470498837 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff8980c1f60 0x7ff89809e300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:22.353 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:22.356 7ff89d2be700 1 --2- 172.21.2.102:0/1470498837 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff87c0186c0 0x7ff87c01abc0 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:22.354 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:22.356 7ff89d2be700 1 --2- 172.21.2.102:0/1470498837 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff89809f140 0x7ff8980c1940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:22.354 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:22.356 7ff89d2be700 1 -- 172.21.2.102:0/1470498837 >> 172.21.2.102:0/1470498837 conn(0x7ff89809fc10 msgr2=0x7ff8980b3250 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:22.354 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:22.356 7ff89d2be700 1 -- 172.21.2.102:0/1470498837 shutdown_connections 2021-02-10T11:22:22.354 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:22.356 7ff89d2be700 1 -- 172.21.2.102:0/1470498837 wait complete. 2021-02-10T11:22:22.361 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --format=json-pretty osd lspools 2021-02-10T11:22:22.789 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:22.776 7f8a15998700 1 Processor -- start 2021-02-10T11:22:22.789 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:22.776 7f8a15998700 1 -- start start 2021-02-10T11:22:22.789 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:22.777 7f8a15998700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f8a100e5470 0x7f8a1007e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:22.789 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:22.777 7f8a15998700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8a1007f300 0x7f8a1007f700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:22.789 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:22.777 7f8a15998700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a1007ed80 con 0x7f8a100e5070 2021-02-10T11:22:22.789 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:22.777 7f8a15998700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a101b4010 con 0x7f8a100e5470 2021-02-10T11:22:22.790 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:22.777 7f8a15998700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a100d1790 con 0x7f8a1007f300 2021-02-10T11:22:22.790 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:22.777 7f8a0ed73700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f8a100e5470 0x7f8a1007e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:22.790 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:22.777 7f8a0ed73700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f8a100e5470 0x7f8a1007e6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:55362/0 (socket says 172.21.2.102:55362) 2021-02-10T11:22:22.790 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:22.777 7f8a0ed73700 1 -- 172.21.2.102:0/2690569561 learned_addr learned my addr 172.21.2.102:0/2690569561 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:22:22.790 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:22.777 7f8a0ed73700 1 -- 172.21.2.102:0/2690569561 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8a1007f300 msgr2=0x7f8a1007f700 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:22:22.790 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:22.777 7f8a0ed73700 1 --2- 172.21.2.102:0/2690569561 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8a1007f300 0x7f8a1007f700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:22.790 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:22.777 7f8a0ed73700 1 -- 172.21.2.102:0/2690569561 >> v1:172.21.2.113:6789/0 conn(0x7f8a100e5070 legacy=0x7f8a101b3820 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:22:22.790 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:22.777 7f8a0ed73700 1 -- 172.21.2.102:0/2690569561 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a100d19b0 con 0x7f8a100e5470 2021-02-10T11:22:22.790 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:22.777 7f8a0ed73700 1 --2- 172.21.2.102:0/2690569561 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8a100e5470 0x7f8a1007e6a0 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 rx=0x7f8a04006700 tx=0x7f8a04009b80).ready entity=mon.0 client_cookie=44a10fbd419d90a0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:22.790 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:22.778 7f8a0e572700 1 -- 172.21.2.102:0/2690569561 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8a0400c0c0 con 0x7f8a100e5470 2021-02-10T11:22:22.790 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:22.778 7f8a0e572700 1 -- 172.21.2.102:0/2690569561 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a0400c0c0 con 0x7f8a100e5470 2021-02-10T11:22:22.790 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:22.778 7f8a0e572700 1 -- 172.21.2.102:0/2690569561 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8a0400c340 con 0x7f8a100e5470 2021-02-10T11:22:22.790 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:22.778 7f8a15998700 1 -- 172.21.2.102:0/2690569561 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8a100e5470 msgr2=0x7f8a1007e6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:22.791 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:22.778 7f8a15998700 1 --2- 172.21.2.102:0/2690569561 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8a100e5470 0x7f8a1007e6a0 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 rx=0x7f8a04006700 tx=0x7f8a04009b80).stop 2021-02-10T11:22:22.791 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:22.778 7f8a15998700 1 -- 172.21.2.102:0/2690569561 shutdown_connections 2021-02-10T11:22:22.791 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:22.778 7f8a15998700 1 --2- 172.21.2.102:0/2690569561 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8a1007f300 0x7f8a1007f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:22.791 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:22.778 7f8a15998700 1 --2- 172.21.2.102:0/2690569561 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8a100e5470 0x7f8a1007e6a0 unknown :-1 s=CLOSED pgs=1053 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:22.791 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:22.778 7f8a15998700 1 -- 172.21.2.102:0/2690569561 >> 172.21.2.102:0/2690569561 conn(0x7f8a100ce690 msgr2=0x7f8a100e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:22.791 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:22.778 7f8a15998700 1 -- 172.21.2.102:0/2690569561 shutdown_connections 2021-02-10T11:22:22.791 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:22.778 7f8a15998700 1 -- 172.21.2.102:0/2690569561 wait complete. 2021-02-10T11:22:22.792 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:22.793 7f8a15998700 1 Processor -- start 2021-02-10T11:22:22.792 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:22.793 7f8a15998700 1 -- start start 2021-02-10T11:22:22.792 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:22.793 7f8a15998700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8a100b7a90 0x7f8a100b7e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:22.792 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:22.793 7f8a15998700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8a10084180 0x7f8a10084580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:22.793 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:22.793 7f8a15998700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8a10084ba0 0x7f8a10084fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:22.793 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:22.793 7f8a15998700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a1009d0f0 con 0x7f8a100b7a90 2021-02-10T11:22:22.793 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:22.793 7f8a15998700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a10191ed0 con 0x7f8a10084180 2021-02-10T11:22:22.793 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:22.793 7f8a15998700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a101a9190 con 0x7f8a10084ba0 2021-02-10T11:22:22.793 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:22.793 7f8a0f574700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8a100b7a90 0x7f8a100b7e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:22.793 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:22.793 7f8a0f574700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8a100b7a90 0x7f8a100b7e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:55368/0 (socket says 172.21.2.102:55368) 2021-02-10T11:22:22.793 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:22.793 7f8a0f574700 1 -- 172.21.2.102:0/3636708316 learned_addr learned my addr 172.21.2.102:0/3636708316 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:22:22.793 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:22.794 7f8a0f574700 1 -- 172.21.2.102:0/3636708316 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8a10084ba0 msgr2=0x7f8a10084fa0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:22:22.793 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:22.794 7f8a0f574700 1 --2- 172.21.2.102:0/3636708316 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8a10084ba0 0x7f8a10084fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:22.793 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:22.794 7f8a0f574700 1 -- 172.21.2.102:0/3636708316 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8a10084180 msgr2=0x7f8a10084580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:22:22.793 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:22.794 7f8a0f574700 1 --2- 172.21.2.102:0/3636708316 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8a10084180 0x7f8a10084580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:22.793 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:22.794 7f8a0f574700 1 -- 172.21.2.102:0/3636708316 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a101a93b0 con 0x7f8a100b7a90 2021-02-10T11:22:22.793 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:22.794 7f8a0f574700 1 --2- 172.21.2.102:0/3636708316 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8a100b7a90 0x7f8a100b7e90 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 rx=0x7f8a0000b5b0 tx=0x7f8a0000b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:22.793 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:22.794 7f89f7fff700 1 -- 172.21.2.102:0/3636708316 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8a0000bd40 con 0x7f8a100b7a90 2021-02-10T11:22:22.794 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:22.794 7f89f7fff700 1 -- 172.21.2.102:0/3636708316 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a00006020 con 0x7f8a100b7a90 2021-02-10T11:22:22.794 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:22.794 7f89f7fff700 1 -- 172.21.2.102:0/3636708316 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8a000054a0 con 0x7f8a100b7a90 2021-02-10T11:22:22.794 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:22.794 7f8a15998700 1 -- 172.21.2.102:0/3636708316 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a101a93b0 con 0x7f8a100b7a90 2021-02-10T11:22:22.794 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:22.794 7f8a15998700 1 -- 172.21.2.102:0/3636708316 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a101a93b0 con 0x7f8a100b7a90 2021-02-10T11:22:22.794 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:22.795 7f89f7fff700 1 -- 172.21.2.102:0/3636708316 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f8a00006200 con 0x7f8a100b7a90 2021-02-10T11:22:22.794 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:22.795 7f89f7fff700 1 --2- 172.21.2.102:0/3636708316 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f89f00186c0 0x7f89f001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:22.794 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:22.795 7f89f7fff700 1 -- 172.21.2.102:0/3636708316 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 4753+0+0 (secure 0 0 0) 0x7f8a000322a0 con 0x7f8a100b7a90 2021-02-10T11:22:22.794 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:22.795 7f8a0ed73700 1 --2- 172.21.2.102:0/3636708316 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f89f00186c0 0x7f89f001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:22.794 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:22.795 7f8a0ed73700 1 --2- 172.21.2.102:0/3636708316 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f89f00186c0 0x7f89f001abc0 crc :-1 s=READY pgs=507 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:22.823 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:22.827 7f8a15998700 1 -- 172.21.2.102:0/3636708316 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a101aa850 con 0x7f8a100b7a90 2021-02-10T11:22:22.826 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:22.829 7f89f7fff700 1 -- 172.21.2.102:0/3636708316 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f8a00030d80 con 0x7f8a100b7a90 2021-02-10T11:22:22.950 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:22.953 7f8a15998700 1 -- 172.21.2.102:0/3636708316 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f8a101aa850 con 0x7f8a100b7a90 2021-02-10T11:22:22.950 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:22.953 7f89f7fff700 1 -- 172.21.2.102:0/3636708316 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v34) v1 ==== 84+0+64 (secure 0 0 0) 0x7f8a0000bd40 con 0x7f8a100b7a90 2021-02-10T11:22:22.951 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:22:22.951 INFO:teuthology.orchestra.run.gibba002.stdout:[ 2021-02-10T11:22:22.951 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:22:22.951 INFO:teuthology.orchestra.run.gibba002.stdout: "poolnum": 1, 2021-02-10T11:22:22.951 INFO:teuthology.orchestra.run.gibba002.stdout: "poolname": "rbd" 2021-02-10T11:22:22.951 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:22:22.951 INFO:teuthology.orchestra.run.gibba002.stdout:] 2021-02-10T11:22:22.952 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:22.955 7f8a15998700 1 -- 172.21.2.102:0/3636708316 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f89f00186c0 msgr2=0x7f89f001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:22.952 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:22.955 7f8a15998700 1 --2- 172.21.2.102:0/3636708316 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f89f00186c0 0x7f89f001abc0 crc :-1 s=READY pgs=507 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:22.952 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:22.955 7f8a15998700 1 -- 172.21.2.102:0/3636708316 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8a100b7a90 msgr2=0x7f8a100b7e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:22.952 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:22.955 7f8a15998700 1 --2- 172.21.2.102:0/3636708316 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8a100b7a90 0x7f8a100b7e90 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 rx=0x7f8a0000b5b0 tx=0x7f8a0000b950).stop 2021-02-10T11:22:22.952 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:22.955 7f8a15998700 1 -- 172.21.2.102:0/3636708316 shutdown_connections 2021-02-10T11:22:22.953 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:22.956 7f8a15998700 1 --2- 172.21.2.102:0/3636708316 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8a10084ba0 0x7f8a10084fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:22.953 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:22.956 7f8a15998700 1 --2- 172.21.2.102:0/3636708316 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f89f00186c0 0x7f89f001abc0 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:22.953 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:22.956 7f8a15998700 1 --2- 172.21.2.102:0/3636708316 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8a10084180 0x7f8a10084580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:22.953 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:22.956 7f8a15998700 1 --2- 172.21.2.102:0/3636708316 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8a100b7a90 0x7f8a100b7e90 unknown :-1 s=CLOSED pgs=1054 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:22.953 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:22.956 7f8a15998700 1 -- 172.21.2.102:0/3636708316 >> 172.21.2.102:0/3636708316 conn(0x7f8a101b2dc0 msgr2=0x7f8a10092c70 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:22.953 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:22.956 7f8a15998700 1 -- 172.21.2.102:0/3636708316 shutdown_connections 2021-02-10T11:22:22.953 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:22.956 7f8a15998700 1 -- 172.21.2.102:0/3636708316 wait complete. 2021-02-10T11:22:22.961 INFO:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2021-02-10T11:22:22.961 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool create cephfs_metadata 8 2021-02-10T11:22:23.391 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:23.377 7f51558cf700 1 Processor -- start 2021-02-10T11:22:23.391 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:23.378 7f51558cf700 1 -- start start 2021-02-10T11:22:23.391 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:23.378 7f51558cf700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f51501b0470 0x7f51501b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:23.391 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:23.378 7f51558cf700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f51500f20e0 0x7f51500f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:23.391 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:23.378 7f51558cf700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51500f0b50 con 0x7f51500ef610 2021-02-10T11:22:23.391 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:23.378 7f51558cf700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51500f0820 con 0x7f51501b0470 2021-02-10T11:22:23.391 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:23.378 7f51558cf700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51500f24e0 con 0x7f51500f20e0 2021-02-10T11:22:23.392 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:23.378 7f514f574700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f51501b0470 0x7f51501b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:23.392 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:23.378 7f514ed73700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f51500ef610 0x7f51500efa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:47598/0 (socket says 172.21.2.102:47598) 2021-02-10T11:22:23.392 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:23.378 7f514ed73700 1 -- 172.21.2.102:0/2690049506 learned_addr learned my addr 172.21.2.102:0/2690049506 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:22:23.392 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:23.379 7f514f574700 1 -- 172.21.2.102:0/2690049506 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f51500f20e0 msgr2=0x7f51500f0200 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:22:23.392 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:23.379 7f514f574700 1 --2- 172.21.2.102:0/2690049506 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f51500f20e0 0x7f51500f0200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:23.392 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:23.379 7f514f574700 1 -- 172.21.2.102:0/2690049506 >> v1:172.21.2.102:6789/0 conn(0x7f51500ef610 legacy=0x7f51500efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:23.392 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:23.379 7f514f574700 1 -- 172.21.2.102:0/2690049506 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51500e52e0 con 0x7f51501b0470 2021-02-10T11:22:23.392 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:23.379 7f514f574700 1 --2- 172.21.2.102:0/2690049506 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f51501b0470 0x7f51501b0870 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 rx=0x7f5138006650 tx=0x7f5138009a80).ready entity=mon.0 client_cookie=34f695b47de35d4d server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:23.392 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:23.379 7f514e572700 1 -- 172.21.2.102:0/2690049506 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f513800c0c0 con 0x7f51501b0470 2021-02-10T11:22:23.392 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:23.379 7f514e572700 1 -- 172.21.2.102:0/2690049506 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f513800c340 con 0x7f51501b0470 2021-02-10T11:22:23.392 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:23.379 7f514e572700 1 -- 172.21.2.102:0/2690049506 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f513800c860 con 0x7f51501b0470 2021-02-10T11:22:23.392 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:23.379 7f51558cf700 1 -- 172.21.2.102:0/2690049506 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f51501b0470 msgr2=0x7f51501b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:23.392 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:23.379 7f51558cf700 1 --2- 172.21.2.102:0/2690049506 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f51501b0470 0x7f51501b0870 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 rx=0x7f5138006650 tx=0x7f5138009a80).stop 2021-02-10T11:22:23.393 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:23.380 7f51558cf700 1 -- 172.21.2.102:0/2690049506 shutdown_connections 2021-02-10T11:22:23.393 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:23.380 7f51558cf700 1 --2- 172.21.2.102:0/2690049506 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f51500f20e0 0x7f51500f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:23.393 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:23.380 7f51558cf700 1 --2- 172.21.2.102:0/2690049506 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f51501b0470 0x7f51501b0870 unknown :-1 s=CLOSED pgs=1056 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:23.393 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:23.380 7f51558cf700 1 -- 172.21.2.102:0/2690049506 >> 172.21.2.102:0/2690049506 conn(0x7f51500ce3e0 msgr2=0x7f51500e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:23.393 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:23.380 7f51558cf700 1 -- 172.21.2.102:0/2690049506 shutdown_connections 2021-02-10T11:22:23.393 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:23.380 7f51558cf700 1 -- 172.21.2.102:0/2690049506 wait complete. 2021-02-10T11:22:23.394 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:23.394 7f51558cf700 1 Processor -- start 2021-02-10T11:22:23.394 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:23.395 7f51558cf700 1 -- start start 2021-02-10T11:22:23.395 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:23.395 7f51558cf700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5150094fb0 0x7f515009e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:23.395 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:23.395 7f51558cf700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f51500c2be0 0x7f51500c2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:23.395 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:23.395 7f51558cf700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f51500c3600 0x7f51500a9c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:23.395 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:23.395 7f51558cf700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f515009ebf0 con 0x7f51500c3600 2021-02-10T11:22:23.395 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:23.395 7f51558cf700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51500aa2a0 con 0x7f51500c2be0 2021-02-10T11:22:23.395 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:23.395 7f51558cf700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51500aa4c0 con 0x7f5150094fb0 2021-02-10T11:22:23.395 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:23.395 7f514fd75700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f51500c3600 0x7f51500a9c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:23.395 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:23.395 7f514fd75700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f51500c3600 0x7f51500a9c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:55382/0 (socket says 172.21.2.102:55382) 2021-02-10T11:22:23.395 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:23.395 7f514fd75700 1 -- 172.21.2.102:0/3461818412 learned_addr learned my addr 172.21.2.102:0/3461818412 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:22:23.395 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:23.395 7f514fd75700 1 -- 172.21.2.102:0/3461818412 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5150094fb0 msgr2=0x7f515009e5d0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:22:23.395 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:23.395 7f514fd75700 1 --2- 172.21.2.102:0/3461818412 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5150094fb0 0x7f515009e5d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:23.395 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:23.395 7f514fd75700 1 -- 172.21.2.102:0/3461818412 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f51500c2be0 msgr2=0x7f51500c2fe0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:22:23.395 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:23.395 7f514fd75700 1 --2- 172.21.2.102:0/3461818412 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f51500c2be0 0x7f51500c2fe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:23.396 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:23.395 7f514fd75700 1 -- 172.21.2.102:0/3461818412 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51500aa6e0 con 0x7f51500c3600 2021-02-10T11:22:23.396 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:23.396 7f514fd75700 1 --2- 172.21.2.102:0/3461818412 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f51500c3600 0x7f51500a9c80 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 rx=0x7f5148009960 tx=0x7f514800bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:23.396 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:23.396 7f5147fff700 1 -- 172.21.2.102:0/3461818412 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5148003030 con 0x7f51500c3600 2021-02-10T11:22:23.396 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:23.396 7f5147fff700 1 -- 172.21.2.102:0/3461818412 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5148001310 con 0x7f51500c3600 2021-02-10T11:22:23.396 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:23.396 7f51558cf700 1 -- 172.21.2.102:0/3461818412 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51500aa6e0 con 0x7f51500c3600 2021-02-10T11:22:23.396 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:23.396 7f51558cf700 1 -- 172.21.2.102:0/3461818412 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51500aac70 con 0x7f51500c3600 2021-02-10T11:22:23.396 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:23.396 7f5147fff700 1 -- 172.21.2.102:0/3461818412 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5148001830 con 0x7f51500c3600 2021-02-10T11:22:23.396 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:23.396 7f5147fff700 1 -- 172.21.2.102:0/3461818412 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f51480014f0 con 0x7f51500c3600 2021-02-10T11:22:23.396 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:23.397 7f5147fff700 1 --2- 172.21.2.102:0/3461818412 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f51340186c0 0x7f513401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:23.396 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:23.397 7f5147fff700 1 -- 172.21.2.102:0/3461818412 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 4753+0+0 (secure 0 0 0) 0x7f5148032640 con 0x7f51500c3600 2021-02-10T11:22:23.396 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:23.397 7f514f574700 1 --2- 172.21.2.102:0/3461818412 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f51340186c0 0x7f513401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:23.396 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:23.397 7f514f574700 1 --2- 172.21.2.102:0/3461818412 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f51340186c0 0x7f513401abc0 crc :-1 s=READY pgs=508 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:23.425 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:23.428 7f51558cf700 1 -- 172.21.2.102:0/3461818412 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51500a9750 con 0x7f51500c3600 2021-02-10T11:22:23.427 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:23.431 7f5147fff700 1 -- 172.21.2.102:0/3461818412 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f514802cfc0 con 0x7f51500c3600 2021-02-10T11:22:23.556 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:23.560 7f51558cf700 1 -- 172.21.2.102:0/3461818412 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"} v 0) v1 -- 0x7f51500a9750 con 0x7f51500c3600 2021-02-10T11:22:24.472 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:24.476 7f5147fff700 1 -- 172.21.2.102:0/3461818412 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"}]=0 pool 'cephfs_metadata' created v36) v1 ==== 133+0+0 (secure 0 0 0) 0x7f514802cfc0 con 0x7f51500c3600 2021-02-10T11:22:24.476 INFO:teuthology.orchestra.run.gibba002.stderr:pool 'cephfs_metadata' created 2021-02-10T11:22:24.478 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:24.481 7f51558cf700 1 -- 172.21.2.102:0/3461818412 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f51340186c0 msgr2=0x7f513401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:24.478 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:24.481 7f51558cf700 1 --2- 172.21.2.102:0/3461818412 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f51340186c0 0x7f513401abc0 crc :-1 s=READY pgs=508 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:24.478 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:24.481 7f51558cf700 1 -- 172.21.2.102:0/3461818412 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f51500c3600 msgr2=0x7f51500a9c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:24.479 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:24.481 7f51558cf700 1 --2- 172.21.2.102:0/3461818412 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f51500c3600 0x7f51500a9c80 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 rx=0x7f5148009960 tx=0x7f514800bcf0).stop 2021-02-10T11:22:24.479 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:24.481 7f51558cf700 1 -- 172.21.2.102:0/3461818412 shutdown_connections 2021-02-10T11:22:24.479 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:24.481 7f51558cf700 1 --2- 172.21.2.102:0/3461818412 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f51500c3600 0x7f51500a9c80 unknown :-1 s=CLOSED pgs=1057 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:24.479 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:24.481 7f51558cf700 1 --2- 172.21.2.102:0/3461818412 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f51340186c0 0x7f513401abc0 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:24.479 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:24.481 7f51558cf700 1 --2- 172.21.2.102:0/3461818412 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f51500c2be0 0x7f51500c2fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:24.479 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:24.481 7f51558cf700 1 --2- 172.21.2.102:0/3461818412 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5150094fb0 0x7f515009e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:24.479 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:24.481 7f51558cf700 1 -- 172.21.2.102:0/3461818412 >> 172.21.2.102:0/3461818412 conn(0x7f515009f630 msgr2=0x7f51500a0130 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:24.480 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:24.482 7f51558cf700 1 -- 172.21.2.102:0/3461818412 shutdown_connections 2021-02-10T11:22:24.480 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:24.482 7f51558cf700 1 -- 172.21.2.102:0/3461818412 wait complete. 2021-02-10T11:22:24.487 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool create cephfs_data 8 2021-02-10T11:22:24.926 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:24.912 7f01e016b700 1 Processor -- start 2021-02-10T11:22:24.927 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:24.913 7f01e016b700 1 -- start start 2021-02-10T11:22:24.927 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:24.913 7f01e016b700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f01d81b0470 0x7f01d81b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:24.927 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:24.913 7f01e016b700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f01d80f20e0 0x7f01d80f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:24.927 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:24.913 7f01e016b700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01d80f0b50 con 0x7f01d80ef610 2021-02-10T11:22:24.927 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:24.913 7f01e016b700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01d80f0820 con 0x7f01d81b0470 2021-02-10T11:22:24.927 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:24.913 7f01e016b700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01d80f24e0 con 0x7f01d80f20e0 2021-02-10T11:22:24.927 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:24.913 7f01ddf87700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f01d81b0470 0x7f01d81b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:24.927 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:24.913 7f01ddf87700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f01d81b0470 0x7f01d81b0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:55388/0 (socket says 172.21.2.102:55388) 2021-02-10T11:22:24.927 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:24.913 7f01ddf87700 1 -- 172.21.2.102:0/1016902521 learned_addr learned my addr 172.21.2.102:0/1016902521 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:22:24.927 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:24.914 7f01de788700 1 --2- 172.21.2.102:0/1016902521 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f01d80f20e0 0x7f01d80f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:24.927 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:24.914 7f01dcf85700 1 -- 172.21.2.102:0/1016902521 <== mon.1 v1:172.21.2.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2430980708 0 0) 0x7f01d4000ff0 con 0x7f01d80ef610 2021-02-10T11:22:24.928 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:24.914 7f01de788700 1 -- 172.21.2.102:0/1016902521 >> v2:172.21.2.102:3300/0 conn(0x7f01d81b0470 msgr2=0x7f01d81b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:24.928 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:24.914 7f01de788700 1 --2- 172.21.2.102:0/1016902521 >> v2:172.21.2.102:3300/0 conn(0x7f01d81b0470 0x7f01d81b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:24.928 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:24.914 7f01de788700 1 -- 172.21.2.102:0/1016902521 >> v1:172.21.2.113:6789/0 conn(0x7f01d80ef610 legacy=0x7f01d80efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:24.928 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:24.914 7f01de788700 1 -- 172.21.2.102:0/1016902521 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01d80e52e0 con 0x7f01d80f20e0 2021-02-10T11:22:24.928 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:24.914 7f01ddf87700 1 --2- 172.21.2.102:0/1016902521 >> v2:172.21.2.102:3300/0 conn(0x7f01d81b0470 0x7f01d81b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:22:24.928 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:24.914 7f01de788700 1 --2- 172.21.2.102:0/1016902521 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f01d80f20e0 0x7f01d80f0200 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7f01d0006e50 tx=0x7f01d0009e70).ready entity=mon.2 client_cookie=54923a54461f7746 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:24.928 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:24.914 7f01dcf85700 1 -- 172.21.2.102:0/1016902521 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f01d0006a20 con 0x7f01d80f20e0 2021-02-10T11:22:24.928 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:24.914 7f01dcf85700 1 -- 172.21.2.102:0/1016902521 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01d0006790 con 0x7f01d80f20e0 2021-02-10T11:22:24.928 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:24.914 7f01dcf85700 1 -- 172.21.2.102:0/1016902521 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f01d000b700 con 0x7f01d80f20e0 2021-02-10T11:22:24.928 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:24.915 7f01e016b700 1 -- 172.21.2.102:0/1016902521 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f01d80f20e0 msgr2=0x7f01d80f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:24.928 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:24.915 7f01e016b700 1 --2- 172.21.2.102:0/1016902521 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f01d80f20e0 0x7f01d80f0200 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7f01d0006e50 tx=0x7f01d0009e70).stop 2021-02-10T11:22:24.928 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:24.915 7f01e016b700 1 -- 172.21.2.102:0/1016902521 shutdown_connections 2021-02-10T11:22:24.928 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:24.915 7f01e016b700 1 --2- 172.21.2.102:0/1016902521 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f01d80f20e0 0x7f01d80f0200 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:24.928 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:24.915 7f01e016b700 1 --2- 172.21.2.102:0/1016902521 >> v2:172.21.2.102:3300/0 conn(0x7f01d81b0470 0x7f01d81b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:24.929 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:24.915 7f01e016b700 1 -- 172.21.2.102:0/1016902521 >> 172.21.2.102:0/1016902521 conn(0x7f01d80ce3e0 msgr2=0x7f01d80e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:24.929 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:24.916 7f01e016b700 1 -- 172.21.2.102:0/1016902521 shutdown_connections 2021-02-10T11:22:24.929 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:24.916 7f01e016b700 1 -- 172.21.2.102:0/1016902521 wait complete. 2021-02-10T11:22:24.930 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:24.930 7f01e016b700 1 Processor -- start 2021-02-10T11:22:24.930 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:24.930 7f01e016b700 1 -- start start 2021-02-10T11:22:24.930 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:24.931 7f01e016b700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f01d81a42e0 0x7f01d809ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:24.930 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:24.931 7f01e016b700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f01d81a46e0 0x7f01d809f310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:24.930 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:24.931 7f01e016b700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f01d808ad30 0x7f01d808b130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:24.930 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:24.931 7f01e016b700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01d808a7b0 con 0x7f01d81a46e0 2021-02-10T11:22:24.930 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:24.931 7f01e016b700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01d80a65f0 con 0x7f01d81a42e0 2021-02-10T11:22:24.930 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:24.931 7f01e016b700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01d808a430 con 0x7f01d808ad30 2021-02-10T11:22:24.930 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:24.931 7f01dd786700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f01d81a46e0 0x7f01d809f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:24.930 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:24.931 7f01ddf87700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f01d81a42e0 0x7f01d809ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:24.930 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:24.931 7f01ddf87700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f01d81a42e0 0x7f01d809ecf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.113:3300/0 says I am v2:172.21.2.102:58822/0 (socket says 172.21.2.102:58822) 2021-02-10T11:22:24.930 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:24.931 7f01ddf87700 1 -- 172.21.2.102:0/950630959 learned_addr learned my addr 172.21.2.102:0/950630959 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:22:24.930 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:24.931 7f01dd786700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f01d81a46e0 0x7f01d809f310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:55394/0 (socket says 172.21.2.102:55394) 2021-02-10T11:22:24.931 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:24.931 7f01de788700 1 --2- 172.21.2.102:0/950630959 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f01d808ad30 0x7f01d808b130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:24.931 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:24.931 7f01dd786700 1 -- 172.21.2.102:0/950630959 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f01d808ad30 msgr2=0x7f01d808b130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:24.931 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:24.931 7f01dd786700 1 --2- 172.21.2.102:0/950630959 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f01d808ad30 0x7f01d808b130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:24.931 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:24.931 7f01dd786700 1 -- 172.21.2.102:0/950630959 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f01d81a42e0 msgr2=0x7f01d809ecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:24.931 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:24.931 7f01dd786700 1 --2- 172.21.2.102:0/950630959 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f01d81a42e0 0x7f01d809ecf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:24.931 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:24.931 7f01dd786700 1 -- 172.21.2.102:0/950630959 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01d809fa50 con 0x7f01d81a46e0 2021-02-10T11:22:24.932 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:24.932 7f01dd786700 1 --2- 172.21.2.102:0/950630959 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f01d81a46e0 0x7f01d809f310 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 rx=0x7f01d400a4a0 tx=0x7f01d400a840).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:24.932 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:24.932 7f01ca7fc700 1 -- 172.21.2.102:0/950630959 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f01d400ac30 con 0x7f01d81a46e0 2021-02-10T11:22:24.932 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:24.932 7f01ca7fc700 1 -- 172.21.2.102:0/950630959 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01d4007240 con 0x7f01d81a46e0 2021-02-10T11:22:24.932 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:24.932 7f01e016b700 1 -- 172.21.2.102:0/950630959 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01d809fa50 con 0x7f01d81a46e0 2021-02-10T11:22:24.932 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:24.932 7f01ca7fc700 1 -- 172.21.2.102:0/950630959 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f01d4006540 con 0x7f01d81a46e0 2021-02-10T11:22:24.932 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:24.932 7f01e016b700 1 -- 172.21.2.102:0/950630959 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01d809fa50 con 0x7f01d81a46e0 2021-02-10T11:22:24.932 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:24.933 7f01ca7fc700 1 -- 172.21.2.102:0/950630959 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f01d400ac30 con 0x7f01d81a46e0 2021-02-10T11:22:24.932 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:24.933 7f01ca7fc700 1 --2- 172.21.2.102:0/950630959 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f01c0018730 0x7f01c001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:24.932 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:24.933 7f01ca7fc700 1 -- 172.21.2.102:0/950630959 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5107+0+0 (secure 0 0 0) 0x7f01d4032500 con 0x7f01d81a46e0 2021-02-10T11:22:24.933 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:24.933 7f01ddf87700 1 --2- 172.21.2.102:0/950630959 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f01c0018730 0x7f01c001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:24.933 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:24.933 7f01ddf87700 1 --2- 172.21.2.102:0/950630959 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f01c0018730 0x7f01c001ac10 crc :-1 s=READY pgs=511 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:24.960 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:24.963 7f01e016b700 1 -- 172.21.2.102:0/950630959 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01d8105ae0 con 0x7f01d81a46e0 2021-02-10T11:22:24.962 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:24.966 7f01ca7fc700 1 -- 172.21.2.102:0/950630959 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f01d400ac30 con 0x7f01d81a46e0 2021-02-10T11:22:25.090 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:25.093 7f01e016b700 1 -- 172.21.2.102:0/950630959 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_data"} v 0) v1 -- 0x7f01d80f0b50 con 0x7f01d81a46e0 2021-02-10T11:22:25.479 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:25.482 7f01ca7fc700 1 -- 172.21.2.102:0/950630959 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_data"}]=0 pool 'cephfs_data' created v37) v1 ==== 125+0+0 (secure 0 0 0) 0x7f01d4007030 con 0x7f01d81a46e0 2021-02-10T11:22:25.505 INFO:teuthology.orchestra.run.gibba002.stderr:pool 'cephfs_data' created 2021-02-10T11:22:25.507 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:25.510 7f01e016b700 1 -- 172.21.2.102:0/950630959 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f01c0018730 msgr2=0x7f01c001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:25.507 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:25.510 7f01e016b700 1 --2- 172.21.2.102:0/950630959 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f01c0018730 0x7f01c001ac10 crc :-1 s=READY pgs=511 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:25.508 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:25.510 7f01e016b700 1 -- 172.21.2.102:0/950630959 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f01d81a46e0 msgr2=0x7f01d809f310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:25.508 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:25.510 7f01e016b700 1 --2- 172.21.2.102:0/950630959 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f01d81a46e0 0x7f01d809f310 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 rx=0x7f01d400a4a0 tx=0x7f01d400a840).stop 2021-02-10T11:22:25.508 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:25.510 7f01e016b700 1 -- 172.21.2.102:0/950630959 shutdown_connections 2021-02-10T11:22:25.508 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:25.510 7f01e016b700 1 --2- 172.21.2.102:0/950630959 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f01d808ad30 0x7f01d808b130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:25.508 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:25.510 7f01e016b700 1 --2- 172.21.2.102:0/950630959 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f01d81a46e0 0x7f01d809f310 unknown :-1 s=CLOSED pgs=1058 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:25.508 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:25.510 7f01e016b700 1 --2- 172.21.2.102:0/950630959 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f01c0018730 0x7f01c001ac10 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:25.508 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:25.510 7f01e016b700 1 --2- 172.21.2.102:0/950630959 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f01d81a42e0 0x7f01d809ecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:25.508 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:25.510 7f01e016b700 1 -- 172.21.2.102:0/950630959 >> 172.21.2.102:0/950630959 conn(0x7f01d8095a70 msgr2=0x7f01d80bf050 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:25.508 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:25.510 7f01e016b700 1 -- 172.21.2.102:0/950630959 shutdown_connections 2021-02-10T11:22:25.508 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:25.510 7f01e016b700 1 -- 172.21.2.102:0/950630959 wait complete. 2021-02-10T11:22:25.515 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2021-02-10T11:22:25.950 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:25.936 7fec8bbbd700 1 Processor -- start 2021-02-10T11:22:25.950 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:25.936 7fec8bbbd700 1 -- start start 2021-02-10T11:22:25.950 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:25.937 7fec8bbbd700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fec840d1870 0x7fec841b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:25.950 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:25.937 7fec8bbbd700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fec841b0ae0 0x7fec84108940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:25.950 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:25.937 7fec8bbbd700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec84109a60 con 0x7fec840d1c70 2021-02-10T11:22:25.950 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:25.937 7fec8bbbd700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec840e5ef0 con 0x7fec841b0ae0 2021-02-10T11:22:25.951 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:25.937 7fec8bbbd700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec84109750 con 0x7fec840d1870 2021-02-10T11:22:25.951 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:25.937 7fec891d8700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fec841b0ae0 0x7fec84108940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:25.951 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:25.937 7fec891d8700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fec841b0ae0 0x7fec84108940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:55400/0 (socket says 172.21.2.102:55400) 2021-02-10T11:22:25.951 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:25.937 7fec891d8700 1 -- 172.21.2.102:0/3945799477 learned_addr learned my addr 172.21.2.102:0/3945799477 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:22:25.951 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:25.937 7fec891d8700 1 -- 172.21.2.102:0/3945799477 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fec840d1870 msgr2=0x7fec841b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:25.951 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:25.937 7fec891d8700 1 --2- 172.21.2.102:0/3945799477 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fec840d1870 0x7fec841b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:25.951 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:25.937 7fec891d8700 1 -- 172.21.2.102:0/3945799477 >> v1:172.21.2.113:6789/0 conn(0x7fec840d1c70 legacy=0x7fec84108f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:25.951 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:25.937 7fec891d8700 1 -- 172.21.2.102:0/3945799477 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec84088020 con 0x7fec841b0ae0 2021-02-10T11:22:25.952 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:25.938 7fec891d8700 1 --2- 172.21.2.102:0/3945799477 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fec841b0ae0 0x7fec84108940 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 rx=0x7fec74006700 tx=0x7fec74009b80).ready entity=mon.0 client_cookie=3b92338488a07234 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:25.952 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:25.938 7fec889d7700 1 -- 172.21.2.102:0/3945799477 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fec7400c0c0 con 0x7fec841b0ae0 2021-02-10T11:22:25.952 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:25.938 7fec889d7700 1 -- 172.21.2.102:0/3945799477 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec7400c440 con 0x7fec841b0ae0 2021-02-10T11:22:25.952 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:25.938 7fec889d7700 1 -- 172.21.2.102:0/3945799477 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fec7400c8a0 con 0x7fec841b0ae0 2021-02-10T11:22:25.952 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:25.938 7fec8bbbd700 1 -- 172.21.2.102:0/3945799477 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fec841b0ae0 msgr2=0x7fec84108940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:25.952 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:25.938 7fec8bbbd700 1 --2- 172.21.2.102:0/3945799477 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fec841b0ae0 0x7fec84108940 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 rx=0x7fec74006700 tx=0x7fec74009b80).stop 2021-02-10T11:22:25.952 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:25.939 7fec8bbbd700 1 -- 172.21.2.102:0/3945799477 shutdown_connections 2021-02-10T11:22:25.952 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:25.939 7fec8bbbd700 1 --2- 172.21.2.102:0/3945799477 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fec841b0ae0 0x7fec84108940 unknown :-1 s=CLOSED pgs=1059 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:25.952 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:25.939 7fec8bbbd700 1 --2- 172.21.2.102:0/3945799477 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fec840d1870 0x7fec841b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:25.952 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:25.939 7fec8bbbd700 1 -- 172.21.2.102:0/3945799477 >> 172.21.2.102:0/3945799477 conn(0x7fec840ce420 msgr2=0x7fec840d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:25.952 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:25.939 7fec8bbbd700 1 -- 172.21.2.102:0/3945799477 shutdown_connections 2021-02-10T11:22:25.953 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:25.939 7fec8bbbd700 1 -- 172.21.2.102:0/3945799477 wait complete. 2021-02-10T11:22:25.955 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:25.953 7fec8bbbd700 1 Processor -- start 2021-02-10T11:22:25.955 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:25.953 7fec8bbbd700 1 -- start start 2021-02-10T11:22:25.955 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:25.954 7fec8bbbd700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fec841a9930 0x7fec841a9080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:25.955 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:25.954 7fec8bbbd700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fec841abd60 0x7fec84192a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:25.955 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:25.954 7fec8bbbd700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fec840b03e0 0x7fec840b07e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:25.955 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:25.954 7fec8bbbd700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec841932c0 con 0x7fec841a9930 2021-02-10T11:22:25.956 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:25.954 7fec8bbbd700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec84193840 con 0x7fec841abd60 2021-02-10T11:22:25.956 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:25.954 7fec8bbbd700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec84193a20 con 0x7fec840b03e0 2021-02-10T11:22:25.956 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:25.954 7fec899d9700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fec841a9930 0x7fec841a9080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:25.956 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:25.954 7fec899d9700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fec841a9930 0x7fec841a9080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:55406/0 (socket says 172.21.2.102:55406) 2021-02-10T11:22:25.956 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:25.954 7fec899d9700 1 -- 172.21.2.102:0/4013137211 learned_addr learned my addr 172.21.2.102:0/4013137211 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:22:25.956 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:25.954 7fec899d9700 1 -- 172.21.2.102:0/4013137211 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fec840b03e0 msgr2=0x7fec840b07e0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:22:25.956 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:25.954 7fec899d9700 1 --2- 172.21.2.102:0/4013137211 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fec840b03e0 0x7fec840b07e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:25.956 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:25.954 7fec899d9700 1 -- 172.21.2.102:0/4013137211 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fec841abd60 msgr2=0x7fec84192a60 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:22:25.956 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:25.954 7fec899d9700 1 --2- 172.21.2.102:0/4013137211 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fec841abd60 0x7fec84192a60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:25.956 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:25.954 7fec899d9700 1 -- 172.21.2.102:0/4013137211 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec84193c40 con 0x7fec841a9930 2021-02-10T11:22:25.956 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:25.955 7fec899d9700 1 --2- 172.21.2.102:0/4013137211 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fec841a9930 0x7fec841a9080 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 rx=0x7fec8000b8c0 tx=0x7fec8000bc60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:25.956 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:25.955 7fec7a7fc700 1 -- 172.21.2.102:0/4013137211 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fec8000c2f0 con 0x7fec841a9930 2021-02-10T11:22:25.957 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:25.955 7fec7a7fc700 1 -- 172.21.2.102:0/4013137211 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec8000c570 con 0x7fec841a9930 2021-02-10T11:22:25.957 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:25.955 7fec8bbbd700 1 -- 172.21.2.102:0/4013137211 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec84193c40 con 0x7fec841a9930 2021-02-10T11:22:25.957 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:25.955 7fec7a7fc700 1 -- 172.21.2.102:0/4013137211 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fec8000ca90 con 0x7fec841a9930 2021-02-10T11:22:25.957 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:25.955 7fec8bbbd700 1 -- 172.21.2.102:0/4013137211 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec84193c40 con 0x7fec841a9930 2021-02-10T11:22:25.957 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:25.956 7fec7a7fc700 1 -- 172.21.2.102:0/4013137211 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fec80001010 con 0x7fec841a9930 2021-02-10T11:22:25.957 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:25.956 7fec7a7fc700 1 --2- 172.21.2.102:0/4013137211 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fec6c018730 0x7fec6c01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:25.957 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:25.956 7fec7a7fc700 1 -- 172.21.2.102:0/4013137211 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5457+0+0 (secure 0 0 0) 0x7fec80032ae0 con 0x7fec841a9930 2021-02-10T11:22:25.957 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:25.956 7fec891d8700 1 --2- 172.21.2.102:0/4013137211 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fec6c018730 0x7fec6c01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:25.957 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:25.956 7fec891d8700 1 --2- 172.21.2.102:0/4013137211 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fec6c018730 0x7fec6c01ac10 crc :-1 s=READY pgs=512 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:25.984 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:25.987 7fec8bbbd700 1 -- 172.21.2.102:0/4013137211 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec840828e0 con 0x7fec841a9930 2021-02-10T11:22:25.987 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:25.990 7fec7a7fc700 1 -- 172.21.2.102:0/4013137211 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fec8000bda0 con 0x7fec841a9930 2021-02-10T11:22:26.118 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:26.120 7fec8bbbd700 1 -- 172.21.2.102:0/4013137211 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"data": "cephfs_data", "prefix": "fs new", "force": true, "fs_name": "cephfs", "metadata": "cephfs_metadata"} v 0) v1 -- 0x7fec840828e0 con 0x7fec841a9930 2021-02-10T11:22:26.121 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:26.124 7fec7a7fc700 1 -- 172.21.2.102:0/4013137211 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"data": "cephfs_data", "prefix": "fs new", "force": true, "fs_name": "cephfs", "metadata": "cephfs_metadata"}]=0 new fs with metadata pool 6 and data pool 7 v19) v1 ==== 187+0+0 (secure 0 0 0) 0x7fec80001010 con 0x7fec841a9930 2021-02-10T11:22:26.125 INFO:teuthology.orchestra.run.gibba002.stderr:new fs with metadata pool 6 and data pool 7 2021-02-10T11:22:26.128 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:26.130 7fec8bbbd700 1 -- 172.21.2.102:0/4013137211 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fec6c018730 msgr2=0x7fec6c01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:26.128 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:26.130 7fec8bbbd700 1 --2- 172.21.2.102:0/4013137211 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fec6c018730 0x7fec6c01ac10 crc :-1 s=READY pgs=512 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:26.128 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:26.130 7fec8bbbd700 1 -- 172.21.2.102:0/4013137211 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fec841a9930 msgr2=0x7fec841a9080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:26.128 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:26.130 7fec8bbbd700 1 --2- 172.21.2.102:0/4013137211 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fec841a9930 0x7fec841a9080 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 rx=0x7fec8000b8c0 tx=0x7fec8000bc60).stop 2021-02-10T11:22:26.128 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:26.130 7fec8bbbd700 1 -- 172.21.2.102:0/4013137211 shutdown_connections 2021-02-10T11:22:26.128 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:26.130 7fec8bbbd700 1 --2- 172.21.2.102:0/4013137211 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fec840b03e0 0x7fec840b07e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:26.129 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:26.130 7fec8bbbd700 1 --2- 172.21.2.102:0/4013137211 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fec6c018730 0x7fec6c01ac10 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:26.129 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:26.130 7fec8bbbd700 1 --2- 172.21.2.102:0/4013137211 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fec841abd60 0x7fec84192a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:26.129 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:26.130 7fec8bbbd700 1 --2- 172.21.2.102:0/4013137211 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fec841a9930 0x7fec841a9080 unknown :-1 s=CLOSED pgs=1060 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:26.129 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:26.130 7fec8bbbd700 1 -- 172.21.2.102:0/4013137211 >> 172.21.2.102:0/4013137211 conn(0x7fec8419f820 msgr2=0x7fec84082b10 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:26.129 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:26.130 7fec8bbbd700 1 -- 172.21.2.102:0/4013137211 shutdown_connections 2021-02-10T11:22:26.129 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:26.130 7fec8bbbd700 1 -- 172.21.2.102:0/4013137211 wait complete. 2021-02-10T11:22:26.136 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:22:26.562 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:26.549 7f6f966d9700 1 Processor -- start 2021-02-10T11:22:26.563 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:26.549 7f6f966d9700 1 -- start start 2021-02-10T11:22:26.563 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:26.549 7f6f966d9700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6f900e5470 0x7f6f9007e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:26.563 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:26.549 7f6f966d9700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f9007ed80 con 0x7f6f9007f300 2021-02-10T11:22:26.563 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:26.549 7f6f966d9700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f901b4010 con 0x7f6f900e5070 2021-02-10T11:22:26.563 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:26.549 7f6f966d9700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f900d1790 con 0x7f6f900e5470 2021-02-10T11:22:26.564 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:26.550 7f6f94cf6700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f6f9007f300 0x7f6f9007f700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:47640/0 (socket says 172.21.2.102:47640) 2021-02-10T11:22:26.564 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:26.550 7f6f94cf6700 1 -- 172.21.2.102:0/4099601593 learned_addr learned my addr 172.21.2.102:0/4099601593 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:22:26.564 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:26.550 7f6f8f7fe700 1 --2- 172.21.2.102:0/4099601593 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6f900e5470 0x7f6f9007e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:26.564 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:26.550 7f6f8effd700 1 -- 172.21.2.102:0/4099601593 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2780621894 0 0) 0x7f6f88000f90 con 0x7f6f9007f300 2021-02-10T11:22:26.564 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:26.550 7f6f8effd700 1 -- 172.21.2.102:0/4099601593 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f78002fe0 con 0x7f6f9007f300 2021-02-10T11:22:26.564 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:26.550 7f6f8f7fe700 1 -- 172.21.2.102:0/4099601593 >> v1:172.21.2.113:6789/0 conn(0x7f6f900e5070 legacy=0x7f6f901b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:26.564 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:26.550 7f6f8f7fe700 1 -- 172.21.2.102:0/4099601593 >> v1:172.21.2.102:6789/0 conn(0x7f6f9007f300 legacy=0x7f6f9007f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:26.564 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:26.550 7f6f8f7fe700 1 -- 172.21.2.102:0/4099601593 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f900d1980 con 0x7f6f900e5470 2021-02-10T11:22:26.565 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:26.550 7f6f8f7fe700 1 --2- 172.21.2.102:0/4099601593 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6f900e5470 0x7f6f9007e6a0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f6f84006650 tx=0x7f6f84009a80).ready entity=mon.2 client_cookie=764486c240c63d48 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:26.565 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:26.551 7f6f8effd700 1 -- 172.21.2.102:0/4099601593 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6f8400c0c0 con 0x7f6f900e5470 2021-02-10T11:22:26.565 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:26.551 7f6f8effd700 1 -- 172.21.2.102:0/4099601593 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f8400c340 con 0x7f6f900e5470 2021-02-10T11:22:26.565 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:26.551 7f6f8effd700 1 -- 172.21.2.102:0/4099601593 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6f8400c860 con 0x7f6f900e5470 2021-02-10T11:22:26.565 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:26.551 7f6f966d9700 1 -- 172.21.2.102:0/4099601593 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6f900e5470 msgr2=0x7f6f9007e6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:26.565 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:26.551 7f6f966d9700 1 --2- 172.21.2.102:0/4099601593 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6f900e5470 0x7f6f9007e6a0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f6f84006650 tx=0x7f6f84009a80).stop 2021-02-10T11:22:26.565 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:26.551 7f6f966d9700 1 -- 172.21.2.102:0/4099601593 shutdown_connections 2021-02-10T11:22:26.566 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:26.551 7f6f966d9700 1 --2- 172.21.2.102:0/4099601593 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6f900e5470 0x7f6f9007e6a0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:26.566 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:26.551 7f6f966d9700 1 -- 172.21.2.102:0/4099601593 >> 172.21.2.102:0/4099601593 conn(0x7f6f900ce690 msgr2=0x7f6f900e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:26.566 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:26.552 7f6f966d9700 1 -- 172.21.2.102:0/4099601593 shutdown_connections 2021-02-10T11:22:26.566 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:26.552 7f6f966d9700 1 -- 172.21.2.102:0/4099601593 wait complete. 2021-02-10T11:22:26.570 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:26.566 7f6f966d9700 1 Processor -- start 2021-02-10T11:22:26.570 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:26.566 7f6f966d9700 1 -- start start 2021-02-10T11:22:26.571 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:26.567 7f6f966d9700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6f901a2a90 0x7f6f901aa0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:26.571 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:26.567 7f6f966d9700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6f901aa700 0x7f6f901a3a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:26.571 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:26.567 7f6f966d9700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6f900a55e0 0x7f6f901a4040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:26.571 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:26.567 7f6f966d9700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f901a34a0 con 0x7f6f901aa700 2021-02-10T11:22:26.571 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:26.567 7f6f966d9700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f901a2e90 con 0x7f6f901a2a90 2021-02-10T11:22:26.571 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:26.567 7f6f966d9700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f901a30b0 con 0x7f6f900a55e0 2021-02-10T11:22:26.571 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:26.567 7f6f8f7fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6f901aa700 0x7f6f901a3a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:26.572 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:26.567 7f6f8f7fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6f901aa700 0x7f6f901a3a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:55422/0 (socket says 172.21.2.102:55422) 2021-02-10T11:22:26.572 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:26.567 7f6f8f7fe700 1 -- 172.21.2.102:0/608211424 learned_addr learned my addr 172.21.2.102:0/608211424 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:22:26.572 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:26.567 7f6f8f7fe700 1 -- 172.21.2.102:0/608211424 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6f900a55e0 msgr2=0x7f6f901a4040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:26.573 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:26.567 7f6f8f7fe700 1 --2- 172.21.2.102:0/608211424 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6f900a55e0 0x7f6f901a4040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:26.573 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:26.567 7f6f8f7fe700 1 -- 172.21.2.102:0/608211424 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6f901a2a90 msgr2=0x7f6f901aa0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:26.573 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:26.567 7f6f8f7fe700 1 --2- 172.21.2.102:0/608211424 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6f901a2a90 0x7f6f901aa0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:26.573 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:26.567 7f6f8f7fe700 1 -- 172.21.2.102:0/608211424 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f901a4660 con 0x7f6f901aa700 2021-02-10T11:22:26.573 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:26.568 7f6f8f7fe700 1 --2- 172.21.2.102:0/608211424 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6f901aa700 0x7f6f901a3a20 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 rx=0x7f6f84009630 tx=0x7f6f8400b900).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:26.573 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:26.568 7f6f8cff9700 1 -- 172.21.2.102:0/608211424 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6f840093a0 con 0x7f6f901aa700 2021-02-10T11:22:26.574 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:26.568 7f6f8cff9700 1 -- 172.21.2.102:0/608211424 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f8400bcf0 con 0x7f6f901aa700 2021-02-10T11:22:26.574 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:26.568 7f6f8cff9700 1 -- 172.21.2.102:0/608211424 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6f84005560 con 0x7f6f901aa700 2021-02-10T11:22:26.574 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:26.568 7f6f966d9700 1 -- 172.21.2.102:0/608211424 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f901a4660 con 0x7f6f901aa700 2021-02-10T11:22:26.574 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:26.568 7f6f966d9700 1 -- 172.21.2.102:0/608211424 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f901a4660 con 0x7f6f901aa700 2021-02-10T11:22:26.574 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:26.569 7f6f8cff9700 1 -- 172.21.2.102:0/608211424 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f6f84008240 con 0x7f6f901aa700 2021-02-10T11:22:26.574 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:26.569 7f6f8cff9700 1 --2- 172.21.2.102:0/608211424 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6f700186c0 0x7f6f7001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:26.574 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:26.569 7f6f8cff9700 1 -- 172.21.2.102:0/608211424 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f6f84032500 con 0x7f6f901aa700 2021-02-10T11:22:26.574 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:26.569 7f6f8ffff700 1 --2- 172.21.2.102:0/608211424 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6f700186c0 0x7f6f7001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:26.574 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:26.569 7f6f8ffff700 1 --2- 172.21.2.102:0/608211424 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6f700186c0 0x7f6f7001abc0 crc :-1 s=READY pgs=513 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:26.596 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:26.599 7f6f966d9700 1 -- 172.21.2.102:0/608211424 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f90083da0 con 0x7f6f901aa700 2021-02-10T11:22:26.599 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:26.602 7f6f8cff9700 1 -- 172.21.2.102:0/608211424 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f6f8402d010 con 0x7f6f901aa700 2021-02-10T11:22:26.722 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:26.725 7f6f966d9700 1 -- 172.21.2.102:0/608211424 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6f9007ed80 con 0x7f6f901aa700 2021-02-10T11:22:26.722 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:26.726 7f6f8cff9700 1 -- 172.21.2.102:0/608211424 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v38) v1 ==== 74+0+14006 (secure 0 0 0) 0x7f6f8402d010 con 0x7f6f901aa700 2021-02-10T11:22:26.723 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:22:26.723 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":38,"fsid":"68db30ea-dbce-4aef-a6df-f01c6fea0513","created":"2021-02-10 11:03:06.331557","modified":"2021-02-10 11:22:26.122344","last_up_change":"2021-02-10 11:03:32.690562","last_in_change":"2021-02-10 11:03:32.690562","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:03:37.322519","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:22:23.578113","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":7,"pool_name":"cephfs_data","create_time":"2021-02-10 11:22:25.111303","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"be691b4d-a039-408a-af80-9d53f02b6f1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6818","nonce":36088},{"type":"v1","addr":"172.21.2.102:6820","nonce":36088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6822","nonce":36088},{"type":"v1","addr":"172.21.2.102:6824","nonce":36088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6830","nonce":36088},{"type":"v1","addr":"172.21.2.102:6832","nonce":36088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6826","nonce":36088},{"type":"v1","addr":"172.21.2.102:6828","nonce":36088}]},"public_addr":"172.21.2.102:6820/36088","cluster_addr":"172.21.2.102:6824/36088","heartbeat_back_addr":"172.21.2.102:6832/36088","heartbeat_front_addr":"172.21.2.102:6828/36088","state":["exists","up"]},{"osd":1,"uuid":"64e3ccc8-9c69-4d7c-865e-af8e83822635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6819","nonce":36087},{"type":"v1","addr":"172.21.2.102:6821","nonce":36087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6823","nonce":36087},{"type":"v1","addr":"172.21.2.102:6825","nonce":36087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6831","nonce":36087},{"type":"v1","addr":"172.21.2.102:6833","nonce":36087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6827","nonce":36087},{"type":"v1","addr":"172.21.2.102:6829","nonce":36087}]},"public_addr":"172.21.2.102:6821/36087","cluster_addr":"172.21.2.102:6825/36087","heartbeat_back_addr":"172.21.2.102:6833/36087","heartbeat_front_addr":"172.21.2.102:6829/36087","state":["exists","up"]},{"osd":2,"uuid":"153700af-747f-481f-bdf4-d6072d3a842f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6810","nonce":36089},{"type":"v1","addr":"172.21.2.102:6811","nonce":36089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6812","nonce":36089},{"type":"v1","addr":"172.21.2.102:6813","nonce":36089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6816","nonce":36089},{"type":"v1","addr":"172.21.2.102:6817","nonce":36089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6814","nonce":36089},{"type":"v1","addr":"172.21.2.102:6815","nonce":36089}]},"public_addr":"172.21.2.102:6811/36089","cluster_addr":"172.21.2.102:6813/36089","heartbeat_back_addr":"172.21.2.102:6817/36089","heartbeat_front_addr":"172.21.2.102:6815/36089","state":["exists","up"]},{"osd":3,"uuid":"cbc678f4-83a9-45e6-8830-2f7c232b1ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6802","nonce":36090},{"type":"v1","addr":"172.21.2.102:6803","nonce":36090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6804","nonce":36090},{"type":"v1","addr":"172.21.2.102:6805","nonce":36090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6808","nonce":36090},{"type":"v1","addr":"172.21.2.102:6809","nonce":36090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6806","nonce":36090},{"type":"v1","addr":"172.21.2.102:6807","nonce":36090}]},"public_addr":"172.21.2.102:6803/36090","cluster_addr":"172.21.2.102:6805/36090","heartbeat_back_addr":"172.21.2.102:6809/36090","heartbeat_front_addr":"172.21.2.102:6807/36090","state":["exists","up"]},{"osd":4,"uuid":"c01e4008-b258-4f10-afd2-9e7f37214351","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6808","nonce":36368},{"type":"v1","addr":"172.21.2.113:6811","nonce":36368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6814","nonce":36368},{"type":"v1","addr":"172.21.2.113:6818","nonce":36368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6826","nonce":36368},{"type":"v1","addr":"172.21.2.113:6828","nonce":36368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6820","nonce":36368},{"type":"v1","addr":"172.21.2.113:6824","nonce":36368}]},"public_addr":"172.21.2.113:6811/36368","cluster_addr":"172.21.2.113:6818/36368","heartbeat_back_addr":"172.21.2.113:6828/36368","heartbeat_front_addr":"172.21.2.113:6824/36368","state":["exists","up"]},{"osd":5,"uuid":"551f6007-daf8-4a98-8ee0-c2b16bcccb5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6802","nonce":36366},{"type":"v1","addr":"172.21.2.113:6805","nonce":36366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6807","nonce":36366},{"type":"v1","addr":"172.21.2.113:6810","nonce":36366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6819","nonce":36366},{"type":"v1","addr":"172.21.2.113:6822","nonce":36366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6812","nonce":36366},{"type":"v1","addr":"172.21.2.113:6815","nonce":36366}]},"public_addr":"172.21.2.113:6805/36366","cluster_addr":"172.21.2.113:6810/36366","heartbeat_back_addr":"172.21.2.113:6822/36366","heartbeat_front_addr":"172.21.2.113:6815/36366","state":["exists","up"]},{"osd":6,"uuid":"bf67c040-0e50-48be-b102-c3943bc9ebef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6800","nonce":36367},{"type":"v1","addr":"172.21.2.113:6801","nonce":36367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6803","nonce":36367},{"type":"v1","addr":"172.21.2.113:6804","nonce":36367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6813","nonce":36367},{"type":"v1","addr":"172.21.2.113:6816","nonce":36367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6806","nonce":36367},{"type":"v1","addr":"172.21.2.113:6809","nonce":36367}]},"public_addr":"172.21.2.113:6801/36367","cluster_addr":"172.21.2.113:6804/36367","heartbeat_back_addr":"172.21.2.113:6816/36367","heartbeat_front_addr":"172.21.2.113:6809/36367","state":["exists","up"]},{"osd":7,"uuid":"f5461f80-018d-428f-be89-2eb2bcbe7bb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6817","nonce":36365},{"type":"v1","addr":"172.21.2.113:6821","nonce":36365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6823","nonce":36365},{"type":"v1","addr":"172.21.2.113:6825","nonce":36365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6830","nonce":36365},{"type":"v1","addr":"172.21.2.113:6831","nonce":36365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6827","nonce":36365},{"type":"v1","addr":"172.21.2.113:6829","nonce":36365}]},"public_addr":"172.21.2.113:6821/36365","cluster_addr":"172.21.2.113:6825/36365","heartbeat_back_addr":"172.21.2.113:6831/36365","heartbeat_front_addr":"172.21.2.113:6829/36365","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:22:26.724 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:26.727 7f6f966d9700 1 -- 172.21.2.102:0/608211424 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6f700186c0 msgr2=0x7f6f7001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:26.724 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:26.727 7f6f966d9700 1 --2- 172.21.2.102:0/608211424 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6f700186c0 0x7f6f7001abc0 crc :-1 s=READY pgs=513 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:26.724 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:26.727 7f6f966d9700 1 -- 172.21.2.102:0/608211424 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6f901aa700 msgr2=0x7f6f901a3a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:26.724 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:26.727 7f6f966d9700 1 --2- 172.21.2.102:0/608211424 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6f901aa700 0x7f6f901a3a20 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 rx=0x7f6f84009630 tx=0x7f6f8400b900).stop 2021-02-10T11:22:26.724 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:26.727 7f6f966d9700 1 -- 172.21.2.102:0/608211424 shutdown_connections 2021-02-10T11:22:26.724 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:26.727 7f6f966d9700 1 --2- 172.21.2.102:0/608211424 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6f900a55e0 0x7f6f901a4040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:26.724 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:26.727 7f6f966d9700 1 --2- 172.21.2.102:0/608211424 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6f901aa700 0x7f6f901a3a20 unknown :-1 s=CLOSED pgs=1062 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:26.725 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:26.727 7f6f966d9700 1 --2- 172.21.2.102:0/608211424 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6f700186c0 0x7f6f7001abc0 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:26.725 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:26.727 7f6f966d9700 1 --2- 172.21.2.102:0/608211424 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6f901a2a90 0x7f6f901aa0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:26.725 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:26.727 7f6f966d9700 1 -- 172.21.2.102:0/608211424 >> 172.21.2.102:0/608211424 conn(0x7f6f9008fe10 msgr2=0x7f6f9019c020 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:26.725 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:26.728 7f6f966d9700 1 -- 172.21.2.102:0/608211424 shutdown_connections 2021-02-10T11:22:26.725 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:26.728 7f6f966d9700 1 -- 172.21.2.102:0/608211424 wait complete. 2021-02-10T11:22:26.733 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:22:27.157 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:27.144 7fead414b700 1 Processor -- start 2021-02-10T11:22:27.158 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:27.144 7fead414b700 1 -- start start 2021-02-10T11:22:27.158 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:27.145 7fead414b700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7feacc0e5070 0x7feacc1b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:27.158 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:27.145 7fead414b700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7feacc1b3e40 0x7feacc07d690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:27.158 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:27.145 7fead414b700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feacc07ed80 con 0x7feacc0e5470 2021-02-10T11:22:27.158 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:27.145 7fead414b700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feacc080310 con 0x7feacc0e5070 2021-02-10T11:22:27.158 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:27.145 7fead414b700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feacc080530 con 0x7feacc1b3e40 2021-02-10T11:22:27.158 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:27.145 7fead1f67700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7feacc0e5070 0x7feacc1b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:27.158 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:27.145 7fead1766700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7feacc1b3e40 0x7feacc07d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:27.158 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:27.145 7fead2768700 1 --1- >> v1:172.21.2.113:6789/0 conn(0x7feacc0e5470 0x7feacc07dcb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.113:6789/0 says I am v1:172.21.2.102:54974/0 (socket says 172.21.2.102:54974) 2021-02-10T11:22:27.158 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:27.145 7fead1f67700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7feacc0e5070 0x7feacc1b3820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:55426/0 (socket says 172.21.2.102:55426) 2021-02-10T11:22:27.159 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:27.145 7fead2768700 1 -- 172.21.2.102:0/1649563462 learned_addr learned my addr 172.21.2.102:0/1649563462 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:22:27.159 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:27.145 7fead1f67700 1 -- 172.21.2.102:0/1649563462 >> v2:172.21.2.113:3300/0 conn(0x7feacc1b3e40 msgr2=0x7feacc07d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:27.159 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:27.145 7fead1f67700 1 --2- 172.21.2.102:0/1649563462 >> v2:172.21.2.113:3300/0 conn(0x7feacc1b3e40 0x7feacc07d690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:27.159 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:27.145 7fead1f67700 1 -- 172.21.2.102:0/1649563462 >> v1:172.21.2.113:6789/0 conn(0x7feacc0e5470 legacy=0x7feacc07dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:27.159 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:27.145 7fead1f67700 1 -- 172.21.2.102:0/1649563462 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feacc080750 con 0x7feacc0e5070 2021-02-10T11:22:27.159 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:27.145 7fead1766700 1 --2- 172.21.2.102:0/1649563462 >> v2:172.21.2.113:3300/0 conn(0x7feacc1b3e40 0x7feacc07d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:22:27.159 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:27.146 7fead1f67700 1 --2- 172.21.2.102:0/1649563462 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7feacc0e5070 0x7feacc1b3820 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 rx=0x7feac8006650 tx=0x7feac8009a80).ready entity=mon.0 client_cookie=abc6afb0535aa38 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:27.159 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:27.146 7fead0f65700 1 -- 172.21.2.102:0/1649563462 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7feac80009b0 con 0x7feacc0e5070 2021-02-10T11:22:27.159 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:27.146 7fead0f65700 1 -- 172.21.2.102:0/1649563462 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feac800c0c0 con 0x7feacc0e5070 2021-02-10T11:22:27.159 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:27.146 7fead0f65700 1 -- 172.21.2.102:0/1649563462 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7feac800c520 con 0x7feacc0e5070 2021-02-10T11:22:27.159 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:27.146 7fead414b700 1 -- 172.21.2.102:0/1649563462 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7feacc0e5070 msgr2=0x7feacc1b3820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:27.159 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:27.146 7fead414b700 1 --2- 172.21.2.102:0/1649563462 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7feacc0e5070 0x7feacc1b3820 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 rx=0x7feac8006650 tx=0x7feac8009a80).stop 2021-02-10T11:22:27.160 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:27.146 7fead414b700 1 -- 172.21.2.102:0/1649563462 shutdown_connections 2021-02-10T11:22:27.160 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:27.146 7fead414b700 1 --2- 172.21.2.102:0/1649563462 >> v2:172.21.2.113:3300/0 conn(0x7feacc1b3e40 0x7feacc07d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:27.160 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:27.146 7fead414b700 1 --2- 172.21.2.102:0/1649563462 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7feacc0e5070 0x7feacc1b3820 unknown :-1 s=CLOSED pgs=1063 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:27.160 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:27.146 7fead414b700 1 -- 172.21.2.102:0/1649563462 >> 172.21.2.102:0/1649563462 conn(0x7feacc0ce690 msgr2=0x7feacc0e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:27.160 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:27.147 7fead414b700 1 -- 172.21.2.102:0/1649563462 shutdown_connections 2021-02-10T11:22:27.160 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:27.147 7fead414b700 1 -- 172.21.2.102:0/1649563462 wait complete. 2021-02-10T11:22:27.161 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:27.161 7fead414b700 1 Processor -- start 2021-02-10T11:22:27.161 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:27.161 7fead414b700 1 -- start start 2021-02-10T11:22:27.161 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:27.162 7fead414b700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7feacc08d400 0x7feacc096c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:27.162 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:27.162 7fead414b700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7feacc0b1450 0x7feacc0b1850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:27.162 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:27.162 7fead414b700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7feacc0b1e70 0x7feacc1b1f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:27.162 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:27.162 7fead414b700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feacc097310 con 0x7feacc0b1e70 2021-02-10T11:22:27.162 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:27.162 7fead414b700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feacc08d800 con 0x7feacc08d400 2021-02-10T11:22:27.162 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:27.162 7fead414b700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feacc08d9f0 con 0x7feacc0b1450 2021-02-10T11:22:27.162 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:27.162 7fead2768700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7feacc0b1e70 0x7feacc1b1f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:27.162 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:27.162 7fead1f67700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7feacc08d400 0x7feacc096c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:27.162 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:27.162 7fead2768700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7feacc0b1e70 0x7feacc1b1f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:55434/0 (socket says 172.21.2.102:55434) 2021-02-10T11:22:27.162 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:27.162 7fead2768700 1 -- 172.21.2.102:0/1876913403 learned_addr learned my addr 172.21.2.102:0/1876913403 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:22:27.162 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:27.162 7fead1766700 1 --2- 172.21.2.102:0/1876913403 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7feacc0b1450 0x7feacc0b1850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:27.162 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:27.162 7fead1f67700 1 -- 172.21.2.102:0/1876913403 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7feacc0b1450 msgr2=0x7feacc0b1850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:27.162 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:27.162 7fead1f67700 1 --2- 172.21.2.102:0/1876913403 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7feacc0b1450 0x7feacc0b1850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:27.162 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:27.162 7fead1f67700 1 -- 172.21.2.102:0/1876913403 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7feacc0b1e70 msgr2=0x7feacc1b1f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:27.163 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:27.162 7fead1f67700 1 --2- 172.21.2.102:0/1876913403 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7feacc0b1e70 0x7feacc1b1f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:27.163 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:27.162 7fead1f67700 1 -- 172.21.2.102:0/1876913403 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feacc1b25f0 con 0x7feacc08d400 2021-02-10T11:22:27.163 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:27.162 7fead1f67700 1 --2- 172.21.2.102:0/1876913403 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7feacc08d400 0x7feacc096c70 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 rx=0x7feac800b530 tx=0x7feac800b8d0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:27.163 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:27.163 7feabe7fc700 1 -- 172.21.2.102:0/1876913403 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7feac8000a10 con 0x7feacc08d400 2021-02-10T11:22:27.163 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:27.163 7feabe7fc700 1 -- 172.21.2.102:0/1876913403 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feac8009310 con 0x7feacc08d400 2021-02-10T11:22:27.163 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:27.163 7feabe7fc700 1 -- 172.21.2.102:0/1876913403 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7feac8002030 con 0x7feacc08d400 2021-02-10T11:22:27.163 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:27.163 7fead414b700 1 -- 172.21.2.102:0/1876913403 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feacc1b25f0 con 0x7feacc08d400 2021-02-10T11:22:27.163 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:27.163 7fead414b700 1 -- 172.21.2.102:0/1876913403 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feacc1b25f0 con 0x7feacc08d400 2021-02-10T11:22:27.163 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:27.163 7feabe7fc700 1 -- 172.21.2.102:0/1876913403 <== mon.1 v2:172.21.2.113:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7feac80052e0 con 0x7feacc08d400 2021-02-10T11:22:27.163 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:27.164 7feabe7fc700 1 --2- 172.21.2.102:0/1876913403 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7feab40186c0 0x7feab401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:27.163 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:27.164 7feabe7fc700 1 -- 172.21.2.102:0/1876913403 <== mon.1 v2:172.21.2.113:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 5573+0+0 (secure 0 0 0) 0x7feac8032490 con 0x7feacc08d400 2021-02-10T11:22:27.163 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:27.164 7fead1766700 1 --2- 172.21.2.102:0/1876913403 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7feab40186c0 0x7feab401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:27.163 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:27.164 7fead1766700 1 --2- 172.21.2.102:0/1876913403 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7feab40186c0 0x7feab401abc0 crc :-1 s=READY pgs=514 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:27.191 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:27.195 7fead414b700 1 -- 172.21.2.102:0/1876913403 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feacc07ed80 con 0x7feacc08d400 2021-02-10T11:22:27.195 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:27.200 7feabe7fc700 1 -- 172.21.2.102:0/1876913403 <== mon.1 v2:172.21.2.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7feac8000c90 con 0x7feacc08d400 2021-02-10T11:22:27.324 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:27.327 7fead414b700 1 -- 172.21.2.102:0/1876913403 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7feacc07ed80 con 0x7feacc08d400 2021-02-10T11:22:27.325 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:27.329 7feabe7fc700 1 -- 172.21.2.102:0/1876913403 <== mon.1 v2:172.21.2.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v39) v1 ==== 74+0+13993 (secure 0 0 0) 0x7feac8000f00 con 0x7feacc08d400 2021-02-10T11:22:27.325 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:22:27.325 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":39,"fsid":"68db30ea-dbce-4aef-a6df-f01c6fea0513","created":"2021-02-10 11:03:06.331557","modified":"2021-02-10 11:22:27.123992","last_up_change":"2021-02-10 11:03:32.690562","last_in_change":"2021-02-10 11:03:32.690562","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:03:37.322519","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:22:23.578113","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":7,"pool_name":"cephfs_data","create_time":"2021-02-10 11:22:25.111303","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"be691b4d-a039-408a-af80-9d53f02b6f1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6818","nonce":36088},{"type":"v1","addr":"172.21.2.102:6820","nonce":36088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6822","nonce":36088},{"type":"v1","addr":"172.21.2.102:6824","nonce":36088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6830","nonce":36088},{"type":"v1","addr":"172.21.2.102:6832","nonce":36088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6826","nonce":36088},{"type":"v1","addr":"172.21.2.102:6828","nonce":36088}]},"public_addr":"172.21.2.102:6820/36088","cluster_addr":"172.21.2.102:6824/36088","heartbeat_back_addr":"172.21.2.102:6832/36088","heartbeat_front_addr":"172.21.2.102:6828/36088","state":["exists","up"]},{"osd":1,"uuid":"64e3ccc8-9c69-4d7c-865e-af8e83822635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6819","nonce":36087},{"type":"v1","addr":"172.21.2.102:6821","nonce":36087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6823","nonce":36087},{"type":"v1","addr":"172.21.2.102:6825","nonce":36087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6831","nonce":36087},{"type":"v1","addr":"172.21.2.102:6833","nonce":36087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6827","nonce":36087},{"type":"v1","addr":"172.21.2.102:6829","nonce":36087}]},"public_addr":"172.21.2.102:6821/36087","cluster_addr":"172.21.2.102:6825/36087","heartbeat_back_addr":"172.21.2.102:6833/36087","heartbeat_front_addr":"172.21.2.102:6829/36087","state":["exists","up"]},{"osd":2,"uuid":"153700af-747f-481f-bdf4-d6072d3a842f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6810","nonce":36089},{"type":"v1","addr":"172.21.2.102:6811","nonce":36089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6812","nonce":36089},{"type":"v1","addr":"172.21.2.102:6813","nonce":36089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6816","nonce":36089},{"type":"v1","addr":"172.21.2.102:6817","nonce":36089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6814","nonce":36089},{"type":"v1","addr":"172.21.2.102:6815","nonce":36089}]},"public_addr":"172.21.2.102:6811/36089","cluster_addr":"172.21.2.102:6813/36089","heartbeat_back_addr":"172.21.2.102:6817/36089","heartbeat_front_addr":"172.21.2.102:6815/36089","state":["exists","up"]},{"osd":3,"uuid":"cbc678f4-83a9-45e6-8830-2f7c232b1ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6802","nonce":36090},{"type":"v1","addr":"172.21.2.102:6803","nonce":36090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6804","nonce":36090},{"type":"v1","addr":"172.21.2.102:6805","nonce":36090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6808","nonce":36090},{"type":"v1","addr":"172.21.2.102:6809","nonce":36090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6806","nonce":36090},{"type":"v1","addr":"172.21.2.102:6807","nonce":36090}]},"public_addr":"172.21.2.102:6803/36090","cluster_addr":"172.21.2.102:6805/36090","heartbeat_back_addr":"172.21.2.102:6809/36090","heartbeat_front_addr":"172.21.2.102:6807/36090","state":["exists","up"]},{"osd":4,"uuid":"c01e4008-b258-4f10-afd2-9e7f37214351","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6808","nonce":36368},{"type":"v1","addr":"172.21.2.113:6811","nonce":36368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6814","nonce":36368},{"type":"v1","addr":"172.21.2.113:6818","nonce":36368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6826","nonce":36368},{"type":"v1","addr":"172.21.2.113:6828","nonce":36368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6820","nonce":36368},{"type":"v1","addr":"172.21.2.113:6824","nonce":36368}]},"public_addr":"172.21.2.113:6811/36368","cluster_addr":"172.21.2.113:6818/36368","heartbeat_back_addr":"172.21.2.113:6828/36368","heartbeat_front_addr":"172.21.2.113:6824/36368","state":["exists","up"]},{"osd":5,"uuid":"551f6007-daf8-4a98-8ee0-c2b16bcccb5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6802","nonce":36366},{"type":"v1","addr":"172.21.2.113:6805","nonce":36366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6807","nonce":36366},{"type":"v1","addr":"172.21.2.113:6810","nonce":36366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6819","nonce":36366},{"type":"v1","addr":"172.21.2.113:6822","nonce":36366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6812","nonce":36366},{"type":"v1","addr":"172.21.2.113:6815","nonce":36366}]},"public_addr":"172.21.2.113:6805/36366","cluster_addr":"172.21.2.113:6810/36366","heartbeat_back_addr":"172.21.2.113:6822/36366","heartbeat_front_addr":"172.21.2.113:6815/36366","state":["exists","up"]},{"osd":6,"uuid":"bf67c040-0e50-48be-b102-c3943bc9ebef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6800","nonce":36367},{"type":"v1","addr":"172.21.2.113:6801","nonce":36367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6803","nonce":36367},{"type":"v1","addr":"172.21.2.113:6804","nonce":36367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6813","nonce":36367},{"type":"v1","addr":"172.21.2.113:6816","nonce":36367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6806","nonce":36367},{"type":"v1","addr":"172.21.2.113:6809","nonce":36367}]},"public_addr":"172.21.2.113:6801/36367","cluster_addr":"172.21.2.113:6804/36367","heartbeat_back_addr":"172.21.2.113:6816/36367","heartbeat_front_addr":"172.21.2.113:6809/36367","state":["exists","up"]},{"osd":7,"uuid":"f5461f80-018d-428f-be89-2eb2bcbe7bb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6817","nonce":36365},{"type":"v1","addr":"172.21.2.113:6821","nonce":36365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6823","nonce":36365},{"type":"v1","addr":"172.21.2.113:6825","nonce":36365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6830","nonce":36365},{"type":"v1","addr":"172.21.2.113:6831","nonce":36365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6827","nonce":36365},{"type":"v1","addr":"172.21.2.113:6829","nonce":36365}]},"public_addr":"172.21.2.113:6821/36365","cluster_addr":"172.21.2.113:6825/36365","heartbeat_back_addr":"172.21.2.113:6831/36365","heartbeat_front_addr":"172.21.2.113:6829/36365","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:22:27.326 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:27.330 7fead414b700 1 -- 172.21.2.102:0/1876913403 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7feab40186c0 msgr2=0x7feab401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:27.326 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:27.330 7fead414b700 1 --2- 172.21.2.102:0/1876913403 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7feab40186c0 0x7feab401abc0 crc :-1 s=READY pgs=514 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:27.327 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:27.330 7fead414b700 1 -- 172.21.2.102:0/1876913403 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7feacc08d400 msgr2=0x7feacc096c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:27.327 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:27.330 7fead414b700 1 --2- 172.21.2.102:0/1876913403 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7feacc08d400 0x7feacc096c70 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 rx=0x7feac800b530 tx=0x7feac800b8d0).stop 2021-02-10T11:22:27.327 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:27.330 7fead414b700 1 -- 172.21.2.102:0/1876913403 shutdown_connections 2021-02-10T11:22:27.327 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:27.330 7fead414b700 1 --2- 172.21.2.102:0/1876913403 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7feacc0b1e70 0x7feacc1b1f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:27.327 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:27.330 7fead414b700 1 --2- 172.21.2.102:0/1876913403 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7feacc0b1450 0x7feacc0b1850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:27.327 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:27.330 7fead414b700 1 --2- 172.21.2.102:0/1876913403 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7feab40186c0 0x7feab401abc0 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:27.328 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:27.330 7fead414b700 1 --2- 172.21.2.102:0/1876913403 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7feacc08d400 0x7feacc096c70 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:27.328 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:27.330 7fead414b700 1 -- 172.21.2.102:0/1876913403 >> 172.21.2.102:0/1876913403 conn(0x7feacc1ac620 msgr2=0x7feacc0896e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:27.328 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:27.330 7fead414b700 1 -- 172.21.2.102:0/1876913403 shutdown_connections 2021-02-10T11:22:27.328 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:27.330 7fead414b700 1 -- 172.21.2.102:0/1876913403 wait complete. 2021-02-10T11:22:27.335 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2021-02-10T11:22:27.766 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:27.753 7f3c10994700 1 Processor -- start 2021-02-10T11:22:27.766 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:27.753 7f3c10994700 1 -- start start 2021-02-10T11:22:27.766 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:27.754 7f3c10994700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f3c0c1b0870 0x7f3c0c0f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:27.766 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:27.754 7f3c10994700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f3c0c0efe00 0x7f3c0c0f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:27.767 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:27.754 7f3c10994700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c0c1b0c70 con 0x7f3c0c1b0470 2021-02-10T11:22:27.767 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:27.754 7f3c10994700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c0c0e5ea0 con 0x7f3c0c1b0870 2021-02-10T11:22:27.767 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:27.754 7f3c10994700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c0c0f0b50 con 0x7f3c0c0efe00 2021-02-10T11:22:27.767 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:27.754 7f3c01e1b700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f3c0c1b0870 0x7f3c0c0f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:27.767 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:27.754 7f3c0a61c700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f3c0c1b0470 0x7f3c0c0ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:47662/0 (socket says 172.21.2.102:47662) 2021-02-10T11:22:27.767 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:27.754 7f3c0a61c700 1 -- 172.21.2.102:0/1511685766 learned_addr learned my addr 172.21.2.102:0/1511685766 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:22:27.767 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:27.754 7f3c0ae1d700 1 --2- 172.21.2.102:0/1511685766 >> v2:172.21.2.113:3300/0 conn(0x7f3c0c0efe00 0x7f3c0c0f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:27.767 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:27.754 7f3c01e1b700 1 -- 172.21.2.102:0/1511685766 >> v2:172.21.2.113:3300/0 conn(0x7f3c0c0efe00 msgr2=0x7f3c0c0f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:27.767 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:27.754 7f3c01e1b700 1 --2- 172.21.2.102:0/1511685766 >> v2:172.21.2.113:3300/0 conn(0x7f3c0c0efe00 0x7f3c0c0f0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:27.767 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:27.754 7f3c01e1b700 1 -- 172.21.2.102:0/1511685766 >> v1:172.21.2.102:6789/0 conn(0x7f3c0c1b0470 legacy=0x7f3c0c0ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:27.767 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:27.754 7f3c09e1b700 1 -- 172.21.2.102:0/1511685766 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2483898553 0 0) 0x7f3bfc001060 con 0x7f3c0c1b0470 2021-02-10T11:22:27.768 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:27.755 7f3c01e1b700 1 -- 172.21.2.102:0/1511685766 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c0c0f0820 con 0x7f3c0c1b0870 2021-02-10T11:22:27.768 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:27.755 7f3c0ae1d700 1 --2- 172.21.2.102:0/1511685766 >> v2:172.21.2.113:3300/0 conn(0x7f3c0c0efe00 0x7f3c0c0f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:22:27.768 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:27.755 7f3c01e1b700 1 --2- 172.21.2.102:0/1511685766 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3c0c1b0870 0x7f3c0c0f20e0 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 rx=0x7f3bf4006650 tx=0x7f3bf4009a80).ready entity=mon.0 client_cookie=73d8ea8a7cba6a9c server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:27.768 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:27.755 7f3c09e1b700 1 -- 172.21.2.102:0/1511685766 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3bf40009b0 con 0x7f3c0c1b0870 2021-02-10T11:22:27.768 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:27.755 7f3c09e1b700 1 -- 172.21.2.102:0/1511685766 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bf400c0c0 con 0x7f3c0c1b0870 2021-02-10T11:22:27.768 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:27.755 7f3c09e1b700 1 -- 172.21.2.102:0/1511685766 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3bf400c520 con 0x7f3c0c1b0870 2021-02-10T11:22:27.768 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:27.755 7f3c10994700 1 -- 172.21.2.102:0/1511685766 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3c0c1b0870 msgr2=0x7f3c0c0f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:27.768 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:27.755 7f3c10994700 1 --2- 172.21.2.102:0/1511685766 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3c0c1b0870 0x7f3c0c0f20e0 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 rx=0x7f3bf4006650 tx=0x7f3bf4009a80).stop 2021-02-10T11:22:27.768 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:27.756 7f3c10994700 1 -- 172.21.2.102:0/1511685766 shutdown_connections 2021-02-10T11:22:27.768 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:27.756 7f3c10994700 1 --2- 172.21.2.102:0/1511685766 >> v2:172.21.2.113:3300/0 conn(0x7f3c0c0efe00 0x7f3c0c0f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:27.768 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:27.756 7f3c10994700 1 --2- 172.21.2.102:0/1511685766 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3c0c1b0870 0x7f3c0c0f20e0 unknown :-1 s=CLOSED pgs=1065 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:27.768 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:27.756 7f3c10994700 1 -- 172.21.2.102:0/1511685766 >> 172.21.2.102:0/1511685766 conn(0x7f3c0c0ce3e0 msgr2=0x7f3c0c0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:27.768 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:27.756 7f3c10994700 1 -- 172.21.2.102:0/1511685766 shutdown_connections 2021-02-10T11:22:27.769 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:27.756 7f3c10994700 1 -- 172.21.2.102:0/1511685766 wait complete. 2021-02-10T11:22:27.770 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:27.771 7f3c10994700 1 Processor -- start 2021-02-10T11:22:27.770 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:27.771 7f3c10994700 1 -- start start 2021-02-10T11:22:27.770 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:27.771 7f3c10994700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3c0c0a0420 0x7f3c0c0a0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:27.770 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:27.771 7f3c10994700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3c0c189d00 0x7f3c0c18a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:27.770 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:27.771 7f3c10994700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3c0c18a720 0x7f3c0c0b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:27.770 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:27.771 7f3c10994700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c0c189800 con 0x7f3c0c18a720 2021-02-10T11:22:27.771 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:27.771 7f3c10994700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c0c18ab20 con 0x7f3c0c189d00 2021-02-10T11:22:27.771 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:27.771 7f3c10994700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c0c0a1140 con 0x7f3c0c0a0420 2021-02-10T11:22:27.771 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:27.771 7f3c0ae1d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3c0c18a720 0x7f3c0c0b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:27.771 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:27.771 7f3c0ae1d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3c0c18a720 0x7f3c0c0b7be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:55448/0 (socket says 172.21.2.102:55448) 2021-02-10T11:22:27.771 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:27.772 7f3c0ae1d700 1 -- 172.21.2.102:0/2232839378 learned_addr learned my addr 172.21.2.102:0/2232839378 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:22:27.771 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:27.772 7f3c0a61c700 1 --2- 172.21.2.102:0/2232839378 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3c0c0a0420 0x7f3c0c0a0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:27.771 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:27.772 7f3c01e1b700 1 --2- 172.21.2.102:0/2232839378 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3c0c189d00 0x7f3c0c18a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:27.771 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:27.772 7f3c0ae1d700 1 -- 172.21.2.102:0/2232839378 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3c0c0a0420 msgr2=0x7f3c0c0a0820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:27.771 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:27.772 7f3c0ae1d700 1 --2- 172.21.2.102:0/2232839378 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3c0c0a0420 0x7f3c0c0a0820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:27.771 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:27.772 7f3c0ae1d700 1 -- 172.21.2.102:0/2232839378 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3c0c189d00 msgr2=0x7f3c0c18a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:27.771 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:27.772 7f3c0ae1d700 1 --2- 172.21.2.102:0/2232839378 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3c0c189d00 0x7f3c0c18a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:27.771 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:27.772 7f3c0ae1d700 1 -- 172.21.2.102:0/2232839378 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c0c0b8200 con 0x7f3c0c18a720 2021-02-10T11:22:27.771 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:27.772 7f3c0ae1d700 1 --2- 172.21.2.102:0/2232839378 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3c0c18a720 0x7f3c0c0b7be0 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 rx=0x7f3c0400b5e0 tx=0x7f3c0400b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:27.772 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:27.772 7f3c037fe700 1 -- 172.21.2.102:0/2232839378 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3c0400bd70 con 0x7f3c0c18a720 2021-02-10T11:22:27.772 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:27.772 7f3c037fe700 1 -- 172.21.2.102:0/2232839378 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c04006410 con 0x7f3c0c18a720 2021-02-10T11:22:27.772 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:27.772 7f3c10994700 1 -- 172.21.2.102:0/2232839378 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c0c0b8200 con 0x7f3c0c18a720 2021-02-10T11:22:27.772 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:27.772 7f3c037fe700 1 -- 172.21.2.102:0/2232839378 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3c04002560 con 0x7f3c0c18a720 2021-02-10T11:22:27.772 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:27.772 7f3c10994700 1 -- 172.21.2.102:0/2232839378 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c0c0b8200 con 0x7f3c0c18a720 2021-02-10T11:22:27.772 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:27.773 7f3c037fe700 1 -- 172.21.2.102:0/2232839378 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f3c04005330 con 0x7f3c0c18a720 2021-02-10T11:22:27.772 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:27.773 7f3c037fe700 1 --2- 172.21.2.102:0/2232839378 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3bf00186c0 0x7f3bf001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:27.772 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:27.773 7f3c037fe700 1 -- 172.21.2.102:0/2232839378 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f3c040325f0 con 0x7f3c0c18a720 2021-02-10T11:22:27.772 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:27.773 7f3c0a61c700 1 --2- 172.21.2.102:0/2232839378 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3bf00186c0 0x7f3bf001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:27.772 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:27.774 7f3c0a61c700 1 --2- 172.21.2.102:0/2232839378 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3bf00186c0 0x7f3bf001abc0 crc :-1 s=READY pgs=515 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:27.800 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:27.804 7f3c10994700 1 -- 172.21.2.102:0/2232839378 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c0c18ab20 con 0x7f3c0c18a720 2021-02-10T11:22:27.802 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:27.806 7f3c037fe700 1 -- 172.21.2.102:0/2232839378 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f3c0402df50 con 0x7f3c0c18a720 2021-02-10T11:22:27.939 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:27.943 7f3c10994700 1 -- 172.21.2.102:0/2232839378 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"} v 0) v1 -- 0x7f3c0c18ab20 con 0x7f3c0c18a720 2021-02-10T11:22:28.125 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:28.129 7f3c037fe700 1 -- 172.21.2.102:0/2232839378 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"}]=0 v21) v1 ==== 118+0+0 (secure 0 0 0) 0x7f3c0402df50 con 0x7f3c0c18a720 2021-02-10T11:22:28.156 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:28.160 7f3c10994700 1 -- 172.21.2.102:0/2232839378 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3bf00186c0 msgr2=0x7f3bf001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:28.156 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:28.160 7f3c10994700 1 --2- 172.21.2.102:0/2232839378 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3bf00186c0 0x7f3bf001abc0 crc :-1 s=READY pgs=515 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:28.156 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:28.160 7f3c10994700 1 -- 172.21.2.102:0/2232839378 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3c0c18a720 msgr2=0x7f3c0c0b7be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:28.156 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:28.160 7f3c10994700 1 --2- 172.21.2.102:0/2232839378 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3c0c18a720 0x7f3c0c0b7be0 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 rx=0x7f3c0400b5e0 tx=0x7f3c0400b980).stop 2021-02-10T11:22:28.157 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:28.160 7f3c10994700 1 -- 172.21.2.102:0/2232839378 shutdown_connections 2021-02-10T11:22:28.157 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:28.160 7f3c10994700 1 --2- 172.21.2.102:0/2232839378 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3c0c18a720 0x7f3c0c0b7be0 unknown :-1 s=CLOSED pgs=1066 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:28.157 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:28.160 7f3c10994700 1 --2- 172.21.2.102:0/2232839378 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3bf00186c0 0x7f3bf001abc0 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:28.157 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:28.160 7f3c10994700 1 --2- 172.21.2.102:0/2232839378 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3c0c189d00 0x7f3c0c18a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:28.157 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:28.160 7f3c10994700 1 --2- 172.21.2.102:0/2232839378 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3c0c0a0420 0x7f3c0c0a0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:28.157 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:28.160 7f3c10994700 1 -- 172.21.2.102:0/2232839378 >> 172.21.2.102:0/2232839378 conn(0x7f3c0c094e10 msgr2=0x7f3c0c0a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:28.157 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:28.160 7f3c10994700 1 -- 172.21.2.102:0/2232839378 shutdown_connections 2021-02-10T11:22:28.157 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:28.160 7f3c10994700 1 -- 172.21.2.102:0/2232839378 wait complete. 2021-02-10T11:22:28.164 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:22:28.592 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:28.579 7f436364b700 1 Processor -- start 2021-02-10T11:22:28.592 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:28.579 7f436364b700 1 -- start start 2021-02-10T11:22:28.592 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:28.579 7f436364b700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f435c0e5070 0x7f435c1b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:28.592 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:28.579 7f436364b700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f435c1b3e40 0x7f435c07d690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:28.592 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:28.579 7f436364b700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f435c0e5470 0x7f435c07dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:28.592 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:28.579 7f436364b700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f435c07ed80 con 0x7f435c1b3e40 2021-02-10T11:22:28.592 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:28.579 7f436364b700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f435c07e2d0 con 0x7f435c0e5070 2021-02-10T11:22:28.592 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:28.579 7f436364b700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f435c07e4f0 con 0x7f435c0e5470 2021-02-10T11:22:28.592 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:28.579 7f4360c66700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f435c1b3e40 0x7f435c07d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:28.592 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:28.579 7f4360c66700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f435c1b3e40 0x7f435c07d690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:55452/0 (socket says 172.21.2.102:55452) 2021-02-10T11:22:28.593 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:28.579 7f4360c66700 1 -- 172.21.2.102:0/520773473 learned_addr learned my addr 172.21.2.102:0/520773473 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:22:28.593 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:28.580 7f4360c66700 1 -- 172.21.2.102:0/520773473 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f435c0e5470 msgr2=0x7f435c07dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:28.593 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:28.580 7f4360c66700 1 --2- 172.21.2.102:0/520773473 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f435c0e5470 0x7f435c07dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:28.593 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:28.580 7f4360c66700 1 -- 172.21.2.102:0/520773473 >> v2:172.21.2.113:3300/0 conn(0x7f435c0e5070 msgr2=0x7f435c1b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:28.593 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:28.580 7f4360c66700 1 --2- 172.21.2.102:0/520773473 >> v2:172.21.2.113:3300/0 conn(0x7f435c0e5070 0x7f435c1b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:28.593 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:28.580 7f4360c66700 1 -- 172.21.2.102:0/520773473 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f435c07e710 con 0x7f435c1b3e40 2021-02-10T11:22:28.593 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:28.580 7f4360c66700 1 --2- 172.21.2.102:0/520773473 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f435c1b3e40 0x7f435c07d690 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 rx=0x7f434c006700 tx=0x7f434c009b80).ready entity=mon.0 client_cookie=67576c159b95f0f6 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:28.593 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:28.580 7f4353fff700 1 -- 172.21.2.102:0/520773473 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f434c00c0c0 con 0x7f435c1b3e40 2021-02-10T11:22:28.593 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:28.580 7f4353fff700 1 -- 172.21.2.102:0/520773473 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f434c00c340 con 0x7f435c1b3e40 2021-02-10T11:22:28.593 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:28.580 7f4353fff700 1 -- 172.21.2.102:0/520773473 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f434c00c860 con 0x7f435c1b3e40 2021-02-10T11:22:28.593 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:28.580 7f436364b700 1 -- 172.21.2.102:0/520773473 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f435c1b3e40 msgr2=0x7f435c07d690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:28.593 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:28.580 7f436364b700 1 --2- 172.21.2.102:0/520773473 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f435c1b3e40 0x7f435c07d690 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 rx=0x7f434c006700 tx=0x7f434c009b80).stop 2021-02-10T11:22:28.593 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:28.581 7f436364b700 1 -- 172.21.2.102:0/520773473 shutdown_connections 2021-02-10T11:22:28.594 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:28.581 7f436364b700 1 --2- 172.21.2.102:0/520773473 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f435c0e5470 0x7f435c07dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:28.594 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:28.581 7f436364b700 1 --2- 172.21.2.102:0/520773473 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f435c1b3e40 0x7f435c07d690 unknown :-1 s=CLOSED pgs=1067 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:28.594 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:28.581 7f436364b700 1 --2- 172.21.2.102:0/520773473 >> v2:172.21.2.113:3300/0 conn(0x7f435c0e5070 0x7f435c1b3820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:28.594 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:28.581 7f436364b700 1 -- 172.21.2.102:0/520773473 >> 172.21.2.102:0/520773473 conn(0x7f435c0ce690 msgr2=0x7f435c0e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:28.594 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:28.581 7f436364b700 1 -- 172.21.2.102:0/520773473 shutdown_connections 2021-02-10T11:22:28.594 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:28.581 7f436364b700 1 -- 172.21.2.102:0/520773473 wait complete. 2021-02-10T11:22:28.594 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:28.596 7f436364b700 1 Processor -- start 2021-02-10T11:22:28.594 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:28.596 7f436364b700 1 -- start start 2021-02-10T11:22:28.595 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:28.596 7f436364b700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f435c192960 0x7f435c192d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:28.596 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:28.596 7f436364b700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f435c0ad110 0x7f435c0ad510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:28.596 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:28.596 7f436364b700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f435c0adb30 0x7f435c08e460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:28.596 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:28.596 7f436364b700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f435c1a03e0 con 0x7f435c192960 2021-02-10T11:22:28.596 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:28.596 7f436364b700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f435c193380 con 0x7f435c0adb30 2021-02-10T11:22:28.596 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:28.596 7f436364b700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f435c08ea80 con 0x7f435c0ad110 2021-02-10T11:22:28.596 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:28.597 7f4361467700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f435c192960 0x7f435c192d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:28.596 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:28.597 7f4361467700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f435c192960 0x7f435c192d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:55456/0 (socket says 172.21.2.102:55456) 2021-02-10T11:22:28.596 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:28.597 7f4361467700 1 -- 172.21.2.102:0/3366296973 learned_addr learned my addr 172.21.2.102:0/3366296973 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:22:28.596 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:28.597 7f4360c66700 1 --2- 172.21.2.102:0/3366296973 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f435c0ad110 0x7f435c0ad510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:28.596 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:28.597 7f4361c68700 1 --2- 172.21.2.102:0/3366296973 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f435c0adb30 0x7f435c08e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:28.596 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:28.597 7f4361467700 1 -- 172.21.2.102:0/3366296973 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f435c0ad110 msgr2=0x7f435c0ad510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:28.596 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:28.597 7f4361467700 1 --2- 172.21.2.102:0/3366296973 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f435c0ad110 0x7f435c0ad510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:28.596 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:28.597 7f4361467700 1 -- 172.21.2.102:0/3366296973 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f435c0adb30 msgr2=0x7f435c08e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:28.597 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:28.597 7f4361467700 1 --2- 172.21.2.102:0/3366296973 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f435c0adb30 0x7f435c08e460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:28.597 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:28.597 7f4361467700 1 -- 172.21.2.102:0/3366296973 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f435c08eca0 con 0x7f435c192960 2021-02-10T11:22:28.597 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:28.597 7f4361467700 1 --2- 172.21.2.102:0/3366296973 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f435c192960 0x7f435c192d60 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 rx=0x7f4358006f00 tx=0x7f435800b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:28.597 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:28.598 7f4351ffb700 1 -- 172.21.2.102:0/3366296973 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f435800bd30 con 0x7f435c192960 2021-02-10T11:22:28.597 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:28.598 7f4351ffb700 1 -- 172.21.2.102:0/3366296973 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4358006c20 con 0x7f435c192960 2021-02-10T11:22:28.597 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:28.598 7f4351ffb700 1 -- 172.21.2.102:0/3366296973 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4358002020 con 0x7f435c192960 2021-02-10T11:22:28.597 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:28.598 7f436364b700 1 -- 172.21.2.102:0/3366296973 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f435c08eca0 con 0x7f435c192960 2021-02-10T11:22:28.597 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:28.598 7f436364b700 1 -- 172.21.2.102:0/3366296973 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f435c08f230 con 0x7f435c192960 2021-02-10T11:22:28.597 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:28.598 7f4351ffb700 1 -- 172.21.2.102:0/3366296973 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f435800bd30 con 0x7f435c192960 2021-02-10T11:22:28.597 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:28.598 7f4351ffb700 1 --2- 172.21.2.102:0/3366296973 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f433c0186c0 0x7f433c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:28.597 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:28.598 7f4351ffb700 1 -- 172.21.2.102:0/3366296973 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f4358032570 con 0x7f435c192960 2021-02-10T11:22:28.597 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:28.599 7f4360c66700 1 --2- 172.21.2.102:0/3366296973 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f433c0186c0 0x7f433c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:28.597 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:28.599 7f4360c66700 1 --2- 172.21.2.102:0/3366296973 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f433c0186c0 0x7f433c01abc0 crc :-1 s=READY pgs=516 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:28.625 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:28.629 7f436364b700 1 -- 172.21.2.102:0/3366296973 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f435c107120 con 0x7f435c192960 2021-02-10T11:22:28.627 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:28.632 7f4351ffb700 1 -- 172.21.2.102:0/3366296973 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f43580009b0 con 0x7f435c192960 2021-02-10T11:22:28.756 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:28.760 7f436364b700 1 -- 172.21.2.102:0/3366296973 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f435c07ed80 con 0x7f435c192960 2021-02-10T11:22:28.756 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:28.760 7f4351ffb700 1 -- 172.21.2.102:0/3366296973 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) v1 ==== 94+0+2099 (secure 0 0 0) 0x7f435800bd30 con 0x7f435c192960 2021-02-10T11:22:28.757 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:22:28.757 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":21,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7194,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.102:6835/2295196020","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6834","nonce":2295196020},{"type":"v1","addr":"172.21.2.102:6835","nonce":2295196020}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":18}],"filesystems":[{"mdsmap":{"epoch":21,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:22:26.122430","modified":"2021-02-10 11:22:28.126179","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7191},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7191":{"gid":7191,"name":"b","rank":0,"incarnation":19,"state":"up:active","state_seq":3,"addr":"172.21.2.113:6833/3694995761","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6832","nonce":3694995761},{"type":"v1","addr":"172.21.2.113:6833","nonce":3694995761}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":3}]} 2021-02-10T11:22:28.757 INFO:teuthology.orchestra.run.gibba002.stderr:dumped fsmap epoch 21 2021-02-10T11:22:28.758 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:28.762 7f436364b700 1 -- 172.21.2.102:0/3366296973 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f433c0186c0 msgr2=0x7f433c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:28.759 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:28.762 7f436364b700 1 --2- 172.21.2.102:0/3366296973 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f433c0186c0 0x7f433c01abc0 crc :-1 s=READY pgs=516 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:28.759 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:28.762 7f436364b700 1 -- 172.21.2.102:0/3366296973 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f435c192960 msgr2=0x7f435c192d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:28.759 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:28.762 7f436364b700 1 --2- 172.21.2.102:0/3366296973 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f435c192960 0x7f435c192d60 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 rx=0x7f4358006f00 tx=0x7f435800b940).stop 2021-02-10T11:22:28.759 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:28.763 7f436364b700 1 -- 172.21.2.102:0/3366296973 shutdown_connections 2021-02-10T11:22:28.759 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:28.763 7f436364b700 1 --2- 172.21.2.102:0/3366296973 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f433c0186c0 0x7f433c01abc0 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:28.759 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:28.763 7f436364b700 1 --2- 172.21.2.102:0/3366296973 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f435c0adb30 0x7f435c08e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:28.759 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:28.763 7f436364b700 1 --2- 172.21.2.102:0/3366296973 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f435c0ad110 0x7f435c0ad510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:28.759 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:28.763 7f436364b700 1 --2- 172.21.2.102:0/3366296973 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f435c192960 0x7f435c192d60 unknown :-1 s=CLOSED pgs=1068 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:28.759 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:28.763 7f436364b700 1 -- 172.21.2.102:0/3366296973 >> 172.21.2.102:0/3366296973 conn(0x7f435c0891f0 msgr2=0x7f435c1a0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:28.759 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:28.763 7f436364b700 1 -- 172.21.2.102:0/3366296973 shutdown_connections 2021-02-10T11:22:28.759 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:28.763 7f436364b700 1 -- 172.21.2.102:0/3366296973 wait complete. 2021-02-10T11:22:28.767 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:22:29.196 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.182 7f993582e700 1 Processor -- start 2021-02-10T11:22:29.196 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.182 7f993582e700 1 -- start start 2021-02-10T11:22:29.196 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.183 7f993582e700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f993007e6a0 0x7f99300d1790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:29.196 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.183 7f993582e700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f993007ed80 con 0x7f99300e5470 2021-02-10T11:22:29.196 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.183 7f993582e700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f993007eb10 con 0x7f99300e5070 2021-02-10T11:22:29.196 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.183 7f993582e700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99301b4080 con 0x7f993007e6a0 2021-02-10T11:22:29.196 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.183 7f992ed73700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f99300e5470 0x7f993007f300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:47688/0 (socket says 172.21.2.102:47688) 2021-02-10T11:22:29.196 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.183 7f992ed73700 1 -- 172.21.2.102:0/1857689185 learned_addr learned my addr 172.21.2.102:0/1857689185 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:22:29.196 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.184 7f992e572700 1 -- 172.21.2.102:0/1857689185 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 528688093 0 0) 0x7f9918000ce0 con 0x7f99300e5470 2021-02-10T11:22:29.196 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.184 7f992e572700 1 -- 172.21.2.102:0/1857689185 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f991c002fe0 con 0x7f99300e5470 2021-02-10T11:22:29.196 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.184 7f992fd75700 1 --2- 172.21.2.102:0/1857689185 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f993007e6a0 0x7f99300d1790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:29.197 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.184 7f992e572700 1 -- 172.21.2.102:0/1857689185 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 268279685 0 0) 0x7f99180010e0 con 0x7f99300e5470 2021-02-10T11:22:29.197 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.184 7f992e572700 1 -- 172.21.2.102:0/1857689185 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f993007e6a0 msgr2=0x7f99300d1790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:29.197 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.184 7f992e572700 1 --2- 172.21.2.102:0/1857689185 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f993007e6a0 0x7f99300d1790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:29.197 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.184 7f992e572700 1 -- 172.21.2.102:0/1857689185 >> v1:172.21.2.113:6789/0 conn(0x7f99300e5070 legacy=0x7f99301b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:29.197 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.184 7f992e572700 1 -- 172.21.2.102:0/1857689185 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99300d1db0 con 0x7f99300e5470 2021-02-10T11:22:29.197 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.184 7f992e572700 1 -- 172.21.2.102:0/1857689185 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7f9918001600 con 0x7f99300e5470 2021-02-10T11:22:29.197 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.184 7f992e572700 1 -- 172.21.2.102:0/1857689185 >> v1:172.21.2.102:6789/0 conn(0x7f99300e5470 legacy=0x7f993007f300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:29.197 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.184 7f992e572700 1 --2- 172.21.2.102:0/1857689185 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f991c003aa0 0x7f991c005ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:29.197 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.184 7f992e572700 1 --2- 172.21.2.102:0/1857689185 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f991c006770 0x7f991c008c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:29.197 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.184 7f992e572700 1 --2- 172.21.2.102:0/1857689185 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f991c009590 0x7f991c00ba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:29.197 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.184 7f992e572700 1 -- 172.21.2.102:0/1857689185 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f991c00c2d0 con 0x7f991c006770 2021-02-10T11:22:29.197 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.184 7f992e572700 1 -- 172.21.2.102:0/1857689185 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f991c00c4f0 con 0x7f991c003aa0 2021-02-10T11:22:29.197 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.184 7f992e572700 1 -- 172.21.2.102:0/1857689185 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f991c00c710 con 0x7f991c009590 2021-02-10T11:22:29.198 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.184 7f992ed73700 1 --2- 172.21.2.102:0/1857689185 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f991c006770 0x7f991c008c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:29.198 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.184 7f992fd75700 1 --2- 172.21.2.102:0/1857689185 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f991c009590 0x7f991c00ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:29.198 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.184 7f992f574700 1 --2- 172.21.2.102:0/1857689185 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f991c003aa0 0x7f991c005ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:29.198 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.185 7f992ed73700 1 -- 172.21.2.102:0/1857689185 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f991c009590 msgr2=0x7f991c00ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:29.198 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.185 7f992ed73700 1 --2- 172.21.2.102:0/1857689185 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f991c009590 0x7f991c00ba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:29.198 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.185 7f992ed73700 1 -- 172.21.2.102:0/1857689185 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f991c003aa0 msgr2=0x7f991c005ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:29.198 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.185 7f992ed73700 1 --2- 172.21.2.102:0/1857689185 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f991c003aa0 0x7f991c005ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:29.198 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.185 7f992ed73700 1 -- 172.21.2.102:0/1857689185 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f991c050950 con 0x7f991c006770 2021-02-10T11:22:29.198 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.185 7f992f574700 1 --2- 172.21.2.102:0/1857689185 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f991c003aa0 0x7f991c005ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:22:29.198 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.185 7f992ed73700 1 --2- 172.21.2.102:0/1857689185 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f991c006770 0x7f991c008c90 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 rx=0x7f99180078f0 tx=0x7f9918007c90).ready entity=mon.0 client_cookie=1ad730335fdea692 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:29.198 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.185 7f992e572700 1 -- 172.21.2.102:0/1857689185 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f99180070c0 con 0x7f991c006770 2021-02-10T11:22:29.198 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.185 7f992e572700 1 -- 172.21.2.102:0/1857689185 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f991800a520 con 0x7f991c006770 2021-02-10T11:22:29.198 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.185 7f992e572700 1 -- 172.21.2.102:0/1857689185 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f99180010e0 con 0x7f991c006770 2021-02-10T11:22:29.199 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.185 7f993582e700 1 -- 172.21.2.102:0/1857689185 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f991c006770 msgr2=0x7f991c008c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:29.199 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.185 7f993582e700 1 --2- 172.21.2.102:0/1857689185 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f991c006770 0x7f991c008c90 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 rx=0x7f99180078f0 tx=0x7f9918007c90).stop 2021-02-10T11:22:29.199 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.185 7f992fd75700 1 -- 172.21.2.102:0/1857689185 reap_dead start 2021-02-10T11:22:29.199 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.185 7f993582e700 1 -- 172.21.2.102:0/1857689185 shutdown_connections 2021-02-10T11:22:29.199 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.185 7f993582e700 1 -- 172.21.2.102:0/1857689185 >> 172.21.2.102:0/1857689185 conn(0x7f99300ce690 msgr2=0x7f99300e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:29.199 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.185 7f993582e700 1 -- 172.21.2.102:0/1857689185 shutdown_connections 2021-02-10T11:22:29.199 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.185 7f993582e700 1 -- 172.21.2.102:0/1857689185 wait complete. 2021-02-10T11:22:29.199 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.200 7f993582e700 1 Processor -- start 2021-02-10T11:22:29.200 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.200 7f993582e700 1 -- start start 2021-02-10T11:22:29.201 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.200 7f993582e700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f99301acb70 0x7f993008da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:29.201 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.200 7f993582e700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f9930187eb0 0x7f99300b85a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:29.201 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.200 7f993582e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f99300b8bc0 0x7f9930188e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:29.201 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.200 7f993582e700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9930188940 con 0x7f99300b8bc0 2021-02-10T11:22:29.201 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.200 7f993582e700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f993008e070 con 0x7f9930187eb0 2021-02-10T11:22:29.201 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.200 7f993582e700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99301ac6d0 con 0x7f99301acb70 2021-02-10T11:22:29.201 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.201 7f992fd75700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f99300b8bc0 0x7f9930188e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:29.201 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.201 7f992fd75700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f99300b8bc0 0x7f9930188e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:55478/0 (socket says 172.21.2.102:55478) 2021-02-10T11:22:29.201 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.201 7f992fd75700 1 -- 172.21.2.102:0/1891753634 learned_addr learned my addr 172.21.2.102:0/1891753634 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:22:29.201 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.201 7f992f574700 1 --2- 172.21.2.102:0/1891753634 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f99301acb70 0x7f993008da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:29.201 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.201 7f992ed73700 1 --2- 172.21.2.102:0/1891753634 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f9930187eb0 0x7f99300b85a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:29.201 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.201 7f992fd75700 1 -- 172.21.2.102:0/1891753634 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f99301acb70 msgr2=0x7f993008da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:29.202 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.201 7f992fd75700 1 --2- 172.21.2.102:0/1891753634 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f99301acb70 0x7f993008da10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:29.202 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.201 7f992fd75700 1 -- 172.21.2.102:0/1891753634 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f9930187eb0 msgr2=0x7f99300b85a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:29.202 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.201 7f992fd75700 1 --2- 172.21.2.102:0/1891753634 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f9930187eb0 0x7f99300b85a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:29.202 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.201 7f992fd75700 1 -- 172.21.2.102:0/1891753634 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99301882b0 con 0x7f99300b8bc0 2021-02-10T11:22:29.202 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.201 7f992fd75700 1 --2- 172.21.2.102:0/1891753634 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f99300b8bc0 0x7f9930188e40 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 rx=0x7f992800a940 tx=0x7f992800ace0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:29.202 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.201 7f9927fff700 1 -- 172.21.2.102:0/1891753634 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f99280066f0 con 0x7f99300b8bc0 2021-02-10T11:22:29.202 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.201 7f9927fff700 1 -- 172.21.2.102:0/1891753634 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9928003030 con 0x7f99300b8bc0 2021-02-10T11:22:29.202 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.201 7f993582e700 1 -- 172.21.2.102:0/1891753634 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99301882b0 con 0x7f99300b8bc0 2021-02-10T11:22:29.202 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.201 7f993582e700 1 -- 172.21.2.102:0/1891753634 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9930189460 con 0x7f99300b8bc0 2021-02-10T11:22:29.202 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.201 7f9927fff700 1 -- 172.21.2.102:0/1891753634 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f99280015f0 con 0x7f99300b8bc0 2021-02-10T11:22:29.202 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.202 7f9927fff700 1 -- 172.21.2.102:0/1891753634 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f99280011f0 con 0x7f99300b8bc0 2021-02-10T11:22:29.202 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.202 7f9927fff700 1 --2- 172.21.2.102:0/1891753634 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f99140186c0 0x7f991401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:29.202 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.202 7f9927fff700 1 -- 172.21.2.102:0/1891753634 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f99280328f0 con 0x7f99300b8bc0 2021-02-10T11:22:29.203 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.202 7f992f574700 1 --2- 172.21.2.102:0/1891753634 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f99140186c0 0x7f991401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:29.203 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.203 7f992f574700 1 --2- 172.21.2.102:0/1891753634 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f99140186c0 0x7f991401abc0 crc :-1 s=READY pgs=517 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:29.228 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.232 7f993582e700 1 -- 172.21.2.102:0/1891753634 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f993007ed80 con 0x7f99300b8bc0 2021-02-10T11:22:29.231 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.235 7f9927fff700 1 -- 172.21.2.102:0/1891753634 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f992802e5b0 con 0x7f99300b8bc0 2021-02-10T11:22:29.355 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.359 7f993582e700 1 -- 172.21.2.102:0/1891753634 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f99301888c0 con 0x7f99300b8bc0 2021-02-10T11:22:29.356 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.360 7f9927fff700 1 -- 172.21.2.102:0/1891753634 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v39) v1 ==== 74+0+13993 (secure 0 0 0) 0x7f992802e5b0 con 0x7f99300b8bc0 2021-02-10T11:22:29.356 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:22:29.356 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":39,"fsid":"68db30ea-dbce-4aef-a6df-f01c6fea0513","created":"2021-02-10 11:03:06.331557","modified":"2021-02-10 11:22:27.123992","last_up_change":"2021-02-10 11:03:32.690562","last_in_change":"2021-02-10 11:03:32.690562","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:03:37.322519","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:22:23.578113","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":7,"pool_name":"cephfs_data","create_time":"2021-02-10 11:22:25.111303","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"be691b4d-a039-408a-af80-9d53f02b6f1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6818","nonce":36088},{"type":"v1","addr":"172.21.2.102:6820","nonce":36088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6822","nonce":36088},{"type":"v1","addr":"172.21.2.102:6824","nonce":36088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6830","nonce":36088},{"type":"v1","addr":"172.21.2.102:6832","nonce":36088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6826","nonce":36088},{"type":"v1","addr":"172.21.2.102:6828","nonce":36088}]},"public_addr":"172.21.2.102:6820/36088","cluster_addr":"172.21.2.102:6824/36088","heartbeat_back_addr":"172.21.2.102:6832/36088","heartbeat_front_addr":"172.21.2.102:6828/36088","state":["exists","up"]},{"osd":1,"uuid":"64e3ccc8-9c69-4d7c-865e-af8e83822635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6819","nonce":36087},{"type":"v1","addr":"172.21.2.102:6821","nonce":36087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6823","nonce":36087},{"type":"v1","addr":"172.21.2.102:6825","nonce":36087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6831","nonce":36087},{"type":"v1","addr":"172.21.2.102:6833","nonce":36087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6827","nonce":36087},{"type":"v1","addr":"172.21.2.102:6829","nonce":36087}]},"public_addr":"172.21.2.102:6821/36087","cluster_addr":"172.21.2.102:6825/36087","heartbeat_back_addr":"172.21.2.102:6833/36087","heartbeat_front_addr":"172.21.2.102:6829/36087","state":["exists","up"]},{"osd":2,"uuid":"153700af-747f-481f-bdf4-d6072d3a842f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6810","nonce":36089},{"type":"v1","addr":"172.21.2.102:6811","nonce":36089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6812","nonce":36089},{"type":"v1","addr":"172.21.2.102:6813","nonce":36089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6816","nonce":36089},{"type":"v1","addr":"172.21.2.102:6817","nonce":36089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6814","nonce":36089},{"type":"v1","addr":"172.21.2.102:6815","nonce":36089}]},"public_addr":"172.21.2.102:6811/36089","cluster_addr":"172.21.2.102:6813/36089","heartbeat_back_addr":"172.21.2.102:6817/36089","heartbeat_front_addr":"172.21.2.102:6815/36089","state":["exists","up"]},{"osd":3,"uuid":"cbc678f4-83a9-45e6-8830-2f7c232b1ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6802","nonce":36090},{"type":"v1","addr":"172.21.2.102:6803","nonce":36090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6804","nonce":36090},{"type":"v1","addr":"172.21.2.102:6805","nonce":36090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6808","nonce":36090},{"type":"v1","addr":"172.21.2.102:6809","nonce":36090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6806","nonce":36090},{"type":"v1","addr":"172.21.2.102:6807","nonce":36090}]},"public_addr":"172.21.2.102:6803/36090","cluster_addr":"172.21.2.102:6805/36090","heartbeat_back_addr":"172.21.2.102:6809/36090","heartbeat_front_addr":"172.21.2.102:6807/36090","state":["exists","up"]},{"osd":4,"uuid":"c01e4008-b258-4f10-afd2-9e7f37214351","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6808","nonce":36368},{"type":"v1","addr":"172.21.2.113:6811","nonce":36368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6814","nonce":36368},{"type":"v1","addr":"172.21.2.113:6818","nonce":36368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6826","nonce":36368},{"type":"v1","addr":"172.21.2.113:6828","nonce":36368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6820","nonce":36368},{"type":"v1","addr":"172.21.2.113:6824","nonce":36368}]},"public_addr":"172.21.2.113:6811/36368","cluster_addr":"172.21.2.113:6818/36368","heartbeat_back_addr":"172.21.2.113:6828/36368","heartbeat_front_addr":"172.21.2.113:6824/36368","state":["exists","up"]},{"osd":5,"uuid":"551f6007-daf8-4a98-8ee0-c2b16bcccb5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6802","nonce":36366},{"type":"v1","addr":"172.21.2.113:6805","nonce":36366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6807","nonce":36366},{"type":"v1","addr":"172.21.2.113:6810","nonce":36366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6819","nonce":36366},{"type":"v1","addr":"172.21.2.113:6822","nonce":36366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6812","nonce":36366},{"type":"v1","addr":"172.21.2.113:6815","nonce":36366}]},"public_addr":"172.21.2.113:6805/36366","cluster_addr":"172.21.2.113:6810/36366","heartbeat_back_addr":"172.21.2.113:6822/36366","heartbeat_front_addr":"172.21.2.113:6815/36366","state":["exists","up"]},{"osd":6,"uuid":"bf67c040-0e50-48be-b102-c3943bc9ebef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6800","nonce":36367},{"type":"v1","addr":"172.21.2.113:6801","nonce":36367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6803","nonce":36367},{"type":"v1","addr":"172.21.2.113:6804","nonce":36367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6813","nonce":36367},{"type":"v1","addr":"172.21.2.113:6816","nonce":36367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6806","nonce":36367},{"type":"v1","addr":"172.21.2.113:6809","nonce":36367}]},"public_addr":"172.21.2.113:6801/36367","cluster_addr":"172.21.2.113:6804/36367","heartbeat_back_addr":"172.21.2.113:6816/36367","heartbeat_front_addr":"172.21.2.113:6809/36367","state":["exists","up"]},{"osd":7,"uuid":"f5461f80-018d-428f-be89-2eb2bcbe7bb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6817","nonce":36365},{"type":"v1","addr":"172.21.2.113:6821","nonce":36365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6823","nonce":36365},{"type":"v1","addr":"172.21.2.113:6825","nonce":36365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6830","nonce":36365},{"type":"v1","addr":"172.21.2.113:6831","nonce":36365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6827","nonce":36365},{"type":"v1","addr":"172.21.2.113:6829","nonce":36365}]},"public_addr":"172.21.2.113:6821/36365","cluster_addr":"172.21.2.113:6825/36365","heartbeat_back_addr":"172.21.2.113:6831/36365","heartbeat_front_addr":"172.21.2.113:6829/36365","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:22:29.357 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.361 7f993582e700 1 -- 172.21.2.102:0/1891753634 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f99140186c0 msgr2=0x7f991401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:29.357 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.361 7f993582e700 1 --2- 172.21.2.102:0/1891753634 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f99140186c0 0x7f991401abc0 crc :-1 s=READY pgs=517 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:29.358 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.361 7f993582e700 1 -- 172.21.2.102:0/1891753634 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f99300b8bc0 msgr2=0x7f9930188e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:29.358 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.361 7f993582e700 1 --2- 172.21.2.102:0/1891753634 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f99300b8bc0 0x7f9930188e40 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 rx=0x7f992800a940 tx=0x7f992800ace0).stop 2021-02-10T11:22:29.358 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.361 7f993582e700 1 -- 172.21.2.102:0/1891753634 shutdown_connections 2021-02-10T11:22:29.358 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.361 7f993582e700 1 --2- 172.21.2.102:0/1891753634 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f99300b8bc0 0x7f9930188e40 unknown :-1 s=CLOSED pgs=1071 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:29.358 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.361 7f993582e700 1 --2- 172.21.2.102:0/1891753634 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f99140186c0 0x7f991401abc0 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:29.358 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.361 7f993582e700 1 --2- 172.21.2.102:0/1891753634 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f9930187eb0 0x7f99300b85a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:29.358 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.361 7f993582e700 1 --2- 172.21.2.102:0/1891753634 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f99301acb70 0x7f993008da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:29.358 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.361 7f993582e700 1 -- 172.21.2.102:0/1891753634 >> 172.21.2.102:0/1891753634 conn(0x7f99301a31b0 msgr2=0x7f993018c8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:29.358 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.362 7f993582e700 1 -- 172.21.2.102:0/1891753634 shutdown_connections 2021-02-10T11:22:29.358 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.362 7f993582e700 1 -- 172.21.2.102:0/1891753634 wait complete. 2021-02-10T11:22:29.366 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:22:29.796 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.783 7f9845b80700 1 Processor -- start 2021-02-10T11:22:29.796 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.783 7f9845b80700 1 -- start start 2021-02-10T11:22:29.796 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.783 7f9845b80700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f98401b0470 0x7f98401b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:29.796 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.783 7f9845b80700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98400f0b50 con 0x7f98400f20e0 2021-02-10T11:22:29.796 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.783 7f9845b80700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98400f24e0 con 0x7f98400ef610 2021-02-10T11:22:29.796 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.783 7f9845b80700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98400e5ea0 con 0x7f98401b0470 2021-02-10T11:22:29.797 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.784 7f983ffff700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f98400f20e0 0x7f98400f0200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:47708/0 (socket says 172.21.2.102:47708) 2021-02-10T11:22:29.797 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.784 7f983ffff700 1 -- 172.21.2.102:0/2170240251 learned_addr learned my addr 172.21.2.102:0/2170240251 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:22:29.797 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.784 7f983e7fc700 1 -- 172.21.2.102:0/2170240251 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2472987821 0 0) 0x7f9838001500 con 0x7f98400f20e0 2021-02-10T11:22:29.797 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.784 7f983e7fc700 1 -- 172.21.2.102:0/2170240251 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f982c002fe0 con 0x7f98400f20e0 2021-02-10T11:22:29.797 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.784 7f983f7fe700 1 --2- 172.21.2.102:0/2170240251 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f98401b0470 0x7f98401b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:29.797 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.784 7f983e7fc700 1 -- 172.21.2.102:0/2170240251 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4026431983 0 0) 0x7f9838001460 con 0x7f98400f20e0 2021-02-10T11:22:29.797 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.784 7f983e7fc700 1 -- 172.21.2.102:0/2170240251 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f98401b0470 msgr2=0x7f98401b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:29.797 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.784 7f983e7fc700 1 --2- 172.21.2.102:0/2170240251 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f98401b0470 0x7f98401b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:29.797 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.784 7f983e7fc700 1 -- 172.21.2.102:0/2170240251 >> v1:172.21.2.113:6789/0 conn(0x7f98400ef610 legacy=0x7f98400efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:29.797 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.784 7f983e7fc700 1 -- 172.21.2.102:0/2170240251 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98400e52e0 con 0x7f98400f20e0 2021-02-10T11:22:29.797 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.784 7f983e7fc700 1 -- 172.21.2.102:0/2170240251 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7f9838001980 con 0x7f98400f20e0 2021-02-10T11:22:29.797 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.784 7f983e7fc700 1 -- 172.21.2.102:0/2170240251 >> v1:172.21.2.102:6789/0 conn(0x7f98400f20e0 legacy=0x7f98400f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:29.798 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.784 7f983e7fc700 1 --2- 172.21.2.102:0/2170240251 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f982c003a60 0x7f982c005ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:29.798 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.784 7f983e7fc700 1 --2- 172.21.2.102:0/2170240251 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f982c006790 0x7f982c008c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:29.798 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.784 7f983e7fc700 1 --2- 172.21.2.102:0/2170240251 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f982c009500 0x7f982c00ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:29.798 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.784 7f983e7fc700 1 -- 172.21.2.102:0/2170240251 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f982c00c2d0 con 0x7f982c003a60 2021-02-10T11:22:29.798 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.784 7f983e7fc700 1 -- 172.21.2.102:0/2170240251 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f982c00c4f0 con 0x7f982c009500 2021-02-10T11:22:29.798 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.784 7f983e7fc700 1 -- 172.21.2.102:0/2170240251 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f982c00c710 con 0x7f982c006790 2021-02-10T11:22:29.798 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.785 7f983f7fe700 1 --2- 172.21.2.102:0/2170240251 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f982c003a60 0x7f982c005ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:29.798 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.785 7f983f7fe700 1 -- 172.21.2.102:0/2170240251 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f982c006790 msgr2=0x7f982c008c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:29.798 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.785 7f983f7fe700 1 --2- 172.21.2.102:0/2170240251 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f982c006790 0x7f982c008c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:29.798 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.785 7f983f7fe700 1 -- 172.21.2.102:0/2170240251 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f982c009500 msgr2=0x7f982c00ba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:29.798 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.785 7f983f7fe700 1 --2- 172.21.2.102:0/2170240251 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f982c009500 0x7f982c00ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:29.798 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.785 7f983f7fe700 1 -- 172.21.2.102:0/2170240251 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f982c00c9b0 con 0x7f982c003a60 2021-02-10T11:22:29.799 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.785 7f983f7fe700 1 --2- 172.21.2.102:0/2170240251 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f982c003a60 0x7f982c005ef0 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 rx=0x7f9828006640 tx=0x7f982804da90).ready entity=mon.0 client_cookie=f959829461b58e55 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:29.799 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.785 7f983e7fc700 1 -- 172.21.2.102:0/2170240251 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f98280500c0 con 0x7f982c003a60 2021-02-10T11:22:29.799 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.785 7f983e7fc700 1 -- 172.21.2.102:0/2170240251 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9828050440 con 0x7f982c003a60 2021-02-10T11:22:29.799 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.785 7f983e7fc700 1 -- 172.21.2.102:0/2170240251 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9828050960 con 0x7f982c003a60 2021-02-10T11:22:29.799 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.785 7f9845b80700 1 -- 172.21.2.102:0/2170240251 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f982c003a60 msgr2=0x7f982c005ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:29.799 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.785 7f9845b80700 1 --2- 172.21.2.102:0/2170240251 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f982c003a60 0x7f982c005ef0 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 rx=0x7f9828006640 tx=0x7f982804da90).stop 2021-02-10T11:22:29.799 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.785 7f983ffff700 1 -- 172.21.2.102:0/2170240251 reap_dead start 2021-02-10T11:22:29.799 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.786 7f9845b80700 1 -- 172.21.2.102:0/2170240251 shutdown_connections 2021-02-10T11:22:29.799 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.786 7f9845b80700 1 -- 172.21.2.102:0/2170240251 >> 172.21.2.102:0/2170240251 conn(0x7f98400ce3e0 msgr2=0x7f98400e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:29.799 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.786 7f9845b80700 1 -- 172.21.2.102:0/2170240251 shutdown_connections 2021-02-10T11:22:29.799 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.786 7f9845b80700 1 -- 172.21.2.102:0/2170240251 wait complete. 2021-02-10T11:22:29.799 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.800 7f9845b80700 1 Processor -- start 2021-02-10T11:22:29.801 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.800 7f9845b80700 1 -- start start 2021-02-10T11:22:29.801 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.801 7f9845b80700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f98400b8530 0x7f98400b8930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:29.801 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.801 7f9845b80700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f98400b8f50 0x7f98400b9350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:29.802 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.801 7f9845b80700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f98400b65c0 0x7f98400b69c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:29.802 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.801 7f9845b80700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98400b99f0 con 0x7f98400b8f50 2021-02-10T11:22:29.802 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.801 7f9845b80700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98400b8020 con 0x7f98400b8530 2021-02-10T11:22:29.802 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.801 7f9845b80700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9840091650 con 0x7f98400b65c0 2021-02-10T11:22:29.802 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.801 7f983effd700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f98400b8f50 0x7f98400b9350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:29.802 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.801 7f983effd700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f98400b8f50 0x7f98400b9350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:55498/0 (socket says 172.21.2.102:55498) 2021-02-10T11:22:29.802 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.801 7f983effd700 1 -- 172.21.2.102:0/1823985212 learned_addr learned my addr 172.21.2.102:0/1823985212 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:22:29.802 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.801 7f983effd700 1 -- 172.21.2.102:0/1823985212 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f98400b65c0 msgr2=0x7f98400b69c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:29.803 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.801 7f983effd700 1 --2- 172.21.2.102:0/1823985212 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f98400b65c0 0x7f98400b69c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:29.803 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.801 7f983effd700 1 -- 172.21.2.102:0/1823985212 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f98400b8530 msgr2=0x7f98400b8930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:29.803 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.801 7f983effd700 1 --2- 172.21.2.102:0/1823985212 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f98400b8530 0x7f98400b8930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:29.803 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.801 7f983effd700 1 -- 172.21.2.102:0/1823985212 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98400b6fe0 con 0x7f98400b8f50 2021-02-10T11:22:29.803 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.802 7f983effd700 1 --2- 172.21.2.102:0/1823985212 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f98400b8f50 0x7f98400b9350 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 rx=0x7f9830006ed0 tx=0x7f983000b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:29.803 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.802 7f9837fff700 1 -- 172.21.2.102:0/1823985212 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9830006240 con 0x7f98400b8f50 2021-02-10T11:22:29.803 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.802 7f9837fff700 1 -- 172.21.2.102:0/1823985212 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98300054b0 con 0x7f98400b8f50 2021-02-10T11:22:29.803 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.802 7f9837fff700 1 -- 172.21.2.102:0/1823985212 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9830005910 con 0x7f98400b8f50 2021-02-10T11:22:29.803 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.802 7f9845b80700 1 -- 172.21.2.102:0/1823985212 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98400b6fe0 con 0x7f98400b8f50 2021-02-10T11:22:29.803 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.802 7f9845b80700 1 -- 172.21.2.102:0/1823985212 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98400b6fe0 con 0x7f98400b8f50 2021-02-10T11:22:29.804 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.802 7f9837fff700 1 -- 172.21.2.102:0/1823985212 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f9830005690 con 0x7f98400b8f50 2021-02-10T11:22:29.804 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.802 7f9837fff700 1 --2- 172.21.2.102:0/1823985212 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f98240186c0 0x7f982401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:29.804 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.802 7f9837fff700 1 -- 172.21.2.102:0/1823985212 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f98300323c0 con 0x7f98400b8f50 2021-02-10T11:22:29.804 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.803 7f983f7fe700 1 --2- 172.21.2.102:0/1823985212 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f98240186c0 0x7f982401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:29.804 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.803 7f983f7fe700 1 --2- 172.21.2.102:0/1823985212 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f98240186c0 0x7f982401abc0 crc :-1 s=READY pgs=518 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:29.829 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.833 7f9845b80700 1 -- 172.21.2.102:0/1823985212 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9840105a30 con 0x7f98400b8f50 2021-02-10T11:22:29.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.835 7f9837fff700 1 -- 172.21.2.102:0/1823985212 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f9830030cf0 con 0x7f98400b8f50 2021-02-10T11:22:29.961 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.965 7f9845b80700 1 -- 172.21.2.102:0/1823985212 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f98400f0b50 con 0x7f98400b8f50 2021-02-10T11:22:29.964 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.968 7f9837fff700 1 -- 172.21.2.102:0/1823985212 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.0 v15) v1 ==== 184+0+0 (secure 0 0 0) 0x7f983001f4d0 con 0x7f98400b8f50 2021-02-10T11:22:29.968 INFO:teuthology.orchestra.run.gibba002.stderr:updated caps for client.0 2021-02-10T11:22:29.969 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.973 7f9845b80700 1 -- 172.21.2.102:0/1823985212 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f98240186c0 msgr2=0x7f982401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:29.970 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.973 7f9845b80700 1 --2- 172.21.2.102:0/1823985212 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f98240186c0 0x7f982401abc0 crc :-1 s=READY pgs=518 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:29.970 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.973 7f9845b80700 1 -- 172.21.2.102:0/1823985212 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f98400b8f50 msgr2=0x7f98400b9350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:29.970 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.973 7f9845b80700 1 --2- 172.21.2.102:0/1823985212 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f98400b8f50 0x7f98400b9350 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 rx=0x7f9830006ed0 tx=0x7f983000b960).stop 2021-02-10T11:22:29.970 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.973 7f9845b80700 1 -- 172.21.2.102:0/1823985212 shutdown_connections 2021-02-10T11:22:29.970 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.973 7f9845b80700 1 --2- 172.21.2.102:0/1823985212 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f98400b65c0 0x7f98400b69c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:29.970 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.973 7f9845b80700 1 --2- 172.21.2.102:0/1823985212 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f98400b8f50 0x7f98400b9350 unknown :-1 s=CLOSED pgs=1074 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:29.970 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.973 7f9845b80700 1 --2- 172.21.2.102:0/1823985212 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f98240186c0 0x7f982401abc0 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:29.970 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.973 7f9845b80700 1 --2- 172.21.2.102:0/1823985212 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f98400b8530 0x7f98400b8930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:29.970 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.973 7f9845b80700 1 -- 172.21.2.102:0/1823985212 >> 172.21.2.102:0/1823985212 conn(0x7f98400b2800 msgr2=0x7f9840091a50 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:29.970 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.974 7f9845b80700 1 -- 172.21.2.102:0/1823985212 shutdown_connections 2021-02-10T11:22:29.970 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:29.974 7f9845b80700 1 -- 172.21.2.102:0/1823985212 wait complete. 2021-02-10T11:22:29.978 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.1 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:22:30.417 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:30.403 7fdd67aca700 1 Processor -- start 2021-02-10T11:22:30.417 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:30.403 7fdd67aca700 1 -- start start 2021-02-10T11:22:30.417 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:30.404 7fdd67aca700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdd600f05f0 0x7fdd600f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:30.418 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:30.404 7fdd67aca700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd601b0c70 con 0x7fdd601b0470 2021-02-10T11:22:30.418 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:30.404 7fdd67aca700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd600f0b50 con 0x7fdd601b0870 2021-02-10T11:22:30.418 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:30.404 7fdd67aca700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd600e5ea0 con 0x7fdd600f05f0 2021-02-10T11:22:30.418 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:30.404 7fdd658e6700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fdd601b0470 0x7fdd600ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:47726/0 (socket says 172.21.2.102:47726) 2021-02-10T11:22:30.418 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:30.404 7fdd658e6700 1 -- 172.21.2.102:0/4284644469 learned_addr learned my addr 172.21.2.102:0/4284644469 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:22:30.418 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:30.404 7fdd648e4700 1 -- 172.21.2.102:0/4284644469 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 321202037 0 0) 0x7fdd54000ce0 con 0x7fdd601b0470 2021-02-10T11:22:30.418 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:30.404 7fdd648e4700 1 -- 172.21.2.102:0/4284644469 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd4c002fe0 con 0x7fdd601b0470 2021-02-10T11:22:30.418 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:30.404 7fdd660e7700 1 --2- 172.21.2.102:0/4284644469 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdd600f05f0 0x7fdd600f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:30.418 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:30.405 7fdd648e4700 1 -- 172.21.2.102:0/4284644469 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 325609574 0 0) 0x7fdd540010e0 con 0x7fdd601b0470 2021-02-10T11:22:30.418 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:30.405 7fdd648e4700 1 -- 172.21.2.102:0/4284644469 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdd600f05f0 msgr2=0x7fdd600f20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:30.418 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:30.405 7fdd648e4700 1 --2- 172.21.2.102:0/4284644469 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdd600f05f0 0x7fdd600f20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:30.418 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:30.405 7fdd648e4700 1 -- 172.21.2.102:0/4284644469 >> v1:172.21.2.113:6789/0 conn(0x7fdd601b0870 legacy=0x7fdd600efe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:30.418 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:30.405 7fdd648e4700 1 -- 172.21.2.102:0/4284644469 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd600e52e0 con 0x7fdd601b0470 2021-02-10T11:22:30.419 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:30.405 7fdd648e4700 1 -- 172.21.2.102:0/4284644469 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7fdd54001660 con 0x7fdd601b0470 2021-02-10T11:22:30.419 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:30.405 7fdd648e4700 1 -- 172.21.2.102:0/4284644469 >> v1:172.21.2.102:6789/0 conn(0x7fdd601b0470 legacy=0x7fdd600ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:30.419 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:30.405 7fdd648e4700 1 --2- 172.21.2.102:0/4284644469 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdd4c003ae0 0x7fdd4c005ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:30.419 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:30.405 7fdd648e4700 1 --2- 172.21.2.102:0/4284644469 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fdd4c006770 0x7fdd4c008c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:30.419 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:30.405 7fdd648e4700 1 --2- 172.21.2.102:0/4284644469 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdd4c009590 0x7fdd4c00ba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:30.419 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:30.405 7fdd648e4700 1 -- 172.21.2.102:0/4284644469 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd4c00c2d0 con 0x7fdd4c003ae0 2021-02-10T11:22:30.419 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:30.405 7fdd648e4700 1 -- 172.21.2.102:0/4284644469 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd4c00c4f0 con 0x7fdd4c006770 2021-02-10T11:22:30.419 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:30.405 7fdd648e4700 1 -- 172.21.2.102:0/4284644469 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd4c00c710 con 0x7fdd4c009590 2021-02-10T11:22:30.419 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:30.405 7fdd658e6700 1 --2- 172.21.2.102:0/4284644469 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdd4c003ae0 0x7fdd4c005ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:30.419 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:30.405 7fdd650e5700 1 --2- 172.21.2.102:0/4284644469 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fdd4c006770 0x7fdd4c008c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:30.419 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:30.405 7fdd660e7700 1 --2- 172.21.2.102:0/4284644469 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdd4c009590 0x7fdd4c00ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:30.419 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:30.405 7fdd658e6700 1 -- 172.21.2.102:0/4284644469 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdd4c009590 msgr2=0x7fdd4c00ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:30.419 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:30.405 7fdd658e6700 1 --2- 172.21.2.102:0/4284644469 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdd4c009590 0x7fdd4c00ba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:30.419 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:30.405 7fdd658e6700 1 -- 172.21.2.102:0/4284644469 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fdd4c006770 msgr2=0x7fdd4c008c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:30.420 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:30.405 7fdd658e6700 1 --2- 172.21.2.102:0/4284644469 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fdd4c006770 0x7fdd4c008c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:30.420 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:30.405 7fdd658e6700 1 -- 172.21.2.102:0/4284644469 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd4c050980 con 0x7fdd4c003ae0 2021-02-10T11:22:30.420 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:30.405 7fdd658e6700 1 --2- 172.21.2.102:0/4284644469 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdd4c003ae0 0x7fdd4c005ef0 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 rx=0x7fdd540016f0 tx=0x7fdd54007d00).ready entity=mon.0 client_cookie=7f1c7e6dd2f0a06d server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:30.420 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:30.406 7fdd648e4700 1 -- 172.21.2.102:0/4284644469 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdd540013a0 con 0x7fdd4c003ae0 2021-02-10T11:22:30.420 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:30.406 7fdd648e4700 1 -- 172.21.2.102:0/4284644469 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd540070c0 con 0x7fdd4c003ae0 2021-02-10T11:22:30.420 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:30.406 7fdd648e4700 1 -- 172.21.2.102:0/4284644469 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdd54004590 con 0x7fdd4c003ae0 2021-02-10T11:22:30.420 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:30.406 7fdd67aca700 1 -- 172.21.2.102:0/4284644469 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdd4c003ae0 msgr2=0x7fdd4c005ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:30.420 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:30.406 7fdd67aca700 1 --2- 172.21.2.102:0/4284644469 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdd4c003ae0 0x7fdd4c005ef0 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 rx=0x7fdd540016f0 tx=0x7fdd54007d00).stop 2021-02-10T11:22:30.420 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:30.406 7fdd660e7700 1 -- 172.21.2.102:0/4284644469 reap_dead start 2021-02-10T11:22:30.420 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:30.406 7fdd67aca700 1 -- 172.21.2.102:0/4284644469 shutdown_connections 2021-02-10T11:22:30.420 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:30.406 7fdd67aca700 1 -- 172.21.2.102:0/4284644469 >> 172.21.2.102:0/4284644469 conn(0x7fdd600ce3e0 msgr2=0x7fdd600e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:30.420 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:30.406 7fdd67aca700 1 -- 172.21.2.102:0/4284644469 shutdown_connections 2021-02-10T11:22:30.420 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:30.406 7fdd67aca700 1 -- 172.21.2.102:0/4284644469 wait complete. 2021-02-10T11:22:30.420 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:30.421 7fdd67aca700 1 Processor -- start 2021-02-10T11:22:30.421 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:30.421 7fdd67aca700 1 -- start start 2021-02-10T11:22:30.422 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:30.422 7fdd67aca700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdd60090660 0x7fdd6008fae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:30.422 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:30.422 7fdd67aca700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fdd60090100 0x7fdd60090b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:30.422 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:30.422 7fdd67aca700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdd60091150 0x7fdd600b8850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:30.422 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:30.422 7fdd67aca700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd60091550 con 0x7fdd60091150 2021-02-10T11:22:30.422 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:30.422 7fdd67aca700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd60091740 con 0x7fdd60090100 2021-02-10T11:22:30.422 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:30.422 7fdd67aca700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd600b8e70 con 0x7fdd60090660 2021-02-10T11:22:30.422 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:30.422 7fdd660e7700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdd60091150 0x7fdd600b8850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:30.422 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:30.422 7fdd660e7700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdd60091150 0x7fdd600b8850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:55512/0 (socket says 172.21.2.102:55512) 2021-02-10T11:22:30.422 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:30.422 7fdd660e7700 1 -- 172.21.2.102:0/1003928789 learned_addr learned my addr 172.21.2.102:0/1003928789 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:22:30.423 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:30.422 7fdd660e7700 1 -- 172.21.2.102:0/1003928789 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdd60090660 msgr2=0x7fdd6008fae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:30.423 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:30.422 7fdd660e7700 1 --2- 172.21.2.102:0/1003928789 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdd60090660 0x7fdd6008fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:30.423 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:30.422 7fdd660e7700 1 -- 172.21.2.102:0/1003928789 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fdd60090100 msgr2=0x7fdd60090b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:30.423 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:30.422 7fdd660e7700 1 --2- 172.21.2.102:0/1003928789 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fdd60090100 0x7fdd60090b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:30.423 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:30.422 7fdd660e7700 1 -- 172.21.2.102:0/1003928789 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd600b9090 con 0x7fdd60091150 2021-02-10T11:22:30.423 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:30.422 7fdd660e7700 1 --2- 172.21.2.102:0/1003928789 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdd60091150 0x7fdd600b8850 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 rx=0x7fdd58009a30 tx=0x7fdd5800bd20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:30.423 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:30.422 7fdd527fc700 1 -- 172.21.2.102:0/1003928789 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdd58001400 con 0x7fdd60091150 2021-02-10T11:22:30.423 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:30.422 7fdd527fc700 1 -- 172.21.2.102:0/1003928789 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd5800be60 con 0x7fdd60091150 2021-02-10T11:22:30.423 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:30.422 7fdd527fc700 1 -- 172.21.2.102:0/1003928789 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdd58001900 con 0x7fdd60091150 2021-02-10T11:22:30.423 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:30.422 7fdd67aca700 1 -- 172.21.2.102:0/1003928789 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd600b9090 con 0x7fdd60091150 2021-02-10T11:22:30.423 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:30.423 7fdd67aca700 1 -- 172.21.2.102:0/1003928789 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd600b9090 con 0x7fdd60091150 2021-02-10T11:22:30.423 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:30.423 7fdd527fc700 1 -- 172.21.2.102:0/1003928789 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fdd580069b0 con 0x7fdd60091150 2021-02-10T11:22:30.424 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:30.423 7fdd527fc700 1 --2- 172.21.2.102:0/1003928789 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fdd480186c0 0x7fdd4801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:30.424 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:30.423 7fdd527fc700 1 -- 172.21.2.102:0/1003928789 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fdd58032a10 con 0x7fdd60091150 2021-02-10T11:22:30.424 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:30.423 7fdd658e6700 1 --2- 172.21.2.102:0/1003928789 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fdd480186c0 0x7fdd4801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:30.424 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:30.424 7fdd658e6700 1 --2- 172.21.2.102:0/1003928789 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fdd480186c0 0x7fdd4801abc0 crc :-1 s=READY pgs=519 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:30.449 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:30.453 7fdd67aca700 1 -- 172.21.2.102:0/1003928789 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd60105a30 con 0x7fdd60091150 2021-02-10T11:22:30.452 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:30.456 7fdd527fc700 1 -- 172.21.2.102:0/1003928789 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fdd58032c20 con 0x7fdd60091150 2021-02-10T11:22:30.582 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:30.586 7fdd67aca700 1 -- 172.21.2.102:0/1003928789 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7fdd600f0b50 con 0x7fdd60091150 2021-02-10T11:22:30.586 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:30.590 7fdd527fc700 1 -- 172.21.2.102:0/1003928789 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.1 v16) v1 ==== 184+0+0 (secure 0 0 0) 0x7fdd58006bc0 con 0x7fdd60091150 2021-02-10T11:22:30.589 INFO:teuthology.orchestra.run.gibba002.stderr:updated caps for client.1 2021-02-10T11:22:30.591 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:30.595 7fdd67aca700 1 -- 172.21.2.102:0/1003928789 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fdd480186c0 msgr2=0x7fdd4801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:30.592 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:30.595 7fdd67aca700 1 --2- 172.21.2.102:0/1003928789 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fdd480186c0 0x7fdd4801abc0 crc :-1 s=READY pgs=519 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:30.592 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:30.595 7fdd67aca700 1 -- 172.21.2.102:0/1003928789 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdd60091150 msgr2=0x7fdd600b8850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:30.592 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:30.595 7fdd67aca700 1 --2- 172.21.2.102:0/1003928789 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdd60091150 0x7fdd600b8850 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 rx=0x7fdd58009a30 tx=0x7fdd5800bd20).stop 2021-02-10T11:22:30.592 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:30.595 7fdd67aca700 1 -- 172.21.2.102:0/1003928789 shutdown_connections 2021-02-10T11:22:30.592 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:30.595 7fdd67aca700 1 --2- 172.21.2.102:0/1003928789 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdd60091150 0x7fdd600b8850 unknown :-1 s=CLOSED pgs=1077 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:30.592 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:30.595 7fdd67aca700 1 --2- 172.21.2.102:0/1003928789 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fdd480186c0 0x7fdd4801abc0 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:30.593 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:30.595 7fdd67aca700 1 --2- 172.21.2.102:0/1003928789 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fdd60090100 0x7fdd60090b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:30.593 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:30.595 7fdd67aca700 1 --2- 172.21.2.102:0/1003928789 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdd60090660 0x7fdd6008fae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:30.593 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:30.595 7fdd67aca700 1 -- 172.21.2.102:0/1003928789 >> 172.21.2.102:0/1003928789 conn(0x7fdd600b2800 msgr2=0x7fdd60091a50 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:30.593 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:30.595 7fdd67aca700 1 -- 172.21.2.102:0/1003928789 shutdown_connections 2021-02-10T11:22:30.593 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:30.595 7fdd67aca700 1 -- 172.21.2.102:0/1003928789 wait complete. 2021-02-10T11:22:30.600 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.2 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:22:31.038 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:31.025 7f285a0bb700 1 Processor -- start 2021-02-10T11:22:31.038 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:31.025 7f285a0bb700 1 -- start start 2021-02-10T11:22:31.038 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:31.026 7f285a0bb700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f285407e6a0 0x7f28540d1790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:31.038 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:31.026 7f285a0bb700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f285407ed80 con 0x7f28540e5070 2021-02-10T11:22:31.038 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:31.026 7f285a0bb700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f285407eaa0 con 0x7f28540e5470 2021-02-10T11:22:31.038 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:31.026 7f285a0bb700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28541b4080 con 0x7f285407e6a0 2021-02-10T11:22:31.038 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:31.026 7f28537fe700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f28540e5070 0x7f28541b3820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:47744/0 (socket says 172.21.2.102:47744) 2021-02-10T11:22:31.038 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:31.026 7f28537fe700 1 -- 172.21.2.102:0/3143498427 learned_addr learned my addr 172.21.2.102:0/3143498427 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:22:31.038 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:31.026 7f2853fff700 1 --2- 172.21.2.102:0/3143498427 >> v2:172.21.2.113:3300/0 conn(0x7f285407e6a0 0x7f28540d1790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:31.038 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:31.026 7f28527fc700 1 -- 172.21.2.102:0/3143498427 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1191061761 0 0) 0x7f284c001500 con 0x7f28540e5070 2021-02-10T11:22:31.039 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:31.026 7f28527fc700 1 -- 172.21.2.102:0/3143498427 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f283c002fe0 con 0x7f28540e5070 2021-02-10T11:22:31.039 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:31.026 7f28527fc700 1 -- 172.21.2.102:0/3143498427 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 774862620 0 0) 0x7f284c001460 con 0x7f28540e5070 2021-02-10T11:22:31.039 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:31.026 7f2853fff700 1 -- 172.21.2.102:0/3143498427 >> v1:172.21.2.113:6789/0 conn(0x7f28540e5470 legacy=0x7f285407f300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:31.039 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:31.026 7f2853fff700 1 -- 172.21.2.102:0/3143498427 >> v1:172.21.2.102:6789/0 conn(0x7f28540e5070 legacy=0x7f28541b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:31.039 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:31.026 7f2853fff700 1 -- 172.21.2.102:0/3143498427 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28540d1db0 con 0x7f285407e6a0 2021-02-10T11:22:31.039 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:31.027 7f2853fff700 1 --2- 172.21.2.102:0/3143498427 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f285407e6a0 0x7f28540d1790 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 rx=0x7f2844006ad0 tx=0x7f2844009ad0).ready entity=mon.1 client_cookie=1f8ffd76ce538b6d server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:31.039 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:31.027 7f28527fc700 1 -- 172.21.2.102:0/3143498427 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f28440022e0 con 0x7f285407e6a0 2021-02-10T11:22:31.039 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:31.027 7f28527fc700 1 -- 172.21.2.102:0/3143498427 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2844006410 con 0x7f285407e6a0 2021-02-10T11:22:31.039 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:31.027 7f28527fc700 1 -- 172.21.2.102:0/3143498427 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2844002560 con 0x7f285407e6a0 2021-02-10T11:22:31.039 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:31.027 7f285a0bb700 1 -- 172.21.2.102:0/3143498427 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f285407e6a0 msgr2=0x7f28540d1790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:31.039 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:31.027 7f285a0bb700 1 --2- 172.21.2.102:0/3143498427 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f285407e6a0 0x7f28540d1790 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 rx=0x7f2844006ad0 tx=0x7f2844009ad0).stop 2021-02-10T11:22:31.039 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:31.027 7f285a0bb700 1 -- 172.21.2.102:0/3143498427 shutdown_connections 2021-02-10T11:22:31.039 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:31.027 7f285a0bb700 1 --2- 172.21.2.102:0/3143498427 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f285407e6a0 0x7f28540d1790 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:31.040 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:31.027 7f285a0bb700 1 -- 172.21.2.102:0/3143498427 >> 172.21.2.102:0/3143498427 conn(0x7f28540ce690 msgr2=0x7f28540e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:31.040 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:31.027 7f285a0bb700 1 -- 172.21.2.102:0/3143498427 shutdown_connections 2021-02-10T11:22:31.040 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:31.027 7f285a0bb700 1 -- 172.21.2.102:0/3143498427 wait complete. 2021-02-10T11:22:31.040 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:31.042 7f285a0bb700 1 Processor -- start 2021-02-10T11:22:31.040 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:31.042 7f285a0bb700 1 -- start start 2021-02-10T11:22:31.041 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:31.042 7f285a0bb700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f285408ce50 0x7f2854093020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:31.041 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:31.042 7f285a0bb700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f285408d250 0x7f2854093640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:31.041 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:31.042 7f285a0bb700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f285408c440 0x7f2854087ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:31.041 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:31.042 7f285a0bb700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f285408c8d0 con 0x7f285408ce50 2021-02-10T11:22:31.041 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:31.042 7f285a0bb700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2854089430 con 0x7f285408d250 2021-02-10T11:22:31.041 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:31.042 7f285a0bb700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2854088f40 con 0x7f285408c440 2021-02-10T11:22:31.041 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:31.042 7f28537fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f285408ce50 0x7f2854093020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:31.042 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:31.042 7f28537fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f285408ce50 0x7f2854093020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:55526/0 (socket says 172.21.2.102:55526) 2021-02-10T11:22:31.042 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:31.042 7f28537fe700 1 -- 172.21.2.102:0/2570241392 learned_addr learned my addr 172.21.2.102:0/2570241392 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:22:31.042 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:31.043 7f28537fe700 1 -- 172.21.2.102:0/2570241392 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f285408c440 msgr2=0x7f2854087ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:31.042 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:31.043 7f28537fe700 1 --2- 172.21.2.102:0/2570241392 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f285408c440 0x7f2854087ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:31.042 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:31.043 7f28537fe700 1 -- 172.21.2.102:0/2570241392 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f285408d250 msgr2=0x7f2854093640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:31.042 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:31.043 7f28537fe700 1 --2- 172.21.2.102:0/2570241392 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f285408d250 0x7f2854093640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:31.042 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:31.043 7f28537fe700 1 -- 172.21.2.102:0/2570241392 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2854088510 con 0x7f285408ce50 2021-02-10T11:22:31.042 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:31.043 7f28537fe700 1 --2- 172.21.2.102:0/2570241392 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f285408ce50 0x7f2854093020 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 rx=0x7f284c007fc0 tx=0x7f284c00ac90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:31.042 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:31.043 7f283bfff700 1 -- 172.21.2.102:0/2570241392 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f284c001e70 con 0x7f285408ce50 2021-02-10T11:22:31.042 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:31.043 7f283bfff700 1 -- 172.21.2.102:0/2570241392 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f284c004930 con 0x7f285408ce50 2021-02-10T11:22:31.042 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:31.043 7f283bfff700 1 -- 172.21.2.102:0/2570241392 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f284c004d90 con 0x7f285408ce50 2021-02-10T11:22:31.043 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:31.043 7f285a0bb700 1 -- 172.21.2.102:0/2570241392 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2854088510 con 0x7f285408ce50 2021-02-10T11:22:31.043 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:31.043 7f285a0bb700 1 -- 172.21.2.102:0/2570241392 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2854088aa0 con 0x7f285408ce50 2021-02-10T11:22:31.043 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:31.044 7f283bfff700 1 -- 172.21.2.102:0/2570241392 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f284c00add0 con 0x7f285408ce50 2021-02-10T11:22:31.043 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:31.044 7f283bfff700 1 --2- 172.21.2.102:0/2570241392 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2834018730 0x7f283401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:31.043 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:31.044 7f283bfff700 1 -- 172.21.2.102:0/2570241392 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f284c0329e0 con 0x7f285408ce50 2021-02-10T11:22:31.043 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:31.044 7f2852ffd700 1 --2- 172.21.2.102:0/2570241392 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2834018730 0x7f283401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:31.043 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:31.044 7f2852ffd700 1 --2- 172.21.2.102:0/2570241392 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2834018730 0x7f283401ac10 crc :-1 s=READY pgs=520 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:31.071 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:31.076 7f285a0bb700 1 -- 172.21.2.102:0/2570241392 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28540893c0 con 0x7f285408ce50 2021-02-10T11:22:31.074 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:31.078 7f283bfff700 1 -- 172.21.2.102:0/2570241392 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f284c004930 con 0x7f285408ce50 2021-02-10T11:22:31.206 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:31.210 7f285a0bb700 1 -- 172.21.2.102:0/2570241392 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f28540893c0 con 0x7f285408ce50 2021-02-10T11:22:31.209 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:31.214 7f283bfff700 1 -- 172.21.2.102:0/2570241392 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.2 v17) v1 ==== 184+0+0 (secure 0 0 0) 0x7f284c004b40 con 0x7f285408ce50 2021-02-10T11:22:31.214 INFO:teuthology.orchestra.run.gibba002.stderr:updated caps for client.2 2021-02-10T11:22:31.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:31.219 7f285a0bb700 1 -- 172.21.2.102:0/2570241392 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2834018730 msgr2=0x7f283401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:31.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:31.219 7f285a0bb700 1 --2- 172.21.2.102:0/2570241392 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2834018730 0x7f283401ac10 crc :-1 s=READY pgs=520 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:31.216 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:31.219 7f285a0bb700 1 -- 172.21.2.102:0/2570241392 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f285408ce50 msgr2=0x7f2854093020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:31.216 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:31.219 7f285a0bb700 1 --2- 172.21.2.102:0/2570241392 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f285408ce50 0x7f2854093020 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 rx=0x7f284c007fc0 tx=0x7f284c00ac90).stop 2021-02-10T11:22:31.216 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:31.220 7f285a0bb700 1 -- 172.21.2.102:0/2570241392 shutdown_connections 2021-02-10T11:22:31.216 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:31.220 7f285a0bb700 1 --2- 172.21.2.102:0/2570241392 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f285408c440 0x7f2854087ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:31.216 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:31.220 7f285a0bb700 1 --2- 172.21.2.102:0/2570241392 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2834018730 0x7f283401ac10 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:31.216 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:31.220 7f285a0bb700 1 --2- 172.21.2.102:0/2570241392 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f285408d250 0x7f2854093640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:31.216 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:31.220 7f285a0bb700 1 --2- 172.21.2.102:0/2570241392 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f285408ce50 0x7f2854093020 unknown :-1 s=CLOSED pgs=1079 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:31.216 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:31.220 7f285a0bb700 1 -- 172.21.2.102:0/2570241392 >> 172.21.2.102:0/2570241392 conn(0x7f285408dde0 msgr2=0x7f28541ac5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:31.217 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:31.220 7f285a0bb700 1 -- 172.21.2.102:0/2570241392 shutdown_connections 2021-02-10T11:22:31.217 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:31.220 7f285a0bb700 1 -- 172.21.2.102:0/2570241392 wait complete. 2021-02-10T11:22:31.224 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.3 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:22:31.654 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:31.641 7fcd24460700 1 Processor -- start 2021-02-10T11:22:31.654 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:31.641 7fcd24460700 1 -- start start 2021-02-10T11:22:31.654 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:31.641 7fcd24460700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fcd1c1b0870 0x7fcd1c0f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:31.654 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:31.641 7fcd24460700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd1c1b0c70 con 0x7fcd1c0efe00 2021-02-10T11:22:31.654 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:31.641 7fcd24460700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd1c0f0b50 con 0x7fcd1c1b0470 2021-02-10T11:22:31.655 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:31.641 7fcd24460700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd1c0e5ea0 con 0x7fcd1c1b0870 2021-02-10T11:22:31.655 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:31.642 7fcd22a7d700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fcd1c0efe00 0x7fcd1c0f0200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:47762/0 (socket says 172.21.2.102:47762) 2021-02-10T11:22:31.655 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:31.642 7fcd22a7d700 1 -- 172.21.2.102:0/918897903 learned_addr learned my addr 172.21.2.102:0/918897903 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:22:31.655 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:31.642 7fcd21a7b700 1 --2- 172.21.2.102:0/918897903 >> v2:172.21.2.113:3300/0 conn(0x7fcd1c1b0870 0x7fcd1c0f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:31.655 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:31.642 7fcd2127a700 1 -- 172.21.2.102:0/918897903 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4247098510 0 0) 0x7fcd14001500 con 0x7fcd1c0efe00 2021-02-10T11:22:31.655 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:31.642 7fcd2127a700 1 -- 172.21.2.102:0/918897903 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd08002fe0 con 0x7fcd1c0efe00 2021-02-10T11:22:31.655 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:31.642 7fcd2127a700 1 -- 172.21.2.102:0/918897903 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 60795882 0 0) 0x7fcd14001500 con 0x7fcd1c0efe00 2021-02-10T11:22:31.655 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:31.642 7fcd21a7b700 1 -- 172.21.2.102:0/918897903 >> v1:172.21.2.113:6789/0 conn(0x7fcd1c1b0470 legacy=0x7fcd1c0ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:31.655 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:31.642 7fcd21a7b700 1 -- 172.21.2.102:0/918897903 >> v1:172.21.2.102:6789/0 conn(0x7fcd1c0efe00 legacy=0x7fcd1c0f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:31.656 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:31.643 7fcd21a7b700 1 -- 172.21.2.102:0/918897903 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd1c0e52e0 con 0x7fcd1c1b0870 2021-02-10T11:22:31.656 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:31.643 7fcd21a7b700 1 --2- 172.21.2.102:0/918897903 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fcd1c1b0870 0x7fcd1c0f20e0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 rx=0x7fcd18006650 tx=0x7fcd18009a80).ready entity=mon.1 client_cookie=6d3eb4fd2abf806f server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:31.656 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:31.643 7fcd2127a700 1 -- 172.21.2.102:0/918897903 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcd1800c0c0 con 0x7fcd1c1b0870 2021-02-10T11:22:31.656 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:31.643 7fcd2127a700 1 -- 172.21.2.102:0/918897903 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd1800c340 con 0x7fcd1c1b0870 2021-02-10T11:22:31.656 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:31.643 7fcd2127a700 1 -- 172.21.2.102:0/918897903 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcd1800c860 con 0x7fcd1c1b0870 2021-02-10T11:22:31.656 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:31.643 7fcd24460700 1 -- 172.21.2.102:0/918897903 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fcd1c1b0870 msgr2=0x7fcd1c0f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:31.656 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:31.643 7fcd24460700 1 --2- 172.21.2.102:0/918897903 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fcd1c1b0870 0x7fcd1c0f20e0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 rx=0x7fcd18006650 tx=0x7fcd18009a80).stop 2021-02-10T11:22:31.656 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:31.644 7fcd24460700 1 -- 172.21.2.102:0/918897903 shutdown_connections 2021-02-10T11:22:31.656 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:31.644 7fcd24460700 1 --2- 172.21.2.102:0/918897903 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fcd1c1b0870 0x7fcd1c0f20e0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:31.656 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:31.644 7fcd24460700 1 -- 172.21.2.102:0/918897903 >> 172.21.2.102:0/918897903 conn(0x7fcd1c0ce3e0 msgr2=0x7fcd1c0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:31.656 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:31.644 7fcd24460700 1 -- 172.21.2.102:0/918897903 shutdown_connections 2021-02-10T11:22:31.656 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:31.644 7fcd24460700 1 -- 172.21.2.102:0/918897903 wait complete. 2021-02-10T11:22:31.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:31.659 7fcd24460700 1 Processor -- start 2021-02-10T11:22:31.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:31.659 7fcd24460700 1 -- start start 2021-02-10T11:22:31.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:31.659 7fcd24460700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcd1c1ac0b0 0x7fcd1c0ad920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:31.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:31.659 7fcd24460700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcd1c0adf40 0x7fcd1c1b7040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:31.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:31.659 7fcd24460700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fcd1c1b7660 0x7fcd1c1b7a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:31.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:31.659 7fcd24460700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd1c1b8080 con 0x7fcd1c1ac0b0 2021-02-10T11:22:31.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:31.659 7fcd24460700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd1c1b8270 con 0x7fcd1c1b7660 2021-02-10T11:22:31.659 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:31.659 7fcd24460700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd1c18d950 con 0x7fcd1c0adf40 2021-02-10T11:22:31.659 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:31.659 7fcd2227c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcd1c1ac0b0 0x7fcd1c0ad920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:31.659 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:31.659 7fcd2227c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcd1c1ac0b0 0x7fcd1c0ad920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:55540/0 (socket says 172.21.2.102:55540) 2021-02-10T11:22:31.659 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:31.659 7fcd2227c700 1 -- 172.21.2.102:0/966883698 learned_addr learned my addr 172.21.2.102:0/966883698 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:22:31.659 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:31.660 7fcd2227c700 1 -- 172.21.2.102:0/966883698 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcd1c0adf40 msgr2=0x7fcd1c1b7040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:31.659 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:31.660 7fcd2227c700 1 --2- 172.21.2.102:0/966883698 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcd1c0adf40 0x7fcd1c1b7040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:31.659 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:31.660 7fcd2227c700 1 -- 172.21.2.102:0/966883698 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fcd1c1b7660 msgr2=0x7fcd1c1b7a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:31.659 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:31.660 7fcd2227c700 1 --2- 172.21.2.102:0/966883698 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fcd1c1b7660 0x7fcd1c1b7a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:31.659 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:31.660 7fcd2227c700 1 -- 172.21.2.102:0/966883698 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd1c18db70 con 0x7fcd1c1ac0b0 2021-02-10T11:22:31.659 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:31.660 7fcd2227c700 1 --2- 172.21.2.102:0/966883698 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcd1c1ac0b0 0x7fcd1c0ad920 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 rx=0x7fcd1000b530 tx=0x7fcd1000b8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:31.659 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:31.660 7fcd0effd700 1 -- 172.21.2.102:0/966883698 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcd1000bcc0 con 0x7fcd1c1ac0b0 2021-02-10T11:22:31.659 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:31.660 7fcd0effd700 1 -- 172.21.2.102:0/966883698 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd10006020 con 0x7fcd1c1ac0b0 2021-02-10T11:22:31.659 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:31.660 7fcd0effd700 1 -- 172.21.2.102:0/966883698 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcd100054a0 con 0x7fcd1c1ac0b0 2021-02-10T11:22:31.660 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:31.660 7fcd24460700 1 -- 172.21.2.102:0/966883698 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd1c18db70 con 0x7fcd1c1ac0b0 2021-02-10T11:22:31.660 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:31.660 7fcd24460700 1 -- 172.21.2.102:0/966883698 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd1c18e100 con 0x7fcd1c1ac0b0 2021-02-10T11:22:31.660 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:31.661 7fcd0effd700 1 -- 172.21.2.102:0/966883698 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fcd10006200 con 0x7fcd1c1ac0b0 2021-02-10T11:22:31.660 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:31.661 7fcd0effd700 1 --2- 172.21.2.102:0/966883698 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcd040186c0 0x7fcd0401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:31.660 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:31.661 7fcd0effd700 1 -- 172.21.2.102:0/966883698 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fcd100325c0 con 0x7fcd1c1ac0b0 2021-02-10T11:22:31.660 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:31.661 7fcd21a7b700 1 --2- 172.21.2.102:0/966883698 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcd040186c0 0x7fcd0401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:31.660 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:31.661 7fcd21a7b700 1 --2- 172.21.2.102:0/966883698 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcd040186c0 0x7fcd0401abc0 crc :-1 s=READY pgs=521 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:31.687 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:31.691 7fcd24460700 1 -- 172.21.2.102:0/966883698 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd1c1059e0 con 0x7fcd1c1ac0b0 2021-02-10T11:22:31.689 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:31.694 7fcd0effd700 1 -- 172.21.2.102:0/966883698 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fcd10000940 con 0x7fcd1c1ac0b0 2021-02-10T11:22:31.819 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:31.823 7fcd24460700 1 -- 172.21.2.102:0/966883698 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.3", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7fcd1c0f0b50 con 0x7fcd1c1ac0b0 2021-02-10T11:22:31.823 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:31.827 7fcd0effd700 1 -- 172.21.2.102:0/966883698 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.3", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.3 v18) v1 ==== 184+0+0 (secure 0 0 0) 0x7fcd10006200 con 0x7fcd1c1ac0b0 2021-02-10T11:22:31.826 INFO:teuthology.orchestra.run.gibba002.stderr:updated caps for client.3 2021-02-10T11:22:31.828 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:31.832 7fcd24460700 1 -- 172.21.2.102:0/966883698 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcd040186c0 msgr2=0x7fcd0401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:31.828 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:31.832 7fcd24460700 1 --2- 172.21.2.102:0/966883698 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcd040186c0 0x7fcd0401abc0 crc :-1 s=READY pgs=521 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:31.828 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:31.832 7fcd24460700 1 -- 172.21.2.102:0/966883698 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcd1c1ac0b0 msgr2=0x7fcd1c0ad920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:31.828 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:31.832 7fcd24460700 1 --2- 172.21.2.102:0/966883698 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcd1c1ac0b0 0x7fcd1c0ad920 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 rx=0x7fcd1000b530 tx=0x7fcd1000b8d0).stop 2021-02-10T11:22:31.828 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:31.832 7fcd24460700 1 -- 172.21.2.102:0/966883698 shutdown_connections 2021-02-10T11:22:31.828 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:31.832 7fcd24460700 1 --2- 172.21.2.102:0/966883698 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcd040186c0 0x7fcd0401abc0 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:31.828 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:31.832 7fcd24460700 1 --2- 172.21.2.102:0/966883698 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fcd1c1b7660 0x7fcd1c1b7a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:31.829 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:31.832 7fcd24460700 1 --2- 172.21.2.102:0/966883698 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcd1c0adf40 0x7fcd1c1b7040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:31.829 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:31.832 7fcd24460700 1 --2- 172.21.2.102:0/966883698 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcd1c1ac0b0 0x7fcd1c0ad920 unknown :-1 s=CLOSED pgs=1081 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:31.829 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:31.832 7fcd24460700 1 -- 172.21.2.102:0/966883698 >> 172.21.2.102:0/966883698 conn(0x7fcd1c1a37a0 msgr2=0x7fcd1c18c230 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:31.829 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:31.832 7fcd24460700 1 -- 172.21.2.102:0/966883698 shutdown_connections 2021-02-10T11:22:31.829 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:31.832 7fcd24460700 1 -- 172.21.2.102:0/966883698 wait complete. 2021-02-10T11:22:31.836 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:22:32.263 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:32.249 7f1a060d6700 1 Processor -- start 2021-02-10T11:22:32.264 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:32.249 7f1a060d6700 1 -- start start 2021-02-10T11:22:32.264 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:32.250 7f1a060d6700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1a000e5470 0x7f1a0007e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:32.264 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:32.250 7f1a060d6700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a0007ed80 con 0x7f1a0007f300 2021-02-10T11:22:32.264 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:32.250 7f1a060d6700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a001b4010 con 0x7f1a000e5070 2021-02-10T11:22:32.264 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:32.250 7f1a060d6700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a000d1790 con 0x7f1a000e5470 2021-02-10T11:22:32.264 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:32.250 7f19fffff700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f1a0007f300 0x7f1a0007f700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:47776/0 (socket says 172.21.2.102:47776) 2021-02-10T11:22:32.264 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:32.250 7f19fffff700 1 -- 172.21.2.102:0/3026379892 learned_addr learned my addr 172.21.2.102:0/3026379892 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:22:32.264 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:32.251 7f19fe7fc700 1 -- 172.21.2.102:0/3026379892 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3031854280 0 0) 0x7f19f8001500 con 0x7f1a0007f300 2021-02-10T11:22:32.264 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:32.251 7f19fe7fc700 1 -- 172.21.2.102:0/3026379892 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19ec002fe0 con 0x7f1a0007f300 2021-02-10T11:22:32.264 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:32.251 7f19feffd700 1 --2- 172.21.2.102:0/3026379892 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1a000e5470 0x7f1a0007e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:32.264 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:32.251 7f19fe7fc700 1 -- 172.21.2.102:0/3026379892 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1703814904 0 0) 0x7f19f8001500 con 0x7f1a0007f300 2021-02-10T11:22:32.264 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:32.251 7f19fe7fc700 1 -- 172.21.2.102:0/3026379892 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1a000e5470 msgr2=0x7f1a0007e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:32.265 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:32.251 7f19fe7fc700 1 --2- 172.21.2.102:0/3026379892 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1a000e5470 0x7f1a0007e6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:32.265 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:32.251 7f19fe7fc700 1 -- 172.21.2.102:0/3026379892 >> v1:172.21.2.113:6789/0 conn(0x7f1a000e5070 legacy=0x7f1a001b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:32.265 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:32.251 7f19fe7fc700 1 -- 172.21.2.102:0/3026379892 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a000d1980 con 0x7f1a0007f300 2021-02-10T11:22:32.265 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:32.251 7f19fe7fc700 1 -- 172.21.2.102:0/3026379892 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7f19f8001a00 con 0x7f1a0007f300 2021-02-10T11:22:32.265 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:32.251 7f19fe7fc700 1 -- 172.21.2.102:0/3026379892 >> v1:172.21.2.102:6789/0 conn(0x7f1a0007f300 legacy=0x7f1a0007f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:32.265 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:32.251 7f19fe7fc700 1 --2- 172.21.2.102:0/3026379892 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f19ec003a60 0x7f19ec005f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:32.265 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:32.251 7f19fe7fc700 1 --2- 172.21.2.102:0/3026379892 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f19ec006780 0x7f19ec008c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:32.265 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:32.251 7f19fe7fc700 1 --2- 172.21.2.102:0/3026379892 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f19ec009580 0x7f19ec00ba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:32.265 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:32.251 7f19fe7fc700 1 -- 172.21.2.102:0/3026379892 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19ec00c2c0 con 0x7f19ec006780 2021-02-10T11:22:32.265 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:32.251 7f19fe7fc700 1 -- 172.21.2.102:0/3026379892 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19ec00c4e0 con 0x7f19ec003a60 2021-02-10T11:22:32.265 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:32.251 7f19fe7fc700 1 -- 172.21.2.102:0/3026379892 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19ec050710 con 0x7f19ec009580 2021-02-10T11:22:32.265 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:32.251 7f19feffd700 1 --2- 172.21.2.102:0/3026379892 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f19ec006780 0x7f19ec008c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:32.265 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:32.252 7f19ff7fe700 1 --2- 172.21.2.102:0/3026379892 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f19ec003a60 0x7f19ec005f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:32.266 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:32.252 7f19fffff700 1 --2- 172.21.2.102:0/3026379892 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f19ec009580 0x7f19ec00ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:32.266 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:32.252 7f19feffd700 1 -- 172.21.2.102:0/3026379892 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f19ec009580 msgr2=0x7f19ec00ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:32.266 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:32.252 7f19feffd700 1 --2- 172.21.2.102:0/3026379892 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f19ec009580 0x7f19ec00ba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:32.266 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:32.252 7f19feffd700 1 -- 172.21.2.102:0/3026379892 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f19ec003a60 msgr2=0x7f19ec005f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:32.266 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:32.252 7f19feffd700 1 --2- 172.21.2.102:0/3026379892 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f19ec003a60 0x7f19ec005f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:32.266 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:32.252 7f19feffd700 1 -- 172.21.2.102:0/3026379892 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19ec050970 con 0x7f19ec006780 2021-02-10T11:22:32.266 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:32.252 7f19ff7fe700 1 --2- 172.21.2.102:0/3026379892 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f19ec003a60 0x7f19ec005f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:22:32.266 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:32.252 7f19fffff700 1 --2- 172.21.2.102:0/3026379892 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f19ec009580 0x7f19ec00ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:22:32.266 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:32.252 7f19feffd700 1 --2- 172.21.2.102:0/3026379892 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f19ec006780 0x7f19ec008c80 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 rx=0x7f19f0009310 tx=0x7f19f0009a90).ready entity=mon.0 client_cookie=5eafed90656d04fe server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:32.266 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:32.252 7f19fe7fc700 1 -- 172.21.2.102:0/3026379892 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f19f000c0c0 con 0x7f19ec006780 2021-02-10T11:22:32.266 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:32.252 7f19fe7fc700 1 -- 172.21.2.102:0/3026379892 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19f0006400 con 0x7f19ec006780 2021-02-10T11:22:32.266 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:32.252 7f19fe7fc700 1 -- 172.21.2.102:0/3026379892 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f19f000c400 con 0x7f19ec006780 2021-02-10T11:22:32.266 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:32.252 7f1a060d6700 1 -- 172.21.2.102:0/3026379892 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f19ec006780 msgr2=0x7f19ec008c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:32.267 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:32.252 7f1a060d6700 1 --2- 172.21.2.102:0/3026379892 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f19ec006780 0x7f19ec008c80 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 rx=0x7f19f0009310 tx=0x7f19f0009a90).stop 2021-02-10T11:22:32.267 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:32.252 7f19fffff700 1 -- 172.21.2.102:0/3026379892 reap_dead start 2021-02-10T11:22:32.267 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:32.253 7f1a060d6700 1 -- 172.21.2.102:0/3026379892 shutdown_connections 2021-02-10T11:22:32.267 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:32.253 7f1a060d6700 1 -- 172.21.2.102:0/3026379892 >> 172.21.2.102:0/3026379892 conn(0x7f1a000ce690 msgr2=0x7f1a000e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:32.267 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:32.253 7f1a060d6700 1 -- 172.21.2.102:0/3026379892 shutdown_connections 2021-02-10T11:22:32.267 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:32.253 7f1a060d6700 1 -- 172.21.2.102:0/3026379892 wait complete. 2021-02-10T11:22:32.267 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:32.268 7f1a060d6700 1 Processor -- start 2021-02-10T11:22:32.267 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:32.268 7f1a060d6700 1 -- start start 2021-02-10T11:22:32.268 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:32.268 7f1a060d6700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1a0018d210 0x7f1a001ab320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:32.268 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:32.268 7f1a060d6700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1a000984c0 0x7f1a00097290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:32.269 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:32.268 7f1a060d6700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1a000978b0 0x7f1a00097cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:32.269 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:32.268 7f1a060d6700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a0009a540 con 0x7f1a000978b0 2021-02-10T11:22:32.269 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:32.268 7f1a060d6700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a001aba60 con 0x7f1a000984c0 2021-02-10T11:22:32.269 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:32.268 7f1a060d6700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a0009aac0 con 0x7f1a0018d210 2021-02-10T11:22:32.269 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:32.268 7f19fffff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1a000978b0 0x7f1a00097cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:32.269 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:32.268 7f19fffff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1a000978b0 0x7f1a00097cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:55564/0 (socket says 172.21.2.102:55564) 2021-02-10T11:22:32.269 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:32.268 7f19fffff700 1 -- 172.21.2.102:0/96958953 learned_addr learned my addr 172.21.2.102:0/96958953 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:22:32.269 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:32.269 7f19fffff700 1 -- 172.21.2.102:0/96958953 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1a0018d210 msgr2=0x7f1a001ab320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:32.269 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:32.269 7f19fffff700 1 --2- 172.21.2.102:0/96958953 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1a0018d210 0x7f1a001ab320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:32.269 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:32.269 7f19fffff700 1 -- 172.21.2.102:0/96958953 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1a000984c0 msgr2=0x7f1a00097290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:32.269 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:32.269 7f19fffff700 1 --2- 172.21.2.102:0/96958953 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1a000984c0 0x7f1a00097290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:32.269 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:32.269 7f19fffff700 1 -- 172.21.2.102:0/96958953 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a0009ace0 con 0x7f1a000978b0 2021-02-10T11:22:32.269 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:32.269 7f19fffff700 1 --2- 172.21.2.102:0/96958953 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1a000978b0 0x7f1a00097cb0 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 rx=0x7f19f80077a0 tx=0x7f19f800bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:32.270 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:32.269 7f19f7fff700 1 -- 172.21.2.102:0/96958953 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f19f8006030 con 0x7f1a000978b0 2021-02-10T11:22:32.270 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:32.269 7f19f7fff700 1 -- 172.21.2.102:0/96958953 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19f80062f0 con 0x7f1a000978b0 2021-02-10T11:22:32.270 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:32.269 7f19f7fff700 1 -- 172.21.2.102:0/96958953 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f19f8007ab0 con 0x7f1a000978b0 2021-02-10T11:22:32.270 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:32.269 7f1a060d6700 1 -- 172.21.2.102:0/96958953 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a0009ace0 con 0x7f1a000978b0 2021-02-10T11:22:32.270 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:32.269 7f1a060d6700 1 -- 172.21.2.102:0/96958953 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a0009ace0 con 0x7f1a000978b0 2021-02-10T11:22:32.270 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:32.270 7f19f7fff700 1 -- 172.21.2.102:0/96958953 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f19f8004720 con 0x7f1a000978b0 2021-02-10T11:22:32.270 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:32.270 7f19f7fff700 1 --2- 172.21.2.102:0/96958953 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f19e40186c0 0x7f19e401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:32.270 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:32.270 7f19f7fff700 1 -- 172.21.2.102:0/96958953 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f19f80328e0 con 0x7f1a000978b0 2021-02-10T11:22:32.270 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:32.270 7f19ff7fe700 1 --2- 172.21.2.102:0/96958953 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f19e40186c0 0x7f19e401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:32.270 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:32.271 7f19ff7fe700 1 --2- 172.21.2.102:0/96958953 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f19e40186c0 0x7f19e401abc0 crc :-1 s=READY pgs=522 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:32.296 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:32.300 7f1a060d6700 1 -- 172.21.2.102:0/96958953 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a00084b20 con 0x7f1a000978b0 2021-02-10T11:22:32.298 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:32.302 7f19f7fff700 1 -- 172.21.2.102:0/96958953 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f19f800be30 con 0x7f1a000978b0 2021-02-10T11:22:32.343 DEBUG:teuthology.orchestra.run.gibba002:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:22:32.345 DEBUG:teuthology.orchestra.run.gibba013:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:22:32.429 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:32.433 7f1a060d6700 1 -- 172.21.2.102:0/96958953 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1a00084b20 con 0x7f1a000978b0 2021-02-10T11:22:32.430 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:32.434 7f19f7fff700 1 -- 172.21.2.102:0/96958953 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) v1 ==== 94+0+2099 (secure 0 0 0) 0x7f19f80063c0 con 0x7f1a000978b0 2021-02-10T11:22:32.430 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:22:32.430 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":21,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7194,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.102:6835/2295196020","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6834","nonce":2295196020},{"type":"v1","addr":"172.21.2.102:6835","nonce":2295196020}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":18}],"filesystems":[{"mdsmap":{"epoch":21,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:22:26.122430","modified":"2021-02-10 11:22:28.126179","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7191},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7191":{"gid":7191,"name":"b","rank":0,"incarnation":19,"state":"up:active","state_seq":3,"addr":"172.21.2.113:6833/3694995761","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6832","nonce":3694995761},{"type":"v1","addr":"172.21.2.113:6833","nonce":3694995761}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":3}]} 2021-02-10T11:22:32.431 INFO:teuthology.orchestra.run.gibba002.stderr:dumped fsmap epoch 21 2021-02-10T11:22:32.432 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:32.436 7f1a060d6700 1 -- 172.21.2.102:0/96958953 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f19e40186c0 msgr2=0x7f19e401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:32.432 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:32.436 7f1a060d6700 1 --2- 172.21.2.102:0/96958953 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f19e40186c0 0x7f19e401abc0 crc :-1 s=READY pgs=522 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:32.432 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:32.436 7f1a060d6700 1 -- 172.21.2.102:0/96958953 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1a000978b0 msgr2=0x7f1a00097cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:32.432 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:32.436 7f1a060d6700 1 --2- 172.21.2.102:0/96958953 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1a000978b0 0x7f1a00097cb0 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 rx=0x7f19f80077a0 tx=0x7f19f800bcf0).stop 2021-02-10T11:22:32.432 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:32.436 7f1a060d6700 1 -- 172.21.2.102:0/96958953 shutdown_connections 2021-02-10T11:22:32.432 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:32.436 7f1a060d6700 1 --2- 172.21.2.102:0/96958953 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1a000978b0 0x7f1a00097cb0 unknown :-1 s=CLOSED pgs=1084 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:32.432 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:32.436 7f1a060d6700 1 --2- 172.21.2.102:0/96958953 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f19e40186c0 0x7f19e401abc0 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:32.432 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:32.436 7f1a060d6700 1 --2- 172.21.2.102:0/96958953 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1a000984c0 0x7f1a00097290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:32.432 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:32.436 7f1a060d6700 1 --2- 172.21.2.102:0/96958953 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1a0018d210 0x7f1a001ab320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:32.432 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:32.436 7f1a060d6700 1 -- 172.21.2.102:0/96958953 >> 172.21.2.102:0/96958953 conn(0x7f1a001b2a40 msgr2=0x7f1a00198bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:32.433 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:32.436 7f1a060d6700 1 -- 172.21.2.102:0/96958953 shutdown_connections 2021-02-10T11:22:32.433 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:32.436 7f1a060d6700 1 -- 172.21.2.102:0/96958953 wait complete. 2021-02-10T11:22:32.643 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 21, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:22:26.122430', 'modified': '2021-02-10 11:22:28.126179', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7191}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7191': {'gid': 7191, 'name': 'b', 'rank': 0, 'incarnation': 19, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.113:6833/3694995761', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.113:6832', 'nonce': 3694995761}, {'type': 'v1', 'addr': '172.21.2.113:6833', 'nonce': 3694995761}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [7], 'metadata_pool': 6, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:22:32.643 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:22:32.643 DEBUG:teuthology.orchestra.run.gibba013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:22:33.041 INFO:teuthology.orchestra.run.gibba013.stdout:{ 2021-02-10T11:22:33.041 INFO:teuthology.orchestra.run.gibba013.stdout: "cluster_fsid": "68db30ea-dbce-4aef-a6df-f01c6fea0513", 2021-02-10T11:22:33.041 INFO:teuthology.orchestra.run.gibba013.stdout: "whoami": 0, 2021-02-10T11:22:33.042 INFO:teuthology.orchestra.run.gibba013.stdout: "id": 7191, 2021-02-10T11:22:33.042 INFO:teuthology.orchestra.run.gibba013.stdout: "want_state": "up:active", 2021-02-10T11:22:33.042 INFO:teuthology.orchestra.run.gibba013.stdout: "state": "up:active", 2021-02-10T11:22:33.042 INFO:teuthology.orchestra.run.gibba013.stdout: "rank_uptime": 6.921608516, 2021-02-10T11:22:33.042 INFO:teuthology.orchestra.run.gibba013.stdout: "mdsmap_epoch": 21, 2021-02-10T11:22:33.042 INFO:teuthology.orchestra.run.gibba013.stdout: "osdmap_epoch": 39, 2021-02-10T11:22:33.042 INFO:teuthology.orchestra.run.gibba013.stdout: "osdmap_epoch_barrier": 39, 2021-02-10T11:22:33.042 INFO:teuthology.orchestra.run.gibba013.stdout: "uptime": 13.580912373 2021-02-10T11:22:33.042 INFO:teuthology.orchestra.run.gibba013.stdout:} 2021-02-10T11:22:33.045 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "68db30ea-dbce-4aef-a6df-f01c6fea0513", "whoami": 0, "id": 7191, "want_state": "up:active", "state": "up:active", "rank_uptime": 6.921608516, "mdsmap_epoch": 21, "osdmap_epoch": 39, "osdmap_epoch_barrier": 39, "uptime": 13.580912373 } 2021-02-10T11:22:33.046 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:22:33.472 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:33.460 7fb66a937700 1 Processor -- start 2021-02-10T11:22:33.472 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:33.460 7fb66a937700 1 -- start start 2021-02-10T11:22:33.473 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:33.461 7fb66a937700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fb6640e5470 0x7fb66407e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:33.473 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:33.461 7fb66a937700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb66407ed80 con 0x7fb6640e5070 2021-02-10T11:22:33.473 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:33.461 7fb66a937700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6641b4010 con 0x7fb66407f300 2021-02-10T11:22:33.473 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:33.461 7fb66a937700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6640d1790 con 0x7fb6640e5470 2021-02-10T11:22:33.473 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:33.461 7fb6637fe700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fb6640e5470 0x7fb66407e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:33.473 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:33.461 7fb663fff700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fb6640e5070 0x7fb6641b3820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:47792/0 (socket says 172.21.2.102:47792) 2021-02-10T11:22:33.473 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:33.461 7fb663fff700 1 -- 172.21.2.102:0/2806203612 learned_addr learned my addr 172.21.2.102:0/2806203612 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:22:33.473 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:33.461 7fb6637fe700 1 -- 172.21.2.102:0/2806203612 >> v1:172.21.2.113:6789/0 conn(0x7fb66407f300 legacy=0x7fb66407f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:33.473 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:33.461 7fb6637fe700 1 -- 172.21.2.102:0/2806203612 >> v1:172.21.2.102:6789/0 conn(0x7fb6640e5070 legacy=0x7fb6641b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:33.474 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:33.462 7fb6637fe700 1 -- 172.21.2.102:0/2806203612 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6640d19b0 con 0x7fb6640e5470 2021-02-10T11:22:33.474 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:33.461 7fb662ffd700 1 -- 172.21.2.102:0/2806203612 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3816673369 0 0) 0x7fb654001180 con 0x7fb6640e5070 2021-02-10T11:22:33.474 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:33.462 7fb6637fe700 1 --2- 172.21.2.102:0/2806203612 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb6640e5470 0x7fb66407e6a0 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 rx=0x7fb658006650 tx=0x7fb658009a00).ready entity=mon.0 client_cookie=13352df443e3bb88 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:33.474 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:33.462 7fb662ffd700 1 -- 172.21.2.102:0/2806203612 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb6580009b0 con 0x7fb6640e5470 2021-02-10T11:22:33.474 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:33.462 7fb662ffd700 1 -- 172.21.2.102:0/2806203612 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb65800c0c0 con 0x7fb6640e5470 2021-02-10T11:22:33.474 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:33.462 7fb662ffd700 1 -- 172.21.2.102:0/2806203612 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb65800c5a0 con 0x7fb6640e5470 2021-02-10T11:22:33.474 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:33.462 7fb66a937700 1 -- 172.21.2.102:0/2806203612 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb6640e5470 msgr2=0x7fb66407e6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:33.474 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:33.462 7fb66a937700 1 --2- 172.21.2.102:0/2806203612 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb6640e5470 0x7fb66407e6a0 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 rx=0x7fb658006650 tx=0x7fb658009a00).stop 2021-02-10T11:22:33.474 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:33.462 7fb66a937700 1 -- 172.21.2.102:0/2806203612 shutdown_connections 2021-02-10T11:22:33.474 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:33.462 7fb66a937700 1 --2- 172.21.2.102:0/2806203612 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb6640e5470 0x7fb66407e6a0 unknown :-1 s=CLOSED pgs=1086 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:33.474 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:33.462 7fb66a937700 1 -- 172.21.2.102:0/2806203612 >> 172.21.2.102:0/2806203612 conn(0x7fb6640ce690 msgr2=0x7fb6640e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:33.474 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:33.462 7fb66a937700 1 -- 172.21.2.102:0/2806203612 shutdown_connections 2021-02-10T11:22:33.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:33.462 7fb66a937700 1 -- 172.21.2.102:0/2806203612 wait complete. 2021-02-10T11:22:33.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:33.477 7fb66a937700 1 Processor -- start 2021-02-10T11:22:33.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:33.477 7fb66a937700 1 -- start start 2021-02-10T11:22:33.477 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:33.477 7fb66a937700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb66408c830 0x7fb66408d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:33.477 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:33.477 7fb66a937700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb6641afe20 0x7fb6640b2f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:33.477 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:33.477 7fb66a937700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb6641b0220 0x7fb6640b3550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:33.477 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:33.477 7fb66a937700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb66408cc30 con 0x7fb6641afe20 2021-02-10T11:22:33.477 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:33.477 7fb66a937700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6640b3b70 con 0x7fb66408c830 2021-02-10T11:22:33.477 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:33.477 7fb66a937700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6640b3d90 con 0x7fb6641b0220 2021-02-10T11:22:33.477 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:33.477 7fb6637fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb6641afe20 0x7fb6640b2f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:33.477 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:33.477 7fb6637fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb6641afe20 0x7fb6640b2f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:55578/0 (socket says 172.21.2.102:55578) 2021-02-10T11:22:33.478 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:33.477 7fb6637fe700 1 -- 172.21.2.102:0/1839307028 learned_addr learned my addr 172.21.2.102:0/1839307028 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:22:33.478 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:33.478 7fb6637fe700 1 -- 172.21.2.102:0/1839307028 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb6641b0220 msgr2=0x7fb6640b3550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:33.478 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:33.478 7fb6637fe700 1 --2- 172.21.2.102:0/1839307028 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb6641b0220 0x7fb6640b3550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:33.478 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:33.478 7fb6637fe700 1 -- 172.21.2.102:0/1839307028 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb66408c830 msgr2=0x7fb66408d0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:33.478 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:33.478 7fb6637fe700 1 --2- 172.21.2.102:0/1839307028 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb66408c830 0x7fb66408d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:33.478 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:33.478 7fb6637fe700 1 -- 172.21.2.102:0/1839307028 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6641b0c00 con 0x7fb6641afe20 2021-02-10T11:22:33.478 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:33.478 7fb6637fe700 1 --2- 172.21.2.102:0/1839307028 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb6641afe20 0x7fb6640b2f30 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 rx=0x7fb65800b640 tx=0x7fb65800b9e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:33.478 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:33.478 7fb660ff9700 1 -- 172.21.2.102:0/1839307028 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb658006ae0 con 0x7fb6641afe20 2021-02-10T11:22:33.478 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:33.478 7fb660ff9700 1 -- 172.21.2.102:0/1839307028 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb65800bdd0 con 0x7fb6641afe20 2021-02-10T11:22:33.478 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:33.478 7fb660ff9700 1 -- 172.21.2.102:0/1839307028 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb6580022b0 con 0x7fb6641afe20 2021-02-10T11:22:33.478 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:33.478 7fb66a937700 1 -- 172.21.2.102:0/1839307028 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6641b0c00 con 0x7fb6641afe20 2021-02-10T11:22:33.478 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:33.478 7fb66a937700 1 -- 172.21.2.102:0/1839307028 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6641b1150 con 0x7fb6641afe20 2021-02-10T11:22:33.479 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:33.479 7fb660ff9700 1 -- 172.21.2.102:0/1839307028 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fb658006a10 con 0x7fb6641afe20 2021-02-10T11:22:33.479 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:33.479 7fb660ff9700 1 --2- 172.21.2.102:0/1839307028 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb6440186c0 0x7fb64401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:33.479 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:33.479 7fb660ff9700 1 -- 172.21.2.102:0/1839307028 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fb6580325a0 con 0x7fb6641afe20 2021-02-10T11:22:33.479 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:33.479 7fb663fff700 1 --2- 172.21.2.102:0/1839307028 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb6440186c0 0x7fb64401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:33.479 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:33.480 7fb663fff700 1 --2- 172.21.2.102:0/1839307028 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb6440186c0 0x7fb64401abc0 crc :-1 s=READY pgs=523 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:33.507 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:33.511 7fb66a937700 1 -- 172.21.2.102:0/1839307028 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb664083db0 con 0x7fb6641afe20 2021-02-10T11:22:33.509 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:33.513 7fb660ff9700 1 -- 172.21.2.102:0/1839307028 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fb6580009b0 con 0x7fb6641afe20 2021-02-10T11:22:33.633 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:33.637 7fb66a937700 1 -- 172.21.2.102:0/1839307028 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb66407ed80 con 0x7fb6641afe20 2021-02-10T11:22:33.633 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:33.637 7fb660ff9700 1 -- 172.21.2.102:0/1839307028 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v39) v1 ==== 74+0+13993 (secure 0 0 0) 0x7fb658006a10 con 0x7fb6641afe20 2021-02-10T11:22:33.634 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:22:33.634 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":39,"fsid":"68db30ea-dbce-4aef-a6df-f01c6fea0513","created":"2021-02-10 11:03:06.331557","modified":"2021-02-10 11:22:27.123992","last_up_change":"2021-02-10 11:03:32.690562","last_in_change":"2021-02-10 11:03:32.690562","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:03:37.322519","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:22:23.578113","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":7,"pool_name":"cephfs_data","create_time":"2021-02-10 11:22:25.111303","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"be691b4d-a039-408a-af80-9d53f02b6f1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6818","nonce":36088},{"type":"v1","addr":"172.21.2.102:6820","nonce":36088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6822","nonce":36088},{"type":"v1","addr":"172.21.2.102:6824","nonce":36088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6830","nonce":36088},{"type":"v1","addr":"172.21.2.102:6832","nonce":36088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6826","nonce":36088},{"type":"v1","addr":"172.21.2.102:6828","nonce":36088}]},"public_addr":"172.21.2.102:6820/36088","cluster_addr":"172.21.2.102:6824/36088","heartbeat_back_addr":"172.21.2.102:6832/36088","heartbeat_front_addr":"172.21.2.102:6828/36088","state":["exists","up"]},{"osd":1,"uuid":"64e3ccc8-9c69-4d7c-865e-af8e83822635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6819","nonce":36087},{"type":"v1","addr":"172.21.2.102:6821","nonce":36087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6823","nonce":36087},{"type":"v1","addr":"172.21.2.102:6825","nonce":36087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6831","nonce":36087},{"type":"v1","addr":"172.21.2.102:6833","nonce":36087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6827","nonce":36087},{"type":"v1","addr":"172.21.2.102:6829","nonce":36087}]},"public_addr":"172.21.2.102:6821/36087","cluster_addr":"172.21.2.102:6825/36087","heartbeat_back_addr":"172.21.2.102:6833/36087","heartbeat_front_addr":"172.21.2.102:6829/36087","state":["exists","up"]},{"osd":2,"uuid":"153700af-747f-481f-bdf4-d6072d3a842f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6810","nonce":36089},{"type":"v1","addr":"172.21.2.102:6811","nonce":36089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6812","nonce":36089},{"type":"v1","addr":"172.21.2.102:6813","nonce":36089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6816","nonce":36089},{"type":"v1","addr":"172.21.2.102:6817","nonce":36089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6814","nonce":36089},{"type":"v1","addr":"172.21.2.102:6815","nonce":36089}]},"public_addr":"172.21.2.102:6811/36089","cluster_addr":"172.21.2.102:6813/36089","heartbeat_back_addr":"172.21.2.102:6817/36089","heartbeat_front_addr":"172.21.2.102:6815/36089","state":["exists","up"]},{"osd":3,"uuid":"cbc678f4-83a9-45e6-8830-2f7c232b1ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6802","nonce":36090},{"type":"v1","addr":"172.21.2.102:6803","nonce":36090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6804","nonce":36090},{"type":"v1","addr":"172.21.2.102:6805","nonce":36090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6808","nonce":36090},{"type":"v1","addr":"172.21.2.102:6809","nonce":36090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6806","nonce":36090},{"type":"v1","addr":"172.21.2.102:6807","nonce":36090}]},"public_addr":"172.21.2.102:6803/36090","cluster_addr":"172.21.2.102:6805/36090","heartbeat_back_addr":"172.21.2.102:6809/36090","heartbeat_front_addr":"172.21.2.102:6807/36090","state":["exists","up"]},{"osd":4,"uuid":"c01e4008-b258-4f10-afd2-9e7f37214351","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6808","nonce":36368},{"type":"v1","addr":"172.21.2.113:6811","nonce":36368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6814","nonce":36368},{"type":"v1","addr":"172.21.2.113:6818","nonce":36368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6826","nonce":36368},{"type":"v1","addr":"172.21.2.113:6828","nonce":36368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6820","nonce":36368},{"type":"v1","addr":"172.21.2.113:6824","nonce":36368}]},"public_addr":"172.21.2.113:6811/36368","cluster_addr":"172.21.2.113:6818/36368","heartbeat_back_addr":"172.21.2.113:6828/36368","heartbeat_front_addr":"172.21.2.113:6824/36368","state":["exists","up"]},{"osd":5,"uuid":"551f6007-daf8-4a98-8ee0-c2b16bcccb5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6802","nonce":36366},{"type":"v1","addr":"172.21.2.113:6805","nonce":36366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6807","nonce":36366},{"type":"v1","addr":"172.21.2.113:6810","nonce":36366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6819","nonce":36366},{"type":"v1","addr":"172.21.2.113:6822","nonce":36366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6812","nonce":36366},{"type":"v1","addr":"172.21.2.113:6815","nonce":36366}]},"public_addr":"172.21.2.113:6805/36366","cluster_addr":"172.21.2.113:6810/36366","heartbeat_back_addr":"172.21.2.113:6822/36366","heartbeat_front_addr":"172.21.2.113:6815/36366","state":["exists","up"]},{"osd":6,"uuid":"bf67c040-0e50-48be-b102-c3943bc9ebef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6800","nonce":36367},{"type":"v1","addr":"172.21.2.113:6801","nonce":36367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6803","nonce":36367},{"type":"v1","addr":"172.21.2.113:6804","nonce":36367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6813","nonce":36367},{"type":"v1","addr":"172.21.2.113:6816","nonce":36367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6806","nonce":36367},{"type":"v1","addr":"172.21.2.113:6809","nonce":36367}]},"public_addr":"172.21.2.113:6801/36367","cluster_addr":"172.21.2.113:6804/36367","heartbeat_back_addr":"172.21.2.113:6816/36367","heartbeat_front_addr":"172.21.2.113:6809/36367","state":["exists","up"]},{"osd":7,"uuid":"f5461f80-018d-428f-be89-2eb2bcbe7bb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6817","nonce":36365},{"type":"v1","addr":"172.21.2.113:6821","nonce":36365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6823","nonce":36365},{"type":"v1","addr":"172.21.2.113:6825","nonce":36365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6830","nonce":36365},{"type":"v1","addr":"172.21.2.113:6831","nonce":36365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6827","nonce":36365},{"type":"v1","addr":"172.21.2.113:6829","nonce":36365}]},"public_addr":"172.21.2.113:6821/36365","cluster_addr":"172.21.2.113:6825/36365","heartbeat_back_addr":"172.21.2.113:6831/36365","heartbeat_front_addr":"172.21.2.113:6829/36365","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:22:33.635 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:33.639 7fb66a937700 1 -- 172.21.2.102:0/1839307028 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb6440186c0 msgr2=0x7fb64401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:33.635 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:33.639 7fb66a937700 1 --2- 172.21.2.102:0/1839307028 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb6440186c0 0x7fb64401abc0 crc :-1 s=READY pgs=523 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:33.635 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:33.639 7fb66a937700 1 -- 172.21.2.102:0/1839307028 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb6641afe20 msgr2=0x7fb6640b2f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:33.635 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:33.639 7fb66a937700 1 --2- 172.21.2.102:0/1839307028 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb6641afe20 0x7fb6640b2f30 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 rx=0x7fb65800b640 tx=0x7fb65800b9e0).stop 2021-02-10T11:22:33.635 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:33.639 7fb66a937700 1 -- 172.21.2.102:0/1839307028 shutdown_connections 2021-02-10T11:22:33.636 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:33.639 7fb66a937700 1 --2- 172.21.2.102:0/1839307028 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb6641b0220 0x7fb6640b3550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:33.636 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:33.639 7fb66a937700 1 --2- 172.21.2.102:0/1839307028 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb6641afe20 0x7fb6640b2f30 unknown :-1 s=CLOSED pgs=1087 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:33.636 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:33.639 7fb66a937700 1 --2- 172.21.2.102:0/1839307028 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb6440186c0 0x7fb64401abc0 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:33.636 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:33.639 7fb66a937700 1 --2- 172.21.2.102:0/1839307028 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb66408c830 0x7fb66408d0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:33.636 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:33.639 7fb66a937700 1 -- 172.21.2.102:0/1839307028 >> 172.21.2.102:0/1839307028 conn(0x7fb6640937a0 msgr2=0x7fb664197e80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:33.636 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:33.639 7fb66a937700 1 -- 172.21.2.102:0/1839307028 shutdown_connections 2021-02-10T11:22:33.636 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:33.639 7fb66a937700 1 -- 172.21.2.102:0/1839307028 wait complete. 2021-02-10T11:22:33.644 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:03:37.322519', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 6, 'pool_name': 'cephfs_metadata', 'create_time': '2021-02-10 11:22:23.578113', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '38', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 7, 'pool_name': 'cephfs_data', 'create_time': '2021-02-10 11:22:25.111303', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '39', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-02-10T11:22:33.644 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:22:34.075 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:34.062 7ff5480b4700 1 Processor -- start 2021-02-10T11:22:34.075 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:34.062 7ff5480b4700 1 -- start start 2021-02-10T11:22:34.075 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:34.063 7ff5480b4700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7ff5401b0470 0x7ff5401b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:34.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:34.063 7ff5480b4700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7ff5400f20e0 0x7ff5400f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:34.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:34.063 7ff5480b4700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5400f0b50 con 0x7ff5400ef610 2021-02-10T11:22:34.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:34.063 7ff5480b4700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5400f0820 con 0x7ff5401b0470 2021-02-10T11:22:34.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:34.063 7ff5480b4700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5400f24e0 con 0x7ff5400f20e0 2021-02-10T11:22:34.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:34.063 7ff545ed0700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7ff5401b0470 0x7ff5401b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:34.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:34.063 7ff5456cf700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7ff5400ef610 0x7ff5400efa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:47806/0 (socket says 172.21.2.102:47806) 2021-02-10T11:22:34.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:34.063 7ff5456cf700 1 -- 172.21.2.102:0/1303616448 learned_addr learned my addr 172.21.2.102:0/1303616448 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:22:34.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:34.064 7ff545ed0700 1 -- 172.21.2.102:0/1303616448 >> v2:172.21.2.113:3300/0 conn(0x7ff5400f20e0 msgr2=0x7ff5400f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:34.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:34.064 7ff545ed0700 1 --2- 172.21.2.102:0/1303616448 >> v2:172.21.2.113:3300/0 conn(0x7ff5400f20e0 0x7ff5400f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:34.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:34.064 7ff545ed0700 1 -- 172.21.2.102:0/1303616448 >> v1:172.21.2.102:6789/0 conn(0x7ff5400ef610 legacy=0x7ff5400efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:34.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:34.064 7ff545ed0700 1 -- 172.21.2.102:0/1303616448 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5400e52e0 con 0x7ff5401b0470 2021-02-10T11:22:34.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:34.064 7ff545ed0700 1 --2- 172.21.2.102:0/1303616448 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff5401b0470 0x7ff5401b0870 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 rx=0x7ff53c006650 tx=0x7ff53c009a80).ready entity=mon.0 client_cookie=59b1c3ea64a503e3 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:34.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:34.064 7ff544ece700 1 -- 172.21.2.102:0/1303616448 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff53c0009b0 con 0x7ff5401b0470 2021-02-10T11:22:34.077 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:34.064 7ff544ece700 1 -- 172.21.2.102:0/1303616448 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff53c00c0c0 con 0x7ff5401b0470 2021-02-10T11:22:34.077 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:34.064 7ff544ece700 1 -- 172.21.2.102:0/1303616448 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff53c00c5e0 con 0x7ff5401b0470 2021-02-10T11:22:34.077 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:34.064 7ff5480b4700 1 -- 172.21.2.102:0/1303616448 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff5401b0470 msgr2=0x7ff5401b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:34.077 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:34.064 7ff5480b4700 1 --2- 172.21.2.102:0/1303616448 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff5401b0470 0x7ff5401b0870 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 rx=0x7ff53c006650 tx=0x7ff53c009a80).stop 2021-02-10T11:22:34.077 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:34.065 7ff5480b4700 1 -- 172.21.2.102:0/1303616448 shutdown_connections 2021-02-10T11:22:34.077 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:34.065 7ff5480b4700 1 --2- 172.21.2.102:0/1303616448 >> v2:172.21.2.113:3300/0 conn(0x7ff5400f20e0 0x7ff5400f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:34.077 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:34.065 7ff5480b4700 1 --2- 172.21.2.102:0/1303616448 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff5401b0470 0x7ff5401b0870 unknown :-1 s=CLOSED pgs=1089 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:34.077 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:34.065 7ff5480b4700 1 -- 172.21.2.102:0/1303616448 >> 172.21.2.102:0/1303616448 conn(0x7ff5400ce3e0 msgr2=0x7ff5400e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:34.077 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:34.065 7ff5480b4700 1 -- 172.21.2.102:0/1303616448 shutdown_connections 2021-02-10T11:22:34.077 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:34.065 7ff5480b4700 1 -- 172.21.2.102:0/1303616448 wait complete. 2021-02-10T11:22:34.079 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:34.080 7ff5480b4700 1 Processor -- start 2021-02-10T11:22:34.079 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:34.080 7ff5480b4700 1 -- start start 2021-02-10T11:22:34.079 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:34.080 7ff5480b4700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff5400c36a0 0x7ff5400c2b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:34.079 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:34.080 7ff5480b4700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff5400c3190 0x7ff5400b1d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:34.079 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:34.080 7ff5480b4700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff5400b2370 0x7ff5400b2d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:34.079 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:34.080 7ff5480b4700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5400b27c0 con 0x7ff5400b2370 2021-02-10T11:22:34.079 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:34.080 7ff5480b4700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5400b3360 con 0x7ff5400c36a0 2021-02-10T11:22:34.079 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:34.080 7ff5480b4700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5400b3540 con 0x7ff5400c3190 2021-02-10T11:22:34.079 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:34.080 7ff5466d1700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff5400b2370 0x7ff5400b2d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:34.080 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:34.080 7ff5466d1700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff5400b2370 0x7ff5400b2d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:55590/0 (socket says 172.21.2.102:55590) 2021-02-10T11:22:34.080 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:34.080 7ff5466d1700 1 -- 172.21.2.102:0/3599510842 learned_addr learned my addr 172.21.2.102:0/3599510842 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:22:34.080 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:34.081 7ff5466d1700 1 -- 172.21.2.102:0/3599510842 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff5400c3190 msgr2=0x7ff5400b1d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:34.080 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:34.081 7ff5466d1700 1 --2- 172.21.2.102:0/3599510842 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff5400c3190 0x7ff5400b1d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:34.080 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:34.081 7ff5466d1700 1 -- 172.21.2.102:0/3599510842 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff5400c36a0 msgr2=0x7ff5400c2b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:34.080 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:34.081 7ff5466d1700 1 --2- 172.21.2.102:0/3599510842 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff5400c36a0 0x7ff5400c2b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:34.080 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:34.081 7ff5466d1700 1 -- 172.21.2.102:0/3599510842 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5400b3760 con 0x7ff5400b2370 2021-02-10T11:22:34.080 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:34.081 7ff5466d1700 1 --2- 172.21.2.102:0/3599510842 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff5400b2370 0x7ff5400b2d40 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 rx=0x7ff538009980 tx=0x7ff53800bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:34.080 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:34.081 7ff5367fc700 1 -- 172.21.2.102:0/3599510842 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff538003020 con 0x7ff5400b2370 2021-02-10T11:22:34.080 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:34.081 7ff5367fc700 1 -- 172.21.2.102:0/3599510842 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff538005a80 con 0x7ff5400b2370 2021-02-10T11:22:34.080 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:34.081 7ff5480b4700 1 -- 172.21.2.102:0/3599510842 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5400b3760 con 0x7ff5400b2370 2021-02-10T11:22:34.080 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:34.081 7ff5367fc700 1 -- 172.21.2.102:0/3599510842 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff538006030 con 0x7ff5400b2370 2021-02-10T11:22:34.080 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:34.081 7ff5480b4700 1 -- 172.21.2.102:0/3599510842 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5400b3760 con 0x7ff5400b2370 2021-02-10T11:22:34.081 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:34.082 7ff5367fc700 1 -- 172.21.2.102:0/3599510842 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ff538005c60 con 0x7ff5400b2370 2021-02-10T11:22:34.081 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:34.082 7ff5367fc700 1 --2- 172.21.2.102:0/3599510842 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff5280186c0 0x7ff52801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:34.081 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:34.082 7ff5367fc700 1 -- 172.21.2.102:0/3599510842 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 5573+0+0 (secure 0 0 0) 0x7ff5380328e0 con 0x7ff5400b2370 2021-02-10T11:22:34.081 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:34.082 7ff545ed0700 1 --2- 172.21.2.102:0/3599510842 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff5280186c0 0x7ff52801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:34.081 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:34.082 7ff545ed0700 1 --2- 172.21.2.102:0/3599510842 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff5280186c0 0x7ff52801abc0 crc :-1 s=READY pgs=524 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:34.108 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:34.113 7ff5480b4700 1 -- 172.21.2.102:0/3599510842 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5400edd10 con 0x7ff5400b2370 2021-02-10T11:22:34.111 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:34.115 7ff5367fc700 1 -- 172.21.2.102:0/3599510842 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ff538005870 con 0x7ff5400b2370 2021-02-10T11:22:34.240 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:34.244 7ff5480b4700 1 -- 172.21.2.102:0/3599510842 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7ff5400edd10 con 0x7ff5400b2370 2021-02-10T11:22:34.240 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:34.244 7ff5367fc700 1 -- 172.21.2.102:0/3599510842 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v39) v1 ==== 92+0+10 (secure 0 0 0) 0x7ff538005c60 con 0x7ff5400b2370 2021-02-10T11:22:34.241 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:22:34.242 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:34.246 7ff5480b4700 1 -- 172.21.2.102:0/3599510842 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff5280186c0 msgr2=0x7ff52801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:34.243 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:34.246 7ff5480b4700 1 --2- 172.21.2.102:0/3599510842 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff5280186c0 0x7ff52801abc0 crc :-1 s=READY pgs=524 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:34.243 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:34.246 7ff5480b4700 1 -- 172.21.2.102:0/3599510842 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff5400b2370 msgr2=0x7ff5400b2d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:34.243 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:34.246 7ff5480b4700 1 --2- 172.21.2.102:0/3599510842 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff5400b2370 0x7ff5400b2d40 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 rx=0x7ff538009980 tx=0x7ff53800bcd0).stop 2021-02-10T11:22:34.243 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:34.247 7ff5480b4700 1 -- 172.21.2.102:0/3599510842 shutdown_connections 2021-02-10T11:22:34.243 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:34.247 7ff5480b4700 1 --2- 172.21.2.102:0/3599510842 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff5400b2370 0x7ff5400b2d40 unknown :-1 s=CLOSED pgs=1090 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:34.243 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:34.247 7ff5480b4700 1 --2- 172.21.2.102:0/3599510842 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff5400c3190 0x7ff5400b1d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:34.243 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:34.247 7ff5480b4700 1 --2- 172.21.2.102:0/3599510842 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff5280186c0 0x7ff52801abc0 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:34.243 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:34.247 7ff5480b4700 1 --2- 172.21.2.102:0/3599510842 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff5400c36a0 0x7ff5400c2b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:34.243 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:34.247 7ff5480b4700 1 -- 172.21.2.102:0/3599510842 >> 172.21.2.102:0/3599510842 conn(0x7ff54009f630 msgr2=0x7ff5400a0130 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:34.243 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:34.247 7ff5480b4700 1 -- 172.21.2.102:0/3599510842 shutdown_connections 2021-02-10T11:22:34.243 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:34.247 7ff5480b4700 1 -- 172.21.2.102:0/3599510842 wait complete. 2021-02-10T11:22:34.251 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-02-10T11:22:34.681 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:34.669 7f35871ba700 1 Processor -- start 2021-02-10T11:22:34.681 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:34.669 7f35871ba700 1 -- start start 2021-02-10T11:22:34.682 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:34.670 7f35871ba700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f35801b0870 0x7f35800f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:34.682 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:34.670 7f35871ba700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f35800efe00 0x7f35800f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:34.682 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:34.670 7f35871ba700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35801b0c70 con 0x7f35801b0470 2021-02-10T11:22:34.682 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:34.670 7f35871ba700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35800f0b50 con 0x7f35801b0870 2021-02-10T11:22:34.682 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:34.670 7f35871ba700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35800f0820 con 0x7f35800efe00 2021-02-10T11:22:34.682 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:34.670 7f357ffff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f35801b0870 0x7f35800f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:34.682 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:34.670 7f3584fd6700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f35801b0470 0x7f35800ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:47818/0 (socket says 172.21.2.102:47818) 2021-02-10T11:22:34.682 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:34.670 7f3584fd6700 1 -- 172.21.2.102:0/1884214948 learned_addr learned my addr 172.21.2.102:0/1884214948 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:22:34.682 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:34.670 7f35857d7700 1 --2- 172.21.2.102:0/1884214948 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f35800efe00 0x7f35800f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:34.682 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:34.670 7f357f7fe700 1 -- 172.21.2.102:0/1884214948 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2703055160 0 0) 0x7f3568001180 con 0x7f35801b0470 2021-02-10T11:22:34.682 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:34.670 7f357f7fe700 1 -- 172.21.2.102:0/1884214948 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f356c002fe0 con 0x7f35801b0470 2021-02-10T11:22:34.682 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:34.670 7f35857d7700 1 -- 172.21.2.102:0/1884214948 >> v2:172.21.2.102:3300/0 conn(0x7f35801b0870 msgr2=0x7f35800f20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:34.683 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:34.670 7f35857d7700 1 --2- 172.21.2.102:0/1884214948 >> v2:172.21.2.102:3300/0 conn(0x7f35801b0870 0x7f35800f20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:34.683 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:34.670 7f35857d7700 1 -- 172.21.2.102:0/1884214948 >> v1:172.21.2.102:6789/0 conn(0x7f35801b0470 legacy=0x7f35800ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:34.683 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:34.670 7f35857d7700 1 -- 172.21.2.102:0/1884214948 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35800e52e0 con 0x7f35800efe00 2021-02-10T11:22:34.683 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:34.670 7f357ffff700 1 --2- 172.21.2.102:0/1884214948 >> v2:172.21.2.102:3300/0 conn(0x7f35801b0870 0x7f35800f20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:22:34.683 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:34.671 7f35857d7700 1 --2- 172.21.2.102:0/1884214948 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f35800efe00 0x7f35800f0200 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7f3578006ad0 tx=0x7f3578009ad0).ready entity=mon.2 client_cookie=3694ced5ecdc8a8 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:34.683 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:34.671 7f357f7fe700 1 -- 172.21.2.102:0/1884214948 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f357800c0c0 con 0x7f35800efe00 2021-02-10T11:22:34.683 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:34.671 7f357f7fe700 1 -- 172.21.2.102:0/1884214948 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f357800c440 con 0x7f35800efe00 2021-02-10T11:22:34.683 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:34.671 7f357f7fe700 1 -- 172.21.2.102:0/1884214948 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f357800c8a0 con 0x7f35800efe00 2021-02-10T11:22:34.683 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:34.671 7f35871ba700 1 -- 172.21.2.102:0/1884214948 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f35800efe00 msgr2=0x7f35800f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:34.683 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:34.671 7f35871ba700 1 --2- 172.21.2.102:0/1884214948 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f35800efe00 0x7f35800f0200 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7f3578006ad0 tx=0x7f3578009ad0).stop 2021-02-10T11:22:34.684 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:34.671 7f35871ba700 1 -- 172.21.2.102:0/1884214948 shutdown_connections 2021-02-10T11:22:34.684 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:34.671 7f35871ba700 1 --2- 172.21.2.102:0/1884214948 >> v2:172.21.2.102:3300/0 conn(0x7f35801b0870 0x7f35800f20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:34.684 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:34.671 7f35871ba700 1 --2- 172.21.2.102:0/1884214948 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f35800efe00 0x7f35800f0200 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:34.684 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:34.671 7f35871ba700 1 -- 172.21.2.102:0/1884214948 >> 172.21.2.102:0/1884214948 conn(0x7f35800ce3e0 msgr2=0x7f35800e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:34.684 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:34.671 7f35871ba700 1 -- 172.21.2.102:0/1884214948 shutdown_connections 2021-02-10T11:22:34.684 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:34.672 7f35871ba700 1 -- 172.21.2.102:0/1884214948 wait complete. 2021-02-10T11:22:34.685 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:34.686 7f35871ba700 1 Processor -- start 2021-02-10T11:22:34.685 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:34.686 7f35871ba700 1 -- start start 2021-02-10T11:22:34.685 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:34.686 7f35871ba700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f35800a0420 0x7f35800a0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:34.686 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:34.686 7f35871ba700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3580189d00 0x7f358018a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:34.686 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:34.686 7f35871ba700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f358018a720 0x7f35800b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:34.686 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:34.686 7f35871ba700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3580189800 con 0x7f3580189d00 2021-02-10T11:22:34.686 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:34.686 7f35871ba700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f358018ab20 con 0x7f35800a0420 2021-02-10T11:22:34.686 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:34.686 7f35871ba700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35800a1140 con 0x7f358018a720 2021-02-10T11:22:34.686 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:34.687 7f357ffff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3580189d00 0x7f358018a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:34.686 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:34.687 7f357ffff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3580189d00 0x7f358018a100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:55600/0 (socket says 172.21.2.102:55600) 2021-02-10T11:22:34.686 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:34.687 7f357ffff700 1 -- 172.21.2.102:0/670411236 learned_addr learned my addr 172.21.2.102:0/670411236 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:22:34.686 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:34.687 7f357ffff700 1 -- 172.21.2.102:0/670411236 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f358018a720 msgr2=0x7f35800b7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:34.686 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:34.687 7f357ffff700 1 --2- 172.21.2.102:0/670411236 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f358018a720 0x7f35800b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:34.686 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:34.687 7f357ffff700 1 -- 172.21.2.102:0/670411236 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f35800a0420 msgr2=0x7f35800a0820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:34.686 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:34.687 7f357ffff700 1 --2- 172.21.2.102:0/670411236 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f35800a0420 0x7f35800a0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:34.686 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:34.687 7f357ffff700 1 -- 172.21.2.102:0/670411236 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35800b8200 con 0x7f3580189d00 2021-02-10T11:22:34.687 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:34.687 7f357ffff700 1 --2- 172.21.2.102:0/670411236 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3580189d00 0x7f358018a100 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 rx=0x7f357000b970 tx=0x7f357000bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:34.687 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:34.687 7f357d7fa700 1 -- 172.21.2.102:0/670411236 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3570000d30 con 0x7f3580189d00 2021-02-10T11:22:34.687 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:34.687 7f357d7fa700 1 -- 172.21.2.102:0/670411236 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3570006240 con 0x7f3580189d00 2021-02-10T11:22:34.687 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:34.687 7f35871ba700 1 -- 172.21.2.102:0/670411236 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35800b8200 con 0x7f3580189d00 2021-02-10T11:22:34.687 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:34.687 7f357d7fa700 1 -- 172.21.2.102:0/670411236 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3570005660 con 0x7f3580189d00 2021-02-10T11:22:34.687 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:34.687 7f35871ba700 1 -- 172.21.2.102:0/670411236 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35800b8790 con 0x7f3580189d00 2021-02-10T11:22:34.687 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:34.688 7f357d7fa700 1 -- 172.21.2.102:0/670411236 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f3570006420 con 0x7f3580189d00 2021-02-10T11:22:34.687 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:34.688 7f357d7fa700 1 --2- 172.21.2.102:0/670411236 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3564018730 0x7f356401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:34.687 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:34.688 7f357d7fa700 1 -- 172.21.2.102:0/670411236 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f35700329f0 con 0x7f3580189d00 2021-02-10T11:22:34.687 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:34.688 7f3584fd6700 1 --2- 172.21.2.102:0/670411236 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3564018730 0x7f356401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:34.687 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:34.688 7f3584fd6700 1 --2- 172.21.2.102:0/670411236 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3564018730 0x7f356401ac10 crc :-1 s=READY pgs=525 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:34.715 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:34.719 7f35871ba700 1 -- 172.21.2.102:0/670411236 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3580105ae0 con 0x7f3580189d00 2021-02-10T11:22:34.718 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:34.722 7f357d7fa700 1 -- 172.21.2.102:0/670411236 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f3570000cc0 con 0x7f3580189d00 2021-02-10T11:22:34.846 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:34.850 7f35871ba700 1 -- 172.21.2.102:0/670411236 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7f35800f0b50 con 0x7f3580189d00 2021-02-10T11:22:34.847 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:34.851 7f357d7fa700 1 -- 172.21.2.102:0/670411236 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v39) v1 ==== 104+0+10 (secure 0 0 0) 0x7f3570006240 con 0x7f3580189d00 2021-02-10T11:22:34.847 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:22:34.848 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:34.852 7f35871ba700 1 -- 172.21.2.102:0/670411236 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3564018730 msgr2=0x7f356401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:34.848 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:34.852 7f35871ba700 1 --2- 172.21.2.102:0/670411236 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3564018730 0x7f356401ac10 crc :-1 s=READY pgs=525 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:34.849 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:34.852 7f35871ba700 1 -- 172.21.2.102:0/670411236 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3580189d00 msgr2=0x7f358018a100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:34.849 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:34.852 7f35871ba700 1 --2- 172.21.2.102:0/670411236 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3580189d00 0x7f358018a100 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 rx=0x7f357000b970 tx=0x7f357000bd10).stop 2021-02-10T11:22:34.849 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:34.853 7f35871ba700 1 -- 172.21.2.102:0/670411236 shutdown_connections 2021-02-10T11:22:34.849 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:34.853 7f35871ba700 1 --2- 172.21.2.102:0/670411236 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f358018a720 0x7f35800b7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:34.849 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:34.853 7f35871ba700 1 --2- 172.21.2.102:0/670411236 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3580189d00 0x7f358018a100 unknown :-1 s=CLOSED pgs=1092 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:34.849 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:34.853 7f35871ba700 1 --2- 172.21.2.102:0/670411236 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3564018730 0x7f356401ac10 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:34.849 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:34.853 7f35871ba700 1 --2- 172.21.2.102:0/670411236 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f35800a0420 0x7f35800a0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:34.850 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:34.853 7f35871ba700 1 -- 172.21.2.102:0/670411236 >> 172.21.2.102:0/670411236 conn(0x7f3580094e10 msgr2=0x7f35800a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:34.850 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:34.853 7f35871ba700 1 -- 172.21.2.102:0/670411236 shutdown_connections 2021-02-10T11:22:34.850 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:34.853 7f35871ba700 1 -- 172.21.2.102:0/670411236 wait complete. 2021-02-10T11:22:34.857 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-02-10T11:22:35.287 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:35.273 7f5049d0c700 1 Processor -- start 2021-02-10T11:22:35.287 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:35.273 7f5049d0c700 1 -- start start 2021-02-10T11:22:35.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:35.274 7f5049d0c700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f50440f05f0 0x7f50440f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:35.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:35.274 7f5049d0c700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50441b0c70 con 0x7f50441b0470 2021-02-10T11:22:35.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:35.274 7f5049d0c700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50440f0b50 con 0x7f50441b0870 2021-02-10T11:22:35.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:35.274 7f5049d0c700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50440e5ea0 con 0x7f50440f05f0 2021-02-10T11:22:35.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:35.274 7f50437fe700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f50441b0470 0x7f50440ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:47832/0 (socket says 172.21.2.102:47832) 2021-02-10T11:22:35.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:35.274 7f50437fe700 1 -- 172.21.2.102:0/4002376712 learned_addr learned my addr 172.21.2.102:0/4002376712 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:22:35.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:35.274 7f5043fff700 1 --2- 172.21.2.102:0/4002376712 >> v2:172.21.2.102:3300/0 conn(0x7f50440f05f0 0x7f50440f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:35.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:35.275 7f5043fff700 1 -- 172.21.2.102:0/4002376712 >> v1:172.21.2.113:6789/0 conn(0x7f50441b0870 legacy=0x7f50440efe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:35.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:35.275 7f5043fff700 1 -- 172.21.2.102:0/4002376712 >> v1:172.21.2.102:6789/0 conn(0x7f50441b0470 legacy=0x7f50440ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:35.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:35.275 7f5043fff700 1 -- 172.21.2.102:0/4002376712 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50440e52e0 con 0x7f50440f05f0 2021-02-10T11:22:35.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:35.275 7f5043fff700 1 --2- 172.21.2.102:0/4002376712 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f50440f05f0 0x7f50440f20e0 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 rx=0x7f503c006e30 tx=0x7f503c009de0).ready entity=mon.0 client_cookie=f3c0bb7f15abfbe0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:35.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:35.275 7f50427fc700 1 -- 172.21.2.102:0/4002376712 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f503c00b300 con 0x7f50440f05f0 2021-02-10T11:22:35.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:35.275 7f50427fc700 1 -- 172.21.2.102:0/4002376712 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f503c00b680 con 0x7f50440f05f0 2021-02-10T11:22:35.289 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:35.275 7f50427fc700 1 -- 172.21.2.102:0/4002376712 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f503c00bba0 con 0x7f50440f05f0 2021-02-10T11:22:35.289 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:35.275 7f5049d0c700 1 -- 172.21.2.102:0/4002376712 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f50440f05f0 msgr2=0x7f50440f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:35.289 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:35.275 7f5049d0c700 1 --2- 172.21.2.102:0/4002376712 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f50440f05f0 0x7f50440f20e0 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 rx=0x7f503c006e30 tx=0x7f503c009de0).stop 2021-02-10T11:22:35.289 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:35.276 7f5049d0c700 1 -- 172.21.2.102:0/4002376712 shutdown_connections 2021-02-10T11:22:35.289 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:35.276 7f5049d0c700 1 --2- 172.21.2.102:0/4002376712 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f50440f05f0 0x7f50440f20e0 unknown :-1 s=CLOSED pgs=1094 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:35.289 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:35.276 7f5049d0c700 1 -- 172.21.2.102:0/4002376712 >> 172.21.2.102:0/4002376712 conn(0x7f50440ce3e0 msgr2=0x7f50440e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:35.289 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:35.276 7f5049d0c700 1 -- 172.21.2.102:0/4002376712 shutdown_connections 2021-02-10T11:22:35.289 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:35.276 7f5049d0c700 1 -- 172.21.2.102:0/4002376712 wait complete. 2021-02-10T11:22:35.290 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:35.291 7f5049d0c700 1 Processor -- start 2021-02-10T11:22:35.291 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:35.291 7f5049d0c700 1 -- start start 2021-02-10T11:22:35.291 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:35.291 7f5049d0c700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5044099d70 0x7f5044081e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:35.291 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:35.291 7f5049d0c700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f50440a9bf0 0x7f50440a9ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:35.291 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:35.291 7f5049d0c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f50440aa610 0x7f50440c1830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:35.291 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:35.291 7f5049d0c700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5044082480 con 0x7f50440aa610 2021-02-10T11:22:35.291 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:35.291 7f5049d0c700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50440c1e80 con 0x7f5044099d70 2021-02-10T11:22:35.291 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:35.291 7f5049d0c700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50440c20a0 con 0x7f50440a9bf0 2021-02-10T11:22:35.291 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:35.291 7f5043fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f50440aa610 0x7f50440c1830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:35.291 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:35.291 7f5043fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f50440aa610 0x7f50440c1830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:55616/0 (socket says 172.21.2.102:55616) 2021-02-10T11:22:35.291 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:35.291 7f5043fff700 1 -- 172.21.2.102:0/3757756541 learned_addr learned my addr 172.21.2.102:0/3757756541 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:22:35.291 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:35.292 7f5043fff700 1 -- 172.21.2.102:0/3757756541 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f50440a9bf0 msgr2=0x7f50440a9ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:35.291 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:35.292 7f5043fff700 1 --2- 172.21.2.102:0/3757756541 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f50440a9bf0 0x7f50440a9ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:35.292 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:35.292 7f5043fff700 1 -- 172.21.2.102:0/3757756541 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5044099d70 msgr2=0x7f5044081e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:35.292 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:35.292 7f5043fff700 1 --2- 172.21.2.102:0/3757756541 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5044099d70 0x7f5044081e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:35.292 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:35.292 7f5043fff700 1 -- 172.21.2.102:0/3757756541 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50440c22c0 con 0x7f50440aa610 2021-02-10T11:22:35.292 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:35.292 7f5043fff700 1 --2- 172.21.2.102:0/3757756541 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f50440aa610 0x7f50440c1830 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 rx=0x7f503c00b970 tx=0x7f503c00bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:35.292 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:35.292 7f503bfff700 1 -- 172.21.2.102:0/3757756541 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f503c009380 con 0x7f50440aa610 2021-02-10T11:22:35.292 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:35.292 7f503bfff700 1 -- 172.21.2.102:0/3757756541 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f503c005820 con 0x7f50440aa610 2021-02-10T11:22:35.292 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:35.292 7f5049d0c700 1 -- 172.21.2.102:0/3757756541 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50440c22c0 con 0x7f50440aa610 2021-02-10T11:22:35.292 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:35.292 7f5049d0c700 1 -- 172.21.2.102:0/3757756541 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50440c2850 con 0x7f50440aa610 2021-02-10T11:22:35.292 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:35.292 7f503bfff700 1 -- 172.21.2.102:0/3757756541 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f503c005d00 con 0x7f50440aa610 2021-02-10T11:22:35.292 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:35.293 7f503bfff700 1 -- 172.21.2.102:0/3757756541 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f503c005a00 con 0x7f50440aa610 2021-02-10T11:22:35.292 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:35.293 7f503bfff700 1 --2- 172.21.2.102:0/3757756541 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f50280186c0 0x7f502801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:35.292 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:35.293 7f503bfff700 1 -- 172.21.2.102:0/3757756541 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f503c0329c0 con 0x7f50440aa610 2021-02-10T11:22:35.292 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:35.293 7f50437fe700 1 --2- 172.21.2.102:0/3757756541 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f50280186c0 0x7f502801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:35.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:35.293 7f50437fe700 1 --2- 172.21.2.102:0/3757756541 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f50280186c0 0x7f502801abc0 crc :-1 s=READY pgs=526 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:35.321 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:35.324 7f5049d0c700 1 -- 172.21.2.102:0/3757756541 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f504407e6b0 con 0x7f50440aa610 2021-02-10T11:22:35.323 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:35.326 7f503bfff700 1 -- 172.21.2.102:0/3757756541 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f503c009380 con 0x7f50440aa610 2021-02-10T11:22:35.452 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:35.455 7f5049d0c700 1 -- 172.21.2.102:0/3757756541 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7f50440ce3e0 con 0x7f50440aa610 2021-02-10T11:22:35.452 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:35.455 7f503bfff700 1 -- 172.21.2.102:0/3757756541 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v39) v1 ==== 100+0+10 (secure 0 0 0) 0x7f503c005820 con 0x7f50440aa610 2021-02-10T11:22:35.453 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:22:35.454 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:35.457 7f5049d0c700 1 -- 172.21.2.102:0/3757756541 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f50280186c0 msgr2=0x7f502801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:35.454 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:35.457 7f5049d0c700 1 --2- 172.21.2.102:0/3757756541 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f50280186c0 0x7f502801abc0 crc :-1 s=READY pgs=526 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:35.454 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:35.457 7f5049d0c700 1 -- 172.21.2.102:0/3757756541 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f50440aa610 msgr2=0x7f50440c1830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:35.455 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:35.457 7f5049d0c700 1 --2- 172.21.2.102:0/3757756541 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f50440aa610 0x7f50440c1830 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 rx=0x7f503c00b970 tx=0x7f503c00bd10).stop 2021-02-10T11:22:35.455 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:35.458 7f5049d0c700 1 -- 172.21.2.102:0/3757756541 shutdown_connections 2021-02-10T11:22:35.455 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:35.458 7f5049d0c700 1 --2- 172.21.2.102:0/3757756541 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f50440aa610 0x7f50440c1830 unknown :-1 s=CLOSED pgs=1095 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:35.455 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:35.458 7f5049d0c700 1 --2- 172.21.2.102:0/3757756541 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f50440a9bf0 0x7f50440a9ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:35.455 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:35.458 7f5049d0c700 1 --2- 172.21.2.102:0/3757756541 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f50280186c0 0x7f502801abc0 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:35.455 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:35.458 7f5049d0c700 1 --2- 172.21.2.102:0/3757756541 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5044099d70 0x7f5044081e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:35.455 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:35.458 7f5049d0c700 1 -- 172.21.2.102:0/3757756541 >> 172.21.2.102:0/3757756541 conn(0x7f5044088220 msgr2=0x7f50440a2d50 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:35.455 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:35.458 7f5049d0c700 1 -- 172.21.2.102:0/3757756541 shutdown_connections 2021-02-10T11:22:35.455 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:35.458 7f5049d0c700 1 -- 172.21.2.102:0/3757756541 wait complete. 2021-02-10T11:22:35.463 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-02-10T11:22:35.463 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:22:35.890 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:35.877 7fd90eaae700 1 Processor -- start 2021-02-10T11:22:35.890 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:35.877 7fd90eaae700 1 -- start start 2021-02-10T11:22:35.890 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:35.877 7fd90eaae700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fd9080e5070 0x7fd9081b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:35.890 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:35.877 7fd90eaae700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fd9080e5470 0x7fd908080310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:35.890 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:35.877 7fd90eaae700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd90807ed80 con 0x7fd9081b3e40 2021-02-10T11:22:35.890 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:35.877 7fd90eaae700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9080d1790 con 0x7fd9080e5070 2021-02-10T11:22:35.890 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:35.878 7fd90eaae700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9080d19b0 con 0x7fd9080e5470 2021-02-10T11:22:35.890 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:35.878 7fd90c8ca700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fd9080e5070 0x7fd9081b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:35.891 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:35.878 7fd90c8ca700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fd9080e5070 0x7fd9081b3820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:55624/0 (socket says 172.21.2.102:55624) 2021-02-10T11:22:35.891 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:35.878 7fd90c8ca700 1 -- 172.21.2.102:0/2859446433 learned_addr learned my addr 172.21.2.102:0/2859446433 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:22:35.891 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:35.878 7fd90c8ca700 1 -- 172.21.2.102:0/2859446433 >> v2:172.21.2.113:3300/0 conn(0x7fd9080e5470 msgr2=0x7fd908080310 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:22:35.891 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:35.878 7fd90c8ca700 1 --2- 172.21.2.102:0/2859446433 >> v2:172.21.2.113:3300/0 conn(0x7fd9080e5470 0x7fd908080310 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:35.891 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:35.878 7fd90c8ca700 1 -- 172.21.2.102:0/2859446433 >> v1:172.21.2.113:6789/0 conn(0x7fd9081b3e40 legacy=0x7fd90807d690 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:22:35.891 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:35.878 7fd90c8ca700 1 -- 172.21.2.102:0/2859446433 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9080d1bd0 con 0x7fd9080e5070 2021-02-10T11:22:35.891 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:35.878 7fd90c8ca700 1 --2- 172.21.2.102:0/2859446433 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd9080e5070 0x7fd9081b3820 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 rx=0x7fd8f8006700 tx=0x7fd8f8009b00).ready entity=mon.0 client_cookie=eeef9ad7ad82d004 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:35.891 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:35.878 7fd9077fe700 1 -- 172.21.2.102:0/2859446433 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd8f800c0c0 con 0x7fd9080e5070 2021-02-10T11:22:35.891 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:35.878 7fd9077fe700 1 -- 172.21.2.102:0/2859446433 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8f800c400 con 0x7fd9080e5070 2021-02-10T11:22:35.892 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:35.878 7fd9077fe700 1 -- 172.21.2.102:0/2859446433 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd8f800c8e0 con 0x7fd9080e5070 2021-02-10T11:22:35.892 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:35.878 7fd90eaae700 1 -- 172.21.2.102:0/2859446433 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd9080e5070 msgr2=0x7fd9081b3820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:35.892 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:35.878 7fd90eaae700 1 --2- 172.21.2.102:0/2859446433 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd9080e5070 0x7fd9081b3820 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 rx=0x7fd8f8006700 tx=0x7fd8f8009b00).stop 2021-02-10T11:22:35.892 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:35.879 7fd90eaae700 1 -- 172.21.2.102:0/2859446433 shutdown_connections 2021-02-10T11:22:35.892 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:35.879 7fd90eaae700 1 --2- 172.21.2.102:0/2859446433 >> v2:172.21.2.113:3300/0 conn(0x7fd9080e5470 0x7fd908080310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:35.892 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:35.879 7fd90eaae700 1 --2- 172.21.2.102:0/2859446433 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd9080e5070 0x7fd9081b3820 unknown :-1 s=CLOSED pgs=1096 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:35.892 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:35.879 7fd90eaae700 1 -- 172.21.2.102:0/2859446433 >> 172.21.2.102:0/2859446433 conn(0x7fd9080ce690 msgr2=0x7fd9080e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:35.892 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:35.879 7fd90eaae700 1 -- 172.21.2.102:0/2859446433 shutdown_connections 2021-02-10T11:22:35.892 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:35.879 7fd90eaae700 1 -- 172.21.2.102:0/2859446433 wait complete. 2021-02-10T11:22:35.894 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:35.893 7fd90eaae700 1 Processor -- start 2021-02-10T11:22:35.894 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:35.893 7fd90eaae700 1 -- start start 2021-02-10T11:22:35.894 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:35.894 7fd90eaae700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd9081aad40 0x7fd908192110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:35.894 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:35.894 7fd90eaae700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd9081a1aa0 0x7fd9080b7530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:35.894 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:35.894 7fd90eaae700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd908192730 0x7fd9080b7b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:35.894 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:35.894 7fd90eaae700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd908192d70 con 0x7fd9081a1aa0 2021-02-10T11:22:35.894 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:35.894 7fd90eaae700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9081a1590 con 0x7fd908192730 2021-02-10T11:22:35.894 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:35.894 7fd90eaae700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9081a11b0 con 0x7fd9081aad40 2021-02-10T11:22:35.895 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:35.894 7fd907fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd9081a1aa0 0x7fd9080b7530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:35.895 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:35.894 7fd907fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd9081a1aa0 0x7fd9080b7530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:55628/0 (socket says 172.21.2.102:55628) 2021-02-10T11:22:35.895 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:35.894 7fd907fff700 1 -- 172.21.2.102:0/2184347874 learned_addr learned my addr 172.21.2.102:0/2184347874 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:22:35.895 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:35.894 7fd907fff700 1 -- 172.21.2.102:0/2184347874 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd9081aad40 msgr2=0x7fd908192110 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:22:35.895 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:35.894 7fd907fff700 1 --2- 172.21.2.102:0/2184347874 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd9081aad40 0x7fd908192110 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:35.895 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:35.894 7fd907fff700 1 -- 172.21.2.102:0/2184347874 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd908192730 msgr2=0x7fd9080b7b50 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:22:35.895 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:35.894 7fd907fff700 1 --2- 172.21.2.102:0/2184347874 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd908192730 0x7fd9080b7b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:35.895 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:35.894 7fd907fff700 1 -- 172.21.2.102:0/2184347874 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9080b8170 con 0x7fd9081a1aa0 2021-02-10T11:22:35.895 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:35.894 7fd907fff700 1 --2- 172.21.2.102:0/2184347874 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd9081a1aa0 0x7fd9080b7530 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 rx=0x7fd8fc00b5b0 tx=0x7fd8fc00b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:35.895 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:35.894 7fd9057fa700 1 -- 172.21.2.102:0/2184347874 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd8fc00bd40 con 0x7fd9081a1aa0 2021-02-10T11:22:35.895 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:35.894 7fd9057fa700 1 -- 172.21.2.102:0/2184347874 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8fc006240 con 0x7fd9081a1aa0 2021-02-10T11:22:35.895 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:35.894 7fd90eaae700 1 -- 172.21.2.102:0/2184347874 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9080b8170 con 0x7fd9081a1aa0 2021-02-10T11:22:35.895 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:35.894 7fd9057fa700 1 -- 172.21.2.102:0/2184347874 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd8fc005720 con 0x7fd9081a1aa0 2021-02-10T11:22:35.896 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:35.895 7fd90eaae700 1 -- 172.21.2.102:0/2184347874 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9080b8170 con 0x7fd9081a1aa0 2021-02-10T11:22:35.896 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:35.895 7fd9057fa700 1 -- 172.21.2.102:0/2184347874 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd8fc0054a0 con 0x7fd9081a1aa0 2021-02-10T11:22:35.896 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:35.895 7fd9057fa700 1 --2- 172.21.2.102:0/2184347874 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd8e8018730 0x7fd8e801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:35.896 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:35.895 7fd9057fa700 1 -- 172.21.2.102:0/2184347874 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fd8fc0327f0 con 0x7fd9081a1aa0 2021-02-10T11:22:35.896 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:35.895 7fd90c8ca700 1 --2- 172.21.2.102:0/2184347874 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd8e8018730 0x7fd8e801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:35.896 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:35.896 7fd90c8ca700 1 --2- 172.21.2.102:0/2184347874 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd8e8018730 0x7fd8e801ac10 crc :-1 s=READY pgs=527 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:35.924 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:35.927 7fd90eaae700 1 -- 172.21.2.102:0/2184347874 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd908083da0 con 0x7fd9081a1aa0 2021-02-10T11:22:35.927 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:35.930 7fd9057fa700 1 -- 172.21.2.102:0/2184347874 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd8fc006030 con 0x7fd9081a1aa0 2021-02-10T11:22:36.054 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:36.057 7fd90eaae700 1 -- 172.21.2.102:0/2184347874 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd908083da0 con 0x7fd9081a1aa0 2021-02-10T11:22:36.054 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:36.057 7fd9057fa700 1 -- 172.21.2.102:0/2184347874 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) v1 ==== 94+0+2099 (secure 0 0 0) 0x7fd8fc00bd40 con 0x7fd9081a1aa0 2021-02-10T11:22:36.055 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:22:36.055 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":21,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7194,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.102:6835/2295196020","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6834","nonce":2295196020},{"type":"v1","addr":"172.21.2.102:6835","nonce":2295196020}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":18}],"filesystems":[{"mdsmap":{"epoch":21,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:22:26.122430","modified":"2021-02-10 11:22:28.126179","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7191},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7191":{"gid":7191,"name":"b","rank":0,"incarnation":19,"state":"up:active","state_seq":3,"addr":"172.21.2.113:6833/3694995761","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6832","nonce":3694995761},{"type":"v1","addr":"172.21.2.113:6833","nonce":3694995761}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":3}]} 2021-02-10T11:22:36.055 INFO:teuthology.orchestra.run.gibba002.stderr:dumped fsmap epoch 21 2021-02-10T11:22:36.056 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:36.059 7fd90eaae700 1 -- 172.21.2.102:0/2184347874 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd8e8018730 msgr2=0x7fd8e801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:36.056 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:36.059 7fd90eaae700 1 --2- 172.21.2.102:0/2184347874 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd8e8018730 0x7fd8e801ac10 crc :-1 s=READY pgs=527 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:36.056 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:36.059 7fd90eaae700 1 -- 172.21.2.102:0/2184347874 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd9081a1aa0 msgr2=0x7fd9080b7530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:36.056 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:36.059 7fd90eaae700 1 --2- 172.21.2.102:0/2184347874 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd9081a1aa0 0x7fd9080b7530 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 rx=0x7fd8fc00b5b0 tx=0x7fd8fc00b950).stop 2021-02-10T11:22:36.057 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:36.060 7fd90eaae700 1 -- 172.21.2.102:0/2184347874 shutdown_connections 2021-02-10T11:22:36.057 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:36.060 7fd90eaae700 1 --2- 172.21.2.102:0/2184347874 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd8e8018730 0x7fd8e801ac10 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:36.057 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:36.060 7fd90eaae700 1 --2- 172.21.2.102:0/2184347874 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd908192730 0x7fd9080b7b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:36.057 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:36.060 7fd90eaae700 1 --2- 172.21.2.102:0/2184347874 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd9081a1aa0 0x7fd9080b7530 unknown :-1 s=CLOSED pgs=1097 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:36.057 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:36.060 7fd90eaae700 1 --2- 172.21.2.102:0/2184347874 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd9081aad40 0x7fd908192110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:36.057 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:36.060 7fd90eaae700 1 -- 172.21.2.102:0/2184347874 >> 172.21.2.102:0/2184347874 conn(0x7fd9081a2470 msgr2=0x7fd908194f50 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:36.057 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:36.060 7fd90eaae700 1 -- 172.21.2.102:0/2184347874 shutdown_connections 2021-02-10T11:22:36.057 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:36.060 7fd90eaae700 1 -- 172.21.2.102:0/2184347874 wait complete. 2021-02-10T11:22:36.267 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 21, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:22:26.122430', 'modified': '2021-02-10 11:22:28.126179', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7191}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7191': {'gid': 7191, 'name': 'b', 'rank': 0, 'incarnation': 19, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.113:6833/3694995761', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.113:6832', 'nonce': 3694995761}, {'type': 'v1', 'addr': '172.21.2.113:6833', 'nonce': 3694995761}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [7], 'metadata_pool': 6, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:22:36.267 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:22:36.267 DEBUG:teuthology.orchestra.run.gibba013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:22:36.666 INFO:teuthology.orchestra.run.gibba013.stdout:{ 2021-02-10T11:22:36.666 INFO:teuthology.orchestra.run.gibba013.stdout: "cluster_fsid": "68db30ea-dbce-4aef-a6df-f01c6fea0513", 2021-02-10T11:22:36.667 INFO:teuthology.orchestra.run.gibba013.stdout: "whoami": 0, 2021-02-10T11:22:36.667 INFO:teuthology.orchestra.run.gibba013.stdout: "id": 7191, 2021-02-10T11:22:36.667 INFO:teuthology.orchestra.run.gibba013.stdout: "want_state": "up:active", 2021-02-10T11:22:36.667 INFO:teuthology.orchestra.run.gibba013.stdout: "state": "up:active", 2021-02-10T11:22:36.667 INFO:teuthology.orchestra.run.gibba013.stdout: "rank_uptime": 10.546590721999999, 2021-02-10T11:22:36.667 INFO:teuthology.orchestra.run.gibba013.stdout: "mdsmap_epoch": 21, 2021-02-10T11:22:36.667 INFO:teuthology.orchestra.run.gibba013.stdout: "osdmap_epoch": 39, 2021-02-10T11:22:36.667 INFO:teuthology.orchestra.run.gibba013.stdout: "osdmap_epoch_barrier": 39, 2021-02-10T11:22:36.667 INFO:teuthology.orchestra.run.gibba013.stdout: "uptime": 17.205895272999999 2021-02-10T11:22:36.667 INFO:teuthology.orchestra.run.gibba013.stdout:} 2021-02-10T11:22:36.670 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "68db30ea-dbce-4aef-a6df-f01c6fea0513", "whoami": 0, "id": 7191, "want_state": "up:active", "state": "up:active", "rank_uptime": 10.546590721999999, "mdsmap_epoch": 21, "osdmap_epoch": 39, "osdmap_epoch_barrier": 39, "uptime": 17.205895272999999 } 2021-02-10T11:22:36.670 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2021-02-10T11:22:36.671 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2021-02-10T11:22:36.671 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@gibba002.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2021-02-10T11:22:36.671 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:22:36.671 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec mkdir -p /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:22:37.007 DEBUG:teuthology.orchestra.run.gibba002:> sudo modprobe fuse 2021-02-10T11:22:37.350 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:22:37.351 DEBUG:teuthology.orchestra.run.gibba002:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:22:37.694 INFO:teuthology.orchestra.run.gibba002.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:22:37.694 INFO:teuthology.orchestra.run.gibba002.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:22:37.695 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:22:37.695 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:22:37.695 DEBUG:teuthology.orchestra.run.gibba002:> ls /sys/fs/fuse/connections 2021-02-10T11:22:38.030 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2021-02-10T11:22:38.030 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:22:38.033 DEBUG:teuthology.orchestra.run.gibba002:> sudo modprobe fuse 2021-02-10T11:22:38.375 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:22:38.376 DEBUG:teuthology.orchestra.run.gibba002:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:22:38.443 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba002.stderr:2021-02-10 11:22:38.446 7f16c684cf80 -1 init, newargv = 0x55a0e97be110 newargc=9 2021-02-10T11:22:38.443 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba002.stderr:ceph-fuse[145313]: starting ceph client 2021-02-10T11:22:38.449 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba002.stderr:ceph-fuse[145313]: starting fuse 2021-02-10T11:22:38.719 INFO:teuthology.orchestra.run.gibba002.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:22:38.719 INFO:teuthology.orchestra.run.gibba002.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:22:38.720 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:22:38.720 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:22:38.721 DEBUG:teuthology.orchestra.run.gibba002:> ls /sys/fs/fuse/connections 2021-02-10T11:22:39.054 INFO:teuthology.orchestra.run.gibba002.stdout:43 2021-02-10T11:22:39.054 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2021-02-10T11:22:39.055 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:22:39.055 DEBUG:teuthology.orchestra.run.gibba002:> sudo python3 -c ' 2021-02-10T11:22:39.055 DEBUG:teuthology.orchestra.run.gibba002:> import glob 2021-02-10T11:22:39.055 DEBUG:teuthology.orchestra.run.gibba002:> import re 2021-02-10T11:22:39.055 DEBUG:teuthology.orchestra.run.gibba002:> import os 2021-02-10T11:22:39.055 DEBUG:teuthology.orchestra.run.gibba002:> import subprocess 2021-02-10T11:22:39.055 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:22:39.055 DEBUG:teuthology.orchestra.run.gibba002:> def find_socket(client_name): 2021-02-10T11:22:39.055 DEBUG:teuthology.orchestra.run.gibba002:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2021-02-10T11:22:39.055 DEBUG:teuthology.orchestra.run.gibba002:> files = glob.glob(asok_path) 2021-02-10T11:22:39.055 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:22:39.056 DEBUG:teuthology.orchestra.run.gibba002:> # Given a non-glob path, it better be there 2021-02-10T11:22:39.056 DEBUG:teuthology.orchestra.run.gibba002:> if "*" not in asok_path: 2021-02-10T11:22:39.056 DEBUG:teuthology.orchestra.run.gibba002:> assert(len(files) == 1) 2021-02-10T11:22:39.056 DEBUG:teuthology.orchestra.run.gibba002:> return files[0] 2021-02-10T11:22:39.056 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:22:39.056 DEBUG:teuthology.orchestra.run.gibba002:> for f in files: 2021-02-10T11:22:39.056 DEBUG:teuthology.orchestra.run.gibba002:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2021-02-10T11:22:39.056 DEBUG:teuthology.orchestra.run.gibba002:> if os.path.exists("/proc/{0}".format(pid)): 2021-02-10T11:22:39.056 DEBUG:teuthology.orchestra.run.gibba002:> return f 2021-02-10T11:22:39.056 DEBUG:teuthology.orchestra.run.gibba002:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2021-02-10T11:22:39.056 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:22:39.056 DEBUG:teuthology.orchestra.run.gibba002:> print(find_socket("client.0")) 2021-02-10T11:22:39.056 DEBUG:teuthology.orchestra.run.gibba002:> ' 2021-02-10T11:22:39.417 INFO:teuthology.orchestra.run.gibba002.stdout:/var/run/ceph/ceph-client.0.145313.asok 2021-02-10T11:22:39.420 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.145313.asok 2021-02-10T11:22:39.420 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:22:39.421 DEBUG:teuthology.orchestra.run.gibba002:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.145313.asok status 2021-02-10T11:22:39.810 INFO:teuthology.orchestra.run.gibba002.stdout:{ 2021-02-10T11:22:39.810 INFO:teuthology.orchestra.run.gibba002.stdout: "metadata": { 2021-02-10T11:22:39.810 INFO:teuthology.orchestra.run.gibba002.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:22:39.810 INFO:teuthology.orchestra.run.gibba002.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:22:39.810 INFO:teuthology.orchestra.run.gibba002.stdout: "entity_id": "0", 2021-02-10T11:22:39.810 INFO:teuthology.orchestra.run.gibba002.stdout: "hostname": "gibba002", 2021-02-10T11:22:39.810 INFO:teuthology.orchestra.run.gibba002.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2021-02-10T11:22:39.810 INFO:teuthology.orchestra.run.gibba002.stdout: "pid": "145313", 2021-02-10T11:22:39.811 INFO:teuthology.orchestra.run.gibba002.stdout: "root": "/" 2021-02-10T11:22:39.811 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:22:39.811 INFO:teuthology.orchestra.run.gibba002.stdout: "dentry_count": 0, 2021-02-10T11:22:39.811 INFO:teuthology.orchestra.run.gibba002.stdout: "dentry_pinned_count": 0, 2021-02-10T11:22:39.811 INFO:teuthology.orchestra.run.gibba002.stdout: "id": 7359, 2021-02-10T11:22:39.811 INFO:teuthology.orchestra.run.gibba002.stdout: "inst": { 2021-02-10T11:22:39.811 INFO:teuthology.orchestra.run.gibba002.stdout: "name": { 2021-02-10T11:22:39.811 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "client", 2021-02-10T11:22:39.811 INFO:teuthology.orchestra.run.gibba002.stdout: "num": 7359 2021-02-10T11:22:39.811 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:22:39.811 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": { 2021-02-10T11:22:39.811 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:22:39.811 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:0", 2021-02-10T11:22:39.812 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 937204636 2021-02-10T11:22:39.812 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:22:39.812 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:22:39.812 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": { 2021-02-10T11:22:39.812 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:22:39.812 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:0", 2021-02-10T11:22:39.812 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 937204636 2021-02-10T11:22:39.812 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:22:39.812 INFO:teuthology.orchestra.run.gibba002.stdout: "inst_str": "client.7359 172.21.2.102:0/937204636", 2021-02-10T11:22:39.812 INFO:teuthology.orchestra.run.gibba002.stdout: "addr_str": "172.21.2.102:0/937204636", 2021-02-10T11:22:39.812 INFO:teuthology.orchestra.run.gibba002.stdout: "inode_count": 1, 2021-02-10T11:22:39.812 INFO:teuthology.orchestra.run.gibba002.stdout: "mds_epoch": 21, 2021-02-10T11:22:39.812 INFO:teuthology.orchestra.run.gibba002.stdout: "osd_epoch": 39, 2021-02-10T11:22:39.813 INFO:teuthology.orchestra.run.gibba002.stdout: "osd_epoch_barrier": 0, 2021-02-10T11:22:39.813 INFO:teuthology.orchestra.run.gibba002.stdout: "blacklisted": false 2021-02-10T11:22:39.813 INFO:teuthology.orchestra.run.gibba002.stdout:} 2021-02-10T11:22:39.814 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:22:39.814 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:22:39.814 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:22:40.149 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:22:40.149 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:22:40.149 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:22:40.149 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:22:40.495 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:22:40.921 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:40.907 7f6a4dab7700 1 Processor -- start 2021-02-10T11:22:40.922 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:40.907 7f6a4dab7700 1 -- start start 2021-02-10T11:22:40.922 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:40.908 7f6a4dab7700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f6a480e5070 0x7f6a481b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:40.922 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:40.908 7f6a4dab7700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a4807ed80 con 0x7f6a480e5470 2021-02-10T11:22:40.922 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:40.908 7f6a4dab7700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a4807e670 con 0x7f6a481b3e40 2021-02-10T11:22:40.922 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:40.908 7f6a4dab7700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a4807e890 con 0x7f6a480e5070 2021-02-10T11:22:40.922 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:40.908 7f6a47fff700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f6a480e5470 0x7f6a4807de80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:47880/0 (socket says 172.21.2.102:47880) 2021-02-10T11:22:40.922 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:40.908 7f6a47fff700 1 -- 172.21.2.102:0/556877659 learned_addr learned my addr 172.21.2.102:0/556877659 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:22:40.922 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:40.908 7f6a467fc700 1 -- 172.21.2.102:0/556877659 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1307633815 0 0) 0x7f6a40001500 con 0x7f6a480e5470 2021-02-10T11:22:40.922 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:40.908 7f6a467fc700 1 -- 172.21.2.102:0/556877659 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a34002fe0 con 0x7f6a480e5470 2021-02-10T11:22:40.922 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:40.908 7f6a467fc700 1 -- 172.21.2.102:0/556877659 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2841195560 0 0) 0x7f6a40001460 con 0x7f6a480e5470 2021-02-10T11:22:40.923 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:40.908 7f6a467fc700 1 -- 172.21.2.102:0/556877659 >> v2:172.21.2.113:3300/0 conn(0x7f6a480e5070 msgr2=0x7f6a481b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:40.923 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:40.908 7f6a467fc700 1 --2- 172.21.2.102:0/556877659 >> v2:172.21.2.113:3300/0 conn(0x7f6a480e5070 0x7f6a481b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:40.923 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:40.908 7f6a467fc700 1 -- 172.21.2.102:0/556877659 >> v1:172.21.2.113:6789/0 conn(0x7f6a481b3e40 legacy=0x7f6a4807d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:40.923 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:40.908 7f6a467fc700 1 -- 172.21.2.102:0/556877659 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a4807eab0 con 0x7f6a480e5470 2021-02-10T11:22:40.923 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:40.908 7f6a467fc700 1 -- 172.21.2.102:0/556877659 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7f6a40001980 con 0x7f6a480e5470 2021-02-10T11:22:40.923 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:40.908 7f6a467fc700 1 -- 172.21.2.102:0/556877659 >> v1:172.21.2.102:6789/0 conn(0x7f6a480e5470 legacy=0x7f6a4807de80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:40.923 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:40.908 7f6a467fc700 1 --2- 172.21.2.102:0/556877659 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6a34003ac0 0x7f6a34005f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:40.923 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:40.908 7f6a467fc700 1 --2- 172.21.2.102:0/556877659 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6a34006780 0x7f6a34008ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:40.923 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:40.909 7f6a467fc700 1 --2- 172.21.2.102:0/556877659 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6a34009580 0x7f6a3400baa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:40.923 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:40.909 7f6a467fc700 1 -- 172.21.2.102:0/556877659 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a3400c2f0 con 0x7f6a34009580 2021-02-10T11:22:40.923 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:40.909 7f6a467fc700 1 -- 172.21.2.102:0/556877659 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a3400c510 con 0x7f6a34006780 2021-02-10T11:22:40.923 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:40.909 7f6a467fc700 1 -- 172.21.2.102:0/556877659 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a3400c730 con 0x7f6a34003ac0 2021-02-10T11:22:40.923 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:40.909 7f6a47fff700 1 --2- 172.21.2.102:0/556877659 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6a34009580 0x7f6a3400baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:40.924 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:40.909 7f6a46ffd700 1 --2- 172.21.2.102:0/556877659 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6a34006780 0x7f6a34008ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:40.924 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:40.909 7f6a477fe700 1 --2- 172.21.2.102:0/556877659 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6a34003ac0 0x7f6a34005f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:40.924 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:40.909 7f6a47fff700 1 -- 172.21.2.102:0/556877659 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6a34003ac0 msgr2=0x7f6a34005f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:40.924 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:40.909 7f6a47fff700 1 --2- 172.21.2.102:0/556877659 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6a34003ac0 0x7f6a34005f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:40.924 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:40.909 7f6a47fff700 1 -- 172.21.2.102:0/556877659 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6a34006780 msgr2=0x7f6a34008ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:40.924 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:40.909 7f6a47fff700 1 --2- 172.21.2.102:0/556877659 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6a34006780 0x7f6a34008ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:40.924 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:40.909 7f6a47fff700 1 -- 172.21.2.102:0/556877659 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a3400c9d0 con 0x7f6a34009580 2021-02-10T11:22:40.924 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:40.909 7f6a47fff700 1 -- 172.21.2.102:0/556877659 reap_dead start 2021-02-10T11:22:40.924 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:40.909 7f6a47fff700 1 --2- 172.21.2.102:0/556877659 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6a34009580 0x7f6a3400baa0 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 rx=0x7f6a40001160 tx=0x7f6a4004e320).ready entity=mon.0 client_cookie=80aa32a9c921b104 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:40.924 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:40.909 7f6a467fc700 1 -- 172.21.2.102:0/556877659 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6a40001460 con 0x7f6a34009580 2021-02-10T11:22:40.924 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:40.909 7f6a467fc700 1 -- 172.21.2.102:0/556877659 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a400070c0 con 0x7f6a34009580 2021-02-10T11:22:40.924 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:40.909 7f6a467fc700 1 -- 172.21.2.102:0/556877659 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6a40002070 con 0x7f6a34009580 2021-02-10T11:22:40.924 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:40.909 7f6a4dab7700 1 -- 172.21.2.102:0/556877659 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6a34009580 msgr2=0x7f6a3400baa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:40.925 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:40.909 7f6a4dab7700 1 --2- 172.21.2.102:0/556877659 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6a34009580 0x7f6a3400baa0 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 rx=0x7f6a40001160 tx=0x7f6a4004e320).stop 2021-02-10T11:22:40.925 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:40.910 7f6a4dab7700 1 -- 172.21.2.102:0/556877659 shutdown_connections 2021-02-10T11:22:40.925 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:40.910 7f6a4dab7700 1 --2- 172.21.2.102:0/556877659 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6a34009580 0x7f6a3400baa0 unknown :-1 s=CLOSED pgs=1101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:40.925 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:40.910 7f6a4dab7700 1 -- 172.21.2.102:0/556877659 >> 172.21.2.102:0/556877659 conn(0x7f6a480ce690 msgr2=0x7f6a480e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:40.925 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:40.910 7f6a4dab7700 1 -- 172.21.2.102:0/556877659 shutdown_connections 2021-02-10T11:22:40.925 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:40.910 7f6a4dab7700 1 -- 172.21.2.102:0/556877659 wait complete. 2021-02-10T11:22:40.925 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:40.925 7f6a4dab7700 1 Processor -- start 2021-02-10T11:22:40.926 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:40.925 7f6a4dab7700 1 -- start start 2021-02-10T11:22:40.926 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:40.925 7f6a4dab7700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6a48194000 0x7f6a480b0d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:40.927 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:40.925 7f6a4dab7700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6a480b1330 0x7f6a48087050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:40.927 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:40.925 7f6a4dab7700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6a48087670 0x7f6a48087a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:40.927 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:40.925 7f6a4dab7700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a4809c0e0 con 0x7f6a480b1330 2021-02-10T11:22:40.927 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:40.925 7f6a4dab7700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a48194440 con 0x7f6a48194000 2021-02-10T11:22:40.927 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:40.925 7f6a4dab7700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a4809c5e0 con 0x7f6a48087670 2021-02-10T11:22:40.927 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:40.925 7f6a46ffd700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6a480b1330 0x7f6a48087050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:40.927 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:40.925 7f6a46ffd700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6a480b1330 0x7f6a48087050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:55670/0 (socket says 172.21.2.102:55670) 2021-02-10T11:22:40.927 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:40.925 7f6a46ffd700 1 -- 172.21.2.102:0/801948422 learned_addr learned my addr 172.21.2.102:0/801948422 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:22:40.927 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:40.925 7f6a46ffd700 1 -- 172.21.2.102:0/801948422 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6a48087670 msgr2=0x7f6a48087a70 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:22:40.927 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:40.925 7f6a46ffd700 1 --2- 172.21.2.102:0/801948422 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6a48087670 0x7f6a48087a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:40.927 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:40.925 7f6a46ffd700 1 -- 172.21.2.102:0/801948422 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6a48194000 msgr2=0x7f6a480b0d10 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:22:40.927 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:40.925 7f6a46ffd700 1 --2- 172.21.2.102:0/801948422 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6a48194000 0x7f6a480b0d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:40.927 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:40.926 7f6a46ffd700 1 -- 172.21.2.102:0/801948422 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a4809c7d0 con 0x7f6a480b1330 2021-02-10T11:22:40.928 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:40.926 7f6a46ffd700 1 --2- 172.21.2.102:0/801948422 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6a480b1330 0x7f6a48087050 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 rx=0x7f6a3000b5e0 tx=0x7f6a3000b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:40.928 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:40.926 7f6a3ffff700 1 -- 172.21.2.102:0/801948422 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6a3000bd70 con 0x7f6a480b1330 2021-02-10T11:22:40.928 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:40.926 7f6a3ffff700 1 -- 172.21.2.102:0/801948422 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a300052e0 con 0x7f6a480b1330 2021-02-10T11:22:40.928 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:40.926 7f6a3ffff700 1 -- 172.21.2.102:0/801948422 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6a30005800 con 0x7f6a480b1330 2021-02-10T11:22:40.928 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:40.926 7f6a4dab7700 1 -- 172.21.2.102:0/801948422 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a4809c7d0 con 0x7f6a480b1330 2021-02-10T11:22:40.928 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:40.926 7f6a4dab7700 1 -- 172.21.2.102:0/801948422 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a4809cd60 con 0x7f6a480b1330 2021-02-10T11:22:40.928 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:40.926 7f6a3ffff700 1 -- 172.21.2.102:0/801948422 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f6a300054c0 con 0x7f6a480b1330 2021-02-10T11:22:40.928 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:40.927 7f6a3ffff700 1 --2- 172.21.2.102:0/801948422 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6a2c0186c0 0x7f6a2c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:40.928 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:40.927 7f6a3ffff700 1 -- 172.21.2.102:0/801948422 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f6a300325a0 con 0x7f6a480b1330 2021-02-10T11:22:40.928 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:40.927 7f6a477fe700 1 --2- 172.21.2.102:0/801948422 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6a2c0186c0 0x7f6a2c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:40.928 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:40.927 7f6a477fe700 1 --2- 172.21.2.102:0/801948422 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6a2c0186c0 0x7f6a2c01abc0 crc :-1 s=READY pgs=528 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:40.955 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:40.958 7f6a4dab7700 1 -- 172.21.2.102:0/801948422 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a48084b20 con 0x7f6a480b1330 2021-02-10T11:22:40.957 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:40.960 7f6a3ffff700 1 -- 172.21.2.102:0/801948422 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f6a300276e0 con 0x7f6a480b1330 2021-02-10T11:22:41.082 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:41.085 7f6a4dab7700 1 -- 172.21.2.102:0/801948422 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6a4807ed80 con 0x7f6a480b1330 2021-02-10T11:22:41.082 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:41.085 7f6a3ffff700 1 -- 172.21.2.102:0/801948422 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v39) v1 ==== 74+0+13993 (secure 0 0 0) 0x7f6a300276e0 con 0x7f6a480b1330 2021-02-10T11:22:41.083 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:22:41.083 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":39,"fsid":"68db30ea-dbce-4aef-a6df-f01c6fea0513","created":"2021-02-10 11:03:06.331557","modified":"2021-02-10 11:22:27.123992","last_up_change":"2021-02-10 11:03:32.690562","last_in_change":"2021-02-10 11:03:32.690562","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:03:37.322519","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:22:23.578113","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":7,"pool_name":"cephfs_data","create_time":"2021-02-10 11:22:25.111303","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"be691b4d-a039-408a-af80-9d53f02b6f1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6818","nonce":36088},{"type":"v1","addr":"172.21.2.102:6820","nonce":36088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6822","nonce":36088},{"type":"v1","addr":"172.21.2.102:6824","nonce":36088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6830","nonce":36088},{"type":"v1","addr":"172.21.2.102:6832","nonce":36088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6826","nonce":36088},{"type":"v1","addr":"172.21.2.102:6828","nonce":36088}]},"public_addr":"172.21.2.102:6820/36088","cluster_addr":"172.21.2.102:6824/36088","heartbeat_back_addr":"172.21.2.102:6832/36088","heartbeat_front_addr":"172.21.2.102:6828/36088","state":["exists","up"]},{"osd":1,"uuid":"64e3ccc8-9c69-4d7c-865e-af8e83822635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6819","nonce":36087},{"type":"v1","addr":"172.21.2.102:6821","nonce":36087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6823","nonce":36087},{"type":"v1","addr":"172.21.2.102:6825","nonce":36087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6831","nonce":36087},{"type":"v1","addr":"172.21.2.102:6833","nonce":36087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6827","nonce":36087},{"type":"v1","addr":"172.21.2.102:6829","nonce":36087}]},"public_addr":"172.21.2.102:6821/36087","cluster_addr":"172.21.2.102:6825/36087","heartbeat_back_addr":"172.21.2.102:6833/36087","heartbeat_front_addr":"172.21.2.102:6829/36087","state":["exists","up"]},{"osd":2,"uuid":"153700af-747f-481f-bdf4-d6072d3a842f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6810","nonce":36089},{"type":"v1","addr":"172.21.2.102:6811","nonce":36089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6812","nonce":36089},{"type":"v1","addr":"172.21.2.102:6813","nonce":36089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6816","nonce":36089},{"type":"v1","addr":"172.21.2.102:6817","nonce":36089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6814","nonce":36089},{"type":"v1","addr":"172.21.2.102:6815","nonce":36089}]},"public_addr":"172.21.2.102:6811/36089","cluster_addr":"172.21.2.102:6813/36089","heartbeat_back_addr":"172.21.2.102:6817/36089","heartbeat_front_addr":"172.21.2.102:6815/36089","state":["exists","up"]},{"osd":3,"uuid":"cbc678f4-83a9-45e6-8830-2f7c232b1ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6802","nonce":36090},{"type":"v1","addr":"172.21.2.102:6803","nonce":36090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6804","nonce":36090},{"type":"v1","addr":"172.21.2.102:6805","nonce":36090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6808","nonce":36090},{"type":"v1","addr":"172.21.2.102:6809","nonce":36090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6806","nonce":36090},{"type":"v1","addr":"172.21.2.102:6807","nonce":36090}]},"public_addr":"172.21.2.102:6803/36090","cluster_addr":"172.21.2.102:6805/36090","heartbeat_back_addr":"172.21.2.102:6809/36090","heartbeat_front_addr":"172.21.2.102:6807/36090","state":["exists","up"]},{"osd":4,"uuid":"c01e4008-b258-4f10-afd2-9e7f37214351","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6808","nonce":36368},{"type":"v1","addr":"172.21.2.113:6811","nonce":36368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6814","nonce":36368},{"type":"v1","addr":"172.21.2.113:6818","nonce":36368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6826","nonce":36368},{"type":"v1","addr":"172.21.2.113:6828","nonce":36368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6820","nonce":36368},{"type":"v1","addr":"172.21.2.113:6824","nonce":36368}]},"public_addr":"172.21.2.113:6811/36368","cluster_addr":"172.21.2.113:6818/36368","heartbeat_back_addr":"172.21.2.113:6828/36368","heartbeat_front_addr":"172.21.2.113:6824/36368","state":["exists","up"]},{"osd":5,"uuid":"551f6007-daf8-4a98-8ee0-c2b16bcccb5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6802","nonce":36366},{"type":"v1","addr":"172.21.2.113:6805","nonce":36366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6807","nonce":36366},{"type":"v1","addr":"172.21.2.113:6810","nonce":36366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6819","nonce":36366},{"type":"v1","addr":"172.21.2.113:6822","nonce":36366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6812","nonce":36366},{"type":"v1","addr":"172.21.2.113:6815","nonce":36366}]},"public_addr":"172.21.2.113:6805/36366","cluster_addr":"172.21.2.113:6810/36366","heartbeat_back_addr":"172.21.2.113:6822/36366","heartbeat_front_addr":"172.21.2.113:6815/36366","state":["exists","up"]},{"osd":6,"uuid":"bf67c040-0e50-48be-b102-c3943bc9ebef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6800","nonce":36367},{"type":"v1","addr":"172.21.2.113:6801","nonce":36367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6803","nonce":36367},{"type":"v1","addr":"172.21.2.113:6804","nonce":36367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6813","nonce":36367},{"type":"v1","addr":"172.21.2.113:6816","nonce":36367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6806","nonce":36367},{"type":"v1","addr":"172.21.2.113:6809","nonce":36367}]},"public_addr":"172.21.2.113:6801/36367","cluster_addr":"172.21.2.113:6804/36367","heartbeat_back_addr":"172.21.2.113:6816/36367","heartbeat_front_addr":"172.21.2.113:6809/36367","state":["exists","up"]},{"osd":7,"uuid":"f5461f80-018d-428f-be89-2eb2bcbe7bb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6817","nonce":36365},{"type":"v1","addr":"172.21.2.113:6821","nonce":36365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6823","nonce":36365},{"type":"v1","addr":"172.21.2.113:6825","nonce":36365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6830","nonce":36365},{"type":"v1","addr":"172.21.2.113:6831","nonce":36365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6827","nonce":36365},{"type":"v1","addr":"172.21.2.113:6829","nonce":36365}]},"public_addr":"172.21.2.113:6821/36365","cluster_addr":"172.21.2.113:6825/36365","heartbeat_back_addr":"172.21.2.113:6831/36365","heartbeat_front_addr":"172.21.2.113:6829/36365","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:22:41.084 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:41.087 7f6a4dab7700 1 -- 172.21.2.102:0/801948422 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6a2c0186c0 msgr2=0x7f6a2c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:41.085 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:41.087 7f6a4dab7700 1 --2- 172.21.2.102:0/801948422 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6a2c0186c0 0x7f6a2c01abc0 crc :-1 s=READY pgs=528 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:41.085 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:41.087 7f6a4dab7700 1 -- 172.21.2.102:0/801948422 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6a480b1330 msgr2=0x7f6a48087050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:41.085 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:41.087 7f6a4dab7700 1 --2- 172.21.2.102:0/801948422 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6a480b1330 0x7f6a48087050 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 rx=0x7f6a3000b5e0 tx=0x7f6a3000b980).stop 2021-02-10T11:22:41.085 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:41.087 7f6a4dab7700 1 -- 172.21.2.102:0/801948422 shutdown_connections 2021-02-10T11:22:41.085 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:41.087 7f6a4dab7700 1 --2- 172.21.2.102:0/801948422 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6a48087670 0x7f6a48087a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:41.085 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:41.087 7f6a4dab7700 1 --2- 172.21.2.102:0/801948422 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6a480b1330 0x7f6a48087050 unknown :-1 s=CLOSED pgs=1102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:41.085 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:41.087 7f6a4dab7700 1 --2- 172.21.2.102:0/801948422 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6a2c0186c0 0x7f6a2c01abc0 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:41.085 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:41.087 7f6a4dab7700 1 --2- 172.21.2.102:0/801948422 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6a48194000 0x7f6a480b0d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:41.085 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:41.087 7f6a4dab7700 1 -- 172.21.2.102:0/801948422 >> 172.21.2.102:0/801948422 conn(0x7f6a48092c20 msgr2=0x7f6a48195070 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:41.085 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:41.088 7f6a4dab7700 1 -- 172.21.2.102:0/801948422 shutdown_connections 2021-02-10T11:22:41.086 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:41.088 7f6a4dab7700 1 -- 172.21.2.102:0/801948422 wait complete. 2021-02-10T11:22:41.093 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:03:37.322519', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 6, 'pool_name': 'cephfs_metadata', 'create_time': '2021-02-10 11:22:23.578113', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '38', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 7, 'pool_name': 'cephfs_data', 'create_time': '2021-02-10 11:22:25.111303', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '39', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-02-10T11:22:41.093 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:22:41.524 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:41.511 7f34c44be700 1 Processor -- start 2021-02-10T11:22:41.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:41.511 7f34c44be700 1 -- start start 2021-02-10T11:22:41.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:41.512 7f34c44be700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f34bc1b0470 0x7f34bc1b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:41.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:41.512 7f34c44be700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f34bc0ef610 0x7f34bc0efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:41.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:41.512 7f34c44be700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f34bc0f0030 0x7f34bc0f0430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:41.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:41.512 7f34c44be700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34bc0e5ea0 con 0x7f34bc1b0470 2021-02-10T11:22:41.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:41.512 7f34c44be700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34bc0f0b50 con 0x7f34bc0f0030 2021-02-10T11:22:41.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:41.512 7f34c44be700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34bc0e52e0 con 0x7f34bc0ef610 2021-02-10T11:22:41.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:41.512 7f34c22da700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f34bc1b0470 0x7f34bc1b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:41.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:41.512 7f34c22da700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f34bc1b0470 0x7f34bc1b0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:55674/0 (socket says 172.21.2.102:55674) 2021-02-10T11:22:41.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:41.512 7f34c22da700 1 -- 172.21.2.102:0/4114316071 learned_addr learned my addr 172.21.2.102:0/4114316071 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:22:41.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:41.512 7f34c1ad9700 1 --2- 172.21.2.102:0/4114316071 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f34bc0ef610 0x7f34bc0efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:41.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:41.512 7f34c2adb700 1 --2- 172.21.2.102:0/4114316071 >> v2:172.21.2.113:3300/0 conn(0x7f34bc0f0030 0x7f34bc0f0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:41.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:41.512 7f34c22da700 1 -- 172.21.2.102:0/4114316071 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f34bc0ef610 msgr2=0x7f34bc0efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:41.526 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:41.512 7f34c22da700 1 --2- 172.21.2.102:0/4114316071 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f34bc0ef610 0x7f34bc0efa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:41.526 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:41.512 7f34c22da700 1 -- 172.21.2.102:0/4114316071 >> v2:172.21.2.113:3300/0 conn(0x7f34bc0f0030 msgr2=0x7f34bc0f0430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:41.526 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:41.512 7f34c22da700 1 --2- 172.21.2.102:0/4114316071 >> v2:172.21.2.113:3300/0 conn(0x7f34bc0f0030 0x7f34bc0f0430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:41.526 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:41.512 7f34c22da700 1 -- 172.21.2.102:0/4114316071 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34bc0f10d0 con 0x7f34bc1b0470 2021-02-10T11:22:41.526 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:41.512 7f34c2adb700 1 --2- 172.21.2.102:0/4114316071 >> v2:172.21.2.113:3300/0 conn(0x7f34bc0f0030 0x7f34bc0f0430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:22:41.526 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:41.512 7f34c1ad9700 1 --2- 172.21.2.102:0/4114316071 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f34bc0ef610 0x7f34bc0efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:22:41.526 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:41.512 7f34c22da700 1 --2- 172.21.2.102:0/4114316071 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f34bc1b0470 0x7f34bc1b0870 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 rx=0x7f34b0006700 tx=0x7f34b0009b80).ready entity=mon.0 client_cookie=dc2c7e1869666eea server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:41.526 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:41.512 7f34c12d8700 1 -- 172.21.2.102:0/4114316071 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f34b0006340 con 0x7f34bc1b0470 2021-02-10T11:22:41.526 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:41.513 7f34c12d8700 1 -- 172.21.2.102:0/4114316071 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34b000c0c0 con 0x7f34bc1b0470 2021-02-10T11:22:41.526 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:41.513 7f34c12d8700 1 -- 172.21.2.102:0/4114316071 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f34b000c510 con 0x7f34bc1b0470 2021-02-10T11:22:41.526 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:41.513 7f34c44be700 1 -- 172.21.2.102:0/4114316071 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f34bc1b0470 msgr2=0x7f34bc1b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:41.526 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:41.513 7f34c44be700 1 --2- 172.21.2.102:0/4114316071 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f34bc1b0470 0x7f34bc1b0870 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 rx=0x7f34b0006700 tx=0x7f34b0009b80).stop 2021-02-10T11:22:41.527 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:41.513 7f34c44be700 1 -- 172.21.2.102:0/4114316071 shutdown_connections 2021-02-10T11:22:41.527 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:41.513 7f34c44be700 1 --2- 172.21.2.102:0/4114316071 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f34bc0ef610 0x7f34bc0efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:41.527 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:41.513 7f34c44be700 1 --2- 172.21.2.102:0/4114316071 >> v2:172.21.2.113:3300/0 conn(0x7f34bc0f0030 0x7f34bc0f0430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:41.527 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:41.513 7f34c44be700 1 --2- 172.21.2.102:0/4114316071 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f34bc1b0470 0x7f34bc1b0870 unknown :-1 s=CLOSED pgs=1103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:41.527 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:41.513 7f34c44be700 1 -- 172.21.2.102:0/4114316071 >> 172.21.2.102:0/4114316071 conn(0x7f34bc0ce3e0 msgr2=0x7f34bc0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:41.527 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:41.514 7f34c44be700 1 -- 172.21.2.102:0/4114316071 shutdown_connections 2021-02-10T11:22:41.527 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:41.514 7f34c44be700 1 -- 172.21.2.102:0/4114316071 wait complete. 2021-02-10T11:22:41.528 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:41.528 7f34c44be700 1 Processor -- start 2021-02-10T11:22:41.528 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:41.528 7f34c44be700 1 -- start start 2021-02-10T11:22:41.528 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:41.528 7f34c44be700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f34bc0b7de0 0x7f34bc190920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:41.529 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:41.528 7f34c44be700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f34bc0b82f0 0x7f34bc0b86f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:41.529 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:41.528 7f34c44be700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f34bc0b8d10 0x7f34bc0bf9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:41.529 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:41.528 7f34c44be700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34bc0b7930 con 0x7f34bc0b8d10 2021-02-10T11:22:41.529 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:41.528 7f34c44be700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34bc0b7b20 con 0x7f34bc0b7de0 2021-02-10T11:22:41.529 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:41.528 7f34c44be700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34bc0bffd0 con 0x7f34bc0b82f0 2021-02-10T11:22:41.529 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:41.528 7f34c2adb700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f34bc0b8d10 0x7f34bc0bf9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:41.529 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:41.528 7f34c2adb700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f34bc0b8d10 0x7f34bc0bf9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:55682/0 (socket says 172.21.2.102:55682) 2021-02-10T11:22:41.529 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:41.528 7f34c2adb700 1 -- 172.21.2.102:0/752635144 learned_addr learned my addr 172.21.2.102:0/752635144 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:22:41.529 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:41.529 7f34c2adb700 1 -- 172.21.2.102:0/752635144 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f34bc0b82f0 msgr2=0x7f34bc0b86f0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:22:41.529 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:41.529 7f34c2adb700 1 --2- 172.21.2.102:0/752635144 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f34bc0b82f0 0x7f34bc0b86f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:41.529 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:41.529 7f34c2adb700 1 -- 172.21.2.102:0/752635144 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f34bc0b7de0 msgr2=0x7f34bc190920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:41.529 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:41.529 7f34c2adb700 1 --2- 172.21.2.102:0/752635144 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f34bc0b7de0 0x7f34bc190920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:41.530 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:41.529 7f34c2adb700 1 -- 172.21.2.102:0/752635144 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34bc0c01c0 con 0x7f34bc0b8d10 2021-02-10T11:22:41.530 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:41.529 7f34c2adb700 1 --2- 172.21.2.102:0/752635144 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f34bc0b8d10 0x7f34bc0bf9b0 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 rx=0x7f34b40099a0 tx=0x7f34b400bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:41.530 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:41.529 7f34aeffd700 1 -- 172.21.2.102:0/752635144 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f34b4006030 con 0x7f34bc0b8d10 2021-02-10T11:22:41.530 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:41.529 7f34aeffd700 1 -- 172.21.2.102:0/752635144 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34b4006330 con 0x7f34bc0b8d10 2021-02-10T11:22:41.530 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:41.529 7f34aeffd700 1 -- 172.21.2.102:0/752635144 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f34b40056b0 con 0x7f34bc0b8d10 2021-02-10T11:22:41.530 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:41.529 7f34c44be700 1 -- 172.21.2.102:0/752635144 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34bc0c01c0 con 0x7f34bc0b8d10 2021-02-10T11:22:41.530 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:41.529 7f34c44be700 1 -- 172.21.2.102:0/752635144 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34bc0c0750 con 0x7f34bc0b8d10 2021-02-10T11:22:41.530 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:41.530 7f34aeffd700 1 -- 172.21.2.102:0/752635144 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f34b4006030 con 0x7f34bc0b8d10 2021-02-10T11:22:41.530 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:41.530 7f34aeffd700 1 --2- 172.21.2.102:0/752635144 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f34a40186c0 0x7f34a401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:41.530 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:41.530 7f34aeffd700 1 -- 172.21.2.102:0/752635144 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f34b4032910 con 0x7f34bc0b8d10 2021-02-10T11:22:41.530 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:41.530 7f34c22da700 1 --2- 172.21.2.102:0/752635144 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f34a40186c0 0x7f34a401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:41.530 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:41.530 7f34c22da700 1 --2- 172.21.2.102:0/752635144 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f34a40186c0 0x7f34a401abc0 crc :-1 s=READY pgs=529 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:41.559 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:41.562 7f34c44be700 1 -- 172.21.2.102:0/752635144 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34bc105b30 con 0x7f34bc0b8d10 2021-02-10T11:22:41.561 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:41.564 7f34aeffd700 1 -- 172.21.2.102:0/752635144 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f34b4020ee0 con 0x7f34bc0b8d10 2021-02-10T11:22:41.690 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:41.693 7f34c44be700 1 -- 172.21.2.102:0/752635144 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f34bc0f0b50 con 0x7f34bc0b8d10 2021-02-10T11:22:41.690 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:41.693 7f34aeffd700 1 -- 172.21.2.102:0/752635144 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v39) v1 ==== 92+0+10 (secure 0 0 0) 0x7f34b4020ee0 con 0x7f34bc0b8d10 2021-02-10T11:22:41.691 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:22:41.692 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:41.695 7f34c44be700 1 -- 172.21.2.102:0/752635144 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f34a40186c0 msgr2=0x7f34a401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:41.693 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:41.695 7f34c44be700 1 --2- 172.21.2.102:0/752635144 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f34a40186c0 0x7f34a401abc0 crc :-1 s=READY pgs=529 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:41.693 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:41.695 7f34c44be700 1 -- 172.21.2.102:0/752635144 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f34bc0b8d10 msgr2=0x7f34bc0bf9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:41.693 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:41.695 7f34c44be700 1 --2- 172.21.2.102:0/752635144 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f34bc0b8d10 0x7f34bc0bf9b0 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 rx=0x7f34b40099a0 tx=0x7f34b400bcf0).stop 2021-02-10T11:22:41.693 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:41.695 7f34c44be700 1 -- 172.21.2.102:0/752635144 shutdown_connections 2021-02-10T11:22:41.693 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:41.695 7f34c44be700 1 --2- 172.21.2.102:0/752635144 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f34bc0b8d10 0x7f34bc0bf9b0 unknown :-1 s=CLOSED pgs=1104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:41.693 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:41.696 7f34c44be700 1 --2- 172.21.2.102:0/752635144 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f34bc0b82f0 0x7f34bc0b86f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:41.693 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:41.696 7f34c44be700 1 --2- 172.21.2.102:0/752635144 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f34a40186c0 0x7f34a401abc0 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:41.693 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:41.696 7f34c44be700 1 --2- 172.21.2.102:0/752635144 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f34bc0b7de0 0x7f34bc190920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:41.693 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:41.696 7f34c44be700 1 -- 172.21.2.102:0/752635144 >> 172.21.2.102:0/752635144 conn(0x7f34bc094e10 msgr2=0x7f34bc0a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:41.693 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:41.696 7f34c44be700 1 -- 172.21.2.102:0/752635144 shutdown_connections 2021-02-10T11:22:41.693 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:41.696 7f34c44be700 1 -- 172.21.2.102:0/752635144 wait complete. 2021-02-10T11:22:41.701 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-02-10T11:22:42.131 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:42.117 7f0f9716a700 1 Processor -- start 2021-02-10T11:22:42.132 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:42.117 7f0f9716a700 1 -- start start 2021-02-10T11:22:42.132 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:42.118 7f0f9716a700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f0f901b0870 0x7f0f900f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:42.132 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:42.118 7f0f9716a700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f0f900efe00 0x7f0f900f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:42.132 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:42.118 7f0f9716a700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f901b0c70 con 0x7f0f901b0470 2021-02-10T11:22:42.132 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:42.118 7f0f9716a700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f900e5ea0 con 0x7f0f900efe00 2021-02-10T11:22:42.132 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:42.118 7f0f9716a700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f900f0b50 con 0x7f0f901b0870 2021-02-10T11:22:42.132 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:42.118 7f0f94f86700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f0f901b0470 0x7f0f900ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:47914/0 (socket says 172.21.2.102:47914) 2021-02-10T11:22:42.132 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:42.118 7f0f94f86700 1 -- 172.21.2.102:0/3204643159 learned_addr learned my addr 172.21.2.102:0/3204643159 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:22:42.132 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:42.118 7f0f95787700 1 --2- 172.21.2.102:0/3204643159 >> v2:172.21.2.102:3300/0 conn(0x7f0f900efe00 0x7f0f900f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:42.132 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:42.118 7f0f8ffff700 1 --2- 172.21.2.102:0/3204643159 >> v2:172.21.2.113:3300/0 conn(0x7f0f901b0870 0x7f0f900f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:42.132 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:42.118 7f0f8ffff700 1 -- 172.21.2.102:0/3204643159 >> v2:172.21.2.102:3300/0 conn(0x7f0f900efe00 msgr2=0x7f0f900f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:42.133 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:42.118 7f0f8ffff700 1 --2- 172.21.2.102:0/3204643159 >> v2:172.21.2.102:3300/0 conn(0x7f0f900efe00 0x7f0f900f0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:42.133 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:42.118 7f0f8ffff700 1 -- 172.21.2.102:0/3204643159 >> v1:172.21.2.102:6789/0 conn(0x7f0f901b0470 legacy=0x7f0f900ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:42.133 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:42.118 7f0f8f7fe700 1 -- 172.21.2.102:0/3204643159 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3221840488 0 0) 0x7f0f780009b0 con 0x7f0f901b0470 2021-02-10T11:22:42.133 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:42.119 7f0f8ffff700 1 -- 172.21.2.102:0/3204643159 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f900f0820 con 0x7f0f901b0870 2021-02-10T11:22:42.133 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:42.119 7f0f8ffff700 1 --2- 172.21.2.102:0/3204643159 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0f901b0870 0x7f0f900f20e0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 rx=0x7f0f80006650 tx=0x7f0f80009a80).ready entity=mon.1 client_cookie=4066d2d7bb9e8173 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:42.133 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:42.119 7f0f8f7fe700 1 -- 172.21.2.102:0/3204643159 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0f8000c0c0 con 0x7f0f901b0870 2021-02-10T11:22:42.133 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:42.119 7f0f8f7fe700 1 -- 172.21.2.102:0/3204643159 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f8000c340 con 0x7f0f901b0870 2021-02-10T11:22:42.133 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:42.119 7f0f8f7fe700 1 -- 172.21.2.102:0/3204643159 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0f8000c860 con 0x7f0f901b0870 2021-02-10T11:22:42.133 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:42.119 7f0f9716a700 1 -- 172.21.2.102:0/3204643159 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0f901b0870 msgr2=0x7f0f900f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:42.133 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:42.119 7f0f9716a700 1 --2- 172.21.2.102:0/3204643159 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0f901b0870 0x7f0f900f20e0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 rx=0x7f0f80006650 tx=0x7f0f80009a80).stop 2021-02-10T11:22:42.133 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:42.120 7f0f9716a700 1 -- 172.21.2.102:0/3204643159 shutdown_connections 2021-02-10T11:22:42.133 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:42.120 7f0f9716a700 1 --2- 172.21.2.102:0/3204643159 >> v2:172.21.2.102:3300/0 conn(0x7f0f900efe00 0x7f0f900f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:42.134 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:42.120 7f0f9716a700 1 --2- 172.21.2.102:0/3204643159 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0f901b0870 0x7f0f900f20e0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:42.134 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:42.120 7f0f9716a700 1 -- 172.21.2.102:0/3204643159 >> 172.21.2.102:0/3204643159 conn(0x7f0f900ce3e0 msgr2=0x7f0f900e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:42.134 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:42.120 7f0f9716a700 1 -- 172.21.2.102:0/3204643159 shutdown_connections 2021-02-10T11:22:42.134 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:42.120 7f0f9716a700 1 -- 172.21.2.102:0/3204643159 wait complete. 2021-02-10T11:22:42.135 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:42.135 7f0f9716a700 1 Processor -- start 2021-02-10T11:22:42.135 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:42.135 7f0f9716a700 1 -- start start 2021-02-10T11:22:42.136 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:42.135 7f0f9716a700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0f90190920 0x7f0f900b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:42.136 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:42.135 7f0f9716a700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0f900b8200 0x7f0f900b8600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:42.136 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:42.135 7f0f9716a700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0f900c1d80 0x7f0f900c2180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:42.136 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:42.135 7f0f9716a700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f90190d20 con 0x7f0f900b8200 2021-02-10T11:22:42.136 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:42.135 7f0f9716a700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f900b8c20 con 0x7f0f90190920 2021-02-10T11:22:42.136 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:42.135 7f0f9716a700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f900c27a0 con 0x7f0f900c1d80 2021-02-10T11:22:42.136 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:42.136 7f0f8ffff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0f900b8200 0x7f0f900b8600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:42.136 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:42.136 7f0f8ffff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0f900b8200 0x7f0f900b8600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:55696/0 (socket says 172.21.2.102:55696) 2021-02-10T11:22:42.136 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:42.136 7f0f8ffff700 1 -- 172.21.2.102:0/2392157722 learned_addr learned my addr 172.21.2.102:0/2392157722 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:22:42.136 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:42.136 7f0f8ffff700 1 -- 172.21.2.102:0/2392157722 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0f900c1d80 msgr2=0x7f0f900c2180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:42.136 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:42.136 7f0f8ffff700 1 --2- 172.21.2.102:0/2392157722 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0f900c1d80 0x7f0f900c2180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:42.136 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:42.136 7f0f8ffff700 1 -- 172.21.2.102:0/2392157722 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0f90190920 msgr2=0x7f0f900b7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:42.136 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:42.136 7f0f8ffff700 1 --2- 172.21.2.102:0/2392157722 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0f90190920 0x7f0f900b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:42.137 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:42.136 7f0f8ffff700 1 -- 172.21.2.102:0/2392157722 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f900c29c0 con 0x7f0f900b8200 2021-02-10T11:22:42.137 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:42.136 7f0f8ffff700 1 --2- 172.21.2.102:0/2392157722 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0f900b8200 0x7f0f900b8600 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 rx=0x7f0f80009630 tx=0x7f0f8000b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:42.137 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:42.136 7f0f8d7fa700 1 -- 172.21.2.102:0/2392157722 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0f800093a0 con 0x7f0f900b8200 2021-02-10T11:22:42.137 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:42.136 7f0f8d7fa700 1 -- 172.21.2.102:0/2392157722 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f8000bd70 con 0x7f0f900b8200 2021-02-10T11:22:42.137 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:42.136 7f0f8d7fa700 1 -- 172.21.2.102:0/2392157722 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0f80005560 con 0x7f0f900b8200 2021-02-10T11:22:42.137 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:42.136 7f0f9716a700 1 -- 172.21.2.102:0/2392157722 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f900c29c0 con 0x7f0f900b8200 2021-02-10T11:22:42.137 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:42.136 7f0f9716a700 1 -- 172.21.2.102:0/2392157722 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f900c29c0 con 0x7f0f900b8200 2021-02-10T11:22:42.137 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:42.137 7f0f8d7fa700 1 -- 172.21.2.102:0/2392157722 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f0f80005770 con 0x7f0f900b8200 2021-02-10T11:22:42.137 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:42.137 7f0f8d7fa700 1 --2- 172.21.2.102:0/2392157722 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0f74018730 0x7f0f7401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:42.137 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:42.137 7f0f8d7fa700 1 -- 172.21.2.102:0/2392157722 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f0f80032630 con 0x7f0f900b8200 2021-02-10T11:22:42.137 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:42.137 7f0f94f86700 1 --2- 172.21.2.102:0/2392157722 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0f74018730 0x7f0f7401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:42.137 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:42.137 7f0f94f86700 1 --2- 172.21.2.102:0/2392157722 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0f74018730 0x7f0f7401ac10 crc :-1 s=READY pgs=530 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:42.165 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:42.168 7f0f9716a700 1 -- 172.21.2.102:0/2392157722 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f90105ae0 con 0x7f0f900b8200 2021-02-10T11:22:42.167 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:42.170 7f0f8d7fa700 1 -- 172.21.2.102:0/2392157722 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f0f80032840 con 0x7f0f900b8200 2021-02-10T11:22:42.296 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:42.299 7f0f9716a700 1 -- 172.21.2.102:0/2392157722 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7f0f900f0b50 con 0x7f0f900b8200 2021-02-10T11:22:42.296 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:42.299 7f0f8d7fa700 1 -- 172.21.2.102:0/2392157722 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v39) v1 ==== 104+0+10 (secure 0 0 0) 0x7f0f8000bd70 con 0x7f0f900b8200 2021-02-10T11:22:42.297 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:22:42.299 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:42.301 7f0f9716a700 1 -- 172.21.2.102:0/2392157722 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0f74018730 msgr2=0x7f0f7401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:42.299 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:42.301 7f0f9716a700 1 --2- 172.21.2.102:0/2392157722 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0f74018730 0x7f0f7401ac10 crc :-1 s=READY pgs=530 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:42.299 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:42.301 7f0f9716a700 1 -- 172.21.2.102:0/2392157722 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0f900b8200 msgr2=0x7f0f900b8600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:42.299 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:42.301 7f0f9716a700 1 --2- 172.21.2.102:0/2392157722 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0f900b8200 0x7f0f900b8600 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 rx=0x7f0f80009630 tx=0x7f0f8000b980).stop 2021-02-10T11:22:42.299 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:42.301 7f0f9716a700 1 -- 172.21.2.102:0/2392157722 shutdown_connections 2021-02-10T11:22:42.299 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:42.301 7f0f9716a700 1 --2- 172.21.2.102:0/2392157722 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0f900c1d80 0x7f0f900c2180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:42.299 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:42.301 7f0f9716a700 1 --2- 172.21.2.102:0/2392157722 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0f900b8200 0x7f0f900b8600 unknown :-1 s=CLOSED pgs=1106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:42.299 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:42.301 7f0f9716a700 1 --2- 172.21.2.102:0/2392157722 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0f74018730 0x7f0f7401ac10 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:42.299 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:42.301 7f0f9716a700 1 --2- 172.21.2.102:0/2392157722 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0f90190920 0x7f0f900b7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:42.299 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:42.301 7f0f9716a700 1 -- 172.21.2.102:0/2392157722 >> 172.21.2.102:0/2392157722 conn(0x7f0f90094e10 msgr2=0x7f0f900a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:42.299 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:42.302 7f0f9716a700 1 -- 172.21.2.102:0/2392157722 shutdown_connections 2021-02-10T11:22:42.299 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:42.302 7f0f9716a700 1 -- 172.21.2.102:0/2392157722 wait complete. 2021-02-10T11:22:42.307 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-02-10T11:22:42.736 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:42.722 7f00aa4a2700 1 Processor -- start 2021-02-10T11:22:42.737 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:42.723 7f00aa4a2700 1 -- start start 2021-02-10T11:22:42.737 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:42.723 7f00aa4a2700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f00a41b0470 0x7f00a41b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:42.737 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:42.723 7f00aa4a2700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f00a40ef610 0x7f00a40efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:42.737 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:42.723 7f00aa4a2700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00a40f0b50 con 0x7f00a40f0030 2021-02-10T11:22:42.737 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:42.723 7f00aa4a2700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00a40e52e0 con 0x7f00a41b0470 2021-02-10T11:22:42.737 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:42.723 7f00aa4a2700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00a40e54d0 con 0x7f00a40ef610 2021-02-10T11:22:42.737 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:42.724 7f00a37fe700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f00a40ef610 0x7f00a40efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:42.737 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:42.724 7f00a3fff700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f00a41b0470 0x7f00a41b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:42.737 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:42.724 7f00a8abf700 1 --1- >> v1:172.21.2.113:6789/0 conn(0x7f00a40f0030 0x7f00a40f10d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.113:6789/0 says I am v1:172.21.2.102:55250/0 (socket says 172.21.2.102:55250) 2021-02-10T11:22:42.737 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:42.724 7f00a8abf700 1 -- 172.21.2.102:0/1549929439 learned_addr learned my addr 172.21.2.102:0/1549929439 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:22:42.737 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:42.724 7f00a37fe700 1 -- 172.21.2.102:0/1549929439 >> v2:172.21.2.113:3300/0 conn(0x7f00a41b0470 msgr2=0x7f00a41b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:42.738 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:42.724 7f00a37fe700 1 --2- 172.21.2.102:0/1549929439 >> v2:172.21.2.113:3300/0 conn(0x7f00a41b0470 0x7f00a41b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:42.738 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:42.724 7f00a37fe700 1 -- 172.21.2.102:0/1549929439 >> v1:172.21.2.113:6789/0 conn(0x7f00a40f0030 legacy=0x7f00a40f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:42.738 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:42.724 7f00a37fe700 1 -- 172.21.2.102:0/1549929439 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00a40f0430 con 0x7f00a40ef610 2021-02-10T11:22:42.738 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:42.724 7f00a3fff700 1 --2- 172.21.2.102:0/1549929439 >> v2:172.21.2.113:3300/0 conn(0x7f00a41b0470 0x7f00a41b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:22:42.738 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:42.724 7f00a37fe700 1 --2- 172.21.2.102:0/1549929439 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f00a40ef610 0x7f00a40efa10 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f0098006650 tx=0x7f0098009a80).ready entity=mon.2 client_cookie=2f4dfd3a15209496 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:42.738 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:42.725 7f00a2ffd700 1 -- 172.21.2.102:0/1549929439 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f00980009b0 con 0x7f00a40ef610 2021-02-10T11:22:42.738 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:42.725 7f00a2ffd700 1 -- 172.21.2.102:0/1549929439 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0098006390 con 0x7f00a40ef610 2021-02-10T11:22:42.738 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:42.725 7f00a2ffd700 1 -- 172.21.2.102:0/1549929439 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f009800c0c0 con 0x7f00a40ef610 2021-02-10T11:22:42.738 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:42.725 7f00aa4a2700 1 -- 172.21.2.102:0/1549929439 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f00a40ef610 msgr2=0x7f00a40efa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:42.738 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:42.725 7f00aa4a2700 1 --2- 172.21.2.102:0/1549929439 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f00a40ef610 0x7f00a40efa10 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f0098006650 tx=0x7f0098009a80).stop 2021-02-10T11:22:42.738 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:42.725 7f00aa4a2700 1 -- 172.21.2.102:0/1549929439 shutdown_connections 2021-02-10T11:22:42.738 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:42.725 7f00aa4a2700 1 --2- 172.21.2.102:0/1549929439 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f00a40ef610 0x7f00a40efa10 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:42.738 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:42.725 7f00aa4a2700 1 --2- 172.21.2.102:0/1549929439 >> v2:172.21.2.113:3300/0 conn(0x7f00a41b0470 0x7f00a41b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:42.739 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:42.725 7f00aa4a2700 1 -- 172.21.2.102:0/1549929439 >> 172.21.2.102:0/1549929439 conn(0x7f00a40ce3e0 msgr2=0x7f00a40e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:42.739 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:42.725 7f00aa4a2700 1 -- 172.21.2.102:0/1549929439 shutdown_connections 2021-02-10T11:22:42.739 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:42.725 7f00aa4a2700 1 -- 172.21.2.102:0/1549929439 wait complete. 2021-02-10T11:22:42.740 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:42.740 7f00aa4a2700 1 Processor -- start 2021-02-10T11:22:42.740 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:42.740 7f00aa4a2700 1 -- start start 2021-02-10T11:22:42.740 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:42.740 7f00aa4a2700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f00a40a0420 0x7f00a40a0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:42.740 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:42.740 7f00aa4a2700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f00a4189d00 0x7f00a418a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:42.741 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:42.740 7f00aa4a2700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f00a418a720 0x7f00a40b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:42.741 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:42.740 7f00aa4a2700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00a4189800 con 0x7f00a40a0420 2021-02-10T11:22:42.741 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:42.740 7f00aa4a2700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00a418ab20 con 0x7f00a418a720 2021-02-10T11:22:42.741 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:42.740 7f00aa4a2700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00a40a1140 con 0x7f00a4189d00 2021-02-10T11:22:42.741 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:42.740 7f00a3fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f00a40a0420 0x7f00a40a0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:42.741 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:42.740 7f00a3fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f00a40a0420 0x7f00a40a0820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:55710/0 (socket says 172.21.2.102:55710) 2021-02-10T11:22:42.741 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:42.740 7f00a3fff700 1 -- 172.21.2.102:0/1243626848 learned_addr learned my addr 172.21.2.102:0/1243626848 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:22:42.741 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:42.741 7f00a3fff700 1 -- 172.21.2.102:0/1243626848 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f00a4189d00 msgr2=0x7f00a418a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:42.741 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:42.741 7f00a3fff700 1 --2- 172.21.2.102:0/1243626848 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f00a4189d00 0x7f00a418a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:42.741 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:42.741 7f00a3fff700 1 -- 172.21.2.102:0/1243626848 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f00a418a720 msgr2=0x7f00a40b7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:42.741 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:42.741 7f00a3fff700 1 --2- 172.21.2.102:0/1243626848 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f00a418a720 0x7f00a40b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:42.741 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:42.741 7f00a3fff700 1 -- 172.21.2.102:0/1243626848 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00a40b8200 con 0x7f00a40a0420 2021-02-10T11:22:42.741 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:42.741 7f00a3fff700 1 --2- 172.21.2.102:0/1243626848 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f00a40a0420 0x7f00a40a0820 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 rx=0x7f0094006f40 tx=0x7f009400b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:42.742 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:42.741 7f00a0ff9700 1 -- 172.21.2.102:0/1243626848 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f009400bd80 con 0x7f00a40a0420 2021-02-10T11:22:42.742 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:42.741 7f00a0ff9700 1 -- 172.21.2.102:0/1243626848 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0094000c90 con 0x7f00a40a0420 2021-02-10T11:22:42.742 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:42.741 7f00aa4a2700 1 -- 172.21.2.102:0/1243626848 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00a40b8200 con 0x7f00a40a0420 2021-02-10T11:22:42.742 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:42.741 7f00a0ff9700 1 -- 172.21.2.102:0/1243626848 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0094002220 con 0x7f00a40a0420 2021-02-10T11:22:42.742 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:42.741 7f00aa4a2700 1 -- 172.21.2.102:0/1243626848 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00a40b8790 con 0x7f00a40a0420 2021-02-10T11:22:42.742 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:42.742 7f00a0ff9700 1 -- 172.21.2.102:0/1243626848 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f009400a000 con 0x7f00a40a0420 2021-02-10T11:22:42.742 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:42.742 7f00a0ff9700 1 --2- 172.21.2.102:0/1243626848 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f00840186c0 0x7f008401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:42.742 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:42.742 7f00a0ff9700 1 -- 172.21.2.102:0/1243626848 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f00940323f0 con 0x7f00a40a0420 2021-02-10T11:22:42.742 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:42.742 7f00a37fe700 1 --2- 172.21.2.102:0/1243626848 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f00840186c0 0x7f008401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:42.742 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:42.742 7f00a37fe700 1 --2- 172.21.2.102:0/1243626848 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f00840186c0 0x7f008401abc0 crc :-1 s=READY pgs=531 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:42.770 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:42.773 7f00aa4a2700 1 -- 172.21.2.102:0/1243626848 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00a4105ae0 con 0x7f00a40a0420 2021-02-10T11:22:42.772 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:42.775 7f00a0ff9700 1 -- 172.21.2.102:0/1243626848 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f009402f240 con 0x7f00a40a0420 2021-02-10T11:22:42.901 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:42.904 7f00aa4a2700 1 -- 172.21.2.102:0/1243626848 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7f00a40f0b50 con 0x7f00a40a0420 2021-02-10T11:22:42.902 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:42.905 7f00a0ff9700 1 -- 172.21.2.102:0/1243626848 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v39) v1 ==== 100+0+10 (secure 0 0 0) 0x7f009402f240 con 0x7f00a40a0420 2021-02-10T11:22:42.902 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:22:42.904 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:42.906 7f00aa4a2700 1 -- 172.21.2.102:0/1243626848 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f00840186c0 msgr2=0x7f008401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:42.904 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:42.906 7f00aa4a2700 1 --2- 172.21.2.102:0/1243626848 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f00840186c0 0x7f008401abc0 crc :-1 s=READY pgs=531 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:42.904 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:42.906 7f00aa4a2700 1 -- 172.21.2.102:0/1243626848 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f00a40a0420 msgr2=0x7f00a40a0820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:42.904 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:42.906 7f00aa4a2700 1 --2- 172.21.2.102:0/1243626848 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f00a40a0420 0x7f00a40a0820 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 rx=0x7f0094006f40 tx=0x7f009400b990).stop 2021-02-10T11:22:42.904 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:42.907 7f00aa4a2700 1 -- 172.21.2.102:0/1243626848 shutdown_connections 2021-02-10T11:22:42.904 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:42.907 7f00aa4a2700 1 --2- 172.21.2.102:0/1243626848 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f00840186c0 0x7f008401abc0 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:42.904 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:42.907 7f00aa4a2700 1 --2- 172.21.2.102:0/1243626848 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f00a418a720 0x7f00a40b7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:42.904 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:42.907 7f00aa4a2700 1 --2- 172.21.2.102:0/1243626848 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f00a4189d00 0x7f00a418a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:42.904 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:42.907 7f00aa4a2700 1 --2- 172.21.2.102:0/1243626848 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f00a40a0420 0x7f00a40a0820 unknown :-1 s=CLOSED pgs=1107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:42.904 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:42.907 7f00aa4a2700 1 -- 172.21.2.102:0/1243626848 >> 172.21.2.102:0/1243626848 conn(0x7f00a4094e10 msgr2=0x7f00a40a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:42.905 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:42.907 7f00aa4a2700 1 -- 172.21.2.102:0/1243626848 shutdown_connections 2021-02-10T11:22:42.905 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:42.907 7f00aa4a2700 1 -- 172.21.2.102:0/1243626848 wait complete. 2021-02-10T11:22:42.912 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-02-10T11:22:42.913 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:22:43.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:43.324 7f83fe0fb700 1 Processor -- start 2021-02-10T11:22:43.338 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:43.324 7f83fe0fb700 1 -- start start 2021-02-10T11:22:43.338 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:43.325 7f83fe0fb700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f83f80e5470 0x7f83f807e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:43.338 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:43.325 7f83fe0fb700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f83f807f300 0x7f83f807f700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:43.338 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:43.325 7f83fe0fb700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83f807ed80 con 0x7f83f80e5070 2021-02-10T11:22:43.338 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:43.325 7f83fe0fb700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83f81b4010 con 0x7f83f80e5470 2021-02-10T11:22:43.338 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:43.325 7f83fe0fb700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83f80d1790 con 0x7f83f807f300 2021-02-10T11:22:43.338 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:43.325 7f83f6ffd700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f83f80e5470 0x7f83f807e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:43.338 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:43.325 7f83f6ffd700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f83f80e5470 0x7f83f807e6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:55714/0 (socket says 172.21.2.102:55714) 2021-02-10T11:22:43.339 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:43.325 7f83f6ffd700 1 -- 172.21.2.102:0/1515363135 learned_addr learned my addr 172.21.2.102:0/1515363135 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:22:43.339 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:43.325 7f83f7fff700 1 --2- 172.21.2.102:0/1515363135 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f83f807f300 0x7f83f807f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:43.339 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:43.325 7f83f6ffd700 1 -- 172.21.2.102:0/1515363135 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f83f807f300 msgr2=0x7f83f807f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:43.339 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:43.325 7f83f6ffd700 1 --2- 172.21.2.102:0/1515363135 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f83f807f300 0x7f83f807f700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:43.339 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:43.325 7f83f6ffd700 1 -- 172.21.2.102:0/1515363135 >> v1:172.21.2.113:6789/0 conn(0x7f83f80e5070 legacy=0x7f83f81b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:43.339 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:43.325 7f83f6ffd700 1 -- 172.21.2.102:0/1515363135 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83f80d19b0 con 0x7f83f80e5470 2021-02-10T11:22:43.339 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:43.325 7f83f7fff700 1 --2- 172.21.2.102:0/1515363135 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f83f807f300 0x7f83f807f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:22:43.339 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:43.325 7f83f6ffd700 1 --2- 172.21.2.102:0/1515363135 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f83f80e5470 0x7f83f807e6a0 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 rx=0x7f83e8006700 tx=0x7f83e8009b80).ready entity=mon.0 client_cookie=4009b38af9af0dd1 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:43.339 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:43.326 7f83f67fc700 1 -- 172.21.2.102:0/1515363135 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f83e800c0c0 con 0x7f83f80e5470 2021-02-10T11:22:43.339 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:43.326 7f83f67fc700 1 -- 172.21.2.102:0/1515363135 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83e800c3c0 con 0x7f83f80e5470 2021-02-10T11:22:43.339 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:43.326 7f83f67fc700 1 -- 172.21.2.102:0/1515363135 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f83e800c8a0 con 0x7f83f80e5470 2021-02-10T11:22:43.339 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:43.326 7f83fe0fb700 1 -- 172.21.2.102:0/1515363135 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f83f80e5470 msgr2=0x7f83f807e6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:43.339 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:43.326 7f83fe0fb700 1 --2- 172.21.2.102:0/1515363135 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f83f80e5470 0x7f83f807e6a0 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 rx=0x7f83e8006700 tx=0x7f83e8009b80).stop 2021-02-10T11:22:43.339 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:43.326 7f83fe0fb700 1 -- 172.21.2.102:0/1515363135 shutdown_connections 2021-02-10T11:22:43.340 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:43.326 7f83fe0fb700 1 --2- 172.21.2.102:0/1515363135 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f83f807f300 0x7f83f807f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:43.340 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:43.326 7f83fe0fb700 1 --2- 172.21.2.102:0/1515363135 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f83f80e5470 0x7f83f807e6a0 unknown :-1 s=CLOSED pgs=1108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:43.340 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:43.326 7f83fe0fb700 1 -- 172.21.2.102:0/1515363135 >> 172.21.2.102:0/1515363135 conn(0x7f83f80ce690 msgr2=0x7f83f80e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:43.340 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:43.326 7f83fe0fb700 1 -- 172.21.2.102:0/1515363135 shutdown_connections 2021-02-10T11:22:43.340 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:43.326 7f83fe0fb700 1 -- 172.21.2.102:0/1515363135 wait complete. 2021-02-10T11:22:43.341 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:43.341 7f83fe0fb700 1 Processor -- start 2021-02-10T11:22:43.341 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:43.341 7f83fe0fb700 1 -- start start 2021-02-10T11:22:43.341 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:43.341 7f83fe0fb700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f83f808d400 0x7f83f8096c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:43.341 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:43.341 7f83fe0fb700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f83f80b1450 0x7f83f80b1850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:43.342 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:43.341 7f83fe0fb700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f83f80b1e70 0x7f83f81b1f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:43.342 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:43.341 7f83fe0fb700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83f8097310 con 0x7f83f80b1450 2021-02-10T11:22:43.342 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:43.341 7f83fe0fb700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83f808d800 con 0x7f83f808d400 2021-02-10T11:22:43.342 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:43.341 7f83fe0fb700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83f808d9f0 con 0x7f83f80b1e70 2021-02-10T11:22:43.342 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:43.341 7f83f6ffd700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f83f80b1450 0x7f83f80b1850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:43.342 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:43.341 7f83f6ffd700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f83f80b1450 0x7f83f80b1850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:55724/0 (socket says 172.21.2.102:55724) 2021-02-10T11:22:43.342 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:43.341 7f83f6ffd700 1 -- 172.21.2.102:0/2683827203 learned_addr learned my addr 172.21.2.102:0/2683827203 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:22:43.342 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:43.342 7f83f6ffd700 1 -- 172.21.2.102:0/2683827203 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f83f80b1e70 msgr2=0x7f83f81b1f60 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:22:43.342 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:43.342 7f83f6ffd700 1 --2- 172.21.2.102:0/2683827203 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f83f80b1e70 0x7f83f81b1f60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:43.342 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:43.342 7f83f6ffd700 1 -- 172.21.2.102:0/2683827203 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f83f808d400 msgr2=0x7f83f8096c70 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:22:43.342 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:43.342 7f83f6ffd700 1 --2- 172.21.2.102:0/2683827203 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f83f808d400 0x7f83f8096c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:43.342 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:43.342 7f83f6ffd700 1 -- 172.21.2.102:0/2683827203 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83f81b25f0 con 0x7f83f80b1450 2021-02-10T11:22:43.342 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:43.342 7f83f6ffd700 1 --2- 172.21.2.102:0/2683827203 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f83f80b1450 0x7f83f80b1850 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 rx=0x7f83e8009720 tx=0x7f83e800b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:43.343 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:43.342 7f83effff700 1 -- 172.21.2.102:0/2683827203 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f83e80090c0 con 0x7f83f80b1450 2021-02-10T11:22:43.343 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:43.342 7f83effff700 1 -- 172.21.2.102:0/2683827203 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83e8002030 con 0x7f83f80b1450 2021-02-10T11:22:43.343 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:43.342 7f83effff700 1 -- 172.21.2.102:0/2683827203 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f83e80024c0 con 0x7f83f80b1450 2021-02-10T11:22:43.343 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:43.342 7f83fe0fb700 1 -- 172.21.2.102:0/2683827203 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83f81b25f0 con 0x7f83f80b1450 2021-02-10T11:22:43.343 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:43.342 7f83fe0fb700 1 -- 172.21.2.102:0/2683827203 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83f81b25f0 con 0x7f83f80b1450 2021-02-10T11:22:43.343 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:43.343 7f83effff700 1 -- 172.21.2.102:0/2683827203 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f83e8000c90 con 0x7f83f80b1450 2021-02-10T11:22:43.343 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:43.343 7f83effff700 1 --2- 172.21.2.102:0/2683827203 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f83dc0186c0 0x7f83dc01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:43.343 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:43.343 7f83effff700 1 -- 172.21.2.102:0/2683827203 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f83e80325a0 con 0x7f83f80b1450 2021-02-10T11:22:43.343 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:43.343 7f83f77fe700 1 --2- 172.21.2.102:0/2683827203 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f83dc0186c0 0x7f83dc01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:43.343 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:43.344 7f83f77fe700 1 --2- 172.21.2.102:0/2683827203 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f83dc0186c0 0x7f83dc01abc0 crc :-1 s=READY pgs=532 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:43.372 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:43.375 7f83fe0fb700 1 -- 172.21.2.102:0/2683827203 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83f807ed80 con 0x7f83f80b1450 2021-02-10T11:22:43.374 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:43.377 7f83effff700 1 -- 172.21.2.102:0/2683827203 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f83e802cf50 con 0x7f83f80b1450 2021-02-10T11:22:43.503 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:43.506 7f83fe0fb700 1 -- 172.21.2.102:0/2683827203 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f83f807ed80 con 0x7f83f80b1450 2021-02-10T11:22:43.503 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:43.506 7f83effff700 1 -- 172.21.2.102:0/2683827203 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) v1 ==== 94+0+2099 (secure 0 0 0) 0x7f83e802cf50 con 0x7f83f80b1450 2021-02-10T11:22:43.504 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:22:43.504 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":21,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7194,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.102:6835/2295196020","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6834","nonce":2295196020},{"type":"v1","addr":"172.21.2.102:6835","nonce":2295196020}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":18}],"filesystems":[{"mdsmap":{"epoch":21,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:22:26.122430","modified":"2021-02-10 11:22:28.126179","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7191},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7191":{"gid":7191,"name":"b","rank":0,"incarnation":19,"state":"up:active","state_seq":3,"addr":"172.21.2.113:6833/3694995761","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6832","nonce":3694995761},{"type":"v1","addr":"172.21.2.113:6833","nonce":3694995761}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":3}]} 2021-02-10T11:22:43.504 INFO:teuthology.orchestra.run.gibba002.stderr:dumped fsmap epoch 21 2021-02-10T11:22:43.506 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:43.508 7f83fe0fb700 1 -- 172.21.2.102:0/2683827203 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f83dc0186c0 msgr2=0x7f83dc01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:43.506 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:43.508 7f83fe0fb700 1 --2- 172.21.2.102:0/2683827203 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f83dc0186c0 0x7f83dc01abc0 crc :-1 s=READY pgs=532 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:43.506 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:43.508 7f83fe0fb700 1 -- 172.21.2.102:0/2683827203 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f83f80b1450 msgr2=0x7f83f80b1850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:43.506 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:43.508 7f83fe0fb700 1 --2- 172.21.2.102:0/2683827203 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f83f80b1450 0x7f83f80b1850 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 rx=0x7f83e8009720 tx=0x7f83e800b960).stop 2021-02-10T11:22:43.506 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:43.508 7f83fe0fb700 1 -- 172.21.2.102:0/2683827203 shutdown_connections 2021-02-10T11:22:43.506 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:43.508 7f83fe0fb700 1 --2- 172.21.2.102:0/2683827203 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f83f80b1e70 0x7f83f81b1f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:43.506 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:43.508 7f83fe0fb700 1 --2- 172.21.2.102:0/2683827203 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f83f80b1450 0x7f83f80b1850 unknown :-1 s=CLOSED pgs=1109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:43.506 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:43.508 7f83fe0fb700 1 --2- 172.21.2.102:0/2683827203 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f83dc0186c0 0x7f83dc01abc0 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:43.506 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:43.508 7f83fe0fb700 1 --2- 172.21.2.102:0/2683827203 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f83f808d400 0x7f83f8096c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:43.507 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:43.508 7f83fe0fb700 1 -- 172.21.2.102:0/2683827203 >> 172.21.2.102:0/2683827203 conn(0x7f83f81ac620 msgr2=0x7f83f80896e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:43.507 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:43.509 7f83fe0fb700 1 -- 172.21.2.102:0/2683827203 shutdown_connections 2021-02-10T11:22:43.507 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:43.509 7f83fe0fb700 1 -- 172.21.2.102:0/2683827203 wait complete. 2021-02-10T11:22:43.716 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 21, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:22:26.122430', 'modified': '2021-02-10 11:22:28.126179', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7191}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7191': {'gid': 7191, 'name': 'b', 'rank': 0, 'incarnation': 19, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.113:6833/3694995761', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.113:6832', 'nonce': 3694995761}, {'type': 'v1', 'addr': '172.21.2.113:6833', 'nonce': 3694995761}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [7], 'metadata_pool': 6, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:22:43.716 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:22:43.716 DEBUG:teuthology.orchestra.run.gibba013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:22:44.115 INFO:teuthology.orchestra.run.gibba013.stdout:{ 2021-02-10T11:22:44.115 INFO:teuthology.orchestra.run.gibba013.stdout: "cluster_fsid": "68db30ea-dbce-4aef-a6df-f01c6fea0513", 2021-02-10T11:22:44.115 INFO:teuthology.orchestra.run.gibba013.stdout: "whoami": 0, 2021-02-10T11:22:44.115 INFO:teuthology.orchestra.run.gibba013.stdout: "id": 7191, 2021-02-10T11:22:44.115 INFO:teuthology.orchestra.run.gibba013.stdout: "want_state": "up:active", 2021-02-10T11:22:44.115 INFO:teuthology.orchestra.run.gibba013.stdout: "state": "up:active", 2021-02-10T11:22:44.115 INFO:teuthology.orchestra.run.gibba013.stdout: "rank_uptime": 17.994991249000002, 2021-02-10T11:22:44.115 INFO:teuthology.orchestra.run.gibba013.stdout: "mdsmap_epoch": 21, 2021-02-10T11:22:44.115 INFO:teuthology.orchestra.run.gibba013.stdout: "osdmap_epoch": 39, 2021-02-10T11:22:44.115 INFO:teuthology.orchestra.run.gibba013.stdout: "osdmap_epoch_barrier": 39, 2021-02-10T11:22:44.115 INFO:teuthology.orchestra.run.gibba013.stdout: "uptime": 24.654297021000001 2021-02-10T11:22:44.116 INFO:teuthology.orchestra.run.gibba013.stdout:} 2021-02-10T11:22:44.119 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "68db30ea-dbce-4aef-a6df-f01c6fea0513", "whoami": 0, "id": 7191, "want_state": "up:active", "state": "up:active", "rank_uptime": 17.994991249000002, "mdsmap_epoch": 21, "osdmap_epoch": 39, "osdmap_epoch_barrier": 39, "uptime": 24.654297021000001 } 2021-02-10T11:22:44.119 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2021-02-10T11:22:44.119 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2021-02-10T11:22:44.119 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.1 at ubuntu@gibba002.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.1... 2021-02-10T11:22:44.119 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:22:44.119 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec mkdir -p /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:22:44.455 DEBUG:teuthology.orchestra.run.gibba002:> sudo modprobe fuse 2021-02-10T11:22:44.802 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:22:44.802 DEBUG:teuthology.orchestra.run.gibba002:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:22:45.145 INFO:teuthology.orchestra.run.gibba002.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:22:45.145 INFO:teuthology.orchestra.run.gibba002.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:22:45.146 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:22:45.146 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:22:45.146 DEBUG:teuthology.orchestra.run.gibba002:> ls /sys/fs/fuse/connections 2021-02-10T11:22:45.481 INFO:teuthology.orchestra.run.gibba002.stdout:43 2021-02-10T11:22:45.482 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [43] 2021-02-10T11:22:45.482 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.1 /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:22:45.484 DEBUG:teuthology.orchestra.run.gibba002:> sudo modprobe fuse 2021-02-10T11:22:45.828 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:22:45.828 DEBUG:teuthology.orchestra.run.gibba002:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:22:45.895 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba002.stderr:ceph-fuse[145953]: starting ceph client 2021-02-10T11:22:45.895 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba002.stderr:2021-02-10 11:22:45.898 7f6543402f80 -1 init, newargv = 0x55b95e78d120 newargc=9 2021-02-10T11:22:45.900 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba002.stderr:ceph-fuse[145953]: starting fuse 2021-02-10T11:22:46.171 INFO:teuthology.orchestra.run.gibba002.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:22:46.171 INFO:teuthology.orchestra.run.gibba002.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:22:46.172 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:22:46.172 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:22:46.172 DEBUG:teuthology.orchestra.run.gibba002:> ls /sys/fs/fuse/connections 2021-02-10T11:22:46.506 INFO:teuthology.orchestra.run.gibba002.stdout:43 2021-02-10T11:22:46.506 INFO:teuthology.orchestra.run.gibba002.stdout:44 2021-02-10T11:22:46.507 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43, 44] 2021-02-10T11:22:46.507 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:22:46.507 DEBUG:teuthology.orchestra.run.gibba002:> sudo python3 -c ' 2021-02-10T11:22:46.507 DEBUG:teuthology.orchestra.run.gibba002:> import glob 2021-02-10T11:22:46.507 DEBUG:teuthology.orchestra.run.gibba002:> import re 2021-02-10T11:22:46.507 DEBUG:teuthology.orchestra.run.gibba002:> import os 2021-02-10T11:22:46.507 DEBUG:teuthology.orchestra.run.gibba002:> import subprocess 2021-02-10T11:22:46.507 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:22:46.507 DEBUG:teuthology.orchestra.run.gibba002:> def find_socket(client_name): 2021-02-10T11:22:46.507 DEBUG:teuthology.orchestra.run.gibba002:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2021-02-10T11:22:46.508 DEBUG:teuthology.orchestra.run.gibba002:> files = glob.glob(asok_path) 2021-02-10T11:22:46.508 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:22:46.508 DEBUG:teuthology.orchestra.run.gibba002:> # Given a non-glob path, it better be there 2021-02-10T11:22:46.508 DEBUG:teuthology.orchestra.run.gibba002:> if "*" not in asok_path: 2021-02-10T11:22:46.508 DEBUG:teuthology.orchestra.run.gibba002:> assert(len(files) == 1) 2021-02-10T11:22:46.508 DEBUG:teuthology.orchestra.run.gibba002:> return files[0] 2021-02-10T11:22:46.508 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:22:46.508 DEBUG:teuthology.orchestra.run.gibba002:> for f in files: 2021-02-10T11:22:46.508 DEBUG:teuthology.orchestra.run.gibba002:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2021-02-10T11:22:46.508 DEBUG:teuthology.orchestra.run.gibba002:> if os.path.exists("/proc/{0}".format(pid)): 2021-02-10T11:22:46.508 DEBUG:teuthology.orchestra.run.gibba002:> return f 2021-02-10T11:22:46.508 DEBUG:teuthology.orchestra.run.gibba002:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2021-02-10T11:22:46.508 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:22:46.508 DEBUG:teuthology.orchestra.run.gibba002:> print(find_socket("client.1")) 2021-02-10T11:22:46.508 DEBUG:teuthology.orchestra.run.gibba002:> ' 2021-02-10T11:22:46.869 INFO:teuthology.orchestra.run.gibba002.stdout:/var/run/ceph/ceph-client.1.145953.asok 2021-02-10T11:22:46.872 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.145953.asok 2021-02-10T11:22:46.872 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:22:46.872 DEBUG:teuthology.orchestra.run.gibba002:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.145953.asok status 2021-02-10T11:22:47.262 INFO:teuthology.orchestra.run.gibba002.stdout:{ 2021-02-10T11:22:47.262 INFO:teuthology.orchestra.run.gibba002.stdout: "metadata": { 2021-02-10T11:22:47.262 INFO:teuthology.orchestra.run.gibba002.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:22:47.262 INFO:teuthology.orchestra.run.gibba002.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:22:47.262 INFO:teuthology.orchestra.run.gibba002.stdout: "entity_id": "1", 2021-02-10T11:22:47.262 INFO:teuthology.orchestra.run.gibba002.stdout: "hostname": "gibba002", 2021-02-10T11:22:47.262 INFO:teuthology.orchestra.run.gibba002.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2021-02-10T11:22:47.262 INFO:teuthology.orchestra.run.gibba002.stdout: "pid": "145953", 2021-02-10T11:22:47.262 INFO:teuthology.orchestra.run.gibba002.stdout: "root": "/" 2021-02-10T11:22:47.263 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:22:47.263 INFO:teuthology.orchestra.run.gibba002.stdout: "dentry_count": 0, 2021-02-10T11:22:47.263 INFO:teuthology.orchestra.run.gibba002.stdout: "dentry_pinned_count": 0, 2021-02-10T11:22:47.263 INFO:teuthology.orchestra.run.gibba002.stdout: "id": 7395, 2021-02-10T11:22:47.263 INFO:teuthology.orchestra.run.gibba002.stdout: "inst": { 2021-02-10T11:22:47.263 INFO:teuthology.orchestra.run.gibba002.stdout: "name": { 2021-02-10T11:22:47.263 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "client", 2021-02-10T11:22:47.263 INFO:teuthology.orchestra.run.gibba002.stdout: "num": 7395 2021-02-10T11:22:47.263 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:22:47.263 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": { 2021-02-10T11:22:47.263 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:22:47.263 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:0", 2021-02-10T11:22:47.263 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 3053135689 2021-02-10T11:22:47.263 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:22:47.264 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:22:47.264 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": { 2021-02-10T11:22:47.264 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:22:47.264 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:0", 2021-02-10T11:22:47.264 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 3053135689 2021-02-10T11:22:47.264 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:22:47.264 INFO:teuthology.orchestra.run.gibba002.stdout: "inst_str": "client.7395 172.21.2.102:0/3053135689", 2021-02-10T11:22:47.264 INFO:teuthology.orchestra.run.gibba002.stdout: "addr_str": "172.21.2.102:0/3053135689", 2021-02-10T11:22:47.264 INFO:teuthology.orchestra.run.gibba002.stdout: "inode_count": 1, 2021-02-10T11:22:47.264 INFO:teuthology.orchestra.run.gibba002.stdout: "mds_epoch": 21, 2021-02-10T11:22:47.264 INFO:teuthology.orchestra.run.gibba002.stdout: "osd_epoch": 39, 2021-02-10T11:22:47.264 INFO:teuthology.orchestra.run.gibba002.stdout: "osd_epoch_barrier": 0, 2021-02-10T11:22:47.264 INFO:teuthology.orchestra.run.gibba002.stdout: "blacklisted": false 2021-02-10T11:22:47.264 INFO:teuthology.orchestra.run.gibba002.stdout:} 2021-02-10T11:22:47.266 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:22:47.266 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:22:47.266 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:22:47.602 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:22:47.602 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2021-02-10T11:22:47.602 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:22:47.603 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:22:47.948 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2021-02-10T11:22:48.379 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:48.364 7ffa4596f700 1 Processor -- start 2021-02-10T11:22:48.380 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:48.364 7ffa4596f700 1 -- start start 2021-02-10T11:22:48.380 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:48.365 7ffa4596f700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ffa400d1870 0x7ffa401b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:48.380 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:48.365 7ffa4596f700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7ffa400cff70 0x7ffa40109130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:48.380 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:48.365 7ffa4596f700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa40109a60 con 0x7ffa401b0ae0 2021-02-10T11:22:48.380 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:48.365 7ffa4596f700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa40109750 con 0x7ffa400cff70 2021-02-10T11:22:48.380 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:48.365 7ffa4596f700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa40088020 con 0x7ffa400d1870 2021-02-10T11:22:48.380 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:48.366 7ffa3f574700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ffa400d1870 0x7ffa401b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:48.380 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:48.366 7ffa3ed73700 1 --1- >> v1:172.21.2.113:6789/0 conn(0x7ffa401b0ae0 0x7ffa40108940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.113:6789/0 says I am v1:172.21.2.102:55296/0 (socket says 172.21.2.102:55296) 2021-02-10T11:22:48.380 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:48.366 7ffa3fd75700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7ffa400cff70 0x7ffa40109130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:48.380 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:48.366 7ffa3ed73700 1 -- 172.21.2.102:0/4246517596 learned_addr learned my addr 172.21.2.102:0/4246517596 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:22:48.380 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:48.366 7ffa3fd75700 1 -- 172.21.2.102:0/4246517596 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ffa400d1870 msgr2=0x7ffa401b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:48.380 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:48.366 7ffa3fd75700 1 --2- 172.21.2.102:0/4246517596 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ffa400d1870 0x7ffa401b04c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:48.381 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:48.366 7ffa3fd75700 1 -- 172.21.2.102:0/4246517596 >> v1:172.21.2.113:6789/0 conn(0x7ffa401b0ae0 legacy=0x7ffa40108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:48.381 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:48.366 7ffa3e572700 1 -- 172.21.2.102:0/4246517596 <== mon.1 v1:172.21.2.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3455946243 0 0) 0x7ffa340009b0 con 0x7ffa401b0ae0 2021-02-10T11:22:48.381 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:48.366 7ffa3fd75700 1 -- 172.21.2.102:0/4246517596 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa40088210 con 0x7ffa400cff70 2021-02-10T11:22:48.381 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:48.367 7ffa3fd75700 1 --2- 172.21.2.102:0/4246517596 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ffa400cff70 0x7ffa40109130 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 rx=0x7ffa38006e50 tx=0x7ffa38009e70).ready entity=mon.1 client_cookie=d6cdc37b62862106 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:48.381 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:48.367 7ffa3e572700 1 -- 172.21.2.102:0/4246517596 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ffa38006a20 con 0x7ffa400cff70 2021-02-10T11:22:48.381 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:48.367 7ffa3e572700 1 -- 172.21.2.102:0/4246517596 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa38006790 con 0x7ffa400cff70 2021-02-10T11:22:48.381 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:48.367 7ffa3e572700 1 -- 172.21.2.102:0/4246517596 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ffa3800b700 con 0x7ffa400cff70 2021-02-10T11:22:48.381 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:48.367 7ffa4596f700 1 -- 172.21.2.102:0/4246517596 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ffa400cff70 msgr2=0x7ffa40109130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:48.381 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:48.367 7ffa4596f700 1 --2- 172.21.2.102:0/4246517596 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ffa400cff70 0x7ffa40109130 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 rx=0x7ffa38006e50 tx=0x7ffa38009e70).stop 2021-02-10T11:22:48.381 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:48.368 7ffa4596f700 1 -- 172.21.2.102:0/4246517596 shutdown_connections 2021-02-10T11:22:48.381 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:48.368 7ffa4596f700 1 --2- 172.21.2.102:0/4246517596 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ffa400cff70 0x7ffa40109130 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:48.381 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:48.368 7ffa4596f700 1 --2- 172.21.2.102:0/4246517596 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ffa400d1870 0x7ffa401b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:48.381 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:48.368 7ffa4596f700 1 -- 172.21.2.102:0/4246517596 >> 172.21.2.102:0/4246517596 conn(0x7ffa400ce420 msgr2=0x7ffa400d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:48.382 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:48.368 7ffa4596f700 1 -- 172.21.2.102:0/4246517596 shutdown_connections 2021-02-10T11:22:48.382 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:48.368 7ffa4596f700 1 -- 172.21.2.102:0/4246517596 wait complete. 2021-02-10T11:22:48.383 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:48.383 7ffa4596f700 1 Processor -- start 2021-02-10T11:22:48.383 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:48.383 7ffa4596f700 1 -- start start 2021-02-10T11:22:48.384 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:48.383 7ffa4596f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ffa400aa9e0 0x7ffa4018ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:48.384 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:48.383 7ffa4596f700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ffa4018d4d0 0x7ffa40098df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:48.384 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:48.383 7ffa4596f700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ffa40099410 0x7ffa401a6cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:48.384 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:48.383 7ffa4596f700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa40098870 con 0x7ffa400aa9e0 2021-02-10T11:22:48.384 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:48.383 7ffa4596f700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa401a72e0 con 0x7ffa40099410 2021-02-10T11:22:48.384 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:48.383 7ffa4596f700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa401a74c0 con 0x7ffa4018d4d0 2021-02-10T11:22:48.384 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:48.383 7ffa3f574700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ffa400aa9e0 0x7ffa4018ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:48.384 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:48.383 7ffa3f574700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ffa400aa9e0 0x7ffa4018ceb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:55752/0 (socket says 172.21.2.102:55752) 2021-02-10T11:22:48.384 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:48.383 7ffa3f574700 1 -- 172.21.2.102:0/44666383 learned_addr learned my addr 172.21.2.102:0/44666383 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:22:48.384 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:48.383 7ffa3ed73700 1 --2- 172.21.2.102:0/44666383 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ffa4018d4d0 0x7ffa40098df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:48.384 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:48.384 7ffa3fd75700 1 --2- 172.21.2.102:0/44666383 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ffa40099410 0x7ffa401a6cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:48.384 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:48.384 7ffa3ed73700 1 -- 172.21.2.102:0/44666383 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ffa40099410 msgr2=0x7ffa401a6cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:48.384 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:48.384 7ffa3ed73700 1 --2- 172.21.2.102:0/44666383 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ffa40099410 0x7ffa401a6cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:48.385 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:48.384 7ffa3ed73700 1 -- 172.21.2.102:0/44666383 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ffa400aa9e0 msgr2=0x7ffa4018ceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:48.385 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:48.384 7ffa3ed73700 1 --2- 172.21.2.102:0/44666383 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ffa400aa9e0 0x7ffa4018ceb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:48.385 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:48.384 7ffa3ed73700 1 -- 172.21.2.102:0/44666383 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa401a76e0 con 0x7ffa4018d4d0 2021-02-10T11:22:48.385 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:48.384 7ffa3f574700 1 --2- 172.21.2.102:0/44666383 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ffa400aa9e0 0x7ffa4018ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:22:48.385 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:48.384 7ffa3ed73700 1 --2- 172.21.2.102:0/44666383 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ffa4018d4d0 0x7ffa40098df0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7ffa3400a4c0 tx=0x7ffa3400a860).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:48.385 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:48.384 7ffa27fff700 1 -- 172.21.2.102:0/44666383 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ffa34000e20 con 0x7ffa4018d4d0 2021-02-10T11:22:48.385 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:48.384 7ffa27fff700 1 -- 172.21.2.102:0/44666383 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa34000940 con 0x7ffa4018d4d0 2021-02-10T11:22:48.385 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:48.384 7ffa27fff700 1 -- 172.21.2.102:0/44666383 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ffa3400ac50 con 0x7ffa4018d4d0 2021-02-10T11:22:48.385 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:48.384 7ffa4596f700 1 -- 172.21.2.102:0/44666383 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa401a76e0 con 0x7ffa4018d4d0 2021-02-10T11:22:48.385 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:48.384 7ffa4596f700 1 -- 172.21.2.102:0/44666383 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa40097c90 con 0x7ffa4018d4d0 2021-02-10T11:22:48.385 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:48.385 7ffa27fff700 1 -- 172.21.2.102:0/44666383 <== mon.2 v2:172.21.2.113:3301/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ffa34004140 con 0x7ffa4018d4d0 2021-02-10T11:22:48.385 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:48.385 7ffa27fff700 1 --2- 172.21.2.102:0/44666383 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ffa200186c0 0x7ffa2001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:48.386 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:48.385 7ffa27fff700 1 -- 172.21.2.102:0/44666383 <== mon.2 v2:172.21.2.113:3301/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 5573+0+0 (secure 0 0 0) 0x7ffa34032300 con 0x7ffa4018d4d0 2021-02-10T11:22:48.386 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:48.385 7ffa3f574700 1 --2- 172.21.2.102:0/44666383 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ffa200186c0 0x7ffa2001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:48.386 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:48.385 7ffa3f574700 1 --2- 172.21.2.102:0/44666383 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ffa200186c0 0x7ffa2001abc0 crc :-1 s=READY pgs=533 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:48.412 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:48.415 7ffa4596f700 1 -- 172.21.2.102:0/44666383 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa400828f0 con 0x7ffa4018d4d0 2021-02-10T11:22:48.415 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:48.418 7ffa27fff700 1 -- 172.21.2.102:0/44666383 <== mon.2 v2:172.21.2.113:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ffa34030c30 con 0x7ffa4018d4d0 2021-02-10T11:22:48.555 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:48.558 7ffa4596f700 1 -- 172.21.2.102:0/44666383 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7ffa40109a60 con 0x7ffa4018d4d0 2021-02-10T11:22:48.978 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:48.980 7ffa27fff700 1 -- 172.21.2.102:0/44666383 <== mon.2 v2:172.21.2.113:3301/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v22) v1 ==== 136+0+0 (secure 0 0 0) 0x7ffa34004140 con 0x7ffa4018d4d0 2021-02-10T11:22:49.025 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:49.027 7ffa4596f700 1 -- 172.21.2.102:0/44666383 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ffa200186c0 msgr2=0x7ffa2001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:49.025 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:49.027 7ffa4596f700 1 --2- 172.21.2.102:0/44666383 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ffa200186c0 0x7ffa2001abc0 crc :-1 s=READY pgs=533 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:49.026 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:49.027 7ffa4596f700 1 -- 172.21.2.102:0/44666383 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ffa4018d4d0 msgr2=0x7ffa40098df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:49.026 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:49.027 7ffa4596f700 1 --2- 172.21.2.102:0/44666383 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ffa4018d4d0 0x7ffa40098df0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7ffa3400a4c0 tx=0x7ffa3400a860).stop 2021-02-10T11:22:49.026 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:49.028 7ffa4596f700 1 -- 172.21.2.102:0/44666383 shutdown_connections 2021-02-10T11:22:49.026 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:49.028 7ffa4596f700 1 --2- 172.21.2.102:0/44666383 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ffa200186c0 0x7ffa2001abc0 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:49.026 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:49.028 7ffa4596f700 1 --2- 172.21.2.102:0/44666383 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ffa40099410 0x7ffa401a6cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:49.026 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:49.028 7ffa4596f700 1 --2- 172.21.2.102:0/44666383 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ffa4018d4d0 0x7ffa40098df0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:49.026 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:49.028 7ffa4596f700 1 --2- 172.21.2.102:0/44666383 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ffa400aa9e0 0x7ffa4018ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:49.026 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:49.028 7ffa4596f700 1 -- 172.21.2.102:0/44666383 >> 172.21.2.102:0/44666383 conn(0x7ffa4019af60 msgr2=0x7ffa4018c280 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:49.026 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:49.028 7ffa4596f700 1 -- 172.21.2.102:0/44666383 shutdown_connections 2021-02-10T11:22:49.026 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:49.028 7ffa4596f700 1 -- 172.21.2.102:0/44666383 wait complete. 2021-02-10T11:22:49.033 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs volume ls 2021-02-10T11:22:49.457 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:49.444 7f67c6b8a700 1 Processor -- start 2021-02-10T11:22:49.458 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:49.444 7f67c6b8a700 1 -- start start 2021-02-10T11:22:49.458 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:49.445 7f67c6b8a700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f67c00e5470 0x7f67c007e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:49.458 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:49.445 7f67c6b8a700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f67c007f300 0x7f67c007f700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:49.458 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:49.445 7f67c6b8a700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67c007ed80 con 0x7f67c00e5070 2021-02-10T11:22:49.458 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:49.445 7f67c6b8a700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67c01b4010 con 0x7f67c007f300 2021-02-10T11:22:49.458 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:49.445 7f67c6b8a700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67c00d1790 con 0x7f67c00e5470 2021-02-10T11:22:49.458 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:49.445 7f67c51a7700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f67c007f300 0x7f67c007f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:49.458 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:49.445 7f67c51a7700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f67c007f300 0x7f67c007f700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:55764/0 (socket says 172.21.2.102:55764) 2021-02-10T11:22:49.458 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:49.445 7f67c51a7700 1 -- 172.21.2.102:0/1195031658 learned_addr learned my addr 172.21.2.102:0/1195031658 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:22:49.458 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:49.445 7f67c51a7700 1 -- 172.21.2.102:0/1195031658 >> v2:172.21.2.113:3300/0 conn(0x7f67c00e5470 msgr2=0x7f67c007e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:49.459 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:49.445 7f67c51a7700 1 --2- 172.21.2.102:0/1195031658 >> v2:172.21.2.113:3300/0 conn(0x7f67c00e5470 0x7f67c007e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:49.459 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:49.445 7f67c51a7700 1 -- 172.21.2.102:0/1195031658 >> v1:172.21.2.113:6789/0 conn(0x7f67c00e5070 legacy=0x7f67c01b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:49.459 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:49.445 7f67c51a7700 1 -- 172.21.2.102:0/1195031658 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67c00d19b0 con 0x7f67c007f300 2021-02-10T11:22:49.459 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:49.445 7f67c51a7700 1 --2- 172.21.2.102:0/1195031658 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f67c007f300 0x7f67c007f700 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 rx=0x7f67b8006b80 tx=0x7f67b8009b90).ready entity=mon.0 client_cookie=5606034ef16fc0ac server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:49.459 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:49.446 7f67bf7fe700 1 -- 172.21.2.102:0/1195031658 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f67b80022e0 con 0x7f67c007f300 2021-02-10T11:22:49.459 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:49.446 7f67bf7fe700 1 -- 172.21.2.102:0/1195031658 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67b80022e0 con 0x7f67c007f300 2021-02-10T11:22:49.459 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:49.446 7f67bf7fe700 1 -- 172.21.2.102:0/1195031658 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f67b8002620 con 0x7f67c007f300 2021-02-10T11:22:49.459 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:49.446 7f67c6b8a700 1 -- 172.21.2.102:0/1195031658 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f67c007f300 msgr2=0x7f67c007f700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:49.459 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:49.446 7f67c6b8a700 1 --2- 172.21.2.102:0/1195031658 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f67c007f300 0x7f67c007f700 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 rx=0x7f67b8006b80 tx=0x7f67b8009b90).stop 2021-02-10T11:22:49.459 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:49.446 7f67c6b8a700 1 -- 172.21.2.102:0/1195031658 shutdown_connections 2021-02-10T11:22:49.459 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:49.446 7f67c6b8a700 1 --2- 172.21.2.102:0/1195031658 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f67c007f300 0x7f67c007f700 unknown :-1 s=CLOSED pgs=1113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:49.459 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:49.446 7f67c6b8a700 1 --2- 172.21.2.102:0/1195031658 >> v2:172.21.2.113:3300/0 conn(0x7f67c00e5470 0x7f67c007e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:49.459 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:49.446 7f67c6b8a700 1 -- 172.21.2.102:0/1195031658 >> 172.21.2.102:0/1195031658 conn(0x7f67c00ce690 msgr2=0x7f67c00e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:49.459 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:49.446 7f67c6b8a700 1 -- 172.21.2.102:0/1195031658 shutdown_connections 2021-02-10T11:22:49.460 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:49.446 7f67c6b8a700 1 -- 172.21.2.102:0/1195031658 wait complete. 2021-02-10T11:22:49.461 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:49.461 7f67c6b8a700 1 Processor -- start 2021-02-10T11:22:49.461 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:49.461 7f67c6b8a700 1 -- start start 2021-02-10T11:22:49.461 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:49.461 7f67c6b8a700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f67c01a2fa0 0x7f67c00914f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:49.461 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:49.461 7f67c6b8a700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f67c01a33a0 0x7f67c0091b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:49.462 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:49.461 7f67c6b8a700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f67c00921c0 0x7f67c00925c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:49.462 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:49.461 7f67c6b8a700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67c0092c60 con 0x7f67c00921c0 2021-02-10T11:22:49.462 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:49.461 7f67c6b8a700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67c0092e80 con 0x7f67c01a2fa0 2021-02-10T11:22:49.462 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:49.461 7f67c6b8a700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67c00930a0 con 0x7f67c01a33a0 2021-02-10T11:22:49.462 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:49.461 7f67c51a7700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f67c00921c0 0x7f67c00925c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:49.462 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:49.461 7f67c51a7700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f67c00921c0 0x7f67c00925c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:55768/0 (socket says 172.21.2.102:55768) 2021-02-10T11:22:49.462 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:49.461 7f67c51a7700 1 -- 172.21.2.102:0/3146373036 learned_addr learned my addr 172.21.2.102:0/3146373036 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:22:49.462 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:49.462 7f67bffff700 1 --2- 172.21.2.102:0/3146373036 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f67c01a33a0 0x7f67c0091b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:49.462 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:49.462 7f67c49a6700 1 --2- 172.21.2.102:0/3146373036 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f67c01a2fa0 0x7f67c00914f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:49.462 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:49.462 7f67c51a7700 1 -- 172.21.2.102:0/3146373036 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f67c01a33a0 msgr2=0x7f67c0091b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:49.463 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:49.462 7f67c51a7700 1 --2- 172.21.2.102:0/3146373036 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f67c01a33a0 0x7f67c0091b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:49.463 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:49.462 7f67c51a7700 1 -- 172.21.2.102:0/3146373036 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f67c01a2fa0 msgr2=0x7f67c00914f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:49.463 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:49.462 7f67c51a7700 1 --2- 172.21.2.102:0/3146373036 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f67c01a2fa0 0x7f67c00914f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:49.463 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:49.462 7f67c51a7700 1 -- 172.21.2.102:0/3146373036 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67c00932c0 con 0x7f67c00921c0 2021-02-10T11:22:49.463 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:49.462 7f67c51a7700 1 --2- 172.21.2.102:0/3146373036 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f67c00921c0 0x7f67c00925c0 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 rx=0x7f67b8009640 tx=0x7f67b800b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:49.463 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:49.462 7f67bd7fa700 1 -- 172.21.2.102:0/3146373036 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f67b8009100 con 0x7f67c00921c0 2021-02-10T11:22:49.464 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:49.462 7f67bd7fa700 1 -- 172.21.2.102:0/3146373036 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67b800bde0 con 0x7f67c00921c0 2021-02-10T11:22:49.464 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:49.462 7f67c6b8a700 1 -- 172.21.2.102:0/3146373036 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67c00932c0 con 0x7f67c00921c0 2021-02-10T11:22:49.464 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:49.462 7f67c6b8a700 1 -- 172.21.2.102:0/3146373036 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67c00ba3b0 con 0x7f67c00921c0 2021-02-10T11:22:49.465 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:49.462 7f67bd7fa700 1 -- 172.21.2.102:0/3146373036 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f67b8006470 con 0x7f67c00921c0 2021-02-10T11:22:49.465 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:49.463 7f67bd7fa700 1 -- 172.21.2.102:0/3146373036 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f67b8006030 con 0x7f67c00921c0 2021-02-10T11:22:49.465 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:49.463 7f67bd7fa700 1 --2- 172.21.2.102:0/3146373036 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f67a40186d0 0x7f67a401abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:49.465 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:49.463 7f67bd7fa700 1 -- 172.21.2.102:0/3146373036 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f67b80327a0 con 0x7f67c00921c0 2021-02-10T11:22:49.465 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:49.463 7f67c49a6700 1 --2- 172.21.2.102:0/3146373036 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f67a40186d0 0x7f67a401abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:49.465 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:49.463 7f67c49a6700 1 --2- 172.21.2.102:0/3146373036 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f67a40186d0 0x7f67a401abd0 crc :-1 s=READY pgs=534 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:49.492 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:49.495 7f67c6b8a700 1 -- 172.21.2.102:0/3146373036 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67c0083da0 con 0x7f67c00921c0 2021-02-10T11:22:49.494 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:49.497 7f67bd7fa700 1 -- 172.21.2.102:0/3146373036 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f67b8009100 con 0x7f67c00921c0 2021-02-10T11:22:49.629 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:49.632 7f67c6b8a700 1 -- 172.21.2.102:0/3146373036 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs volume ls", "target": ["mgr", ""]}) v1 -- 0x7f67c007ed80 con 0x7f67a40186d0 2021-02-10T11:22:49.629 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:49.632 7f67bd7fa700 1 -- 172.21.2.102:0/3146373036 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+41 (crc 0 0 0) 0x7f67a8006b40 con 0x7f67a40186d0 2021-02-10T11:22:49.630 INFO:teuthology.orchestra.run.gibba002.stdout:[ 2021-02-10T11:22:49.630 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:22:49.630 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "cephfs" 2021-02-10T11:22:49.630 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:22:49.630 INFO:teuthology.orchestra.run.gibba002.stdout:] 2021-02-10T11:22:49.631 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:49.634 7f67c6b8a700 1 -- 172.21.2.102:0/3146373036 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f67a40186d0 msgr2=0x7f67a401abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:49.631 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:49.634 7f67c6b8a700 1 --2- 172.21.2.102:0/3146373036 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f67a40186d0 0x7f67a401abd0 crc :-1 s=READY pgs=534 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:49.631 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:49.634 7f67c6b8a700 1 -- 172.21.2.102:0/3146373036 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f67c00921c0 msgr2=0x7f67c00925c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:49.631 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:49.634 7f67c6b8a700 1 --2- 172.21.2.102:0/3146373036 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f67c00921c0 0x7f67c00925c0 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 rx=0x7f67b8009640 tx=0x7f67b800b960).stop 2021-02-10T11:22:49.632 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:49.634 7f67c6b8a700 1 -- 172.21.2.102:0/3146373036 shutdown_connections 2021-02-10T11:22:49.632 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:49.634 7f67c6b8a700 1 --2- 172.21.2.102:0/3146373036 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f67c00921c0 0x7f67c00925c0 unknown :-1 s=CLOSED pgs=1114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:49.632 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:49.634 7f67c6b8a700 1 --2- 172.21.2.102:0/3146373036 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f67c01a33a0 0x7f67c0091b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:49.632 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:49.634 7f67c6b8a700 1 --2- 172.21.2.102:0/3146373036 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f67a40186d0 0x7f67a401abd0 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:49.632 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:49.634 7f67c6b8a700 1 --2- 172.21.2.102:0/3146373036 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f67c01a2fa0 0x7f67c00914f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:49.632 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:49.634 7f67c6b8a700 1 -- 172.21.2.102:0/3146373036 >> 172.21.2.102:0/3146373036 conn(0x7f67c00bde70 msgr2=0x7f67c00c3970 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:49.633 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:49.634 7f67c6b8a700 1 -- 172.21.2.102:0/3146373036 shutdown_connections 2021-02-10T11:22:49.633 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:49.634 7f67c6b8a700 1 -- 172.21.2.102:0/3146373036 wait complete. 2021-02-10T11:22:49.640 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config set mon mon_allow_pool_delete True 2021-02-10T11:22:50.074 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.059 7f34d7996700 1 Processor -- start 2021-02-10T11:22:50.318 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.059 7f34d7996700 1 -- start start 2021-02-10T11:22:50.318 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.059 7f34d7996700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f34d01b0870 0x7f34d00f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:50.319 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.059 7f34d7996700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34d01b0c70 con 0x7f34d01b0470 2021-02-10T11:22:50.319 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.059 7f34d7996700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34d00f0b50 con 0x7f34d00efe00 2021-02-10T11:22:50.319 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.059 7f34d7996700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34d00e5ea0 con 0x7f34d01b0870 2021-02-10T11:22:50.319 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.059 7f34d57b2700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f34d01b0470 0x7f34d00ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:47998/0 (socket says 172.21.2.102:47998) 2021-02-10T11:22:50.319 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.059 7f34d57b2700 1 -- 172.21.2.102:0/3265799923 learned_addr learned my addr 172.21.2.102:0/3265799923 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:22:50.319 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.060 7f34c7fff700 1 -- 172.21.2.102:0/3265799923 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3918164494 0 0) 0x7f34cc001180 con 0x7f34d01b0470 2021-02-10T11:22:50.319 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.060 7f34c7fff700 1 -- 172.21.2.102:0/3265799923 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34bc002fe0 con 0x7f34d01b0470 2021-02-10T11:22:50.319 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.060 7f34d4fb1700 1 --2- 172.21.2.102:0/3265799923 >> v2:172.21.2.113:3300/0 conn(0x7f34d01b0870 0x7f34d00f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:50.319 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.060 7f34c7fff700 1 -- 172.21.2.102:0/3265799923 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1901620354 0 0) 0x7f34cc001180 con 0x7f34d01b0470 2021-02-10T11:22:50.319 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.060 7f34c7fff700 1 -- 172.21.2.102:0/3265799923 >> v2:172.21.2.113:3300/0 conn(0x7f34d01b0870 msgr2=0x7f34d00f20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:50.319 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.060 7f34c7fff700 1 --2- 172.21.2.102:0/3265799923 >> v2:172.21.2.113:3300/0 conn(0x7f34d01b0870 0x7f34d00f20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:50.319 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.060 7f34c7fff700 1 -- 172.21.2.102:0/3265799923 >> v1:172.21.2.113:6789/0 conn(0x7f34d00efe00 legacy=0x7f34d00f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:50.319 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.060 7f34c7fff700 1 -- 172.21.2.102:0/3265799923 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34d00e52e0 con 0x7f34d01b0470 2021-02-10T11:22:50.320 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.061 7f34c7fff700 1 -- 172.21.2.102:0/3265799923 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7f34cc001680 con 0x7f34d01b0470 2021-02-10T11:22:50.320 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.061 7f34c7fff700 1 -- 172.21.2.102:0/3265799923 >> v1:172.21.2.102:6789/0 conn(0x7f34d01b0470 legacy=0x7f34d00ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:50.320 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.061 7f34c7fff700 1 --2- 172.21.2.102:0/3265799923 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f34bc003aa0 0x7f34bc005f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:50.320 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.061 7f34c7fff700 1 --2- 172.21.2.102:0/3265799923 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f34bc006780 0x7f34bc008ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:50.320 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.061 7f34c7fff700 1 --2- 172.21.2.102:0/3265799923 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f34bc0095a0 0x7f34bc00baa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:50.320 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.061 7f34c7fff700 1 -- 172.21.2.102:0/3265799923 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34bc00c2f0 con 0x7f34bc0095a0 2021-02-10T11:22:50.320 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.061 7f34c7fff700 1 -- 172.21.2.102:0/3265799923 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34bc00c510 con 0x7f34bc006780 2021-02-10T11:22:50.320 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.061 7f34c7fff700 1 -- 172.21.2.102:0/3265799923 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34bc050740 con 0x7f34bc003aa0 2021-02-10T11:22:50.320 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.061 7f34d4fb1700 1 --2- 172.21.2.102:0/3265799923 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f34bc006780 0x7f34bc008ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:50.320 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.061 7f34d57b2700 1 --2- 172.21.2.102:0/3265799923 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f34bc003aa0 0x7f34bc005f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:50.320 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.061 7f34d5fb3700 1 --2- 172.21.2.102:0/3265799923 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f34bc0095a0 0x7f34bc00baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:50.320 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.061 7f34d5fb3700 1 -- 172.21.2.102:0/3265799923 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f34bc003aa0 msgr2=0x7f34bc005f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:50.320 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.061 7f34d5fb3700 1 --2- 172.21.2.102:0/3265799923 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f34bc003aa0 0x7f34bc005f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:50.320 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.061 7f34d5fb3700 1 -- 172.21.2.102:0/3265799923 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f34bc006780 msgr2=0x7f34bc008ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:50.321 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.061 7f34d5fb3700 1 --2- 172.21.2.102:0/3265799923 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f34bc006780 0x7f34bc008ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:50.321 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.061 7f34d5fb3700 1 -- 172.21.2.102:0/3265799923 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34bc050970 con 0x7f34bc0095a0 2021-02-10T11:22:50.321 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.061 7f34d5fb3700 1 -- 172.21.2.102:0/3265799923 reap_dead start 2021-02-10T11:22:50.321 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.061 7f34d5fb3700 1 --2- 172.21.2.102:0/3265799923 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f34bc0095a0 0x7f34bc00baa0 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 rx=0x7f34c8006e30 tx=0x7f34c8009de0).ready entity=mon.0 client_cookie=f9748812bd54e082 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:50.321 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.061 7f34c7fff700 1 -- 172.21.2.102:0/3265799923 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f34c800b300 con 0x7f34bc0095a0 2021-02-10T11:22:50.321 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.061 7f34c7fff700 1 -- 172.21.2.102:0/3265799923 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34c800b680 con 0x7f34bc0095a0 2021-02-10T11:22:50.321 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.062 7f34c7fff700 1 -- 172.21.2.102:0/3265799923 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f34c800bba0 con 0x7f34bc0095a0 2021-02-10T11:22:50.321 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.062 7f34d7996700 1 -- 172.21.2.102:0/3265799923 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f34bc0095a0 msgr2=0x7f34bc00baa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:50.321 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.062 7f34d7996700 1 --2- 172.21.2.102:0/3265799923 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f34bc0095a0 0x7f34bc00baa0 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 rx=0x7f34c8006e30 tx=0x7f34c8009de0).stop 2021-02-10T11:22:50.321 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.062 7f34d7996700 1 -- 172.21.2.102:0/3265799923 shutdown_connections 2021-02-10T11:22:50.321 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.062 7f34d7996700 1 --2- 172.21.2.102:0/3265799923 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f34bc0095a0 0x7f34bc00baa0 unknown :-1 s=CLOSED pgs=1116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:50.321 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.062 7f34d7996700 1 -- 172.21.2.102:0/3265799923 >> 172.21.2.102:0/3265799923 conn(0x7f34d00ce3e0 msgr2=0x7f34d00e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:50.321 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.062 7f34d7996700 1 -- 172.21.2.102:0/3265799923 shutdown_connections 2021-02-10T11:22:50.322 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.062 7f34d7996700 1 -- 172.21.2.102:0/3265799923 wait complete. 2021-02-10T11:22:50.324 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.077 7f34d7996700 1 Processor -- start 2021-02-10T11:22:50.324 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.077 7f34d7996700 1 -- start start 2021-02-10T11:22:50.324 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.077 7f34d7996700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f34d009b930 0x7f34d00b7c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:50.324 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.077 7f34d7996700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f34d00b8410 0x7f34d00b8810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:50.325 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.077 7f34d7996700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f34d00c1f80 0x7f34d00c2380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:50.325 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.077 7f34d7996700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34d00c1a00 con 0x7f34d00c1f80 2021-02-10T11:22:50.325 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.077 7f34d7996700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34d00b8e30 con 0x7f34d00b8410 2021-02-10T11:22:50.325 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.077 7f34d7996700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34d00b9050 con 0x7f34d009b930 2021-02-10T11:22:50.325 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.077 7f34d5fb3700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f34d00c1f80 0x7f34d00c2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:50.325 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.077 7f34d5fb3700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f34d00c1f80 0x7f34d00c2380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:55790/0 (socket says 172.21.2.102:55790) 2021-02-10T11:22:50.325 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.077 7f34d5fb3700 1 -- 172.21.2.102:0/3669727162 learned_addr learned my addr 172.21.2.102:0/3669727162 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:22:50.325 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.078 7f34d57b2700 1 --2- 172.21.2.102:0/3669727162 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f34d009b930 0x7f34d00b7c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:50.325 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.078 7f34d4fb1700 1 --2- 172.21.2.102:0/3669727162 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f34d00b8410 0x7f34d00b8810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:50.325 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.078 7f34d5fb3700 1 -- 172.21.2.102:0/3669727162 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f34d009b930 msgr2=0x7f34d00b7c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:50.325 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.078 7f34d5fb3700 1 --2- 172.21.2.102:0/3669727162 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f34d009b930 0x7f34d00b7c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:50.325 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.078 7f34d5fb3700 1 -- 172.21.2.102:0/3669727162 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f34d00b8410 msgr2=0x7f34d00b8810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:50.325 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.078 7f34d5fb3700 1 --2- 172.21.2.102:0/3669727162 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f34d00b8410 0x7f34d00b8810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:50.325 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.078 7f34d5fb3700 1 -- 172.21.2.102:0/3669727162 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34d00c29a0 con 0x7f34d00c1f80 2021-02-10T11:22:50.326 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.078 7f34d5fb3700 1 --2- 172.21.2.102:0/3669727162 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f34d00c1f80 0x7f34d00c2380 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 rx=0x7f34c8006e40 tx=0x7f34c800bd40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:50.326 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.078 7f34c5ffb700 1 -- 172.21.2.102:0/3669727162 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f34c8005a30 con 0x7f34d00c1f80 2021-02-10T11:22:50.326 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.078 7f34c5ffb700 1 -- 172.21.2.102:0/3669727162 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34c8005c70 con 0x7f34d00c1f80 2021-02-10T11:22:50.326 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.078 7f34c5ffb700 1 -- 172.21.2.102:0/3669727162 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f34c80062b0 con 0x7f34d00c1f80 2021-02-10T11:22:50.326 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.078 7f34d7996700 1 -- 172.21.2.102:0/3669727162 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34d00c29a0 con 0x7f34d00c1f80 2021-02-10T11:22:50.326 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.078 7f34d7996700 1 -- 172.21.2.102:0/3669727162 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34d00c2f30 con 0x7f34d00c1f80 2021-02-10T11:22:50.326 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.079 7f34c5ffb700 1 -- 172.21.2.102:0/3669727162 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f34c8005820 con 0x7f34d00c1f80 2021-02-10T11:22:50.326 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.079 7f34c5ffb700 1 --2- 172.21.2.102:0/3669727162 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f34b00186c0 0x7f34b001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:50.326 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.079 7f34c5ffb700 1 -- 172.21.2.102:0/3669727162 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f34c8032910 con 0x7f34d00c1f80 2021-02-10T11:22:50.326 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.079 7f34d57b2700 1 --2- 172.21.2.102:0/3669727162 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f34b00186c0 0x7f34b001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:50.326 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.079 7f34d57b2700 1 --2- 172.21.2.102:0/3669727162 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f34b00186c0 0x7f34b001abc0 crc :-1 s=READY pgs=535 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:50.326 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.109 7f34d7996700 1 -- 172.21.2.102:0/3669727162 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34d0105b30 con 0x7f34d00c1f80 2021-02-10T11:22:50.326 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.112 7f34c5ffb700 1 -- 172.21.2.102:0/3669727162 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f34c80093d0 con 0x7f34d00c1f80 2021-02-10T11:22:50.327 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.261 7f34d7996700 1 -- 172.21.2.102:0/3669727162 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f34d00f0b50 con 0x7f34d00c1f80 2021-02-10T11:22:50.327 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.262 7f34c5ffb700 1 -- 172.21.2.102:0/3669727162 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v4)=0 v4) v1 ==== 122+0+0 (secure 0 0 0) 0x7f34c8005820 con 0x7f34d00c1f80 2021-02-10T11:22:50.327 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.264 7f34d7996700 1 -- 172.21.2.102:0/3669727162 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f34b00186c0 msgr2=0x7f34b001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:50.327 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.264 7f34d7996700 1 --2- 172.21.2.102:0/3669727162 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f34b00186c0 0x7f34b001abc0 crc :-1 s=READY pgs=535 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:50.327 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.264 7f34d7996700 1 -- 172.21.2.102:0/3669727162 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f34d00c1f80 msgr2=0x7f34d00c2380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:50.327 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.264 7f34d7996700 1 --2- 172.21.2.102:0/3669727162 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f34d00c1f80 0x7f34d00c2380 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 rx=0x7f34c8006e40 tx=0x7f34c800bd40).stop 2021-02-10T11:22:50.327 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.264 7f34d7996700 1 -- 172.21.2.102:0/3669727162 shutdown_connections 2021-02-10T11:22:50.327 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.264 7f34d7996700 1 --2- 172.21.2.102:0/3669727162 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f34d00c1f80 0x7f34d00c2380 unknown :-1 s=CLOSED pgs=1117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:50.327 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.264 7f34d7996700 1 --2- 172.21.2.102:0/3669727162 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f34b00186c0 0x7f34b001abc0 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:50.327 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.264 7f34d7996700 1 --2- 172.21.2.102:0/3669727162 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f34d00b8410 0x7f34d00b8810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:50.327 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.264 7f34d7996700 1 --2- 172.21.2.102:0/3669727162 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f34d009b930 0x7f34d00b7c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:50.327 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.264 7f34d7996700 1 -- 172.21.2.102:0/3669727162 >> 172.21.2.102:0/3669727162 conn(0x7f34d0094e10 msgr2=0x7f34d00a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:50.327 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.264 7f34d7996700 1 -- 172.21.2.102:0/3669727162 shutdown_connections 2021-02-10T11:22:50.327 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.264 7f34d7996700 1 -- 172.21.2.102:0/3669727162 wait complete. 2021-02-10T11:22:50.328 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth get-or-create client.guest1 mds 'allow *' osd 'allow rw' mon 'allow *' 2021-02-10T11:22:50.769 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.754 7f1568ed8700 1 Processor -- start 2021-02-10T11:22:50.770 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.754 7f1568ed8700 1 -- start start 2021-02-10T11:22:50.770 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.755 7f1568ed8700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f15640f05f0 0x7f15640f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:50.770 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.755 7f1568ed8700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15641b0c70 con 0x7f15641b0470 2021-02-10T11:22:50.770 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.755 7f1568ed8700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15640f0b50 con 0x7f15641b0870 2021-02-10T11:22:50.770 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.755 7f1568ed8700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15640e5ea0 con 0x7f15640f05f0 2021-02-10T11:22:50.770 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.755 7f1562b13700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f15641b0470 0x7f15640ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:48018/0 (socket says 172.21.2.102:48018) 2021-02-10T11:22:50.770 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.755 7f1562b13700 1 -- 172.21.2.102:0/2426530516 learned_addr learned my addr 172.21.2.102:0/2426530516 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:22:50.770 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.755 7f1561b11700 1 -- 172.21.2.102:0/2426530516 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2701694759 0 0) 0x7f154c001180 con 0x7f15641b0470 2021-02-10T11:22:50.770 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.755 7f1561b11700 1 -- 172.21.2.102:0/2426530516 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1550002fe0 con 0x7f15641b0470 2021-02-10T11:22:50.770 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.755 7f1563314700 1 --2- 172.21.2.102:0/2426530516 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f15640f05f0 0x7f15640f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:50.770 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.756 7f1561b11700 1 -- 172.21.2.102:0/2426530516 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1692145516 0 0) 0x7f154c001180 con 0x7f15641b0470 2021-02-10T11:22:50.770 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.756 7f1561b11700 1 -- 172.21.2.102:0/2426530516 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f15640f05f0 msgr2=0x7f15640f20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:50.771 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.756 7f1561b11700 1 --2- 172.21.2.102:0/2426530516 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f15640f05f0 0x7f15640f20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:50.771 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.756 7f1561b11700 1 -- 172.21.2.102:0/2426530516 >> v1:172.21.2.113:6789/0 conn(0x7f15641b0870 legacy=0x7f15640efe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:50.771 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.756 7f1561b11700 1 -- 172.21.2.102:0/2426530516 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15640e52e0 con 0x7f15641b0470 2021-02-10T11:22:50.771 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.756 7f1561b11700 1 -- 172.21.2.102:0/2426530516 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7f154c001680 con 0x7f15641b0470 2021-02-10T11:22:50.771 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.756 7f1561b11700 1 -- 172.21.2.102:0/2426530516 >> v1:172.21.2.102:6789/0 conn(0x7f15641b0470 legacy=0x7f15640ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:50.771 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.756 7f1561b11700 1 --2- 172.21.2.102:0/2426530516 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1550003ac0 0x7f1550005ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:50.771 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.756 7f1561b11700 1 --2- 172.21.2.102:0/2426530516 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1550006770 0x7f1550008c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:50.771 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.756 7f1561b11700 1 --2- 172.21.2.102:0/2426530516 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1550009500 0x7f155000ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:50.771 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.756 7f1561b11700 1 -- 172.21.2.102:0/2426530516 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f155000c2d0 con 0x7f1550009500 2021-02-10T11:22:50.771 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.756 7f1561b11700 1 -- 172.21.2.102:0/2426530516 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f155000c4f0 con 0x7f1550003ac0 2021-02-10T11:22:50.771 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.756 7f1561b11700 1 -- 172.21.2.102:0/2426530516 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f155000c710 con 0x7f1550006770 2021-02-10T11:22:50.771 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.756 7f1563314700 1 --2- 172.21.2.102:0/2426530516 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1550009500 0x7f155000ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:50.771 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.756 7f1562b13700 1 --2- 172.21.2.102:0/2426530516 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1550003ac0 0x7f1550005ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:50.771 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.756 7f1562312700 1 --2- 172.21.2.102:0/2426530516 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1550006770 0x7f1550008c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:50.772 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.756 7f1563314700 1 -- 172.21.2.102:0/2426530516 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1550006770 msgr2=0x7f1550008c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:50.772 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.756 7f1563314700 1 --2- 172.21.2.102:0/2426530516 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1550006770 0x7f1550008c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:50.772 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.756 7f1563314700 1 -- 172.21.2.102:0/2426530516 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1550003ac0 msgr2=0x7f1550005ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:50.772 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.756 7f1563314700 1 --2- 172.21.2.102:0/2426530516 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1550003ac0 0x7f1550005ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:50.772 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.756 7f1563314700 1 -- 172.21.2.102:0/2426530516 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1550050950 con 0x7f1550009500 2021-02-10T11:22:50.772 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.756 7f1563314700 1 --2- 172.21.2.102:0/2426530516 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1550009500 0x7f155000ba20 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 rx=0x7f155c009b10 tx=0x7f155c009eb0).ready entity=mon.0 client_cookie=ee93e4b224caf84f server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:50.772 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.757 7f1561b11700 1 -- 172.21.2.102:0/2426530516 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f155c00b510 con 0x7f1550009500 2021-02-10T11:22:50.772 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.757 7f1561b11700 1 -- 172.21.2.102:0/2426530516 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f155c003030 con 0x7f1550009500 2021-02-10T11:22:50.772 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.757 7f1561b11700 1 -- 172.21.2.102:0/2426530516 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f155c00b790 con 0x7f1550009500 2021-02-10T11:22:50.772 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.757 7f1568ed8700 1 -- 172.21.2.102:0/2426530516 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1550009500 msgr2=0x7f155000ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:50.772 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.757 7f1568ed8700 1 --2- 172.21.2.102:0/2426530516 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1550009500 0x7f155000ba20 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 rx=0x7f155c009b10 tx=0x7f155c009eb0).stop 2021-02-10T11:22:50.772 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.757 7f1563314700 1 -- 172.21.2.102:0/2426530516 reap_dead start 2021-02-10T11:22:50.772 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.758 7f1568ed8700 1 -- 172.21.2.102:0/2426530516 shutdown_connections 2021-02-10T11:22:50.773 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.758 7f1568ed8700 1 -- 172.21.2.102:0/2426530516 >> 172.21.2.102:0/2426530516 conn(0x7f15640ce3e0 msgr2=0x7f15640e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:50.773 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.758 7f1568ed8700 1 -- 172.21.2.102:0/2426530516 shutdown_connections 2021-02-10T11:22:50.773 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.758 7f1568ed8700 1 -- 172.21.2.102:0/2426530516 wait complete. 2021-02-10T11:22:50.773 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.772 7f1568ed8700 1 Processor -- start 2021-02-10T11:22:50.773 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.772 7f1568ed8700 1 -- start start 2021-02-10T11:22:50.774 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.773 7f1568ed8700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f15640b8530 0x7f15640b8930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:50.774 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.773 7f1568ed8700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f15640b8f50 0x7f15640b9350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:50.774 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.773 7f1568ed8700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f15640b65c0 0x7f15640b69c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:50.774 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.773 7f1568ed8700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15640b99f0 con 0x7f15640b65c0 2021-02-10T11:22:50.774 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.773 7f1568ed8700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15640b8020 con 0x7f15640b8530 2021-02-10T11:22:50.774 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.773 7f1568ed8700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1564091650 con 0x7f15640b8f50 2021-02-10T11:22:50.775 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.773 7f1563314700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f15640b65c0 0x7f15640b69c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:50.775 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.773 7f1563314700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f15640b65c0 0x7f15640b69c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:55808/0 (socket says 172.21.2.102:55808) 2021-02-10T11:22:50.775 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.773 7f1563314700 1 -- 172.21.2.102:0/2295709366 learned_addr learned my addr 172.21.2.102:0/2295709366 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:22:50.775 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.773 7f1563314700 1 -- 172.21.2.102:0/2295709366 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f15640b8f50 msgr2=0x7f15640b9350 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:22:50.775 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.773 7f1563314700 1 --2- 172.21.2.102:0/2295709366 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f15640b8f50 0x7f15640b9350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:50.775 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.773 7f1563314700 1 -- 172.21.2.102:0/2295709366 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f15640b8530 msgr2=0x7f15640b8930 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:22:50.775 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.773 7f1563314700 1 --2- 172.21.2.102:0/2295709366 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f15640b8530 0x7f15640b8930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:50.775 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.773 7f1563314700 1 -- 172.21.2.102:0/2295709366 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15640b6fe0 con 0x7f15640b65c0 2021-02-10T11:22:50.775 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.774 7f1563314700 1 --2- 172.21.2.102:0/2295709366 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f15640b65c0 0x7f15640b69c0 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 rx=0x7f155c0099f0 tx=0x7f155c00bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:50.775 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.774 7f155b7fe700 1 -- 172.21.2.102:0/2295709366 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f155c0093d0 con 0x7f15640b65c0 2021-02-10T11:22:50.775 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.774 7f155b7fe700 1 -- 172.21.2.102:0/2295709366 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f155c009020 con 0x7f15640b65c0 2021-02-10T11:22:50.775 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.774 7f155b7fe700 1 -- 172.21.2.102:0/2295709366 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f155c00c4e0 con 0x7f15640b65c0 2021-02-10T11:22:50.775 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.774 7f1568ed8700 1 -- 172.21.2.102:0/2295709366 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15640b6fe0 con 0x7f15640b65c0 2021-02-10T11:22:50.776 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.774 7f1568ed8700 1 -- 172.21.2.102:0/2295709366 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15640b7570 con 0x7f15640b65c0 2021-02-10T11:22:50.776 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.775 7f155b7fe700 1 -- 172.21.2.102:0/2295709366 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f155c009380 con 0x7f15640b65c0 2021-02-10T11:22:50.776 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.775 7f155b7fe700 1 --2- 172.21.2.102:0/2295709366 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f15480186c0 0x7f154801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:50.776 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.775 7f155b7fe700 1 -- 172.21.2.102:0/2295709366 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f155c032960 con 0x7f15640b65c0 2021-02-10T11:22:50.776 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.775 7f1562b13700 1 --2- 172.21.2.102:0/2295709366 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f15480186c0 0x7f154801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:50.776 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.775 7f1562b13700 1 --2- 172.21.2.102:0/2295709366 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f15480186c0 0x7f154801abc0 crc :-1 s=READY pgs=536 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:50.802 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.804 7f1568ed8700 1 -- 172.21.2.102:0/2295709366 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1564105a30 con 0x7f15640b65c0 2021-02-10T11:22:50.804 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.807 7f155b7fe700 1 -- 172.21.2.102:0/2295709366 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f155c001ee0 con 0x7f15640b65c0 2021-02-10T11:22:50.933 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.936 7f1568ed8700 1 -- 172.21.2.102:0/2295709366 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.guest1", "caps": ["mds", "allow *", "osd", "allow rw", "mon", "allow *"]} v 0) v1 -- 0x7f15640f0b50 con 0x7f15640b65c0 2021-02-10T11:22:50.937 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.940 7f155b7fe700 1 -- 172.21.2.102:0/2295709366 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.guest1", "caps": ["mds", "allow *", "osd", "allow rw", "mon", "allow *"]}]=0 v19) v1 ==== 158+0+64 (secure 0 0 0) 0x7f155c009380 con 0x7f15640b65c0 2021-02-10T11:22:50.941 INFO:teuthology.orchestra.run.gibba002.stdout:[client.guest1] 2021-02-10T11:22:50.941 INFO:teuthology.orchestra.run.gibba002.stdout: key = AQAKwiNgLvDsNxAAQDxLzGC+52prgobglejwyw== 2021-02-10T11:22:50.942 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.945 7f1568ed8700 1 -- 172.21.2.102:0/2295709366 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f15480186c0 msgr2=0x7f154801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:50.943 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.945 7f1568ed8700 1 --2- 172.21.2.102:0/2295709366 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f15480186c0 0x7f154801abc0 crc :-1 s=READY pgs=536 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:50.943 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.945 7f1568ed8700 1 -- 172.21.2.102:0/2295709366 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f15640b65c0 msgr2=0x7f15640b69c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:50.943 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.945 7f1568ed8700 1 --2- 172.21.2.102:0/2295709366 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f15640b65c0 0x7f15640b69c0 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 rx=0x7f155c0099f0 tx=0x7f155c00bcd0).stop 2021-02-10T11:22:50.943 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.945 7f1568ed8700 1 -- 172.21.2.102:0/2295709366 shutdown_connections 2021-02-10T11:22:50.943 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.945 7f1568ed8700 1 --2- 172.21.2.102:0/2295709366 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f15640b65c0 0x7f15640b69c0 unknown :-1 s=CLOSED pgs=1120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:50.943 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.945 7f1568ed8700 1 --2- 172.21.2.102:0/2295709366 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f15640b8f50 0x7f15640b9350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:50.943 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.945 7f1568ed8700 1 --2- 172.21.2.102:0/2295709366 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f15480186c0 0x7f154801abc0 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:50.943 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.945 7f1568ed8700 1 --2- 172.21.2.102:0/2295709366 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f15640b8530 0x7f15640b8930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:50.943 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.945 7f1568ed8700 1 -- 172.21.2.102:0/2295709366 >> 172.21.2.102:0/2295709366 conn(0x7f15640b2800 msgr2=0x7f1564091a50 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:50.943 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.945 7f1568ed8700 1 -- 172.21.2.102:0/2295709366 shutdown_connections 2021-02-10T11:22:50.943 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:50.945 7f1568ed8700 1 -- 172.21.2.102:0/2295709366 wait complete. 2021-02-10T11:22:50.951 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolumegroup create cephfs group_0000000000760661 2021-02-10T11:22:51.382 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:51.367 7f9cf50d4700 1 Processor -- start 2021-02-10T11:22:51.382 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:51.367 7f9cf50d4700 1 -- start start 2021-02-10T11:22:51.382 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:51.367 7f9cf50d4700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f9cf01b0470 0x7f9cf01b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:51.382 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:51.367 7f9cf50d4700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f9cf00ef610 0x7f9cf00efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:51.382 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:51.367 7f9cf50d4700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9cf00f0b50 con 0x7f9cf00f0030 2021-02-10T11:22:51.382 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:51.367 7f9cf50d4700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9cf00e52e0 con 0x7f9cf00ef610 2021-02-10T11:22:51.382 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:51.367 7f9cf50d4700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9cf00e5490 con 0x7f9cf01b0470 2021-02-10T11:22:51.383 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:51.368 7f9cee59c700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f9cf00ef610 0x7f9cf00efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:51.383 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:51.368 7f9cee59c700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f9cf00ef610 0x7f9cf00efa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:55814/0 (socket says 172.21.2.102:55814) 2021-02-10T11:22:51.383 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:51.368 7f9cee59c700 1 -- 172.21.2.102:0/3487055828 learned_addr learned my addr 172.21.2.102:0/3487055828 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:22:51.383 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:51.368 7f9ceed9d700 1 --2- 172.21.2.102:0/3487055828 >> v2:172.21.2.113:3300/0 conn(0x7f9cf01b0470 0x7f9cf01b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:51.383 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:51.368 7f9ceed9d700 1 -- 172.21.2.102:0/3487055828 >> v2:172.21.2.102:3300/0 conn(0x7f9cf00ef610 msgr2=0x7f9cf00efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:51.383 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:51.368 7f9ceed9d700 1 --2- 172.21.2.102:0/3487055828 >> v2:172.21.2.102:3300/0 conn(0x7f9cf00ef610 0x7f9cf00efa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:51.383 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:51.368 7f9ceed9d700 1 -- 172.21.2.102:0/3487055828 >> v1:172.21.2.113:6789/0 conn(0x7f9cf00f0030 legacy=0x7f9cf00f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:51.383 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:51.368 7f9ceed9d700 1 -- 172.21.2.102:0/3487055828 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cf00f0430 con 0x7f9cf01b0470 2021-02-10T11:22:51.383 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:51.368 7f9cee59c700 1 --2- 172.21.2.102:0/3487055828 >> v2:172.21.2.102:3300/0 conn(0x7f9cf00ef610 0x7f9cf00efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:22:51.383 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:51.369 7f9ceed9d700 1 --2- 172.21.2.102:0/3487055828 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f9cf01b0470 0x7f9cf01b0870 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 rx=0x7f9ce0006650 tx=0x7f9ce0009a80).ready entity=mon.1 client_cookie=b4deaea93bc20037 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:51.383 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:51.369 7f9cedd9b700 1 -- 172.21.2.102:0/3487055828 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9ce00009b0 con 0x7f9cf01b0470 2021-02-10T11:22:51.383 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:51.369 7f9cedd9b700 1 -- 172.21.2.102:0/3487055828 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ce000c0c0 con 0x7f9cf01b0470 2021-02-10T11:22:51.383 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:51.369 7f9cedd9b700 1 -- 172.21.2.102:0/3487055828 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9ce000c5e0 con 0x7f9cf01b0470 2021-02-10T11:22:51.383 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:51.369 7f9cf50d4700 1 -- 172.21.2.102:0/3487055828 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f9cf01b0470 msgr2=0x7f9cf01b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:51.384 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:51.369 7f9cf50d4700 1 --2- 172.21.2.102:0/3487055828 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f9cf01b0470 0x7f9cf01b0870 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 rx=0x7f9ce0006650 tx=0x7f9ce0009a80).stop 2021-02-10T11:22:51.384 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:51.370 7f9cf50d4700 1 -- 172.21.2.102:0/3487055828 shutdown_connections 2021-02-10T11:22:51.384 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:51.370 7f9cf50d4700 1 --2- 172.21.2.102:0/3487055828 >> v2:172.21.2.102:3300/0 conn(0x7f9cf00ef610 0x7f9cf00efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:51.384 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:51.370 7f9cf50d4700 1 --2- 172.21.2.102:0/3487055828 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f9cf01b0470 0x7f9cf01b0870 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:51.384 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:51.370 7f9cf50d4700 1 -- 172.21.2.102:0/3487055828 >> 172.21.2.102:0/3487055828 conn(0x7f9cf00ce3e0 msgr2=0x7f9cf00e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:51.384 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:51.370 7f9cf50d4700 1 -- 172.21.2.102:0/3487055828 shutdown_connections 2021-02-10T11:22:51.384 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:51.370 7f9cf50d4700 1 -- 172.21.2.102:0/3487055828 wait complete. 2021-02-10T11:22:51.384 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:51.385 7f9cf50d4700 1 Processor -- start 2021-02-10T11:22:51.384 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:51.385 7f9cf50d4700 1 -- start start 2021-02-10T11:22:51.385 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:51.385 7f9cf50d4700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9cf008ece0 0x7f9cf00a6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:51.385 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:51.385 7f9cf50d4700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9cf00a6860 0x7f9cf00a6c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:51.385 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:51.385 7f9cf50d4700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f9cf00a7280 0x7f9cf00a4590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:51.386 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:51.385 7f9cf50d4700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cf00a5cc0 con 0x7f9cf00a6860 2021-02-10T11:22:51.386 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:51.385 7f9cf50d4700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cf008f0e0 con 0x7f9cf00a7280 2021-02-10T11:22:51.386 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:51.385 7f9cf50d4700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9cf008f2c0 con 0x7f9cf008ece0 2021-02-10T11:22:51.386 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:51.385 7f9cee59c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9cf00a6860 0x7f9cf00a6c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:51.386 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:51.385 7f9cee59c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9cf00a6860 0x7f9cf00a6c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:55818/0 (socket says 172.21.2.102:55818) 2021-02-10T11:22:51.386 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:51.385 7f9cee59c700 1 -- 172.21.2.102:0/979668427 learned_addr learned my addr 172.21.2.102:0/979668427 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:22:51.386 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:51.385 7f9cef59e700 1 --2- 172.21.2.102:0/979668427 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f9cf00a7280 0x7f9cf00a4590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:51.386 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:51.385 7f9ceed9d700 1 --2- 172.21.2.102:0/979668427 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9cf008ece0 0x7f9cf00a6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:51.386 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:51.386 7f9cee59c700 1 -- 172.21.2.102:0/979668427 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9cf008ece0 msgr2=0x7f9cf00a6240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:51.387 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:51.386 7f9cee59c700 1 --2- 172.21.2.102:0/979668427 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9cf008ece0 0x7f9cf00a6240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:51.387 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:51.386 7f9cee59c700 1 -- 172.21.2.102:0/979668427 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f9cf00a7280 msgr2=0x7f9cf00a4590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:51.387 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:51.386 7f9cee59c700 1 --2- 172.21.2.102:0/979668427 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f9cf00a7280 0x7f9cf00a4590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:51.387 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:51.386 7f9cee59c700 1 -- 172.21.2.102:0/979668427 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cf00a4bb0 con 0x7f9cf00a6860 2021-02-10T11:22:51.387 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:51.386 7f9cee59c700 1 --2- 172.21.2.102:0/979668427 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9cf00a6860 0x7f9cf00a6c60 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 rx=0x7f9ce4006ee0 tx=0x7f9ce400b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:51.387 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:51.386 7f9cdf7fe700 1 -- 172.21.2.102:0/979668427 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9ce400bd50 con 0x7f9cf00a6860 2021-02-10T11:22:51.387 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:51.386 7f9cdf7fe700 1 -- 172.21.2.102:0/979668427 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ce40052e0 con 0x7f9cf00a6860 2021-02-10T11:22:51.387 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:51.386 7f9cf50d4700 1 -- 172.21.2.102:0/979668427 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9cf00a4bb0 con 0x7f9cf00a6860 2021-02-10T11:22:51.387 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:51.386 7f9cf50d4700 1 -- 172.21.2.102:0/979668427 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9cf00a5140 con 0x7f9cf00a6860 2021-02-10T11:22:51.387 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:51.386 7f9cdf7fe700 1 -- 172.21.2.102:0/979668427 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9ce4005800 con 0x7f9cf00a6860 2021-02-10T11:22:51.387 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:51.387 7f9cdf7fe700 1 -- 172.21.2.102:0/979668427 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f9ce40054c0 con 0x7f9cf00a6860 2021-02-10T11:22:51.387 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:51.387 7f9cdf7fe700 1 --2- 172.21.2.102:0/979668427 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f9cd4018700 0x7f9cd401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:51.387 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:51.387 7f9cdf7fe700 1 -- 172.21.2.102:0/979668427 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f9ce40327f0 con 0x7f9cf00a6860 2021-02-10T11:22:51.387 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:51.387 7f9ceed9d700 1 --2- 172.21.2.102:0/979668427 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f9cd4018700 0x7f9cd401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:51.388 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:51.388 7f9ceed9d700 1 --2- 172.21.2.102:0/979668427 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f9cd4018700 0x7f9cd401abc0 crc :-1 s=READY pgs=537 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:51.415 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:51.417 7f9cf50d4700 1 -- 172.21.2.102:0/979668427 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9cf019d1a0 con 0x7f9cf00a6860 2021-02-10T11:22:51.417 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:51.420 7f9cdf7fe700 1 -- 172.21.2.102:0/979668427 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f9ce40009b0 con 0x7f9cf00a6860 2021-02-10T11:22:51.544 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:51.547 7f9cf50d4700 1 -- 172.21.2.102:0/979668427 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "target": ["mgr", ""], "group_name": "group_0000000000760661"}) v1 -- 0x7f9cf00a5cc0 con 0x7f9cd4018700 2021-02-10T11:22:51.548 INFO:tasks.ceph.mgr.y.gibba002.stderr:2021-02-10 11:22:51.551 7fd94bc53700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:22:51.548 INFO:tasks.ceph.mgr.y.gibba002.stderr:2021-02-10 11:22:51.551 7fd94bc53700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:22:51.548 INFO:tasks.ceph.mgr.y.gibba002.stderr:2021-02-10 11:22:51.551 7fd94bc53700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:22:51.548 INFO:tasks.ceph.mgr.y.gibba002.stderr:2021-02-10 11:22:51.551 7fd94bc53700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:22:51.548 INFO:tasks.ceph.mgr.y.gibba002.stderr:2021-02-10 11:22:51.551 7fd94bc53700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:22:51.561 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:51.564 7f9cdf7fe700 1 -- 172.21.2.102:0/979668427 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f9ce0000c90 con 0x7f9cd4018700 2021-02-10T11:22:51.578 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:51.580 7f9cf50d4700 1 -- 172.21.2.102:0/979668427 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f9cd4018700 msgr2=0x7f9cd401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:51.578 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:51.580 7f9cf50d4700 1 --2- 172.21.2.102:0/979668427 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f9cd4018700 0x7f9cd401abc0 crc :-1 s=READY pgs=537 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:51.578 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:51.580 7f9cf50d4700 1 -- 172.21.2.102:0/979668427 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9cf00a6860 msgr2=0x7f9cf00a6c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:51.578 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:51.580 7f9cf50d4700 1 --2- 172.21.2.102:0/979668427 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9cf00a6860 0x7f9cf00a6c60 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 rx=0x7f9ce4006ee0 tx=0x7f9ce400b960).stop 2021-02-10T11:22:51.578 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:51.580 7f9cf50d4700 1 -- 172.21.2.102:0/979668427 shutdown_connections 2021-02-10T11:22:51.578 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:51.580 7f9cf50d4700 1 --2- 172.21.2.102:0/979668427 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f9cd4018700 0x7f9cd401abc0 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:51.578 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:51.580 7f9cf50d4700 1 --2- 172.21.2.102:0/979668427 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f9cf00a7280 0x7f9cf00a4590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:51.578 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:51.580 7f9cf50d4700 1 --2- 172.21.2.102:0/979668427 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9cf00a6860 0x7f9cf00a6c60 unknown :-1 s=CLOSED pgs=1121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:51.579 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:51.580 7f9cf50d4700 1 --2- 172.21.2.102:0/979668427 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9cf008ece0 0x7f9cf00a6240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:51.579 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:51.580 7f9cf50d4700 1 -- 172.21.2.102:0/979668427 >> 172.21.2.102:0/979668427 conn(0x7f9cf008ffe0 msgr2=0x7f9cf00bf050 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:51.579 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:51.580 7f9cf50d4700 1 -- 172.21.2.102:0/979668427 shutdown_connections 2021-02-10T11:22:51.579 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:51.580 7f9cf50d4700 1 -- 172.21.2.102:0/979668427 wait complete. 2021-02-10T11:22:51.586 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000793468 --group_name group_0000000000760661 2021-02-10T11:22:52.019 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:52.004 7f0917f2e700 1 Processor -- start 2021-02-10T11:22:52.019 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:52.005 7f0917f2e700 1 -- start start 2021-02-10T11:22:52.019 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:52.005 7f0917f2e700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f09100d1860 0x7f09101b03e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:52.019 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:52.005 7f0917f2e700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09100f0b00 con 0x7f09100f0640 2021-02-10T11:22:52.019 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:52.005 7f0917f2e700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09100d0300 con 0x7f09101b0a00 2021-02-10T11:22:52.019 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:52.005 7f0917f2e700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09100d0520 con 0x7f09100d1860 2021-02-10T11:22:52.019 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:52.005 7f0915d4a700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f09100d1860 0x7f09101b03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:52.019 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:52.005 7f091654b700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f09100f0640 0x7f09100cfb10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:48068/0 (socket says 172.21.2.102:48068) 2021-02-10T11:22:52.019 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:52.005 7f091654b700 1 -- 172.21.2.102:0/3260931719 learned_addr learned my addr 172.21.2.102:0/3260931719 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:22:52.019 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:52.006 7f0914d48700 1 -- 172.21.2.102:0/3260931719 <== mon.1 v1:172.21.2.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2762964885 0 0) 0x7f090c000f30 con 0x7f09101b0a00 2021-02-10T11:22:52.019 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:52.006 7f0915d4a700 1 -- 172.21.2.102:0/3260931719 >> v1:172.21.2.113:6789/0 conn(0x7f09101b0a00 legacy=0x7f09100cf320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:52.020 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:52.006 7f0915d4a700 1 -- 172.21.2.102:0/3260931719 >> v1:172.21.2.102:6789/0 conn(0x7f09100f0640 legacy=0x7f09100cfb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:52.020 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:52.006 7f0915d4a700 1 -- 172.21.2.102:0/3260931719 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09100d1c60 con 0x7f09100d1860 2021-02-10T11:22:52.020 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:52.006 7f0915d4a700 1 --2- 172.21.2.102:0/3260931719 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f09100d1860 0x7f09101b03e0 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 rx=0x7f0904006650 tx=0x7f0904009a80).ready entity=mon.0 client_cookie=61c7940b647418b8 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:52.020 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:52.006 7f0914d48700 1 -- 172.21.2.102:0/3260931719 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f090400c0c0 con 0x7f09100d1860 2021-02-10T11:22:52.020 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:52.006 7f0914d48700 1 -- 172.21.2.102:0/3260931719 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f090400c340 con 0x7f09100d1860 2021-02-10T11:22:52.020 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:52.006 7f0914d48700 1 -- 172.21.2.102:0/3260931719 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f090400c860 con 0x7f09100d1860 2021-02-10T11:22:52.020 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:52.006 7f0917f2e700 1 -- 172.21.2.102:0/3260931719 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f09100d1860 msgr2=0x7f09101b03e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:52.020 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:52.006 7f0917f2e700 1 --2- 172.21.2.102:0/3260931719 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f09100d1860 0x7f09101b03e0 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 rx=0x7f0904006650 tx=0x7f0904009a80).stop 2021-02-10T11:22:52.020 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:52.007 7f0917f2e700 1 -- 172.21.2.102:0/3260931719 shutdown_connections 2021-02-10T11:22:52.020 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:52.007 7f0917f2e700 1 --2- 172.21.2.102:0/3260931719 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f09100d1860 0x7f09101b03e0 unknown :-1 s=CLOSED pgs=1125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:52.020 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:52.007 7f0917f2e700 1 -- 172.21.2.102:0/3260931719 >> 172.21.2.102:0/3260931719 conn(0x7f09100ce440 msgr2=0x7f09100d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:52.020 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:52.007 7f0917f2e700 1 -- 172.21.2.102:0/3260931719 shutdown_connections 2021-02-10T11:22:52.020 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:52.007 7f0917f2e700 1 -- 172.21.2.102:0/3260931719 wait complete. 2021-02-10T11:22:52.022 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:52.022 7f0917f2e700 1 Processor -- start 2021-02-10T11:22:52.022 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:52.022 7f0917f2e700 1 -- start start 2021-02-10T11:22:52.022 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:52.022 7f0917f2e700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f09100998a0 0x7f0910099ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:52.022 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:52.022 7f0917f2e700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f091019f390 0x7f091009a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:52.022 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:52.022 7f0917f2e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0910095b90 0x7f0910095f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:52.022 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:52.022 7f0917f2e700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0910098ee0 con 0x7f0910095b90 2021-02-10T11:22:52.023 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:52.022 7f0917f2e700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09100965b0 con 0x7f09100998a0 2021-02-10T11:22:52.023 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:52.022 7f0917f2e700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09100967d0 con 0x7f091019f390 2021-02-10T11:22:52.023 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:52.022 7f091654b700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0910095b90 0x7f0910095f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:52.023 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:52.022 7f091654b700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0910095b90 0x7f0910095f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:55850/0 (socket says 172.21.2.102:55850) 2021-02-10T11:22:52.023 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:52.022 7f091654b700 1 -- 172.21.2.102:0/1984667002 learned_addr learned my addr 172.21.2.102:0/1984667002 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:22:52.023 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:52.022 7f0915549700 1 --2- 172.21.2.102:0/1984667002 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f091019f390 0x7f091009a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:52.023 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:52.022 7f0915d4a700 1 --2- 172.21.2.102:0/1984667002 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f09100998a0 0x7f0910099ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:52.023 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:52.023 7f091654b700 1 -- 172.21.2.102:0/1984667002 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f091019f390 msgr2=0x7f091009a2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:52.023 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:52.023 7f091654b700 1 --2- 172.21.2.102:0/1984667002 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f091019f390 0x7f091009a2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:52.023 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:52.023 7f091654b700 1 -- 172.21.2.102:0/1984667002 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f09100998a0 msgr2=0x7f0910099ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:52.023 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:52.023 7f091654b700 1 --2- 172.21.2.102:0/1984667002 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f09100998a0 0x7f0910099ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:52.023 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:52.023 7f091654b700 1 -- 172.21.2.102:0/1984667002 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09100969f0 con 0x7f0910095b90 2021-02-10T11:22:52.023 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:52.023 7f091654b700 1 --2- 172.21.2.102:0/1984667002 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0910095b90 0x7f0910095f90 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 rx=0x7f0908001bf0 tx=0x7f090800abd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:52.024 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:52.023 7f09027fc700 1 -- 172.21.2.102:0/1984667002 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0908000f00 con 0x7f0910095b90 2021-02-10T11:22:52.024 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:52.023 7f09027fc700 1 -- 172.21.2.102:0/1984667002 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0908007350 con 0x7f0910095b90 2021-02-10T11:22:52.024 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:52.023 7f0917f2e700 1 -- 172.21.2.102:0/1984667002 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09100969f0 con 0x7f0910095b90 2021-02-10T11:22:52.024 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:52.023 7f09027fc700 1 -- 172.21.2.102:0/1984667002 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0908004720 con 0x7f0910095b90 2021-02-10T11:22:52.024 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:52.023 7f0917f2e700 1 -- 172.21.2.102:0/1984667002 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09100969f0 con 0x7f0910095b90 2021-02-10T11:22:52.024 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:52.024 7f09027fc700 1 -- 172.21.2.102:0/1984667002 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f0908007530 con 0x7f0910095b90 2021-02-10T11:22:52.024 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:52.024 7f09027fc700 1 --2- 172.21.2.102:0/1984667002 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f08f8018730 0x7f08f801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:52.024 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:52.024 7f09027fc700 1 -- 172.21.2.102:0/1984667002 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f0908032990 con 0x7f0910095b90 2021-02-10T11:22:52.024 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:52.024 7f0915d4a700 1 --2- 172.21.2.102:0/1984667002 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f08f8018730 0x7f08f801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:52.024 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:52.024 7f0915d4a700 1 --2- 172.21.2.102:0/1984667002 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f08f8018730 0x7f08f801ac10 crc :-1 s=READY pgs=538 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:52.052 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:52.055 7f0917f2e700 1 -- 172.21.2.102:0/1984667002 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09100ce440 con 0x7f0910095b90 2021-02-10T11:22:52.054 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:52.057 7f09027fc700 1 -- 172.21.2.102:0/1984667002 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f0908007070 con 0x7f0910095b90 2021-02-10T11:22:52.180 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:52.183 7f0917f2e700 1 -- 172.21.2.102:0/1984667002 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000793468", "target": ["mgr", ""], "group_name": "group_0000000000760661"}) v1 -- 0x7f09100ce440 con 0x7f08f8018730 2021-02-10T11:22:52.204 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:52.207 7f09027fc700 1 -- 172.21.2.102:0/1984667002 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f0904000c90 con 0x7f08f8018730 2021-02-10T11:22:52.213 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:52.216 7f0917f2e700 1 -- 172.21.2.102:0/1984667002 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f08f8018730 msgr2=0x7f08f801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:52.214 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:52.216 7f0917f2e700 1 --2- 172.21.2.102:0/1984667002 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f08f8018730 0x7f08f801ac10 crc :-1 s=READY pgs=538 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:52.214 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:52.216 7f0917f2e700 1 -- 172.21.2.102:0/1984667002 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0910095b90 msgr2=0x7f0910095f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:52.214 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:52.216 7f0917f2e700 1 --2- 172.21.2.102:0/1984667002 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0910095b90 0x7f0910095f90 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 rx=0x7f0908001bf0 tx=0x7f090800abd0).stop 2021-02-10T11:22:52.214 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:52.216 7f0917f2e700 1 -- 172.21.2.102:0/1984667002 shutdown_connections 2021-02-10T11:22:52.214 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:52.216 7f0917f2e700 1 --2- 172.21.2.102:0/1984667002 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0910095b90 0x7f0910095f90 unknown :-1 s=CLOSED pgs=1126 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:52.214 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:52.216 7f0917f2e700 1 --2- 172.21.2.102:0/1984667002 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f091019f390 0x7f091009a2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:52.214 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:52.216 7f0917f2e700 1 --2- 172.21.2.102:0/1984667002 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f08f8018730 0x7f08f801ac10 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:52.214 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:52.216 7f0917f2e700 1 --2- 172.21.2.102:0/1984667002 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f09100998a0 0x7f0910099ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:52.214 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:52.216 7f0917f2e700 1 -- 172.21.2.102:0/1984667002 >> 172.21.2.102:0/1984667002 conn(0x7f091019f870 msgr2=0x7f0910094d10 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:52.214 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:52.216 7f0917f2e700 1 -- 172.21.2.102:0/1984667002 shutdown_connections 2021-02-10T11:22:52.214 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:52.217 7f0917f2e700 1 -- 172.21.2.102:0/1984667002 wait complete. 2021-02-10T11:22:52.222 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume authorize cephfs subvolume_0000000000793468 guest1 --group_name group_0000000000760661 --tenant_id tenant1 --allow-existing-id 2021-02-10T11:22:52.663 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:52.647 7fc83a4c4700 1 Processor -- start 2021-02-10T11:22:52.663 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:52.647 7fc83a4c4700 1 -- start start 2021-02-10T11:22:52.663 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:52.647 7fc83a4c4700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fc8340eb220 0x7fc8340eb620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:52.663 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:52.647 7fc83a4c4700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc83410a550 con 0x7fc83410aad0 2021-02-10T11:22:52.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:52.647 7fc83a4c4700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc834088180 con 0x7fc8340dce80 2021-02-10T11:22:52.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:52.647 7fc83a4c4700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8340d1ce0 con 0x7fc8340eb220 2021-02-10T11:22:52.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:52.648 7fc838ae1700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fc83410aad0 0x7fc83410aed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:48080/0 (socket says 172.21.2.102:48080) 2021-02-10T11:22:52.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:52.648 7fc838ae1700 1 -- 172.21.2.102:0/4013614485 learned_addr learned my addr 172.21.2.102:0/4013614485 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:22:52.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:52.648 7fc8337fe700 1 --2- 172.21.2.102:0/4013614485 >> v2:172.21.2.113:3300/0 conn(0x7fc8340eb220 0x7fc8340eb620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:52.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:52.648 7fc832ffd700 1 -- 172.21.2.102:0/4013614485 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 164877965 0 0) 0x7fc82c000d30 con 0x7fc83410aad0 2021-02-10T11:22:52.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:52.648 7fc832ffd700 1 -- 172.21.2.102:0/4013614485 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc81c002fe0 con 0x7fc83410aad0 2021-02-10T11:22:52.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:52.648 7fc832ffd700 1 -- 172.21.2.102:0/4013614485 <== mon.1 v1:172.21.2.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1079864353 0 0) 0x7fc824000c60 con 0x7fc8340dce80 2021-02-10T11:22:52.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:52.648 7fc832ffd700 1 -- 172.21.2.102:0/4013614485 --> v1:172.21.2.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc81c003210 con 0x7fc8340dce80 2021-02-10T11:22:52.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:52.648 7fc832ffd700 1 -- 172.21.2.102:0/4013614485 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2601560011 0 0) 0x7fc82c001500 con 0x7fc83410aad0 2021-02-10T11:22:52.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:52.649 7fc832ffd700 1 -- 172.21.2.102:0/4013614485 >> v2:172.21.2.113:3300/0 conn(0x7fc8340eb220 msgr2=0x7fc8340eb620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:52.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:52.649 7fc832ffd700 1 --2- 172.21.2.102:0/4013614485 >> v2:172.21.2.113:3300/0 conn(0x7fc8340eb220 0x7fc8340eb620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:52.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:52.649 7fc832ffd700 1 -- 172.21.2.102:0/4013614485 >> v1:172.21.2.113:6789/0 conn(0x7fc8340dce80 legacy=0x7fc8340d14f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:52.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:52.649 7fc832ffd700 1 -- 172.21.2.102:0/4013614485 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8340d1f60 con 0x7fc83410aad0 2021-02-10T11:22:52.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:52.649 7fc832ffd700 1 -- 172.21.2.102:0/4013614485 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7fc82c001a00 con 0x7fc83410aad0 2021-02-10T11:22:52.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:52.649 7fc832ffd700 1 -- 172.21.2.102:0/4013614485 >> v1:172.21.2.102:6789/0 conn(0x7fc83410aad0 legacy=0x7fc83410aed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:52.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:52.649 7fc832ffd700 1 --2- 172.21.2.102:0/4013614485 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc81c0037e0 0x7fc81c003c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:52.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:52.649 7fc832ffd700 1 --2- 172.21.2.102:0/4013614485 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc81c006860 0x7fc81c008d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:52.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:52.649 7fc832ffd700 1 --2- 172.21.2.102:0/4013614485 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc81c009620 0x7fc81c00bb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:52.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:52.649 7fc832ffd700 1 -- 172.21.2.102:0/4013614485 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc81c00c390 con 0x7fc81c0037e0 2021-02-10T11:22:52.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:52.649 7fc832ffd700 1 -- 172.21.2.102:0/4013614485 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc81c050580 con 0x7fc81c009620 2021-02-10T11:22:52.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:52.649 7fc832ffd700 1 -- 172.21.2.102:0/4013614485 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc81c0507a0 con 0x7fc81c006860 2021-02-10T11:22:52.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:52.649 7fc8337fe700 1 --2- 172.21.2.102:0/4013614485 >> v2:172.21.2.113:3300/0 conn(0x7fc8340eb220 0x7fc8340eb620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:22:52.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:52.649 7fc833fff700 1 --2- 172.21.2.102:0/4013614485 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc81c0037e0 0x7fc81c003c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:52.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:52.649 7fc838ae1700 1 --2- 172.21.2.102:0/4013614485 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc81c009620 0x7fc81c00bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:52.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:52.649 7fc8337fe700 1 --2- 172.21.2.102:0/4013614485 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc81c006860 0x7fc81c008d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:52.666 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:52.649 7fc838ae1700 1 -- 172.21.2.102:0/4013614485 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc81c006860 msgr2=0x7fc81c008d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:52.666 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:52.649 7fc838ae1700 1 --2- 172.21.2.102:0/4013614485 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc81c006860 0x7fc81c008d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:52.666 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:52.649 7fc838ae1700 1 -- 172.21.2.102:0/4013614485 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc81c0037e0 msgr2=0x7fc81c003c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:52.666 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:52.649 7fc838ae1700 1 --2- 172.21.2.102:0/4013614485 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc81c0037e0 0x7fc81c003c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:52.666 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:52.649 7fc838ae1700 1 -- 172.21.2.102:0/4013614485 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc81c050a40 con 0x7fc81c009620 2021-02-10T11:22:52.666 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:52.649 7fc833fff700 1 --2- 172.21.2.102:0/4013614485 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc81c0037e0 0x7fc81c003c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:22:52.666 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:52.649 7fc838ae1700 1 -- 172.21.2.102:0/4013614485 reap_dead start 2021-02-10T11:22:52.666 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:52.650 7fc838ae1700 1 --2- 172.21.2.102:0/4013614485 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc81c009620 0x7fc81c00bb40 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 rx=0x7fc82c007ce0 tx=0x7fc82c00a310).ready entity=mon.1 client_cookie=2fc44ff9c3acc090 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:52.666 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:52.650 7fc832ffd700 1 -- 172.21.2.102:0/4013614485 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc82c007d70 con 0x7fc81c009620 2021-02-10T11:22:52.666 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:52.650 7fc832ffd700 1 -- 172.21.2.102:0/4013614485 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc82c0070c0 con 0x7fc81c009620 2021-02-10T11:22:52.666 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:52.650 7fc832ffd700 1 -- 172.21.2.102:0/4013614485 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc82c007550 con 0x7fc81c009620 2021-02-10T11:22:52.666 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:52.650 7fc83a4c4700 1 -- 172.21.2.102:0/4013614485 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc81c009620 msgr2=0x7fc81c00bb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:52.667 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:52.650 7fc83a4c4700 1 --2- 172.21.2.102:0/4013614485 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc81c009620 0x7fc81c00bb40 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 rx=0x7fc82c007ce0 tx=0x7fc82c00a310).stop 2021-02-10T11:22:52.667 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:52.651 7fc83a4c4700 1 -- 172.21.2.102:0/4013614485 shutdown_connections 2021-02-10T11:22:52.667 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:52.651 7fc83a4c4700 1 --2- 172.21.2.102:0/4013614485 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc81c009620 0x7fc81c00bb40 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:52.667 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:52.651 7fc83a4c4700 1 -- 172.21.2.102:0/4013614485 >> 172.21.2.102:0/4013614485 conn(0x7fc8340ea630 msgr2=0x7fc8340d0800 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:52.667 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:52.651 7fc83a4c4700 1 -- 172.21.2.102:0/4013614485 shutdown_connections 2021-02-10T11:22:52.667 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:52.651 7fc83a4c4700 1 -- 172.21.2.102:0/4013614485 wait complete. 2021-02-10T11:22:52.667 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:52.666 7fc83a4c4700 1 Processor -- start 2021-02-10T11:22:52.667 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:52.666 7fc83a4c4700 1 -- start start 2021-02-10T11:22:52.670 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:52.666 7fc83a4c4700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc834093450 0x7fc8341064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:52.670 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:52.666 7fc83a4c4700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc834093850 0x7fc8340ffb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:52.670 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:52.666 7fc83a4c4700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc834100150 0x7fc834100df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:52.671 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:52.666 7fc83a4c4700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc834100550 con 0x7fc834093450 2021-02-10T11:22:52.671 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:52.666 7fc83a4c4700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc834093d70 con 0x7fc834100150 2021-02-10T11:22:52.671 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:52.666 7fc83a4c4700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc834101410 con 0x7fc834093850 2021-02-10T11:22:52.671 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:52.666 7fc833fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc834093450 0x7fc8341064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:52.671 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:52.666 7fc833fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc834093450 0x7fc8341064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:55866/0 (socket says 172.21.2.102:55866) 2021-02-10T11:22:52.671 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:52.666 7fc833fff700 1 -- 172.21.2.102:0/1275319053 learned_addr learned my addr 172.21.2.102:0/1275319053 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:22:52.671 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:52.667 7fc8337fe700 1 --2- 172.21.2.102:0/1275319053 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc834093850 0x7fc8340ffb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:52.671 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:52.667 7fc838ae1700 1 --2- 172.21.2.102:0/1275319053 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc834100150 0x7fc834100df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:52.671 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:52.667 7fc833fff700 1 -- 172.21.2.102:0/1275319053 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc834093850 msgr2=0x7fc8340ffb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:52.671 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:52.667 7fc833fff700 1 --2- 172.21.2.102:0/1275319053 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc834093850 0x7fc8340ffb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:52.671 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:52.667 7fc833fff700 1 -- 172.21.2.102:0/1275319053 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc834100150 msgr2=0x7fc834100df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:52.671 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:52.667 7fc833fff700 1 --2- 172.21.2.102:0/1275319053 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc834100150 0x7fc834100df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:52.671 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:52.667 7fc833fff700 1 -- 172.21.2.102:0/1275319053 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc834101630 con 0x7fc834093450 2021-02-10T11:22:52.672 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:52.667 7fc833fff700 1 --2- 172.21.2.102:0/1275319053 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc834093450 0x7fc8341064b0 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 rx=0x7fc824006f40 tx=0x7fc82400b9b0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:52.672 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:52.667 7fc830ff9700 1 -- 172.21.2.102:0/1275319053 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc82400bda0 con 0x7fc834093450 2021-02-10T11:22:52.672 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:52.667 7fc830ff9700 1 -- 172.21.2.102:0/1275319053 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc824006030 con 0x7fc834093450 2021-02-10T11:22:52.672 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:52.667 7fc830ff9700 1 -- 172.21.2.102:0/1275319053 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc824005540 con 0x7fc834093450 2021-02-10T11:22:52.672 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:52.667 7fc83a4c4700 1 -- 172.21.2.102:0/1275319053 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc834101630 con 0x7fc834093450 2021-02-10T11:22:52.672 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:52.668 7fc83a4c4700 1 -- 172.21.2.102:0/1275319053 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc834101630 con 0x7fc834093450 2021-02-10T11:22:52.672 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:52.668 7fc830ff9700 1 -- 172.21.2.102:0/1275319053 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fc824006210 con 0x7fc834093450 2021-02-10T11:22:52.672 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:52.668 7fc830ff9700 1 --2- 172.21.2.102:0/1275319053 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc8140186c0 0x7fc81401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:52.672 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:52.668 7fc830ff9700 1 -- 172.21.2.102:0/1275319053 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fc824032440 con 0x7fc834093450 2021-02-10T11:22:52.672 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:52.668 7fc8337fe700 1 --2- 172.21.2.102:0/1275319053 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc8140186c0 0x7fc81401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:52.672 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:52.669 7fc8337fe700 1 --2- 172.21.2.102:0/1275319053 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc8140186c0 0x7fc81401abc0 crc :-1 s=READY pgs=539 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:52.695 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:52.697 7fc83a4c4700 1 -- 172.21.2.102:0/1275319053 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc834100550 con 0x7fc834093450 2021-02-10T11:22:52.697 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:52.700 7fc830ff9700 1 -- 172.21.2.102:0/1275319053 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fc824030d70 con 0x7fc834093450 2021-02-10T11:22:52.824 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:52.827 7fc83a4c4700 1 -- 172.21.2.102:0/1275319053 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"allow_existing_id": true, "vol_name": "cephfs", "sub_name": "subvolume_0000000000793468", "target": ["mgr", ""], "tenant_id": "tenant1", "group_name": "group_0000000000760661", "prefix": "fs subvolume authorize", "auth_id": "guest1"}) v1 -- 0x7fc834100550 con 0x7fc8140186c0 2021-02-10T11:22:52.830 INFO:tasks.ceph.mgr.y.gibba002.stderr:Creating meta for ID guest1 with tenant tenant1 2021-02-10T11:22:52.858 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:52.861 7fc830ff9700 1 -- 172.21.2.102:0/1275319053 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+41 (crc 0 0 0) 0x7fc8280069d0 con 0x7fc8140186c0 2021-02-10T11:22:52.888 INFO:teuthology.orchestra.run.gibba002.stdout:AQAKwiNgLvDsNxAAQDxLzGC+52prgobglejwyw== 2021-02-10T11:22:52.889 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:52.892 7fc83a4c4700 1 -- 172.21.2.102:0/1275319053 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc8140186c0 msgr2=0x7fc81401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:52.889 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:52.892 7fc83a4c4700 1 --2- 172.21.2.102:0/1275319053 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc8140186c0 0x7fc81401abc0 crc :-1 s=READY pgs=539 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:52.889 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:52.892 7fc83a4c4700 1 -- 172.21.2.102:0/1275319053 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc834093450 msgr2=0x7fc8341064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:52.889 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:52.892 7fc83a4c4700 1 --2- 172.21.2.102:0/1275319053 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc834093450 0x7fc8341064b0 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 rx=0x7fc824006f40 tx=0x7fc82400b9b0).stop 2021-02-10T11:22:52.889 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:52.892 7fc83a4c4700 1 -- 172.21.2.102:0/1275319053 shutdown_connections 2021-02-10T11:22:52.890 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:52.892 7fc83a4c4700 1 --2- 172.21.2.102:0/1275319053 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc8140186c0 0x7fc81401abc0 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:52.890 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:52.892 7fc83a4c4700 1 --2- 172.21.2.102:0/1275319053 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc834100150 0x7fc834100df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:52.890 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:52.892 7fc83a4c4700 1 --2- 172.21.2.102:0/1275319053 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc834093850 0x7fc8340ffb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:52.890 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:52.892 7fc83a4c4700 1 --2- 172.21.2.102:0/1275319053 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc834093450 0x7fc8341064b0 unknown :-1 s=CLOSED pgs=1128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:52.890 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:52.892 7fc83a4c4700 1 -- 172.21.2.102:0/1275319053 >> 172.21.2.102:0/1275319053 conn(0x7fc8340b0b50 msgr2=0x7fc8340bbf00 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:52.890 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:52.892 7fc83a4c4700 1 -- 172.21.2.102:0/1275319053 shutdown_connections 2021-02-10T11:22:52.890 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:52.892 7fc83a4c4700 1 -- 172.21.2.102:0/1275319053 wait complete. 2021-02-10T11:22:52.898 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume deauthorize cephfs subvolume_0000000000793468 guest1 --group_name group_0000000000760661 2021-02-10T11:22:53.329 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:53.313 7f5b350f5700 1 Processor -- start 2021-02-10T11:22:53.329 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:53.313 7f5b350f5700 1 -- start start 2021-02-10T11:22:53.329 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:53.314 7f5b350f5700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5b3010afe0 0x7f5b300cf740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:53.329 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:53.314 7f5b350f5700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b300fe1b0 con 0x7f5b300d1860 2021-02-10T11:22:53.330 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:53.314 7f5b350f5700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b301b0ca0 con 0x7f5b300d1c60 2021-02-10T11:22:53.330 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:53.314 7f5b350f5700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b30109a50 con 0x7f5b3010afe0 2021-02-10T11:22:53.330 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:53.314 7f5b2ed9d700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f5b300d1860 0x7f5b301b04b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:48100/0 (socket says 172.21.2.102:48100) 2021-02-10T11:22:53.330 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:53.314 7f5b2ed9d700 1 -- 172.21.2.102:0/3082360404 learned_addr learned my addr 172.21.2.102:0/3082360404 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:22:53.330 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:53.315 7f5b2dd9b700 1 -- 172.21.2.102:0/3082360404 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3428218374 0 0) 0x7f5b18001060 con 0x7f5b300d1860 2021-02-10T11:22:53.330 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:53.315 7f5b2dd9b700 1 -- 172.21.2.102:0/3082360404 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b1c002fe0 con 0x7f5b300d1860 2021-02-10T11:22:53.330 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:53.315 7f5b2f59e700 1 --2- 172.21.2.102:0/3082360404 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5b3010afe0 0x7f5b300cf740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:53.330 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:53.315 7f5b2dd9b700 1 -- 172.21.2.102:0/3082360404 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2066112224 0 0) 0x7f5b18001500 con 0x7f5b300d1860 2021-02-10T11:22:53.330 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:53.315 7f5b2dd9b700 1 -- 172.21.2.102:0/3082360404 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5b3010afe0 msgr2=0x7f5b300cf740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:53.330 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:53.315 7f5b2dd9b700 1 --2- 172.21.2.102:0/3082360404 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5b3010afe0 0x7f5b300cf740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:53.330 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:53.315 7f5b2dd9b700 1 -- 172.21.2.102:0/3082360404 >> v1:172.21.2.113:6789/0 conn(0x7f5b300d1c60 legacy=0x7f5b300cef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:53.330 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:53.315 7f5b2dd9b700 1 -- 172.21.2.102:0/3082360404 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b3010b3e0 con 0x7f5b300d1860 2021-02-10T11:22:53.330 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:53.315 7f5b2dd9b700 1 -- 172.21.2.102:0/3082360404 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7f5b18001a00 con 0x7f5b300d1860 2021-02-10T11:22:53.331 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:53.315 7f5b2dd9b700 1 -- 172.21.2.102:0/3082360404 >> v1:172.21.2.102:6789/0 conn(0x7f5b300d1860 legacy=0x7f5b301b04b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:53.331 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:53.315 7f5b2dd9b700 1 --2- 172.21.2.102:0/3082360404 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5b1c003b10 0x7f5b1c005f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:53.331 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:53.315 7f5b2dd9b700 1 --2- 172.21.2.102:0/3082360404 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5b1c006820 0x7f5b1c008d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:53.331 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:53.315 7f5b2dd9b700 1 --2- 172.21.2.102:0/3082360404 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5b1c009620 0x7f5b1c00bb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:53.331 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:53.315 7f5b2dd9b700 1 -- 172.21.2.102:0/3082360404 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b1c00c360 con 0x7f5b1c006820 2021-02-10T11:22:53.331 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:53.315 7f5b2dd9b700 1 -- 172.21.2.102:0/3082360404 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b1c00c580 con 0x7f5b1c003b10 2021-02-10T11:22:53.331 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:53.315 7f5b2dd9b700 1 -- 172.21.2.102:0/3082360404 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b1c00c7a0 con 0x7f5b1c009620 2021-02-10T11:22:53.331 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:53.315 7f5b2f59e700 1 --2- 172.21.2.102:0/3082360404 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5b3010afe0 0x7f5b300cf740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:22:53.331 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:53.315 7f5b2ed9d700 1 --2- 172.21.2.102:0/3082360404 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5b1c003b10 0x7f5b1c005f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:53.331 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:53.315 7f5b2e59c700 1 --2- 172.21.2.102:0/3082360404 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5b1c006820 0x7f5b1c008d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:53.331 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:53.316 7f5b2f59e700 1 --2- 172.21.2.102:0/3082360404 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5b1c009620 0x7f5b1c00bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:53.331 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:53.316 7f5b2ed9d700 1 -- 172.21.2.102:0/3082360404 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5b1c009620 msgr2=0x7f5b1c00bb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:53.331 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:53.316 7f5b2ed9d700 1 --2- 172.21.2.102:0/3082360404 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5b1c009620 0x7f5b1c00bb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:53.332 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:53.316 7f5b2ed9d700 1 -- 172.21.2.102:0/3082360404 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5b1c006820 msgr2=0x7f5b1c008d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:53.332 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:53.316 7f5b2ed9d700 1 --2- 172.21.2.102:0/3082360404 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5b1c006820 0x7f5b1c008d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:53.332 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:53.316 7f5b2ed9d700 1 -- 172.21.2.102:0/3082360404 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b1c00ca40 con 0x7f5b1c003b10 2021-02-10T11:22:53.332 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:53.316 7f5b2e59c700 1 --2- 172.21.2.102:0/3082360404 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5b1c006820 0x7f5b1c008d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:22:53.332 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:53.316 7f5b2ed9d700 1 --2- 172.21.2.102:0/3082360404 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5b1c003b10 0x7f5b1c005f80 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 rx=0x7f5b18001bb0 tx=0x7f5b1804e310).ready entity=mon.1 client_cookie=78cc2c00e58aff71 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:53.332 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:53.316 7f5b2dd9b700 1 -- 172.21.2.102:0/3082360404 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5b18004720 con 0x7f5b1c003b10 2021-02-10T11:22:53.332 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:53.316 7f5b2dd9b700 1 -- 172.21.2.102:0/3082360404 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b18001050 con 0x7f5b1c003b10 2021-02-10T11:22:53.332 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:53.316 7f5b2dd9b700 1 -- 172.21.2.102:0/3082360404 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5b18004c20 con 0x7f5b1c003b10 2021-02-10T11:22:53.332 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:53.316 7f5b350f5700 1 -- 172.21.2.102:0/3082360404 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5b1c003b10 msgr2=0x7f5b1c005f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:53.332 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:53.316 7f5b350f5700 1 --2- 172.21.2.102:0/3082360404 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5b1c003b10 0x7f5b1c005f80 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 rx=0x7f5b18001bb0 tx=0x7f5b1804e310).stop 2021-02-10T11:22:53.332 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:53.316 7f5b2f59e700 1 -- 172.21.2.102:0/3082360404 reap_dead start 2021-02-10T11:22:53.332 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:53.317 7f5b350f5700 1 -- 172.21.2.102:0/3082360404 shutdown_connections 2021-02-10T11:22:53.332 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:53.317 7f5b350f5700 1 -- 172.21.2.102:0/3082360404 >> 172.21.2.102:0/3082360404 conn(0x7f5b300ce410 msgr2=0x7f5b300d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:53.333 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:53.317 7f5b350f5700 1 -- 172.21.2.102:0/3082360404 shutdown_connections 2021-02-10T11:22:53.333 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:53.317 7f5b350f5700 1 -- 172.21.2.102:0/3082360404 wait complete. 2021-02-10T11:22:53.333 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:53.332 7f5b350f5700 1 Processor -- start 2021-02-10T11:22:53.333 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:53.332 7f5b350f5700 1 -- start start 2021-02-10T11:22:53.334 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:53.332 7f5b350f5700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5b301a49c0 0x7f5b3008c0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:53.334 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:53.332 7f5b350f5700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5b3008c6d0 0x7f5b300b4070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:53.334 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:53.332 7f5b350f5700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5b300b4690 0x7f5b3007e9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:53.334 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:53.332 7f5b350f5700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b3008cb50 con 0x7f5b3008c6d0 2021-02-10T11:22:53.334 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:53.332 7f5b350f5700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b300b4a90 con 0x7f5b301a49c0 2021-02-10T11:22:53.334 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:53.332 7f5b350f5700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b300b4cb0 con 0x7f5b300b4690 2021-02-10T11:22:53.335 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:53.333 7f5b2e59c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5b3008c6d0 0x7f5b300b4070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:53.335 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:53.333 7f5b2e59c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5b3008c6d0 0x7f5b300b4070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:55886/0 (socket says 172.21.2.102:55886) 2021-02-10T11:22:53.335 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:53.333 7f5b2e59c700 1 -- 172.21.2.102:0/335499808 learned_addr learned my addr 172.21.2.102:0/335499808 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:22:53.335 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:53.333 7f5b2e59c700 1 -- 172.21.2.102:0/335499808 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5b300b4690 msgr2=0x7f5b3007e9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:53.335 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:53.333 7f5b2f59e700 1 --2- 172.21.2.102:0/335499808 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5b300b4690 0x7f5b3007e9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:53.335 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:53.333 7f5b2ed9d700 1 --2- 172.21.2.102:0/335499808 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5b301a49c0 0x7f5b3008c0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:53.335 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:53.333 7f5b2e59c700 1 --2- 172.21.2.102:0/335499808 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5b300b4690 0x7f5b3007e9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:53.335 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:53.333 7f5b2e59c700 1 -- 172.21.2.102:0/335499808 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5b301a49c0 msgr2=0x7f5b3008c0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:53.335 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:53.333 7f5b2e59c700 1 --2- 172.21.2.102:0/335499808 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5b301a49c0 0x7f5b3008c0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:53.335 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:53.333 7f5b2e59c700 1 -- 172.21.2.102:0/335499808 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b3007efe0 con 0x7f5b3008c6d0 2021-02-10T11:22:53.335 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:53.333 7f5b2f59e700 1 --2- 172.21.2.102:0/335499808 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5b300b4690 0x7f5b3007e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:22:53.335 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:53.333 7f5b2ed9d700 1 --2- 172.21.2.102:0/335499808 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5b301a49c0 0x7f5b3008c0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:22:53.335 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:53.333 7f5b2e59c700 1 --2- 172.21.2.102:0/335499808 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5b3008c6d0 0x7f5b300b4070 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 rx=0x7f5b2000b610 tx=0x7f5b2000b9b0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:53.336 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:53.334 7f5b277fe700 1 -- 172.21.2.102:0/335499808 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5b20006bc0 con 0x7f5b3008c6d0 2021-02-10T11:22:53.336 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:53.334 7f5b277fe700 1 -- 172.21.2.102:0/335499808 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b2000bda0 con 0x7f5b3008c6d0 2021-02-10T11:22:53.336 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:53.334 7f5b350f5700 1 -- 172.21.2.102:0/335499808 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b3007efe0 con 0x7f5b3008c6d0 2021-02-10T11:22:53.336 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:53.334 7f5b350f5700 1 -- 172.21.2.102:0/335499808 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b3007f570 con 0x7f5b3008c6d0 2021-02-10T11:22:53.336 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:53.334 7f5b277fe700 1 -- 172.21.2.102:0/335499808 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5b200054a0 con 0x7f5b3008c6d0 2021-02-10T11:22:53.336 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:53.334 7f5b277fe700 1 -- 172.21.2.102:0/335499808 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f5b20006030 con 0x7f5b3008c6d0 2021-02-10T11:22:53.336 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:53.335 7f5b277fe700 1 --2- 172.21.2.102:0/335499808 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5b14018730 0x7f5b1401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:53.336 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:53.335 7f5b277fe700 1 -- 172.21.2.102:0/335499808 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f5b20032800 con 0x7f5b3008c6d0 2021-02-10T11:22:53.336 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:53.335 7f5b2ed9d700 1 --2- 172.21.2.102:0/335499808 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5b14018730 0x7f5b1401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:53.336 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:53.335 7f5b2ed9d700 1 --2- 172.21.2.102:0/335499808 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5b14018730 0x7f5b1401ac10 crc :-1 s=READY pgs=540 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:53.362 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:53.364 7f5b350f5700 1 -- 172.21.2.102:0/335499808 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b300fe1b0 con 0x7f5b3008c6d0 2021-02-10T11:22:53.364 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:53.367 7f5b277fe700 1 -- 172.21.2.102:0/335499808 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f5b20000940 con 0x7f5b3008c6d0 2021-02-10T11:22:53.490 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:53.493 7f5b350f5700 1 -- 172.21.2.102:0/335499808 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume deauthorize", "vol_name": "cephfs", "sub_name": "subvolume_0000000000793468", "target": ["mgr", ""], "auth_id": "guest1", "group_name": "group_0000000000760661"}) v1 -- 0x7f5b300fe1b0 con 0x7f5b14018730 2021-02-10T11:22:53.524 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:53.527 7f5b277fe700 1 -- 172.21.2.102:0/335499808 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f5b18001500 con 0x7f5b14018730 2021-02-10T11:22:53.556 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:53.558 7f5b350f5700 1 -- 172.21.2.102:0/335499808 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5b14018730 msgr2=0x7f5b1401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:53.556 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:53.558 7f5b350f5700 1 --2- 172.21.2.102:0/335499808 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5b14018730 0x7f5b1401ac10 crc :-1 s=READY pgs=540 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:53.556 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:53.558 7f5b350f5700 1 -- 172.21.2.102:0/335499808 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5b3008c6d0 msgr2=0x7f5b300b4070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:53.556 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:53.558 7f5b350f5700 1 --2- 172.21.2.102:0/335499808 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5b3008c6d0 0x7f5b300b4070 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 rx=0x7f5b2000b610 tx=0x7f5b2000b9b0).stop 2021-02-10T11:22:53.556 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:53.558 7f5b350f5700 1 -- 172.21.2.102:0/335499808 shutdown_connections 2021-02-10T11:22:53.557 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:53.558 7f5b350f5700 1 --2- 172.21.2.102:0/335499808 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5b300b4690 0x7f5b3007e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:53.557 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:53.558 7f5b350f5700 1 --2- 172.21.2.102:0/335499808 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5b3008c6d0 0x7f5b300b4070 unknown :-1 s=CLOSED pgs=1130 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:53.557 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:53.558 7f5b350f5700 1 --2- 172.21.2.102:0/335499808 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5b14018730 0x7f5b1401ac10 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:53.557 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:53.558 7f5b350f5700 1 --2- 172.21.2.102:0/335499808 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5b301a49c0 0x7f5b3008c0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:53.557 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:53.558 7f5b350f5700 1 -- 172.21.2.102:0/335499808 >> 172.21.2.102:0/335499808 conn(0x7f5b300985a0 msgr2=0x7f5b300beed0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:53.557 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:53.558 7f5b350f5700 1 -- 172.21.2.102:0/335499808 shutdown_connections 2021-02-10T11:22:53.557 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:53.558 7f5b350f5700 1 -- 172.21.2.102:0/335499808 wait complete. 2021-02-10T11:22:53.564 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth rm client.guest1 2021-02-10T11:22:53.990 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:53.976 7f8ffec26700 1 Processor -- start 2021-02-10T11:22:53.991 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:53.977 7f8ffec26700 1 -- start start 2021-02-10T11:22:53.991 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:53.977 7f8ffec26700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f8ff80e5070 0x7f8ff81b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:53.991 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:53.977 7f8ffec26700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8ff81b3e40 0x7f8ff807d690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:53.991 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:53.977 7f8ffec26700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ff807ed80 con 0x7f8ff80e5470 2021-02-10T11:22:53.991 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:53.977 7f8ffec26700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ff8080310 con 0x7f8ff80e5070 2021-02-10T11:22:53.991 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:53.977 7f8ffec26700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ff8080500 con 0x7f8ff81b3e40 2021-02-10T11:22:53.991 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:53.977 7f8ffca42700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f8ff80e5070 0x7f8ff81b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:53.991 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:53.977 7f8ffca42700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f8ff80e5070 0x7f8ff81b3820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:55892/0 (socket says 172.21.2.102:55892) 2021-02-10T11:22:53.991 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:53.977 7f8ffca42700 1 -- 172.21.2.102:0/4130699870 learned_addr learned my addr 172.21.2.102:0/4130699870 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:22:53.991 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:53.978 7f8ffca42700 1 -- 172.21.2.102:0/4130699870 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8ff81b3e40 msgr2=0x7f8ff807d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:53.992 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:53.978 7f8ffca42700 1 --2- 172.21.2.102:0/4130699870 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8ff81b3e40 0x7f8ff807d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:53.992 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:53.978 7f8ffca42700 1 -- 172.21.2.102:0/4130699870 >> v1:172.21.2.113:6789/0 conn(0x7f8ff80e5470 legacy=0x7f8ff807dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:53.992 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:53.978 7f8ffca42700 1 -- 172.21.2.102:0/4130699870 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ff80806f0 con 0x7f8ff80e5070 2021-02-10T11:22:53.992 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:53.978 7f8ffca42700 1 --2- 172.21.2.102:0/4130699870 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8ff80e5070 0x7f8ff81b3820 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 rx=0x7f8fe0006700 tx=0x7f8fe0009b80).ready entity=mon.0 client_cookie=c8772da9eed48f42 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:53.992 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:53.978 7f8ff77fe700 1 -- 172.21.2.102:0/4130699870 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8fe000c0c0 con 0x7f8ff80e5070 2021-02-10T11:22:53.992 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:53.978 7f8ff77fe700 1 -- 172.21.2.102:0/4130699870 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fe000c440 con 0x7f8ff80e5070 2021-02-10T11:22:53.992 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:53.978 7f8ff77fe700 1 -- 172.21.2.102:0/4130699870 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8fe000c920 con 0x7f8ff80e5070 2021-02-10T11:22:53.992 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:53.978 7f8ffec26700 1 -- 172.21.2.102:0/4130699870 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8ff80e5070 msgr2=0x7f8ff81b3820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:53.992 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:53.978 7f8ffec26700 1 --2- 172.21.2.102:0/4130699870 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8ff80e5070 0x7f8ff81b3820 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 rx=0x7f8fe0006700 tx=0x7f8fe0009b80).stop 2021-02-10T11:22:53.992 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:53.979 7f8ffec26700 1 -- 172.21.2.102:0/4130699870 shutdown_connections 2021-02-10T11:22:53.992 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:53.979 7f8ffec26700 1 --2- 172.21.2.102:0/4130699870 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8ff81b3e40 0x7f8ff807d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:53.992 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:53.979 7f8ffec26700 1 --2- 172.21.2.102:0/4130699870 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8ff80e5070 0x7f8ff81b3820 unknown :-1 s=CLOSED pgs=1131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:53.992 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:53.979 7f8ffec26700 1 -- 172.21.2.102:0/4130699870 >> 172.21.2.102:0/4130699870 conn(0x7f8ff80ce690 msgr2=0x7f8ff80e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:53.993 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:53.979 7f8ffec26700 1 -- 172.21.2.102:0/4130699870 shutdown_connections 2021-02-10T11:22:53.993 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:53.979 7f8ffec26700 1 -- 172.21.2.102:0/4130699870 wait complete. 2021-02-10T11:22:53.994 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:53.994 7f8ffec26700 1 Processor -- start 2021-02-10T11:22:53.995 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:53.994 7f8ffec26700 1 -- start start 2021-02-10T11:22:53.995 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:53.994 7f8ffec26700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8ff80b07a0 0x7f8ff819ec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:53.995 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:53.994 7f8ffec26700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8ff819f2b0 0x7f8ff80a0e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:53.995 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:53.994 7f8ffec26700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8ff80a1490 0x7f8ff809fcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:53.995 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:53.994 7f8ffec26700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ff80b0ba0 con 0x7f8ff80b07a0 2021-02-10T11:22:53.995 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:53.994 7f8ffec26700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ff80a1890 con 0x7f8ff80a1490 2021-02-10T11:22:53.995 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:53.994 7f8ffec26700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ff80a1ab0 con 0x7f8ff819f2b0 2021-02-10T11:22:53.995 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:53.994 7f8ffca42700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8ff80b07a0 0x7f8ff819ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:53.995 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:53.994 7f8ffca42700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8ff80b07a0 0x7f8ff819ec90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:55896/0 (socket says 172.21.2.102:55896) 2021-02-10T11:22:53.995 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:53.994 7f8ffca42700 1 -- 172.21.2.102:0/1923105757 learned_addr learned my addr 172.21.2.102:0/1923105757 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:22:53.995 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:53.995 7f8ff7fff700 1 --2- 172.21.2.102:0/1923105757 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8ff819f2b0 0x7f8ff80a0e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:53.995 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:53.995 7f8ffd243700 1 --2- 172.21.2.102:0/1923105757 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8ff80a1490 0x7f8ff809fcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:53.996 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:53.995 7f8ffca42700 1 -- 172.21.2.102:0/1923105757 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8ff819f2b0 msgr2=0x7f8ff80a0e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:53.996 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:53.995 7f8ffca42700 1 --2- 172.21.2.102:0/1923105757 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8ff819f2b0 0x7f8ff80a0e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:53.996 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:53.995 7f8ffca42700 1 -- 172.21.2.102:0/1923105757 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8ff80a1490 msgr2=0x7f8ff809fcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:53.996 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:53.995 7f8ffca42700 1 --2- 172.21.2.102:0/1923105757 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8ff80a1490 0x7f8ff809fcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:53.996 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:53.995 7f8ffca42700 1 -- 172.21.2.102:0/1923105757 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ff80a0310 con 0x7f8ff80b07a0 2021-02-10T11:22:53.996 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:53.995 7f8ffca42700 1 --2- 172.21.2.102:0/1923105757 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8ff80b07a0 0x7f8ff819ec90 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 rx=0x7f8fe000b5e0 tx=0x7f8fe000b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:53.996 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:53.996 7f8ff57fa700 1 -- 172.21.2.102:0/1923105757 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8fe000bd70 con 0x7f8ff80b07a0 2021-02-10T11:22:53.996 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:53.996 7f8ff57fa700 1 -- 172.21.2.102:0/1923105757 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fe0002030 con 0x7f8ff80b07a0 2021-02-10T11:22:53.996 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:53.996 7f8ffec26700 1 -- 172.21.2.102:0/1923105757 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ff80a0310 con 0x7f8ff80b07a0 2021-02-10T11:22:53.996 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:53.996 7f8ffec26700 1 -- 172.21.2.102:0/1923105757 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ff80a0860 con 0x7f8ff80b07a0 2021-02-10T11:22:53.996 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:53.996 7f8ff57fa700 1 -- 172.21.2.102:0/1923105757 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8fe0002550 con 0x7f8ff80b07a0 2021-02-10T11:22:53.997 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:53.997 7f8ff57fa700 1 -- 172.21.2.102:0/1923105757 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f8fe0000c90 con 0x7f8ff80b07a0 2021-02-10T11:22:53.998 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:53.998 7f8ff57fa700 1 --2- 172.21.2.102:0/1923105757 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8fdc0186c0 0x7f8fdc01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:53.998 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:53.998 7f8ff57fa700 1 -- 172.21.2.102:0/1923105757 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f8fe0032540 con 0x7f8ff80b07a0 2021-02-10T11:22:53.998 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:53.998 7f8ff7fff700 1 --2- 172.21.2.102:0/1923105757 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8fdc0186c0 0x7f8fdc01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:53.998 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:53.998 7f8ff7fff700 1 --2- 172.21.2.102:0/1923105757 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8fdc0186c0 0x7f8fdc01abc0 crc :-1 s=READY pgs=541 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:54.024 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:54.027 7f8ffec26700 1 -- 172.21.2.102:0/1923105757 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ff8083e50 con 0x7f8ff80b07a0 2021-02-10T11:22:54.027 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:54.030 7f8ff57fa700 1 -- 172.21.2.102:0/1923105757 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f8fe00009b0 con 0x7f8ff80b07a0 2021-02-10T11:22:54.157 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:54.159 7f8ffec26700 1 -- 172.21.2.102:0/1923105757 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "auth rm", "entity": "client.guest1"} v 0) v1 -- 0x7f8ff807ed80 con 0x7f8ff80b07a0 2021-02-10T11:22:54.160 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:54.162 7f8ff57fa700 1 -- 172.21.2.102:0/1923105757 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "auth rm", "entity": "client.guest1"}]=0 updated v22) v1 ==== 89+0+0 (secure 0 0 0) 0x7f8fe0000c90 con 0x7f8ff80b07a0 2021-02-10T11:22:54.164 INFO:teuthology.orchestra.run.gibba002.stderr:updated 2021-02-10T11:22:54.167 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:54.168 7f8ffec26700 1 -- 172.21.2.102:0/1923105757 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8fdc0186c0 msgr2=0x7f8fdc01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:54.167 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:54.168 7f8ffec26700 1 --2- 172.21.2.102:0/1923105757 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8fdc0186c0 0x7f8fdc01abc0 crc :-1 s=READY pgs=541 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:54.167 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:54.168 7f8ffec26700 1 -- 172.21.2.102:0/1923105757 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8ff80b07a0 msgr2=0x7f8ff819ec90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:54.167 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:54.168 7f8ffec26700 1 --2- 172.21.2.102:0/1923105757 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8ff80b07a0 0x7f8ff819ec90 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 rx=0x7f8fe000b5e0 tx=0x7f8fe000b980).stop 2021-02-10T11:22:54.167 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:54.168 7f8ffec26700 1 -- 172.21.2.102:0/1923105757 shutdown_connections 2021-02-10T11:22:54.168 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:54.168 7f8ffec26700 1 --2- 172.21.2.102:0/1923105757 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8fdc0186c0 0x7f8fdc01abc0 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:54.168 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:54.168 7f8ffec26700 1 --2- 172.21.2.102:0/1923105757 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8ff80a1490 0x7f8ff809fcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:54.168 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:54.168 7f8ffec26700 1 --2- 172.21.2.102:0/1923105757 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8ff819f2b0 0x7f8ff80a0e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:54.168 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:54.168 7f8ffec26700 1 --2- 172.21.2.102:0/1923105757 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8ff80b07a0 0x7f8ff819ec90 unknown :-1 s=CLOSED pgs=1132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:54.168 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:54.168 7f8ffec26700 1 -- 172.21.2.102:0/1923105757 >> 172.21.2.102:0/1923105757 conn(0x7f8ff80bde90 msgr2=0x7f8ff80bd600 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:54.168 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:54.168 7f8ffec26700 1 -- 172.21.2.102:0/1923105757 shutdown_connections 2021-02-10T11:22:54.169 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:54.168 7f8ffec26700 1 -- 172.21.2.102:0/1923105757 wait complete. 2021-02-10T11:22:54.175 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000793468 --group_name group_0000000000760661 2021-02-10T11:22:54.619 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:54.604 7f537ebad700 1 Processor -- start 2021-02-10T11:22:54.619 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:54.604 7f537ebad700 1 -- start start 2021-02-10T11:22:54.619 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:54.605 7f537ebad700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f53780d1c60 0x7f53780f2090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:54.620 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:54.605 7f537ebad700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53780f0b00 con 0x7f53780d1860 2021-02-10T11:22:54.620 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:54.605 7f537ebad700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53781b0bd0 con 0x7f53780d0330 2021-02-10T11:22:54.620 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:54.605 7f537ebad700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53780e5ec0 con 0x7f53780d1c60 2021-02-10T11:22:54.620 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:54.605 7f537c9c9700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f53780d1860 0x7f53781b03e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:48128/0 (socket says 172.21.2.102:48128) 2021-02-10T11:22:54.620 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:54.605 7f537c9c9700 1 -- 172.21.2.102:0/2890855679 learned_addr learned my addr 172.21.2.102:0/2890855679 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:22:54.620 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:54.605 7f53777fe700 1 -- 172.21.2.102:0/2890855679 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3597711822 0 0) 0x7f53600009b0 con 0x7f53780d1860 2021-02-10T11:22:54.620 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:54.605 7f53777fe700 1 -- 172.21.2.102:0/2890855679 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5364002fe0 con 0x7f53780d1860 2021-02-10T11:22:54.620 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:54.606 7f537d1ca700 1 --2- 172.21.2.102:0/2890855679 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f53780d1c60 0x7f53780f2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:54.620 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:54.606 7f53777fe700 1 -- 172.21.2.102:0/2890855679 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 639261454 0 0) 0x7f5360001010 con 0x7f53780d1860 2021-02-10T11:22:54.620 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:54.606 7f53777fe700 1 -- 172.21.2.102:0/2890855679 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f53780d1c60 msgr2=0x7f53780f2090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:54.620 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:54.606 7f53777fe700 1 --2- 172.21.2.102:0/2890855679 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f53780d1c60 0x7f53780f2090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:54.620 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:54.606 7f53777fe700 1 -- 172.21.2.102:0/2890855679 >> v1:172.21.2.113:6789/0 conn(0x7f53780d0330 legacy=0x7f53780ef630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:54.620 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:54.606 7f53777fe700 1 -- 172.21.2.102:0/2890855679 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53780efe20 con 0x7f53780d1860 2021-02-10T11:22:54.621 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:54.606 7f53777fe700 1 -- 172.21.2.102:0/2890855679 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7f5360001590 con 0x7f53780d1860 2021-02-10T11:22:54.621 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:54.606 7f53777fe700 1 -- 172.21.2.102:0/2890855679 >> v1:172.21.2.102:6789/0 conn(0x7f53780d1860 legacy=0x7f53781b03e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:54.621 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:54.606 7f53777fe700 1 --2- 172.21.2.102:0/2890855679 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5364003ae0 0x7f5364005f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:54.621 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:54.606 7f53777fe700 1 --2- 172.21.2.102:0/2890855679 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f53640067f0 0x7f5364008cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:54.621 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:54.606 7f53777fe700 1 --2- 172.21.2.102:0/2890855679 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5364009590 0x7f536400bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:54.621 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:54.606 7f53777fe700 1 -- 172.21.2.102:0/2890855679 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f536400c360 con 0x7f5364009590 2021-02-10T11:22:54.621 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:54.606 7f53777fe700 1 -- 172.21.2.102:0/2890855679 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f536400c580 con 0x7f53640067f0 2021-02-10T11:22:54.621 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:54.606 7f53777fe700 1 -- 172.21.2.102:0/2890855679 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f536400c7a0 con 0x7f5364003ae0 2021-02-10T11:22:54.621 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:54.606 7f537d1ca700 1 --2- 172.21.2.102:0/2890855679 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f53780d1c60 0x7f53780f2090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:22:54.621 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:54.606 7f5377fff700 1 --2- 172.21.2.102:0/2890855679 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f53640067f0 0x7f5364008cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:54.621 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:54.606 7f537d1ca700 1 --2- 172.21.2.102:0/2890855679 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5364009590 0x7f536400bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:54.621 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:54.606 7f537c9c9700 1 --2- 172.21.2.102:0/2890855679 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5364003ae0 0x7f5364005f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:54.621 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:54.606 7f537d1ca700 1 -- 172.21.2.102:0/2890855679 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5364003ae0 msgr2=0x7f5364005f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:54.621 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:54.606 7f537d1ca700 1 --2- 172.21.2.102:0/2890855679 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5364003ae0 0x7f5364005f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:54.622 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:54.606 7f537d1ca700 1 -- 172.21.2.102:0/2890855679 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f53640067f0 msgr2=0x7f5364008cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:54.622 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:54.606 7f537d1ca700 1 --2- 172.21.2.102:0/2890855679 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f53640067f0 0x7f5364008cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:54.622 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:54.606 7f537d1ca700 1 -- 172.21.2.102:0/2890855679 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f536400ca40 con 0x7f5364009590 2021-02-10T11:22:54.622 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:54.607 7f537d1ca700 1 --2- 172.21.2.102:0/2890855679 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5364009590 0x7f536400bab0 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 rx=0x7f5370050ba0 tx=0x7f5370050f40).ready entity=mon.0 client_cookie=c1b6da20907d0616 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:54.622 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:54.607 7f53777fe700 1 -- 172.21.2.102:0/2890855679 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f53700011f0 con 0x7f5364009590 2021-02-10T11:22:54.622 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:54.607 7f53777fe700 1 -- 172.21.2.102:0/2890855679 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5370003030 con 0x7f5364009590 2021-02-10T11:22:54.622 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:54.607 7f53777fe700 1 -- 172.21.2.102:0/2890855679 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f53700016a0 con 0x7f5364009590 2021-02-10T11:22:54.622 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:54.607 7f537ebad700 1 -- 172.21.2.102:0/2890855679 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5364009590 msgr2=0x7f536400bab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:54.622 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:54.607 7f537ebad700 1 --2- 172.21.2.102:0/2890855679 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5364009590 0x7f536400bab0 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 rx=0x7f5370050ba0 tx=0x7f5370050f40).stop 2021-02-10T11:22:54.622 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:54.607 7f537d1ca700 1 -- 172.21.2.102:0/2890855679 reap_dead start 2021-02-10T11:22:54.622 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:54.607 7f537ebad700 1 -- 172.21.2.102:0/2890855679 shutdown_connections 2021-02-10T11:22:54.622 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:54.607 7f537ebad700 1 -- 172.21.2.102:0/2890855679 >> 172.21.2.102:0/2890855679 conn(0x7f53780ce440 msgr2=0x7f53780d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:54.622 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:54.607 7f537ebad700 1 -- 172.21.2.102:0/2890855679 shutdown_connections 2021-02-10T11:22:54.623 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:54.607 7f537ebad700 1 -- 172.21.2.102:0/2890855679 wait complete. 2021-02-10T11:22:54.623 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:54.622 7f537ebad700 1 Processor -- start 2021-02-10T11:22:54.623 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:54.622 7f537ebad700 1 -- start start 2021-02-10T11:22:54.624 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:54.622 7f537ebad700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f53780bc430 0x7f537809ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:54.624 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:54.622 7f537ebad700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f53780bc830 0x7f537809d340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:54.624 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:54.622 7f537ebad700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f537809d960 0x7f537809ba60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:54.624 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:54.623 7f537ebad700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f537809d060 con 0x7f53780bc830 2021-02-10T11:22:54.624 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:54.623 7f537ebad700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f537809c580 con 0x7f537809d960 2021-02-10T11:22:54.624 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:54.623 7f537ebad700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f537809c080 con 0x7f53780bc430 2021-02-10T11:22:54.624 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:54.623 7f5377fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f53780bc830 0x7f537809d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:54.624 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:54.623 7f5377fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f53780bc830 0x7f537809d340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:55916/0 (socket says 172.21.2.102:55916) 2021-02-10T11:22:54.625 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:54.623 7f5377fff700 1 -- 172.21.2.102:0/3286591196 learned_addr learned my addr 172.21.2.102:0/3286591196 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:22:54.625 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:54.623 7f5377fff700 1 -- 172.21.2.102:0/3286591196 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f53780bc430 msgr2=0x7f537809ca40 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:22:54.625 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:54.623 7f5377fff700 1 --2- 172.21.2.102:0/3286591196 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f53780bc430 0x7f537809ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:54.625 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:54.623 7f5377fff700 1 -- 172.21.2.102:0/3286591196 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f537809d960 msgr2=0x7f537809ba60 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:22:54.625 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:54.623 7f5377fff700 1 --2- 172.21.2.102:0/3286591196 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f537809d960 0x7f537809ba60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:54.625 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:54.623 7f5377fff700 1 -- 172.21.2.102:0/3286591196 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f537809c2a0 con 0x7f53780bc830 2021-02-10T11:22:54.625 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:54.623 7f5377fff700 1 --2- 172.21.2.102:0/3286591196 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f53780bc830 0x7f537809d340 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 rx=0x7f536800b5e0 tx=0x7f536800b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:54.625 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:54.623 7f53757fa700 1 -- 172.21.2.102:0/3286591196 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5368002030 con 0x7f53780bc830 2021-02-10T11:22:54.625 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:54.623 7f53757fa700 1 -- 172.21.2.102:0/3286591196 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5368002330 con 0x7f53780bc830 2021-02-10T11:22:54.625 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:54.623 7f53757fa700 1 -- 172.21.2.102:0/3286591196 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5368002850 con 0x7f53780bc830 2021-02-10T11:22:54.625 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:54.623 7f537ebad700 1 -- 172.21.2.102:0/3286591196 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f537809c2a0 con 0x7f53780bc830 2021-02-10T11:22:54.625 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:54.624 7f537ebad700 1 -- 172.21.2.102:0/3286591196 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f537809c2a0 con 0x7f53780bc830 2021-02-10T11:22:54.625 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:54.624 7f53757fa700 1 -- 172.21.2.102:0/3286591196 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f5368002510 con 0x7f53780bc830 2021-02-10T11:22:54.626 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:54.624 7f53757fa700 1 --2- 172.21.2.102:0/3286591196 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f535c0186c0 0x7f535c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:54.626 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:54.624 7f53757fa700 1 -- 172.21.2.102:0/3286591196 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f53680323b0 con 0x7f53780bc830 2021-02-10T11:22:54.626 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:54.624 7f537c9c9700 1 --2- 172.21.2.102:0/3286591196 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f535c0186c0 0x7f535c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:54.626 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:54.624 7f537c9c9700 1 --2- 172.21.2.102:0/3286591196 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f535c0186c0 0x7f535c01abc0 crc :-1 s=READY pgs=542 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:54.651 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:54.654 7f537ebad700 1 -- 172.21.2.102:0/3286591196 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f537809ddb0 con 0x7f53780bc830 2021-02-10T11:22:54.654 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:54.657 7f53757fa700 1 -- 172.21.2.102:0/3286591196 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f5368002620 con 0x7f53780bc830 2021-02-10T11:22:54.779 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:54.782 7f537ebad700 1 -- 172.21.2.102:0/3286591196 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000793468", "target": ["mgr", ""], "group_name": "group_0000000000760661"}) v1 -- 0x7f537809ddb0 con 0x7f535c0186c0 2021-02-10T11:22:54.788 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:54.791 7f53757fa700 1 -- 172.21.2.102:0/3286591196 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f5360001590 con 0x7f535c0186c0 2021-02-10T11:22:54.796 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:54.798 7f537ebad700 1 -- 172.21.2.102:0/3286591196 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f535c0186c0 msgr2=0x7f535c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:54.796 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:54.798 7f537ebad700 1 --2- 172.21.2.102:0/3286591196 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f535c0186c0 0x7f535c01abc0 crc :-1 s=READY pgs=542 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:54.796 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:54.799 7f537ebad700 1 -- 172.21.2.102:0/3286591196 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f53780bc830 msgr2=0x7f537809d340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:54.796 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:54.799 7f537ebad700 1 --2- 172.21.2.102:0/3286591196 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f53780bc830 0x7f537809d340 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 rx=0x7f536800b5e0 tx=0x7f536800b980).stop 2021-02-10T11:22:54.796 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:54.799 7f537ebad700 1 -- 172.21.2.102:0/3286591196 shutdown_connections 2021-02-10T11:22:54.797 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:54.799 7f537ebad700 1 --2- 172.21.2.102:0/3286591196 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f535c0186c0 0x7f535c01abc0 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:54.797 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:54.799 7f537ebad700 1 --2- 172.21.2.102:0/3286591196 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f537809d960 0x7f537809ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:54.797 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:54.799 7f537ebad700 1 --2- 172.21.2.102:0/3286591196 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f53780bc830 0x7f537809d340 unknown :-1 s=CLOSED pgs=1135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:54.797 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:54.799 7f537ebad700 1 --2- 172.21.2.102:0/3286591196 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f53780bc430 0x7f537809ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:54.797 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:54.799 7f537ebad700 1 -- 172.21.2.102:0/3286591196 >> 172.21.2.102:0/3286591196 conn(0x7f5378099c60 msgr2=0x7f53780b2920 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:54.797 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:54.799 7f537ebad700 1 -- 172.21.2.102:0/3286591196 shutdown_connections 2021-02-10T11:22:54.797 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:54.799 7f537ebad700 1 -- 172.21.2.102:0/3286591196 wait complete. 2021-02-10T11:22:54.806 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolumegroup rm cephfs group_0000000000760661 2021-02-10T11:22:55.236 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:55.221 7fde3354e700 1 Processor -- start 2021-02-10T11:22:55.236 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:55.221 7fde3354e700 1 -- start start 2021-02-10T11:22:55.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:55.222 7fde3354e700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fde2c1b0470 0x7fde2c1b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:55.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:55.222 7fde3354e700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fde2c0ef610 0x7fde2c0efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:55.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:55.222 7fde3354e700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde2c0f0b50 con 0x7fde2c0f0030 2021-02-10T11:22:55.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:55.222 7fde3354e700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde2c0e52e0 con 0x7fde2c0ef610 2021-02-10T11:22:55.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:55.222 7fde3354e700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde2c0e5490 con 0x7fde2c1b0470 2021-02-10T11:22:55.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:55.222 7fde31b6b700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fde2c0f0030 0x7fde2c0f10d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:48148/0 (socket says 172.21.2.102:48148) 2021-02-10T11:22:55.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:55.222 7fde31b6b700 1 -- 172.21.2.102:0/3561467449 learned_addr learned my addr 172.21.2.102:0/3561467449 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:22:55.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:55.222 7fde3136a700 1 --2- 172.21.2.102:0/3561467449 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fde2c1b0470 0x7fde2c1b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:55.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:55.222 7fde30b69700 1 --2- 172.21.2.102:0/3561467449 >> v2:172.21.2.113:3300/0 conn(0x7fde2c0ef610 0x7fde2c0efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:55.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:55.222 7fde1ffff700 1 -- 172.21.2.102:0/3561467449 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 814158832 0 0) 0x7fde24000f90 con 0x7fde2c0f0030 2021-02-10T11:22:55.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:55.223 7fde1ffff700 1 -- 172.21.2.102:0/3561467449 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde18002fe0 con 0x7fde2c0f0030 2021-02-10T11:22:55.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:55.223 7fde3136a700 1 -- 172.21.2.102:0/3561467449 >> v2:172.21.2.113:3300/0 conn(0x7fde2c0ef610 msgr2=0x7fde2c0efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:55.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:55.223 7fde3136a700 1 --2- 172.21.2.102:0/3561467449 >> v2:172.21.2.113:3300/0 conn(0x7fde2c0ef610 0x7fde2c0efa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:55.238 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:55.223 7fde3136a700 1 -- 172.21.2.102:0/3561467449 >> v1:172.21.2.102:6789/0 conn(0x7fde2c0f0030 legacy=0x7fde2c0f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:55.238 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:55.223 7fde3136a700 1 -- 172.21.2.102:0/3561467449 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde2c0f0430 con 0x7fde2c1b0470 2021-02-10T11:22:55.238 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:55.223 7fde3136a700 1 --2- 172.21.2.102:0/3561467449 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fde2c1b0470 0x7fde2c1b0870 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7fde28006650 tx=0x7fde28009a80).ready entity=mon.2 client_cookie=694032f8c6a59989 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:55.238 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:55.223 7fde1ffff700 1 -- 172.21.2.102:0/3561467449 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fde280009b0 con 0x7fde2c1b0470 2021-02-10T11:22:55.238 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:55.223 7fde1ffff700 1 -- 172.21.2.102:0/3561467449 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde28006740 con 0x7fde2c1b0470 2021-02-10T11:22:55.238 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:55.223 7fde1ffff700 1 -- 172.21.2.102:0/3561467449 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fde2800c0c0 con 0x7fde2c1b0470 2021-02-10T11:22:55.238 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:55.223 7fde3354e700 1 -- 172.21.2.102:0/3561467449 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fde2c1b0470 msgr2=0x7fde2c1b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:55.238 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:55.223 7fde3354e700 1 --2- 172.21.2.102:0/3561467449 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fde2c1b0470 0x7fde2c1b0870 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7fde28006650 tx=0x7fde28009a80).stop 2021-02-10T11:22:55.238 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:55.224 7fde3354e700 1 -- 172.21.2.102:0/3561467449 shutdown_connections 2021-02-10T11:22:55.238 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:55.224 7fde3354e700 1 --2- 172.21.2.102:0/3561467449 >> v2:172.21.2.113:3300/0 conn(0x7fde2c0ef610 0x7fde2c0efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:55.238 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:55.224 7fde3354e700 1 --2- 172.21.2.102:0/3561467449 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fde2c1b0470 0x7fde2c1b0870 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:55.238 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:55.224 7fde3354e700 1 -- 172.21.2.102:0/3561467449 >> 172.21.2.102:0/3561467449 conn(0x7fde2c0ce3e0 msgr2=0x7fde2c0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:55.238 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:55.224 7fde3354e700 1 -- 172.21.2.102:0/3561467449 shutdown_connections 2021-02-10T11:22:55.239 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:55.224 7fde3354e700 1 -- 172.21.2.102:0/3561467449 wait complete. 2021-02-10T11:22:55.239 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:55.239 7fde3354e700 1 Processor -- start 2021-02-10T11:22:55.239 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:55.239 7fde3354e700 1 -- start start 2021-02-10T11:22:55.240 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:55.239 7fde3354e700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fde2c0a0420 0x7fde2c0a0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:55.240 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:55.239 7fde3354e700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fde2c189d00 0x7fde2c18a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:55.240 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:55.239 7fde3354e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fde2c18a720 0x7fde2c0b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:55.240 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:55.239 7fde3354e700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde2c189800 con 0x7fde2c18a720 2021-02-10T11:22:55.241 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:55.239 7fde3354e700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde2c18ab20 con 0x7fde2c0a0420 2021-02-10T11:22:55.241 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:55.239 7fde3354e700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde2c0a1140 con 0x7fde2c189d00 2021-02-10T11:22:55.241 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:55.240 7fde3136a700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fde2c0a0420 0x7fde2c0a0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:55.241 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:55.240 7fde31b6b700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fde2c18a720 0x7fde2c0b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:55.241 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:55.240 7fde31b6b700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fde2c18a720 0x7fde2c0b7be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:55930/0 (socket says 172.21.2.102:55930) 2021-02-10T11:22:55.241 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:55.240 7fde31b6b700 1 -- 172.21.2.102:0/2098326731 learned_addr learned my addr 172.21.2.102:0/2098326731 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:22:55.241 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:55.240 7fde3136a700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fde2c0a0420 0x7fde2c0a0820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.113:3300/0 says I am v2:172.21.2.102:59358/0 (socket says 172.21.2.102:59358) 2021-02-10T11:22:55.241 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:55.240 7fde30b69700 1 --2- 172.21.2.102:0/2098326731 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fde2c189d00 0x7fde2c18a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:55.241 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:55.240 7fde31b6b700 1 -- 172.21.2.102:0/2098326731 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fde2c189d00 msgr2=0x7fde2c18a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:55.241 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:55.240 7fde31b6b700 1 --2- 172.21.2.102:0/2098326731 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fde2c189d00 0x7fde2c18a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:55.241 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:55.240 7fde31b6b700 1 -- 172.21.2.102:0/2098326731 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fde2c0a0420 msgr2=0x7fde2c0a0820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:55.241 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:55.240 7fde31b6b700 1 --2- 172.21.2.102:0/2098326731 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fde2c0a0420 0x7fde2c0a0820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:55.241 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:55.240 7fde31b6b700 1 -- 172.21.2.102:0/2098326731 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde2c0b8200 con 0x7fde2c18a720 2021-02-10T11:22:55.242 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:55.240 7fde31b6b700 1 --2- 172.21.2.102:0/2098326731 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fde2c18a720 0x7fde2c0b7be0 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 rx=0x7fde24009980 tx=0x7fde2400bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:55.242 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:55.240 7fde1dffb700 1 -- 172.21.2.102:0/2098326731 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fde24006030 con 0x7fde2c18a720 2021-02-10T11:22:55.242 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:55.240 7fde1dffb700 1 -- 172.21.2.102:0/2098326731 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde240063b0 con 0x7fde2c18a720 2021-02-10T11:22:55.242 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:55.240 7fde1dffb700 1 -- 172.21.2.102:0/2098326731 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fde24004720 con 0x7fde2c18a720 2021-02-10T11:22:55.242 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:55.240 7fde3354e700 1 -- 172.21.2.102:0/2098326731 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde2c0b8200 con 0x7fde2c18a720 2021-02-10T11:22:55.242 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:55.240 7fde3354e700 1 -- 172.21.2.102:0/2098326731 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde2c0b8200 con 0x7fde2c18a720 2021-02-10T11:22:55.242 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:55.241 7fde1dffb700 1 -- 172.21.2.102:0/2098326731 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fde24000e40 con 0x7fde2c18a720 2021-02-10T11:22:55.242 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:55.241 7fde1dffb700 1 --2- 172.21.2.102:0/2098326731 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fde0c018730 0x7fde0c01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:55.242 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:55.241 7fde1dffb700 1 -- 172.21.2.102:0/2098326731 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fde24032a20 con 0x7fde2c18a720 2021-02-10T11:22:55.242 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:55.241 7fde3136a700 1 --2- 172.21.2.102:0/2098326731 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fde0c018730 0x7fde0c01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:55.242 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:55.241 7fde3136a700 1 --2- 172.21.2.102:0/2098326731 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fde0c018730 0x7fde0c01ac10 crc :-1 s=READY pgs=543 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:55.269 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:55.271 7fde3354e700 1 -- 172.21.2.102:0/2098326731 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde2c18ab20 con 0x7fde2c18a720 2021-02-10T11:22:55.271 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:55.274 7fde1dffb700 1 -- 172.21.2.102:0/2098326731 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fde2400be10 con 0x7fde2c18a720 2021-02-10T11:22:55.400 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:55.403 7fde3354e700 1 -- 172.21.2.102:0/2098326731 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolumegroup rm", "vol_name": "cephfs", "target": ["mgr", ""], "group_name": "group_0000000000760661"}) v1 -- 0x7fde2c18ab20 con 0x7fde0c018730 2021-02-10T11:22:55.404 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:55.407 7fde1dffb700 1 -- 172.21.2.102:0/2098326731 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fde28009180 con 0x7fde0c018730 2021-02-10T11:22:55.409 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:55.411 7fde3354e700 1 -- 172.21.2.102:0/2098326731 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fde0c018730 msgr2=0x7fde0c01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:55.409 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:55.411 7fde3354e700 1 --2- 172.21.2.102:0/2098326731 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fde0c018730 0x7fde0c01ac10 crc :-1 s=READY pgs=543 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:55.409 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:55.411 7fde3354e700 1 -- 172.21.2.102:0/2098326731 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fde2c18a720 msgr2=0x7fde2c0b7be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:55.409 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:55.411 7fde3354e700 1 --2- 172.21.2.102:0/2098326731 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fde2c18a720 0x7fde2c0b7be0 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 rx=0x7fde24009980 tx=0x7fde2400bcd0).stop 2021-02-10T11:22:55.410 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:55.412 7fde3354e700 1 -- 172.21.2.102:0/2098326731 shutdown_connections 2021-02-10T11:22:55.410 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:55.412 7fde3354e700 1 --2- 172.21.2.102:0/2098326731 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fde2c18a720 0x7fde2c0b7be0 unknown :-1 s=CLOSED pgs=1137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:55.410 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:55.412 7fde3354e700 1 --2- 172.21.2.102:0/2098326731 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fde2c189d00 0x7fde2c18a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:55.410 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:55.412 7fde3354e700 1 --2- 172.21.2.102:0/2098326731 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fde0c018730 0x7fde0c01ac10 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:55.410 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:55.412 7fde3354e700 1 --2- 172.21.2.102:0/2098326731 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fde2c0a0420 0x7fde2c0a0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:55.410 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:55.412 7fde3354e700 1 -- 172.21.2.102:0/2098326731 >> 172.21.2.102:0/2098326731 conn(0x7fde2c094e10 msgr2=0x7fde2c0a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:55.410 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:55.412 7fde3354e700 1 -- 172.21.2.102:0/2098326731 shutdown_connections 2021-02-10T11:22:55.410 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:22:55.412 7fde3354e700 1 -- 172.21.2.102:0/2098326731 wait complete. 2021-02-10T11:22:55.417 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2021-02-10T11:22:55.418 DEBUG:teuthology.orchestra.run.gibba002:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:22:55.765 DEBUG:teuthology.orchestra.run.gibba013:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:22:56.110 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2021-02-10T11:22:56.110 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:22:56.110 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:22:56.110 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:22:56.446 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:22:56.446 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:22:56.447 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@gibba002.front.sepia.ceph.com... 2021-02-10T11:22:56.447 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-02-10T11:22:56.447 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:22:56.798 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba002.stderr:ceph-fuse[145313]: fuse finished with error 0 and tester_r 0 2021-02-10T11:22:56.800 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:22:56.800 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:22:56.800 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:22:57.135 INFO:teuthology.orchestra.run.gibba002.stdout:ext2/ext3 2021-02-10T11:22:57.136 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2021-02-10T11:22:57.136 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:22:57.136 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:22:57.474 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:22:57.474 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:22:57.474 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:22:57.813 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:22:57.814 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2021-02-10T11:22:57.814 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@gibba002.front.sepia.ceph.com... 2021-02-10T11:22:57.814 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-02-10T11:22:57.814 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:22:58.175 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba002.stderr:ceph-fuse[145953]: fuse finished with error 0 and tester_r 0 2021-02-10T11:22:58.177 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:22:58.177 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:22:58.177 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:22:58.515 INFO:teuthology.orchestra.run.gibba002.stdout:ext2/ext3 2021-02-10T11:22:58.515 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2021-02-10T11:22:58.515 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:22:58.515 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:22:58.853 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:22:58.853 DEBUG:teuthology.orchestra.run.gibba013:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:22:58.853 DEBUG:teuthology.orchestra.run.gibba013:> ' -- /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:22:59.192 INFO:teuthology.orchestra.run.gibba013.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.2’: No such file or directory 2021-02-10T11:22:59.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:22:59.192 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2021-02-10T11:22:59.192 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:22:59.193 DEBUG:teuthology.orchestra.run.gibba013:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:22:59.527 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:22:59.527 DEBUG:teuthology.orchestra.run.gibba013:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:22:59.527 DEBUG:teuthology.orchestra.run.gibba013:> ' -- /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:22:59.862 INFO:teuthology.orchestra.run.gibba013.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.3’: No such file or directory 2021-02-10T11:22:59.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:22:59.863 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.3 2021-02-10T11:22:59.863 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:22:59.863 DEBUG:teuthology.orchestra.run.gibba013:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:23:00.197 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config rm mon mon_allow_pool_delete 2021-02-10T11:23:00.628 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:00.614 7f3f083f9700 1 Processor -- start 2021-02-10T11:23:00.628 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:00.614 7f3f083f9700 1 -- start start 2021-02-10T11:23:00.629 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:00.615 7f3f083f9700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f3f000d17f0 0x7f3f001b31e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:00.629 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:00.615 7f3f083f9700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f3f000d1bf0 0x7f3f001b3800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:00.629 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:00.615 7f3f083f9700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3f001b3e20 0x7f3f000ceef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:00.629 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:00.615 7f3f083f9700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f000e5eb0 con 0x7f3f000d1bf0 2021-02-10T11:23:00.629 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:00.615 7f3f083f9700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f000f0b20 con 0x7f3f000d17f0 2021-02-10T11:23:00.629 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:00.615 7f3f083f9700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f000cf510 con 0x7f3f001b3e20 2021-02-10T11:23:00.629 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:00.615 7f3f05a14700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f3f000d1bf0 0x7f3f001b3800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:00.629 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:00.615 7f3f05a14700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f3f000d1bf0 0x7f3f001b3800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:55936/0 (socket says 172.21.2.102:55936) 2021-02-10T11:23:00.629 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:00.615 7f3f05a14700 1 -- 172.21.2.102:0/2619898940 learned_addr learned my addr 172.21.2.102:0/2619898940 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:23:00.629 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:00.615 7f3f06215700 1 --2- 172.21.2.102:0/2619898940 >> v2:172.21.2.113:3300/0 conn(0x7f3f000d17f0 0x7f3f001b31e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:00.629 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:00.615 7f3f06a16700 1 --2- 172.21.2.102:0/2619898940 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3f001b3e20 0x7f3f000ceef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:00.630 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:00.615 7f3f05a14700 1 -- 172.21.2.102:0/2619898940 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3f001b3e20 msgr2=0x7f3f000ceef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:00.630 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:00.615 7f3f05a14700 1 --2- 172.21.2.102:0/2619898940 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3f001b3e20 0x7f3f000ceef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:00.630 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:00.615 7f3f05a14700 1 -- 172.21.2.102:0/2619898940 >> v2:172.21.2.113:3300/0 conn(0x7f3f000d17f0 msgr2=0x7f3f001b31e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:00.630 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:00.615 7f3f05a14700 1 --2- 172.21.2.102:0/2619898940 >> v2:172.21.2.113:3300/0 conn(0x7f3f000d17f0 0x7f3f001b31e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:00.630 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:00.615 7f3f05a14700 1 -- 172.21.2.102:0/2619898940 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f000cf6f0 con 0x7f3f000d1bf0 2021-02-10T11:23:00.630 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:00.615 7f3f05a14700 1 --2- 172.21.2.102:0/2619898940 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3f000d1bf0 0x7f3f001b3800 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 rx=0x7f3efc006700 tx=0x7f3efc009b80).ready entity=mon.0 client_cookie=73b3028c0c0e1f64 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:00.630 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:00.616 7f3f05213700 1 -- 172.21.2.102:0/2619898940 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3efc00c0c0 con 0x7f3f000d1bf0 2021-02-10T11:23:00.630 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:00.616 7f3f05213700 1 -- 172.21.2.102:0/2619898940 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3efc00c340 con 0x7f3f000d1bf0 2021-02-10T11:23:00.630 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:00.616 7f3f05213700 1 -- 172.21.2.102:0/2619898940 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3efc00c860 con 0x7f3f000d1bf0 2021-02-10T11:23:00.630 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:00.616 7f3f083f9700 1 -- 172.21.2.102:0/2619898940 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3f000d1bf0 msgr2=0x7f3f001b3800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:00.630 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:00.616 7f3f083f9700 1 --2- 172.21.2.102:0/2619898940 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3f000d1bf0 0x7f3f001b3800 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 rx=0x7f3efc006700 tx=0x7f3efc009b80).stop 2021-02-10T11:23:00.630 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:00.616 7f3f083f9700 1 -- 172.21.2.102:0/2619898940 shutdown_connections 2021-02-10T11:23:00.631 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:00.616 7f3f083f9700 1 --2- 172.21.2.102:0/2619898940 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3f001b3e20 0x7f3f000ceef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:00.631 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:00.616 7f3f083f9700 1 --2- 172.21.2.102:0/2619898940 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3f000d1bf0 0x7f3f001b3800 unknown :-1 s=CLOSED pgs=1138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:00.631 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:00.616 7f3f083f9700 1 --2- 172.21.2.102:0/2619898940 >> v2:172.21.2.113:3300/0 conn(0x7f3f000d17f0 0x7f3f001b31e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:00.631 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:00.616 7f3f083f9700 1 -- 172.21.2.102:0/2619898940 >> 172.21.2.102:0/2619898940 conn(0x7f3f000fde40 msgr2=0x7f3f000ce390 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:00.631 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:00.617 7f3f083f9700 1 -- 172.21.2.102:0/2619898940 shutdown_connections 2021-02-10T11:23:00.631 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:00.617 7f3f083f9700 1 -- 172.21.2.102:0/2619898940 wait complete. 2021-02-10T11:23:00.632 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:00.631 7f3f083f9700 1 Processor -- start 2021-02-10T11:23:00.633 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:00.631 7f3f083f9700 1 -- start start 2021-02-10T11:23:00.633 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:00.632 7f3f083f9700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3f0019ffe0 0x7f3f001a03e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:00.633 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:00.632 7f3f083f9700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3f00190340 0x7f3f00190740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:00.633 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:00.632 7f3f083f9700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3f00190d60 0x7f3f00191160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:00.633 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:00.632 7f3f083f9700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f00191850 con 0x7f3f00190340 2021-02-10T11:23:00.633 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:00.632 7f3f083f9700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f000c3c30 con 0x7f3f00190d60 2021-02-10T11:23:00.633 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:00.632 7f3f083f9700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f00191d50 con 0x7f3f0019ffe0 2021-02-10T11:23:00.633 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:00.632 7f3f05a14700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3f00190340 0x7f3f00190740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:00.633 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:00.632 7f3f05a14700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3f00190340 0x7f3f00190740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:55944/0 (socket says 172.21.2.102:55944) 2021-02-10T11:23:00.633 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:00.632 7f3f05a14700 1 -- 172.21.2.102:0/2407485527 learned_addr learned my addr 172.21.2.102:0/2407485527 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:23:00.633 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:00.632 7f3f06a16700 1 --2- 172.21.2.102:0/2407485527 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3f00190d60 0x7f3f00191160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:00.633 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:00.632 7f3f06215700 1 --2- 172.21.2.102:0/2407485527 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3f0019ffe0 0x7f3f001a03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:00.634 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:00.632 7f3f05a14700 1 -- 172.21.2.102:0/2407485527 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3f0019ffe0 msgr2=0x7f3f001a03e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:00.634 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:00.632 7f3f05a14700 1 --2- 172.21.2.102:0/2407485527 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3f0019ffe0 0x7f3f001a03e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:00.634 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:00.632 7f3f05a14700 1 -- 172.21.2.102:0/2407485527 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3f00190d60 msgr2=0x7f3f00191160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:00.634 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:00.632 7f3f05a14700 1 --2- 172.21.2.102:0/2407485527 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3f00190d60 0x7f3f00191160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:00.634 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:00.632 7f3f05a14700 1 -- 172.21.2.102:0/2407485527 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f00191f70 con 0x7f3f00190340 2021-02-10T11:23:00.634 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:00.632 7f3f05a14700 1 --2- 172.21.2.102:0/2407485527 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3f00190340 0x7f3f00190740 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 rx=0x7f3efc00b5e0 tx=0x7f3efc00b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:00.634 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:00.633 7f3ef2ffd700 1 -- 172.21.2.102:0/2407485527 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3efc00bd70 con 0x7f3f00190340 2021-02-10T11:23:00.634 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:00.633 7f3ef2ffd700 1 -- 172.21.2.102:0/2407485527 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3efc002030 con 0x7f3f00190340 2021-02-10T11:23:00.634 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:00.633 7f3ef2ffd700 1 -- 172.21.2.102:0/2407485527 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3efc002550 con 0x7f3f00190340 2021-02-10T11:23:00.634 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:00.633 7f3f083f9700 1 -- 172.21.2.102:0/2407485527 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f00191f70 con 0x7f3f00190340 2021-02-10T11:23:00.634 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:00.633 7f3f083f9700 1 -- 172.21.2.102:0/2407485527 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f00191f70 con 0x7f3f00190340 2021-02-10T11:23:00.634 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:00.633 7f3ef2ffd700 1 -- 172.21.2.102:0/2407485527 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f3efc000c90 con 0x7f3f00190340 2021-02-10T11:23:00.634 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:00.633 7f3ef2ffd700 1 --2- 172.21.2.102:0/2407485527 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3ee80186c0 0x7f3ee801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:00.635 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:00.633 7f3ef2ffd700 1 -- 172.21.2.102:0/2407485527 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f3efc032620 con 0x7f3f00190340 2021-02-10T11:23:00.635 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:00.633 7f3f06215700 1 --2- 172.21.2.102:0/2407485527 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3ee80186c0 0x7f3ee801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:00.635 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:00.634 7f3f06215700 1 --2- 172.21.2.102:0/2407485527 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3ee80186c0 0x7f3ee801abc0 crc :-1 s=READY pgs=544 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:00.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:00.664 7f3f083f9700 1 -- 172.21.2.102:0/2407485527 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f00103690 con 0x7f3f00190340 2021-02-10T11:23:00.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:00.667 7f3ef2ffd700 1 -- 172.21.2.102:0/2407485527 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f3efc030dd0 con 0x7f3f00190340 2021-02-10T11:23:00.812 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:00.815 7f3f083f9700 1 -- 172.21.2.102:0/2407485527 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f3f00103690 con 0x7f3f00190340 2021-02-10T11:23:00.816 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:00.819 7f3ef2ffd700 1 -- 172.21.2.102:0/2407485527 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v5) v1 ==== 104+0+0 (secure 0 0 0) 0x7f3efc000c90 con 0x7f3f00190340 2021-02-10T11:23:00.822 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:00.824 7f3f083f9700 1 -- 172.21.2.102:0/2407485527 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3ee80186c0 msgr2=0x7f3ee801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:00.822 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:00.824 7f3f083f9700 1 --2- 172.21.2.102:0/2407485527 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3ee80186c0 0x7f3ee801abc0 crc :-1 s=READY pgs=544 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:00.822 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:00.824 7f3f083f9700 1 -- 172.21.2.102:0/2407485527 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3f00190340 msgr2=0x7f3f00190740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:00.822 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:00.824 7f3f083f9700 1 --2- 172.21.2.102:0/2407485527 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3f00190340 0x7f3f00190740 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 rx=0x7f3efc00b5e0 tx=0x7f3efc00b980).stop 2021-02-10T11:23:00.822 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:00.824 7f3f083f9700 1 -- 172.21.2.102:0/2407485527 shutdown_connections 2021-02-10T11:23:00.822 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:00.824 7f3f083f9700 1 --2- 172.21.2.102:0/2407485527 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3ee80186c0 0x7f3ee801abc0 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:00.822 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:00.824 7f3f083f9700 1 --2- 172.21.2.102:0/2407485527 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3f00190d60 0x7f3f00191160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:00.822 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:00.824 7f3f083f9700 1 --2- 172.21.2.102:0/2407485527 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3f00190340 0x7f3f00190740 unknown :-1 s=CLOSED pgs=1139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:00.822 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:00.824 7f3f083f9700 1 --2- 172.21.2.102:0/2407485527 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3f0019ffe0 0x7f3f001a03e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:00.822 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:00.824 7f3f083f9700 1 -- 172.21.2.102:0/2407485527 >> 172.21.2.102:0/2407485527 conn(0x7f3f000a1690 msgr2=0x7f3f000a94c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:00.822 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:00.824 7f3f083f9700 1 -- 172.21.2.102:0/2407485527 shutdown_connections 2021-02-10T11:23:00.822 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:00.824 7f3f083f9700 1 -- 172.21.2.102:0/2407485527 wait complete. 2021-02-10T11:23:00.831 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestVolumes.test_authorize_allow_existing_id_option' 2021-02-10T11:23:01.257 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:01.243 7fa314a70700 1 Processor -- start 2021-02-10T11:23:01.257 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:01.243 7fa314a70700 1 -- start start 2021-02-10T11:23:01.257 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:01.243 7fa314a70700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa31007e6a0 0x7fa3100d1790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:01.257 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:01.243 7fa314a70700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa31007ed80 con 0x7fa3100e5470 2021-02-10T11:23:01.257 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:01.243 7fa314a70700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa31007eb10 con 0x7fa3100e5070 2021-02-10T11:23:01.257 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:01.243 7fa314a70700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3101b4080 con 0x7fa31007e6a0 2021-02-10T11:23:01.257 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:01.243 7fa30e05d700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fa3100e5470 0x7fa31007f300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:48172/0 (socket says 172.21.2.102:48172) 2021-02-10T11:23:01.258 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:01.243 7fa30e05d700 1 -- 172.21.2.102:0/1980327669 learned_addr learned my addr 172.21.2.102:0/1980327669 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:23:01.258 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:01.243 7fa30f05f700 1 --2- 172.21.2.102:0/1980327669 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa31007e6a0 0x7fa3100d1790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:01.258 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:01.244 7fa30d85c700 1 -- 172.21.2.102:0/1980327669 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 384804493 0 0) 0x7fa2f8000ce0 con 0x7fa3100e5470 2021-02-10T11:23:01.258 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:01.244 7fa30d85c700 1 -- 172.21.2.102:0/1980327669 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2fc002fe0 con 0x7fa3100e5470 2021-02-10T11:23:01.258 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:01.244 7fa30d85c700 1 -- 172.21.2.102:0/1980327669 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2824937856 0 0) 0x7fa2f80010e0 con 0x7fa3100e5470 2021-02-10T11:23:01.258 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:01.244 7fa30f05f700 1 -- 172.21.2.102:0/1980327669 >> v1:172.21.2.113:6789/0 conn(0x7fa3100e5070 legacy=0x7fa3101b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:01.258 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:01.244 7fa30f05f700 1 -- 172.21.2.102:0/1980327669 >> v1:172.21.2.102:6789/0 conn(0x7fa3100e5470 legacy=0x7fa31007f300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:01.258 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:01.244 7fa30f05f700 1 -- 172.21.2.102:0/1980327669 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3100d1db0 con 0x7fa31007e6a0 2021-02-10T11:23:01.258 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:01.244 7fa30f05f700 1 --2- 172.21.2.102:0/1980327669 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa31007e6a0 0x7fa3100d1790 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7fa308006e50 tx=0x7fa308009df0).ready entity=mon.2 client_cookie=d7237185535c607 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:01.258 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:01.244 7fa30d85c700 1 -- 172.21.2.102:0/1980327669 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa30800b510 con 0x7fa31007e6a0 2021-02-10T11:23:01.258 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:01.244 7fa30d85c700 1 -- 172.21.2.102:0/1980327669 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa308006790 con 0x7fa31007e6a0 2021-02-10T11:23:01.258 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:01.244 7fa30d85c700 1 -- 172.21.2.102:0/1980327669 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa30800b750 con 0x7fa31007e6a0 2021-02-10T11:23:01.258 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:01.244 7fa314a70700 1 -- 172.21.2.102:0/1980327669 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa31007e6a0 msgr2=0x7fa3100d1790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:01.259 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:01.244 7fa314a70700 1 --2- 172.21.2.102:0/1980327669 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa31007e6a0 0x7fa3100d1790 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7fa308006e50 tx=0x7fa308009df0).stop 2021-02-10T11:23:01.259 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:01.245 7fa314a70700 1 -- 172.21.2.102:0/1980327669 shutdown_connections 2021-02-10T11:23:01.259 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:01.245 7fa314a70700 1 --2- 172.21.2.102:0/1980327669 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa31007e6a0 0x7fa3100d1790 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:01.259 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:01.245 7fa314a70700 1 -- 172.21.2.102:0/1980327669 >> 172.21.2.102:0/1980327669 conn(0x7fa3100ce690 msgr2=0x7fa3100e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:01.259 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:01.245 7fa314a70700 1 -- 172.21.2.102:0/1980327669 shutdown_connections 2021-02-10T11:23:01.259 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:01.245 7fa314a70700 1 -- 172.21.2.102:0/1980327669 wait complete. 2021-02-10T11:23:01.260 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:01.260 7fa314a70700 1 Processor -- start 2021-02-10T11:23:01.261 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:01.260 7fa314a70700 1 -- start start 2021-02-10T11:23:01.261 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:01.260 7fa314a70700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa310192960 0x7fa310192d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:01.261 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:01.260 7fa314a70700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa3100ad110 0x7fa3100ad510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:01.261 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:01.260 7fa314a70700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa3100adb30 0x7fa31008e460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:01.261 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:01.260 7fa314a70700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3101a03e0 con 0x7fa310192960 2021-02-10T11:23:01.261 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:01.260 7fa314a70700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa310193380 con 0x7fa3100adb30 2021-02-10T11:23:01.261 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:01.260 7fa314a70700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa31008ea80 con 0x7fa3100ad110 2021-02-10T11:23:01.261 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:01.261 7fa30e85e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa310192960 0x7fa310192d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:01.261 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:01.261 7fa30e85e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa310192960 0x7fa310192d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:55954/0 (socket says 172.21.2.102:55954) 2021-02-10T11:23:01.261 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:01.261 7fa30e85e700 1 -- 172.21.2.102:0/160686697 learned_addr learned my addr 172.21.2.102:0/160686697 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:23:01.261 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:01.261 7fa30e05d700 1 --2- 172.21.2.102:0/160686697 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa3100ad110 0x7fa3100ad510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:01.261 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:01.261 7fa30f05f700 1 --2- 172.21.2.102:0/160686697 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa3100adb30 0x7fa31008e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:01.261 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:01.261 7fa30e85e700 1 -- 172.21.2.102:0/160686697 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa3100ad110 msgr2=0x7fa3100ad510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:01.262 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:01.261 7fa30e85e700 1 --2- 172.21.2.102:0/160686697 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa3100ad110 0x7fa3100ad510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:01.262 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:01.261 7fa30e85e700 1 -- 172.21.2.102:0/160686697 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa3100adb30 msgr2=0x7fa31008e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:01.262 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:01.261 7fa30e85e700 1 --2- 172.21.2.102:0/160686697 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa3100adb30 0x7fa31008e460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:01.262 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:01.261 7fa30e85e700 1 -- 172.21.2.102:0/160686697 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa31008eca0 con 0x7fa310192960 2021-02-10T11:23:01.262 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:01.261 7fa30e05d700 1 --2- 172.21.2.102:0/160686697 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa3100ad110 0x7fa3100ad510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:23:01.262 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:01.261 7fa30f05f700 1 --2- 172.21.2.102:0/160686697 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa3100adb30 0x7fa31008e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:23:01.262 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:01.261 7fa30e85e700 1 --2- 172.21.2.102:0/160686697 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa310192960 0x7fa310192d60 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 rx=0x7fa30000b5b0 tx=0x7fa30000b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:01.262 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:01.262 7fa3077fe700 1 -- 172.21.2.102:0/160686697 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa30000bd40 con 0x7fa310192960 2021-02-10T11:23:01.262 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:01.262 7fa3077fe700 1 -- 172.21.2.102:0/160686697 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa300006020 con 0x7fa310192960 2021-02-10T11:23:01.262 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:01.262 7fa3077fe700 1 -- 172.21.2.102:0/160686697 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa3000054a0 con 0x7fa310192960 2021-02-10T11:23:01.262 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:01.262 7fa314a70700 1 -- 172.21.2.102:0/160686697 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa31008eca0 con 0x7fa310192960 2021-02-10T11:23:01.262 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:01.262 7fa314a70700 1 -- 172.21.2.102:0/160686697 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa31008f230 con 0x7fa310192960 2021-02-10T11:23:01.262 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:01.262 7fa3077fe700 1 -- 172.21.2.102:0/160686697 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa300006200 con 0x7fa310192960 2021-02-10T11:23:01.263 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:01.262 7fa3077fe700 1 --2- 172.21.2.102:0/160686697 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa2f40186c0 0x7fa2f401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:01.263 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:01.262 7fa3077fe700 1 -- 172.21.2.102:0/160686697 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fa300032450 con 0x7fa310192960 2021-02-10T11:23:01.263 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:01.263 7fa30e05d700 1 --2- 172.21.2.102:0/160686697 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa2f40186c0 0x7fa2f401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:01.263 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:01.263 7fa30e05d700 1 --2- 172.21.2.102:0/160686697 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa2f40186c0 0x7fa2f401abc0 crc :-1 s=READY pgs=545 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:01.290 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:01.293 7fa314a70700 1 -- 172.21.2.102:0/160686697 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa310107120 con 0x7fa310192960 2021-02-10T11:23:01.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:01.296 7fa3077fe700 1 -- 172.21.2.102:0/160686697 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fa300000940 con 0x7fa310192960 2021-02-10T11:23:01.423 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:01.425 7fa314a70700 1 -- 172.21.2.102:0/160686697 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestVolumes.test_authorize_allow_existing_id_option"]} v 0) v1 -- 0x7fa31007ed80 con 0x7fa310192960 2021-02-10T11:23:01.455 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:01.458 7fa3077fe700 1 -- 172.21.2.102:0/160686697 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestVolumes.test_authorize_allow_existing_id_option"]}]=0 v250) v1 ==== 156+0+0 (secure 0 0 0) 0x7fa30001f530 con 0x7fa310192960 2021-02-10T11:23:01.490 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:01.492 7fa314a70700 1 -- 172.21.2.102:0/160686697 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa2f40186c0 msgr2=0x7fa2f401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:01.490 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:01.492 7fa314a70700 1 --2- 172.21.2.102:0/160686697 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa2f40186c0 0x7fa2f401abc0 crc :-1 s=READY pgs=545 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:01.490 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:01.492 7fa314a70700 1 -- 172.21.2.102:0/160686697 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa310192960 msgr2=0x7fa310192d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:01.490 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:01.492 7fa314a70700 1 --2- 172.21.2.102:0/160686697 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa310192960 0x7fa310192d60 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 rx=0x7fa30000b5b0 tx=0x7fa30000b950).stop 2021-02-10T11:23:01.490 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:01.492 7fa314a70700 1 -- 172.21.2.102:0/160686697 shutdown_connections 2021-02-10T11:23:01.490 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:01.492 7fa314a70700 1 --2- 172.21.2.102:0/160686697 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa2f40186c0 0x7fa2f401abc0 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:01.490 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:01.492 7fa314a70700 1 --2- 172.21.2.102:0/160686697 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa3100adb30 0x7fa31008e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:01.490 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:01.492 7fa314a70700 1 --2- 172.21.2.102:0/160686697 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa3100ad110 0x7fa3100ad510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:01.490 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:01.492 7fa314a70700 1 --2- 172.21.2.102:0/160686697 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa310192960 0x7fa310192d60 unknown :-1 s=CLOSED pgs=1141 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:01.491 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:01.492 7fa314a70700 1 -- 172.21.2.102:0/160686697 >> 172.21.2.102:0/160686697 conn(0x7fa3100891f0 msgr2=0x7fa3101a0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:01.491 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:01.492 7fa314a70700 1 -- 172.21.2.102:0/160686697 shutdown_connections 2021-02-10T11:23:01.491 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:01.492 7fa314a70700 1 -- 172.21.2.102:0/160686697 wait complete. 2021-02-10T11:23:01.498 INFO:tasks.cephfs_test_runner:test_authorize_allow_existing_id_option (tasks.cephfs.test_volumes.TestVolumes) ... ok 2021-02-10T11:23:01.499 INFO:tasks.cephfs_test_runner:Starting test: test_authorize_auth_id_not_created_by_mgr_volumes (tasks.cephfs.test_volumes.TestVolumes) 2021-02-10T11:23:01.499 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestVolumes.test_authorize_auth_id_not_created_by_mgr_volumes' 2021-02-10T11:23:01.925 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:01.910 7f486e2e1700 1 Processor -- start 2021-02-10T11:23:01.925 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:01.910 7f486e2e1700 1 -- start start 2021-02-10T11:23:01.925 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:01.911 7f486e2e1700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f48680e5070 0x7f48681b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:01.925 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:01.911 7f486e2e1700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f48681b3e40 0x7f486807d690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:01.925 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:01.911 7f486e2e1700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f48680e5470 0x7f486807dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:01.925 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:01.911 7f486e2e1700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f486807ed80 con 0x7f48680e5070 2021-02-10T11:23:01.925 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:01.911 7f486e2e1700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f486807e2d0 con 0x7f48680e5470 2021-02-10T11:23:01.925 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:01.911 7f486e2e1700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f486807e4c0 con 0x7f48681b3e40 2021-02-10T11:23:01.926 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:01.911 7f4867fff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f48680e5070 0x7f48681b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:01.926 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:01.911 7f4867fff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f48680e5070 0x7f48681b3820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:55960/0 (socket says 172.21.2.102:55960) 2021-02-10T11:23:01.926 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:01.911 7f4867fff700 1 -- 172.21.2.102:0/860427 learned_addr learned my addr 172.21.2.102:0/860427 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:23:01.926 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:01.911 7f486c8fe700 1 --2- 172.21.2.102:0/860427 >> v2:172.21.2.113:3300/0 conn(0x7f48680e5470 0x7f486807dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:01.926 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:01.911 7f48677fe700 1 --2- 172.21.2.102:0/860427 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f48681b3e40 0x7f486807d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:01.926 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:01.913 7f4867fff700 1 -- 172.21.2.102:0/860427 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f48681b3e40 msgr2=0x7f486807d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:01.926 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:01.913 7f4867fff700 1 --2- 172.21.2.102:0/860427 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f48681b3e40 0x7f486807d690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:01.926 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:01.913 7f4867fff700 1 -- 172.21.2.102:0/860427 >> v2:172.21.2.113:3300/0 conn(0x7f48680e5470 msgr2=0x7f486807dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:01.926 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:01.913 7f4867fff700 1 --2- 172.21.2.102:0/860427 >> v2:172.21.2.113:3300/0 conn(0x7f48680e5470 0x7f486807dcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:01.926 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:01.913 7f4867fff700 1 -- 172.21.2.102:0/860427 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f486807e6e0 con 0x7f48680e5070 2021-02-10T11:23:01.926 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:01.913 7f4867fff700 1 --2- 172.21.2.102:0/860427 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f48680e5070 0x7f48681b3820 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 rx=0x7f4858006700 tx=0x7f4858009b80).ready entity=mon.0 client_cookie=e601018580157f27 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:01.926 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:01.913 7f4866ffd700 1 -- 172.21.2.102:0/860427 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f485800c0c0 con 0x7f48680e5070 2021-02-10T11:23:01.926 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:01.913 7f4866ffd700 1 -- 172.21.2.102:0/860427 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f485800c400 con 0x7f48680e5070 2021-02-10T11:23:01.927 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:01.913 7f4866ffd700 1 -- 172.21.2.102:0/860427 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f485800c920 con 0x7f48680e5070 2021-02-10T11:23:01.927 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:01.913 7f486e2e1700 1 -- 172.21.2.102:0/860427 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f48680e5070 msgr2=0x7f48681b3820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:01.927 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:01.913 7f486e2e1700 1 --2- 172.21.2.102:0/860427 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f48680e5070 0x7f48681b3820 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 rx=0x7f4858006700 tx=0x7f4858009b80).stop 2021-02-10T11:23:01.927 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:01.913 7f486e2e1700 1 -- 172.21.2.102:0/860427 shutdown_connections 2021-02-10T11:23:01.927 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:01.913 7f486e2e1700 1 --2- 172.21.2.102:0/860427 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f48681b3e40 0x7f486807d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:01.927 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:01.913 7f486e2e1700 1 --2- 172.21.2.102:0/860427 >> v2:172.21.2.113:3300/0 conn(0x7f48680e5470 0x7f486807dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:01.927 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:01.913 7f486e2e1700 1 --2- 172.21.2.102:0/860427 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f48680e5070 0x7f48681b3820 unknown :-1 s=CLOSED pgs=1142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:01.927 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:01.913 7f486e2e1700 1 -- 172.21.2.102:0/860427 >> 172.21.2.102:0/860427 conn(0x7f48680ce690 msgr2=0x7f48680e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:01.927 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:01.913 7f486e2e1700 1 -- 172.21.2.102:0/860427 shutdown_connections 2021-02-10T11:23:01.927 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:01.913 7f486e2e1700 1 -- 172.21.2.102:0/860427 wait complete. 2021-02-10T11:23:01.927 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:01.928 7f486e2e1700 1 Processor -- start 2021-02-10T11:23:01.929 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:01.928 7f486e2e1700 1 -- start start 2021-02-10T11:23:01.929 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:01.928 7f486e2e1700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f48681918e0 0x7f4868189c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:01.929 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:01.928 7f486e2e1700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f486818a2a0 0x7f486818a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:01.929 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:01.928 7f486e2e1700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f486818acc0 0x7f4868190070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:01.929 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:01.928 7f486e2e1700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48681a2f00 con 0x7f486818acc0 2021-02-10T11:23:01.929 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:01.928 7f486e2e1700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4868191ce0 con 0x7f486818a2a0 2021-02-10T11:23:01.929 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:01.928 7f486e2e1700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4868191f00 con 0x7f48681918e0 2021-02-10T11:23:01.930 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:01.928 7f486c8fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f486818acc0 0x7f4868190070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:01.930 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:01.928 7f486c8fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f486818acc0 0x7f4868190070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:55968/0 (socket says 172.21.2.102:55968) 2021-02-10T11:23:01.930 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:01.928 7f486c8fe700 1 -- 172.21.2.102:0/3574501754 learned_addr learned my addr 172.21.2.102:0/3574501754 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:23:01.930 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:01.929 7f48677fe700 1 --2- 172.21.2.102:0/3574501754 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f486818a2a0 0x7f486818a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:01.930 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:01.929 7f4867fff700 1 --2- 172.21.2.102:0/3574501754 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f48681918e0 0x7f4868189c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:01.930 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:01.929 7f486c8fe700 1 -- 172.21.2.102:0/3574501754 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f48681918e0 msgr2=0x7f4868189c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:01.930 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:01.929 7f486c8fe700 1 --2- 172.21.2.102:0/3574501754 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f48681918e0 0x7f4868189c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:01.930 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:01.929 7f486c8fe700 1 -- 172.21.2.102:0/3574501754 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f486818a2a0 msgr2=0x7f486818a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:01.930 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:01.929 7f486c8fe700 1 --2- 172.21.2.102:0/3574501754 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f486818a2a0 0x7f486818a6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:01.930 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:01.929 7f486c8fe700 1 -- 172.21.2.102:0/3574501754 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4868190690 con 0x7f486818acc0 2021-02-10T11:23:01.930 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:01.929 7f48677fe700 1 --2- 172.21.2.102:0/3574501754 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f486818a2a0 0x7f486818a6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:23:01.930 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:01.929 7f486c8fe700 1 --2- 172.21.2.102:0/3574501754 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f486818acc0 0x7f4868190070 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 rx=0x7f486000b970 tx=0x7f486000bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:01.930 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:01.929 7f4864ff9700 1 -- 172.21.2.102:0/3574501754 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4860000f90 con 0x7f486818acc0 2021-02-10T11:23:01.931 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:01.929 7f4864ff9700 1 -- 172.21.2.102:0/3574501754 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4860001400 con 0x7f486818acc0 2021-02-10T11:23:01.931 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:01.929 7f4864ff9700 1 -- 172.21.2.102:0/3574501754 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f48600018e0 con 0x7f486818acc0 2021-02-10T11:23:01.931 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:01.929 7f486e2e1700 1 -- 172.21.2.102:0/3574501754 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4868190690 con 0x7f486818acc0 2021-02-10T11:23:01.931 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:01.929 7f486e2e1700 1 -- 172.21.2.102:0/3574501754 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4868190c20 con 0x7f486818acc0 2021-02-10T11:23:01.931 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:01.930 7f4864ff9700 1 -- 172.21.2.102:0/3574501754 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4860000df0 con 0x7f486818acc0 2021-02-10T11:23:01.931 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:01.930 7f4864ff9700 1 --2- 172.21.2.102:0/3574501754 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f48480186d0 0x7f484801abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:01.931 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:01.930 7f4864ff9700 1 -- 172.21.2.102:0/3574501754 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f4860032a20 con 0x7f486818acc0 2021-02-10T11:23:01.931 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:01.930 7f4867fff700 1 --2- 172.21.2.102:0/3574501754 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f48480186d0 0x7f484801abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:01.931 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:01.930 7f4867fff700 1 --2- 172.21.2.102:0/3574501754 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f48480186d0 0x7f484801abd0 crc :-1 s=READY pgs=546 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:01.958 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:01.961 7f486e2e1700 1 -- 172.21.2.102:0/3574501754 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f486807ed80 con 0x7f486818acc0 2021-02-10T11:23:01.960 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:01.963 7f4864ff9700 1 -- 172.21.2.102:0/3574501754 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f486001f8e0 con 0x7f486818acc0 2021-02-10T11:23:02.092 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:02.096 7f486e2e1700 1 -- 172.21.2.102:0/3574501754 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestVolumes.test_authorize_auth_id_not_created_by_mgr_volumes"]} v 0) v1 -- 0x7f486807ed80 con 0x7f486818acc0 2021-02-10T11:23:02.461 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:02.464 7f4864ff9700 1 -- 172.21.2.102:0/3574501754 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestVolumes.test_authorize_auth_id_not_created_by_mgr_volumes"]}]=0 v251) v1 ==== 169+0+0 (secure 0 0 0) 0x7f4860000df0 con 0x7f486818acc0 2021-02-10T11:23:02.511 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:02.515 7f486e2e1700 1 -- 172.21.2.102:0/3574501754 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f48480186d0 msgr2=0x7f484801abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:02.512 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:02.515 7f486e2e1700 1 --2- 172.21.2.102:0/3574501754 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f48480186d0 0x7f484801abd0 crc :-1 s=READY pgs=546 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:02.512 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:02.515 7f486e2e1700 1 -- 172.21.2.102:0/3574501754 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f486818acc0 msgr2=0x7f4868190070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:02.512 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:02.515 7f486e2e1700 1 --2- 172.21.2.102:0/3574501754 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f486818acc0 0x7f4868190070 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 rx=0x7f486000b970 tx=0x7f486000bd10).stop 2021-02-10T11:23:02.512 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:02.515 7f486e2e1700 1 -- 172.21.2.102:0/3574501754 shutdown_connections 2021-02-10T11:23:02.512 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:02.515 7f486e2e1700 1 --2- 172.21.2.102:0/3574501754 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f486818acc0 0x7f4868190070 unknown :-1 s=CLOSED pgs=1143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:02.512 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:02.515 7f486e2e1700 1 --2- 172.21.2.102:0/3574501754 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f48480186d0 0x7f484801abd0 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:02.512 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:02.515 7f486e2e1700 1 --2- 172.21.2.102:0/3574501754 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f486818a2a0 0x7f486818a6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:02.512 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:02.515 7f486e2e1700 1 --2- 172.21.2.102:0/3574501754 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f48681918e0 0x7f4868189c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:02.512 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:02.515 7f486e2e1700 1 -- 172.21.2.102:0/3574501754 >> 172.21.2.102:0/3574501754 conn(0x7f48681a31d0 msgr2=0x7f486808b830 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:02.512 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:02.515 7f486e2e1700 1 -- 172.21.2.102:0/3574501754 shutdown_connections 2021-02-10T11:23:02.513 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:02.515 7f486e2e1700 1 -- 172.21.2.102:0/3574501754 wait complete. 2021-02-10T11:23:02.520 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config set mon mon_allow_pool_delete True 2021-02-10T11:23:02.712 DEBUG:teuthology.orchestra.run.gibba002:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:23:02.715 DEBUG:teuthology.orchestra.run.gibba013:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:23:02.949 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:02.936 7fc183f41700 1 Processor -- start 2021-02-10T11:23:02.949 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:02.936 7fc183f41700 1 -- start start 2021-02-10T11:23:02.949 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:02.936 7fc183f41700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fc17c1b0870 0x7fc17c0f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:02.949 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:02.936 7fc183f41700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fc17c0efe00 0x7fc17c0f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:02.950 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:02.936 7fc183f41700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc17c1b0c70 con 0x7fc17c1b0470 2021-02-10T11:23:02.950 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:02.936 7fc183f41700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc17c0e5ea0 con 0x7fc17c0efe00 2021-02-10T11:23:02.950 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:02.936 7fc183f41700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc17c0f0b50 con 0x7fc17c1b0870 2021-02-10T11:23:02.950 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:02.937 7fc181d5d700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fc17c1b0470 0x7fc17c0ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:48196/0 (socket says 172.21.2.102:48196) 2021-02-10T11:23:02.950 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:02.937 7fc181d5d700 1 -- 172.21.2.102:0/2490817244 learned_addr learned my addr 172.21.2.102:0/2490817244 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:23:02.950 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:02.937 7fc18255e700 1 --2- 172.21.2.102:0/2490817244 >> v2:172.21.2.102:3300/0 conn(0x7fc17c0efe00 0x7fc17c0f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:02.950 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:02.937 7fc180d5b700 1 -- 172.21.2.102:0/2490817244 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3108030955 0 0) 0x7fc1700010b0 con 0x7fc17c1b0470 2021-02-10T11:23:02.950 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:02.937 7fc180d5b700 1 -- 172.21.2.102:0/2490817244 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc168002fe0 con 0x7fc17c1b0470 2021-02-10T11:23:02.950 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:02.937 7fc18255e700 1 -- 172.21.2.102:0/2490817244 >> v2:172.21.2.113:3300/0 conn(0x7fc17c1b0870 msgr2=0x7fc17c0f20e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:23:02.950 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:02.937 7fc18255e700 1 --2- 172.21.2.102:0/2490817244 >> v2:172.21.2.113:3300/0 conn(0x7fc17c1b0870 0x7fc17c0f20e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:02.950 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:02.937 7fc18255e700 1 -- 172.21.2.102:0/2490817244 >> v1:172.21.2.102:6789/0 conn(0x7fc17c1b0470 legacy=0x7fc17c0ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:02.950 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:02.937 7fc18255e700 1 -- 172.21.2.102:0/2490817244 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc17c0f0820 con 0x7fc17c0efe00 2021-02-10T11:23:02.950 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:02.937 7fc18255e700 1 --2- 172.21.2.102:0/2490817244 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc17c0efe00 0x7fc17c0f0200 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 rx=0x7fc174006e30 tx=0x7fc174009e60).ready entity=mon.0 client_cookie=be9603345b039b5c server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:02.951 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:02.937 7fc180d5b700 1 -- 172.21.2.102:0/2490817244 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc174006a50 con 0x7fc17c0efe00 2021-02-10T11:23:02.951 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:02.937 7fc180d5b700 1 -- 172.21.2.102:0/2490817244 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc17400b300 con 0x7fc17c0efe00 2021-02-10T11:23:02.951 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:02.937 7fc180d5b700 1 -- 172.21.2.102:0/2490817244 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc17400b820 con 0x7fc17c0efe00 2021-02-10T11:23:02.951 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:02.937 7fc183f41700 1 -- 172.21.2.102:0/2490817244 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc17c0efe00 msgr2=0x7fc17c0f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:02.951 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:02.937 7fc183f41700 1 --2- 172.21.2.102:0/2490817244 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc17c0efe00 0x7fc17c0f0200 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 rx=0x7fc174006e30 tx=0x7fc174009e60).stop 2021-02-10T11:23:02.951 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:02.938 7fc183f41700 1 -- 172.21.2.102:0/2490817244 shutdown_connections 2021-02-10T11:23:02.951 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:02.938 7fc183f41700 1 --2- 172.21.2.102:0/2490817244 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc17c0efe00 0x7fc17c0f0200 unknown :-1 s=CLOSED pgs=1145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:02.951 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:02.938 7fc183f41700 1 --2- 172.21.2.102:0/2490817244 >> v2:172.21.2.113:3300/0 conn(0x7fc17c1b0870 0x7fc17c0f20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:02.951 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:02.938 7fc183f41700 1 -- 172.21.2.102:0/2490817244 >> 172.21.2.102:0/2490817244 conn(0x7fc17c0ce3e0 msgr2=0x7fc17c0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:02.951 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:02.938 7fc183f41700 1 -- 172.21.2.102:0/2490817244 shutdown_connections 2021-02-10T11:23:02.951 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:02.938 7fc183f41700 1 -- 172.21.2.102:0/2490817244 wait complete. 2021-02-10T11:23:02.953 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:02.953 7fc183f41700 1 Processor -- start 2021-02-10T11:23:02.953 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:02.953 7fc183f41700 1 -- start start 2021-02-10T11:23:02.953 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:02.953 7fc183f41700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc17c0a4840 0x7fc17c10a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:02.953 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:02.953 7fc183f41700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc17c08d6c0 0x7fc17c0a5490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:02.953 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:02.953 7fc183f41700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc17c0a5ab0 0x7fc17c0a5eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:02.953 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:02.953 7fc183f41700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc17c10a7f0 con 0x7fc17c0a4840 2021-02-10T11:23:02.953 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:02.953 7fc183f41700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc17c0a4c40 con 0x7fc17c0a5ab0 2021-02-10T11:23:02.953 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:02.953 7fc183f41700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc17c1906b0 con 0x7fc17c08d6c0 2021-02-10T11:23:02.953 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:02.954 7fc181d5d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc17c0a4840 0x7fc17c10a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:02.954 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:02.954 7fc181d5d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc17c0a4840 0x7fc17c10a1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:55978/0 (socket says 172.21.2.102:55978) 2021-02-10T11:23:02.954 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:02.954 7fc181d5d700 1 -- 172.21.2.102:0/1916258595 learned_addr learned my addr 172.21.2.102:0/1916258595 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:23:02.954 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:02.954 7fc18255e700 1 --2- 172.21.2.102:0/1916258595 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc17c0a5ab0 0x7fc17c0a5eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:02.954 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:02.954 7fc18155c700 1 --2- 172.21.2.102:0/1916258595 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc17c08d6c0 0x7fc17c0a5490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:02.954 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:02.954 7fc181d5d700 1 -- 172.21.2.102:0/1916258595 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc17c08d6c0 msgr2=0x7fc17c0a5490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:02.954 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:02.954 7fc181d5d700 1 --2- 172.21.2.102:0/1916258595 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc17c08d6c0 0x7fc17c0a5490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:02.954 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:02.954 7fc181d5d700 1 -- 172.21.2.102:0/1916258595 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc17c0a5ab0 msgr2=0x7fc17c0a5eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:02.954 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:02.954 7fc181d5d700 1 --2- 172.21.2.102:0/1916258595 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc17c0a5ab0 0x7fc17c0a5eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:02.955 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:02.954 7fc181d5d700 1 -- 172.21.2.102:0/1916258595 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc17c190b50 con 0x7fc17c0a4840 2021-02-10T11:23:02.955 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:02.954 7fc181d5d700 1 --2- 172.21.2.102:0/1916258595 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc17c0a4840 0x7fc17c10a1d0 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 rx=0x7fc170001c60 tx=0x7fc17000a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:02.955 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:02.955 7fc16e7fc700 1 -- 172.21.2.102:0/1916258595 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc1700010b0 con 0x7fc17c0a4840 2021-02-10T11:23:02.955 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:02.955 7fc16e7fc700 1 -- 172.21.2.102:0/1916258595 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc170004140 con 0x7fc17c0a4840 2021-02-10T11:23:02.955 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:02.955 7fc16e7fc700 1 -- 172.21.2.102:0/1916258595 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc170004660 con 0x7fc17c0a4840 2021-02-10T11:23:02.955 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:02.955 7fc183f41700 1 -- 172.21.2.102:0/1916258595 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc17c190b50 con 0x7fc17c0a4840 2021-02-10T11:23:02.955 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:02.955 7fc183f41700 1 -- 172.21.2.102:0/1916258595 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc17c190b50 con 0x7fc17c0a4840 2021-02-10T11:23:02.955 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:02.955 7fc16e7fc700 1 -- 172.21.2.102:0/1916258595 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fc170001090 con 0x7fc17c0a4840 2021-02-10T11:23:02.955 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:02.955 7fc16e7fc700 1 --2- 172.21.2.102:0/1916258595 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc1640186c0 0x7fc16401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:02.955 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:02.955 7fc16e7fc700 1 -- 172.21.2.102:0/1916258595 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fc170032540 con 0x7fc17c0a4840 2021-02-10T11:23:02.955 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:02.956 7fc18155c700 1 --2- 172.21.2.102:0/1916258595 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc1640186c0 0x7fc16401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:02.955 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:02.956 7fc18155c700 1 --2- 172.21.2.102:0/1916258595 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc1640186c0 0x7fc16401abc0 crc :-1 s=READY pgs=547 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:02.983 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:02.986 7fc183f41700 1 -- 172.21.2.102:0/1916258595 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc17c0a4f60 con 0x7fc17c0a4840 2021-02-10T11:23:02.985 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:02.989 7fc16e7fc700 1 -- 172.21.2.102:0/1916258595 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fc170000940 con 0x7fc17c0a4840 2021-02-10T11:23:03.136 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:03.139 7fc183f41700 1 -- 172.21.2.102:0/1916258595 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fc17c0a4f60 con 0x7fc17c0a4840 2021-02-10T11:23:03.140 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:03.143 7fc16e7fc700 1 -- 172.21.2.102:0/1916258595 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v6)=0 v6) v1 ==== 122+0+0 (secure 0 0 0) 0x7fc170001090 con 0x7fc17c0a4840 2021-02-10T11:23:03.145 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:03.148 7fc183f41700 1 -- 172.21.2.102:0/1916258595 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc1640186c0 msgr2=0x7fc16401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:03.145 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:03.148 7fc183f41700 1 --2- 172.21.2.102:0/1916258595 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc1640186c0 0x7fc16401abc0 crc :-1 s=READY pgs=547 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:03.145 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:03.148 7fc183f41700 1 -- 172.21.2.102:0/1916258595 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc17c0a4840 msgr2=0x7fc17c10a1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:03.145 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:03.148 7fc183f41700 1 --2- 172.21.2.102:0/1916258595 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc17c0a4840 0x7fc17c10a1d0 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 rx=0x7fc170001c60 tx=0x7fc17000a8d0).stop 2021-02-10T11:23:03.145 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:03.148 7fc183f41700 1 -- 172.21.2.102:0/1916258595 shutdown_connections 2021-02-10T11:23:03.145 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:03.148 7fc183f41700 1 --2- 172.21.2.102:0/1916258595 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc1640186c0 0x7fc16401abc0 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:03.145 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:03.148 7fc183f41700 1 --2- 172.21.2.102:0/1916258595 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc17c0a5ab0 0x7fc17c0a5eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:03.146 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:03.148 7fc183f41700 1 --2- 172.21.2.102:0/1916258595 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc17c08d6c0 0x7fc17c0a5490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:03.146 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:03.148 7fc183f41700 1 --2- 172.21.2.102:0/1916258595 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc17c0a4840 0x7fc17c10a1d0 unknown :-1 s=CLOSED pgs=1146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:03.146 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:03.148 7fc183f41700 1 -- 172.21.2.102:0/1916258595 >> 172.21.2.102:0/1916258595 conn(0x7fc17c0bbf60 msgr2=0x7fc17c105d10 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:03.146 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:03.148 7fc183f41700 1 -- 172.21.2.102:0/1916258595 shutdown_connections 2021-02-10T11:23:03.146 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:03.148 7fc183f41700 1 -- 172.21.2.102:0/1916258595 wait complete. 2021-02-10T11:23:03.153 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2021-02-10T11:23:03.153 DEBUG:teuthology.orchestra.run.gibba002:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:23:03.498 DEBUG:teuthology.orchestra.run.gibba013:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:23:03.843 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2021-02-10T11:23:03.843 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:23:03.843 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:23:03.843 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:23:04.178 INFO:teuthology.orchestra.run.gibba002.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.0’: No such file or directory 2021-02-10T11:23:04.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:23:04.179 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.0 2021-02-10T11:23:04.179 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:23:04.179 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:23:04.179 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:23:04.512 INFO:teuthology.orchestra.run.gibba002.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.1’: No such file or directory 2021-02-10T11:23:04.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:23:04.513 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2021-02-10T11:23:04.513 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:23:04.514 DEBUG:teuthology.orchestra.run.gibba013:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:23:04.514 DEBUG:teuthology.orchestra.run.gibba013:> ' -- /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:23:04.848 INFO:teuthology.orchestra.run.gibba013.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.2’: No such file or directory 2021-02-10T11:23:04.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:23:04.849 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2021-02-10T11:23:04.849 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:23:04.849 DEBUG:teuthology.orchestra.run.gibba013:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:23:04.849 DEBUG:teuthology.orchestra.run.gibba013:> ' -- /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:23:05.183 INFO:teuthology.orchestra.run.gibba013.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.3’: No such file or directory 2021-02-10T11:23:05.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:23:05.184 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.3 2021-02-10T11:23:05.184 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json-pretty 2021-02-10T11:23:05.613 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:05.599 7fb8c1702700 1 Processor -- start 2021-02-10T11:23:05.613 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:05.599 7fb8c1702700 1 -- start start 2021-02-10T11:23:05.613 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:05.599 7fb8c1702700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fb8bc0e5070 0x7fb8bc1b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:05.613 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:05.599 7fb8c1702700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb8bc0e5470 0x7fb8bc080310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:05.614 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:05.599 7fb8c1702700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8bc07ed80 con 0x7fb8bc1b3e40 2021-02-10T11:23:05.614 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:05.599 7fb8c1702700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8bc0d1790 con 0x7fb8bc0e5070 2021-02-10T11:23:05.614 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:05.599 7fb8c1702700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8bc0d19b0 con 0x7fb8bc0e5470 2021-02-10T11:23:05.614 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:05.600 7fb8baabe700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fb8bc1b3e40 0x7fb8bc07d690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:48210/0 (socket says 172.21.2.102:48210) 2021-02-10T11:23:05.614 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:05.600 7fb8baabe700 1 -- 172.21.2.102:0/1821594583 learned_addr learned my addr 172.21.2.102:0/1821594583 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:23:05.614 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:05.600 7fb8ba2bd700 1 -- 172.21.2.102:0/1821594583 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2739748152 0 0) 0x7fb8b00010b0 con 0x7fb8bc1b3e40 2021-02-10T11:23:05.614 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:05.600 7fb8ba2bd700 1 -- 172.21.2.102:0/1821594583 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8a4002fe0 con 0x7fb8bc1b3e40 2021-02-10T11:23:05.614 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:05.600 7fb8bb2bf700 1 --2- 172.21.2.102:0/1821594583 >> v2:172.21.2.113:3300/0 conn(0x7fb8bc0e5070 0x7fb8bc1b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:05.614 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:05.600 7fb8bbac0700 1 --2- 172.21.2.102:0/1821594583 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb8bc0e5470 0x7fb8bc080310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:05.614 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:05.600 7fb8ba2bd700 1 -- 172.21.2.102:0/1821594583 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3638897429 0 0) 0x7fb8b00010b0 con 0x7fb8bc1b3e40 2021-02-10T11:23:05.614 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:05.600 7fb8ba2bd700 1 -- 172.21.2.102:0/1821594583 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb8bc0e5470 msgr2=0x7fb8bc080310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:05.614 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:05.600 7fb8ba2bd700 1 --2- 172.21.2.102:0/1821594583 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb8bc0e5470 0x7fb8bc080310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:05.615 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:05.600 7fb8ba2bd700 1 -- 172.21.2.102:0/1821594583 >> v2:172.21.2.113:3300/0 conn(0x7fb8bc0e5070 msgr2=0x7fb8bc1b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:05.615 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:05.600 7fb8ba2bd700 1 --2- 172.21.2.102:0/1821594583 >> v2:172.21.2.113:3300/0 conn(0x7fb8bc0e5070 0x7fb8bc1b3820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:05.615 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:05.600 7fb8ba2bd700 1 -- 172.21.2.102:0/1821594583 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8bc0d1bd0 con 0x7fb8bc1b3e40 2021-02-10T11:23:05.615 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:05.600 7fb8ba2bd700 1 -- 172.21.2.102:0/1821594583 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7fb8b00015b0 con 0x7fb8bc1b3e40 2021-02-10T11:23:05.615 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:05.600 7fb8ba2bd700 1 -- 172.21.2.102:0/1821594583 >> v1:172.21.2.102:6789/0 conn(0x7fb8bc1b3e40 legacy=0x7fb8bc07d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:05.615 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:05.600 7fb8ba2bd700 1 --2- 172.21.2.102:0/1821594583 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb8a4003b90 0x7fb8a4006040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:05.615 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:05.601 7fb8ba2bd700 1 --2- 172.21.2.102:0/1821594583 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb8a404a820 0x7fb8a404cca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:05.615 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:05.601 7fb8ba2bd700 1 --2- 172.21.2.102:0/1821594583 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb8a404d5a0 0x7fb8a404fa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:05.615 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:05.601 7fb8ba2bd700 1 -- 172.21.2.102:0/1821594583 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8a40502e0 con 0x7fb8a4003b90 2021-02-10T11:23:05.615 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:05.601 7fb8ba2bd700 1 -- 172.21.2.102:0/1821594583 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8a4050500 con 0x7fb8a404a820 2021-02-10T11:23:05.615 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:05.601 7fb8ba2bd700 1 -- 172.21.2.102:0/1821594583 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8a4050720 con 0x7fb8a404d5a0 2021-02-10T11:23:05.615 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:05.601 7fb8bb2bf700 1 --2- 172.21.2.102:0/1821594583 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb8a4003b90 0x7fb8a4006040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:05.615 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:05.601 7fb8bbac0700 1 --2- 172.21.2.102:0/1821594583 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb8a404d5a0 0x7fb8a404fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:05.615 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:05.601 7fb8baabe700 1 --2- 172.21.2.102:0/1821594583 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb8a404a820 0x7fb8a404cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:05.616 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:05.601 7fb8bb2bf700 1 -- 172.21.2.102:0/1821594583 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb8a404d5a0 msgr2=0x7fb8a404fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:05.616 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:05.601 7fb8bb2bf700 1 --2- 172.21.2.102:0/1821594583 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb8a404d5a0 0x7fb8a404fa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:05.616 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:05.601 7fb8bb2bf700 1 -- 172.21.2.102:0/1821594583 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb8a404a820 msgr2=0x7fb8a404cca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:05.616 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:05.601 7fb8bb2bf700 1 --2- 172.21.2.102:0/1821594583 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb8a404a820 0x7fb8a404cca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:05.616 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:05.601 7fb8bb2bf700 1 -- 172.21.2.102:0/1821594583 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8a40509c0 con 0x7fb8a4003b90 2021-02-10T11:23:05.616 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:05.601 7fb8bb2bf700 1 --2- 172.21.2.102:0/1821594583 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb8a4003b90 0x7fb8a4006040 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 rx=0x7fb8ac000b50 tx=0x7fb8ac00cc80).ready entity=mon.0 client_cookie=5111aab2f9764a7f server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:05.616 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:05.601 7fb8ba2bd700 1 -- 172.21.2.102:0/1821594583 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb8ac006cb0 con 0x7fb8a4003b90 2021-02-10T11:23:05.616 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:05.601 7fb8ba2bd700 1 -- 172.21.2.102:0/1821594583 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8ac009310 con 0x7fb8a4003b90 2021-02-10T11:23:05.616 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:05.601 7fb8ba2bd700 1 -- 172.21.2.102:0/1821594583 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb8ac0097f0 con 0x7fb8a4003b90 2021-02-10T11:23:05.616 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:05.601 7fb8c1702700 1 -- 172.21.2.102:0/1821594583 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb8a4003b90 msgr2=0x7fb8a4006040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:05.616 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:05.601 7fb8c1702700 1 --2- 172.21.2.102:0/1821594583 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb8a4003b90 0x7fb8a4006040 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 rx=0x7fb8ac000b50 tx=0x7fb8ac00cc80).stop 2021-02-10T11:23:05.616 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:05.601 7fb8bbac0700 1 -- 172.21.2.102:0/1821594583 reap_dead start 2021-02-10T11:23:05.616 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:05.602 7fb8c1702700 1 -- 172.21.2.102:0/1821594583 shutdown_connections 2021-02-10T11:23:05.617 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:05.602 7fb8c1702700 1 -- 172.21.2.102:0/1821594583 >> 172.21.2.102:0/1821594583 conn(0x7fb8bc0ce690 msgr2=0x7fb8bc0e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:05.617 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:05.602 7fb8c1702700 1 -- 172.21.2.102:0/1821594583 shutdown_connections 2021-02-10T11:23:05.617 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:05.602 7fb8c1702700 1 -- 172.21.2.102:0/1821594583 wait complete. 2021-02-10T11:23:05.617 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:05.617 7fb8c1702700 1 Processor -- start 2021-02-10T11:23:05.617 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:05.617 7fb8c1702700 1 -- start start 2021-02-10T11:23:05.618 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:05.617 7fb8c1702700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb8bc194000 0x7fb8bc19f680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:05.618 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:05.617 7fb8c1702700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb8bc19fde0 0x7fb8bc09bed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:05.618 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:05.617 7fb8c1702700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb8bc1a01e0 0x7fb8bc0895c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:05.618 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:05.617 7fb8c1702700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8bc194400 con 0x7fb8bc1a01e0 2021-02-10T11:23:05.618 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:05.617 7fb8c1702700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8bc09c4f0 con 0x7fb8bc19fde0 2021-02-10T11:23:05.619 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:05.617 7fb8c1702700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8bc089be0 con 0x7fb8bc194000 2021-02-10T11:23:05.619 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:05.617 7fb8bbac0700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb8bc1a01e0 0x7fb8bc0895c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:05.619 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:05.617 7fb8bbac0700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb8bc1a01e0 0x7fb8bc0895c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:55998/0 (socket says 172.21.2.102:55998) 2021-02-10T11:23:05.619 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:05.617 7fb8bbac0700 1 -- 172.21.2.102:0/564380757 learned_addr learned my addr 172.21.2.102:0/564380757 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:23:05.619 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:05.618 7fb8baabe700 1 --2- 172.21.2.102:0/564380757 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb8bc19fde0 0x7fb8bc09bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:05.619 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:05.618 7fb8bb2bf700 1 --2- 172.21.2.102:0/564380757 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb8bc194000 0x7fb8bc19f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:05.619 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:05.618 7fb8bbac0700 1 -- 172.21.2.102:0/564380757 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb8bc194000 msgr2=0x7fb8bc19f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:05.619 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:05.618 7fb8bbac0700 1 --2- 172.21.2.102:0/564380757 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb8bc194000 0x7fb8bc19f680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:05.619 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:05.618 7fb8bbac0700 1 -- 172.21.2.102:0/564380757 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb8bc19fde0 msgr2=0x7fb8bc09bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:05.619 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:05.618 7fb8bbac0700 1 --2- 172.21.2.102:0/564380757 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb8bc19fde0 0x7fb8bc09bed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:05.619 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:05.618 7fb8bbac0700 1 -- 172.21.2.102:0/564380757 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8bc089e00 con 0x7fb8bc1a01e0 2021-02-10T11:23:05.619 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:05.618 7fb8baabe700 1 --2- 172.21.2.102:0/564380757 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb8bc19fde0 0x7fb8bc09bed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:23:05.619 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:05.618 7fb8bbac0700 1 --2- 172.21.2.102:0/564380757 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb8bc1a01e0 0x7fb8bc0895c0 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 rx=0x7fb8b4009a30 tx=0x7fb8b400bd20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:05.620 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:05.618 7fb8a3fff700 1 -- 172.21.2.102:0/564380757 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb8b4001400 con 0x7fb8bc1a01e0 2021-02-10T11:23:05.620 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:05.618 7fb8a3fff700 1 -- 172.21.2.102:0/564380757 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8b400be60 con 0x7fb8bc1a01e0 2021-02-10T11:23:05.620 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:05.618 7fb8a3fff700 1 -- 172.21.2.102:0/564380757 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb8b4001640 con 0x7fb8bc1a01e0 2021-02-10T11:23:05.620 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:05.618 7fb8c1702700 1 -- 172.21.2.102:0/564380757 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8bc089e00 con 0x7fb8bc1a01e0 2021-02-10T11:23:05.620 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:05.618 7fb8c1702700 1 -- 172.21.2.102:0/564380757 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8bc089e00 con 0x7fb8bc1a01e0 2021-02-10T11:23:05.620 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:05.619 7fb8a3fff700 1 -- 172.21.2.102:0/564380757 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fb8b40011f0 con 0x7fb8bc1a01e0 2021-02-10T11:23:05.620 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:05.619 7fb8a3fff700 1 --2- 172.21.2.102:0/564380757 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb89c0186c0 0x7fb89c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:05.620 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:05.619 7fb8a3fff700 1 -- 172.21.2.102:0/564380757 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fb8b4032990 con 0x7fb8bc1a01e0 2021-02-10T11:23:05.620 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:05.619 7fb8bb2bf700 1 --2- 172.21.2.102:0/564380757 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb89c0186c0 0x7fb89c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:05.620 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:05.619 7fb8bb2bf700 1 --2- 172.21.2.102:0/564380757 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb89c0186c0 0x7fb89c01abc0 crc :-1 s=READY pgs=548 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:05.646 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:05.649 7fb8c1702700 1 -- 172.21.2.102:0/564380757 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8bc09c710 con 0x7fb8bc1a01e0 2021-02-10T11:23:05.648 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:05.652 7fb8a3fff700 1 -- 172.21.2.102:0/564380757 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fb8b4001f00 con 0x7fb8bc1a01e0 2021-02-10T11:23:05.775 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:05.778 7fb8c1702700 1 -- 172.21.2.102:0/564380757 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7fb8bc089090 con 0x7fb8bc1a01e0 2021-02-10T11:23:05.776 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:05.779 7fb8a3fff700 1 -- 172.21.2.102:0/564380757 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v39) v1 ==== 81+0+30360 (secure 0 0 0) 0x7fb8b40011f0 con 0x7fb8bc1a01e0 2021-02-10T11:23:05.778 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:23:05.779 INFO:teuthology.orchestra.run.gibba002.stdout:{ 2021-02-10T11:23:05.779 INFO:teuthology.orchestra.run.gibba002.stdout: "epoch": 39, 2021-02-10T11:23:05.779 INFO:teuthology.orchestra.run.gibba002.stdout: "fsid": "68db30ea-dbce-4aef-a6df-f01c6fea0513", 2021-02-10T11:23:05.779 INFO:teuthology.orchestra.run.gibba002.stdout: "created": "2021-02-10 11:03:06.331557", 2021-02-10T11:23:05.779 INFO:teuthology.orchestra.run.gibba002.stdout: "modified": "2021-02-10 11:22:27.123992", 2021-02-10T11:23:05.779 INFO:teuthology.orchestra.run.gibba002.stdout: "last_up_change": "2021-02-10 11:03:32.690562", 2021-02-10T11:23:05.779 INFO:teuthology.orchestra.run.gibba002.stdout: "last_in_change": "2021-02-10 11:03:32.690562", 2021-02-10T11:23:05.779 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2021-02-10T11:23:05.779 INFO:teuthology.orchestra.run.gibba002.stdout: "flags_num": 5799936, 2021-02-10T11:23:05.779 INFO:teuthology.orchestra.run.gibba002.stdout: "flags_set": [ 2021-02-10T11:23:05.779 INFO:teuthology.orchestra.run.gibba002.stdout: "pglog_hardlimit", 2021-02-10T11:23:05.779 INFO:teuthology.orchestra.run.gibba002.stdout: "purged_snapdirs", 2021-02-10T11:23:05.779 INFO:teuthology.orchestra.run.gibba002.stdout: "recovery_deletes", 2021-02-10T11:23:05.780 INFO:teuthology.orchestra.run.gibba002.stdout: "sortbitwise" 2021-02-10T11:23:05.780 INFO:teuthology.orchestra.run.gibba002.stdout: ], 2021-02-10T11:23:05.780 INFO:teuthology.orchestra.run.gibba002.stdout: "crush_version": 4, 2021-02-10T11:23:05.780 INFO:teuthology.orchestra.run.gibba002.stdout: "full_ratio": 0.94999998807907104, 2021-02-10T11:23:05.780 INFO:teuthology.orchestra.run.gibba002.stdout: "backfillfull_ratio": 0.89999997615814209, 2021-02-10T11:23:05.780 INFO:teuthology.orchestra.run.gibba002.stdout: "nearfull_ratio": 0.85000002384185791, 2021-02-10T11:23:05.780 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_snapshot": "", 2021-02-10T11:23:05.780 INFO:teuthology.orchestra.run.gibba002.stdout: "pool_max": 7, 2021-02-10T11:23:05.780 INFO:teuthology.orchestra.run.gibba002.stdout: "max_osd": 8, 2021-02-10T11:23:05.780 INFO:teuthology.orchestra.run.gibba002.stdout: "require_min_compat_client": "jewel", 2021-02-10T11:23:05.780 INFO:teuthology.orchestra.run.gibba002.stdout: "min_compat_client": "jewel", 2021-02-10T11:23:05.780 INFO:teuthology.orchestra.run.gibba002.stdout: "require_osd_release": "nautilus", 2021-02-10T11:23:05.780 INFO:teuthology.orchestra.run.gibba002.stdout: "pools": [ 2021-02-10T11:23:05.781 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:05.781 INFO:teuthology.orchestra.run.gibba002.stdout: "pool": 1, 2021-02-10T11:23:05.781 INFO:teuthology.orchestra.run.gibba002.stdout: "pool_name": "rbd", 2021-02-10T11:23:05.781 INFO:teuthology.orchestra.run.gibba002.stdout: "create_time": "2021-02-10 11:03:37.322519", 2021-02-10T11:23:05.781 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 1, 2021-02-10T11:23:05.781 INFO:teuthology.orchestra.run.gibba002.stdout: "flags_names": "hashpspool", 2021-02-10T11:23:05.781 INFO:teuthology.orchestra.run.gibba002.stdout: "type": 1, 2021-02-10T11:23:05.781 INFO:teuthology.orchestra.run.gibba002.stdout: "size": 2, 2021-02-10T11:23:05.781 INFO:teuthology.orchestra.run.gibba002.stdout: "min_size": 1, 2021-02-10T11:23:05.781 INFO:teuthology.orchestra.run.gibba002.stdout: "crush_rule": 0, 2021-02-10T11:23:05.781 INFO:teuthology.orchestra.run.gibba002.stdout: "object_hash": 2, 2021-02-10T11:23:05.781 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:23:05.782 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num": 8, 2021-02-10T11:23:05.782 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_placement_num": 8, 2021-02-10T11:23:05.782 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_placement_num_target": 8, 2021-02-10T11:23:05.782 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num_target": 8, 2021-02-10T11:23:05.782 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num_pending": 8, 2021-02-10T11:23:05.782 INFO:teuthology.orchestra.run.gibba002.stdout: "last_pg_merge_meta": { 2021-02-10T11:23:05.782 INFO:teuthology.orchestra.run.gibba002.stdout: "source_pgid": "0.0", 2021-02-10T11:23:05.782 INFO:teuthology.orchestra.run.gibba002.stdout: "ready_epoch": 0, 2021-02-10T11:23:05.782 INFO:teuthology.orchestra.run.gibba002.stdout: "last_epoch_started": 0, 2021-02-10T11:23:05.782 INFO:teuthology.orchestra.run.gibba002.stdout: "last_epoch_clean": 0, 2021-02-10T11:23:05.783 INFO:teuthology.orchestra.run.gibba002.stdout: "source_version": "0'0", 2021-02-10T11:23:05.783 INFO:teuthology.orchestra.run.gibba002.stdout: "target_version": "0'0" 2021-02-10T11:23:05.783 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:05.783 INFO:teuthology.orchestra.run.gibba002.stdout: "last_change": "16", 2021-02-10T11:23:05.783 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend": "0", 2021-02-10T11:23:05.783 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:23:05.783 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:23:05.783 INFO:teuthology.orchestra.run.gibba002.stdout: "auid": 0, 2021-02-10T11:23:05.783 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:23:05.783 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_seq": 0, 2021-02-10T11:23:05.783 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_epoch": 0, 2021-02-10T11:23:05.784 INFO:teuthology.orchestra.run.gibba002.stdout: "pool_snaps": [], 2021-02-10T11:23:05.784 INFO:teuthology.orchestra.run.gibba002.stdout: "removed_snaps": "[]", 2021-02-10T11:23:05.784 INFO:teuthology.orchestra.run.gibba002.stdout: "quota_max_bytes": 0, 2021-02-10T11:23:05.784 INFO:teuthology.orchestra.run.gibba002.stdout: "quota_max_objects": 0, 2021-02-10T11:23:05.784 INFO:teuthology.orchestra.run.gibba002.stdout: "tiers": [], 2021-02-10T11:23:05.784 INFO:teuthology.orchestra.run.gibba002.stdout: "tier_of": -1, 2021-02-10T11:23:05.784 INFO:teuthology.orchestra.run.gibba002.stdout: "read_tier": -1, 2021-02-10T11:23:05.784 INFO:teuthology.orchestra.run.gibba002.stdout: "write_tier": -1, 2021-02-10T11:23:05.784 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_mode": "none", 2021-02-10T11:23:05.784 INFO:teuthology.orchestra.run.gibba002.stdout: "target_max_bytes": 0, 2021-02-10T11:23:05.785 INFO:teuthology.orchestra.run.gibba002.stdout: "target_max_objects": 0, 2021-02-10T11:23:05.785 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:23:05.785 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:23:05.785 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:23:05.785 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_min_flush_age": 0, 2021-02-10T11:23:05.785 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_min_evict_age": 0, 2021-02-10T11:23:05.785 INFO:teuthology.orchestra.run.gibba002.stdout: "erasure_code_profile": "", 2021-02-10T11:23:05.785 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_params": { 2021-02-10T11:23:05.785 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "none" 2021-02-10T11:23:05.785 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:05.785 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_period": 0, 2021-02-10T11:23:05.785 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_count": 0, 2021-02-10T11:23:05.785 INFO:teuthology.orchestra.run.gibba002.stdout: "use_gmt_hitset": true, 2021-02-10T11:23:05.786 INFO:teuthology.orchestra.run.gibba002.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:23:05.786 INFO:teuthology.orchestra.run.gibba002.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:23:05.786 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:23:05.786 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:23:05.786 INFO:teuthology.orchestra.run.gibba002.stdout: "grade_table": [], 2021-02-10T11:23:05.786 INFO:teuthology.orchestra.run.gibba002.stdout: "stripe_width": 0, 2021-02-10T11:23:05.786 INFO:teuthology.orchestra.run.gibba002.stdout: "expected_num_objects": 0, 2021-02-10T11:23:05.786 INFO:teuthology.orchestra.run.gibba002.stdout: "fast_read": false, 2021-02-10T11:23:05.786 INFO:teuthology.orchestra.run.gibba002.stdout: "options": {}, 2021-02-10T11:23:05.786 INFO:teuthology.orchestra.run.gibba002.stdout: "application_metadata": { 2021-02-10T11:23:05.786 INFO:teuthology.orchestra.run.gibba002.stdout: "rbd": {} 2021-02-10T11:23:05.786 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:23:05.786 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:05.786 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:05.787 INFO:teuthology.orchestra.run.gibba002.stdout: "pool": 6, 2021-02-10T11:23:05.787 INFO:teuthology.orchestra.run.gibba002.stdout: "pool_name": "cephfs_metadata", 2021-02-10T11:23:05.787 INFO:teuthology.orchestra.run.gibba002.stdout: "create_time": "2021-02-10 11:22:23.578113", 2021-02-10T11:23:05.787 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 1, 2021-02-10T11:23:05.787 INFO:teuthology.orchestra.run.gibba002.stdout: "flags_names": "hashpspool", 2021-02-10T11:23:05.787 INFO:teuthology.orchestra.run.gibba002.stdout: "type": 1, 2021-02-10T11:23:05.787 INFO:teuthology.orchestra.run.gibba002.stdout: "size": 2, 2021-02-10T11:23:05.787 INFO:teuthology.orchestra.run.gibba002.stdout: "min_size": 1, 2021-02-10T11:23:05.787 INFO:teuthology.orchestra.run.gibba002.stdout: "crush_rule": 0, 2021-02-10T11:23:05.787 INFO:teuthology.orchestra.run.gibba002.stdout: "object_hash": 2, 2021-02-10T11:23:05.787 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:23:05.787 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num": 8, 2021-02-10T11:23:05.787 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_placement_num": 8, 2021-02-10T11:23:05.788 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_placement_num_target": 8, 2021-02-10T11:23:05.788 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num_target": 8, 2021-02-10T11:23:05.788 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num_pending": 8, 2021-02-10T11:23:05.788 INFO:teuthology.orchestra.run.gibba002.stdout: "last_pg_merge_meta": { 2021-02-10T11:23:05.788 INFO:teuthology.orchestra.run.gibba002.stdout: "source_pgid": "0.0", 2021-02-10T11:23:05.788 INFO:teuthology.orchestra.run.gibba002.stdout: "ready_epoch": 0, 2021-02-10T11:23:05.788 INFO:teuthology.orchestra.run.gibba002.stdout: "last_epoch_started": 0, 2021-02-10T11:23:05.788 INFO:teuthology.orchestra.run.gibba002.stdout: "last_epoch_clean": 0, 2021-02-10T11:23:05.788 INFO:teuthology.orchestra.run.gibba002.stdout: "source_version": "0'0", 2021-02-10T11:23:05.788 INFO:teuthology.orchestra.run.gibba002.stdout: "target_version": "0'0" 2021-02-10T11:23:05.788 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:05.788 INFO:teuthology.orchestra.run.gibba002.stdout: "last_change": "38", 2021-02-10T11:23:05.788 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend": "0", 2021-02-10T11:23:05.788 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:23:05.789 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:23:05.789 INFO:teuthology.orchestra.run.gibba002.stdout: "auid": 0, 2021-02-10T11:23:05.789 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:23:05.789 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_seq": 0, 2021-02-10T11:23:05.789 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_epoch": 0, 2021-02-10T11:23:05.789 INFO:teuthology.orchestra.run.gibba002.stdout: "pool_snaps": [], 2021-02-10T11:23:05.789 INFO:teuthology.orchestra.run.gibba002.stdout: "removed_snaps": "[]", 2021-02-10T11:23:05.789 INFO:teuthology.orchestra.run.gibba002.stdout: "quota_max_bytes": 0, 2021-02-10T11:23:05.789 INFO:teuthology.orchestra.run.gibba002.stdout: "quota_max_objects": 0, 2021-02-10T11:23:05.789 INFO:teuthology.orchestra.run.gibba002.stdout: "tiers": [], 2021-02-10T11:23:05.789 INFO:teuthology.orchestra.run.gibba002.stdout: "tier_of": -1, 2021-02-10T11:23:05.789 INFO:teuthology.orchestra.run.gibba002.stdout: "read_tier": -1, 2021-02-10T11:23:05.789 INFO:teuthology.orchestra.run.gibba002.stdout: "write_tier": -1, 2021-02-10T11:23:05.790 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_mode": "none", 2021-02-10T11:23:05.790 INFO:teuthology.orchestra.run.gibba002.stdout: "target_max_bytes": 0, 2021-02-10T11:23:05.790 INFO:teuthology.orchestra.run.gibba002.stdout: "target_max_objects": 0, 2021-02-10T11:23:05.790 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:23:05.790 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:23:05.790 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:23:05.790 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_min_flush_age": 0, 2021-02-10T11:23:05.790 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_min_evict_age": 0, 2021-02-10T11:23:05.790 INFO:teuthology.orchestra.run.gibba002.stdout: "erasure_code_profile": "", 2021-02-10T11:23:05.790 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_params": { 2021-02-10T11:23:05.790 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "none" 2021-02-10T11:23:05.790 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:05.790 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_period": 0, 2021-02-10T11:23:05.790 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_count": 0, 2021-02-10T11:23:05.791 INFO:teuthology.orchestra.run.gibba002.stdout: "use_gmt_hitset": true, 2021-02-10T11:23:05.791 INFO:teuthology.orchestra.run.gibba002.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:23:05.791 INFO:teuthology.orchestra.run.gibba002.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:23:05.791 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:23:05.791 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:23:05.791 INFO:teuthology.orchestra.run.gibba002.stdout: "grade_table": [], 2021-02-10T11:23:05.791 INFO:teuthology.orchestra.run.gibba002.stdout: "stripe_width": 0, 2021-02-10T11:23:05.791 INFO:teuthology.orchestra.run.gibba002.stdout: "expected_num_objects": 0, 2021-02-10T11:23:05.791 INFO:teuthology.orchestra.run.gibba002.stdout: "fast_read": false, 2021-02-10T11:23:05.791 INFO:teuthology.orchestra.run.gibba002.stdout: "options": { 2021-02-10T11:23:05.791 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_autoscale_bias": 4, 2021-02-10T11:23:05.791 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num_min": 16, 2021-02-10T11:23:05.791 INFO:teuthology.orchestra.run.gibba002.stdout: "recovery_priority": 5 2021-02-10T11:23:05.791 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:05.792 INFO:teuthology.orchestra.run.gibba002.stdout: "application_metadata": { 2021-02-10T11:23:05.792 INFO:teuthology.orchestra.run.gibba002.stdout: "cephfs": { 2021-02-10T11:23:05.792 INFO:teuthology.orchestra.run.gibba002.stdout: "metadata": "cephfs" 2021-02-10T11:23:05.792 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:23:05.792 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:23:05.792 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:05.792 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:05.792 INFO:teuthology.orchestra.run.gibba002.stdout: "pool": 7, 2021-02-10T11:23:05.792 INFO:teuthology.orchestra.run.gibba002.stdout: "pool_name": "cephfs_data", 2021-02-10T11:23:05.792 INFO:teuthology.orchestra.run.gibba002.stdout: "create_time": "2021-02-10 11:22:25.111303", 2021-02-10T11:23:05.792 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 1, 2021-02-10T11:23:05.792 INFO:teuthology.orchestra.run.gibba002.stdout: "flags_names": "hashpspool", 2021-02-10T11:23:05.792 INFO:teuthology.orchestra.run.gibba002.stdout: "type": 1, 2021-02-10T11:23:05.792 INFO:teuthology.orchestra.run.gibba002.stdout: "size": 2, 2021-02-10T11:23:05.793 INFO:teuthology.orchestra.run.gibba002.stdout: "min_size": 1, 2021-02-10T11:23:05.793 INFO:teuthology.orchestra.run.gibba002.stdout: "crush_rule": 0, 2021-02-10T11:23:05.793 INFO:teuthology.orchestra.run.gibba002.stdout: "object_hash": 2, 2021-02-10T11:23:05.793 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:23:05.793 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num": 8, 2021-02-10T11:23:05.793 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_placement_num": 8, 2021-02-10T11:23:05.793 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_placement_num_target": 8, 2021-02-10T11:23:05.793 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num_target": 8, 2021-02-10T11:23:05.793 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num_pending": 8, 2021-02-10T11:23:05.793 INFO:teuthology.orchestra.run.gibba002.stdout: "last_pg_merge_meta": { 2021-02-10T11:23:05.793 INFO:teuthology.orchestra.run.gibba002.stdout: "source_pgid": "0.0", 2021-02-10T11:23:05.793 INFO:teuthology.orchestra.run.gibba002.stdout: "ready_epoch": 0, 2021-02-10T11:23:05.793 INFO:teuthology.orchestra.run.gibba002.stdout: "last_epoch_started": 0, 2021-02-10T11:23:05.794 INFO:teuthology.orchestra.run.gibba002.stdout: "last_epoch_clean": 0, 2021-02-10T11:23:05.794 INFO:teuthology.orchestra.run.gibba002.stdout: "source_version": "0'0", 2021-02-10T11:23:05.794 INFO:teuthology.orchestra.run.gibba002.stdout: "target_version": "0'0" 2021-02-10T11:23:05.794 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:05.794 INFO:teuthology.orchestra.run.gibba002.stdout: "last_change": "39", 2021-02-10T11:23:05.794 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend": "0", 2021-02-10T11:23:05.794 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:23:05.794 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:23:05.794 INFO:teuthology.orchestra.run.gibba002.stdout: "auid": 0, 2021-02-10T11:23:05.794 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:23:05.794 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_seq": 0, 2021-02-10T11:23:05.794 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_epoch": 0, 2021-02-10T11:23:05.795 INFO:teuthology.orchestra.run.gibba002.stdout: "pool_snaps": [], 2021-02-10T11:23:05.795 INFO:teuthology.orchestra.run.gibba002.stdout: "removed_snaps": "[]", 2021-02-10T11:23:05.795 INFO:teuthology.orchestra.run.gibba002.stdout: "quota_max_bytes": 0, 2021-02-10T11:23:05.795 INFO:teuthology.orchestra.run.gibba002.stdout: "quota_max_objects": 0, 2021-02-10T11:23:05.795 INFO:teuthology.orchestra.run.gibba002.stdout: "tiers": [], 2021-02-10T11:23:05.795 INFO:teuthology.orchestra.run.gibba002.stdout: "tier_of": -1, 2021-02-10T11:23:05.795 INFO:teuthology.orchestra.run.gibba002.stdout: "read_tier": -1, 2021-02-10T11:23:05.795 INFO:teuthology.orchestra.run.gibba002.stdout: "write_tier": -1, 2021-02-10T11:23:05.795 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_mode": "none", 2021-02-10T11:23:05.795 INFO:teuthology.orchestra.run.gibba002.stdout: "target_max_bytes": 0, 2021-02-10T11:23:05.795 INFO:teuthology.orchestra.run.gibba002.stdout: "target_max_objects": 0, 2021-02-10T11:23:05.795 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:23:05.795 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:23:05.795 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:23:05.796 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_min_flush_age": 0, 2021-02-10T11:23:05.796 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_min_evict_age": 0, 2021-02-10T11:23:05.796 INFO:teuthology.orchestra.run.gibba002.stdout: "erasure_code_profile": "", 2021-02-10T11:23:05.796 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_params": { 2021-02-10T11:23:05.796 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "none" 2021-02-10T11:23:05.796 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:05.796 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_period": 0, 2021-02-10T11:23:05.796 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_count": 0, 2021-02-10T11:23:05.796 INFO:teuthology.orchestra.run.gibba002.stdout: "use_gmt_hitset": true, 2021-02-10T11:23:05.796 INFO:teuthology.orchestra.run.gibba002.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:23:05.796 INFO:teuthology.orchestra.run.gibba002.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:23:05.796 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:23:05.796 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:23:05.797 INFO:teuthology.orchestra.run.gibba002.stdout: "grade_table": [], 2021-02-10T11:23:05.797 INFO:teuthology.orchestra.run.gibba002.stdout: "stripe_width": 0, 2021-02-10T11:23:05.797 INFO:teuthology.orchestra.run.gibba002.stdout: "expected_num_objects": 0, 2021-02-10T11:23:05.797 INFO:teuthology.orchestra.run.gibba002.stdout: "fast_read": false, 2021-02-10T11:23:05.797 INFO:teuthology.orchestra.run.gibba002.stdout: "options": {}, 2021-02-10T11:23:05.797 INFO:teuthology.orchestra.run.gibba002.stdout: "application_metadata": { 2021-02-10T11:23:05.797 INFO:teuthology.orchestra.run.gibba002.stdout: "cephfs": { 2021-02-10T11:23:05.797 INFO:teuthology.orchestra.run.gibba002.stdout: "data": "cephfs" 2021-02-10T11:23:05.797 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:23:05.797 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:23:05.797 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:23:05.797 INFO:teuthology.orchestra.run.gibba002.stdout: ], 2021-02-10T11:23:05.797 INFO:teuthology.orchestra.run.gibba002.stdout: "osds": [ 2021-02-10T11:23:05.797 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:05.798 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 0, 2021-02-10T11:23:05.798 INFO:teuthology.orchestra.run.gibba002.stdout: "uuid": "be691b4d-a039-408a-af80-9d53f02b6f1f", 2021-02-10T11:23:05.798 INFO:teuthology.orchestra.run.gibba002.stdout: "up": 1, 2021-02-10T11:23:05.798 INFO:teuthology.orchestra.run.gibba002.stdout: "in": 1, 2021-02-10T11:23:05.798 INFO:teuthology.orchestra.run.gibba002.stdout: "weight": 1, 2021-02-10T11:23:05.798 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_affinity": 1, 2021-02-10T11:23:05.798 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_begin": 0, 2021-02-10T11:23:05.798 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_end": 0, 2021-02-10T11:23:05.798 INFO:teuthology.orchestra.run.gibba002.stdout: "up_from": 13, 2021-02-10T11:23:05.798 INFO:teuthology.orchestra.run.gibba002.stdout: "up_thru": 37, 2021-02-10T11:23:05.798 INFO:teuthology.orchestra.run.gibba002.stdout: "down_at": 0, 2021-02-10T11:23:05.798 INFO:teuthology.orchestra.run.gibba002.stdout: "lost_at": 0, 2021-02-10T11:23:05.798 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addrs": { 2021-02-10T11:23:05.799 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:23:05.799 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:05.799 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:23:05.799 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6818", 2021-02-10T11:23:05.799 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36088 2021-02-10T11:23:05.799 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:05.799 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:05.799 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:23:05.799 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6820", 2021-02-10T11:23:05.799 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36088 2021-02-10T11:23:05.799 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:23:05.799 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:23:05.799 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:05.799 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addrs": { 2021-02-10T11:23:05.800 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:23:05.800 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:05.800 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:23:05.800 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6822", 2021-02-10T11:23:05.800 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36088 2021-02-10T11:23:05.800 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:05.800 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:05.800 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:23:05.800 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6824", 2021-02-10T11:23:05.800 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36088 2021-02-10T11:23:05.800 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:23:05.800 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:23:05.800 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:05.801 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addrs": { 2021-02-10T11:23:05.801 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:23:05.801 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:05.801 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:23:05.801 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6830", 2021-02-10T11:23:05.801 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36088 2021-02-10T11:23:05.801 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:05.801 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:05.801 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:23:05.801 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6832", 2021-02-10T11:23:05.801 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36088 2021-02-10T11:23:05.801 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:23:05.801 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:23:05.801 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:05.802 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addrs": { 2021-02-10T11:23:05.802 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:23:05.802 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:05.802 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:23:05.802 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6826", 2021-02-10T11:23:05.802 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36088 2021-02-10T11:23:05.802 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:05.802 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:05.802 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:23:05.802 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6828", 2021-02-10T11:23:05.802 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36088 2021-02-10T11:23:05.802 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:23:05.802 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:23:05.802 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:05.803 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addr": "172.21.2.102:6820/36088", 2021-02-10T11:23:05.803 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addr": "172.21.2.102:6824/36088", 2021-02-10T11:23:05.803 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addr": "172.21.2.102:6832/36088", 2021-02-10T11:23:05.803 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addr": "172.21.2.102:6828/36088", 2021-02-10T11:23:05.803 INFO:teuthology.orchestra.run.gibba002.stdout: "state": [ 2021-02-10T11:23:05.803 INFO:teuthology.orchestra.run.gibba002.stdout: "exists", 2021-02-10T11:23:05.803 INFO:teuthology.orchestra.run.gibba002.stdout: "up" 2021-02-10T11:23:05.803 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:23:05.803 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:05.803 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:05.803 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 1, 2021-02-10T11:23:05.803 INFO:teuthology.orchestra.run.gibba002.stdout: "uuid": "64e3ccc8-9c69-4d7c-865e-af8e83822635", 2021-02-10T11:23:05.803 INFO:teuthology.orchestra.run.gibba002.stdout: "up": 1, 2021-02-10T11:23:05.804 INFO:teuthology.orchestra.run.gibba002.stdout: "in": 1, 2021-02-10T11:23:05.804 INFO:teuthology.orchestra.run.gibba002.stdout: "weight": 1, 2021-02-10T11:23:05.804 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_affinity": 1, 2021-02-10T11:23:05.804 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_begin": 0, 2021-02-10T11:23:05.804 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_end": 0, 2021-02-10T11:23:05.804 INFO:teuthology.orchestra.run.gibba002.stdout: "up_from": 13, 2021-02-10T11:23:05.804 INFO:teuthology.orchestra.run.gibba002.stdout: "up_thru": 36, 2021-02-10T11:23:05.804 INFO:teuthology.orchestra.run.gibba002.stdout: "down_at": 0, 2021-02-10T11:23:05.804 INFO:teuthology.orchestra.run.gibba002.stdout: "lost_at": 0, 2021-02-10T11:23:05.804 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addrs": { 2021-02-10T11:23:05.804 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:23:05.804 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:05.804 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:23:05.804 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6819", 2021-02-10T11:23:05.805 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36087 2021-02-10T11:23:05.805 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:05.805 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:05.805 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:23:05.805 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6821", 2021-02-10T11:23:05.805 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36087 2021-02-10T11:23:05.805 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:23:05.805 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:23:05.805 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:05.805 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addrs": { 2021-02-10T11:23:05.805 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:23:05.805 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:05.805 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:23:05.805 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6823", 2021-02-10T11:23:05.806 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36087 2021-02-10T11:23:05.806 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:05.806 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:05.806 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:23:05.806 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6825", 2021-02-10T11:23:05.806 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36087 2021-02-10T11:23:05.806 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:23:05.806 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:23:05.806 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:05.806 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addrs": { 2021-02-10T11:23:05.806 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:23:05.806 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:05.806 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:23:05.806 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6831", 2021-02-10T11:23:05.807 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36087 2021-02-10T11:23:05.807 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:05.807 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:05.807 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:23:05.807 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6833", 2021-02-10T11:23:05.807 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36087 2021-02-10T11:23:05.807 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:23:05.807 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:23:05.807 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:05.807 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addrs": { 2021-02-10T11:23:05.807 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:23:05.807 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:05.807 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:23:05.807 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6827", 2021-02-10T11:23:05.808 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36087 2021-02-10T11:23:05.808 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:05.808 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:05.808 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:23:05.808 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6829", 2021-02-10T11:23:05.808 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36087 2021-02-10T11:23:05.808 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:23:05.808 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:23:05.808 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:05.808 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addr": "172.21.2.102:6821/36087", 2021-02-10T11:23:05.808 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addr": "172.21.2.102:6825/36087", 2021-02-10T11:23:05.808 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addr": "172.21.2.102:6833/36087", 2021-02-10T11:23:05.808 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addr": "172.21.2.102:6829/36087", 2021-02-10T11:23:05.808 INFO:teuthology.orchestra.run.gibba002.stdout: "state": [ 2021-02-10T11:23:05.809 INFO:teuthology.orchestra.run.gibba002.stdout: "exists", 2021-02-10T11:23:05.809 INFO:teuthology.orchestra.run.gibba002.stdout: "up" 2021-02-10T11:23:05.809 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:23:05.809 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:05.809 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:05.809 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 2, 2021-02-10T11:23:05.809 INFO:teuthology.orchestra.run.gibba002.stdout: "uuid": "153700af-747f-481f-bdf4-d6072d3a842f", 2021-02-10T11:23:05.809 INFO:teuthology.orchestra.run.gibba002.stdout: "up": 1, 2021-02-10T11:23:05.809 INFO:teuthology.orchestra.run.gibba002.stdout: "in": 1, 2021-02-10T11:23:05.809 INFO:teuthology.orchestra.run.gibba002.stdout: "weight": 1, 2021-02-10T11:23:05.809 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_affinity": 1, 2021-02-10T11:23:05.809 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_begin": 0, 2021-02-10T11:23:05.809 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_end": 0, 2021-02-10T11:23:05.809 INFO:teuthology.orchestra.run.gibba002.stdout: "up_from": 13, 2021-02-10T11:23:05.810 INFO:teuthology.orchestra.run.gibba002.stdout: "up_thru": 28, 2021-02-10T11:23:05.810 INFO:teuthology.orchestra.run.gibba002.stdout: "down_at": 0, 2021-02-10T11:23:05.810 INFO:teuthology.orchestra.run.gibba002.stdout: "lost_at": 0, 2021-02-10T11:23:05.810 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addrs": { 2021-02-10T11:23:05.810 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:23:05.810 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:05.811 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:23:05.811 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6810", 2021-02-10T11:23:05.811 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36089 2021-02-10T11:23:05.811 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:05.811 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:05.811 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:23:05.811 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6811", 2021-02-10T11:23:05.811 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36089 2021-02-10T11:23:05.811 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:23:05.812 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:23:05.812 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:05.812 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addrs": { 2021-02-10T11:23:05.812 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:23:05.812 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:05.812 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:23:05.812 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6812", 2021-02-10T11:23:05.812 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36089 2021-02-10T11:23:05.812 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:05.812 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:05.813 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:23:05.813 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6813", 2021-02-10T11:23:05.813 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36089 2021-02-10T11:23:05.813 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:23:05.813 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:23:05.813 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:05.813 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addrs": { 2021-02-10T11:23:05.813 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:23:05.813 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:05.813 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:23:05.814 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6816", 2021-02-10T11:23:05.814 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36089 2021-02-10T11:23:05.814 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:05.814 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:05.814 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:23:05.814 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6817", 2021-02-10T11:23:05.814 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36089 2021-02-10T11:23:05.814 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:23:05.814 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:23:05.814 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:05.814 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addrs": { 2021-02-10T11:23:05.815 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:23:05.815 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:05.815 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:23:05.815 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6814", 2021-02-10T11:23:05.815 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36089 2021-02-10T11:23:05.815 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:05.815 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:05.815 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:23:05.815 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6815", 2021-02-10T11:23:05.815 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36089 2021-02-10T11:23:05.815 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:23:05.815 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:23:05.816 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:05.816 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addr": "172.21.2.102:6811/36089", 2021-02-10T11:23:05.816 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addr": "172.21.2.102:6813/36089", 2021-02-10T11:23:05.816 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addr": "172.21.2.102:6817/36089", 2021-02-10T11:23:05.816 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addr": "172.21.2.102:6815/36089", 2021-02-10T11:23:05.816 INFO:teuthology.orchestra.run.gibba002.stdout: "state": [ 2021-02-10T11:23:05.816 INFO:teuthology.orchestra.run.gibba002.stdout: "exists", 2021-02-10T11:23:05.816 INFO:teuthology.orchestra.run.gibba002.stdout: "up" 2021-02-10T11:23:05.816 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:23:05.816 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:05.816 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:05.817 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 3, 2021-02-10T11:23:05.817 INFO:teuthology.orchestra.run.gibba002.stdout: "uuid": "cbc678f4-83a9-45e6-8830-2f7c232b1ea7", 2021-02-10T11:23:05.817 INFO:teuthology.orchestra.run.gibba002.stdout: "up": 1, 2021-02-10T11:23:05.817 INFO:teuthology.orchestra.run.gibba002.stdout: "in": 1, 2021-02-10T11:23:05.817 INFO:teuthology.orchestra.run.gibba002.stdout: "weight": 1, 2021-02-10T11:23:05.817 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_affinity": 1, 2021-02-10T11:23:05.817 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_begin": 0, 2021-02-10T11:23:05.817 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_end": 0, 2021-02-10T11:23:05.817 INFO:teuthology.orchestra.run.gibba002.stdout: "up_from": 13, 2021-02-10T11:23:05.817 INFO:teuthology.orchestra.run.gibba002.stdout: "up_thru": 37, 2021-02-10T11:23:05.817 INFO:teuthology.orchestra.run.gibba002.stdout: "down_at": 0, 2021-02-10T11:23:05.818 INFO:teuthology.orchestra.run.gibba002.stdout: "lost_at": 0, 2021-02-10T11:23:05.818 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addrs": { 2021-02-10T11:23:05.818 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:23:05.818 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:05.818 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:23:05.818 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6802", 2021-02-10T11:23:05.818 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36090 2021-02-10T11:23:05.818 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:05.818 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:05.818 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:23:05.819 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6803", 2021-02-10T11:23:05.819 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36090 2021-02-10T11:23:05.819 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:23:05.819 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:23:05.819 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:05.820 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addrs": { 2021-02-10T11:23:05.820 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:23:05.820 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:05.820 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:23:05.820 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6804", 2021-02-10T11:23:05.820 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36090 2021-02-10T11:23:05.820 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:05.820 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:05.820 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:23:05.820 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6805", 2021-02-10T11:23:05.820 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36090 2021-02-10T11:23:05.821 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:23:05.821 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:23:05.821 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:05.821 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addrs": { 2021-02-10T11:23:05.821 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:23:05.821 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:05.821 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:23:05.821 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6808", 2021-02-10T11:23:05.821 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36090 2021-02-10T11:23:05.821 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:05.821 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:05.821 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:23:05.822 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6809", 2021-02-10T11:23:05.822 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36090 2021-02-10T11:23:05.822 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:23:05.822 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:23:05.822 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:05.822 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addrs": { 2021-02-10T11:23:05.822 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:23:05.822 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:05.822 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:23:05.822 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6806", 2021-02-10T11:23:05.823 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36090 2021-02-10T11:23:05.823 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:05.823 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:05.823 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:23:05.823 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6807", 2021-02-10T11:23:05.823 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36090 2021-02-10T11:23:05.823 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:23:05.823 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:23:05.823 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:05.823 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addr": "172.21.2.102:6803/36090", 2021-02-10T11:23:05.823 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addr": "172.21.2.102:6805/36090", 2021-02-10T11:23:05.823 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addr": "172.21.2.102:6809/36090", 2021-02-10T11:23:05.824 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addr": "172.21.2.102:6807/36090", 2021-02-10T11:23:05.824 INFO:teuthology.orchestra.run.gibba002.stdout: "state": [ 2021-02-10T11:23:05.824 INFO:teuthology.orchestra.run.gibba002.stdout: "exists", 2021-02-10T11:23:05.824 INFO:teuthology.orchestra.run.gibba002.stdout: "up" 2021-02-10T11:23:05.824 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:23:05.824 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:05.824 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:05.824 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 4, 2021-02-10T11:23:05.824 INFO:teuthology.orchestra.run.gibba002.stdout: "uuid": "c01e4008-b258-4f10-afd2-9e7f37214351", 2021-02-10T11:23:05.824 INFO:teuthology.orchestra.run.gibba002.stdout: "up": 1, 2021-02-10T11:23:05.824 INFO:teuthology.orchestra.run.gibba002.stdout: "in": 1, 2021-02-10T11:23:05.824 INFO:teuthology.orchestra.run.gibba002.stdout: "weight": 1, 2021-02-10T11:23:05.825 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_affinity": 1, 2021-02-10T11:23:05.825 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_begin": 0, 2021-02-10T11:23:05.825 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_end": 0, 2021-02-10T11:23:05.825 INFO:teuthology.orchestra.run.gibba002.stdout: "up_from": 13, 2021-02-10T11:23:05.825 INFO:teuthology.orchestra.run.gibba002.stdout: "up_thru": 37, 2021-02-10T11:23:05.825 INFO:teuthology.orchestra.run.gibba002.stdout: "down_at": 0, 2021-02-10T11:23:05.825 INFO:teuthology.orchestra.run.gibba002.stdout: "lost_at": 0, 2021-02-10T11:23:05.825 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addrs": { 2021-02-10T11:23:05.825 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:23:05.825 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:05.825 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:23:05.825 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6808", 2021-02-10T11:23:05.826 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36368 2021-02-10T11:23:05.826 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:05.826 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:05.826 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:23:05.826 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6811", 2021-02-10T11:23:05.826 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36368 2021-02-10T11:23:05.826 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:23:05.826 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:23:05.826 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:05.826 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addrs": { 2021-02-10T11:23:05.826 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:23:05.826 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:05.827 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:23:05.827 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6814", 2021-02-10T11:23:05.827 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36368 2021-02-10T11:23:05.827 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:05.827 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:05.827 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:23:05.827 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6818", 2021-02-10T11:23:05.827 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36368 2021-02-10T11:23:05.827 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:23:05.827 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:23:05.828 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:05.828 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addrs": { 2021-02-10T11:23:05.828 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:23:05.828 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:05.828 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:23:05.829 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6826", 2021-02-10T11:23:05.829 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36368 2021-02-10T11:23:05.829 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:05.829 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:05.829 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:23:05.829 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6828", 2021-02-10T11:23:05.829 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36368 2021-02-10T11:23:05.829 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:23:05.829 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:23:05.829 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:05.829 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addrs": { 2021-02-10T11:23:05.830 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:23:05.830 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:05.830 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:23:05.830 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6820", 2021-02-10T11:23:05.830 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36368 2021-02-10T11:23:05.830 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:05.830 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:05.830 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:23:05.830 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6824", 2021-02-10T11:23:05.830 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36368 2021-02-10T11:23:05.830 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:23:05.830 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:23:05.831 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:05.831 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addr": "172.21.2.113:6811/36368", 2021-02-10T11:23:05.831 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addr": "172.21.2.113:6818/36368", 2021-02-10T11:23:05.831 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addr": "172.21.2.113:6828/36368", 2021-02-10T11:23:05.831 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addr": "172.21.2.113:6824/36368", 2021-02-10T11:23:05.831 INFO:teuthology.orchestra.run.gibba002.stdout: "state": [ 2021-02-10T11:23:05.831 INFO:teuthology.orchestra.run.gibba002.stdout: "exists", 2021-02-10T11:23:05.831 INFO:teuthology.orchestra.run.gibba002.stdout: "up" 2021-02-10T11:23:05.831 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:23:05.831 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:05.831 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:05.831 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 5, 2021-02-10T11:23:05.832 INFO:teuthology.orchestra.run.gibba002.stdout: "uuid": "551f6007-daf8-4a98-8ee0-c2b16bcccb5e", 2021-02-10T11:23:05.832 INFO:teuthology.orchestra.run.gibba002.stdout: "up": 1, 2021-02-10T11:23:05.832 INFO:teuthology.orchestra.run.gibba002.stdout: "in": 1, 2021-02-10T11:23:05.832 INFO:teuthology.orchestra.run.gibba002.stdout: "weight": 1, 2021-02-10T11:23:05.832 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_affinity": 1, 2021-02-10T11:23:05.832 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_begin": 0, 2021-02-10T11:23:05.832 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_end": 0, 2021-02-10T11:23:05.832 INFO:teuthology.orchestra.run.gibba002.stdout: "up_from": 13, 2021-02-10T11:23:05.832 INFO:teuthology.orchestra.run.gibba002.stdout: "up_thru": 37, 2021-02-10T11:23:05.832 INFO:teuthology.orchestra.run.gibba002.stdout: "down_at": 0, 2021-02-10T11:23:05.832 INFO:teuthology.orchestra.run.gibba002.stdout: "lost_at": 0, 2021-02-10T11:23:05.832 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addrs": { 2021-02-10T11:23:05.832 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:23:05.833 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:05.833 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:23:05.833 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6802", 2021-02-10T11:23:05.833 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36366 2021-02-10T11:23:05.833 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:05.833 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:05.833 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:23:05.833 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6805", 2021-02-10T11:23:05.833 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36366 2021-02-10T11:23:05.833 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:23:05.833 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:23:05.833 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:05.833 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addrs": { 2021-02-10T11:23:05.833 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:23:05.834 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:05.834 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:23:05.834 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6807", 2021-02-10T11:23:05.834 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36366 2021-02-10T11:23:05.834 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:05.834 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:05.834 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:23:05.834 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6810", 2021-02-10T11:23:05.834 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36366 2021-02-10T11:23:05.834 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:23:05.835 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:23:05.835 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:05.835 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addrs": { 2021-02-10T11:23:05.835 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:23:05.835 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:05.835 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:23:05.835 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6819", 2021-02-10T11:23:05.835 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36366 2021-02-10T11:23:05.835 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:05.835 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:05.835 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:23:05.835 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6822", 2021-02-10T11:23:05.835 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36366 2021-02-10T11:23:05.836 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:23:05.836 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:23:05.836 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:05.836 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addrs": { 2021-02-10T11:23:05.836 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:23:05.836 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:05.836 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:23:05.836 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6812", 2021-02-10T11:23:05.836 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36366 2021-02-10T11:23:05.836 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:05.836 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:05.836 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:23:05.837 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6815", 2021-02-10T11:23:05.837 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36366 2021-02-10T11:23:05.837 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:23:05.837 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:23:05.837 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:05.837 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addr": "172.21.2.113:6805/36366", 2021-02-10T11:23:05.837 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addr": "172.21.2.113:6810/36366", 2021-02-10T11:23:05.837 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addr": "172.21.2.113:6822/36366", 2021-02-10T11:23:05.837 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addr": "172.21.2.113:6815/36366", 2021-02-10T11:23:05.837 INFO:teuthology.orchestra.run.gibba002.stdout: "state": [ 2021-02-10T11:23:05.837 INFO:teuthology.orchestra.run.gibba002.stdout: "exists", 2021-02-10T11:23:05.837 INFO:teuthology.orchestra.run.gibba002.stdout: "up" 2021-02-10T11:23:05.838 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:23:05.838 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:05.838 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:05.838 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 6, 2021-02-10T11:23:05.838 INFO:teuthology.orchestra.run.gibba002.stdout: "uuid": "bf67c040-0e50-48be-b102-c3943bc9ebef", 2021-02-10T11:23:05.838 INFO:teuthology.orchestra.run.gibba002.stdout: "up": 1, 2021-02-10T11:23:05.838 INFO:teuthology.orchestra.run.gibba002.stdout: "in": 1, 2021-02-10T11:23:05.838 INFO:teuthology.orchestra.run.gibba002.stdout: "weight": 1, 2021-02-10T11:23:05.838 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_affinity": 1, 2021-02-10T11:23:05.838 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_begin": 0, 2021-02-10T11:23:05.839 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_end": 0, 2021-02-10T11:23:05.839 INFO:teuthology.orchestra.run.gibba002.stdout: "up_from": 13, 2021-02-10T11:23:05.839 INFO:teuthology.orchestra.run.gibba002.stdout: "up_thru": 37, 2021-02-10T11:23:05.839 INFO:teuthology.orchestra.run.gibba002.stdout: "down_at": 0, 2021-02-10T11:23:05.839 INFO:teuthology.orchestra.run.gibba002.stdout: "lost_at": 0, 2021-02-10T11:23:05.839 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addrs": { 2021-02-10T11:23:05.839 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:23:05.839 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:05.839 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:23:05.839 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6800", 2021-02-10T11:23:05.840 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36367 2021-02-10T11:23:05.840 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:05.840 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:05.840 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:23:05.840 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6801", 2021-02-10T11:23:05.840 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36367 2021-02-10T11:23:05.840 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:23:05.840 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:23:05.840 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:05.841 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addrs": { 2021-02-10T11:23:05.841 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:23:05.841 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:05.841 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:23:05.841 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6803", 2021-02-10T11:23:05.841 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36367 2021-02-10T11:23:05.841 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:05.841 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:05.841 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:23:05.841 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6804", 2021-02-10T11:23:05.841 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36367 2021-02-10T11:23:05.842 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:23:05.842 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:23:05.842 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:05.842 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addrs": { 2021-02-10T11:23:05.842 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:23:05.842 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:05.842 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:23:05.842 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6813", 2021-02-10T11:23:05.842 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36367 2021-02-10T11:23:05.842 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:05.842 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:05.843 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:23:05.843 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6816", 2021-02-10T11:23:05.843 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36367 2021-02-10T11:23:05.843 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:23:05.843 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:23:05.843 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:05.843 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addrs": { 2021-02-10T11:23:05.843 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:23:05.843 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:05.843 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:23:05.843 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6806", 2021-02-10T11:23:05.844 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36367 2021-02-10T11:23:05.844 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:05.844 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:05.844 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:23:05.844 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6809", 2021-02-10T11:23:05.844 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36367 2021-02-10T11:23:05.844 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:23:05.844 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:23:05.844 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:05.844 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addr": "172.21.2.113:6801/36367", 2021-02-10T11:23:05.844 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addr": "172.21.2.113:6804/36367", 2021-02-10T11:23:05.845 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addr": "172.21.2.113:6816/36367", 2021-02-10T11:23:05.845 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addr": "172.21.2.113:6809/36367", 2021-02-10T11:23:05.845 INFO:teuthology.orchestra.run.gibba002.stdout: "state": [ 2021-02-10T11:23:05.845 INFO:teuthology.orchestra.run.gibba002.stdout: "exists", 2021-02-10T11:23:05.845 INFO:teuthology.orchestra.run.gibba002.stdout: "up" 2021-02-10T11:23:05.845 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:23:05.845 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:05.845 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:05.845 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 7, 2021-02-10T11:23:05.845 INFO:teuthology.orchestra.run.gibba002.stdout: "uuid": "f5461f80-018d-428f-be89-2eb2bcbe7bb6", 2021-02-10T11:23:05.845 INFO:teuthology.orchestra.run.gibba002.stdout: "up": 1, 2021-02-10T11:23:05.845 INFO:teuthology.orchestra.run.gibba002.stdout: "in": 1, 2021-02-10T11:23:05.846 INFO:teuthology.orchestra.run.gibba002.stdout: "weight": 1, 2021-02-10T11:23:05.846 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_affinity": 1, 2021-02-10T11:23:05.846 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_begin": 0, 2021-02-10T11:23:05.846 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_end": 0, 2021-02-10T11:23:05.846 INFO:teuthology.orchestra.run.gibba002.stdout: "up_from": 13, 2021-02-10T11:23:05.846 INFO:teuthology.orchestra.run.gibba002.stdout: "up_thru": 37, 2021-02-10T11:23:05.846 INFO:teuthology.orchestra.run.gibba002.stdout: "down_at": 0, 2021-02-10T11:23:05.846 INFO:teuthology.orchestra.run.gibba002.stdout: "lost_at": 0, 2021-02-10T11:23:05.846 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addrs": { 2021-02-10T11:23:05.846 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:23:05.846 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:05.847 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:23:05.847 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6817", 2021-02-10T11:23:05.847 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36365 2021-02-10T11:23:05.847 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:05.847 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:05.847 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:23:05.847 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6821", 2021-02-10T11:23:05.847 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36365 2021-02-10T11:23:05.847 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:23:05.847 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:23:05.847 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:05.847 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addrs": { 2021-02-10T11:23:05.847 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:23:05.848 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:05.848 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:23:05.848 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6823", 2021-02-10T11:23:05.848 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36365 2021-02-10T11:23:05.848 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:05.848 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:05.848 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:23:05.848 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6825", 2021-02-10T11:23:05.848 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36365 2021-02-10T11:23:05.849 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:23:05.849 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:23:05.849 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:05.849 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addrs": { 2021-02-10T11:23:05.849 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:23:05.849 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:05.849 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:23:05.849 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6830", 2021-02-10T11:23:05.849 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36365 2021-02-10T11:23:05.849 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:05.849 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:05.849 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:23:05.849 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6831", 2021-02-10T11:23:05.850 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36365 2021-02-10T11:23:05.850 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:23:05.850 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:23:05.850 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:05.850 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addrs": { 2021-02-10T11:23:05.850 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:23:05.850 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:05.850 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:23:05.850 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6827", 2021-02-10T11:23:05.850 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36365 2021-02-10T11:23:05.850 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:05.850 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:05.851 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:23:05.851 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6829", 2021-02-10T11:23:05.851 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36365 2021-02-10T11:23:05.851 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:23:05.851 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:23:05.851 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:05.851 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addr": "172.21.2.113:6821/36365", 2021-02-10T11:23:05.851 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addr": "172.21.2.113:6825/36365", 2021-02-10T11:23:05.851 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addr": "172.21.2.113:6831/36365", 2021-02-10T11:23:05.851 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addr": "172.21.2.113:6829/36365", 2021-02-10T11:23:05.851 INFO:teuthology.orchestra.run.gibba002.stdout: "state": [ 2021-02-10T11:23:05.851 INFO:teuthology.orchestra.run.gibba002.stdout: "exists", 2021-02-10T11:23:05.852 INFO:teuthology.orchestra.run.gibba002.stdout: "up" 2021-02-10T11:23:05.852 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:23:05.852 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:23:05.852 INFO:teuthology.orchestra.run.gibba002.stdout: ], 2021-02-10T11:23:05.852 INFO:teuthology.orchestra.run.gibba002.stdout: "osd_xinfo": [ 2021-02-10T11:23:05.852 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:05.852 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 0, 2021-02-10T11:23:05.852 INFO:teuthology.orchestra.run.gibba002.stdout: "down_stamp": "0.000000", 2021-02-10T11:23:05.853 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_probability": 0, 2021-02-10T11:23:05.853 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_interval": 0, 2021-02-10T11:23:05.853 INFO:teuthology.orchestra.run.gibba002.stdout: "features": 4611087854035861503, 2021-02-10T11:23:05.853 INFO:teuthology.orchestra.run.gibba002.stdout: "old_weight": 0 2021-02-10T11:23:05.853 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:05.853 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:05.853 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 1, 2021-02-10T11:23:05.853 INFO:teuthology.orchestra.run.gibba002.stdout: "down_stamp": "0.000000", 2021-02-10T11:23:05.853 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_probability": 0, 2021-02-10T11:23:05.853 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_interval": 0, 2021-02-10T11:23:05.853 INFO:teuthology.orchestra.run.gibba002.stdout: "features": 4611087854035861503, 2021-02-10T11:23:05.853 INFO:teuthology.orchestra.run.gibba002.stdout: "old_weight": 0 2021-02-10T11:23:05.853 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:05.854 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:05.854 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 2, 2021-02-10T11:23:05.854 INFO:teuthology.orchestra.run.gibba002.stdout: "down_stamp": "0.000000", 2021-02-10T11:23:05.854 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_probability": 0, 2021-02-10T11:23:05.854 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_interval": 0, 2021-02-10T11:23:05.854 INFO:teuthology.orchestra.run.gibba002.stdout: "features": 4611087854035861503, 2021-02-10T11:23:05.854 INFO:teuthology.orchestra.run.gibba002.stdout: "old_weight": 0 2021-02-10T11:23:05.854 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:05.854 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:05.854 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 3, 2021-02-10T11:23:05.854 INFO:teuthology.orchestra.run.gibba002.stdout: "down_stamp": "0.000000", 2021-02-10T11:23:05.854 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_probability": 0, 2021-02-10T11:23:05.855 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_interval": 0, 2021-02-10T11:23:05.855 INFO:teuthology.orchestra.run.gibba002.stdout: "features": 4611087854035861503, 2021-02-10T11:23:05.855 INFO:teuthology.orchestra.run.gibba002.stdout: "old_weight": 0 2021-02-10T11:23:05.855 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:05.855 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:05.855 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 4, 2021-02-10T11:23:05.855 INFO:teuthology.orchestra.run.gibba002.stdout: "down_stamp": "0.000000", 2021-02-10T11:23:05.855 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_probability": 0, 2021-02-10T11:23:05.855 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_interval": 0, 2021-02-10T11:23:05.855 INFO:teuthology.orchestra.run.gibba002.stdout: "features": 4611087854035861503, 2021-02-10T11:23:05.855 INFO:teuthology.orchestra.run.gibba002.stdout: "old_weight": 0 2021-02-10T11:23:05.855 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:05.855 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:05.856 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 5, 2021-02-10T11:23:05.856 INFO:teuthology.orchestra.run.gibba002.stdout: "down_stamp": "0.000000", 2021-02-10T11:23:05.856 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_probability": 0, 2021-02-10T11:23:05.856 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_interval": 0, 2021-02-10T11:23:05.856 INFO:teuthology.orchestra.run.gibba002.stdout: "features": 4611087854035861503, 2021-02-10T11:23:05.856 INFO:teuthology.orchestra.run.gibba002.stdout: "old_weight": 0 2021-02-10T11:23:05.856 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:05.856 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:05.856 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 6, 2021-02-10T11:23:05.856 INFO:teuthology.orchestra.run.gibba002.stdout: "down_stamp": "0.000000", 2021-02-10T11:23:05.856 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_probability": 0, 2021-02-10T11:23:05.856 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_interval": 0, 2021-02-10T11:23:05.857 INFO:teuthology.orchestra.run.gibba002.stdout: "features": 4611087854035861503, 2021-02-10T11:23:05.857 INFO:teuthology.orchestra.run.gibba002.stdout: "old_weight": 0 2021-02-10T11:23:05.857 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:05.857 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:05.857 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 7, 2021-02-10T11:23:05.857 INFO:teuthology.orchestra.run.gibba002.stdout: "down_stamp": "0.000000", 2021-02-10T11:23:05.857 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_probability": 0, 2021-02-10T11:23:05.857 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_interval": 0, 2021-02-10T11:23:05.857 INFO:teuthology.orchestra.run.gibba002.stdout: "features": 4611087854035861503, 2021-02-10T11:23:05.857 INFO:teuthology.orchestra.run.gibba002.stdout: "old_weight": 0 2021-02-10T11:23:05.857 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:23:05.858 INFO:teuthology.orchestra.run.gibba002.stdout: ], 2021-02-10T11:23:05.858 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_upmap": [], 2021-02-10T11:23:05.858 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_upmap_items": [], 2021-02-10T11:23:05.858 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_temp": [], 2021-02-10T11:23:05.858 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_temp": [], 2021-02-10T11:23:05.858 INFO:teuthology.orchestra.run.gibba002.stdout: "blacklist": {}, 2021-02-10T11:23:05.858 INFO:teuthology.orchestra.run.gibba002.stdout: "erasure_code_profiles": { 2021-02-10T11:23:05.858 INFO:teuthology.orchestra.run.gibba002.stdout: "default": { 2021-02-10T11:23:05.858 INFO:teuthology.orchestra.run.gibba002.stdout: "crush-failure-domain": "osd", 2021-02-10T11:23:05.858 INFO:teuthology.orchestra.run.gibba002.stdout: "k": "2", 2021-02-10T11:23:05.859 INFO:teuthology.orchestra.run.gibba002.stdout: "m": "1", 2021-02-10T11:23:05.859 INFO:teuthology.orchestra.run.gibba002.stdout: "plugin": "jerasure", 2021-02-10T11:23:05.859 INFO:teuthology.orchestra.run.gibba002.stdout: "ruleset-failure-domain": "osd", 2021-02-10T11:23:05.859 INFO:teuthology.orchestra.run.gibba002.stdout: "technique": "reed_sol_van" 2021-02-10T11:23:05.859 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:23:05.859 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:05.859 INFO:teuthology.orchestra.run.gibba002.stdout: "removed_snaps_queue": [], 2021-02-10T11:23:05.859 INFO:teuthology.orchestra.run.gibba002.stdout: "new_removed_snaps": [], 2021-02-10T11:23:05.859 INFO:teuthology.orchestra.run.gibba002.stdout: "new_purged_snaps": [], 2021-02-10T11:23:05.859 INFO:teuthology.orchestra.run.gibba002.stdout: "crush_node_flags": {}, 2021-02-10T11:23:05.859 INFO:teuthology.orchestra.run.gibba002.stdout: "device_class_flags": {} 2021-02-10T11:23:05.859 INFO:teuthology.orchestra.run.gibba002.stdout:} 2021-02-10T11:23:05.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:05.783 7fb8c1702700 1 -- 172.21.2.102:0/564380757 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb89c0186c0 msgr2=0x7fb89c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:05.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:05.783 7fb8c1702700 1 --2- 172.21.2.102:0/564380757 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb89c0186c0 0x7fb89c01abc0 crc :-1 s=READY pgs=548 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:05.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:05.783 7fb8c1702700 1 -- 172.21.2.102:0/564380757 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb8bc1a01e0 msgr2=0x7fb8bc0895c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:05.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:05.783 7fb8c1702700 1 --2- 172.21.2.102:0/564380757 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb8bc1a01e0 0x7fb8bc0895c0 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 rx=0x7fb8b4009a30 tx=0x7fb8b400bd20).stop 2021-02-10T11:23:05.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:05.783 7fb8c1702700 1 -- 172.21.2.102:0/564380757 shutdown_connections 2021-02-10T11:23:05.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:05.783 7fb8c1702700 1 --2- 172.21.2.102:0/564380757 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb8bc1a01e0 0x7fb8bc0895c0 unknown :-1 s=CLOSED pgs=1149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:05.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:05.783 7fb8c1702700 1 --2- 172.21.2.102:0/564380757 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb89c0186c0 0x7fb89c01abc0 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:05.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:05.783 7fb8c1702700 1 --2- 172.21.2.102:0/564380757 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb8bc19fde0 0x7fb8bc09bed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:05.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:05.783 7fb8c1702700 1 --2- 172.21.2.102:0/564380757 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb8bc194000 0x7fb8bc19f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:05.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:05.783 7fb8c1702700 1 -- 172.21.2.102:0/564380757 >> 172.21.2.102:0/564380757 conn(0x7fb8bc092c20 msgr2=0x7fb8bc195070 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:05.862 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:05.784 7fb8c1702700 1 -- 172.21.2.102:0/564380757 shutdown_connections 2021-02-10T11:23:05.862 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:05.784 7fb8c1702700 1 -- 172.21.2.102:0/564380757 wait complete. 2021-02-10T11:23:05.862 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:23:06.289 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:06.276 7f0b6853a700 1 Processor -- start 2021-02-10T11:23:06.290 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:06.276 7f0b6853a700 1 -- start start 2021-02-10T11:23:06.290 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:06.277 7f0b6853a700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0b600d17f0 0x7f0b601b31e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:06.290 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:06.277 7f0b6853a700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f0b600d1bf0 0x7f0b601b3800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:06.290 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:06.277 7f0b6853a700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b600f0b20 con 0x7f0b601b3e20 2021-02-10T11:23:06.290 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:06.277 7f0b6853a700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b600e5eb0 con 0x7f0b600d1bf0 2021-02-10T11:23:06.290 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:06.277 7f0b6853a700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b600f0630 con 0x7f0b600d17f0 2021-02-10T11:23:06.290 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:06.277 7f0b66b57700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f0b601b3e20 0x7f0b600ceef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:48230/0 (socket says 172.21.2.102:48230) 2021-02-10T11:23:06.290 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:06.277 7f0b66b57700 1 -- 172.21.2.102:0/4046638957 learned_addr learned my addr 172.21.2.102:0/4046638957 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:23:06.290 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:06.277 7f0b66356700 1 --2- 172.21.2.102:0/4046638957 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0b600d17f0 0x7f0b601b31e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:06.290 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:06.277 7f0b65b55700 1 --2- 172.21.2.102:0/4046638957 >> v2:172.21.2.113:3300/0 conn(0x7f0b600d1bf0 0x7f0b601b3800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:06.291 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:06.277 7f0b65354700 1 -- 172.21.2.102:0/4046638957 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2926975972 0 0) 0x7f0b58001500 con 0x7f0b601b3e20 2021-02-10T11:23:06.291 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:06.277 7f0b65354700 1 -- 172.21.2.102:0/4046638957 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b4c002fe0 con 0x7f0b601b3e20 2021-02-10T11:23:06.291 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:06.278 7f0b66356700 1 -- 172.21.2.102:0/4046638957 >> v2:172.21.2.113:3300/0 conn(0x7f0b600d1bf0 msgr2=0x7f0b601b3800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:06.291 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:06.278 7f0b66356700 1 --2- 172.21.2.102:0/4046638957 >> v2:172.21.2.113:3300/0 conn(0x7f0b600d1bf0 0x7f0b601b3800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:06.291 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:06.278 7f0b66356700 1 -- 172.21.2.102:0/4046638957 >> v1:172.21.2.102:6789/0 conn(0x7f0b601b3e20 legacy=0x7f0b600ceef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:06.291 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:06.278 7f0b66356700 1 -- 172.21.2.102:0/4046638957 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b600f0820 con 0x7f0b600d17f0 2021-02-10T11:23:06.291 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:06.278 7f0b66356700 1 --2- 172.21.2.102:0/4046638957 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0b600d17f0 0x7f0b601b31e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f0b54006650 tx=0x7f0b54009a80).ready entity=mon.2 client_cookie=89f5952a0c53f532 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:06.291 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:06.278 7f0b65354700 1 -- 172.21.2.102:0/4046638957 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0b5400c0c0 con 0x7f0b600d17f0 2021-02-10T11:23:06.291 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:06.278 7f0b65354700 1 -- 172.21.2.102:0/4046638957 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b5400c340 con 0x7f0b600d17f0 2021-02-10T11:23:06.291 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:06.278 7f0b65354700 1 -- 172.21.2.102:0/4046638957 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0b5400c860 con 0x7f0b600d17f0 2021-02-10T11:23:06.291 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:06.278 7f0b6853a700 1 -- 172.21.2.102:0/4046638957 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0b600d17f0 msgr2=0x7f0b601b31e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:06.291 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:06.278 7f0b6853a700 1 --2- 172.21.2.102:0/4046638957 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0b600d17f0 0x7f0b601b31e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f0b54006650 tx=0x7f0b54009a80).stop 2021-02-10T11:23:06.291 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:06.279 7f0b6853a700 1 -- 172.21.2.102:0/4046638957 shutdown_connections 2021-02-10T11:23:06.292 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:06.279 7f0b6853a700 1 --2- 172.21.2.102:0/4046638957 >> v2:172.21.2.113:3300/0 conn(0x7f0b600d1bf0 0x7f0b601b3800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:06.292 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:06.279 7f0b6853a700 1 --2- 172.21.2.102:0/4046638957 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0b600d17f0 0x7f0b601b31e0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:06.292 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:06.279 7f0b6853a700 1 -- 172.21.2.102:0/4046638957 >> 172.21.2.102:0/4046638957 conn(0x7f0b600fde40 msgr2=0x7f0b600ce390 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:06.292 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:06.279 7f0b6853a700 1 -- 172.21.2.102:0/4046638957 shutdown_connections 2021-02-10T11:23:06.292 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:06.279 7f0b6853a700 1 -- 172.21.2.102:0/4046638957 wait complete. 2021-02-10T11:23:06.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:06.293 7f0b6853a700 1 Processor -- start 2021-02-10T11:23:06.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:06.293 7f0b6853a700 1 -- start start 2021-02-10T11:23:06.294 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:06.294 7f0b6853a700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0b60103900 0x7f0b600c2840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:06.294 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:06.294 7f0b6853a700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0b60103d00 0x7f0b600bfcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:06.294 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:06.294 7f0b6853a700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0b600c02d0 0x7f0b601047c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:06.294 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:06.294 7f0b6853a700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b60104240 con 0x7f0b60103d00 2021-02-10T11:23:06.294 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:06.294 7f0b6853a700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b60104de0 con 0x7f0b60103900 2021-02-10T11:23:06.294 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:06.294 7f0b6853a700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b60105000 con 0x7f0b600c02d0 2021-02-10T11:23:06.294 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:06.294 7f0b65b55700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0b60103d00 0x7f0b600bfcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:06.294 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:06.294 7f0b65b55700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0b60103d00 0x7f0b600bfcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:56010/0 (socket says 172.21.2.102:56010) 2021-02-10T11:23:06.294 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:06.294 7f0b65b55700 1 -- 172.21.2.102:0/4007452814 learned_addr learned my addr 172.21.2.102:0/4007452814 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:23:06.294 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:06.294 7f0b66b57700 1 --2- 172.21.2.102:0/4007452814 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0b600c02d0 0x7f0b601047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:06.294 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:06.294 7f0b66356700 1 --2- 172.21.2.102:0/4007452814 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0b60103900 0x7f0b600c2840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:06.294 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:06.294 7f0b65b55700 1 -- 172.21.2.102:0/4007452814 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0b600c02d0 msgr2=0x7f0b601047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:06.295 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:06.294 7f0b65b55700 1 --2- 172.21.2.102:0/4007452814 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0b600c02d0 0x7f0b601047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:06.295 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:06.294 7f0b65b55700 1 -- 172.21.2.102:0/4007452814 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0b60103900 msgr2=0x7f0b600c2840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:06.295 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:06.294 7f0b65b55700 1 --2- 172.21.2.102:0/4007452814 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0b60103900 0x7f0b600c2840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:06.295 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:06.294 7f0b65b55700 1 -- 172.21.2.102:0/4007452814 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b60105220 con 0x7f0b60103d00 2021-02-10T11:23:06.295 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:06.295 7f0b65b55700 1 --2- 172.21.2.102:0/4007452814 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0b60103d00 0x7f0b600bfcb0 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 rx=0x7f0b5c006f60 tx=0x7f0b5c00b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:06.295 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:06.295 7f0b52ffd700 1 -- 172.21.2.102:0/4007452814 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0b5c00bd60 con 0x7f0b60103d00 2021-02-10T11:23:06.295 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:06.295 7f0b52ffd700 1 -- 172.21.2.102:0/4007452814 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b5c000c90 con 0x7f0b60103d00 2021-02-10T11:23:06.295 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:06.295 7f0b6853a700 1 -- 172.21.2.102:0/4007452814 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b60105220 con 0x7f0b60103d00 2021-02-10T11:23:06.295 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:06.295 7f0b6853a700 1 -- 172.21.2.102:0/4007452814 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b601057b0 con 0x7f0b60103d00 2021-02-10T11:23:06.295 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:06.295 7f0b52ffd700 1 -- 172.21.2.102:0/4007452814 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0b5c0022b0 con 0x7f0b60103d00 2021-02-10T11:23:06.295 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:06.296 7f0b52ffd700 1 -- 172.21.2.102:0/4007452814 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f0b5c0052e0 con 0x7f0b60103d00 2021-02-10T11:23:06.295 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:06.296 7f0b52ffd700 1 --2- 172.21.2.102:0/4007452814 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0b48018730 0x7f0b4801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:06.295 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:06.296 7f0b52ffd700 1 -- 172.21.2.102:0/4007452814 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f0b5c0327e0 con 0x7f0b60103d00 2021-02-10T11:23:06.296 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:06.296 7f0b66356700 1 --2- 172.21.2.102:0/4007452814 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0b48018730 0x7f0b4801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:06.296 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:06.296 7f0b66356700 1 --2- 172.21.2.102:0/4007452814 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0b48018730 0x7f0b4801ac10 crc :-1 s=READY pgs=549 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:06.323 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:06.327 7f0b6853a700 1 -- 172.21.2.102:0/4007452814 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b601036d0 con 0x7f0b60103d00 2021-02-10T11:23:06.326 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:06.329 7f0b52ffd700 1 -- 172.21.2.102:0/4007452814 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f0b5c000c90 con 0x7f0b60103d00 2021-02-10T11:23:06.454 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:06.457 7f0b6853a700 1 -- 172.21.2.102:0/4007452814 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0b600f0b20 con 0x7f0b60103d00 2021-02-10T11:23:06.455 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:06.458 7f0b52ffd700 1 -- 172.21.2.102:0/4007452814 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+2099 (secure 0 0 0) 0x7f0b5c00bd60 con 0x7f0b60103d00 2021-02-10T11:23:06.455 INFO:teuthology.orchestra.run.gibba002.stderr:dumped fsmap epoch 22 2021-02-10T11:23:06.455 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:23:06.455 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":22,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7194,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.102:6835/2295196020","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6834","nonce":2295196020},{"type":"v1","addr":"172.21.2.102:6835","nonce":2295196020}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":18}],"filesystems":[{"mdsmap":{"epoch":21,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:22:26.122430","modified":"2021-02-10 11:22:28.126179","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7191},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7191":{"gid":7191,"name":"b","rank":0,"incarnation":19,"state":"up:active","state_seq":3,"addr":"172.21.2.113:6833/3694995761","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6832","nonce":3694995761},{"type":"v1","addr":"172.21.2.113:6833","nonce":3694995761}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":3}]} 2021-02-10T11:23:06.456 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:06.460 7f0b6853a700 1 -- 172.21.2.102:0/4007452814 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0b48018730 msgr2=0x7f0b4801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:06.456 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:06.460 7f0b6853a700 1 --2- 172.21.2.102:0/4007452814 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0b48018730 0x7f0b4801ac10 crc :-1 s=READY pgs=549 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:06.457 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:06.460 7f0b6853a700 1 -- 172.21.2.102:0/4007452814 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0b60103d00 msgr2=0x7f0b600bfcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:06.457 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:06.460 7f0b6853a700 1 --2- 172.21.2.102:0/4007452814 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0b60103d00 0x7f0b600bfcb0 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 rx=0x7f0b5c006f60 tx=0x7f0b5c00b970).stop 2021-02-10T11:23:06.457 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:06.460 7f0b6853a700 1 -- 172.21.2.102:0/4007452814 shutdown_connections 2021-02-10T11:23:06.457 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:06.460 7f0b6853a700 1 --2- 172.21.2.102:0/4007452814 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0b600c02d0 0x7f0b601047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:06.457 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:06.460 7f0b6853a700 1 --2- 172.21.2.102:0/4007452814 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0b60103d00 0x7f0b600bfcb0 unknown :-1 s=CLOSED pgs=1151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:06.457 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:06.460 7f0b6853a700 1 --2- 172.21.2.102:0/4007452814 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0b48018730 0x7f0b4801ac10 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:06.457 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:06.460 7f0b6853a700 1 --2- 172.21.2.102:0/4007452814 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0b60103900 0x7f0b600c2840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:06.457 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:06.460 7f0b6853a700 1 -- 172.21.2.102:0/4007452814 >> 172.21.2.102:0/4007452814 conn(0x7f0b600b5e30 msgr2=0x7f0b600c3c90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:06.457 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:06.460 7f0b6853a700 1 -- 172.21.2.102:0/4007452814 shutdown_connections 2021-02-10T11:23:06.458 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:06.460 7f0b6853a700 1 -- 172.21.2.102:0/4007452814 wait complete. 2021-02-10T11:23:06.465 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs fail cephfs 2021-02-10T11:23:06.893 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:06.880 7f79f95df700 1 Processor -- start 2021-02-10T11:23:06.893 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:06.880 7f79f95df700 1 -- start start 2021-02-10T11:23:06.893 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:06.881 7f79f95df700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f79f40e5070 0x7f79f41b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:06.893 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:06.881 7f79f95df700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f79f40e5470 0x7f79f4080310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:06.894 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:06.881 7f79f95df700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79f407ed80 con 0x7f79f41b3e40 2021-02-10T11:23:06.894 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:06.881 7f79f95df700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79f40d1790 con 0x7f79f40e5070 2021-02-10T11:23:06.894 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:06.881 7f79f95df700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79f40d19b0 con 0x7f79f40e5470 2021-02-10T11:23:06.894 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:06.881 7f79f2abe700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f79f41b3e40 0x7f79f407d690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:48242/0 (socket says 172.21.2.102:48242) 2021-02-10T11:23:06.894 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:06.881 7f79f2abe700 1 -- 172.21.2.102:0/2945211089 learned_addr learned my addr 172.21.2.102:0/2945211089 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:23:06.894 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:06.881 7f79f32bf700 1 --2- 172.21.2.102:0/2945211089 >> v2:172.21.2.102:3300/0 conn(0x7f79f40e5070 0x7f79f41b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:06.894 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:06.881 7f79f22bd700 1 -- 172.21.2.102:0/2945211089 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3303346443 0 0) 0x7f79dc0010b0 con 0x7f79f41b3e40 2021-02-10T11:23:06.894 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:06.881 7f79f22bd700 1 -- 172.21.2.102:0/2945211089 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f79e0002fe0 con 0x7f79f41b3e40 2021-02-10T11:23:06.895 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:06.881 7f79f32bf700 1 -- 172.21.2.102:0/2945211089 >> v2:172.21.2.113:3300/0 conn(0x7f79f40e5470 msgr2=0x7f79f4080310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:06.895 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:06.881 7f79f32bf700 1 --2- 172.21.2.102:0/2945211089 >> v2:172.21.2.113:3300/0 conn(0x7f79f40e5470 0x7f79f4080310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:06.895 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:06.881 7f79f32bf700 1 -- 172.21.2.102:0/2945211089 >> v1:172.21.2.102:6789/0 conn(0x7f79f41b3e40 legacy=0x7f79f407d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:06.895 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:06.881 7f79f32bf700 1 -- 172.21.2.102:0/2945211089 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79f40d1bd0 con 0x7f79f40e5070 2021-02-10T11:23:06.895 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:06.882 7f79f32bf700 1 --2- 172.21.2.102:0/2945211089 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f79f40e5070 0x7f79f41b3820 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 rx=0x7f79e4006650 tx=0x7f79e4009a80).ready entity=mon.0 client_cookie=8a219a581fb3feab server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:06.895 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:06.882 7f79f22bd700 1 -- 172.21.2.102:0/2945211089 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f79e400c0c0 con 0x7f79f40e5070 2021-02-10T11:23:06.895 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:06.882 7f79f22bd700 1 -- 172.21.2.102:0/2945211089 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79e400c340 con 0x7f79f40e5070 2021-02-10T11:23:06.895 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:06.882 7f79f22bd700 1 -- 172.21.2.102:0/2945211089 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f79e400c860 con 0x7f79f40e5070 2021-02-10T11:23:06.895 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:06.882 7f79f95df700 1 -- 172.21.2.102:0/2945211089 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f79f40e5070 msgr2=0x7f79f41b3820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:06.895 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:06.882 7f79f95df700 1 --2- 172.21.2.102:0/2945211089 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f79f40e5070 0x7f79f41b3820 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 rx=0x7f79e4006650 tx=0x7f79e4009a80).stop 2021-02-10T11:23:06.895 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:06.882 7f79f95df700 1 -- 172.21.2.102:0/2945211089 shutdown_connections 2021-02-10T11:23:06.895 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:06.882 7f79f95df700 1 --2- 172.21.2.102:0/2945211089 >> v2:172.21.2.113:3300/0 conn(0x7f79f40e5470 0x7f79f4080310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:06.895 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:06.882 7f79f95df700 1 --2- 172.21.2.102:0/2945211089 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f79f40e5070 0x7f79f41b3820 unknown :-1 s=CLOSED pgs=1153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:06.896 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:06.882 7f79f95df700 1 -- 172.21.2.102:0/2945211089 >> 172.21.2.102:0/2945211089 conn(0x7f79f40ce690 msgr2=0x7f79f40e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:06.896 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:06.882 7f79f95df700 1 -- 172.21.2.102:0/2945211089 shutdown_connections 2021-02-10T11:23:06.896 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:06.882 7f79f95df700 1 -- 172.21.2.102:0/2945211089 wait complete. 2021-02-10T11:23:06.897 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:06.897 7f79f95df700 1 Processor -- start 2021-02-10T11:23:06.897 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:06.897 7f79f95df700 1 -- start start 2021-02-10T11:23:06.897 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:06.897 7f79f95df700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f79f408a430 0x7f79f41a73e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:06.897 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:06.897 7f79f95df700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f79f41a7a00 0x7f79f409c040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:06.898 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:06.897 7f79f95df700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f79f409c660 0x7f79f409ca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:06.898 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:06.897 7f79f95df700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79f408a9b0 con 0x7f79f41a7a00 2021-02-10T11:23:06.898 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:06.897 7f79f95df700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79f409d080 con 0x7f79f409c660 2021-02-10T11:23:06.898 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:06.897 7f79f95df700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79f4192b00 con 0x7f79f408a430 2021-02-10T11:23:06.898 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:06.898 7f79f2abe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f79f41a7a00 0x7f79f409c040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:06.898 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:06.898 7f79f2abe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f79f41a7a00 0x7f79f409c040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:56022/0 (socket says 172.21.2.102:56022) 2021-02-10T11:23:06.898 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:06.898 7f79f2abe700 1 -- 172.21.2.102:0/3750675819 learned_addr learned my addr 172.21.2.102:0/3750675819 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:23:06.898 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:06.898 7f79f2abe700 1 -- 172.21.2.102:0/3750675819 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f79f408a430 msgr2=0x7f79f41a73e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:06.898 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:06.898 7f79f2abe700 1 --2- 172.21.2.102:0/3750675819 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f79f408a430 0x7f79f41a73e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:06.898 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:06.898 7f79f2abe700 1 -- 172.21.2.102:0/3750675819 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f79f409c660 msgr2=0x7f79f409ca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:06.899 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:06.898 7f79f2abe700 1 --2- 172.21.2.102:0/3750675819 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f79f409c660 0x7f79f409ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:06.899 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:06.898 7f79f2abe700 1 -- 172.21.2.102:0/3750675819 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79f408af30 con 0x7f79f41a7a00 2021-02-10T11:23:06.899 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:06.898 7f79f2abe700 1 --2- 172.21.2.102:0/3750675819 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f79f41a7a00 0x7f79f409c040 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 rx=0x7f79dc00a4c0 tx=0x7f79dc00a860).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:06.899 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:06.898 7f79ebfff700 1 -- 172.21.2.102:0/3750675819 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f79dc000e50 con 0x7f79f41a7a00 2021-02-10T11:23:06.899 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:06.898 7f79ebfff700 1 -- 172.21.2.102:0/3750675819 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79dc00ac50 con 0x7f79f41a7a00 2021-02-10T11:23:06.899 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:06.898 7f79ebfff700 1 -- 172.21.2.102:0/3750675819 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f79dc0045d0 con 0x7f79f41a7a00 2021-02-10T11:23:06.899 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:06.898 7f79f95df700 1 -- 172.21.2.102:0/3750675819 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79f408af30 con 0x7f79f41a7a00 2021-02-10T11:23:06.899 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:06.898 7f79f95df700 1 -- 172.21.2.102:0/3750675819 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79f408af30 con 0x7f79f41a7a00 2021-02-10T11:23:06.899 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:06.899 7f79ebfff700 1 -- 172.21.2.102:0/3750675819 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f79dc004350 con 0x7f79f41a7a00 2021-02-10T11:23:06.899 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:06.899 7f79ebfff700 1 --2- 172.21.2.102:0/3750675819 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f79d80186c0 0x7f79d801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:06.899 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:06.899 7f79ebfff700 1 -- 172.21.2.102:0/3750675819 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f79dc0325c0 con 0x7f79f41a7a00 2021-02-10T11:23:06.899 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:06.899 7f79f32bf700 1 --2- 172.21.2.102:0/3750675819 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f79d80186c0 0x7f79d801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:06.899 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:06.900 7f79f32bf700 1 --2- 172.21.2.102:0/3750675819 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f79d80186c0 0x7f79d801abc0 crc :-1 s=READY pgs=550 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:06.927 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:06.930 7f79f95df700 1 -- 172.21.2.102:0/3750675819 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79f4084ab0 con 0x7f79f41a7a00 2021-02-10T11:23:06.929 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:06.933 7f79ebfff700 1 -- 172.21.2.102:0/3750675819 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f79dc004140 con 0x7f79f41a7a00 2021-02-10T11:23:07.057 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:07.061 7f79f95df700 1 -- 172.21.2.102:0/3750675819 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f79f407ed80 con 0x7f79f41a7a00 2021-02-10T11:23:07.833 INFO:tasks.ceph.mon.a.gibba002.stderr:2021-02-10 11:23:07.836 7f8dd2755700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-02-10T11:23:07.838 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:07.842 7f79ebfff700 1 -- 172.21.2.102:0/3750675819 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v23) v1 ==== 161+0+0 (secure 0 0 0) 0x7f79dc004350 con 0x7f79f41a7a00 2021-02-10T11:23:07.876 INFO:teuthology.orchestra.run.gibba002.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2021-02-10T11:23:07.878 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:07.881 7f79f95df700 1 -- 172.21.2.102:0/3750675819 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f79d80186c0 msgr2=0x7f79d801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:07.879 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:07.881 7f79f95df700 1 --2- 172.21.2.102:0/3750675819 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f79d80186c0 0x7f79d801abc0 crc :-1 s=READY pgs=550 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:07.879 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:07.881 7f79f95df700 1 -- 172.21.2.102:0/3750675819 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f79f41a7a00 msgr2=0x7f79f409c040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:07.879 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:07.881 7f79f95df700 1 --2- 172.21.2.102:0/3750675819 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f79f41a7a00 0x7f79f409c040 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 rx=0x7f79dc00a4c0 tx=0x7f79dc00a860).stop 2021-02-10T11:23:07.879 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:07.881 7f79f95df700 1 -- 172.21.2.102:0/3750675819 shutdown_connections 2021-02-10T11:23:07.879 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:07.881 7f79f95df700 1 --2- 172.21.2.102:0/3750675819 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f79d80186c0 0x7f79d801abc0 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:07.879 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:07.881 7f79f95df700 1 --2- 172.21.2.102:0/3750675819 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f79f409c660 0x7f79f409ca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:07.879 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:07.881 7f79f95df700 1 --2- 172.21.2.102:0/3750675819 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f79f41a7a00 0x7f79f409c040 unknown :-1 s=CLOSED pgs=1154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:07.879 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:07.881 7f79f95df700 1 --2- 172.21.2.102:0/3750675819 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f79f408a430 0x7f79f41a73e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:07.879 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:07.881 7f79f95df700 1 -- 172.21.2.102:0/3750675819 >> 172.21.2.102:0/3750675819 conn(0x7f79f40aac70 msgr2=0x7f79f4097bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:07.879 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:07.882 7f79f95df700 1 -- 172.21.2.102:0/3750675819 shutdown_connections 2021-02-10T11:23:07.879 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:07.882 7f79f95df700 1 -- 172.21.2.102:0/3750675819 wait complete. 2021-02-10T11:23:07.887 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:23:07.888 INFO:tasks.ceph.mds.b.gibba013.stdout:starting mds.b at 2021-02-10T11:23:08.314 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:08.300 7f1697133700 1 Processor -- start 2021-02-10T11:23:08.314 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:08.300 7f1697133700 1 -- start start 2021-02-10T11:23:08.314 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:08.301 7f1697133700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f16900d17f0 0x7f16901b31e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:08.314 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:08.301 7f1697133700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f16900cff00 0x7f16900ef620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:08.314 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:08.301 7f1697133700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16900f0b20 con 0x7f16900d1bf0 2021-02-10T11:23:08.314 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:08.301 7f1697133700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16901b3ff0 con 0x7f16900d17f0 2021-02-10T11:23:08.314 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:08.301 7f1697133700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16900f20b0 con 0x7f16900cff00 2021-02-10T11:23:08.314 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:08.301 7f1695750700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f16900cff00 0x7f16900ef620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:08.314 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:08.301 7f1695750700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f16900cff00 0x7f16900ef620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.113:3301/0 says I am v2:172.21.2.102:33338/0 (socket says 172.21.2.102:33338) 2021-02-10T11:23:08.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:08.301 7f1695750700 1 -- 172.21.2.102:0/130501276 learned_addr learned my addr 172.21.2.102:0/130501276 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:23:08.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:08.301 7f1694f4f700 1 --2- 172.21.2.102:0/130501276 >> v2:172.21.2.113:3300/0 conn(0x7f16900d17f0 0x7f16901b31e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:08.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:08.302 7f1694f4f700 1 -- 172.21.2.102:0/130501276 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f16900cff00 msgr2=0x7f16900ef620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:08.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:08.302 7f1694f4f700 1 --2- 172.21.2.102:0/130501276 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f16900cff00 0x7f16900ef620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:08.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:08.302 7f1694f4f700 1 -- 172.21.2.102:0/130501276 >> v1:172.21.2.113:6789/0 conn(0x7f16900d1bf0 legacy=0x7f16901b3800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:08.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:08.302 7f1694f4f700 1 -- 172.21.2.102:0/130501276 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16900f22d0 con 0x7f16900d17f0 2021-02-10T11:23:08.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:08.302 7f1695750700 1 --2- 172.21.2.102:0/130501276 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f16900cff00 0x7f16900ef620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:23:08.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:08.302 7f1694f4f700 1 --2- 172.21.2.102:0/130501276 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f16900d17f0 0x7f16901b31e0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 rx=0x7f1678006650 tx=0x7f1678009a80).ready entity=mon.1 client_cookie=69e929beedc48318 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:08.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:08.302 7f168f7fe700 1 -- 172.21.2.102:0/130501276 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f16780009b0 con 0x7f16900d17f0 2021-02-10T11:23:08.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:08.302 7f168f7fe700 1 -- 172.21.2.102:0/130501276 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f167800c0c0 con 0x7f16900d17f0 2021-02-10T11:23:08.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:08.302 7f168f7fe700 1 -- 172.21.2.102:0/130501276 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f167800c5e0 con 0x7f16900d17f0 2021-02-10T11:23:08.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:08.302 7f1697133700 1 -- 172.21.2.102:0/130501276 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f16900d17f0 msgr2=0x7f16901b31e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:08.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:08.302 7f1697133700 1 --2- 172.21.2.102:0/130501276 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f16900d17f0 0x7f16901b31e0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 rx=0x7f1678006650 tx=0x7f1678009a80).stop 2021-02-10T11:23:08.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:08.303 7f1697133700 1 -- 172.21.2.102:0/130501276 shutdown_connections 2021-02-10T11:23:08.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:08.303 7f1697133700 1 --2- 172.21.2.102:0/130501276 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f16900cff00 0x7f16900ef620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:08.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:08.303 7f1697133700 1 --2- 172.21.2.102:0/130501276 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f16900d17f0 0x7f16901b31e0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:08.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:08.303 7f1697133700 1 -- 172.21.2.102:0/130501276 >> 172.21.2.102:0/130501276 conn(0x7f16900fde40 msgr2=0x7f16900ce390 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:08.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:08.303 7f1697133700 1 -- 172.21.2.102:0/130501276 shutdown_connections 2021-02-10T11:23:08.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:08.303 7f1697133700 1 -- 172.21.2.102:0/130501276 wait complete. 2021-02-10T11:23:08.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:08.318 7f1697133700 1 Processor -- start 2021-02-10T11:23:08.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:08.318 7f1697133700 1 -- start start 2021-02-10T11:23:08.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:08.318 7f1697133700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f16901986f0 0x7f169019fea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:08.318 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:08.318 7f1697133700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f16901980d0 0x7f16900a9e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:08.318 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:08.318 7f1697133700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f16900aa470 0x7f16901a0af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:08.318 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:08.318 7f1697133700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16901a05f0 con 0x7f16901986f0 2021-02-10T11:23:08.318 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:08.318 7f1697133700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f169009dab0 con 0x7f16901980d0 2021-02-10T11:23:08.318 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:08.318 7f1697133700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f169018eec0 con 0x7f16900aa470 2021-02-10T11:23:08.318 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:08.318 7f1694f4f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f16901986f0 0x7f169019fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:08.318 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:08.318 7f1694f4f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f16901986f0 0x7f169019fea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:56040/0 (socket says 172.21.2.102:56040) 2021-02-10T11:23:08.318 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:08.318 7f1694f4f700 1 -- 172.21.2.102:0/402937334 learned_addr learned my addr 172.21.2.102:0/402937334 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:23:08.318 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:08.319 7f1694f4f700 1 -- 172.21.2.102:0/402937334 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f16900aa470 msgr2=0x7f16901a0af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:08.318 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:08.319 7f1694f4f700 1 --2- 172.21.2.102:0/402937334 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f16900aa470 0x7f16901a0af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:08.318 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:08.319 7f1694f4f700 1 -- 172.21.2.102:0/402937334 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f16901980d0 msgr2=0x7f16900a9e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:08.318 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:08.319 7f1694f4f700 1 --2- 172.21.2.102:0/402937334 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f16901980d0 0x7f16900a9e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:08.318 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:08.319 7f1694f4f700 1 -- 172.21.2.102:0/402937334 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f169018f0a0 con 0x7f16901986f0 2021-02-10T11:23:08.319 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:08.319 7f1694f4f700 1 --2- 172.21.2.102:0/402937334 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f16901986f0 0x7f169019fea0 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 rx=0x7f1678009630 tx=0x7f167800b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:08.319 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:08.319 7f168d7fa700 1 -- 172.21.2.102:0/402937334 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f16780093a0 con 0x7f16901986f0 2021-02-10T11:23:08.319 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:08.319 7f168d7fa700 1 -- 172.21.2.102:0/402937334 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f167800bd70 con 0x7f16901986f0 2021-02-10T11:23:08.319 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:08.319 7f1697133700 1 -- 172.21.2.102:0/402937334 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f169018f0a0 con 0x7f16901986f0 2021-02-10T11:23:08.319 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:08.319 7f168d7fa700 1 -- 172.21.2.102:0/402937334 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1678005560 con 0x7f16901986f0 2021-02-10T11:23:08.319 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:08.319 7f1697133700 1 -- 172.21.2.102:0/402937334 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16901a1400 con 0x7f16901986f0 2021-02-10T11:23:08.319 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:08.320 7f168d7fa700 1 -- 172.21.2.102:0/402937334 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f167800a000 con 0x7f16901986f0 2021-02-10T11:23:08.319 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:08.320 7f168d7fa700 1 --2- 172.21.2.102:0/402937334 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f16740186c0 0x7f167401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:08.319 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:08.320 7f168d7fa700 1 -- 172.21.2.102:0/402937334 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 5659+0+0 (secure 0 0 0) 0x7f16780325d0 con 0x7f16901986f0 2021-02-10T11:23:08.319 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:08.320 7f168ffff700 1 --2- 172.21.2.102:0/402937334 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f16740186c0 0x7f167401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:08.319 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:08.320 7f168ffff700 1 --2- 172.21.2.102:0/402937334 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f16740186c0 0x7f167401abc0 crc :-1 s=READY pgs=551 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:08.347 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:08.350 7f1697133700 1 -- 172.21.2.102:0/402937334 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16900f0b20 con 0x7f16901986f0 2021-02-10T11:23:08.349 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:08.353 7f168d7fa700 1 -- 172.21.2.102:0/402937334 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f1678009340 con 0x7f16901986f0 2021-02-10T11:23:08.477 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:08.480 7f1697133700 1 -- 172.21.2.102:0/402937334 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f16901a0570 con 0x7f16901986f0 2021-02-10T11:23:08.477 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:08.481 7f168d7fa700 1 -- 172.21.2.102:0/402937334 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+1754 (secure 0 0 0) 0x7f167802d830 con 0x7f16901986f0 2021-02-10T11:23:08.478 INFO:teuthology.orchestra.run.gibba002.stderr:dumped fsmap epoch 23 2021-02-10T11:23:08.478 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:23:08.478 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":23,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7194,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.102:6835/2295196020","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6834","nonce":2295196020},{"type":"v1","addr":"172.21.2.102:6835","nonce":2295196020}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":18}],"filesystems":[{"mdsmap":{"epoch":23,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:22:26.122430","modified":"2021-02-10 11:23:07.837009","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":3}]} 2021-02-10T11:23:08.479 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:08.483 7f1697133700 1 -- 172.21.2.102:0/402937334 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f16740186c0 msgr2=0x7f167401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:08.479 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:08.483 7f1697133700 1 --2- 172.21.2.102:0/402937334 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f16740186c0 0x7f167401abc0 crc :-1 s=READY pgs=551 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:08.479 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:08.483 7f1697133700 1 -- 172.21.2.102:0/402937334 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f16901986f0 msgr2=0x7f169019fea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:08.479 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:08.483 7f1697133700 1 --2- 172.21.2.102:0/402937334 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f16901986f0 0x7f169019fea0 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 rx=0x7f1678009630 tx=0x7f167800b980).stop 2021-02-10T11:23:08.480 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:08.483 7f1697133700 1 -- 172.21.2.102:0/402937334 shutdown_connections 2021-02-10T11:23:08.480 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:08.483 7f1697133700 1 --2- 172.21.2.102:0/402937334 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f16900aa470 0x7f16901a0af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:08.480 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:08.483 7f1697133700 1 --2- 172.21.2.102:0/402937334 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f16740186c0 0x7f167401abc0 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:08.480 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:08.483 7f1697133700 1 --2- 172.21.2.102:0/402937334 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f16901980d0 0x7f16900a9e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:08.480 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:08.483 7f1697133700 1 --2- 172.21.2.102:0/402937334 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f16901986f0 0x7f169019fea0 unknown :-1 s=CLOSED pgs=1157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:08.480 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:08.483 7f1697133700 1 -- 172.21.2.102:0/402937334 >> 172.21.2.102:0/402937334 conn(0x7f16901a3780 msgr2=0x7f1690198c50 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:08.480 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:08.483 7f1697133700 1 -- 172.21.2.102:0/402937334 shutdown_connections 2021-02-10T11:23:08.480 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:08.483 7f1697133700 1 -- 172.21.2.102:0/402937334 wait complete. 2021-02-10T11:23:08.488 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2021-02-10T11:23:08.919 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:08.906 7fab8c0e8700 1 Processor -- start 2021-02-10T11:23:08.920 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:08.906 7fab8c0e8700 1 -- start start 2021-02-10T11:23:08.920 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:08.906 7fab8c0e8700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fab840d17f0 0x7fab841b31e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:08.920 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:08.906 7fab8c0e8700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fab840d1bf0 0x7fab841b3800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:08.920 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:08.906 7fab8c0e8700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fab841b3e20 0x7fab840ceef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:08.920 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:08.906 7fab8c0e8700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab840e5eb0 con 0x7fab840d1bf0 2021-02-10T11:23:08.920 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:08.906 7fab8c0e8700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab840f0b20 con 0x7fab841b3e20 2021-02-10T11:23:08.920 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:08.906 7fab8c0e8700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab840cf510 con 0x7fab840d17f0 2021-02-10T11:23:08.920 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:08.906 7fab89703700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fab840d1bf0 0x7fab841b3800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:08.920 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:08.906 7fab8a705700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fab841b3e20 0x7fab840ceef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:08.921 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:08.906 7fab89703700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fab840d1bf0 0x7fab841b3800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:56044/0 (socket says 172.21.2.102:56044) 2021-02-10T11:23:08.921 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:08.907 7fab89703700 1 -- 172.21.2.102:0/856348790 learned_addr learned my addr 172.21.2.102:0/856348790 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:23:08.921 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:08.907 7fab89f04700 1 --2- 172.21.2.102:0/856348790 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fab840d17f0 0x7fab841b31e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:08.921 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:08.907 7fab89703700 1 -- 172.21.2.102:0/856348790 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fab840d17f0 msgr2=0x7fab841b31e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:08.921 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:08.907 7fab89703700 1 --2- 172.21.2.102:0/856348790 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fab840d17f0 0x7fab841b31e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:08.921 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:08.907 7fab89703700 1 -- 172.21.2.102:0/856348790 >> v2:172.21.2.113:3300/0 conn(0x7fab841b3e20 msgr2=0x7fab840ceef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:08.921 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:08.907 7fab89703700 1 --2- 172.21.2.102:0/856348790 >> v2:172.21.2.113:3300/0 conn(0x7fab841b3e20 0x7fab840ceef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:08.921 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:08.907 7fab89703700 1 -- 172.21.2.102:0/856348790 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab840cf730 con 0x7fab840d1bf0 2021-02-10T11:23:08.921 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:08.907 7fab8a705700 1 --2- 172.21.2.102:0/856348790 >> v2:172.21.2.113:3300/0 conn(0x7fab841b3e20 0x7fab840ceef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:23:08.921 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:08.907 7fab89703700 1 --2- 172.21.2.102:0/856348790 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fab840d1bf0 0x7fab841b3800 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 rx=0x7fab78006700 tx=0x7fab78009b80).ready entity=mon.0 client_cookie=add9d7a8ba564fb9 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:08.921 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:08.907 7fab88f02700 1 -- 172.21.2.102:0/856348790 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fab7800c0c0 con 0x7fab840d1bf0 2021-02-10T11:23:08.921 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:08.907 7fab88f02700 1 -- 172.21.2.102:0/856348790 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab7800c340 con 0x7fab840d1bf0 2021-02-10T11:23:08.921 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:08.907 7fab88f02700 1 -- 172.21.2.102:0/856348790 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fab7800c860 con 0x7fab840d1bf0 2021-02-10T11:23:08.922 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:08.907 7fab8c0e8700 1 -- 172.21.2.102:0/856348790 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fab840d1bf0 msgr2=0x7fab841b3800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:08.922 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:08.907 7fab8c0e8700 1 --2- 172.21.2.102:0/856348790 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fab840d1bf0 0x7fab841b3800 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 rx=0x7fab78006700 tx=0x7fab78009b80).stop 2021-02-10T11:23:08.922 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:08.908 7fab8c0e8700 1 -- 172.21.2.102:0/856348790 shutdown_connections 2021-02-10T11:23:08.922 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:08.908 7fab8c0e8700 1 --2- 172.21.2.102:0/856348790 >> v2:172.21.2.113:3300/0 conn(0x7fab841b3e20 0x7fab840ceef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:08.922 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:08.908 7fab8c0e8700 1 --2- 172.21.2.102:0/856348790 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fab840d1bf0 0x7fab841b3800 unknown :-1 s=CLOSED pgs=1158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:08.922 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:08.908 7fab8c0e8700 1 --2- 172.21.2.102:0/856348790 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fab840d17f0 0x7fab841b31e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:08.922 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:08.908 7fab8c0e8700 1 -- 172.21.2.102:0/856348790 >> 172.21.2.102:0/856348790 conn(0x7fab840fde40 msgr2=0x7fab840ce390 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:08.922 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:08.908 7fab8c0e8700 1 -- 172.21.2.102:0/856348790 shutdown_connections 2021-02-10T11:23:08.922 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:08.908 7fab8c0e8700 1 -- 172.21.2.102:0/856348790 wait complete. 2021-02-10T11:23:08.922 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:08.923 7fab8c0e8700 1 Processor -- start 2021-02-10T11:23:08.922 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:08.923 7fab8c0e8700 1 -- start start 2021-02-10T11:23:08.924 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:08.923 7fab8c0e8700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fab841a7a20 0x7fab841a7e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:08.924 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:08.923 7fab8c0e8700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fab840aeb20 0x7fab840aef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:08.924 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:08.923 7fab8c0e8700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fab840af540 0x7fab840af940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:08.924 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:08.923 7fab8c0e8700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab840ae5a0 con 0x7fab840aeb20 2021-02-10T11:23:08.925 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:08.924 7fab8c0e8700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab840a43d0 con 0x7fab840af540 2021-02-10T11:23:08.925 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:08.924 7fab8c0e8700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab840a45f0 con 0x7fab841a7a20 2021-02-10T11:23:08.925 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:08.924 7fab89703700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fab840aeb20 0x7fab840aef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:08.925 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:08.924 7fab89703700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fab840aeb20 0x7fab840aef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:56048/0 (socket says 172.21.2.102:56048) 2021-02-10T11:23:08.925 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:08.924 7fab89703700 1 -- 172.21.2.102:0/2388991566 learned_addr learned my addr 172.21.2.102:0/2388991566 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:23:08.925 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:08.924 7fab89f04700 1 --2- 172.21.2.102:0/2388991566 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fab841a7a20 0x7fab841a7e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:08.925 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:08.924 7fab8a705700 1 --2- 172.21.2.102:0/2388991566 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fab840af540 0x7fab840af940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:08.925 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:08.924 7fab89703700 1 -- 172.21.2.102:0/2388991566 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fab841a7a20 msgr2=0x7fab841a7e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:08.925 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:08.924 7fab89703700 1 --2- 172.21.2.102:0/2388991566 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fab841a7a20 0x7fab841a7e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:08.925 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:08.924 7fab89703700 1 -- 172.21.2.102:0/2388991566 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fab840af540 msgr2=0x7fab840af940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:08.925 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:08.924 7fab89703700 1 --2- 172.21.2.102:0/2388991566 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fab840af540 0x7fab840af940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:08.925 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:08.924 7fab89703700 1 -- 172.21.2.102:0/2388991566 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab840a4810 con 0x7fab840aeb20 2021-02-10T11:23:08.926 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:08.924 7fab89703700 1 --2- 172.21.2.102:0/2388991566 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fab840aeb20 0x7fab840aef20 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 rx=0x7fab7800b650 tx=0x7fab7800b9f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:08.926 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:08.924 7fab767fc700 1 -- 172.21.2.102:0/2388991566 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fab7800bde0 con 0x7fab840aeb20 2021-02-10T11:23:08.926 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:08.924 7fab767fc700 1 -- 172.21.2.102:0/2388991566 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab78006570 con 0x7fab840aeb20 2021-02-10T11:23:08.926 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:08.924 7fab8c0e8700 1 -- 172.21.2.102:0/2388991566 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab840a4810 con 0x7fab840aeb20 2021-02-10T11:23:08.926 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:08.924 7fab767fc700 1 -- 172.21.2.102:0/2388991566 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fab780022b0 con 0x7fab840aeb20 2021-02-10T11:23:08.926 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:08.925 7fab8c0e8700 1 -- 172.21.2.102:0/2388991566 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab840a4810 con 0x7fab840aeb20 2021-02-10T11:23:08.926 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:08.925 7fab767fc700 1 -- 172.21.2.102:0/2388991566 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fab78007000 con 0x7fab840aeb20 2021-02-10T11:23:08.926 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:08.925 7fab767fc700 1 --2- 172.21.2.102:0/2388991566 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fab6c018730 0x7fab6c01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:08.926 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:08.925 7fab767fc700 1 -- 172.21.2.102:0/2388991566 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 5659+0+0 (secure 0 0 0) 0x7fab7801f590 con 0x7fab840aeb20 2021-02-10T11:23:08.926 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:08.925 7fab89f04700 1 --2- 172.21.2.102:0/2388991566 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fab6c018730 0x7fab6c01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:08.927 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:08.926 7fab89f04700 1 --2- 172.21.2.102:0/2388991566 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fab6c018730 0x7fab6c01ac10 crc :-1 s=READY pgs=552 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:08.952 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:08.956 7fab8c0e8700 1 -- 172.21.2.102:0/2388991566 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab84103690 con 0x7fab840aeb20 2021-02-10T11:23:08.955 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:08.958 7fab767fc700 1 -- 172.21.2.102:0/2388991566 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fab78006570 con 0x7fab840aeb20 2021-02-10T11:23:09.082 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:09.085 7fab8c0e8700 1 -- 172.21.2.102:0/2388991566 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fab840f0b20 con 0x7fab840aeb20 2021-02-10T11:23:09.846 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:09.849 7fab767fc700 1 -- 172.21.2.102:0/2388991566 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v24) v1 ==== 104+0+0 (secure 0 0 0) 0x7fab780009b0 con 0x7fab840aeb20 2021-02-10T11:23:09.852 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:09.856 7fab8c0e8700 1 -- 172.21.2.102:0/2388991566 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fab6c018730 msgr2=0x7fab6c01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:09.853 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:09.856 7fab8c0e8700 1 --2- 172.21.2.102:0/2388991566 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fab6c018730 0x7fab6c01ac10 crc :-1 s=READY pgs=552 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:09.853 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:09.856 7fab8c0e8700 1 -- 172.21.2.102:0/2388991566 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fab840aeb20 msgr2=0x7fab840aef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:09.853 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:09.856 7fab8c0e8700 1 --2- 172.21.2.102:0/2388991566 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fab840aeb20 0x7fab840aef20 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 rx=0x7fab7800b650 tx=0x7fab7800b9f0).stop 2021-02-10T11:23:09.853 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:09.856 7fab8c0e8700 1 -- 172.21.2.102:0/2388991566 shutdown_connections 2021-02-10T11:23:09.853 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:09.856 7fab8c0e8700 1 --2- 172.21.2.102:0/2388991566 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fab6c018730 0x7fab6c01ac10 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:09.853 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:09.856 7fab8c0e8700 1 --2- 172.21.2.102:0/2388991566 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fab840af540 0x7fab840af940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:09.854 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:09.856 7fab8c0e8700 1 --2- 172.21.2.102:0/2388991566 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fab840aeb20 0x7fab840aef20 unknown :-1 s=CLOSED pgs=1159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:09.854 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:09.856 7fab8c0e8700 1 --2- 172.21.2.102:0/2388991566 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fab841a7a20 0x7fab841a7e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:09.854 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:09.856 7fab8c0e8700 1 -- 172.21.2.102:0/2388991566 >> 172.21.2.102:0/2388991566 conn(0x7fab840b5de0 msgr2=0x7fab840a6510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:09.854 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:09.856 7fab8c0e8700 1 -- 172.21.2.102:0/2388991566 shutdown_connections 2021-02-10T11:23:09.854 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:09.856 7fab8c0e8700 1 -- 172.21.2.102:0/2388991566 wait complete. 2021-02-10T11:23:09.862 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool delete cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2021-02-10T11:23:10.292 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:10.280 7ff172a3c700 1 Processor -- start 2021-02-10T11:23:10.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:10.280 7ff172a3c700 1 -- start start 2021-02-10T11:23:10.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:10.281 7ff172a3c700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff16c0d1c70 0x7ff16c0cef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:10.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:10.281 7ff172a3c700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7ff16c0cf580 0x7ff16c0cf980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:10.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:10.281 7ff172a3c700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff16c0fe1c0 con 0x7ff16c0d1870 2021-02-10T11:23:10.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:10.281 7ff172a3c700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff16c1b0cb0 con 0x7ff16c0cf580 2021-02-10T11:23:10.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:10.281 7ff172a3c700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff16c109a60 con 0x7ff16c0d1c70 2021-02-10T11:23:10.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:10.281 7ff171059700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7ff16c0cf580 0x7ff16c0cf980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:10.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:10.281 7ff171059700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7ff16c0cf580 0x7ff16c0cf980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:56056/0 (socket says 172.21.2.102:56056) 2021-02-10T11:23:10.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:10.281 7ff171059700 1 -- 172.21.2.102:0/1836491041 learned_addr learned my addr 172.21.2.102:0/1836491041 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:23:10.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:10.281 7ff171059700 1 -- 172.21.2.102:0/1836491041 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff16c0d1c70 msgr2=0x7ff16c0cef60 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:23:10.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:10.281 7ff171059700 1 --2- 172.21.2.102:0/1836491041 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff16c0d1c70 0x7ff16c0cef60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:10.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:10.281 7ff171059700 1 -- 172.21.2.102:0/1836491041 >> v1:172.21.2.113:6789/0 conn(0x7ff16c0d1870 legacy=0x7ff16c1b04c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:23:10.294 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:10.281 7ff171059700 1 -- 172.21.2.102:0/1836491041 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff16c088020 con 0x7ff16c0cf580 2021-02-10T11:23:10.294 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:10.281 7ff171059700 1 --2- 172.21.2.102:0/1836491041 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff16c0cf580 0x7ff16c0cf980 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 rx=0x7ff164006f00 tx=0x7ff164009f10).ready entity=mon.0 client_cookie=eaf39aff92bf4f7e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:10.294 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:10.281 7ff16b7fe700 1 -- 172.21.2.102:0/1836491041 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff16400b3b0 con 0x7ff16c0cf580 2021-02-10T11:23:10.294 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:10.281 7ff16b7fe700 1 -- 172.21.2.102:0/1836491041 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff16400b630 con 0x7ff16c0cf580 2021-02-10T11:23:10.294 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:10.281 7ff16b7fe700 1 -- 172.21.2.102:0/1836491041 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff16400bb50 con 0x7ff16c0cf580 2021-02-10T11:23:10.294 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:10.281 7ff172a3c700 1 -- 172.21.2.102:0/1836491041 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff16c0cf580 msgr2=0x7ff16c0cf980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:10.294 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:10.281 7ff172a3c700 1 --2- 172.21.2.102:0/1836491041 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff16c0cf580 0x7ff16c0cf980 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 rx=0x7ff164006f00 tx=0x7ff164009f10).stop 2021-02-10T11:23:10.294 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:10.282 7ff172a3c700 1 -- 172.21.2.102:0/1836491041 shutdown_connections 2021-02-10T11:23:10.294 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:10.282 7ff172a3c700 1 --2- 172.21.2.102:0/1836491041 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff16c0d1c70 0x7ff16c0cef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:10.294 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:10.282 7ff172a3c700 1 --2- 172.21.2.102:0/1836491041 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff16c0cf580 0x7ff16c0cf980 unknown :-1 s=CLOSED pgs=1160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:10.294 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:10.282 7ff172a3c700 1 -- 172.21.2.102:0/1836491041 >> 172.21.2.102:0/1836491041 conn(0x7ff16c0ce420 msgr2=0x7ff16c0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:10.294 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:10.282 7ff172a3c700 1 -- 172.21.2.102:0/1836491041 shutdown_connections 2021-02-10T11:23:10.295 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:10.282 7ff172a3c700 1 -- 172.21.2.102:0/1836491041 wait complete. 2021-02-10T11:23:10.296 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:10.296 7ff172a3c700 1 Processor -- start 2021-02-10T11:23:10.296 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:10.296 7ff172a3c700 1 -- start start 2021-02-10T11:23:10.296 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:10.297 7ff172a3c700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff16c191690 0x7ff16c099560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:10.296 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:10.297 7ff172a3c700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff16c187e70 0x7ff16c188270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:10.296 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:10.297 7ff172a3c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff16c188890 0x7ff16c189560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:10.296 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:10.297 7ff172a3c700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff16c099b80 con 0x7ff16c188890 2021-02-10T11:23:10.296 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:10.297 7ff172a3c700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff16c188c90 con 0x7ff16c187e70 2021-02-10T11:23:10.296 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:10.297 7ff172a3c700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff16c189b80 con 0x7ff16c191690 2021-02-10T11:23:10.296 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:10.297 7ff171059700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff16c188890 0x7ff16c189560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:10.297 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:10.297 7ff171059700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff16c188890 0x7ff16c189560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:56062/0 (socket says 172.21.2.102:56062) 2021-02-10T11:23:10.297 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:10.297 7ff171059700 1 -- 172.21.2.102:0/3025841290 learned_addr learned my addr 172.21.2.102:0/3025841290 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:23:10.297 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:10.297 7ff171059700 1 -- 172.21.2.102:0/3025841290 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff16c191690 msgr2=0x7ff16c099560 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:23:10.297 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:10.297 7ff171059700 1 --2- 172.21.2.102:0/3025841290 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff16c191690 0x7ff16c099560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:10.297 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:10.297 7ff171059700 1 -- 172.21.2.102:0/3025841290 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff16c187e70 msgr2=0x7ff16c188270 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:23:10.297 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:10.297 7ff171059700 1 --2- 172.21.2.102:0/3025841290 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff16c187e70 0x7ff16c188270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:10.297 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:10.297 7ff171059700 1 -- 172.21.2.102:0/3025841290 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff16c189da0 con 0x7ff16c188890 2021-02-10T11:23:10.297 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:10.297 7ff171059700 1 --2- 172.21.2.102:0/3025841290 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff16c188890 0x7ff16c189560 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 rx=0x7ff1640099f0 tx=0x7ff16400bcc0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:10.297 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:10.298 7ff1697fa700 1 -- 172.21.2.102:0/3025841290 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff16400be00 con 0x7ff16c188890 2021-02-10T11:23:10.297 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:10.298 7ff1697fa700 1 -- 172.21.2.102:0/3025841290 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff164005820 con 0x7ff16c188890 2021-02-10T11:23:10.297 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:10.298 7ff1697fa700 1 -- 172.21.2.102:0/3025841290 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff164005d40 con 0x7ff16c188890 2021-02-10T11:23:10.297 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:10.298 7ff172a3c700 1 -- 172.21.2.102:0/3025841290 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff16c189da0 con 0x7ff16c188890 2021-02-10T11:23:10.297 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:10.298 7ff172a3c700 1 -- 172.21.2.102:0/3025841290 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff16c189da0 con 0x7ff16c188890 2021-02-10T11:23:10.297 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:10.298 7ff1697fa700 1 -- 172.21.2.102:0/3025841290 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ff164005a00 con 0x7ff16c188890 2021-02-10T11:23:10.298 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:10.298 7ff1697fa700 1 --2- 172.21.2.102:0/3025841290 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff150018730 0x7ff15001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:10.298 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:10.299 7ff1697fa700 1 -- 172.21.2.102:0/3025841290 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 5659+0+0 (secure 0 0 0) 0x7ff164032b90 con 0x7ff16c188890 2021-02-10T11:23:10.298 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:10.299 7ff170858700 1 --2- 172.21.2.102:0/3025841290 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff150018730 0x7ff15001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:10.298 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:10.299 7ff170858700 1 --2- 172.21.2.102:0/3025841290 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff150018730 0x7ff15001abc0 crc :-1 s=READY pgs=553 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:10.327 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:10.330 7ff172a3c700 1 -- 172.21.2.102:0/3025841290 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff16c099b80 con 0x7ff16c188890 2021-02-10T11:23:10.329 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:10.333 7ff1697fa700 1 -- 172.21.2.102:0/3025841290 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ff16400be00 con 0x7ff16c188890 2021-02-10T11:23:10.455 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:10.459 7ff172a3c700 1 -- 172.21.2.102:0/3025841290 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"pool2": "cephfs_metadata", "prefix": "osd pool delete", "pool": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7ff16c099b80 con 0x7ff16c188890 2021-02-10T11:23:10.850 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:10.853 7ff1697fa700 1 -- 172.21.2.102:0/3025841290 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"pool2": "cephfs_metadata", "prefix": "osd pool delete", "pool": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v41) v1 ==== 185+0+0 (secure 0 0 0) 0x7ff1640059e0 con 0x7ff16c188890 2021-02-10T11:23:10.873 INFO:teuthology.orchestra.run.gibba002.stderr:pool 'cephfs_metadata' removed 2021-02-10T11:23:10.875 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:10.877 7ff172a3c700 1 -- 172.21.2.102:0/3025841290 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff150018730 msgr2=0x7ff15001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:10.875 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:10.877 7ff172a3c700 1 --2- 172.21.2.102:0/3025841290 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff150018730 0x7ff15001abc0 crc :-1 s=READY pgs=553 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:10.875 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:10.877 7ff172a3c700 1 -- 172.21.2.102:0/3025841290 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff16c188890 msgr2=0x7ff16c189560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:10.875 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:10.877 7ff172a3c700 1 --2- 172.21.2.102:0/3025841290 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff16c188890 0x7ff16c189560 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 rx=0x7ff1640099f0 tx=0x7ff16400bcc0).stop 2021-02-10T11:23:10.875 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:10.877 7ff172a3c700 1 -- 172.21.2.102:0/3025841290 shutdown_connections 2021-02-10T11:23:10.876 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:10.877 7ff172a3c700 1 --2- 172.21.2.102:0/3025841290 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff16c188890 0x7ff16c189560 unknown :-1 s=CLOSED pgs=1161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:10.876 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:10.877 7ff172a3c700 1 --2- 172.21.2.102:0/3025841290 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff150018730 0x7ff15001abc0 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:10.876 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:10.877 7ff172a3c700 1 --2- 172.21.2.102:0/3025841290 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff16c187e70 0x7ff16c188270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:10.876 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:10.877 7ff172a3c700 1 --2- 172.21.2.102:0/3025841290 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff16c191690 0x7ff16c099560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:10.876 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:10.877 7ff172a3c700 1 -- 172.21.2.102:0/3025841290 >> 172.21.2.102:0/3025841290 conn(0x7ff16c19f870 msgr2=0x7ff16c1a5f70 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:10.876 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:10.878 7ff172a3c700 1 -- 172.21.2.102:0/3025841290 shutdown_connections 2021-02-10T11:23:10.876 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:10.878 7ff172a3c700 1 -- 172.21.2.102:0/3025841290 wait complete. 2021-02-10T11:23:10.884 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool delete cephfs_data cephfs_data --yes-i-really-really-mean-it 2021-02-10T11:23:11.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:11.301 7fc111cc6700 1 Processor -- start 2021-02-10T11:23:11.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:11.301 7fc111cc6700 1 -- start start 2021-02-10T11:23:11.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:11.301 7fc111cc6700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc10c0d1c60 0x7fc10c0cef50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:11.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:11.301 7fc111cc6700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fc10c0cf570 0x7fc10c0cf970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:11.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:11.301 7fc111cc6700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc10c0fe1b0 con 0x7fc10c0d1860 2021-02-10T11:23:11.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:11.301 7fc111cc6700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc10c1b0ca0 con 0x7fc10c0cf570 2021-02-10T11:23:11.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:11.301 7fc111cc6700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc10c109a50 con 0x7fc10c0d1c60 2021-02-10T11:23:11.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:11.302 7fc10bfff700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fc10c0cf570 0x7fc10c0cf970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:11.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:11.302 7fc10affd700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc10c0d1c60 0x7fc10c0cef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:11.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:11.302 7fc10b7fe700 1 --1- >> v1:172.21.2.113:6789/0 conn(0x7fc10c0d1860 0x7fc10c1b04b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.113:6789/0 says I am v1:172.21.2.102:55614/0 (socket says 172.21.2.102:55614) 2021-02-10T11:23:11.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:11.302 7fc10b7fe700 1 -- 172.21.2.102:0/2170820300 learned_addr learned my addr 172.21.2.102:0/2170820300 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:23:11.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:11.303 7fc10bfff700 1 -- 172.21.2.102:0/2170820300 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc10c0d1c60 msgr2=0x7fc10c0cef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:11.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:11.303 7fc10bfff700 1 --2- 172.21.2.102:0/2170820300 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc10c0d1c60 0x7fc10c0cef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:11.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:11.303 7fc10bfff700 1 -- 172.21.2.102:0/2170820300 >> v1:172.21.2.113:6789/0 conn(0x7fc10c0d1860 legacy=0x7fc10c1b04b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:11.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:11.303 7fc10bfff700 1 -- 172.21.2.102:0/2170820300 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc10c088020 con 0x7fc10c0cf570 2021-02-10T11:23:11.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:11.303 7fc10affd700 1 --2- 172.21.2.102:0/2170820300 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc10c0d1c60 0x7fc10c0cef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:23:11.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:11.303 7fc10bfff700 1 --2- 172.21.2.102:0/2170820300 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc10c0cf570 0x7fc10c0cf970 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 rx=0x7fc104006e50 tx=0x7fc104009e70).ready entity=mon.1 client_cookie=5491f0caa0689761 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:11.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:11.304 7fc10a7fc700 1 -- 172.21.2.102:0/2170820300 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc10400b510 con 0x7fc10c0cf570 2021-02-10T11:23:11.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:11.304 7fc10a7fc700 1 -- 172.21.2.102:0/2170820300 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc104006790 con 0x7fc10c0cf570 2021-02-10T11:23:11.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:11.304 7fc10a7fc700 1 -- 172.21.2.102:0/2170820300 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc10400b940 con 0x7fc10c0cf570 2021-02-10T11:23:11.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:11.304 7fc111cc6700 1 -- 172.21.2.102:0/2170820300 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc10c0cf570 msgr2=0x7fc10c0cf970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:11.318 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:11.304 7fc111cc6700 1 --2- 172.21.2.102:0/2170820300 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc10c0cf570 0x7fc10c0cf970 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 rx=0x7fc104006e50 tx=0x7fc104009e70).stop 2021-02-10T11:23:11.318 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:11.304 7fc111cc6700 1 -- 172.21.2.102:0/2170820300 shutdown_connections 2021-02-10T11:23:11.318 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:11.304 7fc111cc6700 1 --2- 172.21.2.102:0/2170820300 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc10c0d1c60 0x7fc10c0cef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:11.318 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:11.304 7fc111cc6700 1 --2- 172.21.2.102:0/2170820300 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc10c0cf570 0x7fc10c0cf970 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:11.318 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:11.304 7fc111cc6700 1 -- 172.21.2.102:0/2170820300 >> 172.21.2.102:0/2170820300 conn(0x7fc10c0ce410 msgr2=0x7fc10c0d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:11.318 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:11.304 7fc111cc6700 1 -- 172.21.2.102:0/2170820300 shutdown_connections 2021-02-10T11:23:11.318 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:11.305 7fc111cc6700 1 -- 172.21.2.102:0/2170820300 wait complete. 2021-02-10T11:23:11.319 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:11.319 7fc111cc6700 1 Processor -- start 2021-02-10T11:23:11.320 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:11.319 7fc111cc6700 1 -- start start 2021-02-10T11:23:11.320 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:11.319 7fc111cc6700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc10c1a2920 0x7fc10c1a1c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:11.320 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:11.319 7fc111cc6700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc10c1a5810 0x7fc10c19a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:11.320 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:11.319 7fc111cc6700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc10c09d670 0x7fc10c09da70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:11.320 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:11.319 7fc111cc6700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc10c19a960 con 0x7fc10c09d670 2021-02-10T11:23:11.320 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:11.319 7fc111cc6700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc10c1a23d0 con 0x7fc10c1a2920 2021-02-10T11:23:11.320 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:11.319 7fc111cc6700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc10c1a5370 con 0x7fc10c1a5810 2021-02-10T11:23:11.320 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:11.319 7fc10bfff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc10c09d670 0x7fc10c09da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:11.320 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:11.319 7fc10bfff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc10c09d670 0x7fc10c09da70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:56072/0 (socket says 172.21.2.102:56072) 2021-02-10T11:23:11.320 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:11.319 7fc10bfff700 1 -- 172.21.2.102:0/3630353512 learned_addr learned my addr 172.21.2.102:0/3630353512 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:23:11.320 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:11.320 7fc10bfff700 1 -- 172.21.2.102:0/3630353512 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc10c1a5810 msgr2=0x7fc10c19a2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:11.320 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:11.320 7fc10bfff700 1 --2- 172.21.2.102:0/3630353512 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc10c1a5810 0x7fc10c19a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:11.320 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:11.320 7fc10bfff700 1 -- 172.21.2.102:0/3630353512 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc10c1a2920 msgr2=0x7fc10c1a1c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:11.321 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:11.320 7fc10bfff700 1 --2- 172.21.2.102:0/3630353512 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc10c1a2920 0x7fc10c1a1c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:11.321 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:11.320 7fc10bfff700 1 -- 172.21.2.102:0/3630353512 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc10c1a5c10 con 0x7fc10c09d670 2021-02-10T11:23:11.321 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:11.320 7fc10bfff700 1 --2- 172.21.2.102:0/3630353512 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc10c09d670 0x7fc10c09da70 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 rx=0x7fc104009a90 tx=0x7fc10400bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:11.321 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:11.320 7fc103fff700 1 -- 172.21.2.102:0/3630353512 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc104002f10 con 0x7fc10c09d670 2021-02-10T11:23:11.321 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:11.320 7fc103fff700 1 -- 172.21.2.102:0/3630353512 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc104009290 con 0x7fc10c09d670 2021-02-10T11:23:11.321 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:11.320 7fc103fff700 1 -- 172.21.2.102:0/3630353512 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc10400c4e0 con 0x7fc10c09d670 2021-02-10T11:23:11.321 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:11.320 7fc111cc6700 1 -- 172.21.2.102:0/3630353512 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc10c1a5c10 con 0x7fc10c09d670 2021-02-10T11:23:11.321 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:11.320 7fc111cc6700 1 -- 172.21.2.102:0/3630353512 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc10c09e380 con 0x7fc10c09d670 2021-02-10T11:23:11.321 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:11.321 7fc103fff700 1 -- 172.21.2.102:0/3630353512 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fc1040056b0 con 0x7fc10c09d670 2021-02-10T11:23:11.321 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:11.321 7fc103fff700 1 --2- 172.21.2.102:0/3630353512 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc0f00186c0 0x7fc0f001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:11.321 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:11.321 7fc103fff700 1 -- 172.21.2.102:0/3630353512 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 5221+0+0 (secure 0 0 0) 0x7fc104032600 con 0x7fc10c09d670 2021-02-10T11:23:11.321 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:11.321 7fc10b7fe700 1 --2- 172.21.2.102:0/3630353512 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc0f00186c0 0x7fc0f001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:11.321 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:11.321 7fc10b7fe700 1 --2- 172.21.2.102:0/3630353512 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc0f00186c0 0x7fc0f001abc0 crc :-1 s=READY pgs=554 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:11.349 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:11.351 7fc111cc6700 1 -- 172.21.2.102:0/3630353512 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc10c0fe1b0 con 0x7fc10c09d670 2021-02-10T11:23:11.352 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:11.354 7fc103fff700 1 -- 172.21.2.102:0/3630353512 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fc10400be30 con 0x7fc10c09d670 2021-02-10T11:23:11.479 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:11.481 7fc111cc6700 1 -- 172.21.2.102:0/3630353512 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"pool2": "cephfs_data", "prefix": "osd pool delete", "pool": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fc10c0fe1b0 con 0x7fc10c09d670 2021-02-10T11:23:11.857 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:11.859 7fc103fff700 1 -- 172.21.2.102:0/3630353512 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"pool2": "cephfs_data", "prefix": "osd pool delete", "pool": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v42) v1 ==== 173+0+0 (secure 0 0 0) 0x7fc1040056b0 con 0x7fc10c09d670 2021-02-10T11:23:11.896 INFO:teuthology.orchestra.run.gibba002.stderr:pool 'cephfs_data' removed 2021-02-10T11:23:11.898 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:11.900 7fc111cc6700 1 -- 172.21.2.102:0/3630353512 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc0f00186c0 msgr2=0x7fc0f001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:11.898 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:11.900 7fc111cc6700 1 --2- 172.21.2.102:0/3630353512 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc0f00186c0 0x7fc0f001abc0 crc :-1 s=READY pgs=554 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:11.898 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:11.900 7fc111cc6700 1 -- 172.21.2.102:0/3630353512 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc10c09d670 msgr2=0x7fc10c09da70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:11.898 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:11.900 7fc111cc6700 1 --2- 172.21.2.102:0/3630353512 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc10c09d670 0x7fc10c09da70 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 rx=0x7fc104009a90 tx=0x7fc10400bcf0).stop 2021-02-10T11:23:11.898 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:11.900 7fc111cc6700 1 -- 172.21.2.102:0/3630353512 shutdown_connections 2021-02-10T11:23:11.898 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:11.900 7fc111cc6700 1 --2- 172.21.2.102:0/3630353512 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc10c09d670 0x7fc10c09da70 unknown :-1 s=CLOSED pgs=1162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:11.899 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:11.900 7fc111cc6700 1 --2- 172.21.2.102:0/3630353512 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc10c1a5810 0x7fc10c19a2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:11.899 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:11.900 7fc111cc6700 1 --2- 172.21.2.102:0/3630353512 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc0f00186c0 0x7fc0f001abc0 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:11.899 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:11.900 7fc111cc6700 1 --2- 172.21.2.102:0/3630353512 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc10c1a2920 0x7fc10c1a1c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:11.899 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:11.900 7fc111cc6700 1 -- 172.21.2.102:0/3630353512 >> 172.21.2.102:0/3630353512 conn(0x7fc10c094ea0 msgr2=0x7fc10c19ace0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:11.899 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:11.901 7fc111cc6700 1 -- 172.21.2.102:0/3630353512 shutdown_connections 2021-02-10T11:23:11.899 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:11.901 7fc111cc6700 1 -- 172.21.2.102:0/3630353512 wait complete. 2021-02-10T11:23:11.907 INFO:tasks.ceph.mds.a:Restarting daemon 2021-02-10T11:23:11.907 INFO:tasks.ceph.mds.a:Stopping old one... 2021-02-10T11:23:11.907 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2021-02-10T11:23:11.907 INFO:teuthology.orchestra.run:waiting for 300 2021-02-10T11:23:11.907 INFO:tasks.ceph.mds.b:Restarting daemon 2021-02-10T11:23:11.907 INFO:tasks.ceph.mds.b:Stopping old one... 2021-02-10T11:23:11.908 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2021-02-10T11:23:11.908 INFO:teuthology.orchestra.run:waiting for 300 2021-02-10T11:23:11.913 INFO:tasks.ceph.mds.a:Stopped 2021-02-10T11:23:11.913 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2021-02-10T11:23:11.914 INFO:tasks.ceph.mds.b:Stopped 2021-02-10T11:23:11.914 DEBUG:teuthology.orchestra.run.gibba013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2021-02-10T11:23:11.955 INFO:tasks.ceph.mds.a:Started 2021-02-10T11:23:11.956 DEBUG:teuthology.parallel:result is None 2021-02-10T11:23:11.956 INFO:tasks.ceph.mds.b:Started 2021-02-10T11:23:11.956 DEBUG:teuthology.parallel:result is None 2021-02-10T11:23:11.957 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd blacklist clear 2021-02-10T11:23:12.360 INFO:tasks.ceph.mds.b.gibba013.stdout:starting mds.b at 2021-02-10T11:23:12.363 INFO:tasks.ceph.mds.a.gibba002.stdout:starting mds.a at 2021-02-10T11:23:12.387 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:12.372 7fba656de700 1 Processor -- start 2021-02-10T11:23:12.387 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:12.372 7fba656de700 1 -- start start 2021-02-10T11:23:12.387 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:12.373 7fba656de700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fba6007e6a0 0x7fba600d1790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:12.387 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:12.373 7fba656de700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba6007ed80 con 0x7fba600e5470 2021-02-10T11:23:12.387 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:12.373 7fba656de700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba6007eb10 con 0x7fba600e5070 2021-02-10T11:23:12.387 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:12.373 7fba656de700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba601b4010 con 0x7fba6007e6a0 2021-02-10T11:23:12.387 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:12.373 7fba5eabe700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fba600e5470 0x7fba6007f300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:48318/0 (socket says 172.21.2.102:48318) 2021-02-10T11:23:12.387 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:12.373 7fba5eabe700 1 -- 172.21.2.102:0/65417039 learned_addr learned my addr 172.21.2.102:0/65417039 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:23:12.387 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:12.373 7fba5fac0700 1 --2- 172.21.2.102:0/65417039 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fba6007e6a0 0x7fba600d1790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:12.387 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:12.373 7fba5e2bd700 1 -- 172.21.2.102:0/65417039 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1147724482 0 0) 0x7fba50001180 con 0x7fba600e5470 2021-02-10T11:23:12.388 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:12.374 7fba5e2bd700 1 -- 172.21.2.102:0/65417039 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba4c002fe0 con 0x7fba600e5470 2021-02-10T11:23:12.388 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:12.374 7fba5fac0700 1 -- 172.21.2.102:0/65417039 >> v1:172.21.2.113:6789/0 conn(0x7fba600e5070 legacy=0x7fba601b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:12.388 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:12.374 7fba5fac0700 1 -- 172.21.2.102:0/65417039 >> v1:172.21.2.102:6789/0 conn(0x7fba600e5470 legacy=0x7fba6007f300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:12.388 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:12.374 7fba5fac0700 1 -- 172.21.2.102:0/65417039 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba600d1db0 con 0x7fba6007e6a0 2021-02-10T11:23:12.388 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:12.374 7fba5fac0700 1 --2- 172.21.2.102:0/65417039 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fba6007e6a0 0x7fba600d1790 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7fba58006e50 tx=0x7fba58009e70).ready entity=mon.2 client_cookie=3ee4e4d14f44ef1 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:12.388 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:12.374 7fba5e2bd700 1 -- 172.21.2.102:0/65417039 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fba58006a20 con 0x7fba6007e6a0 2021-02-10T11:23:12.388 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:12.374 7fba5e2bd700 1 -- 172.21.2.102:0/65417039 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba58006790 con 0x7fba6007e6a0 2021-02-10T11:23:12.388 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:12.374 7fba5e2bd700 1 -- 172.21.2.102:0/65417039 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fba5800b700 con 0x7fba6007e6a0 2021-02-10T11:23:12.388 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:12.374 7fba656de700 1 -- 172.21.2.102:0/65417039 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fba6007e6a0 msgr2=0x7fba600d1790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:12.388 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:12.374 7fba656de700 1 --2- 172.21.2.102:0/65417039 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fba6007e6a0 0x7fba600d1790 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7fba58006e50 tx=0x7fba58009e70).stop 2021-02-10T11:23:12.388 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:12.375 7fba656de700 1 -- 172.21.2.102:0/65417039 shutdown_connections 2021-02-10T11:23:12.388 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:12.375 7fba656de700 1 --2- 172.21.2.102:0/65417039 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fba6007e6a0 0x7fba600d1790 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:12.389 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:12.375 7fba656de700 1 -- 172.21.2.102:0/65417039 >> 172.21.2.102:0/65417039 conn(0x7fba600ce690 msgr2=0x7fba600e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:12.389 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:12.375 7fba656de700 1 -- 172.21.2.102:0/65417039 shutdown_connections 2021-02-10T11:23:12.389 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:12.375 7fba656de700 1 -- 172.21.2.102:0/65417039 wait complete. 2021-02-10T11:23:12.390 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:12.390 7fba656de700 1 Processor -- start 2021-02-10T11:23:12.390 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:12.390 7fba656de700 1 -- start start 2021-02-10T11:23:12.390 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:12.390 7fba656de700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fba6009b070 0x7fba600ad190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:12.390 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:12.390 7fba656de700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fba601a1a70 0x7fba601a1e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:12.391 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:12.390 7fba656de700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fba601a2490 0x7fba600abce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:12.391 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:12.390 7fba656de700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba601a2910 con 0x7fba6009b070 2021-02-10T11:23:12.391 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:12.390 7fba656de700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba600ad7b0 con 0x7fba601a1a70 2021-02-10T11:23:12.391 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:12.390 7fba656de700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba600ac300 con 0x7fba601a2490 2021-02-10T11:23:12.391 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:12.390 7fba5f2bf700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fba6009b070 0x7fba600ad190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:12.391 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:12.390 7fba5f2bf700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fba6009b070 0x7fba600ad190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:56100/0 (socket says 172.21.2.102:56100) 2021-02-10T11:23:12.391 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:12.390 7fba5f2bf700 1 -- 172.21.2.102:0/685969448 learned_addr learned my addr 172.21.2.102:0/685969448 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:23:12.391 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:12.391 7fba5f2bf700 1 -- 172.21.2.102:0/685969448 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fba601a2490 msgr2=0x7fba600abce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:12.391 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:12.391 7fba5f2bf700 1 --2- 172.21.2.102:0/685969448 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fba601a2490 0x7fba600abce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:12.391 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:12.391 7fba5f2bf700 1 -- 172.21.2.102:0/685969448 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fba601a1a70 msgr2=0x7fba601a1e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:12.391 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:12.391 7fba5f2bf700 1 --2- 172.21.2.102:0/685969448 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fba601a1a70 0x7fba601a1e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:12.391 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:12.391 7fba5f2bf700 1 -- 172.21.2.102:0/685969448 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba600ac520 con 0x7fba6009b070 2021-02-10T11:23:12.391 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:12.391 7fba5f2bf700 1 --2- 172.21.2.102:0/685969448 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fba6009b070 0x7fba600ad190 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 rx=0x7fba48006f50 tx=0x7fba4800b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:12.392 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:12.391 7fba57fff700 1 -- 172.21.2.102:0/685969448 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fba4800bd50 con 0x7fba6009b070 2021-02-10T11:23:12.392 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:12.391 7fba57fff700 1 -- 172.21.2.102:0/685969448 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba480080d0 con 0x7fba6009b070 2021-02-10T11:23:12.392 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:12.391 7fba57fff700 1 -- 172.21.2.102:0/685969448 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fba48008520 con 0x7fba6009b070 2021-02-10T11:23:12.392 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:12.391 7fba656de700 1 -- 172.21.2.102:0/685969448 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba600ac520 con 0x7fba6009b070 2021-02-10T11:23:12.392 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:12.391 7fba656de700 1 -- 172.21.2.102:0/685969448 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba600acab0 con 0x7fba6009b070 2021-02-10T11:23:12.392 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:12.392 7fba57fff700 1 -- 172.21.2.102:0/685969448 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fba480082b0 con 0x7fba6009b070 2021-02-10T11:23:12.392 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:12.392 7fba57fff700 1 --2- 172.21.2.102:0/685969448 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fba440186c0 0x7fba4401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:12.392 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:12.392 7fba57fff700 1 -- 172.21.2.102:0/685969448 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 4839+0+0 (secure 0 0 0) 0x7fba48032220 con 0x7fba6009b070 2021-02-10T11:23:12.392 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:12.392 7fba5eabe700 1 --2- 172.21.2.102:0/685969448 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fba440186c0 0x7fba4401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:12.392 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:12.392 7fba5eabe700 1 --2- 172.21.2.102:0/685969448 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fba440186c0 0x7fba4401abc0 crc :-1 s=READY pgs=555 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:12.419 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:12.421 7fba656de700 1 -- 172.21.2.102:0/685969448 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba6007ed80 con 0x7fba6009b070 2021-02-10T11:23:12.421 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:12.424 7fba57fff700 1 -- 172.21.2.102:0/685969448 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fba48026f00 con 0x7fba6009b070 2021-02-10T11:23:12.560 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:12.562 7fba656de700 1 -- 172.21.2.102:0/685969448 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd blacklist clear"} v 0) v1 -- 0x7fba6007ed80 con 0x7fba6009b070 2021-02-10T11:23:12.900 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:12.902 7fba57fff700 1 -- 172.21.2.102:0/685969448 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd blacklist clear"}]=0 removed all blacklist entries v43) v1 ==== 97+0+0 (secure 0 0 0) 0x7fba48026f00 con 0x7fba6009b070 2021-02-10T11:23:12.927 INFO:teuthology.orchestra.run.gibba002.stderr: removed all blacklist entries 2021-02-10T11:23:12.929 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:12.931 7fba656de700 1 -- 172.21.2.102:0/685969448 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fba440186c0 msgr2=0x7fba4401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:12.929 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:12.931 7fba656de700 1 --2- 172.21.2.102:0/685969448 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fba440186c0 0x7fba4401abc0 crc :-1 s=READY pgs=555 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:12.929 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:12.931 7fba656de700 1 -- 172.21.2.102:0/685969448 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fba6009b070 msgr2=0x7fba600ad190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:12.929 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:12.931 7fba656de700 1 --2- 172.21.2.102:0/685969448 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fba6009b070 0x7fba600ad190 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 rx=0x7fba48006f50 tx=0x7fba4800b960).stop 2021-02-10T11:23:12.929 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:12.931 7fba656de700 1 -- 172.21.2.102:0/685969448 shutdown_connections 2021-02-10T11:23:12.930 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:12.931 7fba656de700 1 --2- 172.21.2.102:0/685969448 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fba601a2490 0x7fba600abce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:12.930 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:12.931 7fba656de700 1 --2- 172.21.2.102:0/685969448 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fba440186c0 0x7fba4401abc0 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:12.930 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:12.931 7fba656de700 1 --2- 172.21.2.102:0/685969448 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fba601a1a70 0x7fba601a1e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:12.930 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:12.931 7fba656de700 1 --2- 172.21.2.102:0/685969448 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fba6009b070 0x7fba600ad190 unknown :-1 s=CLOSED pgs=1167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:12.930 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:12.931 7fba656de700 1 -- 172.21.2.102:0/685969448 >> 172.21.2.102:0/685969448 conn(0x7fba60085960 msgr2=0x7fba600a1c70 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:12.930 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:12.932 7fba656de700 1 -- 172.21.2.102:0/685969448 shutdown_connections 2021-02-10T11:23:12.930 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:12.932 7fba656de700 1 -- 172.21.2.102:0/685969448 wait complete. 2021-02-10T11:23:12.938 INFO:tasks.cephfs.cephfs_test_case:['0', '1', '2', '3'] 2021-02-10T11:23:12.938 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth ls --format=json-pretty 2021-02-10T11:23:13.364 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:13.350 7f1ac0c03700 1 Processor -- start 2021-02-10T11:23:13.364 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:13.350 7f1ac0c03700 1 -- start start 2021-02-10T11:23:13.364 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:13.350 7f1ac0c03700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f1abc0e5070 0x7f1abc1b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:13.364 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:13.350 7f1ac0c03700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f1abc0e5470 0x7f1abc080310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:13.364 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:13.350 7f1ac0c03700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1abc07ed80 con 0x7f1abc1b3e40 2021-02-10T11:23:13.364 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:13.350 7f1ac0c03700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1abc0d1790 con 0x7f1abc0e5070 2021-02-10T11:23:13.364 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:13.350 7f1ac0c03700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1abc0d19b0 con 0x7f1abc0e5470 2021-02-10T11:23:13.364 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:13.350 7f1aba85e700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f1abc0e5070 0x7f1abc1b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:13.365 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:13.350 7f1aba85e700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f1abc0e5070 0x7f1abc1b3820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:56106/0 (socket says 172.21.2.102:56106) 2021-02-10T11:23:13.365 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:13.350 7f1aba85e700 1 -- 172.21.2.102:0/822571233 learned_addr learned my addr 172.21.2.102:0/822571233 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:23:13.365 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:13.350 7f1abb05f700 1 --2- 172.21.2.102:0/822571233 >> v2:172.21.2.113:3300/0 conn(0x7f1abc0e5470 0x7f1abc080310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:13.365 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:13.351 7f1aba85e700 1 -- 172.21.2.102:0/822571233 >> v2:172.21.2.113:3300/0 conn(0x7f1abc0e5470 msgr2=0x7f1abc080310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:13.365 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:13.351 7f1aba85e700 1 --2- 172.21.2.102:0/822571233 >> v2:172.21.2.113:3300/0 conn(0x7f1abc0e5470 0x7f1abc080310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:13.365 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:13.351 7f1aba85e700 1 -- 172.21.2.102:0/822571233 >> v1:172.21.2.113:6789/0 conn(0x7f1abc1b3e40 legacy=0x7f1abc07d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:13.365 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:13.351 7f1aba85e700 1 -- 172.21.2.102:0/822571233 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1abc0d1bd0 con 0x7f1abc0e5070 2021-02-10T11:23:13.365 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:13.351 7f1aba85e700 1 --2- 172.21.2.102:0/822571233 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1abc0e5070 0x7f1abc1b3820 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 rx=0x7f1aa4006700 tx=0x7f1aa4009b80).ready entity=mon.0 client_cookie=db59fe36f51d1263 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:13.365 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:13.351 7f1aba05d700 1 -- 172.21.2.102:0/822571233 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1aa400c0c0 con 0x7f1abc0e5070 2021-02-10T11:23:13.365 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:13.351 7f1aba05d700 1 -- 172.21.2.102:0/822571233 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1aa400c440 con 0x7f1abc0e5070 2021-02-10T11:23:13.365 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:13.351 7f1aba05d700 1 -- 172.21.2.102:0/822571233 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1aa400c8a0 con 0x7f1abc0e5070 2021-02-10T11:23:13.365 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:13.352 7f1ac0c03700 1 -- 172.21.2.102:0/822571233 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1abc0e5070 msgr2=0x7f1abc1b3820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:13.365 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:13.352 7f1ac0c03700 1 --2- 172.21.2.102:0/822571233 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1abc0e5070 0x7f1abc1b3820 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 rx=0x7f1aa4006700 tx=0x7f1aa4009b80).stop 2021-02-10T11:23:13.366 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:13.352 7f1ac0c03700 1 -- 172.21.2.102:0/822571233 shutdown_connections 2021-02-10T11:23:13.366 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:13.352 7f1ac0c03700 1 --2- 172.21.2.102:0/822571233 >> v2:172.21.2.113:3300/0 conn(0x7f1abc0e5470 0x7f1abc080310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:13.366 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:13.352 7f1ac0c03700 1 --2- 172.21.2.102:0/822571233 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1abc0e5070 0x7f1abc1b3820 unknown :-1 s=CLOSED pgs=1168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:13.366 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:13.352 7f1ac0c03700 1 -- 172.21.2.102:0/822571233 >> 172.21.2.102:0/822571233 conn(0x7f1abc0ce690 msgr2=0x7f1abc0e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:13.366 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:13.352 7f1ac0c03700 1 -- 172.21.2.102:0/822571233 shutdown_connections 2021-02-10T11:23:13.366 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:13.352 7f1ac0c03700 1 -- 172.21.2.102:0/822571233 wait complete. 2021-02-10T11:23:13.368 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:13.367 7f1ac0c03700 1 Processor -- start 2021-02-10T11:23:13.368 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:13.367 7f1ac0c03700 1 -- start start 2021-02-10T11:23:13.368 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:13.367 7f1ac0c03700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1abc08d400 0x7f1abc081230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:13.368 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:13.367 7f1ac0c03700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1abc081850 0x7f1abc0acf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:13.368 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:13.367 7f1ac0c03700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1abc081c50 0x7f1abc0ad5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:13.368 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:13.368 7f1ac0c03700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1abc082110 con 0x7f1abc081c50 2021-02-10T11:23:13.368 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:13.368 7f1ac0c03700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1abc08d800 con 0x7f1abc081850 2021-02-10T11:23:13.368 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:13.368 7f1ac0c03700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1abc0adbc0 con 0x7f1abc08d400 2021-02-10T11:23:13.368 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:13.368 7f1abb05f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1abc081c50 0x7f1abc0ad5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:13.368 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:13.368 7f1abb05f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1abc081c50 0x7f1abc0ad5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:56114/0 (socket says 172.21.2.102:56114) 2021-02-10T11:23:13.368 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:13.368 7f1abb05f700 1 -- 172.21.2.102:0/2111572423 learned_addr learned my addr 172.21.2.102:0/2111572423 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:23:13.368 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:13.368 7f1aba85e700 1 --2- 172.21.2.102:0/2111572423 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1abc08d400 0x7f1abc081230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:13.368 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:13.368 7f1ab205d700 1 --2- 172.21.2.102:0/2111572423 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1abc081850 0x7f1abc0acf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:13.369 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:13.368 7f1abb05f700 1 -- 172.21.2.102:0/2111572423 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1abc08d400 msgr2=0x7f1abc081230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:13.369 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:13.368 7f1abb05f700 1 --2- 172.21.2.102:0/2111572423 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1abc08d400 0x7f1abc081230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:13.369 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:13.368 7f1abb05f700 1 -- 172.21.2.102:0/2111572423 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1abc081850 msgr2=0x7f1abc0acf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:13.369 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:13.368 7f1abb05f700 1 --2- 172.21.2.102:0/2111572423 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1abc081850 0x7f1abc0acf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:13.369 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:13.368 7f1abb05f700 1 -- 172.21.2.102:0/2111572423 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1abc187fc0 con 0x7f1abc081c50 2021-02-10T11:23:13.369 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:13.368 7f1abb05f700 1 --2- 172.21.2.102:0/2111572423 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1abc081c50 0x7f1abc0ad5a0 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 rx=0x7f1ab400b910 tx=0x7f1ab400bcb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:13.369 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:13.369 7f1ab3fff700 1 -- 172.21.2.102:0/2111572423 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1ab4006aa0 con 0x7f1abc081c50 2021-02-10T11:23:13.369 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:13.369 7f1ab3fff700 1 -- 172.21.2.102:0/2111572423 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ab4003030 con 0x7f1abc081c50 2021-02-10T11:23:13.369 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:13.369 7f1ab3fff700 1 -- 172.21.2.102:0/2111572423 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1ab40013e0 con 0x7f1abc081c50 2021-02-10T11:23:13.369 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:13.369 7f1ac0c03700 1 -- 172.21.2.102:0/2111572423 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1abc187fc0 con 0x7f1abc081c50 2021-02-10T11:23:13.369 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:13.369 7f1ac0c03700 1 -- 172.21.2.102:0/2111572423 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1abc188550 con 0x7f1abc081c50 2021-02-10T11:23:13.369 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:13.370 7f1ab3fff700 1 -- 172.21.2.102:0/2111572423 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1ab4006aa0 con 0x7f1abc081c50 2021-02-10T11:23:13.369 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:13.370 7f1ab3fff700 1 --2- 172.21.2.102:0/2111572423 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1aa00186c0 0x7f1aa001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:13.370 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:13.370 7f1ab3fff700 1 -- 172.21.2.102:0/2111572423 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 4753+0+0 (secure 0 0 0) 0x7f1ab4032610 con 0x7f1abc081c50 2021-02-10T11:23:13.370 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:13.370 7f1aba85e700 1 --2- 172.21.2.102:0/2111572423 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1aa00186c0 0x7f1aa001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:13.370 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:13.370 7f1aba85e700 1 --2- 172.21.2.102:0/2111572423 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1aa00186c0 0x7f1aa001abc0 crc :-1 s=READY pgs=556 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:13.398 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:13.400 7f1ac0c03700 1 -- 172.21.2.102:0/2111572423 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1abc07ed80 con 0x7f1abc081c50 2021-02-10T11:23:13.401 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:13.403 7f1ab3fff700 1 -- 172.21.2.102:0/2111572423 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f1ab401b660 con 0x7f1abc081c50 2021-02-10T11:23:13.533 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:13.536 7f1ac0c03700 1 -- 172.21.2.102:0/2111572423 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f1abc07ed80 con 0x7f1abc081c50 2021-02-10T11:23:13.534 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:13.536 7f1ab3fff700 1 -- 172.21.2.102:0/2111572423 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v22) v1 ==== 80+0+4513 (secure 0 0 0) 0x7f1ab401b660 con 0x7f1abc081c50 2021-02-10T11:23:13.535 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:23:13.535 INFO:teuthology.orchestra.run.gibba002.stdout:{ 2021-02-10T11:23:13.535 INFO:teuthology.orchestra.run.gibba002.stdout: "auth_dump": [ 2021-02-10T11:23:13.535 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:13.535 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "mds.a", 2021-02-10T11:23:13.535 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAevSNgsgyEARAAvr0cUjjCoEHLxaZSZ2HtVA==", 2021-02-10T11:23:13.535 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:23:13.535 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow", 2021-02-10T11:23:13.535 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:23:13.535 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:23:13.535 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:23:13.536 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:23:13.536 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:13.536 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:13.536 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "mds.b", 2021-02-10T11:23:13.536 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAevSNgwOFMMRAALfb1fbX56ao1TU8ClnTW1A==", 2021-02-10T11:23:13.536 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:23:13.536 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow", 2021-02-10T11:23:13.536 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:23:13.536 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:23:13.536 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:23:13.536 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:23:13.536 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:13.537 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:13.537 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.0", 2021-02-10T11:23:13.537 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAmvSNgj+bMMBAAD3HUD6fLiMAg+++iNzLJKQ==", 2021-02-10T11:23:13.537 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:23:13.537 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:23:13.537 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:23:13.537 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:23:13.537 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:23:13.537 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:13.537 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:13.537 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.1", 2021-02-10T11:23:13.537 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQApvSNgnvc+MRAAAOxqlNwiyLHqNITOrYeFKQ==", 2021-02-10T11:23:13.538 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:23:13.538 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:23:13.538 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:23:13.538 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:23:13.538 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:23:13.538 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:13.538 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:13.538 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.2", 2021-02-10T11:23:13.538 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAsvSNgZ4SbMRAA76IOYWfbiAuBoM/4k81GxA==", 2021-02-10T11:23:13.538 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:23:13.538 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:23:13.539 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:23:13.539 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:23:13.539 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:23:13.539 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:13.539 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:13.539 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.3", 2021-02-10T11:23:13.539 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAvvSNgFk3rMRAAWCmKKyw0LDQ+UlAHZAITOw==", 2021-02-10T11:23:13.540 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:23:13.540 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:23:13.540 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:23:13.540 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:23:13.540 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:23:13.540 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:13.540 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:13.540 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.4", 2021-02-10T11:23:13.540 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQA4vSNgpKcNJBAApJMmvj5YxVHPDOmvmQONOg==", 2021-02-10T11:23:13.540 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:23:13.540 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:23:13.540 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:23:13.541 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:23:13.541 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:23:13.541 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:13.541 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:13.541 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.5", 2021-02-10T11:23:13.541 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQA7vSNgf2ujJBAAfkuYpyb3WTzc8V5kw+Wktg==", 2021-02-10T11:23:13.541 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:23:13.541 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:23:13.541 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:23:13.541 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:23:13.541 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:23:13.541 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:13.541 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:13.542 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.6", 2021-02-10T11:23:13.542 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQA+vSNgqw9uJRAAFMU0R+UxU9C3o/9byb3e0g==", 2021-02-10T11:23:13.542 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:23:13.542 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:23:13.542 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:23:13.542 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:23:13.542 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:23:13.542 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:13.542 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:13.542 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.7", 2021-02-10T11:23:13.542 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQBBvSNgvWSrJRAA2KaCHmLpzovRgYzLvyofTQ==", 2021-02-10T11:23:13.542 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:23:13.543 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:23:13.543 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:23:13.543 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:23:13.543 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:23:13.543 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:13.543 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:13.543 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "client.0", 2021-02-10T11:23:13.544 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAfvSNgZd9XIBAAvyrvl8fdKuHnJZfczfRoFw==", 2021-02-10T11:23:13.544 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:23:13.544 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow", 2021-02-10T11:23:13.544 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow r", 2021-02-10T11:23:13.544 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:23:13.544 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:23:13.544 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:13.544 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:13.545 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "client.1", 2021-02-10T11:23:13.545 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAfvSNgSyfXNhAAIduPnF7Vd/6HSkJ0MlcMSA==", 2021-02-10T11:23:13.545 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:23:13.545 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow", 2021-02-10T11:23:13.545 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow r", 2021-02-10T11:23:13.545 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:23:13.545 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:23:13.545 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:13.546 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:13.546 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "client.2", 2021-02-10T11:23:13.546 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAgvSNgiIG/ERAA9ZTamaTI7ZsPSnxBqjplCg==", 2021-02-10T11:23:13.546 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:23:13.546 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow", 2021-02-10T11:23:13.546 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow r", 2021-02-10T11:23:13.546 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:23:13.546 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:23:13.547 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:13.547 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:13.547 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "client.3", 2021-02-10T11:23:13.547 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAgvSNgOWpOKBAAlXsEB04Q/GvtL06PqRMLeA==", 2021-02-10T11:23:13.547 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:23:13.547 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow", 2021-02-10T11:23:13.547 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow r", 2021-02-10T11:23:13.548 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:23:13.548 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:23:13.548 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:13.548 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:13.548 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "client.admin", 2021-02-10T11:23:13.548 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAavSNgO3GQBhAAL+NVnYskQRKVWLgyDPY/xw==", 2021-02-10T11:23:13.548 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:23:13.549 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow *", 2021-02-10T11:23:13.549 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:23:13.549 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:23:13.549 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:23:13.549 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:23:13.549 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:13.550 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:13.550 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "mgr.x", 2021-02-10T11:23:13.550 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAdvSNgXZZgJhAA7dtBSGzH+DsBwnddZpOBJA==", 2021-02-10T11:23:13.550 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:23:13.550 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow *", 2021-02-10T11:23:13.550 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow profile mgr", 2021-02-10T11:23:13.550 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:23:13.551 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:23:13.551 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:13.551 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:13.551 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "mgr.y", 2021-02-10T11:23:13.551 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAdvSNgVqQFDRAAwo6YEdWZCAr5GdOWVL/Qvg==", 2021-02-10T11:23:13.551 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:23:13.552 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow *", 2021-02-10T11:23:13.552 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow profile mgr", 2021-02-10T11:23:13.552 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:23:13.552 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:23:13.552 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:23:13.552 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:23:13.552 INFO:teuthology.orchestra.run.gibba002.stdout:} 2021-02-10T11:23:13.554 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:13.538 7f1ac0c03700 1 -- 172.21.2.102:0/2111572423 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1aa00186c0 msgr2=0x7f1aa001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:13.555 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:13.538 7f1ac0c03700 1 --2- 172.21.2.102:0/2111572423 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1aa00186c0 0x7f1aa001abc0 crc :-1 s=READY pgs=556 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:13.555 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:13.538 7f1ac0c03700 1 -- 172.21.2.102:0/2111572423 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1abc081c50 msgr2=0x7f1abc0ad5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:13.555 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:13.538 7f1ac0c03700 1 --2- 172.21.2.102:0/2111572423 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1abc081c50 0x7f1abc0ad5a0 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 rx=0x7f1ab400b910 tx=0x7f1ab400bcb0).stop 2021-02-10T11:23:13.555 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:13.539 7f1ac0c03700 1 -- 172.21.2.102:0/2111572423 shutdown_connections 2021-02-10T11:23:13.555 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:13.539 7f1ac0c03700 1 --2- 172.21.2.102:0/2111572423 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1abc081c50 0x7f1abc0ad5a0 unknown :-1 s=CLOSED pgs=1169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:13.555 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:13.539 7f1ac0c03700 1 --2- 172.21.2.102:0/2111572423 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1aa00186c0 0x7f1aa001abc0 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:13.555 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:13.539 7f1ac0c03700 1 --2- 172.21.2.102:0/2111572423 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1abc081850 0x7f1abc0acf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:13.556 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:13.539 7f1ac0c03700 1 --2- 172.21.2.102:0/2111572423 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1abc08d400 0x7f1abc081230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:13.556 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:13.539 7f1ac0c03700 1 -- 172.21.2.102:0/2111572423 >> 172.21.2.102:0/2111572423 conn(0x7f1abc1ac620 msgr2=0x7f1abc0896e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:13.556 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:13.539 7f1ac0c03700 1 -- 172.21.2.102:0/2111572423 shutdown_connections 2021-02-10T11:23:13.556 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:13.539 7f1ac0c03700 1 -- 172.21.2.102:0/2111572423 wait complete. 2021-02-10T11:23:13.557 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:23:13.984 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:13.970 7f6cf183e700 1 Processor -- start 2021-02-10T11:23:13.984 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:13.970 7f6cf183e700 1 -- start start 2021-02-10T11:23:13.984 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:13.970 7f6cf183e700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6cec0e5070 0x7f6cec1b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:13.984 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:13.970 7f6cf183e700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f6cec0e5470 0x7f6cec07d690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:13.984 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:13.970 7f6cf183e700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cec07ed80 con 0x7f6cec1b3e40 2021-02-10T11:23:13.985 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:13.970 7f6cf183e700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6cec080310 con 0x7f6cec0e5470 2021-02-10T11:23:13.985 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:13.970 7f6cf183e700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6cec080500 con 0x7f6cec0e5070 2021-02-10T11:23:13.985 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:13.971 7f6cead73700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f6cec0e5470 0x7f6cec07d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:13.985 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:13.971 7f6cead73700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f6cec0e5470 0x7f6cec07d690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:56118/0 (socket says 172.21.2.102:56118) 2021-02-10T11:23:13.985 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:13.971 7f6cead73700 1 -- 172.21.2.102:0/1757499993 learned_addr learned my addr 172.21.2.102:0/1757499993 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:23:13.985 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:13.971 7f6cead73700 1 -- 172.21.2.102:0/1757499993 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6cec0e5070 msgr2=0x7f6cec1b3820 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:23:13.985 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:13.971 7f6cead73700 1 --2- 172.21.2.102:0/1757499993 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6cec0e5070 0x7f6cec1b3820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:13.985 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:13.971 7f6cead73700 1 -- 172.21.2.102:0/1757499993 >> v1:172.21.2.113:6789/0 conn(0x7f6cec1b3e40 legacy=0x7f6cec07dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:13.985 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:13.971 7f6cead73700 1 -- 172.21.2.102:0/1757499993 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cec0806f0 con 0x7f6cec0e5470 2021-02-10T11:23:13.985 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:13.971 7f6cead73700 1 --2- 172.21.2.102:0/1757499993 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6cec0e5470 0x7f6cec07d690 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 rx=0x7f6cdc006700 tx=0x7f6cdc009b00).ready entity=mon.0 client_cookie=dfa9026ca31bf06b server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:13.985 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:13.971 7f6cea572700 1 -- 172.21.2.102:0/1757499993 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6cdc00c0c0 con 0x7f6cec0e5470 2021-02-10T11:23:13.985 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:13.971 7f6cea572700 1 -- 172.21.2.102:0/1757499993 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cdc00c400 con 0x7f6cec0e5470 2021-02-10T11:23:13.986 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:13.971 7f6cea572700 1 -- 172.21.2.102:0/1757499993 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6cdc00c920 con 0x7f6cec0e5470 2021-02-10T11:23:13.986 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:13.971 7f6cf183e700 1 -- 172.21.2.102:0/1757499993 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6cec0e5470 msgr2=0x7f6cec07d690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:13.986 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:13.971 7f6cf183e700 1 --2- 172.21.2.102:0/1757499993 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6cec0e5470 0x7f6cec07d690 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 rx=0x7f6cdc006700 tx=0x7f6cdc009b00).stop 2021-02-10T11:23:13.986 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:13.972 7f6cf183e700 1 -- 172.21.2.102:0/1757499993 shutdown_connections 2021-02-10T11:23:13.986 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:13.972 7f6cf183e700 1 --2- 172.21.2.102:0/1757499993 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6cec0e5470 0x7f6cec07d690 unknown :-1 s=CLOSED pgs=1170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:13.986 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:13.972 7f6cf183e700 1 --2- 172.21.2.102:0/1757499993 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6cec0e5070 0x7f6cec1b3820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:13.986 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:13.972 7f6cf183e700 1 -- 172.21.2.102:0/1757499993 >> 172.21.2.102:0/1757499993 conn(0x7f6cec0ce690 msgr2=0x7f6cec0e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:13.986 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:13.972 7f6cf183e700 1 -- 172.21.2.102:0/1757499993 shutdown_connections 2021-02-10T11:23:13.986 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:13.972 7f6cf183e700 1 -- 172.21.2.102:0/1757499993 wait complete. 2021-02-10T11:23:13.988 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:13.987 7f6cf183e700 1 Processor -- start 2021-02-10T11:23:13.988 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:13.987 7f6cf183e700 1 -- start start 2021-02-10T11:23:13.988 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:13.987 7f6cf183e700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6cec1a2a90 0x7f6cec1aa0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:13.988 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:13.987 7f6cf183e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6cec1aa700 0x7f6cec1a3a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:13.988 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:13.987 7f6cf183e700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6cec0a55e0 0x7f6cec1a4040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:13.988 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:13.987 7f6cf183e700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cec1a34a0 con 0x7f6cec1aa700 2021-02-10T11:23:13.988 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:13.987 7f6cf183e700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cec1a2e90 con 0x7f6cec1a2a90 2021-02-10T11:23:13.988 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:13.987 7f6cf183e700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6cec1a30b0 con 0x7f6cec0a55e0 2021-02-10T11:23:13.988 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:13.987 7f6cead73700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6cec1aa700 0x7f6cec1a3a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:13.988 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:13.987 7f6cead73700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6cec1aa700 0x7f6cec1a3a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:56122/0 (socket says 172.21.2.102:56122) 2021-02-10T11:23:13.989 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:13.987 7f6cead73700 1 -- 172.21.2.102:0/1492166934 learned_addr learned my addr 172.21.2.102:0/1492166934 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:23:13.989 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:13.988 7f6cead73700 1 -- 172.21.2.102:0/1492166934 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6cec0a55e0 msgr2=0x7f6cec1a4040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:13.989 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:13.988 7f6cead73700 1 --2- 172.21.2.102:0/1492166934 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6cec0a55e0 0x7f6cec1a4040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:13.989 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:13.988 7f6cead73700 1 -- 172.21.2.102:0/1492166934 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6cec1a2a90 msgr2=0x7f6cec1aa0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:13.989 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:13.988 7f6cead73700 1 --2- 172.21.2.102:0/1492166934 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6cec1a2a90 0x7f6cec1aa0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:13.989 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:13.988 7f6cead73700 1 -- 172.21.2.102:0/1492166934 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cec1a4660 con 0x7f6cec1aa700 2021-02-10T11:23:13.989 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:13.988 7f6cead73700 1 --2- 172.21.2.102:0/1492166934 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6cec1aa700 0x7f6cec1a3a20 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 rx=0x7f6cdc006f20 tx=0x7f6cdc00b9e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:13.989 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:13.988 7f6ce3fff700 1 -- 172.21.2.102:0/1492166934 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6cdc00bdd0 con 0x7f6cec1aa700 2021-02-10T11:23:13.989 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:13.988 7f6ce3fff700 1 -- 172.21.2.102:0/1492166934 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cdc006440 con 0x7f6cec1aa700 2021-02-10T11:23:13.989 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:13.988 7f6cf183e700 1 -- 172.21.2.102:0/1492166934 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6cec1a4660 con 0x7f6cec1aa700 2021-02-10T11:23:13.989 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:13.988 7f6cf183e700 1 -- 172.21.2.102:0/1492166934 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6cec1a4bf0 con 0x7f6cec1aa700 2021-02-10T11:23:13.989 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:13.988 7f6ce3fff700 1 -- 172.21.2.102:0/1492166934 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6cdc0052e0 con 0x7f6cec1aa700 2021-02-10T11:23:13.989 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:13.989 7f6ce3fff700 1 -- 172.21.2.102:0/1492166934 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f6cdc006030 con 0x7f6cec1aa700 2021-02-10T11:23:13.990 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:13.989 7f6ce3fff700 1 --2- 172.21.2.102:0/1492166934 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6cd0018700 0x7f6cd001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:13.990 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:13.989 7f6ce3fff700 1 -- 172.21.2.102:0/1492166934 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 4753+0+0 (secure 0 0 0) 0x7f6cdc0324a0 con 0x7f6cec1aa700 2021-02-10T11:23:13.990 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:13.989 7f6ceb574700 1 --2- 172.21.2.102:0/1492166934 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6cd0018700 0x7f6cd001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:13.990 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:13.989 7f6ceb574700 1 --2- 172.21.2.102:0/1492166934 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6cd0018700 0x7f6cd001abc0 crc :-1 s=READY pgs=557 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:14.018 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:14.020 7f6cf183e700 1 -- 172.21.2.102:0/1492166934 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6cec083da0 con 0x7f6cec1aa700 2021-02-10T11:23:14.020 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:14.023 7f6ce3fff700 1 -- 172.21.2.102:0/1492166934 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f6cdc006030 con 0x7f6cec1aa700 2021-02-10T11:23:14.144 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:14.146 7f6cf183e700 1 -- 172.21.2.102:0/1492166934 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6cec07ed80 con 0x7f6cec1aa700 2021-02-10T11:23:14.144 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:14.147 7f6ce3fff700 1 -- 172.21.2.102:0/1492166934 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v43) v1 ==== 74+0+11144 (secure 0 0 0) 0x7f6cdc006440 con 0x7f6cec1aa700 2021-02-10T11:23:14.145 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:23:14.145 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":43,"fsid":"68db30ea-dbce-4aef-a6df-f01c6fea0513","created":"2021-02-10 11:03:06.331557","modified":"2021-02-10 11:23:12.898530","last_up_change":"2021-02-10 11:03:32.690562","last_in_change":"2021-02-10 11:03:32.690562","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:03:37.322519","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"be691b4d-a039-408a-af80-9d53f02b6f1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6818","nonce":36088},{"type":"v1","addr":"172.21.2.102:6820","nonce":36088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6822","nonce":36088},{"type":"v1","addr":"172.21.2.102:6824","nonce":36088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6830","nonce":36088},{"type":"v1","addr":"172.21.2.102:6832","nonce":36088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6826","nonce":36088},{"type":"v1","addr":"172.21.2.102:6828","nonce":36088}]},"public_addr":"172.21.2.102:6820/36088","cluster_addr":"172.21.2.102:6824/36088","heartbeat_back_addr":"172.21.2.102:6832/36088","heartbeat_front_addr":"172.21.2.102:6828/36088","state":["exists","up"]},{"osd":1,"uuid":"64e3ccc8-9c69-4d7c-865e-af8e83822635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6819","nonce":36087},{"type":"v1","addr":"172.21.2.102:6821","nonce":36087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6823","nonce":36087},{"type":"v1","addr":"172.21.2.102:6825","nonce":36087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6831","nonce":36087},{"type":"v1","addr":"172.21.2.102:6833","nonce":36087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6827","nonce":36087},{"type":"v1","addr":"172.21.2.102:6829","nonce":36087}]},"public_addr":"172.21.2.102:6821/36087","cluster_addr":"172.21.2.102:6825/36087","heartbeat_back_addr":"172.21.2.102:6833/36087","heartbeat_front_addr":"172.21.2.102:6829/36087","state":["exists","up"]},{"osd":2,"uuid":"153700af-747f-481f-bdf4-d6072d3a842f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6810","nonce":36089},{"type":"v1","addr":"172.21.2.102:6811","nonce":36089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6812","nonce":36089},{"type":"v1","addr":"172.21.2.102:6813","nonce":36089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6816","nonce":36089},{"type":"v1","addr":"172.21.2.102:6817","nonce":36089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6814","nonce":36089},{"type":"v1","addr":"172.21.2.102:6815","nonce":36089}]},"public_addr":"172.21.2.102:6811/36089","cluster_addr":"172.21.2.102:6813/36089","heartbeat_back_addr":"172.21.2.102:6817/36089","heartbeat_front_addr":"172.21.2.102:6815/36089","state":["exists","up"]},{"osd":3,"uuid":"cbc678f4-83a9-45e6-8830-2f7c232b1ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6802","nonce":36090},{"type":"v1","addr":"172.21.2.102:6803","nonce":36090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6804","nonce":36090},{"type":"v1","addr":"172.21.2.102:6805","nonce":36090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6808","nonce":36090},{"type":"v1","addr":"172.21.2.102:6809","nonce":36090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6806","nonce":36090},{"type":"v1","addr":"172.21.2.102:6807","nonce":36090}]},"public_addr":"172.21.2.102:6803/36090","cluster_addr":"172.21.2.102:6805/36090","heartbeat_back_addr":"172.21.2.102:6809/36090","heartbeat_front_addr":"172.21.2.102:6807/36090","state":["exists","up"]},{"osd":4,"uuid":"c01e4008-b258-4f10-afd2-9e7f37214351","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6808","nonce":36368},{"type":"v1","addr":"172.21.2.113:6811","nonce":36368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6814","nonce":36368},{"type":"v1","addr":"172.21.2.113:6818","nonce":36368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6826","nonce":36368},{"type":"v1","addr":"172.21.2.113:6828","nonce":36368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6820","nonce":36368},{"type":"v1","addr":"172.21.2.113:6824","nonce":36368}]},"public_addr":"172.21.2.113:6811/36368","cluster_addr":"172.21.2.113:6818/36368","heartbeat_back_addr":"172.21.2.113:6828/36368","heartbeat_front_addr":"172.21.2.113:6824/36368","state":["exists","up"]},{"osd":5,"uuid":"551f6007-daf8-4a98-8ee0-c2b16bcccb5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6802","nonce":36366},{"type":"v1","addr":"172.21.2.113:6805","nonce":36366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6807","nonce":36366},{"type":"v1","addr":"172.21.2.113:6810","nonce":36366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6819","nonce":36366},{"type":"v1","addr":"172.21.2.113:6822","nonce":36366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6812","nonce":36366},{"type":"v1","addr":"172.21.2.113:6815","nonce":36366}]},"public_addr":"172.21.2.113:6805/36366","cluster_addr":"172.21.2.113:6810/36366","heartbeat_back_addr":"172.21.2.113:6822/36366","heartbeat_front_addr":"172.21.2.113:6815/36366","state":["exists","up"]},{"osd":6,"uuid":"bf67c040-0e50-48be-b102-c3943bc9ebef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6800","nonce":36367},{"type":"v1","addr":"172.21.2.113:6801","nonce":36367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6803","nonce":36367},{"type":"v1","addr":"172.21.2.113:6804","nonce":36367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6813","nonce":36367},{"type":"v1","addr":"172.21.2.113:6816","nonce":36367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6806","nonce":36367},{"type":"v1","addr":"172.21.2.113:6809","nonce":36367}]},"public_addr":"172.21.2.113:6801/36367","cluster_addr":"172.21.2.113:6804/36367","heartbeat_back_addr":"172.21.2.113:6816/36367","heartbeat_front_addr":"172.21.2.113:6809/36367","state":["exists","up"]},{"osd":7,"uuid":"f5461f80-018d-428f-be89-2eb2bcbe7bb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6817","nonce":36365},{"type":"v1","addr":"172.21.2.113:6821","nonce":36365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6823","nonce":36365},{"type":"v1","addr":"172.21.2.113:6825","nonce":36365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6830","nonce":36365},{"type":"v1","addr":"172.21.2.113:6831","nonce":36365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6827","nonce":36365},{"type":"v1","addr":"172.21.2.113:6829","nonce":36365}]},"public_addr":"172.21.2.113:6821/36365","cluster_addr":"172.21.2.113:6825/36365","heartbeat_back_addr":"172.21.2.113:6831/36365","heartbeat_front_addr":"172.21.2.113:6829/36365","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:23:14.146 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:14.148 7f6cf183e700 1 -- 172.21.2.102:0/1492166934 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6cd0018700 msgr2=0x7f6cd001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:14.146 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:14.148 7f6cf183e700 1 --2- 172.21.2.102:0/1492166934 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6cd0018700 0x7f6cd001abc0 crc :-1 s=READY pgs=557 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:14.146 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:14.148 7f6cf183e700 1 -- 172.21.2.102:0/1492166934 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6cec1aa700 msgr2=0x7f6cec1a3a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:14.146 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:14.148 7f6cf183e700 1 --2- 172.21.2.102:0/1492166934 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6cec1aa700 0x7f6cec1a3a20 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 rx=0x7f6cdc006f20 tx=0x7f6cdc00b9e0).stop 2021-02-10T11:23:14.147 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:14.148 7f6cf183e700 1 -- 172.21.2.102:0/1492166934 shutdown_connections 2021-02-10T11:23:14.147 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:14.148 7f6cf183e700 1 --2- 172.21.2.102:0/1492166934 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6cec0a55e0 0x7f6cec1a4040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:14.147 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:14.148 7f6cf183e700 1 --2- 172.21.2.102:0/1492166934 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6cec1aa700 0x7f6cec1a3a20 unknown :-1 s=CLOSED pgs=1171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:14.147 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:14.148 7f6cf183e700 1 --2- 172.21.2.102:0/1492166934 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6cd0018700 0x7f6cd001abc0 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:14.147 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:14.148 7f6cf183e700 1 --2- 172.21.2.102:0/1492166934 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6cec1a2a90 0x7f6cec1aa0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:14.147 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:14.148 7f6cf183e700 1 -- 172.21.2.102:0/1492166934 >> 172.21.2.102:0/1492166934 conn(0x7f6cec08fe10 msgr2=0x7f6cec19c020 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:14.147 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:14.149 7f6cf183e700 1 -- 172.21.2.102:0/1492166934 shutdown_connections 2021-02-10T11:23:14.147 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:14.149 7f6cf183e700 1 -- 172.21.2.102:0/1492166934 wait complete. 2021-02-10T11:23:14.155 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:03:37.322519', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2021-02-10T11:23:14.155 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:23:14.584 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:14.570 7f33fd170700 1 Processor -- start 2021-02-10T11:23:14.585 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:14.570 7f33fd170700 1 -- start start 2021-02-10T11:23:14.585 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:14.571 7f33fd170700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f33f81b0470 0x7f33f81b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:14.585 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:14.571 7f33fd170700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f33f80f20e0 0x7f33f80f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:14.585 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:14.571 7f33fd170700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33f80f0b50 con 0x7f33f80ef610 2021-02-10T11:23:14.585 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:14.571 7f33fd170700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33f80f0820 con 0x7f33f80f20e0 2021-02-10T11:23:14.585 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:14.571 7f33fd170700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33f80f24e0 con 0x7f33f81b0470 2021-02-10T11:23:14.585 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:14.571 7f33f759e700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f33f80f20e0 0x7f33f80f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:14.585 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:14.571 7f33f759e700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f33f80f20e0 0x7f33f80f0200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:56134/0 (socket says 172.21.2.102:56134) 2021-02-10T11:23:14.585 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:14.571 7f33f759e700 1 -- 172.21.2.102:0/3875454471 learned_addr learned my addr 172.21.2.102:0/3875454471 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:23:14.585 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:14.571 7f33f759e700 1 -- 172.21.2.102:0/3875454471 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f33f81b0470 msgr2=0x7f33f81b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:14.585 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:14.571 7f33f759e700 1 --2- 172.21.2.102:0/3875454471 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f33f81b0470 0x7f33f81b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:14.585 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:14.571 7f33f759e700 1 -- 172.21.2.102:0/3875454471 >> v1:172.21.2.113:6789/0 conn(0x7f33f80ef610 legacy=0x7f33f80efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:14.586 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:14.571 7f33f759e700 1 -- 172.21.2.102:0/3875454471 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33f80e52e0 con 0x7f33f80f20e0 2021-02-10T11:23:14.586 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:14.572 7f33f759e700 1 --2- 172.21.2.102:0/3875454471 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f33f80f20e0 0x7f33f80f0200 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 rx=0x7f33f0006f00 tx=0x7f33f0009f10).ready entity=mon.0 client_cookie=687e8aa8758a01a3 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:14.586 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:14.572 7f33f5d9b700 1 -- 172.21.2.102:0/3875454471 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f33f000b3b0 con 0x7f33f80f20e0 2021-02-10T11:23:14.586 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:14.572 7f33f5d9b700 1 -- 172.21.2.102:0/3875454471 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33f000b730 con 0x7f33f80f20e0 2021-02-10T11:23:14.586 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:14.572 7f33f5d9b700 1 -- 172.21.2.102:0/3875454471 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f33f000bc10 con 0x7f33f80f20e0 2021-02-10T11:23:14.586 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:14.572 7f33fd170700 1 -- 172.21.2.102:0/3875454471 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f33f80f20e0 msgr2=0x7f33f80f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:14.586 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:14.572 7f33fd170700 1 --2- 172.21.2.102:0/3875454471 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f33f80f20e0 0x7f33f80f0200 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 rx=0x7f33f0006f00 tx=0x7f33f0009f10).stop 2021-02-10T11:23:14.586 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:14.572 7f33fd170700 1 -- 172.21.2.102:0/3875454471 shutdown_connections 2021-02-10T11:23:14.586 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:14.572 7f33fd170700 1 --2- 172.21.2.102:0/3875454471 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f33f80f20e0 0x7f33f80f0200 unknown :-1 s=CLOSED pgs=1172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:14.586 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:14.572 7f33fd170700 1 --2- 172.21.2.102:0/3875454471 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f33f81b0470 0x7f33f81b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:14.586 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:14.572 7f33fd170700 1 -- 172.21.2.102:0/3875454471 >> 172.21.2.102:0/3875454471 conn(0x7f33f80ce3e0 msgr2=0x7f33f80e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:14.586 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:14.573 7f33fd170700 1 -- 172.21.2.102:0/3875454471 shutdown_connections 2021-02-10T11:23:14.587 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:14.573 7f33fd170700 1 -- 172.21.2.102:0/3875454471 wait complete. 2021-02-10T11:23:14.588 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:14.587 7f33fd170700 1 Processor -- start 2021-02-10T11:23:14.588 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:14.587 7f33fd170700 1 -- start start 2021-02-10T11:23:14.588 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:14.588 7f33fd170700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f33f8094fb0 0x7f33f809e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:14.588 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:14.588 7f33fd170700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f33f80c2be0 0x7f33f80c2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:14.588 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:14.588 7f33fd170700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f33f80c3600 0x7f33f80a9c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:14.588 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:14.588 7f33fd170700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33f809ebf0 con 0x7f33f80c3600 2021-02-10T11:23:14.589 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:14.588 7f33fd170700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33f80aa2a0 con 0x7f33f8094fb0 2021-02-10T11:23:14.589 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:14.588 7f33fd170700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33f80aa4c0 con 0x7f33f80c2be0 2021-02-10T11:23:14.589 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:14.588 7f33f759e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f33f80c3600 0x7f33f80a9c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:14.589 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:14.588 7f33f759e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f33f80c3600 0x7f33f80a9c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:56138/0 (socket says 172.21.2.102:56138) 2021-02-10T11:23:14.589 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:14.588 7f33f759e700 1 -- 172.21.2.102:0/2030061591 learned_addr learned my addr 172.21.2.102:0/2030061591 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:23:14.589 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:14.588 7f33f759e700 1 -- 172.21.2.102:0/2030061591 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f33f80c2be0 msgr2=0x7f33f80c2fe0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:23:14.589 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:14.588 7f33f759e700 1 --2- 172.21.2.102:0/2030061591 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f33f80c2be0 0x7f33f80c2fe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:14.589 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:14.588 7f33f759e700 1 -- 172.21.2.102:0/2030061591 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f33f8094fb0 msgr2=0x7f33f809e5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:14.589 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:14.588 7f33f759e700 1 --2- 172.21.2.102:0/2030061591 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f33f8094fb0 0x7f33f809e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:14.589 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:14.588 7f33f759e700 1 -- 172.21.2.102:0/2030061591 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33f80aa6e0 con 0x7f33f80c3600 2021-02-10T11:23:14.589 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:14.588 7f33f759e700 1 --2- 172.21.2.102:0/2030061591 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f33f80c3600 0x7f33f80a9c80 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 rx=0x7f33f000b8e0 tx=0x7f33f000bc80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:14.589 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:14.589 7f33ef7fe700 1 -- 172.21.2.102:0/2030061591 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f33f0009450 con 0x7f33f80c3600 2021-02-10T11:23:14.590 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:14.589 7f33ef7fe700 1 -- 172.21.2.102:0/2030061591 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33f00058b0 con 0x7f33f80c3600 2021-02-10T11:23:14.590 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:14.589 7f33ef7fe700 1 -- 172.21.2.102:0/2030061591 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f33f0005c80 con 0x7f33f80c3600 2021-02-10T11:23:14.590 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:14.589 7f33fd170700 1 -- 172.21.2.102:0/2030061591 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33f80aa6e0 con 0x7f33f80c3600 2021-02-10T11:23:14.590 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:14.589 7f33fd170700 1 -- 172.21.2.102:0/2030061591 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33f80aa6e0 con 0x7f33f80c3600 2021-02-10T11:23:14.590 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:14.589 7f33ef7fe700 1 -- 172.21.2.102:0/2030061591 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f33f0009450 con 0x7f33f80c3600 2021-02-10T11:23:14.590 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:14.589 7f33ef7fe700 1 --2- 172.21.2.102:0/2030061591 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f33dc018700 0x7f33dc01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:14.590 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:14.589 7f33ef7fe700 1 -- 172.21.2.102:0/2030061591 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 4753+0+0 (secure 0 0 0) 0x7f33f0032680 con 0x7f33f80c3600 2021-02-10T11:23:14.590 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:14.590 7f33f6d9d700 1 --2- 172.21.2.102:0/2030061591 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f33dc018700 0x7f33dc01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:14.590 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:14.590 7f33f6d9d700 1 --2- 172.21.2.102:0/2030061591 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f33dc018700 0x7f33dc01abc0 crc :-1 s=READY pgs=558 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:14.618 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:14.621 7f33fd170700 1 -- 172.21.2.102:0/2030061591 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33f80edd10 con 0x7f33f80c3600 2021-02-10T11:23:14.621 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:14.624 7f33ef7fe700 1 -- 172.21.2.102:0/2030061591 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f33f0000f90 con 0x7f33f80c3600 2021-02-10T11:23:14.749 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:14.751 7f33fd170700 1 -- 172.21.2.102:0/2030061591 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f33f80e52e0 con 0x7f33f80c3600 2021-02-10T11:23:14.749 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:14.751 7f33ef7fe700 1 -- 172.21.2.102:0/2030061591 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v43) v1 ==== 92+0+10 (secure 0 0 0) 0x7f33f00058b0 con 0x7f33f80c3600 2021-02-10T11:23:14.750 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:23:14.751 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:14.753 7f33fd170700 1 -- 172.21.2.102:0/2030061591 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f33dc018700 msgr2=0x7f33dc01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:14.751 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:14.753 7f33fd170700 1 --2- 172.21.2.102:0/2030061591 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f33dc018700 0x7f33dc01abc0 crc :-1 s=READY pgs=558 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:14.751 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:14.753 7f33fd170700 1 -- 172.21.2.102:0/2030061591 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f33f80c3600 msgr2=0x7f33f80a9c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:14.751 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:14.753 7f33fd170700 1 --2- 172.21.2.102:0/2030061591 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f33f80c3600 0x7f33f80a9c80 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 rx=0x7f33f000b8e0 tx=0x7f33f000bc80).stop 2021-02-10T11:23:14.752 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:14.753 7f33fd170700 1 -- 172.21.2.102:0/2030061591 shutdown_connections 2021-02-10T11:23:14.752 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:14.753 7f33fd170700 1 --2- 172.21.2.102:0/2030061591 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f33f80c3600 0x7f33f80a9c80 unknown :-1 s=CLOSED pgs=1173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:14.752 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:14.753 7f33fd170700 1 --2- 172.21.2.102:0/2030061591 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f33f80c2be0 0x7f33f80c2fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:14.752 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:14.753 7f33fd170700 1 --2- 172.21.2.102:0/2030061591 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f33dc018700 0x7f33dc01abc0 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:14.752 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:14.753 7f33fd170700 1 --2- 172.21.2.102:0/2030061591 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f33f8094fb0 0x7f33f809e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:14.752 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:14.753 7f33fd170700 1 -- 172.21.2.102:0/2030061591 >> 172.21.2.102:0/2030061591 conn(0x7f33f809f630 msgr2=0x7f33f80a0130 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:14.752 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:14.754 7f33fd170700 1 -- 172.21.2.102:0/2030061591 shutdown_connections 2021-02-10T11:23:14.752 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:14.754 7f33fd170700 1 -- 172.21.2.102:0/2030061591 wait complete. 2021-02-10T11:23:14.760 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --format=json-pretty osd lspools 2021-02-10T11:23:15.186 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:15.172 7efbed8e9700 1 Processor -- start 2021-02-10T11:23:15.187 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:15.172 7efbed8e9700 1 -- start start 2021-02-10T11:23:15.187 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:15.173 7efbed8e9700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7efbe80e5070 0x7efbe81b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:15.187 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:15.173 7efbed8e9700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efbe807ed80 con 0x7efbe81b3e40 2021-02-10T11:23:15.187 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:15.173 7efbed8e9700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efbe807e670 con 0x7efbe80e5470 2021-02-10T11:23:15.187 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:15.173 7efbed8e9700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efbe807e890 con 0x7efbe80e5070 2021-02-10T11:23:15.187 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:15.173 7efbe7574700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7efbe80e5070 0x7efbe81b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:15.187 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:15.173 7efbe6d73700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7efbe81b3e40 0x7efbe807d690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:48368/0 (socket says 172.21.2.102:48368) 2021-02-10T11:23:15.187 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:15.173 7efbe6d73700 1 -- 172.21.2.102:0/2266448554 learned_addr learned my addr 172.21.2.102:0/2266448554 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:23:15.187 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:15.173 7efbe6572700 1 -- 172.21.2.102:0/2266448554 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2454010560 0 0) 0x7efbdc0009b0 con 0x7efbe81b3e40 2021-02-10T11:23:15.187 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:15.173 7efbe6572700 1 -- 172.21.2.102:0/2266448554 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efbd0002fe0 con 0x7efbe81b3e40 2021-02-10T11:23:15.187 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:15.173 7efbe6572700 1 -- 172.21.2.102:0/2266448554 <== mon.1 v1:172.21.2.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4103982389 0 0) 0x7efbe0000f90 con 0x7efbe80e5470 2021-02-10T11:23:15.188 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:15.173 7efbe6572700 1 -- 172.21.2.102:0/2266448554 --> v1:172.21.2.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efbd0004370 con 0x7efbe80e5470 2021-02-10T11:23:15.188 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:15.174 7efbe7574700 1 -- 172.21.2.102:0/2266448554 >> v1:172.21.2.113:6789/0 conn(0x7efbe80e5470 legacy=0x7efbe807de80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:15.188 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:15.174 7efbe7574700 1 -- 172.21.2.102:0/2266448554 >> v1:172.21.2.102:6789/0 conn(0x7efbe81b3e40 legacy=0x7efbe807d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:15.188 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:15.174 7efbe7574700 1 -- 172.21.2.102:0/2266448554 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efbe807eab0 con 0x7efbe80e5070 2021-02-10T11:23:15.188 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:15.174 7efbe7574700 1 --2- 172.21.2.102:0/2266448554 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efbe80e5070 0x7efbe81b3820 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 rx=0x7efbd8006650 tx=0x7efbd8009a80).ready entity=mon.0 client_cookie=d00db200260ed9e6 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:15.188 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:15.174 7efbe6572700 1 -- 172.21.2.102:0/2266448554 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efbd80009b0 con 0x7efbe80e5070 2021-02-10T11:23:15.188 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:15.174 7efbe6572700 1 -- 172.21.2.102:0/2266448554 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efbd800c0c0 con 0x7efbe80e5070 2021-02-10T11:23:15.188 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:15.174 7efbe6572700 1 -- 172.21.2.102:0/2266448554 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efbd800c5e0 con 0x7efbe80e5070 2021-02-10T11:23:15.188 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:15.174 7efbed8e9700 1 -- 172.21.2.102:0/2266448554 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efbe80e5070 msgr2=0x7efbe81b3820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:15.188 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:15.174 7efbed8e9700 1 --2- 172.21.2.102:0/2266448554 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efbe80e5070 0x7efbe81b3820 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 rx=0x7efbd8006650 tx=0x7efbd8009a80).stop 2021-02-10T11:23:15.188 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:15.175 7efbed8e9700 1 -- 172.21.2.102:0/2266448554 shutdown_connections 2021-02-10T11:23:15.188 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:15.175 7efbed8e9700 1 --2- 172.21.2.102:0/2266448554 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efbe80e5070 0x7efbe81b3820 unknown :-1 s=CLOSED pgs=1175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:15.188 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:15.175 7efbed8e9700 1 -- 172.21.2.102:0/2266448554 >> 172.21.2.102:0/2266448554 conn(0x7efbe80ce690 msgr2=0x7efbe80e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:15.189 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:15.175 7efbed8e9700 1 -- 172.21.2.102:0/2266448554 shutdown_connections 2021-02-10T11:23:15.189 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:15.175 7efbed8e9700 1 -- 172.21.2.102:0/2266448554 wait complete. 2021-02-10T11:23:15.190 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:15.189 7efbed8e9700 1 Processor -- start 2021-02-10T11:23:15.190 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:15.189 7efbed8e9700 1 -- start start 2021-02-10T11:23:15.190 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:15.190 7efbed8e9700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efbe8192960 0x7efbe8192d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:15.190 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:15.190 7efbed8e9700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7efbe80ad110 0x7efbe80ad510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:15.190 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:15.190 7efbed8e9700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7efbe80adb30 0x7efbe808e460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:15.191 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:15.190 7efbed8e9700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efbe81a03e0 con 0x7efbe8192960 2021-02-10T11:23:15.191 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:15.190 7efbed8e9700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efbe8193380 con 0x7efbe80adb30 2021-02-10T11:23:15.191 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:15.190 7efbed8e9700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efbe808ea80 con 0x7efbe80ad110 2021-02-10T11:23:15.191 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:15.190 7efbe7574700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efbe8192960 0x7efbe8192d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:15.191 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:15.190 7efbe7574700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efbe8192960 0x7efbe8192d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:56146/0 (socket says 172.21.2.102:56146) 2021-02-10T11:23:15.191 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:15.190 7efbe7574700 1 -- 172.21.2.102:0/655331816 learned_addr learned my addr 172.21.2.102:0/655331816 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:23:15.191 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:15.190 7efbe6d73700 1 --2- 172.21.2.102:0/655331816 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7efbe80ad110 0x7efbe80ad510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:15.191 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:15.190 7efbe7d75700 1 --2- 172.21.2.102:0/655331816 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7efbe80adb30 0x7efbe808e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:15.191 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:15.190 7efbe7574700 1 -- 172.21.2.102:0/655331816 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7efbe80ad110 msgr2=0x7efbe80ad510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:15.191 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:15.190 7efbe7574700 1 --2- 172.21.2.102:0/655331816 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7efbe80ad110 0x7efbe80ad510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:15.191 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:15.190 7efbe7574700 1 -- 172.21.2.102:0/655331816 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7efbe80adb30 msgr2=0x7efbe808e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:15.191 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:15.190 7efbe7574700 1 --2- 172.21.2.102:0/655331816 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7efbe80adb30 0x7efbe808e460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:15.191 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:15.190 7efbe7574700 1 -- 172.21.2.102:0/655331816 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efbe808eca0 con 0x7efbe8192960 2021-02-10T11:23:15.192 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:15.190 7efbe7574700 1 --2- 172.21.2.102:0/655331816 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efbe8192960 0x7efbe8192d60 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 rx=0x7efbd8009650 tx=0x7efbd800b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:15.192 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:15.191 7efbcffff700 1 -- 172.21.2.102:0/655331816 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efbd80093a0 con 0x7efbe8192960 2021-02-10T11:23:15.192 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:15.191 7efbcffff700 1 -- 172.21.2.102:0/655331816 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efbd800bde0 con 0x7efbe8192960 2021-02-10T11:23:15.192 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:15.191 7efbcffff700 1 -- 172.21.2.102:0/655331816 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efbd80052e0 con 0x7efbe8192960 2021-02-10T11:23:15.192 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:15.191 7efbed8e9700 1 -- 172.21.2.102:0/655331816 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efbe808eca0 con 0x7efbe8192960 2021-02-10T11:23:15.192 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:15.191 7efbed8e9700 1 -- 172.21.2.102:0/655331816 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efbe808f230 con 0x7efbe8192960 2021-02-10T11:23:15.192 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:15.191 7efbcffff700 1 -- 172.21.2.102:0/655331816 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7efbd8009300 con 0x7efbe8192960 2021-02-10T11:23:15.192 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:15.191 7efbcffff700 1 --2- 172.21.2.102:0/655331816 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7efbc80186c0 0x7efbc801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:15.192 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:15.191 7efbcffff700 1 -- 172.21.2.102:0/655331816 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 4753+0+0 (secure 0 0 0) 0x7efbd80320b0 con 0x7efbe8192960 2021-02-10T11:23:15.192 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:15.192 7efbe6d73700 1 --2- 172.21.2.102:0/655331816 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7efbc80186c0 0x7efbc801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:15.192 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:15.192 7efbe6d73700 1 --2- 172.21.2.102:0/655331816 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7efbc80186c0 0x7efbc801abc0 crc :-1 s=READY pgs=559 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:15.220 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:15.223 7efbed8e9700 1 -- 172.21.2.102:0/655331816 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efbe8107120 con 0x7efbe8192960 2021-02-10T11:23:15.223 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:15.225 7efbcffff700 1 -- 172.21.2.102:0/655331816 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7efbd802cf10 con 0x7efbe8192960 2021-02-10T11:23:15.346 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:15.349 7efbed8e9700 1 -- 172.21.2.102:0/655331816 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7efbe807ed80 con 0x7efbe8192960 2021-02-10T11:23:15.347 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:15.349 7efbcffff700 1 -- 172.21.2.102:0/655331816 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v43) v1 ==== 84+0+64 (secure 0 0 0) 0x7efbd802cf10 con 0x7efbe8192960 2021-02-10T11:23:15.347 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:23:15.348 INFO:teuthology.orchestra.run.gibba002.stdout:[ 2021-02-10T11:23:15.348 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:15.348 INFO:teuthology.orchestra.run.gibba002.stdout: "poolnum": 1, 2021-02-10T11:23:15.348 INFO:teuthology.orchestra.run.gibba002.stdout: "poolname": "rbd" 2021-02-10T11:23:15.348 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:23:15.348 INFO:teuthology.orchestra.run.gibba002.stdout:] 2021-02-10T11:23:15.349 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:15.351 7efbed8e9700 1 -- 172.21.2.102:0/655331816 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7efbc80186c0 msgr2=0x7efbc801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:15.349 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:15.351 7efbed8e9700 1 --2- 172.21.2.102:0/655331816 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7efbc80186c0 0x7efbc801abc0 crc :-1 s=READY pgs=559 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:15.349 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:15.351 7efbed8e9700 1 -- 172.21.2.102:0/655331816 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efbe8192960 msgr2=0x7efbe8192d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:15.349 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:15.351 7efbed8e9700 1 --2- 172.21.2.102:0/655331816 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efbe8192960 0x7efbe8192d60 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 rx=0x7efbd8009650 tx=0x7efbd800b960).stop 2021-02-10T11:23:15.349 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:15.351 7efbed8e9700 1 -- 172.21.2.102:0/655331816 shutdown_connections 2021-02-10T11:23:15.349 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:15.351 7efbed8e9700 1 --2- 172.21.2.102:0/655331816 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7efbc80186c0 0x7efbc801abc0 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:15.349 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:15.351 7efbed8e9700 1 --2- 172.21.2.102:0/655331816 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7efbe80adb30 0x7efbe808e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:15.350 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:15.351 7efbed8e9700 1 --2- 172.21.2.102:0/655331816 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7efbe80ad110 0x7efbe80ad510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:15.350 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:15.351 7efbed8e9700 1 --2- 172.21.2.102:0/655331816 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efbe8192960 0x7efbe8192d60 unknown :-1 s=CLOSED pgs=1176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:15.350 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:15.351 7efbed8e9700 1 -- 172.21.2.102:0/655331816 >> 172.21.2.102:0/655331816 conn(0x7efbe80891f0 msgr2=0x7efbe81a0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:15.350 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:15.352 7efbed8e9700 1 -- 172.21.2.102:0/655331816 shutdown_connections 2021-02-10T11:23:15.350 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:15.352 7efbed8e9700 1 -- 172.21.2.102:0/655331816 wait complete. 2021-02-10T11:23:15.358 INFO:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2021-02-10T11:23:15.358 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool create cephfs_metadata 8 2021-02-10T11:23:15.788 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:15.774 7fd54270c700 1 Processor -- start 2021-02-10T11:23:15.788 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:15.774 7fd54270c700 1 -- start start 2021-02-10T11:23:15.788 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:15.774 7fd54270c700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fd53c1b0470 0x7fd53c1b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:15.788 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:15.774 7fd54270c700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fd53c0ef610 0x7fd53c0efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:15.788 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:15.774 7fd54270c700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd53c0f0b50 con 0x7fd53c0f0030 2021-02-10T11:23:15.788 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:15.774 7fd54270c700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd53c0e52e0 con 0x7fd53c0ef610 2021-02-10T11:23:15.789 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:15.774 7fd54270c700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd53c0e5490 con 0x7fd53c1b0470 2021-02-10T11:23:15.789 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:15.775 7fd53b7fe700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fd53c0ef610 0x7fd53c0efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:15.789 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:15.775 7fd53b7fe700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fd53c0ef610 0x7fd53c0efa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:56154/0 (socket says 172.21.2.102:56154) 2021-02-10T11:23:15.789 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:15.775 7fd53b7fe700 1 -- 172.21.2.102:0/3876566843 learned_addr learned my addr 172.21.2.102:0/3876566843 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:23:15.789 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:15.775 7fd53b7fe700 1 -- 172.21.2.102:0/3876566843 >> v2:172.21.2.113:3300/0 conn(0x7fd53c1b0470 msgr2=0x7fd53c1b0870 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:23:15.789 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:15.775 7fd53b7fe700 1 --2- 172.21.2.102:0/3876566843 >> v2:172.21.2.113:3300/0 conn(0x7fd53c1b0470 0x7fd53c1b0870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:15.789 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:15.775 7fd53b7fe700 1 -- 172.21.2.102:0/3876566843 >> v1:172.21.2.113:6789/0 conn(0x7fd53c0f0030 legacy=0x7fd53c0f10d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:23:15.789 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:15.775 7fd53b7fe700 1 -- 172.21.2.102:0/3876566843 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd53c0f0430 con 0x7fd53c0ef610 2021-02-10T11:23:15.789 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:15.775 7fd53b7fe700 1 --2- 172.21.2.102:0/3876566843 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd53c0ef610 0x7fd53c0efa10 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 rx=0x7fd530006700 tx=0x7fd530009b00).ready entity=mon.0 client_cookie=2597bf3384b5493f server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:15.789 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:15.775 7fd53affd700 1 -- 172.21.2.102:0/3876566843 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd53000c0c0 con 0x7fd53c0ef610 2021-02-10T11:23:15.789 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:15.775 7fd53affd700 1 -- 172.21.2.102:0/3876566843 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd53000c340 con 0x7fd53c0ef610 2021-02-10T11:23:15.789 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:15.775 7fd53affd700 1 -- 172.21.2.102:0/3876566843 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd53000c820 con 0x7fd53c0ef610 2021-02-10T11:23:15.789 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:15.775 7fd54270c700 1 -- 172.21.2.102:0/3876566843 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd53c0ef610 msgr2=0x7fd53c0efa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:15.789 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:15.775 7fd54270c700 1 --2- 172.21.2.102:0/3876566843 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd53c0ef610 0x7fd53c0efa10 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 rx=0x7fd530006700 tx=0x7fd530009b00).stop 2021-02-10T11:23:15.790 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:15.776 7fd54270c700 1 -- 172.21.2.102:0/3876566843 shutdown_connections 2021-02-10T11:23:15.790 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:15.776 7fd54270c700 1 --2- 172.21.2.102:0/3876566843 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd53c0ef610 0x7fd53c0efa10 unknown :-1 s=CLOSED pgs=1177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:15.790 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:15.776 7fd54270c700 1 --2- 172.21.2.102:0/3876566843 >> v2:172.21.2.113:3300/0 conn(0x7fd53c1b0470 0x7fd53c1b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:15.790 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:15.776 7fd54270c700 1 -- 172.21.2.102:0/3876566843 >> 172.21.2.102:0/3876566843 conn(0x7fd53c0ce3e0 msgr2=0x7fd53c0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:15.790 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:15.776 7fd54270c700 1 -- 172.21.2.102:0/3876566843 shutdown_connections 2021-02-10T11:23:15.790 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:15.776 7fd54270c700 1 -- 172.21.2.102:0/3876566843 wait complete. 2021-02-10T11:23:15.791 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:15.791 7fd54270c700 1 Processor -- start 2021-02-10T11:23:15.791 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:15.791 7fd54270c700 1 -- start start 2021-02-10T11:23:15.792 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:15.791 7fd54270c700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd53c0c36a0 0x7fd53c0c2b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:15.792 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:15.791 7fd54270c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd53c0c3190 0x7fd53c0b1d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:15.792 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:15.791 7fd54270c700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd53c0b2370 0x7fd53c0b2d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:15.792 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:15.791 7fd54270c700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd53c0b27c0 con 0x7fd53c0c3190 2021-02-10T11:23:15.792 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:15.791 7fd54270c700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd53c0b3360 con 0x7fd53c0c36a0 2021-02-10T11:23:15.792 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:15.791 7fd54270c700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd53c0b3540 con 0x7fd53c0b2370 2021-02-10T11:23:15.792 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:15.791 7fd53b7fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd53c0c3190 0x7fd53c0b1d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:15.792 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:15.791 7fd53b7fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd53c0c3190 0x7fd53c0b1d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:56164/0 (socket says 172.21.2.102:56164) 2021-02-10T11:23:15.792 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:15.791 7fd53b7fe700 1 -- 172.21.2.102:0/2322542124 learned_addr learned my addr 172.21.2.102:0/2322542124 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:23:15.792 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:15.792 7fd53b7fe700 1 -- 172.21.2.102:0/2322542124 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd53c0b2370 msgr2=0x7fd53c0b2d40 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:23:15.792 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:15.792 7fd53b7fe700 1 --2- 172.21.2.102:0/2322542124 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd53c0b2370 0x7fd53c0b2d40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:15.792 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:15.792 7fd53b7fe700 1 -- 172.21.2.102:0/2322542124 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd53c0c36a0 msgr2=0x7fd53c0c2b70 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:23:15.792 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:15.792 7fd53b7fe700 1 --2- 172.21.2.102:0/2322542124 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd53c0c36a0 0x7fd53c0c2b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:15.792 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:15.792 7fd53b7fe700 1 -- 172.21.2.102:0/2322542124 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd53c0b3760 con 0x7fd53c0c3190 2021-02-10T11:23:15.793 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:15.792 7fd53b7fe700 1 --2- 172.21.2.102:0/2322542124 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd53c0c3190 0x7fd53c0b1d50 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 rx=0x7fd530006f20 tx=0x7fd53000b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:15.793 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:15.792 7fd538ff9700 1 -- 172.21.2.102:0/2322542124 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd53000bd80 con 0x7fd53c0c3190 2021-02-10T11:23:15.793 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:15.792 7fd538ff9700 1 -- 172.21.2.102:0/2322542124 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd530000c90 con 0x7fd53c0c3190 2021-02-10T11:23:15.793 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:15.792 7fd538ff9700 1 -- 172.21.2.102:0/2322542124 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd5300022b0 con 0x7fd53c0c3190 2021-02-10T11:23:15.793 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:15.792 7fd54270c700 1 -- 172.21.2.102:0/2322542124 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd53c0b3760 con 0x7fd53c0c3190 2021-02-10T11:23:15.793 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:15.792 7fd54270c700 1 -- 172.21.2.102:0/2322542124 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd53c0b3cf0 con 0x7fd53c0c3190 2021-02-10T11:23:15.793 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:15.793 7fd538ff9700 1 -- 172.21.2.102:0/2322542124 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd53000a000 con 0x7fd53c0c3190 2021-02-10T11:23:15.793 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:15.793 7fd538ff9700 1 --2- 172.21.2.102:0/2322542124 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd51c0186c0 0x7fd51c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:15.793 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:15.793 7fd538ff9700 1 -- 172.21.2.102:0/2322542124 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 4753+0+0 (secure 0 0 0) 0x7fd5300322f0 con 0x7fd53c0c3190 2021-02-10T11:23:15.793 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:15.793 7fd53bfff700 1 --2- 172.21.2.102:0/2322542124 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd51c0186c0 0x7fd51c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:15.793 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:15.793 7fd53bfff700 1 --2- 172.21.2.102:0/2322542124 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd51c0186c0 0x7fd51c01abc0 crc :-1 s=READY pgs=560 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:15.822 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:15.824 7fd54270c700 1 -- 172.21.2.102:0/2322542124 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd53c0edd10 con 0x7fd53c0c3190 2021-02-10T11:23:15.825 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:15.827 7fd538ff9700 1 -- 172.21.2.102:0/2322542124 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd530009100 con 0x7fd53c0c3190 2021-02-10T11:23:15.952 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:15.955 7fd54270c700 1 -- 172.21.2.102:0/2322542124 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"} v 0) v1 -- 0x7fd53c0e52e0 con 0x7fd53c0c3190 2021-02-10T11:23:16.368 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:16.371 7fd538ff9700 1 -- 172.21.2.102:0/2322542124 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"}]=0 pool 'cephfs_metadata' created v44) v1 ==== 133+0+0 (secure 0 0 0) 0x7fd53000a000 con 0x7fd53c0c3190 2021-02-10T11:23:16.417 INFO:teuthology.orchestra.run.gibba002.stderr:pool 'cephfs_metadata' created 2021-02-10T11:23:16.419 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:16.422 7fd54270c700 1 -- 172.21.2.102:0/2322542124 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd51c0186c0 msgr2=0x7fd51c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:16.420 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:16.422 7fd54270c700 1 --2- 172.21.2.102:0/2322542124 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd51c0186c0 0x7fd51c01abc0 crc :-1 s=READY pgs=560 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:16.420 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:16.422 7fd54270c700 1 -- 172.21.2.102:0/2322542124 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd53c0c3190 msgr2=0x7fd53c0b1d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:16.420 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:16.422 7fd54270c700 1 --2- 172.21.2.102:0/2322542124 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd53c0c3190 0x7fd53c0b1d50 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 rx=0x7fd530006f20 tx=0x7fd53000b990).stop 2021-02-10T11:23:16.420 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:16.422 7fd54270c700 1 -- 172.21.2.102:0/2322542124 shutdown_connections 2021-02-10T11:23:16.420 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:16.422 7fd54270c700 1 --2- 172.21.2.102:0/2322542124 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd53c0b2370 0x7fd53c0b2d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:16.420 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:16.422 7fd54270c700 1 --2- 172.21.2.102:0/2322542124 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd53c0c3190 0x7fd53c0b1d50 unknown :-1 s=CLOSED pgs=1178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:16.420 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:16.422 7fd54270c700 1 --2- 172.21.2.102:0/2322542124 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd51c0186c0 0x7fd51c01abc0 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:16.420 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:16.422 7fd54270c700 1 --2- 172.21.2.102:0/2322542124 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd53c0c36a0 0x7fd53c0c2b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:16.420 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:16.422 7fd54270c700 1 -- 172.21.2.102:0/2322542124 >> 172.21.2.102:0/2322542124 conn(0x7fd53c09f630 msgr2=0x7fd53c0a0130 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:16.420 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:16.422 7fd54270c700 1 -- 172.21.2.102:0/2322542124 shutdown_connections 2021-02-10T11:23:16.420 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:16.422 7fd54270c700 1 -- 172.21.2.102:0/2322542124 wait complete. 2021-02-10T11:23:16.428 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool create cephfs_data 8 2021-02-10T11:23:16.859 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:16.845 7fc3ad548700 1 Processor -- start 2021-02-10T11:23:16.860 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:16.845 7fc3ad548700 1 -- start start 2021-02-10T11:23:16.860 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:16.846 7fc3ad548700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fc3a81b0470 0x7fc3a81b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:16.860 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:16.846 7fc3ad548700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fc3a80ef610 0x7fc3a80efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:16.860 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:16.846 7fc3ad548700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3a80f0b50 con 0x7fc3a80f0030 2021-02-10T11:23:16.860 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:16.846 7fc3ad548700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3a80e52e0 con 0x7fc3a80ef610 2021-02-10T11:23:16.860 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:16.846 7fc3ad548700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3a80e5490 con 0x7fc3a81b0470 2021-02-10T11:23:16.860 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:16.846 7fc3a6abe700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fc3a80ef610 0x7fc3a80efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:16.860 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:16.846 7fc3a6abe700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fc3a80ef610 0x7fc3a80efa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:56168/0 (socket says 172.21.2.102:56168) 2021-02-10T11:23:16.860 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:16.846 7fc3a6abe700 1 -- 172.21.2.102:0/3811568686 learned_addr learned my addr 172.21.2.102:0/3811568686 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:23:16.860 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:16.846 7fc3a7ac0700 1 --1- 172.21.2.102:0/3811568686 >> v1:172.21.2.102:6789/0 conn(0x7fc3a80f0030 0x7fc3a80f10d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:48392/0 (socket says 172.21.2.102:48392) 2021-02-10T11:23:16.860 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:16.846 7fc3a72bf700 1 --2- 172.21.2.102:0/3811568686 >> v2:172.21.2.113:3300/0 conn(0x7fc3a81b0470 0x7fc3a81b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:16.860 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:16.846 7fc3a6abe700 1 -- 172.21.2.102:0/3811568686 >> v2:172.21.2.113:3300/0 conn(0x7fc3a81b0470 msgr2=0x7fc3a81b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:16.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:16.846 7fc3a6abe700 1 --2- 172.21.2.102:0/3811568686 >> v2:172.21.2.113:3300/0 conn(0x7fc3a81b0470 0x7fc3a81b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:16.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:16.846 7fc3a6abe700 1 -- 172.21.2.102:0/3811568686 >> v1:172.21.2.102:6789/0 conn(0x7fc3a80f0030 legacy=0x7fc3a80f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:16.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:16.846 7fc3a6abe700 1 -- 172.21.2.102:0/3811568686 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3a80f0430 con 0x7fc3a80ef610 2021-02-10T11:23:16.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:16.846 7fc3a6abe700 1 --2- 172.21.2.102:0/3811568686 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc3a80ef610 0x7fc3a80efa10 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 rx=0x7fc398006700 tx=0x7fc398009b80).ready entity=mon.0 client_cookie=c4b489d03eb8a062 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:16.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:16.847 7fc3a62bd700 1 -- 172.21.2.102:0/3811568686 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc39800c0c0 con 0x7fc3a80ef610 2021-02-10T11:23:16.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:16.847 7fc3a62bd700 1 -- 172.21.2.102:0/3811568686 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc39800c340 con 0x7fc3a80ef610 2021-02-10T11:23:16.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:16.847 7fc3a62bd700 1 -- 172.21.2.102:0/3811568686 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc39800c860 con 0x7fc3a80ef610 2021-02-10T11:23:16.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:16.847 7fc3ad548700 1 -- 172.21.2.102:0/3811568686 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc3a80ef610 msgr2=0x7fc3a80efa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:16.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:16.847 7fc3ad548700 1 --2- 172.21.2.102:0/3811568686 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc3a80ef610 0x7fc3a80efa10 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 rx=0x7fc398006700 tx=0x7fc398009b80).stop 2021-02-10T11:23:16.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:16.847 7fc3ad548700 1 -- 172.21.2.102:0/3811568686 shutdown_connections 2021-02-10T11:23:16.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:16.847 7fc3ad548700 1 --2- 172.21.2.102:0/3811568686 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc3a80ef610 0x7fc3a80efa10 unknown :-1 s=CLOSED pgs=1180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:16.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:16.847 7fc3ad548700 1 --2- 172.21.2.102:0/3811568686 >> v2:172.21.2.113:3300/0 conn(0x7fc3a81b0470 0x7fc3a81b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:16.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:16.847 7fc3ad548700 1 -- 172.21.2.102:0/3811568686 >> 172.21.2.102:0/3811568686 conn(0x7fc3a80ce3e0 msgr2=0x7fc3a80e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:16.862 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:16.847 7fc3ad548700 1 -- 172.21.2.102:0/3811568686 shutdown_connections 2021-02-10T11:23:16.862 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:16.847 7fc3ad548700 1 -- 172.21.2.102:0/3811568686 wait complete. 2021-02-10T11:23:16.863 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:16.862 7fc3ad548700 1 Processor -- start 2021-02-10T11:23:16.863 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:16.862 7fc3ad548700 1 -- start start 2021-02-10T11:23:16.863 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:16.863 7fc3ad548700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc3a81a48b0 0x7fc3a80bf930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:16.863 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:16.863 7fc3ad548700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc3a80a73c0 0x7fc3a80bff50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:16.864 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:16.863 7fc3ad548700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc3a80a5260 0x7fc3a80a5660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:16.864 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:16.863 7fc3ad548700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3a80a6ec0 con 0x7fc3a81a48b0 2021-02-10T11:23:16.864 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:16.863 7fc3ad548700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3a80bc010 con 0x7fc3a80a5260 2021-02-10T11:23:16.864 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:16.863 7fc3ad548700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3a80c0570 con 0x7fc3a80a73c0 2021-02-10T11:23:16.864 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:16.863 7fc3a72bf700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc3a81a48b0 0x7fc3a80bf930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:16.864 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:16.863 7fc3a6abe700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc3a80a73c0 0x7fc3a80bff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:16.864 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:16.863 7fc3a72bf700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc3a81a48b0 0x7fc3a80bf930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:56174/0 (socket says 172.21.2.102:56174) 2021-02-10T11:23:16.864 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:16.863 7fc3a72bf700 1 -- 172.21.2.102:0/2266547266 learned_addr learned my addr 172.21.2.102:0/2266547266 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:23:16.864 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:16.863 7fc3a6abe700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc3a80a73c0 0x7fc3a80bff50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.113:3301/0 says I am v2:172.21.2.102:33476/0 (socket says 172.21.2.102:33476) 2021-02-10T11:23:16.864 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:16.863 7fc3a7ac0700 1 --2- 172.21.2.102:0/2266547266 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc3a80a5260 0x7fc3a80a5660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:16.864 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:16.863 7fc3a72bf700 1 -- 172.21.2.102:0/2266547266 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc3a80a73c0 msgr2=0x7fc3a80bff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:16.864 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:16.863 7fc3a72bf700 1 --2- 172.21.2.102:0/2266547266 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc3a80a73c0 0x7fc3a80bff50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:16.864 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:16.863 7fc3a72bf700 1 -- 172.21.2.102:0/2266547266 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc3a80a5260 msgr2=0x7fc3a80a5660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:16.865 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:16.863 7fc3a72bf700 1 --2- 172.21.2.102:0/2266547266 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc3a80a5260 0x7fc3a80a5660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:16.865 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:16.863 7fc3a72bf700 1 -- 172.21.2.102:0/2266547266 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3a80a5c80 con 0x7fc3a81a48b0 2021-02-10T11:23:16.865 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:16.863 7fc3a72bf700 1 --2- 172.21.2.102:0/2266547266 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc3a81a48b0 0x7fc3a80bf930 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 rx=0x7fc390006f60 tx=0x7fc39000b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:16.865 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:16.864 7fc39ffff700 1 -- 172.21.2.102:0/2266547266 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc39000bd60 con 0x7fc3a81a48b0 2021-02-10T11:23:16.865 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:16.864 7fc39ffff700 1 -- 172.21.2.102:0/2266547266 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc390000c90 con 0x7fc3a81a48b0 2021-02-10T11:23:16.865 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:16.864 7fc39ffff700 1 -- 172.21.2.102:0/2266547266 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc3900022b0 con 0x7fc3a81a48b0 2021-02-10T11:23:16.865 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:16.864 7fc3ad548700 1 -- 172.21.2.102:0/2266547266 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3a80a5c80 con 0x7fc3a81a48b0 2021-02-10T11:23:16.865 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:16.864 7fc3ad548700 1 -- 172.21.2.102:0/2266547266 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3a80a6210 con 0x7fc3a81a48b0 2021-02-10T11:23:16.865 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:16.864 7fc39ffff700 1 -- 172.21.2.102:0/2266547266 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fc390000c90 con 0x7fc3a81a48b0 2021-02-10T11:23:16.865 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:16.864 7fc39ffff700 1 --2- 172.21.2.102:0/2266547266 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc38c0186c0 0x7fc38c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:16.865 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:16.864 7fc39ffff700 1 -- 172.21.2.102:0/2266547266 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5107+0+0 (secure 0 0 0) 0x7fc390032270 con 0x7fc3a81a48b0 2021-02-10T11:23:16.865 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:16.865 7fc3a6abe700 1 --2- 172.21.2.102:0/2266547266 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc38c0186c0 0x7fc38c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:16.865 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:16.865 7fc3a6abe700 1 --2- 172.21.2.102:0/2266547266 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc38c0186c0 0x7fc38c01abc0 crc :-1 s=READY pgs=561 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:16.893 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:16.895 7fc3ad548700 1 -- 172.21.2.102:0/2266547266 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3a80c0790 con 0x7fc3a81a48b0 2021-02-10T11:23:16.895 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:16.898 7fc39ffff700 1 -- 172.21.2.102:0/2266547266 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fc3900009b0 con 0x7fc3a81a48b0 2021-02-10T11:23:17.024 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:17.027 7fc3ad548700 1 -- 172.21.2.102:0/2266547266 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_data"} v 0) v1 -- 0x7fc3a80c0790 con 0x7fc3a81a48b0 2021-02-10T11:23:17.377 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:17.379 7fc39ffff700 1 -- 172.21.2.102:0/2266547266 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_data"}]=0 pool 'cephfs_data' created v46) v1 ==== 125+0+0 (secure 0 0 0) 0x7fc390000c90 con 0x7fc3a81a48b0 2021-02-10T11:23:17.390 INFO:teuthology.orchestra.run.gibba002.stderr:pool 'cephfs_data' created 2021-02-10T11:23:17.392 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:17.394 7fc3ad548700 1 -- 172.21.2.102:0/2266547266 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc38c0186c0 msgr2=0x7fc38c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:17.392 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:17.394 7fc3ad548700 1 --2- 172.21.2.102:0/2266547266 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc38c0186c0 0x7fc38c01abc0 crc :-1 s=READY pgs=561 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:17.392 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:17.394 7fc3ad548700 1 -- 172.21.2.102:0/2266547266 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc3a81a48b0 msgr2=0x7fc3a80bf930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:17.392 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:17.394 7fc3ad548700 1 --2- 172.21.2.102:0/2266547266 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc3a81a48b0 0x7fc3a80bf930 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 rx=0x7fc390006f60 tx=0x7fc39000b970).stop 2021-02-10T11:23:17.392 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:17.394 7fc3ad548700 1 -- 172.21.2.102:0/2266547266 shutdown_connections 2021-02-10T11:23:17.392 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:17.394 7fc3ad548700 1 --2- 172.21.2.102:0/2266547266 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc38c0186c0 0x7fc38c01abc0 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:17.393 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:17.394 7fc3ad548700 1 --2- 172.21.2.102:0/2266547266 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc3a80a5260 0x7fc3a80a5660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:17.393 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:17.394 7fc3ad548700 1 --2- 172.21.2.102:0/2266547266 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc3a80a73c0 0x7fc3a80bff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:17.393 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:17.394 7fc3ad548700 1 --2- 172.21.2.102:0/2266547266 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc3a81a48b0 0x7fc3a80bf930 unknown :-1 s=CLOSED pgs=1181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:17.393 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:17.394 7fc3ad548700 1 -- 172.21.2.102:0/2266547266 >> 172.21.2.102:0/2266547266 conn(0x7fc3a8109e80 msgr2=0x7fc3a80af060 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:17.393 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:17.395 7fc3ad548700 1 -- 172.21.2.102:0/2266547266 shutdown_connections 2021-02-10T11:23:17.393 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:17.395 7fc3ad548700 1 -- 172.21.2.102:0/2266547266 wait complete. 2021-02-10T11:23:17.401 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2021-02-10T11:23:17.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:17.817 7fb3814fe700 1 Processor -- start 2021-02-10T11:23:17.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:17.817 7fb3814fe700 1 -- start start 2021-02-10T11:23:17.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:17.818 7fb3814fe700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb37c0d1870 0x7fb37c1b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:17.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:17.818 7fb3814fe700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fb37c0cff70 0x7fb37c109130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:17.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:17.818 7fb3814fe700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb37c109a60 con 0x7fb37c1b0ae0 2021-02-10T11:23:17.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:17.818 7fb3814fe700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb37c109750 con 0x7fb37c0cff70 2021-02-10T11:23:17.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:17.818 7fb3814fe700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb37c088020 con 0x7fb37c0d1870 2021-02-10T11:23:17.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:17.818 7fb37bac0700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fb37c0cff70 0x7fb37c109130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:17.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:17.818 7fb37bac0700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fb37c0cff70 0x7fb37c109130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:56182/0 (socket says 172.21.2.102:56182) 2021-02-10T11:23:17.832 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:17.818 7fb37bac0700 1 -- 172.21.2.102:0/2952817552 learned_addr learned my addr 172.21.2.102:0/2952817552 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:23:17.832 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:17.818 7fb37bac0700 1 -- 172.21.2.102:0/2952817552 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb37c0d1870 msgr2=0x7fb37c1b04c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:23:17.832 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:17.818 7fb37bac0700 1 --2- 172.21.2.102:0/2952817552 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb37c0d1870 0x7fb37c1b04c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:17.832 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:17.818 7fb37bac0700 1 -- 172.21.2.102:0/2952817552 >> v1:172.21.2.113:6789/0 conn(0x7fb37c1b0ae0 legacy=0x7fb37c108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:17.832 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:17.818 7fb37bac0700 1 -- 172.21.2.102:0/2952817552 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb37c088210 con 0x7fb37c0cff70 2021-02-10T11:23:17.832 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:17.818 7fb37bac0700 1 --2- 172.21.2.102:0/2952817552 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb37c0cff70 0x7fb37c109130 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 rx=0x7fb374006f00 tx=0x7fb374009f10).ready entity=mon.0 client_cookie=5efecc6758417475 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:17.832 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:17.818 7fb37a2bd700 1 -- 172.21.2.102:0/2952817552 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb37400b3b0 con 0x7fb37c0cff70 2021-02-10T11:23:17.832 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:17.818 7fb37a2bd700 1 -- 172.21.2.102:0/2952817552 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb37400b630 con 0x7fb37c0cff70 2021-02-10T11:23:17.832 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:17.818 7fb37a2bd700 1 -- 172.21.2.102:0/2952817552 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb37400bb50 con 0x7fb37c0cff70 2021-02-10T11:23:17.832 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:17.818 7fb3814fe700 1 -- 172.21.2.102:0/2952817552 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb37c0cff70 msgr2=0x7fb37c109130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:17.832 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:17.818 7fb3814fe700 1 --2- 172.21.2.102:0/2952817552 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb37c0cff70 0x7fb37c109130 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 rx=0x7fb374006f00 tx=0x7fb374009f10).stop 2021-02-10T11:23:17.832 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:17.819 7fb3814fe700 1 -- 172.21.2.102:0/2952817552 shutdown_connections 2021-02-10T11:23:17.832 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:17.819 7fb3814fe700 1 --2- 172.21.2.102:0/2952817552 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb37c0cff70 0x7fb37c109130 unknown :-1 s=CLOSED pgs=1182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:17.833 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:17.819 7fb3814fe700 1 --2- 172.21.2.102:0/2952817552 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb37c0d1870 0x7fb37c1b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:17.833 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:17.819 7fb3814fe700 1 -- 172.21.2.102:0/2952817552 >> 172.21.2.102:0/2952817552 conn(0x7fb37c0ce420 msgr2=0x7fb37c0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:17.833 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:17.819 7fb3814fe700 1 -- 172.21.2.102:0/2952817552 shutdown_connections 2021-02-10T11:23:17.833 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:17.819 7fb3814fe700 1 -- 172.21.2.102:0/2952817552 wait complete. 2021-02-10T11:23:17.834 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:17.833 7fb3814fe700 1 Processor -- start 2021-02-10T11:23:17.834 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:17.834 7fb3814fe700 1 -- start start 2021-02-10T11:23:17.834 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:17.834 7fb3814fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb37c19af50 0x7fb37c0b0420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:17.835 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:17.834 7fb3814fe700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb37c0b0a40 0x7fb37c19b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:17.835 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:17.834 7fb3814fe700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb37c19be00 0x7fb37c19c200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:17.835 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:17.834 7fb3814fe700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb37c19b3d0 con 0x7fb37c19af50 2021-02-10T11:23:17.835 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:17.834 7fb3814fe700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb37c18f4d0 con 0x7fb37c19be00 2021-02-10T11:23:17.835 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:17.834 7fb3814fe700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb37c18f6f0 con 0x7fb37c0b0a40 2021-02-10T11:23:17.835 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:17.834 7fb37b2bf700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb37c19af50 0x7fb37c0b0420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:17.835 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:17.834 7fb37b2bf700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb37c19af50 0x7fb37c0b0420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:56186/0 (socket says 172.21.2.102:56186) 2021-02-10T11:23:17.835 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:17.834 7fb37b2bf700 1 -- 172.21.2.102:0/1492169173 learned_addr learned my addr 172.21.2.102:0/1492169173 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:23:17.835 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:17.834 7fb37bac0700 1 --2- 172.21.2.102:0/1492169173 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb37c19be00 0x7fb37c19c200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:17.835 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:17.834 7fb37aabe700 1 --2- 172.21.2.102:0/1492169173 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb37c0b0a40 0x7fb37c19b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:17.835 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:17.835 7fb37b2bf700 1 -- 172.21.2.102:0/1492169173 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb37c0b0a40 msgr2=0x7fb37c19b7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:17.835 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:17.835 7fb37b2bf700 1 --2- 172.21.2.102:0/1492169173 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb37c0b0a40 0x7fb37c19b7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:17.835 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:17.835 7fb37b2bf700 1 -- 172.21.2.102:0/1492169173 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb37c19be00 msgr2=0x7fb37c19c200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:17.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:17.835 7fb37b2bf700 1 --2- 172.21.2.102:0/1492169173 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb37c19be00 0x7fb37c19c200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:17.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:17.835 7fb37b2bf700 1 -- 172.21.2.102:0/1492169173 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb37c18f910 con 0x7fb37c19af50 2021-02-10T11:23:17.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:17.835 7fb37b2bf700 1 --2- 172.21.2.102:0/1492169173 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb37c19af50 0x7fb37c0b0420 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 rx=0x7fb36c00b5c0 tx=0x7fb36c00b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:17.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:17.835 7fb373fff700 1 -- 172.21.2.102:0/1492169173 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb36c00bd50 con 0x7fb37c19af50 2021-02-10T11:23:17.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:17.835 7fb373fff700 1 -- 172.21.2.102:0/1492169173 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb36c006030 con 0x7fb37c19af50 2021-02-10T11:23:17.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:17.835 7fb373fff700 1 -- 172.21.2.102:0/1492169173 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb36c0054d0 con 0x7fb37c19af50 2021-02-10T11:23:17.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:17.835 7fb3814fe700 1 -- 172.21.2.102:0/1492169173 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb37c18f910 con 0x7fb37c19af50 2021-02-10T11:23:17.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:17.835 7fb3814fe700 1 -- 172.21.2.102:0/1492169173 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb37c18fea0 con 0x7fb37c19af50 2021-02-10T11:23:17.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:17.835 7fb373fff700 1 -- 172.21.2.102:0/1492169173 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fb36c006210 con 0x7fb37c19af50 2021-02-10T11:23:17.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:17.836 7fb373fff700 1 --2- 172.21.2.102:0/1492169173 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb3600186c0 0x7fb36001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:17.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:17.836 7fb373fff700 1 -- 172.21.2.102:0/1492169173 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5457+0+0 (secure 0 0 0) 0x7fb36c032530 con 0x7fb37c19af50 2021-02-10T11:23:17.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:17.836 7fb37aabe700 1 --2- 172.21.2.102:0/1492169173 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb3600186c0 0x7fb36001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:17.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:17.836 7fb37aabe700 1 --2- 172.21.2.102:0/1492169173 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb3600186c0 0x7fb36001abc0 crc :-1 s=READY pgs=564 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:17.865 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:17.867 7fb3814fe700 1 -- 172.21.2.102:0/1492169173 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb37c0fe1c0 con 0x7fb37c19af50 2021-02-10T11:23:17.868 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:17.870 7fb373fff700 1 -- 172.21.2.102:0/1492169173 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fb36c000960 con 0x7fb37c19af50 2021-02-10T11:23:17.998 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:18.001 7fb3814fe700 1 -- 172.21.2.102:0/1492169173 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"data": "cephfs_data", "prefix": "fs new", "force": true, "fs_name": "cephfs", "metadata": "cephfs_metadata"} v 0) v1 -- 0x7fb37c0fe1c0 con 0x7fb37c19af50 2021-02-10T11:23:18.002 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:18.005 7fb373fff700 1 -- 172.21.2.102:0/1492169173 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"data": "cephfs_data", "prefix": "fs new", "force": true, "fs_name": "cephfs", "metadata": "cephfs_metadata"}]=0 new fs with metadata pool 8 and data pool 9 v27) v1 ==== 187+0+0 (secure 0 0 0) 0x7fb36c006210 con 0x7fb37c19af50 2021-02-10T11:23:18.005 INFO:teuthology.orchestra.run.gibba002.stderr:new fs with metadata pool 8 and data pool 9 2021-02-10T11:23:18.008 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:18.010 7fb3814fe700 1 -- 172.21.2.102:0/1492169173 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb3600186c0 msgr2=0x7fb36001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:18.008 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:18.010 7fb3814fe700 1 --2- 172.21.2.102:0/1492169173 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb3600186c0 0x7fb36001abc0 crc :-1 s=READY pgs=564 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:18.008 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:18.010 7fb3814fe700 1 -- 172.21.2.102:0/1492169173 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb37c19af50 msgr2=0x7fb37c0b0420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:18.008 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:18.010 7fb3814fe700 1 --2- 172.21.2.102:0/1492169173 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb37c19af50 0x7fb37c0b0420 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 rx=0x7fb36c00b5c0 tx=0x7fb36c00b960).stop 2021-02-10T11:23:18.008 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:18.010 7fb3814fe700 1 -- 172.21.2.102:0/1492169173 shutdown_connections 2021-02-10T11:23:18.008 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:18.010 7fb3814fe700 1 --2- 172.21.2.102:0/1492169173 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb3600186c0 0x7fb36001abc0 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:18.008 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:18.010 7fb3814fe700 1 --2- 172.21.2.102:0/1492169173 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb37c19be00 0x7fb37c19c200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:18.008 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:18.010 7fb3814fe700 1 --2- 172.21.2.102:0/1492169173 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb37c0b0a40 0x7fb37c19b7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:18.008 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:18.010 7fb3814fe700 1 --2- 172.21.2.102:0/1492169173 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb37c19af50 0x7fb37c0b0420 unknown :-1 s=CLOSED pgs=1183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:18.008 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:18.010 7fb3814fe700 1 -- 172.21.2.102:0/1492169173 >> 172.21.2.102:0/1492169173 conn(0x7fb37c19f870 msgr2=0x7fb37c1a5f70 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:18.009 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:18.010 7fb3814fe700 1 -- 172.21.2.102:0/1492169173 shutdown_connections 2021-02-10T11:23:18.009 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:18.010 7fb3814fe700 1 -- 172.21.2.102:0/1492169173 wait complete. 2021-02-10T11:23:18.016 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:23:18.442 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:18.427 7f24b3bbd700 1 Processor -- start 2021-02-10T11:23:18.442 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:18.428 7f24b3bbd700 1 -- start start 2021-02-10T11:23:18.442 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:18.428 7f24b3bbd700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f24ac0e5070 0x7f24ac1b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:18.442 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:18.428 7f24b3bbd700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f24ac0e5470 0x7f24ac080310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:18.442 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:18.428 7f24b3bbd700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24ac07ed80 con 0x7f24ac1b3e40 2021-02-10T11:23:18.442 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:18.428 7f24b3bbd700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24ac0d1790 con 0x7f24ac0e5070 2021-02-10T11:23:18.443 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:18.428 7f24b3bbd700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24ac0d19b0 con 0x7f24ac0e5470 2021-02-10T11:23:18.443 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:18.428 7f24b19d9700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f24ac0e5070 0x7f24ac1b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:18.443 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:18.428 7f24b19d9700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f24ac0e5070 0x7f24ac1b3820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:56192/0 (socket says 172.21.2.102:56192) 2021-02-10T11:23:18.443 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:18.428 7f24b19d9700 1 -- 172.21.2.102:0/1885003005 learned_addr learned my addr 172.21.2.102:0/1885003005 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:23:18.443 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:18.428 7f24b21da700 1 --2- 172.21.2.102:0/1885003005 >> v2:172.21.2.113:3300/0 conn(0x7f24ac0e5470 0x7f24ac080310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:18.443 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:18.429 7f24b19d9700 1 -- 172.21.2.102:0/1885003005 >> v2:172.21.2.113:3300/0 conn(0x7f24ac0e5470 msgr2=0x7f24ac080310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:18.443 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:18.429 7f24b19d9700 1 --2- 172.21.2.102:0/1885003005 >> v2:172.21.2.113:3300/0 conn(0x7f24ac0e5470 0x7f24ac080310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:18.443 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:18.429 7f24b19d9700 1 -- 172.21.2.102:0/1885003005 >> v1:172.21.2.113:6789/0 conn(0x7f24ac1b3e40 legacy=0x7f24ac07d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:18.443 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:18.429 7f24b19d9700 1 -- 172.21.2.102:0/1885003005 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24ac0d1bd0 con 0x7f24ac0e5070 2021-02-10T11:23:18.443 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:18.429 7f24b19d9700 1 --2- 172.21.2.102:0/1885003005 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f24ac0e5070 0x7f24ac1b3820 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 rx=0x7f249c006700 tx=0x7f249c009b80).ready entity=mon.0 client_cookie=eb940deedbf117de server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:18.443 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:18.429 7f24b09d7700 1 -- 172.21.2.102:0/1885003005 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f249c00c0c0 con 0x7f24ac0e5070 2021-02-10T11:23:18.443 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:18.429 7f24b09d7700 1 -- 172.21.2.102:0/1885003005 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f249c00c340 con 0x7f24ac0e5070 2021-02-10T11:23:18.443 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:18.429 7f24b09d7700 1 -- 172.21.2.102:0/1885003005 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f249c00c860 con 0x7f24ac0e5070 2021-02-10T11:23:18.444 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:18.429 7f24b3bbd700 1 -- 172.21.2.102:0/1885003005 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f24ac0e5070 msgr2=0x7f24ac1b3820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:18.444 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:18.429 7f24b3bbd700 1 --2- 172.21.2.102:0/1885003005 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f24ac0e5070 0x7f24ac1b3820 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 rx=0x7f249c006700 tx=0x7f249c009b80).stop 2021-02-10T11:23:18.444 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:18.430 7f24b3bbd700 1 -- 172.21.2.102:0/1885003005 shutdown_connections 2021-02-10T11:23:18.444 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:18.430 7f24b3bbd700 1 --2- 172.21.2.102:0/1885003005 >> v2:172.21.2.113:3300/0 conn(0x7f24ac0e5470 0x7f24ac080310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:18.444 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:18.430 7f24b3bbd700 1 --2- 172.21.2.102:0/1885003005 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f24ac0e5070 0x7f24ac1b3820 unknown :-1 s=CLOSED pgs=1184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:18.444 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:18.430 7f24b3bbd700 1 -- 172.21.2.102:0/1885003005 >> 172.21.2.102:0/1885003005 conn(0x7f24ac0ce690 msgr2=0x7f24ac0e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:18.444 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:18.430 7f24b3bbd700 1 -- 172.21.2.102:0/1885003005 shutdown_connections 2021-02-10T11:23:18.444 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:18.430 7f24b3bbd700 1 -- 172.21.2.102:0/1885003005 wait complete. 2021-02-10T11:23:18.446 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:18.445 7f24b3bbd700 1 Processor -- start 2021-02-10T11:23:18.446 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:18.445 7f24b3bbd700 1 -- start start 2021-02-10T11:23:18.446 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:18.445 7f24b3bbd700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f24ac08d400 0x7f24ac096c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:18.446 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:18.445 7f24b3bbd700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f24ac0b1450 0x7f24ac0b1850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:18.446 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:18.445 7f24b3bbd700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f24ac0b1e70 0x7f24ac1b1f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:18.446 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:18.445 7f24b3bbd700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24ac097310 con 0x7f24ac0b1e70 2021-02-10T11:23:18.446 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:18.445 7f24b3bbd700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24ac08d800 con 0x7f24ac08d400 2021-02-10T11:23:18.446 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:18.445 7f24b3bbd700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24ac08d9f0 con 0x7f24ac0b1450 2021-02-10T11:23:18.446 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:18.445 7f24b19d9700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f24ac08d400 0x7f24ac096c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:18.446 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:18.445 7f24b19d9700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f24ac08d400 0x7f24ac096c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.113:3300/0 says I am v2:172.21.2.102:59628/0 (socket says 172.21.2.102:59628) 2021-02-10T11:23:18.446 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:18.445 7f24b19d9700 1 -- 172.21.2.102:0/1427679056 learned_addr learned my addr 172.21.2.102:0/1427679056 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:23:18.447 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:18.445 7f24b21da700 1 --2- 172.21.2.102:0/1427679056 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f24ac0b1e70 0x7f24ac1b1f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:18.447 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:18.445 7f24b11d8700 1 --2- 172.21.2.102:0/1427679056 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f24ac0b1450 0x7f24ac0b1850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:18.447 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:18.446 7f24b21da700 1 -- 172.21.2.102:0/1427679056 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f24ac0b1450 msgr2=0x7f24ac0b1850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:18.447 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:18.446 7f24b21da700 1 --2- 172.21.2.102:0/1427679056 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f24ac0b1450 0x7f24ac0b1850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:18.447 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:18.446 7f24b21da700 1 -- 172.21.2.102:0/1427679056 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f24ac08d400 msgr2=0x7f24ac096c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:18.447 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:18.446 7f24b21da700 1 --2- 172.21.2.102:0/1427679056 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f24ac08d400 0x7f24ac096c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:18.447 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:18.446 7f24b21da700 1 -- 172.21.2.102:0/1427679056 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24ac1b25f0 con 0x7f24ac0b1e70 2021-02-10T11:23:18.447 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:18.446 7f24b21da700 1 --2- 172.21.2.102:0/1427679056 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f24ac0b1e70 0x7f24ac1b1f60 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 rx=0x7f24a8006ed0 tx=0x7f24a800b840).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:18.447 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:18.446 7f24a27fc700 1 -- 172.21.2.102:0/1427679056 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f24a800bc30 con 0x7f24ac0b1e70 2021-02-10T11:23:18.447 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:18.446 7f24a27fc700 1 -- 172.21.2.102:0/1427679056 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24a80066c0 con 0x7f24ac0b1e70 2021-02-10T11:23:18.447 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:18.446 7f24b3bbd700 1 -- 172.21.2.102:0/1427679056 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24ac1b25f0 con 0x7f24ac0b1e70 2021-02-10T11:23:18.447 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:18.446 7f24a27fc700 1 -- 172.21.2.102:0/1427679056 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f24a8002560 con 0x7f24ac0b1e70 2021-02-10T11:23:18.447 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:18.446 7f24b3bbd700 1 -- 172.21.2.102:0/1427679056 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24ac1b25f0 con 0x7f24ac0b1e70 2021-02-10T11:23:18.448 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:18.447 7f24a27fc700 1 -- 172.21.2.102:0/1427679056 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f24a8005330 con 0x7f24ac0b1e70 2021-02-10T11:23:18.448 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:18.447 7f24a27fc700 1 --2- 172.21.2.102:0/1427679056 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2494018730 0x7f249401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:18.448 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:18.447 7f24a27fc700 1 -- 172.21.2.102:0/1427679056 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f24a80326e0 con 0x7f24ac0b1e70 2021-02-10T11:23:18.448 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:18.447 7f24b19d9700 1 --2- 172.21.2.102:0/1427679056 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2494018730 0x7f249401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:18.448 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:18.447 7f24b19d9700 1 --2- 172.21.2.102:0/1427679056 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2494018730 0x7f249401ac10 crc :-1 s=READY pgs=565 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:18.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:18.477 7f24b3bbd700 1 -- 172.21.2.102:0/1427679056 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24ac07ed80 con 0x7f24ac0b1e70 2021-02-10T11:23:18.478 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:18.480 7f24a27fc700 1 -- 172.21.2.102:0/1427679056 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f24a80066c0 con 0x7f24ac0b1e70 2021-02-10T11:23:18.603 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:18.606 7f24b3bbd700 1 -- 172.21.2.102:0/1427679056 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f24ac07ed80 con 0x7f24ac0b1e70 2021-02-10T11:23:18.604 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:18.606 7f24a27fc700 1 -- 172.21.2.102:0/1427679056 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v47) v1 ==== 74+0+14006 (secure 0 0 0) 0x7f24a8006370 con 0x7f24ac0b1e70 2021-02-10T11:23:18.604 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:23:18.604 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":47,"fsid":"68db30ea-dbce-4aef-a6df-f01c6fea0513","created":"2021-02-10 11:03:06.331557","modified":"2021-02-10 11:23:18.002592","last_up_change":"2021-02-10 11:03:32.690562","last_in_change":"2021-02-10 11:03:32.690562","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:03:37.322519","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:23:15.973296","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"47","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":9,"pool_name":"cephfs_data","create_time":"2021-02-10 11:23:17.045434","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"47","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"be691b4d-a039-408a-af80-9d53f02b6f1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6818","nonce":36088},{"type":"v1","addr":"172.21.2.102:6820","nonce":36088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6822","nonce":36088},{"type":"v1","addr":"172.21.2.102:6824","nonce":36088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6830","nonce":36088},{"type":"v1","addr":"172.21.2.102:6832","nonce":36088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6826","nonce":36088},{"type":"v1","addr":"172.21.2.102:6828","nonce":36088}]},"public_addr":"172.21.2.102:6820/36088","cluster_addr":"172.21.2.102:6824/36088","heartbeat_back_addr":"172.21.2.102:6832/36088","heartbeat_front_addr":"172.21.2.102:6828/36088","state":["exists","up"]},{"osd":1,"uuid":"64e3ccc8-9c69-4d7c-865e-af8e83822635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6819","nonce":36087},{"type":"v1","addr":"172.21.2.102:6821","nonce":36087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6823","nonce":36087},{"type":"v1","addr":"172.21.2.102:6825","nonce":36087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6831","nonce":36087},{"type":"v1","addr":"172.21.2.102:6833","nonce":36087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6827","nonce":36087},{"type":"v1","addr":"172.21.2.102:6829","nonce":36087}]},"public_addr":"172.21.2.102:6821/36087","cluster_addr":"172.21.2.102:6825/36087","heartbeat_back_addr":"172.21.2.102:6833/36087","heartbeat_front_addr":"172.21.2.102:6829/36087","state":["exists","up"]},{"osd":2,"uuid":"153700af-747f-481f-bdf4-d6072d3a842f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6810","nonce":36089},{"type":"v1","addr":"172.21.2.102:6811","nonce":36089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6812","nonce":36089},{"type":"v1","addr":"172.21.2.102:6813","nonce":36089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6816","nonce":36089},{"type":"v1","addr":"172.21.2.102:6817","nonce":36089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6814","nonce":36089},{"type":"v1","addr":"172.21.2.102:6815","nonce":36089}]},"public_addr":"172.21.2.102:6811/36089","cluster_addr":"172.21.2.102:6813/36089","heartbeat_back_addr":"172.21.2.102:6817/36089","heartbeat_front_addr":"172.21.2.102:6815/36089","state":["exists","up"]},{"osd":3,"uuid":"cbc678f4-83a9-45e6-8830-2f7c232b1ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6802","nonce":36090},{"type":"v1","addr":"172.21.2.102:6803","nonce":36090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6804","nonce":36090},{"type":"v1","addr":"172.21.2.102:6805","nonce":36090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6808","nonce":36090},{"type":"v1","addr":"172.21.2.102:6809","nonce":36090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6806","nonce":36090},{"type":"v1","addr":"172.21.2.102:6807","nonce":36090}]},"public_addr":"172.21.2.102:6803/36090","cluster_addr":"172.21.2.102:6805/36090","heartbeat_back_addr":"172.21.2.102:6809/36090","heartbeat_front_addr":"172.21.2.102:6807/36090","state":["exists","up"]},{"osd":4,"uuid":"c01e4008-b258-4f10-afd2-9e7f37214351","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6808","nonce":36368},{"type":"v1","addr":"172.21.2.113:6811","nonce":36368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6814","nonce":36368},{"type":"v1","addr":"172.21.2.113:6818","nonce":36368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6826","nonce":36368},{"type":"v1","addr":"172.21.2.113:6828","nonce":36368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6820","nonce":36368},{"type":"v1","addr":"172.21.2.113:6824","nonce":36368}]},"public_addr":"172.21.2.113:6811/36368","cluster_addr":"172.21.2.113:6818/36368","heartbeat_back_addr":"172.21.2.113:6828/36368","heartbeat_front_addr":"172.21.2.113:6824/36368","state":["exists","up"]},{"osd":5,"uuid":"551f6007-daf8-4a98-8ee0-c2b16bcccb5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6802","nonce":36366},{"type":"v1","addr":"172.21.2.113:6805","nonce":36366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6807","nonce":36366},{"type":"v1","addr":"172.21.2.113:6810","nonce":36366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6819","nonce":36366},{"type":"v1","addr":"172.21.2.113:6822","nonce":36366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6812","nonce":36366},{"type":"v1","addr":"172.21.2.113:6815","nonce":36366}]},"public_addr":"172.21.2.113:6805/36366","cluster_addr":"172.21.2.113:6810/36366","heartbeat_back_addr":"172.21.2.113:6822/36366","heartbeat_front_addr":"172.21.2.113:6815/36366","state":["exists","up"]},{"osd":6,"uuid":"bf67c040-0e50-48be-b102-c3943bc9ebef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6800","nonce":36367},{"type":"v1","addr":"172.21.2.113:6801","nonce":36367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6803","nonce":36367},{"type":"v1","addr":"172.21.2.113:6804","nonce":36367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6813","nonce":36367},{"type":"v1","addr":"172.21.2.113:6816","nonce":36367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6806","nonce":36367},{"type":"v1","addr":"172.21.2.113:6809","nonce":36367}]},"public_addr":"172.21.2.113:6801/36367","cluster_addr":"172.21.2.113:6804/36367","heartbeat_back_addr":"172.21.2.113:6816/36367","heartbeat_front_addr":"172.21.2.113:6809/36367","state":["exists","up"]},{"osd":7,"uuid":"f5461f80-018d-428f-be89-2eb2bcbe7bb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6817","nonce":36365},{"type":"v1","addr":"172.21.2.113:6821","nonce":36365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6823","nonce":36365},{"type":"v1","addr":"172.21.2.113:6825","nonce":36365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6830","nonce":36365},{"type":"v1","addr":"172.21.2.113:6831","nonce":36365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6827","nonce":36365},{"type":"v1","addr":"172.21.2.113:6829","nonce":36365}]},"public_addr":"172.21.2.113:6821/36365","cluster_addr":"172.21.2.113:6825/36365","heartbeat_back_addr":"172.21.2.113:6831/36365","heartbeat_front_addr":"172.21.2.113:6829/36365","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:23:18.605 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:18.608 7f24b3bbd700 1 -- 172.21.2.102:0/1427679056 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2494018730 msgr2=0x7f249401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:18.606 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:18.608 7f24b3bbd700 1 --2- 172.21.2.102:0/1427679056 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2494018730 0x7f249401ac10 crc :-1 s=READY pgs=565 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:18.606 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:18.608 7f24b3bbd700 1 -- 172.21.2.102:0/1427679056 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f24ac0b1e70 msgr2=0x7f24ac1b1f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:18.606 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:18.608 7f24b3bbd700 1 --2- 172.21.2.102:0/1427679056 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f24ac0b1e70 0x7f24ac1b1f60 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 rx=0x7f24a8006ed0 tx=0x7f24a800b840).stop 2021-02-10T11:23:18.606 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:18.608 7f24b3bbd700 1 -- 172.21.2.102:0/1427679056 shutdown_connections 2021-02-10T11:23:18.606 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:18.608 7f24b3bbd700 1 --2- 172.21.2.102:0/1427679056 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f24ac0b1e70 0x7f24ac1b1f60 unknown :-1 s=CLOSED pgs=1185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:18.606 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:18.608 7f24b3bbd700 1 --2- 172.21.2.102:0/1427679056 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f24ac0b1450 0x7f24ac0b1850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:18.606 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:18.608 7f24b3bbd700 1 --2- 172.21.2.102:0/1427679056 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2494018730 0x7f249401ac10 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:18.606 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:18.608 7f24b3bbd700 1 --2- 172.21.2.102:0/1427679056 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f24ac08d400 0x7f24ac096c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:18.606 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:18.608 7f24b3bbd700 1 -- 172.21.2.102:0/1427679056 >> 172.21.2.102:0/1427679056 conn(0x7f24ac1ac620 msgr2=0x7f24ac0896e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:18.607 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:18.608 7f24b3bbd700 1 -- 172.21.2.102:0/1427679056 shutdown_connections 2021-02-10T11:23:18.607 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:18.608 7f24b3bbd700 1 -- 172.21.2.102:0/1427679056 wait complete. 2021-02-10T11:23:18.614 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:23:19.041 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:19.027 7f559c2cf700 1 Processor -- start 2021-02-10T11:23:19.042 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:19.027 7f559c2cf700 1 -- start start 2021-02-10T11:23:19.042 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:19.027 7f559c2cf700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f55940e5070 0x7f55941b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:19.042 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:19.027 7f559c2cf700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f55941b3e40 0x7f559407d690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:19.042 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:19.027 7f559c2cf700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f559407ed80 con 0x7f55940e5470 2021-02-10T11:23:19.042 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:19.027 7f559c2cf700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5594080310 con 0x7f55940e5070 2021-02-10T11:23:19.042 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:19.027 7f559c2cf700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5594080530 con 0x7f55941b3e40 2021-02-10T11:23:19.042 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:19.028 7f559a0eb700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f55940e5070 0x7f55941b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:19.042 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:19.028 7f559a0eb700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f55940e5070 0x7f55941b3820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:56206/0 (socket says 172.21.2.102:56206) 2021-02-10T11:23:19.042 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:19.028 7f559a0eb700 1 -- 172.21.2.102:0/457917749 learned_addr learned my addr 172.21.2.102:0/457917749 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:23:19.042 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:19.028 7f55998ea700 1 --2- 172.21.2.102:0/457917749 >> v2:172.21.2.113:3300/0 conn(0x7f55941b3e40 0x7f559407d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:19.042 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:19.028 7f559a0eb700 1 -- 172.21.2.102:0/457917749 >> v2:172.21.2.113:3300/0 conn(0x7f55941b3e40 msgr2=0x7f559407d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:19.042 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:19.028 7f559a0eb700 1 --2- 172.21.2.102:0/457917749 >> v2:172.21.2.113:3300/0 conn(0x7f55941b3e40 0x7f559407d690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:19.043 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:19.028 7f559a0eb700 1 -- 172.21.2.102:0/457917749 >> v1:172.21.2.113:6789/0 conn(0x7f55940e5470 legacy=0x7f559407dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:19.043 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:19.028 7f559a0eb700 1 -- 172.21.2.102:0/457917749 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5594080750 con 0x7f55940e5070 2021-02-10T11:23:19.043 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:19.028 7f559a0eb700 1 --2- 172.21.2.102:0/457917749 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f55940e5070 0x7f55941b3820 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 rx=0x7f558c006a80 tx=0x7f558c009f00).ready entity=mon.0 client_cookie=96c531fe7d981543 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:19.043 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:19.028 7f55990e9700 1 -- 172.21.2.102:0/457917749 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f558c00b5c0 con 0x7f55940e5070 2021-02-10T11:23:19.043 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:19.028 7f55990e9700 1 -- 172.21.2.102:0/457917749 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f558c00b8c0 con 0x7f55940e5070 2021-02-10T11:23:19.043 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:19.028 7f55990e9700 1 -- 172.21.2.102:0/457917749 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f558c00bde0 con 0x7f55940e5070 2021-02-10T11:23:19.043 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:19.028 7f559c2cf700 1 -- 172.21.2.102:0/457917749 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f55940e5070 msgr2=0x7f55941b3820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:19.043 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:19.028 7f559c2cf700 1 --2- 172.21.2.102:0/457917749 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f55940e5070 0x7f55941b3820 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 rx=0x7f558c006a80 tx=0x7f558c009f00).stop 2021-02-10T11:23:19.043 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:19.029 7f559c2cf700 1 -- 172.21.2.102:0/457917749 shutdown_connections 2021-02-10T11:23:19.043 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:19.029 7f559c2cf700 1 --2- 172.21.2.102:0/457917749 >> v2:172.21.2.113:3300/0 conn(0x7f55941b3e40 0x7f559407d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:19.043 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:19.029 7f559c2cf700 1 --2- 172.21.2.102:0/457917749 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f55940e5070 0x7f55941b3820 unknown :-1 s=CLOSED pgs=1186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:19.043 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:19.029 7f559c2cf700 1 -- 172.21.2.102:0/457917749 >> 172.21.2.102:0/457917749 conn(0x7f55940ce690 msgr2=0x7f55940e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:19.043 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:19.029 7f559c2cf700 1 -- 172.21.2.102:0/457917749 shutdown_connections 2021-02-10T11:23:19.044 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:19.029 7f559c2cf700 1 -- 172.21.2.102:0/457917749 wait complete. 2021-02-10T11:23:19.044 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:19.044 7f559c2cf700 1 Processor -- start 2021-02-10T11:23:19.044 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:19.044 7f559c2cf700 1 -- start start 2021-02-10T11:23:19.045 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:19.044 7f559c2cf700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f559408d400 0x7f5594081230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:19.045 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:19.044 7f559c2cf700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5594081850 0x7f55940acf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:19.045 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:19.044 7f559c2cf700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5594081c50 0x7f55940ad5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:19.045 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:19.044 7f559c2cf700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5594082110 con 0x7f559408d400 2021-02-10T11:23:19.045 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:19.044 7f559c2cf700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f559408d800 con 0x7f5594081850 2021-02-10T11:23:19.045 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:19.044 7f559c2cf700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55940adbc0 con 0x7f5594081c50 2021-02-10T11:23:19.046 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:19.044 7f559a8ec700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5594081c50 0x7f55940ad5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:19.046 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:19.044 7f559a8ec700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5594081c50 0x7f55940ad5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.113:3301/0 says I am v2:172.21.2.102:33516/0 (socket says 172.21.2.102:33516) 2021-02-10T11:23:19.046 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:19.044 7f559a8ec700 1 -- 172.21.2.102:0/3217848357 learned_addr learned my addr 172.21.2.102:0/3217848357 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:23:19.046 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:19.044 7f559a0eb700 1 --2- 172.21.2.102:0/3217848357 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f559408d400 0x7f5594081230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:19.046 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:19.045 7f55998ea700 1 --2- 172.21.2.102:0/3217848357 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5594081850 0x7f55940acf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:19.046 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:19.045 7f559a8ec700 1 -- 172.21.2.102:0/3217848357 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5594081850 msgr2=0x7f55940acf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:19.046 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:19.045 7f559a8ec700 1 --2- 172.21.2.102:0/3217848357 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5594081850 0x7f55940acf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:19.046 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:19.045 7f559a8ec700 1 -- 172.21.2.102:0/3217848357 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f559408d400 msgr2=0x7f5594081230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:19.046 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:19.045 7f559a8ec700 1 --2- 172.21.2.102:0/3217848357 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f559408d400 0x7f5594081230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:19.046 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:19.045 7f559a8ec700 1 -- 172.21.2.102:0/3217848357 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5594187fc0 con 0x7f5594081c50 2021-02-10T11:23:19.046 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:19.045 7f55998ea700 1 --2- 172.21.2.102:0/3217848357 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5594081850 0x7f55940acf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:23:19.046 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:19.045 7f559a0eb700 1 --2- 172.21.2.102:0/3217848357 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f559408d400 0x7f5594081230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:23:19.047 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:19.045 7f559a8ec700 1 --2- 172.21.2.102:0/3217848357 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5594081c50 0x7f55940ad5a0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7f5590006f10 tx=0x7f559000b990).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:19.047 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:19.045 7f558affd700 1 -- 172.21.2.102:0/3217848357 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f55900063b0 con 0x7f5594081c50 2021-02-10T11:23:19.047 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:19.045 7f558affd700 1 -- 172.21.2.102:0/3217848357 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5590000960 con 0x7f5594081c50 2021-02-10T11:23:19.047 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:19.045 7f558affd700 1 -- 172.21.2.102:0/3217848357 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5590002030 con 0x7f5594081c50 2021-02-10T11:23:19.047 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:19.045 7f559c2cf700 1 -- 172.21.2.102:0/3217848357 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5594187fc0 con 0x7f5594081c50 2021-02-10T11:23:19.047 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:19.045 7f559c2cf700 1 -- 172.21.2.102:0/3217848357 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5594188550 con 0x7f5594081c50 2021-02-10T11:23:19.047 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:19.046 7f558affd700 1 -- 172.21.2.102:0/3217848357 <== mon.2 v2:172.21.2.113:3301/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f5590005320 con 0x7f5594081c50 2021-02-10T11:23:19.047 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:19.046 7f558affd700 1 --2- 172.21.2.102:0/3217848357 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f557c018700 0x7f557c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:19.047 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:19.046 7f558affd700 1 -- 172.21.2.102:0/3217848357 <== mon.2 v2:172.21.2.113:3301/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f5590032680 con 0x7f5594081c50 2021-02-10T11:23:19.047 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:19.046 7f559a0eb700 1 --2- 172.21.2.102:0/3217848357 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f557c018700 0x7f557c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:19.047 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:19.046 7f559a0eb700 1 --2- 172.21.2.102:0/3217848357 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f557c018700 0x7f557c01abc0 crc :-1 s=READY pgs=566 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:19.075 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:19.077 7f559c2cf700 1 -- 172.21.2.102:0/3217848357 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f559407ed80 con 0x7f5594081c50 2021-02-10T11:23:19.077 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:19.080 7f558affd700 1 -- 172.21.2.102:0/3217848357 <== mon.2 v2:172.21.2.113:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f55900063b0 con 0x7f5594081c50 2021-02-10T11:23:19.207 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:19.207 7f559c2cf700 1 -- 172.21.2.102:0/3217848357 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f559407ed80 con 0x7f5594081c50 2021-02-10T11:23:19.207 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:19.208 7f558affd700 1 -- 172.21.2.102:0/3217848357 <== mon.2 v2:172.21.2.113:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v48) v1 ==== 74+0+13993 (secure 0 0 0) 0x7f5590000960 con 0x7f5594081c50 2021-02-10T11:23:19.208 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:23:19.208 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":48,"fsid":"68db30ea-dbce-4aef-a6df-f01c6fea0513","created":"2021-02-10 11:03:06.331557","modified":"2021-02-10 11:23:19.005019","last_up_change":"2021-02-10 11:03:32.690562","last_in_change":"2021-02-10 11:03:32.690562","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:03:37.322519","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:23:15.973296","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"47","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":9,"pool_name":"cephfs_data","create_time":"2021-02-10 11:23:17.045434","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"48","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"be691b4d-a039-408a-af80-9d53f02b6f1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6818","nonce":36088},{"type":"v1","addr":"172.21.2.102:6820","nonce":36088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6822","nonce":36088},{"type":"v1","addr":"172.21.2.102:6824","nonce":36088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6830","nonce":36088},{"type":"v1","addr":"172.21.2.102:6832","nonce":36088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6826","nonce":36088},{"type":"v1","addr":"172.21.2.102:6828","nonce":36088}]},"public_addr":"172.21.2.102:6820/36088","cluster_addr":"172.21.2.102:6824/36088","heartbeat_back_addr":"172.21.2.102:6832/36088","heartbeat_front_addr":"172.21.2.102:6828/36088","state":["exists","up"]},{"osd":1,"uuid":"64e3ccc8-9c69-4d7c-865e-af8e83822635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6819","nonce":36087},{"type":"v1","addr":"172.21.2.102:6821","nonce":36087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6823","nonce":36087},{"type":"v1","addr":"172.21.2.102:6825","nonce":36087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6831","nonce":36087},{"type":"v1","addr":"172.21.2.102:6833","nonce":36087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6827","nonce":36087},{"type":"v1","addr":"172.21.2.102:6829","nonce":36087}]},"public_addr":"172.21.2.102:6821/36087","cluster_addr":"172.21.2.102:6825/36087","heartbeat_back_addr":"172.21.2.102:6833/36087","heartbeat_front_addr":"172.21.2.102:6829/36087","state":["exists","up"]},{"osd":2,"uuid":"153700af-747f-481f-bdf4-d6072d3a842f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6810","nonce":36089},{"type":"v1","addr":"172.21.2.102:6811","nonce":36089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6812","nonce":36089},{"type":"v1","addr":"172.21.2.102:6813","nonce":36089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6816","nonce":36089},{"type":"v1","addr":"172.21.2.102:6817","nonce":36089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6814","nonce":36089},{"type":"v1","addr":"172.21.2.102:6815","nonce":36089}]},"public_addr":"172.21.2.102:6811/36089","cluster_addr":"172.21.2.102:6813/36089","heartbeat_back_addr":"172.21.2.102:6817/36089","heartbeat_front_addr":"172.21.2.102:6815/36089","state":["exists","up"]},{"osd":3,"uuid":"cbc678f4-83a9-45e6-8830-2f7c232b1ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6802","nonce":36090},{"type":"v1","addr":"172.21.2.102:6803","nonce":36090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6804","nonce":36090},{"type":"v1","addr":"172.21.2.102:6805","nonce":36090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6808","nonce":36090},{"type":"v1","addr":"172.21.2.102:6809","nonce":36090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6806","nonce":36090},{"type":"v1","addr":"172.21.2.102:6807","nonce":36090}]},"public_addr":"172.21.2.102:6803/36090","cluster_addr":"172.21.2.102:6805/36090","heartbeat_back_addr":"172.21.2.102:6809/36090","heartbeat_front_addr":"172.21.2.102:6807/36090","state":["exists","up"]},{"osd":4,"uuid":"c01e4008-b258-4f10-afd2-9e7f37214351","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6808","nonce":36368},{"type":"v1","addr":"172.21.2.113:6811","nonce":36368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6814","nonce":36368},{"type":"v1","addr":"172.21.2.113:6818","nonce":36368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6826","nonce":36368},{"type":"v1","addr":"172.21.2.113:6828","nonce":36368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6820","nonce":36368},{"type":"v1","addr":"172.21.2.113:6824","nonce":36368}]},"public_addr":"172.21.2.113:6811/36368","cluster_addr":"172.21.2.113:6818/36368","heartbeat_back_addr":"172.21.2.113:6828/36368","heartbeat_front_addr":"172.21.2.113:6824/36368","state":["exists","up"]},{"osd":5,"uuid":"551f6007-daf8-4a98-8ee0-c2b16bcccb5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6802","nonce":36366},{"type":"v1","addr":"172.21.2.113:6805","nonce":36366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6807","nonce":36366},{"type":"v1","addr":"172.21.2.113:6810","nonce":36366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6819","nonce":36366},{"type":"v1","addr":"172.21.2.113:6822","nonce":36366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6812","nonce":36366},{"type":"v1","addr":"172.21.2.113:6815","nonce":36366}]},"public_addr":"172.21.2.113:6805/36366","cluster_addr":"172.21.2.113:6810/36366","heartbeat_back_addr":"172.21.2.113:6822/36366","heartbeat_front_addr":"172.21.2.113:6815/36366","state":["exists","up"]},{"osd":6,"uuid":"bf67c040-0e50-48be-b102-c3943bc9ebef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6800","nonce":36367},{"type":"v1","addr":"172.21.2.113:6801","nonce":36367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6803","nonce":36367},{"type":"v1","addr":"172.21.2.113:6804","nonce":36367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6813","nonce":36367},{"type":"v1","addr":"172.21.2.113:6816","nonce":36367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6806","nonce":36367},{"type":"v1","addr":"172.21.2.113:6809","nonce":36367}]},"public_addr":"172.21.2.113:6801/36367","cluster_addr":"172.21.2.113:6804/36367","heartbeat_back_addr":"172.21.2.113:6816/36367","heartbeat_front_addr":"172.21.2.113:6809/36367","state":["exists","up"]},{"osd":7,"uuid":"f5461f80-018d-428f-be89-2eb2bcbe7bb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6817","nonce":36365},{"type":"v1","addr":"172.21.2.113:6821","nonce":36365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6823","nonce":36365},{"type":"v1","addr":"172.21.2.113:6825","nonce":36365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6830","nonce":36365},{"type":"v1","addr":"172.21.2.113:6831","nonce":36365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6827","nonce":36365},{"type":"v1","addr":"172.21.2.113:6829","nonce":36365}]},"public_addr":"172.21.2.113:6821/36365","cluster_addr":"172.21.2.113:6825/36365","heartbeat_back_addr":"172.21.2.113:6831/36365","heartbeat_front_addr":"172.21.2.113:6829/36365","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:23:19.209 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:19.211 7f559c2cf700 1 -- 172.21.2.102:0/3217848357 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f557c018700 msgr2=0x7f557c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:19.209 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:19.211 7f559c2cf700 1 --2- 172.21.2.102:0/3217848357 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f557c018700 0x7f557c01abc0 crc :-1 s=READY pgs=566 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:19.210 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:19.211 7f559c2cf700 1 -- 172.21.2.102:0/3217848357 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5594081c50 msgr2=0x7f55940ad5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:19.210 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:19.211 7f559c2cf700 1 --2- 172.21.2.102:0/3217848357 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5594081c50 0x7f55940ad5a0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7f5590006f10 tx=0x7f559000b990).stop 2021-02-10T11:23:19.210 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:19.211 7f559c2cf700 1 -- 172.21.2.102:0/3217848357 shutdown_connections 2021-02-10T11:23:19.210 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:19.211 7f559c2cf700 1 --2- 172.21.2.102:0/3217848357 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5594081c50 0x7f55940ad5a0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:19.210 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:19.211 7f559c2cf700 1 --2- 172.21.2.102:0/3217848357 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f557c018700 0x7f557c01abc0 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:19.210 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:19.211 7f559c2cf700 1 --2- 172.21.2.102:0/3217848357 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5594081850 0x7f55940acf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:19.210 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:19.211 7f559c2cf700 1 --2- 172.21.2.102:0/3217848357 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f559408d400 0x7f5594081230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:19.210 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:19.211 7f559c2cf700 1 -- 172.21.2.102:0/3217848357 >> 172.21.2.102:0/3217848357 conn(0x7f55941ac620 msgr2=0x7f55940896e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:19.210 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:19.212 7f559c2cf700 1 -- 172.21.2.102:0/3217848357 shutdown_connections 2021-02-10T11:23:19.210 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:19.212 7f559c2cf700 1 -- 172.21.2.102:0/3217848357 wait complete. 2021-02-10T11:23:19.218 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2021-02-10T11:23:19.650 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:19.635 7f15ed9a5700 1 Processor -- start 2021-02-10T11:23:19.650 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:19.635 7f15ed9a5700 1 -- start start 2021-02-10T11:23:19.650 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:19.636 7f15ed9a5700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f15e81b0870 0x7f15e80f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:19.650 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:19.636 7f15ed9a5700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f15e80efe00 0x7f15e80f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:19.651 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:19.636 7f15ed9a5700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15e81b0c70 con 0x7f15e81b0470 2021-02-10T11:23:19.651 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:19.636 7f15ed9a5700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15e80f0b50 con 0x7f15e80efe00 2021-02-10T11:23:19.651 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:19.636 7f15ed9a5700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15e80d2fb0 con 0x7f15e81b0870 2021-02-10T11:23:19.651 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:19.636 7f15e7574700 1 --1- >> v1:172.21.2.113:6789/0 conn(0x7f15e81b0470 0x7f15e80ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.113:6789/0 says I am v1:172.21.2.102:55768/0 (socket says 172.21.2.102:55768) 2021-02-10T11:23:19.651 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:19.636 7f15e7574700 1 -- 172.21.2.102:0/2149193247 learned_addr learned my addr 172.21.2.102:0/2149193247 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:23:19.651 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:19.636 7f15e7d75700 1 --2- 172.21.2.102:0/2149193247 >> v2:172.21.2.113:3300/0 conn(0x7f15e80efe00 0x7f15e80f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:19.651 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:19.636 7f15ded73700 1 --2- 172.21.2.102:0/2149193247 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f15e81b0870 0x7f15e80f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:19.651 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:19.637 7f15e7d75700 1 -- 172.21.2.102:0/2149193247 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f15e81b0870 msgr2=0x7f15e80f20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:19.651 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:19.637 7f15e7d75700 1 --2- 172.21.2.102:0/2149193247 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f15e81b0870 0x7f15e80f20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:19.651 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:19.637 7f15e6d73700 1 -- 172.21.2.102:0/2149193247 <== mon.1 v1:172.21.2.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2749287461 0 0) 0x7f15d8001440 con 0x7f15e81b0470 2021-02-10T11:23:19.651 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:19.637 7f15e7d75700 1 -- 172.21.2.102:0/2149193247 >> v1:172.21.2.113:6789/0 conn(0x7f15e81b0470 legacy=0x7f15e80ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:19.651 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:19.637 7f15e7d75700 1 -- 172.21.2.102:0/2149193247 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15e80d3330 con 0x7f15e80efe00 2021-02-10T11:23:19.651 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:19.637 7f15e7d75700 1 --2- 172.21.2.102:0/2149193247 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f15e80efe00 0x7f15e80f0200 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 rx=0x7f15e0006ad0 tx=0x7f15e0009ad0).ready entity=mon.1 client_cookie=5c2e482a545000f0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:19.652 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:19.637 7f15e6d73700 1 -- 172.21.2.102:0/2149193247 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f15e00022e0 con 0x7f15e80efe00 2021-02-10T11:23:19.652 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:19.637 7f15e6d73700 1 -- 172.21.2.102:0/2149193247 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15e0006410 con 0x7f15e80efe00 2021-02-10T11:23:19.652 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:19.637 7f15e6d73700 1 -- 172.21.2.102:0/2149193247 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f15e0002620 con 0x7f15e80efe00 2021-02-10T11:23:19.652 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:19.637 7f15ed9a5700 1 -- 172.21.2.102:0/2149193247 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f15e80efe00 msgr2=0x7f15e80f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:19.652 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:19.637 7f15ed9a5700 1 --2- 172.21.2.102:0/2149193247 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f15e80efe00 0x7f15e80f0200 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 rx=0x7f15e0006ad0 tx=0x7f15e0009ad0).stop 2021-02-10T11:23:19.652 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:19.638 7f15ed9a5700 1 -- 172.21.2.102:0/2149193247 shutdown_connections 2021-02-10T11:23:19.652 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:19.638 7f15ed9a5700 1 --2- 172.21.2.102:0/2149193247 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f15e81b0870 0x7f15e80f20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:19.652 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:19.638 7f15ed9a5700 1 --2- 172.21.2.102:0/2149193247 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f15e80efe00 0x7f15e80f0200 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:19.652 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:19.638 7f15ed9a5700 1 -- 172.21.2.102:0/2149193247 >> 172.21.2.102:0/2149193247 conn(0x7f15e80ce3e0 msgr2=0x7f15e80e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:19.652 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:19.638 7f15ed9a5700 1 -- 172.21.2.102:0/2149193247 shutdown_connections 2021-02-10T11:23:19.652 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:19.638 7f15ed9a5700 1 -- 172.21.2.102:0/2149193247 wait complete. 2021-02-10T11:23:19.654 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:19.653 7f15ed9a5700 1 Processor -- start 2021-02-10T11:23:19.654 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:19.653 7f15ed9a5700 1 -- start start 2021-02-10T11:23:19.654 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:19.653 7f15ed9a5700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f15e808ece0 0x7f15e80a6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:19.654 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:19.653 7f15ed9a5700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f15e80a6860 0x7f15e80a6c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:19.655 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:19.653 7f15ed9a5700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f15e80a7280 0x7f15e80a4590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:19.655 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:19.653 7f15ed9a5700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15e80a5cc0 con 0x7f15e80a6860 2021-02-10T11:23:19.655 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:19.653 7f15ed9a5700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15e808f0e0 con 0x7f15e808ece0 2021-02-10T11:23:19.655 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:19.653 7f15ed9a5700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15e808f2c0 con 0x7f15e80a7280 2021-02-10T11:23:19.655 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:19.654 7f15e7574700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f15e808ece0 0x7f15e80a6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:19.655 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:19.654 7f15e7574700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f15e808ece0 0x7f15e80a6240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.113:3300/0 says I am v2:172.21.2.102:59656/0 (socket says 172.21.2.102:59656) 2021-02-10T11:23:19.655 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:19.654 7f15e7d75700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f15e80a7280 0x7f15e80a4590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:19.655 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:19.654 7f15ded73700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f15e80a6860 0x7f15e80a6c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:19.655 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:19.654 7f15e7574700 1 -- 172.21.2.102:0/2749530944 learned_addr learned my addr 172.21.2.102:0/2749530944 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:23:19.655 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:19.654 7f15ded73700 1 -- 172.21.2.102:0/2749530944 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f15e80a7280 msgr2=0x7f15e80a4590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:19.655 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:19.654 7f15ded73700 1 --2- 172.21.2.102:0/2749530944 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f15e80a7280 0x7f15e80a4590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:19.655 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:19.654 7f15ded73700 1 -- 172.21.2.102:0/2749530944 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f15e808ece0 msgr2=0x7f15e80a6240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:19.655 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:19.654 7f15ded73700 1 --2- 172.21.2.102:0/2749530944 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f15e808ece0 0x7f15e80a6240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:19.656 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:19.654 7f15ded73700 1 -- 172.21.2.102:0/2749530944 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15e80a4bb0 con 0x7f15e80a6860 2021-02-10T11:23:19.656 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:19.654 7f15e7d75700 1 --2- 172.21.2.102:0/2749530944 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f15e80a7280 0x7f15e80a4590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:23:19.656 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:19.654 7f15e7574700 1 --2- 172.21.2.102:0/2749530944 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f15e808ece0 0x7f15e80a6240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:23:19.656 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:19.654 7f15ded73700 1 --2- 172.21.2.102:0/2749530944 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f15e80a6860 0x7f15e80a6c60 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 rx=0x7f15d0009540 tx=0x7f15d000b8e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:19.656 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:19.654 7f15e4d6f700 1 -- 172.21.2.102:0/2749530944 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f15d00066d0 con 0x7f15e80a6860 2021-02-10T11:23:19.656 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:19.654 7f15e4d6f700 1 -- 172.21.2.102:0/2749530944 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15d000bd60 con 0x7f15e80a6860 2021-02-10T11:23:19.656 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:19.654 7f15ed9a5700 1 -- 172.21.2.102:0/2749530944 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15e80a4bb0 con 0x7f15e80a6860 2021-02-10T11:23:19.656 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:19.654 7f15e4d6f700 1 -- 172.21.2.102:0/2749530944 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f15d0002220 con 0x7f15e80a6860 2021-02-10T11:23:19.656 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:19.655 7f15ed9a5700 1 -- 172.21.2.102:0/2749530944 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15e80a4bb0 con 0x7f15e80a6860 2021-02-10T11:23:19.656 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:19.655 7f15e4d6f700 1 -- 172.21.2.102:0/2749530944 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f15d0000c90 con 0x7f15e80a6860 2021-02-10T11:23:19.656 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:19.655 7f15e4d6f700 1 --2- 172.21.2.102:0/2749530944 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f15cc0186d0 0x7f15cc01abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:19.656 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:19.655 7f15e4d6f700 1 -- 172.21.2.102:0/2749530944 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f15d00325a0 con 0x7f15e80a6860 2021-02-10T11:23:19.656 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:19.655 7f15e7574700 1 --2- 172.21.2.102:0/2749530944 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f15cc0186d0 0x7f15cc01abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:19.657 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:19.656 7f15e7574700 1 --2- 172.21.2.102:0/2749530944 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f15cc0186d0 0x7f15cc01abd0 crc :-1 s=READY pgs=567 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:19.682 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:19.685 7f15ed9a5700 1 -- 172.21.2.102:0/2749530944 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15e8105ae0 con 0x7f15e80a6860 2021-02-10T11:23:19.685 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:19.687 7f15e4d6f700 1 -- 172.21.2.102:0/2749530944 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f15d00066d0 con 0x7f15e80a6860 2021-02-10T11:23:19.822 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:19.824 7f15ed9a5700 1 -- 172.21.2.102:0/2749530944 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"} v 0) v1 -- 0x7f15e80f0b50 con 0x7f15e80a6860 2021-02-10T11:23:20.010 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:20.012 7f15e4d6f700 1 -- 172.21.2.102:0/2749530944 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"}]=0 v30) v1 ==== 118+0+0 (secure 0 0 0) 0x7f15d0000c90 con 0x7f15e80a6860 2021-02-10T11:23:20.039 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:20.041 7f15ed9a5700 1 -- 172.21.2.102:0/2749530944 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f15cc0186d0 msgr2=0x7f15cc01abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:20.039 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:20.041 7f15ed9a5700 1 --2- 172.21.2.102:0/2749530944 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f15cc0186d0 0x7f15cc01abd0 crc :-1 s=READY pgs=567 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:20.039 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:20.041 7f15ed9a5700 1 -- 172.21.2.102:0/2749530944 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f15e80a6860 msgr2=0x7f15e80a6c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:20.039 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:20.041 7f15ed9a5700 1 --2- 172.21.2.102:0/2749530944 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f15e80a6860 0x7f15e80a6c60 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 rx=0x7f15d0009540 tx=0x7f15d000b8e0).stop 2021-02-10T11:23:20.039 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:20.041 7f15ed9a5700 1 -- 172.21.2.102:0/2749530944 shutdown_connections 2021-02-10T11:23:20.039 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:20.041 7f15ed9a5700 1 --2- 172.21.2.102:0/2749530944 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f15e80a7280 0x7f15e80a4590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:20.040 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:20.041 7f15ed9a5700 1 --2- 172.21.2.102:0/2749530944 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f15e80a6860 0x7f15e80a6c60 unknown :-1 s=CLOSED pgs=1187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:20.040 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:20.041 7f15ed9a5700 1 --2- 172.21.2.102:0/2749530944 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f15cc0186d0 0x7f15cc01abd0 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:20.040 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:20.041 7f15ed9a5700 1 --2- 172.21.2.102:0/2749530944 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f15e808ece0 0x7f15e80a6240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:20.040 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:20.041 7f15ed9a5700 1 -- 172.21.2.102:0/2749530944 >> 172.21.2.102:0/2749530944 conn(0x7f15e808ffe0 msgr2=0x7f15e80bf050 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:20.040 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:20.041 7f15ed9a5700 1 -- 172.21.2.102:0/2749530944 shutdown_connections 2021-02-10T11:23:20.040 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:20.041 7f15ed9a5700 1 -- 172.21.2.102:0/2749530944 wait complete. 2021-02-10T11:23:20.047 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:23:20.474 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:20.458 7fa99e6d4700 1 Processor -- start 2021-02-10T11:23:20.474 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:20.458 7fa99e6d4700 1 -- start start 2021-02-10T11:23:20.474 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:20.459 7fa99e6d4700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fa9980e5070 0x7fa9981b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:20.474 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:20.459 7fa99e6d4700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa99807ed80 con 0x7fa9981b3e40 2021-02-10T11:23:20.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:20.459 7fa99e6d4700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa99807e670 con 0x7fa9980e5470 2021-02-10T11:23:20.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:20.459 7fa99e6d4700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa99807e890 con 0x7fa9980e5070 2021-02-10T11:23:20.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:20.459 7fa9977fe700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fa9981b3e40 0x7fa99807d690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:48458/0 (socket says 172.21.2.102:48458) 2021-02-10T11:23:20.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:20.459 7fa9977fe700 1 -- 172.21.2.102:0/1632023221 learned_addr learned my addr 172.21.2.102:0/1632023221 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:23:20.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:20.459 7fa996ffd700 1 -- 172.21.2.102:0/1632023221 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 45962251 0 0) 0x7fa98c0010b0 con 0x7fa9981b3e40 2021-02-10T11:23:20.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:20.459 7fa996ffd700 1 -- 172.21.2.102:0/1632023221 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa980002fe0 con 0x7fa9981b3e40 2021-02-10T11:23:20.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:20.459 7fa997fff700 1 --2- 172.21.2.102:0/1632023221 >> v2:172.21.2.113:3300/0 conn(0x7fa9980e5070 0x7fa9981b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:20.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:20.460 7fa996ffd700 1 -- 172.21.2.102:0/1632023221 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 278315171 0 0) 0x7fa98c0010b0 con 0x7fa9981b3e40 2021-02-10T11:23:20.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:20.460 7fa996ffd700 1 -- 172.21.2.102:0/1632023221 >> v2:172.21.2.113:3300/0 conn(0x7fa9980e5070 msgr2=0x7fa9981b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:20.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:20.460 7fa996ffd700 1 --2- 172.21.2.102:0/1632023221 >> v2:172.21.2.113:3300/0 conn(0x7fa9980e5070 0x7fa9981b3820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:20.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:20.460 7fa996ffd700 1 -- 172.21.2.102:0/1632023221 >> v1:172.21.2.113:6789/0 conn(0x7fa9980e5470 legacy=0x7fa99807de80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:20.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:20.460 7fa996ffd700 1 -- 172.21.2.102:0/1632023221 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa99807eab0 con 0x7fa9981b3e40 2021-02-10T11:23:20.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:20.460 7fa996ffd700 1 -- 172.21.2.102:0/1632023221 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7fa98c0015b0 con 0x7fa9981b3e40 2021-02-10T11:23:20.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:20.460 7fa996ffd700 1 -- 172.21.2.102:0/1632023221 >> v1:172.21.2.102:6789/0 conn(0x7fa9981b3e40 legacy=0x7fa99807d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:20.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:20.460 7fa996ffd700 1 --2- 172.21.2.102:0/1632023221 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa980003ac0 0x7fa980005ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:20.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:20.460 7fa996ffd700 1 --2- 172.21.2.102:0/1632023221 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa980006790 0x7fa980008c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:20.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:20.460 7fa996ffd700 1 --2- 172.21.2.102:0/1632023221 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa980009590 0x7fa98000bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:20.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:20.460 7fa996ffd700 1 -- 172.21.2.102:0/1632023221 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa98000c300 con 0x7fa980009590 2021-02-10T11:23:20.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:20.460 7fa996ffd700 1 -- 172.21.2.102:0/1632023221 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa98000c520 con 0x7fa980006790 2021-02-10T11:23:20.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:20.460 7fa996ffd700 1 -- 172.21.2.102:0/1632023221 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa980050750 con 0x7fa980003ac0 2021-02-10T11:23:20.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:20.460 7fa997fff700 1 --2- 172.21.2.102:0/1632023221 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa980003ac0 0x7fa980005ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:20.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:20.460 7fa99ccf1700 1 --2- 172.21.2.102:0/1632023221 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa980009590 0x7fa98000bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:20.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:20.460 7fa9977fe700 1 --2- 172.21.2.102:0/1632023221 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa980006790 0x7fa980008c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:20.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:20.460 7fa997fff700 1 -- 172.21.2.102:0/1632023221 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa980006790 msgr2=0x7fa980008c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:20.477 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:20.461 7fa997fff700 1 --2- 172.21.2.102:0/1632023221 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa980006790 0x7fa980008c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:20.477 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:20.461 7fa997fff700 1 -- 172.21.2.102:0/1632023221 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa980009590 msgr2=0x7fa98000bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:20.477 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:20.461 7fa997fff700 1 --2- 172.21.2.102:0/1632023221 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa980009590 0x7fa98000bab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:20.477 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:20.461 7fa997fff700 1 -- 172.21.2.102:0/1632023221 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9800509b0 con 0x7fa980003ac0 2021-02-10T11:23:20.477 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:20.461 7fa99ccf1700 1 -- 172.21.2.102:0/1632023221 reap_dead start 2021-02-10T11:23:20.477 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:20.461 7fa997fff700 1 --2- 172.21.2.102:0/1632023221 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa980003ac0 0x7fa980005ef0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7fa98800c8f0 tx=0x7fa98800cc90).ready entity=mon.2 client_cookie=8d6c931f70ae0237 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:20.477 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:20.461 7fa996ffd700 1 -- 172.21.2.102:0/1632023221 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa988009310 con 0x7fa980003ac0 2021-02-10T11:23:20.477 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:20.461 7fa996ffd700 1 -- 172.21.2.102:0/1632023221 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa988009610 con 0x7fa980003ac0 2021-02-10T11:23:20.477 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:20.461 7fa996ffd700 1 -- 172.21.2.102:0/1632023221 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa988009b30 con 0x7fa980003ac0 2021-02-10T11:23:20.477 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:20.461 7fa99e6d4700 1 -- 172.21.2.102:0/1632023221 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa980003ac0 msgr2=0x7fa980005ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:20.477 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:20.461 7fa99e6d4700 1 --2- 172.21.2.102:0/1632023221 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa980003ac0 0x7fa980005ef0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7fa98800c8f0 tx=0x7fa98800cc90).stop 2021-02-10T11:23:20.477 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:20.462 7fa99e6d4700 1 -- 172.21.2.102:0/1632023221 shutdown_connections 2021-02-10T11:23:20.477 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:20.462 7fa99e6d4700 1 --2- 172.21.2.102:0/1632023221 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa980003ac0 0x7fa980005ef0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:20.478 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:20.462 7fa99e6d4700 1 -- 172.21.2.102:0/1632023221 >> 172.21.2.102:0/1632023221 conn(0x7fa9980ce690 msgr2=0x7fa9980e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:20.478 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:20.462 7fa99e6d4700 1 -- 172.21.2.102:0/1632023221 shutdown_connections 2021-02-10T11:23:20.478 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:20.462 7fa99e6d4700 1 -- 172.21.2.102:0/1632023221 wait complete. 2021-02-10T11:23:20.478 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:20.476 7fa99e6d4700 1 Processor -- start 2021-02-10T11:23:20.478 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:20.476 7fa99e6d4700 1 -- start start 2021-02-10T11:23:20.479 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:20.477 7fa99e6d4700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa99819acd0 0x7fa9980a03d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:20.480 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:20.477 7fa99e6d4700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa9980a09f0 0x7fa9980b3ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:20.480 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:20.477 7fa99e6d4700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa9980b40e0 0x7fa9980b44e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:20.480 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:20.477 7fa99e6d4700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9980a1240 con 0x7fa99819acd0 2021-02-10T11:23:20.480 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:20.477 7fa99e6d4700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa99819b1e0 con 0x7fa9980a09f0 2021-02-10T11:23:20.480 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:20.477 7fa99e6d4700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9980a0df0 con 0x7fa9980b40e0 2021-02-10T11:23:20.480 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:20.477 7fa997fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa99819acd0 0x7fa9980a03d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:20.480 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:20.477 7fa997fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa99819acd0 0x7fa9980a03d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:56242/0 (socket says 172.21.2.102:56242) 2021-02-10T11:23:20.480 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:20.477 7fa997fff700 1 -- 172.21.2.102:0/2528506737 learned_addr learned my addr 172.21.2.102:0/2528506737 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:23:20.480 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:20.477 7fa997fff700 1 -- 172.21.2.102:0/2528506737 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa9980b40e0 msgr2=0x7fa9980b44e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:20.480 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:20.477 7fa997fff700 1 --2- 172.21.2.102:0/2528506737 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa9980b40e0 0x7fa9980b44e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:20.480 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:20.477 7fa997fff700 1 -- 172.21.2.102:0/2528506737 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa9980a09f0 msgr2=0x7fa9980b3ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:20.481 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:20.477 7fa997fff700 1 --2- 172.21.2.102:0/2528506737 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa9980a09f0 0x7fa9980b3ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:20.481 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:20.477 7fa997fff700 1 -- 172.21.2.102:0/2528506737 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9980a1740 con 0x7fa99819acd0 2021-02-10T11:23:20.481 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:20.478 7fa997fff700 1 --2- 172.21.2.102:0/2528506737 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa99819acd0 0x7fa9980a03d0 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 rx=0x7fa988006fc0 tx=0x7fa98800b300).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:20.481 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:20.478 7fa994ff9700 1 -- 172.21.2.102:0/2528506737 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa988006030 con 0x7fa99819acd0 2021-02-10T11:23:20.481 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:20.478 7fa994ff9700 1 -- 172.21.2.102:0/2528506737 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa988006330 con 0x7fa99819acd0 2021-02-10T11:23:20.481 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:20.478 7fa994ff9700 1 -- 172.21.2.102:0/2528506737 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa98800c960 con 0x7fa99819acd0 2021-02-10T11:23:20.481 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:20.478 7fa99e6d4700 1 -- 172.21.2.102:0/2528506737 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9980a1740 con 0x7fa99819acd0 2021-02-10T11:23:20.481 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:20.478 7fa99e6d4700 1 -- 172.21.2.102:0/2528506737 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9980a1740 con 0x7fa99819acd0 2021-02-10T11:23:20.481 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:20.479 7fa994ff9700 1 -- 172.21.2.102:0/2528506737 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa98800b6f0 con 0x7fa99819acd0 2021-02-10T11:23:20.481 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:20.479 7fa994ff9700 1 --2- 172.21.2.102:0/2528506737 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa9780186d0 0x7fa97801abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:20.481 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:20.479 7fa994ff9700 1 -- 172.21.2.102:0/2528506737 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fa988032610 con 0x7fa99819acd0 2021-02-10T11:23:20.481 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:20.479 7fa9977fe700 1 --2- 172.21.2.102:0/2528506737 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa9780186d0 0x7fa97801abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:20.481 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:20.479 7fa9977fe700 1 --2- 172.21.2.102:0/2528506737 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa9780186d0 0x7fa97801abd0 crc :-1 s=READY pgs=568 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:20.506 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:20.508 7fa99e6d4700 1 -- 172.21.2.102:0/2528506737 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa99807ed80 con 0x7fa99819acd0 2021-02-10T11:23:20.509 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:20.511 7fa994ff9700 1 -- 172.21.2.102:0/2528506737 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fa988006030 con 0x7fa99819acd0 2021-02-10T11:23:20.637 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:20.640 7fa99e6d4700 1 -- 172.21.2.102:0/2528506737 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa99807ed80 con 0x7fa99819acd0 2021-02-10T11:23:20.638 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:20.640 7fa994ff9700 1 -- 172.21.2.102:0/2528506737 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+2099 (secure 0 0 0) 0x7fa9880067c0 con 0x7fa99819acd0 2021-02-10T11:23:20.638 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:23:20.639 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":30,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7557,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.102:6835/1941781958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6834","nonce":1941781958},{"type":"v1","addr":"172.21.2.102:6835","nonce":1941781958}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":26}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:23:18.002680","modified":"2021-02-10 11:23:20.009191","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5606},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5606":{"gid":5606,"name":"b","rank":0,"incarnation":27,"state":"up:active","state_seq":3,"addr":"172.21.2.113:6833/1872558991","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6832","nonce":1872558991},{"type":"v1","addr":"172.21.2.113:6833","nonce":1872558991}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":4}]} 2021-02-10T11:23:20.639 INFO:teuthology.orchestra.run.gibba002.stderr:dumped fsmap epoch 30 2021-02-10T11:23:20.640 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:20.642 7fa99e6d4700 1 -- 172.21.2.102:0/2528506737 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa9780186d0 msgr2=0x7fa97801abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:20.640 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:20.642 7fa99e6d4700 1 --2- 172.21.2.102:0/2528506737 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa9780186d0 0x7fa97801abd0 crc :-1 s=READY pgs=568 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:20.640 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:20.642 7fa99e6d4700 1 -- 172.21.2.102:0/2528506737 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa99819acd0 msgr2=0x7fa9980a03d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:20.640 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:20.642 7fa99e6d4700 1 --2- 172.21.2.102:0/2528506737 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa99819acd0 0x7fa9980a03d0 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 rx=0x7fa988006fc0 tx=0x7fa98800b300).stop 2021-02-10T11:23:20.640 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:20.642 7fa99e6d4700 1 -- 172.21.2.102:0/2528506737 shutdown_connections 2021-02-10T11:23:20.640 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:20.642 7fa99e6d4700 1 --2- 172.21.2.102:0/2528506737 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa9980b40e0 0x7fa9980b44e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:20.641 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:20.642 7fa99e6d4700 1 --2- 172.21.2.102:0/2528506737 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa9780186d0 0x7fa97801abd0 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:20.641 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:20.642 7fa99e6d4700 1 --2- 172.21.2.102:0/2528506737 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa9980a09f0 0x7fa9980b3ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:20.641 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:20.642 7fa99e6d4700 1 --2- 172.21.2.102:0/2528506737 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa99819acd0 0x7fa9980a03d0 unknown :-1 s=CLOSED pgs=1189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:20.641 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:20.642 7fa99e6d4700 1 -- 172.21.2.102:0/2528506737 >> 172.21.2.102:0/2528506737 conn(0x7fa9981a31d0 msgr2=0x7fa99808b830 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:20.641 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:20.642 7fa99e6d4700 1 -- 172.21.2.102:0/2528506737 shutdown_connections 2021-02-10T11:23:20.641 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:20.643 7fa99e6d4700 1 -- 172.21.2.102:0/2528506737 wait complete. 2021-02-10T11:23:20.649 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:23:21.078 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:21.063 7f30621fe700 1 Processor -- start 2021-02-10T11:23:21.078 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:21.063 7f30621fe700 1 -- start start 2021-02-10T11:23:21.078 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:21.063 7f30621fe700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f305c0e5070 0x7f305c1b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:21.078 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:21.063 7f30621fe700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f305c1b3e40 0x7f305c07d690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:21.078 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:21.063 7f30621fe700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f305c07ed80 con 0x7f305c0e5470 2021-02-10T11:23:21.078 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:21.063 7f30621fe700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f305c080310 con 0x7f305c0e5070 2021-02-10T11:23:21.078 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:21.063 7f30621fe700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f305c080500 con 0x7f305c1b3e40 2021-02-10T11:23:21.078 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:21.064 7f306081b700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f305c0e5470 0x7f305c07dcb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:48474/0 (socket says 172.21.2.102:48474) 2021-02-10T11:23:21.078 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:21.064 7f306081b700 1 -- 172.21.2.102:0/844947015 learned_addr learned my addr 172.21.2.102:0/844947015 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:23:21.079 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:21.064 7f305affd700 1 -- 172.21.2.102:0/844947015 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2764528102 0 0) 0x7f3054000f90 con 0x7f305c0e5470 2021-02-10T11:23:21.079 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:21.064 7f305bfff700 1 --2- 172.21.2.102:0/844947015 >> v2:172.21.2.113:3300/0 conn(0x7f305c0e5070 0x7f305c1b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:21.079 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:21.064 7f305b7fe700 1 --2- 172.21.2.102:0/844947015 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f305c1b3e40 0x7f305c07d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:21.079 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:21.064 7f305affd700 1 -- 172.21.2.102:0/844947015 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3044002fe0 con 0x7f305c0e5470 2021-02-10T11:23:21.079 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:21.064 7f305affd700 1 -- 172.21.2.102:0/844947015 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3026048740 0 0) 0x7f3054001500 con 0x7f305c0e5470 2021-02-10T11:23:21.079 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:21.064 7f305b7fe700 1 -- 172.21.2.102:0/844947015 >> v2:172.21.2.113:3300/0 conn(0x7f305c0e5070 msgr2=0x7f305c1b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:21.079 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:21.064 7f305b7fe700 1 --2- 172.21.2.102:0/844947015 >> v2:172.21.2.113:3300/0 conn(0x7f305c0e5070 0x7f305c1b3820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:21.079 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:21.064 7f305b7fe700 1 -- 172.21.2.102:0/844947015 >> v1:172.21.2.102:6789/0 conn(0x7f305c0e5470 legacy=0x7f305c07dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:21.079 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:21.064 7f305b7fe700 1 -- 172.21.2.102:0/844947015 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f305c0806f0 con 0x7f305c1b3e40 2021-02-10T11:23:21.079 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:21.065 7f305bfff700 1 --2- 172.21.2.102:0/844947015 >> v2:172.21.2.113:3300/0 conn(0x7f305c0e5070 0x7f305c1b3820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:23:21.079 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:21.065 7f305b7fe700 1 --2- 172.21.2.102:0/844947015 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f305c1b3e40 0x7f305c07d690 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f3050006650 tx=0x7f3050009a80).ready entity=mon.2 client_cookie=c37daa06505790bf server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:21.079 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:21.065 7f305affd700 1 -- 172.21.2.102:0/844947015 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f305000c0c0 con 0x7f305c1b3e40 2021-02-10T11:23:21.080 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:21.065 7f305affd700 1 -- 172.21.2.102:0/844947015 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f305000c0c0 con 0x7f305c1b3e40 2021-02-10T11:23:21.080 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:21.065 7f305affd700 1 -- 172.21.2.102:0/844947015 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f305000c340 con 0x7f305c1b3e40 2021-02-10T11:23:21.080 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:21.065 7f30621fe700 1 -- 172.21.2.102:0/844947015 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f305c1b3e40 msgr2=0x7f305c07d690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:21.080 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:21.065 7f30621fe700 1 --2- 172.21.2.102:0/844947015 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f305c1b3e40 0x7f305c07d690 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f3050006650 tx=0x7f3050009a80).stop 2021-02-10T11:23:21.080 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:21.066 7f30621fe700 1 -- 172.21.2.102:0/844947015 shutdown_connections 2021-02-10T11:23:21.080 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:21.066 7f30621fe700 1 --2- 172.21.2.102:0/844947015 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f305c1b3e40 0x7f305c07d690 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:21.080 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:21.066 7f30621fe700 1 --2- 172.21.2.102:0/844947015 >> v2:172.21.2.113:3300/0 conn(0x7f305c0e5070 0x7f305c1b3820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:21.080 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:21.066 7f30621fe700 1 -- 172.21.2.102:0/844947015 >> 172.21.2.102:0/844947015 conn(0x7f305c0ce690 msgr2=0x7f305c0e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:21.080 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:21.066 7f30621fe700 1 -- 172.21.2.102:0/844947015 shutdown_connections 2021-02-10T11:23:21.080 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:21.066 7f30621fe700 1 -- 172.21.2.102:0/844947015 wait complete. 2021-02-10T11:23:21.082 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:21.080 7f30621fe700 1 Processor -- start 2021-02-10T11:23:21.082 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:21.080 7f30621fe700 1 -- start start 2021-02-10T11:23:21.082 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:21.081 7f30621fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f305c191bb0 0x7f305c0a2380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:21.082 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:21.081 7f30621fe700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f305c0a29a0 0x7f305c0a2da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:21.082 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:21.081 7f30621fe700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f305c0a33c0 0x7f305c0a03d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:21.082 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:21.081 7f30621fe700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f305c0adf90 con 0x7f305c191bb0 2021-02-10T11:23:21.082 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:21.081 7f30621fe700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f305c0a1e00 con 0x7f305c0a29a0 2021-02-10T11:23:21.082 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:21.081 7f30621fe700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f305c0a09f0 con 0x7f305c0a33c0 2021-02-10T11:23:21.082 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:21.081 7f305bfff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f305c191bb0 0x7f305c0a2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:21.083 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:21.081 7f305bfff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f305c191bb0 0x7f305c0a2380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:56254/0 (socket says 172.21.2.102:56254) 2021-02-10T11:23:21.083 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:21.081 7f305bfff700 1 -- 172.21.2.102:0/2518411503 learned_addr learned my addr 172.21.2.102:0/2518411503 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:23:21.083 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:21.081 7f305b7fe700 1 --2- 172.21.2.102:0/2518411503 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f305c0a29a0 0x7f305c0a2da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:21.083 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:21.081 7f306081b700 1 --2- 172.21.2.102:0/2518411503 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f305c0a33c0 0x7f305c0a03d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:21.083 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:21.081 7f305bfff700 1 -- 172.21.2.102:0/2518411503 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f305c0a33c0 msgr2=0x7f305c0a03d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:21.083 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:21.081 7f305bfff700 1 --2- 172.21.2.102:0/2518411503 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f305c0a33c0 0x7f305c0a03d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:21.083 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:21.081 7f305bfff700 1 -- 172.21.2.102:0/2518411503 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f305c0a29a0 msgr2=0x7f305c0a2da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:21.083 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:21.081 7f305bfff700 1 --2- 172.21.2.102:0/2518411503 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f305c0a29a0 0x7f305c0a2da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:21.083 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:21.081 7f305bfff700 1 -- 172.21.2.102:0/2518411503 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f305c0a0bd0 con 0x7f305c191bb0 2021-02-10T11:23:21.083 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:21.081 7f306081b700 1 --2- 172.21.2.102:0/2518411503 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f305c0a33c0 0x7f305c0a03d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:23:21.083 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:21.081 7f305b7fe700 1 --2- 172.21.2.102:0/2518411503 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f305c0a29a0 0x7f305c0a2da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:23:21.084 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:21.082 7f305bfff700 1 --2- 172.21.2.102:0/2518411503 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f305c191bb0 0x7f305c0a2380 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 rx=0x7f304c006f40 tx=0x7f304c00b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:21.084 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:21.082 7f3058ff9700 1 -- 172.21.2.102:0/2518411503 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f304c0009b0 con 0x7f305c191bb0 2021-02-10T11:23:21.084 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:21.082 7f3058ff9700 1 -- 172.21.2.102:0/2518411503 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f304c00bd80 con 0x7f305c191bb0 2021-02-10T11:23:21.084 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:21.082 7f30621fe700 1 -- 172.21.2.102:0/2518411503 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f305c0a0bd0 con 0x7f305c191bb0 2021-02-10T11:23:21.084 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:21.082 7f3058ff9700 1 -- 172.21.2.102:0/2518411503 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f304c002220 con 0x7f305c191bb0 2021-02-10T11:23:21.084 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:21.082 7f30621fe700 1 -- 172.21.2.102:0/2518411503 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f305c0a0bd0 con 0x7f305c191bb0 2021-02-10T11:23:21.084 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:21.082 7f3058ff9700 1 -- 172.21.2.102:0/2518411503 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f304c00a000 con 0x7f305c191bb0 2021-02-10T11:23:21.084 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:21.082 7f3058ff9700 1 --2- 172.21.2.102:0/2518411503 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f303c018730 0x7f303c01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:21.084 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:21.082 7f3058ff9700 1 -- 172.21.2.102:0/2518411503 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f304c032660 con 0x7f305c191bb0 2021-02-10T11:23:21.084 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:21.083 7f305b7fe700 1 --2- 172.21.2.102:0/2518411503 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f303c018730 0x7f303c01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:21.084 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:21.083 7f305b7fe700 1 --2- 172.21.2.102:0/2518411503 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f303c018730 0x7f303c01ac10 crc :-1 s=READY pgs=569 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:21.111 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:21.113 7f30621fe700 1 -- 172.21.2.102:0/2518411503 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f305c0a1e00 con 0x7f305c191bb0 2021-02-10T11:23:21.114 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:21.116 7f3058ff9700 1 -- 172.21.2.102:0/2518411503 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f304c00a000 con 0x7f305c191bb0 2021-02-10T11:23:21.241 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:21.243 7f30621fe700 1 -- 172.21.2.102:0/2518411503 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f305c0a1e00 con 0x7f305c191bb0 2021-02-10T11:23:21.241 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:21.243 7f3058ff9700 1 -- 172.21.2.102:0/2518411503 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v48) v1 ==== 74+0+13993 (secure 0 0 0) 0x7f304c006b00 con 0x7f305c191bb0 2021-02-10T11:23:21.242 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:23:21.242 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":48,"fsid":"68db30ea-dbce-4aef-a6df-f01c6fea0513","created":"2021-02-10 11:03:06.331557","modified":"2021-02-10 11:23:19.005019","last_up_change":"2021-02-10 11:03:32.690562","last_in_change":"2021-02-10 11:03:32.690562","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:03:37.322519","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:23:15.973296","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"47","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":9,"pool_name":"cephfs_data","create_time":"2021-02-10 11:23:17.045434","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"48","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"be691b4d-a039-408a-af80-9d53f02b6f1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6818","nonce":36088},{"type":"v1","addr":"172.21.2.102:6820","nonce":36088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6822","nonce":36088},{"type":"v1","addr":"172.21.2.102:6824","nonce":36088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6830","nonce":36088},{"type":"v1","addr":"172.21.2.102:6832","nonce":36088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6826","nonce":36088},{"type":"v1","addr":"172.21.2.102:6828","nonce":36088}]},"public_addr":"172.21.2.102:6820/36088","cluster_addr":"172.21.2.102:6824/36088","heartbeat_back_addr":"172.21.2.102:6832/36088","heartbeat_front_addr":"172.21.2.102:6828/36088","state":["exists","up"]},{"osd":1,"uuid":"64e3ccc8-9c69-4d7c-865e-af8e83822635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6819","nonce":36087},{"type":"v1","addr":"172.21.2.102:6821","nonce":36087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6823","nonce":36087},{"type":"v1","addr":"172.21.2.102:6825","nonce":36087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6831","nonce":36087},{"type":"v1","addr":"172.21.2.102:6833","nonce":36087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6827","nonce":36087},{"type":"v1","addr":"172.21.2.102:6829","nonce":36087}]},"public_addr":"172.21.2.102:6821/36087","cluster_addr":"172.21.2.102:6825/36087","heartbeat_back_addr":"172.21.2.102:6833/36087","heartbeat_front_addr":"172.21.2.102:6829/36087","state":["exists","up"]},{"osd":2,"uuid":"153700af-747f-481f-bdf4-d6072d3a842f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6810","nonce":36089},{"type":"v1","addr":"172.21.2.102:6811","nonce":36089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6812","nonce":36089},{"type":"v1","addr":"172.21.2.102:6813","nonce":36089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6816","nonce":36089},{"type":"v1","addr":"172.21.2.102:6817","nonce":36089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6814","nonce":36089},{"type":"v1","addr":"172.21.2.102:6815","nonce":36089}]},"public_addr":"172.21.2.102:6811/36089","cluster_addr":"172.21.2.102:6813/36089","heartbeat_back_addr":"172.21.2.102:6817/36089","heartbeat_front_addr":"172.21.2.102:6815/36089","state":["exists","up"]},{"osd":3,"uuid":"cbc678f4-83a9-45e6-8830-2f7c232b1ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6802","nonce":36090},{"type":"v1","addr":"172.21.2.102:6803","nonce":36090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6804","nonce":36090},{"type":"v1","addr":"172.21.2.102:6805","nonce":36090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6808","nonce":36090},{"type":"v1","addr":"172.21.2.102:6809","nonce":36090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6806","nonce":36090},{"type":"v1","addr":"172.21.2.102:6807","nonce":36090}]},"public_addr":"172.21.2.102:6803/36090","cluster_addr":"172.21.2.102:6805/36090","heartbeat_back_addr":"172.21.2.102:6809/36090","heartbeat_front_addr":"172.21.2.102:6807/36090","state":["exists","up"]},{"osd":4,"uuid":"c01e4008-b258-4f10-afd2-9e7f37214351","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6808","nonce":36368},{"type":"v1","addr":"172.21.2.113:6811","nonce":36368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6814","nonce":36368},{"type":"v1","addr":"172.21.2.113:6818","nonce":36368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6826","nonce":36368},{"type":"v1","addr":"172.21.2.113:6828","nonce":36368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6820","nonce":36368},{"type":"v1","addr":"172.21.2.113:6824","nonce":36368}]},"public_addr":"172.21.2.113:6811/36368","cluster_addr":"172.21.2.113:6818/36368","heartbeat_back_addr":"172.21.2.113:6828/36368","heartbeat_front_addr":"172.21.2.113:6824/36368","state":["exists","up"]},{"osd":5,"uuid":"551f6007-daf8-4a98-8ee0-c2b16bcccb5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6802","nonce":36366},{"type":"v1","addr":"172.21.2.113:6805","nonce":36366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6807","nonce":36366},{"type":"v1","addr":"172.21.2.113:6810","nonce":36366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6819","nonce":36366},{"type":"v1","addr":"172.21.2.113:6822","nonce":36366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6812","nonce":36366},{"type":"v1","addr":"172.21.2.113:6815","nonce":36366}]},"public_addr":"172.21.2.113:6805/36366","cluster_addr":"172.21.2.113:6810/36366","heartbeat_back_addr":"172.21.2.113:6822/36366","heartbeat_front_addr":"172.21.2.113:6815/36366","state":["exists","up"]},{"osd":6,"uuid":"bf67c040-0e50-48be-b102-c3943bc9ebef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6800","nonce":36367},{"type":"v1","addr":"172.21.2.113:6801","nonce":36367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6803","nonce":36367},{"type":"v1","addr":"172.21.2.113:6804","nonce":36367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6813","nonce":36367},{"type":"v1","addr":"172.21.2.113:6816","nonce":36367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6806","nonce":36367},{"type":"v1","addr":"172.21.2.113:6809","nonce":36367}]},"public_addr":"172.21.2.113:6801/36367","cluster_addr":"172.21.2.113:6804/36367","heartbeat_back_addr":"172.21.2.113:6816/36367","heartbeat_front_addr":"172.21.2.113:6809/36367","state":["exists","up"]},{"osd":7,"uuid":"f5461f80-018d-428f-be89-2eb2bcbe7bb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6817","nonce":36365},{"type":"v1","addr":"172.21.2.113:6821","nonce":36365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6823","nonce":36365},{"type":"v1","addr":"172.21.2.113:6825","nonce":36365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6830","nonce":36365},{"type":"v1","addr":"172.21.2.113:6831","nonce":36365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6827","nonce":36365},{"type":"v1","addr":"172.21.2.113:6829","nonce":36365}]},"public_addr":"172.21.2.113:6821/36365","cluster_addr":"172.21.2.113:6825/36365","heartbeat_back_addr":"172.21.2.113:6831/36365","heartbeat_front_addr":"172.21.2.113:6829/36365","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:23:21.243 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:21.245 7f30621fe700 1 -- 172.21.2.102:0/2518411503 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f303c018730 msgr2=0x7f303c01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:21.243 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:21.245 7f30621fe700 1 --2- 172.21.2.102:0/2518411503 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f303c018730 0x7f303c01ac10 crc :-1 s=READY pgs=569 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:21.243 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:21.245 7f30621fe700 1 -- 172.21.2.102:0/2518411503 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f305c191bb0 msgr2=0x7f305c0a2380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:21.243 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:21.245 7f30621fe700 1 --2- 172.21.2.102:0/2518411503 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f305c191bb0 0x7f305c0a2380 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 rx=0x7f304c006f40 tx=0x7f304c00b990).stop 2021-02-10T11:23:21.243 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:21.245 7f30621fe700 1 -- 172.21.2.102:0/2518411503 shutdown_connections 2021-02-10T11:23:21.244 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:21.245 7f30621fe700 1 --2- 172.21.2.102:0/2518411503 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f305c0a33c0 0x7f305c0a03d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:21.244 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:21.245 7f30621fe700 1 --2- 172.21.2.102:0/2518411503 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f303c018730 0x7f303c01ac10 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:21.244 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:21.245 7f30621fe700 1 --2- 172.21.2.102:0/2518411503 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f305c0a29a0 0x7f305c0a2da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:21.244 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:21.245 7f30621fe700 1 --2- 172.21.2.102:0/2518411503 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f305c191bb0 0x7f305c0a2380 unknown :-1 s=CLOSED pgs=1191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:21.244 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:21.245 7f30621fe700 1 -- 172.21.2.102:0/2518411503 >> 172.21.2.102:0/2518411503 conn(0x7f305c1a31d0 msgr2=0x7f305c08b830 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:21.244 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:21.246 7f30621fe700 1 -- 172.21.2.102:0/2518411503 shutdown_connections 2021-02-10T11:23:21.244 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:21.246 7f30621fe700 1 -- 172.21.2.102:0/2518411503 wait complete. 2021-02-10T11:23:21.252 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:23:21.690 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:21.676 7fa0fb08f700 1 Processor -- start 2021-02-10T11:23:21.690 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:21.676 7fa0fb08f700 1 -- start start 2021-02-10T11:23:21.691 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:21.677 7fa0fb08f700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa0f41b0470 0x7fa0f41b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:21.691 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:21.677 7fa0fb08f700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fa0f40ef610 0x7fa0f40efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:21.691 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:21.677 7fa0fb08f700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0f40f0b50 con 0x7fa0f40f0030 2021-02-10T11:23:21.691 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:21.677 7fa0fb08f700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0f40e52e0 con 0x7fa0f40ef610 2021-02-10T11:23:21.691 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:21.677 7fa0fb08f700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0f40e5490 con 0x7fa0f41b0470 2021-02-10T11:23:21.691 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:21.677 7fa0f3fff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fa0f40ef610 0x7fa0f40efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:21.691 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:21.677 7fa0f3fff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fa0f40ef610 0x7fa0f40efa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:56262/0 (socket says 172.21.2.102:56262) 2021-02-10T11:23:21.691 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:21.677 7fa0f3fff700 1 -- 172.21.2.102:0/1935854290 learned_addr learned my addr 172.21.2.102:0/1935854290 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:23:21.691 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:21.677 7fa0f3fff700 1 -- 172.21.2.102:0/1935854290 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa0f41b0470 msgr2=0x7fa0f41b0870 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:23:21.691 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:21.677 7fa0f3fff700 1 --2- 172.21.2.102:0/1935854290 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa0f41b0470 0x7fa0f41b0870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:21.691 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:21.677 7fa0f3fff700 1 -- 172.21.2.102:0/1935854290 >> v1:172.21.2.113:6789/0 conn(0x7fa0f40f0030 legacy=0x7fa0f40f10d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:23:21.691 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:21.677 7fa0f3fff700 1 -- 172.21.2.102:0/1935854290 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0f40f0430 con 0x7fa0f40ef610 2021-02-10T11:23:21.691 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:21.677 7fa0f3fff700 1 --2- 172.21.2.102:0/1935854290 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa0f40ef610 0x7fa0f40efa10 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 rx=0x7fa0e4006700 tx=0x7fa0e4009b00).ready entity=mon.0 client_cookie=e1ab37f8d92610f2 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:21.692 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:21.678 7fa0f37fe700 1 -- 172.21.2.102:0/1935854290 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa0e400c0c0 con 0x7fa0f40ef610 2021-02-10T11:23:21.692 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:21.678 7fa0f37fe700 1 -- 172.21.2.102:0/1935854290 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0e400c400 con 0x7fa0f40ef610 2021-02-10T11:23:21.692 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:21.678 7fa0f37fe700 1 -- 172.21.2.102:0/1935854290 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa0e400c8e0 con 0x7fa0f40ef610 2021-02-10T11:23:21.692 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:21.678 7fa0fb08f700 1 -- 172.21.2.102:0/1935854290 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa0f40ef610 msgr2=0x7fa0f40efa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:21.692 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:21.678 7fa0fb08f700 1 --2- 172.21.2.102:0/1935854290 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa0f40ef610 0x7fa0f40efa10 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 rx=0x7fa0e4006700 tx=0x7fa0e4009b00).stop 2021-02-10T11:23:21.692 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:21.678 7fa0fb08f700 1 -- 172.21.2.102:0/1935854290 shutdown_connections 2021-02-10T11:23:21.692 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:21.678 7fa0fb08f700 1 --2- 172.21.2.102:0/1935854290 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa0f40ef610 0x7fa0f40efa10 unknown :-1 s=CLOSED pgs=1192 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:21.692 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:21.678 7fa0fb08f700 1 --2- 172.21.2.102:0/1935854290 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa0f41b0470 0x7fa0f41b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:21.692 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:21.678 7fa0fb08f700 1 -- 172.21.2.102:0/1935854290 >> 172.21.2.102:0/1935854290 conn(0x7fa0f40ce3e0 msgr2=0x7fa0f40e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:21.692 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:21.678 7fa0fb08f700 1 -- 172.21.2.102:0/1935854290 shutdown_connections 2021-02-10T11:23:21.692 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:21.678 7fa0fb08f700 1 -- 172.21.2.102:0/1935854290 wait complete. 2021-02-10T11:23:21.694 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:21.693 7fa0fb08f700 1 Processor -- start 2021-02-10T11:23:21.694 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:21.693 7fa0fb08f700 1 -- start start 2021-02-10T11:23:21.694 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:21.693 7fa0fb08f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa0f408dad0 0x7fa0f408a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:21.694 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:21.693 7fa0fb08f700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa0f408aa50 0x7fa0f408e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:21.694 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:21.693 7fa0fb08f700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa0f408e7d0 0x7fa0f408b460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:21.694 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:21.693 7fa0fb08f700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0f408ded0 con 0x7fa0f408dad0 2021-02-10T11:23:21.694 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:21.693 7fa0fb08f700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0f408ebd0 con 0x7fa0f408aa50 2021-02-10T11:23:21.695 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:21.693 7fa0fb08f700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0f408ba80 con 0x7fa0f408e7d0 2021-02-10T11:23:21.695 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:21.694 7fa0f8eab700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa0f408dad0 0x7fa0f408a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:21.695 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:21.694 7fa0f8eab700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa0f408dad0 0x7fa0f408a430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:56270/0 (socket says 172.21.2.102:56270) 2021-02-10T11:23:21.695 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:21.694 7fa0f8eab700 1 -- 172.21.2.102:0/2162251140 learned_addr learned my addr 172.21.2.102:0/2162251140 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:23:21.695 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:21.694 7fa0f96ac700 1 --2- 172.21.2.102:0/2162251140 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa0f408e7d0 0x7fa0f408b460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:21.695 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:21.694 7fa0f3fff700 1 --2- 172.21.2.102:0/2162251140 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa0f408aa50 0x7fa0f408e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:21.695 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:21.694 7fa0f8eab700 1 -- 172.21.2.102:0/2162251140 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa0f408e7d0 msgr2=0x7fa0f408b460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:21.695 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:21.694 7fa0f8eab700 1 --2- 172.21.2.102:0/2162251140 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa0f408e7d0 0x7fa0f408b460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:21.695 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:21.694 7fa0f8eab700 1 -- 172.21.2.102:0/2162251140 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa0f408aa50 msgr2=0x7fa0f408e1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:21.695 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:21.694 7fa0f8eab700 1 --2- 172.21.2.102:0/2162251140 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa0f408aa50 0x7fa0f408e1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:21.695 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:21.694 7fa0f8eab700 1 -- 172.21.2.102:0/2162251140 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0f408bca0 con 0x7fa0f408dad0 2021-02-10T11:23:21.695 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:21.695 7fa0f8eab700 1 --2- 172.21.2.102:0/2162251140 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa0f408dad0 0x7fa0f408a430 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 rx=0x7fa0dc00b5c0 tx=0x7fa0dc00b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:21.696 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:21.695 7fa0f17fa700 1 -- 172.21.2.102:0/2162251140 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa0dc00bd50 con 0x7fa0f408dad0 2021-02-10T11:23:21.696 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:21.695 7fa0f17fa700 1 -- 172.21.2.102:0/2162251140 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0dc006030 con 0x7fa0f408dad0 2021-02-10T11:23:21.696 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:21.695 7fa0f17fa700 1 -- 172.21.2.102:0/2162251140 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa0dc0054d0 con 0x7fa0f408dad0 2021-02-10T11:23:21.696 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:21.695 7fa0fb08f700 1 -- 172.21.2.102:0/2162251140 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0f408bca0 con 0x7fa0f408dad0 2021-02-10T11:23:21.696 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:21.695 7fa0fb08f700 1 -- 172.21.2.102:0/2162251140 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0f408c230 con 0x7fa0f408dad0 2021-02-10T11:23:21.696 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:21.695 7fa0f17fa700 1 -- 172.21.2.102:0/2162251140 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa0dc006210 con 0x7fa0f408dad0 2021-02-10T11:23:21.696 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:21.695 7fa0f17fa700 1 --2- 172.21.2.102:0/2162251140 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa0d80186c0 0x7fa0d801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:21.696 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:21.696 7fa0f17fa700 1 -- 172.21.2.102:0/2162251140 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fa0dc0325b0 con 0x7fa0f408dad0 2021-02-10T11:23:21.696 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:21.696 7fa0f3fff700 1 --2- 172.21.2.102:0/2162251140 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa0d80186c0 0x7fa0d801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:21.696 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:21.696 7fa0f3fff700 1 --2- 172.21.2.102:0/2162251140 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa0d80186c0 0x7fa0d801abc0 crc :-1 s=READY pgs=570 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:21.724 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:21.727 7fa0fb08f700 1 -- 172.21.2.102:0/2162251140 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0f40e52e0 con 0x7fa0f408dad0 2021-02-10T11:23:21.727 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:21.729 7fa0f17fa700 1 -- 172.21.2.102:0/2162251140 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fa0dc000940 con 0x7fa0f408dad0 2021-02-10T11:23:21.857 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:21.859 7fa0fb08f700 1 -- 172.21.2.102:0/2162251140 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7fa0f40e52e0 con 0x7fa0f408dad0 2021-02-10T11:23:21.860 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:21.863 7fa0f17fa700 1 -- 172.21.2.102:0/2162251140 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.0 v23) v1 ==== 184+0+0 (secure 0 0 0) 0x7fa0dc006210 con 0x7fa0f408dad0 2021-02-10T11:23:21.864 INFO:teuthology.orchestra.run.gibba002.stderr:updated caps for client.0 2021-02-10T11:23:21.865 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:21.867 7fa0fb08f700 1 -- 172.21.2.102:0/2162251140 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa0d80186c0 msgr2=0x7fa0d801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:21.865 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:21.867 7fa0fb08f700 1 --2- 172.21.2.102:0/2162251140 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa0d80186c0 0x7fa0d801abc0 crc :-1 s=READY pgs=570 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:21.865 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:21.867 7fa0fb08f700 1 -- 172.21.2.102:0/2162251140 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa0f408dad0 msgr2=0x7fa0f408a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:21.865 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:21.867 7fa0fb08f700 1 --2- 172.21.2.102:0/2162251140 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa0f408dad0 0x7fa0f408a430 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 rx=0x7fa0dc00b5c0 tx=0x7fa0dc00b960).stop 2021-02-10T11:23:21.866 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:21.868 7fa0fb08f700 1 -- 172.21.2.102:0/2162251140 shutdown_connections 2021-02-10T11:23:21.866 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:21.868 7fa0fb08f700 1 --2- 172.21.2.102:0/2162251140 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa0f408e7d0 0x7fa0f408b460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:21.866 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:21.868 7fa0fb08f700 1 --2- 172.21.2.102:0/2162251140 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa0d80186c0 0x7fa0d801abc0 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:21.866 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:21.868 7fa0fb08f700 1 --2- 172.21.2.102:0/2162251140 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa0f408aa50 0x7fa0f408e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:21.866 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:21.868 7fa0fb08f700 1 --2- 172.21.2.102:0/2162251140 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa0f408dad0 0x7fa0f408a430 unknown :-1 s=CLOSED pgs=1193 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:21.866 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:21.868 7fa0fb08f700 1 -- 172.21.2.102:0/2162251140 >> 172.21.2.102:0/2162251140 conn(0x7fa0f40ef940 msgr2=0x7fa0f40a3720 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:21.867 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:21.868 7fa0fb08f700 1 -- 172.21.2.102:0/2162251140 shutdown_connections 2021-02-10T11:23:21.867 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:21.868 7fa0fb08f700 1 -- 172.21.2.102:0/2162251140 wait complete. 2021-02-10T11:23:21.874 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.1 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:23:22.312 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:22.298 7f85734b4700 1 Processor -- start 2021-02-10T11:23:22.312 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:22.298 7f85734b4700 1 -- start start 2021-02-10T11:23:22.312 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:22.298 7f85734b4700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f856c1b0870 0x7f856c0f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:22.313 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:22.298 7f85734b4700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f856c0efe00 0x7f856c0f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:22.313 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:22.298 7f85734b4700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f856c1b0c70 con 0x7f856c1b0470 2021-02-10T11:23:22.313 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:22.298 7f85734b4700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f856c0f0b50 con 0x7f856c1b0870 2021-02-10T11:23:22.313 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:22.298 7f85734b4700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f856c0f0820 con 0x7f856c0efe00 2021-02-10T11:23:22.313 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:22.298 7f8570acf700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f856c1b0870 0x7f856c0f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:22.313 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:22.298 7f8570acf700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f856c1b0870 0x7f856c0f20e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:56276/0 (socket says 172.21.2.102:56276) 2021-02-10T11:23:22.313 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:22.299 7f8570acf700 1 -- 172.21.2.102:0/2736450562 learned_addr learned my addr 172.21.2.102:0/2736450562 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:23:22.313 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:22.299 7f8570acf700 1 -- 172.21.2.102:0/2736450562 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f856c0efe00 msgr2=0x7f856c0f0200 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:23:22.313 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:22.299 7f8570acf700 1 --2- 172.21.2.102:0/2736450562 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f856c0efe00 0x7f856c0f0200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:22.313 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:22.299 7f8570acf700 1 -- 172.21.2.102:0/2736450562 >> v1:172.21.2.113:6789/0 conn(0x7f856c1b0470 legacy=0x7f856c0ef610 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:23:22.313 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:22.299 7f8570acf700 1 -- 172.21.2.102:0/2736450562 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f856c0e52e0 con 0x7f856c1b0870 2021-02-10T11:23:22.313 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:22.299 7f8570acf700 1 --2- 172.21.2.102:0/2736450562 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f856c1b0870 0x7f856c0f20e0 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 rx=0x7f8568006700 tx=0x7f8568009b80).ready entity=mon.0 client_cookie=554c16d697af7a0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:22.313 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:22.299 7f855ffff700 1 -- 172.21.2.102:0/2736450562 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f856800c0c0 con 0x7f856c1b0870 2021-02-10T11:23:22.314 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:22.299 7f855ffff700 1 -- 172.21.2.102:0/2736450562 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f856800c340 con 0x7f856c1b0870 2021-02-10T11:23:22.314 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:22.299 7f855ffff700 1 -- 172.21.2.102:0/2736450562 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f856800c820 con 0x7f856c1b0870 2021-02-10T11:23:22.314 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:22.299 7f85734b4700 1 -- 172.21.2.102:0/2736450562 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f856c1b0870 msgr2=0x7f856c0f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:22.314 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:22.299 7f85734b4700 1 --2- 172.21.2.102:0/2736450562 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f856c1b0870 0x7f856c0f20e0 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 rx=0x7f8568006700 tx=0x7f8568009b80).stop 2021-02-10T11:23:22.314 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:22.300 7f85734b4700 1 -- 172.21.2.102:0/2736450562 shutdown_connections 2021-02-10T11:23:22.314 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:22.300 7f85734b4700 1 --2- 172.21.2.102:0/2736450562 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f856c0efe00 0x7f856c0f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:22.314 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:22.300 7f85734b4700 1 --2- 172.21.2.102:0/2736450562 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f856c1b0870 0x7f856c0f20e0 unknown :-1 s=CLOSED pgs=1194 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:22.314 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:22.300 7f85734b4700 1 -- 172.21.2.102:0/2736450562 >> 172.21.2.102:0/2736450562 conn(0x7f856c0ce3e0 msgr2=0x7f856c0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:22.314 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:22.300 7f85734b4700 1 -- 172.21.2.102:0/2736450562 shutdown_connections 2021-02-10T11:23:22.314 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:22.300 7f85734b4700 1 -- 172.21.2.102:0/2736450562 wait complete. 2021-02-10T11:23:22.314 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:22.314 7f85734b4700 1 Processor -- start 2021-02-10T11:23:22.314 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:22.314 7f85734b4700 1 -- start start 2021-02-10T11:23:22.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:22.315 7f85734b4700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f856c1981f0 0x7f856c18ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:22.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:22.315 7f85734b4700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f856c0b75f0 0x7f856c0b79f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:22.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:22.315 7f85734b4700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f856c0b8010 0x7f856c18d700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:22.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:22.315 7f85734b4700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f856c18bb80 con 0x7f856c0b8010 2021-02-10T11:23:22.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:22.315 7f85734b4700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f856c0b8410 con 0x7f856c0b75f0 2021-02-10T11:23:22.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:22.315 7f85734b4700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f856c18dd20 con 0x7f856c1981f0 2021-02-10T11:23:22.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:22.315 7f8571ad1700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f856c0b8010 0x7f856c18d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:22.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:22.315 7f8571ad1700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f856c0b8010 0x7f856c18d700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:56286/0 (socket says 172.21.2.102:56286) 2021-02-10T11:23:22.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:22.315 7f8571ad1700 1 -- 172.21.2.102:0/2703963682 learned_addr learned my addr 172.21.2.102:0/2703963682 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:23:22.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:22.315 7f8571ad1700 1 -- 172.21.2.102:0/2703963682 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f856c1981f0 msgr2=0x7f856c18ebb0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:23:22.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:22.315 7f8571ad1700 1 --2- 172.21.2.102:0/2703963682 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f856c1981f0 0x7f856c18ebb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:22.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:22.315 7f8571ad1700 1 -- 172.21.2.102:0/2703963682 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f856c0b75f0 msgr2=0x7f856c0b79f0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:23:22.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:22.315 7f8571ad1700 1 --2- 172.21.2.102:0/2703963682 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f856c0b75f0 0x7f856c0b79f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:22.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:22.315 7f8571ad1700 1 -- 172.21.2.102:0/2703963682 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f856c18df40 con 0x7f856c0b8010 2021-02-10T11:23:22.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:22.315 7f8571ad1700 1 --2- 172.21.2.102:0/2703963682 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f856c0b8010 0x7f856c18d700 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 rx=0x7f8564009960 tx=0x7f856400bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:22.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:22.315 7f855dffb700 1 -- 172.21.2.102:0/2703963682 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8564003030 con 0x7f856c0b8010 2021-02-10T11:23:22.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:22.315 7f855dffb700 1 -- 172.21.2.102:0/2703963682 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8564001310 con 0x7f856c0b8010 2021-02-10T11:23:22.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:22.315 7f85734b4700 1 -- 172.21.2.102:0/2703963682 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f856c18df40 con 0x7f856c0b8010 2021-02-10T11:23:22.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:22.315 7f855dffb700 1 -- 172.21.2.102:0/2703963682 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f85640017f0 con 0x7f856c0b8010 2021-02-10T11:23:22.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:22.316 7f85734b4700 1 -- 172.21.2.102:0/2703963682 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f856c18df40 con 0x7f856c0b8010 2021-02-10T11:23:22.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:22.316 7f855dffb700 1 -- 172.21.2.102:0/2703963682 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f85640014f0 con 0x7f856c0b8010 2021-02-10T11:23:22.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:22.316 7f855dffb700 1 --2- 172.21.2.102:0/2703963682 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f854c0186d0 0x7f854c01abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:22.318 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:22.316 7f855dffb700 1 -- 172.21.2.102:0/2703963682 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f8564032b40 con 0x7f856c0b8010 2021-02-10T11:23:22.318 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:22.316 7f85712d0700 1 --2- 172.21.2.102:0/2703963682 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f854c0186d0 0x7f854c01abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:22.318 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:22.317 7f85712d0700 1 --2- 172.21.2.102:0/2703963682 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f854c0186d0 0x7f854c01abd0 crc :-1 s=READY pgs=571 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:22.346 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:22.348 7f85734b4700 1 -- 172.21.2.102:0/2703963682 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f856c0e52e0 con 0x7f856c0b8010 2021-02-10T11:23:22.348 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:22.350 7f855dffb700 1 -- 172.21.2.102:0/2703963682 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f8564001520 con 0x7f856c0b8010 2021-02-10T11:23:22.479 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:22.481 7f85734b4700 1 -- 172.21.2.102:0/2703963682 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f856c0e52e0 con 0x7f856c0b8010 2021-02-10T11:23:22.482 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:22.484 7f855dffb700 1 -- 172.21.2.102:0/2703963682 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.1 v24) v1 ==== 184+0+0 (secure 0 0 0) 0x7f8564000d10 con 0x7f856c0b8010 2021-02-10T11:23:22.486 INFO:teuthology.orchestra.run.gibba002.stderr:updated caps for client.1 2021-02-10T11:23:22.488 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:22.489 7f85734b4700 1 -- 172.21.2.102:0/2703963682 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f854c0186d0 msgr2=0x7f854c01abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:22.488 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:22.489 7f85734b4700 1 --2- 172.21.2.102:0/2703963682 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f854c0186d0 0x7f854c01abd0 crc :-1 s=READY pgs=571 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:22.488 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:22.489 7f85734b4700 1 -- 172.21.2.102:0/2703963682 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f856c0b8010 msgr2=0x7f856c18d700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:22.489 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:22.489 7f85734b4700 1 --2- 172.21.2.102:0/2703963682 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f856c0b8010 0x7f856c18d700 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 rx=0x7f8564009960 tx=0x7f856400bcf0).stop 2021-02-10T11:23:22.489 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:22.490 7f85734b4700 1 -- 172.21.2.102:0/2703963682 shutdown_connections 2021-02-10T11:23:22.489 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:22.490 7f85734b4700 1 --2- 172.21.2.102:0/2703963682 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f856c0b8010 0x7f856c18d700 unknown :-1 s=CLOSED pgs=1195 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:22.489 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:22.490 7f85734b4700 1 --2- 172.21.2.102:0/2703963682 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f854c0186d0 0x7f854c01abd0 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:22.489 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:22.490 7f85734b4700 1 --2- 172.21.2.102:0/2703963682 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f856c0b75f0 0x7f856c0b79f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:22.490 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:22.490 7f85734b4700 1 --2- 172.21.2.102:0/2703963682 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f856c1981f0 0x7f856c18ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:22.490 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:22.490 7f85734b4700 1 -- 172.21.2.102:0/2703963682 >> 172.21.2.102:0/2703963682 conn(0x7f856c0ef940 msgr2=0x7f856c0a3720 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:22.490 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:22.490 7f85734b4700 1 -- 172.21.2.102:0/2703963682 shutdown_connections 2021-02-10T11:23:22.490 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:22.490 7f85734b4700 1 -- 172.21.2.102:0/2703963682 wait complete. 2021-02-10T11:23:22.497 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.2 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:23:22.936 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:22.922 7f686466b700 1 Processor -- start 2021-02-10T11:23:22.936 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:22.922 7f686466b700 1 -- start start 2021-02-10T11:23:22.936 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:22.922 7f686466b700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f685c1b0470 0x7f685c1b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:22.936 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:22.922 7f686466b700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f685c0ef610 0x7f685c0efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:22.936 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:22.922 7f686466b700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f685c0f0b50 con 0x7f685c0f0030 2021-02-10T11:23:22.936 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:22.922 7f686466b700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f685c0e52e0 con 0x7f685c0ef610 2021-02-10T11:23:22.936 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:22.922 7f686466b700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f685c0e5490 con 0x7f685c1b0470 2021-02-10T11:23:22.936 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:22.922 7f6861c86700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f685c0ef610 0x7f685c0efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:22.937 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:22.922 7f6861c86700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f685c0ef610 0x7f685c0efa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:56290/0 (socket says 172.21.2.102:56290) 2021-02-10T11:23:22.937 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:22.922 7f6861c86700 1 -- 172.21.2.102:0/2463583223 learned_addr learned my addr 172.21.2.102:0/2463583223 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:23:22.937 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:22.923 7f6861c86700 1 -- 172.21.2.102:0/2463583223 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f685c1b0470 msgr2=0x7f685c1b0870 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:23:22.937 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:22.923 7f6861c86700 1 --2- 172.21.2.102:0/2463583223 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f685c1b0470 0x7f685c1b0870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:22.937 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:22.923 7f6861c86700 1 -- 172.21.2.102:0/2463583223 >> v1:172.21.2.113:6789/0 conn(0x7f685c0f0030 legacy=0x7f685c0f10d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:23:22.937 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:22.923 7f6861c86700 1 -- 172.21.2.102:0/2463583223 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f685c0f0430 con 0x7f685c0ef610 2021-02-10T11:23:22.937 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:22.923 7f6861c86700 1 --2- 172.21.2.102:0/2463583223 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f685c0ef610 0x7f685c0efa10 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 rx=0x7f6858006700 tx=0x7f6858009b00).ready entity=mon.0 client_cookie=58aa6a5fd5b3414c server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:22.937 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:22.923 7f6861485700 1 -- 172.21.2.102:0/2463583223 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f685800c0c0 con 0x7f685c0ef610 2021-02-10T11:23:22.937 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:22.923 7f6861485700 1 -- 172.21.2.102:0/2463583223 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f685800c400 con 0x7f685c0ef610 2021-02-10T11:23:22.937 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:22.923 7f6861485700 1 -- 172.21.2.102:0/2463583223 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f685800c920 con 0x7f685c0ef610 2021-02-10T11:23:22.937 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:22.923 7f686466b700 1 -- 172.21.2.102:0/2463583223 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f685c0ef610 msgr2=0x7f685c0efa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:22.937 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:22.923 7f686466b700 1 --2- 172.21.2.102:0/2463583223 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f685c0ef610 0x7f685c0efa10 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 rx=0x7f6858006700 tx=0x7f6858009b00).stop 2021-02-10T11:23:22.937 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:22.924 7f686466b700 1 -- 172.21.2.102:0/2463583223 shutdown_connections 2021-02-10T11:23:22.937 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:22.924 7f686466b700 1 --2- 172.21.2.102:0/2463583223 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f685c0ef610 0x7f685c0efa10 unknown :-1 s=CLOSED pgs=1196 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:22.938 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:22.924 7f686466b700 1 --2- 172.21.2.102:0/2463583223 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f685c1b0470 0x7f685c1b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:22.938 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:22.924 7f686466b700 1 -- 172.21.2.102:0/2463583223 >> 172.21.2.102:0/2463583223 conn(0x7f685c0ce3e0 msgr2=0x7f685c0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:22.938 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:22.924 7f686466b700 1 -- 172.21.2.102:0/2463583223 shutdown_connections 2021-02-10T11:23:22.938 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:22.924 7f686466b700 1 -- 172.21.2.102:0/2463583223 wait complete. 2021-02-10T11:23:22.939 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:22.938 7f686466b700 1 Processor -- start 2021-02-10T11:23:22.939 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:22.939 7f686466b700 1 -- start start 2021-02-10T11:23:22.939 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:22.939 7f686466b700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f685c08dad0 0x7f685c08a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:22.939 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:22.939 7f686466b700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f685c08aa50 0x7f685c08e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:22.940 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:22.939 7f686466b700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f685c08e7d0 0x7f685c08b460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:22.940 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:22.939 7f686466b700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f685c08ded0 con 0x7f685c08aa50 2021-02-10T11:23:22.940 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:22.939 7f686466b700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f685c08ebd0 con 0x7f685c08e7d0 2021-02-10T11:23:22.940 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:22.939 7f686466b700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f685c08ba80 con 0x7f685c08dad0 2021-02-10T11:23:22.940 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:22.939 7f6861c86700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f685c08aa50 0x7f685c08e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:22.940 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:22.939 7f6861c86700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f685c08aa50 0x7f685c08e1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:56296/0 (socket says 172.21.2.102:56296) 2021-02-10T11:23:22.940 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:22.939 7f6861c86700 1 -- 172.21.2.102:0/1363895989 learned_addr learned my addr 172.21.2.102:0/1363895989 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:23:22.940 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:22.939 7f6861c86700 1 -- 172.21.2.102:0/1363895989 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f685c08dad0 msgr2=0x7f685c08a430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:22.940 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:22.939 7f6861c86700 1 --2- 172.21.2.102:0/1363895989 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f685c08dad0 0x7f685c08a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:22.940 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:22.939 7f6861c86700 1 -- 172.21.2.102:0/1363895989 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f685c08e7d0 msgr2=0x7f685c08b460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:22.940 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:22.939 7f6861c86700 1 --2- 172.21.2.102:0/1363895989 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f685c08e7d0 0x7f685c08b460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:22.940 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:22.939 7f6861c86700 1 -- 172.21.2.102:0/1363895989 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f685c08bca0 con 0x7f685c08aa50 2021-02-10T11:23:22.940 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:22.940 7f6861c86700 1 --2- 172.21.2.102:0/1363895989 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f685c08aa50 0x7f685c08e1b0 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 rx=0x7f685800b660 tx=0x7f685800ba00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:22.941 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:22.940 7f684effd700 1 -- 172.21.2.102:0/1363895989 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6858009440 con 0x7f685c08aa50 2021-02-10T11:23:22.941 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:22.940 7f684effd700 1 -- 172.21.2.102:0/1363895989 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6858000c90 con 0x7f685c08aa50 2021-02-10T11:23:22.941 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:22.940 7f684effd700 1 -- 172.21.2.102:0/1363895989 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6858002030 con 0x7f685c08aa50 2021-02-10T11:23:22.941 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:22.940 7f686466b700 1 -- 172.21.2.102:0/1363895989 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f685c08bca0 con 0x7f685c08aa50 2021-02-10T11:23:22.941 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:22.940 7f686466b700 1 -- 172.21.2.102:0/1363895989 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f685c08bca0 con 0x7f685c08aa50 2021-02-10T11:23:22.941 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:22.940 7f684effd700 1 -- 172.21.2.102:0/1363895989 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f68580093e0 con 0x7f685c08aa50 2021-02-10T11:23:22.941 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:22.940 7f684effd700 1 --2- 172.21.2.102:0/1363895989 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f68440186d0 0x7f684401abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:22.941 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:22.940 7f684effd700 1 -- 172.21.2.102:0/1363895989 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f6858032650 con 0x7f685c08aa50 2021-02-10T11:23:22.941 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:22.941 7f6862487700 1 --2- 172.21.2.102:0/1363895989 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f68440186d0 0x7f684401abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:22.941 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:22.941 7f6862487700 1 --2- 172.21.2.102:0/1363895989 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f68440186d0 0x7f684401abd0 crc :-1 s=READY pgs=572 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:22.970 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:22.972 7f686466b700 1 -- 172.21.2.102:0/1363895989 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f685c08ded0 con 0x7f685c08aa50 2021-02-10T11:23:22.972 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:22.974 7f684effd700 1 -- 172.21.2.102:0/1363895989 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f68580090c0 con 0x7f685c08aa50 2021-02-10T11:23:23.104 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:23.106 7f686466b700 1 -- 172.21.2.102:0/1363895989 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f685c08ded0 con 0x7f685c08aa50 2021-02-10T11:23:23.107 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:23.109 7f684effd700 1 -- 172.21.2.102:0/1363895989 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.2 v25) v1 ==== 184+0+0 (secure 0 0 0) 0x7f68580093e0 con 0x7f685c08aa50 2021-02-10T11:23:23.111 INFO:teuthology.orchestra.run.gibba002.stderr:updated caps for client.2 2021-02-10T11:23:23.113 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:23.114 7f686466b700 1 -- 172.21.2.102:0/1363895989 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f68440186d0 msgr2=0x7f684401abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:23.114 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:23.114 7f686466b700 1 --2- 172.21.2.102:0/1363895989 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f68440186d0 0x7f684401abd0 crc :-1 s=READY pgs=572 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:23.114 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:23.114 7f686466b700 1 -- 172.21.2.102:0/1363895989 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f685c08aa50 msgr2=0x7f685c08e1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:23.114 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:23.114 7f686466b700 1 --2- 172.21.2.102:0/1363895989 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f685c08aa50 0x7f685c08e1b0 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 rx=0x7f685800b660 tx=0x7f685800ba00).stop 2021-02-10T11:23:23.114 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:23.115 7f686466b700 1 -- 172.21.2.102:0/1363895989 shutdown_connections 2021-02-10T11:23:23.114 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:23.115 7f686466b700 1 --2- 172.21.2.102:0/1363895989 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f68440186d0 0x7f684401abd0 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:23.115 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:23.115 7f686466b700 1 --2- 172.21.2.102:0/1363895989 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f685c08e7d0 0x7f685c08b460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:23.115 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:23.115 7f686466b700 1 --2- 172.21.2.102:0/1363895989 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f685c08aa50 0x7f685c08e1b0 unknown :-1 s=CLOSED pgs=1197 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:23.115 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:23.115 7f686466b700 1 --2- 172.21.2.102:0/1363895989 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f685c08dad0 0x7f685c08a430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:23.115 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:23.115 7f686466b700 1 -- 172.21.2.102:0/1363895989 >> 172.21.2.102:0/1363895989 conn(0x7f685c0ef940 msgr2=0x7f685c0a3720 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:23.115 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:23.115 7f686466b700 1 -- 172.21.2.102:0/1363895989 shutdown_connections 2021-02-10T11:23:23.115 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:23.115 7f686466b700 1 -- 172.21.2.102:0/1363895989 wait complete. 2021-02-10T11:23:23.122 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.3 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:23:23.561 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:23.545 7f7887821700 1 Processor -- start 2021-02-10T11:23:23.562 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:23.546 7f7887821700 1 -- start start 2021-02-10T11:23:23.562 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:23.546 7f7887821700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f78800f05f0 0x7f78800f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:23.562 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:23.546 7f7887821700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78801b0c70 con 0x7f78801b0870 2021-02-10T11:23:23.562 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:23.546 7f7887821700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78800f0b50 con 0x7f78801b0470 2021-02-10T11:23:23.562 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:23.546 7f7887821700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78800e5ea0 con 0x7f78800f05f0 2021-02-10T11:23:23.562 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:23.546 7f7884e3c700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f78801b0870 0x7f78800efe00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:48526/0 (socket says 172.21.2.102:48526) 2021-02-10T11:23:23.562 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:23.546 7f7884e3c700 1 -- 172.21.2.102:0/3388512520 learned_addr learned my addr 172.21.2.102:0/3388512520 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:23:23.562 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:23.547 7f7873fff700 1 -- 172.21.2.102:0/3388512520 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3666394225 0 0) 0x7f7874001180 con 0x7f78801b0870 2021-02-10T11:23:23.563 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:23.547 7f7885e3e700 1 --2- 172.21.2.102:0/3388512520 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f78800f05f0 0x7f78800f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:23.563 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:23.547 7f7873fff700 1 -- 172.21.2.102:0/3388512520 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f786c002fe0 con 0x7f78801b0870 2021-02-10T11:23:23.563 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:23.547 7f7873fff700 1 -- 172.21.2.102:0/3388512520 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3946981378 0 0) 0x7f7874001180 con 0x7f78801b0870 2021-02-10T11:23:23.563 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:23.547 7f7873fff700 1 -- 172.21.2.102:0/3388512520 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f78800f05f0 msgr2=0x7f78800f20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:23.563 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:23.547 7f7873fff700 1 --2- 172.21.2.102:0/3388512520 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f78800f05f0 0x7f78800f20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:23.563 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:23.547 7f7873fff700 1 -- 172.21.2.102:0/3388512520 >> v1:172.21.2.113:6789/0 conn(0x7f78801b0470 legacy=0x7f78800ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:23.563 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:23.547 7f7873fff700 1 -- 172.21.2.102:0/3388512520 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78800e52e0 con 0x7f78801b0870 2021-02-10T11:23:23.563 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:23.547 7f7873fff700 1 -- 172.21.2.102:0/3388512520 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7f7874001680 con 0x7f78801b0870 2021-02-10T11:23:23.563 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:23.547 7f7873fff700 1 -- 172.21.2.102:0/3388512520 >> v1:172.21.2.102:6789/0 conn(0x7f78801b0870 legacy=0x7f78800efe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:23.563 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:23.547 7f7873fff700 1 --2- 172.21.2.102:0/3388512520 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f786c003a60 0x7f786c005ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:23.563 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:23.547 7f7873fff700 1 --2- 172.21.2.102:0/3388512520 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f786c006790 0x7f786c008c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:23.564 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:23.547 7f7873fff700 1 --2- 172.21.2.102:0/3388512520 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f786c009500 0x7f786c00ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:23.564 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:23.547 7f7873fff700 1 -- 172.21.2.102:0/3388512520 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f786c00c2d0 con 0x7f786c003a60 2021-02-10T11:23:23.564 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:23.547 7f7873fff700 1 -- 172.21.2.102:0/3388512520 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f786c00c4f0 con 0x7f786c009500 2021-02-10T11:23:23.564 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:23.547 7f7873fff700 1 -- 172.21.2.102:0/3388512520 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f786c00c710 con 0x7f786c006790 2021-02-10T11:23:23.564 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:23.548 7f788563d700 1 --2- 172.21.2.102:0/3388512520 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f786c003a60 0x7f786c005ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:23.564 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:23.548 7f7885e3e700 1 --2- 172.21.2.102:0/3388512520 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f786c009500 0x7f786c00ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:23.564 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:23.548 7f7884e3c700 1 --2- 172.21.2.102:0/3388512520 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f786c006790 0x7f786c008c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:23.564 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:23.548 7f788563d700 1 -- 172.21.2.102:0/3388512520 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f786c006790 msgr2=0x7f786c008c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:23.564 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:23.548 7f788563d700 1 --2- 172.21.2.102:0/3388512520 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f786c006790 0x7f786c008c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:23.565 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:23.548 7f788563d700 1 -- 172.21.2.102:0/3388512520 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f786c009500 msgr2=0x7f786c00ba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:23.565 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:23.548 7f788563d700 1 --2- 172.21.2.102:0/3388512520 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f786c009500 0x7f786c00ba20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:23.565 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:23.548 7f788563d700 1 -- 172.21.2.102:0/3388512520 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f786c050950 con 0x7f786c003a60 2021-02-10T11:23:23.565 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:23.548 7f788563d700 1 --2- 172.21.2.102:0/3388512520 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f786c003a60 0x7f786c005ef0 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 rx=0x7f787c000e30 tx=0x7f787c009aa0).ready entity=mon.0 client_cookie=82e5ce80ef1bc474 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:23.565 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:23.548 7f7873fff700 1 -- 172.21.2.102:0/3388512520 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f787c0066e0 con 0x7f786c003a60 2021-02-10T11:23:23.565 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:23.548 7f7873fff700 1 -- 172.21.2.102:0/3388512520 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f787c00c0c0 con 0x7f786c003a60 2021-02-10T11:23:23.565 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:23.548 7f7873fff700 1 -- 172.21.2.102:0/3388512520 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f787c00c560 con 0x7f786c003a60 2021-02-10T11:23:23.565 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:23.549 7f7887821700 1 -- 172.21.2.102:0/3388512520 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f786c003a60 msgr2=0x7f786c005ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:23.565 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:23.549 7f7887821700 1 --2- 172.21.2.102:0/3388512520 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f786c003a60 0x7f786c005ef0 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 rx=0x7f787c000e30 tx=0x7f787c009aa0).stop 2021-02-10T11:23:23.565 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:23.549 7f7885e3e700 1 -- 172.21.2.102:0/3388512520 reap_dead start 2021-02-10T11:23:23.565 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:23.549 7f7887821700 1 -- 172.21.2.102:0/3388512520 shutdown_connections 2021-02-10T11:23:23.565 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:23.549 7f7887821700 1 -- 172.21.2.102:0/3388512520 >> 172.21.2.102:0/3388512520 conn(0x7f78800ce3e0 msgr2=0x7f78800e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:23.565 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:23.549 7f7887821700 1 -- 172.21.2.102:0/3388512520 shutdown_connections 2021-02-10T11:23:23.566 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:23.549 7f7887821700 1 -- 172.21.2.102:0/3388512520 wait complete. 2021-02-10T11:23:23.566 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:23.564 7f7887821700 1 Processor -- start 2021-02-10T11:23:23.566 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:23.564 7f7887821700 1 -- start start 2021-02-10T11:23:23.567 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:23.564 7f7887821700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f78801a7960 0x7f78801a7d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:23.567 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:23.564 7f7887821700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f788008c440 0x7f7880189710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:23.567 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:23.564 7f7887821700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7880189d30 0x7f788018a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:23.567 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:23.564 7f7887821700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f788018a850 con 0x7f7880189d30 2021-02-10T11:23:23.567 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:23.565 7f7887821700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f788008c840 con 0x7f78801a7960 2021-02-10T11:23:23.568 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:23.565 7f7887821700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f788008cbb0 con 0x7f788008c440 2021-02-10T11:23:23.568 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:23.565 7f7885e3e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7880189d30 0x7f788018a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:23.568 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:23.565 7f7885e3e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7880189d30 0x7f788018a130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:56318/0 (socket says 172.21.2.102:56318) 2021-02-10T11:23:23.568 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:23.565 7f7885e3e700 1 -- 172.21.2.102:0/364402219 learned_addr learned my addr 172.21.2.102:0/364402219 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:23:23.568 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:23.565 7f7885e3e700 1 -- 172.21.2.102:0/364402219 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f788008c440 msgr2=0x7f7880189710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:23.568 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:23.565 7f7885e3e700 1 --2- 172.21.2.102:0/364402219 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f788008c440 0x7f7880189710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:23.568 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:23.565 7f7885e3e700 1 -- 172.21.2.102:0/364402219 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f78801a7960 msgr2=0x7f78801a7d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:23.568 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:23.565 7f7885e3e700 1 --2- 172.21.2.102:0/364402219 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f78801a7960 0x7f78801a7d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:23.568 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:23.565 7f7885e3e700 1 -- 172.21.2.102:0/364402219 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f788018ad50 con 0x7f7880189d30 2021-02-10T11:23:23.568 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:23.565 7f7885e3e700 1 --2- 172.21.2.102:0/364402219 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7880189d30 0x7f788018a130 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 rx=0x7f787800a940 tx=0x7f787800ace0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:23.568 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:23.566 7f7871ffb700 1 -- 172.21.2.102:0/364402219 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7878001400 con 0x7f7880189d30 2021-02-10T11:23:23.569 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:23.566 7f7871ffb700 1 -- 172.21.2.102:0/364402219 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7878003030 con 0x7f7880189d30 2021-02-10T11:23:23.569 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:23.566 7f7871ffb700 1 -- 172.21.2.102:0/364402219 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f78780018b0 con 0x7f7880189d30 2021-02-10T11:23:23.569 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:23.566 7f7887821700 1 -- 172.21.2.102:0/364402219 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f788018ad50 con 0x7f7880189d30 2021-02-10T11:23:23.569 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:23.566 7f7887821700 1 -- 172.21.2.102:0/364402219 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f788018b2e0 con 0x7f7880189d30 2021-02-10T11:23:23.569 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:23.566 7f7871ffb700 1 -- 172.21.2.102:0/364402219 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f78780011f0 con 0x7f7880189d30 2021-02-10T11:23:23.569 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:23.566 7f7871ffb700 1 --2- 172.21.2.102:0/364402219 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7860018730 0x7f786001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:23.569 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:23.566 7f7871ffb700 1 -- 172.21.2.102:0/364402219 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f7878032a00 con 0x7f7880189d30 2021-02-10T11:23:23.569 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:23.566 7f788563d700 1 --2- 172.21.2.102:0/364402219 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7860018730 0x7f786001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:23.570 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:23.567 7f788563d700 1 --2- 172.21.2.102:0/364402219 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7860018730 0x7f786001abc0 crc :-1 s=READY pgs=573 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:23.594 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:23.596 7f7887821700 1 -- 172.21.2.102:0/364402219 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f788008c840 con 0x7f7880189d30 2021-02-10T11:23:23.597 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:23.599 7f7871ffb700 1 -- 172.21.2.102:0/364402219 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f7878003010 con 0x7f7880189d30 2021-02-10T11:23:23.728 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:23.730 7f7887821700 1 -- 172.21.2.102:0/364402219 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.3", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f788008c840 con 0x7f7880189d30 2021-02-10T11:23:23.731 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:23.734 7f7871ffb700 1 -- 172.21.2.102:0/364402219 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.3", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.3 v26) v1 ==== 184+0+0 (secure 0 0 0) 0x7f7878003010 con 0x7f7880189d30 2021-02-10T11:23:23.735 INFO:teuthology.orchestra.run.gibba002.stderr:updated caps for client.3 2021-02-10T11:23:23.736 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:23.738 7f7887821700 1 -- 172.21.2.102:0/364402219 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7860018730 msgr2=0x7f786001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:23.737 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:23.738 7f7887821700 1 --2- 172.21.2.102:0/364402219 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7860018730 0x7f786001abc0 crc :-1 s=READY pgs=573 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:23.737 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:23.738 7f7887821700 1 -- 172.21.2.102:0/364402219 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7880189d30 msgr2=0x7f788018a130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:23.737 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:23.738 7f7887821700 1 --2- 172.21.2.102:0/364402219 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7880189d30 0x7f788018a130 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 rx=0x7f787800a940 tx=0x7f787800ace0).stop 2021-02-10T11:23:23.737 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:23.739 7f7887821700 1 -- 172.21.2.102:0/364402219 shutdown_connections 2021-02-10T11:23:23.737 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:23.739 7f7887821700 1 --2- 172.21.2.102:0/364402219 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7880189d30 0x7f788018a130 unknown :-1 s=CLOSED pgs=1200 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:23.737 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:23.739 7f7887821700 1 --2- 172.21.2.102:0/364402219 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f788008c440 0x7f7880189710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:23.737 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:23.739 7f7887821700 1 --2- 172.21.2.102:0/364402219 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7860018730 0x7f786001abc0 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:23.737 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:23.739 7f7887821700 1 --2- 172.21.2.102:0/364402219 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f78801a7960 0x7f78801a7d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:23.737 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:23.739 7f7887821700 1 -- 172.21.2.102:0/364402219 >> 172.21.2.102:0/364402219 conn(0x7f78800b5850 msgr2=0x7f7880091a50 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:23.737 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:23.739 7f7887821700 1 -- 172.21.2.102:0/364402219 shutdown_connections 2021-02-10T11:23:23.737 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:23.739 7f7887821700 1 -- 172.21.2.102:0/364402219 wait complete. 2021-02-10T11:23:23.746 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:23:24.173 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:24.158 7f45a2d25700 1 Processor -- start 2021-02-10T11:23:24.174 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:24.158 7f45a2d25700 1 -- start start 2021-02-10T11:23:24.174 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:24.159 7f45a2d25700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f459c0e5070 0x7f459c1b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:24.174 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:24.159 7f45a2d25700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f459c1b3e40 0x7f459c07d690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:24.174 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:24.159 7f45a2d25700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f459c0e5470 0x7f459c07dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:24.174 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:24.159 7f45a2d25700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f459c07ed80 con 0x7f459c0e5070 2021-02-10T11:23:24.174 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:24.159 7f45a2d25700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f459c07e2d0 con 0x7f459c0e5470 2021-02-10T11:23:24.175 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:24.159 7f45a2d25700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f459c07e4c0 con 0x7f459c1b3e40 2021-02-10T11:23:24.175 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:24.159 7f45a0b41700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f459c0e5070 0x7f459c1b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:24.175 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:24.159 7f45a0b41700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f459c0e5070 0x7f459c1b3820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:56324/0 (socket says 172.21.2.102:56324) 2021-02-10T11:23:24.175 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:24.159 7f45a0b41700 1 -- 172.21.2.102:0/2459966994 learned_addr learned my addr 172.21.2.102:0/2459966994 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:23:24.175 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:24.159 7f45a0b41700 1 -- 172.21.2.102:0/2459966994 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f459c1b3e40 msgr2=0x7f459c07d690 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:23:24.175 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:24.159 7f45a0b41700 1 --2- 172.21.2.102:0/2459966994 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f459c1b3e40 0x7f459c07d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:24.176 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:24.159 7f45a0b41700 1 -- 172.21.2.102:0/2459966994 >> v2:172.21.2.113:3300/0 conn(0x7f459c0e5470 msgr2=0x7f459c07dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:24.176 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:24.159 7f45a0b41700 1 --2- 172.21.2.102:0/2459966994 >> v2:172.21.2.113:3300/0 conn(0x7f459c0e5470 0x7f459c07dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:24.176 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:24.159 7f45a0b41700 1 -- 172.21.2.102:0/2459966994 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f459c07e6e0 con 0x7f459c0e5070 2021-02-10T11:23:24.176 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:24.160 7f45a0b41700 1 --2- 172.21.2.102:0/2459966994 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f459c0e5070 0x7f459c1b3820 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 rx=0x7f4584006700 tx=0x7f4584009b80).ready entity=mon.0 client_cookie=8c2a9fff543459ad server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:24.176 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:24.160 7f459b7fe700 1 -- 172.21.2.102:0/2459966994 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f458400c0c0 con 0x7f459c0e5070 2021-02-10T11:23:24.176 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:24.160 7f459b7fe700 1 -- 172.21.2.102:0/2459966994 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f458400c440 con 0x7f459c0e5070 2021-02-10T11:23:24.176 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:24.160 7f459b7fe700 1 -- 172.21.2.102:0/2459966994 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f458400c960 con 0x7f459c0e5070 2021-02-10T11:23:24.176 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:24.160 7f45a2d25700 1 -- 172.21.2.102:0/2459966994 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f459c0e5070 msgr2=0x7f459c1b3820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:24.176 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:24.160 7f45a2d25700 1 --2- 172.21.2.102:0/2459966994 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f459c0e5070 0x7f459c1b3820 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 rx=0x7f4584006700 tx=0x7f4584009b80).stop 2021-02-10T11:23:24.176 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:24.161 7f45a2d25700 1 -- 172.21.2.102:0/2459966994 shutdown_connections 2021-02-10T11:23:24.177 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:24.161 7f45a2d25700 1 --2- 172.21.2.102:0/2459966994 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f459c1b3e40 0x7f459c07d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:24.177 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:24.161 7f45a2d25700 1 --2- 172.21.2.102:0/2459966994 >> v2:172.21.2.113:3300/0 conn(0x7f459c0e5470 0x7f459c07dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:24.177 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:24.161 7f45a2d25700 1 --2- 172.21.2.102:0/2459966994 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f459c0e5070 0x7f459c1b3820 unknown :-1 s=CLOSED pgs=1201 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:24.177 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:24.161 7f45a2d25700 1 -- 172.21.2.102:0/2459966994 >> 172.21.2.102:0/2459966994 conn(0x7f459c0ce690 msgr2=0x7f459c0e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:24.177 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:24.161 7f45a2d25700 1 -- 172.21.2.102:0/2459966994 shutdown_connections 2021-02-10T11:23:24.177 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:24.161 7f45a2d25700 1 -- 172.21.2.102:0/2459966994 wait complete. 2021-02-10T11:23:24.177 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:24.175 7f45a2d25700 1 Processor -- start 2021-02-10T11:23:24.177 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:24.175 7f45a2d25700 1 -- start start 2021-02-10T11:23:24.178 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:24.176 7f45a2d25700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f459c192990 0x7f459c19e530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:24.179 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:24.176 7f45a2d25700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f459c19eb50 0x7f459c0b1390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:24.179 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:24.176 7f45a2d25700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f459c0b19b0 0x7f459c08a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:24.179 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:24.176 7f45a2d25700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f459c19f6b0 con 0x7f459c19eb50 2021-02-10T11:23:24.179 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:24.176 7f45a2d25700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f459c19f130 con 0x7f459c0b19b0 2021-02-10T11:23:24.179 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:24.176 7f45a2d25700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f459c1931d0 con 0x7f459c192990 2021-02-10T11:23:24.179 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:24.176 7f459bfff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f459c19eb50 0x7f459c0b1390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:24.179 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:24.176 7f459bfff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f459c19eb50 0x7f459c0b1390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:56332/0 (socket says 172.21.2.102:56332) 2021-02-10T11:23:24.179 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:24.176 7f459bfff700 1 -- 172.21.2.102:0/1672586605 learned_addr learned my addr 172.21.2.102:0/1672586605 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:23:24.180 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:24.176 7f45a0b41700 1 --2- 172.21.2.102:0/1672586605 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f459c192990 0x7f459c19e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:24.180 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:24.176 7f45a1342700 1 --2- 172.21.2.102:0/1672586605 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f459c0b19b0 0x7f459c08a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:24.180 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:24.177 7f459bfff700 1 -- 172.21.2.102:0/1672586605 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f459c192990 msgr2=0x7f459c19e530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:24.180 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:24.177 7f459bfff700 1 --2- 172.21.2.102:0/1672586605 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f459c192990 0x7f459c19e530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:24.180 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:24.177 7f459bfff700 1 -- 172.21.2.102:0/1672586605 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f459c0b19b0 msgr2=0x7f459c08a460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:24.180 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:24.177 7f459bfff700 1 --2- 172.21.2.102:0/1672586605 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f459c0b19b0 0x7f459c08a460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:24.181 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:24.177 7f459bfff700 1 -- 172.21.2.102:0/1672586605 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f459c192d90 con 0x7f459c19eb50 2021-02-10T11:23:24.181 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:24.177 7f45a1342700 1 --2- 172.21.2.102:0/1672586605 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f459c0b19b0 0x7f459c08a460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:23:24.181 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:24.177 7f459bfff700 1 --2- 172.21.2.102:0/1672586605 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f459c19eb50 0x7f459c0b1390 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 rx=0x7f458c009980 tx=0x7f458c00bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:24.181 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:24.177 7f45997fa700 1 -- 172.21.2.102:0/1672586605 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f458c0066d0 con 0x7f459c19eb50 2021-02-10T11:23:24.181 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:24.177 7f45997fa700 1 -- 172.21.2.102:0/1672586605 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f458c00c2f0 con 0x7f459c19eb50 2021-02-10T11:23:24.181 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:24.177 7f45997fa700 1 -- 172.21.2.102:0/1672586605 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f458c00c810 con 0x7f459c19eb50 2021-02-10T11:23:24.182 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:24.177 7f45a2d25700 1 -- 172.21.2.102:0/1672586605 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f459c192d90 con 0x7f459c19eb50 2021-02-10T11:23:24.182 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:24.177 7f45a2d25700 1 -- 172.21.2.102:0/1672586605 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f459c0b20a0 con 0x7f459c19eb50 2021-02-10T11:23:24.182 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:24.178 7f45997fa700 1 -- 172.21.2.102:0/1672586605 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f458c001010 con 0x7f459c19eb50 2021-02-10T11:23:24.182 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:24.178 7f45997fa700 1 --2- 172.21.2.102:0/1672586605 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f45800186c0 0x7f458001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:24.182 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:24.178 7f45997fa700 1 -- 172.21.2.102:0/1672586605 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f458c032790 con 0x7f459c19eb50 2021-02-10T11:23:24.182 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:24.178 7f45a0b41700 1 --2- 172.21.2.102:0/1672586605 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f45800186c0 0x7f458001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:24.183 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:24.178 7f45a0b41700 1 --2- 172.21.2.102:0/1672586605 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f45800186c0 0x7f458001abc0 crc :-1 s=READY pgs=574 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:24.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:24.209 7f45a2d25700 1 -- 172.21.2.102:0/1672586605 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f459c107120 con 0x7f459c19eb50 2021-02-10T11:23:24.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:24.212 7f45997fa700 1 -- 172.21.2.102:0/1672586605 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f458c000cc0 con 0x7f459c19eb50 2021-02-10T11:23:24.338 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:24.340 7f45a2d25700 1 -- 172.21.2.102:0/1672586605 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f459c07ed80 con 0x7f459c19eb50 2021-02-10T11:23:24.338 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:24.340 7f45997fa700 1 -- 172.21.2.102:0/1672586605 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+2099 (secure 0 0 0) 0x7f458c001010 con 0x7f459c19eb50 2021-02-10T11:23:24.339 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:23:24.339 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":30,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7557,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.102:6835/1941781958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6834","nonce":1941781958},{"type":"v1","addr":"172.21.2.102:6835","nonce":1941781958}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":26}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:23:18.002680","modified":"2021-02-10 11:23:20.009191","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5606},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5606":{"gid":5606,"name":"b","rank":0,"incarnation":27,"state":"up:active","state_seq":3,"addr":"172.21.2.113:6833/1872558991","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6832","nonce":1872558991},{"type":"v1","addr":"172.21.2.113:6833","nonce":1872558991}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":4}]} 2021-02-10T11:23:24.339 INFO:teuthology.orchestra.run.gibba002.stderr:dumped fsmap epoch 30 2021-02-10T11:23:24.341 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:24.342 7f45a2d25700 1 -- 172.21.2.102:0/1672586605 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f45800186c0 msgr2=0x7f458001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:24.341 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:24.342 7f45a2d25700 1 --2- 172.21.2.102:0/1672586605 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f45800186c0 0x7f458001abc0 crc :-1 s=READY pgs=574 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:24.341 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:24.342 7f45a2d25700 1 -- 172.21.2.102:0/1672586605 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f459c19eb50 msgr2=0x7f459c0b1390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:24.341 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:24.342 7f45a2d25700 1 --2- 172.21.2.102:0/1672586605 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f459c19eb50 0x7f459c0b1390 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 rx=0x7f458c009980 tx=0x7f458c00bcd0).stop 2021-02-10T11:23:24.341 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:24.343 7f45a2d25700 1 -- 172.21.2.102:0/1672586605 shutdown_connections 2021-02-10T11:23:24.341 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:24.343 7f45a2d25700 1 --2- 172.21.2.102:0/1672586605 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f45800186c0 0x7f458001abc0 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:24.341 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:24.343 7f45a2d25700 1 --2- 172.21.2.102:0/1672586605 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f459c0b19b0 0x7f459c08a460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:24.341 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:24.343 7f45a2d25700 1 --2- 172.21.2.102:0/1672586605 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f459c19eb50 0x7f459c0b1390 unknown :-1 s=CLOSED pgs=1202 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:24.341 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:24.343 7f45a2d25700 1 --2- 172.21.2.102:0/1672586605 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f459c192990 0x7f459c19e530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:24.341 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:24.343 7f45a2d25700 1 -- 172.21.2.102:0/1672586605 >> 172.21.2.102:0/1672586605 conn(0x7f459c1ac620 msgr2=0x7f459c19fee0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:24.341 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:24.343 7f45a2d25700 1 -- 172.21.2.102:0/1672586605 shutdown_connections 2021-02-10T11:23:24.341 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:24.343 7f45a2d25700 1 -- 172.21.2.102:0/1672586605 wait complete. 2021-02-10T11:23:24.551 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 30, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:23:18.002680', 'modified': '2021-02-10 11:23:20.009191', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5606}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5606': {'gid': 5606, 'name': 'b', 'rank': 0, 'incarnation': 27, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.113:6833/1872558991', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.113:6832', 'nonce': 1872558991}, {'type': 'v1', 'addr': '172.21.2.113:6833', 'nonce': 1872558991}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [9], 'metadata_pool': 8, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:23:24.551 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:23:24.551 DEBUG:teuthology.orchestra.run.gibba013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:23:24.951 INFO:teuthology.orchestra.run.gibba013.stdout:{ 2021-02-10T11:23:24.951 INFO:teuthology.orchestra.run.gibba013.stdout: "cluster_fsid": "68db30ea-dbce-4aef-a6df-f01c6fea0513", 2021-02-10T11:23:24.951 INFO:teuthology.orchestra.run.gibba013.stdout: "whoami": 0, 2021-02-10T11:23:24.951 INFO:teuthology.orchestra.run.gibba013.stdout: "id": 5606, 2021-02-10T11:23:24.951 INFO:teuthology.orchestra.run.gibba013.stdout: "want_state": "up:active", 2021-02-10T11:23:24.951 INFO:teuthology.orchestra.run.gibba013.stdout: "state": "up:active", 2021-02-10T11:23:24.951 INFO:teuthology.orchestra.run.gibba013.stdout: "rank_uptime": 6.948980272, 2021-02-10T11:23:24.952 INFO:teuthology.orchestra.run.gibba013.stdout: "mdsmap_epoch": 30, 2021-02-10T11:23:24.952 INFO:teuthology.orchestra.run.gibba013.stdout: "osdmap_epoch": 48, 2021-02-10T11:23:24.952 INFO:teuthology.orchestra.run.gibba013.stdout: "osdmap_epoch_barrier": 48, 2021-02-10T11:23:24.952 INFO:teuthology.orchestra.run.gibba013.stdout: "uptime": 12.590274594 2021-02-10T11:23:24.952 INFO:teuthology.orchestra.run.gibba013.stdout:} 2021-02-10T11:23:24.955 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "68db30ea-dbce-4aef-a6df-f01c6fea0513", "whoami": 0, "id": 5606, "want_state": "up:active", "state": "up:active", "rank_uptime": 6.948980272, "mdsmap_epoch": 30, "osdmap_epoch": 48, "osdmap_epoch_barrier": 48, "uptime": 12.590274594 } 2021-02-10T11:23:24.955 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:23:25.383 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:25.366 7f6f75b25700 1 Processor -- start 2021-02-10T11:23:25.383 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:25.367 7f6f75b25700 1 -- start start 2021-02-10T11:23:25.383 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:25.367 7f6f75b25700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f6f700e5070 0x7f6f701b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:25.383 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:25.367 7f6f75b25700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f7007ed80 con 0x7f6f701b3e40 2021-02-10T11:23:25.383 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:25.367 7f6f75b25700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f7007e670 con 0x7f6f700e5470 2021-02-10T11:23:25.383 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:25.367 7f6f75b25700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f7007e890 con 0x7f6f700e5070 2021-02-10T11:23:25.383 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:25.367 7f6f6effd700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f6f701b3e40 0x7f6f7007d690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:48560/0 (socket says 172.21.2.102:48560) 2021-02-10T11:23:25.383 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:25.367 7f6f6effd700 1 -- 172.21.2.102:0/2784848134 learned_addr learned my addr 172.21.2.102:0/2784848134 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:23:25.383 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:25.368 7f6f6e7fc700 1 -- 172.21.2.102:0/2784848134 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 643401629 0 0) 0x7f6f640010b0 con 0x7f6f701b3e40 2021-02-10T11:23:25.383 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:25.368 7f6f6f7fe700 1 --2- 172.21.2.102:0/2784848134 >> v2:172.21.2.113:3300/0 conn(0x7f6f700e5070 0x7f6f701b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:25.383 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:25.368 7f6f6e7fc700 1 -- 172.21.2.102:0/2784848134 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f58002fe0 con 0x7f6f701b3e40 2021-02-10T11:23:25.384 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:25.368 7f6f6e7fc700 1 -- 172.21.2.102:0/2784848134 <== mon.1 v1:172.21.2.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2994577169 0 0) 0x7f6f68001400 con 0x7f6f700e5470 2021-02-10T11:23:25.384 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:25.368 7f6f6e7fc700 1 -- 172.21.2.102:0/2784848134 --> v1:172.21.2.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f58003210 con 0x7f6f700e5470 2021-02-10T11:23:25.384 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:25.368 7f6f6e7fc700 1 -- 172.21.2.102:0/2784848134 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2283040669 0 0) 0x7f6f640010b0 con 0x7f6f701b3e40 2021-02-10T11:23:25.384 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:25.368 7f6f6e7fc700 1 -- 172.21.2.102:0/2784848134 >> v2:172.21.2.113:3300/0 conn(0x7f6f700e5070 msgr2=0x7f6f701b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:25.384 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:25.368 7f6f6e7fc700 1 --2- 172.21.2.102:0/2784848134 >> v2:172.21.2.113:3300/0 conn(0x7f6f700e5070 0x7f6f701b3820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:25.384 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:25.368 7f6f6e7fc700 1 -- 172.21.2.102:0/2784848134 >> v1:172.21.2.113:6789/0 conn(0x7f6f700e5470 legacy=0x7f6f7007de80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:25.384 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:25.368 7f6f6e7fc700 1 -- 172.21.2.102:0/2784848134 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f7007eab0 con 0x7f6f701b3e40 2021-02-10T11:23:25.384 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:25.368 7f6f6e7fc700 1 -- 172.21.2.102:0/2784848134 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7f6f640015b0 con 0x7f6f701b3e40 2021-02-10T11:23:25.384 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:25.368 7f6f6e7fc700 1 -- 172.21.2.102:0/2784848134 >> v1:172.21.2.102:6789/0 conn(0x7f6f701b3e40 legacy=0x7f6f7007d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:25.384 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:25.368 7f6f6e7fc700 1 --2- 172.21.2.102:0/2784848134 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6f58003980 0x7f6f58003d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:25.384 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:25.368 7f6f6e7fc700 1 --2- 172.21.2.102:0/2784848134 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6f58006880 0x7f6f58008d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:25.384 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:25.368 7f6f6e7fc700 1 --2- 172.21.2.102:0/2784848134 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6f58009640 0x7f6f5800bb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:25.384 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:25.368 7f6f6e7fc700 1 -- 172.21.2.102:0/2784848134 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f5800c390 con 0x7f6f58003980 2021-02-10T11:23:25.385 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:25.368 7f6f6e7fc700 1 -- 172.21.2.102:0/2784848134 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f58050580 con 0x7f6f58009640 2021-02-10T11:23:25.385 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:25.368 7f6f6e7fc700 1 -- 172.21.2.102:0/2784848134 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f580507a0 con 0x7f6f58006880 2021-02-10T11:23:25.385 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:25.368 7f6f6f7fe700 1 --2- 172.21.2.102:0/2784848134 >> v2:172.21.2.113:3300/0 conn(0x7f6f700e5070 0x7f6f701b3820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:23:25.385 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:25.368 7f6f6f7fe700 1 --2- 172.21.2.102:0/2784848134 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6f58003980 0x7f6f58003d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:25.385 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:25.369 7f6f6effd700 1 --2- 172.21.2.102:0/2784848134 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6f58006880 0x7f6f58008d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:25.385 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:25.369 7f6f6ffff700 1 --2- 172.21.2.102:0/2784848134 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6f58009640 0x7f6f5800bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:25.385 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:25.369 7f6f6f7fe700 1 -- 172.21.2.102:0/2784848134 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6f58006880 msgr2=0x7f6f58008d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:25.385 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:25.369 7f6f6f7fe700 1 --2- 172.21.2.102:0/2784848134 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6f58006880 0x7f6f58008d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:25.385 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:25.369 7f6f6f7fe700 1 -- 172.21.2.102:0/2784848134 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6f58009640 msgr2=0x7f6f5800bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:25.385 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:25.369 7f6f6f7fe700 1 --2- 172.21.2.102:0/2784848134 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6f58009640 0x7f6f5800bb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:25.385 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:25.369 7f6f6f7fe700 1 -- 172.21.2.102:0/2784848134 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f58050a40 con 0x7f6f58003980 2021-02-10T11:23:25.385 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:25.369 7f6f6ffff700 1 --2- 172.21.2.102:0/2784848134 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6f58009640 0x7f6f5800bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:23:25.385 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:25.369 7f6f6ffff700 1 -- 172.21.2.102:0/2784848134 reap_dead start 2021-02-10T11:23:25.385 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:25.369 7f6f6effd700 1 --2- 172.21.2.102:0/2784848134 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6f58006880 0x7f6f58008d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:23:25.386 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:25.369 7f6f6f7fe700 1 --2- 172.21.2.102:0/2784848134 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6f58003980 0x7f6f58003d80 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 rx=0x7f6f60000b80 tx=0x7f6f60009e80).ready entity=mon.0 client_cookie=13f567e8591c58d5 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:25.386 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:25.369 7f6f6e7fc700 1 -- 172.21.2.102:0/2784848134 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6f60006dc0 con 0x7f6f58003980 2021-02-10T11:23:25.386 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:25.369 7f6f6e7fc700 1 -- 172.21.2.102:0/2784848134 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f60007070 con 0x7f6f58003980 2021-02-10T11:23:25.386 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:25.369 7f6f6e7fc700 1 -- 172.21.2.102:0/2784848134 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6f60007590 con 0x7f6f58003980 2021-02-10T11:23:25.386 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:25.369 7f6f75b25700 1 -- 172.21.2.102:0/2784848134 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6f58003980 msgr2=0x7f6f58003d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:25.386 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:25.369 7f6f75b25700 1 --2- 172.21.2.102:0/2784848134 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6f58003980 0x7f6f58003d80 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 rx=0x7f6f60000b80 tx=0x7f6f60009e80).stop 2021-02-10T11:23:25.386 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:25.370 7f6f75b25700 1 -- 172.21.2.102:0/2784848134 shutdown_connections 2021-02-10T11:23:25.386 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:25.370 7f6f75b25700 1 --2- 172.21.2.102:0/2784848134 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6f58003980 0x7f6f58003d80 unknown :-1 s=CLOSED pgs=1204 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:25.386 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:25.370 7f6f75b25700 1 -- 172.21.2.102:0/2784848134 >> 172.21.2.102:0/2784848134 conn(0x7f6f700ce690 msgr2=0x7f6f700e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:25.386 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:25.370 7f6f75b25700 1 -- 172.21.2.102:0/2784848134 shutdown_connections 2021-02-10T11:23:25.386 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:25.370 7f6f75b25700 1 -- 172.21.2.102:0/2784848134 wait complete. 2021-02-10T11:23:25.386 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:25.385 7f6f75b25700 1 Processor -- start 2021-02-10T11:23:25.386 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:25.385 7f6f75b25700 1 -- start start 2021-02-10T11:23:25.388 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:25.385 7f6f75b25700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6f70194000 0x7f6f7019f680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:25.388 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:25.385 7f6f75b25700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6f7019fde0 0x7f6f7009bed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:25.388 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:25.385 7f6f75b25700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6f701a01e0 0x7f6f700895c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:25.388 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:25.385 7f6f75b25700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f70194400 con 0x7f6f701a01e0 2021-02-10T11:23:25.388 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:25.385 7f6f75b25700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f7009c4f0 con 0x7f6f70194000 2021-02-10T11:23:25.388 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:25.385 7f6f75b25700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f70089be0 con 0x7f6f7019fde0 2021-02-10T11:23:25.388 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:25.385 7f6f6ffff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6f701a01e0 0x7f6f700895c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:25.388 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:25.385 7f6f6ffff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6f701a01e0 0x7f6f700895c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:56346/0 (socket says 172.21.2.102:56346) 2021-02-10T11:23:25.388 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:25.385 7f6f6ffff700 1 -- 172.21.2.102:0/571380579 learned_addr learned my addr 172.21.2.102:0/571380579 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:23:25.389 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:25.386 7f6f6ffff700 1 -- 172.21.2.102:0/571380579 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6f7019fde0 msgr2=0x7f6f7009bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:25.389 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:25.386 7f6f6ffff700 1 --2- 172.21.2.102:0/571380579 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6f7019fde0 0x7f6f7009bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:25.389 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:25.386 7f6f6ffff700 1 -- 172.21.2.102:0/571380579 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6f70194000 msgr2=0x7f6f7019f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:25.389 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:25.386 7f6f6ffff700 1 --2- 172.21.2.102:0/571380579 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6f70194000 0x7f6f7019f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:25.389 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:25.386 7f6f6ffff700 1 -- 172.21.2.102:0/571380579 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f70089e00 con 0x7f6f701a01e0 2021-02-10T11:23:25.389 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:25.386 7f6f6ffff700 1 --2- 172.21.2.102:0/571380579 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6f701a01e0 0x7f6f700895c0 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 rx=0x7f6f68009940 tx=0x7f6f6800cc70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:25.389 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:25.386 7f6f57fff700 1 -- 172.21.2.102:0/571380579 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6f680011f0 con 0x7f6f701a01e0 2021-02-10T11:23:25.389 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:25.386 7f6f57fff700 1 -- 172.21.2.102:0/571380579 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f68003030 con 0x7f6f701a01e0 2021-02-10T11:23:25.389 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:25.386 7f6f57fff700 1 -- 172.21.2.102:0/571380579 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6f68001660 con 0x7f6f701a01e0 2021-02-10T11:23:25.389 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:25.386 7f6f75b25700 1 -- 172.21.2.102:0/571380579 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f70089e00 con 0x7f6f701a01e0 2021-02-10T11:23:25.389 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:25.386 7f6f75b25700 1 -- 172.21.2.102:0/571380579 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f700b8230 con 0x7f6f701a01e0 2021-02-10T11:23:25.389 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:25.387 7f6f57fff700 1 -- 172.21.2.102:0/571380579 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f6f68000de0 con 0x7f6f701a01e0 2021-02-10T11:23:25.389 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:25.387 7f6f57fff700 1 --2- 172.21.2.102:0/571380579 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6f500186c0 0x7f6f5001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:25.389 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:25.387 7f6f57fff700 1 -- 172.21.2.102:0/571380579 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f6f68032900 con 0x7f6f701a01e0 2021-02-10T11:23:25.390 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:25.387 7f6f6f7fe700 1 --2- 172.21.2.102:0/571380579 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6f500186c0 0x7f6f5001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:25.390 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:25.387 7f6f6f7fe700 1 --2- 172.21.2.102:0/571380579 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6f500186c0 0x7f6f5001abc0 crc :-1 s=READY pgs=575 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:25.415 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:25.417 7f6f75b25700 1 -- 172.21.2.102:0/571380579 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f70089090 con 0x7f6f701a01e0 2021-02-10T11:23:25.417 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:25.419 7f6f57fff700 1 -- 172.21.2.102:0/571380579 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f6f6801f6a0 con 0x7f6f701a01e0 2021-02-10T11:23:25.540 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:25.542 7f6f75b25700 1 -- 172.21.2.102:0/571380579 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6f70089090 con 0x7f6f701a01e0 2021-02-10T11:23:25.541 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:25.543 7f6f57fff700 1 -- 172.21.2.102:0/571380579 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v48) v1 ==== 74+0+13993 (secure 0 0 0) 0x7f6f68000de0 con 0x7f6f701a01e0 2021-02-10T11:23:25.541 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:23:25.541 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":48,"fsid":"68db30ea-dbce-4aef-a6df-f01c6fea0513","created":"2021-02-10 11:03:06.331557","modified":"2021-02-10 11:23:19.005019","last_up_change":"2021-02-10 11:03:32.690562","last_in_change":"2021-02-10 11:03:32.690562","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:03:37.322519","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:23:15.973296","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"47","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":9,"pool_name":"cephfs_data","create_time":"2021-02-10 11:23:17.045434","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"48","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"be691b4d-a039-408a-af80-9d53f02b6f1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6818","nonce":36088},{"type":"v1","addr":"172.21.2.102:6820","nonce":36088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6822","nonce":36088},{"type":"v1","addr":"172.21.2.102:6824","nonce":36088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6830","nonce":36088},{"type":"v1","addr":"172.21.2.102:6832","nonce":36088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6826","nonce":36088},{"type":"v1","addr":"172.21.2.102:6828","nonce":36088}]},"public_addr":"172.21.2.102:6820/36088","cluster_addr":"172.21.2.102:6824/36088","heartbeat_back_addr":"172.21.2.102:6832/36088","heartbeat_front_addr":"172.21.2.102:6828/36088","state":["exists","up"]},{"osd":1,"uuid":"64e3ccc8-9c69-4d7c-865e-af8e83822635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6819","nonce":36087},{"type":"v1","addr":"172.21.2.102:6821","nonce":36087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6823","nonce":36087},{"type":"v1","addr":"172.21.2.102:6825","nonce":36087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6831","nonce":36087},{"type":"v1","addr":"172.21.2.102:6833","nonce":36087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6827","nonce":36087},{"type":"v1","addr":"172.21.2.102:6829","nonce":36087}]},"public_addr":"172.21.2.102:6821/36087","cluster_addr":"172.21.2.102:6825/36087","heartbeat_back_addr":"172.21.2.102:6833/36087","heartbeat_front_addr":"172.21.2.102:6829/36087","state":["exists","up"]},{"osd":2,"uuid":"153700af-747f-481f-bdf4-d6072d3a842f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6810","nonce":36089},{"type":"v1","addr":"172.21.2.102:6811","nonce":36089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6812","nonce":36089},{"type":"v1","addr":"172.21.2.102:6813","nonce":36089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6816","nonce":36089},{"type":"v1","addr":"172.21.2.102:6817","nonce":36089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6814","nonce":36089},{"type":"v1","addr":"172.21.2.102:6815","nonce":36089}]},"public_addr":"172.21.2.102:6811/36089","cluster_addr":"172.21.2.102:6813/36089","heartbeat_back_addr":"172.21.2.102:6817/36089","heartbeat_front_addr":"172.21.2.102:6815/36089","state":["exists","up"]},{"osd":3,"uuid":"cbc678f4-83a9-45e6-8830-2f7c232b1ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6802","nonce":36090},{"type":"v1","addr":"172.21.2.102:6803","nonce":36090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6804","nonce":36090},{"type":"v1","addr":"172.21.2.102:6805","nonce":36090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6808","nonce":36090},{"type":"v1","addr":"172.21.2.102:6809","nonce":36090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6806","nonce":36090},{"type":"v1","addr":"172.21.2.102:6807","nonce":36090}]},"public_addr":"172.21.2.102:6803/36090","cluster_addr":"172.21.2.102:6805/36090","heartbeat_back_addr":"172.21.2.102:6809/36090","heartbeat_front_addr":"172.21.2.102:6807/36090","state":["exists","up"]},{"osd":4,"uuid":"c01e4008-b258-4f10-afd2-9e7f37214351","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6808","nonce":36368},{"type":"v1","addr":"172.21.2.113:6811","nonce":36368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6814","nonce":36368},{"type":"v1","addr":"172.21.2.113:6818","nonce":36368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6826","nonce":36368},{"type":"v1","addr":"172.21.2.113:6828","nonce":36368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6820","nonce":36368},{"type":"v1","addr":"172.21.2.113:6824","nonce":36368}]},"public_addr":"172.21.2.113:6811/36368","cluster_addr":"172.21.2.113:6818/36368","heartbeat_back_addr":"172.21.2.113:6828/36368","heartbeat_front_addr":"172.21.2.113:6824/36368","state":["exists","up"]},{"osd":5,"uuid":"551f6007-daf8-4a98-8ee0-c2b16bcccb5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6802","nonce":36366},{"type":"v1","addr":"172.21.2.113:6805","nonce":36366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6807","nonce":36366},{"type":"v1","addr":"172.21.2.113:6810","nonce":36366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6819","nonce":36366},{"type":"v1","addr":"172.21.2.113:6822","nonce":36366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6812","nonce":36366},{"type":"v1","addr":"172.21.2.113:6815","nonce":36366}]},"public_addr":"172.21.2.113:6805/36366","cluster_addr":"172.21.2.113:6810/36366","heartbeat_back_addr":"172.21.2.113:6822/36366","heartbeat_front_addr":"172.21.2.113:6815/36366","state":["exists","up"]},{"osd":6,"uuid":"bf67c040-0e50-48be-b102-c3943bc9ebef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6800","nonce":36367},{"type":"v1","addr":"172.21.2.113:6801","nonce":36367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6803","nonce":36367},{"type":"v1","addr":"172.21.2.113:6804","nonce":36367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6813","nonce":36367},{"type":"v1","addr":"172.21.2.113:6816","nonce":36367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6806","nonce":36367},{"type":"v1","addr":"172.21.2.113:6809","nonce":36367}]},"public_addr":"172.21.2.113:6801/36367","cluster_addr":"172.21.2.113:6804/36367","heartbeat_back_addr":"172.21.2.113:6816/36367","heartbeat_front_addr":"172.21.2.113:6809/36367","state":["exists","up"]},{"osd":7,"uuid":"f5461f80-018d-428f-be89-2eb2bcbe7bb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6817","nonce":36365},{"type":"v1","addr":"172.21.2.113:6821","nonce":36365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6823","nonce":36365},{"type":"v1","addr":"172.21.2.113:6825","nonce":36365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6830","nonce":36365},{"type":"v1","addr":"172.21.2.113:6831","nonce":36365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6827","nonce":36365},{"type":"v1","addr":"172.21.2.113:6829","nonce":36365}]},"public_addr":"172.21.2.113:6821/36365","cluster_addr":"172.21.2.113:6825/36365","heartbeat_back_addr":"172.21.2.113:6831/36365","heartbeat_front_addr":"172.21.2.113:6829/36365","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:23:25.542 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:25.544 7f6f75b25700 1 -- 172.21.2.102:0/571380579 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6f500186c0 msgr2=0x7f6f5001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:25.542 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:25.544 7f6f75b25700 1 --2- 172.21.2.102:0/571380579 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6f500186c0 0x7f6f5001abc0 crc :-1 s=READY pgs=575 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:25.543 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:25.544 7f6f75b25700 1 -- 172.21.2.102:0/571380579 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6f701a01e0 msgr2=0x7f6f700895c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:25.543 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:25.544 7f6f75b25700 1 --2- 172.21.2.102:0/571380579 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6f701a01e0 0x7f6f700895c0 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 rx=0x7f6f68009940 tx=0x7f6f6800cc70).stop 2021-02-10T11:23:25.543 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:25.545 7f6f75b25700 1 -- 172.21.2.102:0/571380579 shutdown_connections 2021-02-10T11:23:25.543 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:25.545 7f6f75b25700 1 --2- 172.21.2.102:0/571380579 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6f701a01e0 0x7f6f700895c0 unknown :-1 s=CLOSED pgs=1205 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:25.543 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:25.545 7f6f75b25700 1 --2- 172.21.2.102:0/571380579 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6f7019fde0 0x7f6f7009bed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:25.543 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:25.545 7f6f75b25700 1 --2- 172.21.2.102:0/571380579 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6f500186c0 0x7f6f5001abc0 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:25.543 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:25.545 7f6f75b25700 1 --2- 172.21.2.102:0/571380579 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6f70194000 0x7f6f7019f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:25.543 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:25.545 7f6f75b25700 1 -- 172.21.2.102:0/571380579 >> 172.21.2.102:0/571380579 conn(0x7f6f70092c20 msgr2=0x7f6f70195070 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:25.543 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:25.545 7f6f75b25700 1 -- 172.21.2.102:0/571380579 shutdown_connections 2021-02-10T11:23:25.543 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:25.545 7f6f75b25700 1 -- 172.21.2.102:0/571380579 wait complete. 2021-02-10T11:23:25.551 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:03:37.322519', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 8, 'pool_name': 'cephfs_metadata', 'create_time': '2021-02-10 11:23:15.973296', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '47', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 9, 'pool_name': 'cephfs_data', 'create_time': '2021-02-10 11:23:17.045434', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '48', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-02-10T11:23:25.552 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:23:25.982 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:25.968 7f63bf6dd700 1 Processor -- start 2021-02-10T11:23:25.983 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:25.968 7f63bf6dd700 1 -- start start 2021-02-10T11:23:25.983 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:25.969 7f63bf6dd700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f63b81b0470 0x7f63b81b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:25.983 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:25.969 7f63bf6dd700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f63b80ef610 0x7f63b80efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:25.983 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:25.969 7f63bf6dd700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f63b80f0030 0x7f63b80f0430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:25.983 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:25.969 7f63bf6dd700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63b80e5ea0 con 0x7f63b80ef610 2021-02-10T11:23:25.983 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:25.969 7f63bf6dd700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63b80f0b50 con 0x7f63b81b0470 2021-02-10T11:23:25.983 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:25.969 7f63bf6dd700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63b80e52e0 con 0x7f63b80f0030 2021-02-10T11:23:25.983 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:25.969 7f63bccf8700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f63b80ef610 0x7f63b80efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:25.983 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:25.969 7f63bccf8700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f63b80ef610 0x7f63b80efa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:56352/0 (socket says 172.21.2.102:56352) 2021-02-10T11:23:25.983 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:25.969 7f63bccf8700 1 -- 172.21.2.102:0/3833041536 learned_addr learned my addr 172.21.2.102:0/3833041536 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:23:25.983 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:25.969 7f63bccf8700 1 -- 172.21.2.102:0/3833041536 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f63b80f0030 msgr2=0x7f63b80f0430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:25.983 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:25.969 7f63bccf8700 1 --2- 172.21.2.102:0/3833041536 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f63b80f0030 0x7f63b80f0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:25.984 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:25.969 7f63bccf8700 1 -- 172.21.2.102:0/3833041536 >> v2:172.21.2.113:3300/0 conn(0x7f63b81b0470 msgr2=0x7f63b81b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:25.984 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:25.969 7f63bccf8700 1 --2- 172.21.2.102:0/3833041536 >> v2:172.21.2.113:3300/0 conn(0x7f63b81b0470 0x7f63b81b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:25.984 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:25.969 7f63bccf8700 1 -- 172.21.2.102:0/3833041536 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63b80f10d0 con 0x7f63b80ef610 2021-02-10T11:23:25.984 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:25.970 7f63bccf8700 1 --2- 172.21.2.102:0/3833041536 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f63b80ef610 0x7f63b80efa10 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 rx=0x7f63b4006a80 tx=0x7f63b4009f00).ready entity=mon.0 client_cookie=da7114aa8246a957 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:25.984 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:25.970 7f63affff700 1 -- 172.21.2.102:0/3833041536 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f63b400b5c0 con 0x7f63b80ef610 2021-02-10T11:23:25.984 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:25.970 7f63affff700 1 -- 172.21.2.102:0/3833041536 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63b400b8c0 con 0x7f63b80ef610 2021-02-10T11:23:25.984 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:25.970 7f63affff700 1 -- 172.21.2.102:0/3833041536 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f63b400bda0 con 0x7f63b80ef610 2021-02-10T11:23:25.984 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:25.970 7f63bf6dd700 1 -- 172.21.2.102:0/3833041536 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f63b80ef610 msgr2=0x7f63b80efa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:25.984 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:25.970 7f63bf6dd700 1 --2- 172.21.2.102:0/3833041536 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f63b80ef610 0x7f63b80efa10 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 rx=0x7f63b4006a80 tx=0x7f63b4009f00).stop 2021-02-10T11:23:25.984 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:25.970 7f63bf6dd700 1 -- 172.21.2.102:0/3833041536 shutdown_connections 2021-02-10T11:23:25.984 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:25.970 7f63bf6dd700 1 --2- 172.21.2.102:0/3833041536 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f63b80f0030 0x7f63b80f0430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:25.985 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:25.970 7f63bf6dd700 1 --2- 172.21.2.102:0/3833041536 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f63b80ef610 0x7f63b80efa10 unknown :-1 s=CLOSED pgs=1206 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:25.985 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:25.970 7f63bf6dd700 1 --2- 172.21.2.102:0/3833041536 >> v2:172.21.2.113:3300/0 conn(0x7f63b81b0470 0x7f63b81b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:25.985 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:25.970 7f63bf6dd700 1 -- 172.21.2.102:0/3833041536 >> 172.21.2.102:0/3833041536 conn(0x7f63b80ce3e0 msgr2=0x7f63b80e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:25.985 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:25.971 7f63bf6dd700 1 -- 172.21.2.102:0/3833041536 shutdown_connections 2021-02-10T11:23:25.985 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:25.971 7f63bf6dd700 1 -- 172.21.2.102:0/3833041536 wait complete. 2021-02-10T11:23:25.986 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:25.985 7f63bf6dd700 1 Processor -- start 2021-02-10T11:23:25.987 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:25.985 7f63bf6dd700 1 -- start start 2021-02-10T11:23:25.987 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:25.985 7f63bf6dd700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f63b8190920 0x7f63b808f600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:25.987 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:25.985 7f63bf6dd700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f63b8190d20 0x7f63b808fc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:25.987 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:25.985 7f63bf6dd700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f63b8193270 0x7f63b8193670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:25.987 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:25.985 7f63bf6dd700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63b8090470 con 0x7f63b8190d20 2021-02-10T11:23:25.987 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:25.985 7f63bf6dd700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63b8193c90 con 0x7f63b8193270 2021-02-10T11:23:25.987 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:25.985 7f63bf6dd700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63b8193e70 con 0x7f63b8190920 2021-02-10T11:23:25.987 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:25.986 7f63bccf8700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f63b8190d20 0x7f63b808fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:25.987 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:25.986 7f63bccf8700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f63b8190d20 0x7f63b808fc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:56358/0 (socket says 172.21.2.102:56358) 2021-02-10T11:23:25.987 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:25.986 7f63bccf8700 1 -- 172.21.2.102:0/992449307 learned_addr learned my addr 172.21.2.102:0/992449307 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:23:25.987 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:25.986 7f63bccf8700 1 -- 172.21.2.102:0/992449307 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f63b8190920 msgr2=0x7f63b808f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:25.988 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:25.986 7f63bccf8700 1 --2- 172.21.2.102:0/992449307 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f63b8190920 0x7f63b808f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:25.988 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:25.986 7f63bccf8700 1 -- 172.21.2.102:0/992449307 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f63b8193270 msgr2=0x7f63b8193670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:25.988 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:25.986 7f63bdcfa700 1 --2- 172.21.2.102:0/992449307 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f63b8193270 0x7f63b8193670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:25.988 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:25.986 7f63bccf8700 1 --2- 172.21.2.102:0/992449307 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f63b8193270 0x7f63b8193670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:25.988 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:25.986 7f63bccf8700 1 -- 172.21.2.102:0/992449307 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63b8194090 con 0x7f63b8190d20 2021-02-10T11:23:25.988 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:25.986 7f63bdcfa700 1 --2- 172.21.2.102:0/992449307 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f63b8193270 0x7f63b8193670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:23:25.988 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:25.986 7f63bccf8700 1 --2- 172.21.2.102:0/992449307 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f63b8190d20 0x7f63b808fc20 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 rx=0x7f63b4006a90 tx=0x7f63b400bc90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:25.988 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:25.986 7f63adffb700 1 -- 172.21.2.102:0/992449307 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f63b400c2f0 con 0x7f63b8190d20 2021-02-10T11:23:25.988 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:25.986 7f63adffb700 1 -- 172.21.2.102:0/992449307 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63b4006710 con 0x7f63b8190d20 2021-02-10T11:23:25.988 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:25.986 7f63bf6dd700 1 -- 172.21.2.102:0/992449307 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63b8194090 con 0x7f63b8190d20 2021-02-10T11:23:25.988 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:25.986 7f63adffb700 1 -- 172.21.2.102:0/992449307 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f63b400c870 con 0x7f63b8190d20 2021-02-10T11:23:25.988 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:25.987 7f63bf6dd700 1 -- 172.21.2.102:0/992449307 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63b8194090 con 0x7f63b8190d20 2021-02-10T11:23:25.989 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:25.987 7f63adffb700 1 -- 172.21.2.102:0/992449307 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f63b4001010 con 0x7f63b8190d20 2021-02-10T11:23:25.989 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:25.987 7f63adffb700 1 --2- 172.21.2.102:0/992449307 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6398018730 0x7f639801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:25.989 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:25.987 7f63adffb700 1 -- 172.21.2.102:0/992449307 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f63b40329c0 con 0x7f63b8190d20 2021-02-10T11:23:25.989 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:25.987 7f63bd4f9700 1 --2- 172.21.2.102:0/992449307 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6398018730 0x7f639801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:25.989 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:25.987 7f63bd4f9700 1 --2- 172.21.2.102:0/992449307 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6398018730 0x7f639801ac10 crc :-1 s=READY pgs=576 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:26.016 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:26.019 7f63bf6dd700 1 -- 172.21.2.102:0/992449307 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63b8105af0 con 0x7f63b8190d20 2021-02-10T11:23:26.019 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:26.021 7f63adffb700 1 -- 172.21.2.102:0/992449307 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f63b400bdd0 con 0x7f63b8190d20 2021-02-10T11:23:26.147 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:26.149 7f63bf6dd700 1 -- 172.21.2.102:0/992449307 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f63b80f0b50 con 0x7f63b8190d20 2021-02-10T11:23:26.147 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:26.150 7f63adffb700 1 -- 172.21.2.102:0/992449307 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v48) v1 ==== 92+0+10 (secure 0 0 0) 0x7f63b4001010 con 0x7f63b8190d20 2021-02-10T11:23:26.148 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:23:26.149 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:26.151 7f63bf6dd700 1 -- 172.21.2.102:0/992449307 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6398018730 msgr2=0x7f639801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:26.150 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:26.151 7f63bf6dd700 1 --2- 172.21.2.102:0/992449307 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6398018730 0x7f639801ac10 crc :-1 s=READY pgs=576 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:26.150 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:26.151 7f63bf6dd700 1 -- 172.21.2.102:0/992449307 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f63b8190d20 msgr2=0x7f63b808fc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:26.150 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:26.151 7f63bf6dd700 1 --2- 172.21.2.102:0/992449307 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f63b8190d20 0x7f63b808fc20 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 rx=0x7f63b4006a90 tx=0x7f63b400bc90).stop 2021-02-10T11:23:26.150 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:26.152 7f63bf6dd700 1 -- 172.21.2.102:0/992449307 shutdown_connections 2021-02-10T11:23:26.150 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:26.152 7f63bf6dd700 1 --2- 172.21.2.102:0/992449307 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6398018730 0x7f639801ac10 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:26.150 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:26.152 7f63bf6dd700 1 --2- 172.21.2.102:0/992449307 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f63b8193270 0x7f63b8193670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:26.150 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:26.152 7f63bf6dd700 1 --2- 172.21.2.102:0/992449307 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f63b8190d20 0x7f63b808fc20 unknown :-1 s=CLOSED pgs=1207 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:26.150 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:26.152 7f63bf6dd700 1 --2- 172.21.2.102:0/992449307 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f63b8190920 0x7f63b808f600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:26.150 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:26.152 7f63bf6dd700 1 -- 172.21.2.102:0/992449307 >> 172.21.2.102:0/992449307 conn(0x7f63b80bf4e0 msgr2=0x7f63b80be3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:26.150 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:26.152 7f63bf6dd700 1 -- 172.21.2.102:0/992449307 shutdown_connections 2021-02-10T11:23:26.150 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:26.152 7f63bf6dd700 1 -- 172.21.2.102:0/992449307 wait complete. 2021-02-10T11:23:26.158 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-02-10T11:23:26.588 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:26.573 7f1fcd0b5700 1 Processor -- start 2021-02-10T11:23:26.588 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:26.573 7f1fcd0b5700 1 -- start start 2021-02-10T11:23:26.588 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:26.573 7f1fcd0b5700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1fc81b0870 0x7f1fc80f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:26.589 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:26.573 7f1fcd0b5700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fc81b0c70 con 0x7f1fc80efe00 2021-02-10T11:23:26.589 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:26.573 7f1fcd0b5700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fc80f0b50 con 0x7f1fc81b0470 2021-02-10T11:23:26.589 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:26.573 7f1fcd0b5700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fc80e5ea0 con 0x7f1fc81b0870 2021-02-10T11:23:26.589 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:26.574 7f1fc759e700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f1fc80efe00 0x7f1fc80f0200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:48594/0 (socket says 172.21.2.102:48594) 2021-02-10T11:23:26.589 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:26.574 7f1fc759e700 1 -- 172.21.2.102:0/4293028409 learned_addr learned my addr 172.21.2.102:0/4293028409 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:23:26.589 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:26.574 7f1fc5d9b700 1 -- 172.21.2.102:0/4293028409 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3902881980 0 0) 0x7f1fc0001500 con 0x7f1fc80efe00 2021-02-10T11:23:26.589 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:26.574 7f1fc5d9b700 1 -- 172.21.2.102:0/4293028409 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1fb0002fe0 con 0x7f1fc80efe00 2021-02-10T11:23:26.589 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:26.574 7f1fc659c700 1 --2- 172.21.2.102:0/4293028409 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1fc81b0870 0x7f1fc80f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:26.589 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:26.574 7f1fc5d9b700 1 -- 172.21.2.102:0/4293028409 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1505380348 0 0) 0x7f1fc0001460 con 0x7f1fc80efe00 2021-02-10T11:23:26.589 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:26.574 7f1fc5d9b700 1 -- 172.21.2.102:0/4293028409 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1fc81b0870 msgr2=0x7f1fc80f20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:26.589 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:26.574 7f1fc5d9b700 1 --2- 172.21.2.102:0/4293028409 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1fc81b0870 0x7f1fc80f20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:26.589 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:26.574 7f1fc5d9b700 1 -- 172.21.2.102:0/4293028409 >> v1:172.21.2.113:6789/0 conn(0x7f1fc81b0470 legacy=0x7f1fc80ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:26.590 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:26.574 7f1fc5d9b700 1 -- 172.21.2.102:0/4293028409 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fc80e52e0 con 0x7f1fc80efe00 2021-02-10T11:23:26.590 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:26.574 7f1fc5d9b700 1 -- 172.21.2.102:0/4293028409 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7f1fc0001980 con 0x7f1fc80efe00 2021-02-10T11:23:26.590 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:26.574 7f1fc5d9b700 1 -- 172.21.2.102:0/4293028409 >> v1:172.21.2.102:6789/0 conn(0x7f1fc80efe00 legacy=0x7f1fc80f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:26.590 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:26.574 7f1fc5d9b700 1 --2- 172.21.2.102:0/4293028409 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1fb0003aa0 0x7f1fb0005ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:26.590 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:26.574 7f1fc5d9b700 1 --2- 172.21.2.102:0/4293028409 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1fb0006770 0x7f1fb0008c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:26.590 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:26.574 7f1fc5d9b700 1 --2- 172.21.2.102:0/4293028409 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1fb0009590 0x7f1fb000ba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:26.590 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:26.574 7f1fc5d9b700 1 -- 172.21.2.102:0/4293028409 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fb000c2d0 con 0x7f1fb0003aa0 2021-02-10T11:23:26.590 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:26.574 7f1fc5d9b700 1 -- 172.21.2.102:0/4293028409 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fb000c4f0 con 0x7f1fb0006770 2021-02-10T11:23:26.590 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:26.574 7f1fc5d9b700 1 -- 172.21.2.102:0/4293028409 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fb000c710 con 0x7f1fb0009590 2021-02-10T11:23:26.590 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:26.575 7f1fc6d9d700 1 --2- 172.21.2.102:0/4293028409 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1fb0003aa0 0x7f1fb0005ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:26.590 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:26.575 7f1fc659c700 1 --2- 172.21.2.102:0/4293028409 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1fb0006770 0x7f1fb0008c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:26.590 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:26.575 7f1fc759e700 1 --2- 172.21.2.102:0/4293028409 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1fb0009590 0x7f1fb000ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:26.591 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:26.575 7f1fc6d9d700 1 -- 172.21.2.102:0/4293028409 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1fb0009590 msgr2=0x7f1fb000ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:26.591 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:26.575 7f1fc6d9d700 1 --2- 172.21.2.102:0/4293028409 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1fb0009590 0x7f1fb000ba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:26.591 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:26.575 7f1fc6d9d700 1 -- 172.21.2.102:0/4293028409 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1fb0006770 msgr2=0x7f1fb0008c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:26.591 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:26.575 7f1fc6d9d700 1 --2- 172.21.2.102:0/4293028409 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1fb0006770 0x7f1fb0008c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:26.591 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:26.575 7f1fc6d9d700 1 -- 172.21.2.102:0/4293028409 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fb000c9b0 con 0x7f1fb0003aa0 2021-02-10T11:23:26.591 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:26.575 7f1fc6d9d700 1 --2- 172.21.2.102:0/4293028409 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1fb0003aa0 0x7f1fb0005ef0 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 rx=0x7f1fb8006ad0 tx=0x7f1fb804db50).ready entity=mon.0 client_cookie=984732ecf3a66740 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:26.591 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:26.575 7f1fc5d9b700 1 -- 172.21.2.102:0/4293028409 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1fb80066d0 con 0x7f1fb0003aa0 2021-02-10T11:23:26.591 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:26.575 7f1fc5d9b700 1 -- 172.21.2.102:0/4293028409 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fb8002280 con 0x7f1fb0003aa0 2021-02-10T11:23:26.591 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:26.575 7f1fc5d9b700 1 -- 172.21.2.102:0/4293028409 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1fb8002710 con 0x7f1fb0003aa0 2021-02-10T11:23:26.591 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:26.575 7f1fcd0b5700 1 -- 172.21.2.102:0/4293028409 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1fb0003aa0 msgr2=0x7f1fb0005ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:26.591 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:26.575 7f1fcd0b5700 1 --2- 172.21.2.102:0/4293028409 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1fb0003aa0 0x7f1fb0005ef0 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 rx=0x7f1fb8006ad0 tx=0x7f1fb804db50).stop 2021-02-10T11:23:26.591 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:26.575 7f1fc759e700 1 -- 172.21.2.102:0/4293028409 reap_dead start 2021-02-10T11:23:26.591 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:26.576 7f1fcd0b5700 1 -- 172.21.2.102:0/4293028409 shutdown_connections 2021-02-10T11:23:26.592 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:26.576 7f1fcd0b5700 1 -- 172.21.2.102:0/4293028409 >> 172.21.2.102:0/4293028409 conn(0x7f1fc80ce3e0 msgr2=0x7f1fc80e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:26.592 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:26.576 7f1fcd0b5700 1 -- 172.21.2.102:0/4293028409 shutdown_connections 2021-02-10T11:23:26.592 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:26.576 7f1fcd0b5700 1 -- 172.21.2.102:0/4293028409 wait complete. 2021-02-10T11:23:26.592 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:26.591 7f1fcd0b5700 1 Processor -- start 2021-02-10T11:23:26.592 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:26.591 7f1fcd0b5700 1 -- start start 2021-02-10T11:23:26.593 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:26.591 7f1fcd0b5700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1fc809b930 0x7f1fc80b7c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:26.593 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:26.591 7f1fcd0b5700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1fc80b8410 0x7f1fc80b8810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:26.593 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:26.591 7f1fcd0b5700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1fc80c1f80 0x7f1fc80c2380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:26.593 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:26.591 7f1fcd0b5700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fc80c1a00 con 0x7f1fc809b930 2021-02-10T11:23:26.593 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:26.591 7f1fcd0b5700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fc80b8e30 con 0x7f1fc80c1f80 2021-02-10T11:23:26.593 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:26.591 7f1fcd0b5700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fc80b9050 con 0x7f1fc80b8410 2021-02-10T11:23:26.593 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:26.591 7f1fc6d9d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1fc809b930 0x7f1fc80b7c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:26.594 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:26.591 7f1fc6d9d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1fc809b930 0x7f1fc80b7c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:56378/0 (socket says 172.21.2.102:56378) 2021-02-10T11:23:26.594 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:26.591 7f1fc6d9d700 1 -- 172.21.2.102:0/4001228778 learned_addr learned my addr 172.21.2.102:0/4001228778 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:23:26.594 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:26.591 7f1fc6d9d700 1 -- 172.21.2.102:0/4001228778 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1fc80b8410 msgr2=0x7f1fc80b8810 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:23:26.594 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:26.591 7f1fc6d9d700 1 --2- 172.21.2.102:0/4001228778 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1fc80b8410 0x7f1fc80b8810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:26.594 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:26.591 7f1fc6d9d700 1 -- 172.21.2.102:0/4001228778 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1fc80c1f80 msgr2=0x7f1fc80c2380 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:23:26.594 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:26.591 7f1fc6d9d700 1 --2- 172.21.2.102:0/4001228778 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1fc80c1f80 0x7f1fc80c2380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:26.594 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:26.591 7f1fc6d9d700 1 -- 172.21.2.102:0/4001228778 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fc80c29a0 con 0x7f1fc809b930 2021-02-10T11:23:26.594 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:26.592 7f1fc6d9d700 1 --2- 172.21.2.102:0/4001228778 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1fc809b930 0x7f1fc80b7c20 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 rx=0x7f1fb804c5a0 tx=0x7f1fb804c940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:26.594 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:26.592 7f1fb77fe700 1 -- 172.21.2.102:0/4001228778 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1fb804cd30 con 0x7f1fc809b930 2021-02-10T11:23:26.594 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:26.592 7f1fb77fe700 1 -- 172.21.2.102:0/4001228778 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fb8002030 con 0x7f1fc809b930 2021-02-10T11:23:26.594 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:26.592 7f1fb77fe700 1 -- 172.21.2.102:0/4001228778 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1fb8002490 con 0x7f1fc809b930 2021-02-10T11:23:26.594 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:26.592 7f1fcd0b5700 1 -- 172.21.2.102:0/4001228778 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fc80c29a0 con 0x7f1fc809b930 2021-02-10T11:23:26.594 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:26.592 7f1fcd0b5700 1 -- 172.21.2.102:0/4001228778 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fc80c2f30 con 0x7f1fc809b930 2021-02-10T11:23:26.595 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:26.593 7f1fb77fe700 1 -- 172.21.2.102:0/4001228778 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1fb8002210 con 0x7f1fc809b930 2021-02-10T11:23:26.595 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:26.593 7f1fb77fe700 1 --2- 172.21.2.102:0/4001228778 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1fac0186c0 0x7f1fac01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:26.595 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:26.593 7f1fb77fe700 1 -- 172.21.2.102:0/4001228778 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f1fb80765f0 con 0x7f1fc809b930 2021-02-10T11:23:26.595 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:26.593 7f1fc659c700 1 --2- 172.21.2.102:0/4001228778 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1fac0186c0 0x7f1fac01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:26.595 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:26.593 7f1fc659c700 1 --2- 172.21.2.102:0/4001228778 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1fac0186c0 0x7f1fac01abc0 crc :-1 s=READY pgs=577 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:26.621 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:26.623 7f1fcd0b5700 1 -- 172.21.2.102:0/4001228778 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fc8105b30 con 0x7f1fc809b930 2021-02-10T11:23:26.623 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:26.626 7f1fb77fe700 1 -- 172.21.2.102:0/4001228778 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f1fb8074da0 con 0x7f1fc809b930 2021-02-10T11:23:26.750 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:26.752 7f1fcd0b5700 1 -- 172.21.2.102:0/4001228778 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7f1fc80f0b50 con 0x7f1fc809b930 2021-02-10T11:23:26.750 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:26.752 7f1fb77fe700 1 -- 172.21.2.102:0/4001228778 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v48) v1 ==== 104+0+10 (secure 0 0 0) 0x7f1fb80021f0 con 0x7f1fc809b930 2021-02-10T11:23:26.751 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:23:26.752 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:26.754 7f1fcd0b5700 1 -- 172.21.2.102:0/4001228778 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1fac0186c0 msgr2=0x7f1fac01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:26.752 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:26.754 7f1fcd0b5700 1 --2- 172.21.2.102:0/4001228778 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1fac0186c0 0x7f1fac01abc0 crc :-1 s=READY pgs=577 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:26.752 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:26.754 7f1fcd0b5700 1 -- 172.21.2.102:0/4001228778 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1fc809b930 msgr2=0x7f1fc80b7c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:26.752 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:26.754 7f1fcd0b5700 1 --2- 172.21.2.102:0/4001228778 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1fc809b930 0x7f1fc80b7c20 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 rx=0x7f1fb804c5a0 tx=0x7f1fb804c940).stop 2021-02-10T11:23:26.753 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:26.755 7f1fcd0b5700 1 -- 172.21.2.102:0/4001228778 shutdown_connections 2021-02-10T11:23:26.753 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:26.755 7f1fcd0b5700 1 --2- 172.21.2.102:0/4001228778 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1fac0186c0 0x7f1fac01abc0 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:26.753 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:26.755 7f1fcd0b5700 1 --2- 172.21.2.102:0/4001228778 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1fc80c1f80 0x7f1fc80c2380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:26.753 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:26.755 7f1fcd0b5700 1 --2- 172.21.2.102:0/4001228778 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1fc80b8410 0x7f1fc80b8810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:26.753 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:26.755 7f1fcd0b5700 1 --2- 172.21.2.102:0/4001228778 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1fc809b930 0x7f1fc80b7c20 unknown :-1 s=CLOSED pgs=1210 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:26.753 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:26.755 7f1fcd0b5700 1 -- 172.21.2.102:0/4001228778 >> 172.21.2.102:0/4001228778 conn(0x7f1fc8094e10 msgr2=0x7f1fc80a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:26.753 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:26.755 7f1fcd0b5700 1 -- 172.21.2.102:0/4001228778 shutdown_connections 2021-02-10T11:23:26.753 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:26.755 7f1fcd0b5700 1 -- 172.21.2.102:0/4001228778 wait complete. 2021-02-10T11:23:26.761 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-02-10T11:23:27.189 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:27.175 7f6190d3f700 1 Processor -- start 2021-02-10T11:23:27.189 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:27.175 7f6190d3f700 1 -- start start 2021-02-10T11:23:27.189 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:27.175 7f6190d3f700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f618c1b0870 0x7f618c0f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:27.189 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:27.175 7f6190d3f700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f618c1b0c70 con 0x7f618c0efe00 2021-02-10T11:23:27.190 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:27.175 7f6190d3f700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f618c0f0b50 con 0x7f618c1b0470 2021-02-10T11:23:27.190 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:27.175 7f6190d3f700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f618c0e5ea0 con 0x7f618c1b0870 2021-02-10T11:23:27.190 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:27.175 7f618b314700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f618c0efe00 0x7f618c0f0200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:48610/0 (socket says 172.21.2.102:48610) 2021-02-10T11:23:27.190 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:27.175 7f618b314700 1 -- 172.21.2.102:0/3559107330 learned_addr learned my addr 172.21.2.102:0/3559107330 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:23:27.190 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:27.175 7f618a312700 1 --2- 172.21.2.102:0/3559107330 >> v2:172.21.2.102:3300/0 conn(0x7f618c1b0870 0x7f618c0f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:27.190 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:27.176 7f6189b11700 1 -- 172.21.2.102:0/3559107330 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2012605805 0 0) 0x7f6184000d30 con 0x7f618c0efe00 2021-02-10T11:23:27.190 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:27.176 7f618a312700 1 -- 172.21.2.102:0/3559107330 >> v1:172.21.2.113:6789/0 conn(0x7f618c1b0470 legacy=0x7f618c0ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:27.190 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:27.176 7f618a312700 1 -- 172.21.2.102:0/3559107330 >> v1:172.21.2.102:6789/0 conn(0x7f618c0efe00 legacy=0x7f618c0f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:27.190 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:27.176 7f618a312700 1 -- 172.21.2.102:0/3559107330 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f618c0e52e0 con 0x7f618c1b0870 2021-02-10T11:23:27.190 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:27.176 7f618a312700 1 --2- 172.21.2.102:0/3559107330 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f618c1b0870 0x7f618c0f20e0 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 rx=0x7f6174006650 tx=0x7f6174009a80).ready entity=mon.0 client_cookie=cc0fc7328f81cde7 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:27.190 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:27.176 7f6189b11700 1 -- 172.21.2.102:0/3559107330 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f61740009b0 con 0x7f618c1b0870 2021-02-10T11:23:27.190 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:27.176 7f6189b11700 1 -- 172.21.2.102:0/3559107330 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f617400c0c0 con 0x7f618c1b0870 2021-02-10T11:23:27.190 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:27.176 7f6189b11700 1 -- 172.21.2.102:0/3559107330 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f617400c520 con 0x7f618c1b0870 2021-02-10T11:23:27.191 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:27.176 7f6190d3f700 1 -- 172.21.2.102:0/3559107330 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f618c1b0870 msgr2=0x7f618c0f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:27.191 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:27.176 7f6190d3f700 1 --2- 172.21.2.102:0/3559107330 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f618c1b0870 0x7f618c0f20e0 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 rx=0x7f6174006650 tx=0x7f6174009a80).stop 2021-02-10T11:23:27.191 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:27.177 7f6190d3f700 1 -- 172.21.2.102:0/3559107330 shutdown_connections 2021-02-10T11:23:27.191 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:27.177 7f6190d3f700 1 --2- 172.21.2.102:0/3559107330 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f618c1b0870 0x7f618c0f20e0 unknown :-1 s=CLOSED pgs=1212 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:27.191 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:27.177 7f6190d3f700 1 -- 172.21.2.102:0/3559107330 >> 172.21.2.102:0/3559107330 conn(0x7f618c0ce3e0 msgr2=0x7f618c0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:27.191 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:27.177 7f6190d3f700 1 -- 172.21.2.102:0/3559107330 shutdown_connections 2021-02-10T11:23:27.191 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:27.177 7f6190d3f700 1 -- 172.21.2.102:0/3559107330 wait complete. 2021-02-10T11:23:27.193 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:27.192 7f6190d3f700 1 Processor -- start 2021-02-10T11:23:27.193 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:27.192 7f6190d3f700 1 -- start start 2021-02-10T11:23:27.193 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:27.192 7f6190d3f700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f618c0a1dd0 0x7f618c190920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:27.193 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:27.192 7f6190d3f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f618c08fd30 0x7f618c0effc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:27.193 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:27.192 7f6190d3f700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f618c0f0e90 0x7f618c0f1290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:27.193 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:27.192 7f6190d3f700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f618c0f06f0 con 0x7f618c08fd30 2021-02-10T11:23:27.193 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:27.192 7f6190d3f700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f618c090160 con 0x7f618c0f0e90 2021-02-10T11:23:27.193 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:27.192 7f6190d3f700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f618c0f18b0 con 0x7f618c0a1dd0 2021-02-10T11:23:27.193 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:27.192 7f618a312700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f618c08fd30 0x7f618c0effc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:27.193 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:27.193 7f618a312700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f618c08fd30 0x7f618c0effc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:56392/0 (socket says 172.21.2.102:56392) 2021-02-10T11:23:27.193 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:27.193 7f618a312700 1 -- 172.21.2.102:0/1024939446 learned_addr learned my addr 172.21.2.102:0/1024939446 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:23:27.193 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:27.193 7f618b314700 1 --2- 172.21.2.102:0/1024939446 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f618c0f0e90 0x7f618c0f1290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:27.194 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:27.193 7f618ab13700 1 --2- 172.21.2.102:0/1024939446 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f618c0a1dd0 0x7f618c190920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:27.194 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:27.193 7f618a312700 1 -- 172.21.2.102:0/1024939446 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f618c0a1dd0 msgr2=0x7f618c190920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:27.194 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:27.193 7f618a312700 1 --2- 172.21.2.102:0/1024939446 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f618c0a1dd0 0x7f618c190920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:27.194 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:27.193 7f618a312700 1 -- 172.21.2.102:0/1024939446 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f618c0f0e90 msgr2=0x7f618c0f1290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:27.194 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:27.193 7f618a312700 1 --2- 172.21.2.102:0/1024939446 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f618c0f0e90 0x7f618c0f1290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:27.194 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:27.193 7f618a312700 1 -- 172.21.2.102:0/1024939446 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f618c0f1ad0 con 0x7f618c08fd30 2021-02-10T11:23:27.194 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:27.193 7f618a312700 1 --2- 172.21.2.102:0/1024939446 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f618c08fd30 0x7f618c0effc0 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 rx=0x7f6174009630 tx=0x7f617400b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:27.194 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:27.193 7f61837fe700 1 -- 172.21.2.102:0/1024939446 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f61740093a0 con 0x7f618c08fd30 2021-02-10T11:23:27.194 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:27.193 7f61837fe700 1 -- 172.21.2.102:0/1024939446 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f617400bd70 con 0x7f618c08fd30 2021-02-10T11:23:27.194 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:27.193 7f6190d3f700 1 -- 172.21.2.102:0/1024939446 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f618c0f1ad0 con 0x7f618c08fd30 2021-02-10T11:23:27.194 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:27.193 7f6190d3f700 1 -- 172.21.2.102:0/1024939446 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f618c0f1ad0 con 0x7f618c08fd30 2021-02-10T11:23:27.194 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:27.194 7f61837fe700 1 -- 172.21.2.102:0/1024939446 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f61740054d0 con 0x7f618c08fd30 2021-02-10T11:23:27.194 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:27.194 7f61837fe700 1 -- 172.21.2.102:0/1024939446 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f61740052e0 con 0x7f618c08fd30 2021-02-10T11:23:27.195 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:27.194 7f61837fe700 1 --2- 172.21.2.102:0/1024939446 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6170018730 0x7f617001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:27.195 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:27.194 7f61837fe700 1 -- 172.21.2.102:0/1024939446 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f61740327d0 con 0x7f618c08fd30 2021-02-10T11:23:27.195 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:27.194 7f618ab13700 1 --2- 172.21.2.102:0/1024939446 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6170018730 0x7f617001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:27.195 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:27.195 7f618ab13700 1 --2- 172.21.2.102:0/1024939446 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6170018730 0x7f617001ac10 crc :-1 s=READY pgs=578 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:27.223 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:27.225 7f6190d3f700 1 -- 172.21.2.102:0/1024939446 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f618c0e52e0 con 0x7f618c08fd30 2021-02-10T11:23:27.225 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:27.228 7f61837fe700 1 -- 172.21.2.102:0/1024939446 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f617401f510 con 0x7f618c08fd30 2021-02-10T11:23:27.353 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:27.355 7f6190d3f700 1 -- 172.21.2.102:0/1024939446 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7f618c0e52e0 con 0x7f618c08fd30 2021-02-10T11:23:27.353 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:27.355 7f61837fe700 1 -- 172.21.2.102:0/1024939446 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v48) v1 ==== 100+0+10 (secure 0 0 0) 0x7f617400bd70 con 0x7f618c08fd30 2021-02-10T11:23:27.354 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:23:27.355 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:27.357 7f6190d3f700 1 -- 172.21.2.102:0/1024939446 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6170018730 msgr2=0x7f617001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:27.355 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:27.357 7f6190d3f700 1 --2- 172.21.2.102:0/1024939446 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6170018730 0x7f617001ac10 crc :-1 s=READY pgs=578 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:27.356 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:27.357 7f6190d3f700 1 -- 172.21.2.102:0/1024939446 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f618c08fd30 msgr2=0x7f618c0effc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:27.356 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:27.357 7f6190d3f700 1 --2- 172.21.2.102:0/1024939446 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f618c08fd30 0x7f618c0effc0 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 rx=0x7f6174009630 tx=0x7f617400b980).stop 2021-02-10T11:23:27.356 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:27.357 7f6190d3f700 1 -- 172.21.2.102:0/1024939446 shutdown_connections 2021-02-10T11:23:27.356 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:27.358 7f6190d3f700 1 --2- 172.21.2.102:0/1024939446 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6170018730 0x7f617001ac10 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:27.356 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:27.358 7f6190d3f700 1 --2- 172.21.2.102:0/1024939446 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f618c0f0e90 0x7f618c0f1290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:27.356 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:27.358 7f6190d3f700 1 --2- 172.21.2.102:0/1024939446 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f618c08fd30 0x7f618c0effc0 unknown :-1 s=CLOSED pgs=1213 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:27.356 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:27.358 7f6190d3f700 1 --2- 172.21.2.102:0/1024939446 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f618c0a1dd0 0x7f618c190920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:27.356 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:27.358 7f6190d3f700 1 -- 172.21.2.102:0/1024939446 >> 172.21.2.102:0/1024939446 conn(0x7f618c0958b0 msgr2=0x7f618c0bf040 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:27.356 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:27.358 7f6190d3f700 1 -- 172.21.2.102:0/1024939446 shutdown_connections 2021-02-10T11:23:27.356 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:27.358 7f6190d3f700 1 -- 172.21.2.102:0/1024939446 wait complete. 2021-02-10T11:23:27.364 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-02-10T11:23:27.364 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:23:27.792 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:27.776 7f467c3be700 1 Processor -- start 2021-02-10T11:23:27.792 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:27.776 7f467c3be700 1 -- start start 2021-02-10T11:23:27.792 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:27.776 7f467c3be700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f46740e5470 0x7f467407e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:27.792 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:27.776 7f467c3be700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f467407ed80 con 0x7f467407f300 2021-02-10T11:23:27.792 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:27.776 7f467c3be700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46741b4010 con 0x7f46740e5070 2021-02-10T11:23:27.792 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:27.776 7f467c3be700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46740d1790 con 0x7f46740e5470 2021-02-10T11:23:27.792 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:27.777 7f467a9db700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f467407f300 0x7f467407f700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:48628/0 (socket says 172.21.2.102:48628) 2021-02-10T11:23:27.793 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:27.777 7f467a9db700 1 -- 172.21.2.102:0/938791533 learned_addr learned my addr 172.21.2.102:0/938791533 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:23:27.793 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:27.777 7f46791d8700 1 -- 172.21.2.102:0/938791533 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 873383152 0 0) 0x7f466c001500 con 0x7f467407f300 2021-02-10T11:23:27.793 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:27.777 7f46791d8700 1 -- 172.21.2.102:0/938791533 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4660002fe0 con 0x7f467407f300 2021-02-10T11:23:27.793 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:27.777 7f46799d9700 1 --2- 172.21.2.102:0/938791533 >> v2:172.21.2.113:3300/0 conn(0x7f46740e5470 0x7f467407e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:27.793 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:27.777 7f46791d8700 1 -- 172.21.2.102:0/938791533 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3042844224 0 0) 0x7f466c001460 con 0x7f467407f300 2021-02-10T11:23:27.793 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:27.777 7f46791d8700 1 -- 172.21.2.102:0/938791533 >> v2:172.21.2.113:3300/0 conn(0x7f46740e5470 msgr2=0x7f467407e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:27.793 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:27.777 7f46791d8700 1 --2- 172.21.2.102:0/938791533 >> v2:172.21.2.113:3300/0 conn(0x7f46740e5470 0x7f467407e6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:27.793 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:27.777 7f46791d8700 1 -- 172.21.2.102:0/938791533 >> v1:172.21.2.113:6789/0 conn(0x7f46740e5070 legacy=0x7f46741b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:27.793 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:27.777 7f46791d8700 1 -- 172.21.2.102:0/938791533 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46740d19b0 con 0x7f467407f300 2021-02-10T11:23:27.794 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:27.777 7f46791d8700 1 -- 172.21.2.102:0/938791533 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7f466c001980 con 0x7f467407f300 2021-02-10T11:23:27.794 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:27.777 7f46791d8700 1 -- 172.21.2.102:0/938791533 >> v1:172.21.2.102:6789/0 conn(0x7f467407f300 legacy=0x7f467407f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:27.794 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:27.777 7f46791d8700 1 --2- 172.21.2.102:0/938791533 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4660003ac0 0x7f4660005ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:27.794 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:27.777 7f46791d8700 1 --2- 172.21.2.102:0/938791533 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4660006770 0x7f4660008c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:27.794 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:27.777 7f46791d8700 1 --2- 172.21.2.102:0/938791533 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4660009590 0x7f466000bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:27.794 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:27.777 7f46791d8700 1 -- 172.21.2.102:0/938791533 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f466000c300 con 0x7f4660006770 2021-02-10T11:23:27.794 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:27.777 7f46791d8700 1 -- 172.21.2.102:0/938791533 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f466000c520 con 0x7f4660009590 2021-02-10T11:23:27.794 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:27.777 7f46791d8700 1 -- 172.21.2.102:0/938791533 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f466000c740 con 0x7f4660003ac0 2021-02-10T11:23:27.794 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:27.778 7f46799d9700 1 --2- 172.21.2.102:0/938791533 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4660006770 0x7f4660008c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:27.794 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:27.778 7f467a9db700 1 --2- 172.21.2.102:0/938791533 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4660009590 0x7f466000bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:27.794 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:27.778 7f467a1da700 1 --2- 172.21.2.102:0/938791533 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4660003ac0 0x7f4660005ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:27.795 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:27.778 7f46799d9700 1 -- 172.21.2.102:0/938791533 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4660003ac0 msgr2=0x7f4660005ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:27.795 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:27.778 7f46799d9700 1 --2- 172.21.2.102:0/938791533 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4660003ac0 0x7f4660005ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:27.795 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:27.778 7f46799d9700 1 -- 172.21.2.102:0/938791533 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4660009590 msgr2=0x7f466000bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:27.795 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:27.778 7f46799d9700 1 --2- 172.21.2.102:0/938791533 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4660009590 0x7f466000bab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:27.795 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:27.778 7f467a9db700 1 -- 172.21.2.102:0/938791533 reap_dead start 2021-02-10T11:23:27.795 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:27.778 7f46799d9700 1 -- 172.21.2.102:0/938791533 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46600509b0 con 0x7f4660006770 2021-02-10T11:23:27.795 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:27.778 7f46799d9700 1 --2- 172.21.2.102:0/938791533 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4660006770 0x7f4660008c90 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 rx=0x7f466800c470 tx=0x7f4668006c20).ready entity=mon.0 client_cookie=e2007c2213ad01c9 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:27.795 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:27.778 7f46791d8700 1 -- 172.21.2.102:0/938791533 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f46680090d0 con 0x7f4660006770 2021-02-10T11:23:27.795 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:27.778 7f46791d8700 1 -- 172.21.2.102:0/938791533 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4668009450 con 0x7f4660006770 2021-02-10T11:23:27.795 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:27.778 7f46791d8700 1 -- 172.21.2.102:0/938791533 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f46680098b0 con 0x7f4660006770 2021-02-10T11:23:27.795 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:27.778 7f467c3be700 1 -- 172.21.2.102:0/938791533 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4660006770 msgr2=0x7f4660008c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:27.795 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:27.778 7f467c3be700 1 --2- 172.21.2.102:0/938791533 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4660006770 0x7f4660008c90 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 rx=0x7f466800c470 tx=0x7f4668006c20).stop 2021-02-10T11:23:27.795 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:27.779 7f467c3be700 1 -- 172.21.2.102:0/938791533 shutdown_connections 2021-02-10T11:23:27.795 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:27.779 7f467c3be700 1 --2- 172.21.2.102:0/938791533 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4660006770 0x7f4660008c90 unknown :-1 s=CLOSED pgs=1215 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:27.796 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:27.779 7f467c3be700 1 -- 172.21.2.102:0/938791533 >> 172.21.2.102:0/938791533 conn(0x7f46740ce690 msgr2=0x7f46740e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:27.796 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:27.779 7f467c3be700 1 -- 172.21.2.102:0/938791533 shutdown_connections 2021-02-10T11:23:27.796 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:27.779 7f467c3be700 1 -- 172.21.2.102:0/938791533 wait complete. 2021-02-10T11:23:27.796 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:27.794 7f467c3be700 1 Processor -- start 2021-02-10T11:23:27.796 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:27.794 7f467c3be700 1 -- start start 2021-02-10T11:23:27.797 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:27.794 7f467c3be700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4674194000 0x7f467419f680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:27.797 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:27.794 7f467c3be700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f467419fde0 0x7f467409bed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:27.797 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:27.794 7f467c3be700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f46741a01e0 0x7f46740895c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:27.797 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:27.794 7f467c3be700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4674194400 con 0x7f46741a01e0 2021-02-10T11:23:27.797 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:27.794 7f467c3be700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f467409c4f0 con 0x7f4674194000 2021-02-10T11:23:27.798 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:27.794 7f467c3be700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4674089be0 con 0x7f467419fde0 2021-02-10T11:23:27.798 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:27.795 7f467a9db700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f46741a01e0 0x7f46740895c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:27.798 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:27.795 7f467a9db700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f46741a01e0 0x7f46740895c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:56416/0 (socket says 172.21.2.102:56416) 2021-02-10T11:23:27.798 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:27.795 7f467a9db700 1 -- 172.21.2.102:0/3075957514 learned_addr learned my addr 172.21.2.102:0/3075957514 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:23:27.798 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:27.795 7f467a9db700 1 -- 172.21.2.102:0/3075957514 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f467419fde0 msgr2=0x7f467409bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:27.798 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:27.795 7f467a9db700 1 --2- 172.21.2.102:0/3075957514 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f467419fde0 0x7f467409bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:27.798 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:27.795 7f467a9db700 1 -- 172.21.2.102:0/3075957514 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4674194000 msgr2=0x7f467419f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:27.798 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:27.795 7f467a9db700 1 --2- 172.21.2.102:0/3075957514 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4674194000 0x7f467419f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:27.798 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:27.795 7f467a9db700 1 -- 172.21.2.102:0/3075957514 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4674089e00 con 0x7f46741a01e0 2021-02-10T11:23:27.798 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:27.795 7f467a9db700 1 --2- 172.21.2.102:0/3075957514 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f46741a01e0 0x7f46740895c0 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 rx=0x7f466c0017d0 tx=0x7f466c00ac80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:27.799 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:27.795 7f4666ffd700 1 -- 172.21.2.102:0/3075957514 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f466c001ed0 con 0x7f46741a01e0 2021-02-10T11:23:27.799 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:27.795 7f4666ffd700 1 -- 172.21.2.102:0/3075957514 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f466c0070d0 con 0x7f46741a01e0 2021-02-10T11:23:27.799 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:27.795 7f4666ffd700 1 -- 172.21.2.102:0/3075957514 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f466c004930 con 0x7f46741a01e0 2021-02-10T11:23:27.799 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:27.795 7f467c3be700 1 -- 172.21.2.102:0/3075957514 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4674089e00 con 0x7f46741a01e0 2021-02-10T11:23:27.799 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:27.795 7f467c3be700 1 -- 172.21.2.102:0/3075957514 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4674089e00 con 0x7f46741a01e0 2021-02-10T11:23:27.799 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:27.796 7f4666ffd700 1 -- 172.21.2.102:0/3075957514 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f466c0072b0 con 0x7f46741a01e0 2021-02-10T11:23:27.799 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:27.796 7f4666ffd700 1 --2- 172.21.2.102:0/3075957514 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f465c0186c0 0x7f465c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:27.799 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:27.796 7f4666ffd700 1 -- 172.21.2.102:0/3075957514 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f466c032b80 con 0x7f46741a01e0 2021-02-10T11:23:27.799 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:27.796 7f467a1da700 1 --2- 172.21.2.102:0/3075957514 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f465c0186c0 0x7f465c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:27.799 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:27.797 7f467a1da700 1 --2- 172.21.2.102:0/3075957514 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f465c0186c0 0x7f465c01abc0 crc :-1 s=READY pgs=579 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:27.825 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:27.826 7f467c3be700 1 -- 172.21.2.102:0/3075957514 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f467409c710 con 0x7f46741a01e0 2021-02-10T11:23:27.827 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:27.829 7f4666ffd700 1 -- 172.21.2.102:0/3075957514 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f466c001ed0 con 0x7f46741a01e0 2021-02-10T11:23:27.957 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:27.959 7f467c3be700 1 -- 172.21.2.102:0/3075957514 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4674089090 con 0x7f46741a01e0 2021-02-10T11:23:27.958 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:27.960 7f4666ffd700 1 -- 172.21.2.102:0/3075957514 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+2099 (secure 0 0 0) 0x7f466c0070d0 con 0x7f46741a01e0 2021-02-10T11:23:27.958 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:23:27.959 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":30,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7557,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.102:6835/1941781958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6834","nonce":1941781958},{"type":"v1","addr":"172.21.2.102:6835","nonce":1941781958}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":26}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:23:18.002680","modified":"2021-02-10 11:23:20.009191","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5606},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5606":{"gid":5606,"name":"b","rank":0,"incarnation":27,"state":"up:active","state_seq":3,"addr":"172.21.2.113:6833/1872558991","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6832","nonce":1872558991},{"type":"v1","addr":"172.21.2.113:6833","nonce":1872558991}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":4}]} 2021-02-10T11:23:27.959 INFO:teuthology.orchestra.run.gibba002.stderr:dumped fsmap epoch 30 2021-02-10T11:23:27.960 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:27.962 7f467c3be700 1 -- 172.21.2.102:0/3075957514 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f465c0186c0 msgr2=0x7f465c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:27.960 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:27.962 7f467c3be700 1 --2- 172.21.2.102:0/3075957514 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f465c0186c0 0x7f465c01abc0 crc :-1 s=READY pgs=579 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:27.960 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:27.962 7f467c3be700 1 -- 172.21.2.102:0/3075957514 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f46741a01e0 msgr2=0x7f46740895c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:27.960 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:27.962 7f467c3be700 1 --2- 172.21.2.102:0/3075957514 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f46741a01e0 0x7f46740895c0 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 rx=0x7f466c0017d0 tx=0x7f466c00ac80).stop 2021-02-10T11:23:27.960 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:27.962 7f467c3be700 1 -- 172.21.2.102:0/3075957514 shutdown_connections 2021-02-10T11:23:27.960 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:27.962 7f467c3be700 1 --2- 172.21.2.102:0/3075957514 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f46741a01e0 0x7f46740895c0 unknown :-1 s=CLOSED pgs=1216 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:27.960 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:27.962 7f467c3be700 1 --2- 172.21.2.102:0/3075957514 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f467419fde0 0x7f467409bed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:27.960 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:27.962 7f467c3be700 1 --2- 172.21.2.102:0/3075957514 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f465c0186c0 0x7f465c01abc0 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:27.960 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:27.962 7f467c3be700 1 --2- 172.21.2.102:0/3075957514 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4674194000 0x7f467419f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:27.960 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:27.962 7f467c3be700 1 -- 172.21.2.102:0/3075957514 >> 172.21.2.102:0/3075957514 conn(0x7f4674092c20 msgr2=0x7f4674195070 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:27.961 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:27.962 7f467c3be700 1 -- 172.21.2.102:0/3075957514 shutdown_connections 2021-02-10T11:23:27.961 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:27.962 7f467c3be700 1 -- 172.21.2.102:0/3075957514 wait complete. 2021-02-10T11:23:28.170 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 30, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:23:18.002680', 'modified': '2021-02-10 11:23:20.009191', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5606}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5606': {'gid': 5606, 'name': 'b', 'rank': 0, 'incarnation': 27, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.113:6833/1872558991', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.113:6832', 'nonce': 1872558991}, {'type': 'v1', 'addr': '172.21.2.113:6833', 'nonce': 1872558991}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [9], 'metadata_pool': 8, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:23:28.170 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:23:28.170 DEBUG:teuthology.orchestra.run.gibba013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:23:28.569 INFO:teuthology.orchestra.run.gibba013.stdout:{ 2021-02-10T11:23:28.569 INFO:teuthology.orchestra.run.gibba013.stdout: "cluster_fsid": "68db30ea-dbce-4aef-a6df-f01c6fea0513", 2021-02-10T11:23:28.569 INFO:teuthology.orchestra.run.gibba013.stdout: "whoami": 0, 2021-02-10T11:23:28.569 INFO:teuthology.orchestra.run.gibba013.stdout: "id": 5606, 2021-02-10T11:23:28.569 INFO:teuthology.orchestra.run.gibba013.stdout: "want_state": "up:active", 2021-02-10T11:23:28.570 INFO:teuthology.orchestra.run.gibba013.stdout: "state": "up:active", 2021-02-10T11:23:28.570 INFO:teuthology.orchestra.run.gibba013.stdout: "rank_uptime": 10.567150198, 2021-02-10T11:23:28.570 INFO:teuthology.orchestra.run.gibba013.stdout: "mdsmap_epoch": 30, 2021-02-10T11:23:28.570 INFO:teuthology.orchestra.run.gibba013.stdout: "osdmap_epoch": 48, 2021-02-10T11:23:28.570 INFO:teuthology.orchestra.run.gibba013.stdout: "osdmap_epoch_barrier": 48, 2021-02-10T11:23:28.570 INFO:teuthology.orchestra.run.gibba013.stdout: "uptime": 16.208460795000001 2021-02-10T11:23:28.570 INFO:teuthology.orchestra.run.gibba013.stdout:} 2021-02-10T11:23:28.573 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "68db30ea-dbce-4aef-a6df-f01c6fea0513", "whoami": 0, "id": 5606, "want_state": "up:active", "state": "up:active", "rank_uptime": 10.567150198, "mdsmap_epoch": 30, "osdmap_epoch": 48, "osdmap_epoch_barrier": 48, "uptime": 16.208460795000001 } 2021-02-10T11:23:28.574 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2021-02-10T11:23:28.574 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2021-02-10T11:23:28.574 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@gibba002.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2021-02-10T11:23:28.574 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:23:28.574 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec mkdir -p /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:23:28.915 DEBUG:teuthology.orchestra.run.gibba002:> sudo modprobe fuse 2021-02-10T11:23:29.261 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:23:29.262 DEBUG:teuthology.orchestra.run.gibba002:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:23:29.605 INFO:teuthology.orchestra.run.gibba002.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:23:29.606 INFO:teuthology.orchestra.run.gibba002.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:23:29.606 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:23:29.607 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:23:29.607 DEBUG:teuthology.orchestra.run.gibba002:> ls /sys/fs/fuse/connections 2021-02-10T11:23:29.941 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2021-02-10T11:23:29.941 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:23:29.944 DEBUG:teuthology.orchestra.run.gibba002:> sudo modprobe fuse 2021-02-10T11:23:30.287 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:23:30.287 DEBUG:teuthology.orchestra.run.gibba002:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:23:30.353 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba002.stderr:ceph-fuse[2021-02-10 11:23:30.355 7f2162cfef80 -1 init, newargv = 0x561047f09100 newargc=9149289 2021-02-10T11:23:30.353 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba002.stderr:]: starting ceph client 2021-02-10T11:23:30.358 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba002.stderr:ceph-fuse[149289]: starting fuse 2021-02-10T11:23:30.630 INFO:teuthology.orchestra.run.gibba002.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:23:30.630 INFO:teuthology.orchestra.run.gibba002.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:23:30.630 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:23:30.631 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:23:30.631 DEBUG:teuthology.orchestra.run.gibba002:> ls /sys/fs/fuse/connections 2021-02-10T11:23:30.966 INFO:teuthology.orchestra.run.gibba002.stdout:43 2021-02-10T11:23:30.966 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2021-02-10T11:23:30.967 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:23:30.967 DEBUG:teuthology.orchestra.run.gibba002:> sudo python3 -c ' 2021-02-10T11:23:30.967 DEBUG:teuthology.orchestra.run.gibba002:> import glob 2021-02-10T11:23:30.967 DEBUG:teuthology.orchestra.run.gibba002:> import re 2021-02-10T11:23:30.967 DEBUG:teuthology.orchestra.run.gibba002:> import os 2021-02-10T11:23:30.967 DEBUG:teuthology.orchestra.run.gibba002:> import subprocess 2021-02-10T11:23:30.967 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:23:30.967 DEBUG:teuthology.orchestra.run.gibba002:> def find_socket(client_name): 2021-02-10T11:23:30.967 DEBUG:teuthology.orchestra.run.gibba002:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2021-02-10T11:23:30.967 DEBUG:teuthology.orchestra.run.gibba002:> files = glob.glob(asok_path) 2021-02-10T11:23:30.967 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:23:30.967 DEBUG:teuthology.orchestra.run.gibba002:> # Given a non-glob path, it better be there 2021-02-10T11:23:30.968 DEBUG:teuthology.orchestra.run.gibba002:> if "*" not in asok_path: 2021-02-10T11:23:30.968 DEBUG:teuthology.orchestra.run.gibba002:> assert(len(files) == 1) 2021-02-10T11:23:30.968 DEBUG:teuthology.orchestra.run.gibba002:> return files[0] 2021-02-10T11:23:30.968 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:23:30.968 DEBUG:teuthology.orchestra.run.gibba002:> for f in files: 2021-02-10T11:23:30.968 DEBUG:teuthology.orchestra.run.gibba002:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2021-02-10T11:23:30.968 DEBUG:teuthology.orchestra.run.gibba002:> if os.path.exists("/proc/{0}".format(pid)): 2021-02-10T11:23:30.968 DEBUG:teuthology.orchestra.run.gibba002:> return f 2021-02-10T11:23:30.968 DEBUG:teuthology.orchestra.run.gibba002:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2021-02-10T11:23:30.968 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:23:30.968 DEBUG:teuthology.orchestra.run.gibba002:> print(find_socket("client.0")) 2021-02-10T11:23:30.968 DEBUG:teuthology.orchestra.run.gibba002:> ' 2021-02-10T11:23:31.330 INFO:teuthology.orchestra.run.gibba002.stdout:/var/run/ceph/ceph-client.0.149289.asok 2021-02-10T11:23:31.333 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.149289.asok 2021-02-10T11:23:31.333 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:23:31.333 DEBUG:teuthology.orchestra.run.gibba002:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.149289.asok status 2021-02-10T11:23:31.723 INFO:teuthology.orchestra.run.gibba002.stdout:{ 2021-02-10T11:23:31.723 INFO:teuthology.orchestra.run.gibba002.stdout: "metadata": { 2021-02-10T11:23:31.723 INFO:teuthology.orchestra.run.gibba002.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:23:31.723 INFO:teuthology.orchestra.run.gibba002.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:23:31.724 INFO:teuthology.orchestra.run.gibba002.stdout: "entity_id": "0", 2021-02-10T11:23:31.724 INFO:teuthology.orchestra.run.gibba002.stdout: "hostname": "gibba002", 2021-02-10T11:23:31.724 INFO:teuthology.orchestra.run.gibba002.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2021-02-10T11:23:31.724 INFO:teuthology.orchestra.run.gibba002.stdout: "pid": "149289", 2021-02-10T11:23:31.724 INFO:teuthology.orchestra.run.gibba002.stdout: "root": "/" 2021-02-10T11:23:31.724 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:31.724 INFO:teuthology.orchestra.run.gibba002.stdout: "dentry_count": 0, 2021-02-10T11:23:31.724 INFO:teuthology.orchestra.run.gibba002.stdout: "dentry_pinned_count": 0, 2021-02-10T11:23:31.724 INFO:teuthology.orchestra.run.gibba002.stdout: "id": 7707, 2021-02-10T11:23:31.724 INFO:teuthology.orchestra.run.gibba002.stdout: "inst": { 2021-02-10T11:23:31.724 INFO:teuthology.orchestra.run.gibba002.stdout: "name": { 2021-02-10T11:23:31.724 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "client", 2021-02-10T11:23:31.724 INFO:teuthology.orchestra.run.gibba002.stdout: "num": 7707 2021-02-10T11:23:31.724 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:31.725 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": { 2021-02-10T11:23:31.725 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:23:31.725 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:0", 2021-02-10T11:23:31.725 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 3759339251 2021-02-10T11:23:31.725 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:23:31.725 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:31.725 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": { 2021-02-10T11:23:31.725 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:23:31.725 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:0", 2021-02-10T11:23:31.725 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 3759339251 2021-02-10T11:23:31.725 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:31.726 INFO:teuthology.orchestra.run.gibba002.stdout: "inst_str": "client.7707 172.21.2.102:0/3759339251", 2021-02-10T11:23:31.726 INFO:teuthology.orchestra.run.gibba002.stdout: "addr_str": "172.21.2.102:0/3759339251", 2021-02-10T11:23:31.726 INFO:teuthology.orchestra.run.gibba002.stdout: "inode_count": 1, 2021-02-10T11:23:31.726 INFO:teuthology.orchestra.run.gibba002.stdout: "mds_epoch": 30, 2021-02-10T11:23:31.726 INFO:teuthology.orchestra.run.gibba002.stdout: "osd_epoch": 48, 2021-02-10T11:23:31.726 INFO:teuthology.orchestra.run.gibba002.stdout: "osd_epoch_barrier": 0, 2021-02-10T11:23:31.727 INFO:teuthology.orchestra.run.gibba002.stdout: "blacklisted": false 2021-02-10T11:23:31.727 INFO:teuthology.orchestra.run.gibba002.stdout:} 2021-02-10T11:23:31.728 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:23:31.728 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:23:31.729 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:23:32.063 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:23:32.064 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:23:32.064 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:23:32.064 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:23:32.410 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:23:32.839 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:32.823 7fe747cd8700 1 Processor -- start 2021-02-10T11:23:32.839 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:32.824 7fe747cd8700 1 -- start start 2021-02-10T11:23:32.839 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:32.824 7fe747cd8700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fe7400e5070 0x7fe7401b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:32.839 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:32.824 7fe747cd8700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe7401b3e40 0x7fe74007d690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:32.839 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:32.824 7fe747cd8700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fe7400e5470 0x7fe74007dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:32.839 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:32.824 7fe747cd8700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe74007ed80 con 0x7fe7400e5070 2021-02-10T11:23:32.839 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:32.824 7fe747cd8700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe74007e2d0 con 0x7fe7400e5470 2021-02-10T11:23:32.840 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:32.824 7fe747cd8700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe74007e4c0 con 0x7fe7401b3e40 2021-02-10T11:23:32.840 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:32.824 7fe745af4700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fe7400e5070 0x7fe7401b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:32.840 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:32.824 7fe745af4700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fe7400e5070 0x7fe7401b3820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:56434/0 (socket says 172.21.2.102:56434) 2021-02-10T11:23:32.840 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:32.824 7fe745af4700 1 -- 172.21.2.102:0/90142994 learned_addr learned my addr 172.21.2.102:0/90142994 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:23:32.840 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:32.825 7fe745af4700 1 -- 172.21.2.102:0/90142994 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe7401b3e40 msgr2=0x7fe74007d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:32.840 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:32.825 7fe745af4700 1 --2- 172.21.2.102:0/90142994 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe7401b3e40 0x7fe74007d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:32.840 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:32.825 7fe745af4700 1 -- 172.21.2.102:0/90142994 >> v2:172.21.2.113:3300/0 conn(0x7fe7400e5470 msgr2=0x7fe74007dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:32.840 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:32.825 7fe745af4700 1 --2- 172.21.2.102:0/90142994 >> v2:172.21.2.113:3300/0 conn(0x7fe7400e5470 0x7fe74007dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:32.840 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:32.825 7fe745af4700 1 -- 172.21.2.102:0/90142994 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe74007e6e0 con 0x7fe7400e5070 2021-02-10T11:23:32.840 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:32.825 7fe745af4700 1 --2- 172.21.2.102:0/90142994 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe7400e5070 0x7fe7401b3820 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 rx=0x7fe734006700 tx=0x7fe734009b80).ready entity=mon.0 client_cookie=57510c89b6d99e0e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:32.841 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:32.825 7fe744af2700 1 -- 172.21.2.102:0/90142994 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe73400c0c0 con 0x7fe7400e5070 2021-02-10T11:23:32.841 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:32.825 7fe744af2700 1 -- 172.21.2.102:0/90142994 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe73400c3c0 con 0x7fe7400e5070 2021-02-10T11:23:32.841 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:32.825 7fe744af2700 1 -- 172.21.2.102:0/90142994 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe73400c820 con 0x7fe7400e5070 2021-02-10T11:23:32.841 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:32.825 7fe747cd8700 1 -- 172.21.2.102:0/90142994 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe7400e5070 msgr2=0x7fe7401b3820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:32.841 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:32.825 7fe747cd8700 1 --2- 172.21.2.102:0/90142994 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe7400e5070 0x7fe7401b3820 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 rx=0x7fe734006700 tx=0x7fe734009b80).stop 2021-02-10T11:23:32.841 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:32.826 7fe747cd8700 1 -- 172.21.2.102:0/90142994 shutdown_connections 2021-02-10T11:23:32.841 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:32.826 7fe747cd8700 1 --2- 172.21.2.102:0/90142994 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe7401b3e40 0x7fe74007d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:32.841 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:32.826 7fe747cd8700 1 --2- 172.21.2.102:0/90142994 >> v2:172.21.2.113:3300/0 conn(0x7fe7400e5470 0x7fe74007dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:32.841 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:32.826 7fe747cd8700 1 --2- 172.21.2.102:0/90142994 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe7400e5070 0x7fe7401b3820 unknown :-1 s=CLOSED pgs=1219 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:32.841 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:32.826 7fe747cd8700 1 -- 172.21.2.102:0/90142994 >> 172.21.2.102:0/90142994 conn(0x7fe7400ce690 msgr2=0x7fe7400e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:32.841 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:32.826 7fe747cd8700 1 -- 172.21.2.102:0/90142994 shutdown_connections 2021-02-10T11:23:32.842 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:32.826 7fe747cd8700 1 -- 172.21.2.102:0/90142994 wait complete. 2021-02-10T11:23:32.842 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:32.841 7fe747cd8700 1 Processor -- start 2021-02-10T11:23:32.842 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:32.841 7fe747cd8700 1 -- start start 2021-02-10T11:23:32.844 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:32.841 7fe747cd8700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe740192960 0x7fe740192d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:32.844 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:32.841 7fe747cd8700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe7400ad110 0x7fe7400ad510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:32.844 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:32.841 7fe747cd8700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe7400adb30 0x7fe74008e460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:32.844 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:32.841 7fe747cd8700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7401a03e0 con 0x7fe7400ad110 2021-02-10T11:23:32.844 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:32.841 7fe747cd8700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe740193380 con 0x7fe7400adb30 2021-02-10T11:23:32.844 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:32.841 7fe747cd8700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe74008ea80 con 0x7fe740192960 2021-02-10T11:23:32.844 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:32.841 7fe7452f3700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe7400ad110 0x7fe7400ad510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:32.844 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:32.841 7fe7452f3700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe7400ad110 0x7fe7400ad510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:56442/0 (socket says 172.21.2.102:56442) 2021-02-10T11:23:32.844 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:32.841 7fe7452f3700 1 -- 172.21.2.102:0/194425896 learned_addr learned my addr 172.21.2.102:0/194425896 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:23:32.845 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:32.841 7fe745af4700 1 --2- 172.21.2.102:0/194425896 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe740192960 0x7fe740192d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:32.845 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:32.841 7fe7462f5700 1 --2- 172.21.2.102:0/194425896 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe7400adb30 0x7fe74008e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:32.845 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:32.842 7fe7452f3700 1 -- 172.21.2.102:0/194425896 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe740192960 msgr2=0x7fe740192d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:32.845 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:32.842 7fe7452f3700 1 --2- 172.21.2.102:0/194425896 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe740192960 0x7fe740192d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:32.845 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:32.842 7fe7452f3700 1 -- 172.21.2.102:0/194425896 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe7400adb30 msgr2=0x7fe74008e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:32.845 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:32.842 7fe7452f3700 1 --2- 172.21.2.102:0/194425896 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe7400adb30 0x7fe74008e460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:32.845 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:32.842 7fe7452f3700 1 -- 172.21.2.102:0/194425896 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe74008eca0 con 0x7fe7400ad110 2021-02-10T11:23:32.845 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:32.842 7fe7452f3700 1 --2- 172.21.2.102:0/194425896 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe7400ad110 0x7fe7400ad510 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 rx=0x7fe73c006f00 tx=0x7fe73c00b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:32.846 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:32.842 7fe7327fc700 1 -- 172.21.2.102:0/194425896 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe73c00bd30 con 0x7fe7400ad110 2021-02-10T11:23:32.846 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:32.842 7fe7327fc700 1 -- 172.21.2.102:0/194425896 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe73c000c90 con 0x7fe7400ad110 2021-02-10T11:23:32.846 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:32.842 7fe7327fc700 1 -- 172.21.2.102:0/194425896 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe73c0022a0 con 0x7fe7400ad110 2021-02-10T11:23:32.846 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:32.842 7fe747cd8700 1 -- 172.21.2.102:0/194425896 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe74008eca0 con 0x7fe7400ad110 2021-02-10T11:23:32.846 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:32.842 7fe747cd8700 1 -- 172.21.2.102:0/194425896 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe74008eca0 con 0x7fe7400ad110 2021-02-10T11:23:32.846 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:32.843 7fe7327fc700 1 -- 172.21.2.102:0/194425896 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fe73c005310 con 0x7fe7400ad110 2021-02-10T11:23:32.846 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:32.843 7fe7327fc700 1 --2- 172.21.2.102:0/194425896 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe7280186c0 0x7fe72801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:32.847 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:32.843 7fe7327fc700 1 -- 172.21.2.102:0/194425896 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fe73c032530 con 0x7fe7400ad110 2021-02-10T11:23:32.847 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:32.843 7fe745af4700 1 --2- 172.21.2.102:0/194425896 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe7280186c0 0x7fe72801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:32.847 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:32.843 7fe745af4700 1 --2- 172.21.2.102:0/194425896 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe7280186c0 0x7fe72801abc0 crc :-1 s=READY pgs=580 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:32.872 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:32.874 7fe747cd8700 1 -- 172.21.2.102:0/194425896 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe740107120 con 0x7fe7400ad110 2021-02-10T11:23:32.874 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:32.877 7fe7327fc700 1 -- 172.21.2.102:0/194425896 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fe73c006b90 con 0x7fe7400ad110 2021-02-10T11:23:32.998 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:33.000 7fe747cd8700 1 -- 172.21.2.102:0/194425896 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe74007ed80 con 0x7fe7400ad110 2021-02-10T11:23:32.999 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:33.001 7fe7327fc700 1 -- 172.21.2.102:0/194425896 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v48) v1 ==== 74+0+13993 (secure 0 0 0) 0x7fe73c005310 con 0x7fe7400ad110 2021-02-10T11:23:32.999 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:23:33.000 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":48,"fsid":"68db30ea-dbce-4aef-a6df-f01c6fea0513","created":"2021-02-10 11:03:06.331557","modified":"2021-02-10 11:23:19.005019","last_up_change":"2021-02-10 11:03:32.690562","last_in_change":"2021-02-10 11:03:32.690562","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:03:37.322519","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:23:15.973296","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"47","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":9,"pool_name":"cephfs_data","create_time":"2021-02-10 11:23:17.045434","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"48","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"be691b4d-a039-408a-af80-9d53f02b6f1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6818","nonce":36088},{"type":"v1","addr":"172.21.2.102:6820","nonce":36088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6822","nonce":36088},{"type":"v1","addr":"172.21.2.102:6824","nonce":36088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6830","nonce":36088},{"type":"v1","addr":"172.21.2.102:6832","nonce":36088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6826","nonce":36088},{"type":"v1","addr":"172.21.2.102:6828","nonce":36088}]},"public_addr":"172.21.2.102:6820/36088","cluster_addr":"172.21.2.102:6824/36088","heartbeat_back_addr":"172.21.2.102:6832/36088","heartbeat_front_addr":"172.21.2.102:6828/36088","state":["exists","up"]},{"osd":1,"uuid":"64e3ccc8-9c69-4d7c-865e-af8e83822635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6819","nonce":36087},{"type":"v1","addr":"172.21.2.102:6821","nonce":36087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6823","nonce":36087},{"type":"v1","addr":"172.21.2.102:6825","nonce":36087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6831","nonce":36087},{"type":"v1","addr":"172.21.2.102:6833","nonce":36087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6827","nonce":36087},{"type":"v1","addr":"172.21.2.102:6829","nonce":36087}]},"public_addr":"172.21.2.102:6821/36087","cluster_addr":"172.21.2.102:6825/36087","heartbeat_back_addr":"172.21.2.102:6833/36087","heartbeat_front_addr":"172.21.2.102:6829/36087","state":["exists","up"]},{"osd":2,"uuid":"153700af-747f-481f-bdf4-d6072d3a842f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6810","nonce":36089},{"type":"v1","addr":"172.21.2.102:6811","nonce":36089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6812","nonce":36089},{"type":"v1","addr":"172.21.2.102:6813","nonce":36089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6816","nonce":36089},{"type":"v1","addr":"172.21.2.102:6817","nonce":36089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6814","nonce":36089},{"type":"v1","addr":"172.21.2.102:6815","nonce":36089}]},"public_addr":"172.21.2.102:6811/36089","cluster_addr":"172.21.2.102:6813/36089","heartbeat_back_addr":"172.21.2.102:6817/36089","heartbeat_front_addr":"172.21.2.102:6815/36089","state":["exists","up"]},{"osd":3,"uuid":"cbc678f4-83a9-45e6-8830-2f7c232b1ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6802","nonce":36090},{"type":"v1","addr":"172.21.2.102:6803","nonce":36090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6804","nonce":36090},{"type":"v1","addr":"172.21.2.102:6805","nonce":36090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6808","nonce":36090},{"type":"v1","addr":"172.21.2.102:6809","nonce":36090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6806","nonce":36090},{"type":"v1","addr":"172.21.2.102:6807","nonce":36090}]},"public_addr":"172.21.2.102:6803/36090","cluster_addr":"172.21.2.102:6805/36090","heartbeat_back_addr":"172.21.2.102:6809/36090","heartbeat_front_addr":"172.21.2.102:6807/36090","state":["exists","up"]},{"osd":4,"uuid":"c01e4008-b258-4f10-afd2-9e7f37214351","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6808","nonce":36368},{"type":"v1","addr":"172.21.2.113:6811","nonce":36368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6814","nonce":36368},{"type":"v1","addr":"172.21.2.113:6818","nonce":36368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6826","nonce":36368},{"type":"v1","addr":"172.21.2.113:6828","nonce":36368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6820","nonce":36368},{"type":"v1","addr":"172.21.2.113:6824","nonce":36368}]},"public_addr":"172.21.2.113:6811/36368","cluster_addr":"172.21.2.113:6818/36368","heartbeat_back_addr":"172.21.2.113:6828/36368","heartbeat_front_addr":"172.21.2.113:6824/36368","state":["exists","up"]},{"osd":5,"uuid":"551f6007-daf8-4a98-8ee0-c2b16bcccb5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6802","nonce":36366},{"type":"v1","addr":"172.21.2.113:6805","nonce":36366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6807","nonce":36366},{"type":"v1","addr":"172.21.2.113:6810","nonce":36366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6819","nonce":36366},{"type":"v1","addr":"172.21.2.113:6822","nonce":36366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6812","nonce":36366},{"type":"v1","addr":"172.21.2.113:6815","nonce":36366}]},"public_addr":"172.21.2.113:6805/36366","cluster_addr":"172.21.2.113:6810/36366","heartbeat_back_addr":"172.21.2.113:6822/36366","heartbeat_front_addr":"172.21.2.113:6815/36366","state":["exists","up"]},{"osd":6,"uuid":"bf67c040-0e50-48be-b102-c3943bc9ebef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6800","nonce":36367},{"type":"v1","addr":"172.21.2.113:6801","nonce":36367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6803","nonce":36367},{"type":"v1","addr":"172.21.2.113:6804","nonce":36367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6813","nonce":36367},{"type":"v1","addr":"172.21.2.113:6816","nonce":36367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6806","nonce":36367},{"type":"v1","addr":"172.21.2.113:6809","nonce":36367}]},"public_addr":"172.21.2.113:6801/36367","cluster_addr":"172.21.2.113:6804/36367","heartbeat_back_addr":"172.21.2.113:6816/36367","heartbeat_front_addr":"172.21.2.113:6809/36367","state":["exists","up"]},{"osd":7,"uuid":"f5461f80-018d-428f-be89-2eb2bcbe7bb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6817","nonce":36365},{"type":"v1","addr":"172.21.2.113:6821","nonce":36365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6823","nonce":36365},{"type":"v1","addr":"172.21.2.113:6825","nonce":36365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6830","nonce":36365},{"type":"v1","addr":"172.21.2.113:6831","nonce":36365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6827","nonce":36365},{"type":"v1","addr":"172.21.2.113:6829","nonce":36365}]},"public_addr":"172.21.2.113:6821/36365","cluster_addr":"172.21.2.113:6825/36365","heartbeat_back_addr":"172.21.2.113:6831/36365","heartbeat_front_addr":"172.21.2.113:6829/36365","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:23:33.001 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:33.003 7fe747cd8700 1 -- 172.21.2.102:0/194425896 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe7280186c0 msgr2=0x7fe72801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:33.001 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:33.003 7fe747cd8700 1 --2- 172.21.2.102:0/194425896 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe7280186c0 0x7fe72801abc0 crc :-1 s=READY pgs=580 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:33.001 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:33.003 7fe747cd8700 1 -- 172.21.2.102:0/194425896 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe7400ad110 msgr2=0x7fe7400ad510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:33.001 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:33.003 7fe747cd8700 1 --2- 172.21.2.102:0/194425896 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe7400ad110 0x7fe7400ad510 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 rx=0x7fe73c006f00 tx=0x7fe73c00b940).stop 2021-02-10T11:23:33.001 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:33.003 7fe747cd8700 1 -- 172.21.2.102:0/194425896 shutdown_connections 2021-02-10T11:23:33.001 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:33.003 7fe747cd8700 1 --2- 172.21.2.102:0/194425896 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe7280186c0 0x7fe72801abc0 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:33.001 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:33.003 7fe747cd8700 1 --2- 172.21.2.102:0/194425896 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe7400adb30 0x7fe74008e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:33.002 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:33.003 7fe747cd8700 1 --2- 172.21.2.102:0/194425896 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe7400ad110 0x7fe7400ad510 unknown :-1 s=CLOSED pgs=1220 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:33.002 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:33.003 7fe747cd8700 1 --2- 172.21.2.102:0/194425896 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe740192960 0x7fe740192d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:33.002 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:33.003 7fe747cd8700 1 -- 172.21.2.102:0/194425896 >> 172.21.2.102:0/194425896 conn(0x7fe7400891f0 msgr2=0x7fe7401a0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:33.002 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:33.003 7fe747cd8700 1 -- 172.21.2.102:0/194425896 shutdown_connections 2021-02-10T11:23:33.002 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:33.003 7fe747cd8700 1 -- 172.21.2.102:0/194425896 wait complete. 2021-02-10T11:23:33.010 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:03:37.322519', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 8, 'pool_name': 'cephfs_metadata', 'create_time': '2021-02-10 11:23:15.973296', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '47', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 9, 'pool_name': 'cephfs_data', 'create_time': '2021-02-10 11:23:17.045434', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '48', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-02-10T11:23:33.010 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:23:33.174 DEBUG:teuthology.orchestra.run.gibba002:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:23:33.178 DEBUG:teuthology.orchestra.run.gibba013:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:23:33.440 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:33.425 7efc23266700 1 Processor -- start 2021-02-10T11:23:33.440 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:33.425 7efc23266700 1 -- start start 2021-02-10T11:23:33.440 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:33.426 7efc23266700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7efc1c1b0870 0x7efc1c0f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:33.440 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:33.426 7efc23266700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7efc1c0efe00 0x7efc1c0f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:33.441 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:33.426 7efc23266700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc1c1b0c70 con 0x7efc1c1b0470 2021-02-10T11:23:33.441 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:33.426 7efc23266700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc1c0e5ea0 con 0x7efc1c1b0870 2021-02-10T11:23:33.441 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:33.426 7efc23266700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc1c0f0b50 con 0x7efc1c0efe00 2021-02-10T11:23:33.441 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:33.426 7efc20881700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7efc1c1b0870 0x7efc1c0f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:33.441 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:33.426 7efc21082700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7efc1c1b0470 0x7efc1c0ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:48670/0 (socket says 172.21.2.102:48670) 2021-02-10T11:23:33.441 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:33.426 7efc21082700 1 -- 172.21.2.102:0/1194338320 learned_addr learned my addr 172.21.2.102:0/1194338320 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:23:33.441 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:33.426 7efc21883700 1 --2- 172.21.2.102:0/1194338320 >> v2:172.21.2.113:3300/0 conn(0x7efc1c0efe00 0x7efc1c0f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:33.441 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:33.426 7efc20881700 1 -- 172.21.2.102:0/1194338320 >> v2:172.21.2.113:3300/0 conn(0x7efc1c0efe00 msgr2=0x7efc1c0f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:33.441 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:33.426 7efc20881700 1 --2- 172.21.2.102:0/1194338320 >> v2:172.21.2.113:3300/0 conn(0x7efc1c0efe00 0x7efc1c0f0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:33.441 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:33.426 7efc20881700 1 -- 172.21.2.102:0/1194338320 >> v1:172.21.2.102:6789/0 conn(0x7efc1c1b0470 legacy=0x7efc1c0ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:33.441 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:33.426 7efc20881700 1 -- 172.21.2.102:0/1194338320 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc1c0f0820 con 0x7efc1c1b0870 2021-02-10T11:23:33.441 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:33.427 7efc20881700 1 --2- 172.21.2.102:0/1194338320 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efc1c1b0870 0x7efc1c0f20e0 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 rx=0x7efc18006650 tx=0x7efc18009a80).ready entity=mon.0 client_cookie=19d83ae11ea2c5d9 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:33.441 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:33.427 7efc0ffff700 1 -- 172.21.2.102:0/1194338320 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efc180009b0 con 0x7efc1c1b0870 2021-02-10T11:23:33.442 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:33.427 7efc0ffff700 1 -- 172.21.2.102:0/1194338320 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc1800c0c0 con 0x7efc1c1b0870 2021-02-10T11:23:33.442 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:33.427 7efc0ffff700 1 -- 172.21.2.102:0/1194338320 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efc1800c5e0 con 0x7efc1c1b0870 2021-02-10T11:23:33.442 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:33.427 7efc23266700 1 -- 172.21.2.102:0/1194338320 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efc1c1b0870 msgr2=0x7efc1c0f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:33.442 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:33.427 7efc23266700 1 --2- 172.21.2.102:0/1194338320 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efc1c1b0870 0x7efc1c0f20e0 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 rx=0x7efc18006650 tx=0x7efc18009a80).stop 2021-02-10T11:23:33.442 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:33.428 7efc23266700 1 -- 172.21.2.102:0/1194338320 shutdown_connections 2021-02-10T11:23:33.442 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:33.428 7efc23266700 1 --2- 172.21.2.102:0/1194338320 >> v2:172.21.2.113:3300/0 conn(0x7efc1c0efe00 0x7efc1c0f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:33.442 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:33.428 7efc23266700 1 --2- 172.21.2.102:0/1194338320 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efc1c1b0870 0x7efc1c0f20e0 unknown :-1 s=CLOSED pgs=1222 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:33.442 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:33.428 7efc23266700 1 -- 172.21.2.102:0/1194338320 >> 172.21.2.102:0/1194338320 conn(0x7efc1c0ce3e0 msgr2=0x7efc1c0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:33.442 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:33.428 7efc23266700 1 -- 172.21.2.102:0/1194338320 shutdown_connections 2021-02-10T11:23:33.442 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:33.428 7efc23266700 1 -- 172.21.2.102:0/1194338320 wait complete. 2021-02-10T11:23:33.444 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:33.443 7efc23266700 1 Processor -- start 2021-02-10T11:23:33.444 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:33.443 7efc23266700 1 -- start start 2021-02-10T11:23:33.444 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:33.443 7efc23266700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7efc1c0a6a70 0x7efc1c0a5f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:33.444 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:33.443 7efc23266700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efc1c0a6e70 0x7efc1c0ba3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:33.444 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:33.443 7efc23266700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7efc1c0b9810 0x7efc1c0ba9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:33.444 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:33.443 7efc23266700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc1c0b9c10 con 0x7efc1c0a6e70 2021-02-10T11:23:33.444 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:33.443 7efc23266700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc1c0b9e20 con 0x7efc1c0b9810 2021-02-10T11:23:33.444 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:33.443 7efc23266700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc1c0a0a80 con 0x7efc1c0a6a70 2021-02-10T11:23:33.444 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:33.443 7efc20881700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efc1c0a6e70 0x7efc1c0ba3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:33.445 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:33.443 7efc20881700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efc1c0a6e70 0x7efc1c0ba3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:56452/0 (socket says 172.21.2.102:56452) 2021-02-10T11:23:33.445 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:33.443 7efc20881700 1 -- 172.21.2.102:0/3349736842 learned_addr learned my addr 172.21.2.102:0/3349736842 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:23:33.445 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:33.444 7efc20881700 1 -- 172.21.2.102:0/3349736842 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7efc1c0a6a70 msgr2=0x7efc1c0a5f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:33.445 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:33.444 7efc20881700 1 --2- 172.21.2.102:0/3349736842 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7efc1c0a6a70 0x7efc1c0a5f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:33.445 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:33.444 7efc20881700 1 -- 172.21.2.102:0/3349736842 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7efc1c0b9810 msgr2=0x7efc1c0ba9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:33.445 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:33.444 7efc20881700 1 --2- 172.21.2.102:0/3349736842 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7efc1c0b9810 0x7efc1c0ba9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:33.445 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:33.444 7efc20881700 1 -- 172.21.2.102:0/3349736842 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc1c0a6540 con 0x7efc1c0a6e70 2021-02-10T11:23:33.445 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:33.444 7efc20881700 1 --2- 172.21.2.102:0/3349736842 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efc1c0a6e70 0x7efc1c0ba3a0 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 rx=0x7efc18009630 tx=0x7efc1800b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:33.445 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:33.444 7efc0dffb700 1 -- 172.21.2.102:0/3349736842 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efc180093a0 con 0x7efc1c0a6e70 2021-02-10T11:23:33.445 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:33.444 7efc0dffb700 1 -- 172.21.2.102:0/3349736842 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc1800bd70 con 0x7efc1c0a6e70 2021-02-10T11:23:33.445 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:33.444 7efc0dffb700 1 -- 172.21.2.102:0/3349736842 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efc18005560 con 0x7efc1c0a6e70 2021-02-10T11:23:33.445 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:33.444 7efc23266700 1 -- 172.21.2.102:0/3349736842 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc1c0a6540 con 0x7efc1c0a6e70 2021-02-10T11:23:33.445 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:33.444 7efc23266700 1 -- 172.21.2.102:0/3349736842 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc1c0a6540 con 0x7efc1c0a6e70 2021-02-10T11:23:33.446 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:33.445 7efc0dffb700 1 -- 172.21.2.102:0/3349736842 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7efc18008240 con 0x7efc1c0a6e70 2021-02-10T11:23:33.446 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:33.445 7efc0dffb700 1 --2- 172.21.2.102:0/3349736842 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7efbfc0186c0 0x7efbfc01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:33.446 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:33.445 7efc0dffb700 1 -- 172.21.2.102:0/3349736842 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5573+0+0 (secure 0 0 0) 0x7efc18032570 con 0x7efc1c0a6e70 2021-02-10T11:23:33.446 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:33.445 7efc21082700 1 --2- 172.21.2.102:0/3349736842 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7efbfc0186c0 0x7efbfc01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:33.446 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:33.445 7efc21082700 1 --2- 172.21.2.102:0/3349736842 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7efbfc0186c0 0x7efbfc01abc0 crc :-1 s=READY pgs=581 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:33.473 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:33.475 7efc23266700 1 -- 172.21.2.102:0/3349736842 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc1c105ae0 con 0x7efc1c0a6e70 2021-02-10T11:23:33.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:33.478 7efc0dffb700 1 -- 172.21.2.102:0/3349736842 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7efc18024ee0 con 0x7efc1c0a6e70 2021-02-10T11:23:33.608 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:33.610 7efc23266700 1 -- 172.21.2.102:0/3349736842 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7efc1c0f0b50 con 0x7efc1c0a6e70 2021-02-10T11:23:33.608 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:33.610 7efc0dffb700 1 -- 172.21.2.102:0/3349736842 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v48) v1 ==== 92+0+10 (secure 0 0 0) 0x7efc18024ee0 con 0x7efc1c0a6e70 2021-02-10T11:23:33.608 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:23:33.610 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:33.612 7efc23266700 1 -- 172.21.2.102:0/3349736842 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7efbfc0186c0 msgr2=0x7efbfc01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:33.610 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:33.612 7efc23266700 1 --2- 172.21.2.102:0/3349736842 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7efbfc0186c0 0x7efbfc01abc0 crc :-1 s=READY pgs=581 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:33.610 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:33.612 7efc23266700 1 -- 172.21.2.102:0/3349736842 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efc1c0a6e70 msgr2=0x7efc1c0ba3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:33.610 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:33.612 7efc23266700 1 --2- 172.21.2.102:0/3349736842 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efc1c0a6e70 0x7efc1c0ba3a0 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 rx=0x7efc18009630 tx=0x7efc1800b980).stop 2021-02-10T11:23:33.610 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:33.612 7efc23266700 1 -- 172.21.2.102:0/3349736842 shutdown_connections 2021-02-10T11:23:33.611 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:33.612 7efc23266700 1 --2- 172.21.2.102:0/3349736842 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7efbfc0186c0 0x7efbfc01abc0 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:33.611 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:33.612 7efc23266700 1 --2- 172.21.2.102:0/3349736842 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7efc1c0b9810 0x7efc1c0ba9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:33.611 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:33.612 7efc23266700 1 --2- 172.21.2.102:0/3349736842 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efc1c0a6e70 0x7efc1c0ba3a0 unknown :-1 s=CLOSED pgs=1223 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:33.611 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:33.612 7efc23266700 1 --2- 172.21.2.102:0/3349736842 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7efc1c0a6a70 0x7efc1c0a5f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:33.611 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:33.612 7efc23266700 1 -- 172.21.2.102:0/3349736842 >> 172.21.2.102:0/3349736842 conn(0x7efc1c0bbf60 msgr2=0x7efc1c105d10 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:33.611 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:33.612 7efc23266700 1 -- 172.21.2.102:0/3349736842 shutdown_connections 2021-02-10T11:23:33.611 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:33.612 7efc23266700 1 -- 172.21.2.102:0/3349736842 wait complete. 2021-02-10T11:23:33.619 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-02-10T11:23:34.087 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:34.073 7f48b538d700 1 Processor -- start 2021-02-10T11:23:34.088 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:34.073 7f48b538d700 1 -- start start 2021-02-10T11:23:34.088 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:34.073 7f48b538d700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f48b01b0870 0x7f48b00f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:34.088 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:34.073 7f48b538d700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48b01b0c70 con 0x7f48b00efe00 2021-02-10T11:23:34.088 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:34.073 7f48b538d700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48b00f0b50 con 0x7f48b01b0470 2021-02-10T11:23:34.088 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:34.073 7f48b538d700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48b00e5ea0 con 0x7f48b01b0870 2021-02-10T11:23:34.088 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:34.073 7f48af87e700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f48b00efe00 0x7f48b00f0200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:48688/0 (socket says 172.21.2.102:48688) 2021-02-10T11:23:34.088 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:34.073 7f48af87e700 1 -- 172.21.2.102:0/3141179922 learned_addr learned my addr 172.21.2.102:0/3141179922 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:23:34.088 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:34.073 7f48ae87c700 1 --2- 172.21.2.102:0/3141179922 >> v2:172.21.2.102:3300/0 conn(0x7f48b01b0870 0x7f48b00f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:34.088 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:34.074 7f48ae07b700 1 -- 172.21.2.102:0/3141179922 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2914321152 0 0) 0x7f48a8001500 con 0x7f48b00efe00 2021-02-10T11:23:34.088 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:34.074 7f48ae07b700 1 -- 172.21.2.102:0/3141179922 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f489c002fe0 con 0x7f48b00efe00 2021-02-10T11:23:34.088 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:34.074 7f48ae87c700 1 -- 172.21.2.102:0/3141179922 >> v1:172.21.2.113:6789/0 conn(0x7f48b01b0470 legacy=0x7f48b00ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:34.089 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:34.074 7f48ae87c700 1 -- 172.21.2.102:0/3141179922 >> v1:172.21.2.102:6789/0 conn(0x7f48b00efe00 legacy=0x7f48b00f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:34.089 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:34.074 7f48ae87c700 1 -- 172.21.2.102:0/3141179922 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48b00e52e0 con 0x7f48b01b0870 2021-02-10T11:23:34.089 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:34.074 7f48ae87c700 1 --2- 172.21.2.102:0/3141179922 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f48b01b0870 0x7f48b00f20e0 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 rx=0x7f48a0006650 tx=0x7f48a0009a80).ready entity=mon.0 client_cookie=c6777f7884ca26b server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:34.089 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:34.074 7f48ae07b700 1 -- 172.21.2.102:0/3141179922 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f48a000c0c0 con 0x7f48b01b0870 2021-02-10T11:23:34.089 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:34.074 7f48ae07b700 1 -- 172.21.2.102:0/3141179922 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48a000c340 con 0x7f48b01b0870 2021-02-10T11:23:34.089 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:34.074 7f48ae07b700 1 -- 172.21.2.102:0/3141179922 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f48a000c860 con 0x7f48b01b0870 2021-02-10T11:23:34.089 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:34.074 7f48b538d700 1 -- 172.21.2.102:0/3141179922 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f48b01b0870 msgr2=0x7f48b00f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:34.089 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:34.074 7f48b538d700 1 --2- 172.21.2.102:0/3141179922 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f48b01b0870 0x7f48b00f20e0 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 rx=0x7f48a0006650 tx=0x7f48a0009a80).stop 2021-02-10T11:23:34.089 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:34.075 7f48b538d700 1 -- 172.21.2.102:0/3141179922 shutdown_connections 2021-02-10T11:23:34.089 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:34.075 7f48b538d700 1 --2- 172.21.2.102:0/3141179922 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f48b01b0870 0x7f48b00f20e0 unknown :-1 s=CLOSED pgs=1225 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:34.089 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:34.075 7f48b538d700 1 -- 172.21.2.102:0/3141179922 >> 172.21.2.102:0/3141179922 conn(0x7f48b00ce3e0 msgr2=0x7f48b00e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:34.089 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:34.075 7f48b538d700 1 -- 172.21.2.102:0/3141179922 shutdown_connections 2021-02-10T11:23:34.089 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:34.075 7f48b538d700 1 -- 172.21.2.102:0/3141179922 wait complete. 2021-02-10T11:23:34.091 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:34.090 7f48b538d700 1 Processor -- start 2021-02-10T11:23:34.091 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:34.090 7f48b538d700 1 -- start start 2021-02-10T11:23:34.091 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:34.090 7f48b538d700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f48b00c36a0 0x7f48b00c2b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:34.091 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:34.090 7f48b538d700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f48b00c3190 0x7f48b00b1d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:34.091 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:34.090 7f48b538d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f48b00b2370 0x7f48b00b2d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:34.091 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:34.090 7f48b538d700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48b00b27c0 con 0x7f48b00b2370 2021-02-10T11:23:34.091 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:34.090 7f48b538d700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48b00b3360 con 0x7f48b00c36a0 2021-02-10T11:23:34.091 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:34.090 7f48b538d700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48b00b3540 con 0x7f48b00c3190 2021-02-10T11:23:34.091 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:34.090 7f48af87e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f48b00b2370 0x7f48b00b2d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:34.092 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:34.090 7f48af87e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f48b00b2370 0x7f48b00b2d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:56466/0 (socket says 172.21.2.102:56466) 2021-02-10T11:23:34.092 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:34.090 7f48af87e700 1 -- 172.21.2.102:0/3653512885 learned_addr learned my addr 172.21.2.102:0/3653512885 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:23:34.092 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:34.091 7f48af87e700 1 -- 172.21.2.102:0/3653512885 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f48b00c3190 msgr2=0x7f48b00b1d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:34.092 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:34.091 7f48af87e700 1 --2- 172.21.2.102:0/3653512885 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f48b00c3190 0x7f48b00b1d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:34.092 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:34.091 7f48af87e700 1 -- 172.21.2.102:0/3653512885 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f48b00c36a0 msgr2=0x7f48b00c2b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:34.092 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:34.091 7f48af87e700 1 --2- 172.21.2.102:0/3653512885 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f48b00c36a0 0x7f48b00c2b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:34.092 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:34.091 7f48af87e700 1 -- 172.21.2.102:0/3653512885 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48b00b3760 con 0x7f48b00b2370 2021-02-10T11:23:34.092 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:34.091 7f48af87e700 1 --2- 172.21.2.102:0/3653512885 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f48b00b2370 0x7f48b00b2d40 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 rx=0x7f48a8001c10 tx=0x7f48a800aca0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:34.092 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:34.091 7f48a7fff700 1 -- 172.21.2.102:0/3653512885 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f48a8007ce0 con 0x7f48b00b2370 2021-02-10T11:23:34.092 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:34.091 7f48a7fff700 1 -- 172.21.2.102:0/3653512885 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48a80070d0 con 0x7f48b00b2370 2021-02-10T11:23:34.092 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:34.091 7f48a7fff700 1 -- 172.21.2.102:0/3653512885 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f48a8004930 con 0x7f48b00b2370 2021-02-10T11:23:34.092 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:34.091 7f48b538d700 1 -- 172.21.2.102:0/3653512885 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48b00b3760 con 0x7f48b00b2370 2021-02-10T11:23:34.092 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:34.091 7f48b538d700 1 -- 172.21.2.102:0/3653512885 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48b00b3cf0 con 0x7f48b00b2370 2021-02-10T11:23:34.092 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:34.092 7f48a7fff700 1 -- 172.21.2.102:0/3653512885 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f48a80072b0 con 0x7f48b00b2370 2021-02-10T11:23:34.093 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:34.092 7f48a7fff700 1 --2- 172.21.2.102:0/3653512885 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f48940186c0 0x7f489401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:34.093 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:34.092 7f48a7fff700 1 -- 172.21.2.102:0/3653512885 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f48a80328c0 con 0x7f48b00b2370 2021-02-10T11:23:34.093 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:34.092 7f48af07d700 1 --2- 172.21.2.102:0/3653512885 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f48940186c0 0x7f489401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:34.093 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:34.092 7f48af07d700 1 --2- 172.21.2.102:0/3653512885 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f48940186c0 0x7f489401abc0 crc :-1 s=READY pgs=582 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:34.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:34.122 7f48b538d700 1 -- 172.21.2.102:0/3653512885 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48b00edd10 con 0x7f48b00b2370 2021-02-10T11:23:34.123 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:34.125 7f48a7fff700 1 -- 172.21.2.102:0/3653512885 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f48a8025d40 con 0x7f48b00b2370 2021-02-10T11:23:34.252 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:34.254 7f48b538d700 1 -- 172.21.2.102:0/3653512885 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7f48b00e52e0 con 0x7f48b00b2370 2021-02-10T11:23:34.252 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:34.254 7f48a7fff700 1 -- 172.21.2.102:0/3653512885 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v48) v1 ==== 104+0+10 (secure 0 0 0) 0x7f48a8025d40 con 0x7f48b00b2370 2021-02-10T11:23:34.253 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:23:34.254 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:34.256 7f48b538d700 1 -- 172.21.2.102:0/3653512885 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f48940186c0 msgr2=0x7f489401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:34.254 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:34.256 7f48b538d700 1 --2- 172.21.2.102:0/3653512885 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f48940186c0 0x7f489401abc0 crc :-1 s=READY pgs=582 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:34.254 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:34.256 7f48b538d700 1 -- 172.21.2.102:0/3653512885 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f48b00b2370 msgr2=0x7f48b00b2d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:34.255 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:34.256 7f48b538d700 1 --2- 172.21.2.102:0/3653512885 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f48b00b2370 0x7f48b00b2d40 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 rx=0x7f48a8001c10 tx=0x7f48a800aca0).stop 2021-02-10T11:23:34.255 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:34.256 7f48b538d700 1 -- 172.21.2.102:0/3653512885 shutdown_connections 2021-02-10T11:23:34.255 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:34.256 7f48b538d700 1 --2- 172.21.2.102:0/3653512885 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f48b00b2370 0x7f48b00b2d40 unknown :-1 s=CLOSED pgs=1226 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:34.255 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:34.256 7f48b538d700 1 --2- 172.21.2.102:0/3653512885 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f48b00c3190 0x7f48b00b1d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:34.255 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:34.256 7f48b538d700 1 --2- 172.21.2.102:0/3653512885 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f48940186c0 0x7f489401abc0 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:34.255 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:34.256 7f48b538d700 1 --2- 172.21.2.102:0/3653512885 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f48b00c36a0 0x7f48b00c2b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:34.255 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:34.256 7f48b538d700 1 -- 172.21.2.102:0/3653512885 >> 172.21.2.102:0/3653512885 conn(0x7f48b009f630 msgr2=0x7f48b00a0130 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:34.255 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:34.256 7f48b538d700 1 -- 172.21.2.102:0/3653512885 shutdown_connections 2021-02-10T11:23:34.255 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:34.257 7f48b538d700 1 -- 172.21.2.102:0/3653512885 wait complete. 2021-02-10T11:23:34.264 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-02-10T11:23:34.694 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:34.680 7fa252a34700 1 Processor -- start 2021-02-10T11:23:34.694 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:34.680 7fa252a34700 1 -- start start 2021-02-10T11:23:34.694 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:34.680 7fa252a34700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa24c1b0470 0x7fa24c1b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:34.694 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:34.680 7fa252a34700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fa24c0f20e0 0x7fa24c0f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:34.694 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:34.680 7fa252a34700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa24c0f0b50 con 0x7fa24c0ef610 2021-02-10T11:23:34.694 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:34.680 7fa252a34700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa24c0f0820 con 0x7fa24c0f20e0 2021-02-10T11:23:34.694 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:34.680 7fa252a34700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa24c0f24e0 con 0x7fa24c1b0470 2021-02-10T11:23:34.694 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:34.680 7fa251051700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fa24c0f20e0 0x7fa24c0f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:34.694 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:34.680 7fa251051700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fa24c0f20e0 0x7fa24c0f0200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:56478/0 (socket says 172.21.2.102:56478) 2021-02-10T11:23:34.695 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:34.680 7fa251051700 1 -- 172.21.2.102:0/443992378 learned_addr learned my addr 172.21.2.102:0/443992378 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:23:34.695 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:34.681 7fa251051700 1 -- 172.21.2.102:0/443992378 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa24c1b0470 msgr2=0x7fa24c1b0870 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:23:34.695 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:34.681 7fa251051700 1 --2- 172.21.2.102:0/443992378 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa24c1b0470 0x7fa24c1b0870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:34.695 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:34.681 7fa251051700 1 -- 172.21.2.102:0/443992378 >> v1:172.21.2.113:6789/0 conn(0x7fa24c0ef610 legacy=0x7fa24c0efa10 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:23:34.695 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:34.681 7fa251051700 1 -- 172.21.2.102:0/443992378 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa24c0e52e0 con 0x7fa24c0f20e0 2021-02-10T11:23:34.695 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:34.681 7fa251051700 1 --2- 172.21.2.102:0/443992378 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa24c0f20e0 0x7fa24c0f0200 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 rx=0x7fa244006b80 tx=0x7fa244009b90).ready entity=mon.0 client_cookie=1a4f3f2e283d6b95 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:34.695 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:34.681 7fa24b7fe700 1 -- 172.21.2.102:0/443992378 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa2440022e0 con 0x7fa24c0f20e0 2021-02-10T11:23:34.695 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:34.681 7fa24b7fe700 1 -- 172.21.2.102:0/443992378 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa244002560 con 0x7fa24c0f20e0 2021-02-10T11:23:34.695 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:34.681 7fa24b7fe700 1 -- 172.21.2.102:0/443992378 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa244002a80 con 0x7fa24c0f20e0 2021-02-10T11:23:34.695 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:34.681 7fa252a34700 1 -- 172.21.2.102:0/443992378 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa24c0f20e0 msgr2=0x7fa24c0f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:34.695 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:34.681 7fa252a34700 1 --2- 172.21.2.102:0/443992378 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa24c0f20e0 0x7fa24c0f0200 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 rx=0x7fa244006b80 tx=0x7fa244009b90).stop 2021-02-10T11:23:34.695 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:34.682 7fa252a34700 1 -- 172.21.2.102:0/443992378 shutdown_connections 2021-02-10T11:23:34.695 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:34.682 7fa252a34700 1 --2- 172.21.2.102:0/443992378 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa24c0f20e0 0x7fa24c0f0200 unknown :-1 s=CLOSED pgs=1227 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:34.695 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:34.682 7fa252a34700 1 --2- 172.21.2.102:0/443992378 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa24c1b0470 0x7fa24c1b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:34.696 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:34.682 7fa252a34700 1 -- 172.21.2.102:0/443992378 >> 172.21.2.102:0/443992378 conn(0x7fa24c0ce3e0 msgr2=0x7fa24c0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:34.696 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:34.682 7fa252a34700 1 -- 172.21.2.102:0/443992378 shutdown_connections 2021-02-10T11:23:34.696 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:34.682 7fa252a34700 1 -- 172.21.2.102:0/443992378 wait complete. 2021-02-10T11:23:34.697 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:34.696 7fa252a34700 1 Processor -- start 2021-02-10T11:23:34.697 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:34.696 7fa252a34700 1 -- start start 2021-02-10T11:23:34.697 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:34.697 7fa252a34700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa24c0c36a0 0x7fa24c0c2b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:34.697 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:34.697 7fa252a34700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa24c0c3190 0x7fa24c0b1d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:34.697 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:34.697 7fa252a34700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa24c0b2370 0x7fa24c0b2d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:34.697 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:34.697 7fa252a34700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa24c0b27c0 con 0x7fa24c0b2370 2021-02-10T11:23:34.698 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:34.697 7fa252a34700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa24c0b3360 con 0x7fa24c0c3190 2021-02-10T11:23:34.698 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:34.697 7fa252a34700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa24c0b3540 con 0x7fa24c0c36a0 2021-02-10T11:23:34.698 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:34.697 7fa251051700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa24c0b2370 0x7fa24c0b2d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:34.698 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:34.697 7fa251051700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa24c0b2370 0x7fa24c0b2d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:56480/0 (socket says 172.21.2.102:56480) 2021-02-10T11:23:34.698 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:34.697 7fa251051700 1 -- 172.21.2.102:0/161541586 learned_addr learned my addr 172.21.2.102:0/161541586 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:23:34.698 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:34.697 7fa251051700 1 -- 172.21.2.102:0/161541586 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa24c0c36a0 msgr2=0x7fa24c0c2b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:34.698 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:34.697 7fa251051700 1 --2- 172.21.2.102:0/161541586 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa24c0c36a0 0x7fa24c0c2b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:34.698 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:34.697 7fa251051700 1 -- 172.21.2.102:0/161541586 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa24c0c3190 msgr2=0x7fa24c0b1d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:34.698 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:34.697 7fa251051700 1 --2- 172.21.2.102:0/161541586 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa24c0c3190 0x7fa24c0b1d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:34.698 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:34.697 7fa251051700 1 -- 172.21.2.102:0/161541586 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa24c0b3760 con 0x7fa24c0b2370 2021-02-10T11:23:34.698 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:34.697 7fa251051700 1 --2- 172.21.2.102:0/161541586 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa24c0b2370 0x7fa24c0b2d40 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 rx=0x7fa244006f10 tx=0x7fa24400b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:34.698 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:34.697 7fa2497fa700 1 -- 172.21.2.102:0/161541586 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa244000b90 con 0x7fa24c0b2370 2021-02-10T11:23:34.698 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:34.698 7fa2497fa700 1 -- 172.21.2.102:0/161541586 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa24400bd50 con 0x7fa24c0b2370 2021-02-10T11:23:34.699 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:34.698 7fa2497fa700 1 -- 172.21.2.102:0/161541586 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa244002220 con 0x7fa24c0b2370 2021-02-10T11:23:34.699 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:34.698 7fa252a34700 1 -- 172.21.2.102:0/161541586 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa24c0b3760 con 0x7fa24c0b2370 2021-02-10T11:23:34.699 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:34.698 7fa252a34700 1 -- 172.21.2.102:0/161541586 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa24c0b3760 con 0x7fa24c0b2370 2021-02-10T11:23:34.699 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:34.698 7fa2497fa700 1 -- 172.21.2.102:0/161541586 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa244009100 con 0x7fa24c0b2370 2021-02-10T11:23:34.699 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:34.698 7fa2497fa700 1 --2- 172.21.2.102:0/161541586 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa2300186c0 0x7fa23001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:34.699 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:34.698 7fa2497fa700 1 -- 172.21.2.102:0/161541586 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fa2440323f0 con 0x7fa24c0b2370 2021-02-10T11:23:34.699 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:34.698 7fa250850700 1 --2- 172.21.2.102:0/161541586 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa2300186c0 0x7fa23001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:34.699 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:34.699 7fa250850700 1 --2- 172.21.2.102:0/161541586 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa2300186c0 0x7fa23001abc0 crc :-1 s=READY pgs=583 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:34.729 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:34.730 7fa252a34700 1 -- 172.21.2.102:0/161541586 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa24c0edd10 con 0x7fa24c0b2370 2021-02-10T11:23:34.730 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:34.732 7fa2497fa700 1 -- 172.21.2.102:0/161541586 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fa244025fa0 con 0x7fa24c0b2370 2021-02-10T11:23:34.858 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:34.860 7fa252a34700 1 -- 172.21.2.102:0/161541586 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7fa24c0edd10 con 0x7fa24c0b2370 2021-02-10T11:23:34.859 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:34.861 7fa2497fa700 1 -- 172.21.2.102:0/161541586 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v48) v1 ==== 100+0+10 (secure 0 0 0) 0x7fa244025fa0 con 0x7fa24c0b2370 2021-02-10T11:23:34.859 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:23:34.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:34.863 7fa252a34700 1 -- 172.21.2.102:0/161541586 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa2300186c0 msgr2=0x7fa23001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:34.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:34.863 7fa252a34700 1 --2- 172.21.2.102:0/161541586 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa2300186c0 0x7fa23001abc0 crc :-1 s=READY pgs=583 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:34.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:34.863 7fa252a34700 1 -- 172.21.2.102:0/161541586 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa24c0b2370 msgr2=0x7fa24c0b2d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:34.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:34.863 7fa252a34700 1 --2- 172.21.2.102:0/161541586 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa24c0b2370 0x7fa24c0b2d40 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 rx=0x7fa244006f10 tx=0x7fa24400b960).stop 2021-02-10T11:23:34.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:34.863 7fa252a34700 1 -- 172.21.2.102:0/161541586 shutdown_connections 2021-02-10T11:23:34.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:34.863 7fa252a34700 1 --2- 172.21.2.102:0/161541586 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa24c0b2370 0x7fa24c0b2d40 unknown :-1 s=CLOSED pgs=1228 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:34.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:34.863 7fa252a34700 1 --2- 172.21.2.102:0/161541586 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa2300186c0 0x7fa23001abc0 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:34.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:34.863 7fa252a34700 1 --2- 172.21.2.102:0/161541586 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa24c0c3190 0x7fa24c0b1d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:34.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:34.863 7fa252a34700 1 --2- 172.21.2.102:0/161541586 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa24c0c36a0 0x7fa24c0c2b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:34.862 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:34.863 7fa252a34700 1 -- 172.21.2.102:0/161541586 >> 172.21.2.102:0/161541586 conn(0x7fa24c09f630 msgr2=0x7fa24c0a0130 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:34.862 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:34.863 7fa252a34700 1 -- 172.21.2.102:0/161541586 shutdown_connections 2021-02-10T11:23:34.862 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:34.863 7fa252a34700 1 -- 172.21.2.102:0/161541586 wait complete. 2021-02-10T11:23:34.869 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-02-10T11:23:34.869 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:23:35.294 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:35.279 7f6ffead5700 1 Processor -- start 2021-02-10T11:23:35.294 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:35.280 7f6ffead5700 1 -- start start 2021-02-10T11:23:35.294 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:35.280 7f6ffead5700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f6ff80e5070 0x7f6ff81b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:35.294 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:35.280 7f6ffead5700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6ff80e5470 0x7f6ff8080310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:35.294 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:35.280 7f6ffead5700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ff807ed80 con 0x7f6ff81b3e40 2021-02-10T11:23:35.294 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:35.280 7f6ffead5700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ff80d1790 con 0x7f6ff80e5070 2021-02-10T11:23:35.295 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:35.280 7f6ffead5700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ff80d19b0 con 0x7f6ff80e5470 2021-02-10T11:23:35.295 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:35.280 7f6ff7fff700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f6ff81b3e40 0x7f6ff807d690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:48714/0 (socket says 172.21.2.102:48714) 2021-02-10T11:23:35.295 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:35.280 7f6ff7fff700 1 -- 172.21.2.102:0/3378807567 learned_addr learned my addr 172.21.2.102:0/3378807567 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:23:35.295 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:35.280 7f6ffc8f1700 1 --2- 172.21.2.102:0/3378807567 >> v2:172.21.2.102:3300/0 conn(0x7f6ff80e5070 0x7f6ff81b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:35.295 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:35.280 7f6ff77fe700 1 -- 172.21.2.102:0/3378807567 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 617585020 0 0) 0x7f6fe80010b0 con 0x7f6ff81b3e40 2021-02-10T11:23:35.295 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:35.280 7f6ff77fe700 1 -- 172.21.2.102:0/3378807567 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6fe4002fe0 con 0x7f6ff81b3e40 2021-02-10T11:23:35.295 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:35.281 7f6ffc8f1700 1 -- 172.21.2.102:0/3378807567 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6ff80e5470 msgr2=0x7f6ff8080310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:35.295 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:35.281 7f6ffc8f1700 1 --2- 172.21.2.102:0/3378807567 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6ff80e5470 0x7f6ff8080310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:35.295 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:35.281 7f6ffc8f1700 1 -- 172.21.2.102:0/3378807567 >> v1:172.21.2.102:6789/0 conn(0x7f6ff81b3e40 legacy=0x7f6ff807d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:35.296 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:35.281 7f6ffc8f1700 1 -- 172.21.2.102:0/3378807567 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ff80d1bd0 con 0x7f6ff80e5070 2021-02-10T11:23:35.296 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:35.281 7f6ffc8f1700 1 --2- 172.21.2.102:0/3378807567 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6ff80e5070 0x7f6ff81b3820 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 rx=0x7f6ff00069d0 tx=0x7f6ff0009e40).ready entity=mon.0 client_cookie=f81a15d910bc2c5a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:35.296 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:35.281 7f6ff77fe700 1 -- 172.21.2.102:0/3378807567 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6ff000b510 con 0x7f6ff80e5070 2021-02-10T11:23:35.296 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:35.281 7f6ff77fe700 1 -- 172.21.2.102:0/3378807567 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ff000b790 con 0x7f6ff80e5070 2021-02-10T11:23:35.296 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:35.281 7f6ff77fe700 1 -- 172.21.2.102:0/3378807567 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6ff000bcb0 con 0x7f6ff80e5070 2021-02-10T11:23:35.296 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:35.281 7f6ffead5700 1 -- 172.21.2.102:0/3378807567 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6ff80e5070 msgr2=0x7f6ff81b3820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:35.296 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:35.281 7f6ffead5700 1 --2- 172.21.2.102:0/3378807567 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6ff80e5070 0x7f6ff81b3820 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 rx=0x7f6ff00069d0 tx=0x7f6ff0009e40).stop 2021-02-10T11:23:35.296 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:35.282 7f6ffead5700 1 -- 172.21.2.102:0/3378807567 shutdown_connections 2021-02-10T11:23:35.296 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:35.282 7f6ffead5700 1 --2- 172.21.2.102:0/3378807567 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6ff80e5470 0x7f6ff8080310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:35.296 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:35.282 7f6ffead5700 1 --2- 172.21.2.102:0/3378807567 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6ff80e5070 0x7f6ff81b3820 unknown :-1 s=CLOSED pgs=1230 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:35.296 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:35.282 7f6ffead5700 1 -- 172.21.2.102:0/3378807567 >> 172.21.2.102:0/3378807567 conn(0x7f6ff80ce690 msgr2=0x7f6ff80e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:35.296 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:35.282 7f6ffead5700 1 -- 172.21.2.102:0/3378807567 shutdown_connections 2021-02-10T11:23:35.296 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:35.282 7f6ffead5700 1 -- 172.21.2.102:0/3378807567 wait complete. 2021-02-10T11:23:35.298 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:35.296 7f6ffead5700 1 Processor -- start 2021-02-10T11:23:35.298 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:35.296 7f6ffead5700 1 -- start start 2021-02-10T11:23:35.298 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:35.297 7f6ffead5700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6ff80b1970 0x7f6ff80b1d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:35.299 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:35.297 7f6ffead5700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6ff808acb0 0x7f6ff808b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:35.299 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:35.297 7f6ffead5700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6ff808b6d0 0x7f6ff81aac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:35.299 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:35.297 7f6ffead5700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ff808bba0 con 0x7f6ff80b1970 2021-02-10T11:23:35.299 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:35.297 7f6ffead5700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ff81ab280 con 0x7f6ff808acb0 2021-02-10T11:23:35.299 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:35.297 7f6ffead5700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ff81ab4a0 con 0x7f6ff808b6d0 2021-02-10T11:23:35.299 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:35.297 7f6ffc8f1700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6ff80b1970 0x7f6ff80b1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:35.299 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:35.297 7f6ffc8f1700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6ff80b1970 0x7f6ff80b1d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:56496/0 (socket says 172.21.2.102:56496) 2021-02-10T11:23:35.299 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:35.297 7f6ffc8f1700 1 -- 172.21.2.102:0/2070113006 learned_addr learned my addr 172.21.2.102:0/2070113006 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:23:35.299 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:35.297 7f6ffc8f1700 1 -- 172.21.2.102:0/2070113006 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6ff808b6d0 msgr2=0x7f6ff81aac60 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:23:35.299 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:35.297 7f6ffc8f1700 1 --2- 172.21.2.102:0/2070113006 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6ff808b6d0 0x7f6ff81aac60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:35.299 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:35.297 7f6ffc8f1700 1 -- 172.21.2.102:0/2070113006 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6ff808acb0 msgr2=0x7f6ff808b0b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:23:35.299 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:35.297 7f6ffc8f1700 1 --2- 172.21.2.102:0/2070113006 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6ff808acb0 0x7f6ff808b0b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:35.299 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:35.297 7f6ffc8f1700 1 -- 172.21.2.102:0/2070113006 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ff81ab6c0 con 0x7f6ff80b1970 2021-02-10T11:23:35.300 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:35.298 7f6ffc8f1700 1 --2- 172.21.2.102:0/2070113006 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6ff80b1970 0x7f6ff80b1d70 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 rx=0x7f6ff0009f40 tx=0x7f6ff000bd50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:35.300 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:35.298 7f6ff57fa700 1 -- 172.21.2.102:0/2070113006 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6ff0006240 con 0x7f6ff80b1970 2021-02-10T11:23:35.300 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:35.298 7f6ff57fa700 1 -- 172.21.2.102:0/2070113006 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ff00064c0 con 0x7f6ff80b1970 2021-02-10T11:23:35.300 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:35.298 7f6ff57fa700 1 -- 172.21.2.102:0/2070113006 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6ff00058e0 con 0x7f6ff80b1970 2021-02-10T11:23:35.300 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:35.298 7f6ffead5700 1 -- 172.21.2.102:0/2070113006 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ff81ab6c0 con 0x7f6ff80b1970 2021-02-10T11:23:35.300 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:35.298 7f6ffead5700 1 -- 172.21.2.102:0/2070113006 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ff808c0a0 con 0x7f6ff80b1970 2021-02-10T11:23:35.300 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:35.298 7f6ff57fa700 1 -- 172.21.2.102:0/2070113006 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f6ff0006ca0 con 0x7f6ff80b1970 2021-02-10T11:23:35.300 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:35.299 7f6ff57fa700 1 --2- 172.21.2.102:0/2070113006 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6fdc0186c0 0x7f6fdc01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:35.301 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:35.299 7f6ff57fa700 1 -- 172.21.2.102:0/2070113006 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f6ff00328e0 con 0x7f6ff80b1970 2021-02-10T11:23:35.301 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:35.299 7f6ff7fff700 1 --2- 172.21.2.102:0/2070113006 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6fdc0186c0 0x7f6fdc01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:35.301 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:35.299 7f6ff7fff700 1 --2- 172.21.2.102:0/2070113006 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6fdc0186c0 0x7f6fdc01abc0 crc :-1 s=READY pgs=584 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:35.328 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:35.330 7f6ffead5700 1 -- 172.21.2.102:0/2070113006 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ff8107180 con 0x7f6ff80b1970 2021-02-10T11:23:35.330 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:35.333 7f6ff57fa700 1 -- 172.21.2.102:0/2070113006 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f6ff0000d30 con 0x7f6ff80b1970 2021-02-10T11:23:35.458 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:35.460 7f6ffead5700 1 -- 172.21.2.102:0/2070113006 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6ff808bb20 con 0x7f6ff80b1970 2021-02-10T11:23:35.459 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:35.461 7f6ff57fa700 1 -- 172.21.2.102:0/2070113006 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+2099 (secure 0 0 0) 0x7f6ff0006ca0 con 0x7f6ff80b1970 2021-02-10T11:23:35.459 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:23:35.459 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":30,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7557,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.102:6835/1941781958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6834","nonce":1941781958},{"type":"v1","addr":"172.21.2.102:6835","nonce":1941781958}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":26}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:23:18.002680","modified":"2021-02-10 11:23:20.009191","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5606},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5606":{"gid":5606,"name":"b","rank":0,"incarnation":27,"state":"up:active","state_seq":3,"addr":"172.21.2.113:6833/1872558991","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6832","nonce":1872558991},{"type":"v1","addr":"172.21.2.113:6833","nonce":1872558991}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":4}]} 2021-02-10T11:23:35.459 INFO:teuthology.orchestra.run.gibba002.stderr:dumped fsmap epoch 30 2021-02-10T11:23:35.461 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:35.462 7f6ffead5700 1 -- 172.21.2.102:0/2070113006 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6fdc0186c0 msgr2=0x7f6fdc01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:35.461 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:35.462 7f6ffead5700 1 --2- 172.21.2.102:0/2070113006 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6fdc0186c0 0x7f6fdc01abc0 crc :-1 s=READY pgs=584 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:35.461 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:35.462 7f6ffead5700 1 -- 172.21.2.102:0/2070113006 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6ff80b1970 msgr2=0x7f6ff80b1d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:35.461 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:35.462 7f6ffead5700 1 --2- 172.21.2.102:0/2070113006 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6ff80b1970 0x7f6ff80b1d70 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 rx=0x7f6ff0009f40 tx=0x7f6ff000bd50).stop 2021-02-10T11:23:35.461 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:35.463 7f6ffead5700 1 -- 172.21.2.102:0/2070113006 shutdown_connections 2021-02-10T11:23:35.461 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:35.463 7f6ffead5700 1 --2- 172.21.2.102:0/2070113006 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6ff808b6d0 0x7f6ff81aac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:35.461 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:35.463 7f6ffead5700 1 --2- 172.21.2.102:0/2070113006 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6fdc0186c0 0x7f6fdc01abc0 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:35.461 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:35.463 7f6ffead5700 1 --2- 172.21.2.102:0/2070113006 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6ff808acb0 0x7f6ff808b0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:35.461 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:35.463 7f6ffead5700 1 --2- 172.21.2.102:0/2070113006 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6ff80b1970 0x7f6ff80b1d70 unknown :-1 s=CLOSED pgs=1231 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:35.461 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:35.463 7f6ffead5700 1 -- 172.21.2.102:0/2070113006 >> 172.21.2.102:0/2070113006 conn(0x7f6ff81ac620 msgr2=0x7f6ff80893d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:35.462 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:35.463 7f6ffead5700 1 -- 172.21.2.102:0/2070113006 shutdown_connections 2021-02-10T11:23:35.462 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:35.463 7f6ffead5700 1 -- 172.21.2.102:0/2070113006 wait complete. 2021-02-10T11:23:35.671 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 30, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:23:18.002680', 'modified': '2021-02-10 11:23:20.009191', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5606}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5606': {'gid': 5606, 'name': 'b', 'rank': 0, 'incarnation': 27, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.113:6833/1872558991', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.113:6832', 'nonce': 1872558991}, {'type': 'v1', 'addr': '172.21.2.113:6833', 'nonce': 1872558991}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [9], 'metadata_pool': 8, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:23:35.671 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:23:35.672 DEBUG:teuthology.orchestra.run.gibba013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:23:36.072 INFO:teuthology.orchestra.run.gibba013.stdout:{ 2021-02-10T11:23:36.072 INFO:teuthology.orchestra.run.gibba013.stdout: "cluster_fsid": "68db30ea-dbce-4aef-a6df-f01c6fea0513", 2021-02-10T11:23:36.072 INFO:teuthology.orchestra.run.gibba013.stdout: "whoami": 0, 2021-02-10T11:23:36.072 INFO:teuthology.orchestra.run.gibba013.stdout: "id": 5606, 2021-02-10T11:23:36.072 INFO:teuthology.orchestra.run.gibba013.stdout: "want_state": "up:active", 2021-02-10T11:23:36.072 INFO:teuthology.orchestra.run.gibba013.stdout: "state": "up:active", 2021-02-10T11:23:36.073 INFO:teuthology.orchestra.run.gibba013.stdout: "rank_uptime": 18.069863745999999, 2021-02-10T11:23:36.073 INFO:teuthology.orchestra.run.gibba013.stdout: "mdsmap_epoch": 30, 2021-02-10T11:23:36.073 INFO:teuthology.orchestra.run.gibba013.stdout: "osdmap_epoch": 48, 2021-02-10T11:23:36.073 INFO:teuthology.orchestra.run.gibba013.stdout: "osdmap_epoch_barrier": 48, 2021-02-10T11:23:36.073 INFO:teuthology.orchestra.run.gibba013.stdout: "uptime": 23.711189833999999 2021-02-10T11:23:36.073 INFO:teuthology.orchestra.run.gibba013.stdout:} 2021-02-10T11:23:36.076 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "68db30ea-dbce-4aef-a6df-f01c6fea0513", "whoami": 0, "id": 5606, "want_state": "up:active", "state": "up:active", "rank_uptime": 18.069863745999999, "mdsmap_epoch": 30, "osdmap_epoch": 48, "osdmap_epoch_barrier": 48, "uptime": 23.711189833999999 } 2021-02-10T11:23:36.076 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2021-02-10T11:23:36.077 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2021-02-10T11:23:36.077 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.1 at ubuntu@gibba002.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.1... 2021-02-10T11:23:36.077 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:23:36.077 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec mkdir -p /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:23:36.411 DEBUG:teuthology.orchestra.run.gibba002:> sudo modprobe fuse 2021-02-10T11:23:36.756 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:23:36.756 DEBUG:teuthology.orchestra.run.gibba002:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:23:37.100 INFO:teuthology.orchestra.run.gibba002.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:23:37.100 INFO:teuthology.orchestra.run.gibba002.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:23:37.101 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:23:37.101 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:23:37.101 DEBUG:teuthology.orchestra.run.gibba002:> ls /sys/fs/fuse/connections 2021-02-10T11:23:37.435 INFO:teuthology.orchestra.run.gibba002.stdout:43 2021-02-10T11:23:37.435 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [43] 2021-02-10T11:23:37.436 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.1 /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:23:37.438 DEBUG:teuthology.orchestra.run.gibba002:> sudo modprobe fuse 2021-02-10T11:23:37.781 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:23:37.782 DEBUG:teuthology.orchestra.run.gibba002:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:23:37.846 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba002.stderr:ceph-fuse[149956]: starting ceph client 2021-02-10T11:23:37.847 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba002.stderr:2021-02-10 11:23:37.848 7f8c7be3af80 -1 init, newargv = 0x56212db9bcf0 newargc=9 2021-02-10T11:23:37.852 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba002.stderr:ceph-fuse[149956]: starting fuse 2021-02-10T11:23:38.124 INFO:teuthology.orchestra.run.gibba002.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:23:38.124 INFO:teuthology.orchestra.run.gibba002.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:23:38.125 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:23:38.125 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:23:38.126 DEBUG:teuthology.orchestra.run.gibba002:> ls /sys/fs/fuse/connections 2021-02-10T11:23:38.460 INFO:teuthology.orchestra.run.gibba002.stdout:43 2021-02-10T11:23:38.460 INFO:teuthology.orchestra.run.gibba002.stdout:44 2021-02-10T11:23:38.460 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43, 44] 2021-02-10T11:23:38.460 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:23:38.460 DEBUG:teuthology.orchestra.run.gibba002:> sudo python3 -c ' 2021-02-10T11:23:38.461 DEBUG:teuthology.orchestra.run.gibba002:> import glob 2021-02-10T11:23:38.461 DEBUG:teuthology.orchestra.run.gibba002:> import re 2021-02-10T11:23:38.461 DEBUG:teuthology.orchestra.run.gibba002:> import os 2021-02-10T11:23:38.461 DEBUG:teuthology.orchestra.run.gibba002:> import subprocess 2021-02-10T11:23:38.461 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:23:38.461 DEBUG:teuthology.orchestra.run.gibba002:> def find_socket(client_name): 2021-02-10T11:23:38.461 DEBUG:teuthology.orchestra.run.gibba002:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2021-02-10T11:23:38.461 DEBUG:teuthology.orchestra.run.gibba002:> files = glob.glob(asok_path) 2021-02-10T11:23:38.461 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:23:38.461 DEBUG:teuthology.orchestra.run.gibba002:> # Given a non-glob path, it better be there 2021-02-10T11:23:38.461 DEBUG:teuthology.orchestra.run.gibba002:> if "*" not in asok_path: 2021-02-10T11:23:38.461 DEBUG:teuthology.orchestra.run.gibba002:> assert(len(files) == 1) 2021-02-10T11:23:38.461 DEBUG:teuthology.orchestra.run.gibba002:> return files[0] 2021-02-10T11:23:38.461 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:23:38.461 DEBUG:teuthology.orchestra.run.gibba002:> for f in files: 2021-02-10T11:23:38.462 DEBUG:teuthology.orchestra.run.gibba002:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2021-02-10T11:23:38.462 DEBUG:teuthology.orchestra.run.gibba002:> if os.path.exists("/proc/{0}".format(pid)): 2021-02-10T11:23:38.462 DEBUG:teuthology.orchestra.run.gibba002:> return f 2021-02-10T11:23:38.462 DEBUG:teuthology.orchestra.run.gibba002:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2021-02-10T11:23:38.462 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:23:38.462 DEBUG:teuthology.orchestra.run.gibba002:> print(find_socket("client.1")) 2021-02-10T11:23:38.462 DEBUG:teuthology.orchestra.run.gibba002:> ' 2021-02-10T11:23:38.823 INFO:teuthology.orchestra.run.gibba002.stdout:/var/run/ceph/ceph-client.1.149956.asok 2021-02-10T11:23:38.826 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.149956.asok 2021-02-10T11:23:38.826 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:23:38.826 DEBUG:teuthology.orchestra.run.gibba002:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.149956.asok status 2021-02-10T11:23:39.217 INFO:teuthology.orchestra.run.gibba002.stdout:{ 2021-02-10T11:23:39.217 INFO:teuthology.orchestra.run.gibba002.stdout: "metadata": { 2021-02-10T11:23:39.217 INFO:teuthology.orchestra.run.gibba002.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:23:39.217 INFO:teuthology.orchestra.run.gibba002.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:23:39.217 INFO:teuthology.orchestra.run.gibba002.stdout: "entity_id": "1", 2021-02-10T11:23:39.217 INFO:teuthology.orchestra.run.gibba002.stdout: "hostname": "gibba002", 2021-02-10T11:23:39.217 INFO:teuthology.orchestra.run.gibba002.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2021-02-10T11:23:39.217 INFO:teuthology.orchestra.run.gibba002.stdout: "pid": "149956", 2021-02-10T11:23:39.217 INFO:teuthology.orchestra.run.gibba002.stdout: "root": "/" 2021-02-10T11:23:39.217 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:39.217 INFO:teuthology.orchestra.run.gibba002.stdout: "dentry_count": 0, 2021-02-10T11:23:39.218 INFO:teuthology.orchestra.run.gibba002.stdout: "dentry_pinned_count": 0, 2021-02-10T11:23:39.218 INFO:teuthology.orchestra.run.gibba002.stdout: "id": 7755, 2021-02-10T11:23:39.218 INFO:teuthology.orchestra.run.gibba002.stdout: "inst": { 2021-02-10T11:23:39.218 INFO:teuthology.orchestra.run.gibba002.stdout: "name": { 2021-02-10T11:23:39.218 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "client", 2021-02-10T11:23:39.218 INFO:teuthology.orchestra.run.gibba002.stdout: "num": 7755 2021-02-10T11:23:39.218 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:39.218 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": { 2021-02-10T11:23:39.218 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:23:39.218 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:0", 2021-02-10T11:23:39.218 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 2653736768 2021-02-10T11:23:39.218 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:23:39.218 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:39.219 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": { 2021-02-10T11:23:39.219 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:23:39.219 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:0", 2021-02-10T11:23:39.219 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 2653736768 2021-02-10T11:23:39.219 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:39.219 INFO:teuthology.orchestra.run.gibba002.stdout: "inst_str": "client.7755 172.21.2.102:0/2653736768", 2021-02-10T11:23:39.219 INFO:teuthology.orchestra.run.gibba002.stdout: "addr_str": "172.21.2.102:0/2653736768", 2021-02-10T11:23:39.219 INFO:teuthology.orchestra.run.gibba002.stdout: "inode_count": 1, 2021-02-10T11:23:39.219 INFO:teuthology.orchestra.run.gibba002.stdout: "mds_epoch": 30, 2021-02-10T11:23:39.219 INFO:teuthology.orchestra.run.gibba002.stdout: "osd_epoch": 48, 2021-02-10T11:23:39.219 INFO:teuthology.orchestra.run.gibba002.stdout: "osd_epoch_barrier": 0, 2021-02-10T11:23:39.219 INFO:teuthology.orchestra.run.gibba002.stdout: "blacklisted": false 2021-02-10T11:23:39.219 INFO:teuthology.orchestra.run.gibba002.stdout:} 2021-02-10T11:23:39.221 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:23:39.221 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:23:39.221 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:23:39.559 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:23:39.560 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2021-02-10T11:23:39.560 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:23:39.560 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:23:39.906 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2021-02-10T11:23:40.338 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:40.324 7f76731dc700 1 Processor -- start 2021-02-10T11:23:40.338 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:40.324 7f76731dc700 1 -- start start 2021-02-10T11:23:40.338 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:40.324 7f76731dc700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f766c0d1870 0x7f766c1b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:40.338 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:40.324 7f76731dc700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f766c0cff70 0x7f766c109130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:40.338 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:40.324 7f76731dc700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f766c0fe1c0 con 0x7f766c1b0ae0 2021-02-10T11:23:40.338 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:40.324 7f76731dc700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f766c109ad0 con 0x7f766c0d1870 2021-02-10T11:23:40.338 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:40.324 7f76731dc700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f766c109750 con 0x7f766c0cff70 2021-02-10T11:23:40.338 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:40.324 7f7670ff8700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f766c0d1870 0x7f766c1b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:40.338 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:40.324 7f766bfff700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f766c1b0ae0 0x7f766c108940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:48740/0 (socket says 172.21.2.102:48740) 2021-02-10T11:23:40.339 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:40.324 7f766bfff700 1 -- 172.21.2.102:0/3767816637 learned_addr learned my addr 172.21.2.102:0/3767816637 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:23:40.339 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:40.325 7f766b7fe700 1 -- 172.21.2.102:0/3767816637 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2551545356 0 0) 0x7f76600009b0 con 0x7f766c1b0ae0 2021-02-10T11:23:40.339 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:40.325 7f766b7fe700 1 -- 172.21.2.102:0/3767816637 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7654002fe0 con 0x7f766c1b0ae0 2021-02-10T11:23:40.339 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:40.325 7f76717f9700 1 --2- 172.21.2.102:0/3767816637 >> v2:172.21.2.113:3300/0 conn(0x7f766c0cff70 0x7f766c109130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:40.339 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:40.325 7f7670ff8700 1 -- 172.21.2.102:0/3767816637 >> v2:172.21.2.113:3300/0 conn(0x7f766c0cff70 msgr2=0x7f766c109130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:40.339 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:40.325 7f766b7fe700 1 -- 172.21.2.102:0/3767816637 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 42353140 0 0) 0x7f7660001010 con 0x7f766c1b0ae0 2021-02-10T11:23:40.339 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:40.325 7f7670ff8700 1 --2- 172.21.2.102:0/3767816637 >> v2:172.21.2.113:3300/0 conn(0x7f766c0cff70 0x7f766c109130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:40.339 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:40.325 7f7670ff8700 1 -- 172.21.2.102:0/3767816637 >> v1:172.21.2.102:6789/0 conn(0x7f766c1b0ae0 legacy=0x7f766c108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:40.339 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:40.325 7f7670ff8700 1 -- 172.21.2.102:0/3767816637 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f766c088020 con 0x7f766c0d1870 2021-02-10T11:23:40.339 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:40.325 7f76717f9700 1 --2- 172.21.2.102:0/3767816637 >> v2:172.21.2.113:3300/0 conn(0x7f766c0cff70 0x7f766c109130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:23:40.339 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:40.325 7f7670ff8700 1 --2- 172.21.2.102:0/3767816637 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f766c0d1870 0x7f766c1b04c0 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 rx=0x7f765c006650 tx=0x7f765c009a80).ready entity=mon.0 client_cookie=1b9dfcf4d41cdbcd server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:40.339 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:40.325 7f766b7fe700 1 -- 172.21.2.102:0/3767816637 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f765c00c0c0 con 0x7f766c0d1870 2021-02-10T11:23:40.339 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:40.325 7f766b7fe700 1 -- 172.21.2.102:0/3767816637 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f765c00c340 con 0x7f766c0d1870 2021-02-10T11:23:40.340 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:40.325 7f766b7fe700 1 -- 172.21.2.102:0/3767816637 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f765c00c860 con 0x7f766c0d1870 2021-02-10T11:23:40.340 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:40.325 7f76731dc700 1 -- 172.21.2.102:0/3767816637 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f766c0d1870 msgr2=0x7f766c1b04c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:40.340 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:40.325 7f76731dc700 1 --2- 172.21.2.102:0/3767816637 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f766c0d1870 0x7f766c1b04c0 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 rx=0x7f765c006650 tx=0x7f765c009a80).stop 2021-02-10T11:23:40.340 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:40.326 7f76731dc700 1 -- 172.21.2.102:0/3767816637 shutdown_connections 2021-02-10T11:23:40.340 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:40.326 7f76731dc700 1 --2- 172.21.2.102:0/3767816637 >> v2:172.21.2.113:3300/0 conn(0x7f766c0cff70 0x7f766c109130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:40.340 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:40.326 7f76731dc700 1 --2- 172.21.2.102:0/3767816637 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f766c0d1870 0x7f766c1b04c0 unknown :-1 s=CLOSED pgs=1236 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:40.340 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:40.326 7f76731dc700 1 -- 172.21.2.102:0/3767816637 >> 172.21.2.102:0/3767816637 conn(0x7f766c0ce420 msgr2=0x7f766c0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:40.340 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:40.326 7f76731dc700 1 -- 172.21.2.102:0/3767816637 shutdown_connections 2021-02-10T11:23:40.340 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:40.326 7f76731dc700 1 -- 172.21.2.102:0/3767816637 wait complete. 2021-02-10T11:23:40.340 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:40.341 7f76731dc700 1 Processor -- start 2021-02-10T11:23:40.340 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:40.341 7f76731dc700 1 -- start start 2021-02-10T11:23:40.341 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:40.341 7f76731dc700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f766c0b8070 0x7f766c0b77c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:40.342 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:40.341 7f76731dc700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f766c0ba4a0 0x7f766c0ba8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:40.342 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:40.341 7f76731dc700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f766c1a2fd0 0x7f766c1a33d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:40.342 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:40.341 7f76731dc700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f766c0baec0 con 0x7f766c0b8070 2021-02-10T11:23:40.342 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:40.341 7f76731dc700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f766c1a39f0 con 0x7f766c1a2fd0 2021-02-10T11:23:40.342 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:40.341 7f76731dc700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f766c1a3c10 con 0x7f766c0ba4a0 2021-02-10T11:23:40.342 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:40.342 7f7670ff8700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f766c0b8070 0x7f766c0b77c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:40.342 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:40.342 7f7670ff8700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f766c0b8070 0x7f766c0b77c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:56520/0 (socket says 172.21.2.102:56520) 2021-02-10T11:23:40.342 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:40.342 7f7670ff8700 1 -- 172.21.2.102:0/2956855561 learned_addr learned my addr 172.21.2.102:0/2956855561 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:23:40.342 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:40.342 7f7670ff8700 1 -- 172.21.2.102:0/2956855561 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f766c0ba4a0 msgr2=0x7f766c0ba8a0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:23:40.342 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:40.342 7f7670ff8700 1 --2- 172.21.2.102:0/2956855561 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f766c0ba4a0 0x7f766c0ba8a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:40.342 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:40.342 7f7670ff8700 1 -- 172.21.2.102:0/2956855561 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f766c1a2fd0 msgr2=0x7f766c1a33d0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:23:40.342 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:40.342 7f7670ff8700 1 --2- 172.21.2.102:0/2956855561 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f766c1a2fd0 0x7f766c1a33d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:40.343 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:40.342 7f7670ff8700 1 -- 172.21.2.102:0/2956855561 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f766c0ad700 con 0x7f766c0b8070 2021-02-10T11:23:40.343 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:40.342 7f7670ff8700 1 --2- 172.21.2.102:0/2956855561 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f766c0b8070 0x7f766c0b77c0 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 rx=0x7f765c009650 tx=0x7f765c00b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:40.343 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:40.342 7f76697fa700 1 -- 172.21.2.102:0/2956855561 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f765c00bd50 con 0x7f766c0b8070 2021-02-10T11:23:40.343 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:40.342 7f76697fa700 1 -- 172.21.2.102:0/2956855561 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f765c006030 con 0x7f766c0b8070 2021-02-10T11:23:40.343 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:40.342 7f76697fa700 1 -- 172.21.2.102:0/2956855561 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f765c005560 con 0x7f766c0b8070 2021-02-10T11:23:40.343 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:40.342 7f76731dc700 1 -- 172.21.2.102:0/2956855561 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f766c0ad700 con 0x7f766c0b8070 2021-02-10T11:23:40.343 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:40.342 7f76731dc700 1 -- 172.21.2.102:0/2956855561 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f766c0adc90 con 0x7f766c0b8070 2021-02-10T11:23:40.343 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:40.343 7f76697fa700 1 -- 172.21.2.102:0/2956855561 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f765c0052e0 con 0x7f766c0b8070 2021-02-10T11:23:40.343 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:40.343 7f76697fa700 1 --2- 172.21.2.102:0/2956855561 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f764c0186c0 0x7f764c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:40.343 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:40.343 7f76697fa700 1 -- 172.21.2.102:0/2956855561 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f765c032550 con 0x7f766c0b8070 2021-02-10T11:23:40.343 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:40.343 7f766bfff700 1 --2- 172.21.2.102:0/2956855561 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f764c0186c0 0x7f764c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:40.343 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:40.343 7f766bfff700 1 --2- 172.21.2.102:0/2956855561 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f764c0186c0 0x7f764c01abc0 crc :-1 s=READY pgs=585 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:40.371 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:40.374 7f76731dc700 1 -- 172.21.2.102:0/2956855561 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f766c0828e0 con 0x7f766c0b8070 2021-02-10T11:23:40.373 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:40.376 7f76697fa700 1 -- 172.21.2.102:0/2956855561 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f765c0093a0 con 0x7f766c0b8070 2021-02-10T11:23:40.510 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:40.513 7f76731dc700 1 -- 172.21.2.102:0/2956855561 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f766c0828e0 con 0x7f766c0b8070 2021-02-10T11:23:41.460 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:41.463 7f76697fa700 1 -- 172.21.2.102:0/2956855561 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v31) v1 ==== 136+0+0 (secure 0 0 0) 0x7f765c0052e0 con 0x7f766c0b8070 2021-02-10T11:23:41.484 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:41.486 7f76731dc700 1 -- 172.21.2.102:0/2956855561 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f764c0186c0 msgr2=0x7f764c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:41.484 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:41.486 7f76731dc700 1 --2- 172.21.2.102:0/2956855561 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f764c0186c0 0x7f764c01abc0 crc :-1 s=READY pgs=585 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:41.484 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:41.486 7f76731dc700 1 -- 172.21.2.102:0/2956855561 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f766c0b8070 msgr2=0x7f766c0b77c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:41.484 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:41.486 7f76731dc700 1 --2- 172.21.2.102:0/2956855561 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f766c0b8070 0x7f766c0b77c0 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 rx=0x7f765c009650 tx=0x7f765c00b960).stop 2021-02-10T11:23:41.484 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:41.487 7f76731dc700 1 -- 172.21.2.102:0/2956855561 shutdown_connections 2021-02-10T11:23:41.484 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:41.487 7f76731dc700 1 --2- 172.21.2.102:0/2956855561 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f764c0186c0 0x7f764c01abc0 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:41.484 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:41.487 7f76731dc700 1 --2- 172.21.2.102:0/2956855561 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f766c1a2fd0 0x7f766c1a33d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:41.484 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:41.487 7f76731dc700 1 --2- 172.21.2.102:0/2956855561 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f766c0ba4a0 0x7f766c0ba8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:41.484 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:41.487 7f76731dc700 1 --2- 172.21.2.102:0/2956855561 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f766c0b8070 0x7f766c0b77c0 unknown :-1 s=CLOSED pgs=1237 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:41.484 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:41.487 7f76731dc700 1 -- 172.21.2.102:0/2956855561 >> 172.21.2.102:0/2956855561 conn(0x7f766c0b58d0 msgr2=0x7f766c0b5010 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:41.485 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:41.487 7f76731dc700 1 -- 172.21.2.102:0/2956855561 shutdown_connections 2021-02-10T11:23:41.485 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:41.487 7f76731dc700 1 -- 172.21.2.102:0/2956855561 wait complete. 2021-02-10T11:23:41.492 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs volume ls 2021-02-10T11:23:41.918 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:41.904 7fe0c5dd8700 1 Processor -- start 2021-02-10T11:23:41.919 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:41.904 7fe0c5dd8700 1 -- start start 2021-02-10T11:23:41.919 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:41.905 7fe0c5dd8700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fe0c00e5470 0x7fe0c007e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:41.919 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:41.905 7fe0c5dd8700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe0c007f300 0x7fe0c007f700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:41.919 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:41.905 7fe0c5dd8700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0c007ed80 con 0x7fe0c00e5070 2021-02-10T11:23:41.919 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:41.905 7fe0c5dd8700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0c01b4010 con 0x7fe0c00e5470 2021-02-10T11:23:41.919 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:41.905 7fe0c5dd8700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0c00d1790 con 0x7fe0c007f300 2021-02-10T11:23:41.919 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:41.905 7fe0bf7fe700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fe0c00e5070 0x7fe0c01b3820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:48752/0 (socket says 172.21.2.102:48752) 2021-02-10T11:23:41.919 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:41.905 7fe0bf7fe700 1 -- 172.21.2.102:0/727144871 learned_addr learned my addr 172.21.2.102:0/727144871 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:23:41.919 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:41.905 7fe0be7fc700 1 -- 172.21.2.102:0/727144871 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3683099861 0 0) 0x7fe0b00010b0 con 0x7fe0c00e5070 2021-02-10T11:23:41.919 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:41.905 7fe0be7fc700 1 -- 172.21.2.102:0/727144871 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0ac002fe0 con 0x7fe0c00e5070 2021-02-10T11:23:41.920 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:41.905 7fe0beffd700 1 --2- 172.21.2.102:0/727144871 >> v2:172.21.2.113:3300/0 conn(0x7fe0c00e5470 0x7fe0c007e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:41.920 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:41.905 7fe0bffff700 1 --2- 172.21.2.102:0/727144871 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe0c007f300 0x7fe0c007f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:41.920 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:41.906 7fe0be7fc700 1 -- 172.21.2.102:0/727144871 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1635329453 0 0) 0x7fe0b00010b0 con 0x7fe0c00e5070 2021-02-10T11:23:41.920 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:41.906 7fe0be7fc700 1 -- 172.21.2.102:0/727144871 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe0c007f300 msgr2=0x7fe0c007f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:41.920 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:41.906 7fe0be7fc700 1 --2- 172.21.2.102:0/727144871 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe0c007f300 0x7fe0c007f700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:41.920 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:41.906 7fe0be7fc700 1 -- 172.21.2.102:0/727144871 >> v2:172.21.2.113:3300/0 conn(0x7fe0c00e5470 msgr2=0x7fe0c007e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:41.920 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:41.906 7fe0be7fc700 1 --2- 172.21.2.102:0/727144871 >> v2:172.21.2.113:3300/0 conn(0x7fe0c00e5470 0x7fe0c007e6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:41.920 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:41.906 7fe0be7fc700 1 -- 172.21.2.102:0/727144871 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0c00d19b0 con 0x7fe0c00e5070 2021-02-10T11:23:41.920 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:41.906 7fe0be7fc700 1 -- 172.21.2.102:0/727144871 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7fe0b00015b0 con 0x7fe0c00e5070 2021-02-10T11:23:41.920 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:41.906 7fe0be7fc700 1 -- 172.21.2.102:0/727144871 >> v1:172.21.2.102:6789/0 conn(0x7fe0c00e5070 legacy=0x7fe0c01b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:41.920 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:41.906 7fe0be7fc700 1 --2- 172.21.2.102:0/727144871 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe0ac003ac0 0x7fe0ac005f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:41.920 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:41.906 7fe0be7fc700 1 --2- 172.21.2.102:0/727144871 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe0ac0067d0 0x7fe0ac008cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:41.920 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:41.906 7fe0be7fc700 1 --2- 172.21.2.102:0/727144871 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe0ac009590 0x7fe0ac00bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:41.920 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:41.906 7fe0be7fc700 1 -- 172.21.2.102:0/727144871 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0ac00c360 con 0x7fe0ac0067d0 2021-02-10T11:23:41.921 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:41.906 7fe0be7fc700 1 -- 172.21.2.102:0/727144871 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0ac00c580 con 0x7fe0ac009590 2021-02-10T11:23:41.921 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:41.906 7fe0be7fc700 1 -- 172.21.2.102:0/727144871 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0ac0507b0 con 0x7fe0ac003ac0 2021-02-10T11:23:41.921 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:41.906 7fe0bffff700 1 --2- 172.21.2.102:0/727144871 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe0c007f300 0x7fe0c007f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:23:41.921 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:41.906 7fe0beffd700 1 --2- 172.21.2.102:0/727144871 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe0ac0067d0 0x7fe0ac008cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:41.921 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:41.906 7fe0bf7fe700 1 --2- 172.21.2.102:0/727144871 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe0ac003ac0 0x7fe0ac005f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:41.921 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:41.906 7fe0bffff700 1 --2- 172.21.2.102:0/727144871 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe0ac009590 0x7fe0ac00bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:41.921 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:41.906 7fe0beffd700 1 -- 172.21.2.102:0/727144871 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe0ac003ac0 msgr2=0x7fe0ac005f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:41.921 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:41.906 7fe0beffd700 1 --2- 172.21.2.102:0/727144871 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe0ac003ac0 0x7fe0ac005f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:41.921 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:41.906 7fe0beffd700 1 -- 172.21.2.102:0/727144871 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe0ac009590 msgr2=0x7fe0ac00bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:41.921 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:41.906 7fe0beffd700 1 --2- 172.21.2.102:0/727144871 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe0ac009590 0x7fe0ac00bab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:41.921 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:41.906 7fe0beffd700 1 -- 172.21.2.102:0/727144871 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0ac050a10 con 0x7fe0ac0067d0 2021-02-10T11:23:41.921 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:41.906 7fe0bf7fe700 1 --2- 172.21.2.102:0/727144871 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe0ac003ac0 0x7fe0ac005f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:23:41.922 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:41.906 7fe0beffd700 1 --2- 172.21.2.102:0/727144871 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe0ac0067d0 0x7fe0ac008cf0 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 rx=0x7fe0a8006360 tx=0x7fe0a8009c30).ready entity=mon.0 client_cookie=cdad234b8d1f39c9 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:41.922 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:41.906 7fe0be7fc700 1 -- 172.21.2.102:0/727144871 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe0a80009b0 con 0x7fe0ac0067d0 2021-02-10T11:23:41.922 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:41.906 7fe0be7fc700 1 -- 172.21.2.102:0/727144871 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0a8000d20 con 0x7fe0ac0067d0 2021-02-10T11:23:41.922 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:41.906 7fe0be7fc700 1 -- 172.21.2.102:0/727144871 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe0a8002220 con 0x7fe0ac0067d0 2021-02-10T11:23:41.922 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:41.906 7fe0c5dd8700 1 -- 172.21.2.102:0/727144871 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe0ac0067d0 msgr2=0x7fe0ac008cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:41.922 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:41.906 7fe0c5dd8700 1 --2- 172.21.2.102:0/727144871 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe0ac0067d0 0x7fe0ac008cf0 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 rx=0x7fe0a8006360 tx=0x7fe0a8009c30).stop 2021-02-10T11:23:41.922 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:41.906 7fe0bffff700 1 -- 172.21.2.102:0/727144871 reap_dead start 2021-02-10T11:23:41.922 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:41.907 7fe0c5dd8700 1 -- 172.21.2.102:0/727144871 shutdown_connections 2021-02-10T11:23:41.922 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:41.907 7fe0c5dd8700 1 -- 172.21.2.102:0/727144871 >> 172.21.2.102:0/727144871 conn(0x7fe0c00ce690 msgr2=0x7fe0c00e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:41.922 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:41.907 7fe0c5dd8700 1 -- 172.21.2.102:0/727144871 shutdown_connections 2021-02-10T11:23:41.922 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:41.907 7fe0c5dd8700 1 -- 172.21.2.102:0/727144871 wait complete. 2021-02-10T11:23:41.922 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:41.922 7fe0c5dd8700 1 Processor -- start 2021-02-10T11:23:41.924 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:41.922 7fe0c5dd8700 1 -- start start 2021-02-10T11:23:41.924 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:41.922 7fe0c5dd8700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe0c019f960 0x7fe0c019fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:41.924 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:41.922 7fe0c5dd8700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe0c00a4e30 0x7fe0c00a5230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:41.924 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:41.922 7fe0c5dd8700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe0c00a5b00 0x7fe0c00a5f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:41.924 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:41.922 7fe0c5dd8700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0c00a6520 con 0x7fe0c00a4e30 2021-02-10T11:23:41.924 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:41.922 7fe0c5dd8700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0c00a6710 con 0x7fe0c019f960 2021-02-10T11:23:41.925 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:41.922 7fe0c5dd8700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0c00a6930 con 0x7fe0c00a5b00 2021-02-10T11:23:41.925 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:41.922 7fe0beffd700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe0c00a4e30 0x7fe0c00a5230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:41.925 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:41.922 7fe0beffd700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe0c00a4e30 0x7fe0c00a5230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:56544/0 (socket says 172.21.2.102:56544) 2021-02-10T11:23:41.925 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:41.922 7fe0beffd700 1 -- 172.21.2.102:0/2052879187 learned_addr learned my addr 172.21.2.102:0/2052879187 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:23:41.925 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:41.923 7fe0bf7fe700 1 --2- 172.21.2.102:0/2052879187 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe0c019f960 0x7fe0c019fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:41.925 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:41.923 7fe0bffff700 1 --2- 172.21.2.102:0/2052879187 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe0c00a5b00 0x7fe0c00a5f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:41.925 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:41.923 7fe0beffd700 1 -- 172.21.2.102:0/2052879187 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe0c00a5b00 msgr2=0x7fe0c00a5f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:41.925 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:41.923 7fe0beffd700 1 --2- 172.21.2.102:0/2052879187 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe0c00a5b00 0x7fe0c00a5f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:41.925 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:41.923 7fe0beffd700 1 -- 172.21.2.102:0/2052879187 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe0c019f960 msgr2=0x7fe0c019fd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:41.925 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:41.923 7fe0beffd700 1 --2- 172.21.2.102:0/2052879187 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe0c019f960 0x7fe0c019fd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:41.925 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:41.923 7fe0beffd700 1 -- 172.21.2.102:0/2052879187 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0c00a6b50 con 0x7fe0c00a4e30 2021-02-10T11:23:41.925 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:41.923 7fe0bf7fe700 1 --2- 172.21.2.102:0/2052879187 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe0c019f960 0x7fe0c019fd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:23:41.925 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:41.923 7fe0beffd700 1 --2- 172.21.2.102:0/2052879187 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe0c00a4e30 0x7fe0c00a5230 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 rx=0x7fe0a800b610 tx=0x7fe0a800b9b0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:41.926 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:41.923 7fe0b7fff700 1 -- 172.21.2.102:0/2052879187 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe0a800bda0 con 0x7fe0c00a4e30 2021-02-10T11:23:41.926 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:41.923 7fe0b7fff700 1 -- 172.21.2.102:0/2052879187 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0a8002030 con 0x7fe0c00a4e30 2021-02-10T11:23:41.926 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:41.923 7fe0b7fff700 1 -- 172.21.2.102:0/2052879187 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe0a8002550 con 0x7fe0c00a4e30 2021-02-10T11:23:41.926 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:41.923 7fe0c5dd8700 1 -- 172.21.2.102:0/2052879187 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0c00a6b50 con 0x7fe0c00a4e30 2021-02-10T11:23:41.926 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:41.923 7fe0c5dd8700 1 -- 172.21.2.102:0/2052879187 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0c0091aa0 con 0x7fe0c00a4e30 2021-02-10T11:23:41.926 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:41.924 7fe0b7fff700 1 -- 172.21.2.102:0/2052879187 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fe0a8000c90 con 0x7fe0c00a4e30 2021-02-10T11:23:41.926 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:41.924 7fe0b7fff700 1 --2- 172.21.2.102:0/2052879187 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe0a40186c0 0x7fe0a401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:41.926 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:41.924 7fe0b7fff700 1 -- 172.21.2.102:0/2052879187 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fe0a80325c0 con 0x7fe0c00a4e30 2021-02-10T11:23:41.926 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:41.924 7fe0bf7fe700 1 --2- 172.21.2.102:0/2052879187 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe0a40186c0 0x7fe0a401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:41.926 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:41.924 7fe0bf7fe700 1 --2- 172.21.2.102:0/2052879187 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe0a40186c0 0x7fe0a401abc0 crc :-1 s=READY pgs=586 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:41.952 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:41.955 7fe0c5dd8700 1 -- 172.21.2.102:0/2052879187 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0c0084ad0 con 0x7fe0c00a4e30 2021-02-10T11:23:41.959 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:41.957 7fe0b7fff700 1 -- 172.21.2.102:0/2052879187 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fe0a8009550 con 0x7fe0c00a4e30 2021-02-10T11:23:42.090 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:42.093 7fe0c5dd8700 1 -- 172.21.2.102:0/2052879187 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs volume ls", "target": ["mgr", ""]}) v1 -- 0x7fe0c0091aa0 con 0x7fe0a40186c0 2021-02-10T11:23:42.091 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:42.094 7fe0b7fff700 1 -- 172.21.2.102:0/2052879187 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+41 (crc 0 0 0) 0x7fe0b00010b0 con 0x7fe0a40186c0 2021-02-10T11:23:42.091 INFO:teuthology.orchestra.run.gibba002.stdout:[ 2021-02-10T11:23:42.091 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:42.091 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "cephfs" 2021-02-10T11:23:42.091 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:23:42.091 INFO:teuthology.orchestra.run.gibba002.stdout:] 2021-02-10T11:23:42.092 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:42.095 7fe0c5dd8700 1 -- 172.21.2.102:0/2052879187 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe0a40186c0 msgr2=0x7fe0a401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:42.092 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:42.095 7fe0c5dd8700 1 --2- 172.21.2.102:0/2052879187 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe0a40186c0 0x7fe0a401abc0 crc :-1 s=READY pgs=586 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:42.092 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:42.095 7fe0c5dd8700 1 -- 172.21.2.102:0/2052879187 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe0c00a4e30 msgr2=0x7fe0c00a5230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:42.093 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:42.095 7fe0c5dd8700 1 --2- 172.21.2.102:0/2052879187 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe0c00a4e30 0x7fe0c00a5230 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 rx=0x7fe0a800b610 tx=0x7fe0a800b9b0).stop 2021-02-10T11:23:42.093 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:42.095 7fe0c5dd8700 1 -- 172.21.2.102:0/2052879187 shutdown_connections 2021-02-10T11:23:42.093 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:42.095 7fe0c5dd8700 1 --2- 172.21.2.102:0/2052879187 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe0c00a5b00 0x7fe0c00a5f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:42.093 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:42.095 7fe0c5dd8700 1 --2- 172.21.2.102:0/2052879187 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe0c00a4e30 0x7fe0c00a5230 unknown :-1 s=CLOSED pgs=1240 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:42.093 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:42.095 7fe0c5dd8700 1 --2- 172.21.2.102:0/2052879187 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe0a40186c0 0x7fe0a401abc0 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:42.093 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:42.095 7fe0c5dd8700 1 --2- 172.21.2.102:0/2052879187 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe0c019f960 0x7fe0c019fd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:42.093 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:42.095 7fe0c5dd8700 1 -- 172.21.2.102:0/2052879187 >> 172.21.2.102:0/2052879187 conn(0x7fe0c00a3580 msgr2=0x7fe0c00bd230 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:42.093 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:42.096 7fe0c5dd8700 1 -- 172.21.2.102:0/2052879187 shutdown_connections 2021-02-10T11:23:42.094 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:42.096 7fe0c5dd8700 1 -- 172.21.2.102:0/2052879187 wait complete. 2021-02-10T11:23:42.101 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config set mon mon_allow_pool_delete True 2021-02-10T11:23:42.533 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:42.520 7f0e25866700 1 Processor -- start 2021-02-10T11:23:42.534 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:42.520 7f0e25866700 1 -- start start 2021-02-10T11:23:42.534 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:42.520 7f0e25866700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f0e201b0470 0x7f0e201b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:42.534 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:42.520 7f0e25866700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0e200ef610 0x7f0e200efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:42.534 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:42.520 7f0e25866700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e200f0b50 con 0x7f0e200f0030 2021-02-10T11:23:42.534 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:42.520 7f0e25866700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e200e52e0 con 0x7f0e201b0470 2021-02-10T11:23:42.534 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:42.520 7f0e25866700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e200e54d0 con 0x7f0e200ef610 2021-02-10T11:23:42.534 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:42.520 7f0e1f574700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f0e201b0470 0x7f0e201b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:42.534 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:42.520 7f0e1f574700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f0e201b0470 0x7f0e201b0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:56548/0 (socket says 172.21.2.102:56548) 2021-02-10T11:23:42.534 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:42.520 7f0e1f574700 1 -- 172.21.2.102:0/754018947 learned_addr learned my addr 172.21.2.102:0/754018947 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:23:42.534 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:42.521 7f0e1f574700 1 -- 172.21.2.102:0/754018947 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0e200ef610 msgr2=0x7f0e200efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:42.534 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:42.521 7f0e1f574700 1 --2- 172.21.2.102:0/754018947 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0e200ef610 0x7f0e200efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:42.534 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:42.521 7f0e1f574700 1 -- 172.21.2.102:0/754018947 >> v1:172.21.2.102:6789/0 conn(0x7f0e200f0030 legacy=0x7f0e200f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:42.535 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:42.521 7f0e1f574700 1 -- 172.21.2.102:0/754018947 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e200f0430 con 0x7f0e201b0470 2021-02-10T11:23:42.535 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:42.521 7f0e1f574700 1 --2- 172.21.2.102:0/754018947 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0e201b0470 0x7f0e201b0870 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 rx=0x7f0e10006700 tx=0x7f0e10009b80).ready entity=mon.0 client_cookie=36e14ae92675df97 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:42.535 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:42.521 7f0e1e572700 1 -- 172.21.2.102:0/754018947 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0e1000c0c0 con 0x7f0e201b0470 2021-02-10T11:23:42.535 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:42.521 7f0e1e572700 1 -- 172.21.2.102:0/754018947 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e1000c340 con 0x7f0e201b0470 2021-02-10T11:23:42.535 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:42.521 7f0e1e572700 1 -- 172.21.2.102:0/754018947 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0e1000c860 con 0x7f0e201b0470 2021-02-10T11:23:42.535 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:42.521 7f0e25866700 1 -- 172.21.2.102:0/754018947 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0e201b0470 msgr2=0x7f0e201b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:42.535 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:42.521 7f0e25866700 1 --2- 172.21.2.102:0/754018947 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0e201b0470 0x7f0e201b0870 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 rx=0x7f0e10006700 tx=0x7f0e10009b80).stop 2021-02-10T11:23:42.535 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:42.522 7f0e25866700 1 -- 172.21.2.102:0/754018947 shutdown_connections 2021-02-10T11:23:42.535 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:42.522 7f0e25866700 1 --2- 172.21.2.102:0/754018947 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0e200ef610 0x7f0e200efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:42.535 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:42.522 7f0e25866700 1 --2- 172.21.2.102:0/754018947 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0e201b0470 0x7f0e201b0870 unknown :-1 s=CLOSED pgs=1242 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:42.535 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:42.522 7f0e25866700 1 -- 172.21.2.102:0/754018947 >> 172.21.2.102:0/754018947 conn(0x7f0e200ce3e0 msgr2=0x7f0e200e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:42.535 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:42.522 7f0e25866700 1 -- 172.21.2.102:0/754018947 shutdown_connections 2021-02-10T11:23:42.535 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:42.522 7f0e25866700 1 -- 172.21.2.102:0/754018947 wait complete. 2021-02-10T11:23:42.537 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:42.537 7f0e25866700 1 Processor -- start 2021-02-10T11:23:42.537 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:42.537 7f0e25866700 1 -- start start 2021-02-10T11:23:42.537 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:42.537 7f0e25866700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0e200acc50 0x7f0e20098de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:42.537 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:42.537 7f0e25866700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0e200eec20 0x7f0e200ef020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:42.538 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:42.537 7f0e25866700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0e200ef640 0x7f0e201a0c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:42.538 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:42.537 7f0e25866700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e20090cf0 con 0x7f0e200acc50 2021-02-10T11:23:42.538 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:42.537 7f0e25866700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e200ad050 con 0x7f0e200ef640 2021-02-10T11:23:42.538 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:42.537 7f0e25866700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e201a1250 con 0x7f0e200eec20 2021-02-10T11:23:42.538 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:42.537 7f0e1f574700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0e200acc50 0x7f0e20098de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:42.538 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:42.537 7f0e1f574700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0e200acc50 0x7f0e20098de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:56552/0 (socket says 172.21.2.102:56552) 2021-02-10T11:23:42.538 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:42.537 7f0e1f574700 1 -- 172.21.2.102:0/3507891700 learned_addr learned my addr 172.21.2.102:0/3507891700 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:23:42.538 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:42.538 7f0e1f574700 1 -- 172.21.2.102:0/3507891700 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0e200eec20 msgr2=0x7f0e200ef020 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:23:42.538 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:42.538 7f0e1f574700 1 --2- 172.21.2.102:0/3507891700 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0e200eec20 0x7f0e200ef020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:42.538 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:42.538 7f0e1f574700 1 -- 172.21.2.102:0/3507891700 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0e200ef640 msgr2=0x7f0e201a0c30 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:23:42.538 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:42.538 7f0e1f574700 1 --2- 172.21.2.102:0/3507891700 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0e200ef640 0x7f0e201a0c30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:42.538 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:42.538 7f0e1f574700 1 -- 172.21.2.102:0/3507891700 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e201a1470 con 0x7f0e200acc50 2021-02-10T11:23:42.538 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:42.538 7f0e1f574700 1 --2- 172.21.2.102:0/3507891700 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0e200acc50 0x7f0e20098de0 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 rx=0x7f0e1000b5e0 tx=0x7f0e1000b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:42.539 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:42.538 7f0e17fff700 1 -- 172.21.2.102:0/3507891700 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0e1000bd70 con 0x7f0e200acc50 2021-02-10T11:23:42.539 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:42.538 7f0e17fff700 1 -- 172.21.2.102:0/3507891700 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e10002030 con 0x7f0e200acc50 2021-02-10T11:23:42.539 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:42.538 7f0e17fff700 1 -- 172.21.2.102:0/3507891700 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0e100024d0 con 0x7f0e200acc50 2021-02-10T11:23:42.539 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:42.538 7f0e25866700 1 -- 172.21.2.102:0/3507891700 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e201a1470 con 0x7f0e200acc50 2021-02-10T11:23:42.539 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:42.538 7f0e25866700 1 -- 172.21.2.102:0/3507891700 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e201a1470 con 0x7f0e200acc50 2021-02-10T11:23:42.539 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:42.538 7f0e17fff700 1 -- 172.21.2.102:0/3507891700 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f0e10000c90 con 0x7f0e200acc50 2021-02-10T11:23:42.539 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:42.538 7f0e17fff700 1 --2- 172.21.2.102:0/3507891700 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0e04018700 0x7f0e0401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:42.539 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:42.538 7f0e17fff700 1 -- 172.21.2.102:0/3507891700 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f0e10032610 con 0x7f0e200acc50 2021-02-10T11:23:42.539 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:42.538 7f0e1ed73700 1 --2- 172.21.2.102:0/3507891700 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0e04018700 0x7f0e0401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:42.539 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:42.539 7f0e1ed73700 1 --2- 172.21.2.102:0/3507891700 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0e04018700 0x7f0e0401abc0 crc :-1 s=READY pgs=587 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:42.567 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:42.570 7f0e25866700 1 -- 172.21.2.102:0/3507891700 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e200edd10 con 0x7f0e200acc50 2021-02-10T11:23:42.569 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:42.572 7f0e17fff700 1 -- 172.21.2.102:0/3507891700 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f0e1001f4d0 con 0x7f0e200acc50 2021-02-10T11:23:42.717 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:42.720 7f0e25866700 1 -- 172.21.2.102:0/3507891700 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f0e200e52e0 con 0x7f0e200acc50 2021-02-10T11:23:42.718 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:42.720 7f0e17fff700 1 -- 172.21.2.102:0/3507891700 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v6)=0 v6) v1 ==== 122+0+0 (secure 0 0 0) 0x7f0e10000c90 con 0x7f0e200acc50 2021-02-10T11:23:42.720 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:42.722 7f0e25866700 1 -- 172.21.2.102:0/3507891700 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0e04018700 msgr2=0x7f0e0401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:42.720 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:42.722 7f0e25866700 1 --2- 172.21.2.102:0/3507891700 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0e04018700 0x7f0e0401abc0 crc :-1 s=READY pgs=587 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:42.720 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:42.722 7f0e25866700 1 -- 172.21.2.102:0/3507891700 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0e200acc50 msgr2=0x7f0e20098de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:42.720 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:42.722 7f0e25866700 1 --2- 172.21.2.102:0/3507891700 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0e200acc50 0x7f0e20098de0 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 rx=0x7f0e1000b5e0 tx=0x7f0e1000b980).stop 2021-02-10T11:23:42.720 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:42.722 7f0e25866700 1 -- 172.21.2.102:0/3507891700 shutdown_connections 2021-02-10T11:23:42.720 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:42.722 7f0e25866700 1 --2- 172.21.2.102:0/3507891700 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0e04018700 0x7f0e0401abc0 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:42.720 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:42.722 7f0e25866700 1 --2- 172.21.2.102:0/3507891700 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0e200ef640 0x7f0e201a0c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:42.720 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:42.722 7f0e25866700 1 --2- 172.21.2.102:0/3507891700 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0e200eec20 0x7f0e200ef020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:42.720 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:42.722 7f0e25866700 1 --2- 172.21.2.102:0/3507891700 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0e200acc50 0x7f0e20098de0 unknown :-1 s=CLOSED pgs=1243 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:42.720 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:42.722 7f0e25866700 1 -- 172.21.2.102:0/3507891700 >> 172.21.2.102:0/3507891700 conn(0x7f0e200c40e0 msgr2=0x7f0e200f22c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:42.720 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:42.723 7f0e25866700 1 -- 172.21.2.102:0/3507891700 shutdown_connections 2021-02-10T11:23:42.721 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:42.723 7f0e25866700 1 -- 172.21.2.102:0/3507891700 wait complete. 2021-02-10T11:23:42.729 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth get-or-create client.guest1 mds 'allow *' osd 'allow rw' mon 'allow *' 2021-02-10T11:23:43.160 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:43.146 7fa6a10c2700 1 Processor -- start 2021-02-10T11:23:43.161 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:43.146 7fa6a10c2700 1 -- start start 2021-02-10T11:23:43.161 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:43.147 7fa6a10c2700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fa69c1b0470 0x7fa69c1b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:43.161 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:43.147 7fa6a10c2700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa69c0ef610 0x7fa69c0efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:43.161 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:43.147 7fa6a10c2700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa69c0f0b50 con 0x7fa69c0f0030 2021-02-10T11:23:43.161 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:43.147 7fa6a10c2700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa69c0e52e0 con 0x7fa69c1b0470 2021-02-10T11:23:43.161 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:43.147 7fa6a10c2700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa69c0e54d0 con 0x7fa69c0ef610 2021-02-10T11:23:43.161 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:43.147 7fa69b59e700 1 --1- >> v1:172.21.2.113:6789/0 conn(0x7fa69c0f0030 0x7fa69c0f10d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.113:6789/0 says I am v1:172.21.2.102:56108/0 (socket says 172.21.2.102:56108) 2021-02-10T11:23:43.161 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:43.147 7fa69b59e700 1 -- 172.21.2.102:0/2329476680 learned_addr learned my addr 172.21.2.102:0/2329476680 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:23:43.161 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:43.147 7fa69ad9d700 1 --2- 172.21.2.102:0/2329476680 >> v2:172.21.2.113:3300/0 conn(0x7fa69c1b0470 0x7fa69c1b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:43.161 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:43.147 7fa69259c700 1 --2- 172.21.2.102:0/2329476680 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa69c0ef610 0x7fa69c0efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:43.162 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:43.148 7fa69259c700 1 -- 172.21.2.102:0/2329476680 >> v2:172.21.2.113:3300/0 conn(0x7fa69c1b0470 msgr2=0x7fa69c1b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:43.162 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:43.148 7fa69259c700 1 --2- 172.21.2.102:0/2329476680 >> v2:172.21.2.113:3300/0 conn(0x7fa69c1b0470 0x7fa69c1b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:43.162 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:43.148 7fa69259c700 1 -- 172.21.2.102:0/2329476680 >> v1:172.21.2.113:6789/0 conn(0x7fa69c0f0030 legacy=0x7fa69c0f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:43.162 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:43.148 7fa69259c700 1 -- 172.21.2.102:0/2329476680 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa69c0f0430 con 0x7fa69c0ef610 2021-02-10T11:23:43.162 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:43.148 7fa69259c700 1 --2- 172.21.2.102:0/2329476680 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa69c0ef610 0x7fa69c0efa10 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7fa684006650 tx=0x7fa684009a80).ready entity=mon.2 client_cookie=60be0c29b72a6bf9 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:43.162 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:43.148 7fa69a59c700 1 -- 172.21.2.102:0/2329476680 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa6840009b0 con 0x7fa69c0ef610 2021-02-10T11:23:43.162 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:43.148 7fa69a59c700 1 -- 172.21.2.102:0/2329476680 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa68400c0c0 con 0x7fa69c0ef610 2021-02-10T11:23:43.162 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:43.148 7fa69a59c700 1 -- 172.21.2.102:0/2329476680 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa68400c5e0 con 0x7fa69c0ef610 2021-02-10T11:23:43.162 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:43.149 7fa6a10c2700 1 -- 172.21.2.102:0/2329476680 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa69c0ef610 msgr2=0x7fa69c0efa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:43.162 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:43.149 7fa6a10c2700 1 --2- 172.21.2.102:0/2329476680 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa69c0ef610 0x7fa69c0efa10 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7fa684006650 tx=0x7fa684009a80).stop 2021-02-10T11:23:43.162 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:43.149 7fa6a10c2700 1 -- 172.21.2.102:0/2329476680 shutdown_connections 2021-02-10T11:23:43.163 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:43.149 7fa6a10c2700 1 --2- 172.21.2.102:0/2329476680 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa69c0ef610 0x7fa69c0efa10 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:43.163 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:43.149 7fa6a10c2700 1 --2- 172.21.2.102:0/2329476680 >> v2:172.21.2.113:3300/0 conn(0x7fa69c1b0470 0x7fa69c1b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:43.163 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:43.149 7fa6a10c2700 1 -- 172.21.2.102:0/2329476680 >> 172.21.2.102:0/2329476680 conn(0x7fa69c0ce3e0 msgr2=0x7fa69c0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:43.163 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:43.149 7fa6a10c2700 1 -- 172.21.2.102:0/2329476680 shutdown_connections 2021-02-10T11:23:43.163 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:43.149 7fa6a10c2700 1 -- 172.21.2.102:0/2329476680 wait complete. 2021-02-10T11:23:43.164 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:43.164 7fa6a10c2700 1 Processor -- start 2021-02-10T11:23:43.165 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:43.164 7fa6a10c2700 1 -- start start 2021-02-10T11:23:43.165 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:43.164 7fa6a10c2700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa69c1a0480 0x7fa69c08e840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:43.165 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:43.164 7fa6a10c2700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa69c10a4b0 0x7fa69c08ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:43.165 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:43.164 7fa6a10c2700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa69c18ebf0 0x7fa69c18eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:43.165 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:43.164 7fa6a10c2700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa69c18f720 con 0x7fa69c10a4b0 2021-02-10T11:23:43.165 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:43.164 7fa6a10c2700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa69c18f940 con 0x7fa69c18ebf0 2021-02-10T11:23:43.165 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:43.164 7fa6a10c2700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa69c18fb60 con 0x7fa69c1a0480 2021-02-10T11:23:43.165 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:43.165 7fa69259c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa69c10a4b0 0x7fa69c08ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:43.165 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:43.165 7fa69259c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa69c10a4b0 0x7fa69c08ee60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:56564/0 (socket says 172.21.2.102:56564) 2021-02-10T11:23:43.165 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:43.165 7fa69259c700 1 -- 172.21.2.102:0/1725684325 learned_addr learned my addr 172.21.2.102:0/1725684325 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:23:43.166 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:43.165 7fa69ad9d700 1 --2- 172.21.2.102:0/1725684325 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa69c1a0480 0x7fa69c08e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:43.166 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:43.165 7fa69b59e700 1 --2- 172.21.2.102:0/1725684325 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa69c18ebf0 0x7fa69c18eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:43.166 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:43.165 7fa69259c700 1 -- 172.21.2.102:0/1725684325 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa69c1a0480 msgr2=0x7fa69c08e840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:43.166 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:43.165 7fa69259c700 1 --2- 172.21.2.102:0/1725684325 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa69c1a0480 0x7fa69c08e840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:43.166 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:43.165 7fa69259c700 1 -- 172.21.2.102:0/1725684325 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa69c18ebf0 msgr2=0x7fa69c18eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:43.166 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:43.165 7fa69259c700 1 --2- 172.21.2.102:0/1725684325 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa69c18ebf0 0x7fa69c18eff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:43.166 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:43.165 7fa69259c700 1 -- 172.21.2.102:0/1725684325 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa69c18fd80 con 0x7fa69c10a4b0 2021-02-10T11:23:43.166 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:43.165 7fa69259c700 1 --2- 172.21.2.102:0/1725684325 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa69c10a4b0 0x7fa69c08ee60 secure :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 rx=0x7fa684009630 tx=0x7fa68400b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:43.166 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:43.165 7fa693fff700 1 -- 172.21.2.102:0/1725684325 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa6840093a0 con 0x7fa69c10a4b0 2021-02-10T11:23:43.166 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:43.166 7fa693fff700 1 -- 172.21.2.102:0/1725684325 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa68400bd70 con 0x7fa69c10a4b0 2021-02-10T11:23:43.166 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:43.166 7fa693fff700 1 -- 172.21.2.102:0/1725684325 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa6840054d0 con 0x7fa69c10a4b0 2021-02-10T11:23:43.167 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:43.166 7fa6a10c2700 1 -- 172.21.2.102:0/1725684325 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa69c18fd80 con 0x7fa69c10a4b0 2021-02-10T11:23:43.167 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:43.166 7fa6a10c2700 1 -- 172.21.2.102:0/1725684325 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa69c08fdb0 con 0x7fa69c10a4b0 2021-02-10T11:23:43.167 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:43.166 7fa693fff700 1 -- 172.21.2.102:0/1725684325 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa684007020 con 0x7fa69c10a4b0 2021-02-10T11:23:43.167 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:43.166 7fa693fff700 1 --2- 172.21.2.102:0/1725684325 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa6740186c0 0x7fa67401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:43.167 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:43.166 7fa693fff700 1 -- 172.21.2.102:0/1725684325 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fa684032410 con 0x7fa69c10a4b0 2021-02-10T11:23:43.167 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:43.166 7fa69ad9d700 1 --2- 172.21.2.102:0/1725684325 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa6740186c0 0x7fa67401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:43.168 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:43.167 7fa69ad9d700 1 --2- 172.21.2.102:0/1725684325 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa6740186c0 0x7fa67401abc0 crc :-1 s=READY pgs=588 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:43.194 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:43.197 7fa6a10c2700 1 -- 172.21.2.102:0/1725684325 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa69c1059f0 con 0x7fa69c10a4b0 2021-02-10T11:23:43.196 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:43.199 7fa693fff700 1 -- 172.21.2.102:0/1725684325 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fa684009340 con 0x7fa69c10a4b0 2021-02-10T11:23:43.325 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:43.328 7fa6a10c2700 1 -- 172.21.2.102:0/1725684325 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.guest1", "caps": ["mds", "allow *", "osd", "allow rw", "mon", "allow *"]} v 0) v1 -- 0x7fa69c0f0b50 con 0x7fa69c10a4b0 2021-02-10T11:23:43.328 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:43.331 7fa693fff700 1 -- 172.21.2.102:0/1725684325 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.guest1", "caps": ["mds", "allow *", "osd", "allow rw", "mon", "allow *"]}]=0 v27) v1 ==== 158+0+64 (secure 0 0 0) 0x7fa684007000 con 0x7fa69c10a4b0 2021-02-10T11:23:43.329 INFO:teuthology.orchestra.run.gibba002.stdout:[client.guest1] 2021-02-10T11:23:43.329 INFO:teuthology.orchestra.run.gibba002.stdout: key = AQA/wiNgkL+jExAAUDxsjotx/qtQi0uWaNqyPg== 2021-02-10T11:23:43.330 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:43.333 7fa6a10c2700 1 -- 172.21.2.102:0/1725684325 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa6740186c0 msgr2=0x7fa67401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:43.330 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:43.333 7fa6a10c2700 1 --2- 172.21.2.102:0/1725684325 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa6740186c0 0x7fa67401abc0 crc :-1 s=READY pgs=588 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:43.330 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:43.333 7fa6a10c2700 1 -- 172.21.2.102:0/1725684325 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa69c10a4b0 msgr2=0x7fa69c08ee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:43.330 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:43.333 7fa6a10c2700 1 --2- 172.21.2.102:0/1725684325 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa69c10a4b0 0x7fa69c08ee60 secure :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 rx=0x7fa684009630 tx=0x7fa68400b980).stop 2021-02-10T11:23:43.330 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:43.333 7fa6a10c2700 1 -- 172.21.2.102:0/1725684325 shutdown_connections 2021-02-10T11:23:43.330 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:43.333 7fa6a10c2700 1 --2- 172.21.2.102:0/1725684325 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa6740186c0 0x7fa67401abc0 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:43.331 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:43.333 7fa6a10c2700 1 --2- 172.21.2.102:0/1725684325 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa69c18ebf0 0x7fa69c18eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:43.331 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:43.333 7fa6a10c2700 1 --2- 172.21.2.102:0/1725684325 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa69c10a4b0 0x7fa69c08ee60 unknown :-1 s=CLOSED pgs=1244 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:43.331 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:43.333 7fa6a10c2700 1 --2- 172.21.2.102:0/1725684325 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa69c1a0480 0x7fa69c08e840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:43.331 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:43.333 7fa6a10c2700 1 -- 172.21.2.102:0/1725684325 >> 172.21.2.102:0/1725684325 conn(0x7fa69c09cd10 msgr2=0x7fa69c093a20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:43.331 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:43.333 7fa6a10c2700 1 -- 172.21.2.102:0/1725684325 shutdown_connections 2021-02-10T11:23:43.331 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:43.334 7fa6a10c2700 1 -- 172.21.2.102:0/1725684325 wait complete. 2021-02-10T11:23:43.338 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolumegroup create cephfs group_0000000000353920 2021-02-10T11:23:43.769 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:43.756 7fa866473700 1 Processor -- start 2021-02-10T11:23:43.770 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:43.756 7fa866473700 1 -- start start 2021-02-10T11:23:43.770 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:43.756 7fa866473700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa8601b0470 0x7fa8601b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:43.770 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:43.756 7fa866473700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fa8600ef610 0x7fa8600efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:43.770 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:43.756 7fa866473700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8600f0b50 con 0x7fa8600f0030 2021-02-10T11:23:43.770 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:43.757 7fa866473700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8600e52e0 con 0x7fa8600ef610 2021-02-10T11:23:43.770 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:43.757 7fa866473700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8600e5490 con 0x7fa8601b0470 2021-02-10T11:23:43.770 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:43.757 7fa864a90700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fa8600f0030 0x7fa8600f10d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:48796/0 (socket says 172.21.2.102:48796) 2021-02-10T11:23:43.770 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:43.757 7fa864a90700 1 -- 172.21.2.102:0/2224904872 learned_addr learned my addr 172.21.2.102:0/2224904872 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:23:43.770 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:43.757 7fa85ffff700 1 --2- 172.21.2.102:0/2224904872 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa8601b0470 0x7fa8601b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:43.770 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:43.757 7fa85f7fe700 1 --2- 172.21.2.102:0/2224904872 >> v2:172.21.2.113:3300/0 conn(0x7fa8600ef610 0x7fa8600efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:43.770 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:43.757 7fa85effd700 1 -- 172.21.2.102:0/2224904872 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3444913270 0 0) 0x7fa858000c10 con 0x7fa8600f0030 2021-02-10T11:23:43.771 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:43.757 7fa85effd700 1 -- 172.21.2.102:0/2224904872 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa84c002fe0 con 0x7fa8600f0030 2021-02-10T11:23:43.771 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:43.757 7fa85f7fe700 1 -- 172.21.2.102:0/2224904872 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa8601b0470 msgr2=0x7fa8601b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:43.771 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:43.757 7fa85f7fe700 1 --2- 172.21.2.102:0/2224904872 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa8601b0470 0x7fa8601b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:43.771 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:43.757 7fa85f7fe700 1 -- 172.21.2.102:0/2224904872 >> v1:172.21.2.102:6789/0 conn(0x7fa8600f0030 legacy=0x7fa8600f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:43.771 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:43.757 7fa85f7fe700 1 -- 172.21.2.102:0/2224904872 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8600f0430 con 0x7fa8600ef610 2021-02-10T11:23:43.771 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:43.757 7fa85f7fe700 1 --2- 172.21.2.102:0/2224904872 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa8600ef610 0x7fa8600efa10 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 rx=0x7fa8500069d0 tx=0x7fa850009e40).ready entity=mon.1 client_cookie=f80e20e4eeee81c6 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:43.772 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:43.758 7fa85effd700 1 -- 172.21.2.102:0/2224904872 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa85000b510 con 0x7fa8600ef610 2021-02-10T11:23:43.772 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:43.758 7fa85effd700 1 -- 172.21.2.102:0/2224904872 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa85000b790 con 0x7fa8600ef610 2021-02-10T11:23:43.772 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:43.758 7fa85effd700 1 -- 172.21.2.102:0/2224904872 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa85000bcb0 con 0x7fa8600ef610 2021-02-10T11:23:43.772 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:43.758 7fa866473700 1 -- 172.21.2.102:0/2224904872 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa8600ef610 msgr2=0x7fa8600efa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:43.772 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:43.758 7fa866473700 1 --2- 172.21.2.102:0/2224904872 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa8600ef610 0x7fa8600efa10 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 rx=0x7fa8500069d0 tx=0x7fa850009e40).stop 2021-02-10T11:23:43.773 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:43.758 7fa866473700 1 -- 172.21.2.102:0/2224904872 shutdown_connections 2021-02-10T11:23:43.773 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:43.758 7fa866473700 1 --2- 172.21.2.102:0/2224904872 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa8600ef610 0x7fa8600efa10 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:43.773 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:43.758 7fa866473700 1 --2- 172.21.2.102:0/2224904872 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa8601b0470 0x7fa8601b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:43.774 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:43.758 7fa866473700 1 -- 172.21.2.102:0/2224904872 >> 172.21.2.102:0/2224904872 conn(0x7fa8600ce3e0 msgr2=0x7fa8600e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:43.774 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:43.758 7fa866473700 1 -- 172.21.2.102:0/2224904872 shutdown_connections 2021-02-10T11:23:43.774 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:43.758 7fa866473700 1 -- 172.21.2.102:0/2224904872 wait complete. 2021-02-10T11:23:43.777 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:43.773 7fa866473700 1 Processor -- start 2021-02-10T11:23:43.777 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:43.773 7fa866473700 1 -- start start 2021-02-10T11:23:43.777 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:43.774 7fa866473700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa8600a0420 0x7fa8600a0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:43.777 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:43.774 7fa866473700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa860189d00 0x7fa86018a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:43.777 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:43.774 7fa866473700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa86018a720 0x7fa8600b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:43.778 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:43.774 7fa866473700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa860189800 con 0x7fa86018a720 2021-02-10T11:23:43.778 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:43.774 7fa866473700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa86018ab20 con 0x7fa8600a0420 2021-02-10T11:23:43.778 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:43.774 7fa866473700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8600a1140 con 0x7fa860189d00 2021-02-10T11:23:43.779 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:43.774 7fa864a90700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa86018a720 0x7fa8600b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:43.779 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:43.774 7fa864a90700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa86018a720 0x7fa8600b7be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:56578/0 (socket says 172.21.2.102:56578) 2021-02-10T11:23:43.779 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:43.774 7fa864a90700 1 -- 172.21.2.102:0/176919700 learned_addr learned my addr 172.21.2.102:0/176919700 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:23:43.779 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:43.774 7fa85ffff700 1 --2- 172.21.2.102:0/176919700 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa8600a0420 0x7fa8600a0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:43.779 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:43.774 7fa85f7fe700 1 --2- 172.21.2.102:0/176919700 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa860189d00 0x7fa86018a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:43.780 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:43.774 7fa864a90700 1 -- 172.21.2.102:0/176919700 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa860189d00 msgr2=0x7fa86018a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:43.780 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:43.774 7fa864a90700 1 --2- 172.21.2.102:0/176919700 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa860189d00 0x7fa86018a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:43.781 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:43.774 7fa864a90700 1 -- 172.21.2.102:0/176919700 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa8600a0420 msgr2=0x7fa8600a0820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:43.782 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:43.774 7fa864a90700 1 --2- 172.21.2.102:0/176919700 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa8600a0420 0x7fa8600a0820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:43.783 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:43.774 7fa864a90700 1 -- 172.21.2.102:0/176919700 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8600b8200 con 0x7fa86018a720 2021-02-10T11:23:43.783 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:43.774 7fa864a90700 1 --2- 172.21.2.102:0/176919700 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa86018a720 0x7fa8600b7be0 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 rx=0x7fa858006f00 tx=0x7fa85800b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:43.783 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:43.775 7fa85cff9700 1 -- 172.21.2.102:0/176919700 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa85800bd30 con 0x7fa86018a720 2021-02-10T11:23:43.783 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:43.775 7fa85cff9700 1 -- 172.21.2.102:0/176919700 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8580060d0 con 0x7fa86018a720 2021-02-10T11:23:43.784 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:43.775 7fa85cff9700 1 -- 172.21.2.102:0/176919700 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa8580043a0 con 0x7fa86018a720 2021-02-10T11:23:43.784 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:43.775 7fa866473700 1 -- 172.21.2.102:0/176919700 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8600b8200 con 0x7fa86018a720 2021-02-10T11:23:43.784 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:43.775 7fa866473700 1 -- 172.21.2.102:0/176919700 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8600b8200 con 0x7fa86018a720 2021-02-10T11:23:43.785 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:43.775 7fa85cff9700 1 -- 172.21.2.102:0/176919700 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa8580062b0 con 0x7fa86018a720 2021-02-10T11:23:43.785 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:43.775 7fa85cff9700 1 --2- 172.21.2.102:0/176919700 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa8440186c0 0x7fa84401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:43.785 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:43.775 7fa85cff9700 1 -- 172.21.2.102:0/176919700 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fa8580327b0 con 0x7fa86018a720 2021-02-10T11:23:43.785 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:43.776 7fa85ffff700 1 --2- 172.21.2.102:0/176919700 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa8440186c0 0x7fa84401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:43.785 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:43.776 7fa85ffff700 1 --2- 172.21.2.102:0/176919700 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa8440186c0 0x7fa84401abc0 crc :-1 s=READY pgs=589 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:43.803 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:43.806 7fa866473700 1 -- 172.21.2.102:0/176919700 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa860105ae0 con 0x7fa86018a720 2021-02-10T11:23:43.806 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:43.809 7fa85cff9700 1 -- 172.21.2.102:0/176919700 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fa85800bd30 con 0x7fa86018a720 2021-02-10T11:23:43.934 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:43.937 7fa866473700 1 -- 172.21.2.102:0/176919700 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "target": ["mgr", ""], "group_name": "group_0000000000353920"}) v1 -- 0x7fa860105ae0 con 0x7fa8440186c0 2021-02-10T11:23:43.938 INFO:tasks.ceph.mgr.y.gibba002.stderr:2021-02-10 11:23:43.941 7fd94bc53700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:23:43.938 INFO:tasks.ceph.mgr.y.gibba002.stderr:2021-02-10 11:23:43.941 7fd94bc53700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:23:43.939 INFO:tasks.ceph.mgr.y.gibba002.stderr:2021-02-10 11:23:43.941 7fd94bc53700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:23:43.939 INFO:tasks.ceph.mgr.y.gibba002.stderr:2021-02-10 11:23:43.941 7fd94bc53700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:23:43.939 INFO:tasks.ceph.mgr.y.gibba002.stderr:2021-02-10 11:23:43.941 7fd94bc53700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:23:43.953 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:43.956 7fa85cff9700 1 -- 172.21.2.102:0/176919700 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fa848006a70 con 0x7fa8440186c0 2021-02-10T11:23:43.967 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:43.970 7fa866473700 1 -- 172.21.2.102:0/176919700 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa8440186c0 msgr2=0x7fa84401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:43.968 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:43.970 7fa866473700 1 --2- 172.21.2.102:0/176919700 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa8440186c0 0x7fa84401abc0 crc :-1 s=READY pgs=589 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:43.968 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:43.970 7fa866473700 1 -- 172.21.2.102:0/176919700 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa86018a720 msgr2=0x7fa8600b7be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:43.968 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:43.970 7fa866473700 1 --2- 172.21.2.102:0/176919700 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa86018a720 0x7fa8600b7be0 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 rx=0x7fa858006f00 tx=0x7fa85800b940).stop 2021-02-10T11:23:43.968 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:43.970 7fa866473700 1 -- 172.21.2.102:0/176919700 shutdown_connections 2021-02-10T11:23:43.968 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:43.970 7fa866473700 1 --2- 172.21.2.102:0/176919700 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa86018a720 0x7fa8600b7be0 unknown :-1 s=CLOSED pgs=1246 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:43.968 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:43.970 7fa866473700 1 --2- 172.21.2.102:0/176919700 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa860189d00 0x7fa86018a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:43.968 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:43.970 7fa866473700 1 --2- 172.21.2.102:0/176919700 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa8440186c0 0x7fa84401abc0 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:43.968 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:43.970 7fa866473700 1 --2- 172.21.2.102:0/176919700 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa8600a0420 0x7fa8600a0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:43.968 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:43.971 7fa866473700 1 -- 172.21.2.102:0/176919700 >> 172.21.2.102:0/176919700 conn(0x7fa860094e10 msgr2=0x7fa8600a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:43.969 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:43.971 7fa866473700 1 -- 172.21.2.102:0/176919700 shutdown_connections 2021-02-10T11:23:43.969 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:43.971 7fa866473700 1 -- 172.21.2.102:0/176919700 wait complete. 2021-02-10T11:23:43.978 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000647253 --group_name group_0000000000353920 2021-02-10T11:23:44.413 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:44.399 7f1b98c76700 1 Processor -- start 2021-02-10T11:23:44.413 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:44.399 7f1b98c76700 1 -- start start 2021-02-10T11:23:44.413 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:44.400 7f1b98c76700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f1b940d1860 0x7f1b941b03e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:44.413 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:44.400 7f1b98c76700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1b941b0a00 0x7f1b940cf320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:44.413 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:44.400 7f1b98c76700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b940f0b00 con 0x7f1b940d1c60 2021-02-10T11:23:44.413 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:44.400 7f1b98c76700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b940e5ec0 con 0x7f1b940d1860 2021-02-10T11:23:44.414 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:44.400 7f1b98c76700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b940f0640 con 0x7f1b941b0a00 2021-02-10T11:23:44.414 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:44.400 7f1b9285e700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f1b940d1860 0x7f1b941b03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:44.414 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:44.400 7f1b9285e700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f1b940d1860 0x7f1b941b03e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:56596/0 (socket says 172.21.2.102:56596) 2021-02-10T11:23:44.414 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:44.400 7f1b9285e700 1 -- 172.21.2.102:0/2647568116 learned_addr learned my addr 172.21.2.102:0/2647568116 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:23:44.414 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:44.400 7f1b9285e700 1 -- 172.21.2.102:0/2647568116 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1b941b0a00 msgr2=0x7f1b940cf320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:44.414 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:44.400 7f1b9285e700 1 --2- 172.21.2.102:0/2647568116 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1b941b0a00 0x7f1b940cf320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:44.414 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:44.400 7f1b9285e700 1 -- 172.21.2.102:0/2647568116 >> v1:172.21.2.102:6789/0 conn(0x7f1b940d1c60 legacy=0x7f1b940cf940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:44.414 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:44.400 7f1b9285e700 1 -- 172.21.2.102:0/2647568116 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b940f0860 con 0x7f1b940d1860 2021-02-10T11:23:44.414 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:44.400 7f1b9185c700 1 -- 172.21.2.102:0/2647568116 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2911429229 0 0) 0x7f1b8c000f90 con 0x7f1b940d1c60 2021-02-10T11:23:44.415 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:44.401 7f1b9285e700 1 --2- 172.21.2.102:0/2647568116 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1b940d1860 0x7f1b941b03e0 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 rx=0x7f1b7c006700 tx=0x7f1b7c009b00).ready entity=mon.0 client_cookie=ade3a6e24236a2e2 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:44.415 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:44.401 7f1b9185c700 1 -- 172.21.2.102:0/2647568116 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1b7c00c0c0 con 0x7f1b940d1860 2021-02-10T11:23:44.415 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:44.401 7f1b9185c700 1 -- 172.21.2.102:0/2647568116 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b7c00c400 con 0x7f1b940d1860 2021-02-10T11:23:44.415 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:44.401 7f1b9185c700 1 -- 172.21.2.102:0/2647568116 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1b7c00c860 con 0x7f1b940d1860 2021-02-10T11:23:44.415 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:44.401 7f1b98c76700 1 -- 172.21.2.102:0/2647568116 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1b940d1860 msgr2=0x7f1b941b03e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:44.415 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:44.401 7f1b98c76700 1 --2- 172.21.2.102:0/2647568116 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1b940d1860 0x7f1b941b03e0 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 rx=0x7f1b7c006700 tx=0x7f1b7c009b00).stop 2021-02-10T11:23:44.415 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:44.401 7f1b98c76700 1 -- 172.21.2.102:0/2647568116 shutdown_connections 2021-02-10T11:23:44.415 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:44.402 7f1b98c76700 1 --2- 172.21.2.102:0/2647568116 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1b941b0a00 0x7f1b940cf320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:44.415 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:44.402 7f1b98c76700 1 --2- 172.21.2.102:0/2647568116 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1b940d1860 0x7f1b941b03e0 unknown :-1 s=CLOSED pgs=1250 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:44.415 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:44.402 7f1b98c76700 1 -- 172.21.2.102:0/2647568116 >> 172.21.2.102:0/2647568116 conn(0x7f1b940ce440 msgr2=0x7f1b940d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:44.415 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:44.402 7f1b98c76700 1 -- 172.21.2.102:0/2647568116 shutdown_connections 2021-02-10T11:23:44.415 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:44.402 7f1b98c76700 1 -- 172.21.2.102:0/2647568116 wait complete. 2021-02-10T11:23:44.415 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:44.416 7f1b98c76700 1 Processor -- start 2021-02-10T11:23:44.415 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:44.416 7f1b98c76700 1 -- start start 2021-02-10T11:23:44.417 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:44.416 7f1b98c76700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1b940ef8d0 0x7f1b941a8730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:44.418 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:44.416 7f1b98c76700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1b940efcd0 0x7f1b941a8d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:44.418 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:44.416 7f1b98c76700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1b940aa240 0x7f1b940aa640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:44.418 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:44.416 7f1b98c76700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b940f0270 con 0x7f1b940efcd0 2021-02-10T11:23:44.418 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:44.416 7f1b98c76700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b940aac60 con 0x7f1b940ef8d0 2021-02-10T11:23:44.418 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:44.416 7f1b98c76700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b940aae80 con 0x7f1b940aa240 2021-02-10T11:23:44.418 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:44.417 7f1b9205d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1b940efcd0 0x7f1b941a8d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:44.418 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:44.417 7f1b9205d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1b940efcd0 0x7f1b941a8d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:56606/0 (socket says 172.21.2.102:56606) 2021-02-10T11:23:44.418 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:44.417 7f1b9205d700 1 -- 172.21.2.102:0/2745846747 learned_addr learned my addr 172.21.2.102:0/2745846747 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:23:44.418 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:44.417 7f1b9305f700 1 --2- 172.21.2.102:0/2745846747 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1b940aa240 0x7f1b940aa640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:44.418 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:44.417 7f1b9285e700 1 --2- 172.21.2.102:0/2745846747 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1b940ef8d0 0x7f1b941a8730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:44.418 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:44.417 7f1b9205d700 1 -- 172.21.2.102:0/2745846747 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1b940aa240 msgr2=0x7f1b940aa640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:44.418 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:44.417 7f1b9205d700 1 --2- 172.21.2.102:0/2745846747 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1b940aa240 0x7f1b940aa640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:44.418 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:44.417 7f1b9205d700 1 -- 172.21.2.102:0/2745846747 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1b940ef8d0 msgr2=0x7f1b941a8730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:44.419 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:44.417 7f1b9205d700 1 --2- 172.21.2.102:0/2745846747 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1b940ef8d0 0x7f1b941a8730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:44.419 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:44.417 7f1b9205d700 1 -- 172.21.2.102:0/2745846747 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b940ebe10 con 0x7f1b940efcd0 2021-02-10T11:23:44.419 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:44.417 7f1b9305f700 1 --2- 172.21.2.102:0/2745846747 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1b940aa240 0x7f1b940aa640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:23:44.419 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:44.417 7f1b9285e700 1 --2- 172.21.2.102:0/2745846747 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1b940ef8d0 0x7f1b941a8730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:23:44.419 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:44.417 7f1b9205d700 1 --2- 172.21.2.102:0/2745846747 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1b940efcd0 0x7f1b941a8d50 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 rx=0x7f1b84006f00 tx=0x7f1b8400b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:44.419 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:44.417 7f1b8b7fe700 1 -- 172.21.2.102:0/2745846747 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1b8400bd30 con 0x7f1b940efcd0 2021-02-10T11:23:44.419 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:44.417 7f1b8b7fe700 1 -- 172.21.2.102:0/2745846747 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b84000c90 con 0x7f1b940efcd0 2021-02-10T11:23:44.419 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:44.417 7f1b98c76700 1 -- 172.21.2.102:0/2745846747 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b940ebe10 con 0x7f1b940efcd0 2021-02-10T11:23:44.419 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:44.417 7f1b8b7fe700 1 -- 172.21.2.102:0/2745846747 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1b840022a0 con 0x7f1b940efcd0 2021-02-10T11:23:44.419 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:44.417 7f1b98c76700 1 -- 172.21.2.102:0/2745846747 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b940ebe10 con 0x7f1b940efcd0 2021-02-10T11:23:44.419 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:44.418 7f1b8b7fe700 1 -- 172.21.2.102:0/2745846747 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1b84005310 con 0x7f1b940efcd0 2021-02-10T11:23:44.419 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:44.418 7f1b8b7fe700 1 --2- 172.21.2.102:0/2745846747 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1b780186c0 0x7f1b7801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:44.419 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:44.418 7f1b8b7fe700 1 -- 172.21.2.102:0/2745846747 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f1b840327b0 con 0x7f1b940efcd0 2021-02-10T11:23:44.419 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:44.418 7f1b9285e700 1 --2- 172.21.2.102:0/2745846747 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1b780186c0 0x7f1b7801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:44.420 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:44.418 7f1b9285e700 1 --2- 172.21.2.102:0/2745846747 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1b780186c0 0x7f1b7801abc0 crc :-1 s=READY pgs=590 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:44.446 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:44.449 7f1b98c76700 1 -- 172.21.2.102:0/2745846747 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b940eb8f0 con 0x7f1b940efcd0 2021-02-10T11:23:44.449 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:44.452 7f1b8b7fe700 1 -- 172.21.2.102:0/2745846747 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f1b84006b90 con 0x7f1b940efcd0 2021-02-10T11:23:44.577 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:44.580 7f1b98c76700 1 -- 172.21.2.102:0/2745846747 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000647253", "target": ["mgr", ""], "group_name": "group_0000000000353920"}) v1 -- 0x7f1b940eb8f0 con 0x7f1b780186c0 2021-02-10T11:23:44.602 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:44.605 7f1b8b7fe700 1 -- 172.21.2.102:0/2745846747 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f1b7c006e80 con 0x7f1b780186c0 2021-02-10T11:23:44.611 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:44.613 7f1b98c76700 1 -- 172.21.2.102:0/2745846747 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1b780186c0 msgr2=0x7f1b7801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:44.611 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:44.613 7f1b98c76700 1 --2- 172.21.2.102:0/2745846747 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1b780186c0 0x7f1b7801abc0 crc :-1 s=READY pgs=590 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:44.611 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:44.613 7f1b98c76700 1 -- 172.21.2.102:0/2745846747 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1b940efcd0 msgr2=0x7f1b941a8d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:44.611 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:44.613 7f1b98c76700 1 --2- 172.21.2.102:0/2745846747 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1b940efcd0 0x7f1b941a8d50 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 rx=0x7f1b84006f00 tx=0x7f1b8400b940).stop 2021-02-10T11:23:44.611 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:44.613 7f1b98c76700 1 -- 172.21.2.102:0/2745846747 shutdown_connections 2021-02-10T11:23:44.611 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:44.613 7f1b98c76700 1 --2- 172.21.2.102:0/2745846747 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1b940aa240 0x7f1b940aa640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:44.611 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:44.613 7f1b98c76700 1 --2- 172.21.2.102:0/2745846747 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1b940efcd0 0x7f1b941a8d50 unknown :-1 s=CLOSED pgs=1251 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:44.611 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:44.613 7f1b98c76700 1 --2- 172.21.2.102:0/2745846747 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1b780186c0 0x7f1b7801abc0 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:44.611 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:44.613 7f1b98c76700 1 --2- 172.21.2.102:0/2745846747 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1b940ef8d0 0x7f1b941a8730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:44.611 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:44.613 7f1b98c76700 1 -- 172.21.2.102:0/2745846747 >> 172.21.2.102:0/2745846747 conn(0x7f1b9419d4d0 msgr2=0x7f1b941b1970 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:44.611 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:44.613 7f1b98c76700 1 -- 172.21.2.102:0/2745846747 shutdown_connections 2021-02-10T11:23:44.611 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:44.613 7f1b98c76700 1 -- 172.21.2.102:0/2745846747 wait complete. 2021-02-10T11:23:44.619 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume authorize cephfs subvolume_0000000000647253 guest1 --group_name group_0000000000353920 --tenant_id tenant1 2021-02-10T11:23:45.060 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:45.047 7f1c0da77700 1 Processor -- start 2021-02-10T11:23:45.061 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:45.047 7f1c0da77700 1 -- start start 2021-02-10T11:23:45.061 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:45.048 7f1c0da77700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1c080e5010 0x7f1c081b37c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:45.061 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:45.048 7f1c0da77700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f1c081b3de0 0x7f1c080f5a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:45.061 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:45.048 7f1c0da77700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c080f5480 con 0x7f1c080e5410 2021-02-10T11:23:45.061 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:45.048 7f1c0da77700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c080e5e40 con 0x7f1c081b3de0 2021-02-10T11:23:45.061 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:45.048 7f1c0da77700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c080f4de0 con 0x7f1c080e5010 2021-02-10T11:23:45.061 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:45.048 7f1c06ffd700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f1c081b3de0 0x7f1c080f5a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:45.061 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:45.048 7f1c06ffd700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f1c081b3de0 0x7f1c080f5a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:56612/0 (socket says 172.21.2.102:56612) 2021-02-10T11:23:45.061 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:45.048 7f1c06ffd700 1 -- 172.21.2.102:0/320165360 learned_addr learned my addr 172.21.2.102:0/320165360 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:23:45.061 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:45.048 7f1c07fff700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f1c080e5410 0x7f1c080f6020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:48836/0 (socket says 172.21.2.102:48836) 2021-02-10T11:23:45.062 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:45.048 7f1c06ffd700 1 -- 172.21.2.102:0/320165360 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1c080e5010 msgr2=0x7f1c081b37c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:45.062 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:45.048 7f1c06ffd700 1 --2- 172.21.2.102:0/320165360 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1c080e5010 0x7f1c081b37c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:45.062 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:45.048 7f1c06ffd700 1 -- 172.21.2.102:0/320165360 >> v1:172.21.2.102:6789/0 conn(0x7f1c080e5410 legacy=0x7f1c080f6020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:45.062 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:45.048 7f1c067fc700 1 -- 172.21.2.102:0/320165360 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3095205296 0 0) 0x7f1c00001480 con 0x7f1c080e5410 2021-02-10T11:23:45.062 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:45.048 7f1c06ffd700 1 -- 172.21.2.102:0/320165360 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c080f4fd0 con 0x7f1c081b3de0 2021-02-10T11:23:45.062 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:45.048 7f1c06ffd700 1 --2- 172.21.2.102:0/320165360 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1c081b3de0 0x7f1c080f5a00 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 rx=0x7f1bf8006700 tx=0x7f1bf8009b00).ready entity=mon.0 client_cookie=1059d3c3b154936f server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:45.062 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:45.049 7f1c067fc700 1 -- 172.21.2.102:0/320165360 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1bf800c0c0 con 0x7f1c081b3de0 2021-02-10T11:23:45.062 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:45.049 7f1c067fc700 1 -- 172.21.2.102:0/320165360 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bf800c0c0 con 0x7f1c081b3de0 2021-02-10T11:23:45.062 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:45.049 7f1c067fc700 1 -- 172.21.2.102:0/320165360 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1bf800c400 con 0x7f1c081b3de0 2021-02-10T11:23:45.062 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:45.049 7f1c0da77700 1 -- 172.21.2.102:0/320165360 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1c081b3de0 msgr2=0x7f1c080f5a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:45.062 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:45.049 7f1c0da77700 1 --2- 172.21.2.102:0/320165360 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1c081b3de0 0x7f1c080f5a00 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 rx=0x7f1bf8006700 tx=0x7f1bf8009b00).stop 2021-02-10T11:23:45.062 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:45.049 7f1c0da77700 1 -- 172.21.2.102:0/320165360 shutdown_connections 2021-02-10T11:23:45.062 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:45.049 7f1c0da77700 1 --2- 172.21.2.102:0/320165360 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1c081b3de0 0x7f1c080f5a00 unknown :-1 s=CLOSED pgs=1253 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:45.063 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:45.049 7f1c0da77700 1 --2- 172.21.2.102:0/320165360 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1c080e5010 0x7f1c081b37c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:45.063 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:45.049 7f1c0da77700 1 -- 172.21.2.102:0/320165360 >> 172.21.2.102:0/320165360 conn(0x7f1c080ce670 msgr2=0x7f1c080e44b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:45.063 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:45.049 7f1c0da77700 1 -- 172.21.2.102:0/320165360 shutdown_connections 2021-02-10T11:23:45.063 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:45.049 7f1c0da77700 1 -- 172.21.2.102:0/320165360 wait complete. 2021-02-10T11:23:45.064 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:45.064 7f1c0da77700 1 Processor -- start 2021-02-10T11:23:45.064 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:45.064 7f1c0da77700 1 -- start start 2021-02-10T11:23:45.065 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:45.064 7f1c0da77700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1c0808d400 0x7f1c080ad4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:45.065 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:45.064 7f1c0da77700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1c080adaf0 0x7f1c08190330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:45.065 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:45.064 7f1c0da77700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1c08190950 0x7f1c08190d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:45.065 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:45.064 7f1c0da77700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c0808d880 con 0x7f1c08190950 2021-02-10T11:23:45.065 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:45.064 7f1c0da77700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c0818fee0 con 0x7f1c080adaf0 2021-02-10T11:23:45.065 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:45.064 7f1c0da77700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c0808b360 con 0x7f1c0808d400 2021-02-10T11:23:45.065 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:45.065 7f1c07fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1c08190950 0x7f1c08190d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:45.065 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:45.065 7f1c07fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1c08190950 0x7f1c08190d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:56618/0 (socket says 172.21.2.102:56618) 2021-02-10T11:23:45.065 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:45.065 7f1c07fff700 1 -- 172.21.2.102:0/3049457475 learned_addr learned my addr 172.21.2.102:0/3049457475 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:23:45.065 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:45.065 7f1c07fff700 1 -- 172.21.2.102:0/3049457475 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1c0808d400 msgr2=0x7f1c080ad4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:45.065 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:45.065 7f1c07fff700 1 --2- 172.21.2.102:0/3049457475 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1c0808d400 0x7f1c080ad4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:45.065 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:45.065 7f1c07fff700 1 -- 172.21.2.102:0/3049457475 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1c080adaf0 msgr2=0x7f1c08190330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:45.066 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:45.065 7f1c07fff700 1 --2- 172.21.2.102:0/3049457475 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1c080adaf0 0x7f1c08190330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:45.066 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:45.065 7f1c07fff700 1 -- 172.21.2.102:0/3049457475 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c0808b580 con 0x7f1c08190950 2021-02-10T11:23:45.066 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:45.065 7f1c07fff700 1 --2- 172.21.2.102:0/3049457475 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1c08190950 0x7f1c08190d50 secure :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 rx=0x7f1c0000a8f0 tx=0x7f1c0000ac90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:45.066 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:45.065 7f1bfffff700 1 -- 172.21.2.102:0/3049457475 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1c000070d0 con 0x7f1c08190950 2021-02-10T11:23:45.066 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:45.066 7f1bfffff700 1 -- 172.21.2.102:0/3049457475 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c00007310 con 0x7f1c08190950 2021-02-10T11:23:45.066 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:45.066 7f1bfffff700 1 -- 172.21.2.102:0/3049457475 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1c00004720 con 0x7f1c08190950 2021-02-10T11:23:45.066 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:45.066 7f1c0da77700 1 -- 172.21.2.102:0/3049457475 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c0808b580 con 0x7f1c08190950 2021-02-10T11:23:45.066 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:45.066 7f1c0da77700 1 -- 172.21.2.102:0/3049457475 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c0808bb10 con 0x7f1c08190950 2021-02-10T11:23:45.066 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:45.066 7f1bfffff700 1 -- 172.21.2.102:0/3049457475 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1c00000e70 con 0x7f1c08190950 2021-02-10T11:23:45.066 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:45.066 7f1bfffff700 1 --2- 172.21.2.102:0/3049457475 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1bec0186c0 0x7f1bec01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:45.066 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:45.066 7f1bfffff700 1 -- 172.21.2.102:0/3049457475 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f1c000328c0 con 0x7f1c08190950 2021-02-10T11:23:45.066 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:45.066 7f1c077fe700 1 --2- 172.21.2.102:0/3049457475 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1bec0186c0 0x7f1bec01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:45.067 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:45.067 7f1c077fe700 1 --2- 172.21.2.102:0/3049457475 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1bec0186c0 0x7f1bec01abc0 crc :-1 s=READY pgs=591 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:45.098 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:45.097 7f1c0da77700 1 -- 172.21.2.102:0/3049457475 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c0807e100 con 0x7f1c08190950 2021-02-10T11:23:45.098 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:45.100 7f1bfffff700 1 -- 172.21.2.102:0/3049457475 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f1c00001270 con 0x7f1c08190950 2021-02-10T11:23:45.222 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:45.225 7f1c0da77700 1 -- 172.21.2.102:0/3049457475 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume authorize", "vol_name": "cephfs", "sub_name": "subvolume_0000000000647253", "target": ["mgr", ""], "tenant_id": "tenant1", "auth_id": "guest1", "group_name": "group_0000000000353920"}) v1 -- 0x7f1c0808bb10 con 0x7f1bec0186c0 2021-02-10T11:23:45.228 INFO:tasks.ceph.mgr.y.gibba002.stderr:2021-02-10 11:23:45.231 7fd94bc53700 -1 mgr.server reply reply (1) Operation not permitted auth ID: guest1 exists and not created by mgr plugin. Not allowed to modify 2021-02-10T11:23:45.229 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:45.231 7f1bfffff700 1 -- 172.21.2.102:0/3049457475 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: -1 auth ID: guest1 exists and not created by mgr plugin. Not allowed to modify) v1 ==== 83+0+0 (crc 0 0 0) 0x7f1bf00009b0 con 0x7f1bec0186c0 2021-02-10T11:23:45.229 INFO:teuthology.orchestra.run.gibba002.stderr:Error EPERM: auth ID: guest1 exists and not created by mgr plugin. Not allowed to modify 2021-02-10T11:23:45.231 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:45.233 7f1c0da77700 1 -- 172.21.2.102:0/3049457475 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1bec0186c0 msgr2=0x7f1bec01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:45.231 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:45.233 7f1c0da77700 1 --2- 172.21.2.102:0/3049457475 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1bec0186c0 0x7f1bec01abc0 crc :-1 s=READY pgs=591 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:45.231 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:45.233 7f1c0da77700 1 -- 172.21.2.102:0/3049457475 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1c08190950 msgr2=0x7f1c08190d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:45.231 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:45.233 7f1c0da77700 1 --2- 172.21.2.102:0/3049457475 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1c08190950 0x7f1c08190d50 secure :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 rx=0x7f1c0000a8f0 tx=0x7f1c0000ac90).stop 2021-02-10T11:23:45.231 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:45.233 7f1c0da77700 1 -- 172.21.2.102:0/3049457475 shutdown_connections 2021-02-10T11:23:45.231 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:45.233 7f1c0da77700 1 --2- 172.21.2.102:0/3049457475 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1c08190950 0x7f1c08190d50 unknown :-1 s=CLOSED pgs=1254 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:45.231 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:45.233 7f1c0da77700 1 --2- 172.21.2.102:0/3049457475 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1bec0186c0 0x7f1bec01abc0 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:45.231 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:45.233 7f1c0da77700 1 --2- 172.21.2.102:0/3049457475 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1c080adaf0 0x7f1c08190330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:45.231 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:45.233 7f1c0da77700 1 --2- 172.21.2.102:0/3049457475 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1c0808d400 0x7f1c080ad4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:45.231 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:45.233 7f1c0da77700 1 -- 172.21.2.102:0/3049457475 >> 172.21.2.102:0/3049457475 conn(0x7f1c080ffe70 msgr2=0x7f1c08080340 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:45.231 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:45.234 7f1c0da77700 1 -- 172.21.2.102:0/3049457475 shutdown_connections 2021-02-10T11:23:45.232 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:45.234 7f1c0da77700 1 -- 172.21.2.102:0/3049457475 wait complete. 2021-02-10T11:23:45.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:23:45.240 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth rm client.guest1 2021-02-10T11:23:45.666 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:45.652 7eff39424700 1 Processor -- start 2021-02-10T11:23:45.666 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:45.652 7eff39424700 1 -- start start 2021-02-10T11:23:45.666 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:45.653 7eff39424700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7eff340e5070 0x7eff341b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:45.666 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:45.653 7eff39424700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7eff340e5470 0x7eff34080310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:45.666 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:45.653 7eff39424700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff3407ed80 con 0x7eff341b3e40 2021-02-10T11:23:45.666 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:45.653 7eff39424700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff340d1790 con 0x7eff340e5070 2021-02-10T11:23:45.667 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:45.653 7eff39424700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff340d19b0 con 0x7eff340e5470 2021-02-10T11:23:45.667 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:45.653 7eff3287c700 1 --1- >> v1:172.21.2.113:6789/0 conn(0x7eff341b3e40 0x7eff3407d690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.113:6789/0 says I am v1:172.21.2.102:56172/0 (socket says 172.21.2.102:56172) 2021-02-10T11:23:45.667 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:45.653 7eff3387e700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7eff340e5470 0x7eff34080310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:45.667 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:45.653 7eff3307d700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7eff340e5070 0x7eff341b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:45.667 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:45.653 7eff3287c700 1 -- 172.21.2.102:0/2294327636 learned_addr learned my addr 172.21.2.102:0/2294327636 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:23:45.667 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:45.654 7eff3387e700 1 -- 172.21.2.102:0/2294327636 >> v2:172.21.2.113:3300/0 conn(0x7eff340e5070 msgr2=0x7eff341b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:45.667 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:45.654 7eff3387e700 1 --2- 172.21.2.102:0/2294327636 >> v2:172.21.2.113:3300/0 conn(0x7eff340e5070 0x7eff341b3820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:45.667 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:45.654 7eff3387e700 1 -- 172.21.2.102:0/2294327636 >> v1:172.21.2.113:6789/0 conn(0x7eff341b3e40 legacy=0x7eff3407d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:45.667 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:45.654 7eff3387e700 1 -- 172.21.2.102:0/2294327636 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff340d1bd0 con 0x7eff340e5470 2021-02-10T11:23:45.667 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:45.654 7eff3387e700 1 --2- 172.21.2.102:0/2294327636 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7eff340e5470 0x7eff34080310 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7eff2c006e50 tx=0x7eff2c009e70).ready entity=mon.2 client_cookie=99319aacaa7fa5b1 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:45.667 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:45.654 7eff3207b700 1 -- 172.21.2.102:0/2294327636 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7eff2c00b510 con 0x7eff340e5470 2021-02-10T11:23:45.667 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:45.654 7eff3207b700 1 -- 172.21.2.102:0/2294327636 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff2c006790 con 0x7eff340e5470 2021-02-10T11:23:45.667 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:45.654 7eff3207b700 1 -- 172.21.2.102:0/2294327636 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7eff2c000f90 con 0x7eff340e5470 2021-02-10T11:23:45.668 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:45.654 7eff39424700 1 -- 172.21.2.102:0/2294327636 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7eff340e5470 msgr2=0x7eff34080310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:45.668 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:45.654 7eff39424700 1 --2- 172.21.2.102:0/2294327636 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7eff340e5470 0x7eff34080310 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7eff2c006e50 tx=0x7eff2c009e70).stop 2021-02-10T11:23:45.668 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:45.655 7eff39424700 1 -- 172.21.2.102:0/2294327636 shutdown_connections 2021-02-10T11:23:45.668 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:45.655 7eff39424700 1 --2- 172.21.2.102:0/2294327636 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7eff340e5470 0x7eff34080310 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:45.668 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:45.655 7eff39424700 1 --2- 172.21.2.102:0/2294327636 >> v2:172.21.2.113:3300/0 conn(0x7eff340e5070 0x7eff341b3820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:45.668 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:45.655 7eff39424700 1 -- 172.21.2.102:0/2294327636 >> 172.21.2.102:0/2294327636 conn(0x7eff340ce690 msgr2=0x7eff340e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:45.668 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:45.655 7eff39424700 1 -- 172.21.2.102:0/2294327636 shutdown_connections 2021-02-10T11:23:45.668 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:45.655 7eff39424700 1 -- 172.21.2.102:0/2294327636 wait complete. 2021-02-10T11:23:45.670 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:45.669 7eff39424700 1 Processor -- start 2021-02-10T11:23:45.670 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:45.669 7eff39424700 1 -- start start 2021-02-10T11:23:45.670 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:45.670 7eff39424700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7eff34082520 0x7eff34087ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:45.670 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:45.670 7eff39424700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7eff340880d0 0x7eff340b1d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:45.670 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:45.670 7eff39424700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7eff340884d0 0x7eff340b2370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:45.670 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:45.670 7eff39424700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff340b17d0 con 0x7eff340884d0 2021-02-10T11:23:45.670 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:45.670 7eff39424700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff340b2990 con 0x7eff340880d0 2021-02-10T11:23:45.670 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:45.670 7eff39424700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff340b2b70 con 0x7eff34082520 2021-02-10T11:23:45.671 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:45.670 7eff3387e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7eff340884d0 0x7eff340b2370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:45.671 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:45.670 7eff3387e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7eff340884d0 0x7eff340b2370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:56632/0 (socket says 172.21.2.102:56632) 2021-02-10T11:23:45.671 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:45.670 7eff3387e700 1 -- 172.21.2.102:0/760065233 learned_addr learned my addr 172.21.2.102:0/760065233 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:23:45.671 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:45.670 7eff3307d700 1 --2- 172.21.2.102:0/760065233 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7eff34082520 0x7eff34087ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:45.671 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:45.670 7eff3287c700 1 --2- 172.21.2.102:0/760065233 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7eff340880d0 0x7eff340b1d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:45.671 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:45.670 7eff3307d700 1 -- 172.21.2.102:0/760065233 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7eff340880d0 msgr2=0x7eff340b1d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:45.671 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:45.670 7eff3307d700 1 --2- 172.21.2.102:0/760065233 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7eff340880d0 0x7eff340b1d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:45.671 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:45.670 7eff3307d700 1 -- 172.21.2.102:0/760065233 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7eff340884d0 msgr2=0x7eff340b2370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:45.671 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:45.670 7eff3307d700 1 --2- 172.21.2.102:0/760065233 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7eff340884d0 0x7eff340b2370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:45.671 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:45.670 7eff3307d700 1 -- 172.21.2.102:0/760065233 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff34088a70 con 0x7eff34082520 2021-02-10T11:23:45.671 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:45.670 7eff3387e700 1 --2- 172.21.2.102:0/760065233 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7eff340884d0 0x7eff340b2370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:23:45.671 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:45.671 7eff3307d700 1 --2- 172.21.2.102:0/760065233 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7eff34082520 0x7eff34087ab0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7eff24009540 tx=0x7eff2400b8e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:45.671 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:45.671 7eff2bfff700 1 -- 172.21.2.102:0/760065233 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7eff240009b0 con 0x7eff34082520 2021-02-10T11:23:45.672 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:45.671 7eff2bfff700 1 -- 172.21.2.102:0/760065233 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff2400bd60 con 0x7eff34082520 2021-02-10T11:23:45.672 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:45.671 7eff2bfff700 1 -- 172.21.2.102:0/760065233 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7eff24002030 con 0x7eff34082520 2021-02-10T11:23:45.672 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:45.671 7eff39424700 1 -- 172.21.2.102:0/760065233 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff34088a70 con 0x7eff34082520 2021-02-10T11:23:45.672 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:45.671 7eff39424700 1 -- 172.21.2.102:0/760065233 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff34088a70 con 0x7eff34082520 2021-02-10T11:23:45.672 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:45.672 7eff2bfff700 1 -- 172.21.2.102:0/760065233 <== mon.2 v2:172.21.2.113:3301/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7eff24000c90 con 0x7eff34082520 2021-02-10T11:23:45.672 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:45.672 7eff2bfff700 1 --2- 172.21.2.102:0/760065233 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7eff180186c0 0x7eff1801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:45.672 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:45.672 7eff2bfff700 1 -- 172.21.2.102:0/760065233 <== mon.2 v2:172.21.2.113:3301/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5573+0+0 (secure 0 0 0) 0x7eff240324c0 con 0x7eff34082520 2021-02-10T11:23:45.672 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:45.672 7eff3287c700 1 --2- 172.21.2.102:0/760065233 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7eff180186c0 0x7eff1801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:45.672 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:45.672 7eff3287c700 1 --2- 172.21.2.102:0/760065233 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7eff180186c0 0x7eff1801abc0 crc :-1 s=READY pgs=592 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:45.700 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:45.703 7eff39424700 1 -- 172.21.2.102:0/760065233 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff34107180 con 0x7eff34082520 2021-02-10T11:23:45.703 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:45.706 7eff2bfff700 1 -- 172.21.2.102:0/760065233 <== mon.2 v2:172.21.2.113:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7eff2402f670 con 0x7eff34082520 2021-02-10T11:23:45.837 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:45.840 7eff39424700 1 -- 172.21.2.102:0/760065233 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_command({"prefix": "auth rm", "entity": "client.guest1"} v 0) v1 -- 0x7eff34107180 con 0x7eff34082520 2021-02-10T11:23:45.843 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:45.846 7eff2bfff700 1 -- 172.21.2.102:0/760065233 <== mon.2 v2:172.21.2.113:3301/0 7 ==== mon_command_ack([{"prefix": "auth rm", "entity": "client.guest1"}]=0 updated v28) v1 ==== 89+0+0 (secure 0 0 0) 0x7eff2402f670 con 0x7eff34082520 2021-02-10T11:23:45.844 INFO:teuthology.orchestra.run.gibba002.stderr:updated 2021-02-10T11:23:45.846 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:45.849 7eff39424700 1 -- 172.21.2.102:0/760065233 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7eff180186c0 msgr2=0x7eff1801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:45.846 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:45.849 7eff39424700 1 --2- 172.21.2.102:0/760065233 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7eff180186c0 0x7eff1801abc0 crc :-1 s=READY pgs=592 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:45.846 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:45.849 7eff39424700 1 -- 172.21.2.102:0/760065233 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7eff34082520 msgr2=0x7eff34087ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:45.846 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:45.849 7eff39424700 1 --2- 172.21.2.102:0/760065233 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7eff34082520 0x7eff34087ab0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7eff24009540 tx=0x7eff2400b8e0).stop 2021-02-10T11:23:45.846 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:45.849 7eff39424700 1 -- 172.21.2.102:0/760065233 shutdown_connections 2021-02-10T11:23:45.846 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:45.849 7eff39424700 1 --2- 172.21.2.102:0/760065233 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7eff340884d0 0x7eff340b2370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:45.846 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:45.849 7eff39424700 1 --2- 172.21.2.102:0/760065233 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7eff180186c0 0x7eff1801abc0 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:45.846 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:45.849 7eff39424700 1 --2- 172.21.2.102:0/760065233 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7eff340880d0 0x7eff340b1d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:45.846 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:45.849 7eff39424700 1 --2- 172.21.2.102:0/760065233 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7eff34082520 0x7eff34087ab0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:45.847 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:45.849 7eff39424700 1 -- 172.21.2.102:0/760065233 >> 172.21.2.102:0/760065233 conn(0x7eff34098740 msgr2=0x7eff340c3590 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:45.847 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:45.849 7eff39424700 1 -- 172.21.2.102:0/760065233 shutdown_connections 2021-02-10T11:23:45.847 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:45.849 7eff39424700 1 -- 172.21.2.102:0/760065233 wait complete. 2021-02-10T11:23:45.854 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000647253 --group_name group_0000000000353920 2021-02-10T11:23:46.286 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:46.271 7f32e628c700 1 Processor -- start 2021-02-10T11:23:46.286 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:46.271 7f32e628c700 1 -- start start 2021-02-10T11:23:46.286 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:46.272 7f32e628c700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f32e00d1860 0x7f32e01b03e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:46.286 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:46.272 7f32e628c700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f32e01b0a00 0x7f32e00cf320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:46.286 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:46.272 7f32e628c700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32e00f0b00 con 0x7f32e00d1c60 2021-02-10T11:23:46.286 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:46.272 7f32e628c700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32e00fe220 con 0x7f32e01b0a00 2021-02-10T11:23:46.286 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:46.272 7f32e628c700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32e00e5ec0 con 0x7f32e00d1860 2021-02-10T11:23:46.286 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:46.273 7f32dffff700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f32e00d1860 0x7f32e01b03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:46.286 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:46.273 7f32e48a9700 1 --1- >> v1:172.21.2.113:6789/0 conn(0x7f32e00d1c60 0x7f32e00cf940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.113:6789/0 says I am v1:172.21.2.102:56188/0 (socket says 172.21.2.102:56188) 2021-02-10T11:23:46.287 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:46.273 7f32e48a9700 1 -- 172.21.2.102:0/4020936278 learned_addr learned my addr 172.21.2.102:0/4020936278 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:23:46.287 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:46.273 7f32df7fe700 1 --2- 172.21.2.102:0/4020936278 >> v2:172.21.2.113:3300/0 conn(0x7f32e01b0a00 0x7f32e00cf320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:46.287 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:46.273 7f32dffff700 1 -- 172.21.2.102:0/4020936278 >> v2:172.21.2.113:3300/0 conn(0x7f32e01b0a00 msgr2=0x7f32e00cf320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:46.287 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:46.273 7f32dffff700 1 --2- 172.21.2.102:0/4020936278 >> v2:172.21.2.113:3300/0 conn(0x7f32e01b0a00 0x7f32e00cf320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:46.287 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:46.273 7f32dffff700 1 -- 172.21.2.102:0/4020936278 >> v1:172.21.2.113:6789/0 conn(0x7f32e00d1c60 legacy=0x7f32e00cf940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:46.287 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:46.273 7f32dffff700 1 -- 172.21.2.102:0/4020936278 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32e00f0640 con 0x7f32e00d1860 2021-02-10T11:23:46.287 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:46.273 7f32df7fe700 1 --2- 172.21.2.102:0/4020936278 >> v2:172.21.2.113:3300/0 conn(0x7f32e01b0a00 0x7f32e00cf320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:23:46.287 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:46.273 7f32dffff700 1 --2- 172.21.2.102:0/4020936278 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f32e00d1860 0x7f32e01b03e0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f32d0006650 tx=0x7f32d0009a80).ready entity=mon.2 client_cookie=a921da7e369a5d19 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:46.287 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:46.274 7f32deffd700 1 -- 172.21.2.102:0/4020936278 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f32d00009b0 con 0x7f32e00d1860 2021-02-10T11:23:46.287 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:46.274 7f32deffd700 1 -- 172.21.2.102:0/4020936278 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32d000c0c0 con 0x7f32e00d1860 2021-02-10T11:23:46.287 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:46.274 7f32deffd700 1 -- 172.21.2.102:0/4020936278 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f32d000c5e0 con 0x7f32e00d1860 2021-02-10T11:23:46.287 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:46.274 7f32e628c700 1 -- 172.21.2.102:0/4020936278 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f32e00d1860 msgr2=0x7f32e01b03e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:46.287 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:46.274 7f32e628c700 1 --2- 172.21.2.102:0/4020936278 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f32e00d1860 0x7f32e01b03e0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f32d0006650 tx=0x7f32d0009a80).stop 2021-02-10T11:23:46.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:46.274 7f32e628c700 1 -- 172.21.2.102:0/4020936278 shutdown_connections 2021-02-10T11:23:46.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:46.274 7f32e628c700 1 --2- 172.21.2.102:0/4020936278 >> v2:172.21.2.113:3300/0 conn(0x7f32e01b0a00 0x7f32e00cf320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:46.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:46.274 7f32e628c700 1 --2- 172.21.2.102:0/4020936278 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f32e00d1860 0x7f32e01b03e0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:46.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:46.274 7f32e628c700 1 -- 172.21.2.102:0/4020936278 >> 172.21.2.102:0/4020936278 conn(0x7f32e00ce440 msgr2=0x7f32e00d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:46.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:46.274 7f32e628c700 1 -- 172.21.2.102:0/4020936278 shutdown_connections 2021-02-10T11:23:46.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:46.274 7f32e628c700 1 -- 172.21.2.102:0/4020936278 wait complete. 2021-02-10T11:23:46.289 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:46.289 7f32e628c700 1 Processor -- start 2021-02-10T11:23:46.289 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:46.289 7f32e628c700 1 -- start start 2021-02-10T11:23:46.289 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:46.289 7f32e628c700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f32e00b0a20 0x7f32e00afe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:46.290 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:46.289 7f32e628c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f32e00ebe60 0x7f32e00ec260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:46.290 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:46.289 7f32e628c700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f32e00ecf20 0x7f32e00ed320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:46.290 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:46.289 7f32e628c700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32e00ec9a0 con 0x7f32e00ebe60 2021-02-10T11:23:46.290 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:46.289 7f32e628c700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32e00b04a0 con 0x7f32e00ecf20 2021-02-10T11:23:46.290 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:46.289 7f32e628c700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32e00b0680 con 0x7f32e00b0a20 2021-02-10T11:23:46.290 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:46.290 7f32df7fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f32e00ebe60 0x7f32e00ec260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:46.290 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:46.290 7f32df7fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f32e00ebe60 0x7f32e00ec260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:56646/0 (socket says 172.21.2.102:56646) 2021-02-10T11:23:46.290 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:46.290 7f32df7fe700 1 -- 172.21.2.102:0/3381536076 learned_addr learned my addr 172.21.2.102:0/3381536076 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:23:46.290 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:46.290 7f32df7fe700 1 -- 172.21.2.102:0/3381536076 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f32e00b0a20 msgr2=0x7f32e00afe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:46.290 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:46.290 7f32df7fe700 1 --2- 172.21.2.102:0/3381536076 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f32e00b0a20 0x7f32e00afe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:46.290 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:46.290 7f32df7fe700 1 -- 172.21.2.102:0/3381536076 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f32e00ecf20 msgr2=0x7f32e00ed320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:46.290 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:46.290 7f32df7fe700 1 --2- 172.21.2.102:0/3381536076 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f32e00ecf20 0x7f32e00ed320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:46.291 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:46.290 7f32df7fe700 1 -- 172.21.2.102:0/3381536076 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32e00ed970 con 0x7f32e00ebe60 2021-02-10T11:23:46.291 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:46.290 7f32df7fe700 1 --2- 172.21.2.102:0/3381536076 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f32e00ebe60 0x7f32e00ec260 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 rx=0x7f32d4006f40 tx=0x7f32d400b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:46.291 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:46.290 7f32dcff9700 1 -- 172.21.2.102:0/3381536076 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f32d40009b0 con 0x7f32e00ebe60 2021-02-10T11:23:46.291 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:46.290 7f32dcff9700 1 -- 172.21.2.102:0/3381536076 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32d400bd80 con 0x7f32e00ebe60 2021-02-10T11:23:46.291 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:46.290 7f32dcff9700 1 -- 172.21.2.102:0/3381536076 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f32d40022b0 con 0x7f32e00ebe60 2021-02-10T11:23:46.291 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:46.290 7f32e628c700 1 -- 172.21.2.102:0/3381536076 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32e00ed970 con 0x7f32e00ebe60 2021-02-10T11:23:46.291 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:46.290 7f32e628c700 1 -- 172.21.2.102:0/3381536076 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32e00ed970 con 0x7f32e00ebe60 2021-02-10T11:23:46.291 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:46.291 7f32dcff9700 1 -- 172.21.2.102:0/3381536076 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f32d4000c90 con 0x7f32e00ebe60 2021-02-10T11:23:46.291 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:46.291 7f32dcff9700 1 --2- 172.21.2.102:0/3381536076 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f32c00186c0 0x7f32c001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:46.291 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:46.291 7f32dcff9700 1 -- 172.21.2.102:0/3381536076 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f32d4032410 con 0x7f32e00ebe60 2021-02-10T11:23:46.291 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:46.291 7f32dffff700 1 --2- 172.21.2.102:0/3381536076 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f32c00186c0 0x7f32c001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:46.291 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:46.292 7f32dffff700 1 --2- 172.21.2.102:0/3381536076 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f32c00186c0 0x7f32c001abc0 crc :-1 s=READY pgs=593 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:46.319 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:46.322 7f32e628c700 1 -- 172.21.2.102:0/3381536076 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32e00ebc30 con 0x7f32e00ebe60 2021-02-10T11:23:46.321 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:46.324 7f32dcff9700 1 -- 172.21.2.102:0/3381536076 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f32d4006bd0 con 0x7f32e00ebe60 2021-02-10T11:23:46.449 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:46.452 7f32e628c700 1 -- 172.21.2.102:0/3381536076 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000647253", "target": ["mgr", ""], "group_name": "group_0000000000353920"}) v1 -- 0x7f32e00b04a0 con 0x7f32c00186c0 2021-02-10T11:23:46.458 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:46.461 7f32dcff9700 1 -- 172.21.2.102:0/3381536076 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f32d0000940 con 0x7f32c00186c0 2021-02-10T11:23:46.466 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:46.468 7f32e628c700 1 -- 172.21.2.102:0/3381536076 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f32c00186c0 msgr2=0x7f32c001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:46.466 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:46.468 7f32e628c700 1 --2- 172.21.2.102:0/3381536076 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f32c00186c0 0x7f32c001abc0 crc :-1 s=READY pgs=593 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:46.466 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:46.468 7f32e628c700 1 -- 172.21.2.102:0/3381536076 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f32e00ebe60 msgr2=0x7f32e00ec260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:46.466 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:46.468 7f32e628c700 1 --2- 172.21.2.102:0/3381536076 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f32e00ebe60 0x7f32e00ec260 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 rx=0x7f32d4006f40 tx=0x7f32d400b990).stop 2021-02-10T11:23:46.466 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:46.469 7f32e628c700 1 -- 172.21.2.102:0/3381536076 shutdown_connections 2021-02-10T11:23:46.466 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:46.469 7f32e628c700 1 --2- 172.21.2.102:0/3381536076 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f32c00186c0 0x7f32c001abc0 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:46.466 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:46.469 7f32e628c700 1 --2- 172.21.2.102:0/3381536076 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f32e00ecf20 0x7f32e00ed320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:46.466 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:46.469 7f32e628c700 1 --2- 172.21.2.102:0/3381536076 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f32e00ebe60 0x7f32e00ec260 unknown :-1 s=CLOSED pgs=1255 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:46.466 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:46.469 7f32e628c700 1 --2- 172.21.2.102:0/3381536076 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f32e00b0a20 0x7f32e00afe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:46.466 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:46.469 7f32e628c700 1 -- 172.21.2.102:0/3381536076 >> 172.21.2.102:0/3381536076 conn(0x7f32e00c2a00 msgr2=0x7f32e00ad7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:46.467 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:46.469 7f32e628c700 1 -- 172.21.2.102:0/3381536076 shutdown_connections 2021-02-10T11:23:46.467 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:46.469 7f32e628c700 1 -- 172.21.2.102:0/3381536076 wait complete. 2021-02-10T11:23:46.475 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolumegroup rm cephfs group_0000000000353920 2021-02-10T11:23:46.907 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:46.892 7f3d01f18700 1 Processor -- start 2021-02-10T11:23:46.907 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:46.892 7f3d01f18700 1 -- start start 2021-02-10T11:23:46.907 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:46.893 7f3d01f18700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3cfc1b0470 0x7f3cfc1b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:46.907 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:46.893 7f3d01f18700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f3cfc0f20e0 0x7f3cfc0f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:46.907 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:46.893 7f3d01f18700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cfc0f0b50 con 0x7f3cfc0ef610 2021-02-10T11:23:46.907 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:46.893 7f3d01f18700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3cfc0f0820 con 0x7f3cfc0f20e0 2021-02-10T11:23:46.907 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:46.893 7f3d01f18700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3cfc0f24e0 con 0x7f3cfc1b0470 2021-02-10T11:23:46.907 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:46.893 7f3cfaffd700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f3cfc0ef610 0x7f3cfc0efa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:48874/0 (socket says 172.21.2.102:48874) 2021-02-10T11:23:46.908 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:46.893 7f3cfaffd700 1 -- 172.21.2.102:0/702101177 learned_addr learned my addr 172.21.2.102:0/702101177 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:23:46.908 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:46.893 7f3cfa7fc700 1 -- 172.21.2.102:0/702101177 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 892970088 0 0) 0x7f3ce40010b0 con 0x7f3cfc0ef610 2021-02-10T11:23:46.908 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:46.893 7f3cfa7fc700 1 -- 172.21.2.102:0/702101177 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ce8002fe0 con 0x7f3cfc0ef610 2021-02-10T11:23:46.908 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:46.893 7f3cfb7fe700 1 --2- 172.21.2.102:0/702101177 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3cfc1b0470 0x7f3cfc1b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:46.908 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:46.893 7f3cfbfff700 1 --2- 172.21.2.102:0/702101177 >> v2:172.21.2.113:3300/0 conn(0x7f3cfc0f20e0 0x7f3cfc0f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:46.908 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:46.894 7f3cfa7fc700 1 -- 172.21.2.102:0/702101177 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1983645251 0 0) 0x7f3ce40010b0 con 0x7f3cfc0ef610 2021-02-10T11:23:46.908 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:46.894 7f3cfa7fc700 1 -- 172.21.2.102:0/702101177 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3cfc1b0470 msgr2=0x7f3cfc1b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:46.908 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:46.894 7f3cfa7fc700 1 --2- 172.21.2.102:0/702101177 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3cfc1b0470 0x7f3cfc1b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:46.908 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:46.894 7f3cfa7fc700 1 -- 172.21.2.102:0/702101177 >> v2:172.21.2.113:3300/0 conn(0x7f3cfc0f20e0 msgr2=0x7f3cfc0f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:46.908 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:46.894 7f3cfa7fc700 1 --2- 172.21.2.102:0/702101177 >> v2:172.21.2.113:3300/0 conn(0x7f3cfc0f20e0 0x7f3cfc0f0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:46.908 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:46.894 7f3cfa7fc700 1 -- 172.21.2.102:0/702101177 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cfc0e52e0 con 0x7f3cfc0ef610 2021-02-10T11:23:46.908 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:46.894 7f3cfa7fc700 1 -- 172.21.2.102:0/702101177 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7f3ce40015b0 con 0x7f3cfc0ef610 2021-02-10T11:23:46.908 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:46.894 7f3cfa7fc700 1 -- 172.21.2.102:0/702101177 >> v1:172.21.2.102:6789/0 conn(0x7f3cfc0ef610 legacy=0x7f3cfc0efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:46.908 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:46.894 7f3cfa7fc700 1 --2- 172.21.2.102:0/702101177 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3ce8003a60 0x7f3ce8005f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:46.909 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:46.894 7f3cfa7fc700 1 --2- 172.21.2.102:0/702101177 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3ce804a760 0x7f3ce804cc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:46.909 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:46.894 7f3cfa7fc700 1 --2- 172.21.2.102:0/702101177 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3ce804d500 0x7f3ce804fa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:46.909 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:46.894 7f3cfa7fc700 1 -- 172.21.2.102:0/702101177 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ce80502d0 con 0x7f3ce804d500 2021-02-10T11:23:46.909 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:46.894 7f3cfa7fc700 1 -- 172.21.2.102:0/702101177 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ce80504f0 con 0x7f3ce804a760 2021-02-10T11:23:46.909 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:46.894 7f3cfa7fc700 1 -- 172.21.2.102:0/702101177 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ce8050710 con 0x7f3ce8003a60 2021-02-10T11:23:46.909 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:46.894 7f3cfbfff700 1 --2- 172.21.2.102:0/702101177 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3ce804d500 0x7f3ce804fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:46.909 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:46.894 7f3cfb7fe700 1 --2- 172.21.2.102:0/702101177 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3ce8003a60 0x7f3ce8005f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:46.909 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:46.894 7f3cfaffd700 1 --2- 172.21.2.102:0/702101177 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3ce804a760 0x7f3ce804cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:46.909 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:46.894 7f3cfbfff700 1 -- 172.21.2.102:0/702101177 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3ce8003a60 msgr2=0x7f3ce8005f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:46.909 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:46.894 7f3cfbfff700 1 --2- 172.21.2.102:0/702101177 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3ce8003a60 0x7f3ce8005f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:46.909 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:46.894 7f3cfbfff700 1 -- 172.21.2.102:0/702101177 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3ce804a760 msgr2=0x7f3ce804cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:46.909 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:46.894 7f3cfbfff700 1 --2- 172.21.2.102:0/702101177 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3ce804a760 0x7f3ce804cc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:46.909 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:46.894 7f3cfbfff700 1 -- 172.21.2.102:0/702101177 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ce80509b0 con 0x7f3ce804d500 2021-02-10T11:23:46.909 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:46.894 7f3cfb7fe700 1 --2- 172.21.2.102:0/702101177 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3ce8003a60 0x7f3ce8005f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:23:46.910 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:46.894 7f3cfaffd700 1 --2- 172.21.2.102:0/702101177 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3ce804a760 0x7f3ce804cc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:23:46.910 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:46.894 7f3cfbfff700 1 --2- 172.21.2.102:0/702101177 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3ce804d500 0x7f3ce804fa20 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 rx=0x7f3cf4006a40 tx=0x7f3cf400c0c0).ready entity=mon.0 client_cookie=483ff0da9abcbf5e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:46.910 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:46.895 7f3cfa7fc700 1 -- 172.21.2.102:0/702101177 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3cf400c4b0 con 0x7f3ce804d500 2021-02-10T11:23:46.910 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:46.895 7f3cfa7fc700 1 -- 172.21.2.102:0/702101177 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cf400c770 con 0x7f3ce804d500 2021-02-10T11:23:46.910 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:46.895 7f3cfa7fc700 1 -- 172.21.2.102:0/702101177 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3cf400cbd0 con 0x7f3ce804d500 2021-02-10T11:23:46.910 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:46.895 7f3d01f18700 1 -- 172.21.2.102:0/702101177 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3ce804d500 msgr2=0x7f3ce804fa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:46.910 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:46.895 7f3d01f18700 1 --2- 172.21.2.102:0/702101177 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3ce804d500 0x7f3ce804fa20 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 rx=0x7f3cf4006a40 tx=0x7f3cf400c0c0).stop 2021-02-10T11:23:46.910 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:46.895 7f3cfbfff700 1 -- 172.21.2.102:0/702101177 reap_dead start 2021-02-10T11:23:46.910 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:46.895 7f3d01f18700 1 -- 172.21.2.102:0/702101177 shutdown_connections 2021-02-10T11:23:46.910 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:46.895 7f3d01f18700 1 -- 172.21.2.102:0/702101177 >> 172.21.2.102:0/702101177 conn(0x7f3cfc0ce3e0 msgr2=0x7f3cfc0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:46.910 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:46.895 7f3d01f18700 1 -- 172.21.2.102:0/702101177 shutdown_connections 2021-02-10T11:23:46.910 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:46.895 7f3d01f18700 1 -- 172.21.2.102:0/702101177 wait complete. 2021-02-10T11:23:46.910 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:46.910 7f3d01f18700 1 Processor -- start 2021-02-10T11:23:46.911 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:46.910 7f3d01f18700 1 -- start start 2021-02-10T11:23:46.912 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:46.910 7f3d01f18700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3cfc09ba50 0x7f3cfc10a110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:46.912 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:46.910 7f3d01f18700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3cfc0c27f0 0x7f3cfc1095c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:46.912 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:46.910 7f3d01f18700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3cfc0c32d0 0x7f3cfc0c36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:46.912 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:46.910 7f3d01f18700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cfc199f50 con 0x7f3cfc09ba50 2021-02-10T11:23:46.912 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:46.910 7f3d01f18700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cfc0c2d50 con 0x7f3cfc0c32d0 2021-02-10T11:23:46.912 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:46.910 7f3d01f18700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3cfc10a730 con 0x7f3cfc0c27f0 2021-02-10T11:23:46.912 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:46.910 7f3cfb7fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3cfc09ba50 0x7f3cfc10a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:46.912 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:46.910 7f3cfb7fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3cfc09ba50 0x7f3cfc10a110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:56664/0 (socket says 172.21.2.102:56664) 2021-02-10T11:23:46.913 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:46.911 7f3cfb7fe700 1 -- 172.21.2.102:0/3327281981 learned_addr learned my addr 172.21.2.102:0/3327281981 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:23:46.913 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:46.911 7f3cfb7fe700 1 -- 172.21.2.102:0/3327281981 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3cfc0c27f0 msgr2=0x7f3cfc1095c0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:23:46.913 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:46.911 7f3cfb7fe700 1 --2- 172.21.2.102:0/3327281981 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3cfc0c27f0 0x7f3cfc1095c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:46.913 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:46.911 7f3cfb7fe700 1 -- 172.21.2.102:0/3327281981 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3cfc0c32d0 msgr2=0x7f3cfc0c36d0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:23:46.913 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:46.911 7f3cfb7fe700 1 --2- 172.21.2.102:0/3327281981 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3cfc0c32d0 0x7f3cfc0c36d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:46.913 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:46.911 7f3cfb7fe700 1 -- 172.21.2.102:0/3327281981 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cfc09be50 con 0x7f3cfc09ba50 2021-02-10T11:23:46.913 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:46.911 7f3cfb7fe700 1 --2- 172.21.2.102:0/3327281981 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3cfc09ba50 0x7f3cfc10a110 secure :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 rx=0x7f3cec00b510 tx=0x7f3cec00b8b0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:46.913 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:46.911 7f3cf3fff700 1 -- 172.21.2.102:0/3327281981 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3cec000c90 con 0x7f3cfc09ba50 2021-02-10T11:23:46.913 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:46.911 7f3cf3fff700 1 -- 172.21.2.102:0/3327281981 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cec00bca0 con 0x7f3cfc09ba50 2021-02-10T11:23:46.913 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:46.911 7f3cf3fff700 1 -- 172.21.2.102:0/3327281981 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3cec002220 con 0x7f3cfc09ba50 2021-02-10T11:23:46.913 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:46.911 7f3d01f18700 1 -- 172.21.2.102:0/3327281981 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3cfc09be50 con 0x7f3cfc09ba50 2021-02-10T11:23:46.913 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:46.911 7f3d01f18700 1 -- 172.21.2.102:0/3327281981 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3cfc0c3cf0 con 0x7f3cfc09ba50 2021-02-10T11:23:46.913 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:46.912 7f3cf3fff700 1 -- 172.21.2.102:0/3327281981 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f3cec000c90 con 0x7f3cfc09ba50 2021-02-10T11:23:46.914 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:46.912 7f3cf3fff700 1 --2- 172.21.2.102:0/3327281981 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3ce00186c0 0x7f3ce001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:46.914 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:46.912 7f3cf3fff700 1 -- 172.21.2.102:0/3327281981 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f3cec034790 con 0x7f3cfc09ba50 2021-02-10T11:23:46.914 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:46.912 7f3cfaffd700 1 --2- 172.21.2.102:0/3327281981 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3ce00186c0 0x7f3ce001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:46.914 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:46.912 7f3cfaffd700 1 --2- 172.21.2.102:0/3327281981 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3ce00186c0 0x7f3ce001abc0 crc :-1 s=READY pgs=594 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:46.939 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:46.942 7f3d01f18700 1 -- 172.21.2.102:0/3327281981 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3cfc199f50 con 0x7f3cfc09ba50 2021-02-10T11:23:46.942 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:46.945 7f3cf3fff700 1 -- 172.21.2.102:0/3327281981 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f3cec006b00 con 0x7f3cfc09ba50 2021-02-10T11:23:47.072 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:47.075 7f3d01f18700 1 -- 172.21.2.102:0/3327281981 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolumegroup rm", "vol_name": "cephfs", "target": ["mgr", ""], "group_name": "group_0000000000353920"}) v1 -- 0x7f3cfc199f50 con 0x7f3ce00186c0 2021-02-10T11:23:47.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:47.079 7f3cf3fff700 1 -- 172.21.2.102:0/3327281981 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f3ce4000e10 con 0x7f3ce00186c0 2021-02-10T11:23:47.081 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:47.084 7f3d01f18700 1 -- 172.21.2.102:0/3327281981 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3ce00186c0 msgr2=0x7f3ce001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:47.082 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:47.084 7f3d01f18700 1 --2- 172.21.2.102:0/3327281981 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3ce00186c0 0x7f3ce001abc0 crc :-1 s=READY pgs=594 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:47.082 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:47.084 7f3d01f18700 1 -- 172.21.2.102:0/3327281981 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3cfc09ba50 msgr2=0x7f3cfc10a110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:47.082 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:47.084 7f3d01f18700 1 --2- 172.21.2.102:0/3327281981 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3cfc09ba50 0x7f3cfc10a110 secure :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 rx=0x7f3cec00b510 tx=0x7f3cec00b8b0).stop 2021-02-10T11:23:47.082 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:47.084 7f3d01f18700 1 -- 172.21.2.102:0/3327281981 shutdown_connections 2021-02-10T11:23:47.082 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:47.084 7f3d01f18700 1 --2- 172.21.2.102:0/3327281981 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3ce00186c0 0x7f3ce001abc0 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:47.082 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:47.084 7f3d01f18700 1 --2- 172.21.2.102:0/3327281981 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3cfc0c32d0 0x7f3cfc0c36d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:47.082 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:47.084 7f3d01f18700 1 --2- 172.21.2.102:0/3327281981 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3cfc0c27f0 0x7f3cfc1095c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:47.082 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:47.084 7f3d01f18700 1 --2- 172.21.2.102:0/3327281981 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3cfc09ba50 0x7f3cfc10a110 unknown :-1 s=CLOSED pgs=1258 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:47.082 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:47.084 7f3d01f18700 1 -- 172.21.2.102:0/3327281981 >> 172.21.2.102:0/3327281981 conn(0x7f3cfc0c40e0 msgr2=0x7f3cfc0bf620 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:47.082 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:47.085 7f3d01f18700 1 -- 172.21.2.102:0/3327281981 shutdown_connections 2021-02-10T11:23:47.082 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:47.085 7f3d01f18700 1 -- 172.21.2.102:0/3327281981 wait complete. 2021-02-10T11:23:47.090 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2021-02-10T11:23:47.090 DEBUG:teuthology.orchestra.run.gibba002:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:23:47.435 DEBUG:teuthology.orchestra.run.gibba013:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:23:47.780 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2021-02-10T11:23:47.780 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:23:47.780 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:23:47.781 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:23:48.117 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:23:48.117 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:23:48.117 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@gibba002.front.sepia.ceph.com... 2021-02-10T11:23:48.117 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-02-10T11:23:48.117 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:23:48.472 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba002.stderr:ceph-fuse[149289]: fuse finished with error 0 and tester_r 0 2021-02-10T11:23:48.474 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:23:48.474 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:23:48.474 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:23:48.807 INFO:teuthology.orchestra.run.gibba002.stdout:ext2/ext3 2021-02-10T11:23:48.808 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2021-02-10T11:23:48.808 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:23:48.808 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:23:49.144 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:23:49.144 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:23:49.144 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:23:49.479 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:23:49.479 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2021-02-10T11:23:49.479 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@gibba002.front.sepia.ceph.com... 2021-02-10T11:23:49.479 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-02-10T11:23:49.479 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:23:49.829 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba002.stderr:ceph-fuse[149956]: fuse finished with error 0 and tester_r 0 2021-02-10T11:23:49.831 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:23:49.831 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:23:49.831 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:23:50.166 INFO:teuthology.orchestra.run.gibba002.stdout:ext2/ext3 2021-02-10T11:23:50.166 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2021-02-10T11:23:50.166 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:23:50.167 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:23:50.502 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:23:50.503 DEBUG:teuthology.orchestra.run.gibba013:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:23:50.503 DEBUG:teuthology.orchestra.run.gibba013:> ' -- /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:23:50.838 INFO:teuthology.orchestra.run.gibba013.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.2’: No such file or directory 2021-02-10T11:23:50.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:23:50.839 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2021-02-10T11:23:50.839 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:23:50.839 DEBUG:teuthology.orchestra.run.gibba013:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:23:51.174 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:23:51.174 DEBUG:teuthology.orchestra.run.gibba013:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:23:51.174 DEBUG:teuthology.orchestra.run.gibba013:> ' -- /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:23:51.508 INFO:teuthology.orchestra.run.gibba013.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.3’: No such file or directory 2021-02-10T11:23:51.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:23:51.509 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.3 2021-02-10T11:23:51.509 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:23:51.509 DEBUG:teuthology.orchestra.run.gibba013:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:23:51.844 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config rm mon mon_allow_pool_delete 2021-02-10T11:23:52.276 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:52.261 7f1f353b1700 1 Processor -- start 2021-02-10T11:23:52.277 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:52.261 7f1f353b1700 1 -- start start 2021-02-10T11:23:52.277 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:52.262 7f1f353b1700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f1f300d17f0 0x7f1f301b31e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:52.277 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:52.262 7f1f353b1700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1f300d1bf0 0x7f1f301b3800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:52.277 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:52.262 7f1f353b1700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f300f0b20 con 0x7f1f301b3e20 2021-02-10T11:23:52.277 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:52.262 7f1f353b1700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f300e5eb0 con 0x7f1f300d17f0 2021-02-10T11:23:52.277 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:52.262 7f1f353b1700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f300f0630 con 0x7f1f300d1bf0 2021-02-10T11:23:52.277 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:52.262 7f1f2f87e700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f1f301b3e20 0x7f1f300ceef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:48894/0 (socket says 172.21.2.102:48894) 2021-02-10T11:23:52.277 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:52.262 7f1f2f87e700 1 -- 172.21.2.102:0/2212579160 learned_addr learned my addr 172.21.2.102:0/2212579160 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:23:52.277 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:52.262 7f1f2f07d700 1 --2- 172.21.2.102:0/2212579160 >> v2:172.21.2.102:3300/0 conn(0x7f1f300d17f0 0x7f1f301b31e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:52.277 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:52.262 7f1f2e07b700 1 -- 172.21.2.102:0/2212579160 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3282869396 0 0) 0x7f1f28001500 con 0x7f1f301b3e20 2021-02-10T11:23:52.277 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:52.262 7f1f2e07b700 1 -- 172.21.2.102:0/2212579160 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f18002fe0 con 0x7f1f301b3e20 2021-02-10T11:23:52.278 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:52.262 7f1f2f07d700 1 -- 172.21.2.102:0/2212579160 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1f300d1bf0 msgr2=0x7f1f301b3800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:52.278 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:52.262 7f1f2e87c700 1 --2- 172.21.2.102:0/2212579160 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1f300d1bf0 0x7f1f301b3800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:52.278 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:52.262 7f1f2f07d700 1 --2- 172.21.2.102:0/2212579160 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1f300d1bf0 0x7f1f301b3800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:52.278 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:52.263 7f1f2f07d700 1 -- 172.21.2.102:0/2212579160 >> v1:172.21.2.102:6789/0 conn(0x7f1f301b3e20 legacy=0x7f1f300ceef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:52.278 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:52.263 7f1f2f07d700 1 -- 172.21.2.102:0/2212579160 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f300f0820 con 0x7f1f300d17f0 2021-02-10T11:23:52.278 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:52.263 7f1f2e87c700 1 --2- 172.21.2.102:0/2212579160 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1f300d1bf0 0x7f1f301b3800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:23:52.278 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:52.263 7f1f2f07d700 1 --2- 172.21.2.102:0/2212579160 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1f300d17f0 0x7f1f301b31e0 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 rx=0x7f1f20006650 tx=0x7f1f20009a80).ready entity=mon.0 client_cookie=861b84a6d8d7e0bb server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:52.278 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:52.263 7f1f2e07b700 1 -- 172.21.2.102:0/2212579160 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1f200009b0 con 0x7f1f300d17f0 2021-02-10T11:23:52.278 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:52.263 7f1f2e07b700 1 -- 172.21.2.102:0/2212579160 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f2000c0c0 con 0x7f1f300d17f0 2021-02-10T11:23:52.278 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:52.263 7f1f2e07b700 1 -- 172.21.2.102:0/2212579160 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1f2000c5e0 con 0x7f1f300d17f0 2021-02-10T11:23:52.278 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:52.263 7f1f353b1700 1 -- 172.21.2.102:0/2212579160 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1f300d17f0 msgr2=0x7f1f301b31e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:52.278 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:52.263 7f1f353b1700 1 --2- 172.21.2.102:0/2212579160 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1f300d17f0 0x7f1f301b31e0 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 rx=0x7f1f20006650 tx=0x7f1f20009a80).stop 2021-02-10T11:23:52.279 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:52.264 7f1f353b1700 1 -- 172.21.2.102:0/2212579160 shutdown_connections 2021-02-10T11:23:52.279 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:52.264 7f1f353b1700 1 --2- 172.21.2.102:0/2212579160 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1f300d1bf0 0x7f1f301b3800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:52.279 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:52.264 7f1f353b1700 1 --2- 172.21.2.102:0/2212579160 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1f300d17f0 0x7f1f301b31e0 unknown :-1 s=CLOSED pgs=1260 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:52.279 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:52.264 7f1f353b1700 1 -- 172.21.2.102:0/2212579160 >> 172.21.2.102:0/2212579160 conn(0x7f1f300fde40 msgr2=0x7f1f300ce390 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:52.279 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:52.264 7f1f353b1700 1 -- 172.21.2.102:0/2212579160 shutdown_connections 2021-02-10T11:23:52.279 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:52.264 7f1f353b1700 1 -- 172.21.2.102:0/2212579160 wait complete. 2021-02-10T11:23:52.280 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:52.279 7f1f353b1700 1 Processor -- start 2021-02-10T11:23:52.280 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:52.279 7f1f353b1700 1 -- start start 2021-02-10T11:23:52.280 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:52.279 7f1f353b1700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1f301a0510 0x7f1f300b2450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:52.280 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:52.279 7f1f353b1700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1f300ac630 0x7f1f300aca30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:52.280 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:52.279 7f1f353b1700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1f300ad050 0x7f1f300a2800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:52.281 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:52.279 7f1f353b1700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f300b2a70 con 0x7f1f301a0510 2021-02-10T11:23:52.281 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:52.279 7f1f353b1700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f300b2c90 con 0x7f1f300ad050 2021-02-10T11:23:52.281 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:52.279 7f1f353b1700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f300ad4c0 con 0x7f1f300ac630 2021-02-10T11:23:52.281 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:52.279 7f1f2f07d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1f301a0510 0x7f1f300b2450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:52.281 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:52.279 7f1f2f07d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1f301a0510 0x7f1f300b2450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:56676/0 (socket says 172.21.2.102:56676) 2021-02-10T11:23:52.281 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:52.279 7f1f2f07d700 1 -- 172.21.2.102:0/65790821 learned_addr learned my addr 172.21.2.102:0/65790821 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:23:52.281 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:52.280 7f1f2f07d700 1 -- 172.21.2.102:0/65790821 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1f300ac630 msgr2=0x7f1f300aca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:52.281 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:52.280 7f1f2f07d700 1 --2- 172.21.2.102:0/65790821 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1f300ac630 0x7f1f300aca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:52.281 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:52.280 7f1f2f07d700 1 -- 172.21.2.102:0/65790821 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1f300ad050 msgr2=0x7f1f300a2800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:52.281 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:52.280 7f1f2f07d700 1 --2- 172.21.2.102:0/65790821 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1f300ad050 0x7f1f300a2800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:52.281 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:52.280 7f1f2f07d700 1 -- 172.21.2.102:0/65790821 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f300ad6e0 con 0x7f1f301a0510 2021-02-10T11:23:52.281 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:52.280 7f1f2f07d700 1 --2- 172.21.2.102:0/65790821 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1f301a0510 0x7f1f300b2450 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 rx=0x7f1f20009630 tx=0x7f1f2000b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:52.281 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:52.280 7f1f17fff700 1 -- 172.21.2.102:0/65790821 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1f200093a0 con 0x7f1f301a0510 2021-02-10T11:23:52.282 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:52.280 7f1f17fff700 1 -- 172.21.2.102:0/65790821 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f2000bd70 con 0x7f1f301a0510 2021-02-10T11:23:52.282 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:52.280 7f1f17fff700 1 -- 172.21.2.102:0/65790821 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1f20005560 con 0x7f1f301a0510 2021-02-10T11:23:52.282 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:52.280 7f1f353b1700 1 -- 172.21.2.102:0/65790821 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f300ad6e0 con 0x7f1f301a0510 2021-02-10T11:23:52.282 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:52.280 7f1f353b1700 1 -- 172.21.2.102:0/65790821 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f300a3140 con 0x7f1f301a0510 2021-02-10T11:23:52.282 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:52.281 7f1f17fff700 1 -- 172.21.2.102:0/65790821 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1f2000a000 con 0x7f1f301a0510 2021-02-10T11:23:52.282 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:52.281 7f1f17fff700 1 --2- 172.21.2.102:0/65790821 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1f100186c0 0x7f1f1001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:52.282 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:52.281 7f1f17fff700 1 -- 172.21.2.102:0/65790821 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f1f20032570 con 0x7f1f301a0510 2021-02-10T11:23:52.282 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:52.281 7f1f2e87c700 1 --2- 172.21.2.102:0/65790821 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1f100186c0 0x7f1f1001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:52.282 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:52.281 7f1f2e87c700 1 --2- 172.21.2.102:0/65790821 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1f100186c0 0x7f1f1001abc0 crc :-1 s=READY pgs=595 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:52.310 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:52.312 7f1f353b1700 1 -- 172.21.2.102:0/65790821 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f30103670 con 0x7f1f301a0510 2021-02-10T11:23:52.312 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:52.314 7f1f17fff700 1 -- 172.21.2.102:0/65790821 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f1f20024ee0 con 0x7f1f301a0510 2021-02-10T11:23:52.463 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:52.465 7f1f353b1700 1 -- 172.21.2.102:0/65790821 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f1f300f0b20 con 0x7f1f301a0510 2021-02-10T11:23:52.466 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:52.468 7f1f17fff700 1 -- 172.21.2.102:0/65790821 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v7) v1 ==== 104+0+0 (secure 0 0 0) 0x7f1f20024ee0 con 0x7f1f301a0510 2021-02-10T11:23:52.467 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:52.469 7f1f353b1700 1 -- 172.21.2.102:0/65790821 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1f100186c0 msgr2=0x7f1f1001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:52.468 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:52.469 7f1f353b1700 1 --2- 172.21.2.102:0/65790821 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1f100186c0 0x7f1f1001abc0 crc :-1 s=READY pgs=595 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:52.468 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:52.469 7f1f353b1700 1 -- 172.21.2.102:0/65790821 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1f301a0510 msgr2=0x7f1f300b2450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:52.468 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:52.469 7f1f353b1700 1 --2- 172.21.2.102:0/65790821 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1f301a0510 0x7f1f300b2450 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 rx=0x7f1f20009630 tx=0x7f1f2000b980).stop 2021-02-10T11:23:52.468 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:52.469 7f1f353b1700 1 -- 172.21.2.102:0/65790821 shutdown_connections 2021-02-10T11:23:52.468 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:52.469 7f1f353b1700 1 --2- 172.21.2.102:0/65790821 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1f100186c0 0x7f1f1001abc0 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:52.468 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:52.469 7f1f353b1700 1 --2- 172.21.2.102:0/65790821 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1f300ad050 0x7f1f300a2800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:52.468 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:52.469 7f1f353b1700 1 --2- 172.21.2.102:0/65790821 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1f300ac630 0x7f1f300aca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:52.468 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:52.469 7f1f353b1700 1 --2- 172.21.2.102:0/65790821 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1f301a0510 0x7f1f300b2450 unknown :-1 s=CLOSED pgs=1261 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:52.468 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:52.469 7f1f353b1700 1 -- 172.21.2.102:0/65790821 >> 172.21.2.102:0/65790821 conn(0x7f1f300afaf0 msgr2=0x7f1f300c0c00 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:52.468 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:52.470 7f1f353b1700 1 -- 172.21.2.102:0/65790821 shutdown_connections 2021-02-10T11:23:52.468 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:52.470 7f1f353b1700 1 -- 172.21.2.102:0/65790821 wait complete. 2021-02-10T11:23:52.477 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestVolumes.test_authorize_auth_id_not_created_by_mgr_volumes' 2021-02-10T11:23:52.907 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:52.889 7f6d52928700 1 Processor -- start 2021-02-10T11:23:52.907 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:52.889 7f6d52928700 1 -- start start 2021-02-10T11:23:52.907 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:52.889 7f6d52928700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f6d4c0f0630 0x7f6d4c0f10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:52.907 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:52.889 7f6d52928700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d4c0f0b20 con 0x7f6d4c0cff00 2021-02-10T11:23:52.907 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:52.889 7f6d52928700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d4c0d1bf0 con 0x7f6d4c0d17f0 2021-02-10T11:23:52.907 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:52.889 7f6d52928700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d4c0e5eb0 con 0x7f6d4c0f0630 2021-02-10T11:23:52.908 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:52.890 7f6d4b7fe700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f6d4c0cff00 0x7f6d4c1b39d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:48910/0 (socket says 172.21.2.102:48910) 2021-02-10T11:23:52.908 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:52.890 7f6d4b7fe700 1 -- 172.21.2.102:0/2914519382 learned_addr learned my addr 172.21.2.102:0/2914519382 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:23:52.908 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:52.890 7f6d50f45700 1 --2- 172.21.2.102:0/2914519382 >> v2:172.21.2.113:3300/0 conn(0x7f6d4c0f0630 0x7f6d4c0f10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:52.908 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:52.890 7f6d4affd700 1 -- 172.21.2.102:0/2914519382 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 957226637 0 0) 0x7f6d400009b0 con 0x7f6d4c0cff00 2021-02-10T11:23:52.908 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:52.890 7f6d4affd700 1 -- 172.21.2.102:0/2914519382 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d34002fe0 con 0x7f6d4c0cff00 2021-02-10T11:23:52.908 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:52.890 7f6d4affd700 1 -- 172.21.2.102:0/2914519382 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1295608139 0 0) 0x7f6d40001010 con 0x7f6d4c0cff00 2021-02-10T11:23:52.908 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:52.890 7f6d4affd700 1 -- 172.21.2.102:0/2914519382 >> v2:172.21.2.113:3300/0 conn(0x7f6d4c0f0630 msgr2=0x7f6d4c0f10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:52.908 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:52.890 7f6d4affd700 1 --2- 172.21.2.102:0/2914519382 >> v2:172.21.2.113:3300/0 conn(0x7f6d4c0f0630 0x7f6d4c0f10a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:52.908 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:52.891 7f6d4affd700 1 -- 172.21.2.102:0/2914519382 >> v1:172.21.2.113:6789/0 conn(0x7f6d4c0d17f0 legacy=0x7f6d4c1b31e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:52.908 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:52.891 7f6d4affd700 1 -- 172.21.2.102:0/2914519382 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d4c0d1e10 con 0x7f6d4c0cff00 2021-02-10T11:23:52.908 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:52.891 7f6d4affd700 1 -- 172.21.2.102:0/2914519382 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7f6d40001590 con 0x7f6d4c0cff00 2021-02-10T11:23:52.908 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:52.891 7f6d4affd700 1 -- 172.21.2.102:0/2914519382 >> v1:172.21.2.102:6789/0 conn(0x7f6d4c0cff00 legacy=0x7f6d4c1b39d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:52.908 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:52.891 7f6d4affd700 1 --2- 172.21.2.102:0/2914519382 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6d34003ac0 0x7f6d34005f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:52.909 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:52.891 7f6d4affd700 1 --2- 172.21.2.102:0/2914519382 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6d34006780 0x7f6d34008c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:52.909 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:52.891 7f6d4affd700 1 --2- 172.21.2.102:0/2914519382 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6d34009580 0x7f6d3400baa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:52.909 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:52.891 7f6d4affd700 1 -- 172.21.2.102:0/2914519382 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d3400c2f0 con 0x7f6d34003ac0 2021-02-10T11:23:52.909 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:52.891 7f6d4affd700 1 -- 172.21.2.102:0/2914519382 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d3400c510 con 0x7f6d34009580 2021-02-10T11:23:52.909 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:52.891 7f6d4affd700 1 -- 172.21.2.102:0/2914519382 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d34050740 con 0x7f6d34006780 2021-02-10T11:23:52.909 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:52.891 7f6d4bfff700 1 --2- 172.21.2.102:0/2914519382 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6d34003ac0 0x7f6d34005f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:52.909 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:52.891 7f6d50f45700 1 --2- 172.21.2.102:0/2914519382 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6d34009580 0x7f6d3400baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:52.909 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:52.891 7f6d4b7fe700 1 --2- 172.21.2.102:0/2914519382 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6d34006780 0x7f6d34008c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:52.909 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:52.891 7f6d4bfff700 1 -- 172.21.2.102:0/2914519382 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6d34006780 msgr2=0x7f6d34008c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:52.909 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:52.891 7f6d4bfff700 1 --2- 172.21.2.102:0/2914519382 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6d34006780 0x7f6d34008c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:52.909 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:52.891 7f6d4bfff700 1 -- 172.21.2.102:0/2914519382 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6d34009580 msgr2=0x7f6d3400baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:52.909 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:52.891 7f6d4bfff700 1 --2- 172.21.2.102:0/2914519382 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6d34009580 0x7f6d3400baa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:52.909 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:52.891 7f6d4bfff700 1 -- 172.21.2.102:0/2914519382 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d340509a0 con 0x7f6d34003ac0 2021-02-10T11:23:52.909 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:52.891 7f6d50f45700 1 -- 172.21.2.102:0/2914519382 reap_dead start 2021-02-10T11:23:52.910 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:52.891 7f6d4b7fe700 1 --2- 172.21.2.102:0/2914519382 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6d34006780 0x7f6d34008c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:23:52.910 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:52.891 7f6d4bfff700 1 --2- 172.21.2.102:0/2914519382 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6d34003ac0 0x7f6d34005f60 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 rx=0x7f6d3c000e30 tx=0x7f6d3c009aa0).ready entity=mon.0 client_cookie=cb494cf34593f840 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:52.910 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:52.891 7f6d4affd700 1 -- 172.21.2.102:0/2914519382 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6d3c000e70 con 0x7f6d34003ac0 2021-02-10T11:23:52.910 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:52.891 7f6d4affd700 1 -- 172.21.2.102:0/2914519382 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d3c00c0c0 con 0x7f6d34003ac0 2021-02-10T11:23:52.910 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:52.891 7f6d4affd700 1 -- 172.21.2.102:0/2914519382 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6d3c00c5e0 con 0x7f6d34003ac0 2021-02-10T11:23:52.910 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:52.892 7f6d52928700 1 -- 172.21.2.102:0/2914519382 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6d34003ac0 msgr2=0x7f6d34005f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:52.910 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:52.892 7f6d52928700 1 --2- 172.21.2.102:0/2914519382 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6d34003ac0 0x7f6d34005f60 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 rx=0x7f6d3c000e30 tx=0x7f6d3c009aa0).stop 2021-02-10T11:23:52.910 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:52.892 7f6d52928700 1 -- 172.21.2.102:0/2914519382 shutdown_connections 2021-02-10T11:23:52.910 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:52.892 7f6d52928700 1 --2- 172.21.2.102:0/2914519382 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6d34003ac0 0x7f6d34005f60 unknown :-1 s=CLOSED pgs=1263 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:52.910 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:52.892 7f6d52928700 1 -- 172.21.2.102:0/2914519382 >> 172.21.2.102:0/2914519382 conn(0x7f6d4c0fde40 msgr2=0x7f6d4c0ce390 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:52.910 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:52.892 7f6d52928700 1 -- 172.21.2.102:0/2914519382 shutdown_connections 2021-02-10T11:23:52.910 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:52.892 7f6d52928700 1 -- 172.21.2.102:0/2914519382 wait complete. 2021-02-10T11:23:52.910 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:52.907 7f6d52928700 1 Processor -- start 2021-02-10T11:23:52.911 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:52.907 7f6d52928700 1 -- start start 2021-02-10T11:23:52.911 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:52.907 7f6d52928700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6d4c08ccc0 0x7f6d4c08b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:52.911 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:52.907 7f6d52928700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6d4c08bf30 0x7f6d4c1a7440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:52.911 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:52.907 7f6d52928700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6d4c1a7a60 0x7f6d4c104e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:52.911 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:52.907 7f6d52928700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d4c08c330 con 0x7f6d4c1a7a60 2021-02-10T11:23:52.911 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:52.907 7f6d52928700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d4c1a7e60 con 0x7f6d4c08ccc0 2021-02-10T11:23:52.911 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:52.907 7f6d52928700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d4c1a8080 con 0x7f6d4c08bf30 2021-02-10T11:23:52.911 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:52.908 7f6d50f45700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6d4c1a7a60 0x7f6d4c104e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:52.911 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:52.908 7f6d50f45700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6d4c1a7a60 0x7f6d4c104e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:56698/0 (socket says 172.21.2.102:56698) 2021-02-10T11:23:52.911 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:52.908 7f6d50f45700 1 -- 172.21.2.102:0/1002506016 learned_addr learned my addr 172.21.2.102:0/1002506016 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:23:52.911 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:52.908 7f6d50f45700 1 -- 172.21.2.102:0/1002506016 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6d4c08bf30 msgr2=0x7f6d4c1a7440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:52.911 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:52.908 7f6d50f45700 1 --2- 172.21.2.102:0/1002506016 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6d4c08bf30 0x7f6d4c1a7440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:52.911 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:52.908 7f6d50f45700 1 -- 172.21.2.102:0/1002506016 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6d4c08ccc0 msgr2=0x7f6d4c08b910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:52.911 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:52.908 7f6d50f45700 1 --2- 172.21.2.102:0/1002506016 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6d4c08ccc0 0x7f6d4c08b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:52.912 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:52.908 7f6d50f45700 1 -- 172.21.2.102:0/1002506016 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d4c105490 con 0x7f6d4c1a7a60 2021-02-10T11:23:52.912 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:52.908 7f6d50f45700 1 --2- 172.21.2.102:0/1002506016 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6d4c1a7a60 0x7f6d4c104e70 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 rx=0x7f6d4400b970 tx=0x7f6d4400bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:52.912 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:52.908 7f6d48ff9700 1 -- 172.21.2.102:0/1002506016 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6d44000f90 con 0x7f6d4c1a7a60 2021-02-10T11:23:52.912 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:52.908 7f6d48ff9700 1 -- 172.21.2.102:0/1002506016 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d44003030 con 0x7f6d4c1a7a60 2021-02-10T11:23:52.912 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:52.908 7f6d52928700 1 -- 172.21.2.102:0/1002506016 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d4c105490 con 0x7f6d4c1a7a60 2021-02-10T11:23:52.912 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:52.908 7f6d48ff9700 1 -- 172.21.2.102:0/1002506016 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6d440013e0 con 0x7f6d4c1a7a60 2021-02-10T11:23:52.912 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:52.908 7f6d52928700 1 -- 172.21.2.102:0/1002506016 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d4c105490 con 0x7f6d4c1a7a60 2021-02-10T11:23:52.912 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:52.909 7f6d48ff9700 1 -- 172.21.2.102:0/1002506016 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f6d44000df0 con 0x7f6d4c1a7a60 2021-02-10T11:23:52.912 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:52.909 7f6d48ff9700 1 --2- 172.21.2.102:0/1002506016 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6d2c0186d0 0x7f6d2c01abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:52.912 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:52.909 7f6d48ff9700 1 -- 172.21.2.102:0/1002506016 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f6d440329f0 con 0x7f6d4c1a7a60 2021-02-10T11:23:52.912 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:52.909 7f6d4bfff700 1 --2- 172.21.2.102:0/1002506016 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6d2c0186d0 0x7f6d2c01abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:52.913 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:52.909 7f6d4bfff700 1 --2- 172.21.2.102:0/1002506016 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6d2c0186d0 0x7f6d2c01abd0 crc :-1 s=READY pgs=596 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:52.937 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:52.939 7f6d52928700 1 -- 172.21.2.102:0/1002506016 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d4c08c330 con 0x7f6d4c1a7a60 2021-02-10T11:23:52.940 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:52.942 7f6d48ff9700 1 -- 172.21.2.102:0/1002506016 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f6d44003010 con 0x7f6d4c1a7a60 2021-02-10T11:23:53.074 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:53.076 7f6d52928700 1 -- 172.21.2.102:0/1002506016 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestVolumes.test_authorize_auth_id_not_created_by_mgr_volumes"]} v 0) v1 -- 0x7f6d4c08c330 con 0x7f6d4c1a7a60 2021-02-10T11:23:53.998 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:53.999 7f6d48ff9700 1 -- 172.21.2.102:0/1002506016 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestVolumes.test_authorize_auth_id_not_created_by_mgr_volumes"]}]=0 v289) v1 ==== 166+0+0 (secure 0 0 0) 0x7f6d44003010 con 0x7f6d4c1a7a60 2021-02-10T11:23:54.045 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:54.046 7f6d52928700 1 -- 172.21.2.102:0/1002506016 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6d2c0186d0 msgr2=0x7f6d2c01abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:54.045 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:54.046 7f6d52928700 1 --2- 172.21.2.102:0/1002506016 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6d2c0186d0 0x7f6d2c01abd0 crc :-1 s=READY pgs=596 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:54.045 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:54.046 7f6d52928700 1 -- 172.21.2.102:0/1002506016 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6d4c1a7a60 msgr2=0x7f6d4c104e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:54.045 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:54.046 7f6d52928700 1 --2- 172.21.2.102:0/1002506016 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6d4c1a7a60 0x7f6d4c104e70 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 rx=0x7f6d4400b970 tx=0x7f6d4400bd10).stop 2021-02-10T11:23:54.045 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:54.046 7f6d52928700 1 -- 172.21.2.102:0/1002506016 shutdown_connections 2021-02-10T11:23:54.045 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:54.046 7f6d52928700 1 --2- 172.21.2.102:0/1002506016 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6d4c1a7a60 0x7f6d4c104e70 unknown :-1 s=CLOSED pgs=1264 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:54.046 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:54.046 7f6d52928700 1 --2- 172.21.2.102:0/1002506016 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6d4c08bf30 0x7f6d4c1a7440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:54.046 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:54.046 7f6d52928700 1 --2- 172.21.2.102:0/1002506016 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6d2c0186d0 0x7f6d2c01abd0 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:54.046 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:54.046 7f6d52928700 1 --2- 172.21.2.102:0/1002506016 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6d4c08ccc0 0x7f6d4c08b910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:54.046 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:54.047 7f6d52928700 1 -- 172.21.2.102:0/1002506016 >> 172.21.2.102:0/1002506016 conn(0x7f6d4c0a16a0 msgr2=0x7f6d4c0a09c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:54.046 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:54.047 7f6d52928700 1 -- 172.21.2.102:0/1002506016 shutdown_connections 2021-02-10T11:23:54.046 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:54.047 7f6d52928700 1 -- 172.21.2.102:0/1002506016 wait complete. 2021-02-10T11:23:54.056 INFO:tasks.cephfs_test_runner:test_authorize_auth_id_not_created_by_mgr_volumes (tasks.cephfs.test_volumes.TestVolumes) ... ok 2021-02-10T11:23:54.056 INFO:tasks.cephfs_test_runner:Starting test: test_authorize_deauthorize_legacy_subvolume (tasks.cephfs.test_volumes.TestVolumes) 2021-02-10T11:23:54.056 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestVolumes.test_authorize_deauthorize_legacy_subvolume' 2021-02-10T11:23:54.483 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:54.468 7f09ca876700 1 Processor -- start 2021-02-10T11:23:54.566 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:54.468 7f09ca876700 1 -- start start 2021-02-10T11:23:54.566 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:54.468 7f09ca876700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f09c40e5070 0x7f09c41b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:54.566 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:54.468 7f09ca876700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f09c41b3e40 0x7f09c407d690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:54.566 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:54.468 7f09ca876700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f09c40e5470 0x7f09c407dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:54.566 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:54.468 7f09ca876700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09c407ed80 con 0x7f09c41b3e40 2021-02-10T11:23:54.566 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:54.468 7f09ca876700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09c407e2d0 con 0x7f09c40e5070 2021-02-10T11:23:54.566 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:54.468 7f09ca876700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09c407e4f0 con 0x7f09c40e5470 2021-02-10T11:23:54.566 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:54.469 7f09c37fe700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f09c41b3e40 0x7f09c407d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:54.566 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:54.469 7f09c37fe700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f09c41b3e40 0x7f09c407d690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:56702/0 (socket says 172.21.2.102:56702) 2021-02-10T11:23:54.566 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:54.469 7f09c37fe700 1 -- 172.21.2.102:0/529206221 learned_addr learned my addr 172.21.2.102:0/529206221 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:23:54.567 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:54.469 7f09c37fe700 1 -- 172.21.2.102:0/529206221 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f09c40e5470 msgr2=0x7f09c407dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:54.567 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:54.469 7f09c37fe700 1 --2- 172.21.2.102:0/529206221 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f09c40e5470 0x7f09c407dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:54.567 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:54.469 7f09c37fe700 1 -- 172.21.2.102:0/529206221 >> v2:172.21.2.113:3300/0 conn(0x7f09c40e5070 msgr2=0x7f09c41b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:54.567 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:54.469 7f09c37fe700 1 --2- 172.21.2.102:0/529206221 >> v2:172.21.2.113:3300/0 conn(0x7f09c40e5070 0x7f09c41b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:54.567 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:54.469 7f09c37fe700 1 -- 172.21.2.102:0/529206221 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09c407e710 con 0x7f09c41b3e40 2021-02-10T11:23:54.567 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:54.469 7f09c37fe700 1 --2- 172.21.2.102:0/529206221 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f09c41b3e40 0x7f09c407d690 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 rx=0x7f09b8006700 tx=0x7f09b8009b80).ready entity=mon.0 client_cookie=cb48016c1b8e0e39 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:54.567 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:54.470 7f09c2ffd700 1 -- 172.21.2.102:0/529206221 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f09b800c0c0 con 0x7f09c41b3e40 2021-02-10T11:23:54.567 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:54.470 7f09c2ffd700 1 -- 172.21.2.102:0/529206221 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09b800c400 con 0x7f09c41b3e40 2021-02-10T11:23:54.567 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:54.470 7f09c2ffd700 1 -- 172.21.2.102:0/529206221 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f09b800c920 con 0x7f09c41b3e40 2021-02-10T11:23:54.567 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:54.470 7f09ca876700 1 -- 172.21.2.102:0/529206221 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f09c41b3e40 msgr2=0x7f09c407d690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:54.567 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:54.470 7f09ca876700 1 --2- 172.21.2.102:0/529206221 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f09c41b3e40 0x7f09c407d690 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 rx=0x7f09b8006700 tx=0x7f09b8009b80).stop 2021-02-10T11:23:54.567 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:54.470 7f09ca876700 1 -- 172.21.2.102:0/529206221 shutdown_connections 2021-02-10T11:23:54.567 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:54.470 7f09ca876700 1 --2- 172.21.2.102:0/529206221 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f09c40e5470 0x7f09c407dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:54.568 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:54.470 7f09ca876700 1 --2- 172.21.2.102:0/529206221 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f09c41b3e40 0x7f09c407d690 unknown :-1 s=CLOSED pgs=1265 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:54.568 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:54.470 7f09ca876700 1 --2- 172.21.2.102:0/529206221 >> v2:172.21.2.113:3300/0 conn(0x7f09c40e5070 0x7f09c41b3820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:54.568 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:54.470 7f09ca876700 1 -- 172.21.2.102:0/529206221 >> 172.21.2.102:0/529206221 conn(0x7f09c40ce690 msgr2=0x7f09c40e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:54.568 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:54.470 7f09ca876700 1 -- 172.21.2.102:0/529206221 shutdown_connections 2021-02-10T11:23:54.568 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:54.471 7f09ca876700 1 -- 172.21.2.102:0/529206221 wait complete. 2021-02-10T11:23:54.569 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:54.485 7f09ca876700 1 Processor -- start 2021-02-10T11:23:54.570 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:54.485 7f09ca876700 1 -- start start 2021-02-10T11:23:54.570 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:54.486 7f09ca876700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f09c4192960 0x7f09c4192d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:54.570 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:54.486 7f09ca876700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f09c40ad110 0x7f09c40ad510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:54.570 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:54.486 7f09ca876700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f09c40adb30 0x7f09c408e460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:54.570 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:54.486 7f09ca876700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09c41a03e0 con 0x7f09c40ad110 2021-02-10T11:23:54.570 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:54.486 7f09ca876700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09c4193380 con 0x7f09c4192960 2021-02-10T11:23:54.570 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:54.486 7f09ca876700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09c408ea80 con 0x7f09c40adb30 2021-02-10T11:23:54.570 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:54.486 7f09c37fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f09c40ad110 0x7f09c40ad510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:54.570 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:54.486 7f09c37fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f09c40ad110 0x7f09c40ad510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:56708/0 (socket says 172.21.2.102:56708) 2021-02-10T11:23:54.570 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:54.486 7f09c37fe700 1 -- 172.21.2.102:0/3468423000 learned_addr learned my addr 172.21.2.102:0/3468423000 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:23:54.570 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:54.486 7f09c37fe700 1 -- 172.21.2.102:0/3468423000 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f09c40adb30 msgr2=0x7f09c408e460 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:23:54.570 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:54.486 7f09c37fe700 1 --2- 172.21.2.102:0/3468423000 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f09c40adb30 0x7f09c408e460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:54.571 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:54.486 7f09c37fe700 1 -- 172.21.2.102:0/3468423000 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f09c4192960 msgr2=0x7f09c4192d60 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:23:54.571 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:54.486 7f09c37fe700 1 --2- 172.21.2.102:0/3468423000 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f09c4192960 0x7f09c4192d60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:54.571 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:54.486 7f09c37fe700 1 -- 172.21.2.102:0/3468423000 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09c408eca0 con 0x7f09c40ad110 2021-02-10T11:23:54.571 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:54.486 7f09c37fe700 1 --2- 172.21.2.102:0/3468423000 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f09c40ad110 0x7f09c40ad510 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 rx=0x7f09b80066a0 tx=0x7f09b800b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:54.571 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:54.486 7f09c0ff9700 1 -- 172.21.2.102:0/3468423000 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f09b800bd80 con 0x7f09c40ad110 2021-02-10T11:23:54.571 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:54.486 7f09c0ff9700 1 -- 172.21.2.102:0/3468423000 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09b80052e0 con 0x7f09c40ad110 2021-02-10T11:23:54.571 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:54.486 7f09c0ff9700 1 -- 172.21.2.102:0/3468423000 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f09b8005740 con 0x7f09c40ad110 2021-02-10T11:23:54.571 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:54.486 7f09ca876700 1 -- 172.21.2.102:0/3468423000 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09c408eca0 con 0x7f09c40ad110 2021-02-10T11:23:54.571 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:54.486 7f09ca876700 1 -- 172.21.2.102:0/3468423000 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09c408f230 con 0x7f09c40ad110 2021-02-10T11:23:54.571 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:54.487 7f09c0ff9700 1 -- 172.21.2.102:0/3468423000 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f09b8006030 con 0x7f09c40ad110 2021-02-10T11:23:54.571 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:54.487 7f09c0ff9700 1 --2- 172.21.2.102:0/3468423000 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f09a40186c0 0x7f09a401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:54.571 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:54.487 7f09c0ff9700 1 -- 172.21.2.102:0/3468423000 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f09b80325a0 con 0x7f09c40ad110 2021-02-10T11:23:54.571 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:54.487 7f09c3fff700 1 --2- 172.21.2.102:0/3468423000 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f09a40186c0 0x7f09a401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:54.572 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:54.487 7f09c3fff700 1 --2- 172.21.2.102:0/3468423000 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f09a40186c0 0x7f09a401abc0 crc :-1 s=READY pgs=597 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:54.572 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:54.518 7f09ca876700 1 -- 172.21.2.102:0/3468423000 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09c4107120 con 0x7f09c40ad110 2021-02-10T11:23:54.572 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:54.521 7f09c0ff9700 1 -- 172.21.2.102:0/3468423000 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f09b8030d50 con 0x7f09c40ad110 2021-02-10T11:23:54.652 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:54.654 7f09ca876700 1 -- 172.21.2.102:0/3468423000 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestVolumes.test_authorize_deauthorize_legacy_subvolume"]} v 0) v1 -- 0x7f09c4107120 con 0x7f09c40ad110 2021-02-10T11:23:55.001 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:55.002 7f09c0ff9700 1 -- 172.21.2.102:0/3468423000 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestVolumes.test_authorize_deauthorize_legacy_subvolume"]}]=0 v290) v1 ==== 163+0+0 (secure 0 0 0) 0x7f09b8006030 con 0x7f09c40ad110 2021-02-10T11:23:55.019 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:55.021 7f09ca876700 1 -- 172.21.2.102:0/3468423000 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f09a40186c0 msgr2=0x7f09a401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:55.019 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:55.021 7f09ca876700 1 --2- 172.21.2.102:0/3468423000 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f09a40186c0 0x7f09a401abc0 crc :-1 s=READY pgs=597 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:55.019 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:55.021 7f09ca876700 1 -- 172.21.2.102:0/3468423000 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f09c40ad110 msgr2=0x7f09c40ad510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:55.019 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:55.021 7f09ca876700 1 --2- 172.21.2.102:0/3468423000 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f09c40ad110 0x7f09c40ad510 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 rx=0x7f09b80066a0 tx=0x7f09b800b990).stop 2021-02-10T11:23:55.020 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:55.021 7f09ca876700 1 -- 172.21.2.102:0/3468423000 shutdown_connections 2021-02-10T11:23:55.020 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:55.021 7f09ca876700 1 --2- 172.21.2.102:0/3468423000 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f09c40adb30 0x7f09c408e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:55.020 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:55.021 7f09ca876700 1 --2- 172.21.2.102:0/3468423000 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f09c40ad110 0x7f09c40ad510 unknown :-1 s=CLOSED pgs=1266 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:55.020 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:55.021 7f09ca876700 1 --2- 172.21.2.102:0/3468423000 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f09a40186c0 0x7f09a401abc0 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:55.020 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:55.021 7f09ca876700 1 --2- 172.21.2.102:0/3468423000 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f09c4192960 0x7f09c4192d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:55.020 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:55.021 7f09ca876700 1 -- 172.21.2.102:0/3468423000 >> 172.21.2.102:0/3468423000 conn(0x7f09c40891f0 msgr2=0x7f09c41a0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:55.020 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:55.021 7f09ca876700 1 -- 172.21.2.102:0/3468423000 shutdown_connections 2021-02-10T11:23:55.020 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:55.021 7f09ca876700 1 -- 172.21.2.102:0/3468423000 wait complete. 2021-02-10T11:23:55.028 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config set mon mon_allow_pool_delete True 2021-02-10T11:23:55.459 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:55.443 7fd23fad9700 1 Processor -- start 2021-02-10T11:23:55.459 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:55.443 7fd23fad9700 1 -- start start 2021-02-10T11:23:55.459 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:55.443 7fd23fad9700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd2381b0870 0x7fd2380f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:55.459 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:55.443 7fd23fad9700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2381b0c70 con 0x7fd2380efe00 2021-02-10T11:23:55.459 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:55.443 7fd23fad9700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2380f0b50 con 0x7fd2381b0470 2021-02-10T11:23:55.459 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:55.443 7fd23fad9700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2380e5ea0 con 0x7fd2381b0870 2021-02-10T11:23:55.459 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:55.443 7fd23e0f6700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fd2380efe00 0x7fd2380f0200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:48938/0 (socket says 172.21.2.102:48938) 2021-02-10T11:23:55.459 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:55.443 7fd23e0f6700 1 -- 172.21.2.102:0/3321935592 learned_addr learned my addr 172.21.2.102:0/3321935592 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:23:55.459 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:55.444 7fd23c8f3700 1 -- 172.21.2.102:0/3321935592 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2079664241 0 0) 0x7fd230000f90 con 0x7fd2380efe00 2021-02-10T11:23:55.459 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:55.444 7fd23c8f3700 1 -- 172.21.2.102:0/3321935592 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd224002fe0 con 0x7fd2380efe00 2021-02-10T11:23:55.460 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:55.444 7fd23d0f4700 1 --2- 172.21.2.102:0/3321935592 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd2381b0870 0x7fd2380f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:55.460 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:55.444 7fd23c8f3700 1 -- 172.21.2.102:0/3321935592 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3103170605 0 0) 0x7fd230001460 con 0x7fd2380efe00 2021-02-10T11:23:55.460 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:55.444 7fd23c8f3700 1 -- 172.21.2.102:0/3321935592 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd2381b0870 msgr2=0x7fd2380f20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:55.460 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:55.444 7fd23c8f3700 1 --2- 172.21.2.102:0/3321935592 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd2381b0870 0x7fd2380f20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:55.460 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:55.444 7fd23c8f3700 1 -- 172.21.2.102:0/3321935592 >> v1:172.21.2.113:6789/0 conn(0x7fd2381b0470 legacy=0x7fd2380ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:55.460 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:55.444 7fd23c8f3700 1 -- 172.21.2.102:0/3321935592 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2380e52e0 con 0x7fd2380efe00 2021-02-10T11:23:55.460 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:55.444 7fd23c8f3700 1 -- 172.21.2.102:0/3321935592 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7fd230001980 con 0x7fd2380efe00 2021-02-10T11:23:55.460 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:55.444 7fd23c8f3700 1 -- 172.21.2.102:0/3321935592 >> v1:172.21.2.102:6789/0 conn(0x7fd2380efe00 legacy=0x7fd2380f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:55.460 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:55.444 7fd23c8f3700 1 --2- 172.21.2.102:0/3321935592 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd224003ae0 0x7fd224005f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:55.460 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:55.444 7fd23c8f3700 1 --2- 172.21.2.102:0/3321935592 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd224006830 0x7fd224008d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:55.460 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:55.444 7fd23c8f3700 1 --2- 172.21.2.102:0/3321935592 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd224009630 0x7fd22400bb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:55.460 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:55.444 7fd23c8f3700 1 -- 172.21.2.102:0/3321935592 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd22400c370 con 0x7fd224003ae0 2021-02-10T11:23:55.460 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:55.444 7fd23c8f3700 1 -- 172.21.2.102:0/3321935592 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd22400c590 con 0x7fd224006830 2021-02-10T11:23:55.460 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:55.444 7fd23c8f3700 1 -- 172.21.2.102:0/3321935592 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd22400c7b0 con 0x7fd224009630 2021-02-10T11:23:55.461 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:55.444 7fd23d0f4700 1 --2- 172.21.2.102:0/3321935592 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd2381b0870 0x7fd2380f20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:23:55.461 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:55.445 7fd23d8f5700 1 --2- 172.21.2.102:0/3321935592 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd224003ae0 0x7fd224005f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:55.461 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:55.445 7fd23d8f5700 1 -- 172.21.2.102:0/3321935592 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd224009630 msgr2=0x7fd22400bb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:55.461 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:55.445 7fd23d8f5700 1 --2- 172.21.2.102:0/3321935592 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd224009630 0x7fd22400bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:55.461 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:55.445 7fd23d8f5700 1 -- 172.21.2.102:0/3321935592 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd224006830 msgr2=0x7fd224008d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:55.461 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:55.445 7fd23d8f5700 1 --2- 172.21.2.102:0/3321935592 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd224006830 0x7fd224008d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:55.461 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:55.445 7fd23d8f5700 1 -- 172.21.2.102:0/3321935592 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd22400ca50 con 0x7fd224003ae0 2021-02-10T11:23:55.461 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:55.445 7fd23d8f5700 1 --2- 172.21.2.102:0/3321935592 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd224003ae0 0x7fd224005f90 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 rx=0x7fd22c006ab0 tx=0x7fd22c009b40).ready entity=mon.0 client_cookie=de92b9936b6393ee server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:55.461 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:55.445 7fd23c8f3700 1 -- 172.21.2.102:0/3321935592 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd22c00b2e0 con 0x7fd224003ae0 2021-02-10T11:23:55.461 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:55.445 7fd23c8f3700 1 -- 172.21.2.102:0/3321935592 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd22c00b560 con 0x7fd224003ae0 2021-02-10T11:23:55.461 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:55.445 7fd23c8f3700 1 -- 172.21.2.102:0/3321935592 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd22c00ba40 con 0x7fd224003ae0 2021-02-10T11:23:55.461 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:55.445 7fd23fad9700 1 -- 172.21.2.102:0/3321935592 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd224003ae0 msgr2=0x7fd224005f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:55.461 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:55.445 7fd23fad9700 1 --2- 172.21.2.102:0/3321935592 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd224003ae0 0x7fd224005f90 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 rx=0x7fd22c006ab0 tx=0x7fd22c009b40).stop 2021-02-10T11:23:55.461 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:55.445 7fd23e0f6700 1 -- 172.21.2.102:0/3321935592 reap_dead start 2021-02-10T11:23:55.462 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:55.446 7fd23fad9700 1 -- 172.21.2.102:0/3321935592 shutdown_connections 2021-02-10T11:23:55.462 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:55.446 7fd23fad9700 1 -- 172.21.2.102:0/3321935592 >> 172.21.2.102:0/3321935592 conn(0x7fd2380ce3e0 msgr2=0x7fd2380e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:55.462 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:55.446 7fd23fad9700 1 -- 172.21.2.102:0/3321935592 shutdown_connections 2021-02-10T11:23:55.462 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:55.446 7fd23fad9700 1 -- 172.21.2.102:0/3321935592 wait complete. 2021-02-10T11:23:55.462 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:55.460 7fd23fad9700 1 Processor -- start 2021-02-10T11:23:55.462 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:55.460 7fd23fad9700 1 -- start start 2021-02-10T11:23:55.463 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:55.461 7fd23fad9700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd2380b7db0 0x7fd2380b8260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:55.463 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:55.461 7fd23fad9700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd2380b8880 0x7fd2381a7450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:55.463 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:55.461 7fd23fad9700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd2380b8c80 0x7fd2381a7a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:55.464 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:55.461 7fd23fad9700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2380b79b0 con 0x7fd2380b8c80 2021-02-10T11:23:55.464 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:55.461 7fd23fad9700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2381a8090 con 0x7fd2380b7db0 2021-02-10T11:23:55.464 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:55.461 7fd23fad9700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2381a82b0 con 0x7fd2380b8880 2021-02-10T11:23:55.464 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:55.461 7fd23e0f6700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd2380b8c80 0x7fd2381a7a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:55.464 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:55.461 7fd23e0f6700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd2380b8c80 0x7fd2381a7a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:56726/0 (socket says 172.21.2.102:56726) 2021-02-10T11:23:55.464 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:55.461 7fd23e0f6700 1 -- 172.21.2.102:0/2656533013 learned_addr learned my addr 172.21.2.102:0/2656533013 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:23:55.464 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:55.461 7fd23e0f6700 1 -- 172.21.2.102:0/2656533013 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd2380b8880 msgr2=0x7fd2381a7450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:55.464 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:55.461 7fd23e0f6700 1 --2- 172.21.2.102:0/2656533013 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd2380b8880 0x7fd2381a7450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:55.464 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:55.461 7fd23e0f6700 1 -- 172.21.2.102:0/2656533013 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd2380b7db0 msgr2=0x7fd2380b8260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:55.464 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:55.461 7fd23e0f6700 1 --2- 172.21.2.102:0/2656533013 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd2380b7db0 0x7fd2380b8260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:55.464 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:55.461 7fd23e0f6700 1 -- 172.21.2.102:0/2656533013 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd238190c10 con 0x7fd2380b8c80 2021-02-10T11:23:55.464 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:55.462 7fd23e0f6700 1 --2- 172.21.2.102:0/2656533013 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd2380b8c80 0x7fd2381a7a70 secure :-1 s=READY pgs=1269 cs=0 l=1 rev1=1 rx=0x7fd2300099a0 tx=0x7fd23000bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:55.464 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:55.462 7fd22a7fc700 1 -- 172.21.2.102:0/2656533013 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd2300060c0 con 0x7fd2380b8c80 2021-02-10T11:23:55.465 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:55.462 7fd22a7fc700 1 -- 172.21.2.102:0/2656533013 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd230006440 con 0x7fd2380b8c80 2021-02-10T11:23:55.465 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:55.462 7fd23fad9700 1 -- 172.21.2.102:0/2656533013 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd238190c10 con 0x7fd2380b8c80 2021-02-10T11:23:55.465 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:55.462 7fd22a7fc700 1 -- 172.21.2.102:0/2656533013 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd230004720 con 0x7fd2380b8c80 2021-02-10T11:23:55.465 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:55.462 7fd23fad9700 1 -- 172.21.2.102:0/2656533013 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd238190c10 con 0x7fd2380b8c80 2021-02-10T11:23:55.465 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:55.463 7fd22a7fc700 1 -- 172.21.2.102:0/2656533013 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd230006620 con 0x7fd2380b8c80 2021-02-10T11:23:55.465 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:55.463 7fd22a7fc700 1 --2- 172.21.2.102:0/2656533013 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd220018730 0x7fd22001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:55.465 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:55.463 7fd22a7fc700 1 -- 172.21.2.102:0/2656533013 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fd2300329b0 con 0x7fd2380b8c80 2021-02-10T11:23:55.465 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:55.463 7fd23d8f5700 1 --2- 172.21.2.102:0/2656533013 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd220018730 0x7fd22001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:55.465 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:55.463 7fd23d8f5700 1 --2- 172.21.2.102:0/2656533013 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd220018730 0x7fd22001ac10 crc :-1 s=READY pgs=598 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:55.491 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:55.493 7fd23fad9700 1 -- 172.21.2.102:0/2656533013 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd238105b30 con 0x7fd2380b8c80 2021-02-10T11:23:55.494 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:55.496 7fd22a7fc700 1 -- 172.21.2.102:0/2656533013 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd23000be30 con 0x7fd2380b8c80 2021-02-10T11:23:55.643 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:55.645 7fd23fad9700 1 -- 172.21.2.102:0/2656533013 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fd2380f0b50 con 0x7fd2380b8c80 2021-02-10T11:23:55.647 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:55.649 7fd22a7fc700 1 -- 172.21.2.102:0/2656533013 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v8)=0 v8) v1 ==== 122+0+0 (secure 0 0 0) 0x7fd230006440 con 0x7fd2380b8c80 2021-02-10T11:23:55.652 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:55.653 7fd23fad9700 1 -- 172.21.2.102:0/2656533013 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd220018730 msgr2=0x7fd22001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:55.652 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:55.653 7fd23fad9700 1 --2- 172.21.2.102:0/2656533013 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd220018730 0x7fd22001ac10 crc :-1 s=READY pgs=598 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:55.652 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:55.653 7fd23fad9700 1 -- 172.21.2.102:0/2656533013 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd2380b8c80 msgr2=0x7fd2381a7a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:55.652 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:55.653 7fd23fad9700 1 --2- 172.21.2.102:0/2656533013 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd2380b8c80 0x7fd2381a7a70 secure :-1 s=READY pgs=1269 cs=0 l=1 rev1=1 rx=0x7fd2300099a0 tx=0x7fd23000bcf0).stop 2021-02-10T11:23:55.652 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:55.653 7fd23fad9700 1 -- 172.21.2.102:0/2656533013 shutdown_connections 2021-02-10T11:23:55.652 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:55.653 7fd23fad9700 1 --2- 172.21.2.102:0/2656533013 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd2380b8c80 0x7fd2381a7a70 unknown :-1 s=CLOSED pgs=1269 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:55.653 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:55.653 7fd23fad9700 1 --2- 172.21.2.102:0/2656533013 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd2380b8880 0x7fd2381a7450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:55.653 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:55.653 7fd23fad9700 1 --2- 172.21.2.102:0/2656533013 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd220018730 0x7fd22001ac10 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:55.653 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:55.653 7fd23fad9700 1 --2- 172.21.2.102:0/2656533013 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd2380b7db0 0x7fd2380b8260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:55.653 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:55.653 7fd23fad9700 1 -- 172.21.2.102:0/2656533013 >> 172.21.2.102:0/2656533013 conn(0x7fd238094e10 msgr2=0x7fd2380a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:55.653 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:55.654 7fd23fad9700 1 -- 172.21.2.102:0/2656533013 shutdown_connections 2021-02-10T11:23:55.653 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:55.654 7fd23fad9700 1 -- 172.21.2.102:0/2656533013 wait complete. 2021-02-10T11:23:55.660 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2021-02-10T11:23:55.660 DEBUG:teuthology.orchestra.run.gibba002:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:23:56.008 DEBUG:teuthology.orchestra.run.gibba013:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:23:56.352 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2021-02-10T11:23:56.352 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:23:56.352 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:23:56.352 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:23:56.687 INFO:teuthology.orchestra.run.gibba002.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.0’: No such file or directory 2021-02-10T11:23:56.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:23:56.688 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.0 2021-02-10T11:23:56.688 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:23:56.688 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:23:56.688 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:23:57.023 INFO:teuthology.orchestra.run.gibba002.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.1’: No such file or directory 2021-02-10T11:23:57.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:23:57.023 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2021-02-10T11:23:57.023 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:23:57.024 DEBUG:teuthology.orchestra.run.gibba013:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:23:57.024 DEBUG:teuthology.orchestra.run.gibba013:> ' -- /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:23:57.358 INFO:teuthology.orchestra.run.gibba013.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.2’: No such file or directory 2021-02-10T11:23:57.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:23:57.359 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2021-02-10T11:23:57.359 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:23:57.359 DEBUG:teuthology.orchestra.run.gibba013:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:23:57.359 DEBUG:teuthology.orchestra.run.gibba013:> ' -- /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:23:57.694 INFO:teuthology.orchestra.run.gibba013.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.3’: No such file or directory 2021-02-10T11:23:57.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:23:57.694 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.3 2021-02-10T11:23:57.695 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json-pretty 2021-02-10T11:23:58.121 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:58.107 7f6cd3d6b700 1 Processor -- start 2021-02-10T11:23:58.122 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:58.107 7f6cd3d6b700 1 -- start start 2021-02-10T11:23:58.122 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:58.107 7f6cd3d6b700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f6ccc0e5070 0x7f6ccc1b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:58.122 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:58.107 7f6cd3d6b700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6ccc0e5470 0x7f6ccc080310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:58.122 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:58.107 7f6cd3d6b700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ccc07ed80 con 0x7f6ccc1b3e40 2021-02-10T11:23:58.122 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:58.107 7f6cd3d6b700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ccc0d1790 con 0x7f6ccc0e5070 2021-02-10T11:23:58.122 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:58.107 7f6cd3d6b700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ccc0d19b0 con 0x7f6ccc0e5470 2021-02-10T11:23:58.122 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:58.107 7f6cd1b87700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f6ccc0e5070 0x7f6ccc1b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:58.122 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:58.107 7f6cd1386700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f6ccc1b3e40 0x7f6ccc07d690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:48954/0 (socket says 172.21.2.102:48954) 2021-02-10T11:23:58.122 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:58.107 7f6cd1386700 1 -- 172.21.2.102:0/180875897 learned_addr learned my addr 172.21.2.102:0/180875897 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:23:58.122 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:58.108 7f6cd0b85700 1 -- 172.21.2.102:0/180875897 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3974927573 0 0) 0x7f6cbc0010b0 con 0x7f6ccc1b3e40 2021-02-10T11:23:58.122 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:58.108 7f6cd0b85700 1 -- 172.21.2.102:0/180875897 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6cb8002fe0 con 0x7f6ccc1b3e40 2021-02-10T11:23:58.122 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:58.108 7f6cd1b87700 1 -- 172.21.2.102:0/180875897 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6ccc0e5470 msgr2=0x7f6ccc080310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:58.122 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:58.108 7f6cd1b87700 1 --2- 172.21.2.102:0/180875897 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6ccc0e5470 0x7f6ccc080310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:58.123 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:58.108 7f6cd1b87700 1 -- 172.21.2.102:0/180875897 >> v1:172.21.2.102:6789/0 conn(0x7f6ccc1b3e40 legacy=0x7f6ccc07d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:58.123 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:58.108 7f6cd1b87700 1 -- 172.21.2.102:0/180875897 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ccc0d1bd0 con 0x7f6ccc0e5070 2021-02-10T11:23:58.123 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:58.108 7f6cd1b87700 1 --2- 172.21.2.102:0/180875897 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6ccc0e5070 0x7f6ccc1b3820 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 rx=0x7f6cc8006650 tx=0x7f6cc8009a80).ready entity=mon.0 client_cookie=dcf2836377595863 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:58.123 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:58.108 7f6cd0b85700 1 -- 172.21.2.102:0/180875897 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6cc800c0c0 con 0x7f6ccc0e5070 2021-02-10T11:23:58.123 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:58.108 7f6cd0b85700 1 -- 172.21.2.102:0/180875897 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cc800c340 con 0x7f6ccc0e5070 2021-02-10T11:23:58.123 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:58.108 7f6cd0b85700 1 -- 172.21.2.102:0/180875897 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6cc800c860 con 0x7f6ccc0e5070 2021-02-10T11:23:58.123 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:58.108 7f6cd3d6b700 1 -- 172.21.2.102:0/180875897 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6ccc0e5070 msgr2=0x7f6ccc1b3820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:58.123 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:58.108 7f6cd3d6b700 1 --2- 172.21.2.102:0/180875897 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6ccc0e5070 0x7f6ccc1b3820 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 rx=0x7f6cc8006650 tx=0x7f6cc8009a80).stop 2021-02-10T11:23:58.123 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:58.109 7f6cd3d6b700 1 -- 172.21.2.102:0/180875897 shutdown_connections 2021-02-10T11:23:58.123 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:58.109 7f6cd3d6b700 1 --2- 172.21.2.102:0/180875897 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6ccc0e5470 0x7f6ccc080310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:58.123 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:58.109 7f6cd3d6b700 1 --2- 172.21.2.102:0/180875897 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6ccc0e5070 0x7f6ccc1b3820 unknown :-1 s=CLOSED pgs=1271 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:58.123 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:58.109 7f6cd3d6b700 1 -- 172.21.2.102:0/180875897 >> 172.21.2.102:0/180875897 conn(0x7f6ccc0ce690 msgr2=0x7f6ccc0e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:58.123 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:58.109 7f6cd3d6b700 1 -- 172.21.2.102:0/180875897 shutdown_connections 2021-02-10T11:23:58.124 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:58.109 7f6cd3d6b700 1 -- 172.21.2.102:0/180875897 wait complete. 2021-02-10T11:23:58.125 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:58.124 7f6cd3d6b700 1 Processor -- start 2021-02-10T11:23:58.125 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:58.124 7f6cd3d6b700 1 -- start start 2021-02-10T11:23:58.125 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:58.124 7f6cd3d6b700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6ccc0969a0 0x7f6ccc096110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:58.125 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:58.124 7f6cd3d6b700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6ccc08a460 0x7f6ccc1a1440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:58.125 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:58.124 7f6cd3d6b700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6ccc0b1440 0x7f6ccc0b1840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:58.125 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:58.124 7f6cd3d6b700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ccc0b1e60 con 0x7f6ccc0b1440 2021-02-10T11:23:58.125 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:58.124 7f6cd3d6b700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ccc0b2080 con 0x7f6ccc0969a0 2021-02-10T11:23:58.126 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:58.124 7f6cd3d6b700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ccc0b22a0 con 0x7f6ccc08a460 2021-02-10T11:23:58.126 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:58.124 7f6cd2388700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6ccc0b1440 0x7f6ccc0b1840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:58.126 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:58.124 7f6cd2388700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6ccc0b1440 0x7f6ccc0b1840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:56740/0 (socket says 172.21.2.102:56740) 2021-02-10T11:23:58.126 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:58.124 7f6cd2388700 1 -- 172.21.2.102:0/1303804882 learned_addr learned my addr 172.21.2.102:0/1303804882 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:23:58.126 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:58.125 7f6cd2388700 1 -- 172.21.2.102:0/1303804882 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6ccc08a460 msgr2=0x7f6ccc1a1440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:58.126 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:58.125 7f6cd2388700 1 --2- 172.21.2.102:0/1303804882 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6ccc08a460 0x7f6ccc1a1440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:58.126 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:58.125 7f6cd2388700 1 -- 172.21.2.102:0/1303804882 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6ccc0969a0 msgr2=0x7f6ccc096110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:58.126 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:58.125 7f6cd2388700 1 --2- 172.21.2.102:0/1303804882 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6ccc0969a0 0x7f6ccc096110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:58.126 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:58.125 7f6cd2388700 1 -- 172.21.2.102:0/1303804882 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ccc08ae90 con 0x7f6ccc0b1440 2021-02-10T11:23:58.126 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:58.125 7f6cd2388700 1 --2- 172.21.2.102:0/1303804882 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6ccc0b1440 0x7f6ccc0b1840 secure :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 rx=0x7f6cc4009980 tx=0x7f6cc400bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:58.126 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:58.125 7f6cc27fc700 1 -- 172.21.2.102:0/1303804882 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6cc4003020 con 0x7f6ccc0b1440 2021-02-10T11:23:58.126 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:58.125 7f6cc27fc700 1 -- 172.21.2.102:0/1303804882 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cc4005a80 con 0x7f6ccc0b1440 2021-02-10T11:23:58.126 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:58.125 7f6cc27fc700 1 -- 172.21.2.102:0/1303804882 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6cc4006030 con 0x7f6ccc0b1440 2021-02-10T11:23:58.127 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:58.125 7f6cd3d6b700 1 -- 172.21.2.102:0/1303804882 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ccc08ae90 con 0x7f6ccc0b1440 2021-02-10T11:23:58.127 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:58.125 7f6cd3d6b700 1 -- 172.21.2.102:0/1303804882 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ccc08ae90 con 0x7f6ccc0b1440 2021-02-10T11:23:58.127 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:58.126 7f6cc27fc700 1 -- 172.21.2.102:0/1303804882 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f6cc4005c60 con 0x7f6ccc0b1440 2021-02-10T11:23:58.127 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:58.126 7f6cc27fc700 1 --2- 172.21.2.102:0/1303804882 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6cb4018730 0x7f6cb401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:58.127 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:58.126 7f6cc27fc700 1 -- 172.21.2.102:0/1303804882 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f6cc4032b30 con 0x7f6ccc0b1440 2021-02-10T11:23:58.127 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:58.126 7f6cd1b87700 1 --2- 172.21.2.102:0/1303804882 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6cb4018730 0x7f6cb401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:58.127 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:58.126 7f6cd1b87700 1 --2- 172.21.2.102:0/1303804882 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6cb4018730 0x7f6cb401ac10 crc :-1 s=READY pgs=599 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:58.155 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:58.157 7f6cd3d6b700 1 -- 172.21.2.102:0/1303804882 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ccc08a960 con 0x7f6ccc0b1440 2021-02-10T11:23:58.157 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:58.159 7f6cc27fc700 1 -- 172.21.2.102:0/1303804882 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f6cc4005870 con 0x7f6ccc0b1440 2021-02-10T11:23:58.283 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:58.285 7f6cd3d6b700 1 -- 172.21.2.102:0/1303804882 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7f6ccc08a960 con 0x7f6ccc0b1440 2021-02-10T11:23:58.283 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:58.285 7f6cc27fc700 1 -- 172.21.2.102:0/1303804882 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v48) v1 ==== 81+0+30360 (secure 0 0 0) 0x7f6cc4005a80 con 0x7f6ccc0b1440 2021-02-10T11:23:58.284 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:23:58.284 INFO:teuthology.orchestra.run.gibba002.stdout:{ 2021-02-10T11:23:58.285 INFO:teuthology.orchestra.run.gibba002.stdout: "epoch": 48, 2021-02-10T11:23:58.285 INFO:teuthology.orchestra.run.gibba002.stdout: "fsid": "68db30ea-dbce-4aef-a6df-f01c6fea0513", 2021-02-10T11:23:58.285 INFO:teuthology.orchestra.run.gibba002.stdout: "created": "2021-02-10 11:03:06.331557", 2021-02-10T11:23:58.285 INFO:teuthology.orchestra.run.gibba002.stdout: "modified": "2021-02-10 11:23:19.005019", 2021-02-10T11:23:58.285 INFO:teuthology.orchestra.run.gibba002.stdout: "last_up_change": "2021-02-10 11:03:32.690562", 2021-02-10T11:23:58.285 INFO:teuthology.orchestra.run.gibba002.stdout: "last_in_change": "2021-02-10 11:03:32.690562", 2021-02-10T11:23:58.285 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2021-02-10T11:23:58.286 INFO:teuthology.orchestra.run.gibba002.stdout: "flags_num": 5799936, 2021-02-10T11:23:58.286 INFO:teuthology.orchestra.run.gibba002.stdout: "flags_set": [ 2021-02-10T11:23:58.286 INFO:teuthology.orchestra.run.gibba002.stdout: "pglog_hardlimit", 2021-02-10T11:23:58.286 INFO:teuthology.orchestra.run.gibba002.stdout: "purged_snapdirs", 2021-02-10T11:23:58.286 INFO:teuthology.orchestra.run.gibba002.stdout: "recovery_deletes", 2021-02-10T11:23:58.286 INFO:teuthology.orchestra.run.gibba002.stdout: "sortbitwise" 2021-02-10T11:23:58.287 INFO:teuthology.orchestra.run.gibba002.stdout: ], 2021-02-10T11:23:58.287 INFO:teuthology.orchestra.run.gibba002.stdout: "crush_version": 4, 2021-02-10T11:23:58.287 INFO:teuthology.orchestra.run.gibba002.stdout: "full_ratio": 0.94999998807907104, 2021-02-10T11:23:58.287 INFO:teuthology.orchestra.run.gibba002.stdout: "backfillfull_ratio": 0.89999997615814209, 2021-02-10T11:23:58.287 INFO:teuthology.orchestra.run.gibba002.stdout: "nearfull_ratio": 0.85000002384185791, 2021-02-10T11:23:58.287 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_snapshot": "", 2021-02-10T11:23:58.287 INFO:teuthology.orchestra.run.gibba002.stdout: "pool_max": 9, 2021-02-10T11:23:58.287 INFO:teuthology.orchestra.run.gibba002.stdout: "max_osd": 8, 2021-02-10T11:23:58.287 INFO:teuthology.orchestra.run.gibba002.stdout: "require_min_compat_client": "jewel", 2021-02-10T11:23:58.287 INFO:teuthology.orchestra.run.gibba002.stdout: "min_compat_client": "jewel", 2021-02-10T11:23:58.287 INFO:teuthology.orchestra.run.gibba002.stdout: "require_osd_release": "nautilus", 2021-02-10T11:23:58.287 INFO:teuthology.orchestra.run.gibba002.stdout: "pools": [ 2021-02-10T11:23:58.288 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:58.288 INFO:teuthology.orchestra.run.gibba002.stdout: "pool": 1, 2021-02-10T11:23:58.288 INFO:teuthology.orchestra.run.gibba002.stdout: "pool_name": "rbd", 2021-02-10T11:23:58.288 INFO:teuthology.orchestra.run.gibba002.stdout: "create_time": "2021-02-10 11:03:37.322519", 2021-02-10T11:23:58.288 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 1, 2021-02-10T11:23:58.288 INFO:teuthology.orchestra.run.gibba002.stdout: "flags_names": "hashpspool", 2021-02-10T11:23:58.289 INFO:teuthology.orchestra.run.gibba002.stdout: "type": 1, 2021-02-10T11:23:58.289 INFO:teuthology.orchestra.run.gibba002.stdout: "size": 2, 2021-02-10T11:23:58.289 INFO:teuthology.orchestra.run.gibba002.stdout: "min_size": 1, 2021-02-10T11:23:58.289 INFO:teuthology.orchestra.run.gibba002.stdout: "crush_rule": 0, 2021-02-10T11:23:58.289 INFO:teuthology.orchestra.run.gibba002.stdout: "object_hash": 2, 2021-02-10T11:23:58.289 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:23:58.289 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num": 8, 2021-02-10T11:23:58.289 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_placement_num": 8, 2021-02-10T11:23:58.289 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_placement_num_target": 8, 2021-02-10T11:23:58.289 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num_target": 8, 2021-02-10T11:23:58.289 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num_pending": 8, 2021-02-10T11:23:58.289 INFO:teuthology.orchestra.run.gibba002.stdout: "last_pg_merge_meta": { 2021-02-10T11:23:58.290 INFO:teuthology.orchestra.run.gibba002.stdout: "source_pgid": "0.0", 2021-02-10T11:23:58.290 INFO:teuthology.orchestra.run.gibba002.stdout: "ready_epoch": 0, 2021-02-10T11:23:58.290 INFO:teuthology.orchestra.run.gibba002.stdout: "last_epoch_started": 0, 2021-02-10T11:23:58.290 INFO:teuthology.orchestra.run.gibba002.stdout: "last_epoch_clean": 0, 2021-02-10T11:23:58.290 INFO:teuthology.orchestra.run.gibba002.stdout: "source_version": "0'0", 2021-02-10T11:23:58.290 INFO:teuthology.orchestra.run.gibba002.stdout: "target_version": "0'0" 2021-02-10T11:23:58.290 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:58.290 INFO:teuthology.orchestra.run.gibba002.stdout: "last_change": "16", 2021-02-10T11:23:58.290 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend": "0", 2021-02-10T11:23:58.290 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:23:58.290 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:23:58.291 INFO:teuthology.orchestra.run.gibba002.stdout: "auid": 0, 2021-02-10T11:23:58.291 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:23:58.291 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_seq": 0, 2021-02-10T11:23:58.291 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_epoch": 0, 2021-02-10T11:23:58.291 INFO:teuthology.orchestra.run.gibba002.stdout: "pool_snaps": [], 2021-02-10T11:23:58.291 INFO:teuthology.orchestra.run.gibba002.stdout: "removed_snaps": "[]", 2021-02-10T11:23:58.291 INFO:teuthology.orchestra.run.gibba002.stdout: "quota_max_bytes": 0, 2021-02-10T11:23:58.291 INFO:teuthology.orchestra.run.gibba002.stdout: "quota_max_objects": 0, 2021-02-10T11:23:58.291 INFO:teuthology.orchestra.run.gibba002.stdout: "tiers": [], 2021-02-10T11:23:58.291 INFO:teuthology.orchestra.run.gibba002.stdout: "tier_of": -1, 2021-02-10T11:23:58.291 INFO:teuthology.orchestra.run.gibba002.stdout: "read_tier": -1, 2021-02-10T11:23:58.291 INFO:teuthology.orchestra.run.gibba002.stdout: "write_tier": -1, 2021-02-10T11:23:58.291 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_mode": "none", 2021-02-10T11:23:58.292 INFO:teuthology.orchestra.run.gibba002.stdout: "target_max_bytes": 0, 2021-02-10T11:23:58.292 INFO:teuthology.orchestra.run.gibba002.stdout: "target_max_objects": 0, 2021-02-10T11:23:58.292 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:23:58.292 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:23:58.292 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:23:58.292 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_min_flush_age": 0, 2021-02-10T11:23:58.292 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_min_evict_age": 0, 2021-02-10T11:23:58.292 INFO:teuthology.orchestra.run.gibba002.stdout: "erasure_code_profile": "", 2021-02-10T11:23:58.292 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_params": { 2021-02-10T11:23:58.292 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "none" 2021-02-10T11:23:58.292 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:58.292 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_period": 0, 2021-02-10T11:23:58.292 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_count": 0, 2021-02-10T11:23:58.292 INFO:teuthology.orchestra.run.gibba002.stdout: "use_gmt_hitset": true, 2021-02-10T11:23:58.293 INFO:teuthology.orchestra.run.gibba002.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:23:58.293 INFO:teuthology.orchestra.run.gibba002.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:23:58.293 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:23:58.293 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:23:58.293 INFO:teuthology.orchestra.run.gibba002.stdout: "grade_table": [], 2021-02-10T11:23:58.293 INFO:teuthology.orchestra.run.gibba002.stdout: "stripe_width": 0, 2021-02-10T11:23:58.293 INFO:teuthology.orchestra.run.gibba002.stdout: "expected_num_objects": 0, 2021-02-10T11:23:58.293 INFO:teuthology.orchestra.run.gibba002.stdout: "fast_read": false, 2021-02-10T11:23:58.293 INFO:teuthology.orchestra.run.gibba002.stdout: "options": {}, 2021-02-10T11:23:58.293 INFO:teuthology.orchestra.run.gibba002.stdout: "application_metadata": { 2021-02-10T11:23:58.293 INFO:teuthology.orchestra.run.gibba002.stdout: "rbd": {} 2021-02-10T11:23:58.293 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:23:58.293 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:58.293 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:58.294 INFO:teuthology.orchestra.run.gibba002.stdout: "pool": 8, 2021-02-10T11:23:58.294 INFO:teuthology.orchestra.run.gibba002.stdout: "pool_name": "cephfs_metadata", 2021-02-10T11:23:58.294 INFO:teuthology.orchestra.run.gibba002.stdout: "create_time": "2021-02-10 11:23:15.973296", 2021-02-10T11:23:58.294 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 1, 2021-02-10T11:23:58.294 INFO:teuthology.orchestra.run.gibba002.stdout: "flags_names": "hashpspool", 2021-02-10T11:23:58.294 INFO:teuthology.orchestra.run.gibba002.stdout: "type": 1, 2021-02-10T11:23:58.294 INFO:teuthology.orchestra.run.gibba002.stdout: "size": 2, 2021-02-10T11:23:58.294 INFO:teuthology.orchestra.run.gibba002.stdout: "min_size": 1, 2021-02-10T11:23:58.294 INFO:teuthology.orchestra.run.gibba002.stdout: "crush_rule": 0, 2021-02-10T11:23:58.294 INFO:teuthology.orchestra.run.gibba002.stdout: "object_hash": 2, 2021-02-10T11:23:58.295 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:23:58.295 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num": 8, 2021-02-10T11:23:58.295 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_placement_num": 8, 2021-02-10T11:23:58.295 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_placement_num_target": 8, 2021-02-10T11:23:58.295 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num_target": 8, 2021-02-10T11:23:58.295 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num_pending": 8, 2021-02-10T11:23:58.295 INFO:teuthology.orchestra.run.gibba002.stdout: "last_pg_merge_meta": { 2021-02-10T11:23:58.295 INFO:teuthology.orchestra.run.gibba002.stdout: "source_pgid": "0.0", 2021-02-10T11:23:58.295 INFO:teuthology.orchestra.run.gibba002.stdout: "ready_epoch": 0, 2021-02-10T11:23:58.295 INFO:teuthology.orchestra.run.gibba002.stdout: "last_epoch_started": 0, 2021-02-10T11:23:58.295 INFO:teuthology.orchestra.run.gibba002.stdout: "last_epoch_clean": 0, 2021-02-10T11:23:58.295 INFO:teuthology.orchestra.run.gibba002.stdout: "source_version": "0'0", 2021-02-10T11:23:58.296 INFO:teuthology.orchestra.run.gibba002.stdout: "target_version": "0'0" 2021-02-10T11:23:58.296 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:58.296 INFO:teuthology.orchestra.run.gibba002.stdout: "last_change": "47", 2021-02-10T11:23:58.296 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend": "0", 2021-02-10T11:23:58.296 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:23:58.296 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:23:58.296 INFO:teuthology.orchestra.run.gibba002.stdout: "auid": 0, 2021-02-10T11:23:58.296 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:23:58.296 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_seq": 0, 2021-02-10T11:23:58.297 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_epoch": 0, 2021-02-10T11:23:58.297 INFO:teuthology.orchestra.run.gibba002.stdout: "pool_snaps": [], 2021-02-10T11:23:58.297 INFO:teuthology.orchestra.run.gibba002.stdout: "removed_snaps": "[]", 2021-02-10T11:23:58.297 INFO:teuthology.orchestra.run.gibba002.stdout: "quota_max_bytes": 0, 2021-02-10T11:23:58.297 INFO:teuthology.orchestra.run.gibba002.stdout: "quota_max_objects": 0, 2021-02-10T11:23:58.297 INFO:teuthology.orchestra.run.gibba002.stdout: "tiers": [], 2021-02-10T11:23:58.297 INFO:teuthology.orchestra.run.gibba002.stdout: "tier_of": -1, 2021-02-10T11:23:58.297 INFO:teuthology.orchestra.run.gibba002.stdout: "read_tier": -1, 2021-02-10T11:23:58.297 INFO:teuthology.orchestra.run.gibba002.stdout: "write_tier": -1, 2021-02-10T11:23:58.298 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_mode": "none", 2021-02-10T11:23:58.298 INFO:teuthology.orchestra.run.gibba002.stdout: "target_max_bytes": 0, 2021-02-10T11:23:58.298 INFO:teuthology.orchestra.run.gibba002.stdout: "target_max_objects": 0, 2021-02-10T11:23:58.298 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:23:58.298 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:23:58.298 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:23:58.298 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_min_flush_age": 0, 2021-02-10T11:23:58.298 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_min_evict_age": 0, 2021-02-10T11:23:58.298 INFO:teuthology.orchestra.run.gibba002.stdout: "erasure_code_profile": "", 2021-02-10T11:23:58.298 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_params": { 2021-02-10T11:23:58.298 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "none" 2021-02-10T11:23:58.298 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:58.299 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_period": 0, 2021-02-10T11:23:58.299 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_count": 0, 2021-02-10T11:23:58.299 INFO:teuthology.orchestra.run.gibba002.stdout: "use_gmt_hitset": true, 2021-02-10T11:23:58.299 INFO:teuthology.orchestra.run.gibba002.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:23:58.299 INFO:teuthology.orchestra.run.gibba002.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:23:58.299 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:23:58.299 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:23:58.299 INFO:teuthology.orchestra.run.gibba002.stdout: "grade_table": [], 2021-02-10T11:23:58.299 INFO:teuthology.orchestra.run.gibba002.stdout: "stripe_width": 0, 2021-02-10T11:23:58.299 INFO:teuthology.orchestra.run.gibba002.stdout: "expected_num_objects": 0, 2021-02-10T11:23:58.299 INFO:teuthology.orchestra.run.gibba002.stdout: "fast_read": false, 2021-02-10T11:23:58.299 INFO:teuthology.orchestra.run.gibba002.stdout: "options": { 2021-02-10T11:23:58.300 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_autoscale_bias": 4, 2021-02-10T11:23:58.300 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num_min": 16, 2021-02-10T11:23:58.300 INFO:teuthology.orchestra.run.gibba002.stdout: "recovery_priority": 5 2021-02-10T11:23:58.300 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:58.300 INFO:teuthology.orchestra.run.gibba002.stdout: "application_metadata": { 2021-02-10T11:23:58.300 INFO:teuthology.orchestra.run.gibba002.stdout: "cephfs": { 2021-02-10T11:23:58.300 INFO:teuthology.orchestra.run.gibba002.stdout: "metadata": "cephfs" 2021-02-10T11:23:58.300 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:23:58.300 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:23:58.300 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:58.300 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:58.300 INFO:teuthology.orchestra.run.gibba002.stdout: "pool": 9, 2021-02-10T11:23:58.300 INFO:teuthology.orchestra.run.gibba002.stdout: "pool_name": "cephfs_data", 2021-02-10T11:23:58.300 INFO:teuthology.orchestra.run.gibba002.stdout: "create_time": "2021-02-10 11:23:17.045434", 2021-02-10T11:23:58.301 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 1, 2021-02-10T11:23:58.301 INFO:teuthology.orchestra.run.gibba002.stdout: "flags_names": "hashpspool", 2021-02-10T11:23:58.301 INFO:teuthology.orchestra.run.gibba002.stdout: "type": 1, 2021-02-10T11:23:58.301 INFO:teuthology.orchestra.run.gibba002.stdout: "size": 2, 2021-02-10T11:23:58.301 INFO:teuthology.orchestra.run.gibba002.stdout: "min_size": 1, 2021-02-10T11:23:58.301 INFO:teuthology.orchestra.run.gibba002.stdout: "crush_rule": 0, 2021-02-10T11:23:58.301 INFO:teuthology.orchestra.run.gibba002.stdout: "object_hash": 2, 2021-02-10T11:23:58.301 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:23:58.301 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num": 8, 2021-02-10T11:23:58.301 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_placement_num": 8, 2021-02-10T11:23:58.301 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_placement_num_target": 8, 2021-02-10T11:23:58.301 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num_target": 8, 2021-02-10T11:23:58.301 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num_pending": 8, 2021-02-10T11:23:58.301 INFO:teuthology.orchestra.run.gibba002.stdout: "last_pg_merge_meta": { 2021-02-10T11:23:58.302 INFO:teuthology.orchestra.run.gibba002.stdout: "source_pgid": "0.0", 2021-02-10T11:23:58.302 INFO:teuthology.orchestra.run.gibba002.stdout: "ready_epoch": 0, 2021-02-10T11:23:58.302 INFO:teuthology.orchestra.run.gibba002.stdout: "last_epoch_started": 0, 2021-02-10T11:23:58.302 INFO:teuthology.orchestra.run.gibba002.stdout: "last_epoch_clean": 0, 2021-02-10T11:23:58.302 INFO:teuthology.orchestra.run.gibba002.stdout: "source_version": "0'0", 2021-02-10T11:23:58.302 INFO:teuthology.orchestra.run.gibba002.stdout: "target_version": "0'0" 2021-02-10T11:23:58.302 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:58.302 INFO:teuthology.orchestra.run.gibba002.stdout: "last_change": "48", 2021-02-10T11:23:58.302 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend": "0", 2021-02-10T11:23:58.302 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:23:58.302 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:23:58.302 INFO:teuthology.orchestra.run.gibba002.stdout: "auid": 0, 2021-02-10T11:23:58.302 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:23:58.302 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_seq": 0, 2021-02-10T11:23:58.303 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_epoch": 0, 2021-02-10T11:23:58.303 INFO:teuthology.orchestra.run.gibba002.stdout: "pool_snaps": [], 2021-02-10T11:23:58.303 INFO:teuthology.orchestra.run.gibba002.stdout: "removed_snaps": "[]", 2021-02-10T11:23:58.303 INFO:teuthology.orchestra.run.gibba002.stdout: "quota_max_bytes": 0, 2021-02-10T11:23:58.303 INFO:teuthology.orchestra.run.gibba002.stdout: "quota_max_objects": 0, 2021-02-10T11:23:58.303 INFO:teuthology.orchestra.run.gibba002.stdout: "tiers": [], 2021-02-10T11:23:58.303 INFO:teuthology.orchestra.run.gibba002.stdout: "tier_of": -1, 2021-02-10T11:23:58.303 INFO:teuthology.orchestra.run.gibba002.stdout: "read_tier": -1, 2021-02-10T11:23:58.303 INFO:teuthology.orchestra.run.gibba002.stdout: "write_tier": -1, 2021-02-10T11:23:58.303 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_mode": "none", 2021-02-10T11:23:58.303 INFO:teuthology.orchestra.run.gibba002.stdout: "target_max_bytes": 0, 2021-02-10T11:23:58.303 INFO:teuthology.orchestra.run.gibba002.stdout: "target_max_objects": 0, 2021-02-10T11:23:58.304 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:23:58.304 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:23:58.304 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:23:58.304 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_min_flush_age": 0, 2021-02-10T11:23:58.304 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_min_evict_age": 0, 2021-02-10T11:23:58.304 INFO:teuthology.orchestra.run.gibba002.stdout: "erasure_code_profile": "", 2021-02-10T11:23:58.304 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_params": { 2021-02-10T11:23:58.304 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "none" 2021-02-10T11:23:58.304 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:58.304 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_period": 0, 2021-02-10T11:23:58.304 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_count": 0, 2021-02-10T11:23:58.304 INFO:teuthology.orchestra.run.gibba002.stdout: "use_gmt_hitset": true, 2021-02-10T11:23:58.304 INFO:teuthology.orchestra.run.gibba002.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:23:58.304 INFO:teuthology.orchestra.run.gibba002.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:23:58.305 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:23:58.305 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:23:58.305 INFO:teuthology.orchestra.run.gibba002.stdout: "grade_table": [], 2021-02-10T11:23:58.305 INFO:teuthology.orchestra.run.gibba002.stdout: "stripe_width": 0, 2021-02-10T11:23:58.305 INFO:teuthology.orchestra.run.gibba002.stdout: "expected_num_objects": 0, 2021-02-10T11:23:58.305 INFO:teuthology.orchestra.run.gibba002.stdout: "fast_read": false, 2021-02-10T11:23:58.305 INFO:teuthology.orchestra.run.gibba002.stdout: "options": {}, 2021-02-10T11:23:58.305 INFO:teuthology.orchestra.run.gibba002.stdout: "application_metadata": { 2021-02-10T11:23:58.305 INFO:teuthology.orchestra.run.gibba002.stdout: "cephfs": { 2021-02-10T11:23:58.305 INFO:teuthology.orchestra.run.gibba002.stdout: "data": "cephfs" 2021-02-10T11:23:58.305 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:23:58.305 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:23:58.306 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:23:58.306 INFO:teuthology.orchestra.run.gibba002.stdout: ], 2021-02-10T11:23:58.306 INFO:teuthology.orchestra.run.gibba002.stdout: "osds": [ 2021-02-10T11:23:58.306 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:58.306 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 0, 2021-02-10T11:23:58.306 INFO:teuthology.orchestra.run.gibba002.stdout: "uuid": "be691b4d-a039-408a-af80-9d53f02b6f1f", 2021-02-10T11:23:58.306 INFO:teuthology.orchestra.run.gibba002.stdout: "up": 1, 2021-02-10T11:23:58.306 INFO:teuthology.orchestra.run.gibba002.stdout: "in": 1, 2021-02-10T11:23:58.306 INFO:teuthology.orchestra.run.gibba002.stdout: "weight": 1, 2021-02-10T11:23:58.306 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_affinity": 1, 2021-02-10T11:23:58.306 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_begin": 0, 2021-02-10T11:23:58.306 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_end": 0, 2021-02-10T11:23:58.306 INFO:teuthology.orchestra.run.gibba002.stdout: "up_from": 13, 2021-02-10T11:23:58.307 INFO:teuthology.orchestra.run.gibba002.stdout: "up_thru": 46, 2021-02-10T11:23:58.307 INFO:teuthology.orchestra.run.gibba002.stdout: "down_at": 0, 2021-02-10T11:23:58.307 INFO:teuthology.orchestra.run.gibba002.stdout: "lost_at": 0, 2021-02-10T11:23:58.307 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addrs": { 2021-02-10T11:23:58.307 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:23:58.307 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:58.307 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:23:58.307 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6818", 2021-02-10T11:23:58.307 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36088 2021-02-10T11:23:58.307 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:58.308 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:58.308 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:23:58.308 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6820", 2021-02-10T11:23:58.308 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36088 2021-02-10T11:23:58.308 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:23:58.308 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:23:58.308 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:58.308 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addrs": { 2021-02-10T11:23:58.308 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:23:58.308 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:58.308 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:23:58.308 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6822", 2021-02-10T11:23:58.309 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36088 2021-02-10T11:23:58.309 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:58.309 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:58.309 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:23:58.309 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6824", 2021-02-10T11:23:58.309 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36088 2021-02-10T11:23:58.309 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:23:58.309 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:23:58.309 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:58.309 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addrs": { 2021-02-10T11:23:58.309 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:23:58.309 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:58.310 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:23:58.310 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6830", 2021-02-10T11:23:58.310 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36088 2021-02-10T11:23:58.310 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:58.310 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:58.310 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:23:58.310 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6832", 2021-02-10T11:23:58.310 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36088 2021-02-10T11:23:58.310 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:23:58.310 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:23:58.310 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:58.310 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addrs": { 2021-02-10T11:23:58.310 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:23:58.311 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:58.311 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:23:58.311 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6826", 2021-02-10T11:23:58.311 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36088 2021-02-10T11:23:58.311 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:58.311 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:58.311 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:23:58.311 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6828", 2021-02-10T11:23:58.311 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36088 2021-02-10T11:23:58.313 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:23:58.313 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:23:58.313 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:58.313 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addr": "172.21.2.102:6820/36088", 2021-02-10T11:23:58.314 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addr": "172.21.2.102:6824/36088", 2021-02-10T11:23:58.314 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addr": "172.21.2.102:6832/36088", 2021-02-10T11:23:58.314 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addr": "172.21.2.102:6828/36088", 2021-02-10T11:23:58.314 INFO:teuthology.orchestra.run.gibba002.stdout: "state": [ 2021-02-10T11:23:58.314 INFO:teuthology.orchestra.run.gibba002.stdout: "exists", 2021-02-10T11:23:58.314 INFO:teuthology.orchestra.run.gibba002.stdout: "up" 2021-02-10T11:23:58.314 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:23:58.314 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:58.314 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:58.315 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 1, 2021-02-10T11:23:58.315 INFO:teuthology.orchestra.run.gibba002.stdout: "uuid": "64e3ccc8-9c69-4d7c-865e-af8e83822635", 2021-02-10T11:23:58.315 INFO:teuthology.orchestra.run.gibba002.stdout: "up": 1, 2021-02-10T11:23:58.315 INFO:teuthology.orchestra.run.gibba002.stdout: "in": 1, 2021-02-10T11:23:58.315 INFO:teuthology.orchestra.run.gibba002.stdout: "weight": 1, 2021-02-10T11:23:58.315 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_affinity": 1, 2021-02-10T11:23:58.315 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_begin": 0, 2021-02-10T11:23:58.315 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_end": 0, 2021-02-10T11:23:58.315 INFO:teuthology.orchestra.run.gibba002.stdout: "up_from": 13, 2021-02-10T11:23:58.315 INFO:teuthology.orchestra.run.gibba002.stdout: "up_thru": 46, 2021-02-10T11:23:58.315 INFO:teuthology.orchestra.run.gibba002.stdout: "down_at": 0, 2021-02-10T11:23:58.315 INFO:teuthology.orchestra.run.gibba002.stdout: "lost_at": 0, 2021-02-10T11:23:58.315 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addrs": { 2021-02-10T11:23:58.315 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:23:58.316 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:58.316 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:23:58.316 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6819", 2021-02-10T11:23:58.316 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36087 2021-02-10T11:23:58.316 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:58.316 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:58.316 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:23:58.316 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6821", 2021-02-10T11:23:58.316 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36087 2021-02-10T11:23:58.316 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:23:58.316 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:23:58.316 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:58.316 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addrs": { 2021-02-10T11:23:58.317 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:23:58.317 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:58.317 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:23:58.317 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6823", 2021-02-10T11:23:58.317 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36087 2021-02-10T11:23:58.317 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:58.317 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:58.317 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:23:58.317 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6825", 2021-02-10T11:23:58.317 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36087 2021-02-10T11:23:58.317 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:23:58.317 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:23:58.317 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:58.317 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addrs": { 2021-02-10T11:23:58.318 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:23:58.318 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:58.318 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:23:58.318 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6831", 2021-02-10T11:23:58.318 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36087 2021-02-10T11:23:58.318 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:58.318 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:58.318 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:23:58.318 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6833", 2021-02-10T11:23:58.318 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36087 2021-02-10T11:23:58.318 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:23:58.318 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:23:58.318 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:58.318 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addrs": { 2021-02-10T11:23:58.319 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:23:58.319 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:58.319 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:23:58.319 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6827", 2021-02-10T11:23:58.319 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36087 2021-02-10T11:23:58.319 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:58.319 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:58.319 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:23:58.319 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6829", 2021-02-10T11:23:58.319 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36087 2021-02-10T11:23:58.319 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:23:58.319 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:23:58.319 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:58.319 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addr": "172.21.2.102:6821/36087", 2021-02-10T11:23:58.320 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addr": "172.21.2.102:6825/36087", 2021-02-10T11:23:58.320 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addr": "172.21.2.102:6833/36087", 2021-02-10T11:23:58.320 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addr": "172.21.2.102:6829/36087", 2021-02-10T11:23:58.320 INFO:teuthology.orchestra.run.gibba002.stdout: "state": [ 2021-02-10T11:23:58.320 INFO:teuthology.orchestra.run.gibba002.stdout: "exists", 2021-02-10T11:23:58.320 INFO:teuthology.orchestra.run.gibba002.stdout: "up" 2021-02-10T11:23:58.320 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:23:58.320 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:58.320 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:58.320 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 2, 2021-02-10T11:23:58.320 INFO:teuthology.orchestra.run.gibba002.stdout: "uuid": "153700af-747f-481f-bdf4-d6072d3a842f", 2021-02-10T11:23:58.320 INFO:teuthology.orchestra.run.gibba002.stdout: "up": 1, 2021-02-10T11:23:58.320 INFO:teuthology.orchestra.run.gibba002.stdout: "in": 1, 2021-02-10T11:23:58.320 INFO:teuthology.orchestra.run.gibba002.stdout: "weight": 1, 2021-02-10T11:23:58.321 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_affinity": 1, 2021-02-10T11:23:58.321 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_begin": 0, 2021-02-10T11:23:58.321 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_end": 0, 2021-02-10T11:23:58.321 INFO:teuthology.orchestra.run.gibba002.stdout: "up_from": 13, 2021-02-10T11:23:58.321 INFO:teuthology.orchestra.run.gibba002.stdout: "up_thru": 28, 2021-02-10T11:23:58.321 INFO:teuthology.orchestra.run.gibba002.stdout: "down_at": 0, 2021-02-10T11:23:58.321 INFO:teuthology.orchestra.run.gibba002.stdout: "lost_at": 0, 2021-02-10T11:23:58.321 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addrs": { 2021-02-10T11:23:58.321 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:23:58.321 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:58.321 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:23:58.321 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6810", 2021-02-10T11:23:58.321 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36089 2021-02-10T11:23:58.321 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:58.322 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:58.322 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:23:58.322 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6811", 2021-02-10T11:23:58.322 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36089 2021-02-10T11:23:58.323 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:23:58.323 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:23:58.323 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:58.323 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addrs": { 2021-02-10T11:23:58.323 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:23:58.323 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:58.323 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:23:58.323 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6812", 2021-02-10T11:23:58.323 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36089 2021-02-10T11:23:58.323 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:58.324 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:58.324 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:23:58.324 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6813", 2021-02-10T11:23:58.324 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36089 2021-02-10T11:23:58.324 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:23:58.324 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:23:58.324 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:58.324 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addrs": { 2021-02-10T11:23:58.324 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:23:58.324 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:58.324 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:23:58.324 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6816", 2021-02-10T11:23:58.324 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36089 2021-02-10T11:23:58.324 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:58.325 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:58.325 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:23:58.325 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6817", 2021-02-10T11:23:58.325 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36089 2021-02-10T11:23:58.325 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:23:58.325 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:23:58.325 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:58.325 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addrs": { 2021-02-10T11:23:58.325 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:23:58.325 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:58.325 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:23:58.325 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6814", 2021-02-10T11:23:58.325 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36089 2021-02-10T11:23:58.325 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:58.326 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:58.326 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:23:58.326 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6815", 2021-02-10T11:23:58.326 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36089 2021-02-10T11:23:58.326 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:23:58.326 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:23:58.326 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:58.326 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addr": "172.21.2.102:6811/36089", 2021-02-10T11:23:58.326 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addr": "172.21.2.102:6813/36089", 2021-02-10T11:23:58.326 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addr": "172.21.2.102:6817/36089", 2021-02-10T11:23:58.326 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addr": "172.21.2.102:6815/36089", 2021-02-10T11:23:58.327 INFO:teuthology.orchestra.run.gibba002.stdout: "state": [ 2021-02-10T11:23:58.327 INFO:teuthology.orchestra.run.gibba002.stdout: "exists", 2021-02-10T11:23:58.327 INFO:teuthology.orchestra.run.gibba002.stdout: "up" 2021-02-10T11:23:58.327 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:23:58.327 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:58.327 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:58.327 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 3, 2021-02-10T11:23:58.327 INFO:teuthology.orchestra.run.gibba002.stdout: "uuid": "cbc678f4-83a9-45e6-8830-2f7c232b1ea7", 2021-02-10T11:23:58.327 INFO:teuthology.orchestra.run.gibba002.stdout: "up": 1, 2021-02-10T11:23:58.327 INFO:teuthology.orchestra.run.gibba002.stdout: "in": 1, 2021-02-10T11:23:58.327 INFO:teuthology.orchestra.run.gibba002.stdout: "weight": 1, 2021-02-10T11:23:58.327 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_affinity": 1, 2021-02-10T11:23:58.327 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_begin": 0, 2021-02-10T11:23:58.327 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_end": 0, 2021-02-10T11:23:58.328 INFO:teuthology.orchestra.run.gibba002.stdout: "up_from": 13, 2021-02-10T11:23:58.328 INFO:teuthology.orchestra.run.gibba002.stdout: "up_thru": 46, 2021-02-10T11:23:58.328 INFO:teuthology.orchestra.run.gibba002.stdout: "down_at": 0, 2021-02-10T11:23:58.328 INFO:teuthology.orchestra.run.gibba002.stdout: "lost_at": 0, 2021-02-10T11:23:58.328 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addrs": { 2021-02-10T11:23:58.328 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:23:58.328 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:58.328 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:23:58.328 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6802", 2021-02-10T11:23:58.328 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36090 2021-02-10T11:23:58.328 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:58.328 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:58.328 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:23:58.329 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6803", 2021-02-10T11:23:58.329 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36090 2021-02-10T11:23:58.329 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:23:58.329 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:23:58.329 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:58.329 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addrs": { 2021-02-10T11:23:58.329 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:23:58.329 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:58.329 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:23:58.329 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6804", 2021-02-10T11:23:58.329 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36090 2021-02-10T11:23:58.329 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:58.330 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:58.330 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:23:58.330 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6805", 2021-02-10T11:23:58.330 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36090 2021-02-10T11:23:58.330 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:23:58.331 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:23:58.331 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:58.331 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addrs": { 2021-02-10T11:23:58.331 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:23:58.331 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:58.331 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:23:58.331 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6808", 2021-02-10T11:23:58.331 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36090 2021-02-10T11:23:58.331 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:58.331 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:58.331 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:23:58.331 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6809", 2021-02-10T11:23:58.332 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36090 2021-02-10T11:23:58.332 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:23:58.332 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:23:58.332 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:58.332 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addrs": { 2021-02-10T11:23:58.332 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:23:58.332 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:58.332 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:23:58.332 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6806", 2021-02-10T11:23:58.332 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36090 2021-02-10T11:23:58.333 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:58.333 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:58.333 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:23:58.333 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6807", 2021-02-10T11:23:58.333 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36090 2021-02-10T11:23:58.333 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:23:58.333 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:23:58.333 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:58.333 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addr": "172.21.2.102:6803/36090", 2021-02-10T11:23:58.333 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addr": "172.21.2.102:6805/36090", 2021-02-10T11:23:58.333 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addr": "172.21.2.102:6809/36090", 2021-02-10T11:23:58.334 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addr": "172.21.2.102:6807/36090", 2021-02-10T11:23:58.334 INFO:teuthology.orchestra.run.gibba002.stdout: "state": [ 2021-02-10T11:23:58.334 INFO:teuthology.orchestra.run.gibba002.stdout: "exists", 2021-02-10T11:23:58.334 INFO:teuthology.orchestra.run.gibba002.stdout: "up" 2021-02-10T11:23:58.334 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:23:58.334 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:58.334 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:58.334 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 4, 2021-02-10T11:23:58.334 INFO:teuthology.orchestra.run.gibba002.stdout: "uuid": "c01e4008-b258-4f10-afd2-9e7f37214351", 2021-02-10T11:23:58.334 INFO:teuthology.orchestra.run.gibba002.stdout: "up": 1, 2021-02-10T11:23:58.334 INFO:teuthology.orchestra.run.gibba002.stdout: "in": 1, 2021-02-10T11:23:58.335 INFO:teuthology.orchestra.run.gibba002.stdout: "weight": 1, 2021-02-10T11:23:58.335 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_affinity": 1, 2021-02-10T11:23:58.335 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_begin": 0, 2021-02-10T11:23:58.335 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_end": 0, 2021-02-10T11:23:58.335 INFO:teuthology.orchestra.run.gibba002.stdout: "up_from": 13, 2021-02-10T11:23:58.335 INFO:teuthology.orchestra.run.gibba002.stdout: "up_thru": 44, 2021-02-10T11:23:58.335 INFO:teuthology.orchestra.run.gibba002.stdout: "down_at": 0, 2021-02-10T11:23:58.335 INFO:teuthology.orchestra.run.gibba002.stdout: "lost_at": 0, 2021-02-10T11:23:58.335 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addrs": { 2021-02-10T11:23:58.335 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:23:58.335 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:58.335 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:23:58.336 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6808", 2021-02-10T11:23:58.336 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36368 2021-02-10T11:23:58.336 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:58.336 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:58.336 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:23:58.336 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6811", 2021-02-10T11:23:58.336 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36368 2021-02-10T11:23:58.336 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:23:58.336 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:23:58.336 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:58.336 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addrs": { 2021-02-10T11:23:58.336 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:23:58.336 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:58.336 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:23:58.337 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6814", 2021-02-10T11:23:58.337 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36368 2021-02-10T11:23:58.337 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:58.337 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:58.337 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:23:58.337 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6818", 2021-02-10T11:23:58.337 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36368 2021-02-10T11:23:58.337 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:23:58.337 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:23:58.337 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:58.337 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addrs": { 2021-02-10T11:23:58.337 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:23:58.337 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:58.337 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:23:58.338 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6826", 2021-02-10T11:23:58.338 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36368 2021-02-10T11:23:58.338 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:58.338 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:58.338 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:23:58.338 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6828", 2021-02-10T11:23:58.338 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36368 2021-02-10T11:23:58.338 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:23:58.338 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:23:58.338 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:58.338 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addrs": { 2021-02-10T11:23:58.338 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:23:58.338 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:58.339 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:23:58.339 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6820", 2021-02-10T11:23:58.339 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36368 2021-02-10T11:23:58.339 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:58.339 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:58.339 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:23:58.339 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6824", 2021-02-10T11:23:58.339 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36368 2021-02-10T11:23:58.339 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:23:58.339 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:23:58.339 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:58.339 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addr": "172.21.2.113:6811/36368", 2021-02-10T11:23:58.339 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addr": "172.21.2.113:6818/36368", 2021-02-10T11:23:58.339 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addr": "172.21.2.113:6828/36368", 2021-02-10T11:23:58.340 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addr": "172.21.2.113:6824/36368", 2021-02-10T11:23:58.340 INFO:teuthology.orchestra.run.gibba002.stdout: "state": [ 2021-02-10T11:23:58.340 INFO:teuthology.orchestra.run.gibba002.stdout: "exists", 2021-02-10T11:23:58.340 INFO:teuthology.orchestra.run.gibba002.stdout: "up" 2021-02-10T11:23:58.340 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:23:58.340 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:58.340 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:58.340 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 5, 2021-02-10T11:23:58.340 INFO:teuthology.orchestra.run.gibba002.stdout: "uuid": "551f6007-daf8-4a98-8ee0-c2b16bcccb5e", 2021-02-10T11:23:58.340 INFO:teuthology.orchestra.run.gibba002.stdout: "up": 1, 2021-02-10T11:23:58.340 INFO:teuthology.orchestra.run.gibba002.stdout: "in": 1, 2021-02-10T11:23:58.340 INFO:teuthology.orchestra.run.gibba002.stdout: "weight": 1, 2021-02-10T11:23:58.340 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_affinity": 1, 2021-02-10T11:23:58.341 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_begin": 0, 2021-02-10T11:23:58.341 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_end": 0, 2021-02-10T11:23:58.341 INFO:teuthology.orchestra.run.gibba002.stdout: "up_from": 13, 2021-02-10T11:23:58.341 INFO:teuthology.orchestra.run.gibba002.stdout: "up_thru": 46, 2021-02-10T11:23:58.341 INFO:teuthology.orchestra.run.gibba002.stdout: "down_at": 0, 2021-02-10T11:23:58.341 INFO:teuthology.orchestra.run.gibba002.stdout: "lost_at": 0, 2021-02-10T11:23:58.341 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addrs": { 2021-02-10T11:23:58.341 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:23:58.341 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:58.341 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:23:58.341 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6802", 2021-02-10T11:23:58.341 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36366 2021-02-10T11:23:58.341 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:58.341 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:58.342 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:23:58.342 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6805", 2021-02-10T11:23:58.342 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36366 2021-02-10T11:23:58.342 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:23:58.342 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:23:58.342 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:58.342 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addrs": { 2021-02-10T11:23:58.342 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:23:58.342 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:58.342 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:23:58.342 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6807", 2021-02-10T11:23:58.342 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36366 2021-02-10T11:23:58.342 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:58.343 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:58.343 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:23:58.343 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6810", 2021-02-10T11:23:58.343 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36366 2021-02-10T11:23:58.343 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:23:58.343 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:23:58.344 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:58.344 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addrs": { 2021-02-10T11:23:58.344 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:23:58.344 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:58.344 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:23:58.344 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6819", 2021-02-10T11:23:58.344 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36366 2021-02-10T11:23:58.345 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:58.345 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:58.345 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:23:58.345 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6822", 2021-02-10T11:23:58.345 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36366 2021-02-10T11:23:58.345 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:23:58.345 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:23:58.345 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:58.345 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addrs": { 2021-02-10T11:23:58.345 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:23:58.345 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:58.345 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:23:58.345 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6812", 2021-02-10T11:23:58.345 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36366 2021-02-10T11:23:58.346 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:58.346 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:58.346 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:23:58.346 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6815", 2021-02-10T11:23:58.346 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36366 2021-02-10T11:23:58.346 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:23:58.346 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:23:58.346 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:58.346 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addr": "172.21.2.113:6805/36366", 2021-02-10T11:23:58.346 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addr": "172.21.2.113:6810/36366", 2021-02-10T11:23:58.346 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addr": "172.21.2.113:6822/36366", 2021-02-10T11:23:58.346 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addr": "172.21.2.113:6815/36366", 2021-02-10T11:23:58.346 INFO:teuthology.orchestra.run.gibba002.stdout: "state": [ 2021-02-10T11:23:58.347 INFO:teuthology.orchestra.run.gibba002.stdout: "exists", 2021-02-10T11:23:58.347 INFO:teuthology.orchestra.run.gibba002.stdout: "up" 2021-02-10T11:23:58.347 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:23:58.347 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:58.347 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:58.347 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 6, 2021-02-10T11:23:58.347 INFO:teuthology.orchestra.run.gibba002.stdout: "uuid": "bf67c040-0e50-48be-b102-c3943bc9ebef", 2021-02-10T11:23:58.347 INFO:teuthology.orchestra.run.gibba002.stdout: "up": 1, 2021-02-10T11:23:58.347 INFO:teuthology.orchestra.run.gibba002.stdout: "in": 1, 2021-02-10T11:23:58.347 INFO:teuthology.orchestra.run.gibba002.stdout: "weight": 1, 2021-02-10T11:23:58.347 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_affinity": 1, 2021-02-10T11:23:58.347 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_begin": 0, 2021-02-10T11:23:58.347 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_end": 0, 2021-02-10T11:23:58.347 INFO:teuthology.orchestra.run.gibba002.stdout: "up_from": 13, 2021-02-10T11:23:58.348 INFO:teuthology.orchestra.run.gibba002.stdout: "up_thru": 44, 2021-02-10T11:23:58.348 INFO:teuthology.orchestra.run.gibba002.stdout: "down_at": 0, 2021-02-10T11:23:58.348 INFO:teuthology.orchestra.run.gibba002.stdout: "lost_at": 0, 2021-02-10T11:23:58.348 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addrs": { 2021-02-10T11:23:58.348 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:23:58.348 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:58.348 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:23:58.348 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6800", 2021-02-10T11:23:58.348 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36367 2021-02-10T11:23:58.348 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:58.348 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:58.348 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:23:58.348 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6801", 2021-02-10T11:23:58.348 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36367 2021-02-10T11:23:58.349 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:23:58.349 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:23:58.349 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:58.349 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addrs": { 2021-02-10T11:23:58.349 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:23:58.349 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:58.349 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:23:58.349 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6803", 2021-02-10T11:23:58.349 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36367 2021-02-10T11:23:58.349 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:58.349 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:58.349 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:23:58.349 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6804", 2021-02-10T11:23:58.350 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36367 2021-02-10T11:23:58.350 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:23:58.350 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:23:58.350 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:58.350 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addrs": { 2021-02-10T11:23:58.350 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:23:58.350 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:58.350 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:23:58.350 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6813", 2021-02-10T11:23:58.350 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36367 2021-02-10T11:23:58.350 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:58.350 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:58.350 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:23:58.351 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6816", 2021-02-10T11:23:58.351 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36367 2021-02-10T11:23:58.351 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:23:58.351 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:23:58.351 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:58.351 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addrs": { 2021-02-10T11:23:58.351 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:23:58.351 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:58.351 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:23:58.351 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6806", 2021-02-10T11:23:58.351 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36367 2021-02-10T11:23:58.351 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:58.351 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:58.352 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:23:58.352 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6809", 2021-02-10T11:23:58.352 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36367 2021-02-10T11:23:58.352 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:23:58.352 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:23:58.352 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:58.352 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addr": "172.21.2.113:6801/36367", 2021-02-10T11:23:58.352 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addr": "172.21.2.113:6804/36367", 2021-02-10T11:23:58.352 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addr": "172.21.2.113:6816/36367", 2021-02-10T11:23:58.352 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addr": "172.21.2.113:6809/36367", 2021-02-10T11:23:58.352 INFO:teuthology.orchestra.run.gibba002.stdout: "state": [ 2021-02-10T11:23:58.353 INFO:teuthology.orchestra.run.gibba002.stdout: "exists", 2021-02-10T11:23:58.353 INFO:teuthology.orchestra.run.gibba002.stdout: "up" 2021-02-10T11:23:58.353 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:23:58.353 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:58.353 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:58.353 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 7, 2021-02-10T11:23:58.353 INFO:teuthology.orchestra.run.gibba002.stdout: "uuid": "f5461f80-018d-428f-be89-2eb2bcbe7bb6", 2021-02-10T11:23:58.353 INFO:teuthology.orchestra.run.gibba002.stdout: "up": 1, 2021-02-10T11:23:58.353 INFO:teuthology.orchestra.run.gibba002.stdout: "in": 1, 2021-02-10T11:23:58.353 INFO:teuthology.orchestra.run.gibba002.stdout: "weight": 1, 2021-02-10T11:23:58.354 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_affinity": 1, 2021-02-10T11:23:58.354 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_begin": 0, 2021-02-10T11:23:58.354 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_end": 0, 2021-02-10T11:23:58.354 INFO:teuthology.orchestra.run.gibba002.stdout: "up_from": 13, 2021-02-10T11:23:58.354 INFO:teuthology.orchestra.run.gibba002.stdout: "up_thru": 46, 2021-02-10T11:23:58.354 INFO:teuthology.orchestra.run.gibba002.stdout: "down_at": 0, 2021-02-10T11:23:58.354 INFO:teuthology.orchestra.run.gibba002.stdout: "lost_at": 0, 2021-02-10T11:23:58.354 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addrs": { 2021-02-10T11:23:58.354 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:23:58.355 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:58.355 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:23:58.355 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6817", 2021-02-10T11:23:58.355 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36365 2021-02-10T11:23:58.355 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:58.355 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:58.355 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:23:58.355 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6821", 2021-02-10T11:23:58.355 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36365 2021-02-10T11:23:58.355 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:23:58.355 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:23:58.355 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:58.355 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addrs": { 2021-02-10T11:23:58.355 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:23:58.356 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:58.356 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:23:58.356 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6823", 2021-02-10T11:23:58.356 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36365 2021-02-10T11:23:58.356 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:58.356 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:58.356 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:23:58.356 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6825", 2021-02-10T11:23:58.356 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36365 2021-02-10T11:23:58.356 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:23:58.356 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:23:58.356 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:58.357 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addrs": { 2021-02-10T11:23:58.357 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:23:58.357 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:58.357 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:23:58.357 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6830", 2021-02-10T11:23:58.357 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36365 2021-02-10T11:23:58.357 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:58.357 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:58.357 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:23:58.357 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6831", 2021-02-10T11:23:58.357 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36365 2021-02-10T11:23:58.357 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:23:58.357 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:23:58.357 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:58.358 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addrs": { 2021-02-10T11:23:58.358 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:23:58.358 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:58.358 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:23:58.358 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6827", 2021-02-10T11:23:58.358 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36365 2021-02-10T11:23:58.358 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:58.358 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:58.358 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:23:58.358 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6829", 2021-02-10T11:23:58.358 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36365 2021-02-10T11:23:58.358 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:23:58.358 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:23:58.358 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:58.359 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addr": "172.21.2.113:6821/36365", 2021-02-10T11:23:58.359 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addr": "172.21.2.113:6825/36365", 2021-02-10T11:23:58.359 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addr": "172.21.2.113:6831/36365", 2021-02-10T11:23:58.359 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addr": "172.21.2.113:6829/36365", 2021-02-10T11:23:58.359 INFO:teuthology.orchestra.run.gibba002.stdout: "state": [ 2021-02-10T11:23:58.359 INFO:teuthology.orchestra.run.gibba002.stdout: "exists", 2021-02-10T11:23:58.359 INFO:teuthology.orchestra.run.gibba002.stdout: "up" 2021-02-10T11:23:58.359 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:23:58.359 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:23:58.359 INFO:teuthology.orchestra.run.gibba002.stdout: ], 2021-02-10T11:23:58.359 INFO:teuthology.orchestra.run.gibba002.stdout: "osd_xinfo": [ 2021-02-10T11:23:58.359 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:58.359 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 0, 2021-02-10T11:23:58.359 INFO:teuthology.orchestra.run.gibba002.stdout: "down_stamp": "0.000000", 2021-02-10T11:23:58.360 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_probability": 0, 2021-02-10T11:23:58.360 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_interval": 0, 2021-02-10T11:23:58.360 INFO:teuthology.orchestra.run.gibba002.stdout: "features": 4611087854035861503, 2021-02-10T11:23:58.360 INFO:teuthology.orchestra.run.gibba002.stdout: "old_weight": 0 2021-02-10T11:23:58.360 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:58.360 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:58.360 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 1, 2021-02-10T11:23:58.360 INFO:teuthology.orchestra.run.gibba002.stdout: "down_stamp": "0.000000", 2021-02-10T11:23:58.360 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_probability": 0, 2021-02-10T11:23:58.360 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_interval": 0, 2021-02-10T11:23:58.360 INFO:teuthology.orchestra.run.gibba002.stdout: "features": 4611087854035861503, 2021-02-10T11:23:58.360 INFO:teuthology.orchestra.run.gibba002.stdout: "old_weight": 0 2021-02-10T11:23:58.360 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:58.361 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:58.361 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 2, 2021-02-10T11:23:58.361 INFO:teuthology.orchestra.run.gibba002.stdout: "down_stamp": "0.000000", 2021-02-10T11:23:58.361 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_probability": 0, 2021-02-10T11:23:58.361 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_interval": 0, 2021-02-10T11:23:58.361 INFO:teuthology.orchestra.run.gibba002.stdout: "features": 4611087854035861503, 2021-02-10T11:23:58.361 INFO:teuthology.orchestra.run.gibba002.stdout: "old_weight": 0 2021-02-10T11:23:58.361 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:58.361 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:58.361 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 3, 2021-02-10T11:23:58.361 INFO:teuthology.orchestra.run.gibba002.stdout: "down_stamp": "0.000000", 2021-02-10T11:23:58.361 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_probability": 0, 2021-02-10T11:23:58.361 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_interval": 0, 2021-02-10T11:23:58.361 INFO:teuthology.orchestra.run.gibba002.stdout: "features": 4611087854035861503, 2021-02-10T11:23:58.362 INFO:teuthology.orchestra.run.gibba002.stdout: "old_weight": 0 2021-02-10T11:23:58.362 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:58.362 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:58.362 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 4, 2021-02-10T11:23:58.362 INFO:teuthology.orchestra.run.gibba002.stdout: "down_stamp": "0.000000", 2021-02-10T11:23:58.362 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_probability": 0, 2021-02-10T11:23:58.362 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_interval": 0, 2021-02-10T11:23:58.362 INFO:teuthology.orchestra.run.gibba002.stdout: "features": 4611087854035861503, 2021-02-10T11:23:58.362 INFO:teuthology.orchestra.run.gibba002.stdout: "old_weight": 0 2021-02-10T11:23:58.362 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:58.362 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:58.362 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 5, 2021-02-10T11:23:58.362 INFO:teuthology.orchestra.run.gibba002.stdout: "down_stamp": "0.000000", 2021-02-10T11:23:58.363 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_probability": 0, 2021-02-10T11:23:58.363 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_interval": 0, 2021-02-10T11:23:58.363 INFO:teuthology.orchestra.run.gibba002.stdout: "features": 4611087854035861503, 2021-02-10T11:23:58.363 INFO:teuthology.orchestra.run.gibba002.stdout: "old_weight": 0 2021-02-10T11:23:58.363 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:58.363 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:58.363 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 6, 2021-02-10T11:23:58.363 INFO:teuthology.orchestra.run.gibba002.stdout: "down_stamp": "0.000000", 2021-02-10T11:23:58.363 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_probability": 0, 2021-02-10T11:23:58.363 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_interval": 0, 2021-02-10T11:23:58.363 INFO:teuthology.orchestra.run.gibba002.stdout: "features": 4611087854035861503, 2021-02-10T11:23:58.363 INFO:teuthology.orchestra.run.gibba002.stdout: "old_weight": 0 2021-02-10T11:23:58.363 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:58.363 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:23:58.364 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 7, 2021-02-10T11:23:58.364 INFO:teuthology.orchestra.run.gibba002.stdout: "down_stamp": "0.000000", 2021-02-10T11:23:58.364 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_probability": 0, 2021-02-10T11:23:58.364 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_interval": 0, 2021-02-10T11:23:58.364 INFO:teuthology.orchestra.run.gibba002.stdout: "features": 4611087854035861503, 2021-02-10T11:23:58.364 INFO:teuthology.orchestra.run.gibba002.stdout: "old_weight": 0 2021-02-10T11:23:58.364 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:23:58.364 INFO:teuthology.orchestra.run.gibba002.stdout: ], 2021-02-10T11:23:58.364 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_upmap": [], 2021-02-10T11:23:58.364 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_upmap_items": [], 2021-02-10T11:23:58.365 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_temp": [], 2021-02-10T11:23:58.365 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_temp": [], 2021-02-10T11:23:58.365 INFO:teuthology.orchestra.run.gibba002.stdout: "blacklist": {}, 2021-02-10T11:23:58.365 INFO:teuthology.orchestra.run.gibba002.stdout: "erasure_code_profiles": { 2021-02-10T11:23:58.365 INFO:teuthology.orchestra.run.gibba002.stdout: "default": { 2021-02-10T11:23:58.365 INFO:teuthology.orchestra.run.gibba002.stdout: "crush-failure-domain": "osd", 2021-02-10T11:23:58.365 INFO:teuthology.orchestra.run.gibba002.stdout: "k": "2", 2021-02-10T11:23:58.365 INFO:teuthology.orchestra.run.gibba002.stdout: "m": "1", 2021-02-10T11:23:58.365 INFO:teuthology.orchestra.run.gibba002.stdout: "plugin": "jerasure", 2021-02-10T11:23:58.365 INFO:teuthology.orchestra.run.gibba002.stdout: "ruleset-failure-domain": "osd", 2021-02-10T11:23:58.365 INFO:teuthology.orchestra.run.gibba002.stdout: "technique": "reed_sol_van" 2021-02-10T11:23:58.365 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:23:58.365 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:23:58.365 INFO:teuthology.orchestra.run.gibba002.stdout: "removed_snaps_queue": [], 2021-02-10T11:23:58.366 INFO:teuthology.orchestra.run.gibba002.stdout: "new_removed_snaps": [], 2021-02-10T11:23:58.366 INFO:teuthology.orchestra.run.gibba002.stdout: "new_purged_snaps": [], 2021-02-10T11:23:58.366 INFO:teuthology.orchestra.run.gibba002.stdout: "crush_node_flags": {}, 2021-02-10T11:23:58.366 INFO:teuthology.orchestra.run.gibba002.stdout: "device_class_flags": {} 2021-02-10T11:23:58.366 INFO:teuthology.orchestra.run.gibba002.stdout:} 2021-02-10T11:23:58.367 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:58.287 7f6cd3d6b700 1 -- 172.21.2.102:0/1303804882 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6cb4018730 msgr2=0x7f6cb401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:58.367 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:58.287 7f6cd3d6b700 1 --2- 172.21.2.102:0/1303804882 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6cb4018730 0x7f6cb401ac10 crc :-1 s=READY pgs=599 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:58.367 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:58.287 7f6cd3d6b700 1 -- 172.21.2.102:0/1303804882 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6ccc0b1440 msgr2=0x7f6ccc0b1840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:58.367 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:58.287 7f6cd3d6b700 1 --2- 172.21.2.102:0/1303804882 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6ccc0b1440 0x7f6ccc0b1840 secure :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 rx=0x7f6cc4009980 tx=0x7f6cc400bcd0).stop 2021-02-10T11:23:58.367 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:58.287 7f6cd3d6b700 1 -- 172.21.2.102:0/1303804882 shutdown_connections 2021-02-10T11:23:58.367 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:58.287 7f6cd3d6b700 1 --2- 172.21.2.102:0/1303804882 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6ccc0b1440 0x7f6ccc0b1840 unknown :-1 s=CLOSED pgs=1272 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:58.368 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:58.287 7f6cd3d6b700 1 --2- 172.21.2.102:0/1303804882 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6ccc08a460 0x7f6ccc1a1440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:58.368 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:58.287 7f6cd3d6b700 1 --2- 172.21.2.102:0/1303804882 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6cb4018730 0x7f6cb401ac10 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:58.368 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:58.287 7f6cd3d6b700 1 --2- 172.21.2.102:0/1303804882 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6ccc0969a0 0x7f6ccc096110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:58.368 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:58.287 7f6cd3d6b700 1 -- 172.21.2.102:0/1303804882 >> 172.21.2.102:0/1303804882 conn(0x7f6ccc1909e0 msgr2=0x7f6ccc0895a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:58.368 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:58.287 7f6cd3d6b700 1 -- 172.21.2.102:0/1303804882 shutdown_connections 2021-02-10T11:23:58.368 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:58.287 7f6cd3d6b700 1 -- 172.21.2.102:0/1303804882 wait complete. 2021-02-10T11:23:58.369 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:23:58.797 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:58.782 7f74fe606700 1 Processor -- start 2021-02-10T11:23:58.797 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:58.782 7f74fe606700 1 -- start start 2021-02-10T11:23:58.798 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:58.782 7f74fe606700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f74f80e5470 0x7f74f807e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:58.798 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:58.782 7f74fe606700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f74f807f300 0x7f74f807f700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:58.798 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:58.782 7f74fe606700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74f807ed80 con 0x7f74f80e5070 2021-02-10T11:23:58.798 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:58.782 7f74fe606700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74f81b4010 con 0x7f74f807f300 2021-02-10T11:23:58.798 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:58.782 7f74fe606700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74f80d1790 con 0x7f74f80e5470 2021-02-10T11:23:58.798 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:58.783 7f74fcc23700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f74f807f300 0x7f74f807f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:58.798 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:58.783 7f74f7fff700 1 --1- >> v1:172.21.2.113:6789/0 conn(0x7f74f80e5070 0x7f74f81b3820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.113:6789/0 says I am v1:172.21.2.102:56292/0 (socket says 172.21.2.102:56292) 2021-02-10T11:23:58.799 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:58.783 7f74f77fe700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f74f80e5470 0x7f74f807e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:58.799 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:58.783 7f74fcc23700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f74f807f300 0x7f74f807f700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.113:3300/0 says I am v2:172.21.2.102:60178/0 (socket says 172.21.2.102:60178) 2021-02-10T11:23:58.799 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:58.783 7f74f7fff700 1 -- 172.21.2.102:0/2880687818 learned_addr learned my addr 172.21.2.102:0/2880687818 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:23:58.799 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:58.784 7f74f77fe700 1 -- 172.21.2.102:0/2880687818 >> v2:172.21.2.113:3300/0 conn(0x7f74f807f300 msgr2=0x7f74f807f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:58.799 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:58.784 7f74f77fe700 1 --2- 172.21.2.102:0/2880687818 >> v2:172.21.2.113:3300/0 conn(0x7f74f807f300 0x7f74f807f700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:58.799 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:58.784 7f74f77fe700 1 -- 172.21.2.102:0/2880687818 >> v1:172.21.2.113:6789/0 conn(0x7f74f80e5070 legacy=0x7f74f81b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:58.799 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:58.784 7f74f77fe700 1 -- 172.21.2.102:0/2880687818 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74f80d1980 con 0x7f74f80e5470 2021-02-10T11:23:58.799 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:58.784 7f74f77fe700 1 --2- 172.21.2.102:0/2880687818 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f74f80e5470 0x7f74f807e6a0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7f74ec006650 tx=0x7f74ec009a80).ready entity=mon.2 client_cookie=3160f34e62883624 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:58.799 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:58.784 7f74f6ffd700 1 -- 172.21.2.102:0/2880687818 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f74ec00c0c0 con 0x7f74f80e5470 2021-02-10T11:23:58.799 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:58.784 7f74f6ffd700 1 -- 172.21.2.102:0/2880687818 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74ec00c340 con 0x7f74f80e5470 2021-02-10T11:23:58.799 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:58.784 7f74f6ffd700 1 -- 172.21.2.102:0/2880687818 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f74ec0009b0 con 0x7f74f80e5470 2021-02-10T11:23:58.799 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:58.784 7f74fe606700 1 -- 172.21.2.102:0/2880687818 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f74f80e5470 msgr2=0x7f74f807e6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:58.799 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:58.784 7f74fe606700 1 --2- 172.21.2.102:0/2880687818 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f74f80e5470 0x7f74f807e6a0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7f74ec006650 tx=0x7f74ec009a80).stop 2021-02-10T11:23:58.800 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:58.785 7f74fe606700 1 -- 172.21.2.102:0/2880687818 shutdown_connections 2021-02-10T11:23:58.800 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:58.785 7f74fe606700 1 --2- 172.21.2.102:0/2880687818 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f74f80e5470 0x7f74f807e6a0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:58.800 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:58.785 7f74fe606700 1 --2- 172.21.2.102:0/2880687818 >> v2:172.21.2.113:3300/0 conn(0x7f74f807f300 0x7f74f807f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:58.800 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:58.785 7f74fe606700 1 -- 172.21.2.102:0/2880687818 >> 172.21.2.102:0/2880687818 conn(0x7f74f80ce690 msgr2=0x7f74f80e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:58.800 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:58.785 7f74fe606700 1 -- 172.21.2.102:0/2880687818 shutdown_connections 2021-02-10T11:23:58.800 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:58.785 7f74fe606700 1 -- 172.21.2.102:0/2880687818 wait complete. 2021-02-10T11:23:58.802 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:58.799 7f74fe606700 1 Processor -- start 2021-02-10T11:23:58.802 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:58.799 7f74fe606700 1 -- start start 2021-02-10T11:23:58.802 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:58.800 7f74fe606700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f74f808d5a0 0x7f74f80ad560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:58.802 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:58.800 7f74fe606700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f74f80adb80 0x7f74f81a1b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:58.802 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:58.800 7f74fe606700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f74f81a21b0 0x7f74f81a25b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:58.802 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:58.800 7f74fe606700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74f81b2f70 con 0x7f74f80adb80 2021-02-10T11:23:58.803 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:58.800 7f74fe606700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74f808d9a0 con 0x7f74f808d5a0 2021-02-10T11:23:58.803 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:58.800 7f74fe606700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74f808dbc0 con 0x7f74f81a21b0 2021-02-10T11:23:58.803 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:58.800 7f74f77fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f74f80adb80 0x7f74f81a1b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:58.803 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:58.800 7f74f77fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f74f80adb80 0x7f74f81a1b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:56752/0 (socket says 172.21.2.102:56752) 2021-02-10T11:23:58.803 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:58.800 7f74f77fe700 1 -- 172.21.2.102:0/1232569678 learned_addr learned my addr 172.21.2.102:0/1232569678 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:23:58.803 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:58.800 7f74f77fe700 1 -- 172.21.2.102:0/1232569678 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f74f81a21b0 msgr2=0x7f74f81a25b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:23:58.803 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:58.800 7f74f77fe700 1 --2- 172.21.2.102:0/1232569678 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f74f81a21b0 0x7f74f81a25b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:58.803 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:58.800 7f74f77fe700 1 -- 172.21.2.102:0/1232569678 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f74f808d5a0 msgr2=0x7f74f80ad560 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:23:58.803 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:58.800 7f74f77fe700 1 --2- 172.21.2.102:0/1232569678 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f74f808d5a0 0x7f74f80ad560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:58.803 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:58.800 7f74f77fe700 1 -- 172.21.2.102:0/1232569678 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74f81a2bd0 con 0x7f74f80adb80 2021-02-10T11:23:58.803 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:58.800 7f74f77fe700 1 --2- 172.21.2.102:0/1232569678 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f74f80adb80 0x7f74f81a1b90 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 rx=0x7f74ec00b660 tx=0x7f74ec00ba00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:58.804 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:58.801 7f74f4ff9700 1 -- 172.21.2.102:0/1232569678 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f74ec0009b0 con 0x7f74f80adb80 2021-02-10T11:23:58.804 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:58.801 7f74f4ff9700 1 -- 172.21.2.102:0/1232569678 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74ec006760 con 0x7f74f80adb80 2021-02-10T11:23:58.804 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:58.801 7f74fe606700 1 -- 172.21.2.102:0/1232569678 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74f81a2bd0 con 0x7f74f80adb80 2021-02-10T11:23:58.804 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:58.801 7f74f4ff9700 1 -- 172.21.2.102:0/1232569678 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f74ec002030 con 0x7f74f80adb80 2021-02-10T11:23:58.804 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:58.801 7f74fe606700 1 -- 172.21.2.102:0/1232569678 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74f81a3960 con 0x7f74f80adb80 2021-02-10T11:23:58.804 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:58.801 7f74f4ff9700 1 -- 172.21.2.102:0/1232569678 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f74ec000c90 con 0x7f74f80adb80 2021-02-10T11:23:58.804 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:58.801 7f74f4ff9700 1 --2- 172.21.2.102:0/1232569678 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f74d8018700 0x7f74d801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:58.804 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:58.801 7f74f4ff9700 1 -- 172.21.2.102:0/1232569678 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f74ec0327f0 con 0x7f74f80adb80 2021-02-10T11:23:58.804 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:58.802 7f74f7fff700 1 --2- 172.21.2.102:0/1232569678 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f74d8018700 0x7f74d801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:58.804 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:58.802 7f74f7fff700 1 --2- 172.21.2.102:0/1232569678 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f74d8018700 0x7f74d801abc0 crc :-1 s=READY pgs=600 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:58.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:58.832 7f74fe606700 1 -- 172.21.2.102:0/1232569678 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74f8107170 con 0x7f74f80adb80 2021-02-10T11:23:58.833 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:58.835 7f74f4ff9700 1 -- 172.21.2.102:0/1232569678 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f74ec006740 con 0x7f74f80adb80 2021-02-10T11:23:58.961 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:58.962 7f74fe606700 1 -- 172.21.2.102:0/1232569678 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f74f807ed80 con 0x7f74f80adb80 2021-02-10T11:23:58.961 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:58.963 7f74f4ff9700 1 -- 172.21.2.102:0/1232569678 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) v1 ==== 94+0+2099 (secure 0 0 0) 0x7f74ec006740 con 0x7f74f80adb80 2021-02-10T11:23:58.962 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:23:58.962 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":31,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7557,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.102:6835/1941781958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6834","nonce":1941781958},{"type":"v1","addr":"172.21.2.102:6835","nonce":1941781958}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":26}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:23:18.002680","modified":"2021-02-10 11:23:20.009191","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5606},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5606":{"gid":5606,"name":"b","rank":0,"incarnation":27,"state":"up:active","state_seq":3,"addr":"172.21.2.113:6833/1872558991","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6832","nonce":1872558991},{"type":"v1","addr":"172.21.2.113:6833","nonce":1872558991}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":4}]} 2021-02-10T11:23:58.963 INFO:teuthology.orchestra.run.gibba002.stderr:dumped fsmap epoch 31 2021-02-10T11:23:58.963 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:58.964 7f74fe606700 1 -- 172.21.2.102:0/1232569678 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f74d8018700 msgr2=0x7f74d801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:58.964 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:58.964 7f74fe606700 1 --2- 172.21.2.102:0/1232569678 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f74d8018700 0x7f74d801abc0 crc :-1 s=READY pgs=600 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:58.964 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:58.964 7f74fe606700 1 -- 172.21.2.102:0/1232569678 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f74f80adb80 msgr2=0x7f74f81a1b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:58.964 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:58.964 7f74fe606700 1 --2- 172.21.2.102:0/1232569678 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f74f80adb80 0x7f74f81a1b90 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 rx=0x7f74ec00b660 tx=0x7f74ec00ba00).stop 2021-02-10T11:23:58.964 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:58.965 7f74fe606700 1 -- 172.21.2.102:0/1232569678 shutdown_connections 2021-02-10T11:23:58.964 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:58.965 7f74fe606700 1 --2- 172.21.2.102:0/1232569678 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f74f81a21b0 0x7f74f81a25b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:58.964 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:58.965 7f74fe606700 1 --2- 172.21.2.102:0/1232569678 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f74f80adb80 0x7f74f81a1b90 unknown :-1 s=CLOSED pgs=1273 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:58.964 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:58.965 7f74fe606700 1 --2- 172.21.2.102:0/1232569678 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f74d8018700 0x7f74d801abc0 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:58.964 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:58.965 7f74fe606700 1 --2- 172.21.2.102:0/1232569678 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f74f808d5a0 0x7f74f80ad560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:58.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:58.965 7f74fe606700 1 -- 172.21.2.102:0/1232569678 >> 172.21.2.102:0/1232569678 conn(0x7f74f81ac620 msgr2=0x7f74f80896e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:58.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:58.965 7f74fe606700 1 -- 172.21.2.102:0/1232569678 shutdown_connections 2021-02-10T11:23:58.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:58.965 7f74fe606700 1 -- 172.21.2.102:0/1232569678 wait complete. 2021-02-10T11:23:58.972 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs fail cephfs 2021-02-10T11:23:59.401 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:59.384 7f3261368700 1 Processor -- start 2021-02-10T11:23:59.402 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:59.384 7f3261368700 1 -- start start 2021-02-10T11:23:59.402 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:59.384 7f3261368700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f325c0e5070 0x7f325c1b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:59.402 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:59.384 7f3261368700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f325c07ed80 con 0x7f325c1b3e40 2021-02-10T11:23:59.402 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:59.384 7f3261368700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f325c07e670 con 0x7f325c0e5470 2021-02-10T11:23:59.402 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:59.384 7f3261368700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f325c07e940 con 0x7f325c0e5070 2021-02-10T11:23:59.402 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:59.384 7f325b87e700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f325c1b3e40 0x7f325c07de80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:48986/0 (socket says 172.21.2.102:48986) 2021-02-10T11:23:59.402 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:59.384 7f325b87e700 1 -- 172.21.2.102:0/3292932693 learned_addr learned my addr 172.21.2.102:0/3292932693 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:23:59.402 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:59.385 7f325a07b700 1 -- 172.21.2.102:0/3292932693 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2974679441 0 0) 0x7f3254000f90 con 0x7f325c1b3e40 2021-02-10T11:23:59.402 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:59.385 7f325a07b700 1 -- 172.21.2.102:0/3292932693 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3248002fe0 con 0x7f325c1b3e40 2021-02-10T11:23:59.402 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:59.385 7f325b07d700 1 --2- 172.21.2.102:0/3292932693 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f325c0e5070 0x7f325c1b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:59.402 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:59.385 7f325a07b700 1 -- 172.21.2.102:0/3292932693 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1593524518 0 0) 0x7f3254001460 con 0x7f325c1b3e40 2021-02-10T11:23:59.403 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:59.385 7f325a07b700 1 -- 172.21.2.102:0/3292932693 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f325c0e5070 msgr2=0x7f325c1b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:59.403 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:59.385 7f325a07b700 1 --2- 172.21.2.102:0/3292932693 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f325c0e5070 0x7f325c1b3820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:59.403 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:59.385 7f325a07b700 1 -- 172.21.2.102:0/3292932693 >> v1:172.21.2.113:6789/0 conn(0x7f325c0e5470 legacy=0x7f325c07d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:59.403 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:59.385 7f325a07b700 1 -- 172.21.2.102:0/3292932693 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f325c07eb30 con 0x7f325c1b3e40 2021-02-10T11:23:59.403 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:59.385 7f325a07b700 1 -- 172.21.2.102:0/3292932693 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7f3254001980 con 0x7f325c1b3e40 2021-02-10T11:23:59.403 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:59.385 7f325a07b700 1 -- 172.21.2.102:0/3292932693 >> v1:172.21.2.102:6789/0 conn(0x7f325c1b3e40 legacy=0x7f325c07de80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:59.403 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:59.385 7f325a07b700 1 --2- 172.21.2.102:0/3292932693 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3248003a60 0x7f3248005f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:59.403 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:59.385 7f325a07b700 1 --2- 172.21.2.102:0/3292932693 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3248006780 0x7f3248008c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:59.403 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:59.385 7f325a07b700 1 --2- 172.21.2.102:0/3292932693 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3248009510 0x7f324800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:59.403 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:59.385 7f325a07b700 1 -- 172.21.2.102:0/3292932693 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f324800c2c0 con 0x7f3248003a60 2021-02-10T11:23:59.403 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:59.385 7f325a07b700 1 -- 172.21.2.102:0/3292932693 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f324800c4e0 con 0x7f3248009510 2021-02-10T11:23:59.404 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:59.385 7f325a07b700 1 -- 172.21.2.102:0/3292932693 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f324800c700 con 0x7f3248006780 2021-02-10T11:23:59.404 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:59.385 7f325b07d700 1 --2- 172.21.2.102:0/3292932693 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3248003a60 0x7f3248005f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:59.404 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:59.385 7f325b87e700 1 --2- 172.21.2.102:0/3292932693 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3248009510 0x7f324800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:59.404 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:59.385 7f325b07d700 1 -- 172.21.2.102:0/3292932693 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3248006780 msgr2=0x7f3248008c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:59.404 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:59.385 7f325b07d700 1 --2- 172.21.2.102:0/3292932693 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3248006780 0x7f3248008c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:59.404 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:59.385 7f325b07d700 1 -- 172.21.2.102:0/3292932693 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3248009510 msgr2=0x7f324800ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:59.404 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:59.385 7f325b07d700 1 --2- 172.21.2.102:0/3292932693 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3248009510 0x7f324800ba10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:59.404 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:59.385 7f325b07d700 1 -- 172.21.2.102:0/3292932693 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f324800c9a0 con 0x7f3248003a60 2021-02-10T11:23:59.404 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:59.386 7f325b07d700 1 --2- 172.21.2.102:0/3292932693 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3248003a60 0x7f3248005f60 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 rx=0x7f3244006640 tx=0x7f324404da90).ready entity=mon.0 client_cookie=eac0d32487cc78fc server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:59.404 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:59.386 7f325a07b700 1 -- 172.21.2.102:0/3292932693 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f32440500c0 con 0x7f3248003a60 2021-02-10T11:23:59.404 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:59.386 7f325a07b700 1 -- 172.21.2.102:0/3292932693 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3244050340 con 0x7f3248003a60 2021-02-10T11:23:59.404 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:59.386 7f325a07b700 1 -- 172.21.2.102:0/3292932693 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3244050880 con 0x7f3248003a60 2021-02-10T11:23:59.404 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:59.386 7f3261368700 1 -- 172.21.2.102:0/3292932693 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3248003a60 msgr2=0x7f3248005f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:59.404 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:59.386 7f3261368700 1 --2- 172.21.2.102:0/3292932693 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3248003a60 0x7f3248005f60 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 rx=0x7f3244006640 tx=0x7f324404da90).stop 2021-02-10T11:23:59.405 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:59.386 7f325b87e700 1 -- 172.21.2.102:0/3292932693 reap_dead start 2021-02-10T11:23:59.405 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:59.386 7f3261368700 1 -- 172.21.2.102:0/3292932693 shutdown_connections 2021-02-10T11:23:59.405 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:59.386 7f3261368700 1 -- 172.21.2.102:0/3292932693 >> 172.21.2.102:0/3292932693 conn(0x7f325c0ce690 msgr2=0x7f325c0e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:59.405 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:59.386 7f3261368700 1 -- 172.21.2.102:0/3292932693 shutdown_connections 2021-02-10T11:23:59.405 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:59.386 7f3261368700 1 -- 172.21.2.102:0/3292932693 wait complete. 2021-02-10T11:23:59.405 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:59.401 7f3261368700 1 Processor -- start 2021-02-10T11:23:59.405 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:59.402 7f3261368700 1 -- start start 2021-02-10T11:23:59.405 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:59.402 7f3261368700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f325c0a6f60 0x7f325c0a6430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:59.405 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:59.402 7f3261368700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f325c0a6a50 0x7f325c08a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:59.405 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:59.402 7f3261368700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f325c08aa50 0x7f325c08b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:59.405 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:59.402 7f3261368700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f325c08af40 con 0x7f325c0a6a50 2021-02-10T11:23:59.405 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:59.402 7f3261368700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f325c08bae0 con 0x7f325c08aa50 2021-02-10T11:23:59.405 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:59.402 7f3261368700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f325c08bd00 con 0x7f325c0a6f60 2021-02-10T11:23:59.406 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:59.402 7f325a87c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f325c0a6a50 0x7f325c08a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:59.406 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:59.402 7f325a87c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f325c0a6a50 0x7f325c08a430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:56772/0 (socket says 172.21.2.102:56772) 2021-02-10T11:23:59.406 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:59.402 7f325a87c700 1 -- 172.21.2.102:0/2976331452 learned_addr learned my addr 172.21.2.102:0/2976331452 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:23:59.406 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:59.402 7f325b07d700 1 --2- 172.21.2.102:0/2976331452 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f325c0a6f60 0x7f325c0a6430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:59.406 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:59.402 7f325b87e700 1 --2- 172.21.2.102:0/2976331452 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f325c08aa50 0x7f325c08b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:59.406 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:59.403 7f325a87c700 1 -- 172.21.2.102:0/2976331452 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f325c0a6f60 msgr2=0x7f325c0a6430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:59.406 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:59.403 7f325a87c700 1 --2- 172.21.2.102:0/2976331452 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f325c0a6f60 0x7f325c0a6430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:59.406 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:59.403 7f325a87c700 1 -- 172.21.2.102:0/2976331452 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f325c08aa50 msgr2=0x7f325c08b4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:59.406 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:59.403 7f325a87c700 1 --2- 172.21.2.102:0/2976331452 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f325c08aa50 0x7f325c08b4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:59.406 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:59.403 7f325a87c700 1 -- 172.21.2.102:0/2976331452 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f325c08bf20 con 0x7f325c0a6a50 2021-02-10T11:23:59.406 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:59.403 7f325a87c700 1 --2- 172.21.2.102:0/2976331452 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f325c0a6a50 0x7f325c08a430 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 rx=0x7f324c006ed0 tx=0x7f324c00b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:59.406 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:59.403 7f3253fff700 1 -- 172.21.2.102:0/2976331452 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f324c006240 con 0x7f325c0a6a50 2021-02-10T11:23:59.406 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:59.403 7f3253fff700 1 -- 172.21.2.102:0/2976331452 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f324c0054b0 con 0x7f325c0a6a50 2021-02-10T11:23:59.407 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:59.403 7f3253fff700 1 -- 172.21.2.102:0/2976331452 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f324c005990 con 0x7f325c0a6a50 2021-02-10T11:23:59.407 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:59.403 7f3261368700 1 -- 172.21.2.102:0/2976331452 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f325c08bf20 con 0x7f325c0a6a50 2021-02-10T11:23:59.407 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:59.403 7f3261368700 1 -- 172.21.2.102:0/2976331452 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f325c08bf20 con 0x7f325c0a6a50 2021-02-10T11:23:59.407 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:59.404 7f3253fff700 1 -- 172.21.2.102:0/2976331452 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f324c006030 con 0x7f325c0a6a50 2021-02-10T11:23:59.407 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:59.404 7f3253fff700 1 --2- 172.21.2.102:0/2976331452 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f32400186c0 0x7f324001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:59.408 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:59.404 7f3253fff700 1 -- 172.21.2.102:0/2976331452 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f324c032640 con 0x7f325c0a6a50 2021-02-10T11:23:59.408 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:59.404 7f325b07d700 1 --2- 172.21.2.102:0/2976331452 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f32400186c0 0x7f324001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:59.408 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:59.404 7f325b07d700 1 --2- 172.21.2.102:0/2976331452 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f32400186c0 0x7f324001abc0 crc :-1 s=READY pgs=601 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:59.432 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:59.434 7f3261368700 1 -- 172.21.2.102:0/2976331452 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f325c084a90 con 0x7f325c0a6a50 2021-02-10T11:23:59.435 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:59.437 7f3253fff700 1 -- 172.21.2.102:0/2976331452 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f324c000de0 con 0x7f325c0a6a50 2021-02-10T11:23:59.565 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:23:59.567 7f3261368700 1 -- 172.21.2.102:0/2976331452 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f325c07ed80 con 0x7f325c0a6a50 2021-02-10T11:23:59.994 INFO:tasks.ceph.mon.a.gibba002.stderr:2021-02-10 11:23:59.996 7f8dd2755700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-02-10T11:24:00.000 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:00.002 7f3253fff700 1 -- 172.21.2.102:0/2976331452 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v32) v1 ==== 161+0+0 (secure 0 0 0) 0x7f324c01f500 con 0x7f325c0a6a50 2021-02-10T11:24:00.031 INFO:teuthology.orchestra.run.gibba002.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2021-02-10T11:24:00.033 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:00.035 7f3261368700 1 -- 172.21.2.102:0/2976331452 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f32400186c0 msgr2=0x7f324001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:00.033 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:00.035 7f3261368700 1 --2- 172.21.2.102:0/2976331452 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f32400186c0 0x7f324001abc0 crc :-1 s=READY pgs=601 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:00.033 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:00.035 7f3261368700 1 -- 172.21.2.102:0/2976331452 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f325c0a6a50 msgr2=0x7f325c08a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:00.033 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:00.035 7f3261368700 1 --2- 172.21.2.102:0/2976331452 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f325c0a6a50 0x7f325c08a430 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 rx=0x7f324c006ed0 tx=0x7f324c00b960).stop 2021-02-10T11:24:00.034 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:00.035 7f3261368700 1 -- 172.21.2.102:0/2976331452 shutdown_connections 2021-02-10T11:24:00.034 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:00.035 7f3261368700 1 --2- 172.21.2.102:0/2976331452 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f32400186c0 0x7f324001abc0 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:00.034 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:00.035 7f3261368700 1 --2- 172.21.2.102:0/2976331452 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f325c08aa50 0x7f325c08b4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:00.034 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:00.035 7f3261368700 1 --2- 172.21.2.102:0/2976331452 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f325c0a6a50 0x7f325c08a430 unknown :-1 s=CLOSED pgs=1276 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:00.034 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:00.035 7f3261368700 1 --2- 172.21.2.102:0/2976331452 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f325c0a6f60 0x7f325c0a6430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:00.034 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:00.035 7f3261368700 1 -- 172.21.2.102:0/2976331452 >> 172.21.2.102:0/2976331452 conn(0x7f325c0ba9d0 msgr2=0x7f325c088cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:00.034 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:00.035 7f3261368700 1 -- 172.21.2.102:0/2976331452 shutdown_connections 2021-02-10T11:24:00.034 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:00.035 7f3261368700 1 -- 172.21.2.102:0/2976331452 wait complete. 2021-02-10T11:24:00.042 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:24:00.052 INFO:tasks.ceph.mds.b.gibba013.stdout:starting mds.b at 2021-02-10T11:24:00.470 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:00.455 7fc142dd9700 1 Processor -- start 2021-02-10T11:24:00.471 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:00.455 7fc142dd9700 1 -- start start 2021-02-10T11:24:00.471 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:00.456 7fc142dd9700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fc13c0e5070 0x7fc13c1b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:00.471 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:00.456 7fc142dd9700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc13c0e5470 0x7fc13c080310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:00.471 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:00.456 7fc142dd9700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc13c07ed80 con 0x7fc13c1b3e40 2021-02-10T11:24:00.471 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:00.456 7fc142dd9700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc13c0d1790 con 0x7fc13c0e5070 2021-02-10T11:24:00.471 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:00.456 7fc142dd9700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc13c0d19b0 con 0x7fc13c0e5470 2021-02-10T11:24:00.471 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:00.456 7fc13bfff700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fc13c1b3e40 0x7fc13c07d690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:49002/0 (socket says 172.21.2.102:49002) 2021-02-10T11:24:00.471 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:00.456 7fc13bfff700 1 -- 172.21.2.102:0/777452705 learned_addr learned my addr 172.21.2.102:0/777452705 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:24:00.472 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:00.456 7fc140bf5700 1 --2- 172.21.2.102:0/777452705 >> v2:172.21.2.102:3300/0 conn(0x7fc13c0e5070 0x7fc13c1b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:00.472 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:00.456 7fc140bf5700 1 -- 172.21.2.102:0/777452705 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc13c0e5470 msgr2=0x7fc13c080310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:24:00.472 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:00.456 7fc140bf5700 1 --2- 172.21.2.102:0/777452705 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc13c0e5470 0x7fc13c080310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:00.472 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:00.456 7fc140bf5700 1 -- 172.21.2.102:0/777452705 >> v1:172.21.2.102:6789/0 conn(0x7fc13c1b3e40 legacy=0x7fc13c07d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:00.472 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:00.456 7fc13b7fe700 1 -- 172.21.2.102:0/777452705 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2632950677 0 0) 0x7fc12c001430 con 0x7fc13c1b3e40 2021-02-10T11:24:00.472 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:00.456 7fc140bf5700 1 -- 172.21.2.102:0/777452705 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc13c0d1bd0 con 0x7fc13c0e5070 2021-02-10T11:24:00.472 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:00.457 7fc140bf5700 1 --2- 172.21.2.102:0/777452705 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc13c0e5070 0x7fc13c1b3820 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 rx=0x7fc124006650 tx=0x7fc124009a00).ready entity=mon.0 client_cookie=fc3fd4879438dc32 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:00.472 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:00.457 7fc13b7fe700 1 -- 172.21.2.102:0/777452705 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc124009df0 con 0x7fc13c0e5070 2021-02-10T11:24:00.472 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:00.457 7fc13b7fe700 1 -- 172.21.2.102:0/777452705 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1240009b0 con 0x7fc13c0e5070 2021-02-10T11:24:00.472 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:00.457 7fc13b7fe700 1 -- 172.21.2.102:0/777452705 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc12400c2b0 con 0x7fc13c0e5070 2021-02-10T11:24:00.472 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:00.457 7fc142dd9700 1 -- 172.21.2.102:0/777452705 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc13c0e5070 msgr2=0x7fc13c1b3820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:00.472 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:00.457 7fc142dd9700 1 --2- 172.21.2.102:0/777452705 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc13c0e5070 0x7fc13c1b3820 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 rx=0x7fc124006650 tx=0x7fc124009a00).stop 2021-02-10T11:24:00.473 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:00.458 7fc142dd9700 1 -- 172.21.2.102:0/777452705 shutdown_connections 2021-02-10T11:24:00.473 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:00.458 7fc142dd9700 1 --2- 172.21.2.102:0/777452705 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc13c0e5470 0x7fc13c080310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:00.473 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:00.458 7fc142dd9700 1 --2- 172.21.2.102:0/777452705 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc13c0e5070 0x7fc13c1b3820 unknown :-1 s=CLOSED pgs=1280 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:00.473 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:00.458 7fc142dd9700 1 -- 172.21.2.102:0/777452705 >> 172.21.2.102:0/777452705 conn(0x7fc13c0ce690 msgr2=0x7fc13c0e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:00.473 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:00.458 7fc142dd9700 1 -- 172.21.2.102:0/777452705 shutdown_connections 2021-02-10T11:24:00.473 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:00.458 7fc142dd9700 1 -- 172.21.2.102:0/777452705 wait complete. 2021-02-10T11:24:00.473 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:00.472 7fc142dd9700 1 Processor -- start 2021-02-10T11:24:00.473 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:00.472 7fc142dd9700 1 -- start start 2021-02-10T11:24:00.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:00.473 7fc142dd9700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc13c0969a0 0x7fc13c096110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:00.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:00.473 7fc142dd9700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc13c08a460 0x7fc13c1a1440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:00.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:00.473 7fc142dd9700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc13c0b1440 0x7fc13c0b1840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:00.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:00.473 7fc142dd9700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc13c0b1e60 con 0x7fc13c08a460 2021-02-10T11:24:00.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:00.473 7fc142dd9700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc13c0b2080 con 0x7fc13c0b1440 2021-02-10T11:24:00.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:00.473 7fc142dd9700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc13c0b22a0 con 0x7fc13c0969a0 2021-02-10T11:24:00.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:00.473 7fc13bfff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc13c08a460 0x7fc13c1a1440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:00.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:00.473 7fc13bfff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc13c08a460 0x7fc13c1a1440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:56784/0 (socket says 172.21.2.102:56784) 2021-02-10T11:24:00.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:00.473 7fc13bfff700 1 -- 172.21.2.102:0/159881468 learned_addr learned my addr 172.21.2.102:0/159881468 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:24:00.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:00.474 7fc13bfff700 1 -- 172.21.2.102:0/159881468 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc13c0969a0 msgr2=0x7fc13c096110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:00.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:00.474 7fc13bfff700 1 --2- 172.21.2.102:0/159881468 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc13c0969a0 0x7fc13c096110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:00.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:00.474 7fc13bfff700 1 -- 172.21.2.102:0/159881468 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc13c0b1440 msgr2=0x7fc13c0b1840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:00.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:00.474 7fc13bfff700 1 --2- 172.21.2.102:0/159881468 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc13c0b1440 0x7fc13c0b1840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:00.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:00.474 7fc13bfff700 1 -- 172.21.2.102:0/159881468 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc13c08ae90 con 0x7fc13c08a460 2021-02-10T11:24:00.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:00.474 7fc13bfff700 1 --2- 172.21.2.102:0/159881468 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc13c08a460 0x7fc13c1a1440 secure :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 rx=0x7fc12c001fe0 tx=0x7fc12c00ac50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:00.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:00.474 7fc1397fa700 1 -- 172.21.2.102:0/159881468 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc12c0070d0 con 0x7fc13c08a460 2021-02-10T11:24:00.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:00.474 7fc142dd9700 1 -- 172.21.2.102:0/159881468 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc13c08ae90 con 0x7fc13c08a460 2021-02-10T11:24:00.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:00.474 7fc1397fa700 1 -- 172.21.2.102:0/159881468 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc12c0070d0 con 0x7fc13c08a460 2021-02-10T11:24:00.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:00.474 7fc1397fa700 1 -- 172.21.2.102:0/159881468 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc12c001620 con 0x7fc13c08a460 2021-02-10T11:24:00.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:00.474 7fc142dd9700 1 -- 172.21.2.102:0/159881468 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc13c08ae90 con 0x7fc13c08a460 2021-02-10T11:24:00.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:00.475 7fc1397fa700 1 -- 172.21.2.102:0/159881468 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fc12c001410 con 0x7fc13c08a460 2021-02-10T11:24:00.477 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:00.475 7fc1397fa700 1 --2- 172.21.2.102:0/159881468 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc1200186c0 0x7fc12001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:00.477 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:00.475 7fc1397fa700 1 -- 172.21.2.102:0/159881468 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5659+0+0 (secure 0 0 0) 0x7fc12c032b40 con 0x7fc13c08a460 2021-02-10T11:24:00.477 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:00.475 7fc140bf5700 1 --2- 172.21.2.102:0/159881468 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc1200186c0 0x7fc12001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:00.477 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:00.475 7fc140bf5700 1 --2- 172.21.2.102:0/159881468 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc1200186c0 0x7fc12001abc0 crc :-1 s=READY pgs=602 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:00.504 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:00.505 7fc142dd9700 1 -- 172.21.2.102:0/159881468 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc13c08a960 con 0x7fc13c08a460 2021-02-10T11:24:00.506 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:00.508 7fc1397fa700 1 -- 172.21.2.102:0/159881468 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fc12c000d00 con 0x7fc13c08a460 2021-02-10T11:24:00.633 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:00.635 7fc142dd9700 1 -- 172.21.2.102:0/159881468 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc13c08a960 con 0x7fc13c08a460 2021-02-10T11:24:00.634 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:00.635 7fc1397fa700 1 -- 172.21.2.102:0/159881468 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) v1 ==== 94+0+1754 (secure 0 0 0) 0x7fc12c0070b0 con 0x7fc13c08a460 2021-02-10T11:24:00.634 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:24:00.634 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":32,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7557,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.102:6835/1941781958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6834","nonce":1941781958},{"type":"v1","addr":"172.21.2.102:6835","nonce":1941781958}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":26}],"filesystems":[{"mdsmap":{"epoch":32,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:23:18.002680","modified":"2021-02-10 11:23:59.997216","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":4}]} 2021-02-10T11:24:00.634 INFO:teuthology.orchestra.run.gibba002.stderr:dumped fsmap epoch 32 2021-02-10T11:24:00.635 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:00.637 7fc142dd9700 1 -- 172.21.2.102:0/159881468 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc1200186c0 msgr2=0x7fc12001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:00.635 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:00.637 7fc142dd9700 1 --2- 172.21.2.102:0/159881468 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc1200186c0 0x7fc12001abc0 crc :-1 s=READY pgs=602 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:00.636 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:00.637 7fc142dd9700 1 -- 172.21.2.102:0/159881468 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc13c08a460 msgr2=0x7fc13c1a1440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:00.636 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:00.637 7fc142dd9700 1 --2- 172.21.2.102:0/159881468 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc13c08a460 0x7fc13c1a1440 secure :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 rx=0x7fc12c001fe0 tx=0x7fc12c00ac50).stop 2021-02-10T11:24:00.636 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:00.638 7fc142dd9700 1 -- 172.21.2.102:0/159881468 shutdown_connections 2021-02-10T11:24:00.636 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:00.638 7fc142dd9700 1 --2- 172.21.2.102:0/159881468 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc1200186c0 0x7fc12001abc0 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:00.636 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:00.638 7fc142dd9700 1 --2- 172.21.2.102:0/159881468 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc13c0b1440 0x7fc13c0b1840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:00.636 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:00.638 7fc142dd9700 1 --2- 172.21.2.102:0/159881468 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc13c08a460 0x7fc13c1a1440 unknown :-1 s=CLOSED pgs=1281 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:00.636 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:00.638 7fc142dd9700 1 --2- 172.21.2.102:0/159881468 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc13c0969a0 0x7fc13c096110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:00.637 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:00.638 7fc142dd9700 1 -- 172.21.2.102:0/159881468 >> 172.21.2.102:0/159881468 conn(0x7fc13c1909e0 msgr2=0x7fc13c0895a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:00.637 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:00.638 7fc142dd9700 1 -- 172.21.2.102:0/159881468 shutdown_connections 2021-02-10T11:24:00.637 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:00.638 7fc142dd9700 1 -- 172.21.2.102:0/159881468 wait complete. 2021-02-10T11:24:00.644 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2021-02-10T11:24:01.075 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:01.060 7f26b6518700 1 Processor -- start 2021-02-10T11:24:01.075 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:01.060 7f26b6518700 1 -- start start 2021-02-10T11:24:01.075 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:01.061 7f26b6518700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f26b00d17f0 0x7f26b01b31e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:01.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:01.061 7f26b6518700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f26b00d1bf0 0x7f26b01b3800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:01.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:01.061 7f26b6518700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f26b01b3e20 0x7f26b00ceef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:01.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:01.061 7f26b6518700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26b00e5eb0 con 0x7f26b00d1bf0 2021-02-10T11:24:01.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:01.061 7f26b6518700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26b00f0b20 con 0x7f26b01b3e20 2021-02-10T11:24:01.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:01.061 7f26b6518700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26b00cf510 con 0x7f26b00d17f0 2021-02-10T11:24:01.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:01.061 7f26af7fe700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f26b00d1bf0 0x7f26b01b3800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:01.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:01.061 7f26af7fe700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f26b00d1bf0 0x7f26b01b3800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:56788/0 (socket says 172.21.2.102:56788) 2021-02-10T11:24:01.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:01.061 7f26af7fe700 1 -- 172.21.2.102:0/3632895597 learned_addr learned my addr 172.21.2.102:0/3632895597 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:24:01.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:01.061 7f26af7fe700 1 -- 172.21.2.102:0/3632895597 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f26b00d17f0 msgr2=0x7f26b01b31e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:24:01.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:01.061 7f26af7fe700 1 --2- 172.21.2.102:0/3632895597 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f26b00d17f0 0x7f26b01b31e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:01.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:01.062 7f26af7fe700 1 -- 172.21.2.102:0/3632895597 >> v2:172.21.2.113:3300/0 conn(0x7f26b01b3e20 msgr2=0x7f26b00ceef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:24:01.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:01.062 7f26af7fe700 1 --2- 172.21.2.102:0/3632895597 >> v2:172.21.2.113:3300/0 conn(0x7f26b01b3e20 0x7f26b00ceef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:01.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:01.062 7f26af7fe700 1 -- 172.21.2.102:0/3632895597 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26b00cf730 con 0x7f26b00d1bf0 2021-02-10T11:24:01.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:01.062 7f26af7fe700 1 --2- 172.21.2.102:0/3632895597 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f26b00d1bf0 0x7f26b01b3800 secure :-1 s=READY pgs=1282 cs=0 l=1 rev1=1 rx=0x7f26a4006700 tx=0x7f26a4009b00).ready entity=mon.0 client_cookie=7f1d028116acc891 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:01.077 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:01.062 7f26aeffd700 1 -- 172.21.2.102:0/3632895597 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f26a400c0c0 con 0x7f26b00d1bf0 2021-02-10T11:24:01.077 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:01.062 7f26aeffd700 1 -- 172.21.2.102:0/3632895597 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26a400c400 con 0x7f26b00d1bf0 2021-02-10T11:24:01.077 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:01.062 7f26aeffd700 1 -- 172.21.2.102:0/3632895597 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f26a400c8e0 con 0x7f26b00d1bf0 2021-02-10T11:24:01.077 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:01.062 7f26b6518700 1 -- 172.21.2.102:0/3632895597 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f26b00d1bf0 msgr2=0x7f26b01b3800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:01.077 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:01.062 7f26b6518700 1 --2- 172.21.2.102:0/3632895597 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f26b00d1bf0 0x7f26b01b3800 secure :-1 s=READY pgs=1282 cs=0 l=1 rev1=1 rx=0x7f26a4006700 tx=0x7f26a4009b00).stop 2021-02-10T11:24:01.077 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:01.063 7f26b6518700 1 -- 172.21.2.102:0/3632895597 shutdown_connections 2021-02-10T11:24:01.077 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:01.063 7f26b6518700 1 --2- 172.21.2.102:0/3632895597 >> v2:172.21.2.113:3300/0 conn(0x7f26b01b3e20 0x7f26b00ceef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:01.077 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:01.063 7f26b6518700 1 --2- 172.21.2.102:0/3632895597 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f26b00d1bf0 0x7f26b01b3800 unknown :-1 s=CLOSED pgs=1282 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:01.077 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:01.063 7f26b6518700 1 --2- 172.21.2.102:0/3632895597 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f26b00d17f0 0x7f26b01b31e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:01.077 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:01.063 7f26b6518700 1 -- 172.21.2.102:0/3632895597 >> 172.21.2.102:0/3632895597 conn(0x7f26b00fde40 msgr2=0x7f26b00ce390 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:01.077 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:01.063 7f26b6518700 1 -- 172.21.2.102:0/3632895597 shutdown_connections 2021-02-10T11:24:01.077 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:01.063 7f26b6518700 1 -- 172.21.2.102:0/3632895597 wait complete. 2021-02-10T11:24:01.079 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:01.077 7f26b6518700 1 Processor -- start 2021-02-10T11:24:01.079 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:01.077 7f26b6518700 1 -- start start 2021-02-10T11:24:01.079 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:01.078 7f26b6518700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f26b01a0210 0x7f26b019edf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:01.079 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:01.078 7f26b6518700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f26b00ae9c0 0x7f26b019f410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:01.079 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:01.078 7f26b6518700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f26b019fa30 0x7f26b019d400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:01.079 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:01.078 7f26b6518700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26b019cdc0 con 0x7f26b00ae9c0 2021-02-10T11:24:01.079 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:01.078 7f26b6518700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26b019e870 con 0x7f26b01a0210 2021-02-10T11:24:01.079 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:01.078 7f26b6518700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26b0197cb0 con 0x7f26b019fa30 2021-02-10T11:24:01.080 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:01.078 7f26af7fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f26b00ae9c0 0x7f26b019f410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:01.080 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:01.078 7f26af7fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f26b00ae9c0 0x7f26b019f410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:56796/0 (socket says 172.21.2.102:56796) 2021-02-10T11:24:01.080 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:01.078 7f26af7fe700 1 -- 172.21.2.102:0/2206024077 learned_addr learned my addr 172.21.2.102:0/2206024077 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:24:01.080 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:01.078 7f26af7fe700 1 -- 172.21.2.102:0/2206024077 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f26b019fa30 msgr2=0x7f26b019d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:01.080 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:01.078 7f26af7fe700 1 --2- 172.21.2.102:0/2206024077 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f26b019fa30 0x7f26b019d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:01.080 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:01.078 7f26af7fe700 1 -- 172.21.2.102:0/2206024077 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f26b01a0210 msgr2=0x7f26b019edf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:01.080 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:01.078 7f26af7fe700 1 --2- 172.21.2.102:0/2206024077 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f26b01a0210 0x7f26b019edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:01.080 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:01.078 7f26af7fe700 1 -- 172.21.2.102:0/2206024077 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26b019cfd0 con 0x7f26b00ae9c0 2021-02-10T11:24:01.080 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:01.079 7f26af7fe700 1 --2- 172.21.2.102:0/2206024077 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f26b00ae9c0 0x7f26b019f410 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 rx=0x7f26a400b660 tx=0x7f26a400ba00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:01.080 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:01.079 7f26acff9700 1 -- 172.21.2.102:0/2206024077 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f26a4009440 con 0x7f26b00ae9c0 2021-02-10T11:24:01.080 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:01.079 7f26acff9700 1 -- 172.21.2.102:0/2206024077 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26a4000c90 con 0x7f26b00ae9c0 2021-02-10T11:24:01.080 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:01.079 7f26acff9700 1 -- 172.21.2.102:0/2206024077 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f26a4002030 con 0x7f26b00ae9c0 2021-02-10T11:24:01.080 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:01.079 7f26b6518700 1 -- 172.21.2.102:0/2206024077 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26b019cfd0 con 0x7f26b00ae9c0 2021-02-10T11:24:01.081 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:01.079 7f26b6518700 1 -- 172.21.2.102:0/2206024077 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26b019dd10 con 0x7f26b00ae9c0 2021-02-10T11:24:01.081 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:01.079 7f26acff9700 1 -- 172.21.2.102:0/2206024077 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f26a40093e0 con 0x7f26b00ae9c0 2021-02-10T11:24:01.081 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:01.080 7f26acff9700 1 --2- 172.21.2.102:0/2206024077 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f26900186c0 0x7f269001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:01.081 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:01.080 7f26acff9700 1 -- 172.21.2.102:0/2206024077 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5659+0+0 (secure 0 0 0) 0x7f26a4032640 con 0x7f26b00ae9c0 2021-02-10T11:24:01.081 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:01.080 7f26affff700 1 --2- 172.21.2.102:0/2206024077 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f26900186c0 0x7f269001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:01.081 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:01.080 7f26affff700 1 --2- 172.21.2.102:0/2206024077 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f26900186c0 0x7f269001abc0 crc :-1 s=READY pgs=603 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:01.109 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:01.111 7f26b6518700 1 -- 172.21.2.102:0/2206024077 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26b0103700 con 0x7f26b00ae9c0 2021-02-10T11:24:01.112 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:01.114 7f26acff9700 1 -- 172.21.2.102:0/2206024077 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f26a4005770 con 0x7f26b00ae9c0 2021-02-10T11:24:01.240 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:01.241 7f26b6518700 1 -- 172.21.2.102:0/2206024077 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f26b00f0b20 con 0x7f26b00ae9c0 2021-02-10T11:24:02.007 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:02.008 7f26acff9700 1 -- 172.21.2.102:0/2206024077 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v33) v1 ==== 104+0+0 (secure 0 0 0) 0x7f26a4005770 con 0x7f26b00ae9c0 2021-02-10T11:24:02.011 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:02.012 7f26b6518700 1 -- 172.21.2.102:0/2206024077 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f26900186c0 msgr2=0x7f269001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:02.011 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:02.012 7f26b6518700 1 --2- 172.21.2.102:0/2206024077 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f26900186c0 0x7f269001abc0 crc :-1 s=READY pgs=603 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:02.011 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:02.012 7f26b6518700 1 -- 172.21.2.102:0/2206024077 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f26b00ae9c0 msgr2=0x7f26b019f410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:02.011 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:02.012 7f26b6518700 1 --2- 172.21.2.102:0/2206024077 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f26b00ae9c0 0x7f26b019f410 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 rx=0x7f26a400b660 tx=0x7f26a400ba00).stop 2021-02-10T11:24:02.011 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:02.012 7f26b6518700 1 -- 172.21.2.102:0/2206024077 shutdown_connections 2021-02-10T11:24:02.011 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:02.012 7f26b6518700 1 --2- 172.21.2.102:0/2206024077 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f26b019fa30 0x7f26b019d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:02.011 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:02.012 7f26b6518700 1 --2- 172.21.2.102:0/2206024077 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f26b00ae9c0 0x7f26b019f410 unknown :-1 s=CLOSED pgs=1283 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:02.011 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:02.012 7f26b6518700 1 --2- 172.21.2.102:0/2206024077 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f26900186c0 0x7f269001abc0 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:02.011 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:02.012 7f26b6518700 1 --2- 172.21.2.102:0/2206024077 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f26b01a0210 0x7f26b019edf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:02.011 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:02.012 7f26b6518700 1 -- 172.21.2.102:0/2206024077 >> 172.21.2.102:0/2206024077 conn(0x7f26b0197fe0 msgr2=0x7f26b0096c20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:02.011 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:02.012 7f26b6518700 1 -- 172.21.2.102:0/2206024077 shutdown_connections 2021-02-10T11:24:02.012 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:02.012 7f26b6518700 1 -- 172.21.2.102:0/2206024077 wait complete. 2021-02-10T11:24:02.019 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool delete cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2021-02-10T11:24:02.451 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:02.435 7fa8dcf61700 1 Processor -- start 2021-02-10T11:24:02.451 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:02.435 7fa8dcf61700 1 -- start start 2021-02-10T11:24:02.451 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:02.435 7fa8dcf61700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fa8d80d1870 0x7fa8d81b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:02.451 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:02.435 7fa8dcf61700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fa8d80cff70 0x7fa8d8109130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:02.451 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:02.436 7fa8dcf61700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8d80fe1c0 con 0x7fa8d81b0ae0 2021-02-10T11:24:02.451 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:02.436 7fa8dcf61700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8d8109ad0 con 0x7fa8d80d1870 2021-02-10T11:24:02.451 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:02.436 7fa8dcf61700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8d8109750 con 0x7fa8d80cff70 2021-02-10T11:24:02.452 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:02.436 7fa8d6b13700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fa8d80d1870 0x7fa8d81b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:02.452 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:02.436 7fa8d6b13700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fa8d80d1870 0x7fa8d81b04c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:56800/0 (socket says 172.21.2.102:56800) 2021-02-10T11:24:02.452 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:02.436 7fa8d6b13700 1 -- 172.21.2.102:0/1006850806 learned_addr learned my addr 172.21.2.102:0/1006850806 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:24:02.452 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:02.436 7fa8d7314700 1 --2- 172.21.2.102:0/1006850806 >> v2:172.21.2.113:3300/0 conn(0x7fa8d80cff70 0x7fa8d8109130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:02.452 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:02.436 7fa8d6b13700 1 -- 172.21.2.102:0/1006850806 >> v2:172.21.2.113:3300/0 conn(0x7fa8d80cff70 msgr2=0x7fa8d8109130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:24:02.452 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:02.436 7fa8d6b13700 1 --2- 172.21.2.102:0/1006850806 >> v2:172.21.2.113:3300/0 conn(0x7fa8d80cff70 0x7fa8d8109130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:02.452 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:02.436 7fa8d6b13700 1 -- 172.21.2.102:0/1006850806 >> v1:172.21.2.113:6789/0 conn(0x7fa8d81b0ae0 legacy=0x7fa8d8108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:02.452 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:02.436 7fa8d6b13700 1 -- 172.21.2.102:0/1006850806 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8d8088020 con 0x7fa8d80d1870 2021-02-10T11:24:02.453 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:02.436 7fa8d6b13700 1 --2- 172.21.2.102:0/1006850806 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa8d80d1870 0x7fa8d81b04c0 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 rx=0x7fa8c0006700 tx=0x7fa8c0009b80).ready entity=mon.0 client_cookie=1b243079dd44cec2 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:02.453 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:02.437 7fa8d5b11700 1 -- 172.21.2.102:0/1006850806 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa8c000c0c0 con 0x7fa8d80d1870 2021-02-10T11:24:02.453 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:02.437 7fa8d5b11700 1 -- 172.21.2.102:0/1006850806 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8c000c440 con 0x7fa8d80d1870 2021-02-10T11:24:02.453 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:02.437 7fa8d5b11700 1 -- 172.21.2.102:0/1006850806 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa8c000c920 con 0x7fa8d80d1870 2021-02-10T11:24:02.453 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:02.437 7fa8dcf61700 1 -- 172.21.2.102:0/1006850806 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa8d80d1870 msgr2=0x7fa8d81b04c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:02.453 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:02.437 7fa8dcf61700 1 --2- 172.21.2.102:0/1006850806 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa8d80d1870 0x7fa8d81b04c0 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 rx=0x7fa8c0006700 tx=0x7fa8c0009b80).stop 2021-02-10T11:24:02.454 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:02.437 7fa8dcf61700 1 -- 172.21.2.102:0/1006850806 shutdown_connections 2021-02-10T11:24:02.454 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:02.437 7fa8dcf61700 1 --2- 172.21.2.102:0/1006850806 >> v2:172.21.2.113:3300/0 conn(0x7fa8d80cff70 0x7fa8d8109130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:02.454 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:02.437 7fa8dcf61700 1 --2- 172.21.2.102:0/1006850806 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa8d80d1870 0x7fa8d81b04c0 unknown :-1 s=CLOSED pgs=1284 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:02.454 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:02.437 7fa8dcf61700 1 -- 172.21.2.102:0/1006850806 >> 172.21.2.102:0/1006850806 conn(0x7fa8d80ce420 msgr2=0x7fa8d80d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:02.454 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:02.437 7fa8dcf61700 1 -- 172.21.2.102:0/1006850806 shutdown_connections 2021-02-10T11:24:02.454 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:02.438 7fa8dcf61700 1 -- 172.21.2.102:0/1006850806 wait complete. 2021-02-10T11:24:02.454 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:02.452 7fa8dcf61700 1 Processor -- start 2021-02-10T11:24:02.454 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:02.452 7fa8dcf61700 1 -- start start 2021-02-10T11:24:02.454 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:02.452 7fa8dcf61700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa8d81b92d0 0x7fa8d819b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:02.455 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:02.453 7fa8dcf61700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa8d81a1c70 0x7fa8d80aa400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:02.455 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:02.453 7fa8dcf61700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa8d80aaa20 0x7fa8d819c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:02.455 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:02.453 7fa8dcf61700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8d819c240 con 0x7fa8d81b92d0 2021-02-10T11:24:02.455 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:02.453 7fa8dcf61700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8d819bf10 con 0x7fa8d80aaa20 2021-02-10T11:24:02.455 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:02.453 7fa8dcf61700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8d81a57c0 con 0x7fa8d81a1c70 2021-02-10T11:24:02.455 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:02.453 7fa8d6b13700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa8d81b92d0 0x7fa8d819b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:02.456 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:02.453 7fa8d6b13700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa8d81b92d0 0x7fa8d819b8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:56804/0 (socket says 172.21.2.102:56804) 2021-02-10T11:24:02.456 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:02.453 7fa8d6b13700 1 -- 172.21.2.102:0/2351261387 learned_addr learned my addr 172.21.2.102:0/2351261387 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:24:02.457 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:02.453 7fa8d6312700 1 --2- 172.21.2.102:0/2351261387 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa8d81a1c70 0x7fa8d80aa400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:02.457 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:02.453 7fa8d7314700 1 --2- 172.21.2.102:0/2351261387 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa8d80aaa20 0x7fa8d819c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:02.457 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:02.453 7fa8d6b13700 1 -- 172.21.2.102:0/2351261387 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa8d81a1c70 msgr2=0x7fa8d80aa400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:02.457 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:02.453 7fa8d6b13700 1 --2- 172.21.2.102:0/2351261387 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa8d81a1c70 0x7fa8d80aa400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:02.457 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:02.453 7fa8d6b13700 1 -- 172.21.2.102:0/2351261387 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa8d80aaa20 msgr2=0x7fa8d819c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:02.457 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:02.453 7fa8d6b13700 1 --2- 172.21.2.102:0/2351261387 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa8d80aaa20 0x7fa8d819c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:02.457 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:02.453 7fa8d6b13700 1 -- 172.21.2.102:0/2351261387 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8d819cde0 con 0x7fa8d81b92d0 2021-02-10T11:24:02.457 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:02.454 7fa8d6b13700 1 --2- 172.21.2.102:0/2351261387 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa8d81b92d0 0x7fa8d819b8f0 secure :-1 s=READY pgs=1285 cs=0 l=1 rev1=1 rx=0x7fa8c000b5e0 tx=0x7fa8c000b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:02.457 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:02.454 7fa8cf7fe700 1 -- 172.21.2.102:0/2351261387 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa8c000bd70 con 0x7fa8d81b92d0 2021-02-10T11:24:02.457 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:02.454 7fa8cf7fe700 1 -- 172.21.2.102:0/2351261387 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8c0002030 con 0x7fa8d81b92d0 2021-02-10T11:24:02.458 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:02.454 7fa8cf7fe700 1 -- 172.21.2.102:0/2351261387 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa8c0002550 con 0x7fa8d81b92d0 2021-02-10T11:24:02.458 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:02.454 7fa8dcf61700 1 -- 172.21.2.102:0/2351261387 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8d819cde0 con 0x7fa8d81b92d0 2021-02-10T11:24:02.458 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:02.454 7fa8dcf61700 1 -- 172.21.2.102:0/2351261387 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8d819d370 con 0x7fa8d81b92d0 2021-02-10T11:24:02.458 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:02.455 7fa8cf7fe700 1 -- 172.21.2.102:0/2351261387 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa8c0000c90 con 0x7fa8d81b92d0 2021-02-10T11:24:02.458 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:02.455 7fa8cf7fe700 1 --2- 172.21.2.102:0/2351261387 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa8bc0186c0 0x7fa8bc01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:02.458 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:02.455 7fa8cf7fe700 1 -- 172.21.2.102:0/2351261387 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5659+0+0 (secure 0 0 0) 0x7fa8c0032640 con 0x7fa8d81b92d0 2021-02-10T11:24:02.458 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:02.455 7fa8d6312700 1 --2- 172.21.2.102:0/2351261387 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa8bc0186c0 0x7fa8bc01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:02.458 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:02.455 7fa8d6312700 1 --2- 172.21.2.102:0/2351261387 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa8bc0186c0 0x7fa8bc01abc0 crc :-1 s=READY pgs=604 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:02.483 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:02.485 7fa8dcf61700 1 -- 172.21.2.102:0/2351261387 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8d819dc00 con 0x7fa8d81b92d0 2021-02-10T11:24:02.486 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:02.487 7fa8cf7fe700 1 -- 172.21.2.102:0/2351261387 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fa8c00009b0 con 0x7fa8d81b92d0 2021-02-10T11:24:02.614 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:02.616 7fa8dcf61700 1 -- 172.21.2.102:0/2351261387 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"pool2": "cephfs_metadata", "prefix": "osd pool delete", "pool": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fa8d819dc00 con 0x7fa8d81b92d0 2021-02-10T11:24:03.011 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:03.012 7fa8cf7fe700 1 -- 172.21.2.102:0/2351261387 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"pool2": "cephfs_metadata", "prefix": "osd pool delete", "pool": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v50) v1 ==== 185+0+0 (secure 0 0 0) 0x7fa8c00021f0 con 0x7fa8d81b92d0 2021-02-10T11:24:03.032 INFO:teuthology.orchestra.run.gibba002.stderr:pool 'cephfs_metadata' removed 2021-02-10T11:24:03.034 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:03.035 7fa8dcf61700 1 -- 172.21.2.102:0/2351261387 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa8bc0186c0 msgr2=0x7fa8bc01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:03.034 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:03.035 7fa8dcf61700 1 --2- 172.21.2.102:0/2351261387 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa8bc0186c0 0x7fa8bc01abc0 crc :-1 s=READY pgs=604 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:03.034 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:03.035 7fa8dcf61700 1 -- 172.21.2.102:0/2351261387 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa8d81b92d0 msgr2=0x7fa8d819b8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:03.034 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:03.035 7fa8dcf61700 1 --2- 172.21.2.102:0/2351261387 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa8d81b92d0 0x7fa8d819b8f0 secure :-1 s=READY pgs=1285 cs=0 l=1 rev1=1 rx=0x7fa8c000b5e0 tx=0x7fa8c000b980).stop 2021-02-10T11:24:03.034 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:03.035 7fa8dcf61700 1 -- 172.21.2.102:0/2351261387 shutdown_connections 2021-02-10T11:24:03.034 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:03.035 7fa8dcf61700 1 --2- 172.21.2.102:0/2351261387 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa8bc0186c0 0x7fa8bc01abc0 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:03.034 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:03.035 7fa8dcf61700 1 --2- 172.21.2.102:0/2351261387 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa8d80aaa20 0x7fa8d819c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:03.034 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:03.035 7fa8dcf61700 1 --2- 172.21.2.102:0/2351261387 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa8d81a1c70 0x7fa8d80aa400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:03.034 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:03.035 7fa8dcf61700 1 --2- 172.21.2.102:0/2351261387 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa8d81b92d0 0x7fa8d819b8f0 unknown :-1 s=CLOSED pgs=1285 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:03.034 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:03.035 7fa8dcf61700 1 -- 172.21.2.102:0/2351261387 >> 172.21.2.102:0/2351261387 conn(0x7fa8d819f840 msgr2=0x7fa8d819a940 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:03.035 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:03.036 7fa8dcf61700 1 -- 172.21.2.102:0/2351261387 shutdown_connections 2021-02-10T11:24:03.035 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:03.036 7fa8dcf61700 1 -- 172.21.2.102:0/2351261387 wait complete. 2021-02-10T11:24:03.042 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool delete cephfs_data cephfs_data --yes-i-really-really-mean-it 2021-02-10T11:24:03.474 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:03.459 7f352b312700 1 Processor -- start 2021-02-10T11:24:03.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:03.459 7f352b312700 1 -- start start 2021-02-10T11:24:03.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:03.460 7f352b312700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f35240d1870 0x7f35241b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:03.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:03.460 7f352b312700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f35240cff70 0x7f3524109130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:03.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:03.460 7f352b312700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3524109a60 con 0x7f35241b0ae0 2021-02-10T11:24:03.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:03.460 7f352b312700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3524109750 con 0x7f35240d1870 2021-02-10T11:24:03.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:03.460 7f352b312700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3524088020 con 0x7f35240cff70 2021-02-10T11:24:03.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:03.460 7f352892d700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f35241b0ae0 0x7f3524108940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:49036/0 (socket says 172.21.2.102:49036) 2021-02-10T11:24:03.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:03.460 7f352892d700 1 -- 172.21.2.102:0/2865306214 learned_addr learned my addr 172.21.2.102:0/2865306214 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:24:03.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:03.460 7f352912e700 1 --2- 172.21.2.102:0/2865306214 >> v2:172.21.2.113:3300/0 conn(0x7f35240d1870 0x7f35241b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:03.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:03.460 7f352992f700 1 --2- 172.21.2.102:0/2865306214 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f35240cff70 0x7f3524109130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:03.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:03.460 7f351bfff700 1 -- 172.21.2.102:0/2865306214 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1414894378 0 0) 0x7f35140009b0 con 0x7f35241b0ae0 2021-02-10T11:24:03.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:03.460 7f351bfff700 1 -- 172.21.2.102:0/2865306214 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3510002fe0 con 0x7f35241b0ae0 2021-02-10T11:24:03.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:03.461 7f352912e700 1 -- 172.21.2.102:0/2865306214 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f35240cff70 msgr2=0x7f3524109130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:24:03.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:03.461 7f352912e700 1 --2- 172.21.2.102:0/2865306214 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f35240cff70 0x7f3524109130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:03.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:03.461 7f352912e700 1 -- 172.21.2.102:0/2865306214 >> v1:172.21.2.102:6789/0 conn(0x7f35241b0ae0 legacy=0x7f3524108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:03.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:03.461 7f352912e700 1 -- 172.21.2.102:0/2865306214 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35240d1c70 con 0x7f35240d1870 2021-02-10T11:24:03.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:03.461 7f352992f700 1 --2- 172.21.2.102:0/2865306214 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f35240cff70 0x7f3524109130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:24:03.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:03.461 7f352912e700 1 --2- 172.21.2.102:0/2865306214 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f35240d1870 0x7f35241b04c0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 rx=0x7f3520006650 tx=0x7f3520009a80).ready entity=mon.1 client_cookie=98e69525b0fbf75e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:03.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:03.461 7f351bfff700 1 -- 172.21.2.102:0/2865306214 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f352000c0c0 con 0x7f35240d1870 2021-02-10T11:24:03.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:03.461 7f351bfff700 1 -- 172.21.2.102:0/2865306214 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f352000c340 con 0x7f35240d1870 2021-02-10T11:24:03.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:03.461 7f352b312700 1 -- 172.21.2.102:0/2865306214 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f35240d1870 msgr2=0x7f35241b04c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:03.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:03.461 7f352b312700 1 --2- 172.21.2.102:0/2865306214 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f35240d1870 0x7f35241b04c0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 rx=0x7f3520006650 tx=0x7f3520009a80).stop 2021-02-10T11:24:03.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:03.461 7f351bfff700 1 -- 172.21.2.102:0/2865306214 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f35200009b0 con 0x7f35240d1870 2021-02-10T11:24:03.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:03.462 7f352b312700 1 -- 172.21.2.102:0/2865306214 shutdown_connections 2021-02-10T11:24:03.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:03.462 7f352b312700 1 --2- 172.21.2.102:0/2865306214 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f35240cff70 0x7f3524109130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:03.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:03.462 7f352b312700 1 --2- 172.21.2.102:0/2865306214 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f35240d1870 0x7f35241b04c0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:03.477 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:03.462 7f352b312700 1 -- 172.21.2.102:0/2865306214 >> 172.21.2.102:0/2865306214 conn(0x7f35240ce420 msgr2=0x7f35240d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:03.477 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:03.462 7f352b312700 1 -- 172.21.2.102:0/2865306214 shutdown_connections 2021-02-10T11:24:03.477 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:03.462 7f352b312700 1 -- 172.21.2.102:0/2865306214 wait complete. 2021-02-10T11:24:03.478 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:03.477 7f352b312700 1 Processor -- start 2021-02-10T11:24:03.478 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:03.477 7f352b312700 1 -- start start 2021-02-10T11:24:03.478 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:03.477 7f352b312700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f35240b1670 0x7f35240b1a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:03.478 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:03.477 7f352b312700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f35240b2090 0x7f352408c710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:03.478 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:03.477 7f352b312700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f352408cd30 0x7f352408d130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:03.478 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:03.477 7f352b312700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f352419b870 con 0x7f35240b1670 2021-02-10T11:24:03.478 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:03.477 7f352b312700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f352419b400 con 0x7f35240b2090 2021-02-10T11:24:03.479 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:03.477 7f352b312700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35240b2490 con 0x7f352408cd30 2021-02-10T11:24:03.479 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:03.477 7f352912e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f35240b1670 0x7f35240b1a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:03.479 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:03.477 7f352912e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f35240b1670 0x7f35240b1a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:56818/0 (socket says 172.21.2.102:56818) 2021-02-10T11:24:03.479 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:03.477 7f352912e700 1 -- 172.21.2.102:0/3661979924 learned_addr learned my addr 172.21.2.102:0/3661979924 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:24:03.479 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:03.477 7f352912e700 1 -- 172.21.2.102:0/3661979924 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f352408cd30 msgr2=0x7f352408d130 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:24:03.479 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:03.477 7f352912e700 1 --2- 172.21.2.102:0/3661979924 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f352408cd30 0x7f352408d130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:03.479 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:03.477 7f352912e700 1 -- 172.21.2.102:0/3661979924 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f35240b2090 msgr2=0x7f352408c710 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:24:03.479 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:03.477 7f352912e700 1 --2- 172.21.2.102:0/3661979924 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f35240b2090 0x7f352408c710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:03.479 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:03.477 7f352912e700 1 -- 172.21.2.102:0/3661979924 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f352408d750 con 0x7f35240b1670 2021-02-10T11:24:03.479 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:03.478 7f352912e700 1 --2- 172.21.2.102:0/3661979924 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f35240b1670 0x7f35240b1a70 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 rx=0x7f352000b660 tx=0x7f352000ba00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:03.479 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:03.478 7f3519ffb700 1 -- 172.21.2.102:0/3661979924 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f352000bdf0 con 0x7f35240b1670 2021-02-10T11:24:03.479 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:03.478 7f3519ffb700 1 -- 172.21.2.102:0/3661979924 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35200009b0 con 0x7f35240b1670 2021-02-10T11:24:03.479 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:03.478 7f352b312700 1 -- 172.21.2.102:0/3661979924 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f352408d750 con 0x7f35240b1670 2021-02-10T11:24:03.480 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:03.478 7f352b312700 1 -- 172.21.2.102:0/3661979924 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f352408d750 con 0x7f35240b1670 2021-02-10T11:24:03.480 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:03.478 7f3519ffb700 1 -- 172.21.2.102:0/3661979924 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3520002030 con 0x7f35240b1670 2021-02-10T11:24:03.480 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:03.478 7f3519ffb700 1 -- 172.21.2.102:0/3661979924 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f35200052e0 con 0x7f35240b1670 2021-02-10T11:24:03.480 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:03.478 7f3519ffb700 1 --2- 172.21.2.102:0/3661979924 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3504018730 0x7f350401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:03.480 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:03.478 7f3519ffb700 1 -- 172.21.2.102:0/3661979924 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5221+0+0 (secure 0 0 0) 0x7f35200324c0 con 0x7f35240b1670 2021-02-10T11:24:03.480 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:03.478 7f352892d700 1 --2- 172.21.2.102:0/3661979924 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3504018730 0x7f350401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:03.480 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:03.479 7f352892d700 1 --2- 172.21.2.102:0/3661979924 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3504018730 0x7f350401ac10 crc :-1 s=READY pgs=605 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:03.507 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:03.509 7f352b312700 1 -- 172.21.2.102:0/3661979924 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35240828e0 con 0x7f35240b1670 2021-02-10T11:24:03.510 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:03.512 7f3519ffb700 1 -- 172.21.2.102:0/3661979924 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f3520000940 con 0x7f35240b1670 2021-02-10T11:24:03.579 DEBUG:teuthology.orchestra.run.gibba002:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:24:03.582 DEBUG:teuthology.orchestra.run.gibba013:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:24:03.641 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:03.643 7f352b312700 1 -- 172.21.2.102:0/3661979924 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"pool2": "cephfs_data", "prefix": "osd pool delete", "pool": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f35240828e0 con 0x7f35240b1670 2021-02-10T11:24:04.016 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:04.017 7f3519ffb700 1 -- 172.21.2.102:0/3661979924 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"pool2": "cephfs_data", "prefix": "osd pool delete", "pool": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v51) v1 ==== 173+0+0 (secure 0 0 0) 0x7f35200052e0 con 0x7f35240b1670 2021-02-10T11:24:04.057 INFO:teuthology.orchestra.run.gibba002.stderr:pool 'cephfs_data' removed 2021-02-10T11:24:04.059 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:04.061 7f352b312700 1 -- 172.21.2.102:0/3661979924 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3504018730 msgr2=0x7f350401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:04.060 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:04.061 7f352b312700 1 --2- 172.21.2.102:0/3661979924 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3504018730 0x7f350401ac10 crc :-1 s=READY pgs=605 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:04.060 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:04.061 7f352b312700 1 -- 172.21.2.102:0/3661979924 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f35240b1670 msgr2=0x7f35240b1a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:04.060 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:04.061 7f352b312700 1 --2- 172.21.2.102:0/3661979924 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f35240b1670 0x7f35240b1a70 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 rx=0x7f352000b660 tx=0x7f352000ba00).stop 2021-02-10T11:24:04.060 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:04.061 7f352b312700 1 -- 172.21.2.102:0/3661979924 shutdown_connections 2021-02-10T11:24:04.060 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:04.061 7f352b312700 1 --2- 172.21.2.102:0/3661979924 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f352408cd30 0x7f352408d130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:04.060 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:04.061 7f352b312700 1 --2- 172.21.2.102:0/3661979924 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3504018730 0x7f350401ac10 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:04.060 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:04.061 7f352b312700 1 --2- 172.21.2.102:0/3661979924 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f35240b2090 0x7f352408c710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:04.060 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:04.061 7f352b312700 1 --2- 172.21.2.102:0/3661979924 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f35240b1670 0x7f35240b1a70 unknown :-1 s=CLOSED pgs=1287 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:04.060 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:04.061 7f352b312700 1 -- 172.21.2.102:0/3661979924 >> 172.21.2.102:0/3661979924 conn(0x7f35240844f0 msgr2=0x7f352419bf80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:04.060 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:04.061 7f352b312700 1 -- 172.21.2.102:0/3661979924 shutdown_connections 2021-02-10T11:24:04.060 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:04.061 7f352b312700 1 -- 172.21.2.102:0/3661979924 wait complete. 2021-02-10T11:24:04.068 INFO:tasks.ceph.mds.a:Restarting daemon 2021-02-10T11:24:04.068 INFO:tasks.ceph.mds.a:Stopping old one... 2021-02-10T11:24:04.068 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2021-02-10T11:24:04.068 INFO:teuthology.orchestra.run:waiting for 300 2021-02-10T11:24:04.068 INFO:tasks.ceph.mds.b:Restarting daemon 2021-02-10T11:24:04.068 INFO:tasks.ceph.mds.b:Stopping old one... 2021-02-10T11:24:04.069 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2021-02-10T11:24:04.069 INFO:teuthology.orchestra.run:waiting for 300 2021-02-10T11:24:04.075 INFO:tasks.ceph.mds.b:Stopped 2021-02-10T11:24:04.075 DEBUG:teuthology.orchestra.run.gibba013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2021-02-10T11:24:04.113 INFO:tasks.ceph.mds.a:Stopped 2021-02-10T11:24:04.113 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2021-02-10T11:24:04.117 INFO:tasks.ceph.mds.b:Started 2021-02-10T11:24:04.117 DEBUG:teuthology.parallel:result is None 2021-02-10T11:24:04.155 INFO:tasks.ceph.mds.a:Started 2021-02-10T11:24:04.155 DEBUG:teuthology.parallel:result is None 2021-02-10T11:24:04.156 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd blacklist clear 2021-02-10T11:24:04.521 INFO:tasks.ceph.mds.b.gibba013.stdout:starting mds.b at 2021-02-10T11:24:04.563 INFO:tasks.ceph.mds.a.gibba002.stdout:starting mds.a at 2021-02-10T11:24:04.586 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:04.571 7fc2f5990700 1 Processor -- start 2021-02-10T11:24:04.586 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:04.571 7fc2f5990700 1 -- start start 2021-02-10T11:24:04.586 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:04.572 7fc2f5990700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fc2f00e5470 0x7fc2f007e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:04.586 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:04.572 7fc2f5990700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fc2f007f300 0x7fc2f007f700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:04.587 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:04.572 7fc2f5990700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2f007ed80 con 0x7fc2f00e5070 2021-02-10T11:24:04.587 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:04.572 7fc2f5990700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2f01b4080 con 0x7fc2f007f300 2021-02-10T11:24:04.587 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:04.572 7fc2f5990700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2f00e5f70 con 0x7fc2f00e5470 2021-02-10T11:24:04.587 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:04.572 7fc2efd75700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fc2f007f300 0x7fc2f007f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:04.587 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:04.572 7fc2efd75700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fc2f007f300 0x7fc2f007f700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:56838/0 (socket says 172.21.2.102:56838) 2021-02-10T11:24:04.587 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:04.572 7fc2efd75700 1 -- 172.21.2.102:0/3628161243 learned_addr learned my addr 172.21.2.102:0/3628161243 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:24:04.587 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:04.572 7fc2eed73700 1 --2- 172.21.2.102:0/3628161243 >> v2:172.21.2.113:3300/0 conn(0x7fc2f00e5470 0x7fc2f007e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:04.587 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:04.572 7fc2efd75700 1 -- 172.21.2.102:0/3628161243 >> v2:172.21.2.113:3300/0 conn(0x7fc2f00e5470 msgr2=0x7fc2f007e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:24:04.587 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:04.572 7fc2efd75700 1 --2- 172.21.2.102:0/3628161243 >> v2:172.21.2.113:3300/0 conn(0x7fc2f00e5470 0x7fc2f007e6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:04.587 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:04.572 7fc2efd75700 1 -- 172.21.2.102:0/3628161243 >> v1:172.21.2.113:6789/0 conn(0x7fc2f00e5070 legacy=0x7fc2f01b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:24:04.587 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:04.572 7fc2efd75700 1 -- 172.21.2.102:0/3628161243 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2f00d1790 con 0x7fc2f007f300 2021-02-10T11:24:04.587 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:04.572 7fc2eed73700 1 --2- 172.21.2.102:0/3628161243 >> v2:172.21.2.113:3300/0 conn(0x7fc2f00e5470 0x7fc2f007e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:24:04.587 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:04.573 7fc2efd75700 1 --2- 172.21.2.102:0/3628161243 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc2f007f300 0x7fc2f007f700 secure :-1 s=READY pgs=1292 cs=0 l=1 rev1=1 rx=0x7fc2e0006b80 tx=0x7fc2e0009b90).ready entity=mon.0 client_cookie=8c0e4c8b4d4f9c62 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:04.588 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:04.573 7fc2ee572700 1 -- 172.21.2.102:0/3628161243 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc2e00022e0 con 0x7fc2f007f300 2021-02-10T11:24:04.588 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:04.573 7fc2ee572700 1 -- 172.21.2.102:0/3628161243 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2e0002620 con 0x7fc2f007f300 2021-02-10T11:24:04.588 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:04.573 7fc2ee572700 1 -- 172.21.2.102:0/3628161243 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc2e0002b00 con 0x7fc2f007f300 2021-02-10T11:24:04.588 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:04.573 7fc2f5990700 1 -- 172.21.2.102:0/3628161243 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc2f007f300 msgr2=0x7fc2f007f700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:04.588 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:04.573 7fc2f5990700 1 --2- 172.21.2.102:0/3628161243 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc2f007f300 0x7fc2f007f700 secure :-1 s=READY pgs=1292 cs=0 l=1 rev1=1 rx=0x7fc2e0006b80 tx=0x7fc2e0009b90).stop 2021-02-10T11:24:04.588 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:04.573 7fc2f5990700 1 -- 172.21.2.102:0/3628161243 shutdown_connections 2021-02-10T11:24:04.588 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:04.573 7fc2f5990700 1 --2- 172.21.2.102:0/3628161243 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc2f007f300 0x7fc2f007f700 unknown :-1 s=CLOSED pgs=1292 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:04.588 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:04.573 7fc2f5990700 1 --2- 172.21.2.102:0/3628161243 >> v2:172.21.2.113:3300/0 conn(0x7fc2f00e5470 0x7fc2f007e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:04.588 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:04.573 7fc2f5990700 1 -- 172.21.2.102:0/3628161243 >> 172.21.2.102:0/3628161243 conn(0x7fc2f00ce690 msgr2=0x7fc2f00e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:04.588 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:04.573 7fc2f5990700 1 -- 172.21.2.102:0/3628161243 shutdown_connections 2021-02-10T11:24:04.588 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:04.573 7fc2f5990700 1 -- 172.21.2.102:0/3628161243 wait complete. 2021-02-10T11:24:04.590 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:04.588 7fc2f5990700 1 Processor -- start 2021-02-10T11:24:04.590 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:04.588 7fc2f5990700 1 -- start start 2021-02-10T11:24:04.590 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:04.589 7fc2f5990700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc2f0082140 0x7fc2f00937b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:04.590 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:04.589 7fc2f5990700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc2f0094350 0x7fc2f0094770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:04.590 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:04.589 7fc2f5990700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc2f0094d90 0x7fc2f00956e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:04.590 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:04.589 7fc2f5990700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2f0093dd0 con 0x7fc2f0082140 2021-02-10T11:24:04.590 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:04.589 7fc2f5990700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2f0093ff0 con 0x7fc2f0094350 2021-02-10T11:24:04.591 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:04.589 7fc2f5990700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2f0095190 con 0x7fc2f0094d90 2021-02-10T11:24:04.591 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:04.589 7fc2ef574700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc2f0082140 0x7fc2f00937b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:04.591 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:04.589 7fc2ef574700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc2f0082140 0x7fc2f00937b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:56844/0 (socket says 172.21.2.102:56844) 2021-02-10T11:24:04.591 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:04.589 7fc2ef574700 1 -- 172.21.2.102:0/4105764595 learned_addr learned my addr 172.21.2.102:0/4105764595 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:24:04.591 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:04.589 7fc2efd75700 1 --2- 172.21.2.102:0/4105764595 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc2f0094d90 0x7fc2f00956e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:04.591 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:04.589 7fc2eed73700 1 --2- 172.21.2.102:0/4105764595 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc2f0094350 0x7fc2f0094770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:04.591 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:04.589 7fc2ef574700 1 -- 172.21.2.102:0/4105764595 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc2f0094d90 msgr2=0x7fc2f00956e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:04.591 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:04.589 7fc2ef574700 1 --2- 172.21.2.102:0/4105764595 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc2f0094d90 0x7fc2f00956e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:04.591 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:04.589 7fc2ef574700 1 -- 172.21.2.102:0/4105764595 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc2f0094350 msgr2=0x7fc2f0094770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:04.591 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:04.589 7fc2ef574700 1 --2- 172.21.2.102:0/4105764595 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc2f0094350 0x7fc2f0094770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:04.591 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:04.589 7fc2ef574700 1 -- 172.21.2.102:0/4105764595 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2f00953b0 con 0x7fc2f0082140 2021-02-10T11:24:04.591 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:04.590 7fc2ef574700 1 --2- 172.21.2.102:0/4105764595 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc2f0082140 0x7fc2f00937b0 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 rx=0x7fc2e4009940 tx=0x7fc2e400bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:04.591 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:04.590 7fc2d7fff700 1 -- 172.21.2.102:0/4105764595 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc2e4000fe0 con 0x7fc2f0082140 2021-02-10T11:24:04.592 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:04.590 7fc2d7fff700 1 -- 172.21.2.102:0/4105764595 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2e400c2f0 con 0x7fc2f0082140 2021-02-10T11:24:04.592 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:04.590 7fc2d7fff700 1 -- 172.21.2.102:0/4105764595 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc2e400c810 con 0x7fc2f0082140 2021-02-10T11:24:04.592 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:04.590 7fc2f5990700 1 -- 172.21.2.102:0/4105764595 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2f00953b0 con 0x7fc2f0082140 2021-02-10T11:24:04.592 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:04.590 7fc2f5990700 1 -- 172.21.2.102:0/4105764595 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2f0095ff0 con 0x7fc2f0082140 2021-02-10T11:24:04.592 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:04.590 7fc2d7fff700 1 -- 172.21.2.102:0/4105764595 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fc2e4001230 con 0x7fc2f0082140 2021-02-10T11:24:04.592 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:04.590 7fc2d7fff700 1 --2- 172.21.2.102:0/4105764595 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc2d00186c0 0x7fc2d001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:04.592 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:04.590 7fc2d7fff700 1 -- 172.21.2.102:0/4105764595 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 4839+0+0 (secure 0 0 0) 0x7fc2e4032500 con 0x7fc2f0082140 2021-02-10T11:24:04.592 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:04.590 7fc2eed73700 1 --2- 172.21.2.102:0/4105764595 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc2d00186c0 0x7fc2d001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:04.592 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:04.591 7fc2eed73700 1 --2- 172.21.2.102:0/4105764595 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc2d00186c0 0x7fc2d001abc0 crc :-1 s=READY pgs=606 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:04.619 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:04.621 7fc2f5990700 1 -- 172.21.2.102:0/4105764595 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2f0107150 con 0x7fc2f0082140 2021-02-10T11:24:04.621 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:04.623 7fc2d7fff700 1 -- 172.21.2.102:0/4105764595 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fc2e400be30 con 0x7fc2f0082140 2021-02-10T11:24:04.761 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:04.761 7fc2f5990700 1 -- 172.21.2.102:0/4105764595 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd blacklist clear"} v 0) v1 -- 0x7fc2f007ed80 con 0x7fc2f0082140 2021-02-10T11:24:05.061 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:05.062 7fc2d7fff700 1 -- 172.21.2.102:0/4105764595 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd blacklist clear"}]=0 removed all blacklist entries v52) v1 ==== 97+0+0 (secure 0 0 0) 0x7fc2e4001230 con 0x7fc2f0082140 2021-02-10T11:24:05.075 INFO:teuthology.orchestra.run.gibba002.stderr: removed all blacklist entries 2021-02-10T11:24:05.077 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:05.078 7fc2f5990700 1 -- 172.21.2.102:0/4105764595 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc2d00186c0 msgr2=0x7fc2d001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:05.077 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:05.078 7fc2f5990700 1 --2- 172.21.2.102:0/4105764595 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc2d00186c0 0x7fc2d001abc0 crc :-1 s=READY pgs=606 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:05.077 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:05.078 7fc2f5990700 1 -- 172.21.2.102:0/4105764595 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc2f0082140 msgr2=0x7fc2f00937b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:05.077 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:05.078 7fc2f5990700 1 --2- 172.21.2.102:0/4105764595 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc2f0082140 0x7fc2f00937b0 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 rx=0x7fc2e4009940 tx=0x7fc2e400bcf0).stop 2021-02-10T11:24:05.077 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:05.078 7fc2f5990700 1 -- 172.21.2.102:0/4105764595 shutdown_connections 2021-02-10T11:24:05.077 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:05.078 7fc2f5990700 1 --2- 172.21.2.102:0/4105764595 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc2f0094d90 0x7fc2f00956e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:05.078 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:05.078 7fc2f5990700 1 --2- 172.21.2.102:0/4105764595 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc2d00186c0 0x7fc2d001abc0 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:05.078 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:05.078 7fc2f5990700 1 --2- 172.21.2.102:0/4105764595 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc2f0094350 0x7fc2f0094770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:05.078 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:05.078 7fc2f5990700 1 --2- 172.21.2.102:0/4105764595 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc2f0082140 0x7fc2f00937b0 unknown :-1 s=CLOSED pgs=1293 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:05.078 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:05.078 7fc2f5990700 1 -- 172.21.2.102:0/4105764595 >> 172.21.2.102:0/4105764595 conn(0x7fc2f00a6c50 msgr2=0x7fc2f00a6230 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:05.078 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:05.079 7fc2f5990700 1 -- 172.21.2.102:0/4105764595 shutdown_connections 2021-02-10T11:24:05.078 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:05.079 7fc2f5990700 1 -- 172.21.2.102:0/4105764595 wait complete. 2021-02-10T11:24:05.086 INFO:tasks.cephfs.cephfs_test_case:['0', '1', '2', '3'] 2021-02-10T11:24:05.086 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth ls --format=json-pretty 2021-02-10T11:24:05.522 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:05.507 7f1afc4ab700 1 Processor -- start 2021-02-10T11:24:05.523 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:05.507 7f1afc4ab700 1 -- start start 2021-02-10T11:24:05.523 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:05.508 7f1afc4ab700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1af40e5470 0x7f1af407e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:05.523 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:05.508 7f1afc4ab700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f1af407f300 0x7f1af407f700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:05.523 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:05.508 7f1afc4ab700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1af407ed80 con 0x7f1af40e5070 2021-02-10T11:24:05.523 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:05.508 7f1afc4ab700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1af41b4010 con 0x7f1af407f300 2021-02-10T11:24:05.523 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:05.508 7f1afc4ab700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1af40d1790 con 0x7f1af40e5470 2021-02-10T11:24:05.523 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:05.508 7f1afa2c7700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f1af40e5070 0x7f1af41b3820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:49072/0 (socket says 172.21.2.102:49072) 2021-02-10T11:24:05.523 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:05.508 7f1afa2c7700 1 -- 172.21.2.102:0/3701818551 learned_addr learned my addr 172.21.2.102:0/3701818551 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:24:05.523 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:05.508 7f1afaac8700 1 --2- 172.21.2.102:0/3701818551 >> v2:172.21.2.102:3300/0 conn(0x7f1af407f300 0x7f1af407f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:05.523 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:05.508 7f1af9ac6700 1 --2- 172.21.2.102:0/3701818551 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1af40e5470 0x7f1af407e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:05.524 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:05.508 7f1afaac8700 1 -- 172.21.2.102:0/3701818551 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1af40e5470 msgr2=0x7f1af407e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:24:05.524 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:05.508 7f1afaac8700 1 --2- 172.21.2.102:0/3701818551 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1af40e5470 0x7f1af407e6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:05.524 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:05.508 7f1afaac8700 1 -- 172.21.2.102:0/3701818551 >> v1:172.21.2.102:6789/0 conn(0x7f1af40e5070 legacy=0x7f1af41b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:05.524 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:05.508 7f1afaac8700 1 -- 172.21.2.102:0/3701818551 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1af40d1980 con 0x7f1af407f300 2021-02-10T11:24:05.524 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:05.508 7f1afaac8700 1 --2- 172.21.2.102:0/3701818551 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1af407f300 0x7f1af407f700 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 rx=0x7f1aec006e30 tx=0x7f1aec009e60).ready entity=mon.0 client_cookie=5baa5b4a2bf6e7a9 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:05.524 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:05.509 7f1af92c5700 1 -- 172.21.2.102:0/3701818551 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1aec006a50 con 0x7f1af407f300 2021-02-10T11:24:05.524 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:05.509 7f1af92c5700 1 -- 172.21.2.102:0/3701818551 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1aec00b300 con 0x7f1af407f300 2021-02-10T11:24:05.524 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:05.509 7f1af92c5700 1 -- 172.21.2.102:0/3701818551 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1aec00b820 con 0x7f1af407f300 2021-02-10T11:24:05.524 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:05.509 7f1afc4ab700 1 -- 172.21.2.102:0/3701818551 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1af407f300 msgr2=0x7f1af407f700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:05.524 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:05.509 7f1afc4ab700 1 --2- 172.21.2.102:0/3701818551 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1af407f300 0x7f1af407f700 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 rx=0x7f1aec006e30 tx=0x7f1aec009e60).stop 2021-02-10T11:24:05.524 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:05.509 7f1afc4ab700 1 -- 172.21.2.102:0/3701818551 shutdown_connections 2021-02-10T11:24:05.524 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:05.509 7f1afc4ab700 1 --2- 172.21.2.102:0/3701818551 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1af407f300 0x7f1af407f700 unknown :-1 s=CLOSED pgs=1295 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:05.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:05.509 7f1afc4ab700 1 --2- 172.21.2.102:0/3701818551 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1af40e5470 0x7f1af407e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:05.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:05.509 7f1afc4ab700 1 -- 172.21.2.102:0/3701818551 >> 172.21.2.102:0/3701818551 conn(0x7f1af40ce690 msgr2=0x7f1af40e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:05.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:05.510 7f1afc4ab700 1 -- 172.21.2.102:0/3701818551 shutdown_connections 2021-02-10T11:24:05.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:05.510 7f1afc4ab700 1 -- 172.21.2.102:0/3701818551 wait complete. 2021-02-10T11:24:05.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:05.524 7f1afc4ab700 1 Processor -- start 2021-02-10T11:24:05.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:05.524 7f1afc4ab700 1 -- start start 2021-02-10T11:24:05.526 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:05.525 7f1afc4ab700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1af408d400 0x7f1af4096c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:05.526 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:05.525 7f1afc4ab700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1af40b1450 0x7f1af40b1850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:05.526 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:05.525 7f1afc4ab700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1af40b1e70 0x7f1af41b1f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:05.526 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:05.525 7f1afc4ab700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1af4097310 con 0x7f1af40b1450 2021-02-10T11:24:05.526 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:05.525 7f1afc4ab700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1af408d800 con 0x7f1af408d400 2021-02-10T11:24:05.527 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:05.525 7f1afc4ab700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1af408d9f0 con 0x7f1af40b1e70 2021-02-10T11:24:05.527 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:05.525 7f1af9ac6700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1af40b1450 0x7f1af40b1850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:05.527 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:05.525 7f1af9ac6700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1af40b1450 0x7f1af40b1850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:56858/0 (socket says 172.21.2.102:56858) 2021-02-10T11:24:05.527 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:05.525 7f1af9ac6700 1 -- 172.21.2.102:0/912026432 learned_addr learned my addr 172.21.2.102:0/912026432 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:24:05.527 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:05.525 7f1af9ac6700 1 -- 172.21.2.102:0/912026432 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1af40b1e70 msgr2=0x7f1af41b1f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:05.527 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:05.525 7f1af9ac6700 1 --2- 172.21.2.102:0/912026432 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1af40b1e70 0x7f1af41b1f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:05.527 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:05.525 7f1af9ac6700 1 -- 172.21.2.102:0/912026432 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1af408d400 msgr2=0x7f1af4096c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:05.527 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:05.525 7f1af9ac6700 1 --2- 172.21.2.102:0/912026432 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1af408d400 0x7f1af4096c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:05.527 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:05.525 7f1af9ac6700 1 -- 172.21.2.102:0/912026432 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1af41b25f0 con 0x7f1af40b1450 2021-02-10T11:24:05.527 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:05.525 7f1af9ac6700 1 --2- 172.21.2.102:0/912026432 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1af40b1450 0x7f1af40b1850 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 rx=0x7f1ae8006f60 tx=0x7f1ae800b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:05.527 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:05.526 7f1ae6ffd700 1 -- 172.21.2.102:0/912026432 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1ae800bd60 con 0x7f1af40b1450 2021-02-10T11:24:05.527 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:05.526 7f1ae6ffd700 1 -- 172.21.2.102:0/912026432 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ae8000c90 con 0x7f1af40b1450 2021-02-10T11:24:05.527 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:05.526 7f1afc4ab700 1 -- 172.21.2.102:0/912026432 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1af41b25f0 con 0x7f1af40b1450 2021-02-10T11:24:05.528 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:05.526 7f1afc4ab700 1 -- 172.21.2.102:0/912026432 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1af41b2b80 con 0x7f1af40b1450 2021-02-10T11:24:05.528 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:05.526 7f1ae6ffd700 1 -- 172.21.2.102:0/912026432 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1ae8002220 con 0x7f1af40b1450 2021-02-10T11:24:05.528 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:05.526 7f1ae6ffd700 1 -- 172.21.2.102:0/912026432 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1ae80052e0 con 0x7f1af40b1450 2021-02-10T11:24:05.528 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:05.526 7f1ae6ffd700 1 --2- 172.21.2.102:0/912026432 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1adc018730 0x7f1adc01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:05.528 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:05.526 7f1ae6ffd700 1 -- 172.21.2.102:0/912026432 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 4753+0+0 (secure 0 0 0) 0x7f1ae8032450 con 0x7f1af40b1450 2021-02-10T11:24:05.528 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:05.527 7f1afa2c7700 1 --2- 172.21.2.102:0/912026432 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1adc018730 0x7f1adc01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:05.528 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:05.527 7f1afa2c7700 1 --2- 172.21.2.102:0/912026432 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1adc018730 0x7f1adc01ac10 crc :-1 s=READY pgs=607 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:05.555 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:05.557 7f1afc4ab700 1 -- 172.21.2.102:0/912026432 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1af407ed80 con 0x7f1af40b1450 2021-02-10T11:24:05.558 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:05.560 7f1ae6ffd700 1 -- 172.21.2.102:0/912026432 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f1ae8000940 con 0x7f1af40b1450 2021-02-10T11:24:05.691 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:05.693 7f1afc4ab700 1 -- 172.21.2.102:0/912026432 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f1af4097290 con 0x7f1af40b1450 2021-02-10T11:24:05.692 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:05.694 7f1ae6ffd700 1 -- 172.21.2.102:0/912026432 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v28) v1 ==== 80+0+4513 (secure 0 0 0) 0x7f1ae80052e0 con 0x7f1af40b1450 2021-02-10T11:24:05.692 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:24:05.692 INFO:teuthology.orchestra.run.gibba002.stdout:{ 2021-02-10T11:24:05.693 INFO:teuthology.orchestra.run.gibba002.stdout: "auth_dump": [ 2021-02-10T11:24:05.693 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:24:05.693 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "mds.a", 2021-02-10T11:24:05.693 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAevSNgsgyEARAAvr0cUjjCoEHLxaZSZ2HtVA==", 2021-02-10T11:24:05.693 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:24:05.693 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow", 2021-02-10T11:24:05.693 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:24:05.693 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:24:05.693 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:24:05.693 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:24:05.693 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:24:05.693 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:24:05.693 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "mds.b", 2021-02-10T11:24:05.694 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAevSNgwOFMMRAALfb1fbX56ao1TU8ClnTW1A==", 2021-02-10T11:24:05.694 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:24:05.694 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow", 2021-02-10T11:24:05.694 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:24:05.694 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:24:05.694 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:24:05.694 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:24:05.694 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:24:05.694 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:24:05.694 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.0", 2021-02-10T11:24:05.694 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAmvSNgj+bMMBAAD3HUD6fLiMAg+++iNzLJKQ==", 2021-02-10T11:24:05.694 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:24:05.694 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:24:05.695 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:24:05.695 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:24:05.695 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:24:05.695 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:24:05.695 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:24:05.695 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.1", 2021-02-10T11:24:05.695 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQApvSNgnvc+MRAAAOxqlNwiyLHqNITOrYeFKQ==", 2021-02-10T11:24:05.695 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:24:05.695 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:24:05.695 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:24:05.695 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:24:05.695 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:24:05.695 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:24:05.696 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:24:05.696 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.2", 2021-02-10T11:24:05.696 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAsvSNgZ4SbMRAA76IOYWfbiAuBoM/4k81GxA==", 2021-02-10T11:24:05.696 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:24:05.696 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:24:05.696 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:24:05.696 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:24:05.696 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:24:05.696 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:24:05.696 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:24:05.696 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.3", 2021-02-10T11:24:05.696 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAvvSNgFk3rMRAAWCmKKyw0LDQ+UlAHZAITOw==", 2021-02-10T11:24:05.696 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:24:05.696 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:24:05.697 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:24:05.697 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:24:05.697 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:24:05.697 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:24:05.697 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:24:05.697 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.4", 2021-02-10T11:24:05.697 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQA4vSNgpKcNJBAApJMmvj5YxVHPDOmvmQONOg==", 2021-02-10T11:24:05.697 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:24:05.697 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:24:05.697 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:24:05.697 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:24:05.697 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:24:05.697 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:24:05.697 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:24:05.698 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.5", 2021-02-10T11:24:05.698 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQA7vSNgf2ujJBAAfkuYpyb3WTzc8V5kw+Wktg==", 2021-02-10T11:24:05.698 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:24:05.698 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:24:05.698 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:24:05.698 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:24:05.698 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:24:05.698 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:24:05.698 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:24:05.698 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.6", 2021-02-10T11:24:05.698 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQA+vSNgqw9uJRAAFMU0R+UxU9C3o/9byb3e0g==", 2021-02-10T11:24:05.698 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:24:05.699 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:24:05.699 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:24:05.699 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:24:05.699 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:24:05.699 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:24:05.699 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:24:05.699 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.7", 2021-02-10T11:24:05.699 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQBBvSNgvWSrJRAA2KaCHmLpzovRgYzLvyofTQ==", 2021-02-10T11:24:05.699 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:24:05.699 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:24:05.699 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:24:05.699 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:24:05.699 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:24:05.699 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:24:05.700 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:24:05.700 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "client.0", 2021-02-10T11:24:05.700 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAfvSNgZd9XIBAAvyrvl8fdKuHnJZfczfRoFw==", 2021-02-10T11:24:05.700 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:24:05.700 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow", 2021-02-10T11:24:05.700 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow r", 2021-02-10T11:24:05.700 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:24:05.700 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:24:05.700 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:24:05.700 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:24:05.700 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "client.1", 2021-02-10T11:24:05.700 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAfvSNgSyfXNhAAIduPnF7Vd/6HSkJ0MlcMSA==", 2021-02-10T11:24:05.700 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:24:05.700 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow", 2021-02-10T11:24:05.701 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow r", 2021-02-10T11:24:05.701 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:24:05.701 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:24:05.701 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:24:05.701 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:24:05.701 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "client.2", 2021-02-10T11:24:05.701 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAgvSNgiIG/ERAA9ZTamaTI7ZsPSnxBqjplCg==", 2021-02-10T11:24:05.701 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:24:05.701 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow", 2021-02-10T11:24:05.701 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow r", 2021-02-10T11:24:05.701 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:24:05.701 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:24:05.701 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:24:05.701 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:24:05.702 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "client.3", 2021-02-10T11:24:05.702 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAgvSNgOWpOKBAAlXsEB04Q/GvtL06PqRMLeA==", 2021-02-10T11:24:05.702 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:24:05.702 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow", 2021-02-10T11:24:05.702 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow r", 2021-02-10T11:24:05.702 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:24:05.702 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:24:05.702 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:24:05.702 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:24:05.702 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "client.admin", 2021-02-10T11:24:05.702 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAavSNgO3GQBhAAL+NVnYskQRKVWLgyDPY/xw==", 2021-02-10T11:24:05.702 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:24:05.703 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow *", 2021-02-10T11:24:05.703 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:24:05.703 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:24:05.703 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:24:05.703 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:24:05.703 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:24:05.703 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:24:05.703 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "mgr.x", 2021-02-10T11:24:05.703 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAdvSNgXZZgJhAA7dtBSGzH+DsBwnddZpOBJA==", 2021-02-10T11:24:05.703 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:24:05.703 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow *", 2021-02-10T11:24:05.703 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow profile mgr", 2021-02-10T11:24:05.703 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:24:05.703 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:24:05.703 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:24:05.704 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:24:05.704 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "mgr.y", 2021-02-10T11:24:05.704 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAdvSNgVqQFDRAAwo6YEdWZCAr5GdOWVL/Qvg==", 2021-02-10T11:24:05.704 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:24:05.704 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow *", 2021-02-10T11:24:05.704 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow profile mgr", 2021-02-10T11:24:05.704 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:24:05.704 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:24:05.704 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:24:05.704 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:24:05.704 INFO:teuthology.orchestra.run.gibba002.stdout:} 2021-02-10T11:24:05.705 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:05.695 7f1afc4ab700 1 -- 172.21.2.102:0/912026432 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1adc018730 msgr2=0x7f1adc01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:05.705 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:05.695 7f1afc4ab700 1 --2- 172.21.2.102:0/912026432 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1adc018730 0x7f1adc01ac10 crc :-1 s=READY pgs=607 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:05.706 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:05.695 7f1afc4ab700 1 -- 172.21.2.102:0/912026432 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1af40b1450 msgr2=0x7f1af40b1850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:05.706 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:05.695 7f1afc4ab700 1 --2- 172.21.2.102:0/912026432 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1af40b1450 0x7f1af40b1850 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 rx=0x7f1ae8006f60 tx=0x7f1ae800b970).stop 2021-02-10T11:24:05.706 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:05.696 7f1afc4ab700 1 -- 172.21.2.102:0/912026432 shutdown_connections 2021-02-10T11:24:05.706 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:05.696 7f1afc4ab700 1 --2- 172.21.2.102:0/912026432 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1af40b1e70 0x7f1af41b1f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:05.706 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:05.696 7f1afc4ab700 1 --2- 172.21.2.102:0/912026432 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1af40b1450 0x7f1af40b1850 unknown :-1 s=CLOSED pgs=1296 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:05.706 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:05.696 7f1afc4ab700 1 --2- 172.21.2.102:0/912026432 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1adc018730 0x7f1adc01ac10 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:05.706 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:05.696 7f1afc4ab700 1 --2- 172.21.2.102:0/912026432 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1af408d400 0x7f1af4096c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:05.706 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:05.696 7f1afc4ab700 1 -- 172.21.2.102:0/912026432 >> 172.21.2.102:0/912026432 conn(0x7f1af41ac620 msgr2=0x7f1af40896e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:05.706 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:05.696 7f1afc4ab700 1 -- 172.21.2.102:0/912026432 shutdown_connections 2021-02-10T11:24:05.706 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:05.696 7f1afc4ab700 1 -- 172.21.2.102:0/912026432 wait complete. 2021-02-10T11:24:05.707 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:24:06.134 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:06.118 7f6d06cd6700 1 Processor -- start 2021-02-10T11:24:06.134 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:06.118 7f6d06cd6700 1 -- start start 2021-02-10T11:24:06.134 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:06.119 7f6d06cd6700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6d000e5070 0x7f6d001b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:06.134 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:06.119 7f6d06cd6700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f6d000e5470 0x7f6d0007d690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:06.134 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:06.119 7f6d06cd6700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d0007ed80 con 0x7f6d001b3e40 2021-02-10T11:24:06.134 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:06.119 7f6d06cd6700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d00080310 con 0x7f6d000e5470 2021-02-10T11:24:06.134 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:06.119 7f6d06cd6700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d00080500 con 0x7f6d000e5070 2021-02-10T11:24:06.134 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:06.119 7f6d052f3700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f6d001b3e40 0x7f6d0007dcb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:49090/0 (socket says 172.21.2.102:49090) 2021-02-10T11:24:06.134 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:06.119 7f6d052f3700 1 -- 172.21.2.102:0/1180991777 learned_addr learned my addr 172.21.2.102:0/1180991777 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:24:06.134 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:06.119 7f6cfffff700 1 --2- 172.21.2.102:0/1180991777 >> v2:172.21.2.113:3300/0 conn(0x7f6d000e5470 0x7f6d0007d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:06.135 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:06.119 7f6d04af2700 1 --2- 172.21.2.102:0/1180991777 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6d000e5070 0x7f6d001b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:06.135 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:06.119 7f6cff7fe700 1 -- 172.21.2.102:0/1180991777 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2426449728 0 0) 0x7f6cf8001500 con 0x7f6d001b3e40 2021-02-10T11:24:06.135 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:06.119 7f6cff7fe700 1 -- 172.21.2.102:0/1180991777 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6cec002fe0 con 0x7f6d001b3e40 2021-02-10T11:24:06.135 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:06.120 7f6cfffff700 1 -- 172.21.2.102:0/1180991777 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6d000e5070 msgr2=0x7f6d001b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:24:06.135 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:06.120 7f6cfffff700 1 --2- 172.21.2.102:0/1180991777 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6d000e5070 0x7f6d001b3820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:06.135 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:06.120 7f6cfffff700 1 -- 172.21.2.102:0/1180991777 >> v1:172.21.2.102:6789/0 conn(0x7f6d001b3e40 legacy=0x7f6d0007dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:06.135 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:06.120 7f6cfffff700 1 -- 172.21.2.102:0/1180991777 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d000806f0 con 0x7f6d000e5470 2021-02-10T11:24:06.135 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:06.120 7f6cfffff700 1 --2- 172.21.2.102:0/1180991777 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6d000e5470 0x7f6d0007d690 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 rx=0x7f6cf0006650 tx=0x7f6cf0009a80).ready entity=mon.1 client_cookie=ab9b9cd83c409ba7 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:06.135 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:06.120 7f6cff7fe700 1 -- 172.21.2.102:0/1180991777 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6cf000c0c0 con 0x7f6d000e5470 2021-02-10T11:24:06.135 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:06.120 7f6cff7fe700 1 -- 172.21.2.102:0/1180991777 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cf000c340 con 0x7f6d000e5470 2021-02-10T11:24:06.135 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:06.120 7f6cff7fe700 1 -- 172.21.2.102:0/1180991777 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6cf000c860 con 0x7f6d000e5470 2021-02-10T11:24:06.135 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:06.120 7f6d06cd6700 1 -- 172.21.2.102:0/1180991777 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6d000e5470 msgr2=0x7f6d0007d690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:06.135 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:06.120 7f6d06cd6700 1 --2- 172.21.2.102:0/1180991777 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6d000e5470 0x7f6d0007d690 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 rx=0x7f6cf0006650 tx=0x7f6cf0009a80).stop 2021-02-10T11:24:06.136 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:06.121 7f6d06cd6700 1 -- 172.21.2.102:0/1180991777 shutdown_connections 2021-02-10T11:24:06.136 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:06.121 7f6d06cd6700 1 --2- 172.21.2.102:0/1180991777 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6d000e5470 0x7f6d0007d690 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:06.136 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:06.121 7f6d06cd6700 1 --2- 172.21.2.102:0/1180991777 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6d000e5070 0x7f6d001b3820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:06.136 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:06.121 7f6d06cd6700 1 -- 172.21.2.102:0/1180991777 >> 172.21.2.102:0/1180991777 conn(0x7f6d000ce690 msgr2=0x7f6d000e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:06.136 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:06.121 7f6d06cd6700 1 -- 172.21.2.102:0/1180991777 shutdown_connections 2021-02-10T11:24:06.136 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:06.121 7f6d06cd6700 1 -- 172.21.2.102:0/1180991777 wait complete. 2021-02-10T11:24:06.138 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:06.136 7f6d06cd6700 1 Processor -- start 2021-02-10T11:24:06.138 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:06.136 7f6d06cd6700 1 -- start start 2021-02-10T11:24:06.138 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:06.136 7f6d06cd6700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6d001918e0 0x7f6d00189c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:06.138 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:06.136 7f6d06cd6700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6d0018a2a0 0x7f6d0018a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:06.138 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:06.136 7f6d06cd6700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6d0018acc0 0x7f6d00190070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:06.138 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:06.136 7f6d06cd6700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d001a2f00 con 0x7f6d001918e0 2021-02-10T11:24:06.138 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:06.136 7f6d06cd6700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d00191ce0 con 0x7f6d0018acc0 2021-02-10T11:24:06.138 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:06.136 7f6d06cd6700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d00191f00 con 0x7f6d0018a2a0 2021-02-10T11:24:06.138 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:06.136 7f6d04af2700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6d001918e0 0x7f6d00189c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:06.138 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:06.137 7f6d04af2700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6d001918e0 0x7f6d00189c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:56870/0 (socket says 172.21.2.102:56870) 2021-02-10T11:24:06.138 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:06.137 7f6d04af2700 1 -- 172.21.2.102:0/1366095749 learned_addr learned my addr 172.21.2.102:0/1366095749 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:24:06.139 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:06.137 7f6cfffff700 1 --2- 172.21.2.102:0/1366095749 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6d0018a2a0 0x7f6d0018a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:06.139 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:06.137 7f6d052f3700 1 --2- 172.21.2.102:0/1366095749 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6d0018acc0 0x7f6d00190070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:06.139 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:06.137 7f6d04af2700 1 -- 172.21.2.102:0/1366095749 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6d0018a2a0 msgr2=0x7f6d0018a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:06.139 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:06.137 7f6d04af2700 1 --2- 172.21.2.102:0/1366095749 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6d0018a2a0 0x7f6d0018a6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:06.139 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:06.137 7f6d04af2700 1 -- 172.21.2.102:0/1366095749 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6d0018acc0 msgr2=0x7f6d00190070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:06.139 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:06.137 7f6d04af2700 1 --2- 172.21.2.102:0/1366095749 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6d0018acc0 0x7f6d00190070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:06.139 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:06.137 7f6d04af2700 1 -- 172.21.2.102:0/1366095749 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d00190690 con 0x7f6d001918e0 2021-02-10T11:24:06.139 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:06.137 7f6d04af2700 1 --2- 172.21.2.102:0/1366095749 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6d001918e0 0x7f6d00189c80 secure :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 rx=0x7f6ce8006f60 tx=0x7f6ce800b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:06.139 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:06.137 7f6cfd7fa700 1 -- 172.21.2.102:0/1366095749 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6ce800bd60 con 0x7f6d001918e0 2021-02-10T11:24:06.139 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:06.137 7f6cfd7fa700 1 -- 172.21.2.102:0/1366095749 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ce8000c90 con 0x7f6d001918e0 2021-02-10T11:24:06.139 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:06.137 7f6cfd7fa700 1 -- 172.21.2.102:0/1366095749 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6ce8002220 con 0x7f6d001918e0 2021-02-10T11:24:06.139 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:06.137 7f6d06cd6700 1 -- 172.21.2.102:0/1366095749 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d00190690 con 0x7f6d001918e0 2021-02-10T11:24:06.139 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:06.138 7f6d06cd6700 1 -- 172.21.2.102:0/1366095749 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d00190c20 con 0x7f6d001918e0 2021-02-10T11:24:06.139 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:06.138 7f6cfd7fa700 1 -- 172.21.2.102:0/1366095749 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f6ce80052e0 con 0x7f6d001918e0 2021-02-10T11:24:06.140 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:06.138 7f6cfd7fa700 1 --2- 172.21.2.102:0/1366095749 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6ce40186c0 0x7f6ce401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:06.140 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:06.138 7f6cfd7fa700 1 -- 172.21.2.102:0/1366095749 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 4753+0+0 (secure 0 0 0) 0x7f6ce8032230 con 0x7f6d001918e0 2021-02-10T11:24:06.140 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:06.138 7f6cfffff700 1 --2- 172.21.2.102:0/1366095749 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6ce40186c0 0x7f6ce401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:06.140 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:06.139 7f6cfffff700 1 --2- 172.21.2.102:0/1366095749 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6ce40186c0 0x7f6ce401abc0 crc :-1 s=READY pgs=608 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:06.167 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:06.169 7f6d06cd6700 1 -- 172.21.2.102:0/1366095749 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d00189750 con 0x7f6d001918e0 2021-02-10T11:24:06.169 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:06.171 7f6cfd7fa700 1 -- 172.21.2.102:0/1366095749 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f6ce802dc30 con 0x7f6d001918e0 2021-02-10T11:24:06.294 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:06.296 7f6d06cd6700 1 -- 172.21.2.102:0/1366095749 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6d00189750 con 0x7f6d001918e0 2021-02-10T11:24:06.295 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:06.297 7f6cfd7fa700 1 -- 172.21.2.102:0/1366095749 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v52) v1 ==== 74+0+11144 (secure 0 0 0) 0x7f6ce802dc30 con 0x7f6d001918e0 2021-02-10T11:24:06.295 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:24:06.295 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":52,"fsid":"68db30ea-dbce-4aef-a6df-f01c6fea0513","created":"2021-02-10 11:03:06.331557","modified":"2021-02-10 11:24:05.060596","last_up_change":"2021-02-10 11:03:32.690562","last_in_change":"2021-02-10 11:03:32.690562","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:03:37.322519","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"be691b4d-a039-408a-af80-9d53f02b6f1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6818","nonce":36088},{"type":"v1","addr":"172.21.2.102:6820","nonce":36088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6822","nonce":36088},{"type":"v1","addr":"172.21.2.102:6824","nonce":36088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6830","nonce":36088},{"type":"v1","addr":"172.21.2.102:6832","nonce":36088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6826","nonce":36088},{"type":"v1","addr":"172.21.2.102:6828","nonce":36088}]},"public_addr":"172.21.2.102:6820/36088","cluster_addr":"172.21.2.102:6824/36088","heartbeat_back_addr":"172.21.2.102:6832/36088","heartbeat_front_addr":"172.21.2.102:6828/36088","state":["exists","up"]},{"osd":1,"uuid":"64e3ccc8-9c69-4d7c-865e-af8e83822635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6819","nonce":36087},{"type":"v1","addr":"172.21.2.102:6821","nonce":36087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6823","nonce":36087},{"type":"v1","addr":"172.21.2.102:6825","nonce":36087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6831","nonce":36087},{"type":"v1","addr":"172.21.2.102:6833","nonce":36087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6827","nonce":36087},{"type":"v1","addr":"172.21.2.102:6829","nonce":36087}]},"public_addr":"172.21.2.102:6821/36087","cluster_addr":"172.21.2.102:6825/36087","heartbeat_back_addr":"172.21.2.102:6833/36087","heartbeat_front_addr":"172.21.2.102:6829/36087","state":["exists","up"]},{"osd":2,"uuid":"153700af-747f-481f-bdf4-d6072d3a842f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6810","nonce":36089},{"type":"v1","addr":"172.21.2.102:6811","nonce":36089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6812","nonce":36089},{"type":"v1","addr":"172.21.2.102:6813","nonce":36089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6816","nonce":36089},{"type":"v1","addr":"172.21.2.102:6817","nonce":36089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6814","nonce":36089},{"type":"v1","addr":"172.21.2.102:6815","nonce":36089}]},"public_addr":"172.21.2.102:6811/36089","cluster_addr":"172.21.2.102:6813/36089","heartbeat_back_addr":"172.21.2.102:6817/36089","heartbeat_front_addr":"172.21.2.102:6815/36089","state":["exists","up"]},{"osd":3,"uuid":"cbc678f4-83a9-45e6-8830-2f7c232b1ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6802","nonce":36090},{"type":"v1","addr":"172.21.2.102:6803","nonce":36090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6804","nonce":36090},{"type":"v1","addr":"172.21.2.102:6805","nonce":36090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6808","nonce":36090},{"type":"v1","addr":"172.21.2.102:6809","nonce":36090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6806","nonce":36090},{"type":"v1","addr":"172.21.2.102:6807","nonce":36090}]},"public_addr":"172.21.2.102:6803/36090","cluster_addr":"172.21.2.102:6805/36090","heartbeat_back_addr":"172.21.2.102:6809/36090","heartbeat_front_addr":"172.21.2.102:6807/36090","state":["exists","up"]},{"osd":4,"uuid":"c01e4008-b258-4f10-afd2-9e7f37214351","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6808","nonce":36368},{"type":"v1","addr":"172.21.2.113:6811","nonce":36368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6814","nonce":36368},{"type":"v1","addr":"172.21.2.113:6818","nonce":36368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6826","nonce":36368},{"type":"v1","addr":"172.21.2.113:6828","nonce":36368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6820","nonce":36368},{"type":"v1","addr":"172.21.2.113:6824","nonce":36368}]},"public_addr":"172.21.2.113:6811/36368","cluster_addr":"172.21.2.113:6818/36368","heartbeat_back_addr":"172.21.2.113:6828/36368","heartbeat_front_addr":"172.21.2.113:6824/36368","state":["exists","up"]},{"osd":5,"uuid":"551f6007-daf8-4a98-8ee0-c2b16bcccb5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6802","nonce":36366},{"type":"v1","addr":"172.21.2.113:6805","nonce":36366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6807","nonce":36366},{"type":"v1","addr":"172.21.2.113:6810","nonce":36366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6819","nonce":36366},{"type":"v1","addr":"172.21.2.113:6822","nonce":36366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6812","nonce":36366},{"type":"v1","addr":"172.21.2.113:6815","nonce":36366}]},"public_addr":"172.21.2.113:6805/36366","cluster_addr":"172.21.2.113:6810/36366","heartbeat_back_addr":"172.21.2.113:6822/36366","heartbeat_front_addr":"172.21.2.113:6815/36366","state":["exists","up"]},{"osd":6,"uuid":"bf67c040-0e50-48be-b102-c3943bc9ebef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6800","nonce":36367},{"type":"v1","addr":"172.21.2.113:6801","nonce":36367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6803","nonce":36367},{"type":"v1","addr":"172.21.2.113:6804","nonce":36367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6813","nonce":36367},{"type":"v1","addr":"172.21.2.113:6816","nonce":36367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6806","nonce":36367},{"type":"v1","addr":"172.21.2.113:6809","nonce":36367}]},"public_addr":"172.21.2.113:6801/36367","cluster_addr":"172.21.2.113:6804/36367","heartbeat_back_addr":"172.21.2.113:6816/36367","heartbeat_front_addr":"172.21.2.113:6809/36367","state":["exists","up"]},{"osd":7,"uuid":"f5461f80-018d-428f-be89-2eb2bcbe7bb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6817","nonce":36365},{"type":"v1","addr":"172.21.2.113:6821","nonce":36365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6823","nonce":36365},{"type":"v1","addr":"172.21.2.113:6825","nonce":36365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6830","nonce":36365},{"type":"v1","addr":"172.21.2.113:6831","nonce":36365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6827","nonce":36365},{"type":"v1","addr":"172.21.2.113:6829","nonce":36365}]},"public_addr":"172.21.2.113:6821/36365","cluster_addr":"172.21.2.113:6825/36365","heartbeat_back_addr":"172.21.2.113:6831/36365","heartbeat_front_addr":"172.21.2.113:6829/36365","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:24:06.296 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:06.298 7f6d06cd6700 1 -- 172.21.2.102:0/1366095749 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6ce40186c0 msgr2=0x7f6ce401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:06.296 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:06.298 7f6d06cd6700 1 --2- 172.21.2.102:0/1366095749 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6ce40186c0 0x7f6ce401abc0 crc :-1 s=READY pgs=608 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:06.296 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:06.298 7f6d06cd6700 1 -- 172.21.2.102:0/1366095749 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6d001918e0 msgr2=0x7f6d00189c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:06.297 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:06.298 7f6d06cd6700 1 --2- 172.21.2.102:0/1366095749 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6d001918e0 0x7f6d00189c80 secure :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 rx=0x7f6ce8006f60 tx=0x7f6ce800b970).stop 2021-02-10T11:24:06.297 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:06.298 7f6d06cd6700 1 -- 172.21.2.102:0/1366095749 shutdown_connections 2021-02-10T11:24:06.297 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:06.298 7f6d06cd6700 1 --2- 172.21.2.102:0/1366095749 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6ce40186c0 0x7f6ce401abc0 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:06.297 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:06.298 7f6d06cd6700 1 --2- 172.21.2.102:0/1366095749 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6d0018acc0 0x7f6d00190070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:06.297 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:06.298 7f6d06cd6700 1 --2- 172.21.2.102:0/1366095749 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6d0018a2a0 0x7f6d0018a6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:06.297 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:06.298 7f6d06cd6700 1 --2- 172.21.2.102:0/1366095749 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6d001918e0 0x7f6d00189c80 unknown :-1 s=CLOSED pgs=1298 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:06.298 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:06.298 7f6d06cd6700 1 -- 172.21.2.102:0/1366095749 >> 172.21.2.102:0/1366095749 conn(0x7f6d001a31d0 msgr2=0x7f6d0008b830 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:06.298 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:06.299 7f6d06cd6700 1 -- 172.21.2.102:0/1366095749 shutdown_connections 2021-02-10T11:24:06.298 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:06.299 7f6d06cd6700 1 -- 172.21.2.102:0/1366095749 wait complete. 2021-02-10T11:24:06.306 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:03:37.322519', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2021-02-10T11:24:06.306 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:24:06.736 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:06.721 7f4566f20700 1 Processor -- start 2021-02-10T11:24:06.736 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:06.721 7f4566f20700 1 -- start start 2021-02-10T11:24:06.736 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:06.722 7f4566f20700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f45601b0470 0x7f45601b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:06.736 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:06.722 7f4566f20700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f45600ef610 0x7f45600efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:06.736 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:06.722 7f4566f20700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45600f0b50 con 0x7f45600f0030 2021-02-10T11:24:06.736 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:06.722 7f4566f20700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45600e52e0 con 0x7f45600ef610 2021-02-10T11:24:06.736 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:06.722 7f4566f20700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45600e5490 con 0x7f45601b0470 2021-02-10T11:24:06.736 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:06.722 7f455ffff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f45600ef610 0x7f45600efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:06.736 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:06.722 7f455ffff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f45600ef610 0x7f45600efa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:56878/0 (socket says 172.21.2.102:56878) 2021-02-10T11:24:06.737 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:06.722 7f455ffff700 1 -- 172.21.2.102:0/655198727 learned_addr learned my addr 172.21.2.102:0/655198727 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:24:06.737 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:06.722 7f455ffff700 1 -- 172.21.2.102:0/655198727 >> v2:172.21.2.113:3300/0 conn(0x7f45601b0470 msgr2=0x7f45601b0870 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:24:06.737 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:06.722 7f455ffff700 1 --2- 172.21.2.102:0/655198727 >> v2:172.21.2.113:3300/0 conn(0x7f45601b0470 0x7f45601b0870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:06.737 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:06.722 7f455ffff700 1 -- 172.21.2.102:0/655198727 >> v1:172.21.2.113:6789/0 conn(0x7f45600f0030 legacy=0x7f45600f10d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:24:06.737 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:06.722 7f455ffff700 1 -- 172.21.2.102:0/655198727 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45600f0430 con 0x7f45600ef610 2021-02-10T11:24:06.737 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:06.722 7f455ffff700 1 --2- 172.21.2.102:0/655198727 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f45600ef610 0x7f45600efa10 secure :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 rx=0x7f4550006700 tx=0x7f4550009b80).ready entity=mon.0 client_cookie=ebc4939cbe5f84b6 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:06.737 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:06.722 7f455f7fe700 1 -- 172.21.2.102:0/655198727 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f455000c0c0 con 0x7f45600ef610 2021-02-10T11:24:06.737 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:06.722 7f455f7fe700 1 -- 172.21.2.102:0/655198727 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f455000c340 con 0x7f45600ef610 2021-02-10T11:24:06.737 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:06.722 7f455f7fe700 1 -- 172.21.2.102:0/655198727 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f455000c7a0 con 0x7f45600ef610 2021-02-10T11:24:06.737 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:06.722 7f4566f20700 1 -- 172.21.2.102:0/655198727 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f45600ef610 msgr2=0x7f45600efa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:06.737 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:06.722 7f4566f20700 1 --2- 172.21.2.102:0/655198727 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f45600ef610 0x7f45600efa10 secure :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 rx=0x7f4550006700 tx=0x7f4550009b80).stop 2021-02-10T11:24:06.737 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:06.723 7f4566f20700 1 -- 172.21.2.102:0/655198727 shutdown_connections 2021-02-10T11:24:06.737 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:06.723 7f4566f20700 1 --2- 172.21.2.102:0/655198727 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f45600ef610 0x7f45600efa10 unknown :-1 s=CLOSED pgs=1299 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:06.738 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:06.723 7f4566f20700 1 --2- 172.21.2.102:0/655198727 >> v2:172.21.2.113:3300/0 conn(0x7f45601b0470 0x7f45601b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:06.738 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:06.723 7f4566f20700 1 -- 172.21.2.102:0/655198727 >> 172.21.2.102:0/655198727 conn(0x7f45600ce3e0 msgr2=0x7f45600e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:06.738 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:06.723 7f4566f20700 1 -- 172.21.2.102:0/655198727 shutdown_connections 2021-02-10T11:24:06.738 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:06.723 7f4566f20700 1 -- 172.21.2.102:0/655198727 wait complete. 2021-02-10T11:24:06.738 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:06.737 7f4566f20700 1 Processor -- start 2021-02-10T11:24:06.738 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:06.737 7f4566f20700 1 -- start start 2021-02-10T11:24:06.739 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:06.738 7f4566f20700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f45600c36a0 0x7f45600c2b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:06.739 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:06.738 7f4566f20700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f45600c3190 0x7f45600b1d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:06.739 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:06.738 7f4566f20700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f45600b2370 0x7f45600b2d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:06.739 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:06.738 7f4566f20700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45600b27c0 con 0x7f45600c3190 2021-02-10T11:24:06.740 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:06.738 7f4566f20700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45600b3360 con 0x7f45600c36a0 2021-02-10T11:24:06.740 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:06.738 7f4566f20700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45600b3540 con 0x7f45600b2370 2021-02-10T11:24:06.740 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:06.738 7f455ffff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f45600c3190 0x7f45600b1d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:06.740 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:06.738 7f455ffff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f45600c3190 0x7f45600b1d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:56880/0 (socket says 172.21.2.102:56880) 2021-02-10T11:24:06.740 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:06.738 7f455ffff700 1 -- 172.21.2.102:0/3662318566 learned_addr learned my addr 172.21.2.102:0/3662318566 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:24:06.740 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:06.738 7f455ffff700 1 -- 172.21.2.102:0/3662318566 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f45600b2370 msgr2=0x7f45600b2d40 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:24:06.740 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:06.738 7f455ffff700 1 --2- 172.21.2.102:0/3662318566 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f45600b2370 0x7f45600b2d40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:06.740 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:06.738 7f455ffff700 1 -- 172.21.2.102:0/3662318566 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f45600c36a0 msgr2=0x7f45600c2b70 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:24:06.740 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:06.738 7f455ffff700 1 --2- 172.21.2.102:0/3662318566 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f45600c36a0 0x7f45600c2b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:06.740 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:06.738 7f455ffff700 1 -- 172.21.2.102:0/3662318566 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45600b3760 con 0x7f45600c3190 2021-02-10T11:24:06.740 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:06.739 7f455ffff700 1 --2- 172.21.2.102:0/3662318566 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f45600c3190 0x7f45600b1d50 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 rx=0x7f455000b5e0 tx=0x7f455000b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:06.740 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:06.739 7f455d7fa700 1 -- 172.21.2.102:0/3662318566 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f455000bd70 con 0x7f45600c3190 2021-02-10T11:24:06.741 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:06.739 7f455d7fa700 1 -- 172.21.2.102:0/3662318566 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4550002030 con 0x7f45600c3190 2021-02-10T11:24:06.741 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:06.739 7f455d7fa700 1 -- 172.21.2.102:0/3662318566 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4550002550 con 0x7f45600c3190 2021-02-10T11:24:06.741 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:06.739 7f4566f20700 1 -- 172.21.2.102:0/3662318566 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45600b3760 con 0x7f45600c3190 2021-02-10T11:24:06.741 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:06.739 7f4566f20700 1 -- 172.21.2.102:0/3662318566 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45600b3cf0 con 0x7f45600c3190 2021-02-10T11:24:06.741 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:06.740 7f455d7fa700 1 -- 172.21.2.102:0/3662318566 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4550000c90 con 0x7f45600c3190 2021-02-10T11:24:06.741 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:06.740 7f455d7fa700 1 --2- 172.21.2.102:0/3662318566 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f45440186c0 0x7f454401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:06.741 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:06.740 7f455d7fa700 1 -- 172.21.2.102:0/3662318566 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 4753+0+0 (secure 0 0 0) 0x7f4550032290 con 0x7f45600c3190 2021-02-10T11:24:06.741 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:06.740 7f4564d3c700 1 --2- 172.21.2.102:0/3662318566 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f45440186c0 0x7f454401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:06.741 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:06.740 7f4564d3c700 1 --2- 172.21.2.102:0/3662318566 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f45440186c0 0x7f454401abc0 crc :-1 s=READY pgs=609 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:06.769 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:06.771 7f4566f20700 1 -- 172.21.2.102:0/3662318566 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45600edd10 con 0x7f45600c3190 2021-02-10T11:24:06.772 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:06.774 7f455d7fa700 1 -- 172.21.2.102:0/3662318566 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f45500009b0 con 0x7f45600c3190 2021-02-10T11:24:06.898 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:06.899 7f4566f20700 1 -- 172.21.2.102:0/3662318566 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f45600e52e0 con 0x7f45600c3190 2021-02-10T11:24:06.898 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:06.900 7f455d7fa700 1 -- 172.21.2.102:0/3662318566 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v52) v1 ==== 92+0+10 (secure 0 0 0) 0x7f4550000c90 con 0x7f45600c3190 2021-02-10T11:24:06.899 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:24:06.900 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:06.902 7f4566f20700 1 -- 172.21.2.102:0/3662318566 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f45440186c0 msgr2=0x7f454401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:06.900 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:06.902 7f4566f20700 1 --2- 172.21.2.102:0/3662318566 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f45440186c0 0x7f454401abc0 crc :-1 s=READY pgs=609 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:06.900 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:06.902 7f4566f20700 1 -- 172.21.2.102:0/3662318566 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f45600c3190 msgr2=0x7f45600b1d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:06.900 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:06.902 7f4566f20700 1 --2- 172.21.2.102:0/3662318566 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f45600c3190 0x7f45600b1d50 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 rx=0x7f455000b5e0 tx=0x7f455000b980).stop 2021-02-10T11:24:06.901 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:06.902 7f4566f20700 1 -- 172.21.2.102:0/3662318566 shutdown_connections 2021-02-10T11:24:06.901 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:06.902 7f4566f20700 1 --2- 172.21.2.102:0/3662318566 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f45600b2370 0x7f45600b2d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:06.901 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:06.902 7f4566f20700 1 --2- 172.21.2.102:0/3662318566 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f45600c3190 0x7f45600b1d50 unknown :-1 s=CLOSED pgs=1300 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:06.901 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:06.902 7f4566f20700 1 --2- 172.21.2.102:0/3662318566 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f45440186c0 0x7f454401abc0 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:06.901 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:06.902 7f4566f20700 1 --2- 172.21.2.102:0/3662318566 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f45600c36a0 0x7f45600c2b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:06.901 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:06.902 7f4566f20700 1 -- 172.21.2.102:0/3662318566 >> 172.21.2.102:0/3662318566 conn(0x7f456009f630 msgr2=0x7f45600a0130 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:06.901 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:06.902 7f4566f20700 1 -- 172.21.2.102:0/3662318566 shutdown_connections 2021-02-10T11:24:06.901 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:06.902 7f4566f20700 1 -- 172.21.2.102:0/3662318566 wait complete. 2021-02-10T11:24:06.909 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --format=json-pretty osd lspools 2021-02-10T11:24:07.335 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:07.321 7f3ec6a49700 1 Processor -- start 2021-02-10T11:24:07.335 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:07.321 7f3ec6a49700 1 -- start start 2021-02-10T11:24:07.336 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:07.322 7f3ec6a49700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3ec00d17f0 0x7f3ec01b31e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:07.336 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:07.322 7f3ec6a49700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f3ec00d1bf0 0x7f3ec01b3800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:07.336 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:07.322 7f3ec6a49700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f3ec01b3e20 0x7f3ec00ceef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:07.336 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:07.322 7f3ec6a49700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ec00e5eb0 con 0x7f3ec00d1bf0 2021-02-10T11:24:07.336 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:07.322 7f3ec6a49700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ec00f0b20 con 0x7f3ec01b3e20 2021-02-10T11:24:07.336 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:07.322 7f3ec6a49700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ec00cf510 con 0x7f3ec00d17f0 2021-02-10T11:24:07.336 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:07.322 7f3ebffff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f3ec00d1bf0 0x7f3ec01b3800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:07.336 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:07.322 7f3ebffff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f3ec00d1bf0 0x7f3ec01b3800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:56892/0 (socket says 172.21.2.102:56892) 2021-02-10T11:24:07.336 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:07.322 7f3ebffff700 1 -- 172.21.2.102:0/382745682 learned_addr learned my addr 172.21.2.102:0/382745682 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:24:07.336 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:07.322 7f3ebffff700 1 -- 172.21.2.102:0/382745682 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3ec00d17f0 msgr2=0x7f3ec01b31e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:24:07.336 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:07.322 7f3ebffff700 1 --2- 172.21.2.102:0/382745682 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3ec00d17f0 0x7f3ec01b31e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:07.336 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:07.322 7f3ebffff700 1 -- 172.21.2.102:0/382745682 >> v2:172.21.2.113:3300/0 conn(0x7f3ec01b3e20 msgr2=0x7f3ec00ceef0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:24:07.336 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:07.322 7f3ebffff700 1 --2- 172.21.2.102:0/382745682 >> v2:172.21.2.113:3300/0 conn(0x7f3ec01b3e20 0x7f3ec00ceef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:07.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:07.322 7f3ebffff700 1 -- 172.21.2.102:0/382745682 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ec00cf730 con 0x7f3ec00d1bf0 2021-02-10T11:24:07.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:07.322 7f3ebffff700 1 --2- 172.21.2.102:0/382745682 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3ec00d1bf0 0x7f3ec01b3800 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 rx=0x7f3eb0006700 tx=0x7f3eb0009b80).ready entity=mon.0 client_cookie=e2e5472469b1b747 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:07.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:07.322 7f3ebf7fe700 1 -- 172.21.2.102:0/382745682 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3eb000c0c0 con 0x7f3ec00d1bf0 2021-02-10T11:24:07.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:07.322 7f3ebf7fe700 1 -- 172.21.2.102:0/382745682 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3eb000c340 con 0x7f3ec00d1bf0 2021-02-10T11:24:07.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:07.322 7f3ebf7fe700 1 -- 172.21.2.102:0/382745682 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3eb000c860 con 0x7f3ec00d1bf0 2021-02-10T11:24:07.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:07.322 7f3ec6a49700 1 -- 172.21.2.102:0/382745682 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3ec00d1bf0 msgr2=0x7f3ec01b3800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:07.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:07.322 7f3ec6a49700 1 --2- 172.21.2.102:0/382745682 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3ec00d1bf0 0x7f3ec01b3800 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 rx=0x7f3eb0006700 tx=0x7f3eb0009b80).stop 2021-02-10T11:24:07.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:07.323 7f3ec6a49700 1 -- 172.21.2.102:0/382745682 shutdown_connections 2021-02-10T11:24:07.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:07.323 7f3ec6a49700 1 --2- 172.21.2.102:0/382745682 >> v2:172.21.2.113:3300/0 conn(0x7f3ec01b3e20 0x7f3ec00ceef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:07.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:07.323 7f3ec6a49700 1 --2- 172.21.2.102:0/382745682 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3ec00d1bf0 0x7f3ec01b3800 unknown :-1 s=CLOSED pgs=1301 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:07.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:07.323 7f3ec6a49700 1 --2- 172.21.2.102:0/382745682 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3ec00d17f0 0x7f3ec01b31e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:07.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:07.323 7f3ec6a49700 1 -- 172.21.2.102:0/382745682 >> 172.21.2.102:0/382745682 conn(0x7f3ec00fde40 msgr2=0x7f3ec00ce390 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:07.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:07.323 7f3ec6a49700 1 -- 172.21.2.102:0/382745682 shutdown_connections 2021-02-10T11:24:07.338 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:07.323 7f3ec6a49700 1 -- 172.21.2.102:0/382745682 wait complete. 2021-02-10T11:24:07.339 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:07.337 7f3ec6a49700 1 Processor -- start 2021-02-10T11:24:07.339 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:07.337 7f3ec6a49700 1 -- start start 2021-02-10T11:24:07.339 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:07.338 7f3ec6a49700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3ec00906f0 0x7f3ec008fe60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:07.340 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:07.338 7f3ec6a49700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3ec01a3240 0x7f3ec00ae230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:07.340 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:07.338 7f3ec6a49700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3ec0090bc0 0x7f3ec0090fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:07.340 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:07.338 7f3ec6a49700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ec00ae850 con 0x7f3ec0090bc0 2021-02-10T11:24:07.340 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:07.338 7f3ec6a49700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ec00aea30 con 0x7f3ec01a3240 2021-02-10T11:24:07.340 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:07.338 7f3ec6a49700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ec009a0c0 con 0x7f3ec00906f0 2021-02-10T11:24:07.340 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:07.338 7f3ec5066700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3ec0090bc0 0x7f3ec0090fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:07.340 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:07.338 7f3ec5066700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3ec0090bc0 0x7f3ec0090fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:56898/0 (socket says 172.21.2.102:56898) 2021-02-10T11:24:07.340 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:07.338 7f3ec5066700 1 -- 172.21.2.102:0/1740137168 learned_addr learned my addr 172.21.2.102:0/1740137168 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:24:07.340 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:07.338 7f3ec5066700 1 -- 172.21.2.102:0/1740137168 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3ec00906f0 msgr2=0x7f3ec008fe60 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:24:07.340 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:07.338 7f3ec5066700 1 --2- 172.21.2.102:0/1740137168 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3ec00906f0 0x7f3ec008fe60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:07.340 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:07.338 7f3ec5066700 1 -- 172.21.2.102:0/1740137168 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3ec01a3240 msgr2=0x7f3ec00ae230 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:24:07.340 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:07.338 7f3ec5066700 1 --2- 172.21.2.102:0/1740137168 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3ec01a3240 0x7f3ec00ae230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:07.340 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:07.338 7f3ec5066700 1 -- 172.21.2.102:0/1740137168 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ec009a2e0 con 0x7f3ec0090bc0 2021-02-10T11:24:07.341 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:07.339 7f3ec5066700 1 --2- 172.21.2.102:0/1740137168 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3ec0090bc0 0x7f3ec0090fc0 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 rx=0x7f3eb8009960 tx=0x7f3eb800bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:07.341 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:07.339 7f3ebd7fa700 1 -- 172.21.2.102:0/1740137168 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3eb8000d30 con 0x7f3ec0090bc0 2021-02-10T11:24:07.341 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:07.339 7f3ebd7fa700 1 -- 172.21.2.102:0/1740137168 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3eb80030c0 con 0x7f3ec0090bc0 2021-02-10T11:24:07.341 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:07.339 7f3ec6a49700 1 -- 172.21.2.102:0/1740137168 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ec009a2e0 con 0x7f3ec0090bc0 2021-02-10T11:24:07.341 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:07.339 7f3ec6a49700 1 -- 172.21.2.102:0/1740137168 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ec009a870 con 0x7f3ec0090bc0 2021-02-10T11:24:07.341 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:07.339 7f3ebd7fa700 1 -- 172.21.2.102:0/1740137168 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3eb8001500 con 0x7f3ec0090bc0 2021-02-10T11:24:07.341 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:07.339 7f3ebd7fa700 1 -- 172.21.2.102:0/1740137168 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f3eb8001310 con 0x7f3ec0090bc0 2021-02-10T11:24:07.341 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:07.339 7f3ebd7fa700 1 --2- 172.21.2.102:0/1740137168 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3ea40186d0 0x7f3ea401abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:07.341 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:07.339 7f3ebd7fa700 1 -- 172.21.2.102:0/1740137168 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 4753+0+0 (secure 0 0 0) 0x7f3eb8032840 con 0x7f3ec0090bc0 2021-02-10T11:24:07.341 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:07.340 7f3ec4865700 1 --2- 172.21.2.102:0/1740137168 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3ea40186d0 0x7f3ea401abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:07.341 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:07.340 7f3ec4865700 1 --2- 172.21.2.102:0/1740137168 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3ea40186d0 0x7f3ea401abd0 crc :-1 s=READY pgs=610 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:07.370 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:07.372 7f3ec6a49700 1 -- 172.21.2.102:0/1740137168 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ec01036a0 con 0x7f3ec0090bc0 2021-02-10T11:24:07.372 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:07.374 7f3ebd7fa700 1 -- 172.21.2.102:0/1740137168 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f3eb800be30 con 0x7f3ec0090bc0 2021-02-10T11:24:07.494 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:07.496 7f3ec6a49700 1 -- 172.21.2.102:0/1740137168 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f3ec00f0b20 con 0x7f3ec0090bc0 2021-02-10T11:24:07.495 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:07.497 7f3ebd7fa700 1 -- 172.21.2.102:0/1740137168 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v52) v1 ==== 84+0+64 (secure 0 0 0) 0x7f3eb8001310 con 0x7f3ec0090bc0 2021-02-10T11:24:07.495 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:24:07.495 INFO:teuthology.orchestra.run.gibba002.stdout:[ 2021-02-10T11:24:07.496 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:24:07.496 INFO:teuthology.orchestra.run.gibba002.stdout: "poolnum": 1, 2021-02-10T11:24:07.496 INFO:teuthology.orchestra.run.gibba002.stdout: "poolname": "rbd" 2021-02-10T11:24:07.496 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:24:07.496 INFO:teuthology.orchestra.run.gibba002.stdout:] 2021-02-10T11:24:07.497 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:07.498 7f3ec6a49700 1 -- 172.21.2.102:0/1740137168 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3ea40186d0 msgr2=0x7f3ea401abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:07.497 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:07.498 7f3ec6a49700 1 --2- 172.21.2.102:0/1740137168 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3ea40186d0 0x7f3ea401abd0 crc :-1 s=READY pgs=610 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:07.497 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:07.498 7f3ec6a49700 1 -- 172.21.2.102:0/1740137168 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3ec0090bc0 msgr2=0x7f3ec0090fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:07.497 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:07.498 7f3ec6a49700 1 --2- 172.21.2.102:0/1740137168 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3ec0090bc0 0x7f3ec0090fc0 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 rx=0x7f3eb8009960 tx=0x7f3eb800bcf0).stop 2021-02-10T11:24:07.498 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:07.499 7f3ec6a49700 1 -- 172.21.2.102:0/1740137168 shutdown_connections 2021-02-10T11:24:07.498 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:07.499 7f3ec6a49700 1 --2- 172.21.2.102:0/1740137168 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3ec0090bc0 0x7f3ec0090fc0 unknown :-1 s=CLOSED pgs=1302 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:07.498 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:07.499 7f3ec6a49700 1 --2- 172.21.2.102:0/1740137168 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3ea40186d0 0x7f3ea401abd0 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:07.498 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:07.499 7f3ec6a49700 1 --2- 172.21.2.102:0/1740137168 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3ec01a3240 0x7f3ec00ae230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:07.498 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:07.499 7f3ec6a49700 1 --2- 172.21.2.102:0/1740137168 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3ec00906f0 0x7f3ec008fe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:07.498 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:07.499 7f3ec6a49700 1 -- 172.21.2.102:0/1740137168 >> 172.21.2.102:0/1740137168 conn(0x7f3ec018efb0 msgr2=0x7f3ec0198710 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:07.498 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:07.499 7f3ec6a49700 1 -- 172.21.2.102:0/1740137168 shutdown_connections 2021-02-10T11:24:07.498 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:07.499 7f3ec6a49700 1 -- 172.21.2.102:0/1740137168 wait complete. 2021-02-10T11:24:07.506 INFO:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2021-02-10T11:24:07.506 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool create cephfs_metadata 8 2021-02-10T11:24:07.936 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:07.922 7f72b8861700 1 Processor -- start 2021-02-10T11:24:07.937 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:07.922 7f72b8861700 1 -- start start 2021-02-10T11:24:07.937 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:07.922 7f72b8861700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f72b41b0470 0x7f72b41b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:07.937 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:07.922 7f72b8861700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f72b40f20e0 0x7f72b40f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:07.937 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:07.922 7f72b8861700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72b40f0b50 con 0x7f72b40ef610 2021-02-10T11:24:07.937 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:07.922 7f72b8861700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72b40f0820 con 0x7f72b40f20e0 2021-02-10T11:24:07.937 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:07.922 7f72b8861700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72b40f24e0 con 0x7f72b41b0470 2021-02-10T11:24:07.937 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:07.923 7f72b2e1d700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f72b40f20e0 0x7f72b40f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:07.937 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:07.923 7f72b2e1d700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f72b40f20e0 0x7f72b40f0200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:56902/0 (socket says 172.21.2.102:56902) 2021-02-10T11:24:07.937 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:07.923 7f72b2e1d700 1 -- 172.21.2.102:0/3126498762 learned_addr learned my addr 172.21.2.102:0/3126498762 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:24:07.937 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:07.923 7f72b2e1d700 1 -- 172.21.2.102:0/3126498762 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f72b41b0470 msgr2=0x7f72b41b0870 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:24:07.937 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:07.923 7f72b2e1d700 1 --2- 172.21.2.102:0/3126498762 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f72b41b0470 0x7f72b41b0870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:07.938 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:07.923 7f72b2e1d700 1 -- 172.21.2.102:0/3126498762 >> v1:172.21.2.113:6789/0 conn(0x7f72b40ef610 legacy=0x7f72b40efa10 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:24:07.938 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:07.923 7f72b2e1d700 1 -- 172.21.2.102:0/3126498762 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72b40e52e0 con 0x7f72b40f20e0 2021-02-10T11:24:07.938 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:07.923 7f72b2e1d700 1 --2- 172.21.2.102:0/3126498762 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f72b40f20e0 0x7f72b40f0200 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 rx=0x7f72a4006b80 tx=0x7f72a4009b90).ready entity=mon.0 client_cookie=b322c639937f7e0f server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:07.938 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:07.923 7f72b161a700 1 -- 172.21.2.102:0/3126498762 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f72a40022e0 con 0x7f72b40f20e0 2021-02-10T11:24:07.938 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:07.923 7f72b161a700 1 -- 172.21.2.102:0/3126498762 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72a4002560 con 0x7f72b40f20e0 2021-02-10T11:24:07.938 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:07.923 7f72b161a700 1 -- 172.21.2.102:0/3126498762 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f72a4002a80 con 0x7f72b40f20e0 2021-02-10T11:24:07.938 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:07.923 7f72b8861700 1 -- 172.21.2.102:0/3126498762 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f72b40f20e0 msgr2=0x7f72b40f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:07.938 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:07.923 7f72b8861700 1 --2- 172.21.2.102:0/3126498762 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f72b40f20e0 0x7f72b40f0200 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 rx=0x7f72a4006b80 tx=0x7f72a4009b90).stop 2021-02-10T11:24:07.938 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:07.924 7f72b8861700 1 -- 172.21.2.102:0/3126498762 shutdown_connections 2021-02-10T11:24:07.938 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:07.924 7f72b8861700 1 --2- 172.21.2.102:0/3126498762 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f72b40f20e0 0x7f72b40f0200 unknown :-1 s=CLOSED pgs=1303 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:07.938 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:07.924 7f72b8861700 1 --2- 172.21.2.102:0/3126498762 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f72b41b0470 0x7f72b41b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:07.938 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:07.924 7f72b8861700 1 -- 172.21.2.102:0/3126498762 >> 172.21.2.102:0/3126498762 conn(0x7f72b40ce3e0 msgr2=0x7f72b40e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:07.939 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:07.924 7f72b8861700 1 -- 172.21.2.102:0/3126498762 shutdown_connections 2021-02-10T11:24:07.939 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:07.924 7f72b8861700 1 -- 172.21.2.102:0/3126498762 wait complete. 2021-02-10T11:24:07.940 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:07.938 7f72b8861700 1 Processor -- start 2021-02-10T11:24:07.940 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:07.939 7f72b8861700 1 -- start start 2021-02-10T11:24:07.940 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:07.939 7f72b8861700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f72b4094fb0 0x7f72b409e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:07.940 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:07.939 7f72b8861700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f72b40c2be0 0x7f72b40c2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:07.940 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:07.939 7f72b8861700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f72b40c3600 0x7f72b40a9c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:07.941 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:07.939 7f72b8861700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72b409ebf0 con 0x7f72b4094fb0 2021-02-10T11:24:07.941 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:07.939 7f72b8861700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72b40aa2a0 con 0x7f72b40c3600 2021-02-10T11:24:07.941 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:07.939 7f72b8861700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72b40aa4c0 con 0x7f72b40c2be0 2021-02-10T11:24:07.941 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:07.939 7f72b261c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f72b4094fb0 0x7f72b409e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:07.941 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:07.939 7f72b261c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f72b4094fb0 0x7f72b409e5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:56906/0 (socket says 172.21.2.102:56906) 2021-02-10T11:24:07.941 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:07.939 7f72b261c700 1 -- 172.21.2.102:0/2296310021 learned_addr learned my addr 172.21.2.102:0/2296310021 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:24:07.941 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:07.940 7f72b261c700 1 -- 172.21.2.102:0/2296310021 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f72b40c2be0 msgr2=0x7f72b40c2fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:07.941 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:07.940 7f72b261c700 1 --2- 172.21.2.102:0/2296310021 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f72b40c2be0 0x7f72b40c2fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:07.941 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:07.940 7f72b261c700 1 -- 172.21.2.102:0/2296310021 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f72b40c3600 msgr2=0x7f72b40a9c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:07.941 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:07.940 7f72b261c700 1 --2- 172.21.2.102:0/2296310021 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f72b40c3600 0x7f72b40a9c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:07.941 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:07.940 7f72b261c700 1 -- 172.21.2.102:0/2296310021 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72b40aa6e0 con 0x7f72b4094fb0 2021-02-10T11:24:07.941 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:07.940 7f72b261c700 1 --2- 172.21.2.102:0/2296310021 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f72b4094fb0 0x7f72b409e5d0 secure :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 rx=0x7f72a800b5c0 tx=0x7f72a800b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:07.941 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:07.940 7f72a2ffd700 1 -- 172.21.2.102:0/2296310021 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f72a800bd50 con 0x7f72b4094fb0 2021-02-10T11:24:07.942 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:07.940 7f72a2ffd700 1 -- 172.21.2.102:0/2296310021 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72a8006030 con 0x7f72b4094fb0 2021-02-10T11:24:07.942 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:07.940 7f72a2ffd700 1 -- 172.21.2.102:0/2296310021 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f72a80054d0 con 0x7f72b4094fb0 2021-02-10T11:24:07.942 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:07.940 7f72b8861700 1 -- 172.21.2.102:0/2296310021 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72b40aa6e0 con 0x7f72b4094fb0 2021-02-10T11:24:07.942 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:07.940 7f72b8861700 1 -- 172.21.2.102:0/2296310021 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72b40aac70 con 0x7f72b4094fb0 2021-02-10T11:24:07.942 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:07.941 7f72a2ffd700 1 -- 172.21.2.102:0/2296310021 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f72a8006210 con 0x7f72b4094fb0 2021-02-10T11:24:07.942 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:07.941 7f72a2ffd700 1 --2- 172.21.2.102:0/2296310021 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f72980186c0 0x7f729801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:07.942 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:07.941 7f72a2ffd700 1 -- 172.21.2.102:0/2296310021 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 4753+0+0 (secure 0 0 0) 0x7f72a80320a0 con 0x7f72b4094fb0 2021-02-10T11:24:07.942 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:07.941 7f72b1e1b700 1 --2- 172.21.2.102:0/2296310021 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f72980186c0 0x7f729801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:07.942 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:07.941 7f72b1e1b700 1 --2- 172.21.2.102:0/2296310021 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f72980186c0 0x7f729801abc0 crc :-1 s=READY pgs=611 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:07.970 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:07.972 7f72b8861700 1 -- 172.21.2.102:0/2296310021 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72b40edd10 con 0x7f72b4094fb0 2021-02-10T11:24:07.973 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:07.975 7f72a2ffd700 1 -- 172.21.2.102:0/2296310021 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f72a8000c40 con 0x7f72b4094fb0 2021-02-10T11:24:08.102 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:08.103 7f72b8861700 1 -- 172.21.2.102:0/2296310021 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"} v 0) v1 -- 0x7f72b40e52e0 con 0x7f72b4094fb0 2021-02-10T11:24:08.529 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:08.531 7f72a2ffd700 1 -- 172.21.2.102:0/2296310021 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"}]=0 pool 'cephfs_metadata' created v53) v1 ==== 133+0+0 (secure 0 0 0) 0x7f72a8006210 con 0x7f72b4094fb0 2021-02-10T11:24:08.567 INFO:teuthology.orchestra.run.gibba002.stderr:pool 'cephfs_metadata' created 2021-02-10T11:24:08.570 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:08.571 7f72b8861700 1 -- 172.21.2.102:0/2296310021 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f72980186c0 msgr2=0x7f729801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:08.570 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:08.571 7f72b8861700 1 --2- 172.21.2.102:0/2296310021 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f72980186c0 0x7f729801abc0 crc :-1 s=READY pgs=611 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:08.570 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:08.571 7f72b8861700 1 -- 172.21.2.102:0/2296310021 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f72b4094fb0 msgr2=0x7f72b409e5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:08.570 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:08.571 7f72b8861700 1 --2- 172.21.2.102:0/2296310021 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f72b4094fb0 0x7f72b409e5d0 secure :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 rx=0x7f72a800b5c0 tx=0x7f72a800b960).stop 2021-02-10T11:24:08.570 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:08.571 7f72b8861700 1 -- 172.21.2.102:0/2296310021 shutdown_connections 2021-02-10T11:24:08.570 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:08.571 7f72b8861700 1 --2- 172.21.2.102:0/2296310021 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f72980186c0 0x7f729801abc0 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:08.571 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:08.571 7f72b8861700 1 --2- 172.21.2.102:0/2296310021 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f72b40c3600 0x7f72b40a9c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:08.571 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:08.571 7f72b8861700 1 --2- 172.21.2.102:0/2296310021 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f72b40c2be0 0x7f72b40c2fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:08.571 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:08.571 7f72b8861700 1 --2- 172.21.2.102:0/2296310021 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f72b4094fb0 0x7f72b409e5d0 unknown :-1 s=CLOSED pgs=1304 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:08.571 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:08.571 7f72b8861700 1 -- 172.21.2.102:0/2296310021 >> 172.21.2.102:0/2296310021 conn(0x7f72b409f630 msgr2=0x7f72b40a0130 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:08.571 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:08.572 7f72b8861700 1 -- 172.21.2.102:0/2296310021 shutdown_connections 2021-02-10T11:24:08.571 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:08.572 7f72b8861700 1 -- 172.21.2.102:0/2296310021 wait complete. 2021-02-10T11:24:08.578 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool create cephfs_data 8 2021-02-10T11:24:09.007 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:08.992 7f98d737a700 1 Processor -- start 2021-02-10T11:24:09.007 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:08.992 7f98d737a700 1 -- start start 2021-02-10T11:24:09.007 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:08.992 7f98d737a700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f98d01b0470 0x7f98d01b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:09.008 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:08.992 7f98d737a700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f98d00ef610 0x7f98d00efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:09.008 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:08.992 7f98d737a700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98d00f0b50 con 0x7f98d00f0030 2021-02-10T11:24:09.008 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:08.992 7f98d737a700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98d00e52e0 con 0x7f98d00ef610 2021-02-10T11:24:09.008 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:08.992 7f98d737a700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98d00e5490 con 0x7f98d01b0470 2021-02-10T11:24:09.008 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:08.992 7f98d5997700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f98d00f0030 0x7f98d00f10d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:49138/0 (socket says 172.21.2.102:49138) 2021-02-10T11:24:09.008 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:08.992 7f98d5997700 1 -- 172.21.2.102:0/85347627 learned_addr learned my addr 172.21.2.102:0/85347627 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:24:09.009 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:08.992 7f98d4995700 1 --2- 172.21.2.102:0/85347627 >> v2:172.21.2.102:3300/0 conn(0x7f98d00ef610 0x7f98d00efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:09.009 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:08.993 7f98d4995700 1 -- 172.21.2.102:0/85347627 >> v2:172.21.2.113:3300/0 conn(0x7f98d01b0470 msgr2=0x7f98d01b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:24:09.009 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:08.993 7f98d4995700 1 --2- 172.21.2.102:0/85347627 >> v2:172.21.2.113:3300/0 conn(0x7f98d01b0470 0x7f98d01b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:09.009 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:08.993 7f98d4995700 1 -- 172.21.2.102:0/85347627 >> v1:172.21.2.102:6789/0 conn(0x7f98d00f0030 legacy=0x7f98d00f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:09.009 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:08.993 7f98d4995700 1 -- 172.21.2.102:0/85347627 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98d00f0430 con 0x7f98d00ef610 2021-02-10T11:24:09.009 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:08.993 7f98d4995700 1 --2- 172.21.2.102:0/85347627 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f98d00ef610 0x7f98d00efa10 secure :-1 s=READY pgs=1306 cs=0 l=1 rev1=1 rx=0x7f98c0006650 tx=0x7f98c0009a80).ready entity=mon.0 client_cookie=317279f25f7f0d9e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:09.009 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:08.993 7f98c7fff700 1 -- 172.21.2.102:0/85347627 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f98c000c0c0 con 0x7f98d00ef610 2021-02-10T11:24:09.010 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:08.993 7f98c7fff700 1 -- 172.21.2.102:0/85347627 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98c000c340 con 0x7f98d00ef610 2021-02-10T11:24:09.010 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:08.993 7f98c7fff700 1 -- 172.21.2.102:0/85347627 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f98c000c860 con 0x7f98d00ef610 2021-02-10T11:24:09.010 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:08.993 7f98d737a700 1 -- 172.21.2.102:0/85347627 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f98d00ef610 msgr2=0x7f98d00efa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:09.010 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:08.993 7f98d737a700 1 --2- 172.21.2.102:0/85347627 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f98d00ef610 0x7f98d00efa10 secure :-1 s=READY pgs=1306 cs=0 l=1 rev1=1 rx=0x7f98c0006650 tx=0x7f98c0009a80).stop 2021-02-10T11:24:09.010 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:08.994 7f98d737a700 1 -- 172.21.2.102:0/85347627 shutdown_connections 2021-02-10T11:24:09.010 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:08.994 7f98d737a700 1 --2- 172.21.2.102:0/85347627 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f98d00ef610 0x7f98d00efa10 unknown :-1 s=CLOSED pgs=1306 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:09.010 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:08.994 7f98d737a700 1 --2- 172.21.2.102:0/85347627 >> v2:172.21.2.113:3300/0 conn(0x7f98d01b0470 0x7f98d01b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:09.010 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:08.994 7f98d737a700 1 -- 172.21.2.102:0/85347627 >> 172.21.2.102:0/85347627 conn(0x7f98d00ce3e0 msgr2=0x7f98d00e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:09.011 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:08.994 7f98d737a700 1 -- 172.21.2.102:0/85347627 shutdown_connections 2021-02-10T11:24:09.011 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:08.994 7f98d737a700 1 -- 172.21.2.102:0/85347627 wait complete. 2021-02-10T11:24:09.011 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:09.009 7f98d737a700 1 Processor -- start 2021-02-10T11:24:09.011 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:09.009 7f98d737a700 1 -- start start 2021-02-10T11:24:09.013 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:09.009 7f98d737a700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f98d00c36a0 0x7f98d00c2b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:09.013 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:09.009 7f98d737a700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f98d00c3190 0x7f98d00b1d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:09.013 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:09.009 7f98d737a700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f98d00b2370 0x7f98d00b2d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:09.013 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:09.009 7f98d737a700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98d00b27c0 con 0x7f98d00b2370 2021-02-10T11:24:09.014 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:09.009 7f98d737a700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98d00b3360 con 0x7f98d00c3190 2021-02-10T11:24:09.014 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:09.009 7f98d737a700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98d00b3540 con 0x7f98d00c36a0 2021-02-10T11:24:09.014 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:09.009 7f98d5997700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f98d00b2370 0x7f98d00b2d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:09.014 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:09.009 7f98d5997700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f98d00b2370 0x7f98d00b2d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:56918/0 (socket says 172.21.2.102:56918) 2021-02-10T11:24:09.014 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:09.009 7f98d5997700 1 -- 172.21.2.102:0/3525874579 learned_addr learned my addr 172.21.2.102:0/3525874579 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:24:09.014 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:09.009 7f98d5997700 1 -- 172.21.2.102:0/3525874579 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f98d00c36a0 msgr2=0x7f98d00c2b70 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:24:09.014 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:09.009 7f98d5997700 1 --2- 172.21.2.102:0/3525874579 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f98d00c36a0 0x7f98d00c2b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:09.014 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:09.009 7f98d5997700 1 -- 172.21.2.102:0/3525874579 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f98d00c3190 msgr2=0x7f98d00b1d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:09.014 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:09.010 7f98d5997700 1 --2- 172.21.2.102:0/3525874579 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f98d00c3190 0x7f98d00b1d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:09.014 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:09.010 7f98d5997700 1 -- 172.21.2.102:0/3525874579 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98d00b3760 con 0x7f98d00b2370 2021-02-10T11:24:09.014 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:09.010 7f98d5997700 1 --2- 172.21.2.102:0/3525874579 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f98d00b2370 0x7f98d00b2d40 secure :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 rx=0x7f98c800a8a0 tx=0x7f98c800ac40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:09.014 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:09.010 7f98c5ffb700 1 -- 172.21.2.102:0/3525874579 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f98c8001270 con 0x7f98d00b2370 2021-02-10T11:24:09.015 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:09.010 7f98c5ffb700 1 -- 172.21.2.102:0/3525874579 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98c80070b0 con 0x7f98d00b2370 2021-02-10T11:24:09.015 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:09.010 7f98c5ffb700 1 -- 172.21.2.102:0/3525874579 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f98c8007540 con 0x7f98d00b2370 2021-02-10T11:24:09.015 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:09.010 7f98d737a700 1 -- 172.21.2.102:0/3525874579 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98d00b3760 con 0x7f98d00b2370 2021-02-10T11:24:09.015 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:09.010 7f98d737a700 1 -- 172.21.2.102:0/3525874579 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98d00b3760 con 0x7f98d00b2370 2021-02-10T11:24:09.015 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:09.011 7f98c5ffb700 1 -- 172.21.2.102:0/3525874579 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f98c8007290 con 0x7f98d00b2370 2021-02-10T11:24:09.015 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:09.011 7f98c5ffb700 1 --2- 172.21.2.102:0/3525874579 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f98b00186c0 0x7f98b001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:09.015 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:09.011 7f98c5ffb700 1 -- 172.21.2.102:0/3525874579 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 5107+0+0 (secure 0 0 0) 0x7f98c80326e0 con 0x7f98d00b2370 2021-02-10T11:24:09.015 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:09.011 7f98d5196700 1 --2- 172.21.2.102:0/3525874579 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f98b00186c0 0x7f98b001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:09.015 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:09.011 7f98d5196700 1 --2- 172.21.2.102:0/3525874579 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f98b00186c0 0x7f98b001abc0 crc :-1 s=READY pgs=612 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:09.040 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:09.042 7f98d737a700 1 -- 172.21.2.102:0/3525874579 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98d00edd10 con 0x7f98d00b2370 2021-02-10T11:24:09.043 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:09.045 7f98c5ffb700 1 -- 172.21.2.102:0/3525874579 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f98c8007cb0 con 0x7f98d00b2370 2021-02-10T11:24:09.171 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:09.173 7f98d737a700 1 -- 172.21.2.102:0/3525874579 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_data"} v 0) v1 -- 0x7f98d00edd10 con 0x7f98d00b2370 2021-02-10T11:24:09.575 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:09.576 7f98c5ffb700 1 -- 172.21.2.102:0/3525874579 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_data"}]=0 pool 'cephfs_data' created v55) v1 ==== 125+0+0 (secure 0 0 0) 0x7f98c8007290 con 0x7f98d00b2370 2021-02-10T11:24:09.588 INFO:teuthology.orchestra.run.gibba002.stderr:pool 'cephfs_data' created 2021-02-10T11:24:09.589 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:09.591 7f98d737a700 1 -- 172.21.2.102:0/3525874579 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f98b00186c0 msgr2=0x7f98b001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:09.589 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:09.591 7f98d737a700 1 --2- 172.21.2.102:0/3525874579 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f98b00186c0 0x7f98b001abc0 crc :-1 s=READY pgs=612 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:09.590 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:09.591 7f98d737a700 1 -- 172.21.2.102:0/3525874579 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f98d00b2370 msgr2=0x7f98d00b2d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:09.590 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:09.591 7f98d737a700 1 --2- 172.21.2.102:0/3525874579 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f98d00b2370 0x7f98d00b2d40 secure :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 rx=0x7f98c800a8a0 tx=0x7f98c800ac40).stop 2021-02-10T11:24:09.590 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:09.591 7f98d737a700 1 -- 172.21.2.102:0/3525874579 shutdown_connections 2021-02-10T11:24:09.590 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:09.591 7f98d737a700 1 --2- 172.21.2.102:0/3525874579 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f98d00b2370 0x7f98d00b2d40 unknown :-1 s=CLOSED pgs=1307 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:09.590 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:09.591 7f98d737a700 1 --2- 172.21.2.102:0/3525874579 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f98b00186c0 0x7f98b001abc0 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:09.590 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:09.591 7f98d737a700 1 --2- 172.21.2.102:0/3525874579 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f98d00c3190 0x7f98d00b1d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:09.590 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:09.591 7f98d737a700 1 --2- 172.21.2.102:0/3525874579 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f98d00c36a0 0x7f98d00c2b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:09.590 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:09.591 7f98d737a700 1 -- 172.21.2.102:0/3525874579 >> 172.21.2.102:0/3525874579 conn(0x7f98d009f630 msgr2=0x7f98d00a0130 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:09.590 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:09.591 7f98d737a700 1 -- 172.21.2.102:0/3525874579 shutdown_connections 2021-02-10T11:24:09.590 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:09.591 7f98d737a700 1 -- 172.21.2.102:0/3525874579 wait complete. 2021-02-10T11:24:09.598 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2021-02-10T11:24:10.029 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:10.013 7fa22db1a700 1 Processor -- start 2021-02-10T11:24:10.029 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:10.013 7fa22db1a700 1 -- start start 2021-02-10T11:24:10.029 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:10.014 7fa22db1a700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fa2280d1c70 0x7fa2280cef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:10.029 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:10.014 7fa22db1a700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa2280cf580 0x7fa2280cf980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:10.030 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:10.014 7fa22db1a700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2280fe1c0 con 0x7fa2280d1870 2021-02-10T11:24:10.030 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:10.014 7fa22db1a700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2281b0cb0 con 0x7fa2280d1c70 2021-02-10T11:24:10.030 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:10.014 7fa22db1a700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa228109a60 con 0x7fa2280cf580 2021-02-10T11:24:10.030 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:10.014 7fa226ffd700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fa2280d1c70 0x7fa2280cef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:10.030 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:10.014 7fa226ffd700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fa2280d1c70 0x7fa2280cef60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:56928/0 (socket says 172.21.2.102:56928) 2021-02-10T11:24:10.030 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:10.014 7fa226ffd700 1 -- 172.21.2.102:0/1002453321 learned_addr learned my addr 172.21.2.102:0/1002453321 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:24:10.030 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:10.014 7fa227fff700 1 --2- 172.21.2.102:0/1002453321 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa2280cf580 0x7fa2280cf980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:10.030 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:10.015 7fa226ffd700 1 -- 172.21.2.102:0/1002453321 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa2280cf580 msgr2=0x7fa2280cf980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:24:10.030 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:10.015 7fa226ffd700 1 --2- 172.21.2.102:0/1002453321 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa2280cf580 0x7fa2280cf980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:10.030 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:10.015 7fa226ffd700 1 -- 172.21.2.102:0/1002453321 >> v1:172.21.2.113:6789/0 conn(0x7fa2280d1870 legacy=0x7fa2281b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:24:10.031 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:10.015 7fa226ffd700 1 -- 172.21.2.102:0/1002453321 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa228088020 con 0x7fa2280d1c70 2021-02-10T11:24:10.031 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:10.015 7fa227fff700 1 --2- 172.21.2.102:0/1002453321 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa2280cf580 0x7fa2280cf980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:24:10.031 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:10.015 7fa226ffd700 1 --2- 172.21.2.102:0/1002453321 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa2280d1c70 0x7fa2280cef60 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 rx=0x7fa210006700 tx=0x7fa210009b80).ready entity=mon.0 client_cookie=9c95e46af7393417 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:10.031 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:10.015 7fa2267fc700 1 -- 172.21.2.102:0/1002453321 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa21000c0c0 con 0x7fa2280d1c70 2021-02-10T11:24:10.031 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:10.015 7fa2267fc700 1 -- 172.21.2.102:0/1002453321 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa21000c340 con 0x7fa2280d1c70 2021-02-10T11:24:10.031 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:10.015 7fa2267fc700 1 -- 172.21.2.102:0/1002453321 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa21000c860 con 0x7fa2280d1c70 2021-02-10T11:24:10.031 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:10.015 7fa22db1a700 1 -- 172.21.2.102:0/1002453321 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa2280d1c70 msgr2=0x7fa2280cef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:10.031 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:10.015 7fa22db1a700 1 --2- 172.21.2.102:0/1002453321 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa2280d1c70 0x7fa2280cef60 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 rx=0x7fa210006700 tx=0x7fa210009b80).stop 2021-02-10T11:24:10.031 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:10.015 7fa22db1a700 1 -- 172.21.2.102:0/1002453321 shutdown_connections 2021-02-10T11:24:10.031 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:10.016 7fa22db1a700 1 --2- 172.21.2.102:0/1002453321 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa2280cf580 0x7fa2280cf980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:10.031 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:10.016 7fa22db1a700 1 --2- 172.21.2.102:0/1002453321 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa2280d1c70 0x7fa2280cef60 unknown :-1 s=CLOSED pgs=1308 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:10.031 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:10.016 7fa22db1a700 1 -- 172.21.2.102:0/1002453321 >> 172.21.2.102:0/1002453321 conn(0x7fa2280ce420 msgr2=0x7fa2280d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:10.031 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:10.016 7fa22db1a700 1 -- 172.21.2.102:0/1002453321 shutdown_connections 2021-02-10T11:24:10.032 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:10.016 7fa22db1a700 1 -- 172.21.2.102:0/1002453321 wait complete. 2021-02-10T11:24:10.032 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:10.030 7fa22db1a700 1 Processor -- start 2021-02-10T11:24:10.032 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:10.030 7fa22db1a700 1 -- start start 2021-02-10T11:24:10.032 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:10.031 7fa22db1a700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa2280b1670 0x7fa2280b1a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:10.032 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:10.031 7fa22db1a700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa2280b2090 0x7fa22808c710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:10.032 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:10.031 7fa22db1a700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa22808cd30 0x7fa22808d130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:10.032 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:10.031 7fa22db1a700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa22819b870 con 0x7fa2280b2090 2021-02-10T11:24:10.032 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:10.031 7fa22db1a700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa22819b400 con 0x7fa2280b1670 2021-02-10T11:24:10.032 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:10.031 7fa22db1a700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2280b2490 con 0x7fa22808cd30 2021-02-10T11:24:10.032 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:10.031 7fa226ffd700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa2280b2090 0x7fa22808c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:10.032 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:10.031 7fa226ffd700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa2280b2090 0x7fa22808c710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:56936/0 (socket says 172.21.2.102:56936) 2021-02-10T11:24:10.032 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:10.031 7fa226ffd700 1 -- 172.21.2.102:0/2862316749 learned_addr learned my addr 172.21.2.102:0/2862316749 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:24:10.033 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:10.031 7fa226ffd700 1 -- 172.21.2.102:0/2862316749 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa22808cd30 msgr2=0x7fa22808d130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:10.033 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:10.031 7fa226ffd700 1 --2- 172.21.2.102:0/2862316749 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa22808cd30 0x7fa22808d130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:10.033 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:10.031 7fa226ffd700 1 -- 172.21.2.102:0/2862316749 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa2280b1670 msgr2=0x7fa2280b1a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:10.034 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:10.031 7fa226ffd700 1 --2- 172.21.2.102:0/2862316749 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa2280b1670 0x7fa2280b1a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:10.034 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:10.031 7fa226ffd700 1 -- 172.21.2.102:0/2862316749 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa22808d750 con 0x7fa2280b2090 2021-02-10T11:24:10.034 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:10.032 7fa226ffd700 1 --2- 172.21.2.102:0/2862316749 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa2280b2090 0x7fa22808c710 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 rx=0x7fa210009720 tx=0x7fa21000b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:10.034 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:10.032 7fa21ffff700 1 -- 172.21.2.102:0/2862316749 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa2100090c0 con 0x7fa2280b2090 2021-02-10T11:24:10.034 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:10.032 7fa21ffff700 1 -- 172.21.2.102:0/2862316749 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa210000940 con 0x7fa2280b2090 2021-02-10T11:24:10.034 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:10.032 7fa22db1a700 1 -- 172.21.2.102:0/2862316749 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa22808d750 con 0x7fa2280b2090 2021-02-10T11:24:10.034 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:10.032 7fa21ffff700 1 -- 172.21.2.102:0/2862316749 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa210002220 con 0x7fa2280b2090 2021-02-10T11:24:10.034 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:10.032 7fa22db1a700 1 -- 172.21.2.102:0/2862316749 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa228097970 con 0x7fa2280b2090 2021-02-10T11:24:10.034 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:10.032 7fa21ffff700 1 -- 172.21.2.102:0/2862316749 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa210000c90 con 0x7fa2280b2090 2021-02-10T11:24:10.034 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:10.032 7fa21ffff700 1 --2- 172.21.2.102:0/2862316749 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa20c0186c0 0x7fa20c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:10.034 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:10.032 7fa21ffff700 1 -- 172.21.2.102:0/2862316749 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 5457+0+0 (secure 0 0 0) 0x7fa2100324a0 con 0x7fa2280b2090 2021-02-10T11:24:10.035 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:10.033 7fa2277fe700 1 --2- 172.21.2.102:0/2862316749 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa20c0186c0 0x7fa20c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:10.035 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:10.033 7fa2277fe700 1 --2- 172.21.2.102:0/2862316749 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa20c0186c0 0x7fa20c01abc0 crc :-1 s=READY pgs=615 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:10.062 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:10.064 7fa22db1a700 1 -- 172.21.2.102:0/2862316749 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2280828e0 con 0x7fa2280b2090 2021-02-10T11:24:10.065 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:10.067 7fa21ffff700 1 -- 172.21.2.102:0/2862316749 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fa2100063a0 con 0x7fa2280b2090 2021-02-10T11:24:10.195 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:10.197 7fa22db1a700 1 -- 172.21.2.102:0/2862316749 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"data": "cephfs_data", "prefix": "fs new", "force": true, "fs_name": "cephfs", "metadata": "cephfs_metadata"} v 0) v1 -- 0x7fa2280828e0 con 0x7fa2280b2090 2021-02-10T11:24:10.198 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:10.200 7fa21ffff700 1 -- 172.21.2.102:0/2862316749 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"data": "cephfs_data", "prefix": "fs new", "force": true, "fs_name": "cephfs", "metadata": "cephfs_metadata"}]=0 new fs with metadata pool 10 and data pool 11 v36) v1 ==== 189+0+0 (secure 0 0 0) 0x7fa210000c90 con 0x7fa2280b2090 2021-02-10T11:24:10.204 INFO:teuthology.orchestra.run.gibba002.stderr:new fs with metadata pool 10 and data pool 11 2021-02-10T11:24:10.205 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:10.206 7fa22db1a700 1 -- 172.21.2.102:0/2862316749 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa20c0186c0 msgr2=0x7fa20c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:10.205 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:10.206 7fa22db1a700 1 --2- 172.21.2.102:0/2862316749 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa20c0186c0 0x7fa20c01abc0 crc :-1 s=READY pgs=615 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:10.205 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:10.206 7fa22db1a700 1 -- 172.21.2.102:0/2862316749 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa2280b2090 msgr2=0x7fa22808c710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:10.205 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:10.206 7fa22db1a700 1 --2- 172.21.2.102:0/2862316749 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa2280b2090 0x7fa22808c710 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 rx=0x7fa210009720 tx=0x7fa21000b960).stop 2021-02-10T11:24:10.205 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:10.206 7fa22db1a700 1 -- 172.21.2.102:0/2862316749 shutdown_connections 2021-02-10T11:24:10.205 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:10.206 7fa22db1a700 1 --2- 172.21.2.102:0/2862316749 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa22808cd30 0x7fa22808d130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:10.205 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:10.206 7fa22db1a700 1 --2- 172.21.2.102:0/2862316749 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa2280b2090 0x7fa22808c710 secure :-1 s=CLOSED pgs=1309 cs=0 l=1 rev1=1 rx=0x7fa210009720 tx=0x7fa21000b960).stop 2021-02-10T11:24:10.205 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:10.206 7fa22db1a700 1 --2- 172.21.2.102:0/2862316749 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa20c0186c0 0x7fa20c01abc0 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:10.206 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:10.206 7fa22db1a700 1 --2- 172.21.2.102:0/2862316749 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa2280b1670 0x7fa2280b1a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:10.206 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:10.206 7fa22db1a700 1 -- 172.21.2.102:0/2862316749 >> 172.21.2.102:0/2862316749 conn(0x7fa2280844f0 msgr2=0x7fa22819bf80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:10.206 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:10.206 7fa22db1a700 1 -- 172.21.2.102:0/2862316749 shutdown_connections 2021-02-10T11:24:10.206 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:10.206 7fa22db1a700 1 -- 172.21.2.102:0/2862316749 wait complete. 2021-02-10T11:24:10.213 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:24:10.638 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:10.624 7f815d637700 1 Processor -- start 2021-02-10T11:24:10.638 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:10.624 7f815d637700 1 -- start start 2021-02-10T11:24:10.639 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:10.624 7f815d637700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f815807e6a0 0x7f81580d1790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:10.639 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:10.624 7f815d637700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f815807ed80 con 0x7f81580e5070 2021-02-10T11:24:10.639 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:10.624 7f815d637700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f815807eaa0 con 0x7f81580e5470 2021-02-10T11:24:10.639 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:10.624 7f815d637700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81581b4080 con 0x7f815807e6a0 2021-02-10T11:24:10.639 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:10.624 7f81572bf700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f81580e5070 0x7f81581b3820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:49164/0 (socket says 172.21.2.102:49164) 2021-02-10T11:24:10.639 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:10.624 7f8157ac0700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f815807e6a0 0x7f81580d1790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:10.639 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:10.624 7f81572bf700 1 -- 172.21.2.102:0/930707922 learned_addr learned my addr 172.21.2.102:0/930707922 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:24:10.639 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:10.625 7f81562bd700 1 -- 172.21.2.102:0/930707922 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1286670182 0 0) 0x7f81400010b0 con 0x7f81580e5070 2021-02-10T11:24:10.639 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:10.625 7f81562bd700 1 -- 172.21.2.102:0/930707922 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8144002fe0 con 0x7f81580e5070 2021-02-10T11:24:10.639 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:10.625 7f8157ac0700 1 -- 172.21.2.102:0/930707922 >> v1:172.21.2.113:6789/0 conn(0x7f81580e5470 legacy=0x7f815807f300 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:24:10.639 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:10.625 7f8157ac0700 1 -- 172.21.2.102:0/930707922 >> v1:172.21.2.102:6789/0 conn(0x7f81580e5070 legacy=0x7f81581b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:10.639 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:10.625 7f8157ac0700 1 -- 172.21.2.102:0/930707922 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81580d1db0 con 0x7f815807e6a0 2021-02-10T11:24:10.640 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:10.625 7f8157ac0700 1 --2- 172.21.2.102:0/930707922 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f815807e6a0 0x7f81580d1790 secure :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 rx=0x7f8150006e30 tx=0x7f8150009e60).ready entity=mon.0 client_cookie=e981d7d971007307 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:10.640 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:10.625 7f81562bd700 1 -- 172.21.2.102:0/930707922 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8150006a50 con 0x7f815807e6a0 2021-02-10T11:24:10.640 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:10.625 7f81562bd700 1 -- 172.21.2.102:0/930707922 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f815000b300 con 0x7f815807e6a0 2021-02-10T11:24:10.640 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:10.625 7f81562bd700 1 -- 172.21.2.102:0/930707922 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f815000b820 con 0x7f815807e6a0 2021-02-10T11:24:10.640 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:10.625 7f815d637700 1 -- 172.21.2.102:0/930707922 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f815807e6a0 msgr2=0x7f81580d1790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:10.640 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:10.625 7f815d637700 1 --2- 172.21.2.102:0/930707922 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f815807e6a0 0x7f81580d1790 secure :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 rx=0x7f8150006e30 tx=0x7f8150009e60).stop 2021-02-10T11:24:10.640 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:10.626 7f815d637700 1 -- 172.21.2.102:0/930707922 shutdown_connections 2021-02-10T11:24:10.640 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:10.626 7f815d637700 1 --2- 172.21.2.102:0/930707922 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f815807e6a0 0x7f81580d1790 unknown :-1 s=CLOSED pgs=1311 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:10.640 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:10.626 7f815d637700 1 -- 172.21.2.102:0/930707922 >> 172.21.2.102:0/930707922 conn(0x7f81580ce690 msgr2=0x7f81580e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:10.640 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:10.626 7f815d637700 1 -- 172.21.2.102:0/930707922 shutdown_connections 2021-02-10T11:24:10.640 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:10.626 7f815d637700 1 -- 172.21.2.102:0/930707922 wait complete. 2021-02-10T11:24:10.640 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:10.640 7f815d637700 1 Processor -- start 2021-02-10T11:24:10.642 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:10.640 7f815d637700 1 -- start start 2021-02-10T11:24:10.642 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:10.641 7f815d637700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f815808c830 0x7f815808d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:10.642 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:10.641 7f815d637700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f81581afe20 0x7f81580b2f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:10.642 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:10.641 7f815d637700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f81581b0220 0x7f81580b3550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:10.642 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:10.641 7f815d637700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f815808cc30 con 0x7f81581afe20 2021-02-10T11:24:10.642 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:10.641 7f815d637700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81580b3b70 con 0x7f81581b0220 2021-02-10T11:24:10.642 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:10.641 7f815d637700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81580b3d90 con 0x7f815808c830 2021-02-10T11:24:10.642 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:10.641 7f8156abe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f81581afe20 0x7f81580b2f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:10.642 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:10.641 7f8156abe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f81581afe20 0x7f81580b2f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:56948/0 (socket says 172.21.2.102:56948) 2021-02-10T11:24:10.643 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:10.641 7f8156abe700 1 -- 172.21.2.102:0/3733717981 learned_addr learned my addr 172.21.2.102:0/3733717981 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:24:10.643 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:10.641 7f8156abe700 1 -- 172.21.2.102:0/3733717981 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f815808c830 msgr2=0x7f815808d0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:10.643 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:10.641 7f8156abe700 1 --2- 172.21.2.102:0/3733717981 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f815808c830 0x7f815808d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:10.643 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:10.641 7f8156abe700 1 -- 172.21.2.102:0/3733717981 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f81581b0220 msgr2=0x7f81580b3550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:10.643 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:10.641 7f8156abe700 1 --2- 172.21.2.102:0/3733717981 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f81581b0220 0x7f81580b3550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:10.643 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:10.641 7f8156abe700 1 -- 172.21.2.102:0/3733717981 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81581b0c00 con 0x7f81581afe20 2021-02-10T11:24:10.643 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:10.641 7f8156abe700 1 --2- 172.21.2.102:0/3733717981 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f81581afe20 0x7f81580b2f30 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 rx=0x7f814800b530 tx=0x7f814800b8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:10.643 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:10.642 7f814ffff700 1 -- 172.21.2.102:0/3733717981 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f814800bcc0 con 0x7f81581afe20 2021-02-10T11:24:10.643 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:10.642 7f814ffff700 1 -- 172.21.2.102:0/3733717981 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8148006020 con 0x7f81581afe20 2021-02-10T11:24:10.643 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:10.642 7f815d637700 1 -- 172.21.2.102:0/3733717981 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81581b0c00 con 0x7f81581afe20 2021-02-10T11:24:10.643 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:10.642 7f815d637700 1 -- 172.21.2.102:0/3733717981 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81581b1150 con 0x7f81581afe20 2021-02-10T11:24:10.643 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:10.642 7f814ffff700 1 -- 172.21.2.102:0/3733717981 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f81480054a0 con 0x7f81581afe20 2021-02-10T11:24:10.643 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:10.642 7f814ffff700 1 -- 172.21.2.102:0/3733717981 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f8148006260 con 0x7f81581afe20 2021-02-10T11:24:10.643 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:10.642 7f814ffff700 1 --2- 172.21.2.102:0/3733717981 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f813c0186c0 0x7f813c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:10.643 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:10.642 7f814ffff700 1 -- 172.21.2.102:0/3733717981 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f81480325c0 con 0x7f81581afe20 2021-02-10T11:24:10.644 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:10.642 7f81572bf700 1 --2- 172.21.2.102:0/3733717981 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f813c0186c0 0x7f813c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:10.644 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:10.643 7f81572bf700 1 --2- 172.21.2.102:0/3733717981 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f813c0186c0 0x7f813c01abc0 crc :-1 s=READY pgs=616 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:10.672 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:10.674 7f815d637700 1 -- 172.21.2.102:0/3733717981 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8158083db0 con 0x7f81581afe20 2021-02-10T11:24:10.675 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:10.677 7f814ffff700 1 -- 172.21.2.102:0/3733717981 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f814802cf40 con 0x7f81581afe20 2021-02-10T11:24:10.799 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:10.800 7f815d637700 1 -- 172.21.2.102:0/3733717981 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f815807ed80 con 0x7f81581afe20 2021-02-10T11:24:10.799 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:10.801 7f814ffff700 1 -- 172.21.2.102:0/3733717981 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v56) v1 ==== 74+0+14009 (secure 0 0 0) 0x7f814802cf40 con 0x7f81581afe20 2021-02-10T11:24:10.800 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:24:10.800 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":56,"fsid":"68db30ea-dbce-4aef-a6df-f01c6fea0513","created":"2021-02-10 11:03:06.331557","modified":"2021-02-10 11:24:10.198528","last_up_change":"2021-02-10 11:03:32.690562","last_in_change":"2021-02-10 11:03:32.690562","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:03:37.322519","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:24:08.122188","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":11,"pool_name":"cephfs_data","create_time":"2021-02-10 11:24:09.191606","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"be691b4d-a039-408a-af80-9d53f02b6f1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6818","nonce":36088},{"type":"v1","addr":"172.21.2.102:6820","nonce":36088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6822","nonce":36088},{"type":"v1","addr":"172.21.2.102:6824","nonce":36088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6830","nonce":36088},{"type":"v1","addr":"172.21.2.102:6832","nonce":36088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6826","nonce":36088},{"type":"v1","addr":"172.21.2.102:6828","nonce":36088}]},"public_addr":"172.21.2.102:6820/36088","cluster_addr":"172.21.2.102:6824/36088","heartbeat_back_addr":"172.21.2.102:6832/36088","heartbeat_front_addr":"172.21.2.102:6828/36088","state":["exists","up"]},{"osd":1,"uuid":"64e3ccc8-9c69-4d7c-865e-af8e83822635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6819","nonce":36087},{"type":"v1","addr":"172.21.2.102:6821","nonce":36087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6823","nonce":36087},{"type":"v1","addr":"172.21.2.102:6825","nonce":36087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6831","nonce":36087},{"type":"v1","addr":"172.21.2.102:6833","nonce":36087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6827","nonce":36087},{"type":"v1","addr":"172.21.2.102:6829","nonce":36087}]},"public_addr":"172.21.2.102:6821/36087","cluster_addr":"172.21.2.102:6825/36087","heartbeat_back_addr":"172.21.2.102:6833/36087","heartbeat_front_addr":"172.21.2.102:6829/36087","state":["exists","up"]},{"osd":2,"uuid":"153700af-747f-481f-bdf4-d6072d3a842f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6810","nonce":36089},{"type":"v1","addr":"172.21.2.102:6811","nonce":36089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6812","nonce":36089},{"type":"v1","addr":"172.21.2.102:6813","nonce":36089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6816","nonce":36089},{"type":"v1","addr":"172.21.2.102:6817","nonce":36089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6814","nonce":36089},{"type":"v1","addr":"172.21.2.102:6815","nonce":36089}]},"public_addr":"172.21.2.102:6811/36089","cluster_addr":"172.21.2.102:6813/36089","heartbeat_back_addr":"172.21.2.102:6817/36089","heartbeat_front_addr":"172.21.2.102:6815/36089","state":["exists","up"]},{"osd":3,"uuid":"cbc678f4-83a9-45e6-8830-2f7c232b1ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6802","nonce":36090},{"type":"v1","addr":"172.21.2.102:6803","nonce":36090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6804","nonce":36090},{"type":"v1","addr":"172.21.2.102:6805","nonce":36090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6808","nonce":36090},{"type":"v1","addr":"172.21.2.102:6809","nonce":36090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6806","nonce":36090},{"type":"v1","addr":"172.21.2.102:6807","nonce":36090}]},"public_addr":"172.21.2.102:6803/36090","cluster_addr":"172.21.2.102:6805/36090","heartbeat_back_addr":"172.21.2.102:6809/36090","heartbeat_front_addr":"172.21.2.102:6807/36090","state":["exists","up"]},{"osd":4,"uuid":"c01e4008-b258-4f10-afd2-9e7f37214351","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6808","nonce":36368},{"type":"v1","addr":"172.21.2.113:6811","nonce":36368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6814","nonce":36368},{"type":"v1","addr":"172.21.2.113:6818","nonce":36368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6826","nonce":36368},{"type":"v1","addr":"172.21.2.113:6828","nonce":36368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6820","nonce":36368},{"type":"v1","addr":"172.21.2.113:6824","nonce":36368}]},"public_addr":"172.21.2.113:6811/36368","cluster_addr":"172.21.2.113:6818/36368","heartbeat_back_addr":"172.21.2.113:6828/36368","heartbeat_front_addr":"172.21.2.113:6824/36368","state":["exists","up"]},{"osd":5,"uuid":"551f6007-daf8-4a98-8ee0-c2b16bcccb5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6802","nonce":36366},{"type":"v1","addr":"172.21.2.113:6805","nonce":36366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6807","nonce":36366},{"type":"v1","addr":"172.21.2.113:6810","nonce":36366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6819","nonce":36366},{"type":"v1","addr":"172.21.2.113:6822","nonce":36366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6812","nonce":36366},{"type":"v1","addr":"172.21.2.113:6815","nonce":36366}]},"public_addr":"172.21.2.113:6805/36366","cluster_addr":"172.21.2.113:6810/36366","heartbeat_back_addr":"172.21.2.113:6822/36366","heartbeat_front_addr":"172.21.2.113:6815/36366","state":["exists","up"]},{"osd":6,"uuid":"bf67c040-0e50-48be-b102-c3943bc9ebef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6800","nonce":36367},{"type":"v1","addr":"172.21.2.113:6801","nonce":36367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6803","nonce":36367},{"type":"v1","addr":"172.21.2.113:6804","nonce":36367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6813","nonce":36367},{"type":"v1","addr":"172.21.2.113:6816","nonce":36367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6806","nonce":36367},{"type":"v1","addr":"172.21.2.113:6809","nonce":36367}]},"public_addr":"172.21.2.113:6801/36367","cluster_addr":"172.21.2.113:6804/36367","heartbeat_back_addr":"172.21.2.113:6816/36367","heartbeat_front_addr":"172.21.2.113:6809/36367","state":["exists","up"]},{"osd":7,"uuid":"f5461f80-018d-428f-be89-2eb2bcbe7bb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6817","nonce":36365},{"type":"v1","addr":"172.21.2.113:6821","nonce":36365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6823","nonce":36365},{"type":"v1","addr":"172.21.2.113:6825","nonce":36365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6830","nonce":36365},{"type":"v1","addr":"172.21.2.113:6831","nonce":36365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6827","nonce":36365},{"type":"v1","addr":"172.21.2.113:6829","nonce":36365}]},"public_addr":"172.21.2.113:6821/36365","cluster_addr":"172.21.2.113:6825/36365","heartbeat_back_addr":"172.21.2.113:6831/36365","heartbeat_front_addr":"172.21.2.113:6829/36365","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:24:10.801 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:10.803 7f815d637700 1 -- 172.21.2.102:0/3733717981 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f813c0186c0 msgr2=0x7f813c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:10.801 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:10.803 7f815d637700 1 --2- 172.21.2.102:0/3733717981 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f813c0186c0 0x7f813c01abc0 crc :-1 s=READY pgs=616 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:10.801 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:10.803 7f815d637700 1 -- 172.21.2.102:0/3733717981 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f81581afe20 msgr2=0x7f81580b2f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:10.801 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:10.803 7f815d637700 1 --2- 172.21.2.102:0/3733717981 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f81581afe20 0x7f81580b2f30 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 rx=0x7f814800b530 tx=0x7f814800b8d0).stop 2021-02-10T11:24:10.802 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:10.803 7f815d637700 1 -- 172.21.2.102:0/3733717981 shutdown_connections 2021-02-10T11:24:10.802 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:10.803 7f815d637700 1 --2- 172.21.2.102:0/3733717981 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f813c0186c0 0x7f813c01abc0 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:10.802 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:10.803 7f815d637700 1 --2- 172.21.2.102:0/3733717981 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f81581b0220 0x7f81580b3550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:10.802 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:10.803 7f815d637700 1 --2- 172.21.2.102:0/3733717981 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f81581afe20 0x7f81580b2f30 unknown :-1 s=CLOSED pgs=1312 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:10.802 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:10.803 7f815d637700 1 --2- 172.21.2.102:0/3733717981 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f815808c830 0x7f815808d0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:10.802 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:10.803 7f815d637700 1 -- 172.21.2.102:0/3733717981 >> 172.21.2.102:0/3733717981 conn(0x7f81580937a0 msgr2=0x7f8158197e80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:10.802 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:10.803 7f815d637700 1 -- 172.21.2.102:0/3733717981 shutdown_connections 2021-02-10T11:24:10.802 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:10.803 7f815d637700 1 -- 172.21.2.102:0/3733717981 wait complete. 2021-02-10T11:24:10.810 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:24:11.236 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:11.221 7fa0fe31f700 1 Processor -- start 2021-02-10T11:24:11.236 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:11.221 7fa0fe31f700 1 -- start start 2021-02-10T11:24:11.236 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:11.222 7fa0fe31f700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa0f80e5470 0x7fa0f807e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:11.236 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:11.222 7fa0fe31f700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fa0f807f300 0x7fa0f807f700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:11.236 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:11.222 7fa0fe31f700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0f807ed80 con 0x7fa0f80e5070 2021-02-10T11:24:11.236 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:11.222 7fa0fe31f700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0f81b4010 con 0x7fa0f807f300 2021-02-10T11:24:11.236 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:11.222 7fa0fe31f700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0f80d1790 con 0x7fa0f80e5470 2021-02-10T11:24:11.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:11.222 7fa0fc93c700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fa0f807f300 0x7fa0f807f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:11.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:11.222 7fa0f7fff700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fa0f80e5070 0x7fa0f81b3820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:49176/0 (socket says 172.21.2.102:49176) 2021-02-10T11:24:11.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:11.222 7fa0f7fff700 1 -- 172.21.2.102:0/2956748174 learned_addr learned my addr 172.21.2.102:0/2956748174 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:24:11.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:11.222 7fa0fc93c700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fa0f807f300 0x7fa0f807f700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:56956/0 (socket says 172.21.2.102:56956) 2021-02-10T11:24:11.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:11.222 7fa0f77fe700 1 --2- 172.21.2.102:0/2956748174 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa0f80e5470 0x7fa0f807e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:11.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:11.222 7fa0fc93c700 1 -- 172.21.2.102:0/2956748174 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa0f80e5470 msgr2=0x7fa0f807e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:24:11.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:11.222 7fa0fc93c700 1 --2- 172.21.2.102:0/2956748174 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa0f80e5470 0x7fa0f807e6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:11.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:11.222 7fa0fc93c700 1 -- 172.21.2.102:0/2956748174 >> v1:172.21.2.102:6789/0 conn(0x7fa0f80e5070 legacy=0x7fa0f81b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:11.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:11.222 7fa0fc93c700 1 -- 172.21.2.102:0/2956748174 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0f80d1980 con 0x7fa0f807f300 2021-02-10T11:24:11.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:11.222 7fa0fc93c700 1 --2- 172.21.2.102:0/2956748174 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa0f807f300 0x7fa0f807f700 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 rx=0x7fa0f0006ab0 tx=0x7fa0f0009ac0).ready entity=mon.0 client_cookie=f81882c6c2057325 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:11.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:11.222 7fa0f6ffd700 1 -- 172.21.2.102:0/2956748174 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa0f00066d0 con 0x7fa0f807f300 2021-02-10T11:24:11.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:11.223 7fa0f6ffd700 1 -- 172.21.2.102:0/2956748174 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0f00022e0 con 0x7fa0f807f300 2021-02-10T11:24:11.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:11.223 7fa0f6ffd700 1 -- 172.21.2.102:0/2956748174 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa0f0002740 con 0x7fa0f807f300 2021-02-10T11:24:11.238 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:11.223 7fa0fe31f700 1 -- 172.21.2.102:0/2956748174 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa0f807f300 msgr2=0x7fa0f807f700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:11.238 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:11.223 7fa0fe31f700 1 --2- 172.21.2.102:0/2956748174 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa0f807f300 0x7fa0f807f700 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 rx=0x7fa0f0006ab0 tx=0x7fa0f0009ac0).stop 2021-02-10T11:24:11.238 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:11.223 7fa0fe31f700 1 -- 172.21.2.102:0/2956748174 shutdown_connections 2021-02-10T11:24:11.238 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:11.223 7fa0fe31f700 1 --2- 172.21.2.102:0/2956748174 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa0f807f300 0x7fa0f807f700 unknown :-1 s=CLOSED pgs=1314 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:11.238 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:11.223 7fa0fe31f700 1 --2- 172.21.2.102:0/2956748174 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa0f80e5470 0x7fa0f807e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:11.238 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:11.223 7fa0fe31f700 1 -- 172.21.2.102:0/2956748174 >> 172.21.2.102:0/2956748174 conn(0x7fa0f80ce690 msgr2=0x7fa0f80e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:11.238 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:11.223 7fa0fe31f700 1 -- 172.21.2.102:0/2956748174 shutdown_connections 2021-02-10T11:24:11.238 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:11.223 7fa0fe31f700 1 -- 172.21.2.102:0/2956748174 wait complete. 2021-02-10T11:24:11.240 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:11.238 7fa0fe31f700 1 Processor -- start 2021-02-10T11:24:11.240 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:11.238 7fa0fe31f700 1 -- start start 2021-02-10T11:24:11.240 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:11.238 7fa0fe31f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa0f808d400 0x7fa0f8081230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:11.240 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:11.238 7fa0fe31f700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa0f8081850 0x7fa0f80acf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:11.240 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:11.238 7fa0fe31f700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa0f8081c50 0x7fa0f80ad5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:11.240 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:11.238 7fa0fe31f700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0f8082110 con 0x7fa0f808d400 2021-02-10T11:24:11.240 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:11.238 7fa0fe31f700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0f808d800 con 0x7fa0f8081850 2021-02-10T11:24:11.240 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:11.238 7fa0fe31f700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0f80adbc0 con 0x7fa0f8081c50 2021-02-10T11:24:11.240 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:11.239 7fa0f7fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa0f808d400 0x7fa0f8081230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:11.240 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:11.239 7fa0fc93c700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa0f8081c50 0x7fa0f80ad5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:11.241 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:11.239 7fa0fc93c700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa0f8081c50 0x7fa0f80ad5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.113:3301/0 says I am v2:172.21.2.102:34266/0 (socket says 172.21.2.102:34266) 2021-02-10T11:24:11.241 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:11.239 7fa0f7fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa0f808d400 0x7fa0f8081230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:56958/0 (socket says 172.21.2.102:56958) 2021-02-10T11:24:11.241 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:11.239 7fa0fc93c700 1 -- 172.21.2.102:0/2477413429 learned_addr learned my addr 172.21.2.102:0/2477413429 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:24:11.241 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:11.239 7fa0f77fe700 1 --2- 172.21.2.102:0/2477413429 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa0f8081850 0x7fa0f80acf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:11.241 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:11.239 7fa0fc93c700 1 -- 172.21.2.102:0/2477413429 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa0f8081850 msgr2=0x7fa0f80acf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:11.241 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:11.239 7fa0fc93c700 1 --2- 172.21.2.102:0/2477413429 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa0f8081850 0x7fa0f80acf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:11.241 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:11.239 7fa0fc93c700 1 -- 172.21.2.102:0/2477413429 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa0f808d400 msgr2=0x7fa0f8081230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:11.241 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:11.239 7fa0fc93c700 1 --2- 172.21.2.102:0/2477413429 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa0f808d400 0x7fa0f8081230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:11.241 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:11.239 7fa0fc93c700 1 -- 172.21.2.102:0/2477413429 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0f8187fc0 con 0x7fa0f8081c50 2021-02-10T11:24:11.241 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:11.239 7fa0fc93c700 1 --2- 172.21.2.102:0/2477413429 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa0f8081c50 0x7fa0f80ad5a0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7fa0f000afc0 tx=0x7fa0f0002850).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:11.241 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:11.239 7fa0f4ff9700 1 -- 172.21.2.102:0/2477413429 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa0f0009390 con 0x7fa0f8081c50 2021-02-10T11:24:11.241 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:11.239 7fa0f4ff9700 1 -- 172.21.2.102:0/2477413429 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0f000a070 con 0x7fa0f8081c50 2021-02-10T11:24:11.241 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:11.239 7fa0f4ff9700 1 -- 172.21.2.102:0/2477413429 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa0f000a590 con 0x7fa0f8081c50 2021-02-10T11:24:11.242 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:11.239 7fa0fe31f700 1 -- 172.21.2.102:0/2477413429 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0f8187fc0 con 0x7fa0f8081c50 2021-02-10T11:24:11.242 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:11.239 7fa0fe31f700 1 -- 172.21.2.102:0/2477413429 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0f8187fc0 con 0x7fa0f8081c50 2021-02-10T11:24:11.242 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:11.240 7fa0f4ff9700 1 -- 172.21.2.102:0/2477413429 <== mon.2 v2:172.21.2.113:3301/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa0f000a250 con 0x7fa0f8081c50 2021-02-10T11:24:11.242 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:11.240 7fa0f4ff9700 1 --2- 172.21.2.102:0/2477413429 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa0dc018730 0x7fa0dc01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:11.242 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:11.240 7fa0f4ff9700 1 -- 172.21.2.102:0/2477413429 <== mon.2 v2:172.21.2.113:3301/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fa0f00327c0 con 0x7fa0f8081c50 2021-02-10T11:24:11.242 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:11.240 7fa0f7fff700 1 --2- 172.21.2.102:0/2477413429 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa0dc018730 0x7fa0dc01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:11.242 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:11.240 7fa0f7fff700 1 --2- 172.21.2.102:0/2477413429 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa0dc018730 0x7fa0dc01abc0 crc :-1 s=READY pgs=617 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:11.269 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:11.270 7fa0fe31f700 1 -- 172.21.2.102:0/2477413429 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0f808d800 con 0x7fa0f8081c50 2021-02-10T11:24:11.273 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:11.274 7fa0f4ff9700 1 -- 172.21.2.102:0/2477413429 <== mon.2 v2:172.21.2.113:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fa0f0002c40 con 0x7fa0f8081c50 2021-02-10T11:24:11.402 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:11.403 7fa0fe31f700 1 -- 172.21.2.102:0/2477413429 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa0f808d800 con 0x7fa0f8081c50 2021-02-10T11:24:11.403 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:11.404 7fa0f4ff9700 1 -- 172.21.2.102:0/2477413429 <== mon.2 v2:172.21.2.113:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v57) v1 ==== 74+0+13996 (secure 0 0 0) 0x7fa0f000a070 con 0x7fa0f8081c50 2021-02-10T11:24:11.405 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:24:11.405 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":57,"fsid":"68db30ea-dbce-4aef-a6df-f01c6fea0513","created":"2021-02-10 11:03:06.331557","modified":"2021-02-10 11:24:11.200509","last_up_change":"2021-02-10 11:03:32.690562","last_in_change":"2021-02-10 11:03:32.690562","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:03:37.322519","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:24:08.122188","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":11,"pool_name":"cephfs_data","create_time":"2021-02-10 11:24:09.191606","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"be691b4d-a039-408a-af80-9d53f02b6f1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6818","nonce":36088},{"type":"v1","addr":"172.21.2.102:6820","nonce":36088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6822","nonce":36088},{"type":"v1","addr":"172.21.2.102:6824","nonce":36088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6830","nonce":36088},{"type":"v1","addr":"172.21.2.102:6832","nonce":36088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6826","nonce":36088},{"type":"v1","addr":"172.21.2.102:6828","nonce":36088}]},"public_addr":"172.21.2.102:6820/36088","cluster_addr":"172.21.2.102:6824/36088","heartbeat_back_addr":"172.21.2.102:6832/36088","heartbeat_front_addr":"172.21.2.102:6828/36088","state":["exists","up"]},{"osd":1,"uuid":"64e3ccc8-9c69-4d7c-865e-af8e83822635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6819","nonce":36087},{"type":"v1","addr":"172.21.2.102:6821","nonce":36087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6823","nonce":36087},{"type":"v1","addr":"172.21.2.102:6825","nonce":36087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6831","nonce":36087},{"type":"v1","addr":"172.21.2.102:6833","nonce":36087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6827","nonce":36087},{"type":"v1","addr":"172.21.2.102:6829","nonce":36087}]},"public_addr":"172.21.2.102:6821/36087","cluster_addr":"172.21.2.102:6825/36087","heartbeat_back_addr":"172.21.2.102:6833/36087","heartbeat_front_addr":"172.21.2.102:6829/36087","state":["exists","up"]},{"osd":2,"uuid":"153700af-747f-481f-bdf4-d6072d3a842f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6810","nonce":36089},{"type":"v1","addr":"172.21.2.102:6811","nonce":36089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6812","nonce":36089},{"type":"v1","addr":"172.21.2.102:6813","nonce":36089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6816","nonce":36089},{"type":"v1","addr":"172.21.2.102:6817","nonce":36089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6814","nonce":36089},{"type":"v1","addr":"172.21.2.102:6815","nonce":36089}]},"public_addr":"172.21.2.102:6811/36089","cluster_addr":"172.21.2.102:6813/36089","heartbeat_back_addr":"172.21.2.102:6817/36089","heartbeat_front_addr":"172.21.2.102:6815/36089","state":["exists","up"]},{"osd":3,"uuid":"cbc678f4-83a9-45e6-8830-2f7c232b1ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6802","nonce":36090},{"type":"v1","addr":"172.21.2.102:6803","nonce":36090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6804","nonce":36090},{"type":"v1","addr":"172.21.2.102:6805","nonce":36090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6808","nonce":36090},{"type":"v1","addr":"172.21.2.102:6809","nonce":36090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6806","nonce":36090},{"type":"v1","addr":"172.21.2.102:6807","nonce":36090}]},"public_addr":"172.21.2.102:6803/36090","cluster_addr":"172.21.2.102:6805/36090","heartbeat_back_addr":"172.21.2.102:6809/36090","heartbeat_front_addr":"172.21.2.102:6807/36090","state":["exists","up"]},{"osd":4,"uuid":"c01e4008-b258-4f10-afd2-9e7f37214351","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6808","nonce":36368},{"type":"v1","addr":"172.21.2.113:6811","nonce":36368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6814","nonce":36368},{"type":"v1","addr":"172.21.2.113:6818","nonce":36368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6826","nonce":36368},{"type":"v1","addr":"172.21.2.113:6828","nonce":36368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6820","nonce":36368},{"type":"v1","addr":"172.21.2.113:6824","nonce":36368}]},"public_addr":"172.21.2.113:6811/36368","cluster_addr":"172.21.2.113:6818/36368","heartbeat_back_addr":"172.21.2.113:6828/36368","heartbeat_front_addr":"172.21.2.113:6824/36368","state":["exists","up"]},{"osd":5,"uuid":"551f6007-daf8-4a98-8ee0-c2b16bcccb5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6802","nonce":36366},{"type":"v1","addr":"172.21.2.113:6805","nonce":36366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6807","nonce":36366},{"type":"v1","addr":"172.21.2.113:6810","nonce":36366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6819","nonce":36366},{"type":"v1","addr":"172.21.2.113:6822","nonce":36366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6812","nonce":36366},{"type":"v1","addr":"172.21.2.113:6815","nonce":36366}]},"public_addr":"172.21.2.113:6805/36366","cluster_addr":"172.21.2.113:6810/36366","heartbeat_back_addr":"172.21.2.113:6822/36366","heartbeat_front_addr":"172.21.2.113:6815/36366","state":["exists","up"]},{"osd":6,"uuid":"bf67c040-0e50-48be-b102-c3943bc9ebef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6800","nonce":36367},{"type":"v1","addr":"172.21.2.113:6801","nonce":36367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6803","nonce":36367},{"type":"v1","addr":"172.21.2.113:6804","nonce":36367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6813","nonce":36367},{"type":"v1","addr":"172.21.2.113:6816","nonce":36367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6806","nonce":36367},{"type":"v1","addr":"172.21.2.113:6809","nonce":36367}]},"public_addr":"172.21.2.113:6801/36367","cluster_addr":"172.21.2.113:6804/36367","heartbeat_back_addr":"172.21.2.113:6816/36367","heartbeat_front_addr":"172.21.2.113:6809/36367","state":["exists","up"]},{"osd":7,"uuid":"f5461f80-018d-428f-be89-2eb2bcbe7bb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6817","nonce":36365},{"type":"v1","addr":"172.21.2.113:6821","nonce":36365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6823","nonce":36365},{"type":"v1","addr":"172.21.2.113:6825","nonce":36365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6830","nonce":36365},{"type":"v1","addr":"172.21.2.113:6831","nonce":36365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6827","nonce":36365},{"type":"v1","addr":"172.21.2.113:6829","nonce":36365}]},"public_addr":"172.21.2.113:6821/36365","cluster_addr":"172.21.2.113:6825/36365","heartbeat_back_addr":"172.21.2.113:6831/36365","heartbeat_front_addr":"172.21.2.113:6829/36365","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:24:11.406 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:11.408 7fa0fe31f700 1 -- 172.21.2.102:0/2477413429 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa0dc018730 msgr2=0x7fa0dc01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:11.406 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:11.408 7fa0fe31f700 1 --2- 172.21.2.102:0/2477413429 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa0dc018730 0x7fa0dc01abc0 crc :-1 s=READY pgs=617 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:11.407 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:11.408 7fa0fe31f700 1 -- 172.21.2.102:0/2477413429 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa0f8081c50 msgr2=0x7fa0f80ad5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:11.407 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:11.408 7fa0fe31f700 1 --2- 172.21.2.102:0/2477413429 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa0f8081c50 0x7fa0f80ad5a0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7fa0f000afc0 tx=0x7fa0f0002850).stop 2021-02-10T11:24:11.407 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:11.408 7fa0fe31f700 1 -- 172.21.2.102:0/2477413429 shutdown_connections 2021-02-10T11:24:11.407 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:11.408 7fa0fe31f700 1 --2- 172.21.2.102:0/2477413429 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa0f8081c50 0x7fa0f80ad5a0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:11.407 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:11.408 7fa0fe31f700 1 --2- 172.21.2.102:0/2477413429 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa0dc018730 0x7fa0dc01abc0 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:11.407 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:11.408 7fa0fe31f700 1 --2- 172.21.2.102:0/2477413429 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa0f8081850 0x7fa0f80acf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:11.407 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:11.408 7fa0fe31f700 1 --2- 172.21.2.102:0/2477413429 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa0f808d400 0x7fa0f8081230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:11.407 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:11.408 7fa0fe31f700 1 -- 172.21.2.102:0/2477413429 >> 172.21.2.102:0/2477413429 conn(0x7fa0f81ac620 msgr2=0x7fa0f80896e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:11.408 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:11.408 7fa0fe31f700 1 -- 172.21.2.102:0/2477413429 shutdown_connections 2021-02-10T11:24:11.408 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:11.409 7fa0fe31f700 1 -- 172.21.2.102:0/2477413429 wait complete. 2021-02-10T11:24:11.416 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2021-02-10T11:24:11.846 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:11.831 7f06f0b21700 1 Processor -- start 2021-02-10T11:24:11.847 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:11.831 7f06f0b21700 1 -- start start 2021-02-10T11:24:11.847 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:11.832 7f06f0b21700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f06ec1b0870 0x7f06ec0f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:11.847 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:11.832 7f06f0b21700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f06ec0efe00 0x7f06ec0f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:11.847 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:11.832 7f06f0b21700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06ec1b0c70 con 0x7f06ec1b0470 2021-02-10T11:24:11.847 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:11.832 7f06f0b21700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06ec0e5ea0 con 0x7f06ec1b0870 2021-02-10T11:24:11.847 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:11.832 7f06f0b21700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06ec0f0b50 con 0x7f06ec0efe00 2021-02-10T11:24:11.847 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:11.832 7f06ea05d700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f06ec1b0870 0x7f06ec0f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:11.847 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:11.832 7f06ea85e700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f06ec1b0470 0x7f06ec0ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:49190/0 (socket says 172.21.2.102:49190) 2021-02-10T11:24:11.847 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:11.832 7f06ea85e700 1 -- 172.21.2.102:0/1641852294 learned_addr learned my addr 172.21.2.102:0/1641852294 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:24:11.847 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:11.832 7f06ea05d700 1 -- 172.21.2.102:0/1641852294 >> v2:172.21.2.113:3300/0 conn(0x7f06ec0efe00 msgr2=0x7f06ec0f0200 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:24:11.847 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:11.832 7f06ea05d700 1 --2- 172.21.2.102:0/1641852294 >> v2:172.21.2.113:3300/0 conn(0x7f06ec0efe00 0x7f06ec0f0200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:11.847 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:11.832 7f06ea05d700 1 -- 172.21.2.102:0/1641852294 >> v1:172.21.2.102:6789/0 conn(0x7f06ec1b0470 legacy=0x7f06ec0ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:11.848 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:11.832 7f06ea05d700 1 -- 172.21.2.102:0/1641852294 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06ec0f0820 con 0x7f06ec1b0870 2021-02-10T11:24:11.848 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:11.833 7f06ea05d700 1 --2- 172.21.2.102:0/1641852294 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f06ec1b0870 0x7f06ec0f20e0 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 rx=0x7f06d4006650 tx=0x7f06d4009a80).ready entity=mon.0 client_cookie=dbfd5080376e78ca server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:11.848 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:11.833 7f06e985c700 1 -- 172.21.2.102:0/1641852294 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f06d400c0c0 con 0x7f06ec1b0870 2021-02-10T11:24:11.848 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:11.833 7f06e985c700 1 -- 172.21.2.102:0/1641852294 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06d400c340 con 0x7f06ec1b0870 2021-02-10T11:24:11.848 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:11.833 7f06e985c700 1 -- 172.21.2.102:0/1641852294 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f06d400c860 con 0x7f06ec1b0870 2021-02-10T11:24:11.848 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:11.833 7f06f0b21700 1 -- 172.21.2.102:0/1641852294 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f06ec1b0870 msgr2=0x7f06ec0f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:11.848 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:11.833 7f06f0b21700 1 --2- 172.21.2.102:0/1641852294 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f06ec1b0870 0x7f06ec0f20e0 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 rx=0x7f06d4006650 tx=0x7f06d4009a80).stop 2021-02-10T11:24:11.848 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:11.833 7f06f0b21700 1 -- 172.21.2.102:0/1641852294 shutdown_connections 2021-02-10T11:24:11.848 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:11.833 7f06f0b21700 1 --2- 172.21.2.102:0/1641852294 >> v2:172.21.2.113:3300/0 conn(0x7f06ec0efe00 0x7f06ec0f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:11.848 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:11.833 7f06f0b21700 1 --2- 172.21.2.102:0/1641852294 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f06ec1b0870 0x7f06ec0f20e0 unknown :-1 s=CLOSED pgs=1316 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:11.848 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:11.833 7f06f0b21700 1 -- 172.21.2.102:0/1641852294 >> 172.21.2.102:0/1641852294 conn(0x7f06ec0ce3e0 msgr2=0x7f06ec0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:11.848 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:11.834 7f06f0b21700 1 -- 172.21.2.102:0/1641852294 shutdown_connections 2021-02-10T11:24:11.848 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:11.834 7f06f0b21700 1 -- 172.21.2.102:0/1641852294 wait complete. 2021-02-10T11:24:11.850 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:11.848 7f06f0b21700 1 Processor -- start 2021-02-10T11:24:11.850 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:11.848 7f06f0b21700 1 -- start start 2021-02-10T11:24:11.850 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:11.849 7f06f0b21700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f06ec094fb0 0x7f06ec09e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:11.850 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:11.849 7f06f0b21700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f06ec0c2be0 0x7f06ec0c2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:11.850 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:11.849 7f06f0b21700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f06ec0c3600 0x7f06ec0a9c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:11.850 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:11.849 7f06f0b21700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06ec09ebf0 con 0x7f06ec0c3600 2021-02-10T11:24:11.850 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:11.849 7f06f0b21700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06ec0aa2a0 con 0x7f06ec0c2be0 2021-02-10T11:24:11.851 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:11.849 7f06f0b21700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06ec0aa4c0 con 0x7f06ec094fb0 2021-02-10T11:24:11.851 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:11.849 7f06eb05f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f06ec0c3600 0x7f06ec0a9c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:11.851 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:11.849 7f06eb05f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f06ec0c3600 0x7f06ec0a9c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:56976/0 (socket says 172.21.2.102:56976) 2021-02-10T11:24:11.851 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:11.849 7f06eb05f700 1 -- 172.21.2.102:0/1678591831 learned_addr learned my addr 172.21.2.102:0/1678591831 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:24:11.851 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:11.849 7f06eb05f700 1 -- 172.21.2.102:0/1678591831 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f06ec094fb0 msgr2=0x7f06ec09e5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:11.851 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:11.849 7f06eb05f700 1 --2- 172.21.2.102:0/1678591831 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f06ec094fb0 0x7f06ec09e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:11.851 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:11.849 7f06eb05f700 1 -- 172.21.2.102:0/1678591831 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f06ec0c2be0 msgr2=0x7f06ec0c2fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:11.851 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:11.849 7f06eb05f700 1 --2- 172.21.2.102:0/1678591831 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f06ec0c2be0 0x7f06ec0c2fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:11.851 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:11.849 7f06eb05f700 1 -- 172.21.2.102:0/1678591831 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06ec0aa6e0 con 0x7f06ec0c3600 2021-02-10T11:24:11.851 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:11.850 7f06eb05f700 1 --2- 172.21.2.102:0/1678591831 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f06ec0c3600 0x7f06ec0a9c80 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 rx=0x7f06e4009960 tx=0x7f06e400bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:11.851 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:11.850 7f06e37fe700 1 -- 172.21.2.102:0/1678591831 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f06e4000d10 con 0x7f06ec0c3600 2021-02-10T11:24:11.851 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:11.850 7f06e37fe700 1 -- 172.21.2.102:0/1678591831 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06e4003030 con 0x7f06ec0c3600 2021-02-10T11:24:11.852 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:11.850 7f06f0b21700 1 -- 172.21.2.102:0/1678591831 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06ec0aa6e0 con 0x7f06ec0c3600 2021-02-10T11:24:11.852 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:11.850 7f06f0b21700 1 -- 172.21.2.102:0/1678591831 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06ec0aac70 con 0x7f06ec0c3600 2021-02-10T11:24:11.852 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:11.850 7f06e37fe700 1 -- 172.21.2.102:0/1678591831 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f06e4001590 con 0x7f06ec0c3600 2021-02-10T11:24:11.852 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:11.851 7f06e37fe700 1 -- 172.21.2.102:0/1678591831 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f06e4001310 con 0x7f06ec0c3600 2021-02-10T11:24:11.852 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:11.851 7f06e37fe700 1 --2- 172.21.2.102:0/1678591831 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f06d0018730 0x7f06d001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:11.852 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:11.851 7f06e37fe700 1 -- 172.21.2.102:0/1678591831 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f06e4032b30 con 0x7f06ec0c3600 2021-02-10T11:24:11.852 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:11.851 7f06ea85e700 1 --2- 172.21.2.102:0/1678591831 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f06d0018730 0x7f06d001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:11.852 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:11.851 7f06ea85e700 1 --2- 172.21.2.102:0/1678591831 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f06d0018730 0x7f06d001ac10 crc :-1 s=READY pgs=618 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:11.880 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:11.882 7f06f0b21700 1 -- 172.21.2.102:0/1678591831 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06ec0a9750 con 0x7f06ec0c3600 2021-02-10T11:24:11.883 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:11.885 7f06e37fe700 1 -- 172.21.2.102:0/1678591831 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f06e4000f40 con 0x7f06ec0c3600 2021-02-10T11:24:12.022 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:12.024 7f06f0b21700 1 -- 172.21.2.102:0/1678591831 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"} v 0) v1 -- 0x7f06ec0a9750 con 0x7f06ec0c3600 2021-02-10T11:24:12.207 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:12.208 7f06e37fe700 1 -- 172.21.2.102:0/1678591831 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"}]=0 v39) v1 ==== 118+0+0 (secure 0 0 0) 0x7f06e4003030 con 0x7f06ec0c3600 2021-02-10T11:24:12.241 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:12.242 7f06f0b21700 1 -- 172.21.2.102:0/1678591831 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f06d0018730 msgr2=0x7f06d001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:12.241 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:12.242 7f06f0b21700 1 --2- 172.21.2.102:0/1678591831 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f06d0018730 0x7f06d001ac10 crc :-1 s=READY pgs=618 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:12.241 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:12.242 7f06f0b21700 1 -- 172.21.2.102:0/1678591831 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f06ec0c3600 msgr2=0x7f06ec0a9c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:12.241 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:12.242 7f06f0b21700 1 --2- 172.21.2.102:0/1678591831 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f06ec0c3600 0x7f06ec0a9c80 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 rx=0x7f06e4009960 tx=0x7f06e400bcf0).stop 2021-02-10T11:24:12.241 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:12.242 7f06f0b21700 1 -- 172.21.2.102:0/1678591831 shutdown_connections 2021-02-10T11:24:12.241 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:12.242 7f06f0b21700 1 --2- 172.21.2.102:0/1678591831 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f06ec0c3600 0x7f06ec0a9c80 unknown :-1 s=CLOSED pgs=1317 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:12.241 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:12.242 7f06f0b21700 1 --2- 172.21.2.102:0/1678591831 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f06d0018730 0x7f06d001ac10 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:12.241 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:12.242 7f06f0b21700 1 --2- 172.21.2.102:0/1678591831 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f06ec0c2be0 0x7f06ec0c2fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:12.242 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:12.242 7f06f0b21700 1 --2- 172.21.2.102:0/1678591831 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f06ec094fb0 0x7f06ec09e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:12.242 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:12.242 7f06f0b21700 1 -- 172.21.2.102:0/1678591831 >> 172.21.2.102:0/1678591831 conn(0x7f06ec09f630 msgr2=0x7f06ec0a0130 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:12.242 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:12.242 7f06f0b21700 1 -- 172.21.2.102:0/1678591831 shutdown_connections 2021-02-10T11:24:12.242 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:12.242 7f06f0b21700 1 -- 172.21.2.102:0/1678591831 wait complete. 2021-02-10T11:24:12.249 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:24:12.678 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:12.661 7f52bf090700 1 Processor -- start 2021-02-10T11:24:12.679 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:12.661 7f52bf090700 1 -- start start 2021-02-10T11:24:12.679 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:12.662 7f52bf090700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f52b80e5070 0x7f52b81b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:12.679 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:12.662 7f52bf090700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f52b80e5470 0x7f52b807d690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:12.679 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:12.662 7f52bf090700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52b807ed80 con 0x7f52b81b3e40 2021-02-10T11:24:12.679 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:12.662 7f52bf090700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52b8080310 con 0x7f52b80e5470 2021-02-10T11:24:12.679 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:12.662 7f52bf090700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52b8080500 con 0x7f52b80e5070 2021-02-10T11:24:12.679 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:12.662 7f52bd6ad700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f52b81b3e40 0x7f52b807dcb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:49204/0 (socket says 172.21.2.102:49204) 2021-02-10T11:24:12.679 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:12.662 7f52bd6ad700 1 -- 172.21.2.102:0/988503007 learned_addr learned my addr 172.21.2.102:0/988503007 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:24:12.679 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:12.662 7f52b77fe700 1 -- 172.21.2.102:0/988503007 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3971111717 0 0) 0x7f52b0001180 con 0x7f52b81b3e40 2021-02-10T11:24:12.679 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:12.663 7f52b77fe700 1 -- 172.21.2.102:0/988503007 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52a4002fe0 con 0x7f52b81b3e40 2021-02-10T11:24:12.679 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:12.663 7f52bceac700 1 --2- 172.21.2.102:0/988503007 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f52b80e5070 0x7f52b81b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:12.679 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:12.663 7f52b7fff700 1 --2- 172.21.2.102:0/988503007 >> v2:172.21.2.113:3300/0 conn(0x7f52b80e5470 0x7f52b807d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:12.680 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:12.663 7f52b77fe700 1 -- 172.21.2.102:0/988503007 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3509751436 0 0) 0x7f52b0001180 con 0x7f52b81b3e40 2021-02-10T11:24:12.680 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:12.663 7f52b77fe700 1 -- 172.21.2.102:0/988503007 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f52b80e5070 msgr2=0x7f52b81b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:24:12.680 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:12.663 7f52b77fe700 1 --2- 172.21.2.102:0/988503007 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f52b80e5070 0x7f52b81b3820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:12.680 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:12.663 7f52b77fe700 1 -- 172.21.2.102:0/988503007 >> v2:172.21.2.113:3300/0 conn(0x7f52b80e5470 msgr2=0x7f52b807d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:24:12.680 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:12.663 7f52b77fe700 1 --2- 172.21.2.102:0/988503007 >> v2:172.21.2.113:3300/0 conn(0x7f52b80e5470 0x7f52b807d690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:12.680 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:12.663 7f52b77fe700 1 -- 172.21.2.102:0/988503007 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52b80806f0 con 0x7f52b81b3e40 2021-02-10T11:24:12.680 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:12.663 7f52b77fe700 1 -- 172.21.2.102:0/988503007 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7f52b0001680 con 0x7f52b81b3e40 2021-02-10T11:24:12.680 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:12.663 7f52b77fe700 1 -- 172.21.2.102:0/988503007 >> v1:172.21.2.102:6789/0 conn(0x7f52b81b3e40 legacy=0x7f52b807dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:12.680 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:12.663 7f52b77fe700 1 --2- 172.21.2.102:0/988503007 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f52a4003b90 0x7f52a4006040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:12.680 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:12.663 7f52b77fe700 1 --2- 172.21.2.102:0/988503007 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f52a404a860 0x7f52a404cd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:12.681 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:12.663 7f52b77fe700 1 --2- 172.21.2.102:0/988503007 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f52a404d5b0 0x7f52a404fad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:12.681 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:12.663 7f52b77fe700 1 -- 172.21.2.102:0/988503007 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52a4050380 con 0x7f52a4003b90 2021-02-10T11:24:12.681 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:12.663 7f52b77fe700 1 -- 172.21.2.102:0/988503007 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52a40505a0 con 0x7f52a404d5b0 2021-02-10T11:24:12.681 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:12.663 7f52b77fe700 1 -- 172.21.2.102:0/988503007 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52a40507c0 con 0x7f52a404a860 2021-02-10T11:24:12.681 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:12.663 7f52bceac700 1 --2- 172.21.2.102:0/988503007 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f52b80e5070 0x7f52b81b3820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:24:12.681 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:12.663 7f52bceac700 1 --2- 172.21.2.102:0/988503007 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f52a4003b90 0x7f52a4006040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:12.681 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:12.663 7f52b7fff700 1 --2- 172.21.2.102:0/988503007 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f52a404a860 0x7f52a404cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:12.681 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:12.664 7f52bd6ad700 1 --2- 172.21.2.102:0/988503007 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f52a404d5b0 0x7f52a404fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:12.681 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:12.664 7f52bceac700 1 -- 172.21.2.102:0/988503007 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f52a404a860 msgr2=0x7f52a404cd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:24:12.681 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:12.664 7f52bceac700 1 --2- 172.21.2.102:0/988503007 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f52a404a860 0x7f52a404cd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:12.681 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:12.664 7f52bceac700 1 -- 172.21.2.102:0/988503007 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f52a404d5b0 msgr2=0x7f52a404fad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:24:12.681 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:12.664 7f52bceac700 1 --2- 172.21.2.102:0/988503007 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f52a404d5b0 0x7f52a404fad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:12.681 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:12.664 7f52bceac700 1 -- 172.21.2.102:0/988503007 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52a4050a60 con 0x7f52a4003b90 2021-02-10T11:24:12.682 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:12.664 7f52bd6ad700 1 --2- 172.21.2.102:0/988503007 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f52a404d5b0 0x7f52a404fad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:24:12.682 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:12.664 7f52bceac700 1 --2- 172.21.2.102:0/988503007 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f52a4003b90 0x7f52a4006040 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 rx=0x7f52a00063a0 tx=0x7f52a0009aa0).ready entity=mon.0 client_cookie=9f85a88801e2be7c server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:12.682 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:12.664 7f52b77fe700 1 -- 172.21.2.102:0/988503007 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f52a000c0c0 con 0x7f52a4003b90 2021-02-10T11:24:12.682 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:12.664 7f52b77fe700 1 -- 172.21.2.102:0/988503007 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52a000c380 con 0x7f52a4003b90 2021-02-10T11:24:12.682 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:12.664 7f52b77fe700 1 -- 172.21.2.102:0/988503007 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f52a000c8a0 con 0x7f52a4003b90 2021-02-10T11:24:12.682 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:12.664 7f52bf090700 1 -- 172.21.2.102:0/988503007 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f52a4003b90 msgr2=0x7f52a4006040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:12.682 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:12.664 7f52bf090700 1 --2- 172.21.2.102:0/988503007 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f52a4003b90 0x7f52a4006040 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 rx=0x7f52a00063a0 tx=0x7f52a0009aa0).stop 2021-02-10T11:24:12.682 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:12.664 7f52bd6ad700 1 -- 172.21.2.102:0/988503007 reap_dead start 2021-02-10T11:24:12.682 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:12.665 7f52bf090700 1 -- 172.21.2.102:0/988503007 shutdown_connections 2021-02-10T11:24:12.682 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:12.665 7f52bf090700 1 -- 172.21.2.102:0/988503007 >> 172.21.2.102:0/988503007 conn(0x7f52b80ce690 msgr2=0x7f52b80e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:12.682 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:12.665 7f52bf090700 1 -- 172.21.2.102:0/988503007 shutdown_connections 2021-02-10T11:24:12.682 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:12.665 7f52bf090700 1 -- 172.21.2.102:0/988503007 wait complete. 2021-02-10T11:24:12.683 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:12.680 7f52bf090700 1 Processor -- start 2021-02-10T11:24:12.683 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:12.680 7f52bf090700 1 -- start start 2021-02-10T11:24:12.684 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:12.680 7f52bf090700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f52b81ac6c0 0x7f52b809bc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:12.684 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:12.680 7f52bf090700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f52b81acac0 0x7f52b809c280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:12.684 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:12.680 7f52bf090700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f52b808ab70 0x7f52b808af70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:12.684 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:12.680 7f52bf090700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52b808c0f0 con 0x7f52b81acac0 2021-02-10T11:24:12.684 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:12.680 7f52bf090700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52b8088c80 con 0x7f52b81ac6c0 2021-02-10T11:24:12.684 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:12.680 7f52bf090700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52b809ca80 con 0x7f52b808ab70 2021-02-10T11:24:12.685 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:12.681 7f52b7fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f52b81acac0 0x7f52b809c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:12.685 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:12.681 7f52b7fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f52b81acac0 0x7f52b809c280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:56996/0 (socket says 172.21.2.102:56996) 2021-02-10T11:24:12.685 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:12.681 7f52b7fff700 1 -- 172.21.2.102:0/1253665614 learned_addr learned my addr 172.21.2.102:0/1253665614 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:24:12.685 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:12.681 7f52bceac700 1 --2- 172.21.2.102:0/1253665614 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f52b81ac6c0 0x7f52b809bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:12.685 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:12.681 7f52bd6ad700 1 --2- 172.21.2.102:0/1253665614 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f52b808ab70 0x7f52b808af70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:12.685 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:12.681 7f52b7fff700 1 -- 172.21.2.102:0/1253665614 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f52b808ab70 msgr2=0x7f52b808af70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:12.685 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:12.681 7f52b7fff700 1 --2- 172.21.2.102:0/1253665614 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f52b808ab70 0x7f52b808af70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:12.685 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:12.681 7f52b7fff700 1 -- 172.21.2.102:0/1253665614 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f52b81ac6c0 msgr2=0x7f52b809bc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:12.685 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:12.681 7f52b7fff700 1 --2- 172.21.2.102:0/1253665614 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f52b81ac6c0 0x7f52b809bc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:12.685 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:12.681 7f52b7fff700 1 -- 172.21.2.102:0/1253665614 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52b80b5690 con 0x7f52b81acac0 2021-02-10T11:24:12.685 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:12.681 7f52b7fff700 1 --2- 172.21.2.102:0/1253665614 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f52b81acac0 0x7f52b809c280 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 rx=0x7f52a80099e0 tx=0x7f52a800bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:12.685 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:12.682 7f52b57fa700 1 -- 172.21.2.102:0/1253665614 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f52a800c2f0 con 0x7f52b81acac0 2021-02-10T11:24:12.686 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:12.682 7f52b57fa700 1 -- 172.21.2.102:0/1253665614 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52a800c630 con 0x7f52b81acac0 2021-02-10T11:24:12.686 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:12.682 7f52b57fa700 1 -- 172.21.2.102:0/1253665614 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f52a800ca80 con 0x7f52b81acac0 2021-02-10T11:24:12.686 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:12.682 7f52bf090700 1 -- 172.21.2.102:0/1253665614 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52b80b5690 con 0x7f52b81acac0 2021-02-10T11:24:12.686 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:12.682 7f52bf090700 1 -- 172.21.2.102:0/1253665614 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52b808b880 con 0x7f52b81acac0 2021-02-10T11:24:12.686 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:12.682 7f52b57fa700 1 -- 172.21.2.102:0/1253665614 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f52a800c2f0 con 0x7f52b81acac0 2021-02-10T11:24:12.686 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:12.682 7f52b57fa700 1 --2- 172.21.2.102:0/1253665614 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f529c0186c0 0x7f529c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:12.686 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:12.682 7f52b57fa700 1 -- 172.21.2.102:0/1253665614 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f52a80328e0 con 0x7f52b81acac0 2021-02-10T11:24:12.686 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:12.682 7f52bceac700 1 --2- 172.21.2.102:0/1253665614 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f529c0186c0 0x7f529c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:12.686 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:12.683 7f52bceac700 1 --2- 172.21.2.102:0/1253665614 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f529c0186c0 0x7f529c01abc0 crc :-1 s=READY pgs=619 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:12.710 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:12.711 7f52bf090700 1 -- 172.21.2.102:0/1253665614 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52b8088c80 con 0x7f52b81acac0 2021-02-10T11:24:12.712 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:12.714 7f52b57fa700 1 -- 172.21.2.102:0/1253665614 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f52a8028650 con 0x7f52b81acac0 2021-02-10T11:24:12.842 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:12.843 7f52bf090700 1 -- 172.21.2.102:0/1253665614 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f52b8088c80 con 0x7f52b81acac0 2021-02-10T11:24:12.842 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:12.844 7f52b57fa700 1 -- 172.21.2.102:0/1253665614 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 39 v39) v1 ==== 94+0+2101 (secure 0 0 0) 0x7f52a8028650 con 0x7f52b81acac0 2021-02-10T11:24:12.843 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:24:12.843 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":39,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7920,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.102:6835/1829204031","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6834","nonce":1829204031},{"type":"v1","addr":"172.21.2.102:6835","nonce":1829204031}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":35}],"filesystems":[{"mdsmap":{"epoch":39,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:24:10.198618","modified":"2021-02-10 11:24:12.204882","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5714},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5714":{"gid":5714,"name":"b","rank":0,"incarnation":36,"state":"up:active","state_seq":3,"addr":"172.21.2.113:6833/4013137507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6832","nonce":4013137507},{"type":"v1","addr":"172.21.2.113:6833","nonce":4013137507}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":5}]} 2021-02-10T11:24:12.843 INFO:teuthology.orchestra.run.gibba002.stderr:dumped fsmap epoch 39 2021-02-10T11:24:12.844 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:12.846 7f52bf090700 1 -- 172.21.2.102:0/1253665614 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f529c0186c0 msgr2=0x7f529c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:12.844 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:12.846 7f52bf090700 1 --2- 172.21.2.102:0/1253665614 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f529c0186c0 0x7f529c01abc0 crc :-1 s=READY pgs=619 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:12.844 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:12.846 7f52bf090700 1 -- 172.21.2.102:0/1253665614 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f52b81acac0 msgr2=0x7f52b809c280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:12.844 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:12.846 7f52bf090700 1 --2- 172.21.2.102:0/1253665614 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f52b81acac0 0x7f52b809c280 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 rx=0x7f52a80099e0 tx=0x7f52a800bcf0).stop 2021-02-10T11:24:12.845 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:12.846 7f52bf090700 1 -- 172.21.2.102:0/1253665614 shutdown_connections 2021-02-10T11:24:12.845 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:12.846 7f52bf090700 1 --2- 172.21.2.102:0/1253665614 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f52b808ab70 0x7f52b808af70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:12.845 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:12.846 7f52bf090700 1 --2- 172.21.2.102:0/1253665614 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f52b81acac0 0x7f52b809c280 unknown :-1 s=CLOSED pgs=1320 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:12.845 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:12.846 7f52bf090700 1 --2- 172.21.2.102:0/1253665614 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f529c0186c0 0x7f529c01abc0 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:12.845 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:12.846 7f52bf090700 1 --2- 172.21.2.102:0/1253665614 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f52b81ac6c0 0x7f52b809bc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:12.845 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:12.846 7f52bf090700 1 -- 172.21.2.102:0/1253665614 >> 172.21.2.102:0/1253665614 conn(0x7f52b81a31a0 msgr2=0x7f52b8084da0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:12.845 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:12.846 7f52bf090700 1 -- 172.21.2.102:0/1253665614 shutdown_connections 2021-02-10T11:24:12.845 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:12.846 7f52bf090700 1 -- 172.21.2.102:0/1253665614 wait complete. 2021-02-10T11:24:12.854 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:24:13.283 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:13.266 7f1471219700 1 Processor -- start 2021-02-10T11:24:13.284 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:13.266 7f1471219700 1 -- start start 2021-02-10T11:24:13.284 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:13.267 7f1471219700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f146c0e5470 0x7f146c07e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:13.284 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:13.267 7f1471219700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f146c07ed80 con 0x7f146c0e5070 2021-02-10T11:24:13.284 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:13.267 7f1471219700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f146c1b4010 con 0x7f146c07f300 2021-02-10T11:24:13.284 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:13.267 7f1471219700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f146c0d1790 con 0x7f146c0e5470 2021-02-10T11:24:13.284 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:13.267 7f146ad9d700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f146c0e5070 0x7f146c1b3820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:49224/0 (socket says 172.21.2.102:49224) 2021-02-10T11:24:13.284 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:13.267 7f146ad9d700 1 -- 172.21.2.102:0/3739703021 learned_addr learned my addr 172.21.2.102:0/3739703021 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:24:13.284 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:13.267 7f146a59c700 1 --2- 172.21.2.102:0/3739703021 >> v2:172.21.2.113:3300/0 conn(0x7f146c0e5470 0x7f146c07e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:13.284 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:13.268 7f1469d9b700 1 -- 172.21.2.102:0/3739703021 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3969517464 0 0) 0x7f145c001180 con 0x7f146c0e5070 2021-02-10T11:24:13.284 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:13.268 7f1469d9b700 1 -- 172.21.2.102:0/3739703021 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1458002fe0 con 0x7f146c0e5070 2021-02-10T11:24:13.284 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:13.268 7f1469d9b700 1 -- 172.21.2.102:0/3739703021 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2572100834 0 0) 0x7f145c0010e0 con 0x7f146c0e5070 2021-02-10T11:24:13.285 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:13.268 7f1469d9b700 1 -- 172.21.2.102:0/3739703021 >> v2:172.21.2.113:3300/0 conn(0x7f146c0e5470 msgr2=0x7f146c07e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:24:13.285 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:13.268 7f1469d9b700 1 --2- 172.21.2.102:0/3739703021 >> v2:172.21.2.113:3300/0 conn(0x7f146c0e5470 0x7f146c07e6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:13.285 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:13.268 7f1469d9b700 1 -- 172.21.2.102:0/3739703021 >> v1:172.21.2.113:6789/0 conn(0x7f146c07f300 legacy=0x7f146c07f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:13.285 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:13.268 7f1469d9b700 1 -- 172.21.2.102:0/3739703021 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f146c0d19b0 con 0x7f146c0e5070 2021-02-10T11:24:13.285 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:13.268 7f1469d9b700 1 -- 172.21.2.102:0/3739703021 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7f145c001600 con 0x7f146c0e5070 2021-02-10T11:24:13.285 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:13.268 7f1469d9b700 1 -- 172.21.2.102:0/3739703021 >> v1:172.21.2.102:6789/0 conn(0x7f146c0e5070 legacy=0x7f146c1b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:13.286 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:13.268 7f1469d9b700 1 --2- 172.21.2.102:0/3739703021 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1458003b20 0x7f1458005f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:13.286 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:13.268 7f1469d9b700 1 --2- 172.21.2.102:0/3739703021 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1458006830 0x7f1458008d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:13.286 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:13.268 7f1469d9b700 1 --2- 172.21.2.102:0/3739703021 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1458009630 0x7f145800bb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:13.286 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:13.268 7f1469d9b700 1 -- 172.21.2.102:0/3739703021 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f145800c3a0 con 0x7f1458003b20 2021-02-10T11:24:13.286 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:13.268 7f1469d9b700 1 -- 172.21.2.102:0/3739703021 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f145800c5c0 con 0x7f1458009630 2021-02-10T11:24:13.286 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:13.268 7f1469d9b700 1 -- 172.21.2.102:0/3739703021 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14580507f0 con 0x7f1458006830 2021-02-10T11:24:13.287 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:13.268 7f146ad9d700 1 --2- 172.21.2.102:0/3739703021 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1458003b20 0x7f1458005f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:13.287 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:13.268 7f146a59c700 1 --2- 172.21.2.102:0/3739703021 >> v2:172.21.2.113:3300/0 conn(0x7f146c0e5470 0x7f146c07e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:24:13.287 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:13.269 7f146ad9d700 1 -- 172.21.2.102:0/3739703021 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1458006830 msgr2=0x7f1458008d30 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:24:13.287 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:13.269 7f146ad9d700 1 --2- 172.21.2.102:0/3739703021 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1458006830 0x7f1458008d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:13.287 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:13.269 7f146ad9d700 1 -- 172.21.2.102:0/3739703021 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1458009630 msgr2=0x7f145800bb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:24:13.287 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:13.269 7f146ad9d700 1 --2- 172.21.2.102:0/3739703021 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1458009630 0x7f145800bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:13.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:13.269 7f146ad9d700 1 -- 172.21.2.102:0/3739703021 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1458050a50 con 0x7f1458003b20 2021-02-10T11:24:13.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:13.269 7f146b59e700 1 -- 172.21.2.102:0/3739703021 reap_dead start 2021-02-10T11:24:13.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:13.269 7f146ad9d700 1 --2- 172.21.2.102:0/3739703021 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1458003b20 0x7f1458005f90 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 rx=0x7f145c000e80 tx=0x7f145c007cf0).ready entity=mon.0 client_cookie=ac24f93cae3cbec1 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:13.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:13.269 7f1469d9b700 1 -- 172.21.2.102:0/3739703021 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f145c0070c0 con 0x7f1458003b20 2021-02-10T11:24:13.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:13.269 7f1469d9b700 1 -- 172.21.2.102:0/3739703021 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f145c00a520 con 0x7f1458003b20 2021-02-10T11:24:13.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:13.269 7f1469d9b700 1 -- 172.21.2.102:0/3739703021 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f145c00aa40 con 0x7f1458003b20 2021-02-10T11:24:13.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:13.269 7f1471219700 1 -- 172.21.2.102:0/3739703021 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1458003b20 msgr2=0x7f1458005f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:13.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:13.269 7f1471219700 1 --2- 172.21.2.102:0/3739703021 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1458003b20 0x7f1458005f90 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 rx=0x7f145c000e80 tx=0x7f145c007cf0).stop 2021-02-10T11:24:13.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:13.270 7f1471219700 1 -- 172.21.2.102:0/3739703021 shutdown_connections 2021-02-10T11:24:13.289 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:13.270 7f1471219700 1 --2- 172.21.2.102:0/3739703021 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1458003b20 0x7f1458005f90 unknown :-1 s=CLOSED pgs=1322 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:13.289 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:13.270 7f1471219700 1 -- 172.21.2.102:0/3739703021 >> 172.21.2.102:0/3739703021 conn(0x7f146c0ce690 msgr2=0x7f146c0e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:13.289 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:13.270 7f1471219700 1 -- 172.21.2.102:0/3739703021 shutdown_connections 2021-02-10T11:24:13.289 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:13.270 7f1471219700 1 -- 172.21.2.102:0/3739703021 wait complete. 2021-02-10T11:24:13.289 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:13.285 7f1471219700 1 Processor -- start 2021-02-10T11:24:13.289 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:13.285 7f1471219700 1 -- start start 2021-02-10T11:24:13.289 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:13.285 7f1471219700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f146c194000 0x7f146c0b0d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:13.289 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:13.285 7f1471219700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f146c0b1330 0x7f146c087050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:13.289 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:13.285 7f1471219700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f146c087670 0x7f146c087a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:13.289 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:13.285 7f1471219700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f146c09c0e0 con 0x7f146c0b1330 2021-02-10T11:24:13.289 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:13.285 7f1471219700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f146c194440 con 0x7f146c194000 2021-02-10T11:24:13.289 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:13.285 7f1471219700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f146c09c5e0 con 0x7f146c087670 2021-02-10T11:24:13.291 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:13.285 7f146a59c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f146c0b1330 0x7f146c087050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:13.291 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:13.285 7f146a59c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f146c0b1330 0x7f146c087050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:57014/0 (socket says 172.21.2.102:57014) 2021-02-10T11:24:13.291 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:13.285 7f146a59c700 1 -- 172.21.2.102:0/3929096523 learned_addr learned my addr 172.21.2.102:0/3929096523 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:24:13.291 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:13.285 7f146a59c700 1 -- 172.21.2.102:0/3929096523 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f146c087670 msgr2=0x7f146c087a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:13.292 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:13.285 7f146a59c700 1 --2- 172.21.2.102:0/3929096523 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f146c087670 0x7f146c087a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:13.292 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:13.285 7f146a59c700 1 -- 172.21.2.102:0/3929096523 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f146c194000 msgr2=0x7f146c0b0d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:13.292 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:13.285 7f146a59c700 1 --2- 172.21.2.102:0/3929096523 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f146c194000 0x7f146c0b0d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:13.292 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:13.285 7f146a59c700 1 -- 172.21.2.102:0/3929096523 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f146c09c7d0 con 0x7f146c0b1330 2021-02-10T11:24:13.292 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:13.286 7f146a59c700 1 --2- 172.21.2.102:0/3929096523 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f146c0b1330 0x7f146c087050 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 rx=0x7f1454006b10 tx=0x7f145400b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:13.292 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:13.286 7f14637fe700 1 -- 172.21.2.102:0/3929096523 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f145400bd50 con 0x7f146c0b1330 2021-02-10T11:24:13.292 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:13.286 7f14637fe700 1 -- 172.21.2.102:0/3929096523 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1454002030 con 0x7f146c0b1330 2021-02-10T11:24:13.292 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:13.286 7f1471219700 1 -- 172.21.2.102:0/3929096523 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f146c09c7d0 con 0x7f146c0b1330 2021-02-10T11:24:13.292 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:13.286 7f1471219700 1 -- 172.21.2.102:0/3929096523 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f146c09cd60 con 0x7f146c0b1330 2021-02-10T11:24:13.292 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:13.286 7f14637fe700 1 -- 172.21.2.102:0/3929096523 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1454002550 con 0x7f146c0b1330 2021-02-10T11:24:13.292 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:13.286 7f14637fe700 1 -- 172.21.2.102:0/3929096523 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1454002210 con 0x7f146c0b1330 2021-02-10T11:24:13.292 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:13.286 7f14637fe700 1 --2- 172.21.2.102:0/3929096523 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1450018730 0x7f145001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:13.292 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:13.287 7f14637fe700 1 -- 172.21.2.102:0/3929096523 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f1454032790 con 0x7f146c0b1330 2021-02-10T11:24:13.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:13.287 7f146ad9d700 1 --2- 172.21.2.102:0/3929096523 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1450018730 0x7f145001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:13.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:13.287 7f146ad9d700 1 --2- 172.21.2.102:0/3929096523 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1450018730 0x7f145001ac10 crc :-1 s=READY pgs=620 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:13.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:13.317 7f1471219700 1 -- 172.21.2.102:0/3929096523 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f146c084b20 con 0x7f146c0b1330 2021-02-10T11:24:13.318 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:13.320 7f14637fe700 1 -- 172.21.2.102:0/3929096523 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f1454006c20 con 0x7f146c0b1330 2021-02-10T11:24:13.444 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:13.445 7f1471219700 1 -- 172.21.2.102:0/3929096523 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f146c07ed80 con 0x7f146c0b1330 2021-02-10T11:24:13.444 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:13.446 7f14637fe700 1 -- 172.21.2.102:0/3929096523 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v57) v1 ==== 74+0+13996 (secure 0 0 0) 0x7f1454002030 con 0x7f146c0b1330 2021-02-10T11:24:13.447 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:24:13.447 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":57,"fsid":"68db30ea-dbce-4aef-a6df-f01c6fea0513","created":"2021-02-10 11:03:06.331557","modified":"2021-02-10 11:24:11.200509","last_up_change":"2021-02-10 11:03:32.690562","last_in_change":"2021-02-10 11:03:32.690562","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:03:37.322519","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:24:08.122188","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":11,"pool_name":"cephfs_data","create_time":"2021-02-10 11:24:09.191606","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"be691b4d-a039-408a-af80-9d53f02b6f1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6818","nonce":36088},{"type":"v1","addr":"172.21.2.102:6820","nonce":36088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6822","nonce":36088},{"type":"v1","addr":"172.21.2.102:6824","nonce":36088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6830","nonce":36088},{"type":"v1","addr":"172.21.2.102:6832","nonce":36088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6826","nonce":36088},{"type":"v1","addr":"172.21.2.102:6828","nonce":36088}]},"public_addr":"172.21.2.102:6820/36088","cluster_addr":"172.21.2.102:6824/36088","heartbeat_back_addr":"172.21.2.102:6832/36088","heartbeat_front_addr":"172.21.2.102:6828/36088","state":["exists","up"]},{"osd":1,"uuid":"64e3ccc8-9c69-4d7c-865e-af8e83822635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6819","nonce":36087},{"type":"v1","addr":"172.21.2.102:6821","nonce":36087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6823","nonce":36087},{"type":"v1","addr":"172.21.2.102:6825","nonce":36087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6831","nonce":36087},{"type":"v1","addr":"172.21.2.102:6833","nonce":36087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6827","nonce":36087},{"type":"v1","addr":"172.21.2.102:6829","nonce":36087}]},"public_addr":"172.21.2.102:6821/36087","cluster_addr":"172.21.2.102:6825/36087","heartbeat_back_addr":"172.21.2.102:6833/36087","heartbeat_front_addr":"172.21.2.102:6829/36087","state":["exists","up"]},{"osd":2,"uuid":"153700af-747f-481f-bdf4-d6072d3a842f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6810","nonce":36089},{"type":"v1","addr":"172.21.2.102:6811","nonce":36089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6812","nonce":36089},{"type":"v1","addr":"172.21.2.102:6813","nonce":36089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6816","nonce":36089},{"type":"v1","addr":"172.21.2.102:6817","nonce":36089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6814","nonce":36089},{"type":"v1","addr":"172.21.2.102:6815","nonce":36089}]},"public_addr":"172.21.2.102:6811/36089","cluster_addr":"172.21.2.102:6813/36089","heartbeat_back_addr":"172.21.2.102:6817/36089","heartbeat_front_addr":"172.21.2.102:6815/36089","state":["exists","up"]},{"osd":3,"uuid":"cbc678f4-83a9-45e6-8830-2f7c232b1ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6802","nonce":36090},{"type":"v1","addr":"172.21.2.102:6803","nonce":36090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6804","nonce":36090},{"type":"v1","addr":"172.21.2.102:6805","nonce":36090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6808","nonce":36090},{"type":"v1","addr":"172.21.2.102:6809","nonce":36090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6806","nonce":36090},{"type":"v1","addr":"172.21.2.102:6807","nonce":36090}]},"public_addr":"172.21.2.102:6803/36090","cluster_addr":"172.21.2.102:6805/36090","heartbeat_back_addr":"172.21.2.102:6809/36090","heartbeat_front_addr":"172.21.2.102:6807/36090","state":["exists","up"]},{"osd":4,"uuid":"c01e4008-b258-4f10-afd2-9e7f37214351","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6808","nonce":36368},{"type":"v1","addr":"172.21.2.113:6811","nonce":36368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6814","nonce":36368},{"type":"v1","addr":"172.21.2.113:6818","nonce":36368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6826","nonce":36368},{"type":"v1","addr":"172.21.2.113:6828","nonce":36368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6820","nonce":36368},{"type":"v1","addr":"172.21.2.113:6824","nonce":36368}]},"public_addr":"172.21.2.113:6811/36368","cluster_addr":"172.21.2.113:6818/36368","heartbeat_back_addr":"172.21.2.113:6828/36368","heartbeat_front_addr":"172.21.2.113:6824/36368","state":["exists","up"]},{"osd":5,"uuid":"551f6007-daf8-4a98-8ee0-c2b16bcccb5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6802","nonce":36366},{"type":"v1","addr":"172.21.2.113:6805","nonce":36366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6807","nonce":36366},{"type":"v1","addr":"172.21.2.113:6810","nonce":36366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6819","nonce":36366},{"type":"v1","addr":"172.21.2.113:6822","nonce":36366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6812","nonce":36366},{"type":"v1","addr":"172.21.2.113:6815","nonce":36366}]},"public_addr":"172.21.2.113:6805/36366","cluster_addr":"172.21.2.113:6810/36366","heartbeat_back_addr":"172.21.2.113:6822/36366","heartbeat_front_addr":"172.21.2.113:6815/36366","state":["exists","up"]},{"osd":6,"uuid":"bf67c040-0e50-48be-b102-c3943bc9ebef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6800","nonce":36367},{"type":"v1","addr":"172.21.2.113:6801","nonce":36367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6803","nonce":36367},{"type":"v1","addr":"172.21.2.113:6804","nonce":36367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6813","nonce":36367},{"type":"v1","addr":"172.21.2.113:6816","nonce":36367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6806","nonce":36367},{"type":"v1","addr":"172.21.2.113:6809","nonce":36367}]},"public_addr":"172.21.2.113:6801/36367","cluster_addr":"172.21.2.113:6804/36367","heartbeat_back_addr":"172.21.2.113:6816/36367","heartbeat_front_addr":"172.21.2.113:6809/36367","state":["exists","up"]},{"osd":7,"uuid":"f5461f80-018d-428f-be89-2eb2bcbe7bb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6817","nonce":36365},{"type":"v1","addr":"172.21.2.113:6821","nonce":36365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6823","nonce":36365},{"type":"v1","addr":"172.21.2.113:6825","nonce":36365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6830","nonce":36365},{"type":"v1","addr":"172.21.2.113:6831","nonce":36365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6827","nonce":36365},{"type":"v1","addr":"172.21.2.113:6829","nonce":36365}]},"public_addr":"172.21.2.113:6821/36365","cluster_addr":"172.21.2.113:6825/36365","heartbeat_back_addr":"172.21.2.113:6831/36365","heartbeat_front_addr":"172.21.2.113:6829/36365","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:24:13.447 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:13.447 7f1471219700 1 -- 172.21.2.102:0/3929096523 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1450018730 msgr2=0x7f145001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:13.448 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:13.447 7f1471219700 1 --2- 172.21.2.102:0/3929096523 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1450018730 0x7f145001ac10 crc :-1 s=READY pgs=620 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:13.448 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:13.447 7f1471219700 1 -- 172.21.2.102:0/3929096523 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f146c0b1330 msgr2=0x7f146c087050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:13.448 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:13.447 7f1471219700 1 --2- 172.21.2.102:0/3929096523 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f146c0b1330 0x7f146c087050 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 rx=0x7f1454006b10 tx=0x7f145400b960).stop 2021-02-10T11:24:13.448 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:13.447 7f1471219700 1 -- 172.21.2.102:0/3929096523 shutdown_connections 2021-02-10T11:24:13.448 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:13.447 7f1471219700 1 --2- 172.21.2.102:0/3929096523 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f146c087670 0x7f146c087a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:13.448 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:13.447 7f1471219700 1 --2- 172.21.2.102:0/3929096523 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f146c0b1330 0x7f146c087050 unknown :-1 s=CLOSED pgs=1323 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:13.448 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:13.447 7f1471219700 1 --2- 172.21.2.102:0/3929096523 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1450018730 0x7f145001ac10 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:13.448 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:13.447 7f1471219700 1 --2- 172.21.2.102:0/3929096523 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f146c194000 0x7f146c0b0d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:13.448 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:13.447 7f1471219700 1 -- 172.21.2.102:0/3929096523 >> 172.21.2.102:0/3929096523 conn(0x7f146c092c20 msgr2=0x7f146c195070 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:13.448 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:13.447 7f1471219700 1 -- 172.21.2.102:0/3929096523 shutdown_connections 2021-02-10T11:24:13.448 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:13.447 7f1471219700 1 -- 172.21.2.102:0/3929096523 wait complete. 2021-02-10T11:24:13.455 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:24:13.893 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:13.877 7f6fcb50b700 1 Processor -- start 2021-02-10T11:24:13.893 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:13.877 7f6fcb50b700 1 -- start start 2021-02-10T11:24:13.893 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:13.878 7f6fcb50b700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f6fc41b0470 0x7f6fc41b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:13.893 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:13.878 7f6fcb50b700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f6fc40f20e0 0x7f6fc40f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:13.893 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:13.878 7f6fcb50b700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fc40f0b50 con 0x7f6fc40ef610 2021-02-10T11:24:13.893 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:13.878 7f6fcb50b700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6fc40f0820 con 0x7f6fc40f20e0 2021-02-10T11:24:13.893 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:13.878 7f6fcb50b700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6fc40f24e0 con 0x7f6fc41b0470 2021-02-10T11:24:13.893 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:13.878 7f6fc8b26700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f6fc40ef610 0x7f6fc40efa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:49244/0 (socket says 172.21.2.102:49244) 2021-02-10T11:24:13.893 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:13.878 7f6fc8b26700 1 -- 172.21.2.102:0/2353080255 learned_addr learned my addr 172.21.2.102:0/2353080255 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:24:13.894 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:13.878 7f6fc9b28700 1 --2- 172.21.2.102:0/2353080255 >> v2:172.21.2.102:3300/0 conn(0x7f6fc40f20e0 0x7f6fc40f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:13.894 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:13.878 7f6fb7fff700 1 -- 172.21.2.102:0/2353080255 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3293383914 0 0) 0x7f6fb80010b0 con 0x7f6fc40ef610 2021-02-10T11:24:13.894 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:13.878 7f6fc9327700 1 --2- 172.21.2.102:0/2353080255 >> v2:172.21.2.113:3300/0 conn(0x7f6fc41b0470 0x7f6fc41b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:13.894 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:13.879 7f6fc9b28700 1 -- 172.21.2.102:0/2353080255 >> v2:172.21.2.113:3300/0 conn(0x7f6fc41b0470 msgr2=0x7f6fc41b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:24:13.894 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:13.879 7f6fc9b28700 1 --2- 172.21.2.102:0/2353080255 >> v2:172.21.2.113:3300/0 conn(0x7f6fc41b0470 0x7f6fc41b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:13.894 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:13.879 7f6fc9b28700 1 -- 172.21.2.102:0/2353080255 >> v1:172.21.2.102:6789/0 conn(0x7f6fc40ef610 legacy=0x7f6fc40efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:13.894 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:13.879 7f6fc9b28700 1 -- 172.21.2.102:0/2353080255 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fc40e52e0 con 0x7f6fc40f20e0 2021-02-10T11:24:13.894 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:13.879 7f6fc9327700 1 --2- 172.21.2.102:0/2353080255 >> v2:172.21.2.113:3300/0 conn(0x7f6fc41b0470 0x7f6fc41b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:24:13.894 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:13.879 7f6fc9b28700 1 --2- 172.21.2.102:0/2353080255 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6fc40f20e0 0x7f6fc40f0200 secure :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 rx=0x7f6fbc006e30 tx=0x7f6fbc009e60).ready entity=mon.0 client_cookie=23491479989abf1d server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:13.894 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:13.879 7f6fb7fff700 1 -- 172.21.2.102:0/2353080255 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6fbc00b300 con 0x7f6fc40f20e0 2021-02-10T11:24:13.894 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:13.879 7f6fb7fff700 1 -- 172.21.2.102:0/2353080255 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fbc000d30 con 0x7f6fc40f20e0 2021-02-10T11:24:13.894 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:13.879 7f6fb7fff700 1 -- 172.21.2.102:0/2353080255 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6fbc00b870 con 0x7f6fc40f20e0 2021-02-10T11:24:13.895 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:13.879 7f6fcb50b700 1 -- 172.21.2.102:0/2353080255 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6fc40f20e0 msgr2=0x7f6fc40f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:13.895 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:13.879 7f6fcb50b700 1 --2- 172.21.2.102:0/2353080255 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6fc40f20e0 0x7f6fc40f0200 secure :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 rx=0x7f6fbc006e30 tx=0x7f6fbc009e60).stop 2021-02-10T11:24:13.895 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:13.880 7f6fcb50b700 1 -- 172.21.2.102:0/2353080255 shutdown_connections 2021-02-10T11:24:13.895 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:13.880 7f6fcb50b700 1 --2- 172.21.2.102:0/2353080255 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6fc40f20e0 0x7f6fc40f0200 unknown :-1 s=CLOSED pgs=1325 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:13.895 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:13.880 7f6fcb50b700 1 --2- 172.21.2.102:0/2353080255 >> v2:172.21.2.113:3300/0 conn(0x7f6fc41b0470 0x7f6fc41b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:13.895 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:13.880 7f6fcb50b700 1 -- 172.21.2.102:0/2353080255 >> 172.21.2.102:0/2353080255 conn(0x7f6fc40ce3e0 msgr2=0x7f6fc40e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:13.895 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:13.880 7f6fcb50b700 1 -- 172.21.2.102:0/2353080255 shutdown_connections 2021-02-10T11:24:13.895 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:13.880 7f6fcb50b700 1 -- 172.21.2.102:0/2353080255 wait complete. 2021-02-10T11:24:13.897 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:13.895 7f6fcb50b700 1 Processor -- start 2021-02-10T11:24:13.897 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:13.895 7f6fcb50b700 1 -- start start 2021-02-10T11:24:13.897 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:13.895 7f6fcb50b700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6fc41ac0b0 0x7f6fc40ad920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:13.897 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:13.895 7f6fcb50b700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6fc40adf40 0x7f6fc41b7040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:13.897 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:13.895 7f6fcb50b700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6fc41b7660 0x7f6fc41b7a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:13.897 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:13.895 7f6fcb50b700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fc41b8080 con 0x7f6fc41b7660 2021-02-10T11:24:13.897 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:13.895 7f6fcb50b700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fc41b8270 con 0x7f6fc40adf40 2021-02-10T11:24:13.897 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:13.895 7f6fcb50b700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fc418d950 con 0x7f6fc41ac0b0 2021-02-10T11:24:13.897 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:13.895 7f6fc9b28700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6fc41b7660 0x7f6fc41b7a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:13.897 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:13.895 7f6fc9b28700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6fc41b7660 0x7f6fc41b7a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:57026/0 (socket says 172.21.2.102:57026) 2021-02-10T11:24:13.897 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:13.895 7f6fc9b28700 1 -- 172.21.2.102:0/3691170709 learned_addr learned my addr 172.21.2.102:0/3691170709 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:24:13.898 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:13.896 7f6fc9b28700 1 -- 172.21.2.102:0/3691170709 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6fc41ac0b0 msgr2=0x7f6fc40ad920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:13.898 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:13.896 7f6fc9b28700 1 --2- 172.21.2.102:0/3691170709 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6fc41ac0b0 0x7f6fc40ad920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:13.898 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:13.896 7f6fc9b28700 1 -- 172.21.2.102:0/3691170709 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6fc40adf40 msgr2=0x7f6fc41b7040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:13.898 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:13.896 7f6fc9b28700 1 --2- 172.21.2.102:0/3691170709 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6fc40adf40 0x7f6fc41b7040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:13.898 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:13.896 7f6fc9b28700 1 -- 172.21.2.102:0/3691170709 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fc418db70 con 0x7f6fc41b7660 2021-02-10T11:24:13.899 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:13.896 7f6fc9b28700 1 --2- 172.21.2.102:0/3691170709 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6fc41b7660 0x7f6fc41b7a60 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 rx=0x7f6fbc0099e0 tx=0x7f6fbc00bd40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:13.899 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:13.896 7f6fb5ffb700 1 -- 172.21.2.102:0/3691170709 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6fbc009750 con 0x7f6fc41b7660 2021-02-10T11:24:13.899 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:13.896 7f6fb5ffb700 1 -- 172.21.2.102:0/3691170709 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fbc000d90 con 0x7f6fc41b7660 2021-02-10T11:24:13.899 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:13.896 7f6fb5ffb700 1 -- 172.21.2.102:0/3691170709 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6fbc001700 con 0x7f6fc41b7660 2021-02-10T11:24:13.899 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:13.896 7f6fcb50b700 1 -- 172.21.2.102:0/3691170709 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fc418db70 con 0x7f6fc41b7660 2021-02-10T11:24:13.899 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:13.896 7f6fcb50b700 1 -- 172.21.2.102:0/3691170709 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fc418e100 con 0x7f6fc41b7660 2021-02-10T11:24:13.899 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:13.897 7f6fb5ffb700 1 -- 172.21.2.102:0/3691170709 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f6fbc001270 con 0x7f6fc41b7660 2021-02-10T11:24:13.899 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:13.897 7f6fb5ffb700 1 --2- 172.21.2.102:0/3691170709 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6fa40186c0 0x7f6fa401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:13.899 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:13.897 7f6fb5ffb700 1 -- 172.21.2.102:0/3691170709 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f6fbc032920 con 0x7f6fc41b7660 2021-02-10T11:24:13.899 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:13.897 7f6fc9327700 1 --2- 172.21.2.102:0/3691170709 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6fa40186c0 0x7f6fa401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:13.899 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:13.897 7f6fc9327700 1 --2- 172.21.2.102:0/3691170709 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6fa40186c0 0x7f6fa401abc0 crc :-1 s=READY pgs=621 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:13.927 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:13.928 7f6fcb50b700 1 -- 172.21.2.102:0/3691170709 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fc41059e0 con 0x7f6fc41b7660 2021-02-10T11:24:13.929 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:13.930 7f6fb5ffb700 1 -- 172.21.2.102:0/3691170709 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f6fbc009700 con 0x7f6fc41b7660 2021-02-10T11:24:14.059 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:14.061 7f6fcb50b700 1 -- 172.21.2.102:0/3691170709 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f6fc40f0b50 con 0x7f6fc41b7660 2021-02-10T11:24:14.062 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:14.064 7f6fb5ffb700 1 -- 172.21.2.102:0/3691170709 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.0 v29) v1 ==== 184+0+0 (secure 0 0 0) 0x7f6fbc02e020 con 0x7f6fc41b7660 2021-02-10T11:24:14.066 INFO:teuthology.orchestra.run.gibba002.stderr:updated caps for client.0 2021-02-10T11:24:14.068 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:14.069 7f6fcb50b700 1 -- 172.21.2.102:0/3691170709 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6fa40186c0 msgr2=0x7f6fa401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:14.068 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:14.069 7f6fcb50b700 1 --2- 172.21.2.102:0/3691170709 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6fa40186c0 0x7f6fa401abc0 crc :-1 s=READY pgs=621 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:14.068 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:14.069 7f6fcb50b700 1 -- 172.21.2.102:0/3691170709 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6fc41b7660 msgr2=0x7f6fc41b7a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:14.068 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:14.069 7f6fcb50b700 1 --2- 172.21.2.102:0/3691170709 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6fc41b7660 0x7f6fc41b7a60 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 rx=0x7f6fbc0099e0 tx=0x7f6fbc00bd40).stop 2021-02-10T11:24:14.069 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:14.070 7f6fcb50b700 1 -- 172.21.2.102:0/3691170709 shutdown_connections 2021-02-10T11:24:14.069 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:14.070 7f6fcb50b700 1 --2- 172.21.2.102:0/3691170709 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6fc41b7660 0x7f6fc41b7a60 unknown :-1 s=CLOSED pgs=1326 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:14.069 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:14.070 7f6fcb50b700 1 --2- 172.21.2.102:0/3691170709 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6fa40186c0 0x7f6fa401abc0 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:14.069 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:14.070 7f6fcb50b700 1 --2- 172.21.2.102:0/3691170709 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6fc40adf40 0x7f6fc41b7040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:14.069 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:14.070 7f6fcb50b700 1 --2- 172.21.2.102:0/3691170709 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6fc41ac0b0 0x7f6fc40ad920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:14.069 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:14.070 7f6fcb50b700 1 -- 172.21.2.102:0/3691170709 >> 172.21.2.102:0/3691170709 conn(0x7f6fc41a37a0 msgr2=0x7f6fc418c230 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:14.069 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:14.070 7f6fcb50b700 1 -- 172.21.2.102:0/3691170709 shutdown_connections 2021-02-10T11:24:14.069 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:14.070 7f6fcb50b700 1 -- 172.21.2.102:0/3691170709 wait complete. 2021-02-10T11:24:14.077 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.1 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:24:14.516 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:14.500 7fca2cea4700 1 Processor -- start 2021-02-10T11:24:14.516 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:14.500 7fca2cea4700 1 -- start start 2021-02-10T11:24:14.516 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:14.500 7fca2cea4700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fca281b0870 0x7fca280f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:14.516 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:14.500 7fca2cea4700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fca280efe00 0x7fca280f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:14.516 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:14.500 7fca2cea4700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca281b0c70 con 0x7fca281b0470 2021-02-10T11:24:14.516 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:14.500 7fca2cea4700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca280f0b50 con 0x7fca280efe00 2021-02-10T11:24:14.516 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:14.500 7fca2cea4700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca280f0820 con 0x7fca281b0870 2021-02-10T11:24:14.516 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:14.501 7fca26b13700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fca281b0470 0x7fca280ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:49254/0 (socket says 172.21.2.102:49254) 2021-02-10T11:24:14.516 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:14.501 7fca26b13700 1 -- 172.21.2.102:0/2737678338 learned_addr learned my addr 172.21.2.102:0/2737678338 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:24:14.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:14.501 7fca26312700 1 --2- 172.21.2.102:0/2737678338 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fca281b0870 0x7fca280f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:14.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:14.501 7fca27314700 1 --2- 172.21.2.102:0/2737678338 >> v2:172.21.2.113:3300/0 conn(0x7fca280efe00 0x7fca280f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:14.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:14.501 7fca25b11700 1 -- 172.21.2.102:0/2737678338 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3218786830 0 0) 0x7fca18000ce0 con 0x7fca281b0470 2021-02-10T11:24:14.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:14.501 7fca25b11700 1 -- 172.21.2.102:0/2737678338 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca14002fe0 con 0x7fca281b0470 2021-02-10T11:24:14.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:14.501 7fca26312700 1 -- 172.21.2.102:0/2737678338 >> v2:172.21.2.113:3300/0 conn(0x7fca280efe00 msgr2=0x7fca280f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:24:14.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:14.501 7fca26312700 1 --2- 172.21.2.102:0/2737678338 >> v2:172.21.2.113:3300/0 conn(0x7fca280efe00 0x7fca280f0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:14.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:14.501 7fca26312700 1 -- 172.21.2.102:0/2737678338 >> v1:172.21.2.102:6789/0 conn(0x7fca281b0470 legacy=0x7fca280ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:14.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:14.501 7fca26312700 1 -- 172.21.2.102:0/2737678338 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca280e52e0 con 0x7fca281b0870 2021-02-10T11:24:14.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:14.501 7fca27314700 1 --2- 172.21.2.102:0/2737678338 >> v2:172.21.2.113:3300/0 conn(0x7fca280efe00 0x7fca280f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:24:14.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:14.501 7fca26312700 1 --2- 172.21.2.102:0/2737678338 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fca281b0870 0x7fca280f20e0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7fca10006650 tx=0x7fca10009a80).ready entity=mon.2 client_cookie=23c60a0e2e64ba0a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:14.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:14.502 7fca25b11700 1 -- 172.21.2.102:0/2737678338 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fca10006390 con 0x7fca281b0870 2021-02-10T11:24:14.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:14.502 7fca25b11700 1 -- 172.21.2.102:0/2737678338 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca100009b0 con 0x7fca281b0870 2021-02-10T11:24:14.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:14.502 7fca25b11700 1 -- 172.21.2.102:0/2737678338 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fca1000c2b0 con 0x7fca281b0870 2021-02-10T11:24:14.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:14.502 7fca2cea4700 1 -- 172.21.2.102:0/2737678338 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fca281b0870 msgr2=0x7fca280f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:14.518 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:14.502 7fca2cea4700 1 --2- 172.21.2.102:0/2737678338 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fca281b0870 0x7fca280f20e0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7fca10006650 tx=0x7fca10009a80).stop 2021-02-10T11:24:14.518 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:14.502 7fca2cea4700 1 -- 172.21.2.102:0/2737678338 shutdown_connections 2021-02-10T11:24:14.518 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:14.502 7fca2cea4700 1 --2- 172.21.2.102:0/2737678338 >> v2:172.21.2.113:3300/0 conn(0x7fca280efe00 0x7fca280f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:14.518 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:14.502 7fca2cea4700 1 --2- 172.21.2.102:0/2737678338 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fca281b0870 0x7fca280f20e0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:14.518 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:14.502 7fca2cea4700 1 -- 172.21.2.102:0/2737678338 >> 172.21.2.102:0/2737678338 conn(0x7fca280ce3e0 msgr2=0x7fca280e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:14.518 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:14.502 7fca2cea4700 1 -- 172.21.2.102:0/2737678338 shutdown_connections 2021-02-10T11:24:14.518 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:14.502 7fca2cea4700 1 -- 172.21.2.102:0/2737678338 wait complete. 2021-02-10T11:24:14.518 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:14.517 7fca2cea4700 1 Processor -- start 2021-02-10T11:24:14.518 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:14.517 7fca2cea4700 1 -- start start 2021-02-10T11:24:14.519 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:14.518 7fca2cea4700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fca281a0480 0x7fca2808e840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:14.519 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:14.518 7fca2cea4700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fca2810a4b0 0x7fca2808ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:14.519 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:14.518 7fca2cea4700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fca2818ebf0 0x7fca2818eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:14.520 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:14.518 7fca2cea4700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca2818f720 con 0x7fca281a0480 2021-02-10T11:24:14.520 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:14.518 7fca2cea4700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca2818f940 con 0x7fca2818ebf0 2021-02-10T11:24:14.520 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:14.518 7fca2cea4700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca2818fb60 con 0x7fca2810a4b0 2021-02-10T11:24:14.520 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:14.518 7fca26b13700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fca281a0480 0x7fca2808e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:14.520 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:14.518 7fca26b13700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fca281a0480 0x7fca2808e840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:57036/0 (socket says 172.21.2.102:57036) 2021-02-10T11:24:14.520 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:14.518 7fca26b13700 1 -- 172.21.2.102:0/1571370662 learned_addr learned my addr 172.21.2.102:0/1571370662 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:24:14.520 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:14.518 7fca26b13700 1 -- 172.21.2.102:0/1571370662 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fca2810a4b0 msgr2=0x7fca2808ee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:14.520 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:14.518 7fca26b13700 1 --2- 172.21.2.102:0/1571370662 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fca2810a4b0 0x7fca2808ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:14.520 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:14.518 7fca26b13700 1 -- 172.21.2.102:0/1571370662 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fca2818ebf0 msgr2=0x7fca2818eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:14.520 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:14.518 7fca26b13700 1 --2- 172.21.2.102:0/1571370662 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fca2818ebf0 0x7fca2818eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:14.520 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:14.518 7fca26b13700 1 -- 172.21.2.102:0/1571370662 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca2818fd80 con 0x7fca281a0480 2021-02-10T11:24:14.520 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:14.518 7fca26b13700 1 --2- 172.21.2.102:0/1571370662 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fca281a0480 0x7fca2808e840 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 rx=0x7fca1800b5c0 tx=0x7fca1800b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:14.520 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:14.519 7fca1f7fe700 1 -- 172.21.2.102:0/1571370662 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fca18006030 con 0x7fca281a0480 2021-02-10T11:24:14.520 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:14.519 7fca1f7fe700 1 -- 172.21.2.102:0/1571370662 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca180043a0 con 0x7fca281a0480 2021-02-10T11:24:14.521 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:14.519 7fca2cea4700 1 -- 172.21.2.102:0/1571370662 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca2818fd80 con 0x7fca281a0480 2021-02-10T11:24:14.521 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:14.519 7fca1f7fe700 1 -- 172.21.2.102:0/1571370662 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fca180048c0 con 0x7fca281a0480 2021-02-10T11:24:14.521 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:14.519 7fca2cea4700 1 -- 172.21.2.102:0/1571370662 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca2808fdb0 con 0x7fca281a0480 2021-02-10T11:24:14.521 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:14.519 7fca1f7fe700 1 -- 172.21.2.102:0/1571370662 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fca18004580 con 0x7fca281a0480 2021-02-10T11:24:14.521 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:14.519 7fca1f7fe700 1 --2- 172.21.2.102:0/1571370662 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fca0c018730 0x7fca0c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:14.521 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:14.519 7fca1f7fe700 1 -- 172.21.2.102:0/1571370662 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fca18032660 con 0x7fca281a0480 2021-02-10T11:24:14.521 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:14.519 7fca26312700 1 --2- 172.21.2.102:0/1571370662 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fca0c018730 0x7fca0c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:14.522 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:14.520 7fca26312700 1 --2- 172.21.2.102:0/1571370662 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fca0c018730 0x7fca0c01abc0 crc :-1 s=READY pgs=622 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:14.548 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:14.550 7fca2cea4700 1 -- 172.21.2.102:0/1571370662 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca281059f0 con 0x7fca281a0480 2021-02-10T11:24:14.551 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:14.553 7fca1f7fe700 1 -- 172.21.2.102:0/1571370662 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fca180045b0 con 0x7fca281a0480 2021-02-10T11:24:14.681 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:14.683 7fca2cea4700 1 -- 172.21.2.102:0/1571370662 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7fca280f0b50 con 0x7fca281a0480 2021-02-10T11:24:14.685 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:14.686 7fca1f7fe700 1 -- 172.21.2.102:0/1571370662 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.1 v30) v1 ==== 184+0+0 (secure 0 0 0) 0x7fca180043a0 con 0x7fca281a0480 2021-02-10T11:24:14.688 INFO:teuthology.orchestra.run.gibba002.stderr:updated caps for client.1 2021-02-10T11:24:14.690 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:14.692 7fca2cea4700 1 -- 172.21.2.102:0/1571370662 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fca0c018730 msgr2=0x7fca0c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:14.690 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:14.692 7fca2cea4700 1 --2- 172.21.2.102:0/1571370662 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fca0c018730 0x7fca0c01abc0 crc :-1 s=READY pgs=622 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:14.690 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:14.692 7fca2cea4700 1 -- 172.21.2.102:0/1571370662 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fca281a0480 msgr2=0x7fca2808e840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:14.690 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:14.692 7fca2cea4700 1 --2- 172.21.2.102:0/1571370662 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fca281a0480 0x7fca2808e840 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 rx=0x7fca1800b5c0 tx=0x7fca1800b960).stop 2021-02-10T11:24:14.690 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:14.692 7fca2cea4700 1 -- 172.21.2.102:0/1571370662 shutdown_connections 2021-02-10T11:24:14.690 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:14.692 7fca2cea4700 1 --2- 172.21.2.102:0/1571370662 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fca0c018730 0x7fca0c01abc0 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:14.691 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:14.692 7fca2cea4700 1 --2- 172.21.2.102:0/1571370662 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fca2818ebf0 0x7fca2818eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:14.691 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:14.692 7fca2cea4700 1 --2- 172.21.2.102:0/1571370662 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fca2810a4b0 0x7fca2808ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:14.691 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:14.692 7fca2cea4700 1 --2- 172.21.2.102:0/1571370662 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fca281a0480 0x7fca2808e840 unknown :-1 s=CLOSED pgs=1328 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:14.691 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:14.692 7fca2cea4700 1 -- 172.21.2.102:0/1571370662 >> 172.21.2.102:0/1571370662 conn(0x7fca2809cd10 msgr2=0x7fca28093a20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:14.691 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:14.692 7fca2cea4700 1 -- 172.21.2.102:0/1571370662 shutdown_connections 2021-02-10T11:24:14.691 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:14.692 7fca2cea4700 1 -- 172.21.2.102:0/1571370662 wait complete. 2021-02-10T11:24:14.699 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.2 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:24:15.137 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:15.122 7f82cb919700 1 Processor -- start 2021-02-10T11:24:15.138 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:15.122 7f82cb919700 1 -- start start 2021-02-10T11:24:15.138 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:15.123 7f82cb919700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f82c41b0470 0x7f82c41b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:15.138 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:15.123 7f82cb919700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f82c40ef610 0x7f82c40efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:15.138 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:15.123 7f82cb919700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82c40f0b50 con 0x7f82c40f0030 2021-02-10T11:24:15.138 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:15.123 7f82cb919700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82c40e52e0 con 0x7f82c40ef610 2021-02-10T11:24:15.138 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:15.123 7f82cb919700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82c40e5490 con 0x7f82c41b0470 2021-02-10T11:24:15.138 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:15.123 7f82c9f36700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f82c40f0030 0x7f82c40f10d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:49268/0 (socket says 172.21.2.102:49268) 2021-02-10T11:24:15.138 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:15.123 7f82c9f36700 1 -- 172.21.2.102:0/2660636819 learned_addr learned my addr 172.21.2.102:0/2660636819 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:24:15.138 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:15.123 7f82c8f34700 1 --2- 172.21.2.102:0/2660636819 >> v2:172.21.2.113:3300/0 conn(0x7f82c40ef610 0x7f82c40efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:15.139 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:15.123 7f82c9735700 1 --2- 172.21.2.102:0/2660636819 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f82c41b0470 0x7f82c41b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:15.139 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:15.123 7f82bbfff700 1 -- 172.21.2.102:0/2660636819 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 874208006 0 0) 0x7f82bc001500 con 0x7f82c40f0030 2021-02-10T11:24:15.139 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:15.123 7f82bbfff700 1 -- 172.21.2.102:0/2660636819 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82b0002fe0 con 0x7f82c40f0030 2021-02-10T11:24:15.139 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:15.124 7f82c8f34700 1 -- 172.21.2.102:0/2660636819 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f82c41b0470 msgr2=0x7f82c41b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:24:15.139 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:15.124 7f82c8f34700 1 --2- 172.21.2.102:0/2660636819 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f82c41b0470 0x7f82c41b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:15.139 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:15.124 7f82c8f34700 1 -- 172.21.2.102:0/2660636819 >> v1:172.21.2.102:6789/0 conn(0x7f82c40f0030 legacy=0x7f82c40f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:15.139 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:15.124 7f82c8f34700 1 -- 172.21.2.102:0/2660636819 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82c40f0430 con 0x7f82c40ef610 2021-02-10T11:24:15.139 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:15.124 7f82c8f34700 1 --2- 172.21.2.102:0/2660636819 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f82c40ef610 0x7f82c40efa10 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 rx=0x7f82c0006650 tx=0x7f82c0009a80).ready entity=mon.1 client_cookie=7d05f3fd2f950844 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:15.139 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:15.124 7f82bbfff700 1 -- 172.21.2.102:0/2660636819 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f82c000c0c0 con 0x7f82c40ef610 2021-02-10T11:24:15.139 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:15.124 7f82bbfff700 1 -- 172.21.2.102:0/2660636819 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82c000c340 con 0x7f82c40ef610 2021-02-10T11:24:15.139 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:15.124 7f82bbfff700 1 -- 172.21.2.102:0/2660636819 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f82c00009b0 con 0x7f82c40ef610 2021-02-10T11:24:15.139 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:15.124 7f82cb919700 1 -- 172.21.2.102:0/2660636819 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f82c40ef610 msgr2=0x7f82c40efa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:15.139 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:15.124 7f82cb919700 1 --2- 172.21.2.102:0/2660636819 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f82c40ef610 0x7f82c40efa10 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 rx=0x7f82c0006650 tx=0x7f82c0009a80).stop 2021-02-10T11:24:15.140 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:15.125 7f82cb919700 1 -- 172.21.2.102:0/2660636819 shutdown_connections 2021-02-10T11:24:15.140 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:15.125 7f82cb919700 1 --2- 172.21.2.102:0/2660636819 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f82c40ef610 0x7f82c40efa10 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:15.140 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:15.125 7f82cb919700 1 --2- 172.21.2.102:0/2660636819 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f82c41b0470 0x7f82c41b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:15.140 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:15.125 7f82cb919700 1 -- 172.21.2.102:0/2660636819 >> 172.21.2.102:0/2660636819 conn(0x7f82c40ce3e0 msgr2=0x7f82c40e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:15.140 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:15.125 7f82cb919700 1 -- 172.21.2.102:0/2660636819 shutdown_connections 2021-02-10T11:24:15.140 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:15.125 7f82cb919700 1 -- 172.21.2.102:0/2660636819 wait complete. 2021-02-10T11:24:15.141 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:15.140 7f82cb919700 1 Processor -- start 2021-02-10T11:24:15.141 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:15.140 7f82cb919700 1 -- start start 2021-02-10T11:24:15.142 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:15.140 7f82cb919700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f82c41a0480 0x7f82c408e840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:15.142 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:15.140 7f82cb919700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f82c410a4b0 0x7f82c408ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:15.142 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:15.140 7f82cb919700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f82c418ebf0 0x7f82c418eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:15.142 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:15.140 7f82cb919700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82c418f720 con 0x7f82c41a0480 2021-02-10T11:24:15.142 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:15.140 7f82cb919700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82c418f940 con 0x7f82c418ebf0 2021-02-10T11:24:15.142 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:15.140 7f82cb919700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82c418fb60 con 0x7f82c410a4b0 2021-02-10T11:24:15.142 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:15.140 7f82c9735700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f82c41a0480 0x7f82c408e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:15.142 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:15.140 7f82c9735700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f82c41a0480 0x7f82c408e840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:57050/0 (socket says 172.21.2.102:57050) 2021-02-10T11:24:15.142 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:15.140 7f82c9735700 1 -- 172.21.2.102:0/2906087965 learned_addr learned my addr 172.21.2.102:0/2906087965 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:24:15.142 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:15.141 7f82c9735700 1 -- 172.21.2.102:0/2906087965 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f82c410a4b0 msgr2=0x7f82c408ee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:15.142 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:15.141 7f82c9735700 1 --2- 172.21.2.102:0/2906087965 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f82c410a4b0 0x7f82c408ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:15.142 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:15.141 7f82c9735700 1 -- 172.21.2.102:0/2906087965 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f82c418ebf0 msgr2=0x7f82c418eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:15.143 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:15.141 7f82c9735700 1 --2- 172.21.2.102:0/2906087965 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f82c418ebf0 0x7f82c418eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:15.143 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:15.141 7f82c9735700 1 -- 172.21.2.102:0/2906087965 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82c418fd80 con 0x7f82c41a0480 2021-02-10T11:24:15.143 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:15.141 7f82c9735700 1 --2- 172.21.2.102:0/2906087965 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f82c41a0480 0x7f82c408e840 secure :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 rx=0x7f82b4006f60 tx=0x7f82b400b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:15.143 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:15.141 7f82b9ffb700 1 -- 172.21.2.102:0/2906087965 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f82b400bd60 con 0x7f82c41a0480 2021-02-10T11:24:15.143 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:15.141 7f82b9ffb700 1 -- 172.21.2.102:0/2906087965 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82b4000c90 con 0x7f82c41a0480 2021-02-10T11:24:15.143 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:15.141 7f82b9ffb700 1 -- 172.21.2.102:0/2906087965 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f82b4002220 con 0x7f82c41a0480 2021-02-10T11:24:15.143 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:15.141 7f82cb919700 1 -- 172.21.2.102:0/2906087965 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82c418fd80 con 0x7f82c41a0480 2021-02-10T11:24:15.143 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:15.141 7f82cb919700 1 -- 172.21.2.102:0/2906087965 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82c408fdb0 con 0x7f82c41a0480 2021-02-10T11:24:15.143 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:15.142 7f82b9ffb700 1 -- 172.21.2.102:0/2906087965 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f82b4002020 con 0x7f82c41a0480 2021-02-10T11:24:15.143 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:15.142 7f82b9ffb700 1 --2- 172.21.2.102:0/2906087965 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f82a40186c0 0x7f82a401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:15.143 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:15.142 7f82b9ffb700 1 -- 172.21.2.102:0/2906087965 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f82b40325b0 con 0x7f82c41a0480 2021-02-10T11:24:15.143 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:15.142 7f82c8f34700 1 --2- 172.21.2.102:0/2906087965 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f82a40186c0 0x7f82a401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:15.143 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:15.142 7f82c8f34700 1 --2- 172.21.2.102:0/2906087965 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f82a40186c0 0x7f82a401abc0 crc :-1 s=READY pgs=623 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:15.171 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:15.173 7f82cb919700 1 -- 172.21.2.102:0/2906087965 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82c41059f0 con 0x7f82c41a0480 2021-02-10T11:24:15.173 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:15.175 7f82b9ffb700 1 -- 172.21.2.102:0/2906087965 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f82b40009b0 con 0x7f82c41a0480 2021-02-10T11:24:15.307 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:15.307 7f82cb919700 1 -- 172.21.2.102:0/2906087965 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f82c41059f0 con 0x7f82c41a0480 2021-02-10T11:24:15.310 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:15.311 7f82b9ffb700 1 -- 172.21.2.102:0/2906087965 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.2 v31) v1 ==== 184+0+0 (secure 0 0 0) 0x7f82b4002020 con 0x7f82c41a0480 2021-02-10T11:24:15.313 INFO:teuthology.orchestra.run.gibba002.stderr:updated caps for client.2 2021-02-10T11:24:15.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:15.316 7f82cb919700 1 -- 172.21.2.102:0/2906087965 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f82a40186c0 msgr2=0x7f82a401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:15.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:15.316 7f82cb919700 1 --2- 172.21.2.102:0/2906087965 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f82a40186c0 0x7f82a401abc0 crc :-1 s=READY pgs=623 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:15.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:15.316 7f82cb919700 1 -- 172.21.2.102:0/2906087965 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f82c41a0480 msgr2=0x7f82c408e840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:15.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:15.316 7f82cb919700 1 --2- 172.21.2.102:0/2906087965 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f82c41a0480 0x7f82c408e840 secure :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 rx=0x7f82b4006f60 tx=0x7f82b400b970).stop 2021-02-10T11:24:15.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:15.317 7f82cb919700 1 -- 172.21.2.102:0/2906087965 shutdown_connections 2021-02-10T11:24:15.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:15.317 7f82cb919700 1 --2- 172.21.2.102:0/2906087965 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f82a40186c0 0x7f82a401abc0 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:15.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:15.317 7f82cb919700 1 --2- 172.21.2.102:0/2906087965 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f82c418ebf0 0x7f82c418eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:15.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:15.317 7f82cb919700 1 --2- 172.21.2.102:0/2906087965 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f82c410a4b0 0x7f82c408ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:15.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:15.317 7f82cb919700 1 --2- 172.21.2.102:0/2906087965 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f82c41a0480 0x7f82c408e840 unknown :-1 s=CLOSED pgs=1330 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:15.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:15.317 7f82cb919700 1 -- 172.21.2.102:0/2906087965 >> 172.21.2.102:0/2906087965 conn(0x7f82c409cd10 msgr2=0x7f82c4093a20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:15.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:15.317 7f82cb919700 1 -- 172.21.2.102:0/2906087965 shutdown_connections 2021-02-10T11:24:15.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:15.317 7f82cb919700 1 -- 172.21.2.102:0/2906087965 wait complete. 2021-02-10T11:24:15.324 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.3 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:24:15.761 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:15.746 7f9b4db8c700 1 Processor -- start 2021-02-10T11:24:15.761 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:15.746 7f9b4db8c700 1 -- start start 2021-02-10T11:24:15.761 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:15.746 7f9b4db8c700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f9b481b0470 0x7f9b481b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:15.761 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:15.746 7f9b4db8c700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9b480ef610 0x7f9b480efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:15.761 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:15.746 7f9b4db8c700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f9b480f0030 0x7f9b480f0430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:15.761 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:15.746 7f9b4db8c700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b480e5ea0 con 0x7f9b480f0030 2021-02-10T11:24:15.761 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:15.746 7f9b4db8c700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b480f0b50 con 0x7f9b481b0470 2021-02-10T11:24:15.761 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:15.746 7f9b4db8c700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b480e52e0 con 0x7f9b480ef610 2021-02-10T11:24:15.762 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:15.746 7f9b47fff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f9b480f0030 0x7f9b480f0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:15.762 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:15.746 7f9b47fff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f9b480f0030 0x7f9b480f0430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:57058/0 (socket says 172.21.2.102:57058) 2021-02-10T11:24:15.762 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:15.746 7f9b47fff700 1 -- 172.21.2.102:0/505038567 learned_addr learned my addr 172.21.2.102:0/505038567 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:24:15.762 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:15.747 7f9b46ffd700 1 --2- 172.21.2.102:0/505038567 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9b480ef610 0x7f9b480efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:15.762 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:15.747 7f9b477fe700 1 --2- 172.21.2.102:0/505038567 >> v2:172.21.2.113:3300/0 conn(0x7f9b481b0470 0x7f9b481b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:15.762 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:15.747 7f9b47fff700 1 -- 172.21.2.102:0/505038567 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9b480ef610 msgr2=0x7f9b480efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:24:15.762 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:15.747 7f9b47fff700 1 --2- 172.21.2.102:0/505038567 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9b480ef610 0x7f9b480efa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:15.762 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:15.747 7f9b47fff700 1 -- 172.21.2.102:0/505038567 >> v2:172.21.2.113:3300/0 conn(0x7f9b481b0470 msgr2=0x7f9b481b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:24:15.762 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:15.747 7f9b47fff700 1 --2- 172.21.2.102:0/505038567 >> v2:172.21.2.113:3300/0 conn(0x7f9b481b0470 0x7f9b481b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:15.762 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:15.747 7f9b47fff700 1 -- 172.21.2.102:0/505038567 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b480f10d0 con 0x7f9b480f0030 2021-02-10T11:24:15.762 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:15.747 7f9b477fe700 1 --2- 172.21.2.102:0/505038567 >> v2:172.21.2.113:3300/0 conn(0x7f9b481b0470 0x7f9b481b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:24:15.762 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:15.747 7f9b47fff700 1 --2- 172.21.2.102:0/505038567 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9b480f0030 0x7f9b480f0430 secure :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 rx=0x7f9b40006f00 tx=0x7f9b40009f10).ready entity=mon.0 client_cookie=adae56dc6adab095 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:15.763 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:15.747 7f9b467fc700 1 -- 172.21.2.102:0/505038567 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9b4000b3b0 con 0x7f9b480f0030 2021-02-10T11:24:15.763 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:15.747 7f9b467fc700 1 -- 172.21.2.102:0/505038567 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b4000b630 con 0x7f9b480f0030 2021-02-10T11:24:15.763 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:15.747 7f9b467fc700 1 -- 172.21.2.102:0/505038567 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9b4000bb50 con 0x7f9b480f0030 2021-02-10T11:24:15.763 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:15.747 7f9b4db8c700 1 -- 172.21.2.102:0/505038567 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9b480f0030 msgr2=0x7f9b480f0430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:15.763 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:15.747 7f9b4db8c700 1 --2- 172.21.2.102:0/505038567 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9b480f0030 0x7f9b480f0430 secure :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 rx=0x7f9b40006f00 tx=0x7f9b40009f10).stop 2021-02-10T11:24:15.763 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:15.748 7f9b4db8c700 1 -- 172.21.2.102:0/505038567 shutdown_connections 2021-02-10T11:24:15.763 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:15.748 7f9b4db8c700 1 --2- 172.21.2.102:0/505038567 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9b480ef610 0x7f9b480efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:15.763 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:15.748 7f9b4db8c700 1 --2- 172.21.2.102:0/505038567 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9b480f0030 0x7f9b480f0430 unknown :-1 s=CLOSED pgs=1331 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:15.763 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:15.748 7f9b4db8c700 1 --2- 172.21.2.102:0/505038567 >> v2:172.21.2.113:3300/0 conn(0x7f9b481b0470 0x7f9b481b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:15.763 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:15.748 7f9b4db8c700 1 -- 172.21.2.102:0/505038567 >> 172.21.2.102:0/505038567 conn(0x7f9b480ce3e0 msgr2=0x7f9b480e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:15.763 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:15.748 7f9b4db8c700 1 -- 172.21.2.102:0/505038567 shutdown_connections 2021-02-10T11:24:15.763 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:15.748 7f9b4db8c700 1 -- 172.21.2.102:0/505038567 wait complete. 2021-02-10T11:24:15.765 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:15.763 7f9b4db8c700 1 Processor -- start 2021-02-10T11:24:15.765 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:15.763 7f9b4db8c700 1 -- start start 2021-02-10T11:24:15.765 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:15.763 7f9b4db8c700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9b48090660 0x7f9b4808fae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:15.765 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:15.763 7f9b4db8c700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f9b48090100 0x7f9b48090b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:15.765 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:15.763 7f9b4db8c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9b48091150 0x7f9b480b8850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:15.765 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:15.763 7f9b4db8c700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b48091550 con 0x7f9b48091150 2021-02-10T11:24:15.765 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:15.763 7f9b4db8c700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b48091740 con 0x7f9b48090100 2021-02-10T11:24:15.765 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:15.763 7f9b4db8c700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b480b8e70 con 0x7f9b48090660 2021-02-10T11:24:15.766 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:15.763 7f9b47fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9b48091150 0x7f9b480b8850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:15.766 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:15.764 7f9b47fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9b48091150 0x7f9b480b8850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:57066/0 (socket says 172.21.2.102:57066) 2021-02-10T11:24:15.766 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:15.764 7f9b47fff700 1 -- 172.21.2.102:0/1776601062 learned_addr learned my addr 172.21.2.102:0/1776601062 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:24:15.766 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:15.764 7f9b46ffd700 1 --2- 172.21.2.102:0/1776601062 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f9b48090100 0x7f9b48090b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:15.766 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:15.764 7f9b477fe700 1 --2- 172.21.2.102:0/1776601062 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9b48090660 0x7f9b4808fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:15.766 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:15.764 7f9b47fff700 1 -- 172.21.2.102:0/1776601062 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9b48090660 msgr2=0x7f9b4808fae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:15.766 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:15.764 7f9b47fff700 1 --2- 172.21.2.102:0/1776601062 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9b48090660 0x7f9b4808fae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:15.766 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:15.764 7f9b47fff700 1 -- 172.21.2.102:0/1776601062 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f9b48090100 msgr2=0x7f9b48090b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:15.766 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:15.764 7f9b47fff700 1 --2- 172.21.2.102:0/1776601062 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f9b48090100 0x7f9b48090b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:15.766 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:15.764 7f9b47fff700 1 -- 172.21.2.102:0/1776601062 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b480b9090 con 0x7f9b48091150 2021-02-10T11:24:15.766 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:15.764 7f9b47fff700 1 --2- 172.21.2.102:0/1776601062 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9b48091150 0x7f9b480b8850 secure :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 rx=0x7f9b40009390 tx=0x7f9b4000bd40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:15.766 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:15.764 7f9b3ffff700 1 -- 172.21.2.102:0/1776601062 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9b40009450 con 0x7f9b48091150 2021-02-10T11:24:15.766 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:15.764 7f9b3ffff700 1 -- 172.21.2.102:0/1776601062 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b40006960 con 0x7f9b48091150 2021-02-10T11:24:15.767 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:15.764 7f9b3ffff700 1 -- 172.21.2.102:0/1776601062 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9b4000c4e0 con 0x7f9b48091150 2021-02-10T11:24:15.767 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:15.764 7f9b4db8c700 1 -- 172.21.2.102:0/1776601062 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b480b9090 con 0x7f9b48091150 2021-02-10T11:24:15.767 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:15.764 7f9b4db8c700 1 -- 172.21.2.102:0/1776601062 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b480b9090 con 0x7f9b48091150 2021-02-10T11:24:15.767 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:15.765 7f9b3ffff700 1 -- 172.21.2.102:0/1776601062 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f9b40001270 con 0x7f9b48091150 2021-02-10T11:24:15.767 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:15.765 7f9b3ffff700 1 --2- 172.21.2.102:0/1776601062 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f9b2c018730 0x7f9b2c01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:15.767 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:15.765 7f9b3ffff700 1 -- 172.21.2.102:0/1776601062 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f9b400329c0 con 0x7f9b48091150 2021-02-10T11:24:15.767 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:15.765 7f9b477fe700 1 --2- 172.21.2.102:0/1776601062 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f9b2c018730 0x7f9b2c01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:15.767 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:15.765 7f9b477fe700 1 --2- 172.21.2.102:0/1776601062 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f9b2c018730 0x7f9b2c01ac10 crc :-1 s=READY pgs=624 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:15.795 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:15.796 7f9b4db8c700 1 -- 172.21.2.102:0/1776601062 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b48105a30 con 0x7f9b48091150 2021-02-10T11:24:15.797 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:15.799 7f9b3ffff700 1 -- 172.21.2.102:0/1776601062 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f9b40009450 con 0x7f9b48091150 2021-02-10T11:24:15.928 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:15.929 7f9b4db8c700 1 -- 172.21.2.102:0/1776601062 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.3", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f9b480f0b50 con 0x7f9b48091150 2021-02-10T11:24:15.931 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:15.932 7f9b3ffff700 1 -- 172.21.2.102:0/1776601062 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.3", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.3 v32) v1 ==== 184+0+0 (secure 0 0 0) 0x7f9b40001270 con 0x7f9b48091150 2021-02-10T11:24:15.935 INFO:teuthology.orchestra.run.gibba002.stderr:updated caps for client.3 2021-02-10T11:24:15.936 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:15.937 7f9b4db8c700 1 -- 172.21.2.102:0/1776601062 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f9b2c018730 msgr2=0x7f9b2c01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:15.936 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:15.937 7f9b4db8c700 1 --2- 172.21.2.102:0/1776601062 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f9b2c018730 0x7f9b2c01ac10 crc :-1 s=READY pgs=624 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:15.936 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:15.937 7f9b4db8c700 1 -- 172.21.2.102:0/1776601062 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9b48091150 msgr2=0x7f9b480b8850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:15.936 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:15.937 7f9b4db8c700 1 --2- 172.21.2.102:0/1776601062 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9b48091150 0x7f9b480b8850 secure :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 rx=0x7f9b40009390 tx=0x7f9b4000bd40).stop 2021-02-10T11:24:15.937 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:15.938 7f9b4db8c700 1 -- 172.21.2.102:0/1776601062 shutdown_connections 2021-02-10T11:24:15.937 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:15.938 7f9b4db8c700 1 --2- 172.21.2.102:0/1776601062 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9b48091150 0x7f9b480b8850 unknown :-1 s=CLOSED pgs=1332 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:15.937 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:15.938 7f9b4db8c700 1 --2- 172.21.2.102:0/1776601062 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f9b2c018730 0x7f9b2c01ac10 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:15.937 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:15.938 7f9b4db8c700 1 --2- 172.21.2.102:0/1776601062 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f9b48090100 0x7f9b48090b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:15.937 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:15.938 7f9b4db8c700 1 --2- 172.21.2.102:0/1776601062 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9b48090660 0x7f9b4808fae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:15.937 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:15.938 7f9b4db8c700 1 -- 172.21.2.102:0/1776601062 >> 172.21.2.102:0/1776601062 conn(0x7f9b480b2800 msgr2=0x7f9b48091a50 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:15.937 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:15.938 7f9b4db8c700 1 -- 172.21.2.102:0/1776601062 shutdown_connections 2021-02-10T11:24:15.937 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:15.938 7f9b4db8c700 1 -- 172.21.2.102:0/1776601062 wait complete. 2021-02-10T11:24:15.945 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:24:16.373 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:16.357 7f6b1c510700 1 Processor -- start 2021-02-10T11:24:16.373 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:16.357 7f6b1c510700 1 -- start start 2021-02-10T11:24:16.374 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:16.357 7f6b1c510700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6b140e5070 0x7f6b141b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:16.374 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:16.357 7f6b1c510700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b1407ed80 con 0x7f6b141b3e40 2021-02-10T11:24:16.374 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:16.357 7f6b1c510700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b1407e670 con 0x7f6b140e5470 2021-02-10T11:24:16.374 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:16.357 7f6b1c510700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b1407e940 con 0x7f6b140e5070 2021-02-10T11:24:16.374 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:16.357 7f6b1ab2d700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f6b141b3e40 0x7f6b1407de80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:49294/0 (socket says 172.21.2.102:49294) 2021-02-10T11:24:16.374 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:16.357 7f6b1ab2d700 1 -- 172.21.2.102:0/2745037937 learned_addr learned my addr 172.21.2.102:0/2745037937 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:24:16.374 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:16.358 7f6b1932a700 1 -- 172.21.2.102:0/2745037937 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3802916890 0 0) 0x7f6b0c001500 con 0x7f6b141b3e40 2021-02-10T11:24:16.374 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:16.358 7f6b1932a700 1 -- 172.21.2.102:0/2745037937 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b00002fe0 con 0x7f6b141b3e40 2021-02-10T11:24:16.374 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:16.358 7f6b1a32c700 1 --2- 172.21.2.102:0/2745037937 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6b140e5070 0x7f6b141b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:16.374 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:16.358 7f6b1932a700 1 -- 172.21.2.102:0/2745037937 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 276259767 0 0) 0x7f6b0c001500 con 0x7f6b141b3e40 2021-02-10T11:24:16.374 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:16.358 7f6b1932a700 1 -- 172.21.2.102:0/2745037937 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6b140e5070 msgr2=0x7f6b141b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:24:16.375 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:16.358 7f6b1932a700 1 --2- 172.21.2.102:0/2745037937 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6b140e5070 0x7f6b141b3820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:16.375 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:16.358 7f6b1932a700 1 -- 172.21.2.102:0/2745037937 >> v1:172.21.2.113:6789/0 conn(0x7f6b140e5470 legacy=0x7f6b1407d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:16.375 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:16.358 7f6b1932a700 1 -- 172.21.2.102:0/2745037937 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b1407eb30 con 0x7f6b141b3e40 2021-02-10T11:24:16.375 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:16.358 7f6b1932a700 1 -- 172.21.2.102:0/2745037937 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7f6b0c001a00 con 0x7f6b141b3e40 2021-02-10T11:24:16.375 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:16.358 7f6b1a32c700 1 --2- 172.21.2.102:0/2745037937 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6b140e5070 0x7f6b141b3820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:24:16.375 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:16.358 7f6b1932a700 1 -- 172.21.2.102:0/2745037937 >> v1:172.21.2.102:6789/0 conn(0x7f6b141b3e40 legacy=0x7f6b1407de80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:16.375 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:16.358 7f6b1932a700 1 --2- 172.21.2.102:0/2745037937 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6b00003a70 0x7f6b00005f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:16.375 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:16.358 7f6b1932a700 1 --2- 172.21.2.102:0/2745037937 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6b00006790 0x7f6b00008cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:16.375 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:16.358 7f6b1932a700 1 --2- 172.21.2.102:0/2745037937 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6b000095b0 0x7f6b0000ba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:16.375 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:16.358 7f6b1932a700 1 -- 172.21.2.102:0/2745037937 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b0000c2d0 con 0x7f6b00006790 2021-02-10T11:24:16.375 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:16.358 7f6b1932a700 1 -- 172.21.2.102:0/2745037937 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b0000c4f0 con 0x7f6b00003a70 2021-02-10T11:24:16.375 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:16.358 7f6b1932a700 1 -- 172.21.2.102:0/2745037937 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b0000c710 con 0x7f6b000095b0 2021-02-10T11:24:16.375 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:16.359 7f6b19b2b700 1 --2- 172.21.2.102:0/2745037937 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6b00006790 0x7f6b00008cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:16.375 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:16.359 7f6b1a32c700 1 --2- 172.21.2.102:0/2745037937 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6b00003a70 0x7f6b00005f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:16.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:16.359 7f6b1ab2d700 1 --2- 172.21.2.102:0/2745037937 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6b000095b0 0x7f6b0000ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:16.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:16.359 7f6b19b2b700 1 -- 172.21.2.102:0/2745037937 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6b000095b0 msgr2=0x7f6b0000ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:24:16.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:16.359 7f6b19b2b700 1 --2- 172.21.2.102:0/2745037937 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6b000095b0 0x7f6b0000ba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:16.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:16.359 7f6b19b2b700 1 -- 172.21.2.102:0/2745037937 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6b00003a70 msgr2=0x7f6b00005f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:24:16.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:16.359 7f6b19b2b700 1 --2- 172.21.2.102:0/2745037937 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6b00003a70 0x7f6b00005f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:16.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:16.359 7f6b19b2b700 1 -- 172.21.2.102:0/2745037937 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b0000c9b0 con 0x7f6b00006790 2021-02-10T11:24:16.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:16.359 7f6b1a32c700 1 --2- 172.21.2.102:0/2745037937 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6b00003a70 0x7f6b00005f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:24:16.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:16.359 7f6b19b2b700 1 --2- 172.21.2.102:0/2745037937 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6b00006790 0x7f6b00008cb0 secure :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 rx=0x7f6b0804aad0 tx=0x7f6b0804dad0).ready entity=mon.0 client_cookie=c4c5c0c492d1a0a1 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:16.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:16.359 7f6b1932a700 1 -- 172.21.2.102:0/2745037937 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6b0804b280 con 0x7f6b00006790 2021-02-10T11:24:16.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:16.359 7f6b1932a700 1 -- 172.21.2.102:0/2745037937 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b0804b4c0 con 0x7f6b00006790 2021-02-10T11:24:16.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:16.359 7f6b1932a700 1 -- 172.21.2.102:0/2745037937 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6b0804b9e0 con 0x7f6b00006790 2021-02-10T11:24:16.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:16.359 7f6b1c510700 1 -- 172.21.2.102:0/2745037937 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6b00006790 msgr2=0x7f6b00008cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:16.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:16.359 7f6b1c510700 1 --2- 172.21.2.102:0/2745037937 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6b00006790 0x7f6b00008cb0 secure :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 rx=0x7f6b0804aad0 tx=0x7f6b0804dad0).stop 2021-02-10T11:24:16.377 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:16.359 7f6b1ab2d700 1 -- 172.21.2.102:0/2745037937 reap_dead start 2021-02-10T11:24:16.377 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:16.360 7f6b1c510700 1 -- 172.21.2.102:0/2745037937 shutdown_connections 2021-02-10T11:24:16.377 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:16.360 7f6b1c510700 1 -- 172.21.2.102:0/2745037937 >> 172.21.2.102:0/2745037937 conn(0x7f6b140ce690 msgr2=0x7f6b140e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:16.377 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:16.360 7f6b1c510700 1 -- 172.21.2.102:0/2745037937 shutdown_connections 2021-02-10T11:24:16.377 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:16.360 7f6b1c510700 1 -- 172.21.2.102:0/2745037937 wait complete. 2021-02-10T11:24:16.377 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:16.375 7f6b1c510700 1 Processor -- start 2021-02-10T11:24:16.377 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:16.375 7f6b1c510700 1 -- start start 2021-02-10T11:24:16.378 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:16.375 7f6b1c510700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6b14194000 0x7f6b140b0d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:16.379 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:16.375 7f6b1c510700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6b140b1330 0x7f6b14087050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:16.379 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:16.375 7f6b1c510700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6b14087670 0x7f6b14087a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:16.379 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:16.375 7f6b1c510700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b1409c0e0 con 0x7f6b14087670 2021-02-10T11:24:16.379 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:16.375 7f6b1c510700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b14194440 con 0x7f6b140b1330 2021-02-10T11:24:16.379 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:16.375 7f6b1c510700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b1409c5e0 con 0x7f6b14194000 2021-02-10T11:24:16.379 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:16.375 7f6b1ab2d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6b14087670 0x7f6b14087a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:16.379 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:16.375 7f6b1ab2d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6b14087670 0x7f6b14087a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:57086/0 (socket says 172.21.2.102:57086) 2021-02-10T11:24:16.379 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:16.376 7f6b1ab2d700 1 -- 172.21.2.102:0/1101263510 learned_addr learned my addr 172.21.2.102:0/1101263510 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:24:16.379 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:16.376 7f6b1ab2d700 1 -- 172.21.2.102:0/1101263510 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6b14194000 msgr2=0x7f6b140b0d10 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:24:16.379 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:16.376 7f6b1ab2d700 1 --2- 172.21.2.102:0/1101263510 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6b14194000 0x7f6b140b0d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:16.379 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:16.376 7f6b1ab2d700 1 -- 172.21.2.102:0/1101263510 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6b140b1330 msgr2=0x7f6b14087050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:16.380 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:16.376 7f6b1ab2d700 1 --2- 172.21.2.102:0/1101263510 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6b140b1330 0x7f6b14087050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:16.380 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:16.376 7f6b1ab2d700 1 -- 172.21.2.102:0/1101263510 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b1409c7d0 con 0x7f6b14087670 2021-02-10T11:24:16.380 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:16.376 7f6b1ab2d700 1 --2- 172.21.2.102:0/1101263510 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6b14087670 0x7f6b14087a70 secure :-1 s=READY pgs=1335 cs=0 l=1 rev1=1 rx=0x7f6b0c00b940 tx=0x7f6b0c00bce0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:16.380 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:16.376 7f6b06ffd700 1 -- 172.21.2.102:0/1101263510 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6b0c007ca0 con 0x7f6b14087670 2021-02-10T11:24:16.380 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:16.376 7f6b06ffd700 1 -- 172.21.2.102:0/1101263510 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b0c006880 con 0x7f6b14087670 2021-02-10T11:24:16.380 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:16.376 7f6b1c510700 1 -- 172.21.2.102:0/1101263510 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b1409c7d0 con 0x7f6b14087670 2021-02-10T11:24:16.380 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:16.376 7f6b1c510700 1 -- 172.21.2.102:0/1101263510 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b1409cd60 con 0x7f6b14087670 2021-02-10T11:24:16.380 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:16.376 7f6b06ffd700 1 -- 172.21.2.102:0/1101263510 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6b0c006270 con 0x7f6b14087670 2021-02-10T11:24:16.380 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:16.377 7f6b06ffd700 1 -- 172.21.2.102:0/1101263510 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f6b0c006080 con 0x7f6b14087670 2021-02-10T11:24:16.380 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:16.377 7f6b06ffd700 1 --2- 172.21.2.102:0/1101263510 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6afc0186d0 0x7f6afc01abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:16.380 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:16.377 7f6b06ffd700 1 -- 172.21.2.102:0/1101263510 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f6b0c032b20 con 0x7f6b14087670 2021-02-10T11:24:16.380 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:16.377 7f6b1a32c700 1 --2- 172.21.2.102:0/1101263510 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6afc0186d0 0x7f6afc01abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:16.380 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:16.377 7f6b1a32c700 1 --2- 172.21.2.102:0/1101263510 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6afc0186d0 0x7f6afc01abd0 crc :-1 s=READY pgs=625 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:16.405 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:16.407 7f6b1c510700 1 -- 172.21.2.102:0/1101263510 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b14084b20 con 0x7f6b14087670 2021-02-10T11:24:16.408 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:16.409 7f6b06ffd700 1 -- 172.21.2.102:0/1101263510 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f6b0c006530 con 0x7f6b14087670 2021-02-10T11:24:16.537 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:16.539 7f6b1c510700 1 -- 172.21.2.102:0/1101263510 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6b1407ed80 con 0x7f6b14087670 2021-02-10T11:24:16.538 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:16.539 7f6b06ffd700 1 -- 172.21.2.102:0/1101263510 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 39 v39) v1 ==== 94+0+2101 (secure 0 0 0) 0x7f6b0c006000 con 0x7f6b14087670 2021-02-10T11:24:16.538 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:24:16.538 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":39,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7920,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.102:6835/1829204031","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6834","nonce":1829204031},{"type":"v1","addr":"172.21.2.102:6835","nonce":1829204031}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":35}],"filesystems":[{"mdsmap":{"epoch":39,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:24:10.198618","modified":"2021-02-10 11:24:12.204882","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5714},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5714":{"gid":5714,"name":"b","rank":0,"incarnation":36,"state":"up:active","state_seq":3,"addr":"172.21.2.113:6833/4013137507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6832","nonce":4013137507},{"type":"v1","addr":"172.21.2.113:6833","nonce":4013137507}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":5}]} 2021-02-10T11:24:16.538 INFO:teuthology.orchestra.run.gibba002.stderr:dumped fsmap epoch 39 2021-02-10T11:24:16.539 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:16.541 7f6b1c510700 1 -- 172.21.2.102:0/1101263510 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6afc0186d0 msgr2=0x7f6afc01abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:16.539 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:16.541 7f6b1c510700 1 --2- 172.21.2.102:0/1101263510 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6afc0186d0 0x7f6afc01abd0 crc :-1 s=READY pgs=625 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:16.539 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:16.541 7f6b1c510700 1 -- 172.21.2.102:0/1101263510 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6b14087670 msgr2=0x7f6b14087a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:16.540 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:16.541 7f6b1c510700 1 --2- 172.21.2.102:0/1101263510 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6b14087670 0x7f6b14087a70 secure :-1 s=READY pgs=1335 cs=0 l=1 rev1=1 rx=0x7f6b0c00b940 tx=0x7f6b0c00bce0).stop 2021-02-10T11:24:16.540 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:16.541 7f6b1c510700 1 -- 172.21.2.102:0/1101263510 shutdown_connections 2021-02-10T11:24:16.540 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:16.541 7f6b1c510700 1 --2- 172.21.2.102:0/1101263510 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6b14087670 0x7f6b14087a70 unknown :-1 s=CLOSED pgs=1335 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:16.540 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:16.541 7f6b1c510700 1 --2- 172.21.2.102:0/1101263510 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6afc0186d0 0x7f6afc01abd0 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:16.540 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:16.541 7f6b1c510700 1 --2- 172.21.2.102:0/1101263510 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6b140b1330 0x7f6b14087050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:16.540 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:16.541 7f6b1c510700 1 --2- 172.21.2.102:0/1101263510 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6b14194000 0x7f6b140b0d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:16.540 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:16.541 7f6b1c510700 1 -- 172.21.2.102:0/1101263510 >> 172.21.2.102:0/1101263510 conn(0x7f6b14092c20 msgr2=0x7f6b14195070 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:16.541 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:16.541 7f6b1c510700 1 -- 172.21.2.102:0/1101263510 shutdown_connections 2021-02-10T11:24:16.541 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:16.541 7f6b1c510700 1 -- 172.21.2.102:0/1101263510 wait complete. 2021-02-10T11:24:16.749 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 39, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:24:10.198618', 'modified': '2021-02-10 11:24:12.204882', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5714}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5714': {'gid': 5714, 'name': 'b', 'rank': 0, 'incarnation': 36, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.113:6833/4013137507', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.113:6832', 'nonce': 4013137507}, {'type': 'v1', 'addr': '172.21.2.113:6833', 'nonce': 4013137507}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [11], 'metadata_pool': 10, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:24:16.750 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:24:16.750 DEBUG:teuthology.orchestra.run.gibba013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:24:17.147 INFO:teuthology.orchestra.run.gibba013.stdout:{ 2021-02-10T11:24:17.147 INFO:teuthology.orchestra.run.gibba013.stdout: "cluster_fsid": "68db30ea-dbce-4aef-a6df-f01c6fea0513", 2021-02-10T11:24:17.147 INFO:teuthology.orchestra.run.gibba013.stdout: "whoami": 0, 2021-02-10T11:24:17.147 INFO:teuthology.orchestra.run.gibba013.stdout: "id": 5714, 2021-02-10T11:24:17.147 INFO:teuthology.orchestra.run.gibba013.stdout: "want_state": "up:active", 2021-02-10T11:24:17.147 INFO:teuthology.orchestra.run.gibba013.stdout: "state": "up:active", 2021-02-10T11:24:17.148 INFO:teuthology.orchestra.run.gibba013.stdout: "rank_uptime": 6.9489703499999997, 2021-02-10T11:24:17.148 INFO:teuthology.orchestra.run.gibba013.stdout: "mdsmap_epoch": 39, 2021-02-10T11:24:17.148 INFO:teuthology.orchestra.run.gibba013.stdout: "osdmap_epoch": 57, 2021-02-10T11:24:17.148 INFO:teuthology.orchestra.run.gibba013.stdout: "osdmap_epoch_barrier": 57, 2021-02-10T11:24:17.148 INFO:teuthology.orchestra.run.gibba013.stdout: "uptime": 12.625336647999999 2021-02-10T11:24:17.148 INFO:teuthology.orchestra.run.gibba013.stdout:} 2021-02-10T11:24:17.151 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "68db30ea-dbce-4aef-a6df-f01c6fea0513", "whoami": 0, "id": 5714, "want_state": "up:active", "state": "up:active", "rank_uptime": 6.9489703499999997, "mdsmap_epoch": 39, "osdmap_epoch": 57, "osdmap_epoch_barrier": 57, "uptime": 12.625336647999999 } 2021-02-10T11:24:17.151 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:24:17.581 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:17.564 7f136d9f9700 1 Processor -- start 2021-02-10T11:24:17.581 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:17.564 7f136d9f9700 1 -- start start 2021-02-10T11:24:17.582 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:17.565 7f136d9f9700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f13680e5470 0x7f136807e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:17.582 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:17.565 7f136d9f9700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f136807ed80 con 0x7f136807f300 2021-02-10T11:24:17.582 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:17.565 7f136d9f9700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13681b4010 con 0x7f13680e5070 2021-02-10T11:24:17.582 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:17.565 7f136d9f9700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13680d1790 con 0x7f13680e5470 2021-02-10T11:24:17.582 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:17.565 7f1367fff700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f136807f300 0x7f136807f700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:49318/0 (socket says 172.21.2.102:49318) 2021-02-10T11:24:17.582 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:17.565 7f1367fff700 1 -- 172.21.2.102:0/1967935954 learned_addr learned my addr 172.21.2.102:0/1967935954 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:24:17.582 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:17.566 7f1366ffd700 1 --2- 172.21.2.102:0/1967935954 >> v2:172.21.2.113:3300/0 conn(0x7f13680e5470 0x7f136807e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:17.582 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:17.566 7f13667fc700 1 -- 172.21.2.102:0/1967935954 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 906479140 0 0) 0x7f1360000f90 con 0x7f136807f300 2021-02-10T11:24:17.582 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:17.566 7f13667fc700 1 -- 172.21.2.102:0/1967935954 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1354002fe0 con 0x7f136807f300 2021-02-10T11:24:17.582 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:17.566 7f13667fc700 1 -- 172.21.2.102:0/1967935954 <== mon.1 v1:172.21.2.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 151550101 0 0) 0x7f1350000c60 con 0x7f13680e5070 2021-02-10T11:24:17.582 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:17.566 7f13667fc700 1 -- 172.21.2.102:0/1967935954 --> v1:172.21.2.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1354003210 con 0x7f13680e5070 2021-02-10T11:24:17.582 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:17.566 7f13667fc700 1 -- 172.21.2.102:0/1967935954 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3166205414 0 0) 0x7f1360001460 con 0x7f136807f300 2021-02-10T11:24:17.583 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:17.566 7f13667fc700 1 -- 172.21.2.102:0/1967935954 >> v2:172.21.2.113:3300/0 conn(0x7f13680e5470 msgr2=0x7f136807e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:24:17.583 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:17.566 7f13667fc700 1 --2- 172.21.2.102:0/1967935954 >> v2:172.21.2.113:3300/0 conn(0x7f13680e5470 0x7f136807e6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:17.583 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:17.566 7f13667fc700 1 -- 172.21.2.102:0/1967935954 >> v1:172.21.2.113:6789/0 conn(0x7f13680e5070 legacy=0x7f13681b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:17.583 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:17.566 7f13667fc700 1 -- 172.21.2.102:0/1967935954 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13680d19b0 con 0x7f136807f300 2021-02-10T11:24:17.583 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:17.566 7f13667fc700 1 -- 172.21.2.102:0/1967935954 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7f1360001980 con 0x7f136807f300 2021-02-10T11:24:17.583 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:17.566 7f13667fc700 1 -- 172.21.2.102:0/1967935954 >> v1:172.21.2.102:6789/0 conn(0x7f136807f300 legacy=0x7f136807f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:17.583 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:17.566 7f13667fc700 1 --2- 172.21.2.102:0/1967935954 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1354003b90 0x7f1354005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:17.583 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:17.566 7f13667fc700 1 --2- 172.21.2.102:0/1967935954 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f13540067a0 0x7f1354008cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:17.583 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:17.566 7f13667fc700 1 --2- 172.21.2.102:0/1967935954 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f13540095c0 0x7f135400bac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:17.583 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:17.566 7f13667fc700 1 -- 172.21.2.102:0/1967935954 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f135400c310 con 0x7f13540095c0 2021-02-10T11:24:17.583 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:17.566 7f13667fc700 1 -- 172.21.2.102:0/1967935954 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1354050500 con 0x7f1354003b90 2021-02-10T11:24:17.583 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:17.566 7f13667fc700 1 -- 172.21.2.102:0/1967935954 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1354050720 con 0x7f13540067a0 2021-02-10T11:24:17.583 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:17.566 7f1367fff700 1 --2- 172.21.2.102:0/1967935954 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f13540095c0 0x7f135400bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:17.584 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:17.567 7f1366ffd700 1 --2- 172.21.2.102:0/1967935954 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f13540067a0 0x7f1354008cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:17.584 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:17.567 7f13677fe700 1 --2- 172.21.2.102:0/1967935954 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1354003b90 0x7f1354005fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:17.584 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:17.567 7f1367fff700 1 -- 172.21.2.102:0/1967935954 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f13540067a0 msgr2=0x7f1354008cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:24:17.584 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:17.567 7f1367fff700 1 --2- 172.21.2.102:0/1967935954 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f13540067a0 0x7f1354008cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:17.584 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:17.567 7f1367fff700 1 -- 172.21.2.102:0/1967935954 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1354003b90 msgr2=0x7f1354005fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:24:17.584 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:17.567 7f1367fff700 1 --2- 172.21.2.102:0/1967935954 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1354003b90 0x7f1354005fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:17.584 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:17.567 7f1367fff700 1 -- 172.21.2.102:0/1967935954 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13540509c0 con 0x7f13540095c0 2021-02-10T11:24:17.584 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:17.567 7f13677fe700 1 --2- 172.21.2.102:0/1967935954 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1354003b90 0x7f1354005fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:24:17.584 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:17.567 7f1367fff700 1 -- 172.21.2.102:0/1967935954 reap_dead start 2021-02-10T11:24:17.584 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:17.567 7f1367fff700 1 --2- 172.21.2.102:0/1967935954 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f13540095c0 0x7f135400bac0 secure :-1 s=READY pgs=1337 cs=0 l=1 rev1=1 rx=0x7f1360001130 tx=0x7f136000a310).ready entity=mon.0 client_cookie=975a095dd2d435f7 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:17.584 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:17.567 7f13667fc700 1 -- 172.21.2.102:0/1967935954 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f13600070c0 con 0x7f13540095c0 2021-02-10T11:24:17.584 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:17.567 7f13667fc700 1 -- 172.21.2.102:0/1967935954 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1360007400 con 0x7f13540095c0 2021-02-10T11:24:17.584 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:17.567 7f136d9f9700 1 -- 172.21.2.102:0/1967935954 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f13540095c0 msgr2=0x7f135400bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:17.585 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:17.567 7f136d9f9700 1 --2- 172.21.2.102:0/1967935954 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f13540095c0 0x7f135400bac0 secure :-1 s=READY pgs=1337 cs=0 l=1 rev1=1 rx=0x7f1360001130 tx=0x7f136000a310).stop 2021-02-10T11:24:17.585 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:17.567 7f13667fc700 1 -- 172.21.2.102:0/1967935954 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f13600049a0 con 0x7f13540095c0 2021-02-10T11:24:17.585 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:17.568 7f136d9f9700 1 -- 172.21.2.102:0/1967935954 shutdown_connections 2021-02-10T11:24:17.585 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:17.568 7f136d9f9700 1 --2- 172.21.2.102:0/1967935954 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f13540095c0 0x7f135400bac0 unknown :-1 s=CLOSED pgs=1337 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:17.585 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:17.568 7f136d9f9700 1 -- 172.21.2.102:0/1967935954 >> 172.21.2.102:0/1967935954 conn(0x7f13680ce690 msgr2=0x7f13680e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:17.585 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:17.568 7f136d9f9700 1 -- 172.21.2.102:0/1967935954 shutdown_connections 2021-02-10T11:24:17.585 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:17.568 7f136d9f9700 1 -- 172.21.2.102:0/1967935954 wait complete. 2021-02-10T11:24:17.585 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:17.583 7f136d9f9700 1 Processor -- start 2021-02-10T11:24:17.585 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:17.583 7f136d9f9700 1 -- start start 2021-02-10T11:24:17.586 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:17.583 7f136d9f9700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1368194000 0x7f13680b0d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:17.587 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:17.583 7f136d9f9700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f13680b1330 0x7f1368087050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:17.587 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:17.583 7f136d9f9700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1368087670 0x7f1368087a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:17.587 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:17.583 7f136d9f9700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f136809c0e0 con 0x7f13680b1330 2021-02-10T11:24:17.587 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:17.583 7f136d9f9700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1368194440 con 0x7f1368194000 2021-02-10T11:24:17.587 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:17.583 7f136d9f9700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f136809c5e0 con 0x7f1368087670 2021-02-10T11:24:17.587 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:17.583 7f1366ffd700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f13680b1330 0x7f1368087050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:17.587 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:17.583 7f1366ffd700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f13680b1330 0x7f1368087050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:57102/0 (socket says 172.21.2.102:57102) 2021-02-10T11:24:17.587 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:17.583 7f1366ffd700 1 -- 172.21.2.102:0/579179664 learned_addr learned my addr 172.21.2.102:0/579179664 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:24:17.587 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:17.584 7f1366ffd700 1 -- 172.21.2.102:0/579179664 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1368087670 msgr2=0x7f1368087a70 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:24:17.587 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:17.584 7f1366ffd700 1 --2- 172.21.2.102:0/579179664 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1368087670 0x7f1368087a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:17.587 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:17.584 7f1366ffd700 1 -- 172.21.2.102:0/579179664 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1368194000 msgr2=0x7f13680b0d10 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:24:17.587 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:17.584 7f1366ffd700 1 --2- 172.21.2.102:0/579179664 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1368194000 0x7f13680b0d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:17.588 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:17.584 7f1366ffd700 1 -- 172.21.2.102:0/579179664 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f136809c7d0 con 0x7f13680b1330 2021-02-10T11:24:17.588 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:17.584 7f1366ffd700 1 --2- 172.21.2.102:0/579179664 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f13680b1330 0x7f1368087050 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 rx=0x7f1358006f60 tx=0x7f135800b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:17.588 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:17.584 7f135ffff700 1 -- 172.21.2.102:0/579179664 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f135800bd60 con 0x7f13680b1330 2021-02-10T11:24:17.588 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:17.584 7f135ffff700 1 -- 172.21.2.102:0/579179664 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1358000c90 con 0x7f13680b1330 2021-02-10T11:24:17.588 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:17.584 7f136d9f9700 1 -- 172.21.2.102:0/579179664 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f136809c7d0 con 0x7f13680b1330 2021-02-10T11:24:17.588 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:17.584 7f136d9f9700 1 -- 172.21.2.102:0/579179664 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f136809cd60 con 0x7f13680b1330 2021-02-10T11:24:17.588 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:17.584 7f135ffff700 1 -- 172.21.2.102:0/579179664 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f13580022b0 con 0x7f13680b1330 2021-02-10T11:24:17.588 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:17.585 7f135ffff700 1 -- 172.21.2.102:0/579179664 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1358006b00 con 0x7f13680b1330 2021-02-10T11:24:17.588 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:17.585 7f135ffff700 1 --2- 172.21.2.102:0/579179664 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f134c0186c0 0x7f134c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:17.588 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:17.585 7f135ffff700 1 -- 172.21.2.102:0/579179664 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f13580323b0 con 0x7f13680b1330 2021-02-10T11:24:17.588 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:17.585 7f13677fe700 1 --2- 172.21.2.102:0/579179664 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f134c0186c0 0x7f134c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:17.588 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:17.585 7f13677fe700 1 --2- 172.21.2.102:0/579179664 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f134c0186c0 0x7f134c01abc0 crc :-1 s=READY pgs=626 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:17.613 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:17.615 7f136d9f9700 1 -- 172.21.2.102:0/579179664 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1368084b20 con 0x7f13680b1330 2021-02-10T11:24:17.616 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:17.617 7f135ffff700 1 -- 172.21.2.102:0/579179664 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f1358000940 con 0x7f13680b1330 2021-02-10T11:24:17.741 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:17.742 7f136d9f9700 1 -- 172.21.2.102:0/579179664 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f136807ed80 con 0x7f13680b1330 2021-02-10T11:24:17.741 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:17.743 7f135ffff700 1 -- 172.21.2.102:0/579179664 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v57) v1 ==== 74+0+13996 (secure 0 0 0) 0x7f1358006b00 con 0x7f13680b1330 2021-02-10T11:24:17.742 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:24:17.742 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":57,"fsid":"68db30ea-dbce-4aef-a6df-f01c6fea0513","created":"2021-02-10 11:03:06.331557","modified":"2021-02-10 11:24:11.200509","last_up_change":"2021-02-10 11:03:32.690562","last_in_change":"2021-02-10 11:03:32.690562","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:03:37.322519","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:24:08.122188","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":11,"pool_name":"cephfs_data","create_time":"2021-02-10 11:24:09.191606","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"be691b4d-a039-408a-af80-9d53f02b6f1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6818","nonce":36088},{"type":"v1","addr":"172.21.2.102:6820","nonce":36088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6822","nonce":36088},{"type":"v1","addr":"172.21.2.102:6824","nonce":36088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6830","nonce":36088},{"type":"v1","addr":"172.21.2.102:6832","nonce":36088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6826","nonce":36088},{"type":"v1","addr":"172.21.2.102:6828","nonce":36088}]},"public_addr":"172.21.2.102:6820/36088","cluster_addr":"172.21.2.102:6824/36088","heartbeat_back_addr":"172.21.2.102:6832/36088","heartbeat_front_addr":"172.21.2.102:6828/36088","state":["exists","up"]},{"osd":1,"uuid":"64e3ccc8-9c69-4d7c-865e-af8e83822635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6819","nonce":36087},{"type":"v1","addr":"172.21.2.102:6821","nonce":36087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6823","nonce":36087},{"type":"v1","addr":"172.21.2.102:6825","nonce":36087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6831","nonce":36087},{"type":"v1","addr":"172.21.2.102:6833","nonce":36087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6827","nonce":36087},{"type":"v1","addr":"172.21.2.102:6829","nonce":36087}]},"public_addr":"172.21.2.102:6821/36087","cluster_addr":"172.21.2.102:6825/36087","heartbeat_back_addr":"172.21.2.102:6833/36087","heartbeat_front_addr":"172.21.2.102:6829/36087","state":["exists","up"]},{"osd":2,"uuid":"153700af-747f-481f-bdf4-d6072d3a842f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6810","nonce":36089},{"type":"v1","addr":"172.21.2.102:6811","nonce":36089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6812","nonce":36089},{"type":"v1","addr":"172.21.2.102:6813","nonce":36089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6816","nonce":36089},{"type":"v1","addr":"172.21.2.102:6817","nonce":36089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6814","nonce":36089},{"type":"v1","addr":"172.21.2.102:6815","nonce":36089}]},"public_addr":"172.21.2.102:6811/36089","cluster_addr":"172.21.2.102:6813/36089","heartbeat_back_addr":"172.21.2.102:6817/36089","heartbeat_front_addr":"172.21.2.102:6815/36089","state":["exists","up"]},{"osd":3,"uuid":"cbc678f4-83a9-45e6-8830-2f7c232b1ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6802","nonce":36090},{"type":"v1","addr":"172.21.2.102:6803","nonce":36090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6804","nonce":36090},{"type":"v1","addr":"172.21.2.102:6805","nonce":36090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6808","nonce":36090},{"type":"v1","addr":"172.21.2.102:6809","nonce":36090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6806","nonce":36090},{"type":"v1","addr":"172.21.2.102:6807","nonce":36090}]},"public_addr":"172.21.2.102:6803/36090","cluster_addr":"172.21.2.102:6805/36090","heartbeat_back_addr":"172.21.2.102:6809/36090","heartbeat_front_addr":"172.21.2.102:6807/36090","state":["exists","up"]},{"osd":4,"uuid":"c01e4008-b258-4f10-afd2-9e7f37214351","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6808","nonce":36368},{"type":"v1","addr":"172.21.2.113:6811","nonce":36368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6814","nonce":36368},{"type":"v1","addr":"172.21.2.113:6818","nonce":36368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6826","nonce":36368},{"type":"v1","addr":"172.21.2.113:6828","nonce":36368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6820","nonce":36368},{"type":"v1","addr":"172.21.2.113:6824","nonce":36368}]},"public_addr":"172.21.2.113:6811/36368","cluster_addr":"172.21.2.113:6818/36368","heartbeat_back_addr":"172.21.2.113:6828/36368","heartbeat_front_addr":"172.21.2.113:6824/36368","state":["exists","up"]},{"osd":5,"uuid":"551f6007-daf8-4a98-8ee0-c2b16bcccb5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6802","nonce":36366},{"type":"v1","addr":"172.21.2.113:6805","nonce":36366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6807","nonce":36366},{"type":"v1","addr":"172.21.2.113:6810","nonce":36366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6819","nonce":36366},{"type":"v1","addr":"172.21.2.113:6822","nonce":36366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6812","nonce":36366},{"type":"v1","addr":"172.21.2.113:6815","nonce":36366}]},"public_addr":"172.21.2.113:6805/36366","cluster_addr":"172.21.2.113:6810/36366","heartbeat_back_addr":"172.21.2.113:6822/36366","heartbeat_front_addr":"172.21.2.113:6815/36366","state":["exists","up"]},{"osd":6,"uuid":"bf67c040-0e50-48be-b102-c3943bc9ebef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6800","nonce":36367},{"type":"v1","addr":"172.21.2.113:6801","nonce":36367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6803","nonce":36367},{"type":"v1","addr":"172.21.2.113:6804","nonce":36367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6813","nonce":36367},{"type":"v1","addr":"172.21.2.113:6816","nonce":36367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6806","nonce":36367},{"type":"v1","addr":"172.21.2.113:6809","nonce":36367}]},"public_addr":"172.21.2.113:6801/36367","cluster_addr":"172.21.2.113:6804/36367","heartbeat_back_addr":"172.21.2.113:6816/36367","heartbeat_front_addr":"172.21.2.113:6809/36367","state":["exists","up"]},{"osd":7,"uuid":"f5461f80-018d-428f-be89-2eb2bcbe7bb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6817","nonce":36365},{"type":"v1","addr":"172.21.2.113:6821","nonce":36365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6823","nonce":36365},{"type":"v1","addr":"172.21.2.113:6825","nonce":36365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6830","nonce":36365},{"type":"v1","addr":"172.21.2.113:6831","nonce":36365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6827","nonce":36365},{"type":"v1","addr":"172.21.2.113:6829","nonce":36365}]},"public_addr":"172.21.2.113:6821/36365","cluster_addr":"172.21.2.113:6825/36365","heartbeat_back_addr":"172.21.2.113:6831/36365","heartbeat_front_addr":"172.21.2.113:6829/36365","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:24:17.743 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:17.745 7f136d9f9700 1 -- 172.21.2.102:0/579179664 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f134c0186c0 msgr2=0x7f134c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:17.743 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:17.745 7f136d9f9700 1 --2- 172.21.2.102:0/579179664 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f134c0186c0 0x7f134c01abc0 crc :-1 s=READY pgs=626 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:17.743 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:17.745 7f136d9f9700 1 -- 172.21.2.102:0/579179664 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f13680b1330 msgr2=0x7f1368087050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:17.743 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:17.745 7f136d9f9700 1 --2- 172.21.2.102:0/579179664 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f13680b1330 0x7f1368087050 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 rx=0x7f1358006f60 tx=0x7f135800b970).stop 2021-02-10T11:24:17.744 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:17.745 7f136d9f9700 1 -- 172.21.2.102:0/579179664 shutdown_connections 2021-02-10T11:24:17.744 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:17.745 7f136d9f9700 1 --2- 172.21.2.102:0/579179664 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1368087670 0x7f1368087a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:17.744 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:17.745 7f136d9f9700 1 --2- 172.21.2.102:0/579179664 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f13680b1330 0x7f1368087050 unknown :-1 s=CLOSED pgs=1338 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:17.744 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:17.745 7f136d9f9700 1 --2- 172.21.2.102:0/579179664 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f134c0186c0 0x7f134c01abc0 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:17.744 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:17.745 7f136d9f9700 1 --2- 172.21.2.102:0/579179664 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1368194000 0x7f13680b0d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:17.744 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:17.745 7f136d9f9700 1 -- 172.21.2.102:0/579179664 >> 172.21.2.102:0/579179664 conn(0x7f1368092c20 msgr2=0x7f1368195070 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:17.744 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:17.745 7f136d9f9700 1 -- 172.21.2.102:0/579179664 shutdown_connections 2021-02-10T11:24:17.744 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:17.745 7f136d9f9700 1 -- 172.21.2.102:0/579179664 wait complete. 2021-02-10T11:24:17.752 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:03:37.322519', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 10, 'pool_name': 'cephfs_metadata', 'create_time': '2021-02-10 11:24:08.122188', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '56', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 11, 'pool_name': 'cephfs_data', 'create_time': '2021-02-10 11:24:09.191606', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '57', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-02-10T11:24:17.753 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:24:18.185 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:18.169 7efe59e7b700 1 Processor -- start 2021-02-10T11:24:18.185 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:18.169 7efe59e7b700 1 -- start start 2021-02-10T11:24:18.185 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:18.170 7efe59e7b700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7efe541b0470 0x7efe541b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:18.185 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:18.170 7efe59e7b700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7efe540ef610 0x7efe540efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:18.185 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:18.170 7efe59e7b700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe540f0b50 con 0x7efe540f0030 2021-02-10T11:24:18.185 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:18.170 7efe59e7b700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe540e52e0 con 0x7efe541b0470 2021-02-10T11:24:18.185 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:18.170 7efe59e7b700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe540e5490 con 0x7efe540ef610 2021-02-10T11:24:18.186 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:18.170 7efe537fe700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7efe541b0470 0x7efe541b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:18.186 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:18.170 7efe537fe700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7efe541b0470 0x7efe541b0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:57110/0 (socket says 172.21.2.102:57110) 2021-02-10T11:24:18.186 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:18.170 7efe537fe700 1 -- 172.21.2.102:0/2965364702 learned_addr learned my addr 172.21.2.102:0/2965364702 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:24:18.186 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:18.171 7efe537fe700 1 -- 172.21.2.102:0/2965364702 >> v2:172.21.2.113:3300/0 conn(0x7efe540ef610 msgr2=0x7efe540efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:24:18.186 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:18.171 7efe537fe700 1 --2- 172.21.2.102:0/2965364702 >> v2:172.21.2.113:3300/0 conn(0x7efe540ef610 0x7efe540efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:18.186 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:18.171 7efe537fe700 1 -- 172.21.2.102:0/2965364702 >> v1:172.21.2.113:6789/0 conn(0x7efe540f0030 legacy=0x7efe540f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:24:18.186 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:18.171 7efe537fe700 1 -- 172.21.2.102:0/2965364702 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe540f0430 con 0x7efe541b0470 2021-02-10T11:24:18.186 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:18.171 7efe537fe700 1 --2- 172.21.2.102:0/2965364702 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efe541b0470 0x7efe541b0870 secure :-1 s=READY pgs=1339 cs=0 l=1 rev1=1 rx=0x7efe44006700 tx=0x7efe44009b00).ready entity=mon.0 client_cookie=90eba35c78b7ee6f server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:18.186 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:18.171 7efe527fc700 1 -- 172.21.2.102:0/2965364702 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efe4400c0c0 con 0x7efe541b0470 2021-02-10T11:24:18.186 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:18.171 7efe527fc700 1 -- 172.21.2.102:0/2965364702 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe4400c380 con 0x7efe541b0470 2021-02-10T11:24:18.186 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:18.171 7efe527fc700 1 -- 172.21.2.102:0/2965364702 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efe4400c860 con 0x7efe541b0470 2021-02-10T11:24:18.186 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:18.171 7efe59e7b700 1 -- 172.21.2.102:0/2965364702 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efe541b0470 msgr2=0x7efe541b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:18.186 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:18.171 7efe59e7b700 1 --2- 172.21.2.102:0/2965364702 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efe541b0470 0x7efe541b0870 secure :-1 s=READY pgs=1339 cs=0 l=1 rev1=1 rx=0x7efe44006700 tx=0x7efe44009b00).stop 2021-02-10T11:24:18.187 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:18.172 7efe59e7b700 1 -- 172.21.2.102:0/2965364702 shutdown_connections 2021-02-10T11:24:18.187 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:18.172 7efe59e7b700 1 --2- 172.21.2.102:0/2965364702 >> v2:172.21.2.113:3300/0 conn(0x7efe540ef610 0x7efe540efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:18.187 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:18.172 7efe59e7b700 1 --2- 172.21.2.102:0/2965364702 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efe541b0470 0x7efe541b0870 unknown :-1 s=CLOSED pgs=1339 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:18.187 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:18.172 7efe59e7b700 1 -- 172.21.2.102:0/2965364702 >> 172.21.2.102:0/2965364702 conn(0x7efe540ce3e0 msgr2=0x7efe540e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:18.187 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:18.172 7efe59e7b700 1 -- 172.21.2.102:0/2965364702 shutdown_connections 2021-02-10T11:24:18.187 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:18.172 7efe59e7b700 1 -- 172.21.2.102:0/2965364702 wait complete. 2021-02-10T11:24:18.188 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:18.187 7efe59e7b700 1 Processor -- start 2021-02-10T11:24:18.189 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:18.187 7efe59e7b700 1 -- start start 2021-02-10T11:24:18.189 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:18.187 7efe59e7b700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efe540c36a0 0x7efe540c2b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:18.189 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:18.187 7efe59e7b700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7efe540c3190 0x7efe540b1d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:18.189 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:18.187 7efe59e7b700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7efe540b2370 0x7efe540b2d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:18.189 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:18.187 7efe59e7b700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe540b27c0 con 0x7efe540c36a0 2021-02-10T11:24:18.189 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:18.187 7efe59e7b700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe540b3360 con 0x7efe540c3190 2021-02-10T11:24:18.189 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:18.187 7efe59e7b700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe540b3540 con 0x7efe540b2370 2021-02-10T11:24:18.189 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:18.187 7efe537fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efe540c36a0 0x7efe540c2b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:18.189 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:18.187 7efe537fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efe540c36a0 0x7efe540c2b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:57114/0 (socket says 172.21.2.102:57114) 2021-02-10T11:24:18.189 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:18.188 7efe537fe700 1 -- 172.21.2.102:0/1747348886 learned_addr learned my addr 172.21.2.102:0/1747348886 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:24:18.189 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:18.188 7efe537fe700 1 -- 172.21.2.102:0/1747348886 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7efe540b2370 msgr2=0x7efe540b2d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:18.189 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:18.188 7efe537fe700 1 --2- 172.21.2.102:0/1747348886 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7efe540b2370 0x7efe540b2d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:18.189 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:18.188 7efe537fe700 1 -- 172.21.2.102:0/1747348886 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7efe540c3190 msgr2=0x7efe540b1d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:18.190 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:18.188 7efe537fe700 1 --2- 172.21.2.102:0/1747348886 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7efe540c3190 0x7efe540b1d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:18.190 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:18.188 7efe537fe700 1 -- 172.21.2.102:0/1747348886 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe540b3760 con 0x7efe540c36a0 2021-02-10T11:24:18.190 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:18.188 7efe537fe700 1 --2- 172.21.2.102:0/1747348886 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efe540c36a0 0x7efe540c2b70 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 rx=0x7efe4400b660 tx=0x7efe4400ba00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:18.190 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:18.188 7efe3bfff700 1 -- 172.21.2.102:0/1747348886 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efe44009440 con 0x7efe540c36a0 2021-02-10T11:24:18.190 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:18.188 7efe3bfff700 1 -- 172.21.2.102:0/1747348886 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe44000c90 con 0x7efe540c36a0 2021-02-10T11:24:18.190 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:18.188 7efe59e7b700 1 -- 172.21.2.102:0/1747348886 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe540b3760 con 0x7efe540c36a0 2021-02-10T11:24:18.190 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:18.188 7efe59e7b700 1 -- 172.21.2.102:0/1747348886 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe540b3cf0 con 0x7efe540c36a0 2021-02-10T11:24:18.190 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:18.188 7efe3bfff700 1 -- 172.21.2.102:0/1747348886 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efe44002030 con 0x7efe540c36a0 2021-02-10T11:24:18.190 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:18.189 7efe3bfff700 1 -- 172.21.2.102:0/1747348886 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7efe440093e0 con 0x7efe540c36a0 2021-02-10T11:24:18.190 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:18.189 7efe3bfff700 1 --2- 172.21.2.102:0/1747348886 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7efe340186c0 0x7efe3401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:18.190 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:18.189 7efe3bfff700 1 -- 172.21.2.102:0/1747348886 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5573+0+0 (secure 0 0 0) 0x7efe440327b0 con 0x7efe540c36a0 2021-02-10T11:24:18.190 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:18.189 7efe52ffd700 1 --2- 172.21.2.102:0/1747348886 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7efe340186c0 0x7efe3401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:18.190 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:18.190 7efe52ffd700 1 --2- 172.21.2.102:0/1747348886 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7efe340186c0 0x7efe3401abc0 crc :-1 s=READY pgs=627 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:18.218 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:18.220 7efe59e7b700 1 -- 172.21.2.102:0/1747348886 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe540edd10 con 0x7efe540c36a0 2021-02-10T11:24:18.220 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:18.222 7efe3bfff700 1 -- 172.21.2.102:0/1747348886 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7efe44000c90 con 0x7efe540c36a0 2021-02-10T11:24:18.350 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:18.351 7efe59e7b700 1 -- 172.21.2.102:0/1747348886 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7efe540e52e0 con 0x7efe540c36a0 2021-02-10T11:24:18.350 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:18.352 7efe3bfff700 1 -- 172.21.2.102:0/1747348886 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v57) v1 ==== 92+0+10 (secure 0 0 0) 0x7efe4400bdf0 con 0x7efe540c36a0 2021-02-10T11:24:18.351 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:24:18.353 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:18.354 7efe59e7b700 1 -- 172.21.2.102:0/1747348886 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7efe340186c0 msgr2=0x7efe3401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:18.353 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:18.354 7efe59e7b700 1 --2- 172.21.2.102:0/1747348886 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7efe340186c0 0x7efe3401abc0 crc :-1 s=READY pgs=627 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:18.353 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:18.354 7efe59e7b700 1 -- 172.21.2.102:0/1747348886 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efe540c36a0 msgr2=0x7efe540c2b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:18.353 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:18.354 7efe59e7b700 1 --2- 172.21.2.102:0/1747348886 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efe540c36a0 0x7efe540c2b70 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 rx=0x7efe4400b660 tx=0x7efe4400ba00).stop 2021-02-10T11:24:18.353 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:18.354 7efe59e7b700 1 -- 172.21.2.102:0/1747348886 shutdown_connections 2021-02-10T11:24:18.353 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:18.354 7efe59e7b700 1 --2- 172.21.2.102:0/1747348886 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7efe540b2370 0x7efe540b2d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:18.353 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:18.354 7efe59e7b700 1 --2- 172.21.2.102:0/1747348886 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7efe340186c0 0x7efe3401abc0 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:18.353 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:18.354 7efe59e7b700 1 --2- 172.21.2.102:0/1747348886 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7efe540c3190 0x7efe540b1d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:18.354 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:18.354 7efe59e7b700 1 --2- 172.21.2.102:0/1747348886 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efe540c36a0 0x7efe540c2b70 unknown :-1 s=CLOSED pgs=1340 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:18.354 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:18.354 7efe59e7b700 1 -- 172.21.2.102:0/1747348886 >> 172.21.2.102:0/1747348886 conn(0x7efe5409f630 msgr2=0x7efe540a0130 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:18.354 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:18.354 7efe59e7b700 1 -- 172.21.2.102:0/1747348886 shutdown_connections 2021-02-10T11:24:18.354 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:18.354 7efe59e7b700 1 -- 172.21.2.102:0/1747348886 wait complete. 2021-02-10T11:24:18.361 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-02-10T11:24:18.793 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:18.777 7fe95103b700 1 Processor -- start 2021-02-10T11:24:18.793 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:18.777 7fe95103b700 1 -- start start 2021-02-10T11:24:18.793 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:18.778 7fe95103b700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fe94c1b0870 0x7fe94c0f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:18.794 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:18.778 7fe95103b700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe94c1b0c70 con 0x7fe94c0efe00 2021-02-10T11:24:18.794 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:18.778 7fe95103b700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe94c0f0b50 con 0x7fe94c1b0470 2021-02-10T11:24:18.794 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:18.778 7fe95103b700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe94c0e5ea0 con 0x7fe94c1b0870 2021-02-10T11:24:18.794 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:18.778 7fe94b59e700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fe94c0efe00 0x7fe94c0f0200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:49350/0 (socket says 172.21.2.102:49350) 2021-02-10T11:24:18.794 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:18.778 7fe94b59e700 1 -- 172.21.2.102:0/4122785691 learned_addr learned my addr 172.21.2.102:0/4122785691 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:24:18.794 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:18.778 7fe94a59c700 1 --2- 172.21.2.102:0/4122785691 >> v2:172.21.2.113:3300/0 conn(0x7fe94c1b0870 0x7fe94c0f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:18.794 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:18.779 7fe949d9b700 1 -- 172.21.2.102:0/4122785691 <== mon.1 v1:172.21.2.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 639369564 0 0) 0x7fe93c000c60 con 0x7fe94c1b0470 2021-02-10T11:24:18.794 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:18.779 7fe949d9b700 1 -- 172.21.2.102:0/4122785691 --> v1:172.21.2.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe938002fe0 con 0x7fe94c1b0470 2021-02-10T11:24:18.794 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:18.779 7fe949d9b700 1 -- 172.21.2.102:0/4122785691 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3288423970 0 0) 0x7fe944001500 con 0x7fe94c0efe00 2021-02-10T11:24:18.794 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:18.779 7fe949d9b700 1 -- 172.21.2.102:0/4122785691 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe938004370 con 0x7fe94c0efe00 2021-02-10T11:24:18.794 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:18.779 7fe94a59c700 1 -- 172.21.2.102:0/4122785691 >> v1:172.21.2.113:6789/0 conn(0x7fe94c1b0470 legacy=0x7fe94c0ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:18.795 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:18.779 7fe94a59c700 1 -- 172.21.2.102:0/4122785691 >> v1:172.21.2.102:6789/0 conn(0x7fe94c0efe00 legacy=0x7fe94c0f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:18.795 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:18.779 7fe94a59c700 1 -- 172.21.2.102:0/4122785691 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe94c0e52e0 con 0x7fe94c1b0870 2021-02-10T11:24:18.795 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:18.779 7fe94a59c700 1 --2- 172.21.2.102:0/4122785691 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe94c1b0870 0x7fe94c0f20e0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 rx=0x7fe934006650 tx=0x7fe934009a80).ready entity=mon.1 client_cookie=96146fd8d726c537 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:18.795 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:18.779 7fe949d9b700 1 -- 172.21.2.102:0/4122785691 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe9340009b0 con 0x7fe94c1b0870 2021-02-10T11:24:18.795 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:18.779 7fe949d9b700 1 -- 172.21.2.102:0/4122785691 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe93400c0c0 con 0x7fe94c1b0870 2021-02-10T11:24:18.795 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:18.779 7fe949d9b700 1 -- 172.21.2.102:0/4122785691 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe93400c5e0 con 0x7fe94c1b0870 2021-02-10T11:24:18.795 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:18.779 7fe95103b700 1 -- 172.21.2.102:0/4122785691 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe94c1b0870 msgr2=0x7fe94c0f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:18.795 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:18.779 7fe95103b700 1 --2- 172.21.2.102:0/4122785691 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe94c1b0870 0x7fe94c0f20e0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 rx=0x7fe934006650 tx=0x7fe934009a80).stop 2021-02-10T11:24:18.795 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:18.780 7fe95103b700 1 -- 172.21.2.102:0/4122785691 shutdown_connections 2021-02-10T11:24:18.795 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:18.780 7fe95103b700 1 --2- 172.21.2.102:0/4122785691 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe94c1b0870 0x7fe94c0f20e0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:18.795 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:18.780 7fe95103b700 1 -- 172.21.2.102:0/4122785691 >> 172.21.2.102:0/4122785691 conn(0x7fe94c0ce3e0 msgr2=0x7fe94c0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:18.795 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:18.780 7fe95103b700 1 -- 172.21.2.102:0/4122785691 shutdown_connections 2021-02-10T11:24:18.795 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:18.780 7fe95103b700 1 -- 172.21.2.102:0/4122785691 wait complete. 2021-02-10T11:24:18.796 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:18.795 7fe95103b700 1 Processor -- start 2021-02-10T11:24:18.796 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:18.795 7fe95103b700 1 -- start start 2021-02-10T11:24:18.797 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:18.795 7fe95103b700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe94c09e400 0x7fe94c1a7a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:18.797 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:18.795 7fe95103b700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe94c1a8050 0x7fe94c0b2fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:18.797 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:18.795 7fe95103b700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe94c0b35c0 0x7fe94c0b39c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:18.797 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:18.795 7fe95103b700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe94c0bfe70 con 0x7fe94c09e400 2021-02-10T11:24:18.797 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:18.795 7fe95103b700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe94c09e980 con 0x7fe94c0b35c0 2021-02-10T11:24:18.797 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:18.795 7fe95103b700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe94c0bf9e0 con 0x7fe94c1a8050 2021-02-10T11:24:18.797 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:18.796 7fe94ad9d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe94c09e400 0x7fe94c1a7a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:18.798 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:18.796 7fe94ad9d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe94c09e400 0x7fe94c1a7a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:57128/0 (socket says 172.21.2.102:57128) 2021-02-10T11:24:18.798 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:18.796 7fe94ad9d700 1 -- 172.21.2.102:0/214513107 learned_addr learned my addr 172.21.2.102:0/214513107 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:24:18.798 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:18.796 7fe94ad9d700 1 -- 172.21.2.102:0/214513107 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe94c1a8050 msgr2=0x7fe94c0b2fa0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:24:18.798 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:18.796 7fe94ad9d700 1 --2- 172.21.2.102:0/214513107 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe94c1a8050 0x7fe94c0b2fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:18.798 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:18.796 7fe94ad9d700 1 -- 172.21.2.102:0/214513107 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe94c0b35c0 msgr2=0x7fe94c0b39c0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:24:18.798 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:18.796 7fe94ad9d700 1 --2- 172.21.2.102:0/214513107 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe94c0b35c0 0x7fe94c0b39c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:18.798 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:18.796 7fe94ad9d700 1 -- 172.21.2.102:0/214513107 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe94c0c0370 con 0x7fe94c09e400 2021-02-10T11:24:18.798 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:18.796 7fe94ad9d700 1 --2- 172.21.2.102:0/214513107 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe94c09e400 0x7fe94c1a7a30 secure :-1 s=READY pgs=1342 cs=0 l=1 rev1=1 rx=0x7fe93c006f10 tx=0x7fe93c00b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:18.798 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:18.796 7fe9437fe700 1 -- 172.21.2.102:0/214513107 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe93c00bd80 con 0x7fe94c09e400 2021-02-10T11:24:18.798 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:18.796 7fe9437fe700 1 -- 172.21.2.102:0/214513107 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe93c000e20 con 0x7fe94c09e400 2021-02-10T11:24:18.798 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:18.796 7fe9437fe700 1 -- 172.21.2.102:0/214513107 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe93c0054f0 con 0x7fe94c09e400 2021-02-10T11:24:18.798 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:18.796 7fe95103b700 1 -- 172.21.2.102:0/214513107 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe94c0c0370 con 0x7fe94c09e400 2021-02-10T11:24:18.799 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:18.796 7fe95103b700 1 -- 172.21.2.102:0/214513107 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe94c0c0370 con 0x7fe94c09e400 2021-02-10T11:24:18.799 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:18.797 7fe9437fe700 1 -- 172.21.2.102:0/214513107 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fe93c009fe0 con 0x7fe94c09e400 2021-02-10T11:24:18.799 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:18.797 7fe9437fe700 1 --2- 172.21.2.102:0/214513107 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe9300186c0 0x7fe93001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:18.799 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:18.797 7fe9437fe700 1 -- 172.21.2.102:0/214513107 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fe93c0325b0 con 0x7fe94c09e400 2021-02-10T11:24:18.799 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:18.797 7fe94a59c700 1 --2- 172.21.2.102:0/214513107 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe9300186c0 0x7fe93001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:18.799 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:18.797 7fe94a59c700 1 --2- 172.21.2.102:0/214513107 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe9300186c0 0x7fe93001abc0 crc :-1 s=READY pgs=628 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:18.826 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:18.828 7fe95103b700 1 -- 172.21.2.102:0/214513107 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe94c105610 con 0x7fe94c09e400 2021-02-10T11:24:18.828 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:18.830 7fe9437fe700 1 -- 172.21.2.102:0/214513107 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fe93c030de0 con 0x7fe94c09e400 2021-02-10T11:24:18.956 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:18.958 7fe95103b700 1 -- 172.21.2.102:0/214513107 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7fe94c105610 con 0x7fe94c09e400 2021-02-10T11:24:18.957 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:18.958 7fe9437fe700 1 -- 172.21.2.102:0/214513107 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v57) v1 ==== 104+0+10 (secure 0 0 0) 0x7fe93c00bd80 con 0x7fe94c09e400 2021-02-10T11:24:18.958 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:24:18.959 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:18.960 7fe95103b700 1 -- 172.21.2.102:0/214513107 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe9300186c0 msgr2=0x7fe93001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:18.959 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:18.960 7fe95103b700 1 --2- 172.21.2.102:0/214513107 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe9300186c0 0x7fe93001abc0 crc :-1 s=READY pgs=628 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:18.959 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:18.960 7fe95103b700 1 -- 172.21.2.102:0/214513107 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe94c09e400 msgr2=0x7fe94c1a7a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:18.959 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:18.960 7fe95103b700 1 --2- 172.21.2.102:0/214513107 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe94c09e400 0x7fe94c1a7a30 secure :-1 s=READY pgs=1342 cs=0 l=1 rev1=1 rx=0x7fe93c006f10 tx=0x7fe93c00b990).stop 2021-02-10T11:24:18.959 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:18.960 7fe95103b700 1 -- 172.21.2.102:0/214513107 shutdown_connections 2021-02-10T11:24:18.959 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:18.960 7fe95103b700 1 --2- 172.21.2.102:0/214513107 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe9300186c0 0x7fe93001abc0 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:18.960 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:18.960 7fe95103b700 1 --2- 172.21.2.102:0/214513107 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe94c0b35c0 0x7fe94c0b39c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:18.960 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:18.960 7fe95103b700 1 --2- 172.21.2.102:0/214513107 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe94c1a8050 0x7fe94c0b2fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:18.960 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:18.960 7fe95103b700 1 --2- 172.21.2.102:0/214513107 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe94c09e400 0x7fe94c1a7a30 unknown :-1 s=CLOSED pgs=1342 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:18.960 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:18.960 7fe95103b700 1 -- 172.21.2.102:0/214513107 >> 172.21.2.102:0/214513107 conn(0x7fe94c0a6b10 msgr2=0x7fe94c0bf0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:18.960 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:18.961 7fe95103b700 1 -- 172.21.2.102:0/214513107 shutdown_connections 2021-02-10T11:24:18.960 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:18.961 7fe95103b700 1 -- 172.21.2.102:0/214513107 wait complete. 2021-02-10T11:24:18.968 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-02-10T11:24:19.399 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:19.383 7f8c4fb23700 1 Processor -- start 2021-02-10T11:24:19.399 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:19.383 7f8c4fb23700 1 -- start start 2021-02-10T11:24:19.399 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:19.383 7f8c4fb23700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f8c481b0470 0x7f8c481b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:19.399 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:19.383 7f8c4fb23700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8c480ef610 0x7f8c480efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:19.399 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:19.383 7f8c4fb23700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c480f0b50 con 0x7f8c480f0030 2021-02-10T11:24:19.399 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:19.383 7f8c4fb23700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c480e52e0 con 0x7f8c481b0470 2021-02-10T11:24:19.399 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:19.383 7f8c4fb23700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c480e54d0 con 0x7f8c480ef610 2021-02-10T11:24:19.399 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:19.383 7f8c4e140700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f8c480f0030 0x7f8c480f10d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:49364/0 (socket says 172.21.2.102:49364) 2021-02-10T11:24:19.400 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:19.383 7f8c4e140700 1 -- 172.21.2.102:0/4144866410 learned_addr learned my addr 172.21.2.102:0/4144866410 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:24:19.400 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:19.384 7f8c4c93d700 1 -- 172.21.2.102:0/4144866410 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3065045816 0 0) 0x7f8c40001500 con 0x7f8c480f0030 2021-02-10T11:24:19.400 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:19.384 7f8c4c93d700 1 -- 172.21.2.102:0/4144866410 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c34002fe0 con 0x7f8c480f0030 2021-02-10T11:24:19.400 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:19.384 7f8c4d93f700 1 --2- 172.21.2.102:0/4144866410 >> v2:172.21.2.113:3300/0 conn(0x7f8c481b0470 0x7f8c481b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:19.400 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:19.384 7f8c4d13e700 1 --2- 172.21.2.102:0/4144866410 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8c480ef610 0x7f8c480efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:19.400 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:19.384 7f8c4c93d700 1 -- 172.21.2.102:0/4144866410 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3491944987 0 0) 0x7f8c40001460 con 0x7f8c480f0030 2021-02-10T11:24:19.400 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:19.384 7f8c4c93d700 1 -- 172.21.2.102:0/4144866410 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8c480ef610 msgr2=0x7f8c480efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:24:19.400 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:19.384 7f8c4c93d700 1 --2- 172.21.2.102:0/4144866410 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8c480ef610 0x7f8c480efa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:19.400 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:19.384 7f8c4c93d700 1 -- 172.21.2.102:0/4144866410 >> v2:172.21.2.113:3300/0 conn(0x7f8c481b0470 msgr2=0x7f8c481b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:24:19.400 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:19.384 7f8c4c93d700 1 --2- 172.21.2.102:0/4144866410 >> v2:172.21.2.113:3300/0 conn(0x7f8c481b0470 0x7f8c481b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:19.400 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:19.384 7f8c4c93d700 1 -- 172.21.2.102:0/4144866410 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c480f0430 con 0x7f8c480f0030 2021-02-10T11:24:19.400 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:19.384 7f8c4c93d700 1 -- 172.21.2.102:0/4144866410 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7f8c40001980 con 0x7f8c480f0030 2021-02-10T11:24:19.401 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:19.384 7f8c4c93d700 1 -- 172.21.2.102:0/4144866410 >> v1:172.21.2.102:6789/0 conn(0x7f8c480f0030 legacy=0x7f8c480f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:19.401 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:19.384 7f8c4c93d700 1 --2- 172.21.2.102:0/4144866410 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8c34003ae0 0x7f8c34006000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:19.401 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:19.384 7f8c4c93d700 1 --2- 172.21.2.102:0/4144866410 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8c34006820 0x7f8c34008d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:19.401 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:19.384 7f8c4c93d700 1 --2- 172.21.2.102:0/4144866410 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8c34009640 0x7f8c3400bb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:19.401 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:19.384 7f8c4c93d700 1 -- 172.21.2.102:0/4144866410 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c34050370 con 0x7f8c34003ae0 2021-02-10T11:24:19.401 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:19.384 7f8c4c93d700 1 -- 172.21.2.102:0/4144866410 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c34050590 con 0x7f8c34006820 2021-02-10T11:24:19.401 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:19.384 7f8c4c93d700 1 -- 172.21.2.102:0/4144866410 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c340507b0 con 0x7f8c34009640 2021-02-10T11:24:19.401 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:19.384 7f8c4d13e700 1 --2- 172.21.2.102:0/4144866410 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8c480ef610 0x7f8c480efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:24:19.401 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:19.384 7f8c4d93f700 1 --2- 172.21.2.102:0/4144866410 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8c34003ae0 0x7f8c34006000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:19.401 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:19.384 7f8c4e140700 1 --2- 172.21.2.102:0/4144866410 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8c34009640 0x7f8c3400bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:19.401 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:19.385 7f8c4d93f700 1 -- 172.21.2.102:0/4144866410 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8c34009640 msgr2=0x7f8c3400bb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:24:19.401 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:19.385 7f8c4d93f700 1 --2- 172.21.2.102:0/4144866410 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8c34009640 0x7f8c3400bb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:19.401 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:19.385 7f8c4d93f700 1 -- 172.21.2.102:0/4144866410 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8c34006820 msgr2=0x7f8c34008d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:24:19.402 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:19.385 7f8c4d93f700 1 --2- 172.21.2.102:0/4144866410 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8c34006820 0x7f8c34008d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:19.402 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:19.385 7f8c4d93f700 1 -- 172.21.2.102:0/4144866410 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c34050a50 con 0x7f8c34003ae0 2021-02-10T11:24:19.402 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:19.385 7f8c4d93f700 1 --2- 172.21.2.102:0/4144866410 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8c34003ae0 0x7f8c34006000 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 rx=0x7f8c4400c8f0 tx=0x7f8c4400cc90).ready entity=mon.0 client_cookie=9998df3c206ee616 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:19.402 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:19.385 7f8c4c93d700 1 -- 172.21.2.102:0/4144866410 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8c4400cdd0 con 0x7f8c34003ae0 2021-02-10T11:24:19.402 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:19.385 7f8c4c93d700 1 -- 172.21.2.102:0/4144866410 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c44009310 con 0x7f8c34003ae0 2021-02-10T11:24:19.402 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:19.385 7f8c4c93d700 1 -- 172.21.2.102:0/4144866410 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8c44009830 con 0x7f8c34003ae0 2021-02-10T11:24:19.402 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:19.385 7f8c4fb23700 1 -- 172.21.2.102:0/4144866410 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8c34003ae0 msgr2=0x7f8c34006000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:19.402 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:19.385 7f8c4fb23700 1 --2- 172.21.2.102:0/4144866410 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8c34003ae0 0x7f8c34006000 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 rx=0x7f8c4400c8f0 tx=0x7f8c4400cc90).stop 2021-02-10T11:24:19.402 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:19.385 7f8c4e140700 1 -- 172.21.2.102:0/4144866410 reap_dead start 2021-02-10T11:24:19.402 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:19.385 7f8c4fb23700 1 -- 172.21.2.102:0/4144866410 shutdown_connections 2021-02-10T11:24:19.402 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:19.385 7f8c4fb23700 1 -- 172.21.2.102:0/4144866410 >> 172.21.2.102:0/4144866410 conn(0x7f8c480ce3e0 msgr2=0x7f8c480e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:19.402 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:19.386 7f8c4fb23700 1 -- 172.21.2.102:0/4144866410 shutdown_connections 2021-02-10T11:24:19.403 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:19.386 7f8c4fb23700 1 -- 172.21.2.102:0/4144866410 wait complete. 2021-02-10T11:24:19.403 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:19.400 7f8c4fb23700 1 Processor -- start 2021-02-10T11:24:19.403 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:19.400 7f8c4fb23700 1 -- start start 2021-02-10T11:24:19.404 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:19.401 7f8c4fb23700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8c4809b930 0x7f8c480b7c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:19.404 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:19.401 7f8c4fb23700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8c480b8410 0x7f8c480b8810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:19.404 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:19.401 7f8c4fb23700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8c480c1f80 0x7f8c480c2380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:19.404 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:19.401 7f8c4fb23700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c480c1a00 con 0x7f8c4809b930 2021-02-10T11:24:19.404 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:19.401 7f8c4fb23700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c480b8e30 con 0x7f8c480b8410 2021-02-10T11:24:19.404 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:19.401 7f8c4fb23700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c480b9050 con 0x7f8c480c1f80 2021-02-10T11:24:19.404 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:19.401 7f8c4d93f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8c4809b930 0x7f8c480b7c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:19.404 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:19.401 7f8c4d93f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8c4809b930 0x7f8c480b7c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:57148/0 (socket says 172.21.2.102:57148) 2021-02-10T11:24:19.404 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:19.401 7f8c4d93f700 1 -- 172.21.2.102:0/201295546 learned_addr learned my addr 172.21.2.102:0/201295546 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:24:19.405 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:19.401 7f8c4d93f700 1 -- 172.21.2.102:0/201295546 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8c480c1f80 msgr2=0x7f8c480c2380 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:24:19.405 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:19.401 7f8c4d93f700 1 --2- 172.21.2.102:0/201295546 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8c480c1f80 0x7f8c480c2380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:19.405 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:19.401 7f8c4d93f700 1 -- 172.21.2.102:0/201295546 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8c480b8410 msgr2=0x7f8c480b8810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:19.405 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:19.401 7f8c4d93f700 1 --2- 172.21.2.102:0/201295546 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8c480b8410 0x7f8c480b8810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:19.405 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:19.401 7f8c4d93f700 1 -- 172.21.2.102:0/201295546 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c480c29a0 con 0x7f8c4809b930 2021-02-10T11:24:19.405 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:19.402 7f8c4d93f700 1 --2- 172.21.2.102:0/201295546 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8c4809b930 0x7f8c480b7c20 secure :-1 s=READY pgs=1345 cs=0 l=1 rev1=1 rx=0x7f8c4400c160 tx=0x7f8c44009f70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:19.405 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:19.402 7f8c3a7fc700 1 -- 172.21.2.102:0/201295546 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8c44006030 con 0x7f8c4809b930 2021-02-10T11:24:19.405 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:19.402 7f8c3a7fc700 1 -- 172.21.2.102:0/201295546 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c4400b940 con 0x7f8c4809b930 2021-02-10T11:24:19.405 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:19.402 7f8c3a7fc700 1 -- 172.21.2.102:0/201295546 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8c4400bde0 con 0x7f8c4809b930 2021-02-10T11:24:19.405 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:19.402 7f8c4fb23700 1 -- 172.21.2.102:0/201295546 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c480c29a0 con 0x7f8c4809b930 2021-02-10T11:24:19.405 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:19.402 7f8c4fb23700 1 -- 172.21.2.102:0/201295546 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c480c29a0 con 0x7f8c4809b930 2021-02-10T11:24:19.405 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:19.402 7f8c3a7fc700 1 -- 172.21.2.102:0/201295546 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f8c44006030 con 0x7f8c4809b930 2021-02-10T11:24:19.405 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:19.403 7f8c3a7fc700 1 --2- 172.21.2.102:0/201295546 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8c300186c0 0x7f8c3001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:19.406 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:19.403 7f8c3a7fc700 1 -- 172.21.2.102:0/201295546 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f8c44032450 con 0x7f8c4809b930 2021-02-10T11:24:19.406 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:19.403 7f8c4d13e700 1 --2- 172.21.2.102:0/201295546 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8c300186c0 0x7f8c3001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:19.406 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:19.403 7f8c4d13e700 1 --2- 172.21.2.102:0/201295546 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8c300186c0 0x7f8c3001abc0 crc :-1 s=READY pgs=629 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:19.431 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:19.433 7f8c4fb23700 1 -- 172.21.2.102:0/201295546 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c48105b30 con 0x7f8c4809b930 2021-02-10T11:24:19.434 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:19.435 7f8c3a7fc700 1 -- 172.21.2.102:0/201295546 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f8c4400c1a0 con 0x7f8c4809b930 2021-02-10T11:24:19.561 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:19.562 7f8c4fb23700 1 -- 172.21.2.102:0/201295546 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7f8c480f0b50 con 0x7f8c4809b930 2021-02-10T11:24:19.561 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:19.563 7f8c3a7fc700 1 -- 172.21.2.102:0/201295546 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v57) v1 ==== 100+0+10 (secure 0 0 0) 0x7f8c44006030 con 0x7f8c4809b930 2021-02-10T11:24:19.562 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:24:19.563 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:19.564 7f8c4fb23700 1 -- 172.21.2.102:0/201295546 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8c300186c0 msgr2=0x7f8c3001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:19.563 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:19.564 7f8c4fb23700 1 --2- 172.21.2.102:0/201295546 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8c300186c0 0x7f8c3001abc0 crc :-1 s=READY pgs=629 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:19.563 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:19.564 7f8c4fb23700 1 -- 172.21.2.102:0/201295546 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8c4809b930 msgr2=0x7f8c480b7c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:19.564 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:19.564 7f8c4fb23700 1 --2- 172.21.2.102:0/201295546 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8c4809b930 0x7f8c480b7c20 secure :-1 s=READY pgs=1345 cs=0 l=1 rev1=1 rx=0x7f8c4400c160 tx=0x7f8c44009f70).stop 2021-02-10T11:24:19.564 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:19.565 7f8c4fb23700 1 -- 172.21.2.102:0/201295546 shutdown_connections 2021-02-10T11:24:19.564 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:19.565 7f8c4fb23700 1 --2- 172.21.2.102:0/201295546 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8c480c1f80 0x7f8c480c2380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:19.564 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:19.565 7f8c4fb23700 1 --2- 172.21.2.102:0/201295546 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8c300186c0 0x7f8c3001abc0 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:19.564 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:19.565 7f8c4fb23700 1 --2- 172.21.2.102:0/201295546 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8c480b8410 0x7f8c480b8810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:19.564 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:19.565 7f8c4fb23700 1 --2- 172.21.2.102:0/201295546 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8c4809b930 0x7f8c480b7c20 unknown :-1 s=CLOSED pgs=1345 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:19.564 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:19.565 7f8c4fb23700 1 -- 172.21.2.102:0/201295546 >> 172.21.2.102:0/201295546 conn(0x7f8c48094e10 msgr2=0x7f8c480a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:19.564 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:19.565 7f8c4fb23700 1 -- 172.21.2.102:0/201295546 shutdown_connections 2021-02-10T11:24:19.564 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:19.565 7f8c4fb23700 1 -- 172.21.2.102:0/201295546 wait complete. 2021-02-10T11:24:19.572 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-02-10T11:24:19.572 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:24:19.998 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:19.983 7fe61881f700 1 Processor -- start 2021-02-10T11:24:19.998 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:19.983 7fe61881f700 1 -- start start 2021-02-10T11:24:19.998 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:19.984 7fe61881f700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe6140e5470 0x7fe61407e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:19.999 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:19.984 7fe61881f700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fe61407f300 0x7fe61407f700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:19.999 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:19.984 7fe61881f700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe61407ed80 con 0x7fe6140e5070 2021-02-10T11:24:19.999 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:19.984 7fe61881f700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6141b4010 con 0x7fe61407f300 2021-02-10T11:24:19.999 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:19.984 7fe61881f700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6140d1790 con 0x7fe6140e5470 2021-02-10T11:24:19.999 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:19.984 7fe61261c700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fe6140e5070 0x7fe6141b3820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:49380/0 (socket says 172.21.2.102:49380) 2021-02-10T11:24:19.999 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:19.984 7fe61261c700 1 -- 172.21.2.102:0/1045359030 learned_addr learned my addr 172.21.2.102:0/1045359030 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:24:19.999 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:19.984 7fe612e1d700 1 --2- 172.21.2.102:0/1045359030 >> v2:172.21.2.102:3300/0 conn(0x7fe61407f300 0x7fe61407f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:19.999 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:19.984 7fe61161a700 1 -- 172.21.2.102:0/1045359030 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1230878573 0 0) 0x7fe6040010b0 con 0x7fe6140e5070 2021-02-10T11:24:19.999 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:19.984 7fe612e1d700 1 -- 172.21.2.102:0/1045359030 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe6140e5470 msgr2=0x7fe61407e6a0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:24:19.999 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:19.984 7fe612e1d700 1 --2- 172.21.2.102:0/1045359030 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe6140e5470 0x7fe61407e6a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:19.999 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:19.984 7fe612e1d700 1 -- 172.21.2.102:0/1045359030 >> v1:172.21.2.102:6789/0 conn(0x7fe6140e5070 legacy=0x7fe6141b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:19.999 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:19.984 7fe612e1d700 1 -- 172.21.2.102:0/1045359030 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6140d1980 con 0x7fe61407f300 2021-02-10T11:24:20.000 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:19.985 7fe612e1d700 1 --2- 172.21.2.102:0/1045359030 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe61407f300 0x7fe61407f700 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 rx=0x7fe60c006e30 tx=0x7fe60c009e60).ready entity=mon.0 client_cookie=fc5b36df9f7651be server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:20.000 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:19.985 7fe61161a700 1 -- 172.21.2.102:0/1045359030 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe60c006a50 con 0x7fe61407f300 2021-02-10T11:24:20.000 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:19.985 7fe61161a700 1 -- 172.21.2.102:0/1045359030 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe60c00b300 con 0x7fe61407f300 2021-02-10T11:24:20.000 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:19.985 7fe61161a700 1 -- 172.21.2.102:0/1045359030 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe60c00b820 con 0x7fe61407f300 2021-02-10T11:24:20.000 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:19.985 7fe61881f700 1 -- 172.21.2.102:0/1045359030 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe61407f300 msgr2=0x7fe61407f700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:20.000 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:19.985 7fe61881f700 1 --2- 172.21.2.102:0/1045359030 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe61407f300 0x7fe61407f700 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 rx=0x7fe60c006e30 tx=0x7fe60c009e60).stop 2021-02-10T11:24:20.000 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:19.986 7fe61881f700 1 -- 172.21.2.102:0/1045359030 shutdown_connections 2021-02-10T11:24:20.000 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:19.986 7fe61881f700 1 --2- 172.21.2.102:0/1045359030 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe61407f300 0x7fe61407f700 unknown :-1 s=CLOSED pgs=1347 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:20.000 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:19.986 7fe61881f700 1 --2- 172.21.2.102:0/1045359030 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe6140e5470 0x7fe61407e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:20.000 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:19.986 7fe61881f700 1 -- 172.21.2.102:0/1045359030 >> 172.21.2.102:0/1045359030 conn(0x7fe6140ce690 msgr2=0x7fe6140e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:20.000 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:19.986 7fe61881f700 1 -- 172.21.2.102:0/1045359030 shutdown_connections 2021-02-10T11:24:20.000 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:19.986 7fe61881f700 1 -- 172.21.2.102:0/1045359030 wait complete. 2021-02-10T11:24:20.002 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:20.000 7fe61881f700 1 Processor -- start 2021-02-10T11:24:20.002 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:20.000 7fe61881f700 1 -- start start 2021-02-10T11:24:20.002 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:20.001 7fe61881f700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe61418c700 0x7fe614096c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:20.002 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:20.001 7fe61881f700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe6140ae400 0x7fe6140ae800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:20.002 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:20.001 7fe61881f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe614192380 0x7fe614192780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:20.002 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:20.001 7fe61881f700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe614097290 con 0x7fe614192380 2021-02-10T11:24:20.002 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:20.001 7fe61881f700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe614192da0 con 0x7fe6140ae400 2021-02-10T11:24:20.002 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:20.001 7fe61881f700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe614192f80 con 0x7fe61418c700 2021-02-10T11:24:20.002 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:20.001 7fe612e1d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe614192380 0x7fe614192780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:20.002 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:20.001 7fe612e1d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe614192380 0x7fe614192780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:57162/0 (socket says 172.21.2.102:57162) 2021-02-10T11:24:20.003 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:20.001 7fe612e1d700 1 -- 172.21.2.102:0/298511988 learned_addr learned my addr 172.21.2.102:0/298511988 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:24:20.003 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:20.001 7fe612e1d700 1 -- 172.21.2.102:0/298511988 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe61418c700 msgr2=0x7fe614096c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:20.003 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:20.001 7fe612e1d700 1 --2- 172.21.2.102:0/298511988 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe61418c700 0x7fe614096c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:20.003 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:20.001 7fe612e1d700 1 -- 172.21.2.102:0/298511988 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe6140ae400 msgr2=0x7fe6140ae800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:20.003 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:20.001 7fe612e1d700 1 --2- 172.21.2.102:0/298511988 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe6140ae400 0x7fe6140ae800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:20.003 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:20.001 7fe612e1d700 1 -- 172.21.2.102:0/298511988 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6141931a0 con 0x7fe614192380 2021-02-10T11:24:20.003 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:20.001 7fe612e1d700 1 --2- 172.21.2.102:0/298511988 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe614192380 0x7fe614192780 secure :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 rx=0x7fe60c00b970 tx=0x7fe60c00bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:20.003 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:20.002 7fe60affd700 1 -- 172.21.2.102:0/298511988 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe60c0093d0 con 0x7fe614192380 2021-02-10T11:24:20.003 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:20.002 7fe60affd700 1 -- 172.21.2.102:0/298511988 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe60c005820 con 0x7fe614192380 2021-02-10T11:24:20.003 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:20.002 7fe60affd700 1 -- 172.21.2.102:0/298511988 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe60c005d00 con 0x7fe614192380 2021-02-10T11:24:20.003 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:20.002 7fe61881f700 1 -- 172.21.2.102:0/298511988 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6141931a0 con 0x7fe614192380 2021-02-10T11:24:20.003 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:20.002 7fe61881f700 1 -- 172.21.2.102:0/298511988 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe61407f510 con 0x7fe614192380 2021-02-10T11:24:20.003 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:20.002 7fe60affd700 1 -- 172.21.2.102:0/298511988 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fe60c005a00 con 0x7fe614192380 2021-02-10T11:24:20.004 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:20.002 7fe60affd700 1 --2- 172.21.2.102:0/298511988 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe5f80186c0 0x7fe5f801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:20.004 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:20.002 7fe60affd700 1 -- 172.21.2.102:0/298511988 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fe60c032940 con 0x7fe614192380 2021-02-10T11:24:20.004 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:20.003 7fe61261c700 1 --2- 172.21.2.102:0/298511988 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe5f80186c0 0x7fe5f801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:20.004 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:20.003 7fe61261c700 1 --2- 172.21.2.102:0/298511988 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe5f80186c0 0x7fe5f801abc0 crc :-1 s=READY pgs=630 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:20.032 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:20.034 7fe61881f700 1 -- 172.21.2.102:0/298511988 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe614107120 con 0x7fe614192380 2021-02-10T11:24:20.035 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:20.036 7fe60affd700 1 -- 172.21.2.102:0/298511988 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fe60c009380 con 0x7fe614192380 2021-02-10T11:24:20.163 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:20.165 7fe61881f700 1 -- 172.21.2.102:0/298511988 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe61407ed80 con 0x7fe614192380 2021-02-10T11:24:20.164 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:20.165 7fe60affd700 1 -- 172.21.2.102:0/298511988 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 39 v39) v1 ==== 94+0+2101 (secure 0 0 0) 0x7fe60c005a00 con 0x7fe614192380 2021-02-10T11:24:20.164 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:24:20.164 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":39,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7920,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.102:6835/1829204031","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6834","nonce":1829204031},{"type":"v1","addr":"172.21.2.102:6835","nonce":1829204031}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":35}],"filesystems":[{"mdsmap":{"epoch":39,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:24:10.198618","modified":"2021-02-10 11:24:12.204882","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5714},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5714":{"gid":5714,"name":"b","rank":0,"incarnation":36,"state":"up:active","state_seq":3,"addr":"172.21.2.113:6833/4013137507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6832","nonce":4013137507},{"type":"v1","addr":"172.21.2.113:6833","nonce":4013137507}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":5}]} 2021-02-10T11:24:20.165 INFO:teuthology.orchestra.run.gibba002.stderr:dumped fsmap epoch 39 2021-02-10T11:24:20.165 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:20.167 7fe61881f700 1 -- 172.21.2.102:0/298511988 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe5f80186c0 msgr2=0x7fe5f801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:20.166 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:20.167 7fe61881f700 1 --2- 172.21.2.102:0/298511988 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe5f80186c0 0x7fe5f801abc0 crc :-1 s=READY pgs=630 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:20.166 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:20.167 7fe61881f700 1 -- 172.21.2.102:0/298511988 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe614192380 msgr2=0x7fe614192780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:20.166 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:20.167 7fe61881f700 1 --2- 172.21.2.102:0/298511988 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe614192380 0x7fe614192780 secure :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 rx=0x7fe60c00b970 tx=0x7fe60c00bd10).stop 2021-02-10T11:24:20.166 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:20.167 7fe61881f700 1 -- 172.21.2.102:0/298511988 shutdown_connections 2021-02-10T11:24:20.166 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:20.167 7fe61881f700 1 --2- 172.21.2.102:0/298511988 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe614192380 0x7fe614192780 unknown :-1 s=CLOSED pgs=1348 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:20.166 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:20.167 7fe61881f700 1 --2- 172.21.2.102:0/298511988 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe5f80186c0 0x7fe5f801abc0 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:20.167 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:20.167 7fe61881f700 1 --2- 172.21.2.102:0/298511988 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe6140ae400 0x7fe6140ae800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:20.167 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:20.167 7fe61881f700 1 --2- 172.21.2.102:0/298511988 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe61418c700 0x7fe614096c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:20.167 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:20.167 7fe61881f700 1 -- 172.21.2.102:0/298511988 >> 172.21.2.102:0/298511988 conn(0x7fe6140891f0 msgr2=0x7fe6141a0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:20.167 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:20.167 7fe61881f700 1 -- 172.21.2.102:0/298511988 shutdown_connections 2021-02-10T11:24:20.167 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:20.167 7fe61881f700 1 -- 172.21.2.102:0/298511988 wait complete. 2021-02-10T11:24:20.376 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 39, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:24:10.198618', 'modified': '2021-02-10 11:24:12.204882', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5714}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5714': {'gid': 5714, 'name': 'b', 'rank': 0, 'incarnation': 36, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.113:6833/4013137507', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.113:6832', 'nonce': 4013137507}, {'type': 'v1', 'addr': '172.21.2.113:6833', 'nonce': 4013137507}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [11], 'metadata_pool': 10, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:24:20.376 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:24:20.377 DEBUG:teuthology.orchestra.run.gibba013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:24:20.776 INFO:teuthology.orchestra.run.gibba013.stdout:{ 2021-02-10T11:24:20.776 INFO:teuthology.orchestra.run.gibba013.stdout: "cluster_fsid": "68db30ea-dbce-4aef-a6df-f01c6fea0513", 2021-02-10T11:24:20.776 INFO:teuthology.orchestra.run.gibba013.stdout: "whoami": 0, 2021-02-10T11:24:20.776 INFO:teuthology.orchestra.run.gibba013.stdout: "id": 5714, 2021-02-10T11:24:20.776 INFO:teuthology.orchestra.run.gibba013.stdout: "want_state": "up:active", 2021-02-10T11:24:20.776 INFO:teuthology.orchestra.run.gibba013.stdout: "state": "up:active", 2021-02-10T11:24:20.776 INFO:teuthology.orchestra.run.gibba013.stdout: "rank_uptime": 10.577754001000001, 2021-02-10T11:24:20.776 INFO:teuthology.orchestra.run.gibba013.stdout: "mdsmap_epoch": 39, 2021-02-10T11:24:20.777 INFO:teuthology.orchestra.run.gibba013.stdout: "osdmap_epoch": 57, 2021-02-10T11:24:20.777 INFO:teuthology.orchestra.run.gibba013.stdout: "osdmap_epoch_barrier": 57, 2021-02-10T11:24:20.777 INFO:teuthology.orchestra.run.gibba013.stdout: "uptime": 16.254122216999999 2021-02-10T11:24:20.777 INFO:teuthology.orchestra.run.gibba013.stdout:} 2021-02-10T11:24:20.780 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "68db30ea-dbce-4aef-a6df-f01c6fea0513", "whoami": 0, "id": 5714, "want_state": "up:active", "state": "up:active", "rank_uptime": 10.577754001000001, "mdsmap_epoch": 39, "osdmap_epoch": 57, "osdmap_epoch_barrier": 57, "uptime": 16.254122216999999 } 2021-02-10T11:24:20.780 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2021-02-10T11:24:20.780 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2021-02-10T11:24:20.780 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@gibba002.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2021-02-10T11:24:20.780 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:24:20.781 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec mkdir -p /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:24:21.117 DEBUG:teuthology.orchestra.run.gibba002:> sudo modprobe fuse 2021-02-10T11:24:21.461 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:24:21.461 DEBUG:teuthology.orchestra.run.gibba002:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:24:21.804 INFO:teuthology.orchestra.run.gibba002.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:24:21.804 INFO:teuthology.orchestra.run.gibba002.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:24:21.805 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:24:21.805 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:24:21.805 DEBUG:teuthology.orchestra.run.gibba002:> ls /sys/fs/fuse/connections 2021-02-10T11:24:22.139 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2021-02-10T11:24:22.140 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:24:22.142 DEBUG:teuthology.orchestra.run.gibba002:> sudo modprobe fuse 2021-02-10T11:24:22.486 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:24:22.486 DEBUG:teuthology.orchestra.run.gibba002:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:24:22.553 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba002.stderr:ceph-fuse[153237]: starting ceph client 2021-02-10T11:24:22.553 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba002.stderr:2021-02-10 11:24:22.555 7fc8e3592f80 -1 init, newargv = 0x558961184cc0 newargc=9 2021-02-10T11:24:22.559 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba002.stderr:ceph-fuse[153237]: starting fuse 2021-02-10T11:24:22.828 INFO:teuthology.orchestra.run.gibba002.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:24:22.829 INFO:teuthology.orchestra.run.gibba002.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:24:22.829 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:24:22.830 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:24:22.830 DEBUG:teuthology.orchestra.run.gibba002:> ls /sys/fs/fuse/connections 2021-02-10T11:24:23.164 INFO:teuthology.orchestra.run.gibba002.stdout:43 2021-02-10T11:24:23.164 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2021-02-10T11:24:23.164 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:24:23.164 DEBUG:teuthology.orchestra.run.gibba002:> sudo python3 -c ' 2021-02-10T11:24:23.165 DEBUG:teuthology.orchestra.run.gibba002:> import glob 2021-02-10T11:24:23.165 DEBUG:teuthology.orchestra.run.gibba002:> import re 2021-02-10T11:24:23.165 DEBUG:teuthology.orchestra.run.gibba002:> import os 2021-02-10T11:24:23.165 DEBUG:teuthology.orchestra.run.gibba002:> import subprocess 2021-02-10T11:24:23.165 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:24:23.165 DEBUG:teuthology.orchestra.run.gibba002:> def find_socket(client_name): 2021-02-10T11:24:23.165 DEBUG:teuthology.orchestra.run.gibba002:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2021-02-10T11:24:23.165 DEBUG:teuthology.orchestra.run.gibba002:> files = glob.glob(asok_path) 2021-02-10T11:24:23.165 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:24:23.165 DEBUG:teuthology.orchestra.run.gibba002:> # Given a non-glob path, it better be there 2021-02-10T11:24:23.165 DEBUG:teuthology.orchestra.run.gibba002:> if "*" not in asok_path: 2021-02-10T11:24:23.165 DEBUG:teuthology.orchestra.run.gibba002:> assert(len(files) == 1) 2021-02-10T11:24:23.165 DEBUG:teuthology.orchestra.run.gibba002:> return files[0] 2021-02-10T11:24:23.165 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:24:23.166 DEBUG:teuthology.orchestra.run.gibba002:> for f in files: 2021-02-10T11:24:23.166 DEBUG:teuthology.orchestra.run.gibba002:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2021-02-10T11:24:23.166 DEBUG:teuthology.orchestra.run.gibba002:> if os.path.exists("/proc/{0}".format(pid)): 2021-02-10T11:24:23.166 DEBUG:teuthology.orchestra.run.gibba002:> return f 2021-02-10T11:24:23.166 DEBUG:teuthology.orchestra.run.gibba002:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2021-02-10T11:24:23.166 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:24:23.166 DEBUG:teuthology.orchestra.run.gibba002:> print(find_socket("client.0")) 2021-02-10T11:24:23.166 DEBUG:teuthology.orchestra.run.gibba002:> ' 2021-02-10T11:24:23.526 INFO:teuthology.orchestra.run.gibba002.stdout:/var/run/ceph/ceph-client.0.153237.asok 2021-02-10T11:24:23.529 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.153237.asok 2021-02-10T11:24:23.530 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:24:23.530 DEBUG:teuthology.orchestra.run.gibba002:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.153237.asok status 2021-02-10T11:24:23.918 INFO:teuthology.orchestra.run.gibba002.stdout:{ 2021-02-10T11:24:23.918 INFO:teuthology.orchestra.run.gibba002.stdout: "metadata": { 2021-02-10T11:24:23.918 INFO:teuthology.orchestra.run.gibba002.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:24:23.918 INFO:teuthology.orchestra.run.gibba002.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:24:23.918 INFO:teuthology.orchestra.run.gibba002.stdout: "entity_id": "0", 2021-02-10T11:24:23.918 INFO:teuthology.orchestra.run.gibba002.stdout: "hostname": "gibba002", 2021-02-10T11:24:23.919 INFO:teuthology.orchestra.run.gibba002.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2021-02-10T11:24:23.919 INFO:teuthology.orchestra.run.gibba002.stdout: "pid": "153237", 2021-02-10T11:24:23.919 INFO:teuthology.orchestra.run.gibba002.stdout: "root": "/" 2021-02-10T11:24:23.919 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:24:23.919 INFO:teuthology.orchestra.run.gibba002.stdout: "dentry_count": 0, 2021-02-10T11:24:23.919 INFO:teuthology.orchestra.run.gibba002.stdout: "dentry_pinned_count": 0, 2021-02-10T11:24:23.919 INFO:teuthology.orchestra.run.gibba002.stdout: "id": 8085, 2021-02-10T11:24:23.919 INFO:teuthology.orchestra.run.gibba002.stdout: "inst": { 2021-02-10T11:24:23.919 INFO:teuthology.orchestra.run.gibba002.stdout: "name": { 2021-02-10T11:24:23.919 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "client", 2021-02-10T11:24:23.919 INFO:teuthology.orchestra.run.gibba002.stdout: "num": 8085 2021-02-10T11:24:23.919 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:24:23.919 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": { 2021-02-10T11:24:23.920 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:24:23.920 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:0", 2021-02-10T11:24:23.920 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 175669434 2021-02-10T11:24:23.920 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:24:23.920 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:24:23.920 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": { 2021-02-10T11:24:23.920 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:24:23.920 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:0", 2021-02-10T11:24:23.920 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 175669434 2021-02-10T11:24:23.920 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:24:23.920 INFO:teuthology.orchestra.run.gibba002.stdout: "inst_str": "client.8085 172.21.2.102:0/175669434", 2021-02-10T11:24:23.920 INFO:teuthology.orchestra.run.gibba002.stdout: "addr_str": "172.21.2.102:0/175669434", 2021-02-10T11:24:23.920 INFO:teuthology.orchestra.run.gibba002.stdout: "inode_count": 1, 2021-02-10T11:24:23.920 INFO:teuthology.orchestra.run.gibba002.stdout: "mds_epoch": 39, 2021-02-10T11:24:23.921 INFO:teuthology.orchestra.run.gibba002.stdout: "osd_epoch": 57, 2021-02-10T11:24:23.921 INFO:teuthology.orchestra.run.gibba002.stdout: "osd_epoch_barrier": 0, 2021-02-10T11:24:23.921 INFO:teuthology.orchestra.run.gibba002.stdout: "blacklisted": false 2021-02-10T11:24:23.921 INFO:teuthology.orchestra.run.gibba002.stdout:} 2021-02-10T11:24:23.922 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:24:23.922 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:24:23.923 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:24:24.258 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:24:24.259 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:24:24.259 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:24:24.259 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:24:24.606 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:24:25.033 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:25.019 7f8e21183700 1 Processor -- start 2021-02-10T11:24:25.033 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:25.019 7f8e21183700 1 -- start start 2021-02-10T11:24:25.033 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:25.020 7f8e21183700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f8e1c0e5070 0x7f8e1c1b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:25.033 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:25.020 7f8e21183700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8e1c1b3e40 0x7f8e1c07d690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:25.033 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:25.020 7f8e21183700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e1c07ed80 con 0x7f8e1c0e5470 2021-02-10T11:24:25.034 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:25.020 7f8e21183700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e1c080310 con 0x7f8e1c0e5070 2021-02-10T11:24:25.034 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:25.020 7f8e21183700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e1c080500 con 0x7f8e1c1b3e40 2021-02-10T11:24:25.034 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:25.020 7f8e1ad9d700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f8e1c0e5070 0x7f8e1c1b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:25.034 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:25.020 7f8e1ad9d700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f8e1c0e5070 0x7f8e1c1b3820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:57182/0 (socket says 172.21.2.102:57182) 2021-02-10T11:24:25.034 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:25.020 7f8e1ad9d700 1 -- 172.21.2.102:0/4147228522 learned_addr learned my addr 172.21.2.102:0/4147228522 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:24:25.034 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:25.020 7f8e1ad9d700 1 -- 172.21.2.102:0/4147228522 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8e1c1b3e40 msgr2=0x7f8e1c07d690 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:24:25.034 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:25.020 7f8e1ad9d700 1 --2- 172.21.2.102:0/4147228522 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8e1c1b3e40 0x7f8e1c07d690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:25.034 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:25.020 7f8e1ad9d700 1 -- 172.21.2.102:0/4147228522 >> v1:172.21.2.113:6789/0 conn(0x7f8e1c0e5470 legacy=0x7f8e1c07dcb0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:24:25.034 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:25.020 7f8e1ad9d700 1 -- 172.21.2.102:0/4147228522 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e1c0806f0 con 0x7f8e1c0e5070 2021-02-10T11:24:25.034 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:25.020 7f8e1ad9d700 1 --2- 172.21.2.102:0/4147228522 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8e1c0e5070 0x7f8e1c1b3820 secure :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 rx=0x7f8e04006700 tx=0x7f8e04009b80).ready entity=mon.0 client_cookie=996ce7ee142c3db7 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:25.034 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:25.021 7f8e19d9b700 1 -- 172.21.2.102:0/4147228522 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8e0400c0c0 con 0x7f8e1c0e5070 2021-02-10T11:24:25.034 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:25.021 7f8e19d9b700 1 -- 172.21.2.102:0/4147228522 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e0400c400 con 0x7f8e1c0e5070 2021-02-10T11:24:25.035 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:25.021 7f8e19d9b700 1 -- 172.21.2.102:0/4147228522 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8e0400c920 con 0x7f8e1c0e5070 2021-02-10T11:24:25.035 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:25.021 7f8e21183700 1 -- 172.21.2.102:0/4147228522 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8e1c0e5070 msgr2=0x7f8e1c1b3820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:25.035 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:25.021 7f8e21183700 1 --2- 172.21.2.102:0/4147228522 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8e1c0e5070 0x7f8e1c1b3820 secure :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 rx=0x7f8e04006700 tx=0x7f8e04009b80).stop 2021-02-10T11:24:25.035 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:25.021 7f8e21183700 1 -- 172.21.2.102:0/4147228522 shutdown_connections 2021-02-10T11:24:25.035 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:25.021 7f8e21183700 1 --2- 172.21.2.102:0/4147228522 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8e1c1b3e40 0x7f8e1c07d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:25.035 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:25.021 7f8e21183700 1 --2- 172.21.2.102:0/4147228522 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8e1c0e5070 0x7f8e1c1b3820 unknown :-1 s=CLOSED pgs=1351 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:25.035 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:25.021 7f8e21183700 1 -- 172.21.2.102:0/4147228522 >> 172.21.2.102:0/4147228522 conn(0x7f8e1c0ce690 msgr2=0x7f8e1c0e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:25.035 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:25.021 7f8e21183700 1 -- 172.21.2.102:0/4147228522 shutdown_connections 2021-02-10T11:24:25.035 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:25.021 7f8e21183700 1 -- 172.21.2.102:0/4147228522 wait complete. 2021-02-10T11:24:25.037 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:25.036 7f8e21183700 1 Processor -- start 2021-02-10T11:24:25.037 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:25.036 7f8e21183700 1 -- start start 2021-02-10T11:24:25.037 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:25.036 7f8e21183700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8e1c1ade10 0x7f8e1c09d250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:25.037 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:25.036 7f8e21183700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8e1c0b4440 0x7f8e1c0b4840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:25.037 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:25.036 7f8e21183700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8e1c1952d0 0x7f8e1c1956d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:25.037 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:25.036 7f8e21183700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e1c194d50 con 0x7f8e1c0b4440 2021-02-10T11:24:25.037 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:25.036 7f8e21183700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e1c1932c0 con 0x7f8e1c1952d0 2021-02-10T11:24:25.037 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:25.036 7f8e21183700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e1c195cf0 con 0x7f8e1c1ade10 2021-02-10T11:24:25.038 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:25.036 7f8e1a59c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8e1c0b4440 0x7f8e1c0b4840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:25.038 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:25.036 7f8e1a59c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8e1c0b4440 0x7f8e1c0b4840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:57192/0 (socket says 172.21.2.102:57192) 2021-02-10T11:24:25.038 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:25.036 7f8e1a59c700 1 -- 172.21.2.102:0/532923020 learned_addr learned my addr 172.21.2.102:0/532923020 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:24:25.038 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:25.037 7f8e1a59c700 1 -- 172.21.2.102:0/532923020 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8e1c1ade10 msgr2=0x7f8e1c09d250 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:24:25.038 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:25.037 7f8e1a59c700 1 --2- 172.21.2.102:0/532923020 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8e1c1ade10 0x7f8e1c09d250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:25.038 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:25.037 7f8e1a59c700 1 -- 172.21.2.102:0/532923020 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8e1c1952d0 msgr2=0x7f8e1c1956d0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:24:25.038 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:25.037 7f8e1a59c700 1 --2- 172.21.2.102:0/532923020 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8e1c1952d0 0x7f8e1c1956d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:25.038 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:25.037 7f8e1a59c700 1 -- 172.21.2.102:0/532923020 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e1c195f10 con 0x7f8e1c0b4440 2021-02-10T11:24:25.038 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:25.037 7f8e1a59c700 1 --2- 172.21.2.102:0/532923020 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8e1c0b4440 0x7f8e1c0b4840 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 rx=0x7f8e0c00b5c0 tx=0x7f8e0c00b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:25.038 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:25.037 7f8e137fe700 1 -- 172.21.2.102:0/532923020 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8e0c00bd50 con 0x7f8e1c0b4440 2021-02-10T11:24:25.038 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:25.037 7f8e137fe700 1 -- 172.21.2.102:0/532923020 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e0c006030 con 0x7f8e1c0b4440 2021-02-10T11:24:25.038 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:25.037 7f8e137fe700 1 -- 172.21.2.102:0/532923020 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8e0c0054d0 con 0x7f8e1c0b4440 2021-02-10T11:24:25.039 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:25.037 7f8e21183700 1 -- 172.21.2.102:0/532923020 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e1c195f10 con 0x7f8e1c0b4440 2021-02-10T11:24:25.039 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:25.037 7f8e21183700 1 -- 172.21.2.102:0/532923020 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e1c195f10 con 0x7f8e1c0b4440 2021-02-10T11:24:25.039 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:25.038 7f8e137fe700 1 -- 172.21.2.102:0/532923020 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f8e0c006210 con 0x7f8e1c0b4440 2021-02-10T11:24:25.039 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:25.038 7f8e137fe700 1 --2- 172.21.2.102:0/532923020 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8e000186c0 0x7f8e0001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:25.039 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:25.038 7f8e137fe700 1 -- 172.21.2.102:0/532923020 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f8e0c032420 con 0x7f8e1c0b4440 2021-02-10T11:24:25.039 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:25.038 7f8e1ad9d700 1 --2- 172.21.2.102:0/532923020 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8e000186c0 0x7f8e0001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:25.039 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:25.038 7f8e1ad9d700 1 --2- 172.21.2.102:0/532923020 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8e000186c0 0x7f8e0001abc0 crc :-1 s=READY pgs=631 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:25.067 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:25.070 7f8e21183700 1 -- 172.21.2.102:0/532923020 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e1c083db0 con 0x7f8e1c0b4440 2021-02-10T11:24:25.070 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:25.073 7f8e137fe700 1 -- 172.21.2.102:0/532923020 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f8e0c006b80 con 0x7f8e1c0b4440 2021-02-10T11:24:25.195 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:25.197 7f8e21183700 1 -- 172.21.2.102:0/532923020 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8e1c07ed80 con 0x7f8e1c0b4440 2021-02-10T11:24:25.195 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:25.198 7f8e137fe700 1 -- 172.21.2.102:0/532923020 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v57) v1 ==== 74+0+13996 (secure 0 0 0) 0x7f8e0c006100 con 0x7f8e1c0b4440 2021-02-10T11:24:25.196 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:24:25.196 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":57,"fsid":"68db30ea-dbce-4aef-a6df-f01c6fea0513","created":"2021-02-10 11:03:06.331557","modified":"2021-02-10 11:24:11.200509","last_up_change":"2021-02-10 11:03:32.690562","last_in_change":"2021-02-10 11:03:32.690562","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:03:37.322519","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:24:08.122188","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":11,"pool_name":"cephfs_data","create_time":"2021-02-10 11:24:09.191606","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"be691b4d-a039-408a-af80-9d53f02b6f1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6818","nonce":36088},{"type":"v1","addr":"172.21.2.102:6820","nonce":36088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6822","nonce":36088},{"type":"v1","addr":"172.21.2.102:6824","nonce":36088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6830","nonce":36088},{"type":"v1","addr":"172.21.2.102:6832","nonce":36088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6826","nonce":36088},{"type":"v1","addr":"172.21.2.102:6828","nonce":36088}]},"public_addr":"172.21.2.102:6820/36088","cluster_addr":"172.21.2.102:6824/36088","heartbeat_back_addr":"172.21.2.102:6832/36088","heartbeat_front_addr":"172.21.2.102:6828/36088","state":["exists","up"]},{"osd":1,"uuid":"64e3ccc8-9c69-4d7c-865e-af8e83822635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6819","nonce":36087},{"type":"v1","addr":"172.21.2.102:6821","nonce":36087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6823","nonce":36087},{"type":"v1","addr":"172.21.2.102:6825","nonce":36087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6831","nonce":36087},{"type":"v1","addr":"172.21.2.102:6833","nonce":36087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6827","nonce":36087},{"type":"v1","addr":"172.21.2.102:6829","nonce":36087}]},"public_addr":"172.21.2.102:6821/36087","cluster_addr":"172.21.2.102:6825/36087","heartbeat_back_addr":"172.21.2.102:6833/36087","heartbeat_front_addr":"172.21.2.102:6829/36087","state":["exists","up"]},{"osd":2,"uuid":"153700af-747f-481f-bdf4-d6072d3a842f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6810","nonce":36089},{"type":"v1","addr":"172.21.2.102:6811","nonce":36089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6812","nonce":36089},{"type":"v1","addr":"172.21.2.102:6813","nonce":36089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6816","nonce":36089},{"type":"v1","addr":"172.21.2.102:6817","nonce":36089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6814","nonce":36089},{"type":"v1","addr":"172.21.2.102:6815","nonce":36089}]},"public_addr":"172.21.2.102:6811/36089","cluster_addr":"172.21.2.102:6813/36089","heartbeat_back_addr":"172.21.2.102:6817/36089","heartbeat_front_addr":"172.21.2.102:6815/36089","state":["exists","up"]},{"osd":3,"uuid":"cbc678f4-83a9-45e6-8830-2f7c232b1ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6802","nonce":36090},{"type":"v1","addr":"172.21.2.102:6803","nonce":36090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6804","nonce":36090},{"type":"v1","addr":"172.21.2.102:6805","nonce":36090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6808","nonce":36090},{"type":"v1","addr":"172.21.2.102:6809","nonce":36090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6806","nonce":36090},{"type":"v1","addr":"172.21.2.102:6807","nonce":36090}]},"public_addr":"172.21.2.102:6803/36090","cluster_addr":"172.21.2.102:6805/36090","heartbeat_back_addr":"172.21.2.102:6809/36090","heartbeat_front_addr":"172.21.2.102:6807/36090","state":["exists","up"]},{"osd":4,"uuid":"c01e4008-b258-4f10-afd2-9e7f37214351","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6808","nonce":36368},{"type":"v1","addr":"172.21.2.113:6811","nonce":36368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6814","nonce":36368},{"type":"v1","addr":"172.21.2.113:6818","nonce":36368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6826","nonce":36368},{"type":"v1","addr":"172.21.2.113:6828","nonce":36368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6820","nonce":36368},{"type":"v1","addr":"172.21.2.113:6824","nonce":36368}]},"public_addr":"172.21.2.113:6811/36368","cluster_addr":"172.21.2.113:6818/36368","heartbeat_back_addr":"172.21.2.113:6828/36368","heartbeat_front_addr":"172.21.2.113:6824/36368","state":["exists","up"]},{"osd":5,"uuid":"551f6007-daf8-4a98-8ee0-c2b16bcccb5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6802","nonce":36366},{"type":"v1","addr":"172.21.2.113:6805","nonce":36366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6807","nonce":36366},{"type":"v1","addr":"172.21.2.113:6810","nonce":36366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6819","nonce":36366},{"type":"v1","addr":"172.21.2.113:6822","nonce":36366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6812","nonce":36366},{"type":"v1","addr":"172.21.2.113:6815","nonce":36366}]},"public_addr":"172.21.2.113:6805/36366","cluster_addr":"172.21.2.113:6810/36366","heartbeat_back_addr":"172.21.2.113:6822/36366","heartbeat_front_addr":"172.21.2.113:6815/36366","state":["exists","up"]},{"osd":6,"uuid":"bf67c040-0e50-48be-b102-c3943bc9ebef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6800","nonce":36367},{"type":"v1","addr":"172.21.2.113:6801","nonce":36367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6803","nonce":36367},{"type":"v1","addr":"172.21.2.113:6804","nonce":36367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6813","nonce":36367},{"type":"v1","addr":"172.21.2.113:6816","nonce":36367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6806","nonce":36367},{"type":"v1","addr":"172.21.2.113:6809","nonce":36367}]},"public_addr":"172.21.2.113:6801/36367","cluster_addr":"172.21.2.113:6804/36367","heartbeat_back_addr":"172.21.2.113:6816/36367","heartbeat_front_addr":"172.21.2.113:6809/36367","state":["exists","up"]},{"osd":7,"uuid":"f5461f80-018d-428f-be89-2eb2bcbe7bb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6817","nonce":36365},{"type":"v1","addr":"172.21.2.113:6821","nonce":36365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6823","nonce":36365},{"type":"v1","addr":"172.21.2.113:6825","nonce":36365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6830","nonce":36365},{"type":"v1","addr":"172.21.2.113:6831","nonce":36365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6827","nonce":36365},{"type":"v1","addr":"172.21.2.113:6829","nonce":36365}]},"public_addr":"172.21.2.113:6821/36365","cluster_addr":"172.21.2.113:6825/36365","heartbeat_back_addr":"172.21.2.113:6831/36365","heartbeat_front_addr":"172.21.2.113:6829/36365","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:24:25.197 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:25.200 7f8e21183700 1 -- 172.21.2.102:0/532923020 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8e000186c0 msgr2=0x7f8e0001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:25.198 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:25.200 7f8e21183700 1 --2- 172.21.2.102:0/532923020 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8e000186c0 0x7f8e0001abc0 crc :-1 s=READY pgs=631 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:25.198 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:25.200 7f8e21183700 1 -- 172.21.2.102:0/532923020 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8e1c0b4440 msgr2=0x7f8e1c0b4840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:25.198 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:25.200 7f8e21183700 1 --2- 172.21.2.102:0/532923020 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8e1c0b4440 0x7f8e1c0b4840 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 rx=0x7f8e0c00b5c0 tx=0x7f8e0c00b960).stop 2021-02-10T11:24:25.198 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:25.200 7f8e21183700 1 -- 172.21.2.102:0/532923020 shutdown_connections 2021-02-10T11:24:25.198 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:25.200 7f8e21183700 1 --2- 172.21.2.102:0/532923020 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8e000186c0 0x7f8e0001abc0 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:25.198 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:25.200 7f8e21183700 1 --2- 172.21.2.102:0/532923020 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8e1c1952d0 0x7f8e1c1956d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:25.198 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:25.200 7f8e21183700 1 --2- 172.21.2.102:0/532923020 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8e1c0b4440 0x7f8e1c0b4840 unknown :-1 s=CLOSED pgs=1352 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:25.198 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:25.200 7f8e21183700 1 --2- 172.21.2.102:0/532923020 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8e1c1ade10 0x7f8e1c09d250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:25.198 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:25.200 7f8e21183700 1 -- 172.21.2.102:0/532923020 >> 172.21.2.102:0/532923020 conn(0x7f8e1c0937a0 msgr2=0x7f8e1c197e80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:25.198 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:25.200 7f8e21183700 1 -- 172.21.2.102:0/532923020 shutdown_connections 2021-02-10T11:24:25.199 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:25.200 7f8e21183700 1 -- 172.21.2.102:0/532923020 wait complete. 2021-02-10T11:24:25.207 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:03:37.322519', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 10, 'pool_name': 'cephfs_metadata', 'create_time': '2021-02-10 11:24:08.122188', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '56', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 11, 'pool_name': 'cephfs_data', 'create_time': '2021-02-10 11:24:09.191606', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '57', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-02-10T11:24:25.207 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:24:25.638 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:25.624 7f7fd2cab700 1 Processor -- start 2021-02-10T11:24:25.638 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:25.624 7f7fd2cab700 1 -- start start 2021-02-10T11:24:25.638 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:25.624 7f7fd2cab700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f7fcc1b0470 0x7f7fcc1b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:25.638 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:25.624 7f7fd2cab700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f7fcc0ef610 0x7f7fcc0efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:25.638 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:25.624 7f7fd2cab700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fcc0f0b50 con 0x7f7fcc0f0030 2021-02-10T11:24:25.638 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:25.624 7f7fd2cab700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7fcc0e52e0 con 0x7f7fcc0ef610 2021-02-10T11:24:25.638 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:25.624 7f7fd2cab700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7fcc0e5490 con 0x7f7fcc1b0470 2021-02-10T11:24:25.638 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:25.624 7f7fcbfff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f7fcc0ef610 0x7f7fcc0efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:25.638 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:25.625 7f7fcbfff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f7fcc0ef610 0x7f7fcc0efa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:57200/0 (socket says 172.21.2.102:57200) 2021-02-10T11:24:25.639 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:25.625 7f7fcbfff700 1 -- 172.21.2.102:0/2966321391 learned_addr learned my addr 172.21.2.102:0/2966321391 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:24:25.639 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:25.625 7f7fcbfff700 1 -- 172.21.2.102:0/2966321391 >> v2:172.21.2.113:3300/0 conn(0x7f7fcc1b0470 msgr2=0x7f7fcc1b0870 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:24:25.639 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:25.625 7f7fcbfff700 1 --2- 172.21.2.102:0/2966321391 >> v2:172.21.2.113:3300/0 conn(0x7f7fcc1b0470 0x7f7fcc1b0870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:25.639 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:25.625 7f7fcbfff700 1 -- 172.21.2.102:0/2966321391 >> v1:172.21.2.102:6789/0 conn(0x7f7fcc0f0030 legacy=0x7f7fcc0f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:25.639 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:25.625 7f7fcbfff700 1 -- 172.21.2.102:0/2966321391 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fcc0f0430 con 0x7f7fcc0ef610 2021-02-10T11:24:25.639 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:25.625 7f7fcbfff700 1 --2- 172.21.2.102:0/2966321391 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7fcc0ef610 0x7f7fcc0efa10 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 rx=0x7f7fb8006700 tx=0x7f7fb8009b80).ready entity=mon.0 client_cookie=ece09d92e9938a9b server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:25.639 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:25.625 7f7fcb7fe700 1 -- 172.21.2.102:0/2966321391 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7fb800c0c0 con 0x7f7fcc0ef610 2021-02-10T11:24:25.639 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:25.625 7f7fcb7fe700 1 -- 172.21.2.102:0/2966321391 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fb800c340 con 0x7f7fcc0ef610 2021-02-10T11:24:25.639 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:25.625 7f7fcb7fe700 1 -- 172.21.2.102:0/2966321391 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7fb800c820 con 0x7f7fcc0ef610 2021-02-10T11:24:25.639 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:25.625 7f7fd2cab700 1 -- 172.21.2.102:0/2966321391 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7fcc0ef610 msgr2=0x7f7fcc0efa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:25.639 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:25.625 7f7fd2cab700 1 --2- 172.21.2.102:0/2966321391 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7fcc0ef610 0x7f7fcc0efa10 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 rx=0x7f7fb8006700 tx=0x7f7fb8009b80).stop 2021-02-10T11:24:25.639 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:25.626 7f7fd2cab700 1 -- 172.21.2.102:0/2966321391 shutdown_connections 2021-02-10T11:24:25.639 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:25.626 7f7fd2cab700 1 --2- 172.21.2.102:0/2966321391 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7fcc0ef610 0x7f7fcc0efa10 unknown :-1 s=CLOSED pgs=1354 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:25.640 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:25.626 7f7fd2cab700 1 --2- 172.21.2.102:0/2966321391 >> v2:172.21.2.113:3300/0 conn(0x7f7fcc1b0470 0x7f7fcc1b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:25.640 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:25.626 7f7fd2cab700 1 -- 172.21.2.102:0/2966321391 >> 172.21.2.102:0/2966321391 conn(0x7f7fcc0ce3e0 msgr2=0x7f7fcc0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:25.640 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:25.626 7f7fd2cab700 1 -- 172.21.2.102:0/2966321391 shutdown_connections 2021-02-10T11:24:25.640 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:25.626 7f7fd2cab700 1 -- 172.21.2.102:0/2966321391 wait complete. 2021-02-10T11:24:25.641 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:25.641 7f7fd2cab700 1 Processor -- start 2021-02-10T11:24:25.641 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:25.641 7f7fd2cab700 1 -- start start 2021-02-10T11:24:25.641 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:25.641 7f7fd2cab700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7fcc0c36a0 0x7f7fcc0c2b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:25.641 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:25.641 7f7fd2cab700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7fcc0c3190 0x7f7fcc0b1d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:25.641 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:25.641 7f7fd2cab700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7fcc0b2370 0x7f7fcc0b2d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:25.641 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:25.641 7f7fd2cab700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fcc0b27c0 con 0x7f7fcc0b2370 2021-02-10T11:24:25.641 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:25.641 7f7fd2cab700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fcc0b3360 con 0x7f7fcc0c36a0 2021-02-10T11:24:25.642 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:25.641 7f7fd2cab700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fcc0b3540 con 0x7f7fcc0c3190 2021-02-10T11:24:25.642 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:25.641 7f7fd12c8700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7fcc0b2370 0x7f7fcc0b2d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:25.642 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:25.641 7f7fd12c8700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7fcc0b2370 0x7f7fcc0b2d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:57202/0 (socket says 172.21.2.102:57202) 2021-02-10T11:24:25.642 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:25.641 7f7fd12c8700 1 -- 172.21.2.102:0/196144467 learned_addr learned my addr 172.21.2.102:0/196144467 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:24:25.642 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:25.641 7f7fd12c8700 1 -- 172.21.2.102:0/196144467 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7fcc0c3190 msgr2=0x7f7fcc0b1d50 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:24:25.642 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:25.641 7f7fd12c8700 1 --2- 172.21.2.102:0/196144467 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7fcc0c3190 0x7f7fcc0b1d50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:25.642 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:25.641 7f7fd12c8700 1 -- 172.21.2.102:0/196144467 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7fcc0c36a0 msgr2=0x7f7fcc0c2b70 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:24:25.642 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:25.641 7f7fd12c8700 1 --2- 172.21.2.102:0/196144467 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7fcc0c36a0 0x7f7fcc0c2b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:25.642 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:25.641 7f7fd12c8700 1 -- 172.21.2.102:0/196144467 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fcc0b3760 con 0x7f7fcc0b2370 2021-02-10T11:24:25.642 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:25.642 7f7fd12c8700 1 --2- 172.21.2.102:0/196144467 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7fcc0b2370 0x7f7fcc0b2d40 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 rx=0x7f7fc400b920 tx=0x7f7fc400bcc0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:25.642 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:25.642 7f7fc97fa700 1 -- 172.21.2.102:0/196144467 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7fc4005840 con 0x7f7fcc0b2370 2021-02-10T11:24:25.642 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:25.642 7f7fc97fa700 1 -- 172.21.2.102:0/196144467 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fc4005ac0 con 0x7f7fcc0b2370 2021-02-10T11:24:25.643 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:25.642 7f7fd2cab700 1 -- 172.21.2.102:0/196144467 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fcc0b3760 con 0x7f7fcc0b2370 2021-02-10T11:24:25.643 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:25.642 7f7fd2cab700 1 -- 172.21.2.102:0/196144467 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fcc0b3cf0 con 0x7f7fcc0b2370 2021-02-10T11:24:25.643 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:25.642 7f7fc97fa700 1 -- 172.21.2.102:0/196144467 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7fc4006030 con 0x7f7fcc0b2370 2021-02-10T11:24:25.643 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:25.642 7f7fc97fa700 1 -- 172.21.2.102:0/196144467 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f7fc4005ca0 con 0x7f7fcc0b2370 2021-02-10T11:24:25.643 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:25.642 7f7fc97fa700 1 --2- 172.21.2.102:0/196144467 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7fa80186c0 0x7f7fa801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:25.643 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:25.642 7f7fc97fa700 1 -- 172.21.2.102:0/196144467 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f7fc4032920 con 0x7f7fcc0b2370 2021-02-10T11:24:25.643 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:25.643 7f7fd0ac7700 1 --2- 172.21.2.102:0/196144467 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7fa80186c0 0x7f7fa801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:25.643 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:25.643 7f7fd0ac7700 1 --2- 172.21.2.102:0/196144467 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7fa80186c0 0x7f7fa801abc0 crc :-1 s=READY pgs=632 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:25.671 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:25.674 7f7fd2cab700 1 -- 172.21.2.102:0/196144467 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fcc0edd10 con 0x7f7fcc0b2370 2021-02-10T11:24:25.674 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:25.676 7f7fc97fa700 1 -- 172.21.2.102:0/196144467 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f7fc4003000 con 0x7f7fcc0b2370 2021-02-10T11:24:25.848 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:25.803 7f7fd2cab700 1 -- 172.21.2.102:0/196144467 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f7fcc0e52e0 con 0x7f7fcc0b2370 2021-02-10T11:24:25.849 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:25.803 7f7fc97fa700 1 -- 172.21.2.102:0/196144467 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v57) v1 ==== 92+0+10 (secure 0 0 0) 0x7f7fc4005ca0 con 0x7f7fcc0b2370 2021-02-10T11:24:25.849 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:25.805 7f7fd2cab700 1 -- 172.21.2.102:0/196144467 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7fa80186c0 msgr2=0x7f7fa801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:25.849 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:25.805 7f7fd2cab700 1 --2- 172.21.2.102:0/196144467 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7fa80186c0 0x7f7fa801abc0 crc :-1 s=READY pgs=632 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:25.849 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:25.805 7f7fd2cab700 1 -- 172.21.2.102:0/196144467 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7fcc0b2370 msgr2=0x7f7fcc0b2d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:25.849 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:25.805 7f7fd2cab700 1 --2- 172.21.2.102:0/196144467 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7fcc0b2370 0x7f7fcc0b2d40 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 rx=0x7f7fc400b920 tx=0x7f7fc400bcc0).stop 2021-02-10T11:24:25.849 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:25.806 7f7fd2cab700 1 -- 172.21.2.102:0/196144467 shutdown_connections 2021-02-10T11:24:25.849 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:25.806 7f7fd2cab700 1 --2- 172.21.2.102:0/196144467 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7fcc0b2370 0x7f7fcc0b2d40 unknown :-1 s=CLOSED pgs=1355 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:25.849 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:25.806 7f7fd2cab700 1 --2- 172.21.2.102:0/196144467 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7fcc0c3190 0x7f7fcc0b1d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:25.849 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:25.806 7f7fd2cab700 1 --2- 172.21.2.102:0/196144467 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7fa80186c0 0x7f7fa801abc0 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:25.849 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:25.806 7f7fd2cab700 1 --2- 172.21.2.102:0/196144467 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7fcc0c36a0 0x7f7fcc0c2b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:25.849 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:25.806 7f7fd2cab700 1 -- 172.21.2.102:0/196144467 >> 172.21.2.102:0/196144467 conn(0x7f7fcc09f630 msgr2=0x7f7fcc0a0130 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:25.849 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:25.806 7f7fd2cab700 1 -- 172.21.2.102:0/196144467 shutdown_connections 2021-02-10T11:24:25.850 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:25.806 7f7fd2cab700 1 -- 172.21.2.102:0/196144467 wait complete. 2021-02-10T11:24:25.850 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:24:25.850 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-02-10T11:24:26.282 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:26.268 7f1e9fb6e700 1 Processor -- start 2021-02-10T11:24:26.282 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:26.268 7f1e9fb6e700 1 -- start start 2021-02-10T11:24:26.282 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:26.268 7f1e9fb6e700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1e981b0470 0x7f1e981b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:26.282 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:26.268 7f1e9fb6e700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f1e980ef610 0x7f1e980efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:26.282 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:26.268 7f1e9fb6e700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e980f0b50 con 0x7f1e980f0030 2021-02-10T11:24:26.282 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:26.268 7f1e9fb6e700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e980e52e0 con 0x7f1e980ef610 2021-02-10T11:24:26.283 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:26.268 7f1e9fb6e700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e980e5490 con 0x7f1e981b0470 2021-02-10T11:24:26.283 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:26.269 7f1e9d189700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f1e980ef610 0x7f1e980efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:26.283 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:26.269 7f1e9d189700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f1e980ef610 0x7f1e980efa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:57210/0 (socket says 172.21.2.102:57210) 2021-02-10T11:24:26.283 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:26.269 7f1e9e18b700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f1e980f0030 0x7f1e980f10d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:49434/0 (socket says 172.21.2.102:49434) 2021-02-10T11:24:26.283 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:26.269 7f1e9d189700 1 -- 172.21.2.102:0/3767770370 learned_addr learned my addr 172.21.2.102:0/3767770370 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:24:26.283 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:26.269 7f1e9d98a700 1 --2- 172.21.2.102:0/3767770370 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1e981b0470 0x7f1e981b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:26.283 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:26.269 7f1e9d189700 1 -- 172.21.2.102:0/3767770370 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1e981b0470 msgr2=0x7f1e981b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:24:26.283 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:26.269 7f1e9d189700 1 --2- 172.21.2.102:0/3767770370 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1e981b0470 0x7f1e981b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:26.283 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:26.269 7f1e9d189700 1 -- 172.21.2.102:0/3767770370 >> v1:172.21.2.102:6789/0 conn(0x7f1e980f0030 legacy=0x7f1e980f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:26.283 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:26.269 7f1e9d189700 1 -- 172.21.2.102:0/3767770370 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e980f0430 con 0x7f1e980ef610 2021-02-10T11:24:26.283 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:26.269 7f1e9d189700 1 --2- 172.21.2.102:0/3767770370 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1e980ef610 0x7f1e980efa10 secure :-1 s=READY pgs=1357 cs=0 l=1 rev1=1 rx=0x7f1e88006700 tx=0x7f1e88009b80).ready entity=mon.0 client_cookie=d6284973053e8067 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:26.283 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:26.269 7f1e9c988700 1 -- 172.21.2.102:0/3767770370 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1e8800c0c0 con 0x7f1e980ef610 2021-02-10T11:24:26.283 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:26.269 7f1e9c988700 1 -- 172.21.2.102:0/3767770370 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e8800c380 con 0x7f1e980ef610 2021-02-10T11:24:26.284 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:26.269 7f1e9c988700 1 -- 172.21.2.102:0/3767770370 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1e8800c7d0 con 0x7f1e980ef610 2021-02-10T11:24:26.284 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:26.269 7f1e9fb6e700 1 -- 172.21.2.102:0/3767770370 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1e980ef610 msgr2=0x7f1e980efa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:26.284 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:26.269 7f1e9fb6e700 1 --2- 172.21.2.102:0/3767770370 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1e980ef610 0x7f1e980efa10 secure :-1 s=READY pgs=1357 cs=0 l=1 rev1=1 rx=0x7f1e88006700 tx=0x7f1e88009b80).stop 2021-02-10T11:24:26.284 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:26.270 7f1e9fb6e700 1 -- 172.21.2.102:0/3767770370 shutdown_connections 2021-02-10T11:24:26.284 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:26.270 7f1e9fb6e700 1 --2- 172.21.2.102:0/3767770370 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1e980ef610 0x7f1e980efa10 unknown :-1 s=CLOSED pgs=1357 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:26.284 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:26.270 7f1e9fb6e700 1 --2- 172.21.2.102:0/3767770370 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1e981b0470 0x7f1e981b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:26.284 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:26.270 7f1e9fb6e700 1 -- 172.21.2.102:0/3767770370 >> 172.21.2.102:0/3767770370 conn(0x7f1e980ce3e0 msgr2=0x7f1e980e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:26.284 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:26.270 7f1e9fb6e700 1 -- 172.21.2.102:0/3767770370 shutdown_connections 2021-02-10T11:24:26.284 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:26.270 7f1e9fb6e700 1 -- 172.21.2.102:0/3767770370 wait complete. 2021-02-10T11:24:26.286 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:26.285 7f1e9fb6e700 1 Processor -- start 2021-02-10T11:24:26.286 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:26.285 7f1e9fb6e700 1 -- start start 2021-02-10T11:24:26.286 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:26.285 7f1e9fb6e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1e9818a2b0 0x7f1e980ab870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:26.286 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:26.285 7f1e9fb6e700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1e9818ce80 0x7f1e980ac9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:26.286 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:26.285 7f1e9fb6e700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1e980abe90 0x7f1e980acff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:26.286 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:26.285 7f1e9fb6e700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e980ac450 con 0x7f1e9818a2b0 2021-02-10T11:24:26.286 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:26.285 7f1e9fb6e700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e9818a6b0 con 0x7f1e9818ce80 2021-02-10T11:24:26.286 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:26.285 7f1e9fb6e700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e9818a960 con 0x7f1e980abe90 2021-02-10T11:24:26.286 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:26.285 7f1e9d98a700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1e9818a2b0 0x7f1e980ab870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:26.287 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:26.285 7f1e9d98a700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1e9818a2b0 0x7f1e980ab870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:57216/0 (socket says 172.21.2.102:57216) 2021-02-10T11:24:26.287 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:26.285 7f1e9d98a700 1 -- 172.21.2.102:0/196521167 learned_addr learned my addr 172.21.2.102:0/196521167 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:24:26.287 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:26.286 7f1e9e18b700 1 --2- 172.21.2.102:0/196521167 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1e980abe90 0x7f1e980acff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:26.287 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:26.286 7f1e9d98a700 1 -- 172.21.2.102:0/196521167 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1e980abe90 msgr2=0x7f1e980acff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:26.287 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:26.286 7f1e9d189700 1 --2- 172.21.2.102:0/196521167 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1e9818ce80 0x7f1e980ac9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:26.287 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:26.286 7f1e9d98a700 1 --2- 172.21.2.102:0/196521167 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1e980abe90 0x7f1e980acff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:26.287 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:26.286 7f1e9d98a700 1 -- 172.21.2.102:0/196521167 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1e9818ce80 msgr2=0x7f1e980ac9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:26.287 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:26.286 7f1e9d98a700 1 --2- 172.21.2.102:0/196521167 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1e9818ce80 0x7f1e980ac9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:26.287 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:26.286 7f1e9d98a700 1 -- 172.21.2.102:0/196521167 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e980ad610 con 0x7f1e9818a2b0 2021-02-10T11:24:26.287 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:26.286 7f1e9e18b700 1 --2- 172.21.2.102:0/196521167 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1e980abe90 0x7f1e980acff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:24:26.287 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:26.286 7f1e9d189700 1 --2- 172.21.2.102:0/196521167 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1e9818ce80 0x7f1e980ac9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:24:26.287 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:26.286 7f1e9d98a700 1 --2- 172.21.2.102:0/196521167 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1e9818a2b0 0x7f1e980ab870 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 rx=0x7f1e94006f00 tx=0x7f1e9400b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:26.287 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:26.286 7f1e8e7fc700 1 -- 172.21.2.102:0/196521167 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1e9400bd30 con 0x7f1e9818a2b0 2021-02-10T11:24:26.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:26.286 7f1e8e7fc700 1 -- 172.21.2.102:0/196521167 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e94002030 con 0x7f1e9818a2b0 2021-02-10T11:24:26.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:26.286 7f1e9fb6e700 1 -- 172.21.2.102:0/196521167 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e980ad610 con 0x7f1e9818a2b0 2021-02-10T11:24:26.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:26.286 7f1e8e7fc700 1 -- 172.21.2.102:0/196521167 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1e940024c0 con 0x7f1e9818a2b0 2021-02-10T11:24:26.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:26.286 7f1e9fb6e700 1 -- 172.21.2.102:0/196521167 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e980ad610 con 0x7f1e9818a2b0 2021-02-10T11:24:26.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:26.287 7f1e8e7fc700 1 -- 172.21.2.102:0/196521167 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1e94002210 con 0x7f1e9818a2b0 2021-02-10T11:24:26.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:26.287 7f1e8e7fc700 1 --2- 172.21.2.102:0/196521167 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1e80018730 0x7f1e8001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:26.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:26.287 7f1e8e7fc700 1 -- 172.21.2.102:0/196521167 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f1e94032610 con 0x7f1e9818a2b0 2021-02-10T11:24:26.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:26.287 7f1e9d189700 1 --2- 172.21.2.102:0/196521167 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1e80018730 0x7f1e8001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:26.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:26.287 7f1e9d189700 1 --2- 172.21.2.102:0/196521167 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1e80018730 0x7f1e8001ac10 crc :-1 s=READY pgs=633 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:26.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:26.317 7f1e9fb6e700 1 -- 172.21.2.102:0/196521167 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e98105ae0 con 0x7f1e9818a2b0 2021-02-10T11:24:26.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:26.320 7f1e8e7fc700 1 -- 172.21.2.102:0/196521167 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f1e94002240 con 0x7f1e9818a2b0 2021-02-10T11:24:26.447 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:26.449 7f1e9fb6e700 1 -- 172.21.2.102:0/196521167 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7f1e980f0b50 con 0x7f1e9818a2b0 2021-02-10T11:24:26.447 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:26.449 7f1e8e7fc700 1 -- 172.21.2.102:0/196521167 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v57) v1 ==== 104+0+10 (secure 0 0 0) 0x7f1e94002030 con 0x7f1e9818a2b0 2021-02-10T11:24:26.448 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:24:26.450 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:26.451 7f1e9fb6e700 1 -- 172.21.2.102:0/196521167 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1e80018730 msgr2=0x7f1e8001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:26.450 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:26.451 7f1e9fb6e700 1 --2- 172.21.2.102:0/196521167 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1e80018730 0x7f1e8001ac10 crc :-1 s=READY pgs=633 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:26.450 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:26.452 7f1e9fb6e700 1 -- 172.21.2.102:0/196521167 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1e9818a2b0 msgr2=0x7f1e980ab870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:26.450 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:26.452 7f1e9fb6e700 1 --2- 172.21.2.102:0/196521167 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1e9818a2b0 0x7f1e980ab870 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 rx=0x7f1e94006f00 tx=0x7f1e9400b940).stop 2021-02-10T11:24:26.450 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:26.452 7f1e9fb6e700 1 -- 172.21.2.102:0/196521167 shutdown_connections 2021-02-10T11:24:26.450 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:26.452 7f1e9fb6e700 1 --2- 172.21.2.102:0/196521167 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1e980abe90 0x7f1e980acff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:26.450 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:26.452 7f1e9fb6e700 1 --2- 172.21.2.102:0/196521167 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1e80018730 0x7f1e8001ac10 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:26.450 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:26.452 7f1e9fb6e700 1 --2- 172.21.2.102:0/196521167 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1e9818ce80 0x7f1e980ac9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:26.450 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:26.452 7f1e9fb6e700 1 --2- 172.21.2.102:0/196521167 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1e9818a2b0 0x7f1e980ab870 unknown :-1 s=CLOSED pgs=1358 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:26.450 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:26.452 7f1e9fb6e700 1 -- 172.21.2.102:0/196521167 >> 172.21.2.102:0/196521167 conn(0x7f1e980991b0 msgr2=0x7f1e980a7b20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:26.450 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:26.452 7f1e9fb6e700 1 -- 172.21.2.102:0/196521167 shutdown_connections 2021-02-10T11:24:26.451 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:26.452 7f1e9fb6e700 1 -- 172.21.2.102:0/196521167 wait complete. 2021-02-10T11:24:26.458 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-02-10T11:24:26.889 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:26.874 7f4b8bc28700 1 Processor -- start 2021-02-10T11:24:26.889 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:26.874 7f4b8bc28700 1 -- start start 2021-02-10T11:24:26.890 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:26.875 7f4b8bc28700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4b841b0870 0x7f4b840f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:26.890 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:26.875 7f4b8bc28700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b841b0c70 con 0x7f4b840efe00 2021-02-10T11:24:26.890 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:26.875 7f4b8bc28700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b840f0b50 con 0x7f4b841b0470 2021-02-10T11:24:26.890 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:26.875 7f4b8bc28700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b840e5ea0 con 0x7f4b841b0870 2021-02-10T11:24:26.890 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:26.875 7f4b8a245700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f4b840efe00 0x7f4b840f0200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:49450/0 (socket says 172.21.2.102:49450) 2021-02-10T11:24:26.890 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:26.875 7f4b8a245700 1 -- 172.21.2.102:0/2243748795 learned_addr learned my addr 172.21.2.102:0/2243748795 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:24:26.890 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:26.875 7f4b88a42700 1 -- 172.21.2.102:0/2243748795 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 941949819 0 0) 0x7f4b7c001500 con 0x7f4b840efe00 2021-02-10T11:24:26.890 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:26.875 7f4b88a42700 1 -- 172.21.2.102:0/2243748795 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b70002fe0 con 0x7f4b840efe00 2021-02-10T11:24:26.890 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:26.875 7f4b89243700 1 --2- 172.21.2.102:0/2243748795 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4b841b0870 0x7f4b840f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:26.890 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:26.876 7f4b88a42700 1 -- 172.21.2.102:0/2243748795 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3648034779 0 0) 0x7f4b7c001460 con 0x7f4b840efe00 2021-02-10T11:24:26.890 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:26.876 7f4b88a42700 1 -- 172.21.2.102:0/2243748795 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4b841b0870 msgr2=0x7f4b840f20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:24:26.890 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:26.876 7f4b88a42700 1 --2- 172.21.2.102:0/2243748795 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4b841b0870 0x7f4b840f20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:26.891 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:26.876 7f4b88a42700 1 -- 172.21.2.102:0/2243748795 >> v1:172.21.2.113:6789/0 conn(0x7f4b841b0470 legacy=0x7f4b840ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:26.891 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:26.876 7f4b88a42700 1 -- 172.21.2.102:0/2243748795 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b840e52e0 con 0x7f4b840efe00 2021-02-10T11:24:26.891 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:26.876 7f4b88a42700 1 -- 172.21.2.102:0/2243748795 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7f4b7c001980 con 0x7f4b840efe00 2021-02-10T11:24:26.891 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:26.876 7f4b88a42700 1 -- 172.21.2.102:0/2243748795 >> v1:172.21.2.102:6789/0 conn(0x7f4b840efe00 legacy=0x7f4b840f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:26.891 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:26.876 7f4b88a42700 1 --2- 172.21.2.102:0/2243748795 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4b70003af0 0x7f4b70005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:26.891 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:26.876 7f4b88a42700 1 --2- 172.21.2.102:0/2243748795 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4b70006790 0x7f4b70008c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:26.891 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:26.876 7f4b88a42700 1 --2- 172.21.2.102:0/2243748795 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4b70009590 0x7f4b7000ba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:26.891 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:26.876 7f4b88a42700 1 -- 172.21.2.102:0/2243748795 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b7000c2d0 con 0x7f4b70006790 2021-02-10T11:24:26.891 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:26.876 7f4b88a42700 1 -- 172.21.2.102:0/2243748795 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b7000c4f0 con 0x7f4b70003af0 2021-02-10T11:24:26.891 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:26.876 7f4b88a42700 1 -- 172.21.2.102:0/2243748795 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b7000c710 con 0x7f4b70009590 2021-02-10T11:24:26.891 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:26.876 7f4b89243700 1 --2- 172.21.2.102:0/2243748795 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4b70006790 0x7f4b70008c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:26.891 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:26.876 7f4b89a44700 1 --2- 172.21.2.102:0/2243748795 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4b70003af0 0x7f4b70005fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:26.891 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:26.876 7f4b8a245700 1 --2- 172.21.2.102:0/2243748795 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4b70009590 0x7f4b7000ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:26.892 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:26.876 7f4b89243700 1 -- 172.21.2.102:0/2243748795 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4b70009590 msgr2=0x7f4b7000ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:24:26.892 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:26.876 7f4b89243700 1 --2- 172.21.2.102:0/2243748795 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4b70009590 0x7f4b7000ba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:26.892 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:26.876 7f4b89243700 1 -- 172.21.2.102:0/2243748795 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4b70003af0 msgr2=0x7f4b70005fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:24:26.892 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:26.876 7f4b89243700 1 --2- 172.21.2.102:0/2243748795 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4b70003af0 0x7f4b70005fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:26.892 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:26.876 7f4b89243700 1 -- 172.21.2.102:0/2243748795 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b70050980 con 0x7f4b70006790 2021-02-10T11:24:26.892 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:26.876 7f4b8a245700 1 --2- 172.21.2.102:0/2243748795 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4b70009590 0x7f4b7000ba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:24:26.892 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:26.876 7f4b89243700 1 --2- 172.21.2.102:0/2243748795 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4b70006790 0x7f4b70008c90 secure :-1 s=READY pgs=1360 cs=0 l=1 rev1=1 rx=0x7f4b74009310 tx=0x7f4b74009a90).ready entity=mon.0 client_cookie=5c3d622e12fd9228 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:26.892 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:26.876 7f4b88a42700 1 -- 172.21.2.102:0/2243748795 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4b740063c0 con 0x7f4b70006790 2021-02-10T11:24:26.892 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:26.876 7f4b88a42700 1 -- 172.21.2.102:0/2243748795 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b7400c0c0 con 0x7f4b70006790 2021-02-10T11:24:26.892 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:26.876 7f4b88a42700 1 -- 172.21.2.102:0/2243748795 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4b7400c550 con 0x7f4b70006790 2021-02-10T11:24:26.892 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:26.876 7f4b8bc28700 1 -- 172.21.2.102:0/2243748795 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4b70006790 msgr2=0x7f4b70008c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:26.892 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:26.876 7f4b8bc28700 1 --2- 172.21.2.102:0/2243748795 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4b70006790 0x7f4b70008c90 secure :-1 s=READY pgs=1360 cs=0 l=1 rev1=1 rx=0x7f4b74009310 tx=0x7f4b74009a90).stop 2021-02-10T11:24:26.892 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:26.876 7f4b8a245700 1 -- 172.21.2.102:0/2243748795 reap_dead start 2021-02-10T11:24:26.893 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:26.877 7f4b8bc28700 1 -- 172.21.2.102:0/2243748795 shutdown_connections 2021-02-10T11:24:26.893 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:26.877 7f4b8bc28700 1 -- 172.21.2.102:0/2243748795 >> 172.21.2.102:0/2243748795 conn(0x7f4b840ce3e0 msgr2=0x7f4b840e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:26.893 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:26.877 7f4b8bc28700 1 -- 172.21.2.102:0/2243748795 shutdown_connections 2021-02-10T11:24:26.893 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:26.877 7f4b8bc28700 1 -- 172.21.2.102:0/2243748795 wait complete. 2021-02-10T11:24:26.893 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:26.892 7f4b8bc28700 1 Processor -- start 2021-02-10T11:24:26.893 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:26.892 7f4b8bc28700 1 -- start start 2021-02-10T11:24:26.895 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:26.892 7f4b8bc28700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4b840bf920 0x7f4b840bfd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:26.895 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:26.892 7f4b8bc28700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4b840b6570 0x7f4b840b6970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:26.895 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:26.892 7f4b8bc28700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4b841a7430 0x7f4b841a7830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:26.895 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:26.892 7f4b8bc28700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b840c0440 con 0x7f4b840bf920 2021-02-10T11:24:26.895 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:26.892 7f4b8bc28700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b840924b0 con 0x7f4b841a7430 2021-02-10T11:24:26.895 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:26.892 7f4b8bc28700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b84092970 con 0x7f4b840b6570 2021-02-10T11:24:26.895 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:26.892 7f4b89a44700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4b840bf920 0x7f4b840bfd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:26.895 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:26.893 7f4b89a44700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4b840bf920 0x7f4b840bfd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:57236/0 (socket says 172.21.2.102:57236) 2021-02-10T11:24:26.895 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:26.893 7f4b89a44700 1 -- 172.21.2.102:0/1796612153 learned_addr learned my addr 172.21.2.102:0/1796612153 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:24:26.896 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:26.893 7f4b89a44700 1 -- 172.21.2.102:0/1796612153 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4b840b6570 msgr2=0x7f4b840b6970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:26.896 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:26.893 7f4b89a44700 1 --2- 172.21.2.102:0/1796612153 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4b840b6570 0x7f4b840b6970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:26.896 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:26.893 7f4b89a44700 1 -- 172.21.2.102:0/1796612153 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4b841a7430 msgr2=0x7f4b841a7830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:26.896 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:26.893 7f4b89a44700 1 --2- 172.21.2.102:0/1796612153 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4b841a7430 0x7f4b841a7830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:26.896 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:26.893 7f4b89a44700 1 -- 172.21.2.102:0/1796612153 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b841a7e50 con 0x7f4b840bf920 2021-02-10T11:24:26.896 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:26.893 7f4b89a44700 1 --2- 172.21.2.102:0/1796612153 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4b840bf920 0x7f4b840bfd20 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 rx=0x7f4b80006f10 tx=0x7f4b8000b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:26.896 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:26.893 7f4b7a7fc700 1 -- 172.21.2.102:0/1796612153 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4b8000bd80 con 0x7f4b840bf920 2021-02-10T11:24:26.896 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:26.893 7f4b7a7fc700 1 -- 172.21.2.102:0/1796612153 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b80006030 con 0x7f4b840bf920 2021-02-10T11:24:26.896 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:26.893 7f4b7a7fc700 1 -- 172.21.2.102:0/1796612153 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4b800052e0 con 0x7f4b840bf920 2021-02-10T11:24:26.898 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:26.893 7f4b8bc28700 1 -- 172.21.2.102:0/1796612153 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b841a7e50 con 0x7f4b840bf920 2021-02-10T11:24:26.898 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:26.893 7f4b8bc28700 1 -- 172.21.2.102:0/1796612153 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b841a7e50 con 0x7f4b840bf920 2021-02-10T11:24:26.898 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:26.894 7f4b7a7fc700 1 -- 172.21.2.102:0/1796612153 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4b80006210 con 0x7f4b840bf920 2021-02-10T11:24:26.898 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:26.894 7f4b7a7fc700 1 --2- 172.21.2.102:0/1796612153 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4b6c018730 0x7f4b6c01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:26.898 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:26.894 7f4b7a7fc700 1 -- 172.21.2.102:0/1796612153 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f4b80032610 con 0x7f4b840bf920 2021-02-10T11:24:26.898 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:26.894 7f4b89243700 1 --2- 172.21.2.102:0/1796612153 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4b6c018730 0x7f4b6c01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:26.899 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:26.895 7f4b89243700 1 --2- 172.21.2.102:0/1796612153 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4b6c018730 0x7f4b6c01ac10 crc :-1 s=READY pgs=634 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:26.922 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:26.925 7f4b8bc28700 1 -- 172.21.2.102:0/1796612153 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b84105b40 con 0x7f4b840bf920 2021-02-10T11:24:26.925 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:26.927 7f4b7a7fc700 1 -- 172.21.2.102:0/1796612153 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f4b80006ba0 con 0x7f4b840bf920 2021-02-10T11:24:27.053 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:27.056 7f4b8bc28700 1 -- 172.21.2.102:0/1796612153 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7f4b840f0b50 con 0x7f4b840bf920 2021-02-10T11:24:27.054 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:27.056 7f4b7a7fc700 1 -- 172.21.2.102:0/1796612153 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v57) v1 ==== 100+0+10 (secure 0 0 0) 0x7f4b8000bd80 con 0x7f4b840bf920 2021-02-10T11:24:27.054 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:24:27.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:27.058 7f4b8bc28700 1 -- 172.21.2.102:0/1796612153 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4b6c018730 msgr2=0x7f4b6c01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:27.056 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:27.058 7f4b8bc28700 1 --2- 172.21.2.102:0/1796612153 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4b6c018730 0x7f4b6c01ac10 crc :-1 s=READY pgs=634 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:27.056 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:27.058 7f4b8bc28700 1 -- 172.21.2.102:0/1796612153 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4b840bf920 msgr2=0x7f4b840bfd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:27.056 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:27.058 7f4b8bc28700 1 --2- 172.21.2.102:0/1796612153 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4b840bf920 0x7f4b840bfd20 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 rx=0x7f4b80006f10 tx=0x7f4b8000b990).stop 2021-02-10T11:24:27.056 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:27.058 7f4b8bc28700 1 -- 172.21.2.102:0/1796612153 shutdown_connections 2021-02-10T11:24:27.056 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:27.058 7f4b8bc28700 1 --2- 172.21.2.102:0/1796612153 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4b6c018730 0x7f4b6c01ac10 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:27.056 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:27.058 7f4b8bc28700 1 --2- 172.21.2.102:0/1796612153 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4b841a7430 0x7f4b841a7830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:27.057 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:27.058 7f4b8bc28700 1 --2- 172.21.2.102:0/1796612153 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4b840b6570 0x7f4b840b6970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:27.057 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:27.058 7f4b8bc28700 1 --2- 172.21.2.102:0/1796612153 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4b840bf920 0x7f4b840bfd20 unknown :-1 s=CLOSED pgs=1361 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:27.057 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:27.058 7f4b8bc28700 1 -- 172.21.2.102:0/1796612153 >> 172.21.2.102:0/1796612153 conn(0x7f4b840b50c0 msgr2=0x7f4b84092d90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:27.057 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:27.058 7f4b8bc28700 1 -- 172.21.2.102:0/1796612153 shutdown_connections 2021-02-10T11:24:27.057 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:27.058 7f4b8bc28700 1 -- 172.21.2.102:0/1796612153 wait complete. 2021-02-10T11:24:27.064 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-02-10T11:24:27.064 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:24:27.489 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:27.475 7fe947a12700 1 Processor -- start 2021-02-10T11:24:27.489 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:27.475 7fe947a12700 1 -- start start 2021-02-10T11:24:27.489 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:27.476 7fe947a12700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fe9400e5470 0x7fe94007e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:27.489 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:27.476 7fe947a12700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe94007ed80 con 0x7fe94007f300 2021-02-10T11:24:27.489 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:27.476 7fe947a12700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9401b4010 con 0x7fe9400e5070 2021-02-10T11:24:27.489 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:27.476 7fe947a12700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9400d1790 con 0x7fe9400e5470 2021-02-10T11:24:27.489 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:27.476 7fe94502d700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fe9400e5470 0x7fe94007e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:27.490 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:27.476 7fe94602f700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fe94007f300 0x7fe94007f700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:49470/0 (socket says 172.21.2.102:49470) 2021-02-10T11:24:27.490 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:27.476 7fe94602f700 1 -- 172.21.2.102:0/3401372015 learned_addr learned my addr 172.21.2.102:0/3401372015 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:24:27.490 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:27.476 7fe94502d700 1 -- 172.21.2.102:0/3401372015 >> v1:172.21.2.113:6789/0 conn(0x7fe9400e5070 legacy=0x7fe9401b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:24:27.490 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:27.476 7fe94502d700 1 -- 172.21.2.102:0/3401372015 >> v1:172.21.2.102:6789/0 conn(0x7fe94007f300 legacy=0x7fe94007f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:27.490 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:27.476 7fe94502d700 1 -- 172.21.2.102:0/3401372015 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9400d19b0 con 0x7fe9400e5470 2021-02-10T11:24:27.490 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:27.477 7fe94502d700 1 --2- 172.21.2.102:0/3401372015 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe9400e5470 0x7fe94007e6a0 secure :-1 s=READY pgs=1363 cs=0 l=1 rev1=1 rx=0x7fe93c006650 tx=0x7fe93c009a80).ready entity=mon.0 client_cookie=c53a9da981a2aa4b server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:27.490 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:27.477 7fe94482c700 1 -- 172.21.2.102:0/3401372015 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe93c00c0c0 con 0x7fe9400e5470 2021-02-10T11:24:27.490 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:27.477 7fe94482c700 1 -- 172.21.2.102:0/3401372015 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe93c00c340 con 0x7fe9400e5470 2021-02-10T11:24:27.490 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:27.477 7fe94482c700 1 -- 172.21.2.102:0/3401372015 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe93c00c860 con 0x7fe9400e5470 2021-02-10T11:24:27.490 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:27.477 7fe947a12700 1 -- 172.21.2.102:0/3401372015 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe9400e5470 msgr2=0x7fe94007e6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:27.490 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:27.477 7fe947a12700 1 --2- 172.21.2.102:0/3401372015 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe9400e5470 0x7fe94007e6a0 secure :-1 s=READY pgs=1363 cs=0 l=1 rev1=1 rx=0x7fe93c006650 tx=0x7fe93c009a80).stop 2021-02-10T11:24:27.490 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:27.477 7fe947a12700 1 -- 172.21.2.102:0/3401372015 shutdown_connections 2021-02-10T11:24:27.491 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:27.477 7fe947a12700 1 --2- 172.21.2.102:0/3401372015 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe9400e5470 0x7fe94007e6a0 unknown :-1 s=CLOSED pgs=1363 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:27.491 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:27.477 7fe947a12700 1 -- 172.21.2.102:0/3401372015 >> 172.21.2.102:0/3401372015 conn(0x7fe9400ce690 msgr2=0x7fe9400e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:27.491 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:27.477 7fe947a12700 1 -- 172.21.2.102:0/3401372015 shutdown_connections 2021-02-10T11:24:27.491 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:27.478 7fe947a12700 1 -- 172.21.2.102:0/3401372015 wait complete. 2021-02-10T11:24:27.494 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:27.492 7fe947a12700 1 Processor -- start 2021-02-10T11:24:27.494 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:27.492 7fe947a12700 1 -- start start 2021-02-10T11:24:27.494 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:27.492 7fe947a12700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe940096380 0x7fe940090ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:27.494 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:27.492 7fe947a12700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe940096780 0x7fe940091100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:27.495 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:27.492 7fe947a12700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe940091720 0x7fe940097ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:27.495 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:27.492 7fe947a12700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe940108870 con 0x7fe940096780 2021-02-10T11:24:27.495 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:27.492 7fe947a12700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe940096b80 con 0x7fe940091720 2021-02-10T11:24:27.495 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:27.492 7fe947a12700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe940090560 con 0x7fe940096380 2021-02-10T11:24:27.495 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:27.492 7fe94502d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe940096780 0x7fe940091100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:27.495 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:27.492 7fe94502d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe940096780 0x7fe940091100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:57252/0 (socket says 172.21.2.102:57252) 2021-02-10T11:24:27.496 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:27.492 7fe94502d700 1 -- 172.21.2.102:0/69639525 learned_addr learned my addr 172.21.2.102:0/69639525 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:24:27.496 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:27.493 7fe94502d700 1 -- 172.21.2.102:0/69639525 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe940096380 msgr2=0x7fe940090ae0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:24:27.496 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:27.493 7fe94502d700 1 --2- 172.21.2.102:0/69639525 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe940096380 0x7fe940090ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:27.496 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:27.493 7fe94502d700 1 -- 172.21.2.102:0/69639525 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe940091720 msgr2=0x7fe940097ad0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:24:27.496 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:27.493 7fe94502d700 1 --2- 172.21.2.102:0/69639525 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe940091720 0x7fe940097ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:27.497 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:27.493 7fe94502d700 1 -- 172.21.2.102:0/69639525 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe940108440 con 0x7fe940096780 2021-02-10T11:24:27.497 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:27.493 7fe94502d700 1 --2- 172.21.2.102:0/69639525 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe940096780 0x7fe940091100 secure :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 rx=0x7fe93c009650 tx=0x7fe93c00b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:27.497 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:27.493 7fe9327fc700 1 -- 172.21.2.102:0/69639525 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe93c009360 con 0x7fe940096780 2021-02-10T11:24:27.497 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:27.493 7fe9327fc700 1 -- 172.21.2.102:0/69639525 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe93c00bde0 con 0x7fe940096780 2021-02-10T11:24:27.497 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:27.493 7fe947a12700 1 -- 172.21.2.102:0/69639525 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe940108440 con 0x7fe940096780 2021-02-10T11:24:27.497 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:27.493 7fe9327fc700 1 -- 172.21.2.102:0/69639525 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe93c0052e0 con 0x7fe940096780 2021-02-10T11:24:27.498 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:27.493 7fe947a12700 1 -- 172.21.2.102:0/69639525 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe940108440 con 0x7fe940096780 2021-02-10T11:24:27.498 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:27.494 7fe9327fc700 1 -- 172.21.2.102:0/69639525 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fe93c006030 con 0x7fe940096780 2021-02-10T11:24:27.498 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:27.494 7fe9327fc700 1 --2- 172.21.2.102:0/69639525 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe928018730 0x7fe92801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:27.498 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:27.494 7fe9327fc700 1 -- 172.21.2.102:0/69639525 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fe93c006030 con 0x7fe940096780 2021-02-10T11:24:27.498 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:27.494 7fe94582e700 1 --2- 172.21.2.102:0/69639525 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe928018730 0x7fe92801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:27.499 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:27.494 7fe94582e700 1 --2- 172.21.2.102:0/69639525 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe928018730 0x7fe92801ac10 crc :-1 s=READY pgs=635 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:27.523 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:27.526 7fe947a12700 1 -- 172.21.2.102:0/69639525 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9400891c0 con 0x7fe940096780 2021-02-10T11:24:27.526 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:27.528 7fe9327fc700 1 -- 172.21.2.102:0/69639525 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fe93c006030 con 0x7fe940096780 2021-02-10T11:24:27.656 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:27.658 7fe947a12700 1 -- 172.21.2.102:0/69639525 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe9400ce690 con 0x7fe940096780 2021-02-10T11:24:27.656 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:27.659 7fe9327fc700 1 -- 172.21.2.102:0/69639525 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 39 v39) v1 ==== 94+0+2101 (secure 0 0 0) 0x7fe93c00bd50 con 0x7fe940096780 2021-02-10T11:24:27.657 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:24:27.657 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":39,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7920,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.102:6835/1829204031","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6834","nonce":1829204031},{"type":"v1","addr":"172.21.2.102:6835","nonce":1829204031}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":35}],"filesystems":[{"mdsmap":{"epoch":39,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:24:10.198618","modified":"2021-02-10 11:24:12.204882","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5714},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5714":{"gid":5714,"name":"b","rank":0,"incarnation":36,"state":"up:active","state_seq":3,"addr":"172.21.2.113:6833/4013137507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6832","nonce":4013137507},{"type":"v1","addr":"172.21.2.113:6833","nonce":4013137507}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":5}]} 2021-02-10T11:24:27.657 INFO:teuthology.orchestra.run.gibba002.stderr:dumped fsmap epoch 39 2021-02-10T11:24:27.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:27.661 7fe947a12700 1 -- 172.21.2.102:0/69639525 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe928018730 msgr2=0x7fe92801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:27.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:27.661 7fe947a12700 1 --2- 172.21.2.102:0/69639525 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe928018730 0x7fe92801ac10 crc :-1 s=READY pgs=635 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:27.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:27.661 7fe947a12700 1 -- 172.21.2.102:0/69639525 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe940096780 msgr2=0x7fe940091100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:27.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:27.661 7fe947a12700 1 --2- 172.21.2.102:0/69639525 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe940096780 0x7fe940091100 secure :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 rx=0x7fe93c009650 tx=0x7fe93c00b960).stop 2021-02-10T11:24:27.659 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:27.661 7fe947a12700 1 -- 172.21.2.102:0/69639525 shutdown_connections 2021-02-10T11:24:27.659 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:27.661 7fe947a12700 1 --2- 172.21.2.102:0/69639525 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe928018730 0x7fe92801ac10 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:27.659 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:27.661 7fe947a12700 1 --2- 172.21.2.102:0/69639525 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe940091720 0x7fe940097ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:27.659 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:27.661 7fe947a12700 1 --2- 172.21.2.102:0/69639525 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe940096780 0x7fe940091100 unknown :-1 s=CLOSED pgs=1364 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:27.659 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:27.661 7fe947a12700 1 --2- 172.21.2.102:0/69639525 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe940096380 0x7fe940090ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:27.659 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:27.661 7fe947a12700 1 -- 172.21.2.102:0/69639525 >> 172.21.2.102:0/69639525 conn(0x7fe9400b07f0 msgr2=0x7fe940107250 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:27.659 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:27.661 7fe947a12700 1 -- 172.21.2.102:0/69639525 shutdown_connections 2021-02-10T11:24:27.659 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:27.661 7fe947a12700 1 -- 172.21.2.102:0/69639525 wait complete. 2021-02-10T11:24:27.868 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 39, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:24:10.198618', 'modified': '2021-02-10 11:24:12.204882', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5714}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5714': {'gid': 5714, 'name': 'b', 'rank': 0, 'incarnation': 36, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.113:6833/4013137507', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.113:6832', 'nonce': 4013137507}, {'type': 'v1', 'addr': '172.21.2.113:6833', 'nonce': 4013137507}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [11], 'metadata_pool': 10, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:24:27.868 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:24:27.869 DEBUG:teuthology.orchestra.run.gibba013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:24:28.269 INFO:teuthology.orchestra.run.gibba013.stdout:{ 2021-02-10T11:24:28.269 INFO:teuthology.orchestra.run.gibba013.stdout: "cluster_fsid": "68db30ea-dbce-4aef-a6df-f01c6fea0513", 2021-02-10T11:24:28.269 INFO:teuthology.orchestra.run.gibba013.stdout: "whoami": 0, 2021-02-10T11:24:28.269 INFO:teuthology.orchestra.run.gibba013.stdout: "id": 5714, 2021-02-10T11:24:28.269 INFO:teuthology.orchestra.run.gibba013.stdout: "want_state": "up:active", 2021-02-10T11:24:28.270 INFO:teuthology.orchestra.run.gibba013.stdout: "state": "up:active", 2021-02-10T11:24:28.270 INFO:teuthology.orchestra.run.gibba013.stdout: "rank_uptime": 18.070835833, 2021-02-10T11:24:28.270 INFO:teuthology.orchestra.run.gibba013.stdout: "mdsmap_epoch": 39, 2021-02-10T11:24:28.270 INFO:teuthology.orchestra.run.gibba013.stdout: "osdmap_epoch": 57, 2021-02-10T11:24:28.270 INFO:teuthology.orchestra.run.gibba013.stdout: "osdmap_epoch_barrier": 57, 2021-02-10T11:24:28.270 INFO:teuthology.orchestra.run.gibba013.stdout: "uptime": 23.747203842000001 2021-02-10T11:24:28.270 INFO:teuthology.orchestra.run.gibba013.stdout:} 2021-02-10T11:24:28.273 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "68db30ea-dbce-4aef-a6df-f01c6fea0513", "whoami": 0, "id": 5714, "want_state": "up:active", "state": "up:active", "rank_uptime": 18.070835833, "mdsmap_epoch": 39, "osdmap_epoch": 57, "osdmap_epoch_barrier": 57, "uptime": 23.747203842000001 } 2021-02-10T11:24:28.274 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2021-02-10T11:24:28.274 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2021-02-10T11:24:28.274 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.1 at ubuntu@gibba002.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.1... 2021-02-10T11:24:28.274 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:24:28.274 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec mkdir -p /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:24:28.608 DEBUG:teuthology.orchestra.run.gibba002:> sudo modprobe fuse 2021-02-10T11:24:28.952 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:24:28.952 DEBUG:teuthology.orchestra.run.gibba002:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:24:29.295 INFO:teuthology.orchestra.run.gibba002.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:24:29.296 INFO:teuthology.orchestra.run.gibba002.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:24:29.296 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:24:29.297 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:24:29.297 DEBUG:teuthology.orchestra.run.gibba002:> ls /sys/fs/fuse/connections 2021-02-10T11:24:29.630 INFO:teuthology.orchestra.run.gibba002.stdout:43 2021-02-10T11:24:29.631 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [43] 2021-02-10T11:24:29.631 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.1 /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:24:29.634 DEBUG:teuthology.orchestra.run.gibba002:> sudo modprobe fuse 2021-02-10T11:24:29.977 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:24:29.977 DEBUG:teuthology.orchestra.run.gibba002:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:24:30.042 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba002.stderr:2021-02-10 11:24:30.045 7f6ddf879f80 -1 init, newargv = 0x556e399e7b20 newargc=9 2021-02-10T11:24:30.042 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba002.stderr:ceph-fuse[153877]: starting ceph client 2021-02-10T11:24:30.048 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba002.stderr:ceph-fuse[153877]: starting fuse 2021-02-10T11:24:30.319 INFO:teuthology.orchestra.run.gibba002.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:24:30.320 INFO:teuthology.orchestra.run.gibba002.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:24:30.320 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:24:30.321 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:24:30.321 DEBUG:teuthology.orchestra.run.gibba002:> ls /sys/fs/fuse/connections 2021-02-10T11:24:30.655 INFO:teuthology.orchestra.run.gibba002.stdout:43 2021-02-10T11:24:30.655 INFO:teuthology.orchestra.run.gibba002.stdout:44 2021-02-10T11:24:30.655 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43, 44] 2021-02-10T11:24:30.655 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:24:30.655 DEBUG:teuthology.orchestra.run.gibba002:> sudo python3 -c ' 2021-02-10T11:24:30.656 DEBUG:teuthology.orchestra.run.gibba002:> import glob 2021-02-10T11:24:30.656 DEBUG:teuthology.orchestra.run.gibba002:> import re 2021-02-10T11:24:30.656 DEBUG:teuthology.orchestra.run.gibba002:> import os 2021-02-10T11:24:30.656 DEBUG:teuthology.orchestra.run.gibba002:> import subprocess 2021-02-10T11:24:30.656 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:24:30.656 DEBUG:teuthology.orchestra.run.gibba002:> def find_socket(client_name): 2021-02-10T11:24:30.656 DEBUG:teuthology.orchestra.run.gibba002:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2021-02-10T11:24:30.656 DEBUG:teuthology.orchestra.run.gibba002:> files = glob.glob(asok_path) 2021-02-10T11:24:30.656 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:24:30.656 DEBUG:teuthology.orchestra.run.gibba002:> # Given a non-glob path, it better be there 2021-02-10T11:24:30.656 DEBUG:teuthology.orchestra.run.gibba002:> if "*" not in asok_path: 2021-02-10T11:24:30.656 DEBUG:teuthology.orchestra.run.gibba002:> assert(len(files) == 1) 2021-02-10T11:24:30.656 DEBUG:teuthology.orchestra.run.gibba002:> return files[0] 2021-02-10T11:24:30.656 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:24:30.657 DEBUG:teuthology.orchestra.run.gibba002:> for f in files: 2021-02-10T11:24:30.657 DEBUG:teuthology.orchestra.run.gibba002:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2021-02-10T11:24:30.657 DEBUG:teuthology.orchestra.run.gibba002:> if os.path.exists("/proc/{0}".format(pid)): 2021-02-10T11:24:30.657 DEBUG:teuthology.orchestra.run.gibba002:> return f 2021-02-10T11:24:30.657 DEBUG:teuthology.orchestra.run.gibba002:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2021-02-10T11:24:30.657 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:24:30.657 DEBUG:teuthology.orchestra.run.gibba002:> print(find_socket("client.1")) 2021-02-10T11:24:30.657 DEBUG:teuthology.orchestra.run.gibba002:> ' 2021-02-10T11:24:31.018 INFO:teuthology.orchestra.run.gibba002.stdout:/var/run/ceph/ceph-client.1.153877.asok 2021-02-10T11:24:31.021 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.153877.asok 2021-02-10T11:24:31.021 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:24:31.021 DEBUG:teuthology.orchestra.run.gibba002:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.153877.asok status 2021-02-10T11:24:31.410 INFO:teuthology.orchestra.run.gibba002.stdout:{ 2021-02-10T11:24:31.411 INFO:teuthology.orchestra.run.gibba002.stdout: "metadata": { 2021-02-10T11:24:31.411 INFO:teuthology.orchestra.run.gibba002.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:24:31.411 INFO:teuthology.orchestra.run.gibba002.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:24:31.411 INFO:teuthology.orchestra.run.gibba002.stdout: "entity_id": "1", 2021-02-10T11:24:31.411 INFO:teuthology.orchestra.run.gibba002.stdout: "hostname": "gibba002", 2021-02-10T11:24:31.411 INFO:teuthology.orchestra.run.gibba002.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2021-02-10T11:24:31.411 INFO:teuthology.orchestra.run.gibba002.stdout: "pid": "153877", 2021-02-10T11:24:31.411 INFO:teuthology.orchestra.run.gibba002.stdout: "root": "/" 2021-02-10T11:24:31.411 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:24:31.411 INFO:teuthology.orchestra.run.gibba002.stdout: "dentry_count": 0, 2021-02-10T11:24:31.411 INFO:teuthology.orchestra.run.gibba002.stdout: "dentry_pinned_count": 0, 2021-02-10T11:24:31.412 INFO:teuthology.orchestra.run.gibba002.stdout: "id": 8133, 2021-02-10T11:24:31.412 INFO:teuthology.orchestra.run.gibba002.stdout: "inst": { 2021-02-10T11:24:31.412 INFO:teuthology.orchestra.run.gibba002.stdout: "name": { 2021-02-10T11:24:31.412 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "client", 2021-02-10T11:24:31.412 INFO:teuthology.orchestra.run.gibba002.stdout: "num": 8133 2021-02-10T11:24:31.412 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:24:31.412 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": { 2021-02-10T11:24:31.412 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:24:31.412 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:0", 2021-02-10T11:24:31.412 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 4158403420 2021-02-10T11:24:31.412 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:24:31.412 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:24:31.412 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": { 2021-02-10T11:24:31.413 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:24:31.413 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:0", 2021-02-10T11:24:31.413 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 4158403420 2021-02-10T11:24:31.413 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:24:31.413 INFO:teuthology.orchestra.run.gibba002.stdout: "inst_str": "client.8133 172.21.2.102:0/4158403420", 2021-02-10T11:24:31.413 INFO:teuthology.orchestra.run.gibba002.stdout: "addr_str": "172.21.2.102:0/4158403420", 2021-02-10T11:24:31.413 INFO:teuthology.orchestra.run.gibba002.stdout: "inode_count": 1, 2021-02-10T11:24:31.413 INFO:teuthology.orchestra.run.gibba002.stdout: "mds_epoch": 39, 2021-02-10T11:24:31.413 INFO:teuthology.orchestra.run.gibba002.stdout: "osd_epoch": 57, 2021-02-10T11:24:31.413 INFO:teuthology.orchestra.run.gibba002.stdout: "osd_epoch_barrier": 0, 2021-02-10T11:24:31.413 INFO:teuthology.orchestra.run.gibba002.stdout: "blacklisted": false 2021-02-10T11:24:31.413 INFO:teuthology.orchestra.run.gibba002.stdout:} 2021-02-10T11:24:31.415 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:24:31.415 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:24:31.415 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:24:31.749 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:24:31.750 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2021-02-10T11:24:31.750 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:24:31.750 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:24:32.095 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2021-02-10T11:24:32.524 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:32.509 7fb358de5700 1 Processor -- start 2021-02-10T11:24:32.524 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:32.509 7fb358de5700 1 -- start start 2021-02-10T11:24:32.524 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:32.510 7fb358de5700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fb3540d1870 0x7fb3541b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:32.524 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:32.510 7fb358de5700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fb3540cff70 0x7fb354109130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:32.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:32.510 7fb358de5700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3540fe1c0 con 0x7fb3541b0ae0 2021-02-10T11:24:32.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:32.510 7fb358de5700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb354109ad0 con 0x7fb3540cff70 2021-02-10T11:24:32.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:32.510 7fb358de5700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb354109750 con 0x7fb3540d1870 2021-02-10T11:24:32.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:32.510 7fb352312700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fb3541b0ae0 0x7fb354108940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:49494/0 (socket says 172.21.2.102:49494) 2021-02-10T11:24:32.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:32.510 7fb352312700 1 -- 172.21.2.102:0/232435291 learned_addr learned my addr 172.21.2.102:0/232435291 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:24:32.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:32.510 7fb353314700 1 --2- 172.21.2.102:0/232435291 >> v2:172.21.2.102:3300/0 conn(0x7fb3540cff70 0x7fb354109130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:32.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:32.510 7fb353314700 1 -- 172.21.2.102:0/232435291 >> v2:172.21.2.113:3300/0 conn(0x7fb3540d1870 msgr2=0x7fb3541b04c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:24:32.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:32.510 7fb353314700 1 --2- 172.21.2.102:0/232435291 >> v2:172.21.2.113:3300/0 conn(0x7fb3540d1870 0x7fb3541b04c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:32.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:32.510 7fb353314700 1 -- 172.21.2.102:0/232435291 >> v1:172.21.2.102:6789/0 conn(0x7fb3541b0ae0 legacy=0x7fb354108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:32.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:32.510 7fb351b11700 1 -- 172.21.2.102:0/232435291 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3271856154 0 0) 0x7fb33c0010b0 con 0x7fb3541b0ae0 2021-02-10T11:24:32.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:32.510 7fb353314700 1 -- 172.21.2.102:0/232435291 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb354088020 con 0x7fb3540cff70 2021-02-10T11:24:32.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:32.511 7fb353314700 1 --2- 172.21.2.102:0/232435291 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb3540cff70 0x7fb354109130 secure :-1 s=READY pgs=1369 cs=0 l=1 rev1=1 rx=0x7fb34c006e30 tx=0x7fb34c009de0).ready entity=mon.0 client_cookie=13b17ccc042accd2 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:32.526 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:32.511 7fb351b11700 1 -- 172.21.2.102:0/232435291 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb34c00b300 con 0x7fb3540cff70 2021-02-10T11:24:32.526 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:32.511 7fb351b11700 1 -- 172.21.2.102:0/232435291 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb34c00b580 con 0x7fb3540cff70 2021-02-10T11:24:32.526 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:32.511 7fb351b11700 1 -- 172.21.2.102:0/232435291 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb34c00baa0 con 0x7fb3540cff70 2021-02-10T11:24:32.526 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:32.511 7fb358de5700 1 -- 172.21.2.102:0/232435291 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb3540cff70 msgr2=0x7fb354109130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:32.526 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:32.511 7fb358de5700 1 --2- 172.21.2.102:0/232435291 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb3540cff70 0x7fb354109130 secure :-1 s=READY pgs=1369 cs=0 l=1 rev1=1 rx=0x7fb34c006e30 tx=0x7fb34c009de0).stop 2021-02-10T11:24:32.526 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:32.511 7fb358de5700 1 -- 172.21.2.102:0/232435291 shutdown_connections 2021-02-10T11:24:32.526 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:32.511 7fb358de5700 1 --2- 172.21.2.102:0/232435291 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb3540cff70 0x7fb354109130 unknown :-1 s=CLOSED pgs=1369 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:32.526 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:32.511 7fb358de5700 1 --2- 172.21.2.102:0/232435291 >> v2:172.21.2.113:3300/0 conn(0x7fb3540d1870 0x7fb3541b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:32.526 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:32.511 7fb358de5700 1 -- 172.21.2.102:0/232435291 >> 172.21.2.102:0/232435291 conn(0x7fb3540ce420 msgr2=0x7fb3540d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:32.526 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:32.512 7fb358de5700 1 -- 172.21.2.102:0/232435291 shutdown_connections 2021-02-10T11:24:32.526 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:32.512 7fb358de5700 1 -- 172.21.2.102:0/232435291 wait complete. 2021-02-10T11:24:32.528 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:32.526 7fb358de5700 1 Processor -- start 2021-02-10T11:24:32.529 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:32.526 7fb358de5700 1 -- start start 2021-02-10T11:24:32.529 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:32.526 7fb358de5700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb3541a2930 0x7fb3541a1c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:32.529 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:32.526 7fb358de5700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb3541a5820 0x7fb35419a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:32.529 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:32.526 7fb358de5700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb35409d670 0x7fb35409da70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:32.529 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:32.526 7fb358de5700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb35419a970 con 0x7fb3541a5820 2021-02-10T11:24:32.529 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:32.526 7fb358de5700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3541a23e0 con 0x7fb35409d670 2021-02-10T11:24:32.529 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:32.526 7fb358de5700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3541a5380 con 0x7fb3541a2930 2021-02-10T11:24:32.529 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:32.526 7fb352312700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb3541a5820 0x7fb35419a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:32.529 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:32.527 7fb352312700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb3541a5820 0x7fb35419a2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:57278/0 (socket says 172.21.2.102:57278) 2021-02-10T11:24:32.529 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:32.527 7fb352312700 1 -- 172.21.2.102:0/884590121 learned_addr learned my addr 172.21.2.102:0/884590121 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:24:32.530 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:32.527 7fb352312700 1 -- 172.21.2.102:0/884590121 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb3541a2930 msgr2=0x7fb3541a1c50 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:24:32.530 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:32.527 7fb352312700 1 --2- 172.21.2.102:0/884590121 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb3541a2930 0x7fb3541a1c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:32.530 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:32.527 7fb352312700 1 -- 172.21.2.102:0/884590121 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb35409d670 msgr2=0x7fb35409da70 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:24:32.530 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:32.527 7fb352312700 1 --2- 172.21.2.102:0/884590121 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb35409d670 0x7fb35409da70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:32.530 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:32.527 7fb352312700 1 -- 172.21.2.102:0/884590121 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3541a5c20 con 0x7fb3541a5820 2021-02-10T11:24:32.530 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:32.527 7fb352312700 1 --2- 172.21.2.102:0/884590121 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb3541a5820 0x7fb35419a2d0 secure :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 rx=0x7fb33c001c60 tx=0x7fb33c00a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:32.530 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:32.527 7fb34b7fe700 1 -- 172.21.2.102:0/884590121 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb33c00acc0 con 0x7fb3541a5820 2021-02-10T11:24:32.530 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:32.527 7fb34b7fe700 1 -- 172.21.2.102:0/884590121 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb33c004140 con 0x7fb3541a5820 2021-02-10T11:24:32.530 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:32.527 7fb34b7fe700 1 -- 172.21.2.102:0/884590121 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb33c0045a0 con 0x7fb3541a5820 2021-02-10T11:24:32.530 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:32.527 7fb358de5700 1 -- 172.21.2.102:0/884590121 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3541a5c20 con 0x7fb3541a5820 2021-02-10T11:24:32.530 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:32.527 7fb358de5700 1 -- 172.21.2.102:0/884590121 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb35409e380 con 0x7fb3541a5820 2021-02-10T11:24:32.531 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:32.528 7fb34b7fe700 1 -- 172.21.2.102:0/884590121 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fb33c004320 con 0x7fb3541a5820 2021-02-10T11:24:32.531 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:32.528 7fb34b7fe700 1 --2- 172.21.2.102:0/884590121 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb3380186c0 0x7fb33801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:32.531 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:32.528 7fb34b7fe700 1 -- 172.21.2.102:0/884590121 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fb33c0323f0 con 0x7fb3541a5820 2021-02-10T11:24:32.531 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:32.528 7fb352b13700 1 --2- 172.21.2.102:0/884590121 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb3380186c0 0x7fb33801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:32.531 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:32.528 7fb352b13700 1 --2- 172.21.2.102:0/884590121 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb3380186c0 0x7fb33801abc0 crc :-1 s=READY pgs=636 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:32.558 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:32.560 7fb358de5700 1 -- 172.21.2.102:0/884590121 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3541a23e0 con 0x7fb3541a5820 2021-02-10T11:24:32.561 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:32.563 7fb34b7fe700 1 -- 172.21.2.102:0/884590121 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fb33c030d20 con 0x7fb3541a5820 2021-02-10T11:24:32.695 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:32.696 7fb358de5700 1 -- 172.21.2.102:0/884590121 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fb3541a23e0 con 0x7fb3541a5820 2021-02-10T11:24:33.005 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:33.006 7fb34b7fe700 1 -- 172.21.2.102:0/884590121 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v40) v1 ==== 136+0+0 (secure 0 0 0) 0x7fb33c004320 con 0x7fb3541a5820 2021-02-10T11:24:33.013 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:33.014 7fb358de5700 1 -- 172.21.2.102:0/884590121 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb3380186c0 msgr2=0x7fb33801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:33.013 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:33.014 7fb358de5700 1 --2- 172.21.2.102:0/884590121 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb3380186c0 0x7fb33801abc0 crc :-1 s=READY pgs=636 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:33.013 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:33.014 7fb358de5700 1 -- 172.21.2.102:0/884590121 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb3541a5820 msgr2=0x7fb35419a2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:33.013 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:33.014 7fb358de5700 1 --2- 172.21.2.102:0/884590121 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb3541a5820 0x7fb35419a2d0 secure :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 rx=0x7fb33c001c60 tx=0x7fb33c00a8d0).stop 2021-02-10T11:24:33.014 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:33.015 7fb358de5700 1 -- 172.21.2.102:0/884590121 shutdown_connections 2021-02-10T11:24:33.014 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:33.015 7fb358de5700 1 --2- 172.21.2.102:0/884590121 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb3380186c0 0x7fb33801abc0 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:33.014 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:33.015 7fb358de5700 1 --2- 172.21.2.102:0/884590121 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb35409d670 0x7fb35409da70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:33.014 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:33.015 7fb358de5700 1 --2- 172.21.2.102:0/884590121 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb3541a5820 0x7fb35419a2d0 unknown :-1 s=CLOSED pgs=1370 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:33.014 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:33.015 7fb358de5700 1 --2- 172.21.2.102:0/884590121 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb3541a2930 0x7fb3541a1c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:33.014 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:33.015 7fb358de5700 1 -- 172.21.2.102:0/884590121 >> 172.21.2.102:0/884590121 conn(0x7fb354094ea0 msgr2=0x7fb35419acf0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:33.014 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:33.015 7fb358de5700 1 -- 172.21.2.102:0/884590121 shutdown_connections 2021-02-10T11:24:33.014 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:33.015 7fb358de5700 1 -- 172.21.2.102:0/884590121 wait complete. 2021-02-10T11:24:33.022 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs volume ls 2021-02-10T11:24:33.449 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:33.433 7f59f19c3700 1 Processor -- start 2021-02-10T11:24:33.449 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:33.433 7f59f19c3700 1 -- start start 2021-02-10T11:24:33.449 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:33.433 7f59f19c3700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f59ec0e5470 0x7f59ec07e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:33.449 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:33.433 7f59f19c3700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59ec07ed80 con 0x7f59ec07f300 2021-02-10T11:24:33.449 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:33.433 7f59f19c3700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59ec1b4010 con 0x7f59ec0e5070 2021-02-10T11:24:33.449 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:33.433 7f59f19c3700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59ec0d1790 con 0x7f59ec0e5470 2021-02-10T11:24:33.450 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:33.433 7f59ebd75700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f59ec07f300 0x7f59ec07f700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:49506/0 (socket says 172.21.2.102:49506) 2021-02-10T11:24:33.450 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:33.433 7f59ebd75700 1 -- 172.21.2.102:0/2749483238 learned_addr learned my addr 172.21.2.102:0/2749483238 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:24:33.450 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:33.434 7f59ea572700 1 -- 172.21.2.102:0/2749483238 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1681985223 0 0) 0x7f59e4001500 con 0x7f59ec07f300 2021-02-10T11:24:33.450 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:33.434 7f59ea572700 1 -- 172.21.2.102:0/2749483238 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59d8002fe0 con 0x7f59ec07f300 2021-02-10T11:24:33.450 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:33.434 7f59ea572700 1 -- 172.21.2.102:0/2749483238 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3025894685 0 0) 0x7f59e4001460 con 0x7f59ec07f300 2021-02-10T11:24:33.450 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:33.434 7f59ead73700 1 --2- 172.21.2.102:0/2749483238 >> v2:172.21.2.113:3300/0 conn(0x7f59ec0e5470 0x7f59ec07e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:33.450 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:33.434 7f59ea572700 1 -- 172.21.2.102:0/2749483238 >> v2:172.21.2.113:3300/0 conn(0x7f59ec0e5470 msgr2=0x7f59ec07e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:24:33.450 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:33.434 7f59ea572700 1 --2- 172.21.2.102:0/2749483238 >> v2:172.21.2.113:3300/0 conn(0x7f59ec0e5470 0x7f59ec07e6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:33.450 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:33.434 7f59ea572700 1 -- 172.21.2.102:0/2749483238 >> v1:172.21.2.113:6789/0 conn(0x7f59ec0e5070 legacy=0x7f59ec1b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:33.450 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:33.434 7f59ea572700 1 -- 172.21.2.102:0/2749483238 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59ec0d19b0 con 0x7f59ec07f300 2021-02-10T11:24:33.450 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:33.434 7f59ea572700 1 -- 172.21.2.102:0/2749483238 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7f59e4001980 con 0x7f59ec07f300 2021-02-10T11:24:33.450 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:33.434 7f59ea572700 1 -- 172.21.2.102:0/2749483238 >> v1:172.21.2.102:6789/0 conn(0x7f59ec07f300 legacy=0x7f59ec07f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:33.451 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:33.434 7f59ea572700 1 --2- 172.21.2.102:0/2749483238 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f59d8003b90 0x7f59d8006040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:33.451 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:33.434 7f59ea572700 1 --2- 172.21.2.102:0/2749483238 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f59d8006890 0x7f59d8008d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:33.451 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:33.434 7f59ea572700 1 --2- 172.21.2.102:0/2749483238 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f59d8009650 0x7f59d800bb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:33.451 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:33.434 7f59ea572700 1 -- 172.21.2.102:0/2749483238 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59d800c3a0 con 0x7f59d8003b90 2021-02-10T11:24:33.451 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:33.434 7f59ea572700 1 -- 172.21.2.102:0/2749483238 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59d800c5c0 con 0x7f59d8006890 2021-02-10T11:24:33.451 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:33.434 7f59ea572700 1 -- 172.21.2.102:0/2749483238 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59d800c7e0 con 0x7f59d8009650 2021-02-10T11:24:33.451 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:33.434 7f59ead73700 1 --2- 172.21.2.102:0/2749483238 >> v2:172.21.2.113:3300/0 conn(0x7f59ec0e5470 0x7f59ec07e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:24:33.451 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:33.434 7f59eb574700 1 --2- 172.21.2.102:0/2749483238 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f59d8003b90 0x7f59d8006040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:33.451 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:33.434 7f59ead73700 1 --2- 172.21.2.102:0/2749483238 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f59d8006890 0x7f59d8008d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:33.451 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:33.435 7f59ebd75700 1 --2- 172.21.2.102:0/2749483238 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f59d8009650 0x7f59d800bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:33.451 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:33.435 7f59eb574700 1 -- 172.21.2.102:0/2749483238 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f59d8009650 msgr2=0x7f59d800bb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:24:33.451 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:33.435 7f59eb574700 1 --2- 172.21.2.102:0/2749483238 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f59d8009650 0x7f59d800bb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:33.451 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:33.435 7f59eb574700 1 -- 172.21.2.102:0/2749483238 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f59d8006890 msgr2=0x7f59d8008d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:24:33.452 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:33.435 7f59eb574700 1 --2- 172.21.2.102:0/2749483238 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f59d8006890 0x7f59d8008d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:33.452 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:33.435 7f59eb574700 1 -- 172.21.2.102:0/2749483238 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59d800ca80 con 0x7f59d8003b90 2021-02-10T11:24:33.452 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:33.435 7f59ebd75700 1 -- 172.21.2.102:0/2749483238 reap_dead start 2021-02-10T11:24:33.452 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:33.435 7f59eb574700 1 --2- 172.21.2.102:0/2749483238 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f59d8003b90 0x7f59d8006040 secure :-1 s=READY pgs=1372 cs=0 l=1 rev1=1 rx=0x7f59dc001040 tx=0x7f59dc04db00).ready entity=mon.0 client_cookie=6d25499b21735001 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:33.452 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:33.435 7f59ea572700 1 -- 172.21.2.102:0/2749483238 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f59dc006700 con 0x7f59d8003b90 2021-02-10T11:24:33.452 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:33.435 7f59ea572700 1 -- 172.21.2.102:0/2749483238 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59dc0500c0 con 0x7f59d8003b90 2021-02-10T11:24:33.452 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:33.435 7f59ea572700 1 -- 172.21.2.102:0/2749483238 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f59dc0505e0 con 0x7f59d8003b90 2021-02-10T11:24:33.452 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:33.435 7f59f19c3700 1 -- 172.21.2.102:0/2749483238 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f59d8003b90 msgr2=0x7f59d8006040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:33.452 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:33.435 7f59f19c3700 1 --2- 172.21.2.102:0/2749483238 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f59d8003b90 0x7f59d8006040 secure :-1 s=READY pgs=1372 cs=0 l=1 rev1=1 rx=0x7f59dc001040 tx=0x7f59dc04db00).stop 2021-02-10T11:24:33.452 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:33.436 7f59f19c3700 1 -- 172.21.2.102:0/2749483238 shutdown_connections 2021-02-10T11:24:33.452 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:33.436 7f59f19c3700 1 --2- 172.21.2.102:0/2749483238 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f59d8003b90 0x7f59d8006040 unknown :-1 s=CLOSED pgs=1372 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:33.452 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:33.436 7f59f19c3700 1 -- 172.21.2.102:0/2749483238 >> 172.21.2.102:0/2749483238 conn(0x7f59ec0ce690 msgr2=0x7f59ec0e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:33.452 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:33.436 7f59f19c3700 1 -- 172.21.2.102:0/2749483238 shutdown_connections 2021-02-10T11:24:33.452 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:33.436 7f59f19c3700 1 -- 172.21.2.102:0/2749483238 wait complete. 2021-02-10T11:24:33.453 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:33.451 7f59f19c3700 1 Processor -- start 2021-02-10T11:24:33.454 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:33.451 7f59f19c3700 1 -- start start 2021-02-10T11:24:33.454 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:33.451 7f59f19c3700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f59ec0b4ad0 0x7f59ec0a78d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:33.454 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:33.451 7f59f19c3700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f59ec0a7ef0 0x7f59ec1a6d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:33.454 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:33.451 7f59f19c3700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f59ec0a82f0 0x7f59ec1a73b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:33.454 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:33.451 7f59f19c3700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59ec0aa3c0 con 0x7f59ec0b4ad0 2021-02-10T11:24:33.454 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:33.451 7f59f19c3700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59ec0a88c0 con 0x7f59ec0a82f0 2021-02-10T11:24:33.454 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:33.451 7f59f19c3700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59ec0b4ed0 con 0x7f59ec0a7ef0 2021-02-10T11:24:33.454 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:33.451 7f59eb574700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f59ec0b4ad0 0x7f59ec0a78d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:33.454 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:33.451 7f59eb574700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f59ec0b4ad0 0x7f59ec0a78d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:57296/0 (socket says 172.21.2.102:57296) 2021-02-10T11:24:33.454 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:33.451 7f59eb574700 1 -- 172.21.2.102:0/3634945489 learned_addr learned my addr 172.21.2.102:0/3634945489 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:24:33.455 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:33.452 7f59eb574700 1 -- 172.21.2.102:0/3634945489 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f59ec0a7ef0 msgr2=0x7f59ec1a6d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:33.455 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:33.452 7f59eb574700 1 --2- 172.21.2.102:0/3634945489 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f59ec0a7ef0 0x7f59ec1a6d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:33.455 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:33.452 7f59eb574700 1 -- 172.21.2.102:0/3634945489 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f59ec0a82f0 msgr2=0x7f59ec1a73b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:33.455 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:33.452 7f59eb574700 1 --2- 172.21.2.102:0/3634945489 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f59ec0a82f0 0x7f59ec1a73b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:33.455 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:33.452 7f59eb574700 1 -- 172.21.2.102:0/3634945489 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59ec1a79d0 con 0x7f59ec0b4ad0 2021-02-10T11:24:33.455 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:33.452 7f59eb574700 1 --2- 172.21.2.102:0/3634945489 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f59ec0b4ad0 0x7f59ec0a78d0 secure :-1 s=READY pgs=1373 cs=0 l=1 rev1=1 rx=0x7f59dc04e6b0 tx=0x7f59dc050970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:33.455 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:33.452 7f59e3fff700 1 -- 172.21.2.102:0/3634945489 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f59dc050d60 con 0x7f59ec0b4ad0 2021-02-10T11:24:33.455 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:33.452 7f59e3fff700 1 -- 172.21.2.102:0/3634945489 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59dc006030 con 0x7f59ec0b4ad0 2021-02-10T11:24:33.455 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:33.452 7f59e3fff700 1 -- 172.21.2.102:0/3634945489 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f59dc005540 con 0x7f59ec0b4ad0 2021-02-10T11:24:33.455 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:33.452 7f59f19c3700 1 -- 172.21.2.102:0/3634945489 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59ec1a79d0 con 0x7f59ec0b4ad0 2021-02-10T11:24:33.455 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:33.452 7f59f19c3700 1 -- 172.21.2.102:0/3634945489 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59ec0a9130 con 0x7f59ec0b4ad0 2021-02-10T11:24:33.455 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:33.453 7f59e3fff700 1 -- 172.21.2.102:0/3634945489 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f59dc006210 con 0x7f59ec0b4ad0 2021-02-10T11:24:33.455 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:33.453 7f59e3fff700 1 --2- 172.21.2.102:0/3634945489 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f59d00186c0 0x7f59d001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:33.456 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:33.453 7f59e3fff700 1 -- 172.21.2.102:0/3634945489 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f59dc0765e0 con 0x7f59ec0b4ad0 2021-02-10T11:24:33.456 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:33.453 7f59ead73700 1 --2- 172.21.2.102:0/3634945489 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f59d00186c0 0x7f59d001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:33.456 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:33.453 7f59ead73700 1 --2- 172.21.2.102:0/3634945489 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f59d00186c0 0x7f59d001abc0 crc :-1 s=READY pgs=637 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:33.482 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:33.483 7f59f19c3700 1 -- 172.21.2.102:0/3634945489 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59ec084a90 con 0x7f59ec0b4ad0 2021-02-10T11:24:33.484 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:33.486 7f59e3fff700 1 -- 172.21.2.102:0/3634945489 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f59dc074d90 con 0x7f59ec0b4ad0 2021-02-10T11:24:33.617 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:33.619 7f59f19c3700 1 -- 172.21.2.102:0/3634945489 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs volume ls", "target": ["mgr", ""]}) v1 -- 0x7f59ec0b4ed0 con 0x7f59d00186c0 2021-02-10T11:24:33.618 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:33.620 7f59e3fff700 1 -- 172.21.2.102:0/3634945489 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+41 (crc 0 0 0) 0x7f59d4007810 con 0x7f59d00186c0 2021-02-10T11:24:33.618 INFO:teuthology.orchestra.run.gibba002.stdout:[ 2021-02-10T11:24:33.619 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:24:33.619 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "cephfs" 2021-02-10T11:24:33.619 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:24:33.619 INFO:teuthology.orchestra.run.gibba002.stdout:] 2021-02-10T11:24:33.619 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:33.621 7f59f19c3700 1 -- 172.21.2.102:0/3634945489 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f59d00186c0 msgr2=0x7f59d001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:33.620 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:33.621 7f59f19c3700 1 --2- 172.21.2.102:0/3634945489 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f59d00186c0 0x7f59d001abc0 crc :-1 s=READY pgs=637 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:33.620 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:33.621 7f59f19c3700 1 -- 172.21.2.102:0/3634945489 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f59ec0b4ad0 msgr2=0x7f59ec0a78d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:33.620 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:33.621 7f59f19c3700 1 --2- 172.21.2.102:0/3634945489 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f59ec0b4ad0 0x7f59ec0a78d0 secure :-1 s=READY pgs=1373 cs=0 l=1 rev1=1 rx=0x7f59dc04e6b0 tx=0x7f59dc050970).stop 2021-02-10T11:24:33.620 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:33.621 7f59f19c3700 1 -- 172.21.2.102:0/3634945489 shutdown_connections 2021-02-10T11:24:33.620 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:33.621 7f59f19c3700 1 --2- 172.21.2.102:0/3634945489 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f59d00186c0 0x7f59d001abc0 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:33.620 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:33.621 7f59f19c3700 1 --2- 172.21.2.102:0/3634945489 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f59ec0a82f0 0x7f59ec1a73b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:33.621 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:33.621 7f59f19c3700 1 --2- 172.21.2.102:0/3634945489 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f59ec0a7ef0 0x7f59ec1a6d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:33.621 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:33.621 7f59f19c3700 1 --2- 172.21.2.102:0/3634945489 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f59ec0b4ad0 0x7f59ec0a78d0 unknown :-1 s=CLOSED pgs=1373 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:33.621 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:33.621 7f59f19c3700 1 -- 172.21.2.102:0/3634945489 >> 172.21.2.102:0/3634945489 conn(0x7f59ec0b5560 msgr2=0x7f59ec0a6200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:33.621 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:33.622 7f59f19c3700 1 -- 172.21.2.102:0/3634945489 shutdown_connections 2021-02-10T11:24:33.621 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:33.622 7f59f19c3700 1 -- 172.21.2.102:0/3634945489 wait complete. 2021-02-10T11:24:33.629 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config set mon mon_allow_pool_delete True 2021-02-10T11:24:34.022 DEBUG:teuthology.orchestra.run.gibba002:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:24:34.025 DEBUG:teuthology.orchestra.run.gibba013:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:24:34.060 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:34.045 7f37d0a72700 1 Processor -- start 2021-02-10T11:24:34.060 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:34.045 7f37d0a72700 1 -- start start 2021-02-10T11:24:34.061 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:34.046 7f37d0a72700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f37cc1b0870 0x7f37cc0f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:34.061 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:34.046 7f37d0a72700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37cc1b0c70 con 0x7f37cc1b0470 2021-02-10T11:24:34.061 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:34.046 7f37d0a72700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37cc0f0b50 con 0x7f37cc0efe00 2021-02-10T11:24:34.061 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:34.046 7f37d0a72700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37cc0e5ea0 con 0x7f37cc1b0870 2021-02-10T11:24:34.061 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:34.046 7f37ca05d700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f37cc1b0870 0x7f37cc0f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:34.061 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:34.046 7f37ca85e700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f37cc1b0470 0x7f37cc0ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:49526/0 (socket says 172.21.2.102:49526) 2021-02-10T11:24:34.061 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:34.046 7f37ca85e700 1 -- 172.21.2.102:0/140790422 learned_addr learned my addr 172.21.2.102:0/140790422 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:24:34.061 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:34.046 7f37ca05d700 1 -- 172.21.2.102:0/140790422 >> v1:172.21.2.113:6789/0 conn(0x7f37cc0efe00 legacy=0x7f37cc0f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:34.061 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:34.046 7f37ca05d700 1 -- 172.21.2.102:0/140790422 >> v1:172.21.2.102:6789/0 conn(0x7f37cc1b0470 legacy=0x7f37cc0ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:34.061 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:34.046 7f37ca05d700 1 -- 172.21.2.102:0/140790422 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37cc0e52e0 con 0x7f37cc1b0870 2021-02-10T11:24:34.061 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:34.046 7f37ca05d700 1 --2- 172.21.2.102:0/140790422 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f37cc1b0870 0x7f37cc0f20e0 secure :-1 s=READY pgs=1375 cs=0 l=1 rev1=1 rx=0x7f37bc006650 tx=0x7f37bc009a80).ready entity=mon.0 client_cookie=250de110502ee7ee server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:34.061 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:34.046 7f37c985c700 1 -- 172.21.2.102:0/140790422 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f37bc0009b0 con 0x7f37cc1b0870 2021-02-10T11:24:34.062 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:34.047 7f37c985c700 1 -- 172.21.2.102:0/140790422 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37bc006740 con 0x7f37cc1b0870 2021-02-10T11:24:34.062 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:34.047 7f37c985c700 1 -- 172.21.2.102:0/140790422 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f37bc00c0c0 con 0x7f37cc1b0870 2021-02-10T11:24:34.062 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:34.047 7f37d0a72700 1 -- 172.21.2.102:0/140790422 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f37cc1b0870 msgr2=0x7f37cc0f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:34.062 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:34.047 7f37d0a72700 1 --2- 172.21.2.102:0/140790422 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f37cc1b0870 0x7f37cc0f20e0 secure :-1 s=READY pgs=1375 cs=0 l=1 rev1=1 rx=0x7f37bc006650 tx=0x7f37bc009a80).stop 2021-02-10T11:24:34.062 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:34.047 7f37d0a72700 1 -- 172.21.2.102:0/140790422 shutdown_connections 2021-02-10T11:24:34.062 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:34.047 7f37d0a72700 1 --2- 172.21.2.102:0/140790422 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f37cc1b0870 0x7f37cc0f20e0 unknown :-1 s=CLOSED pgs=1375 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:34.062 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:34.047 7f37d0a72700 1 -- 172.21.2.102:0/140790422 >> 172.21.2.102:0/140790422 conn(0x7f37cc0ce3e0 msgr2=0x7f37cc0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:34.062 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:34.047 7f37d0a72700 1 -- 172.21.2.102:0/140790422 shutdown_connections 2021-02-10T11:24:34.062 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:34.047 7f37d0a72700 1 -- 172.21.2.102:0/140790422 wait complete. 2021-02-10T11:24:34.064 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:34.062 7f37d0a72700 1 Processor -- start 2021-02-10T11:24:34.064 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:34.062 7f37d0a72700 1 -- start start 2021-02-10T11:24:34.064 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:34.063 7f37d0a72700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f37cc0a3b50 0x7f37cc1b3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:34.064 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:34.063 7f37d0a72700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f37cc1c25b0 0x7f37cc1b3d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:34.064 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:34.063 7f37d0a72700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f37cc1b4350 0x7f37cc1b18a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:34.064 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:34.063 7f37d0a72700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37cc1b3190 con 0x7f37cc1b4350 2021-02-10T11:24:34.064 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:34.063 7f37d0a72700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37cc087750 con 0x7f37cc1c25b0 2021-02-10T11:24:34.064 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:34.063 7f37d0a72700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37cc0a6bf0 con 0x7f37cc0a3b50 2021-02-10T11:24:34.064 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:34.063 7f37cb05f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f37cc1b4350 0x7f37cc1b18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:34.065 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:34.063 7f37cb05f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f37cc1b4350 0x7f37cc1b18a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:57308/0 (socket says 172.21.2.102:57308) 2021-02-10T11:24:34.065 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:34.063 7f37cb05f700 1 -- 172.21.2.102:0/3004883859 learned_addr learned my addr 172.21.2.102:0/3004883859 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:24:34.065 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:34.063 7f37cb05f700 1 -- 172.21.2.102:0/3004883859 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f37cc0a3b50 msgr2=0x7f37cc1b3710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:34.065 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:34.063 7f37cb05f700 1 --2- 172.21.2.102:0/3004883859 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f37cc0a3b50 0x7f37cc1b3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:34.065 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:34.063 7f37cb05f700 1 -- 172.21.2.102:0/3004883859 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f37cc1c25b0 msgr2=0x7f37cc1b3d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:34.065 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:34.063 7f37cb05f700 1 --2- 172.21.2.102:0/3004883859 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f37cc1c25b0 0x7f37cc1b3d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:34.065 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:34.063 7f37cb05f700 1 -- 172.21.2.102:0/3004883859 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37cc1b1ec0 con 0x7f37cc1b4350 2021-02-10T11:24:34.065 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:34.064 7f37cb05f700 1 --2- 172.21.2.102:0/3004883859 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f37cc1b4350 0x7f37cc1b18a0 secure :-1 s=READY pgs=1376 cs=0 l=1 rev1=1 rx=0x7f37c4009940 tx=0x7f37c400bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:34.065 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:34.064 7f37c37fe700 1 -- 172.21.2.102:0/3004883859 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f37c4000f40 con 0x7f37cc1b4350 2021-02-10T11:24:34.065 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:34.064 7f37c37fe700 1 -- 172.21.2.102:0/3004883859 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37c400c2f0 con 0x7f37cc1b4350 2021-02-10T11:24:34.065 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:34.064 7f37c37fe700 1 -- 172.21.2.102:0/3004883859 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f37c400c810 con 0x7f37cc1b4350 2021-02-10T11:24:34.065 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:34.064 7f37d0a72700 1 -- 172.21.2.102:0/3004883859 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37cc1b1ec0 con 0x7f37cc1b4350 2021-02-10T11:24:34.065 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:34.064 7f37d0a72700 1 -- 172.21.2.102:0/3004883859 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37cc1b1ec0 con 0x7f37cc1b4350 2021-02-10T11:24:34.066 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:34.064 7f37c37fe700 1 -- 172.21.2.102:0/3004883859 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f37c4001220 con 0x7f37cc1b4350 2021-02-10T11:24:34.066 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:34.064 7f37c37fe700 1 --2- 172.21.2.102:0/3004883859 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f37b0018700 0x7f37b001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:34.066 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:34.065 7f37c37fe700 1 -- 172.21.2.102:0/3004883859 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f37c4032a00 con 0x7f37cc1b4350 2021-02-10T11:24:34.066 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:34.065 7f37ca85e700 1 --2- 172.21.2.102:0/3004883859 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f37b0018700 0x7f37b001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:34.066 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:34.065 7f37ca85e700 1 --2- 172.21.2.102:0/3004883859 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f37b0018700 0x7f37b001abc0 crc :-1 s=READY pgs=638 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:34.093 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:34.095 7f37d0a72700 1 -- 172.21.2.102:0/3004883859 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37cc07e6b0 con 0x7f37cc1b4350 2021-02-10T11:24:34.096 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:34.097 7f37c37fe700 1 -- 172.21.2.102:0/3004883859 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f37c400be30 con 0x7f37cc1b4350 2021-02-10T11:24:34.244 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:34.245 7f37d0a72700 1 -- 172.21.2.102:0/3004883859 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f37cc0ce3e0 con 0x7f37cc1b4350 2021-02-10T11:24:34.244 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:34.245 7f37c37fe700 1 -- 172.21.2.102:0/3004883859 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v8)=0 v8) v1 ==== 122+0+0 (secure 0 0 0) 0x7f37c4001220 con 0x7f37cc1b4350 2021-02-10T11:24:34.246 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:34.247 7f37d0a72700 1 -- 172.21.2.102:0/3004883859 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f37b0018700 msgr2=0x7f37b001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:34.246 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:34.247 7f37d0a72700 1 --2- 172.21.2.102:0/3004883859 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f37b0018700 0x7f37b001abc0 crc :-1 s=READY pgs=638 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:34.246 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:34.247 7f37d0a72700 1 -- 172.21.2.102:0/3004883859 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f37cc1b4350 msgr2=0x7f37cc1b18a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:34.246 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:34.247 7f37d0a72700 1 --2- 172.21.2.102:0/3004883859 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f37cc1b4350 0x7f37cc1b18a0 secure :-1 s=READY pgs=1376 cs=0 l=1 rev1=1 rx=0x7f37c4009940 tx=0x7f37c400bcf0).stop 2021-02-10T11:24:34.247 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:34.248 7f37d0a72700 1 -- 172.21.2.102:0/3004883859 shutdown_connections 2021-02-10T11:24:34.247 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:34.248 7f37d0a72700 1 --2- 172.21.2.102:0/3004883859 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f37cc1b4350 0x7f37cc1b18a0 unknown :-1 s=CLOSED pgs=1376 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:34.247 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:34.248 7f37d0a72700 1 --2- 172.21.2.102:0/3004883859 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f37b0018700 0x7f37b001abc0 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:34.247 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:34.248 7f37d0a72700 1 --2- 172.21.2.102:0/3004883859 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f37cc1c25b0 0x7f37cc1b3d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:34.247 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:34.248 7f37d0a72700 1 --2- 172.21.2.102:0/3004883859 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f37cc0a3b50 0x7f37cc1b3710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:34.247 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:34.248 7f37d0a72700 1 -- 172.21.2.102:0/3004883859 >> 172.21.2.102:0/3004883859 conn(0x7f37cc0886d0 msgr2=0x7f37cc081c40 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:34.247 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:34.248 7f37d0a72700 1 -- 172.21.2.102:0/3004883859 shutdown_connections 2021-02-10T11:24:34.247 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:34.248 7f37d0a72700 1 -- 172.21.2.102:0/3004883859 wait complete. 2021-02-10T11:24:34.254 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:24:34.255 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:24:34.255 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:24:34.628 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:24:34.629 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2021-02-10T11:24:34.629 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@gibba002.front.sepia.ceph.com... 2021-02-10T11:24:34.629 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-02-10T11:24:34.629 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:24:34.986 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba002.stderr:ceph-fuse[153877]: fuse finished with error 0 and tester_r 0 2021-02-10T11:24:34.987 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:24:34.988 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:24:34.988 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:24:35.322 INFO:teuthology.orchestra.run.gibba002.stdout:ext2/ext3 2021-02-10T11:24:35.322 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2021-02-10T11:24:35.322 INFO:teuthology.orchestra.run:waiting for 900 2021-02-10T11:24:35.323 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:24:35.323 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:24:35.658 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo mkdir -p ./volumes/group_0000000000878856/subvolume_0000000000190758 2021-02-10T11:24:36.012 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo getfattr --only-values -n ceph.dir.layout.pool . 2021-02-10T11:24:36.373 INFO:teuthology.orchestra.run.gibba002.stdout:cephfs_data 2021-02-10T11:24:36.373 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo setfattr -n ceph.dir.layout.pool -v cephfs_data ./volumes/group_0000000000878856/subvolume_0000000000190758 2021-02-10T11:24:36.737 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume authorize cephfs subvolume_0000000000190758 alice --group_name group_0000000000878856 --tenant_id tenant_id 2021-02-10T11:24:37.175 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:37.160 7f850716c700 1 Processor -- start 2021-02-10T11:24:37.175 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:37.160 7f850716c700 1 -- start start 2021-02-10T11:24:37.175 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:37.160 7f850716c700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f85000e4ff0 0x7f85001b37a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:37.175 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:37.160 7f850716c700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f85001b3dc0 0x7f85000f3db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:37.175 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:37.160 7f850716c700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85000f5460 con 0x7f85000e53f0 2021-02-10T11:24:37.175 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:37.160 7f850716c700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85000f69f0 con 0x7f85000e4ff0 2021-02-10T11:24:37.175 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:37.160 7f850716c700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85000f6c10 con 0x7f85001b3dc0 2021-02-10T11:24:37.175 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:37.161 7f8504f88700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f85000e4ff0 0x7f85001b37a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:37.176 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:37.161 7f8504f88700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f85000e4ff0 0x7f85001b37a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:57312/0 (socket says 172.21.2.102:57312) 2021-02-10T11:24:37.176 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:37.161 7f8504f88700 1 -- 172.21.2.102:0/1522308015 learned_addr learned my addr 172.21.2.102:0/1522308015 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:24:37.176 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:37.161 7f8504f88700 1 -- 172.21.2.102:0/1522308015 >> v2:172.21.2.113:3300/0 conn(0x7f85001b3dc0 msgr2=0x7f85000f3db0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:24:37.176 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:37.161 7f8504f88700 1 --2- 172.21.2.102:0/1522308015 >> v2:172.21.2.113:3300/0 conn(0x7f85001b3dc0 0x7f85000f3db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:37.176 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:37.161 7f8504f88700 1 -- 172.21.2.102:0/1522308015 >> v1:172.21.2.113:6789/0 conn(0x7f85000e53f0 legacy=0x7f85000f43d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:24:37.176 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:37.161 7f8504f88700 1 -- 172.21.2.102:0/1522308015 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85000f6e30 con 0x7f85000e4ff0 2021-02-10T11:24:37.176 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:37.161 7f8504f88700 1 --2- 172.21.2.102:0/1522308015 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f85000e4ff0 0x7f85001b37a0 secure :-1 s=READY pgs=1377 cs=0 l=1 rev1=1 rx=0x7f84e8006700 tx=0x7f84e8009b80).ready entity=mon.0 client_cookie=ad89a7229a53b6ab server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:37.176 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:37.161 7f84ff7fe700 1 -- 172.21.2.102:0/1522308015 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f84e800c0c0 con 0x7f85000e4ff0 2021-02-10T11:24:37.176 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:37.161 7f84ff7fe700 1 -- 172.21.2.102:0/1522308015 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84e800c340 con 0x7f85000e4ff0 2021-02-10T11:24:37.176 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:37.161 7f84ff7fe700 1 -- 172.21.2.102:0/1522308015 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f84e800c820 con 0x7f85000e4ff0 2021-02-10T11:24:37.176 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:37.161 7f850716c700 1 -- 172.21.2.102:0/1522308015 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f85000e4ff0 msgr2=0x7f85001b37a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:37.176 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:37.161 7f850716c700 1 --2- 172.21.2.102:0/1522308015 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f85000e4ff0 0x7f85001b37a0 secure :-1 s=READY pgs=1377 cs=0 l=1 rev1=1 rx=0x7f84e8006700 tx=0x7f84e8009b80).stop 2021-02-10T11:24:37.177 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:37.162 7f850716c700 1 -- 172.21.2.102:0/1522308015 shutdown_connections 2021-02-10T11:24:37.177 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:37.162 7f850716c700 1 --2- 172.21.2.102:0/1522308015 >> v2:172.21.2.113:3300/0 conn(0x7f85001b3dc0 0x7f85000f3db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:37.177 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:37.162 7f850716c700 1 --2- 172.21.2.102:0/1522308015 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f85000e4ff0 0x7f85001b37a0 unknown :-1 s=CLOSED pgs=1377 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:37.177 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:37.162 7f850716c700 1 -- 172.21.2.102:0/1522308015 >> 172.21.2.102:0/1522308015 conn(0x7f85000ce650 msgr2=0x7f85000e4490 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:37.177 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:37.162 7f850716c700 1 -- 172.21.2.102:0/1522308015 shutdown_connections 2021-02-10T11:24:37.177 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:37.162 7f850716c700 1 -- 172.21.2.102:0/1522308015 wait complete. 2021-02-10T11:24:37.179 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:37.177 7f850716c700 1 Processor -- start 2021-02-10T11:24:37.179 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:37.177 7f850716c700 1 -- start start 2021-02-10T11:24:37.179 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:37.177 7f850716c700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f850019a1d0 0x7f85000a45a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:37.179 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:37.177 7f850716c700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f85000a4bc0 0x7f850018e720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:37.179 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:37.177 7f850716c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f850018ed40 0x7f85000a5120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:37.179 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:37.177 7f850716c700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f850018e1a0 con 0x7f850018ed40 2021-02-10T11:24:37.179 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:37.177 7f850716c700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f850018f140 con 0x7f850019a1d0 2021-02-10T11:24:37.179 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:37.177 7f850716c700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f850018f320 con 0x7f85000a4bc0 2021-02-10T11:24:37.179 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:37.177 7f8505789700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f850018ed40 0x7f85000a5120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:37.179 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:37.177 7f8505789700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f850018ed40 0x7f85000a5120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:57320/0 (socket says 172.21.2.102:57320) 2021-02-10T11:24:37.179 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:37.177 7f8505789700 1 -- 172.21.2.102:0/3528199306 learned_addr learned my addr 172.21.2.102:0/3528199306 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:24:37.180 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:37.177 7f8505789700 1 -- 172.21.2.102:0/3528199306 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f85000a4bc0 msgr2=0x7f850018e720 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:24:37.180 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:37.178 7f8505789700 1 --2- 172.21.2.102:0/3528199306 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f85000a4bc0 0x7f850018e720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:37.180 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:37.178 7f8505789700 1 -- 172.21.2.102:0/3528199306 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f850019a1d0 msgr2=0x7f85000a45a0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:24:37.180 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:37.178 7f8505789700 1 --2- 172.21.2.102:0/3528199306 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f850019a1d0 0x7f85000a45a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:37.180 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:37.178 7f8505789700 1 -- 172.21.2.102:0/3528199306 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85000a5740 con 0x7f850018ed40 2021-02-10T11:24:37.180 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:37.178 7f8505789700 1 --2- 172.21.2.102:0/3528199306 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f850018ed40 0x7f85000a5120 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 rx=0x7f84f800b5b0 tx=0x7f84f800b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:37.180 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:37.178 7f84fd7fa700 1 -- 172.21.2.102:0/3528199306 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f84f800bd40 con 0x7f850018ed40 2021-02-10T11:24:37.180 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:37.178 7f84fd7fa700 1 -- 172.21.2.102:0/3528199306 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84f80022e0 con 0x7f850018ed40 2021-02-10T11:24:37.180 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:37.178 7f850716c700 1 -- 172.21.2.102:0/3528199306 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85000a5740 con 0x7f850018ed40 2021-02-10T11:24:37.180 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:37.178 7f850716c700 1 -- 172.21.2.102:0/3528199306 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85000a5cd0 con 0x7f850018ed40 2021-02-10T11:24:37.180 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:37.178 7f84fd7fa700 1 -- 172.21.2.102:0/3528199306 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f84f8002770 con 0x7f850018ed40 2021-02-10T11:24:37.180 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:37.178 7f84fd7fa700 1 -- 172.21.2.102:0/3528199306 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f84f80024c0 con 0x7f850018ed40 2021-02-10T11:24:37.181 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:37.178 7f84fd7fa700 1 --2- 172.21.2.102:0/3528199306 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f84e40186c0 0x7f84e401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:37.181 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:37.178 7f84fd7fa700 1 -- 172.21.2.102:0/3528199306 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f84f80327e0 con 0x7f850018ed40 2021-02-10T11:24:37.181 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:37.179 7f8504f88700 1 --2- 172.21.2.102:0/3528199306 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f84e40186c0 0x7f84e401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:37.181 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:37.179 7f8504f88700 1 --2- 172.21.2.102:0/3528199306 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f84e40186c0 0x7f84e401abc0 crc :-1 s=READY pgs=639 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:37.209 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:37.210 7f850716c700 1 -- 172.21.2.102:0/3528199306 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8500101680 con 0x7f850018ed40 2021-02-10T11:24:37.211 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:37.213 7f84fd7fa700 1 -- 172.21.2.102:0/3528199306 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f84f8000ef0 con 0x7f850018ed40 2021-02-10T11:24:37.336 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:37.338 7f850716c700 1 -- 172.21.2.102:0/3528199306 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume authorize", "vol_name": "cephfs", "sub_name": "subvolume_0000000000190758", "target": ["mgr", ""], "tenant_id": "tenant_id", "auth_id": "alice", "group_name": "group_0000000000878856"}) v1 -- 0x7f85000a5740 con 0x7f84e40186c0 2021-02-10T11:24:37.340 INFO:tasks.ceph.mgr.y.gibba002.stderr:2021-02-10 11:24:37.341 7fd94bc53700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:24:37.340 INFO:tasks.ceph.mgr.y.gibba002.stderr:2021-02-10 11:24:37.341 7fd94bc53700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:24:37.340 INFO:tasks.ceph.mgr.y.gibba002.stderr:2021-02-10 11:24:37.341 7fd94bc53700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:24:37.340 INFO:tasks.ceph.mgr.y.gibba002.stderr:2021-02-10 11:24:37.341 7fd94bc53700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:24:37.341 INFO:tasks.ceph.mgr.y.gibba002.stderr:2021-02-10 11:24:37.341 7fd94bc53700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:24:37.368 INFO:tasks.ceph.mgr.y.gibba002.stderr:Creating meta for ID alice with tenant tenant_id 2021-02-10T11:24:37.395 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:37.397 7f84fd7fa700 1 -- 172.21.2.102:0/3528199306 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+41 (crc 0 0 0) 0x7f84e80090b0 con 0x7f84e40186c0 2021-02-10T11:24:37.400 INFO:teuthology.orchestra.run.gibba002.stdout:AQB1wiNgRCisFhAAXTTug/zXvbqGFfamBUFBxQ== 2021-02-10T11:24:37.402 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:37.403 7f850716c700 1 -- 172.21.2.102:0/3528199306 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f84e40186c0 msgr2=0x7f84e401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:37.402 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:37.403 7f850716c700 1 --2- 172.21.2.102:0/3528199306 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f84e40186c0 0x7f84e401abc0 crc :-1 s=READY pgs=639 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:37.402 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:37.403 7f850716c700 1 -- 172.21.2.102:0/3528199306 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f850018ed40 msgr2=0x7f85000a5120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:37.402 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:37.403 7f850716c700 1 --2- 172.21.2.102:0/3528199306 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f850018ed40 0x7f85000a5120 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 rx=0x7f84f800b5b0 tx=0x7f84f800b950).stop 2021-02-10T11:24:37.402 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:37.403 7f850716c700 1 -- 172.21.2.102:0/3528199306 shutdown_connections 2021-02-10T11:24:37.402 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:37.403 7f850716c700 1 --2- 172.21.2.102:0/3528199306 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f850018ed40 0x7f85000a5120 unknown :-1 s=CLOSED pgs=1378 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:37.402 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:37.403 7f850716c700 1 --2- 172.21.2.102:0/3528199306 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f85000a4bc0 0x7f850018e720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:37.402 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:37.403 7f850716c700 1 --2- 172.21.2.102:0/3528199306 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f84e40186c0 0x7f84e401abc0 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:37.402 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:37.403 7f850716c700 1 --2- 172.21.2.102:0/3528199306 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f850019a1d0 0x7f85000a45a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:37.402 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:37.403 7f850716c700 1 -- 172.21.2.102:0/3528199306 >> 172.21.2.102:0/3528199306 conn(0x7f8500107720 msgr2=0x7f85000a3930 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:37.403 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:37.403 7f850716c700 1 -- 172.21.2.102:0/3528199306 shutdown_connections 2021-02-10T11:24:37.403 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:37.403 7f850716c700 1 -- 172.21.2.102:0/3528199306 wait complete. 2021-02-10T11:24:37.410 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth ls --format=json-pretty 2021-02-10T11:24:37.838 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:37.822 7f3ff3ca5700 1 Processor -- start 2021-02-10T11:24:37.838 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:37.822 7f3ff3ca5700 1 -- start start 2021-02-10T11:24:37.838 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:37.822 7f3ff3ca5700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f3fec0e5070 0x7f3fec1b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:37.838 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:37.822 7f3ff3ca5700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f3fec1b3e40 0x7f3fec07d690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:37.838 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:37.822 7f3ff3ca5700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fec07ed80 con 0x7f3fec0e5470 2021-02-10T11:24:37.839 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:37.822 7f3ff3ca5700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3fec080310 con 0x7f3fec1b3e40 2021-02-10T11:24:37.839 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:37.822 7f3ff3ca5700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3fec080530 con 0x7f3fec0e5070 2021-02-10T11:24:37.839 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:37.823 7f3ff12c0700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f3fec1b3e40 0x7f3fec07d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:37.839 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:37.823 7f3ff12c0700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f3fec1b3e40 0x7f3fec07d690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:57342/0 (socket says 172.21.2.102:57342) 2021-02-10T11:24:37.839 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:37.823 7f3ff12c0700 1 -- 172.21.2.102:0/3942571059 learned_addr learned my addr 172.21.2.102:0/3942571059 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:24:37.839 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:37.823 7f3ff12c0700 1 -- 172.21.2.102:0/3942571059 >> v2:172.21.2.113:3300/0 conn(0x7f3fec0e5070 msgr2=0x7f3fec1b3820 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:24:37.839 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:37.823 7f3ff12c0700 1 --2- 172.21.2.102:0/3942571059 >> v2:172.21.2.113:3300/0 conn(0x7f3fec0e5070 0x7f3fec1b3820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:37.840 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:37.823 7f3ff12c0700 1 -- 172.21.2.102:0/3942571059 >> v1:172.21.2.102:6789/0 conn(0x7f3fec0e5470 legacy=0x7f3fec07dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:37.840 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:37.823 7f3ff12c0700 1 -- 172.21.2.102:0/3942571059 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fec080750 con 0x7f3fec1b3e40 2021-02-10T11:24:37.840 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:37.823 7f3ff12c0700 1 --2- 172.21.2.102:0/3942571059 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3fec1b3e40 0x7f3fec07d690 secure :-1 s=READY pgs=1382 cs=0 l=1 rev1=1 rx=0x7f3fe8006700 tx=0x7f3fe8009b00).ready entity=mon.0 client_cookie=4408e771b90347b1 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:37.840 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:37.823 7f3ff0abf700 1 -- 172.21.2.102:0/3942571059 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3fe800c0c0 con 0x7f3fec1b3e40 2021-02-10T11:24:37.840 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:37.823 7f3ff0abf700 1 -- 172.21.2.102:0/3942571059 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fe800c340 con 0x7f3fec1b3e40 2021-02-10T11:24:37.840 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:37.823 7f3ff0abf700 1 -- 172.21.2.102:0/3942571059 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3fe800c820 con 0x7f3fec1b3e40 2021-02-10T11:24:37.841 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:37.823 7f3ff3ca5700 1 -- 172.21.2.102:0/3942571059 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3fec1b3e40 msgr2=0x7f3fec07d690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:37.841 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:37.823 7f3ff3ca5700 1 --2- 172.21.2.102:0/3942571059 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3fec1b3e40 0x7f3fec07d690 secure :-1 s=READY pgs=1382 cs=0 l=1 rev1=1 rx=0x7f3fe8006700 tx=0x7f3fe8009b00).stop 2021-02-10T11:24:37.841 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:37.824 7f3ff3ca5700 1 -- 172.21.2.102:0/3942571059 shutdown_connections 2021-02-10T11:24:37.841 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:37.824 7f3ff3ca5700 1 --2- 172.21.2.102:0/3942571059 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3fec1b3e40 0x7f3fec07d690 unknown :-1 s=CLOSED pgs=1382 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:37.841 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:37.824 7f3ff3ca5700 1 --2- 172.21.2.102:0/3942571059 >> v2:172.21.2.113:3300/0 conn(0x7f3fec0e5070 0x7f3fec1b3820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:37.841 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:37.824 7f3ff3ca5700 1 -- 172.21.2.102:0/3942571059 >> 172.21.2.102:0/3942571059 conn(0x7f3fec0ce690 msgr2=0x7f3fec0e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:37.842 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:37.824 7f3ff3ca5700 1 -- 172.21.2.102:0/3942571059 shutdown_connections 2021-02-10T11:24:37.842 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:37.824 7f3ff3ca5700 1 -- 172.21.2.102:0/3942571059 wait complete. 2021-02-10T11:24:37.842 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:37.839 7f3ff3ca5700 1 Processor -- start 2021-02-10T11:24:37.842 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:37.839 7f3ff3ca5700 1 -- start start 2021-02-10T11:24:37.844 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:37.839 7f3ff3ca5700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3fec08c830 0x7f3fec08d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:37.844 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:37.839 7f3ff3ca5700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3fec1afe20 0x7f3fec0b2f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:37.844 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:37.839 7f3ff3ca5700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3fec1b0220 0x7f3fec0b3550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:37.844 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:37.839 7f3ff3ca5700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fec08cc30 con 0x7f3fec1afe20 2021-02-10T11:24:37.845 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:37.839 7f3ff3ca5700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fec0b3b70 con 0x7f3fec1b0220 2021-02-10T11:24:37.845 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:37.839 7f3ff3ca5700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fec0b3d90 con 0x7f3fec08c830 2021-02-10T11:24:37.845 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:37.839 7f3ff12c0700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3fec1afe20 0x7f3fec0b2f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:37.845 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:37.839 7f3ff12c0700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3fec1afe20 0x7f3fec0b2f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:57350/0 (socket says 172.21.2.102:57350) 2021-02-10T11:24:37.845 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:37.839 7f3ff12c0700 1 -- 172.21.2.102:0/2770440881 learned_addr learned my addr 172.21.2.102:0/2770440881 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:24:37.845 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:37.840 7f3ff12c0700 1 -- 172.21.2.102:0/2770440881 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3fec08c830 msgr2=0x7f3fec08d0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:37.845 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:37.840 7f3ff12c0700 1 --2- 172.21.2.102:0/2770440881 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3fec08c830 0x7f3fec08d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:37.846 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:37.840 7f3ff12c0700 1 -- 172.21.2.102:0/2770440881 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3fec1b0220 msgr2=0x7f3fec0b3550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:37.846 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:37.840 7f3ff12c0700 1 --2- 172.21.2.102:0/2770440881 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3fec1b0220 0x7f3fec0b3550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:37.846 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:37.840 7f3ff12c0700 1 -- 172.21.2.102:0/2770440881 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fec1b0c00 con 0x7f3fec1afe20 2021-02-10T11:24:37.846 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:37.840 7f3ff12c0700 1 --2- 172.21.2.102:0/2770440881 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3fec1afe20 0x7f3fec0b2f30 secure :-1 s=READY pgs=1383 cs=0 l=1 rev1=1 rx=0x7f3fe8006f20 tx=0x7f3fe800b9b0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:37.846 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:37.840 7f3fde7fc700 1 -- 172.21.2.102:0/2770440881 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3fe8009100 con 0x7f3fec1afe20 2021-02-10T11:24:37.846 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:37.840 7f3fde7fc700 1 -- 172.21.2.102:0/2770440881 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fe800bda0 con 0x7f3fec1afe20 2021-02-10T11:24:37.847 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:37.840 7f3fde7fc700 1 -- 172.21.2.102:0/2770440881 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3fe80054d0 con 0x7f3fec1afe20 2021-02-10T11:24:37.847 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:37.840 7f3ff3ca5700 1 -- 172.21.2.102:0/2770440881 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fec1b0c00 con 0x7f3fec1afe20 2021-02-10T11:24:37.847 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:37.840 7f3ff3ca5700 1 -- 172.21.2.102:0/2770440881 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fec1b0c00 con 0x7f3fec1afe20 2021-02-10T11:24:37.847 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:37.841 7f3fde7fc700 1 -- 172.21.2.102:0/2770440881 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f3fe8006030 con 0x7f3fec1afe20 2021-02-10T11:24:37.847 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:37.841 7f3fde7fc700 1 --2- 172.21.2.102:0/2770440881 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3fd40186c0 0x7f3fd401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:37.847 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:37.841 7f3fde7fc700 1 -- 172.21.2.102:0/2770440881 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f3fe80327c0 con 0x7f3fec1afe20 2021-02-10T11:24:37.848 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:37.841 7f3ff1ac1700 1 --2- 172.21.2.102:0/2770440881 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3fd40186c0 0x7f3fd401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:37.848 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:37.842 7f3ff1ac1700 1 --2- 172.21.2.102:0/2770440881 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3fd40186c0 0x7f3fd401abc0 crc :-1 s=READY pgs=640 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:37.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:37.872 7f3ff3ca5700 1 -- 172.21.2.102:0/2770440881 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fec083db0 con 0x7f3fec1afe20 2021-02-10T11:24:37.873 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:37.875 7f3fde7fc700 1 -- 172.21.2.102:0/2770440881 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f3fe80064c0 con 0x7f3fec1afe20 2021-02-10T11:24:38.006 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:38.007 7f3ff3ca5700 1 -- 172.21.2.102:0/2770440881 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f3fec07ed80 con 0x7f3fec1afe20 2021-02-10T11:24:38.007 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:38.008 7f3fde7fc700 1 -- 172.21.2.102:0/2770440881 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v33) v1 ==== 80+0+4855 (secure 0 0 0) 0x7f3fe80064c0 con 0x7f3fec1afe20 2021-02-10T11:24:38.007 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:24:38.007 INFO:teuthology.orchestra.run.gibba002.stdout:{ 2021-02-10T11:24:38.007 INFO:teuthology.orchestra.run.gibba002.stdout: "auth_dump": [ 2021-02-10T11:24:38.007 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:24:38.008 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "mds.a", 2021-02-10T11:24:38.008 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAevSNgsgyEARAAvr0cUjjCoEHLxaZSZ2HtVA==", 2021-02-10T11:24:38.008 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:24:38.008 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow", 2021-02-10T11:24:38.008 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:24:38.008 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:24:38.008 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:24:38.008 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:24:38.008 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:24:38.008 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:24:38.008 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "mds.b", 2021-02-10T11:24:38.008 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAevSNgwOFMMRAALfb1fbX56ao1TU8ClnTW1A==", 2021-02-10T11:24:38.009 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:24:38.009 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow", 2021-02-10T11:24:38.009 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:24:38.009 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:24:38.009 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:24:38.009 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:24:38.009 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:24:38.009 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:24:38.009 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.0", 2021-02-10T11:24:38.009 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAmvSNgj+bMMBAAD3HUD6fLiMAg+++iNzLJKQ==", 2021-02-10T11:24:38.009 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:24:38.009 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:24:38.009 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:24:38.010 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:24:38.010 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:24:38.010 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:24:38.010 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:24:38.010 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.1", 2021-02-10T11:24:38.010 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQApvSNgnvc+MRAAAOxqlNwiyLHqNITOrYeFKQ==", 2021-02-10T11:24:38.010 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:24:38.010 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:24:38.010 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:24:38.010 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:24:38.010 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:24:38.010 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:24:38.010 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:24:38.011 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.2", 2021-02-10T11:24:38.011 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAsvSNgZ4SbMRAA76IOYWfbiAuBoM/4k81GxA==", 2021-02-10T11:24:38.011 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:24:38.011 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:24:38.011 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:24:38.011 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:24:38.011 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:24:38.011 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:24:38.011 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:24:38.011 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.3", 2021-02-10T11:24:38.011 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAvvSNgFk3rMRAAWCmKKyw0LDQ+UlAHZAITOw==", 2021-02-10T11:24:38.011 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:24:38.011 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:24:38.012 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:24:38.012 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:24:38.012 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:24:38.012 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:24:38.012 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:24:38.012 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.4", 2021-02-10T11:24:38.012 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQA4vSNgpKcNJBAApJMmvj5YxVHPDOmvmQONOg==", 2021-02-10T11:24:38.012 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:24:38.012 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:24:38.012 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:24:38.012 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:24:38.012 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:24:38.013 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:24:38.013 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:24:38.013 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.5", 2021-02-10T11:24:38.013 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQA7vSNgf2ujJBAAfkuYpyb3WTzc8V5kw+Wktg==", 2021-02-10T11:24:38.013 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:24:38.013 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:24:38.013 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:24:38.013 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:24:38.013 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:24:38.013 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:24:38.013 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:24:38.014 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.6", 2021-02-10T11:24:38.014 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQA+vSNgqw9uJRAAFMU0R+UxU9C3o/9byb3e0g==", 2021-02-10T11:24:38.014 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:24:38.014 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:24:38.014 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:24:38.014 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:24:38.014 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:24:38.014 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:24:38.014 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:24:38.014 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.7", 2021-02-10T11:24:38.015 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQBBvSNgvWSrJRAA2KaCHmLpzovRgYzLvyofTQ==", 2021-02-10T11:24:38.015 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:24:38.015 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:24:38.015 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:24:38.015 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:24:38.015 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:24:38.015 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:24:38.015 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:24:38.015 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "client.0", 2021-02-10T11:24:38.015 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAfvSNgZd9XIBAAvyrvl8fdKuHnJZfczfRoFw==", 2021-02-10T11:24:38.015 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:24:38.015 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow", 2021-02-10T11:24:38.015 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow r", 2021-02-10T11:24:38.016 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:24:38.016 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:24:38.016 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:24:38.016 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:24:38.016 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "client.1", 2021-02-10T11:24:38.016 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAfvSNgSyfXNhAAIduPnF7Vd/6HSkJ0MlcMSA==", 2021-02-10T11:24:38.016 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:24:38.016 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow", 2021-02-10T11:24:38.016 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow r", 2021-02-10T11:24:38.016 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:24:38.016 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:24:38.016 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:24:38.016 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:24:38.017 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "client.2", 2021-02-10T11:24:38.017 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAgvSNgiIG/ERAA9ZTamaTI7ZsPSnxBqjplCg==", 2021-02-10T11:24:38.017 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:24:38.017 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow", 2021-02-10T11:24:38.017 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow r", 2021-02-10T11:24:38.017 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:24:38.017 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:24:38.017 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:24:38.017 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:24:38.017 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "client.3", 2021-02-10T11:24:38.017 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAgvSNgOWpOKBAAlXsEB04Q/GvtL06PqRMLeA==", 2021-02-10T11:24:38.017 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:24:38.018 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow", 2021-02-10T11:24:38.018 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow r", 2021-02-10T11:24:38.018 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:24:38.018 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:24:38.018 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:24:38.018 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:24:38.018 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "client.admin", 2021-02-10T11:24:38.018 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAavSNgO3GQBhAAL+NVnYskQRKVWLgyDPY/xw==", 2021-02-10T11:24:38.018 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:24:38.018 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow *", 2021-02-10T11:24:38.018 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:24:38.018 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:24:38.018 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:24:38.018 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:24:38.019 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:24:38.019 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:24:38.019 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "client.alice", 2021-02-10T11:24:38.019 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQB1wiNgRCisFhAAXTTug/zXvbqGFfamBUFBxQ==", 2021-02-10T11:24:38.019 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:24:38.019 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow rw path=/volumes/group_0000000000878856/subvolume_0000000000190758", 2021-02-10T11:24:38.019 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow r", 2021-02-10T11:24:38.019 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:24:38.019 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:24:38.019 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:24:38.019 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:24:38.019 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "mgr.x", 2021-02-10T11:24:38.019 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAdvSNgXZZgJhAA7dtBSGzH+DsBwnddZpOBJA==", 2021-02-10T11:24:38.020 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:24:38.020 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow *", 2021-02-10T11:24:38.020 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow profile mgr", 2021-02-10T11:24:38.020 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:24:38.020 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:24:38.020 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:24:38.020 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:24:38.020 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "mgr.y", 2021-02-10T11:24:38.020 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAdvSNgVqQFDRAAwo6YEdWZCAr5GdOWVL/Qvg==", 2021-02-10T11:24:38.020 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:24:38.020 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow *", 2021-02-10T11:24:38.020 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow profile mgr", 2021-02-10T11:24:38.020 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:24:38.020 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:24:38.021 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:24:38.021 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:24:38.021 INFO:teuthology.orchestra.run.gibba002.stdout:} 2021-02-10T11:24:38.022 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:38.010 7f3ff3ca5700 1 -- 172.21.2.102:0/2770440881 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3fd40186c0 msgr2=0x7f3fd401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:38.022 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:38.010 7f3ff3ca5700 1 --2- 172.21.2.102:0/2770440881 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3fd40186c0 0x7f3fd401abc0 crc :-1 s=READY pgs=640 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:38.022 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:38.010 7f3ff3ca5700 1 -- 172.21.2.102:0/2770440881 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3fec1afe20 msgr2=0x7f3fec0b2f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:38.022 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:38.010 7f3ff3ca5700 1 --2- 172.21.2.102:0/2770440881 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3fec1afe20 0x7f3fec0b2f30 secure :-1 s=READY pgs=1383 cs=0 l=1 rev1=1 rx=0x7f3fe8006f20 tx=0x7f3fe800b9b0).stop 2021-02-10T11:24:38.022 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:38.010 7f3ff3ca5700 1 -- 172.21.2.102:0/2770440881 shutdown_connections 2021-02-10T11:24:38.022 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:38.010 7f3ff3ca5700 1 --2- 172.21.2.102:0/2770440881 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3fd40186c0 0x7f3fd401abc0 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:38.022 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:38.010 7f3ff3ca5700 1 --2- 172.21.2.102:0/2770440881 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3fec1b0220 0x7f3fec0b3550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:38.023 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:38.010 7f3ff3ca5700 1 --2- 172.21.2.102:0/2770440881 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3fec1afe20 0x7f3fec0b2f30 unknown :-1 s=CLOSED pgs=1383 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:38.023 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:38.010 7f3ff3ca5700 1 --2- 172.21.2.102:0/2770440881 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3fec08c830 0x7f3fec08d0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:38.023 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:38.010 7f3ff3ca5700 1 -- 172.21.2.102:0/2770440881 >> 172.21.2.102:0/2770440881 conn(0x7f3fec0937a0 msgr2=0x7f3fec197e80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:38.023 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:38.010 7f3ff3ca5700 1 -- 172.21.2.102:0/2770440881 shutdown_connections 2021-02-10T11:24:38.023 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:38.010 7f3ff3ca5700 1 -- 172.21.2.102:0/2770440881 wait complete. 2021-02-10T11:24:38.024 DEBUG:teuthology.orchestra.run.gibba002:> set -ex 2021-02-10T11:24:38.024 DEBUG:teuthology.orchestra.run.gibba002:> sudo dd of=/etc/ceph/ceph.client.alice.keyring 2021-02-10T11:24:38.367 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config set client.alice debug_client 20 2021-02-10T11:24:38.798 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:38.782 7fd158aeb700 1 Processor -- start 2021-02-10T11:24:38.798 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:38.782 7fd158aeb700 1 -- start start 2021-02-10T11:24:38.798 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:38.783 7fd158aeb700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd1541b0470 0x7fd1541b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:38.798 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:38.783 7fd158aeb700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fd1540ef610 0x7fd1540efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:38.798 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:38.783 7fd158aeb700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1540f0b50 con 0x7fd1540f0030 2021-02-10T11:24:38.798 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:38.783 7fd158aeb700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1540e52e0 con 0x7fd1540ef610 2021-02-10T11:24:38.798 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:38.783 7fd158aeb700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1540e5490 con 0x7fd1541b0470 2021-02-10T11:24:38.799 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:38.783 7fd15205d700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fd1540ef610 0x7fd1540efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:38.799 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:38.783 7fd15305f700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fd1540f0030 0x7fd1540f10d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:49582/0 (socket says 172.21.2.102:49582) 2021-02-10T11:24:38.799 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:38.783 7fd15305f700 1 -- 172.21.2.102:0/2597162265 learned_addr learned my addr 172.21.2.102:0/2597162265 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:24:38.799 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:38.783 7fd15205d700 1 -- 172.21.2.102:0/2597162265 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd1541b0470 msgr2=0x7fd1541b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:24:38.799 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:38.783 7fd15205d700 1 --2- 172.21.2.102:0/2597162265 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd1541b0470 0x7fd1541b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:38.799 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:38.783 7fd15205d700 1 -- 172.21.2.102:0/2597162265 >> v1:172.21.2.102:6789/0 conn(0x7fd1540f0030 legacy=0x7fd1540f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:38.799 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:38.783 7fd15205d700 1 -- 172.21.2.102:0/2597162265 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1540f0430 con 0x7fd1540ef610 2021-02-10T11:24:38.799 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:38.783 7fd15185c700 1 -- 172.21.2.102:0/2597162265 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2832796037 0 0) 0x7fd14c001500 con 0x7fd1540f0030 2021-02-10T11:24:38.799 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:38.784 7fd15205d700 1 --2- 172.21.2.102:0/2597162265 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd1540ef610 0x7fd1540efa10 secure :-1 s=READY pgs=1385 cs=0 l=1 rev1=1 rx=0x7fd144006650 tx=0x7fd144009a80).ready entity=mon.0 client_cookie=7b0e32b756105603 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:38.799 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:38.784 7fd15185c700 1 -- 172.21.2.102:0/2597162265 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd1440009b0 con 0x7fd1540ef610 2021-02-10T11:24:38.799 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:38.784 7fd15185c700 1 -- 172.21.2.102:0/2597162265 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd14400c0c0 con 0x7fd1540ef610 2021-02-10T11:24:38.799 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:38.784 7fd15185c700 1 -- 172.21.2.102:0/2597162265 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd14400c520 con 0x7fd1540ef610 2021-02-10T11:24:38.799 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:38.784 7fd158aeb700 1 -- 172.21.2.102:0/2597162265 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd1540ef610 msgr2=0x7fd1540efa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:38.800 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:38.784 7fd158aeb700 1 --2- 172.21.2.102:0/2597162265 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd1540ef610 0x7fd1540efa10 secure :-1 s=READY pgs=1385 cs=0 l=1 rev1=1 rx=0x7fd144006650 tx=0x7fd144009a80).stop 2021-02-10T11:24:38.800 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:38.785 7fd158aeb700 1 -- 172.21.2.102:0/2597162265 shutdown_connections 2021-02-10T11:24:38.800 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:38.785 7fd158aeb700 1 --2- 172.21.2.102:0/2597162265 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd1540ef610 0x7fd1540efa10 unknown :-1 s=CLOSED pgs=1385 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:38.800 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:38.785 7fd158aeb700 1 --2- 172.21.2.102:0/2597162265 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd1541b0470 0x7fd1541b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:38.800 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:38.785 7fd158aeb700 1 -- 172.21.2.102:0/2597162265 >> 172.21.2.102:0/2597162265 conn(0x7fd1540ce3e0 msgr2=0x7fd1540e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:38.800 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:38.785 7fd158aeb700 1 -- 172.21.2.102:0/2597162265 shutdown_connections 2021-02-10T11:24:38.800 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:38.785 7fd158aeb700 1 -- 172.21.2.102:0/2597162265 wait complete. 2021-02-10T11:24:38.800 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:38.799 7fd158aeb700 1 Processor -- start 2021-02-10T11:24:38.800 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:38.799 7fd158aeb700 1 -- start start 2021-02-10T11:24:38.801 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:38.800 7fd158aeb700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd1540b20b0 0x7fd1540ac970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:38.802 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:38.800 7fd158aeb700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd1540b24b0 0x7fd154191dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:38.802 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:38.800 7fd158aeb700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd1540acf90 0x7fd1541923e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:38.802 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:38.800 7fd158aeb700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1540b1b30 con 0x7fd1540b24b0 2021-02-10T11:24:38.802 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:38.800 7fd158aeb700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1541a04e0 con 0x7fd1540acf90 2021-02-10T11:24:38.802 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:38.800 7fd158aeb700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1541a06c0 con 0x7fd1540b20b0 2021-02-10T11:24:38.802 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:38.800 7fd15205d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd1540b24b0 0x7fd154191dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:38.802 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:38.800 7fd15205d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd1540b24b0 0x7fd154191dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:57362/0 (socket says 172.21.2.102:57362) 2021-02-10T11:24:38.802 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:38.800 7fd15205d700 1 -- 172.21.2.102:0/1560036757 learned_addr learned my addr 172.21.2.102:0/1560036757 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:24:38.802 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:38.800 7fd15205d700 1 -- 172.21.2.102:0/1560036757 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd1540b20b0 msgr2=0x7fd1540ac970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:38.802 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:38.800 7fd15205d700 1 --2- 172.21.2.102:0/1560036757 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd1540b20b0 0x7fd1540ac970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:38.802 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:38.800 7fd15205d700 1 -- 172.21.2.102:0/1560036757 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd1540acf90 msgr2=0x7fd1541923e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:38.803 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:38.800 7fd15205d700 1 --2- 172.21.2.102:0/1560036757 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd1540acf90 0x7fd1541923e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:38.803 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:38.800 7fd15205d700 1 -- 172.21.2.102:0/1560036757 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd154192be0 con 0x7fd1540b24b0 2021-02-10T11:24:38.803 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:38.800 7fd15205d700 1 --2- 172.21.2.102:0/1560036757 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd1540b24b0 0x7fd154191dc0 secure :-1 s=READY pgs=1386 cs=0 l=1 rev1=1 rx=0x7fd144009630 tx=0x7fd14400b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:38.803 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:38.800 7fd14b7fe700 1 -- 172.21.2.102:0/1560036757 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd1440093a0 con 0x7fd1540b24b0 2021-02-10T11:24:38.803 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:38.801 7fd14b7fe700 1 -- 172.21.2.102:0/1560036757 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd14400bd70 con 0x7fd1540b24b0 2021-02-10T11:24:38.803 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:38.801 7fd14b7fe700 1 -- 172.21.2.102:0/1560036757 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd1440054d0 con 0x7fd1540b24b0 2021-02-10T11:24:38.803 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:38.801 7fd158aeb700 1 -- 172.21.2.102:0/1560036757 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd154192be0 con 0x7fd1540b24b0 2021-02-10T11:24:38.803 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:38.801 7fd158aeb700 1 -- 172.21.2.102:0/1560036757 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd154193170 con 0x7fd1540b24b0 2021-02-10T11:24:38.803 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:38.801 7fd14b7fe700 1 -- 172.21.2.102:0/1560036757 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd14400a000 con 0x7fd1540b24b0 2021-02-10T11:24:38.803 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:38.801 7fd14b7fe700 1 --2- 172.21.2.102:0/1560036757 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd138018730 0x7fd13801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:38.803 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:38.801 7fd14b7fe700 1 -- 172.21.2.102:0/1560036757 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fd1440327d0 con 0x7fd1540b24b0 2021-02-10T11:24:38.803 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:38.801 7fd15285e700 1 --2- 172.21.2.102:0/1560036757 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd138018730 0x7fd13801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:38.803 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:38.802 7fd15285e700 1 --2- 172.21.2.102:0/1560036757 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd138018730 0x7fd13801ac10 crc :-1 s=READY pgs=641 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:38.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:38.833 7fd158aeb700 1 -- 172.21.2.102:0/1560036757 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd154105af0 con 0x7fd1540b24b0 2021-02-10T11:24:38.834 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:38.835 7fd14b7fe700 1 -- 172.21.2.102:0/1560036757 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd144009340 con 0x7fd1540b24b0 2021-02-10T11:24:38.985 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:38.986 7fd158aeb700 1 -- 172.21.2.102:0/1560036757 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command([{prefix=config set, name=debug_client}] v 0) v1 -- 0x7fd1540f0b50 con 0x7fd1540b24b0 2021-02-10T11:24:38.989 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:38.990 7fd14b7fe700 1 -- 172.21.2.102:0/1560036757 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{prefix=config set, name=debug_client}]=0 v9)=0 v9) v1 ==== 120+0+0 (secure 0 0 0) 0x7fd14400a000 con 0x7fd1540b24b0 2021-02-10T11:24:38.994 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:38.995 7fd158aeb700 1 -- 172.21.2.102:0/1560036757 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd138018730 msgr2=0x7fd13801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:38.994 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:38.995 7fd158aeb700 1 --2- 172.21.2.102:0/1560036757 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd138018730 0x7fd13801ac10 crc :-1 s=READY pgs=641 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:38.994 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:38.995 7fd158aeb700 1 -- 172.21.2.102:0/1560036757 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd1540b24b0 msgr2=0x7fd154191dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:38.994 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:38.995 7fd158aeb700 1 --2- 172.21.2.102:0/1560036757 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd1540b24b0 0x7fd154191dc0 secure :-1 s=READY pgs=1386 cs=0 l=1 rev1=1 rx=0x7fd144009630 tx=0x7fd14400b980).stop 2021-02-10T11:24:38.994 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:38.995 7fd158aeb700 1 -- 172.21.2.102:0/1560036757 shutdown_connections 2021-02-10T11:24:38.995 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:38.995 7fd158aeb700 1 --2- 172.21.2.102:0/1560036757 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd138018730 0x7fd13801ac10 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:38.995 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:38.995 7fd158aeb700 1 --2- 172.21.2.102:0/1560036757 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd1540acf90 0x7fd1541923e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:38.995 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:38.995 7fd158aeb700 1 --2- 172.21.2.102:0/1560036757 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd1540b24b0 0x7fd154191dc0 unknown :-1 s=CLOSED pgs=1386 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:38.995 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:38.995 7fd158aeb700 1 --2- 172.21.2.102:0/1560036757 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd1540b20b0 0x7fd1540ac970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:38.995 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:38.995 7fd158aeb700 1 -- 172.21.2.102:0/1560036757 >> 172.21.2.102:0/1560036757 conn(0x7fd1540bf4e0 msgr2=0x7fd1540be3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:38.995 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:38.995 7fd158aeb700 1 -- 172.21.2.102:0/1560036757 shutdown_connections 2021-02-10T11:24:38.995 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:38.995 7fd158aeb700 1 -- 172.21.2.102:0/1560036757 wait complete. 2021-02-10T11:24:39.002 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config set client.alice debug_objecter 20 2021-02-10T11:24:39.431 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:39.415 7fecbbc88700 1 Processor -- start 2021-02-10T11:24:39.431 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:39.416 7fecbbc88700 1 -- start start 2021-02-10T11:24:39.432 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:39.416 7fecbbc88700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fecb41b0470 0x7fecb41b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:39.432 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:39.416 7fecbbc88700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fecb40f20e0 0x7fecb40f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:39.432 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:39.416 7fecbbc88700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fecb40f0b50 con 0x7fecb40ef610 2021-02-10T11:24:39.432 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:39.416 7fecbbc88700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fecb40f0820 con 0x7fecb41b0470 2021-02-10T11:24:39.432 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:39.416 7fecbbc88700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fecb40f24e0 con 0x7fecb40f20e0 2021-02-10T11:24:39.432 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:39.416 7fecb92a3700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fecb40ef610 0x7fecb40efa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:49592/0 (socket says 172.21.2.102:49592) 2021-02-10T11:24:39.432 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:39.416 7fecb92a3700 1 -- 172.21.2.102:0/3281197395 learned_addr learned my addr 172.21.2.102:0/3281197395 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:24:39.432 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:39.416 7fecba2a5700 1 --2- 172.21.2.102:0/3281197395 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fecb40f20e0 0x7fecb40f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:39.432 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:39.417 7fecb9aa4700 1 --2- 172.21.2.102:0/3281197395 >> v2:172.21.2.113:3300/0 conn(0x7fecb41b0470 0x7fecb41b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:39.432 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:39.417 7fecb8aa2700 1 -- 172.21.2.102:0/3281197395 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2571956526 0 0) 0x7feca80010b0 con 0x7fecb40ef610 2021-02-10T11:24:39.432 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:39.417 7fecb8aa2700 1 -- 172.21.2.102:0/3281197395 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feca0002fe0 con 0x7fecb40ef610 2021-02-10T11:24:39.432 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:39.417 7fecba2a5700 1 -- 172.21.2.102:0/3281197395 >> v2:172.21.2.113:3300/0 conn(0x7fecb41b0470 msgr2=0x7fecb41b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:24:39.433 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:39.417 7fecba2a5700 1 --2- 172.21.2.102:0/3281197395 >> v2:172.21.2.113:3300/0 conn(0x7fecb41b0470 0x7fecb41b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:39.433 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:39.417 7fecba2a5700 1 -- 172.21.2.102:0/3281197395 >> v1:172.21.2.102:6789/0 conn(0x7fecb40ef610 legacy=0x7fecb40efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:39.433 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:39.417 7fecba2a5700 1 -- 172.21.2.102:0/3281197395 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecb40e52e0 con 0x7fecb40f20e0 2021-02-10T11:24:39.433 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:39.417 7fecb9aa4700 1 --2- 172.21.2.102:0/3281197395 >> v2:172.21.2.113:3300/0 conn(0x7fecb41b0470 0x7fecb41b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:24:39.433 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:39.417 7fecba2a5700 1 --2- 172.21.2.102:0/3281197395 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fecb40f20e0 0x7fecb40f0200 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7fecac006e50 tx=0x7fecac009e70).ready entity=mon.2 client_cookie=9692b3421824fd12 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:39.433 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:39.417 7fecb8aa2700 1 -- 172.21.2.102:0/3281197395 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fecac00b510 con 0x7fecb40f20e0 2021-02-10T11:24:39.433 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:39.417 7fecb8aa2700 1 -- 172.21.2.102:0/3281197395 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fecac006790 con 0x7fecb40f20e0 2021-02-10T11:24:39.433 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:39.417 7fecb8aa2700 1 -- 172.21.2.102:0/3281197395 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fecac00b940 con 0x7fecb40f20e0 2021-02-10T11:24:39.433 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:39.417 7fecbbc88700 1 -- 172.21.2.102:0/3281197395 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fecb40f20e0 msgr2=0x7fecb40f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:39.433 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:39.417 7fecbbc88700 1 --2- 172.21.2.102:0/3281197395 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fecb40f20e0 0x7fecb40f0200 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7fecac006e50 tx=0x7fecac009e70).stop 2021-02-10T11:24:39.433 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:39.418 7fecbbc88700 1 -- 172.21.2.102:0/3281197395 shutdown_connections 2021-02-10T11:24:39.433 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:39.418 7fecbbc88700 1 --2- 172.21.2.102:0/3281197395 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fecb40f20e0 0x7fecb40f0200 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:39.433 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:39.418 7fecbbc88700 1 --2- 172.21.2.102:0/3281197395 >> v2:172.21.2.113:3300/0 conn(0x7fecb41b0470 0x7fecb41b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:39.433 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:39.418 7fecbbc88700 1 -- 172.21.2.102:0/3281197395 >> 172.21.2.102:0/3281197395 conn(0x7fecb40ce3e0 msgr2=0x7fecb40e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:39.434 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:39.418 7fecbbc88700 1 -- 172.21.2.102:0/3281197395 shutdown_connections 2021-02-10T11:24:39.434 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:39.418 7fecbbc88700 1 -- 172.21.2.102:0/3281197395 wait complete. 2021-02-10T11:24:39.435 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:39.433 7fecbbc88700 1 Processor -- start 2021-02-10T11:24:39.436 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:39.433 7fecbbc88700 1 -- start start 2021-02-10T11:24:39.436 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:39.433 7fecbbc88700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fecb409f140 0x7fecb40c1940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:39.436 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:39.433 7fecbbc88700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fecb40c1f60 0x7fecb409e300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:39.436 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:39.433 7fecbbc88700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fecb409e920 0x7fecb40c2960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:39.436 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:39.433 7fecbbc88700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecb40c23e0 con 0x7fecb40c1f60 2021-02-10T11:24:39.436 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:39.433 7fecbbc88700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecb409ed20 con 0x7fecb409f140 2021-02-10T11:24:39.436 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:39.433 7fecbbc88700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fecb4105d10 con 0x7fecb409e920 2021-02-10T11:24:39.436 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:39.434 7fecb92a3700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fecb40c1f60 0x7fecb409e300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:39.436 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:39.434 7fecb92a3700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fecb40c1f60 0x7fecb409e300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:57370/0 (socket says 172.21.2.102:57370) 2021-02-10T11:24:39.436 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:39.434 7fecb92a3700 1 -- 172.21.2.102:0/1647581331 learned_addr learned my addr 172.21.2.102:0/1647581331 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:24:39.436 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:39.434 7fecb92a3700 1 -- 172.21.2.102:0/1647581331 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fecb409e920 msgr2=0x7fecb40c2960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:39.437 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:39.434 7fecb92a3700 1 --2- 172.21.2.102:0/1647581331 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fecb409e920 0x7fecb40c2960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:39.437 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:39.434 7fecb92a3700 1 -- 172.21.2.102:0/1647581331 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fecb409f140 msgr2=0x7fecb40c1940 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:24:39.437 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:39.434 7fecb92a3700 1 --2- 172.21.2.102:0/1647581331 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fecb409f140 0x7fecb40c1940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:39.437 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:39.434 7fecb92a3700 1 -- 172.21.2.102:0/1647581331 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecb41a4910 con 0x7fecb40c1f60 2021-02-10T11:24:39.437 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:39.434 7fecb92a3700 1 --2- 172.21.2.102:0/1647581331 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fecb40c1f60 0x7fecb409e300 secure :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 rx=0x7feca800a4c0 tx=0x7feca800a860).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:39.437 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:39.434 7feca67fc700 1 -- 172.21.2.102:0/1647581331 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7feca8000e50 con 0x7fecb40c1f60 2021-02-10T11:24:39.437 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:39.434 7feca67fc700 1 -- 172.21.2.102:0/1647581331 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feca800ac50 con 0x7fecb40c1f60 2021-02-10T11:24:39.437 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:39.434 7feca67fc700 1 -- 172.21.2.102:0/1647581331 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7feca80045d0 con 0x7fecb40c1f60 2021-02-10T11:24:39.437 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:39.434 7fecbbc88700 1 -- 172.21.2.102:0/1647581331 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fecb41a4910 con 0x7fecb40c1f60 2021-02-10T11:24:39.437 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:39.435 7fecbbc88700 1 -- 172.21.2.102:0/1647581331 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fecb41a4910 con 0x7fecb40c1f60 2021-02-10T11:24:39.437 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:39.435 7feca67fc700 1 -- 172.21.2.102:0/1647581331 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7feca8007f30 con 0x7fecb40c1f60 2021-02-10T11:24:39.437 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:39.435 7feca67fc700 1 --2- 172.21.2.102:0/1647581331 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fec9c018700 0x7fec9c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:39.437 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:39.435 7feca67fc700 1 -- 172.21.2.102:0/1647581331 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5573+0+0 (secure 0 0 0) 0x7feca80325f0 con 0x7fecb40c1f60 2021-02-10T11:24:39.438 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:39.435 7fecb9aa4700 1 --2- 172.21.2.102:0/1647581331 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fec9c018700 0x7fec9c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:39.438 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:39.436 7fecb9aa4700 1 --2- 172.21.2.102:0/1647581331 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fec9c018700 0x7fec9c01abc0 crc :-1 s=READY pgs=642 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:39.465 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:39.466 7fecbbc88700 1 -- 172.21.2.102:0/1647581331 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fecb40f0b50 con 0x7fecb40c1f60 2021-02-10T11:24:39.467 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:39.468 7feca67fc700 1 -- 172.21.2.102:0/1647581331 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7feca8000e50 con 0x7fecb40c1f60 2021-02-10T11:24:39.615 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:39.617 7fecbbc88700 1 -- 172.21.2.102:0/1647581331 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command([{prefix=config set, name=debug_objecter}] v 0) v1 -- 0x7fecb40f0b50 con 0x7fecb40c1f60 2021-02-10T11:24:39.619 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:39.621 7feca67fc700 1 -- 172.21.2.102:0/1647581331 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{prefix=config set, name=debug_objecter}]=0 v10)=0 v10) v1 ==== 122+0+0 (secure 0 0 0) 0x7feca8007f30 con 0x7fecb40c1f60 2021-02-10T11:24:39.624 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:39.625 7fecbbc88700 1 -- 172.21.2.102:0/1647581331 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fec9c018700 msgr2=0x7fec9c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:39.624 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:39.625 7fecbbc88700 1 --2- 172.21.2.102:0/1647581331 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fec9c018700 0x7fec9c01abc0 crc :-1 s=READY pgs=642 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:39.624 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:39.625 7fecbbc88700 1 -- 172.21.2.102:0/1647581331 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fecb40c1f60 msgr2=0x7fecb409e300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:39.624 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:39.625 7fecbbc88700 1 --2- 172.21.2.102:0/1647581331 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fecb40c1f60 0x7fecb409e300 secure :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 rx=0x7feca800a4c0 tx=0x7feca800a860).stop 2021-02-10T11:24:39.625 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:39.626 7fecbbc88700 1 -- 172.21.2.102:0/1647581331 shutdown_connections 2021-02-10T11:24:39.625 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:39.626 7fecbbc88700 1 --2- 172.21.2.102:0/1647581331 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fecb409e920 0x7fecb40c2960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:39.625 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:39.626 7fecbbc88700 1 --2- 172.21.2.102:0/1647581331 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fecb40c1f60 0x7fecb409e300 unknown :-1 s=CLOSED pgs=1388 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:39.625 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:39.626 7fecbbc88700 1 --2- 172.21.2.102:0/1647581331 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fec9c018700 0x7fec9c01abc0 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:39.625 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:39.626 7fecbbc88700 1 --2- 172.21.2.102:0/1647581331 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fecb409f140 0x7fecb40c1940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:39.625 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:39.626 7fecbbc88700 1 -- 172.21.2.102:0/1647581331 >> 172.21.2.102:0/1647581331 conn(0x7fecb409fc10 msgr2=0x7fecb40b3250 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:39.625 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:39.626 7fecbbc88700 1 -- 172.21.2.102:0/1647581331 shutdown_connections 2021-02-10T11:24:39.625 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:39.626 7fecbbc88700 1 -- 172.21.2.102:0/1647581331 wait complete. 2021-02-10T11:24:39.633 DEBUG:teuthology.orchestra.run.gibba002:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2021-02-10T11:24:39.636 DEBUG:teuthology.orchestra.run.gibba013:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2021-02-10T11:24:39.997 INFO:teuthology.orchestra.run.gibba002.stdout:[global] 2021-02-10T11:24:39.997 INFO:teuthology.orchestra.run.gibba002.stdout: chdir = "" 2021-02-10T11:24:39.997 INFO:teuthology.orchestra.run.gibba002.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2021-02-10T11:24:39.997 INFO:teuthology.orchestra.run.gibba002.stdout: auth supported = cephx 2021-02-10T11:24:39.997 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:24:39.997 INFO:teuthology.orchestra.run.gibba002.stdout: filestore xattr use omap = true 2021-02-10T11:24:39.997 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:24:39.997 INFO:teuthology.orchestra.run.gibba002.stdout: mon clock drift allowed = 1.000 2021-02-10T11:24:39.997 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:24:39.997 INFO:teuthology.orchestra.run.gibba002.stdout: osd crush chooseleaf type = 0 2021-02-10T11:24:39.998 INFO:teuthology.orchestra.run.gibba002.stdout: auth debug = true 2021-02-10T11:24:39.998 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:24:39.998 INFO:teuthology.orchestra.run.gibba002.stdout: ms die on old message = true 2021-02-10T11:24:39.998 INFO:teuthology.orchestra.run.gibba002.stdout: ms die on bug = true 2021-02-10T11:24:39.998 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:24:39.998 INFO:teuthology.orchestra.run.gibba002.stdout: mon max pg per osd = 10000 # >= luminous 2021-02-10T11:24:39.998 INFO:teuthology.orchestra.run.gibba002.stdout: mon pg warn max object skew = 0 2021-02-10T11:24:39.998 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:24:39.998 INFO:teuthology.orchestra.run.gibba002.stdout: osd pool default size = 2 2021-02-10T11:24:39.998 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:24:39.998 INFO:teuthology.orchestra.run.gibba002.stdout: mon osd allow primary affinity = true 2021-02-10T11:24:39.998 INFO:teuthology.orchestra.run.gibba002.stdout: mon osd allow pg remap = true 2021-02-10T11:24:39.998 INFO:teuthology.orchestra.run.gibba002.stdout: mon warn on legacy crush tunables = false 2021-02-10T11:24:39.999 INFO:teuthology.orchestra.run.gibba002.stdout: mon warn on crush straw calc version zero = false 2021-02-10T11:24:39.999 INFO:teuthology.orchestra.run.gibba002.stdout: mon warn on no sortbitwise = false 2021-02-10T11:24:39.999 INFO:teuthology.orchestra.run.gibba002.stdout: mon warn on osd down out interval zero = false 2021-02-10T11:24:39.999 INFO:teuthology.orchestra.run.gibba002.stdout: mon warn on too few osds = false 2021-02-10T11:24:39.999 INFO:teuthology.orchestra.run.gibba002.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2021-02-10T11:24:39.999 INFO:teuthology.orchestra.run.gibba002.stdout: mon_warn_on_pool_no_redundancy = false 2021-02-10T11:24:39.999 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:24:39.999 INFO:teuthology.orchestra.run.gibba002.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2021-02-10T11:24:39.999 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:24:39.999 INFO:teuthology.orchestra.run.gibba002.stdout: osd default data pool replay window = 5 2021-02-10T11:24:39.999 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:24:39.999 INFO:teuthology.orchestra.run.gibba002.stdout: mon allow pool delete = true 2021-02-10T11:24:39.999 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:24:40.000 INFO:teuthology.orchestra.run.gibba002.stdout: mon cluster log file level = debug 2021-02-10T11:24:40.000 INFO:teuthology.orchestra.run.gibba002.stdout: debug asserts on shutdown = true 2021-02-10T11:24:40.000 INFO:teuthology.orchestra.run.gibba002.stdout: mon health detail to clog = false 2021-02-10T11:24:40.000 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:24:40.000 INFO:teuthology.orchestra.run.gibba002.stdout: # we see this fail in qa on *nautilus*; bump up retries 2021-02-10T11:24:40.000 INFO:teuthology.orchestra.run.gibba002.stdout: mon_client_directed_command_retry = 4 2021-02-10T11:24:40.000 INFO:teuthology.orchestra.run.gibba002.stdout: mon host = "172.21.2.102,172.21.2.113,[v2:172.21.2.113:3301,v1:172.21.2.113:6790]" 2021-02-10T11:24:40.000 INFO:teuthology.orchestra.run.gibba002.stdout: fsid = 68db30ea-dbce-4aef-a6df-f01c6fea0513 2021-02-10T11:24:40.000 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:24:40.000 INFO:teuthology.orchestra.run.gibba002.stdout:[osd] 2021-02-10T11:24:40.000 INFO:teuthology.orchestra.run.gibba002.stdout: osd journal size = 100 2021-02-10T11:24:40.000 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:24:40.000 INFO:teuthology.orchestra.run.gibba002.stdout: osd scrub load threshold = 5.0 2021-02-10T11:24:40.000 INFO:teuthology.orchestra.run.gibba002.stdout: osd scrub max interval = 600 2021-02-10T11:24:40.001 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:24:40.001 INFO:teuthology.orchestra.run.gibba002.stdout: osd recover clone overlap = true 2021-02-10T11:24:40.001 INFO:teuthology.orchestra.run.gibba002.stdout: osd recovery max chunk = 1048576 2021-02-10T11:24:40.001 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:24:40.001 INFO:teuthology.orchestra.run.gibba002.stdout: osd debug shutdown = true 2021-02-10T11:24:40.001 INFO:teuthology.orchestra.run.gibba002.stdout: osd debug op order = true 2021-02-10T11:24:40.001 INFO:teuthology.orchestra.run.gibba002.stdout: osd debug verify stray on activate = true 2021-02-10T11:24:40.001 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:24:40.001 INFO:teuthology.orchestra.run.gibba002.stdout: osd open classes on start = true 2021-02-10T11:24:40.001 INFO:teuthology.orchestra.run.gibba002.stdout: osd debug pg log writeout = true 2021-02-10T11:24:40.001 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:24:40.001 INFO:teuthology.orchestra.run.gibba002.stdout: osd deep scrub update digest min age = 30 2021-02-10T11:24:40.001 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:24:40.001 INFO:teuthology.orchestra.run.gibba002.stdout: osd map max advance = 10 2021-02-10T11:24:40.002 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:24:40.002 INFO:teuthology.orchestra.run.gibba002.stdout: journal zero on create = true 2021-02-10T11:24:40.002 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:24:40.002 INFO:teuthology.orchestra.run.gibba002.stdout: filestore ondisk finisher threads = 3 2021-02-10T11:24:40.002 INFO:teuthology.orchestra.run.gibba002.stdout: filestore apply finisher threads = 3 2021-02-10T11:24:40.002 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:24:40.002 INFO:teuthology.orchestra.run.gibba002.stdout: bdev debug aio = true 2021-02-10T11:24:40.002 INFO:teuthology.orchestra.run.gibba002.stdout: osd debug misdirected ops = true 2021-02-10T11:24:40.002 INFO:teuthology.orchestra.run.gibba002.stdout: bdev async discard = True 2021-02-10T11:24:40.002 INFO:teuthology.orchestra.run.gibba002.stdout: bdev enable discard = True 2021-02-10T11:24:40.002 INFO:teuthology.orchestra.run.gibba002.stdout: bluestore allocator = bitmap 2021-02-10T11:24:40.002 INFO:teuthology.orchestra.run.gibba002.stdout: bluestore block size = 96636764160 2021-02-10T11:24:40.002 INFO:teuthology.orchestra.run.gibba002.stdout: bluestore fsck on mount = True 2021-02-10T11:24:40.002 INFO:teuthology.orchestra.run.gibba002.stdout: debug bluefs = 20 2021-02-10T11:24:40.003 INFO:teuthology.orchestra.run.gibba002.stdout: debug bluestore = 20 2021-02-10T11:24:40.003 INFO:teuthology.orchestra.run.gibba002.stdout: debug filestore = 20 2021-02-10T11:24:40.003 INFO:teuthology.orchestra.run.gibba002.stdout: debug journal = 20 2021-02-10T11:24:40.003 INFO:teuthology.orchestra.run.gibba002.stdout: debug ms = 20 2021-02-10T11:24:40.003 INFO:teuthology.orchestra.run.gibba002.stdout: debug osd = 25 2021-02-10T11:24:40.003 INFO:teuthology.orchestra.run.gibba002.stdout: debug rocksdb = 10 2021-02-10T11:24:40.003 INFO:teuthology.orchestra.run.gibba002.stdout: mon osd backfillfull_ratio = 0.85 2021-02-10T11:24:40.003 INFO:teuthology.orchestra.run.gibba002.stdout: mon osd full ratio = 0.9 2021-02-10T11:24:40.003 INFO:teuthology.orchestra.run.gibba002.stdout: mon osd nearfull ratio = 0.8 2021-02-10T11:24:40.003 INFO:teuthology.orchestra.run.gibba002.stdout: osd failsafe full ratio = 0.95 2021-02-10T11:24:40.003 INFO:teuthology.orchestra.run.gibba002.stdout: osd objectstore = bluestore 2021-02-10T11:24:40.003 INFO:teuthology.orchestra.run.gibba002.stdout: osd op complaint time = 180 2021-02-10T11:24:40.003 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:24:40.003 INFO:teuthology.orchestra.run.gibba002.stdout:[mgr] 2021-02-10T11:24:40.004 INFO:teuthology.orchestra.run.gibba002.stdout: debug ms = 1 2021-02-10T11:24:40.004 INFO:teuthology.orchestra.run.gibba002.stdout: debug mgr = 20 2021-02-10T11:24:40.004 INFO:teuthology.orchestra.run.gibba002.stdout: debug mon = 20 2021-02-10T11:24:40.004 INFO:teuthology.orchestra.run.gibba002.stdout: debug auth = 20 2021-02-10T11:24:40.004 INFO:teuthology.orchestra.run.gibba002.stdout: mon reweight min pgs per osd = 4 2021-02-10T11:24:40.004 INFO:teuthology.orchestra.run.gibba002.stdout: mon reweight min bytes per osd = 10 2021-02-10T11:24:40.004 INFO:teuthology.orchestra.run.gibba002.stdout: debug client = 10 2021-02-10T11:24:40.004 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:24:40.004 INFO:teuthology.orchestra.run.gibba002.stdout:[mon] 2021-02-10T11:24:40.004 INFO:teuthology.orchestra.run.gibba002.stdout: debug ms = 1 2021-02-10T11:24:40.004 INFO:teuthology.orchestra.run.gibba002.stdout: debug mon = 20 2021-02-10T11:24:40.004 INFO:teuthology.orchestra.run.gibba002.stdout: debug paxos = 20 2021-02-10T11:24:40.004 INFO:teuthology.orchestra.run.gibba002.stdout: debug auth = 20 2021-02-10T11:24:40.004 INFO:teuthology.orchestra.run.gibba002.stdout: mon data avail warn = 5 2021-02-10T11:24:40.005 INFO:teuthology.orchestra.run.gibba002.stdout: mon mgr mkfs grace = 240 2021-02-10T11:24:40.005 INFO:teuthology.orchestra.run.gibba002.stdout: mon reweight min pgs per osd = 4 2021-02-10T11:24:40.005 INFO:teuthology.orchestra.run.gibba002.stdout: mon osd reporter subtree level = osd 2021-02-10T11:24:40.005 INFO:teuthology.orchestra.run.gibba002.stdout: mon osd prime pg temp = true 2021-02-10T11:24:40.005 INFO:teuthology.orchestra.run.gibba002.stdout: mon reweight min bytes per osd = 10 2021-02-10T11:24:40.005 INFO:teuthology.orchestra.run.gibba002.stdout: mon op complaint time = 120 2021-02-10T11:24:40.005 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:24:40.005 INFO:teuthology.orchestra.run.gibba002.stdout:[client] 2021-02-10T11:24:40.005 INFO:teuthology.orchestra.run.gibba002.stdout: rgw cache enabled = true 2021-02-10T11:24:40.005 INFO:teuthology.orchestra.run.gibba002.stdout: rgw enable ops log = true 2021-02-10T11:24:40.006 INFO:teuthology.orchestra.run.gibba002.stdout: rgw enable usage log = true 2021-02-10T11:24:40.006 INFO:teuthology.orchestra.run.gibba002.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2021-02-10T11:24:40.006 INFO:teuthology.orchestra.run.gibba002.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2021-02-10T11:24:40.006 INFO:teuthology.orchestra.run.gibba002.stdout: client mount timeout = 600 2021-02-10T11:24:40.006 INFO:teuthology.orchestra.run.gibba002.stdout: debug client = 20 2021-02-10T11:24:40.006 INFO:teuthology.orchestra.run.gibba002.stdout: debug ms = 1 2021-02-10T11:24:40.006 INFO:teuthology.orchestra.run.gibba002.stdout: pid file = "" 2021-02-10T11:24:40.006 INFO:teuthology.orchestra.run.gibba002.stdout:[mon.a] 2021-02-10T11:24:40.006 INFO:teuthology.orchestra.run.gibba002.stdout:[mon.b] 2021-02-10T11:24:40.007 INFO:teuthology.orchestra.run.gibba002.stdout:[mon.c] 2021-02-10T11:24:40.007 INFO:teuthology.orchestra.run.gibba002.stdout:[mds.a] 2021-02-10T11:24:40.007 INFO:teuthology.orchestra.run.gibba002.stdout:[mds.b] 2021-02-10T11:24:40.007 INFO:teuthology.orchestra.run.gibba002.stdout:[mds] 2021-02-10T11:24:40.007 INFO:teuthology.orchestra.run.gibba002.stdout: debug mds = 20 2021-02-10T11:24:40.007 INFO:teuthology.orchestra.run.gibba002.stdout: debug ms = 1 2021-02-10T11:24:40.007 INFO:teuthology.orchestra.run.gibba002.stdout: mds bal frag = True 2021-02-10T11:24:40.007 INFO:teuthology.orchestra.run.gibba002.stdout: mds bal fragment size max = 10000 2021-02-10T11:24:40.007 INFO:teuthology.orchestra.run.gibba002.stdout: mds bal merge size = 5 2021-02-10T11:24:40.007 INFO:teuthology.orchestra.run.gibba002.stdout: mds bal split bits = 3 2021-02-10T11:24:40.007 INFO:teuthology.orchestra.run.gibba002.stdout: mds bal split size = 100 2021-02-10T11:24:40.007 INFO:teuthology.orchestra.run.gibba002.stdout: mds debug frag = True 2021-02-10T11:24:40.007 INFO:teuthology.orchestra.run.gibba002.stdout: mds debug scatterstat = True 2021-02-10T11:24:40.008 INFO:teuthology.orchestra.run.gibba002.stdout: mds op complaint time = 180 2021-02-10T11:24:40.008 INFO:teuthology.orchestra.run.gibba002.stdout: mds verify scatter = True 2021-02-10T11:24:40.008 INFO:teuthology.orchestra.run.gibba002.stdout: osd op complaint time = 180 2021-02-10T11:24:40.008 INFO:teuthology.orchestra.run.gibba002.stdout:[client.alice] 2021-02-10T11:24:40.008 INFO:teuthology.orchestra.run.gibba002.stdout: keyring = /etc/ceph/ceph.client.alice.keyring 2021-02-10T11:24:40.009 INFO:teuthology.orchestra.run.gibba013.stdout:[global] 2021-02-10T11:24:40.009 INFO:teuthology.orchestra.run.gibba013.stdout: chdir = "" 2021-02-10T11:24:40.009 INFO:teuthology.orchestra.run.gibba013.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2021-02-10T11:24:40.009 INFO:teuthology.orchestra.run.gibba013.stdout: auth supported = cephx 2021-02-10T11:24:40.009 INFO:teuthology.orchestra.run.gibba013.stdout: 2021-02-10T11:24:40.009 INFO:teuthology.orchestra.run.gibba013.stdout: filestore xattr use omap = true 2021-02-10T11:24:40.009 INFO:teuthology.orchestra.run.gibba013.stdout: 2021-02-10T11:24:40.009 INFO:teuthology.orchestra.run.gibba013.stdout: mon clock drift allowed = 1.000 2021-02-10T11:24:40.009 INFO:teuthology.orchestra.run.gibba013.stdout: 2021-02-10T11:24:40.009 INFO:teuthology.orchestra.run.gibba013.stdout: osd crush chooseleaf type = 0 2021-02-10T11:24:40.010 INFO:teuthology.orchestra.run.gibba013.stdout: auth debug = true 2021-02-10T11:24:40.010 INFO:teuthology.orchestra.run.gibba013.stdout: 2021-02-10T11:24:40.010 INFO:teuthology.orchestra.run.gibba013.stdout: ms die on old message = true 2021-02-10T11:24:40.010 INFO:teuthology.orchestra.run.gibba013.stdout: ms die on bug = true 2021-02-10T11:24:40.010 INFO:teuthology.orchestra.run.gibba013.stdout: 2021-02-10T11:24:40.010 INFO:teuthology.orchestra.run.gibba013.stdout: mon max pg per osd = 10000 # >= luminous 2021-02-10T11:24:40.010 INFO:teuthology.orchestra.run.gibba013.stdout: mon pg warn max object skew = 0 2021-02-10T11:24:40.010 INFO:teuthology.orchestra.run.gibba013.stdout: 2021-02-10T11:24:40.010 INFO:teuthology.orchestra.run.gibba013.stdout: osd pool default size = 2 2021-02-10T11:24:40.010 INFO:teuthology.orchestra.run.gibba013.stdout: 2021-02-10T11:24:40.010 INFO:teuthology.orchestra.run.gibba013.stdout: mon osd allow primary affinity = true 2021-02-10T11:24:40.010 INFO:teuthology.orchestra.run.gibba013.stdout: mon osd allow pg remap = true 2021-02-10T11:24:40.010 INFO:teuthology.orchestra.run.gibba013.stdout: mon warn on legacy crush tunables = false 2021-02-10T11:24:40.010 INFO:teuthology.orchestra.run.gibba013.stdout: mon warn on crush straw calc version zero = false 2021-02-10T11:24:40.011 INFO:teuthology.orchestra.run.gibba013.stdout: mon warn on no sortbitwise = false 2021-02-10T11:24:40.011 INFO:teuthology.orchestra.run.gibba013.stdout: mon warn on osd down out interval zero = false 2021-02-10T11:24:40.011 INFO:teuthology.orchestra.run.gibba013.stdout: mon warn on too few osds = false 2021-02-10T11:24:40.011 INFO:teuthology.orchestra.run.gibba013.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2021-02-10T11:24:40.011 INFO:teuthology.orchestra.run.gibba013.stdout: mon_warn_on_pool_no_redundancy = false 2021-02-10T11:24:40.011 INFO:teuthology.orchestra.run.gibba013.stdout: 2021-02-10T11:24:40.011 INFO:teuthology.orchestra.run.gibba013.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2021-02-10T11:24:40.011 INFO:teuthology.orchestra.run.gibba013.stdout: 2021-02-10T11:24:40.011 INFO:teuthology.orchestra.run.gibba013.stdout: osd default data pool replay window = 5 2021-02-10T11:24:40.011 INFO:teuthology.orchestra.run.gibba013.stdout: 2021-02-10T11:24:40.011 INFO:teuthology.orchestra.run.gibba013.stdout: mon allow pool delete = true 2021-02-10T11:24:40.011 INFO:teuthology.orchestra.run.gibba013.stdout: 2021-02-10T11:24:40.011 INFO:teuthology.orchestra.run.gibba013.stdout: mon cluster log file level = debug 2021-02-10T11:24:40.011 INFO:teuthology.orchestra.run.gibba013.stdout: debug asserts on shutdown = true 2021-02-10T11:24:40.012 INFO:teuthology.orchestra.run.gibba013.stdout: mon health detail to clog = false 2021-02-10T11:24:40.012 INFO:teuthology.orchestra.run.gibba013.stdout: 2021-02-10T11:24:40.012 INFO:teuthology.orchestra.run.gibba013.stdout: # we see this fail in qa on *nautilus*; bump up retries 2021-02-10T11:24:40.012 INFO:teuthology.orchestra.run.gibba013.stdout: mon_client_directed_command_retry = 4 2021-02-10T11:24:40.012 INFO:teuthology.orchestra.run.gibba013.stdout: mon host = "172.21.2.102,172.21.2.113,[v2:172.21.2.113:3301,v1:172.21.2.113:6790]" 2021-02-10T11:24:40.012 INFO:teuthology.orchestra.run.gibba013.stdout: fsid = 68db30ea-dbce-4aef-a6df-f01c6fea0513 2021-02-10T11:24:40.012 INFO:teuthology.orchestra.run.gibba013.stdout: 2021-02-10T11:24:40.012 INFO:teuthology.orchestra.run.gibba013.stdout:[osd] 2021-02-10T11:24:40.012 INFO:teuthology.orchestra.run.gibba013.stdout: osd journal size = 100 2021-02-10T11:24:40.012 INFO:teuthology.orchestra.run.gibba013.stdout: 2021-02-10T11:24:40.012 INFO:teuthology.orchestra.run.gibba013.stdout: osd scrub load threshold = 5.0 2021-02-10T11:24:40.012 INFO:teuthology.orchestra.run.gibba013.stdout: osd scrub max interval = 600 2021-02-10T11:24:40.012 INFO:teuthology.orchestra.run.gibba013.stdout: 2021-02-10T11:24:40.012 INFO:teuthology.orchestra.run.gibba013.stdout: osd recover clone overlap = true 2021-02-10T11:24:40.013 INFO:teuthology.orchestra.run.gibba013.stdout: osd recovery max chunk = 1048576 2021-02-10T11:24:40.013 INFO:teuthology.orchestra.run.gibba013.stdout: 2021-02-10T11:24:40.013 INFO:teuthology.orchestra.run.gibba013.stdout: osd debug shutdown = true 2021-02-10T11:24:40.013 INFO:teuthology.orchestra.run.gibba013.stdout: osd debug op order = true 2021-02-10T11:24:40.013 INFO:teuthology.orchestra.run.gibba013.stdout: osd debug verify stray on activate = true 2021-02-10T11:24:40.013 INFO:teuthology.orchestra.run.gibba013.stdout: 2021-02-10T11:24:40.013 INFO:teuthology.orchestra.run.gibba013.stdout: osd open classes on start = true 2021-02-10T11:24:40.013 INFO:teuthology.orchestra.run.gibba013.stdout: osd debug pg log writeout = true 2021-02-10T11:24:40.013 INFO:teuthology.orchestra.run.gibba013.stdout: 2021-02-10T11:24:40.013 INFO:teuthology.orchestra.run.gibba013.stdout: osd deep scrub update digest min age = 30 2021-02-10T11:24:40.013 INFO:teuthology.orchestra.run.gibba013.stdout: 2021-02-10T11:24:40.013 INFO:teuthology.orchestra.run.gibba013.stdout: osd map max advance = 10 2021-02-10T11:24:40.013 INFO:teuthology.orchestra.run.gibba013.stdout: 2021-02-10T11:24:40.014 INFO:teuthology.orchestra.run.gibba013.stdout: journal zero on create = true 2021-02-10T11:24:40.014 INFO:teuthology.orchestra.run.gibba013.stdout: 2021-02-10T11:24:40.014 INFO:teuthology.orchestra.run.gibba013.stdout: filestore ondisk finisher threads = 3 2021-02-10T11:24:40.014 INFO:teuthology.orchestra.run.gibba013.stdout: filestore apply finisher threads = 3 2021-02-10T11:24:40.014 INFO:teuthology.orchestra.run.gibba013.stdout: 2021-02-10T11:24:40.014 INFO:teuthology.orchestra.run.gibba013.stdout: bdev debug aio = true 2021-02-10T11:24:40.014 INFO:teuthology.orchestra.run.gibba013.stdout: osd debug misdirected ops = true 2021-02-10T11:24:40.014 INFO:teuthology.orchestra.run.gibba013.stdout: bdev async discard = True 2021-02-10T11:24:40.015 INFO:teuthology.orchestra.run.gibba013.stdout: bdev enable discard = True 2021-02-10T11:24:40.015 INFO:teuthology.orchestra.run.gibba013.stdout: bluestore allocator = bitmap 2021-02-10T11:24:40.015 INFO:teuthology.orchestra.run.gibba013.stdout: bluestore block size = 96636764160 2021-02-10T11:24:40.015 INFO:teuthology.orchestra.run.gibba013.stdout: bluestore fsck on mount = True 2021-02-10T11:24:40.015 INFO:teuthology.orchestra.run.gibba013.stdout: debug bluefs = 20 2021-02-10T11:24:40.015 INFO:teuthology.orchestra.run.gibba013.stdout: debug bluestore = 20 2021-02-10T11:24:40.015 INFO:teuthology.orchestra.run.gibba013.stdout: debug filestore = 20 2021-02-10T11:24:40.015 INFO:teuthology.orchestra.run.gibba013.stdout: debug journal = 20 2021-02-10T11:24:40.015 INFO:teuthology.orchestra.run.gibba013.stdout: debug ms = 20 2021-02-10T11:24:40.015 INFO:teuthology.orchestra.run.gibba013.stdout: debug osd = 25 2021-02-10T11:24:40.015 INFO:teuthology.orchestra.run.gibba013.stdout: debug rocksdb = 10 2021-02-10T11:24:40.015 INFO:teuthology.orchestra.run.gibba013.stdout: mon osd backfillfull_ratio = 0.85 2021-02-10T11:24:40.015 INFO:teuthology.orchestra.run.gibba013.stdout: mon osd full ratio = 0.9 2021-02-10T11:24:40.015 INFO:teuthology.orchestra.run.gibba013.stdout: mon osd nearfull ratio = 0.8 2021-02-10T11:24:40.016 INFO:teuthology.orchestra.run.gibba013.stdout: osd failsafe full ratio = 0.95 2021-02-10T11:24:40.016 INFO:teuthology.orchestra.run.gibba013.stdout: osd objectstore = bluestore 2021-02-10T11:24:40.016 INFO:teuthology.orchestra.run.gibba013.stdout: osd op complaint time = 180 2021-02-10T11:24:40.016 INFO:teuthology.orchestra.run.gibba013.stdout: 2021-02-10T11:24:40.016 INFO:teuthology.orchestra.run.gibba013.stdout:[mgr] 2021-02-10T11:24:40.016 INFO:teuthology.orchestra.run.gibba013.stdout: debug ms = 1 2021-02-10T11:24:40.016 INFO:teuthology.orchestra.run.gibba013.stdout: debug mgr = 20 2021-02-10T11:24:40.016 INFO:teuthology.orchestra.run.gibba013.stdout: debug mon = 20 2021-02-10T11:24:40.016 INFO:teuthology.orchestra.run.gibba013.stdout: debug auth = 20 2021-02-10T11:24:40.016 INFO:teuthology.orchestra.run.gibba013.stdout: mon reweight min pgs per osd = 4 2021-02-10T11:24:40.016 INFO:teuthology.orchestra.run.gibba013.stdout: mon reweight min bytes per osd = 10 2021-02-10T11:24:40.016 INFO:teuthology.orchestra.run.gibba013.stdout: debug client = 10 2021-02-10T11:24:40.016 INFO:teuthology.orchestra.run.gibba013.stdout: 2021-02-10T11:24:40.016 INFO:teuthology.orchestra.run.gibba013.stdout:[mon] 2021-02-10T11:24:40.017 INFO:teuthology.orchestra.run.gibba013.stdout: debug ms = 1 2021-02-10T11:24:40.017 INFO:teuthology.orchestra.run.gibba013.stdout: debug mon = 20 2021-02-10T11:24:40.017 INFO:teuthology.orchestra.run.gibba013.stdout: debug paxos = 20 2021-02-10T11:24:40.017 INFO:teuthology.orchestra.run.gibba013.stdout: debug auth = 20 2021-02-10T11:24:40.017 INFO:teuthology.orchestra.run.gibba013.stdout: mon data avail warn = 5 2021-02-10T11:24:40.017 INFO:teuthology.orchestra.run.gibba013.stdout: mon mgr mkfs grace = 240 2021-02-10T11:24:40.017 INFO:teuthology.orchestra.run.gibba013.stdout: mon reweight min pgs per osd = 4 2021-02-10T11:24:40.017 INFO:teuthology.orchestra.run.gibba013.stdout: mon osd reporter subtree level = osd 2021-02-10T11:24:40.017 INFO:teuthology.orchestra.run.gibba013.stdout: mon osd prime pg temp = true 2021-02-10T11:24:40.017 INFO:teuthology.orchestra.run.gibba013.stdout: mon reweight min bytes per osd = 10 2021-02-10T11:24:40.017 INFO:teuthology.orchestra.run.gibba013.stdout: mon op complaint time = 120 2021-02-10T11:24:40.017 INFO:teuthology.orchestra.run.gibba013.stdout: 2021-02-10T11:24:40.017 INFO:teuthology.orchestra.run.gibba013.stdout:[client] 2021-02-10T11:24:40.017 INFO:teuthology.orchestra.run.gibba013.stdout: rgw cache enabled = true 2021-02-10T11:24:40.018 INFO:teuthology.orchestra.run.gibba013.stdout: rgw enable ops log = true 2021-02-10T11:24:40.018 INFO:teuthology.orchestra.run.gibba013.stdout: rgw enable usage log = true 2021-02-10T11:24:40.018 INFO:teuthology.orchestra.run.gibba013.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2021-02-10T11:24:40.018 INFO:teuthology.orchestra.run.gibba013.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2021-02-10T11:24:40.018 INFO:teuthology.orchestra.run.gibba013.stdout: client mount timeout = 600 2021-02-10T11:24:40.018 INFO:teuthology.orchestra.run.gibba013.stdout: debug client = 20 2021-02-10T11:24:40.018 INFO:teuthology.orchestra.run.gibba013.stdout: debug ms = 1 2021-02-10T11:24:40.018 INFO:teuthology.orchestra.run.gibba013.stdout: pid file = "" 2021-02-10T11:24:40.018 INFO:teuthology.orchestra.run.gibba013.stdout:[mon.a] 2021-02-10T11:24:40.018 INFO:teuthology.orchestra.run.gibba013.stdout:[mon.b] 2021-02-10T11:24:40.018 INFO:teuthology.orchestra.run.gibba013.stdout:[mon.c] 2021-02-10T11:24:40.018 INFO:teuthology.orchestra.run.gibba013.stdout:[mds.a] 2021-02-10T11:24:40.018 INFO:teuthology.orchestra.run.gibba013.stdout:[mds.b] 2021-02-10T11:24:40.019 INFO:teuthology.orchestra.run.gibba013.stdout:[mds] 2021-02-10T11:24:40.019 INFO:teuthology.orchestra.run.gibba013.stdout: debug mds = 20 2021-02-10T11:24:40.019 INFO:teuthology.orchestra.run.gibba013.stdout: debug ms = 1 2021-02-10T11:24:40.019 INFO:teuthology.orchestra.run.gibba013.stdout: mds bal frag = True 2021-02-10T11:24:40.019 INFO:teuthology.orchestra.run.gibba013.stdout: mds bal fragment size max = 10000 2021-02-10T11:24:40.019 INFO:teuthology.orchestra.run.gibba013.stdout: mds bal merge size = 5 2021-02-10T11:24:40.019 INFO:teuthology.orchestra.run.gibba013.stdout: mds bal split bits = 3 2021-02-10T11:24:40.019 INFO:teuthology.orchestra.run.gibba013.stdout: mds bal split size = 100 2021-02-10T11:24:40.019 INFO:teuthology.orchestra.run.gibba013.stdout: mds debug frag = True 2021-02-10T11:24:40.019 INFO:teuthology.orchestra.run.gibba013.stdout: mds debug scatterstat = True 2021-02-10T11:24:40.019 INFO:teuthology.orchestra.run.gibba013.stdout: mds op complaint time = 180 2021-02-10T11:24:40.019 INFO:teuthology.orchestra.run.gibba013.stdout: mds verify scatter = True 2021-02-10T11:24:40.019 INFO:teuthology.orchestra.run.gibba013.stdout: osd op complaint time = 180 2021-02-10T11:24:40.020 INFO:teuthology.orchestra.run.gibba013.stdout:[client.alice] 2021-02-10T11:24:40.020 INFO:teuthology.orchestra.run.gibba013.stdout: keyring = /etc/ceph/ceph.client.alice.keyring 2021-02-10T11:24:40.021 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:24:40.448 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:40.432 7fb126a2d700 1 Processor -- start 2021-02-10T11:24:40.448 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:40.432 7fb126a2d700 1 -- start start 2021-02-10T11:24:40.448 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:40.433 7fb126a2d700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fb1200e53e0 0x7fb1201b0550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:40.448 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:40.433 7fb126a2d700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fb1201b0b70 0x7fb120081910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:40.449 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:40.433 7fb126a2d700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb120081f30 0x7fb120082330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:40.449 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:40.433 7fb126a2d700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1200e5fe0 con 0x7fb1201b0b70 2021-02-10T11:24:40.449 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:40.433 7fb126a2d700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb120082d50 con 0x7fb1200e53e0 2021-02-10T11:24:40.449 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:40.433 7fb126a2d700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1200e57e0 con 0x7fb120081f30 2021-02-10T11:24:40.449 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:40.433 7fb11ffff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fb1201b0b70 0x7fb120081910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:40.449 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:40.433 7fb11ffff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fb1201b0b70 0x7fb120081910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:57378/0 (socket says 172.21.2.102:57378) 2021-02-10T11:24:40.449 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:40.433 7fb11ffff700 1 -- 172.21.2.102:0/3945788161 learned_addr learned my addr 172.21.2.102:0/3945788161 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:24:40.449 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:40.433 7fb11ffff700 1 -- 172.21.2.102:0/3945788161 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb120081f30 msgr2=0x7fb120082330 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:24:40.449 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:40.433 7fb11ffff700 1 --2- 172.21.2.102:0/3945788161 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb120081f30 0x7fb120082330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:40.449 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:40.433 7fb11ffff700 1 -- 172.21.2.102:0/3945788161 >> v2:172.21.2.113:3300/0 conn(0x7fb1200e53e0 msgr2=0x7fb1201b0550 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:24:40.449 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:40.433 7fb11ffff700 1 --2- 172.21.2.102:0/3945788161 >> v2:172.21.2.113:3300/0 conn(0x7fb1200e53e0 0x7fb1201b0550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:40.449 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:40.433 7fb11ffff700 1 -- 172.21.2.102:0/3945788161 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb120082950 con 0x7fb1201b0b70 2021-02-10T11:24:40.449 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:40.433 7fb11ffff700 1 --2- 172.21.2.102:0/3945788161 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb1201b0b70 0x7fb120081910 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 rx=0x7fb110006700 tx=0x7fb110009b80).ready entity=mon.0 client_cookie=f8a29f8796a844d7 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:40.450 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:40.434 7fb11f7fe700 1 -- 172.21.2.102:0/3945788161 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb11000c0c0 con 0x7fb1201b0b70 2021-02-10T11:24:40.450 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:40.434 7fb11f7fe700 1 -- 172.21.2.102:0/3945788161 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb11000c400 con 0x7fb1201b0b70 2021-02-10T11:24:40.450 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:40.434 7fb11f7fe700 1 -- 172.21.2.102:0/3945788161 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb11000c920 con 0x7fb1201b0b70 2021-02-10T11:24:40.450 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:40.434 7fb126a2d700 1 -- 172.21.2.102:0/3945788161 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb1201b0b70 msgr2=0x7fb120081910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:40.450 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:40.434 7fb126a2d700 1 --2- 172.21.2.102:0/3945788161 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb1201b0b70 0x7fb120081910 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 rx=0x7fb110006700 tx=0x7fb110009b80).stop 2021-02-10T11:24:40.450 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:40.434 7fb126a2d700 1 -- 172.21.2.102:0/3945788161 shutdown_connections 2021-02-10T11:24:40.450 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:40.434 7fb126a2d700 1 --2- 172.21.2.102:0/3945788161 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb120081f30 0x7fb120082330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:40.450 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:40.434 7fb126a2d700 1 --2- 172.21.2.102:0/3945788161 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb1201b0b70 0x7fb120081910 unknown :-1 s=CLOSED pgs=1389 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:40.450 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:40.434 7fb126a2d700 1 --2- 172.21.2.102:0/3945788161 >> v2:172.21.2.113:3300/0 conn(0x7fb1200e53e0 0x7fb1201b0550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:40.450 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:40.434 7fb126a2d700 1 -- 172.21.2.102:0/3945788161 >> 172.21.2.102:0/3945788161 conn(0x7fb1200ce500 msgr2=0x7fb1200e4610 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:40.450 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:40.435 7fb126a2d700 1 -- 172.21.2.102:0/3945788161 shutdown_connections 2021-02-10T11:24:40.450 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:40.435 7fb126a2d700 1 -- 172.21.2.102:0/3945788161 wait complete. 2021-02-10T11:24:40.450 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:40.449 7fb126a2d700 1 Processor -- start 2021-02-10T11:24:40.451 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:40.449 7fb126a2d700 1 -- start start 2021-02-10T11:24:40.452 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:40.450 7fb126a2d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb1201b1fb0 0x7fb12019b400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:40.452 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:40.450 7fb126a2d700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb1201b23b0 0x7fb12019ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:40.452 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:40.450 7fb126a2d700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb12019c750 0x7fb12019cb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:40.452 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:40.450 7fb126a2d700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb12019dcd0 con 0x7fb1201b1fb0 2021-02-10T11:24:40.452 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:40.450 7fb126a2d700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1201b27e0 con 0x7fb12019c750 2021-02-10T11:24:40.452 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:40.450 7fb126a2d700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb12019c240 con 0x7fb1201b23b0 2021-02-10T11:24:40.452 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:40.450 7fb124849700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb1201b1fb0 0x7fb12019b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:40.453 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:40.450 7fb124849700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb1201b1fb0 0x7fb12019b400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:57388/0 (socket says 172.21.2.102:57388) 2021-02-10T11:24:40.453 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:40.450 7fb12504a700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb12019c750 0x7fb12019cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:40.453 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:40.450 7fb124849700 1 -- 172.21.2.102:0/894459981 learned_addr learned my addr 172.21.2.102:0/894459981 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:24:40.453 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:40.450 7fb11ffff700 1 --2- 172.21.2.102:0/894459981 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb1201b23b0 0x7fb12019ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:40.453 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:40.451 7fb124849700 1 -- 172.21.2.102:0/894459981 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb1201b23b0 msgr2=0x7fb12019ba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:40.453 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:40.451 7fb124849700 1 --2- 172.21.2.102:0/894459981 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb1201b23b0 0x7fb12019ba20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:40.453 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:40.451 7fb124849700 1 -- 172.21.2.102:0/894459981 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb12019c750 msgr2=0x7fb12019cb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:40.453 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:40.451 7fb124849700 1 --2- 172.21.2.102:0/894459981 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb12019c750 0x7fb12019cb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:40.453 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:40.451 7fb124849700 1 -- 172.21.2.102:0/894459981 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb12019d1a0 con 0x7fb1201b1fb0 2021-02-10T11:24:40.453 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:40.451 7fb124849700 1 --2- 172.21.2.102:0/894459981 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb1201b1fb0 0x7fb12019b400 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 rx=0x7fb10800b5c0 tx=0x7fb10800b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:40.453 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:40.451 7fb11d7fa700 1 -- 172.21.2.102:0/894459981 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb10800bd50 con 0x7fb1201b1fb0 2021-02-10T11:24:40.453 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:40.451 7fb11d7fa700 1 -- 172.21.2.102:0/894459981 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb108006030 con 0x7fb1201b1fb0 2021-02-10T11:24:40.453 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:40.451 7fb11d7fa700 1 -- 172.21.2.102:0/894459981 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb1080054d0 con 0x7fb1201b1fb0 2021-02-10T11:24:40.454 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:40.451 7fb126a2d700 1 -- 172.21.2.102:0/894459981 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb12019d1a0 con 0x7fb1201b1fb0 2021-02-10T11:24:40.454 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:40.451 7fb126a2d700 1 -- 172.21.2.102:0/894459981 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb12019d1a0 con 0x7fb1201b1fb0 2021-02-10T11:24:40.454 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:40.452 7fb11d7fa700 1 -- 172.21.2.102:0/894459981 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fb108006210 con 0x7fb1201b1fb0 2021-02-10T11:24:40.454 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:40.452 7fb11d7fa700 1 --2- 172.21.2.102:0/894459981 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb1040186c0 0x7fb10401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:40.454 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:40.452 7fb11d7fa700 1 -- 172.21.2.102:0/894459981 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fb108032580 con 0x7fb1201b1fb0 2021-02-10T11:24:40.454 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:40.452 7fb11ffff700 1 --2- 172.21.2.102:0/894459981 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb1040186c0 0x7fb10401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:40.454 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:40.452 7fb11ffff700 1 --2- 172.21.2.102:0/894459981 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb1040186c0 0x7fb10401abc0 crc :-1 s=READY pgs=643 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:40.481 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:40.483 7fb126a2d700 1 -- 172.21.2.102:0/894459981 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1200ed660 con 0x7fb1201b1fb0 2021-02-10T11:24:40.484 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:40.485 7fb11d7fa700 1 -- 172.21.2.102:0/894459981 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fb108006b80 con 0x7fb1201b1fb0 2021-02-10T11:24:40.610 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:40.611 7fb126a2d700 1 -- 172.21.2.102:0/894459981 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb120082d50 con 0x7fb1201b1fb0 2021-02-10T11:24:40.610 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:40.612 7fb11d7fa700 1 -- 172.21.2.102:0/894459981 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v57) v1 ==== 74+0+13996 (secure 0 0 0) 0x7fb1080060f0 con 0x7fb1201b1fb0 2021-02-10T11:24:40.611 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:24:40.611 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":57,"fsid":"68db30ea-dbce-4aef-a6df-f01c6fea0513","created":"2021-02-10 11:03:06.331557","modified":"2021-02-10 11:24:11.200509","last_up_change":"2021-02-10 11:03:32.690562","last_in_change":"2021-02-10 11:03:32.690562","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:03:37.322519","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:24:08.122188","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":11,"pool_name":"cephfs_data","create_time":"2021-02-10 11:24:09.191606","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"be691b4d-a039-408a-af80-9d53f02b6f1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6818","nonce":36088},{"type":"v1","addr":"172.21.2.102:6820","nonce":36088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6822","nonce":36088},{"type":"v1","addr":"172.21.2.102:6824","nonce":36088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6830","nonce":36088},{"type":"v1","addr":"172.21.2.102:6832","nonce":36088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6826","nonce":36088},{"type":"v1","addr":"172.21.2.102:6828","nonce":36088}]},"public_addr":"172.21.2.102:6820/36088","cluster_addr":"172.21.2.102:6824/36088","heartbeat_back_addr":"172.21.2.102:6832/36088","heartbeat_front_addr":"172.21.2.102:6828/36088","state":["exists","up"]},{"osd":1,"uuid":"64e3ccc8-9c69-4d7c-865e-af8e83822635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6819","nonce":36087},{"type":"v1","addr":"172.21.2.102:6821","nonce":36087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6823","nonce":36087},{"type":"v1","addr":"172.21.2.102:6825","nonce":36087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6831","nonce":36087},{"type":"v1","addr":"172.21.2.102:6833","nonce":36087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6827","nonce":36087},{"type":"v1","addr":"172.21.2.102:6829","nonce":36087}]},"public_addr":"172.21.2.102:6821/36087","cluster_addr":"172.21.2.102:6825/36087","heartbeat_back_addr":"172.21.2.102:6833/36087","heartbeat_front_addr":"172.21.2.102:6829/36087","state":["exists","up"]},{"osd":2,"uuid":"153700af-747f-481f-bdf4-d6072d3a842f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6810","nonce":36089},{"type":"v1","addr":"172.21.2.102:6811","nonce":36089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6812","nonce":36089},{"type":"v1","addr":"172.21.2.102:6813","nonce":36089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6816","nonce":36089},{"type":"v1","addr":"172.21.2.102:6817","nonce":36089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6814","nonce":36089},{"type":"v1","addr":"172.21.2.102:6815","nonce":36089}]},"public_addr":"172.21.2.102:6811/36089","cluster_addr":"172.21.2.102:6813/36089","heartbeat_back_addr":"172.21.2.102:6817/36089","heartbeat_front_addr":"172.21.2.102:6815/36089","state":["exists","up"]},{"osd":3,"uuid":"cbc678f4-83a9-45e6-8830-2f7c232b1ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6802","nonce":36090},{"type":"v1","addr":"172.21.2.102:6803","nonce":36090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6804","nonce":36090},{"type":"v1","addr":"172.21.2.102:6805","nonce":36090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6808","nonce":36090},{"type":"v1","addr":"172.21.2.102:6809","nonce":36090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6806","nonce":36090},{"type":"v1","addr":"172.21.2.102:6807","nonce":36090}]},"public_addr":"172.21.2.102:6803/36090","cluster_addr":"172.21.2.102:6805/36090","heartbeat_back_addr":"172.21.2.102:6809/36090","heartbeat_front_addr":"172.21.2.102:6807/36090","state":["exists","up"]},{"osd":4,"uuid":"c01e4008-b258-4f10-afd2-9e7f37214351","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6808","nonce":36368},{"type":"v1","addr":"172.21.2.113:6811","nonce":36368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6814","nonce":36368},{"type":"v1","addr":"172.21.2.113:6818","nonce":36368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6826","nonce":36368},{"type":"v1","addr":"172.21.2.113:6828","nonce":36368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6820","nonce":36368},{"type":"v1","addr":"172.21.2.113:6824","nonce":36368}]},"public_addr":"172.21.2.113:6811/36368","cluster_addr":"172.21.2.113:6818/36368","heartbeat_back_addr":"172.21.2.113:6828/36368","heartbeat_front_addr":"172.21.2.113:6824/36368","state":["exists","up"]},{"osd":5,"uuid":"551f6007-daf8-4a98-8ee0-c2b16bcccb5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6802","nonce":36366},{"type":"v1","addr":"172.21.2.113:6805","nonce":36366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6807","nonce":36366},{"type":"v1","addr":"172.21.2.113:6810","nonce":36366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6819","nonce":36366},{"type":"v1","addr":"172.21.2.113:6822","nonce":36366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6812","nonce":36366},{"type":"v1","addr":"172.21.2.113:6815","nonce":36366}]},"public_addr":"172.21.2.113:6805/36366","cluster_addr":"172.21.2.113:6810/36366","heartbeat_back_addr":"172.21.2.113:6822/36366","heartbeat_front_addr":"172.21.2.113:6815/36366","state":["exists","up"]},{"osd":6,"uuid":"bf67c040-0e50-48be-b102-c3943bc9ebef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6800","nonce":36367},{"type":"v1","addr":"172.21.2.113:6801","nonce":36367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6803","nonce":36367},{"type":"v1","addr":"172.21.2.113:6804","nonce":36367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6813","nonce":36367},{"type":"v1","addr":"172.21.2.113:6816","nonce":36367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6806","nonce":36367},{"type":"v1","addr":"172.21.2.113:6809","nonce":36367}]},"public_addr":"172.21.2.113:6801/36367","cluster_addr":"172.21.2.113:6804/36367","heartbeat_back_addr":"172.21.2.113:6816/36367","heartbeat_front_addr":"172.21.2.113:6809/36367","state":["exists","up"]},{"osd":7,"uuid":"f5461f80-018d-428f-be89-2eb2bcbe7bb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6817","nonce":36365},{"type":"v1","addr":"172.21.2.113:6821","nonce":36365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6823","nonce":36365},{"type":"v1","addr":"172.21.2.113:6825","nonce":36365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6830","nonce":36365},{"type":"v1","addr":"172.21.2.113:6831","nonce":36365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6827","nonce":36365},{"type":"v1","addr":"172.21.2.113:6829","nonce":36365}]},"public_addr":"172.21.2.113:6821/36365","cluster_addr":"172.21.2.113:6825/36365","heartbeat_back_addr":"172.21.2.113:6831/36365","heartbeat_front_addr":"172.21.2.113:6829/36365","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:24:40.612 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:40.613 7fb126a2d700 1 -- 172.21.2.102:0/894459981 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb1040186c0 msgr2=0x7fb10401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:40.612 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:40.613 7fb126a2d700 1 --2- 172.21.2.102:0/894459981 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb1040186c0 0x7fb10401abc0 crc :-1 s=READY pgs=643 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:40.612 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:40.613 7fb126a2d700 1 -- 172.21.2.102:0/894459981 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb1201b1fb0 msgr2=0x7fb12019b400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:40.612 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:40.613 7fb126a2d700 1 --2- 172.21.2.102:0/894459981 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb1201b1fb0 0x7fb12019b400 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 rx=0x7fb10800b5c0 tx=0x7fb10800b960).stop 2021-02-10T11:24:40.613 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:40.614 7fb126a2d700 1 -- 172.21.2.102:0/894459981 shutdown_connections 2021-02-10T11:24:40.613 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:40.614 7fb126a2d700 1 --2- 172.21.2.102:0/894459981 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb1040186c0 0x7fb10401abc0 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:40.613 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:40.614 7fb126a2d700 1 --2- 172.21.2.102:0/894459981 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb12019c750 0x7fb12019cb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:40.613 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:40.614 7fb126a2d700 1 --2- 172.21.2.102:0/894459981 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb1201b23b0 0x7fb12019ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:40.613 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:40.614 7fb126a2d700 1 --2- 172.21.2.102:0/894459981 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb1201b1fb0 0x7fb12019b400 unknown :-1 s=CLOSED pgs=1390 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:40.613 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:40.614 7fb126a2d700 1 -- 172.21.2.102:0/894459981 >> 172.21.2.102:0/894459981 conn(0x7fb1200f2240 msgr2=0x7fb12019a9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:40.613 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:40.614 7fb126a2d700 1 -- 172.21.2.102:0/894459981 shutdown_connections 2021-02-10T11:24:40.613 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:40.614 7fb126a2d700 1 -- 172.21.2.102:0/894459981 wait complete. 2021-02-10T11:24:40.622 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:03:37.322519', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 10, 'pool_name': 'cephfs_metadata', 'create_time': '2021-02-10 11:24:08.122188', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '56', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 11, 'pool_name': 'cephfs_data', 'create_time': '2021-02-10 11:24:09.191606', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '57', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-02-10T11:24:40.623 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:24:41.054 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:41.038 7fdf010fa700 1 Processor -- start 2021-02-10T11:24:41.054 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:41.039 7fdf010fa700 1 -- start start 2021-02-10T11:24:41.054 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:41.039 7fdf010fa700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fdefc0e5170 0x7fdefc1b38e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:41.054 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:41.039 7fdf010fa700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdefc0e5570 0x7fdefc07a120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:41.054 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:41.039 7fdf010fa700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdefc078b90 con 0x7fdefc1b3f00 2021-02-10T11:24:41.054 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:41.039 7fdf010fa700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdefc0d1900 con 0x7fdefc0e5170 2021-02-10T11:24:41.054 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:41.039 7fdf010fa700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdefc0d1b20 con 0x7fdefc0e5570 2021-02-10T11:24:41.054 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:41.039 7fdefad9d700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fdefc0e5170 0x7fdefc1b38e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:41.054 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:41.039 7fdefad9d700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fdefc0e5170 0x7fdefc1b38e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:57392/0 (socket says 172.21.2.102:57392) 2021-02-10T11:24:41.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:41.039 7fdefad9d700 1 -- 172.21.2.102:0/3807247973 learned_addr learned my addr 172.21.2.102:0/3807247973 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:24:41.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:41.039 7fdefad9d700 1 -- 172.21.2.102:0/3807247973 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdefc0e5570 msgr2=0x7fdefc07a120 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:24:41.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:41.039 7fdefad9d700 1 --2- 172.21.2.102:0/3807247973 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdefc0e5570 0x7fdefc07a120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:41.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:41.039 7fdefad9d700 1 -- 172.21.2.102:0/3807247973 >> v1:172.21.2.113:6789/0 conn(0x7fdefc1b3f00 legacy=0x7fdefc077510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:24:41.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:41.039 7fdefad9d700 1 -- 172.21.2.102:0/3807247973 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdefc0d1d40 con 0x7fdefc0e5170 2021-02-10T11:24:41.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:41.040 7fdefad9d700 1 --2- 172.21.2.102:0/3807247973 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdefc0e5170 0x7fdefc1b38e0 secure :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 rx=0x7fdee4006700 tx=0x7fdee4009b80).ready entity=mon.0 client_cookie=951bb62dc83e97ca server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:41.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:41.040 7fdef9d9b700 1 -- 172.21.2.102:0/3807247973 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdee400c0c0 con 0x7fdefc0e5170 2021-02-10T11:24:41.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:41.040 7fdef9d9b700 1 -- 172.21.2.102:0/3807247973 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdee400c340 con 0x7fdefc0e5170 2021-02-10T11:24:41.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:41.040 7fdef9d9b700 1 -- 172.21.2.102:0/3807247973 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdee400c860 con 0x7fdefc0e5170 2021-02-10T11:24:41.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:41.040 7fdf010fa700 1 -- 172.21.2.102:0/3807247973 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdefc0e5170 msgr2=0x7fdefc1b38e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:41.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:41.040 7fdf010fa700 1 --2- 172.21.2.102:0/3807247973 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdefc0e5170 0x7fdefc1b38e0 secure :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 rx=0x7fdee4006700 tx=0x7fdee4009b80).stop 2021-02-10T11:24:41.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:41.041 7fdf010fa700 1 -- 172.21.2.102:0/3807247973 shutdown_connections 2021-02-10T11:24:41.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:41.041 7fdf010fa700 1 --2- 172.21.2.102:0/3807247973 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdefc0e5570 0x7fdefc07a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:41.056 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:41.041 7fdf010fa700 1 --2- 172.21.2.102:0/3807247973 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdefc0e5170 0x7fdefc1b38e0 unknown :-1 s=CLOSED pgs=1391 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:41.056 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:41.041 7fdf010fa700 1 -- 172.21.2.102:0/3807247973 >> 172.21.2.102:0/3807247973 conn(0x7fdefc0ce820 msgr2=0x7fdefc0e4660 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:41.056 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:41.041 7fdf010fa700 1 -- 172.21.2.102:0/3807247973 shutdown_connections 2021-02-10T11:24:41.056 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:41.041 7fdf010fa700 1 -- 172.21.2.102:0/3807247973 wait complete. 2021-02-10T11:24:41.057 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:41.055 7fdf010fa700 1 Processor -- start 2021-02-10T11:24:41.057 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:41.055 7fdf010fa700 1 -- start start 2021-02-10T11:24:41.057 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:41.056 7fdf010fa700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdefc0aeed0 0x7fdefc111280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:41.057 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:41.056 7fdf010fa700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdefc1119f0 0x7fdefc0be9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:41.057 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:41.056 7fdf010fa700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fdefc0ad2d0 0x7fdefc0ad6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:41.058 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:41.056 7fdf010fa700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdefc0add80 con 0x7fdefc1119f0 2021-02-10T11:24:41.058 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:41.056 7fdf010fa700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdefc0befc0 con 0x7fdefc0ad2d0 2021-02-10T11:24:41.058 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:41.056 7fdf010fa700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdefc0ae300 con 0x7fdefc0aeed0 2021-02-10T11:24:41.058 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:41.056 7fdefa59c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdefc1119f0 0x7fdefc0be9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:41.058 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:41.056 7fdefa59c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdefc1119f0 0x7fdefc0be9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:57398/0 (socket says 172.21.2.102:57398) 2021-02-10T11:24:41.058 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:41.056 7fdefa59c700 1 -- 172.21.2.102:0/55437101 learned_addr learned my addr 172.21.2.102:0/55437101 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:24:41.058 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:41.056 7fdefa59c700 1 -- 172.21.2.102:0/55437101 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdefc0aeed0 msgr2=0x7fdefc111280 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:24:41.058 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:41.056 7fdefa59c700 1 --2- 172.21.2.102:0/55437101 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdefc0aeed0 0x7fdefc111280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:41.058 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:41.056 7fdefa59c700 1 -- 172.21.2.102:0/55437101 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fdefc0ad2d0 msgr2=0x7fdefc0ad6d0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:24:41.058 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:41.056 7fdefa59c700 1 --2- 172.21.2.102:0/55437101 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fdefc0ad2d0 0x7fdefc0ad6d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:41.058 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:41.056 7fdefa59c700 1 -- 172.21.2.102:0/55437101 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdefc0ae520 con 0x7fdefc1119f0 2021-02-10T11:24:41.058 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:41.056 7fdefa59c700 1 --2- 172.21.2.102:0/55437101 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdefc1119f0 0x7fdefc0be9a0 secure :-1 s=READY pgs=1392 cs=0 l=1 rev1=1 rx=0x7fdeec00b5b0 tx=0x7fdeec00b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:41.059 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:41.056 7fdef37fe700 1 -- 172.21.2.102:0/55437101 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdeec00bd40 con 0x7fdefc1119f0 2021-02-10T11:24:41.059 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:41.056 7fdef37fe700 1 -- 172.21.2.102:0/55437101 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdeec002030 con 0x7fdefc1119f0 2021-02-10T11:24:41.059 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:41.056 7fdef37fe700 1 -- 172.21.2.102:0/55437101 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdeec002550 con 0x7fdefc1119f0 2021-02-10T11:24:41.059 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:41.056 7fdf010fa700 1 -- 172.21.2.102:0/55437101 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdefc0ae520 con 0x7fdefc1119f0 2021-02-10T11:24:41.059 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:41.056 7fdf010fa700 1 -- 172.21.2.102:0/55437101 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdefc0ae520 con 0x7fdefc1119f0 2021-02-10T11:24:41.059 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:41.057 7fdef37fe700 1 -- 172.21.2.102:0/55437101 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fdeec002210 con 0x7fdefc1119f0 2021-02-10T11:24:41.059 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:41.057 7fdef37fe700 1 --2- 172.21.2.102:0/55437101 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fdee00186c0 0x7fdee001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:41.059 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:41.057 7fdef37fe700 1 -- 172.21.2.102:0/55437101 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fdeec0327e0 con 0x7fdefc1119f0 2021-02-10T11:24:41.059 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:41.057 7fdefad9d700 1 --2- 172.21.2.102:0/55437101 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fdee00186c0 0x7fdee001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:41.059 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:41.057 7fdefad9d700 1 --2- 172.21.2.102:0/55437101 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fdee00186c0 0x7fdee001abc0 crc :-1 s=READY pgs=644 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:41.088 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:41.089 7fdf010fa700 1 -- 172.21.2.102:0/55437101 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdefc10bc00 con 0x7fdefc1119f0 2021-02-10T11:24:41.090 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:41.092 7fdef37fe700 1 -- 172.21.2.102:0/55437101 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fdeec002240 con 0x7fdefc1119f0 2021-02-10T11:24:41.217 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:41.218 7fdf010fa700 1 -- 172.21.2.102:0/55437101 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7fdefc0fe290 con 0x7fdefc1119f0 2021-02-10T11:24:41.217 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:41.218 7fdef37fe700 1 -- 172.21.2.102:0/55437101 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v57) v1 ==== 92+0+10 (secure 0 0 0) 0x7fdeec002030 con 0x7fdefc1119f0 2021-02-10T11:24:41.218 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:24:41.219 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:41.220 7fdf010fa700 1 -- 172.21.2.102:0/55437101 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fdee00186c0 msgr2=0x7fdee001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:41.220 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:41.220 7fdf010fa700 1 --2- 172.21.2.102:0/55437101 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fdee00186c0 0x7fdee001abc0 crc :-1 s=READY pgs=644 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:41.220 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:41.220 7fdf010fa700 1 -- 172.21.2.102:0/55437101 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdefc1119f0 msgr2=0x7fdefc0be9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:41.220 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:41.220 7fdf010fa700 1 --2- 172.21.2.102:0/55437101 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdefc1119f0 0x7fdefc0be9a0 secure :-1 s=READY pgs=1392 cs=0 l=1 rev1=1 rx=0x7fdeec00b5b0 tx=0x7fdeec00b950).stop 2021-02-10T11:24:41.220 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:41.220 7fdf010fa700 1 -- 172.21.2.102:0/55437101 shutdown_connections 2021-02-10T11:24:41.220 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:41.220 7fdf010fa700 1 --2- 172.21.2.102:0/55437101 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fdee00186c0 0x7fdee001abc0 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:41.220 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:41.220 7fdf010fa700 1 --2- 172.21.2.102:0/55437101 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fdefc0ad2d0 0x7fdefc0ad6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:41.220 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:41.221 7fdf010fa700 1 --2- 172.21.2.102:0/55437101 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdefc1119f0 0x7fdefc0be9a0 unknown :-1 s=CLOSED pgs=1392 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:41.220 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:41.221 7fdf010fa700 1 --2- 172.21.2.102:0/55437101 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdefc0aeed0 0x7fdefc111280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:41.220 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:41.221 7fdf010fa700 1 -- 172.21.2.102:0/55437101 >> 172.21.2.102:0/55437101 conn(0x7fdefc0b6c50 msgr2=0x7fdefc097160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:41.220 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:41.221 7fdf010fa700 1 -- 172.21.2.102:0/55437101 shutdown_connections 2021-02-10T11:24:41.220 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:41.221 7fdf010fa700 1 -- 172.21.2.102:0/55437101 wait complete. 2021-02-10T11:24:41.228 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-02-10T11:24:41.659 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:41.642 7f173b0c8700 1 Processor -- start 2021-02-10T11:24:41.659 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:41.642 7f173b0c8700 1 -- start start 2021-02-10T11:24:41.659 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:41.643 7f173b0c8700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f17340e5570 0x7f1734078520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:41.659 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:41.643 7f173b0c8700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1734078b90 con 0x7f17340e5170 2021-02-10T11:24:41.659 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:41.643 7f173b0c8700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17341b40d0 con 0x7f1734079110 2021-02-10T11:24:41.659 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:41.643 7f173b0c8700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17340d1900 con 0x7f17340e5570 2021-02-10T11:24:41.659 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:41.643 7f1738ee4700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f17340e5170 0x7f17341b38e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:49630/0 (socket says 172.21.2.102:49630) 2021-02-10T11:24:41.659 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:41.643 7f1738ee4700 1 -- 172.21.2.102:0/29503277 learned_addr learned my addr 172.21.2.102:0/29503277 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:24:41.659 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:41.644 7f17337fe700 1 -- 172.21.2.102:0/29503277 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3542389134 0 0) 0x7f171c000ce0 con 0x7f17340e5170 2021-02-10T11:24:41.659 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:41.644 7f17337fe700 1 -- 172.21.2.102:0/29503277 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1720002fe0 con 0x7f17340e5170 2021-02-10T11:24:41.660 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:41.644 7f1733fff700 1 --2- 172.21.2.102:0/29503277 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f17340e5570 0x7f1734078520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:41.660 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:41.644 7f17337fe700 1 -- 172.21.2.102:0/29503277 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3454153811 0 0) 0x7f171c0010e0 con 0x7f17340e5170 2021-02-10T11:24:41.660 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:41.644 7f17337fe700 1 -- 172.21.2.102:0/29503277 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f17340e5570 msgr2=0x7f1734078520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:24:41.660 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:41.644 7f17337fe700 1 --2- 172.21.2.102:0/29503277 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f17340e5570 0x7f1734078520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:41.660 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:41.644 7f17337fe700 1 -- 172.21.2.102:0/29503277 >> v1:172.21.2.113:6789/0 conn(0x7f1734079110 legacy=0x7f1734079510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:41.660 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:41.644 7f17337fe700 1 -- 172.21.2.102:0/29503277 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17340d1ae0 con 0x7f17340e5170 2021-02-10T11:24:41.660 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:41.644 7f17337fe700 1 -- 172.21.2.102:0/29503277 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7f171c001660 con 0x7f17340e5170 2021-02-10T11:24:41.660 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:41.644 7f17337fe700 1 -- 172.21.2.102:0/29503277 >> v1:172.21.2.102:6789/0 conn(0x7f17340e5170 legacy=0x7f17341b38e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:41.660 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:41.644 7f17337fe700 1 --2- 172.21.2.102:0/29503277 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1720003b10 0x7f1720005f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:41.660 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:41.644 7f17337fe700 1 --2- 172.21.2.102:0/29503277 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1720006770 0x7f1720008c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:41.660 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:41.644 7f17337fe700 1 --2- 172.21.2.102:0/29503277 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f17200094f0 0x7f172000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:41.660 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:41.644 7f17337fe700 1 -- 172.21.2.102:0/29503277 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f172000c2c0 con 0x7f1720006770 2021-02-10T11:24:41.660 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:41.644 7f17337fe700 1 -- 172.21.2.102:0/29503277 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f172000c4e0 con 0x7f17200094f0 2021-02-10T11:24:41.660 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:41.644 7f17337fe700 1 -- 172.21.2.102:0/29503277 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f172000c700 con 0x7f1720003b10 2021-02-10T11:24:41.661 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:41.644 7f1733fff700 1 --2- 172.21.2.102:0/29503277 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1720006770 0x7f1720008c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:41.661 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:41.644 7f17396e5700 1 --2- 172.21.2.102:0/29503277 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f17200094f0 0x7f172000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:41.661 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:41.645 7f1738ee4700 1 --2- 172.21.2.102:0/29503277 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1720003b10 0x7f1720005f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:41.661 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:41.645 7f1733fff700 1 -- 172.21.2.102:0/29503277 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1720003b10 msgr2=0x7f1720005f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:24:41.661 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:41.645 7f1733fff700 1 --2- 172.21.2.102:0/29503277 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1720003b10 0x7f1720005f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:41.661 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:41.645 7f1733fff700 1 -- 172.21.2.102:0/29503277 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f17200094f0 msgr2=0x7f172000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:24:41.661 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:41.645 7f1733fff700 1 --2- 172.21.2.102:0/29503277 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f17200094f0 0x7f172000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:41.661 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:41.645 7f1733fff700 1 -- 172.21.2.102:0/29503277 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1720050970 con 0x7f1720006770 2021-02-10T11:24:41.661 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:41.645 7f17396e5700 1 --2- 172.21.2.102:0/29503277 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f17200094f0 0x7f172000ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:24:41.661 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:41.645 7f1733fff700 1 --2- 172.21.2.102:0/29503277 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1720006770 0x7f1720008c70 secure :-1 s=READY pgs=1394 cs=0 l=1 rev1=1 rx=0x7f1724006f80 tx=0x7f1724009e20).ready entity=mon.0 client_cookie=41485d82e6c7e46 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:41.661 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:41.645 7f17337fe700 1 -- 172.21.2.102:0/29503277 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f172400b510 con 0x7f1720006770 2021-02-10T11:24:41.661 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:41.645 7f17337fe700 1 -- 172.21.2.102:0/29503277 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f172400b850 con 0x7f1720006770 2021-02-10T11:24:41.661 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:41.645 7f17337fe700 1 -- 172.21.2.102:0/29503277 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f172400bd30 con 0x7f1720006770 2021-02-10T11:24:41.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:41.645 7f173b0c8700 1 -- 172.21.2.102:0/29503277 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1720006770 msgr2=0x7f1720008c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:41.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:41.645 7f173b0c8700 1 --2- 172.21.2.102:0/29503277 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1720006770 0x7f1720008c70 secure :-1 s=READY pgs=1394 cs=0 l=1 rev1=1 rx=0x7f1724006f80 tx=0x7f1724009e20).stop 2021-02-10T11:24:41.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:41.645 7f17396e5700 1 -- 172.21.2.102:0/29503277 reap_dead start 2021-02-10T11:24:41.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:41.645 7f173b0c8700 1 -- 172.21.2.102:0/29503277 shutdown_connections 2021-02-10T11:24:41.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:41.645 7f173b0c8700 1 -- 172.21.2.102:0/29503277 >> 172.21.2.102:0/29503277 conn(0x7f17340ce820 msgr2=0x7f17340e4660 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:41.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:41.645 7f173b0c8700 1 -- 172.21.2.102:0/29503277 shutdown_connections 2021-02-10T11:24:41.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:41.645 7f173b0c8700 1 -- 172.21.2.102:0/29503277 wait complete. 2021-02-10T11:24:41.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:41.660 7f173b0c8700 1 Processor -- start 2021-02-10T11:24:41.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:41.660 7f173b0c8700 1 -- start start 2021-02-10T11:24:41.663 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:41.661 7f173b0c8700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f17340a0c00 0x7f1734093490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:41.663 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:41.661 7f173b0c8700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f17340a5580 0x7f17340a5980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:41.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:41.661 7f173b0c8700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f17340ac950 0x7f17340acd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:41.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:41.661 7f173b0c8700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1734090ac0 con 0x7f17340a5580 2021-02-10T11:24:41.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:41.661 7f173b0c8700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17340a5fa0 con 0x7f17340ac950 2021-02-10T11:24:41.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:41.661 7f173b0c8700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17340ad370 con 0x7f17340a0c00 2021-02-10T11:24:41.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:41.661 7f1733fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f17340a5580 0x7f17340a5980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:41.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:41.661 7f1733fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f17340a5580 0x7f17340a5980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:57416/0 (socket says 172.21.2.102:57416) 2021-02-10T11:24:41.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:41.661 7f1733fff700 1 -- 172.21.2.102:0/2841005803 learned_addr learned my addr 172.21.2.102:0/2841005803 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:24:41.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:41.661 7f1733fff700 1 -- 172.21.2.102:0/2841005803 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f17340a0c00 msgr2=0x7f1734093490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:41.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:41.661 7f1733fff700 1 --2- 172.21.2.102:0/2841005803 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f17340a0c00 0x7f1734093490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:41.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:41.661 7f1733fff700 1 -- 172.21.2.102:0/2841005803 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f17340ac950 msgr2=0x7f17340acd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:41.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:41.661 7f1733fff700 1 --2- 172.21.2.102:0/2841005803 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f17340ac950 0x7f17340acd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:41.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:41.661 7f1733fff700 1 -- 172.21.2.102:0/2841005803 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17340ad590 con 0x7f17340a5580 2021-02-10T11:24:41.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:41.661 7f1733fff700 1 --2- 172.21.2.102:0/2841005803 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f17340a5580 0x7f17340a5980 secure :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 rx=0x7f1724006e90 tx=0x7f172400bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:41.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:41.662 7f17317fa700 1 -- 172.21.2.102:0/2841005803 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1724006860 con 0x7f17340a5580 2021-02-10T11:24:41.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:41.662 7f17317fa700 1 -- 172.21.2.102:0/2841005803 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f172400c2f0 con 0x7f17340a5580 2021-02-10T11:24:41.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:41.662 7f17317fa700 1 -- 172.21.2.102:0/2841005803 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f172400c740 con 0x7f17340a5580 2021-02-10T11:24:41.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:41.662 7f173b0c8700 1 -- 172.21.2.102:0/2841005803 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17340ad590 con 0x7f17340a5580 2021-02-10T11:24:41.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:41.662 7f173b0c8700 1 -- 172.21.2.102:0/2841005803 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17341a7080 con 0x7f17340a5580 2021-02-10T11:24:41.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:41.662 7f17317fa700 1 -- 172.21.2.102:0/2841005803 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1724001010 con 0x7f17340a5580 2021-02-10T11:24:41.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:41.662 7f17317fa700 1 --2- 172.21.2.102:0/2841005803 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f17180186c0 0x7f171801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:41.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:41.663 7f17317fa700 1 -- 172.21.2.102:0/2841005803 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f1724032910 con 0x7f17340a5580 2021-02-10T11:24:41.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:41.663 7f1738ee4700 1 --2- 172.21.2.102:0/2841005803 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f17180186c0 0x7f171801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:41.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:41.663 7f1738ee4700 1 --2- 172.21.2.102:0/2841005803 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f17180186c0 0x7f171801abc0 crc :-1 s=READY pgs=645 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:41.692 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:41.693 7f173b0c8700 1 -- 172.21.2.102:0/2841005803 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1734090ac0 con 0x7f17340a5580 2021-02-10T11:24:41.694 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:41.695 7f17317fa700 1 -- 172.21.2.102:0/2841005803 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f172400be30 con 0x7f17340a5580 2021-02-10T11:24:41.821 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:41.823 7f173b0c8700 1 -- 172.21.2.102:0/2841005803 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7f1734090ac0 con 0x7f17340a5580 2021-02-10T11:24:41.822 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:41.823 7f17317fa700 1 -- 172.21.2.102:0/2841005803 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v57) v1 ==== 104+0+10 (secure 0 0 0) 0x7f172401f6b0 con 0x7f17340a5580 2021-02-10T11:24:41.822 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:24:41.823 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:41.825 7f173b0c8700 1 -- 172.21.2.102:0/2841005803 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f17180186c0 msgr2=0x7f171801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:41.823 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:41.825 7f173b0c8700 1 --2- 172.21.2.102:0/2841005803 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f17180186c0 0x7f171801abc0 crc :-1 s=READY pgs=645 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:41.824 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:41.825 7f173b0c8700 1 -- 172.21.2.102:0/2841005803 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f17340a5580 msgr2=0x7f17340a5980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:41.824 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:41.825 7f173b0c8700 1 --2- 172.21.2.102:0/2841005803 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f17340a5580 0x7f17340a5980 secure :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 rx=0x7f1724006e90 tx=0x7f172400bcf0).stop 2021-02-10T11:24:41.824 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:41.825 7f173b0c8700 1 -- 172.21.2.102:0/2841005803 shutdown_connections 2021-02-10T11:24:41.824 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:41.825 7f173b0c8700 1 --2- 172.21.2.102:0/2841005803 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f17180186c0 0x7f171801abc0 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:41.824 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:41.825 7f173b0c8700 1 --2- 172.21.2.102:0/2841005803 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f17340ac950 0x7f17340acd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:41.824 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:41.825 7f173b0c8700 1 --2- 172.21.2.102:0/2841005803 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f17340a5580 0x7f17340a5980 unknown :-1 s=CLOSED pgs=1395 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:41.825 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:41.825 7f173b0c8700 1 --2- 172.21.2.102:0/2841005803 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f17340a0c00 0x7f1734093490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:41.825 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:41.825 7f173b0c8700 1 -- 172.21.2.102:0/2841005803 >> 172.21.2.102:0/2841005803 conn(0x7f1734188880 msgr2=0x7f17340a4380 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:41.825 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:41.825 7f173b0c8700 1 -- 172.21.2.102:0/2841005803 shutdown_connections 2021-02-10T11:24:41.825 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:41.825 7f173b0c8700 1 -- 172.21.2.102:0/2841005803 wait complete. 2021-02-10T11:24:41.832 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-02-10T11:24:42.264 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:42.247 7f0fd14da700 1 Processor -- start 2021-02-10T11:24:42.264 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:42.247 7f0fd14da700 1 -- start start 2021-02-10T11:24:42.264 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:42.248 7f0fd14da700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f0fcc0e5170 0x7f0fcc1b38e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:42.264 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:42.248 7f0fd14da700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fcc078b90 con 0x7f0fcc0e5570 2021-02-10T11:24:42.264 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:42.248 7f0fd14da700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fcc0784f0 con 0x7f0fcc1b3f00 2021-02-10T11:24:42.264 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:42.248 7f0fd14da700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0fcc078710 con 0x7f0fcc0e5170 2021-02-10T11:24:42.264 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:42.248 7f0fcbac0700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f0fcc0e5570 0x7f0fcc077d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:49646/0 (socket says 172.21.2.102:49646) 2021-02-10T11:24:42.264 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:42.248 7f0fcbac0700 1 -- 172.21.2.102:0/3751565823 learned_addr learned my addr 172.21.2.102:0/3751565823 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:24:42.264 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:42.248 7f0fcb2bf700 1 --2- 172.21.2.102:0/3751565823 >> v2:172.21.2.113:3300/0 conn(0x7f0fcc0e5170 0x7f0fcc1b38e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:42.265 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:42.248 7f0fcaabe700 1 -- 172.21.2.102:0/3751565823 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1524906109 0 0) 0x7f0fc4001480 con 0x7f0fcc0e5570 2021-02-10T11:24:42.265 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:42.248 7f0fcaabe700 1 -- 172.21.2.102:0/3751565823 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0fb8002fe0 con 0x7f0fcc0e5570 2021-02-10T11:24:42.265 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:42.248 7f0fcaabe700 1 -- 172.21.2.102:0/3751565823 <== mon.1 v1:172.21.2.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2277833047 0 0) 0x7f0fb4001080 con 0x7f0fcc1b3f00 2021-02-10T11:24:42.265 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:42.248 7f0fcaabe700 1 -- 172.21.2.102:0/3751565823 --> v1:172.21.2.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0fb8003210 con 0x7f0fcc1b3f00 2021-02-10T11:24:42.265 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:42.248 7f0fcaabe700 1 -- 172.21.2.102:0/3751565823 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3179503329 0 0) 0x7f0fc4001270 con 0x7f0fcc0e5570 2021-02-10T11:24:42.265 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:42.248 7f0fcaabe700 1 -- 172.21.2.102:0/3751565823 >> v2:172.21.2.113:3300/0 conn(0x7f0fcc0e5170 msgr2=0x7f0fcc1b38e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:24:42.265 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:42.248 7f0fcaabe700 1 --2- 172.21.2.102:0/3751565823 >> v2:172.21.2.113:3300/0 conn(0x7f0fcc0e5170 0x7f0fcc1b38e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:42.265 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:42.248 7f0fcaabe700 1 -- 172.21.2.102:0/3751565823 >> v1:172.21.2.113:6789/0 conn(0x7f0fcc1b3f00 legacy=0x7f0fcc077510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:42.265 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:42.248 7f0fcaabe700 1 -- 172.21.2.102:0/3751565823 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fcc078930 con 0x7f0fcc0e5570 2021-02-10T11:24:42.265 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:42.248 7f0fcaabe700 1 -- 172.21.2.102:0/3751565823 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7f0fc4001b90 con 0x7f0fcc0e5570 2021-02-10T11:24:42.265 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:42.248 7f0fcaabe700 1 -- 172.21.2.102:0/3751565823 >> v1:172.21.2.102:6789/0 conn(0x7f0fcc0e5570 legacy=0x7f0fcc077d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:42.265 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:42.248 7f0fcaabe700 1 --2- 172.21.2.102:0/3751565823 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0fb80038e0 0x7f0fb8003ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:42.265 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:42.248 7f0fcaabe700 1 --2- 172.21.2.102:0/3751565823 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0fb80067c0 0x7f0fb8008c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:42.265 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:42.248 7f0fcaabe700 1 --2- 172.21.2.102:0/3751565823 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0fb8009580 0x7f0fb800baa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:42.266 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:42.248 7f0fcaabe700 1 -- 172.21.2.102:0/3751565823 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fb8050300 con 0x7f0fb80067c0 2021-02-10T11:24:42.266 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:42.248 7f0fcaabe700 1 -- 172.21.2.102:0/3751565823 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fb80504e0 con 0x7f0fb8009580 2021-02-10T11:24:42.266 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:42.248 7f0fcaabe700 1 -- 172.21.2.102:0/3751565823 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fb8050700 con 0x7f0fb80038e0 2021-02-10T11:24:42.266 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:42.248 7f0fc2abe700 1 --2- 172.21.2.102:0/3751565823 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0fb80067c0 0x7f0fb8008c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:42.266 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:42.249 7f0fcb2bf700 1 --2- 172.21.2.102:0/3751565823 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0fb80038e0 0x7f0fb8003ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:42.266 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:42.249 7f0fcbac0700 1 --2- 172.21.2.102:0/3751565823 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0fb8009580 0x7f0fb800baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:42.266 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:42.249 7f0fc2abe700 1 -- 172.21.2.102:0/3751565823 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0fb80038e0 msgr2=0x7f0fb8003ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:24:42.266 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:42.249 7f0fc2abe700 1 --2- 172.21.2.102:0/3751565823 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0fb80038e0 0x7f0fb8003ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:42.266 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:42.249 7f0fc2abe700 1 -- 172.21.2.102:0/3751565823 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0fb8009580 msgr2=0x7f0fb800baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:24:42.266 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:42.249 7f0fc2abe700 1 --2- 172.21.2.102:0/3751565823 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0fb8009580 0x7f0fb800baa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:42.266 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:42.249 7f0fc2abe700 1 -- 172.21.2.102:0/3751565823 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fb80509a0 con 0x7f0fb80067c0 2021-02-10T11:24:42.266 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:42.249 7f0fcbac0700 1 --2- 172.21.2.102:0/3751565823 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0fb8009580 0x7f0fb800baa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:24:42.266 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:42.249 7f0fcbac0700 1 -- 172.21.2.102:0/3751565823 reap_dead start 2021-02-10T11:24:42.267 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:42.249 7f0fc2abe700 1 --2- 172.21.2.102:0/3751565823 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0fb80067c0 0x7f0fb8008c80 secure :-1 s=READY pgs=1397 cs=0 l=1 rev1=1 rx=0x7f0fb4006ad0 tx=0x7f0fb4009b40).ready entity=mon.0 client_cookie=d11f7a768ec770c0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:42.267 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:42.249 7f0fcaabe700 1 -- 172.21.2.102:0/3751565823 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0fb4000c10 con 0x7f0fb80067c0 2021-02-10T11:24:42.267 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:42.249 7f0fcaabe700 1 -- 172.21.2.102:0/3751565823 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fb400b2e0 con 0x7f0fb80067c0 2021-02-10T11:24:42.267 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:42.249 7f0fcaabe700 1 -- 172.21.2.102:0/3751565823 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0fb400b800 con 0x7f0fb80067c0 2021-02-10T11:24:42.267 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:42.249 7f0fd14da700 1 -- 172.21.2.102:0/3751565823 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0fb80067c0 msgr2=0x7f0fb8008c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:42.267 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:42.249 7f0fd14da700 1 --2- 172.21.2.102:0/3751565823 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0fb80067c0 0x7f0fb8008c80 secure :-1 s=READY pgs=1397 cs=0 l=1 rev1=1 rx=0x7f0fb4006ad0 tx=0x7f0fb4009b40).stop 2021-02-10T11:24:42.267 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:42.250 7f0fd14da700 1 -- 172.21.2.102:0/3751565823 shutdown_connections 2021-02-10T11:24:42.267 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:42.250 7f0fd14da700 1 --2- 172.21.2.102:0/3751565823 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0fb80067c0 0x7f0fb8008c80 unknown :-1 s=CLOSED pgs=1397 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:42.267 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:42.250 7f0fd14da700 1 -- 172.21.2.102:0/3751565823 >> 172.21.2.102:0/3751565823 conn(0x7f0fcc0ce820 msgr2=0x7f0fcc0e4660 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:42.267 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:42.250 7f0fd14da700 1 -- 172.21.2.102:0/3751565823 shutdown_connections 2021-02-10T11:24:42.267 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:42.250 7f0fd14da700 1 -- 172.21.2.102:0/3751565823 wait complete. 2021-02-10T11:24:42.267 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:42.264 7f0fd14da700 1 Processor -- start 2021-02-10T11:24:42.267 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:42.264 7f0fd14da700 1 -- start start 2021-02-10T11:24:42.268 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:42.265 7f0fd14da700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0fcc10cf50 0x7f0fcc10d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:42.268 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:42.265 7f0fd14da700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0fcc096680 0x7f0fcc096a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:42.268 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:42.265 7f0fd14da700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0fcc0970a0 0x7f0fcc0974a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:42.268 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:42.265 7f0fd14da700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fcc10da40 con 0x7f0fcc10cf50 2021-02-10T11:24:42.268 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:42.265 7f0fd14da700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fcc10dc60 con 0x7f0fcc0970a0 2021-02-10T11:24:42.268 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:42.265 7f0fd14da700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fcc10de80 con 0x7f0fcc096680 2021-02-10T11:24:42.268 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:42.265 7f0fcb2bf700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0fcc10cf50 0x7f0fcc10d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:42.268 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:42.265 7f0fcb2bf700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0fcc10cf50 0x7f0fcc10d350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:57434/0 (socket says 172.21.2.102:57434) 2021-02-10T11:24:42.268 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:42.265 7f0fcb2bf700 1 -- 172.21.2.102:0/2864968335 learned_addr learned my addr 172.21.2.102:0/2864968335 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:24:42.269 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:42.265 7f0fcb2bf700 1 -- 172.21.2.102:0/2864968335 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0fcc096680 msgr2=0x7f0fcc096a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:42.269 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:42.266 7f0fcb2bf700 1 --2- 172.21.2.102:0/2864968335 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0fcc096680 0x7f0fcc096a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:42.269 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:42.266 7f0fcb2bf700 1 -- 172.21.2.102:0/2864968335 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0fcc0970a0 msgr2=0x7f0fcc0974a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:42.269 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:42.266 7f0fcb2bf700 1 --2- 172.21.2.102:0/2864968335 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0fcc0970a0 0x7f0fcc0974a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:42.270 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:42.266 7f0fcb2bf700 1 -- 172.21.2.102:0/2864968335 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fcc10e0a0 con 0x7f0fcc10cf50 2021-02-10T11:24:42.270 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:42.266 7f0fcb2bf700 1 --2- 172.21.2.102:0/2864968335 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0fcc10cf50 0x7f0fcc10d350 secure :-1 s=READY pgs=1398 cs=0 l=1 rev1=1 rx=0x7f0fbc00b560 tx=0x7f0fbc00b900).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:42.270 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:42.266 7f0fc8aba700 1 -- 172.21.2.102:0/2864968335 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0fbc00bcf0 con 0x7f0fcc10cf50 2021-02-10T11:24:42.270 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:42.266 7f0fc8aba700 1 -- 172.21.2.102:0/2864968335 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fbc000c90 con 0x7f0fcc10cf50 2021-02-10T11:24:42.270 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:42.266 7f0fc8aba700 1 -- 172.21.2.102:0/2864968335 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0fbc0022b0 con 0x7f0fcc10cf50 2021-02-10T11:24:42.270 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:42.266 7f0fd14da700 1 -- 172.21.2.102:0/2864968335 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fcc10e0a0 con 0x7f0fcc10cf50 2021-02-10T11:24:42.270 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:42.266 7f0fd14da700 1 -- 172.21.2.102:0/2864968335 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fcc10e0a0 con 0x7f0fcc10cf50 2021-02-10T11:24:42.270 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:42.267 7f0fc8aba700 1 -- 172.21.2.102:0/2864968335 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f0fbc002020 con 0x7f0fcc10cf50 2021-02-10T11:24:42.270 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:42.267 7f0fc8aba700 1 --2- 172.21.2.102:0/2864968335 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0fb00186c0 0x7f0fb001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:42.270 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:42.267 7f0fc8aba700 1 -- 172.21.2.102:0/2864968335 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f0fbc032430 con 0x7f0fcc10cf50 2021-02-10T11:24:42.270 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:42.267 7f0fc2abe700 1 --2- 172.21.2.102:0/2864968335 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0fb00186c0 0x7f0fb001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:42.270 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:42.267 7f0fc2abe700 1 --2- 172.21.2.102:0/2864968335 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0fb00186c0 0x7f0fb001abc0 crc :-1 s=READY pgs=646 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:42.296 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:42.297 7f0fd14da700 1 -- 172.21.2.102:0/2864968335 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fcc10ca30 con 0x7f0fcc10cf50 2021-02-10T11:24:42.298 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:42.300 7f0fc8aba700 1 -- 172.21.2.102:0/2864968335 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f0fbc02d0f0 con 0x7f0fcc10cf50 2021-02-10T11:24:42.429 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:42.431 7f0fd14da700 1 -- 172.21.2.102:0/2864968335 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7f0fcc10ca30 con 0x7f0fcc10cf50 2021-02-10T11:24:42.430 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:42.431 7f0fc8aba700 1 -- 172.21.2.102:0/2864968335 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v57) v1 ==== 100+0+10 (secure 0 0 0) 0x7f0fbc02df10 con 0x7f0fcc10cf50 2021-02-10T11:24:42.430 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:24:42.432 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:42.433 7f0fd14da700 1 -- 172.21.2.102:0/2864968335 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0fb00186c0 msgr2=0x7f0fb001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:42.432 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:42.433 7f0fd14da700 1 --2- 172.21.2.102:0/2864968335 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0fb00186c0 0x7f0fb001abc0 crc :-1 s=READY pgs=646 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:42.432 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:42.433 7f0fd14da700 1 -- 172.21.2.102:0/2864968335 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0fcc10cf50 msgr2=0x7f0fcc10d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:42.432 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:42.433 7f0fd14da700 1 --2- 172.21.2.102:0/2864968335 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0fcc10cf50 0x7f0fcc10d350 secure :-1 s=READY pgs=1398 cs=0 l=1 rev1=1 rx=0x7f0fbc00b560 tx=0x7f0fbc00b900).stop 2021-02-10T11:24:42.432 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:42.433 7f0fd14da700 1 -- 172.21.2.102:0/2864968335 shutdown_connections 2021-02-10T11:24:42.432 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:42.433 7f0fd14da700 1 --2- 172.21.2.102:0/2864968335 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0fb00186c0 0x7f0fb001abc0 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:42.432 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:42.433 7f0fd14da700 1 --2- 172.21.2.102:0/2864968335 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0fcc0970a0 0x7f0fcc0974a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:42.432 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:42.433 7f0fd14da700 1 --2- 172.21.2.102:0/2864968335 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0fcc096680 0x7f0fcc096a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:42.432 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:42.433 7f0fd14da700 1 --2- 172.21.2.102:0/2864968335 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0fcc10cf50 0x7f0fcc10d350 unknown :-1 s=CLOSED pgs=1398 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:42.432 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:42.433 7f0fd14da700 1 -- 172.21.2.102:0/2864968335 >> 172.21.2.102:0/2864968335 conn(0x7f0fcc1900f0 msgr2=0x7f0fcc0acea0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:42.433 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:42.433 7f0fd14da700 1 -- 172.21.2.102:0/2864968335 shutdown_connections 2021-02-10T11:24:42.433 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:42.434 7f0fd14da700 1 -- 172.21.2.102:0/2864968335 wait complete. 2021-02-10T11:24:42.441 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-02-10T11:24:42.441 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:24:42.867 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:42.852 7fb1dea37700 1 Processor -- start 2021-02-10T11:24:42.867 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:42.852 7fb1dea37700 1 -- start start 2021-02-10T11:24:42.867 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:42.853 7fb1dea37700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fb1d80e53e0 0x7fb1d81b0550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:42.867 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:42.853 7fb1dea37700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fb1d81b0b70 0x7fb1d8081910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:42.867 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:42.853 7fb1dea37700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1d8082d50 con 0x7fb1d8081f30 2021-02-10T11:24:42.867 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:42.853 7fb1dea37700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1d8082b20 con 0x7fb1d80e53e0 2021-02-10T11:24:42.868 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:42.853 7fb1dea37700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1d80e5fe0 con 0x7fb1d81b0b70 2021-02-10T11:24:42.868 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:42.853 7fb1dc853700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fb1d80e53e0 0x7fb1d81b0550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:42.868 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:42.853 7fb1dc853700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fb1d80e53e0 0x7fb1d81b0550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:57440/0 (socket says 172.21.2.102:57440) 2021-02-10T11:24:42.868 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:42.853 7fb1dc853700 1 -- 172.21.2.102:0/3392261731 learned_addr learned my addr 172.21.2.102:0/3392261731 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:24:42.868 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:42.853 7fb1dc853700 1 -- 172.21.2.102:0/3392261731 >> v2:172.21.2.113:3300/0 conn(0x7fb1d81b0b70 msgr2=0x7fb1d8081910 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:24:42.868 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:42.853 7fb1dc853700 1 --2- 172.21.2.102:0/3392261731 >> v2:172.21.2.113:3300/0 conn(0x7fb1d81b0b70 0x7fb1d8081910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:42.868 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:42.853 7fb1dc853700 1 -- 172.21.2.102:0/3392261731 >> v1:172.21.2.113:6789/0 conn(0x7fb1d8081f30 legacy=0x7fb1d8082330 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:24:42.868 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:42.853 7fb1dc853700 1 -- 172.21.2.102:0/3392261731 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1d80e57e0 con 0x7fb1d80e53e0 2021-02-10T11:24:42.868 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:42.853 7fb1dc853700 1 --2- 172.21.2.102:0/3392261731 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb1d80e53e0 0x7fb1d81b0550 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 rx=0x7fb1c0006700 tx=0x7fb1c0009b00).ready entity=mon.0 client_cookie=87a4661ed5c976fe server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:42.868 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:42.854 7fb1d77fe700 1 -- 172.21.2.102:0/3392261731 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb1c000c0c0 con 0x7fb1d80e53e0 2021-02-10T11:24:42.868 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:42.854 7fb1d77fe700 1 -- 172.21.2.102:0/3392261731 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1c000c400 con 0x7fb1d80e53e0 2021-02-10T11:24:42.868 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:42.854 7fb1d77fe700 1 -- 172.21.2.102:0/3392261731 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb1c000c920 con 0x7fb1d80e53e0 2021-02-10T11:24:42.868 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:42.854 7fb1dea37700 1 -- 172.21.2.102:0/3392261731 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb1d80e53e0 msgr2=0x7fb1d81b0550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:42.869 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:42.854 7fb1dea37700 1 --2- 172.21.2.102:0/3392261731 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb1d80e53e0 0x7fb1d81b0550 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 rx=0x7fb1c0006700 tx=0x7fb1c0009b00).stop 2021-02-10T11:24:42.869 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:42.854 7fb1dea37700 1 -- 172.21.2.102:0/3392261731 shutdown_connections 2021-02-10T11:24:42.869 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:42.854 7fb1dea37700 1 --2- 172.21.2.102:0/3392261731 >> v2:172.21.2.113:3300/0 conn(0x7fb1d81b0b70 0x7fb1d8081910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:42.869 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:42.854 7fb1dea37700 1 --2- 172.21.2.102:0/3392261731 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb1d80e53e0 0x7fb1d81b0550 unknown :-1 s=CLOSED pgs=1399 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:42.869 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:42.854 7fb1dea37700 1 -- 172.21.2.102:0/3392261731 >> 172.21.2.102:0/3392261731 conn(0x7fb1d80ce500 msgr2=0x7fb1d80e4610 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:42.869 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:42.854 7fb1dea37700 1 -- 172.21.2.102:0/3392261731 shutdown_connections 2021-02-10T11:24:42.869 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:42.854 7fb1dea37700 1 -- 172.21.2.102:0/3392261731 wait complete. 2021-02-10T11:24:42.870 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:42.869 7fb1dea37700 1 Processor -- start 2021-02-10T11:24:42.870 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:42.869 7fb1dea37700 1 -- start start 2021-02-10T11:24:42.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:42.869 7fb1dea37700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb1d81882a0 0x7fb1d81886a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:42.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:42.869 7fb1dea37700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb1d81a5d90 0x7fb1d81a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:42.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:42.869 7fb1dea37700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb1d81af7c0 0x7fb1d81afbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:42.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:42.869 7fb1dea37700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1d8188dd0 con 0x7fb1d81af7c0 2021-02-10T11:24:42.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:42.869 7fb1dea37700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1d81b01e0 con 0x7fb1d81882a0 2021-02-10T11:24:42.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:42.869 7fb1dea37700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1d81b0400 con 0x7fb1d81a5d90 2021-02-10T11:24:42.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:42.870 7fb1dd054700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb1d81af7c0 0x7fb1d81afbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:42.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:42.870 7fb1dd054700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb1d81af7c0 0x7fb1d81afbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:57446/0 (socket says 172.21.2.102:57446) 2021-02-10T11:24:42.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:42.870 7fb1dd054700 1 -- 172.21.2.102:0/4054570036 learned_addr learned my addr 172.21.2.102:0/4054570036 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:24:42.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:42.870 7fb1dd054700 1 -- 172.21.2.102:0/4054570036 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb1d81a5d90 msgr2=0x7fb1d81a6190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:42.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:42.870 7fb1dd054700 1 --2- 172.21.2.102:0/4054570036 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb1d81a5d90 0x7fb1d81a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:42.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:42.870 7fb1dd054700 1 -- 172.21.2.102:0/4054570036 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb1d81882a0 msgr2=0x7fb1d81886a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:42.872 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:42.870 7fb1dd054700 1 --2- 172.21.2.102:0/4054570036 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb1d81882a0 0x7fb1d81886a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:42.872 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:42.870 7fb1dd054700 1 -- 172.21.2.102:0/4054570036 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1d81b0620 con 0x7fb1d81af7c0 2021-02-10T11:24:42.872 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:42.870 7fb1dd054700 1 --2- 172.21.2.102:0/4054570036 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb1d81af7c0 0x7fb1d81afbc0 secure :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 rx=0x7fb1d0009940 tx=0x7fb1d000bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:42.872 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:42.871 7fb1d57fa700 1 -- 172.21.2.102:0/4054570036 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb1d0003030 con 0x7fb1d81af7c0 2021-02-10T11:24:42.872 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:42.871 7fb1d57fa700 1 -- 172.21.2.102:0/4054570036 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1d0001270 con 0x7fb1d81af7c0 2021-02-10T11:24:42.872 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:42.871 7fb1d57fa700 1 -- 172.21.2.102:0/4054570036 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb1d0001790 con 0x7fb1d81af7c0 2021-02-10T11:24:42.872 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:42.871 7fb1dea37700 1 -- 172.21.2.102:0/4054570036 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1d81b0620 con 0x7fb1d81af7c0 2021-02-10T11:24:42.872 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:42.871 7fb1dea37700 1 -- 172.21.2.102:0/4054570036 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1d81b0620 con 0x7fb1d81af7c0 2021-02-10T11:24:42.872 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:42.871 7fb1d57fa700 1 -- 172.21.2.102:0/4054570036 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fb1d0001450 con 0x7fb1d81af7c0 2021-02-10T11:24:42.872 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:42.871 7fb1d57fa700 1 --2- 172.21.2.102:0/4054570036 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb1bc0186c0 0x7fb1bc01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:42.872 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:42.872 7fb1d57fa700 1 -- 172.21.2.102:0/4054570036 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fb1d0032910 con 0x7fb1d81af7c0 2021-02-10T11:24:42.872 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:42.872 7fb1dc853700 1 --2- 172.21.2.102:0/4054570036 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb1bc0186c0 0x7fb1bc01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:42.873 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:42.872 7fb1dc853700 1 --2- 172.21.2.102:0/4054570036 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb1bc0186c0 0x7fb1bc01abc0 crc :-1 s=READY pgs=647 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:42.902 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:42.903 7fb1dea37700 1 -- 172.21.2.102:0/4054570036 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1d80ed660 con 0x7fb1d81af7c0 2021-02-10T11:24:42.904 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:42.906 7fb1d57fa700 1 -- 172.21.2.102:0/4054570036 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fb1d000be30 con 0x7fb1d81af7c0 2021-02-10T11:24:43.032 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:43.033 7fb1dea37700 1 -- 172.21.2.102:0/4054570036 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb1d80ed660 con 0x7fb1d81af7c0 2021-02-10T11:24:43.032 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:43.034 7fb1d57fa700 1 -- 172.21.2.102:0/4054570036 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) v1 ==== 94+0+2101 (secure 0 0 0) 0x7fb1d001f870 con 0x7fb1d81af7c0 2021-02-10T11:24:43.033 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:24:43.033 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":40,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7920,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.102:6835/1829204031","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6834","nonce":1829204031},{"type":"v1","addr":"172.21.2.102:6835","nonce":1829204031}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":35}],"filesystems":[{"mdsmap":{"epoch":39,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:24:10.198618","modified":"2021-02-10 11:24:12.204882","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5714},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5714":{"gid":5714,"name":"b","rank":0,"incarnation":36,"state":"up:active","state_seq":3,"addr":"172.21.2.113:6833/4013137507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6832","nonce":4013137507},{"type":"v1","addr":"172.21.2.113:6833","nonce":4013137507}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":5}]} 2021-02-10T11:24:43.033 INFO:teuthology.orchestra.run.gibba002.stderr:dumped fsmap epoch 40 2021-02-10T11:24:43.034 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:43.035 7fb1dea37700 1 -- 172.21.2.102:0/4054570036 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb1bc0186c0 msgr2=0x7fb1bc01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:43.034 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:43.035 7fb1dea37700 1 --2- 172.21.2.102:0/4054570036 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb1bc0186c0 0x7fb1bc01abc0 crc :-1 s=READY pgs=647 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:43.034 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:43.035 7fb1dea37700 1 -- 172.21.2.102:0/4054570036 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb1d81af7c0 msgr2=0x7fb1d81afbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:43.034 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:43.035 7fb1dea37700 1 --2- 172.21.2.102:0/4054570036 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb1d81af7c0 0x7fb1d81afbc0 secure :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 rx=0x7fb1d0009940 tx=0x7fb1d000bcf0).stop 2021-02-10T11:24:43.035 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:43.036 7fb1dea37700 1 -- 172.21.2.102:0/4054570036 shutdown_connections 2021-02-10T11:24:43.035 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:43.036 7fb1dea37700 1 --2- 172.21.2.102:0/4054570036 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb1d81af7c0 0x7fb1d81afbc0 unknown :-1 s=CLOSED pgs=1400 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:43.035 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:43.036 7fb1dea37700 1 --2- 172.21.2.102:0/4054570036 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb1d81a5d90 0x7fb1d81a6190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:43.035 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:43.036 7fb1dea37700 1 --2- 172.21.2.102:0/4054570036 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb1bc0186c0 0x7fb1bc01abc0 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:43.035 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:43.036 7fb1dea37700 1 --2- 172.21.2.102:0/4054570036 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb1d81882a0 0x7fb1d81886a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:43.035 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:43.036 7fb1dea37700 1 -- 172.21.2.102:0/4054570036 >> 172.21.2.102:0/4054570036 conn(0x7fb1d80a0c60 msgr2=0x7fb1d81ae520 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:43.036 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:43.036 7fb1dea37700 1 -- 172.21.2.102:0/4054570036 shutdown_connections 2021-02-10T11:24:43.036 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:43.036 7fb1dea37700 1 -- 172.21.2.102:0/4054570036 wait complete. 2021-02-10T11:24:43.244 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 39, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:24:10.198618', 'modified': '2021-02-10 11:24:12.204882', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5714}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5714': {'gid': 5714, 'name': 'b', 'rank': 0, 'incarnation': 36, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.113:6833/4013137507', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.113:6832', 'nonce': 4013137507}, {'type': 'v1', 'addr': '172.21.2.113:6833', 'nonce': 4013137507}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [11], 'metadata_pool': 10, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:24:43.244 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:24:43.245 DEBUG:teuthology.orchestra.run.gibba013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:24:43.643 INFO:teuthology.orchestra.run.gibba013.stdout:{ 2021-02-10T11:24:43.643 INFO:teuthology.orchestra.run.gibba013.stdout: "cluster_fsid": "68db30ea-dbce-4aef-a6df-f01c6fea0513", 2021-02-10T11:24:43.643 INFO:teuthology.orchestra.run.gibba013.stdout: "whoami": 0, 2021-02-10T11:24:43.643 INFO:teuthology.orchestra.run.gibba013.stdout: "id": 5714, 2021-02-10T11:24:43.643 INFO:teuthology.orchestra.run.gibba013.stdout: "want_state": "up:active", 2021-02-10T11:24:43.643 INFO:teuthology.orchestra.run.gibba013.stdout: "state": "up:active", 2021-02-10T11:24:43.643 INFO:teuthology.orchestra.run.gibba013.stdout: "rank_uptime": 33.444484164000002, 2021-02-10T11:24:43.643 INFO:teuthology.orchestra.run.gibba013.stdout: "mdsmap_epoch": 39, 2021-02-10T11:24:43.643 INFO:teuthology.orchestra.run.gibba013.stdout: "osdmap_epoch": 57, 2021-02-10T11:24:43.643 INFO:teuthology.orchestra.run.gibba013.stdout: "osdmap_epoch_barrier": 57, 2021-02-10T11:24:43.643 INFO:teuthology.orchestra.run.gibba013.stdout: "uptime": 39.120852499999998 2021-02-10T11:24:43.644 INFO:teuthology.orchestra.run.gibba013.stdout:} 2021-02-10T11:24:43.647 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "68db30ea-dbce-4aef-a6df-f01c6fea0513", "whoami": 0, "id": 5714, "want_state": "up:active", "state": "up:active", "rank_uptime": 33.444484164000002, "mdsmap_epoch": 39, "osdmap_epoch": 57, "osdmap_epoch_barrier": 57, "uptime": 39.120852499999998 } 2021-02-10T11:24:43.647 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2021-02-10T11:24:43.647 INFO:tasks.cephfs.fuse_mount:Client client.alice config is {} 2021-02-10T11:24:43.647 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.alice at ubuntu@gibba002.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.1... 2021-02-10T11:24:43.647 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:24:43.647 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec mkdir -p /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:24:43.982 DEBUG:teuthology.orchestra.run.gibba002:> sudo modprobe fuse 2021-02-10T11:24:44.328 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:24:44.328 DEBUG:teuthology.orchestra.run.gibba002:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:24:44.671 INFO:teuthology.orchestra.run.gibba002.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:24:44.672 INFO:teuthology.orchestra.run.gibba002.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:24:44.672 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:24:44.673 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:24:44.673 DEBUG:teuthology.orchestra.run.gibba002:> ls /sys/fs/fuse/connections 2021-02-10T11:24:45.008 INFO:teuthology.orchestra.run.gibba002.stdout:43 2021-02-10T11:24:45.008 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [43] 2021-02-10T11:24:45.009 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --client_mountpoint=/volumes/group_0000000000878856/subvolume_0000000000190758 --name client.alice /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:24:45.011 DEBUG:teuthology.orchestra.run.gibba002:> sudo modprobe fuse 2021-02-10T11:24:45.355 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:24:45.356 DEBUG:teuthology.orchestra.run.gibba002:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:24:45.422 INFO:tasks.cephfs.fuse_mount.ceph-fuse.alice.gibba002.stderr:ceph-fuse[155213]: starting ceph client 2021-02-10T11:24:45.423 INFO:tasks.cephfs.fuse_mount.ceph-fuse.alice.gibba002.stderr:2021-02-10 11:24:45.423 7f3466a74f80 -1 init, newargv = 0x562a779d4440 newargc=9 2021-02-10T11:24:45.429 INFO:tasks.cephfs.fuse_mount.ceph-fuse.alice.gibba002.stderr:ceph-fuse[155213]: starting fuse 2021-02-10T11:24:45.699 INFO:teuthology.orchestra.run.gibba002.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:24:45.699 INFO:teuthology.orchestra.run.gibba002.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:24:45.700 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:24:45.700 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:24:45.700 DEBUG:teuthology.orchestra.run.gibba002:> ls /sys/fs/fuse/connections 2021-02-10T11:24:46.035 INFO:teuthology.orchestra.run.gibba002.stdout:43 2021-02-10T11:24:46.036 INFO:teuthology.orchestra.run.gibba002.stdout:44 2021-02-10T11:24:46.036 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43, 44] 2021-02-10T11:24:46.036 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:24:46.036 DEBUG:teuthology.orchestra.run.gibba002:> sudo python3 -c ' 2021-02-10T11:24:46.037 DEBUG:teuthology.orchestra.run.gibba002:> import glob 2021-02-10T11:24:46.037 DEBUG:teuthology.orchestra.run.gibba002:> import re 2021-02-10T11:24:46.037 DEBUG:teuthology.orchestra.run.gibba002:> import os 2021-02-10T11:24:46.037 DEBUG:teuthology.orchestra.run.gibba002:> import subprocess 2021-02-10T11:24:46.037 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:24:46.037 DEBUG:teuthology.orchestra.run.gibba002:> def find_socket(client_name): 2021-02-10T11:24:46.038 DEBUG:teuthology.orchestra.run.gibba002:> asok_path = "/var/run/ceph/ceph-client.alice.*.asok" 2021-02-10T11:24:46.038 DEBUG:teuthology.orchestra.run.gibba002:> files = glob.glob(asok_path) 2021-02-10T11:24:46.038 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:24:46.038 DEBUG:teuthology.orchestra.run.gibba002:> # Given a non-glob path, it better be there 2021-02-10T11:24:46.038 DEBUG:teuthology.orchestra.run.gibba002:> if "*" not in asok_path: 2021-02-10T11:24:46.038 DEBUG:teuthology.orchestra.run.gibba002:> assert(len(files) == 1) 2021-02-10T11:24:46.038 DEBUG:teuthology.orchestra.run.gibba002:> return files[0] 2021-02-10T11:24:46.038 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:24:46.038 DEBUG:teuthology.orchestra.run.gibba002:> for f in files: 2021-02-10T11:24:46.038 DEBUG:teuthology.orchestra.run.gibba002:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2021-02-10T11:24:46.038 DEBUG:teuthology.orchestra.run.gibba002:> if os.path.exists("/proc/{0}".format(pid)): 2021-02-10T11:24:46.038 DEBUG:teuthology.orchestra.run.gibba002:> return f 2021-02-10T11:24:46.039 DEBUG:teuthology.orchestra.run.gibba002:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2021-02-10T11:24:46.039 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:24:46.039 DEBUG:teuthology.orchestra.run.gibba002:> print(find_socket("client.alice")) 2021-02-10T11:24:46.039 DEBUG:teuthology.orchestra.run.gibba002:> ' 2021-02-10T11:24:46.399 INFO:teuthology.orchestra.run.gibba002.stdout:/var/run/ceph/ceph-client.alice.155213.asok 2021-02-10T11:24:46.402 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.alice.155213.asok 2021-02-10T11:24:46.402 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:24:46.402 DEBUG:teuthology.orchestra.run.gibba002:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.alice.155213.asok status 2021-02-10T11:24:46.813 INFO:teuthology.orchestra.run.gibba002.stdout:{ 2021-02-10T11:24:46.813 INFO:teuthology.orchestra.run.gibba002.stdout: "metadata": { 2021-02-10T11:24:46.813 INFO:teuthology.orchestra.run.gibba002.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:24:46.814 INFO:teuthology.orchestra.run.gibba002.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:24:46.814 INFO:teuthology.orchestra.run.gibba002.stdout: "entity_id": "alice", 2021-02-10T11:24:46.814 INFO:teuthology.orchestra.run.gibba002.stdout: "hostname": "gibba002", 2021-02-10T11:24:46.814 INFO:teuthology.orchestra.run.gibba002.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2021-02-10T11:24:46.814 INFO:teuthology.orchestra.run.gibba002.stdout: "pid": "155213", 2021-02-10T11:24:46.814 INFO:teuthology.orchestra.run.gibba002.stdout: "root": "/volumes/group_0000000000878856/subvolume_0000000000190758" 2021-02-10T11:24:46.814 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:24:46.815 INFO:teuthology.orchestra.run.gibba002.stdout: "dentry_count": 0, 2021-02-10T11:24:46.815 INFO:teuthology.orchestra.run.gibba002.stdout: "dentry_pinned_count": 0, 2021-02-10T11:24:46.815 INFO:teuthology.orchestra.run.gibba002.stdout: "id": 8229, 2021-02-10T11:24:46.815 INFO:teuthology.orchestra.run.gibba002.stdout: "inst": { 2021-02-10T11:24:46.815 INFO:teuthology.orchestra.run.gibba002.stdout: "name": { 2021-02-10T11:24:46.816 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "client", 2021-02-10T11:24:46.816 INFO:teuthology.orchestra.run.gibba002.stdout: "num": 8229 2021-02-10T11:24:46.816 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:24:46.816 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": { 2021-02-10T11:24:46.816 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:24:46.816 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:0", 2021-02-10T11:24:46.817 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 1149444066 2021-02-10T11:24:46.817 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:24:46.818 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:24:46.818 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": { 2021-02-10T11:24:46.819 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:24:46.819 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:0", 2021-02-10T11:24:46.819 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 1149444066 2021-02-10T11:24:46.819 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:24:46.819 INFO:teuthology.orchestra.run.gibba002.stdout: "inst_str": "client.8229 172.21.2.102:0/1149444066", 2021-02-10T11:24:46.819 INFO:teuthology.orchestra.run.gibba002.stdout: "addr_str": "172.21.2.102:0/1149444066", 2021-02-10T11:24:46.819 INFO:teuthology.orchestra.run.gibba002.stdout: "inode_count": 1, 2021-02-10T11:24:46.820 INFO:teuthology.orchestra.run.gibba002.stdout: "mds_epoch": 39, 2021-02-10T11:24:46.820 INFO:teuthology.orchestra.run.gibba002.stdout: "osd_epoch": 57, 2021-02-10T11:24:46.820 INFO:teuthology.orchestra.run.gibba002.stdout: "osd_epoch_barrier": 0, 2021-02-10T11:24:46.820 INFO:teuthology.orchestra.run.gibba002.stdout: "blacklisted": false 2021-02-10T11:24:46.820 INFO:teuthology.orchestra.run.gibba002.stdout:} 2021-02-10T11:24:46.822 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:24:46.823 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:24:46.823 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:24:47.161 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:24:47.162 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2021-02-10T11:24:47.162 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.1 && sudo dd if=/dev/urandom of=data.bin bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:24:47.538 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:24:47.538 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:24:47.538 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0263944 s, 39.7 MB/s 2021-02-10T11:24:47.539 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume authorize cephfs subvolume_0000000000190758 alice --group_name group_0000000000878856 --tenant_id tenant_id --access_level r 2021-02-10T11:24:47.979 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:47.963 7f83b4ada700 1 Processor -- start 2021-02-10T11:24:47.979 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:47.963 7f83b4ada700 1 -- start start 2021-02-10T11:24:47.979 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:47.964 7f83b4ada700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f83b00e5170 0x7f83b01b38e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:47.979 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:47.964 7f83b4ada700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f83b00e5570 0x7f83b007a120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:47.979 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:47.964 7f83b4ada700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83b0078b90 con 0x7f83b01b3f00 2021-02-10T11:24:47.979 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:47.964 7f83b4ada700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83b00d1900 con 0x7f83b00e5570 2021-02-10T11:24:47.979 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:47.964 7f83b4ada700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83b00d1af0 con 0x7f83b00e5170 2021-02-10T11:24:47.980 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:47.964 7f83ae05d700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f83b01b3f00 0x7f83b0077510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:49700/0 (socket says 172.21.2.102:49700) 2021-02-10T11:24:47.980 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:47.964 7f83ae05d700 1 -- 172.21.2.102:0/2782393236 learned_addr learned my addr 172.21.2.102:0/2782393236 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:24:47.980 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:47.964 7f83af05f700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f83b00e5570 0x7f83b007a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:47.980 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:47.964 7f83af05f700 1 -- 172.21.2.102:0/2782393236 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f83b00e5170 msgr2=0x7f83b01b38e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:24:47.980 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:47.964 7f83af05f700 1 --2- 172.21.2.102:0/2782393236 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f83b00e5170 0x7f83b01b38e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:47.980 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:47.964 7f83af05f700 1 -- 172.21.2.102:0/2782393236 >> v1:172.21.2.102:6789/0 conn(0x7f83b01b3f00 legacy=0x7f83b0077510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:47.980 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:47.964 7f83af05f700 1 -- 172.21.2.102:0/2782393236 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83b00d1ce0 con 0x7f83b00e5570 2021-02-10T11:24:47.980 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:47.964 7f83af05f700 1 --2- 172.21.2.102:0/2782393236 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f83b00e5570 0x7f83b007a120 secure :-1 s=READY pgs=1405 cs=0 l=1 rev1=1 rx=0x7f83a8006ab0 tx=0x7f83a8009b40).ready entity=mon.0 client_cookie=1368d363c0799c78 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:47.980 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:47.965 7f83ad85c700 1 -- 172.21.2.102:0/2782393236 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f83a80022e0 con 0x7f83b00e5570 2021-02-10T11:24:47.980 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:47.965 7f83ad85c700 1 -- 172.21.2.102:0/2782393236 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83a8002560 con 0x7f83b00e5570 2021-02-10T11:24:47.980 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:47.965 7f83ad85c700 1 -- 172.21.2.102:0/2782393236 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f83a80029f0 con 0x7f83b00e5570 2021-02-10T11:24:47.980 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:47.965 7f83b4ada700 1 -- 172.21.2.102:0/2782393236 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f83b00e5570 msgr2=0x7f83b007a120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:47.980 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:47.965 7f83b4ada700 1 --2- 172.21.2.102:0/2782393236 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f83b00e5570 0x7f83b007a120 secure :-1 s=READY pgs=1405 cs=0 l=1 rev1=1 rx=0x7f83a8006ab0 tx=0x7f83a8009b40).stop 2021-02-10T11:24:47.981 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:47.966 7f83b4ada700 1 -- 172.21.2.102:0/2782393236 shutdown_connections 2021-02-10T11:24:47.981 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:47.966 7f83b4ada700 1 --2- 172.21.2.102:0/2782393236 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f83b00e5570 0x7f83b007a120 unknown :-1 s=CLOSED pgs=1405 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:47.981 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:47.966 7f83b4ada700 1 --2- 172.21.2.102:0/2782393236 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f83b00e5170 0x7f83b01b38e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:47.981 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:47.966 7f83b4ada700 1 -- 172.21.2.102:0/2782393236 >> 172.21.2.102:0/2782393236 conn(0x7f83b00ce820 msgr2=0x7f83b00e4660 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:47.981 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:47.966 7f83b4ada700 1 -- 172.21.2.102:0/2782393236 shutdown_connections 2021-02-10T11:24:47.981 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:47.966 7f83b4ada700 1 -- 172.21.2.102:0/2782393236 wait complete. 2021-02-10T11:24:47.982 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:47.980 7f83b4ada700 1 Processor -- start 2021-02-10T11:24:47.982 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:47.981 7f83b4ada700 1 -- start start 2021-02-10T11:24:47.982 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:47.981 7f83b4ada700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f83b00ac260 0x7f83b01b1da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:47.982 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:47.981 7f83b4ada700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f83b01b23c0 0x7f83b008a570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:47.983 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:47.981 7f83b4ada700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f83b008ab90 0x7f83b008af90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:47.983 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:47.981 7f83b4ada700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83b008b5b0 con 0x7f83b01b23c0 2021-02-10T11:24:47.983 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:47.981 7f83b4ada700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83b008bed0 con 0x7f83b008ab90 2021-02-10T11:24:47.983 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:47.981 7f83b4ada700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83b008c0f0 con 0x7f83b00ac260 2021-02-10T11:24:47.983 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:47.981 7f83ae05d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f83b01b23c0 0x7f83b008a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:47.983 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:47.981 7f83ae05d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f83b01b23c0 0x7f83b008a570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:57486/0 (socket says 172.21.2.102:57486) 2021-02-10T11:24:47.983 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:47.981 7f83ae05d700 1 -- 172.21.2.102:0/599758812 learned_addr learned my addr 172.21.2.102:0/599758812 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:24:47.983 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:47.981 7f83ae05d700 1 -- 172.21.2.102:0/599758812 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f83b00ac260 msgr2=0x7f83b01b1da0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:24:47.983 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:47.982 7f83ae05d700 1 --2- 172.21.2.102:0/599758812 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f83b00ac260 0x7f83b01b1da0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:47.983 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:47.982 7f83ae05d700 1 -- 172.21.2.102:0/599758812 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f83b008ab90 msgr2=0x7f83b008af90 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:24:47.983 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:47.982 7f83ae05d700 1 --2- 172.21.2.102:0/599758812 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f83b008ab90 0x7f83b008af90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:47.983 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:47.982 7f83ae05d700 1 -- 172.21.2.102:0/599758812 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83b008c310 con 0x7f83b01b23c0 2021-02-10T11:24:47.984 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:47.982 7f83ae05d700 1 --2- 172.21.2.102:0/599758812 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f83b01b23c0 0x7f83b008a570 secure :-1 s=READY pgs=1406 cs=0 l=1 rev1=1 rx=0x7f83a4001c60 tx=0x7f83a400a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:47.984 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:47.982 7f839f7fe700 1 -- 172.21.2.102:0/599758812 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f83a400acc0 con 0x7f83b01b23c0 2021-02-10T11:24:47.984 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:47.982 7f839f7fe700 1 -- 172.21.2.102:0/599758812 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83a4004140 con 0x7f83b01b23c0 2021-02-10T11:24:47.984 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:47.982 7f839f7fe700 1 -- 172.21.2.102:0/599758812 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f83a4004620 con 0x7f83b01b23c0 2021-02-10T11:24:47.984 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:47.982 7f83b4ada700 1 -- 172.21.2.102:0/599758812 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83b008c310 con 0x7f83b01b23c0 2021-02-10T11:24:47.984 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:47.982 7f83b4ada700 1 -- 172.21.2.102:0/599758812 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83b008c8a0 con 0x7f83b01b23c0 2021-02-10T11:24:47.984 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:47.983 7f839f7fe700 1 -- 172.21.2.102:0/599758812 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f83a4004320 con 0x7f83b01b23c0 2021-02-10T11:24:47.984 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:47.983 7f839f7fe700 1 --2- 172.21.2.102:0/599758812 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f83940186c0 0x7f839401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:47.984 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:47.983 7f839f7fe700 1 -- 172.21.2.102:0/599758812 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f83a4032540 con 0x7f83b01b23c0 2021-02-10T11:24:47.984 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:47.983 7f83ae85e700 1 --2- 172.21.2.102:0/599758812 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f83940186c0 0x7f839401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:47.984 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:47.983 7f83ae85e700 1 --2- 172.21.2.102:0/599758812 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f83940186c0 0x7f839401abc0 crc :-1 s=READY pgs=648 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:48.012 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:48.014 7f83b4ada700 1 -- 172.21.2.102:0/599758812 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83b010bc10 con 0x7f83b01b23c0 2021-02-10T11:24:48.015 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:48.016 7f839f7fe700 1 -- 172.21.2.102:0/599758812 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f83a4000d90 con 0x7f83b01b23c0 2021-02-10T11:24:48.142 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:48.143 7f83b4ada700 1 -- 172.21.2.102:0/599758812 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"vol_name": "cephfs", "sub_name": "subvolume_0000000000190758", "target": ["mgr", ""], "tenant_id": "tenant_id", "group_name": "group_0000000000878856", "access_level": "r", "prefix": "fs subvolume authorize", "auth_id": "alice"}) v1 -- 0x7f83b008c8a0 con 0x7f83940186c0 2021-02-10T11:24:48.183 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:48.184 7f839f7fe700 1 -- 172.21.2.102:0/599758812 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+41 (crc 0 0 0) 0x7f83a0006ce0 con 0x7f83940186c0 2021-02-10T11:24:48.205 INFO:teuthology.orchestra.run.gibba002.stdout:AQB1wiNgRCisFhAAXTTug/zXvbqGFfamBUFBxQ== 2021-02-10T11:24:48.207 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:48.208 7f83b4ada700 1 -- 172.21.2.102:0/599758812 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f83940186c0 msgr2=0x7f839401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:48.207 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:48.208 7f83b4ada700 1 --2- 172.21.2.102:0/599758812 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f83940186c0 0x7f839401abc0 crc :-1 s=READY pgs=648 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:48.207 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:48.208 7f83b4ada700 1 -- 172.21.2.102:0/599758812 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f83b01b23c0 msgr2=0x7f83b008a570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:48.207 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:48.208 7f83b4ada700 1 --2- 172.21.2.102:0/599758812 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f83b01b23c0 0x7f83b008a570 secure :-1 s=READY pgs=1406 cs=0 l=1 rev1=1 rx=0x7f83a4001c60 tx=0x7f83a400a8d0).stop 2021-02-10T11:24:48.207 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:48.208 7f83b4ada700 1 -- 172.21.2.102:0/599758812 shutdown_connections 2021-02-10T11:24:48.207 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:48.208 7f83b4ada700 1 --2- 172.21.2.102:0/599758812 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f83940186c0 0x7f839401abc0 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:48.207 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:48.208 7f83b4ada700 1 --2- 172.21.2.102:0/599758812 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f83b008ab90 0x7f83b008af90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:48.207 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:48.208 7f83b4ada700 1 --2- 172.21.2.102:0/599758812 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f83b01b23c0 0x7f83b008a570 unknown :-1 s=CLOSED pgs=1406 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:48.207 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:48.208 7f83b4ada700 1 --2- 172.21.2.102:0/599758812 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f83b00ac260 0x7f83b01b1da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:48.207 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:48.208 7f83b4ada700 1 -- 172.21.2.102:0/599758812 >> 172.21.2.102:0/599758812 conn(0x7f83b010eab0 msgr2=0x7f83b01a3c70 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:48.208 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:48.208 7f83b4ada700 1 -- 172.21.2.102:0/599758812 shutdown_connections 2021-02-10T11:24:48.208 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:48.208 7f83b4ada700 1 -- 172.21.2.102:0/599758812 wait complete. 2021-02-10T11:24:48.215 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:24:48.215 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:24:48.215 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:24:48.549 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:24:48.550 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2021-02-10T11:24:48.550 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@gibba002.front.sepia.ceph.com... 2021-02-10T11:24:48.550 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-02-10T11:24:48.550 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:24:48.905 INFO:tasks.cephfs.fuse_mount.ceph-fuse.alice.gibba002.stderr:ceph-fuse[155213]: fuse finished with error 0 and tester_r 0 2021-02-10T11:24:48.906 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:24:48.906 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:24:48.906 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:24:49.241 INFO:teuthology.orchestra.run.gibba002.stdout:ext2/ext3 2021-02-10T11:24:49.241 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2021-02-10T11:24:49.241 INFO:teuthology.orchestra.run:waiting for 900 2021-02-10T11:24:55.343 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:24:55.343 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:24:55.677 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:24:56.104 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:56.089 7fe69fea6700 1 Processor -- start 2021-02-10T11:24:56.104 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:56.089 7fe69fea6700 1 -- start start 2021-02-10T11:24:56.104 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:56.089 7fe69fea6700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fe6980e53e0 0x7fe6981b0550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:56.104 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:56.089 7fe69fea6700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fe6980842e0 0x7fe698082100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:56.105 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:56.089 7fe69fea6700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe698082d50 con 0x7fe6981b0b70 2021-02-10T11:24:56.105 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:56.089 7fe69fea6700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6980e5fe0 con 0x7fe6980e53e0 2021-02-10T11:24:56.105 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:56.089 7fe69fea6700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6980e57e0 con 0x7fe6980842e0 2021-02-10T11:24:56.105 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:56.090 7fe69dcc2700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fe6980e53e0 0x7fe6981b0550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:56.105 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:56.090 7fe69d4c1700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fe6981b0b70 0x7fe698081910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:49716/0 (socket says 172.21.2.102:49716) 2021-02-10T11:24:56.105 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:56.090 7fe69d4c1700 1 -- 172.21.2.102:0/1900273500 learned_addr learned my addr 172.21.2.102:0/1900273500 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:24:56.105 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:56.090 7fe69dcc2700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fe6980e53e0 0x7fe6981b0550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:57492/0 (socket says 172.21.2.102:57492) 2021-02-10T11:24:56.105 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:56.090 7fe69dcc2700 1 -- 172.21.2.102:0/1900273500 >> v2:172.21.2.113:3300/0 conn(0x7fe6980842e0 msgr2=0x7fe698082100 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:24:56.105 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:56.090 7fe69dcc2700 1 --2- 172.21.2.102:0/1900273500 >> v2:172.21.2.113:3300/0 conn(0x7fe6980842e0 0x7fe698082100 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:56.105 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:56.090 7fe69dcc2700 1 -- 172.21.2.102:0/1900273500 >> v1:172.21.2.102:6789/0 conn(0x7fe6981b0b70 legacy=0x7fe698081910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:56.106 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:56.090 7fe69dcc2700 1 -- 172.21.2.102:0/1900273500 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6980846e0 con 0x7fe6980e53e0 2021-02-10T11:24:56.106 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:56.090 7fe69dcc2700 1 --2- 172.21.2.102:0/1900273500 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe6980e53e0 0x7fe6981b0550 secure :-1 s=READY pgs=1408 cs=0 l=1 rev1=1 rx=0x7fe68c006650 tx=0x7fe68c009a80).ready entity=mon.0 client_cookie=a6909ed3a716dbae server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:56.106 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:56.090 7fe69ccc0700 1 -- 172.21.2.102:0/1900273500 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe68c00c0c0 con 0x7fe6980e53e0 2021-02-10T11:24:56.106 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:56.090 7fe69ccc0700 1 -- 172.21.2.102:0/1900273500 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe68c00c340 con 0x7fe6980e53e0 2021-02-10T11:24:56.106 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:56.090 7fe69ccc0700 1 -- 172.21.2.102:0/1900273500 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe68c00c860 con 0x7fe6980e53e0 2021-02-10T11:24:56.106 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:56.090 7fe69fea6700 1 -- 172.21.2.102:0/1900273500 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe6980e53e0 msgr2=0x7fe6981b0550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:56.106 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:56.090 7fe69fea6700 1 --2- 172.21.2.102:0/1900273500 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe6980e53e0 0x7fe6981b0550 secure :-1 s=READY pgs=1408 cs=0 l=1 rev1=1 rx=0x7fe68c006650 tx=0x7fe68c009a80).stop 2021-02-10T11:24:56.106 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:56.091 7fe69fea6700 1 -- 172.21.2.102:0/1900273500 shutdown_connections 2021-02-10T11:24:56.106 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:56.091 7fe69fea6700 1 --2- 172.21.2.102:0/1900273500 >> v2:172.21.2.113:3300/0 conn(0x7fe6980842e0 0x7fe698082100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:56.106 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:56.091 7fe69fea6700 1 --2- 172.21.2.102:0/1900273500 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe6980e53e0 0x7fe6981b0550 unknown :-1 s=CLOSED pgs=1408 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:56.107 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:56.091 7fe69fea6700 1 -- 172.21.2.102:0/1900273500 >> 172.21.2.102:0/1900273500 conn(0x7fe6980ce500 msgr2=0x7fe6980e4610 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:56.107 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:56.091 7fe69fea6700 1 -- 172.21.2.102:0/1900273500 shutdown_connections 2021-02-10T11:24:56.107 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:56.091 7fe69fea6700 1 -- 172.21.2.102:0/1900273500 wait complete. 2021-02-10T11:24:56.109 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:56.106 7fe69fea6700 1 Processor -- start 2021-02-10T11:24:56.109 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:56.106 7fe69fea6700 1 -- start start 2021-02-10T11:24:56.109 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:56.106 7fe69fea6700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe69819b810 0x7fe6980ada70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:56.109 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:56.106 7fe69fea6700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe6980ae090 0x7fe6981a2900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:56.109 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:56.106 7fe69fea6700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe6981a2f20 0x7fe69809e270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:56.110 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:56.106 7fe69fea6700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe69819be20 con 0x7fe6981a2f20 2021-02-10T11:24:56.110 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:56.106 7fe69fea6700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6981a3390 con 0x7fe69819b810 2021-02-10T11:24:56.110 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:56.106 7fe69fea6700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6981a35b0 con 0x7fe6980ae090 2021-02-10T11:24:56.110 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:56.106 7fe69e4c3700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe6981a2f20 0x7fe69809e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:56.110 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:56.106 7fe69e4c3700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe6981a2f20 0x7fe69809e270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:57498/0 (socket says 172.21.2.102:57498) 2021-02-10T11:24:56.111 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:56.106 7fe69e4c3700 1 -- 172.21.2.102:0/2843809605 learned_addr learned my addr 172.21.2.102:0/2843809605 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:24:56.111 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:56.107 7fe69d4c1700 1 --2- 172.21.2.102:0/2843809605 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe6980ae090 0x7fe6981a2900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:56.111 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:56.107 7fe69dcc2700 1 --2- 172.21.2.102:0/2843809605 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe69819b810 0x7fe6980ada70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:56.111 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:56.107 7fe69e4c3700 1 -- 172.21.2.102:0/2843809605 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe6980ae090 msgr2=0x7fe6981a2900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:56.111 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:56.107 7fe69e4c3700 1 --2- 172.21.2.102:0/2843809605 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe6980ae090 0x7fe6981a2900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:56.111 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:56.107 7fe69e4c3700 1 -- 172.21.2.102:0/2843809605 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe69819b810 msgr2=0x7fe6980ada70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:56.111 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:56.107 7fe69e4c3700 1 --2- 172.21.2.102:0/2843809605 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe69819b810 0x7fe6980ada70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:56.112 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:56.107 7fe69e4c3700 1 -- 172.21.2.102:0/2843809605 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe69809e890 con 0x7fe6981a2f20 2021-02-10T11:24:56.112 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:56.107 7fe69dcc2700 1 --2- 172.21.2.102:0/2843809605 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe69819b810 0x7fe6980ada70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:24:56.112 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:56.107 7fe69e4c3700 1 --2- 172.21.2.102:0/2843809605 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe6981a2f20 0x7fe69809e270 secure :-1 s=READY pgs=1409 cs=0 l=1 rev1=1 rx=0x7fe690009960 tx=0x7fe69000bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:56.112 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:56.107 7fe68a7fc700 1 -- 172.21.2.102:0/2843809605 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe690003030 con 0x7fe6981a2f20 2021-02-10T11:24:56.112 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:56.107 7fe68a7fc700 1 -- 172.21.2.102:0/2843809605 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe690001310 con 0x7fe6981a2f20 2021-02-10T11:24:56.112 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:56.107 7fe68a7fc700 1 -- 172.21.2.102:0/2843809605 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe6900017f0 con 0x7fe6981a2f20 2021-02-10T11:24:56.112 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:56.107 7fe69fea6700 1 -- 172.21.2.102:0/2843809605 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe69809e890 con 0x7fe6981a2f20 2021-02-10T11:24:56.112 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:56.107 7fe69fea6700 1 -- 172.21.2.102:0/2843809605 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe69809e890 con 0x7fe6981a2f20 2021-02-10T11:24:56.112 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:56.108 7fe68a7fc700 1 -- 172.21.2.102:0/2843809605 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fe690000d10 con 0x7fe6981a2f20 2021-02-10T11:24:56.112 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:56.108 7fe68a7fc700 1 --2- 172.21.2.102:0/2843809605 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe6800186c0 0x7fe68001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:56.112 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:56.108 7fe68a7fc700 1 -- 172.21.2.102:0/2843809605 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fe6900327c0 con 0x7fe6981a2f20 2021-02-10T11:24:56.112 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:56.108 7fe69dcc2700 1 --2- 172.21.2.102:0/2843809605 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe6800186c0 0x7fe68001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:56.112 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:56.108 7fe69dcc2700 1 --2- 172.21.2.102:0/2843809605 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe6800186c0 0x7fe68001abc0 crc :-1 s=READY pgs=649 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:56.138 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:56.139 7fe69fea6700 1 -- 172.21.2.102:0/2843809605 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6980ed670 con 0x7fe6981a2f20 2021-02-10T11:24:56.141 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:56.142 7fe68a7fc700 1 -- 172.21.2.102:0/2843809605 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fe69000be30 con 0x7fe6981a2f20 2021-02-10T11:24:56.265 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:56.266 7fe69fea6700 1 -- 172.21.2.102:0/2843809605 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe6980ed670 con 0x7fe6981a2f20 2021-02-10T11:24:56.266 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:56.267 7fe68a7fc700 1 -- 172.21.2.102:0/2843809605 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v57) v1 ==== 74+0+13996 (secure 0 0 0) 0x7fe690000d10 con 0x7fe6981a2f20 2021-02-10T11:24:56.266 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:24:56.266 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":57,"fsid":"68db30ea-dbce-4aef-a6df-f01c6fea0513","created":"2021-02-10 11:03:06.331557","modified":"2021-02-10 11:24:11.200509","last_up_change":"2021-02-10 11:03:32.690562","last_in_change":"2021-02-10 11:03:32.690562","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:03:37.322519","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:24:08.122188","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":11,"pool_name":"cephfs_data","create_time":"2021-02-10 11:24:09.191606","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"be691b4d-a039-408a-af80-9d53f02b6f1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6818","nonce":36088},{"type":"v1","addr":"172.21.2.102:6820","nonce":36088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6822","nonce":36088},{"type":"v1","addr":"172.21.2.102:6824","nonce":36088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6830","nonce":36088},{"type":"v1","addr":"172.21.2.102:6832","nonce":36088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6826","nonce":36088},{"type":"v1","addr":"172.21.2.102:6828","nonce":36088}]},"public_addr":"172.21.2.102:6820/36088","cluster_addr":"172.21.2.102:6824/36088","heartbeat_back_addr":"172.21.2.102:6832/36088","heartbeat_front_addr":"172.21.2.102:6828/36088","state":["exists","up"]},{"osd":1,"uuid":"64e3ccc8-9c69-4d7c-865e-af8e83822635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6819","nonce":36087},{"type":"v1","addr":"172.21.2.102:6821","nonce":36087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6823","nonce":36087},{"type":"v1","addr":"172.21.2.102:6825","nonce":36087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6831","nonce":36087},{"type":"v1","addr":"172.21.2.102:6833","nonce":36087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6827","nonce":36087},{"type":"v1","addr":"172.21.2.102:6829","nonce":36087}]},"public_addr":"172.21.2.102:6821/36087","cluster_addr":"172.21.2.102:6825/36087","heartbeat_back_addr":"172.21.2.102:6833/36087","heartbeat_front_addr":"172.21.2.102:6829/36087","state":["exists","up"]},{"osd":2,"uuid":"153700af-747f-481f-bdf4-d6072d3a842f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6810","nonce":36089},{"type":"v1","addr":"172.21.2.102:6811","nonce":36089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6812","nonce":36089},{"type":"v1","addr":"172.21.2.102:6813","nonce":36089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6816","nonce":36089},{"type":"v1","addr":"172.21.2.102:6817","nonce":36089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6814","nonce":36089},{"type":"v1","addr":"172.21.2.102:6815","nonce":36089}]},"public_addr":"172.21.2.102:6811/36089","cluster_addr":"172.21.2.102:6813/36089","heartbeat_back_addr":"172.21.2.102:6817/36089","heartbeat_front_addr":"172.21.2.102:6815/36089","state":["exists","up"]},{"osd":3,"uuid":"cbc678f4-83a9-45e6-8830-2f7c232b1ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6802","nonce":36090},{"type":"v1","addr":"172.21.2.102:6803","nonce":36090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6804","nonce":36090},{"type":"v1","addr":"172.21.2.102:6805","nonce":36090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6808","nonce":36090},{"type":"v1","addr":"172.21.2.102:6809","nonce":36090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6806","nonce":36090},{"type":"v1","addr":"172.21.2.102:6807","nonce":36090}]},"public_addr":"172.21.2.102:6803/36090","cluster_addr":"172.21.2.102:6805/36090","heartbeat_back_addr":"172.21.2.102:6809/36090","heartbeat_front_addr":"172.21.2.102:6807/36090","state":["exists","up"]},{"osd":4,"uuid":"c01e4008-b258-4f10-afd2-9e7f37214351","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6808","nonce":36368},{"type":"v1","addr":"172.21.2.113:6811","nonce":36368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6814","nonce":36368},{"type":"v1","addr":"172.21.2.113:6818","nonce":36368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6826","nonce":36368},{"type":"v1","addr":"172.21.2.113:6828","nonce":36368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6820","nonce":36368},{"type":"v1","addr":"172.21.2.113:6824","nonce":36368}]},"public_addr":"172.21.2.113:6811/36368","cluster_addr":"172.21.2.113:6818/36368","heartbeat_back_addr":"172.21.2.113:6828/36368","heartbeat_front_addr":"172.21.2.113:6824/36368","state":["exists","up"]},{"osd":5,"uuid":"551f6007-daf8-4a98-8ee0-c2b16bcccb5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6802","nonce":36366},{"type":"v1","addr":"172.21.2.113:6805","nonce":36366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6807","nonce":36366},{"type":"v1","addr":"172.21.2.113:6810","nonce":36366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6819","nonce":36366},{"type":"v1","addr":"172.21.2.113:6822","nonce":36366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6812","nonce":36366},{"type":"v1","addr":"172.21.2.113:6815","nonce":36366}]},"public_addr":"172.21.2.113:6805/36366","cluster_addr":"172.21.2.113:6810/36366","heartbeat_back_addr":"172.21.2.113:6822/36366","heartbeat_front_addr":"172.21.2.113:6815/36366","state":["exists","up"]},{"osd":6,"uuid":"bf67c040-0e50-48be-b102-c3943bc9ebef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6800","nonce":36367},{"type":"v1","addr":"172.21.2.113:6801","nonce":36367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6803","nonce":36367},{"type":"v1","addr":"172.21.2.113:6804","nonce":36367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6813","nonce":36367},{"type":"v1","addr":"172.21.2.113:6816","nonce":36367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6806","nonce":36367},{"type":"v1","addr":"172.21.2.113:6809","nonce":36367}]},"public_addr":"172.21.2.113:6801/36367","cluster_addr":"172.21.2.113:6804/36367","heartbeat_back_addr":"172.21.2.113:6816/36367","heartbeat_front_addr":"172.21.2.113:6809/36367","state":["exists","up"]},{"osd":7,"uuid":"f5461f80-018d-428f-be89-2eb2bcbe7bb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6817","nonce":36365},{"type":"v1","addr":"172.21.2.113:6821","nonce":36365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6823","nonce":36365},{"type":"v1","addr":"172.21.2.113:6825","nonce":36365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6830","nonce":36365},{"type":"v1","addr":"172.21.2.113:6831","nonce":36365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6827","nonce":36365},{"type":"v1","addr":"172.21.2.113:6829","nonce":36365}]},"public_addr":"172.21.2.113:6821/36365","cluster_addr":"172.21.2.113:6825/36365","heartbeat_back_addr":"172.21.2.113:6831/36365","heartbeat_front_addr":"172.21.2.113:6829/36365","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:24:56.268 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:56.269 7fe69fea6700 1 -- 172.21.2.102:0/2843809605 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe6800186c0 msgr2=0x7fe68001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:56.268 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:56.269 7fe69fea6700 1 --2- 172.21.2.102:0/2843809605 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe6800186c0 0x7fe68001abc0 crc :-1 s=READY pgs=649 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:56.268 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:56.269 7fe69fea6700 1 -- 172.21.2.102:0/2843809605 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe6981a2f20 msgr2=0x7fe69809e270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:56.268 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:56.269 7fe69fea6700 1 --2- 172.21.2.102:0/2843809605 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe6981a2f20 0x7fe69809e270 secure :-1 s=READY pgs=1409 cs=0 l=1 rev1=1 rx=0x7fe690009960 tx=0x7fe69000bcf0).stop 2021-02-10T11:24:56.268 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:56.269 7fe69fea6700 1 -- 172.21.2.102:0/2843809605 shutdown_connections 2021-02-10T11:24:56.268 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:56.269 7fe69fea6700 1 --2- 172.21.2.102:0/2843809605 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe6981a2f20 0x7fe69809e270 unknown :-1 s=CLOSED pgs=1409 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:56.268 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:56.269 7fe69fea6700 1 --2- 172.21.2.102:0/2843809605 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe6980ae090 0x7fe6981a2900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:56.268 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:56.269 7fe69fea6700 1 --2- 172.21.2.102:0/2843809605 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe6800186c0 0x7fe68001abc0 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:56.268 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:56.269 7fe69fea6700 1 --2- 172.21.2.102:0/2843809605 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe69819b810 0x7fe6980ada70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:56.268 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:56.269 7fe69fea6700 1 -- 172.21.2.102:0/2843809605 >> 172.21.2.102:0/2843809605 conn(0x7fe6981b92f0 msgr2=0x7fe6981ac4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:56.269 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:56.269 7fe69fea6700 1 -- 172.21.2.102:0/2843809605 shutdown_connections 2021-02-10T11:24:56.269 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:56.269 7fe69fea6700 1 -- 172.21.2.102:0/2843809605 wait complete. 2021-02-10T11:24:56.276 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:03:37.322519', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 10, 'pool_name': 'cephfs_metadata', 'create_time': '2021-02-10 11:24:08.122188', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '56', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 11, 'pool_name': 'cephfs_data', 'create_time': '2021-02-10 11:24:09.191606', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '57', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-02-10T11:24:56.277 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:24:56.707 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:56.692 7eff6e636700 1 Processor -- start 2021-02-10T11:24:56.716 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:56.692 7eff6e636700 1 -- start start 2021-02-10T11:24:56.716 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:56.692 7eff6e636700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7eff680e5170 0x7eff681b38e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:56.717 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:56.692 7eff6e636700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7eff680e5570 0x7eff6807a120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:56.717 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:56.692 7eff6e636700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff68078b90 con 0x7eff681b3f00 2021-02-10T11:24:56.717 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:56.692 7eff6e636700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff680d1900 con 0x7eff680e5570 2021-02-10T11:24:56.717 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:56.692 7eff6e636700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff680d1af0 con 0x7eff680e5170 2021-02-10T11:24:56.717 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:56.692 7eff6cc53700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7eff680e5570 0x7eff6807a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:56.717 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:56.692 7eff6cc53700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7eff680e5570 0x7eff6807a120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:57504/0 (socket says 172.21.2.102:57504) 2021-02-10T11:24:56.717 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:56.692 7eff6cc53700 1 -- 172.21.2.102:0/676188356 learned_addr learned my addr 172.21.2.102:0/676188356 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:24:56.717 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:56.693 7eff6cc53700 1 -- 172.21.2.102:0/676188356 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7eff680e5170 msgr2=0x7eff681b38e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:24:56.717 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:56.693 7eff6cc53700 1 --2- 172.21.2.102:0/676188356 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7eff680e5170 0x7eff681b38e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:56.717 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:56.693 7eff6cc53700 1 -- 172.21.2.102:0/676188356 >> v1:172.21.2.113:6789/0 conn(0x7eff681b3f00 legacy=0x7eff68077510 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:24:56.718 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:56.693 7eff6cc53700 1 -- 172.21.2.102:0/676188356 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff680d1ce0 con 0x7eff680e5570 2021-02-10T11:24:56.718 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:56.693 7eff6cc53700 1 --2- 172.21.2.102:0/676188356 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7eff680e5570 0x7eff6807a120 secure :-1 s=READY pgs=1410 cs=0 l=1 rev1=1 rx=0x7eff60006f00 tx=0x7eff60009f10).ready entity=mon.0 client_cookie=e237f674074e53b6 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:56.718 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:56.693 7eff66ffd700 1 -- 172.21.2.102:0/676188356 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7eff6000b3b0 con 0x7eff680e5570 2021-02-10T11:24:56.718 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:56.693 7eff66ffd700 1 -- 172.21.2.102:0/676188356 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff6000b630 con 0x7eff680e5570 2021-02-10T11:24:56.718 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:56.693 7eff66ffd700 1 -- 172.21.2.102:0/676188356 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7eff6000bb10 con 0x7eff680e5570 2021-02-10T11:24:56.718 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:56.693 7eff6e636700 1 -- 172.21.2.102:0/676188356 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7eff680e5570 msgr2=0x7eff6807a120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:56.718 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:56.693 7eff6e636700 1 --2- 172.21.2.102:0/676188356 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7eff680e5570 0x7eff6807a120 secure :-1 s=READY pgs=1410 cs=0 l=1 rev1=1 rx=0x7eff60006f00 tx=0x7eff60009f10).stop 2021-02-10T11:24:56.718 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:56.694 7eff6e636700 1 -- 172.21.2.102:0/676188356 shutdown_connections 2021-02-10T11:24:56.718 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:56.694 7eff6e636700 1 --2- 172.21.2.102:0/676188356 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7eff680e5570 0x7eff6807a120 unknown :-1 s=CLOSED pgs=1410 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:56.719 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:56.694 7eff6e636700 1 --2- 172.21.2.102:0/676188356 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7eff680e5170 0x7eff681b38e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:56.719 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:56.694 7eff6e636700 1 -- 172.21.2.102:0/676188356 >> 172.21.2.102:0/676188356 conn(0x7eff680ce820 msgr2=0x7eff680e4660 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:56.719 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:56.694 7eff6e636700 1 -- 172.21.2.102:0/676188356 shutdown_connections 2021-02-10T11:24:56.719 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:56.694 7eff6e636700 1 -- 172.21.2.102:0/676188356 wait complete. 2021-02-10T11:24:56.719 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:56.708 7eff6e636700 1 Processor -- start 2021-02-10T11:24:56.719 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:56.708 7eff6e636700 1 -- start start 2021-02-10T11:24:56.721 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:56.708 7eff6e636700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7eff680bb960 0x7eff6818eab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:56.721 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:56.708 7eff6e636700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7eff6819fb50 0x7eff6819ff50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:56.721 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:56.708 7eff6e636700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7eff681a0570 0x7eff680bc640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:56.721 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:56.708 7eff6e636700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff680bbd60 con 0x7eff680bb960 2021-02-10T11:24:56.721 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:56.708 7eff6e636700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff6818f0d0 con 0x7eff6819fb50 2021-02-10T11:24:56.721 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:56.708 7eff6e636700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff6818c590 con 0x7eff681a0570 2021-02-10T11:24:56.722 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:56.709 7eff67fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7eff680bb960 0x7eff6818eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:56.722 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:56.709 7eff67fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7eff680bb960 0x7eff6818eab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:57512/0 (socket says 172.21.2.102:57512) 2021-02-10T11:24:56.722 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:56.709 7eff67fff700 1 -- 172.21.2.102:0/2289814360 learned_addr learned my addr 172.21.2.102:0/2289814360 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:24:56.722 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:56.709 7eff6cc53700 1 --2- 172.21.2.102:0/2289814360 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7eff681a0570 0x7eff680bc640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:56.722 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:56.709 7eff677fe700 1 --2- 172.21.2.102:0/2289814360 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7eff6819fb50 0x7eff6819ff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:56.722 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:56.709 7eff67fff700 1 -- 172.21.2.102:0/2289814360 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7eff681a0570 msgr2=0x7eff680bc640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:56.722 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:56.709 7eff67fff700 1 --2- 172.21.2.102:0/2289814360 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7eff681a0570 0x7eff680bc640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:56.722 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:56.709 7eff67fff700 1 -- 172.21.2.102:0/2289814360 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7eff6819fb50 msgr2=0x7eff6819ff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:56.722 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:56.709 7eff67fff700 1 --2- 172.21.2.102:0/2289814360 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7eff6819fb50 0x7eff6819ff50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:56.722 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:56.709 7eff67fff700 1 -- 172.21.2.102:0/2289814360 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff6818c1d0 con 0x7eff680bb960 2021-02-10T11:24:56.722 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:56.709 7eff67fff700 1 --2- 172.21.2.102:0/2289814360 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7eff680bb960 0x7eff6818eab0 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 rx=0x7eff5800b5c0 tx=0x7eff5800b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:56.723 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:56.709 7eff64ff9700 1 -- 172.21.2.102:0/2289814360 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7eff580009b0 con 0x7eff680bb960 2021-02-10T11:24:56.723 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:56.709 7eff64ff9700 1 -- 172.21.2.102:0/2289814360 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff5800bd50 con 0x7eff680bb960 2021-02-10T11:24:56.723 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:56.709 7eff64ff9700 1 -- 172.21.2.102:0/2289814360 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7eff58006240 con 0x7eff680bb960 2021-02-10T11:24:56.723 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:56.709 7eff6e636700 1 -- 172.21.2.102:0/2289814360 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff6818c1d0 con 0x7eff680bb960 2021-02-10T11:24:56.723 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:56.709 7eff6e636700 1 -- 172.21.2.102:0/2289814360 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff6818c1d0 con 0x7eff680bb960 2021-02-10T11:24:56.723 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:56.710 7eff64ff9700 1 -- 172.21.2.102:0/2289814360 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7eff58006030 con 0x7eff680bb960 2021-02-10T11:24:56.723 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:56.710 7eff64ff9700 1 --2- 172.21.2.102:0/2289814360 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7eff480186b0 0x7eff4801abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:56.723 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:56.710 7eff64ff9700 1 -- 172.21.2.102:0/2289814360 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5573+0+0 (secure 0 0 0) 0x7eff58032600 con 0x7eff680bb960 2021-02-10T11:24:56.723 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:56.710 7eff677fe700 1 --2- 172.21.2.102:0/2289814360 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7eff480186b0 0x7eff4801abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:56.724 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:56.710 7eff677fe700 1 --2- 172.21.2.102:0/2289814360 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7eff480186b0 0x7eff4801abd0 crc :-1 s=READY pgs=650 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:56.741 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:56.742 7eff6e636700 1 -- 172.21.2.102:0/2289814360 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff6810b960 con 0x7eff680bb960 2021-02-10T11:24:56.743 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:56.744 7eff64ff9700 1 -- 172.21.2.102:0/2289814360 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7eff58006b80 con 0x7eff680bb960 2021-02-10T11:24:56.870 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:56.871 7eff6e636700 1 -- 172.21.2.102:0/2289814360 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7eff6810b960 con 0x7eff680bb960 2021-02-10T11:24:56.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:56.872 7eff64ff9700 1 -- 172.21.2.102:0/2289814360 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v57) v1 ==== 92+0+10 (secure 0 0 0) 0x7eff5800bd50 con 0x7eff680bb960 2021-02-10T11:24:56.871 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:24:56.872 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:56.874 7eff6e636700 1 -- 172.21.2.102:0/2289814360 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7eff480186b0 msgr2=0x7eff4801abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:56.872 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:56.874 7eff6e636700 1 --2- 172.21.2.102:0/2289814360 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7eff480186b0 0x7eff4801abd0 crc :-1 s=READY pgs=650 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:56.872 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:56.874 7eff6e636700 1 -- 172.21.2.102:0/2289814360 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7eff680bb960 msgr2=0x7eff6818eab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:56.873 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:56.874 7eff6e636700 1 --2- 172.21.2.102:0/2289814360 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7eff680bb960 0x7eff6818eab0 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 rx=0x7eff5800b5c0 tx=0x7eff5800b960).stop 2021-02-10T11:24:56.873 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:56.874 7eff6e636700 1 -- 172.21.2.102:0/2289814360 shutdown_connections 2021-02-10T11:24:56.873 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:56.874 7eff6e636700 1 --2- 172.21.2.102:0/2289814360 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7eff681a0570 0x7eff680bc640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:56.873 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:56.874 7eff6e636700 1 --2- 172.21.2.102:0/2289814360 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7eff480186b0 0x7eff4801abd0 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:56.873 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:56.874 7eff6e636700 1 --2- 172.21.2.102:0/2289814360 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7eff6819fb50 0x7eff6819ff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:56.873 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:56.874 7eff6e636700 1 --2- 172.21.2.102:0/2289814360 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7eff680bb960 0x7eff6818eab0 unknown :-1 s=CLOSED pgs=1411 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:56.874 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:56.874 7eff6e636700 1 -- 172.21.2.102:0/2289814360 >> 172.21.2.102:0/2289814360 conn(0x7eff6809ffc0 msgr2=0x7eff6810be80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:56.874 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:56.874 7eff6e636700 1 -- 172.21.2.102:0/2289814360 shutdown_connections 2021-02-10T11:24:56.874 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:56.874 7eff6e636700 1 -- 172.21.2.102:0/2289814360 wait complete. 2021-02-10T11:24:56.882 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-02-10T11:24:57.313 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:57.296 7ff0bab7f700 1 Processor -- start 2021-02-10T11:24:57.313 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:57.296 7ff0bab7f700 1 -- start start 2021-02-10T11:24:57.313 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:57.297 7ff0bab7f700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7ff0b40e5170 0x7ff0b41b38e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:57.313 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:57.297 7ff0bab7f700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0b4078b90 con 0x7ff0b41b3f00 2021-02-10T11:24:57.313 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:57.297 7ff0bab7f700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0b40784f0 con 0x7ff0b40e5570 2021-02-10T11:24:57.313 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:57.297 7ff0bab7f700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0b4078710 con 0x7ff0b40e5170 2021-02-10T11:24:57.314 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:57.297 7ff0b3fff700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7ff0b41b3f00 0x7ff0b4077510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:49742/0 (socket says 172.21.2.102:49742) 2021-02-10T11:24:57.314 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:57.297 7ff0b3fff700 1 -- 172.21.2.102:0/1796014757 learned_addr learned my addr 172.21.2.102:0/1796014757 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:24:57.314 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:57.297 7ff0b37fe700 1 -- 172.21.2.102:0/1796014757 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3482417848 0 0) 0x7ff0a4001060 con 0x7ff0b41b3f00 2021-02-10T11:24:57.314 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:57.297 7ff0b37fe700 1 -- 172.21.2.102:0/1796014757 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0a0002fe0 con 0x7ff0b41b3f00 2021-02-10T11:24:57.314 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:57.298 7ff0b37fe700 1 -- 172.21.2.102:0/1796014757 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2505497830 0 0) 0x7ff0a4001460 con 0x7ff0b41b3f00 2021-02-10T11:24:57.314 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:57.298 7ff0b899b700 1 --2- 172.21.2.102:0/1796014757 >> v2:172.21.2.113:3300/0 conn(0x7ff0b40e5170 0x7ff0b41b38e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:57.314 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:57.298 7ff0b37fe700 1 -- 172.21.2.102:0/1796014757 >> v2:172.21.2.113:3300/0 conn(0x7ff0b40e5170 msgr2=0x7ff0b41b38e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:24:57.314 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:57.298 7ff0b37fe700 1 --2- 172.21.2.102:0/1796014757 >> v2:172.21.2.113:3300/0 conn(0x7ff0b40e5170 0x7ff0b41b38e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:57.314 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:57.298 7ff0b37fe700 1 -- 172.21.2.102:0/1796014757 >> v1:172.21.2.113:6789/0 conn(0x7ff0b40e5570 legacy=0x7ff0b4077d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:24:57.314 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:57.298 7ff0b37fe700 1 -- 172.21.2.102:0/1796014757 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0b4078930 con 0x7ff0b41b3f00 2021-02-10T11:24:57.314 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:57.298 7ff0b37fe700 1 -- 172.21.2.102:0/1796014757 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7ff0a40019e0 con 0x7ff0b41b3f00 2021-02-10T11:24:57.314 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:57.298 7ff0b37fe700 1 -- 172.21.2.102:0/1796014757 >> v1:172.21.2.102:6789/0 conn(0x7ff0b41b3f00 legacy=0x7ff0b4077510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:57.314 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:57.298 7ff0b37fe700 1 --2- 172.21.2.102:0/1796014757 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff0a0003b10 0x7ff0a0006010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:57.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:57.298 7ff0b37fe700 1 --2- 172.21.2.102:0/1796014757 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff0a00068b0 0x7ff0a0008dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:57.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:57.298 7ff0b37fe700 1 --2- 172.21.2.102:0/1796014757 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff0a0009640 0x7ff0a000bb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:57.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:57.298 7ff0b37fe700 1 -- 172.21.2.102:0/1796014757 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0a000c3b0 con 0x7ff0a0009640 2021-02-10T11:24:57.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:57.298 7ff0b37fe700 1 -- 172.21.2.102:0/1796014757 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0a000c5d0 con 0x7ff0a0003b10 2021-02-10T11:24:57.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:57.298 7ff0b37fe700 1 -- 172.21.2.102:0/1796014757 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0a000c7f0 con 0x7ff0a00068b0 2021-02-10T11:24:57.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:57.298 7ff0b899b700 1 --2- 172.21.2.102:0/1796014757 >> v2:172.21.2.113:3300/0 conn(0x7ff0b40e5170 0x7ff0b41b38e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:24:57.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:57.298 7ff0b919c700 1 --2- 172.21.2.102:0/1796014757 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff0a0009640 0x7ff0a000bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:57.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:57.298 7ff0b3fff700 1 --2- 172.21.2.102:0/1796014757 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff0a00068b0 0x7ff0a0008dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:57.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:57.298 7ff0b919c700 1 -- 172.21.2.102:0/1796014757 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff0a00068b0 msgr2=0x7ff0a0008dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:24:57.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:57.298 7ff0b919c700 1 --2- 172.21.2.102:0/1796014757 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff0a00068b0 0x7ff0a0008dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:57.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:57.298 7ff0b919c700 1 -- 172.21.2.102:0/1796014757 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff0a0003b10 msgr2=0x7ff0a0006010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:24:57.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:57.298 7ff0b919c700 1 --2- 172.21.2.102:0/1796014757 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff0a0003b10 0x7ff0a0006010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:57.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:57.298 7ff0b919c700 1 -- 172.21.2.102:0/1796014757 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0a000ca90 con 0x7ff0a0009640 2021-02-10T11:24:57.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:57.299 7ff0b919c700 1 -- 172.21.2.102:0/1796014757 reap_dead start 2021-02-10T11:24:57.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:57.299 7ff0b919c700 1 --2- 172.21.2.102:0/1796014757 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff0a0009640 0x7ff0a000bb60 secure :-1 s=READY pgs=1413 cs=0 l=1 rev1=1 rx=0x7ff0ac006ab0 tx=0x7ff0ac04da20).ready entity=mon.0 client_cookie=a1fb5c6f72f8386d server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:57.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:57.299 7ff0b37fe700 1 -- 172.21.2.102:0/1796014757 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff0ac0500c0 con 0x7ff0a0009640 2021-02-10T11:24:57.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:57.299 7ff0b37fe700 1 -- 172.21.2.102:0/1796014757 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0ac0066f0 con 0x7ff0a0009640 2021-02-10T11:24:57.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:57.299 7ff0b37fe700 1 -- 172.21.2.102:0/1796014757 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff0ac050640 con 0x7ff0a0009640 2021-02-10T11:24:57.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:57.299 7ff0bab7f700 1 -- 172.21.2.102:0/1796014757 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff0a0009640 msgr2=0x7ff0a000bb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:57.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:57.299 7ff0bab7f700 1 --2- 172.21.2.102:0/1796014757 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff0a0009640 0x7ff0a000bb60 secure :-1 s=READY pgs=1413 cs=0 l=1 rev1=1 rx=0x7ff0ac006ab0 tx=0x7ff0ac04da20).stop 2021-02-10T11:24:57.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:57.299 7ff0bab7f700 1 -- 172.21.2.102:0/1796014757 shutdown_connections 2021-02-10T11:24:57.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:57.299 7ff0bab7f700 1 --2- 172.21.2.102:0/1796014757 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff0a0009640 0x7ff0a000bb60 unknown :-1 s=CLOSED pgs=1413 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:57.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:57.299 7ff0bab7f700 1 -- 172.21.2.102:0/1796014757 >> 172.21.2.102:0/1796014757 conn(0x7ff0b40ce820 msgr2=0x7ff0b40e4660 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:57.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:57.299 7ff0bab7f700 1 -- 172.21.2.102:0/1796014757 shutdown_connections 2021-02-10T11:24:57.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:57.299 7ff0bab7f700 1 -- 172.21.2.102:0/1796014757 wait complete. 2021-02-10T11:24:57.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:57.314 7ff0bab7f700 1 Processor -- start 2021-02-10T11:24:57.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:57.314 7ff0bab7f700 1 -- start start 2021-02-10T11:24:57.318 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:57.314 7ff0bab7f700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff0b410cf50 0x7ff0b410d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:57.318 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:57.314 7ff0bab7f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff0b4096680 0x7ff0b4096a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:57.318 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:57.314 7ff0bab7f700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff0b40970a0 0x7ff0b40974a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:57.318 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:57.314 7ff0bab7f700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0b410da40 con 0x7ff0b4096680 2021-02-10T11:24:57.318 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:57.314 7ff0bab7f700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0b410dc60 con 0x7ff0b40970a0 2021-02-10T11:24:57.318 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:57.314 7ff0bab7f700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0b410de80 con 0x7ff0b410cf50 2021-02-10T11:24:57.318 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:57.315 7ff0b3fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff0b4096680 0x7ff0b4096a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:57.318 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:57.315 7ff0b3fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff0b4096680 0x7ff0b4096a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:57528/0 (socket says 172.21.2.102:57528) 2021-02-10T11:24:57.318 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:57.315 7ff0b3fff700 1 -- 172.21.2.102:0/3331917092 learned_addr learned my addr 172.21.2.102:0/3331917092 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:24:57.318 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:57.315 7ff0b3fff700 1 -- 172.21.2.102:0/3331917092 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff0b410cf50 msgr2=0x7ff0b410d350 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:24:57.318 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:57.315 7ff0b3fff700 1 --2- 172.21.2.102:0/3331917092 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff0b410cf50 0x7ff0b410d350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:57.319 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:57.315 7ff0b3fff700 1 -- 172.21.2.102:0/3331917092 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff0b40970a0 msgr2=0x7ff0b40974a0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:24:57.319 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:57.315 7ff0b3fff700 1 --2- 172.21.2.102:0/3331917092 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff0b40970a0 0x7ff0b40974a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:57.319 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:57.315 7ff0b3fff700 1 -- 172.21.2.102:0/3331917092 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0b410e0a0 con 0x7ff0b4096680 2021-02-10T11:24:57.319 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:57.315 7ff0b3fff700 1 --2- 172.21.2.102:0/3331917092 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff0b4096680 0x7ff0b4096a80 secure :-1 s=READY pgs=1414 cs=0 l=1 rev1=1 rx=0x7ff0a400a8e0 tx=0x7ff0a400ac80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:57.319 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:57.315 7ff0b17fa700 1 -- 172.21.2.102:0/3331917092 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff0a4004930 con 0x7ff0b4096680 2021-02-10T11:24:57.319 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:57.315 7ff0b17fa700 1 -- 172.21.2.102:0/3331917092 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0a4004cb0 con 0x7ff0b4096680 2021-02-10T11:24:57.319 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:57.315 7ff0b17fa700 1 -- 172.21.2.102:0/3331917092 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff0a4007350 con 0x7ff0b4096680 2021-02-10T11:24:57.319 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:57.315 7ff0bab7f700 1 -- 172.21.2.102:0/3331917092 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0b410e0a0 con 0x7ff0b4096680 2021-02-10T11:24:57.319 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:57.315 7ff0bab7f700 1 -- 172.21.2.102:0/3331917092 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0b410e630 con 0x7ff0b4096680 2021-02-10T11:24:57.319 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:57.316 7ff0b17fa700 1 -- 172.21.2.102:0/3331917092 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ff0a4007820 con 0x7ff0b4096680 2021-02-10T11:24:57.319 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:57.316 7ff0b17fa700 1 --2- 172.21.2.102:0/3331917092 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff0980186c0 0x7ff09801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:57.319 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:57.316 7ff0b17fa700 1 -- 172.21.2.102:0/3331917092 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5573+0+0 (secure 0 0 0) 0x7ff0a4032990 con 0x7ff0b4096680 2021-02-10T11:24:57.320 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:57.316 7ff0b899b700 1 --2- 172.21.2.102:0/3331917092 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff0980186c0 0x7ff09801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:57.320 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:57.316 7ff0b899b700 1 --2- 172.21.2.102:0/3331917092 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff0980186c0 0x7ff09801abc0 crc :-1 s=READY pgs=651 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:57.345 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:57.347 7ff0bab7f700 1 -- 172.21.2.102:0/3331917092 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0b410ca30 con 0x7ff0b4096680 2021-02-10T11:24:57.348 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:57.349 7ff0b17fa700 1 -- 172.21.2.102:0/3331917092 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ff0a40075f0 con 0x7ff0b4096680 2021-02-10T11:24:57.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:57.476 7ff0bab7f700 1 -- 172.21.2.102:0/3331917092 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7ff0b40fe290 con 0x7ff0b4096680 2021-02-10T11:24:57.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:57.477 7ff0b17fa700 1 -- 172.21.2.102:0/3331917092 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v57) v1 ==== 104+0+10 (secure 0 0 0) 0x7ff0a4004cb0 con 0x7ff0b4096680 2021-02-10T11:24:57.476 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:24:57.477 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:57.478 7ff0bab7f700 1 -- 172.21.2.102:0/3331917092 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff0980186c0 msgr2=0x7ff09801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:57.477 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:57.478 7ff0bab7f700 1 --2- 172.21.2.102:0/3331917092 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff0980186c0 0x7ff09801abc0 crc :-1 s=READY pgs=651 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:57.477 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:57.478 7ff0bab7f700 1 -- 172.21.2.102:0/3331917092 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff0b4096680 msgr2=0x7ff0b4096a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:57.477 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:57.478 7ff0bab7f700 1 --2- 172.21.2.102:0/3331917092 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff0b4096680 0x7ff0b4096a80 secure :-1 s=READY pgs=1414 cs=0 l=1 rev1=1 rx=0x7ff0a400a8e0 tx=0x7ff0a400ac80).stop 2021-02-10T11:24:57.478 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:57.479 7ff0bab7f700 1 -- 172.21.2.102:0/3331917092 shutdown_connections 2021-02-10T11:24:57.478 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:57.479 7ff0bab7f700 1 --2- 172.21.2.102:0/3331917092 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff0980186c0 0x7ff09801abc0 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:57.478 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:57.479 7ff0bab7f700 1 --2- 172.21.2.102:0/3331917092 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff0b40970a0 0x7ff0b40974a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:57.478 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:57.479 7ff0bab7f700 1 --2- 172.21.2.102:0/3331917092 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff0b4096680 0x7ff0b4096a80 unknown :-1 s=CLOSED pgs=1414 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:57.478 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:57.479 7ff0bab7f700 1 --2- 172.21.2.102:0/3331917092 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff0b410cf50 0x7ff0b410d350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:57.478 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:57.479 7ff0bab7f700 1 -- 172.21.2.102:0/3331917092 >> 172.21.2.102:0/3331917092 conn(0x7ff0b41900f0 msgr2=0x7ff0b40acea0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:57.478 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:57.479 7ff0bab7f700 1 -- 172.21.2.102:0/3331917092 shutdown_connections 2021-02-10T11:24:57.478 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:57.479 7ff0bab7f700 1 -- 172.21.2.102:0/3331917092 wait complete. 2021-02-10T11:24:57.486 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-02-10T11:24:57.917 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:57.902 7fb93c84a700 1 Processor -- start 2021-02-10T11:24:57.917 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:57.902 7fb93c84a700 1 -- start start 2021-02-10T11:24:57.917 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:57.902 7fb93c84a700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fb938078520 0x7fb9380d1900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:57.917 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:57.902 7fb93c84a700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb938078920 con 0x7fb9380e5170 2021-02-10T11:24:57.917 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:57.902 7fb93c84a700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb938078b90 con 0x7fb9380e5570 2021-02-10T11:24:57.917 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:57.902 7fb93c84a700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9381b40d0 con 0x7fb938078520 2021-02-10T11:24:57.917 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:57.903 7fb936e1d700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fb938078520 0x7fb9380d1900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:57.917 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:57.903 7fb936e1d700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fb938078520 0x7fb9380d1900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:57540/0 (socket says 172.21.2.102:57540) 2021-02-10T11:24:57.918 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:57.903 7fb936e1d700 1 -- 172.21.2.102:0/4036222789 learned_addr learned my addr 172.21.2.102:0/4036222789 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:24:57.918 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:57.903 7fb936e1d700 1 -- 172.21.2.102:0/4036222789 >> v1:172.21.2.113:6789/0 conn(0x7fb9380e5570 legacy=0x7fb938079110 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:24:57.918 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:57.903 7fb936e1d700 1 -- 172.21.2.102:0/4036222789 >> v1:172.21.2.102:6789/0 conn(0x7fb9380e5170 legacy=0x7fb9381b38e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:57.918 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:57.903 7fb936e1d700 1 -- 172.21.2.102:0/4036222789 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9380d1f20 con 0x7fb938078520 2021-02-10T11:24:57.918 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:57.903 7fb936e1d700 1 --2- 172.21.2.102:0/4036222789 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb938078520 0x7fb9380d1900 secure :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 rx=0x7fb930006f00 tx=0x7fb930009f10).ready entity=mon.0 client_cookie=9807dbf7f7da35 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:57.918 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:57.903 7fb93561a700 1 -- 172.21.2.102:0/4036222789 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb93000b3b0 con 0x7fb938078520 2021-02-10T11:24:57.918 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:57.903 7fb93561a700 1 -- 172.21.2.102:0/4036222789 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb930000d30 con 0x7fb938078520 2021-02-10T11:24:57.918 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:57.903 7fb93561a700 1 -- 172.21.2.102:0/4036222789 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb93000b9b0 con 0x7fb938078520 2021-02-10T11:24:57.918 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:57.903 7fb93c84a700 1 -- 172.21.2.102:0/4036222789 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb938078520 msgr2=0x7fb9380d1900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:57.918 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:57.903 7fb93c84a700 1 --2- 172.21.2.102:0/4036222789 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb938078520 0x7fb9380d1900 secure :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 rx=0x7fb930006f00 tx=0x7fb930009f10).stop 2021-02-10T11:24:57.918 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:57.904 7fb93c84a700 1 -- 172.21.2.102:0/4036222789 shutdown_connections 2021-02-10T11:24:57.918 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:57.904 7fb93c84a700 1 --2- 172.21.2.102:0/4036222789 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb938078520 0x7fb9380d1900 unknown :-1 s=CLOSED pgs=1416 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:57.918 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:57.904 7fb93c84a700 1 -- 172.21.2.102:0/4036222789 >> 172.21.2.102:0/4036222789 conn(0x7fb9380ce820 msgr2=0x7fb9380e4660 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:57.919 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:57.904 7fb93c84a700 1 -- 172.21.2.102:0/4036222789 shutdown_connections 2021-02-10T11:24:57.919 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:57.904 7fb93c84a700 1 -- 172.21.2.102:0/4036222789 wait complete. 2021-02-10T11:24:57.920 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:57.918 7fb93c84a700 1 Processor -- start 2021-02-10T11:24:57.920 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:57.918 7fb93c84a700 1 -- start start 2021-02-10T11:24:57.920 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:57.919 7fb93c84a700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb9381046a0 0x7fb9381c2100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:57.920 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:57.919 7fb93c84a700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb9381c2720 0x7fb9380b1dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:57.920 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:57.919 7fb93c84a700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb9380b23e0 0x7fb9380b27e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:57.921 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:57.919 7fb93c84a700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb93810fae0 con 0x7fb9380b23e0 2021-02-10T11:24:57.921 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:57.919 7fb93c84a700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9380b2e00 con 0x7fb9381046a0 2021-02-10T11:24:57.921 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:57.919 7fb93c84a700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb938110060 con 0x7fb9381c2720 2021-02-10T11:24:57.921 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:57.919 7fb936e1d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb9380b23e0 0x7fb9380b27e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:57.921 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:57.919 7fb936e1d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb9380b23e0 0x7fb9380b27e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:57544/0 (socket says 172.21.2.102:57544) 2021-02-10T11:24:57.921 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:57.919 7fb936e1d700 1 -- 172.21.2.102:0/3434257534 learned_addr learned my addr 172.21.2.102:0/3434257534 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:24:57.921 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:57.919 7fb936e1d700 1 -- 172.21.2.102:0/3434257534 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb9381c2720 msgr2=0x7fb9380b1dc0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:24:57.921 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:57.919 7fb936e1d700 1 --2- 172.21.2.102:0/3434257534 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb9381c2720 0x7fb9380b1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:57.921 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:57.919 7fb936e1d700 1 -- 172.21.2.102:0/3434257534 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb9381046a0 msgr2=0x7fb9381c2100 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:24:57.921 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:57.919 7fb936e1d700 1 --2- 172.21.2.102:0/3434257534 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb9381046a0 0x7fb9381c2100 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:57.921 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:57.919 7fb936e1d700 1 -- 172.21.2.102:0/3434257534 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb938110280 con 0x7fb9380b23e0 2021-02-10T11:24:57.921 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:57.919 7fb936e1d700 1 --2- 172.21.2.102:0/3434257534 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb9380b23e0 0x7fb9380b27e0 secure :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 rx=0x7fb93000b990 tx=0x7fb93000bd30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:57.921 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:57.919 7fb92effd700 1 -- 172.21.2.102:0/3434257534 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb930000f90 con 0x7fb9380b23e0 2021-02-10T11:24:57.922 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:57.920 7fb92effd700 1 -- 172.21.2.102:0/3434257534 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb930001270 con 0x7fb9380b23e0 2021-02-10T11:24:57.922 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:57.920 7fb92effd700 1 -- 172.21.2.102:0/3434257534 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb930001710 con 0x7fb9380b23e0 2021-02-10T11:24:57.922 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:57.920 7fb93c84a700 1 -- 172.21.2.102:0/3434257534 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb938110280 con 0x7fb9380b23e0 2021-02-10T11:24:57.922 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:57.920 7fb93c84a700 1 -- 172.21.2.102:0/3434257534 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb938110280 con 0x7fb9380b23e0 2021-02-10T11:24:57.922 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:57.920 7fb92effd700 1 -- 172.21.2.102:0/3434257534 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fb930001450 con 0x7fb9380b23e0 2021-02-10T11:24:57.922 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:57.920 7fb92effd700 1 --2- 172.21.2.102:0/3434257534 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb91c018730 0x7fb91c01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:57.922 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:57.920 7fb92effd700 1 -- 172.21.2.102:0/3434257534 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fb9300329a0 con 0x7fb9380b23e0 2021-02-10T11:24:57.922 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:57.920 7fb93661c700 1 --2- 172.21.2.102:0/3434257534 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb91c018730 0x7fb91c01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:57.922 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:57.921 7fb93661c700 1 --2- 172.21.2.102:0/3434257534 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb91c018730 0x7fb91c01ac10 crc :-1 s=READY pgs=652 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:57.951 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:57.952 7fb93c84a700 1 -- 172.21.2.102:0/3434257534 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9381035c0 con 0x7fb9380b23e0 2021-02-10T11:24:57.954 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:57.955 7fb92effd700 1 -- 172.21.2.102:0/3434257534 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fb930000f90 con 0x7fb9380b23e0 2021-02-10T11:24:58.081 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:58.082 7fb93c84a700 1 -- 172.21.2.102:0/3434257534 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7fb9381035c0 con 0x7fb9380b23e0 2021-02-10T11:24:58.081 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:58.083 7fb92effd700 1 -- 172.21.2.102:0/3434257534 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v57) v1 ==== 100+0+10 (secure 0 0 0) 0x7fb930001270 con 0x7fb9380b23e0 2021-02-10T11:24:58.082 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:24:58.084 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:58.085 7fb93c84a700 1 -- 172.21.2.102:0/3434257534 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb91c018730 msgr2=0x7fb91c01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:58.084 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:58.085 7fb93c84a700 1 --2- 172.21.2.102:0/3434257534 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb91c018730 0x7fb91c01ac10 crc :-1 s=READY pgs=652 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:58.084 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:58.085 7fb93c84a700 1 -- 172.21.2.102:0/3434257534 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb9380b23e0 msgr2=0x7fb9380b27e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:58.084 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:58.085 7fb93c84a700 1 --2- 172.21.2.102:0/3434257534 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb9380b23e0 0x7fb9380b27e0 secure :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 rx=0x7fb93000b990 tx=0x7fb93000bd30).stop 2021-02-10T11:24:58.084 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:58.085 7fb93c84a700 1 -- 172.21.2.102:0/3434257534 shutdown_connections 2021-02-10T11:24:58.084 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:58.085 7fb93c84a700 1 --2- 172.21.2.102:0/3434257534 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb9380b23e0 0x7fb9380b27e0 unknown :-1 s=CLOSED pgs=1417 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:58.084 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:58.085 7fb93c84a700 1 --2- 172.21.2.102:0/3434257534 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb9381c2720 0x7fb9380b1dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:58.084 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:58.085 7fb93c84a700 1 --2- 172.21.2.102:0/3434257534 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb91c018730 0x7fb91c01ac10 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:58.084 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:58.085 7fb93c84a700 1 --2- 172.21.2.102:0/3434257534 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb9381046a0 0x7fb9381c2100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:58.084 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:58.085 7fb93c84a700 1 -- 172.21.2.102:0/3434257534 >> 172.21.2.102:0/3434257534 conn(0x7fb9380984c0 msgr2=0x7fb938097990 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:58.085 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:58.085 7fb93c84a700 1 -- 172.21.2.102:0/3434257534 shutdown_connections 2021-02-10T11:24:58.085 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:58.085 7fb93c84a700 1 -- 172.21.2.102:0/3434257534 wait complete. 2021-02-10T11:24:58.092 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-02-10T11:24:58.093 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:24:58.518 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:58.502 7fb704512700 1 Processor -- start 2021-02-10T11:24:58.518 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:58.502 7fb704512700 1 -- start start 2021-02-10T11:24:58.518 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:58.503 7fb704512700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fb6fc0e53e0 0x7fb6fc1b0550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:58.519 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:58.503 7fb704512700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fb6fc0842e0 0x7fb6fc082100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:58.519 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:58.503 7fb704512700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6fc082d50 con 0x7fb6fc1b0b70 2021-02-10T11:24:58.519 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:58.503 7fb704512700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6fc0e5fe0 con 0x7fb6fc0e53e0 2021-02-10T11:24:58.519 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:58.503 7fb704512700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6fc0e57e0 con 0x7fb6fc0842e0 2021-02-10T11:24:58.519 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:58.503 7fb70232e700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fb6fc0e53e0 0x7fb6fc1b0550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:58.519 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:58.503 7fb70232e700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fb6fc0e53e0 0x7fb6fc1b0550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:57550/0 (socket says 172.21.2.102:57550) 2021-02-10T11:24:58.519 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:58.503 7fb70232e700 1 -- 172.21.2.102:0/1334218198 learned_addr learned my addr 172.21.2.102:0/1334218198 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:24:58.519 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:58.504 7fb70232e700 1 -- 172.21.2.102:0/1334218198 >> v2:172.21.2.113:3300/0 conn(0x7fb6fc0842e0 msgr2=0x7fb6fc082100 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:24:58.519 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:58.504 7fb70232e700 1 --2- 172.21.2.102:0/1334218198 >> v2:172.21.2.113:3300/0 conn(0x7fb6fc0842e0 0x7fb6fc082100 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:58.519 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:58.504 7fb70232e700 1 -- 172.21.2.102:0/1334218198 >> v1:172.21.2.113:6789/0 conn(0x7fb6fc1b0b70 legacy=0x7fb6fc081910 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:24:58.519 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:58.504 7fb70232e700 1 -- 172.21.2.102:0/1334218198 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6fc0846e0 con 0x7fb6fc0e53e0 2021-02-10T11:24:58.519 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:58.504 7fb70232e700 1 --2- 172.21.2.102:0/1334218198 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb6fc0e53e0 0x7fb6fc1b0550 secure :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 rx=0x7fb6f0006700 tx=0x7fb6f0009b80).ready entity=mon.0 client_cookie=5f3d2b6954e6f149 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:58.520 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:58.504 7fb70132c700 1 -- 172.21.2.102:0/1334218198 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb6f000c0c0 con 0x7fb6fc0e53e0 2021-02-10T11:24:58.520 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:58.504 7fb70132c700 1 -- 172.21.2.102:0/1334218198 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6f000c340 con 0x7fb6fc0e53e0 2021-02-10T11:24:58.520 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:58.504 7fb70132c700 1 -- 172.21.2.102:0/1334218198 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb6f000c860 con 0x7fb6fc0e53e0 2021-02-10T11:24:58.520 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:58.504 7fb704512700 1 -- 172.21.2.102:0/1334218198 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb6fc0e53e0 msgr2=0x7fb6fc1b0550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:58.520 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:58.504 7fb704512700 1 --2- 172.21.2.102:0/1334218198 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb6fc0e53e0 0x7fb6fc1b0550 secure :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 rx=0x7fb6f0006700 tx=0x7fb6f0009b80).stop 2021-02-10T11:24:58.520 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:58.505 7fb704512700 1 -- 172.21.2.102:0/1334218198 shutdown_connections 2021-02-10T11:24:58.520 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:58.505 7fb704512700 1 --2- 172.21.2.102:0/1334218198 >> v2:172.21.2.113:3300/0 conn(0x7fb6fc0842e0 0x7fb6fc082100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:58.520 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:58.505 7fb704512700 1 --2- 172.21.2.102:0/1334218198 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb6fc0e53e0 0x7fb6fc1b0550 unknown :-1 s=CLOSED pgs=1418 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:58.520 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:58.505 7fb704512700 1 -- 172.21.2.102:0/1334218198 >> 172.21.2.102:0/1334218198 conn(0x7fb6fc0ce500 msgr2=0x7fb6fc0e4610 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:58.520 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:58.505 7fb704512700 1 -- 172.21.2.102:0/1334218198 shutdown_connections 2021-02-10T11:24:58.520 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:58.505 7fb704512700 1 -- 172.21.2.102:0/1334218198 wait complete. 2021-02-10T11:24:58.522 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:58.520 7fb704512700 1 Processor -- start 2021-02-10T11:24:58.522 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:58.520 7fb704512700 1 -- start start 2021-02-10T11:24:58.522 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:58.520 7fb704512700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb6fc19e3b0 0x7fb6fc1b19f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:58.522 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:58.520 7fb704512700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb6fc1b2010 0x7fb6fc1b2410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:58.522 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:58.520 7fb704512700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb6fc199d80 0x7fb6fc19a180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:58.522 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:58.520 7fb704512700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6fc19e7b0 con 0x7fb6fc199d80 2021-02-10T11:24:58.522 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:58.520 7fb704512700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6fc1af520 con 0x7fb6fc19e3b0 2021-02-10T11:24:58.522 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:58.520 7fb704512700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6fc19a7a0 con 0x7fb6fc1b2010 2021-02-10T11:24:58.523 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:58.520 7fb702b2f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb6fc199d80 0x7fb6fc19a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:58.523 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:58.520 7fb702b2f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb6fc199d80 0x7fb6fc19a180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:57556/0 (socket says 172.21.2.102:57556) 2021-02-10T11:24:58.523 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:58.520 7fb702b2f700 1 -- 172.21.2.102:0/3484638654 learned_addr learned my addr 172.21.2.102:0/3484638654 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:24:58.523 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:58.521 7fb702b2f700 1 -- 172.21.2.102:0/3484638654 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb6fc1b2010 msgr2=0x7fb6fc1b2410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:58.523 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:58.521 7fb702b2f700 1 --2- 172.21.2.102:0/3484638654 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb6fc1b2010 0x7fb6fc1b2410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:58.523 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:58.521 7fb702b2f700 1 -- 172.21.2.102:0/3484638654 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb6fc19e3b0 msgr2=0x7fb6fc1b19f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:58.523 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:58.521 7fb702b2f700 1 --2- 172.21.2.102:0/3484638654 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb6fc19e3b0 0x7fb6fc1b19f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:58.523 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:58.521 7fb702b2f700 1 -- 172.21.2.102:0/3484638654 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6fc19a9c0 con 0x7fb6fc199d80 2021-02-10T11:24:58.523 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:58.521 7fb702b2f700 1 --2- 172.21.2.102:0/3484638654 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb6fc199d80 0x7fb6fc19a180 secure :-1 s=READY pgs=1419 cs=0 l=1 rev1=1 rx=0x7fb6f4009960 tx=0x7fb6f400bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:58.523 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:58.521 7fb6eeffd700 1 -- 172.21.2.102:0/3484638654 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb6f4003030 con 0x7fb6fc199d80 2021-02-10T11:24:58.523 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:58.521 7fb6eeffd700 1 -- 172.21.2.102:0/3484638654 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6f4001310 con 0x7fb6fc199d80 2021-02-10T11:24:58.523 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:58.521 7fb6eeffd700 1 -- 172.21.2.102:0/3484638654 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb6f4001830 con 0x7fb6fc199d80 2021-02-10T11:24:58.523 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:58.521 7fb704512700 1 -- 172.21.2.102:0/3484638654 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6fc19a9c0 con 0x7fb6fc199d80 2021-02-10T11:24:58.523 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:58.521 7fb704512700 1 -- 172.21.2.102:0/3484638654 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6fc19a9c0 con 0x7fb6fc199d80 2021-02-10T11:24:58.524 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:58.522 7fb6eeffd700 1 -- 172.21.2.102:0/3484638654 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fb6f4000d10 con 0x7fb6fc199d80 2021-02-10T11:24:58.524 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:58.522 7fb6eeffd700 1 --2- 172.21.2.102:0/3484638654 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb6e40186c0 0x7fb6e401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:58.524 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:58.522 7fb6eeffd700 1 -- 172.21.2.102:0/3484638654 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fb6f4032b30 con 0x7fb6fc199d80 2021-02-10T11:24:58.524 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:58.522 7fb70232e700 1 --2- 172.21.2.102:0/3484638654 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb6e40186c0 0x7fb6e401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:58.524 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:58.522 7fb70232e700 1 --2- 172.21.2.102:0/3484638654 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb6e40186c0 0x7fb6e401abc0 crc :-1 s=READY pgs=653 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:58.552 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:58.553 7fb704512700 1 -- 172.21.2.102:0/3484638654 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6fc0ed660 con 0x7fb6fc199d80 2021-02-10T11:24:58.555 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:58.556 7fb6eeffd700 1 -- 172.21.2.102:0/3484638654 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fb6f400be30 con 0x7fb6fc199d80 2021-02-10T11:24:58.683 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:58.684 7fb704512700 1 -- 172.21.2.102:0/3484638654 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb6fc082d50 con 0x7fb6fc199d80 2021-02-10T11:24:58.683 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:58.684 7fb6eeffd700 1 -- 172.21.2.102:0/3484638654 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) v1 ==== 94+0+2101 (secure 0 0 0) 0x7fb6f4000d10 con 0x7fb6fc199d80 2021-02-10T11:24:58.684 INFO:teuthology.orchestra.run.gibba002.stderr:dumped fsmap epoch 40 2021-02-10T11:24:58.684 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:24:58.684 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":40,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7920,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.102:6835/1829204031","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6834","nonce":1829204031},{"type":"v1","addr":"172.21.2.102:6835","nonce":1829204031}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":35}],"filesystems":[{"mdsmap":{"epoch":39,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:24:10.198618","modified":"2021-02-10 11:24:12.204882","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5714},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5714":{"gid":5714,"name":"b","rank":0,"incarnation":36,"state":"up:active","state_seq":3,"addr":"172.21.2.113:6833/4013137507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6832","nonce":4013137507},{"type":"v1","addr":"172.21.2.113:6833","nonce":4013137507}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":5}]} 2021-02-10T11:24:58.686 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:58.686 7fb704512700 1 -- 172.21.2.102:0/3484638654 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb6e40186c0 msgr2=0x7fb6e401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:58.686 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:58.686 7fb704512700 1 --2- 172.21.2.102:0/3484638654 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb6e40186c0 0x7fb6e401abc0 crc :-1 s=READY pgs=653 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:58.686 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:58.686 7fb704512700 1 -- 172.21.2.102:0/3484638654 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb6fc199d80 msgr2=0x7fb6fc19a180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:58.686 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:58.686 7fb704512700 1 --2- 172.21.2.102:0/3484638654 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb6fc199d80 0x7fb6fc19a180 secure :-1 s=READY pgs=1419 cs=0 l=1 rev1=1 rx=0x7fb6f4009960 tx=0x7fb6f400bcf0).stop 2021-02-10T11:24:58.686 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:58.686 7fb704512700 1 -- 172.21.2.102:0/3484638654 shutdown_connections 2021-02-10T11:24:58.686 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:58.686 7fb704512700 1 --2- 172.21.2.102:0/3484638654 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb6fc199d80 0x7fb6fc19a180 unknown :-1 s=CLOSED pgs=1419 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:58.686 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:58.686 7fb704512700 1 --2- 172.21.2.102:0/3484638654 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb6fc1b2010 0x7fb6fc1b2410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:58.686 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:58.686 7fb704512700 1 --2- 172.21.2.102:0/3484638654 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb6e40186c0 0x7fb6e401abc0 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:58.686 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:58.686 7fb704512700 1 --2- 172.21.2.102:0/3484638654 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb6fc19e3b0 0x7fb6fc1b19f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:58.686 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:58.686 7fb704512700 1 -- 172.21.2.102:0/3484638654 >> 172.21.2.102:0/3484638654 conn(0x7fb6fc1b2f00 msgr2=0x7fb6fc1a1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:58.686 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:58.687 7fb704512700 1 -- 172.21.2.102:0/3484638654 shutdown_connections 2021-02-10T11:24:58.687 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:24:58.687 7fb704512700 1 -- 172.21.2.102:0/3484638654 wait complete. 2021-02-10T11:24:58.896 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 39, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:24:10.198618', 'modified': '2021-02-10 11:24:12.204882', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5714}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5714': {'gid': 5714, 'name': 'b', 'rank': 0, 'incarnation': 36, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.113:6833/4013137507', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.113:6832', 'nonce': 4013137507}, {'type': 'v1', 'addr': '172.21.2.113:6833', 'nonce': 4013137507}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [11], 'metadata_pool': 10, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:24:58.896 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:24:58.896 DEBUG:teuthology.orchestra.run.gibba013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:24:59.295 INFO:teuthology.orchestra.run.gibba013.stdout:{ 2021-02-10T11:24:59.295 INFO:teuthology.orchestra.run.gibba013.stdout: "cluster_fsid": "68db30ea-dbce-4aef-a6df-f01c6fea0513", 2021-02-10T11:24:59.295 INFO:teuthology.orchestra.run.gibba013.stdout: "whoami": 0, 2021-02-10T11:24:59.295 INFO:teuthology.orchestra.run.gibba013.stdout: "id": 5714, 2021-02-10T11:24:59.296 INFO:teuthology.orchestra.run.gibba013.stdout: "want_state": "up:active", 2021-02-10T11:24:59.296 INFO:teuthology.orchestra.run.gibba013.stdout: "state": "up:active", 2021-02-10T11:24:59.296 INFO:teuthology.orchestra.run.gibba013.stdout: "rank_uptime": 49.097053273999997, 2021-02-10T11:24:59.296 INFO:teuthology.orchestra.run.gibba013.stdout: "mdsmap_epoch": 39, 2021-02-10T11:24:59.296 INFO:teuthology.orchestra.run.gibba013.stdout: "osdmap_epoch": 57, 2021-02-10T11:24:59.296 INFO:teuthology.orchestra.run.gibba013.stdout: "osdmap_epoch_barrier": 57, 2021-02-10T11:24:59.296 INFO:teuthology.orchestra.run.gibba013.stdout: "uptime": 54.773421745999997 2021-02-10T11:24:59.296 INFO:teuthology.orchestra.run.gibba013.stdout:} 2021-02-10T11:24:59.301 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "68db30ea-dbce-4aef-a6df-f01c6fea0513", "whoami": 0, "id": 5714, "want_state": "up:active", "state": "up:active", "rank_uptime": 49.097053273999997, "mdsmap_epoch": 39, "osdmap_epoch": 57, "osdmap_epoch_barrier": 57, "uptime": 54.773421745999997 } 2021-02-10T11:24:59.301 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2021-02-10T11:24:59.302 INFO:tasks.cephfs.fuse_mount:Client client.alice config is {} 2021-02-10T11:24:59.302 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.alice at ubuntu@gibba002.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.1... 2021-02-10T11:24:59.302 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:24:59.302 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec mkdir -p /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:24:59.638 DEBUG:teuthology.orchestra.run.gibba002:> sudo modprobe fuse 2021-02-10T11:24:59.981 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:24:59.982 DEBUG:teuthology.orchestra.run.gibba002:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:25:00.324 INFO:teuthology.orchestra.run.gibba002.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:25:00.325 INFO:teuthology.orchestra.run.gibba002.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:25:00.325 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:25:00.326 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:00.326 DEBUG:teuthology.orchestra.run.gibba002:> ls /sys/fs/fuse/connections 2021-02-10T11:25:00.661 INFO:teuthology.orchestra.run.gibba002.stdout:43 2021-02-10T11:25:00.661 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [43] 2021-02-10T11:25:00.661 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --client_mountpoint=/volumes/group_0000000000878856/subvolume_0000000000190758 --name client.alice /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:25:00.664 DEBUG:teuthology.orchestra.run.gibba002:> sudo modprobe fuse 2021-02-10T11:25:01.007 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:25:01.008 DEBUG:teuthology.orchestra.run.gibba002:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:25:01.074 INFO:tasks.cephfs.fuse_mount.ceph-fuse.alice.gibba002.stderr:2021-02-10 11:25:01.075 7f2061557f80 -1 init, newargv = 0x55c1e276d440 newargc=9 2021-02-10T11:25:01.075 INFO:tasks.cephfs.fuse_mount.ceph-fuse.alice.gibba002.stderr:ceph-fuse[156017]: starting ceph client 2021-02-10T11:25:01.081 INFO:tasks.cephfs.fuse_mount.ceph-fuse.alice.gibba002.stderr:ceph-fuse[156017]: starting fuse 2021-02-10T11:25:01.350 INFO:teuthology.orchestra.run.gibba002.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:25:01.350 INFO:teuthology.orchestra.run.gibba002.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:25:01.350 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:25:01.351 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:01.351 DEBUG:teuthology.orchestra.run.gibba002:> ls /sys/fs/fuse/connections 2021-02-10T11:25:01.685 INFO:teuthology.orchestra.run.gibba002.stdout:43 2021-02-10T11:25:01.685 INFO:teuthology.orchestra.run.gibba002.stdout:44 2021-02-10T11:25:01.686 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43, 44] 2021-02-10T11:25:01.686 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:01.686 DEBUG:teuthology.orchestra.run.gibba002:> sudo python3 -c ' 2021-02-10T11:25:01.686 DEBUG:teuthology.orchestra.run.gibba002:> import glob 2021-02-10T11:25:01.686 DEBUG:teuthology.orchestra.run.gibba002:> import re 2021-02-10T11:25:01.686 DEBUG:teuthology.orchestra.run.gibba002:> import os 2021-02-10T11:25:01.686 DEBUG:teuthology.orchestra.run.gibba002:> import subprocess 2021-02-10T11:25:01.686 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:25:01.686 DEBUG:teuthology.orchestra.run.gibba002:> def find_socket(client_name): 2021-02-10T11:25:01.686 DEBUG:teuthology.orchestra.run.gibba002:> asok_path = "/var/run/ceph/ceph-client.alice.*.asok" 2021-02-10T11:25:01.687 DEBUG:teuthology.orchestra.run.gibba002:> files = glob.glob(asok_path) 2021-02-10T11:25:01.687 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:25:01.687 DEBUG:teuthology.orchestra.run.gibba002:> # Given a non-glob path, it better be there 2021-02-10T11:25:01.687 DEBUG:teuthology.orchestra.run.gibba002:> if "*" not in asok_path: 2021-02-10T11:25:01.687 DEBUG:teuthology.orchestra.run.gibba002:> assert(len(files) == 1) 2021-02-10T11:25:01.687 DEBUG:teuthology.orchestra.run.gibba002:> return files[0] 2021-02-10T11:25:01.687 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:25:01.687 DEBUG:teuthology.orchestra.run.gibba002:> for f in files: 2021-02-10T11:25:01.687 DEBUG:teuthology.orchestra.run.gibba002:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2021-02-10T11:25:01.687 DEBUG:teuthology.orchestra.run.gibba002:> if os.path.exists("/proc/{0}".format(pid)): 2021-02-10T11:25:01.687 DEBUG:teuthology.orchestra.run.gibba002:> return f 2021-02-10T11:25:01.687 DEBUG:teuthology.orchestra.run.gibba002:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2021-02-10T11:25:01.687 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:25:01.687 DEBUG:teuthology.orchestra.run.gibba002:> print(find_socket("client.alice")) 2021-02-10T11:25:01.687 DEBUG:teuthology.orchestra.run.gibba002:> ' 2021-02-10T11:25:02.049 INFO:teuthology.orchestra.run.gibba002.stdout:/var/run/ceph/ceph-client.alice.156017.asok 2021-02-10T11:25:02.051 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.alice.156017.asok 2021-02-10T11:25:02.051 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:02.052 DEBUG:teuthology.orchestra.run.gibba002:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.alice.156017.asok status 2021-02-10T11:25:02.442 INFO:teuthology.orchestra.run.gibba002.stdout:{ 2021-02-10T11:25:02.443 INFO:teuthology.orchestra.run.gibba002.stdout: "metadata": { 2021-02-10T11:25:02.443 INFO:teuthology.orchestra.run.gibba002.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:25:02.443 INFO:teuthology.orchestra.run.gibba002.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:25:02.443 INFO:teuthology.orchestra.run.gibba002.stdout: "entity_id": "alice", 2021-02-10T11:25:02.443 INFO:teuthology.orchestra.run.gibba002.stdout: "hostname": "gibba002", 2021-02-10T11:25:02.443 INFO:teuthology.orchestra.run.gibba002.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2021-02-10T11:25:02.443 INFO:teuthology.orchestra.run.gibba002.stdout: "pid": "156017", 2021-02-10T11:25:02.443 INFO:teuthology.orchestra.run.gibba002.stdout: "root": "/volumes/group_0000000000878856/subvolume_0000000000190758" 2021-02-10T11:25:02.443 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:25:02.443 INFO:teuthology.orchestra.run.gibba002.stdout: "dentry_count": 0, 2021-02-10T11:25:02.443 INFO:teuthology.orchestra.run.gibba002.stdout: "dentry_pinned_count": 0, 2021-02-10T11:25:02.444 INFO:teuthology.orchestra.run.gibba002.stdout: "id": 8280, 2021-02-10T11:25:02.444 INFO:teuthology.orchestra.run.gibba002.stdout: "inst": { 2021-02-10T11:25:02.444 INFO:teuthology.orchestra.run.gibba002.stdout: "name": { 2021-02-10T11:25:02.444 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "client", 2021-02-10T11:25:02.444 INFO:teuthology.orchestra.run.gibba002.stdout: "num": 8280 2021-02-10T11:25:02.444 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:25:02.444 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": { 2021-02-10T11:25:02.444 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:25:02.444 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:0", 2021-02-10T11:25:02.444 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 4033890702 2021-02-10T11:25:02.444 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:25:02.444 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:25:02.444 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": { 2021-02-10T11:25:02.445 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:25:02.445 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:0", 2021-02-10T11:25:02.445 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 4033890702 2021-02-10T11:25:02.445 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:25:02.445 INFO:teuthology.orchestra.run.gibba002.stdout: "inst_str": "client.8280 172.21.2.102:0/4033890702", 2021-02-10T11:25:02.445 INFO:teuthology.orchestra.run.gibba002.stdout: "addr_str": "172.21.2.102:0/4033890702", 2021-02-10T11:25:02.445 INFO:teuthology.orchestra.run.gibba002.stdout: "inode_count": 1, 2021-02-10T11:25:02.445 INFO:teuthology.orchestra.run.gibba002.stdout: "mds_epoch": 39, 2021-02-10T11:25:02.445 INFO:teuthology.orchestra.run.gibba002.stdout: "osd_epoch": 57, 2021-02-10T11:25:02.445 INFO:teuthology.orchestra.run.gibba002.stdout: "osd_epoch_barrier": 0, 2021-02-10T11:25:02.445 INFO:teuthology.orchestra.run.gibba002.stdout: "blacklisted": false 2021-02-10T11:25:02.445 INFO:teuthology.orchestra.run.gibba002.stdout:} 2021-02-10T11:25:02.447 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.1 && sudo ls /home/ubuntu/cephtest/mnt.1 2021-02-10T11:25:02.793 INFO:teuthology.orchestra.run.gibba002.stdout:data.bin 2021-02-10T11:25:02.794 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:02.794 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:25:02.794 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:25:03.129 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:25:03.129 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2021-02-10T11:25:03.130 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.1 && sudo dd if=/dev/urandom of=rogue.bin bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:25:03.475 INFO:teuthology.orchestra.run.gibba002.stderr:dd: failed to open ‘rogue.bin’: Permission denied 2021-02-10T11:25:03.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:25:03.477 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:03.477 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:25:03.477 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:25:03.812 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:25:03.813 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2021-02-10T11:25:03.813 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@gibba002.front.sepia.ceph.com... 2021-02-10T11:25:03.813 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-02-10T11:25:03.813 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:25:04.164 INFO:tasks.cephfs.fuse_mount.ceph-fuse.alice.gibba002.stderr:ceph-fuse[156017]: fuse finished with error 0 and tester_r 0 2021-02-10T11:25:04.165 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:04.165 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:25:04.165 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:25:04.432 DEBUG:teuthology.orchestra.run.gibba002:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:25:04.436 DEBUG:teuthology.orchestra.run.gibba013:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:25:04.499 INFO:teuthology.orchestra.run.gibba002.stdout:ext2/ext3 2021-02-10T11:25:04.499 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2021-02-10T11:25:04.499 INFO:teuthology.orchestra.run:waiting for 900 2021-02-10T11:25:04.500 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:25:04.500 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:25:04.768 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume deauthorize cephfs subvolume_0000000000190758 alice --group_name group_0000000000878856 2021-02-10T11:25:05.198 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:05.183 7f6372a5f700 1 Processor -- start 2021-02-10T11:25:05.199 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:05.183 7f6372a5f700 1 -- start start 2021-02-10T11:25:05.199 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:05.183 7f6372a5f700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f636c0e51a0 0x7f636c1b3910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:05.199 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:05.183 7f6372a5f700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f636c1b3f30 0x7f636c077510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:05.199 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:05.183 7f6372a5f700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f636c0e55a0 0x7f636c077b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:05.199 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:05.183 7f6372a5f700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f636c078b90 con 0x7f636c1b3f30 2021-02-10T11:25:05.199 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:05.183 7f6372a5f700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f636c078150 con 0x7f636c0e55a0 2021-02-10T11:25:05.199 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:05.183 7f6372a5f700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f636c078340 con 0x7f636c0e51a0 2021-02-10T11:25:05.199 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:05.183 7f636bfff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f636c1b3f30 0x7f636c077510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:05.199 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:05.183 7f636bfff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f636c1b3f30 0x7f636c077510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:57584/0 (socket says 172.21.2.102:57584) 2021-02-10T11:25:05.199 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:05.183 7f636bfff700 1 -- 172.21.2.102:0/46747311 learned_addr learned my addr 172.21.2.102:0/46747311 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:25:05.199 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:05.184 7f636bfff700 1 -- 172.21.2.102:0/46747311 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f636c0e51a0 msgr2=0x7f636c1b3910 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:25:05.200 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:05.184 7f636bfff700 1 --2- 172.21.2.102:0/46747311 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f636c0e51a0 0x7f636c1b3910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:05.200 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:05.184 7f636bfff700 1 -- 172.21.2.102:0/46747311 >> v2:172.21.2.113:3300/0 conn(0x7f636c0e55a0 msgr2=0x7f636c077b30 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:25:05.200 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:05.184 7f636bfff700 1 --2- 172.21.2.102:0/46747311 >> v2:172.21.2.113:3300/0 conn(0x7f636c0e55a0 0x7f636c077b30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:05.200 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:05.184 7f636bfff700 1 -- 172.21.2.102:0/46747311 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f636c078560 con 0x7f636c1b3f30 2021-02-10T11:25:05.200 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:05.184 7f636bfff700 1 --2- 172.21.2.102:0/46747311 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f636c1b3f30 0x7f636c077510 secure :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 rx=0x7f635c006700 tx=0x7f635c009b80).ready entity=mon.0 client_cookie=920aeacc4d50e5f6 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:05.200 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:05.184 7f636b7fe700 1 -- 172.21.2.102:0/46747311 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f635c00c0c0 con 0x7f636c1b3f30 2021-02-10T11:25:05.200 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:05.184 7f636b7fe700 1 -- 172.21.2.102:0/46747311 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f635c00c440 con 0x7f636c1b3f30 2021-02-10T11:25:05.200 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:05.184 7f636b7fe700 1 -- 172.21.2.102:0/46747311 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f635c00c8a0 con 0x7f636c1b3f30 2021-02-10T11:25:05.200 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:05.184 7f6372a5f700 1 -- 172.21.2.102:0/46747311 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f636c1b3f30 msgr2=0x7f636c077510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:05.200 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:05.184 7f6372a5f700 1 --2- 172.21.2.102:0/46747311 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f636c1b3f30 0x7f636c077510 secure :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 rx=0x7f635c006700 tx=0x7f635c009b80).stop 2021-02-10T11:25:05.200 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:05.185 7f6372a5f700 1 -- 172.21.2.102:0/46747311 shutdown_connections 2021-02-10T11:25:05.200 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:05.185 7f6372a5f700 1 --2- 172.21.2.102:0/46747311 >> v2:172.21.2.113:3300/0 conn(0x7f636c0e55a0 0x7f636c077b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:05.200 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:05.185 7f6372a5f700 1 --2- 172.21.2.102:0/46747311 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f636c1b3f30 0x7f636c077510 unknown :-1 s=CLOSED pgs=1423 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:05.201 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:05.185 7f6372a5f700 1 --2- 172.21.2.102:0/46747311 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f636c0e51a0 0x7f636c1b3910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:05.201 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:05.185 7f6372a5f700 1 -- 172.21.2.102:0/46747311 >> 172.21.2.102:0/46747311 conn(0x7f636c0ce850 msgr2=0x7f636c0e4690 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:25:05.201 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:05.185 7f6372a5f700 1 -- 172.21.2.102:0/46747311 shutdown_connections 2021-02-10T11:25:05.201 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:05.185 7f6372a5f700 1 -- 172.21.2.102:0/46747311 wait complete. 2021-02-10T11:25:05.201 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:05.200 7f6372a5f700 1 Processor -- start 2021-02-10T11:25:05.201 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:05.200 7f6372a5f700 1 -- start start 2021-02-10T11:25:05.202 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:05.200 7f6372a5f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f636c098c00 0x7f636c098430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:05.202 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:05.200 7f6372a5f700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f636c10c0b0 0x7f636c10c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:05.202 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:05.200 7f6372a5f700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f636c10cad0 0x7f636c0afef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:05.202 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:05.200 7f6372a5f700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f636c1abbe0 con 0x7f636c098c00 2021-02-10T11:25:05.203 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:05.200 7f6372a5f700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f636c1abe00 con 0x7f636c10cad0 2021-02-10T11:25:05.203 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:05.200 7f6372a5f700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f636c0b0510 con 0x7f636c10c0b0 2021-02-10T11:25:05.203 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:05.200 7f637087b700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f636c098c00 0x7f636c098430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:05.203 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:05.200 7f637087b700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f636c098c00 0x7f636c098430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:57594/0 (socket says 172.21.2.102:57594) 2021-02-10T11:25:05.203 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:05.200 7f637087b700 1 -- 172.21.2.102:0/931692660 learned_addr learned my addr 172.21.2.102:0/931692660 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:25:05.203 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:05.201 7f637087b700 1 -- 172.21.2.102:0/931692660 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f636c10c0b0 msgr2=0x7f636c10c4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:05.203 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:05.201 7f637087b700 1 --2- 172.21.2.102:0/931692660 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f636c10c0b0 0x7f636c10c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:05.203 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:05.201 7f637087b700 1 -- 172.21.2.102:0/931692660 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f636c10cad0 msgr2=0x7f636c0afef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:05.203 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:05.201 7f637087b700 1 --2- 172.21.2.102:0/931692660 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f636c10cad0 0x7f636c0afef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:05.203 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:05.201 7f637087b700 1 -- 172.21.2.102:0/931692660 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f636c0b06f0 con 0x7f636c098c00 2021-02-10T11:25:05.204 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:05.201 7f637087b700 1 --2- 172.21.2.102:0/931692660 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f636c098c00 0x7f636c098430 secure :-1 s=READY pgs=1424 cs=0 l=1 rev1=1 rx=0x7f635400b5c0 tx=0x7f635400b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:05.204 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:05.201 7f63697fa700 1 -- 172.21.2.102:0/931692660 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6354000c40 con 0x7f636c098c00 2021-02-10T11:25:05.204 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:05.201 7f63697fa700 1 -- 172.21.2.102:0/931692660 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f635400be20 con 0x7f636c098c00 2021-02-10T11:25:05.204 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:05.201 7f6372a5f700 1 -- 172.21.2.102:0/931692660 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f636c0b06f0 con 0x7f636c098c00 2021-02-10T11:25:05.204 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:05.201 7f6372a5f700 1 -- 172.21.2.102:0/931692660 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f636c0b0c80 con 0x7f636c098c00 2021-02-10T11:25:05.204 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:05.201 7f63697fa700 1 -- 172.21.2.102:0/931692660 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f63540062e0 con 0x7f636c098c00 2021-02-10T11:25:05.204 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:05.202 7f63697fa700 1 -- 172.21.2.102:0/931692660 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f6354006030 con 0x7f636c098c00 2021-02-10T11:25:05.204 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:05.202 7f63697fa700 1 --2- 172.21.2.102:0/931692660 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f63500186c0 0x7f635001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:05.204 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:05.202 7f63697fa700 1 -- 172.21.2.102:0/931692660 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f6354032820 con 0x7f636c098c00 2021-02-10T11:25:05.204 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:05.202 7f636bfff700 1 --2- 172.21.2.102:0/931692660 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f63500186c0 0x7f635001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:05.204 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:05.202 7f636bfff700 1 --2- 172.21.2.102:0/931692660 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f63500186c0 0x7f635001abc0 crc :-1 s=READY pgs=654 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:05.232 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:05.233 7f6372a5f700 1 -- 172.21.2.102:0/931692660 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f636c103bd0 con 0x7f636c098c00 2021-02-10T11:25:05.235 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:05.236 7f63697fa700 1 -- 172.21.2.102:0/931692660 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f6354000940 con 0x7f636c098c00 2021-02-10T11:25:05.359 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:05.360 7f6372a5f700 1 -- 172.21.2.102:0/931692660 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume deauthorize", "vol_name": "cephfs", "sub_name": "subvolume_0000000000190758", "target": ["mgr", ""], "auth_id": "alice", "group_name": "group_0000000000878856"}) v1 -- 0x7f636c0b0c80 con 0x7f63500186c0 2021-02-10T11:25:05.396 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:05.397 7f63697fa700 1 -- 172.21.2.102:0/931692660 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f635c000990 con 0x7f63500186c0 2021-02-10T11:25:05.425 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:05.426 7f6372a5f700 1 -- 172.21.2.102:0/931692660 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f63500186c0 msgr2=0x7f635001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:05.425 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:05.426 7f6372a5f700 1 --2- 172.21.2.102:0/931692660 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f63500186c0 0x7f635001abc0 crc :-1 s=READY pgs=654 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:05.425 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:05.426 7f6372a5f700 1 -- 172.21.2.102:0/931692660 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f636c098c00 msgr2=0x7f636c098430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:05.425 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:05.426 7f6372a5f700 1 --2- 172.21.2.102:0/931692660 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f636c098c00 0x7f636c098430 secure :-1 s=READY pgs=1424 cs=0 l=1 rev1=1 rx=0x7f635400b5c0 tx=0x7f635400b960).stop 2021-02-10T11:25:05.425 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:05.426 7f6372a5f700 1 -- 172.21.2.102:0/931692660 shutdown_connections 2021-02-10T11:25:05.426 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:05.426 7f6372a5f700 1 --2- 172.21.2.102:0/931692660 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f63500186c0 0x7f635001abc0 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:05.426 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:05.426 7f6372a5f700 1 --2- 172.21.2.102:0/931692660 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f636c10cad0 0x7f636c0afef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:05.426 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:05.426 7f6372a5f700 1 --2- 172.21.2.102:0/931692660 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f636c10c0b0 0x7f636c10c4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:05.426 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:05.426 7f6372a5f700 1 --2- 172.21.2.102:0/931692660 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f636c098c00 0x7f636c098430 unknown :-1 s=CLOSED pgs=1424 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:05.426 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:05.426 7f6372a5f700 1 -- 172.21.2.102:0/931692660 >> 172.21.2.102:0/931692660 conn(0x7f636c1aa7d0 msgr2=0x7f636c19b850 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:25:05.426 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:05.426 7f6372a5f700 1 -- 172.21.2.102:0/931692660 shutdown_connections 2021-02-10T11:25:05.426 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:05.426 7f6372a5f700 1 -- 172.21.2.102:0/931692660 wait complete. 2021-02-10T11:25:05.433 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth ls --format=json-pretty 2021-02-10T11:25:05.859 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:05.843 7f1e685f3700 1 Processor -- start 2021-02-10T11:25:05.860 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:05.843 7f1e685f3700 1 -- start start 2021-02-10T11:25:05.860 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:05.844 7f1e685f3700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1e60082920 0x7f1e600832d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:05.860 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:05.844 7f1e685f3700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e60082d50 con 0x7f1e600838f0 2021-02-10T11:25:05.860 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:05.844 7f1e685f3700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e600e57e0 con 0x7f1e600e53e0 2021-02-10T11:25:05.860 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:05.844 7f1e685f3700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e601b0d40 con 0x7f1e60082920 2021-02-10T11:25:05.860 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:05.844 7f1e66c10700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f1e600838f0 0x7f1e60083cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:49826/0 (socket says 172.21.2.102:49826) 2021-02-10T11:25:05.860 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:05.844 7f1e66c10700 1 -- 172.21.2.102:0/4152033981 learned_addr learned my addr 172.21.2.102:0/4152033981 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:25:05.860 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:05.844 7f1e65c0e700 1 --2- 172.21.2.102:0/4152033981 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1e60082920 0x7f1e600832d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:05.860 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:05.844 7f1e6540d700 1 -- 172.21.2.102:0/4152033981 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2483618989 0 0) 0x7f1e58001500 con 0x7f1e600838f0 2021-02-10T11:25:05.860 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:05.844 7f1e6540d700 1 -- 172.21.2.102:0/4152033981 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e4c002fe0 con 0x7f1e600838f0 2021-02-10T11:25:05.860 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:05.844 7f1e6540d700 1 -- 172.21.2.102:0/4152033981 <== mon.1 v1:172.21.2.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3472628125 0 0) 0x7f1e5c001080 con 0x7f1e600e53e0 2021-02-10T11:25:05.860 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:05.845 7f1e6540d700 1 -- 172.21.2.102:0/4152033981 --> v1:172.21.2.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e4c003210 con 0x7f1e600e53e0 2021-02-10T11:25:05.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:05.845 7f1e65c0e700 1 -- 172.21.2.102:0/4152033981 >> v1:172.21.2.113:6789/0 conn(0x7f1e600e53e0 legacy=0x7f1e601b0550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:05.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:05.845 7f1e65c0e700 1 -- 172.21.2.102:0/4152033981 >> v1:172.21.2.102:6789/0 conn(0x7f1e600838f0 legacy=0x7f1e60083cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:05.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:05.845 7f1e65c0e700 1 -- 172.21.2.102:0/4152033981 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e600844e0 con 0x7f1e60082920 2021-02-10T11:25:05.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:05.845 7f1e65c0e700 1 --2- 172.21.2.102:0/4152033981 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1e60082920 0x7f1e600832d0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7f1e50006650 tx=0x7f1e50009a80).ready entity=mon.2 client_cookie=71352344ea6a3d50 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:05.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:05.845 7f1e6540d700 1 -- 172.21.2.102:0/4152033981 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1e5000c0c0 con 0x7f1e60082920 2021-02-10T11:25:05.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:05.845 7f1e6540d700 1 -- 172.21.2.102:0/4152033981 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e5000c340 con 0x7f1e60082920 2021-02-10T11:25:05.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:05.845 7f1e6540d700 1 -- 172.21.2.102:0/4152033981 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1e5000c820 con 0x7f1e60082920 2021-02-10T11:25:05.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:05.845 7f1e685f3700 1 -- 172.21.2.102:0/4152033981 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1e60082920 msgr2=0x7f1e600832d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:05.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:05.845 7f1e685f3700 1 --2- 172.21.2.102:0/4152033981 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1e60082920 0x7f1e600832d0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7f1e50006650 tx=0x7f1e50009a80).stop 2021-02-10T11:25:05.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:05.846 7f1e685f3700 1 -- 172.21.2.102:0/4152033981 shutdown_connections 2021-02-10T11:25:05.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:05.846 7f1e685f3700 1 --2- 172.21.2.102:0/4152033981 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1e60082920 0x7f1e600832d0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:05.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:05.846 7f1e685f3700 1 -- 172.21.2.102:0/4152033981 >> 172.21.2.102:0/4152033981 conn(0x7f1e600ce500 msgr2=0x7f1e600e4610 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:25:05.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:05.846 7f1e685f3700 1 -- 172.21.2.102:0/4152033981 shutdown_connections 2021-02-10T11:25:05.862 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:05.846 7f1e685f3700 1 -- 172.21.2.102:0/4152033981 wait complete. 2021-02-10T11:25:05.862 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:05.860 7f1e685f3700 1 Processor -- start 2021-02-10T11:25:05.862 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:05.860 7f1e685f3700 1 -- start start 2021-02-10T11:25:05.863 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:05.861 7f1e685f3700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1e601b8850 0x7f1e601b7ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:05.863 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:05.861 7f1e685f3700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1e601960f0 0x7f1e6018f4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:05.864 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:05.861 7f1e685f3700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1e6018fad0 0x7f1e60196fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:05.864 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:05.861 7f1e685f3700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e601b8d20 con 0x7f1e6018fad0 2021-02-10T11:25:05.864 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:05.861 7f1e685f3700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e601b8fb0 con 0x7f1e601960f0 2021-02-10T11:25:05.864 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:05.861 7f1e685f3700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e601964f0 con 0x7f1e601b8850 2021-02-10T11:25:05.864 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:05.861 7f1e66c10700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1e6018fad0 0x7f1e60196fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:05.864 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:05.861 7f1e66c10700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1e6018fad0 0x7f1e60196fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:57604/0 (socket says 172.21.2.102:57604) 2021-02-10T11:25:05.864 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:05.861 7f1e66c10700 1 -- 172.21.2.102:0/3366722123 learned_addr learned my addr 172.21.2.102:0/3366722123 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:25:05.864 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:05.861 7f1e66c10700 1 -- 172.21.2.102:0/3366722123 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1e601b8850 msgr2=0x7f1e601b7ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:05.864 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:05.861 7f1e65c0e700 1 --2- 172.21.2.102:0/3366722123 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1e601960f0 0x7f1e6018f4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:05.864 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:05.861 7f1e66c10700 1 --2- 172.21.2.102:0/3366722123 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1e601b8850 0x7f1e601b7ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:05.864 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:05.861 7f1e66c10700 1 -- 172.21.2.102:0/3366722123 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1e601960f0 msgr2=0x7f1e6018f4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:05.864 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:05.861 7f1e66c10700 1 --2- 172.21.2.102:0/3366722123 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1e601960f0 0x7f1e6018f4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:05.865 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:05.861 7f1e66c10700 1 -- 172.21.2.102:0/3366722123 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e60196710 con 0x7f1e6018fad0 2021-02-10T11:25:05.865 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:05.861 7f1e65c0e700 1 --2- 172.21.2.102:0/3366722123 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1e601960f0 0x7f1e6018f4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:25:05.865 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:05.862 7f1e66c10700 1 --2- 172.21.2.102:0/3366722123 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1e6018fad0 0x7f1e60196fa0 secure :-1 s=READY pgs=1426 cs=0 l=1 rev1=1 rx=0x7f1e58009980 tx=0x7f1e5800bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:05.865 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:05.862 7f1e56ffd700 1 -- 172.21.2.102:0/3366722123 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1e58004720 con 0x7f1e6018fad0 2021-02-10T11:25:05.865 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:05.862 7f1e56ffd700 1 -- 172.21.2.102:0/3366722123 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e580049a0 con 0x7f1e6018fad0 2021-02-10T11:25:05.865 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:05.862 7f1e685f3700 1 -- 172.21.2.102:0/3366722123 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e60196710 con 0x7f1e6018fad0 2021-02-10T11:25:05.865 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:05.862 7f1e685f3700 1 -- 172.21.2.102:0/3366722123 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e60196710 con 0x7f1e6018fad0 2021-02-10T11:25:05.865 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:05.862 7f1e56ffd700 1 -- 172.21.2.102:0/3366722123 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1e58006030 con 0x7f1e6018fad0 2021-02-10T11:25:05.866 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:05.862 7f1e56ffd700 1 -- 172.21.2.102:0/3366722123 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1e58004b80 con 0x7f1e6018fad0 2021-02-10T11:25:05.866 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:05.862 7f1e56ffd700 1 --2- 172.21.2.102:0/3366722123 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1e48018730 0x7f1e4801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:05.866 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:05.862 7f1e56ffd700 1 -- 172.21.2.102:0/3366722123 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f1e580329f0 con 0x7f1e6018fad0 2021-02-10T11:25:05.866 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:05.863 7f1e6640f700 1 --2- 172.21.2.102:0/3366722123 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1e48018730 0x7f1e4801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:05.866 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:05.863 7f1e6640f700 1 --2- 172.21.2.102:0/3366722123 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1e48018730 0x7f1e4801ac10 crc :-1 s=READY pgs=655 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:05.893 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:05.894 7f1e685f3700 1 -- 172.21.2.102:0/3366722123 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e600ed630 con 0x7f1e6018fad0 2021-02-10T11:25:05.895 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:05.896 7f1e56ffd700 1 -- 172.21.2.102:0/3366722123 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f1e5800be10 con 0x7f1e6018fad0 2021-02-10T11:25:06.026 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:06.027 7f1e685f3700 1 -- 172.21.2.102:0/3366722123 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f1e600f3a10 con 0x7f1e6018fad0 2021-02-10T11:25:06.026 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:06.027 7f1e56ffd700 1 -- 172.21.2.102:0/3366722123 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v35) v1 ==== 80+0+4513 (secure 0 0 0) 0x7f1e580049a0 con 0x7f1e6018fad0 2021-02-10T11:25:06.027 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:25:06.027 INFO:teuthology.orchestra.run.gibba002.stdout:{ 2021-02-10T11:25:06.027 INFO:teuthology.orchestra.run.gibba002.stdout: "auth_dump": [ 2021-02-10T11:25:06.027 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:25:06.027 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "mds.a", 2021-02-10T11:25:06.027 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAevSNgsgyEARAAvr0cUjjCoEHLxaZSZ2HtVA==", 2021-02-10T11:25:06.027 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:25:06.027 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow", 2021-02-10T11:25:06.028 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:25:06.028 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:25:06.028 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:25:06.028 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:25:06.028 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:25:06.028 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:25:06.028 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "mds.b", 2021-02-10T11:25:06.028 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAevSNgwOFMMRAALfb1fbX56ao1TU8ClnTW1A==", 2021-02-10T11:25:06.028 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:25:06.028 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow", 2021-02-10T11:25:06.028 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:25:06.028 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:25:06.029 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:25:06.029 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:25:06.029 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:25:06.029 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:25:06.029 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.0", 2021-02-10T11:25:06.029 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAmvSNgj+bMMBAAD3HUD6fLiMAg+++iNzLJKQ==", 2021-02-10T11:25:06.029 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:25:06.029 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:25:06.029 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:25:06.029 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:25:06.029 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:25:06.029 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:25:06.029 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:25:06.030 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.1", 2021-02-10T11:25:06.030 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQApvSNgnvc+MRAAAOxqlNwiyLHqNITOrYeFKQ==", 2021-02-10T11:25:06.030 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:25:06.030 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:25:06.030 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:25:06.030 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:25:06.030 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:25:06.030 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:25:06.030 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:25:06.030 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.2", 2021-02-10T11:25:06.030 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAsvSNgZ4SbMRAA76IOYWfbiAuBoM/4k81GxA==", 2021-02-10T11:25:06.030 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:25:06.031 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:25:06.031 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:25:06.031 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:25:06.031 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:25:06.031 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:25:06.031 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:25:06.031 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.3", 2021-02-10T11:25:06.031 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAvvSNgFk3rMRAAWCmKKyw0LDQ+UlAHZAITOw==", 2021-02-10T11:25:06.031 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:25:06.031 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:25:06.031 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:25:06.031 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:25:06.033 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:25:06.033 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:25:06.033 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:25:06.033 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.4", 2021-02-10T11:25:06.033 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQA4vSNgpKcNJBAApJMmvj5YxVHPDOmvmQONOg==", 2021-02-10T11:25:06.033 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:25:06.033 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:25:06.033 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:25:06.033 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:25:06.033 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:25:06.033 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:25:06.034 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:25:06.034 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.5", 2021-02-10T11:25:06.034 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQA7vSNgf2ujJBAAfkuYpyb3WTzc8V5kw+Wktg==", 2021-02-10T11:25:06.034 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:25:06.034 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:25:06.035 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:25:06.035 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:25:06.035 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:25:06.035 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:25:06.035 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:25:06.035 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.6", 2021-02-10T11:25:06.035 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQA+vSNgqw9uJRAAFMU0R+UxU9C3o/9byb3e0g==", 2021-02-10T11:25:06.036 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:25:06.036 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:25:06.036 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:25:06.036 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:25:06.036 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:25:06.036 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:25:06.036 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:25:06.036 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.7", 2021-02-10T11:25:06.036 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQBBvSNgvWSrJRAA2KaCHmLpzovRgYzLvyofTQ==", 2021-02-10T11:25:06.036 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:25:06.036 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:25:06.036 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:25:06.037 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:25:06.037 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:25:06.037 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:25:06.037 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:25:06.037 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "client.0", 2021-02-10T11:25:06.037 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAfvSNgZd9XIBAAvyrvl8fdKuHnJZfczfRoFw==", 2021-02-10T11:25:06.037 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:25:06.037 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow", 2021-02-10T11:25:06.037 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow r", 2021-02-10T11:25:06.037 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:25:06.037 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:25:06.037 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:25:06.037 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:25:06.037 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "client.1", 2021-02-10T11:25:06.038 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAfvSNgSyfXNhAAIduPnF7Vd/6HSkJ0MlcMSA==", 2021-02-10T11:25:06.038 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:25:06.038 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow", 2021-02-10T11:25:06.038 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow r", 2021-02-10T11:25:06.038 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:25:06.038 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:25:06.038 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:25:06.038 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:25:06.038 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "client.2", 2021-02-10T11:25:06.038 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAgvSNgiIG/ERAA9ZTamaTI7ZsPSnxBqjplCg==", 2021-02-10T11:25:06.038 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:25:06.038 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow", 2021-02-10T11:25:06.038 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow r", 2021-02-10T11:25:06.039 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:25:06.039 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:25:06.039 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:25:06.039 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:25:06.039 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "client.3", 2021-02-10T11:25:06.039 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAgvSNgOWpOKBAAlXsEB04Q/GvtL06PqRMLeA==", 2021-02-10T11:25:06.040 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:25:06.040 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow", 2021-02-10T11:25:06.040 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow r", 2021-02-10T11:25:06.040 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:25:06.040 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:25:06.040 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:25:06.041 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:25:06.041 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "client.admin", 2021-02-10T11:25:06.042 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAavSNgO3GQBhAAL+NVnYskQRKVWLgyDPY/xw==", 2021-02-10T11:25:06.042 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:25:06.042 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow *", 2021-02-10T11:25:06.042 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:25:06.042 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:25:06.042 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:25:06.042 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:25:06.042 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:25:06.042 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:25:06.042 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "mgr.x", 2021-02-10T11:25:06.042 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAdvSNgXZZgJhAA7dtBSGzH+DsBwnddZpOBJA==", 2021-02-10T11:25:06.043 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:25:06.043 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow *", 2021-02-10T11:25:06.043 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow profile mgr", 2021-02-10T11:25:06.043 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:25:06.043 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:25:06.043 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:25:06.043 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:25:06.043 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "mgr.y", 2021-02-10T11:25:06.043 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAdvSNgVqQFDRAAwo6YEdWZCAr5GdOWVL/Qvg==", 2021-02-10T11:25:06.043 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:25:06.043 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow *", 2021-02-10T11:25:06.043 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow profile mgr", 2021-02-10T11:25:06.043 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:25:06.044 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:25:06.044 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:25:06.044 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:25:06.044 INFO:teuthology.orchestra.run.gibba002.stdout:} 2021-02-10T11:25:06.045 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:06.029 7f1e685f3700 1 -- 172.21.2.102:0/3366722123 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1e48018730 msgr2=0x7f1e4801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:06.045 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:06.029 7f1e685f3700 1 --2- 172.21.2.102:0/3366722123 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1e48018730 0x7f1e4801ac10 crc :-1 s=READY pgs=655 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:06.045 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:06.029 7f1e685f3700 1 -- 172.21.2.102:0/3366722123 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1e6018fad0 msgr2=0x7f1e60196fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:06.045 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:06.029 7f1e685f3700 1 --2- 172.21.2.102:0/3366722123 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1e6018fad0 0x7f1e60196fa0 secure :-1 s=READY pgs=1426 cs=0 l=1 rev1=1 rx=0x7f1e58009980 tx=0x7f1e5800bcd0).stop 2021-02-10T11:25:06.046 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:06.029 7f1e685f3700 1 -- 172.21.2.102:0/3366722123 shutdown_connections 2021-02-10T11:25:06.046 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:06.029 7f1e685f3700 1 --2- 172.21.2.102:0/3366722123 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1e6018fad0 0x7f1e60196fa0 unknown :-1 s=CLOSED pgs=1426 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:06.046 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:06.029 7f1e685f3700 1 --2- 172.21.2.102:0/3366722123 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1e48018730 0x7f1e4801ac10 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:06.046 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:06.029 7f1e685f3700 1 --2- 172.21.2.102:0/3366722123 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1e601960f0 0x7f1e6018f4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:06.046 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:06.029 7f1e685f3700 1 --2- 172.21.2.102:0/3366722123 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1e601b8850 0x7f1e601b7ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:06.046 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:06.029 7f1e685f3700 1 -- 172.21.2.102:0/3366722123 >> 172.21.2.102:0/3366722123 conn(0x7f1e6019f890 msgr2=0x7f1e6018c3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:25:06.046 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:06.030 7f1e685f3700 1 -- 172.21.2.102:0/3366722123 shutdown_connections 2021-02-10T11:25:06.046 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:06.030 7f1e685f3700 1 -- 172.21.2.102:0/3366722123 wait complete. 2021-02-10T11:25:06.047 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000190758 --group_name group_0000000000878856 2021-02-10T11:25:06.477 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:06.462 7fb9b3734700 1 Processor -- start 2021-02-10T11:25:06.478 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:06.462 7fb9b3734700 1 -- start start 2021-02-10T11:25:06.478 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:06.462 7fb9b3734700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fb9ac0e5180 0x7fb9ac1b38f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:06.478 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:06.462 7fb9b3734700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fb9ac0e5580 0x7fb9ac07a120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:06.478 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:06.462 7fb9b3734700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9ac078b90 con 0x7fb9ac1b3f10 2021-02-10T11:25:06.478 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:06.462 7fb9b3734700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9ac0d1910 con 0x7fb9ac0e5180 2021-02-10T11:25:06.478 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:06.462 7fb9b3734700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9ac0d1b00 con 0x7fb9ac0e5580 2021-02-10T11:25:06.478 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:06.463 7fb9b1550700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fb9ac0e5180 0x7fb9ac1b38f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:06.478 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:06.463 7fb9b0d4f700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fb9ac1b3f10 0x7fb9ac077510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:49836/0 (socket says 172.21.2.102:49836) 2021-02-10T11:25:06.478 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:06.463 7fb9b0d4f700 1 -- 172.21.2.102:0/3658556784 learned_addr learned my addr 172.21.2.102:0/3658556784 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:25:06.478 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:06.463 7fb9b1550700 1 -- 172.21.2.102:0/3658556784 >> v2:172.21.2.113:3300/0 conn(0x7fb9ac0e5580 msgr2=0x7fb9ac07a120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:25:06.479 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:06.463 7fb9b1550700 1 --2- 172.21.2.102:0/3658556784 >> v2:172.21.2.113:3300/0 conn(0x7fb9ac0e5580 0x7fb9ac07a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:06.479 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:06.463 7fb9b1550700 1 -- 172.21.2.102:0/3658556784 >> v1:172.21.2.102:6789/0 conn(0x7fb9ac1b3f10 legacy=0x7fb9ac077510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:06.479 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:06.463 7fb9b1550700 1 -- 172.21.2.102:0/3658556784 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9ac0d1d20 con 0x7fb9ac0e5180 2021-02-10T11:25:06.479 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:06.463 7fb9b1550700 1 --2- 172.21.2.102:0/3658556784 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb9ac0e5180 0x7fb9ac1b38f0 secure :-1 s=READY pgs=1428 cs=0 l=1 rev1=1 rx=0x7fb9a8006650 tx=0x7fb9a8009a80).ready entity=mon.0 client_cookie=b65f3dba74210fab server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:06.479 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:06.463 7fb9a3fff700 1 -- 172.21.2.102:0/3658556784 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb9a800c0c0 con 0x7fb9ac0e5180 2021-02-10T11:25:06.479 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:06.463 7fb9a3fff700 1 -- 172.21.2.102:0/3658556784 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9a800c340 con 0x7fb9ac0e5180 2021-02-10T11:25:06.479 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:06.463 7fb9a3fff700 1 -- 172.21.2.102:0/3658556784 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb9a800c860 con 0x7fb9ac0e5180 2021-02-10T11:25:06.479 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:06.463 7fb9b3734700 1 -- 172.21.2.102:0/3658556784 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb9ac0e5180 msgr2=0x7fb9ac1b38f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:06.479 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:06.463 7fb9b3734700 1 --2- 172.21.2.102:0/3658556784 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb9ac0e5180 0x7fb9ac1b38f0 secure :-1 s=READY pgs=1428 cs=0 l=1 rev1=1 rx=0x7fb9a8006650 tx=0x7fb9a8009a80).stop 2021-02-10T11:25:06.479 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:06.464 7fb9b3734700 1 -- 172.21.2.102:0/3658556784 shutdown_connections 2021-02-10T11:25:06.479 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:06.464 7fb9b3734700 1 --2- 172.21.2.102:0/3658556784 >> v2:172.21.2.113:3300/0 conn(0x7fb9ac0e5580 0x7fb9ac07a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:06.479 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:06.464 7fb9b3734700 1 --2- 172.21.2.102:0/3658556784 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb9ac0e5180 0x7fb9ac1b38f0 unknown :-1 s=CLOSED pgs=1428 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:06.479 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:06.464 7fb9b3734700 1 -- 172.21.2.102:0/3658556784 >> 172.21.2.102:0/3658556784 conn(0x7fb9ac0ce830 msgr2=0x7fb9ac0e4670 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:25:06.480 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:06.464 7fb9b3734700 1 -- 172.21.2.102:0/3658556784 shutdown_connections 2021-02-10T11:25:06.480 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:06.464 7fb9b3734700 1 -- 172.21.2.102:0/3658556784 wait complete. 2021-02-10T11:25:06.480 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:06.479 7fb9b3734700 1 Processor -- start 2021-02-10T11:25:06.480 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:06.479 7fb9b3734700 1 -- start start 2021-02-10T11:25:06.481 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:06.479 7fb9b3734700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb9ac18e7c0 0x7fb9ac18ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:06.482 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:06.479 7fb9b3734700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb9ac1a5fc0 0x7fb9ac1a63c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:06.482 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:06.479 7fb9b3734700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb9ac1a69e0 0x7fb9ac0b5390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:06.482 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:06.479 7fb9b3734700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9ac1a6ef0 con 0x7fb9ac1a5fc0 2021-02-10T11:25:06.482 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:06.479 7fb9b3734700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9ac18f1e0 con 0x7fb9ac1a69e0 2021-02-10T11:25:06.482 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:06.479 7fb9b3734700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9ac0c1250 con 0x7fb9ac18e7c0 2021-02-10T11:25:06.482 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:06.479 7fb9b0d4f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb9ac1a5fc0 0x7fb9ac1a63c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:06.482 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:06.479 7fb9b0d4f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb9ac1a5fc0 0x7fb9ac1a63c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:57620/0 (socket says 172.21.2.102:57620) 2021-02-10T11:25:06.482 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:06.479 7fb9b0d4f700 1 -- 172.21.2.102:0/3286996205 learned_addr learned my addr 172.21.2.102:0/3286996205 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:25:06.482 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:06.480 7fb9b0d4f700 1 -- 172.21.2.102:0/3286996205 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb9ac18e7c0 msgr2=0x7fb9ac18ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:06.482 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:06.480 7fb9b0d4f700 1 --2- 172.21.2.102:0/3286996205 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb9ac18e7c0 0x7fb9ac18ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:06.482 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:06.480 7fb9b0d4f700 1 -- 172.21.2.102:0/3286996205 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb9ac1a69e0 msgr2=0x7fb9ac0b5390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:06.482 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:06.480 7fb9b0d4f700 1 --2- 172.21.2.102:0/3286996205 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb9ac1a69e0 0x7fb9ac0b5390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:06.482 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:06.480 7fb9b0d4f700 1 -- 172.21.2.102:0/3286996205 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9ac0b59b0 con 0x7fb9ac1a5fc0 2021-02-10T11:25:06.483 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:06.480 7fb9b0d4f700 1 --2- 172.21.2.102:0/3286996205 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb9ac1a5fc0 0x7fb9ac1a63c0 secure :-1 s=READY pgs=1429 cs=0 l=1 rev1=1 rx=0x7fb99c001ec0 tx=0x7fb99c00a8a0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:06.483 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:06.480 7fb9a1ffb700 1 -- 172.21.2.102:0/3286996205 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb99c00ac90 con 0x7fb9ac1a5fc0 2021-02-10T11:25:06.483 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:06.480 7fb9a1ffb700 1 -- 172.21.2.102:0/3286996205 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb99c0009b0 con 0x7fb9ac1a5fc0 2021-02-10T11:25:06.483 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:06.480 7fb9b3734700 1 -- 172.21.2.102:0/3286996205 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9ac0b59b0 con 0x7fb9ac1a5fc0 2021-02-10T11:25:06.483 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:06.480 7fb9b3734700 1 -- 172.21.2.102:0/3286996205 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9ac0b59b0 con 0x7fb9ac1a5fc0 2021-02-10T11:25:06.483 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:06.480 7fb9a1ffb700 1 -- 172.21.2.102:0/3286996205 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb99c004590 con 0x7fb9ac1a5fc0 2021-02-10T11:25:06.483 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:06.481 7fb9a1ffb700 1 -- 172.21.2.102:0/3286996205 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fb99c0043a0 con 0x7fb9ac1a5fc0 2021-02-10T11:25:06.483 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:06.481 7fb9a1ffb700 1 --2- 172.21.2.102:0/3286996205 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb98c0186d0 0x7fb98c01abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:06.483 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:06.481 7fb9a1ffb700 1 -- 172.21.2.102:0/3286996205 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fb99c0327c0 con 0x7fb9ac1a5fc0 2021-02-10T11:25:06.483 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:06.481 7fb9b1550700 1 --2- 172.21.2.102:0/3286996205 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb98c0186d0 0x7fb98c01abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:06.483 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:06.481 7fb9b1550700 1 --2- 172.21.2.102:0/3286996205 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb98c0186d0 0x7fb98c01abd0 crc :-1 s=READY pgs=656 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:06.511 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:06.513 7fb9b3734700 1 -- 172.21.2.102:0/3286996205 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9ac10bd10 con 0x7fb9ac1a5fc0 2021-02-10T11:25:06.514 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:06.515 7fb9a1ffb700 1 -- 172.21.2.102:0/3286996205 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fb99c00ac90 con 0x7fb9ac1a5fc0 2021-02-10T11:25:06.640 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:06.641 7fb9b3734700 1 -- 172.21.2.102:0/3286996205 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000190758", "target": ["mgr", ""], "group_name": "group_0000000000878856"}) v1 -- 0x7fb9ac0fe2a0 con 0x7fb98c0186d0 2021-02-10T11:25:06.651 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:06.652 7fb9a1ffb700 1 -- 172.21.2.102:0/3286996205 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fb9a8000940 con 0x7fb98c0186d0 2021-02-10T11:25:06.657 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:06.658 7fb9b3734700 1 -- 172.21.2.102:0/3286996205 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb98c0186d0 msgr2=0x7fb98c01abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:06.657 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:06.658 7fb9b3734700 1 --2- 172.21.2.102:0/3286996205 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb98c0186d0 0x7fb98c01abd0 crc :-1 s=READY pgs=656 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:06.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:06.658 7fb9b3734700 1 -- 172.21.2.102:0/3286996205 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb9ac1a5fc0 msgr2=0x7fb9ac1a63c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:06.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:06.658 7fb9b3734700 1 --2- 172.21.2.102:0/3286996205 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb9ac1a5fc0 0x7fb9ac1a63c0 secure :-1 s=READY pgs=1429 cs=0 l=1 rev1=1 rx=0x7fb99c001ec0 tx=0x7fb99c00a8a0).stop 2021-02-10T11:25:06.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:06.658 7fb9b3734700 1 -- 172.21.2.102:0/3286996205 shutdown_connections 2021-02-10T11:25:06.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:06.658 7fb9b3734700 1 --2- 172.21.2.102:0/3286996205 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb98c0186d0 0x7fb98c01abd0 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:06.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:06.658 7fb9b3734700 1 --2- 172.21.2.102:0/3286996205 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb9ac1a69e0 0x7fb9ac0b5390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:06.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:06.658 7fb9b3734700 1 --2- 172.21.2.102:0/3286996205 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb9ac1a5fc0 0x7fb9ac1a63c0 unknown :-1 s=CLOSED pgs=1429 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:06.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:06.658 7fb9b3734700 1 --2- 172.21.2.102:0/3286996205 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb9ac18e7c0 0x7fb9ac18ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:06.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:06.658 7fb9b3734700 1 -- 172.21.2.102:0/3286996205 >> 172.21.2.102:0/3286996205 conn(0x7fb9ac08b870 msgr2=0x7fb9ac1108f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:25:06.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:06.658 7fb9b3734700 1 -- 172.21.2.102:0/3286996205 shutdown_connections 2021-02-10T11:25:06.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:06.658 7fb9b3734700 1 -- 172.21.2.102:0/3286996205 wait complete. 2021-02-10T11:25:06.666 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolumegroup rm cephfs group_0000000000878856 2021-02-10T11:25:07.097 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:07.081 7f74f9520700 1 Processor -- start 2021-02-10T11:25:07.098 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:07.081 7f74f9520700 1 -- start start 2021-02-10T11:25:07.098 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:07.081 7f74f9520700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f74f40e5170 0x7f74f41b38e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:07.098 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:07.081 7f74f9520700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74f4078b90 con 0x7f74f41b3f00 2021-02-10T11:25:07.098 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:07.081 7f74f9520700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74f40784f0 con 0x7f74f40e5570 2021-02-10T11:25:07.098 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:07.081 7f74f9520700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74f4078710 con 0x7f74f40e5170 2021-02-10T11:25:07.099 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:07.081 7f74f32bf700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f74f40e5170 0x7f74f41b38e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:07.099 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:07.081 7f74f32bf700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f74f40e5170 0x7f74f41b38e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:57624/0 (socket says 172.21.2.102:57624) 2021-02-10T11:25:07.099 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:07.081 7f74f32bf700 1 -- 172.21.2.102:0/1553250290 learned_addr learned my addr 172.21.2.102:0/1553250290 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:25:07.099 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:07.081 7f74f2abe700 1 --1- 172.21.2.102:0/1553250290 >> v1:172.21.2.102:6789/0 conn(0x7f74f41b3f00 0x7f74f4077510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:49848/0 (socket says 172.21.2.102:49848) 2021-02-10T11:25:07.099 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:07.082 7f74f32bf700 1 -- 172.21.2.102:0/1553250290 >> v1:172.21.2.113:6789/0 conn(0x7f74f40e5570 legacy=0x7f74f4077d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:25:07.100 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:07.082 7f74f32bf700 1 -- 172.21.2.102:0/1553250290 >> v1:172.21.2.102:6789/0 conn(0x7f74f41b3f00 legacy=0x7f74f4077510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:07.100 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:07.082 7f74f32bf700 1 -- 172.21.2.102:0/1553250290 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74f4078930 con 0x7f74f40e5170 2021-02-10T11:25:07.100 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:07.082 7f74f32bf700 1 --2- 172.21.2.102:0/1553250290 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f74f40e5170 0x7f74f41b38e0 secure :-1 s=READY pgs=1431 cs=0 l=1 rev1=1 rx=0x7f74e4006700 tx=0x7f74e4009b80).ready entity=mon.0 client_cookie=2464910b7b4b7874 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:07.100 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:07.082 7f74f22bd700 1 -- 172.21.2.102:0/1553250290 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f74e400c0c0 con 0x7f74f40e5170 2021-02-10T11:25:07.100 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:07.082 7f74f22bd700 1 -- 172.21.2.102:0/1553250290 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74e400c340 con 0x7f74f40e5170 2021-02-10T11:25:07.100 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:07.082 7f74f22bd700 1 -- 172.21.2.102:0/1553250290 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f74e400c860 con 0x7f74f40e5170 2021-02-10T11:25:07.101 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:07.082 7f74f9520700 1 -- 172.21.2.102:0/1553250290 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f74f40e5170 msgr2=0x7f74f41b38e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:07.101 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:07.082 7f74f9520700 1 --2- 172.21.2.102:0/1553250290 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f74f40e5170 0x7f74f41b38e0 secure :-1 s=READY pgs=1431 cs=0 l=1 rev1=1 rx=0x7f74e4006700 tx=0x7f74e4009b80).stop 2021-02-10T11:25:07.101 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:07.083 7f74f9520700 1 -- 172.21.2.102:0/1553250290 shutdown_connections 2021-02-10T11:25:07.101 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:07.083 7f74f9520700 1 --2- 172.21.2.102:0/1553250290 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f74f40e5170 0x7f74f41b38e0 unknown :-1 s=CLOSED pgs=1431 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:07.101 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:07.083 7f74f9520700 1 -- 172.21.2.102:0/1553250290 >> 172.21.2.102:0/1553250290 conn(0x7f74f40ce820 msgr2=0x7f74f40e4660 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:25:07.101 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:07.083 7f74f9520700 1 -- 172.21.2.102:0/1553250290 shutdown_connections 2021-02-10T11:25:07.102 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:07.083 7f74f9520700 1 -- 172.21.2.102:0/1553250290 wait complete. 2021-02-10T11:25:07.102 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:07.097 7f74f9520700 1 Processor -- start 2021-02-10T11:25:07.102 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:07.097 7f74f9520700 1 -- start start 2021-02-10T11:25:07.102 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:07.098 7f74f9520700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f74f41046a0 0x7f74f40c38d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:07.102 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:07.098 7f74f9520700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f74f4090df0 0x7f74f410f8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:07.103 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:07.098 7f74f9520700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f74f40911f0 0x7f74f410fee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:07.103 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:07.098 7f74f9520700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74f4110500 con 0x7f74f41046a0 2021-02-10T11:25:07.103 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:07.098 7f74f9520700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74f41c9420 con 0x7f74f40911f0 2021-02-10T11:25:07.103 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:07.098 7f74f9520700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74f41c9640 con 0x7f74f4090df0 2021-02-10T11:25:07.103 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:07.098 7f74f32bf700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f74f41046a0 0x7f74f40c38d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:07.103 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:07.098 7f74f32bf700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f74f41046a0 0x7f74f40c38d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:57630/0 (socket says 172.21.2.102:57630) 2021-02-10T11:25:07.104 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:07.098 7f74f32bf700 1 -- 172.21.2.102:0/2773146286 learned_addr learned my addr 172.21.2.102:0/2773146286 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:25:07.104 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:07.098 7f74f32bf700 1 -- 172.21.2.102:0/2773146286 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f74f4090df0 msgr2=0x7f74f410f8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:07.104 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:07.098 7f74f32bf700 1 --2- 172.21.2.102:0/2773146286 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f74f4090df0 0x7f74f410f8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:07.104 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:07.098 7f74f32bf700 1 -- 172.21.2.102:0/2773146286 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f74f40911f0 msgr2=0x7f74f410fee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:07.104 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:07.098 7f74f32bf700 1 --2- 172.21.2.102:0/2773146286 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f74f40911f0 0x7f74f410fee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:07.107 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:07.098 7f74f32bf700 1 -- 172.21.2.102:0/2773146286 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74f41c9860 con 0x7f74f41046a0 2021-02-10T11:25:07.107 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:07.099 7f74f32bf700 1 --2- 172.21.2.102:0/2773146286 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f74f41046a0 0x7f74f40c38d0 secure :-1 s=READY pgs=1432 cs=0 l=1 rev1=1 rx=0x7f74e400b5e0 tx=0x7f74e400b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:07.108 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:07.099 7f74ebfff700 1 -- 172.21.2.102:0/2773146286 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f74e400bd70 con 0x7f74f41046a0 2021-02-10T11:25:07.108 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:07.099 7f74ebfff700 1 -- 172.21.2.102:0/2773146286 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74e4002030 con 0x7f74f41046a0 2021-02-10T11:25:07.108 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:07.099 7f74f9520700 1 -- 172.21.2.102:0/2773146286 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74f41c9860 con 0x7f74f41046a0 2021-02-10T11:25:07.108 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:07.099 7f74f9520700 1 -- 172.21.2.102:0/2773146286 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74f41c9df0 con 0x7f74f41046a0 2021-02-10T11:25:07.108 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:07.099 7f74ebfff700 1 -- 172.21.2.102:0/2773146286 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f74e4002550 con 0x7f74f41046a0 2021-02-10T11:25:07.108 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:07.099 7f74ebfff700 1 -- 172.21.2.102:0/2773146286 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f74e4000c90 con 0x7f74f41046a0 2021-02-10T11:25:07.109 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:07.099 7f74ebfff700 1 --2- 172.21.2.102:0/2773146286 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f74d80186c0 0x7f74d801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:07.109 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:07.099 7f74ebfff700 1 -- 172.21.2.102:0/2773146286 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f74e40325e0 con 0x7f74f41046a0 2021-02-10T11:25:07.109 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:07.100 7f74f2abe700 1 --2- 172.21.2.102:0/2773146286 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f74d80186c0 0x7f74d801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:07.109 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:07.100 7f74f2abe700 1 --2- 172.21.2.102:0/2773146286 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f74d80186c0 0x7f74d801abc0 crc :-1 s=READY pgs=657 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:07.131 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:07.131 7f74f9520700 1 -- 172.21.2.102:0/2773146286 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74f41035c0 con 0x7f74f41046a0 2021-02-10T11:25:07.133 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:07.134 7f74ebfff700 1 -- 172.21.2.102:0/2773146286 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f74e4030d90 con 0x7f74f41046a0 2021-02-10T11:25:07.259 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:07.260 7f74f9520700 1 -- 172.21.2.102:0/2773146286 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolumegroup rm", "vol_name": "cephfs", "target": ["mgr", ""], "group_name": "group_0000000000878856"}) v1 -- 0x7f74f41c9df0 con 0x7f74d80186c0 2021-02-10T11:25:07.262 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:07.264 7f74ebfff700 1 -- 172.21.2.102:0/2773146286 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f74dc001590 con 0x7f74d80186c0 2021-02-10T11:25:07.269 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:07.269 7f74f9520700 1 -- 172.21.2.102:0/2773146286 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f74d80186c0 msgr2=0x7f74d801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:07.269 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:07.269 7f74f9520700 1 --2- 172.21.2.102:0/2773146286 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f74d80186c0 0x7f74d801abc0 crc :-1 s=READY pgs=657 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:07.269 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:07.269 7f74f9520700 1 -- 172.21.2.102:0/2773146286 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f74f41046a0 msgr2=0x7f74f40c38d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:07.269 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:07.269 7f74f9520700 1 --2- 172.21.2.102:0/2773146286 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f74f41046a0 0x7f74f40c38d0 secure :-1 s=READY pgs=1432 cs=0 l=1 rev1=1 rx=0x7f74e400b5e0 tx=0x7f74e400b980).stop 2021-02-10T11:25:07.269 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:07.269 7f74f9520700 1 -- 172.21.2.102:0/2773146286 shutdown_connections 2021-02-10T11:25:07.269 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:07.269 7f74f9520700 1 --2- 172.21.2.102:0/2773146286 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f74d80186c0 0x7f74d801abc0 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:07.269 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:07.269 7f74f9520700 1 --2- 172.21.2.102:0/2773146286 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f74f40911f0 0x7f74f410fee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:07.269 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:07.269 7f74f9520700 1 --2- 172.21.2.102:0/2773146286 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f74f4090df0 0x7f74f410f8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:07.269 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:07.269 7f74f9520700 1 --2- 172.21.2.102:0/2773146286 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f74f41046a0 0x7f74f40c38d0 unknown :-1 s=CLOSED pgs=1432 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:07.269 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:07.269 7f74f9520700 1 -- 172.21.2.102:0/2773146286 >> 172.21.2.102:0/2773146286 conn(0x7f74f40984c0 msgr2=0x7f74f4097990 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:25:07.270 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:07.270 7f74f9520700 1 -- 172.21.2.102:0/2773146286 shutdown_connections 2021-02-10T11:25:07.270 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:07.270 7f74f9520700 1 -- 172.21.2.102:0/2773146286 wait complete. 2021-02-10T11:25:07.277 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2021-02-10T11:25:07.277 DEBUG:teuthology.orchestra.run.gibba002:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:25:07.623 DEBUG:teuthology.orchestra.run.gibba013:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:25:07.968 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2021-02-10T11:25:07.969 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:07.969 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:25:07.969 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:25:08.304 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:25:08.304 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:25:08.305 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@gibba002.front.sepia.ceph.com... 2021-02-10T11:25:08.305 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-02-10T11:25:08.305 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:25:08.665 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba002.stderr:ceph-fuse[153237]: fuse finished with error 0 and tester_r 0 2021-02-10T11:25:08.667 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:08.667 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:25:08.668 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:25:09.003 INFO:teuthology.orchestra.run.gibba002.stdout:ext2/ext3 2021-02-10T11:25:09.004 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2021-02-10T11:25:09.004 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:25:09.004 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:25:09.338 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:09.338 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:25:09.338 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:25:09.672 INFO:teuthology.orchestra.run.gibba002.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.1’: No such file or directory 2021-02-10T11:25:09.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:25:09.673 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2021-02-10T11:25:09.673 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:25:09.673 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:25:10.007 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:10.008 DEBUG:teuthology.orchestra.run.gibba013:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:25:10.008 DEBUG:teuthology.orchestra.run.gibba013:> ' -- /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:25:10.343 INFO:teuthology.orchestra.run.gibba013.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.2’: No such file or directory 2021-02-10T11:25:10.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:25:10.343 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2021-02-10T11:25:10.343 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:25:10.344 DEBUG:teuthology.orchestra.run.gibba013:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:25:10.677 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:10.678 DEBUG:teuthology.orchestra.run.gibba013:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:25:10.678 DEBUG:teuthology.orchestra.run.gibba013:> ' -- /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:25:11.012 INFO:teuthology.orchestra.run.gibba013.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.3’: No such file or directory 2021-02-10T11:25:11.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:25:11.013 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.3 2021-02-10T11:25:11.013 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:25:11.013 DEBUG:teuthology.orchestra.run.gibba013:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:25:11.348 DEBUG:teuthology.orchestra.run.gibba002:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2021-02-10T11:25:11.352 DEBUG:teuthology.orchestra.run.gibba013:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2021-02-10T11:25:11.711 INFO:teuthology.orchestra.run.gibba002.stdout:[global] 2021-02-10T11:25:11.711 INFO:teuthology.orchestra.run.gibba002.stdout: chdir = "" 2021-02-10T11:25:11.711 INFO:teuthology.orchestra.run.gibba002.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2021-02-10T11:25:11.712 INFO:teuthology.orchestra.run.gibba002.stdout: auth supported = cephx 2021-02-10T11:25:11.712 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:25:11.712 INFO:teuthology.orchestra.run.gibba002.stdout: filestore xattr use omap = true 2021-02-10T11:25:11.712 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:25:11.712 INFO:teuthology.orchestra.run.gibba002.stdout: mon clock drift allowed = 1.000 2021-02-10T11:25:11.712 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:25:11.712 INFO:teuthology.orchestra.run.gibba002.stdout: osd crush chooseleaf type = 0 2021-02-10T11:25:11.712 INFO:teuthology.orchestra.run.gibba002.stdout: auth debug = true 2021-02-10T11:25:11.712 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:25:11.712 INFO:teuthology.orchestra.run.gibba002.stdout: ms die on old message = true 2021-02-10T11:25:11.712 INFO:teuthology.orchestra.run.gibba002.stdout: ms die on bug = true 2021-02-10T11:25:11.712 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:25:11.713 INFO:teuthology.orchestra.run.gibba002.stdout: mon max pg per osd = 10000 # >= luminous 2021-02-10T11:25:11.713 INFO:teuthology.orchestra.run.gibba002.stdout: mon pg warn max object skew = 0 2021-02-10T11:25:11.713 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:25:11.713 INFO:teuthology.orchestra.run.gibba002.stdout: osd pool default size = 2 2021-02-10T11:25:11.713 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:25:11.713 INFO:teuthology.orchestra.run.gibba002.stdout: mon osd allow primary affinity = true 2021-02-10T11:25:11.713 INFO:teuthology.orchestra.run.gibba002.stdout: mon osd allow pg remap = true 2021-02-10T11:25:11.713 INFO:teuthology.orchestra.run.gibba002.stdout: mon warn on legacy crush tunables = false 2021-02-10T11:25:11.713 INFO:teuthology.orchestra.run.gibba002.stdout: mon warn on crush straw calc version zero = false 2021-02-10T11:25:11.713 INFO:teuthology.orchestra.run.gibba002.stdout: mon warn on no sortbitwise = false 2021-02-10T11:25:11.713 INFO:teuthology.orchestra.run.gibba002.stdout: mon warn on osd down out interval zero = false 2021-02-10T11:25:11.713 INFO:teuthology.orchestra.run.gibba002.stdout: mon warn on too few osds = false 2021-02-10T11:25:11.714 INFO:teuthology.orchestra.run.gibba002.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2021-02-10T11:25:11.714 INFO:teuthology.orchestra.run.gibba002.stdout: mon_warn_on_pool_no_redundancy = false 2021-02-10T11:25:11.714 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:25:11.714 INFO:teuthology.orchestra.run.gibba002.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2021-02-10T11:25:11.714 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:25:11.714 INFO:teuthology.orchestra.run.gibba002.stdout: osd default data pool replay window = 5 2021-02-10T11:25:11.714 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:25:11.714 INFO:teuthology.orchestra.run.gibba002.stdout: mon allow pool delete = true 2021-02-10T11:25:11.714 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:25:11.714 INFO:teuthology.orchestra.run.gibba002.stdout: mon cluster log file level = debug 2021-02-10T11:25:11.714 INFO:teuthology.orchestra.run.gibba002.stdout: debug asserts on shutdown = true 2021-02-10T11:25:11.714 INFO:teuthology.orchestra.run.gibba002.stdout: mon health detail to clog = false 2021-02-10T11:25:11.715 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:25:11.715 INFO:teuthology.orchestra.run.gibba002.stdout: # we see this fail in qa on *nautilus*; bump up retries 2021-02-10T11:25:11.715 INFO:teuthology.orchestra.run.gibba002.stdout: mon_client_directed_command_retry = 4 2021-02-10T11:25:11.715 INFO:teuthology.orchestra.run.gibba002.stdout: mon host = "172.21.2.102,172.21.2.113,[v2:172.21.2.113:3301,v1:172.21.2.113:6790]" 2021-02-10T11:25:11.715 INFO:teuthology.orchestra.run.gibba002.stdout: fsid = 68db30ea-dbce-4aef-a6df-f01c6fea0513 2021-02-10T11:25:11.715 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:25:11.715 INFO:teuthology.orchestra.run.gibba002.stdout:[osd] 2021-02-10T11:25:11.715 INFO:teuthology.orchestra.run.gibba002.stdout: osd journal size = 100 2021-02-10T11:25:11.715 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:25:11.715 INFO:teuthology.orchestra.run.gibba002.stdout: osd scrub load threshold = 5.0 2021-02-10T11:25:11.715 INFO:teuthology.orchestra.run.gibba002.stdout: osd scrub max interval = 600 2021-02-10T11:25:11.715 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:25:11.715 INFO:teuthology.orchestra.run.gibba002.stdout: osd recover clone overlap = true 2021-02-10T11:25:11.716 INFO:teuthology.orchestra.run.gibba002.stdout: osd recovery max chunk = 1048576 2021-02-10T11:25:11.716 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:25:11.716 INFO:teuthology.orchestra.run.gibba002.stdout: osd debug shutdown = true 2021-02-10T11:25:11.716 INFO:teuthology.orchestra.run.gibba002.stdout: osd debug op order = true 2021-02-10T11:25:11.716 INFO:teuthology.orchestra.run.gibba002.stdout: osd debug verify stray on activate = true 2021-02-10T11:25:11.716 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:25:11.716 INFO:teuthology.orchestra.run.gibba002.stdout: osd open classes on start = true 2021-02-10T11:25:11.716 INFO:teuthology.orchestra.run.gibba002.stdout: osd debug pg log writeout = true 2021-02-10T11:25:11.716 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:25:11.716 INFO:teuthology.orchestra.run.gibba002.stdout: osd deep scrub update digest min age = 30 2021-02-10T11:25:11.716 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:25:11.716 INFO:teuthology.orchestra.run.gibba002.stdout: osd map max advance = 10 2021-02-10T11:25:11.716 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:25:11.717 INFO:teuthology.orchestra.run.gibba002.stdout: journal zero on create = true 2021-02-10T11:25:11.717 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:25:11.717 INFO:teuthology.orchestra.run.gibba002.stdout: filestore ondisk finisher threads = 3 2021-02-10T11:25:11.717 INFO:teuthology.orchestra.run.gibba002.stdout: filestore apply finisher threads = 3 2021-02-10T11:25:11.717 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:25:11.717 INFO:teuthology.orchestra.run.gibba002.stdout: bdev debug aio = true 2021-02-10T11:25:11.717 INFO:teuthology.orchestra.run.gibba002.stdout: osd debug misdirected ops = true 2021-02-10T11:25:11.717 INFO:teuthology.orchestra.run.gibba002.stdout: bdev async discard = True 2021-02-10T11:25:11.717 INFO:teuthology.orchestra.run.gibba002.stdout: bdev enable discard = True 2021-02-10T11:25:11.717 INFO:teuthology.orchestra.run.gibba002.stdout: bluestore allocator = bitmap 2021-02-10T11:25:11.717 INFO:teuthology.orchestra.run.gibba002.stdout: bluestore block size = 96636764160 2021-02-10T11:25:11.717 INFO:teuthology.orchestra.run.gibba002.stdout: bluestore fsck on mount = True 2021-02-10T11:25:11.717 INFO:teuthology.orchestra.run.gibba002.stdout: debug bluefs = 20 2021-02-10T11:25:11.717 INFO:teuthology.orchestra.run.gibba002.stdout: debug bluestore = 20 2021-02-10T11:25:11.718 INFO:teuthology.orchestra.run.gibba002.stdout: debug filestore = 20 2021-02-10T11:25:11.718 INFO:teuthology.orchestra.run.gibba002.stdout: debug journal = 20 2021-02-10T11:25:11.718 INFO:teuthology.orchestra.run.gibba002.stdout: debug ms = 20 2021-02-10T11:25:11.718 INFO:teuthology.orchestra.run.gibba002.stdout: debug osd = 25 2021-02-10T11:25:11.718 INFO:teuthology.orchestra.run.gibba002.stdout: debug rocksdb = 10 2021-02-10T11:25:11.718 INFO:teuthology.orchestra.run.gibba002.stdout: mon osd backfillfull_ratio = 0.85 2021-02-10T11:25:11.718 INFO:teuthology.orchestra.run.gibba002.stdout: mon osd full ratio = 0.9 2021-02-10T11:25:11.718 INFO:teuthology.orchestra.run.gibba002.stdout: mon osd nearfull ratio = 0.8 2021-02-10T11:25:11.718 INFO:teuthology.orchestra.run.gibba002.stdout: osd failsafe full ratio = 0.95 2021-02-10T11:25:11.718 INFO:teuthology.orchestra.run.gibba002.stdout: osd objectstore = bluestore 2021-02-10T11:25:11.718 INFO:teuthology.orchestra.run.gibba002.stdout: osd op complaint time = 180 2021-02-10T11:25:11.718 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:25:11.718 INFO:teuthology.orchestra.run.gibba002.stdout:[mgr] 2021-02-10T11:25:11.719 INFO:teuthology.orchestra.run.gibba002.stdout: debug ms = 1 2021-02-10T11:25:11.719 INFO:teuthology.orchestra.run.gibba002.stdout: debug mgr = 20 2021-02-10T11:25:11.719 INFO:teuthology.orchestra.run.gibba002.stdout: debug mon = 20 2021-02-10T11:25:11.719 INFO:teuthology.orchestra.run.gibba002.stdout: debug auth = 20 2021-02-10T11:25:11.719 INFO:teuthology.orchestra.run.gibba002.stdout: mon reweight min pgs per osd = 4 2021-02-10T11:25:11.719 INFO:teuthology.orchestra.run.gibba002.stdout: mon reweight min bytes per osd = 10 2021-02-10T11:25:11.719 INFO:teuthology.orchestra.run.gibba002.stdout: debug client = 10 2021-02-10T11:25:11.719 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:25:11.719 INFO:teuthology.orchestra.run.gibba002.stdout:[mon] 2021-02-10T11:25:11.719 INFO:teuthology.orchestra.run.gibba002.stdout: debug ms = 1 2021-02-10T11:25:11.719 INFO:teuthology.orchestra.run.gibba002.stdout: debug mon = 20 2021-02-10T11:25:11.719 INFO:teuthology.orchestra.run.gibba002.stdout: debug paxos = 20 2021-02-10T11:25:11.719 INFO:teuthology.orchestra.run.gibba002.stdout: debug auth = 20 2021-02-10T11:25:11.720 INFO:teuthology.orchestra.run.gibba002.stdout: mon data avail warn = 5 2021-02-10T11:25:11.720 INFO:teuthology.orchestra.run.gibba002.stdout: mon mgr mkfs grace = 240 2021-02-10T11:25:11.720 INFO:teuthology.orchestra.run.gibba002.stdout: mon reweight min pgs per osd = 4 2021-02-10T11:25:11.720 INFO:teuthology.orchestra.run.gibba002.stdout: mon osd reporter subtree level = osd 2021-02-10T11:25:11.720 INFO:teuthology.orchestra.run.gibba002.stdout: mon osd prime pg temp = true 2021-02-10T11:25:11.720 INFO:teuthology.orchestra.run.gibba002.stdout: mon reweight min bytes per osd = 10 2021-02-10T11:25:11.720 INFO:teuthology.orchestra.run.gibba002.stdout: mon op complaint time = 120 2021-02-10T11:25:11.720 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:25:11.720 INFO:teuthology.orchestra.run.gibba002.stdout:[client] 2021-02-10T11:25:11.720 INFO:teuthology.orchestra.run.gibba002.stdout: rgw cache enabled = true 2021-02-10T11:25:11.720 INFO:teuthology.orchestra.run.gibba002.stdout: rgw enable ops log = true 2021-02-10T11:25:11.720 INFO:teuthology.orchestra.run.gibba002.stdout: rgw enable usage log = true 2021-02-10T11:25:11.720 INFO:teuthology.orchestra.run.gibba002.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2021-02-10T11:25:11.720 INFO:teuthology.orchestra.run.gibba002.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2021-02-10T11:25:11.721 INFO:teuthology.orchestra.run.gibba002.stdout: client mount timeout = 600 2021-02-10T11:25:11.721 INFO:teuthology.orchestra.run.gibba002.stdout: debug client = 20 2021-02-10T11:25:11.721 INFO:teuthology.orchestra.run.gibba002.stdout: debug ms = 1 2021-02-10T11:25:11.721 INFO:teuthology.orchestra.run.gibba002.stdout: pid file = "" 2021-02-10T11:25:11.721 INFO:teuthology.orchestra.run.gibba002.stdout:[mon.a] 2021-02-10T11:25:11.721 INFO:teuthology.orchestra.run.gibba002.stdout:[mon.b] 2021-02-10T11:25:11.721 INFO:teuthology.orchestra.run.gibba002.stdout:[mon.c] 2021-02-10T11:25:11.721 INFO:teuthology.orchestra.run.gibba002.stdout:[mds.a] 2021-02-10T11:25:11.721 INFO:teuthology.orchestra.run.gibba002.stdout:[mds.b] 2021-02-10T11:25:11.721 INFO:teuthology.orchestra.run.gibba002.stdout:[mds] 2021-02-10T11:25:11.721 INFO:teuthology.orchestra.run.gibba002.stdout: debug mds = 20 2021-02-10T11:25:11.721 INFO:teuthology.orchestra.run.gibba002.stdout: debug ms = 1 2021-02-10T11:25:11.721 INFO:teuthology.orchestra.run.gibba002.stdout: mds bal frag = True 2021-02-10T11:25:11.722 INFO:teuthology.orchestra.run.gibba002.stdout: mds bal fragment size max = 10000 2021-02-10T11:25:11.722 INFO:teuthology.orchestra.run.gibba002.stdout: mds bal merge size = 5 2021-02-10T11:25:11.722 INFO:teuthology.orchestra.run.gibba002.stdout: mds bal split bits = 3 2021-02-10T11:25:11.722 INFO:teuthology.orchestra.run.gibba002.stdout: mds bal split size = 100 2021-02-10T11:25:11.722 INFO:teuthology.orchestra.run.gibba002.stdout: mds debug frag = True 2021-02-10T11:25:11.722 INFO:teuthology.orchestra.run.gibba002.stdout: mds debug scatterstat = True 2021-02-10T11:25:11.722 INFO:teuthology.orchestra.run.gibba002.stdout: mds op complaint time = 180 2021-02-10T11:25:11.722 INFO:teuthology.orchestra.run.gibba002.stdout: mds verify scatter = True 2021-02-10T11:25:11.722 INFO:teuthology.orchestra.run.gibba002.stdout: osd op complaint time = 180 2021-02-10T11:25:11.722 INFO:teuthology.orchestra.run.gibba002.stdout:[client.alice] 2021-02-10T11:25:11.723 INFO:teuthology.orchestra.run.gibba013.stdout:[global] 2021-02-10T11:25:11.723 INFO:teuthology.orchestra.run.gibba013.stdout: chdir = "" 2021-02-10T11:25:11.723 INFO:teuthology.orchestra.run.gibba013.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2021-02-10T11:25:11.723 INFO:teuthology.orchestra.run.gibba013.stdout: auth supported = cephx 2021-02-10T11:25:11.723 INFO:teuthology.orchestra.run.gibba013.stdout: 2021-02-10T11:25:11.724 INFO:teuthology.orchestra.run.gibba013.stdout: filestore xattr use omap = true 2021-02-10T11:25:11.724 INFO:teuthology.orchestra.run.gibba013.stdout: 2021-02-10T11:25:11.724 INFO:teuthology.orchestra.run.gibba013.stdout: mon clock drift allowed = 1.000 2021-02-10T11:25:11.724 INFO:teuthology.orchestra.run.gibba013.stdout: 2021-02-10T11:25:11.724 INFO:teuthology.orchestra.run.gibba013.stdout: osd crush chooseleaf type = 0 2021-02-10T11:25:11.724 INFO:teuthology.orchestra.run.gibba013.stdout: auth debug = true 2021-02-10T11:25:11.724 INFO:teuthology.orchestra.run.gibba013.stdout: 2021-02-10T11:25:11.724 INFO:teuthology.orchestra.run.gibba013.stdout: ms die on old message = true 2021-02-10T11:25:11.724 INFO:teuthology.orchestra.run.gibba013.stdout: ms die on bug = true 2021-02-10T11:25:11.724 INFO:teuthology.orchestra.run.gibba013.stdout: 2021-02-10T11:25:11.724 INFO:teuthology.orchestra.run.gibba013.stdout: mon max pg per osd = 10000 # >= luminous 2021-02-10T11:25:11.724 INFO:teuthology.orchestra.run.gibba013.stdout: mon pg warn max object skew = 0 2021-02-10T11:25:11.724 INFO:teuthology.orchestra.run.gibba013.stdout: 2021-02-10T11:25:11.725 INFO:teuthology.orchestra.run.gibba013.stdout: osd pool default size = 2 2021-02-10T11:25:11.725 INFO:teuthology.orchestra.run.gibba013.stdout: 2021-02-10T11:25:11.725 INFO:teuthology.orchestra.run.gibba013.stdout: mon osd allow primary affinity = true 2021-02-10T11:25:11.725 INFO:teuthology.orchestra.run.gibba013.stdout: mon osd allow pg remap = true 2021-02-10T11:25:11.725 INFO:teuthology.orchestra.run.gibba013.stdout: mon warn on legacy crush tunables = false 2021-02-10T11:25:11.725 INFO:teuthology.orchestra.run.gibba013.stdout: mon warn on crush straw calc version zero = false 2021-02-10T11:25:11.725 INFO:teuthology.orchestra.run.gibba013.stdout: mon warn on no sortbitwise = false 2021-02-10T11:25:11.725 INFO:teuthology.orchestra.run.gibba013.stdout: mon warn on osd down out interval zero = false 2021-02-10T11:25:11.725 INFO:teuthology.orchestra.run.gibba013.stdout: mon warn on too few osds = false 2021-02-10T11:25:11.725 INFO:teuthology.orchestra.run.gibba013.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2021-02-10T11:25:11.725 INFO:teuthology.orchestra.run.gibba013.stdout: mon_warn_on_pool_no_redundancy = false 2021-02-10T11:25:11.725 INFO:teuthology.orchestra.run.gibba013.stdout: 2021-02-10T11:25:11.725 INFO:teuthology.orchestra.run.gibba013.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2021-02-10T11:25:11.725 INFO:teuthology.orchestra.run.gibba013.stdout: 2021-02-10T11:25:11.726 INFO:teuthology.orchestra.run.gibba013.stdout: osd default data pool replay window = 5 2021-02-10T11:25:11.726 INFO:teuthology.orchestra.run.gibba013.stdout: 2021-02-10T11:25:11.726 INFO:teuthology.orchestra.run.gibba013.stdout: mon allow pool delete = true 2021-02-10T11:25:11.726 INFO:teuthology.orchestra.run.gibba013.stdout: 2021-02-10T11:25:11.726 INFO:teuthology.orchestra.run.gibba013.stdout: mon cluster log file level = debug 2021-02-10T11:25:11.726 INFO:teuthology.orchestra.run.gibba013.stdout: debug asserts on shutdown = true 2021-02-10T11:25:11.726 INFO:teuthology.orchestra.run.gibba013.stdout: mon health detail to clog = false 2021-02-10T11:25:11.726 INFO:teuthology.orchestra.run.gibba013.stdout: 2021-02-10T11:25:11.726 INFO:teuthology.orchestra.run.gibba013.stdout: # we see this fail in qa on *nautilus*; bump up retries 2021-02-10T11:25:11.726 INFO:teuthology.orchestra.run.gibba013.stdout: mon_client_directed_command_retry = 4 2021-02-10T11:25:11.726 INFO:teuthology.orchestra.run.gibba013.stdout: mon host = "172.21.2.102,172.21.2.113,[v2:172.21.2.113:3301,v1:172.21.2.113:6790]" 2021-02-10T11:25:11.727 INFO:teuthology.orchestra.run.gibba013.stdout: fsid = 68db30ea-dbce-4aef-a6df-f01c6fea0513 2021-02-10T11:25:11.727 INFO:teuthology.orchestra.run.gibba013.stdout: 2021-02-10T11:25:11.727 INFO:teuthology.orchestra.run.gibba013.stdout:[osd] 2021-02-10T11:25:11.727 INFO:teuthology.orchestra.run.gibba013.stdout: osd journal size = 100 2021-02-10T11:25:11.727 INFO:teuthology.orchestra.run.gibba013.stdout: 2021-02-10T11:25:11.727 INFO:teuthology.orchestra.run.gibba013.stdout: osd scrub load threshold = 5.0 2021-02-10T11:25:11.727 INFO:teuthology.orchestra.run.gibba013.stdout: osd scrub max interval = 600 2021-02-10T11:25:11.727 INFO:teuthology.orchestra.run.gibba013.stdout: 2021-02-10T11:25:11.727 INFO:teuthology.orchestra.run.gibba013.stdout: osd recover clone overlap = true 2021-02-10T11:25:11.727 INFO:teuthology.orchestra.run.gibba013.stdout: osd recovery max chunk = 1048576 2021-02-10T11:25:11.727 INFO:teuthology.orchestra.run.gibba013.stdout: 2021-02-10T11:25:11.727 INFO:teuthology.orchestra.run.gibba013.stdout: osd debug shutdown = true 2021-02-10T11:25:11.727 INFO:teuthology.orchestra.run.gibba013.stdout: osd debug op order = true 2021-02-10T11:25:11.728 INFO:teuthology.orchestra.run.gibba013.stdout: osd debug verify stray on activate = true 2021-02-10T11:25:11.728 INFO:teuthology.orchestra.run.gibba013.stdout: 2021-02-10T11:25:11.728 INFO:teuthology.orchestra.run.gibba013.stdout: osd open classes on start = true 2021-02-10T11:25:11.728 INFO:teuthology.orchestra.run.gibba013.stdout: osd debug pg log writeout = true 2021-02-10T11:25:11.728 INFO:teuthology.orchestra.run.gibba013.stdout: 2021-02-10T11:25:11.728 INFO:teuthology.orchestra.run.gibba013.stdout: osd deep scrub update digest min age = 30 2021-02-10T11:25:11.728 INFO:teuthology.orchestra.run.gibba013.stdout: 2021-02-10T11:25:11.728 INFO:teuthology.orchestra.run.gibba013.stdout: osd map max advance = 10 2021-02-10T11:25:11.728 INFO:teuthology.orchestra.run.gibba013.stdout: 2021-02-10T11:25:11.728 INFO:teuthology.orchestra.run.gibba013.stdout: journal zero on create = true 2021-02-10T11:25:11.728 INFO:teuthology.orchestra.run.gibba013.stdout: 2021-02-10T11:25:11.728 INFO:teuthology.orchestra.run.gibba013.stdout: filestore ondisk finisher threads = 3 2021-02-10T11:25:11.728 INFO:teuthology.orchestra.run.gibba013.stdout: filestore apply finisher threads = 3 2021-02-10T11:25:11.729 INFO:teuthology.orchestra.run.gibba013.stdout: 2021-02-10T11:25:11.729 INFO:teuthology.orchestra.run.gibba013.stdout: bdev debug aio = true 2021-02-10T11:25:11.729 INFO:teuthology.orchestra.run.gibba013.stdout: osd debug misdirected ops = true 2021-02-10T11:25:11.729 INFO:teuthology.orchestra.run.gibba013.stdout: bdev async discard = True 2021-02-10T11:25:11.729 INFO:teuthology.orchestra.run.gibba013.stdout: bdev enable discard = True 2021-02-10T11:25:11.729 INFO:teuthology.orchestra.run.gibba013.stdout: bluestore allocator = bitmap 2021-02-10T11:25:11.730 INFO:teuthology.orchestra.run.gibba013.stdout: bluestore block size = 96636764160 2021-02-10T11:25:11.730 INFO:teuthology.orchestra.run.gibba013.stdout: bluestore fsck on mount = True 2021-02-10T11:25:11.730 INFO:teuthology.orchestra.run.gibba013.stdout: debug bluefs = 20 2021-02-10T11:25:11.730 INFO:teuthology.orchestra.run.gibba013.stdout: debug bluestore = 20 2021-02-10T11:25:11.730 INFO:teuthology.orchestra.run.gibba013.stdout: debug filestore = 20 2021-02-10T11:25:11.730 INFO:teuthology.orchestra.run.gibba013.stdout: debug journal = 20 2021-02-10T11:25:11.730 INFO:teuthology.orchestra.run.gibba013.stdout: debug ms = 20 2021-02-10T11:25:11.730 INFO:teuthology.orchestra.run.gibba013.stdout: debug osd = 25 2021-02-10T11:25:11.730 INFO:teuthology.orchestra.run.gibba013.stdout: debug rocksdb = 10 2021-02-10T11:25:11.730 INFO:teuthology.orchestra.run.gibba013.stdout: mon osd backfillfull_ratio = 0.85 2021-02-10T11:25:11.730 INFO:teuthology.orchestra.run.gibba013.stdout: mon osd full ratio = 0.9 2021-02-10T11:25:11.730 INFO:teuthology.orchestra.run.gibba013.stdout: mon osd nearfull ratio = 0.8 2021-02-10T11:25:11.730 INFO:teuthology.orchestra.run.gibba013.stdout: osd failsafe full ratio = 0.95 2021-02-10T11:25:11.731 INFO:teuthology.orchestra.run.gibba013.stdout: osd objectstore = bluestore 2021-02-10T11:25:11.731 INFO:teuthology.orchestra.run.gibba013.stdout: osd op complaint time = 180 2021-02-10T11:25:11.731 INFO:teuthology.orchestra.run.gibba013.stdout: 2021-02-10T11:25:11.731 INFO:teuthology.orchestra.run.gibba013.stdout:[mgr] 2021-02-10T11:25:11.731 INFO:teuthology.orchestra.run.gibba013.stdout: debug ms = 1 2021-02-10T11:25:11.731 INFO:teuthology.orchestra.run.gibba013.stdout: debug mgr = 20 2021-02-10T11:25:11.731 INFO:teuthology.orchestra.run.gibba013.stdout: debug mon = 20 2021-02-10T11:25:11.731 INFO:teuthology.orchestra.run.gibba013.stdout: debug auth = 20 2021-02-10T11:25:11.731 INFO:teuthology.orchestra.run.gibba013.stdout: mon reweight min pgs per osd = 4 2021-02-10T11:25:11.731 INFO:teuthology.orchestra.run.gibba013.stdout: mon reweight min bytes per osd = 10 2021-02-10T11:25:11.731 INFO:teuthology.orchestra.run.gibba013.stdout: debug client = 10 2021-02-10T11:25:11.731 INFO:teuthology.orchestra.run.gibba013.stdout: 2021-02-10T11:25:11.731 INFO:teuthology.orchestra.run.gibba013.stdout:[mon] 2021-02-10T11:25:11.731 INFO:teuthology.orchestra.run.gibba013.stdout: debug ms = 1 2021-02-10T11:25:11.732 INFO:teuthology.orchestra.run.gibba013.stdout: debug mon = 20 2021-02-10T11:25:11.732 INFO:teuthology.orchestra.run.gibba013.stdout: debug paxos = 20 2021-02-10T11:25:11.732 INFO:teuthology.orchestra.run.gibba013.stdout: debug auth = 20 2021-02-10T11:25:11.732 INFO:teuthology.orchestra.run.gibba013.stdout: mon data avail warn = 5 2021-02-10T11:25:11.732 INFO:teuthology.orchestra.run.gibba013.stdout: mon mgr mkfs grace = 240 2021-02-10T11:25:11.732 INFO:teuthology.orchestra.run.gibba013.stdout: mon reweight min pgs per osd = 4 2021-02-10T11:25:11.732 INFO:teuthology.orchestra.run.gibba013.stdout: mon osd reporter subtree level = osd 2021-02-10T11:25:11.732 INFO:teuthology.orchestra.run.gibba013.stdout: mon osd prime pg temp = true 2021-02-10T11:25:11.732 INFO:teuthology.orchestra.run.gibba013.stdout: mon reweight min bytes per osd = 10 2021-02-10T11:25:11.732 INFO:teuthology.orchestra.run.gibba013.stdout: mon op complaint time = 120 2021-02-10T11:25:11.732 INFO:teuthology.orchestra.run.gibba013.stdout: 2021-02-10T11:25:11.732 INFO:teuthology.orchestra.run.gibba013.stdout:[client] 2021-02-10T11:25:11.732 INFO:teuthology.orchestra.run.gibba013.stdout: rgw cache enabled = true 2021-02-10T11:25:11.733 INFO:teuthology.orchestra.run.gibba013.stdout: rgw enable ops log = true 2021-02-10T11:25:11.733 INFO:teuthology.orchestra.run.gibba013.stdout: rgw enable usage log = true 2021-02-10T11:25:11.733 INFO:teuthology.orchestra.run.gibba013.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2021-02-10T11:25:11.733 INFO:teuthology.orchestra.run.gibba013.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2021-02-10T11:25:11.733 INFO:teuthology.orchestra.run.gibba013.stdout: client mount timeout = 600 2021-02-10T11:25:11.733 INFO:teuthology.orchestra.run.gibba013.stdout: debug client = 20 2021-02-10T11:25:11.733 INFO:teuthology.orchestra.run.gibba013.stdout: debug ms = 1 2021-02-10T11:25:11.733 INFO:teuthology.orchestra.run.gibba013.stdout: pid file = "" 2021-02-10T11:25:11.733 INFO:teuthology.orchestra.run.gibba013.stdout:[mon.a] 2021-02-10T11:25:11.733 INFO:teuthology.orchestra.run.gibba013.stdout:[mon.b] 2021-02-10T11:25:11.733 INFO:teuthology.orchestra.run.gibba013.stdout:[mon.c] 2021-02-10T11:25:11.733 INFO:teuthology.orchestra.run.gibba013.stdout:[mds.a] 2021-02-10T11:25:11.733 INFO:teuthology.orchestra.run.gibba013.stdout:[mds.b] 2021-02-10T11:25:11.733 INFO:teuthology.orchestra.run.gibba013.stdout:[mds] 2021-02-10T11:25:11.734 INFO:teuthology.orchestra.run.gibba013.stdout: debug mds = 20 2021-02-10T11:25:11.734 INFO:teuthology.orchestra.run.gibba013.stdout: debug ms = 1 2021-02-10T11:25:11.734 INFO:teuthology.orchestra.run.gibba013.stdout: mds bal frag = True 2021-02-10T11:25:11.734 INFO:teuthology.orchestra.run.gibba013.stdout: mds bal fragment size max = 10000 2021-02-10T11:25:11.734 INFO:teuthology.orchestra.run.gibba013.stdout: mds bal merge size = 5 2021-02-10T11:25:11.734 INFO:teuthology.orchestra.run.gibba013.stdout: mds bal split bits = 3 2021-02-10T11:25:11.734 INFO:teuthology.orchestra.run.gibba013.stdout: mds bal split size = 100 2021-02-10T11:25:11.734 INFO:teuthology.orchestra.run.gibba013.stdout: mds debug frag = True 2021-02-10T11:25:11.734 INFO:teuthology.orchestra.run.gibba013.stdout: mds debug scatterstat = True 2021-02-10T11:25:11.734 INFO:teuthology.orchestra.run.gibba013.stdout: mds op complaint time = 180 2021-02-10T11:25:11.734 INFO:teuthology.orchestra.run.gibba013.stdout: mds verify scatter = True 2021-02-10T11:25:11.734 INFO:teuthology.orchestra.run.gibba013.stdout: osd op complaint time = 180 2021-02-10T11:25:11.734 INFO:teuthology.orchestra.run.gibba013.stdout:[client.alice] 2021-02-10T11:25:11.735 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config rm mon mon_allow_pool_delete 2021-02-10T11:25:12.165 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:12.150 7fceeb9ef700 1 Processor -- start 2021-02-10T11:25:12.165 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:12.150 7fceeb9ef700 1 -- start start 2021-02-10T11:25:12.165 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:12.151 7fceeb9ef700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fcee40d1870 0x7fcee41b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:12.165 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:12.151 7fceeb9ef700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fcee41b0a10 0x7fcee40cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:12.165 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:12.151 7fceeb9ef700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcee40f0b10 con 0x7fcee40d1c70 2021-02-10T11:25:12.165 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:12.151 7fceeb9ef700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcee40e5ed0 con 0x7fcee40d1870 2021-02-10T11:25:12.165 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:12.151 7fceeb9ef700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcee40f0650 con 0x7fcee41b0a10 2021-02-10T11:25:12.165 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:12.151 7fcee980b700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fcee40d1870 0x7fcee41b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:12.166 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:12.151 7fcee980b700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fcee40d1870 0x7fcee41b03f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:57636/0 (socket says 172.21.2.102:57636) 2021-02-10T11:25:12.166 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:12.151 7fcee980b700 1 -- 172.21.2.102:0/3745648183 learned_addr learned my addr 172.21.2.102:0/3745648183 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:25:12.166 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:12.151 7fcee980b700 1 -- 172.21.2.102:0/3745648183 >> v2:172.21.2.113:3300/0 conn(0x7fcee41b0a10 msgr2=0x7fcee40cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:25:12.166 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:12.151 7fcee980b700 1 --2- 172.21.2.102:0/3745648183 >> v2:172.21.2.113:3300/0 conn(0x7fcee41b0a10 0x7fcee40cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:12.166 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:12.151 7fcee980b700 1 -- 172.21.2.102:0/3745648183 >> v1:172.21.2.113:6789/0 conn(0x7fcee40d1c70 legacy=0x7fcee40cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:25:12.166 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:12.151 7fcee980b700 1 -- 172.21.2.102:0/3745648183 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcee40f0840 con 0x7fcee40d1870 2021-02-10T11:25:12.166 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:12.151 7fcee980b700 1 --2- 172.21.2.102:0/3745648183 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcee40d1870 0x7fcee41b03f0 secure :-1 s=READY pgs=1433 cs=0 l=1 rev1=1 rx=0x7fcee0006700 tx=0x7fcee0009b80).ready entity=mon.0 client_cookie=b3a3b1ad313fa876 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:12.166 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:12.152 7fcee8809700 1 -- 172.21.2.102:0/3745648183 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcee000c0c0 con 0x7fcee40d1870 2021-02-10T11:25:12.166 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:12.152 7fcee8809700 1 -- 172.21.2.102:0/3745648183 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcee000c440 con 0x7fcee40d1870 2021-02-10T11:25:12.166 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:12.152 7fcee8809700 1 -- 172.21.2.102:0/3745648183 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcee000c960 con 0x7fcee40d1870 2021-02-10T11:25:12.166 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:12.152 7fceeb9ef700 1 -- 172.21.2.102:0/3745648183 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcee40d1870 msgr2=0x7fcee41b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:12.167 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:12.152 7fceeb9ef700 1 --2- 172.21.2.102:0/3745648183 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcee40d1870 0x7fcee41b03f0 secure :-1 s=READY pgs=1433 cs=0 l=1 rev1=1 rx=0x7fcee0006700 tx=0x7fcee0009b80).stop 2021-02-10T11:25:12.167 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:12.152 7fceeb9ef700 1 -- 172.21.2.102:0/3745648183 shutdown_connections 2021-02-10T11:25:12.167 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:12.152 7fceeb9ef700 1 --2- 172.21.2.102:0/3745648183 >> v2:172.21.2.113:3300/0 conn(0x7fcee41b0a10 0x7fcee40cf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:12.167 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:12.152 7fceeb9ef700 1 --2- 172.21.2.102:0/3745648183 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcee40d1870 0x7fcee41b03f0 unknown :-1 s=CLOSED pgs=1433 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:12.167 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:12.152 7fceeb9ef700 1 -- 172.21.2.102:0/3745648183 >> 172.21.2.102:0/3745648183 conn(0x7fcee40ce450 msgr2=0x7fcee40d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:25:12.167 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:12.153 7fceeb9ef700 1 -- 172.21.2.102:0/3745648183 shutdown_connections 2021-02-10T11:25:12.167 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:12.153 7fceeb9ef700 1 -- 172.21.2.102:0/3745648183 wait complete. 2021-02-10T11:25:12.169 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:12.167 7fceeb9ef700 1 Processor -- start 2021-02-10T11:25:12.169 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:12.167 7fceeb9ef700 1 -- start start 2021-02-10T11:25:12.169 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:12.168 7fceeb9ef700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcee40a2fb0 0x7fcee40a33b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:12.169 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:12.168 7fceeb9ef700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcee40a39d0 0x7fcee40a4860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:12.169 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:12.168 7fceeb9ef700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fcee40a3dd0 0x7fcee40a4e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:12.169 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:12.168 7fceeb9ef700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcee40a2620 con 0x7fcee40a39d0 2021-02-10T11:25:12.169 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:12.168 7fceeb9ef700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcee40a2840 con 0x7fcee40a3dd0 2021-02-10T11:25:12.169 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:12.168 7fceeb9ef700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcee40a2b80 con 0x7fcee40a2fb0 2021-02-10T11:25:12.169 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:12.168 7fcee900a700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcee40a39d0 0x7fcee40a4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:12.169 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:12.168 7fcee900a700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcee40a39d0 0x7fcee40a4860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:57642/0 (socket says 172.21.2.102:57642) 2021-02-10T11:25:12.170 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:12.168 7fcee900a700 1 -- 172.21.2.102:0/4033274538 learned_addr learned my addr 172.21.2.102:0/4033274538 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:25:12.170 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:12.168 7fcee900a700 1 -- 172.21.2.102:0/4033274538 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcee40a2fb0 msgr2=0x7fcee40a33b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:25:12.170 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:12.168 7fcee900a700 1 --2- 172.21.2.102:0/4033274538 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcee40a2fb0 0x7fcee40a33b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:12.170 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:12.168 7fcee900a700 1 -- 172.21.2.102:0/4033274538 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fcee40a3dd0 msgr2=0x7fcee40a4e80 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:25:12.170 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:12.168 7fcee900a700 1 --2- 172.21.2.102:0/4033274538 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fcee40a3dd0 0x7fcee40a4e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:12.170 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:12.168 7fcee900a700 1 -- 172.21.2.102:0/4033274538 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcee40a54a0 con 0x7fcee40a39d0 2021-02-10T11:25:12.170 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:12.169 7fcee900a700 1 --2- 172.21.2.102:0/4033274538 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcee40a39d0 0x7fcee40a4860 secure :-1 s=READY pgs=1434 cs=0 l=1 rev1=1 rx=0x7fced8006f00 tx=0x7fced800b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:12.170 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:12.169 7fced67fc700 1 -- 172.21.2.102:0/4033274538 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fced800bd30 con 0x7fcee40a39d0 2021-02-10T11:25:12.170 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:12.169 7fced67fc700 1 -- 172.21.2.102:0/4033274538 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fced8000c90 con 0x7fcee40a39d0 2021-02-10T11:25:12.170 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:12.169 7fceeb9ef700 1 -- 172.21.2.102:0/4033274538 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcee40a54a0 con 0x7fcee40a39d0 2021-02-10T11:25:12.170 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:12.169 7fced67fc700 1 -- 172.21.2.102:0/4033274538 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fced80022a0 con 0x7fcee40a39d0 2021-02-10T11:25:12.170 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:12.169 7fceeb9ef700 1 -- 172.21.2.102:0/4033274538 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcee40a54a0 con 0x7fcee40a39d0 2021-02-10T11:25:12.171 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:12.169 7fced67fc700 1 -- 172.21.2.102:0/4033274538 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fced8005310 con 0x7fcee40a39d0 2021-02-10T11:25:12.171 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:12.170 7fced67fc700 1 --2- 172.21.2.102:0/4033274538 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcecc018730 0x7fcecc01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:12.171 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:12.170 7fced67fc700 1 -- 172.21.2.102:0/4033274538 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fced8032600 con 0x7fcee40a39d0 2021-02-10T11:25:12.171 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:12.170 7fcee980b700 1 --2- 172.21.2.102:0/4033274538 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcecc018730 0x7fcecc01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:12.171 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:12.170 7fcee980b700 1 --2- 172.21.2.102:0/4033274538 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcecc018730 0x7fcecc01ac10 crc :-1 s=READY pgs=658 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:12.198 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:12.200 7fceeb9ef700 1 -- 172.21.2.102:0/4033274538 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcee40a4330 con 0x7fcee40a39d0 2021-02-10T11:25:12.201 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:12.203 7fced67fc700 1 -- 172.21.2.102:0/4033274538 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fced8006b90 con 0x7fcee40a39d0 2021-02-10T11:25:12.352 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:12.354 7fceeb9ef700 1 -- 172.21.2.102:0/4033274538 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fcee40a4330 con 0x7fcee40a39d0 2021-02-10T11:25:12.356 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:12.358 7fced67fc700 1 -- 172.21.2.102:0/4033274538 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v11) v1 ==== 104+0+0 (secure 0 0 0) 0x7fced8005310 con 0x7fcee40a39d0 2021-02-10T11:25:12.361 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:12.363 7fceeb9ef700 1 -- 172.21.2.102:0/4033274538 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcecc018730 msgr2=0x7fcecc01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:12.361 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:12.363 7fceeb9ef700 1 --2- 172.21.2.102:0/4033274538 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcecc018730 0x7fcecc01ac10 crc :-1 s=READY pgs=658 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:12.361 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:12.363 7fceeb9ef700 1 -- 172.21.2.102:0/4033274538 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcee40a39d0 msgr2=0x7fcee40a4860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:12.361 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:12.363 7fceeb9ef700 1 --2- 172.21.2.102:0/4033274538 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcee40a39d0 0x7fcee40a4860 secure :-1 s=READY pgs=1434 cs=0 l=1 rev1=1 rx=0x7fced8006f00 tx=0x7fced800b940).stop 2021-02-10T11:25:12.362 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:12.363 7fceeb9ef700 1 -- 172.21.2.102:0/4033274538 shutdown_connections 2021-02-10T11:25:12.362 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:12.363 7fceeb9ef700 1 --2- 172.21.2.102:0/4033274538 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcecc018730 0x7fcecc01ac10 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:12.362 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:12.363 7fceeb9ef700 1 --2- 172.21.2.102:0/4033274538 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fcee40a3dd0 0x7fcee40a4e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:12.362 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:12.363 7fceeb9ef700 1 --2- 172.21.2.102:0/4033274538 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcee40a39d0 0x7fcee40a4860 unknown :-1 s=CLOSED pgs=1434 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:12.362 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:12.363 7fceeb9ef700 1 --2- 172.21.2.102:0/4033274538 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcee40a2fb0 0x7fcee40a33b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:12.362 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:12.363 7fceeb9ef700 1 -- 172.21.2.102:0/4033274538 >> 172.21.2.102:0/4033274538 conn(0x7fcee408f030 msgr2=0x7fcee40b5e80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:25:12.362 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:12.363 7fceeb9ef700 1 -- 172.21.2.102:0/4033274538 shutdown_connections 2021-02-10T11:25:12.362 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:12.363 7fceeb9ef700 1 -- 172.21.2.102:0/4033274538 wait complete. 2021-02-10T11:25:12.369 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config rm client.alice debug_client 2021-02-10T11:25:12.801 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:12.785 7f75c37a1700 1 Processor -- start 2021-02-10T11:25:12.801 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:12.785 7f75c37a1700 1 -- start start 2021-02-10T11:25:12.801 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:12.785 7f75c37a1700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f75bc0d1870 0x7f75bc1b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:12.801 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:12.785 7f75c37a1700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f75bc1b0a10 0x7f75bc0cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:12.801 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:12.785 7f75c37a1700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75bc0f0b10 con 0x7f75bc0d1c70 2021-02-10T11:25:12.801 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:12.785 7f75c37a1700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75bc0fe230 con 0x7f75bc1b0a10 2021-02-10T11:25:12.801 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:12.785 7f75c37a1700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75bc0e5ed0 con 0x7f75bc0d1870 2021-02-10T11:25:12.801 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:12.785 7f75c1dbe700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f75bc0d1c70 0x7f75bc0cf950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:49872/0 (socket says 172.21.2.102:49872) 2021-02-10T11:25:12.801 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:12.785 7f75c1dbe700 1 -- 172.21.2.102:0/96976406 learned_addr learned my addr 172.21.2.102:0/96976406 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:25:12.801 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:12.786 7f75affff700 1 -- 172.21.2.102:0/96976406 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 721152664 0 0) 0x7f75b4001500 con 0x7f75bc0d1c70 2021-02-10T11:25:12.802 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:12.786 7f75affff700 1 -- 172.21.2.102:0/96976406 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75a8002fe0 con 0x7f75bc0d1c70 2021-02-10T11:25:12.802 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:12.786 7f75c15bd700 1 --2- 172.21.2.102:0/96976406 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f75bc0d1870 0x7f75bc1b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:12.802 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:12.786 7f75c0dbc700 1 --2- 172.21.2.102:0/96976406 >> v2:172.21.2.113:3300/0 conn(0x7f75bc1b0a10 0x7f75bc0cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:12.802 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:12.786 7f75affff700 1 -- 172.21.2.102:0/96976406 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1889524215 0 0) 0x7f75b4001460 con 0x7f75bc0d1c70 2021-02-10T11:25:12.802 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:12.786 7f75affff700 1 -- 172.21.2.102:0/96976406 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f75bc0d1870 msgr2=0x7f75bc1b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:25:12.802 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:12.786 7f75affff700 1 --2- 172.21.2.102:0/96976406 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f75bc0d1870 0x7f75bc1b03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:12.802 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:12.786 7f75affff700 1 -- 172.21.2.102:0/96976406 >> v2:172.21.2.113:3300/0 conn(0x7f75bc1b0a10 msgr2=0x7f75bc0cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:25:12.802 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:12.786 7f75affff700 1 --2- 172.21.2.102:0/96976406 >> v2:172.21.2.113:3300/0 conn(0x7f75bc1b0a10 0x7f75bc0cf330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:12.802 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:12.786 7f75affff700 1 -- 172.21.2.102:0/96976406 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75bc0f0650 con 0x7f75bc0d1c70 2021-02-10T11:25:12.802 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:12.786 7f75affff700 1 -- 172.21.2.102:0/96976406 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7f75b4001980 con 0x7f75bc0d1c70 2021-02-10T11:25:12.802 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:12.786 7f75affff700 1 -- 172.21.2.102:0/96976406 >> v1:172.21.2.102:6789/0 conn(0x7f75bc0d1c70 legacy=0x7f75bc0cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:12.802 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:12.786 7f75affff700 1 --2- 172.21.2.102:0/96976406 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f75a8003b90 0x7f75a8006040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:12.803 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:12.786 7f75affff700 1 --2- 172.21.2.102:0/96976406 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f75a80068e0 0x7f75a8008de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:12.803 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:12.786 7f75affff700 1 --2- 172.21.2.102:0/96976406 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f75a80096e0 0x7f75a800bbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:12.803 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:12.786 7f75affff700 1 -- 172.21.2.102:0/96976406 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75a800c420 con 0x7f75a80068e0 2021-02-10T11:25:12.803 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:12.786 7f75affff700 1 -- 172.21.2.102:0/96976406 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75a800c640 con 0x7f75a8003b90 2021-02-10T11:25:12.803 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:12.786 7f75affff700 1 -- 172.21.2.102:0/96976406 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75a800c860 con 0x7f75a80096e0 2021-02-10T11:25:12.803 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:12.786 7f75c15bd700 1 --2- 172.21.2.102:0/96976406 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f75bc0d1870 0x7f75bc1b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:25:12.803 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:12.786 7f75c0dbc700 1 --2- 172.21.2.102:0/96976406 >> v2:172.21.2.113:3300/0 conn(0x7f75bc1b0a10 0x7f75bc0cf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:25:12.803 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:12.786 7f75c0dbc700 1 --2- 172.21.2.102:0/96976406 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f75a80068e0 0x7f75a8008de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:12.803 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:12.786 7f75c0dbc700 1 -- 172.21.2.102:0/96976406 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f75a80096e0 msgr2=0x7f75a800bbd0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:25:12.803 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:12.787 7f75c0dbc700 1 --2- 172.21.2.102:0/96976406 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f75a80096e0 0x7f75a800bbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:12.803 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:12.787 7f75c0dbc700 1 -- 172.21.2.102:0/96976406 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f75a8003b90 msgr2=0x7f75a8006040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:25:12.803 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:12.787 7f75c0dbc700 1 --2- 172.21.2.102:0/96976406 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f75a8003b90 0x7f75a8006040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:12.804 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:12.787 7f75c0dbc700 1 -- 172.21.2.102:0/96976406 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75a800cb00 con 0x7f75a80068e0 2021-02-10T11:25:12.804 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:12.787 7f75c0dbc700 1 --2- 172.21.2.102:0/96976406 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f75a80068e0 0x7f75a8008de0 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 rx=0x7f75b80098a0 tx=0x7f75b8009c40).ready entity=mon.0 client_cookie=14e62c4fd3361637 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:12.804 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:12.787 7f75affff700 1 -- 172.21.2.102:0/96976406 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f75b80065d0 con 0x7f75a80068e0 2021-02-10T11:25:12.804 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:12.787 7f75affff700 1 -- 172.21.2.102:0/96976406 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75b8000cd0 con 0x7f75a80068e0 2021-02-10T11:25:12.804 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:12.787 7f75affff700 1 -- 172.21.2.102:0/96976406 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f75b8002220 con 0x7f75a80068e0 2021-02-10T11:25:12.804 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:12.787 7f75c37a1700 1 -- 172.21.2.102:0/96976406 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f75a80068e0 msgr2=0x7f75a8008de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:12.804 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:12.787 7f75c37a1700 1 --2- 172.21.2.102:0/96976406 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f75a80068e0 0x7f75a8008de0 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 rx=0x7f75b80098a0 tx=0x7f75b8009c40).stop 2021-02-10T11:25:12.804 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:12.787 7f75c1dbe700 1 -- 172.21.2.102:0/96976406 reap_dead start 2021-02-10T11:25:12.804 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:12.788 7f75c37a1700 1 -- 172.21.2.102:0/96976406 shutdown_connections 2021-02-10T11:25:12.804 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:12.788 7f75c37a1700 1 -- 172.21.2.102:0/96976406 >> 172.21.2.102:0/96976406 conn(0x7f75bc0ce450 msgr2=0x7f75bc0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:25:12.804 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:12.788 7f75c37a1700 1 -- 172.21.2.102:0/96976406 shutdown_connections 2021-02-10T11:25:12.804 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:12.788 7f75c37a1700 1 -- 172.21.2.102:0/96976406 wait complete. 2021-02-10T11:25:12.804 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:12.802 7f75c37a1700 1 Processor -- start 2021-02-10T11:25:12.805 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:12.802 7f75c37a1700 1 -- start start 2021-02-10T11:25:12.806 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:12.803 7f75c37a1700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f75bc0bc940 0x7f75bc092cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:12.806 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:12.803 7f75c37a1700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f75bc0932e0 0x7f75bc0936e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:12.806 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:12.803 7f75c37a1700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f75bc093d00 0x7f75bc0bd480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:12.806 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:12.803 7f75c37a1700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75bc092740 con 0x7f75bc0bc940 2021-02-10T11:25:12.806 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:12.803 7f75c37a1700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75bc0bcd40 con 0x7f75bc093d00 2021-02-10T11:25:12.806 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:12.803 7f75c37a1700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75bc0bcff0 con 0x7f75bc0932e0 2021-02-10T11:25:12.806 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:12.803 7f75c15bd700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f75bc0bc940 0x7f75bc092cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:12.806 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:12.803 7f75c15bd700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f75bc0bc940 0x7f75bc092cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:57658/0 (socket says 172.21.2.102:57658) 2021-02-10T11:25:12.806 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:12.803 7f75c15bd700 1 -- 172.21.2.102:0/2029142563 learned_addr learned my addr 172.21.2.102:0/2029142563 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:25:12.806 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:12.803 7f75c15bd700 1 -- 172.21.2.102:0/2029142563 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f75bc0932e0 msgr2=0x7f75bc0936e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:12.806 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:12.803 7f75c15bd700 1 --2- 172.21.2.102:0/2029142563 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f75bc0932e0 0x7f75bc0936e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:12.807 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:12.803 7f75c15bd700 1 -- 172.21.2.102:0/2029142563 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f75bc093d00 msgr2=0x7f75bc0bd480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:12.807 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:12.803 7f75c15bd700 1 --2- 172.21.2.102:0/2029142563 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f75bc093d00 0x7f75bc0bd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:12.807 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:12.803 7f75c15bd700 1 -- 172.21.2.102:0/2029142563 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75bc0bdaa0 con 0x7f75bc0bc940 2021-02-10T11:25:12.807 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:12.804 7f75c15bd700 1 --2- 172.21.2.102:0/2029142563 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f75bc0bc940 0x7f75bc092cc0 secure :-1 s=READY pgs=1437 cs=0 l=1 rev1=1 rx=0x7f75b0050610 tx=0x7f75b00509b0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:12.807 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:12.804 7f75adffb700 1 -- 172.21.2.102:0/2029142563 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f75b0050da0 con 0x7f75bc0bc940 2021-02-10T11:25:12.807 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:12.804 7f75adffb700 1 -- 172.21.2.102:0/2029142563 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75b0006030 con 0x7f75bc0bc940 2021-02-10T11:25:12.807 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:12.804 7f75adffb700 1 -- 172.21.2.102:0/2029142563 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f75b00052e0 con 0x7f75bc0bc940 2021-02-10T11:25:12.807 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:12.804 7f75c37a1700 1 -- 172.21.2.102:0/2029142563 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75bc0bdaa0 con 0x7f75bc0bc940 2021-02-10T11:25:12.807 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:12.804 7f75c37a1700 1 -- 172.21.2.102:0/2029142563 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75bc0be030 con 0x7f75bc0bc940 2021-02-10T11:25:12.807 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:12.804 7f75adffb700 1 -- 172.21.2.102:0/2029142563 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f75b0006ac0 con 0x7f75bc0bc940 2021-02-10T11:25:12.807 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:12.804 7f75adffb700 1 --2- 172.21.2.102:0/2029142563 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f759c0186c0 0x7f759c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:12.807 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:12.805 7f75adffb700 1 -- 172.21.2.102:0/2029142563 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f75b0076550 con 0x7f75bc0bc940 2021-02-10T11:25:12.807 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:12.805 7f75c0dbc700 1 --2- 172.21.2.102:0/2029142563 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f759c0186c0 0x7f759c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:12.808 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:12.805 7f75c0dbc700 1 --2- 172.21.2.102:0/2029142563 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f759c0186c0 0x7f759c01abc0 crc :-1 s=READY pgs=659 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:12.833 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:12.835 7f75c37a1700 1 -- 172.21.2.102:0/2029142563 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75bc0ebc40 con 0x7f75bc0bc940 2021-02-10T11:25:12.835 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:12.838 7f75adffb700 1 -- 172.21.2.102:0/2029142563 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f75b0074d40 con 0x7f75bc0bc940 2021-02-10T11:25:12.986 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:12.987 7f75c37a1700 1 -- 172.21.2.102:0/2029142563 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "config rm", "who": "client.alice", "name": "debug_client"} v 0) v1 -- 0x7f75bc0f0b10 con 0x7f75bc0bc940 2021-02-10T11:25:12.989 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:12.991 7f75adffb700 1 -- 172.21.2.102:0/2029142563 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "client.alice", "name": "debug_client"}]=0 v12) v1 ==== 104+0+0 (secure 0 0 0) 0x7f75b0006ac0 con 0x7f75bc0bc940 2021-02-10T11:25:12.994 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:12.996 7f75c37a1700 1 -- 172.21.2.102:0/2029142563 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f759c0186c0 msgr2=0x7f759c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:12.994 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:12.996 7f75c37a1700 1 --2- 172.21.2.102:0/2029142563 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f759c0186c0 0x7f759c01abc0 crc :-1 s=READY pgs=659 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:12.994 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:12.996 7f75c37a1700 1 -- 172.21.2.102:0/2029142563 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f75bc0bc940 msgr2=0x7f75bc092cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:12.994 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:12.996 7f75c37a1700 1 --2- 172.21.2.102:0/2029142563 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f75bc0bc940 0x7f75bc092cc0 secure :-1 s=READY pgs=1437 cs=0 l=1 rev1=1 rx=0x7f75b0050610 tx=0x7f75b00509b0).stop 2021-02-10T11:25:12.995 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:12.996 7f75c37a1700 1 -- 172.21.2.102:0/2029142563 shutdown_connections 2021-02-10T11:25:12.995 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:12.996 7f75c37a1700 1 --2- 172.21.2.102:0/2029142563 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f759c0186c0 0x7f759c01abc0 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:12.995 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:12.996 7f75c37a1700 1 --2- 172.21.2.102:0/2029142563 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f75bc093d00 0x7f75bc0bd480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:12.995 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:12.996 7f75c37a1700 1 --2- 172.21.2.102:0/2029142563 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f75bc0932e0 0x7f75bc0936e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:12.995 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:12.996 7f75c37a1700 1 --2- 172.21.2.102:0/2029142563 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f75bc0bc940 0x7f75bc092cc0 unknown :-1 s=CLOSED pgs=1437 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:12.995 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:12.996 7f75c37a1700 1 -- 172.21.2.102:0/2029142563 >> 172.21.2.102:0/2029142563 conn(0x7f75bc08fa10 msgr2=0x7f75bc0c2b00 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:25:12.995 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:12.996 7f75c37a1700 1 -- 172.21.2.102:0/2029142563 shutdown_connections 2021-02-10T11:25:12.995 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:12.997 7f75c37a1700 1 -- 172.21.2.102:0/2029142563 wait complete. 2021-02-10T11:25:13.003 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config rm client.alice debug_objecter 2021-02-10T11:25:13.434 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:13.419 7f9faeb87700 1 Processor -- start 2021-02-10T11:25:13.434 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:13.419 7f9faeb87700 1 -- start start 2021-02-10T11:25:13.434 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:13.419 7f9faeb87700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9fa80d0340 0x7f9fa80ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:13.434 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:13.419 7f9faeb87700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fa80f0b10 con 0x7f9fa80d1c70 2021-02-10T11:25:13.434 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:13.419 7f9faeb87700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fa81b0be0 con 0x7f9fa80d1870 2021-02-10T11:25:13.434 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:13.419 7f9faeb87700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9fa80e5ed0 con 0x7f9fa80d0340 2021-02-10T11:25:13.434 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:13.419 7f9fad1a4700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f9fa80d1c70 0x7f9fa80efc60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:49892/0 (socket says 172.21.2.102:49892) 2021-02-10T11:25:13.434 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:13.419 7f9fad1a4700 1 -- 172.21.2.102:0/605179902 learned_addr learned my addr 172.21.2.102:0/605179902 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:25:13.434 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:13.420 7f9f9f7fe700 1 -- 172.21.2.102:0/605179902 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1872885602 0 0) 0x7f9fa0000d30 con 0x7f9fa80d1c70 2021-02-10T11:25:13.434 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:13.420 7f9f9f7fe700 1 -- 172.21.2.102:0/605179902 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f90002fe0 con 0x7f9fa80d1c70 2021-02-10T11:25:13.435 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:13.420 7f9f9ffff700 1 --2- 172.21.2.102:0/605179902 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9fa80d0340 0x7f9fa80ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:13.435 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:13.420 7f9f9f7fe700 1 -- 172.21.2.102:0/605179902 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2081201319 0 0) 0x7f9fa0001460 con 0x7f9fa80d1c70 2021-02-10T11:25:13.435 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:13.420 7f9f9f7fe700 1 -- 172.21.2.102:0/605179902 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9fa80d0340 msgr2=0x7f9fa80ef640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:25:13.435 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:13.420 7f9f9f7fe700 1 --2- 172.21.2.102:0/605179902 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9fa80d0340 0x7f9fa80ef640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:13.435 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:13.420 7f9f9f7fe700 1 -- 172.21.2.102:0/605179902 >> v1:172.21.2.113:6789/0 conn(0x7f9fa80d1870 legacy=0x7f9fa81b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:25:13.435 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:13.420 7f9f9f7fe700 1 -- 172.21.2.102:0/605179902 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fa80d2fe0 con 0x7f9fa80d1c70 2021-02-10T11:25:13.435 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:13.420 7f9f9f7fe700 1 -- 172.21.2.102:0/605179902 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7f9fa0001980 con 0x7f9fa80d1c70 2021-02-10T11:25:13.435 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:13.420 7f9f9f7fe700 1 -- 172.21.2.102:0/605179902 >> v1:172.21.2.102:6789/0 conn(0x7f9fa80d1c70 legacy=0x7f9fa80efc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:13.435 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:13.420 7f9f9f7fe700 1 --2- 172.21.2.102:0/605179902 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f9f90003a60 0x7f9f90005ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:13.435 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:13.420 7f9f9f7fe700 1 --2- 172.21.2.102:0/605179902 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9f90006770 0x7f9f90008c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:13.435 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:13.420 7f9f9f7fe700 1 --2- 172.21.2.102:0/605179902 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9f90009500 0x7f9f9000ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:13.435 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:13.420 7f9f9f7fe700 1 -- 172.21.2.102:0/605179902 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f9000c2d0 con 0x7f9f90009500 2021-02-10T11:25:13.435 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:13.420 7f9f9f7fe700 1 -- 172.21.2.102:0/605179902 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f9000c4f0 con 0x7f9f90003a60 2021-02-10T11:25:13.436 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:13.420 7f9f9f7fe700 1 -- 172.21.2.102:0/605179902 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f9000c710 con 0x7f9f90006770 2021-02-10T11:25:13.436 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:13.420 7f9fad1a4700 1 --2- 172.21.2.102:0/605179902 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9f90009500 0x7f9f9000ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:13.436 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:13.421 7f9fad1a4700 1 -- 172.21.2.102:0/605179902 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9f90006770 msgr2=0x7f9f90008c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:25:13.436 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:13.421 7f9fad1a4700 1 --2- 172.21.2.102:0/605179902 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9f90006770 0x7f9f90008c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:13.436 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:13.421 7f9fad1a4700 1 -- 172.21.2.102:0/605179902 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f9f90003a60 msgr2=0x7f9f90005ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:25:13.436 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:13.421 7f9fad1a4700 1 --2- 172.21.2.102:0/605179902 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f9f90003a60 0x7f9f90005ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:13.436 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:13.421 7f9fad1a4700 1 -- 172.21.2.102:0/605179902 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f9000c9b0 con 0x7f9f90009500 2021-02-10T11:25:13.436 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:13.421 7f9fad1a4700 1 --2- 172.21.2.102:0/605179902 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9f90009500 0x7f9f9000ba20 secure :-1 s=READY pgs=1439 cs=0 l=1 rev1=1 rx=0x7f9fa0000f00 tx=0x7f9fa004e320).ready entity=mon.0 client_cookie=a228d20b434b0bb4 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:13.436 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:13.421 7f9f9f7fe700 1 -- 172.21.2.102:0/605179902 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9fa0001800 con 0x7f9f90009500 2021-02-10T11:25:13.436 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:13.421 7f9f9f7fe700 1 -- 172.21.2.102:0/605179902 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fa00070d0 con 0x7f9f90009500 2021-02-10T11:25:13.436 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:13.421 7f9f9f7fe700 1 -- 172.21.2.102:0/605179902 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9fa0004720 con 0x7f9f90009500 2021-02-10T11:25:13.436 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:13.421 7f9faeb87700 1 -- 172.21.2.102:0/605179902 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9f90009500 msgr2=0x7f9f9000ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:13.436 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:13.421 7f9faeb87700 1 --2- 172.21.2.102:0/605179902 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9f90009500 0x7f9f9000ba20 secure :-1 s=READY pgs=1439 cs=0 l=1 rev1=1 rx=0x7f9fa0000f00 tx=0x7f9fa004e320).stop 2021-02-10T11:25:13.436 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:13.421 7f9fad1a4700 1 -- 172.21.2.102:0/605179902 reap_dead start 2021-02-10T11:25:13.437 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:13.422 7f9faeb87700 1 -- 172.21.2.102:0/605179902 shutdown_connections 2021-02-10T11:25:13.437 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:13.422 7f9faeb87700 1 -- 172.21.2.102:0/605179902 >> 172.21.2.102:0/605179902 conn(0x7f9fa80ce450 msgr2=0x7f9fa80d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:25:13.437 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:13.422 7f9faeb87700 1 -- 172.21.2.102:0/605179902 shutdown_connections 2021-02-10T11:25:13.437 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:13.422 7f9faeb87700 1 -- 172.21.2.102:0/605179902 wait complete. 2021-02-10T11:25:13.438 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:13.436 7f9faeb87700 1 Processor -- start 2021-02-10T11:25:13.438 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:13.436 7f9faeb87700 1 -- start start 2021-02-10T11:25:13.438 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:13.437 7f9faeb87700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f9fa8090bb0 0x7f9fa8090fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:13.439 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:13.437 7f9faeb87700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9fa819fb70 0x7f9fa819ff70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:13.439 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:13.437 7f9faeb87700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9fa81a0590 0x7f9fa80ecb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:13.439 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:13.437 7f9faeb87700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fa80915d0 con 0x7f9fa819fb70 2021-02-10T11:25:13.439 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:13.437 7f9faeb87700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fa81a0990 con 0x7f9fa8090bb0 2021-02-10T11:25:13.439 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:13.437 7f9faeb87700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9fa80ee5b0 con 0x7f9fa81a0590 2021-02-10T11:25:13.439 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:13.437 7f9f9ffff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9fa819fb70 0x7f9fa819ff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:13.439 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:13.437 7f9f9ffff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9fa819fb70 0x7f9fa819ff70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:57680/0 (socket says 172.21.2.102:57680) 2021-02-10T11:25:13.439 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:13.437 7f9f9ffff700 1 -- 172.21.2.102:0/2273416433 learned_addr learned my addr 172.21.2.102:0/2273416433 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:25:13.439 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:13.437 7f9f9ffff700 1 -- 172.21.2.102:0/2273416433 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9fa81a0590 msgr2=0x7f9fa80ecb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:13.439 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:13.437 7f9f9ffff700 1 --2- 172.21.2.102:0/2273416433 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9fa81a0590 0x7f9fa80ecb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:13.439 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:13.437 7f9f9ffff700 1 -- 172.21.2.102:0/2273416433 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f9fa8090bb0 msgr2=0x7f9fa8090fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:13.439 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:13.437 7f9f9ffff700 1 --2- 172.21.2.102:0/2273416433 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f9fa8090bb0 0x7f9fa8090fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:13.439 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:13.437 7f9f9ffff700 1 -- 172.21.2.102:0/2273416433 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fa80eea00 con 0x7f9fa819fb70 2021-02-10T11:25:13.440 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:13.437 7f9f9ffff700 1 --2- 172.21.2.102:0/2273416433 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9fa819fb70 0x7f9fa819ff70 secure :-1 s=READY pgs=1440 cs=0 l=1 rev1=1 rx=0x7f9fa400b580 tx=0x7f9fa400b920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:13.440 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:13.438 7f9f9d7fa700 1 -- 172.21.2.102:0/2273416433 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9fa400bd10 con 0x7f9fa819fb70 2021-02-10T11:25:13.440 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:13.438 7f9f9d7fa700 1 -- 172.21.2.102:0/2273416433 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fa4002030 con 0x7f9fa819fb70 2021-02-10T11:25:13.440 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:13.438 7f9faeb87700 1 -- 172.21.2.102:0/2273416433 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9fa80eea00 con 0x7f9fa819fb70 2021-02-10T11:25:13.440 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:13.438 7f9f9d7fa700 1 -- 172.21.2.102:0/2273416433 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9fa40024d0 con 0x7f9fa819fb70 2021-02-10T11:25:13.440 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:13.438 7f9faeb87700 1 -- 172.21.2.102:0/2273416433 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9fa80eea00 con 0x7f9fa819fb70 2021-02-10T11:25:13.440 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:13.438 7f9f9d7fa700 1 -- 172.21.2.102:0/2273416433 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f9fa4002210 con 0x7f9fa819fb70 2021-02-10T11:25:13.440 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:13.438 7f9f9d7fa700 1 --2- 172.21.2.102:0/2273416433 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f9f880186c0 0x7f9f8801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:13.440 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:13.438 7f9f9d7fa700 1 -- 172.21.2.102:0/2273416433 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f9fa4032580 con 0x7f9fa819fb70 2021-02-10T11:25:13.440 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:13.438 7f9fac9a3700 1 --2- 172.21.2.102:0/2273416433 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f9f880186c0 0x7f9f8801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:13.440 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:13.439 7f9fac9a3700 1 --2- 172.21.2.102:0/2273416433 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f9f880186c0 0x7f9f8801abc0 crc :-1 s=READY pgs=660 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:13.467 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:13.469 7f9faeb87700 1 -- 172.21.2.102:0/2273416433 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9fa80a55f0 con 0x7f9fa819fb70 2021-02-10T11:25:13.470 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:13.472 7f9f9d7fa700 1 -- 172.21.2.102:0/2273416433 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f9fa40279e0 con 0x7f9fa819fb70 2021-02-10T11:25:13.617 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:13.619 7f9faeb87700 1 -- 172.21.2.102:0/2273416433 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "config rm", "who": "client.alice", "name": "debug_objecter"} v 0) v1 -- 0x7f9fa80a55f0 con 0x7f9fa819fb70 2021-02-10T11:25:13.622 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:13.624 7f9f9d7fa700 1 -- 172.21.2.102:0/2273416433 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "client.alice", "name": "debug_objecter"}]=0 v13) v1 ==== 106+0+0 (secure 0 0 0) 0x7f9fa40279e0 con 0x7f9fa819fb70 2021-02-10T11:25:13.626 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:13.628 7f9faeb87700 1 -- 172.21.2.102:0/2273416433 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f9f880186c0 msgr2=0x7f9f8801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:13.626 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:13.628 7f9faeb87700 1 --2- 172.21.2.102:0/2273416433 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f9f880186c0 0x7f9f8801abc0 crc :-1 s=READY pgs=660 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:13.626 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:13.628 7f9faeb87700 1 -- 172.21.2.102:0/2273416433 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9fa819fb70 msgr2=0x7f9fa819ff70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:13.627 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:13.628 7f9faeb87700 1 --2- 172.21.2.102:0/2273416433 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9fa819fb70 0x7f9fa819ff70 secure :-1 s=READY pgs=1440 cs=0 l=1 rev1=1 rx=0x7f9fa400b580 tx=0x7f9fa400b920).stop 2021-02-10T11:25:13.627 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:13.629 7f9faeb87700 1 -- 172.21.2.102:0/2273416433 shutdown_connections 2021-02-10T11:25:13.627 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:13.629 7f9faeb87700 1 --2- 172.21.2.102:0/2273416433 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9fa81a0590 0x7f9fa80ecb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:13.627 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:13.629 7f9faeb87700 1 --2- 172.21.2.102:0/2273416433 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9fa819fb70 0x7f9fa819ff70 unknown :-1 s=CLOSED pgs=1440 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:13.627 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:13.629 7f9faeb87700 1 --2- 172.21.2.102:0/2273416433 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f9f880186c0 0x7f9f8801abc0 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:13.627 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:13.629 7f9faeb87700 1 --2- 172.21.2.102:0/2273416433 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f9fa8090bb0 0x7f9fa8090fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:13.628 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:13.629 7f9faeb87700 1 -- 172.21.2.102:0/2273416433 >> 172.21.2.102:0/2273416433 conn(0x7f9fa80bb350 msgr2=0x7f9fa80af280 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:25:13.628 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:13.629 7f9faeb87700 1 -- 172.21.2.102:0/2273416433 shutdown_connections 2021-02-10T11:25:13.628 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:13.629 7f9faeb87700 1 -- 172.21.2.102:0/2273416433 wait complete. 2021-02-10T11:25:13.636 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestVolumes.test_authorize_deauthorize_legacy_subvolume' 2021-02-10T11:25:14.063 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:14.047 7f44fd20c700 1 Processor -- start 2021-02-10T11:25:14.063 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:14.047 7f44fd20c700 1 -- start start 2021-02-10T11:25:14.063 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:14.048 7f44fd20c700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f44f80d1870 0x7f44f81b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:14.063 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:14.048 7f44fd20c700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44f80f0b10 con 0x7f44f80f0650 2021-02-10T11:25:14.064 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:14.048 7f44fd20c700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44f80d0310 con 0x7f44f81b0a10 2021-02-10T11:25:14.064 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:14.048 7f44fd20c700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44f80e5ed0 con 0x7f44f80d1870 2021-02-10T11:25:14.064 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:14.048 7f44f759e700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f44f80f0650 0x7f44f80cfb20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:49910/0 (socket says 172.21.2.102:49910) 2021-02-10T11:25:14.064 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:14.048 7f44f759e700 1 -- 172.21.2.102:0/3542372705 learned_addr learned my addr 172.21.2.102:0/3542372705 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:25:14.064 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:14.048 7f44f5d9b700 1 -- 172.21.2.102:0/3542372705 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3092342621 0 0) 0x7f44f0000d30 con 0x7f44f80f0650 2021-02-10T11:25:14.064 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:14.048 7f44f5d9b700 1 -- 172.21.2.102:0/3542372705 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44e4002fe0 con 0x7f44f80f0650 2021-02-10T11:25:14.064 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:14.048 7f44f6d9d700 1 --2- 172.21.2.102:0/3542372705 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f44f80d1870 0x7f44f81b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:14.064 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:14.048 7f44f5d9b700 1 -- 172.21.2.102:0/3542372705 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 315802149 0 0) 0x7f44f0001270 con 0x7f44f80f0650 2021-02-10T11:25:14.064 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:14.048 7f44f5d9b700 1 -- 172.21.2.102:0/3542372705 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f44f80d1870 msgr2=0x7f44f81b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:25:14.064 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:14.048 7f44f5d9b700 1 --2- 172.21.2.102:0/3542372705 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f44f80d1870 0x7f44f81b03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:14.064 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:14.048 7f44f5d9b700 1 -- 172.21.2.102:0/3542372705 >> v1:172.21.2.113:6789/0 conn(0x7f44f81b0a10 legacy=0x7f44f80cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:25:14.064 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:14.048 7f44f5d9b700 1 -- 172.21.2.102:0/3542372705 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44f80d0530 con 0x7f44f80f0650 2021-02-10T11:25:14.065 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:14.049 7f44f5d9b700 1 -- 172.21.2.102:0/3542372705 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7f44f0001a00 con 0x7f44f80f0650 2021-02-10T11:25:14.065 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:14.049 7f44f5d9b700 1 -- 172.21.2.102:0/3542372705 >> v1:172.21.2.102:6789/0 conn(0x7f44f80f0650 legacy=0x7f44f80cfb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:14.065 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:14.049 7f44f5d9b700 1 --2- 172.21.2.102:0/3542372705 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f44e4003aa0 0x7f44e4005eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:14.065 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:14.049 7f44f5d9b700 1 --2- 172.21.2.102:0/3542372705 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f44e4006750 0x7f44e4008c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:14.065 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:14.049 7f44f5d9b700 1 --2- 172.21.2.102:0/3542372705 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f44e40094f0 0x7f44e400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:14.065 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:14.049 7f44f5d9b700 1 -- 172.21.2.102:0/3542372705 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44e400c2c0 con 0x7f44e4006750 2021-02-10T11:25:14.065 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:14.049 7f44f5d9b700 1 -- 172.21.2.102:0/3542372705 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44e400c4e0 con 0x7f44e40094f0 2021-02-10T11:25:14.065 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:14.049 7f44f5d9b700 1 -- 172.21.2.102:0/3542372705 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44e400c700 con 0x7f44e4003aa0 2021-02-10T11:25:14.065 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:14.049 7f44f6d9d700 1 --2- 172.21.2.102:0/3542372705 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f44e4003aa0 0x7f44e4005eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:14.065 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:14.049 7f44f659c700 1 --2- 172.21.2.102:0/3542372705 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f44e4006750 0x7f44e4008c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:14.065 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:14.049 7f44f759e700 1 --2- 172.21.2.102:0/3542372705 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f44e40094f0 0x7f44e400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:14.066 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:14.049 7f44f659c700 1 -- 172.21.2.102:0/3542372705 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f44e4003aa0 msgr2=0x7f44e4005eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:25:14.066 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:14.049 7f44f659c700 1 --2- 172.21.2.102:0/3542372705 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f44e4003aa0 0x7f44e4005eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:14.066 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:14.049 7f44f659c700 1 -- 172.21.2.102:0/3542372705 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f44e40094f0 msgr2=0x7f44e400ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:25:14.066 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:14.049 7f44f659c700 1 --2- 172.21.2.102:0/3542372705 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f44e40094f0 0x7f44e400ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:14.066 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:14.049 7f44f659c700 1 -- 172.21.2.102:0/3542372705 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44e400c9a0 con 0x7f44e4006750 2021-02-10T11:25:14.066 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:14.049 7f44f659c700 1 --2- 172.21.2.102:0/3542372705 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f44e4006750 0x7f44e4008c50 secure :-1 s=READY pgs=1442 cs=0 l=1 rev1=1 rx=0x7f44e0006ab0 tx=0x7f44e0009a90).ready entity=mon.0 client_cookie=51e0b9dd2a132b77 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:14.066 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:14.050 7f44f5d9b700 1 -- 172.21.2.102:0/3542372705 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f44e000c0c0 con 0x7f44e4006750 2021-02-10T11:25:14.066 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:14.050 7f44f5d9b700 1 -- 172.21.2.102:0/3542372705 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44e00066f0 con 0x7f44e4006750 2021-02-10T11:25:14.066 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:14.050 7f44f5d9b700 1 -- 172.21.2.102:0/3542372705 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f44e000c640 con 0x7f44e4006750 2021-02-10T11:25:14.066 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:14.050 7f44fd20c700 1 -- 172.21.2.102:0/3542372705 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f44e4006750 msgr2=0x7f44e4008c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:14.066 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:14.050 7f44fd20c700 1 --2- 172.21.2.102:0/3542372705 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f44e4006750 0x7f44e4008c50 secure :-1 s=READY pgs=1442 cs=0 l=1 rev1=1 rx=0x7f44e0006ab0 tx=0x7f44e0009a90).stop 2021-02-10T11:25:14.067 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:14.050 7f44f759e700 1 -- 172.21.2.102:0/3542372705 reap_dead start 2021-02-10T11:25:14.067 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:14.050 7f44fd20c700 1 -- 172.21.2.102:0/3542372705 shutdown_connections 2021-02-10T11:25:14.067 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:14.050 7f44fd20c700 1 -- 172.21.2.102:0/3542372705 >> 172.21.2.102:0/3542372705 conn(0x7f44f80ce450 msgr2=0x7f44f80d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:25:14.067 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:14.050 7f44fd20c700 1 -- 172.21.2.102:0/3542372705 shutdown_connections 2021-02-10T11:25:14.067 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:14.051 7f44fd20c700 1 -- 172.21.2.102:0/3542372705 wait complete. 2021-02-10T11:25:14.067 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:14.065 7f44fd20c700 1 Processor -- start 2021-02-10T11:25:14.067 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:14.065 7f44fd20c700 1 -- start start 2021-02-10T11:25:14.068 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:14.066 7f44fd20c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f44f80a8da0 0x7f44f81a4490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:14.069 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:14.066 7f44fd20c700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f44f81a4ab0 0x7f44f80b3060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:14.069 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:14.066 7f44fd20c700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f44f80b3680 0x7f44f80b3a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:14.069 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:14.066 7f44fd20c700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44f80af760 con 0x7f44f80a8da0 2021-02-10T11:25:14.069 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:14.066 7f44fd20c700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44f80a91a0 con 0x7f44f80b3680 2021-02-10T11:25:14.069 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:14.066 7f44fd20c700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44f80afc60 con 0x7f44f81a4ab0 2021-02-10T11:25:14.069 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:14.066 7f44f6d9d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f44f80a8da0 0x7f44f81a4490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:14.069 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:14.066 7f44f6d9d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f44f80a8da0 0x7f44f81a4490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:57698/0 (socket says 172.21.2.102:57698) 2021-02-10T11:25:14.069 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:14.066 7f44f6d9d700 1 -- 172.21.2.102:0/3259755370 learned_addr learned my addr 172.21.2.102:0/3259755370 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:25:14.069 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:14.066 7f44f759e700 1 --2- 172.21.2.102:0/3259755370 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f44f80b3680 0x7f44f80b3a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:14.069 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:14.066 7f44f6d9d700 1 -- 172.21.2.102:0/3259755370 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f44f81a4ab0 msgr2=0x7f44f80b3060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:14.069 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:14.066 7f44f659c700 1 --2- 172.21.2.102:0/3259755370 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f44f81a4ab0 0x7f44f80b3060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:14.070 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:14.066 7f44f6d9d700 1 --2- 172.21.2.102:0/3259755370 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f44f81a4ab0 0x7f44f80b3060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:14.070 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:14.066 7f44f6d9d700 1 -- 172.21.2.102:0/3259755370 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f44f80b3680 msgr2=0x7f44f80b3a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:14.070 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:14.066 7f44f6d9d700 1 --2- 172.21.2.102:0/3259755370 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f44f80b3680 0x7f44f80b3a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:14.070 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:14.066 7f44f6d9d700 1 -- 172.21.2.102:0/3259755370 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44f80afe80 con 0x7f44f80a8da0 2021-02-10T11:25:14.070 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:14.066 7f44f759e700 1 --2- 172.21.2.102:0/3259755370 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f44f80b3680 0x7f44f80b3a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:25:14.070 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:14.066 7f44f659c700 1 --2- 172.21.2.102:0/3259755370 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f44f81a4ab0 0x7f44f80b3060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:25:14.070 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:14.067 7f44f6d9d700 1 --2- 172.21.2.102:0/3259755370 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f44f80a8da0 0x7f44f81a4490 secure :-1 s=READY pgs=1443 cs=0 l=1 rev1=1 rx=0x7f44e804f5e0 tx=0x7f44e804f980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:14.070 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:14.067 7f44ef7fe700 1 -- 172.21.2.102:0/3259755370 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f44e804fd70 con 0x7f44f80a8da0 2021-02-10T11:25:14.070 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:14.067 7f44ef7fe700 1 -- 172.21.2.102:0/3259755370 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44e8002030 con 0x7f44f80a8da0 2021-02-10T11:25:14.070 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:14.067 7f44fd20c700 1 -- 172.21.2.102:0/3259755370 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44f80afe80 con 0x7f44f80a8da0 2021-02-10T11:25:14.070 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:14.067 7f44fd20c700 1 -- 172.21.2.102:0/3259755370 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44f80b0410 con 0x7f44f80a8da0 2021-02-10T11:25:14.071 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:14.067 7f44ef7fe700 1 -- 172.21.2.102:0/3259755370 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f44e8002550 con 0x7f44f80a8da0 2021-02-10T11:25:14.071 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:14.068 7f44ef7fe700 1 -- 172.21.2.102:0/3259755370 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f44e8000c90 con 0x7f44f80a8da0 2021-02-10T11:25:14.071 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:14.068 7f44ef7fe700 1 --2- 172.21.2.102:0/3259755370 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f44dc0186d0 0x7f44dc01abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:14.071 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:14.068 7f44ef7fe700 1 -- 172.21.2.102:0/3259755370 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f44e8076820 con 0x7f44f80a8da0 2021-02-10T11:25:14.071 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:14.068 7f44f659c700 1 --2- 172.21.2.102:0/3259755370 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f44dc0186d0 0x7f44dc01abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:14.071 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:14.068 7f44f659c700 1 --2- 172.21.2.102:0/3259755370 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f44dc0186d0 0x7f44dc01abd0 crc :-1 s=READY pgs=661 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:14.096 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:14.098 7f44fd20c700 1 -- 172.21.2.102:0/3259755370 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44f80ebc80 con 0x7f44f80a8da0 2021-02-10T11:25:14.099 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:14.100 7f44ef7fe700 1 -- 172.21.2.102:0/3259755370 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f44e8000940 con 0x7f44f80a8da0 2021-02-10T11:25:14.233 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:14.235 7f44fd20c700 1 -- 172.21.2.102:0/3259755370 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestVolumes.test_authorize_deauthorize_legacy_subvolume"]} v 0) v1 -- 0x7f44f80f0b10 con 0x7f44f80a8da0 2021-02-10T11:25:15.017 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:15.019 7f44ef7fe700 1 -- 172.21.2.102:0/3259755370 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestVolumes.test_authorize_deauthorize_legacy_subvolume"]}]=0 v339) v1 ==== 160+0+0 (secure 0 0 0) 0x7f44e8000c90 con 0x7f44f80a8da0 2021-02-10T11:25:15.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:15.055 7f44fd20c700 1 -- 172.21.2.102:0/3259755370 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f44dc0186d0 msgr2=0x7f44dc01abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:15.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:15.055 7f44fd20c700 1 --2- 172.21.2.102:0/3259755370 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f44dc0186d0 0x7f44dc01abd0 crc :-1 s=READY pgs=661 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:15.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:15.055 7f44fd20c700 1 -- 172.21.2.102:0/3259755370 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f44f80a8da0 msgr2=0x7f44f81a4490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:15.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:15.055 7f44fd20c700 1 --2- 172.21.2.102:0/3259755370 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f44f80a8da0 0x7f44f81a4490 secure :-1 s=READY pgs=1443 cs=0 l=1 rev1=1 rx=0x7f44e804f5e0 tx=0x7f44e804f980).stop 2021-02-10T11:25:15.056 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:15.055 7f44fd20c700 1 -- 172.21.2.102:0/3259755370 shutdown_connections 2021-02-10T11:25:15.056 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:15.055 7f44fd20c700 1 --2- 172.21.2.102:0/3259755370 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f44dc0186d0 0x7f44dc01abd0 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:15.056 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:15.055 7f44fd20c700 1 --2- 172.21.2.102:0/3259755370 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f44f80b3680 0x7f44f80b3a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:15.056 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:15.055 7f44fd20c700 1 --2- 172.21.2.102:0/3259755370 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f44f81a4ab0 0x7f44f80b3060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:15.056 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:15.055 7f44fd20c700 1 --2- 172.21.2.102:0/3259755370 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f44f80a8da0 0x7f44f81a4490 unknown :-1 s=CLOSED pgs=1443 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:15.057 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:15.055 7f44fd20c700 1 -- 172.21.2.102:0/3259755370 >> 172.21.2.102:0/3259755370 conn(0x7f44f80a9bf0 msgr2=0x7f44f80ebeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:25:15.057 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:15.056 7f44fd20c700 1 -- 172.21.2.102:0/3259755370 shutdown_connections 2021-02-10T11:25:15.057 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:15.056 7f44fd20c700 1 -- 172.21.2.102:0/3259755370 wait complete. 2021-02-10T11:25:15.062 INFO:tasks.cephfs_test_runner:test_authorize_deauthorize_legacy_subvolume (tasks.cephfs.test_volumes.TestVolumes) ... ok 2021-02-10T11:25:15.063 INFO:tasks.cephfs_test_runner:Starting test: test_authorize_deauthorize_subvolume (tasks.cephfs.test_volumes.TestVolumes) 2021-02-10T11:25:15.063 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestVolumes.test_authorize_deauthorize_subvolume' 2021-02-10T11:25:15.490 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:15.475 7f7c4bdd3700 1 Processor -- start 2021-02-10T11:25:15.490 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:15.475 7f7c4bdd3700 1 -- start start 2021-02-10T11:25:15.491 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:15.475 7f7c4bdd3700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f7c440d1870 0x7f7c441b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:15.491 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:15.475 7f7c4bdd3700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7c440f0650 0x7f7c440cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:15.491 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:15.475 7f7c4bdd3700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c440f0b10 con 0x7f7c441b0a10 2021-02-10T11:25:15.491 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:15.475 7f7c4bdd3700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c440e5ed0 con 0x7f7c440d1870 2021-02-10T11:25:15.491 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:15.475 7f7c4bdd3700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c440d1c70 con 0x7f7c440f0650 2021-02-10T11:25:15.491 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:15.475 7f7c493ee700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f7c441b0a10 0x7f7c440cf330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:49930/0 (socket says 172.21.2.102:49930) 2021-02-10T11:25:15.491 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:15.475 7f7c493ee700 1 -- 172.21.2.102:0/4069754363 learned_addr learned my addr 172.21.2.102:0/4069754363 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:25:15.491 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:15.476 7f7c48bed700 1 -- 172.21.2.102:0/4069754363 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2954136294 0 0) 0x7f7c400010b0 con 0x7f7c441b0a10 2021-02-10T11:25:15.491 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:15.476 7f7c48bed700 1 -- 172.21.2.102:0/4069754363 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c30002fe0 con 0x7f7c441b0a10 2021-02-10T11:25:15.491 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:15.476 7f7c49bef700 1 --2- 172.21.2.102:0/4069754363 >> v2:172.21.2.113:3300/0 conn(0x7f7c440d1870 0x7f7c441b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:15.491 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:15.476 7f7c4a3f0700 1 --2- 172.21.2.102:0/4069754363 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7c440f0650 0x7f7c440cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:15.491 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:15.476 7f7c48bed700 1 -- 172.21.2.102:0/4069754363 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2324888644 0 0) 0x7f7c400010b0 con 0x7f7c441b0a10 2021-02-10T11:25:15.492 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:15.476 7f7c48bed700 1 -- 172.21.2.102:0/4069754363 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7c440f0650 msgr2=0x7f7c440cfb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:25:15.492 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:15.476 7f7c48bed700 1 --2- 172.21.2.102:0/4069754363 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7c440f0650 0x7f7c440cfb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:15.492 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:15.476 7f7c48bed700 1 -- 172.21.2.102:0/4069754363 >> v2:172.21.2.113:3300/0 conn(0x7f7c440d1870 msgr2=0x7f7c441b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:25:15.492 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:15.476 7f7c48bed700 1 --2- 172.21.2.102:0/4069754363 >> v2:172.21.2.113:3300/0 conn(0x7f7c440d1870 0x7f7c441b03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:15.492 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:15.476 7f7c48bed700 1 -- 172.21.2.102:0/4069754363 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c440d1e90 con 0x7f7c441b0a10 2021-02-10T11:25:15.492 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:15.476 7f7c48bed700 1 -- 172.21.2.102:0/4069754363 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7f7c400015b0 con 0x7f7c441b0a10 2021-02-10T11:25:15.492 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:15.476 7f7c48bed700 1 -- 172.21.2.102:0/4069754363 >> v1:172.21.2.102:6789/0 conn(0x7f7c441b0a10 legacy=0x7f7c440cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:15.492 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:15.476 7f7c48bed700 1 --2- 172.21.2.102:0/4069754363 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7c30003b90 0x7f7c30005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:15.492 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:15.476 7f7c48bed700 1 --2- 172.21.2.102:0/4069754363 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7c30006840 0x7f7c30008d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:15.492 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:15.476 7f7c48bed700 1 --2- 172.21.2.102:0/4069754363 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7c300095b0 0x7f7c3000bad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:15.492 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:15.476 7f7c48bed700 1 -- 172.21.2.102:0/4069754363 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c30050390 con 0x7f7c30003b90 2021-02-10T11:25:15.492 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:15.476 7f7c48bed700 1 -- 172.21.2.102:0/4069754363 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c300505b0 con 0x7f7c300095b0 2021-02-10T11:25:15.492 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:15.476 7f7c48bed700 1 -- 172.21.2.102:0/4069754363 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c300507d0 con 0x7f7c30006840 2021-02-10T11:25:15.493 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:15.476 7f7c49bef700 1 --2- 172.21.2.102:0/4069754363 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7c30003b90 0x7f7c30005fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:15.493 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:15.476 7f7c4a3f0700 1 --2- 172.21.2.102:0/4069754363 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7c440f0650 0x7f7c440cfb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:25:15.493 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:15.476 7f7c493ee700 1 --2- 172.21.2.102:0/4069754363 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7c30006840 0x7f7c30008d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:15.493 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:15.476 7f7c49bef700 1 -- 172.21.2.102:0/4069754363 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7c30006840 msgr2=0x7f7c30008d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:25:15.493 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:15.477 7f7c49bef700 1 --2- 172.21.2.102:0/4069754363 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7c30006840 0x7f7c30008d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:15.493 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:15.477 7f7c49bef700 1 -- 172.21.2.102:0/4069754363 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7c300095b0 msgr2=0x7f7c3000bad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:25:15.493 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:15.477 7f7c49bef700 1 --2- 172.21.2.102:0/4069754363 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7c300095b0 0x7f7c3000bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:15.493 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:15.477 7f7c49bef700 1 -- 172.21.2.102:0/4069754363 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c30050a70 con 0x7f7c30003b90 2021-02-10T11:25:15.493 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:15.477 7f7c493ee700 1 --2- 172.21.2.102:0/4069754363 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7c30006840 0x7f7c30008d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:25:15.494 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:15.477 7f7c49bef700 1 --2- 172.21.2.102:0/4069754363 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7c30003b90 0x7f7c30005fa0 secure :-1 s=READY pgs=1445 cs=0 l=1 rev1=1 rx=0x7f7c3800c8f0 tx=0x7f7c3800cc90).ready entity=mon.0 client_cookie=efe8d7b37fb2fc9c server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:15.494 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:15.477 7f7c48bed700 1 -- 172.21.2.102:0/4069754363 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7c3800cdd0 con 0x7f7c30003b90 2021-02-10T11:25:15.494 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:15.477 7f7c48bed700 1 -- 172.21.2.102:0/4069754363 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c38009310 con 0x7f7c30003b90 2021-02-10T11:25:15.494 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:15.477 7f7c48bed700 1 -- 172.21.2.102:0/4069754363 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7c380097b0 con 0x7f7c30003b90 2021-02-10T11:25:15.494 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:15.477 7f7c4bdd3700 1 -- 172.21.2.102:0/4069754363 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7c30003b90 msgr2=0x7f7c30005fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:15.494 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:15.477 7f7c4bdd3700 1 --2- 172.21.2.102:0/4069754363 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7c30003b90 0x7f7c30005fa0 secure :-1 s=READY pgs=1445 cs=0 l=1 rev1=1 rx=0x7f7c3800c8f0 tx=0x7f7c3800cc90).stop 2021-02-10T11:25:15.494 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:15.477 7f7c4a3f0700 1 -- 172.21.2.102:0/4069754363 reap_dead start 2021-02-10T11:25:15.494 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:15.477 7f7c4bdd3700 1 -- 172.21.2.102:0/4069754363 shutdown_connections 2021-02-10T11:25:15.494 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:15.477 7f7c4bdd3700 1 -- 172.21.2.102:0/4069754363 >> 172.21.2.102:0/4069754363 conn(0x7f7c440ce450 msgr2=0x7f7c440d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:25:15.494 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:15.478 7f7c4bdd3700 1 -- 172.21.2.102:0/4069754363 shutdown_connections 2021-02-10T11:25:15.494 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:15.478 7f7c4bdd3700 1 -- 172.21.2.102:0/4069754363 wait complete. 2021-02-10T11:25:15.495 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:15.492 7f7c4bdd3700 1 Processor -- start 2021-02-10T11:25:15.495 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:15.492 7f7c4bdd3700 1 -- start start 2021-02-10T11:25:15.496 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:15.493 7f7c4bdd3700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7c441942b0 0x7f7c440b3550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:15.496 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:15.493 7f7c4bdd3700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7c44195070 0x7f7c44195470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:15.496 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:15.493 7f7c4bdd3700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7c440bb8e0 0x7f7c440bbce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:15.496 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:15.493 7f7c4bdd3700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c441946b0 con 0x7f7c441942b0 2021-02-10T11:25:15.496 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:15.493 7f7c4bdd3700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c440b3b70 con 0x7f7c440bb8e0 2021-02-10T11:25:15.496 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:15.493 7f7c4bdd3700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c440b3d90 con 0x7f7c44195070 2021-02-10T11:25:15.496 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:15.493 7f7c49bef700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7c441942b0 0x7f7c440b3550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:15.496 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:15.493 7f7c49bef700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7c441942b0 0x7f7c440b3550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:57718/0 (socket says 172.21.2.102:57718) 2021-02-10T11:25:15.496 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:15.493 7f7c49bef700 1 -- 172.21.2.102:0/2476486720 learned_addr learned my addr 172.21.2.102:0/2476486720 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:25:15.496 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:15.493 7f7c49bef700 1 -- 172.21.2.102:0/2476486720 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7c44195070 msgr2=0x7f7c44195470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:15.497 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:15.493 7f7c49bef700 1 --2- 172.21.2.102:0/2476486720 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7c44195070 0x7f7c44195470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:15.497 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:15.493 7f7c49bef700 1 -- 172.21.2.102:0/2476486720 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7c440bb8e0 msgr2=0x7f7c440bbce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:15.497 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:15.493 7f7c49bef700 1 --2- 172.21.2.102:0/2476486720 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7c440bb8e0 0x7f7c440bbce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:15.497 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:15.493 7f7c49bef700 1 -- 172.21.2.102:0/2476486720 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c441a49e0 con 0x7f7c441942b0 2021-02-10T11:25:15.497 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:15.493 7f7c49bef700 1 --2- 172.21.2.102:0/2476486720 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7c441942b0 0x7f7c440b3550 secure :-1 s=READY pgs=1446 cs=0 l=1 rev1=1 rx=0x7f7c3800c430 tx=0x7f7c3800b4e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:15.497 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:15.494 7f7c367fc700 1 -- 172.21.2.102:0/2476486720 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7c3800b8d0 con 0x7f7c441942b0 2021-02-10T11:25:15.497 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:15.494 7f7c367fc700 1 -- 172.21.2.102:0/2476486720 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c3800bbd0 con 0x7f7c441942b0 2021-02-10T11:25:15.497 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:15.494 7f7c367fc700 1 -- 172.21.2.102:0/2476486720 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7c3800c4b0 con 0x7f7c441942b0 2021-02-10T11:25:15.497 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:15.494 7f7c4bdd3700 1 -- 172.21.2.102:0/2476486720 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c441a49e0 con 0x7f7c441942b0 2021-02-10T11:25:15.497 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:15.494 7f7c4bdd3700 1 -- 172.21.2.102:0/2476486720 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c441a49e0 con 0x7f7c441942b0 2021-02-10T11:25:15.497 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:15.494 7f7c367fc700 1 -- 172.21.2.102:0/2476486720 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f7c3800b8d0 con 0x7f7c441942b0 2021-02-10T11:25:15.498 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:15.494 7f7c367fc700 1 --2- 172.21.2.102:0/2476486720 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7c2c0186d0 0x7f7c2c01abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:15.498 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:15.494 7f7c367fc700 1 -- 172.21.2.102:0/2476486720 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f7c380327a0 con 0x7f7c441942b0 2021-02-10T11:25:15.498 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:15.495 7f7c493ee700 1 --2- 172.21.2.102:0/2476486720 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7c2c0186d0 0x7f7c2c01abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:15.498 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:15.495 7f7c493ee700 1 --2- 172.21.2.102:0/2476486720 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7c2c0186d0 0x7f7c2c01abd0 crc :-1 s=READY pgs=662 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:15.523 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:15.525 7f7c4bdd3700 1 -- 172.21.2.102:0/2476486720 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c441a49e0 con 0x7f7c441942b0 2021-02-10T11:25:15.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:15.528 7f7c367fc700 1 -- 172.21.2.102:0/2476486720 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f7c3800b8d0 con 0x7f7c441942b0 2021-02-10T11:25:15.659 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:15.661 7f7c4bdd3700 1 -- 172.21.2.102:0/2476486720 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestVolumes.test_authorize_deauthorize_subvolume"]} v 0) v1 -- 0x7f7c441a49e0 con 0x7f7c441942b0 2021-02-10T11:25:16.023 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:16.025 7f7c367fc700 1 -- 172.21.2.102:0/2476486720 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestVolumes.test_authorize_deauthorize_subvolume"]}]=0 v340) v1 ==== 156+0+0 (secure 0 0 0) 0x7f7c3800bbd0 con 0x7f7c441942b0 2021-02-10T11:25:16.028 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:16.030 7f7c4bdd3700 1 -- 172.21.2.102:0/2476486720 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7c2c0186d0 msgr2=0x7f7c2c01abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:16.028 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:16.030 7f7c4bdd3700 1 --2- 172.21.2.102:0/2476486720 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7c2c0186d0 0x7f7c2c01abd0 crc :-1 s=READY pgs=662 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:16.028 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:16.030 7f7c4bdd3700 1 -- 172.21.2.102:0/2476486720 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7c441942b0 msgr2=0x7f7c440b3550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:16.028 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:16.030 7f7c4bdd3700 1 --2- 172.21.2.102:0/2476486720 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7c441942b0 0x7f7c440b3550 secure :-1 s=READY pgs=1446 cs=0 l=1 rev1=1 rx=0x7f7c3800c430 tx=0x7f7c3800b4e0).stop 2021-02-10T11:25:16.028 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:16.030 7f7c4bdd3700 1 -- 172.21.2.102:0/2476486720 shutdown_connections 2021-02-10T11:25:16.028 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:16.030 7f7c4bdd3700 1 --2- 172.21.2.102:0/2476486720 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7c2c0186d0 0x7f7c2c01abd0 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:16.028 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:16.030 7f7c4bdd3700 1 --2- 172.21.2.102:0/2476486720 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7c440bb8e0 0x7f7c440bbce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:16.028 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:16.030 7f7c4bdd3700 1 --2- 172.21.2.102:0/2476486720 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7c44195070 0x7f7c44195470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:16.028 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:16.030 7f7c4bdd3700 1 --2- 172.21.2.102:0/2476486720 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7c441942b0 0x7f7c440b3550 unknown :-1 s=CLOSED pgs=1446 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:16.029 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:16.030 7f7c4bdd3700 1 -- 172.21.2.102:0/2476486720 >> 172.21.2.102:0/2476486720 conn(0x7f7c440b3040 msgr2=0x7f7c440c3250 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:25:16.029 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:16.030 7f7c4bdd3700 1 -- 172.21.2.102:0/2476486720 shutdown_connections 2021-02-10T11:25:16.029 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:16.030 7f7c4bdd3700 1 -- 172.21.2.102:0/2476486720 wait complete. 2021-02-10T11:25:16.036 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config set mon mon_allow_pool_delete True 2021-02-10T11:25:16.468 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:16.452 7f646d5cd700 1 Processor -- start 2021-02-10T11:25:16.468 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:16.452 7f646d5cd700 1 -- start start 2021-02-10T11:25:16.468 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:16.453 7f646d5cd700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f646810aff0 0x7f64680cf750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:16.469 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:16.453 7f646d5cd700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64680fe1c0 con 0x7f64680d1c70 2021-02-10T11:25:16.469 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:16.453 7f646d5cd700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64681b0cb0 con 0x7f64680d1870 2021-02-10T11:25:16.469 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:16.453 7f646d5cd700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6468109a60 con 0x7f646810aff0 2021-02-10T11:25:16.469 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:16.453 7f6466abe700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f64680d1c70 0x7f64680cef60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:49950/0 (socket says 172.21.2.102:49950) 2021-02-10T11:25:16.469 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:16.453 7f6466abe700 1 -- 172.21.2.102:0/416326314 learned_addr learned my addr 172.21.2.102:0/416326314 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:25:16.469 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:16.454 7f64662bd700 1 -- 172.21.2.102:0/416326314 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3607117552 0 0) 0x7f6458000ce0 con 0x7f64680d1c70 2021-02-10T11:25:16.469 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:16.454 7f64662bd700 1 -- 172.21.2.102:0/416326314 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6454002fe0 con 0x7f64680d1c70 2021-02-10T11:25:16.469 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:16.454 7f6467ac0700 1 --2- 172.21.2.102:0/416326314 >> v2:172.21.2.113:3300/0 conn(0x7f646810aff0 0x7f64680cf750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:16.469 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:16.454 7f64662bd700 1 -- 172.21.2.102:0/416326314 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1408279982 0 0) 0x7f64580010e0 con 0x7f64680d1c70 2021-02-10T11:25:16.469 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:16.454 7f64662bd700 1 -- 172.21.2.102:0/416326314 >> v2:172.21.2.113:3300/0 conn(0x7f646810aff0 msgr2=0x7f64680cf750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:25:16.470 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:16.454 7f64662bd700 1 --2- 172.21.2.102:0/416326314 >> v2:172.21.2.113:3300/0 conn(0x7f646810aff0 0x7f64680cf750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:16.470 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:16.454 7f64662bd700 1 -- 172.21.2.102:0/416326314 >> v1:172.21.2.113:6789/0 conn(0x7f64680d1870 legacy=0x7f64681b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:16.470 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:16.454 7f64662bd700 1 -- 172.21.2.102:0/416326314 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f646810b3f0 con 0x7f64680d1c70 2021-02-10T11:25:16.470 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:16.454 7f64662bd700 1 -- 172.21.2.102:0/416326314 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7f6458001600 con 0x7f64680d1c70 2021-02-10T11:25:16.470 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:16.454 7f64662bd700 1 -- 172.21.2.102:0/416326314 >> v1:172.21.2.102:6789/0 conn(0x7f64680d1c70 legacy=0x7f64680cef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:16.470 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:16.454 7f64662bd700 1 --2- 172.21.2.102:0/416326314 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6454003aa0 0x7f6454005ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:16.470 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:16.454 7f64662bd700 1 --2- 172.21.2.102:0/416326314 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6454006790 0x7f6454008c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:16.470 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:16.454 7f64662bd700 1 --2- 172.21.2.102:0/416326314 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6454009590 0x7f645400bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:16.470 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:16.454 7f64662bd700 1 -- 172.21.2.102:0/416326314 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f645400c300 con 0x7f6454009590 2021-02-10T11:25:16.470 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:16.454 7f64662bd700 1 -- 172.21.2.102:0/416326314 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f645400c520 con 0x7f6454003aa0 2021-02-10T11:25:16.470 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:16.454 7f64662bd700 1 -- 172.21.2.102:0/416326314 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f645400c740 con 0x7f6454006790 2021-02-10T11:25:16.470 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:16.454 7f6467ac0700 1 --2- 172.21.2.102:0/416326314 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6454009590 0x7f645400bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:16.471 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:16.454 7f6466abe700 1 --2- 172.21.2.102:0/416326314 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6454006790 0x7f6454008c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:16.471 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:16.454 7f64672bf700 1 --2- 172.21.2.102:0/416326314 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6454003aa0 0x7f6454005ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:16.471 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:16.455 7f6467ac0700 1 -- 172.21.2.102:0/416326314 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6454006790 msgr2=0x7f6454008c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:25:16.471 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:16.455 7f6467ac0700 1 --2- 172.21.2.102:0/416326314 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6454006790 0x7f6454008c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:16.471 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:16.455 7f6467ac0700 1 -- 172.21.2.102:0/416326314 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6454003aa0 msgr2=0x7f6454005ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:25:16.471 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:16.455 7f6467ac0700 1 --2- 172.21.2.102:0/416326314 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6454003aa0 0x7f6454005ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:16.471 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:16.455 7f6467ac0700 1 -- 172.21.2.102:0/416326314 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6454050980 con 0x7f6454009590 2021-02-10T11:25:16.471 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:16.455 7f6467ac0700 1 -- 172.21.2.102:0/416326314 reap_dead start 2021-02-10T11:25:16.471 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:16.455 7f6467ac0700 1 --2- 172.21.2.102:0/416326314 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6454009590 0x7f645400bab0 secure :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 rx=0x7f6460006a40 tx=0x7f646000c0c0).ready entity=mon.0 client_cookie=76e606b193ec8603 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:16.471 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:16.455 7f64662bd700 1 -- 172.21.2.102:0/416326314 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6460006a80 con 0x7f6454009590 2021-02-10T11:25:16.471 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:16.455 7f64662bd700 1 -- 172.21.2.102:0/416326314 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6460006a80 con 0x7f6454009590 2021-02-10T11:25:16.471 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:16.455 7f64662bd700 1 -- 172.21.2.102:0/416326314 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6460007070 con 0x7f6454009590 2021-02-10T11:25:16.472 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:16.455 7f646d5cd700 1 -- 172.21.2.102:0/416326314 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6454009590 msgr2=0x7f645400bab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:16.472 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:16.455 7f646d5cd700 1 --2- 172.21.2.102:0/416326314 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6454009590 0x7f645400bab0 secure :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 rx=0x7f6460006a40 tx=0x7f646000c0c0).stop 2021-02-10T11:25:16.472 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:16.456 7f646d5cd700 1 -- 172.21.2.102:0/416326314 shutdown_connections 2021-02-10T11:25:16.472 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:16.456 7f646d5cd700 1 --2- 172.21.2.102:0/416326314 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6454009590 0x7f645400bab0 unknown :-1 s=CLOSED pgs=1448 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:16.472 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:16.456 7f646d5cd700 1 -- 172.21.2.102:0/416326314 >> 172.21.2.102:0/416326314 conn(0x7f64680ce420 msgr2=0x7f64680d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:25:16.472 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:16.456 7f646d5cd700 1 -- 172.21.2.102:0/416326314 shutdown_connections 2021-02-10T11:25:16.472 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:16.456 7f646d5cd700 1 -- 172.21.2.102:0/416326314 wait complete. 2021-02-10T11:25:16.474 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:16.471 7f646d5cd700 1 Processor -- start 2021-02-10T11:25:16.474 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:16.471 7f646d5cd700 1 -- start start 2021-02-10T11:25:16.474 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:16.471 7f646d5cd700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6468085bc0 0x7f64680863e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:16.474 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:16.471 7f646d5cd700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f64680954c0 0x7f6468096300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:16.474 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:16.471 7f646d5cd700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f64680958c0 0x7f6468096920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:16.474 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:16.471 7f646d5cd700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6468095d80 con 0x7f64680958c0 2021-02-10T11:25:16.474 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:16.471 7f646d5cd700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64680bfa60 con 0x7f6468085bc0 2021-02-10T11:25:16.474 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:16.471 7f646d5cd700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6468085fc0 con 0x7f64680954c0 2021-02-10T11:25:16.474 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:16.471 7f6467ac0700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f64680958c0 0x7f6468096920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:16.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:16.472 7f6467ac0700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f64680958c0 0x7f6468096920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:57736/0 (socket says 172.21.2.102:57736) 2021-02-10T11:25:16.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:16.472 7f6467ac0700 1 -- 172.21.2.102:0/3973304229 learned_addr learned my addr 172.21.2.102:0/3973304229 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:25:16.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:16.472 7f6466abe700 1 --2- 172.21.2.102:0/3973304229 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f64680954c0 0x7f6468096300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:16.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:16.472 7f64672bf700 1 --2- 172.21.2.102:0/3973304229 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6468085bc0 0x7f64680863e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:16.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:16.472 7f6467ac0700 1 -- 172.21.2.102:0/3973304229 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f64680954c0 msgr2=0x7f6468096300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:16.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:16.472 7f6467ac0700 1 --2- 172.21.2.102:0/3973304229 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f64680954c0 0x7f6468096300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:16.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:16.472 7f6467ac0700 1 -- 172.21.2.102:0/3973304229 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6468085bc0 msgr2=0x7f64680863e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:16.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:16.472 7f6467ac0700 1 --2- 172.21.2.102:0/3973304229 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6468085bc0 0x7f64680863e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:16.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:16.472 7f6467ac0700 1 -- 172.21.2.102:0/3973304229 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6468086c10 con 0x7f64680958c0 2021-02-10T11:25:16.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:16.472 7f64672bf700 1 --2- 172.21.2.102:0/3973304229 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6468085bc0 0x7f64680863e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:25:16.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:16.472 7f6467ac0700 1 --2- 172.21.2.102:0/3973304229 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f64680958c0 0x7f6468096920 secure :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 rx=0x7f646000b8e0 tx=0x7f646000bc80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:16.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:16.472 7f645ffff700 1 -- 172.21.2.102:0/3973304229 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f646000c2f0 con 0x7f64680958c0 2021-02-10T11:25:16.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:16.472 7f645ffff700 1 -- 172.21.2.102:0/3973304229 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f646000c570 con 0x7f64680958c0 2021-02-10T11:25:16.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:16.472 7f646d5cd700 1 -- 172.21.2.102:0/3973304229 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6468086c10 con 0x7f64680958c0 2021-02-10T11:25:16.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:16.472 7f645ffff700 1 -- 172.21.2.102:0/3973304229 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f646000ca90 con 0x7f64680958c0 2021-02-10T11:25:16.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:16.473 7f646d5cd700 1 -- 172.21.2.102:0/3973304229 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6468097230 con 0x7f64680958c0 2021-02-10T11:25:16.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:16.473 7f645ffff700 1 -- 172.21.2.102:0/3973304229 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f6460002020 con 0x7f64680958c0 2021-02-10T11:25:16.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:16.473 7f645ffff700 1 --2- 172.21.2.102:0/3973304229 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f644c0186c0 0x7f644c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:16.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:16.473 7f645ffff700 1 -- 172.21.2.102:0/3973304229 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f6460032790 con 0x7f64680958c0 2021-02-10T11:25:16.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:16.473 7f64672bf700 1 --2- 172.21.2.102:0/3973304229 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f644c0186c0 0x7f644c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:16.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:16.474 7f64672bf700 1 --2- 172.21.2.102:0/3973304229 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f644c0186c0 0x7f644c01abc0 crc :-1 s=READY pgs=663 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:16.501 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:16.503 7f646d5cd700 1 -- 172.21.2.102:0/3973304229 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64680fe1c0 con 0x7f64680958c0 2021-02-10T11:25:16.503 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:16.505 7f645ffff700 1 -- 172.21.2.102:0/3973304229 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f646002f610 con 0x7f64680958c0 2021-02-10T11:25:16.651 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:16.653 7f646d5cd700 1 -- 172.21.2.102:0/3973304229 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f64680fe1c0 con 0x7f64680958c0 2021-02-10T11:25:16.654 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:16.656 7f645ffff700 1 -- 172.21.2.102:0/3973304229 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v14)=0 v14) v1 ==== 122+0+0 (secure 0 0 0) 0x7f646002f610 con 0x7f64680958c0 2021-02-10T11:25:16.659 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:16.661 7f646d5cd700 1 -- 172.21.2.102:0/3973304229 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f644c0186c0 msgr2=0x7f644c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:16.659 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:16.661 7f646d5cd700 1 --2- 172.21.2.102:0/3973304229 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f644c0186c0 0x7f644c01abc0 crc :-1 s=READY pgs=663 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:16.659 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:16.661 7f646d5cd700 1 -- 172.21.2.102:0/3973304229 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f64680958c0 msgr2=0x7f6468096920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:16.660 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:16.661 7f646d5cd700 1 --2- 172.21.2.102:0/3973304229 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f64680958c0 0x7f6468096920 secure :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 rx=0x7f646000b8e0 tx=0x7f646000bc80).stop 2021-02-10T11:25:16.660 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:16.661 7f646d5cd700 1 -- 172.21.2.102:0/3973304229 shutdown_connections 2021-02-10T11:25:16.660 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:16.661 7f646d5cd700 1 --2- 172.21.2.102:0/3973304229 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f64680958c0 0x7f6468096920 unknown :-1 s=CLOSED pgs=1449 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:16.660 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:16.661 7f646d5cd700 1 --2- 172.21.2.102:0/3973304229 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f64680954c0 0x7f6468096300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:16.660 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:16.661 7f646d5cd700 1 --2- 172.21.2.102:0/3973304229 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f644c0186c0 0x7f644c01abc0 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:16.660 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:16.661 7f646d5cd700 1 --2- 172.21.2.102:0/3973304229 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6468085bc0 0x7f64680863e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:16.660 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:16.661 7f646d5cd700 1 -- 172.21.2.102:0/3973304229 >> 172.21.2.102:0/3973304229 conn(0x7f64680bef00 msgr2=0x7f646809d1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:25:16.660 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:16.661 7f646d5cd700 1 -- 172.21.2.102:0/3973304229 shutdown_connections 2021-02-10T11:25:16.660 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:16.662 7f646d5cd700 1 -- 172.21.2.102:0/3973304229 wait complete. 2021-02-10T11:25:16.668 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2021-02-10T11:25:16.669 DEBUG:teuthology.orchestra.run.gibba002:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:25:17.014 DEBUG:teuthology.orchestra.run.gibba013:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:25:17.360 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2021-02-10T11:25:17.361 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:17.361 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:25:17.361 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:25:17.703 INFO:teuthology.orchestra.run.gibba002.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.0’: No such file or directory 2021-02-10T11:25:17.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:25:17.704 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.0 2021-02-10T11:25:17.704 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:17.704 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:25:17.704 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:25:18.039 INFO:teuthology.orchestra.run.gibba002.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.1’: No such file or directory 2021-02-10T11:25:18.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:25:18.040 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2021-02-10T11:25:18.040 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:18.040 DEBUG:teuthology.orchestra.run.gibba013:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:25:18.040 DEBUG:teuthology.orchestra.run.gibba013:> ' -- /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:25:18.373 INFO:teuthology.orchestra.run.gibba013.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.2’: No such file or directory 2021-02-10T11:25:18.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:25:18.374 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2021-02-10T11:25:18.374 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:18.374 DEBUG:teuthology.orchestra.run.gibba013:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:25:18.374 DEBUG:teuthology.orchestra.run.gibba013:> ' -- /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:25:18.709 INFO:teuthology.orchestra.run.gibba013.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.3’: No such file or directory 2021-02-10T11:25:18.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:25:18.710 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.3 2021-02-10T11:25:18.710 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json-pretty 2021-02-10T11:25:19.138 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:19.122 7f35933ef700 1 Processor -- start 2021-02-10T11:25:19.138 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:19.122 7f35933ef700 1 -- start start 2021-02-10T11:25:19.139 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:19.123 7f35933ef700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f358c0d0340 0x7f358c0ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:19.139 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:19.123 7f35933ef700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f358c0d1c70 0x7f358c0efc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:19.139 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:19.123 7f35933ef700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f358c0f0b10 con 0x7f358c0d1870 2021-02-10T11:25:19.139 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:19.123 7f35933ef700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f358c0e5ed0 con 0x7f358c0d0340 2021-02-10T11:25:19.139 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:19.123 7f35933ef700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f358c1b0be0 con 0x7f358c0d1c70 2021-02-10T11:25:19.139 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:19.123 7f359120b700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f358c0d1870 0x7f358c1b03f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:49968/0 (socket says 172.21.2.102:49968) 2021-02-10T11:25:19.139 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:19.123 7f359120b700 1 -- 172.21.2.102:0/299035362 learned_addr learned my addr 172.21.2.102:0/299035362 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:25:19.139 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:19.123 7f3583fff700 1 -- 172.21.2.102:0/299035362 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2276212106 0 0) 0x7f357c001180 con 0x7f358c0d1870 2021-02-10T11:25:19.139 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:19.123 7f3583fff700 1 -- 172.21.2.102:0/299035362 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3578002fe0 con 0x7f358c0d1870 2021-02-10T11:25:19.139 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:19.124 7f3591a0c700 1 --2- 172.21.2.102:0/299035362 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f358c0d1c70 0x7f358c0efc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:19.139 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:19.124 7f3590a0a700 1 --2- 172.21.2.102:0/299035362 >> v2:172.21.2.113:3300/0 conn(0x7f358c0d0340 0x7f358c0ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:19.139 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:19.124 7f3583fff700 1 -- 172.21.2.102:0/299035362 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1732770158 0 0) 0x7f357c001180 con 0x7f358c0d1870 2021-02-10T11:25:19.140 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:19.124 7f3583fff700 1 -- 172.21.2.102:0/299035362 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f358c0d1c70 msgr2=0x7f358c0efc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:25:19.140 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:19.124 7f3583fff700 1 --2- 172.21.2.102:0/299035362 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f358c0d1c70 0x7f358c0efc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:19.140 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:19.124 7f3583fff700 1 -- 172.21.2.102:0/299035362 >> v2:172.21.2.113:3300/0 conn(0x7f358c0d0340 msgr2=0x7f358c0ef640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:25:19.140 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:19.124 7f3583fff700 1 --2- 172.21.2.102:0/299035362 >> v2:172.21.2.113:3300/0 conn(0x7f358c0d0340 0x7f358c0ef640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:19.140 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:19.124 7f3583fff700 1 -- 172.21.2.102:0/299035362 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f358c0f20a0 con 0x7f358c0d1870 2021-02-10T11:25:19.140 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:19.124 7f3583fff700 1 -- 172.21.2.102:0/299035362 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7f357c001680 con 0x7f358c0d1870 2021-02-10T11:25:19.140 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:19.124 7f3583fff700 1 -- 172.21.2.102:0/299035362 >> v1:172.21.2.102:6789/0 conn(0x7f358c0d1870 legacy=0x7f358c1b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:19.140 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:19.124 7f3583fff700 1 --2- 172.21.2.102:0/299035362 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3578003b10 0x7f3578005f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:19.140 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:19.124 7f3583fff700 1 --2- 172.21.2.102:0/299035362 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f35780067a0 0x7f3578008c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:19.140 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:19.124 7f3583fff700 1 --2- 172.21.2.102:0/299035362 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f35780094f0 0x7f357800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:19.140 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:19.124 7f3583fff700 1 -- 172.21.2.102:0/299035362 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f357800c2c0 con 0x7f3578003b10 2021-02-10T11:25:19.140 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:19.124 7f3583fff700 1 -- 172.21.2.102:0/299035362 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35780504b0 con 0x7f35780094f0 2021-02-10T11:25:19.140 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:19.124 7f3583fff700 1 -- 172.21.2.102:0/299035362 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35780506d0 con 0x7f35780067a0 2021-02-10T11:25:19.141 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:19.124 7f359120b700 1 --2- 172.21.2.102:0/299035362 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3578003b10 0x7f3578005f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:19.141 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:19.124 7f3590a0a700 1 --2- 172.21.2.102:0/299035362 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f35780067a0 0x7f3578008c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:19.141 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:19.124 7f3591a0c700 1 --2- 172.21.2.102:0/299035362 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f35780094f0 0x7f357800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:19.141 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:19.124 7f359120b700 1 -- 172.21.2.102:0/299035362 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f35780067a0 msgr2=0x7f3578008c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:25:19.141 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:19.124 7f359120b700 1 --2- 172.21.2.102:0/299035362 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f35780067a0 0x7f3578008c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:19.141 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:19.124 7f359120b700 1 -- 172.21.2.102:0/299035362 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f35780094f0 msgr2=0x7f357800ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:25:19.141 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:19.124 7f359120b700 1 --2- 172.21.2.102:0/299035362 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f35780094f0 0x7f357800ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:19.141 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:19.124 7f359120b700 1 -- 172.21.2.102:0/299035362 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3578050970 con 0x7f3578003b10 2021-02-10T11:25:19.141 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:19.124 7f3591a0c700 1 --2- 172.21.2.102:0/299035362 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f35780094f0 0x7f357800ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:25:19.141 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:19.124 7f359120b700 1 --2- 172.21.2.102:0/299035362 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3578003b10 0x7f3578005f20 secure :-1 s=READY pgs=1451 cs=0 l=1 rev1=1 rx=0x7f357c007970 tx=0x7f357c007d10).ready entity=mon.0 client_cookie=de3b98edbc7f733e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:19.141 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:19.125 7f3583fff700 1 -- 172.21.2.102:0/299035362 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f357c0070c0 con 0x7f3578003b10 2021-02-10T11:25:19.141 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:19.125 7f3583fff700 1 -- 172.21.2.102:0/299035362 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f357c0070c0 con 0x7f3578003b10 2021-02-10T11:25:19.141 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:19.125 7f3583fff700 1 -- 172.21.2.102:0/299035362 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f357c0043a0 con 0x7f3578003b10 2021-02-10T11:25:19.142 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:19.125 7f35933ef700 1 -- 172.21.2.102:0/299035362 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3578003b10 msgr2=0x7f3578005f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:19.142 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:19.125 7f35933ef700 1 --2- 172.21.2.102:0/299035362 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3578003b10 0x7f3578005f20 secure :-1 s=READY pgs=1451 cs=0 l=1 rev1=1 rx=0x7f357c007970 tx=0x7f357c007d10).stop 2021-02-10T11:25:19.142 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:19.125 7f3591a0c700 1 -- 172.21.2.102:0/299035362 reap_dead start 2021-02-10T11:25:19.142 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:19.125 7f35933ef700 1 -- 172.21.2.102:0/299035362 shutdown_connections 2021-02-10T11:25:19.142 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:19.125 7f35933ef700 1 -- 172.21.2.102:0/299035362 >> 172.21.2.102:0/299035362 conn(0x7f358c0ce450 msgr2=0x7f358c0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:25:19.142 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:19.125 7f35933ef700 1 -- 172.21.2.102:0/299035362 shutdown_connections 2021-02-10T11:25:19.142 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:19.126 7f35933ef700 1 -- 172.21.2.102:0/299035362 wait complete. 2021-02-10T11:25:19.142 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:19.140 7f35933ef700 1 Processor -- start 2021-02-10T11:25:19.142 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:19.140 7f35933ef700 1 -- start start 2021-02-10T11:25:19.143 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:19.141 7f35933ef700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f358c1942b0 0x7f358c0b3550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:19.143 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:19.141 7f35933ef700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f358c195070 0x7f358c195470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:19.144 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:19.141 7f35933ef700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f358c0bb8e0 0x7f358c0bbce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:19.144 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:19.141 7f35933ef700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f358c1946b0 con 0x7f358c0bb8e0 2021-02-10T11:25:19.144 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:19.141 7f35933ef700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f358c0b3b70 con 0x7f358c1942b0 2021-02-10T11:25:19.144 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:19.141 7f35933ef700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f358c0b3d90 con 0x7f358c195070 2021-02-10T11:25:19.144 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:19.141 7f3591a0c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f358c0bb8e0 0x7f358c0bbce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:19.144 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:19.141 7f3591a0c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f358c0bb8e0 0x7f358c0bbce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:57754/0 (socket says 172.21.2.102:57754) 2021-02-10T11:25:19.144 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:19.141 7f3591a0c700 1 -- 172.21.2.102:0/1016283506 learned_addr learned my addr 172.21.2.102:0/1016283506 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:25:19.144 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:19.141 7f3591a0c700 1 -- 172.21.2.102:0/1016283506 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f358c195070 msgr2=0x7f358c195470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:19.144 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:19.141 7f3591a0c700 1 --2- 172.21.2.102:0/1016283506 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f358c195070 0x7f358c195470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:19.144 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:19.141 7f3591a0c700 1 -- 172.21.2.102:0/1016283506 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f358c1942b0 msgr2=0x7f358c0b3550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:19.144 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:19.141 7f3591a0c700 1 --2- 172.21.2.102:0/1016283506 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f358c1942b0 0x7f358c0b3550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:19.144 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:19.141 7f3591a0c700 1 -- 172.21.2.102:0/1016283506 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f358c1a49e0 con 0x7f358c0bb8e0 2021-02-10T11:25:19.144 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:19.141 7f3591a0c700 1 --2- 172.21.2.102:0/1016283506 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f358c0bb8e0 0x7f358c0bbce0 secure :-1 s=READY pgs=1452 cs=0 l=1 rev1=1 rx=0x7f3584009980 tx=0x7f358400bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:19.145 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:19.142 7f3581ffb700 1 -- 172.21.2.102:0/1016283506 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3584006990 con 0x7f358c0bb8e0 2021-02-10T11:25:19.145 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:19.142 7f3581ffb700 1 -- 172.21.2.102:0/1016283506 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f358400c2f0 con 0x7f358c0bb8e0 2021-02-10T11:25:19.145 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:19.142 7f3581ffb700 1 -- 172.21.2.102:0/1016283506 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f358400c750 con 0x7f358c0bb8e0 2021-02-10T11:25:19.145 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:19.142 7f35933ef700 1 -- 172.21.2.102:0/1016283506 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f358c1a49e0 con 0x7f358c0bb8e0 2021-02-10T11:25:19.145 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:19.142 7f35933ef700 1 -- 172.21.2.102:0/1016283506 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f358c0bc5f0 con 0x7f358c0bb8e0 2021-02-10T11:25:19.145 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:19.142 7f3581ffb700 1 -- 172.21.2.102:0/1016283506 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f3584000df0 con 0x7f358c0bb8e0 2021-02-10T11:25:19.145 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:19.142 7f3581ffb700 1 --2- 172.21.2.102:0/1016283506 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f356c0186c0 0x7f356c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:19.145 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:19.142 7f3581ffb700 1 -- 172.21.2.102:0/1016283506 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f3584032900 con 0x7f358c0bb8e0 2021-02-10T11:25:19.145 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:19.142 7f359120b700 1 --2- 172.21.2.102:0/1016283506 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f356c0186c0 0x7f356c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:19.145 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:19.143 7f359120b700 1 --2- 172.21.2.102:0/1016283506 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f356c0186c0 0x7f356c01abc0 crc :-1 s=READY pgs=664 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:19.170 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:19.172 7f35933ef700 1 -- 172.21.2.102:0/1016283506 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f358c0bc5f0 con 0x7f358c0bb8e0 2021-02-10T11:25:19.173 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:19.175 7f3581ffb700 1 -- 172.21.2.102:0/1016283506 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f358401f860 con 0x7f358c0bb8e0 2021-02-10T11:25:19.296 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:19.298 7f35933ef700 1 -- 172.21.2.102:0/1016283506 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7f358c0bc5f0 con 0x7f358c0bb8e0 2021-02-10T11:25:19.297 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:19.299 7f3581ffb700 1 -- 172.21.2.102:0/1016283506 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v57) v1 ==== 81+0+30363 (secure 0 0 0) 0x7f3584000df0 con 0x7f358c0bb8e0 2021-02-10T11:25:19.298 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:25:19.298 INFO:teuthology.orchestra.run.gibba002.stdout:{ 2021-02-10T11:25:19.298 INFO:teuthology.orchestra.run.gibba002.stdout: "epoch": 57, 2021-02-10T11:25:19.298 INFO:teuthology.orchestra.run.gibba002.stdout: "fsid": "68db30ea-dbce-4aef-a6df-f01c6fea0513", 2021-02-10T11:25:19.298 INFO:teuthology.orchestra.run.gibba002.stdout: "created": "2021-02-10 11:03:06.331557", 2021-02-10T11:25:19.298 INFO:teuthology.orchestra.run.gibba002.stdout: "modified": "2021-02-10 11:24:11.200509", 2021-02-10T11:25:19.298 INFO:teuthology.orchestra.run.gibba002.stdout: "last_up_change": "2021-02-10 11:03:32.690562", 2021-02-10T11:25:19.298 INFO:teuthology.orchestra.run.gibba002.stdout: "last_in_change": "2021-02-10 11:03:32.690562", 2021-02-10T11:25:19.298 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2021-02-10T11:25:19.299 INFO:teuthology.orchestra.run.gibba002.stdout: "flags_num": 5799936, 2021-02-10T11:25:19.299 INFO:teuthology.orchestra.run.gibba002.stdout: "flags_set": [ 2021-02-10T11:25:19.299 INFO:teuthology.orchestra.run.gibba002.stdout: "pglog_hardlimit", 2021-02-10T11:25:19.299 INFO:teuthology.orchestra.run.gibba002.stdout: "purged_snapdirs", 2021-02-10T11:25:19.299 INFO:teuthology.orchestra.run.gibba002.stdout: "recovery_deletes", 2021-02-10T11:25:19.299 INFO:teuthology.orchestra.run.gibba002.stdout: "sortbitwise" 2021-02-10T11:25:19.299 INFO:teuthology.orchestra.run.gibba002.stdout: ], 2021-02-10T11:25:19.299 INFO:teuthology.orchestra.run.gibba002.stdout: "crush_version": 4, 2021-02-10T11:25:19.299 INFO:teuthology.orchestra.run.gibba002.stdout: "full_ratio": 0.94999998807907104, 2021-02-10T11:25:19.299 INFO:teuthology.orchestra.run.gibba002.stdout: "backfillfull_ratio": 0.89999997615814209, 2021-02-10T11:25:19.299 INFO:teuthology.orchestra.run.gibba002.stdout: "nearfull_ratio": 0.85000002384185791, 2021-02-10T11:25:19.299 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_snapshot": "", 2021-02-10T11:25:19.300 INFO:teuthology.orchestra.run.gibba002.stdout: "pool_max": 11, 2021-02-10T11:25:19.300 INFO:teuthology.orchestra.run.gibba002.stdout: "max_osd": 8, 2021-02-10T11:25:19.300 INFO:teuthology.orchestra.run.gibba002.stdout: "require_min_compat_client": "jewel", 2021-02-10T11:25:19.300 INFO:teuthology.orchestra.run.gibba002.stdout: "min_compat_client": "jewel", 2021-02-10T11:25:19.300 INFO:teuthology.orchestra.run.gibba002.stdout: "require_osd_release": "nautilus", 2021-02-10T11:25:19.300 INFO:teuthology.orchestra.run.gibba002.stdout: "pools": [ 2021-02-10T11:25:19.300 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:25:19.300 INFO:teuthology.orchestra.run.gibba002.stdout: "pool": 1, 2021-02-10T11:25:19.300 INFO:teuthology.orchestra.run.gibba002.stdout: "pool_name": "rbd", 2021-02-10T11:25:19.300 INFO:teuthology.orchestra.run.gibba002.stdout: "create_time": "2021-02-10 11:03:37.322519", 2021-02-10T11:25:19.300 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 1, 2021-02-10T11:25:19.300 INFO:teuthology.orchestra.run.gibba002.stdout: "flags_names": "hashpspool", 2021-02-10T11:25:19.300 INFO:teuthology.orchestra.run.gibba002.stdout: "type": 1, 2021-02-10T11:25:19.301 INFO:teuthology.orchestra.run.gibba002.stdout: "size": 2, 2021-02-10T11:25:19.301 INFO:teuthology.orchestra.run.gibba002.stdout: "min_size": 1, 2021-02-10T11:25:19.301 INFO:teuthology.orchestra.run.gibba002.stdout: "crush_rule": 0, 2021-02-10T11:25:19.301 INFO:teuthology.orchestra.run.gibba002.stdout: "object_hash": 2, 2021-02-10T11:25:19.301 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:25:19.301 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num": 8, 2021-02-10T11:25:19.301 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_placement_num": 8, 2021-02-10T11:25:19.301 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_placement_num_target": 8, 2021-02-10T11:25:19.301 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num_target": 8, 2021-02-10T11:25:19.301 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num_pending": 8, 2021-02-10T11:25:19.301 INFO:teuthology.orchestra.run.gibba002.stdout: "last_pg_merge_meta": { 2021-02-10T11:25:19.301 INFO:teuthology.orchestra.run.gibba002.stdout: "source_pgid": "0.0", 2021-02-10T11:25:19.301 INFO:teuthology.orchestra.run.gibba002.stdout: "ready_epoch": 0, 2021-02-10T11:25:19.302 INFO:teuthology.orchestra.run.gibba002.stdout: "last_epoch_started": 0, 2021-02-10T11:25:19.302 INFO:teuthology.orchestra.run.gibba002.stdout: "last_epoch_clean": 0, 2021-02-10T11:25:19.302 INFO:teuthology.orchestra.run.gibba002.stdout: "source_version": "0'0", 2021-02-10T11:25:19.302 INFO:teuthology.orchestra.run.gibba002.stdout: "target_version": "0'0" 2021-02-10T11:25:19.302 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:25:19.302 INFO:teuthology.orchestra.run.gibba002.stdout: "last_change": "16", 2021-02-10T11:25:19.302 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend": "0", 2021-02-10T11:25:19.302 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:25:19.302 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:25:19.302 INFO:teuthology.orchestra.run.gibba002.stdout: "auid": 0, 2021-02-10T11:25:19.302 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:25:19.302 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_seq": 0, 2021-02-10T11:25:19.302 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_epoch": 0, 2021-02-10T11:25:19.303 INFO:teuthology.orchestra.run.gibba002.stdout: "pool_snaps": [], 2021-02-10T11:25:19.303 INFO:teuthology.orchestra.run.gibba002.stdout: "removed_snaps": "[]", 2021-02-10T11:25:19.303 INFO:teuthology.orchestra.run.gibba002.stdout: "quota_max_bytes": 0, 2021-02-10T11:25:19.303 INFO:teuthology.orchestra.run.gibba002.stdout: "quota_max_objects": 0, 2021-02-10T11:25:19.303 INFO:teuthology.orchestra.run.gibba002.stdout: "tiers": [], 2021-02-10T11:25:19.303 INFO:teuthology.orchestra.run.gibba002.stdout: "tier_of": -1, 2021-02-10T11:25:19.303 INFO:teuthology.orchestra.run.gibba002.stdout: "read_tier": -1, 2021-02-10T11:25:19.303 INFO:teuthology.orchestra.run.gibba002.stdout: "write_tier": -1, 2021-02-10T11:25:19.303 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_mode": "none", 2021-02-10T11:25:19.303 INFO:teuthology.orchestra.run.gibba002.stdout: "target_max_bytes": 0, 2021-02-10T11:25:19.303 INFO:teuthology.orchestra.run.gibba002.stdout: "target_max_objects": 0, 2021-02-10T11:25:19.303 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:25:19.303 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:25:19.303 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:25:19.304 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_min_flush_age": 0, 2021-02-10T11:25:19.304 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_min_evict_age": 0, 2021-02-10T11:25:19.304 INFO:teuthology.orchestra.run.gibba002.stdout: "erasure_code_profile": "", 2021-02-10T11:25:19.304 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_params": { 2021-02-10T11:25:19.304 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "none" 2021-02-10T11:25:19.304 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:25:19.304 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_period": 0, 2021-02-10T11:25:19.304 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_count": 0, 2021-02-10T11:25:19.304 INFO:teuthology.orchestra.run.gibba002.stdout: "use_gmt_hitset": true, 2021-02-10T11:25:19.304 INFO:teuthology.orchestra.run.gibba002.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:25:19.304 INFO:teuthology.orchestra.run.gibba002.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:25:19.304 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:25:19.304 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:25:19.305 INFO:teuthology.orchestra.run.gibba002.stdout: "grade_table": [], 2021-02-10T11:25:19.305 INFO:teuthology.orchestra.run.gibba002.stdout: "stripe_width": 0, 2021-02-10T11:25:19.305 INFO:teuthology.orchestra.run.gibba002.stdout: "expected_num_objects": 0, 2021-02-10T11:25:19.305 INFO:teuthology.orchestra.run.gibba002.stdout: "fast_read": false, 2021-02-10T11:25:19.305 INFO:teuthology.orchestra.run.gibba002.stdout: "options": {}, 2021-02-10T11:25:19.305 INFO:teuthology.orchestra.run.gibba002.stdout: "application_metadata": { 2021-02-10T11:25:19.305 INFO:teuthology.orchestra.run.gibba002.stdout: "rbd": {} 2021-02-10T11:25:19.305 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:25:19.305 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:25:19.305 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:25:19.305 INFO:teuthology.orchestra.run.gibba002.stdout: "pool": 10, 2021-02-10T11:25:19.305 INFO:teuthology.orchestra.run.gibba002.stdout: "pool_name": "cephfs_metadata", 2021-02-10T11:25:19.305 INFO:teuthology.orchestra.run.gibba002.stdout: "create_time": "2021-02-10 11:24:08.122188", 2021-02-10T11:25:19.305 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 1, 2021-02-10T11:25:19.306 INFO:teuthology.orchestra.run.gibba002.stdout: "flags_names": "hashpspool", 2021-02-10T11:25:19.306 INFO:teuthology.orchestra.run.gibba002.stdout: "type": 1, 2021-02-10T11:25:19.306 INFO:teuthology.orchestra.run.gibba002.stdout: "size": 2, 2021-02-10T11:25:19.306 INFO:teuthology.orchestra.run.gibba002.stdout: "min_size": 1, 2021-02-10T11:25:19.306 INFO:teuthology.orchestra.run.gibba002.stdout: "crush_rule": 0, 2021-02-10T11:25:19.306 INFO:teuthology.orchestra.run.gibba002.stdout: "object_hash": 2, 2021-02-10T11:25:19.306 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:25:19.306 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num": 8, 2021-02-10T11:25:19.306 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_placement_num": 8, 2021-02-10T11:25:19.306 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_placement_num_target": 8, 2021-02-10T11:25:19.306 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num_target": 8, 2021-02-10T11:25:19.306 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num_pending": 8, 2021-02-10T11:25:19.306 INFO:teuthology.orchestra.run.gibba002.stdout: "last_pg_merge_meta": { 2021-02-10T11:25:19.307 INFO:teuthology.orchestra.run.gibba002.stdout: "source_pgid": "0.0", 2021-02-10T11:25:19.307 INFO:teuthology.orchestra.run.gibba002.stdout: "ready_epoch": 0, 2021-02-10T11:25:19.307 INFO:teuthology.orchestra.run.gibba002.stdout: "last_epoch_started": 0, 2021-02-10T11:25:19.307 INFO:teuthology.orchestra.run.gibba002.stdout: "last_epoch_clean": 0, 2021-02-10T11:25:19.307 INFO:teuthology.orchestra.run.gibba002.stdout: "source_version": "0'0", 2021-02-10T11:25:19.307 INFO:teuthology.orchestra.run.gibba002.stdout: "target_version": "0'0" 2021-02-10T11:25:19.307 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:25:19.307 INFO:teuthology.orchestra.run.gibba002.stdout: "last_change": "56", 2021-02-10T11:25:19.307 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend": "0", 2021-02-10T11:25:19.307 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:25:19.307 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:25:19.307 INFO:teuthology.orchestra.run.gibba002.stdout: "auid": 0, 2021-02-10T11:25:19.307 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:25:19.308 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_seq": 0, 2021-02-10T11:25:19.308 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_epoch": 0, 2021-02-10T11:25:19.308 INFO:teuthology.orchestra.run.gibba002.stdout: "pool_snaps": [], 2021-02-10T11:25:19.308 INFO:teuthology.orchestra.run.gibba002.stdout: "removed_snaps": "[]", 2021-02-10T11:25:19.308 INFO:teuthology.orchestra.run.gibba002.stdout: "quota_max_bytes": 0, 2021-02-10T11:25:19.308 INFO:teuthology.orchestra.run.gibba002.stdout: "quota_max_objects": 0, 2021-02-10T11:25:19.308 INFO:teuthology.orchestra.run.gibba002.stdout: "tiers": [], 2021-02-10T11:25:19.308 INFO:teuthology.orchestra.run.gibba002.stdout: "tier_of": -1, 2021-02-10T11:25:19.308 INFO:teuthology.orchestra.run.gibba002.stdout: "read_tier": -1, 2021-02-10T11:25:19.308 INFO:teuthology.orchestra.run.gibba002.stdout: "write_tier": -1, 2021-02-10T11:25:19.308 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_mode": "none", 2021-02-10T11:25:19.308 INFO:teuthology.orchestra.run.gibba002.stdout: "target_max_bytes": 0, 2021-02-10T11:25:19.308 INFO:teuthology.orchestra.run.gibba002.stdout: "target_max_objects": 0, 2021-02-10T11:25:19.308 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:25:19.309 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:25:19.309 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:25:19.309 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_min_flush_age": 0, 2021-02-10T11:25:19.309 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_min_evict_age": 0, 2021-02-10T11:25:19.309 INFO:teuthology.orchestra.run.gibba002.stdout: "erasure_code_profile": "", 2021-02-10T11:25:19.309 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_params": { 2021-02-10T11:25:19.309 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "none" 2021-02-10T11:25:19.309 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:25:19.309 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_period": 0, 2021-02-10T11:25:19.309 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_count": 0, 2021-02-10T11:25:19.309 INFO:teuthology.orchestra.run.gibba002.stdout: "use_gmt_hitset": true, 2021-02-10T11:25:19.309 INFO:teuthology.orchestra.run.gibba002.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:25:19.309 INFO:teuthology.orchestra.run.gibba002.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:25:19.310 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:25:19.310 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:25:19.310 INFO:teuthology.orchestra.run.gibba002.stdout: "grade_table": [], 2021-02-10T11:25:19.310 INFO:teuthology.orchestra.run.gibba002.stdout: "stripe_width": 0, 2021-02-10T11:25:19.310 INFO:teuthology.orchestra.run.gibba002.stdout: "expected_num_objects": 0, 2021-02-10T11:25:19.310 INFO:teuthology.orchestra.run.gibba002.stdout: "fast_read": false, 2021-02-10T11:25:19.310 INFO:teuthology.orchestra.run.gibba002.stdout: "options": { 2021-02-10T11:25:19.310 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_autoscale_bias": 4, 2021-02-10T11:25:19.310 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num_min": 16, 2021-02-10T11:25:19.310 INFO:teuthology.orchestra.run.gibba002.stdout: "recovery_priority": 5 2021-02-10T11:25:19.310 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:25:19.310 INFO:teuthology.orchestra.run.gibba002.stdout: "application_metadata": { 2021-02-10T11:25:19.310 INFO:teuthology.orchestra.run.gibba002.stdout: "cephfs": { 2021-02-10T11:25:19.311 INFO:teuthology.orchestra.run.gibba002.stdout: "metadata": "cephfs" 2021-02-10T11:25:19.311 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:25:19.311 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:25:19.311 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:25:19.311 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:25:19.311 INFO:teuthology.orchestra.run.gibba002.stdout: "pool": 11, 2021-02-10T11:25:19.311 INFO:teuthology.orchestra.run.gibba002.stdout: "pool_name": "cephfs_data", 2021-02-10T11:25:19.311 INFO:teuthology.orchestra.run.gibba002.stdout: "create_time": "2021-02-10 11:24:09.191606", 2021-02-10T11:25:19.311 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 1, 2021-02-10T11:25:19.311 INFO:teuthology.orchestra.run.gibba002.stdout: "flags_names": "hashpspool", 2021-02-10T11:25:19.311 INFO:teuthology.orchestra.run.gibba002.stdout: "type": 1, 2021-02-10T11:25:19.311 INFO:teuthology.orchestra.run.gibba002.stdout: "size": 2, 2021-02-10T11:25:19.311 INFO:teuthology.orchestra.run.gibba002.stdout: "min_size": 1, 2021-02-10T11:25:19.311 INFO:teuthology.orchestra.run.gibba002.stdout: "crush_rule": 0, 2021-02-10T11:25:19.312 INFO:teuthology.orchestra.run.gibba002.stdout: "object_hash": 2, 2021-02-10T11:25:19.312 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:25:19.312 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num": 8, 2021-02-10T11:25:19.312 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_placement_num": 8, 2021-02-10T11:25:19.312 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_placement_num_target": 8, 2021-02-10T11:25:19.312 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num_target": 8, 2021-02-10T11:25:19.312 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num_pending": 8, 2021-02-10T11:25:19.312 INFO:teuthology.orchestra.run.gibba002.stdout: "last_pg_merge_meta": { 2021-02-10T11:25:19.312 INFO:teuthology.orchestra.run.gibba002.stdout: "source_pgid": "0.0", 2021-02-10T11:25:19.312 INFO:teuthology.orchestra.run.gibba002.stdout: "ready_epoch": 0, 2021-02-10T11:25:19.312 INFO:teuthology.orchestra.run.gibba002.stdout: "last_epoch_started": 0, 2021-02-10T11:25:19.312 INFO:teuthology.orchestra.run.gibba002.stdout: "last_epoch_clean": 0, 2021-02-10T11:25:19.312 INFO:teuthology.orchestra.run.gibba002.stdout: "source_version": "0'0", 2021-02-10T11:25:19.312 INFO:teuthology.orchestra.run.gibba002.stdout: "target_version": "0'0" 2021-02-10T11:25:19.313 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:25:19.313 INFO:teuthology.orchestra.run.gibba002.stdout: "last_change": "57", 2021-02-10T11:25:19.313 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend": "0", 2021-02-10T11:25:19.313 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:25:19.313 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:25:19.313 INFO:teuthology.orchestra.run.gibba002.stdout: "auid": 0, 2021-02-10T11:25:19.313 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:25:19.313 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_seq": 0, 2021-02-10T11:25:19.313 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_epoch": 0, 2021-02-10T11:25:19.313 INFO:teuthology.orchestra.run.gibba002.stdout: "pool_snaps": [], 2021-02-10T11:25:19.313 INFO:teuthology.orchestra.run.gibba002.stdout: "removed_snaps": "[]", 2021-02-10T11:25:19.313 INFO:teuthology.orchestra.run.gibba002.stdout: "quota_max_bytes": 0, 2021-02-10T11:25:19.313 INFO:teuthology.orchestra.run.gibba002.stdout: "quota_max_objects": 0, 2021-02-10T11:25:19.314 INFO:teuthology.orchestra.run.gibba002.stdout: "tiers": [], 2021-02-10T11:25:19.314 INFO:teuthology.orchestra.run.gibba002.stdout: "tier_of": -1, 2021-02-10T11:25:19.314 INFO:teuthology.orchestra.run.gibba002.stdout: "read_tier": -1, 2021-02-10T11:25:19.314 INFO:teuthology.orchestra.run.gibba002.stdout: "write_tier": -1, 2021-02-10T11:25:19.314 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_mode": "none", 2021-02-10T11:25:19.314 INFO:teuthology.orchestra.run.gibba002.stdout: "target_max_bytes": 0, 2021-02-10T11:25:19.314 INFO:teuthology.orchestra.run.gibba002.stdout: "target_max_objects": 0, 2021-02-10T11:25:19.314 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:25:19.314 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:25:19.314 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:25:19.314 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_min_flush_age": 0, 2021-02-10T11:25:19.314 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_min_evict_age": 0, 2021-02-10T11:25:19.314 INFO:teuthology.orchestra.run.gibba002.stdout: "erasure_code_profile": "", 2021-02-10T11:25:19.315 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_params": { 2021-02-10T11:25:19.318 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "none" 2021-02-10T11:25:19.318 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:25:19.318 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_period": 0, 2021-02-10T11:25:19.318 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_count": 0, 2021-02-10T11:25:19.318 INFO:teuthology.orchestra.run.gibba002.stdout: "use_gmt_hitset": true, 2021-02-10T11:25:19.318 INFO:teuthology.orchestra.run.gibba002.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:25:19.319 INFO:teuthology.orchestra.run.gibba002.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:25:19.319 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:25:19.319 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:25:19.319 INFO:teuthology.orchestra.run.gibba002.stdout: "grade_table": [], 2021-02-10T11:25:19.319 INFO:teuthology.orchestra.run.gibba002.stdout: "stripe_width": 0, 2021-02-10T11:25:19.319 INFO:teuthology.orchestra.run.gibba002.stdout: "expected_num_objects": 0, 2021-02-10T11:25:19.319 INFO:teuthology.orchestra.run.gibba002.stdout: "fast_read": false, 2021-02-10T11:25:19.319 INFO:teuthology.orchestra.run.gibba002.stdout: "options": {}, 2021-02-10T11:25:19.319 INFO:teuthology.orchestra.run.gibba002.stdout: "application_metadata": { 2021-02-10T11:25:19.319 INFO:teuthology.orchestra.run.gibba002.stdout: "cephfs": { 2021-02-10T11:25:19.319 INFO:teuthology.orchestra.run.gibba002.stdout: "data": "cephfs" 2021-02-10T11:25:19.319 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:25:19.320 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:25:19.320 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:25:19.320 INFO:teuthology.orchestra.run.gibba002.stdout: ], 2021-02-10T11:25:19.320 INFO:teuthology.orchestra.run.gibba002.stdout: "osds": [ 2021-02-10T11:25:19.320 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:25:19.320 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 0, 2021-02-10T11:25:19.320 INFO:teuthology.orchestra.run.gibba002.stdout: "uuid": "be691b4d-a039-408a-af80-9d53f02b6f1f", 2021-02-10T11:25:19.320 INFO:teuthology.orchestra.run.gibba002.stdout: "up": 1, 2021-02-10T11:25:19.320 INFO:teuthology.orchestra.run.gibba002.stdout: "in": 1, 2021-02-10T11:25:19.320 INFO:teuthology.orchestra.run.gibba002.stdout: "weight": 1, 2021-02-10T11:25:19.320 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_affinity": 1, 2021-02-10T11:25:19.320 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_begin": 0, 2021-02-10T11:25:19.321 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_end": 0, 2021-02-10T11:25:19.321 INFO:teuthology.orchestra.run.gibba002.stdout: "up_from": 13, 2021-02-10T11:25:19.321 INFO:teuthology.orchestra.run.gibba002.stdout: "up_thru": 55, 2021-02-10T11:25:19.321 INFO:teuthology.orchestra.run.gibba002.stdout: "down_at": 0, 2021-02-10T11:25:19.321 INFO:teuthology.orchestra.run.gibba002.stdout: "lost_at": 0, 2021-02-10T11:25:19.321 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addrs": { 2021-02-10T11:25:19.321 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:25:19.321 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:25:19.321 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:25:19.321 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6818", 2021-02-10T11:25:19.321 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36088 2021-02-10T11:25:19.321 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:25:19.322 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:25:19.322 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:25:19.322 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6820", 2021-02-10T11:25:19.322 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36088 2021-02-10T11:25:19.322 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:25:19.322 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:25:19.322 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:25:19.322 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addrs": { 2021-02-10T11:25:19.322 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:25:19.322 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:25:19.322 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:25:19.322 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6822", 2021-02-10T11:25:19.323 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36088 2021-02-10T11:25:19.323 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:25:19.323 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:25:19.323 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:25:19.323 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6824", 2021-02-10T11:25:19.323 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36088 2021-02-10T11:25:19.323 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:25:19.323 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:25:19.323 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:25:19.323 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addrs": { 2021-02-10T11:25:19.323 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:25:19.323 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:25:19.323 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:25:19.323 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6830", 2021-02-10T11:25:19.324 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36088 2021-02-10T11:25:19.324 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:25:19.324 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:25:19.324 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:25:19.324 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6832", 2021-02-10T11:25:19.324 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36088 2021-02-10T11:25:19.324 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:25:19.324 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:25:19.324 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:25:19.324 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addrs": { 2021-02-10T11:25:19.324 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:25:19.324 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:25:19.324 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:25:19.325 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6826", 2021-02-10T11:25:19.325 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36088 2021-02-10T11:25:19.325 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:25:19.325 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:25:19.325 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:25:19.325 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6828", 2021-02-10T11:25:19.325 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36088 2021-02-10T11:25:19.325 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:25:19.325 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:25:19.325 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:25:19.325 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addr": "172.21.2.102:6820/36088", 2021-02-10T11:25:19.325 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addr": "172.21.2.102:6824/36088", 2021-02-10T11:25:19.325 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addr": "172.21.2.102:6832/36088", 2021-02-10T11:25:19.326 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addr": "172.21.2.102:6828/36088", 2021-02-10T11:25:19.326 INFO:teuthology.orchestra.run.gibba002.stdout: "state": [ 2021-02-10T11:25:19.326 INFO:teuthology.orchestra.run.gibba002.stdout: "exists", 2021-02-10T11:25:19.326 INFO:teuthology.orchestra.run.gibba002.stdout: "up" 2021-02-10T11:25:19.326 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:25:19.326 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:25:19.326 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:25:19.326 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 1, 2021-02-10T11:25:19.326 INFO:teuthology.orchestra.run.gibba002.stdout: "uuid": "64e3ccc8-9c69-4d7c-865e-af8e83822635", 2021-02-10T11:25:19.326 INFO:teuthology.orchestra.run.gibba002.stdout: "up": 1, 2021-02-10T11:25:19.326 INFO:teuthology.orchestra.run.gibba002.stdout: "in": 1, 2021-02-10T11:25:19.326 INFO:teuthology.orchestra.run.gibba002.stdout: "weight": 1, 2021-02-10T11:25:19.327 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_affinity": 1, 2021-02-10T11:25:19.327 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_begin": 0, 2021-02-10T11:25:19.327 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_end": 0, 2021-02-10T11:25:19.327 INFO:teuthology.orchestra.run.gibba002.stdout: "up_from": 13, 2021-02-10T11:25:19.327 INFO:teuthology.orchestra.run.gibba002.stdout: "up_thru": 55, 2021-02-10T11:25:19.327 INFO:teuthology.orchestra.run.gibba002.stdout: "down_at": 0, 2021-02-10T11:25:19.327 INFO:teuthology.orchestra.run.gibba002.stdout: "lost_at": 0, 2021-02-10T11:25:19.327 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addrs": { 2021-02-10T11:25:19.327 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:25:19.327 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:25:19.327 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:25:19.327 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6819", 2021-02-10T11:25:19.327 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36087 2021-02-10T11:25:19.328 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:25:19.328 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:25:19.328 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:25:19.328 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6821", 2021-02-10T11:25:19.328 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36087 2021-02-10T11:25:19.328 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:25:19.328 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:25:19.328 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:25:19.328 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addrs": { 2021-02-10T11:25:19.328 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:25:19.328 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:25:19.328 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:25:19.328 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6823", 2021-02-10T11:25:19.328 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36087 2021-02-10T11:25:19.329 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:25:19.329 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:25:19.329 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:25:19.329 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6825", 2021-02-10T11:25:19.329 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36087 2021-02-10T11:25:19.329 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:25:19.329 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:25:19.329 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:25:19.329 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addrs": { 2021-02-10T11:25:19.329 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:25:19.329 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:25:19.329 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:25:19.329 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6831", 2021-02-10T11:25:19.330 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36087 2021-02-10T11:25:19.330 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:25:19.330 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:25:19.330 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:25:19.330 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6833", 2021-02-10T11:25:19.330 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36087 2021-02-10T11:25:19.330 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:25:19.330 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:25:19.330 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:25:19.330 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addrs": { 2021-02-10T11:25:19.330 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:25:19.330 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:25:19.330 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:25:19.331 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6827", 2021-02-10T11:25:19.331 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36087 2021-02-10T11:25:19.331 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:25:19.331 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:25:19.331 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:25:19.331 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6829", 2021-02-10T11:25:19.331 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36087 2021-02-10T11:25:19.331 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:25:19.331 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:25:19.331 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:25:19.331 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addr": "172.21.2.102:6821/36087", 2021-02-10T11:25:19.331 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addr": "172.21.2.102:6825/36087", 2021-02-10T11:25:19.331 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addr": "172.21.2.102:6833/36087", 2021-02-10T11:25:19.331 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addr": "172.21.2.102:6829/36087", 2021-02-10T11:25:19.332 INFO:teuthology.orchestra.run.gibba002.stdout: "state": [ 2021-02-10T11:25:19.332 INFO:teuthology.orchestra.run.gibba002.stdout: "exists", 2021-02-10T11:25:19.332 INFO:teuthology.orchestra.run.gibba002.stdout: "up" 2021-02-10T11:25:19.332 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:25:19.332 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:25:19.332 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:25:19.332 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 2, 2021-02-10T11:25:19.332 INFO:teuthology.orchestra.run.gibba002.stdout: "uuid": "153700af-747f-481f-bdf4-d6072d3a842f", 2021-02-10T11:25:19.332 INFO:teuthology.orchestra.run.gibba002.stdout: "up": 1, 2021-02-10T11:25:19.332 INFO:teuthology.orchestra.run.gibba002.stdout: "in": 1, 2021-02-10T11:25:19.332 INFO:teuthology.orchestra.run.gibba002.stdout: "weight": 1, 2021-02-10T11:25:19.332 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_affinity": 1, 2021-02-10T11:25:19.332 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_begin": 0, 2021-02-10T11:25:19.333 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_end": 0, 2021-02-10T11:25:19.333 INFO:teuthology.orchestra.run.gibba002.stdout: "up_from": 13, 2021-02-10T11:25:19.333 INFO:teuthology.orchestra.run.gibba002.stdout: "up_thru": 55, 2021-02-10T11:25:19.333 INFO:teuthology.orchestra.run.gibba002.stdout: "down_at": 0, 2021-02-10T11:25:19.333 INFO:teuthology.orchestra.run.gibba002.stdout: "lost_at": 0, 2021-02-10T11:25:19.333 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addrs": { 2021-02-10T11:25:19.333 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:25:19.333 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:25:19.333 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:25:19.333 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6810", 2021-02-10T11:25:19.333 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36089 2021-02-10T11:25:19.333 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:25:19.333 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:25:19.334 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:25:19.334 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6811", 2021-02-10T11:25:19.334 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36089 2021-02-10T11:25:19.334 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:25:19.334 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:25:19.334 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:25:19.334 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addrs": { 2021-02-10T11:25:19.334 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:25:19.334 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:25:19.334 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:25:19.334 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6812", 2021-02-10T11:25:19.334 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36089 2021-02-10T11:25:19.334 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:25:19.335 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:25:19.335 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:25:19.335 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6813", 2021-02-10T11:25:19.335 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36089 2021-02-10T11:25:19.335 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:25:19.335 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:25:19.335 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:25:19.335 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addrs": { 2021-02-10T11:25:19.335 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:25:19.335 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:25:19.335 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:25:19.335 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6816", 2021-02-10T11:25:19.336 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36089 2021-02-10T11:25:19.336 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:25:19.336 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:25:19.336 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:25:19.336 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6817", 2021-02-10T11:25:19.336 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36089 2021-02-10T11:25:19.336 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:25:19.336 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:25:19.336 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:25:19.336 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addrs": { 2021-02-10T11:25:19.336 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:25:19.336 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:25:19.336 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:25:19.336 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6814", 2021-02-10T11:25:19.337 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36089 2021-02-10T11:25:19.337 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:25:19.337 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:25:19.337 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:25:19.337 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6815", 2021-02-10T11:25:19.337 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36089 2021-02-10T11:25:19.337 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:25:19.337 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:25:19.337 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:25:19.337 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addr": "172.21.2.102:6811/36089", 2021-02-10T11:25:19.337 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addr": "172.21.2.102:6813/36089", 2021-02-10T11:25:19.337 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addr": "172.21.2.102:6817/36089", 2021-02-10T11:25:19.337 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addr": "172.21.2.102:6815/36089", 2021-02-10T11:25:19.337 INFO:teuthology.orchestra.run.gibba002.stdout: "state": [ 2021-02-10T11:25:19.338 INFO:teuthology.orchestra.run.gibba002.stdout: "exists", 2021-02-10T11:25:19.338 INFO:teuthology.orchestra.run.gibba002.stdout: "up" 2021-02-10T11:25:19.338 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:25:19.338 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:25:19.338 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:25:19.338 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 3, 2021-02-10T11:25:19.338 INFO:teuthology.orchestra.run.gibba002.stdout: "uuid": "cbc678f4-83a9-45e6-8830-2f7c232b1ea7", 2021-02-10T11:25:19.338 INFO:teuthology.orchestra.run.gibba002.stdout: "up": 1, 2021-02-10T11:25:19.338 INFO:teuthology.orchestra.run.gibba002.stdout: "in": 1, 2021-02-10T11:25:19.338 INFO:teuthology.orchestra.run.gibba002.stdout: "weight": 1, 2021-02-10T11:25:19.338 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_affinity": 1, 2021-02-10T11:25:19.338 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_begin": 0, 2021-02-10T11:25:19.339 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_end": 0, 2021-02-10T11:25:19.339 INFO:teuthology.orchestra.run.gibba002.stdout: "up_from": 13, 2021-02-10T11:25:19.339 INFO:teuthology.orchestra.run.gibba002.stdout: "up_thru": 46, 2021-02-10T11:25:19.339 INFO:teuthology.orchestra.run.gibba002.stdout: "down_at": 0, 2021-02-10T11:25:19.339 INFO:teuthology.orchestra.run.gibba002.stdout: "lost_at": 0, 2021-02-10T11:25:19.339 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addrs": { 2021-02-10T11:25:19.339 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:25:19.339 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:25:19.339 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:25:19.339 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6802", 2021-02-10T11:25:19.339 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36090 2021-02-10T11:25:19.339 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:25:19.339 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:25:19.340 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:25:19.340 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6803", 2021-02-10T11:25:19.340 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36090 2021-02-10T11:25:19.340 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:25:19.340 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:25:19.340 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:25:19.340 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addrs": { 2021-02-10T11:25:19.340 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:25:19.340 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:25:19.340 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:25:19.340 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6804", 2021-02-10T11:25:19.340 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36090 2021-02-10T11:25:19.341 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:25:19.341 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:25:19.341 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:25:19.341 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6805", 2021-02-10T11:25:19.341 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36090 2021-02-10T11:25:19.341 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:25:19.341 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:25:19.341 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:25:19.341 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addrs": { 2021-02-10T11:25:19.341 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:25:19.341 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:25:19.341 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:25:19.341 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6808", 2021-02-10T11:25:19.341 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36090 2021-02-10T11:25:19.342 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:25:19.342 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:25:19.342 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:25:19.342 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6809", 2021-02-10T11:25:19.342 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36090 2021-02-10T11:25:19.342 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:25:19.342 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:25:19.342 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:25:19.342 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addrs": { 2021-02-10T11:25:19.342 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:25:19.342 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:25:19.342 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:25:19.343 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6806", 2021-02-10T11:25:19.343 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36090 2021-02-10T11:25:19.343 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:25:19.343 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:25:19.343 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:25:19.343 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6807", 2021-02-10T11:25:19.343 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36090 2021-02-10T11:25:19.343 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:25:19.343 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:25:19.343 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:25:19.343 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addr": "172.21.2.102:6803/36090", 2021-02-10T11:25:19.343 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addr": "172.21.2.102:6805/36090", 2021-02-10T11:25:19.343 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addr": "172.21.2.102:6809/36090", 2021-02-10T11:25:19.343 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addr": "172.21.2.102:6807/36090", 2021-02-10T11:25:19.343 INFO:teuthology.orchestra.run.gibba002.stdout: "state": [ 2021-02-10T11:25:19.344 INFO:teuthology.orchestra.run.gibba002.stdout: "exists", 2021-02-10T11:25:19.344 INFO:teuthology.orchestra.run.gibba002.stdout: "up" 2021-02-10T11:25:19.344 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:25:19.344 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:25:19.344 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:25:19.344 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 4, 2021-02-10T11:25:19.344 INFO:teuthology.orchestra.run.gibba002.stdout: "uuid": "c01e4008-b258-4f10-afd2-9e7f37214351", 2021-02-10T11:25:19.344 INFO:teuthology.orchestra.run.gibba002.stdout: "up": 1, 2021-02-10T11:25:19.344 INFO:teuthology.orchestra.run.gibba002.stdout: "in": 1, 2021-02-10T11:25:19.344 INFO:teuthology.orchestra.run.gibba002.stdout: "weight": 1, 2021-02-10T11:25:19.344 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_affinity": 1, 2021-02-10T11:25:19.344 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_begin": 0, 2021-02-10T11:25:19.344 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_end": 0, 2021-02-10T11:25:19.344 INFO:teuthology.orchestra.run.gibba002.stdout: "up_from": 13, 2021-02-10T11:25:19.345 INFO:teuthology.orchestra.run.gibba002.stdout: "up_thru": 53, 2021-02-10T11:25:19.345 INFO:teuthology.orchestra.run.gibba002.stdout: "down_at": 0, 2021-02-10T11:25:19.345 INFO:teuthology.orchestra.run.gibba002.stdout: "lost_at": 0, 2021-02-10T11:25:19.345 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addrs": { 2021-02-10T11:25:19.345 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:25:19.345 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:25:19.345 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:25:19.345 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6808", 2021-02-10T11:25:19.345 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36368 2021-02-10T11:25:19.345 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:25:19.345 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:25:19.345 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:25:19.346 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6811", 2021-02-10T11:25:19.346 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36368 2021-02-10T11:25:19.346 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:25:19.346 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:25:19.346 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:25:19.346 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addrs": { 2021-02-10T11:25:19.346 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:25:19.346 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:25:19.346 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:25:19.346 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6814", 2021-02-10T11:25:19.346 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36368 2021-02-10T11:25:19.346 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:25:19.347 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:25:19.347 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:25:19.347 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6818", 2021-02-10T11:25:19.347 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36368 2021-02-10T11:25:19.347 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:25:19.347 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:25:19.347 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:25:19.347 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addrs": { 2021-02-10T11:25:19.347 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:25:19.347 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:25:19.347 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:25:19.347 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6826", 2021-02-10T11:25:19.347 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36368 2021-02-10T11:25:19.347 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:25:19.348 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:25:19.348 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:25:19.348 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6828", 2021-02-10T11:25:19.348 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36368 2021-02-10T11:25:19.348 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:25:19.348 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:25:19.348 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:25:19.348 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addrs": { 2021-02-10T11:25:19.348 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:25:19.348 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:25:19.348 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:25:19.348 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6820", 2021-02-10T11:25:19.348 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36368 2021-02-10T11:25:19.349 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:25:19.349 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:25:19.349 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:25:19.349 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6824", 2021-02-10T11:25:19.349 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36368 2021-02-10T11:25:19.349 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:25:19.349 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:25:19.349 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:25:19.349 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addr": "172.21.2.113:6811/36368", 2021-02-10T11:25:19.349 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addr": "172.21.2.113:6818/36368", 2021-02-10T11:25:19.349 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addr": "172.21.2.113:6828/36368", 2021-02-10T11:25:19.349 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addr": "172.21.2.113:6824/36368", 2021-02-10T11:25:19.349 INFO:teuthology.orchestra.run.gibba002.stdout: "state": [ 2021-02-10T11:25:19.350 INFO:teuthology.orchestra.run.gibba002.stdout: "exists", 2021-02-10T11:25:19.350 INFO:teuthology.orchestra.run.gibba002.stdout: "up" 2021-02-10T11:25:19.350 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:25:19.350 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:25:19.350 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:25:19.350 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 5, 2021-02-10T11:25:19.350 INFO:teuthology.orchestra.run.gibba002.stdout: "uuid": "551f6007-daf8-4a98-8ee0-c2b16bcccb5e", 2021-02-10T11:25:19.350 INFO:teuthology.orchestra.run.gibba002.stdout: "up": 1, 2021-02-10T11:25:19.350 INFO:teuthology.orchestra.run.gibba002.stdout: "in": 1, 2021-02-10T11:25:19.350 INFO:teuthology.orchestra.run.gibba002.stdout: "weight": 1, 2021-02-10T11:25:19.350 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_affinity": 1, 2021-02-10T11:25:19.350 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_begin": 0, 2021-02-10T11:25:19.351 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_end": 0, 2021-02-10T11:25:19.351 INFO:teuthology.orchestra.run.gibba002.stdout: "up_from": 13, 2021-02-10T11:25:19.351 INFO:teuthology.orchestra.run.gibba002.stdout: "up_thru": 55, 2021-02-10T11:25:19.351 INFO:teuthology.orchestra.run.gibba002.stdout: "down_at": 0, 2021-02-10T11:25:19.351 INFO:teuthology.orchestra.run.gibba002.stdout: "lost_at": 0, 2021-02-10T11:25:19.351 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addrs": { 2021-02-10T11:25:19.351 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:25:19.351 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:25:19.351 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:25:19.351 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6802", 2021-02-10T11:25:19.351 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36366 2021-02-10T11:25:19.351 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:25:19.351 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:25:19.351 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:25:19.352 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6805", 2021-02-10T11:25:19.352 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36366 2021-02-10T11:25:19.352 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:25:19.352 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:25:19.352 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:25:19.352 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addrs": { 2021-02-10T11:25:19.352 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:25:19.352 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:25:19.352 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:25:19.352 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6807", 2021-02-10T11:25:19.352 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36366 2021-02-10T11:25:19.352 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:25:19.352 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:25:19.353 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:25:19.353 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6810", 2021-02-10T11:25:19.353 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36366 2021-02-10T11:25:19.353 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:25:19.353 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:25:19.353 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:25:19.353 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addrs": { 2021-02-10T11:25:19.353 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:25:19.353 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:25:19.353 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:25:19.353 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6819", 2021-02-10T11:25:19.353 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36366 2021-02-10T11:25:19.354 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:25:19.354 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:25:19.354 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:25:19.354 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6822", 2021-02-10T11:25:19.354 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36366 2021-02-10T11:25:19.354 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:25:19.354 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:25:19.354 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:25:19.354 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addrs": { 2021-02-10T11:25:19.354 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:25:19.354 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:25:19.354 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:25:19.354 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6812", 2021-02-10T11:25:19.355 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36366 2021-02-10T11:25:19.355 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:25:19.355 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:25:19.355 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:25:19.355 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6815", 2021-02-10T11:25:19.355 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36366 2021-02-10T11:25:19.355 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:25:19.355 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:25:19.355 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:25:19.355 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addr": "172.21.2.113:6805/36366", 2021-02-10T11:25:19.355 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addr": "172.21.2.113:6810/36366", 2021-02-10T11:25:19.355 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addr": "172.21.2.113:6822/36366", 2021-02-10T11:25:19.355 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addr": "172.21.2.113:6815/36366", 2021-02-10T11:25:19.356 INFO:teuthology.orchestra.run.gibba002.stdout: "state": [ 2021-02-10T11:25:19.356 INFO:teuthology.orchestra.run.gibba002.stdout: "exists", 2021-02-10T11:25:19.356 INFO:teuthology.orchestra.run.gibba002.stdout: "up" 2021-02-10T11:25:19.356 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:25:19.356 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:25:19.356 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:25:19.356 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 6, 2021-02-10T11:25:19.356 INFO:teuthology.orchestra.run.gibba002.stdout: "uuid": "bf67c040-0e50-48be-b102-c3943bc9ebef", 2021-02-10T11:25:19.356 INFO:teuthology.orchestra.run.gibba002.stdout: "up": 1, 2021-02-10T11:25:19.356 INFO:teuthology.orchestra.run.gibba002.stdout: "in": 1, 2021-02-10T11:25:19.356 INFO:teuthology.orchestra.run.gibba002.stdout: "weight": 1, 2021-02-10T11:25:19.356 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_affinity": 1, 2021-02-10T11:25:19.356 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_begin": 0, 2021-02-10T11:25:19.357 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_end": 0, 2021-02-10T11:25:19.357 INFO:teuthology.orchestra.run.gibba002.stdout: "up_from": 13, 2021-02-10T11:25:19.357 INFO:teuthology.orchestra.run.gibba002.stdout: "up_thru": 55, 2021-02-10T11:25:19.357 INFO:teuthology.orchestra.run.gibba002.stdout: "down_at": 0, 2021-02-10T11:25:19.357 INFO:teuthology.orchestra.run.gibba002.stdout: "lost_at": 0, 2021-02-10T11:25:19.357 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addrs": { 2021-02-10T11:25:19.357 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:25:19.357 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:25:19.357 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:25:19.357 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6800", 2021-02-10T11:25:19.357 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36367 2021-02-10T11:25:19.357 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:25:19.357 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:25:19.358 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:25:19.358 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6801", 2021-02-10T11:25:19.358 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36367 2021-02-10T11:25:19.358 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:25:19.358 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:25:19.358 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:25:19.358 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addrs": { 2021-02-10T11:25:19.358 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:25:19.358 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:25:19.358 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:25:19.358 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6803", 2021-02-10T11:25:19.358 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36367 2021-02-10T11:25:19.358 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:25:19.358 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:25:19.359 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:25:19.359 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6804", 2021-02-10T11:25:19.359 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36367 2021-02-10T11:25:19.359 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:25:19.359 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:25:19.359 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:25:19.359 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addrs": { 2021-02-10T11:25:19.359 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:25:19.359 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:25:19.359 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:25:19.359 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6813", 2021-02-10T11:25:19.359 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36367 2021-02-10T11:25:19.359 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:25:19.360 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:25:19.360 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:25:19.360 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6816", 2021-02-10T11:25:19.360 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36367 2021-02-10T11:25:19.360 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:25:19.360 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:25:19.360 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:25:19.360 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addrs": { 2021-02-10T11:25:19.360 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:25:19.360 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:25:19.360 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:25:19.360 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6806", 2021-02-10T11:25:19.360 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36367 2021-02-10T11:25:19.361 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:25:19.361 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:25:19.361 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:25:19.361 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6809", 2021-02-10T11:25:19.361 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36367 2021-02-10T11:25:19.361 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:25:19.361 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:25:19.361 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:25:19.361 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addr": "172.21.2.113:6801/36367", 2021-02-10T11:25:19.361 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addr": "172.21.2.113:6804/36367", 2021-02-10T11:25:19.361 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addr": "172.21.2.113:6816/36367", 2021-02-10T11:25:19.361 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addr": "172.21.2.113:6809/36367", 2021-02-10T11:25:19.362 INFO:teuthology.orchestra.run.gibba002.stdout: "state": [ 2021-02-10T11:25:19.362 INFO:teuthology.orchestra.run.gibba002.stdout: "exists", 2021-02-10T11:25:19.362 INFO:teuthology.orchestra.run.gibba002.stdout: "up" 2021-02-10T11:25:19.362 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:25:19.362 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:25:19.362 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:25:19.362 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 7, 2021-02-10T11:25:19.362 INFO:teuthology.orchestra.run.gibba002.stdout: "uuid": "f5461f80-018d-428f-be89-2eb2bcbe7bb6", 2021-02-10T11:25:19.362 INFO:teuthology.orchestra.run.gibba002.stdout: "up": 1, 2021-02-10T11:25:19.362 INFO:teuthology.orchestra.run.gibba002.stdout: "in": 1, 2021-02-10T11:25:19.362 INFO:teuthology.orchestra.run.gibba002.stdout: "weight": 1, 2021-02-10T11:25:19.362 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_affinity": 1, 2021-02-10T11:25:19.362 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_begin": 0, 2021-02-10T11:25:19.363 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_end": 0, 2021-02-10T11:25:19.363 INFO:teuthology.orchestra.run.gibba002.stdout: "up_from": 13, 2021-02-10T11:25:19.363 INFO:teuthology.orchestra.run.gibba002.stdout: "up_thru": 55, 2021-02-10T11:25:19.363 INFO:teuthology.orchestra.run.gibba002.stdout: "down_at": 0, 2021-02-10T11:25:19.363 INFO:teuthology.orchestra.run.gibba002.stdout: "lost_at": 0, 2021-02-10T11:25:19.363 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addrs": { 2021-02-10T11:25:19.363 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:25:19.363 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:25:19.363 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:25:19.363 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6817", 2021-02-10T11:25:19.363 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36365 2021-02-10T11:25:19.363 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:25:19.363 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:25:19.364 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:25:19.364 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6821", 2021-02-10T11:25:19.364 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36365 2021-02-10T11:25:19.364 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:25:19.364 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:25:19.364 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:25:19.364 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addrs": { 2021-02-10T11:25:19.364 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:25:19.364 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:25:19.364 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:25:19.364 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6823", 2021-02-10T11:25:19.364 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36365 2021-02-10T11:25:19.364 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:25:19.365 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:25:19.365 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:25:19.365 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6825", 2021-02-10T11:25:19.365 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36365 2021-02-10T11:25:19.365 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:25:19.365 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:25:19.365 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:25:19.365 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addrs": { 2021-02-10T11:25:19.365 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:25:19.365 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:25:19.365 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:25:19.365 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6830", 2021-02-10T11:25:19.365 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36365 2021-02-10T11:25:19.366 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:25:19.366 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:25:19.366 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:25:19.366 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6831", 2021-02-10T11:25:19.366 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36365 2021-02-10T11:25:19.366 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:25:19.366 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:25:19.366 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:25:19.366 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addrs": { 2021-02-10T11:25:19.366 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:25:19.366 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:25:19.366 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:25:19.366 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6827", 2021-02-10T11:25:19.367 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36365 2021-02-10T11:25:19.367 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:25:19.367 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:25:19.367 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:25:19.367 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6829", 2021-02-10T11:25:19.367 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36365 2021-02-10T11:25:19.367 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:25:19.367 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:25:19.367 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:25:19.367 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addr": "172.21.2.113:6821/36365", 2021-02-10T11:25:19.367 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addr": "172.21.2.113:6825/36365", 2021-02-10T11:25:19.367 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addr": "172.21.2.113:6831/36365", 2021-02-10T11:25:19.367 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addr": "172.21.2.113:6829/36365", 2021-02-10T11:25:19.368 INFO:teuthology.orchestra.run.gibba002.stdout: "state": [ 2021-02-10T11:25:19.368 INFO:teuthology.orchestra.run.gibba002.stdout: "exists", 2021-02-10T11:25:19.368 INFO:teuthology.orchestra.run.gibba002.stdout: "up" 2021-02-10T11:25:19.368 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:25:19.368 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:25:19.368 INFO:teuthology.orchestra.run.gibba002.stdout: ], 2021-02-10T11:25:19.368 INFO:teuthology.orchestra.run.gibba002.stdout: "osd_xinfo": [ 2021-02-10T11:25:19.368 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:25:19.368 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 0, 2021-02-10T11:25:19.368 INFO:teuthology.orchestra.run.gibba002.stdout: "down_stamp": "0.000000", 2021-02-10T11:25:19.368 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_probability": 0, 2021-02-10T11:25:19.368 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_interval": 0, 2021-02-10T11:25:19.368 INFO:teuthology.orchestra.run.gibba002.stdout: "features": 4611087854035861503, 2021-02-10T11:25:19.369 INFO:teuthology.orchestra.run.gibba002.stdout: "old_weight": 0 2021-02-10T11:25:19.369 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:25:19.369 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:25:19.369 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 1, 2021-02-10T11:25:19.369 INFO:teuthology.orchestra.run.gibba002.stdout: "down_stamp": "0.000000", 2021-02-10T11:25:19.369 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_probability": 0, 2021-02-10T11:25:19.369 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_interval": 0, 2021-02-10T11:25:19.369 INFO:teuthology.orchestra.run.gibba002.stdout: "features": 4611087854035861503, 2021-02-10T11:25:19.369 INFO:teuthology.orchestra.run.gibba002.stdout: "old_weight": 0 2021-02-10T11:25:19.369 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:25:19.369 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:25:19.369 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 2, 2021-02-10T11:25:19.369 INFO:teuthology.orchestra.run.gibba002.stdout: "down_stamp": "0.000000", 2021-02-10T11:25:19.370 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_probability": 0, 2021-02-10T11:25:19.370 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_interval": 0, 2021-02-10T11:25:19.370 INFO:teuthology.orchestra.run.gibba002.stdout: "features": 4611087854035861503, 2021-02-10T11:25:19.370 INFO:teuthology.orchestra.run.gibba002.stdout: "old_weight": 0 2021-02-10T11:25:19.370 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:25:19.370 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:25:19.370 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 3, 2021-02-10T11:25:19.370 INFO:teuthology.orchestra.run.gibba002.stdout: "down_stamp": "0.000000", 2021-02-10T11:25:19.370 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_probability": 0, 2021-02-10T11:25:19.370 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_interval": 0, 2021-02-10T11:25:19.370 INFO:teuthology.orchestra.run.gibba002.stdout: "features": 4611087854035861503, 2021-02-10T11:25:19.370 INFO:teuthology.orchestra.run.gibba002.stdout: "old_weight": 0 2021-02-10T11:25:19.371 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:25:19.371 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:25:19.371 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 4, 2021-02-10T11:25:19.371 INFO:teuthology.orchestra.run.gibba002.stdout: "down_stamp": "0.000000", 2021-02-10T11:25:19.371 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_probability": 0, 2021-02-10T11:25:19.371 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_interval": 0, 2021-02-10T11:25:19.371 INFO:teuthology.orchestra.run.gibba002.stdout: "features": 4611087854035861503, 2021-02-10T11:25:19.371 INFO:teuthology.orchestra.run.gibba002.stdout: "old_weight": 0 2021-02-10T11:25:19.371 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:25:19.371 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:25:19.371 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 5, 2021-02-10T11:25:19.371 INFO:teuthology.orchestra.run.gibba002.stdout: "down_stamp": "0.000000", 2021-02-10T11:25:19.371 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_probability": 0, 2021-02-10T11:25:19.372 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_interval": 0, 2021-02-10T11:25:19.372 INFO:teuthology.orchestra.run.gibba002.stdout: "features": 4611087854035861503, 2021-02-10T11:25:19.372 INFO:teuthology.orchestra.run.gibba002.stdout: "old_weight": 0 2021-02-10T11:25:19.372 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:25:19.372 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:25:19.372 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 6, 2021-02-10T11:25:19.372 INFO:teuthology.orchestra.run.gibba002.stdout: "down_stamp": "0.000000", 2021-02-10T11:25:19.372 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_probability": 0, 2021-02-10T11:25:19.372 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_interval": 0, 2021-02-10T11:25:19.372 INFO:teuthology.orchestra.run.gibba002.stdout: "features": 4611087854035861503, 2021-02-10T11:25:19.372 INFO:teuthology.orchestra.run.gibba002.stdout: "old_weight": 0 2021-02-10T11:25:19.372 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:25:19.372 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:25:19.373 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 7, 2021-02-10T11:25:19.373 INFO:teuthology.orchestra.run.gibba002.stdout: "down_stamp": "0.000000", 2021-02-10T11:25:19.373 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_probability": 0, 2021-02-10T11:25:19.373 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_interval": 0, 2021-02-10T11:25:19.373 INFO:teuthology.orchestra.run.gibba002.stdout: "features": 4611087854035861503, 2021-02-10T11:25:19.373 INFO:teuthology.orchestra.run.gibba002.stdout: "old_weight": 0 2021-02-10T11:25:19.373 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:25:19.373 INFO:teuthology.orchestra.run.gibba002.stdout: ], 2021-02-10T11:25:19.373 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_upmap": [], 2021-02-10T11:25:19.373 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_upmap_items": [], 2021-02-10T11:25:19.373 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_temp": [], 2021-02-10T11:25:19.373 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_temp": [], 2021-02-10T11:25:19.373 INFO:teuthology.orchestra.run.gibba002.stdout: "blacklist": {}, 2021-02-10T11:25:19.373 INFO:teuthology.orchestra.run.gibba002.stdout: "erasure_code_profiles": { 2021-02-10T11:25:19.374 INFO:teuthology.orchestra.run.gibba002.stdout: "default": { 2021-02-10T11:25:19.374 INFO:teuthology.orchestra.run.gibba002.stdout: "crush-failure-domain": "osd", 2021-02-10T11:25:19.374 INFO:teuthology.orchestra.run.gibba002.stdout: "k": "2", 2021-02-10T11:25:19.374 INFO:teuthology.orchestra.run.gibba002.stdout: "m": "1", 2021-02-10T11:25:19.374 INFO:teuthology.orchestra.run.gibba002.stdout: "plugin": "jerasure", 2021-02-10T11:25:19.374 INFO:teuthology.orchestra.run.gibba002.stdout: "ruleset-failure-domain": "osd", 2021-02-10T11:25:19.374 INFO:teuthology.orchestra.run.gibba002.stdout: "technique": "reed_sol_van" 2021-02-10T11:25:19.374 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:25:19.374 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:25:19.374 INFO:teuthology.orchestra.run.gibba002.stdout: "removed_snaps_queue": [], 2021-02-10T11:25:19.374 INFO:teuthology.orchestra.run.gibba002.stdout: "new_removed_snaps": [], 2021-02-10T11:25:19.374 INFO:teuthology.orchestra.run.gibba002.stdout: "new_purged_snaps": [], 2021-02-10T11:25:19.374 INFO:teuthology.orchestra.run.gibba002.stdout: "crush_node_flags": {}, 2021-02-10T11:25:19.375 INFO:teuthology.orchestra.run.gibba002.stdout: "device_class_flags": {} 2021-02-10T11:25:19.375 INFO:teuthology.orchestra.run.gibba002.stdout:} 2021-02-10T11:25:19.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:19.301 7f35933ef700 1 -- 172.21.2.102:0/1016283506 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f356c0186c0 msgr2=0x7f356c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:19.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:19.301 7f35933ef700 1 --2- 172.21.2.102:0/1016283506 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f356c0186c0 0x7f356c01abc0 crc :-1 s=READY pgs=664 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:19.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:19.301 7f35933ef700 1 -- 172.21.2.102:0/1016283506 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f358c0bb8e0 msgr2=0x7f358c0bbce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:19.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:19.301 7f35933ef700 1 --2- 172.21.2.102:0/1016283506 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f358c0bb8e0 0x7f358c0bbce0 secure :-1 s=READY pgs=1452 cs=0 l=1 rev1=1 rx=0x7f3584009980 tx=0x7f358400bcd0).stop 2021-02-10T11:25:19.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:19.301 7f35933ef700 1 -- 172.21.2.102:0/1016283506 shutdown_connections 2021-02-10T11:25:19.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:19.301 7f35933ef700 1 --2- 172.21.2.102:0/1016283506 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f358c0bb8e0 0x7f358c0bbce0 unknown :-1 s=CLOSED pgs=1452 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:19.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:19.301 7f35933ef700 1 --2- 172.21.2.102:0/1016283506 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f358c195070 0x7f358c195470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:19.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:19.301 7f35933ef700 1 --2- 172.21.2.102:0/1016283506 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f356c0186c0 0x7f356c01abc0 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:19.377 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:19.301 7f35933ef700 1 --2- 172.21.2.102:0/1016283506 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f358c1942b0 0x7f358c0b3550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:19.377 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:19.301 7f35933ef700 1 -- 172.21.2.102:0/1016283506 >> 172.21.2.102:0/1016283506 conn(0x7f358c0b3040 msgr2=0x7f358c0c3250 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:25:19.377 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:19.301 7f35933ef700 1 -- 172.21.2.102:0/1016283506 shutdown_connections 2021-02-10T11:25:19.377 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:19.301 7f35933ef700 1 -- 172.21.2.102:0/1016283506 wait complete. 2021-02-10T11:25:19.378 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:25:19.802 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:19.787 7fc1f646d700 1 Processor -- start 2021-02-10T11:25:19.802 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:19.787 7fc1f646d700 1 -- start start 2021-02-10T11:25:19.802 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:19.788 7fc1f646d700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fc1f00d1870 0x7fc1f01b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:19.802 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:19.788 7fc1f646d700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc1f01b0a10 0x7fc1f00cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:19.802 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:19.788 7fc1f646d700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fc1f00d1c70 0x7fc1f00cf950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:19.802 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:19.788 7fc1f646d700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1f00fe200 con 0x7fc1f00d1c70 2021-02-10T11:25:19.803 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:19.788 7fc1f646d700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1f00e5f10 con 0x7fc1f00d1870 2021-02-10T11:25:19.803 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:19.788 7fc1f646d700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1f00f0b50 con 0x7fc1f01b0a10 2021-02-10T11:25:19.803 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:19.788 7fc1f4a8a700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fc1f00d1c70 0x7fc1f00cf950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:19.803 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:19.788 7fc1f4a8a700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fc1f00d1c70 0x7fc1f00cf950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:57762/0 (socket says 172.21.2.102:57762) 2021-02-10T11:25:19.803 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:19.788 7fc1f4a8a700 1 -- 172.21.2.102:0/1974335287 learned_addr learned my addr 172.21.2.102:0/1974335287 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:25:19.803 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:19.788 7fc1f4a8a700 1 -- 172.21.2.102:0/1974335287 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc1f01b0a10 msgr2=0x7fc1f00cf330 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:25:19.803 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:19.788 7fc1f4a8a700 1 --2- 172.21.2.102:0/1974335287 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc1f01b0a10 0x7fc1f00cf330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:19.803 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:19.788 7fc1f4a8a700 1 -- 172.21.2.102:0/1974335287 >> v2:172.21.2.113:3300/0 conn(0x7fc1f00d1870 msgr2=0x7fc1f01b03f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:25:19.803 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:19.788 7fc1f4a8a700 1 --2- 172.21.2.102:0/1974335287 >> v2:172.21.2.113:3300/0 conn(0x7fc1f00d1870 0x7fc1f01b03f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:19.803 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:19.788 7fc1f4a8a700 1 -- 172.21.2.102:0/1974335287 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1f00cff70 con 0x7fc1f00d1c70 2021-02-10T11:25:19.803 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:19.788 7fc1f4a8a700 1 --2- 172.21.2.102:0/1974335287 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc1f00d1c70 0x7fc1f00cf950 secure :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 rx=0x7fc1e8006f00 tx=0x7fc1e8009f10).ready entity=mon.0 client_cookie=b535ef717b211260 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:19.803 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:19.788 7fc1eeffd700 1 -- 172.21.2.102:0/1974335287 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc1e800b3b0 con 0x7fc1f00d1c70 2021-02-10T11:25:19.803 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:19.789 7fc1eeffd700 1 -- 172.21.2.102:0/1974335287 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1e800b730 con 0x7fc1f00d1c70 2021-02-10T11:25:19.803 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:19.789 7fc1eeffd700 1 -- 172.21.2.102:0/1974335287 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc1e800bc10 con 0x7fc1f00d1c70 2021-02-10T11:25:19.804 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:19.789 7fc1f646d700 1 -- 172.21.2.102:0/1974335287 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc1f00d1c70 msgr2=0x7fc1f00cf950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:19.804 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:19.789 7fc1f646d700 1 --2- 172.21.2.102:0/1974335287 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc1f00d1c70 0x7fc1f00cf950 secure :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 rx=0x7fc1e8006f00 tx=0x7fc1e8009f10).stop 2021-02-10T11:25:19.804 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:19.789 7fc1f646d700 1 -- 172.21.2.102:0/1974335287 shutdown_connections 2021-02-10T11:25:19.804 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:19.789 7fc1f646d700 1 --2- 172.21.2.102:0/1974335287 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc1f01b0a10 0x7fc1f00cf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:19.804 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:19.789 7fc1f646d700 1 --2- 172.21.2.102:0/1974335287 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc1f00d1c70 0x7fc1f00cf950 unknown :-1 s=CLOSED pgs=1453 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:19.804 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:19.789 7fc1f646d700 1 --2- 172.21.2.102:0/1974335287 >> v2:172.21.2.113:3300/0 conn(0x7fc1f00d1870 0x7fc1f01b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:19.804 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:19.789 7fc1f646d700 1 -- 172.21.2.102:0/1974335287 >> 172.21.2.102:0/1974335287 conn(0x7fc1f00ce450 msgr2=0x7fc1f00d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:25:19.804 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:19.789 7fc1f646d700 1 -- 172.21.2.102:0/1974335287 shutdown_connections 2021-02-10T11:25:19.804 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:19.789 7fc1f646d700 1 -- 172.21.2.102:0/1974335287 wait complete. 2021-02-10T11:25:19.806 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:19.803 7fc1f646d700 1 Processor -- start 2021-02-10T11:25:19.806 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:19.803 7fc1f646d700 1 -- start start 2021-02-10T11:25:19.806 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:19.804 7fc1f646d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc1f018e930 0x7fc1f00aa4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:19.806 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:19.804 7fc1f646d700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc1f00aaaf0 0x7fc1f0093700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:19.807 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:19.804 7fc1f646d700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc1f018ed30 0x7fc1f0093d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:19.807 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:19.804 7fc1f646d700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1f00944e0 con 0x7fc1f018e930 2021-02-10T11:25:19.807 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:19.804 7fc1f646d700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1f008a590 con 0x7fc1f00aaaf0 2021-02-10T11:25:19.807 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:19.804 7fc1f646d700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1f008a770 con 0x7fc1f018ed30 2021-02-10T11:25:19.807 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:19.804 7fc1effff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc1f018e930 0x7fc1f00aa4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:19.807 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:19.804 7fc1effff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc1f018e930 0x7fc1f00aa4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:57770/0 (socket says 172.21.2.102:57770) 2021-02-10T11:25:19.807 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:19.804 7fc1effff700 1 -- 172.21.2.102:0/2407386012 learned_addr learned my addr 172.21.2.102:0/2407386012 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:25:19.807 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:19.804 7fc1f4a8a700 1 --2- 172.21.2.102:0/2407386012 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc1f018ed30 0x7fc1f0093d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:19.807 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:19.804 7fc1effff700 1 -- 172.21.2.102:0/2407386012 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc1f018ed30 msgr2=0x7fc1f0093d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:19.807 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:19.804 7fc1effff700 1 --2- 172.21.2.102:0/2407386012 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc1f018ed30 0x7fc1f0093d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:19.807 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:19.804 7fc1effff700 1 -- 172.21.2.102:0/2407386012 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc1f00aaaf0 msgr2=0x7fc1f0093700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:19.807 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:19.804 7fc1ef7fe700 1 --2- 172.21.2.102:0/2407386012 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc1f00aaaf0 0x7fc1f0093700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:19.807 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:19.805 7fc1effff700 1 --2- 172.21.2.102:0/2407386012 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc1f00aaaf0 0x7fc1f0093700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:19.808 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:19.805 7fc1effff700 1 -- 172.21.2.102:0/2407386012 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1f008a990 con 0x7fc1f018e930 2021-02-10T11:25:19.808 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:19.805 7fc1f4a8a700 1 --2- 172.21.2.102:0/2407386012 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc1f018ed30 0x7fc1f0093d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:25:19.808 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:19.805 7fc1ef7fe700 1 --2- 172.21.2.102:0/2407386012 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc1f00aaaf0 0x7fc1f0093700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:25:19.808 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:19.805 7fc1effff700 1 --2- 172.21.2.102:0/2407386012 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc1f018e930 0x7fc1f00aa4d0 secure :-1 s=READY pgs=1454 cs=0 l=1 rev1=1 rx=0x7fc1e000b5c0 tx=0x7fc1e000b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:19.808 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:19.805 7fc1ecff9700 1 -- 172.21.2.102:0/2407386012 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc1e000bd50 con 0x7fc1f018e930 2021-02-10T11:25:19.808 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:19.805 7fc1ecff9700 1 -- 172.21.2.102:0/2407386012 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1e0006030 con 0x7fc1f018e930 2021-02-10T11:25:19.808 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:19.805 7fc1f646d700 1 -- 172.21.2.102:0/2407386012 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1f008a990 con 0x7fc1f018e930 2021-02-10T11:25:19.808 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:19.805 7fc1ecff9700 1 -- 172.21.2.102:0/2407386012 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc1e00054d0 con 0x7fc1f018e930 2021-02-10T11:25:19.808 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:19.805 7fc1f646d700 1 -- 172.21.2.102:0/2407386012 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1f008a990 con 0x7fc1f018e930 2021-02-10T11:25:19.808 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:19.805 7fc1ecff9700 1 -- 172.21.2.102:0/2407386012 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fc1e0006210 con 0x7fc1f018e930 2021-02-10T11:25:19.808 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:19.806 7fc1ecff9700 1 --2- 172.21.2.102:0/2407386012 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc1d0018730 0x7fc1d001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:19.808 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:19.806 7fc1ecff9700 1 -- 172.21.2.102:0/2407386012 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fc1e0032610 con 0x7fc1f018e930 2021-02-10T11:25:19.808 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:19.806 7fc1ef7fe700 1 --2- 172.21.2.102:0/2407386012 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc1d0018730 0x7fc1d001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:19.809 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:19.806 7fc1ef7fe700 1 --2- 172.21.2.102:0/2407386012 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc1d0018730 0x7fc1d001abc0 crc :-1 s=READY pgs=665 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:19.837 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:19.837 7fc1f646d700 1 -- 172.21.2.102:0/2407386012 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1f00ee060 con 0x7fc1f018e930 2021-02-10T11:25:19.839 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:19.840 7fc1ecff9700 1 -- 172.21.2.102:0/2407386012 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fc1e0006b80 con 0x7fc1f018e930 2021-02-10T11:25:19.968 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:19.968 7fc1f646d700 1 -- 172.21.2.102:0/2407386012 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc1f00ee060 con 0x7fc1f018e930 2021-02-10T11:25:19.968 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:19.969 7fc1ecff9700 1 -- 172.21.2.102:0/2407386012 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) v1 ==== 94+0+2101 (secure 0 0 0) 0x7fc1e000bd50 con 0x7fc1f018e930 2021-02-10T11:25:19.969 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:25:19.969 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":40,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7920,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.102:6835/1829204031","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6834","nonce":1829204031},{"type":"v1","addr":"172.21.2.102:6835","nonce":1829204031}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":35}],"filesystems":[{"mdsmap":{"epoch":39,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:24:10.198618","modified":"2021-02-10 11:24:12.204882","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5714},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5714":{"gid":5714,"name":"b","rank":0,"incarnation":36,"state":"up:active","state_seq":3,"addr":"172.21.2.113:6833/4013137507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6832","nonce":4013137507},{"type":"v1","addr":"172.21.2.113:6833","nonce":4013137507}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":5}]} 2021-02-10T11:25:19.969 INFO:teuthology.orchestra.run.gibba002.stderr:dumped fsmap epoch 40 2021-02-10T11:25:19.970 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:19.971 7fc1f646d700 1 -- 172.21.2.102:0/2407386012 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc1d0018730 msgr2=0x7fc1d001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:19.970 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:19.971 7fc1f646d700 1 --2- 172.21.2.102:0/2407386012 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc1d0018730 0x7fc1d001abc0 crc :-1 s=READY pgs=665 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:19.970 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:19.971 7fc1f646d700 1 -- 172.21.2.102:0/2407386012 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc1f018e930 msgr2=0x7fc1f00aa4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:19.970 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:19.971 7fc1f646d700 1 --2- 172.21.2.102:0/2407386012 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc1f018e930 0x7fc1f00aa4d0 secure :-1 s=READY pgs=1454 cs=0 l=1 rev1=1 rx=0x7fc1e000b5c0 tx=0x7fc1e000b960).stop 2021-02-10T11:25:19.970 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:19.971 7fc1f646d700 1 -- 172.21.2.102:0/2407386012 shutdown_connections 2021-02-10T11:25:19.971 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:19.971 7fc1f646d700 1 --2- 172.21.2.102:0/2407386012 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc1f018ed30 0x7fc1f0093d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:19.971 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:19.971 7fc1f646d700 1 --2- 172.21.2.102:0/2407386012 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc1d0018730 0x7fc1d001abc0 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:19.971 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:19.971 7fc1f646d700 1 --2- 172.21.2.102:0/2407386012 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc1f00aaaf0 0x7fc1f0093700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:19.971 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:19.971 7fc1f646d700 1 --2- 172.21.2.102:0/2407386012 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc1f018e930 0x7fc1f00aa4d0 unknown :-1 s=CLOSED pgs=1454 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:19.971 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:19.971 7fc1f646d700 1 -- 172.21.2.102:0/2407386012 >> 172.21.2.102:0/2407386012 conn(0x7fc1f019f8f0 msgr2=0x7fc1f018c260 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:25:19.971 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:19.971 7fc1f646d700 1 -- 172.21.2.102:0/2407386012 shutdown_connections 2021-02-10T11:25:19.971 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:19.971 7fc1f646d700 1 -- 172.21.2.102:0/2407386012 wait complete. 2021-02-10T11:25:19.979 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs fail cephfs 2021-02-10T11:25:20.408 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:20.392 7f7f5315f700 1 Processor -- start 2021-02-10T11:25:20.408 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:20.392 7f7f5315f700 1 -- start start 2021-02-10T11:25:20.408 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:20.392 7f7f5315f700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f7f4c0d1860 0x7f7f4c1b03e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:20.408 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:20.392 7f7f5315f700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f4c0f0b00 con 0x7f7f4c1b0a00 2021-02-10T11:25:20.408 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:20.392 7f7f5315f700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f4c0d0300 con 0x7f7f4c0f0640 2021-02-10T11:25:20.408 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:20.392 7f7f5315f700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f4c0d0520 con 0x7f7f4c0d1860 2021-02-10T11:25:20.408 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:20.393 7f7f4bfff700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f7f4c1b0a00 0x7f7f4c0cf320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:49998/0 (socket says 172.21.2.102:49998) 2021-02-10T11:25:20.408 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:20.393 7f7f4bfff700 1 -- 172.21.2.102:0/1126667553 learned_addr learned my addr 172.21.2.102:0/1126667553 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:25:20.408 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:20.393 7f7f50f7b700 1 --2- 172.21.2.102:0/1126667553 >> v2:172.21.2.102:3300/0 conn(0x7f7f4c0d1860 0x7f7f4c1b03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:20.408 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:20.393 7f7f4b7fe700 1 -- 172.21.2.102:0/1126667553 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 182046364 0 0) 0x7f7f3c000ce0 con 0x7f7f4c1b0a00 2021-02-10T11:25:20.409 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:20.393 7f7f50f7b700 1 -- 172.21.2.102:0/1126667553 >> v1:172.21.2.113:6789/0 conn(0x7f7f4c0f0640 legacy=0x7f7f4c0cfb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:25:20.409 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:20.393 7f7f50f7b700 1 -- 172.21.2.102:0/1126667553 >> v1:172.21.2.102:6789/0 conn(0x7f7f4c1b0a00 legacy=0x7f7f4c0cf320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:20.409 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:20.393 7f7f50f7b700 1 -- 172.21.2.102:0/1126667553 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f4c0d1c60 con 0x7f7f4c0d1860 2021-02-10T11:25:20.409 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:20.393 7f7f50f7b700 1 --2- 172.21.2.102:0/1126667553 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7f4c0d1860 0x7f7f4c1b03e0 secure :-1 s=READY pgs=1456 cs=0 l=1 rev1=1 rx=0x7f7f34006650 tx=0x7f7f34009a80).ready entity=mon.0 client_cookie=4f00cd4c3f4a939a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:20.409 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:20.393 7f7f4b7fe700 1 -- 172.21.2.102:0/1126667553 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7f3400c0c0 con 0x7f7f4c0d1860 2021-02-10T11:25:20.409 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:20.393 7f7f4b7fe700 1 -- 172.21.2.102:0/1126667553 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f3400c340 con 0x7f7f4c0d1860 2021-02-10T11:25:20.409 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:20.393 7f7f4b7fe700 1 -- 172.21.2.102:0/1126667553 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7f3400c860 con 0x7f7f4c0d1860 2021-02-10T11:25:20.409 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:20.393 7f7f5315f700 1 -- 172.21.2.102:0/1126667553 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7f4c0d1860 msgr2=0x7f7f4c1b03e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:20.409 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:20.393 7f7f5315f700 1 --2- 172.21.2.102:0/1126667553 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7f4c0d1860 0x7f7f4c1b03e0 secure :-1 s=READY pgs=1456 cs=0 l=1 rev1=1 rx=0x7f7f34006650 tx=0x7f7f34009a80).stop 2021-02-10T11:25:20.409 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:20.394 7f7f5315f700 1 -- 172.21.2.102:0/1126667553 shutdown_connections 2021-02-10T11:25:20.409 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:20.394 7f7f5315f700 1 --2- 172.21.2.102:0/1126667553 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7f4c0d1860 0x7f7f4c1b03e0 unknown :-1 s=CLOSED pgs=1456 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:20.409 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:20.394 7f7f5315f700 1 -- 172.21.2.102:0/1126667553 >> 172.21.2.102:0/1126667553 conn(0x7f7f4c0ce440 msgr2=0x7f7f4c0d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:25:20.409 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:20.394 7f7f5315f700 1 -- 172.21.2.102:0/1126667553 shutdown_connections 2021-02-10T11:25:20.410 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:20.394 7f7f5315f700 1 -- 172.21.2.102:0/1126667553 wait complete. 2021-02-10T11:25:20.411 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:20.409 7f7f5315f700 1 Processor -- start 2021-02-10T11:25:20.411 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:20.409 7f7f5315f700 1 -- start start 2021-02-10T11:25:20.411 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:20.410 7f7f5315f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7f4c0acbb0 0x7f7f4c0ac030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:20.411 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:20.410 7f7f5315f700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7f4c0ac650 0x7f7f4c0ad080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:20.411 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:20.410 7f7f5315f700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7f4c0ad6a0 0x7f7f4c0f19b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:20.411 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:20.410 7f7f5315f700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f4c0f1430 con 0x7f7f4c0acbb0 2021-02-10T11:25:20.411 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:20.410 7f7f5315f700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f4c0adaa0 con 0x7f7f4c0ad6a0 2021-02-10T11:25:20.411 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:20.410 7f7f5315f700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f4c0f1fd0 con 0x7f7f4c0ac650 2021-02-10T11:25:20.411 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:20.410 7f7f50f7b700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7f4c0acbb0 0x7f7f4c0ac030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:20.411 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:20.410 7f7f50f7b700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7f4c0acbb0 0x7f7f4c0ac030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:57780/0 (socket says 172.21.2.102:57780) 2021-02-10T11:25:20.412 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:20.410 7f7f50f7b700 1 -- 172.21.2.102:0/219080751 learned_addr learned my addr 172.21.2.102:0/219080751 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:25:20.412 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:20.410 7f7f50f7b700 1 -- 172.21.2.102:0/219080751 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7f4c0ac650 msgr2=0x7f7f4c0ad080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:20.412 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:20.410 7f7f50f7b700 1 --2- 172.21.2.102:0/219080751 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7f4c0ac650 0x7f7f4c0ad080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:20.412 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:20.410 7f7f50f7b700 1 -- 172.21.2.102:0/219080751 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7f4c0ad6a0 msgr2=0x7f7f4c0f19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:20.412 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:20.410 7f7f50f7b700 1 --2- 172.21.2.102:0/219080751 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7f4c0ad6a0 0x7f7f4c0f19b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:20.412 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:20.410 7f7f50f7b700 1 -- 172.21.2.102:0/219080751 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f4c0f21f0 con 0x7f7f4c0acbb0 2021-02-10T11:25:20.412 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:20.410 7f7f50f7b700 1 --2- 172.21.2.102:0/219080751 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7f4c0acbb0 0x7f7f4c0ac030 secure :-1 s=READY pgs=1457 cs=0 l=1 rev1=1 rx=0x7f7f34009650 tx=0x7f7f3400b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:20.412 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:20.410 7f7f497fa700 1 -- 172.21.2.102:0/219080751 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7f3400bd50 con 0x7f7f4c0acbb0 2021-02-10T11:25:20.412 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:20.411 7f7f497fa700 1 -- 172.21.2.102:0/219080751 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f34006030 con 0x7f7f4c0acbb0 2021-02-10T11:25:20.412 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:20.411 7f7f497fa700 1 -- 172.21.2.102:0/219080751 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7f340054d0 con 0x7f7f4c0acbb0 2021-02-10T11:25:20.412 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:20.411 7f7f5315f700 1 -- 172.21.2.102:0/219080751 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f4c0f21f0 con 0x7f7f4c0acbb0 2021-02-10T11:25:20.412 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:20.411 7f7f5315f700 1 -- 172.21.2.102:0/219080751 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f4c0f2780 con 0x7f7f4c0acbb0 2021-02-10T11:25:20.412 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:20.411 7f7f497fa700 1 -- 172.21.2.102:0/219080751 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f7f340052e0 con 0x7f7f4c0acbb0 2021-02-10T11:25:20.413 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:20.411 7f7f497fa700 1 --2- 172.21.2.102:0/219080751 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7f300186c0 0x7f7f3001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:20.413 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:20.411 7f7f497fa700 1 -- 172.21.2.102:0/219080751 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f7f34032580 con 0x7f7f4c0acbb0 2021-02-10T11:25:20.413 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:20.411 7f7f4bfff700 1 --2- 172.21.2.102:0/219080751 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7f300186c0 0x7f7f3001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:20.413 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:20.412 7f7f4bfff700 1 --2- 172.21.2.102:0/219080751 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7f300186c0 0x7f7f3001abc0 crc :-1 s=READY pgs=666 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:20.441 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:20.442 7f7f5315f700 1 -- 172.21.2.102:0/219080751 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f4c0ee060 con 0x7f7f4c0acbb0 2021-02-10T11:25:20.444 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:20.445 7f7f497fa700 1 -- 172.21.2.102:0/219080751 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f7f34009300 con 0x7f7f4c0acbb0 2021-02-10T11:25:20.571 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:20.572 7f7f5315f700 1 -- 172.21.2.102:0/219080751 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f7f4c0ce440 con 0x7f7f4c0acbb0 2021-02-10T11:25:21.356 INFO:tasks.ceph.mon.a.gibba002.stderr:2021-02-10 11:25:21.356 7f8dd2755700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-02-10T11:25:21.362 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:21.363 7f7f497fa700 1 -- 172.21.2.102:0/219080751 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v41) v1 ==== 161+0+0 (secure 0 0 0) 0x7f7f340052e0 con 0x7f7f4c0acbb0 2021-02-10T11:25:21.389 INFO:teuthology.orchestra.run.gibba002.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2021-02-10T11:25:21.391 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:21.392 7f7f5315f700 1 -- 172.21.2.102:0/219080751 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7f300186c0 msgr2=0x7f7f3001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:21.391 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:21.392 7f7f5315f700 1 --2- 172.21.2.102:0/219080751 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7f300186c0 0x7f7f3001abc0 crc :-1 s=READY pgs=666 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:21.391 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:21.392 7f7f5315f700 1 -- 172.21.2.102:0/219080751 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7f4c0acbb0 msgr2=0x7f7f4c0ac030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:21.391 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:21.392 7f7f5315f700 1 --2- 172.21.2.102:0/219080751 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7f4c0acbb0 0x7f7f4c0ac030 secure :-1 s=READY pgs=1457 cs=0 l=1 rev1=1 rx=0x7f7f34009650 tx=0x7f7f3400b960).stop 2021-02-10T11:25:21.392 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:21.392 7f7f5315f700 1 -- 172.21.2.102:0/219080751 shutdown_connections 2021-02-10T11:25:21.392 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:21.392 7f7f5315f700 1 --2- 172.21.2.102:0/219080751 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7f300186c0 0x7f7f3001abc0 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:21.392 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:21.392 7f7f5315f700 1 --2- 172.21.2.102:0/219080751 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7f4c0ad6a0 0x7f7f4c0f19b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:21.392 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:21.392 7f7f5315f700 1 --2- 172.21.2.102:0/219080751 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7f4c0ac650 0x7f7f4c0ad080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:21.392 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:21.392 7f7f5315f700 1 --2- 172.21.2.102:0/219080751 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7f4c0acbb0 0x7f7f4c0ac030 unknown :-1 s=CLOSED pgs=1457 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:21.392 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:21.392 7f7f5315f700 1 -- 172.21.2.102:0/219080751 >> 172.21.2.102:0/219080751 conn(0x7f7f4c0bea70 msgr2=0x7f7f4c095700 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:25:21.392 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:21.392 7f7f5315f700 1 -- 172.21.2.102:0/219080751 shutdown_connections 2021-02-10T11:25:21.392 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:21.392 7f7f5315f700 1 -- 172.21.2.102:0/219080751 wait complete. 2021-02-10T11:25:21.400 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:25:21.413 INFO:tasks.ceph.mds.b.gibba013.stdout:starting mds.b at 2021-02-10T11:25:21.825 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:21.810 7f016dcb8700 1 Processor -- start 2021-02-10T11:25:21.825 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:21.810 7f016dcb8700 1 -- start start 2021-02-10T11:25:21.825 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:21.811 7f016dcb8700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f01680d1870 0x7f01681b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:21.825 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:21.811 7f016dcb8700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f01680f0650 0x7f01680cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:21.826 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:21.811 7f016dcb8700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01680f0b10 con 0x7f01681b0a10 2021-02-10T11:25:21.826 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:21.811 7f016dcb8700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01680e5ed0 con 0x7f01680d1870 2021-02-10T11:25:21.826 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:21.811 7f016dcb8700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01680d1c70 con 0x7f01680f0650 2021-02-10T11:25:21.826 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:21.811 7f01677fe700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f01680d1870 0x7f01681b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:21.826 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:21.811 7f01677fe700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f01680d1870 0x7f01681b03f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:57788/0 (socket says 172.21.2.102:57788) 2021-02-10T11:25:21.826 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:21.811 7f01677fe700 1 -- 172.21.2.102:0/1302907238 learned_addr learned my addr 172.21.2.102:0/1302907238 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:25:21.826 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:21.811 7f01677fe700 1 -- 172.21.2.102:0/1302907238 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f01680f0650 msgr2=0x7f01680cfb20 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:25:21.826 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:21.811 7f01677fe700 1 --2- 172.21.2.102:0/1302907238 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f01680f0650 0x7f01680cfb20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:21.826 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:21.811 7f01677fe700 1 -- 172.21.2.102:0/1302907238 >> v1:172.21.2.113:6789/0 conn(0x7f01681b0a10 legacy=0x7f01680cf330 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:25:21.826 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:21.811 7f01677fe700 1 -- 172.21.2.102:0/1302907238 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01680d1e90 con 0x7f01680d1870 2021-02-10T11:25:21.826 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:21.811 7f01677fe700 1 --2- 172.21.2.102:0/1302907238 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f01680d1870 0x7f01681b03f0 secure :-1 s=READY pgs=1459 cs=0 l=1 rev1=1 rx=0x7f0150006700 tx=0x7f0150009b00).ready entity=mon.0 client_cookie=dbe4eeb1365545db server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:21.827 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:21.812 7f0166ffd700 1 -- 172.21.2.102:0/1302907238 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f015000c0c0 con 0x7f01680d1870 2021-02-10T11:25:21.827 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:21.812 7f0166ffd700 1 -- 172.21.2.102:0/1302907238 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f015000c400 con 0x7f01680d1870 2021-02-10T11:25:21.827 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:21.812 7f0166ffd700 1 -- 172.21.2.102:0/1302907238 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f015000c860 con 0x7f01680d1870 2021-02-10T11:25:21.827 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:21.812 7f016dcb8700 1 -- 172.21.2.102:0/1302907238 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f01680d1870 msgr2=0x7f01681b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:21.827 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:21.812 7f016dcb8700 1 --2- 172.21.2.102:0/1302907238 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f01680d1870 0x7f01681b03f0 secure :-1 s=READY pgs=1459 cs=0 l=1 rev1=1 rx=0x7f0150006700 tx=0x7f0150009b00).stop 2021-02-10T11:25:21.827 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:21.812 7f016dcb8700 1 -- 172.21.2.102:0/1302907238 shutdown_connections 2021-02-10T11:25:21.827 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:21.812 7f016dcb8700 1 --2- 172.21.2.102:0/1302907238 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f01680f0650 0x7f01680cfb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:21.827 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:21.812 7f016dcb8700 1 --2- 172.21.2.102:0/1302907238 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f01680d1870 0x7f01681b03f0 unknown :-1 s=CLOSED pgs=1459 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:21.827 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:21.812 7f016dcb8700 1 -- 172.21.2.102:0/1302907238 >> 172.21.2.102:0/1302907238 conn(0x7f01680ce450 msgr2=0x7f01680d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:25:21.827 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:21.812 7f016dcb8700 1 -- 172.21.2.102:0/1302907238 shutdown_connections 2021-02-10T11:25:21.827 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:21.812 7f016dcb8700 1 -- 172.21.2.102:0/1302907238 wait complete. 2021-02-10T11:25:21.829 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:21.826 7f016dcb8700 1 Processor -- start 2021-02-10T11:25:21.829 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:21.827 7f016dcb8700 1 -- start start 2021-02-10T11:25:21.829 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:21.827 7f016dcb8700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f01681ae550 0x7f01680ef260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:21.829 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:21.827 7f016dcb8700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f01680ef880 0x7f016808a450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:21.829 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:21.827 7f016dcb8700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f01681a73b0 0x7f016808aa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:21.829 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:21.827 7f016dcb8700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01681a7940 con 0x7f01681a73b0 2021-02-10T11:25:21.829 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:21.827 7f016dcb8700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01681a7e40 con 0x7f01681ae550 2021-02-10T11:25:21.829 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:21.827 7f016dcb8700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01681ae950 con 0x7f01680ef880 2021-02-10T11:25:21.829 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:21.827 7f0167fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f01681a73b0 0x7f016808aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:21.829 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:21.827 7f0167fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f01681a73b0 0x7f016808aa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:57792/0 (socket says 172.21.2.102:57792) 2021-02-10T11:25:21.829 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:21.827 7f0167fff700 1 -- 172.21.2.102:0/3299311527 learned_addr learned my addr 172.21.2.102:0/3299311527 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:25:21.829 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:21.828 7f0167fff700 1 -- 172.21.2.102:0/3299311527 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f01680ef880 msgr2=0x7f016808a450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:21.830 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:21.828 7f0167fff700 1 --2- 172.21.2.102:0/3299311527 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f01680ef880 0x7f016808a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:21.830 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:21.828 7f0167fff700 1 -- 172.21.2.102:0/3299311527 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f01681ae550 msgr2=0x7f01680ef260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:21.830 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:21.828 7f0167fff700 1 --2- 172.21.2.102:0/3299311527 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f01681ae550 0x7f01680ef260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:21.830 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:21.828 7f0167fff700 1 -- 172.21.2.102:0/3299311527 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01681aeb70 con 0x7f01681a73b0 2021-02-10T11:25:21.830 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:21.828 7f0167fff700 1 --2- 172.21.2.102:0/3299311527 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f01681a73b0 0x7f016808aa70 secure :-1 s=READY pgs=1460 cs=0 l=1 rev1=1 rx=0x7f0160009960 tx=0x7f016000bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:21.830 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:21.828 7f0164ff9700 1 -- 172.21.2.102:0/3299311527 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0160003030 con 0x7f01681a73b0 2021-02-10T11:25:21.830 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:21.828 7f0164ff9700 1 -- 172.21.2.102:0/3299311527 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0160001310 con 0x7f01681a73b0 2021-02-10T11:25:21.830 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:21.828 7f016dcb8700 1 -- 172.21.2.102:0/3299311527 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01681aeb70 con 0x7f01681a73b0 2021-02-10T11:25:21.830 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:21.828 7f0164ff9700 1 -- 172.21.2.102:0/3299311527 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0160001770 con 0x7f01681a73b0 2021-02-10T11:25:21.830 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:21.828 7f016dcb8700 1 -- 172.21.2.102:0/3299311527 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f016808b380 con 0x7f01681a73b0 2021-02-10T11:25:21.830 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:21.829 7f0164ff9700 1 -- 172.21.2.102:0/3299311527 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f01600014f0 con 0x7f01681a73b0 2021-02-10T11:25:21.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:21.829 7f0164ff9700 1 --2- 172.21.2.102:0/3299311527 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f014c0186c0 0x7f014c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:21.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:21.829 7f0164ff9700 1 -- 172.21.2.102:0/3299311527 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5659+0+0 (secure 0 0 0) 0x7f01600329b0 con 0x7f01681a73b0 2021-02-10T11:25:21.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:21.829 7f01677fe700 1 --2- 172.21.2.102:0/3299311527 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f014c0186c0 0x7f014c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:21.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:21.829 7f01677fe700 1 --2- 172.21.2.102:0/3299311527 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f014c0186c0 0x7f014c01abc0 crc :-1 s=READY pgs=667 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:21.860 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:21.860 7f016dcb8700 1 -- 172.21.2.102:0/3299311527 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01680ee060 con 0x7f01681a73b0 2021-02-10T11:25:21.862 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:21.863 7f0164ff9700 1 -- 172.21.2.102:0/3299311527 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f0160000ce0 con 0x7f01681a73b0 2021-02-10T11:25:21.991 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:21.992 7f016dcb8700 1 -- 172.21.2.102:0/3299311527 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f01680ce450 con 0x7f01681a73b0 2021-02-10T11:25:21.991 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:21.992 7f0164ff9700 1 -- 172.21.2.102:0/3299311527 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) v1 ==== 94+0+1756 (secure 0 0 0) 0x7f0160001310 con 0x7f01681a73b0 2021-02-10T11:25:21.992 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:25:21.992 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":41,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7920,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.102:6835/1829204031","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6834","nonce":1829204031},{"type":"v1","addr":"172.21.2.102:6835","nonce":1829204031}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":35}],"filesystems":[{"mdsmap":{"epoch":41,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:24:10.198618","modified":"2021-02-10 11:25:21.358146","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":5}]} 2021-02-10T11:25:21.992 INFO:teuthology.orchestra.run.gibba002.stderr:dumped fsmap epoch 41 2021-02-10T11:25:21.993 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:21.994 7f016dcb8700 1 -- 172.21.2.102:0/3299311527 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f014c0186c0 msgr2=0x7f014c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:21.993 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:21.994 7f016dcb8700 1 --2- 172.21.2.102:0/3299311527 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f014c0186c0 0x7f014c01abc0 crc :-1 s=READY pgs=667 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:21.993 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:21.994 7f016dcb8700 1 -- 172.21.2.102:0/3299311527 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f01681a73b0 msgr2=0x7f016808aa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:21.993 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:21.994 7f016dcb8700 1 --2- 172.21.2.102:0/3299311527 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f01681a73b0 0x7f016808aa70 secure :-1 s=READY pgs=1460 cs=0 l=1 rev1=1 rx=0x7f0160009960 tx=0x7f016000bcf0).stop 2021-02-10T11:25:21.994 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:21.994 7f016dcb8700 1 -- 172.21.2.102:0/3299311527 shutdown_connections 2021-02-10T11:25:21.994 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:21.994 7f016dcb8700 1 --2- 172.21.2.102:0/3299311527 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f01681a73b0 0x7f016808aa70 unknown :-1 s=CLOSED pgs=1460 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:21.994 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:21.994 7f016dcb8700 1 --2- 172.21.2.102:0/3299311527 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f01680ef880 0x7f016808a450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:21.994 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:21.994 7f016dcb8700 1 --2- 172.21.2.102:0/3299311527 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f014c0186c0 0x7f014c01abc0 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:21.994 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:21.994 7f016dcb8700 1 --2- 172.21.2.102:0/3299311527 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f01681ae550 0x7f01680ef260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:21.994 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:21.994 7f016dcb8700 1 -- 172.21.2.102:0/3299311527 >> 172.21.2.102:0/3299311527 conn(0x7f01681a8c90 msgr2=0x7f01681b38a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:25:21.994 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:21.994 7f016dcb8700 1 -- 172.21.2.102:0/3299311527 shutdown_connections 2021-02-10T11:25:21.994 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:21.994 7f016dcb8700 1 -- 172.21.2.102:0/3299311527 wait complete. 2021-02-10T11:25:22.002 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2021-02-10T11:25:22.438 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:22.420 7fba1c2af700 1 Processor -- start 2021-02-10T11:25:22.438 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:22.420 7fba1c2af700 1 -- start start 2021-02-10T11:25:22.438 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:22.420 7fba1c2af700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fba140d1870 0x7fba141b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:22.438 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:22.420 7fba1c2af700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fba140f0650 0x7fba140cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:22.438 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:22.420 7fba1c2af700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba140f0b10 con 0x7fba141b0a10 2021-02-10T11:25:22.438 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:22.420 7fba1c2af700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba140fe230 con 0x7fba140f0650 2021-02-10T11:25:22.439 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:22.420 7fba1c2af700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba140e5ed0 con 0x7fba140d1870 2021-02-10T11:25:22.439 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:22.421 7fba1a8cc700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fba140f0650 0x7fba140cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:22.439 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:22.421 7fba1a0cb700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fba140d1870 0x7fba141b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:22.439 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:22.421 7fba198ca700 1 --1- >> v1:172.21.2.113:6789/0 conn(0x7fba141b0a10 0x7fba140cf330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.113:6789/0 says I am v1:172.21.2.102:57348/0 (socket says 172.21.2.102:57348) 2021-02-10T11:25:22.439 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:22.421 7fba198ca700 1 -- 172.21.2.102:0/419126458 learned_addr learned my addr 172.21.2.102:0/419126458 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:25:22.439 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:22.421 7fba1a8cc700 1 -- 172.21.2.102:0/419126458 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fba140d1870 msgr2=0x7fba141b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:25:22.439 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:22.421 7fba1a8cc700 1 --2- 172.21.2.102:0/419126458 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fba140d1870 0x7fba141b03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:22.439 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:22.421 7fba1a8cc700 1 -- 172.21.2.102:0/419126458 >> v1:172.21.2.113:6789/0 conn(0x7fba141b0a10 legacy=0x7fba140cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:22.439 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:22.422 7fba190c9700 1 -- 172.21.2.102:0/419126458 <== mon.1 v1:172.21.2.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1234950780 0 0) 0x7fba080010b0 con 0x7fba141b0a10 2021-02-10T11:25:22.439 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:22.422 7fba1a8cc700 1 -- 172.21.2.102:0/419126458 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba140f20a0 con 0x7fba140f0650 2021-02-10T11:25:22.439 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:22.422 7fba1a8cc700 1 --2- 172.21.2.102:0/419126458 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fba140f0650 0x7fba140cfb20 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 rx=0x7fba0c006e50 tx=0x7fba0c009df0).ready entity=mon.1 client_cookie=d5cbced597c47601 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:22.439 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:22.422 7fba190c9700 1 -- 172.21.2.102:0/419126458 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fba0c00b510 con 0x7fba140f0650 2021-02-10T11:25:22.440 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:22.422 7fba190c9700 1 -- 172.21.2.102:0/419126458 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba0c006790 con 0x7fba140f0650 2021-02-10T11:25:22.440 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:22.422 7fba190c9700 1 -- 172.21.2.102:0/419126458 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fba0c00b750 con 0x7fba140f0650 2021-02-10T11:25:22.440 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:22.422 7fba1c2af700 1 -- 172.21.2.102:0/419126458 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fba140f0650 msgr2=0x7fba140cfb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:22.440 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:22.422 7fba1c2af700 1 --2- 172.21.2.102:0/419126458 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fba140f0650 0x7fba140cfb20 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 rx=0x7fba0c006e50 tx=0x7fba0c009df0).stop 2021-02-10T11:25:22.440 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:22.423 7fba1c2af700 1 -- 172.21.2.102:0/419126458 shutdown_connections 2021-02-10T11:25:22.440 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:22.423 7fba1c2af700 1 --2- 172.21.2.102:0/419126458 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fba140f0650 0x7fba140cfb20 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:22.440 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:22.423 7fba1c2af700 1 --2- 172.21.2.102:0/419126458 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fba140d1870 0x7fba141b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:22.440 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:22.423 7fba1c2af700 1 -- 172.21.2.102:0/419126458 >> 172.21.2.102:0/419126458 conn(0x7fba140ce450 msgr2=0x7fba140d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:25:22.440 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:22.423 7fba1c2af700 1 -- 172.21.2.102:0/419126458 shutdown_connections 2021-02-10T11:25:22.440 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:22.423 7fba1c2af700 1 -- 172.21.2.102:0/419126458 wait complete. 2021-02-10T11:25:22.440 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:22.437 7fba1c2af700 1 Processor -- start 2021-02-10T11:25:22.440 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:22.437 7fba1c2af700 1 -- start start 2021-02-10T11:25:22.440 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:22.438 7fba1c2af700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fba14095190 0x7fba141a4410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:22.441 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:22.438 7fba1c2af700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fba14095740 0x7fba141934b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:22.441 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:22.438 7fba1c2af700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fba14193ad0 0x7fba1409d670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:22.441 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:22.438 7fba1c2af700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba140efd70 con 0x7fba14193ad0 2021-02-10T11:25:22.441 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:22.438 7fba1c2af700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba14193f50 con 0x7fba14095190 2021-02-10T11:25:22.441 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:22.438 7fba1c2af700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba141a4a30 con 0x7fba14095740 2021-02-10T11:25:22.441 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:22.438 7fba1a8cc700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fba14193ad0 0x7fba1409d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:22.441 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:22.438 7fba1a8cc700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fba14193ad0 0x7fba1409d670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:57810/0 (socket says 172.21.2.102:57810) 2021-02-10T11:25:22.441 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:22.438 7fba1a8cc700 1 -- 172.21.2.102:0/1774556842 learned_addr learned my addr 172.21.2.102:0/1774556842 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:25:22.442 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:22.438 7fba1a8cc700 1 -- 172.21.2.102:0/1774556842 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fba14095740 msgr2=0x7fba141934b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:22.442 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:22.438 7fba1a8cc700 1 --2- 172.21.2.102:0/1774556842 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fba14095740 0x7fba141934b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:22.442 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:22.438 7fba1a8cc700 1 -- 172.21.2.102:0/1774556842 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fba14095190 msgr2=0x7fba141a4410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:22.442 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:22.438 7fba1a8cc700 1 --2- 172.21.2.102:0/1774556842 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fba14095190 0x7fba141a4410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:22.443 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:22.438 7fba1a8cc700 1 -- 172.21.2.102:0/1774556842 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba141a4c10 con 0x7fba14193ad0 2021-02-10T11:25:22.443 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:22.438 7fba1a8cc700 1 --2- 172.21.2.102:0/1774556842 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fba14193ad0 0x7fba1409d670 secure :-1 s=READY pgs=1461 cs=0 l=1 rev1=1 rx=0x7fba0c006fa0 tx=0x7fba0c00bd90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:22.443 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:22.439 7fba06ffd700 1 -- 172.21.2.102:0/1774556842 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fba0c0093d0 con 0x7fba14193ad0 2021-02-10T11:25:22.443 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:22.439 7fba06ffd700 1 -- 172.21.2.102:0/1774556842 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba0c0082e0 con 0x7fba14193ad0 2021-02-10T11:25:22.443 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:22.439 7fba06ffd700 1 -- 172.21.2.102:0/1774556842 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fba0c008800 con 0x7fba14193ad0 2021-02-10T11:25:22.443 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:22.439 7fba1c2af700 1 -- 172.21.2.102:0/1774556842 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba141a4c10 con 0x7fba14193ad0 2021-02-10T11:25:22.443 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:22.439 7fba1c2af700 1 -- 172.21.2.102:0/1774556842 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba1409dfb0 con 0x7fba14193ad0 2021-02-10T11:25:22.444 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:22.440 7fba06ffd700 1 -- 172.21.2.102:0/1774556842 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fba0c0084c0 con 0x7fba14193ad0 2021-02-10T11:25:22.444 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:22.440 7fba06ffd700 1 --2- 172.21.2.102:0/1774556842 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb9fc0186c0 0x7fb9fc01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:22.444 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:22.440 7fba06ffd700 1 -- 172.21.2.102:0/1774556842 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5659+0+0 (secure 0 0 0) 0x7fba0c032960 con 0x7fba14193ad0 2021-02-10T11:25:22.444 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:22.440 7fba1a0cb700 1 --2- 172.21.2.102:0/1774556842 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb9fc0186c0 0x7fb9fc01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:22.444 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:22.440 7fba1a0cb700 1 --2- 172.21.2.102:0/1774556842 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb9fc0186c0 0x7fb9fc01abc0 crc :-1 s=READY pgs=668 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:22.469 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:22.470 7fba1c2af700 1 -- 172.21.2.102:0/1774556842 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba140efd70 con 0x7fba14193ad0 2021-02-10T11:25:22.473 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:22.472 7fba06ffd700 1 -- 172.21.2.102:0/1774556842 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fba0c0093d0 con 0x7fba14193ad0 2021-02-10T11:25:22.599 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:22.600 7fba1c2af700 1 -- 172.21.2.102:0/1774556842 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fba140efd70 con 0x7fba14193ad0 2021-02-10T11:25:23.366 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:23.367 7fba06ffd700 1 -- 172.21.2.102:0/1774556842 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v42) v1 ==== 104+0+0 (secure 0 0 0) 0x7fba0c01f650 con 0x7fba14193ad0 2021-02-10T11:25:23.371 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:23.372 7fba1c2af700 1 -- 172.21.2.102:0/1774556842 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb9fc0186c0 msgr2=0x7fb9fc01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:23.372 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:23.372 7fba1c2af700 1 --2- 172.21.2.102:0/1774556842 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb9fc0186c0 0x7fb9fc01abc0 crc :-1 s=READY pgs=668 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:23.372 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:23.372 7fba1c2af700 1 -- 172.21.2.102:0/1774556842 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fba14193ad0 msgr2=0x7fba1409d670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:23.372 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:23.372 7fba1c2af700 1 --2- 172.21.2.102:0/1774556842 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fba14193ad0 0x7fba1409d670 secure :-1 s=READY pgs=1461 cs=0 l=1 rev1=1 rx=0x7fba0c006fa0 tx=0x7fba0c00bd90).stop 2021-02-10T11:25:23.372 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:23.372 7fba1c2af700 1 -- 172.21.2.102:0/1774556842 shutdown_connections 2021-02-10T11:25:23.372 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:23.372 7fba1c2af700 1 --2- 172.21.2.102:0/1774556842 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fba14193ad0 0x7fba1409d670 unknown :-1 s=CLOSED pgs=1461 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:23.372 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:23.372 7fba1c2af700 1 --2- 172.21.2.102:0/1774556842 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fba14095740 0x7fba141934b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:23.372 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:23.372 7fba1c2af700 1 --2- 172.21.2.102:0/1774556842 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb9fc0186c0 0x7fb9fc01abc0 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:23.372 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:23.372 7fba1c2af700 1 --2- 172.21.2.102:0/1774556842 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fba14095190 0x7fba141a4410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:23.372 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:23.372 7fba1c2af700 1 -- 172.21.2.102:0/1774556842 >> 172.21.2.102:0/1774556842 conn(0x7fba140ed860 msgr2=0x7fba140bf0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:25:23.372 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:23.372 7fba1c2af700 1 -- 172.21.2.102:0/1774556842 shutdown_connections 2021-02-10T11:25:23.372 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:23.372 7fba1c2af700 1 -- 172.21.2.102:0/1774556842 wait complete. 2021-02-10T11:25:23.380 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool delete cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2021-02-10T11:25:23.808 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:23.793 7fe9ca8a4700 1 Processor -- start 2021-02-10T11:25:23.809 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:23.793 7fe9ca8a4700 1 -- start start 2021-02-10T11:25:23.809 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:23.794 7fe9ca8a4700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe9c40d1870 0x7fe9c41b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:23.809 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:23.794 7fe9ca8a4700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fe9c41b0ae0 0x7fe9c4108940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:23.809 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:23.794 7fe9ca8a4700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9c4109a60 con 0x7fe9c40d1c70 2021-02-10T11:25:23.809 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:23.794 7fe9ca8a4700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9c40e5ef0 con 0x7fe9c41b0ae0 2021-02-10T11:25:23.809 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:23.794 7fe9ca8a4700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9c4109750 con 0x7fe9c40d1870 2021-02-10T11:25:23.809 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:23.794 7fe9c37fe700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fe9c41b0ae0 0x7fe9c4108940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:23.809 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:23.794 7fe9c37fe700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fe9c41b0ae0 0x7fe9c4108940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:57818/0 (socket says 172.21.2.102:57818) 2021-02-10T11:25:23.809 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:23.794 7fe9c37fe700 1 -- 172.21.2.102:0/3058730317 learned_addr learned my addr 172.21.2.102:0/3058730317 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:25:23.809 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:23.794 7fe9c37fe700 1 -- 172.21.2.102:0/3058730317 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe9c40d1870 msgr2=0x7fe9c41b04c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:25:23.809 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:23.794 7fe9c37fe700 1 --2- 172.21.2.102:0/3058730317 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe9c40d1870 0x7fe9c41b04c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:23.809 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:23.794 7fe9c37fe700 1 -- 172.21.2.102:0/3058730317 >> v1:172.21.2.113:6789/0 conn(0x7fe9c40d1c70 legacy=0x7fe9c4108f60 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:25:23.810 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:23.794 7fe9c37fe700 1 -- 172.21.2.102:0/3058730317 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9c4088070 con 0x7fe9c41b0ae0 2021-02-10T11:25:23.810 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:23.795 7fe9c37fe700 1 --2- 172.21.2.102:0/3058730317 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe9c41b0ae0 0x7fe9c4108940 secure :-1 s=READY pgs=1462 cs=0 l=1 rev1=1 rx=0x7fe9b8006700 tx=0x7fe9b8009b80).ready entity=mon.0 client_cookie=df4ca81a46319cb3 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:23.810 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:23.795 7fe9c2ffd700 1 -- 172.21.2.102:0/3058730317 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe9b800c0c0 con 0x7fe9c41b0ae0 2021-02-10T11:25:23.810 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:23.795 7fe9c2ffd700 1 -- 172.21.2.102:0/3058730317 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9b800c440 con 0x7fe9c41b0ae0 2021-02-10T11:25:23.810 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:23.795 7fe9c2ffd700 1 -- 172.21.2.102:0/3058730317 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe9b800c920 con 0x7fe9c41b0ae0 2021-02-10T11:25:23.810 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:23.795 7fe9ca8a4700 1 -- 172.21.2.102:0/3058730317 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe9c41b0ae0 msgr2=0x7fe9c4108940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:23.810 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:23.795 7fe9ca8a4700 1 --2- 172.21.2.102:0/3058730317 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe9c41b0ae0 0x7fe9c4108940 secure :-1 s=READY pgs=1462 cs=0 l=1 rev1=1 rx=0x7fe9b8006700 tx=0x7fe9b8009b80).stop 2021-02-10T11:25:23.810 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:23.795 7fe9ca8a4700 1 -- 172.21.2.102:0/3058730317 shutdown_connections 2021-02-10T11:25:23.810 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:23.795 7fe9ca8a4700 1 --2- 172.21.2.102:0/3058730317 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe9c41b0ae0 0x7fe9c4108940 unknown :-1 s=CLOSED pgs=1462 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:23.810 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:23.795 7fe9ca8a4700 1 --2- 172.21.2.102:0/3058730317 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe9c40d1870 0x7fe9c41b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:23.810 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:23.795 7fe9ca8a4700 1 -- 172.21.2.102:0/3058730317 >> 172.21.2.102:0/3058730317 conn(0x7fe9c40ce420 msgr2=0x7fe9c40d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:25:23.810 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:23.796 7fe9ca8a4700 1 -- 172.21.2.102:0/3058730317 shutdown_connections 2021-02-10T11:25:23.810 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:23.796 7fe9ca8a4700 1 -- 172.21.2.102:0/3058730317 wait complete. 2021-02-10T11:25:23.812 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:23.810 7fe9ca8a4700 1 Processor -- start 2021-02-10T11:25:23.812 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:23.810 7fe9ca8a4700 1 -- start start 2021-02-10T11:25:23.812 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:23.810 7fe9ca8a4700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe9c41b33e0 0x7fe9c41a2b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:23.812 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:23.810 7fe9ca8a4700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe9c40931c0 0x7fe9c41b2610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:23.812 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:23.810 7fe9ca8a4700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe9c4187e70 0x7fe9c4188270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:23.812 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:23.810 7fe9ca8a4700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9c41b2e60 con 0x7fe9c40931c0 2021-02-10T11:25:23.812 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:23.810 7fe9ca8a4700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9c41b37e0 con 0x7fe9c41b33e0 2021-02-10T11:25:23.812 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:23.810 7fe9ca8a4700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9c40935c0 con 0x7fe9c4187e70 2021-02-10T11:25:23.813 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:23.810 7fe9c37fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe9c40931c0 0x7fe9c41b2610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:23.813 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:23.810 7fe9c37fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe9c40931c0 0x7fe9c41b2610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:57822/0 (socket says 172.21.2.102:57822) 2021-02-10T11:25:23.813 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:23.811 7fe9c37fe700 1 -- 172.21.2.102:0/3260303876 learned_addr learned my addr 172.21.2.102:0/3260303876 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:25:23.813 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:23.811 7fe9c37fe700 1 -- 172.21.2.102:0/3260303876 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe9c4187e70 msgr2=0x7fe9c4188270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:23.813 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:23.811 7fe9c37fe700 1 --2- 172.21.2.102:0/3260303876 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe9c4187e70 0x7fe9c4188270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:23.813 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:23.811 7fe9c37fe700 1 -- 172.21.2.102:0/3260303876 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe9c41b33e0 msgr2=0x7fe9c41a2b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:23.813 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:23.811 7fe9c37fe700 1 --2- 172.21.2.102:0/3260303876 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe9c41b33e0 0x7fe9c41a2b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:23.813 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:23.811 7fe9c37fe700 1 -- 172.21.2.102:0/3260303876 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9c4188890 con 0x7fe9c40931c0 2021-02-10T11:25:23.813 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:23.811 7fe9c37fe700 1 --2- 172.21.2.102:0/3260303876 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe9c40931c0 0x7fe9c41b2610 secure :-1 s=READY pgs=1463 cs=0 l=1 rev1=1 rx=0x7fe9b800b5e0 tx=0x7fe9b800b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:23.813 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:23.811 7fe9c0ff9700 1 -- 172.21.2.102:0/3260303876 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe9b800bd70 con 0x7fe9c40931c0 2021-02-10T11:25:23.813 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:23.811 7fe9c0ff9700 1 -- 172.21.2.102:0/3260303876 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9b8002030 con 0x7fe9c40931c0 2021-02-10T11:25:23.813 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:23.811 7fe9c0ff9700 1 -- 172.21.2.102:0/3260303876 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe9b8002550 con 0x7fe9c40931c0 2021-02-10T11:25:23.813 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:23.811 7fe9ca8a4700 1 -- 172.21.2.102:0/3260303876 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9c4188890 con 0x7fe9c40931c0 2021-02-10T11:25:23.813 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:23.811 7fe9ca8a4700 1 -- 172.21.2.102:0/3260303876 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9c40b7a20 con 0x7fe9c40931c0 2021-02-10T11:25:23.814 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:23.812 7fe9c0ff9700 1 -- 172.21.2.102:0/3260303876 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fe9b8000c90 con 0x7fe9c40931c0 2021-02-10T11:25:23.814 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:23.812 7fe9c0ff9700 1 --2- 172.21.2.102:0/3260303876 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe9a40186c0 0x7fe9a401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:23.814 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:23.812 7fe9c0ff9700 1 -- 172.21.2.102:0/3260303876 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5659+0+0 (secure 0 0 0) 0x7fe9b8032610 con 0x7fe9c40931c0 2021-02-10T11:25:23.814 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:23.812 7fe9c3fff700 1 --2- 172.21.2.102:0/3260303876 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe9a40186c0 0x7fe9a401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:23.814 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:23.812 7fe9c3fff700 1 --2- 172.21.2.102:0/3260303876 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe9a40186c0 0x7fe9a401abc0 crc :-1 s=READY pgs=669 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:23.843 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:23.844 7fe9ca8a4700 1 -- 172.21.2.102:0/3260303876 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9c40825d0 con 0x7fe9c40931c0 2021-02-10T11:25:23.846 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:23.847 7fe9c0ff9700 1 -- 172.21.2.102:0/3260303876 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fe9b80009b0 con 0x7fe9c40931c0 2021-02-10T11:25:23.973 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:23.974 7fe9ca8a4700 1 -- 172.21.2.102:0/3260303876 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"pool2": "cephfs_metadata", "prefix": "osd pool delete", "pool": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fe9c40825d0 con 0x7fe9c40931c0 2021-02-10T11:25:24.375 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:24.376 7fe9c0ff9700 1 -- 172.21.2.102:0/3260303876 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"pool2": "cephfs_metadata", "prefix": "osd pool delete", "pool": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v59) v1 ==== 185+0+0 (secure 0 0 0) 0x7fe9b8000c90 con 0x7fe9c40931c0 2021-02-10T11:25:24.391 INFO:teuthology.orchestra.run.gibba002.stderr:pool 'cephfs_metadata' removed 2021-02-10T11:25:24.393 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:24.394 7fe9ca8a4700 1 -- 172.21.2.102:0/3260303876 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe9a40186c0 msgr2=0x7fe9a401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:24.393 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:24.394 7fe9ca8a4700 1 --2- 172.21.2.102:0/3260303876 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe9a40186c0 0x7fe9a401abc0 crc :-1 s=READY pgs=669 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:24.393 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:24.394 7fe9ca8a4700 1 -- 172.21.2.102:0/3260303876 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe9c40931c0 msgr2=0x7fe9c41b2610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:24.393 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:24.394 7fe9ca8a4700 1 --2- 172.21.2.102:0/3260303876 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe9c40931c0 0x7fe9c41b2610 secure :-1 s=READY pgs=1463 cs=0 l=1 rev1=1 rx=0x7fe9b800b5e0 tx=0x7fe9b800b980).stop 2021-02-10T11:25:24.393 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:24.394 7fe9ca8a4700 1 -- 172.21.2.102:0/3260303876 shutdown_connections 2021-02-10T11:25:24.393 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:24.394 7fe9ca8a4700 1 --2- 172.21.2.102:0/3260303876 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe9c4187e70 0x7fe9c4188270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:24.393 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:24.394 7fe9ca8a4700 1 --2- 172.21.2.102:0/3260303876 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe9c40931c0 0x7fe9c41b2610 unknown :-1 s=CLOSED pgs=1463 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:24.393 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:24.394 7fe9ca8a4700 1 --2- 172.21.2.102:0/3260303876 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe9a40186c0 0x7fe9a401abc0 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:24.393 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:24.394 7fe9ca8a4700 1 --2- 172.21.2.102:0/3260303876 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe9c41b33e0 0x7fe9c41a2b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:24.394 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:24.394 7fe9ca8a4700 1 -- 172.21.2.102:0/3260303876 >> 172.21.2.102:0/3260303876 conn(0x7fe9c40841e0 msgr2=0x7fe9c4082800 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:25:24.394 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:24.394 7fe9ca8a4700 1 -- 172.21.2.102:0/3260303876 shutdown_connections 2021-02-10T11:25:24.394 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:24.394 7fe9ca8a4700 1 -- 172.21.2.102:0/3260303876 wait complete. 2021-02-10T11:25:24.402 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool delete cephfs_data cephfs_data --yes-i-really-really-mean-it 2021-02-10T11:25:24.835 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:24.819 7fe8d79a9700 1 Processor -- start 2021-02-10T11:25:24.835 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:24.819 7fe8d79a9700 1 -- start start 2021-02-10T11:25:24.835 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:24.819 7fe8d79a9700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fe8d00d1870 0x7fe8d01b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:24.835 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:24.819 7fe8d79a9700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe8d00cff70 0x7fe8d0109130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:24.835 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:24.819 7fe8d79a9700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8d0109a60 con 0x7fe8d01b0ae0 2021-02-10T11:25:24.835 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:24.819 7fe8d79a9700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8d0109750 con 0x7fe8d00d1870 2021-02-10T11:25:24.835 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:24.819 7fe8d79a9700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8d0088070 con 0x7fe8d00cff70 2021-02-10T11:25:24.835 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:24.820 7fe8d57c5700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fe8d00d1870 0x7fe8d01b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:24.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:24.820 7fe8d4fc4700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fe8d01b0ae0 0x7fe8d0108940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:50052/0 (socket says 172.21.2.102:50052) 2021-02-10T11:25:24.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:24.820 7fe8d4fc4700 1 -- 172.21.2.102:0/3323516035 learned_addr learned my addr 172.21.2.102:0/3323516035 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:25:24.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:24.820 7fe8d5fc6700 1 --2- 172.21.2.102:0/3323516035 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe8d00cff70 0x7fe8d0109130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:24.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:24.820 7fe8d57c5700 1 -- 172.21.2.102:0/3323516035 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe8d00cff70 msgr2=0x7fe8d0109130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:25:24.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:24.820 7fe8d57c5700 1 --2- 172.21.2.102:0/3323516035 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe8d00cff70 0x7fe8d0109130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:24.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:24.820 7fe8d57c5700 1 -- 172.21.2.102:0/3323516035 >> v1:172.21.2.102:6789/0 conn(0x7fe8d01b0ae0 legacy=0x7fe8d0108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:24.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:24.820 7fe8d57c5700 1 -- 172.21.2.102:0/3323516035 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8d00d1c70 con 0x7fe8d00d1870 2021-02-10T11:25:24.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:24.820 7fe8d57c5700 1 --2- 172.21.2.102:0/3323516035 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe8d00d1870 0x7fe8d01b04c0 secure :-1 s=READY pgs=1465 cs=0 l=1 rev1=1 rx=0x7fe8cc006650 tx=0x7fe8cc009a80).ready entity=mon.0 client_cookie=65f4fa7c54ff8cab server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:24.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:24.820 7fe8c3fff700 1 -- 172.21.2.102:0/3323516035 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe8cc00c0c0 con 0x7fe8d00d1870 2021-02-10T11:25:24.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:24.820 7fe8c3fff700 1 -- 172.21.2.102:0/3323516035 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8cc00c340 con 0x7fe8d00d1870 2021-02-10T11:25:24.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:24.820 7fe8c3fff700 1 -- 172.21.2.102:0/3323516035 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe8cc00c860 con 0x7fe8d00d1870 2021-02-10T11:25:24.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:24.820 7fe8d79a9700 1 -- 172.21.2.102:0/3323516035 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe8d00d1870 msgr2=0x7fe8d01b04c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:24.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:24.820 7fe8d79a9700 1 --2- 172.21.2.102:0/3323516035 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe8d00d1870 0x7fe8d01b04c0 secure :-1 s=READY pgs=1465 cs=0 l=1 rev1=1 rx=0x7fe8cc006650 tx=0x7fe8cc009a80).stop 2021-02-10T11:25:24.837 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:24.821 7fe8d79a9700 1 -- 172.21.2.102:0/3323516035 shutdown_connections 2021-02-10T11:25:24.837 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:24.821 7fe8d79a9700 1 --2- 172.21.2.102:0/3323516035 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe8d00cff70 0x7fe8d0109130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:24.837 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:24.821 7fe8d79a9700 1 --2- 172.21.2.102:0/3323516035 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe8d00d1870 0x7fe8d01b04c0 unknown :-1 s=CLOSED pgs=1465 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:24.837 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:24.821 7fe8d79a9700 1 -- 172.21.2.102:0/3323516035 >> 172.21.2.102:0/3323516035 conn(0x7fe8d00ce420 msgr2=0x7fe8d00d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:25:24.837 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:24.821 7fe8d79a9700 1 -- 172.21.2.102:0/3323516035 shutdown_connections 2021-02-10T11:25:24.837 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:24.821 7fe8d79a9700 1 -- 172.21.2.102:0/3323516035 wait complete. 2021-02-10T11:25:24.839 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:24.836 7fe8d79a9700 1 Processor -- start 2021-02-10T11:25:24.839 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:24.836 7fe8d79a9700 1 -- start start 2021-02-10T11:25:24.839 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:24.836 7fe8d79a9700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe8d019e5c0 0x7fe8d0099fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:24.839 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:24.836 7fe8d79a9700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe8d009a5d0 0x7fe8d009a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:24.839 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:24.836 7fe8d79a9700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe8d00982a0 0x7fe8d00986a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:24.839 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:24.836 7fe8d79a9700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8d019feb0 con 0x7fe8d009a5d0 2021-02-10T11:25:24.839 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:24.836 7fe8d79a9700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8d0099a30 con 0x7fe8d00982a0 2021-02-10T11:25:24.839 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:24.836 7fe8d79a9700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8d019fb30 con 0x7fe8d019e5c0 2021-02-10T11:25:24.839 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:24.837 7fe8d4fc4700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe8d009a5d0 0x7fe8d009a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:24.839 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:24.837 7fe8d4fc4700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe8d009a5d0 0x7fe8d009a9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:57832/0 (socket says 172.21.2.102:57832) 2021-02-10T11:25:24.840 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:24.837 7fe8d4fc4700 1 -- 172.21.2.102:0/2594443278 learned_addr learned my addr 172.21.2.102:0/2594443278 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:25:24.840 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:24.837 7fe8d5fc6700 1 --2- 172.21.2.102:0/2594443278 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe8d00982a0 0x7fe8d00986a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:24.840 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:24.837 7fe8d57c5700 1 --2- 172.21.2.102:0/2594443278 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe8d019e5c0 0x7fe8d0099fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:24.840 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:24.837 7fe8d4fc4700 1 -- 172.21.2.102:0/2594443278 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe8d019e5c0 msgr2=0x7fe8d0099fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:24.840 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:24.837 7fe8d4fc4700 1 --2- 172.21.2.102:0/2594443278 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe8d019e5c0 0x7fe8d0099fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:24.840 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:24.837 7fe8d4fc4700 1 -- 172.21.2.102:0/2594443278 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe8d00982a0 msgr2=0x7fe8d00986a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:24.840 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:24.837 7fe8d4fc4700 1 --2- 172.21.2.102:0/2594443278 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe8d00982a0 0x7fe8d00986a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:24.840 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:24.837 7fe8d4fc4700 1 -- 172.21.2.102:0/2594443278 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8d019e9c0 con 0x7fe8d009a5d0 2021-02-10T11:25:24.840 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:24.837 7fe8d5fc6700 1 --2- 172.21.2.102:0/2594443278 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe8d00982a0 0x7fe8d00986a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:25:24.840 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:24.837 7fe8d4fc4700 1 --2- 172.21.2.102:0/2594443278 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe8d009a5d0 0x7fe8d009a9d0 secure :-1 s=READY pgs=1466 cs=0 l=1 rev1=1 rx=0x7fe8c4001c60 tx=0x7fe8c400a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:24.840 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:24.837 7fe8c1ffb700 1 -- 172.21.2.102:0/2594443278 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe8c400acc0 con 0x7fe8d009a5d0 2021-02-10T11:25:24.840 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:24.837 7fe8c1ffb700 1 -- 172.21.2.102:0/2594443278 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8c4004140 con 0x7fe8d009a5d0 2021-02-10T11:25:24.840 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:24.837 7fe8d79a9700 1 -- 172.21.2.102:0/2594443278 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8d019e9c0 con 0x7fe8d009a5d0 2021-02-10T11:25:24.841 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:24.837 7fe8c1ffb700 1 -- 172.21.2.102:0/2594443278 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe8c40045a0 con 0x7fe8d009a5d0 2021-02-10T11:25:24.841 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:24.837 7fe8d79a9700 1 -- 172.21.2.102:0/2594443278 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8d019e9c0 con 0x7fe8d009a5d0 2021-02-10T11:25:24.841 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:24.838 7fe8c1ffb700 1 -- 172.21.2.102:0/2594443278 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fe8c4004320 con 0x7fe8d009a5d0 2021-02-10T11:25:24.841 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:24.838 7fe8c1ffb700 1 --2- 172.21.2.102:0/2594443278 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe8b0018730 0x7fe8b001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:24.841 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:24.838 7fe8c1ffb700 1 -- 172.21.2.102:0/2594443278 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5221+0+0 (secure 0 0 0) 0x7fe8c40324d0 con 0x7fe8d009a5d0 2021-02-10T11:25:24.841 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:24.838 7fe8d57c5700 1 --2- 172.21.2.102:0/2594443278 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe8b0018730 0x7fe8b001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:24.841 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:24.838 7fe8d57c5700 1 --2- 172.21.2.102:0/2594443278 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe8b0018730 0x7fe8b001ac10 crc :-1 s=READY pgs=670 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:24.868 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:24.869 7fe8d79a9700 1 -- 172.21.2.102:0/2594443278 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8d00825d0 con 0x7fe8d009a5d0 2021-02-10T11:25:24.870 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:24.871 7fe8c1ffb700 1 -- 172.21.2.102:0/2594443278 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fe8c40009b0 con 0x7fe8d009a5d0 2021-02-10T11:25:24.996 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:24.996 7fe8d79a9700 1 -- 172.21.2.102:0/2594443278 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"pool2": "cephfs_data", "prefix": "osd pool delete", "pool": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fe8d00825d0 con 0x7fe8d009a5d0 2021-02-10T11:25:25.381 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:25.382 7fe8c1ffb700 1 -- 172.21.2.102:0/2594443278 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"pool2": "cephfs_data", "prefix": "osd pool delete", "pool": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v60) v1 ==== 173+0+0 (secure 0 0 0) 0x7fe8c40042f0 con 0x7fe8d009a5d0 2021-02-10T11:25:25.412 INFO:teuthology.orchestra.run.gibba002.stderr:pool 'cephfs_data' removed 2021-02-10T11:25:25.414 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:25.414 7fe8d79a9700 1 -- 172.21.2.102:0/2594443278 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe8b0018730 msgr2=0x7fe8b001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:25.414 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:25.414 7fe8d79a9700 1 --2- 172.21.2.102:0/2594443278 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe8b0018730 0x7fe8b001ac10 crc :-1 s=READY pgs=670 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:25.414 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:25.414 7fe8d79a9700 1 -- 172.21.2.102:0/2594443278 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe8d009a5d0 msgr2=0x7fe8d009a9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:25.414 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:25.414 7fe8d79a9700 1 --2- 172.21.2.102:0/2594443278 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe8d009a5d0 0x7fe8d009a9d0 secure :-1 s=READY pgs=1466 cs=0 l=1 rev1=1 rx=0x7fe8c4001c60 tx=0x7fe8c400a8d0).stop 2021-02-10T11:25:25.414 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:25.415 7fe8d79a9700 1 -- 172.21.2.102:0/2594443278 shutdown_connections 2021-02-10T11:25:25.415 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:25.415 7fe8d79a9700 1 --2- 172.21.2.102:0/2594443278 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe8b0018730 0x7fe8b001ac10 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:25.415 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:25.415 7fe8d79a9700 1 --2- 172.21.2.102:0/2594443278 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe8d00982a0 0x7fe8d00986a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:25.415 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:25.415 7fe8d79a9700 1 --2- 172.21.2.102:0/2594443278 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe8d009a5d0 0x7fe8d009a9d0 unknown :-1 s=CLOSED pgs=1466 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:25.415 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:25.415 7fe8d79a9700 1 --2- 172.21.2.102:0/2594443278 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe8d019e5c0 0x7fe8d0099fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:25.415 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:25.415 7fe8d79a9700 1 -- 172.21.2.102:0/2594443278 >> 172.21.2.102:0/2594443278 conn(0x7fe8d01b3560 msgr2=0x7fe8d01b90e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:25:25.415 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:25.415 7fe8d79a9700 1 -- 172.21.2.102:0/2594443278 shutdown_connections 2021-02-10T11:25:25.415 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:25.415 7fe8d79a9700 1 -- 172.21.2.102:0/2594443278 wait complete. 2021-02-10T11:25:25.423 INFO:tasks.ceph.mds.a:Restarting daemon 2021-02-10T11:25:25.423 INFO:tasks.ceph.mds.a:Stopping old one... 2021-02-10T11:25:25.423 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2021-02-10T11:25:25.424 INFO:teuthology.orchestra.run:waiting for 300 2021-02-10T11:25:25.424 INFO:tasks.ceph.mds.b:Restarting daemon 2021-02-10T11:25:25.424 INFO:tasks.ceph.mds.b:Stopping old one... 2021-02-10T11:25:25.424 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2021-02-10T11:25:25.424 INFO:teuthology.orchestra.run:waiting for 300 2021-02-10T11:25:25.430 INFO:tasks.ceph.mds.a:Stopped 2021-02-10T11:25:25.430 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2021-02-10T11:25:25.431 INFO:tasks.ceph.mds.b:Stopped 2021-02-10T11:25:25.432 DEBUG:teuthology.orchestra.run.gibba013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2021-02-10T11:25:25.473 INFO:tasks.ceph.mds.a:Started 2021-02-10T11:25:25.473 DEBUG:teuthology.parallel:result is None 2021-02-10T11:25:25.474 INFO:tasks.ceph.mds.b:Started 2021-02-10T11:25:25.474 DEBUG:teuthology.parallel:result is None 2021-02-10T11:25:25.474 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd blacklist clear 2021-02-10T11:25:25.878 INFO:tasks.ceph.mds.b.gibba013.stdout:starting mds.b at 2021-02-10T11:25:25.878 INFO:tasks.ceph.mds.a.gibba002.stdout:starting mds.a at 2021-02-10T11:25:25.903 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:25.886 7f809ad0d700 1 Processor -- start 2021-02-10T11:25:25.903 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:25.886 7f809ad0d700 1 -- start start 2021-02-10T11:25:25.903 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:25.887 7f809ad0d700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f80940d1870 0x7f80941b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:25.903 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:25.887 7f809ad0d700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f80941b0a10 0x7f80940cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:25.903 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:25.887 7f809ad0d700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80940f0b10 con 0x7f80940d1c70 2021-02-10T11:25:25.903 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:25.887 7f809ad0d700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80940e5ed0 con 0x7f80940d1870 2021-02-10T11:25:25.903 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:25.887 7f809ad0d700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80940d2fe0 con 0x7f80941b0a10 2021-02-10T11:25:25.903 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:25.887 7f809932a700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f80940d1c70 0x7f80940cf950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:50080/0 (socket says 172.21.2.102:50080) 2021-02-10T11:25:25.904 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:25.887 7f809932a700 1 -- 172.21.2.102:0/148172385 learned_addr learned my addr 172.21.2.102:0/148172385 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:25:25.904 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:25.887 7f8098b29700 1 --2- 172.21.2.102:0/148172385 >> v2:172.21.2.113:3300/0 conn(0x7f80940d1870 0x7f80941b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:25.904 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:25.887 7f8093fff700 1 --2- 172.21.2.102:0/148172385 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f80941b0a10 0x7f80940cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:25.904 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:25.887 7f80937fe700 1 -- 172.21.2.102:0/148172385 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3101446474 0 0) 0x7f8084000c10 con 0x7f80940d1c70 2021-02-10T11:25:25.904 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:25.887 7f80937fe700 1 -- 172.21.2.102:0/148172385 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f807c002fe0 con 0x7f80940d1c70 2021-02-10T11:25:25.904 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:25.887 7f8098b29700 1 -- 172.21.2.102:0/148172385 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f80941b0a10 msgr2=0x7f80940cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:25:25.904 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:25.887 7f8098b29700 1 --2- 172.21.2.102:0/148172385 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f80941b0a10 0x7f80940cf330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:25.904 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:25.887 7f8098b29700 1 -- 172.21.2.102:0/148172385 >> v1:172.21.2.102:6789/0 conn(0x7f80940d1c70 legacy=0x7f80940cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:25.904 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:25.888 7f8098b29700 1 -- 172.21.2.102:0/148172385 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80940f0650 con 0x7f80940d1870 2021-02-10T11:25:25.904 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:25.888 7f8098b29700 1 --2- 172.21.2.102:0/148172385 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f80940d1870 0x7f80941b03f0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 rx=0x7f808c0069d0 tx=0x7f808c009dc0).ready entity=mon.1 client_cookie=74825f94649a5e2e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:25.904 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:25.888 7f80937fe700 1 -- 172.21.2.102:0/148172385 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f808c00b510 con 0x7f80940d1870 2021-02-10T11:25:25.904 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:25.888 7f80937fe700 1 -- 172.21.2.102:0/148172385 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f808c00b790 con 0x7f80940d1870 2021-02-10T11:25:25.905 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:25.888 7f80937fe700 1 -- 172.21.2.102:0/148172385 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f808c00bcb0 con 0x7f80940d1870 2021-02-10T11:25:25.905 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:25.888 7f809ad0d700 1 -- 172.21.2.102:0/148172385 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f80940d1870 msgr2=0x7f80941b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:25.905 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:25.888 7f809ad0d700 1 --2- 172.21.2.102:0/148172385 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f80940d1870 0x7f80941b03f0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 rx=0x7f808c0069d0 tx=0x7f808c009dc0).stop 2021-02-10T11:25:25.905 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:25.889 7f809ad0d700 1 -- 172.21.2.102:0/148172385 shutdown_connections 2021-02-10T11:25:25.905 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:25.889 7f809ad0d700 1 --2- 172.21.2.102:0/148172385 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f80941b0a10 0x7f80940cf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:25.905 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:25.889 7f809ad0d700 1 --2- 172.21.2.102:0/148172385 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f80940d1870 0x7f80941b03f0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:25.905 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:25.889 7f809ad0d700 1 -- 172.21.2.102:0/148172385 >> 172.21.2.102:0/148172385 conn(0x7f80940ce450 msgr2=0x7f80940d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:25:25.905 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:25.889 7f809ad0d700 1 -- 172.21.2.102:0/148172385 shutdown_connections 2021-02-10T11:25:25.905 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:25.889 7f809ad0d700 1 -- 172.21.2.102:0/148172385 wait complete. 2021-02-10T11:25:25.905 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:25.903 7f809ad0d700 1 Processor -- start 2021-02-10T11:25:25.905 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:25.904 7f809ad0d700 1 -- start start 2021-02-10T11:25:25.907 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:25.904 7f809ad0d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8094189d00 0x7f809418a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:25.907 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:25.904 7f809ad0d700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f809418a720 0x7f809418ab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:25.907 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:25.904 7f809ad0d700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8094187e50 0x7f8094188250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:25.908 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:25.904 7f809ad0d700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80940c37e0 con 0x7f8094189d00 2021-02-10T11:25:25.908 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:25.904 7f809ad0d700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80940c39d0 con 0x7f8094187e50 2021-02-10T11:25:25.908 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:25.904 7f809ad0d700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8094188870 con 0x7f809418a720 2021-02-10T11:25:25.908 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:25.904 7f8098b29700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8094189d00 0x7f809418a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:25.908 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:25.904 7f8098b29700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8094189d00 0x7f809418a100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:57858/0 (socket says 172.21.2.102:57858) 2021-02-10T11:25:25.908 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:25.904 7f8098b29700 1 -- 172.21.2.102:0/423313847 learned_addr learned my addr 172.21.2.102:0/423313847 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:25:25.908 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:25.904 7f809932a700 1 --2- 172.21.2.102:0/423313847 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8094187e50 0x7f8094188250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:25.908 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:25.904 7f8093fff700 1 --2- 172.21.2.102:0/423313847 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f809418a720 0x7f809418ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:25.908 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:25.904 7f8098b29700 1 -- 172.21.2.102:0/423313847 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f809418a720 msgr2=0x7f809418ab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:25.908 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:25.904 7f8098b29700 1 --2- 172.21.2.102:0/423313847 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f809418a720 0x7f809418ab20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:25.908 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:25.904 7f8098b29700 1 -- 172.21.2.102:0/423313847 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8094187e50 msgr2=0x7f8094188250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:25.908 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:25.904 7f8098b29700 1 --2- 172.21.2.102:0/423313847 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8094187e50 0x7f8094188250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:25.909 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:25.904 7f8098b29700 1 -- 172.21.2.102:0/423313847 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8094188a90 con 0x7f8094189d00 2021-02-10T11:25:25.909 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:25.905 7f8098b29700 1 --2- 172.21.2.102:0/423313847 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8094189d00 0x7f809418a100 secure :-1 s=READY pgs=1472 cs=0 l=1 rev1=1 rx=0x7f808c0099e0 tx=0x7f808c00bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:25.909 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:25.905 7f80917fa700 1 -- 172.21.2.102:0/423313847 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f808c006030 con 0x7f8094189d00 2021-02-10T11:25:25.909 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:25.905 7f80917fa700 1 -- 172.21.2.102:0/423313847 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f808c0062f0 con 0x7f8094189d00 2021-02-10T11:25:25.909 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:25.905 7f809ad0d700 1 -- 172.21.2.102:0/423313847 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8094188a90 con 0x7f8094189d00 2021-02-10T11:25:25.909 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:25.905 7f80917fa700 1 -- 172.21.2.102:0/423313847 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f808c005660 con 0x7f8094189d00 2021-02-10T11:25:25.909 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:25.905 7f809ad0d700 1 -- 172.21.2.102:0/423313847 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8094189020 con 0x7f8094189d00 2021-02-10T11:25:25.909 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:25.906 7f80917fa700 1 -- 172.21.2.102:0/423313847 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f808c0064d0 con 0x7f8094189d00 2021-02-10T11:25:25.909 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:25.906 7f80917fa700 1 --2- 172.21.2.102:0/423313847 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f80740186d0 0x7f807401abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:25.909 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:25.906 7f80917fa700 1 -- 172.21.2.102:0/423313847 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 4839+0+0 (secure 0 0 0) 0x7f808c032740 con 0x7f8094189d00 2021-02-10T11:25:25.909 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:25.906 7f8093fff700 1 --2- 172.21.2.102:0/423313847 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f80740186d0 0x7f807401abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:25.909 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:25.906 7f8093fff700 1 --2- 172.21.2.102:0/423313847 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f80740186d0 0x7f807401abd0 crc :-1 s=READY pgs=671 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:25.935 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:25.936 7f809ad0d700 1 -- 172.21.2.102:0/423313847 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80940ebc50 con 0x7f8094189d00 2021-02-10T11:25:25.937 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:25.938 7f80917fa700 1 -- 172.21.2.102:0/423313847 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f808c000d30 con 0x7f8094189d00 2021-02-10T11:25:26.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:26.077 7f809ad0d700 1 -- 172.21.2.102:0/423313847 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd blacklist clear"} v 0) v1 -- 0x7f80940ebc50 con 0x7f8094189d00 2021-02-10T11:25:26.424 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:26.425 7f80917fa700 1 -- 172.21.2.102:0/423313847 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd blacklist clear"}]=0 removed all blacklist entries v61) v1 ==== 97+0+0 (secure 0 0 0) 0x7f808c0061d0 con 0x7f8094189d00 2021-02-10T11:25:26.442 INFO:teuthology.orchestra.run.gibba002.stderr: removed all blacklist entries 2021-02-10T11:25:26.444 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:26.445 7f809ad0d700 1 -- 172.21.2.102:0/423313847 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f80740186d0 msgr2=0x7f807401abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:26.444 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:26.445 7f809ad0d700 1 --2- 172.21.2.102:0/423313847 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f80740186d0 0x7f807401abd0 crc :-1 s=READY pgs=671 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:26.444 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:26.445 7f809ad0d700 1 -- 172.21.2.102:0/423313847 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8094189d00 msgr2=0x7f809418a100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:26.444 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:26.445 7f809ad0d700 1 --2- 172.21.2.102:0/423313847 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8094189d00 0x7f809418a100 secure :-1 s=READY pgs=1472 cs=0 l=1 rev1=1 rx=0x7f808c0099e0 tx=0x7f808c00bcd0).stop 2021-02-10T11:25:26.445 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:26.445 7f809ad0d700 1 -- 172.21.2.102:0/423313847 shutdown_connections 2021-02-10T11:25:26.445 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:26.445 7f809ad0d700 1 --2- 172.21.2.102:0/423313847 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f80740186d0 0x7f807401abd0 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:26.445 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:26.445 7f809ad0d700 1 --2- 172.21.2.102:0/423313847 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8094187e50 0x7f8094188250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:26.445 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:26.445 7f809ad0d700 1 --2- 172.21.2.102:0/423313847 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f809418a720 0x7f809418ab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:26.445 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:26.445 7f809ad0d700 1 --2- 172.21.2.102:0/423313847 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8094189d00 0x7f809418a100 unknown :-1 s=CLOSED pgs=1472 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:26.445 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:26.445 7f809ad0d700 1 -- 172.21.2.102:0/423313847 >> 172.21.2.102:0/423313847 conn(0x7f80940aed50 msgr2=0x7f809409c9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:25:26.445 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:26.445 7f809ad0d700 1 -- 172.21.2.102:0/423313847 shutdown_connections 2021-02-10T11:25:26.445 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:26.445 7f809ad0d700 1 -- 172.21.2.102:0/423313847 wait complete. 2021-02-10T11:25:26.452 INFO:tasks.cephfs.cephfs_test_case:['0', '1', '2', '3'] 2021-02-10T11:25:26.453 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth ls --format=json-pretty 2021-02-10T11:25:26.880 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:26.864 7fa4b5a8a700 1 Processor -- start 2021-02-10T11:25:26.880 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:26.864 7fa4b5a8a700 1 -- start start 2021-02-10T11:25:26.880 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:26.865 7fa4b5a8a700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fa4b00d1870 0x7fa4b01b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:26.881 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:26.865 7fa4b5a8a700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fa4b00f0650 0x7fa4b00cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:26.881 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:26.865 7fa4b5a8a700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4b00f0b10 con 0x7fa4b01b0a10 2021-02-10T11:25:26.881 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:26.865 7fa4b5a8a700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4b00e5ed0 con 0x7fa4b00d1870 2021-02-10T11:25:26.881 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:26.865 7fa4b5a8a700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4b00d1c70 con 0x7fa4b00f0650 2021-02-10T11:25:26.881 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:26.865 7fa4af7fe700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fa4b00d1870 0x7fa4b01b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:26.881 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:26.865 7fa4af7fe700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fa4b00d1870 0x7fa4b01b03f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:57866/0 (socket says 172.21.2.102:57866) 2021-02-10T11:25:26.881 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:26.865 7fa4af7fe700 1 -- 172.21.2.102:0/745503756 learned_addr learned my addr 172.21.2.102:0/745503756 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:25:26.881 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:26.865 7fa4af7fe700 1 -- 172.21.2.102:0/745503756 >> v2:172.21.2.113:3300/0 conn(0x7fa4b00f0650 msgr2=0x7fa4b00cfb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:25:26.881 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:26.865 7fa4af7fe700 1 --2- 172.21.2.102:0/745503756 >> v2:172.21.2.113:3300/0 conn(0x7fa4b00f0650 0x7fa4b00cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:26.881 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:26.865 7fa4af7fe700 1 -- 172.21.2.102:0/745503756 >> v1:172.21.2.113:6789/0 conn(0x7fa4b01b0a10 legacy=0x7fa4b00cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:25:26.881 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:26.865 7fa4af7fe700 1 -- 172.21.2.102:0/745503756 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4b00d1e90 con 0x7fa4b00d1870 2021-02-10T11:25:26.881 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:26.866 7fa4af7fe700 1 --2- 172.21.2.102:0/745503756 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa4b00d1870 0x7fa4b01b03f0 secure :-1 s=READY pgs=1473 cs=0 l=1 rev1=1 rx=0x7fa4a0006700 tx=0x7fa4a0009b80).ready entity=mon.0 client_cookie=2a8dc089d680f4d7 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:26.882 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:26.866 7fa4ae7fc700 1 -- 172.21.2.102:0/745503756 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa4a000c0c0 con 0x7fa4b00d1870 2021-02-10T11:25:26.882 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:26.866 7fa4ae7fc700 1 -- 172.21.2.102:0/745503756 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4a000c340 con 0x7fa4b00d1870 2021-02-10T11:25:26.882 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:26.866 7fa4ae7fc700 1 -- 172.21.2.102:0/745503756 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa4a000c860 con 0x7fa4b00d1870 2021-02-10T11:25:26.882 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:26.866 7fa4b5a8a700 1 -- 172.21.2.102:0/745503756 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa4b00d1870 msgr2=0x7fa4b01b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:26.882 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:26.866 7fa4b5a8a700 1 --2- 172.21.2.102:0/745503756 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa4b00d1870 0x7fa4b01b03f0 secure :-1 s=READY pgs=1473 cs=0 l=1 rev1=1 rx=0x7fa4a0006700 tx=0x7fa4a0009b80).stop 2021-02-10T11:25:26.882 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:26.867 7fa4b5a8a700 1 -- 172.21.2.102:0/745503756 shutdown_connections 2021-02-10T11:25:26.882 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:26.867 7fa4b5a8a700 1 --2- 172.21.2.102:0/745503756 >> v2:172.21.2.113:3300/0 conn(0x7fa4b00f0650 0x7fa4b00cfb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:26.882 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:26.867 7fa4b5a8a700 1 --2- 172.21.2.102:0/745503756 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa4b00d1870 0x7fa4b01b03f0 unknown :-1 s=CLOSED pgs=1473 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:26.882 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:26.867 7fa4b5a8a700 1 -- 172.21.2.102:0/745503756 >> 172.21.2.102:0/745503756 conn(0x7fa4b00ce450 msgr2=0x7fa4b00d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:25:26.883 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:26.867 7fa4b5a8a700 1 -- 172.21.2.102:0/745503756 shutdown_connections 2021-02-10T11:25:26.883 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:26.867 7fa4b5a8a700 1 -- 172.21.2.102:0/745503756 wait complete. 2021-02-10T11:25:26.884 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:26.881 7fa4b5a8a700 1 Processor -- start 2021-02-10T11:25:26.884 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:26.882 7fa4b5a8a700 1 -- start start 2021-02-10T11:25:26.884 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:26.882 7fa4b5a8a700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa4b018a540 0x7fa4b0197480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:26.885 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:26.882 7fa4b5a8a700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa4b018aa50 0x7fa4b00b77d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:26.885 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:26.882 7fa4b5a8a700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa4b00b7df0 0x7fa4b00b81f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:26.885 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:26.882 7fa4b5a8a700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4b01a79d0 con 0x7fa4b00b7df0 2021-02-10T11:25:26.885 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:26.882 7fa4b5a8a700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4b01a9e20 con 0x7fa4b018a540 2021-02-10T11:25:26.885 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:26.882 7fa4b5a8a700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4b01aa780 con 0x7fa4b018aa50 2021-02-10T11:25:26.885 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:26.882 7fa4affff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa4b00b7df0 0x7fa4b00b81f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:26.885 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:26.882 7fa4affff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa4b00b7df0 0x7fa4b00b81f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:57874/0 (socket says 172.21.2.102:57874) 2021-02-10T11:25:26.885 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:26.882 7fa4affff700 1 -- 172.21.2.102:0/1511992274 learned_addr learned my addr 172.21.2.102:0/1511992274 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:25:26.885 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:26.883 7fa4affff700 1 -- 172.21.2.102:0/1511992274 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa4b018aa50 msgr2=0x7fa4b00b77d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:26.885 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:26.883 7fa4affff700 1 --2- 172.21.2.102:0/1511992274 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa4b018aa50 0x7fa4b00b77d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:26.885 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:26.883 7fa4affff700 1 -- 172.21.2.102:0/1511992274 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa4b018a540 msgr2=0x7fa4b0197480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:26.886 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:26.883 7fa4affff700 1 --2- 172.21.2.102:0/1511992274 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa4b018a540 0x7fa4b0197480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:26.886 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:26.883 7fa4affff700 1 -- 172.21.2.102:0/1511992274 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4b01aa9a0 con 0x7fa4b00b7df0 2021-02-10T11:25:26.886 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:26.883 7fa4affff700 1 --2- 172.21.2.102:0/1511992274 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa4b00b7df0 0x7fa4b00b81f0 secure :-1 s=READY pgs=1474 cs=0 l=1 rev1=1 rx=0x7fa4a8009980 tx=0x7fa4a800bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:26.886 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:26.883 7fa497fff700 1 -- 172.21.2.102:0/1511992274 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa4a8005af0 con 0x7fa4b00b7df0 2021-02-10T11:25:26.886 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:26.883 7fa497fff700 1 -- 172.21.2.102:0/1511992274 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4a8005d70 con 0x7fa4b00b7df0 2021-02-10T11:25:26.886 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:26.883 7fa497fff700 1 -- 172.21.2.102:0/1511992274 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa4a80062b0 con 0x7fa4b00b7df0 2021-02-10T11:25:26.886 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:26.883 7fa4b5a8a700 1 -- 172.21.2.102:0/1511992274 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4b01aa9a0 con 0x7fa4b00b7df0 2021-02-10T11:25:26.886 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:26.883 7fa4b5a8a700 1 -- 172.21.2.102:0/1511992274 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4b01aaf30 con 0x7fa4b00b7df0 2021-02-10T11:25:26.886 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:26.884 7fa497fff700 1 -- 172.21.2.102:0/1511992274 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa4a8006030 con 0x7fa4b00b7df0 2021-02-10T11:25:26.886 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:26.884 7fa497fff700 1 --2- 172.21.2.102:0/1511992274 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa4900186c0 0x7fa49001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:26.887 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:26.884 7fa497fff700 1 -- 172.21.2.102:0/1511992274 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 4753+0+0 (secure 0 0 0) 0x7fa4a80325c0 con 0x7fa4b00b7df0 2021-02-10T11:25:26.887 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:26.884 7fa4af7fe700 1 --2- 172.21.2.102:0/1511992274 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa4900186c0 0x7fa49001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:26.887 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:26.884 7fa4af7fe700 1 --2- 172.21.2.102:0/1511992274 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa4900186c0 0x7fa49001abc0 crc :-1 s=READY pgs=672 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:26.914 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:26.915 7fa4b5a8a700 1 -- 172.21.2.102:0/1511992274 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4b01aa250 con 0x7fa4b00b7df0 2021-02-10T11:25:26.916 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:26.917 7fa497fff700 1 -- 172.21.2.102:0/1511992274 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fa4a8006570 con 0x7fa4b00b7df0 2021-02-10T11:25:27.049 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:27.049 7fa4b5a8a700 1 -- 172.21.2.102:0/1511992274 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fa4b01aa250 con 0x7fa4b00b7df0 2021-02-10T11:25:27.049 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:27.050 7fa497fff700 1 -- 172.21.2.102:0/1511992274 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v35) v1 ==== 80+0+4513 (secure 0 0 0) 0x7fa4a8006030 con 0x7fa4b00b7df0 2021-02-10T11:25:27.050 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:25:27.050 INFO:teuthology.orchestra.run.gibba002.stdout:{ 2021-02-10T11:25:27.050 INFO:teuthology.orchestra.run.gibba002.stdout: "auth_dump": [ 2021-02-10T11:25:27.050 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:25:27.050 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "mds.a", 2021-02-10T11:25:27.050 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAevSNgsgyEARAAvr0cUjjCoEHLxaZSZ2HtVA==", 2021-02-10T11:25:27.050 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:25:27.050 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow", 2021-02-10T11:25:27.050 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:25:27.050 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:25:27.050 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:25:27.051 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:25:27.051 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:25:27.051 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:25:27.051 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "mds.b", 2021-02-10T11:25:27.051 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAevSNgwOFMMRAALfb1fbX56ao1TU8ClnTW1A==", 2021-02-10T11:25:27.051 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:25:27.051 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow", 2021-02-10T11:25:27.051 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:25:27.051 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:25:27.051 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:25:27.051 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:25:27.051 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:25:27.051 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:25:27.052 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.0", 2021-02-10T11:25:27.052 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAmvSNgj+bMMBAAD3HUD6fLiMAg+++iNzLJKQ==", 2021-02-10T11:25:27.052 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:25:27.052 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:25:27.052 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:25:27.052 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:25:27.052 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:25:27.052 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:25:27.052 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:25:27.052 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.1", 2021-02-10T11:25:27.052 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQApvSNgnvc+MRAAAOxqlNwiyLHqNITOrYeFKQ==", 2021-02-10T11:25:27.052 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:25:27.052 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:25:27.052 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:25:27.053 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:25:27.053 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:25:27.053 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:25:27.053 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:25:27.053 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.2", 2021-02-10T11:25:27.053 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAsvSNgZ4SbMRAA76IOYWfbiAuBoM/4k81GxA==", 2021-02-10T11:25:27.053 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:25:27.053 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:25:27.053 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:25:27.053 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:25:27.053 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:25:27.053 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:25:27.053 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:25:27.054 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.3", 2021-02-10T11:25:27.054 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAvvSNgFk3rMRAAWCmKKyw0LDQ+UlAHZAITOw==", 2021-02-10T11:25:27.054 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:25:27.054 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:25:27.054 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:25:27.054 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:25:27.054 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:25:27.054 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:25:27.054 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:25:27.054 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.4", 2021-02-10T11:25:27.054 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQA4vSNgpKcNJBAApJMmvj5YxVHPDOmvmQONOg==", 2021-02-10T11:25:27.054 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:25:27.054 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:25:27.055 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:25:27.055 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:25:27.055 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:25:27.055 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:25:27.055 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:25:27.055 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.5", 2021-02-10T11:25:27.055 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQA7vSNgf2ujJBAAfkuYpyb3WTzc8V5kw+Wktg==", 2021-02-10T11:25:27.055 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:25:27.055 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:25:27.055 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:25:27.055 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:25:27.055 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:25:27.055 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:25:27.056 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:25:27.056 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.6", 2021-02-10T11:25:27.056 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQA+vSNgqw9uJRAAFMU0R+UxU9C3o/9byb3e0g==", 2021-02-10T11:25:27.056 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:25:27.056 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:25:27.056 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:25:27.056 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:25:27.056 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:25:27.056 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:25:27.056 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:25:27.056 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.7", 2021-02-10T11:25:27.056 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQBBvSNgvWSrJRAA2KaCHmLpzovRgYzLvyofTQ==", 2021-02-10T11:25:27.056 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:25:27.057 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:25:27.057 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:25:27.057 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:25:27.057 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:25:27.057 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:25:27.057 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:25:27.057 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "client.0", 2021-02-10T11:25:27.057 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAfvSNgZd9XIBAAvyrvl8fdKuHnJZfczfRoFw==", 2021-02-10T11:25:27.057 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:25:27.057 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow", 2021-02-10T11:25:27.057 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow r", 2021-02-10T11:25:27.057 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:25:27.057 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:25:27.057 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:25:27.058 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:25:27.058 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "client.1", 2021-02-10T11:25:27.058 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAfvSNgSyfXNhAAIduPnF7Vd/6HSkJ0MlcMSA==", 2021-02-10T11:25:27.058 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:25:27.058 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow", 2021-02-10T11:25:27.058 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow r", 2021-02-10T11:25:27.058 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:25:27.058 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:25:27.058 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:25:27.058 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:25:27.058 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "client.2", 2021-02-10T11:25:27.058 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAgvSNgiIG/ERAA9ZTamaTI7ZsPSnxBqjplCg==", 2021-02-10T11:25:27.058 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:25:27.059 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow", 2021-02-10T11:25:27.059 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow r", 2021-02-10T11:25:27.059 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:25:27.059 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:25:27.059 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:25:27.059 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:25:27.059 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "client.3", 2021-02-10T11:25:27.059 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAgvSNgOWpOKBAAlXsEB04Q/GvtL06PqRMLeA==", 2021-02-10T11:25:27.059 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:25:27.059 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow", 2021-02-10T11:25:27.059 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow r", 2021-02-10T11:25:27.059 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:25:27.059 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:25:27.060 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:25:27.060 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:25:27.060 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "client.admin", 2021-02-10T11:25:27.060 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAavSNgO3GQBhAAL+NVnYskQRKVWLgyDPY/xw==", 2021-02-10T11:25:27.060 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:25:27.060 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow *", 2021-02-10T11:25:27.060 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:25:27.060 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:25:27.060 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:25:27.060 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:25:27.060 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:25:27.060 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:25:27.060 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "mgr.x", 2021-02-10T11:25:27.060 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAdvSNgXZZgJhAA7dtBSGzH+DsBwnddZpOBJA==", 2021-02-10T11:25:27.061 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:25:27.061 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow *", 2021-02-10T11:25:27.061 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow profile mgr", 2021-02-10T11:25:27.061 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:25:27.061 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:25:27.061 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:25:27.061 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:25:27.061 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "mgr.y", 2021-02-10T11:25:27.061 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAdvSNgVqQFDRAAwo6YEdWZCAr5GdOWVL/Qvg==", 2021-02-10T11:25:27.061 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:25:27.061 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow *", 2021-02-10T11:25:27.061 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow profile mgr", 2021-02-10T11:25:27.061 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:25:27.062 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:25:27.062 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:25:27.062 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:25:27.062 INFO:teuthology.orchestra.run.gibba002.stdout:} 2021-02-10T11:25:27.063 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:27.052 7fa4b5a8a700 1 -- 172.21.2.102:0/1511992274 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa4900186c0 msgr2=0x7fa49001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:27.063 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:27.052 7fa4b5a8a700 1 --2- 172.21.2.102:0/1511992274 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa4900186c0 0x7fa49001abc0 crc :-1 s=READY pgs=672 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:27.063 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:27.052 7fa4b5a8a700 1 -- 172.21.2.102:0/1511992274 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa4b00b7df0 msgr2=0x7fa4b00b81f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:27.063 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:27.052 7fa4b5a8a700 1 --2- 172.21.2.102:0/1511992274 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa4b00b7df0 0x7fa4b00b81f0 secure :-1 s=READY pgs=1474 cs=0 l=1 rev1=1 rx=0x7fa4a8009980 tx=0x7fa4a800bcd0).stop 2021-02-10T11:25:27.063 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:27.052 7fa4b5a8a700 1 -- 172.21.2.102:0/1511992274 shutdown_connections 2021-02-10T11:25:27.063 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:27.052 7fa4b5a8a700 1 --2- 172.21.2.102:0/1511992274 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa4b00b7df0 0x7fa4b00b81f0 unknown :-1 s=CLOSED pgs=1474 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:27.063 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:27.052 7fa4b5a8a700 1 --2- 172.21.2.102:0/1511992274 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa4b018aa50 0x7fa4b00b77d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:27.063 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:27.052 7fa4b5a8a700 1 --2- 172.21.2.102:0/1511992274 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa4900186c0 0x7fa49001abc0 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:27.063 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:27.052 7fa4b5a8a700 1 --2- 172.21.2.102:0/1511992274 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa4b018a540 0x7fa4b0197480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:27.063 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:27.052 7fa4b5a8a700 1 -- 172.21.2.102:0/1511992274 >> 172.21.2.102:0/1511992274 conn(0x7fa4b00efc70 msgr2=0x7fa4b00ef410 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:25:27.064 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:27.052 7fa4b5a8a700 1 -- 172.21.2.102:0/1511992274 shutdown_connections 2021-02-10T11:25:27.064 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:27.052 7fa4b5a8a700 1 -- 172.21.2.102:0/1511992274 wait complete. 2021-02-10T11:25:27.064 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:25:27.492 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:27.475 7f04c854e700 1 Processor -- start 2021-02-10T11:25:27.492 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:27.475 7f04c854e700 1 -- start start 2021-02-10T11:25:27.492 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:27.476 7f04c854e700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f04c00d1c70 0x7f04c00f20a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:27.493 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:27.476 7f04c854e700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04c00f0b10 con 0x7f04c00d1870 2021-02-10T11:25:27.493 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:27.476 7f04c854e700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04c01b0be0 con 0x7f04c00d0340 2021-02-10T11:25:27.493 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:27.476 7f04c854e700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04c00e5ed0 con 0x7f04c00d1c70 2021-02-10T11:25:27.493 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:27.476 7f04c636a700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f04c00d1870 0x7f04c01b03f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:50102/0 (socket says 172.21.2.102:50102) 2021-02-10T11:25:27.493 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:27.476 7f04c636a700 1 -- 172.21.2.102:0/2591386305 learned_addr learned my addr 172.21.2.102:0/2591386305 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:25:27.493 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:27.476 7f04c5368700 1 -- 172.21.2.102:0/2591386305 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 540397518 0 0) 0x7f04bc0010b0 con 0x7f04c00d1870 2021-02-10T11:25:27.493 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:27.477 7f04c5368700 1 -- 172.21.2.102:0/2591386305 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04ac002fe0 con 0x7f04c00d1870 2021-02-10T11:25:27.493 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:27.477 7f04c6b6b700 1 --2- 172.21.2.102:0/2591386305 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f04c00d1c70 0x7f04c00f20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:27.493 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:27.477 7f04c5368700 1 -- 172.21.2.102:0/2591386305 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 212362332 0 0) 0x7f04bc0010b0 con 0x7f04c00d1870 2021-02-10T11:25:27.493 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:27.477 7f04c5368700 1 -- 172.21.2.102:0/2591386305 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f04c00d1c70 msgr2=0x7f04c00f20a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:25:27.493 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:27.477 7f04c5368700 1 --2- 172.21.2.102:0/2591386305 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f04c00d1c70 0x7f04c00f20a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:27.494 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:27.477 7f04c5368700 1 -- 172.21.2.102:0/2591386305 >> v1:172.21.2.113:6789/0 conn(0x7f04c00d0340 legacy=0x7f04c00ef640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:27.494 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:27.477 7f04c5368700 1 -- 172.21.2.102:0/2591386305 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04c00efe30 con 0x7f04c00d1870 2021-02-10T11:25:27.494 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:27.477 7f04c5368700 1 -- 172.21.2.102:0/2591386305 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7f04bc0015b0 con 0x7f04c00d1870 2021-02-10T11:25:27.494 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:27.477 7f04c5368700 1 -- 172.21.2.102:0/2591386305 >> v1:172.21.2.102:6789/0 conn(0x7f04c00d1870 legacy=0x7f04c01b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:27.494 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:27.477 7f04c5368700 1 --2- 172.21.2.102:0/2591386305 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f04ac003a70 0x7f04ac005ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:27.494 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:27.477 7f04c5368700 1 --2- 172.21.2.102:0/2591386305 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f04ac006780 0x7f04ac008c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:27.494 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:27.477 7f04c5368700 1 --2- 172.21.2.102:0/2591386305 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f04ac009510 0x7f04ac00ba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:27.494 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:27.477 7f04c5368700 1 -- 172.21.2.102:0/2591386305 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04ac00c2e0 con 0x7f04ac009510 2021-02-10T11:25:27.494 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:27.477 7f04c5368700 1 -- 172.21.2.102:0/2591386305 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04ac00c500 con 0x7f04ac003a70 2021-02-10T11:25:27.494 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:27.477 7f04c5368700 1 -- 172.21.2.102:0/2591386305 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04ac00c720 con 0x7f04ac006780 2021-02-10T11:25:27.494 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:27.477 7f04c6b6b700 1 --2- 172.21.2.102:0/2591386305 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f04ac009510 0x7f04ac00ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:27.494 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:27.477 7f04c5b69700 1 --2- 172.21.2.102:0/2591386305 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f04ac006780 0x7f04ac008c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:27.495 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:27.477 7f04c6b6b700 1 -- 172.21.2.102:0/2591386305 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f04ac006780 msgr2=0x7f04ac008c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:25:27.495 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:27.477 7f04c6b6b700 1 --2- 172.21.2.102:0/2591386305 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f04ac006780 0x7f04ac008c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:27.495 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:27.477 7f04c6b6b700 1 -- 172.21.2.102:0/2591386305 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f04ac003a70 msgr2=0x7f04ac005ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:25:27.495 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:27.477 7f04c636a700 1 --2- 172.21.2.102:0/2591386305 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f04ac003a70 0x7f04ac005ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:27.495 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:27.477 7f04c6b6b700 1 --2- 172.21.2.102:0/2591386305 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f04ac003a70 0x7f04ac005ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:27.495 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:27.477 7f04c6b6b700 1 -- 172.21.2.102:0/2591386305 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04ac00c9c0 con 0x7f04ac009510 2021-02-10T11:25:27.495 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:27.477 7f04c636a700 1 --2- 172.21.2.102:0/2591386305 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f04ac003a70 0x7f04ac005ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:25:27.495 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:27.477 7f04c6b6b700 1 --2- 172.21.2.102:0/2591386305 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f04ac009510 0x7f04ac00ba30 secure :-1 s=READY pgs=1476 cs=0 l=1 rev1=1 rx=0x7f04b804da40 tx=0x7f04b804dde0).ready entity=mon.0 client_cookie=2e10b2e35c5ed2d0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:27.495 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:27.478 7f04c5368700 1 -- 172.21.2.102:0/2591386305 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f04b8006ac0 con 0x7f04ac009510 2021-02-10T11:25:27.495 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:27.478 7f04c5368700 1 -- 172.21.2.102:0/2591386305 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04b804f300 con 0x7f04ac009510 2021-02-10T11:25:27.495 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:27.478 7f04c5368700 1 -- 172.21.2.102:0/2591386305 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f04b8000f90 con 0x7f04ac009510 2021-02-10T11:25:27.496 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:27.478 7f04c854e700 1 -- 172.21.2.102:0/2591386305 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f04ac009510 msgr2=0x7f04ac00ba30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:27.496 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:27.478 7f04c854e700 1 --2- 172.21.2.102:0/2591386305 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f04ac009510 0x7f04ac00ba30 secure :-1 s=READY pgs=1476 cs=0 l=1 rev1=1 rx=0x7f04b804da40 tx=0x7f04b804dde0).stop 2021-02-10T11:25:27.496 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:27.478 7f04c6b6b700 1 -- 172.21.2.102:0/2591386305 reap_dead start 2021-02-10T11:25:27.496 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:27.478 7f04c854e700 1 -- 172.21.2.102:0/2591386305 shutdown_connections 2021-02-10T11:25:27.496 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:27.478 7f04c854e700 1 -- 172.21.2.102:0/2591386305 >> 172.21.2.102:0/2591386305 conn(0x7f04c00ce450 msgr2=0x7f04c00d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:25:27.496 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:27.478 7f04c854e700 1 -- 172.21.2.102:0/2591386305 shutdown_connections 2021-02-10T11:25:27.496 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:27.478 7f04c854e700 1 -- 172.21.2.102:0/2591386305 wait complete. 2021-02-10T11:25:27.496 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:27.493 7f04c854e700 1 Processor -- start 2021-02-10T11:25:27.496 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:27.493 7f04c854e700 1 -- start start 2021-02-10T11:25:27.497 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:27.493 7f04c854e700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f04c01942b0 0x7f04c00b3550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:27.498 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:27.493 7f04c854e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f04c0195070 0x7f04c0195470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:27.498 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:27.493 7f04c854e700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f04c00bb8e0 0x7f04c00bbce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:27.498 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:27.493 7f04c854e700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04c01946b0 con 0x7f04c0195070 2021-02-10T11:25:27.498 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:27.493 7f04c854e700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04c00b3b70 con 0x7f04c00bb8e0 2021-02-10T11:25:27.498 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:27.493 7f04c854e700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04c00b3d90 con 0x7f04c01942b0 2021-02-10T11:25:27.498 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:27.494 7f04c5b69700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f04c0195070 0x7f04c0195470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:27.498 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:27.494 7f04c5b69700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f04c0195070 0x7f04c0195470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:57890/0 (socket says 172.21.2.102:57890) 2021-02-10T11:25:27.498 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:27.494 7f04c5b69700 1 -- 172.21.2.102:0/736258625 learned_addr learned my addr 172.21.2.102:0/736258625 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:25:27.498 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:27.494 7f04c5b69700 1 -- 172.21.2.102:0/736258625 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f04c01942b0 msgr2=0x7f04c00b3550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:27.498 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:27.494 7f04c5b69700 1 --2- 172.21.2.102:0/736258625 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f04c01942b0 0x7f04c00b3550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:27.498 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:27.494 7f04c5b69700 1 -- 172.21.2.102:0/736258625 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f04c00bb8e0 msgr2=0x7f04c00bbce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:27.499 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:27.494 7f04c5b69700 1 --2- 172.21.2.102:0/736258625 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f04c00bb8e0 0x7f04c00bbce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:27.499 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:27.494 7f04c5b69700 1 -- 172.21.2.102:0/736258625 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04c01a49e0 con 0x7f04c0195070 2021-02-10T11:25:27.499 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:27.494 7f04c5b69700 1 --2- 172.21.2.102:0/736258625 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f04c0195070 0x7f04c0195470 secure :-1 s=READY pgs=1477 cs=0 l=1 rev1=1 rx=0x7f04b400b650 tx=0x7f04b400b9f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:27.499 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:27.494 7f04b2ffd700 1 -- 172.21.2.102:0/736258625 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f04b4006b60 con 0x7f04c0195070 2021-02-10T11:25:27.499 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:27.494 7f04b2ffd700 1 -- 172.21.2.102:0/736258625 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04b400bde0 con 0x7f04c0195070 2021-02-10T11:25:27.499 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:27.494 7f04b2ffd700 1 -- 172.21.2.102:0/736258625 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f04b4007330 con 0x7f04c0195070 2021-02-10T11:25:27.499 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:27.494 7f04c854e700 1 -- 172.21.2.102:0/736258625 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04c01a49e0 con 0x7f04c0195070 2021-02-10T11:25:27.499 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:27.495 7f04c854e700 1 -- 172.21.2.102:0/736258625 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04c00bc5f0 con 0x7f04c0195070 2021-02-10T11:25:27.499 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:27.495 7f04b2ffd700 1 -- 172.21.2.102:0/736258625 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f04b400a000 con 0x7f04c0195070 2021-02-10T11:25:27.499 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:27.495 7f04b2ffd700 1 --2- 172.21.2.102:0/736258625 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f04a80186c0 0x7f04a801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:27.500 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:27.495 7f04b2ffd700 1 -- 172.21.2.102:0/736258625 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 4753+0+0 (secure 0 0 0) 0x7f04b4032080 con 0x7f04c0195070 2021-02-10T11:25:27.500 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:27.495 7f04c636a700 1 --2- 172.21.2.102:0/736258625 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f04a80186c0 0x7f04a801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:27.500 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:27.496 7f04c636a700 1 --2- 172.21.2.102:0/736258625 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f04a80186c0 0x7f04a801abc0 crc :-1 s=READY pgs=673 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:27.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:27.526 7f04c854e700 1 -- 172.21.2.102:0/736258625 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04c00bc5f0 con 0x7f04c0195070 2021-02-10T11:25:27.527 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:27.528 7f04b2ffd700 1 -- 172.21.2.102:0/736258625 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f04b4006b60 con 0x7f04c0195070 2021-02-10T11:25:27.653 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:27.654 7f04c854e700 1 -- 172.21.2.102:0/736258625 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f04c00bc5f0 con 0x7f04c0195070 2021-02-10T11:25:27.763 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:27.655 7f04b2ffd700 1 -- 172.21.2.102:0/736258625 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v61) v1 ==== 74+0+11145 (secure 0 0 0) 0x7f04b400a000 con 0x7f04c0195070 2021-02-10T11:25:27.763 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:27.656 7f04c854e700 1 -- 172.21.2.102:0/736258625 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f04a80186c0 msgr2=0x7f04a801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:27.763 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:27.656 7f04c854e700 1 --2- 172.21.2.102:0/736258625 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f04a80186c0 0x7f04a801abc0 crc :-1 s=READY pgs=673 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:27.763 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:27.656 7f04c854e700 1 -- 172.21.2.102:0/736258625 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f04c0195070 msgr2=0x7f04c0195470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:27.763 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:27.656 7f04c854e700 1 --2- 172.21.2.102:0/736258625 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f04c0195070 0x7f04c0195470 secure :-1 s=READY pgs=1477 cs=0 l=1 rev1=1 rx=0x7f04b400b650 tx=0x7f04b400b9f0).stop 2021-02-10T11:25:27.763 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:27.657 7f04c854e700 1 -- 172.21.2.102:0/736258625 shutdown_connections 2021-02-10T11:25:27.763 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:27.657 7f04c854e700 1 --2- 172.21.2.102:0/736258625 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f04a80186c0 0x7f04a801abc0 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:27.763 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:27.657 7f04c854e700 1 --2- 172.21.2.102:0/736258625 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f04c00bb8e0 0x7f04c00bbce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:27.763 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:27.657 7f04c854e700 1 --2- 172.21.2.102:0/736258625 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f04c0195070 0x7f04c0195470 unknown :-1 s=CLOSED pgs=1477 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:27.763 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:27.657 7f04c854e700 1 --2- 172.21.2.102:0/736258625 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f04c01942b0 0x7f04c00b3550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:27.763 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:27.657 7f04c854e700 1 -- 172.21.2.102:0/736258625 >> 172.21.2.102:0/736258625 conn(0x7f04c00b3040 msgr2=0x7f04c00c3250 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:25:27.764 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:27.657 7f04c854e700 1 -- 172.21.2.102:0/736258625 shutdown_connections 2021-02-10T11:25:27.764 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:27.657 7f04c854e700 1 -- 172.21.2.102:0/736258625 wait complete. 2021-02-10T11:25:27.764 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:25:27.764 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":61,"fsid":"68db30ea-dbce-4aef-a6df-f01c6fea0513","created":"2021-02-10 11:03:06.331557","modified":"2021-02-10 11:25:26.420898","last_up_change":"2021-02-10 11:03:32.690562","last_in_change":"2021-02-10 11:03:32.690562","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:03:37.322519","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"be691b4d-a039-408a-af80-9d53f02b6f1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6818","nonce":36088},{"type":"v1","addr":"172.21.2.102:6820","nonce":36088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6822","nonce":36088},{"type":"v1","addr":"172.21.2.102:6824","nonce":36088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6830","nonce":36088},{"type":"v1","addr":"172.21.2.102:6832","nonce":36088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6826","nonce":36088},{"type":"v1","addr":"172.21.2.102:6828","nonce":36088}]},"public_addr":"172.21.2.102:6820/36088","cluster_addr":"172.21.2.102:6824/36088","heartbeat_back_addr":"172.21.2.102:6832/36088","heartbeat_front_addr":"172.21.2.102:6828/36088","state":["exists","up"]},{"osd":1,"uuid":"64e3ccc8-9c69-4d7c-865e-af8e83822635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6819","nonce":36087},{"type":"v1","addr":"172.21.2.102:6821","nonce":36087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6823","nonce":36087},{"type":"v1","addr":"172.21.2.102:6825","nonce":36087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6831","nonce":36087},{"type":"v1","addr":"172.21.2.102:6833","nonce":36087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6827","nonce":36087},{"type":"v1","addr":"172.21.2.102:6829","nonce":36087}]},"public_addr":"172.21.2.102:6821/36087","cluster_addr":"172.21.2.102:6825/36087","heartbeat_back_addr":"172.21.2.102:6833/36087","heartbeat_front_addr":"172.21.2.102:6829/36087","state":["exists","up"]},{"osd":2,"uuid":"153700af-747f-481f-bdf4-d6072d3a842f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6810","nonce":36089},{"type":"v1","addr":"172.21.2.102:6811","nonce":36089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6812","nonce":36089},{"type":"v1","addr":"172.21.2.102:6813","nonce":36089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6816","nonce":36089},{"type":"v1","addr":"172.21.2.102:6817","nonce":36089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6814","nonce":36089},{"type":"v1","addr":"172.21.2.102:6815","nonce":36089}]},"public_addr":"172.21.2.102:6811/36089","cluster_addr":"172.21.2.102:6813/36089","heartbeat_back_addr":"172.21.2.102:6817/36089","heartbeat_front_addr":"172.21.2.102:6815/36089","state":["exists","up"]},{"osd":3,"uuid":"cbc678f4-83a9-45e6-8830-2f7c232b1ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6802","nonce":36090},{"type":"v1","addr":"172.21.2.102:6803","nonce":36090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6804","nonce":36090},{"type":"v1","addr":"172.21.2.102:6805","nonce":36090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6808","nonce":36090},{"type":"v1","addr":"172.21.2.102:6809","nonce":36090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6806","nonce":36090},{"type":"v1","addr":"172.21.2.102:6807","nonce":36090}]},"public_addr":"172.21.2.102:6803/36090","cluster_addr":"172.21.2.102:6805/36090","heartbeat_back_addr":"172.21.2.102:6809/36090","heartbeat_front_addr":"172.21.2.102:6807/36090","state":["exists","up"]},{"osd":4,"uuid":"c01e4008-b258-4f10-afd2-9e7f37214351","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6808","nonce":36368},{"type":"v1","addr":"172.21.2.113:6811","nonce":36368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6814","nonce":36368},{"type":"v1","addr":"172.21.2.113:6818","nonce":36368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6826","nonce":36368},{"type":"v1","addr":"172.21.2.113:6828","nonce":36368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6820","nonce":36368},{"type":"v1","addr":"172.21.2.113:6824","nonce":36368}]},"public_addr":"172.21.2.113:6811/36368","cluster_addr":"172.21.2.113:6818/36368","heartbeat_back_addr":"172.21.2.113:6828/36368","heartbeat_front_addr":"172.21.2.113:6824/36368","state":["exists","up"]},{"osd":5,"uuid":"551f6007-daf8-4a98-8ee0-c2b16bcccb5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6802","nonce":36366},{"type":"v1","addr":"172.21.2.113:6805","nonce":36366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6807","nonce":36366},{"type":"v1","addr":"172.21.2.113:6810","nonce":36366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6819","nonce":36366},{"type":"v1","addr":"172.21.2.113:6822","nonce":36366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6812","nonce":36366},{"type":"v1","addr":"172.21.2.113:6815","nonce":36366}]},"public_addr":"172.21.2.113:6805/36366","cluster_addr":"172.21.2.113:6810/36366","heartbeat_back_addr":"172.21.2.113:6822/36366","heartbeat_front_addr":"172.21.2.113:6815/36366","state":["exists","up"]},{"osd":6,"uuid":"bf67c040-0e50-48be-b102-c3943bc9ebef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6800","nonce":36367},{"type":"v1","addr":"172.21.2.113:6801","nonce":36367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6803","nonce":36367},{"type":"v1","addr":"172.21.2.113:6804","nonce":36367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6813","nonce":36367},{"type":"v1","addr":"172.21.2.113:6816","nonce":36367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6806","nonce":36367},{"type":"v1","addr":"172.21.2.113:6809","nonce":36367}]},"public_addr":"172.21.2.113:6801/36367","cluster_addr":"172.21.2.113:6804/36367","heartbeat_back_addr":"172.21.2.113:6816/36367","heartbeat_front_addr":"172.21.2.113:6809/36367","state":["exists","up"]},{"osd":7,"uuid":"f5461f80-018d-428f-be89-2eb2bcbe7bb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6817","nonce":36365},{"type":"v1","addr":"172.21.2.113:6821","nonce":36365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6823","nonce":36365},{"type":"v1","addr":"172.21.2.113:6825","nonce":36365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6830","nonce":36365},{"type":"v1","addr":"172.21.2.113:6831","nonce":36365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6827","nonce":36365},{"type":"v1","addr":"172.21.2.113:6829","nonce":36365}]},"public_addr":"172.21.2.113:6821/36365","cluster_addr":"172.21.2.113:6825/36365","heartbeat_back_addr":"172.21.2.113:6831/36365","heartbeat_front_addr":"172.21.2.113:6829/36365","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:25:27.764 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:03:37.322519', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2021-02-10T11:25:27.765 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:25:28.196 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:28.180 7f49365e4700 1 Processor -- start 2021-02-10T11:25:28.196 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:28.180 7f49365e4700 1 -- start start 2021-02-10T11:25:28.196 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:28.181 7f49365e4700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f49300d1870 0x7f49301b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:28.196 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:28.181 7f49365e4700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49300fe1c0 con 0x7f49301b0ae0 2021-02-10T11:25:28.196 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:28.181 7f49365e4700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4930109a60 con 0x7f49300cff70 2021-02-10T11:25:28.196 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:28.181 7f49365e4700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4930088070 con 0x7f49300d1870 2021-02-10T11:25:28.196 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:28.181 7f492ffff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f49300d1870 0x7f49301b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:28.196 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:28.181 7f492f7fe700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f49301b0ae0 0x7f4930108940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:50122/0 (socket says 172.21.2.102:50122) 2021-02-10T11:25:28.196 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:28.181 7f492f7fe700 1 -- 172.21.2.102:0/3962496339 learned_addr learned my addr 172.21.2.102:0/3962496339 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:25:28.196 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:28.181 7f492effd700 1 -- 172.21.2.102:0/3962496339 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 599675851 0 0) 0x7f49240009b0 con 0x7f49301b0ae0 2021-02-10T11:25:28.197 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:28.181 7f492effd700 1 -- 172.21.2.102:0/3962496339 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4918002fe0 con 0x7f49301b0ae0 2021-02-10T11:25:28.197 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:28.181 7f492ffff700 1 -- 172.21.2.102:0/3962496339 >> v1:172.21.2.113:6789/0 conn(0x7f49300cff70 legacy=0x7f4930109130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:25:28.197 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:28.181 7f492ffff700 1 -- 172.21.2.102:0/3962496339 >> v1:172.21.2.102:6789/0 conn(0x7f49301b0ae0 legacy=0x7f4930108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:28.197 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:28.181 7f492ffff700 1 -- 172.21.2.102:0/3962496339 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4930088260 con 0x7f49300d1870 2021-02-10T11:25:28.197 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:28.181 7f492ffff700 1 --2- 172.21.2.102:0/3962496339 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f49300d1870 0x7f49301b04c0 secure :-1 s=READY pgs=1479 cs=0 l=1 rev1=1 rx=0x7f4920006650 tx=0x7f4920009a80).ready entity=mon.0 client_cookie=389febda33ebba0b server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:28.197 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:28.182 7f492effd700 1 -- 172.21.2.102:0/3962496339 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f49200009b0 con 0x7f49300d1870 2021-02-10T11:25:28.197 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:28.182 7f492effd700 1 -- 172.21.2.102:0/3962496339 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f492000c0c0 con 0x7f49300d1870 2021-02-10T11:25:28.197 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:28.182 7f492effd700 1 -- 172.21.2.102:0/3962496339 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f492000c520 con 0x7f49300d1870 2021-02-10T11:25:28.197 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:28.182 7f49365e4700 1 -- 172.21.2.102:0/3962496339 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f49300d1870 msgr2=0x7f49301b04c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:28.197 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:28.182 7f49365e4700 1 --2- 172.21.2.102:0/3962496339 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f49300d1870 0x7f49301b04c0 secure :-1 s=READY pgs=1479 cs=0 l=1 rev1=1 rx=0x7f4920006650 tx=0x7f4920009a80).stop 2021-02-10T11:25:28.197 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:28.182 7f49365e4700 1 -- 172.21.2.102:0/3962496339 shutdown_connections 2021-02-10T11:25:28.197 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:28.182 7f49365e4700 1 --2- 172.21.2.102:0/3962496339 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f49300d1870 0x7f49301b04c0 unknown :-1 s=CLOSED pgs=1479 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:28.198 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:28.182 7f49365e4700 1 -- 172.21.2.102:0/3962496339 >> 172.21.2.102:0/3962496339 conn(0x7f49300ce420 msgr2=0x7f49300d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:25:28.198 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:28.182 7f49365e4700 1 -- 172.21.2.102:0/3962496339 shutdown_connections 2021-02-10T11:25:28.198 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:28.182 7f49365e4700 1 -- 172.21.2.102:0/3962496339 wait complete. 2021-02-10T11:25:28.199 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:28.197 7f49365e4700 1 Processor -- start 2021-02-10T11:25:28.199 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:28.197 7f49365e4700 1 -- start start 2021-02-10T11:25:28.200 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:28.197 7f49365e4700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f49300c3820 0x7f4930084cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:28.200 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:28.197 7f49365e4700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f49300a0290 0x7f49300852d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:28.200 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:28.197 7f49365e4700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4930086270 0x7f4930086670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:28.200 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:28.197 7f49365e4700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49300c0700 con 0x7f4930086270 2021-02-10T11:25:28.200 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:28.197 7f49365e4700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49300858f0 con 0x7f49300a0290 2021-02-10T11:25:28.200 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:28.197 7f49365e4700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4930085b10 con 0x7f49300c3820 2021-02-10T11:25:28.200 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:28.198 7f4934c01700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4930086270 0x7f4930086670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:28.200 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:28.198 7f4934c01700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4930086270 0x7f4930086670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:57908/0 (socket says 172.21.2.102:57908) 2021-02-10T11:25:28.200 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:28.198 7f4934c01700 1 -- 172.21.2.102:0/3420200090 learned_addr learned my addr 172.21.2.102:0/3420200090 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:25:28.200 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:28.198 7f492f7fe700 1 --2- 172.21.2.102:0/3420200090 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f49300a0290 0x7f49300852d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:28.200 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:28.198 7f492ffff700 1 --2- 172.21.2.102:0/3420200090 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f49300c3820 0x7f4930084cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:28.200 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:28.198 7f4934c01700 1 -- 172.21.2.102:0/3420200090 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f49300c3820 msgr2=0x7f4930084cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:28.200 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:28.198 7f4934c01700 1 --2- 172.21.2.102:0/3420200090 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f49300c3820 0x7f4930084cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:28.201 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:28.198 7f4934c01700 1 -- 172.21.2.102:0/3420200090 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f49300a0290 msgr2=0x7f49300852d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:28.201 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:28.198 7f4934c01700 1 --2- 172.21.2.102:0/3420200090 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f49300a0290 0x7f49300852d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:28.201 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:28.198 7f4934c01700 1 -- 172.21.2.102:0/3420200090 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4930086c90 con 0x7f4930086270 2021-02-10T11:25:28.201 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:28.198 7f492f7fe700 1 --2- 172.21.2.102:0/3420200090 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f49300a0290 0x7f49300852d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:25:28.201 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:28.198 7f4934c01700 1 --2- 172.21.2.102:0/3420200090 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4930086270 0x7f4930086670 secure :-1 s=READY pgs=1480 cs=0 l=1 rev1=1 rx=0x7f4928009940 tx=0x7f492800bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:28.201 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:28.198 7f492cff9700 1 -- 172.21.2.102:0/3420200090 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4928000f90 con 0x7f4930086270 2021-02-10T11:25:28.201 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:28.198 7f492cff9700 1 -- 172.21.2.102:0/3420200090 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f492800c2f0 con 0x7f4930086270 2021-02-10T11:25:28.201 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:28.198 7f492cff9700 1 -- 172.21.2.102:0/3420200090 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f492800c810 con 0x7f4930086270 2021-02-10T11:25:28.201 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:28.198 7f49365e4700 1 -- 172.21.2.102:0/3420200090 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4930086c90 con 0x7f4930086270 2021-02-10T11:25:28.201 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:28.198 7f49365e4700 1 -- 172.21.2.102:0/3420200090 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49301a2f00 con 0x7f4930086270 2021-02-10T11:25:28.201 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:28.199 7f492cff9700 1 -- 172.21.2.102:0/3420200090 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4928001220 con 0x7f4930086270 2021-02-10T11:25:28.201 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:28.199 7f492cff9700 1 --2- 172.21.2.102:0/3420200090 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f49100186c0 0x7f491001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:28.201 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:28.199 7f492cff9700 1 -- 172.21.2.102:0/3420200090 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 4753+0+0 (secure 0 0 0) 0x7f49280325a0 con 0x7f4930086270 2021-02-10T11:25:28.201 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:28.199 7f492ffff700 1 --2- 172.21.2.102:0/3420200090 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f49100186c0 0x7f491001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:28.202 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:28.200 7f492ffff700 1 --2- 172.21.2.102:0/3420200090 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f49100186c0 0x7f491001abc0 crc :-1 s=READY pgs=674 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:28.229 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:28.230 7f49365e4700 1 -- 172.21.2.102:0/3420200090 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4930082a70 con 0x7f4930086270 2021-02-10T11:25:28.232 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:28.233 7f492cff9700 1 -- 172.21.2.102:0/3420200090 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f4928000f40 con 0x7f4930086270 2021-02-10T11:25:28.359 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:28.360 7f49365e4700 1 -- 172.21.2.102:0/3420200090 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f4930082a70 con 0x7f4930086270 2021-02-10T11:25:28.360 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:28.360 7f492cff9700 1 -- 172.21.2.102:0/3420200090 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v61) v1 ==== 92+0+10 (secure 0 0 0) 0x7f492801f860 con 0x7f4930086270 2021-02-10T11:25:28.360 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:25:28.362 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:28.362 7f49365e4700 1 -- 172.21.2.102:0/3420200090 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f49100186c0 msgr2=0x7f491001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:28.362 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:28.362 7f49365e4700 1 --2- 172.21.2.102:0/3420200090 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f49100186c0 0x7f491001abc0 crc :-1 s=READY pgs=674 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:28.362 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:28.362 7f49365e4700 1 -- 172.21.2.102:0/3420200090 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4930086270 msgr2=0x7f4930086670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:28.362 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:28.362 7f49365e4700 1 --2- 172.21.2.102:0/3420200090 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4930086270 0x7f4930086670 secure :-1 s=READY pgs=1480 cs=0 l=1 rev1=1 rx=0x7f4928009940 tx=0x7f492800bcf0).stop 2021-02-10T11:25:28.362 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:28.362 7f49365e4700 1 -- 172.21.2.102:0/3420200090 shutdown_connections 2021-02-10T11:25:28.362 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:28.362 7f49365e4700 1 --2- 172.21.2.102:0/3420200090 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4930086270 0x7f4930086670 unknown :-1 s=CLOSED pgs=1480 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:28.362 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:28.362 7f49365e4700 1 --2- 172.21.2.102:0/3420200090 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f49100186c0 0x7f491001abc0 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:28.362 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:28.362 7f49365e4700 1 --2- 172.21.2.102:0/3420200090 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f49300a0290 0x7f49300852d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:28.362 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:28.362 7f49365e4700 1 --2- 172.21.2.102:0/3420200090 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f49300c3820 0x7f4930084cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:28.363 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:28.362 7f49365e4700 1 -- 172.21.2.102:0/3420200090 >> 172.21.2.102:0/3420200090 conn(0x7f49300872c0 msgr2=0x7f4930095700 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:25:28.363 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:28.363 7f49365e4700 1 -- 172.21.2.102:0/3420200090 shutdown_connections 2021-02-10T11:25:28.363 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:28.363 7f49365e4700 1 -- 172.21.2.102:0/3420200090 wait complete. 2021-02-10T11:25:28.372 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --format=json-pretty osd lspools 2021-02-10T11:25:28.802 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:28.786 7fd72d9a3700 1 Processor -- start 2021-02-10T11:25:28.802 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:28.786 7fd72d9a3700 1 -- start start 2021-02-10T11:25:28.802 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:28.786 7fd72d9a3700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fd7280d1870 0x7fd7281b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:28.802 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:28.786 7fd72d9a3700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd7281b0a10 0x7fd7280cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:28.802 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:28.786 7fd72d9a3700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fd7280d1c70 0x7fd7280cf950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:28.802 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:28.786 7fd72d9a3700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7280fe200 con 0x7fd7280d1c70 2021-02-10T11:25:28.802 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:28.786 7fd72d9a3700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7280e5f10 con 0x7fd7280d1870 2021-02-10T11:25:28.802 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:28.786 7fd72d9a3700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7280f0b50 con 0x7fd7281b0a10 2021-02-10T11:25:28.802 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:28.787 7fd727d75700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fd7280d1c70 0x7fd7280cf950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:28.803 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:28.787 7fd727d75700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fd7280d1c70 0x7fd7280cf950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:57912/0 (socket says 172.21.2.102:57912) 2021-02-10T11:25:28.803 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:28.787 7fd727d75700 1 -- 172.21.2.102:0/2834562167 learned_addr learned my addr 172.21.2.102:0/2834562167 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:25:28.803 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:28.787 7fd727574700 1 --2- 172.21.2.102:0/2834562167 >> v2:172.21.2.113:3300/0 conn(0x7fd7280d1870 0x7fd7281b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:28.803 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:28.787 7fd726d73700 1 --2- 172.21.2.102:0/2834562167 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd7281b0a10 0x7fd7280cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:28.803 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:28.787 7fd727d75700 1 -- 172.21.2.102:0/2834562167 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd7281b0a10 msgr2=0x7fd7280cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:25:28.803 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:28.787 7fd727d75700 1 --2- 172.21.2.102:0/2834562167 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd7281b0a10 0x7fd7280cf330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:28.803 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:28.787 7fd727d75700 1 -- 172.21.2.102:0/2834562167 >> v2:172.21.2.113:3300/0 conn(0x7fd7280d1870 msgr2=0x7fd7281b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:25:28.803 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:28.787 7fd727d75700 1 --2- 172.21.2.102:0/2834562167 >> v2:172.21.2.113:3300/0 conn(0x7fd7280d1870 0x7fd7281b03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:28.803 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:28.787 7fd727d75700 1 -- 172.21.2.102:0/2834562167 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7280cff70 con 0x7fd7280d1c70 2021-02-10T11:25:28.803 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:28.787 7fd727574700 1 --2- 172.21.2.102:0/2834562167 >> v2:172.21.2.113:3300/0 conn(0x7fd7280d1870 0x7fd7281b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:25:28.803 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:28.787 7fd726d73700 1 --2- 172.21.2.102:0/2834562167 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd7281b0a10 0x7fd7280cf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:25:28.803 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:28.787 7fd727d75700 1 --2- 172.21.2.102:0/2834562167 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd7280d1c70 0x7fd7280cf950 secure :-1 s=READY pgs=1481 cs=0 l=1 rev1=1 rx=0x7fd720006f00 tx=0x7fd720009f10).ready entity=mon.0 client_cookie=d03e65629fee1c11 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:28.803 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:28.787 7fd726572700 1 -- 172.21.2.102:0/2834562167 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd72000b3b0 con 0x7fd7280d1c70 2021-02-10T11:25:28.804 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:28.787 7fd726572700 1 -- 172.21.2.102:0/2834562167 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd720000d30 con 0x7fd7280d1c70 2021-02-10T11:25:28.804 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:28.788 7fd726572700 1 -- 172.21.2.102:0/2834562167 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd72000b820 con 0x7fd7280d1c70 2021-02-10T11:25:28.804 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:28.788 7fd72d9a3700 1 -- 172.21.2.102:0/2834562167 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd7280d1c70 msgr2=0x7fd7280cf950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:28.804 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:28.788 7fd72d9a3700 1 --2- 172.21.2.102:0/2834562167 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd7280d1c70 0x7fd7280cf950 secure :-1 s=READY pgs=1481 cs=0 l=1 rev1=1 rx=0x7fd720006f00 tx=0x7fd720009f10).stop 2021-02-10T11:25:28.804 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:28.788 7fd72d9a3700 1 -- 172.21.2.102:0/2834562167 shutdown_connections 2021-02-10T11:25:28.804 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:28.788 7fd72d9a3700 1 --2- 172.21.2.102:0/2834562167 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd7281b0a10 0x7fd7280cf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:28.804 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:28.788 7fd72d9a3700 1 --2- 172.21.2.102:0/2834562167 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd7280d1c70 0x7fd7280cf950 unknown :-1 s=CLOSED pgs=1481 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:28.804 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:28.788 7fd72d9a3700 1 --2- 172.21.2.102:0/2834562167 >> v2:172.21.2.113:3300/0 conn(0x7fd7280d1870 0x7fd7281b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:28.804 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:28.788 7fd72d9a3700 1 -- 172.21.2.102:0/2834562167 >> 172.21.2.102:0/2834562167 conn(0x7fd7280ce450 msgr2=0x7fd7280d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:25:28.804 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:28.788 7fd72d9a3700 1 -- 172.21.2.102:0/2834562167 shutdown_connections 2021-02-10T11:25:28.804 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:28.788 7fd72d9a3700 1 -- 172.21.2.102:0/2834562167 wait complete. 2021-02-10T11:25:28.804 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:28.803 7fd72d9a3700 1 Processor -- start 2021-02-10T11:25:28.806 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:28.803 7fd72d9a3700 1 -- start start 2021-02-10T11:25:28.806 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:28.803 7fd72d9a3700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd7281a4a60 0x7fd7280c1c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:28.806 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:28.803 7fd72d9a3700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd7280c2290 0x7fd7280c2690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:28.806 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:28.803 7fd72d9a3700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd7280c2cb0 0x7fd7280bfd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:28.806 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:28.803 7fd72d9a3700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7280f05b0 con 0x7fd7280c2cb0 2021-02-10T11:25:28.806 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:28.803 7fd72d9a3700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7280f07d0 con 0x7fd7280c2290 2021-02-10T11:25:28.807 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:28.803 7fd72d9a3700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7280c0340 con 0x7fd7281a4a60 2021-02-10T11:25:28.807 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:28.803 7fd727d75700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd7280c2cb0 0x7fd7280bfd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:28.807 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:28.804 7fd727d75700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd7280c2cb0 0x7fd7280bfd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:57922/0 (socket says 172.21.2.102:57922) 2021-02-10T11:25:28.807 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:28.804 7fd727d75700 1 -- 172.21.2.102:0/308822653 learned_addr learned my addr 172.21.2.102:0/308822653 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:25:28.807 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:28.804 7fd727d75700 1 -- 172.21.2.102:0/308822653 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd7281a4a60 msgr2=0x7fd7280c1c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:28.807 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:28.804 7fd727d75700 1 --2- 172.21.2.102:0/308822653 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd7281a4a60 0x7fd7280c1c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:28.807 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:28.804 7fd727d75700 1 -- 172.21.2.102:0/308822653 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd7280c2290 msgr2=0x7fd7280c2690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:28.807 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:28.804 7fd727d75700 1 --2- 172.21.2.102:0/308822653 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd7280c2290 0x7fd7280c2690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:28.807 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:28.804 7fd727d75700 1 -- 172.21.2.102:0/308822653 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7280c0560 con 0x7fd7280c2cb0 2021-02-10T11:25:28.807 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:28.804 7fd727d75700 1 --2- 172.21.2.102:0/308822653 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd7280c2cb0 0x7fd7280bfd20 secure :-1 s=READY pgs=1482 cs=0 l=1 rev1=1 rx=0x7fd72000a8d0 tx=0x7fd72000ac70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:28.807 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:28.804 7fd71ffff700 1 -- 172.21.2.102:0/308822653 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd720000d30 con 0x7fd7280c2cb0 2021-02-10T11:25:28.808 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:28.804 7fd71ffff700 1 -- 172.21.2.102:0/308822653 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd720005820 con 0x7fd7280c2cb0 2021-02-10T11:25:28.808 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:28.804 7fd72d9a3700 1 -- 172.21.2.102:0/308822653 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7280c0560 con 0x7fd7280c2cb0 2021-02-10T11:25:28.808 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:28.804 7fd71ffff700 1 -- 172.21.2.102:0/308822653 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd720005cb0 con 0x7fd7280c2cb0 2021-02-10T11:25:28.808 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:28.804 7fd72d9a3700 1 -- 172.21.2.102:0/308822653 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7280c0560 con 0x7fd7280c2cb0 2021-02-10T11:25:28.808 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:28.805 7fd71ffff700 1 -- 172.21.2.102:0/308822653 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd720005c10 con 0x7fd7280c2cb0 2021-02-10T11:25:28.808 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:28.805 7fd71ffff700 1 --2- 172.21.2.102:0/308822653 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd70c0186c0 0x7fd70c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:28.808 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:28.805 7fd71ffff700 1 -- 172.21.2.102:0/308822653 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 4753+0+0 (secure 0 0 0) 0x7fd720032640 con 0x7fd7280c2cb0 2021-02-10T11:25:28.808 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:28.805 7fd727574700 1 --2- 172.21.2.102:0/308822653 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd70c0186c0 0x7fd70c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:28.808 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:28.805 7fd727574700 1 --2- 172.21.2.102:0/308822653 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd70c0186c0 0x7fd70c01abc0 crc :-1 s=READY pgs=675 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:28.840 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:28.836 7fd72d9a3700 1 -- 172.21.2.102:0/308822653 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7280f0b10 con 0x7fd7280c2cb0 2021-02-10T11:25:28.840 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:28.839 7fd71ffff700 1 -- 172.21.2.102:0/308822653 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd720005820 con 0x7fd7280c2cb0 2021-02-10T11:25:28.962 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:28.963 7fd72d9a3700 1 -- 172.21.2.102:0/308822653 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fd7280f0b10 con 0x7fd7280c2cb0 2021-02-10T11:25:28.962 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:28.963 7fd71ffff700 1 -- 172.21.2.102:0/308822653 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v61) v1 ==== 84+0+64 (secure 0 0 0) 0x7fd72001f830 con 0x7fd7280c2cb0 2021-02-10T11:25:28.963 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:25:28.963 INFO:teuthology.orchestra.run.gibba002.stdout:[ 2021-02-10T11:25:28.963 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:25:28.963 INFO:teuthology.orchestra.run.gibba002.stdout: "poolnum": 1, 2021-02-10T11:25:28.963 INFO:teuthology.orchestra.run.gibba002.stdout: "poolname": "rbd" 2021-02-10T11:25:28.964 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:25:28.964 INFO:teuthology.orchestra.run.gibba002.stdout:] 2021-02-10T11:25:28.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:28.965 7fd72d9a3700 1 -- 172.21.2.102:0/308822653 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd70c0186c0 msgr2=0x7fd70c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:28.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:28.965 7fd72d9a3700 1 --2- 172.21.2.102:0/308822653 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd70c0186c0 0x7fd70c01abc0 crc :-1 s=READY pgs=675 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:28.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:28.965 7fd72d9a3700 1 -- 172.21.2.102:0/308822653 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd7280c2cb0 msgr2=0x7fd7280bfd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:28.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:28.965 7fd72d9a3700 1 --2- 172.21.2.102:0/308822653 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd7280c2cb0 0x7fd7280bfd20 secure :-1 s=READY pgs=1482 cs=0 l=1 rev1=1 rx=0x7fd72000a8d0 tx=0x7fd72000ac70).stop 2021-02-10T11:25:28.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:28.965 7fd72d9a3700 1 -- 172.21.2.102:0/308822653 shutdown_connections 2021-02-10T11:25:28.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:28.965 7fd72d9a3700 1 --2- 172.21.2.102:0/308822653 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd7280c2cb0 0x7fd7280bfd20 unknown :-1 s=CLOSED pgs=1482 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:28.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:28.965 7fd72d9a3700 1 --2- 172.21.2.102:0/308822653 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd70c0186c0 0x7fd70c01abc0 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:28.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:28.965 7fd72d9a3700 1 --2- 172.21.2.102:0/308822653 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd7280c2290 0x7fd7280c2690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:28.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:28.965 7fd72d9a3700 1 --2- 172.21.2.102:0/308822653 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd7281a4a60 0x7fd7280c1c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:28.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:28.965 7fd72d9a3700 1 -- 172.21.2.102:0/308822653 >> 172.21.2.102:0/308822653 conn(0x7fd7280ebe30 msgr2=0x7fd7280bf200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:25:28.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:28.965 7fd72d9a3700 1 -- 172.21.2.102:0/308822653 shutdown_connections 2021-02-10T11:25:28.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:28.965 7fd72d9a3700 1 -- 172.21.2.102:0/308822653 wait complete. 2021-02-10T11:25:28.973 INFO:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2021-02-10T11:25:28.973 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool create cephfs_metadata 8 2021-02-10T11:25:29.403 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:29.387 7fe5f54e2700 1 Processor -- start 2021-02-10T11:25:29.403 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:29.387 7fe5f54e2700 1 -- start start 2021-02-10T11:25:29.403 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:29.387 7fe5f54e2700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fe5f00d1870 0x7fe5f01b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:29.403 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:29.387 7fe5f54e2700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe5f00cff70 0x7fe5f0109130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:29.403 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:29.387 7fe5f54e2700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5f0109a60 con 0x7fe5f01b0ae0 2021-02-10T11:25:29.404 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:29.387 7fe5f54e2700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5f0109750 con 0x7fe5f00d1870 2021-02-10T11:25:29.404 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:29.387 7fe5f54e2700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5f0088070 con 0x7fe5f00cff70 2021-02-10T11:25:29.404 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:29.387 7fe5ef2bf700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fe5f00d1870 0x7fe5f01b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:29.404 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:29.387 7fe5ef2bf700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fe5f00d1870 0x7fe5f01b04c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:57926/0 (socket says 172.21.2.102:57926) 2021-02-10T11:25:29.404 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:29.387 7fe5ef2bf700 1 -- 172.21.2.102:0/3057828808 learned_addr learned my addr 172.21.2.102:0/3057828808 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:25:29.404 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:29.388 7fe5efac0700 1 --2- 172.21.2.102:0/3057828808 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe5f00cff70 0x7fe5f0109130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:29.404 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:29.388 7fe5ef2bf700 1 -- 172.21.2.102:0/3057828808 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe5f00cff70 msgr2=0x7fe5f0109130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:25:29.404 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:29.388 7fe5ef2bf700 1 --2- 172.21.2.102:0/3057828808 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe5f00cff70 0x7fe5f0109130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:29.404 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:29.388 7fe5ef2bf700 1 -- 172.21.2.102:0/3057828808 >> v1:172.21.2.113:6789/0 conn(0x7fe5f01b0ae0 legacy=0x7fe5f0108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:29.404 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:29.388 7fe5ef2bf700 1 -- 172.21.2.102:0/3057828808 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5f00d1c70 con 0x7fe5f00d1870 2021-02-10T11:25:29.404 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:29.388 7fe5ef2bf700 1 --2- 172.21.2.102:0/3057828808 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe5f00d1870 0x7fe5f01b04c0 secure :-1 s=READY pgs=1483 cs=0 l=1 rev1=1 rx=0x7fe5e0006700 tx=0x7fe5e0009b80).ready entity=mon.0 client_cookie=55ad1665d68009f7 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:29.404 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:29.388 7fe5ee2bd700 1 -- 172.21.2.102:0/3057828808 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe5e000c0c0 con 0x7fe5f00d1870 2021-02-10T11:25:29.405 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:29.388 7fe5ee2bd700 1 -- 172.21.2.102:0/3057828808 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5e000c400 con 0x7fe5f00d1870 2021-02-10T11:25:29.405 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:29.388 7fe5ee2bd700 1 -- 172.21.2.102:0/3057828808 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe5e000c920 con 0x7fe5f00d1870 2021-02-10T11:25:29.405 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:29.388 7fe5f54e2700 1 -- 172.21.2.102:0/3057828808 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe5f00d1870 msgr2=0x7fe5f01b04c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:29.405 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:29.388 7fe5f54e2700 1 --2- 172.21.2.102:0/3057828808 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe5f00d1870 0x7fe5f01b04c0 secure :-1 s=READY pgs=1483 cs=0 l=1 rev1=1 rx=0x7fe5e0006700 tx=0x7fe5e0009b80).stop 2021-02-10T11:25:29.405 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:29.389 7fe5f54e2700 1 -- 172.21.2.102:0/3057828808 shutdown_connections 2021-02-10T11:25:29.405 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:29.389 7fe5f54e2700 1 --2- 172.21.2.102:0/3057828808 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe5f00cff70 0x7fe5f0109130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:29.405 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:29.389 7fe5f54e2700 1 --2- 172.21.2.102:0/3057828808 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe5f00d1870 0x7fe5f01b04c0 unknown :-1 s=CLOSED pgs=1483 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:29.405 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:29.389 7fe5f54e2700 1 -- 172.21.2.102:0/3057828808 >> 172.21.2.102:0/3057828808 conn(0x7fe5f00ce420 msgr2=0x7fe5f00d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:25:29.405 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:29.389 7fe5f54e2700 1 -- 172.21.2.102:0/3057828808 shutdown_connections 2021-02-10T11:25:29.405 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:29.389 7fe5f54e2700 1 -- 172.21.2.102:0/3057828808 wait complete. 2021-02-10T11:25:29.405 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:29.404 7fe5f54e2700 1 Processor -- start 2021-02-10T11:25:29.405 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:29.404 7fe5f54e2700 1 -- start start 2021-02-10T11:25:29.407 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:29.404 7fe5f54e2700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe5f009bcc0 0x7fe5f009c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:29.407 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:29.404 7fe5f54e2700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe5f00a0820 0x7fe5f00a0c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:29.407 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:29.404 7fe5f54e2700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe5f00a1240 0x7fe5f00858d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:29.407 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:29.404 7fe5f54e2700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5f00a66b0 con 0x7fe5f00a1240 2021-02-10T11:25:29.407 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:29.404 7fe5f54e2700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5f01a49b0 con 0x7fe5f009bcc0 2021-02-10T11:25:29.407 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:29.404 7fe5f54e2700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5f01a4b90 con 0x7fe5f00a0820 2021-02-10T11:25:29.407 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:29.405 7fe5efac0700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe5f00a1240 0x7fe5f00858d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:29.407 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:29.405 7fe5efac0700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe5f00a1240 0x7fe5f00858d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:57934/0 (socket says 172.21.2.102:57934) 2021-02-10T11:25:29.407 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:29.405 7fe5efac0700 1 -- 172.21.2.102:0/279261301 learned_addr learned my addr 172.21.2.102:0/279261301 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:25:29.408 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:29.405 7fe5ef2bf700 1 --2- 172.21.2.102:0/279261301 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe5f009bcc0 0x7fe5f009c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:29.408 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:29.405 7fe5eeabe700 1 --2- 172.21.2.102:0/279261301 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe5f00a0820 0x7fe5f00a0c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:29.408 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:29.405 7fe5efac0700 1 -- 172.21.2.102:0/279261301 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe5f00a0820 msgr2=0x7fe5f00a0c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:29.408 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:29.405 7fe5efac0700 1 --2- 172.21.2.102:0/279261301 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe5f00a0820 0x7fe5f00a0c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:29.408 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:29.405 7fe5efac0700 1 -- 172.21.2.102:0/279261301 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe5f009bcc0 msgr2=0x7fe5f009c0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:29.408 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:29.405 7fe5efac0700 1 --2- 172.21.2.102:0/279261301 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe5f009bcc0 0x7fe5f009c0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:29.408 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:29.405 7fe5efac0700 1 -- 172.21.2.102:0/279261301 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5f00a1640 con 0x7fe5f00a1240 2021-02-10T11:25:29.408 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:29.405 7fe5ef2bf700 1 --2- 172.21.2.102:0/279261301 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe5f009bcc0 0x7fe5f009c0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:25:29.408 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:29.405 7fe5efac0700 1 --2- 172.21.2.102:0/279261301 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe5f00a1240 0x7fe5f00858d0 secure :-1 s=READY pgs=1484 cs=0 l=1 rev1=1 rx=0x7fe5e80099a0 tx=0x7fe5e800bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:29.408 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:29.406 7fe5d7fff700 1 -- 172.21.2.102:0/279261301 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe5e80066f0 con 0x7fe5f00a1240 2021-02-10T11:25:29.408 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:29.406 7fe5d7fff700 1 -- 172.21.2.102:0/279261301 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5e8003030 con 0x7fe5f00a1240 2021-02-10T11:25:29.408 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:29.406 7fe5d7fff700 1 -- 172.21.2.102:0/279261301 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe5e8001470 con 0x7fe5f00a1240 2021-02-10T11:25:29.409 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:29.406 7fe5f54e2700 1 -- 172.21.2.102:0/279261301 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5f00a1640 con 0x7fe5f00a1240 2021-02-10T11:25:29.409 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:29.406 7fe5f54e2700 1 -- 172.21.2.102:0/279261301 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5f00861e0 con 0x7fe5f00a1240 2021-02-10T11:25:29.409 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:29.406 7fe5d7fff700 1 -- 172.21.2.102:0/279261301 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fe5e80011f0 con 0x7fe5f00a1240 2021-02-10T11:25:29.409 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:29.406 7fe5d7fff700 1 --2- 172.21.2.102:0/279261301 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe5d00186c0 0x7fe5d001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:29.409 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:29.406 7fe5d7fff700 1 -- 172.21.2.102:0/279261301 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 4753+0+0 (secure 0 0 0) 0x7fe5e80325c0 con 0x7fe5f00a1240 2021-02-10T11:25:29.409 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:29.406 7fe5ef2bf700 1 --2- 172.21.2.102:0/279261301 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe5d00186c0 0x7fe5d001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:29.409 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:29.407 7fe5ef2bf700 1 --2- 172.21.2.102:0/279261301 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe5d00186c0 0x7fe5d001abc0 crc :-1 s=READY pgs=676 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:29.436 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:29.437 7fe5f54e2700 1 -- 172.21.2.102:0/279261301 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5f00fe1c0 con 0x7fe5f00a1240 2021-02-10T11:25:29.439 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:29.440 7fe5d7fff700 1 -- 172.21.2.102:0/279261301 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fe5e8000f90 con 0x7fe5f00a1240 2021-02-10T11:25:29.567 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:29.568 7fe5f54e2700 1 -- 172.21.2.102:0/279261301 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"} v 0) v1 -- 0x7fe5f00fe1c0 con 0x7fe5f00a1240 2021-02-10T11:25:29.886 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:29.887 7fe5d7fff700 1 -- 172.21.2.102:0/279261301 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"}]=0 pool 'cephfs_metadata' created v62) v1 ==== 133+0+0 (secure 0 0 0) 0x7fe5e801f880 con 0x7fe5f00a1240 2021-02-10T11:25:29.933 INFO:teuthology.orchestra.run.gibba002.stderr:pool 'cephfs_metadata' created 2021-02-10T11:25:29.935 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:29.935 7fe5f54e2700 1 -- 172.21.2.102:0/279261301 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe5d00186c0 msgr2=0x7fe5d001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:29.935 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:29.935 7fe5f54e2700 1 --2- 172.21.2.102:0/279261301 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe5d00186c0 0x7fe5d001abc0 crc :-1 s=READY pgs=676 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:29.935 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:29.935 7fe5f54e2700 1 -- 172.21.2.102:0/279261301 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe5f00a1240 msgr2=0x7fe5f00858d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:29.935 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:29.935 7fe5f54e2700 1 --2- 172.21.2.102:0/279261301 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe5f00a1240 0x7fe5f00858d0 secure :-1 s=READY pgs=1484 cs=0 l=1 rev1=1 rx=0x7fe5e80099a0 tx=0x7fe5e800bcf0).stop 2021-02-10T11:25:29.935 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:29.935 7fe5f54e2700 1 -- 172.21.2.102:0/279261301 shutdown_connections 2021-02-10T11:25:29.935 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:29.935 7fe5f54e2700 1 --2- 172.21.2.102:0/279261301 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe5f00a1240 0x7fe5f00858d0 unknown :-1 s=CLOSED pgs=1484 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:29.935 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:29.935 7fe5f54e2700 1 --2- 172.21.2.102:0/279261301 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe5f00a0820 0x7fe5f00a0c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:29.935 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:29.935 7fe5f54e2700 1 --2- 172.21.2.102:0/279261301 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe5d00186c0 0x7fe5d001abc0 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:29.936 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:29.935 7fe5f54e2700 1 --2- 172.21.2.102:0/279261301 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe5f009bcc0 0x7fe5f009c0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:29.936 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:29.935 7fe5f54e2700 1 -- 172.21.2.102:0/279261301 >> 172.21.2.102:0/279261301 conn(0x7fe5f00bef00 msgr2=0x7fe5f009d1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:25:29.936 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:29.936 7fe5f54e2700 1 -- 172.21.2.102:0/279261301 shutdown_connections 2021-02-10T11:25:29.936 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:29.936 7fe5f54e2700 1 -- 172.21.2.102:0/279261301 wait complete. 2021-02-10T11:25:29.943 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool create cephfs_data 8 2021-02-10T11:25:30.375 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:30.358 7f90fbda6700 1 Processor -- start 2021-02-10T11:25:30.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:30.358 7f90fbda6700 1 -- start start 2021-02-10T11:25:30.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:30.359 7f90fbda6700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f90f40d1870 0x7f90f41b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:30.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:30.359 7f90fbda6700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f90f41b0ae0 0x7f90f4108940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:30.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:30.359 7f90fbda6700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90f40fe1c0 con 0x7f90f40d1c70 2021-02-10T11:25:30.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:30.359 7f90fbda6700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90f4109a60 con 0x7f90f40d1870 2021-02-10T11:25:30.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:30.359 7f90fbda6700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90f4109750 con 0x7f90f41b0ae0 2021-02-10T11:25:30.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:30.360 7f90fa3c3700 1 --1- >> v1:172.21.2.113:6789/0 conn(0x7f90f40d1c70 0x7f90f4108f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.113:6789/0 says I am v1:172.21.2.102:57486/0 (socket says 172.21.2.102:57486) 2021-02-10T11:25:30.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:30.360 7f90f9bc2700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f90f40d1870 0x7f90f41b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:30.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:30.360 7f90f93c1700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f90f41b0ae0 0x7f90f4108940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:30.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:30.360 7f90fa3c3700 1 -- 172.21.2.102:0/3216915704 learned_addr learned my addr 172.21.2.102:0/3216915704 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:25:30.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:30.361 7f90f93c1700 1 -- 172.21.2.102:0/3216915704 >> v2:172.21.2.113:3300/0 conn(0x7f90f40d1870 msgr2=0x7f90f41b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:25:30.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:30.361 7f90f93c1700 1 --2- 172.21.2.102:0/3216915704 >> v2:172.21.2.113:3300/0 conn(0x7f90f40d1870 0x7f90f41b04c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:30.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:30.361 7f90f93c1700 1 -- 172.21.2.102:0/3216915704 >> v1:172.21.2.113:6789/0 conn(0x7f90f40d1c70 legacy=0x7f90f4108f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:30.377 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:30.361 7f90f93c1700 1 -- 172.21.2.102:0/3216915704 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90f4088070 con 0x7f90f41b0ae0 2021-02-10T11:25:30.377 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:30.361 7f90f9bc2700 1 --2- 172.21.2.102:0/3216915704 >> v2:172.21.2.113:3300/0 conn(0x7f90f40d1870 0x7f90f41b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:25:30.377 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:30.361 7f90f93c1700 1 --2- 172.21.2.102:0/3216915704 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f90f41b0ae0 0x7f90f4108940 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7f90f0006650 tx=0x7f90f0009a80).ready entity=mon.2 client_cookie=e00223ccd8f3b686 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:30.377 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:30.361 7f90f8bc0700 1 -- 172.21.2.102:0/3216915704 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f90f000c0c0 con 0x7f90f41b0ae0 2021-02-10T11:25:30.377 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:30.361 7f90f8bc0700 1 -- 172.21.2.102:0/3216915704 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90f000c340 con 0x7f90f41b0ae0 2021-02-10T11:25:30.377 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:30.361 7f90f8bc0700 1 -- 172.21.2.102:0/3216915704 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f90f000c790 con 0x7f90f41b0ae0 2021-02-10T11:25:30.377 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:30.361 7f90fbda6700 1 -- 172.21.2.102:0/3216915704 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f90f41b0ae0 msgr2=0x7f90f4108940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:30.377 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:30.361 7f90fbda6700 1 --2- 172.21.2.102:0/3216915704 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f90f41b0ae0 0x7f90f4108940 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7f90f0006650 tx=0x7f90f0009a80).stop 2021-02-10T11:25:30.377 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:30.362 7f90fbda6700 1 -- 172.21.2.102:0/3216915704 shutdown_connections 2021-02-10T11:25:30.377 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:30.362 7f90fbda6700 1 --2- 172.21.2.102:0/3216915704 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f90f41b0ae0 0x7f90f4108940 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:30.377 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:30.362 7f90fbda6700 1 --2- 172.21.2.102:0/3216915704 >> v2:172.21.2.113:3300/0 conn(0x7f90f40d1870 0x7f90f41b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:30.377 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:30.362 7f90fbda6700 1 -- 172.21.2.102:0/3216915704 >> 172.21.2.102:0/3216915704 conn(0x7f90f40ce420 msgr2=0x7f90f40d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:25:30.377 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:30.362 7f90fbda6700 1 -- 172.21.2.102:0/3216915704 shutdown_connections 2021-02-10T11:25:30.378 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:30.362 7f90fbda6700 1 -- 172.21.2.102:0/3216915704 wait complete. 2021-02-10T11:25:30.379 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:30.377 7f90fbda6700 1 Processor -- start 2021-02-10T11:25:30.379 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:30.377 7f90fbda6700 1 -- start start 2021-02-10T11:25:30.379 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:30.377 7f90fbda6700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f90f40a7950 0x7f90f40b8df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:30.379 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:30.377 7f90fbda6700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f90f41a7490 0x7f90f41a7890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:30.380 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:30.377 7f90fbda6700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f90f41a7eb0 0x7f90f40ac630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:30.380 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:30.377 7f90fbda6700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90f40a7fb0 con 0x7f90f40a7950 2021-02-10T11:25:30.380 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:30.377 7f90fbda6700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90f40acc50 con 0x7f90f41a7490 2021-02-10T11:25:30.380 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:30.377 7f90fbda6700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90f40ace30 con 0x7f90f41a7eb0 2021-02-10T11:25:30.380 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:30.377 7f90f9bc2700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f90f40a7950 0x7f90f40b8df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:30.380 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:30.377 7f90f9bc2700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f90f40a7950 0x7f90f40b8df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:57942/0 (socket says 172.21.2.102:57942) 2021-02-10T11:25:30.380 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:30.377 7f90f9bc2700 1 -- 172.21.2.102:0/3312212233 learned_addr learned my addr 172.21.2.102:0/3312212233 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:25:30.380 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:30.378 7f90f93c1700 1 --2- 172.21.2.102:0/3312212233 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f90f41a7490 0x7f90f41a7890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:30.380 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:30.378 7f90f9bc2700 1 -- 172.21.2.102:0/3312212233 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f90f41a7eb0 msgr2=0x7f90f40ac630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:30.380 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:30.378 7f90f9bc2700 1 --2- 172.21.2.102:0/3312212233 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f90f41a7eb0 0x7f90f40ac630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:30.380 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:30.378 7f90f9bc2700 1 -- 172.21.2.102:0/3312212233 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f90f41a7490 msgr2=0x7f90f41a7890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:30.380 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:30.378 7f90f9bc2700 1 --2- 172.21.2.102:0/3312212233 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f90f41a7490 0x7f90f41a7890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:30.381 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:30.378 7f90f9bc2700 1 -- 172.21.2.102:0/3312212233 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90f40ad050 con 0x7f90f40a7950 2021-02-10T11:25:30.381 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:30.378 7f90f9bc2700 1 --2- 172.21.2.102:0/3312212233 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f90f40a7950 0x7f90f40b8df0 secure :-1 s=READY pgs=1485 cs=0 l=1 rev1=1 rx=0x7f90e8006f40 tx=0x7f90e800b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:30.381 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:30.378 7f90e67fc700 1 -- 172.21.2.102:0/3312212233 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f90e800bd80 con 0x7f90f40a7950 2021-02-10T11:25:30.381 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:30.378 7f90e67fc700 1 -- 172.21.2.102:0/3312212233 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90e8000c90 con 0x7f90f40a7950 2021-02-10T11:25:30.381 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:30.378 7f90fbda6700 1 -- 172.21.2.102:0/3312212233 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90f40ad050 con 0x7f90f40a7950 2021-02-10T11:25:30.381 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:30.378 7f90e67fc700 1 -- 172.21.2.102:0/3312212233 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f90e8002220 con 0x7f90f40a7950 2021-02-10T11:25:30.381 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:30.378 7f90fbda6700 1 -- 172.21.2.102:0/3312212233 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90f40ad050 con 0x7f90f40a7950 2021-02-10T11:25:30.381 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:30.379 7f90e67fc700 1 -- 172.21.2.102:0/3312212233 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f90e800a000 con 0x7f90f40a7950 2021-02-10T11:25:30.381 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:30.379 7f90e67fc700 1 --2- 172.21.2.102:0/3312212233 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f90dc018730 0x7f90dc01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:30.381 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:30.379 7f90e67fc700 1 -- 172.21.2.102:0/3312212233 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5107+0+0 (secure 0 0 0) 0x7f90e8032460 con 0x7f90f40a7950 2021-02-10T11:25:30.381 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:30.379 7f90f93c1700 1 --2- 172.21.2.102:0/3312212233 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f90dc018730 0x7f90dc01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:30.381 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:30.379 7f90f93c1700 1 --2- 172.21.2.102:0/3312212233 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f90dc018730 0x7f90dc01ac10 crc :-1 s=READY pgs=677 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:30.408 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:30.409 7f90fbda6700 1 -- 172.21.2.102:0/3312212233 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90f4082a70 con 0x7f90f40a7950 2021-02-10T11:25:30.410 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:30.411 7f90e67fc700 1 -- 172.21.2.102:0/3312212233 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f90e8000c90 con 0x7f90f40a7950 2021-02-10T11:25:30.539 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:30.540 7f90fbda6700 1 -- 172.21.2.102:0/3312212233 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_data"} v 0) v1 -- 0x7f90f4082a70 con 0x7f90f40a7950 2021-02-10T11:25:30.893 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:30.894 7f90e67fc700 1 -- 172.21.2.102:0/3312212233 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_data"}]=0 pool 'cephfs_data' created v64) v1 ==== 125+0+0 (secure 0 0 0) 0x7f90e800a000 con 0x7f90f40a7950 2021-02-10T11:25:30.905 INFO:teuthology.orchestra.run.gibba002.stderr:pool 'cephfs_data' created 2021-02-10T11:25:30.907 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:30.907 7f90fbda6700 1 -- 172.21.2.102:0/3312212233 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f90dc018730 msgr2=0x7f90dc01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:30.907 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:30.907 7f90fbda6700 1 --2- 172.21.2.102:0/3312212233 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f90dc018730 0x7f90dc01ac10 crc :-1 s=READY pgs=677 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:30.907 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:30.907 7f90fbda6700 1 -- 172.21.2.102:0/3312212233 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f90f40a7950 msgr2=0x7f90f40b8df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:30.907 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:30.907 7f90fbda6700 1 --2- 172.21.2.102:0/3312212233 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f90f40a7950 0x7f90f40b8df0 secure :-1 s=READY pgs=1485 cs=0 l=1 rev1=1 rx=0x7f90e8006f40 tx=0x7f90e800b990).stop 2021-02-10T11:25:30.907 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:30.908 7f90fbda6700 1 -- 172.21.2.102:0/3312212233 shutdown_connections 2021-02-10T11:25:30.908 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:30.908 7f90fbda6700 1 --2- 172.21.2.102:0/3312212233 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f90f41a7eb0 0x7f90f40ac630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:30.908 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:30.908 7f90fbda6700 1 --2- 172.21.2.102:0/3312212233 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f90dc018730 0x7f90dc01ac10 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:30.908 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:30.908 7f90fbda6700 1 --2- 172.21.2.102:0/3312212233 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f90f41a7490 0x7f90f41a7890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:30.908 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:30.908 7f90fbda6700 1 --2- 172.21.2.102:0/3312212233 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f90f40a7950 0x7f90f40b8df0 unknown :-1 s=CLOSED pgs=1485 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:30.908 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:30.908 7f90fbda6700 1 -- 172.21.2.102:0/3312212233 >> 172.21.2.102:0/3312212233 conn(0x7f90f40a02d0 msgr2=0x7f90f409f200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:25:30.908 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:30.908 7f90fbda6700 1 -- 172.21.2.102:0/3312212233 shutdown_connections 2021-02-10T11:25:30.908 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:30.908 7f90fbda6700 1 -- 172.21.2.102:0/3312212233 wait complete. 2021-02-10T11:25:30.915 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2021-02-10T11:25:31.347 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:31.331 7fc50b49c700 1 Processor -- start 2021-02-10T11:25:31.347 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:31.331 7fc50b49c700 1 -- start start 2021-02-10T11:25:31.347 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:31.332 7fc50b49c700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fc5040d1880 0x7fc5041b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:31.347 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:31.332 7fc50b49c700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc5041b0af0 0x7fc504108950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:31.348 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:31.332 7fc50b49c700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5040fe1d0 con 0x7fc5040d1c80 2021-02-10T11:25:31.348 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:31.332 7fc50b49c700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc504109a70 con 0x7fc5040d1880 2021-02-10T11:25:31.348 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:31.332 7fc50b49c700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc504109760 con 0x7fc5041b0af0 2021-02-10T11:25:31.348 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:31.332 7fc5092b8700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fc5040d1880 0x7fc5041b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:31.348 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:31.332 7fc5092b8700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fc5040d1880 0x7fc5041b04d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:57952/0 (socket says 172.21.2.102:57952) 2021-02-10T11:25:31.348 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:31.332 7fc5092b8700 1 -- 172.21.2.102:0/1682233530 learned_addr learned my addr 172.21.2.102:0/1682233530 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:25:31.348 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:31.332 7fc508ab7700 1 --2- 172.21.2.102:0/1682233530 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc5041b0af0 0x7fc504108950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:31.348 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:31.332 7fc5092b8700 1 -- 172.21.2.102:0/1682233530 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc5041b0af0 msgr2=0x7fc504108950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:25:31.348 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:31.332 7fc5092b8700 1 --2- 172.21.2.102:0/1682233530 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc5041b0af0 0x7fc504108950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:31.348 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:31.332 7fc5092b8700 1 -- 172.21.2.102:0/1682233530 >> v1:172.21.2.113:6789/0 conn(0x7fc5040d1c80 legacy=0x7fc504108f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:25:31.348 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:31.332 7fc5092b8700 1 -- 172.21.2.102:0/1682233530 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc504088070 con 0x7fc5040d1880 2021-02-10T11:25:31.348 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:31.332 7fc5092b8700 1 --2- 172.21.2.102:0/1682233530 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc5040d1880 0x7fc5041b04d0 secure :-1 s=READY pgs=1486 cs=0 l=1 rev1=1 rx=0x7fc500006700 tx=0x7fc500009b00).ready entity=mon.0 client_cookie=3894c7b4fd3416a8 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:31.349 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:31.333 7fc4f7fff700 1 -- 172.21.2.102:0/1682233530 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc50000c0c0 con 0x7fc5040d1880 2021-02-10T11:25:31.349 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:31.333 7fc4f7fff700 1 -- 172.21.2.102:0/1682233530 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc50000c400 con 0x7fc5040d1880 2021-02-10T11:25:31.349 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:31.333 7fc4f7fff700 1 -- 172.21.2.102:0/1682233530 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc50000c920 con 0x7fc5040d1880 2021-02-10T11:25:31.349 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:31.333 7fc50b49c700 1 -- 172.21.2.102:0/1682233530 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc5040d1880 msgr2=0x7fc5041b04d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:31.349 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:31.333 7fc50b49c700 1 --2- 172.21.2.102:0/1682233530 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc5040d1880 0x7fc5041b04d0 secure :-1 s=READY pgs=1486 cs=0 l=1 rev1=1 rx=0x7fc500006700 tx=0x7fc500009b00).stop 2021-02-10T11:25:31.349 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:31.333 7fc50b49c700 1 -- 172.21.2.102:0/1682233530 shutdown_connections 2021-02-10T11:25:31.349 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:31.333 7fc50b49c700 1 --2- 172.21.2.102:0/1682233530 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc5041b0af0 0x7fc504108950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:31.349 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:31.333 7fc50b49c700 1 --2- 172.21.2.102:0/1682233530 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc5040d1880 0x7fc5041b04d0 unknown :-1 s=CLOSED pgs=1486 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:31.349 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:31.333 7fc50b49c700 1 -- 172.21.2.102:0/1682233530 >> 172.21.2.102:0/1682233530 conn(0x7fc5040ce430 msgr2=0x7fc5040d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:25:31.349 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:31.334 7fc50b49c700 1 -- 172.21.2.102:0/1682233530 shutdown_connections 2021-02-10T11:25:31.349 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:31.334 7fc50b49c700 1 -- 172.21.2.102:0/1682233530 wait complete. 2021-02-10T11:25:31.351 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:31.348 7fc50b49c700 1 Processor -- start 2021-02-10T11:25:31.351 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:31.348 7fc50b49c700 1 -- start start 2021-02-10T11:25:31.351 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:31.349 7fc50b49c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc5041b58d0 0x7fc5040aa4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:31.351 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:31.349 7fc50b49c700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc5040aaaf0 0x7fc5041b80a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:31.351 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:31.349 7fc50b49c700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc5041b86c0 0x7fc50419de70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:31.351 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:31.349 7fc50b49c700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5041b8ac0 con 0x7fc5041b58d0 2021-02-10T11:25:31.351 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:31.349 7fc50b49c700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5041b8e20 con 0x7fc5041b86c0 2021-02-10T11:25:31.351 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:31.349 7fc50b49c700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5041b5cd0 con 0x7fc5040aaaf0 2021-02-10T11:25:31.352 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:31.349 7fc5092b8700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc5041b58d0 0x7fc5040aa4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:31.352 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:31.349 7fc5092b8700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc5041b58d0 0x7fc5040aa4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:57954/0 (socket says 172.21.2.102:57954) 2021-02-10T11:25:31.352 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:31.349 7fc5092b8700 1 -- 172.21.2.102:0/2429404341 learned_addr learned my addr 172.21.2.102:0/2429404341 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:25:31.352 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:31.349 7fc5092b8700 1 -- 172.21.2.102:0/2429404341 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc5040aaaf0 msgr2=0x7fc5041b80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:31.352 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:31.349 7fc5092b8700 1 --2- 172.21.2.102:0/2429404341 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc5040aaaf0 0x7fc5041b80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:31.352 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:31.349 7fc5092b8700 1 -- 172.21.2.102:0/2429404341 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc5041b86c0 msgr2=0x7fc50419de70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:31.352 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:31.349 7fc5092b8700 1 --2- 172.21.2.102:0/2429404341 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc5041b86c0 0x7fc50419de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:31.352 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:31.349 7fc5092b8700 1 -- 172.21.2.102:0/2429404341 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc504095710 con 0x7fc5041b58d0 2021-02-10T11:25:31.352 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:31.350 7fc5092b8700 1 --2- 172.21.2.102:0/2429404341 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc5041b58d0 0x7fc5040aa4d0 secure :-1 s=READY pgs=1487 cs=0 l=1 rev1=1 rx=0x7fc50000b660 tx=0x7fc50000ba00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:31.352 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:31.350 7fc4f5ffb700 1 -- 172.21.2.102:0/2429404341 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc500009440 con 0x7fc5041b58d0 2021-02-10T11:25:31.352 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:31.350 7fc4f5ffb700 1 -- 172.21.2.102:0/2429404341 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc500000c90 con 0x7fc5041b58d0 2021-02-10T11:25:31.353 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:31.350 7fc4f5ffb700 1 -- 172.21.2.102:0/2429404341 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc500002030 con 0x7fc5041b58d0 2021-02-10T11:25:31.353 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:31.350 7fc50b49c700 1 -- 172.21.2.102:0/2429404341 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc504095710 con 0x7fc5041b58d0 2021-02-10T11:25:31.353 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:31.350 7fc50b49c700 1 -- 172.21.2.102:0/2429404341 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc50419e780 con 0x7fc5041b58d0 2021-02-10T11:25:31.353 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:31.350 7fc4f5ffb700 1 -- 172.21.2.102:0/2429404341 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fc5000093e0 con 0x7fc5041b58d0 2021-02-10T11:25:31.353 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:31.350 7fc4f5ffb700 1 --2- 172.21.2.102:0/2429404341 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc4e40186c0 0x7fc4e401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:31.353 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:31.350 7fc4f5ffb700 1 -- 172.21.2.102:0/2429404341 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5457+0+0 (secure 0 0 0) 0x7fc500032570 con 0x7fc5041b58d0 2021-02-10T11:25:31.353 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:31.351 7fc508ab7700 1 --2- 172.21.2.102:0/2429404341 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc4e40186c0 0x7fc4e401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:31.353 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:31.351 7fc508ab7700 1 --2- 172.21.2.102:0/2429404341 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc4e40186c0 0x7fc4e401abc0 crc :-1 s=READY pgs=680 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:31.381 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:31.382 7fc50b49c700 1 -- 172.21.2.102:0/2429404341 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc504082aa0 con 0x7fc5041b58d0 2021-02-10T11:25:31.383 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:31.384 7fc4f5ffb700 1 -- 172.21.2.102:0/2429404341 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fc500030d90 con 0x7fc5041b58d0 2021-02-10T11:25:31.511 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:31.512 7fc50b49c700 1 -- 172.21.2.102:0/2429404341 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"data": "cephfs_data", "prefix": "fs new", "force": true, "fs_name": "cephfs", "metadata": "cephfs_metadata"} v 0) v1 -- 0x7fc504082aa0 con 0x7fc5041b58d0 2021-02-10T11:25:31.515 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:31.516 7fc4f5ffb700 1 -- 172.21.2.102:0/2429404341 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"data": "cephfs_data", "prefix": "fs new", "force": true, "fs_name": "cephfs", "metadata": "cephfs_metadata"}]=0 new fs with metadata pool 12 and data pool 13 v45) v1 ==== 189+0+0 (secure 0 0 0) 0x7fc5000093e0 con 0x7fc5041b58d0 2021-02-10T11:25:31.519 INFO:teuthology.orchestra.run.gibba002.stderr:new fs with metadata pool 12 and data pool 13 2021-02-10T11:25:31.521 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:31.521 7fc50b49c700 1 -- 172.21.2.102:0/2429404341 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc4e40186c0 msgr2=0x7fc4e401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:31.521 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:31.521 7fc50b49c700 1 --2- 172.21.2.102:0/2429404341 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc4e40186c0 0x7fc4e401abc0 crc :-1 s=READY pgs=680 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:31.521 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:31.521 7fc50b49c700 1 -- 172.21.2.102:0/2429404341 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc5041b58d0 msgr2=0x7fc5040aa4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:31.521 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:31.521 7fc50b49c700 1 --2- 172.21.2.102:0/2429404341 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc5041b58d0 0x7fc5040aa4d0 secure :-1 s=READY pgs=1487 cs=0 l=1 rev1=1 rx=0x7fc50000b660 tx=0x7fc50000ba00).stop 2021-02-10T11:25:31.521 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:31.521 7fc50b49c700 1 -- 172.21.2.102:0/2429404341 shutdown_connections 2021-02-10T11:25:31.521 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:31.521 7fc50b49c700 1 --2- 172.21.2.102:0/2429404341 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc4e40186c0 0x7fc4e401abc0 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:31.521 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:31.521 7fc50b49c700 1 --2- 172.21.2.102:0/2429404341 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc5041b86c0 0x7fc50419de70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:31.521 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:31.521 7fc50b49c700 1 --2- 172.21.2.102:0/2429404341 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc5040aaaf0 0x7fc5041b80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:31.521 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:31.521 7fc50b49c700 1 --2- 172.21.2.102:0/2429404341 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc5041b58d0 0x7fc5040aa4d0 unknown :-1 s=CLOSED pgs=1487 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:31.521 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:31.521 7fc50b49c700 1 -- 172.21.2.102:0/2429404341 >> 172.21.2.102:0/2429404341 conn(0x7fc50419f940 msgr2=0x7fc50418c290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:25:31.521 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:31.521 7fc50b49c700 1 -- 172.21.2.102:0/2429404341 shutdown_connections 2021-02-10T11:25:31.522 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:31.521 7fc50b49c700 1 -- 172.21.2.102:0/2429404341 wait complete. 2021-02-10T11:25:31.529 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:25:31.955 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:31.939 7f7ae8b26700 1 Processor -- start 2021-02-10T11:25:31.956 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:31.939 7f7ae8b26700 1 -- start start 2021-02-10T11:25:31.956 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:31.940 7f7ae8b26700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7ae40d1c70 0x7f7ae40f20a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:31.956 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:31.940 7f7ae8b26700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ae40f0b10 con 0x7f7ae40d1870 2021-02-10T11:25:31.956 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:31.940 7f7ae8b26700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ae41b0be0 con 0x7f7ae40d0340 2021-02-10T11:25:31.956 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:31.940 7f7ae8b26700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ae40e5ed0 con 0x7f7ae40d1c70 2021-02-10T11:25:31.956 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:31.940 7f7ae285e700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f7ae40d1870 0x7f7ae41b03f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:50186/0 (socket says 172.21.2.102:50186) 2021-02-10T11:25:31.956 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:31.940 7f7ae285e700 1 -- 172.21.2.102:0/2101727102 learned_addr learned my addr 172.21.2.102:0/2101727102 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:25:31.956 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:31.940 7f7ae305f700 1 --2- 172.21.2.102:0/2101727102 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7ae40d1c70 0x7f7ae40f20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:31.956 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:31.941 7f7ae185c700 1 -- 172.21.2.102:0/2101727102 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 606512848 0 0) 0x7f7acc001180 con 0x7f7ae40d1870 2021-02-10T11:25:31.956 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:31.941 7f7ae185c700 1 -- 172.21.2.102:0/2101727102 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ad0002fe0 con 0x7f7ae40d1870 2021-02-10T11:25:31.956 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:31.941 7f7ae185c700 1 -- 172.21.2.102:0/2101727102 <== mon.1 v1:172.21.2.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2341789364 0 0) 0x7f7ad4000ff0 con 0x7f7ae40d0340 2021-02-10T11:25:31.957 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:31.941 7f7ae185c700 1 -- 172.21.2.102:0/2101727102 --> v1:172.21.2.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ad0003210 con 0x7f7ae40d0340 2021-02-10T11:25:31.957 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:31.941 7f7ae305f700 1 -- 172.21.2.102:0/2101727102 >> v1:172.21.2.113:6789/0 conn(0x7f7ae40d0340 legacy=0x7f7ae40ef640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:31.957 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:31.941 7f7ae305f700 1 -- 172.21.2.102:0/2101727102 >> v1:172.21.2.102:6789/0 conn(0x7f7ae40d1870 legacy=0x7f7ae41b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:31.957 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:31.941 7f7ae305f700 1 -- 172.21.2.102:0/2101727102 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ae40efe30 con 0x7f7ae40d1c70 2021-02-10T11:25:31.957 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:31.941 7f7ae305f700 1 --2- 172.21.2.102:0/2101727102 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7ae40d1c70 0x7f7ae40f20a0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f7adc006e50 tx=0x7f7adc009e70).ready entity=mon.2 client_cookie=b669a334c90f916 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:31.957 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:31.941 7f7ae185c700 1 -- 172.21.2.102:0/2101727102 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7adc006a20 con 0x7f7ae40d1c70 2021-02-10T11:25:31.957 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:31.941 7f7ae185c700 1 -- 172.21.2.102:0/2101727102 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7adc006790 con 0x7f7ae40d1c70 2021-02-10T11:25:31.957 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:31.941 7f7ae185c700 1 -- 172.21.2.102:0/2101727102 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7adc00b700 con 0x7f7ae40d1c70 2021-02-10T11:25:31.957 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:31.941 7f7ae8b26700 1 -- 172.21.2.102:0/2101727102 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7ae40d1c70 msgr2=0x7f7ae40f20a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:31.957 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:31.941 7f7ae8b26700 1 --2- 172.21.2.102:0/2101727102 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7ae40d1c70 0x7f7ae40f20a0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f7adc006e50 tx=0x7f7adc009e70).stop 2021-02-10T11:25:31.957 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:31.942 7f7ae8b26700 1 -- 172.21.2.102:0/2101727102 shutdown_connections 2021-02-10T11:25:31.957 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:31.942 7f7ae8b26700 1 --2- 172.21.2.102:0/2101727102 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7ae40d1c70 0x7f7ae40f20a0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:31.957 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:31.942 7f7ae8b26700 1 -- 172.21.2.102:0/2101727102 >> 172.21.2.102:0/2101727102 conn(0x7f7ae40ce450 msgr2=0x7f7ae40d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:25:31.958 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:31.942 7f7ae8b26700 1 -- 172.21.2.102:0/2101727102 shutdown_connections 2021-02-10T11:25:31.958 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:31.942 7f7ae8b26700 1 -- 172.21.2.102:0/2101727102 wait complete. 2021-02-10T11:25:31.959 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:31.957 7f7ae8b26700 1 Processor -- start 2021-02-10T11:25:31.959 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:31.957 7f7ae8b26700 1 -- start start 2021-02-10T11:25:31.959 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:31.957 7f7ae8b26700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7ae41b7cf0 0x7f7ae419cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:31.959 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:31.957 7f7ae8b26700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7ae41b80f0 0x7f7ae41b84f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:31.960 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:31.957 7f7ae8b26700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7ae419da20 0x7f7ae419de20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:31.960 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:31.957 7f7ae8b26700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ae40a77c0 con 0x7f7ae41b80f0 2021-02-10T11:25:31.960 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:31.957 7f7ae8b26700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ae419d4a0 con 0x7f7ae419da20 2021-02-10T11:25:31.960 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:31.957 7f7ae8b26700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ae40954f0 con 0x7f7ae41b7cf0 2021-02-10T11:25:31.960 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:31.957 7f7ae205d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7ae41b80f0 0x7f7ae41b84f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:31.960 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:31.957 7f7ae205d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7ae41b80f0 0x7f7ae41b84f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:57966/0 (socket says 172.21.2.102:57966) 2021-02-10T11:25:31.960 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:31.957 7f7ae205d700 1 -- 172.21.2.102:0/1205882555 learned_addr learned my addr 172.21.2.102:0/1205882555 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:25:31.960 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:31.958 7f7ae305f700 1 --2- 172.21.2.102:0/1205882555 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7ae419da20 0x7f7ae419de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:31.960 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:31.958 7f7ae285e700 1 --2- 172.21.2.102:0/1205882555 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7ae41b7cf0 0x7f7ae419cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:31.960 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:31.958 7f7ae205d700 1 -- 172.21.2.102:0/1205882555 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7ae41b7cf0 msgr2=0x7f7ae419cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:31.960 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:31.958 7f7ae205d700 1 --2- 172.21.2.102:0/1205882555 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7ae41b7cf0 0x7f7ae419cdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:31.960 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:31.958 7f7ae205d700 1 -- 172.21.2.102:0/1205882555 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7ae419da20 msgr2=0x7f7ae419de20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:31.960 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:31.958 7f7ae205d700 1 --2- 172.21.2.102:0/1205882555 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7ae419da20 0x7f7ae419de20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:31.961 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:31.958 7f7ae205d700 1 -- 172.21.2.102:0/1205882555 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ae41b8d60 con 0x7f7ae41b80f0 2021-02-10T11:25:31.961 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:31.958 7f7ae205d700 1 --2- 172.21.2.102:0/1205882555 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7ae41b80f0 0x7f7ae41b84f0 secure :-1 s=READY pgs=1489 cs=0 l=1 rev1=1 rx=0x7f7ad4006f20 tx=0x7f7ad400b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:31.961 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:31.958 7f7adb7fe700 1 -- 172.21.2.102:0/1205882555 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7ad40009b0 con 0x7f7ae41b80f0 2021-02-10T11:25:31.961 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:31.958 7f7adb7fe700 1 -- 172.21.2.102:0/1205882555 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ad400bd50 con 0x7f7ae41b80f0 2021-02-10T11:25:31.961 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:31.958 7f7ae8b26700 1 -- 172.21.2.102:0/1205882555 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ae41b8d60 con 0x7f7ae41b80f0 2021-02-10T11:25:31.961 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:31.958 7f7adb7fe700 1 -- 172.21.2.102:0/1205882555 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7ad4006240 con 0x7f7ae41b80f0 2021-02-10T11:25:31.961 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:31.958 7f7ae8b26700 1 -- 172.21.2.102:0/1205882555 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ae419e730 con 0x7f7ae41b80f0 2021-02-10T11:25:31.961 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:31.959 7f7adb7fe700 1 -- 172.21.2.102:0/1205882555 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f7ad400a000 con 0x7f7ae41b80f0 2021-02-10T11:25:31.961 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:31.959 7f7adb7fe700 1 --2- 172.21.2.102:0/1205882555 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7ac80186c0 0x7f7ac801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:31.961 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:31.959 7f7adb7fe700 1 -- 172.21.2.102:0/1205882555 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f7ad4032430 con 0x7f7ae41b80f0 2021-02-10T11:25:31.961 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:31.959 7f7ae285e700 1 --2- 172.21.2.102:0/1205882555 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7ac80186c0 0x7f7ac801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:31.961 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:31.959 7f7ae285e700 1 --2- 172.21.2.102:0/1205882555 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7ac80186c0 0x7f7ac801abc0 crc :-1 s=READY pgs=681 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:31.989 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:31.990 7f7ae8b26700 1 -- 172.21.2.102:0/1205882555 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ae40ebc00 con 0x7f7ae41b80f0 2021-02-10T11:25:31.991 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:31.992 7f7adb7fe700 1 -- 172.21.2.102:0/1205882555 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f7ad4000940 con 0x7f7ae41b80f0 2021-02-10T11:25:32.118 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:32.119 7f7ae8b26700 1 -- 172.21.2.102:0/1205882555 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7ae40f0b10 con 0x7f7ae41b80f0 2021-02-10T11:25:32.118 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:32.119 7f7adb7fe700 1 -- 172.21.2.102:0/1205882555 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v65) v1 ==== 74+0+14009 (secure 0 0 0) 0x7f7ad400a000 con 0x7f7ae41b80f0 2021-02-10T11:25:32.119 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:25:32.119 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":65,"fsid":"68db30ea-dbce-4aef-a6df-f01c6fea0513","created":"2021-02-10 11:03:06.331557","modified":"2021-02-10 11:25:31.513840","last_up_change":"2021-02-10 11:03:32.690562","last_in_change":"2021-02-10 11:03:32.690562","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:03:37.322519","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:25:29.586765","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":13,"pool_name":"cephfs_data","create_time":"2021-02-10 11:25:30.558626","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"be691b4d-a039-408a-af80-9d53f02b6f1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6818","nonce":36088},{"type":"v1","addr":"172.21.2.102:6820","nonce":36088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6822","nonce":36088},{"type":"v1","addr":"172.21.2.102:6824","nonce":36088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6830","nonce":36088},{"type":"v1","addr":"172.21.2.102:6832","nonce":36088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6826","nonce":36088},{"type":"v1","addr":"172.21.2.102:6828","nonce":36088}]},"public_addr":"172.21.2.102:6820/36088","cluster_addr":"172.21.2.102:6824/36088","heartbeat_back_addr":"172.21.2.102:6832/36088","heartbeat_front_addr":"172.21.2.102:6828/36088","state":["exists","up"]},{"osd":1,"uuid":"64e3ccc8-9c69-4d7c-865e-af8e83822635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6819","nonce":36087},{"type":"v1","addr":"172.21.2.102:6821","nonce":36087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6823","nonce":36087},{"type":"v1","addr":"172.21.2.102:6825","nonce":36087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6831","nonce":36087},{"type":"v1","addr":"172.21.2.102:6833","nonce":36087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6827","nonce":36087},{"type":"v1","addr":"172.21.2.102:6829","nonce":36087}]},"public_addr":"172.21.2.102:6821/36087","cluster_addr":"172.21.2.102:6825/36087","heartbeat_back_addr":"172.21.2.102:6833/36087","heartbeat_front_addr":"172.21.2.102:6829/36087","state":["exists","up"]},{"osd":2,"uuid":"153700af-747f-481f-bdf4-d6072d3a842f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6810","nonce":36089},{"type":"v1","addr":"172.21.2.102:6811","nonce":36089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6812","nonce":36089},{"type":"v1","addr":"172.21.2.102:6813","nonce":36089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6816","nonce":36089},{"type":"v1","addr":"172.21.2.102:6817","nonce":36089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6814","nonce":36089},{"type":"v1","addr":"172.21.2.102:6815","nonce":36089}]},"public_addr":"172.21.2.102:6811/36089","cluster_addr":"172.21.2.102:6813/36089","heartbeat_back_addr":"172.21.2.102:6817/36089","heartbeat_front_addr":"172.21.2.102:6815/36089","state":["exists","up"]},{"osd":3,"uuid":"cbc678f4-83a9-45e6-8830-2f7c232b1ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6802","nonce":36090},{"type":"v1","addr":"172.21.2.102:6803","nonce":36090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6804","nonce":36090},{"type":"v1","addr":"172.21.2.102:6805","nonce":36090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6808","nonce":36090},{"type":"v1","addr":"172.21.2.102:6809","nonce":36090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6806","nonce":36090},{"type":"v1","addr":"172.21.2.102:6807","nonce":36090}]},"public_addr":"172.21.2.102:6803/36090","cluster_addr":"172.21.2.102:6805/36090","heartbeat_back_addr":"172.21.2.102:6809/36090","heartbeat_front_addr":"172.21.2.102:6807/36090","state":["exists","up"]},{"osd":4,"uuid":"c01e4008-b258-4f10-afd2-9e7f37214351","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6808","nonce":36368},{"type":"v1","addr":"172.21.2.113:6811","nonce":36368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6814","nonce":36368},{"type":"v1","addr":"172.21.2.113:6818","nonce":36368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6826","nonce":36368},{"type":"v1","addr":"172.21.2.113:6828","nonce":36368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6820","nonce":36368},{"type":"v1","addr":"172.21.2.113:6824","nonce":36368}]},"public_addr":"172.21.2.113:6811/36368","cluster_addr":"172.21.2.113:6818/36368","heartbeat_back_addr":"172.21.2.113:6828/36368","heartbeat_front_addr":"172.21.2.113:6824/36368","state":["exists","up"]},{"osd":5,"uuid":"551f6007-daf8-4a98-8ee0-c2b16bcccb5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6802","nonce":36366},{"type":"v1","addr":"172.21.2.113:6805","nonce":36366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6807","nonce":36366},{"type":"v1","addr":"172.21.2.113:6810","nonce":36366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6819","nonce":36366},{"type":"v1","addr":"172.21.2.113:6822","nonce":36366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6812","nonce":36366},{"type":"v1","addr":"172.21.2.113:6815","nonce":36366}]},"public_addr":"172.21.2.113:6805/36366","cluster_addr":"172.21.2.113:6810/36366","heartbeat_back_addr":"172.21.2.113:6822/36366","heartbeat_front_addr":"172.21.2.113:6815/36366","state":["exists","up"]},{"osd":6,"uuid":"bf67c040-0e50-48be-b102-c3943bc9ebef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6800","nonce":36367},{"type":"v1","addr":"172.21.2.113:6801","nonce":36367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6803","nonce":36367},{"type":"v1","addr":"172.21.2.113:6804","nonce":36367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6813","nonce":36367},{"type":"v1","addr":"172.21.2.113:6816","nonce":36367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6806","nonce":36367},{"type":"v1","addr":"172.21.2.113:6809","nonce":36367}]},"public_addr":"172.21.2.113:6801/36367","cluster_addr":"172.21.2.113:6804/36367","heartbeat_back_addr":"172.21.2.113:6816/36367","heartbeat_front_addr":"172.21.2.113:6809/36367","state":["exists","up"]},{"osd":7,"uuid":"f5461f80-018d-428f-be89-2eb2bcbe7bb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6817","nonce":36365},{"type":"v1","addr":"172.21.2.113:6821","nonce":36365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6823","nonce":36365},{"type":"v1","addr":"172.21.2.113:6825","nonce":36365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6830","nonce":36365},{"type":"v1","addr":"172.21.2.113:6831","nonce":36365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6827","nonce":36365},{"type":"v1","addr":"172.21.2.113:6829","nonce":36365}]},"public_addr":"172.21.2.113:6821/36365","cluster_addr":"172.21.2.113:6825/36365","heartbeat_back_addr":"172.21.2.113:6831/36365","heartbeat_front_addr":"172.21.2.113:6829/36365","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:25:32.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:32.121 7f7ae8b26700 1 -- 172.21.2.102:0/1205882555 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7ac80186c0 msgr2=0x7f7ac801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:32.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:32.121 7f7ae8b26700 1 --2- 172.21.2.102:0/1205882555 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7ac80186c0 0x7f7ac801abc0 crc :-1 s=READY pgs=681 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:32.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:32.121 7f7ae8b26700 1 -- 172.21.2.102:0/1205882555 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7ae41b80f0 msgr2=0x7f7ae41b84f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:32.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:32.121 7f7ae8b26700 1 --2- 172.21.2.102:0/1205882555 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7ae41b80f0 0x7f7ae41b84f0 secure :-1 s=READY pgs=1489 cs=0 l=1 rev1=1 rx=0x7f7ad4006f20 tx=0x7f7ad400b960).stop 2021-02-10T11:25:32.121 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:32.121 7f7ae8b26700 1 -- 172.21.2.102:0/1205882555 shutdown_connections 2021-02-10T11:25:32.121 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:32.121 7f7ae8b26700 1 --2- 172.21.2.102:0/1205882555 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7ac80186c0 0x7f7ac801abc0 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:32.121 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:32.121 7f7ae8b26700 1 --2- 172.21.2.102:0/1205882555 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7ae419da20 0x7f7ae419de20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:32.121 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:32.121 7f7ae8b26700 1 --2- 172.21.2.102:0/1205882555 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7ae41b80f0 0x7f7ae41b84f0 unknown :-1 s=CLOSED pgs=1489 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:32.121 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:32.121 7f7ae8b26700 1 --2- 172.21.2.102:0/1205882555 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7ae41b7cf0 0x7f7ae419cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:32.121 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:32.121 7f7ae8b26700 1 -- 172.21.2.102:0/1205882555 >> 172.21.2.102:0/1205882555 conn(0x7f7ae40ed810 msgr2=0x7f7ae4094980 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:25:32.121 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:32.121 7f7ae8b26700 1 -- 172.21.2.102:0/1205882555 shutdown_connections 2021-02-10T11:25:32.121 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:32.121 7f7ae8b26700 1 -- 172.21.2.102:0/1205882555 wait complete. 2021-02-10T11:25:32.130 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:25:32.561 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:32.541 7f3b561da700 1 Processor -- start 2021-02-10T11:25:32.561 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:32.541 7f3b561da700 1 -- start start 2021-02-10T11:25:32.562 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:32.542 7f3b561da700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f3b500d1870 0x7f3b501b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:32.562 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:32.542 7f3b561da700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3b501b0a10 0x7f3b500cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:32.562 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:32.542 7f3b561da700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b500f0b10 con 0x7f3b500d1c70 2021-02-10T11:25:32.562 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:32.542 7f3b561da700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b500e5ed0 con 0x7f3b500d1870 2021-02-10T11:25:32.562 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:32.542 7f3b561da700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b500f0650 con 0x7f3b501b0a10 2021-02-10T11:25:32.563 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:32.542 7f3b4f7fe700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f3b500d1870 0x7f3b501b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:32.563 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:32.542 7f3b4f7fe700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f3b500d1870 0x7f3b501b03f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:57974/0 (socket says 172.21.2.102:57974) 2021-02-10T11:25:32.563 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:32.542 7f3b4f7fe700 1 -- 172.21.2.102:0/775864204 learned_addr learned my addr 172.21.2.102:0/775864204 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:25:32.563 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:32.542 7f3b4effd700 1 --2- 172.21.2.102:0/775864204 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3b501b0a10 0x7f3b500cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:32.563 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:32.542 7f3b4f7fe700 1 -- 172.21.2.102:0/775864204 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3b501b0a10 msgr2=0x7f3b500cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:25:32.564 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:32.542 7f3b4f7fe700 1 --2- 172.21.2.102:0/775864204 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3b501b0a10 0x7f3b500cf330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:32.564 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:32.542 7f3b4f7fe700 1 -- 172.21.2.102:0/775864204 >> v1:172.21.2.113:6789/0 conn(0x7f3b500d1c70 legacy=0x7f3b500cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:32.564 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:32.542 7f3b4f7fe700 1 -- 172.21.2.102:0/775864204 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b500f0870 con 0x7f3b500d1870 2021-02-10T11:25:32.564 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:32.542 7f3b4f7fe700 1 --2- 172.21.2.102:0/775864204 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3b500d1870 0x7f3b501b03f0 secure :-1 s=READY pgs=1490 cs=0 l=1 rev1=1 rx=0x7f3b48006a80 tx=0x7f3b48009f00).ready entity=mon.0 client_cookie=5b2a738d13683384 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:32.565 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:32.542 7f3b4e7fc700 1 -- 172.21.2.102:0/775864204 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3b48000d30 con 0x7f3b500d1870 2021-02-10T11:25:32.565 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:32.542 7f3b4e7fc700 1 -- 172.21.2.102:0/775864204 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b4800b5c0 con 0x7f3b500d1870 2021-02-10T11:25:32.565 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:32.542 7f3b4e7fc700 1 -- 172.21.2.102:0/775864204 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3b4800ba50 con 0x7f3b500d1870 2021-02-10T11:25:32.565 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:32.543 7f3b561da700 1 -- 172.21.2.102:0/775864204 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3b500d1870 msgr2=0x7f3b501b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:32.565 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:32.543 7f3b561da700 1 --2- 172.21.2.102:0/775864204 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3b500d1870 0x7f3b501b03f0 secure :-1 s=READY pgs=1490 cs=0 l=1 rev1=1 rx=0x7f3b48006a80 tx=0x7f3b48009f00).stop 2021-02-10T11:25:32.566 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:32.543 7f3b561da700 1 -- 172.21.2.102:0/775864204 shutdown_connections 2021-02-10T11:25:32.566 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:32.543 7f3b561da700 1 --2- 172.21.2.102:0/775864204 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3b501b0a10 0x7f3b500cf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:32.566 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:32.543 7f3b561da700 1 --2- 172.21.2.102:0/775864204 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3b500d1870 0x7f3b501b03f0 unknown :-1 s=CLOSED pgs=1490 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:32.566 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:32.543 7f3b561da700 1 -- 172.21.2.102:0/775864204 >> 172.21.2.102:0/775864204 conn(0x7f3b500ce450 msgr2=0x7f3b500d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:25:32.566 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:32.543 7f3b561da700 1 -- 172.21.2.102:0/775864204 shutdown_connections 2021-02-10T11:25:32.567 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:32.543 7f3b561da700 1 -- 172.21.2.102:0/775864204 wait complete. 2021-02-10T11:25:32.567 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:32.558 7f3b561da700 1 Processor -- start 2021-02-10T11:25:32.567 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:32.558 7f3b561da700 1 -- start start 2021-02-10T11:25:32.567 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:32.558 7f3b561da700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3b50195650 0x7f3b500b5560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:32.567 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:32.558 7f3b561da700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3b500b5b80 0x7f3b501a2e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:32.568 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:32.558 7f3b561da700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3b500acfa0 0x7f3b501a3440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:32.568 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:32.558 7f3b561da700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b500b6010 con 0x7f3b50195650 2021-02-10T11:25:32.568 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:32.558 7f3b561da700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b501a3a60 con 0x7f3b500b5b80 2021-02-10T11:25:32.568 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:32.558 7f3b561da700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b501a3c40 con 0x7f3b500acfa0 2021-02-10T11:25:32.568 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:32.559 7f3b4f7fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3b50195650 0x7f3b500b5560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:32.569 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:32.559 7f3b4f7fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3b50195650 0x7f3b500b5560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:57980/0 (socket says 172.21.2.102:57980) 2021-02-10T11:25:32.569 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:32.559 7f3b4f7fe700 1 -- 172.21.2.102:0/3394233491 learned_addr learned my addr 172.21.2.102:0/3394233491 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:25:32.569 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:32.559 7f3b4effd700 1 --2- 172.21.2.102:0/3394233491 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3b500b5b80 0x7f3b501a2e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:32.569 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:32.559 7f3b4ffff700 1 --2- 172.21.2.102:0/3394233491 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3b500acfa0 0x7f3b501a3440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:32.569 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:32.559 7f3b4f7fe700 1 -- 172.21.2.102:0/3394233491 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3b500acfa0 msgr2=0x7f3b501a3440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:32.570 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:32.559 7f3b4f7fe700 1 --2- 172.21.2.102:0/3394233491 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3b500acfa0 0x7f3b501a3440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:32.570 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:32.559 7f3b4f7fe700 1 -- 172.21.2.102:0/3394233491 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3b500b5b80 msgr2=0x7f3b501a2e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:32.570 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:32.559 7f3b4f7fe700 1 --2- 172.21.2.102:0/3394233491 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3b500b5b80 0x7f3b501a2e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:32.570 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:32.559 7f3b4f7fe700 1 -- 172.21.2.102:0/3394233491 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b500bcae0 con 0x7f3b50195650 2021-02-10T11:25:32.571 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:32.559 7f3b4f7fe700 1 --2- 172.21.2.102:0/3394233491 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3b50195650 0x7f3b500b5560 secure :-1 s=READY pgs=1491 cs=0 l=1 rev1=1 rx=0x7f3b48009450 tx=0x7f3b4800bd20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:32.571 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:32.559 7f3b47fff700 1 -- 172.21.2.102:0/3394233491 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3b4800c2f0 con 0x7f3b50195650 2021-02-10T11:25:32.571 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:32.560 7f3b47fff700 1 -- 172.21.2.102:0/3394233491 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b4800be60 con 0x7f3b50195650 2021-02-10T11:25:32.571 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:32.560 7f3b561da700 1 -- 172.21.2.102:0/3394233491 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b500bcae0 con 0x7f3b50195650 2021-02-10T11:25:32.571 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:32.560 7f3b47fff700 1 -- 172.21.2.102:0/3394233491 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3b4800c760 con 0x7f3b50195650 2021-02-10T11:25:32.571 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:32.560 7f3b561da700 1 -- 172.21.2.102:0/3394233491 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b500bcae0 con 0x7f3b50195650 2021-02-10T11:25:32.572 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:32.560 7f3b47fff700 1 -- 172.21.2.102:0/3394233491 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f3b48008020 con 0x7f3b50195650 2021-02-10T11:25:32.572 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:32.560 7f3b47fff700 1 --2- 172.21.2.102:0/3394233491 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3b2c018730 0x7f3b2c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:32.572 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:32.560 7f3b47fff700 1 -- 172.21.2.102:0/3394233491 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f3b48032a00 con 0x7f3b50195650 2021-02-10T11:25:32.572 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:32.560 7f3b4effd700 1 --2- 172.21.2.102:0/3394233491 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3b2c018730 0x7f3b2c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:32.572 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:32.561 7f3b4effd700 1 --2- 172.21.2.102:0/3394233491 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3b2c018730 0x7f3b2c01abc0 crc :-1 s=READY pgs=682 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:32.590 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:32.590 7f3b561da700 1 -- 172.21.2.102:0/3394233491 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b500b6010 con 0x7f3b50195650 2021-02-10T11:25:32.592 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:32.593 7f3b47fff700 1 -- 172.21.2.102:0/3394233491 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f3b4801f8c0 con 0x7f3b50195650 2021-02-10T11:25:32.719 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:32.720 7f3b561da700 1 -- 172.21.2.102:0/3394233491 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3b500b6010 con 0x7f3b50195650 2021-02-10T11:25:32.720 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:32.721 7f3b47fff700 1 -- 172.21.2.102:0/3394233491 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v66) v1 ==== 74+0+13996 (secure 0 0 0) 0x7f3b48007ee0 con 0x7f3b50195650 2021-02-10T11:25:32.720 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:25:32.720 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":66,"fsid":"68db30ea-dbce-4aef-a6df-f01c6fea0513","created":"2021-02-10 11:03:06.331557","modified":"2021-02-10 11:25:32.516321","last_up_change":"2021-02-10 11:03:32.690562","last_in_change":"2021-02-10 11:03:32.690562","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:03:37.322519","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:25:29.586765","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":13,"pool_name":"cephfs_data","create_time":"2021-02-10 11:25:30.558626","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"be691b4d-a039-408a-af80-9d53f02b6f1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6818","nonce":36088},{"type":"v1","addr":"172.21.2.102:6820","nonce":36088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6822","nonce":36088},{"type":"v1","addr":"172.21.2.102:6824","nonce":36088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6830","nonce":36088},{"type":"v1","addr":"172.21.2.102:6832","nonce":36088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6826","nonce":36088},{"type":"v1","addr":"172.21.2.102:6828","nonce":36088}]},"public_addr":"172.21.2.102:6820/36088","cluster_addr":"172.21.2.102:6824/36088","heartbeat_back_addr":"172.21.2.102:6832/36088","heartbeat_front_addr":"172.21.2.102:6828/36088","state":["exists","up"]},{"osd":1,"uuid":"64e3ccc8-9c69-4d7c-865e-af8e83822635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6819","nonce":36087},{"type":"v1","addr":"172.21.2.102:6821","nonce":36087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6823","nonce":36087},{"type":"v1","addr":"172.21.2.102:6825","nonce":36087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6831","nonce":36087},{"type":"v1","addr":"172.21.2.102:6833","nonce":36087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6827","nonce":36087},{"type":"v1","addr":"172.21.2.102:6829","nonce":36087}]},"public_addr":"172.21.2.102:6821/36087","cluster_addr":"172.21.2.102:6825/36087","heartbeat_back_addr":"172.21.2.102:6833/36087","heartbeat_front_addr":"172.21.2.102:6829/36087","state":["exists","up"]},{"osd":2,"uuid":"153700af-747f-481f-bdf4-d6072d3a842f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6810","nonce":36089},{"type":"v1","addr":"172.21.2.102:6811","nonce":36089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6812","nonce":36089},{"type":"v1","addr":"172.21.2.102:6813","nonce":36089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6816","nonce":36089},{"type":"v1","addr":"172.21.2.102:6817","nonce":36089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6814","nonce":36089},{"type":"v1","addr":"172.21.2.102:6815","nonce":36089}]},"public_addr":"172.21.2.102:6811/36089","cluster_addr":"172.21.2.102:6813/36089","heartbeat_back_addr":"172.21.2.102:6817/36089","heartbeat_front_addr":"172.21.2.102:6815/36089","state":["exists","up"]},{"osd":3,"uuid":"cbc678f4-83a9-45e6-8830-2f7c232b1ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6802","nonce":36090},{"type":"v1","addr":"172.21.2.102:6803","nonce":36090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6804","nonce":36090},{"type":"v1","addr":"172.21.2.102:6805","nonce":36090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6808","nonce":36090},{"type":"v1","addr":"172.21.2.102:6809","nonce":36090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6806","nonce":36090},{"type":"v1","addr":"172.21.2.102:6807","nonce":36090}]},"public_addr":"172.21.2.102:6803/36090","cluster_addr":"172.21.2.102:6805/36090","heartbeat_back_addr":"172.21.2.102:6809/36090","heartbeat_front_addr":"172.21.2.102:6807/36090","state":["exists","up"]},{"osd":4,"uuid":"c01e4008-b258-4f10-afd2-9e7f37214351","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6808","nonce":36368},{"type":"v1","addr":"172.21.2.113:6811","nonce":36368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6814","nonce":36368},{"type":"v1","addr":"172.21.2.113:6818","nonce":36368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6826","nonce":36368},{"type":"v1","addr":"172.21.2.113:6828","nonce":36368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6820","nonce":36368},{"type":"v1","addr":"172.21.2.113:6824","nonce":36368}]},"public_addr":"172.21.2.113:6811/36368","cluster_addr":"172.21.2.113:6818/36368","heartbeat_back_addr":"172.21.2.113:6828/36368","heartbeat_front_addr":"172.21.2.113:6824/36368","state":["exists","up"]},{"osd":5,"uuid":"551f6007-daf8-4a98-8ee0-c2b16bcccb5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6802","nonce":36366},{"type":"v1","addr":"172.21.2.113:6805","nonce":36366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6807","nonce":36366},{"type":"v1","addr":"172.21.2.113:6810","nonce":36366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6819","nonce":36366},{"type":"v1","addr":"172.21.2.113:6822","nonce":36366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6812","nonce":36366},{"type":"v1","addr":"172.21.2.113:6815","nonce":36366}]},"public_addr":"172.21.2.113:6805/36366","cluster_addr":"172.21.2.113:6810/36366","heartbeat_back_addr":"172.21.2.113:6822/36366","heartbeat_front_addr":"172.21.2.113:6815/36366","state":["exists","up"]},{"osd":6,"uuid":"bf67c040-0e50-48be-b102-c3943bc9ebef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6800","nonce":36367},{"type":"v1","addr":"172.21.2.113:6801","nonce":36367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6803","nonce":36367},{"type":"v1","addr":"172.21.2.113:6804","nonce":36367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6813","nonce":36367},{"type":"v1","addr":"172.21.2.113:6816","nonce":36367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6806","nonce":36367},{"type":"v1","addr":"172.21.2.113:6809","nonce":36367}]},"public_addr":"172.21.2.113:6801/36367","cluster_addr":"172.21.2.113:6804/36367","heartbeat_back_addr":"172.21.2.113:6816/36367","heartbeat_front_addr":"172.21.2.113:6809/36367","state":["exists","up"]},{"osd":7,"uuid":"f5461f80-018d-428f-be89-2eb2bcbe7bb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6817","nonce":36365},{"type":"v1","addr":"172.21.2.113:6821","nonce":36365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6823","nonce":36365},{"type":"v1","addr":"172.21.2.113:6825","nonce":36365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6830","nonce":36365},{"type":"v1","addr":"172.21.2.113:6831","nonce":36365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6827","nonce":36365},{"type":"v1","addr":"172.21.2.113:6829","nonce":36365}]},"public_addr":"172.21.2.113:6821/36365","cluster_addr":"172.21.2.113:6825/36365","heartbeat_back_addr":"172.21.2.113:6831/36365","heartbeat_front_addr":"172.21.2.113:6829/36365","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:25:32.722 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:32.722 7f3b561da700 1 -- 172.21.2.102:0/3394233491 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3b2c018730 msgr2=0x7f3b2c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:32.722 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:32.722 7f3b561da700 1 --2- 172.21.2.102:0/3394233491 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3b2c018730 0x7f3b2c01abc0 crc :-1 s=READY pgs=682 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:32.722 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:32.722 7f3b561da700 1 -- 172.21.2.102:0/3394233491 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3b50195650 msgr2=0x7f3b500b5560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:32.722 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:32.722 7f3b561da700 1 --2- 172.21.2.102:0/3394233491 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3b50195650 0x7f3b500b5560 secure :-1 s=READY pgs=1491 cs=0 l=1 rev1=1 rx=0x7f3b48009450 tx=0x7f3b4800bd20).stop 2021-02-10T11:25:32.722 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:32.722 7f3b561da700 1 -- 172.21.2.102:0/3394233491 shutdown_connections 2021-02-10T11:25:32.722 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:32.722 7f3b561da700 1 --2- 172.21.2.102:0/3394233491 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3b500acfa0 0x7f3b501a3440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:32.722 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:32.722 7f3b561da700 1 --2- 172.21.2.102:0/3394233491 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3b2c018730 0x7f3b2c01abc0 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:32.722 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:32.722 7f3b561da700 1 --2- 172.21.2.102:0/3394233491 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3b500b5b80 0x7f3b501a2e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:32.722 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:32.722 7f3b561da700 1 --2- 172.21.2.102:0/3394233491 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3b50195650 0x7f3b500b5560 unknown :-1 s=CLOSED pgs=1491 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:32.722 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:32.722 7f3b561da700 1 -- 172.21.2.102:0/3394233491 >> 172.21.2.102:0/3394233491 conn(0x7f3b500b3040 msgr2=0x7f3b500c3250 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:25:32.723 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:32.723 7f3b561da700 1 -- 172.21.2.102:0/3394233491 shutdown_connections 2021-02-10T11:25:32.723 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:32.723 7f3b561da700 1 -- 172.21.2.102:0/3394233491 wait complete. 2021-02-10T11:25:32.730 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2021-02-10T11:25:33.162 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:33.145 7f7287337700 1 Processor -- start 2021-02-10T11:25:33.162 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:33.146 7f7287337700 1 -- start start 2021-02-10T11:25:33.162 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:33.146 7f7287337700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f72800d1c70 0x7f72800cef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:33.162 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:33.146 7f7287337700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72800d0170 con 0x7f72800cf580 2021-02-10T11:25:33.163 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:33.146 7f7287337700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72800fe1c0 con 0x7f72800d1870 2021-02-10T11:25:33.163 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:33.146 7f7287337700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72801b0cb0 con 0x7f72800d1c70 2021-02-10T11:25:33.163 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:33.146 7f7285954700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f72800cf580 0x7f72800cf980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:50216/0 (socket says 172.21.2.102:50216) 2021-02-10T11:25:33.163 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:33.146 7f7285954700 1 -- 172.21.2.102:0/3644633495 learned_addr learned my addr 172.21.2.102:0/3644633495 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:25:33.163 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:33.146 7f7284952700 1 --2- 172.21.2.102:0/3644633495 >> v2:172.21.2.102:3300/0 conn(0x7f72800d1c70 0x7f72800cef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:33.163 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:33.147 7f7273fff700 1 -- 172.21.2.102:0/3644633495 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3894365950 0 0) 0x7f7278000f90 con 0x7f72800cf580 2021-02-10T11:25:33.163 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:33.147 7f7273fff700 1 -- 172.21.2.102:0/3644633495 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f726c002fe0 con 0x7f72800cf580 2021-02-10T11:25:33.163 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:33.147 7f7284952700 1 -- 172.21.2.102:0/3644633495 >> v1:172.21.2.113:6789/0 conn(0x7f72800d1870 legacy=0x7f72801b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:25:33.163 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:33.147 7f7284952700 1 -- 172.21.2.102:0/3644633495 >> v1:172.21.2.102:6789/0 conn(0x7f72800cf580 legacy=0x7f72800cf980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:33.163 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:33.147 7f7284952700 1 -- 172.21.2.102:0/3644633495 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7280109a60 con 0x7f72800d1c70 2021-02-10T11:25:33.163 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:33.147 7f7284952700 1 --2- 172.21.2.102:0/3644633495 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f72800d1c70 0x7f72800cef60 secure :-1 s=READY pgs=1493 cs=0 l=1 rev1=1 rx=0x7f7274006650 tx=0x7f7274009a00).ready entity=mon.0 client_cookie=323df20fb8c6df76 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:33.163 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:33.147 7f7273fff700 1 -- 172.21.2.102:0/3644633495 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7274009df0 con 0x7f72800d1c70 2021-02-10T11:25:33.163 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:33.147 7f7273fff700 1 -- 172.21.2.102:0/3644633495 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72740009b0 con 0x7f72800d1c70 2021-02-10T11:25:33.164 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:33.147 7f7273fff700 1 -- 172.21.2.102:0/3644633495 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f727400c2b0 con 0x7f72800d1c70 2021-02-10T11:25:33.164 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:33.147 7f7287337700 1 -- 172.21.2.102:0/3644633495 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f72800d1c70 msgr2=0x7f72800cef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:33.164 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:33.147 7f7287337700 1 --2- 172.21.2.102:0/3644633495 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f72800d1c70 0x7f72800cef60 secure :-1 s=READY pgs=1493 cs=0 l=1 rev1=1 rx=0x7f7274006650 tx=0x7f7274009a00).stop 2021-02-10T11:25:33.164 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:33.148 7f7287337700 1 -- 172.21.2.102:0/3644633495 shutdown_connections 2021-02-10T11:25:33.164 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:33.148 7f7287337700 1 --2- 172.21.2.102:0/3644633495 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f72800d1c70 0x7f72800cef60 unknown :-1 s=CLOSED pgs=1493 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:33.164 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:33.148 7f7287337700 1 -- 172.21.2.102:0/3644633495 >> 172.21.2.102:0/3644633495 conn(0x7f72800ce420 msgr2=0x7f72800d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:25:33.164 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:33.148 7f7287337700 1 -- 172.21.2.102:0/3644633495 shutdown_connections 2021-02-10T11:25:33.164 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:33.148 7f7287337700 1 -- 172.21.2.102:0/3644633495 wait complete. 2021-02-10T11:25:33.166 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:33.163 7f7287337700 1 Processor -- start 2021-02-10T11:25:33.166 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:33.163 7f7287337700 1 -- start start 2021-02-10T11:25:33.166 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:33.164 7f7287337700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f72801a49c0 0x7f72800bbf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:33.166 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:33.164 7f7287337700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f72800a7950 0x7f72800a7d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:33.166 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:33.164 7f7287337700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f72800ba040 0x7f72800ba440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:33.166 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:33.164 7f7287337700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7280086740 con 0x7f72801a49c0 2021-02-10T11:25:33.166 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:33.164 7f7287337700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72800bc5a0 con 0x7f72800ba040 2021-02-10T11:25:33.167 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:33.164 7f7287337700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72800baa60 con 0x7f72800a7950 2021-02-10T11:25:33.167 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:33.164 7f7285153700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f72801a49c0 0x7f72800bbf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:33.167 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:33.164 7f7285153700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f72801a49c0 0x7f72800bbf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:57996/0 (socket says 172.21.2.102:57996) 2021-02-10T11:25:33.167 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:33.164 7f7285153700 1 -- 172.21.2.102:0/252978339 learned_addr learned my addr 172.21.2.102:0/252978339 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:25:33.167 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:33.164 7f7285954700 1 --2- 172.21.2.102:0/252978339 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f72800ba040 0x7f72800ba440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:33.167 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:33.164 7f7284952700 1 --2- 172.21.2.102:0/252978339 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f72800a7950 0x7f72800a7d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:33.167 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:33.164 7f7285153700 1 -- 172.21.2.102:0/252978339 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f72800a7950 msgr2=0x7f72800a7d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:33.167 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:33.164 7f7285153700 1 --2- 172.21.2.102:0/252978339 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f72800a7950 0x7f72800a7d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:33.167 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:33.164 7f7285153700 1 -- 172.21.2.102:0/252978339 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f72800ba040 msgr2=0x7f72800ba440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:33.167 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:33.164 7f7285153700 1 --2- 172.21.2.102:0/252978339 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f72800ba040 0x7f72800ba440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:33.167 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:33.164 7f7285153700 1 -- 172.21.2.102:0/252978339 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72800bac80 con 0x7f72801a49c0 2021-02-10T11:25:33.167 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:33.165 7f7285153700 1 --2- 172.21.2.102:0/252978339 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f72801a49c0 0x7f72800bbf80 secure :-1 s=READY pgs=1494 cs=0 l=1 rev1=1 rx=0x7f727c00b5b0 tx=0x7f727c00b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:33.167 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:33.165 7f7271ffb700 1 -- 172.21.2.102:0/252978339 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f727c00bd40 con 0x7f72801a49c0 2021-02-10T11:25:33.168 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:33.165 7f7271ffb700 1 -- 172.21.2.102:0/252978339 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f727c006240 con 0x7f72801a49c0 2021-02-10T11:25:33.168 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:33.165 7f7287337700 1 -- 172.21.2.102:0/252978339 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72800bac80 con 0x7f72801a49c0 2021-02-10T11:25:33.168 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:33.165 7f7287337700 1 -- 172.21.2.102:0/252978339 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72800bb210 con 0x7f72801a49c0 2021-02-10T11:25:33.168 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:33.165 7f7271ffb700 1 -- 172.21.2.102:0/252978339 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f727c005720 con 0x7f72801a49c0 2021-02-10T11:25:33.168 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:33.166 7f7271ffb700 1 -- 172.21.2.102:0/252978339 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f727c0054a0 con 0x7f72801a49c0 2021-02-10T11:25:33.168 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:33.166 7f7271ffb700 1 --2- 172.21.2.102:0/252978339 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f72600186c0 0x7f726001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:33.168 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:33.166 7f7271ffb700 1 -- 172.21.2.102:0/252978339 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f727c0325a0 con 0x7f72801a49c0 2021-02-10T11:25:33.168 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:33.166 7f7284952700 1 --2- 172.21.2.102:0/252978339 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f72600186c0 0x7f726001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:33.168 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:33.166 7f7284952700 1 --2- 172.21.2.102:0/252978339 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f72600186c0 0x7f726001abc0 crc :-1 s=READY pgs=683 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:33.195 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:33.196 7f7287337700 1 -- 172.21.2.102:0/252978339 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72800bba50 con 0x7f72801a49c0 2021-02-10T11:25:33.198 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:33.199 7f7271ffb700 1 -- 172.21.2.102:0/252978339 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f727c02ae30 con 0x7f72801a49c0 2021-02-10T11:25:33.334 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:33.334 7f7287337700 1 -- 172.21.2.102:0/252978339 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"} v 0) v1 -- 0x7f72800bba50 con 0x7f72801a49c0 2021-02-10T11:25:33.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:33.525 7f7271ffb700 1 -- 172.21.2.102:0/252978339 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"}]=0 v48) v1 ==== 118+0+0 (secure 0 0 0) 0x7f727c02ae30 con 0x7f72801a49c0 2021-02-10T11:25:33.552 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:33.552 7f7287337700 1 -- 172.21.2.102:0/252978339 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f72600186c0 msgr2=0x7f726001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:33.552 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:33.552 7f7287337700 1 --2- 172.21.2.102:0/252978339 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f72600186c0 0x7f726001abc0 crc :-1 s=READY pgs=683 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:33.552 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:33.552 7f7287337700 1 -- 172.21.2.102:0/252978339 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f72801a49c0 msgr2=0x7f72800bbf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:33.552 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:33.552 7f7287337700 1 --2- 172.21.2.102:0/252978339 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f72801a49c0 0x7f72800bbf80 secure :-1 s=READY pgs=1494 cs=0 l=1 rev1=1 rx=0x7f727c00b5b0 tx=0x7f727c00b950).stop 2021-02-10T11:25:33.552 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:33.552 7f7287337700 1 -- 172.21.2.102:0/252978339 shutdown_connections 2021-02-10T11:25:33.552 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:33.552 7f7287337700 1 --2- 172.21.2.102:0/252978339 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f72600186c0 0x7f726001abc0 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:33.552 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:33.552 7f7287337700 1 --2- 172.21.2.102:0/252978339 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f72800ba040 0x7f72800ba440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:33.552 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:33.552 7f7287337700 1 --2- 172.21.2.102:0/252978339 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f72800a7950 0x7f72800a7d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:33.552 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:33.552 7f7287337700 1 --2- 172.21.2.102:0/252978339 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f72801a49c0 0x7f72800bbf80 unknown :-1 s=CLOSED pgs=1494 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:33.552 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:33.552 7f7287337700 1 -- 172.21.2.102:0/252978339 >> 172.21.2.102:0/252978339 conn(0x7f72800874f0 msgr2=0x7f72800ad100 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:25:33.553 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:33.552 7f7287337700 1 -- 172.21.2.102:0/252978339 shutdown_connections 2021-02-10T11:25:33.553 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:33.552 7f7287337700 1 -- 172.21.2.102:0/252978339 wait complete. 2021-02-10T11:25:33.560 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:25:33.987 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:33.971 7f9eb83ab700 1 Processor -- start 2021-02-10T11:25:33.987 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:33.971 7f9eb83ab700 1 -- start start 2021-02-10T11:25:33.987 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:33.971 7f9eb83ab700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f9eb00d1870 0x7f9eb01b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:33.987 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:33.971 7f9eb83ab700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f9eb01b0a10 0x7f9eb00cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:33.987 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:33.971 7f9eb83ab700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9eb00f0b10 con 0x7f9eb00d1c70 2021-02-10T11:25:33.987 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:33.971 7f9eb83ab700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9eb00e5ed0 con 0x7f9eb00d1870 2021-02-10T11:25:33.987 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:33.971 7f9eb83ab700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9eb00f0650 con 0x7f9eb01b0a10 2021-02-10T11:25:33.988 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:33.972 7f9eb61c7700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f9eb00d1870 0x7f9eb01b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:33.988 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:33.972 7f9eb69c8700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f9eb00d1c70 0x7f9eb00cf950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:50226/0 (socket says 172.21.2.102:50226) 2021-02-10T11:25:33.988 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:33.972 7f9eb69c8700 1 -- 172.21.2.102:0/992390855 learned_addr learned my addr 172.21.2.102:0/992390855 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:25:33.988 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:33.972 7f9eb61c7700 1 -- 172.21.2.102:0/992390855 >> v2:172.21.2.113:3300/0 conn(0x7f9eb01b0a10 msgr2=0x7f9eb00cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:25:33.988 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:33.972 7f9eb61c7700 1 --2- 172.21.2.102:0/992390855 >> v2:172.21.2.113:3300/0 conn(0x7f9eb01b0a10 0x7f9eb00cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:33.988 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:33.972 7f9eb61c7700 1 -- 172.21.2.102:0/992390855 >> v1:172.21.2.102:6789/0 conn(0x7f9eb00d1c70 legacy=0x7f9eb00cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:33.988 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:33.972 7f9eb61c7700 1 -- 172.21.2.102:0/992390855 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9eb00f0840 con 0x7f9eb00d1870 2021-02-10T11:25:33.988 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:33.972 7f9eb61c7700 1 --2- 172.21.2.102:0/992390855 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9eb00d1870 0x7f9eb01b03f0 secure :-1 s=READY pgs=1496 cs=0 l=1 rev1=1 rx=0x7f9eac006650 tx=0x7f9eac009a80).ready entity=mon.0 client_cookie=1cf3b535142d2151 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:33.988 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:33.973 7f9eb51c5700 1 -- 172.21.2.102:0/992390855 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9eac00c0c0 con 0x7f9eb00d1870 2021-02-10T11:25:33.988 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:33.973 7f9eb51c5700 1 -- 172.21.2.102:0/992390855 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9eac00c340 con 0x7f9eb00d1870 2021-02-10T11:25:33.988 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:33.973 7f9eb51c5700 1 -- 172.21.2.102:0/992390855 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9eac00c820 con 0x7f9eb00d1870 2021-02-10T11:25:33.988 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:33.973 7f9eb83ab700 1 -- 172.21.2.102:0/992390855 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9eb00d1870 msgr2=0x7f9eb01b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:33.989 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:33.973 7f9eb83ab700 1 --2- 172.21.2.102:0/992390855 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9eb00d1870 0x7f9eb01b03f0 secure :-1 s=READY pgs=1496 cs=0 l=1 rev1=1 rx=0x7f9eac006650 tx=0x7f9eac009a80).stop 2021-02-10T11:25:33.989 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:33.973 7f9eb83ab700 1 -- 172.21.2.102:0/992390855 shutdown_connections 2021-02-10T11:25:33.989 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:33.973 7f9eb83ab700 1 --2- 172.21.2.102:0/992390855 >> v2:172.21.2.113:3300/0 conn(0x7f9eb01b0a10 0x7f9eb00cf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:33.989 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:33.973 7f9eb83ab700 1 --2- 172.21.2.102:0/992390855 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9eb00d1870 0x7f9eb01b03f0 unknown :-1 s=CLOSED pgs=1496 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:33.989 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:33.973 7f9eb83ab700 1 -- 172.21.2.102:0/992390855 >> 172.21.2.102:0/992390855 conn(0x7f9eb00ce450 msgr2=0x7f9eb00d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:25:33.989 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:33.974 7f9eb83ab700 1 -- 172.21.2.102:0/992390855 shutdown_connections 2021-02-10T11:25:33.989 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:33.974 7f9eb83ab700 1 -- 172.21.2.102:0/992390855 wait complete. 2021-02-10T11:25:33.990 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:33.988 7f9eb83ab700 1 Processor -- start 2021-02-10T11:25:33.991 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:33.988 7f9eb83ab700 1 -- start start 2021-02-10T11:25:33.991 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:33.988 7f9eb83ab700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9eb01b3460 0x7f9eb01aaf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:33.991 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:33.988 7f9eb83ab700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9eb01ab560 0x7f9eb00b36a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:33.991 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:33.988 7f9eb83ab700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f9eb01a3630 0x7f9eb00b3cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:33.991 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:33.988 7f9eb83ab700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9eb01aba90 con 0x7f9eb01ab560 2021-02-10T11:25:33.991 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:33.988 7f9eb83ab700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9eb01a31f0 con 0x7f9eb01a3630 2021-02-10T11:25:33.991 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:33.988 7f9eb83ab700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9eb00b42e0 con 0x7f9eb01b3460 2021-02-10T11:25:33.991 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:33.989 7f9eb59c6700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9eb01ab560 0x7f9eb00b36a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:33.991 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:33.989 7f9eb59c6700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9eb01ab560 0x7f9eb00b36a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:58010/0 (socket says 172.21.2.102:58010) 2021-02-10T11:25:33.991 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:33.989 7f9eb59c6700 1 -- 172.21.2.102:0/3009912382 learned_addr learned my addr 172.21.2.102:0/3009912382 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:25:33.991 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:33.989 7f9eb59c6700 1 -- 172.21.2.102:0/3009912382 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9eb01b3460 msgr2=0x7f9eb01aaf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:33.991 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:33.989 7f9eb59c6700 1 --2- 172.21.2.102:0/3009912382 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9eb01b3460 0x7f9eb01aaf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:33.992 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:33.989 7f9eb59c6700 1 -- 172.21.2.102:0/3009912382 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f9eb01a3630 msgr2=0x7f9eb00b3cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:33.992 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:33.989 7f9eb59c6700 1 --2- 172.21.2.102:0/3009912382 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f9eb01a3630 0x7f9eb00b3cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:33.992 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:33.989 7f9eb59c6700 1 -- 172.21.2.102:0/3009912382 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9eb01a3c80 con 0x7f9eb01ab560 2021-02-10T11:25:33.992 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:33.989 7f9eb59c6700 1 --2- 172.21.2.102:0/3009912382 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9eb01ab560 0x7f9eb00b36a0 secure :-1 s=READY pgs=1497 cs=0 l=1 rev1=1 rx=0x7f9ea0006f00 tx=0x7f9ea000b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:33.992 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:33.989 7f9ea6ffd700 1 -- 172.21.2.102:0/3009912382 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9ea0006350 con 0x7f9eb01ab560 2021-02-10T11:25:33.992 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:33.990 7f9ea6ffd700 1 -- 172.21.2.102:0/3009912382 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ea000bdc0 con 0x7f9eb01ab560 2021-02-10T11:25:33.992 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:33.990 7f9eb83ab700 1 -- 172.21.2.102:0/3009912382 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9eb01a3c80 con 0x7f9eb01ab560 2021-02-10T11:25:33.992 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:33.990 7f9ea6ffd700 1 -- 172.21.2.102:0/3009912382 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9ea0002020 con 0x7f9eb01ab560 2021-02-10T11:25:33.992 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:33.990 7f9eb83ab700 1 -- 172.21.2.102:0/3009912382 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9eb01a4210 con 0x7f9eb01ab560 2021-02-10T11:25:33.992 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:33.990 7f9ea6ffd700 1 -- 172.21.2.102:0/3009912382 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f9ea0007020 con 0x7f9eb01ab560 2021-02-10T11:25:33.993 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:33.990 7f9ea6ffd700 1 --2- 172.21.2.102:0/3009912382 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f9e980186c0 0x7f9e9801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:33.993 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:33.990 7f9ea6ffd700 1 -- 172.21.2.102:0/3009912382 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f9ea0032510 con 0x7f9eb01ab560 2021-02-10T11:25:33.993 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:33.990 7f9eb61c7700 1 --2- 172.21.2.102:0/3009912382 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f9e980186c0 0x7f9e9801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:33.993 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:33.990 7f9eb61c7700 1 --2- 172.21.2.102:0/3009912382 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f9e980186c0 0x7f9e9801abc0 crc :-1 s=READY pgs=684 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:34.020 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:34.021 7f9eb83ab700 1 -- 172.21.2.102:0/3009912382 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9eb00ee060 con 0x7f9eb01ab560 2021-02-10T11:25:34.023 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:34.024 7f9ea6ffd700 1 -- 172.21.2.102:0/3009912382 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f9ea0020f90 con 0x7f9eb01ab560 2021-02-10T11:25:34.151 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:34.152 7f9eb83ab700 1 -- 172.21.2.102:0/3009912382 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9eb00ce450 con 0x7f9eb01ab560 2021-02-10T11:25:34.151 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:34.152 7f9ea6ffd700 1 -- 172.21.2.102:0/3009912382 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) v1 ==== 94+0+2101 (secure 0 0 0) 0x7f9ea0020f90 con 0x7f9eb01ab560 2021-02-10T11:25:34.152 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:25:34.152 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":48,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8412,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.102:6835/3414701760","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6834","nonce":3414701760},{"type":"v1","addr":"172.21.2.102:6835","nonce":3414701760}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":44}],"filesystems":[{"mdsmap":{"epoch":48,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:25:31.513947","modified":"2021-02-10 11:25:33.524416","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5813},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5813":{"gid":5813,"name":"b","rank":0,"incarnation":45,"state":"up:active","state_seq":3,"addr":"172.21.2.113:6833/2951398038","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6832","nonce":2951398038},{"type":"v1","addr":"172.21.2.113:6833","nonce":2951398038}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":6}]} 2021-02-10T11:25:34.152 INFO:teuthology.orchestra.run.gibba002.stderr:dumped fsmap epoch 48 2021-02-10T11:25:34.153 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:34.154 7f9eb83ab700 1 -- 172.21.2.102:0/3009912382 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f9e980186c0 msgr2=0x7f9e9801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:34.153 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:34.154 7f9eb83ab700 1 --2- 172.21.2.102:0/3009912382 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f9e980186c0 0x7f9e9801abc0 crc :-1 s=READY pgs=684 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:34.153 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:34.154 7f9eb83ab700 1 -- 172.21.2.102:0/3009912382 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9eb01ab560 msgr2=0x7f9eb00b36a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:34.153 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:34.154 7f9eb83ab700 1 --2- 172.21.2.102:0/3009912382 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9eb01ab560 0x7f9eb00b36a0 secure :-1 s=READY pgs=1497 cs=0 l=1 rev1=1 rx=0x7f9ea0006f00 tx=0x7f9ea000b940).stop 2021-02-10T11:25:34.153 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:34.154 7f9eb83ab700 1 -- 172.21.2.102:0/3009912382 shutdown_connections 2021-02-10T11:25:34.154 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:34.154 7f9eb83ab700 1 --2- 172.21.2.102:0/3009912382 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f9e980186c0 0x7f9e9801abc0 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:34.154 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:34.154 7f9eb83ab700 1 --2- 172.21.2.102:0/3009912382 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f9eb01a3630 0x7f9eb00b3cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:34.154 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:34.154 7f9eb83ab700 1 --2- 172.21.2.102:0/3009912382 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9eb01ab560 0x7f9eb00b36a0 unknown :-1 s=CLOSED pgs=1497 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:34.154 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:34.154 7f9eb83ab700 1 --2- 172.21.2.102:0/3009912382 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9eb01b3460 0x7f9eb01aaf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:34.154 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:34.154 7f9eb83ab700 1 -- 172.21.2.102:0/3009912382 >> 172.21.2.102:0/3009912382 conn(0x7f9eb019c5f0 msgr2=0x7f9eb01b38a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:25:34.154 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:34.154 7f9eb83ab700 1 -- 172.21.2.102:0/3009912382 shutdown_connections 2021-02-10T11:25:34.155 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:34.154 7f9eb83ab700 1 -- 172.21.2.102:0/3009912382 wait complete. 2021-02-10T11:25:34.162 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:25:34.591 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:34.574 7fd3685ee700 1 Processor -- start 2021-02-10T11:25:34.591 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:34.574 7fd3685ee700 1 -- start start 2021-02-10T11:25:34.591 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:34.575 7fd3685ee700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fd3600d0340 0x7fd3600ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:34.591 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:34.575 7fd3685ee700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3600f0b10 con 0x7fd3600d1870 2021-02-10T11:25:34.591 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:34.575 7fd3685ee700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3601b0be0 con 0x7fd3600d1c70 2021-02-10T11:25:34.591 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:34.575 7fd3685ee700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3600e5ed0 con 0x7fd3600d0340 2021-02-10T11:25:34.591 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:34.575 7fd36640a700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fd3600d1870 0x7fd3601b03f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:50240/0 (socket says 172.21.2.102:50240) 2021-02-10T11:25:34.592 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:34.575 7fd36640a700 1 -- 172.21.2.102:0/2850561946 learned_addr learned my addr 172.21.2.102:0/2850561946 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:25:34.592 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:34.575 7fd365c09700 1 --2- 172.21.2.102:0/2850561946 >> v2:172.21.2.113:3300/0 conn(0x7fd3600d0340 0x7fd3600ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:34.592 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:34.576 7fd365408700 1 -- 172.21.2.102:0/2850561946 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1314901407 0 0) 0x7fd35c0009b0 con 0x7fd3600d1870 2021-02-10T11:25:34.592 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:34.576 7fd365408700 1 -- 172.21.2.102:0/2850561946 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd34c002fe0 con 0x7fd3600d1870 2021-02-10T11:25:34.592 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:34.576 7fd365c09700 1 -- 172.21.2.102:0/2850561946 >> v1:172.21.2.113:6789/0 conn(0x7fd3600d1c70 legacy=0x7fd3600efc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:34.592 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:34.576 7fd365c09700 1 -- 172.21.2.102:0/2850561946 >> v1:172.21.2.102:6789/0 conn(0x7fd3600d1870 legacy=0x7fd3601b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:34.592 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:34.576 7fd365c09700 1 -- 172.21.2.102:0/2850561946 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3600f0450 con 0x7fd3600d0340 2021-02-10T11:25:34.592 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:34.576 7fd365c09700 1 --2- 172.21.2.102:0/2850561946 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd3600d0340 0x7fd3600ef640 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 rx=0x7fd354006650 tx=0x7fd354009a80).ready entity=mon.1 client_cookie=687738d5b48d619d server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:34.592 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:34.576 7fd365408700 1 -- 172.21.2.102:0/2850561946 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd3540009b0 con 0x7fd3600d0340 2021-02-10T11:25:34.592 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:34.576 7fd365408700 1 -- 172.21.2.102:0/2850561946 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd35400c0c0 con 0x7fd3600d0340 2021-02-10T11:25:34.592 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:34.576 7fd365408700 1 -- 172.21.2.102:0/2850561946 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd35400c5a0 con 0x7fd3600d0340 2021-02-10T11:25:34.592 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:34.576 7fd3685ee700 1 -- 172.21.2.102:0/2850561946 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd3600d0340 msgr2=0x7fd3600ef640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:34.593 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:34.576 7fd3685ee700 1 --2- 172.21.2.102:0/2850561946 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd3600d0340 0x7fd3600ef640 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 rx=0x7fd354006650 tx=0x7fd354009a80).stop 2021-02-10T11:25:34.593 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:34.577 7fd3685ee700 1 -- 172.21.2.102:0/2850561946 shutdown_connections 2021-02-10T11:25:34.593 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:34.577 7fd3685ee700 1 --2- 172.21.2.102:0/2850561946 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd3600d0340 0x7fd3600ef640 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:34.593 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:34.577 7fd3685ee700 1 -- 172.21.2.102:0/2850561946 >> 172.21.2.102:0/2850561946 conn(0x7fd3600ce450 msgr2=0x7fd3600d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:25:34.593 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:34.577 7fd3685ee700 1 -- 172.21.2.102:0/2850561946 shutdown_connections 2021-02-10T11:25:34.593 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:34.577 7fd3685ee700 1 -- 172.21.2.102:0/2850561946 wait complete. 2021-02-10T11:25:34.593 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:34.592 7fd3685ee700 1 Processor -- start 2021-02-10T11:25:34.593 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:34.592 7fd3685ee700 1 -- start start 2021-02-10T11:25:34.595 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:34.592 7fd3685ee700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd360093fb0 0x7fd3600f0ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:34.595 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:34.592 7fd3685ee700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd3600f11c0 0x7fd360092c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:34.595 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:34.592 7fd3685ee700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd3600943b0 0x7fd360093280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:34.595 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:34.592 7fd3685ee700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd360093a30 con 0x7fd360093fb0 2021-02-10T11:25:34.595 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:34.592 7fd3685ee700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd360094f90 con 0x7fd3600f11c0 2021-02-10T11:25:34.595 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:34.592 7fd3685ee700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd360095170 con 0x7fd3600943b0 2021-02-10T11:25:34.596 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:34.592 7fd36640a700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd360093fb0 0x7fd3600f0ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:34.596 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:34.592 7fd36640a700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd360093fb0 0x7fd3600f0ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:58022/0 (socket says 172.21.2.102:58022) 2021-02-10T11:25:34.596 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:34.592 7fd36640a700 1 -- 172.21.2.102:0/2288694437 learned_addr learned my addr 172.21.2.102:0/2288694437 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:25:34.596 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:34.593 7fd366c0b700 1 --2- 172.21.2.102:0/2288694437 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd3600943b0 0x7fd360093280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:34.596 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:34.593 7fd365c09700 1 --2- 172.21.2.102:0/2288694437 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd3600f11c0 0x7fd360092c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:34.596 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:34.593 7fd36640a700 1 -- 172.21.2.102:0/2288694437 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd3600943b0 msgr2=0x7fd360093280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:34.596 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:34.593 7fd36640a700 1 --2- 172.21.2.102:0/2288694437 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd3600943b0 0x7fd360093280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:34.596 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:34.593 7fd36640a700 1 -- 172.21.2.102:0/2288694437 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd3600f11c0 msgr2=0x7fd360092c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:34.596 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:34.593 7fd36640a700 1 --2- 172.21.2.102:0/2288694437 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd3600f11c0 0x7fd360092c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:34.596 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:34.593 7fd36640a700 1 -- 172.21.2.102:0/2288694437 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd360095350 con 0x7fd360093fb0 2021-02-10T11:25:34.596 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:34.593 7fd366c0b700 1 --2- 172.21.2.102:0/2288694437 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd3600943b0 0x7fd360093280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:25:34.597 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:34.593 7fd36640a700 1 --2- 172.21.2.102:0/2288694437 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd360093fb0 0x7fd3600f0ba0 secure :-1 s=READY pgs=1499 cs=0 l=1 rev1=1 rx=0x7fd35c001c60 tx=0x7fd35c00a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:34.597 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:34.593 7fd352ffd700 1 -- 172.21.2.102:0/2288694437 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd35c00acc0 con 0x7fd360093fb0 2021-02-10T11:25:34.597 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:34.593 7fd352ffd700 1 -- 172.21.2.102:0/2288694437 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd35c004140 con 0x7fd360093fb0 2021-02-10T11:25:34.597 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:34.593 7fd352ffd700 1 -- 172.21.2.102:0/2288694437 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd35c004660 con 0x7fd360093fb0 2021-02-10T11:25:34.597 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:34.593 7fd3685ee700 1 -- 172.21.2.102:0/2288694437 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd360095350 con 0x7fd360093fb0 2021-02-10T11:25:34.597 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:34.593 7fd3685ee700 1 -- 172.21.2.102:0/2288694437 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd360095350 con 0x7fd360093fb0 2021-02-10T11:25:34.597 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:34.594 7fd352ffd700 1 -- 172.21.2.102:0/2288694437 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd35c004320 con 0x7fd360093fb0 2021-02-10T11:25:34.597 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:34.594 7fd352ffd700 1 --2- 172.21.2.102:0/2288694437 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd3480186c0 0x7fd34801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:34.597 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:34.594 7fd352ffd700 1 -- 172.21.2.102:0/2288694437 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fd35c032520 con 0x7fd360093fb0 2021-02-10T11:25:34.598 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:34.594 7fd365c09700 1 --2- 172.21.2.102:0/2288694437 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd3480186c0 0x7fd34801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:34.598 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:34.594 7fd365c09700 1 --2- 172.21.2.102:0/2288694437 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd3480186c0 0x7fd34801abc0 crc :-1 s=READY pgs=685 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:34.624 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:34.625 7fd3685ee700 1 -- 172.21.2.102:0/2288694437 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3600eddf0 con 0x7fd360093fb0 2021-02-10T11:25:34.627 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:34.628 7fd352ffd700 1 -- 172.21.2.102:0/2288694437 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd35c000e20 con 0x7fd360093fb0 2021-02-10T11:25:34.752 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:34.753 7fd3685ee700 1 -- 172.21.2.102:0/2288694437 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd3600eddf0 con 0x7fd360093fb0 2021-02-10T11:25:34.753 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:34.754 7fd352ffd700 1 -- 172.21.2.102:0/2288694437 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v66) v1 ==== 74+0+13996 (secure 0 0 0) 0x7fd35c00acc0 con 0x7fd360093fb0 2021-02-10T11:25:34.753 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:25:34.753 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":66,"fsid":"68db30ea-dbce-4aef-a6df-f01c6fea0513","created":"2021-02-10 11:03:06.331557","modified":"2021-02-10 11:25:32.516321","last_up_change":"2021-02-10 11:03:32.690562","last_in_change":"2021-02-10 11:03:32.690562","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:03:37.322519","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:25:29.586765","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":13,"pool_name":"cephfs_data","create_time":"2021-02-10 11:25:30.558626","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"be691b4d-a039-408a-af80-9d53f02b6f1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6818","nonce":36088},{"type":"v1","addr":"172.21.2.102:6820","nonce":36088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6822","nonce":36088},{"type":"v1","addr":"172.21.2.102:6824","nonce":36088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6830","nonce":36088},{"type":"v1","addr":"172.21.2.102:6832","nonce":36088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6826","nonce":36088},{"type":"v1","addr":"172.21.2.102:6828","nonce":36088}]},"public_addr":"172.21.2.102:6820/36088","cluster_addr":"172.21.2.102:6824/36088","heartbeat_back_addr":"172.21.2.102:6832/36088","heartbeat_front_addr":"172.21.2.102:6828/36088","state":["exists","up"]},{"osd":1,"uuid":"64e3ccc8-9c69-4d7c-865e-af8e83822635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6819","nonce":36087},{"type":"v1","addr":"172.21.2.102:6821","nonce":36087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6823","nonce":36087},{"type":"v1","addr":"172.21.2.102:6825","nonce":36087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6831","nonce":36087},{"type":"v1","addr":"172.21.2.102:6833","nonce":36087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6827","nonce":36087},{"type":"v1","addr":"172.21.2.102:6829","nonce":36087}]},"public_addr":"172.21.2.102:6821/36087","cluster_addr":"172.21.2.102:6825/36087","heartbeat_back_addr":"172.21.2.102:6833/36087","heartbeat_front_addr":"172.21.2.102:6829/36087","state":["exists","up"]},{"osd":2,"uuid":"153700af-747f-481f-bdf4-d6072d3a842f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6810","nonce":36089},{"type":"v1","addr":"172.21.2.102:6811","nonce":36089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6812","nonce":36089},{"type":"v1","addr":"172.21.2.102:6813","nonce":36089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6816","nonce":36089},{"type":"v1","addr":"172.21.2.102:6817","nonce":36089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6814","nonce":36089},{"type":"v1","addr":"172.21.2.102:6815","nonce":36089}]},"public_addr":"172.21.2.102:6811/36089","cluster_addr":"172.21.2.102:6813/36089","heartbeat_back_addr":"172.21.2.102:6817/36089","heartbeat_front_addr":"172.21.2.102:6815/36089","state":["exists","up"]},{"osd":3,"uuid":"cbc678f4-83a9-45e6-8830-2f7c232b1ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6802","nonce":36090},{"type":"v1","addr":"172.21.2.102:6803","nonce":36090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6804","nonce":36090},{"type":"v1","addr":"172.21.2.102:6805","nonce":36090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6808","nonce":36090},{"type":"v1","addr":"172.21.2.102:6809","nonce":36090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6806","nonce":36090},{"type":"v1","addr":"172.21.2.102:6807","nonce":36090}]},"public_addr":"172.21.2.102:6803/36090","cluster_addr":"172.21.2.102:6805/36090","heartbeat_back_addr":"172.21.2.102:6809/36090","heartbeat_front_addr":"172.21.2.102:6807/36090","state":["exists","up"]},{"osd":4,"uuid":"c01e4008-b258-4f10-afd2-9e7f37214351","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6808","nonce":36368},{"type":"v1","addr":"172.21.2.113:6811","nonce":36368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6814","nonce":36368},{"type":"v1","addr":"172.21.2.113:6818","nonce":36368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6826","nonce":36368},{"type":"v1","addr":"172.21.2.113:6828","nonce":36368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6820","nonce":36368},{"type":"v1","addr":"172.21.2.113:6824","nonce":36368}]},"public_addr":"172.21.2.113:6811/36368","cluster_addr":"172.21.2.113:6818/36368","heartbeat_back_addr":"172.21.2.113:6828/36368","heartbeat_front_addr":"172.21.2.113:6824/36368","state":["exists","up"]},{"osd":5,"uuid":"551f6007-daf8-4a98-8ee0-c2b16bcccb5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6802","nonce":36366},{"type":"v1","addr":"172.21.2.113:6805","nonce":36366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6807","nonce":36366},{"type":"v1","addr":"172.21.2.113:6810","nonce":36366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6819","nonce":36366},{"type":"v1","addr":"172.21.2.113:6822","nonce":36366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6812","nonce":36366},{"type":"v1","addr":"172.21.2.113:6815","nonce":36366}]},"public_addr":"172.21.2.113:6805/36366","cluster_addr":"172.21.2.113:6810/36366","heartbeat_back_addr":"172.21.2.113:6822/36366","heartbeat_front_addr":"172.21.2.113:6815/36366","state":["exists","up"]},{"osd":6,"uuid":"bf67c040-0e50-48be-b102-c3943bc9ebef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6800","nonce":36367},{"type":"v1","addr":"172.21.2.113:6801","nonce":36367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6803","nonce":36367},{"type":"v1","addr":"172.21.2.113:6804","nonce":36367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6813","nonce":36367},{"type":"v1","addr":"172.21.2.113:6816","nonce":36367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6806","nonce":36367},{"type":"v1","addr":"172.21.2.113:6809","nonce":36367}]},"public_addr":"172.21.2.113:6801/36367","cluster_addr":"172.21.2.113:6804/36367","heartbeat_back_addr":"172.21.2.113:6816/36367","heartbeat_front_addr":"172.21.2.113:6809/36367","state":["exists","up"]},{"osd":7,"uuid":"f5461f80-018d-428f-be89-2eb2bcbe7bb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6817","nonce":36365},{"type":"v1","addr":"172.21.2.113:6821","nonce":36365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6823","nonce":36365},{"type":"v1","addr":"172.21.2.113:6825","nonce":36365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6830","nonce":36365},{"type":"v1","addr":"172.21.2.113:6831","nonce":36365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6827","nonce":36365},{"type":"v1","addr":"172.21.2.113:6829","nonce":36365}]},"public_addr":"172.21.2.113:6821/36365","cluster_addr":"172.21.2.113:6825/36365","heartbeat_back_addr":"172.21.2.113:6831/36365","heartbeat_front_addr":"172.21.2.113:6829/36365","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:25:34.755 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:34.755 7fd3685ee700 1 -- 172.21.2.102:0/2288694437 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd3480186c0 msgr2=0x7fd34801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:34.755 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:34.755 7fd3685ee700 1 --2- 172.21.2.102:0/2288694437 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd3480186c0 0x7fd34801abc0 crc :-1 s=READY pgs=685 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:34.755 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:34.755 7fd3685ee700 1 -- 172.21.2.102:0/2288694437 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd360093fb0 msgr2=0x7fd3600f0ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:34.755 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:34.755 7fd3685ee700 1 --2- 172.21.2.102:0/2288694437 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd360093fb0 0x7fd3600f0ba0 secure :-1 s=READY pgs=1499 cs=0 l=1 rev1=1 rx=0x7fd35c001c60 tx=0x7fd35c00a8d0).stop 2021-02-10T11:25:34.755 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:34.755 7fd3685ee700 1 -- 172.21.2.102:0/2288694437 shutdown_connections 2021-02-10T11:25:34.755 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:34.755 7fd3685ee700 1 --2- 172.21.2.102:0/2288694437 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd3600943b0 0x7fd360093280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:34.755 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:34.755 7fd3685ee700 1 --2- 172.21.2.102:0/2288694437 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd3480186c0 0x7fd34801abc0 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:34.755 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:34.755 7fd3685ee700 1 --2- 172.21.2.102:0/2288694437 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd3600f11c0 0x7fd360092c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:34.755 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:34.755 7fd3685ee700 1 --2- 172.21.2.102:0/2288694437 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd360093fb0 0x7fd3600f0ba0 unknown :-1 s=CLOSED pgs=1499 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:34.756 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:34.755 7fd3685ee700 1 -- 172.21.2.102:0/2288694437 >> 172.21.2.102:0/2288694437 conn(0x7fd36018b320 msgr2=0x7fd3601b1cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:25:34.756 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:34.755 7fd3685ee700 1 -- 172.21.2.102:0/2288694437 shutdown_connections 2021-02-10T11:25:34.756 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:34.756 7fd3685ee700 1 -- 172.21.2.102:0/2288694437 wait complete. 2021-02-10T11:25:34.764 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:25:34.807 DEBUG:teuthology.orchestra.run.gibba002:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:25:34.810 DEBUG:teuthology.orchestra.run.gibba013:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:25:35.196 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:35.180 7f71351b5700 1 Processor -- start 2021-02-10T11:25:35.197 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:35.180 7f71351b5700 1 -- start start 2021-02-10T11:25:35.197 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:35.181 7f71351b5700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f71300d1c80 0x7f71300cef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:35.197 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:35.181 7f71351b5700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f71300cf590 0x7f71300cf990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:35.197 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:35.181 7f71351b5700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71300fe1d0 con 0x7f71300d1880 2021-02-10T11:25:35.197 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:35.181 7f71351b5700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71301b0cc0 con 0x7f71300d1c80 2021-02-10T11:25:35.197 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:35.181 7f71351b5700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7130109a70 con 0x7f71300cf590 2021-02-10T11:25:35.197 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:35.181 7f712e59c700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f71300d1c80 0x7f71300cef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:35.197 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:35.181 7f712e59c700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f71300d1c80 0x7f71300cef70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:58032/0 (socket says 172.21.2.102:58032) 2021-02-10T11:25:35.197 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:35.181 7f712e59c700 1 -- 172.21.2.102:0/3551548975 learned_addr learned my addr 172.21.2.102:0/3551548975 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:25:35.197 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:35.181 7f712f59e700 1 --2- 172.21.2.102:0/3551548975 >> v2:172.21.2.113:3300/0 conn(0x7f71300cf590 0x7f71300cf990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:35.197 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:35.181 7f712e59c700 1 -- 172.21.2.102:0/3551548975 >> v2:172.21.2.113:3300/0 conn(0x7f71300cf590 msgr2=0x7f71300cf990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:25:35.197 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:35.181 7f712e59c700 1 --2- 172.21.2.102:0/3551548975 >> v2:172.21.2.113:3300/0 conn(0x7f71300cf590 0x7f71300cf990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:35.197 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:35.181 7f712e59c700 1 -- 172.21.2.102:0/3551548975 >> v1:172.21.2.113:6789/0 conn(0x7f71300d1880 legacy=0x7f71301b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:25:35.198 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:35.181 7f712e59c700 1 -- 172.21.2.102:0/3551548975 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7130088070 con 0x7f71300d1c80 2021-02-10T11:25:35.198 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:35.181 7f712f59e700 1 --2- 172.21.2.102:0/3551548975 >> v2:172.21.2.113:3300/0 conn(0x7f71300cf590 0x7f71300cf990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:25:35.198 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:35.181 7f712e59c700 1 --2- 172.21.2.102:0/3551548975 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f71300d1c80 0x7f71300cef70 secure :-1 s=READY pgs=1500 cs=0 l=1 rev1=1 rx=0x7f7120006700 tx=0x7f7120009b80).ready entity=mon.0 client_cookie=a87807f41ec6e287 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:35.198 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:35.182 7f712dd9b700 1 -- 172.21.2.102:0/3551548975 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f712000c0c0 con 0x7f71300d1c80 2021-02-10T11:25:35.198 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:35.182 7f712dd9b700 1 -- 172.21.2.102:0/3551548975 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f712000c340 con 0x7f71300d1c80 2021-02-10T11:25:35.198 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:35.182 7f712dd9b700 1 -- 172.21.2.102:0/3551548975 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f712000c880 con 0x7f71300d1c80 2021-02-10T11:25:35.198 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:35.182 7f71351b5700 1 -- 172.21.2.102:0/3551548975 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f71300d1c80 msgr2=0x7f71300cef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:35.198 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:35.182 7f71351b5700 1 --2- 172.21.2.102:0/3551548975 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f71300d1c80 0x7f71300cef70 secure :-1 s=READY pgs=1500 cs=0 l=1 rev1=1 rx=0x7f7120006700 tx=0x7f7120009b80).stop 2021-02-10T11:25:35.198 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:35.183 7f71351b5700 1 -- 172.21.2.102:0/3551548975 shutdown_connections 2021-02-10T11:25:35.198 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:35.183 7f71351b5700 1 --2- 172.21.2.102:0/3551548975 >> v2:172.21.2.113:3300/0 conn(0x7f71300cf590 0x7f71300cf990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:35.198 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:35.183 7f71351b5700 1 --2- 172.21.2.102:0/3551548975 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f71300d1c80 0x7f71300cef70 unknown :-1 s=CLOSED pgs=1500 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:35.198 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:35.183 7f71351b5700 1 -- 172.21.2.102:0/3551548975 >> 172.21.2.102:0/3551548975 conn(0x7f71300ce430 msgr2=0x7f71300d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:25:35.199 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:35.183 7f71351b5700 1 -- 172.21.2.102:0/3551548975 shutdown_connections 2021-02-10T11:25:35.199 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:35.183 7f71351b5700 1 -- 172.21.2.102:0/3551548975 wait complete. 2021-02-10T11:25:35.200 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:35.198 7f71351b5700 1 Processor -- start 2021-02-10T11:25:35.200 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:35.198 7f71351b5700 1 -- start start 2021-02-10T11:25:35.200 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:35.198 7f71351b5700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f71301b9280 0x7f713018f750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:35.201 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:35.198 7f71351b5700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f71301b9680 0x7f713018fd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:35.201 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:35.198 7f71351b5700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f71301acd60 0x7f71301ad160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:35.201 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:35.198 7f71351b5700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7130190390 con 0x7f71301b9280 2021-02-10T11:25:35.201 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:35.198 7f71351b5700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71301907c0 con 0x7f71301acd60 2021-02-10T11:25:35.201 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:35.198 7f71351b5700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71301ac3d0 con 0x7f71301b9680 2021-02-10T11:25:35.201 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:35.198 7f712ed9d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f71301b9280 0x7f713018f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:35.201 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:35.198 7f712ed9d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f71301b9280 0x7f713018f750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:58036/0 (socket says 172.21.2.102:58036) 2021-02-10T11:25:35.201 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:35.198 7f712ed9d700 1 -- 172.21.2.102:0/600219627 learned_addr learned my addr 172.21.2.102:0/600219627 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:25:35.201 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:35.199 7f712ed9d700 1 -- 172.21.2.102:0/600219627 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f71301b9680 msgr2=0x7f713018fd70 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:25:35.201 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:35.199 7f712ed9d700 1 --2- 172.21.2.102:0/600219627 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f71301b9680 0x7f713018fd70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:35.201 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:35.199 7f712ed9d700 1 -- 172.21.2.102:0/600219627 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f71301acd60 msgr2=0x7f71301ad160 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:25:35.201 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:35.199 7f712ed9d700 1 --2- 172.21.2.102:0/600219627 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f71301acd60 0x7f71301ad160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:35.202 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:35.199 7f712ed9d700 1 -- 172.21.2.102:0/600219627 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71301ad780 con 0x7f71301b9280 2021-02-10T11:25:35.202 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:35.199 7f712ed9d700 1 --2- 172.21.2.102:0/600219627 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f71301b9280 0x7f713018f750 secure :-1 s=READY pgs=1501 cs=0 l=1 rev1=1 rx=0x7f711800b5b0 tx=0x7f711800b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:35.202 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:35.199 7f71277fe700 1 -- 172.21.2.102:0/600219627 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f711800bd40 con 0x7f71301b9280 2021-02-10T11:25:35.202 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:35.199 7f71277fe700 1 -- 172.21.2.102:0/600219627 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71180009b0 con 0x7f71301b9280 2021-02-10T11:25:35.202 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:35.199 7f71351b5700 1 -- 172.21.2.102:0/600219627 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71301ad780 con 0x7f71301b9280 2021-02-10T11:25:35.202 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:35.199 7f71351b5700 1 -- 172.21.2.102:0/600219627 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71301ad780 con 0x7f71301b9280 2021-02-10T11:25:35.202 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:35.199 7f71277fe700 1 -- 172.21.2.102:0/600219627 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f71180062a0 con 0x7f71301b9280 2021-02-10T11:25:35.202 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:35.199 7f71277fe700 1 -- 172.21.2.102:0/600219627 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f7118006020 con 0x7f71301b9280 2021-02-10T11:25:35.202 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:35.199 7f71277fe700 1 --2- 172.21.2.102:0/600219627 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7114018730 0x7f711401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:35.202 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:35.199 7f71277fe700 1 -- 172.21.2.102:0/600219627 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f71180327c0 con 0x7f71301b9280 2021-02-10T11:25:35.202 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:35.200 7f712e59c700 1 --2- 172.21.2.102:0/600219627 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7114018730 0x7f711401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:35.202 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:35.200 7f712e59c700 1 --2- 172.21.2.102:0/600219627 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7114018730 0x7f711401ac10 crc :-1 s=READY pgs=686 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:35.229 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:35.230 7f71351b5700 1 -- 172.21.2.102:0/600219627 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7130190390 con 0x7f71301b9280 2021-02-10T11:25:35.232 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:35.233 7f71277fe700 1 -- 172.21.2.102:0/600219627 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f711801f500 con 0x7f71301b9280 2021-02-10T11:25:35.365 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:35.365 7f71351b5700 1 -- 172.21.2.102:0/600219627 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f7130190390 con 0x7f71301b9280 2021-02-10T11:25:35.368 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:35.369 7f71277fe700 1 -- 172.21.2.102:0/600219627 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.0 v36) v1 ==== 184+0+0 (secure 0 0 0) 0x7f7118006020 con 0x7f71301b9280 2021-02-10T11:25:35.372 INFO:teuthology.orchestra.run.gibba002.stderr:updated caps for client.0 2021-02-10T11:25:35.374 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:35.374 7f71351b5700 1 -- 172.21.2.102:0/600219627 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7114018730 msgr2=0x7f711401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:35.374 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:35.374 7f71351b5700 1 --2- 172.21.2.102:0/600219627 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7114018730 0x7f711401ac10 crc :-1 s=READY pgs=686 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:35.374 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:35.374 7f71351b5700 1 -- 172.21.2.102:0/600219627 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f71301b9280 msgr2=0x7f713018f750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:35.374 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:35.374 7f71351b5700 1 --2- 172.21.2.102:0/600219627 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f71301b9280 0x7f713018f750 secure :-1 s=READY pgs=1501 cs=0 l=1 rev1=1 rx=0x7f711800b5b0 tx=0x7f711800b950).stop 2021-02-10T11:25:35.374 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:35.374 7f71351b5700 1 -- 172.21.2.102:0/600219627 shutdown_connections 2021-02-10T11:25:35.374 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:35.374 7f71351b5700 1 --2- 172.21.2.102:0/600219627 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7114018730 0x7f711401ac10 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:35.374 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:35.374 7f71351b5700 1 --2- 172.21.2.102:0/600219627 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f71301acd60 0x7f71301ad160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:35.374 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:35.374 7f71351b5700 1 --2- 172.21.2.102:0/600219627 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f71301b9680 0x7f713018fd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:35.374 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:35.374 7f71351b5700 1 --2- 172.21.2.102:0/600219627 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f71301b9280 0x7f713018f750 unknown :-1 s=CLOSED pgs=1501 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:35.374 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:35.374 7f71351b5700 1 -- 172.21.2.102:0/600219627 >> 172.21.2.102:0/600219627 conn(0x7f71300845b0 msgr2=0x7f713018c290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:25:35.375 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:35.374 7f71351b5700 1 -- 172.21.2.102:0/600219627 shutdown_connections 2021-02-10T11:25:35.375 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:35.374 7f71351b5700 1 -- 172.21.2.102:0/600219627 wait complete. 2021-02-10T11:25:35.382 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.1 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:25:35.860 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:35.844 7fdd3475a700 1 Processor -- start 2021-02-10T11:25:35.860 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:35.844 7fdd3475a700 1 -- start start 2021-02-10T11:25:35.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:35.845 7fdd3475a700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fdd2c0d1880 0x7fdd2c1b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:35.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:35.845 7fdd3475a700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd2c0fe1d0 con 0x7fdd2c1b0af0 2021-02-10T11:25:35.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:35.845 7fdd3475a700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd2c109a70 con 0x7fdd2c0cff80 2021-02-10T11:25:35.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:35.845 7fdd3475a700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd2c088070 con 0x7fdd2c0d1880 2021-02-10T11:25:35.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:35.845 7fdd32576700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fdd2c0d1880 0x7fdd2c1b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:35.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:35.845 7fdd31d75700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fdd2c1b0af0 0x7fdd2c108950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:50268/0 (socket says 172.21.2.102:50268) 2021-02-10T11:25:35.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:35.845 7fdd31d75700 1 -- 172.21.2.102:0/698213815 learned_addr learned my addr 172.21.2.102:0/698213815 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:25:35.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:35.846 7fdd32576700 1 -- 172.21.2.102:0/698213815 >> v1:172.21.2.113:6789/0 conn(0x7fdd2c0cff80 legacy=0x7fdd2c109140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:25:35.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:35.846 7fdd32576700 1 -- 172.21.2.102:0/698213815 >> v1:172.21.2.102:6789/0 conn(0x7fdd2c1b0af0 legacy=0x7fdd2c108950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:35.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:35.846 7fdd32576700 1 -- 172.21.2.102:0/698213815 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd2c088260 con 0x7fdd2c0d1880 2021-02-10T11:25:35.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:35.846 7fdd32576700 1 --2- 172.21.2.102:0/698213815 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdd2c0d1880 0x7fdd2c1b04d0 secure :-1 s=READY pgs=1503 cs=0 l=1 rev1=1 rx=0x7fdd20006650 tx=0x7fdd20009a00).ready entity=mon.0 client_cookie=b728d383d60c7939 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:35.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:35.846 7fdd31574700 1 -- 172.21.2.102:0/698213815 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdd200009b0 con 0x7fdd2c0d1880 2021-02-10T11:25:35.862 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:35.846 7fdd31574700 1 -- 172.21.2.102:0/698213815 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd2000c0c0 con 0x7fdd2c0d1880 2021-02-10T11:25:35.862 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:35.846 7fdd31574700 1 -- 172.21.2.102:0/698213815 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdd2000c5a0 con 0x7fdd2c0d1880 2021-02-10T11:25:35.862 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:35.846 7fdd3475a700 1 -- 172.21.2.102:0/698213815 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdd2c0d1880 msgr2=0x7fdd2c1b04d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:35.862 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:35.846 7fdd3475a700 1 --2- 172.21.2.102:0/698213815 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdd2c0d1880 0x7fdd2c1b04d0 secure :-1 s=READY pgs=1503 cs=0 l=1 rev1=1 rx=0x7fdd20006650 tx=0x7fdd20009a00).stop 2021-02-10T11:25:35.862 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:35.847 7fdd3475a700 1 -- 172.21.2.102:0/698213815 shutdown_connections 2021-02-10T11:25:35.862 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:35.847 7fdd3475a700 1 --2- 172.21.2.102:0/698213815 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdd2c0d1880 0x7fdd2c1b04d0 unknown :-1 s=CLOSED pgs=1503 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:35.862 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:35.847 7fdd3475a700 1 -- 172.21.2.102:0/698213815 >> 172.21.2.102:0/698213815 conn(0x7fdd2c0ce430 msgr2=0x7fdd2c0d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:25:35.862 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:35.847 7fdd3475a700 1 -- 172.21.2.102:0/698213815 shutdown_connections 2021-02-10T11:25:35.862 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:35.847 7fdd3475a700 1 -- 172.21.2.102:0/698213815 wait complete. 2021-02-10T11:25:35.864 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:35.861 7fdd3475a700 1 Processor -- start 2021-02-10T11:25:35.864 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:35.862 7fdd3475a700 1 -- start start 2021-02-10T11:25:35.864 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:35.862 7fdd3475a700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fdd2c0a34f0 0x7fdd2c106430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:35.864 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:35.862 7fdd3475a700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdd2c106a50 0x7fdd2c0a2490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:35.864 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:35.862 7fdd3475a700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdd2c106e50 0x7fdd2c0a2ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:35.864 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:35.862 7fdd3475a700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd2c0a4e50 con 0x7fdd2c106a50 2021-02-10T11:25:35.864 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:35.862 7fdd3475a700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd2c0a30d0 con 0x7fdd2c0a34f0 2021-02-10T11:25:35.864 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:35.862 7fdd3475a700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd2c0a32f0 con 0x7fdd2c106e50 2021-02-10T11:25:35.864 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:35.862 7fdd31d75700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdd2c106a50 0x7fdd2c0a2490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:35.865 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:35.862 7fdd31d75700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdd2c106a50 0x7fdd2c0a2490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:58054/0 (socket says 172.21.2.102:58054) 2021-02-10T11:25:35.865 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:35.862 7fdd31d75700 1 -- 172.21.2.102:0/4055507852 learned_addr learned my addr 172.21.2.102:0/4055507852 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:25:35.865 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:35.863 7fdd32576700 1 --2- 172.21.2.102:0/4055507852 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fdd2c0a34f0 0x7fdd2c106430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:35.865 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:35.863 7fdd32d77700 1 --2- 172.21.2.102:0/4055507852 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdd2c106e50 0x7fdd2c0a2ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:35.865 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:35.863 7fdd31d75700 1 -- 172.21.2.102:0/4055507852 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdd2c106e50 msgr2=0x7fdd2c0a2ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:35.865 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:35.863 7fdd31d75700 1 --2- 172.21.2.102:0/4055507852 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdd2c106e50 0x7fdd2c0a2ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:35.865 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:35.863 7fdd31d75700 1 -- 172.21.2.102:0/4055507852 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fdd2c0a34f0 msgr2=0x7fdd2c106430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:35.865 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:35.863 7fdd31d75700 1 --2- 172.21.2.102:0/4055507852 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fdd2c0a34f0 0x7fdd2c106430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:35.865 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:35.863 7fdd31d75700 1 -- 172.21.2.102:0/4055507852 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd2c093710 con 0x7fdd2c106a50 2021-02-10T11:25:35.865 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:35.863 7fdd31d75700 1 --2- 172.21.2.102:0/4055507852 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdd2c106a50 0x7fdd2c0a2490 secure :-1 s=READY pgs=1504 cs=0 l=1 rev1=1 rx=0x7fdd2800a4c0 tx=0x7fdd2800a860).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:35.865 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:35.863 7fdd1effd700 1 -- 172.21.2.102:0/4055507852 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdd2800ac50 con 0x7fdd2c106a50 2021-02-10T11:25:35.865 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:35.863 7fdd1effd700 1 -- 172.21.2.102:0/4055507852 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd280012a0 con 0x7fdd2c106a50 2021-02-10T11:25:35.865 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:35.863 7fdd3475a700 1 -- 172.21.2.102:0/4055507852 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd2c093710 con 0x7fdd2c106a50 2021-02-10T11:25:35.866 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:35.863 7fdd1effd700 1 -- 172.21.2.102:0/4055507852 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdd280043c0 con 0x7fdd2c106a50 2021-02-10T11:25:35.866 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:35.863 7fdd3475a700 1 -- 172.21.2.102:0/4055507852 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd2c093ca0 con 0x7fdd2c106a50 2021-02-10T11:25:35.866 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:35.864 7fdd1effd700 1 -- 172.21.2.102:0/4055507852 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fdd28004140 con 0x7fdd2c106a50 2021-02-10T11:25:35.866 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:35.864 7fdd1effd700 1 --2- 172.21.2.102:0/4055507852 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fdd140186c0 0x7fdd1401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:35.866 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:35.864 7fdd1effd700 1 -- 172.21.2.102:0/4055507852 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fdd28032530 con 0x7fdd2c106a50 2021-02-10T11:25:35.866 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:35.864 7fdd32576700 1 --2- 172.21.2.102:0/4055507852 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fdd140186c0 0x7fdd1401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:35.866 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:35.864 7fdd32576700 1 --2- 172.21.2.102:0/4055507852 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fdd140186c0 0x7fdd1401abc0 crc :-1 s=READY pgs=687 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:35.894 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:35.895 7fdd3475a700 1 -- 172.21.2.102:0/4055507852 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd2c080710 con 0x7fdd2c106a50 2021-02-10T11:25:35.896 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:35.897 7fdd1effd700 1 -- 172.21.2.102:0/4055507852 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fdd280009b0 con 0x7fdd2c106a50 2021-02-10T11:25:36.029 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:36.030 7fdd3475a700 1 -- 172.21.2.102:0/4055507852 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7fdd2c0ce430 con 0x7fdd2c106a50 2021-02-10T11:25:36.032 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:36.033 7fdd1effd700 1 -- 172.21.2.102:0/4055507852 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.1 v37) v1 ==== 184+0+0 (secure 0 0 0) 0x7fdd2801f4d0 con 0x7fdd2c106a50 2021-02-10T11:25:36.036 INFO:teuthology.orchestra.run.gibba002.stderr:updated caps for client.1 2021-02-10T11:25:36.038 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:36.038 7fdd3475a700 1 -- 172.21.2.102:0/4055507852 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fdd140186c0 msgr2=0x7fdd1401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:36.038 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:36.038 7fdd3475a700 1 --2- 172.21.2.102:0/4055507852 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fdd140186c0 0x7fdd1401abc0 crc :-1 s=READY pgs=687 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:36.038 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:36.038 7fdd3475a700 1 -- 172.21.2.102:0/4055507852 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdd2c106a50 msgr2=0x7fdd2c0a2490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:36.038 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:36.038 7fdd3475a700 1 --2- 172.21.2.102:0/4055507852 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdd2c106a50 0x7fdd2c0a2490 secure :-1 s=READY pgs=1504 cs=0 l=1 rev1=1 rx=0x7fdd2800a4c0 tx=0x7fdd2800a860).stop 2021-02-10T11:25:36.039 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:36.039 7fdd3475a700 1 -- 172.21.2.102:0/4055507852 shutdown_connections 2021-02-10T11:25:36.039 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:36.039 7fdd3475a700 1 --2- 172.21.2.102:0/4055507852 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdd2c106e50 0x7fdd2c0a2ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:36.039 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:36.039 7fdd3475a700 1 --2- 172.21.2.102:0/4055507852 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdd2c106a50 0x7fdd2c0a2490 unknown :-1 s=CLOSED pgs=1504 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:36.039 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:36.039 7fdd3475a700 1 --2- 172.21.2.102:0/4055507852 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fdd140186c0 0x7fdd1401abc0 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:36.039 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:36.039 7fdd3475a700 1 --2- 172.21.2.102:0/4055507852 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fdd2c0a34f0 0x7fdd2c106430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:36.039 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:36.039 7fdd3475a700 1 -- 172.21.2.102:0/4055507852 >> 172.21.2.102:0/4055507852 conn(0x7fdd2c0b35b0 msgr2=0x7fdd2c1009c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:25:36.039 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:36.039 7fdd3475a700 1 -- 172.21.2.102:0/4055507852 shutdown_connections 2021-02-10T11:25:36.039 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:36.039 7fdd3475a700 1 -- 172.21.2.102:0/4055507852 wait complete. 2021-02-10T11:25:36.047 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.2 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:25:36.486 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:36.470 7f520673d700 1 Processor -- start 2021-02-10T11:25:36.486 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:36.470 7f520673d700 1 -- start start 2021-02-10T11:25:36.486 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:36.471 7f520673d700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f52000d1880 0x7f52001b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:36.486 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:36.471 7f520673d700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f52001b0af0 0x7f5200108950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:36.487 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:36.471 7f520673d700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5200109a70 con 0x7f52000d1c80 2021-02-10T11:25:36.487 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:36.471 7f520673d700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52000e5f00 con 0x7f52001b0af0 2021-02-10T11:25:36.487 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:36.471 7f520673d700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5200109760 con 0x7f52000d1880 2021-02-10T11:25:36.487 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:36.471 7f5204d5a700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f52000d1c80 0x7f5200108f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:50282/0 (socket says 172.21.2.102:50282) 2021-02-10T11:25:36.487 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:36.471 7f5204d5a700 1 -- 172.21.2.102:0/3820110517 learned_addr learned my addr 172.21.2.102:0/3820110517 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:25:36.487 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:36.471 7f51fffff700 1 --2- 172.21.2.102:0/3820110517 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f52000d1880 0x7f52001b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:36.487 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:36.471 7f51ff7fe700 1 --2- 172.21.2.102:0/3820110517 >> v2:172.21.2.113:3300/0 conn(0x7f52001b0af0 0x7f5200108950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:36.487 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:36.471 7f51feffd700 1 -- 172.21.2.102:0/3820110517 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1506836285 0 0) 0x7f51f8001180 con 0x7f52000d1c80 2021-02-10T11:25:36.487 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:36.471 7f51feffd700 1 -- 172.21.2.102:0/3820110517 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f51ec002fe0 con 0x7f52000d1c80 2021-02-10T11:25:36.487 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:36.472 7f51fffff700 1 -- 172.21.2.102:0/3820110517 >> v2:172.21.2.113:3300/0 conn(0x7f52001b0af0 msgr2=0x7f5200108950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:25:36.487 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:36.472 7f51fffff700 1 --2- 172.21.2.102:0/3820110517 >> v2:172.21.2.113:3300/0 conn(0x7f52001b0af0 0x7f5200108950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:36.487 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:36.472 7f51fffff700 1 -- 172.21.2.102:0/3820110517 >> v1:172.21.2.102:6789/0 conn(0x7f52000d1c80 legacy=0x7f5200108f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:36.488 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:36.472 7f51fffff700 1 -- 172.21.2.102:0/3820110517 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5200088070 con 0x7f52000d1880 2021-02-10T11:25:36.488 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:36.472 7f51fffff700 1 --2- 172.21.2.102:0/3820110517 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f52000d1880 0x7f52001b04d0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7f51f00069d0 tx=0x7f51f0009e40).ready entity=mon.2 client_cookie=dff4669d4d2bb83d server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:36.488 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:36.472 7f51feffd700 1 -- 172.21.2.102:0/3820110517 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f51f000b510 con 0x7f52000d1880 2021-02-10T11:25:36.488 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:36.472 7f51feffd700 1 -- 172.21.2.102:0/3820110517 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51f000b890 con 0x7f52000d1880 2021-02-10T11:25:36.488 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:36.472 7f51feffd700 1 -- 172.21.2.102:0/3820110517 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f51f000bdb0 con 0x7f52000d1880 2021-02-10T11:25:36.488 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:36.472 7f520673d700 1 -- 172.21.2.102:0/3820110517 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f52000d1880 msgr2=0x7f52001b04d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:36.488 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:36.472 7f520673d700 1 --2- 172.21.2.102:0/3820110517 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f52000d1880 0x7f52001b04d0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7f51f00069d0 tx=0x7f51f0009e40).stop 2021-02-10T11:25:36.488 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:36.473 7f520673d700 1 -- 172.21.2.102:0/3820110517 shutdown_connections 2021-02-10T11:25:36.488 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:36.473 7f520673d700 1 --2- 172.21.2.102:0/3820110517 >> v2:172.21.2.113:3300/0 conn(0x7f52001b0af0 0x7f5200108950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:36.488 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:36.473 7f520673d700 1 --2- 172.21.2.102:0/3820110517 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f52000d1880 0x7f52001b04d0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:36.488 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:36.473 7f520673d700 1 -- 172.21.2.102:0/3820110517 >> 172.21.2.102:0/3820110517 conn(0x7f52000ce430 msgr2=0x7f52000d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:25:36.488 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:36.473 7f520673d700 1 -- 172.21.2.102:0/3820110517 shutdown_connections 2021-02-10T11:25:36.488 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:36.473 7f520673d700 1 -- 172.21.2.102:0/3820110517 wait complete. 2021-02-10T11:25:36.490 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:36.487 7f520673d700 1 Processor -- start 2021-02-10T11:25:36.490 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:36.487 7f520673d700 1 -- start start 2021-02-10T11:25:36.490 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:36.488 7f520673d700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5200085300 0x7f52000bf1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:36.490 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:36.488 7f520673d700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f52000bf7d0 0x7f5200085fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:36.490 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:36.488 7f520673d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f52000bfbd0 0x7f52000865d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:36.490 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:36.488 7f520673d700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5200084d80 con 0x7f52000bfbd0 2021-02-10T11:25:36.490 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:36.488 7f520673d700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5200085700 con 0x7f52000bf7d0 2021-02-10T11:25:36.490 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:36.488 7f520673d700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5200086bf0 con 0x7f5200085300 2021-02-10T11:25:36.491 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:36.488 7f5204d5a700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f52000bfbd0 0x7f52000865d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:36.491 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:36.488 7f5204d5a700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f52000bfbd0 0x7f52000865d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:58062/0 (socket says 172.21.2.102:58062) 2021-02-10T11:25:36.491 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:36.488 7f5204d5a700 1 -- 172.21.2.102:0/2042592156 learned_addr learned my addr 172.21.2.102:0/2042592156 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:25:36.491 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:36.488 7f5204d5a700 1 -- 172.21.2.102:0/2042592156 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5200085300 msgr2=0x7f52000bf1b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:25:36.491 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:36.488 7f5204d5a700 1 --2- 172.21.2.102:0/2042592156 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5200085300 0x7f52000bf1b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:36.491 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:36.488 7f5204d5a700 1 -- 172.21.2.102:0/2042592156 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f52000bf7d0 msgr2=0x7f5200085fb0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:25:36.491 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:36.488 7f5204d5a700 1 --2- 172.21.2.102:0/2042592156 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f52000bf7d0 0x7f5200085fb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:36.491 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:36.488 7f5204d5a700 1 -- 172.21.2.102:0/2042592156 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5200086e10 con 0x7f52000bfbd0 2021-02-10T11:25:36.491 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:36.488 7f5204d5a700 1 --2- 172.21.2.102:0/2042592156 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f52000bfbd0 0x7f52000865d0 secure :-1 s=READY pgs=1506 cs=0 l=1 rev1=1 rx=0x7f51f800a5d0 tx=0x7f51f800a970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:36.491 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:36.488 7f51fcff9700 1 -- 172.21.2.102:0/2042592156 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f51f800ad60 con 0x7f52000bfbd0 2021-02-10T11:25:36.491 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:36.488 7f51fcff9700 1 -- 172.21.2.102:0/2042592156 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51f80043a0 con 0x7f52000bfbd0 2021-02-10T11:25:36.491 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:36.488 7f51fcff9700 1 -- 172.21.2.102:0/2042592156 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f51f80048c0 con 0x7f52000bfbd0 2021-02-10T11:25:36.491 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:36.488 7f520673d700 1 -- 172.21.2.102:0/2042592156 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5200086e10 con 0x7f52000bfbd0 2021-02-10T11:25:36.492 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:36.488 7f520673d700 1 -- 172.21.2.102:0/2042592156 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52000873a0 con 0x7f52000bfbd0 2021-02-10T11:25:36.492 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:36.489 7f51fcff9700 1 -- 172.21.2.102:0/2042592156 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f51f8000b20 con 0x7f52000bfbd0 2021-02-10T11:25:36.492 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:36.489 7f51fcff9700 1 --2- 172.21.2.102:0/2042592156 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f51e40186c0 0x7f51e401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:36.492 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:36.489 7f51fcff9700 1 -- 172.21.2.102:0/2042592156 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f51f8032540 con 0x7f52000bfbd0 2021-02-10T11:25:36.492 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:36.489 7f51fffff700 1 --2- 172.21.2.102:0/2042592156 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f51e40186c0 0x7f51e401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:36.492 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:36.489 7f51fffff700 1 --2- 172.21.2.102:0/2042592156 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f51e40186c0 0x7f51e401abc0 crc :-1 s=READY pgs=688 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:36.519 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:36.520 7f520673d700 1 -- 172.21.2.102:0/2042592156 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52000829a0 con 0x7f52000bfbd0 2021-02-10T11:25:36.522 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:36.523 7f51fcff9700 1 -- 172.21.2.102:0/2042592156 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f51f8030cf0 con 0x7f52000bfbd0 2021-02-10T11:25:36.652 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:36.653 7f520673d700 1 -- 172.21.2.102:0/2042592156 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f5200109a70 con 0x7f52000bfbd0 2021-02-10T11:25:36.655 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:36.656 7f51fcff9700 1 -- 172.21.2.102:0/2042592156 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.2 v38) v1 ==== 184+0+0 (secure 0 0 0) 0x7f51f801f6e0 con 0x7f52000bfbd0 2021-02-10T11:25:36.655 INFO:teuthology.orchestra.run.gibba002.stderr:updated caps for client.2 2021-02-10T11:25:36.656 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:36.657 7f520673d700 1 -- 172.21.2.102:0/2042592156 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f51e40186c0 msgr2=0x7f51e401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:36.656 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:36.657 7f520673d700 1 --2- 172.21.2.102:0/2042592156 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f51e40186c0 0x7f51e401abc0 crc :-1 s=READY pgs=688 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:36.657 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:36.657 7f520673d700 1 -- 172.21.2.102:0/2042592156 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f52000bfbd0 msgr2=0x7f52000865d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:36.657 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:36.657 7f520673d700 1 --2- 172.21.2.102:0/2042592156 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f52000bfbd0 0x7f52000865d0 secure :-1 s=READY pgs=1506 cs=0 l=1 rev1=1 rx=0x7f51f800a5d0 tx=0x7f51f800a970).stop 2021-02-10T11:25:36.657 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:36.657 7f520673d700 1 -- 172.21.2.102:0/2042592156 shutdown_connections 2021-02-10T11:25:36.657 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:36.657 7f520673d700 1 --2- 172.21.2.102:0/2042592156 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f52000bfbd0 0x7f52000865d0 unknown :-1 s=CLOSED pgs=1506 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:36.657 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:36.657 7f520673d700 1 --2- 172.21.2.102:0/2042592156 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f51e40186c0 0x7f51e401abc0 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:36.657 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:36.657 7f520673d700 1 --2- 172.21.2.102:0/2042592156 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f52000bf7d0 0x7f5200085fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:36.657 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:36.657 7f520673d700 1 --2- 172.21.2.102:0/2042592156 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5200085300 0x7f52000bf1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:36.657 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:36.657 7f520673d700 1 -- 172.21.2.102:0/2042592156 >> 172.21.2.102:0/2042592156 conn(0x7f52000bd7a0 msgr2=0x7f52000b5900 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:25:36.657 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:36.658 7f520673d700 1 -- 172.21.2.102:0/2042592156 shutdown_connections 2021-02-10T11:25:36.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:36.658 7f520673d700 1 -- 172.21.2.102:0/2042592156 wait complete. 2021-02-10T11:25:36.665 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.3 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:25:37.104 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:37.088 7fe775065700 1 Processor -- start 2021-02-10T11:25:37.104 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:37.088 7fe775065700 1 -- start start 2021-02-10T11:25:37.104 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:37.088 7fe775065700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe77010b000 0x7fe7700cf760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:37.104 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:37.088 7fe775065700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7700fe1d0 con 0x7fe7700d1880 2021-02-10T11:25:37.104 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:37.088 7fe775065700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7701b0cc0 con 0x7fe7700d1c80 2021-02-10T11:25:37.104 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:37.088 7fe775065700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe770109a70 con 0x7fe77010b000 2021-02-10T11:25:37.104 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:37.089 7fe76ed9d700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fe7700d1880 0x7fe7701b04d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:50294/0 (socket says 172.21.2.102:50294) 2021-02-10T11:25:37.104 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:37.089 7fe76ed9d700 1 -- 172.21.2.102:0/1806661312 learned_addr learned my addr 172.21.2.102:0/1806661312 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:25:37.105 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:37.089 7fe76f59e700 1 --2- 172.21.2.102:0/1806661312 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe77010b000 0x7fe7700cf760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:37.105 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:37.089 7fe76dd9b700 1 -- 172.21.2.102:0/1806661312 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4160000163 0 0) 0x7fe760001180 con 0x7fe7700d1880 2021-02-10T11:25:37.105 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:37.089 7fe76dd9b700 1 -- 172.21.2.102:0/1806661312 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe75c002fe0 con 0x7fe7700d1880 2021-02-10T11:25:37.105 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:37.089 7fe76f59e700 1 -- 172.21.2.102:0/1806661312 >> v1:172.21.2.113:6789/0 conn(0x7fe7700d1c80 legacy=0x7fe7700cef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:37.105 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:37.089 7fe76f59e700 1 -- 172.21.2.102:0/1806661312 >> v1:172.21.2.102:6789/0 conn(0x7fe7700d1880 legacy=0x7fe7701b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:37.105 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:37.089 7fe76f59e700 1 -- 172.21.2.102:0/1806661312 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe77010b400 con 0x7fe77010b000 2021-02-10T11:25:37.105 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:37.089 7fe76f59e700 1 --2- 172.21.2.102:0/1806661312 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe77010b000 0x7fe7700cf760 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7fe768006e50 tx=0x7fe768009e70).ready entity=mon.2 client_cookie=45983d898580743b server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:37.105 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:37.090 7fe76dd9b700 1 -- 172.21.2.102:0/1806661312 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe768006a20 con 0x7fe77010b000 2021-02-10T11:25:37.105 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:37.090 7fe76dd9b700 1 -- 172.21.2.102:0/1806661312 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe768006790 con 0x7fe77010b000 2021-02-10T11:25:37.105 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:37.090 7fe76dd9b700 1 -- 172.21.2.102:0/1806661312 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe76800b700 con 0x7fe77010b000 2021-02-10T11:25:37.105 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:37.090 7fe775065700 1 -- 172.21.2.102:0/1806661312 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe77010b000 msgr2=0x7fe7700cf760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:37.105 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:37.090 7fe775065700 1 --2- 172.21.2.102:0/1806661312 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe77010b000 0x7fe7700cf760 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7fe768006e50 tx=0x7fe768009e70).stop 2021-02-10T11:25:37.105 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:37.090 7fe775065700 1 -- 172.21.2.102:0/1806661312 shutdown_connections 2021-02-10T11:25:37.105 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:37.090 7fe775065700 1 --2- 172.21.2.102:0/1806661312 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe77010b000 0x7fe7700cf760 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:37.106 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:37.090 7fe775065700 1 -- 172.21.2.102:0/1806661312 >> 172.21.2.102:0/1806661312 conn(0x7fe7700ce430 msgr2=0x7fe7700d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:25:37.106 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:37.090 7fe775065700 1 -- 172.21.2.102:0/1806661312 shutdown_connections 2021-02-10T11:25:37.106 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:37.090 7fe775065700 1 -- 172.21.2.102:0/1806661312 wait complete. 2021-02-10T11:25:37.107 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:37.105 7fe775065700 1 Processor -- start 2021-02-10T11:25:37.107 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:37.105 7fe775065700 1 -- start start 2021-02-10T11:25:37.107 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:37.105 7fe775065700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe770086520 0x7fe7701a1cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:37.108 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:37.105 7fe775065700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe770086920 0x7fe770187e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:37.108 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:37.105 7fe775065700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe7701a22d0 0x7fe7701884a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:37.108 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:37.105 7fe775065700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe770099600 con 0x7fe770086920 2021-02-10T11:25:37.108 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:37.105 7fe775065700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7700999f0 con 0x7fe770086520 2021-02-10T11:25:37.108 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:37.105 7fe775065700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe770188ac0 con 0x7fe7701a22d0 2021-02-10T11:25:37.108 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:37.106 7fe76e59c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe770086920 0x7fe770187e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:37.108 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:37.106 7fe76e59c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe770086920 0x7fe770187e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:58078/0 (socket says 172.21.2.102:58078) 2021-02-10T11:25:37.108 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:37.106 7fe76e59c700 1 -- 172.21.2.102:0/745028089 learned_addr learned my addr 172.21.2.102:0/745028089 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:25:37.108 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:37.106 7fe76e59c700 1 -- 172.21.2.102:0/745028089 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe7701a22d0 msgr2=0x7fe7701884a0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:25:37.108 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:37.106 7fe76e59c700 1 --2- 172.21.2.102:0/745028089 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe7701a22d0 0x7fe7701884a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:37.108 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:37.106 7fe76e59c700 1 -- 172.21.2.102:0/745028089 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe770086520 msgr2=0x7fe7701a1cb0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:25:37.108 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:37.106 7fe76e59c700 1 --2- 172.21.2.102:0/745028089 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe770086520 0x7fe7701a1cb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:37.108 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:37.106 7fe76e59c700 1 -- 172.21.2.102:0/745028089 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe770188ce0 con 0x7fe770086920 2021-02-10T11:25:37.108 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:37.106 7fe76e59c700 1 --2- 172.21.2.102:0/745028089 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe770086920 0x7fe770187e80 secure :-1 s=READY pgs=1508 cs=0 l=1 rev1=1 rx=0x7fe758006ed0 tx=0x7fe75800b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:37.109 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:37.106 7fe7677fe700 1 -- 172.21.2.102:0/745028089 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe758006240 con 0x7fe770086920 2021-02-10T11:25:37.109 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:37.106 7fe7677fe700 1 -- 172.21.2.102:0/745028089 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7580054b0 con 0x7fe770086920 2021-02-10T11:25:37.109 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:37.106 7fe775065700 1 -- 172.21.2.102:0/745028089 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe770188ce0 con 0x7fe770086920 2021-02-10T11:25:37.109 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:37.106 7fe7677fe700 1 -- 172.21.2.102:0/745028089 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe7580059d0 con 0x7fe770086920 2021-02-10T11:25:37.109 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:37.106 7fe775065700 1 -- 172.21.2.102:0/745028089 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe770189270 con 0x7fe770086920 2021-02-10T11:25:37.109 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:37.107 7fe7677fe700 1 -- 172.21.2.102:0/745028089 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fe758006030 con 0x7fe770086920 2021-02-10T11:25:37.109 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:37.107 7fe7677fe700 1 --2- 172.21.2.102:0/745028089 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe7540186c0 0x7fe75401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:37.109 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:37.107 7fe7677fe700 1 -- 172.21.2.102:0/745028089 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fe758032590 con 0x7fe770086920 2021-02-10T11:25:37.109 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:37.107 7fe76ed9d700 1 --2- 172.21.2.102:0/745028089 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe7540186c0 0x7fe75401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:37.109 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:37.108 7fe76ed9d700 1 --2- 172.21.2.102:0/745028089 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe7540186c0 0x7fe75401abc0 crc :-1 s=READY pgs=689 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:37.137 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:37.138 7fe775065700 1 -- 172.21.2.102:0/745028089 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7700fe1d0 con 0x7fe770086920 2021-02-10T11:25:37.140 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:37.141 7fe7677fe700 1 -- 172.21.2.102:0/745028089 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fe758030d40 con 0x7fe770086920 2021-02-10T11:25:37.270 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:37.270 7fe775065700 1 -- 172.21.2.102:0/745028089 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.3", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7fe7700fe1d0 con 0x7fe770086920 2021-02-10T11:25:37.273 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:37.273 7fe7677fe700 1 -- 172.21.2.102:0/745028089 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.3", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.3 v39) v1 ==== 184+0+0 (secure 0 0 0) 0x7fe7580058a0 con 0x7fe770086920 2021-02-10T11:25:37.273 INFO:teuthology.orchestra.run.gibba002.stderr:updated caps for client.3 2021-02-10T11:25:37.274 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:37.275 7fe775065700 1 -- 172.21.2.102:0/745028089 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe7540186c0 msgr2=0x7fe75401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:37.274 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:37.275 7fe775065700 1 --2- 172.21.2.102:0/745028089 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe7540186c0 0x7fe75401abc0 crc :-1 s=READY pgs=689 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:37.274 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:37.275 7fe775065700 1 -- 172.21.2.102:0/745028089 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe770086920 msgr2=0x7fe770187e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:37.274 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:37.275 7fe775065700 1 --2- 172.21.2.102:0/745028089 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe770086920 0x7fe770187e80 secure :-1 s=READY pgs=1508 cs=0 l=1 rev1=1 rx=0x7fe758006ed0 tx=0x7fe75800b960).stop 2021-02-10T11:25:37.275 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:37.275 7fe775065700 1 -- 172.21.2.102:0/745028089 shutdown_connections 2021-02-10T11:25:37.275 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:37.275 7fe775065700 1 --2- 172.21.2.102:0/745028089 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe7701a22d0 0x7fe7701884a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:37.275 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:37.275 7fe775065700 1 --2- 172.21.2.102:0/745028089 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe770086920 0x7fe770187e80 unknown :-1 s=CLOSED pgs=1508 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:37.275 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:37.275 7fe775065700 1 --2- 172.21.2.102:0/745028089 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe7540186c0 0x7fe75401abc0 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:37.275 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:37.275 7fe775065700 1 --2- 172.21.2.102:0/745028089 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe770086520 0x7fe7701a1cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:37.276 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:37.275 7fe775065700 1 -- 172.21.2.102:0/745028089 >> 172.21.2.102:0/745028089 conn(0x7fe77019f880 msgr2=0x7fe77008f890 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:25:37.276 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:37.275 7fe775065700 1 -- 172.21.2.102:0/745028089 shutdown_connections 2021-02-10T11:25:37.276 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:37.276 7fe775065700 1 -- 172.21.2.102:0/745028089 wait complete. 2021-02-10T11:25:37.283 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:25:37.710 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:37.694 7fbccaba5700 1 Processor -- start 2021-02-10T11:25:37.711 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:37.694 7fbccaba5700 1 -- start start 2021-02-10T11:25:37.711 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:37.695 7fbccaba5700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fbcc40d1870 0x7fbcc41b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:37.711 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:37.695 7fbccaba5700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fbcc41b0a10 0x7fbcc40cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:37.711 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:37.695 7fbccaba5700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbcc40f0b10 con 0x7fbcc40d1c70 2021-02-10T11:25:37.711 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:37.695 7fbccaba5700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbcc40e5ed0 con 0x7fbcc40d1870 2021-02-10T11:25:37.711 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:37.695 7fbccaba5700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbcc40f0650 con 0x7fbcc41b0a10 2021-02-10T11:25:37.711 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:37.695 7fbcc91c2700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fbcc40d1c70 0x7fbcc40cf950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:50306/0 (socket says 172.21.2.102:50306) 2021-02-10T11:25:37.711 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:37.695 7fbcc91c2700 1 -- 172.21.2.102:0/390758338 learned_addr learned my addr 172.21.2.102:0/390758338 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:25:37.711 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:37.695 7fbcc89c1700 1 --2- 172.21.2.102:0/390758338 >> v2:172.21.2.102:3300/0 conn(0x7fbcc40d1870 0x7fbcc41b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:37.711 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:37.695 7fbcc3fff700 1 --2- 172.21.2.102:0/390758338 >> v2:172.21.2.113:3300/0 conn(0x7fbcc41b0a10 0x7fbcc40cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:37.711 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:37.695 7fbcc37fe700 1 -- 172.21.2.102:0/390758338 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3590010847 0 0) 0x7fbcbc000c10 con 0x7fbcc40d1c70 2021-02-10T11:25:37.711 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:37.695 7fbcc37fe700 1 -- 172.21.2.102:0/390758338 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbcb0002fe0 con 0x7fbcc40d1c70 2021-02-10T11:25:37.712 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:37.696 7fbcc89c1700 1 -- 172.21.2.102:0/390758338 >> v2:172.21.2.113:3300/0 conn(0x7fbcc41b0a10 msgr2=0x7fbcc40cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:25:37.712 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:37.696 7fbcc89c1700 1 --2- 172.21.2.102:0/390758338 >> v2:172.21.2.113:3300/0 conn(0x7fbcc41b0a10 0x7fbcc40cf330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:37.712 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:37.696 7fbcc89c1700 1 -- 172.21.2.102:0/390758338 >> v1:172.21.2.102:6789/0 conn(0x7fbcc40d1c70 legacy=0x7fbcc40cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:37.712 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:37.696 7fbcc89c1700 1 -- 172.21.2.102:0/390758338 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcc40f0840 con 0x7fbcc40d1870 2021-02-10T11:25:37.712 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:37.696 7fbcc89c1700 1 --2- 172.21.2.102:0/390758338 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbcc40d1870 0x7fbcc41b03f0 secure :-1 s=READY pgs=1510 cs=0 l=1 rev1=1 rx=0x7fbcac006650 tx=0x7fbcac009a80).ready entity=mon.0 client_cookie=c526ff0af1cbb4e6 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:37.712 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:37.696 7fbcc37fe700 1 -- 172.21.2.102:0/390758338 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbcac0009b0 con 0x7fbcc40d1870 2021-02-10T11:25:37.712 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:37.696 7fbcc37fe700 1 -- 172.21.2.102:0/390758338 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbcac00c0c0 con 0x7fbcc40d1870 2021-02-10T11:25:37.712 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:37.696 7fbcc37fe700 1 -- 172.21.2.102:0/390758338 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbcac00c510 con 0x7fbcc40d1870 2021-02-10T11:25:37.712 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:37.696 7fbccaba5700 1 -- 172.21.2.102:0/390758338 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbcc40d1870 msgr2=0x7fbcc41b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:37.712 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:37.696 7fbccaba5700 1 --2- 172.21.2.102:0/390758338 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbcc40d1870 0x7fbcc41b03f0 secure :-1 s=READY pgs=1510 cs=0 l=1 rev1=1 rx=0x7fbcac006650 tx=0x7fbcac009a80).stop 2021-02-10T11:25:37.712 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:37.697 7fbccaba5700 1 -- 172.21.2.102:0/390758338 shutdown_connections 2021-02-10T11:25:37.712 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:37.697 7fbccaba5700 1 --2- 172.21.2.102:0/390758338 >> v2:172.21.2.113:3300/0 conn(0x7fbcc41b0a10 0x7fbcc40cf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:37.712 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:37.697 7fbccaba5700 1 --2- 172.21.2.102:0/390758338 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbcc40d1870 0x7fbcc41b03f0 unknown :-1 s=CLOSED pgs=1510 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:37.712 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:37.697 7fbccaba5700 1 -- 172.21.2.102:0/390758338 >> 172.21.2.102:0/390758338 conn(0x7fbcc40ce450 msgr2=0x7fbcc40d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:25:37.713 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:37.697 7fbccaba5700 1 -- 172.21.2.102:0/390758338 shutdown_connections 2021-02-10T11:25:37.713 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:37.697 7fbccaba5700 1 -- 172.21.2.102:0/390758338 wait complete. 2021-02-10T11:25:37.714 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:37.711 7fbccaba5700 1 Processor -- start 2021-02-10T11:25:37.714 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:37.712 7fbccaba5700 1 -- start start 2021-02-10T11:25:37.714 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:37.712 7fbccaba5700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbcc4094040 0x7fbcc4093370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:37.714 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:37.712 7fbccaba5700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbcc4093990 0x7fbcc40ad700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:37.714 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:37.712 7fbccaba5700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fbcc40add20 0x7fbcc41a2770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:37.714 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:37.712 7fbccaba5700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcc4093d90 con 0x7fbcc4093990 2021-02-10T11:25:37.715 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:37.712 7fbccaba5700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcc41a1e50 con 0x7fbcc40add20 2021-02-10T11:25:37.715 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:37.712 7fbccaba5700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbcc4094590 con 0x7fbcc4094040 2021-02-10T11:25:37.715 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:37.712 7fbcc3fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbcc4093990 0x7fbcc40ad700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:37.715 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:37.712 7fbcc3fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbcc4093990 0x7fbcc40ad700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:58090/0 (socket says 172.21.2.102:58090) 2021-02-10T11:25:37.715 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:37.712 7fbcc3fff700 1 -- 172.21.2.102:0/2901795369 learned_addr learned my addr 172.21.2.102:0/2901795369 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:25:37.715 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:37.713 7fbcc3fff700 1 -- 172.21.2.102:0/2901795369 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbcc4094040 msgr2=0x7fbcc4093370 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:25:37.715 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:37.713 7fbcc3fff700 1 --2- 172.21.2.102:0/2901795369 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbcc4094040 0x7fbcc4093370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:37.715 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:37.713 7fbcc3fff700 1 -- 172.21.2.102:0/2901795369 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fbcc40add20 msgr2=0x7fbcc41a2770 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:25:37.715 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:37.713 7fbcc3fff700 1 --2- 172.21.2.102:0/2901795369 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fbcc40add20 0x7fbcc41a2770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:37.715 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:37.713 7fbcc3fff700 1 -- 172.21.2.102:0/2901795369 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcc40ae120 con 0x7fbcc4093990 2021-02-10T11:25:37.715 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:37.713 7fbcc3fff700 1 --2- 172.21.2.102:0/2901795369 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbcc4093990 0x7fbcc40ad700 secure :-1 s=READY pgs=1511 cs=0 l=1 rev1=1 rx=0x7fbcb4009980 tx=0x7fbcb400bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:37.715 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:37.713 7fbcc17fa700 1 -- 172.21.2.102:0/2901795369 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbcb400be10 con 0x7fbcc4093990 2021-02-10T11:25:37.715 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:37.713 7fbcc17fa700 1 -- 172.21.2.102:0/2901795369 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbcb4006240 con 0x7fbcc4093990 2021-02-10T11:25:37.715 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:37.713 7fbcc17fa700 1 -- 172.21.2.102:0/2901795369 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbcb4006720 con 0x7fbcc4093990 2021-02-10T11:25:37.716 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:37.713 7fbccaba5700 1 -- 172.21.2.102:0/2901795369 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbcc40ae120 con 0x7fbcc4093990 2021-02-10T11:25:37.716 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:37.713 7fbccaba5700 1 -- 172.21.2.102:0/2901795369 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbcc41a2d90 con 0x7fbcc4093990 2021-02-10T11:25:37.716 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:37.714 7fbcc17fa700 1 -- 172.21.2.102:0/2901795369 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fbcb4006030 con 0x7fbcc4093990 2021-02-10T11:25:37.716 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:37.714 7fbcc17fa700 1 --2- 172.21.2.102:0/2901795369 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fbca80186c0 0x7fbca801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:37.716 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:37.714 7fbcc17fa700 1 -- 172.21.2.102:0/2901795369 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fbcb40328f0 con 0x7fbcc4093990 2021-02-10T11:25:37.716 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:37.714 7fbcc89c1700 1 --2- 172.21.2.102:0/2901795369 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fbca80186c0 0x7fbca801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:37.716 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:37.714 7fbcc89c1700 1 --2- 172.21.2.102:0/2901795369 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fbca80186c0 0x7fbca801abc0 crc :-1 s=READY pgs=690 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:37.744 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:37.745 7fbccaba5700 1 -- 172.21.2.102:0/2901795369 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbcc40ebc60 con 0x7fbcc4093990 2021-02-10T11:25:37.746 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:37.747 7fbcc17fa700 1 -- 172.21.2.102:0/2901795369 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fbcb4000cc0 con 0x7fbcc4093990 2021-02-10T11:25:37.875 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:37.875 7fbccaba5700 1 -- 172.21.2.102:0/2901795369 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbcc40f0b10 con 0x7fbcc4093990 2021-02-10T11:25:37.875 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:37.876 7fbcc17fa700 1 -- 172.21.2.102:0/2901795369 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) v1 ==== 94+0+2101 (secure 0 0 0) 0x7fbcb4006030 con 0x7fbcc4093990 2021-02-10T11:25:37.876 INFO:teuthology.orchestra.run.gibba002.stderr:dumped fsmap epoch 48 2021-02-10T11:25:37.876 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:25:37.876 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":48,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8412,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.102:6835/3414701760","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6834","nonce":3414701760},{"type":"v1","addr":"172.21.2.102:6835","nonce":3414701760}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":44}],"filesystems":[{"mdsmap":{"epoch":48,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:25:31.513947","modified":"2021-02-10 11:25:33.524416","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5813},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5813":{"gid":5813,"name":"b","rank":0,"incarnation":45,"state":"up:active","state_seq":3,"addr":"172.21.2.113:6833/2951398038","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6832","nonce":2951398038},{"type":"v1","addr":"172.21.2.113:6833","nonce":2951398038}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":6}]} 2021-02-10T11:25:37.876 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:37.877 7fbccaba5700 1 -- 172.21.2.102:0/2901795369 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fbca80186c0 msgr2=0x7fbca801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:37.877 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:37.877 7fbccaba5700 1 --2- 172.21.2.102:0/2901795369 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fbca80186c0 0x7fbca801abc0 crc :-1 s=READY pgs=690 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:37.877 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:37.877 7fbccaba5700 1 -- 172.21.2.102:0/2901795369 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbcc4093990 msgr2=0x7fbcc40ad700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:37.877 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:37.877 7fbccaba5700 1 --2- 172.21.2.102:0/2901795369 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbcc4093990 0x7fbcc40ad700 secure :-1 s=READY pgs=1511 cs=0 l=1 rev1=1 rx=0x7fbcb4009980 tx=0x7fbcb400bcd0).stop 2021-02-10T11:25:37.877 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:37.878 7fbccaba5700 1 -- 172.21.2.102:0/2901795369 shutdown_connections 2021-02-10T11:25:37.878 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:37.878 7fbccaba5700 1 --2- 172.21.2.102:0/2901795369 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fbca80186c0 0x7fbca801abc0 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:37.878 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:37.878 7fbccaba5700 1 --2- 172.21.2.102:0/2901795369 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fbcc40add20 0x7fbcc41a2770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:37.878 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:37.878 7fbccaba5700 1 --2- 172.21.2.102:0/2901795369 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbcc4093990 0x7fbcc40ad700 unknown :-1 s=CLOSED pgs=1511 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:37.878 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:37.878 7fbccaba5700 1 --2- 172.21.2.102:0/2901795369 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbcc4094040 0x7fbcc4093370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:37.878 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:37.878 7fbccaba5700 1 -- 172.21.2.102:0/2901795369 >> 172.21.2.102:0/2901795369 conn(0x7fbcc419f8a0 msgr2=0x7fbcc4094980 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:25:37.878 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:37.878 7fbccaba5700 1 -- 172.21.2.102:0/2901795369 shutdown_connections 2021-02-10T11:25:37.878 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:37.878 7fbccaba5700 1 -- 172.21.2.102:0/2901795369 wait complete. 2021-02-10T11:25:38.087 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 48, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:25:31.513947', 'modified': '2021-02-10 11:25:33.524416', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5813}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5813': {'gid': 5813, 'name': 'b', 'rank': 0, 'incarnation': 45, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.113:6833/2951398038', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.113:6832', 'nonce': 2951398038}, {'type': 'v1', 'addr': '172.21.2.113:6833', 'nonce': 2951398038}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [13], 'metadata_pool': 12, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:25:38.088 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:25:38.088 DEBUG:teuthology.orchestra.run.gibba013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:25:38.487 INFO:teuthology.orchestra.run.gibba013.stdout:{ 2021-02-10T11:25:38.487 INFO:teuthology.orchestra.run.gibba013.stdout: "cluster_fsid": "68db30ea-dbce-4aef-a6df-f01c6fea0513", 2021-02-10T11:25:38.487 INFO:teuthology.orchestra.run.gibba013.stdout: "whoami": 0, 2021-02-10T11:25:38.487 INFO:teuthology.orchestra.run.gibba013.stdout: "id": 5813, 2021-02-10T11:25:38.487 INFO:teuthology.orchestra.run.gibba013.stdout: "want_state": "up:active", 2021-02-10T11:25:38.487 INFO:teuthology.orchestra.run.gibba013.stdout: "state": "up:active", 2021-02-10T11:25:38.487 INFO:teuthology.orchestra.run.gibba013.stdout: "rank_uptime": 6.9717365559999998, 2021-02-10T11:25:38.487 INFO:teuthology.orchestra.run.gibba013.stdout: "mdsmap_epoch": 48, 2021-02-10T11:25:38.487 INFO:teuthology.orchestra.run.gibba013.stdout: "osdmap_epoch": 66, 2021-02-10T11:25:38.487 INFO:teuthology.orchestra.run.gibba013.stdout: "osdmap_epoch_barrier": 66, 2021-02-10T11:25:38.488 INFO:teuthology.orchestra.run.gibba013.stdout: "uptime": 12.607792288000001 2021-02-10T11:25:38.488 INFO:teuthology.orchestra.run.gibba013.stdout:} 2021-02-10T11:25:38.491 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "68db30ea-dbce-4aef-a6df-f01c6fea0513", "whoami": 0, "id": 5813, "want_state": "up:active", "state": "up:active", "rank_uptime": 6.9717365559999998, "mdsmap_epoch": 48, "osdmap_epoch": 66, "osdmap_epoch_barrier": 66, "uptime": 12.607792288000001 } 2021-02-10T11:25:38.491 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:25:38.919 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:38.902 7f75c1be1700 1 Processor -- start 2021-02-10T11:25:38.920 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:38.902 7f75c1be1700 1 -- start start 2021-02-10T11:25:38.920 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:38.902 7f75c1be1700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f75bc0d1870 0x7f75bc1b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:38.920 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:38.902 7f75c1be1700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f75bc1b0a10 0x7f75bc0cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:38.920 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:38.902 7f75c1be1700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75bc0f0b10 con 0x7f75bc0d1c70 2021-02-10T11:25:38.920 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:38.902 7f75c1be1700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75bc0fe230 con 0x7f75bc1b0a10 2021-02-10T11:25:38.920 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:38.902 7f75c1be1700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75bc0e5ed0 con 0x7f75bc0d1870 2021-02-10T11:25:38.920 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:38.904 7f75bb7fe700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f75bc0d1870 0x7f75bc1b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:38.920 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:38.904 7f75baffd700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f75bc1b0a10 0x7f75bc0cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:38.920 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:38.904 7f75bbfff700 1 --1- >> v1:172.21.2.113:6789/0 conn(0x7f75bc0d1c70 0x7f75bc0cf950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.113:6789/0 says I am v1:172.21.2.102:57642/0 (socket says 172.21.2.102:57642) 2021-02-10T11:25:38.920 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:38.904 7f75bbfff700 1 -- 172.21.2.102:0/3747726214 learned_addr learned my addr 172.21.2.102:0/3747726214 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:25:38.920 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:38.905 7f75baffd700 1 -- 172.21.2.102:0/3747726214 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f75bc0d1870 msgr2=0x7f75bc1b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:25:38.920 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:38.905 7f75baffd700 1 --2- 172.21.2.102:0/3747726214 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f75bc0d1870 0x7f75bc1b03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:38.920 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:38.905 7f75baffd700 1 -- 172.21.2.102:0/3747726214 >> v1:172.21.2.113:6789/0 conn(0x7f75bc0d1c70 legacy=0x7f75bc0cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:38.921 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:38.905 7f75baffd700 1 -- 172.21.2.102:0/3747726214 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75bc0f0650 con 0x7f75bc1b0a10 2021-02-10T11:25:38.921 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:38.905 7f75bb7fe700 1 --2- 172.21.2.102:0/3747726214 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f75bc0d1870 0x7f75bc1b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:25:38.921 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:38.905 7f75baffd700 1 --2- 172.21.2.102:0/3747726214 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f75bc1b0a10 0x7f75bc0cf330 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 rx=0x7f75ac006650 tx=0x7f75ac009a80).ready entity=mon.1 client_cookie=ba2b93f99d54806c server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:38.921 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:38.905 7f75ba7fc700 1 -- 172.21.2.102:0/3747726214 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f75ac0009b0 con 0x7f75bc1b0a10 2021-02-10T11:25:38.921 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:38.906 7f75ba7fc700 1 -- 172.21.2.102:0/3747726214 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75ac00c0c0 con 0x7f75bc1b0a10 2021-02-10T11:25:38.921 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:38.906 7f75ba7fc700 1 -- 172.21.2.102:0/3747726214 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f75ac00c5e0 con 0x7f75bc1b0a10 2021-02-10T11:25:38.921 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:38.906 7f75c1be1700 1 -- 172.21.2.102:0/3747726214 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f75bc1b0a10 msgr2=0x7f75bc0cf330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:38.921 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:38.906 7f75c1be1700 1 --2- 172.21.2.102:0/3747726214 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f75bc1b0a10 0x7f75bc0cf330 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 rx=0x7f75ac006650 tx=0x7f75ac009a80).stop 2021-02-10T11:25:38.921 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:38.906 7f75c1be1700 1 -- 172.21.2.102:0/3747726214 shutdown_connections 2021-02-10T11:25:38.921 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:38.906 7f75c1be1700 1 --2- 172.21.2.102:0/3747726214 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f75bc1b0a10 0x7f75bc0cf330 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:38.921 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:38.906 7f75c1be1700 1 --2- 172.21.2.102:0/3747726214 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f75bc0d1870 0x7f75bc1b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:38.921 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:38.906 7f75c1be1700 1 -- 172.21.2.102:0/3747726214 >> 172.21.2.102:0/3747726214 conn(0x7f75bc0ce450 msgr2=0x7f75bc0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:25:38.921 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:38.906 7f75c1be1700 1 -- 172.21.2.102:0/3747726214 shutdown_connections 2021-02-10T11:25:38.922 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:38.906 7f75c1be1700 1 -- 172.21.2.102:0/3747726214 wait complete. 2021-02-10T11:25:38.923 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:38.921 7f75c1be1700 1 Processor -- start 2021-02-10T11:25:38.923 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:38.921 7f75c1be1700 1 -- start start 2021-02-10T11:25:38.923 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:38.921 7f75c1be1700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f75bc18d870 0x7f75bc0b19b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:38.923 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:38.921 7f75c1be1700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f75bc0b1fd0 0x7f75bc099ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:38.923 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:38.921 7f75c1be1700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f75bc0b23d0 0x7f75bc09a510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:38.923 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:38.921 7f75c1be1700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75bc18dc70 con 0x7f75bc0b1fd0 2021-02-10T11:25:38.923 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:38.921 7f75c1be1700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75bc199e60 con 0x7f75bc0b23d0 2021-02-10T11:25:38.923 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:38.921 7f75c1be1700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75bc09ab30 con 0x7f75bc18d870 2021-02-10T11:25:38.924 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:38.921 7f75baffd700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f75bc0b1fd0 0x7f75bc099ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:38.924 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:38.921 7f75baffd700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f75bc0b1fd0 0x7f75bc099ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:58100/0 (socket says 172.21.2.102:58100) 2021-02-10T11:25:38.924 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:38.921 7f75baffd700 1 -- 172.21.2.102:0/2445994176 learned_addr learned my addr 172.21.2.102:0/2445994176 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:25:38.924 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:38.922 7f75bb7fe700 1 --2- 172.21.2.102:0/2445994176 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f75bc18d870 0x7f75bc0b19b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:38.924 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:38.922 7f75bbfff700 1 --2- 172.21.2.102:0/2445994176 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f75bc0b23d0 0x7f75bc09a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:38.924 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:38.922 7f75baffd700 1 -- 172.21.2.102:0/2445994176 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f75bc18d870 msgr2=0x7f75bc0b19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:38.924 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:38.922 7f75baffd700 1 --2- 172.21.2.102:0/2445994176 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f75bc18d870 0x7f75bc0b19b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:38.924 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:38.922 7f75baffd700 1 -- 172.21.2.102:0/2445994176 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f75bc0b23d0 msgr2=0x7f75bc09a510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:38.924 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:38.922 7f75baffd700 1 --2- 172.21.2.102:0/2445994176 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f75bc0b23d0 0x7f75bc09a510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:38.924 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:38.922 7f75baffd700 1 -- 172.21.2.102:0/2445994176 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75bc09ad50 con 0x7f75bc0b1fd0 2021-02-10T11:25:38.924 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:38.922 7f75bbfff700 1 --2- 172.21.2.102:0/2445994176 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f75bc0b23d0 0x7f75bc09a510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:25:38.924 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:38.922 7f75baffd700 1 --2- 172.21.2.102:0/2445994176 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f75bc0b1fd0 0x7f75bc099ef0 secure :-1 s=READY pgs=1512 cs=0 l=1 rev1=1 rx=0x7f75ac009630 tx=0x7f75ac00b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:38.924 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:38.922 7f75b3fff700 1 -- 172.21.2.102:0/2445994176 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f75ac0093a0 con 0x7f75bc0b1fd0 2021-02-10T11:25:38.925 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:38.922 7f75b3fff700 1 -- 172.21.2.102:0/2445994176 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75ac00bd70 con 0x7f75bc0b1fd0 2021-02-10T11:25:38.925 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:38.922 7f75b3fff700 1 -- 172.21.2.102:0/2445994176 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f75ac0054d0 con 0x7f75bc0b1fd0 2021-02-10T11:25:38.925 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:38.922 7f75c1be1700 1 -- 172.21.2.102:0/2445994176 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75bc09ad50 con 0x7f75bc0b1fd0 2021-02-10T11:25:38.925 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:38.922 7f75c1be1700 1 -- 172.21.2.102:0/2445994176 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75bc09ad50 con 0x7f75bc0b1fd0 2021-02-10T11:25:38.925 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:38.923 7f75b3fff700 1 -- 172.21.2.102:0/2445994176 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f75ac008240 con 0x7f75bc0b1fd0 2021-02-10T11:25:38.925 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:38.923 7f75b3fff700 1 --2- 172.21.2.102:0/2445994176 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f75a00186c0 0x7f75a001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:38.925 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:38.923 7f75b3fff700 1 -- 172.21.2.102:0/2445994176 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f75ac0323e0 con 0x7f75bc0b1fd0 2021-02-10T11:25:38.925 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:38.923 7f75bb7fe700 1 --2- 172.21.2.102:0/2445994176 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f75a00186c0 0x7f75a001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:38.925 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:38.923 7f75bb7fe700 1 --2- 172.21.2.102:0/2445994176 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f75a00186c0 0x7f75a001abc0 crc :-1 s=READY pgs=691 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:38.952 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:38.952 7f75c1be1700 1 -- 172.21.2.102:0/2445994176 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75bc0999c0 con 0x7f75bc0b1fd0 2021-02-10T11:25:38.954 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:38.955 7f75b3fff700 1 -- 172.21.2.102:0/2445994176 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f75ac008030 con 0x7f75bc0b1fd0 2021-02-10T11:25:39.079 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:39.080 7f75c1be1700 1 -- 172.21.2.102:0/2445994176 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f75bc0999c0 con 0x7f75bc0b1fd0 2021-02-10T11:25:39.080 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:39.081 7f75b3fff700 1 -- 172.21.2.102:0/2445994176 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v66) v1 ==== 74+0+13996 (secure 0 0 0) 0x7f75ac008240 con 0x7f75bc0b1fd0 2021-02-10T11:25:39.080 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:25:39.081 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":66,"fsid":"68db30ea-dbce-4aef-a6df-f01c6fea0513","created":"2021-02-10 11:03:06.331557","modified":"2021-02-10 11:25:32.516321","last_up_change":"2021-02-10 11:03:32.690562","last_in_change":"2021-02-10 11:03:32.690562","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:03:37.322519","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:25:29.586765","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":13,"pool_name":"cephfs_data","create_time":"2021-02-10 11:25:30.558626","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"be691b4d-a039-408a-af80-9d53f02b6f1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6818","nonce":36088},{"type":"v1","addr":"172.21.2.102:6820","nonce":36088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6822","nonce":36088},{"type":"v1","addr":"172.21.2.102:6824","nonce":36088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6830","nonce":36088},{"type":"v1","addr":"172.21.2.102:6832","nonce":36088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6826","nonce":36088},{"type":"v1","addr":"172.21.2.102:6828","nonce":36088}]},"public_addr":"172.21.2.102:6820/36088","cluster_addr":"172.21.2.102:6824/36088","heartbeat_back_addr":"172.21.2.102:6832/36088","heartbeat_front_addr":"172.21.2.102:6828/36088","state":["exists","up"]},{"osd":1,"uuid":"64e3ccc8-9c69-4d7c-865e-af8e83822635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6819","nonce":36087},{"type":"v1","addr":"172.21.2.102:6821","nonce":36087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6823","nonce":36087},{"type":"v1","addr":"172.21.2.102:6825","nonce":36087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6831","nonce":36087},{"type":"v1","addr":"172.21.2.102:6833","nonce":36087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6827","nonce":36087},{"type":"v1","addr":"172.21.2.102:6829","nonce":36087}]},"public_addr":"172.21.2.102:6821/36087","cluster_addr":"172.21.2.102:6825/36087","heartbeat_back_addr":"172.21.2.102:6833/36087","heartbeat_front_addr":"172.21.2.102:6829/36087","state":["exists","up"]},{"osd":2,"uuid":"153700af-747f-481f-bdf4-d6072d3a842f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6810","nonce":36089},{"type":"v1","addr":"172.21.2.102:6811","nonce":36089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6812","nonce":36089},{"type":"v1","addr":"172.21.2.102:6813","nonce":36089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6816","nonce":36089},{"type":"v1","addr":"172.21.2.102:6817","nonce":36089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6814","nonce":36089},{"type":"v1","addr":"172.21.2.102:6815","nonce":36089}]},"public_addr":"172.21.2.102:6811/36089","cluster_addr":"172.21.2.102:6813/36089","heartbeat_back_addr":"172.21.2.102:6817/36089","heartbeat_front_addr":"172.21.2.102:6815/36089","state":["exists","up"]},{"osd":3,"uuid":"cbc678f4-83a9-45e6-8830-2f7c232b1ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6802","nonce":36090},{"type":"v1","addr":"172.21.2.102:6803","nonce":36090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6804","nonce":36090},{"type":"v1","addr":"172.21.2.102:6805","nonce":36090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6808","nonce":36090},{"type":"v1","addr":"172.21.2.102:6809","nonce":36090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6806","nonce":36090},{"type":"v1","addr":"172.21.2.102:6807","nonce":36090}]},"public_addr":"172.21.2.102:6803/36090","cluster_addr":"172.21.2.102:6805/36090","heartbeat_back_addr":"172.21.2.102:6809/36090","heartbeat_front_addr":"172.21.2.102:6807/36090","state":["exists","up"]},{"osd":4,"uuid":"c01e4008-b258-4f10-afd2-9e7f37214351","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6808","nonce":36368},{"type":"v1","addr":"172.21.2.113:6811","nonce":36368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6814","nonce":36368},{"type":"v1","addr":"172.21.2.113:6818","nonce":36368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6826","nonce":36368},{"type":"v1","addr":"172.21.2.113:6828","nonce":36368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6820","nonce":36368},{"type":"v1","addr":"172.21.2.113:6824","nonce":36368}]},"public_addr":"172.21.2.113:6811/36368","cluster_addr":"172.21.2.113:6818/36368","heartbeat_back_addr":"172.21.2.113:6828/36368","heartbeat_front_addr":"172.21.2.113:6824/36368","state":["exists","up"]},{"osd":5,"uuid":"551f6007-daf8-4a98-8ee0-c2b16bcccb5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6802","nonce":36366},{"type":"v1","addr":"172.21.2.113:6805","nonce":36366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6807","nonce":36366},{"type":"v1","addr":"172.21.2.113:6810","nonce":36366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6819","nonce":36366},{"type":"v1","addr":"172.21.2.113:6822","nonce":36366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6812","nonce":36366},{"type":"v1","addr":"172.21.2.113:6815","nonce":36366}]},"public_addr":"172.21.2.113:6805/36366","cluster_addr":"172.21.2.113:6810/36366","heartbeat_back_addr":"172.21.2.113:6822/36366","heartbeat_front_addr":"172.21.2.113:6815/36366","state":["exists","up"]},{"osd":6,"uuid":"bf67c040-0e50-48be-b102-c3943bc9ebef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6800","nonce":36367},{"type":"v1","addr":"172.21.2.113:6801","nonce":36367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6803","nonce":36367},{"type":"v1","addr":"172.21.2.113:6804","nonce":36367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6813","nonce":36367},{"type":"v1","addr":"172.21.2.113:6816","nonce":36367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6806","nonce":36367},{"type":"v1","addr":"172.21.2.113:6809","nonce":36367}]},"public_addr":"172.21.2.113:6801/36367","cluster_addr":"172.21.2.113:6804/36367","heartbeat_back_addr":"172.21.2.113:6816/36367","heartbeat_front_addr":"172.21.2.113:6809/36367","state":["exists","up"]},{"osd":7,"uuid":"f5461f80-018d-428f-be89-2eb2bcbe7bb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6817","nonce":36365},{"type":"v1","addr":"172.21.2.113:6821","nonce":36365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6823","nonce":36365},{"type":"v1","addr":"172.21.2.113:6825","nonce":36365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6830","nonce":36365},{"type":"v1","addr":"172.21.2.113:6831","nonce":36365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6827","nonce":36365},{"type":"v1","addr":"172.21.2.113:6829","nonce":36365}]},"public_addr":"172.21.2.113:6821/36365","cluster_addr":"172.21.2.113:6825/36365","heartbeat_back_addr":"172.21.2.113:6831/36365","heartbeat_front_addr":"172.21.2.113:6829/36365","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:25:39.082 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:39.082 7f75c1be1700 1 -- 172.21.2.102:0/2445994176 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f75a00186c0 msgr2=0x7f75a001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:39.082 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:39.082 7f75c1be1700 1 --2- 172.21.2.102:0/2445994176 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f75a00186c0 0x7f75a001abc0 crc :-1 s=READY pgs=691 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:39.082 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:39.082 7f75c1be1700 1 -- 172.21.2.102:0/2445994176 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f75bc0b1fd0 msgr2=0x7f75bc099ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:39.082 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:39.082 7f75c1be1700 1 --2- 172.21.2.102:0/2445994176 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f75bc0b1fd0 0x7f75bc099ef0 secure :-1 s=READY pgs=1512 cs=0 l=1 rev1=1 rx=0x7f75ac009630 tx=0x7f75ac00b980).stop 2021-02-10T11:25:39.082 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:39.082 7f75c1be1700 1 -- 172.21.2.102:0/2445994176 shutdown_connections 2021-02-10T11:25:39.082 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:39.082 7f75c1be1700 1 --2- 172.21.2.102:0/2445994176 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f75a00186c0 0x7f75a001abc0 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:39.082 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:39.082 7f75c1be1700 1 --2- 172.21.2.102:0/2445994176 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f75bc0b23d0 0x7f75bc09a510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:39.083 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:39.082 7f75c1be1700 1 --2- 172.21.2.102:0/2445994176 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f75bc0b1fd0 0x7f75bc099ef0 unknown :-1 s=CLOSED pgs=1512 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:39.083 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:39.082 7f75c1be1700 1 --2- 172.21.2.102:0/2445994176 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f75bc18d870 0x7f75bc0b19b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:39.083 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:39.082 7f75c1be1700 1 -- 172.21.2.102:0/2445994176 >> 172.21.2.102:0/2445994176 conn(0x7f75bc0f0550 msgr2=0x7f75bc191730 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:25:39.083 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:39.083 7f75c1be1700 1 -- 172.21.2.102:0/2445994176 shutdown_connections 2021-02-10T11:25:39.083 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:39.083 7f75c1be1700 1 -- 172.21.2.102:0/2445994176 wait complete. 2021-02-10T11:25:39.091 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:03:37.322519', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 12, 'pool_name': 'cephfs_metadata', 'create_time': '2021-02-10 11:25:29.586765', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '65', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 13, 'pool_name': 'cephfs_data', 'create_time': '2021-02-10 11:25:30.558626', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '66', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-02-10T11:25:39.091 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:25:39.524 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:39.506 7f1a1edf5700 1 Processor -- start 2021-02-10T11:25:39.524 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:39.506 7f1a1edf5700 1 -- start start 2021-02-10T11:25:39.524 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:39.507 7f1a1edf5700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1a180d1c70 0x7f1a180cef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:39.524 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:39.507 7f1a1edf5700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f1a180cf580 0x7f1a180cf980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:39.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:39.507 7f1a1edf5700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a180fe1c0 con 0x7f1a180d1870 2021-02-10T11:25:39.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:39.507 7f1a1edf5700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a181b0cb0 con 0x7f1a180cf580 2021-02-10T11:25:39.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:39.507 7f1a1edf5700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a18109a60 con 0x7f1a180d1c70 2021-02-10T11:25:39.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:39.507 7f1a1cc11700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f1a180d1870 0x7f1a181b04c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:50332/0 (socket says 172.21.2.102:50332) 2021-02-10T11:25:39.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:39.507 7f1a1cc11700 1 -- 172.21.2.102:0/3130192863 learned_addr learned my addr 172.21.2.102:0/3130192863 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:25:39.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:39.507 7f1a177fe700 1 -- 172.21.2.102:0/3130192863 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2719214096 0 0) 0x7f1a000009b0 con 0x7f1a180d1870 2021-02-10T11:25:39.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:39.508 7f1a177fe700 1 -- 172.21.2.102:0/3130192863 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a04002fe0 con 0x7f1a180d1870 2021-02-10T11:25:39.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:39.508 7f1a17fff700 1 --2- 172.21.2.102:0/3130192863 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1a180d1c70 0x7f1a180cef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:39.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:39.508 7f1a1d412700 1 --2- 172.21.2.102:0/3130192863 >> v2:172.21.2.113:3300/0 conn(0x7f1a180cf580 0x7f1a180cf980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:39.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:39.508 7f1a177fe700 1 -- 172.21.2.102:0/3130192863 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1393037475 0 0) 0x7f1a00001010 con 0x7f1a180d1870 2021-02-10T11:25:39.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:39.508 7f1a177fe700 1 -- 172.21.2.102:0/3130192863 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1a180d1c70 msgr2=0x7f1a180cef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:25:39.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:39.508 7f1a177fe700 1 --2- 172.21.2.102:0/3130192863 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1a180d1c70 0x7f1a180cef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:39.526 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:39.508 7f1a177fe700 1 -- 172.21.2.102:0/3130192863 >> v2:172.21.2.113:3300/0 conn(0x7f1a180cf580 msgr2=0x7f1a180cf980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:25:39.526 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:39.508 7f1a177fe700 1 --2- 172.21.2.102:0/3130192863 >> v2:172.21.2.113:3300/0 conn(0x7f1a180cf580 0x7f1a180cf980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:39.526 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:39.508 7f1a177fe700 1 -- 172.21.2.102:0/3130192863 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a18088070 con 0x7f1a180d1870 2021-02-10T11:25:39.526 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:39.508 7f1a177fe700 1 -- 172.21.2.102:0/3130192863 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7f1a00001530 con 0x7f1a180d1870 2021-02-10T11:25:39.526 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:39.508 7f1a177fe700 1 -- 172.21.2.102:0/3130192863 >> v1:172.21.2.102:6789/0 conn(0x7f1a180d1870 legacy=0x7f1a181b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:39.526 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:39.508 7f1a177fe700 1 --2- 172.21.2.102:0/3130192863 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1a04003af0 0x7f1a04005f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:39.526 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:39.508 7f1a177fe700 1 --2- 172.21.2.102:0/3130192863 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1a040067e0 0x7f1a04008d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:39.526 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:39.508 7f1a177fe700 1 --2- 172.21.2.102:0/3130192863 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1a04009520 0x7f1a0400ba40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:39.526 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:39.508 7f1a177fe700 1 -- 172.21.2.102:0/3130192863 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a0400c2f0 con 0x7f1a04009520 2021-02-10T11:25:39.526 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:39.508 7f1a177fe700 1 -- 172.21.2.102:0/3130192863 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a040504b0 con 0x7f1a040067e0 2021-02-10T11:25:39.526 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:39.508 7f1a177fe700 1 -- 172.21.2.102:0/3130192863 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a040506d0 con 0x7f1a04003af0 2021-02-10T11:25:39.526 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:39.508 7f1a1d412700 1 --2- 172.21.2.102:0/3130192863 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1a04009520 0x7f1a0400ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:39.526 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:39.508 7f1a17fff700 1 --2- 172.21.2.102:0/3130192863 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1a040067e0 0x7f1a04008d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:39.527 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:39.508 7f1a1cc11700 1 --2- 172.21.2.102:0/3130192863 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1a04003af0 0x7f1a04005f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:39.527 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:39.508 7f1a1d412700 1 -- 172.21.2.102:0/3130192863 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1a04003af0 msgr2=0x7f1a04005f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:25:39.527 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:39.508 7f1a1d412700 1 --2- 172.21.2.102:0/3130192863 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1a04003af0 0x7f1a04005f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:39.527 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:39.508 7f1a1d412700 1 -- 172.21.2.102:0/3130192863 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1a040067e0 msgr2=0x7f1a04008d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:25:39.527 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:39.508 7f1a1d412700 1 --2- 172.21.2.102:0/3130192863 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1a040067e0 0x7f1a04008d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:39.527 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:39.508 7f1a1d412700 1 -- 172.21.2.102:0/3130192863 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a04050970 con 0x7f1a04009520 2021-02-10T11:25:39.527 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:39.509 7f1a1d412700 1 --2- 172.21.2.102:0/3130192863 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1a04009520 0x7f1a0400ba40 secure :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 rx=0x7f1a100067f0 tx=0x7f1a10009cb0).ready entity=mon.0 client_cookie=fd6ca5afda34caf6 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:39.527 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:39.509 7f1a177fe700 1 -- 172.21.2.102:0/3130192863 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1a10006c50 con 0x7f1a04009520 2021-02-10T11:25:39.527 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:39.509 7f1a177fe700 1 -- 172.21.2.102:0/3130192863 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a10006610 con 0x7f1a04009520 2021-02-10T11:25:39.527 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:39.509 7f1a177fe700 1 -- 172.21.2.102:0/3130192863 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1a1000b5b0 con 0x7f1a04009520 2021-02-10T11:25:39.527 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:39.509 7f1a1edf5700 1 -- 172.21.2.102:0/3130192863 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1a04009520 msgr2=0x7f1a0400ba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:39.527 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:39.509 7f1a1edf5700 1 --2- 172.21.2.102:0/3130192863 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1a04009520 0x7f1a0400ba40 secure :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 rx=0x7f1a100067f0 tx=0x7f1a10009cb0).stop 2021-02-10T11:25:39.527 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:39.509 7f1a1d412700 1 -- 172.21.2.102:0/3130192863 reap_dead start 2021-02-10T11:25:39.528 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:39.510 7f1a1edf5700 1 -- 172.21.2.102:0/3130192863 shutdown_connections 2021-02-10T11:25:39.528 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:39.510 7f1a1edf5700 1 -- 172.21.2.102:0/3130192863 >> 172.21.2.102:0/3130192863 conn(0x7f1a180ce420 msgr2=0x7f1a180d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:25:39.528 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:39.510 7f1a1edf5700 1 -- 172.21.2.102:0/3130192863 shutdown_connections 2021-02-10T11:25:39.528 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:39.510 7f1a1edf5700 1 -- 172.21.2.102:0/3130192863 wait complete. 2021-02-10T11:25:39.528 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:39.524 7f1a1edf5700 1 Processor -- start 2021-02-10T11:25:39.528 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:39.524 7f1a1edf5700 1 -- start start 2021-02-10T11:25:39.528 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:39.525 7f1a1edf5700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1a181a79a0 0x7f1a181a7da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:39.528 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:39.525 7f1a1edf5700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1a180bdb40 0x7f1a180bdf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:39.528 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:39.525 7f1a1edf5700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1a18082ca0 0x7f1a180830a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:39.528 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:39.525 7f1a1edf5700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a1819d990 con 0x7f1a180bdb40 2021-02-10T11:25:39.528 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:39.525 7f1a1edf5700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a1819dbb0 con 0x7f1a18082ca0 2021-02-10T11:25:39.528 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:39.525 7f1a1edf5700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a180836c0 con 0x7f1a181a79a0 2021-02-10T11:25:39.530 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:39.525 7f1a17fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1a180bdb40 0x7f1a180bdf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:39.530 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:39.525 7f1a17fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1a180bdb40 0x7f1a180bdf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:58120/0 (socket says 172.21.2.102:58120) 2021-02-10T11:25:39.530 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:39.525 7f1a17fff700 1 -- 172.21.2.102:0/4135325055 learned_addr learned my addr 172.21.2.102:0/4135325055 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:25:39.530 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:39.525 7f1a17fff700 1 -- 172.21.2.102:0/4135325055 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1a181a79a0 msgr2=0x7f1a181a7da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:39.530 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:39.525 7f1a17fff700 1 --2- 172.21.2.102:0/4135325055 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1a181a79a0 0x7f1a181a7da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:39.530 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:39.525 7f1a17fff700 1 -- 172.21.2.102:0/4135325055 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1a18082ca0 msgr2=0x7f1a180830a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:39.530 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:39.525 7f1a17fff700 1 --2- 172.21.2.102:0/4135325055 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1a18082ca0 0x7f1a180830a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:39.530 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:39.525 7f1a17fff700 1 -- 172.21.2.102:0/4135325055 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a180838e0 con 0x7f1a180bdb40 2021-02-10T11:25:39.530 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:39.526 7f1a17fff700 1 --2- 172.21.2.102:0/4135325055 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1a180bdb40 0x7f1a180bdf40 secure :-1 s=READY pgs=1515 cs=0 l=1 rev1=1 rx=0x7f1a0800b990 tx=0x7f1a0800bd30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:39.530 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:39.526 7f1a157fa700 1 -- 172.21.2.102:0/4135325055 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1a0800c2f0 con 0x7f1a180bdb40 2021-02-10T11:25:39.530 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:39.526 7f1a157fa700 1 -- 172.21.2.102:0/4135325055 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a08006b40 con 0x7f1a180bdb40 2021-02-10T11:25:39.531 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:39.526 7f1a1edf5700 1 -- 172.21.2.102:0/4135325055 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a180838e0 con 0x7f1a180bdb40 2021-02-10T11:25:39.531 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:39.526 7f1a1edf5700 1 -- 172.21.2.102:0/4135325055 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a18084420 con 0x7f1a180bdb40 2021-02-10T11:25:39.531 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:39.526 7f1a157fa700 1 -- 172.21.2.102:0/4135325055 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1a0800c820 con 0x7f1a180bdb40 2021-02-10T11:25:39.531 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:39.526 7f1a157fa700 1 -- 172.21.2.102:0/4135325055 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1a0800a000 con 0x7f1a180bdb40 2021-02-10T11:25:39.531 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:39.527 7f1a157fa700 1 --2- 172.21.2.102:0/4135325055 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f19fc018730 0x7f19fc01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:39.531 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:39.527 7f1a157fa700 1 -- 172.21.2.102:0/4135325055 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f1a080329c0 con 0x7f1a180bdb40 2021-02-10T11:25:39.531 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:39.527 7f1a1cc11700 1 --2- 172.21.2.102:0/4135325055 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f19fc018730 0x7f19fc01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:39.531 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:39.527 7f1a1cc11700 1 --2- 172.21.2.102:0/4135325055 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f19fc018730 0x7f19fc01ac10 crc :-1 s=READY pgs=692 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:39.556 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:39.556 7f1a1edf5700 1 -- 172.21.2.102:0/4135325055 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a18082a70 con 0x7f1a180bdb40 2021-02-10T11:25:39.558 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:39.559 7f1a157fa700 1 -- 172.21.2.102:0/4135325055 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f1a08000cc0 con 0x7f1a180bdb40 2021-02-10T11:25:39.685 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:39.686 7f1a1edf5700 1 -- 172.21.2.102:0/4135325055 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f1a18082a70 con 0x7f1a180bdb40 2021-02-10T11:25:39.686 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:39.686 7f1a157fa700 1 -- 172.21.2.102:0/4135325055 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v66) v1 ==== 92+0+10 (secure 0 0 0) 0x7f1a08006b40 con 0x7f1a180bdb40 2021-02-10T11:25:39.686 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:25:39.688 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:39.688 7f1a1edf5700 1 -- 172.21.2.102:0/4135325055 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f19fc018730 msgr2=0x7f19fc01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:39.688 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:39.688 7f1a1edf5700 1 --2- 172.21.2.102:0/4135325055 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f19fc018730 0x7f19fc01ac10 crc :-1 s=READY pgs=692 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:39.688 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:39.688 7f1a1edf5700 1 -- 172.21.2.102:0/4135325055 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1a180bdb40 msgr2=0x7f1a180bdf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:39.688 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:39.688 7f1a1edf5700 1 --2- 172.21.2.102:0/4135325055 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1a180bdb40 0x7f1a180bdf40 secure :-1 s=READY pgs=1515 cs=0 l=1 rev1=1 rx=0x7f1a0800b990 tx=0x7f1a0800bd30).stop 2021-02-10T11:25:39.688 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:39.688 7f1a1edf5700 1 -- 172.21.2.102:0/4135325055 shutdown_connections 2021-02-10T11:25:39.688 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:39.688 7f1a1edf5700 1 --2- 172.21.2.102:0/4135325055 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f19fc018730 0x7f19fc01ac10 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:39.688 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:39.688 7f1a1edf5700 1 --2- 172.21.2.102:0/4135325055 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1a18082ca0 0x7f1a180830a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:39.688 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:39.688 7f1a1edf5700 1 --2- 172.21.2.102:0/4135325055 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1a180bdb40 0x7f1a180bdf40 unknown :-1 s=CLOSED pgs=1515 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:39.688 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:39.688 7f1a1edf5700 1 --2- 172.21.2.102:0/4135325055 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1a181a79a0 0x7f1a181a7da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:39.688 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:39.688 7f1a1edf5700 1 -- 172.21.2.102:0/4135325055 >> 172.21.2.102:0/4135325055 conn(0x7f1a180bef00 msgr2=0x7f1a1809f200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:25:39.688 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:39.688 7f1a1edf5700 1 -- 172.21.2.102:0/4135325055 shutdown_connections 2021-02-10T11:25:39.688 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:39.688 7f1a1edf5700 1 -- 172.21.2.102:0/4135325055 wait complete. 2021-02-10T11:25:39.696 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-02-10T11:25:40.128 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:40.112 7fb01b7d9700 1 Processor -- start 2021-02-10T11:25:40.128 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:40.112 7fb01b7d9700 1 -- start start 2021-02-10T11:25:40.128 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:40.112 7fb01b7d9700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb0140d1870 0x7fb0141b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:40.128 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:40.112 7fb01b7d9700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb014109a60 con 0x7fb0141b0ae0 2021-02-10T11:25:40.128 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:40.112 7fb01b7d9700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb014088070 con 0x7fb0140cff70 2021-02-10T11:25:40.128 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:40.112 7fb01b7d9700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0140e5ef0 con 0x7fb0140d1870 2021-02-10T11:25:40.128 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:40.112 7fb018df4700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fb0141b0ae0 0x7fb014108940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:50350/0 (socket says 172.21.2.102:50350) 2021-02-10T11:25:40.128 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:40.112 7fb018df4700 1 -- 172.21.2.102:0/2987812750 learned_addr learned my addr 172.21.2.102:0/2987812750 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:25:40.129 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:40.113 7fb0195f5700 1 --2- 172.21.2.102:0/2987812750 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb0140d1870 0x7fb0141b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:40.129 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:40.113 7fb007fff700 1 -- 172.21.2.102:0/2987812750 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1230526662 0 0) 0x7fb0080010b0 con 0x7fb0141b0ae0 2021-02-10T11:25:40.129 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:40.113 7fb007fff700 1 -- 172.21.2.102:0/2987812750 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb000002fe0 con 0x7fb0141b0ae0 2021-02-10T11:25:40.129 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:40.113 7fb007fff700 1 -- 172.21.2.102:0/2987812750 <== mon.1 v1:172.21.2.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 853527429 0 0) 0x7fb00c000f90 con 0x7fb0140cff70 2021-02-10T11:25:40.129 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:40.113 7fb007fff700 1 -- 172.21.2.102:0/2987812750 --> v1:172.21.2.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb000003210 con 0x7fb0140cff70 2021-02-10T11:25:40.129 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:40.113 7fb0195f5700 1 -- 172.21.2.102:0/2987812750 >> v1:172.21.2.113:6789/0 conn(0x7fb0140cff70 legacy=0x7fb014109130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:40.129 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:40.113 7fb0195f5700 1 -- 172.21.2.102:0/2987812750 >> v1:172.21.2.102:6789/0 conn(0x7fb0141b0ae0 legacy=0x7fb014108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:40.129 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:40.113 7fb0195f5700 1 -- 172.21.2.102:0/2987812750 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0140d1c70 con 0x7fb0140d1870 2021-02-10T11:25:40.129 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:40.113 7fb0195f5700 1 --2- 172.21.2.102:0/2987812750 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb0140d1870 0x7fb0141b04c0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7fb010006650 tx=0x7fb010009a80).ready entity=mon.2 client_cookie=21bbd0e85c31c0f6 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:40.129 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:40.113 7fb007fff700 1 -- 172.21.2.102:0/2987812750 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb01000c0c0 con 0x7fb0140d1870 2021-02-10T11:25:40.129 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:40.113 7fb007fff700 1 -- 172.21.2.102:0/2987812750 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb01000c340 con 0x7fb0140d1870 2021-02-10T11:25:40.129 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:40.113 7fb007fff700 1 -- 172.21.2.102:0/2987812750 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb01000c820 con 0x7fb0140d1870 2021-02-10T11:25:40.130 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:40.113 7fb01b7d9700 1 -- 172.21.2.102:0/2987812750 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb0140d1870 msgr2=0x7fb0141b04c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:40.130 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:40.113 7fb01b7d9700 1 --2- 172.21.2.102:0/2987812750 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb0140d1870 0x7fb0141b04c0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7fb010006650 tx=0x7fb010009a80).stop 2021-02-10T11:25:40.130 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:40.114 7fb01b7d9700 1 -- 172.21.2.102:0/2987812750 shutdown_connections 2021-02-10T11:25:40.130 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:40.114 7fb01b7d9700 1 --2- 172.21.2.102:0/2987812750 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb0140d1870 0x7fb0141b04c0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:40.130 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:40.114 7fb01b7d9700 1 -- 172.21.2.102:0/2987812750 >> 172.21.2.102:0/2987812750 conn(0x7fb0140ce420 msgr2=0x7fb0140d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:25:40.130 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:40.114 7fb01b7d9700 1 -- 172.21.2.102:0/2987812750 shutdown_connections 2021-02-10T11:25:40.130 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:40.114 7fb01b7d9700 1 -- 172.21.2.102:0/2987812750 wait complete. 2021-02-10T11:25:40.130 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:40.129 7fb01b7d9700 1 Processor -- start 2021-02-10T11:25:40.131 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:40.129 7fb01b7d9700 1 -- start start 2021-02-10T11:25:40.132 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:40.129 7fb01b7d9700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb0140b1ba0 0x7fb0141a78d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:40.132 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:40.129 7fb01b7d9700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb0141a7ef0 0x7fb0140a6a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:40.132 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:40.129 7fb01b7d9700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb0140a7060 0x7fb0140b2920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:40.132 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:40.129 7fb01b7d9700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb014190c20 con 0x7fb0140a7060 2021-02-10T11:25:40.132 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:40.129 7fb01b7d9700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb014190710 con 0x7fb0141a7ef0 2021-02-10T11:25:40.132 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:40.129 7fb01b7d9700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0140b1fa0 con 0x7fb0140b1ba0 2021-02-10T11:25:40.132 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:40.130 7fb019df6700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb0140a7060 0x7fb0140b2920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:40.132 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:40.130 7fb019df6700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb0140a7060 0x7fb0140b2920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:58134/0 (socket says 172.21.2.102:58134) 2021-02-10T11:25:40.132 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:40.130 7fb019df6700 1 -- 172.21.2.102:0/566108654 learned_addr learned my addr 172.21.2.102:0/566108654 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:25:40.132 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:40.130 7fb019df6700 1 -- 172.21.2.102:0/566108654 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb0140b1ba0 msgr2=0x7fb0141a78d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:40.132 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:40.130 7fb019df6700 1 --2- 172.21.2.102:0/566108654 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb0140b1ba0 0x7fb0141a78d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:40.132 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:40.130 7fb019df6700 1 -- 172.21.2.102:0/566108654 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb0141a7ef0 msgr2=0x7fb0140a6a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:40.133 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:40.130 7fb019df6700 1 --2- 172.21.2.102:0/566108654 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb0141a7ef0 0x7fb0140a6a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:40.133 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:40.130 7fb019df6700 1 -- 172.21.2.102:0/566108654 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0140a7460 con 0x7fb0140a7060 2021-02-10T11:25:40.133 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:40.130 7fb019df6700 1 --2- 172.21.2.102:0/566108654 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb0140a7060 0x7fb0140b2920 secure :-1 s=READY pgs=1517 cs=0 l=1 rev1=1 rx=0x7fb00c0099a0 tx=0x7fb00c00bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:40.133 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:40.130 7fb005ffb700 1 -- 172.21.2.102:0/566108654 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb00c003030 con 0x7fb0140a7060 2021-02-10T11:25:40.133 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:40.130 7fb005ffb700 1 -- 172.21.2.102:0/566108654 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb00c001230 con 0x7fb0140a7060 2021-02-10T11:25:40.133 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:40.130 7fb005ffb700 1 -- 172.21.2.102:0/566108654 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb00c001680 con 0x7fb0140a7060 2021-02-10T11:25:40.133 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:40.130 7fb01b7d9700 1 -- 172.21.2.102:0/566108654 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0140a7460 con 0x7fb0140a7060 2021-02-10T11:25:40.133 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:40.130 7fb01b7d9700 1 -- 172.21.2.102:0/566108654 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0140a7460 con 0x7fb0140a7060 2021-02-10T11:25:40.133 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:40.131 7fb005ffb700 1 -- 172.21.2.102:0/566108654 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fb00c000ed0 con 0x7fb0140a7060 2021-02-10T11:25:40.133 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:40.131 7fb005ffb700 1 --2- 172.21.2.102:0/566108654 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7faff40186c0 0x7faff401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:40.133 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:40.131 7fb005ffb700 1 -- 172.21.2.102:0/566108654 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fb00c032770 con 0x7fb0140a7060 2021-02-10T11:25:40.133 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:40.131 7fb0195f5700 1 --2- 172.21.2.102:0/566108654 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7faff40186c0 0x7faff401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:40.133 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:40.131 7fb0195f5700 1 --2- 172.21.2.102:0/566108654 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7faff40186c0 0x7faff401abc0 crc :-1 s=READY pgs=693 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:40.161 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:40.162 7fb01b7d9700 1 -- 172.21.2.102:0/566108654 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb014190710 con 0x7fb0140a7060 2021-02-10T11:25:40.163 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:40.164 7fb005ffb700 1 -- 172.21.2.102:0/566108654 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fb00c02cf40 con 0x7fb0140a7060 2021-02-10T11:25:40.292 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:40.293 7fb01b7d9700 1 -- 172.21.2.102:0/566108654 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7fb014190710 con 0x7fb0140a7060 2021-02-10T11:25:40.292 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:40.293 7fb005ffb700 1 -- 172.21.2.102:0/566108654 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v66) v1 ==== 104+0+10 (secure 0 0 0) 0x7fb00c02cf40 con 0x7fb0140a7060 2021-02-10T11:25:40.293 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:25:40.294 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:40.295 7fb01b7d9700 1 -- 172.21.2.102:0/566108654 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7faff40186c0 msgr2=0x7faff401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:40.294 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:40.295 7fb01b7d9700 1 --2- 172.21.2.102:0/566108654 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7faff40186c0 0x7faff401abc0 crc :-1 s=READY pgs=693 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:40.294 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:40.295 7fb01b7d9700 1 -- 172.21.2.102:0/566108654 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb0140a7060 msgr2=0x7fb0140b2920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:40.294 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:40.295 7fb01b7d9700 1 --2- 172.21.2.102:0/566108654 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb0140a7060 0x7fb0140b2920 secure :-1 s=READY pgs=1517 cs=0 l=1 rev1=1 rx=0x7fb00c0099a0 tx=0x7fb00c00bcf0).stop 2021-02-10T11:25:40.295 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:40.295 7fb01b7d9700 1 -- 172.21.2.102:0/566108654 shutdown_connections 2021-02-10T11:25:40.295 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:40.295 7fb01b7d9700 1 --2- 172.21.2.102:0/566108654 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb0140a7060 0x7fb0140b2920 unknown :-1 s=CLOSED pgs=1517 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:40.295 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:40.295 7fb01b7d9700 1 --2- 172.21.2.102:0/566108654 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7faff40186c0 0x7faff401abc0 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:40.295 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:40.295 7fb01b7d9700 1 --2- 172.21.2.102:0/566108654 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb0141a7ef0 0x7fb0140a6a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:40.295 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:40.295 7fb01b7d9700 1 --2- 172.21.2.102:0/566108654 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb0140b1ba0 0x7fb0141a78d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:40.296 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:40.295 7fb01b7d9700 1 -- 172.21.2.102:0/566108654 >> 172.21.2.102:0/566108654 conn(0x7fb0140b5dc0 msgr2=0x7fb014095960 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:25:40.296 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:40.296 7fb01b7d9700 1 -- 172.21.2.102:0/566108654 shutdown_connections 2021-02-10T11:25:40.296 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:40.296 7fb01b7d9700 1 -- 172.21.2.102:0/566108654 wait complete. 2021-02-10T11:25:40.303 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-02-10T11:25:40.734 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:40.717 7f69e7546700 1 Processor -- start 2021-02-10T11:25:40.734 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:40.717 7f69e7546700 1 -- start start 2021-02-10T11:25:40.734 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:40.717 7f69e7546700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f69e010aff0 0x7f69e00cf750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:40.734 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:40.717 7f69e7546700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69e00fe1c0 con 0x7f69e00d1870 2021-02-10T11:25:40.734 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:40.717 7f69e7546700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69e01b0cb0 con 0x7f69e00d1c70 2021-02-10T11:25:40.734 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:40.718 7f69e7546700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69e0109a60 con 0x7f69e010aff0 2021-02-10T11:25:40.734 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:40.718 7f69e5362700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f69e00d1870 0x7f69e01b04c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:50364/0 (socket says 172.21.2.102:50364) 2021-02-10T11:25:40.734 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:40.718 7f69e5362700 1 -- 172.21.2.102:0/1033372022 learned_addr learned my addr 172.21.2.102:0/1033372022 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:25:40.735 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:40.718 7f69d3fff700 1 -- 172.21.2.102:0/1033372022 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1096244440 0 0) 0x7f69d4001180 con 0x7f69e00d1870 2021-02-10T11:25:40.735 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:40.718 7f69d3fff700 1 -- 172.21.2.102:0/1033372022 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69cc002fe0 con 0x7f69e00d1870 2021-02-10T11:25:40.735 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:40.718 7f69e5b63700 1 --2- 172.21.2.102:0/1033372022 >> v2:172.21.2.113:3300/0 conn(0x7f69e010aff0 0x7f69e00cf750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:40.735 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:40.718 7f69d3fff700 1 -- 172.21.2.102:0/1033372022 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1450056630 0 0) 0x7f69d4001180 con 0x7f69e00d1870 2021-02-10T11:25:40.735 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:40.718 7f69d3fff700 1 -- 172.21.2.102:0/1033372022 >> v2:172.21.2.113:3300/0 conn(0x7f69e010aff0 msgr2=0x7f69e00cf750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:25:40.735 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:40.718 7f69d3fff700 1 --2- 172.21.2.102:0/1033372022 >> v2:172.21.2.113:3300/0 conn(0x7f69e010aff0 0x7f69e00cf750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:40.735 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:40.718 7f69d3fff700 1 -- 172.21.2.102:0/1033372022 >> v1:172.21.2.113:6789/0 conn(0x7f69e00d1c70 legacy=0x7f69e00cef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:40.735 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:40.718 7f69d3fff700 1 -- 172.21.2.102:0/1033372022 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69e010b3f0 con 0x7f69e00d1870 2021-02-10T11:25:40.735 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:40.718 7f69d3fff700 1 -- 172.21.2.102:0/1033372022 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7f69d4001680 con 0x7f69e00d1870 2021-02-10T11:25:40.735 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:40.718 7f69d3fff700 1 -- 172.21.2.102:0/1033372022 >> v1:172.21.2.102:6789/0 conn(0x7f69e00d1870 legacy=0x7f69e01b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:40.735 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:40.718 7f69d3fff700 1 --2- 172.21.2.102:0/1033372022 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f69cc003aa0 0x7f69cc005f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:40.735 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:40.718 7f69d3fff700 1 --2- 172.21.2.102:0/1033372022 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f69cc006780 0x7f69cc008ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:40.735 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:40.718 7f69d3fff700 1 --2- 172.21.2.102:0/1033372022 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f69cc0095a0 0x7f69cc00baa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:40.736 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:40.718 7f69d3fff700 1 -- 172.21.2.102:0/1033372022 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69cc00c2f0 con 0x7f69cc006780 2021-02-10T11:25:40.736 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:40.718 7f69d3fff700 1 -- 172.21.2.102:0/1033372022 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69cc00c510 con 0x7f69cc003aa0 2021-02-10T11:25:40.736 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:40.718 7f69d3fff700 1 -- 172.21.2.102:0/1033372022 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69cc00c730 con 0x7f69cc0095a0 2021-02-10T11:25:40.736 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:40.719 7f69e4b61700 1 --2- 172.21.2.102:0/1033372022 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f69cc006780 0x7f69cc008ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:40.736 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:40.719 7f69e5b63700 1 --2- 172.21.2.102:0/1033372022 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f69cc0095a0 0x7f69cc00baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:40.736 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:40.719 7f69e5362700 1 --2- 172.21.2.102:0/1033372022 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f69cc003aa0 0x7f69cc005f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:40.736 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:40.719 7f69e4b61700 1 -- 172.21.2.102:0/1033372022 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f69cc0095a0 msgr2=0x7f69cc00baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:25:40.736 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:40.719 7f69e4b61700 1 --2- 172.21.2.102:0/1033372022 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f69cc0095a0 0x7f69cc00baa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:40.736 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:40.719 7f69e4b61700 1 -- 172.21.2.102:0/1033372022 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f69cc003aa0 msgr2=0x7f69cc005f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:25:40.736 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:40.719 7f69e4b61700 1 --2- 172.21.2.102:0/1033372022 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f69cc003aa0 0x7f69cc005f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:40.736 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:40.719 7f69e4b61700 1 -- 172.21.2.102:0/1033372022 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69cc00c9d0 con 0x7f69cc006780 2021-02-10T11:25:40.736 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:40.719 7f69e5b63700 1 -- 172.21.2.102:0/1033372022 reap_dead start 2021-02-10T11:25:40.736 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:40.719 7f69e4b61700 1 --2- 172.21.2.102:0/1033372022 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f69cc006780 0x7f69cc008ca0 secure :-1 s=READY pgs=1519 cs=0 l=1 rev1=1 rx=0x7f69dc04a750 tx=0x7f69dc04da60).ready entity=mon.0 client_cookie=41ed9a91a4c86cc8 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:40.737 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:40.719 7f69d3fff700 1 -- 172.21.2.102:0/1033372022 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f69dc0500c0 con 0x7f69cc006780 2021-02-10T11:25:40.737 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:40.719 7f69d3fff700 1 -- 172.21.2.102:0/1033372022 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69dc050380 con 0x7f69cc006780 2021-02-10T11:25:40.737 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:40.719 7f69d3fff700 1 -- 172.21.2.102:0/1033372022 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f69dc0507d0 con 0x7f69cc006780 2021-02-10T11:25:40.737 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:40.719 7f69e7546700 1 -- 172.21.2.102:0/1033372022 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f69cc006780 msgr2=0x7f69cc008ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:40.737 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:40.719 7f69e7546700 1 --2- 172.21.2.102:0/1033372022 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f69cc006780 0x7f69cc008ca0 secure :-1 s=READY pgs=1519 cs=0 l=1 rev1=1 rx=0x7f69dc04a750 tx=0x7f69dc04da60).stop 2021-02-10T11:25:40.737 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:40.720 7f69e7546700 1 -- 172.21.2.102:0/1033372022 shutdown_connections 2021-02-10T11:25:40.737 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:40.720 7f69e7546700 1 --2- 172.21.2.102:0/1033372022 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f69cc006780 0x7f69cc008ca0 unknown :-1 s=CLOSED pgs=1519 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:40.737 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:40.720 7f69e7546700 1 -- 172.21.2.102:0/1033372022 >> 172.21.2.102:0/1033372022 conn(0x7f69e00ce420 msgr2=0x7f69e00d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:25:40.737 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:40.720 7f69e7546700 1 -- 172.21.2.102:0/1033372022 shutdown_connections 2021-02-10T11:25:40.737 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:40.720 7f69e7546700 1 -- 172.21.2.102:0/1033372022 wait complete. 2021-02-10T11:25:40.739 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:40.735 7f69e7546700 1 Processor -- start 2021-02-10T11:25:40.739 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:40.735 7f69e7546700 1 -- start start 2021-02-10T11:25:40.739 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:40.735 7f69e7546700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f69e0085bc0 0x7f69e00863e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:40.739 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:40.735 7f69e7546700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f69e00954c0 0x7f69e0096300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:40.739 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:40.735 7f69e7546700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f69e00958c0 0x7f69e0096920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:40.739 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:40.735 7f69e7546700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69e0095d80 con 0x7f69e00958c0 2021-02-10T11:25:40.739 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:40.735 7f69e7546700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69e00bfa60 con 0x7f69e00954c0 2021-02-10T11:25:40.740 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:40.735 7f69e7546700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69e0085fc0 con 0x7f69e0085bc0 2021-02-10T11:25:40.740 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:40.735 7f69e5362700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f69e0085bc0 0x7f69e00863e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:40.740 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:40.736 7f69e5362700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f69e0085bc0 0x7f69e00863e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.113:3301/0 says I am v2:172.21.2.102:35458/0 (socket says 172.21.2.102:35458) 2021-02-10T11:25:40.740 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:40.736 7f69e5362700 1 -- 172.21.2.102:0/2541254699 learned_addr learned my addr 172.21.2.102:0/2541254699 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:25:40.740 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:40.736 7f69e5b63700 1 --2- 172.21.2.102:0/2541254699 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f69e00958c0 0x7f69e0096920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:40.740 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:40.736 7f69e4b61700 1 --2- 172.21.2.102:0/2541254699 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f69e00954c0 0x7f69e0096300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:40.740 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:40.736 7f69e5b63700 1 -- 172.21.2.102:0/2541254699 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f69e0085bc0 msgr2=0x7f69e00863e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:40.740 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:40.736 7f69e5b63700 1 --2- 172.21.2.102:0/2541254699 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f69e0085bc0 0x7f69e00863e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:40.741 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:40.736 7f69e5b63700 1 -- 172.21.2.102:0/2541254699 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f69e00954c0 msgr2=0x7f69e0096300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:40.741 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:40.736 7f69e5b63700 1 --2- 172.21.2.102:0/2541254699 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f69e00954c0 0x7f69e0096300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:40.741 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:40.736 7f69e5b63700 1 -- 172.21.2.102:0/2541254699 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69e0086c10 con 0x7f69e00958c0 2021-02-10T11:25:40.741 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:40.736 7f69e5b63700 1 --2- 172.21.2.102:0/2541254699 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f69e00958c0 0x7f69e0096920 secure :-1 s=READY pgs=1520 cs=0 l=1 rev1=1 rx=0x7f69d8009fc0 tx=0x7f69d800bc40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:40.741 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:40.736 7f69d1ffb700 1 -- 172.21.2.102:0/2541254699 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f69d80090c0 con 0x7f69e00958c0 2021-02-10T11:25:40.741 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:40.736 7f69d1ffb700 1 -- 172.21.2.102:0/2541254699 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69d8003030 con 0x7f69e00958c0 2021-02-10T11:25:40.741 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:40.736 7f69e7546700 1 -- 172.21.2.102:0/2541254699 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69e0086c10 con 0x7f69e00958c0 2021-02-10T11:25:40.742 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:40.736 7f69d1ffb700 1 -- 172.21.2.102:0/2541254699 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f69d8001470 con 0x7f69e00958c0 2021-02-10T11:25:40.742 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:40.736 7f69e7546700 1 -- 172.21.2.102:0/2541254699 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69e0097230 con 0x7f69e00958c0 2021-02-10T11:25:40.742 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:40.737 7f69d1ffb700 1 -- 172.21.2.102:0/2541254699 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f69d8000df0 con 0x7f69e00958c0 2021-02-10T11:25:40.742 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:40.737 7f69d1ffb700 1 --2- 172.21.2.102:0/2541254699 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f69c00186d0 0x7f69c001abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:40.742 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:40.737 7f69d1ffb700 1 -- 172.21.2.102:0/2541254699 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f69d8032900 con 0x7f69e00958c0 2021-02-10T11:25:40.742 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:40.737 7f69e5362700 1 --2- 172.21.2.102:0/2541254699 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f69c00186d0 0x7f69c001abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:40.742 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:40.737 7f69e5362700 1 --2- 172.21.2.102:0/2541254699 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f69c00186d0 0x7f69c001abd0 crc :-1 s=READY pgs=694 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:40.766 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:40.767 7f69e7546700 1 -- 172.21.2.102:0/2541254699 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69e00fe1c0 con 0x7f69e00958c0 2021-02-10T11:25:40.769 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:40.770 7f69d1ffb700 1 -- 172.21.2.102:0/2541254699 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f69d8003010 con 0x7f69e00958c0 2021-02-10T11:25:40.896 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:40.896 7f69e7546700 1 -- 172.21.2.102:0/2541254699 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7f69e00fe1c0 con 0x7f69e00958c0 2021-02-10T11:25:40.896 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:40.897 7f69d1ffb700 1 -- 172.21.2.102:0/2541254699 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v66) v1 ==== 100+0+10 (secure 0 0 0) 0x7f69d8003010 con 0x7f69e00958c0 2021-02-10T11:25:40.896 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:25:40.898 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:40.898 7f69e7546700 1 -- 172.21.2.102:0/2541254699 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f69c00186d0 msgr2=0x7f69c001abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:40.898 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:40.898 7f69e7546700 1 --2- 172.21.2.102:0/2541254699 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f69c00186d0 0x7f69c001abd0 crc :-1 s=READY pgs=694 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:40.898 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:40.898 7f69e7546700 1 -- 172.21.2.102:0/2541254699 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f69e00958c0 msgr2=0x7f69e0096920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:40.898 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:40.898 7f69e7546700 1 --2- 172.21.2.102:0/2541254699 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f69e00958c0 0x7f69e0096920 secure :-1 s=READY pgs=1520 cs=0 l=1 rev1=1 rx=0x7f69d8009fc0 tx=0x7f69d800bc40).stop 2021-02-10T11:25:40.898 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:40.899 7f69e7546700 1 -- 172.21.2.102:0/2541254699 shutdown_connections 2021-02-10T11:25:40.899 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:40.899 7f69e7546700 1 --2- 172.21.2.102:0/2541254699 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f69e00958c0 0x7f69e0096920 unknown :-1 s=CLOSED pgs=1520 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:40.899 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:40.899 7f69e7546700 1 --2- 172.21.2.102:0/2541254699 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f69c00186d0 0x7f69c001abd0 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:40.899 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:40.899 7f69e7546700 1 --2- 172.21.2.102:0/2541254699 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f69e00954c0 0x7f69e0096300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:40.899 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:40.899 7f69e7546700 1 --2- 172.21.2.102:0/2541254699 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f69e0085bc0 0x7f69e00863e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:40.899 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:40.899 7f69e7546700 1 -- 172.21.2.102:0/2541254699 >> 172.21.2.102:0/2541254699 conn(0x7f69e00bef00 msgr2=0x7f69e009d1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:25:40.899 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:40.899 7f69e7546700 1 -- 172.21.2.102:0/2541254699 shutdown_connections 2021-02-10T11:25:40.899 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:40.899 7f69e7546700 1 -- 172.21.2.102:0/2541254699 wait complete. 2021-02-10T11:25:40.906 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-02-10T11:25:40.907 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:25:41.333 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:41.317 7f484ff4c700 1 Processor -- start 2021-02-10T11:25:41.333 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:41.317 7f484ff4c700 1 -- start start 2021-02-10T11:25:41.333 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:41.318 7f484ff4c700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f48480d1870 0x7f48481b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:41.333 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:41.318 7f484ff4c700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f48481b0a10 0x7f48480cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:41.333 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:41.318 7f484ff4c700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f48480d1c70 0x7f48480cf950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:41.333 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:41.318 7f484ff4c700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48480fe200 con 0x7f48480d1c70 2021-02-10T11:25:41.333 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:41.318 7f484ff4c700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48480e5f10 con 0x7f48480d1870 2021-02-10T11:25:41.333 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:41.318 7f484ff4c700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48480f0b50 con 0x7f48481b0a10 2021-02-10T11:25:41.333 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:41.318 7f484e569700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f48480d1c70 0x7f48480cf950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:41.333 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:41.318 7f484e569700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f48480d1c70 0x7f48480cf950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:58160/0 (socket says 172.21.2.102:58160) 2021-02-10T11:25:41.334 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:41.318 7f484e569700 1 -- 172.21.2.102:0/3116884865 learned_addr learned my addr 172.21.2.102:0/3116884865 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:25:41.334 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:41.318 7f484e569700 1 -- 172.21.2.102:0/3116884865 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f48481b0a10 msgr2=0x7f48480cf330 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:25:41.334 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:41.318 7f484e569700 1 --2- 172.21.2.102:0/3116884865 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f48481b0a10 0x7f48480cf330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:41.334 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:41.318 7f484e569700 1 -- 172.21.2.102:0/3116884865 >> v2:172.21.2.113:3300/0 conn(0x7f48480d1870 msgr2=0x7f48481b03f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:25:41.334 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:41.318 7f484e569700 1 --2- 172.21.2.102:0/3116884865 >> v2:172.21.2.113:3300/0 conn(0x7f48480d1870 0x7f48481b03f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:41.334 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:41.318 7f484e569700 1 -- 172.21.2.102:0/3116884865 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48480cff70 con 0x7f48480d1c70 2021-02-10T11:25:41.334 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:41.318 7f484e569700 1 --2- 172.21.2.102:0/3116884865 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f48480d1c70 0x7f48480cf950 secure :-1 s=READY pgs=1521 cs=0 l=1 rev1=1 rx=0x7f4840006f00 tx=0x7f4840009f10).ready entity=mon.0 client_cookie=e88ef1855eece1c6 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:41.334 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:41.318 7f484cd66700 1 -- 172.21.2.102:0/3116884865 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f484000b3b0 con 0x7f48480d1c70 2021-02-10T11:25:41.334 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:41.318 7f484cd66700 1 -- 172.21.2.102:0/3116884865 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f484000b630 con 0x7f48480d1c70 2021-02-10T11:25:41.334 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:41.318 7f484cd66700 1 -- 172.21.2.102:0/3116884865 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f484000bb50 con 0x7f48480d1c70 2021-02-10T11:25:41.334 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:41.319 7f484ff4c700 1 -- 172.21.2.102:0/3116884865 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f48480d1c70 msgr2=0x7f48480cf950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:41.334 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:41.319 7f484ff4c700 1 --2- 172.21.2.102:0/3116884865 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f48480d1c70 0x7f48480cf950 secure :-1 s=READY pgs=1521 cs=0 l=1 rev1=1 rx=0x7f4840006f00 tx=0x7f4840009f10).stop 2021-02-10T11:25:41.335 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:41.319 7f484ff4c700 1 -- 172.21.2.102:0/3116884865 shutdown_connections 2021-02-10T11:25:41.335 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:41.319 7f484ff4c700 1 --2- 172.21.2.102:0/3116884865 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f48481b0a10 0x7f48480cf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:41.335 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:41.319 7f484ff4c700 1 --2- 172.21.2.102:0/3116884865 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f48480d1c70 0x7f48480cf950 unknown :-1 s=CLOSED pgs=1521 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:41.335 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:41.319 7f484ff4c700 1 --2- 172.21.2.102:0/3116884865 >> v2:172.21.2.113:3300/0 conn(0x7f48480d1870 0x7f48481b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:41.335 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:41.319 7f484ff4c700 1 -- 172.21.2.102:0/3116884865 >> 172.21.2.102:0/3116884865 conn(0x7f48480ce450 msgr2=0x7f48480d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:25:41.335 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:41.319 7f484ff4c700 1 -- 172.21.2.102:0/3116884865 shutdown_connections 2021-02-10T11:25:41.335 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:41.319 7f484ff4c700 1 -- 172.21.2.102:0/3116884865 wait complete. 2021-02-10T11:25:41.336 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:41.334 7f484ff4c700 1 Processor -- start 2021-02-10T11:25:41.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:41.334 7f484ff4c700 1 -- start start 2021-02-10T11:25:41.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:41.334 7f484ff4c700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f484818e930 0x7f48480aa4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:41.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:41.334 7f484ff4c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f48480aaaf0 0x7f4848093700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:41.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:41.334 7f484ff4c700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f484818ed30 0x7f4848093d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:41.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:41.334 7f484ff4c700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48480944e0 con 0x7f48480aaaf0 2021-02-10T11:25:41.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:41.334 7f484ff4c700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f484808a590 con 0x7f484818ed30 2021-02-10T11:25:41.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:41.334 7f484ff4c700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f484808a770 con 0x7f484818e930 2021-02-10T11:25:41.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:41.334 7f484d567700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f48480aaaf0 0x7f4848093700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:41.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:41.335 7f484d567700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f48480aaaf0 0x7f4848093700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:58164/0 (socket says 172.21.2.102:58164) 2021-02-10T11:25:41.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:41.335 7f484d567700 1 -- 172.21.2.102:0/2423395245 learned_addr learned my addr 172.21.2.102:0/2423395245 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:25:41.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:41.335 7f484d567700 1 -- 172.21.2.102:0/2423395245 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f484818e930 msgr2=0x7f48480aa4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:41.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:41.335 7f484d567700 1 --2- 172.21.2.102:0/2423395245 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f484818e930 0x7f48480aa4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:41.338 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:41.335 7f484d567700 1 -- 172.21.2.102:0/2423395245 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f484818ed30 msgr2=0x7f4848093d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:41.338 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:41.335 7f484d567700 1 --2- 172.21.2.102:0/2423395245 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f484818ed30 0x7f4848093d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:41.338 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:41.335 7f484d567700 1 -- 172.21.2.102:0/2423395245 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f484808a990 con 0x7f48480aaaf0 2021-02-10T11:25:41.338 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:41.335 7f484d567700 1 --2- 172.21.2.102:0/2423395245 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f48480aaaf0 0x7f4848093700 secure :-1 s=READY pgs=1522 cs=0 l=1 rev1=1 rx=0x7f483c00b5c0 tx=0x7f483c00b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:41.338 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:41.335 7f483a7fc700 1 -- 172.21.2.102:0/2423395245 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f483c000c40 con 0x7f48480aaaf0 2021-02-10T11:25:41.338 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:41.335 7f483a7fc700 1 -- 172.21.2.102:0/2423395245 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f483c00bd50 con 0x7f48480aaaf0 2021-02-10T11:25:41.338 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:41.335 7f484ff4c700 1 -- 172.21.2.102:0/2423395245 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f484808a990 con 0x7f48480aaaf0 2021-02-10T11:25:41.338 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:41.335 7f483a7fc700 1 -- 172.21.2.102:0/2423395245 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f483c0054d0 con 0x7f48480aaaf0 2021-02-10T11:25:41.338 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:41.335 7f484ff4c700 1 -- 172.21.2.102:0/2423395245 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f484808a990 con 0x7f48480aaaf0 2021-02-10T11:25:41.338 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:41.336 7f483a7fc700 1 -- 172.21.2.102:0/2423395245 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f483c006030 con 0x7f48480aaaf0 2021-02-10T11:25:41.338 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:41.336 7f483a7fc700 1 --2- 172.21.2.102:0/2423395245 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f48300186c0 0x7f483001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:41.338 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:41.336 7f483a7fc700 1 -- 172.21.2.102:0/2423395245 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f483c032570 con 0x7f48480aaaf0 2021-02-10T11:25:41.339 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:41.336 7f484dd68700 1 --2- 172.21.2.102:0/2423395245 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f48300186c0 0x7f483001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:41.339 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:41.337 7f484dd68700 1 --2- 172.21.2.102:0/2423395245 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f48300186c0 0x7f483001abc0 crc :-1 s=READY pgs=695 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:41.366 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:41.367 7f484ff4c700 1 -- 172.21.2.102:0/2423395245 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f484808a530 con 0x7f48480aaaf0 2021-02-10T11:25:41.369 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:41.370 7f483a7fc700 1 -- 172.21.2.102:0/2423395245 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f483c006b80 con 0x7f48480aaaf0 2021-02-10T11:25:41.498 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:41.499 7f484ff4c700 1 -- 172.21.2.102:0/2423395245 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f484808a530 con 0x7f48480aaaf0 2021-02-10T11:25:41.499 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:41.500 7f483a7fc700 1 -- 172.21.2.102:0/2423395245 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) v1 ==== 94+0+2101 (secure 0 0 0) 0x7f483c02e240 con 0x7f48480aaaf0 2021-02-10T11:25:41.499 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:25:41.500 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":48,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8412,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.102:6835/3414701760","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6834","nonce":3414701760},{"type":"v1","addr":"172.21.2.102:6835","nonce":3414701760}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":44}],"filesystems":[{"mdsmap":{"epoch":48,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:25:31.513947","modified":"2021-02-10 11:25:33.524416","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5813},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5813":{"gid":5813,"name":"b","rank":0,"incarnation":45,"state":"up:active","state_seq":3,"addr":"172.21.2.113:6833/2951398038","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6832","nonce":2951398038},{"type":"v1","addr":"172.21.2.113:6833","nonce":2951398038}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":6}]} 2021-02-10T11:25:41.500 INFO:teuthology.orchestra.run.gibba002.stderr:dumped fsmap epoch 48 2021-02-10T11:25:41.501 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:41.501 7f484ff4c700 1 -- 172.21.2.102:0/2423395245 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f48300186c0 msgr2=0x7f483001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:41.502 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:41.501 7f484ff4c700 1 --2- 172.21.2.102:0/2423395245 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f48300186c0 0x7f483001abc0 crc :-1 s=READY pgs=695 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:41.502 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:41.502 7f484ff4c700 1 -- 172.21.2.102:0/2423395245 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f48480aaaf0 msgr2=0x7f4848093700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:41.502 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:41.502 7f484ff4c700 1 --2- 172.21.2.102:0/2423395245 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f48480aaaf0 0x7f4848093700 secure :-1 s=READY pgs=1522 cs=0 l=1 rev1=1 rx=0x7f483c00b5c0 tx=0x7f483c00b960).stop 2021-02-10T11:25:41.502 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:41.502 7f484ff4c700 1 -- 172.21.2.102:0/2423395245 shutdown_connections 2021-02-10T11:25:41.502 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:41.502 7f484ff4c700 1 --2- 172.21.2.102:0/2423395245 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f48300186c0 0x7f483001abc0 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:41.503 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:41.502 7f484ff4c700 1 --2- 172.21.2.102:0/2423395245 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f484818ed30 0x7f4848093d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:41.503 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:41.502 7f484ff4c700 1 --2- 172.21.2.102:0/2423395245 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f48480aaaf0 0x7f4848093700 unknown :-1 s=CLOSED pgs=1522 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:41.503 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:41.502 7f484ff4c700 1 --2- 172.21.2.102:0/2423395245 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f484818e930 0x7f48480aa4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:41.503 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:41.502 7f484ff4c700 1 -- 172.21.2.102:0/2423395245 >> 172.21.2.102:0/2423395245 conn(0x7f484819f8f0 msgr2=0x7f484818c260 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:25:41.503 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:41.502 7f484ff4c700 1 -- 172.21.2.102:0/2423395245 shutdown_connections 2021-02-10T11:25:41.503 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:41.502 7f484ff4c700 1 -- 172.21.2.102:0/2423395245 wait complete. 2021-02-10T11:25:41.711 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 48, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:25:31.513947', 'modified': '2021-02-10 11:25:33.524416', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5813}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5813': {'gid': 5813, 'name': 'b', 'rank': 0, 'incarnation': 45, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.113:6833/2951398038', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.113:6832', 'nonce': 2951398038}, {'type': 'v1', 'addr': '172.21.2.113:6833', 'nonce': 2951398038}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [13], 'metadata_pool': 12, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:25:41.711 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:25:41.712 DEBUG:teuthology.orchestra.run.gibba013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:25:42.110 INFO:teuthology.orchestra.run.gibba013.stdout:{ 2021-02-10T11:25:42.110 INFO:teuthology.orchestra.run.gibba013.stdout: "cluster_fsid": "68db30ea-dbce-4aef-a6df-f01c6fea0513", 2021-02-10T11:25:42.110 INFO:teuthology.orchestra.run.gibba013.stdout: "whoami": 0, 2021-02-10T11:25:42.111 INFO:teuthology.orchestra.run.gibba013.stdout: "id": 5813, 2021-02-10T11:25:42.111 INFO:teuthology.orchestra.run.gibba013.stdout: "want_state": "up:active", 2021-02-10T11:25:42.111 INFO:teuthology.orchestra.run.gibba013.stdout: "state": "up:active", 2021-02-10T11:25:42.111 INFO:teuthology.orchestra.run.gibba013.stdout: "rank_uptime": 10.594973179, 2021-02-10T11:25:42.111 INFO:teuthology.orchestra.run.gibba013.stdout: "mdsmap_epoch": 48, 2021-02-10T11:25:42.111 INFO:teuthology.orchestra.run.gibba013.stdout: "osdmap_epoch": 66, 2021-02-10T11:25:42.111 INFO:teuthology.orchestra.run.gibba013.stdout: "osdmap_epoch_barrier": 66, 2021-02-10T11:25:42.111 INFO:teuthology.orchestra.run.gibba013.stdout: "uptime": 16.231029475 2021-02-10T11:25:42.111 INFO:teuthology.orchestra.run.gibba013.stdout:} 2021-02-10T11:25:42.114 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "68db30ea-dbce-4aef-a6df-f01c6fea0513", "whoami": 0, "id": 5813, "want_state": "up:active", "state": "up:active", "rank_uptime": 10.594973179, "mdsmap_epoch": 48, "osdmap_epoch": 66, "osdmap_epoch_barrier": 66, "uptime": 16.231029475 } 2021-02-10T11:25:42.114 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2021-02-10T11:25:42.114 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2021-02-10T11:25:42.114 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@gibba002.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2021-02-10T11:25:42.114 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:42.114 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec mkdir -p /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:25:42.450 DEBUG:teuthology.orchestra.run.gibba002:> sudo modprobe fuse 2021-02-10T11:25:42.793 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:25:42.794 DEBUG:teuthology.orchestra.run.gibba002:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:25:43.136 INFO:teuthology.orchestra.run.gibba002.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:25:43.136 INFO:teuthology.orchestra.run.gibba002.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:25:43.137 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:25:43.137 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:43.137 DEBUG:teuthology.orchestra.run.gibba002:> ls /sys/fs/fuse/connections 2021-02-10T11:25:43.472 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2021-02-10T11:25:43.473 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:25:43.475 DEBUG:teuthology.orchestra.run.gibba002:> sudo modprobe fuse 2021-02-10T11:25:43.820 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:25:43.820 DEBUG:teuthology.orchestra.run.gibba002:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:25:43.887 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba002.stderr:ceph-fuse[159204]: starting ceph client 2021-02-10T11:25:43.887 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba002.stderr:2021-02-10 11:25:43.887 7f28e36fcf80 -1 init, newargv = 0x55f2243451a0 newargc=9 2021-02-10T11:25:43.892 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba002.stderr:ceph-fuse[159204]: starting fuse 2021-02-10T11:25:44.162 INFO:teuthology.orchestra.run.gibba002.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:25:44.162 INFO:teuthology.orchestra.run.gibba002.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:25:44.163 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:25:44.163 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:44.163 DEBUG:teuthology.orchestra.run.gibba002:> ls /sys/fs/fuse/connections 2021-02-10T11:25:44.496 INFO:teuthology.orchestra.run.gibba002.stdout:43 2021-02-10T11:25:44.496 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2021-02-10T11:25:44.497 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:44.497 DEBUG:teuthology.orchestra.run.gibba002:> sudo python3 -c ' 2021-02-10T11:25:44.497 DEBUG:teuthology.orchestra.run.gibba002:> import glob 2021-02-10T11:25:44.497 DEBUG:teuthology.orchestra.run.gibba002:> import re 2021-02-10T11:25:44.497 DEBUG:teuthology.orchestra.run.gibba002:> import os 2021-02-10T11:25:44.497 DEBUG:teuthology.orchestra.run.gibba002:> import subprocess 2021-02-10T11:25:44.497 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:25:44.497 DEBUG:teuthology.orchestra.run.gibba002:> def find_socket(client_name): 2021-02-10T11:25:44.497 DEBUG:teuthology.orchestra.run.gibba002:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2021-02-10T11:25:44.497 DEBUG:teuthology.orchestra.run.gibba002:> files = glob.glob(asok_path) 2021-02-10T11:25:44.497 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:25:44.497 DEBUG:teuthology.orchestra.run.gibba002:> # Given a non-glob path, it better be there 2021-02-10T11:25:44.498 DEBUG:teuthology.orchestra.run.gibba002:> if "*" not in asok_path: 2021-02-10T11:25:44.498 DEBUG:teuthology.orchestra.run.gibba002:> assert(len(files) == 1) 2021-02-10T11:25:44.498 DEBUG:teuthology.orchestra.run.gibba002:> return files[0] 2021-02-10T11:25:44.498 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:25:44.498 DEBUG:teuthology.orchestra.run.gibba002:> for f in files: 2021-02-10T11:25:44.498 DEBUG:teuthology.orchestra.run.gibba002:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2021-02-10T11:25:44.498 DEBUG:teuthology.orchestra.run.gibba002:> if os.path.exists("/proc/{0}".format(pid)): 2021-02-10T11:25:44.498 DEBUG:teuthology.orchestra.run.gibba002:> return f 2021-02-10T11:25:44.498 DEBUG:teuthology.orchestra.run.gibba002:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2021-02-10T11:25:44.498 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:25:44.498 DEBUG:teuthology.orchestra.run.gibba002:> print(find_socket("client.0")) 2021-02-10T11:25:44.498 DEBUG:teuthology.orchestra.run.gibba002:> ' 2021-02-10T11:25:44.860 INFO:teuthology.orchestra.run.gibba002.stdout:/var/run/ceph/ceph-client.0.159204.asok 2021-02-10T11:25:44.863 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.159204.asok 2021-02-10T11:25:44.863 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:44.863 DEBUG:teuthology.orchestra.run.gibba002:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.159204.asok status 2021-02-10T11:25:45.255 INFO:teuthology.orchestra.run.gibba002.stdout:{ 2021-02-10T11:25:45.255 INFO:teuthology.orchestra.run.gibba002.stdout: "metadata": { 2021-02-10T11:25:45.255 INFO:teuthology.orchestra.run.gibba002.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:25:45.255 INFO:teuthology.orchestra.run.gibba002.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:25:45.255 INFO:teuthology.orchestra.run.gibba002.stdout: "entity_id": "0", 2021-02-10T11:25:45.255 INFO:teuthology.orchestra.run.gibba002.stdout: "hostname": "gibba002", 2021-02-10T11:25:45.255 INFO:teuthology.orchestra.run.gibba002.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2021-02-10T11:25:45.255 INFO:teuthology.orchestra.run.gibba002.stdout: "pid": "159204", 2021-02-10T11:25:45.255 INFO:teuthology.orchestra.run.gibba002.stdout: "root": "/" 2021-02-10T11:25:45.255 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:25:45.256 INFO:teuthology.orchestra.run.gibba002.stdout: "dentry_count": 0, 2021-02-10T11:25:45.256 INFO:teuthology.orchestra.run.gibba002.stdout: "dentry_pinned_count": 0, 2021-02-10T11:25:45.256 INFO:teuthology.orchestra.run.gibba002.stdout: "id": 8562, 2021-02-10T11:25:45.256 INFO:teuthology.orchestra.run.gibba002.stdout: "inst": { 2021-02-10T11:25:45.256 INFO:teuthology.orchestra.run.gibba002.stdout: "name": { 2021-02-10T11:25:45.256 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "client", 2021-02-10T11:25:45.256 INFO:teuthology.orchestra.run.gibba002.stdout: "num": 8562 2021-02-10T11:25:45.256 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:25:45.256 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": { 2021-02-10T11:25:45.256 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:25:45.256 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:0", 2021-02-10T11:25:45.256 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 460692712 2021-02-10T11:25:45.256 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:25:45.257 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:25:45.257 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": { 2021-02-10T11:25:45.257 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:25:45.257 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:0", 2021-02-10T11:25:45.257 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 460692712 2021-02-10T11:25:45.257 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:25:45.257 INFO:teuthology.orchestra.run.gibba002.stdout: "inst_str": "client.8562 172.21.2.102:0/460692712", 2021-02-10T11:25:45.257 INFO:teuthology.orchestra.run.gibba002.stdout: "addr_str": "172.21.2.102:0/460692712", 2021-02-10T11:25:45.257 INFO:teuthology.orchestra.run.gibba002.stdout: "inode_count": 1, 2021-02-10T11:25:45.257 INFO:teuthology.orchestra.run.gibba002.stdout: "mds_epoch": 48, 2021-02-10T11:25:45.257 INFO:teuthology.orchestra.run.gibba002.stdout: "osd_epoch": 66, 2021-02-10T11:25:45.257 INFO:teuthology.orchestra.run.gibba002.stdout: "osd_epoch_barrier": 0, 2021-02-10T11:25:45.257 INFO:teuthology.orchestra.run.gibba002.stdout: "blacklisted": false 2021-02-10T11:25:45.257 INFO:teuthology.orchestra.run.gibba002.stdout:} 2021-02-10T11:25:45.259 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:45.259 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:25:45.259 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:25:45.593 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:25:45.593 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:25:45.593 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:45.594 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:25:45.941 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:25:46.368 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:46.351 7f43245e3700 1 Processor -- start 2021-02-10T11:25:46.368 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:46.351 7f43245e3700 1 -- start start 2021-02-10T11:25:46.368 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:46.351 7f43245e3700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f431c0d1870 0x7f431c1b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:46.368 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:46.351 7f43245e3700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f431c0f0b10 con 0x7f431c1b0a10 2021-02-10T11:25:46.368 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:46.351 7f43245e3700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f431c0d0310 con 0x7f431c0f0650 2021-02-10T11:25:46.368 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:46.351 7f43245e3700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f431c0d0530 con 0x7f431c0d1870 2021-02-10T11:25:46.369 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:46.351 7f4321bfe700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f431c1b0a10 0x7f431c0cf330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:50408/0 (socket says 172.21.2.102:50408) 2021-02-10T11:25:46.369 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:46.351 7f4321bfe700 1 -- 172.21.2.102:0/1022567119 learned_addr learned my addr 172.21.2.102:0/1022567119 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:25:46.369 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:46.352 7f43223ff700 1 --2- 172.21.2.102:0/1022567119 >> v2:172.21.2.113:3300/0 conn(0x7f431c0d1870 0x7f431c1b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:46.369 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:46.352 7f43213fd700 1 -- 172.21.2.102:0/1022567119 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2420392211 0 0) 0x7f430c001180 con 0x7f431c1b0a10 2021-02-10T11:25:46.369 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:46.352 7f43213fd700 1 -- 172.21.2.102:0/1022567119 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4308002fe0 con 0x7f431c1b0a10 2021-02-10T11:25:46.369 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:46.352 7f43213fd700 1 -- 172.21.2.102:0/1022567119 <== mon.1 v1:172.21.2.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1212611356 0 0) 0x7f4314001400 con 0x7f431c0f0650 2021-02-10T11:25:46.369 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:46.352 7f43213fd700 1 -- 172.21.2.102:0/1022567119 --> v1:172.21.2.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4308003210 con 0x7f431c0f0650 2021-02-10T11:25:46.369 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:46.352 7f43213fd700 1 -- 172.21.2.102:0/1022567119 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2299653942 0 0) 0x7f430c001180 con 0x7f431c1b0a10 2021-02-10T11:25:46.369 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:46.352 7f43213fd700 1 -- 172.21.2.102:0/1022567119 >> v2:172.21.2.113:3300/0 conn(0x7f431c0d1870 msgr2=0x7f431c1b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:25:46.369 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:46.352 7f43213fd700 1 --2- 172.21.2.102:0/1022567119 >> v2:172.21.2.113:3300/0 conn(0x7f431c0d1870 0x7f431c1b03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:46.370 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:46.352 7f43213fd700 1 -- 172.21.2.102:0/1022567119 >> v1:172.21.2.113:6789/0 conn(0x7f431c0f0650 legacy=0x7f431c0cfb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:46.370 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:46.352 7f43213fd700 1 -- 172.21.2.102:0/1022567119 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f431c0d1c70 con 0x7f431c1b0a10 2021-02-10T11:25:46.370 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:46.352 7f43213fd700 1 -- 172.21.2.102:0/1022567119 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7f430c001680 con 0x7f431c1b0a10 2021-02-10T11:25:46.370 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:46.352 7f43213fd700 1 -- 172.21.2.102:0/1022567119 >> v1:172.21.2.102:6789/0 conn(0x7f431c1b0a10 legacy=0x7f431c0cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:46.370 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:46.352 7f43213fd700 1 --2- 172.21.2.102:0/1022567119 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4308003bb0 0x7f4308005fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:46.370 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:46.352 7f43213fd700 1 --2- 172.21.2.102:0/1022567119 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4308006840 0x7f4308008d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:46.370 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:46.352 7f43213fd700 1 --2- 172.21.2.102:0/1022567119 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4308009640 0x7f430800bb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:46.370 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:46.352 7f43213fd700 1 -- 172.21.2.102:0/1022567119 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f430800c3b0 con 0x7f4308003bb0 2021-02-10T11:25:46.370 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:46.352 7f43213fd700 1 -- 172.21.2.102:0/1022567119 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43080505a0 con 0x7f4308006840 2021-02-10T11:25:46.370 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:46.352 7f43213fd700 1 -- 172.21.2.102:0/1022567119 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43080507c0 con 0x7f4308009640 2021-02-10T11:25:46.370 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:46.352 7f43223ff700 1 --2- 172.21.2.102:0/1022567119 >> v2:172.21.2.113:3300/0 conn(0x7f431c0d1870 0x7f431c1b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:25:46.371 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:46.353 7f4321bfe700 1 --2- 172.21.2.102:0/1022567119 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4308006840 0x7f4308008d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:46.371 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:46.353 7f4322c00700 1 --2- 172.21.2.102:0/1022567119 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4308009640 0x7f430800bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:46.371 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:46.353 7f43223ff700 1 --2- 172.21.2.102:0/1022567119 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4308003bb0 0x7f4308005fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:46.371 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:46.353 7f43223ff700 1 -- 172.21.2.102:0/1022567119 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4308009640 msgr2=0x7f430800bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:25:46.371 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:46.353 7f43223ff700 1 --2- 172.21.2.102:0/1022567119 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4308009640 0x7f430800bb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:46.371 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:46.353 7f43223ff700 1 -- 172.21.2.102:0/1022567119 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4308006840 msgr2=0x7f4308008d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:25:46.371 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:46.353 7f43223ff700 1 --2- 172.21.2.102:0/1022567119 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4308006840 0x7f4308008d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:46.371 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:46.353 7f43223ff700 1 -- 172.21.2.102:0/1022567119 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4308050a60 con 0x7f4308003bb0 2021-02-10T11:25:46.371 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:46.353 7f4322c00700 1 -- 172.21.2.102:0/1022567119 reap_dead start 2021-02-10T11:25:46.371 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:46.353 7f43223ff700 1 --2- 172.21.2.102:0/1022567119 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4308003bb0 0x7f4308005fc0 secure :-1 s=READY pgs=1525 cs=0 l=1 rev1=1 rx=0x7f4318000b80 tx=0x7f4318009e80).ready entity=mon.0 client_cookie=98f2c730721dcdf3 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:46.371 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:46.353 7f43213fd700 1 -- 172.21.2.102:0/1022567119 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4318006b40 con 0x7f4308003bb0 2021-02-10T11:25:46.371 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:46.353 7f43213fd700 1 -- 172.21.2.102:0/1022567119 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4318006dc0 con 0x7f4308003bb0 2021-02-10T11:25:46.372 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:46.353 7f43213fd700 1 -- 172.21.2.102:0/1022567119 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f43180072f0 con 0x7f4308003bb0 2021-02-10T11:25:46.372 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:46.353 7f43245e3700 1 -- 172.21.2.102:0/1022567119 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4308003bb0 msgr2=0x7f4308005fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:46.372 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:46.353 7f43245e3700 1 --2- 172.21.2.102:0/1022567119 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4308003bb0 0x7f4308005fc0 secure :-1 s=READY pgs=1525 cs=0 l=1 rev1=1 rx=0x7f4318000b80 tx=0x7f4318009e80).stop 2021-02-10T11:25:46.372 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:46.354 7f43245e3700 1 -- 172.21.2.102:0/1022567119 shutdown_connections 2021-02-10T11:25:46.372 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:46.354 7f43245e3700 1 --2- 172.21.2.102:0/1022567119 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4308003bb0 0x7f4308005fc0 unknown :-1 s=CLOSED pgs=1525 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:46.372 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:46.354 7f43245e3700 1 -- 172.21.2.102:0/1022567119 >> 172.21.2.102:0/1022567119 conn(0x7f431c0ce450 msgr2=0x7f431c0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:25:46.372 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:46.354 7f43245e3700 1 -- 172.21.2.102:0/1022567119 shutdown_connections 2021-02-10T11:25:46.372 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:46.354 7f43245e3700 1 -- 172.21.2.102:0/1022567119 wait complete. 2021-02-10T11:25:46.372 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:46.369 7f43245e3700 1 Processor -- start 2021-02-10T11:25:46.372 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:46.369 7f43245e3700 1 -- start start 2021-02-10T11:25:46.374 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:46.369 7f43245e3700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f431c190ae0 0x7f431c090c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:46.374 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:46.369 7f43245e3700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f431c0912a0 0x7f431c0ecb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:46.374 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:46.369 7f43245e3700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f431c0ed130 0x7f431c0ed530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:46.374 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:46.369 7f43245e3700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f431c08e460 con 0x7f431c190ae0 2021-02-10T11:25:46.374 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:46.369 7f43245e3700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f431c1880e0 con 0x7f431c0ed130 2021-02-10T11:25:46.374 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:46.369 7f43245e3700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f431c1882c0 con 0x7f431c0912a0 2021-02-10T11:25:46.374 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:46.369 7f43223ff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f431c190ae0 0x7f431c090c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:46.375 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:46.369 7f43223ff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f431c190ae0 0x7f431c090c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:58196/0 (socket says 172.21.2.102:58196) 2021-02-10T11:25:46.375 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:46.369 7f43223ff700 1 -- 172.21.2.102:0/228456494 learned_addr learned my addr 172.21.2.102:0/228456494 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:25:46.375 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:46.370 7f43223ff700 1 -- 172.21.2.102:0/228456494 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f431c0912a0 msgr2=0x7f431c0ecb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:46.375 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:46.370 7f43223ff700 1 --2- 172.21.2.102:0/228456494 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f431c0912a0 0x7f431c0ecb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:46.375 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:46.370 7f43223ff700 1 -- 172.21.2.102:0/228456494 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f431c0ed130 msgr2=0x7f431c0ed530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:46.375 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:46.370 7f43223ff700 1 --2- 172.21.2.102:0/228456494 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f431c0ed130 0x7f431c0ed530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:46.375 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:46.370 7f43223ff700 1 -- 172.21.2.102:0/228456494 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f431c1884e0 con 0x7f431c190ae0 2021-02-10T11:25:46.375 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:46.370 7f43223ff700 1 --2- 172.21.2.102:0/228456494 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f431c190ae0 0x7f431c090c80 secure :-1 s=READY pgs=1526 cs=0 l=1 rev1=1 rx=0x7f431800b5b0 tx=0x7f431800b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:46.375 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:46.370 7f4312ffd700 1 -- 172.21.2.102:0/228456494 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f431800bd40 con 0x7f431c190ae0 2021-02-10T11:25:46.375 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:46.370 7f4312ffd700 1 -- 172.21.2.102:0/228456494 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4318002030 con 0x7f431c190ae0 2021-02-10T11:25:46.375 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:46.370 7f43245e3700 1 -- 172.21.2.102:0/228456494 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f431c1884e0 con 0x7f431c190ae0 2021-02-10T11:25:46.375 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:46.370 7f43245e3700 1 -- 172.21.2.102:0/228456494 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f431c188a70 con 0x7f431c190ae0 2021-02-10T11:25:46.375 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:46.370 7f4312ffd700 1 -- 172.21.2.102:0/228456494 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4318002490 con 0x7f431c190ae0 2021-02-10T11:25:46.375 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:46.371 7f4312ffd700 1 -- 172.21.2.102:0/228456494 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4318000c90 con 0x7f431c190ae0 2021-02-10T11:25:46.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:46.371 7f4312ffd700 1 --2- 172.21.2.102:0/228456494 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4304018730 0x7f430401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:46.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:46.371 7f4312ffd700 1 -- 172.21.2.102:0/228456494 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f43180327e0 con 0x7f431c190ae0 2021-02-10T11:25:46.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:46.371 7f4321bfe700 1 --2- 172.21.2.102:0/228456494 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4304018730 0x7f430401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:46.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:46.371 7f4321bfe700 1 --2- 172.21.2.102:0/228456494 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4304018730 0x7f430401ac10 crc :-1 s=READY pgs=696 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:46.400 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:46.401 7f43245e3700 1 -- 172.21.2.102:0/228456494 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f431c0ebc00 con 0x7f431c190ae0 2021-02-10T11:25:46.402 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:46.403 7f4312ffd700 1 -- 172.21.2.102:0/228456494 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f4318000940 con 0x7f431c190ae0 2021-02-10T11:25:46.528 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:46.529 7f43245e3700 1 -- 172.21.2.102:0/228456494 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f431c0f0b10 con 0x7f431c190ae0 2021-02-10T11:25:46.529 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:46.529 7f4312ffd700 1 -- 172.21.2.102:0/228456494 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v66) v1 ==== 74+0+13996 (secure 0 0 0) 0x7f4318000c90 con 0x7f431c190ae0 2021-02-10T11:25:46.529 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:25:46.529 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":66,"fsid":"68db30ea-dbce-4aef-a6df-f01c6fea0513","created":"2021-02-10 11:03:06.331557","modified":"2021-02-10 11:25:32.516321","last_up_change":"2021-02-10 11:03:32.690562","last_in_change":"2021-02-10 11:03:32.690562","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:03:37.322519","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:25:29.586765","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":13,"pool_name":"cephfs_data","create_time":"2021-02-10 11:25:30.558626","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"be691b4d-a039-408a-af80-9d53f02b6f1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6818","nonce":36088},{"type":"v1","addr":"172.21.2.102:6820","nonce":36088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6822","nonce":36088},{"type":"v1","addr":"172.21.2.102:6824","nonce":36088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6830","nonce":36088},{"type":"v1","addr":"172.21.2.102:6832","nonce":36088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6826","nonce":36088},{"type":"v1","addr":"172.21.2.102:6828","nonce":36088}]},"public_addr":"172.21.2.102:6820/36088","cluster_addr":"172.21.2.102:6824/36088","heartbeat_back_addr":"172.21.2.102:6832/36088","heartbeat_front_addr":"172.21.2.102:6828/36088","state":["exists","up"]},{"osd":1,"uuid":"64e3ccc8-9c69-4d7c-865e-af8e83822635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6819","nonce":36087},{"type":"v1","addr":"172.21.2.102:6821","nonce":36087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6823","nonce":36087},{"type":"v1","addr":"172.21.2.102:6825","nonce":36087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6831","nonce":36087},{"type":"v1","addr":"172.21.2.102:6833","nonce":36087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6827","nonce":36087},{"type":"v1","addr":"172.21.2.102:6829","nonce":36087}]},"public_addr":"172.21.2.102:6821/36087","cluster_addr":"172.21.2.102:6825/36087","heartbeat_back_addr":"172.21.2.102:6833/36087","heartbeat_front_addr":"172.21.2.102:6829/36087","state":["exists","up"]},{"osd":2,"uuid":"153700af-747f-481f-bdf4-d6072d3a842f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6810","nonce":36089},{"type":"v1","addr":"172.21.2.102:6811","nonce":36089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6812","nonce":36089},{"type":"v1","addr":"172.21.2.102:6813","nonce":36089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6816","nonce":36089},{"type":"v1","addr":"172.21.2.102:6817","nonce":36089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6814","nonce":36089},{"type":"v1","addr":"172.21.2.102:6815","nonce":36089}]},"public_addr":"172.21.2.102:6811/36089","cluster_addr":"172.21.2.102:6813/36089","heartbeat_back_addr":"172.21.2.102:6817/36089","heartbeat_front_addr":"172.21.2.102:6815/36089","state":["exists","up"]},{"osd":3,"uuid":"cbc678f4-83a9-45e6-8830-2f7c232b1ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6802","nonce":36090},{"type":"v1","addr":"172.21.2.102:6803","nonce":36090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6804","nonce":36090},{"type":"v1","addr":"172.21.2.102:6805","nonce":36090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6808","nonce":36090},{"type":"v1","addr":"172.21.2.102:6809","nonce":36090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6806","nonce":36090},{"type":"v1","addr":"172.21.2.102:6807","nonce":36090}]},"public_addr":"172.21.2.102:6803/36090","cluster_addr":"172.21.2.102:6805/36090","heartbeat_back_addr":"172.21.2.102:6809/36090","heartbeat_front_addr":"172.21.2.102:6807/36090","state":["exists","up"]},{"osd":4,"uuid":"c01e4008-b258-4f10-afd2-9e7f37214351","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6808","nonce":36368},{"type":"v1","addr":"172.21.2.113:6811","nonce":36368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6814","nonce":36368},{"type":"v1","addr":"172.21.2.113:6818","nonce":36368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6826","nonce":36368},{"type":"v1","addr":"172.21.2.113:6828","nonce":36368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6820","nonce":36368},{"type":"v1","addr":"172.21.2.113:6824","nonce":36368}]},"public_addr":"172.21.2.113:6811/36368","cluster_addr":"172.21.2.113:6818/36368","heartbeat_back_addr":"172.21.2.113:6828/36368","heartbeat_front_addr":"172.21.2.113:6824/36368","state":["exists","up"]},{"osd":5,"uuid":"551f6007-daf8-4a98-8ee0-c2b16bcccb5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6802","nonce":36366},{"type":"v1","addr":"172.21.2.113:6805","nonce":36366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6807","nonce":36366},{"type":"v1","addr":"172.21.2.113:6810","nonce":36366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6819","nonce":36366},{"type":"v1","addr":"172.21.2.113:6822","nonce":36366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6812","nonce":36366},{"type":"v1","addr":"172.21.2.113:6815","nonce":36366}]},"public_addr":"172.21.2.113:6805/36366","cluster_addr":"172.21.2.113:6810/36366","heartbeat_back_addr":"172.21.2.113:6822/36366","heartbeat_front_addr":"172.21.2.113:6815/36366","state":["exists","up"]},{"osd":6,"uuid":"bf67c040-0e50-48be-b102-c3943bc9ebef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6800","nonce":36367},{"type":"v1","addr":"172.21.2.113:6801","nonce":36367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6803","nonce":36367},{"type":"v1","addr":"172.21.2.113:6804","nonce":36367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6813","nonce":36367},{"type":"v1","addr":"172.21.2.113:6816","nonce":36367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6806","nonce":36367},{"type":"v1","addr":"172.21.2.113:6809","nonce":36367}]},"public_addr":"172.21.2.113:6801/36367","cluster_addr":"172.21.2.113:6804/36367","heartbeat_back_addr":"172.21.2.113:6816/36367","heartbeat_front_addr":"172.21.2.113:6809/36367","state":["exists","up"]},{"osd":7,"uuid":"f5461f80-018d-428f-be89-2eb2bcbe7bb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6817","nonce":36365},{"type":"v1","addr":"172.21.2.113:6821","nonce":36365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6823","nonce":36365},{"type":"v1","addr":"172.21.2.113:6825","nonce":36365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6830","nonce":36365},{"type":"v1","addr":"172.21.2.113:6831","nonce":36365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6827","nonce":36365},{"type":"v1","addr":"172.21.2.113:6829","nonce":36365}]},"public_addr":"172.21.2.113:6821/36365","cluster_addr":"172.21.2.113:6825/36365","heartbeat_back_addr":"172.21.2.113:6831/36365","heartbeat_front_addr":"172.21.2.113:6829/36365","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:25:46.531 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:46.531 7f43245e3700 1 -- 172.21.2.102:0/228456494 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4304018730 msgr2=0x7f430401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:46.531 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:46.531 7f43245e3700 1 --2- 172.21.2.102:0/228456494 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4304018730 0x7f430401ac10 crc :-1 s=READY pgs=696 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:46.531 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:46.531 7f43245e3700 1 -- 172.21.2.102:0/228456494 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f431c190ae0 msgr2=0x7f431c090c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:46.531 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:46.531 7f43245e3700 1 --2- 172.21.2.102:0/228456494 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f431c190ae0 0x7f431c090c80 secure :-1 s=READY pgs=1526 cs=0 l=1 rev1=1 rx=0x7f431800b5b0 tx=0x7f431800b950).stop 2021-02-10T11:25:46.531 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:46.531 7f43245e3700 1 -- 172.21.2.102:0/228456494 shutdown_connections 2021-02-10T11:25:46.531 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:46.531 7f43245e3700 1 --2- 172.21.2.102:0/228456494 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4304018730 0x7f430401ac10 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:46.532 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:46.531 7f43245e3700 1 --2- 172.21.2.102:0/228456494 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f431c0ed130 0x7f431c0ed530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:46.532 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:46.531 7f43245e3700 1 --2- 172.21.2.102:0/228456494 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f431c0912a0 0x7f431c0ecb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:46.532 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:46.531 7f43245e3700 1 --2- 172.21.2.102:0/228456494 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f431c190ae0 0x7f431c090c80 unknown :-1 s=CLOSED pgs=1526 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:46.532 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:46.531 7f43245e3700 1 -- 172.21.2.102:0/228456494 >> 172.21.2.102:0/228456494 conn(0x7f431c0b7e70 msgr2=0x7f431c0c3850 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:25:46.532 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:46.531 7f43245e3700 1 -- 172.21.2.102:0/228456494 shutdown_connections 2021-02-10T11:25:46.532 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:46.531 7f43245e3700 1 -- 172.21.2.102:0/228456494 wait complete. 2021-02-10T11:25:46.540 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:03:37.322519', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 12, 'pool_name': 'cephfs_metadata', 'create_time': '2021-02-10 11:25:29.586765', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '65', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 13, 'pool_name': 'cephfs_data', 'create_time': '2021-02-10 11:25:30.558626', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '66', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-02-10T11:25:46.540 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:25:46.972 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:46.956 7fa979f9b700 1 Processor -- start 2021-02-10T11:25:46.972 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:46.956 7fa979f9b700 1 -- start start 2021-02-10T11:25:46.972 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:46.956 7fa979f9b700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fa9740d1860 0x7fa9741b04b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:46.972 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:46.956 7fa979f9b700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa9741b0ad0 0x7fa974108930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:46.972 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:46.956 7fa979f9b700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9740fe1b0 con 0x7fa9740d1c60 2021-02-10T11:25:46.972 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:46.956 7fa979f9b700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa974109a50 con 0x7fa9740d1860 2021-02-10T11:25:46.972 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:46.956 7fa979f9b700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa974109740 con 0x7fa9741b0ad0 2021-02-10T11:25:46.972 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:46.957 7fa9737fe700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fa9740d1860 0x7fa9741b04b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:46.973 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:46.957 7fa9737fe700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fa9740d1860 0x7fa9741b04b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:58200/0 (socket says 172.21.2.102:58200) 2021-02-10T11:25:46.973 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:46.957 7fa9737fe700 1 -- 172.21.2.102:0/1539558268 learned_addr learned my addr 172.21.2.102:0/1539558268 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:25:46.973 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:46.957 7fa973fff700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fa9740d1c60 0x7fa974108f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:50424/0 (socket says 172.21.2.102:50424) 2021-02-10T11:25:46.973 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:46.957 7fa9737fe700 1 -- 172.21.2.102:0/1539558268 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa9741b0ad0 msgr2=0x7fa974108930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:25:46.973 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:46.957 7fa9737fe700 1 --2- 172.21.2.102:0/1539558268 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa9741b0ad0 0x7fa974108930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:46.973 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:46.957 7fa9737fe700 1 -- 172.21.2.102:0/1539558268 >> v1:172.21.2.102:6789/0 conn(0x7fa9740d1c60 legacy=0x7fa974108f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:46.973 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:46.957 7fa9737fe700 1 -- 172.21.2.102:0/1539558268 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa974088070 con 0x7fa9740d1860 2021-02-10T11:25:46.973 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:46.957 7fa9727fc700 1 -- 172.21.2.102:0/1539558268 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2690249930 0 0) 0x7fa96c000f90 con 0x7fa9740d1c60 2021-02-10T11:25:46.973 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:46.957 7fa9737fe700 1 --2- 172.21.2.102:0/1539558268 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa9740d1860 0x7fa9741b04b0 secure :-1 s=READY pgs=1528 cs=0 l=1 rev1=1 rx=0x7fa964006700 tx=0x7fa964009b80).ready entity=mon.0 client_cookie=94fa48e723b4b4 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:46.973 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:46.958 7fa9727fc700 1 -- 172.21.2.102:0/1539558268 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa96400c0c0 con 0x7fa9740d1860 2021-02-10T11:25:46.973 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:46.958 7fa9727fc700 1 -- 172.21.2.102:0/1539558268 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa96400c440 con 0x7fa9740d1860 2021-02-10T11:25:46.973 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:46.958 7fa9727fc700 1 -- 172.21.2.102:0/1539558268 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa96400c960 con 0x7fa9740d1860 2021-02-10T11:25:46.973 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:46.958 7fa979f9b700 1 -- 172.21.2.102:0/1539558268 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa9740d1860 msgr2=0x7fa9741b04b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:46.974 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:46.958 7fa979f9b700 1 --2- 172.21.2.102:0/1539558268 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa9740d1860 0x7fa9741b04b0 secure :-1 s=READY pgs=1528 cs=0 l=1 rev1=1 rx=0x7fa964006700 tx=0x7fa964009b80).stop 2021-02-10T11:25:46.974 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:46.958 7fa979f9b700 1 -- 172.21.2.102:0/1539558268 shutdown_connections 2021-02-10T11:25:46.974 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:46.958 7fa979f9b700 1 --2- 172.21.2.102:0/1539558268 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa9741b0ad0 0x7fa974108930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:46.974 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:46.958 7fa979f9b700 1 --2- 172.21.2.102:0/1539558268 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa9740d1860 0x7fa9741b04b0 unknown :-1 s=CLOSED pgs=1528 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:46.974 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:46.958 7fa979f9b700 1 -- 172.21.2.102:0/1539558268 >> 172.21.2.102:0/1539558268 conn(0x7fa9740ce410 msgr2=0x7fa9740d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:25:46.974 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:46.958 7fa979f9b700 1 -- 172.21.2.102:0/1539558268 shutdown_connections 2021-02-10T11:25:46.974 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:46.958 7fa979f9b700 1 -- 172.21.2.102:0/1539558268 wait complete. 2021-02-10T11:25:46.975 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:46.973 7fa979f9b700 1 Processor -- start 2021-02-10T11:25:46.975 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:46.973 7fa979f9b700 1 -- start start 2021-02-10T11:25:46.976 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:46.974 7fa979f9b700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa9741a05e0 0x7fa97418b970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:46.976 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:46.974 7fa979f9b700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa9740b8780 0x7fa9740bfa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:46.976 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:46.974 7fa979f9b700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa9740c0030 0x7fa974189e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:46.976 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:46.974 7fa979f9b700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9740b82a0 con 0x7fa9740c0030 2021-02-10T11:25:46.976 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:46.974 7fa979f9b700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9740c0430 con 0x7fa9741a05e0 2021-02-10T11:25:46.976 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:46.974 7fa979f9b700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9740c0650 con 0x7fa9740b8780 2021-02-10T11:25:46.976 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:46.974 7fa973fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa9740c0030 0x7fa974189e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:46.976 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:46.974 7fa973fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa9740c0030 0x7fa974189e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:58208/0 (socket says 172.21.2.102:58208) 2021-02-10T11:25:46.976 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:46.974 7fa973fff700 1 -- 172.21.2.102:0/659917700 learned_addr learned my addr 172.21.2.102:0/659917700 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:25:46.976 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:46.974 7fa973fff700 1 -- 172.21.2.102:0/659917700 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa9740b8780 msgr2=0x7fa9740bfa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:46.976 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:46.974 7fa973fff700 1 --2- 172.21.2.102:0/659917700 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa9740b8780 0x7fa9740bfa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:46.976 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:46.974 7fa973fff700 1 -- 172.21.2.102:0/659917700 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa9741a05e0 msgr2=0x7fa97418b970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:46.977 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:46.974 7fa973fff700 1 --2- 172.21.2.102:0/659917700 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa9741a05e0 0x7fa97418b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:46.977 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:46.974 7fa973fff700 1 -- 172.21.2.102:0/659917700 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa97418a490 con 0x7fa9740c0030 2021-02-10T11:25:46.977 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:46.974 7fa973fff700 1 --2- 172.21.2.102:0/659917700 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa9740c0030 0x7fa974189e70 secure :-1 s=READY pgs=1529 cs=0 l=1 rev1=1 rx=0x7fa96c00a8f0 tx=0x7fa96c00ac90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:46.977 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:46.975 7fa95bfff700 1 -- 172.21.2.102:0/659917700 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa96c007d50 con 0x7fa9740c0030 2021-02-10T11:25:46.977 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:46.975 7fa95bfff700 1 -- 172.21.2.102:0/659917700 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa96c000fb0 con 0x7fa9740c0030 2021-02-10T11:25:46.977 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:46.975 7fa95bfff700 1 -- 172.21.2.102:0/659917700 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa96c007380 con 0x7fa9740c0030 2021-02-10T11:25:46.977 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:46.975 7fa979f9b700 1 -- 172.21.2.102:0/659917700 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa97418a490 con 0x7fa9740c0030 2021-02-10T11:25:46.977 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:46.975 7fa979f9b700 1 -- 172.21.2.102:0/659917700 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa97418aa20 con 0x7fa9740c0030 2021-02-10T11:25:46.977 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:46.975 7fa95bfff700 1 -- 172.21.2.102:0/659917700 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa96c004720 con 0x7fa9740c0030 2021-02-10T11:25:46.977 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:46.975 7fa95bfff700 1 --2- 172.21.2.102:0/659917700 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa9540186c0 0x7fa95401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:46.977 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:46.975 7fa95bfff700 1 -- 172.21.2.102:0/659917700 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fa96c032870 con 0x7fa9740c0030 2021-02-10T11:25:46.977 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:46.976 7fa9737fe700 1 --2- 172.21.2.102:0/659917700 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa9540186c0 0x7fa95401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:46.977 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:46.976 7fa9737fe700 1 --2- 172.21.2.102:0/659917700 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa9540186c0 0x7fa95401abc0 crc :-1 s=READY pgs=697 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:47.005 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:47.006 7fa979f9b700 1 -- 172.21.2.102:0/659917700 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa974082a20 con 0x7fa9740c0030 2021-02-10T11:25:47.008 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:47.009 7fa95bfff700 1 -- 172.21.2.102:0/659917700 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fa96c001270 con 0x7fa9740c0030 2021-02-10T11:25:47.138 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:47.139 7fa979f9b700 1 -- 172.21.2.102:0/659917700 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7fa974082a20 con 0x7fa9740c0030 2021-02-10T11:25:47.138 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:47.139 7fa95bfff700 1 -- 172.21.2.102:0/659917700 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v66) v1 ==== 92+0+10 (secure 0 0 0) 0x7fa96c005ef0 con 0x7fa9740c0030 2021-02-10T11:25:47.139 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:25:47.140 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:47.141 7fa979f9b700 1 -- 172.21.2.102:0/659917700 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa9540186c0 msgr2=0x7fa95401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:47.140 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:47.141 7fa979f9b700 1 --2- 172.21.2.102:0/659917700 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa9540186c0 0x7fa95401abc0 crc :-1 s=READY pgs=697 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:47.140 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:47.141 7fa979f9b700 1 -- 172.21.2.102:0/659917700 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa9740c0030 msgr2=0x7fa974189e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:47.141 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:47.141 7fa979f9b700 1 --2- 172.21.2.102:0/659917700 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa9740c0030 0x7fa974189e70 secure :-1 s=READY pgs=1529 cs=0 l=1 rev1=1 rx=0x7fa96c00a8f0 tx=0x7fa96c00ac90).stop 2021-02-10T11:25:47.141 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:47.141 7fa979f9b700 1 -- 172.21.2.102:0/659917700 shutdown_connections 2021-02-10T11:25:47.141 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:47.141 7fa979f9b700 1 --2- 172.21.2.102:0/659917700 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa9740c0030 0x7fa974189e70 unknown :-1 s=CLOSED pgs=1529 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:47.141 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:47.141 7fa979f9b700 1 --2- 172.21.2.102:0/659917700 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa9740b8780 0x7fa9740bfa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:47.141 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:47.141 7fa979f9b700 1 --2- 172.21.2.102:0/659917700 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa9540186c0 0x7fa95401abc0 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:47.141 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:47.141 7fa979f9b700 1 --2- 172.21.2.102:0/659917700 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa9741a05e0 0x7fa97418b970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:47.141 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:47.141 7fa979f9b700 1 -- 172.21.2.102:0/659917700 >> 172.21.2.102:0/659917700 conn(0x7fa9740a6a80 msgr2=0x7fa9740ae110 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:25:47.141 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:47.141 7fa979f9b700 1 -- 172.21.2.102:0/659917700 shutdown_connections 2021-02-10T11:25:47.141 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:47.141 7fa979f9b700 1 -- 172.21.2.102:0/659917700 wait complete. 2021-02-10T11:25:47.149 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-02-10T11:25:47.580 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:47.562 7f8d9d4ee700 1 Processor -- start 2021-02-10T11:25:47.580 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:47.562 7f8d9d4ee700 1 -- start start 2021-02-10T11:25:47.581 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:47.563 7f8d9d4ee700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f8d980d1870 0x7f8d981b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:47.581 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:47.563 7f8d9d4ee700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8d980cff70 0x7f8d98109130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:47.581 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:47.563 7f8d9d4ee700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d98109a60 con 0x7f8d981b0ae0 2021-02-10T11:25:47.581 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:47.563 7f8d9d4ee700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d98109750 con 0x7f8d980d1870 2021-02-10T11:25:47.581 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:47.563 7f8d9d4ee700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d98088070 con 0x7f8d980cff70 2021-02-10T11:25:47.581 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:47.563 7f8d96abe700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f8d981b0ae0 0x7f8d98108940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:50436/0 (socket says 172.21.2.102:50436) 2021-02-10T11:25:47.582 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:47.563 7f8d96abe700 1 -- 172.21.2.102:0/2932772277 learned_addr learned my addr 172.21.2.102:0/2932772277 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:25:47.582 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:47.563 7f8d97ac0700 1 --2- 172.21.2.102:0/2932772277 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8d980cff70 0x7f8d98109130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:47.582 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:47.563 7f8d962bd700 1 -- 172.21.2.102:0/2932772277 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 425851526 0 0) 0x7f8d8c0010b0 con 0x7f8d981b0ae0 2021-02-10T11:25:47.582 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:47.563 7f8d972bf700 1 --2- 172.21.2.102:0/2932772277 >> v2:172.21.2.113:3300/0 conn(0x7f8d980d1870 0x7f8d981b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:47.582 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:47.563 7f8d962bd700 1 -- 172.21.2.102:0/2932772277 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d80002fe0 con 0x7f8d981b0ae0 2021-02-10T11:25:47.582 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:47.564 7f8d962bd700 1 -- 172.21.2.102:0/2932772277 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1886567013 0 0) 0x7f8d8c0010b0 con 0x7f8d981b0ae0 2021-02-10T11:25:47.583 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:47.564 7f8d962bd700 1 -- 172.21.2.102:0/2932772277 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8d980cff70 msgr2=0x7f8d98109130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:25:47.583 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:47.564 7f8d962bd700 1 --2- 172.21.2.102:0/2932772277 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8d980cff70 0x7f8d98109130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:47.583 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:47.564 7f8d962bd700 1 -- 172.21.2.102:0/2932772277 >> v2:172.21.2.113:3300/0 conn(0x7f8d980d1870 msgr2=0x7f8d981b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:25:47.583 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:47.564 7f8d962bd700 1 --2- 172.21.2.102:0/2932772277 >> v2:172.21.2.113:3300/0 conn(0x7f8d980d1870 0x7f8d981b04c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:47.583 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:47.564 7f8d962bd700 1 -- 172.21.2.102:0/2932772277 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d980d1c70 con 0x7f8d981b0ae0 2021-02-10T11:25:47.583 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:47.564 7f8d962bd700 1 -- 172.21.2.102:0/2932772277 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7f8d8c0015b0 con 0x7f8d981b0ae0 2021-02-10T11:25:47.583 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:47.564 7f8d962bd700 1 -- 172.21.2.102:0/2932772277 >> v1:172.21.2.102:6789/0 conn(0x7f8d981b0ae0 legacy=0x7f8d98108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:47.583 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:47.564 7f8d962bd700 1 --2- 172.21.2.102:0/2932772277 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8d80003a70 0x7f8d80005f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:47.583 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:47.564 7f8d962bd700 1 --2- 172.21.2.102:0/2932772277 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8d8004a770 0x7f8d8004cc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:47.583 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:47.564 7f8d962bd700 1 --2- 172.21.2.102:0/2932772277 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8d8004d510 0x7f8d8004fa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:47.584 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:47.564 7f8d962bd700 1 -- 172.21.2.102:0/2932772277 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d800502e0 con 0x7f8d8004d510 2021-02-10T11:25:47.584 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:47.564 7f8d962bd700 1 -- 172.21.2.102:0/2932772277 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d80050500 con 0x7f8d8004a770 2021-02-10T11:25:47.584 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:47.564 7f8d962bd700 1 -- 172.21.2.102:0/2932772277 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d80050720 con 0x7f8d80003a70 2021-02-10T11:25:47.584 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:47.564 7f8d97ac0700 1 --2- 172.21.2.102:0/2932772277 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8d8004d510 0x7f8d8004fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:47.584 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:47.564 7f8d972bf700 1 --2- 172.21.2.102:0/2932772277 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8d80003a70 0x7f8d80005f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:47.584 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:47.564 7f8d97ac0700 1 -- 172.21.2.102:0/2932772277 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8d80003a70 msgr2=0x7f8d80005f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:25:47.584 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:47.564 7f8d96abe700 1 --2- 172.21.2.102:0/2932772277 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8d8004a770 0x7f8d8004cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:47.584 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:47.564 7f8d97ac0700 1 --2- 172.21.2.102:0/2932772277 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8d80003a70 0x7f8d80005f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:47.584 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:47.564 7f8d97ac0700 1 -- 172.21.2.102:0/2932772277 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8d8004a770 msgr2=0x7f8d8004cc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:25:47.585 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:47.564 7f8d97ac0700 1 --2- 172.21.2.102:0/2932772277 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8d8004a770 0x7f8d8004cc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:47.585 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:47.564 7f8d97ac0700 1 -- 172.21.2.102:0/2932772277 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d800509c0 con 0x7f8d8004d510 2021-02-10T11:25:47.585 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:47.564 7f8d96abe700 1 --2- 172.21.2.102:0/2932772277 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8d8004a770 0x7f8d8004cc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:25:47.585 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:47.565 7f8d97ac0700 1 --2- 172.21.2.102:0/2932772277 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8d8004d510 0x7f8d8004fa30 secure :-1 s=READY pgs=1531 cs=0 l=1 rev1=1 rx=0x7f8d90009a40 tx=0x7f8d90009de0).ready entity=mon.0 client_cookie=4f59a40e4ede2faf server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:47.585 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:47.565 7f8d962bd700 1 -- 172.21.2.102:0/2932772277 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8d90006a90 con 0x7f8d8004d510 2021-02-10T11:25:47.585 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:47.565 7f8d962bd700 1 -- 172.21.2.102:0/2932772277 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d90003030 con 0x7f8d8004d510 2021-02-10T11:25:47.585 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:47.565 7f8d962bd700 1 -- 172.21.2.102:0/2932772277 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8d9000b790 con 0x7f8d8004d510 2021-02-10T11:25:47.585 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:47.565 7f8d9d4ee700 1 -- 172.21.2.102:0/2932772277 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8d8004d510 msgr2=0x7f8d8004fa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:47.585 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:47.565 7f8d9d4ee700 1 --2- 172.21.2.102:0/2932772277 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8d8004d510 0x7f8d8004fa30 secure :-1 s=READY pgs=1531 cs=0 l=1 rev1=1 rx=0x7f8d90009a40 tx=0x7f8d90009de0).stop 2021-02-10T11:25:47.585 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:47.565 7f8d97ac0700 1 -- 172.21.2.102:0/2932772277 reap_dead start 2021-02-10T11:25:47.585 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:47.565 7f8d9d4ee700 1 -- 172.21.2.102:0/2932772277 shutdown_connections 2021-02-10T11:25:47.585 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:47.565 7f8d9d4ee700 1 -- 172.21.2.102:0/2932772277 >> 172.21.2.102:0/2932772277 conn(0x7f8d980ce420 msgr2=0x7f8d980d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:25:47.585 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:47.565 7f8d9d4ee700 1 -- 172.21.2.102:0/2932772277 shutdown_connections 2021-02-10T11:25:47.586 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:47.565 7f8d9d4ee700 1 -- 172.21.2.102:0/2932772277 wait complete. 2021-02-10T11:25:47.586 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:47.580 7f8d9d4ee700 1 Processor -- start 2021-02-10T11:25:47.586 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:47.580 7f8d9d4ee700 1 -- start start 2021-02-10T11:25:47.586 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:47.580 7f8d9d4ee700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8d98085bc0 0x7f8d980863e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:47.586 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:47.580 7f8d9d4ee700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8d980954c0 0x7f8d98096300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:47.586 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:47.580 7f8d9d4ee700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8d980958c0 0x7f8d98096920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:47.586 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:47.580 7f8d9d4ee700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d98095d80 con 0x7f8d980954c0 2021-02-10T11:25:47.586 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:47.580 7f8d9d4ee700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d980bfa60 con 0x7f8d980958c0 2021-02-10T11:25:47.586 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:47.580 7f8d9d4ee700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d98085fc0 con 0x7f8d98085bc0 2021-02-10T11:25:47.587 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:47.580 7f8d96abe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8d980954c0 0x7f8d98096300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:47.587 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:47.580 7f8d96abe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8d980954c0 0x7f8d98096300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:58224/0 (socket says 172.21.2.102:58224) 2021-02-10T11:25:47.587 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:47.580 7f8d96abe700 1 -- 172.21.2.102:0/4194392921 learned_addr learned my addr 172.21.2.102:0/4194392921 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:25:47.587 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:47.581 7f8d96abe700 1 -- 172.21.2.102:0/4194392921 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8d98085bc0 msgr2=0x7f8d980863e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:47.587 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:47.581 7f8d96abe700 1 --2- 172.21.2.102:0/4194392921 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8d98085bc0 0x7f8d980863e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:47.587 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:47.581 7f8d96abe700 1 -- 172.21.2.102:0/4194392921 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8d980958c0 msgr2=0x7f8d98096920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:47.587 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:47.581 7f8d96abe700 1 --2- 172.21.2.102:0/4194392921 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8d980958c0 0x7f8d98096920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:47.587 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:47.581 7f8d96abe700 1 -- 172.21.2.102:0/4194392921 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d98086c10 con 0x7f8d980954c0 2021-02-10T11:25:47.587 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:47.581 7f8d96abe700 1 --2- 172.21.2.102:0/4194392921 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8d980954c0 0x7f8d98096300 secure :-1 s=READY pgs=1532 cs=0 l=1 rev1=1 rx=0x7f8d8c001c60 tx=0x7f8d8c00a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:47.588 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:47.581 7f8d7ffff700 1 -- 172.21.2.102:0/4194392921 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8d8c00acc0 con 0x7f8d980954c0 2021-02-10T11:25:47.588 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:47.581 7f8d7ffff700 1 -- 172.21.2.102:0/4194392921 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d8c004350 con 0x7f8d980954c0 2021-02-10T11:25:47.588 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:47.581 7f8d9d4ee700 1 -- 172.21.2.102:0/4194392921 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d98086c10 con 0x7f8d980954c0 2021-02-10T11:25:47.588 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:47.581 7f8d7ffff700 1 -- 172.21.2.102:0/4194392921 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8d8c0047b0 con 0x7f8d980954c0 2021-02-10T11:25:47.588 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:47.581 7f8d9d4ee700 1 -- 172.21.2.102:0/4194392921 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d98097230 con 0x7f8d980954c0 2021-02-10T11:25:47.588 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:47.582 7f8d7ffff700 1 -- 172.21.2.102:0/4194392921 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f8d8c004740 con 0x7f8d980954c0 2021-02-10T11:25:47.588 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:47.582 7f8d7ffff700 1 --2- 172.21.2.102:0/4194392921 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8d780186c0 0x7f8d7801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:47.588 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:47.582 7f8d7ffff700 1 -- 172.21.2.102:0/4194392921 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f8d8c0323b0 con 0x7f8d980954c0 2021-02-10T11:25:47.589 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:47.582 7f8d972bf700 1 --2- 172.21.2.102:0/4194392921 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8d780186c0 0x7f8d7801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:47.589 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:47.582 7f8d972bf700 1 --2- 172.21.2.102:0/4194392921 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8d780186c0 0x7f8d7801abc0 crc :-1 s=READY pgs=698 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:47.612 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:47.613 7f8d9d4ee700 1 -- 172.21.2.102:0/4194392921 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d980fe1c0 con 0x7f8d980954c0 2021-02-10T11:25:47.614 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:47.615 7f8d7ffff700 1 -- 172.21.2.102:0/4194392921 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f8d8c004530 con 0x7f8d980954c0 2021-02-10T11:25:47.743 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:47.743 7f8d9d4ee700 1 -- 172.21.2.102:0/4194392921 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7f8d980fe1c0 con 0x7f8d980954c0 2021-02-10T11:25:47.743 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:47.743 7f8d7ffff700 1 -- 172.21.2.102:0/4194392921 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v66) v1 ==== 104+0+10 (secure 0 0 0) 0x7f8d8c004720 con 0x7f8d980954c0 2021-02-10T11:25:47.743 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:25:47.745 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:47.745 7f8d9d4ee700 1 -- 172.21.2.102:0/4194392921 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8d780186c0 msgr2=0x7f8d7801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:47.745 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:47.745 7f8d9d4ee700 1 --2- 172.21.2.102:0/4194392921 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8d780186c0 0x7f8d7801abc0 crc :-1 s=READY pgs=698 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:47.745 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:47.745 7f8d9d4ee700 1 -- 172.21.2.102:0/4194392921 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8d980954c0 msgr2=0x7f8d98096300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:47.745 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:47.745 7f8d9d4ee700 1 --2- 172.21.2.102:0/4194392921 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8d980954c0 0x7f8d98096300 secure :-1 s=READY pgs=1532 cs=0 l=1 rev1=1 rx=0x7f8d8c001c60 tx=0x7f8d8c00a8d0).stop 2021-02-10T11:25:47.745 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:47.745 7f8d9d4ee700 1 -- 172.21.2.102:0/4194392921 shutdown_connections 2021-02-10T11:25:47.745 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:47.745 7f8d9d4ee700 1 --2- 172.21.2.102:0/4194392921 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8d780186c0 0x7f8d7801abc0 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:47.745 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:47.745 7f8d9d4ee700 1 --2- 172.21.2.102:0/4194392921 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8d980958c0 0x7f8d98096920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:47.745 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:47.745 7f8d9d4ee700 1 --2- 172.21.2.102:0/4194392921 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8d980954c0 0x7f8d98096300 unknown :-1 s=CLOSED pgs=1532 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:47.745 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:47.745 7f8d9d4ee700 1 --2- 172.21.2.102:0/4194392921 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8d98085bc0 0x7f8d980863e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:47.746 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:47.745 7f8d9d4ee700 1 -- 172.21.2.102:0/4194392921 >> 172.21.2.102:0/4194392921 conn(0x7f8d980bef00 msgr2=0x7f8d9809d1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:25:47.746 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:47.746 7f8d9d4ee700 1 -- 172.21.2.102:0/4194392921 shutdown_connections 2021-02-10T11:25:47.746 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:47.746 7f8d9d4ee700 1 -- 172.21.2.102:0/4194392921 wait complete. 2021-02-10T11:25:47.754 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-02-10T11:25:48.183 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:48.167 7f8951fcf700 1 Processor -- start 2021-02-10T11:25:48.183 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:48.167 7f8951fcf700 1 -- start start 2021-02-10T11:25:48.183 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:48.167 7f8951fcf700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f894c0d1870 0x7f894c1b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:48.183 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:48.167 7f8951fcf700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f894c0cff70 0x7f894c109130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:48.183 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:48.167 7f8951fcf700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f894c109a60 con 0x7f894c1b0ae0 2021-02-10T11:25:48.183 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:48.167 7f8951fcf700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f894c109750 con 0x7f894c0cff70 2021-02-10T11:25:48.183 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:48.167 7f8951fcf700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f894c088070 con 0x7f894c0d1870 2021-02-10T11:25:48.183 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:48.167 7f894affd700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f894c1b0ae0 0x7f894c108940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:50454/0 (socket says 172.21.2.102:50454) 2021-02-10T11:25:48.184 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:48.167 7f894affd700 1 -- 172.21.2.102:0/2403372605 learned_addr learned my addr 172.21.2.102:0/2403372605 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:25:48.184 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:48.168 7f894b7fe700 1 --2- 172.21.2.102:0/2403372605 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f894c0d1870 0x7f894c1b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:48.184 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:48.168 7f894bfff700 1 --2- 172.21.2.102:0/2403372605 >> v2:172.21.2.113:3300/0 conn(0x7f894c0cff70 0x7f894c109130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:48.184 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:48.168 7f894a7fc700 1 -- 172.21.2.102:0/2403372605 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1132539223 0 0) 0x7f893c0009b0 con 0x7f894c1b0ae0 2021-02-10T11:25:48.184 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:48.168 7f894a7fc700 1 -- 172.21.2.102:0/2403372605 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8938002fe0 con 0x7f894c1b0ae0 2021-02-10T11:25:48.184 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:48.168 7f894a7fc700 1 -- 172.21.2.102:0/2403372605 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3411343505 0 0) 0x7f893c001070 con 0x7f894c1b0ae0 2021-02-10T11:25:48.184 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:48.168 7f894b7fe700 1 -- 172.21.2.102:0/2403372605 >> v2:172.21.2.113:3300/0 conn(0x7f894c0cff70 msgr2=0x7f894c109130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:25:48.184 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:48.168 7f894b7fe700 1 --2- 172.21.2.102:0/2403372605 >> v2:172.21.2.113:3300/0 conn(0x7f894c0cff70 0x7f894c109130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:48.184 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:48.168 7f894b7fe700 1 -- 172.21.2.102:0/2403372605 >> v1:172.21.2.102:6789/0 conn(0x7f894c1b0ae0 legacy=0x7f894c108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:48.184 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:48.168 7f894b7fe700 1 -- 172.21.2.102:0/2403372605 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f894c088260 con 0x7f894c0d1870 2021-02-10T11:25:48.184 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:48.168 7f894b7fe700 1 --2- 172.21.2.102:0/2403372605 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f894c0d1870 0x7f894c1b04c0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7f8934006650 tx=0x7f8934009a80).ready entity=mon.2 client_cookie=837b1edc826413a6 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:48.184 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:48.168 7f894a7fc700 1 -- 172.21.2.102:0/2403372605 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f89340009b0 con 0x7f894c0d1870 2021-02-10T11:25:48.185 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:48.168 7f894a7fc700 1 -- 172.21.2.102:0/2403372605 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f893400c0c0 con 0x7f894c0d1870 2021-02-10T11:25:48.185 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:48.168 7f894a7fc700 1 -- 172.21.2.102:0/2403372605 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f893400c5e0 con 0x7f894c0d1870 2021-02-10T11:25:48.185 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:48.169 7f8951fcf700 1 -- 172.21.2.102:0/2403372605 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f894c0d1870 msgr2=0x7f894c1b04c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:48.185 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:48.169 7f8951fcf700 1 --2- 172.21.2.102:0/2403372605 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f894c0d1870 0x7f894c1b04c0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7f8934006650 tx=0x7f8934009a80).stop 2021-02-10T11:25:48.185 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:48.169 7f8951fcf700 1 -- 172.21.2.102:0/2403372605 shutdown_connections 2021-02-10T11:25:48.185 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:48.169 7f8951fcf700 1 --2- 172.21.2.102:0/2403372605 >> v2:172.21.2.113:3300/0 conn(0x7f894c0cff70 0x7f894c109130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:48.185 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:48.169 7f8951fcf700 1 --2- 172.21.2.102:0/2403372605 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f894c0d1870 0x7f894c1b04c0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:48.185 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:48.169 7f8951fcf700 1 -- 172.21.2.102:0/2403372605 >> 172.21.2.102:0/2403372605 conn(0x7f894c0ce420 msgr2=0x7f894c0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:25:48.185 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:48.169 7f8951fcf700 1 -- 172.21.2.102:0/2403372605 shutdown_connections 2021-02-10T11:25:48.185 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:48.169 7f8951fcf700 1 -- 172.21.2.102:0/2403372605 wait complete. 2021-02-10T11:25:48.187 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:48.184 7f8951fcf700 1 Processor -- start 2021-02-10T11:25:48.187 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:48.184 7f8951fcf700 1 -- start start 2021-02-10T11:25:48.187 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:48.184 7f8951fcf700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f894c0c1850 0x7f894c0c1f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:48.187 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:48.184 7f8951fcf700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f894c0c2810 0x7f894c0c2c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:48.187 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:48.184 7f8951fcf700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f894c0c3230 0x7f894c0c3630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:48.187 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:48.184 7f8951fcf700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f894c0c1c50 con 0x7f894c0c3230 2021-02-10T11:25:48.187 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:48.184 7f8951fcf700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f894c084920 con 0x7f894c0c2810 2021-02-10T11:25:48.187 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:48.184 7f8951fcf700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f894c084b40 con 0x7f894c0c1850 2021-02-10T11:25:48.187 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:48.185 7f894bfff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f894c0c3230 0x7f894c0c3630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:48.187 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:48.185 7f894bfff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f894c0c3230 0x7f894c0c3630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:58236/0 (socket says 172.21.2.102:58236) 2021-02-10T11:25:48.187 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:48.185 7f894bfff700 1 -- 172.21.2.102:0/2767946145 learned_addr learned my addr 172.21.2.102:0/2767946145 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:25:48.188 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:48.185 7f894bfff700 1 -- 172.21.2.102:0/2767946145 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f894c0c1850 msgr2=0x7f894c0c1f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:48.188 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:48.185 7f894bfff700 1 --2- 172.21.2.102:0/2767946145 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f894c0c1850 0x7f894c0c1f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:48.188 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:48.185 7f894bfff700 1 -- 172.21.2.102:0/2767946145 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f894c0c2810 msgr2=0x7f894c0c2c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:48.188 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:48.185 7f894bfff700 1 --2- 172.21.2.102:0/2767946145 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f894c0c2810 0x7f894c0c2c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:48.188 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:48.185 7f894bfff700 1 -- 172.21.2.102:0/2767946145 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f894c084d60 con 0x7f894c0c3230 2021-02-10T11:25:48.188 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:48.185 7f894bfff700 1 --2- 172.21.2.102:0/2767946145 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f894c0c3230 0x7f894c0c3630 secure :-1 s=READY pgs=1534 cs=0 l=1 rev1=1 rx=0x7f894400b970 tx=0x7f894400bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:48.188 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:48.185 7f8943fff700 1 -- 172.21.2.102:0/2767946145 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8944000f90 con 0x7f894c0c3230 2021-02-10T11:25:48.188 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:48.185 7f8943fff700 1 -- 172.21.2.102:0/2767946145 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8944003030 con 0x7f894c0c3230 2021-02-10T11:25:48.188 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:48.185 7f8943fff700 1 -- 172.21.2.102:0/2767946145 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8944001400 con 0x7f894c0c3230 2021-02-10T11:25:48.188 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:48.185 7f8951fcf700 1 -- 172.21.2.102:0/2767946145 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f894c084d60 con 0x7f894c0c3230 2021-02-10T11:25:48.188 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:48.185 7f8951fcf700 1 -- 172.21.2.102:0/2767946145 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f894c0852f0 con 0x7f894c0c3230 2021-02-10T11:25:48.188 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:48.186 7f8943fff700 1 -- 172.21.2.102:0/2767946145 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f89440056b0 con 0x7f894c0c3230 2021-02-10T11:25:48.189 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:48.186 7f8943fff700 1 --2- 172.21.2.102:0/2767946145 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f89300186d0 0x7f893001abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:48.189 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:48.186 7f8943fff700 1 -- 172.21.2.102:0/2767946145 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f8944032b30 con 0x7f894c0c3230 2021-02-10T11:25:48.189 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:48.186 7f894b7fe700 1 --2- 172.21.2.102:0/2767946145 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f89300186d0 0x7f893001abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:48.189 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:48.186 7f894b7fe700 1 --2- 172.21.2.102:0/2767946145 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f89300186d0 0x7f893001abd0 crc :-1 s=READY pgs=699 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:48.217 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:48.217 7f8951fcf700 1 -- 172.21.2.102:0/2767946145 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f894c109a60 con 0x7f894c0c3230 2021-02-10T11:25:48.219 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:48.220 7f8943fff700 1 -- 172.21.2.102:0/2767946145 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f8944000f90 con 0x7f894c0c3230 2021-02-10T11:25:48.347 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:48.348 7f8951fcf700 1 -- 172.21.2.102:0/2767946145 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7f894c109a60 con 0x7f894c0c3230 2021-02-10T11:25:48.347 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:48.348 7f8943fff700 1 -- 172.21.2.102:0/2767946145 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v66) v1 ==== 100+0+10 (secure 0 0 0) 0x7f89440056b0 con 0x7f894c0c3230 2021-02-10T11:25:48.348 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:25:48.350 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:48.350 7f8951fcf700 1 -- 172.21.2.102:0/2767946145 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f89300186d0 msgr2=0x7f893001abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:48.350 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:48.350 7f8951fcf700 1 --2- 172.21.2.102:0/2767946145 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f89300186d0 0x7f893001abd0 crc :-1 s=READY pgs=699 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:48.350 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:48.350 7f8951fcf700 1 -- 172.21.2.102:0/2767946145 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f894c0c3230 msgr2=0x7f894c0c3630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:48.350 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:48.350 7f8951fcf700 1 --2- 172.21.2.102:0/2767946145 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f894c0c3230 0x7f894c0c3630 secure :-1 s=READY pgs=1534 cs=0 l=1 rev1=1 rx=0x7f894400b970 tx=0x7f894400bd10).stop 2021-02-10T11:25:48.350 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:48.350 7f8951fcf700 1 -- 172.21.2.102:0/2767946145 shutdown_connections 2021-02-10T11:25:48.350 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:48.350 7f8951fcf700 1 --2- 172.21.2.102:0/2767946145 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f894c0c3230 0x7f894c0c3630 unknown :-1 s=CLOSED pgs=1534 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:48.350 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:48.350 7f8951fcf700 1 --2- 172.21.2.102:0/2767946145 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f89300186d0 0x7f893001abd0 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:48.350 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:48.350 7f8951fcf700 1 --2- 172.21.2.102:0/2767946145 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f894c0c2810 0x7f894c0c2c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:48.350 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:48.350 7f8951fcf700 1 --2- 172.21.2.102:0/2767946145 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f894c0c1850 0x7f894c0c1f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:48.350 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:48.350 7f8951fcf700 1 -- 172.21.2.102:0/2767946145 >> 172.21.2.102:0/2767946145 conn(0x7f894c0af690 msgr2=0x7f894c094c40 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:25:48.351 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:48.351 7f8951fcf700 1 -- 172.21.2.102:0/2767946145 shutdown_connections 2021-02-10T11:25:48.351 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:48.351 7f8951fcf700 1 -- 172.21.2.102:0/2767946145 wait complete. 2021-02-10T11:25:48.358 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-02-10T11:25:48.358 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:25:48.788 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:48.769 7fac44bc3700 1 Processor -- start 2021-02-10T11:25:48.789 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:48.769 7fac44bc3700 1 -- start start 2021-02-10T11:25:48.789 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:48.770 7fac44bc3700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fac400d1870 0x7fac401b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:48.789 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:48.770 7fac44bc3700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fac401b0a10 0x7fac400cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:48.789 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:48.770 7fac44bc3700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac400f0b10 con 0x7fac400d1c70 2021-02-10T11:25:48.790 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:48.770 7fac44bc3700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac400e5ed0 con 0x7fac400d1870 2021-02-10T11:25:48.790 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:48.770 7fac44bc3700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac400f0650 con 0x7fac401b0a10 2021-02-10T11:25:48.790 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:48.770 7fac3e85e700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fac400d1870 0x7fac401b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:48.790 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:48.770 7fac3e05d700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fac401b0a10 0x7fac400cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:48.790 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:48.770 7fac3e85e700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fac400d1870 0x7fac401b03f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:58242/0 (socket says 172.21.2.102:58242) 2021-02-10T11:25:48.790 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:48.770 7fac3e05d700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fac401b0a10 0x7fac400cf330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.113:3301/0 says I am v2:172.21.2.102:35546/0 (socket says 172.21.2.102:35546) 2021-02-10T11:25:48.791 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:48.770 7fac3f05f700 1 --1- >> v1:172.21.2.113:6789/0 conn(0x7fac400d1c70 0x7fac400cf950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.113:6789/0 says I am v1:172.21.2.102:57790/0 (socket says 172.21.2.102:57790) 2021-02-10T11:25:48.791 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:48.770 7fac3e85e700 1 -- 172.21.2.102:0/487700364 learned_addr learned my addr 172.21.2.102:0/487700364 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:25:48.791 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:48.770 7fac3e05d700 1 -- 172.21.2.102:0/487700364 >> v2:172.21.2.102:3300/0 conn(0x7fac400d1870 msgr2=0x7fac401b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:25:48.791 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:48.770 7fac3e05d700 1 --2- 172.21.2.102:0/487700364 >> v2:172.21.2.102:3300/0 conn(0x7fac400d1870 0x7fac401b03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:48.791 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:48.771 7fac3e05d700 1 -- 172.21.2.102:0/487700364 >> v1:172.21.2.113:6789/0 conn(0x7fac400d1c70 legacy=0x7fac400cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:48.791 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:48.771 7fac3e05d700 1 -- 172.21.2.102:0/487700364 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac400f0870 con 0x7fac401b0a10 2021-02-10T11:25:48.792 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:48.771 7fac3e85e700 1 --2- 172.21.2.102:0/487700364 >> v2:172.21.2.102:3300/0 conn(0x7fac400d1870 0x7fac401b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:25:48.792 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:48.771 7fac3e05d700 1 --2- 172.21.2.102:0/487700364 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fac401b0a10 0x7fac400cf330 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7fac34006650 tx=0x7fac34009a80).ready entity=mon.2 client_cookie=c9d14de0715dc3e6 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:48.792 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:48.771 7fac3d85c700 1 -- 172.21.2.102:0/487700364 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fac340009b0 con 0x7fac401b0a10 2021-02-10T11:25:48.792 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:48.771 7fac3d85c700 1 -- 172.21.2.102:0/487700364 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac3400c0c0 con 0x7fac401b0a10 2021-02-10T11:25:48.792 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:48.771 7fac3d85c700 1 -- 172.21.2.102:0/487700364 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fac3400c5e0 con 0x7fac401b0a10 2021-02-10T11:25:48.793 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:48.771 7fac44bc3700 1 -- 172.21.2.102:0/487700364 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fac401b0a10 msgr2=0x7fac400cf330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:48.793 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:48.771 7fac44bc3700 1 --2- 172.21.2.102:0/487700364 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fac401b0a10 0x7fac400cf330 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7fac34006650 tx=0x7fac34009a80).stop 2021-02-10T11:25:48.793 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:48.772 7fac44bc3700 1 -- 172.21.2.102:0/487700364 shutdown_connections 2021-02-10T11:25:48.793 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:48.772 7fac44bc3700 1 --2- 172.21.2.102:0/487700364 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fac401b0a10 0x7fac400cf330 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:48.793 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:48.772 7fac44bc3700 1 --2- 172.21.2.102:0/487700364 >> v2:172.21.2.102:3300/0 conn(0x7fac400d1870 0x7fac401b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:48.793 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:48.772 7fac44bc3700 1 -- 172.21.2.102:0/487700364 >> 172.21.2.102:0/487700364 conn(0x7fac400ce450 msgr2=0x7fac400d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:25:48.794 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:48.772 7fac44bc3700 1 -- 172.21.2.102:0/487700364 shutdown_connections 2021-02-10T11:25:48.794 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:48.772 7fac44bc3700 1 -- 172.21.2.102:0/487700364 wait complete. 2021-02-10T11:25:48.794 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:48.786 7fac44bc3700 1 Processor -- start 2021-02-10T11:25:48.794 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:48.786 7fac44bc3700 1 -- start start 2021-02-10T11:25:48.794 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:48.787 7fac44bc3700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fac400ab7b0 0x7fac400abbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:48.794 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:48.787 7fac44bc3700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fac401aba60 0x7fac401abe60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:48.795 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:48.787 7fac44bc3700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fac401ac480 0x7fac40091390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:48.795 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:48.787 7fac44bc3700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac400919b0 con 0x7fac401aba60 2021-02-10T11:25:48.795 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:48.787 7fac44bc3700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac40091bd0 con 0x7fac400ab7b0 2021-02-10T11:25:48.795 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:48.787 7fac44bc3700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac40091df0 con 0x7fac401ac480 2021-02-10T11:25:48.795 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:48.787 7fac3e05d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fac401aba60 0x7fac401abe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:48.796 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:48.787 7fac3e05d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fac401aba60 0x7fac401abe60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:58246/0 (socket says 172.21.2.102:58246) 2021-02-10T11:25:48.796 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:48.787 7fac3e05d700 1 -- 172.21.2.102:0/2879292719 learned_addr learned my addr 172.21.2.102:0/2879292719 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:25:48.796 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:48.787 7fac3e85e700 1 --2- 172.21.2.102:0/2879292719 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fac400ab7b0 0x7fac400abbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:48.796 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:48.787 7fac3e05d700 1 -- 172.21.2.102:0/2879292719 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fac401ac480 msgr2=0x7fac40091390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:48.796 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:48.787 7fac3e05d700 1 --2- 172.21.2.102:0/2879292719 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fac401ac480 0x7fac40091390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:48.796 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:48.787 7fac3e05d700 1 -- 172.21.2.102:0/2879292719 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fac400ab7b0 msgr2=0x7fac400abbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:48.797 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:48.787 7fac3e05d700 1 --2- 172.21.2.102:0/2879292719 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fac400ab7b0 0x7fac400abbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:48.797 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:48.787 7fac3e05d700 1 -- 172.21.2.102:0/2879292719 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac40092010 con 0x7fac401aba60 2021-02-10T11:25:48.797 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:48.787 7fac3e05d700 1 --2- 172.21.2.102:0/2879292719 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fac401aba60 0x7fac401abe60 secure :-1 s=READY pgs=1535 cs=0 l=1 rev1=1 rx=0x7fac34009630 tx=0x7fac3400b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:48.797 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:48.787 7fac2f7fe700 1 -- 172.21.2.102:0/2879292719 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fac340093a0 con 0x7fac401aba60 2021-02-10T11:25:48.797 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:48.787 7fac2f7fe700 1 -- 172.21.2.102:0/2879292719 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac3400bd70 con 0x7fac401aba60 2021-02-10T11:25:48.798 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:48.787 7fac44bc3700 1 -- 172.21.2.102:0/2879292719 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac40092010 con 0x7fac401aba60 2021-02-10T11:25:48.798 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:48.787 7fac2f7fe700 1 -- 172.21.2.102:0/2879292719 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fac34005560 con 0x7fac401aba60 2021-02-10T11:25:48.798 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:48.787 7fac44bc3700 1 -- 172.21.2.102:0/2879292719 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac40092010 con 0x7fac401aba60 2021-02-10T11:25:48.798 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:48.788 7fac2f7fe700 1 -- 172.21.2.102:0/2879292719 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fac34005770 con 0x7fac401aba60 2021-02-10T11:25:48.798 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:48.788 7fac2f7fe700 1 --2- 172.21.2.102:0/2879292719 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fac240186c0 0x7fac2401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:48.798 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:48.788 7fac2f7fe700 1 -- 172.21.2.102:0/2879292719 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fac34032610 con 0x7fac401aba60 2021-02-10T11:25:48.799 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:48.788 7fac3e85e700 1 --2- 172.21.2.102:0/2879292719 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fac240186c0 0x7fac2401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:48.799 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:48.788 7fac3e85e700 1 --2- 172.21.2.102:0/2879292719 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fac240186c0 0x7fac2401abc0 crc :-1 s=READY pgs=700 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:48.819 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:48.820 7fac44bc3700 1 -- 172.21.2.102:0/2879292719 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac400eb910 con 0x7fac401aba60 2021-02-10T11:25:48.821 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:48.822 7fac2f7fe700 1 -- 172.21.2.102:0/2879292719 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fac34005560 con 0x7fac401aba60 2021-02-10T11:25:48.949 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:48.950 7fac44bc3700 1 -- 172.21.2.102:0/2879292719 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fac400eb910 con 0x7fac401aba60 2021-02-10T11:25:48.950 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:48.950 7fac2f7fe700 1 -- 172.21.2.102:0/2879292719 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) v1 ==== 94+0+2101 (secure 0 0 0) 0x7fac34005770 con 0x7fac401aba60 2021-02-10T11:25:48.950 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:25:48.951 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":48,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8412,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.102:6835/3414701760","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6834","nonce":3414701760},{"type":"v1","addr":"172.21.2.102:6835","nonce":3414701760}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":44}],"filesystems":[{"mdsmap":{"epoch":48,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:25:31.513947","modified":"2021-02-10 11:25:33.524416","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5813},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5813":{"gid":5813,"name":"b","rank":0,"incarnation":45,"state":"up:active","state_seq":3,"addr":"172.21.2.113:6833/2951398038","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6832","nonce":2951398038},{"type":"v1","addr":"172.21.2.113:6833","nonce":2951398038}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":6}]} 2021-02-10T11:25:48.951 INFO:teuthology.orchestra.run.gibba002.stderr:dumped fsmap epoch 48 2021-02-10T11:25:48.952 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:48.952 7fac44bc3700 1 -- 172.21.2.102:0/2879292719 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fac240186c0 msgr2=0x7fac2401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:48.952 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:48.952 7fac44bc3700 1 --2- 172.21.2.102:0/2879292719 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fac240186c0 0x7fac2401abc0 crc :-1 s=READY pgs=700 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:48.952 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:48.952 7fac44bc3700 1 -- 172.21.2.102:0/2879292719 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fac401aba60 msgr2=0x7fac401abe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:48.952 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:48.952 7fac44bc3700 1 --2- 172.21.2.102:0/2879292719 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fac401aba60 0x7fac401abe60 secure :-1 s=READY pgs=1535 cs=0 l=1 rev1=1 rx=0x7fac34009630 tx=0x7fac3400b980).stop 2021-02-10T11:25:48.952 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:48.953 7fac44bc3700 1 -- 172.21.2.102:0/2879292719 shutdown_connections 2021-02-10T11:25:48.952 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:48.953 7fac44bc3700 1 --2- 172.21.2.102:0/2879292719 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fac401ac480 0x7fac40091390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:48.953 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:48.953 7fac44bc3700 1 --2- 172.21.2.102:0/2879292719 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fac401aba60 0x7fac401abe60 unknown :-1 s=CLOSED pgs=1535 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:48.953 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:48.953 7fac44bc3700 1 --2- 172.21.2.102:0/2879292719 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fac240186c0 0x7fac2401abc0 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:48.953 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:48.953 7fac44bc3700 1 --2- 172.21.2.102:0/2879292719 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fac400ab7b0 0x7fac400abbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:48.953 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:48.953 7fac44bc3700 1 -- 172.21.2.102:0/2879292719 >> 172.21.2.102:0/2879292719 conn(0x7fac4019e640 msgr2=0x7fac401b2a60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:25:48.953 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:48.953 7fac44bc3700 1 -- 172.21.2.102:0/2879292719 shutdown_connections 2021-02-10T11:25:48.953 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:48.953 7fac44bc3700 1 -- 172.21.2.102:0/2879292719 wait complete. 2021-02-10T11:25:49.162 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 48, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:25:31.513947', 'modified': '2021-02-10 11:25:33.524416', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5813}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5813': {'gid': 5813, 'name': 'b', 'rank': 0, 'incarnation': 45, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.113:6833/2951398038', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.113:6832', 'nonce': 2951398038}, {'type': 'v1', 'addr': '172.21.2.113:6833', 'nonce': 2951398038}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [13], 'metadata_pool': 12, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:25:49.162 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:25:49.163 DEBUG:teuthology.orchestra.run.gibba013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:25:49.560 INFO:teuthology.orchestra.run.gibba013.stdout:{ 2021-02-10T11:25:49.560 INFO:teuthology.orchestra.run.gibba013.stdout: "cluster_fsid": "68db30ea-dbce-4aef-a6df-f01c6fea0513", 2021-02-10T11:25:49.560 INFO:teuthology.orchestra.run.gibba013.stdout: "whoami": 0, 2021-02-10T11:25:49.560 INFO:teuthology.orchestra.run.gibba013.stdout: "id": 5813, 2021-02-10T11:25:49.561 INFO:teuthology.orchestra.run.gibba013.stdout: "want_state": "up:active", 2021-02-10T11:25:49.561 INFO:teuthology.orchestra.run.gibba013.stdout: "state": "up:active", 2021-02-10T11:25:49.561 INFO:teuthology.orchestra.run.gibba013.stdout: "rank_uptime": 18.04524395, 2021-02-10T11:25:49.561 INFO:teuthology.orchestra.run.gibba013.stdout: "mdsmap_epoch": 48, 2021-02-10T11:25:49.561 INFO:teuthology.orchestra.run.gibba013.stdout: "osdmap_epoch": 66, 2021-02-10T11:25:49.561 INFO:teuthology.orchestra.run.gibba013.stdout: "osdmap_epoch_barrier": 66, 2021-02-10T11:25:49.561 INFO:teuthology.orchestra.run.gibba013.stdout: "uptime": 23.681299999 2021-02-10T11:25:49.561 INFO:teuthology.orchestra.run.gibba013.stdout:} 2021-02-10T11:25:49.565 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "68db30ea-dbce-4aef-a6df-f01c6fea0513", "whoami": 0, "id": 5813, "want_state": "up:active", "state": "up:active", "rank_uptime": 18.04524395, "mdsmap_epoch": 48, "osdmap_epoch": 66, "osdmap_epoch_barrier": 66, "uptime": 23.681299999 } 2021-02-10T11:25:49.565 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2021-02-10T11:25:49.565 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2021-02-10T11:25:49.565 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.1 at ubuntu@gibba002.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.1... 2021-02-10T11:25:49.566 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:49.566 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec mkdir -p /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:25:49.901 DEBUG:teuthology.orchestra.run.gibba002:> sudo modprobe fuse 2021-02-10T11:25:50.245 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:25:50.245 DEBUG:teuthology.orchestra.run.gibba002:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:25:50.588 INFO:teuthology.orchestra.run.gibba002.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:25:50.588 INFO:teuthology.orchestra.run.gibba002.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:25:50.589 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:25:50.589 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:50.590 DEBUG:teuthology.orchestra.run.gibba002:> ls /sys/fs/fuse/connections 2021-02-10T11:25:50.923 INFO:teuthology.orchestra.run.gibba002.stdout:43 2021-02-10T11:25:50.923 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [43] 2021-02-10T11:25:50.924 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.1 /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:25:50.927 DEBUG:teuthology.orchestra.run.gibba002:> sudo modprobe fuse 2021-02-10T11:25:51.273 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:25:51.273 DEBUG:teuthology.orchestra.run.gibba002:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:25:51.338 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba002.stderr:ceph-fuse[159844]: starting ceph client 2021-02-10T11:25:51.339 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba002.stderr:2021-02-10 11:25:51.339 7f4cd7a92f80 -1 init, newargv = 0x55b3500321a0 newargc=9 2021-02-10T11:25:51.344 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba002.stderr:ceph-fuse[159844]: starting fuse 2021-02-10T11:25:51.616 INFO:teuthology.orchestra.run.gibba002.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:25:51.616 INFO:teuthology.orchestra.run.gibba002.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:25:51.617 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:25:51.617 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:51.617 DEBUG:teuthology.orchestra.run.gibba002:> ls /sys/fs/fuse/connections 2021-02-10T11:25:51.951 INFO:teuthology.orchestra.run.gibba002.stdout:43 2021-02-10T11:25:51.951 INFO:teuthology.orchestra.run.gibba002.stdout:44 2021-02-10T11:25:51.952 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43, 44] 2021-02-10T11:25:51.952 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:51.952 DEBUG:teuthology.orchestra.run.gibba002:> sudo python3 -c ' 2021-02-10T11:25:51.952 DEBUG:teuthology.orchestra.run.gibba002:> import glob 2021-02-10T11:25:51.952 DEBUG:teuthology.orchestra.run.gibba002:> import re 2021-02-10T11:25:51.952 DEBUG:teuthology.orchestra.run.gibba002:> import os 2021-02-10T11:25:51.952 DEBUG:teuthology.orchestra.run.gibba002:> import subprocess 2021-02-10T11:25:51.952 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:25:51.952 DEBUG:teuthology.orchestra.run.gibba002:> def find_socket(client_name): 2021-02-10T11:25:51.953 DEBUG:teuthology.orchestra.run.gibba002:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2021-02-10T11:25:51.953 DEBUG:teuthology.orchestra.run.gibba002:> files = glob.glob(asok_path) 2021-02-10T11:25:51.953 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:25:51.953 DEBUG:teuthology.orchestra.run.gibba002:> # Given a non-glob path, it better be there 2021-02-10T11:25:51.953 DEBUG:teuthology.orchestra.run.gibba002:> if "*" not in asok_path: 2021-02-10T11:25:51.953 DEBUG:teuthology.orchestra.run.gibba002:> assert(len(files) == 1) 2021-02-10T11:25:51.953 DEBUG:teuthology.orchestra.run.gibba002:> return files[0] 2021-02-10T11:25:51.953 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:25:51.953 DEBUG:teuthology.orchestra.run.gibba002:> for f in files: 2021-02-10T11:25:51.953 DEBUG:teuthology.orchestra.run.gibba002:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2021-02-10T11:25:51.953 DEBUG:teuthology.orchestra.run.gibba002:> if os.path.exists("/proc/{0}".format(pid)): 2021-02-10T11:25:51.953 DEBUG:teuthology.orchestra.run.gibba002:> return f 2021-02-10T11:25:51.953 DEBUG:teuthology.orchestra.run.gibba002:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2021-02-10T11:25:51.953 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:25:51.953 DEBUG:teuthology.orchestra.run.gibba002:> print(find_socket("client.1")) 2021-02-10T11:25:51.954 DEBUG:teuthology.orchestra.run.gibba002:> ' 2021-02-10T11:25:52.315 INFO:teuthology.orchestra.run.gibba002.stdout:/var/run/ceph/ceph-client.1.159844.asok 2021-02-10T11:25:52.318 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.159844.asok 2021-02-10T11:25:52.318 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:52.318 DEBUG:teuthology.orchestra.run.gibba002:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.159844.asok status 2021-02-10T11:25:52.708 INFO:teuthology.orchestra.run.gibba002.stdout:{ 2021-02-10T11:25:52.709 INFO:teuthology.orchestra.run.gibba002.stdout: "metadata": { 2021-02-10T11:25:52.709 INFO:teuthology.orchestra.run.gibba002.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:25:52.709 INFO:teuthology.orchestra.run.gibba002.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:25:52.709 INFO:teuthology.orchestra.run.gibba002.stdout: "entity_id": "1", 2021-02-10T11:25:52.709 INFO:teuthology.orchestra.run.gibba002.stdout: "hostname": "gibba002", 2021-02-10T11:25:52.709 INFO:teuthology.orchestra.run.gibba002.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2021-02-10T11:25:52.709 INFO:teuthology.orchestra.run.gibba002.stdout: "pid": "159844", 2021-02-10T11:25:52.709 INFO:teuthology.orchestra.run.gibba002.stdout: "root": "/" 2021-02-10T11:25:52.709 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:25:52.709 INFO:teuthology.orchestra.run.gibba002.stdout: "dentry_count": 0, 2021-02-10T11:25:52.709 INFO:teuthology.orchestra.run.gibba002.stdout: "dentry_pinned_count": 0, 2021-02-10T11:25:52.709 INFO:teuthology.orchestra.run.gibba002.stdout: "id": 8601, 2021-02-10T11:25:52.710 INFO:teuthology.orchestra.run.gibba002.stdout: "inst": { 2021-02-10T11:25:52.710 INFO:teuthology.orchestra.run.gibba002.stdout: "name": { 2021-02-10T11:25:52.710 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "client", 2021-02-10T11:25:52.710 INFO:teuthology.orchestra.run.gibba002.stdout: "num": 8601 2021-02-10T11:25:52.710 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:25:52.710 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": { 2021-02-10T11:25:52.710 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:25:52.710 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:0", 2021-02-10T11:25:52.710 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 573498401 2021-02-10T11:25:52.710 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:25:52.710 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:25:52.710 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": { 2021-02-10T11:25:52.711 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:25:52.711 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:0", 2021-02-10T11:25:52.711 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 573498401 2021-02-10T11:25:52.711 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:25:52.711 INFO:teuthology.orchestra.run.gibba002.stdout: "inst_str": "client.8601 172.21.2.102:0/573498401", 2021-02-10T11:25:52.711 INFO:teuthology.orchestra.run.gibba002.stdout: "addr_str": "172.21.2.102:0/573498401", 2021-02-10T11:25:52.711 INFO:teuthology.orchestra.run.gibba002.stdout: "inode_count": 1, 2021-02-10T11:25:52.711 INFO:teuthology.orchestra.run.gibba002.stdout: "mds_epoch": 48, 2021-02-10T11:25:52.711 INFO:teuthology.orchestra.run.gibba002.stdout: "osd_epoch": 66, 2021-02-10T11:25:52.711 INFO:teuthology.orchestra.run.gibba002.stdout: "osd_epoch_barrier": 0, 2021-02-10T11:25:52.711 INFO:teuthology.orchestra.run.gibba002.stdout: "blacklisted": false 2021-02-10T11:25:52.711 INFO:teuthology.orchestra.run.gibba002.stdout:} 2021-02-10T11:25:52.713 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:52.713 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:25:52.713 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:25:53.048 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:25:53.049 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2021-02-10T11:25:53.049 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:53.049 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:25:53.396 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2021-02-10T11:25:53.829 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:53.813 7f8aeaa80700 1 Processor -- start 2021-02-10T11:25:53.829 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:53.813 7f8aeaa80700 1 -- start start 2021-02-10T11:25:53.829 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:53.814 7f8aeaa80700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f8ae40d1c70 0x7f8ae40cef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:53.829 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:53.814 7f8aeaa80700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ae40d0170 con 0x7f8ae40d1870 2021-02-10T11:25:53.830 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:53.814 7f8aeaa80700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ae40fe1c0 con 0x7f8ae40cf580 2021-02-10T11:25:53.830 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:53.814 7f8aeaa80700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ae41b0cb0 con 0x7f8ae40d1c70 2021-02-10T11:25:53.830 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:53.814 7f8ae889c700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f8ae40d1870 0x7f8ae41b04c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:50490/0 (socket says 172.21.2.102:50490) 2021-02-10T11:25:53.830 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:53.814 7f8ae889c700 1 -- 172.21.2.102:0/838775588 learned_addr learned my addr 172.21.2.102:0/838775588 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:25:53.830 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:53.814 7f8ae3fff700 1 --2- 172.21.2.102:0/838775588 >> v2:172.21.2.113:3300/0 conn(0x7f8ae40d1c70 0x7f8ae40cef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:53.830 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:53.814 7f8ae37fe700 1 -- 172.21.2.102:0/838775588 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4285326829 0 0) 0x7f8acc0009b0 con 0x7f8ae40d1870 2021-02-10T11:25:53.830 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:53.814 7f8ae37fe700 1 -- 172.21.2.102:0/838775588 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ad0002fe0 con 0x7f8ae40d1870 2021-02-10T11:25:53.830 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:53.814 7f8ae37fe700 1 -- 172.21.2.102:0/838775588 <== mon.1 v1:172.21.2.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2690736456 0 0) 0x7f8adc000f90 con 0x7f8ae40cf580 2021-02-10T11:25:53.830 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:53.814 7f8ae37fe700 1 -- 172.21.2.102:0/838775588 --> v1:172.21.2.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ad0004370 con 0x7f8ae40cf580 2021-02-10T11:25:53.830 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:53.815 7f8ae3fff700 1 -- 172.21.2.102:0/838775588 >> v1:172.21.2.113:6789/0 conn(0x7f8ae40cf580 legacy=0x7f8ae40cf980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:53.830 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:53.815 7f8ae3fff700 1 -- 172.21.2.102:0/838775588 >> v1:172.21.2.102:6789/0 conn(0x7f8ae40d1870 legacy=0x7f8ae41b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:53.830 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:53.815 7f8ae3fff700 1 -- 172.21.2.102:0/838775588 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ae4109a60 con 0x7f8ae40d1c70 2021-02-10T11:25:53.830 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:53.815 7f8ae3fff700 1 --2- 172.21.2.102:0/838775588 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8ae40d1c70 0x7f8ae40cef60 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 rx=0x7f8ad4006650 tx=0x7f8ad4009a80).ready entity=mon.1 client_cookie=f13fcd2ce2731f9b server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:53.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:53.815 7f8ae37fe700 1 -- 172.21.2.102:0/838775588 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8ad400c0c0 con 0x7f8ae40d1c70 2021-02-10T11:25:53.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:53.815 7f8ae37fe700 1 -- 172.21.2.102:0/838775588 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ad400c340 con 0x7f8ae40d1c70 2021-02-10T11:25:53.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:53.815 7f8ae37fe700 1 -- 172.21.2.102:0/838775588 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8ad400c860 con 0x7f8ae40d1c70 2021-02-10T11:25:53.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:53.815 7f8aeaa80700 1 -- 172.21.2.102:0/838775588 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8ae40d1c70 msgr2=0x7f8ae40cef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:53.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:53.815 7f8aeaa80700 1 --2- 172.21.2.102:0/838775588 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8ae40d1c70 0x7f8ae40cef60 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 rx=0x7f8ad4006650 tx=0x7f8ad4009a80).stop 2021-02-10T11:25:53.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:53.815 7f8aeaa80700 1 -- 172.21.2.102:0/838775588 shutdown_connections 2021-02-10T11:25:53.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:53.815 7f8aeaa80700 1 --2- 172.21.2.102:0/838775588 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8ae40d1c70 0x7f8ae40cef60 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:53.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:53.815 7f8aeaa80700 1 -- 172.21.2.102:0/838775588 >> 172.21.2.102:0/838775588 conn(0x7f8ae40ce420 msgr2=0x7f8ae40d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:25:53.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:53.816 7f8aeaa80700 1 -- 172.21.2.102:0/838775588 shutdown_connections 2021-02-10T11:25:53.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:53.816 7f8aeaa80700 1 -- 172.21.2.102:0/838775588 wait complete. 2021-02-10T11:25:53.833 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:53.830 7f8aeaa80700 1 Processor -- start 2021-02-10T11:25:53.833 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:53.830 7f8aeaa80700 1 -- start start 2021-02-10T11:25:53.833 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:53.831 7f8aeaa80700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8ae419fb30 0x7f8ae419ff30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:53.833 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:53.831 7f8aeaa80700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8ae409a0d0 0x7f8ae409a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:53.833 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:53.831 7f8aeaa80700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8ae409aaf0 0x7f8ae41a0ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:53.833 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:53.831 7f8aeaa80700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ae41a0920 con 0x7f8ae409aaf0 2021-02-10T11:25:53.833 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:53.831 7f8aeaa80700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ae409aef0 con 0x7f8ae409a0d0 2021-02-10T11:25:53.833 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:53.831 7f8aeaa80700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ae41a0550 con 0x7f8ae419fb30 2021-02-10T11:25:53.833 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:53.831 7f8ae909d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8ae409aaf0 0x7f8ae41a0ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:53.833 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:53.831 7f8ae909d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8ae409aaf0 0x7f8ae41a0ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:58272/0 (socket says 172.21.2.102:58272) 2021-02-10T11:25:53.834 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:53.831 7f8ae909d700 1 -- 172.21.2.102:0/1903607929 learned_addr learned my addr 172.21.2.102:0/1903607929 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:25:53.834 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:53.831 7f8ae889c700 1 --2- 172.21.2.102:0/1903607929 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8ae419fb30 0x7f8ae419ff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:53.834 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:53.831 7f8ae3fff700 1 --2- 172.21.2.102:0/1903607929 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8ae409a0d0 0x7f8ae409a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:53.834 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:53.831 7f8ae909d700 1 -- 172.21.2.102:0/1903607929 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8ae419fb30 msgr2=0x7f8ae419ff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:53.834 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:53.831 7f8ae909d700 1 --2- 172.21.2.102:0/1903607929 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8ae419fb30 0x7f8ae419ff30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:53.834 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:53.831 7f8ae909d700 1 -- 172.21.2.102:0/1903607929 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8ae409a0d0 msgr2=0x7f8ae409a4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:53.834 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:53.831 7f8ae909d700 1 --2- 172.21.2.102:0/1903607929 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8ae409a0d0 0x7f8ae409a4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:53.834 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:53.831 7f8ae909d700 1 -- 172.21.2.102:0/1903607929 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ae41a14c0 con 0x7f8ae409aaf0 2021-02-10T11:25:53.834 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:53.831 7f8ae3fff700 1 --2- 172.21.2.102:0/1903607929 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8ae409a0d0 0x7f8ae409a4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:25:53.834 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:53.831 7f8ae909d700 1 --2- 172.21.2.102:0/1903607929 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8ae409aaf0 0x7f8ae41a0ea0 secure :-1 s=READY pgs=1539 cs=0 l=1 rev1=1 rx=0x7f8adc0099a0 tx=0x7f8adc00bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:53.834 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:53.831 7f8ae17fa700 1 -- 172.21.2.102:0/1903607929 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8adc003030 con 0x7f8ae409aaf0 2021-02-10T11:25:53.834 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:53.832 7f8ae17fa700 1 -- 172.21.2.102:0/1903607929 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8adc001230 con 0x7f8ae409aaf0 2021-02-10T11:25:53.834 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:53.832 7f8aeaa80700 1 -- 172.21.2.102:0/1903607929 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ae41a14c0 con 0x7f8ae409aaf0 2021-02-10T11:25:53.835 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:53.832 7f8aeaa80700 1 -- 172.21.2.102:0/1903607929 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ae41a1a50 con 0x7f8ae409aaf0 2021-02-10T11:25:53.835 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:53.832 7f8ae17fa700 1 -- 172.21.2.102:0/1903607929 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8adc001680 con 0x7f8ae409aaf0 2021-02-10T11:25:53.835 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:53.832 7f8ae17fa700 1 -- 172.21.2.102:0/1903607929 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f8adc001410 con 0x7f8ae409aaf0 2021-02-10T11:25:53.835 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:53.832 7f8ae17fa700 1 --2- 172.21.2.102:0/1903607929 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8ac8018730 0x7f8ac801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:53.835 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:53.832 7f8ae17fa700 1 -- 172.21.2.102:0/1903607929 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f8adc032b60 con 0x7f8ae409aaf0 2021-02-10T11:25:53.835 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:53.832 7f8ae889c700 1 --2- 172.21.2.102:0/1903607929 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8ac8018730 0x7f8ac801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:53.835 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:53.833 7f8ae889c700 1 --2- 172.21.2.102:0/1903607929 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8ac8018730 0x7f8ac801abc0 crc :-1 s=READY pgs=701 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:53.863 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:53.864 7f8aeaa80700 1 -- 172.21.2.102:0/1903607929 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ae40825d0 con 0x7f8ae409aaf0 2021-02-10T11:25:53.865 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:53.866 7f8ae17fa700 1 -- 172.21.2.102:0/1903607929 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f8adc000f20 con 0x7f8ae409aaf0 2021-02-10T11:25:54.001 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:54.002 7f8aeaa80700 1 -- 172.21.2.102:0/1903607929 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f8ae40825d0 con 0x7f8ae409aaf0 2021-02-10T11:25:54.056 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:54.056 7f8ae17fa700 1 -- 172.21.2.102:0/1903607929 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v49) v1 ==== 136+0+0 (secure 0 0 0) 0x7f8adc01f8a0 con 0x7f8ae409aaf0 2021-02-10T11:25:54.068 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:54.068 7f8aeaa80700 1 -- 172.21.2.102:0/1903607929 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8ac8018730 msgr2=0x7f8ac801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:54.068 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:54.068 7f8aeaa80700 1 --2- 172.21.2.102:0/1903607929 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8ac8018730 0x7f8ac801abc0 crc :-1 s=READY pgs=701 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:54.068 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:54.068 7f8aeaa80700 1 -- 172.21.2.102:0/1903607929 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8ae409aaf0 msgr2=0x7f8ae41a0ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:54.068 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:54.068 7f8aeaa80700 1 --2- 172.21.2.102:0/1903607929 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8ae409aaf0 0x7f8ae41a0ea0 secure :-1 s=READY pgs=1539 cs=0 l=1 rev1=1 rx=0x7f8adc0099a0 tx=0x7f8adc00bcf0).stop 2021-02-10T11:25:54.068 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:54.068 7f8aeaa80700 1 -- 172.21.2.102:0/1903607929 shutdown_connections 2021-02-10T11:25:54.068 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:54.068 7f8aeaa80700 1 --2- 172.21.2.102:0/1903607929 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8ae409aaf0 0x7f8ae41a0ea0 unknown :-1 s=CLOSED pgs=1539 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:54.068 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:54.068 7f8aeaa80700 1 --2- 172.21.2.102:0/1903607929 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8ac8018730 0x7f8ac801abc0 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:54.068 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:54.068 7f8aeaa80700 1 --2- 172.21.2.102:0/1903607929 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8ae409a0d0 0x7f8ae409a4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:54.068 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:54.068 7f8aeaa80700 1 --2- 172.21.2.102:0/1903607929 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8ae419fb30 0x7f8ae419ff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:54.068 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:54.068 7f8aeaa80700 1 -- 172.21.2.102:0/1903607929 >> 172.21.2.102:0/1903607929 conn(0x7f8ae41b3560 msgr2=0x7f8ae41b90e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:25:54.068 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:54.068 7f8aeaa80700 1 -- 172.21.2.102:0/1903607929 shutdown_connections 2021-02-10T11:25:54.069 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:54.068 7f8aeaa80700 1 -- 172.21.2.102:0/1903607929 wait complete. 2021-02-10T11:25:54.076 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs volume ls 2021-02-10T11:25:54.502 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:54.486 7fca79a0b700 1 Processor -- start 2021-02-10T11:25:54.502 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:54.486 7fca79a0b700 1 -- start start 2021-02-10T11:25:54.502 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:54.486 7fca79a0b700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fca740d0330 0x7fca740ef630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:54.502 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:54.486 7fca79a0b700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fca740d1c60 0x7fca740efc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:54.502 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:54.486 7fca79a0b700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca740f0b00 con 0x7fca740d1860 2021-02-10T11:25:54.502 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:54.486 7fca79a0b700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca740e5ec0 con 0x7fca740d0330 2021-02-10T11:25:54.502 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:54.486 7fca79a0b700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca741b0bd0 con 0x7fca740d1c60 2021-02-10T11:25:54.502 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:54.487 7fca72ffd700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fca740d0330 0x7fca740ef630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:54.503 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:54.487 7fca737fe700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fca740d1860 0x7fca741b03e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:50502/0 (socket says 172.21.2.102:50502) 2021-02-10T11:25:54.503 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:54.487 7fca737fe700 1 -- 172.21.2.102:0/3693640375 learned_addr learned my addr 172.21.2.102:0/3693640375 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:25:54.503 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:54.487 7fca72ffd700 1 -- 172.21.2.102:0/3693640375 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fca740d1c60 msgr2=0x7fca740efc50 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:25:54.503 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:54.487 7fca72ffd700 1 --2- 172.21.2.102:0/3693640375 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fca740d1c60 0x7fca740efc50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:54.503 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:54.487 7fca72ffd700 1 -- 172.21.2.102:0/3693640375 >> v1:172.21.2.102:6789/0 conn(0x7fca740d1860 legacy=0x7fca741b03e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:54.503 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:54.487 7fca72ffd700 1 -- 172.21.2.102:0/3693640375 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca740f2090 con 0x7fca740d0330 2021-02-10T11:25:54.503 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:54.487 7fca72ffd700 1 --2- 172.21.2.102:0/3693640375 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fca740d0330 0x7fca740ef630 secure :-1 s=READY pgs=1541 cs=0 l=1 rev1=1 rx=0x7fca64006650 tx=0x7fca64009a80).ready entity=mon.0 client_cookie=1ec9ba9ca0cea99e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:54.503 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:54.487 7fca727fc700 1 -- 172.21.2.102:0/3693640375 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fca6400c0c0 con 0x7fca740d0330 2021-02-10T11:25:54.503 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:54.487 7fca727fc700 1 -- 172.21.2.102:0/3693640375 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca6400c340 con 0x7fca740d0330 2021-02-10T11:25:54.503 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:54.487 7fca727fc700 1 -- 172.21.2.102:0/3693640375 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fca6400c860 con 0x7fca740d0330 2021-02-10T11:25:54.503 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:54.487 7fca79a0b700 1 -- 172.21.2.102:0/3693640375 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fca740d0330 msgr2=0x7fca740ef630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:54.503 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:54.487 7fca79a0b700 1 --2- 172.21.2.102:0/3693640375 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fca740d0330 0x7fca740ef630 secure :-1 s=READY pgs=1541 cs=0 l=1 rev1=1 rx=0x7fca64006650 tx=0x7fca64009a80).stop 2021-02-10T11:25:54.503 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:54.488 7fca79a0b700 1 -- 172.21.2.102:0/3693640375 shutdown_connections 2021-02-10T11:25:54.504 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:54.488 7fca79a0b700 1 --2- 172.21.2.102:0/3693640375 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fca740d0330 0x7fca740ef630 unknown :-1 s=CLOSED pgs=1541 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:54.504 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:54.488 7fca79a0b700 1 --2- 172.21.2.102:0/3693640375 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fca740d1c60 0x7fca740efc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:54.504 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:54.488 7fca79a0b700 1 -- 172.21.2.102:0/3693640375 >> 172.21.2.102:0/3693640375 conn(0x7fca740ce440 msgr2=0x7fca740d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:25:54.504 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:54.488 7fca79a0b700 1 -- 172.21.2.102:0/3693640375 shutdown_connections 2021-02-10T11:25:54.504 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:54.488 7fca79a0b700 1 -- 172.21.2.102:0/3693640375 wait complete. 2021-02-10T11:25:54.505 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:54.503 7fca79a0b700 1 Processor -- start 2021-02-10T11:25:54.505 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:54.503 7fca79a0b700 1 -- start start 2021-02-10T11:25:54.505 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:54.503 7fca79a0b700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fca740acbb0 0x7fca740ac030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:54.505 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:54.503 7fca79a0b700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fca740ac650 0x7fca740ad080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:54.506 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:54.503 7fca79a0b700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fca740ad6a0 0x7fca740f19b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:54.506 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:54.503 7fca79a0b700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca740f1430 con 0x7fca740acbb0 2021-02-10T11:25:54.506 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:54.503 7fca79a0b700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca740adaa0 con 0x7fca740ad6a0 2021-02-10T11:25:54.506 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:54.503 7fca79a0b700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca740f1fd0 con 0x7fca740ac650 2021-02-10T11:25:54.506 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:54.503 7fca737fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fca740acbb0 0x7fca740ac030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:54.506 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:54.503 7fca737fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fca740acbb0 0x7fca740ac030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:58282/0 (socket says 172.21.2.102:58282) 2021-02-10T11:25:54.506 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:54.503 7fca737fe700 1 -- 172.21.2.102:0/970082372 learned_addr learned my addr 172.21.2.102:0/970082372 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:25:54.506 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:54.504 7fca737fe700 1 -- 172.21.2.102:0/970082372 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fca740ac650 msgr2=0x7fca740ad080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:54.506 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:54.504 7fca737fe700 1 --2- 172.21.2.102:0/970082372 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fca740ac650 0x7fca740ad080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:54.506 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:54.504 7fca737fe700 1 -- 172.21.2.102:0/970082372 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fca740ad6a0 msgr2=0x7fca740f19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:54.506 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:54.504 7fca737fe700 1 --2- 172.21.2.102:0/970082372 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fca740ad6a0 0x7fca740f19b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:54.506 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:54.504 7fca737fe700 1 -- 172.21.2.102:0/970082372 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca740f21f0 con 0x7fca740acbb0 2021-02-10T11:25:54.507 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:54.504 7fca737fe700 1 --2- 172.21.2.102:0/970082372 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fca740acbb0 0x7fca740ac030 secure :-1 s=READY pgs=1542 cs=0 l=1 rev1=1 rx=0x7fca5c001ea0 tx=0x7fca5c00a880).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:54.507 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:54.504 7fca6bfff700 1 -- 172.21.2.102:0/970082372 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fca5c00ac70 con 0x7fca740acbb0 2021-02-10T11:25:54.507 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:54.504 7fca6bfff700 1 -- 172.21.2.102:0/970082372 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca5c000f90 con 0x7fca740acbb0 2021-02-10T11:25:54.507 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:54.504 7fca6bfff700 1 -- 172.21.2.102:0/970082372 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fca5c0045b0 con 0x7fca740acbb0 2021-02-10T11:25:54.507 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:54.504 7fca79a0b700 1 -- 172.21.2.102:0/970082372 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca740f21f0 con 0x7fca740acbb0 2021-02-10T11:25:54.507 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:54.504 7fca79a0b700 1 -- 172.21.2.102:0/970082372 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca740f2780 con 0x7fca740acbb0 2021-02-10T11:25:54.507 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:54.505 7fca6bfff700 1 -- 172.21.2.102:0/970082372 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fca5c00ac70 con 0x7fca740acbb0 2021-02-10T11:25:54.507 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:54.505 7fca6bfff700 1 --2- 172.21.2.102:0/970082372 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fca580186c0 0x7fca5801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:54.507 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:54.505 7fca6bfff700 1 -- 172.21.2.102:0/970082372 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fca5c032400 con 0x7fca740acbb0 2021-02-10T11:25:54.507 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:54.505 7fca72ffd700 1 --2- 172.21.2.102:0/970082372 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fca580186c0 0x7fca5801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:54.507 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:54.505 7fca72ffd700 1 --2- 172.21.2.102:0/970082372 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fca580186c0 0x7fca5801abc0 crc :-1 s=READY pgs=702 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:54.536 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:54.536 7fca79a0b700 1 -- 172.21.2.102:0/970082372 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca740ee060 con 0x7fca740acbb0 2021-02-10T11:25:54.538 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:54.539 7fca6bfff700 1 -- 172.21.2.102:0/970082372 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fca5c0070d0 con 0x7fca740acbb0 2021-02-10T11:25:54.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:54.674 7fca79a0b700 1 -- 172.21.2.102:0/970082372 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs volume ls", "target": ["mgr", ""]}) v1 -- 0x7fca740f2780 con 0x7fca580186c0 2021-02-10T11:25:54.674 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:54.675 7fca6bfff700 1 -- 172.21.2.102:0/970082372 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+41 (crc 0 0 0) 0x7fca64006040 con 0x7fca580186c0 2021-02-10T11:25:54.674 INFO:teuthology.orchestra.run.gibba002.stdout:[ 2021-02-10T11:25:54.674 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:25:54.674 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "cephfs" 2021-02-10T11:25:54.674 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:25:54.674 INFO:teuthology.orchestra.run.gibba002.stdout:] 2021-02-10T11:25:54.675 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:54.676 7fca79a0b700 1 -- 172.21.2.102:0/970082372 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fca580186c0 msgr2=0x7fca5801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:54.675 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:54.676 7fca79a0b700 1 --2- 172.21.2.102:0/970082372 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fca580186c0 0x7fca5801abc0 crc :-1 s=READY pgs=702 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:54.675 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:54.676 7fca79a0b700 1 -- 172.21.2.102:0/970082372 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fca740acbb0 msgr2=0x7fca740ac030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:54.675 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:54.676 7fca79a0b700 1 --2- 172.21.2.102:0/970082372 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fca740acbb0 0x7fca740ac030 secure :-1 s=READY pgs=1542 cs=0 l=1 rev1=1 rx=0x7fca5c001ea0 tx=0x7fca5c00a880).stop 2021-02-10T11:25:54.676 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:54.676 7fca79a0b700 1 -- 172.21.2.102:0/970082372 shutdown_connections 2021-02-10T11:25:54.676 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:54.676 7fca79a0b700 1 --2- 172.21.2.102:0/970082372 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fca580186c0 0x7fca5801abc0 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:54.676 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:54.676 7fca79a0b700 1 --2- 172.21.2.102:0/970082372 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fca740ad6a0 0x7fca740f19b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:54.676 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:54.676 7fca79a0b700 1 --2- 172.21.2.102:0/970082372 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fca740ac650 0x7fca740ad080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:54.676 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:54.676 7fca79a0b700 1 --2- 172.21.2.102:0/970082372 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fca740acbb0 0x7fca740ac030 unknown :-1 s=CLOSED pgs=1542 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:54.676 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:54.676 7fca79a0b700 1 -- 172.21.2.102:0/970082372 >> 172.21.2.102:0/970082372 conn(0x7fca740bea70 msgr2=0x7fca74095700 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:25:54.676 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:54.676 7fca79a0b700 1 -- 172.21.2.102:0/970082372 shutdown_connections 2021-02-10T11:25:54.676 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:54.676 7fca79a0b700 1 -- 172.21.2.102:0/970082372 wait complete. 2021-02-10T11:25:54.685 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config set mon mon_allow_pool_delete True 2021-02-10T11:25:55.117 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:55.101 7f0cbd9ae700 1 Processor -- start 2021-02-10T11:25:55.117 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:55.101 7f0cbd9ae700 1 -- start start 2021-02-10T11:25:55.117 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:55.101 7f0cbd9ae700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f0cb80d1870 0x7f0cb81b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:55.117 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:55.101 7f0cbd9ae700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0cb81b0ae0 0x7f0cb8108940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:55.117 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:55.101 7f0cbd9ae700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f0cb80d1c70 0x7f0cb8108f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:55.117 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:55.101 7f0cbd9ae700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0cb80fe1c0 con 0x7f0cb80d1870 2021-02-10T11:25:55.117 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:55.101 7f0cbd9ae700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0cb8109a60 con 0x7f0cb80d1c70 2021-02-10T11:25:55.117 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:55.101 7f0cbd9ae700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cb8088070 con 0x7f0cb81b0ae0 2021-02-10T11:25:55.117 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:55.102 7f0cb7574700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f0cb80d1870 0x7f0cb81b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:55.117 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:55.102 7f0cb7574700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f0cb80d1870 0x7f0cb81b04c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:58290/0 (socket says 172.21.2.102:58290) 2021-02-10T11:25:55.118 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:55.102 7f0cb7574700 1 -- 172.21.2.102:0/3830107291 learned_addr learned my addr 172.21.2.102:0/3830107291 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:25:55.118 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:55.102 7f0cb7574700 1 -- 172.21.2.102:0/3830107291 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0cb81b0ae0 msgr2=0x7f0cb8108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:25:55.118 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:55.102 7f0cb7574700 1 --2- 172.21.2.102:0/3830107291 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0cb81b0ae0 0x7f0cb8108940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:55.118 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:55.102 7f0cb7574700 1 -- 172.21.2.102:0/3830107291 >> v2:172.21.2.113:3300/0 conn(0x7f0cb80d1c70 msgr2=0x7f0cb8108f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:25:55.118 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:55.102 7f0cb7574700 1 --2- 172.21.2.102:0/3830107291 >> v2:172.21.2.113:3300/0 conn(0x7f0cb80d1c70 0x7f0cb8108f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:55.118 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:55.102 7f0cb7574700 1 -- 172.21.2.102:0/3830107291 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cb8109580 con 0x7f0cb80d1870 2021-02-10T11:25:55.118 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:55.102 7f0cb7574700 1 --2- 172.21.2.102:0/3830107291 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0cb80d1870 0x7f0cb81b04c0 secure :-1 s=READY pgs=1543 cs=0 l=1 rev1=1 rx=0x7f0ca0006700 tx=0x7f0ca0009b80).ready entity=mon.0 client_cookie=c6715f0360675be8 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:55.118 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:55.102 7f0cb6572700 1 -- 172.21.2.102:0/3830107291 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0ca000c0c0 con 0x7f0cb80d1870 2021-02-10T11:25:55.118 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:55.102 7f0cb6572700 1 -- 172.21.2.102:0/3830107291 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ca000c3c0 con 0x7f0cb80d1870 2021-02-10T11:25:55.118 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:55.102 7f0cb6572700 1 -- 172.21.2.102:0/3830107291 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0ca000c8e0 con 0x7f0cb80d1870 2021-02-10T11:25:55.118 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:55.102 7f0cbd9ae700 1 -- 172.21.2.102:0/3830107291 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0cb80d1870 msgr2=0x7f0cb81b04c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:55.118 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:55.102 7f0cbd9ae700 1 --2- 172.21.2.102:0/3830107291 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0cb80d1870 0x7f0cb81b04c0 secure :-1 s=READY pgs=1543 cs=0 l=1 rev1=1 rx=0x7f0ca0006700 tx=0x7f0ca0009b80).stop 2021-02-10T11:25:55.119 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:55.103 7f0cbd9ae700 1 -- 172.21.2.102:0/3830107291 shutdown_connections 2021-02-10T11:25:55.119 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:55.103 7f0cbd9ae700 1 --2- 172.21.2.102:0/3830107291 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0cb81b0ae0 0x7f0cb8108940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:55.119 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:55.103 7f0cbd9ae700 1 --2- 172.21.2.102:0/3830107291 >> v2:172.21.2.113:3300/0 conn(0x7f0cb80d1c70 0x7f0cb8108f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:55.119 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:55.103 7f0cbd9ae700 1 --2- 172.21.2.102:0/3830107291 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0cb80d1870 0x7f0cb81b04c0 unknown :-1 s=CLOSED pgs=1543 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:55.119 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:55.103 7f0cbd9ae700 1 -- 172.21.2.102:0/3830107291 >> 172.21.2.102:0/3830107291 conn(0x7f0cb80ce420 msgr2=0x7f0cb80d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:25:55.119 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:55.103 7f0cbd9ae700 1 -- 172.21.2.102:0/3830107291 shutdown_connections 2021-02-10T11:25:55.119 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:55.103 7f0cbd9ae700 1 -- 172.21.2.102:0/3830107291 wait complete. 2021-02-10T11:25:55.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:55.118 7f0cbd9ae700 1 Processor -- start 2021-02-10T11:25:55.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:55.118 7f0cbd9ae700 1 -- start start 2021-02-10T11:25:55.121 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:55.118 7f0cbd9ae700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0cb80c3820 0x7f0cb8084cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:55.121 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:55.118 7f0cbd9ae700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0cb80a0290 0x7f0cb80852d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:55.121 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:55.118 7f0cbd9ae700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0cb8086270 0x7f0cb8086670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:55.121 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:55.118 7f0cbd9ae700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cb80c0700 con 0x7f0cb80c3820 2021-02-10T11:25:55.121 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:55.118 7f0cbd9ae700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cb80858f0 con 0x7f0cb80a0290 2021-02-10T11:25:55.121 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:55.118 7f0cbd9ae700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cb8085b10 con 0x7f0cb8086270 2021-02-10T11:25:55.121 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:55.118 7f0cb7574700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0cb80c3820 0x7f0cb8084cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:55.121 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:55.118 7f0cb7574700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0cb80c3820 0x7f0cb8084cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:58296/0 (socket says 172.21.2.102:58296) 2021-02-10T11:25:55.121 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:55.118 7f0cb7574700 1 -- 172.21.2.102:0/1919853624 learned_addr learned my addr 172.21.2.102:0/1919853624 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:25:55.121 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:55.119 7f0cb7574700 1 -- 172.21.2.102:0/1919853624 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0cb8086270 msgr2=0x7f0cb8086670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:55.121 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:55.119 7f0cb7574700 1 --2- 172.21.2.102:0/1919853624 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0cb8086270 0x7f0cb8086670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:55.121 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:55.119 7f0cb7574700 1 -- 172.21.2.102:0/1919853624 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0cb80a0290 msgr2=0x7f0cb80852d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:55.121 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:55.119 7f0cb7574700 1 --2- 172.21.2.102:0/1919853624 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0cb80a0290 0x7f0cb80852d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:55.122 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:55.119 7f0cb7574700 1 -- 172.21.2.102:0/1919853624 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cb8086c90 con 0x7f0cb80c3820 2021-02-10T11:25:55.122 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:55.119 7f0cb7574700 1 --2- 172.21.2.102:0/1919853624 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0cb80c3820 0x7f0cb8084cb0 secure :-1 s=READY pgs=1544 cs=0 l=1 rev1=1 rx=0x7f0ca000b650 tx=0x7f0ca000b9f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:55.122 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:55.119 7f0caffff700 1 -- 172.21.2.102:0/1919853624 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0ca000bde0 con 0x7f0cb80c3820 2021-02-10T11:25:55.122 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:55.119 7f0caffff700 1 -- 172.21.2.102:0/1919853624 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ca0006390 con 0x7f0cb80c3820 2021-02-10T11:25:55.122 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:55.119 7f0cbd9ae700 1 -- 172.21.2.102:0/1919853624 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0cb8086c90 con 0x7f0cb80c3820 2021-02-10T11:25:55.122 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:55.119 7f0cbd9ae700 1 -- 172.21.2.102:0/1919853624 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cb81a2f00 con 0x7f0cb80c3820 2021-02-10T11:25:55.122 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:55.119 7f0caffff700 1 -- 172.21.2.102:0/1919853624 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0ca00022b0 con 0x7f0cb80c3820 2021-02-10T11:25:55.122 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:55.120 7f0caffff700 1 -- 172.21.2.102:0/1919853624 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f0ca00052e0 con 0x7f0cb80c3820 2021-02-10T11:25:55.122 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:55.120 7f0caffff700 1 --2- 172.21.2.102:0/1919853624 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0c9c0186c0 0x7f0c9c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:55.122 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:55.120 7f0caffff700 1 -- 172.21.2.102:0/1919853624 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f0ca0032550 con 0x7f0cb80c3820 2021-02-10T11:25:55.122 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:55.120 7f0cb6d73700 1 --2- 172.21.2.102:0/1919853624 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0c9c0186c0 0x7f0c9c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:55.122 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:55.120 7f0cb6d73700 1 --2- 172.21.2.102:0/1919853624 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0c9c0186c0 0x7f0c9c01abc0 crc :-1 s=READY pgs=703 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:55.151 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:55.151 7f0cbd9ae700 1 -- 172.21.2.102:0/1919853624 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0cb8082a70 con 0x7f0cb80c3820 2021-02-10T11:25:55.153 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:55.154 7f0caffff700 1 -- 172.21.2.102:0/1919853624 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f0ca002cc10 con 0x7f0cb80c3820 2021-02-10T11:25:55.302 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:55.303 7f0cbd9ae700 1 -- 172.21.2.102:0/1919853624 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f0cb8082a70 con 0x7f0cb80c3820 2021-02-10T11:25:55.303 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:55.303 7f0caffff700 1 -- 172.21.2.102:0/1919853624 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v14)=0 v14) v1 ==== 122+0+0 (secure 0 0 0) 0x7f0ca002cc10 con 0x7f0cb80c3820 2021-02-10T11:25:55.304 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:55.305 7f0cbd9ae700 1 -- 172.21.2.102:0/1919853624 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0c9c0186c0 msgr2=0x7f0c9c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:55.305 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:55.305 7f0cbd9ae700 1 --2- 172.21.2.102:0/1919853624 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0c9c0186c0 0x7f0c9c01abc0 crc :-1 s=READY pgs=703 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:55.305 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:55.305 7f0cbd9ae700 1 -- 172.21.2.102:0/1919853624 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0cb80c3820 msgr2=0x7f0cb8084cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:55.305 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:55.305 7f0cbd9ae700 1 --2- 172.21.2.102:0/1919853624 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0cb80c3820 0x7f0cb8084cb0 secure :-1 s=READY pgs=1544 cs=0 l=1 rev1=1 rx=0x7f0ca000b650 tx=0x7f0ca000b9f0).stop 2021-02-10T11:25:55.305 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:55.305 7f0cbd9ae700 1 -- 172.21.2.102:0/1919853624 shutdown_connections 2021-02-10T11:25:55.305 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:55.305 7f0cbd9ae700 1 --2- 172.21.2.102:0/1919853624 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0cb8086270 0x7f0cb8086670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:55.305 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:55.305 7f0cbd9ae700 1 --2- 172.21.2.102:0/1919853624 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0c9c0186c0 0x7f0c9c01abc0 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:55.305 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:55.305 7f0cbd9ae700 1 --2- 172.21.2.102:0/1919853624 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0cb80a0290 0x7f0cb80852d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:55.305 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:55.305 7f0cbd9ae700 1 --2- 172.21.2.102:0/1919853624 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0cb80c3820 0x7f0cb8084cb0 unknown :-1 s=CLOSED pgs=1544 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:55.305 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:55.305 7f0cbd9ae700 1 -- 172.21.2.102:0/1919853624 >> 172.21.2.102:0/1919853624 conn(0x7f0cb80872c0 msgr2=0x7f0cb8095700 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:25:55.306 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:55.306 7f0cbd9ae700 1 -- 172.21.2.102:0/1919853624 shutdown_connections 2021-02-10T11:25:55.306 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:55.306 7f0cbd9ae700 1 -- 172.21.2.102:0/1919853624 wait complete. 2021-02-10T11:25:55.313 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:55.313 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:25:55.313 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:25:55.649 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:25:55.649 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2021-02-10T11:25:55.650 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@gibba002.front.sepia.ceph.com... 2021-02-10T11:25:55.650 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-02-10T11:25:55.650 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:25:55.999 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba002.stderr:ceph-fuse[159844]: fuse finished with error 0 and tester_r 0 2021-02-10T11:25:56.001 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:56.001 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:25:56.001 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:25:56.335 INFO:teuthology.orchestra.run.gibba002.stdout:ext2/ext3 2021-02-10T11:25:56.335 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2021-02-10T11:25:56.335 INFO:teuthology.orchestra.run:waiting for 900 2021-02-10T11:25:56.335 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:25:56.336 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:25:56.671 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolumegroup create cephfs group_0000000000664797 2021-02-10T11:25:57.100 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:57.084 7f332ccb3700 1 Processor -- start 2021-02-10T11:25:57.100 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:57.084 7f332ccb3700 1 -- start start 2021-02-10T11:25:57.100 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:57.085 7f332ccb3700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f33280d1870 0x7f33281b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:57.100 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:57.085 7f332ccb3700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f33280cff70 0x7f3328109130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:57.100 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:57.085 7f332ccb3700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33280fe1c0 con 0x7f33281b0ae0 2021-02-10T11:25:57.100 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:57.085 7f332ccb3700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3328109ad0 con 0x7f33280d1870 2021-02-10T11:25:57.100 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:57.085 7f332ccb3700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3328109750 con 0x7f33280cff70 2021-02-10T11:25:57.100 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:57.085 7f332605d700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f33281b0ae0 0x7f3328108940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:50528/0 (socket says 172.21.2.102:50528) 2021-02-10T11:25:57.100 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:57.085 7f332605d700 1 -- 172.21.2.102:0/1440546380 learned_addr learned my addr 172.21.2.102:0/1440546380 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:25:57.101 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:57.085 7f332685e700 1 --2- 172.21.2.102:0/1440546380 >> v2:172.21.2.102:3300/0 conn(0x7f33280d1870 0x7f33281b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:57.101 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:57.085 7f332685e700 1 -- 172.21.2.102:0/1440546380 >> v2:172.21.2.113:3300/0 conn(0x7f33280cff70 msgr2=0x7f3328109130 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:25:57.101 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:57.085 7f332685e700 1 --2- 172.21.2.102:0/1440546380 >> v2:172.21.2.113:3300/0 conn(0x7f33280cff70 0x7f3328109130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:57.101 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:57.085 7f332685e700 1 -- 172.21.2.102:0/1440546380 >> v1:172.21.2.102:6789/0 conn(0x7f33281b0ae0 legacy=0x7f3328108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:57.101 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:57.085 7f332685e700 1 -- 172.21.2.102:0/1440546380 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3328088070 con 0x7f33280d1870 2021-02-10T11:25:57.101 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:57.085 7f332585c700 1 -- 172.21.2.102:0/1440546380 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2470980004 0 0) 0x7f33100010b0 con 0x7f33281b0ae0 2021-02-10T11:25:57.101 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:57.085 7f332685e700 1 --2- 172.21.2.102:0/1440546380 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f33280d1870 0x7f33281b04c0 secure :-1 s=READY pgs=1546 cs=0 l=1 rev1=1 rx=0x7f3318006650 tx=0x7f3318009a80).ready entity=mon.0 client_cookie=c6fcc0a2dba63312 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:57.101 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:57.085 7f332585c700 1 -- 172.21.2.102:0/1440546380 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f33180009b0 con 0x7f33280d1870 2021-02-10T11:25:57.101 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:57.086 7f332585c700 1 -- 172.21.2.102:0/1440546380 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f331800c0c0 con 0x7f33280d1870 2021-02-10T11:25:57.101 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:57.086 7f332585c700 1 -- 172.21.2.102:0/1440546380 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f331800c520 con 0x7f33280d1870 2021-02-10T11:25:57.101 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:57.086 7f332ccb3700 1 -- 172.21.2.102:0/1440546380 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f33280d1870 msgr2=0x7f33281b04c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:57.101 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:57.086 7f332ccb3700 1 --2- 172.21.2.102:0/1440546380 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f33280d1870 0x7f33281b04c0 secure :-1 s=READY pgs=1546 cs=0 l=1 rev1=1 rx=0x7f3318006650 tx=0x7f3318009a80).stop 2021-02-10T11:25:57.101 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:57.086 7f332ccb3700 1 -- 172.21.2.102:0/1440546380 shutdown_connections 2021-02-10T11:25:57.102 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:57.086 7f332ccb3700 1 --2- 172.21.2.102:0/1440546380 >> v2:172.21.2.113:3300/0 conn(0x7f33280cff70 0x7f3328109130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:57.102 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:57.086 7f332ccb3700 1 --2- 172.21.2.102:0/1440546380 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f33280d1870 0x7f33281b04c0 unknown :-1 s=CLOSED pgs=1546 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:57.102 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:57.086 7f332ccb3700 1 -- 172.21.2.102:0/1440546380 >> 172.21.2.102:0/1440546380 conn(0x7f33280ce420 msgr2=0x7f33280d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:25:57.102 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:57.086 7f332ccb3700 1 -- 172.21.2.102:0/1440546380 shutdown_connections 2021-02-10T11:25:57.102 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:57.086 7f332ccb3700 1 -- 172.21.2.102:0/1440546380 wait complete. 2021-02-10T11:25:57.103 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:57.101 7f332ccb3700 1 Processor -- start 2021-02-10T11:25:57.103 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:57.101 7f332ccb3700 1 -- start start 2021-02-10T11:25:57.103 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:57.101 7f332ccb3700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f33280c3820 0x7f33280a7950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:57.103 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:57.101 7f332ccb3700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f33281a44c0 0x7f33281a48c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:57.104 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:57.101 7f332ccb3700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f33280863d0 0x7f33280867d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:57.104 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:57.101 7f332ccb3700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3328085ed0 con 0x7f33280863d0 2021-02-10T11:25:57.104 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:57.101 7f332ccb3700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33280c0740 con 0x7f33281a44c0 2021-02-10T11:25:57.104 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:57.101 7f332ccb3700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3328086df0 con 0x7f33280c3820 2021-02-10T11:25:57.104 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:57.101 7f332705f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f33280863d0 0x7f33280867d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:57.104 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:57.101 7f332705f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f33280863d0 0x7f33280867d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:58310/0 (socket says 172.21.2.102:58310) 2021-02-10T11:25:57.104 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:57.101 7f332705f700 1 -- 172.21.2.102:0/2234962576 learned_addr learned my addr 172.21.2.102:0/2234962576 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:25:57.104 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:57.102 7f332705f700 1 -- 172.21.2.102:0/2234962576 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f33280c3820 msgr2=0x7f33280a7950 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:25:57.104 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:57.102 7f332705f700 1 --2- 172.21.2.102:0/2234962576 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f33280c3820 0x7f33280a7950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:57.104 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:57.102 7f332705f700 1 -- 172.21.2.102:0/2234962576 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f33281a44c0 msgr2=0x7f33281a48c0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:25:57.104 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:57.102 7f332705f700 1 --2- 172.21.2.102:0/2234962576 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f33281a44c0 0x7f33281a48c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:57.104 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:57.102 7f332705f700 1 -- 172.21.2.102:0/2234962576 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3328087010 con 0x7f33280863d0 2021-02-10T11:25:57.104 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:57.102 7f332705f700 1 --2- 172.21.2.102:0/2234962576 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f33280863d0 0x7f33280867d0 secure :-1 s=READY pgs=1547 cs=0 l=1 rev1=1 rx=0x7f3320009960 tx=0x7f332000bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:57.104 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:57.102 7f331f7fe700 1 -- 172.21.2.102:0/2234962576 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3320003030 con 0x7f33280863d0 2021-02-10T11:25:57.105 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:57.102 7f331f7fe700 1 -- 172.21.2.102:0/2234962576 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3320001310 con 0x7f33280863d0 2021-02-10T11:25:57.105 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:57.102 7f331f7fe700 1 -- 172.21.2.102:0/2234962576 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3320001830 con 0x7f33280863d0 2021-02-10T11:25:57.105 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:57.102 7f332ccb3700 1 -- 172.21.2.102:0/2234962576 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3328087010 con 0x7f33280863d0 2021-02-10T11:25:57.105 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:57.102 7f332ccb3700 1 -- 172.21.2.102:0/2234962576 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3328087010 con 0x7f33280863d0 2021-02-10T11:25:57.105 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:57.103 7f331f7fe700 1 -- 172.21.2.102:0/2234962576 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f3320000d10 con 0x7f33280863d0 2021-02-10T11:25:57.105 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:57.103 7f331f7fe700 1 --2- 172.21.2.102:0/2234962576 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f330c018730 0x7f330c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:57.105 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:57.103 7f331f7fe700 1 -- 172.21.2.102:0/2234962576 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f3320032b40 con 0x7f33280863d0 2021-02-10T11:25:57.105 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:57.103 7f332685e700 1 --2- 172.21.2.102:0/2234962576 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f330c018730 0x7f330c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:57.105 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:57.103 7f332685e700 1 --2- 172.21.2.102:0/2234962576 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f330c018730 0x7f330c01abc0 crc :-1 s=READY pgs=704 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:57.134 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:57.135 7f332ccb3700 1 -- 172.21.2.102:0/2234962576 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33280fe1c0 con 0x7f33280863d0 2021-02-10T11:25:57.136 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:57.137 7f331f7fe700 1 -- 172.21.2.102:0/2234962576 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f332000be30 con 0x7f33280863d0 2021-02-10T11:25:57.265 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:57.265 7f332ccb3700 1 -- 172.21.2.102:0/2234962576 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "target": ["mgr", ""], "group_name": "group_0000000000664797"}) v1 -- 0x7f33280fe1c0 con 0x7f330c018730 2021-02-10T11:25:57.268 INFO:tasks.ceph.mgr.y.gibba002.stderr:2021-02-10 11:25:57.269 7fd94bc53700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:25:57.268 INFO:tasks.ceph.mgr.y.gibba002.stderr:2021-02-10 11:25:57.269 7fd94bc53700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:25:57.268 INFO:tasks.ceph.mgr.y.gibba002.stderr:2021-02-10 11:25:57.269 7fd94bc53700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:25:57.269 INFO:tasks.ceph.mgr.y.gibba002.stderr:2021-02-10 11:25:57.269 7fd94bc53700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:25:57.269 INFO:tasks.ceph.mgr.y.gibba002.stderr:2021-02-10 11:25:57.269 7fd94bc53700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:25:57.282 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:57.283 7f331f7fe700 1 -- 172.21.2.102:0/2234962576 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f3318000d10 con 0x7f330c018730 2021-02-10T11:25:57.298 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:57.298 7f332ccb3700 1 -- 172.21.2.102:0/2234962576 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f330c018730 msgr2=0x7f330c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:57.298 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:57.298 7f332ccb3700 1 --2- 172.21.2.102:0/2234962576 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f330c018730 0x7f330c01abc0 crc :-1 s=READY pgs=704 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:57.298 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:57.298 7f332ccb3700 1 -- 172.21.2.102:0/2234962576 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f33280863d0 msgr2=0x7f33280867d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:57.298 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:57.298 7f332ccb3700 1 --2- 172.21.2.102:0/2234962576 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f33280863d0 0x7f33280867d0 secure :-1 s=READY pgs=1547 cs=0 l=1 rev1=1 rx=0x7f3320009960 tx=0x7f332000bcf0).stop 2021-02-10T11:25:57.299 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:57.299 7f332ccb3700 1 -- 172.21.2.102:0/2234962576 shutdown_connections 2021-02-10T11:25:57.299 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:57.299 7f332ccb3700 1 --2- 172.21.2.102:0/2234962576 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f33280863d0 0x7f33280867d0 unknown :-1 s=CLOSED pgs=1547 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:57.299 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:57.299 7f332ccb3700 1 --2- 172.21.2.102:0/2234962576 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f330c018730 0x7f330c01abc0 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:57.299 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:57.299 7f332ccb3700 1 --2- 172.21.2.102:0/2234962576 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f33281a44c0 0x7f33281a48c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:57.299 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:57.299 7f332ccb3700 1 --2- 172.21.2.102:0/2234962576 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f33280c3820 0x7f33280a7950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:57.299 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:57.299 7f332ccb3700 1 -- 172.21.2.102:0/2234962576 >> 172.21.2.102:0/2234962576 conn(0x7f33280872c0 msgr2=0x7f3328095700 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:25:57.299 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:57.299 7f332ccb3700 1 -- 172.21.2.102:0/2234962576 shutdown_connections 2021-02-10T11:25:57.299 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:57.299 7f332ccb3700 1 -- 172.21.2.102:0/2234962576 wait complete. 2021-02-10T11:25:57.306 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000152661 --group_name group_0000000000664797 2021-02-10T11:25:57.739 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:57.723 7f56ae5e1700 1 Processor -- start 2021-02-10T11:25:57.740 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:57.723 7f56ae5e1700 1 -- start start 2021-02-10T11:25:57.740 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:57.723 7f56ae5e1700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f56a80d1870 0x7f56a81b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:57.740 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:57.723 7f56ae5e1700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f56a80cff70 0x7f56a8109130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:57.740 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:57.723 7f56ae5e1700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56a8109a60 con 0x7f56a81b0ae0 2021-02-10T11:25:57.740 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:57.723 7f56ae5e1700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56a8109750 con 0x7f56a80d1870 2021-02-10T11:25:57.740 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:57.723 7f56ae5e1700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56a8088070 con 0x7f56a80cff70 2021-02-10T11:25:57.740 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:57.724 7f56a77fe700 1 --1- >> v1:172.21.2.113:6789/0 conn(0x7f56a81b0ae0 0x7f56a8108940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.113:6789/0 says I am v1:172.21.2.102:57878/0 (socket says 172.21.2.102:57878) 2021-02-10T11:25:57.740 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:57.724 7f56a77fe700 1 -- 172.21.2.102:0/1258997283 learned_addr learned my addr 172.21.2.102:0/1258997283 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:25:57.740 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:57.724 7f56acbfe700 1 --2- 172.21.2.102:0/1258997283 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f56a80cff70 0x7f56a8109130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:57.740 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:57.724 7f56a7fff700 1 --2- 172.21.2.102:0/1258997283 >> v2:172.21.2.113:3300/0 conn(0x7f56a80d1870 0x7f56a81b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:57.740 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:57.725 7f56acbfe700 1 -- 172.21.2.102:0/1258997283 >> v2:172.21.2.113:3300/0 conn(0x7f56a80d1870 msgr2=0x7f56a81b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:25:57.740 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:57.725 7f56acbfe700 1 --2- 172.21.2.102:0/1258997283 >> v2:172.21.2.113:3300/0 conn(0x7f56a80d1870 0x7f56a81b04c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:57.741 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:57.725 7f56acbfe700 1 -- 172.21.2.102:0/1258997283 >> v1:172.21.2.113:6789/0 conn(0x7f56a81b0ae0 legacy=0x7f56a8108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:57.741 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:57.725 7f56acbfe700 1 -- 172.21.2.102:0/1258997283 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56a80d1c70 con 0x7f56a80cff70 2021-02-10T11:25:57.741 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:57.725 7f56acbfe700 1 --2- 172.21.2.102:0/1258997283 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f56a80cff70 0x7f56a8109130 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7f56a0006e50 tx=0x7f56a0009e70).ready entity=mon.2 client_cookie=9b5d806541bd78c0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:57.741 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:57.725 7f56a6ffd700 1 -- 172.21.2.102:0/1258997283 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f56a000b510 con 0x7f56a80cff70 2021-02-10T11:25:57.741 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:57.725 7f56a6ffd700 1 -- 172.21.2.102:0/1258997283 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56a0006790 con 0x7f56a80cff70 2021-02-10T11:25:57.741 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:57.725 7f56a6ffd700 1 -- 172.21.2.102:0/1258997283 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f56a000b940 con 0x7f56a80cff70 2021-02-10T11:25:57.741 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:57.725 7f56ae5e1700 1 -- 172.21.2.102:0/1258997283 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f56a80cff70 msgr2=0x7f56a8109130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:57.741 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:57.725 7f56ae5e1700 1 --2- 172.21.2.102:0/1258997283 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f56a80cff70 0x7f56a8109130 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7f56a0006e50 tx=0x7f56a0009e70).stop 2021-02-10T11:25:57.741 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:57.726 7f56ae5e1700 1 -- 172.21.2.102:0/1258997283 shutdown_connections 2021-02-10T11:25:57.741 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:57.726 7f56ae5e1700 1 --2- 172.21.2.102:0/1258997283 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f56a80cff70 0x7f56a8109130 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:57.741 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:57.726 7f56ae5e1700 1 --2- 172.21.2.102:0/1258997283 >> v2:172.21.2.113:3300/0 conn(0x7f56a80d1870 0x7f56a81b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:57.741 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:57.726 7f56ae5e1700 1 -- 172.21.2.102:0/1258997283 >> 172.21.2.102:0/1258997283 conn(0x7f56a80ce420 msgr2=0x7f56a80d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:25:57.741 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:57.726 7f56ae5e1700 1 -- 172.21.2.102:0/1258997283 shutdown_connections 2021-02-10T11:25:57.742 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:57.726 7f56ae5e1700 1 -- 172.21.2.102:0/1258997283 wait complete. 2021-02-10T11:25:57.743 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:57.740 7f56ae5e1700 1 Processor -- start 2021-02-10T11:25:57.743 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:57.741 7f56ae5e1700 1 -- start start 2021-02-10T11:25:57.743 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:57.741 7f56ae5e1700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f56a81a2a00 0x7f56a8084960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:57.743 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:57.741 7f56ae5e1700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f56a8084f80 0x7f56a80b1700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:57.743 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:57.741 7f56ae5e1700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f56a80b1d20 0x7f56a80b2120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:57.744 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:57.741 7f56ae5e1700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56a80853b0 con 0x7f56a81a2a00 2021-02-10T11:25:57.744 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:57.741 7f56ae5e1700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56a819e2b0 con 0x7f56a8084f80 2021-02-10T11:25:57.744 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:57.741 7f56ae5e1700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56a819e6e0 con 0x7f56a80b1d20 2021-02-10T11:25:57.744 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:57.741 7f56a7fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f56a81a2a00 0x7f56a8084960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:57.744 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:57.741 7f56a7fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f56a81a2a00 0x7f56a8084960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:58338/0 (socket says 172.21.2.102:58338) 2021-02-10T11:25:57.744 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:57.741 7f56a7fff700 1 -- 172.21.2.102:0/2330904355 learned_addr learned my addr 172.21.2.102:0/2330904355 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:25:57.744 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:57.741 7f56acbfe700 1 --2- 172.21.2.102:0/2330904355 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f56a80b1d20 0x7f56a80b2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:57.744 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:57.741 7f56a77fe700 1 --2- 172.21.2.102:0/2330904355 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f56a8084f80 0x7f56a80b1700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:57.744 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:57.742 7f56a7fff700 1 -- 172.21.2.102:0/2330904355 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f56a80b1d20 msgr2=0x7f56a80b2120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:57.744 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:57.742 7f56a7fff700 1 --2- 172.21.2.102:0/2330904355 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f56a80b1d20 0x7f56a80b2120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:57.744 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:57.742 7f56a7fff700 1 -- 172.21.2.102:0/2330904355 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f56a8084f80 msgr2=0x7f56a80b1700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:57.744 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:57.742 7f56a7fff700 1 --2- 172.21.2.102:0/2330904355 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f56a8084f80 0x7f56a80b1700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:57.745 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:57.742 7f56a7fff700 1 -- 172.21.2.102:0/2330904355 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56a8085930 con 0x7f56a81a2a00 2021-02-10T11:25:57.745 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:57.742 7f56acbfe700 1 --2- 172.21.2.102:0/2330904355 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f56a80b1d20 0x7f56a80b2120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:25:57.745 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:57.742 7f56a7fff700 1 --2- 172.21.2.102:0/2330904355 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f56a81a2a00 0x7f56a8084960 secure :-1 s=READY pgs=1550 cs=0 l=1 rev1=1 rx=0x7f5698006f60 tx=0x7f569800b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:57.745 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:57.742 7f56a4ff9700 1 -- 172.21.2.102:0/2330904355 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f569800bd60 con 0x7f56a81a2a00 2021-02-10T11:25:57.745 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:57.742 7f56a4ff9700 1 -- 172.21.2.102:0/2330904355 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5698000c90 con 0x7f56a81a2a00 2021-02-10T11:25:57.745 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:57.742 7f56a4ff9700 1 -- 172.21.2.102:0/2330904355 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f56980022b0 con 0x7f56a81a2a00 2021-02-10T11:25:57.745 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:57.742 7f56ae5e1700 1 -- 172.21.2.102:0/2330904355 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56a8085930 con 0x7f56a81a2a00 2021-02-10T11:25:57.745 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:57.742 7f56ae5e1700 1 -- 172.21.2.102:0/2330904355 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56a8085ec0 con 0x7f56a81a2a00 2021-02-10T11:25:57.745 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:57.743 7f56a4ff9700 1 -- 172.21.2.102:0/2330904355 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f5698006b00 con 0x7f56a81a2a00 2021-02-10T11:25:57.745 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:57.743 7f56a4ff9700 1 --2- 172.21.2.102:0/2330904355 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f56880186c0 0x7f568801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:57.745 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:57.743 7f56a4ff9700 1 -- 172.21.2.102:0/2330904355 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f56980327b0 con 0x7f56a81a2a00 2021-02-10T11:25:57.745 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:57.743 7f56a77fe700 1 --2- 172.21.2.102:0/2330904355 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f56880186c0 0x7f568801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:57.745 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:57.743 7f56a77fe700 1 --2- 172.21.2.102:0/2330904355 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f56880186c0 0x7f568801abc0 crc :-1 s=READY pgs=705 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:57.772 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:57.773 7f56ae5e1700 1 -- 172.21.2.102:0/2330904355 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56a80825c0 con 0x7f56a81a2a00 2021-02-10T11:25:57.775 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:57.775 7f56a4ff9700 1 -- 172.21.2.102:0/2330904355 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f5698000940 con 0x7f56a81a2a00 2021-02-10T11:25:57.901 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:57.901 7f56ae5e1700 1 -- 172.21.2.102:0/2330904355 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000152661", "target": ["mgr", ""], "group_name": "group_0000000000664797"}) v1 -- 0x7f56a8085ec0 con 0x7f56880186c0 2021-02-10T11:25:57.927 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:57.928 7f56a4ff9700 1 -- 172.21.2.102:0/2330904355 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f569c0014c0 con 0x7f56880186c0 2021-02-10T11:25:57.933 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:57.934 7f56ae5e1700 1 -- 172.21.2.102:0/2330904355 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f56880186c0 msgr2=0x7f568801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:57.934 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:57.934 7f56ae5e1700 1 --2- 172.21.2.102:0/2330904355 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f56880186c0 0x7f568801abc0 crc :-1 s=READY pgs=705 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:57.934 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:57.934 7f56ae5e1700 1 -- 172.21.2.102:0/2330904355 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f56a81a2a00 msgr2=0x7f56a8084960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:57.934 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:57.934 7f56ae5e1700 1 --2- 172.21.2.102:0/2330904355 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f56a81a2a00 0x7f56a8084960 secure :-1 s=READY pgs=1550 cs=0 l=1 rev1=1 rx=0x7f5698006f60 tx=0x7f569800b970).stop 2021-02-10T11:25:57.934 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:57.934 7f56ae5e1700 1 -- 172.21.2.102:0/2330904355 shutdown_connections 2021-02-10T11:25:57.934 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:57.934 7f56ae5e1700 1 --2- 172.21.2.102:0/2330904355 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f56a80b1d20 0x7f56a80b2120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:57.934 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:57.934 7f56ae5e1700 1 --2- 172.21.2.102:0/2330904355 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f56880186c0 0x7f568801abc0 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:57.934 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:57.934 7f56ae5e1700 1 --2- 172.21.2.102:0/2330904355 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f56a8084f80 0x7f56a80b1700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:57.934 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:57.934 7f56ae5e1700 1 --2- 172.21.2.102:0/2330904355 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f56a81a2a00 0x7f56a8084960 unknown :-1 s=CLOSED pgs=1550 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:57.934 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:57.934 7f56ae5e1700 1 -- 172.21.2.102:0/2330904355 >> 172.21.2.102:0/2330904355 conn(0x7f56a808dcf0 msgr2=0x7f56a808c930 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:25:57.935 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:57.934 7f56ae5e1700 1 -- 172.21.2.102:0/2330904355 shutdown_connections 2021-02-10T11:25:57.935 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:57.935 7f56ae5e1700 1 -- 172.21.2.102:0/2330904355 wait complete. 2021-02-10T11:25:57.943 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000152661 --group_name group_0000000000664797 2021-02-10T11:25:58.375 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:58.358 7ff7d8f07700 1 Processor -- start 2021-02-10T11:25:58.375 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:58.358 7ff7d8f07700 1 -- start start 2021-02-10T11:25:58.375 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:58.359 7ff7d8f07700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7ff7d40d1c80 0x7ff7d40cef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:58.375 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:58.359 7ff7d8f07700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7ff7d40cf590 0x7ff7d40cf990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:58.375 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:58.359 7ff7d8f07700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7d40fe1d0 con 0x7ff7d40d1880 2021-02-10T11:25:58.375 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:58.359 7ff7d8f07700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7d41b0cc0 con 0x7ff7d40d1c80 2021-02-10T11:25:58.375 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:58.359 7ff7d8f07700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7d4109a70 con 0x7ff7d40cf590 2021-02-10T11:25:58.375 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:58.359 7ff7d2312700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7ff7d40d1c80 0x7ff7d40cef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:58.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:58.359 7ff7d2312700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7ff7d40d1c80 0x7ff7d40cef70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:58344/0 (socket says 172.21.2.102:58344) 2021-02-10T11:25:58.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:58.359 7ff7d2312700 1 -- 172.21.2.102:0/3578238675 learned_addr learned my addr 172.21.2.102:0/3578238675 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:25:58.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:58.359 7ff7d2312700 1 -- 172.21.2.102:0/3578238675 >> v2:172.21.2.113:3300/0 conn(0x7ff7d40cf590 msgr2=0x7ff7d40cf990 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:25:58.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:58.359 7ff7d2312700 1 --2- 172.21.2.102:0/3578238675 >> v2:172.21.2.113:3300/0 conn(0x7ff7d40cf590 0x7ff7d40cf990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:58.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:58.359 7ff7d2312700 1 -- 172.21.2.102:0/3578238675 >> v1:172.21.2.113:6789/0 conn(0x7ff7d40d1880 legacy=0x7ff7d41b04d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:25:58.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:58.359 7ff7d2312700 1 -- 172.21.2.102:0/3578238675 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7d4088070 con 0x7ff7d40d1c80 2021-02-10T11:25:58.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:58.360 7ff7d2312700 1 --2- 172.21.2.102:0/3578238675 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff7d40d1c80 0x7ff7d40cef70 secure :-1 s=READY pgs=1551 cs=0 l=1 rev1=1 rx=0x7ff7c8006700 tx=0x7ff7c8009b80).ready entity=mon.0 client_cookie=6f4b710646a6b9c9 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:58.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:58.360 7ff7d1b11700 1 -- 172.21.2.102:0/3578238675 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff7c800c0c0 con 0x7ff7d40d1c80 2021-02-10T11:25:58.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:58.360 7ff7d1b11700 1 -- 172.21.2.102:0/3578238675 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7c800c440 con 0x7ff7d40d1c80 2021-02-10T11:25:58.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:58.360 7ff7d1b11700 1 -- 172.21.2.102:0/3578238675 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff7c800c920 con 0x7ff7d40d1c80 2021-02-10T11:25:58.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:58.360 7ff7d8f07700 1 -- 172.21.2.102:0/3578238675 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff7d40d1c80 msgr2=0x7ff7d40cef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:58.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:58.360 7ff7d8f07700 1 --2- 172.21.2.102:0/3578238675 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff7d40d1c80 0x7ff7d40cef70 secure :-1 s=READY pgs=1551 cs=0 l=1 rev1=1 rx=0x7ff7c8006700 tx=0x7ff7c8009b80).stop 2021-02-10T11:25:58.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:58.361 7ff7d8f07700 1 -- 172.21.2.102:0/3578238675 shutdown_connections 2021-02-10T11:25:58.377 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:58.361 7ff7d8f07700 1 --2- 172.21.2.102:0/3578238675 >> v2:172.21.2.113:3300/0 conn(0x7ff7d40cf590 0x7ff7d40cf990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:58.377 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:58.361 7ff7d8f07700 1 --2- 172.21.2.102:0/3578238675 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff7d40d1c80 0x7ff7d40cef70 unknown :-1 s=CLOSED pgs=1551 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:58.377 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:58.361 7ff7d8f07700 1 -- 172.21.2.102:0/3578238675 >> 172.21.2.102:0/3578238675 conn(0x7ff7d40ce430 msgr2=0x7ff7d40d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:25:58.377 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:58.361 7ff7d8f07700 1 -- 172.21.2.102:0/3578238675 shutdown_connections 2021-02-10T11:25:58.377 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:58.361 7ff7d8f07700 1 -- 172.21.2.102:0/3578238675 wait complete. 2021-02-10T11:25:58.378 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:58.376 7ff7d8f07700 1 Processor -- start 2021-02-10T11:25:58.379 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:58.376 7ff7d8f07700 1 -- start start 2021-02-10T11:25:58.379 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:58.376 7ff7d8f07700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff7d408fb40 0x7ff7d4195720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:58.379 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:58.376 7ff7d8f07700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff7d409b4f0 0x7ff7d409b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:58.379 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:58.376 7ff7d8f07700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff7d409bf10 0x7ff7d41940a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:58.379 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:58.376 7ff7d8f07700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7d41951a0 con 0x7ff7d409b4f0 2021-02-10T11:25:58.379 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:58.376 7ff7d8f07700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7d4094990 con 0x7ff7d409bf10 2021-02-10T11:25:58.379 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:58.376 7ff7d8f07700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7d41946c0 con 0x7ff7d408fb40 2021-02-10T11:25:58.379 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:58.377 7ff7d2312700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff7d409b4f0 0x7ff7d409b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:58.379 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:58.377 7ff7d2312700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff7d409b4f0 0x7ff7d409b8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:58352/0 (socket says 172.21.2.102:58352) 2021-02-10T11:25:58.379 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:58.377 7ff7d2312700 1 -- 172.21.2.102:0/1245591966 learned_addr learned my addr 172.21.2.102:0/1245591966 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:25:58.379 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:58.377 7ff7d2b13700 1 --2- 172.21.2.102:0/1245591966 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff7d408fb40 0x7ff7d4195720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:58.379 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:58.377 7ff7d3314700 1 --2- 172.21.2.102:0/1245591966 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff7d409bf10 0x7ff7d41940a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:58.380 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:58.377 7ff7d2312700 1 -- 172.21.2.102:0/1245591966 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff7d408fb40 msgr2=0x7ff7d4195720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:58.380 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:58.377 7ff7d2312700 1 --2- 172.21.2.102:0/1245591966 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff7d408fb40 0x7ff7d4195720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:58.380 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:58.377 7ff7d2312700 1 -- 172.21.2.102:0/1245591966 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff7d409bf10 msgr2=0x7ff7d41940a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:58.380 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:58.377 7ff7d2312700 1 --2- 172.21.2.102:0/1245591966 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff7d409bf10 0x7ff7d41940a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:58.380 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:58.377 7ff7d2312700 1 -- 172.21.2.102:0/1245591966 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7d41948e0 con 0x7ff7d409b4f0 2021-02-10T11:25:58.380 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:58.377 7ff7d3314700 1 --2- 172.21.2.102:0/1245591966 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff7d409bf10 0x7ff7d41940a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:25:58.380 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:58.377 7ff7d2312700 1 --2- 172.21.2.102:0/1245591966 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff7d409b4f0 0x7ff7d409b8f0 secure :-1 s=READY pgs=1552 cs=0 l=1 rev1=1 rx=0x7ff7c800b5e0 tx=0x7ff7c800b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:58.380 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:58.377 7ff7c37fe700 1 -- 172.21.2.102:0/1245591966 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff7c800bd70 con 0x7ff7d409b4f0 2021-02-10T11:25:58.380 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:58.377 7ff7c37fe700 1 -- 172.21.2.102:0/1245591966 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7c8002030 con 0x7ff7d409b4f0 2021-02-10T11:25:58.380 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:58.377 7ff7d8f07700 1 -- 172.21.2.102:0/1245591966 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7d41948e0 con 0x7ff7d409b4f0 2021-02-10T11:25:58.380 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:58.377 7ff7c37fe700 1 -- 172.21.2.102:0/1245591966 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff7c8002550 con 0x7ff7d409b4f0 2021-02-10T11:25:58.380 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:58.377 7ff7d8f07700 1 -- 172.21.2.102:0/1245591966 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7d41948e0 con 0x7ff7d409b4f0 2021-02-10T11:25:58.380 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:58.378 7ff7c37fe700 1 -- 172.21.2.102:0/1245591966 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ff7c8002210 con 0x7ff7d409b4f0 2021-02-10T11:25:58.381 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:58.378 7ff7c37fe700 1 --2- 172.21.2.102:0/1245591966 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff7b80186c0 0x7ff7b801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:58.381 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:58.378 7ff7c37fe700 1 -- 172.21.2.102:0/1245591966 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 5573+0+0 (secure 0 0 0) 0x7ff7c8032620 con 0x7ff7d409b4f0 2021-02-10T11:25:58.381 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:58.378 7ff7d2b13700 1 --2- 172.21.2.102:0/1245591966 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff7b80186c0 0x7ff7b801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:58.381 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:58.379 7ff7d2b13700 1 --2- 172.21.2.102:0/1245591966 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff7b80186c0 0x7ff7b801abc0 crc :-1 s=READY pgs=706 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:58.408 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:58.409 7ff7d8f07700 1 -- 172.21.2.102:0/1245591966 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7d4082ab0 con 0x7ff7d409b4f0 2021-02-10T11:25:58.410 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:58.412 7ff7c37fe700 1 -- 172.21.2.102:0/1245591966 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ff7c801f4e0 con 0x7ff7d409b4f0 2021-02-10T11:25:58.539 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:58.540 7ff7d8f07700 1 -- 172.21.2.102:0/1245591966 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000152661", "target": ["mgr", ""], "group_name": "group_0000000000664797"}) v1 -- 0x7ff7d408ff40 con 0x7ff7b80186c0 2021-02-10T11:25:58.542 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:58.544 7ff7c37fe700 1 -- 172.21.2.102:0/1245591966 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+96 (crc 0 0 0) 0x7ff7c4000c10 con 0x7ff7b80186c0 2021-02-10T11:25:58.546 INFO:teuthology.orchestra.run.gibba002.stdout:/volumes/group_0000000000664797/subvolume_0000000000152661/c517f318-1811-438b-85ff-c9e7d6aca87c 2021-02-10T11:25:58.548 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:58.549 7ff7d8f07700 1 -- 172.21.2.102:0/1245591966 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff7b80186c0 msgr2=0x7ff7b801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:58.548 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:58.549 7ff7d8f07700 1 --2- 172.21.2.102:0/1245591966 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff7b80186c0 0x7ff7b801abc0 crc :-1 s=READY pgs=706 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:58.548 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:58.549 7ff7d8f07700 1 -- 172.21.2.102:0/1245591966 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff7d409b4f0 msgr2=0x7ff7d409b8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:58.548 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:58.549 7ff7d8f07700 1 --2- 172.21.2.102:0/1245591966 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff7d409b4f0 0x7ff7d409b8f0 secure :-1 s=READY pgs=1552 cs=0 l=1 rev1=1 rx=0x7ff7c800b5e0 tx=0x7ff7c800b980).stop 2021-02-10T11:25:58.548 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:58.549 7ff7d8f07700 1 -- 172.21.2.102:0/1245591966 shutdown_connections 2021-02-10T11:25:58.548 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:58.549 7ff7d8f07700 1 --2- 172.21.2.102:0/1245591966 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff7b80186c0 0x7ff7b801abc0 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:58.548 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:58.549 7ff7d8f07700 1 --2- 172.21.2.102:0/1245591966 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff7d409bf10 0x7ff7d41940a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:58.548 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:58.549 7ff7d8f07700 1 --2- 172.21.2.102:0/1245591966 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff7d409b4f0 0x7ff7d409b8f0 unknown :-1 s=CLOSED pgs=1552 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:58.549 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:58.549 7ff7d8f07700 1 --2- 172.21.2.102:0/1245591966 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff7d408fb40 0x7ff7d4195720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:58.549 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:58.549 7ff7d8f07700 1 -- 172.21.2.102:0/1245591966 >> 172.21.2.102:0/1245591966 conn(0x7ff7d409afc0 msgr2=0x7ff7d40953e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:25:58.549 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:58.549 7ff7d8f07700 1 -- 172.21.2.102:0/1245591966 shutdown_connections 2021-02-10T11:25:58.549 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:58.549 7ff7d8f07700 1 -- 172.21.2.102:0/1245591966 wait complete. 2021-02-10T11:25:58.556 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume authorize cephfs subvolume_0000000000152661 alice --group_name group_0000000000664797 --tenant_id tenant_id 2021-02-10T11:25:58.996 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:58.979 7f10e031c700 1 Processor -- start 2021-02-10T11:25:58.996 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:58.979 7f10e031c700 1 -- start start 2021-02-10T11:25:58.996 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:58.980 7f10e031c700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f10d80e53a0 0x7f10d81b0550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:58.996 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:58.980 7f10e031c700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f10d81b0b70 0x7f10d81089d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:58.996 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:58.980 7f10e031c700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f10d8108ff0 0x7f10d81093f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:58.996 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:58.980 7f10e031c700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10d80e5f80 con 0x7f10d81b0b70 2021-02-10T11:25:58.996 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:58.980 7f10e031c700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10d8109ac0 con 0x7f10d80e53a0 2021-02-10T11:25:58.996 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:58.980 7f10e031c700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10d80e57a0 con 0x7f10d8108ff0 2021-02-10T11:25:58.997 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:58.980 7f10dd937700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f10d81b0b70 0x7f10d81089d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:58.997 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:58.980 7f10dd937700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f10d81b0b70 0x7f10d81089d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:58358/0 (socket says 172.21.2.102:58358) 2021-02-10T11:25:58.997 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:58.980 7f10dd937700 1 -- 172.21.2.102:0/759000102 learned_addr learned my addr 172.21.2.102:0/759000102 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:25:58.997 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:58.980 7f10dd937700 1 -- 172.21.2.102:0/759000102 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f10d8108ff0 msgr2=0x7f10d81093f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:25:58.997 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:58.981 7f10dd937700 1 --2- 172.21.2.102:0/759000102 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f10d8108ff0 0x7f10d81093f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:58.997 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:58.981 7f10dd937700 1 -- 172.21.2.102:0/759000102 >> v2:172.21.2.113:3300/0 conn(0x7f10d80e53a0 msgr2=0x7f10d81b0550 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:25:58.997 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:58.981 7f10dd937700 1 --2- 172.21.2.102:0/759000102 >> v2:172.21.2.113:3300/0 conn(0x7f10d80e53a0 0x7f10d81b0550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:58.997 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:58.981 7f10dd937700 1 -- 172.21.2.102:0/759000102 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10d8088070 con 0x7f10d81b0b70 2021-02-10T11:25:58.997 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:58.981 7f10dd937700 1 --2- 172.21.2.102:0/759000102 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f10d81b0b70 0x7f10d81089d0 secure :-1 s=READY pgs=1553 cs=0 l=1 rev1=1 rx=0x7f10cc006700 tx=0x7f10cc009b80).ready entity=mon.0 client_cookie=d2b4e1d5f7dd327b server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:58.997 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:58.981 7f10dd136700 1 -- 172.21.2.102:0/759000102 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f10cc00c0c0 con 0x7f10d81b0b70 2021-02-10T11:25:58.997 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:58.981 7f10dd136700 1 -- 172.21.2.102:0/759000102 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10cc00c440 con 0x7f10d81b0b70 2021-02-10T11:25:58.997 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:58.981 7f10dd136700 1 -- 172.21.2.102:0/759000102 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f10cc00c960 con 0x7f10d81b0b70 2021-02-10T11:25:58.997 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:58.981 7f10e031c700 1 -- 172.21.2.102:0/759000102 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f10d81b0b70 msgr2=0x7f10d81089d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:58.998 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:58.981 7f10e031c700 1 --2- 172.21.2.102:0/759000102 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f10d81b0b70 0x7f10d81089d0 secure :-1 s=READY pgs=1553 cs=0 l=1 rev1=1 rx=0x7f10cc006700 tx=0x7f10cc009b80).stop 2021-02-10T11:25:58.998 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:58.983 7f10e031c700 1 -- 172.21.2.102:0/759000102 shutdown_connections 2021-02-10T11:25:58.998 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:58.983 7f10e031c700 1 --2- 172.21.2.102:0/759000102 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f10d8108ff0 0x7f10d81093f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:58.998 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:58.983 7f10e031c700 1 --2- 172.21.2.102:0/759000102 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f10d81b0b70 0x7f10d81089d0 unknown :-1 s=CLOSED pgs=1553 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:58.998 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:58.983 7f10e031c700 1 --2- 172.21.2.102:0/759000102 >> v2:172.21.2.113:3300/0 conn(0x7f10d80e53a0 0x7f10d81b0550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:58.998 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:58.983 7f10e031c700 1 -- 172.21.2.102:0/759000102 >> 172.21.2.102:0/759000102 conn(0x7f10d80ce4b0 msgr2=0x7f10d80e45d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:25:58.998 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:58.983 7f10e031c700 1 -- 172.21.2.102:0/759000102 shutdown_connections 2021-02-10T11:25:58.998 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:58.983 7f10e031c700 1 -- 172.21.2.102:0/759000102 wait complete. 2021-02-10T11:25:58.998 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:58.996 7f10e031c700 1 Processor -- start 2021-02-10T11:25:58.998 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:58.996 7f10e031c700 1 -- start start 2021-02-10T11:25:58.999 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:58.997 7f10e031c700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f10d80b1f80 0x7f10d80b2380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:58.999 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:58.997 7f10e031c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f10d81a0000 0x7f10d81a0400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:59.000 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:58.997 7f10e031c700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f10d81a0a20 0x7f10d81a0e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:59.000 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:58.997 7f10e031c700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10d8094e30 con 0x7f10d81a0000 2021-02-10T11:25:59.000 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:58.997 7f10e031c700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10d81b8660 con 0x7f10d80b1f80 2021-02-10T11:25:59.000 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:58.997 7f10e031c700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10d8103a40 con 0x7f10d81a0a20 2021-02-10T11:25:59.000 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:58.997 7f10dd937700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f10d81a0000 0x7f10d81a0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:59.000 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:58.997 7f10dd937700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f10d81a0000 0x7f10d81a0400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:58366/0 (socket says 172.21.2.102:58366) 2021-02-10T11:25:59.000 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:58.997 7f10dd937700 1 -- 172.21.2.102:0/1707043118 learned_addr learned my addr 172.21.2.102:0/1707043118 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:25:59.000 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:58.997 7f10dd937700 1 -- 172.21.2.102:0/1707043118 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f10d81a0a20 msgr2=0x7f10d81a0e20 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:25:59.000 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:58.997 7f10dd937700 1 --2- 172.21.2.102:0/1707043118 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f10d81a0a20 0x7f10d81a0e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:59.000 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:58.997 7f10dd937700 1 -- 172.21.2.102:0/1707043118 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f10d80b1f80 msgr2=0x7f10d80b2380 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:25:59.000 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:58.997 7f10dd937700 1 --2- 172.21.2.102:0/1707043118 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f10d80b1f80 0x7f10d80b2380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:59.000 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:58.997 7f10dd937700 1 -- 172.21.2.102:0/1707043118 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10d8103c60 con 0x7f10d81a0000 2021-02-10T11:25:59.000 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:58.999 7f10dd937700 1 --2- 172.21.2.102:0/1707043118 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f10d81a0000 0x7f10d81a0400 secure :-1 s=READY pgs=1554 cs=0 l=1 rev1=1 rx=0x7f10cc00b5e0 tx=0x7f10cc00b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:59.001 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:58.999 7f10caffd700 1 -- 172.21.2.102:0/1707043118 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f10cc00bd70 con 0x7f10d81a0000 2021-02-10T11:25:59.001 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:58.999 7f10caffd700 1 -- 172.21.2.102:0/1707043118 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10cc002030 con 0x7f10d81a0000 2021-02-10T11:25:59.001 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:58.999 7f10caffd700 1 -- 172.21.2.102:0/1707043118 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f10cc002550 con 0x7f10d81a0000 2021-02-10T11:25:59.001 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:58.999 7f10e031c700 1 -- 172.21.2.102:0/1707043118 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10d8103c60 con 0x7f10d81a0000 2021-02-10T11:25:59.001 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:58.999 7f10e031c700 1 -- 172.21.2.102:0/1707043118 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10d81041f0 con 0x7f10d81a0000 2021-02-10T11:25:59.001 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:59.000 7f10caffd700 1 -- 172.21.2.102:0/1707043118 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f10cc000c90 con 0x7f10d81a0000 2021-02-10T11:25:59.001 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:59.000 7f10caffd700 1 --2- 172.21.2.102:0/1707043118 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f10c00186c0 0x7f10c001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:59.001 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:59.000 7f10caffd700 1 -- 172.21.2.102:0/1707043118 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f10cc032450 con 0x7f10d81a0000 2021-02-10T11:25:59.001 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:59.000 7f10de138700 1 --2- 172.21.2.102:0/1707043118 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f10c00186c0 0x7f10c001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:59.001 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:59.000 7f10de138700 1 --2- 172.21.2.102:0/1707043118 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f10c00186c0 0x7f10c001abc0 crc :-1 s=READY pgs=707 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:59.029 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:59.030 7f10e031c700 1 -- 172.21.2.102:0/1707043118 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10d8111fa0 con 0x7f10d81a0000 2021-02-10T11:25:59.032 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:59.033 7f10caffd700 1 -- 172.21.2.102:0/1707043118 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f10cc030d80 con 0x7f10d81a0000 2021-02-10T11:25:59.159 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:59.159 7f10e031c700 1 -- 172.21.2.102:0/1707043118 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume authorize", "vol_name": "cephfs", "sub_name": "subvolume_0000000000152661", "target": ["mgr", ""], "tenant_id": "tenant_id", "auth_id": "alice", "group_name": "group_0000000000664797"}) v1 -- 0x7f10d81b8660 con 0x7f10c00186c0 2021-02-10T11:25:59.164 INFO:tasks.ceph.mgr.y.gibba002.stderr:Creating meta for ID alice with tenant tenant_id 2021-02-10T11:25:59.191 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:59.192 7f10caffd700 1 -- 172.21.2.102:0/1707043118 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+41 (crc 0 0 0) 0x7f10d4006970 con 0x7f10c00186c0 2021-02-10T11:25:59.222 INFO:teuthology.orchestra.run.gibba002.stdout:AQDHwiNg5455ChAAnocKD3rX046PozdDDvByNA== 2021-02-10T11:25:59.224 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:59.226 7f10e031c700 1 -- 172.21.2.102:0/1707043118 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f10c00186c0 msgr2=0x7f10c001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:59.224 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:59.226 7f10e031c700 1 --2- 172.21.2.102:0/1707043118 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f10c00186c0 0x7f10c001abc0 crc :-1 s=READY pgs=707 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:59.224 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:59.226 7f10e031c700 1 -- 172.21.2.102:0/1707043118 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f10d81a0000 msgr2=0x7f10d81a0400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:59.224 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:59.226 7f10e031c700 1 --2- 172.21.2.102:0/1707043118 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f10d81a0000 0x7f10d81a0400 secure :-1 s=READY pgs=1554 cs=0 l=1 rev1=1 rx=0x7f10cc00b5e0 tx=0x7f10cc00b980).stop 2021-02-10T11:25:59.224 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:59.226 7f10e031c700 1 -- 172.21.2.102:0/1707043118 shutdown_connections 2021-02-10T11:25:59.224 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:59.226 7f10e031c700 1 --2- 172.21.2.102:0/1707043118 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f10d81a0a20 0x7f10d81a0e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:59.225 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:59.226 7f10e031c700 1 --2- 172.21.2.102:0/1707043118 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f10d81a0000 0x7f10d81a0400 unknown :-1 s=CLOSED pgs=1554 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:59.225 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:59.226 7f10e031c700 1 --2- 172.21.2.102:0/1707043118 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f10c00186c0 0x7f10c001abc0 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:59.225 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:59.226 7f10e031c700 1 --2- 172.21.2.102:0/1707043118 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f10d80b1f80 0x7f10d80b2380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:59.225 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:59.226 7f10e031c700 1 -- 172.21.2.102:0/1707043118 >> 172.21.2.102:0/1707043118 conn(0x7f10d81a38a0 msgr2=0x7f10d819dc20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:25:59.225 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:59.226 7f10e031c700 1 -- 172.21.2.102:0/1707043118 shutdown_connections 2021-02-10T11:25:59.225 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:59.226 7f10e031c700 1 -- 172.21.2.102:0/1707043118 wait complete. 2021-02-10T11:25:59.233 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth ls --format=json-pretty 2021-02-10T11:25:59.660 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:59.645 7f5e1123d700 1 Processor -- start 2021-02-10T11:25:59.660 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:59.645 7f5e1123d700 1 -- start start 2021-02-10T11:25:59.660 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:59.646 7f5e1123d700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f5e0c0d1870 0x7f5e0c1b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:59.660 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:59.646 7f5e1123d700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f5e0c1b0a10 0x7f5e0c0cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:59.660 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:59.646 7f5e1123d700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e0c0f0b10 con 0x7f5e0c0d1c70 2021-02-10T11:25:59.660 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:59.646 7f5e1123d700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e0c0e5ed0 con 0x7f5e0c0d1870 2021-02-10T11:25:59.660 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:59.646 7f5e1123d700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e0c0f0650 con 0x7f5e0c1b0a10 2021-02-10T11:25:59.660 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:59.646 7f5e0ad9d700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f5e0c0d1870 0x7f5e0c1b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:59.660 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:59.646 7f5e0b59e700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f5e0c0d1c70 0x7f5e0c0cf950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:50600/0 (socket says 172.21.2.102:50600) 2021-02-10T11:25:59.660 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:59.646 7f5e0b59e700 1 -- 172.21.2.102:0/2421390945 learned_addr learned my addr 172.21.2.102:0/2421390945 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:25:59.661 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:59.646 7f5e0ad9d700 1 -- 172.21.2.102:0/2421390945 >> v2:172.21.2.113:3300/0 conn(0x7f5e0c1b0a10 msgr2=0x7f5e0c0cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:25:59.661 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:59.646 7f5e0ad9d700 1 --2- 172.21.2.102:0/2421390945 >> v2:172.21.2.113:3300/0 conn(0x7f5e0c1b0a10 0x7f5e0c0cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:59.661 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:59.646 7f5e0ad9d700 1 -- 172.21.2.102:0/2421390945 >> v1:172.21.2.102:6789/0 conn(0x7f5e0c0d1c70 legacy=0x7f5e0c0cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:59.661 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:59.646 7f5e0ad9d700 1 -- 172.21.2.102:0/2421390945 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e0c0f0840 con 0x7f5e0c0d1870 2021-02-10T11:25:59.661 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:59.646 7f5e0ad9d700 1 --2- 172.21.2.102:0/2421390945 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5e0c0d1870 0x7f5e0c1b03f0 secure :-1 s=READY pgs=1556 cs=0 l=1 rev1=1 rx=0x7f5dfc006650 tx=0x7f5dfc009a80).ready entity=mon.0 client_cookie=fd263b3b3d606781 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:59.661 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:59.646 7f5e09d9b700 1 -- 172.21.2.102:0/2421390945 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5dfc0009b0 con 0x7f5e0c0d1870 2021-02-10T11:25:59.661 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:59.646 7f5e09d9b700 1 -- 172.21.2.102:0/2421390945 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5dfc00c0c0 con 0x7f5e0c0d1870 2021-02-10T11:25:59.661 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:59.646 7f5e09d9b700 1 -- 172.21.2.102:0/2421390945 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5dfc00c510 con 0x7f5e0c0d1870 2021-02-10T11:25:59.661 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:59.647 7f5e1123d700 1 -- 172.21.2.102:0/2421390945 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5e0c0d1870 msgr2=0x7f5e0c1b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:59.661 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:59.647 7f5e1123d700 1 --2- 172.21.2.102:0/2421390945 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5e0c0d1870 0x7f5e0c1b03f0 secure :-1 s=READY pgs=1556 cs=0 l=1 rev1=1 rx=0x7f5dfc006650 tx=0x7f5dfc009a80).stop 2021-02-10T11:25:59.661 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:59.647 7f5e1123d700 1 -- 172.21.2.102:0/2421390945 shutdown_connections 2021-02-10T11:25:59.661 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:59.647 7f5e1123d700 1 --2- 172.21.2.102:0/2421390945 >> v2:172.21.2.113:3300/0 conn(0x7f5e0c1b0a10 0x7f5e0c0cf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:59.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:59.647 7f5e1123d700 1 --2- 172.21.2.102:0/2421390945 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5e0c0d1870 0x7f5e0c1b03f0 unknown :-1 s=CLOSED pgs=1556 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:59.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:59.647 7f5e1123d700 1 -- 172.21.2.102:0/2421390945 >> 172.21.2.102:0/2421390945 conn(0x7f5e0c0ce450 msgr2=0x7f5e0c0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:25:59.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:59.647 7f5e1123d700 1 -- 172.21.2.102:0/2421390945 shutdown_connections 2021-02-10T11:25:59.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:59.647 7f5e1123d700 1 -- 172.21.2.102:0/2421390945 wait complete. 2021-02-10T11:25:59.663 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:59.662 7f5e1123d700 1 Processor -- start 2021-02-10T11:25:59.663 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:59.662 7f5e1123d700 1 -- start start 2021-02-10T11:25:59.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:59.662 7f5e1123d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5e0c0b0ad0 0x7f5e0c091f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:59.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:59.662 7f5e1123d700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5e0c1a8850 0x7f5e0c1a8c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:59.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:59.662 7f5e1123d700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5e0c1a6fa0 0x7f5e0c1a73a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:59.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:59.662 7f5e1123d700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e0c1abfe0 con 0x7f5e0c0b0ad0 2021-02-10T11:25:59.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:59.662 7f5e1123d700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e0c1a9c40 con 0x7f5e0c1a6fa0 2021-02-10T11:25:59.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:59.662 7f5e1123d700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e0c1a9730 con 0x7f5e0c1a8850 2021-02-10T11:25:59.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:59.662 7f5e0ad9d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5e0c0b0ad0 0x7f5e0c091f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:59.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:59.662 7f5e0ad9d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5e0c0b0ad0 0x7f5e0c091f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:58378/0 (socket says 172.21.2.102:58378) 2021-02-10T11:25:59.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:59.662 7f5e0ad9d700 1 -- 172.21.2.102:0/1386523544 learned_addr learned my addr 172.21.2.102:0/1386523544 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:25:59.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:59.663 7f5e0ad9d700 1 -- 172.21.2.102:0/1386523544 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5e0c1a8850 msgr2=0x7f5e0c1a8c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:59.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:59.663 7f5e0ad9d700 1 --2- 172.21.2.102:0/1386523544 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5e0c1a8850 0x7f5e0c1a8c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:59.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:59.663 7f5e0ad9d700 1 -- 172.21.2.102:0/1386523544 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5e0c1a6fa0 msgr2=0x7f5e0c1a73a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:59.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:59.663 7f5e0ad9d700 1 --2- 172.21.2.102:0/1386523544 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5e0c1a6fa0 0x7f5e0c1a73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:59.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:59.663 7f5e0ad9d700 1 -- 172.21.2.102:0/1386523544 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e0c1a9270 con 0x7f5e0c0b0ad0 2021-02-10T11:25:59.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:59.663 7f5e0ad9d700 1 --2- 172.21.2.102:0/1386523544 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5e0c0b0ad0 0x7f5e0c091f30 secure :-1 s=READY pgs=1557 cs=0 l=1 rev1=1 rx=0x7f5dfc009630 tx=0x7f5dfc00b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:59.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:59.663 7f5e037fe700 1 -- 172.21.2.102:0/1386523544 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5dfc0093a0 con 0x7f5e0c0b0ad0 2021-02-10T11:25:59.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:59.663 7f5e037fe700 1 -- 172.21.2.102:0/1386523544 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5dfc00bd70 con 0x7f5e0c0b0ad0 2021-02-10T11:25:59.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:59.663 7f5e1123d700 1 -- 172.21.2.102:0/1386523544 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e0c1a9270 con 0x7f5e0c0b0ad0 2021-02-10T11:25:59.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:59.663 7f5e1123d700 1 -- 172.21.2.102:0/1386523544 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e0c1a7cb0 con 0x7f5e0c0b0ad0 2021-02-10T11:25:59.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:59.663 7f5e037fe700 1 -- 172.21.2.102:0/1386523544 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5dfc005560 con 0x7f5e0c0b0ad0 2021-02-10T11:25:59.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:59.664 7f5e037fe700 1 -- 172.21.2.102:0/1386523544 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f5dfc00a000 con 0x7f5e0c0b0ad0 2021-02-10T11:25:59.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:59.664 7f5e037fe700 1 --2- 172.21.2.102:0/1386523544 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5df0018730 0x7f5df001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:59.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:59.664 7f5e037fe700 1 -- 172.21.2.102:0/1386523544 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f5dfc0327a0 con 0x7f5e0c0b0ad0 2021-02-10T11:25:59.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:59.664 7f5e0a59c700 1 --2- 172.21.2.102:0/1386523544 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5df0018730 0x7f5df001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:59.666 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:59.664 7f5e0a59c700 1 --2- 172.21.2.102:0/1386523544 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5df0018730 0x7f5df001ac10 crc :-1 s=READY pgs=708 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:59.694 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:59.696 7f5e1123d700 1 -- 172.21.2.102:0/1386523544 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e0c0ce450 con 0x7f5e0c0b0ad0 2021-02-10T11:25:59.696 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:59.698 7f5e037fe700 1 -- 172.21.2.102:0/1386523544 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f5dfc009340 con 0x7f5e0c0b0ad0 2021-02-10T11:25:59.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:59.832 7f5e1123d700 1 -- 172.21.2.102:0/1386523544 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f5e0c0ce450 con 0x7f5e0c0b0ad0 2021-02-10T11:25:59.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:59.833 7f5e037fe700 1 -- 172.21.2.102:0/1386523544 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v40) v1 ==== 80+0+4892 (secure 0 0 0) 0x7f5dfc01f4e0 con 0x7f5e0c0b0ad0 2021-02-10T11:25:59.832 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:25:59.832 INFO:teuthology.orchestra.run.gibba002.stdout:{ 2021-02-10T11:25:59.832 INFO:teuthology.orchestra.run.gibba002.stdout: "auth_dump": [ 2021-02-10T11:25:59.832 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:25:59.832 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "mds.a", 2021-02-10T11:25:59.832 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAevSNgsgyEARAAvr0cUjjCoEHLxaZSZ2HtVA==", 2021-02-10T11:25:59.832 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:25:59.832 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow", 2021-02-10T11:25:59.833 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:25:59.833 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:25:59.833 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:25:59.833 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:25:59.833 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:25:59.833 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:25:59.833 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "mds.b", 2021-02-10T11:25:59.833 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAevSNgwOFMMRAALfb1fbX56ao1TU8ClnTW1A==", 2021-02-10T11:25:59.833 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:25:59.833 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow", 2021-02-10T11:25:59.833 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:25:59.833 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:25:59.833 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:25:59.834 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:25:59.834 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:25:59.834 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:25:59.834 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.0", 2021-02-10T11:25:59.834 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAmvSNgj+bMMBAAD3HUD6fLiMAg+++iNzLJKQ==", 2021-02-10T11:25:59.834 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:25:59.834 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:25:59.834 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:25:59.834 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:25:59.834 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:25:59.834 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:25:59.834 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:25:59.834 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.1", 2021-02-10T11:25:59.835 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQApvSNgnvc+MRAAAOxqlNwiyLHqNITOrYeFKQ==", 2021-02-10T11:25:59.835 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:25:59.835 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:25:59.835 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:25:59.835 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:25:59.835 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:25:59.835 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:25:59.835 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:25:59.835 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.2", 2021-02-10T11:25:59.835 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAsvSNgZ4SbMRAA76IOYWfbiAuBoM/4k81GxA==", 2021-02-10T11:25:59.835 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:25:59.835 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:25:59.835 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:25:59.835 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:25:59.836 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:25:59.836 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:25:59.836 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:25:59.836 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.3", 2021-02-10T11:25:59.836 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAvvSNgFk3rMRAAWCmKKyw0LDQ+UlAHZAITOw==", 2021-02-10T11:25:59.836 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:25:59.836 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:25:59.836 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:25:59.836 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:25:59.836 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:25:59.836 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:25:59.836 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:25:59.836 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.4", 2021-02-10T11:25:59.837 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQA4vSNgpKcNJBAApJMmvj5YxVHPDOmvmQONOg==", 2021-02-10T11:25:59.837 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:25:59.837 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:25:59.837 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:25:59.837 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:25:59.837 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:25:59.837 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:25:59.837 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:25:59.837 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.5", 2021-02-10T11:25:59.837 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQA7vSNgf2ujJBAAfkuYpyb3WTzc8V5kw+Wktg==", 2021-02-10T11:25:59.837 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:25:59.837 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:25:59.837 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:25:59.837 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:25:59.838 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:25:59.838 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:25:59.838 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:25:59.838 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.6", 2021-02-10T11:25:59.838 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQA+vSNgqw9uJRAAFMU0R+UxU9C3o/9byb3e0g==", 2021-02-10T11:25:59.838 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:25:59.838 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:25:59.838 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:25:59.838 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:25:59.838 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:25:59.838 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:25:59.838 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:25:59.838 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.7", 2021-02-10T11:25:59.839 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQBBvSNgvWSrJRAA2KaCHmLpzovRgYzLvyofTQ==", 2021-02-10T11:25:59.839 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:25:59.839 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:25:59.839 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:25:59.839 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:25:59.839 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:25:59.839 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:25:59.839 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:25:59.839 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "client.0", 2021-02-10T11:25:59.839 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAfvSNgZd9XIBAAvyrvl8fdKuHnJZfczfRoFw==", 2021-02-10T11:25:59.839 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:25:59.839 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow", 2021-02-10T11:25:59.839 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow r", 2021-02-10T11:25:59.839 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:25:59.840 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:25:59.840 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:25:59.840 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:25:59.840 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "client.1", 2021-02-10T11:25:59.840 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAfvSNgSyfXNhAAIduPnF7Vd/6HSkJ0MlcMSA==", 2021-02-10T11:25:59.840 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:25:59.840 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow", 2021-02-10T11:25:59.840 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow r", 2021-02-10T11:25:59.840 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:25:59.840 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:25:59.840 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:25:59.840 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:25:59.840 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "client.2", 2021-02-10T11:25:59.840 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAgvSNgiIG/ERAA9ZTamaTI7ZsPSnxBqjplCg==", 2021-02-10T11:25:59.841 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:25:59.841 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow", 2021-02-10T11:25:59.841 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow r", 2021-02-10T11:25:59.841 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:25:59.841 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:25:59.841 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:25:59.841 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:25:59.841 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "client.3", 2021-02-10T11:25:59.841 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAgvSNgOWpOKBAAlXsEB04Q/GvtL06PqRMLeA==", 2021-02-10T11:25:59.841 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:25:59.841 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow", 2021-02-10T11:25:59.841 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow r", 2021-02-10T11:25:59.842 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:25:59.842 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:25:59.842 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:25:59.842 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:25:59.842 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "client.admin", 2021-02-10T11:25:59.842 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAavSNgO3GQBhAAL+NVnYskQRKVWLgyDPY/xw==", 2021-02-10T11:25:59.842 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:25:59.842 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow *", 2021-02-10T11:25:59.842 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:25:59.842 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:25:59.842 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:25:59.842 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:25:59.843 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:25:59.843 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:25:59.843 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "client.alice", 2021-02-10T11:25:59.843 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQDHwiNg5455ChAAnocKD3rX046PozdDDvByNA==", 2021-02-10T11:25:59.843 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:25:59.843 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow rw path=/volumes/group_0000000000664797/subvolume_0000000000152661/c517f318-1811-438b-85ff-c9e7d6aca87c", 2021-02-10T11:25:59.843 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow r", 2021-02-10T11:25:59.843 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:25:59.843 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:25:59.843 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:25:59.843 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:25:59.843 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "mgr.x", 2021-02-10T11:25:59.843 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAdvSNgXZZgJhAA7dtBSGzH+DsBwnddZpOBJA==", 2021-02-10T11:25:59.843 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:25:59.844 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow *", 2021-02-10T11:25:59.844 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow profile mgr", 2021-02-10T11:25:59.844 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:25:59.844 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:25:59.844 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:25:59.844 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:25:59.844 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "mgr.y", 2021-02-10T11:25:59.844 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAdvSNgVqQFDRAAwo6YEdWZCAr5GdOWVL/Qvg==", 2021-02-10T11:25:59.844 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:25:59.844 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow *", 2021-02-10T11:25:59.844 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow profile mgr", 2021-02-10T11:25:59.844 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:25:59.844 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:25:59.844 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:25:59.845 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:25:59.845 INFO:teuthology.orchestra.run.gibba002.stdout:} 2021-02-10T11:25:59.846 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:59.835 7f5e1123d700 1 -- 172.21.2.102:0/1386523544 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5df0018730 msgr2=0x7f5df001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:59.846 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:59.835 7f5e1123d700 1 --2- 172.21.2.102:0/1386523544 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5df0018730 0x7f5df001ac10 crc :-1 s=READY pgs=708 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:59.846 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:59.835 7f5e1123d700 1 -- 172.21.2.102:0/1386523544 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5e0c0b0ad0 msgr2=0x7f5e0c091f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:59.846 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:59.835 7f5e1123d700 1 --2- 172.21.2.102:0/1386523544 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5e0c0b0ad0 0x7f5e0c091f30 secure :-1 s=READY pgs=1557 cs=0 l=1 rev1=1 rx=0x7f5dfc009630 tx=0x7f5dfc00b980).stop 2021-02-10T11:25:59.846 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:59.835 7f5e1123d700 1 -- 172.21.2.102:0/1386523544 shutdown_connections 2021-02-10T11:25:59.846 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:59.835 7f5e1123d700 1 --2- 172.21.2.102:0/1386523544 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5df0018730 0x7f5df001ac10 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:59.846 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:59.835 7f5e1123d700 1 --2- 172.21.2.102:0/1386523544 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5e0c1a6fa0 0x7f5e0c1a73a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:59.846 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:59.835 7f5e1123d700 1 --2- 172.21.2.102:0/1386523544 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5e0c1a8850 0x7f5e0c1a8c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:59.846 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:59.835 7f5e1123d700 1 --2- 172.21.2.102:0/1386523544 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5e0c0b0ad0 0x7f5e0c091f30 unknown :-1 s=CLOSED pgs=1557 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:59.846 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:59.835 7f5e1123d700 1 -- 172.21.2.102:0/1386523544 >> 172.21.2.102:0/1386523544 conn(0x7f5e0c19f8f0 msgr2=0x7f5e0c1a5f80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:25:59.847 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:59.835 7f5e1123d700 1 -- 172.21.2.102:0/1386523544 shutdown_connections 2021-02-10T11:25:59.847 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:25:59.835 7f5e1123d700 1 -- 172.21.2.102:0/1386523544 wait complete. 2021-02-10T11:25:59.847 DEBUG:teuthology.orchestra.run.gibba002:> set -ex 2021-02-10T11:25:59.847 DEBUG:teuthology.orchestra.run.gibba002:> sudo dd of=/etc/ceph/ceph.client.alice.keyring 2021-02-10T11:26:00.191 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config set client.alice debug_client 20 2021-02-10T11:26:00.621 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:00.607 7f97f30ba700 1 Processor -- start 2021-02-10T11:26:00.621 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:00.607 7f97f30ba700 1 -- start start 2021-02-10T11:26:00.621 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:00.607 7f97f30ba700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f97ec0d1870 0x7f97ec1b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:00.622 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:00.607 7f97f30ba700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f97ec1b0ae0 0x7f97ec108940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:00.622 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:00.607 7f97f30ba700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f97ec0d1c70 0x7f97ec108f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:00.622 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:00.607 7f97f30ba700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97ec0fe1c0 con 0x7f97ec0d1870 2021-02-10T11:26:00.622 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:00.607 7f97f30ba700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97ec109a60 con 0x7f97ec0d1c70 2021-02-10T11:26:00.622 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:00.607 7f97f30ba700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97ec088070 con 0x7f97ec1b0ae0 2021-02-10T11:26:00.622 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:00.608 7f97f0ed6700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f97ec0d1870 0x7f97ec1b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:00.622 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:00.608 7f97f0ed6700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f97ec0d1870 0x7f97ec1b04c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:58388/0 (socket says 172.21.2.102:58388) 2021-02-10T11:26:00.622 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:00.608 7f97f0ed6700 1 -- 172.21.2.102:0/3803241763 learned_addr learned my addr 172.21.2.102:0/3803241763 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:26:00.623 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:00.608 7f97f0ed6700 1 -- 172.21.2.102:0/3803241763 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f97ec1b0ae0 msgr2=0x7f97ec108940 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:26:00.623 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:00.608 7f97f0ed6700 1 --2- 172.21.2.102:0/3803241763 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f97ec1b0ae0 0x7f97ec108940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:00.623 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:00.608 7f97f0ed6700 1 -- 172.21.2.102:0/3803241763 >> v2:172.21.2.113:3300/0 conn(0x7f97ec0d1c70 msgr2=0x7f97ec108f60 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:26:00.623 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:00.608 7f97f0ed6700 1 --2- 172.21.2.102:0/3803241763 >> v2:172.21.2.113:3300/0 conn(0x7f97ec0d1c70 0x7f97ec108f60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:00.623 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:00.608 7f97f0ed6700 1 -- 172.21.2.102:0/3803241763 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97ec109580 con 0x7f97ec0d1870 2021-02-10T11:26:00.623 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:00.608 7f97f0ed6700 1 --2- 172.21.2.102:0/3803241763 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f97ec0d1870 0x7f97ec1b04c0 secure :-1 s=READY pgs=1558 cs=0 l=1 rev1=1 rx=0x7f97d4006700 tx=0x7f97d4009b80).ready entity=mon.0 client_cookie=3b13912616ef6887 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:00.623 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:00.608 7f97eb7fe700 1 -- 172.21.2.102:0/3803241763 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f97d400c0c0 con 0x7f97ec0d1870 2021-02-10T11:26:00.623 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:00.608 7f97eb7fe700 1 -- 172.21.2.102:0/3803241763 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97d400c440 con 0x7f97ec0d1870 2021-02-10T11:26:00.623 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:00.608 7f97eb7fe700 1 -- 172.21.2.102:0/3803241763 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f97d400c920 con 0x7f97ec0d1870 2021-02-10T11:26:00.623 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:00.608 7f97f30ba700 1 -- 172.21.2.102:0/3803241763 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f97ec0d1870 msgr2=0x7f97ec1b04c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:00.623 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:00.608 7f97f30ba700 1 --2- 172.21.2.102:0/3803241763 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f97ec0d1870 0x7f97ec1b04c0 secure :-1 s=READY pgs=1558 cs=0 l=1 rev1=1 rx=0x7f97d4006700 tx=0x7f97d4009b80).stop 2021-02-10T11:26:00.623 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:00.609 7f97f30ba700 1 -- 172.21.2.102:0/3803241763 shutdown_connections 2021-02-10T11:26:00.624 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:00.609 7f97f30ba700 1 --2- 172.21.2.102:0/3803241763 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f97ec1b0ae0 0x7f97ec108940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:00.624 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:00.609 7f97f30ba700 1 --2- 172.21.2.102:0/3803241763 >> v2:172.21.2.113:3300/0 conn(0x7f97ec0d1c70 0x7f97ec108f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:00.624 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:00.609 7f97f30ba700 1 --2- 172.21.2.102:0/3803241763 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f97ec0d1870 0x7f97ec1b04c0 unknown :-1 s=CLOSED pgs=1558 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:00.624 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:00.609 7f97f30ba700 1 -- 172.21.2.102:0/3803241763 >> 172.21.2.102:0/3803241763 conn(0x7f97ec0ce420 msgr2=0x7f97ec0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:26:00.624 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:00.609 7f97f30ba700 1 -- 172.21.2.102:0/3803241763 shutdown_connections 2021-02-10T11:26:00.624 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:00.609 7f97f30ba700 1 -- 172.21.2.102:0/3803241763 wait complete. 2021-02-10T11:26:00.625 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:00.623 7f97f30ba700 1 Processor -- start 2021-02-10T11:26:00.626 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:00.623 7f97f30ba700 1 -- start start 2021-02-10T11:26:00.626 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:00.624 7f97f30ba700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f97ec190680 0x7f97ec0926d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:00.626 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:00.624 7f97f30ba700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f97ec190a80 0x7f97ec1a7b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:00.626 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:00.624 7f97f30ba700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f97ec1a8190 0x7f97ec093660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:00.626 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:00.624 7f97f30ba700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97ec0930e0 con 0x7f97ec1a8190 2021-02-10T11:26:00.626 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:00.624 7f97f30ba700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97ec096180 con 0x7f97ec190a80 2021-02-10T11:26:00.626 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:00.624 7f97f30ba700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97ec096360 con 0x7f97ec190680 2021-02-10T11:26:00.626 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:00.624 7f97f16d7700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f97ec1a8190 0x7f97ec093660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:00.626 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:00.624 7f97f16d7700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f97ec1a8190 0x7f97ec093660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:58396/0 (socket says 172.21.2.102:58396) 2021-02-10T11:26:00.626 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:00.624 7f97f16d7700 1 -- 172.21.2.102:0/1347623405 learned_addr learned my addr 172.21.2.102:0/1347623405 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:26:00.626 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:00.624 7f97f16d7700 1 -- 172.21.2.102:0/1347623405 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f97ec190680 msgr2=0x7f97ec0926d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:00.627 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:00.624 7f97f16d7700 1 --2- 172.21.2.102:0/1347623405 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f97ec190680 0x7f97ec0926d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:00.627 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:00.624 7f97f16d7700 1 -- 172.21.2.102:0/1347623405 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f97ec190a80 msgr2=0x7f97ec1a7b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:00.627 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:00.624 7f97f16d7700 1 --2- 172.21.2.102:0/1347623405 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f97ec190a80 0x7f97ec1a7b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:00.627 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:00.624 7f97f16d7700 1 -- 172.21.2.102:0/1347623405 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97ec092d20 con 0x7f97ec1a8190 2021-02-10T11:26:00.627 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:00.624 7f97f16d7700 1 --2- 172.21.2.102:0/1347623405 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f97ec1a8190 0x7f97ec093660 secure :-1 s=READY pgs=1559 cs=0 l=1 rev1=1 rx=0x7f97e4009960 tx=0x7f97e400bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:00.627 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:00.625 7f97e97fa700 1 -- 172.21.2.102:0/1347623405 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f97e4000d10 con 0x7f97ec1a8190 2021-02-10T11:26:00.627 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:00.625 7f97e97fa700 1 -- 172.21.2.102:0/1347623405 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97e4003020 con 0x7f97ec1a8190 2021-02-10T11:26:00.627 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:00.625 7f97e97fa700 1 -- 172.21.2.102:0/1347623405 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f97e4005a40 con 0x7f97ec1a8190 2021-02-10T11:26:00.627 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:00.625 7f97f30ba700 1 -- 172.21.2.102:0/1347623405 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97ec092d20 con 0x7f97ec1a8190 2021-02-10T11:26:00.627 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:00.625 7f97f30ba700 1 -- 172.21.2.102:0/1347623405 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97ec091620 con 0x7f97ec1a8190 2021-02-10T11:26:00.627 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:00.625 7f97e97fa700 1 -- 172.21.2.102:0/1347623405 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f97e4005850 con 0x7f97ec1a8190 2021-02-10T11:26:00.627 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:00.625 7f97e97fa700 1 --2- 172.21.2.102:0/1347623405 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f97d00186c0 0x7f97d001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:00.627 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:00.625 7f97e97fa700 1 -- 172.21.2.102:0/1347623405 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f97e4032900 con 0x7f97ec1a8190 2021-02-10T11:26:00.628 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:00.625 7f97f0ed6700 1 --2- 172.21.2.102:0/1347623405 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f97d00186c0 0x7f97d001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:00.628 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:00.626 7f97f0ed6700 1 --2- 172.21.2.102:0/1347623405 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f97d00186c0 0x7f97d001abc0 crc :-1 s=READY pgs=709 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:00.656 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:00.657 7f97f30ba700 1 -- 172.21.2.102:0/1347623405 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97ec082a70 con 0x7f97ec1a8190 2021-02-10T11:26:00.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:00.660 7f97e97fa700 1 -- 172.21.2.102:0/1347623405 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f97e4005a60 con 0x7f97ec1a8190 2021-02-10T11:26:00.806 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:00.808 7f97f30ba700 1 -- 172.21.2.102:0/1347623405 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command([{prefix=config set, name=debug_client}] v 0) v1 -- 0x7f97ec082a70 con 0x7f97ec1a8190 2021-02-10T11:26:00.810 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:00.812 7f97e97fa700 1 -- 172.21.2.102:0/1347623405 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{prefix=config set, name=debug_client}]=0 v15)=0 v15) v1 ==== 120+0+0 (secure 0 0 0) 0x7f97e4005850 con 0x7f97ec1a8190 2021-02-10T11:26:00.815 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:00.816 7f97f30ba700 1 -- 172.21.2.102:0/1347623405 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f97d00186c0 msgr2=0x7f97d001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:00.815 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:00.816 7f97f30ba700 1 --2- 172.21.2.102:0/1347623405 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f97d00186c0 0x7f97d001abc0 crc :-1 s=READY pgs=709 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:00.815 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:00.816 7f97f30ba700 1 -- 172.21.2.102:0/1347623405 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f97ec1a8190 msgr2=0x7f97ec093660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:00.815 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:00.816 7f97f30ba700 1 --2- 172.21.2.102:0/1347623405 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f97ec1a8190 0x7f97ec093660 secure :-1 s=READY pgs=1559 cs=0 l=1 rev1=1 rx=0x7f97e4009960 tx=0x7f97e400bcf0).stop 2021-02-10T11:26:00.816 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:00.817 7f97f30ba700 1 -- 172.21.2.102:0/1347623405 shutdown_connections 2021-02-10T11:26:00.816 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:00.817 7f97f30ba700 1 --2- 172.21.2.102:0/1347623405 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f97ec1a8190 0x7f97ec093660 unknown :-1 s=CLOSED pgs=1559 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:00.816 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:00.817 7f97f30ba700 1 --2- 172.21.2.102:0/1347623405 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f97d00186c0 0x7f97d001abc0 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:00.816 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:00.817 7f97f30ba700 1 --2- 172.21.2.102:0/1347623405 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f97ec190a80 0x7f97ec1a7b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:00.816 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:00.817 7f97f30ba700 1 --2- 172.21.2.102:0/1347623405 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f97ec190680 0x7f97ec0926d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:00.816 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:00.817 7f97f30ba700 1 -- 172.21.2.102:0/1347623405 >> 172.21.2.102:0/1347623405 conn(0x7f97ec0bc0c0 msgr2=0x7f97ec082ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:26:00.816 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:00.817 7f97f30ba700 1 -- 172.21.2.102:0/1347623405 shutdown_connections 2021-02-10T11:26:00.816 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:00.817 7f97f30ba700 1 -- 172.21.2.102:0/1347623405 wait complete. 2021-02-10T11:26:00.824 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config set client.alice debug_objecter 20 2021-02-10T11:26:01.253 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:01.238 7fd9f497b700 1 Processor -- start 2021-02-10T11:26:01.253 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:01.238 7fd9f497b700 1 -- start start 2021-02-10T11:26:01.253 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:01.238 7fd9f497b700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fd9f00d1c70 0x7fd9f00cef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:01.253 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:01.238 7fd9f497b700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9f00d0170 con 0x7fd9f00d1870 2021-02-10T11:26:01.253 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:01.238 7fd9f497b700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9f00fe1c0 con 0x7fd9f00cf580 2021-02-10T11:26:01.253 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:01.238 7fd9f497b700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9f01b0cb0 con 0x7fd9f00d1c70 2021-02-10T11:26:01.253 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:01.239 7fd9ee61c700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fd9f00d1870 0x7fd9f01b04c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:50624/0 (socket says 172.21.2.102:50624) 2021-02-10T11:26:01.253 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:01.239 7fd9ee61c700 1 -- 172.21.2.102:0/3961645160 learned_addr learned my addr 172.21.2.102:0/3961645160 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:26:01.253 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:01.239 7fd9ed61a700 1 -- 172.21.2.102:0/3961645160 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1566880996 0 0) 0x7fd9e00010b0 con 0x7fd9f00d1870 2021-02-10T11:26:01.253 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:01.239 7fd9ed61a700 1 -- 172.21.2.102:0/3961645160 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9dc002fe0 con 0x7fd9f00d1870 2021-02-10T11:26:01.254 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:01.239 7fd9ed61a700 1 -- 172.21.2.102:0/3961645160 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3475843009 0 0) 0x7fd9e0001010 con 0x7fd9f00d1870 2021-02-10T11:26:01.254 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:01.239 7fd9ed61a700 1 -- 172.21.2.102:0/3961645160 >> v2:172.21.2.113:3300/0 conn(0x7fd9f00d1c70 msgr2=0x7fd9f00cef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:26:01.254 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:01.239 7fd9ed61a700 1 --2- 172.21.2.102:0/3961645160 >> v2:172.21.2.113:3300/0 conn(0x7fd9f00d1c70 0x7fd9f00cef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:01.254 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:01.239 7fd9ed61a700 1 -- 172.21.2.102:0/3961645160 >> v1:172.21.2.113:6789/0 conn(0x7fd9f00cf580 legacy=0x7fd9f00cf980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:26:01.254 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:01.239 7fd9ed61a700 1 -- 172.21.2.102:0/3961645160 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9f0109a60 con 0x7fd9f00d1870 2021-02-10T11:26:01.254 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:01.239 7fd9ed61a700 1 -- 172.21.2.102:0/3961645160 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7fd9e0001530 con 0x7fd9f00d1870 2021-02-10T11:26:01.254 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:01.239 7fd9ed61a700 1 -- 172.21.2.102:0/3961645160 >> v1:172.21.2.102:6789/0 conn(0x7fd9f00d1870 legacy=0x7fd9f01b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:01.254 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:01.239 7fd9ed61a700 1 --2- 172.21.2.102:0/3961645160 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd9dc003ac0 0x7fd9dc005ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:01.254 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:01.239 7fd9ed61a700 1 --2- 172.21.2.102:0/3961645160 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd9dc006790 0x7fd9dc008c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:01.254 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:01.239 7fd9ed61a700 1 --2- 172.21.2.102:0/3961645160 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd9dc009590 0x7fd9dc00bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:01.254 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:01.239 7fd9ed61a700 1 -- 172.21.2.102:0/3961645160 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9dc00c300 con 0x7fd9dc003ac0 2021-02-10T11:26:01.254 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:01.239 7fd9ed61a700 1 -- 172.21.2.102:0/3961645160 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9dc00c520 con 0x7fd9dc009590 2021-02-10T11:26:01.254 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:01.239 7fd9ed61a700 1 -- 172.21.2.102:0/3961645160 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9dc00c740 con 0x7fd9dc006790 2021-02-10T11:26:01.255 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:01.239 7fd9ee61c700 1 --2- 172.21.2.102:0/3961645160 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd9dc003ac0 0x7fd9dc005ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:01.255 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:01.239 7fd9ee61c700 1 -- 172.21.2.102:0/3961645160 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd9dc006790 msgr2=0x7fd9dc008c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:26:01.255 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:01.239 7fd9ee61c700 1 --2- 172.21.2.102:0/3961645160 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd9dc006790 0x7fd9dc008c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:01.255 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:01.239 7fd9ee61c700 1 -- 172.21.2.102:0/3961645160 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd9dc009590 msgr2=0x7fd9dc00bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:26:01.255 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:01.239 7fd9eee1d700 1 --2- 172.21.2.102:0/3961645160 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd9dc009590 0x7fd9dc00bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:01.255 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:01.239 7fd9ee61c700 1 --2- 172.21.2.102:0/3961645160 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd9dc009590 0x7fd9dc00bab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:01.255 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:01.239 7fd9ee61c700 1 -- 172.21.2.102:0/3961645160 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9dc00c9e0 con 0x7fd9dc003ac0 2021-02-10T11:26:01.255 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:01.239 7fd9eee1d700 1 -- 172.21.2.102:0/3961645160 reap_dead start 2021-02-10T11:26:01.255 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:01.240 7fd9ee61c700 1 --2- 172.21.2.102:0/3961645160 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd9dc003ac0 0x7fd9dc005ef0 secure :-1 s=READY pgs=1561 cs=0 l=1 rev1=1 rx=0x7fd9e00017d0 tx=0x7fd9e0007d30).ready entity=mon.0 client_cookie=cc3f99da16206903 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:01.255 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:01.240 7fd9ed61a700 1 -- 172.21.2.102:0/3961645160 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd9e0001010 con 0x7fd9dc003ac0 2021-02-10T11:26:01.255 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:01.240 7fd9ed61a700 1 -- 172.21.2.102:0/3961645160 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9e00013a0 con 0x7fd9dc003ac0 2021-02-10T11:26:01.255 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:01.240 7fd9ed61a700 1 -- 172.21.2.102:0/3961645160 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd9e000a2b0 con 0x7fd9dc003ac0 2021-02-10T11:26:01.255 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:01.240 7fd9f497b700 1 -- 172.21.2.102:0/3961645160 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd9dc003ac0 msgr2=0x7fd9dc005ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:01.256 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:01.240 7fd9f497b700 1 --2- 172.21.2.102:0/3961645160 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd9dc003ac0 0x7fd9dc005ef0 secure :-1 s=READY pgs=1561 cs=0 l=1 rev1=1 rx=0x7fd9e00017d0 tx=0x7fd9e0007d30).stop 2021-02-10T11:26:01.256 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:01.240 7fd9f497b700 1 -- 172.21.2.102:0/3961645160 shutdown_connections 2021-02-10T11:26:01.256 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:01.240 7fd9f497b700 1 --2- 172.21.2.102:0/3961645160 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd9dc003ac0 0x7fd9dc005ef0 unknown :-1 s=CLOSED pgs=1561 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:01.256 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:01.240 7fd9f497b700 1 -- 172.21.2.102:0/3961645160 >> 172.21.2.102:0/3961645160 conn(0x7fd9f00ce420 msgr2=0x7fd9f00d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:26:01.256 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:01.240 7fd9f497b700 1 -- 172.21.2.102:0/3961645160 shutdown_connections 2021-02-10T11:26:01.256 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:01.240 7fd9f497b700 1 -- 172.21.2.102:0/3961645160 wait complete. 2021-02-10T11:26:01.257 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:01.255 7fd9f497b700 1 Processor -- start 2021-02-10T11:26:01.258 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:01.255 7fd9f497b700 1 -- start start 2021-02-10T11:26:01.258 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:01.255 7fd9f497b700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd9f0085bc0 0x7fd9f00863e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:01.258 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:01.255 7fd9f497b700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd9f00954c0 0x7fd9f0096300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:01.258 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:01.255 7fd9f497b700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd9f00958c0 0x7fd9f0096920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:01.258 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:01.255 7fd9f497b700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9f0095d80 con 0x7fd9f00954c0 2021-02-10T11:26:01.258 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:01.255 7fd9f497b700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9f00bfa60 con 0x7fd9f00958c0 2021-02-10T11:26:01.258 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:01.255 7fd9f497b700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9f0085fc0 con 0x7fd9f0085bc0 2021-02-10T11:26:01.258 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:01.255 7fd9ede1b700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd9f00954c0 0x7fd9f0096300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:01.258 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:01.255 7fd9ede1b700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd9f00954c0 0x7fd9f0096300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:58414/0 (socket says 172.21.2.102:58414) 2021-02-10T11:26:01.258 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:01.255 7fd9ede1b700 1 -- 172.21.2.102:0/2743148364 learned_addr learned my addr 172.21.2.102:0/2743148364 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:26:01.258 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:01.256 7fd9eee1d700 1 --2- 172.21.2.102:0/2743148364 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd9f00958c0 0x7fd9f0096920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:01.258 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:01.256 7fd9ee61c700 1 --2- 172.21.2.102:0/2743148364 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd9f0085bc0 0x7fd9f00863e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:01.259 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:01.256 7fd9ede1b700 1 -- 172.21.2.102:0/2743148364 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd9f0085bc0 msgr2=0x7fd9f00863e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:01.259 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:01.256 7fd9ede1b700 1 --2- 172.21.2.102:0/2743148364 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd9f0085bc0 0x7fd9f00863e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:01.259 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:01.256 7fd9ede1b700 1 -- 172.21.2.102:0/2743148364 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd9f00958c0 msgr2=0x7fd9f0096920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:01.259 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:01.256 7fd9ede1b700 1 --2- 172.21.2.102:0/2743148364 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd9f00958c0 0x7fd9f0096920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:01.259 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:01.256 7fd9ede1b700 1 -- 172.21.2.102:0/2743148364 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9f0086c10 con 0x7fd9f00954c0 2021-02-10T11:26:01.259 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:01.256 7fd9ee61c700 1 --2- 172.21.2.102:0/2743148364 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd9f0085bc0 0x7fd9f00863e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:26:01.259 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:01.256 7fd9ede1b700 1 --2- 172.21.2.102:0/2743148364 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd9f00954c0 0x7fd9f0096300 secure :-1 s=READY pgs=1562 cs=0 l=1 rev1=1 rx=0x7fd9d800a5a0 tx=0x7fd9d800a940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:01.259 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:01.256 7fd9e6ffd700 1 -- 172.21.2.102:0/2743148364 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd9d80009b0 con 0x7fd9f00954c0 2021-02-10T11:26:01.259 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:01.257 7fd9e6ffd700 1 -- 172.21.2.102:0/2743148364 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9d800ad30 con 0x7fd9f00954c0 2021-02-10T11:26:01.259 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:01.257 7fd9e6ffd700 1 -- 172.21.2.102:0/2743148364 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd9d80034c0 con 0x7fd9f00954c0 2021-02-10T11:26:01.259 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:01.257 7fd9f497b700 1 -- 172.21.2.102:0/2743148364 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9f0086c10 con 0x7fd9f00954c0 2021-02-10T11:26:01.259 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:01.257 7fd9f497b700 1 -- 172.21.2.102:0/2743148364 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9f0097230 con 0x7fd9f00954c0 2021-02-10T11:26:01.259 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:01.257 7fd9e6ffd700 1 -- 172.21.2.102:0/2743148364 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd9d8003030 con 0x7fd9f00954c0 2021-02-10T11:26:01.259 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:01.257 7fd9e6ffd700 1 --2- 172.21.2.102:0/2743148364 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd9d40186c0 0x7fd9d401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:01.260 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:01.257 7fd9e6ffd700 1 -- 172.21.2.102:0/2743148364 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fd9d8032460 con 0x7fd9f00954c0 2021-02-10T11:26:01.260 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:01.257 7fd9ee61c700 1 --2- 172.21.2.102:0/2743148364 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd9d40186c0 0x7fd9d401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:01.260 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:01.258 7fd9ee61c700 1 --2- 172.21.2.102:0/2743148364 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd9d40186c0 0x7fd9d401abc0 crc :-1 s=READY pgs=710 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:01.287 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:01.288 7fd9f497b700 1 -- 172.21.2.102:0/2743148364 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9f00fe1c0 con 0x7fd9f00954c0 2021-02-10T11:26:01.290 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:01.291 7fd9e6ffd700 1 -- 172.21.2.102:0/2743148364 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd9d80009b0 con 0x7fd9f00954c0 2021-02-10T11:26:01.437 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:01.438 7fd9f497b700 1 -- 172.21.2.102:0/2743148364 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command([{prefix=config set, name=debug_objecter}] v 0) v1 -- 0x7fd9f00fe1c0 con 0x7fd9f00954c0 2021-02-10T11:26:01.440 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:01.442 7fd9e6ffd700 1 -- 172.21.2.102:0/2743148364 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{prefix=config set, name=debug_objecter}]=0 v16)=0 v16) v1 ==== 122+0+0 (secure 0 0 0) 0x7fd9d8003030 con 0x7fd9f00954c0 2021-02-10T11:26:01.446 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:01.448 7fd9f497b700 1 -- 172.21.2.102:0/2743148364 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd9d40186c0 msgr2=0x7fd9d401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:01.446 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:01.448 7fd9f497b700 1 --2- 172.21.2.102:0/2743148364 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd9d40186c0 0x7fd9d401abc0 crc :-1 s=READY pgs=710 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:01.446 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:01.448 7fd9f497b700 1 -- 172.21.2.102:0/2743148364 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd9f00954c0 msgr2=0x7fd9f0096300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:01.447 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:01.448 7fd9f497b700 1 --2- 172.21.2.102:0/2743148364 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd9f00954c0 0x7fd9f0096300 secure :-1 s=READY pgs=1562 cs=0 l=1 rev1=1 rx=0x7fd9d800a5a0 tx=0x7fd9d800a940).stop 2021-02-10T11:26:01.447 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:01.448 7fd9f497b700 1 -- 172.21.2.102:0/2743148364 shutdown_connections 2021-02-10T11:26:01.447 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:01.448 7fd9f497b700 1 --2- 172.21.2.102:0/2743148364 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd9d40186c0 0x7fd9d401abc0 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:01.447 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:01.448 7fd9f497b700 1 --2- 172.21.2.102:0/2743148364 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd9f00958c0 0x7fd9f0096920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:01.447 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:01.448 7fd9f497b700 1 --2- 172.21.2.102:0/2743148364 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd9f00954c0 0x7fd9f0096300 unknown :-1 s=CLOSED pgs=1562 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:01.447 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:01.448 7fd9f497b700 1 --2- 172.21.2.102:0/2743148364 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd9f0085bc0 0x7fd9f00863e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:01.447 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:01.448 7fd9f497b700 1 -- 172.21.2.102:0/2743148364 >> 172.21.2.102:0/2743148364 conn(0x7fd9f00bef00 msgr2=0x7fd9f009d1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:26:01.447 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:01.448 7fd9f497b700 1 -- 172.21.2.102:0/2743148364 shutdown_connections 2021-02-10T11:26:01.447 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:01.448 7fd9f497b700 1 -- 172.21.2.102:0/2743148364 wait complete. 2021-02-10T11:26:01.456 DEBUG:teuthology.orchestra.run.gibba002:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2021-02-10T11:26:01.458 DEBUG:teuthology.orchestra.run.gibba013:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2021-02-10T11:26:01.818 INFO:teuthology.orchestra.run.gibba002.stdout:[global] 2021-02-10T11:26:01.819 INFO:teuthology.orchestra.run.gibba002.stdout: chdir = "" 2021-02-10T11:26:01.819 INFO:teuthology.orchestra.run.gibba002.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2021-02-10T11:26:01.819 INFO:teuthology.orchestra.run.gibba002.stdout: auth supported = cephx 2021-02-10T11:26:01.819 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:26:01.819 INFO:teuthology.orchestra.run.gibba002.stdout: filestore xattr use omap = true 2021-02-10T11:26:01.819 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:26:01.819 INFO:teuthology.orchestra.run.gibba002.stdout: mon clock drift allowed = 1.000 2021-02-10T11:26:01.819 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:26:01.819 INFO:teuthology.orchestra.run.gibba002.stdout: osd crush chooseleaf type = 0 2021-02-10T11:26:01.819 INFO:teuthology.orchestra.run.gibba002.stdout: auth debug = true 2021-02-10T11:26:01.819 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:26:01.819 INFO:teuthology.orchestra.run.gibba002.stdout: ms die on old message = true 2021-02-10T11:26:01.819 INFO:teuthology.orchestra.run.gibba002.stdout: ms die on bug = true 2021-02-10T11:26:01.820 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:26:01.820 INFO:teuthology.orchestra.run.gibba002.stdout: mon max pg per osd = 10000 # >= luminous 2021-02-10T11:26:01.820 INFO:teuthology.orchestra.run.gibba002.stdout: mon pg warn max object skew = 0 2021-02-10T11:26:01.820 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:26:01.820 INFO:teuthology.orchestra.run.gibba002.stdout: osd pool default size = 2 2021-02-10T11:26:01.820 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:26:01.820 INFO:teuthology.orchestra.run.gibba002.stdout: mon osd allow primary affinity = true 2021-02-10T11:26:01.820 INFO:teuthology.orchestra.run.gibba002.stdout: mon osd allow pg remap = true 2021-02-10T11:26:01.820 INFO:teuthology.orchestra.run.gibba002.stdout: mon warn on legacy crush tunables = false 2021-02-10T11:26:01.820 INFO:teuthology.orchestra.run.gibba002.stdout: mon warn on crush straw calc version zero = false 2021-02-10T11:26:01.820 INFO:teuthology.orchestra.run.gibba002.stdout: mon warn on no sortbitwise = false 2021-02-10T11:26:01.820 INFO:teuthology.orchestra.run.gibba002.stdout: mon warn on osd down out interval zero = false 2021-02-10T11:26:01.820 INFO:teuthology.orchestra.run.gibba002.stdout: mon warn on too few osds = false 2021-02-10T11:26:01.821 INFO:teuthology.orchestra.run.gibba002.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2021-02-10T11:26:01.821 INFO:teuthology.orchestra.run.gibba002.stdout: mon_warn_on_pool_no_redundancy = false 2021-02-10T11:26:01.821 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:26:01.821 INFO:teuthology.orchestra.run.gibba002.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2021-02-10T11:26:01.821 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:26:01.821 INFO:teuthology.orchestra.run.gibba002.stdout: osd default data pool replay window = 5 2021-02-10T11:26:01.821 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:26:01.821 INFO:teuthology.orchestra.run.gibba002.stdout: mon allow pool delete = true 2021-02-10T11:26:01.821 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:26:01.821 INFO:teuthology.orchestra.run.gibba002.stdout: mon cluster log file level = debug 2021-02-10T11:26:01.821 INFO:teuthology.orchestra.run.gibba002.stdout: debug asserts on shutdown = true 2021-02-10T11:26:01.821 INFO:teuthology.orchestra.run.gibba002.stdout: mon health detail to clog = false 2021-02-10T11:26:01.821 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:26:01.822 INFO:teuthology.orchestra.run.gibba002.stdout: # we see this fail in qa on *nautilus*; bump up retries 2021-02-10T11:26:01.822 INFO:teuthology.orchestra.run.gibba002.stdout: mon_client_directed_command_retry = 4 2021-02-10T11:26:01.822 INFO:teuthology.orchestra.run.gibba002.stdout: mon host = "172.21.2.102,172.21.2.113,[v2:172.21.2.113:3301,v1:172.21.2.113:6790]" 2021-02-10T11:26:01.822 INFO:teuthology.orchestra.run.gibba002.stdout: fsid = 68db30ea-dbce-4aef-a6df-f01c6fea0513 2021-02-10T11:26:01.822 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:26:01.822 INFO:teuthology.orchestra.run.gibba002.stdout:[osd] 2021-02-10T11:26:01.822 INFO:teuthology.orchestra.run.gibba002.stdout: osd journal size = 100 2021-02-10T11:26:01.822 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:26:01.822 INFO:teuthology.orchestra.run.gibba002.stdout: osd scrub load threshold = 5.0 2021-02-10T11:26:01.822 INFO:teuthology.orchestra.run.gibba002.stdout: osd scrub max interval = 600 2021-02-10T11:26:01.822 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:26:01.822 INFO:teuthology.orchestra.run.gibba002.stdout: osd recover clone overlap = true 2021-02-10T11:26:01.823 INFO:teuthology.orchestra.run.gibba002.stdout: osd recovery max chunk = 1048576 2021-02-10T11:26:01.823 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:26:01.823 INFO:teuthology.orchestra.run.gibba002.stdout: osd debug shutdown = true 2021-02-10T11:26:01.823 INFO:teuthology.orchestra.run.gibba002.stdout: osd debug op order = true 2021-02-10T11:26:01.823 INFO:teuthology.orchestra.run.gibba002.stdout: osd debug verify stray on activate = true 2021-02-10T11:26:01.823 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:26:01.823 INFO:teuthology.orchestra.run.gibba002.stdout: osd open classes on start = true 2021-02-10T11:26:01.823 INFO:teuthology.orchestra.run.gibba002.stdout: osd debug pg log writeout = true 2021-02-10T11:26:01.823 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:26:01.823 INFO:teuthology.orchestra.run.gibba002.stdout: osd deep scrub update digest min age = 30 2021-02-10T11:26:01.823 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:26:01.823 INFO:teuthology.orchestra.run.gibba002.stdout: osd map max advance = 10 2021-02-10T11:26:01.823 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:26:01.824 INFO:teuthology.orchestra.run.gibba002.stdout: journal zero on create = true 2021-02-10T11:26:01.824 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:26:01.824 INFO:teuthology.orchestra.run.gibba002.stdout: filestore ondisk finisher threads = 3 2021-02-10T11:26:01.824 INFO:teuthology.orchestra.run.gibba002.stdout: filestore apply finisher threads = 3 2021-02-10T11:26:01.824 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:26:01.824 INFO:teuthology.orchestra.run.gibba002.stdout: bdev debug aio = true 2021-02-10T11:26:01.824 INFO:teuthology.orchestra.run.gibba002.stdout: osd debug misdirected ops = true 2021-02-10T11:26:01.824 INFO:teuthology.orchestra.run.gibba002.stdout: bdev async discard = True 2021-02-10T11:26:01.824 INFO:teuthology.orchestra.run.gibba002.stdout: bdev enable discard = True 2021-02-10T11:26:01.824 INFO:teuthology.orchestra.run.gibba002.stdout: bluestore allocator = bitmap 2021-02-10T11:26:01.824 INFO:teuthology.orchestra.run.gibba002.stdout: bluestore block size = 96636764160 2021-02-10T11:26:01.824 INFO:teuthology.orchestra.run.gibba002.stdout: bluestore fsck on mount = True 2021-02-10T11:26:01.825 INFO:teuthology.orchestra.run.gibba002.stdout: debug bluefs = 20 2021-02-10T11:26:01.825 INFO:teuthology.orchestra.run.gibba002.stdout: debug bluestore = 20 2021-02-10T11:26:01.825 INFO:teuthology.orchestra.run.gibba002.stdout: debug filestore = 20 2021-02-10T11:26:01.825 INFO:teuthology.orchestra.run.gibba002.stdout: debug journal = 20 2021-02-10T11:26:01.825 INFO:teuthology.orchestra.run.gibba002.stdout: debug ms = 20 2021-02-10T11:26:01.825 INFO:teuthology.orchestra.run.gibba002.stdout: debug osd = 25 2021-02-10T11:26:01.825 INFO:teuthology.orchestra.run.gibba002.stdout: debug rocksdb = 10 2021-02-10T11:26:01.825 INFO:teuthology.orchestra.run.gibba002.stdout: mon osd backfillfull_ratio = 0.85 2021-02-10T11:26:01.825 INFO:teuthology.orchestra.run.gibba002.stdout: mon osd full ratio = 0.9 2021-02-10T11:26:01.825 INFO:teuthology.orchestra.run.gibba002.stdout: mon osd nearfull ratio = 0.8 2021-02-10T11:26:01.825 INFO:teuthology.orchestra.run.gibba002.stdout: osd failsafe full ratio = 0.95 2021-02-10T11:26:01.825 INFO:teuthology.orchestra.run.gibba002.stdout: osd objectstore = bluestore 2021-02-10T11:26:01.825 INFO:teuthology.orchestra.run.gibba002.stdout: osd op complaint time = 180 2021-02-10T11:26:01.826 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:26:01.826 INFO:teuthology.orchestra.run.gibba002.stdout:[mgr] 2021-02-10T11:26:01.826 INFO:teuthology.orchestra.run.gibba002.stdout: debug ms = 1 2021-02-10T11:26:01.826 INFO:teuthology.orchestra.run.gibba002.stdout: debug mgr = 20 2021-02-10T11:26:01.826 INFO:teuthology.orchestra.run.gibba002.stdout: debug mon = 20 2021-02-10T11:26:01.826 INFO:teuthology.orchestra.run.gibba002.stdout: debug auth = 20 2021-02-10T11:26:01.826 INFO:teuthology.orchestra.run.gibba002.stdout: mon reweight min pgs per osd = 4 2021-02-10T11:26:01.826 INFO:teuthology.orchestra.run.gibba002.stdout: mon reweight min bytes per osd = 10 2021-02-10T11:26:01.826 INFO:teuthology.orchestra.run.gibba002.stdout: debug client = 10 2021-02-10T11:26:01.826 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:26:01.826 INFO:teuthology.orchestra.run.gibba002.stdout:[mon] 2021-02-10T11:26:01.826 INFO:teuthology.orchestra.run.gibba002.stdout: debug ms = 1 2021-02-10T11:26:01.826 INFO:teuthology.orchestra.run.gibba002.stdout: debug mon = 20 2021-02-10T11:26:01.827 INFO:teuthology.orchestra.run.gibba002.stdout: debug paxos = 20 2021-02-10T11:26:01.827 INFO:teuthology.orchestra.run.gibba002.stdout: debug auth = 20 2021-02-10T11:26:01.827 INFO:teuthology.orchestra.run.gibba002.stdout: mon data avail warn = 5 2021-02-10T11:26:01.827 INFO:teuthology.orchestra.run.gibba002.stdout: mon mgr mkfs grace = 240 2021-02-10T11:26:01.827 INFO:teuthology.orchestra.run.gibba002.stdout: mon reweight min pgs per osd = 4 2021-02-10T11:26:01.827 INFO:teuthology.orchestra.run.gibba002.stdout: mon osd reporter subtree level = osd 2021-02-10T11:26:01.827 INFO:teuthology.orchestra.run.gibba002.stdout: mon osd prime pg temp = true 2021-02-10T11:26:01.827 INFO:teuthology.orchestra.run.gibba002.stdout: mon reweight min bytes per osd = 10 2021-02-10T11:26:01.827 INFO:teuthology.orchestra.run.gibba002.stdout: mon op complaint time = 120 2021-02-10T11:26:01.827 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:26:01.827 INFO:teuthology.orchestra.run.gibba002.stdout:[client] 2021-02-10T11:26:01.827 INFO:teuthology.orchestra.run.gibba002.stdout: rgw cache enabled = true 2021-02-10T11:26:01.827 INFO:teuthology.orchestra.run.gibba002.stdout: rgw enable ops log = true 2021-02-10T11:26:01.828 INFO:teuthology.orchestra.run.gibba002.stdout: rgw enable usage log = true 2021-02-10T11:26:01.828 INFO:teuthology.orchestra.run.gibba002.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2021-02-10T11:26:01.828 INFO:teuthology.orchestra.run.gibba002.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2021-02-10T11:26:01.828 INFO:teuthology.orchestra.run.gibba002.stdout: client mount timeout = 600 2021-02-10T11:26:01.828 INFO:teuthology.orchestra.run.gibba002.stdout: debug client = 20 2021-02-10T11:26:01.828 INFO:teuthology.orchestra.run.gibba002.stdout: debug ms = 1 2021-02-10T11:26:01.828 INFO:teuthology.orchestra.run.gibba002.stdout: pid file = "" 2021-02-10T11:26:01.828 INFO:teuthology.orchestra.run.gibba002.stdout:[mon.a] 2021-02-10T11:26:01.828 INFO:teuthology.orchestra.run.gibba002.stdout:[mon.b] 2021-02-10T11:26:01.828 INFO:teuthology.orchestra.run.gibba002.stdout:[mon.c] 2021-02-10T11:26:01.828 INFO:teuthology.orchestra.run.gibba002.stdout:[mds.a] 2021-02-10T11:26:01.828 INFO:teuthology.orchestra.run.gibba002.stdout:[mds.b] 2021-02-10T11:26:01.829 INFO:teuthology.orchestra.run.gibba002.stdout:[mds] 2021-02-10T11:26:01.829 INFO:teuthology.orchestra.run.gibba002.stdout: debug mds = 20 2021-02-10T11:26:01.829 INFO:teuthology.orchestra.run.gibba002.stdout: debug ms = 1 2021-02-10T11:26:01.829 INFO:teuthology.orchestra.run.gibba002.stdout: mds bal frag = True 2021-02-10T11:26:01.829 INFO:teuthology.orchestra.run.gibba002.stdout: mds bal fragment size max = 10000 2021-02-10T11:26:01.829 INFO:teuthology.orchestra.run.gibba002.stdout: mds bal merge size = 5 2021-02-10T11:26:01.829 INFO:teuthology.orchestra.run.gibba002.stdout: mds bal split bits = 3 2021-02-10T11:26:01.829 INFO:teuthology.orchestra.run.gibba002.stdout: mds bal split size = 100 2021-02-10T11:26:01.829 INFO:teuthology.orchestra.run.gibba002.stdout: mds debug frag = True 2021-02-10T11:26:01.829 INFO:teuthology.orchestra.run.gibba002.stdout: mds debug scatterstat = True 2021-02-10T11:26:01.829 INFO:teuthology.orchestra.run.gibba002.stdout: mds op complaint time = 180 2021-02-10T11:26:01.829 INFO:teuthology.orchestra.run.gibba002.stdout: mds verify scatter = True 2021-02-10T11:26:01.829 INFO:teuthology.orchestra.run.gibba002.stdout: osd op complaint time = 180 2021-02-10T11:26:01.829 INFO:teuthology.orchestra.run.gibba002.stdout:[client.alice] 2021-02-10T11:26:01.830 INFO:teuthology.orchestra.run.gibba002.stdout: keyring = /etc/ceph/ceph.client.alice.keyring 2021-02-10T11:26:01.831 INFO:teuthology.orchestra.run.gibba013.stdout:[global] 2021-02-10T11:26:01.831 INFO:teuthology.orchestra.run.gibba013.stdout: chdir = "" 2021-02-10T11:26:01.831 INFO:teuthology.orchestra.run.gibba013.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2021-02-10T11:26:01.831 INFO:teuthology.orchestra.run.gibba013.stdout: auth supported = cephx 2021-02-10T11:26:01.831 INFO:teuthology.orchestra.run.gibba013.stdout: 2021-02-10T11:26:01.831 INFO:teuthology.orchestra.run.gibba013.stdout: filestore xattr use omap = true 2021-02-10T11:26:01.832 INFO:teuthology.orchestra.run.gibba013.stdout: 2021-02-10T11:26:01.832 INFO:teuthology.orchestra.run.gibba013.stdout: mon clock drift allowed = 1.000 2021-02-10T11:26:01.832 INFO:teuthology.orchestra.run.gibba013.stdout: 2021-02-10T11:26:01.832 INFO:teuthology.orchestra.run.gibba013.stdout: osd crush chooseleaf type = 0 2021-02-10T11:26:01.832 INFO:teuthology.orchestra.run.gibba013.stdout: auth debug = true 2021-02-10T11:26:01.832 INFO:teuthology.orchestra.run.gibba013.stdout: 2021-02-10T11:26:01.832 INFO:teuthology.orchestra.run.gibba013.stdout: ms die on old message = true 2021-02-10T11:26:01.832 INFO:teuthology.orchestra.run.gibba013.stdout: ms die on bug = true 2021-02-10T11:26:01.832 INFO:teuthology.orchestra.run.gibba013.stdout: 2021-02-10T11:26:01.832 INFO:teuthology.orchestra.run.gibba013.stdout: mon max pg per osd = 10000 # >= luminous 2021-02-10T11:26:01.832 INFO:teuthology.orchestra.run.gibba013.stdout: mon pg warn max object skew = 0 2021-02-10T11:26:01.832 INFO:teuthology.orchestra.run.gibba013.stdout: 2021-02-10T11:26:01.833 INFO:teuthology.orchestra.run.gibba013.stdout: osd pool default size = 2 2021-02-10T11:26:01.833 INFO:teuthology.orchestra.run.gibba013.stdout: 2021-02-10T11:26:01.833 INFO:teuthology.orchestra.run.gibba013.stdout: mon osd allow primary affinity = true 2021-02-10T11:26:01.833 INFO:teuthology.orchestra.run.gibba013.stdout: mon osd allow pg remap = true 2021-02-10T11:26:01.833 INFO:teuthology.orchestra.run.gibba013.stdout: mon warn on legacy crush tunables = false 2021-02-10T11:26:01.833 INFO:teuthology.orchestra.run.gibba013.stdout: mon warn on crush straw calc version zero = false 2021-02-10T11:26:01.833 INFO:teuthology.orchestra.run.gibba013.stdout: mon warn on no sortbitwise = false 2021-02-10T11:26:01.833 INFO:teuthology.orchestra.run.gibba013.stdout: mon warn on osd down out interval zero = false 2021-02-10T11:26:01.833 INFO:teuthology.orchestra.run.gibba013.stdout: mon warn on too few osds = false 2021-02-10T11:26:01.833 INFO:teuthology.orchestra.run.gibba013.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2021-02-10T11:26:01.833 INFO:teuthology.orchestra.run.gibba013.stdout: mon_warn_on_pool_no_redundancy = false 2021-02-10T11:26:01.833 INFO:teuthology.orchestra.run.gibba013.stdout: 2021-02-10T11:26:01.834 INFO:teuthology.orchestra.run.gibba013.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2021-02-10T11:26:01.834 INFO:teuthology.orchestra.run.gibba013.stdout: 2021-02-10T11:26:01.834 INFO:teuthology.orchestra.run.gibba013.stdout: osd default data pool replay window = 5 2021-02-10T11:26:01.834 INFO:teuthology.orchestra.run.gibba013.stdout: 2021-02-10T11:26:01.834 INFO:teuthology.orchestra.run.gibba013.stdout: mon allow pool delete = true 2021-02-10T11:26:01.834 INFO:teuthology.orchestra.run.gibba013.stdout: 2021-02-10T11:26:01.834 INFO:teuthology.orchestra.run.gibba013.stdout: mon cluster log file level = debug 2021-02-10T11:26:01.834 INFO:teuthology.orchestra.run.gibba013.stdout: debug asserts on shutdown = true 2021-02-10T11:26:01.834 INFO:teuthology.orchestra.run.gibba013.stdout: mon health detail to clog = false 2021-02-10T11:26:01.834 INFO:teuthology.orchestra.run.gibba013.stdout: 2021-02-10T11:26:01.834 INFO:teuthology.orchestra.run.gibba013.stdout: # we see this fail in qa on *nautilus*; bump up retries 2021-02-10T11:26:01.834 INFO:teuthology.orchestra.run.gibba013.stdout: mon_client_directed_command_retry = 4 2021-02-10T11:26:01.835 INFO:teuthology.orchestra.run.gibba013.stdout: mon host = "172.21.2.102,172.21.2.113,[v2:172.21.2.113:3301,v1:172.21.2.113:6790]" 2021-02-10T11:26:01.835 INFO:teuthology.orchestra.run.gibba013.stdout: fsid = 68db30ea-dbce-4aef-a6df-f01c6fea0513 2021-02-10T11:26:01.835 INFO:teuthology.orchestra.run.gibba013.stdout: 2021-02-10T11:26:01.835 INFO:teuthology.orchestra.run.gibba013.stdout:[osd] 2021-02-10T11:26:01.835 INFO:teuthology.orchestra.run.gibba013.stdout: osd journal size = 100 2021-02-10T11:26:01.835 INFO:teuthology.orchestra.run.gibba013.stdout: 2021-02-10T11:26:01.835 INFO:teuthology.orchestra.run.gibba013.stdout: osd scrub load threshold = 5.0 2021-02-10T11:26:01.835 INFO:teuthology.orchestra.run.gibba013.stdout: osd scrub max interval = 600 2021-02-10T11:26:01.835 INFO:teuthology.orchestra.run.gibba013.stdout: 2021-02-10T11:26:01.835 INFO:teuthology.orchestra.run.gibba013.stdout: osd recover clone overlap = true 2021-02-10T11:26:01.835 INFO:teuthology.orchestra.run.gibba013.stdout: osd recovery max chunk = 1048576 2021-02-10T11:26:01.835 INFO:teuthology.orchestra.run.gibba013.stdout: 2021-02-10T11:26:01.835 INFO:teuthology.orchestra.run.gibba013.stdout: osd debug shutdown = true 2021-02-10T11:26:01.835 INFO:teuthology.orchestra.run.gibba013.stdout: osd debug op order = true 2021-02-10T11:26:01.836 INFO:teuthology.orchestra.run.gibba013.stdout: osd debug verify stray on activate = true 2021-02-10T11:26:01.836 INFO:teuthology.orchestra.run.gibba013.stdout: 2021-02-10T11:26:01.836 INFO:teuthology.orchestra.run.gibba013.stdout: osd open classes on start = true 2021-02-10T11:26:01.836 INFO:teuthology.orchestra.run.gibba013.stdout: osd debug pg log writeout = true 2021-02-10T11:26:01.836 INFO:teuthology.orchestra.run.gibba013.stdout: 2021-02-10T11:26:01.836 INFO:teuthology.orchestra.run.gibba013.stdout: osd deep scrub update digest min age = 30 2021-02-10T11:26:01.836 INFO:teuthology.orchestra.run.gibba013.stdout: 2021-02-10T11:26:01.836 INFO:teuthology.orchestra.run.gibba013.stdout: osd map max advance = 10 2021-02-10T11:26:01.836 INFO:teuthology.orchestra.run.gibba013.stdout: 2021-02-10T11:26:01.836 INFO:teuthology.orchestra.run.gibba013.stdout: journal zero on create = true 2021-02-10T11:26:01.836 INFO:teuthology.orchestra.run.gibba013.stdout: 2021-02-10T11:26:01.836 INFO:teuthology.orchestra.run.gibba013.stdout: filestore ondisk finisher threads = 3 2021-02-10T11:26:01.836 INFO:teuthology.orchestra.run.gibba013.stdout: filestore apply finisher threads = 3 2021-02-10T11:26:01.836 INFO:teuthology.orchestra.run.gibba013.stdout: 2021-02-10T11:26:01.837 INFO:teuthology.orchestra.run.gibba013.stdout: bdev debug aio = true 2021-02-10T11:26:01.837 INFO:teuthology.orchestra.run.gibba013.stdout: osd debug misdirected ops = true 2021-02-10T11:26:01.837 INFO:teuthology.orchestra.run.gibba013.stdout: bdev async discard = True 2021-02-10T11:26:01.837 INFO:teuthology.orchestra.run.gibba013.stdout: bdev enable discard = True 2021-02-10T11:26:01.837 INFO:teuthology.orchestra.run.gibba013.stdout: bluestore allocator = bitmap 2021-02-10T11:26:01.837 INFO:teuthology.orchestra.run.gibba013.stdout: bluestore block size = 96636764160 2021-02-10T11:26:01.837 INFO:teuthology.orchestra.run.gibba013.stdout: bluestore fsck on mount = True 2021-02-10T11:26:01.837 INFO:teuthology.orchestra.run.gibba013.stdout: debug bluefs = 20 2021-02-10T11:26:01.837 INFO:teuthology.orchestra.run.gibba013.stdout: debug bluestore = 20 2021-02-10T11:26:01.837 INFO:teuthology.orchestra.run.gibba013.stdout: debug filestore = 20 2021-02-10T11:26:01.837 INFO:teuthology.orchestra.run.gibba013.stdout: debug journal = 20 2021-02-10T11:26:01.837 INFO:teuthology.orchestra.run.gibba013.stdout: debug ms = 20 2021-02-10T11:26:01.837 INFO:teuthology.orchestra.run.gibba013.stdout: debug osd = 25 2021-02-10T11:26:01.837 INFO:teuthology.orchestra.run.gibba013.stdout: debug rocksdb = 10 2021-02-10T11:26:01.838 INFO:teuthology.orchestra.run.gibba013.stdout: mon osd backfillfull_ratio = 0.85 2021-02-10T11:26:01.838 INFO:teuthology.orchestra.run.gibba013.stdout: mon osd full ratio = 0.9 2021-02-10T11:26:01.838 INFO:teuthology.orchestra.run.gibba013.stdout: mon osd nearfull ratio = 0.8 2021-02-10T11:26:01.838 INFO:teuthology.orchestra.run.gibba013.stdout: osd failsafe full ratio = 0.95 2021-02-10T11:26:01.838 INFO:teuthology.orchestra.run.gibba013.stdout: osd objectstore = bluestore 2021-02-10T11:26:01.838 INFO:teuthology.orchestra.run.gibba013.stdout: osd op complaint time = 180 2021-02-10T11:26:01.838 INFO:teuthology.orchestra.run.gibba013.stdout: 2021-02-10T11:26:01.838 INFO:teuthology.orchestra.run.gibba013.stdout:[mgr] 2021-02-10T11:26:01.838 INFO:teuthology.orchestra.run.gibba013.stdout: debug ms = 1 2021-02-10T11:26:01.838 INFO:teuthology.orchestra.run.gibba013.stdout: debug mgr = 20 2021-02-10T11:26:01.838 INFO:teuthology.orchestra.run.gibba013.stdout: debug mon = 20 2021-02-10T11:26:01.838 INFO:teuthology.orchestra.run.gibba013.stdout: debug auth = 20 2021-02-10T11:26:01.838 INFO:teuthology.orchestra.run.gibba013.stdout: mon reweight min pgs per osd = 4 2021-02-10T11:26:01.838 INFO:teuthology.orchestra.run.gibba013.stdout: mon reweight min bytes per osd = 10 2021-02-10T11:26:01.839 INFO:teuthology.orchestra.run.gibba013.stdout: debug client = 10 2021-02-10T11:26:01.839 INFO:teuthology.orchestra.run.gibba013.stdout: 2021-02-10T11:26:01.839 INFO:teuthology.orchestra.run.gibba013.stdout:[mon] 2021-02-10T11:26:01.839 INFO:teuthology.orchestra.run.gibba013.stdout: debug ms = 1 2021-02-10T11:26:01.839 INFO:teuthology.orchestra.run.gibba013.stdout: debug mon = 20 2021-02-10T11:26:01.839 INFO:teuthology.orchestra.run.gibba013.stdout: debug paxos = 20 2021-02-10T11:26:01.839 INFO:teuthology.orchestra.run.gibba013.stdout: debug auth = 20 2021-02-10T11:26:01.839 INFO:teuthology.orchestra.run.gibba013.stdout: mon data avail warn = 5 2021-02-10T11:26:01.839 INFO:teuthology.orchestra.run.gibba013.stdout: mon mgr mkfs grace = 240 2021-02-10T11:26:01.839 INFO:teuthology.orchestra.run.gibba013.stdout: mon reweight min pgs per osd = 4 2021-02-10T11:26:01.839 INFO:teuthology.orchestra.run.gibba013.stdout: mon osd reporter subtree level = osd 2021-02-10T11:26:01.839 INFO:teuthology.orchestra.run.gibba013.stdout: mon osd prime pg temp = true 2021-02-10T11:26:01.839 INFO:teuthology.orchestra.run.gibba013.stdout: mon reweight min bytes per osd = 10 2021-02-10T11:26:01.839 INFO:teuthology.orchestra.run.gibba013.stdout: mon op complaint time = 120 2021-02-10T11:26:01.840 INFO:teuthology.orchestra.run.gibba013.stdout: 2021-02-10T11:26:01.840 INFO:teuthology.orchestra.run.gibba013.stdout:[client] 2021-02-10T11:26:01.840 INFO:teuthology.orchestra.run.gibba013.stdout: rgw cache enabled = true 2021-02-10T11:26:01.840 INFO:teuthology.orchestra.run.gibba013.stdout: rgw enable ops log = true 2021-02-10T11:26:01.840 INFO:teuthology.orchestra.run.gibba013.stdout: rgw enable usage log = true 2021-02-10T11:26:01.840 INFO:teuthology.orchestra.run.gibba013.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2021-02-10T11:26:01.840 INFO:teuthology.orchestra.run.gibba013.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2021-02-10T11:26:01.840 INFO:teuthology.orchestra.run.gibba013.stdout: client mount timeout = 600 2021-02-10T11:26:01.840 INFO:teuthology.orchestra.run.gibba013.stdout: debug client = 20 2021-02-10T11:26:01.840 INFO:teuthology.orchestra.run.gibba013.stdout: debug ms = 1 2021-02-10T11:26:01.840 INFO:teuthology.orchestra.run.gibba013.stdout: pid file = "" 2021-02-10T11:26:01.840 INFO:teuthology.orchestra.run.gibba013.stdout:[mon.a] 2021-02-10T11:26:01.840 INFO:teuthology.orchestra.run.gibba013.stdout:[mon.b] 2021-02-10T11:26:01.840 INFO:teuthology.orchestra.run.gibba013.stdout:[mon.c] 2021-02-10T11:26:01.840 INFO:teuthology.orchestra.run.gibba013.stdout:[mds.a] 2021-02-10T11:26:01.841 INFO:teuthology.orchestra.run.gibba013.stdout:[mds.b] 2021-02-10T11:26:01.841 INFO:teuthology.orchestra.run.gibba013.stdout:[mds] 2021-02-10T11:26:01.841 INFO:teuthology.orchestra.run.gibba013.stdout: debug mds = 20 2021-02-10T11:26:01.841 INFO:teuthology.orchestra.run.gibba013.stdout: debug ms = 1 2021-02-10T11:26:01.841 INFO:teuthology.orchestra.run.gibba013.stdout: mds bal frag = True 2021-02-10T11:26:01.841 INFO:teuthology.orchestra.run.gibba013.stdout: mds bal fragment size max = 10000 2021-02-10T11:26:01.841 INFO:teuthology.orchestra.run.gibba013.stdout: mds bal merge size = 5 2021-02-10T11:26:01.841 INFO:teuthology.orchestra.run.gibba013.stdout: mds bal split bits = 3 2021-02-10T11:26:01.841 INFO:teuthology.orchestra.run.gibba013.stdout: mds bal split size = 100 2021-02-10T11:26:01.841 INFO:teuthology.orchestra.run.gibba013.stdout: mds debug frag = True 2021-02-10T11:26:01.842 INFO:teuthology.orchestra.run.gibba013.stdout: mds debug scatterstat = True 2021-02-10T11:26:01.842 INFO:teuthology.orchestra.run.gibba013.stdout: mds op complaint time = 180 2021-02-10T11:26:01.842 INFO:teuthology.orchestra.run.gibba013.stdout: mds verify scatter = True 2021-02-10T11:26:01.842 INFO:teuthology.orchestra.run.gibba013.stdout: osd op complaint time = 180 2021-02-10T11:26:01.842 INFO:teuthology.orchestra.run.gibba013.stdout:[client.alice] 2021-02-10T11:26:01.842 INFO:teuthology.orchestra.run.gibba013.stdout: keyring = /etc/ceph/ceph.client.alice.keyring 2021-02-10T11:26:01.843 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:26:02.270 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:02.255 7fe610057700 1 Processor -- start 2021-02-10T11:26:02.270 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:02.255 7fe610057700 1 -- start start 2021-02-10T11:26:02.270 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:02.256 7fe610057700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fe6080e53e0 0x7fe6081b0550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:02.270 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:02.256 7fe610057700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fe6081b0b70 0x7fe608081910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:02.270 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:02.256 7fe610057700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe608081f30 0x7fe608082330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:02.270 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:02.256 7fe610057700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6080e5fe0 con 0x7fe6080e53e0 2021-02-10T11:26:02.270 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:02.256 7fe610057700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe608082d50 con 0x7fe6081b0b70 2021-02-10T11:26:02.270 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:02.256 7fe610057700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6080e57e0 con 0x7fe608081f30 2021-02-10T11:26:02.270 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:02.256 7fe60de73700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fe6080e53e0 0x7fe6081b0550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:02.270 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:02.256 7fe60de73700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fe6080e53e0 0x7fe6081b0550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:58420/0 (socket says 172.21.2.102:58420) 2021-02-10T11:26:02.271 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:02.256 7fe60de73700 1 -- 172.21.2.102:0/557389313 learned_addr learned my addr 172.21.2.102:0/557389313 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:26:02.271 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:02.256 7fe60de73700 1 -- 172.21.2.102:0/557389313 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe608081f30 msgr2=0x7fe608082330 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:26:02.271 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:02.256 7fe60de73700 1 --2- 172.21.2.102:0/557389313 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe608081f30 0x7fe608082330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:02.271 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:02.256 7fe60de73700 1 -- 172.21.2.102:0/557389313 >> v2:172.21.2.113:3300/0 conn(0x7fe6081b0b70 msgr2=0x7fe608081910 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:26:02.271 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:02.256 7fe60de73700 1 --2- 172.21.2.102:0/557389313 >> v2:172.21.2.113:3300/0 conn(0x7fe6081b0b70 0x7fe608081910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:02.271 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:02.256 7fe60de73700 1 -- 172.21.2.102:0/557389313 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe608082950 con 0x7fe6080e53e0 2021-02-10T11:26:02.271 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:02.256 7fe60de73700 1 --2- 172.21.2.102:0/557389313 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe6080e53e0 0x7fe6081b0550 secure :-1 s=READY pgs=1563 cs=0 l=1 rev1=1 rx=0x7fe604006700 tx=0x7fe604009b80).ready entity=mon.0 client_cookie=22bc522d687d6eef server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:02.271 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:02.256 7fe60ce71700 1 -- 172.21.2.102:0/557389313 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe60400c0c0 con 0x7fe6080e53e0 2021-02-10T11:26:02.271 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:02.257 7fe60ce71700 1 -- 172.21.2.102:0/557389313 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe60400c3c0 con 0x7fe6080e53e0 2021-02-10T11:26:02.271 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:02.257 7fe60ce71700 1 -- 172.21.2.102:0/557389313 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe60400c820 con 0x7fe6080e53e0 2021-02-10T11:26:02.271 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:02.257 7fe610057700 1 -- 172.21.2.102:0/557389313 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe6080e53e0 msgr2=0x7fe6081b0550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:02.271 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:02.257 7fe610057700 1 --2- 172.21.2.102:0/557389313 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe6080e53e0 0x7fe6081b0550 secure :-1 s=READY pgs=1563 cs=0 l=1 rev1=1 rx=0x7fe604006700 tx=0x7fe604009b80).stop 2021-02-10T11:26:02.271 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:02.257 7fe610057700 1 -- 172.21.2.102:0/557389313 shutdown_connections 2021-02-10T11:26:02.272 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:02.257 7fe610057700 1 --2- 172.21.2.102:0/557389313 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe608081f30 0x7fe608082330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:02.272 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:02.257 7fe610057700 1 --2- 172.21.2.102:0/557389313 >> v2:172.21.2.113:3300/0 conn(0x7fe6081b0b70 0x7fe608081910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:02.272 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:02.257 7fe610057700 1 --2- 172.21.2.102:0/557389313 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe6080e53e0 0x7fe6081b0550 unknown :-1 s=CLOSED pgs=1563 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:02.272 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:02.257 7fe610057700 1 -- 172.21.2.102:0/557389313 >> 172.21.2.102:0/557389313 conn(0x7fe6080ce500 msgr2=0x7fe6080e4610 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:26:02.272 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:02.257 7fe610057700 1 -- 172.21.2.102:0/557389313 shutdown_connections 2021-02-10T11:26:02.272 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:02.257 7fe610057700 1 -- 172.21.2.102:0/557389313 wait complete. 2021-02-10T11:26:02.274 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:02.272 7fe610057700 1 Processor -- start 2021-02-10T11:26:02.274 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:02.272 7fe610057700 1 -- start start 2021-02-10T11:26:02.274 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:02.272 7fe610057700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe6080aaa70 0x7fe6081b1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:02.274 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:02.272 7fe610057700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe6081b22e0 0x7fe608095690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:02.274 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:02.272 7fe610057700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe608095cb0 0x7fe60819da40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:02.274 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:02.272 7fe610057700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6081b26e0 con 0x7fe608095cb0 2021-02-10T11:26:02.274 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:02.272 7fe610057700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6080960b0 con 0x7fe6080aaa70 2021-02-10T11:26:02.274 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:02.272 7fe610057700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6080962d0 con 0x7fe6081b22e0 2021-02-10T11:26:02.274 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:02.272 7fe60e674700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe608095cb0 0x7fe60819da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:02.274 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:02.273 7fe60e674700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe608095cb0 0x7fe60819da40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:58428/0 (socket says 172.21.2.102:58428) 2021-02-10T11:26:02.274 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:02.273 7fe60e674700 1 -- 172.21.2.102:0/3695188514 learned_addr learned my addr 172.21.2.102:0/3695188514 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:26:02.275 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:02.273 7fe60d672700 1 --2- 172.21.2.102:0/3695188514 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe6081b22e0 0x7fe608095690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:02.275 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:02.273 7fe60de73700 1 --2- 172.21.2.102:0/3695188514 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe6080aaa70 0x7fe6081b1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:02.275 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:02.273 7fe60e674700 1 -- 172.21.2.102:0/3695188514 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe6081b22e0 msgr2=0x7fe608095690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:02.275 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:02.273 7fe60e674700 1 --2- 172.21.2.102:0/3695188514 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe6081b22e0 0x7fe608095690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:02.275 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:02.273 7fe60e674700 1 -- 172.21.2.102:0/3695188514 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe6080aaa70 msgr2=0x7fe6081b1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:02.275 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:02.273 7fe60e674700 1 --2- 172.21.2.102:0/3695188514 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe6080aaa70 0x7fe6081b1cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:02.275 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:02.273 7fe60e674700 1 -- 172.21.2.102:0/3695188514 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe60819e060 con 0x7fe608095cb0 2021-02-10T11:26:02.275 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:02.273 7fe60de73700 1 --2- 172.21.2.102:0/3695188514 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe6080aaa70 0x7fe6081b1cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:26:02.275 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:02.273 7fe60d672700 1 --2- 172.21.2.102:0/3695188514 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe6081b22e0 0x7fe608095690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:26:02.275 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:02.273 7fe60e674700 1 --2- 172.21.2.102:0/3695188514 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe608095cb0 0x7fe60819da40 secure :-1 s=READY pgs=1564 cs=0 l=1 rev1=1 rx=0x7fe600009960 tx=0x7fe60000bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:02.275 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:02.273 7fe5fe7fc700 1 -- 172.21.2.102:0/3695188514 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe600003030 con 0x7fe608095cb0 2021-02-10T11:26:02.275 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:02.273 7fe5fe7fc700 1 -- 172.21.2.102:0/3695188514 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe600001310 con 0x7fe608095cb0 2021-02-10T11:26:02.275 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:02.273 7fe610057700 1 -- 172.21.2.102:0/3695188514 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe60819e060 con 0x7fe608095cb0 2021-02-10T11:26:02.276 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:02.273 7fe5fe7fc700 1 -- 172.21.2.102:0/3695188514 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe600001830 con 0x7fe608095cb0 2021-02-10T11:26:02.276 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:02.274 7fe610057700 1 -- 172.21.2.102:0/3695188514 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe60819e060 con 0x7fe608095cb0 2021-02-10T11:26:02.276 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:02.274 7fe5fe7fc700 1 -- 172.21.2.102:0/3695188514 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fe6000014f0 con 0x7fe608095cb0 2021-02-10T11:26:02.276 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:02.274 7fe5fe7fc700 1 --2- 172.21.2.102:0/3695188514 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe5f0018730 0x7fe5f001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:02.276 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:02.274 7fe5fe7fc700 1 -- 172.21.2.102:0/3695188514 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fe600032b70 con 0x7fe608095cb0 2021-02-10T11:26:02.276 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:02.274 7fe60de73700 1 --2- 172.21.2.102:0/3695188514 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe5f0018730 0x7fe5f001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:02.276 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:02.274 7fe60de73700 1 --2- 172.21.2.102:0/3695188514 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe5f0018730 0x7fe5f001ac10 crc :-1 s=READY pgs=711 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:02.304 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:02.305 7fe610057700 1 -- 172.21.2.102:0/3695188514 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6080ed660 con 0x7fe608095cb0 2021-02-10T11:26:02.306 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:02.308 7fe5fe7fc700 1 -- 172.21.2.102:0/3695188514 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fe600001520 con 0x7fe608095cb0 2021-02-10T11:26:02.430 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:02.432 7fe610057700 1 -- 172.21.2.102:0/3695188514 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe608082d50 con 0x7fe608095cb0 2021-02-10T11:26:02.431 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:02.432 7fe5fe7fc700 1 -- 172.21.2.102:0/3695188514 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v66) v1 ==== 74+0+13996 (secure 0 0 0) 0x7fe600001310 con 0x7fe608095cb0 2021-02-10T11:26:02.431 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:26:02.431 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":66,"fsid":"68db30ea-dbce-4aef-a6df-f01c6fea0513","created":"2021-02-10 11:03:06.331557","modified":"2021-02-10 11:25:32.516321","last_up_change":"2021-02-10 11:03:32.690562","last_in_change":"2021-02-10 11:03:32.690562","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:03:37.322519","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:25:29.586765","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":13,"pool_name":"cephfs_data","create_time":"2021-02-10 11:25:30.558626","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"be691b4d-a039-408a-af80-9d53f02b6f1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6818","nonce":36088},{"type":"v1","addr":"172.21.2.102:6820","nonce":36088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6822","nonce":36088},{"type":"v1","addr":"172.21.2.102:6824","nonce":36088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6830","nonce":36088},{"type":"v1","addr":"172.21.2.102:6832","nonce":36088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6826","nonce":36088},{"type":"v1","addr":"172.21.2.102:6828","nonce":36088}]},"public_addr":"172.21.2.102:6820/36088","cluster_addr":"172.21.2.102:6824/36088","heartbeat_back_addr":"172.21.2.102:6832/36088","heartbeat_front_addr":"172.21.2.102:6828/36088","state":["exists","up"]},{"osd":1,"uuid":"64e3ccc8-9c69-4d7c-865e-af8e83822635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6819","nonce":36087},{"type":"v1","addr":"172.21.2.102:6821","nonce":36087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6823","nonce":36087},{"type":"v1","addr":"172.21.2.102:6825","nonce":36087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6831","nonce":36087},{"type":"v1","addr":"172.21.2.102:6833","nonce":36087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6827","nonce":36087},{"type":"v1","addr":"172.21.2.102:6829","nonce":36087}]},"public_addr":"172.21.2.102:6821/36087","cluster_addr":"172.21.2.102:6825/36087","heartbeat_back_addr":"172.21.2.102:6833/36087","heartbeat_front_addr":"172.21.2.102:6829/36087","state":["exists","up"]},{"osd":2,"uuid":"153700af-747f-481f-bdf4-d6072d3a842f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6810","nonce":36089},{"type":"v1","addr":"172.21.2.102:6811","nonce":36089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6812","nonce":36089},{"type":"v1","addr":"172.21.2.102:6813","nonce":36089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6816","nonce":36089},{"type":"v1","addr":"172.21.2.102:6817","nonce":36089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6814","nonce":36089},{"type":"v1","addr":"172.21.2.102:6815","nonce":36089}]},"public_addr":"172.21.2.102:6811/36089","cluster_addr":"172.21.2.102:6813/36089","heartbeat_back_addr":"172.21.2.102:6817/36089","heartbeat_front_addr":"172.21.2.102:6815/36089","state":["exists","up"]},{"osd":3,"uuid":"cbc678f4-83a9-45e6-8830-2f7c232b1ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6802","nonce":36090},{"type":"v1","addr":"172.21.2.102:6803","nonce":36090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6804","nonce":36090},{"type":"v1","addr":"172.21.2.102:6805","nonce":36090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6808","nonce":36090},{"type":"v1","addr":"172.21.2.102:6809","nonce":36090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6806","nonce":36090},{"type":"v1","addr":"172.21.2.102:6807","nonce":36090}]},"public_addr":"172.21.2.102:6803/36090","cluster_addr":"172.21.2.102:6805/36090","heartbeat_back_addr":"172.21.2.102:6809/36090","heartbeat_front_addr":"172.21.2.102:6807/36090","state":["exists","up"]},{"osd":4,"uuid":"c01e4008-b258-4f10-afd2-9e7f37214351","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6808","nonce":36368},{"type":"v1","addr":"172.21.2.113:6811","nonce":36368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6814","nonce":36368},{"type":"v1","addr":"172.21.2.113:6818","nonce":36368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6826","nonce":36368},{"type":"v1","addr":"172.21.2.113:6828","nonce":36368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6820","nonce":36368},{"type":"v1","addr":"172.21.2.113:6824","nonce":36368}]},"public_addr":"172.21.2.113:6811/36368","cluster_addr":"172.21.2.113:6818/36368","heartbeat_back_addr":"172.21.2.113:6828/36368","heartbeat_front_addr":"172.21.2.113:6824/36368","state":["exists","up"]},{"osd":5,"uuid":"551f6007-daf8-4a98-8ee0-c2b16bcccb5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6802","nonce":36366},{"type":"v1","addr":"172.21.2.113:6805","nonce":36366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6807","nonce":36366},{"type":"v1","addr":"172.21.2.113:6810","nonce":36366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6819","nonce":36366},{"type":"v1","addr":"172.21.2.113:6822","nonce":36366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6812","nonce":36366},{"type":"v1","addr":"172.21.2.113:6815","nonce":36366}]},"public_addr":"172.21.2.113:6805/36366","cluster_addr":"172.21.2.113:6810/36366","heartbeat_back_addr":"172.21.2.113:6822/36366","heartbeat_front_addr":"172.21.2.113:6815/36366","state":["exists","up"]},{"osd":6,"uuid":"bf67c040-0e50-48be-b102-c3943bc9ebef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6800","nonce":36367},{"type":"v1","addr":"172.21.2.113:6801","nonce":36367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6803","nonce":36367},{"type":"v1","addr":"172.21.2.113:6804","nonce":36367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6813","nonce":36367},{"type":"v1","addr":"172.21.2.113:6816","nonce":36367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6806","nonce":36367},{"type":"v1","addr":"172.21.2.113:6809","nonce":36367}]},"public_addr":"172.21.2.113:6801/36367","cluster_addr":"172.21.2.113:6804/36367","heartbeat_back_addr":"172.21.2.113:6816/36367","heartbeat_front_addr":"172.21.2.113:6809/36367","state":["exists","up"]},{"osd":7,"uuid":"f5461f80-018d-428f-be89-2eb2bcbe7bb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6817","nonce":36365},{"type":"v1","addr":"172.21.2.113:6821","nonce":36365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6823","nonce":36365},{"type":"v1","addr":"172.21.2.113:6825","nonce":36365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6830","nonce":36365},{"type":"v1","addr":"172.21.2.113:6831","nonce":36365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6827","nonce":36365},{"type":"v1","addr":"172.21.2.113:6829","nonce":36365}]},"public_addr":"172.21.2.113:6821/36365","cluster_addr":"172.21.2.113:6825/36365","heartbeat_back_addr":"172.21.2.113:6831/36365","heartbeat_front_addr":"172.21.2.113:6829/36365","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:26:02.432 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:02.434 7fe610057700 1 -- 172.21.2.102:0/3695188514 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe5f0018730 msgr2=0x7fe5f001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:02.432 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:02.434 7fe610057700 1 --2- 172.21.2.102:0/3695188514 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe5f0018730 0x7fe5f001ac10 crc :-1 s=READY pgs=711 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:02.433 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:02.434 7fe610057700 1 -- 172.21.2.102:0/3695188514 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe608095cb0 msgr2=0x7fe60819da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:02.433 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:02.434 7fe610057700 1 --2- 172.21.2.102:0/3695188514 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe608095cb0 0x7fe60819da40 secure :-1 s=READY pgs=1564 cs=0 l=1 rev1=1 rx=0x7fe600009960 tx=0x7fe60000bcf0).stop 2021-02-10T11:26:02.433 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:02.434 7fe610057700 1 -- 172.21.2.102:0/3695188514 shutdown_connections 2021-02-10T11:26:02.433 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:02.434 7fe610057700 1 --2- 172.21.2.102:0/3695188514 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe608095cb0 0x7fe60819da40 unknown :-1 s=CLOSED pgs=1564 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:02.433 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:02.434 7fe610057700 1 --2- 172.21.2.102:0/3695188514 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe6081b22e0 0x7fe608095690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:02.433 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:02.434 7fe610057700 1 --2- 172.21.2.102:0/3695188514 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe5f0018730 0x7fe5f001ac10 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:02.433 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:02.434 7fe610057700 1 --2- 172.21.2.102:0/3695188514 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe6080aaa70 0x7fe6081b1cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:02.433 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:02.434 7fe610057700 1 -- 172.21.2.102:0/3695188514 >> 172.21.2.102:0/3695188514 conn(0x7fe6080f2240 msgr2=0x7fe60819a9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:26:02.433 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:02.434 7fe610057700 1 -- 172.21.2.102:0/3695188514 shutdown_connections 2021-02-10T11:26:02.433 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:02.434 7fe610057700 1 -- 172.21.2.102:0/3695188514 wait complete. 2021-02-10T11:26:02.441 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:03:37.322519', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 12, 'pool_name': 'cephfs_metadata', 'create_time': '2021-02-10 11:25:29.586765', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '65', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 13, 'pool_name': 'cephfs_data', 'create_time': '2021-02-10 11:25:30.558626', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '66', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-02-10T11:26:02.441 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:26:02.876 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:02.859 7fd983f92700 1 Processor -- start 2021-02-10T11:26:02.876 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:02.859 7fd983f92700 1 -- start start 2021-02-10T11:26:02.876 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:02.860 7fd983f92700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fd97c0d18c0 0x7fd97c083ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:02.876 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:02.860 7fd983f92700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd97c082d50 con 0x7fd97c082920 2021-02-10T11:26:02.876 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:02.860 7fd983f92700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd97c0e57d0 con 0x7fd97c0e53d0 2021-02-10T11:26:02.877 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:02.860 7fd983f92700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd97c1b0d30 con 0x7fd97c0d18c0 2021-02-10T11:26:02.877 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:02.860 7fd9815ad700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fd97c082920 0x7fd97c0832d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:50658/0 (socket says 172.21.2.102:50658) 2021-02-10T11:26:02.877 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:02.860 7fd9815ad700 1 -- 172.21.2.102:0/505832437 learned_addr learned my addr 172.21.2.102:0/505832437 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:26:02.877 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:02.860 7fd9825af700 1 --2- 172.21.2.102:0/505832437 >> v2:172.21.2.113:3300/0 conn(0x7fd97c0d18c0 0x7fd97c083ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:02.877 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:02.861 7fd980dac700 1 -- 172.21.2.102:0/505832437 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2681571507 0 0) 0x7fd96c0009b0 con 0x7fd97c082920 2021-02-10T11:26:02.877 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:02.861 7fd980dac700 1 -- 172.21.2.102:0/505832437 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd968002fe0 con 0x7fd97c082920 2021-02-10T11:26:02.877 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:02.861 7fd980dac700 1 -- 172.21.2.102:0/505832437 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 973722110 0 0) 0x7fd96c001010 con 0x7fd97c082920 2021-02-10T11:26:02.877 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:02.861 7fd980dac700 1 -- 172.21.2.102:0/505832437 >> v2:172.21.2.113:3300/0 conn(0x7fd97c0d18c0 msgr2=0x7fd97c083ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:26:02.877 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:02.861 7fd980dac700 1 --2- 172.21.2.102:0/505832437 >> v2:172.21.2.113:3300/0 conn(0x7fd97c0d18c0 0x7fd97c083ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:02.878 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:02.861 7fd980dac700 1 -- 172.21.2.102:0/505832437 >> v1:172.21.2.113:6789/0 conn(0x7fd97c0e53d0 legacy=0x7fd97c1b0540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:02.878 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:02.861 7fd980dac700 1 -- 172.21.2.102:0/505832437 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd97c0d1cc0 con 0x7fd97c082920 2021-02-10T11:26:02.878 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:02.861 7fd980dac700 1 -- 172.21.2.102:0/505832437 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7fd96c001590 con 0x7fd97c082920 2021-02-10T11:26:02.878 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:02.861 7fd980dac700 1 -- 172.21.2.102:0/505832437 >> v1:172.21.2.102:6789/0 conn(0x7fd97c082920 legacy=0x7fd97c0832d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:02.878 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:02.861 7fd980dac700 1 --2- 172.21.2.102:0/505832437 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd968003a60 0x7fd968005ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:02.878 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:02.861 7fd980dac700 1 --2- 172.21.2.102:0/505832437 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd968006770 0x7fd968008c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:02.878 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:02.861 7fd980dac700 1 --2- 172.21.2.102:0/505832437 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd968009590 0x7fd96800bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:02.878 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:02.861 7fd980dac700 1 -- 172.21.2.102:0/505832437 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd96800c300 con 0x7fd968003a60 2021-02-10T11:26:02.878 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:02.861 7fd980dac700 1 -- 172.21.2.102:0/505832437 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd96800c520 con 0x7fd968009590 2021-02-10T11:26:02.878 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:02.861 7fd980dac700 1 -- 172.21.2.102:0/505832437 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd96800c740 con 0x7fd968006770 2021-02-10T11:26:02.878 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:02.861 7fd981dae700 1 --2- 172.21.2.102:0/505832437 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd968003a60 0x7fd968005ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:02.878 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:02.862 7fd9825af700 1 --2- 172.21.2.102:0/505832437 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd968009590 0x7fd96800bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:02.878 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:02.862 7fd9815ad700 1 --2- 172.21.2.102:0/505832437 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd968006770 0x7fd968008c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:02.879 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:02.862 7fd981dae700 1 -- 172.21.2.102:0/505832437 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd968006770 msgr2=0x7fd968008c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:26:02.879 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:02.862 7fd981dae700 1 --2- 172.21.2.102:0/505832437 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd968006770 0x7fd968008c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:02.879 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:02.862 7fd981dae700 1 -- 172.21.2.102:0/505832437 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd968009590 msgr2=0x7fd96800bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:26:02.879 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:02.862 7fd981dae700 1 --2- 172.21.2.102:0/505832437 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd968009590 0x7fd96800bab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:02.879 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:02.862 7fd981dae700 1 -- 172.21.2.102:0/505832437 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9680509b0 con 0x7fd968003a60 2021-02-10T11:26:02.879 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:02.862 7fd9815ad700 1 --2- 172.21.2.102:0/505832437 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd968006770 0x7fd968008c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:26:02.879 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:02.862 7fd9825af700 1 -- 172.21.2.102:0/505832437 reap_dead start 2021-02-10T11:26:02.879 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:02.862 7fd981dae700 1 --2- 172.21.2.102:0/505832437 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd968003a60 0x7fd968005ed0 secure :-1 s=READY pgs=1566 cs=0 l=1 rev1=1 rx=0x7fd978006ad0 tx=0x7fd978009b40).ready entity=mon.0 client_cookie=667bdfac83459ebb server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:02.879 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:02.862 7fd980dac700 1 -- 172.21.2.102:0/505832437 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd97800c0c0 con 0x7fd968003a60 2021-02-10T11:26:02.879 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:02.862 7fd980dac700 1 -- 172.21.2.102:0/505832437 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd97800c300 con 0x7fd968003a60 2021-02-10T11:26:02.879 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:02.862 7fd980dac700 1 -- 172.21.2.102:0/505832437 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd97800c750 con 0x7fd968003a60 2021-02-10T11:26:02.879 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:02.862 7fd983f92700 1 -- 172.21.2.102:0/505832437 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd968003a60 msgr2=0x7fd968005ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:02.879 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:02.862 7fd983f92700 1 --2- 172.21.2.102:0/505832437 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd968003a60 0x7fd968005ed0 secure :-1 s=READY pgs=1566 cs=0 l=1 rev1=1 rx=0x7fd978006ad0 tx=0x7fd978009b40).stop 2021-02-10T11:26:02.880 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:02.863 7fd983f92700 1 -- 172.21.2.102:0/505832437 shutdown_connections 2021-02-10T11:26:02.880 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:02.863 7fd983f92700 1 --2- 172.21.2.102:0/505832437 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd968003a60 0x7fd968005ed0 unknown :-1 s=CLOSED pgs=1566 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:02.880 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:02.863 7fd983f92700 1 -- 172.21.2.102:0/505832437 >> 172.21.2.102:0/505832437 conn(0x7fd97c0ce4f0 msgr2=0x7fd97c0e4600 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:26:02.880 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:02.863 7fd983f92700 1 -- 172.21.2.102:0/505832437 shutdown_connections 2021-02-10T11:26:02.880 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:02.863 7fd983f92700 1 -- 172.21.2.102:0/505832437 wait complete. 2021-02-10T11:26:02.880 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:02.878 7fd983f92700 1 Processor -- start 2021-02-10T11:26:02.880 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:02.878 7fd983f92700 1 -- start start 2021-02-10T11:26:02.882 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:02.878 7fd983f92700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd97c0a9bc0 0x7fd97c1a0040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:02.883 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:02.878 7fd983f92700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd97c0aa390 0x7fd97c0aa790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:02.883 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:02.878 7fd983f92700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd97c0aadb0 0x7fd97c0c1b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:02.883 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:02.878 7fd983f92700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd97c0aa0e0 con 0x7fd97c0aa390 2021-02-10T11:26:02.883 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:02.878 7fd983f92700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd97c0bf510 con 0x7fd97c0a9bc0 2021-02-10T11:26:02.883 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:02.878 7fd983f92700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd97c0bf730 con 0x7fd97c0aadb0 2021-02-10T11:26:02.883 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:02.878 7fd9815ad700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd97c0aa390 0x7fd97c0aa790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:02.883 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:02.878 7fd9815ad700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd97c0aa390 0x7fd97c0aa790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:58448/0 (socket says 172.21.2.102:58448) 2021-02-10T11:26:02.883 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:02.878 7fd9815ad700 1 -- 172.21.2.102:0/3679100473 learned_addr learned my addr 172.21.2.102:0/3679100473 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:26:02.883 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:02.879 7fd9815ad700 1 -- 172.21.2.102:0/3679100473 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd97c0aadb0 msgr2=0x7fd97c0c1b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:02.883 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:02.879 7fd9815ad700 1 --2- 172.21.2.102:0/3679100473 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd97c0aadb0 0x7fd97c0c1b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:02.883 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:02.879 7fd9815ad700 1 -- 172.21.2.102:0/3679100473 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd97c0a9bc0 msgr2=0x7fd97c1a0040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:02.883 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:02.879 7fd9815ad700 1 --2- 172.21.2.102:0/3679100473 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd97c0a9bc0 0x7fd97c1a0040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:02.884 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:02.879 7fd9815ad700 1 -- 172.21.2.102:0/3679100473 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd97c1a0660 con 0x7fd97c0aa390 2021-02-10T11:26:02.884 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:02.879 7fd9815ad700 1 --2- 172.21.2.102:0/3679100473 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd97c0aa390 0x7fd97c0aa790 secure :-1 s=READY pgs=1567 cs=0 l=1 rev1=1 rx=0x7fd96c001c60 tx=0x7fd96c00a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:02.884 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:02.879 7fd9727fc700 1 -- 172.21.2.102:0/3679100473 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd96c000940 con 0x7fd97c0aa390 2021-02-10T11:26:02.884 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:02.879 7fd9727fc700 1 -- 172.21.2.102:0/3679100473 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd96c00acc0 con 0x7fd97c0aa390 2021-02-10T11:26:02.884 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:02.879 7fd9727fc700 1 -- 172.21.2.102:0/3679100473 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd96c0045d0 con 0x7fd97c0aa390 2021-02-10T11:26:02.884 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:02.879 7fd983f92700 1 -- 172.21.2.102:0/3679100473 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd97c1a0660 con 0x7fd97c0aa390 2021-02-10T11:26:02.884 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:02.879 7fd983f92700 1 -- 172.21.2.102:0/3679100473 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd97c1a0660 con 0x7fd97c0aa390 2021-02-10T11:26:02.884 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:02.880 7fd9727fc700 1 -- 172.21.2.102:0/3679100473 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd96c000dd0 con 0x7fd97c0aa390 2021-02-10T11:26:02.884 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:02.880 7fd9727fc700 1 --2- 172.21.2.102:0/3679100473 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd9640186c0 0x7fd96401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:02.884 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:02.880 7fd9727fc700 1 -- 172.21.2.102:0/3679100473 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fd96c0327a0 con 0x7fd97c0aa390 2021-02-10T11:26:02.884 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:02.880 7fd981dae700 1 --2- 172.21.2.102:0/3679100473 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd9640186c0 0x7fd96401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:02.884 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:02.880 7fd981dae700 1 --2- 172.21.2.102:0/3679100473 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd9640186c0 0x7fd96401abc0 crc :-1 s=READY pgs=712 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:02.908 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:02.910 7fd983f92700 1 -- 172.21.2.102:0/3679100473 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd97c0ed610 con 0x7fd97c0aa390 2021-02-10T11:26:02.911 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:02.912 7fd9727fc700 1 -- 172.21.2.102:0/3679100473 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd96c000940 con 0x7fd97c0aa390 2021-02-10T11:26:03.042 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:03.044 7fd983f92700 1 -- 172.21.2.102:0/3679100473 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7fd97c082d50 con 0x7fd97c0aa390 2021-02-10T11:26:03.042 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:03.044 7fd9727fc700 1 -- 172.21.2.102:0/3679100473 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v66) v1 ==== 92+0+10 (secure 0 0 0) 0x7fd96c000d70 con 0x7fd97c0aa390 2021-02-10T11:26:03.043 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:26:03.044 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:03.046 7fd983f92700 1 -- 172.21.2.102:0/3679100473 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd9640186c0 msgr2=0x7fd96401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:03.045 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:03.046 7fd983f92700 1 --2- 172.21.2.102:0/3679100473 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd9640186c0 0x7fd96401abc0 crc :-1 s=READY pgs=712 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:03.045 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:03.046 7fd983f92700 1 -- 172.21.2.102:0/3679100473 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd97c0aa390 msgr2=0x7fd97c0aa790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:03.045 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:03.046 7fd983f92700 1 --2- 172.21.2.102:0/3679100473 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd97c0aa390 0x7fd97c0aa790 secure :-1 s=READY pgs=1567 cs=0 l=1 rev1=1 rx=0x7fd96c001c60 tx=0x7fd96c00a8d0).stop 2021-02-10T11:26:03.045 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:03.046 7fd983f92700 1 -- 172.21.2.102:0/3679100473 shutdown_connections 2021-02-10T11:26:03.045 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:03.046 7fd983f92700 1 --2- 172.21.2.102:0/3679100473 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd97c0aadb0 0x7fd97c0c1b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:03.045 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:03.046 7fd983f92700 1 --2- 172.21.2.102:0/3679100473 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd97c0aa390 0x7fd97c0aa790 unknown :-1 s=CLOSED pgs=1567 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:03.045 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:03.046 7fd983f92700 1 --2- 172.21.2.102:0/3679100473 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd9640186c0 0x7fd96401abc0 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:03.045 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:03.046 7fd983f92700 1 --2- 172.21.2.102:0/3679100473 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd97c0a9bc0 0x7fd97c1a0040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:03.045 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:03.046 7fd983f92700 1 -- 172.21.2.102:0/3679100473 >> 172.21.2.102:0/3679100473 conn(0x7fd97c0b60c0 msgr2=0x7fd97c0a6390 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:26:03.045 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:03.046 7fd983f92700 1 -- 172.21.2.102:0/3679100473 shutdown_connections 2021-02-10T11:26:03.045 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:03.046 7fd983f92700 1 -- 172.21.2.102:0/3679100473 wait complete. 2021-02-10T11:26:03.053 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-02-10T11:26:03.485 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:03.470 7f9e3ae54700 1 Processor -- start 2021-02-10T11:26:03.485 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:03.470 7f9e3ae54700 1 -- start start 2021-02-10T11:26:03.486 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:03.470 7f9e3ae54700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f9e340e53d0 0x7f9e341b0540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:03.486 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:03.470 7f9e3ae54700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f9e341b0b60 0x7f9e34081910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:03.486 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:03.470 7f9e3ae54700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e34082d50 con 0x7f9e34081f30 2021-02-10T11:26:03.486 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:03.470 7f9e3ae54700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e34082b20 con 0x7f9e340e53d0 2021-02-10T11:26:03.486 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:03.471 7f9e3ae54700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e340e5fd0 con 0x7f9e341b0b60 2021-02-10T11:26:03.486 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:03.471 7f9e38c70700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f9e340e53d0 0x7f9e341b0540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:03.486 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:03.471 7f9e38c70700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f9e340e53d0 0x7f9e341b0540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:58452/0 (socket says 172.21.2.102:58452) 2021-02-10T11:26:03.486 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:03.471 7f9e38c70700 1 -- 172.21.2.102:0/2222174376 learned_addr learned my addr 172.21.2.102:0/2222174376 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:26:03.486 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:03.471 7f9e33fff700 1 --2- 172.21.2.102:0/2222174376 >> v2:172.21.2.113:3300/0 conn(0x7f9e341b0b60 0x7f9e34081910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:03.486 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:03.471 7f9e38c70700 1 -- 172.21.2.102:0/2222174376 >> v2:172.21.2.113:3300/0 conn(0x7f9e341b0b60 msgr2=0x7f9e34081910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:26:03.486 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:03.471 7f9e38c70700 1 --2- 172.21.2.102:0/2222174376 >> v2:172.21.2.113:3300/0 conn(0x7f9e341b0b60 0x7f9e34081910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:03.487 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:03.471 7f9e38c70700 1 -- 172.21.2.102:0/2222174376 >> v1:172.21.2.113:6789/0 conn(0x7f9e34081f30 legacy=0x7f9e34082330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:26:03.487 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:03.471 7f9e38c70700 1 -- 172.21.2.102:0/2222174376 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e340e57d0 con 0x7f9e340e53d0 2021-02-10T11:26:03.487 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:03.471 7f9e33fff700 1 --2- 172.21.2.102:0/2222174376 >> v2:172.21.2.113:3300/0 conn(0x7f9e341b0b60 0x7f9e34081910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:26:03.487 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:03.471 7f9e38c70700 1 --2- 172.21.2.102:0/2222174376 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9e340e53d0 0x7f9e341b0540 secure :-1 s=READY pgs=1568 cs=0 l=1 rev1=1 rx=0x7f9e1c006700 tx=0x7f9e1c009b80).ready entity=mon.0 client_cookie=644a392278b2da97 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:03.487 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:03.472 7f9e337fe700 1 -- 172.21.2.102:0/2222174376 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9e1c00c0c0 con 0x7f9e340e53d0 2021-02-10T11:26:03.487 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:03.472 7f9e337fe700 1 -- 172.21.2.102:0/2222174376 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e1c00c440 con 0x7f9e340e53d0 2021-02-10T11:26:03.487 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:03.472 7f9e337fe700 1 -- 172.21.2.102:0/2222174376 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9e1c00c960 con 0x7f9e340e53d0 2021-02-10T11:26:03.487 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:03.472 7f9e3ae54700 1 -- 172.21.2.102:0/2222174376 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9e340e53d0 msgr2=0x7f9e341b0540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:03.487 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:03.472 7f9e3ae54700 1 --2- 172.21.2.102:0/2222174376 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9e340e53d0 0x7f9e341b0540 secure :-1 s=READY pgs=1568 cs=0 l=1 rev1=1 rx=0x7f9e1c006700 tx=0x7f9e1c009b80).stop 2021-02-10T11:26:03.487 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:03.472 7f9e3ae54700 1 -- 172.21.2.102:0/2222174376 shutdown_connections 2021-02-10T11:26:03.487 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:03.473 7f9e3ae54700 1 --2- 172.21.2.102:0/2222174376 >> v2:172.21.2.113:3300/0 conn(0x7f9e341b0b60 0x7f9e34081910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:03.487 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:03.473 7f9e3ae54700 1 --2- 172.21.2.102:0/2222174376 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9e340e53d0 0x7f9e341b0540 unknown :-1 s=CLOSED pgs=1568 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:03.487 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:03.473 7f9e3ae54700 1 -- 172.21.2.102:0/2222174376 >> 172.21.2.102:0/2222174376 conn(0x7f9e340ce4f0 msgr2=0x7f9e340e4600 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:26:03.488 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:03.473 7f9e3ae54700 1 -- 172.21.2.102:0/2222174376 shutdown_connections 2021-02-10T11:26:03.488 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:03.473 7f9e3ae54700 1 -- 172.21.2.102:0/2222174376 wait complete. 2021-02-10T11:26:03.489 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:03.487 7f9e3ae54700 1 Processor -- start 2021-02-10T11:26:03.489 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:03.487 7f9e3ae54700 1 -- start start 2021-02-10T11:26:03.489 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:03.488 7f9e3ae54700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9e340ad640 0x7f9e340ada40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:03.489 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:03.488 7f9e3ae54700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9e340ede70 0x7f9e340ee270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:03.489 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:03.488 7f9e3ae54700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f9e340ee890 0x7f9e340eec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:03.489 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:03.488 7f9e3ae54700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e340ef370 con 0x7f9e340ad640 2021-02-10T11:26:03.490 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:03.488 7f9e3ae54700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e340be880 con 0x7f9e340ee890 2021-02-10T11:26:03.490 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:03.488 7f9e3ae54700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e340ef8f0 con 0x7f9e340ede70 2021-02-10T11:26:03.490 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:03.488 7f9e38c70700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9e340ad640 0x7f9e340ada40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:03.490 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:03.488 7f9e38c70700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9e340ad640 0x7f9e340ada40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:58460/0 (socket says 172.21.2.102:58460) 2021-02-10T11:26:03.490 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:03.488 7f9e38c70700 1 -- 172.21.2.102:0/3854676653 learned_addr learned my addr 172.21.2.102:0/3854676653 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:26:03.490 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:03.488 7f9e38c70700 1 -- 172.21.2.102:0/3854676653 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9e340ede70 msgr2=0x7f9e340ee270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:03.490 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:03.488 7f9e38c70700 1 --2- 172.21.2.102:0/3854676653 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9e340ede70 0x7f9e340ee270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:03.490 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:03.488 7f9e38c70700 1 -- 172.21.2.102:0/3854676653 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f9e340ee890 msgr2=0x7f9e340eec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:03.490 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:03.488 7f9e38c70700 1 --2- 172.21.2.102:0/3854676653 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f9e340ee890 0x7f9e340eec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:03.490 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:03.488 7f9e38c70700 1 -- 172.21.2.102:0/3854676653 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e340efb10 con 0x7f9e340ad640 2021-02-10T11:26:03.490 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:03.488 7f9e38c70700 1 --2- 172.21.2.102:0/3854676653 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9e340ad640 0x7f9e340ada40 secure :-1 s=READY pgs=1569 cs=0 l=1 rev1=1 rx=0x7f9e1c00b5e0 tx=0x7f9e1c00b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:03.490 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:03.489 7f9e317fa700 1 -- 172.21.2.102:0/3854676653 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9e1c00bd70 con 0x7f9e340ad640 2021-02-10T11:26:03.490 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:03.489 7f9e317fa700 1 -- 172.21.2.102:0/3854676653 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e1c002030 con 0x7f9e340ad640 2021-02-10T11:26:03.491 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:03.489 7f9e317fa700 1 -- 172.21.2.102:0/3854676653 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9e1c002550 con 0x7f9e340ad640 2021-02-10T11:26:03.491 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:03.489 7f9e3ae54700 1 -- 172.21.2.102:0/3854676653 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e340efb10 con 0x7f9e340ad640 2021-02-10T11:26:03.491 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:03.489 7f9e3ae54700 1 -- 172.21.2.102:0/3854676653 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e340efb10 con 0x7f9e340ad640 2021-02-10T11:26:03.491 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:03.489 7f9e317fa700 1 -- 172.21.2.102:0/3854676653 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f9e1c000c90 con 0x7f9e340ad640 2021-02-10T11:26:03.491 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:03.489 7f9e317fa700 1 --2- 172.21.2.102:0/3854676653 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f9e180186c0 0x7f9e1801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:03.491 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:03.489 7f9e317fa700 1 -- 172.21.2.102:0/3854676653 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f9e1c032570 con 0x7f9e340ad640 2021-02-10T11:26:03.491 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:03.489 7f9e33fff700 1 --2- 172.21.2.102:0/3854676653 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f9e180186c0 0x7f9e1801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:03.491 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:03.490 7f9e33fff700 1 --2- 172.21.2.102:0/3854676653 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f9e180186c0 0x7f9e1801abc0 crc :-1 s=READY pgs=713 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:03.519 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:03.520 7f9e3ae54700 1 -- 172.21.2.102:0/3854676653 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e340ed630 con 0x7f9e340ad640 2021-02-10T11:26:03.521 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:03.523 7f9e317fa700 1 -- 172.21.2.102:0/3854676653 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f9e1c025fa0 con 0x7f9e340ad640 2021-02-10T11:26:03.649 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:03.650 7f9e3ae54700 1 -- 172.21.2.102:0/3854676653 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7f9e340f3a00 con 0x7f9e340ad640 2021-02-10T11:26:03.649 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:03.651 7f9e317fa700 1 -- 172.21.2.102:0/3854676653 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v66) v1 ==== 104+0+10 (secure 0 0 0) 0x7f9e1c025fa0 con 0x7f9e340ad640 2021-02-10T11:26:03.650 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:26:03.651 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:03.653 7f9e3ae54700 1 -- 172.21.2.102:0/3854676653 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f9e180186c0 msgr2=0x7f9e1801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:03.651 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:03.653 7f9e3ae54700 1 --2- 172.21.2.102:0/3854676653 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f9e180186c0 0x7f9e1801abc0 crc :-1 s=READY pgs=713 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:03.651 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:03.653 7f9e3ae54700 1 -- 172.21.2.102:0/3854676653 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9e340ad640 msgr2=0x7f9e340ada40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:03.651 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:03.653 7f9e3ae54700 1 --2- 172.21.2.102:0/3854676653 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9e340ad640 0x7f9e340ada40 secure :-1 s=READY pgs=1569 cs=0 l=1 rev1=1 rx=0x7f9e1c00b5e0 tx=0x7f9e1c00b980).stop 2021-02-10T11:26:03.652 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:03.653 7f9e3ae54700 1 -- 172.21.2.102:0/3854676653 shutdown_connections 2021-02-10T11:26:03.652 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:03.653 7f9e3ae54700 1 --2- 172.21.2.102:0/3854676653 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f9e180186c0 0x7f9e1801abc0 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:03.652 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:03.653 7f9e3ae54700 1 --2- 172.21.2.102:0/3854676653 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f9e340ee890 0x7f9e340eec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:03.652 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:03.653 7f9e3ae54700 1 --2- 172.21.2.102:0/3854676653 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9e340ede70 0x7f9e340ee270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:03.652 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:03.653 7f9e3ae54700 1 --2- 172.21.2.102:0/3854676653 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9e340ad640 0x7f9e340ada40 unknown :-1 s=CLOSED pgs=1569 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:03.652 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:03.653 7f9e3ae54700 1 -- 172.21.2.102:0/3854676653 >> 172.21.2.102:0/3854676653 conn(0x7f9e3408f010 msgr2=0x7f9e340c3ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:26:03.652 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:03.653 7f9e3ae54700 1 -- 172.21.2.102:0/3854676653 shutdown_connections 2021-02-10T11:26:03.652 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:03.653 7f9e3ae54700 1 -- 172.21.2.102:0/3854676653 wait complete. 2021-02-10T11:26:03.660 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-02-10T11:26:04.091 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:04.075 7f8bfa5a4700 1 Processor -- start 2021-02-10T11:26:04.091 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:04.075 7f8bfa5a4700 1 -- start start 2021-02-10T11:26:04.091 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:04.076 7f8bfa5a4700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f8bf40e5170 0x7f8bf41b38e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:04.091 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:04.076 7f8bfa5a4700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f8bf41b3f00 0x7f8bf4077510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:04.091 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:04.076 7f8bfa5a4700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8bf40e5570 0x7f8bf4077b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:04.091 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:04.076 7f8bfa5a4700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8bf4078b90 con 0x7f8bf41b3f00 2021-02-10T11:26:04.091 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:04.076 7f8bfa5a4700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8bf4078150 con 0x7f8bf40e5170 2021-02-10T11:26:04.091 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:04.076 7f8bfa5a4700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8bf4078370 con 0x7f8bf40e5570 2021-02-10T11:26:04.092 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:04.076 7f8bf37fe700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f8bf41b3f00 0x7f8bf4077510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:04.092 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:04.076 7f8bf37fe700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f8bf41b3f00 0x7f8bf4077510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:58470/0 (socket says 172.21.2.102:58470) 2021-02-10T11:26:04.092 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:04.076 7f8bf37fe700 1 -- 172.21.2.102:0/4068096637 learned_addr learned my addr 172.21.2.102:0/4068096637 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:26:04.092 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:04.076 7f8bf37fe700 1 -- 172.21.2.102:0/4068096637 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8bf40e5570 msgr2=0x7f8bf4077b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:26:04.092 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:04.076 7f8bf37fe700 1 --2- 172.21.2.102:0/4068096637 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8bf40e5570 0x7f8bf4077b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:04.092 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:04.077 7f8bf37fe700 1 -- 172.21.2.102:0/4068096637 >> v2:172.21.2.113:3300/0 conn(0x7f8bf40e5170 msgr2=0x7f8bf41b38e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:26:04.092 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:04.077 7f8bf37fe700 1 --2- 172.21.2.102:0/4068096637 >> v2:172.21.2.113:3300/0 conn(0x7f8bf40e5170 0x7f8bf41b38e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:04.092 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:04.077 7f8bf37fe700 1 -- 172.21.2.102:0/4068096637 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bf4078590 con 0x7f8bf41b3f00 2021-02-10T11:26:04.092 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:04.077 7f8bf37fe700 1 --2- 172.21.2.102:0/4068096637 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8bf41b3f00 0x7f8bf4077510 secure :-1 s=READY pgs=1570 cs=0 l=1 rev1=1 rx=0x7f8be8006700 tx=0x7f8be8009b80).ready entity=mon.0 client_cookie=64884ebacfb1e0c6 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:04.092 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:04.077 7f8bf2ffd700 1 -- 172.21.2.102:0/4068096637 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8be800c0c0 con 0x7f8bf41b3f00 2021-02-10T11:26:04.092 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:04.077 7f8bf2ffd700 1 -- 172.21.2.102:0/4068096637 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8be800c400 con 0x7f8bf41b3f00 2021-02-10T11:26:04.093 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:04.077 7f8bf2ffd700 1 -- 172.21.2.102:0/4068096637 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8be800c920 con 0x7f8bf41b3f00 2021-02-10T11:26:04.093 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:04.077 7f8bfa5a4700 1 -- 172.21.2.102:0/4068096637 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8bf41b3f00 msgr2=0x7f8bf4077510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:04.093 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:04.077 7f8bfa5a4700 1 --2- 172.21.2.102:0/4068096637 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8bf41b3f00 0x7f8bf4077510 secure :-1 s=READY pgs=1570 cs=0 l=1 rev1=1 rx=0x7f8be8006700 tx=0x7f8be8009b80).stop 2021-02-10T11:26:04.093 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:04.078 7f8bfa5a4700 1 -- 172.21.2.102:0/4068096637 shutdown_connections 2021-02-10T11:26:04.093 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:04.078 7f8bfa5a4700 1 --2- 172.21.2.102:0/4068096637 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8bf40e5570 0x7f8bf4077b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:04.093 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:04.078 7f8bfa5a4700 1 --2- 172.21.2.102:0/4068096637 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8bf41b3f00 0x7f8bf4077510 unknown :-1 s=CLOSED pgs=1570 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:04.093 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:04.078 7f8bfa5a4700 1 --2- 172.21.2.102:0/4068096637 >> v2:172.21.2.113:3300/0 conn(0x7f8bf40e5170 0x7f8bf41b38e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:04.093 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:04.078 7f8bfa5a4700 1 -- 172.21.2.102:0/4068096637 >> 172.21.2.102:0/4068096637 conn(0x7f8bf40ce820 msgr2=0x7f8bf40e4660 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:26:04.093 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:04.078 7f8bfa5a4700 1 -- 172.21.2.102:0/4068096637 shutdown_connections 2021-02-10T11:26:04.093 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:04.078 7f8bfa5a4700 1 -- 172.21.2.102:0/4068096637 wait complete. 2021-02-10T11:26:04.095 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:04.093 7f8bfa5a4700 1 Processor -- start 2021-02-10T11:26:04.095 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:04.093 7f8bfa5a4700 1 -- start start 2021-02-10T11:26:04.095 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:04.093 7f8bfa5a4700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8bf41117f0 0x7f8bf40aeb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:04.095 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:04.093 7f8bfa5a4700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8bf40af160 0x7f8bf40af560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:04.096 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:04.093 7f8bfa5a4700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8bf40bb520 0x7f8bf40bb920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:04.096 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:04.093 7f8bfa5a4700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bf410da90 con 0x7f8bf40bb520 2021-02-10T11:26:04.096 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:04.093 7f8bfa5a4700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bf40afb80 con 0x7f8bf40af160 2021-02-10T11:26:04.096 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:04.093 7f8bfa5a4700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8bf4111bf0 con 0x7f8bf41117f0 2021-02-10T11:26:04.096 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:04.093 7f8bf8bc1700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8bf40bb520 0x7f8bf40bb920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:04.096 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:04.093 7f8bf8bc1700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8bf40bb520 0x7f8bf40bb920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:58476/0 (socket says 172.21.2.102:58476) 2021-02-10T11:26:04.096 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:04.093 7f8bf8bc1700 1 -- 172.21.2.102:0/211516066 learned_addr learned my addr 172.21.2.102:0/211516066 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:26:04.096 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:04.094 7f8bf8bc1700 1 -- 172.21.2.102:0/211516066 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8bf41117f0 msgr2=0x7f8bf40aeb40 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:26:04.096 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:04.094 7f8bf8bc1700 1 --2- 172.21.2.102:0/211516066 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8bf41117f0 0x7f8bf40aeb40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:04.096 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:04.094 7f8bf8bc1700 1 -- 172.21.2.102:0/211516066 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8bf40af160 msgr2=0x7f8bf40af560 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:26:04.096 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:04.094 7f8bf8bc1700 1 --2- 172.21.2.102:0/211516066 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8bf40af160 0x7f8bf40af560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:04.096 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:04.094 7f8bf8bc1700 1 -- 172.21.2.102:0/211516066 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bf40bc9d0 con 0x7f8bf40bb520 2021-02-10T11:26:04.097 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:04.094 7f8bf8bc1700 1 --2- 172.21.2.102:0/211516066 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8bf40bb520 0x7f8bf40bb920 secure :-1 s=READY pgs=1571 cs=0 l=1 rev1=1 rx=0x7f8bec009980 tx=0x7f8bec00bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:04.097 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:04.094 7f8bf0ff9700 1 -- 172.21.2.102:0/211516066 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8bec005870 con 0x7f8bf40bb520 2021-02-10T11:26:04.097 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:04.094 7f8bf0ff9700 1 -- 172.21.2.102:0/211516066 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bec005af0 con 0x7f8bf40bb520 2021-02-10T11:26:04.097 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:04.094 7f8bf0ff9700 1 -- 172.21.2.102:0/211516066 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8bec006030 con 0x7f8bf40bb520 2021-02-10T11:26:04.097 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:04.094 7f8bfa5a4700 1 -- 172.21.2.102:0/211516066 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8bf40bc9d0 con 0x7f8bf40bb520 2021-02-10T11:26:04.097 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:04.094 7f8bfa5a4700 1 -- 172.21.2.102:0/211516066 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8bf40bc9d0 con 0x7f8bf40bb520 2021-02-10T11:26:04.097 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:04.095 7f8bf0ff9700 1 -- 172.21.2.102:0/211516066 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f8bec005cd0 con 0x7f8bf40bb520 2021-02-10T11:26:04.097 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:04.095 7f8bf0ff9700 1 --2- 172.21.2.102:0/211516066 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8bd40186c0 0x7f8bd401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:04.097 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:04.095 7f8bf0ff9700 1 -- 172.21.2.102:0/211516066 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f8bec032910 con 0x7f8bf40bb520 2021-02-10T11:26:04.097 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:04.095 7f8bf3fff700 1 --2- 172.21.2.102:0/211516066 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8bd40186c0 0x7f8bd401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:04.097 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:04.096 7f8bf3fff700 1 --2- 172.21.2.102:0/211516066 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8bd40186c0 0x7f8bd401abc0 crc :-1 s=READY pgs=714 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:04.124 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:04.126 7f8bfa5a4700 1 -- 172.21.2.102:0/211516066 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8bf4105c20 con 0x7f8bf40bb520 2021-02-10T11:26:04.127 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:04.128 7f8bf0ff9700 1 -- 172.21.2.102:0/211516066 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f8bec005dd0 con 0x7f8bf40bb520 2021-02-10T11:26:04.255 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:04.256 7f8bfa5a4700 1 -- 172.21.2.102:0/211516066 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7f8bf40fe290 con 0x7f8bf40bb520 2021-02-10T11:26:04.255 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:04.257 7f8bf0ff9700 1 -- 172.21.2.102:0/211516066 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v66) v1 ==== 100+0+10 (secure 0 0 0) 0x7f8bec005bc0 con 0x7f8bf40bb520 2021-02-10T11:26:04.256 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:26:04.257 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:04.258 7f8bfa5a4700 1 -- 172.21.2.102:0/211516066 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8bd40186c0 msgr2=0x7f8bd401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:04.257 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:04.258 7f8bfa5a4700 1 --2- 172.21.2.102:0/211516066 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8bd40186c0 0x7f8bd401abc0 crc :-1 s=READY pgs=714 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:04.257 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:04.258 7f8bfa5a4700 1 -- 172.21.2.102:0/211516066 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8bf40bb520 msgr2=0x7f8bf40bb920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:04.257 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:04.258 7f8bfa5a4700 1 --2- 172.21.2.102:0/211516066 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8bf40bb520 0x7f8bf40bb920 secure :-1 s=READY pgs=1571 cs=0 l=1 rev1=1 rx=0x7f8bec009980 tx=0x7f8bec00bcd0).stop 2021-02-10T11:26:04.257 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:04.259 7f8bfa5a4700 1 -- 172.21.2.102:0/211516066 shutdown_connections 2021-02-10T11:26:04.257 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:04.259 7f8bfa5a4700 1 --2- 172.21.2.102:0/211516066 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8bf40bb520 0x7f8bf40bb920 unknown :-1 s=CLOSED pgs=1571 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:04.257 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:04.259 7f8bfa5a4700 1 --2- 172.21.2.102:0/211516066 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8bd40186c0 0x7f8bd401abc0 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:04.258 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:04.259 7f8bfa5a4700 1 --2- 172.21.2.102:0/211516066 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8bf40af160 0x7f8bf40af560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:04.258 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:04.259 7f8bfa5a4700 1 --2- 172.21.2.102:0/211516066 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8bf41117f0 0x7f8bf40aeb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:04.258 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:04.259 7f8bfa5a4700 1 -- 172.21.2.102:0/211516066 >> 172.21.2.102:0/211516066 conn(0x7f8bf4105e50 msgr2=0x7f8bf40b22a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:26:04.258 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:04.259 7f8bfa5a4700 1 -- 172.21.2.102:0/211516066 shutdown_connections 2021-02-10T11:26:04.258 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:04.259 7f8bfa5a4700 1 -- 172.21.2.102:0/211516066 wait complete. 2021-02-10T11:26:04.266 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-02-10T11:26:04.266 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:26:04.693 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:04.678 7f2cec253700 1 Processor -- start 2021-02-10T11:26:04.693 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:04.678 7f2cec253700 1 -- start start 2021-02-10T11:26:04.693 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:04.679 7f2cec253700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2ce40e53e0 0x7f2ce41b0550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:04.693 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:04.679 7f2cec253700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f2ce41b0b70 0x7f2ce4081910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:04.693 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:04.679 7f2cec253700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f2ce4081f30 0x7f2ce4082330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:04.693 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:04.679 7f2cec253700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ce4084850 con 0x7f2ce41b0b70 2021-02-10T11:26:04.693 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:04.679 7f2cec253700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ce40e5fe0 con 0x7f2ce4081f30 2021-02-10T11:26:04.694 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:04.679 7f2cec253700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ce4082d50 con 0x7f2ce40e53e0 2021-02-10T11:26:04.694 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:04.679 7f2ce986e700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f2ce41b0b70 0x7f2ce4081910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:04.694 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:04.679 7f2ce986e700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f2ce41b0b70 0x7f2ce4081910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:58480/0 (socket says 172.21.2.102:58480) 2021-02-10T11:26:04.694 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:04.679 7f2ce986e700 1 -- 172.21.2.102:0/604983848 learned_addr learned my addr 172.21.2.102:0/604983848 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:26:04.694 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:04.679 7f2ce986e700 1 -- 172.21.2.102:0/604983848 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2ce40e53e0 msgr2=0x7f2ce41b0550 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:26:04.694 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:04.679 7f2ce986e700 1 --2- 172.21.2.102:0/604983848 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2ce40e53e0 0x7f2ce41b0550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:04.694 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:04.679 7f2ce986e700 1 -- 172.21.2.102:0/604983848 >> v2:172.21.2.113:3300/0 conn(0x7f2ce4081f30 msgr2=0x7f2ce4082330 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:26:04.695 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:04.679 7f2ce986e700 1 --2- 172.21.2.102:0/604983848 >> v2:172.21.2.113:3300/0 conn(0x7f2ce4081f30 0x7f2ce4082330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:04.695 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:04.679 7f2ce986e700 1 -- 172.21.2.102:0/604983848 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ce40e57e0 con 0x7f2ce41b0b70 2021-02-10T11:26:04.695 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:04.679 7f2ce986e700 1 --2- 172.21.2.102:0/604983848 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2ce41b0b70 0x7f2ce4081910 secure :-1 s=READY pgs=1572 cs=0 l=1 rev1=1 rx=0x7f2ce0006700 tx=0x7f2ce0009b00).ready entity=mon.0 client_cookie=35f9abd45d1a2397 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:04.695 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:04.679 7f2ce906d700 1 -- 172.21.2.102:0/604983848 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2ce000c0c0 con 0x7f2ce41b0b70 2021-02-10T11:26:04.695 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:04.680 7f2ce906d700 1 -- 172.21.2.102:0/604983848 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ce000c340 con 0x7f2ce41b0b70 2021-02-10T11:26:04.695 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:04.680 7f2ce906d700 1 -- 172.21.2.102:0/604983848 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2ce000c820 con 0x7f2ce41b0b70 2021-02-10T11:26:04.695 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:04.680 7f2cec253700 1 -- 172.21.2.102:0/604983848 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2ce41b0b70 msgr2=0x7f2ce4081910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:04.695 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:04.680 7f2cec253700 1 --2- 172.21.2.102:0/604983848 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2ce41b0b70 0x7f2ce4081910 secure :-1 s=READY pgs=1572 cs=0 l=1 rev1=1 rx=0x7f2ce0006700 tx=0x7f2ce0009b00).stop 2021-02-10T11:26:04.695 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:04.680 7f2cec253700 1 -- 172.21.2.102:0/604983848 shutdown_connections 2021-02-10T11:26:04.695 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:04.680 7f2cec253700 1 --2- 172.21.2.102:0/604983848 >> v2:172.21.2.113:3300/0 conn(0x7f2ce4081f30 0x7f2ce4082330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:04.695 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:04.680 7f2cec253700 1 --2- 172.21.2.102:0/604983848 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2ce41b0b70 0x7f2ce4081910 unknown :-1 s=CLOSED pgs=1572 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:04.696 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:04.680 7f2cec253700 1 --2- 172.21.2.102:0/604983848 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2ce40e53e0 0x7f2ce41b0550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:04.696 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:04.680 7f2cec253700 1 -- 172.21.2.102:0/604983848 >> 172.21.2.102:0/604983848 conn(0x7f2ce40ce500 msgr2=0x7f2ce40e4610 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:26:04.696 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:04.680 7f2cec253700 1 -- 172.21.2.102:0/604983848 shutdown_connections 2021-02-10T11:26:04.696 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:04.680 7f2cec253700 1 -- 172.21.2.102:0/604983848 wait complete. 2021-02-10T11:26:04.699 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:04.695 7f2cec253700 1 Processor -- start 2021-02-10T11:26:04.699 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:04.695 7f2cec253700 1 -- start start 2021-02-10T11:26:04.699 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:04.695 7f2cec253700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2ce409b400 0x7f2ce418ee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:04.700 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:04.695 7f2cec253700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2ce419e8b0 0x7f2ce40ae940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:04.700 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:04.695 7f2cec253700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2ce40aef60 0x7f2ce409be60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:04.700 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:04.695 7f2cec253700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ce409b8e0 con 0x7f2ce409b400 2021-02-10T11:26:04.700 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:04.695 7f2cec253700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ce409add0 con 0x7f2ce40aef60 2021-02-10T11:26:04.700 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:04.695 7f2cec253700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ce409afb0 con 0x7f2ce419e8b0 2021-02-10T11:26:04.700 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:04.695 7f2cea06f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2ce409b400 0x7f2ce418ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:04.700 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:04.695 7f2cea06f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2ce409b400 0x7f2ce418ee10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:58484/0 (socket says 172.21.2.102:58484) 2021-02-10T11:26:04.700 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:04.695 7f2cea06f700 1 -- 172.21.2.102:0/3367172378 learned_addr learned my addr 172.21.2.102:0/3367172378 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:26:04.700 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:04.695 7f2cea06f700 1 -- 172.21.2.102:0/3367172378 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2ce419e8b0 msgr2=0x7f2ce40ae940 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:26:04.700 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:04.695 7f2cea06f700 1 --2- 172.21.2.102:0/3367172378 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2ce419e8b0 0x7f2ce40ae940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:04.700 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:04.695 7f2cea06f700 1 -- 172.21.2.102:0/3367172378 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2ce40aef60 msgr2=0x7f2ce409be60 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:26:04.700 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:04.695 7f2cea06f700 1 --2- 172.21.2.102:0/3367172378 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2ce40aef60 0x7f2ce409be60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:04.700 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:04.695 7f2cea06f700 1 -- 172.21.2.102:0/3367172378 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ce409c480 con 0x7f2ce409b400 2021-02-10T11:26:04.701 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:04.696 7f2cea06f700 1 --2- 172.21.2.102:0/3367172378 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2ce409b400 0x7f2ce418ee10 secure :-1 s=READY pgs=1573 cs=0 l=1 rev1=1 rx=0x7f2cd800b5c0 tx=0x7f2cd800b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:04.701 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:04.696 7f2cd6ffd700 1 -- 172.21.2.102:0/3367172378 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2cd800bd50 con 0x7f2ce409b400 2021-02-10T11:26:04.701 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:04.696 7f2cd6ffd700 1 -- 172.21.2.102:0/3367172378 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cd8006030 con 0x7f2ce409b400 2021-02-10T11:26:04.701 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:04.696 7f2cec253700 1 -- 172.21.2.102:0/3367172378 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ce409c480 con 0x7f2ce409b400 2021-02-10T11:26:04.701 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:04.696 7f2cec253700 1 -- 172.21.2.102:0/3367172378 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ce409ca10 con 0x7f2ce409b400 2021-02-10T11:26:04.701 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:04.696 7f2cd6ffd700 1 -- 172.21.2.102:0/3367172378 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2cd80054d0 con 0x7f2ce409b400 2021-02-10T11:26:04.701 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:04.697 7f2cd6ffd700 1 -- 172.21.2.102:0/3367172378 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f2cd8006210 con 0x7f2ce409b400 2021-02-10T11:26:04.701 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:04.697 7f2cd6ffd700 1 --2- 172.21.2.102:0/3367172378 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2ccc018730 0x7f2ccc01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:04.701 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:04.697 7f2cd6ffd700 1 -- 172.21.2.102:0/3367172378 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f2cd8032800 con 0x7f2ce409b400 2021-02-10T11:26:04.701 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:04.697 7f2ce986e700 1 --2- 172.21.2.102:0/3367172378 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2ccc018730 0x7f2ccc01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:04.701 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:04.697 7f2ce986e700 1 --2- 172.21.2.102:0/3367172378 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2ccc018730 0x7f2ccc01ac10 crc :-1 s=READY pgs=715 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:04.727 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:04.729 7f2cec253700 1 -- 172.21.2.102:0/3367172378 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ce40ed660 con 0x7f2ce409b400 2021-02-10T11:26:04.729 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:04.731 7f2cd6ffd700 1 -- 172.21.2.102:0/3367172378 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f2cd80009b0 con 0x7f2ce409b400 2021-02-10T11:26:04.859 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:04.861 7f2cec253700 1 -- 172.21.2.102:0/3367172378 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2ce4082d50 con 0x7f2ce409b400 2021-02-10T11:26:04.860 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:04.861 7f2cd6ffd700 1 -- 172.21.2.102:0/3367172378 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) v1 ==== 94+0+2101 (secure 0 0 0) 0x7f2cd800bd50 con 0x7f2ce409b400 2021-02-10T11:26:04.860 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:26:04.860 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":49,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8412,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.102:6835/3414701760","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6834","nonce":3414701760},{"type":"v1","addr":"172.21.2.102:6835","nonce":3414701760}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":44}],"filesystems":[{"mdsmap":{"epoch":48,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:25:31.513947","modified":"2021-02-10 11:25:33.524416","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5813},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5813":{"gid":5813,"name":"b","rank":0,"incarnation":45,"state":"up:active","state_seq":3,"addr":"172.21.2.113:6833/2951398038","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6832","nonce":2951398038},{"type":"v1","addr":"172.21.2.113:6833","nonce":2951398038}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":6}]} 2021-02-10T11:26:04.861 INFO:teuthology.orchestra.run.gibba002.stderr:dumped fsmap epoch 49 2021-02-10T11:26:04.862 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:04.863 7f2cec253700 1 -- 172.21.2.102:0/3367172378 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2ccc018730 msgr2=0x7f2ccc01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:04.862 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:04.863 7f2cec253700 1 --2- 172.21.2.102:0/3367172378 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2ccc018730 0x7f2ccc01ac10 crc :-1 s=READY pgs=715 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:04.862 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:04.863 7f2cec253700 1 -- 172.21.2.102:0/3367172378 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2ce409b400 msgr2=0x7f2ce418ee10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:04.863 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:04.863 7f2cec253700 1 --2- 172.21.2.102:0/3367172378 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2ce409b400 0x7f2ce418ee10 secure :-1 s=READY pgs=1573 cs=0 l=1 rev1=1 rx=0x7f2cd800b5c0 tx=0x7f2cd800b960).stop 2021-02-10T11:26:04.863 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:04.864 7f2cec253700 1 -- 172.21.2.102:0/3367172378 shutdown_connections 2021-02-10T11:26:04.863 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:04.864 7f2cec253700 1 --2- 172.21.2.102:0/3367172378 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2ccc018730 0x7f2ccc01ac10 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:04.863 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:04.864 7f2cec253700 1 --2- 172.21.2.102:0/3367172378 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2ce40aef60 0x7f2ce409be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:04.863 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:04.864 7f2cec253700 1 --2- 172.21.2.102:0/3367172378 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2ce419e8b0 0x7f2ce40ae940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:04.863 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:04.864 7f2cec253700 1 --2- 172.21.2.102:0/3367172378 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2ce409b400 0x7f2ce418ee10 unknown :-1 s=CLOSED pgs=1573 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:04.863 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:04.864 7f2cec253700 1 -- 172.21.2.102:0/3367172378 >> 172.21.2.102:0/3367172378 conn(0x7f2ce40f2240 msgr2=0x7f2ce40ed890 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:26:04.863 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:04.864 7f2cec253700 1 -- 172.21.2.102:0/3367172378 shutdown_connections 2021-02-10T11:26:04.863 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:04.864 7f2cec253700 1 -- 172.21.2.102:0/3367172378 wait complete. 2021-02-10T11:26:05.072 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 48, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:25:31.513947', 'modified': '2021-02-10 11:25:33.524416', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5813}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5813': {'gid': 5813, 'name': 'b', 'rank': 0, 'incarnation': 45, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.113:6833/2951398038', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.113:6832', 'nonce': 2951398038}, {'type': 'v1', 'addr': '172.21.2.113:6833', 'nonce': 2951398038}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [13], 'metadata_pool': 12, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:26:05.072 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:26:05.073 DEBUG:teuthology.orchestra.run.gibba013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:26:05.204 DEBUG:teuthology.orchestra.run.gibba002:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:26:05.208 DEBUG:teuthology.orchestra.run.gibba013:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:26:05.471 INFO:teuthology.orchestra.run.gibba013.stdout:{ 2021-02-10T11:26:05.471 INFO:teuthology.orchestra.run.gibba013.stdout: "cluster_fsid": "68db30ea-dbce-4aef-a6df-f01c6fea0513", 2021-02-10T11:26:05.471 INFO:teuthology.orchestra.run.gibba013.stdout: "whoami": 0, 2021-02-10T11:26:05.472 INFO:teuthology.orchestra.run.gibba013.stdout: "id": 5813, 2021-02-10T11:26:05.472 INFO:teuthology.orchestra.run.gibba013.stdout: "want_state": "up:active", 2021-02-10T11:26:05.472 INFO:teuthology.orchestra.run.gibba013.stdout: "state": "up:active", 2021-02-10T11:26:05.472 INFO:teuthology.orchestra.run.gibba013.stdout: "rank_uptime": 33.956454149999999, 2021-02-10T11:26:05.472 INFO:teuthology.orchestra.run.gibba013.stdout: "mdsmap_epoch": 48, 2021-02-10T11:26:05.472 INFO:teuthology.orchestra.run.gibba013.stdout: "osdmap_epoch": 66, 2021-02-10T11:26:05.472 INFO:teuthology.orchestra.run.gibba013.stdout: "osdmap_epoch_barrier": 66, 2021-02-10T11:26:05.472 INFO:teuthology.orchestra.run.gibba013.stdout: "uptime": 39.592510406999999 2021-02-10T11:26:05.472 INFO:teuthology.orchestra.run.gibba013.stdout:} 2021-02-10T11:26:05.475 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "68db30ea-dbce-4aef-a6df-f01c6fea0513", "whoami": 0, "id": 5813, "want_state": "up:active", "state": "up:active", "rank_uptime": 33.956454149999999, "mdsmap_epoch": 48, "osdmap_epoch": 66, "osdmap_epoch_barrier": 66, "uptime": 39.592510406999999 } 2021-02-10T11:26:05.476 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2021-02-10T11:26:05.476 INFO:tasks.cephfs.fuse_mount:Client client.alice config is {} 2021-02-10T11:26:05.476 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.alice at ubuntu@gibba002.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.1... 2021-02-10T11:26:05.476 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:05.476 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec mkdir -p /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:26:05.811 DEBUG:teuthology.orchestra.run.gibba002:> sudo modprobe fuse 2021-02-10T11:26:06.156 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:26:06.156 DEBUG:teuthology.orchestra.run.gibba002:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:26:06.499 INFO:teuthology.orchestra.run.gibba002.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:26:06.499 INFO:teuthology.orchestra.run.gibba002.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:26:06.499 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:26:06.500 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:06.500 DEBUG:teuthology.orchestra.run.gibba002:> ls /sys/fs/fuse/connections 2021-02-10T11:26:06.834 INFO:teuthology.orchestra.run.gibba002.stdout:43 2021-02-10T11:26:06.834 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [43] 2021-02-10T11:26:06.835 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --client_mountpoint=/volumes/group_0000000000664797/subvolume_0000000000152661/c517f318-1811-438b-85ff-c9e7d6aca87c --name client.alice /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:26:06.838 DEBUG:teuthology.orchestra.run.gibba002:> sudo modprobe fuse 2021-02-10T11:26:07.183 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:26:07.183 DEBUG:teuthology.orchestra.run.gibba002:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:26:07.249 INFO:tasks.cephfs.fuse_mount.ceph-fuse.alice.gibba002.stderr:ceph-fuse[161266]: starting ceph client 2021-02-10T11:26:07.249 INFO:tasks.cephfs.fuse_mount.ceph-fuse.alice.gibba002.stderr:2021-02-10 11:26:07.250 7fe1902f8f80 -1 init, newargv = 0x55c0da676430 newargc=9 2021-02-10T11:26:07.260 INFO:tasks.cephfs.fuse_mount.ceph-fuse.alice.gibba002.stderr:ceph-fuse[161266]: starting fuse 2021-02-10T11:26:07.527 INFO:teuthology.orchestra.run.gibba002.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:26:07.527 INFO:teuthology.orchestra.run.gibba002.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:26:07.527 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:26:07.528 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:07.528 DEBUG:teuthology.orchestra.run.gibba002:> ls /sys/fs/fuse/connections 2021-02-10T11:26:07.862 INFO:teuthology.orchestra.run.gibba002.stdout:43 2021-02-10T11:26:07.862 INFO:teuthology.orchestra.run.gibba002.stdout:44 2021-02-10T11:26:07.863 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43, 44] 2021-02-10T11:26:07.863 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:07.863 DEBUG:teuthology.orchestra.run.gibba002:> sudo python3 -c ' 2021-02-10T11:26:07.863 DEBUG:teuthology.orchestra.run.gibba002:> import glob 2021-02-10T11:26:07.863 DEBUG:teuthology.orchestra.run.gibba002:> import re 2021-02-10T11:26:07.863 DEBUG:teuthology.orchestra.run.gibba002:> import os 2021-02-10T11:26:07.863 DEBUG:teuthology.orchestra.run.gibba002:> import subprocess 2021-02-10T11:26:07.863 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:26:07.863 DEBUG:teuthology.orchestra.run.gibba002:> def find_socket(client_name): 2021-02-10T11:26:07.863 DEBUG:teuthology.orchestra.run.gibba002:> asok_path = "/var/run/ceph/ceph-client.alice.*.asok" 2021-02-10T11:26:07.864 DEBUG:teuthology.orchestra.run.gibba002:> files = glob.glob(asok_path) 2021-02-10T11:26:07.864 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:26:07.864 DEBUG:teuthology.orchestra.run.gibba002:> # Given a non-glob path, it better be there 2021-02-10T11:26:07.864 DEBUG:teuthology.orchestra.run.gibba002:> if "*" not in asok_path: 2021-02-10T11:26:07.864 DEBUG:teuthology.orchestra.run.gibba002:> assert(len(files) == 1) 2021-02-10T11:26:07.864 DEBUG:teuthology.orchestra.run.gibba002:> return files[0] 2021-02-10T11:26:07.864 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:26:07.864 DEBUG:teuthology.orchestra.run.gibba002:> for f in files: 2021-02-10T11:26:07.864 DEBUG:teuthology.orchestra.run.gibba002:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2021-02-10T11:26:07.864 DEBUG:teuthology.orchestra.run.gibba002:> if os.path.exists("/proc/{0}".format(pid)): 2021-02-10T11:26:07.864 DEBUG:teuthology.orchestra.run.gibba002:> return f 2021-02-10T11:26:07.864 DEBUG:teuthology.orchestra.run.gibba002:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2021-02-10T11:26:07.864 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:26:07.864 DEBUG:teuthology.orchestra.run.gibba002:> print(find_socket("client.alice")) 2021-02-10T11:26:07.864 DEBUG:teuthology.orchestra.run.gibba002:> ' 2021-02-10T11:26:08.226 INFO:teuthology.orchestra.run.gibba002.stdout:/var/run/ceph/ceph-client.alice.161266.asok 2021-02-10T11:26:08.228 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.alice.161266.asok 2021-02-10T11:26:08.229 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:08.229 DEBUG:teuthology.orchestra.run.gibba002:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.alice.161266.asok status 2021-02-10T11:26:08.618 INFO:teuthology.orchestra.run.gibba002.stdout:{ 2021-02-10T11:26:08.618 INFO:teuthology.orchestra.run.gibba002.stdout: "metadata": { 2021-02-10T11:26:08.618 INFO:teuthology.orchestra.run.gibba002.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:26:08.618 INFO:teuthology.orchestra.run.gibba002.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:26:08.619 INFO:teuthology.orchestra.run.gibba002.stdout: "entity_id": "alice", 2021-02-10T11:26:08.619 INFO:teuthology.orchestra.run.gibba002.stdout: "hostname": "gibba002", 2021-02-10T11:26:08.619 INFO:teuthology.orchestra.run.gibba002.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2021-02-10T11:26:08.619 INFO:teuthology.orchestra.run.gibba002.stdout: "pid": "161266", 2021-02-10T11:26:08.619 INFO:teuthology.orchestra.run.gibba002.stdout: "root": "/volumes/group_0000000000664797/subvolume_0000000000152661/c517f318-1811-438b-85ff-c9e7d6aca87c" 2021-02-10T11:26:08.619 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:26:08.619 INFO:teuthology.orchestra.run.gibba002.stdout: "dentry_count": 0, 2021-02-10T11:26:08.619 INFO:teuthology.orchestra.run.gibba002.stdout: "dentry_pinned_count": 0, 2021-02-10T11:26:08.619 INFO:teuthology.orchestra.run.gibba002.stdout: "id": 8709, 2021-02-10T11:26:08.619 INFO:teuthology.orchestra.run.gibba002.stdout: "inst": { 2021-02-10T11:26:08.619 INFO:teuthology.orchestra.run.gibba002.stdout: "name": { 2021-02-10T11:26:08.619 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "client", 2021-02-10T11:26:08.619 INFO:teuthology.orchestra.run.gibba002.stdout: "num": 8709 2021-02-10T11:26:08.620 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:26:08.620 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": { 2021-02-10T11:26:08.620 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:26:08.620 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:0", 2021-02-10T11:26:08.620 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 199801610 2021-02-10T11:26:08.620 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:26:08.620 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:26:08.620 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": { 2021-02-10T11:26:08.620 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:26:08.620 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:0", 2021-02-10T11:26:08.620 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 199801610 2021-02-10T11:26:08.620 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:26:08.621 INFO:teuthology.orchestra.run.gibba002.stdout: "inst_str": "client.8709 172.21.2.102:0/199801610", 2021-02-10T11:26:08.621 INFO:teuthology.orchestra.run.gibba002.stdout: "addr_str": "172.21.2.102:0/199801610", 2021-02-10T11:26:08.621 INFO:teuthology.orchestra.run.gibba002.stdout: "inode_count": 1, 2021-02-10T11:26:08.621 INFO:teuthology.orchestra.run.gibba002.stdout: "mds_epoch": 48, 2021-02-10T11:26:08.621 INFO:teuthology.orchestra.run.gibba002.stdout: "osd_epoch": 66, 2021-02-10T11:26:08.621 INFO:teuthology.orchestra.run.gibba002.stdout: "osd_epoch_barrier": 0, 2021-02-10T11:26:08.621 INFO:teuthology.orchestra.run.gibba002.stdout: "blacklisted": false 2021-02-10T11:26:08.621 INFO:teuthology.orchestra.run.gibba002.stdout:} 2021-02-10T11:26:08.622 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:08.623 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:26:08.623 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:26:08.958 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:26:08.959 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2021-02-10T11:26:08.959 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.1 && sudo dd if=/dev/urandom of=data.bin bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:26:09.339 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:26:09.339 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:26:09.339 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0291192 s, 36.0 MB/s 2021-02-10T11:26:09.340 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume authorize cephfs subvolume_0000000000152661 alice --group_name group_0000000000664797 --tenant_id tenant_id --access_level r 2021-02-10T11:26:09.778 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:09.764 7ff746134700 1 Processor -- start 2021-02-10T11:26:09.778 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:09.764 7ff746134700 1 -- start start 2021-02-10T11:26:09.778 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:09.764 7ff746134700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff740082920 0x7ff7400832d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:09.778 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:09.764 7ff746134700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7ff7400838f0 0x7ff740083cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:09.778 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:09.764 7ff746134700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff740082d50 con 0x7ff7400e53d0 2021-02-10T11:26:09.778 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:09.764 7ff746134700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7400e57d0 con 0x7ff7400838f0 2021-02-10T11:26:09.778 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:09.764 7ff746134700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7401b0d30 con 0x7ff740082920 2021-02-10T11:26:09.779 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:09.764 7ff73ffff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7ff7400838f0 0x7ff740083cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:09.779 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:09.764 7ff73ffff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7ff7400838f0 0x7ff740083cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:58510/0 (socket says 172.21.2.102:58510) 2021-02-10T11:26:09.779 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:09.764 7ff73ffff700 1 -- 172.21.2.102:0/1744276767 learned_addr learned my addr 172.21.2.102:0/1744276767 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:26:09.779 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:09.764 7ff73ffff700 1 -- 172.21.2.102:0/1744276767 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff740082920 msgr2=0x7ff7400832d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:26:09.779 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:09.764 7ff73ffff700 1 --2- 172.21.2.102:0/1744276767 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff740082920 0x7ff7400832d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:09.779 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:09.764 7ff73ffff700 1 -- 172.21.2.102:0/1744276767 >> v1:172.21.2.113:6789/0 conn(0x7ff7400e53d0 legacy=0x7ff7401b0540 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:26:09.779 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:09.764 7ff73ffff700 1 -- 172.21.2.102:0/1744276767 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff740084310 con 0x7ff7400838f0 2021-02-10T11:26:09.779 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:09.765 7ff73ffff700 1 --2- 172.21.2.102:0/1744276767 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff7400838f0 0x7ff740083cf0 secure :-1 s=READY pgs=1576 cs=0 l=1 rev1=1 rx=0x7ff738006f00 tx=0x7ff738009f10).ready entity=mon.0 client_cookie=bb32dd44a3f11228 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:09.779 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:09.765 7ff73e7fc700 1 -- 172.21.2.102:0/1744276767 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff73800b3b0 con 0x7ff7400838f0 2021-02-10T11:26:09.779 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:09.765 7ff73e7fc700 1 -- 172.21.2.102:0/1744276767 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff73800b630 con 0x7ff7400838f0 2021-02-10T11:26:09.779 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:09.765 7ff73e7fc700 1 -- 172.21.2.102:0/1744276767 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff73800bb50 con 0x7ff7400838f0 2021-02-10T11:26:09.779 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:09.765 7ff746134700 1 -- 172.21.2.102:0/1744276767 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff7400838f0 msgr2=0x7ff740083cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:09.779 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:09.765 7ff746134700 1 --2- 172.21.2.102:0/1744276767 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff7400838f0 0x7ff740083cf0 secure :-1 s=READY pgs=1576 cs=0 l=1 rev1=1 rx=0x7ff738006f00 tx=0x7ff738009f10).stop 2021-02-10T11:26:09.779 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:09.765 7ff746134700 1 -- 172.21.2.102:0/1744276767 shutdown_connections 2021-02-10T11:26:09.780 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:09.765 7ff746134700 1 --2- 172.21.2.102:0/1744276767 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff740082920 0x7ff7400832d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:09.780 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:09.765 7ff746134700 1 --2- 172.21.2.102:0/1744276767 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff7400838f0 0x7ff740083cf0 unknown :-1 s=CLOSED pgs=1576 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:09.780 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:09.765 7ff746134700 1 -- 172.21.2.102:0/1744276767 >> 172.21.2.102:0/1744276767 conn(0x7ff7400ce4f0 msgr2=0x7ff7400e4600 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:26:09.780 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:09.766 7ff746134700 1 -- 172.21.2.102:0/1744276767 shutdown_connections 2021-02-10T11:26:09.780 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:09.766 7ff746134700 1 -- 172.21.2.102:0/1744276767 wait complete. 2021-02-10T11:26:09.781 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:09.780 7ff746134700 1 Processor -- start 2021-02-10T11:26:09.781 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:09.780 7ff746134700 1 -- start start 2021-02-10T11:26:09.781 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:09.780 7ff746134700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff7401b2a10 0x7ff7401b2160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:09.781 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:09.780 7ff746134700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff74008a570 0x7ff7400b4800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:09.782 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:09.780 7ff746134700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff7401b2e30 0x7ff7401b3230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:09.782 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:09.780 7ff746134700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff740099cb0 con 0x7ff7401b2e30 2021-02-10T11:26:09.782 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:09.780 7ff746134700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7400b4e20 con 0x7ff7401b2a10 2021-02-10T11:26:09.782 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:09.780 7ff746134700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7400b5040 con 0x7ff74008a570 2021-02-10T11:26:09.782 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:09.781 7ff73ffff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff7401b2e30 0x7ff7401b3230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:09.782 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:09.781 7ff73ffff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff7401b2e30 0x7ff7401b3230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:58516/0 (socket says 172.21.2.102:58516) 2021-02-10T11:26:09.782 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:09.781 7ff73ffff700 1 -- 172.21.2.102:0/412888243 learned_addr learned my addr 172.21.2.102:0/412888243 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:26:09.782 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:09.781 7ff73ffff700 1 -- 172.21.2.102:0/412888243 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff74008a570 msgr2=0x7ff7400b4800 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:26:09.782 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:09.781 7ff73ffff700 1 --2- 172.21.2.102:0/412888243 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff74008a570 0x7ff7400b4800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:09.782 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:09.781 7ff73ffff700 1 -- 172.21.2.102:0/412888243 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff7401b2a10 msgr2=0x7ff7401b2160 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:26:09.782 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:09.781 7ff73ffff700 1 --2- 172.21.2.102:0/412888243 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff7401b2a10 0x7ff7401b2160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:09.782 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:09.781 7ff73ffff700 1 -- 172.21.2.102:0/412888243 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff74009a230 con 0x7ff7401b2e30 2021-02-10T11:26:09.782 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:09.781 7ff73ffff700 1 --2- 172.21.2.102:0/412888243 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff7401b2e30 0x7ff7401b3230 secure :-1 s=READY pgs=1577 cs=0 l=1 rev1=1 rx=0x7ff738009390 tx=0x7ff73800bd40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:09.783 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:09.781 7ff737fff700 1 -- 172.21.2.102:0/412888243 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff738003030 con 0x7ff7401b2e30 2021-02-10T11:26:09.783 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:09.781 7ff737fff700 1 -- 172.21.2.102:0/412888243 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff738006c80 con 0x7ff7401b2e30 2021-02-10T11:26:09.783 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:09.781 7ff746134700 1 -- 172.21.2.102:0/412888243 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff74009a230 con 0x7ff7401b2e30 2021-02-10T11:26:09.783 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:09.781 7ff746134700 1 -- 172.21.2.102:0/412888243 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff74009a780 con 0x7ff7401b2e30 2021-02-10T11:26:09.783 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:09.781 7ff737fff700 1 -- 172.21.2.102:0/412888243 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff738001480 con 0x7ff7401b2e30 2021-02-10T11:26:09.783 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:09.782 7ff737fff700 1 -- 172.21.2.102:0/412888243 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ff738001270 con 0x7ff7401b2e30 2021-02-10T11:26:09.783 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:09.782 7ff737fff700 1 --2- 172.21.2.102:0/412888243 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff7240186c0 0x7ff72401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:09.783 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:09.782 7ff737fff700 1 -- 172.21.2.102:0/412888243 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 5573+0+0 (secure 0 0 0) 0x7ff738032b40 con 0x7ff7401b2e30 2021-02-10T11:26:09.783 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:09.782 7ff73f7fe700 1 --2- 172.21.2.102:0/412888243 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff7240186c0 0x7ff72401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:09.783 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:09.782 7ff73f7fe700 1 --2- 172.21.2.102:0/412888243 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff7240186c0 0x7ff72401abc0 crc :-1 s=READY pgs=716 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:09.812 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:09.814 7ff746134700 1 -- 172.21.2.102:0/412888243 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7400ed5e0 con 0x7ff7401b2e30 2021-02-10T11:26:09.815 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:09.817 7ff737fff700 1 -- 172.21.2.102:0/412888243 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ff7380093d0 con 0x7ff7401b2e30 2021-02-10T11:26:09.940 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:09.941 7ff746134700 1 -- 172.21.2.102:0/412888243 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"vol_name": "cephfs", "sub_name": "subvolume_0000000000152661", "target": ["mgr", ""], "tenant_id": "tenant_id", "group_name": "group_0000000000664797", "access_level": "r", "prefix": "fs subvolume authorize", "auth_id": "alice"}) v1 -- 0x7ff74009a780 con 0x7ff7240186c0 2021-02-10T11:26:09.982 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:09.984 7ff737fff700 1 -- 172.21.2.102:0/412888243 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+41 (crc 0 0 0) 0x7ff728006970 con 0x7ff7240186c0 2021-02-10T11:26:10.003 INFO:teuthology.orchestra.run.gibba002.stdout:AQDHwiNg5455ChAAnocKD3rX046PozdDDvByNA== 2021-02-10T11:26:10.005 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:10.006 7ff746134700 1 -- 172.21.2.102:0/412888243 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff7240186c0 msgr2=0x7ff72401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:10.005 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:10.006 7ff746134700 1 --2- 172.21.2.102:0/412888243 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff7240186c0 0x7ff72401abc0 crc :-1 s=READY pgs=716 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:10.005 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:10.006 7ff746134700 1 -- 172.21.2.102:0/412888243 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff7401b2e30 msgr2=0x7ff7401b3230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:10.005 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:10.006 7ff746134700 1 --2- 172.21.2.102:0/412888243 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff7401b2e30 0x7ff7401b3230 secure :-1 s=READY pgs=1577 cs=0 l=1 rev1=1 rx=0x7ff738009390 tx=0x7ff73800bd40).stop 2021-02-10T11:26:10.005 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:10.006 7ff746134700 1 -- 172.21.2.102:0/412888243 shutdown_connections 2021-02-10T11:26:10.005 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:10.006 7ff746134700 1 --2- 172.21.2.102:0/412888243 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff7401b2e30 0x7ff7401b3230 unknown :-1 s=CLOSED pgs=1577 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:10.005 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:10.006 7ff746134700 1 --2- 172.21.2.102:0/412888243 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff74008a570 0x7ff7400b4800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:10.005 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:10.006 7ff746134700 1 --2- 172.21.2.102:0/412888243 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff7240186c0 0x7ff72401abc0 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:10.006 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:10.006 7ff746134700 1 --2- 172.21.2.102:0/412888243 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff7401b2a10 0x7ff7401b2160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:10.006 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:10.006 7ff746134700 1 -- 172.21.2.102:0/412888243 >> 172.21.2.102:0/412888243 conn(0x7ff74008ed60 msgr2=0x7ff7401b38c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:26:10.006 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:10.006 7ff746134700 1 -- 172.21.2.102:0/412888243 shutdown_connections 2021-02-10T11:26:10.006 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:10.007 7ff746134700 1 -- 172.21.2.102:0/412888243 wait complete. 2021-02-10T11:26:10.013 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:10.014 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:26:10.014 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:26:10.348 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:26:10.349 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2021-02-10T11:26:10.349 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@gibba002.front.sepia.ceph.com... 2021-02-10T11:26:10.349 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-02-10T11:26:10.349 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:26:10.705 INFO:tasks.cephfs.fuse_mount.ceph-fuse.alice.gibba002.stderr:ceph-fuse[161266]: fuse finished with error 0 and tester_r 0 2021-02-10T11:26:10.706 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:10.706 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:26:10.706 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:26:11.040 INFO:teuthology.orchestra.run.gibba002.stdout:ext2/ext3 2021-02-10T11:26:11.041 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2021-02-10T11:26:11.041 INFO:teuthology.orchestra.run:waiting for 900 2021-02-10T11:26:11.041 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:26:11.041 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:26:11.416 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:26:11.843 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:11.827 7ff7baaf7700 1 Processor -- start 2021-02-10T11:26:11.843 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:11.827 7ff7baaf7700 1 -- start start 2021-02-10T11:26:11.844 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:11.828 7ff7baaf7700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff7b40e53e0 0x7ff7b41b0550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:11.844 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:11.828 7ff7baaf7700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7ff7b40842e0 0x7ff7b4082100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:11.844 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:11.828 7ff7baaf7700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7b4082d50 con 0x7ff7b41b0b70 2021-02-10T11:26:11.844 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:11.828 7ff7baaf7700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7b4084850 con 0x7ff7b40842e0 2021-02-10T11:26:11.844 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:11.828 7ff7baaf7700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7b40e5fe0 con 0x7ff7b40e53e0 2021-02-10T11:26:11.844 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:11.828 7ff7b3fff700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7ff7b41b0b70 0x7ff7b4081910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:50744/0 (socket says 172.21.2.102:50744) 2021-02-10T11:26:11.844 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:11.828 7ff7b3fff700 1 -- 172.21.2.102:0/2489812592 learned_addr learned my addr 172.21.2.102:0/2489812592 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:26:11.844 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:11.828 7ff7b37fe700 1 -- 172.21.2.102:0/2489812592 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1813532150 0 0) 0x7ff7a4001500 con 0x7ff7b41b0b70 2021-02-10T11:26:11.844 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:11.828 7ff7b37fe700 1 -- 172.21.2.102:0/2489812592 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff7a0002fe0 con 0x7ff7b41b0b70 2021-02-10T11:26:11.844 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:11.829 7ff7b8913700 1 --2- 172.21.2.102:0/2489812592 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff7b40e53e0 0x7ff7b41b0550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:11.844 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:11.828 7ff7b9114700 1 --2- 172.21.2.102:0/2489812592 >> v2:172.21.2.113:3300/0 conn(0x7ff7b40842e0 0x7ff7b4082100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:11.844 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:11.829 7ff7b37fe700 1 -- 172.21.2.102:0/2489812592 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2764145162 0 0) 0x7ff7a4001500 con 0x7ff7b41b0b70 2021-02-10T11:26:11.845 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:11.829 7ff7b37fe700 1 -- 172.21.2.102:0/2489812592 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff7b40e53e0 msgr2=0x7ff7b41b0550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:26:11.845 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:11.829 7ff7b37fe700 1 --2- 172.21.2.102:0/2489812592 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff7b40e53e0 0x7ff7b41b0550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:11.845 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:11.829 7ff7b37fe700 1 -- 172.21.2.102:0/2489812592 >> v2:172.21.2.113:3300/0 conn(0x7ff7b40842e0 msgr2=0x7ff7b4082100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:26:11.845 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:11.829 7ff7b37fe700 1 --2- 172.21.2.102:0/2489812592 >> v2:172.21.2.113:3300/0 conn(0x7ff7b40842e0 0x7ff7b4082100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:11.845 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:11.829 7ff7b37fe700 1 -- 172.21.2.102:0/2489812592 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7b40e57e0 con 0x7ff7b41b0b70 2021-02-10T11:26:11.845 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:11.829 7ff7b37fe700 1 -- 172.21.2.102:0/2489812592 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7ff7a4001a00 con 0x7ff7b41b0b70 2021-02-10T11:26:11.845 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:11.829 7ff7b37fe700 1 -- 172.21.2.102:0/2489812592 >> v1:172.21.2.102:6789/0 conn(0x7ff7b41b0b70 legacy=0x7ff7b4081910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:11.845 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:11.829 7ff7b37fe700 1 --2- 172.21.2.102:0/2489812592 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff7a0003a60 0x7ff7a0005ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:11.845 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:11.829 7ff7b37fe700 1 --2- 172.21.2.102:0/2489812592 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff7a0006790 0x7ff7a0008c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:11.845 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:11.829 7ff7b37fe700 1 --2- 172.21.2.102:0/2489812592 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff7a0009500 0x7ff7a000ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:11.845 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:11.829 7ff7b37fe700 1 -- 172.21.2.102:0/2489812592 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7a00502e0 con 0x7ff7a0003a60 2021-02-10T11:26:11.845 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:11.829 7ff7b37fe700 1 -- 172.21.2.102:0/2489812592 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7a0050500 con 0x7ff7a0009500 2021-02-10T11:26:11.845 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:11.829 7ff7b37fe700 1 -- 172.21.2.102:0/2489812592 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7a0050720 con 0x7ff7a0006790 2021-02-10T11:26:11.846 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:11.829 7ff7b8913700 1 --2- 172.21.2.102:0/2489812592 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff7a0003a60 0x7ff7a0005ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:11.846 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:11.829 7ff7b9114700 1 --2- 172.21.2.102:0/2489812592 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff7a0009500 0x7ff7a000ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:11.846 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:11.829 7ff7b3fff700 1 --2- 172.21.2.102:0/2489812592 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff7a0006790 0x7ff7a0008c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:11.846 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:11.829 7ff7b8913700 1 -- 172.21.2.102:0/2489812592 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff7a0006790 msgr2=0x7ff7a0008c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:26:11.846 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:11.829 7ff7b8913700 1 --2- 172.21.2.102:0/2489812592 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff7a0006790 0x7ff7a0008c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:11.846 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:11.829 7ff7b8913700 1 -- 172.21.2.102:0/2489812592 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff7a0009500 msgr2=0x7ff7a000ba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:26:11.846 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:11.829 7ff7b8913700 1 --2- 172.21.2.102:0/2489812592 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff7a0009500 0x7ff7a000ba20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:11.846 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:11.829 7ff7b8913700 1 -- 172.21.2.102:0/2489812592 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7a00509c0 con 0x7ff7a0003a60 2021-02-10T11:26:11.846 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:11.829 7ff7b9114700 1 --2- 172.21.2.102:0/2489812592 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff7a0009500 0x7ff7a000ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:26:11.846 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:11.829 7ff7b8913700 1 --2- 172.21.2.102:0/2489812592 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff7a0003a60 0x7ff7a0005ef0 secure :-1 s=READY pgs=1579 cs=0 l=1 rev1=1 rx=0x7ff79c006ad0 tx=0x7ff79c009b40).ready entity=mon.0 client_cookie=9c25590fbf615a56 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:11.846 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:11.830 7ff7b37fe700 1 -- 172.21.2.102:0/2489812592 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff79c0066d0 con 0x7ff7a0003a60 2021-02-10T11:26:11.847 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:11.830 7ff7b37fe700 1 -- 172.21.2.102:0/2489812592 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff79c00c0c0 con 0x7ff7a0003a60 2021-02-10T11:26:11.847 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:11.830 7ff7b37fe700 1 -- 172.21.2.102:0/2489812592 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff79c00c5e0 con 0x7ff7a0003a60 2021-02-10T11:26:11.847 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:11.830 7ff7baaf7700 1 -- 172.21.2.102:0/2489812592 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff7a0003a60 msgr2=0x7ff7a0005ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:11.847 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:11.830 7ff7baaf7700 1 --2- 172.21.2.102:0/2489812592 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff7a0003a60 0x7ff7a0005ef0 secure :-1 s=READY pgs=1579 cs=0 l=1 rev1=1 rx=0x7ff79c006ad0 tx=0x7ff79c009b40).stop 2021-02-10T11:26:11.847 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:11.830 7ff7b9114700 1 -- 172.21.2.102:0/2489812592 reap_dead start 2021-02-10T11:26:11.847 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:11.830 7ff7baaf7700 1 -- 172.21.2.102:0/2489812592 shutdown_connections 2021-02-10T11:26:11.847 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:11.830 7ff7baaf7700 1 -- 172.21.2.102:0/2489812592 >> 172.21.2.102:0/2489812592 conn(0x7ff7b40ce500 msgr2=0x7ff7b40e4610 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:26:11.847 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:11.831 7ff7baaf7700 1 -- 172.21.2.102:0/2489812592 shutdown_connections 2021-02-10T11:26:11.847 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:11.831 7ff7baaf7700 1 -- 172.21.2.102:0/2489812592 wait complete. 2021-02-10T11:26:11.849 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:11.845 7ff7baaf7700 1 Processor -- start 2021-02-10T11:26:11.849 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:11.845 7ff7baaf7700 1 -- start start 2021-02-10T11:26:11.849 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:11.846 7ff7baaf7700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff7b4091500 0x7ff7b418f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:11.849 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:11.846 7ff7baaf7700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff7b40a2330 0x7ff7b418f9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:11.849 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:11.846 7ff7baaf7700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff7b40c0f10 0x7ff7b40c1310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:11.849 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:11.846 7ff7baaf7700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7b40c0990 con 0x7ff7b40c0f10 2021-02-10T11:26:11.850 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:11.846 7ff7baaf7700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7b41a4a80 con 0x7ff7b40a2330 2021-02-10T11:26:11.850 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:11.846 7ff7baaf7700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7b41a4c60 con 0x7ff7b4091500 2021-02-10T11:26:11.850 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:11.846 7ff7b9114700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff7b40c0f10 0x7ff7b40c1310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:11.850 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:11.846 7ff7b9114700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff7b40c0f10 0x7ff7b40c1310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:58534/0 (socket says 172.21.2.102:58534) 2021-02-10T11:26:11.850 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:11.846 7ff7b9114700 1 -- 172.21.2.102:0/488246713 learned_addr learned my addr 172.21.2.102:0/488246713 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:26:11.850 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:11.846 7ff7b9114700 1 -- 172.21.2.102:0/488246713 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff7b4091500 msgr2=0x7ff7b418f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:11.850 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:11.846 7ff7b9114700 1 --2- 172.21.2.102:0/488246713 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff7b4091500 0x7ff7b418f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:11.850 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:11.846 7ff7b9114700 1 -- 172.21.2.102:0/488246713 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff7b40a2330 msgr2=0x7ff7b418f9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:11.850 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:11.846 7ff7b9114700 1 --2- 172.21.2.102:0/488246713 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff7b40a2330 0x7ff7b418f9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:11.850 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:11.846 7ff7b9114700 1 -- 172.21.2.102:0/488246713 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7b40c1930 con 0x7ff7b40c0f10 2021-02-10T11:26:11.850 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:11.846 7ff7b9114700 1 --2- 172.21.2.102:0/488246713 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff7b40c0f10 0x7ff7b40c1310 secure :-1 s=READY pgs=1580 cs=0 l=1 rev1=1 rx=0x7ff7ac006f60 tx=0x7ff7ac00b9b0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:11.850 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:11.847 7ff7b17fa700 1 -- 172.21.2.102:0/488246713 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff7ac00bda0 con 0x7ff7b40c0f10 2021-02-10T11:26:11.851 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:11.847 7ff7b17fa700 1 -- 172.21.2.102:0/488246713 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7ac006370 con 0x7ff7b40c0f10 2021-02-10T11:26:11.851 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:11.847 7ff7b17fa700 1 -- 172.21.2.102:0/488246713 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff7ac002220 con 0x7ff7b40c0f10 2021-02-10T11:26:11.851 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:11.847 7ff7baaf7700 1 -- 172.21.2.102:0/488246713 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7b40c1930 con 0x7ff7b40c0f10 2021-02-10T11:26:11.851 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:11.847 7ff7baaf7700 1 -- 172.21.2.102:0/488246713 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7b40c1930 con 0x7ff7b40c0f10 2021-02-10T11:26:11.851 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:11.847 7ff7b17fa700 1 -- 172.21.2.102:0/488246713 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ff7ac002020 con 0x7ff7b40c0f10 2021-02-10T11:26:11.851 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:11.847 7ff7b17fa700 1 --2- 172.21.2.102:0/488246713 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff7980186d0 0x7ff79801abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:11.851 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:11.847 7ff7b17fa700 1 -- 172.21.2.102:0/488246713 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 5573+0+0 (secure 0 0 0) 0x7ff7ac032810 con 0x7ff7b40c0f10 2021-02-10T11:26:11.851 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:11.847 7ff7b8913700 1 --2- 172.21.2.102:0/488246713 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff7980186d0 0x7ff79801abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:11.851 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:11.848 7ff7b8913700 1 --2- 172.21.2.102:0/488246713 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff7980186d0 0x7ff79801abd0 crc :-1 s=READY pgs=717 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:11.876 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:11.878 7ff7baaf7700 1 -- 172.21.2.102:0/488246713 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7b40ed630 con 0x7ff7b40c0f10 2021-02-10T11:26:11.879 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:11.880 7ff7b17fa700 1 -- 172.21.2.102:0/488246713 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ff7ac000a70 con 0x7ff7b40c0f10 2021-02-10T11:26:12.002 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:12.003 7ff7baaf7700 1 -- 172.21.2.102:0/488246713 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff7b40ed630 con 0x7ff7b40c0f10 2021-02-10T11:26:12.002 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:12.004 7ff7b17fa700 1 -- 172.21.2.102:0/488246713 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v66) v1 ==== 74+0+13996 (secure 0 0 0) 0x7ff7ac006370 con 0x7ff7b40c0f10 2021-02-10T11:26:12.002 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:26:12.003 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":66,"fsid":"68db30ea-dbce-4aef-a6df-f01c6fea0513","created":"2021-02-10 11:03:06.331557","modified":"2021-02-10 11:25:32.516321","last_up_change":"2021-02-10 11:03:32.690562","last_in_change":"2021-02-10 11:03:32.690562","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:03:37.322519","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:25:29.586765","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":13,"pool_name":"cephfs_data","create_time":"2021-02-10 11:25:30.558626","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"be691b4d-a039-408a-af80-9d53f02b6f1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6818","nonce":36088},{"type":"v1","addr":"172.21.2.102:6820","nonce":36088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6822","nonce":36088},{"type":"v1","addr":"172.21.2.102:6824","nonce":36088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6830","nonce":36088},{"type":"v1","addr":"172.21.2.102:6832","nonce":36088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6826","nonce":36088},{"type":"v1","addr":"172.21.2.102:6828","nonce":36088}]},"public_addr":"172.21.2.102:6820/36088","cluster_addr":"172.21.2.102:6824/36088","heartbeat_back_addr":"172.21.2.102:6832/36088","heartbeat_front_addr":"172.21.2.102:6828/36088","state":["exists","up"]},{"osd":1,"uuid":"64e3ccc8-9c69-4d7c-865e-af8e83822635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6819","nonce":36087},{"type":"v1","addr":"172.21.2.102:6821","nonce":36087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6823","nonce":36087},{"type":"v1","addr":"172.21.2.102:6825","nonce":36087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6831","nonce":36087},{"type":"v1","addr":"172.21.2.102:6833","nonce":36087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6827","nonce":36087},{"type":"v1","addr":"172.21.2.102:6829","nonce":36087}]},"public_addr":"172.21.2.102:6821/36087","cluster_addr":"172.21.2.102:6825/36087","heartbeat_back_addr":"172.21.2.102:6833/36087","heartbeat_front_addr":"172.21.2.102:6829/36087","state":["exists","up"]},{"osd":2,"uuid":"153700af-747f-481f-bdf4-d6072d3a842f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6810","nonce":36089},{"type":"v1","addr":"172.21.2.102:6811","nonce":36089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6812","nonce":36089},{"type":"v1","addr":"172.21.2.102:6813","nonce":36089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6816","nonce":36089},{"type":"v1","addr":"172.21.2.102:6817","nonce":36089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6814","nonce":36089},{"type":"v1","addr":"172.21.2.102:6815","nonce":36089}]},"public_addr":"172.21.2.102:6811/36089","cluster_addr":"172.21.2.102:6813/36089","heartbeat_back_addr":"172.21.2.102:6817/36089","heartbeat_front_addr":"172.21.2.102:6815/36089","state":["exists","up"]},{"osd":3,"uuid":"cbc678f4-83a9-45e6-8830-2f7c232b1ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6802","nonce":36090},{"type":"v1","addr":"172.21.2.102:6803","nonce":36090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6804","nonce":36090},{"type":"v1","addr":"172.21.2.102:6805","nonce":36090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6808","nonce":36090},{"type":"v1","addr":"172.21.2.102:6809","nonce":36090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6806","nonce":36090},{"type":"v1","addr":"172.21.2.102:6807","nonce":36090}]},"public_addr":"172.21.2.102:6803/36090","cluster_addr":"172.21.2.102:6805/36090","heartbeat_back_addr":"172.21.2.102:6809/36090","heartbeat_front_addr":"172.21.2.102:6807/36090","state":["exists","up"]},{"osd":4,"uuid":"c01e4008-b258-4f10-afd2-9e7f37214351","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6808","nonce":36368},{"type":"v1","addr":"172.21.2.113:6811","nonce":36368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6814","nonce":36368},{"type":"v1","addr":"172.21.2.113:6818","nonce":36368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6826","nonce":36368},{"type":"v1","addr":"172.21.2.113:6828","nonce":36368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6820","nonce":36368},{"type":"v1","addr":"172.21.2.113:6824","nonce":36368}]},"public_addr":"172.21.2.113:6811/36368","cluster_addr":"172.21.2.113:6818/36368","heartbeat_back_addr":"172.21.2.113:6828/36368","heartbeat_front_addr":"172.21.2.113:6824/36368","state":["exists","up"]},{"osd":5,"uuid":"551f6007-daf8-4a98-8ee0-c2b16bcccb5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6802","nonce":36366},{"type":"v1","addr":"172.21.2.113:6805","nonce":36366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6807","nonce":36366},{"type":"v1","addr":"172.21.2.113:6810","nonce":36366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6819","nonce":36366},{"type":"v1","addr":"172.21.2.113:6822","nonce":36366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6812","nonce":36366},{"type":"v1","addr":"172.21.2.113:6815","nonce":36366}]},"public_addr":"172.21.2.113:6805/36366","cluster_addr":"172.21.2.113:6810/36366","heartbeat_back_addr":"172.21.2.113:6822/36366","heartbeat_front_addr":"172.21.2.113:6815/36366","state":["exists","up"]},{"osd":6,"uuid":"bf67c040-0e50-48be-b102-c3943bc9ebef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6800","nonce":36367},{"type":"v1","addr":"172.21.2.113:6801","nonce":36367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6803","nonce":36367},{"type":"v1","addr":"172.21.2.113:6804","nonce":36367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6813","nonce":36367},{"type":"v1","addr":"172.21.2.113:6816","nonce":36367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6806","nonce":36367},{"type":"v1","addr":"172.21.2.113:6809","nonce":36367}]},"public_addr":"172.21.2.113:6801/36367","cluster_addr":"172.21.2.113:6804/36367","heartbeat_back_addr":"172.21.2.113:6816/36367","heartbeat_front_addr":"172.21.2.113:6809/36367","state":["exists","up"]},{"osd":7,"uuid":"f5461f80-018d-428f-be89-2eb2bcbe7bb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6817","nonce":36365},{"type":"v1","addr":"172.21.2.113:6821","nonce":36365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6823","nonce":36365},{"type":"v1","addr":"172.21.2.113:6825","nonce":36365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6830","nonce":36365},{"type":"v1","addr":"172.21.2.113:6831","nonce":36365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6827","nonce":36365},{"type":"v1","addr":"172.21.2.113:6829","nonce":36365}]},"public_addr":"172.21.2.113:6821/36365","cluster_addr":"172.21.2.113:6825/36365","heartbeat_back_addr":"172.21.2.113:6831/36365","heartbeat_front_addr":"172.21.2.113:6829/36365","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:26:12.004 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:12.005 7ff7baaf7700 1 -- 172.21.2.102:0/488246713 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff7980186d0 msgr2=0x7ff79801abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:12.004 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:12.005 7ff7baaf7700 1 --2- 172.21.2.102:0/488246713 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff7980186d0 0x7ff79801abd0 crc :-1 s=READY pgs=717 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:12.004 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:12.005 7ff7baaf7700 1 -- 172.21.2.102:0/488246713 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff7b40c0f10 msgr2=0x7ff7b40c1310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:12.004 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:12.005 7ff7baaf7700 1 --2- 172.21.2.102:0/488246713 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff7b40c0f10 0x7ff7b40c1310 secure :-1 s=READY pgs=1580 cs=0 l=1 rev1=1 rx=0x7ff7ac006f60 tx=0x7ff7ac00b9b0).stop 2021-02-10T11:26:12.004 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:12.006 7ff7baaf7700 1 -- 172.21.2.102:0/488246713 shutdown_connections 2021-02-10T11:26:12.005 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:12.006 7ff7baaf7700 1 --2- 172.21.2.102:0/488246713 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff7b40c0f10 0x7ff7b40c1310 unknown :-1 s=CLOSED pgs=1580 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:12.005 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:12.006 7ff7baaf7700 1 --2- 172.21.2.102:0/488246713 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff7980186d0 0x7ff79801abd0 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:12.005 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:12.006 7ff7baaf7700 1 --2- 172.21.2.102:0/488246713 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff7b40a2330 0x7ff7b418f9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:12.005 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:12.006 7ff7baaf7700 1 --2- 172.21.2.102:0/488246713 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff7b4091500 0x7ff7b418f3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:12.005 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:12.006 7ff7baaf7700 1 -- 172.21.2.102:0/488246713 >> 172.21.2.102:0/488246713 conn(0x7ff7b40a1860 msgr2=0x7ff7b40c3da0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:26:12.005 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:12.006 7ff7baaf7700 1 -- 172.21.2.102:0/488246713 shutdown_connections 2021-02-10T11:26:12.005 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:12.006 7ff7baaf7700 1 -- 172.21.2.102:0/488246713 wait complete. 2021-02-10T11:26:12.013 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:03:37.322519', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 12, 'pool_name': 'cephfs_metadata', 'create_time': '2021-02-10 11:25:29.586765', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '65', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 13, 'pool_name': 'cephfs_data', 'create_time': '2021-02-10 11:25:30.558626', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '66', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-02-10T11:26:12.013 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:26:12.445 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:12.429 7fc566eb4700 1 Processor -- start 2021-02-10T11:26:12.445 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:12.429 7fc566eb4700 1 -- start start 2021-02-10T11:26:12.445 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:12.430 7fc566eb4700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fc5600e5170 0x7fc5601b38e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:12.445 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:12.430 7fc566eb4700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fc5600e5570 0x7fc56007a120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:12.445 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:12.430 7fc566eb4700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc560078b90 con 0x7fc5601b3f00 2021-02-10T11:26:12.445 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:12.430 7fc566eb4700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5600d1900 con 0x7fc5600e5570 2021-02-10T11:26:12.445 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:12.430 7fc566eb4700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5600d1af0 con 0x7fc5600e5170 2021-02-10T11:26:12.445 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:12.430 7fc5654d1700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fc5600e5570 0x7fc56007a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:12.445 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:12.430 7fc5654d1700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fc5600e5570 0x7fc56007a120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:58540/0 (socket says 172.21.2.102:58540) 2021-02-10T11:26:12.445 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:12.430 7fc5654d1700 1 -- 172.21.2.102:0/988100033 learned_addr learned my addr 172.21.2.102:0/988100033 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:26:12.446 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:12.430 7fc5654d1700 1 -- 172.21.2.102:0/988100033 >> v2:172.21.2.113:3300/0 conn(0x7fc5600e5170 msgr2=0x7fc5601b38e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:26:12.446 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:12.430 7fc5654d1700 1 --2- 172.21.2.102:0/988100033 >> v2:172.21.2.113:3300/0 conn(0x7fc5600e5170 0x7fc5601b38e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:12.446 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:12.430 7fc5654d1700 1 -- 172.21.2.102:0/988100033 >> v1:172.21.2.113:6789/0 conn(0x7fc5601b3f00 legacy=0x7fc560077510 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:26:12.446 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:12.430 7fc5654d1700 1 -- 172.21.2.102:0/988100033 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5600d1d10 con 0x7fc5600e5570 2021-02-10T11:26:12.446 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:12.430 7fc5654d1700 1 --2- 172.21.2.102:0/988100033 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc5600e5570 0x7fc56007a120 secure :-1 s=READY pgs=1581 cs=0 l=1 rev1=1 rx=0x7fc558006f00 tx=0x7fc558009f10).ready entity=mon.0 client_cookie=9b2e13af6f66e9f3 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:12.446 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:12.430 7fc55f7fe700 1 -- 172.21.2.102:0/988100033 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc55800b3b0 con 0x7fc5600e5570 2021-02-10T11:26:12.446 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:12.430 7fc55f7fe700 1 -- 172.21.2.102:0/988100033 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc55800b630 con 0x7fc5600e5570 2021-02-10T11:26:12.446 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:12.430 7fc55f7fe700 1 -- 172.21.2.102:0/988100033 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc55800bb50 con 0x7fc5600e5570 2021-02-10T11:26:12.446 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:12.430 7fc566eb4700 1 -- 172.21.2.102:0/988100033 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc5600e5570 msgr2=0x7fc56007a120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:12.446 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:12.430 7fc566eb4700 1 --2- 172.21.2.102:0/988100033 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc5600e5570 0x7fc56007a120 secure :-1 s=READY pgs=1581 cs=0 l=1 rev1=1 rx=0x7fc558006f00 tx=0x7fc558009f10).stop 2021-02-10T11:26:12.446 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:12.431 7fc566eb4700 1 -- 172.21.2.102:0/988100033 shutdown_connections 2021-02-10T11:26:12.446 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:12.431 7fc566eb4700 1 --2- 172.21.2.102:0/988100033 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc5600e5570 0x7fc56007a120 unknown :-1 s=CLOSED pgs=1581 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:12.447 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:12.431 7fc566eb4700 1 --2- 172.21.2.102:0/988100033 >> v2:172.21.2.113:3300/0 conn(0x7fc5600e5170 0x7fc5601b38e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:12.447 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:12.431 7fc566eb4700 1 -- 172.21.2.102:0/988100033 >> 172.21.2.102:0/988100033 conn(0x7fc5600ce820 msgr2=0x7fc5600e4660 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:26:12.447 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:12.431 7fc566eb4700 1 -- 172.21.2.102:0/988100033 shutdown_connections 2021-02-10T11:26:12.447 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:12.431 7fc566eb4700 1 -- 172.21.2.102:0/988100033 wait complete. 2021-02-10T11:26:12.448 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:12.446 7fc566eb4700 1 Processor -- start 2021-02-10T11:26:12.448 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:12.446 7fc566eb4700 1 -- start start 2021-02-10T11:26:12.448 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:12.446 7fc566eb4700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc5600b6d40 0x7fc5601a6640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:12.448 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:12.446 7fc566eb4700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc5600b9510 0x7fc5600b34c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:12.448 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:12.446 7fc566eb4700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc5601a6c60 0x7fc5600b3ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:12.449 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:12.446 7fc566eb4700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5600b4300 con 0x7fc5600b9510 2021-02-10T11:26:12.449 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:12.446 7fc566eb4700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc560111c50 con 0x7fc5601a6c60 2021-02-10T11:26:12.449 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:12.446 7fc566eb4700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5600b7140 con 0x7fc5600b6d40 2021-02-10T11:26:12.449 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:12.446 7fc55ffff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc5600b9510 0x7fc5600b34c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:12.449 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:12.446 7fc55ffff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc5600b9510 0x7fc5600b34c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:58546/0 (socket says 172.21.2.102:58546) 2021-02-10T11:26:12.449 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:12.446 7fc55ffff700 1 -- 172.21.2.102:0/1088024118 learned_addr learned my addr 172.21.2.102:0/1088024118 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:26:12.449 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:12.447 7fc55ffff700 1 -- 172.21.2.102:0/1088024118 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc5600b6d40 msgr2=0x7fc5601a6640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:12.449 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:12.447 7fc55ffff700 1 --2- 172.21.2.102:0/1088024118 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc5600b6d40 0x7fc5601a6640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:12.449 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:12.447 7fc55ffff700 1 -- 172.21.2.102:0/1088024118 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc5601a6c60 msgr2=0x7fc5600b3ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:12.449 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:12.447 7fc55ffff700 1 --2- 172.21.2.102:0/1088024118 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc5601a6c60 0x7fc5600b3ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:12.449 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:12.447 7fc55ffff700 1 -- 172.21.2.102:0/1088024118 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5600b4800 con 0x7fc5600b9510 2021-02-10T11:26:12.450 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:12.447 7fc55ffff700 1 --2- 172.21.2.102:0/1088024118 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc5600b9510 0x7fc5600b34c0 secure :-1 s=READY pgs=1582 cs=0 l=1 rev1=1 rx=0x7fc55000b5b0 tx=0x7fc55000b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:12.450 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:12.447 7fc55d7fa700 1 -- 172.21.2.102:0/1088024118 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc55000bd40 con 0x7fc5600b9510 2021-02-10T11:26:12.450 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:12.447 7fc55d7fa700 1 -- 172.21.2.102:0/1088024118 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc550006020 con 0x7fc5600b9510 2021-02-10T11:26:12.450 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:12.447 7fc566eb4700 1 -- 172.21.2.102:0/1088024118 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5600b4800 con 0x7fc5600b9510 2021-02-10T11:26:12.450 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:12.447 7fc566eb4700 1 -- 172.21.2.102:0/1088024118 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5600b4d90 con 0x7fc5600b9510 2021-02-10T11:26:12.450 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:12.447 7fc55d7fa700 1 -- 172.21.2.102:0/1088024118 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc5500054a0 con 0x7fc5600b9510 2021-02-10T11:26:12.450 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:12.448 7fc55d7fa700 1 -- 172.21.2.102:0/1088024118 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fc550006200 con 0x7fc5600b9510 2021-02-10T11:26:12.450 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:12.448 7fc55d7fa700 1 --2- 172.21.2.102:0/1088024118 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc5440186d0 0x7fc54401abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:12.450 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:12.448 7fc55d7fa700 1 -- 172.21.2.102:0/1088024118 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fc5500327e0 con 0x7fc5600b9510 2021-02-10T11:26:12.450 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:12.448 7fc564cd0700 1 --2- 172.21.2.102:0/1088024118 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc5440186d0 0x7fc54401abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:12.450 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:12.448 7fc564cd0700 1 --2- 172.21.2.102:0/1088024118 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc5440186d0 0x7fc54401abd0 crc :-1 s=READY pgs=718 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:12.479 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:12.479 7fc566eb4700 1 -- 172.21.2.102:0/1088024118 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc56010bc00 con 0x7fc5600b9510 2021-02-10T11:26:12.482 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:12.482 7fc55d7fa700 1 -- 172.21.2.102:0/1088024118 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fc550000c10 con 0x7fc5600b9510 2021-02-10T11:26:12.607 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:12.608 7fc566eb4700 1 -- 172.21.2.102:0/1088024118 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7fc5600fe290 con 0x7fc5600b9510 2021-02-10T11:26:12.608 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:12.608 7fc55d7fa700 1 -- 172.21.2.102:0/1088024118 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v66) v1 ==== 92+0+10 (secure 0 0 0) 0x7fc55000bd40 con 0x7fc5600b9510 2021-02-10T11:26:12.608 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:26:12.609 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:12.610 7fc566eb4700 1 -- 172.21.2.102:0/1088024118 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc5440186d0 msgr2=0x7fc54401abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:12.610 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:12.610 7fc566eb4700 1 --2- 172.21.2.102:0/1088024118 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc5440186d0 0x7fc54401abd0 crc :-1 s=READY pgs=718 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:12.610 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:12.610 7fc566eb4700 1 -- 172.21.2.102:0/1088024118 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc5600b9510 msgr2=0x7fc5600b34c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:12.610 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:12.610 7fc566eb4700 1 --2- 172.21.2.102:0/1088024118 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc5600b9510 0x7fc5600b34c0 secure :-1 s=READY pgs=1582 cs=0 l=1 rev1=1 rx=0x7fc55000b5b0 tx=0x7fc55000b950).stop 2021-02-10T11:26:12.611 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:12.610 7fc566eb4700 1 -- 172.21.2.102:0/1088024118 shutdown_connections 2021-02-10T11:26:12.611 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:12.610 7fc566eb4700 1 --2- 172.21.2.102:0/1088024118 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc5440186d0 0x7fc54401abd0 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:12.611 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:12.610 7fc566eb4700 1 --2- 172.21.2.102:0/1088024118 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc5601a6c60 0x7fc5600b3ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:12.611 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:12.610 7fc566eb4700 1 --2- 172.21.2.102:0/1088024118 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc5600b9510 0x7fc5600b34c0 unknown :-1 s=CLOSED pgs=1582 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:12.611 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:12.610 7fc566eb4700 1 --2- 172.21.2.102:0/1088024118 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc5600b6d40 0x7fc5601a6640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:12.611 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:12.610 7fc566eb4700 1 -- 172.21.2.102:0/1088024118 >> 172.21.2.102:0/1088024118 conn(0x7fc5600bd380 msgr2=0x7fc5600bb9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:26:12.611 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:12.611 7fc566eb4700 1 -- 172.21.2.102:0/1088024118 shutdown_connections 2021-02-10T11:26:12.611 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:12.611 7fc566eb4700 1 -- 172.21.2.102:0/1088024118 wait complete. 2021-02-10T11:26:12.618 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-02-10T11:26:13.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:13.034 7f910adcc700 1 Processor -- start 2021-02-10T11:26:13.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:13.034 7f910adcc700 1 -- start start 2021-02-10T11:26:13.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:13.035 7f910adcc700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f91040e5170 0x7f91041b38e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:13.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:13.035 7f910adcc700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f91041b3f00 0x7f9104077510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:13.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:13.035 7f910adcc700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f91040e5570 0x7f9104077b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:13.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:13.035 7f910adcc700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9104078b90 con 0x7f91040e5570 2021-02-10T11:26:13.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:13.035 7f910adcc700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9104078150 con 0x7f91041b3f00 2021-02-10T11:26:13.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:13.035 7f910adcc700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9104078340 con 0x7f91040e5170 2021-02-10T11:26:13.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:13.035 7f91093e9700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f91040e5570 0x7f9104077b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:13.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:13.035 7f91093e9700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f91040e5570 0x7f9104077b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:58558/0 (socket says 172.21.2.102:58558) 2021-02-10T11:26:13.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:13.035 7f91093e9700 1 -- 172.21.2.102:0/2307520483 learned_addr learned my addr 172.21.2.102:0/2307520483 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:26:13.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:13.035 7f9108be8700 1 --2- 172.21.2.102:0/2307520483 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f91040e5170 0x7f91041b38e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:13.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:13.035 7f9103fff700 1 --2- 172.21.2.102:0/2307520483 >> v2:172.21.2.113:3300/0 conn(0x7f91041b3f00 0x7f9104077510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:13.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:13.035 7f91093e9700 1 -- 172.21.2.102:0/2307520483 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f91040e5170 msgr2=0x7f91041b38e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:26:13.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:13.035 7f91093e9700 1 --2- 172.21.2.102:0/2307520483 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f91040e5170 0x7f91041b38e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:13.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:13.035 7f91093e9700 1 -- 172.21.2.102:0/2307520483 >> v2:172.21.2.113:3300/0 conn(0x7f91041b3f00 msgr2=0x7f9104077510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:26:13.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:13.035 7f91093e9700 1 --2- 172.21.2.102:0/2307520483 >> v2:172.21.2.113:3300/0 conn(0x7f91041b3f00 0x7f9104077510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:13.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:13.035 7f91093e9700 1 -- 172.21.2.102:0/2307520483 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9104078560 con 0x7f91040e5570 2021-02-10T11:26:13.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:13.035 7f9108be8700 1 --2- 172.21.2.102:0/2307520483 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f91040e5170 0x7f91041b38e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:26:13.052 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:13.035 7f91093e9700 1 --2- 172.21.2.102:0/2307520483 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f91040e5570 0x7f9104077b30 secure :-1 s=READY pgs=1583 cs=0 l=1 rev1=1 rx=0x7f90fc006f00 tx=0x7f90fc009f10).ready entity=mon.0 client_cookie=efed2ace377991c9 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:13.052 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:13.035 7f91037fe700 1 -- 172.21.2.102:0/2307520483 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f90fc00b3b0 con 0x7f91040e5570 2021-02-10T11:26:13.052 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:13.035 7f91037fe700 1 -- 172.21.2.102:0/2307520483 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90fc000d30 con 0x7f91040e5570 2021-02-10T11:26:13.052 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:13.035 7f91037fe700 1 -- 172.21.2.102:0/2307520483 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f90fc00b7e0 con 0x7f91040e5570 2021-02-10T11:26:13.052 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:13.035 7f910adcc700 1 -- 172.21.2.102:0/2307520483 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f91040e5570 msgr2=0x7f9104077b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:13.052 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:13.035 7f910adcc700 1 --2- 172.21.2.102:0/2307520483 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f91040e5570 0x7f9104077b30 secure :-1 s=READY pgs=1583 cs=0 l=1 rev1=1 rx=0x7f90fc006f00 tx=0x7f90fc009f10).stop 2021-02-10T11:26:13.052 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:13.036 7f910adcc700 1 -- 172.21.2.102:0/2307520483 shutdown_connections 2021-02-10T11:26:13.052 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:13.036 7f910adcc700 1 --2- 172.21.2.102:0/2307520483 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f91040e5570 0x7f9104077b30 unknown :-1 s=CLOSED pgs=1583 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:13.052 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:13.036 7f910adcc700 1 --2- 172.21.2.102:0/2307520483 >> v2:172.21.2.113:3300/0 conn(0x7f91041b3f00 0x7f9104077510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:13.052 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:13.036 7f910adcc700 1 --2- 172.21.2.102:0/2307520483 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f91040e5170 0x7f91041b38e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:13.052 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:13.036 7f910adcc700 1 -- 172.21.2.102:0/2307520483 >> 172.21.2.102:0/2307520483 conn(0x7f91040ce820 msgr2=0x7f91040e4660 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:26:13.052 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:13.036 7f910adcc700 1 -- 172.21.2.102:0/2307520483 shutdown_connections 2021-02-10T11:26:13.052 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:13.036 7f910adcc700 1 -- 172.21.2.102:0/2307520483 wait complete. 2021-02-10T11:26:13.054 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:13.051 7f910adcc700 1 Processor -- start 2021-02-10T11:26:13.054 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:13.051 7f910adcc700 1 -- start start 2021-02-10T11:26:13.054 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:13.051 7f910adcc700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f91040c3580 0x7f910409f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:13.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:13.051 7f910adcc700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f91040c3980 0x7f910409fd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:13.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:13.051 7f910adcc700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f91040c2b40 0x7f9104108e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:13.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:13.051 7f910adcc700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91040c3000 con 0x7f91040c3580 2021-02-10T11:26:13.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:13.051 7f910adcc700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91040a0320 con 0x7f91040c2b40 2021-02-10T11:26:13.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:13.051 7f910adcc700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91041118b0 con 0x7f91040c3980 2021-02-10T11:26:13.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:13.052 7f9108be8700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f91040c3580 0x7f910409f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:13.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:13.052 7f9108be8700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f91040c3580 0x7f910409f6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:58560/0 (socket says 172.21.2.102:58560) 2021-02-10T11:26:13.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:13.052 7f9108be8700 1 -- 172.21.2.102:0/580966762 learned_addr learned my addr 172.21.2.102:0/580966762 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:26:13.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:13.052 7f91093e9700 1 --2- 172.21.2.102:0/580966762 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f91040c2b40 0x7f9104108e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:13.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:13.052 7f9103fff700 1 --2- 172.21.2.102:0/580966762 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f91040c3980 0x7f910409fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:13.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:13.052 7f9108be8700 1 -- 172.21.2.102:0/580966762 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f91040c3980 msgr2=0x7f910409fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:13.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:13.052 7f9108be8700 1 --2- 172.21.2.102:0/580966762 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f91040c3980 0x7f910409fd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:13.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:13.052 7f9108be8700 1 -- 172.21.2.102:0/580966762 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f91040c2b40 msgr2=0x7f9104108e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:13.056 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:13.052 7f9108be8700 1 --2- 172.21.2.102:0/580966762 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f91040c2b40 0x7f9104108e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:13.056 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:13.052 7f9108be8700 1 -- 172.21.2.102:0/580966762 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9104109460 con 0x7f91040c3580 2021-02-10T11:26:13.056 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:13.052 7f9103fff700 1 --2- 172.21.2.102:0/580966762 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f91040c3980 0x7f910409fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:26:13.056 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:13.052 7f9108be8700 1 --2- 172.21.2.102:0/580966762 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f91040c3580 0x7f910409f6e0 secure :-1 s=READY pgs=1584 cs=0 l=1 rev1=1 rx=0x7f90ec006f20 tx=0x7f90ec00b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:13.056 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:13.052 7f91017fa700 1 -- 172.21.2.102:0/580966762 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f90ec00bd80 con 0x7f91040c3580 2021-02-10T11:26:13.056 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:13.052 7f91017fa700 1 -- 172.21.2.102:0/580966762 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90ec000c90 con 0x7f91040c3580 2021-02-10T11:26:13.056 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:13.052 7f91017fa700 1 -- 172.21.2.102:0/580966762 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f90ec002220 con 0x7f91040c3580 2021-02-10T11:26:13.056 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:13.052 7f910adcc700 1 -- 172.21.2.102:0/580966762 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9104109460 con 0x7f91040c3580 2021-02-10T11:26:13.056 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:13.052 7f910adcc700 1 -- 172.21.2.102:0/580966762 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91041099f0 con 0x7f91040c3580 2021-02-10T11:26:13.056 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:13.053 7f91017fa700 1 -- 172.21.2.102:0/580966762 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f90ec00a000 con 0x7f91040c3580 2021-02-10T11:26:13.056 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:13.053 7f91017fa700 1 --2- 172.21.2.102:0/580966762 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f90e80186d0 0x7f90e801abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:13.056 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:13.053 7f91017fa700 1 -- 172.21.2.102:0/580966762 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f90ec0327c0 con 0x7f91040c3580 2021-02-10T11:26:13.056 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:13.053 7f9103fff700 1 --2- 172.21.2.102:0/580966762 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f90e80186d0 0x7f90e801abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:13.057 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:13.053 7f9103fff700 1 --2- 172.21.2.102:0/580966762 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f90e80186d0 0x7f90e801abd0 crc :-1 s=READY pgs=719 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:13.084 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:13.084 7f910adcc700 1 -- 172.21.2.102:0/580966762 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9104105c60 con 0x7f91040c3580 2021-02-10T11:26:13.086 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:13.087 7f91017fa700 1 -- 172.21.2.102:0/580966762 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f90ec000960 con 0x7f91040c3580 2021-02-10T11:26:13.214 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:13.215 7f910adcc700 1 -- 172.21.2.102:0/580966762 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7f91040fe290 con 0x7f91040c3580 2021-02-10T11:26:13.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:13.215 7f91017fa700 1 -- 172.21.2.102:0/580966762 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v66) v1 ==== 104+0+10 (secure 0 0 0) 0x7f90ec00bd80 con 0x7f91040c3580 2021-02-10T11:26:13.215 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:26:13.216 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:13.217 7f910adcc700 1 -- 172.21.2.102:0/580966762 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f90e80186d0 msgr2=0x7f90e801abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:13.216 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:13.217 7f910adcc700 1 --2- 172.21.2.102:0/580966762 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f90e80186d0 0x7f90e801abd0 crc :-1 s=READY pgs=719 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:13.216 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:13.217 7f910adcc700 1 -- 172.21.2.102:0/580966762 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f91040c3580 msgr2=0x7f910409f6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:13.216 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:13.217 7f910adcc700 1 --2- 172.21.2.102:0/580966762 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f91040c3580 0x7f910409f6e0 secure :-1 s=READY pgs=1584 cs=0 l=1 rev1=1 rx=0x7f90ec006f20 tx=0x7f90ec00b990).stop 2021-02-10T11:26:13.217 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:13.217 7f910adcc700 1 -- 172.21.2.102:0/580966762 shutdown_connections 2021-02-10T11:26:13.217 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:13.217 7f910adcc700 1 --2- 172.21.2.102:0/580966762 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f90e80186d0 0x7f90e801abd0 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:13.217 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:13.217 7f910adcc700 1 --2- 172.21.2.102:0/580966762 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f91040c2b40 0x7f9104108e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:13.217 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:13.217 7f910adcc700 1 --2- 172.21.2.102:0/580966762 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f91040c3980 0x7f910409fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:13.217 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:13.217 7f910adcc700 1 --2- 172.21.2.102:0/580966762 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f91040c3580 0x7f910409f6e0 unknown :-1 s=CLOSED pgs=1584 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:13.217 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:13.217 7f910adcc700 1 -- 172.21.2.102:0/580966762 >> 172.21.2.102:0/580966762 conn(0x7f91040a3c70 msgr2=0x7f91040c0170 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:26:13.217 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:13.217 7f910adcc700 1 -- 172.21.2.102:0/580966762 shutdown_connections 2021-02-10T11:26:13.217 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:13.217 7f910adcc700 1 -- 172.21.2.102:0/580966762 wait complete. 2021-02-10T11:26:13.225 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-02-10T11:26:13.656 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:13.639 7f8aeae12700 1 Processor -- start 2021-02-10T11:26:13.656 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:13.639 7f8aeae12700 1 -- start start 2021-02-10T11:26:13.656 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:13.640 7f8aeae12700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f8ae40e5170 0x7f8ae41b38e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:13.656 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:13.640 7f8aeae12700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8ae41b3f00 0x7f8ae4077510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:13.656 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:13.640 7f8aeae12700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f8ae40e5570 0x7f8ae4077b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:13.656 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:13.640 7f8aeae12700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ae4078b90 con 0x7f8ae40e5570 2021-02-10T11:26:13.656 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:13.640 7f8aeae12700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ae4078150 con 0x7f8ae40e5170 2021-02-10T11:26:13.656 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:13.640 7f8aeae12700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ae4078340 con 0x7f8ae41b3f00 2021-02-10T11:26:13.656 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:13.640 7f8ae942f700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f8ae40e5570 0x7f8ae4077b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:13.656 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:13.640 7f8ae942f700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f8ae40e5570 0x7f8ae4077b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:58568/0 (socket says 172.21.2.102:58568) 2021-02-10T11:26:13.657 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:13.640 7f8ae942f700 1 -- 172.21.2.102:0/487864214 learned_addr learned my addr 172.21.2.102:0/487864214 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:26:13.657 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:13.640 7f8ae3fff700 1 --2- 172.21.2.102:0/487864214 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8ae41b3f00 0x7f8ae4077510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:13.657 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:13.640 7f8ae8c2e700 1 --2- 172.21.2.102:0/487864214 >> v2:172.21.2.113:3300/0 conn(0x7f8ae40e5170 0x7f8ae41b38e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:13.657 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:13.641 7f8ae942f700 1 -- 172.21.2.102:0/487864214 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8ae41b3f00 msgr2=0x7f8ae4077510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:26:13.657 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:13.641 7f8ae942f700 1 --2- 172.21.2.102:0/487864214 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8ae41b3f00 0x7f8ae4077510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:13.657 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:13.641 7f8ae942f700 1 -- 172.21.2.102:0/487864214 >> v2:172.21.2.113:3300/0 conn(0x7f8ae40e5170 msgr2=0x7f8ae41b38e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:26:13.657 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:13.641 7f8ae942f700 1 --2- 172.21.2.102:0/487864214 >> v2:172.21.2.113:3300/0 conn(0x7f8ae40e5170 0x7f8ae41b38e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:13.657 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:13.641 7f8ae942f700 1 -- 172.21.2.102:0/487864214 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ae4078560 con 0x7f8ae40e5570 2021-02-10T11:26:13.657 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:13.641 7f8ae8c2e700 1 --2- 172.21.2.102:0/487864214 >> v2:172.21.2.113:3300/0 conn(0x7f8ae40e5170 0x7f8ae41b38e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:26:13.657 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:13.641 7f8ae3fff700 1 --2- 172.21.2.102:0/487864214 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8ae41b3f00 0x7f8ae4077510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:26:13.657 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:13.641 7f8ae942f700 1 --2- 172.21.2.102:0/487864214 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8ae40e5570 0x7f8ae4077b30 secure :-1 s=READY pgs=1585 cs=0 l=1 rev1=1 rx=0x7f8adc006b80 tx=0x7f8adc009b90).ready entity=mon.0 client_cookie=3271862aac793ed3 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:13.657 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:13.641 7f8ae37fe700 1 -- 172.21.2.102:0/487864214 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8adc0022e0 con 0x7f8ae40e5570 2021-02-10T11:26:13.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:13.641 7f8ae37fe700 1 -- 172.21.2.102:0/487864214 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8adc0009b0 con 0x7f8ae40e5570 2021-02-10T11:26:13.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:13.641 7f8ae37fe700 1 -- 172.21.2.102:0/487864214 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8adc0028a0 con 0x7f8ae40e5570 2021-02-10T11:26:13.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:13.641 7f8aeae12700 1 -- 172.21.2.102:0/487864214 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8ae40e5570 msgr2=0x7f8ae4077b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:13.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:13.641 7f8aeae12700 1 --2- 172.21.2.102:0/487864214 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8ae40e5570 0x7f8ae4077b30 secure :-1 s=READY pgs=1585 cs=0 l=1 rev1=1 rx=0x7f8adc006b80 tx=0x7f8adc009b90).stop 2021-02-10T11:26:13.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:13.642 7f8aeae12700 1 -- 172.21.2.102:0/487864214 shutdown_connections 2021-02-10T11:26:13.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:13.642 7f8aeae12700 1 --2- 172.21.2.102:0/487864214 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8ae41b3f00 0x7f8ae4077510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:13.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:13.642 7f8aeae12700 1 --2- 172.21.2.102:0/487864214 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8ae40e5570 0x7f8ae4077b30 unknown :-1 s=CLOSED pgs=1585 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:13.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:13.642 7f8aeae12700 1 --2- 172.21.2.102:0/487864214 >> v2:172.21.2.113:3300/0 conn(0x7f8ae40e5170 0x7f8ae41b38e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:13.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:13.642 7f8aeae12700 1 -- 172.21.2.102:0/487864214 >> 172.21.2.102:0/487864214 conn(0x7f8ae40ce820 msgr2=0x7f8ae40e4660 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:26:13.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:13.642 7f8aeae12700 1 -- 172.21.2.102:0/487864214 shutdown_connections 2021-02-10T11:26:13.659 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:13.642 7f8aeae12700 1 -- 172.21.2.102:0/487864214 wait complete. 2021-02-10T11:26:13.660 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:13.657 7f8aeae12700 1 Processor -- start 2021-02-10T11:26:13.660 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:13.657 7f8aeae12700 1 -- start start 2021-02-10T11:26:13.660 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:13.657 7f8aeae12700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8ae41a0160 0x7f8ae40ac950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:13.661 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:13.657 7f8aeae12700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8ae40acf70 0x7f8ae4091cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:13.661 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:13.657 7f8aeae12700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8ae40922e0 0x7f8ae40926e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:13.661 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:13.657 7f8aeae12700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ae40ad370 con 0x7f8ae41a0160 2021-02-10T11:26:13.661 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:13.657 7f8aeae12700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ae40ad710 con 0x7f8ae40922e0 2021-02-10T11:26:13.661 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:13.657 7f8aeae12700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ae4092d00 con 0x7f8ae40acf70 2021-02-10T11:26:13.661 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:13.657 7f8ae8c2e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8ae41a0160 0x7f8ae40ac950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:13.661 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:13.657 7f8ae8c2e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8ae41a0160 0x7f8ae40ac950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:58572/0 (socket says 172.21.2.102:58572) 2021-02-10T11:26:13.661 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:13.657 7f8ae8c2e700 1 -- 172.21.2.102:0/61036071 learned_addr learned my addr 172.21.2.102:0/61036071 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:26:13.661 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:13.658 7f8ae3fff700 1 --2- 172.21.2.102:0/61036071 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8ae40acf70 0x7f8ae4091cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:13.661 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:13.658 7f8ae942f700 1 --2- 172.21.2.102:0/61036071 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8ae40922e0 0x7f8ae40926e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:13.661 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:13.658 7f8ae8c2e700 1 -- 172.21.2.102:0/61036071 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8ae40acf70 msgr2=0x7f8ae4091cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:13.661 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:13.658 7f8ae8c2e700 1 --2- 172.21.2.102:0/61036071 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8ae40acf70 0x7f8ae4091cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:13.661 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:13.658 7f8ae8c2e700 1 -- 172.21.2.102:0/61036071 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8ae40922e0 msgr2=0x7f8ae40926e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:13.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:13.658 7f8ae8c2e700 1 --2- 172.21.2.102:0/61036071 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8ae40922e0 0x7f8ae40926e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:13.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:13.658 7f8ae8c2e700 1 -- 172.21.2.102:0/61036071 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ae4092f20 con 0x7f8ae41a0160 2021-02-10T11:26:13.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:13.658 7f8ae8c2e700 1 --2- 172.21.2.102:0/61036071 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8ae41a0160 0x7f8ae40ac950 secure :-1 s=READY pgs=1586 cs=0 l=1 rev1=1 rx=0x7f8acc006f20 tx=0x7f8acc00b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:13.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:13.658 7f8ae17fa700 1 -- 172.21.2.102:0/61036071 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8acc00bd80 con 0x7f8ae41a0160 2021-02-10T11:26:13.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:13.659 7f8ae17fa700 1 -- 172.21.2.102:0/61036071 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8acc000c90 con 0x7f8ae41a0160 2021-02-10T11:26:13.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:13.659 7f8ae17fa700 1 -- 172.21.2.102:0/61036071 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8acc0022b0 con 0x7f8ae41a0160 2021-02-10T11:26:13.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:13.659 7f8aeae12700 1 -- 172.21.2.102:0/61036071 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ae4092f20 con 0x7f8ae41a0160 2021-02-10T11:26:13.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:13.659 7f8aeae12700 1 -- 172.21.2.102:0/61036071 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ae4090450 con 0x7f8ae41a0160 2021-02-10T11:26:13.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:13.659 7f8ae17fa700 1 -- 172.21.2.102:0/61036071 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f8acc00a000 con 0x7f8ae41a0160 2021-02-10T11:26:13.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:13.659 7f8ae17fa700 1 --2- 172.21.2.102:0/61036071 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8ac80186c0 0x7f8ac801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:13.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:13.659 7f8ae17fa700 1 -- 172.21.2.102:0/61036071 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f8acc032540 con 0x7f8ae41a0160 2021-02-10T11:26:13.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:13.659 7f8ae3fff700 1 --2- 172.21.2.102:0/61036071 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8ac80186c0 0x7f8ac801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:13.663 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:13.660 7f8ae3fff700 1 --2- 172.21.2.102:0/61036071 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8ac80186c0 0x7f8ac801abc0 crc :-1 s=READY pgs=720 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:13.690 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:13.690 7f8aeae12700 1 -- 172.21.2.102:0/61036071 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ae410ca30 con 0x7f8ae41a0160 2021-02-10T11:26:13.692 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:13.693 7f8ae17fa700 1 -- 172.21.2.102:0/61036071 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f8acc000940 con 0x7f8ae41a0160 2021-02-10T11:26:13.821 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:13.821 7f8aeae12700 1 -- 172.21.2.102:0/61036071 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7f8ae410ca30 con 0x7f8ae41a0160 2021-02-10T11:26:13.821 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:13.822 7f8ae17fa700 1 -- 172.21.2.102:0/61036071 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v66) v1 ==== 100+0+10 (secure 0 0 0) 0x7f8acc00bd80 con 0x7f8ae41a0160 2021-02-10T11:26:13.822 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:26:13.824 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:13.824 7f8aeae12700 1 -- 172.21.2.102:0/61036071 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8ac80186c0 msgr2=0x7f8ac801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:13.824 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:13.824 7f8aeae12700 1 --2- 172.21.2.102:0/61036071 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8ac80186c0 0x7f8ac801abc0 crc :-1 s=READY pgs=720 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:13.824 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:13.824 7f8aeae12700 1 -- 172.21.2.102:0/61036071 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8ae41a0160 msgr2=0x7f8ae40ac950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:13.824 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:13.824 7f8aeae12700 1 --2- 172.21.2.102:0/61036071 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8ae41a0160 0x7f8ae40ac950 secure :-1 s=READY pgs=1586 cs=0 l=1 rev1=1 rx=0x7f8acc006f20 tx=0x7f8acc00b990).stop 2021-02-10T11:26:13.825 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:13.824 7f8aeae12700 1 -- 172.21.2.102:0/61036071 shutdown_connections 2021-02-10T11:26:13.825 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:13.824 7f8aeae12700 1 --2- 172.21.2.102:0/61036071 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8ac80186c0 0x7f8ac801abc0 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:13.825 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:13.824 7f8aeae12700 1 --2- 172.21.2.102:0/61036071 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8ae40922e0 0x7f8ae40926e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:13.825 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:13.824 7f8aeae12700 1 --2- 172.21.2.102:0/61036071 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8ae40acf70 0x7f8ae4091cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:13.825 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:13.824 7f8aeae12700 1 --2- 172.21.2.102:0/61036071 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8ae41a0160 0x7f8ae40ac950 unknown :-1 s=CLOSED pgs=1586 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:13.825 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:13.824 7f8aeae12700 1 -- 172.21.2.102:0/61036071 >> 172.21.2.102:0/61036071 conn(0x7f8ae4188880 msgr2=0x7f8ae40a4380 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:26:13.825 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:13.824 7f8aeae12700 1 -- 172.21.2.102:0/61036071 shutdown_connections 2021-02-10T11:26:13.826 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:13.824 7f8aeae12700 1 -- 172.21.2.102:0/61036071 wait complete. 2021-02-10T11:26:13.832 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-02-10T11:26:13.832 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:26:14.262 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:14.244 7fad30b98700 1 Processor -- start 2021-02-10T11:26:14.262 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:14.244 7fad30b98700 1 -- start start 2021-02-10T11:26:14.262 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:14.244 7fad30b98700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fad2c0d18d0 0x7fad2c083ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:14.262 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:14.244 7fad30b98700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad2c082d50 con 0x7fad2c0e53e0 2021-02-10T11:26:14.262 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:14.244 7fad30b98700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad2c0e57e0 con 0x7fad2c082920 2021-02-10T11:26:14.262 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:14.244 7fad30b98700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad2c1b0d40 con 0x7fad2c0d18d0 2021-02-10T11:26:14.263 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:14.245 7fad2a85e700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fad2c0e53e0 0x7fad2c1b0550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:50804/0 (socket says 172.21.2.102:50804) 2021-02-10T11:26:14.263 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:14.245 7fad2a85e700 1 -- 172.21.2.102:0/1620328398 learned_addr learned my addr 172.21.2.102:0/1620328398 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:26:14.263 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:14.245 7fad2985c700 1 -- 172.21.2.102:0/1620328398 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 126181962 0 0) 0x7fad14001500 con 0x7fad2c0e53e0 2021-02-10T11:26:14.263 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:14.245 7fad2985c700 1 -- 172.21.2.102:0/1620328398 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad18002fe0 con 0x7fad2c0e53e0 2021-02-10T11:26:14.263 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:14.245 7fad2b05f700 1 --2- 172.21.2.102:0/1620328398 >> v2:172.21.2.113:3300/0 conn(0x7fad2c0d18d0 0x7fad2c083ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:14.263 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:14.245 7fad2985c700 1 -- 172.21.2.102:0/1620328398 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2562353600 0 0) 0x7fad14001500 con 0x7fad2c0e53e0 2021-02-10T11:26:14.263 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:14.245 7fad2985c700 1 -- 172.21.2.102:0/1620328398 >> v2:172.21.2.113:3300/0 conn(0x7fad2c0d18d0 msgr2=0x7fad2c083ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:26:14.263 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:14.245 7fad2985c700 1 --2- 172.21.2.102:0/1620328398 >> v2:172.21.2.113:3300/0 conn(0x7fad2c0d18d0 0x7fad2c083ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:14.263 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:14.245 7fad2985c700 1 -- 172.21.2.102:0/1620328398 >> v1:172.21.2.113:6789/0 conn(0x7fad2c082920 legacy=0x7fad2c0832d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:26:14.263 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:14.245 7fad2985c700 1 -- 172.21.2.102:0/1620328398 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad2c0d1cd0 con 0x7fad2c0e53e0 2021-02-10T11:26:14.263 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:14.245 7fad2985c700 1 -- 172.21.2.102:0/1620328398 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7fad14001a00 con 0x7fad2c0e53e0 2021-02-10T11:26:14.263 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:14.245 7fad2985c700 1 -- 172.21.2.102:0/1620328398 >> v1:172.21.2.102:6789/0 conn(0x7fad2c0e53e0 legacy=0x7fad2c1b0550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:14.264 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:14.245 7fad2985c700 1 --2- 172.21.2.102:0/1620328398 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fad18003b70 0x7fad18005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:14.264 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:14.245 7fad2985c700 1 --2- 172.21.2.102:0/1620328398 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fad18006840 0x7fad18008d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:14.264 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:14.245 7fad2985c700 1 --2- 172.21.2.102:0/1620328398 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fad18009620 0x7fad1800bb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:14.264 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:14.245 7fad2985c700 1 -- 172.21.2.102:0/1620328398 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad1800c390 con 0x7fad18009620 2021-02-10T11:26:14.264 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:14.245 7fad2985c700 1 -- 172.21.2.102:0/1620328398 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad1800c5b0 con 0x7fad18006840 2021-02-10T11:26:14.264 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:14.245 7fad2985c700 1 -- 172.21.2.102:0/1620328398 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad1800c7d0 con 0x7fad18003b70 2021-02-10T11:26:14.264 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:14.245 7fad2b05f700 1 --2- 172.21.2.102:0/1620328398 >> v2:172.21.2.113:3300/0 conn(0x7fad2c0d18d0 0x7fad2c083ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:26:14.264 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:14.246 7fad2b05f700 1 --2- 172.21.2.102:0/1620328398 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fad18009620 0x7fad1800bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:14.264 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:14.246 7fad2a85e700 1 --2- 172.21.2.102:0/1620328398 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fad18003b70 0x7fad18005fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:14.264 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:14.246 7fad2a05d700 1 --2- 172.21.2.102:0/1620328398 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fad18006840 0x7fad18008d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:14.264 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:14.246 7fad2b05f700 1 -- 172.21.2.102:0/1620328398 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fad18003b70 msgr2=0x7fad18005fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:26:14.264 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:14.246 7fad2b05f700 1 --2- 172.21.2.102:0/1620328398 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fad18003b70 0x7fad18005fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:14.265 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:14.246 7fad2b05f700 1 -- 172.21.2.102:0/1620328398 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fad18006840 msgr2=0x7fad18008d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:26:14.265 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:14.246 7fad2b05f700 1 --2- 172.21.2.102:0/1620328398 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fad18006840 0x7fad18008d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:14.265 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:14.246 7fad2b05f700 1 -- 172.21.2.102:0/1620328398 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad1800ca70 con 0x7fad18009620 2021-02-10T11:26:14.265 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:14.246 7fad2a05d700 1 --2- 172.21.2.102:0/1620328398 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fad18006840 0x7fad18008d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:26:14.265 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:14.246 7fad2b05f700 1 -- 172.21.2.102:0/1620328398 reap_dead start 2021-02-10T11:26:14.265 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:14.246 7fad2b05f700 1 --2- 172.21.2.102:0/1620328398 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fad18009620 0x7fad1800bb40 secure :-1 s=READY pgs=1588 cs=0 l=1 rev1=1 rx=0x7fad24000de0 tx=0x7fad2404dd30).ready entity=mon.0 client_cookie=bbe328940d7044b2 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:14.265 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:14.246 7fad2985c700 1 -- 172.21.2.102:0/1620328398 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fad240068b0 con 0x7fad18009620 2021-02-10T11:26:14.265 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:14.246 7fad2985c700 1 -- 172.21.2.102:0/1620328398 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad24000a70 con 0x7fad18009620 2021-02-10T11:26:14.265 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:14.246 7fad2985c700 1 -- 172.21.2.102:0/1620328398 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fad2404f580 con 0x7fad18009620 2021-02-10T11:26:14.265 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:14.246 7fad30b98700 1 -- 172.21.2.102:0/1620328398 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fad18009620 msgr2=0x7fad1800bb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:14.265 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:14.246 7fad30b98700 1 --2- 172.21.2.102:0/1620328398 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fad18009620 0x7fad1800bb40 secure :-1 s=READY pgs=1588 cs=0 l=1 rev1=1 rx=0x7fad24000de0 tx=0x7fad2404dd30).stop 2021-02-10T11:26:14.265 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:14.247 7fad30b98700 1 -- 172.21.2.102:0/1620328398 shutdown_connections 2021-02-10T11:26:14.265 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:14.247 7fad30b98700 1 --2- 172.21.2.102:0/1620328398 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fad18009620 0x7fad1800bb40 unknown :-1 s=CLOSED pgs=1588 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:14.266 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:14.247 7fad30b98700 1 -- 172.21.2.102:0/1620328398 >> 172.21.2.102:0/1620328398 conn(0x7fad2c0ce500 msgr2=0x7fad2c0e4610 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:26:14.266 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:14.247 7fad30b98700 1 -- 172.21.2.102:0/1620328398 shutdown_connections 2021-02-10T11:26:14.266 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:14.247 7fad30b98700 1 -- 172.21.2.102:0/1620328398 wait complete. 2021-02-10T11:26:14.266 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:14.262 7fad30b98700 1 Processor -- start 2021-02-10T11:26:14.266 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:14.262 7fad30b98700 1 -- start start 2021-02-10T11:26:14.266 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:14.262 7fad30b98700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fad2c191bd0 0x7fad2c1a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:14.266 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:14.262 7fad30b98700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fad2c1a3e10 0x7fad2c0b72d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:14.266 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:14.262 7fad30b98700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fad2c0b78f0 0x7fad2c0b7cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:14.266 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:14.262 7fad30b98700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad2c192170 con 0x7fad2c0b78f0 2021-02-10T11:26:14.266 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:14.262 7fad30b98700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad2c0b8310 con 0x7fad2c1a3e10 2021-02-10T11:26:14.266 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:14.262 7fad30b98700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad2c0b8530 con 0x7fad2c191bd0 2021-02-10T11:26:14.266 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:14.262 7fad2b05f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fad2c0b78f0 0x7fad2c0b7cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:14.267 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:14.262 7fad2b05f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fad2c0b78f0 0x7fad2c0b7cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:58590/0 (socket says 172.21.2.102:58590) 2021-02-10T11:26:14.267 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:14.262 7fad2b05f700 1 -- 172.21.2.102:0/3696552492 learned_addr learned my addr 172.21.2.102:0/3696552492 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:26:14.268 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:14.263 7fad2a85e700 1 --2- 172.21.2.102:0/3696552492 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fad2c191bd0 0x7fad2c1a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:14.268 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:14.263 7fad2a05d700 1 --2- 172.21.2.102:0/3696552492 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fad2c1a3e10 0x7fad2c0b72d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:14.268 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:14.263 7fad2b05f700 1 -- 172.21.2.102:0/3696552492 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fad2c191bd0 msgr2=0x7fad2c1a37f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:14.268 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:14.263 7fad2b05f700 1 --2- 172.21.2.102:0/3696552492 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fad2c191bd0 0x7fad2c1a37f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:14.268 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:14.263 7fad2b05f700 1 -- 172.21.2.102:0/3696552492 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fad2c1a3e10 msgr2=0x7fad2c0b72d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:14.268 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:14.263 7fad2b05f700 1 --2- 172.21.2.102:0/3696552492 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fad2c1a3e10 0x7fad2c0b72d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:14.268 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:14.263 7fad2b05f700 1 -- 172.21.2.102:0/3696552492 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad2c1944a0 con 0x7fad2c0b78f0 2021-02-10T11:26:14.268 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:14.263 7fad2a05d700 1 --2- 172.21.2.102:0/3696552492 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fad2c1a3e10 0x7fad2c0b72d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:26:14.268 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:14.263 7fad2b05f700 1 --2- 172.21.2.102:0/3696552492 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fad2c0b78f0 0x7fad2c0b7cf0 secure :-1 s=READY pgs=1589 cs=0 l=1 rev1=1 rx=0x7fad2404ef10 tx=0x7fad240027b0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:14.268 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:14.263 7fad237fe700 1 -- 172.21.2.102:0/3696552492 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fad24006020 con 0x7fad2c0b78f0 2021-02-10T11:26:14.269 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:14.263 7fad237fe700 1 -- 172.21.2.102:0/3696552492 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad240062e0 con 0x7fad2c0b78f0 2021-02-10T11:26:14.269 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:14.263 7fad30b98700 1 -- 172.21.2.102:0/3696552492 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad2c1944a0 con 0x7fad2c0b78f0 2021-02-10T11:26:14.269 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:14.263 7fad30b98700 1 -- 172.21.2.102:0/3696552492 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad2c194a30 con 0x7fad2c0b78f0 2021-02-10T11:26:14.269 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:14.263 7fad237fe700 1 -- 172.21.2.102:0/3696552492 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fad240055b0 con 0x7fad2c0b78f0 2021-02-10T11:26:14.269 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:14.264 7fad237fe700 1 -- 172.21.2.102:0/3696552492 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fad24002ba0 con 0x7fad2c0b78f0 2021-02-10T11:26:14.269 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:14.264 7fad237fe700 1 --2- 172.21.2.102:0/3696552492 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fad10018730 0x7fad1001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:14.269 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:14.264 7fad237fe700 1 -- 172.21.2.102:0/3696552492 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fad240768b0 con 0x7fad2c0b78f0 2021-02-10T11:26:14.269 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:14.264 7fad2a85e700 1 --2- 172.21.2.102:0/3696552492 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fad10018730 0x7fad1001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:14.269 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:14.264 7fad2a85e700 1 --2- 172.21.2.102:0/3696552492 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fad10018730 0x7fad1001ac10 crc :-1 s=READY pgs=721 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:14.294 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:14.294 7fad30b98700 1 -- 172.21.2.102:0/3696552492 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad2c0ed5d0 con 0x7fad2c0b78f0 2021-02-10T11:26:14.296 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:14.297 7fad237fe700 1 -- 172.21.2.102:0/3696552492 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fad24005870 con 0x7fad2c0b78f0 2021-02-10T11:26:14.427 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:14.427 7fad30b98700 1 -- 172.21.2.102:0/3696552492 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fad2c082d50 con 0x7fad2c0b78f0 2021-02-10T11:26:14.428 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:14.428 7fad237fe700 1 -- 172.21.2.102:0/3696552492 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) v1 ==== 94+0+2101 (secure 0 0 0) 0x7fad24002ba0 con 0x7fad2c0b78f0 2021-02-10T11:26:14.430 INFO:teuthology.orchestra.run.gibba002.stderr:dumped fsmap epoch 49 2021-02-10T11:26:14.430 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:14.429 7fad30b98700 1 -- 172.21.2.102:0/3696552492 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fad10018730 msgr2=0x7fad1001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:14.430 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:14.429 7fad30b98700 1 --2- 172.21.2.102:0/3696552492 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fad10018730 0x7fad1001ac10 crc :-1 s=READY pgs=721 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:14.430 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:14.429 7fad30b98700 1 -- 172.21.2.102:0/3696552492 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fad2c0b78f0 msgr2=0x7fad2c0b7cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:14.430 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:14.429 7fad30b98700 1 --2- 172.21.2.102:0/3696552492 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fad2c0b78f0 0x7fad2c0b7cf0 secure :-1 s=READY pgs=1589 cs=0 l=1 rev1=1 rx=0x7fad2404ef10 tx=0x7fad240027b0).stop 2021-02-10T11:26:14.431 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:14.430 7fad30b98700 1 -- 172.21.2.102:0/3696552492 shutdown_connections 2021-02-10T11:26:14.431 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:14.430 7fad30b98700 1 --2- 172.21.2.102:0/3696552492 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fad2c0b78f0 0x7fad2c0b7cf0 unknown :-1 s=CLOSED pgs=1589 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:14.431 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:14.430 7fad30b98700 1 --2- 172.21.2.102:0/3696552492 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fad10018730 0x7fad1001ac10 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:14.431 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:14.430 7fad30b98700 1 --2- 172.21.2.102:0/3696552492 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fad2c1a3e10 0x7fad2c0b72d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:14.431 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:14.430 7fad30b98700 1 --2- 172.21.2.102:0/3696552492 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fad2c191bd0 0x7fad2c1a37f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:14.431 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:14.430 7fad30b98700 1 -- 172.21.2.102:0/3696552492 >> 172.21.2.102:0/3696552492 conn(0x7fad2c0ad180 msgr2=0x7fad2c0b5070 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:26:14.431 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:14.430 7fad30b98700 1 -- 172.21.2.102:0/3696552492 shutdown_connections 2021-02-10T11:26:14.431 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:14.430 7fad30b98700 1 -- 172.21.2.102:0/3696552492 wait complete. 2021-02-10T11:26:14.431 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:26:14.432 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":49,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8412,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.102:6835/3414701760","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6834","nonce":3414701760},{"type":"v1","addr":"172.21.2.102:6835","nonce":3414701760}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":44}],"filesystems":[{"mdsmap":{"epoch":48,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:25:31.513947","modified":"2021-02-10 11:25:33.524416","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5813},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5813":{"gid":5813,"name":"b","rank":0,"incarnation":45,"state":"up:active","state_seq":3,"addr":"172.21.2.113:6833/2951398038","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6832","nonce":2951398038},{"type":"v1","addr":"172.21.2.113:6833","nonce":2951398038}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":6}]} 2021-02-10T11:26:14.639 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 48, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:25:31.513947', 'modified': '2021-02-10 11:25:33.524416', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5813}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5813': {'gid': 5813, 'name': 'b', 'rank': 0, 'incarnation': 45, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.113:6833/2951398038', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.113:6832', 'nonce': 2951398038}, {'type': 'v1', 'addr': '172.21.2.113:6833', 'nonce': 2951398038}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [13], 'metadata_pool': 12, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:26:14.640 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:26:14.640 DEBUG:teuthology.orchestra.run.gibba013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:26:15.038 INFO:teuthology.orchestra.run.gibba013.stdout:{ 2021-02-10T11:26:15.038 INFO:teuthology.orchestra.run.gibba013.stdout: "cluster_fsid": "68db30ea-dbce-4aef-a6df-f01c6fea0513", 2021-02-10T11:26:15.038 INFO:teuthology.orchestra.run.gibba013.stdout: "whoami": 0, 2021-02-10T11:26:15.038 INFO:teuthology.orchestra.run.gibba013.stdout: "id": 5813, 2021-02-10T11:26:15.038 INFO:teuthology.orchestra.run.gibba013.stdout: "want_state": "up:active", 2021-02-10T11:26:15.038 INFO:teuthology.orchestra.run.gibba013.stdout: "state": "up:active", 2021-02-10T11:26:15.039 INFO:teuthology.orchestra.run.gibba013.stdout: "rank_uptime": 43.522845582999999, 2021-02-10T11:26:15.039 INFO:teuthology.orchestra.run.gibba013.stdout: "mdsmap_epoch": 48, 2021-02-10T11:26:15.039 INFO:teuthology.orchestra.run.gibba013.stdout: "osdmap_epoch": 66, 2021-02-10T11:26:15.039 INFO:teuthology.orchestra.run.gibba013.stdout: "osdmap_epoch_barrier": 66, 2021-02-10T11:26:15.039 INFO:teuthology.orchestra.run.gibba013.stdout: "uptime": 49.158902437000002 2021-02-10T11:26:15.040 INFO:teuthology.orchestra.run.gibba013.stdout:} 2021-02-10T11:26:15.042 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "68db30ea-dbce-4aef-a6df-f01c6fea0513", "whoami": 0, "id": 5813, "want_state": "up:active", "state": "up:active", "rank_uptime": 43.522845582999999, "mdsmap_epoch": 48, "osdmap_epoch": 66, "osdmap_epoch_barrier": 66, "uptime": 49.158902437000002 } 2021-02-10T11:26:15.042 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2021-02-10T11:26:15.042 INFO:tasks.cephfs.fuse_mount:Client client.alice config is {} 2021-02-10T11:26:15.042 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.alice at ubuntu@gibba002.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.1... 2021-02-10T11:26:15.042 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:15.042 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec mkdir -p /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:26:15.378 DEBUG:teuthology.orchestra.run.gibba002:> sudo modprobe fuse 2021-02-10T11:26:15.721 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:26:15.721 DEBUG:teuthology.orchestra.run.gibba002:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:26:16.064 INFO:teuthology.orchestra.run.gibba002.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:26:16.064 INFO:teuthology.orchestra.run.gibba002.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:26:16.065 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:26:16.065 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:16.065 DEBUG:teuthology.orchestra.run.gibba002:> ls /sys/fs/fuse/connections 2021-02-10T11:26:16.399 INFO:teuthology.orchestra.run.gibba002.stdout:43 2021-02-10T11:26:16.400 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [43] 2021-02-10T11:26:16.400 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --client_mountpoint=/volumes/group_0000000000664797/subvolume_0000000000152661/c517f318-1811-438b-85ff-c9e7d6aca87c --name client.alice /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:26:16.403 DEBUG:teuthology.orchestra.run.gibba002:> sudo modprobe fuse 2021-02-10T11:26:16.747 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:26:16.747 DEBUG:teuthology.orchestra.run.gibba002:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:26:16.813 INFO:tasks.cephfs.fuse_mount.ceph-fuse.alice.gibba002.stderr:ceph-fuse[162071]: starting ceph client 2021-02-10T11:26:16.813 INFO:tasks.cephfs.fuse_mount.ceph-fuse.alice.gibba002.stderr:2021-02-10 11:26:16.813 7f70f8dd9f80 -1 init, newargv = 0x55f7bdc68430 newargc=9 2021-02-10T11:26:16.820 INFO:tasks.cephfs.fuse_mount.ceph-fuse.alice.gibba002.stderr:ceph-fuse[162071]: starting fuse 2021-02-10T11:26:17.089 INFO:teuthology.orchestra.run.gibba002.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:26:17.089 INFO:teuthology.orchestra.run.gibba002.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:26:17.090 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:26:17.090 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:17.090 DEBUG:teuthology.orchestra.run.gibba002:> ls /sys/fs/fuse/connections 2021-02-10T11:26:17.425 INFO:teuthology.orchestra.run.gibba002.stdout:43 2021-02-10T11:26:17.426 INFO:teuthology.orchestra.run.gibba002.stdout:44 2021-02-10T11:26:17.426 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43, 44] 2021-02-10T11:26:17.426 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:17.426 DEBUG:teuthology.orchestra.run.gibba002:> sudo python3 -c ' 2021-02-10T11:26:17.426 DEBUG:teuthology.orchestra.run.gibba002:> import glob 2021-02-10T11:26:17.426 DEBUG:teuthology.orchestra.run.gibba002:> import re 2021-02-10T11:26:17.426 DEBUG:teuthology.orchestra.run.gibba002:> import os 2021-02-10T11:26:17.427 DEBUG:teuthology.orchestra.run.gibba002:> import subprocess 2021-02-10T11:26:17.427 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:26:17.427 DEBUG:teuthology.orchestra.run.gibba002:> def find_socket(client_name): 2021-02-10T11:26:17.427 DEBUG:teuthology.orchestra.run.gibba002:> asok_path = "/var/run/ceph/ceph-client.alice.*.asok" 2021-02-10T11:26:17.427 DEBUG:teuthology.orchestra.run.gibba002:> files = glob.glob(asok_path) 2021-02-10T11:26:17.427 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:26:17.427 DEBUG:teuthology.orchestra.run.gibba002:> # Given a non-glob path, it better be there 2021-02-10T11:26:17.427 DEBUG:teuthology.orchestra.run.gibba002:> if "*" not in asok_path: 2021-02-10T11:26:17.427 DEBUG:teuthology.orchestra.run.gibba002:> assert(len(files) == 1) 2021-02-10T11:26:17.427 DEBUG:teuthology.orchestra.run.gibba002:> return files[0] 2021-02-10T11:26:17.427 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:26:17.427 DEBUG:teuthology.orchestra.run.gibba002:> for f in files: 2021-02-10T11:26:17.427 DEBUG:teuthology.orchestra.run.gibba002:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2021-02-10T11:26:17.427 DEBUG:teuthology.orchestra.run.gibba002:> if os.path.exists("/proc/{0}".format(pid)): 2021-02-10T11:26:17.427 DEBUG:teuthology.orchestra.run.gibba002:> return f 2021-02-10T11:26:17.428 DEBUG:teuthology.orchestra.run.gibba002:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2021-02-10T11:26:17.428 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:26:17.428 DEBUG:teuthology.orchestra.run.gibba002:> print(find_socket("client.alice")) 2021-02-10T11:26:17.428 DEBUG:teuthology.orchestra.run.gibba002:> ' 2021-02-10T11:26:17.788 INFO:teuthology.orchestra.run.gibba002.stdout:/var/run/ceph/ceph-client.alice.162071.asok 2021-02-10T11:26:17.791 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.alice.162071.asok 2021-02-10T11:26:17.791 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:17.791 DEBUG:teuthology.orchestra.run.gibba002:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.alice.162071.asok status 2021-02-10T11:26:18.180 INFO:teuthology.orchestra.run.gibba002.stdout:{ 2021-02-10T11:26:18.181 INFO:teuthology.orchestra.run.gibba002.stdout: "metadata": { 2021-02-10T11:26:18.181 INFO:teuthology.orchestra.run.gibba002.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:26:18.181 INFO:teuthology.orchestra.run.gibba002.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:26:18.181 INFO:teuthology.orchestra.run.gibba002.stdout: "entity_id": "alice", 2021-02-10T11:26:18.181 INFO:teuthology.orchestra.run.gibba002.stdout: "hostname": "gibba002", 2021-02-10T11:26:18.181 INFO:teuthology.orchestra.run.gibba002.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2021-02-10T11:26:18.181 INFO:teuthology.orchestra.run.gibba002.stdout: "pid": "162071", 2021-02-10T11:26:18.181 INFO:teuthology.orchestra.run.gibba002.stdout: "root": "/volumes/group_0000000000664797/subvolume_0000000000152661/c517f318-1811-438b-85ff-c9e7d6aca87c" 2021-02-10T11:26:18.181 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:26:18.181 INFO:teuthology.orchestra.run.gibba002.stdout: "dentry_count": 0, 2021-02-10T11:26:18.181 INFO:teuthology.orchestra.run.gibba002.stdout: "dentry_pinned_count": 0, 2021-02-10T11:26:18.181 INFO:teuthology.orchestra.run.gibba002.stdout: "id": 8751, 2021-02-10T11:26:18.181 INFO:teuthology.orchestra.run.gibba002.stdout: "inst": { 2021-02-10T11:26:18.182 INFO:teuthology.orchestra.run.gibba002.stdout: "name": { 2021-02-10T11:26:18.182 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "client", 2021-02-10T11:26:18.182 INFO:teuthology.orchestra.run.gibba002.stdout: "num": 8751 2021-02-10T11:26:18.182 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:26:18.182 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": { 2021-02-10T11:26:18.182 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:26:18.182 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:0", 2021-02-10T11:26:18.182 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 538088805 2021-02-10T11:26:18.182 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:26:18.182 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:26:18.182 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": { 2021-02-10T11:26:18.182 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:26:18.182 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:0", 2021-02-10T11:26:18.183 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 538088805 2021-02-10T11:26:18.183 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:26:18.183 INFO:teuthology.orchestra.run.gibba002.stdout: "inst_str": "client.8751 172.21.2.102:0/538088805", 2021-02-10T11:26:18.183 INFO:teuthology.orchestra.run.gibba002.stdout: "addr_str": "172.21.2.102:0/538088805", 2021-02-10T11:26:18.183 INFO:teuthology.orchestra.run.gibba002.stdout: "inode_count": 1, 2021-02-10T11:26:18.183 INFO:teuthology.orchestra.run.gibba002.stdout: "mds_epoch": 48, 2021-02-10T11:26:18.183 INFO:teuthology.orchestra.run.gibba002.stdout: "osd_epoch": 66, 2021-02-10T11:26:18.183 INFO:teuthology.orchestra.run.gibba002.stdout: "osd_epoch_barrier": 0, 2021-02-10T11:26:18.183 INFO:teuthology.orchestra.run.gibba002.stdout: "blacklisted": false 2021-02-10T11:26:18.183 INFO:teuthology.orchestra.run.gibba002.stdout:} 2021-02-10T11:26:18.185 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.1 && sudo ls /home/ubuntu/cephtest/mnt.1 2021-02-10T11:26:18.530 INFO:teuthology.orchestra.run.gibba002.stdout:data.bin 2021-02-10T11:26:18.531 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:18.531 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:26:18.531 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:26:18.867 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:26:18.868 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2021-02-10T11:26:18.868 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.1 && sudo dd if=/dev/urandom of=rogue.bin bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:26:19.214 INFO:teuthology.orchestra.run.gibba002.stderr:dd: failed to open ‘rogue.bin’: Permission denied 2021-02-10T11:26:19.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:26:19.215 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:19.215 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:26:19.215 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:26:19.551 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:26:19.551 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2021-02-10T11:26:19.551 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@gibba002.front.sepia.ceph.com... 2021-02-10T11:26:19.552 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-02-10T11:26:19.552 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:26:19.903 INFO:tasks.cephfs.fuse_mount.ceph-fuse.alice.gibba002.stderr:ceph-fuse[162071]: fuse finished with error 0 and tester_r 0 2021-02-10T11:26:19.905 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:19.905 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:26:19.905 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:26:20.239 INFO:teuthology.orchestra.run.gibba002.stdout:ext2/ext3 2021-02-10T11:26:20.239 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2021-02-10T11:26:20.239 INFO:teuthology.orchestra.run:waiting for 900 2021-02-10T11:26:20.239 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:26:20.239 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:26:20.575 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume deauthorize cephfs subvolume_0000000000152661 alice --group_name group_0000000000664797 2021-02-10T11:26:21.006 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:20.989 7f4cf232f700 1 Processor -- start 2021-02-10T11:26:21.006 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:20.989 7f4cf232f700 1 -- start start 2021-02-10T11:26:21.006 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:20.990 7f4cf232f700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f4cec0e51a0 0x7f4cec1b3910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:21.007 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:20.990 7f4cf232f700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4cec0e55a0 0x7f4cec07a120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:21.007 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:20.990 7f4cf232f700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cec078b90 con 0x7f4cec1b3f30 2021-02-10T11:26:21.007 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:20.990 7f4cf232f700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4cec0d1930 con 0x7f4cec0e51a0 2021-02-10T11:26:21.007 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:20.990 7f4cf232f700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4cec0d1b50 con 0x7f4cec0e55a0 2021-02-10T11:26:21.007 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:20.990 7f4ceb7fe700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f4cec1b3f30 0x7f4cec077510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:50836/0 (socket says 172.21.2.102:50836) 2021-02-10T11:26:21.007 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:20.990 7f4ceb7fe700 1 -- 172.21.2.102:0/3852295039 learned_addr learned my addr 172.21.2.102:0/3852295039 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:26:21.007 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:20.990 7f4cf094c700 1 --2- 172.21.2.102:0/3852295039 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4cec0e55a0 0x7f4cec07a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:21.007 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:20.990 7f4cebfff700 1 --2- 172.21.2.102:0/3852295039 >> v2:172.21.2.113:3300/0 conn(0x7f4cec0e51a0 0x7f4cec1b3910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:21.007 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:20.990 7f4ceaffd700 1 -- 172.21.2.102:0/3852295039 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2685283329 0 0) 0x7f4ce00010b0 con 0x7f4cec1b3f30 2021-02-10T11:26:21.007 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:20.990 7f4ceaffd700 1 -- 172.21.2.102:0/3852295039 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4cd4002fe0 con 0x7f4cec1b3f30 2021-02-10T11:26:21.007 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:20.990 7f4ceaffd700 1 -- 172.21.2.102:0/3852295039 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2258811444 0 0) 0x7f4ce00010b0 con 0x7f4cec1b3f30 2021-02-10T11:26:21.007 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:20.990 7f4ceaffd700 1 -- 172.21.2.102:0/3852295039 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4cec0e55a0 msgr2=0x7f4cec07a120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:26:21.007 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:20.990 7f4ceaffd700 1 --2- 172.21.2.102:0/3852295039 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4cec0e55a0 0x7f4cec07a120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:21.008 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:20.990 7f4ceaffd700 1 -- 172.21.2.102:0/3852295039 >> v2:172.21.2.113:3300/0 conn(0x7f4cec0e51a0 msgr2=0x7f4cec1b3910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:26:21.008 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:20.990 7f4ceaffd700 1 --2- 172.21.2.102:0/3852295039 >> v2:172.21.2.113:3300/0 conn(0x7f4cec0e51a0 0x7f4cec1b3910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:21.008 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:20.990 7f4ceaffd700 1 -- 172.21.2.102:0/3852295039 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cec0d1d70 con 0x7f4cec1b3f30 2021-02-10T11:26:21.008 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:20.990 7f4ceaffd700 1 -- 172.21.2.102:0/3852295039 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7f4ce00015b0 con 0x7f4cec1b3f30 2021-02-10T11:26:21.008 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:20.991 7f4ceaffd700 1 -- 172.21.2.102:0/3852295039 >> v1:172.21.2.102:6789/0 conn(0x7f4cec1b3f30 legacy=0x7f4cec077510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:21.008 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:20.991 7f4ceaffd700 1 --2- 172.21.2.102:0/3852295039 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4cd4003ae0 0x7f4cd4005f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:21.008 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:20.991 7f4ceaffd700 1 --2- 172.21.2.102:0/3852295039 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4cd404a840 0x7f4cd404cd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:21.008 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:20.991 7f4ceaffd700 1 --2- 172.21.2.102:0/3852295039 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4cd404d5b0 0x7f4cd404fad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:21.008 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:20.991 7f4ceaffd700 1 -- 172.21.2.102:0/3852295039 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cd4050380 con 0x7f4cd4003ae0 2021-02-10T11:26:21.008 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:20.991 7f4ceaffd700 1 -- 172.21.2.102:0/3852295039 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cd40505a0 con 0x7f4cd404d5b0 2021-02-10T11:26:21.008 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:20.991 7f4ceaffd700 1 -- 172.21.2.102:0/3852295039 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4cd40507c0 con 0x7f4cd404a840 2021-02-10T11:26:21.008 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:20.991 7f4cf094c700 1 --2- 172.21.2.102:0/3852295039 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4cec0e55a0 0x7f4cec07a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:26:21.008 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:20.991 7f4cebfff700 1 --2- 172.21.2.102:0/3852295039 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4cd4003ae0 0x7f4cd4005f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:21.008 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:20.991 7f4ceb7fe700 1 --2- 172.21.2.102:0/3852295039 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4cd404a840 0x7f4cd404cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:21.009 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:20.991 7f4cf094c700 1 --2- 172.21.2.102:0/3852295039 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4cd404d5b0 0x7f4cd404fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:21.009 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:20.991 7f4cebfff700 1 -- 172.21.2.102:0/3852295039 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4cd404a840 msgr2=0x7f4cd404cd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:26:21.009 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:20.991 7f4cebfff700 1 --2- 172.21.2.102:0/3852295039 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4cd404a840 0x7f4cd404cd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:21.009 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:20.991 7f4cebfff700 1 -- 172.21.2.102:0/3852295039 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4cd404d5b0 msgr2=0x7f4cd404fad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:26:21.009 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:20.991 7f4cebfff700 1 --2- 172.21.2.102:0/3852295039 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4cd404d5b0 0x7f4cd404fad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:21.009 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:20.991 7f4cebfff700 1 -- 172.21.2.102:0/3852295039 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cd4050a60 con 0x7f4cd4003ae0 2021-02-10T11:26:21.009 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:20.991 7f4cebfff700 1 --2- 172.21.2.102:0/3852295039 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4cd4003ae0 0x7f4cd4005f90 secure :-1 s=READY pgs=1593 cs=0 l=1 rev1=1 rx=0x7f4cdc00c8f0 tx=0x7f4cdc00cc90).ready entity=mon.0 client_cookie=1a3010662a498da5 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:21.009 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:20.991 7f4ceaffd700 1 -- 172.21.2.102:0/3852295039 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4cdc00cdd0 con 0x7f4cd4003ae0 2021-02-10T11:26:21.009 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:20.991 7f4ceaffd700 1 -- 172.21.2.102:0/3852295039 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4cdc009310 con 0x7f4cd4003ae0 2021-02-10T11:26:21.009 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:20.991 7f4ceaffd700 1 -- 172.21.2.102:0/3852295039 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4cdc0097b0 con 0x7f4cd4003ae0 2021-02-10T11:26:21.009 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:20.991 7f4cf232f700 1 -- 172.21.2.102:0/3852295039 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4cd4003ae0 msgr2=0x7f4cd4005f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:21.009 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:20.991 7f4cf232f700 1 --2- 172.21.2.102:0/3852295039 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4cd4003ae0 0x7f4cd4005f90 secure :-1 s=READY pgs=1593 cs=0 l=1 rev1=1 rx=0x7f4cdc00c8f0 tx=0x7f4cdc00cc90).stop 2021-02-10T11:26:21.009 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:20.991 7f4cf094c700 1 -- 172.21.2.102:0/3852295039 reap_dead start 2021-02-10T11:26:21.010 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:20.992 7f4cf232f700 1 -- 172.21.2.102:0/3852295039 shutdown_connections 2021-02-10T11:26:21.010 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:20.992 7f4cf232f700 1 -- 172.21.2.102:0/3852295039 >> 172.21.2.102:0/3852295039 conn(0x7f4cec0ce850 msgr2=0x7f4cec0e4690 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:26:21.010 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:20.992 7f4cf232f700 1 -- 172.21.2.102:0/3852295039 shutdown_connections 2021-02-10T11:26:21.010 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:20.992 7f4cf232f700 1 -- 172.21.2.102:0/3852295039 wait complete. 2021-02-10T11:26:21.010 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:21.007 7f4cf232f700 1 Processor -- start 2021-02-10T11:26:21.010 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:21.007 7f4cf232f700 1 -- start start 2021-02-10T11:26:21.012 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:21.007 7f4cf232f700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4cec08db00 0x7f4cec18f290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:21.012 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:21.007 7f4cf232f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4cec09c830 0x7f4cec090a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:21.012 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:21.007 7f4cf232f700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4cec091910 0x7f4cec091d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:21.012 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:21.007 7f4cf232f700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cec091410 con 0x7f4cec09c830 2021-02-10T11:26:21.012 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:21.007 7f4cf232f700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cec08df40 con 0x7f4cec091910 2021-02-10T11:26:21.012 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:21.007 7f4cf232f700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4cec09cc30 con 0x7f4cec08db00 2021-02-10T11:26:21.012 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:21.007 7f4ceb7fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4cec09c830 0x7f4cec090a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:21.012 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:21.007 7f4ceb7fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4cec09c830 0x7f4cec090a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:58622/0 (socket says 172.21.2.102:58622) 2021-02-10T11:26:21.012 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:21.007 7f4ceb7fe700 1 -- 172.21.2.102:0/2644471159 learned_addr learned my addr 172.21.2.102:0/2644471159 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:26:21.012 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:21.008 7f4cf094c700 1 --2- 172.21.2.102:0/2644471159 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4cec091910 0x7f4cec091d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:21.012 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:21.008 7f4cebfff700 1 --2- 172.21.2.102:0/2644471159 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4cec08db00 0x7f4cec18f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:21.012 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:21.008 7f4ceb7fe700 1 -- 172.21.2.102:0/2644471159 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4cec08db00 msgr2=0x7f4cec18f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:21.013 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:21.008 7f4ceb7fe700 1 --2- 172.21.2.102:0/2644471159 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4cec08db00 0x7f4cec18f290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:21.013 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:21.008 7f4ceb7fe700 1 -- 172.21.2.102:0/2644471159 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4cec091910 msgr2=0x7f4cec091d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:21.013 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:21.008 7f4ceb7fe700 1 --2- 172.21.2.102:0/2644471159 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4cec091910 0x7f4cec091d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:21.013 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:21.008 7f4ceb7fe700 1 -- 172.21.2.102:0/2644471159 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cec091060 con 0x7f4cec09c830 2021-02-10T11:26:21.013 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:21.008 7f4cebfff700 1 --2- 172.21.2.102:0/2644471159 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4cec08db00 0x7f4cec18f290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:26:21.013 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:21.008 7f4ceb7fe700 1 --2- 172.21.2.102:0/2644471159 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4cec09c830 0x7f4cec090a40 secure :-1 s=READY pgs=1594 cs=0 l=1 rev1=1 rx=0x7f4ce0001c00 tx=0x7f4ce000a8e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:21.013 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:21.008 7f4ce8ff9700 1 -- 172.21.2.102:0/2644471159 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4ce00009b0 con 0x7f4cec09c830 2021-02-10T11:26:21.013 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:21.008 7f4ce8ff9700 1 -- 172.21.2.102:0/2644471159 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ce000acd0 con 0x7f4cec09c830 2021-02-10T11:26:21.013 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:21.008 7f4ce8ff9700 1 -- 172.21.2.102:0/2644471159 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4ce0004540 con 0x7f4cec09c830 2021-02-10T11:26:21.013 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:21.008 7f4cf232f700 1 -- 172.21.2.102:0/2644471159 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4cec091060 con 0x7f4cec09c830 2021-02-10T11:26:21.013 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:21.008 7f4cf232f700 1 -- 172.21.2.102:0/2644471159 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4cec091060 con 0x7f4cec09c830 2021-02-10T11:26:21.013 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:21.009 7f4ce8ff9700 1 -- 172.21.2.102:0/2644471159 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4ce0004350 con 0x7f4cec09c830 2021-02-10T11:26:21.013 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:21.009 7f4ce8ff9700 1 --2- 172.21.2.102:0/2644471159 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4ccc0186c0 0x7f4ccc01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:21.013 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:21.009 7f4ce8ff9700 1 -- 172.21.2.102:0/2644471159 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f4ce0032400 con 0x7f4cec09c830 2021-02-10T11:26:21.014 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:21.009 7f4cebfff700 1 --2- 172.21.2.102:0/2644471159 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4ccc0186c0 0x7f4ccc01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:21.014 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:21.009 7f4cebfff700 1 --2- 172.21.2.102:0/2644471159 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4ccc0186c0 0x7f4ccc01abc0 crc :-1 s=READY pgs=722 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:21.039 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:21.040 7f4cf232f700 1 -- 172.21.2.102:0/2644471159 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4cec10a8c0 con 0x7f4cec09c830 2021-02-10T11:26:21.042 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:21.042 7f4ce8ff9700 1 -- 172.21.2.102:0/2644471159 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f4ce0004140 con 0x7f4cec09c830 2021-02-10T11:26:21.170 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:21.170 7f4cf232f700 1 -- 172.21.2.102:0/2644471159 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume deauthorize", "vol_name": "cephfs", "sub_name": "subvolume_0000000000152661", "target": ["mgr", ""], "auth_id": "alice", "group_name": "group_0000000000664797"}) v1 -- 0x7f4cec0fe2c0 con 0x7f4ccc0186c0 2021-02-10T11:26:21.204 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:21.204 7f4ce8ff9700 1 -- 172.21.2.102:0/2644471159 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f4cdc0009b0 con 0x7f4ccc0186c0 2021-02-10T11:26:21.236 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:21.236 7f4cf232f700 1 -- 172.21.2.102:0/2644471159 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4ccc0186c0 msgr2=0x7f4ccc01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:21.236 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:21.236 7f4cf232f700 1 --2- 172.21.2.102:0/2644471159 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4ccc0186c0 0x7f4ccc01abc0 crc :-1 s=READY pgs=722 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:21.236 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:21.236 7f4cf232f700 1 -- 172.21.2.102:0/2644471159 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4cec09c830 msgr2=0x7f4cec090a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:21.236 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:21.236 7f4cf232f700 1 --2- 172.21.2.102:0/2644471159 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4cec09c830 0x7f4cec090a40 secure :-1 s=READY pgs=1594 cs=0 l=1 rev1=1 rx=0x7f4ce0001c00 tx=0x7f4ce000a8e0).stop 2021-02-10T11:26:21.236 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:21.236 7f4cf232f700 1 -- 172.21.2.102:0/2644471159 shutdown_connections 2021-02-10T11:26:21.236 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:21.236 7f4cf232f700 1 --2- 172.21.2.102:0/2644471159 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4ccc0186c0 0x7f4ccc01abc0 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:21.236 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:21.236 7f4cf232f700 1 --2- 172.21.2.102:0/2644471159 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4cec091910 0x7f4cec091d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:21.236 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:21.236 7f4cf232f700 1 --2- 172.21.2.102:0/2644471159 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4cec09c830 0x7f4cec090a40 unknown :-1 s=CLOSED pgs=1594 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:21.236 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:21.236 7f4cf232f700 1 --2- 172.21.2.102:0/2644471159 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4cec08db00 0x7f4cec18f290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:21.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:21.236 7f4cf232f700 1 -- 172.21.2.102:0/2644471159 >> 172.21.2.102:0/2644471159 conn(0x7f4cec1a1310 msgr2=0x7f4cec1b2690 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:26:21.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:21.236 7f4cf232f700 1 -- 172.21.2.102:0/2644471159 shutdown_connections 2021-02-10T11:26:21.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:21.236 7f4cf232f700 1 -- 172.21.2.102:0/2644471159 wait complete. 2021-02-10T11:26:21.244 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth ls --format=json-pretty 2021-02-10T11:26:21.672 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:21.655 7ffaeed75700 1 Processor -- start 2021-02-10T11:26:21.672 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:21.655 7ffaeed75700 1 -- start start 2021-02-10T11:26:21.672 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:21.656 7ffaeed75700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ffae80e53e0 0x7ffae81b0550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:21.672 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:21.656 7ffaeed75700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7ffae81b0b70 0x7ffae8081910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:21.672 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:21.656 7ffaeed75700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7ffae8081f30 0x7ffae8082330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:21.672 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:21.656 7ffaeed75700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffae8084850 con 0x7ffae8081f30 2021-02-10T11:26:21.672 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:21.656 7ffaeed75700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffae80e5fe0 con 0x7ffae81b0b70 2021-02-10T11:26:21.672 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:21.656 7ffaeed75700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffae8082d50 con 0x7ffae80e53e0 2021-02-10T11:26:21.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:21.656 7ffaed392700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7ffae8081f30 0x7ffae8082330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:21.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:21.656 7ffaed392700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7ffae8081f30 0x7ffae8082330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:58630/0 (socket says 172.21.2.102:58630) 2021-02-10T11:26:21.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:21.656 7ffaed392700 1 -- 172.21.2.102:0/3167562188 learned_addr learned my addr 172.21.2.102:0/3167562188 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:26:21.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:21.656 7ffae7fff700 1 --2- 172.21.2.102:0/3167562188 >> v2:172.21.2.113:3300/0 conn(0x7ffae81b0b70 0x7ffae8081910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:21.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:21.656 7ffaecb91700 1 --2- 172.21.2.102:0/3167562188 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ffae80e53e0 0x7ffae81b0550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:21.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:21.656 7ffaed392700 1 -- 172.21.2.102:0/3167562188 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ffae80e53e0 msgr2=0x7ffae81b0550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:26:21.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:21.657 7ffaed392700 1 --2- 172.21.2.102:0/3167562188 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ffae80e53e0 0x7ffae81b0550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:21.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:21.657 7ffaed392700 1 -- 172.21.2.102:0/3167562188 >> v2:172.21.2.113:3300/0 conn(0x7ffae81b0b70 msgr2=0x7ffae8081910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:26:21.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:21.657 7ffaed392700 1 --2- 172.21.2.102:0/3167562188 >> v2:172.21.2.113:3300/0 conn(0x7ffae81b0b70 0x7ffae8081910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:21.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:21.657 7ffaed392700 1 -- 172.21.2.102:0/3167562188 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffae80e57e0 con 0x7ffae8081f30 2021-02-10T11:26:21.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:21.657 7ffaed392700 1 --2- 172.21.2.102:0/3167562188 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ffae8081f30 0x7ffae8082330 secure :-1 s=READY pgs=1595 cs=0 l=1 rev1=1 rx=0x7ffae0006b80 tx=0x7ffae0009b90).ready entity=mon.0 client_cookie=1c3e46bb8832c352 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:21.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:21.657 7ffae77fe700 1 -- 172.21.2.102:0/3167562188 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ffae00022e0 con 0x7ffae8081f30 2021-02-10T11:26:21.674 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:21.657 7ffae77fe700 1 -- 172.21.2.102:0/3167562188 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffae0002560 con 0x7ffae8081f30 2021-02-10T11:26:21.674 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:21.657 7ffae77fe700 1 -- 172.21.2.102:0/3167562188 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ffae0002a80 con 0x7ffae8081f30 2021-02-10T11:26:21.674 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:21.657 7ffaeed75700 1 -- 172.21.2.102:0/3167562188 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ffae8081f30 msgr2=0x7ffae8082330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:21.674 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:21.657 7ffaeed75700 1 --2- 172.21.2.102:0/3167562188 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ffae8081f30 0x7ffae8082330 secure :-1 s=READY pgs=1595 cs=0 l=1 rev1=1 rx=0x7ffae0006b80 tx=0x7ffae0009b90).stop 2021-02-10T11:26:21.674 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:21.658 7ffaeed75700 1 -- 172.21.2.102:0/3167562188 shutdown_connections 2021-02-10T11:26:21.674 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:21.658 7ffaeed75700 1 --2- 172.21.2.102:0/3167562188 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ffae8081f30 0x7ffae8082330 unknown :-1 s=CLOSED pgs=1595 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:21.674 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:21.658 7ffaeed75700 1 --2- 172.21.2.102:0/3167562188 >> v2:172.21.2.113:3300/0 conn(0x7ffae81b0b70 0x7ffae8081910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:21.674 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:21.658 7ffaeed75700 1 --2- 172.21.2.102:0/3167562188 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ffae80e53e0 0x7ffae81b0550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:21.674 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:21.658 7ffaeed75700 1 -- 172.21.2.102:0/3167562188 >> 172.21.2.102:0/3167562188 conn(0x7ffae80ce500 msgr2=0x7ffae80e4610 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:26:21.674 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:21.658 7ffaeed75700 1 -- 172.21.2.102:0/3167562188 shutdown_connections 2021-02-10T11:26:21.674 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:21.658 7ffaeed75700 1 -- 172.21.2.102:0/3167562188 wait complete. 2021-02-10T11:26:21.676 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:21.673 7ffaeed75700 1 Processor -- start 2021-02-10T11:26:21.676 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:21.673 7ffaeed75700 1 -- start start 2021-02-10T11:26:21.676 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:21.673 7ffaeed75700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ffae80abd50 0x7ffae81a75c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:21.676 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:21.673 7ffaeed75700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ffae80b2630 0x7ffae81a7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:21.676 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:21.673 7ffaeed75700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ffae80bb770 0x7ffae80bc290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:21.676 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:21.673 7ffaeed75700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffae80bbd90 con 0x7ffae80abd50 2021-02-10T11:26:21.676 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:21.673 7ffaeed75700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffae80ac1a0 con 0x7ffae80bb770 2021-02-10T11:26:21.676 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:21.673 7ffaeed75700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffae81a8200 con 0x7ffae80b2630 2021-02-10T11:26:21.676 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:21.673 7ffaecb91700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ffae80abd50 0x7ffae81a75c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:21.677 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:21.674 7ffaecb91700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ffae80abd50 0x7ffae81a75c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:58636/0 (socket says 172.21.2.102:58636) 2021-02-10T11:26:21.677 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:21.674 7ffaecb91700 1 -- 172.21.2.102:0/3418153667 learned_addr learned my addr 172.21.2.102:0/3418153667 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:26:21.677 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:21.674 7ffaed392700 1 --2- 172.21.2.102:0/3418153667 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ffae80bb770 0x7ffae80bc290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:21.677 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:21.674 7ffae7fff700 1 --2- 172.21.2.102:0/3418153667 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ffae80b2630 0x7ffae81a7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:21.677 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:21.674 7ffaecb91700 1 -- 172.21.2.102:0/3418153667 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ffae80b2630 msgr2=0x7ffae81a7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:21.677 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:21.674 7ffaecb91700 1 --2- 172.21.2.102:0/3418153667 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ffae80b2630 0x7ffae81a7be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:21.677 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:21.674 7ffaecb91700 1 -- 172.21.2.102:0/3418153667 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ffae80bb770 msgr2=0x7ffae80bc290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:21.677 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:21.674 7ffaecb91700 1 --2- 172.21.2.102:0/3418153667 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ffae80bb770 0x7ffae80bc290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:21.677 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:21.674 7ffaecb91700 1 -- 172.21.2.102:0/3418153667 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffae80b9100 con 0x7ffae80abd50 2021-02-10T11:26:21.677 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:21.674 7ffaecb91700 1 --2- 172.21.2.102:0/3418153667 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ffae80abd50 0x7ffae81a75c0 secure :-1 s=READY pgs=1596 cs=0 l=1 rev1=1 rx=0x7ffad0006f60 tx=0x7ffad000b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:21.677 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:21.674 7ffae57fa700 1 -- 172.21.2.102:0/3418153667 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ffad000bd60 con 0x7ffae80abd50 2021-02-10T11:26:21.677 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:21.674 7ffae57fa700 1 -- 172.21.2.102:0/3418153667 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffad0000c90 con 0x7ffae80abd50 2021-02-10T11:26:21.677 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:21.674 7ffaeed75700 1 -- 172.21.2.102:0/3418153667 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffae80b9100 con 0x7ffae80abd50 2021-02-10T11:26:21.678 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:21.674 7ffae57fa700 1 -- 172.21.2.102:0/3418153667 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ffad0002220 con 0x7ffae80abd50 2021-02-10T11:26:21.678 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:21.674 7ffaeed75700 1 -- 172.21.2.102:0/3418153667 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffae80bc8b0 con 0x7ffae80abd50 2021-02-10T11:26:21.678 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:21.675 7ffae57fa700 1 -- 172.21.2.102:0/3418153667 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ffad0006b00 con 0x7ffae80abd50 2021-02-10T11:26:21.678 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:21.675 7ffae57fa700 1 --2- 172.21.2.102:0/3418153667 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ffacc0186c0 0x7ffacc01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:21.678 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:21.675 7ffae57fa700 1 -- 172.21.2.102:0/3418153667 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 5573+0+0 (secure 0 0 0) 0x7ffad0032450 con 0x7ffae80abd50 2021-02-10T11:26:21.678 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:21.675 7ffae7fff700 1 --2- 172.21.2.102:0/3418153667 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ffacc0186c0 0x7ffacc01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:21.678 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:21.675 7ffae7fff700 1 --2- 172.21.2.102:0/3418153667 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ffacc0186c0 0x7ffacc01abc0 crc :-1 s=READY pgs=723 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:21.705 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:21.706 7ffaeed75700 1 -- 172.21.2.102:0/3418153667 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffae80ac180 con 0x7ffae80abd50 2021-02-10T11:26:21.708 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:21.708 7ffae57fa700 1 -- 172.21.2.102:0/3418153667 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ffad002df30 con 0x7ffae80abd50 2021-02-10T11:26:21.839 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:21.840 7ffaeed75700 1 -- 172.21.2.102:0/3418153667 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7ffae80ac180 con 0x7ffae80abd50 2021-02-10T11:26:21.840 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:21.840 7ffae57fa700 1 -- 172.21.2.102:0/3418153667 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v42) v1 ==== 80+0+4513 (secure 0 0 0) 0x7ffad002df30 con 0x7ffae80abd50 2021-02-10T11:26:21.840 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:26:21.840 INFO:teuthology.orchestra.run.gibba002.stdout:{ 2021-02-10T11:26:21.840 INFO:teuthology.orchestra.run.gibba002.stdout: "auth_dump": [ 2021-02-10T11:26:21.841 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:26:21.841 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "mds.a", 2021-02-10T11:26:21.841 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAevSNgsgyEARAAvr0cUjjCoEHLxaZSZ2HtVA==", 2021-02-10T11:26:21.841 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:26:21.841 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow", 2021-02-10T11:26:21.841 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:26:21.841 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:26:21.841 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:26:21.841 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:26:21.841 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:26:21.841 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:26:21.841 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "mds.b", 2021-02-10T11:26:21.841 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAevSNgwOFMMRAALfb1fbX56ao1TU8ClnTW1A==", 2021-02-10T11:26:21.842 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:26:21.842 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow", 2021-02-10T11:26:21.842 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:26:21.842 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:26:21.842 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:26:21.842 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:26:21.842 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:26:21.842 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:26:21.842 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.0", 2021-02-10T11:26:21.842 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAmvSNgj+bMMBAAD3HUD6fLiMAg+++iNzLJKQ==", 2021-02-10T11:26:21.842 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:26:21.842 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:26:21.843 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:26:21.843 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:26:21.843 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:26:21.843 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:26:21.843 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:26:21.843 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.1", 2021-02-10T11:26:21.843 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQApvSNgnvc+MRAAAOxqlNwiyLHqNITOrYeFKQ==", 2021-02-10T11:26:21.843 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:26:21.843 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:26:21.843 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:26:21.843 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:26:21.843 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:26:21.843 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:26:21.843 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:26:21.844 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.2", 2021-02-10T11:26:21.844 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAsvSNgZ4SbMRAA76IOYWfbiAuBoM/4k81GxA==", 2021-02-10T11:26:21.844 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:26:21.844 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:26:21.844 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:26:21.844 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:26:21.844 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:26:21.844 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:26:21.844 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:26:21.844 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.3", 2021-02-10T11:26:21.844 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAvvSNgFk3rMRAAWCmKKyw0LDQ+UlAHZAITOw==", 2021-02-10T11:26:21.844 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:26:21.844 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:26:21.844 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:26:21.845 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:26:21.845 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:26:21.845 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:26:21.845 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:26:21.845 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.4", 2021-02-10T11:26:21.845 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQA4vSNgpKcNJBAApJMmvj5YxVHPDOmvmQONOg==", 2021-02-10T11:26:21.845 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:26:21.845 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:26:21.845 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:26:21.845 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:26:21.845 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:26:21.845 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:26:21.845 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:26:21.846 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.5", 2021-02-10T11:26:21.846 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQA7vSNgf2ujJBAAfkuYpyb3WTzc8V5kw+Wktg==", 2021-02-10T11:26:21.846 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:26:21.846 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:26:21.846 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:26:21.846 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:26:21.846 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:26:21.846 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:26:21.847 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:26:21.847 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.6", 2021-02-10T11:26:21.847 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQA+vSNgqw9uJRAAFMU0R+UxU9C3o/9byb3e0g==", 2021-02-10T11:26:21.847 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:26:21.847 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:26:21.847 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:26:21.847 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:26:21.847 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:26:21.847 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:26:21.847 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:26:21.847 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.7", 2021-02-10T11:26:21.848 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQBBvSNgvWSrJRAA2KaCHmLpzovRgYzLvyofTQ==", 2021-02-10T11:26:21.848 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:26:21.848 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:26:21.848 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:26:21.848 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:26:21.848 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:26:21.848 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:26:21.848 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:26:21.848 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "client.0", 2021-02-10T11:26:21.848 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAfvSNgZd9XIBAAvyrvl8fdKuHnJZfczfRoFw==", 2021-02-10T11:26:21.848 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:26:21.848 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow", 2021-02-10T11:26:21.848 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow r", 2021-02-10T11:26:21.849 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:26:21.849 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:26:21.849 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:26:21.849 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:26:21.849 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "client.1", 2021-02-10T11:26:21.849 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAfvSNgSyfXNhAAIduPnF7Vd/6HSkJ0MlcMSA==", 2021-02-10T11:26:21.849 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:26:21.849 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow", 2021-02-10T11:26:21.849 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow r", 2021-02-10T11:26:21.849 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:26:21.849 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:26:21.849 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:26:21.849 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:26:21.849 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "client.2", 2021-02-10T11:26:21.850 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAgvSNgiIG/ERAA9ZTamaTI7ZsPSnxBqjplCg==", 2021-02-10T11:26:21.850 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:26:21.850 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow", 2021-02-10T11:26:21.850 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow r", 2021-02-10T11:26:21.850 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:26:21.850 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:26:21.850 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:26:21.850 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:26:21.850 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "client.3", 2021-02-10T11:26:21.850 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAgvSNgOWpOKBAAlXsEB04Q/GvtL06PqRMLeA==", 2021-02-10T11:26:21.850 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:26:21.850 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow", 2021-02-10T11:26:21.850 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow r", 2021-02-10T11:26:21.851 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:26:21.851 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:26:21.851 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:26:21.851 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:26:21.851 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "client.admin", 2021-02-10T11:26:21.851 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAavSNgO3GQBhAAL+NVnYskQRKVWLgyDPY/xw==", 2021-02-10T11:26:21.851 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:26:21.851 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow *", 2021-02-10T11:26:21.851 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:26:21.851 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:26:21.851 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:26:21.851 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:26:21.851 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:26:21.852 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:26:21.852 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "mgr.x", 2021-02-10T11:26:21.852 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAdvSNgXZZgJhAA7dtBSGzH+DsBwnddZpOBJA==", 2021-02-10T11:26:21.852 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:26:21.852 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow *", 2021-02-10T11:26:21.852 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow profile mgr", 2021-02-10T11:26:21.852 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:26:21.852 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:26:21.852 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:26:21.852 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:26:21.852 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "mgr.y", 2021-02-10T11:26:21.852 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAdvSNgVqQFDRAAwo6YEdWZCAr5GdOWVL/Qvg==", 2021-02-10T11:26:21.852 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:26:21.852 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow *", 2021-02-10T11:26:21.853 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow profile mgr", 2021-02-10T11:26:21.853 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:26:21.853 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:26:21.853 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:26:21.853 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:26:21.853 INFO:teuthology.orchestra.run.gibba002.stdout:} 2021-02-10T11:26:21.854 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:21.842 7ffaeed75700 1 -- 172.21.2.102:0/3418153667 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ffacc0186c0 msgr2=0x7ffacc01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:21.854 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:21.842 7ffaeed75700 1 --2- 172.21.2.102:0/3418153667 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ffacc0186c0 0x7ffacc01abc0 crc :-1 s=READY pgs=723 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:21.854 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:21.842 7ffaeed75700 1 -- 172.21.2.102:0/3418153667 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ffae80abd50 msgr2=0x7ffae81a75c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:21.854 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:21.842 7ffaeed75700 1 --2- 172.21.2.102:0/3418153667 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ffae80abd50 0x7ffae81a75c0 secure :-1 s=READY pgs=1596 cs=0 l=1 rev1=1 rx=0x7ffad0006f60 tx=0x7ffad000b970).stop 2021-02-10T11:26:21.854 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:21.842 7ffaeed75700 1 -- 172.21.2.102:0/3418153667 shutdown_connections 2021-02-10T11:26:21.855 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:21.842 7ffaeed75700 1 --2- 172.21.2.102:0/3418153667 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ffacc0186c0 0x7ffacc01abc0 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:21.855 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:21.842 7ffaeed75700 1 --2- 172.21.2.102:0/3418153667 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ffae80bb770 0x7ffae80bc290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:21.855 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:21.842 7ffaeed75700 1 --2- 172.21.2.102:0/3418153667 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ffae80b2630 0x7ffae81a7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:21.855 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:21.842 7ffaeed75700 1 --2- 172.21.2.102:0/3418153667 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ffae80abd50 0x7ffae81a75c0 unknown :-1 s=CLOSED pgs=1596 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:21.855 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:21.842 7ffaeed75700 1 -- 172.21.2.102:0/3418153667 >> 172.21.2.102:0/3418153667 conn(0x7ffae80b5fd0 msgr2=0x7ffae80c3cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:26:21.855 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:21.842 7ffaeed75700 1 -- 172.21.2.102:0/3418153667 shutdown_connections 2021-02-10T11:26:21.855 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:21.842 7ffaeed75700 1 -- 172.21.2.102:0/3418153667 wait complete. 2021-02-10T11:26:21.855 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000152661 --group_name group_0000000000664797 2021-02-10T11:26:22.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:22.271 7f8cdcf78700 1 Processor -- start 2021-02-10T11:26:22.289 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:22.271 7f8cdcf78700 1 -- start start 2021-02-10T11:26:22.289 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:22.272 7f8cdcf78700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8cd80e5580 0x7f8cd8078520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:22.289 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:22.272 7f8cdcf78700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cd8078b90 con 0x7f8cd8079110 2021-02-10T11:26:22.289 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:22.272 7f8cdcf78700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cd81b40e0 con 0x7f8cd80e5180 2021-02-10T11:26:22.289 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:22.272 7f8cdcf78700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cd80d1910 con 0x7f8cd80e5580 2021-02-10T11:26:22.289 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:22.272 7f8cd7314700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f8cd8079110 0x7f8cd8079510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:50866/0 (socket says 172.21.2.102:50866) 2021-02-10T11:26:22.289 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:22.272 7f8cd7314700 1 -- 172.21.2.102:0/1863292999 learned_addr learned my addr 172.21.2.102:0/1863292999 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:26:22.289 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:22.272 7f8cd5b11700 1 -- 172.21.2.102:0/1863292999 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3995977026 0 0) 0x7f8cd0001500 con 0x7f8cd8079110 2021-02-10T11:26:22.289 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:22.272 7f8cd5b11700 1 -- 172.21.2.102:0/1863292999 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8cc0002fe0 con 0x7f8cd8079110 2021-02-10T11:26:22.289 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:22.272 7f8cd5b11700 1 -- 172.21.2.102:0/1863292999 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 474512900 0 0) 0x7f8cd0001460 con 0x7f8cd8079110 2021-02-10T11:26:22.289 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:22.272 7f8cd5b11700 1 -- 172.21.2.102:0/1863292999 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8cd80e5580 msgr2=0x7f8cd8078520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:26:22.289 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:22.272 7f8cd5b11700 1 --2- 172.21.2.102:0/1863292999 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8cd80e5580 0x7f8cd8078520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:22.290 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:22.272 7f8cd5b11700 1 -- 172.21.2.102:0/1863292999 >> v1:172.21.2.113:6789/0 conn(0x7f8cd80e5180 legacy=0x7f8cd81b38f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:26:22.290 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:22.272 7f8cd5b11700 1 -- 172.21.2.102:0/1863292999 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cd80d1af0 con 0x7f8cd8079110 2021-02-10T11:26:22.290 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:22.272 7f8cd5b11700 1 -- 172.21.2.102:0/1863292999 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7f8cd0001980 con 0x7f8cd8079110 2021-02-10T11:26:22.290 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:22.272 7f8cd5b11700 1 -- 172.21.2.102:0/1863292999 >> v1:172.21.2.102:6789/0 conn(0x7f8cd8079110 legacy=0x7f8cd8079510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:22.290 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:22.272 7f8cd5b11700 1 --2- 172.21.2.102:0/1863292999 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8cc0003a60 0x7f8cc0005ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:22.290 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:22.272 7f8cd5b11700 1 --2- 172.21.2.102:0/1863292999 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8cc0006790 0x7f8cc0008c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:22.290 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:22.272 7f8cd5b11700 1 --2- 172.21.2.102:0/1863292999 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8cc0009500 0x7f8cc000ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:22.290 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:22.272 7f8cd5b11700 1 -- 172.21.2.102:0/1863292999 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cc000c2d0 con 0x7f8cc0003a60 2021-02-10T11:26:22.290 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:22.272 7f8cd5b11700 1 -- 172.21.2.102:0/1863292999 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cc000c4f0 con 0x7f8cc0009500 2021-02-10T11:26:22.290 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:22.273 7f8cd5b11700 1 -- 172.21.2.102:0/1863292999 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cc000c710 con 0x7f8cc0006790 2021-02-10T11:26:22.290 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:22.273 7f8cd6b13700 1 --2- 172.21.2.102:0/1863292999 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8cc0003a60 0x7f8cc0005ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:22.290 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:22.273 7f8cd6312700 1 --2- 172.21.2.102:0/1863292999 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8cc0006790 0x7f8cc0008c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:22.291 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:22.273 7f8cd7314700 1 --2- 172.21.2.102:0/1863292999 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8cc0009500 0x7f8cc000ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:22.291 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:22.273 7f8cd6b13700 1 -- 172.21.2.102:0/1863292999 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8cc0006790 msgr2=0x7f8cc0008c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:26:22.291 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:22.273 7f8cd6b13700 1 --2- 172.21.2.102:0/1863292999 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8cc0006790 0x7f8cc0008c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:22.291 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:22.273 7f8cd6b13700 1 -- 172.21.2.102:0/1863292999 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8cc0009500 msgr2=0x7f8cc000ba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:26:22.291 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:22.273 7f8cd6b13700 1 --2- 172.21.2.102:0/1863292999 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8cc0009500 0x7f8cc000ba20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:22.291 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:22.273 7f8cd6b13700 1 -- 172.21.2.102:0/1863292999 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cc000c9b0 con 0x7f8cc0003a60 2021-02-10T11:26:22.291 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:22.273 7f8cd6312700 1 --2- 172.21.2.102:0/1863292999 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8cc0006790 0x7f8cc0008c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:26:22.291 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:22.273 7f8cd7314700 1 --2- 172.21.2.102:0/1863292999 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8cc0009500 0x7f8cc000ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:26:22.291 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:22.273 7f8cd6b13700 1 --2- 172.21.2.102:0/1863292999 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8cc0003a60 0x7f8cc0005ef0 secure :-1 s=READY pgs=1598 cs=0 l=1 rev1=1 rx=0x7f8cc8006ab0 tx=0x7f8cc804fb10).ready entity=mon.0 client_cookie=fa392a48952d2542 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:22.291 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:22.273 7f8cd5b11700 1 -- 172.21.2.102:0/1863292999 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8cc8001290 con 0x7f8cc0003a60 2021-02-10T11:26:22.291 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:22.273 7f8cd5b11700 1 -- 172.21.2.102:0/1863292999 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cc80066f0 con 0x7f8cc0003a60 2021-02-10T11:26:22.291 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:22.273 7f8cd5b11700 1 -- 172.21.2.102:0/1863292999 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8cc8001850 con 0x7f8cc0003a60 2021-02-10T11:26:22.291 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:22.273 7f8cdcf78700 1 -- 172.21.2.102:0/1863292999 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8cc0003a60 msgr2=0x7f8cc0005ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:22.292 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:22.273 7f8cdcf78700 1 --2- 172.21.2.102:0/1863292999 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8cc0003a60 0x7f8cc0005ef0 secure :-1 s=READY pgs=1598 cs=0 l=1 rev1=1 rx=0x7f8cc8006ab0 tx=0x7f8cc804fb10).stop 2021-02-10T11:26:22.292 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:22.273 7f8cd7314700 1 -- 172.21.2.102:0/1863292999 reap_dead start 2021-02-10T11:26:22.292 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:22.274 7f8cdcf78700 1 -- 172.21.2.102:0/1863292999 shutdown_connections 2021-02-10T11:26:22.292 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:22.274 7f8cdcf78700 1 -- 172.21.2.102:0/1863292999 >> 172.21.2.102:0/1863292999 conn(0x7f8cd80ce830 msgr2=0x7f8cd80e4670 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:26:22.292 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:22.274 7f8cdcf78700 1 -- 172.21.2.102:0/1863292999 shutdown_connections 2021-02-10T11:26:22.292 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:22.274 7f8cdcf78700 1 -- 172.21.2.102:0/1863292999 wait complete. 2021-02-10T11:26:22.292 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:22.289 7f8cdcf78700 1 Processor -- start 2021-02-10T11:26:22.292 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:22.289 7f8cdcf78700 1 -- start start 2021-02-10T11:26:22.294 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:22.289 7f8cdcf78700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8cd81a3100 0x7f8cd81a3500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:22.294 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:22.289 7f8cdcf78700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8cd810cc70 0x7f8cd810d070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:22.294 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:22.289 7f8cdcf78700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8cd80b6010 0x7f8cd80b6410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:22.294 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:22.289 7f8cdcf78700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cd810d9e0 con 0x7f8cd80b6010 2021-02-10T11:26:22.294 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:22.289 7f8cdcf78700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cd810d690 con 0x7f8cd810cc70 2021-02-10T11:26:22.294 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:22.289 7f8cdcf78700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cd80b6a30 con 0x7f8cd81a3100 2021-02-10T11:26:22.294 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:22.289 7f8cd7314700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8cd80b6010 0x7f8cd80b6410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:22.294 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:22.289 7f8cd7314700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8cd80b6010 0x7f8cd80b6410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:58654/0 (socket says 172.21.2.102:58654) 2021-02-10T11:26:22.294 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:22.289 7f8cd7314700 1 -- 172.21.2.102:0/2833632157 learned_addr learned my addr 172.21.2.102:0/2833632157 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:26:22.294 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:22.290 7f8cd6b13700 1 --2- 172.21.2.102:0/2833632157 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8cd81a3100 0x7f8cd81a3500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:22.295 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:22.290 7f8cd7314700 1 -- 172.21.2.102:0/2833632157 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8cd81a3100 msgr2=0x7f8cd81a3500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:22.295 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:22.290 7f8cd7314700 1 --2- 172.21.2.102:0/2833632157 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8cd81a3100 0x7f8cd81a3500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:22.295 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:22.290 7f8cd7314700 1 -- 172.21.2.102:0/2833632157 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8cd810cc70 msgr2=0x7f8cd810d070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:22.295 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:22.290 7f8cd7314700 1 --2- 172.21.2.102:0/2833632157 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8cd810cc70 0x7f8cd810d070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:22.295 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:22.290 7f8cd7314700 1 -- 172.21.2.102:0/2833632157 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cd80b6c50 con 0x7f8cd80b6010 2021-02-10T11:26:22.295 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:22.290 7f8cd7314700 1 --2- 172.21.2.102:0/2833632157 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8cd80b6010 0x7f8cd80b6410 secure :-1 s=READY pgs=1599 cs=0 l=1 rev1=1 rx=0x7f8cd000a8f0 tx=0x7f8cd000ac90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:22.295 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:22.290 7f8cc77fe700 1 -- 172.21.2.102:0/2833632157 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8cd0007020 con 0x7f8cd80b6010 2021-02-10T11:26:22.295 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:22.290 7f8cc77fe700 1 -- 172.21.2.102:0/2833632157 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cd00072a0 con 0x7f8cd80b6010 2021-02-10T11:26:22.295 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:22.290 7f8cdcf78700 1 -- 172.21.2.102:0/2833632157 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cd80b6c50 con 0x7f8cd80b6010 2021-02-10T11:26:22.295 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:22.290 7f8cdcf78700 1 -- 172.21.2.102:0/2833632157 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cd810e250 con 0x7f8cd80b6010 2021-02-10T11:26:22.295 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:22.290 7f8cc77fe700 1 -- 172.21.2.102:0/2833632157 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8cd0004720 con 0x7f8cd80b6010 2021-02-10T11:26:22.295 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:22.291 7f8cc77fe700 1 -- 172.21.2.102:0/2833632157 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f8cd0007480 con 0x7f8cd80b6010 2021-02-10T11:26:22.295 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:22.291 7f8cc77fe700 1 --2- 172.21.2.102:0/2833632157 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8cbc018730 0x7f8cbc01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:22.296 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:22.291 7f8cc77fe700 1 -- 172.21.2.102:0/2833632157 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f8cd0032b70 con 0x7f8cd80b6010 2021-02-10T11:26:22.296 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:22.291 7f8cd6b13700 1 --2- 172.21.2.102:0/2833632157 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8cbc018730 0x7f8cbc01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:22.296 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:22.291 7f8cd6b13700 1 --2- 172.21.2.102:0/2833632157 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8cbc018730 0x7f8cbc01ac10 crc :-1 s=READY pgs=724 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:22.321 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:22.321 7f8cdcf78700 1 -- 172.21.2.102:0/2833632157 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8cd810ca40 con 0x7f8cd80b6010 2021-02-10T11:26:22.323 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:22.324 7f8cc77fe700 1 -- 172.21.2.102:0/2833632157 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f8cd00019d0 con 0x7f8cd80b6010 2021-02-10T11:26:22.450 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:22.450 7f8cdcf78700 1 -- 172.21.2.102:0/2833632157 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000152661", "target": ["mgr", ""], "group_name": "group_0000000000664797"}) v1 -- 0x7f8cd810e250 con 0x7f8cbc018730 2021-02-10T11:26:22.459 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:22.459 7f8cc77fe700 1 -- 172.21.2.102:0/2833632157 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f8cc8000c60 con 0x7f8cbc018730 2021-02-10T11:26:22.468 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:22.468 7f8cdcf78700 1 -- 172.21.2.102:0/2833632157 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8cbc018730 msgr2=0x7f8cbc01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:22.468 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:22.468 7f8cdcf78700 1 --2- 172.21.2.102:0/2833632157 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8cbc018730 0x7f8cbc01ac10 crc :-1 s=READY pgs=724 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:22.468 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:22.468 7f8cdcf78700 1 -- 172.21.2.102:0/2833632157 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8cd80b6010 msgr2=0x7f8cd80b6410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:22.468 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:22.468 7f8cdcf78700 1 --2- 172.21.2.102:0/2833632157 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8cd80b6010 0x7f8cd80b6410 secure :-1 s=READY pgs=1599 cs=0 l=1 rev1=1 rx=0x7f8cd000a8f0 tx=0x7f8cd000ac90).stop 2021-02-10T11:26:22.468 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:22.468 7f8cdcf78700 1 -- 172.21.2.102:0/2833632157 shutdown_connections 2021-02-10T11:26:22.469 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:22.468 7f8cdcf78700 1 --2- 172.21.2.102:0/2833632157 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8cd80b6010 0x7f8cd80b6410 unknown :-1 s=CLOSED pgs=1599 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:22.469 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:22.468 7f8cdcf78700 1 --2- 172.21.2.102:0/2833632157 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8cbc018730 0x7f8cbc01ac10 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:22.469 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:22.468 7f8cdcf78700 1 --2- 172.21.2.102:0/2833632157 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8cd810cc70 0x7f8cd810d070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:22.469 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:22.468 7f8cdcf78700 1 --2- 172.21.2.102:0/2833632157 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8cd81a3100 0x7f8cd81a3500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:22.469 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:22.468 7f8cdcf78700 1 -- 172.21.2.102:0/2833632157 >> 172.21.2.102:0/2833632157 conn(0x7f8cd80a3880 msgr2=0x7f8cd80ad320 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:26:22.469 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:22.468 7f8cdcf78700 1 -- 172.21.2.102:0/2833632157 shutdown_connections 2021-02-10T11:26:22.469 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:22.468 7f8cdcf78700 1 -- 172.21.2.102:0/2833632157 wait complete. 2021-02-10T11:26:22.476 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolumegroup rm cephfs group_0000000000664797 2021-02-10T11:26:22.906 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:22.889 7fb042464700 1 Processor -- start 2021-02-10T11:26:22.906 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:22.889 7fb042464700 1 -- start start 2021-02-10T11:26:22.906 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:22.889 7fb042464700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb03c0e5170 0x7fb03c1b38e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:22.907 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:22.890 7fb042464700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fb03c0e5570 0x7fb03c07a120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:22.907 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:22.890 7fb042464700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb03c078b90 con 0x7fb03c1b3f00 2021-02-10T11:26:22.907 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:22.890 7fb042464700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb03c0d1900 con 0x7fb03c0e5570 2021-02-10T11:26:22.907 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:22.890 7fb042464700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb03c0d1af0 con 0x7fb03c0e5170 2021-02-10T11:26:22.907 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:22.891 7fb03b7fe700 1 --1- >> v1:172.21.2.113:6789/0 conn(0x7fb03c1b3f00 0x7fb03c077510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.113:6789/0 says I am v1:172.21.2.102:58208/0 (socket says 172.21.2.102:58208) 2021-02-10T11:26:22.907 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:22.891 7fb040a81700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fb03c0e5570 0x7fb03c07a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:22.907 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:22.891 7fb03b7fe700 1 -- 172.21.2.102:0/1268710590 learned_addr learned my addr 172.21.2.102:0/1268710590 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:26:22.907 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:22.891 7fb03bfff700 1 --2- 172.21.2.102:0/1268710590 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb03c0e5170 0x7fb03c1b38e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:22.907 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:22.891 7fb040a81700 1 -- 172.21.2.102:0/1268710590 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb03c0e5170 msgr2=0x7fb03c1b38e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:26:22.907 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:22.891 7fb040a81700 1 --2- 172.21.2.102:0/1268710590 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb03c0e5170 0x7fb03c1b38e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:22.907 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:22.891 7fb040a81700 1 -- 172.21.2.102:0/1268710590 >> v1:172.21.2.113:6789/0 conn(0x7fb03c1b3f00 legacy=0x7fb03c077510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:22.908 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:22.891 7fb03affd700 1 -- 172.21.2.102:0/1268710590 <== mon.1 v1:172.21.2.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 233667592 0 0) 0x7fb0300010b0 con 0x7fb03c1b3f00 2021-02-10T11:26:22.908 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:22.891 7fb040a81700 1 -- 172.21.2.102:0/1268710590 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb03c0d1ce0 con 0x7fb03c0e5570 2021-02-10T11:26:22.908 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:22.891 7fb03bfff700 1 --2- 172.21.2.102:0/1268710590 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb03c0e5170 0x7fb03c1b38e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:26:22.908 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:22.891 7fb040a81700 1 --2- 172.21.2.102:0/1268710590 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb03c0e5570 0x7fb03c07a120 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 rx=0x7fb034006e50 tx=0x7fb034009e70).ready entity=mon.1 client_cookie=80383111d15dfc9 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:22.908 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:22.892 7fb03affd700 1 -- 172.21.2.102:0/1268710590 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb034006aa0 con 0x7fb03c0e5570 2021-02-10T11:26:22.908 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:22.892 7fb03affd700 1 -- 172.21.2.102:0/1268710590 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb034006790 con 0x7fb03c0e5570 2021-02-10T11:26:22.908 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:22.892 7fb03affd700 1 -- 172.21.2.102:0/1268710590 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb03400b700 con 0x7fb03c0e5570 2021-02-10T11:26:22.908 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:22.892 7fb042464700 1 -- 172.21.2.102:0/1268710590 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb03c0e5570 msgr2=0x7fb03c07a120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:22.908 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:22.892 7fb042464700 1 --2- 172.21.2.102:0/1268710590 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb03c0e5570 0x7fb03c07a120 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 rx=0x7fb034006e50 tx=0x7fb034009e70).stop 2021-02-10T11:26:22.908 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:22.892 7fb042464700 1 -- 172.21.2.102:0/1268710590 shutdown_connections 2021-02-10T11:26:22.908 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:22.892 7fb042464700 1 --2- 172.21.2.102:0/1268710590 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb03c0e5570 0x7fb03c07a120 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:22.908 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:22.892 7fb042464700 1 --2- 172.21.2.102:0/1268710590 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb03c0e5170 0x7fb03c1b38e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:22.909 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:22.892 7fb042464700 1 -- 172.21.2.102:0/1268710590 >> 172.21.2.102:0/1268710590 conn(0x7fb03c0ce820 msgr2=0x7fb03c0e4660 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:26:22.909 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:22.892 7fb042464700 1 -- 172.21.2.102:0/1268710590 shutdown_connections 2021-02-10T11:26:22.909 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:22.892 7fb042464700 1 -- 172.21.2.102:0/1268710590 wait complete. 2021-02-10T11:26:22.910 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:22.907 7fb042464700 1 Processor -- start 2021-02-10T11:26:22.910 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:22.907 7fb042464700 1 -- start start 2021-02-10T11:26:22.910 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:22.908 7fb042464700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb03c0a0b80 0x7fb03c1a6d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:22.910 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:22.908 7fb042464700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb03c0bd490 0x7fb03c1a7350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:22.910 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:22.908 7fb042464700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb03c1a7970 0x7fb03c0bbfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:22.911 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:22.908 7fb042464700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb03c10e4d0 con 0x7fb03c1a7970 2021-02-10T11:26:22.911 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:22.908 7fb042464700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb03c0bc600 con 0x7fb03c0a0b80 2021-02-10T11:26:22.911 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:22.908 7fb042464700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb03c0bc820 con 0x7fb03c0bd490 2021-02-10T11:26:22.911 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:22.908 7fb040a81700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb03c1a7970 0x7fb03c0bbfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:22.911 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:22.908 7fb040a81700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb03c1a7970 0x7fb03c0bbfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:58666/0 (socket says 172.21.2.102:58666) 2021-02-10T11:26:22.911 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:22.908 7fb040a81700 1 -- 172.21.2.102:0/1803807774 learned_addr learned my addr 172.21.2.102:0/1803807774 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:26:22.911 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:22.908 7fb040a81700 1 -- 172.21.2.102:0/1803807774 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb03c0bd490 msgr2=0x7fb03c1a7350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:22.911 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:22.908 7fb040a81700 1 --2- 172.21.2.102:0/1803807774 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb03c0bd490 0x7fb03c1a7350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:22.911 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:22.908 7fb040a81700 1 -- 172.21.2.102:0/1803807774 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb03c0a0b80 msgr2=0x7fb03c1a6d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:22.911 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:22.908 7fb040a81700 1 --2- 172.21.2.102:0/1803807774 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb03c0a0b80 0x7fb03c1a6d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:22.911 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:22.908 7fb040a81700 1 -- 172.21.2.102:0/1803807774 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb03c0bca40 con 0x7fb03c1a7970 2021-02-10T11:26:22.911 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:22.908 7fb040a81700 1 --2- 172.21.2.102:0/1803807774 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb03c1a7970 0x7fb03c0bbfe0 secure :-1 s=READY pgs=1600 cs=0 l=1 rev1=1 rx=0x7fb034006e60 tx=0x7fb03400bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:22.912 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:22.909 7fb038ff9700 1 -- 172.21.2.102:0/1803807774 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb034009130 con 0x7fb03c1a7970 2021-02-10T11:26:22.912 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:22.909 7fb042464700 1 -- 172.21.2.102:0/1803807774 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb03c0bca40 con 0x7fb03c1a7970 2021-02-10T11:26:22.912 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:22.909 7fb042464700 1 -- 172.21.2.102:0/1803807774 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb03c0bcfd0 con 0x7fb03c1a7970 2021-02-10T11:26:22.912 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:22.909 7fb038ff9700 1 -- 172.21.2.102:0/1803807774 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb034000df0 con 0x7fb03c1a7970 2021-02-10T11:26:22.912 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:22.909 7fb038ff9700 1 -- 172.21.2.102:0/1803807774 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb034009130 con 0x7fb03c1a7970 2021-02-10T11:26:22.912 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:22.909 7fb038ff9700 1 -- 172.21.2.102:0/1803807774 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fb034009130 con 0x7fb03c1a7970 2021-02-10T11:26:22.912 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:22.909 7fb038ff9700 1 --2- 172.21.2.102:0/1803807774 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb01c0186c0 0x7fb01c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:22.912 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:22.909 7fb038ff9700 1 -- 172.21.2.102:0/1803807774 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fb0340328b0 con 0x7fb03c1a7970 2021-02-10T11:26:22.912 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:22.909 7fb03bfff700 1 --2- 172.21.2.102:0/1803807774 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb01c0186c0 0x7fb01c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:22.912 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:22.910 7fb03bfff700 1 --2- 172.21.2.102:0/1803807774 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb01c0186c0 0x7fb01c01abc0 crc :-1 s=READY pgs=725 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:22.939 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:22.939 7fb042464700 1 -- 172.21.2.102:0/1803807774 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb03c0fe290 con 0x7fb03c1a7970 2021-02-10T11:26:22.941 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:22.942 7fb038ff9700 1 -- 172.21.2.102:0/1803807774 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fb0340011a0 con 0x7fb03c1a7970 2021-02-10T11:26:23.069 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:23.070 7fb042464700 1 -- 172.21.2.102:0/1803807774 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolumegroup rm", "vol_name": "cephfs", "target": ["mgr", ""], "group_name": "group_0000000000664797"}) v1 -- 0x7fb03c0bcfd0 con 0x7fb01c0186c0 2021-02-10T11:26:23.073 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:23.074 7fb038ff9700 1 -- 172.21.2.102:0/1803807774 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fb02c0009b0 con 0x7fb01c0186c0 2021-02-10T11:26:23.078 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:23.078 7fb042464700 1 -- 172.21.2.102:0/1803807774 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb01c0186c0 msgr2=0x7fb01c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:23.078 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:23.078 7fb042464700 1 --2- 172.21.2.102:0/1803807774 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb01c0186c0 0x7fb01c01abc0 crc :-1 s=READY pgs=725 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:23.078 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:23.078 7fb042464700 1 -- 172.21.2.102:0/1803807774 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb03c1a7970 msgr2=0x7fb03c0bbfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:23.078 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:23.078 7fb042464700 1 --2- 172.21.2.102:0/1803807774 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb03c1a7970 0x7fb03c0bbfe0 secure :-1 s=READY pgs=1600 cs=0 l=1 rev1=1 rx=0x7fb034006e60 tx=0x7fb03400bcf0).stop 2021-02-10T11:26:23.078 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:23.079 7fb042464700 1 -- 172.21.2.102:0/1803807774 shutdown_connections 2021-02-10T11:26:23.079 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:23.079 7fb042464700 1 --2- 172.21.2.102:0/1803807774 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb03c1a7970 0x7fb03c0bbfe0 unknown :-1 s=CLOSED pgs=1600 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:23.079 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:23.079 7fb042464700 1 --2- 172.21.2.102:0/1803807774 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb03c0bd490 0x7fb03c1a7350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:23.079 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:23.079 7fb042464700 1 --2- 172.21.2.102:0/1803807774 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb01c0186c0 0x7fb01c01abc0 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:23.079 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:23.079 7fb042464700 1 --2- 172.21.2.102:0/1803807774 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb03c0a0b80 0x7fb03c1a6d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:23.079 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:23.079 7fb042464700 1 -- 172.21.2.102:0/1803807774 >> 172.21.2.102:0/1803807774 conn(0x7fb03c0a1790 msgr2=0x7fb03c187fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:26:23.079 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:23.079 7fb042464700 1 -- 172.21.2.102:0/1803807774 shutdown_connections 2021-02-10T11:26:23.079 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:23.079 7fb042464700 1 -- 172.21.2.102:0/1803807774 wait complete. 2021-02-10T11:26:23.087 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2021-02-10T11:26:23.087 DEBUG:teuthology.orchestra.run.gibba002:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:26:23.433 DEBUG:teuthology.orchestra.run.gibba013:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:26:23.777 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2021-02-10T11:26:23.778 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:23.778 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:26:23.778 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:26:24.114 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:26:24.114 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:26:24.114 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@gibba002.front.sepia.ceph.com... 2021-02-10T11:26:24.114 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-02-10T11:26:24.115 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:26:24.467 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba002.stderr:ceph-fuse[159204]: fuse finished with error 0 and tester_r 0 2021-02-10T11:26:24.468 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:24.468 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:26:24.468 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:26:24.804 INFO:teuthology.orchestra.run.gibba002.stdout:ext2/ext3 2021-02-10T11:26:24.804 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2021-02-10T11:26:24.805 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:26:24.805 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:26:25.142 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:25.142 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:26:25.142 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:26:25.477 INFO:teuthology.orchestra.run.gibba002.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.1’: No such file or directory 2021-02-10T11:26:25.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:26:25.477 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2021-02-10T11:26:25.477 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:26:25.478 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:26:25.812 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:25.812 DEBUG:teuthology.orchestra.run.gibba013:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:26:25.812 DEBUG:teuthology.orchestra.run.gibba013:> ' -- /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:26:26.145 INFO:teuthology.orchestra.run.gibba013.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.2’: No such file or directory 2021-02-10T11:26:26.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:26:26.145 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2021-02-10T11:26:26.145 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:26:26.146 DEBUG:teuthology.orchestra.run.gibba013:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:26:26.479 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:26.480 DEBUG:teuthology.orchestra.run.gibba013:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:26:26.480 DEBUG:teuthology.orchestra.run.gibba013:> ' -- /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:26:26.813 INFO:teuthology.orchestra.run.gibba013.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.3’: No such file or directory 2021-02-10T11:26:26.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:26:26.814 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.3 2021-02-10T11:26:26.814 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:26:26.814 DEBUG:teuthology.orchestra.run.gibba013:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:26:27.149 DEBUG:teuthology.orchestra.run.gibba002:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2021-02-10T11:26:27.151 DEBUG:teuthology.orchestra.run.gibba013:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2021-02-10T11:26:27.509 INFO:teuthology.orchestra.run.gibba002.stdout:[global] 2021-02-10T11:26:27.509 INFO:teuthology.orchestra.run.gibba002.stdout: chdir = "" 2021-02-10T11:26:27.509 INFO:teuthology.orchestra.run.gibba002.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2021-02-10T11:26:27.509 INFO:teuthology.orchestra.run.gibba002.stdout: auth supported = cephx 2021-02-10T11:26:27.509 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:26:27.509 INFO:teuthology.orchestra.run.gibba002.stdout: filestore xattr use omap = true 2021-02-10T11:26:27.509 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:26:27.509 INFO:teuthology.orchestra.run.gibba002.stdout: mon clock drift allowed = 1.000 2021-02-10T11:26:27.509 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:26:27.509 INFO:teuthology.orchestra.run.gibba002.stdout: osd crush chooseleaf type = 0 2021-02-10T11:26:27.510 INFO:teuthology.orchestra.run.gibba002.stdout: auth debug = true 2021-02-10T11:26:27.510 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:26:27.510 INFO:teuthology.orchestra.run.gibba002.stdout: ms die on old message = true 2021-02-10T11:26:27.510 INFO:teuthology.orchestra.run.gibba002.stdout: ms die on bug = true 2021-02-10T11:26:27.510 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:26:27.510 INFO:teuthology.orchestra.run.gibba002.stdout: mon max pg per osd = 10000 # >= luminous 2021-02-10T11:26:27.510 INFO:teuthology.orchestra.run.gibba002.stdout: mon pg warn max object skew = 0 2021-02-10T11:26:27.510 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:26:27.510 INFO:teuthology.orchestra.run.gibba002.stdout: osd pool default size = 2 2021-02-10T11:26:27.510 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:26:27.510 INFO:teuthology.orchestra.run.gibba002.stdout: mon osd allow primary affinity = true 2021-02-10T11:26:27.510 INFO:teuthology.orchestra.run.gibba002.stdout: mon osd allow pg remap = true 2021-02-10T11:26:27.510 INFO:teuthology.orchestra.run.gibba002.stdout: mon warn on legacy crush tunables = false 2021-02-10T11:26:27.511 INFO:teuthology.orchestra.run.gibba002.stdout: mon warn on crush straw calc version zero = false 2021-02-10T11:26:27.511 INFO:teuthology.orchestra.run.gibba002.stdout: mon warn on no sortbitwise = false 2021-02-10T11:26:27.511 INFO:teuthology.orchestra.run.gibba002.stdout: mon warn on osd down out interval zero = false 2021-02-10T11:26:27.511 INFO:teuthology.orchestra.run.gibba002.stdout: mon warn on too few osds = false 2021-02-10T11:26:27.511 INFO:teuthology.orchestra.run.gibba002.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2021-02-10T11:26:27.511 INFO:teuthology.orchestra.run.gibba002.stdout: mon_warn_on_pool_no_redundancy = false 2021-02-10T11:26:27.511 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:26:27.511 INFO:teuthology.orchestra.run.gibba002.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2021-02-10T11:26:27.511 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:26:27.511 INFO:teuthology.orchestra.run.gibba002.stdout: osd default data pool replay window = 5 2021-02-10T11:26:27.511 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:26:27.511 INFO:teuthology.orchestra.run.gibba002.stdout: mon allow pool delete = true 2021-02-10T11:26:27.511 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:26:27.512 INFO:teuthology.orchestra.run.gibba002.stdout: mon cluster log file level = debug 2021-02-10T11:26:27.512 INFO:teuthology.orchestra.run.gibba002.stdout: debug asserts on shutdown = true 2021-02-10T11:26:27.512 INFO:teuthology.orchestra.run.gibba002.stdout: mon health detail to clog = false 2021-02-10T11:26:27.512 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:26:27.512 INFO:teuthology.orchestra.run.gibba002.stdout: # we see this fail in qa on *nautilus*; bump up retries 2021-02-10T11:26:27.512 INFO:teuthology.orchestra.run.gibba002.stdout: mon_client_directed_command_retry = 4 2021-02-10T11:26:27.512 INFO:teuthology.orchestra.run.gibba002.stdout: mon host = "172.21.2.102,172.21.2.113,[v2:172.21.2.113:3301,v1:172.21.2.113:6790]" 2021-02-10T11:26:27.512 INFO:teuthology.orchestra.run.gibba002.stdout: fsid = 68db30ea-dbce-4aef-a6df-f01c6fea0513 2021-02-10T11:26:27.512 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:26:27.512 INFO:teuthology.orchestra.run.gibba002.stdout:[osd] 2021-02-10T11:26:27.512 INFO:teuthology.orchestra.run.gibba002.stdout: osd journal size = 100 2021-02-10T11:26:27.512 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:26:27.512 INFO:teuthology.orchestra.run.gibba002.stdout: osd scrub load threshold = 5.0 2021-02-10T11:26:27.513 INFO:teuthology.orchestra.run.gibba002.stdout: osd scrub max interval = 600 2021-02-10T11:26:27.513 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:26:27.513 INFO:teuthology.orchestra.run.gibba002.stdout: osd recover clone overlap = true 2021-02-10T11:26:27.513 INFO:teuthology.orchestra.run.gibba002.stdout: osd recovery max chunk = 1048576 2021-02-10T11:26:27.513 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:26:27.513 INFO:teuthology.orchestra.run.gibba002.stdout: osd debug shutdown = true 2021-02-10T11:26:27.513 INFO:teuthology.orchestra.run.gibba002.stdout: osd debug op order = true 2021-02-10T11:26:27.513 INFO:teuthology.orchestra.run.gibba002.stdout: osd debug verify stray on activate = true 2021-02-10T11:26:27.513 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:26:27.513 INFO:teuthology.orchestra.run.gibba002.stdout: osd open classes on start = true 2021-02-10T11:26:27.513 INFO:teuthology.orchestra.run.gibba002.stdout: osd debug pg log writeout = true 2021-02-10T11:26:27.514 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:26:27.514 INFO:teuthology.orchestra.run.gibba002.stdout: osd deep scrub update digest min age = 30 2021-02-10T11:26:27.514 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:26:27.514 INFO:teuthology.orchestra.run.gibba002.stdout: osd map max advance = 10 2021-02-10T11:26:27.514 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:26:27.514 INFO:teuthology.orchestra.run.gibba002.stdout: journal zero on create = true 2021-02-10T11:26:27.514 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:26:27.514 INFO:teuthology.orchestra.run.gibba002.stdout: filestore ondisk finisher threads = 3 2021-02-10T11:26:27.514 INFO:teuthology.orchestra.run.gibba002.stdout: filestore apply finisher threads = 3 2021-02-10T11:26:27.514 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:26:27.515 INFO:teuthology.orchestra.run.gibba002.stdout: bdev debug aio = true 2021-02-10T11:26:27.515 INFO:teuthology.orchestra.run.gibba002.stdout: osd debug misdirected ops = true 2021-02-10T11:26:27.515 INFO:teuthology.orchestra.run.gibba002.stdout: bdev async discard = True 2021-02-10T11:26:27.515 INFO:teuthology.orchestra.run.gibba002.stdout: bdev enable discard = True 2021-02-10T11:26:27.515 INFO:teuthology.orchestra.run.gibba002.stdout: bluestore allocator = bitmap 2021-02-10T11:26:27.515 INFO:teuthology.orchestra.run.gibba002.stdout: bluestore block size = 96636764160 2021-02-10T11:26:27.515 INFO:teuthology.orchestra.run.gibba002.stdout: bluestore fsck on mount = True 2021-02-10T11:26:27.515 INFO:teuthology.orchestra.run.gibba002.stdout: debug bluefs = 20 2021-02-10T11:26:27.515 INFO:teuthology.orchestra.run.gibba002.stdout: debug bluestore = 20 2021-02-10T11:26:27.515 INFO:teuthology.orchestra.run.gibba002.stdout: debug filestore = 20 2021-02-10T11:26:27.515 INFO:teuthology.orchestra.run.gibba002.stdout: debug journal = 20 2021-02-10T11:26:27.515 INFO:teuthology.orchestra.run.gibba002.stdout: debug ms = 20 2021-02-10T11:26:27.515 INFO:teuthology.orchestra.run.gibba002.stdout: debug osd = 25 2021-02-10T11:26:27.516 INFO:teuthology.orchestra.run.gibba002.stdout: debug rocksdb = 10 2021-02-10T11:26:27.516 INFO:teuthology.orchestra.run.gibba002.stdout: mon osd backfillfull_ratio = 0.85 2021-02-10T11:26:27.516 INFO:teuthology.orchestra.run.gibba002.stdout: mon osd full ratio = 0.9 2021-02-10T11:26:27.516 INFO:teuthology.orchestra.run.gibba002.stdout: mon osd nearfull ratio = 0.8 2021-02-10T11:26:27.516 INFO:teuthology.orchestra.run.gibba002.stdout: osd failsafe full ratio = 0.95 2021-02-10T11:26:27.516 INFO:teuthology.orchestra.run.gibba002.stdout: osd objectstore = bluestore 2021-02-10T11:26:27.516 INFO:teuthology.orchestra.run.gibba002.stdout: osd op complaint time = 180 2021-02-10T11:26:27.516 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:26:27.516 INFO:teuthology.orchestra.run.gibba002.stdout:[mgr] 2021-02-10T11:26:27.516 INFO:teuthology.orchestra.run.gibba002.stdout: debug ms = 1 2021-02-10T11:26:27.516 INFO:teuthology.orchestra.run.gibba002.stdout: debug mgr = 20 2021-02-10T11:26:27.516 INFO:teuthology.orchestra.run.gibba002.stdout: debug mon = 20 2021-02-10T11:26:27.516 INFO:teuthology.orchestra.run.gibba002.stdout: debug auth = 20 2021-02-10T11:26:27.516 INFO:teuthology.orchestra.run.gibba002.stdout: mon reweight min pgs per osd = 4 2021-02-10T11:26:27.517 INFO:teuthology.orchestra.run.gibba002.stdout: mon reweight min bytes per osd = 10 2021-02-10T11:26:27.517 INFO:teuthology.orchestra.run.gibba002.stdout: debug client = 10 2021-02-10T11:26:27.517 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:26:27.517 INFO:teuthology.orchestra.run.gibba002.stdout:[mon] 2021-02-10T11:26:27.517 INFO:teuthology.orchestra.run.gibba002.stdout: debug ms = 1 2021-02-10T11:26:27.517 INFO:teuthology.orchestra.run.gibba002.stdout: debug mon = 20 2021-02-10T11:26:27.517 INFO:teuthology.orchestra.run.gibba002.stdout: debug paxos = 20 2021-02-10T11:26:27.517 INFO:teuthology.orchestra.run.gibba002.stdout: debug auth = 20 2021-02-10T11:26:27.517 INFO:teuthology.orchestra.run.gibba002.stdout: mon data avail warn = 5 2021-02-10T11:26:27.517 INFO:teuthology.orchestra.run.gibba002.stdout: mon mgr mkfs grace = 240 2021-02-10T11:26:27.517 INFO:teuthology.orchestra.run.gibba002.stdout: mon reweight min pgs per osd = 4 2021-02-10T11:26:27.517 INFO:teuthology.orchestra.run.gibba002.stdout: mon osd reporter subtree level = osd 2021-02-10T11:26:27.517 INFO:teuthology.orchestra.run.gibba002.stdout: mon osd prime pg temp = true 2021-02-10T11:26:27.518 INFO:teuthology.orchestra.run.gibba002.stdout: mon reweight min bytes per osd = 10 2021-02-10T11:26:27.518 INFO:teuthology.orchestra.run.gibba002.stdout: mon op complaint time = 120 2021-02-10T11:26:27.518 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:26:27.518 INFO:teuthology.orchestra.run.gibba002.stdout:[client] 2021-02-10T11:26:27.518 INFO:teuthology.orchestra.run.gibba002.stdout: rgw cache enabled = true 2021-02-10T11:26:27.518 INFO:teuthology.orchestra.run.gibba002.stdout: rgw enable ops log = true 2021-02-10T11:26:27.518 INFO:teuthology.orchestra.run.gibba002.stdout: rgw enable usage log = true 2021-02-10T11:26:27.518 INFO:teuthology.orchestra.run.gibba002.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2021-02-10T11:26:27.518 INFO:teuthology.orchestra.run.gibba002.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2021-02-10T11:26:27.518 INFO:teuthology.orchestra.run.gibba002.stdout: client mount timeout = 600 2021-02-10T11:26:27.518 INFO:teuthology.orchestra.run.gibba002.stdout: debug client = 20 2021-02-10T11:26:27.518 INFO:teuthology.orchestra.run.gibba002.stdout: debug ms = 1 2021-02-10T11:26:27.518 INFO:teuthology.orchestra.run.gibba002.stdout: pid file = "" 2021-02-10T11:26:27.518 INFO:teuthology.orchestra.run.gibba002.stdout:[mon.a] 2021-02-10T11:26:27.519 INFO:teuthology.orchestra.run.gibba002.stdout:[mon.b] 2021-02-10T11:26:27.519 INFO:teuthology.orchestra.run.gibba002.stdout:[mon.c] 2021-02-10T11:26:27.519 INFO:teuthology.orchestra.run.gibba002.stdout:[mds.a] 2021-02-10T11:26:27.519 INFO:teuthology.orchestra.run.gibba002.stdout:[mds.b] 2021-02-10T11:26:27.519 INFO:teuthology.orchestra.run.gibba002.stdout:[mds] 2021-02-10T11:26:27.519 INFO:teuthology.orchestra.run.gibba002.stdout: debug mds = 20 2021-02-10T11:26:27.519 INFO:teuthology.orchestra.run.gibba002.stdout: debug ms = 1 2021-02-10T11:26:27.519 INFO:teuthology.orchestra.run.gibba002.stdout: mds bal frag = True 2021-02-10T11:26:27.519 INFO:teuthology.orchestra.run.gibba002.stdout: mds bal fragment size max = 10000 2021-02-10T11:26:27.519 INFO:teuthology.orchestra.run.gibba002.stdout: mds bal merge size = 5 2021-02-10T11:26:27.519 INFO:teuthology.orchestra.run.gibba002.stdout: mds bal split bits = 3 2021-02-10T11:26:27.519 INFO:teuthology.orchestra.run.gibba002.stdout: mds bal split size = 100 2021-02-10T11:26:27.519 INFO:teuthology.orchestra.run.gibba002.stdout: mds debug frag = True 2021-02-10T11:26:27.519 INFO:teuthology.orchestra.run.gibba002.stdout: mds debug scatterstat = True 2021-02-10T11:26:27.520 INFO:teuthology.orchestra.run.gibba002.stdout: mds op complaint time = 180 2021-02-10T11:26:27.520 INFO:teuthology.orchestra.run.gibba002.stdout: mds verify scatter = True 2021-02-10T11:26:27.520 INFO:teuthology.orchestra.run.gibba002.stdout: osd op complaint time = 180 2021-02-10T11:26:27.520 INFO:teuthology.orchestra.run.gibba002.stdout:[client.alice] 2021-02-10T11:26:27.521 INFO:teuthology.orchestra.run.gibba013.stdout:[global] 2021-02-10T11:26:27.521 INFO:teuthology.orchestra.run.gibba013.stdout: chdir = "" 2021-02-10T11:26:27.521 INFO:teuthology.orchestra.run.gibba013.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2021-02-10T11:26:27.521 INFO:teuthology.orchestra.run.gibba013.stdout: auth supported = cephx 2021-02-10T11:26:27.521 INFO:teuthology.orchestra.run.gibba013.stdout: 2021-02-10T11:26:27.521 INFO:teuthology.orchestra.run.gibba013.stdout: filestore xattr use omap = true 2021-02-10T11:26:27.521 INFO:teuthology.orchestra.run.gibba013.stdout: 2021-02-10T11:26:27.521 INFO:teuthology.orchestra.run.gibba013.stdout: mon clock drift allowed = 1.000 2021-02-10T11:26:27.521 INFO:teuthology.orchestra.run.gibba013.stdout: 2021-02-10T11:26:27.521 INFO:teuthology.orchestra.run.gibba013.stdout: osd crush chooseleaf type = 0 2021-02-10T11:26:27.521 INFO:teuthology.orchestra.run.gibba013.stdout: auth debug = true 2021-02-10T11:26:27.522 INFO:teuthology.orchestra.run.gibba013.stdout: 2021-02-10T11:26:27.522 INFO:teuthology.orchestra.run.gibba013.stdout: ms die on old message = true 2021-02-10T11:26:27.522 INFO:teuthology.orchestra.run.gibba013.stdout: ms die on bug = true 2021-02-10T11:26:27.522 INFO:teuthology.orchestra.run.gibba013.stdout: 2021-02-10T11:26:27.522 INFO:teuthology.orchestra.run.gibba013.stdout: mon max pg per osd = 10000 # >= luminous 2021-02-10T11:26:27.522 INFO:teuthology.orchestra.run.gibba013.stdout: mon pg warn max object skew = 0 2021-02-10T11:26:27.522 INFO:teuthology.orchestra.run.gibba013.stdout: 2021-02-10T11:26:27.522 INFO:teuthology.orchestra.run.gibba013.stdout: osd pool default size = 2 2021-02-10T11:26:27.522 INFO:teuthology.orchestra.run.gibba013.stdout: 2021-02-10T11:26:27.522 INFO:teuthology.orchestra.run.gibba013.stdout: mon osd allow primary affinity = true 2021-02-10T11:26:27.522 INFO:teuthology.orchestra.run.gibba013.stdout: mon osd allow pg remap = true 2021-02-10T11:26:27.522 INFO:teuthology.orchestra.run.gibba013.stdout: mon warn on legacy crush tunables = false 2021-02-10T11:26:27.522 INFO:teuthology.orchestra.run.gibba013.stdout: mon warn on crush straw calc version zero = false 2021-02-10T11:26:27.522 INFO:teuthology.orchestra.run.gibba013.stdout: mon warn on no sortbitwise = false 2021-02-10T11:26:27.523 INFO:teuthology.orchestra.run.gibba013.stdout: mon warn on osd down out interval zero = false 2021-02-10T11:26:27.523 INFO:teuthology.orchestra.run.gibba013.stdout: mon warn on too few osds = false 2021-02-10T11:26:27.523 INFO:teuthology.orchestra.run.gibba013.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2021-02-10T11:26:27.523 INFO:teuthology.orchestra.run.gibba013.stdout: mon_warn_on_pool_no_redundancy = false 2021-02-10T11:26:27.523 INFO:teuthology.orchestra.run.gibba013.stdout: 2021-02-10T11:26:27.523 INFO:teuthology.orchestra.run.gibba013.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2021-02-10T11:26:27.523 INFO:teuthology.orchestra.run.gibba013.stdout: 2021-02-10T11:26:27.523 INFO:teuthology.orchestra.run.gibba013.stdout: osd default data pool replay window = 5 2021-02-10T11:26:27.523 INFO:teuthology.orchestra.run.gibba013.stdout: 2021-02-10T11:26:27.523 INFO:teuthology.orchestra.run.gibba013.stdout: mon allow pool delete = true 2021-02-10T11:26:27.523 INFO:teuthology.orchestra.run.gibba013.stdout: 2021-02-10T11:26:27.523 INFO:teuthology.orchestra.run.gibba013.stdout: mon cluster log file level = debug 2021-02-10T11:26:27.523 INFO:teuthology.orchestra.run.gibba013.stdout: debug asserts on shutdown = true 2021-02-10T11:26:27.524 INFO:teuthology.orchestra.run.gibba013.stdout: mon health detail to clog = false 2021-02-10T11:26:27.524 INFO:teuthology.orchestra.run.gibba013.stdout: 2021-02-10T11:26:27.524 INFO:teuthology.orchestra.run.gibba013.stdout: # we see this fail in qa on *nautilus*; bump up retries 2021-02-10T11:26:27.524 INFO:teuthology.orchestra.run.gibba013.stdout: mon_client_directed_command_retry = 4 2021-02-10T11:26:27.524 INFO:teuthology.orchestra.run.gibba013.stdout: mon host = "172.21.2.102,172.21.2.113,[v2:172.21.2.113:3301,v1:172.21.2.113:6790]" 2021-02-10T11:26:27.524 INFO:teuthology.orchestra.run.gibba013.stdout: fsid = 68db30ea-dbce-4aef-a6df-f01c6fea0513 2021-02-10T11:26:27.524 INFO:teuthology.orchestra.run.gibba013.stdout: 2021-02-10T11:26:27.524 INFO:teuthology.orchestra.run.gibba013.stdout:[osd] 2021-02-10T11:26:27.524 INFO:teuthology.orchestra.run.gibba013.stdout: osd journal size = 100 2021-02-10T11:26:27.524 INFO:teuthology.orchestra.run.gibba013.stdout: 2021-02-10T11:26:27.524 INFO:teuthology.orchestra.run.gibba013.stdout: osd scrub load threshold = 5.0 2021-02-10T11:26:27.524 INFO:teuthology.orchestra.run.gibba013.stdout: osd scrub max interval = 600 2021-02-10T11:26:27.524 INFO:teuthology.orchestra.run.gibba013.stdout: 2021-02-10T11:26:27.525 INFO:teuthology.orchestra.run.gibba013.stdout: osd recover clone overlap = true 2021-02-10T11:26:27.525 INFO:teuthology.orchestra.run.gibba013.stdout: osd recovery max chunk = 1048576 2021-02-10T11:26:27.525 INFO:teuthology.orchestra.run.gibba013.stdout: 2021-02-10T11:26:27.525 INFO:teuthology.orchestra.run.gibba013.stdout: osd debug shutdown = true 2021-02-10T11:26:27.525 INFO:teuthology.orchestra.run.gibba013.stdout: osd debug op order = true 2021-02-10T11:26:27.525 INFO:teuthology.orchestra.run.gibba013.stdout: osd debug verify stray on activate = true 2021-02-10T11:26:27.525 INFO:teuthology.orchestra.run.gibba013.stdout: 2021-02-10T11:26:27.525 INFO:teuthology.orchestra.run.gibba013.stdout: osd open classes on start = true 2021-02-10T11:26:27.525 INFO:teuthology.orchestra.run.gibba013.stdout: osd debug pg log writeout = true 2021-02-10T11:26:27.525 INFO:teuthology.orchestra.run.gibba013.stdout: 2021-02-10T11:26:27.525 INFO:teuthology.orchestra.run.gibba013.stdout: osd deep scrub update digest min age = 30 2021-02-10T11:26:27.525 INFO:teuthology.orchestra.run.gibba013.stdout: 2021-02-10T11:26:27.525 INFO:teuthology.orchestra.run.gibba013.stdout: osd map max advance = 10 2021-02-10T11:26:27.525 INFO:teuthology.orchestra.run.gibba013.stdout: 2021-02-10T11:26:27.525 INFO:teuthology.orchestra.run.gibba013.stdout: journal zero on create = true 2021-02-10T11:26:27.526 INFO:teuthology.orchestra.run.gibba013.stdout: 2021-02-10T11:26:27.526 INFO:teuthology.orchestra.run.gibba013.stdout: filestore ondisk finisher threads = 3 2021-02-10T11:26:27.526 INFO:teuthology.orchestra.run.gibba013.stdout: filestore apply finisher threads = 3 2021-02-10T11:26:27.526 INFO:teuthology.orchestra.run.gibba013.stdout: 2021-02-10T11:26:27.526 INFO:teuthology.orchestra.run.gibba013.stdout: bdev debug aio = true 2021-02-10T11:26:27.526 INFO:teuthology.orchestra.run.gibba013.stdout: osd debug misdirected ops = true 2021-02-10T11:26:27.526 INFO:teuthology.orchestra.run.gibba013.stdout: bdev async discard = True 2021-02-10T11:26:27.526 INFO:teuthology.orchestra.run.gibba013.stdout: bdev enable discard = True 2021-02-10T11:26:27.526 INFO:teuthology.orchestra.run.gibba013.stdout: bluestore allocator = bitmap 2021-02-10T11:26:27.526 INFO:teuthology.orchestra.run.gibba013.stdout: bluestore block size = 96636764160 2021-02-10T11:26:27.526 INFO:teuthology.orchestra.run.gibba013.stdout: bluestore fsck on mount = True 2021-02-10T11:26:27.526 INFO:teuthology.orchestra.run.gibba013.stdout: debug bluefs = 20 2021-02-10T11:26:27.526 INFO:teuthology.orchestra.run.gibba013.stdout: debug bluestore = 20 2021-02-10T11:26:27.527 INFO:teuthology.orchestra.run.gibba013.stdout: debug filestore = 20 2021-02-10T11:26:27.527 INFO:teuthology.orchestra.run.gibba013.stdout: debug journal = 20 2021-02-10T11:26:27.527 INFO:teuthology.orchestra.run.gibba013.stdout: debug ms = 20 2021-02-10T11:26:27.527 INFO:teuthology.orchestra.run.gibba013.stdout: debug osd = 25 2021-02-10T11:26:27.527 INFO:teuthology.orchestra.run.gibba013.stdout: debug rocksdb = 10 2021-02-10T11:26:27.527 INFO:teuthology.orchestra.run.gibba013.stdout: mon osd backfillfull_ratio = 0.85 2021-02-10T11:26:27.527 INFO:teuthology.orchestra.run.gibba013.stdout: mon osd full ratio = 0.9 2021-02-10T11:26:27.527 INFO:teuthology.orchestra.run.gibba013.stdout: mon osd nearfull ratio = 0.8 2021-02-10T11:26:27.527 INFO:teuthology.orchestra.run.gibba013.stdout: osd failsafe full ratio = 0.95 2021-02-10T11:26:27.527 INFO:teuthology.orchestra.run.gibba013.stdout: osd objectstore = bluestore 2021-02-10T11:26:27.527 INFO:teuthology.orchestra.run.gibba013.stdout: osd op complaint time = 180 2021-02-10T11:26:27.527 INFO:teuthology.orchestra.run.gibba013.stdout: 2021-02-10T11:26:27.527 INFO:teuthology.orchestra.run.gibba013.stdout:[mgr] 2021-02-10T11:26:27.527 INFO:teuthology.orchestra.run.gibba013.stdout: debug ms = 1 2021-02-10T11:26:27.528 INFO:teuthology.orchestra.run.gibba013.stdout: debug mgr = 20 2021-02-10T11:26:27.528 INFO:teuthology.orchestra.run.gibba013.stdout: debug mon = 20 2021-02-10T11:26:27.528 INFO:teuthology.orchestra.run.gibba013.stdout: debug auth = 20 2021-02-10T11:26:27.528 INFO:teuthology.orchestra.run.gibba013.stdout: mon reweight min pgs per osd = 4 2021-02-10T11:26:27.528 INFO:teuthology.orchestra.run.gibba013.stdout: mon reweight min bytes per osd = 10 2021-02-10T11:26:27.528 INFO:teuthology.orchestra.run.gibba013.stdout: debug client = 10 2021-02-10T11:26:27.528 INFO:teuthology.orchestra.run.gibba013.stdout: 2021-02-10T11:26:27.528 INFO:teuthology.orchestra.run.gibba013.stdout:[mon] 2021-02-10T11:26:27.528 INFO:teuthology.orchestra.run.gibba013.stdout: debug ms = 1 2021-02-10T11:26:27.528 INFO:teuthology.orchestra.run.gibba013.stdout: debug mon = 20 2021-02-10T11:26:27.528 INFO:teuthology.orchestra.run.gibba013.stdout: debug paxos = 20 2021-02-10T11:26:27.528 INFO:teuthology.orchestra.run.gibba013.stdout: debug auth = 20 2021-02-10T11:26:27.528 INFO:teuthology.orchestra.run.gibba013.stdout: mon data avail warn = 5 2021-02-10T11:26:27.528 INFO:teuthology.orchestra.run.gibba013.stdout: mon mgr mkfs grace = 240 2021-02-10T11:26:27.529 INFO:teuthology.orchestra.run.gibba013.stdout: mon reweight min pgs per osd = 4 2021-02-10T11:26:27.529 INFO:teuthology.orchestra.run.gibba013.stdout: mon osd reporter subtree level = osd 2021-02-10T11:26:27.529 INFO:teuthology.orchestra.run.gibba013.stdout: mon osd prime pg temp = true 2021-02-10T11:26:27.529 INFO:teuthology.orchestra.run.gibba013.stdout: mon reweight min bytes per osd = 10 2021-02-10T11:26:27.529 INFO:teuthology.orchestra.run.gibba013.stdout: mon op complaint time = 120 2021-02-10T11:26:27.529 INFO:teuthology.orchestra.run.gibba013.stdout: 2021-02-10T11:26:27.529 INFO:teuthology.orchestra.run.gibba013.stdout:[client] 2021-02-10T11:26:27.529 INFO:teuthology.orchestra.run.gibba013.stdout: rgw cache enabled = true 2021-02-10T11:26:27.529 INFO:teuthology.orchestra.run.gibba013.stdout: rgw enable ops log = true 2021-02-10T11:26:27.529 INFO:teuthology.orchestra.run.gibba013.stdout: rgw enable usage log = true 2021-02-10T11:26:27.529 INFO:teuthology.orchestra.run.gibba013.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2021-02-10T11:26:27.529 INFO:teuthology.orchestra.run.gibba013.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2021-02-10T11:26:27.529 INFO:teuthology.orchestra.run.gibba013.stdout: client mount timeout = 600 2021-02-10T11:26:27.530 INFO:teuthology.orchestra.run.gibba013.stdout: debug client = 20 2021-02-10T11:26:27.530 INFO:teuthology.orchestra.run.gibba013.stdout: debug ms = 1 2021-02-10T11:26:27.530 INFO:teuthology.orchestra.run.gibba013.stdout: pid file = "" 2021-02-10T11:26:27.530 INFO:teuthology.orchestra.run.gibba013.stdout:[mon.a] 2021-02-10T11:26:27.530 INFO:teuthology.orchestra.run.gibba013.stdout:[mon.b] 2021-02-10T11:26:27.530 INFO:teuthology.orchestra.run.gibba013.stdout:[mon.c] 2021-02-10T11:26:27.530 INFO:teuthology.orchestra.run.gibba013.stdout:[mds.a] 2021-02-10T11:26:27.530 INFO:teuthology.orchestra.run.gibba013.stdout:[mds.b] 2021-02-10T11:26:27.530 INFO:teuthology.orchestra.run.gibba013.stdout:[mds] 2021-02-10T11:26:27.530 INFO:teuthology.orchestra.run.gibba013.stdout: debug mds = 20 2021-02-10T11:26:27.530 INFO:teuthology.orchestra.run.gibba013.stdout: debug ms = 1 2021-02-10T11:26:27.530 INFO:teuthology.orchestra.run.gibba013.stdout: mds bal frag = True 2021-02-10T11:26:27.530 INFO:teuthology.orchestra.run.gibba013.stdout: mds bal fragment size max = 10000 2021-02-10T11:26:27.530 INFO:teuthology.orchestra.run.gibba013.stdout: mds bal merge size = 5 2021-02-10T11:26:27.531 INFO:teuthology.orchestra.run.gibba013.stdout: mds bal split bits = 3 2021-02-10T11:26:27.531 INFO:teuthology.orchestra.run.gibba013.stdout: mds bal split size = 100 2021-02-10T11:26:27.531 INFO:teuthology.orchestra.run.gibba013.stdout: mds debug frag = True 2021-02-10T11:26:27.531 INFO:teuthology.orchestra.run.gibba013.stdout: mds debug scatterstat = True 2021-02-10T11:26:27.531 INFO:teuthology.orchestra.run.gibba013.stdout: mds op complaint time = 180 2021-02-10T11:26:27.531 INFO:teuthology.orchestra.run.gibba013.stdout: mds verify scatter = True 2021-02-10T11:26:27.531 INFO:teuthology.orchestra.run.gibba013.stdout: osd op complaint time = 180 2021-02-10T11:26:27.531 INFO:teuthology.orchestra.run.gibba013.stdout:[client.alice] 2021-02-10T11:26:27.532 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config rm mon mon_allow_pool_delete 2021-02-10T11:26:27.963 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:27.947 7fd2cf74d700 1 Processor -- start 2021-02-10T11:26:27.963 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:27.947 7fd2cf74d700 1 -- start start 2021-02-10T11:26:27.963 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:27.947 7fd2cf74d700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd2c80d0340 0x7fd2c80ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:27.963 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:27.947 7fd2cf74d700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fd2c80d1c70 0x7fd2c80efc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:27.963 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:27.947 7fd2cf74d700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2c80f0b10 con 0x7fd2c80d1870 2021-02-10T11:26:27.964 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:27.947 7fd2cf74d700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2c80e5ed0 con 0x7fd2c80d1c70 2021-02-10T11:26:27.964 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:27.947 7fd2cf74d700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2c81b0be0 con 0x7fd2c80d0340 2021-02-10T11:26:27.964 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:27.948 7fd2cdd6a700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fd2c80d1c70 0x7fd2c80efc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:27.964 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:27.948 7fd2cdd6a700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fd2c80d1c70 0x7fd2c80efc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:58676/0 (socket says 172.21.2.102:58676) 2021-02-10T11:26:27.964 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:27.948 7fd2cdd6a700 1 -- 172.21.2.102:0/2687579419 learned_addr learned my addr 172.21.2.102:0/2687579419 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:26:27.964 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:27.948 7fd2ccd68700 1 --2- 172.21.2.102:0/2687579419 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd2c80d0340 0x7fd2c80ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:27.964 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:27.948 7fd2cdd6a700 1 -- 172.21.2.102:0/2687579419 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd2c80d0340 msgr2=0x7fd2c80ef640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:26:27.964 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:27.948 7fd2cdd6a700 1 --2- 172.21.2.102:0/2687579419 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd2c80d0340 0x7fd2c80ef640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:27.964 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:27.948 7fd2cdd6a700 1 -- 172.21.2.102:0/2687579419 >> v1:172.21.2.113:6789/0 conn(0x7fd2c80d1870 legacy=0x7fd2c81b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:27.964 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:27.948 7fd2cdd6a700 1 -- 172.21.2.102:0/2687579419 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2c80d2fe0 con 0x7fd2c80d1c70 2021-02-10T11:26:27.964 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:27.948 7fd2cdd6a700 1 --2- 172.21.2.102:0/2687579419 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd2c80d1c70 0x7fd2c80efc60 secure :-1 s=READY pgs=1601 cs=0 l=1 rev1=1 rx=0x7fd2c0006f00 tx=0x7fd2c0009f10).ready entity=mon.0 client_cookie=77b145564fbb1186 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:27.964 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:27.948 7fd2bffff700 1 -- 172.21.2.102:0/2687579419 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd2c000b3b0 con 0x7fd2c80d1c70 2021-02-10T11:26:27.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:27.948 7fd2bffff700 1 -- 172.21.2.102:0/2687579419 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2c000b730 con 0x7fd2c80d1c70 2021-02-10T11:26:27.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:27.948 7fd2bffff700 1 -- 172.21.2.102:0/2687579419 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd2c000bc10 con 0x7fd2c80d1c70 2021-02-10T11:26:27.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:27.948 7fd2cf74d700 1 -- 172.21.2.102:0/2687579419 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd2c80d1c70 msgr2=0x7fd2c80efc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:27.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:27.948 7fd2cf74d700 1 --2- 172.21.2.102:0/2687579419 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd2c80d1c70 0x7fd2c80efc60 secure :-1 s=READY pgs=1601 cs=0 l=1 rev1=1 rx=0x7fd2c0006f00 tx=0x7fd2c0009f10).stop 2021-02-10T11:26:27.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:27.949 7fd2cf74d700 1 -- 172.21.2.102:0/2687579419 shutdown_connections 2021-02-10T11:26:27.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:27.949 7fd2cf74d700 1 --2- 172.21.2.102:0/2687579419 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd2c80d0340 0x7fd2c80ef640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:27.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:27.949 7fd2cf74d700 1 --2- 172.21.2.102:0/2687579419 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd2c80d1c70 0x7fd2c80efc60 unknown :-1 s=CLOSED pgs=1601 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:27.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:27.949 7fd2cf74d700 1 -- 172.21.2.102:0/2687579419 >> 172.21.2.102:0/2687579419 conn(0x7fd2c80ce450 msgr2=0x7fd2c80d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:26:27.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:27.949 7fd2cf74d700 1 -- 172.21.2.102:0/2687579419 shutdown_connections 2021-02-10T11:26:27.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:27.949 7fd2cf74d700 1 -- 172.21.2.102:0/2687579419 wait complete. 2021-02-10T11:26:27.967 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:27.964 7fd2cf74d700 1 Processor -- start 2021-02-10T11:26:27.967 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:27.964 7fd2cf74d700 1 -- start start 2021-02-10T11:26:27.967 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:27.964 7fd2cf74d700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd2c8094310 0x7fd2c80ebe70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:27.967 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:27.964 7fd2cf74d700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd2c8094710 0x7fd2c80bca30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:27.967 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:27.964 7fd2cf74d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd2c80ec490 0x7fd2c80bd050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:27.967 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:27.964 7fd2cf74d700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2c8093d90 con 0x7fd2c80ec490 2021-02-10T11:26:27.968 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:27.964 7fd2cf74d700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2c80bd670 con 0x7fd2c8094310 2021-02-10T11:26:27.968 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:27.964 7fd2cf74d700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2c80939f0 con 0x7fd2c8094710 2021-02-10T11:26:27.968 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:27.965 7fd2cdd6a700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd2c80ec490 0x7fd2c80bd050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:27.968 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:27.965 7fd2cdd6a700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd2c80ec490 0x7fd2c80bd050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:58682/0 (socket says 172.21.2.102:58682) 2021-02-10T11:26:27.968 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:27.965 7fd2cdd6a700 1 -- 172.21.2.102:0/2662365821 learned_addr learned my addr 172.21.2.102:0/2662365821 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:26:27.968 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:27.965 7fd2ccd68700 1 --2- 172.21.2.102:0/2662365821 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd2c8094710 0x7fd2c80bca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:27.968 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:27.965 7fd2cd569700 1 --2- 172.21.2.102:0/2662365821 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd2c8094310 0x7fd2c80ebe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:27.968 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:27.965 7fd2ccd68700 1 -- 172.21.2.102:0/2662365821 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd2c8094310 msgr2=0x7fd2c80ebe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:27.968 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:27.965 7fd2ccd68700 1 --2- 172.21.2.102:0/2662365821 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd2c8094310 0x7fd2c80ebe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:27.968 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:27.965 7fd2ccd68700 1 -- 172.21.2.102:0/2662365821 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd2c80ec490 msgr2=0x7fd2c80bd050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:27.968 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:27.965 7fd2ccd68700 1 --2- 172.21.2.102:0/2662365821 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd2c80ec490 0x7fd2c80bd050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:27.968 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:27.965 7fd2ccd68700 1 -- 172.21.2.102:0/2662365821 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2c80bda80 con 0x7fd2c8094710 2021-02-10T11:26:27.969 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:27.965 7fd2cdd6a700 1 --2- 172.21.2.102:0/2662365821 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd2c80ec490 0x7fd2c80bd050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:26:27.969 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:27.965 7fd2cd569700 1 --2- 172.21.2.102:0/2662365821 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd2c8094310 0x7fd2c80ebe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:26:27.969 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:27.965 7fd2ccd68700 1 --2- 172.21.2.102:0/2662365821 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd2c8094710 0x7fd2c80bca30 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7fd2b8009540 tx=0x7fd2b800b8e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:27.969 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:27.966 7fd2bdffb700 1 -- 172.21.2.102:0/2662365821 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd2b80009b0 con 0x7fd2c8094710 2021-02-10T11:26:27.969 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:27.966 7fd2bdffb700 1 -- 172.21.2.102:0/2662365821 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2b8006340 con 0x7fd2c8094710 2021-02-10T11:26:27.969 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:27.966 7fd2bdffb700 1 -- 172.21.2.102:0/2662365821 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd2b8002030 con 0x7fd2c8094710 2021-02-10T11:26:27.969 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:27.966 7fd2cf74d700 1 -- 172.21.2.102:0/2662365821 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2c80bda80 con 0x7fd2c8094710 2021-02-10T11:26:27.969 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:27.966 7fd2cf74d700 1 -- 172.21.2.102:0/2662365821 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2c80bda80 con 0x7fd2c8094710 2021-02-10T11:26:27.969 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:27.966 7fd2bdffb700 1 -- 172.21.2.102:0/2662365821 <== mon.2 v2:172.21.2.113:3301/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd2b8000c90 con 0x7fd2c8094710 2021-02-10T11:26:27.969 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:27.966 7fd2bdffb700 1 --2- 172.21.2.102:0/2662365821 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd2a80186c0 0x7fd2a801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:27.969 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:27.966 7fd2bdffb700 1 -- 172.21.2.102:0/2662365821 <== mon.2 v2:172.21.2.113:3301/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fd2b8032340 con 0x7fd2c8094710 2021-02-10T11:26:27.969 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:27.967 7fd2cd569700 1 --2- 172.21.2.102:0/2662365821 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd2a80186c0 0x7fd2a801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:27.969 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:27.967 7fd2cd569700 1 --2- 172.21.2.102:0/2662365821 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd2a80186c0 0x7fd2a801abc0 crc :-1 s=READY pgs=726 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:27.996 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:27.997 7fd2cf74d700 1 -- 172.21.2.102:0/2662365821 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2c80ebc40 con 0x7fd2c8094710 2021-02-10T11:26:27.999 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:28.000 7fd2bdffb700 1 -- 172.21.2.102:0/2662365821 <== mon.2 v2:172.21.2.113:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd2b8006ae0 con 0x7fd2c8094710 2021-02-10T11:26:28.153 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:28.153 7fd2cf74d700 1 -- 172.21.2.102:0/2662365821 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fd2c80f0b10 con 0x7fd2c8094710 2021-02-10T11:26:28.159 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:28.160 7fd2bdffb700 1 -- 172.21.2.102:0/2662365821 <== mon.2 v2:172.21.2.113:3301/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v17) v1 ==== 104+0+0 (secure 0 0 0) 0x7fd2b8006740 con 0x7fd2c8094710 2021-02-10T11:26:28.161 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:28.162 7fd2cf74d700 1 -- 172.21.2.102:0/2662365821 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd2a80186c0 msgr2=0x7fd2a801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:28.161 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:28.162 7fd2cf74d700 1 --2- 172.21.2.102:0/2662365821 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd2a80186c0 0x7fd2a801abc0 crc :-1 s=READY pgs=726 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:28.162 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:28.162 7fd2cf74d700 1 -- 172.21.2.102:0/2662365821 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd2c8094710 msgr2=0x7fd2c80bca30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:28.162 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:28.162 7fd2cf74d700 1 --2- 172.21.2.102:0/2662365821 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd2c8094710 0x7fd2c80bca30 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7fd2b8009540 tx=0x7fd2b800b8e0).stop 2021-02-10T11:26:28.162 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:28.162 7fd2cf74d700 1 -- 172.21.2.102:0/2662365821 shutdown_connections 2021-02-10T11:26:28.162 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:28.162 7fd2cf74d700 1 --2- 172.21.2.102:0/2662365821 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd2c80ec490 0x7fd2c80bd050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:28.162 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:28.162 7fd2cf74d700 1 --2- 172.21.2.102:0/2662365821 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd2c8094710 0x7fd2c80bca30 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:28.162 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:28.162 7fd2cf74d700 1 --2- 172.21.2.102:0/2662365821 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd2a80186c0 0x7fd2a801abc0 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:28.163 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:28.162 7fd2cf74d700 1 --2- 172.21.2.102:0/2662365821 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd2c8094310 0x7fd2c80ebe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:28.163 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:28.162 7fd2cf74d700 1 -- 172.21.2.102:0/2662365821 >> 172.21.2.102:0/2662365821 conn(0x7fd2c808fa10 msgr2=0x7fd2c80bfa60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:26:28.163 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:28.162 7fd2cf74d700 1 -- 172.21.2.102:0/2662365821 shutdown_connections 2021-02-10T11:26:28.163 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:28.162 7fd2cf74d700 1 -- 172.21.2.102:0/2662365821 wait complete. 2021-02-10T11:26:28.170 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config rm client.alice debug_client 2021-02-10T11:26:28.603 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:28.587 7fb4de1ec700 1 Processor -- start 2021-02-10T11:26:28.603 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:28.587 7fb4de1ec700 1 -- start start 2021-02-10T11:26:28.603 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:28.588 7fb4de1ec700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fb4d80d1870 0x7fb4d81b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:28.603 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:28.588 7fb4de1ec700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fb4d80f0650 0x7fb4d80cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:28.603 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:28.588 7fb4de1ec700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4d80f0b10 con 0x7fb4d81b0a10 2021-02-10T11:26:28.604 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:28.588 7fb4de1ec700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4d80e5ed0 con 0x7fb4d80d1870 2021-02-10T11:26:28.604 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:28.588 7fb4de1ec700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4d80d1c70 con 0x7fb4d80f0650 2021-02-10T11:26:28.604 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:28.588 7fb4d7fff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fb4d80d1870 0x7fb4d81b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:28.604 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:28.588 7fb4d77fe700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fb4d81b0a10 0x7fb4d80cf330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:50910/0 (socket says 172.21.2.102:50910) 2021-02-10T11:26:28.604 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:28.588 7fb4d77fe700 1 -- 172.21.2.102:0/1384984266 learned_addr learned my addr 172.21.2.102:0/1384984266 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:26:28.604 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:28.588 7fb4d7fff700 1 -- 172.21.2.102:0/1384984266 >> v2:172.21.2.113:3300/0 conn(0x7fb4d80f0650 msgr2=0x7fb4d80cfb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:26:28.605 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:28.588 7fb4d7fff700 1 --2- 172.21.2.102:0/1384984266 >> v2:172.21.2.113:3300/0 conn(0x7fb4d80f0650 0x7fb4d80cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:28.605 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:28.588 7fb4d7fff700 1 -- 172.21.2.102:0/1384984266 >> v1:172.21.2.102:6789/0 conn(0x7fb4d81b0a10 legacy=0x7fb4d80cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:28.605 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:28.588 7fb4d7fff700 1 -- 172.21.2.102:0/1384984266 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4d80d1e90 con 0x7fb4d80d1870 2021-02-10T11:26:28.605 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:28.588 7fb4d7fff700 1 --2- 172.21.2.102:0/1384984266 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb4d80d1870 0x7fb4d81b03f0 secure :-1 s=READY pgs=1603 cs=0 l=1 rev1=1 rx=0x7fb4c8006650 tx=0x7fb4c8009a80).ready entity=mon.0 client_cookie=f1fb11356c652450 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:28.605 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:28.588 7fb4d6ffd700 1 -- 172.21.2.102:0/1384984266 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb4c80009b0 con 0x7fb4d80d1870 2021-02-10T11:26:28.606 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:28.588 7fb4d6ffd700 1 -- 172.21.2.102:0/1384984266 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4c800c0c0 con 0x7fb4d80d1870 2021-02-10T11:26:28.606 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:28.588 7fb4d6ffd700 1 -- 172.21.2.102:0/1384984266 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb4c800c5e0 con 0x7fb4d80d1870 2021-02-10T11:26:28.606 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:28.588 7fb4de1ec700 1 -- 172.21.2.102:0/1384984266 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb4d80d1870 msgr2=0x7fb4d81b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:28.606 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:28.588 7fb4de1ec700 1 --2- 172.21.2.102:0/1384984266 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb4d80d1870 0x7fb4d81b03f0 secure :-1 s=READY pgs=1603 cs=0 l=1 rev1=1 rx=0x7fb4c8006650 tx=0x7fb4c8009a80).stop 2021-02-10T11:26:28.606 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:28.589 7fb4de1ec700 1 -- 172.21.2.102:0/1384984266 shutdown_connections 2021-02-10T11:26:28.607 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:28.589 7fb4de1ec700 1 --2- 172.21.2.102:0/1384984266 >> v2:172.21.2.113:3300/0 conn(0x7fb4d80f0650 0x7fb4d80cfb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:28.607 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:28.589 7fb4de1ec700 1 --2- 172.21.2.102:0/1384984266 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb4d80d1870 0x7fb4d81b03f0 unknown :-1 s=CLOSED pgs=1603 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:28.607 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:28.589 7fb4de1ec700 1 -- 172.21.2.102:0/1384984266 >> 172.21.2.102:0/1384984266 conn(0x7fb4d80ce450 msgr2=0x7fb4d80d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:26:28.607 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:28.589 7fb4de1ec700 1 -- 172.21.2.102:0/1384984266 shutdown_connections 2021-02-10T11:26:28.607 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:28.589 7fb4de1ec700 1 -- 172.21.2.102:0/1384984266 wait complete. 2021-02-10T11:26:28.610 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:28.604 7fb4de1ec700 1 Processor -- start 2021-02-10T11:26:28.611 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:28.604 7fb4de1ec700 1 -- start start 2021-02-10T11:26:28.611 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:28.604 7fb4de1ec700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb4d80a2fb0 0x7fb4d80a33b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:28.611 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:28.604 7fb4de1ec700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb4d80a39d0 0x7fb4d80a4860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:28.611 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:28.604 7fb4de1ec700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb4d80a3dd0 0x7fb4d80a4e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:28.611 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:28.604 7fb4de1ec700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4d80a2620 con 0x7fb4d80a2fb0 2021-02-10T11:26:28.611 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:28.604 7fb4de1ec700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4d80a2840 con 0x7fb4d80a3dd0 2021-02-10T11:26:28.612 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:28.604 7fb4de1ec700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4d80a2b80 con 0x7fb4d80a39d0 2021-02-10T11:26:28.612 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:28.604 7fb4d7fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb4d80a2fb0 0x7fb4d80a33b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:28.612 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:28.604 7fb4d7fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb4d80a2fb0 0x7fb4d80a33b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:58692/0 (socket says 172.21.2.102:58692) 2021-02-10T11:26:28.612 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:28.604 7fb4d7fff700 1 -- 172.21.2.102:0/455055599 learned_addr learned my addr 172.21.2.102:0/455055599 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:26:28.612 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:28.605 7fb4d7fff700 1 -- 172.21.2.102:0/455055599 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb4d80a39d0 msgr2=0x7fb4d80a4860 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:26:28.613 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:28.605 7fb4d7fff700 1 --2- 172.21.2.102:0/455055599 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb4d80a39d0 0x7fb4d80a4860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:28.613 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:28.605 7fb4d7fff700 1 -- 172.21.2.102:0/455055599 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb4d80a3dd0 msgr2=0x7fb4d80a4e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:28.613 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:28.605 7fb4d7fff700 1 --2- 172.21.2.102:0/455055599 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb4d80a3dd0 0x7fb4d80a4e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:28.613 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:28.605 7fb4d7fff700 1 -- 172.21.2.102:0/455055599 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4d80a54a0 con 0x7fb4d80a2fb0 2021-02-10T11:26:28.613 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:28.605 7fb4d7fff700 1 --2- 172.21.2.102:0/455055599 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb4d80a2fb0 0x7fb4d80a33b0 secure :-1 s=READY pgs=1604 cs=0 l=1 rev1=1 rx=0x7fb4c8009630 tx=0x7fb4c800b900).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:28.613 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:28.605 7fb4d4ff9700 1 -- 172.21.2.102:0/455055599 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb4c80093a0 con 0x7fb4d80a2fb0 2021-02-10T11:26:28.614 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:28.605 7fb4d4ff9700 1 -- 172.21.2.102:0/455055599 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4c800bcf0 con 0x7fb4d80a2fb0 2021-02-10T11:26:28.614 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:28.605 7fb4de1ec700 1 -- 172.21.2.102:0/455055599 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4d80a54a0 con 0x7fb4d80a2fb0 2021-02-10T11:26:28.614 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:28.605 7fb4d4ff9700 1 -- 172.21.2.102:0/455055599 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb4c8005560 con 0x7fb4d80a2fb0 2021-02-10T11:26:28.614 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:28.605 7fb4de1ec700 1 -- 172.21.2.102:0/455055599 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4d80a5a30 con 0x7fb4d80a2fb0 2021-02-10T11:26:28.614 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:28.606 7fb4d4ff9700 1 -- 172.21.2.102:0/455055599 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fb4c800a000 con 0x7fb4d80a2fb0 2021-02-10T11:26:28.615 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:28.606 7fb4d4ff9700 1 --2- 172.21.2.102:0/455055599 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb4b80186c0 0x7fb4b801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:28.615 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:28.606 7fb4d4ff9700 1 -- 172.21.2.102:0/455055599 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fb4c80323b0 con 0x7fb4d80a2fb0 2021-02-10T11:26:28.615 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:28.606 7fb4d77fe700 1 --2- 172.21.2.102:0/455055599 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb4b80186c0 0x7fb4b801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:28.615 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:28.606 7fb4d77fe700 1 --2- 172.21.2.102:0/455055599 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb4b80186c0 0x7fb4b801abc0 crc :-1 s=READY pgs=727 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:28.637 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:28.637 7fb4de1ec700 1 -- 172.21.2.102:0/455055599 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4d80a4330 con 0x7fb4d80a2fb0 2021-02-10T11:26:28.639 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:28.640 7fb4d4ff9700 1 -- 172.21.2.102:0/455055599 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fb4c8009340 con 0x7fb4d80a2fb0 2021-02-10T11:26:28.789 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:28.789 7fb4de1ec700 1 -- 172.21.2.102:0/455055599 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "config rm", "who": "client.alice", "name": "debug_client"} v 0) v1 -- 0x7fb4d80a4330 con 0x7fb4d80a2fb0 2021-02-10T11:26:28.793 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:28.793 7fb4d4ff9700 1 -- 172.21.2.102:0/455055599 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "client.alice", "name": "debug_client"}]=0 v18) v1 ==== 104+0+0 (secure 0 0 0) 0x7fb4c800a000 con 0x7fb4d80a2fb0 2021-02-10T11:26:28.798 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:28.798 7fb4de1ec700 1 -- 172.21.2.102:0/455055599 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb4b80186c0 msgr2=0x7fb4b801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:28.798 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:28.798 7fb4de1ec700 1 --2- 172.21.2.102:0/455055599 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb4b80186c0 0x7fb4b801abc0 crc :-1 s=READY pgs=727 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:28.798 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:28.798 7fb4de1ec700 1 -- 172.21.2.102:0/455055599 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb4d80a2fb0 msgr2=0x7fb4d80a33b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:28.798 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:28.798 7fb4de1ec700 1 --2- 172.21.2.102:0/455055599 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb4d80a2fb0 0x7fb4d80a33b0 secure :-1 s=READY pgs=1604 cs=0 l=1 rev1=1 rx=0x7fb4c8009630 tx=0x7fb4c800b900).stop 2021-02-10T11:26:28.798 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:28.798 7fb4de1ec700 1 -- 172.21.2.102:0/455055599 shutdown_connections 2021-02-10T11:26:28.798 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:28.798 7fb4de1ec700 1 --2- 172.21.2.102:0/455055599 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb4b80186c0 0x7fb4b801abc0 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:28.798 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:28.798 7fb4de1ec700 1 --2- 172.21.2.102:0/455055599 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb4d80a3dd0 0x7fb4d80a4e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:28.798 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:28.798 7fb4de1ec700 1 --2- 172.21.2.102:0/455055599 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb4d80a39d0 0x7fb4d80a4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:28.798 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:28.798 7fb4de1ec700 1 --2- 172.21.2.102:0/455055599 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb4d80a2fb0 0x7fb4d80a33b0 unknown :-1 s=CLOSED pgs=1604 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:28.798 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:28.798 7fb4de1ec700 1 -- 172.21.2.102:0/455055599 >> 172.21.2.102:0/455055599 conn(0x7fb4d808f030 msgr2=0x7fb4d80b5e80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:26:28.799 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:28.799 7fb4de1ec700 1 -- 172.21.2.102:0/455055599 shutdown_connections 2021-02-10T11:26:28.799 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:28.799 7fb4de1ec700 1 -- 172.21.2.102:0/455055599 wait complete. 2021-02-10T11:26:28.807 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config rm client.alice debug_objecter 2021-02-10T11:26:29.236 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:29.220 7fd83c032700 1 Processor -- start 2021-02-10T11:26:29.236 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:29.220 7fd83c032700 1 -- start start 2021-02-10T11:26:29.236 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:29.221 7fd83c032700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fd8340d1870 0x7fd8341b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:29.236 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:29.221 7fd83c032700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd8341b0a10 0x7fd8340cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:29.236 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:29.221 7fd83c032700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fd8340d1c70 0x7fd8340cf950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:29.236 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:29.221 7fd83c032700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8340fe200 con 0x7fd8340d1c70 2021-02-10T11:26:29.236 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:29.221 7fd83c032700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8340e5f10 con 0x7fd8340d1870 2021-02-10T11:26:29.236 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:29.221 7fd83c032700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8340f0b50 con 0x7fd8341b0a10 2021-02-10T11:26:29.236 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:29.221 7fd83a64f700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fd8340d1c70 0x7fd8340cf950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:29.236 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:29.221 7fd83a64f700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fd8340d1c70 0x7fd8340cf950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:58702/0 (socket says 172.21.2.102:58702) 2021-02-10T11:26:29.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:29.221 7fd83a64f700 1 -- 172.21.2.102:0/3549900514 learned_addr learned my addr 172.21.2.102:0/3549900514 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:26:29.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:29.221 7fd83a64f700 1 -- 172.21.2.102:0/3549900514 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd8341b0a10 msgr2=0x7fd8340cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:26:29.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:29.221 7fd83a64f700 1 --2- 172.21.2.102:0/3549900514 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd8341b0a10 0x7fd8340cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:29.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:29.221 7fd83a64f700 1 -- 172.21.2.102:0/3549900514 >> v2:172.21.2.113:3300/0 conn(0x7fd8340d1870 msgr2=0x7fd8341b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:26:29.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:29.221 7fd83a64f700 1 --2- 172.21.2.102:0/3549900514 >> v2:172.21.2.113:3300/0 conn(0x7fd8340d1870 0x7fd8341b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:29.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:29.221 7fd83a64f700 1 -- 172.21.2.102:0/3549900514 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8340cff70 con 0x7fd8340d1c70 2021-02-10T11:26:29.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:29.221 7fd83a64f700 1 --2- 172.21.2.102:0/3549900514 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd8340d1c70 0x7fd8340cf950 secure :-1 s=READY pgs=1605 cs=0 l=1 rev1=1 rx=0x7fd82c006f00 tx=0x7fd82c009f10).ready entity=mon.0 client_cookie=26768bfdc7e7da87 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:29.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:29.221 7fd838e4c700 1 -- 172.21.2.102:0/3549900514 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd82c00b3b0 con 0x7fd8340d1c70 2021-02-10T11:26:29.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:29.221 7fd838e4c700 1 -- 172.21.2.102:0/3549900514 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd82c00b630 con 0x7fd8340d1c70 2021-02-10T11:26:29.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:29.221 7fd838e4c700 1 -- 172.21.2.102:0/3549900514 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd82c00bb50 con 0x7fd8340d1c70 2021-02-10T11:26:29.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:29.221 7fd83c032700 1 -- 172.21.2.102:0/3549900514 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd8340d1c70 msgr2=0x7fd8340cf950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:29.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:29.221 7fd83c032700 1 --2- 172.21.2.102:0/3549900514 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd8340d1c70 0x7fd8340cf950 secure :-1 s=READY pgs=1605 cs=0 l=1 rev1=1 rx=0x7fd82c006f00 tx=0x7fd82c009f10).stop 2021-02-10T11:26:29.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:29.222 7fd83c032700 1 -- 172.21.2.102:0/3549900514 shutdown_connections 2021-02-10T11:26:29.238 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:29.222 7fd83c032700 1 --2- 172.21.2.102:0/3549900514 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd8341b0a10 0x7fd8340cf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:29.238 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:29.222 7fd83c032700 1 --2- 172.21.2.102:0/3549900514 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd8340d1c70 0x7fd8340cf950 unknown :-1 s=CLOSED pgs=1605 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:29.238 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:29.222 7fd83c032700 1 --2- 172.21.2.102:0/3549900514 >> v2:172.21.2.113:3300/0 conn(0x7fd8340d1870 0x7fd8341b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:29.238 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:29.222 7fd83c032700 1 -- 172.21.2.102:0/3549900514 >> 172.21.2.102:0/3549900514 conn(0x7fd8340ce450 msgr2=0x7fd8340d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:26:29.238 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:29.222 7fd83c032700 1 -- 172.21.2.102:0/3549900514 shutdown_connections 2021-02-10T11:26:29.238 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:29.222 7fd83c032700 1 -- 172.21.2.102:0/3549900514 wait complete. 2021-02-10T11:26:29.239 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:29.236 7fd83c032700 1 Processor -- start 2021-02-10T11:26:29.240 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:29.237 7fd83c032700 1 -- start start 2021-02-10T11:26:29.240 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:29.237 7fd83c032700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd8340bad20 0x7fd8340c2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:29.240 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:29.237 7fd83c032700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd8340bb120 0x7fd8340b9b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:29.240 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:29.237 7fd83c032700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd8340ba1a0 0x7fd8340b7950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:29.240 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:29.237 7fd83c032700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8340ba7a0 con 0x7fd8340bb120 2021-02-10T11:26:29.240 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:29.237 7fd83c032700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8340917c0 con 0x7fd8340bad20 2021-02-10T11:26:29.240 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:29.237 7fd83c032700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8340c26a0 con 0x7fd8340ba1a0 2021-02-10T11:26:29.240 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:29.237 7fd83964d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd8340bb120 0x7fd8340b9b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:29.240 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:29.237 7fd83964d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd8340bb120 0x7fd8340b9b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:58704/0 (socket says 172.21.2.102:58704) 2021-02-10T11:26:29.240 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:29.237 7fd83964d700 1 -- 172.21.2.102:0/1232698984 learned_addr learned my addr 172.21.2.102:0/1232698984 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:26:29.240 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:29.237 7fd83964d700 1 -- 172.21.2.102:0/1232698984 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd8340ba1a0 msgr2=0x7fd8340b7950 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:26:29.240 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:29.237 7fd83964d700 1 --2- 172.21.2.102:0/1232698984 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd8340ba1a0 0x7fd8340b7950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:29.241 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:29.237 7fd83964d700 1 -- 172.21.2.102:0/1232698984 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd8340bad20 msgr2=0x7fd8340c2080 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:26:29.241 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:29.237 7fd83964d700 1 --2- 172.21.2.102:0/1232698984 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd8340bad20 0x7fd8340c2080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:29.241 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:29.237 7fd83964d700 1 -- 172.21.2.102:0/1232698984 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8340b7f70 con 0x7fd8340bb120 2021-02-10T11:26:29.241 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:29.238 7fd83964d700 1 --2- 172.21.2.102:0/1232698984 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd8340bb120 0x7fd8340b9b80 secure :-1 s=READY pgs=1606 cs=0 l=1 rev1=1 rx=0x7fd830006f00 tx=0x7fd83000b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:29.241 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:29.238 7fd8267fc700 1 -- 172.21.2.102:0/1232698984 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd83000bd30 con 0x7fd8340bb120 2021-02-10T11:26:29.241 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:29.238 7fd8267fc700 1 -- 172.21.2.102:0/1232698984 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd830000c90 con 0x7fd8340bb120 2021-02-10T11:26:29.241 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:29.238 7fd83c032700 1 -- 172.21.2.102:0/1232698984 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8340b7f70 con 0x7fd8340bb120 2021-02-10T11:26:29.241 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:29.238 7fd8267fc700 1 -- 172.21.2.102:0/1232698984 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd8300022a0 con 0x7fd8340bb120 2021-02-10T11:26:29.241 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:29.238 7fd83c032700 1 -- 172.21.2.102:0/1232698984 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8340b7f70 con 0x7fd8340bb120 2021-02-10T11:26:29.241 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:29.238 7fd8267fc700 1 -- 172.21.2.102:0/1232698984 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd830005310 con 0x7fd8340bb120 2021-02-10T11:26:29.241 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:29.238 7fd8267fc700 1 --2- 172.21.2.102:0/1232698984 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd81c018730 0x7fd81c01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:29.241 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:29.238 7fd8267fc700 1 -- 172.21.2.102:0/1232698984 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fd8300325f0 con 0x7fd8340bb120 2021-02-10T11:26:29.241 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:29.239 7fd839e4e700 1 --2- 172.21.2.102:0/1232698984 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd81c018730 0x7fd81c01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:29.241 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:29.239 7fd839e4e700 1 --2- 172.21.2.102:0/1232698984 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd81c018730 0x7fd81c01ac10 crc :-1 s=READY pgs=728 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:29.270 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:29.270 7fd83c032700 1 -- 172.21.2.102:0/1232698984 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8340ee060 con 0x7fd8340bb120 2021-02-10T11:26:29.272 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:29.273 7fd8267fc700 1 -- 172.21.2.102:0/1232698984 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd830006b90 con 0x7fd8340bb120 2021-02-10T11:26:29.421 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:29.421 7fd83c032700 1 -- 172.21.2.102:0/1232698984 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "config rm", "who": "client.alice", "name": "debug_objecter"} v 0) v1 -- 0x7fd8340ce450 con 0x7fd8340bb120 2021-02-10T11:26:29.425 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:29.425 7fd8267fc700 1 -- 172.21.2.102:0/1232698984 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "client.alice", "name": "debug_objecter"}]=0 v19) v1 ==== 106+0+0 (secure 0 0 0) 0x7fd830005310 con 0x7fd8340bb120 2021-02-10T11:26:29.429 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:29.430 7fd83c032700 1 -- 172.21.2.102:0/1232698984 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd81c018730 msgr2=0x7fd81c01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:29.430 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:29.430 7fd83c032700 1 --2- 172.21.2.102:0/1232698984 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd81c018730 0x7fd81c01ac10 crc :-1 s=READY pgs=728 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:29.430 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:29.430 7fd83c032700 1 -- 172.21.2.102:0/1232698984 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd8340bb120 msgr2=0x7fd8340b9b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:29.430 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:29.430 7fd83c032700 1 --2- 172.21.2.102:0/1232698984 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd8340bb120 0x7fd8340b9b80 secure :-1 s=READY pgs=1606 cs=0 l=1 rev1=1 rx=0x7fd830006f00 tx=0x7fd83000b940).stop 2021-02-10T11:26:29.430 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:29.430 7fd83c032700 1 -- 172.21.2.102:0/1232698984 shutdown_connections 2021-02-10T11:26:29.430 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:29.430 7fd83c032700 1 --2- 172.21.2.102:0/1232698984 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd8340ba1a0 0x7fd8340b7950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:29.430 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:29.430 7fd83c032700 1 --2- 172.21.2.102:0/1232698984 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd8340bb120 0x7fd8340b9b80 unknown :-1 s=CLOSED pgs=1606 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:29.430 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:29.430 7fd83c032700 1 --2- 172.21.2.102:0/1232698984 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd81c018730 0x7fd81c01ac10 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:29.430 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:29.430 7fd83c032700 1 --2- 172.21.2.102:0/1232698984 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd8340bad20 0x7fd8340c2080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:29.430 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:29.430 7fd83c032700 1 -- 172.21.2.102:0/1232698984 >> 172.21.2.102:0/1232698984 conn(0x7fd8340bc0b0 msgr2=0x7fd8340c29c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:26:29.430 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:29.430 7fd83c032700 1 -- 172.21.2.102:0/1232698984 shutdown_connections 2021-02-10T11:26:29.431 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:29.430 7fd83c032700 1 -- 172.21.2.102:0/1232698984 wait complete. 2021-02-10T11:26:29.438 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestVolumes.test_authorize_deauthorize_subvolume' 2021-02-10T11:26:29.866 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:29.850 7f6723f13700 1 Processor -- start 2021-02-10T11:26:29.866 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:29.850 7f6723f13700 1 -- start start 2021-02-10T11:26:29.867 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:29.851 7f6723f13700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f671c0d1c70 0x7f671c0f20a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:29.867 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:29.851 7f6723f13700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f671c0f0b10 con 0x7f671c0d1870 2021-02-10T11:26:29.867 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:29.851 7f6723f13700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f671c1b0be0 con 0x7f671c0d0340 2021-02-10T11:26:29.867 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:29.851 7f6723f13700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f671c0e5ed0 con 0x7f671c0d1c70 2021-02-10T11:26:29.867 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:29.851 7f6722530700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f671c0d1c70 0x7f671c0f20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:29.867 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:29.851 7f6722530700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f671c0d1c70 0x7f671c0f20a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:58716/0 (socket says 172.21.2.102:58716) 2021-02-10T11:26:29.867 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:29.851 7f6722530700 1 -- 172.21.2.102:0/1603500730 learned_addr learned my addr 172.21.2.102:0/1603500730 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:26:29.867 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:29.851 7f6722530700 1 -- 172.21.2.102:0/1603500730 >> v1:172.21.2.113:6789/0 conn(0x7f671c0d0340 legacy=0x7f671c0ef640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:26:29.867 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:29.851 7f6722530700 1 -- 172.21.2.102:0/1603500730 >> v1:172.21.2.102:6789/0 conn(0x7f671c0d1870 legacy=0x7f671c1b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:29.867 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:29.851 7f6722530700 1 -- 172.21.2.102:0/1603500730 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f671c0efe30 con 0x7f671c0d1c70 2021-02-10T11:26:29.867 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:29.851 7f6722530700 1 --2- 172.21.2.102:0/1603500730 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f671c0d1c70 0x7f671c0f20a0 secure :-1 s=READY pgs=1608 cs=0 l=1 rev1=1 rx=0x7f6714006f00 tx=0x7f6714009f10).ready entity=mon.0 client_cookie=292f15cbffc45e83 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:29.867 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:29.852 7f6720d2d700 1 -- 172.21.2.102:0/1603500730 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f671400b3b0 con 0x7f671c0d1c70 2021-02-10T11:26:29.868 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:29.852 7f6720d2d700 1 -- 172.21.2.102:0/1603500730 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f671400b3b0 con 0x7f671c0d1c70 2021-02-10T11:26:29.868 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:29.852 7f6720d2d700 1 -- 172.21.2.102:0/1603500730 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f671400b730 con 0x7f671c0d1c70 2021-02-10T11:26:29.868 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:29.852 7f6723f13700 1 -- 172.21.2.102:0/1603500730 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f671c0d1c70 msgr2=0x7f671c0f20a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:29.868 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:29.852 7f6723f13700 1 --2- 172.21.2.102:0/1603500730 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f671c0d1c70 0x7f671c0f20a0 secure :-1 s=READY pgs=1608 cs=0 l=1 rev1=1 rx=0x7f6714006f00 tx=0x7f6714009f10).stop 2021-02-10T11:26:29.868 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:29.852 7f6723f13700 1 -- 172.21.2.102:0/1603500730 shutdown_connections 2021-02-10T11:26:29.868 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:29.852 7f6723f13700 1 --2- 172.21.2.102:0/1603500730 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f671c0d1c70 0x7f671c0f20a0 unknown :-1 s=CLOSED pgs=1608 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:29.868 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:29.852 7f6723f13700 1 -- 172.21.2.102:0/1603500730 >> 172.21.2.102:0/1603500730 conn(0x7f671c0ce450 msgr2=0x7f671c0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:26:29.868 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:29.853 7f6723f13700 1 -- 172.21.2.102:0/1603500730 shutdown_connections 2021-02-10T11:26:29.868 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:29.853 7f6723f13700 1 -- 172.21.2.102:0/1603500730 wait complete. 2021-02-10T11:26:29.870 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:29.867 7f6723f13700 1 Processor -- start 2021-02-10T11:26:29.870 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:29.867 7f6723f13700 1 -- start start 2021-02-10T11:26:29.870 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:29.868 7f6723f13700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f671c092010 0x7f671c0c33c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:29.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:29.868 7f6723f13700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f671c0c39e0 0x7f671c0af040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:29.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:29.868 7f6723f13700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f671c0af660 0x7f671c0afa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:29.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:29.868 7f6723f13700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f671c0b0500 con 0x7f671c0c39e0 2021-02-10T11:26:29.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:29.868 7f6723f13700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f671c0b0080 con 0x7f671c092010 2021-02-10T11:26:29.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:29.868 7f6723f13700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f671c0b02a0 con 0x7f671c0af660 2021-02-10T11:26:29.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:29.868 7f672152e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f671c0c39e0 0x7f671c0af040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:29.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:29.868 7f672152e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f671c0c39e0 0x7f671c0af040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:58722/0 (socket says 172.21.2.102:58722) 2021-02-10T11:26:29.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:29.868 7f672152e700 1 -- 172.21.2.102:0/96555555 learned_addr learned my addr 172.21.2.102:0/96555555 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:26:29.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:29.868 7f6721d2f700 1 --2- 172.21.2.102:0/96555555 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f671c092010 0x7f671c0c33c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:29.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:29.868 7f672152e700 1 -- 172.21.2.102:0/96555555 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f671c0af660 msgr2=0x7f671c0afa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:29.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:29.868 7f6722530700 1 --2- 172.21.2.102:0/96555555 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f671c0af660 0x7f671c0afa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:29.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:29.869 7f672152e700 1 --2- 172.21.2.102:0/96555555 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f671c0af660 0x7f671c0afa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:29.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:29.869 7f672152e700 1 -- 172.21.2.102:0/96555555 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f671c092010 msgr2=0x7f671c0c33c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:29.872 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:29.869 7f672152e700 1 --2- 172.21.2.102:0/96555555 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f671c092010 0x7f671c0c33c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:29.872 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:29.869 7f672152e700 1 -- 172.21.2.102:0/96555555 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f671c0b0a70 con 0x7f671c0c39e0 2021-02-10T11:26:29.872 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:29.869 7f6721d2f700 1 --2- 172.21.2.102:0/96555555 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f671c092010 0x7f671c0c33c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:26:29.872 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:29.869 7f6722530700 1 --2- 172.21.2.102:0/96555555 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f671c0af660 0x7f671c0afa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:26:29.872 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:29.869 7f672152e700 1 --2- 172.21.2.102:0/96555555 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f671c0c39e0 0x7f671c0af040 secure :-1 s=READY pgs=1609 cs=0 l=1 rev1=1 rx=0x7f670c00b5b0 tx=0x7f670c00b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:29.872 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:29.869 7f67127fc700 1 -- 172.21.2.102:0/96555555 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f670c00bd40 con 0x7f671c0c39e0 2021-02-10T11:26:29.872 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:29.869 7f67127fc700 1 -- 172.21.2.102:0/96555555 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f670c006240 con 0x7f671c0c39e0 2021-02-10T11:26:29.872 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:29.869 7f6723f13700 1 -- 172.21.2.102:0/96555555 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f671c0b0a70 con 0x7f671c0c39e0 2021-02-10T11:26:29.872 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:29.869 7f67127fc700 1 -- 172.21.2.102:0/96555555 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f670c005720 con 0x7f671c0c39e0 2021-02-10T11:26:29.872 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:29.869 7f6723f13700 1 -- 172.21.2.102:0/96555555 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f671c0b0a70 con 0x7f671c0c39e0 2021-02-10T11:26:29.872 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:29.870 7f67127fc700 1 -- 172.21.2.102:0/96555555 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f670c0054a0 con 0x7f671c0c39e0 2021-02-10T11:26:29.872 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:29.870 7f67127fc700 1 --2- 172.21.2.102:0/96555555 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6704018730 0x7f670401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:29.872 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:29.870 7f67127fc700 1 -- 172.21.2.102:0/96555555 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f670c032670 con 0x7f671c0c39e0 2021-02-10T11:26:29.873 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:29.870 7f6721d2f700 1 --2- 172.21.2.102:0/96555555 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6704018730 0x7f670401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:29.873 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:29.870 7f6721d2f700 1 --2- 172.21.2.102:0/96555555 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6704018730 0x7f670401ac10 crc :-1 s=READY pgs=729 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:29.901 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:29.900 7f6723f13700 1 -- 172.21.2.102:0/96555555 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f671c0e7df0 con 0x7f671c0c39e0 2021-02-10T11:26:29.903 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:29.903 7f67127fc700 1 -- 172.21.2.102:0/96555555 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f670c006030 con 0x7f671c0c39e0 2021-02-10T11:26:30.037 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:30.037 7f6723f13700 1 -- 172.21.2.102:0/96555555 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestVolumes.test_authorize_deauthorize_subvolume"]} v 0) v1 -- 0x7f671c0e7df0 con 0x7f671c0c39e0 2021-02-10T11:26:31.033 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:31.033 7f67127fc700 1 -- 172.21.2.102:0/96555555 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestVolumes.test_authorize_deauthorize_subvolume"]}]=0 v389) v1 ==== 153+0+0 (secure 0 0 0) 0x7f670c0054a0 con 0x7f671c0c39e0 2021-02-10T11:26:31.059 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:31.059 7f6723f13700 1 -- 172.21.2.102:0/96555555 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6704018730 msgr2=0x7f670401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:31.059 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:31.059 7f6723f13700 1 --2- 172.21.2.102:0/96555555 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6704018730 0x7f670401ac10 crc :-1 s=READY pgs=729 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:31.060 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:31.060 7f6723f13700 1 -- 172.21.2.102:0/96555555 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f671c0c39e0 msgr2=0x7f671c0af040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:31.060 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:31.060 7f6723f13700 1 --2- 172.21.2.102:0/96555555 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f671c0c39e0 0x7f671c0af040 secure :-1 s=READY pgs=1609 cs=0 l=1 rev1=1 rx=0x7f670c00b5b0 tx=0x7f670c00b950).stop 2021-02-10T11:26:31.060 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:31.060 7f6723f13700 1 -- 172.21.2.102:0/96555555 shutdown_connections 2021-02-10T11:26:31.060 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:31.060 7f6723f13700 1 --2- 172.21.2.102:0/96555555 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f671c0af660 0x7f671c0afa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:31.060 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:31.060 7f6723f13700 1 --2- 172.21.2.102:0/96555555 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f671c0c39e0 0x7f671c0af040 unknown :-1 s=CLOSED pgs=1609 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:31.060 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:31.060 7f6723f13700 1 --2- 172.21.2.102:0/96555555 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6704018730 0x7f670401ac10 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:31.060 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:31.060 7f6723f13700 1 --2- 172.21.2.102:0/96555555 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f671c092010 0x7f671c0c33c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:31.060 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:31.060 7f6723f13700 1 -- 172.21.2.102:0/96555555 >> 172.21.2.102:0/96555555 conn(0x7f671c0a0030 msgr2=0x7f671c0f22b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:26:31.061 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:31.060 7f6723f13700 1 -- 172.21.2.102:0/96555555 shutdown_connections 2021-02-10T11:26:31.061 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:31.060 7f6723f13700 1 -- 172.21.2.102:0/96555555 wait complete. 2021-02-10T11:26:31.068 INFO:tasks.cephfs_test_runner:test_authorize_deauthorize_subvolume (tasks.cephfs.test_volumes.TestVolumes) ... ok 2021-02-10T11:26:31.068 INFO:tasks.cephfs_test_runner:Starting test: test_clone_subvolume_info (tasks.cephfs.test_volumes.TestVolumes) 2021-02-10T11:26:31.068 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestVolumes.test_clone_subvolume_info' 2021-02-10T11:26:31.495 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:31.479 7f1db2bf0700 1 Processor -- start 2021-02-10T11:26:31.496 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:31.479 7f1db2bf0700 1 -- start start 2021-02-10T11:26:31.496 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:31.479 7f1db2bf0700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f1dac0d1c70 0x7f1dac0f20a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:31.496 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:31.479 7f1db2bf0700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1dac0f0b10 con 0x7f1dac0d0340 2021-02-10T11:26:31.496 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:31.479 7f1db2bf0700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1dac1b0be0 con 0x7f1dac0d1870 2021-02-10T11:26:31.496 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:31.479 7f1db2bf0700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1dac0e5ed0 con 0x7f1dac0d1c70 2021-02-10T11:26:31.496 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:31.479 7f1dabfff700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f1dac0d0340 0x7f1dac0ef640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:50950/0 (socket says 172.21.2.102:50950) 2021-02-10T11:26:31.496 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:31.480 7f1dabfff700 1 -- 172.21.2.102:0/643744084 learned_addr learned my addr 172.21.2.102:0/643744084 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:26:31.496 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:31.480 7f1dab7fe700 1 -- 172.21.2.102:0/643744084 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1474283267 0 0) 0x7f1d9c000ce0 con 0x7f1dac0d0340 2021-02-10T11:26:31.496 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:31.480 7f1dab7fe700 1 -- 172.21.2.102:0/643744084 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d98002fe0 con 0x7f1dac0d0340 2021-02-10T11:26:31.496 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:31.480 7f1db120d700 1 --2- 172.21.2.102:0/643744084 >> v2:172.21.2.113:3300/0 conn(0x7f1dac0d1c70 0x7f1dac0f20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:31.496 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:31.480 7f1dab7fe700 1 -- 172.21.2.102:0/643744084 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1208443301 0 0) 0x7f1d9c0010e0 con 0x7f1dac0d0340 2021-02-10T11:26:31.497 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:31.480 7f1db120d700 1 -- 172.21.2.102:0/643744084 >> v1:172.21.2.113:6789/0 conn(0x7f1dac0d1870 legacy=0x7f1dac1b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:31.497 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:31.480 7f1db120d700 1 -- 172.21.2.102:0/643744084 >> v1:172.21.2.102:6789/0 conn(0x7f1dac0d0340 legacy=0x7f1dac0ef640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:31.497 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:31.480 7f1db120d700 1 -- 172.21.2.102:0/643744084 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dac0efe30 con 0x7f1dac0d1c70 2021-02-10T11:26:31.497 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:31.480 7f1db120d700 1 --2- 172.21.2.102:0/643744084 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1dac0d1c70 0x7f1dac0f20a0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 rx=0x7f1da4006e50 tx=0x7f1da4009e70).ready entity=mon.1 client_cookie=c901ba3efcd171e2 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:31.497 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:31.481 7f1dab7fe700 1 -- 172.21.2.102:0/643744084 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1da400b510 con 0x7f1dac0d1c70 2021-02-10T11:26:31.497 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:31.481 7f1dab7fe700 1 -- 172.21.2.102:0/643744084 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1da4006790 con 0x7f1dac0d1c70 2021-02-10T11:26:31.497 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:31.481 7f1dab7fe700 1 -- 172.21.2.102:0/643744084 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1da400ba00 con 0x7f1dac0d1c70 2021-02-10T11:26:31.497 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:31.481 7f1db2bf0700 1 -- 172.21.2.102:0/643744084 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1dac0d1c70 msgr2=0x7f1dac0f20a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:31.497 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:31.481 7f1db2bf0700 1 --2- 172.21.2.102:0/643744084 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1dac0d1c70 0x7f1dac0f20a0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 rx=0x7f1da4006e50 tx=0x7f1da4009e70).stop 2021-02-10T11:26:31.497 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:31.481 7f1db2bf0700 1 -- 172.21.2.102:0/643744084 shutdown_connections 2021-02-10T11:26:31.497 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:31.481 7f1db2bf0700 1 --2- 172.21.2.102:0/643744084 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1dac0d1c70 0x7f1dac0f20a0 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:31.497 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:31.481 7f1db2bf0700 1 -- 172.21.2.102:0/643744084 >> 172.21.2.102:0/643744084 conn(0x7f1dac0ce450 msgr2=0x7f1dac0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:26:31.497 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:31.481 7f1db2bf0700 1 -- 172.21.2.102:0/643744084 shutdown_connections 2021-02-10T11:26:31.498 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:31.481 7f1db2bf0700 1 -- 172.21.2.102:0/643744084 wait complete. 2021-02-10T11:26:31.498 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:31.496 7f1db2bf0700 1 Processor -- start 2021-02-10T11:26:31.498 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:31.496 7f1db2bf0700 1 -- start start 2021-02-10T11:26:31.499 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:31.496 7f1db2bf0700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1dac18b7f0 0x7f1dac19b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:31.499 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:31.496 7f1db2bf0700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1dac08de10 0x7f1dac08e210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:31.500 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:31.496 7f1db2bf0700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1dac189f50 0x7f1dac18a350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:31.500 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:31.496 7f1db2bf0700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dac18bbf0 con 0x7f1dac18b7f0 2021-02-10T11:26:31.500 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:31.496 7f1db2bf0700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dac1abfc0 con 0x7f1dac08de10 2021-02-10T11:26:31.500 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:31.496 7f1db2bf0700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1dac08e830 con 0x7f1dac189f50 2021-02-10T11:26:31.500 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:31.497 7f1db0a0c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1dac18b7f0 0x7f1dac19b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:31.500 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:31.497 7f1db0a0c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1dac18b7f0 0x7f1dac19b170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:58734/0 (socket says 172.21.2.102:58734) 2021-02-10T11:26:31.500 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:31.497 7f1db0a0c700 1 -- 172.21.2.102:0/1053890259 learned_addr learned my addr 172.21.2.102:0/1053890259 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:26:31.500 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:31.497 7f1dabfff700 1 --2- 172.21.2.102:0/1053890259 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1dac08de10 0x7f1dac08e210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:31.500 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:31.497 7f1db120d700 1 --2- 172.21.2.102:0/1053890259 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1dac189f50 0x7f1dac18a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:31.500 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:31.497 7f1db0a0c700 1 -- 172.21.2.102:0/1053890259 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1dac189f50 msgr2=0x7f1dac18a350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:31.500 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:31.497 7f1db0a0c700 1 --2- 172.21.2.102:0/1053890259 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1dac189f50 0x7f1dac18a350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:31.500 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:31.497 7f1db0a0c700 1 -- 172.21.2.102:0/1053890259 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1dac08de10 msgr2=0x7f1dac08e210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:31.501 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:31.497 7f1db0a0c700 1 --2- 172.21.2.102:0/1053890259 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1dac08de10 0x7f1dac08e210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:31.501 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:31.497 7f1db0a0c700 1 -- 172.21.2.102:0/1053890259 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dac18a970 con 0x7f1dac18b7f0 2021-02-10T11:26:31.501 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:31.497 7f1dabfff700 1 --2- 172.21.2.102:0/1053890259 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1dac08de10 0x7f1dac08e210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:26:31.501 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:31.497 7f1db0a0c700 1 --2- 172.21.2.102:0/1053890259 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1dac18b7f0 0x7f1dac19b170 secure :-1 s=READY pgs=1611 cs=0 l=1 rev1=1 rx=0x7f1d94006ed0 tx=0x7f1d9400b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:31.501 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:31.498 7f1da97fa700 1 -- 172.21.2.102:0/1053890259 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1d94006020 con 0x7f1dac18b7f0 2021-02-10T11:26:31.501 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:31.498 7f1da97fa700 1 -- 172.21.2.102:0/1053890259 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d94006360 con 0x7f1dac18b7f0 2021-02-10T11:26:31.501 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:31.498 7f1da97fa700 1 -- 172.21.2.102:0/1053890259 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1d94005730 con 0x7f1dac18b7f0 2021-02-10T11:26:31.501 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:31.498 7f1db2bf0700 1 -- 172.21.2.102:0/1053890259 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1dac18a970 con 0x7f1dac18b7f0 2021-02-10T11:26:31.501 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:31.498 7f1db2bf0700 1 -- 172.21.2.102:0/1053890259 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1dac18af00 con 0x7f1dac18b7f0 2021-02-10T11:26:31.501 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:31.498 7f1da97fa700 1 -- 172.21.2.102:0/1053890259 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1d94006020 con 0x7f1dac18b7f0 2021-02-10T11:26:31.501 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:31.499 7f1da97fa700 1 --2- 172.21.2.102:0/1053890259 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1d900186c0 0x7f1d9001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:31.501 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:31.499 7f1da97fa700 1 -- 172.21.2.102:0/1053890259 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f1d940325e0 con 0x7f1dac18b7f0 2021-02-10T11:26:31.501 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:31.499 7f1dabfff700 1 --2- 172.21.2.102:0/1053890259 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1d900186c0 0x7f1d9001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:31.502 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:31.499 7f1dabfff700 1 --2- 172.21.2.102:0/1053890259 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1d900186c0 0x7f1d9001abc0 crc :-1 s=READY pgs=730 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:31.529 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:31.529 7f1db2bf0700 1 -- 172.21.2.102:0/1053890259 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1dac18b4d0 con 0x7f1dac18b7f0 2021-02-10T11:26:31.531 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:31.532 7f1da97fa700 1 -- 172.21.2.102:0/1053890259 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f1d94030d90 con 0x7f1dac18b7f0 2021-02-10T11:26:31.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:31.665 7f1db2bf0700 1 -- 172.21.2.102:0/1053890259 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestVolumes.test_clone_subvolume_info"]} v 0) v1 -- 0x7f1dac18b4d0 con 0x7f1dac18b7f0 2021-02-10T11:26:32.038 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:32.038 7f1da97fa700 1 -- 172.21.2.102:0/1053890259 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestVolumes.test_clone_subvolume_info"]}]=0 v390) v1 ==== 145+0+0 (secure 0 0 0) 0x7f1d94005fe0 con 0x7f1dac18b7f0 2021-02-10T11:26:32.083 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:32.084 7f1db2bf0700 1 -- 172.21.2.102:0/1053890259 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1d900186c0 msgr2=0x7f1d9001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:32.083 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:32.084 7f1db2bf0700 1 --2- 172.21.2.102:0/1053890259 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1d900186c0 0x7f1d9001abc0 crc :-1 s=READY pgs=730 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:32.084 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:32.084 7f1db2bf0700 1 -- 172.21.2.102:0/1053890259 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1dac18b7f0 msgr2=0x7f1dac19b170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:32.084 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:32.084 7f1db2bf0700 1 --2- 172.21.2.102:0/1053890259 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1dac18b7f0 0x7f1dac19b170 secure :-1 s=READY pgs=1611 cs=0 l=1 rev1=1 rx=0x7f1d94006ed0 tx=0x7f1d9400b960).stop 2021-02-10T11:26:32.084 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:32.084 7f1db2bf0700 1 -- 172.21.2.102:0/1053890259 shutdown_connections 2021-02-10T11:26:32.084 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:32.084 7f1db2bf0700 1 --2- 172.21.2.102:0/1053890259 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1dac189f50 0x7f1dac18a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:32.084 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:32.084 7f1db2bf0700 1 --2- 172.21.2.102:0/1053890259 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1d900186c0 0x7f1d9001abc0 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:32.084 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:32.084 7f1db2bf0700 1 --2- 172.21.2.102:0/1053890259 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1dac08de10 0x7f1dac08e210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:32.085 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:32.084 7f1db2bf0700 1 --2- 172.21.2.102:0/1053890259 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1dac18b7f0 0x7f1dac19b170 unknown :-1 s=CLOSED pgs=1611 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:32.085 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:32.084 7f1db2bf0700 1 -- 172.21.2.102:0/1053890259 >> 172.21.2.102:0/1053890259 conn(0x7f1dac19c580 msgr2=0x7f1dac0916f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:26:32.085 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:32.084 7f1db2bf0700 1 -- 172.21.2.102:0/1053890259 shutdown_connections 2021-02-10T11:26:32.085 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:32.084 7f1db2bf0700 1 -- 172.21.2.102:0/1053890259 wait complete. 2021-02-10T11:26:32.092 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config set mon mon_allow_pool_delete True 2021-02-10T11:26:32.524 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:32.507 7f8808141700 1 Processor -- start 2021-02-10T11:26:32.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:32.507 7f8808141700 1 -- start start 2021-02-10T11:26:32.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:32.507 7f8808141700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f88000d1870 0x7f88001b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:32.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:32.507 7f8808141700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f88000cff70 0x7f8800109130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:32.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:32.507 7f8808141700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8800109a60 con 0x7f88001b0ae0 2021-02-10T11:26:32.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:32.507 7f8808141700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8800109750 con 0x7f88000d1870 2021-02-10T11:26:32.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:32.507 7f8808141700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8800088070 con 0x7f88000cff70 2021-02-10T11:26:32.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:32.509 7f8805f5d700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f88000d1870 0x7f88001b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:32.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:32.509 7f880575c700 1 --1- >> v1:172.21.2.113:6789/0 conn(0x7f88001b0ae0 0x7f8800108940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.113:6789/0 says I am v1:172.21.2.102:58288/0 (socket says 172.21.2.102:58288) 2021-02-10T11:26:32.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:32.509 7f880575c700 1 -- 172.21.2.102:0/3599277892 learned_addr learned my addr 172.21.2.102:0/3599277892 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:26:32.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:32.509 7f880675e700 1 --2- 172.21.2.102:0/3599277892 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f88000cff70 0x7f8800109130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:32.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:32.509 7f8804f5b700 1 -- 172.21.2.102:0/3599277892 <== mon.1 v1:172.21.2.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3607505398 0 0) 0x7f87f40010b0 con 0x7f88001b0ae0 2021-02-10T11:26:32.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:32.509 7f8804f5b700 1 -- 172.21.2.102:0/3599277892 --> v1:172.21.2.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87ec002fe0 con 0x7f88001b0ae0 2021-02-10T11:26:32.526 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:32.509 7f880675e700 1 -- 172.21.2.102:0/3599277892 >> v2:172.21.2.113:3300/0 conn(0x7f88000d1870 msgr2=0x7f88001b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:26:32.526 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:32.509 7f880675e700 1 --2- 172.21.2.102:0/3599277892 >> v2:172.21.2.113:3300/0 conn(0x7f88000d1870 0x7f88001b04c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:32.526 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:32.509 7f880675e700 1 -- 172.21.2.102:0/3599277892 >> v1:172.21.2.113:6789/0 conn(0x7f88001b0ae0 legacy=0x7f8800108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:32.526 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:32.509 7f880675e700 1 -- 172.21.2.102:0/3599277892 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88000d1c70 con 0x7f88000cff70 2021-02-10T11:26:32.526 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:32.509 7f8805f5d700 1 --2- 172.21.2.102:0/3599277892 >> v2:172.21.2.113:3300/0 conn(0x7f88000d1870 0x7f88001b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:26:32.526 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:32.509 7f880675e700 1 --2- 172.21.2.102:0/3599277892 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f88000cff70 0x7f8800109130 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f87f8006e50 tx=0x7f87f8009e70).ready entity=mon.2 client_cookie=d3ccd38b0978b0ec server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:32.526 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:32.510 7f8804f5b700 1 -- 172.21.2.102:0/3599277892 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f87f800b510 con 0x7f88000cff70 2021-02-10T11:26:32.526 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:32.510 7f8804f5b700 1 -- 172.21.2.102:0/3599277892 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87f8006790 con 0x7f88000cff70 2021-02-10T11:26:32.526 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:32.510 7f8804f5b700 1 -- 172.21.2.102:0/3599277892 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f87f800b940 con 0x7f88000cff70 2021-02-10T11:26:32.526 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:32.510 7f8808141700 1 -- 172.21.2.102:0/3599277892 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f88000cff70 msgr2=0x7f8800109130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:32.526 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:32.510 7f8808141700 1 --2- 172.21.2.102:0/3599277892 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f88000cff70 0x7f8800109130 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f87f8006e50 tx=0x7f87f8009e70).stop 2021-02-10T11:26:32.527 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:32.510 7f8808141700 1 -- 172.21.2.102:0/3599277892 shutdown_connections 2021-02-10T11:26:32.527 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:32.510 7f8808141700 1 --2- 172.21.2.102:0/3599277892 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f88000cff70 0x7f8800109130 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:32.527 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:32.510 7f8808141700 1 --2- 172.21.2.102:0/3599277892 >> v2:172.21.2.113:3300/0 conn(0x7f88000d1870 0x7f88001b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:32.527 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:32.510 7f8808141700 1 -- 172.21.2.102:0/3599277892 >> 172.21.2.102:0/3599277892 conn(0x7f88000ce420 msgr2=0x7f88000d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:26:32.527 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:32.511 7f8808141700 1 -- 172.21.2.102:0/3599277892 shutdown_connections 2021-02-10T11:26:32.527 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:32.511 7f8808141700 1 -- 172.21.2.102:0/3599277892 wait complete. 2021-02-10T11:26:32.528 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:32.525 7f8808141700 1 Processor -- start 2021-02-10T11:26:32.528 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:32.525 7f8808141700 1 -- start start 2021-02-10T11:26:32.528 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:32.526 7f8808141700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8800095cb0 0x7f880008a420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:32.528 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:32.526 7f8808141700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8800084cb0 0x7f88000961c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:32.529 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:32.526 7f8808141700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f88000850b0 0x7f8800085ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:32.529 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:32.526 7f8808141700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88000c08e0 con 0x7f8800095cb0 2021-02-10T11:26:32.529 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:32.526 7f8808141700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88000967e0 con 0x7f8800084cb0 2021-02-10T11:26:32.529 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:32.526 7f8808141700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88000860d0 con 0x7f88000850b0 2021-02-10T11:26:32.529 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:32.526 7f8805f5d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8800095cb0 0x7f880008a420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:32.529 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:32.526 7f8805f5d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8800095cb0 0x7f880008a420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:58744/0 (socket says 172.21.2.102:58744) 2021-02-10T11:26:32.529 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:32.526 7f8805f5d700 1 -- 172.21.2.102:0/1595385722 learned_addr learned my addr 172.21.2.102:0/1595385722 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:26:32.529 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:32.526 7f8805f5d700 1 -- 172.21.2.102:0/1595385722 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f88000850b0 msgr2=0x7f8800085ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:32.529 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:32.526 7f8805f5d700 1 --2- 172.21.2.102:0/1595385722 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f88000850b0 0x7f8800085ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:32.529 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:32.526 7f8805f5d700 1 -- 172.21.2.102:0/1595385722 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8800084cb0 msgr2=0x7f88000961c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:32.529 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:32.526 7f8805f5d700 1 --2- 172.21.2.102:0/1595385722 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8800084cb0 0x7f88000961c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:32.529 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:32.526 7f8805f5d700 1 -- 172.21.2.102:0/1595385722 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88000862f0 con 0x7f8800095cb0 2021-02-10T11:26:32.530 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:32.526 7f8805f5d700 1 --2- 172.21.2.102:0/1595385722 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8800095cb0 0x7f880008a420 secure :-1 s=READY pgs=1612 cs=0 l=1 rev1=1 rx=0x7f87fc006f40 tx=0x7f87fc00b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:32.530 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:32.527 7f87f27fc700 1 -- 172.21.2.102:0/1595385722 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f87fc00bd80 con 0x7f8800095cb0 2021-02-10T11:26:32.530 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:32.527 7f87f27fc700 1 -- 172.21.2.102:0/1595385722 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87fc000c90 con 0x7f8800095cb0 2021-02-10T11:26:32.530 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:32.527 7f87f27fc700 1 -- 172.21.2.102:0/1595385722 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f87fc002220 con 0x7f8800095cb0 2021-02-10T11:26:32.530 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:32.527 7f8808141700 1 -- 172.21.2.102:0/1595385722 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88000862f0 con 0x7f8800095cb0 2021-02-10T11:26:32.530 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:32.527 7f8808141700 1 -- 172.21.2.102:0/1595385722 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88000862f0 con 0x7f8800095cb0 2021-02-10T11:26:32.530 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:32.527 7f87f27fc700 1 -- 172.21.2.102:0/1595385722 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f87fc00a000 con 0x7f8800095cb0 2021-02-10T11:26:32.530 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:32.527 7f87f27fc700 1 --2- 172.21.2.102:0/1595385722 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f87e80186d0 0x7f87e801abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:32.530 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:32.527 7f87f27fc700 1 -- 172.21.2.102:0/1595385722 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f87fc032660 con 0x7f8800095cb0 2021-02-10T11:26:32.530 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:32.527 7f880575c700 1 --2- 172.21.2.102:0/1595385722 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f87e80186d0 0x7f87e801abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:32.530 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:32.528 7f880575c700 1 --2- 172.21.2.102:0/1595385722 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f87e80186d0 0x7f87e801abd0 crc :-1 s=READY pgs=731 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:32.557 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:32.557 7f8808141700 1 -- 172.21.2.102:0/1595385722 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8800082a70 con 0x7f8800095cb0 2021-02-10T11:26:32.559 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:32.560 7f87f27fc700 1 -- 172.21.2.102:0/1595385722 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f87fc000c90 con 0x7f8800095cb0 2021-02-10T11:26:32.711 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:32.712 7f8808141700 1 -- 172.21.2.102:0/1595385722 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f8800082a70 con 0x7f8800095cb0 2021-02-10T11:26:32.715 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:32.716 7f87f27fc700 1 -- 172.21.2.102:0/1595385722 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v20)=0 v20) v1 ==== 122+0+0 (secure 0 0 0) 0x7f87fc00a000 con 0x7f8800095cb0 2021-02-10T11:26:32.720 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:32.720 7f8808141700 1 -- 172.21.2.102:0/1595385722 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f87e80186d0 msgr2=0x7f87e801abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:32.720 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:32.720 7f8808141700 1 --2- 172.21.2.102:0/1595385722 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f87e80186d0 0x7f87e801abd0 crc :-1 s=READY pgs=731 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:32.720 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:32.720 7f8808141700 1 -- 172.21.2.102:0/1595385722 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8800095cb0 msgr2=0x7f880008a420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:32.721 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:32.720 7f8808141700 1 --2- 172.21.2.102:0/1595385722 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8800095cb0 0x7f880008a420 secure :-1 s=READY pgs=1612 cs=0 l=1 rev1=1 rx=0x7f87fc006f40 tx=0x7f87fc00b990).stop 2021-02-10T11:26:32.721 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:32.720 7f8808141700 1 -- 172.21.2.102:0/1595385722 shutdown_connections 2021-02-10T11:26:32.721 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:32.720 7f8808141700 1 --2- 172.21.2.102:0/1595385722 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f88000850b0 0x7f8800085ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:32.721 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:32.720 7f8808141700 1 --2- 172.21.2.102:0/1595385722 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f87e80186d0 0x7f87e801abd0 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:32.721 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:32.720 7f8808141700 1 --2- 172.21.2.102:0/1595385722 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8800084cb0 0x7f88000961c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:32.721 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:32.720 7f8808141700 1 --2- 172.21.2.102:0/1595385722 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8800095cb0 0x7f880008a420 unknown :-1 s=CLOSED pgs=1612 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:32.721 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:32.720 7f8808141700 1 -- 172.21.2.102:0/1595385722 >> 172.21.2.102:0/1595385722 conn(0x7f88000874f0 msgr2=0x7f88000b7420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:26:32.721 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:32.720 7f8808141700 1 -- 172.21.2.102:0/1595385722 shutdown_connections 2021-02-10T11:26:32.721 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:32.720 7f8808141700 1 -- 172.21.2.102:0/1595385722 wait complete. 2021-02-10T11:26:32.729 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2021-02-10T11:26:32.729 DEBUG:teuthology.orchestra.run.gibba002:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:26:33.073 DEBUG:teuthology.orchestra.run.gibba013:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:26:33.418 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2021-02-10T11:26:33.419 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:33.419 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:26:33.419 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:26:33.755 INFO:teuthology.orchestra.run.gibba002.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.0’: No such file or directory 2021-02-10T11:26:33.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:26:33.755 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.0 2021-02-10T11:26:33.755 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:33.755 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:26:33.755 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:26:34.090 INFO:teuthology.orchestra.run.gibba002.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.1’: No such file or directory 2021-02-10T11:26:34.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:26:34.090 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2021-02-10T11:26:34.090 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:34.090 DEBUG:teuthology.orchestra.run.gibba013:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:26:34.091 DEBUG:teuthology.orchestra.run.gibba013:> ' -- /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:26:34.425 INFO:teuthology.orchestra.run.gibba013.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.2’: No such file or directory 2021-02-10T11:26:34.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:26:34.426 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2021-02-10T11:26:34.426 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:34.426 DEBUG:teuthology.orchestra.run.gibba013:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:26:34.426 DEBUG:teuthology.orchestra.run.gibba013:> ' -- /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:26:34.760 INFO:teuthology.orchestra.run.gibba013.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.3’: No such file or directory 2021-02-10T11:26:34.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:26:34.761 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.3 2021-02-10T11:26:34.761 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json-pretty 2021-02-10T11:26:35.188 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:35.173 7f83828db700 1 Processor -- start 2021-02-10T11:26:35.189 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:35.173 7f83828db700 1 -- start start 2021-02-10T11:26:35.189 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:35.174 7f83828db700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f837c0d1870 0x7f837c1b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:35.189 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:35.174 7f83828db700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f837c1b0a10 0x7f837c0cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:35.189 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:35.174 7f83828db700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f837c0d1c70 0x7f837c0cf950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:35.189 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:35.174 7f83828db700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f837c0fe230 con 0x7f837c0d1c70 2021-02-10T11:26:35.189 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:35.174 7f83828db700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f837c0e5ed0 con 0x7f837c1b0a10 2021-02-10T11:26:35.189 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:35.174 7f83828db700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f837c0f0b10 con 0x7f837c0d1870 2021-02-10T11:26:35.189 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:35.174 7f8380ef8700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f837c0d1c70 0x7f837c0cf950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:35.189 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:35.174 7f8380ef8700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f837c0d1c70 0x7f837c0cf950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:58752/0 (socket says 172.21.2.102:58752) 2021-02-10T11:26:35.189 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:35.174 7f8380ef8700 1 -- 172.21.2.102:0/2632039372 learned_addr learned my addr 172.21.2.102:0/2632039372 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:26:35.189 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:35.174 7f8380ef8700 1 -- 172.21.2.102:0/2632039372 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f837c0d1870 msgr2=0x7f837c1b03f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:26:35.190 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:35.174 7f8380ef8700 1 --2- 172.21.2.102:0/2632039372 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f837c0d1870 0x7f837c1b03f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:35.190 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:35.174 7f8380ef8700 1 -- 172.21.2.102:0/2632039372 >> v2:172.21.2.113:3300/0 conn(0x7f837c1b0a10 msgr2=0x7f837c0cf330 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:26:35.190 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:35.174 7f8380ef8700 1 --2- 172.21.2.102:0/2632039372 >> v2:172.21.2.113:3300/0 conn(0x7f837c1b0a10 0x7f837c0cf330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:35.190 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:35.174 7f8380ef8700 1 -- 172.21.2.102:0/2632039372 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f837c0cff70 con 0x7f837c0d1c70 2021-02-10T11:26:35.190 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:35.174 7f8380ef8700 1 --2- 172.21.2.102:0/2632039372 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f837c0d1c70 0x7f837c0cf950 secure :-1 s=READY pgs=1613 cs=0 l=1 rev1=1 rx=0x7f8374006f00 tx=0x7f8374009f10).ready entity=mon.0 client_cookie=9d576104b1ea0762 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:35.190 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:35.174 7f837affd700 1 -- 172.21.2.102:0/2632039372 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f837400b3b0 con 0x7f837c0d1c70 2021-02-10T11:26:35.190 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:35.174 7f837affd700 1 -- 172.21.2.102:0/2632039372 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f837400b630 con 0x7f837c0d1c70 2021-02-10T11:26:35.190 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:35.174 7f837affd700 1 -- 172.21.2.102:0/2632039372 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f837400bb50 con 0x7f837c0d1c70 2021-02-10T11:26:35.190 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:35.174 7f83828db700 1 -- 172.21.2.102:0/2632039372 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f837c0d1c70 msgr2=0x7f837c0cf950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:35.190 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:35.174 7f83828db700 1 --2- 172.21.2.102:0/2632039372 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f837c0d1c70 0x7f837c0cf950 secure :-1 s=READY pgs=1613 cs=0 l=1 rev1=1 rx=0x7f8374006f00 tx=0x7f8374009f10).stop 2021-02-10T11:26:35.190 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:35.175 7f83828db700 1 -- 172.21.2.102:0/2632039372 shutdown_connections 2021-02-10T11:26:35.190 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:35.175 7f83828db700 1 --2- 172.21.2.102:0/2632039372 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f837c0d1c70 0x7f837c0cf950 unknown :-1 s=CLOSED pgs=1613 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:35.190 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:35.175 7f83828db700 1 --2- 172.21.2.102:0/2632039372 >> v2:172.21.2.113:3300/0 conn(0x7f837c1b0a10 0x7f837c0cf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:35.191 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:35.175 7f83828db700 1 --2- 172.21.2.102:0/2632039372 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f837c0d1870 0x7f837c1b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:35.191 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:35.175 7f83828db700 1 -- 172.21.2.102:0/2632039372 >> 172.21.2.102:0/2632039372 conn(0x7f837c0ce450 msgr2=0x7f837c0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:26:35.191 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:35.175 7f83828db700 1 -- 172.21.2.102:0/2632039372 shutdown_connections 2021-02-10T11:26:35.191 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:35.175 7f83828db700 1 -- 172.21.2.102:0/2632039372 wait complete. 2021-02-10T11:26:35.192 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:35.189 7f83828db700 1 Processor -- start 2021-02-10T11:26:35.192 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:35.189 7f83828db700 1 -- start start 2021-02-10T11:26:35.192 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:35.190 7f83828db700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f837c093060 0x7f837c19a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:35.192 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:35.190 7f83828db700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f837c19aeb0 0x7f837c0b1990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:35.193 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:35.190 7f83828db700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f837c19b2b0 0x7f837c0b1fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:35.193 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:35.190 7f83828db700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f837c0efd70 con 0x7f837c093060 2021-02-10T11:26:35.193 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:35.190 7f83828db700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f837c19b8a0 con 0x7f837c19aeb0 2021-02-10T11:26:35.193 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:35.190 7f83828db700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f837c0b25d0 con 0x7f837c19b2b0 2021-02-10T11:26:35.193 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:35.190 7f837bfff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f837c093060 0x7f837c19a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:35.193 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:35.190 7f837bfff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f837c093060 0x7f837c19a890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:58762/0 (socket says 172.21.2.102:58762) 2021-02-10T11:26:35.193 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:35.190 7f837bfff700 1 -- 172.21.2.102:0/3624204280 learned_addr learned my addr 172.21.2.102:0/3624204280 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:26:35.193 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:35.190 7f837bfff700 1 -- 172.21.2.102:0/3624204280 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f837c19b2b0 msgr2=0x7f837c0b1fb0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:26:35.193 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:35.190 7f837bfff700 1 --2- 172.21.2.102:0/3624204280 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f837c19b2b0 0x7f837c0b1fb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:35.193 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:35.190 7f837bfff700 1 -- 172.21.2.102:0/3624204280 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f837c19aeb0 msgr2=0x7f837c0b1990 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:26:35.193 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:35.190 7f837bfff700 1 --2- 172.21.2.102:0/3624204280 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f837c19aeb0 0x7f837c0b1990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:35.193 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:35.190 7f837bfff700 1 -- 172.21.2.102:0/3624204280 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f837c18e410 con 0x7f837c093060 2021-02-10T11:26:35.193 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:35.190 7f837bfff700 1 --2- 172.21.2.102:0/3624204280 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f837c093060 0x7f837c19a890 secure :-1 s=READY pgs=1614 cs=0 l=1 rev1=1 rx=0x7f836c00b560 tx=0x7f836c00b900).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:35.194 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:35.191 7f8378ff9700 1 -- 172.21.2.102:0/3624204280 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f836c00bcf0 con 0x7f837c093060 2021-02-10T11:26:35.194 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:35.191 7f8378ff9700 1 -- 172.21.2.102:0/3624204280 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f836c006030 con 0x7f837c093060 2021-02-10T11:26:35.194 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:35.191 7f83828db700 1 -- 172.21.2.102:0/3624204280 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f837c18e410 con 0x7f837c093060 2021-02-10T11:26:35.194 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:35.191 7f8378ff9700 1 -- 172.21.2.102:0/3624204280 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f836c0054d0 con 0x7f837c093060 2021-02-10T11:26:35.194 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:35.191 7f83828db700 1 -- 172.21.2.102:0/3624204280 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f837c18e9a0 con 0x7f837c093060 2021-02-10T11:26:35.194 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:35.191 7f8378ff9700 1 -- 172.21.2.102:0/3624204280 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f836c006210 con 0x7f837c093060 2021-02-10T11:26:35.194 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:35.191 7f8378ff9700 1 --2- 172.21.2.102:0/3624204280 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f835c018700 0x7f835c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:35.194 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:35.191 7f8378ff9700 1 -- 172.21.2.102:0/3624204280 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f836c032630 con 0x7f837c093060 2021-02-10T11:26:35.194 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:35.191 7f837b7fe700 1 --2- 172.21.2.102:0/3624204280 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f835c018700 0x7f835c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:35.194 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:35.192 7f837b7fe700 1 --2- 172.21.2.102:0/3624204280 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f835c018700 0x7f835c01abc0 crc :-1 s=READY pgs=732 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:35.194 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:35.192 7f83828db700 1 -- 172.21.2.102:0/3624204280 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f837c18f230 con 0x7f837c093060 2021-02-10T11:26:35.195 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:35.194 7f8378ff9700 1 -- 172.21.2.102:0/3624204280 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f836c0009b0 con 0x7f837c093060 2021-02-10T11:26:35.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:35.318 7f83828db700 1 -- 172.21.2.102:0/3624204280 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7f837c18f230 con 0x7f837c093060 2021-02-10T11:26:35.318 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:35.318 7f8378ff9700 1 -- 172.21.2.102:0/3624204280 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v66) v1 ==== 81+0+30363 (secure 0 0 0) 0x7f836c006030 con 0x7f837c093060 2021-02-10T11:26:35.319 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:26:35.319 INFO:teuthology.orchestra.run.gibba002.stdout:{ 2021-02-10T11:26:35.319 INFO:teuthology.orchestra.run.gibba002.stdout: "epoch": 66, 2021-02-10T11:26:35.319 INFO:teuthology.orchestra.run.gibba002.stdout: "fsid": "68db30ea-dbce-4aef-a6df-f01c6fea0513", 2021-02-10T11:26:35.319 INFO:teuthology.orchestra.run.gibba002.stdout: "created": "2021-02-10 11:03:06.331557", 2021-02-10T11:26:35.319 INFO:teuthology.orchestra.run.gibba002.stdout: "modified": "2021-02-10 11:25:32.516321", 2021-02-10T11:26:35.319 INFO:teuthology.orchestra.run.gibba002.stdout: "last_up_change": "2021-02-10 11:03:32.690562", 2021-02-10T11:26:35.319 INFO:teuthology.orchestra.run.gibba002.stdout: "last_in_change": "2021-02-10 11:03:32.690562", 2021-02-10T11:26:35.319 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2021-02-10T11:26:35.319 INFO:teuthology.orchestra.run.gibba002.stdout: "flags_num": 5799936, 2021-02-10T11:26:35.319 INFO:teuthology.orchestra.run.gibba002.stdout: "flags_set": [ 2021-02-10T11:26:35.320 INFO:teuthology.orchestra.run.gibba002.stdout: "pglog_hardlimit", 2021-02-10T11:26:35.320 INFO:teuthology.orchestra.run.gibba002.stdout: "purged_snapdirs", 2021-02-10T11:26:35.320 INFO:teuthology.orchestra.run.gibba002.stdout: "recovery_deletes", 2021-02-10T11:26:35.320 INFO:teuthology.orchestra.run.gibba002.stdout: "sortbitwise" 2021-02-10T11:26:35.320 INFO:teuthology.orchestra.run.gibba002.stdout: ], 2021-02-10T11:26:35.320 INFO:teuthology.orchestra.run.gibba002.stdout: "crush_version": 4, 2021-02-10T11:26:35.320 INFO:teuthology.orchestra.run.gibba002.stdout: "full_ratio": 0.94999998807907104, 2021-02-10T11:26:35.320 INFO:teuthology.orchestra.run.gibba002.stdout: "backfillfull_ratio": 0.89999997615814209, 2021-02-10T11:26:35.320 INFO:teuthology.orchestra.run.gibba002.stdout: "nearfull_ratio": 0.85000002384185791, 2021-02-10T11:26:35.320 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_snapshot": "", 2021-02-10T11:26:35.320 INFO:teuthology.orchestra.run.gibba002.stdout: "pool_max": 13, 2021-02-10T11:26:35.320 INFO:teuthology.orchestra.run.gibba002.stdout: "max_osd": 8, 2021-02-10T11:26:35.320 INFO:teuthology.orchestra.run.gibba002.stdout: "require_min_compat_client": "jewel", 2021-02-10T11:26:35.321 INFO:teuthology.orchestra.run.gibba002.stdout: "min_compat_client": "jewel", 2021-02-10T11:26:35.321 INFO:teuthology.orchestra.run.gibba002.stdout: "require_osd_release": "nautilus", 2021-02-10T11:26:35.321 INFO:teuthology.orchestra.run.gibba002.stdout: "pools": [ 2021-02-10T11:26:35.321 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:26:35.321 INFO:teuthology.orchestra.run.gibba002.stdout: "pool": 1, 2021-02-10T11:26:35.321 INFO:teuthology.orchestra.run.gibba002.stdout: "pool_name": "rbd", 2021-02-10T11:26:35.321 INFO:teuthology.orchestra.run.gibba002.stdout: "create_time": "2021-02-10 11:03:37.322519", 2021-02-10T11:26:35.321 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 1, 2021-02-10T11:26:35.321 INFO:teuthology.orchestra.run.gibba002.stdout: "flags_names": "hashpspool", 2021-02-10T11:26:35.321 INFO:teuthology.orchestra.run.gibba002.stdout: "type": 1, 2021-02-10T11:26:35.321 INFO:teuthology.orchestra.run.gibba002.stdout: "size": 2, 2021-02-10T11:26:35.321 INFO:teuthology.orchestra.run.gibba002.stdout: "min_size": 1, 2021-02-10T11:26:35.321 INFO:teuthology.orchestra.run.gibba002.stdout: "crush_rule": 0, 2021-02-10T11:26:35.321 INFO:teuthology.orchestra.run.gibba002.stdout: "object_hash": 2, 2021-02-10T11:26:35.322 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:26:35.322 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num": 8, 2021-02-10T11:26:35.322 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_placement_num": 8, 2021-02-10T11:26:35.322 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_placement_num_target": 8, 2021-02-10T11:26:35.322 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num_target": 8, 2021-02-10T11:26:35.322 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num_pending": 8, 2021-02-10T11:26:35.322 INFO:teuthology.orchestra.run.gibba002.stdout: "last_pg_merge_meta": { 2021-02-10T11:26:35.322 INFO:teuthology.orchestra.run.gibba002.stdout: "source_pgid": "0.0", 2021-02-10T11:26:35.322 INFO:teuthology.orchestra.run.gibba002.stdout: "ready_epoch": 0, 2021-02-10T11:26:35.322 INFO:teuthology.orchestra.run.gibba002.stdout: "last_epoch_started": 0, 2021-02-10T11:26:35.322 INFO:teuthology.orchestra.run.gibba002.stdout: "last_epoch_clean": 0, 2021-02-10T11:26:35.322 INFO:teuthology.orchestra.run.gibba002.stdout: "source_version": "0'0", 2021-02-10T11:26:35.322 INFO:teuthology.orchestra.run.gibba002.stdout: "target_version": "0'0" 2021-02-10T11:26:35.323 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:26:35.323 INFO:teuthology.orchestra.run.gibba002.stdout: "last_change": "16", 2021-02-10T11:26:35.323 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend": "0", 2021-02-10T11:26:35.323 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:26:35.323 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:26:35.323 INFO:teuthology.orchestra.run.gibba002.stdout: "auid": 0, 2021-02-10T11:26:35.323 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:26:35.323 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_seq": 0, 2021-02-10T11:26:35.323 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_epoch": 0, 2021-02-10T11:26:35.323 INFO:teuthology.orchestra.run.gibba002.stdout: "pool_snaps": [], 2021-02-10T11:26:35.323 INFO:teuthology.orchestra.run.gibba002.stdout: "removed_snaps": "[]", 2021-02-10T11:26:35.323 INFO:teuthology.orchestra.run.gibba002.stdout: "quota_max_bytes": 0, 2021-02-10T11:26:35.323 INFO:teuthology.orchestra.run.gibba002.stdout: "quota_max_objects": 0, 2021-02-10T11:26:35.323 INFO:teuthology.orchestra.run.gibba002.stdout: "tiers": [], 2021-02-10T11:26:35.324 INFO:teuthology.orchestra.run.gibba002.stdout: "tier_of": -1, 2021-02-10T11:26:35.324 INFO:teuthology.orchestra.run.gibba002.stdout: "read_tier": -1, 2021-02-10T11:26:35.324 INFO:teuthology.orchestra.run.gibba002.stdout: "write_tier": -1, 2021-02-10T11:26:35.324 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_mode": "none", 2021-02-10T11:26:35.324 INFO:teuthology.orchestra.run.gibba002.stdout: "target_max_bytes": 0, 2021-02-10T11:26:35.324 INFO:teuthology.orchestra.run.gibba002.stdout: "target_max_objects": 0, 2021-02-10T11:26:35.324 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:26:35.324 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:26:35.324 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:26:35.324 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_min_flush_age": 0, 2021-02-10T11:26:35.324 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_min_evict_age": 0, 2021-02-10T11:26:35.324 INFO:teuthology.orchestra.run.gibba002.stdout: "erasure_code_profile": "", 2021-02-10T11:26:35.324 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_params": { 2021-02-10T11:26:35.325 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "none" 2021-02-10T11:26:35.325 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:26:35.325 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_period": 0, 2021-02-10T11:26:35.325 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_count": 0, 2021-02-10T11:26:35.325 INFO:teuthology.orchestra.run.gibba002.stdout: "use_gmt_hitset": true, 2021-02-10T11:26:35.325 INFO:teuthology.orchestra.run.gibba002.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:26:35.325 INFO:teuthology.orchestra.run.gibba002.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:26:35.325 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:26:35.325 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:26:35.325 INFO:teuthology.orchestra.run.gibba002.stdout: "grade_table": [], 2021-02-10T11:26:35.325 INFO:teuthology.orchestra.run.gibba002.stdout: "stripe_width": 0, 2021-02-10T11:26:35.325 INFO:teuthology.orchestra.run.gibba002.stdout: "expected_num_objects": 0, 2021-02-10T11:26:35.325 INFO:teuthology.orchestra.run.gibba002.stdout: "fast_read": false, 2021-02-10T11:26:35.325 INFO:teuthology.orchestra.run.gibba002.stdout: "options": {}, 2021-02-10T11:26:35.326 INFO:teuthology.orchestra.run.gibba002.stdout: "application_metadata": { 2021-02-10T11:26:35.326 INFO:teuthology.orchestra.run.gibba002.stdout: "rbd": {} 2021-02-10T11:26:35.326 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:26:35.326 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:26:35.326 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:26:35.326 INFO:teuthology.orchestra.run.gibba002.stdout: "pool": 12, 2021-02-10T11:26:35.326 INFO:teuthology.orchestra.run.gibba002.stdout: "pool_name": "cephfs_metadata", 2021-02-10T11:26:35.326 INFO:teuthology.orchestra.run.gibba002.stdout: "create_time": "2021-02-10 11:25:29.586765", 2021-02-10T11:26:35.326 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 1, 2021-02-10T11:26:35.326 INFO:teuthology.orchestra.run.gibba002.stdout: "flags_names": "hashpspool", 2021-02-10T11:26:35.326 INFO:teuthology.orchestra.run.gibba002.stdout: "type": 1, 2021-02-10T11:26:35.326 INFO:teuthology.orchestra.run.gibba002.stdout: "size": 2, 2021-02-10T11:26:35.326 INFO:teuthology.orchestra.run.gibba002.stdout: "min_size": 1, 2021-02-10T11:26:35.327 INFO:teuthology.orchestra.run.gibba002.stdout: "crush_rule": 0, 2021-02-10T11:26:35.327 INFO:teuthology.orchestra.run.gibba002.stdout: "object_hash": 2, 2021-02-10T11:26:35.327 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:26:35.327 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num": 8, 2021-02-10T11:26:35.327 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_placement_num": 8, 2021-02-10T11:26:35.327 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_placement_num_target": 8, 2021-02-10T11:26:35.327 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num_target": 8, 2021-02-10T11:26:35.327 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num_pending": 8, 2021-02-10T11:26:35.327 INFO:teuthology.orchestra.run.gibba002.stdout: "last_pg_merge_meta": { 2021-02-10T11:26:35.327 INFO:teuthology.orchestra.run.gibba002.stdout: "source_pgid": "0.0", 2021-02-10T11:26:35.327 INFO:teuthology.orchestra.run.gibba002.stdout: "ready_epoch": 0, 2021-02-10T11:26:35.327 INFO:teuthology.orchestra.run.gibba002.stdout: "last_epoch_started": 0, 2021-02-10T11:26:35.327 INFO:teuthology.orchestra.run.gibba002.stdout: "last_epoch_clean": 0, 2021-02-10T11:26:35.327 INFO:teuthology.orchestra.run.gibba002.stdout: "source_version": "0'0", 2021-02-10T11:26:35.328 INFO:teuthology.orchestra.run.gibba002.stdout: "target_version": "0'0" 2021-02-10T11:26:35.328 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:26:35.328 INFO:teuthology.orchestra.run.gibba002.stdout: "last_change": "65", 2021-02-10T11:26:35.328 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend": "0", 2021-02-10T11:26:35.328 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:26:35.328 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:26:35.328 INFO:teuthology.orchestra.run.gibba002.stdout: "auid": 0, 2021-02-10T11:26:35.328 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:26:35.328 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_seq": 0, 2021-02-10T11:26:35.328 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_epoch": 0, 2021-02-10T11:26:35.328 INFO:teuthology.orchestra.run.gibba002.stdout: "pool_snaps": [], 2021-02-10T11:26:35.328 INFO:teuthology.orchestra.run.gibba002.stdout: "removed_snaps": "[]", 2021-02-10T11:26:35.328 INFO:teuthology.orchestra.run.gibba002.stdout: "quota_max_bytes": 0, 2021-02-10T11:26:35.328 INFO:teuthology.orchestra.run.gibba002.stdout: "quota_max_objects": 0, 2021-02-10T11:26:35.329 INFO:teuthology.orchestra.run.gibba002.stdout: "tiers": [], 2021-02-10T11:26:35.329 INFO:teuthology.orchestra.run.gibba002.stdout: "tier_of": -1, 2021-02-10T11:26:35.329 INFO:teuthology.orchestra.run.gibba002.stdout: "read_tier": -1, 2021-02-10T11:26:35.329 INFO:teuthology.orchestra.run.gibba002.stdout: "write_tier": -1, 2021-02-10T11:26:35.329 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_mode": "none", 2021-02-10T11:26:35.329 INFO:teuthology.orchestra.run.gibba002.stdout: "target_max_bytes": 0, 2021-02-10T11:26:35.329 INFO:teuthology.orchestra.run.gibba002.stdout: "target_max_objects": 0, 2021-02-10T11:26:35.329 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:26:35.329 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:26:35.329 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:26:35.329 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_min_flush_age": 0, 2021-02-10T11:26:35.329 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_min_evict_age": 0, 2021-02-10T11:26:35.329 INFO:teuthology.orchestra.run.gibba002.stdout: "erasure_code_profile": "", 2021-02-10T11:26:35.329 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_params": { 2021-02-10T11:26:35.330 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "none" 2021-02-10T11:26:35.330 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:26:35.330 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_period": 0, 2021-02-10T11:26:35.330 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_count": 0, 2021-02-10T11:26:35.330 INFO:teuthology.orchestra.run.gibba002.stdout: "use_gmt_hitset": true, 2021-02-10T11:26:35.330 INFO:teuthology.orchestra.run.gibba002.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:26:35.330 INFO:teuthology.orchestra.run.gibba002.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:26:35.330 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:26:35.330 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:26:35.330 INFO:teuthology.orchestra.run.gibba002.stdout: "grade_table": [], 2021-02-10T11:26:35.330 INFO:teuthology.orchestra.run.gibba002.stdout: "stripe_width": 0, 2021-02-10T11:26:35.330 INFO:teuthology.orchestra.run.gibba002.stdout: "expected_num_objects": 0, 2021-02-10T11:26:35.330 INFO:teuthology.orchestra.run.gibba002.stdout: "fast_read": false, 2021-02-10T11:26:35.331 INFO:teuthology.orchestra.run.gibba002.stdout: "options": { 2021-02-10T11:26:35.331 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_autoscale_bias": 4, 2021-02-10T11:26:35.331 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num_min": 16, 2021-02-10T11:26:35.331 INFO:teuthology.orchestra.run.gibba002.stdout: "recovery_priority": 5 2021-02-10T11:26:35.331 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:26:35.331 INFO:teuthology.orchestra.run.gibba002.stdout: "application_metadata": { 2021-02-10T11:26:35.331 INFO:teuthology.orchestra.run.gibba002.stdout: "cephfs": { 2021-02-10T11:26:35.331 INFO:teuthology.orchestra.run.gibba002.stdout: "metadata": "cephfs" 2021-02-10T11:26:35.331 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:26:35.331 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:26:35.331 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:26:35.331 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:26:35.331 INFO:teuthology.orchestra.run.gibba002.stdout: "pool": 13, 2021-02-10T11:26:35.331 INFO:teuthology.orchestra.run.gibba002.stdout: "pool_name": "cephfs_data", 2021-02-10T11:26:35.332 INFO:teuthology.orchestra.run.gibba002.stdout: "create_time": "2021-02-10 11:25:30.558626", 2021-02-10T11:26:35.332 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 1, 2021-02-10T11:26:35.332 INFO:teuthology.orchestra.run.gibba002.stdout: "flags_names": "hashpspool", 2021-02-10T11:26:35.332 INFO:teuthology.orchestra.run.gibba002.stdout: "type": 1, 2021-02-10T11:26:35.332 INFO:teuthology.orchestra.run.gibba002.stdout: "size": 2, 2021-02-10T11:26:35.332 INFO:teuthology.orchestra.run.gibba002.stdout: "min_size": 1, 2021-02-10T11:26:35.332 INFO:teuthology.orchestra.run.gibba002.stdout: "crush_rule": 0, 2021-02-10T11:26:35.332 INFO:teuthology.orchestra.run.gibba002.stdout: "object_hash": 2, 2021-02-10T11:26:35.332 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:26:35.332 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num": 8, 2021-02-10T11:26:35.332 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_placement_num": 8, 2021-02-10T11:26:35.332 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_placement_num_target": 8, 2021-02-10T11:26:35.332 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num_target": 8, 2021-02-10T11:26:35.332 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num_pending": 8, 2021-02-10T11:26:35.333 INFO:teuthology.orchestra.run.gibba002.stdout: "last_pg_merge_meta": { 2021-02-10T11:26:35.333 INFO:teuthology.orchestra.run.gibba002.stdout: "source_pgid": "0.0", 2021-02-10T11:26:35.333 INFO:teuthology.orchestra.run.gibba002.stdout: "ready_epoch": 0, 2021-02-10T11:26:35.333 INFO:teuthology.orchestra.run.gibba002.stdout: "last_epoch_started": 0, 2021-02-10T11:26:35.333 INFO:teuthology.orchestra.run.gibba002.stdout: "last_epoch_clean": 0, 2021-02-10T11:26:35.333 INFO:teuthology.orchestra.run.gibba002.stdout: "source_version": "0'0", 2021-02-10T11:26:35.333 INFO:teuthology.orchestra.run.gibba002.stdout: "target_version": "0'0" 2021-02-10T11:26:35.333 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:26:35.333 INFO:teuthology.orchestra.run.gibba002.stdout: "last_change": "66", 2021-02-10T11:26:35.333 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend": "0", 2021-02-10T11:26:35.333 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:26:35.333 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:26:35.333 INFO:teuthology.orchestra.run.gibba002.stdout: "auid": 0, 2021-02-10T11:26:35.333 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:26:35.334 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_seq": 0, 2021-02-10T11:26:35.334 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_epoch": 0, 2021-02-10T11:26:35.334 INFO:teuthology.orchestra.run.gibba002.stdout: "pool_snaps": [], 2021-02-10T11:26:35.334 INFO:teuthology.orchestra.run.gibba002.stdout: "removed_snaps": "[]", 2021-02-10T11:26:35.334 INFO:teuthology.orchestra.run.gibba002.stdout: "quota_max_bytes": 0, 2021-02-10T11:26:35.334 INFO:teuthology.orchestra.run.gibba002.stdout: "quota_max_objects": 0, 2021-02-10T11:26:35.334 INFO:teuthology.orchestra.run.gibba002.stdout: "tiers": [], 2021-02-10T11:26:35.334 INFO:teuthology.orchestra.run.gibba002.stdout: "tier_of": -1, 2021-02-10T11:26:35.334 INFO:teuthology.orchestra.run.gibba002.stdout: "read_tier": -1, 2021-02-10T11:26:35.334 INFO:teuthology.orchestra.run.gibba002.stdout: "write_tier": -1, 2021-02-10T11:26:35.334 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_mode": "none", 2021-02-10T11:26:35.334 INFO:teuthology.orchestra.run.gibba002.stdout: "target_max_bytes": 0, 2021-02-10T11:26:35.334 INFO:teuthology.orchestra.run.gibba002.stdout: "target_max_objects": 0, 2021-02-10T11:26:35.335 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:26:35.335 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:26:35.335 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:26:35.335 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_min_flush_age": 0, 2021-02-10T11:26:35.335 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_min_evict_age": 0, 2021-02-10T11:26:35.335 INFO:teuthology.orchestra.run.gibba002.stdout: "erasure_code_profile": "", 2021-02-10T11:26:35.335 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_params": { 2021-02-10T11:26:35.335 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "none" 2021-02-10T11:26:35.335 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:26:35.335 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_period": 0, 2021-02-10T11:26:35.335 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_count": 0, 2021-02-10T11:26:35.335 INFO:teuthology.orchestra.run.gibba002.stdout: "use_gmt_hitset": true, 2021-02-10T11:26:35.335 INFO:teuthology.orchestra.run.gibba002.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:26:35.336 INFO:teuthology.orchestra.run.gibba002.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:26:35.336 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:26:35.336 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:26:35.336 INFO:teuthology.orchestra.run.gibba002.stdout: "grade_table": [], 2021-02-10T11:26:35.336 INFO:teuthology.orchestra.run.gibba002.stdout: "stripe_width": 0, 2021-02-10T11:26:35.336 INFO:teuthology.orchestra.run.gibba002.stdout: "expected_num_objects": 0, 2021-02-10T11:26:35.336 INFO:teuthology.orchestra.run.gibba002.stdout: "fast_read": false, 2021-02-10T11:26:35.336 INFO:teuthology.orchestra.run.gibba002.stdout: "options": {}, 2021-02-10T11:26:35.336 INFO:teuthology.orchestra.run.gibba002.stdout: "application_metadata": { 2021-02-10T11:26:35.336 INFO:teuthology.orchestra.run.gibba002.stdout: "cephfs": { 2021-02-10T11:26:35.337 INFO:teuthology.orchestra.run.gibba002.stdout: "data": "cephfs" 2021-02-10T11:26:35.337 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:26:35.337 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:26:35.337 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:26:35.337 INFO:teuthology.orchestra.run.gibba002.stdout: ], 2021-02-10T11:26:35.337 INFO:teuthology.orchestra.run.gibba002.stdout: "osds": [ 2021-02-10T11:26:35.337 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:26:35.337 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 0, 2021-02-10T11:26:35.337 INFO:teuthology.orchestra.run.gibba002.stdout: "uuid": "be691b4d-a039-408a-af80-9d53f02b6f1f", 2021-02-10T11:26:35.337 INFO:teuthology.orchestra.run.gibba002.stdout: "up": 1, 2021-02-10T11:26:35.337 INFO:teuthology.orchestra.run.gibba002.stdout: "in": 1, 2021-02-10T11:26:35.337 INFO:teuthology.orchestra.run.gibba002.stdout: "weight": 1, 2021-02-10T11:26:35.337 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_affinity": 1, 2021-02-10T11:26:35.338 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_begin": 0, 2021-02-10T11:26:35.338 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_end": 0, 2021-02-10T11:26:35.338 INFO:teuthology.orchestra.run.gibba002.stdout: "up_from": 13, 2021-02-10T11:26:35.338 INFO:teuthology.orchestra.run.gibba002.stdout: "up_thru": 64, 2021-02-10T11:26:35.338 INFO:teuthology.orchestra.run.gibba002.stdout: "down_at": 0, 2021-02-10T11:26:35.338 INFO:teuthology.orchestra.run.gibba002.stdout: "lost_at": 0, 2021-02-10T11:26:35.338 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addrs": { 2021-02-10T11:26:35.338 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:26:35.338 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:26:35.338 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:26:35.338 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6818", 2021-02-10T11:26:35.338 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36088 2021-02-10T11:26:35.338 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:26:35.339 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:26:35.339 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:26:35.339 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6820", 2021-02-10T11:26:35.339 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36088 2021-02-10T11:26:35.339 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:26:35.339 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:26:35.339 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:26:35.339 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addrs": { 2021-02-10T11:26:35.339 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:26:35.339 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:26:35.339 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:26:35.339 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6822", 2021-02-10T11:26:35.339 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36088 2021-02-10T11:26:35.339 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:26:35.340 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:26:35.340 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:26:35.340 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6824", 2021-02-10T11:26:35.340 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36088 2021-02-10T11:26:35.340 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:26:35.340 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:26:35.340 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:26:35.340 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addrs": { 2021-02-10T11:26:35.340 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:26:35.340 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:26:35.340 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:26:35.340 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6830", 2021-02-10T11:26:35.340 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36088 2021-02-10T11:26:35.341 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:26:35.341 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:26:35.341 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:26:35.341 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6832", 2021-02-10T11:26:35.341 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36088 2021-02-10T11:26:35.341 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:26:35.341 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:26:35.341 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:26:35.341 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addrs": { 2021-02-10T11:26:35.341 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:26:35.341 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:26:35.341 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:26:35.342 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6826", 2021-02-10T11:26:35.342 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36088 2021-02-10T11:26:35.342 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:26:35.342 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:26:35.342 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:26:35.342 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6828", 2021-02-10T11:26:35.342 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36088 2021-02-10T11:26:35.342 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:26:35.342 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:26:35.342 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:26:35.342 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addr": "172.21.2.102:6820/36088", 2021-02-10T11:26:35.343 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addr": "172.21.2.102:6824/36088", 2021-02-10T11:26:35.343 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addr": "172.21.2.102:6832/36088", 2021-02-10T11:26:35.343 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addr": "172.21.2.102:6828/36088", 2021-02-10T11:26:35.343 INFO:teuthology.orchestra.run.gibba002.stdout: "state": [ 2021-02-10T11:26:35.343 INFO:teuthology.orchestra.run.gibba002.stdout: "exists", 2021-02-10T11:26:35.343 INFO:teuthology.orchestra.run.gibba002.stdout: "up" 2021-02-10T11:26:35.343 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:26:35.343 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:26:35.343 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:26:35.343 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 1, 2021-02-10T11:26:35.343 INFO:teuthology.orchestra.run.gibba002.stdout: "uuid": "64e3ccc8-9c69-4d7c-865e-af8e83822635", 2021-02-10T11:26:35.343 INFO:teuthology.orchestra.run.gibba002.stdout: "up": 1, 2021-02-10T11:26:35.343 INFO:teuthology.orchestra.run.gibba002.stdout: "in": 1, 2021-02-10T11:26:35.343 INFO:teuthology.orchestra.run.gibba002.stdout: "weight": 1, 2021-02-10T11:26:35.344 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_affinity": 1, 2021-02-10T11:26:35.344 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_begin": 0, 2021-02-10T11:26:35.344 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_end": 0, 2021-02-10T11:26:35.344 INFO:teuthology.orchestra.run.gibba002.stdout: "up_from": 13, 2021-02-10T11:26:35.344 INFO:teuthology.orchestra.run.gibba002.stdout: "up_thru": 62, 2021-02-10T11:26:35.344 INFO:teuthology.orchestra.run.gibba002.stdout: "down_at": 0, 2021-02-10T11:26:35.344 INFO:teuthology.orchestra.run.gibba002.stdout: "lost_at": 0, 2021-02-10T11:26:35.344 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addrs": { 2021-02-10T11:26:35.344 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:26:35.344 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:26:35.344 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:26:35.344 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6819", 2021-02-10T11:26:35.344 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36087 2021-02-10T11:26:35.345 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:26:35.345 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:26:35.345 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:26:35.345 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6821", 2021-02-10T11:26:35.345 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36087 2021-02-10T11:26:35.345 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:26:35.345 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:26:35.345 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:26:35.345 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addrs": { 2021-02-10T11:26:35.345 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:26:35.345 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:26:35.345 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:26:35.345 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6823", 2021-02-10T11:26:35.345 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36087 2021-02-10T11:26:35.346 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:26:35.346 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:26:35.346 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:26:35.346 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6825", 2021-02-10T11:26:35.346 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36087 2021-02-10T11:26:35.346 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:26:35.346 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:26:35.346 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:26:35.346 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addrs": { 2021-02-10T11:26:35.346 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:26:35.346 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:26:35.346 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:26:35.346 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6831", 2021-02-10T11:26:35.347 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36087 2021-02-10T11:26:35.347 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:26:35.347 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:26:35.347 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:26:35.347 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6833", 2021-02-10T11:26:35.347 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36087 2021-02-10T11:26:35.347 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:26:35.347 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:26:35.347 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:26:35.347 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addrs": { 2021-02-10T11:26:35.347 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:26:35.347 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:26:35.347 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:26:35.347 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6827", 2021-02-10T11:26:35.348 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36087 2021-02-10T11:26:35.348 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:26:35.348 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:26:35.348 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:26:35.348 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6829", 2021-02-10T11:26:35.348 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36087 2021-02-10T11:26:35.348 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:26:35.348 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:26:35.348 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:26:35.348 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addr": "172.21.2.102:6821/36087", 2021-02-10T11:26:35.348 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addr": "172.21.2.102:6825/36087", 2021-02-10T11:26:35.348 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addr": "172.21.2.102:6833/36087", 2021-02-10T11:26:35.348 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addr": "172.21.2.102:6829/36087", 2021-02-10T11:26:35.349 INFO:teuthology.orchestra.run.gibba002.stdout: "state": [ 2021-02-10T11:26:35.349 INFO:teuthology.orchestra.run.gibba002.stdout: "exists", 2021-02-10T11:26:35.349 INFO:teuthology.orchestra.run.gibba002.stdout: "up" 2021-02-10T11:26:35.349 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:26:35.349 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:26:35.349 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:26:35.349 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 2, 2021-02-10T11:26:35.349 INFO:teuthology.orchestra.run.gibba002.stdout: "uuid": "153700af-747f-481f-bdf4-d6072d3a842f", 2021-02-10T11:26:35.349 INFO:teuthology.orchestra.run.gibba002.stdout: "up": 1, 2021-02-10T11:26:35.349 INFO:teuthology.orchestra.run.gibba002.stdout: "in": 1, 2021-02-10T11:26:35.349 INFO:teuthology.orchestra.run.gibba002.stdout: "weight": 1, 2021-02-10T11:26:35.349 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_affinity": 1, 2021-02-10T11:26:35.349 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_begin": 0, 2021-02-10T11:26:35.350 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_end": 0, 2021-02-10T11:26:35.350 INFO:teuthology.orchestra.run.gibba002.stdout: "up_from": 13, 2021-02-10T11:26:35.350 INFO:teuthology.orchestra.run.gibba002.stdout: "up_thru": 55, 2021-02-10T11:26:35.350 INFO:teuthology.orchestra.run.gibba002.stdout: "down_at": 0, 2021-02-10T11:26:35.350 INFO:teuthology.orchestra.run.gibba002.stdout: "lost_at": 0, 2021-02-10T11:26:35.350 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addrs": { 2021-02-10T11:26:35.350 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:26:35.350 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:26:35.350 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:26:35.350 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6810", 2021-02-10T11:26:35.350 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36089 2021-02-10T11:26:35.350 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:26:35.350 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:26:35.351 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:26:35.351 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6811", 2021-02-10T11:26:35.351 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36089 2021-02-10T11:26:35.351 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:26:35.351 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:26:35.351 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:26:35.351 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addrs": { 2021-02-10T11:26:35.351 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:26:35.351 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:26:35.351 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:26:35.351 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6812", 2021-02-10T11:26:35.351 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36089 2021-02-10T11:26:35.351 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:26:35.351 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:26:35.352 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:26:35.352 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6813", 2021-02-10T11:26:35.352 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36089 2021-02-10T11:26:35.352 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:26:35.352 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:26:35.352 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:26:35.352 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addrs": { 2021-02-10T11:26:35.352 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:26:35.352 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:26:35.352 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:26:35.352 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6816", 2021-02-10T11:26:35.352 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36089 2021-02-10T11:26:35.352 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:26:35.353 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:26:35.353 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:26:35.353 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6817", 2021-02-10T11:26:35.353 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36089 2021-02-10T11:26:35.353 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:26:35.353 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:26:35.353 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:26:35.353 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addrs": { 2021-02-10T11:26:35.353 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:26:35.353 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:26:35.353 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:26:35.353 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6814", 2021-02-10T11:26:35.353 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36089 2021-02-10T11:26:35.353 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:26:35.354 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:26:35.354 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:26:35.354 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6815", 2021-02-10T11:26:35.354 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36089 2021-02-10T11:26:35.354 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:26:35.354 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:26:35.354 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:26:35.354 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addr": "172.21.2.102:6811/36089", 2021-02-10T11:26:35.354 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addr": "172.21.2.102:6813/36089", 2021-02-10T11:26:35.354 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addr": "172.21.2.102:6817/36089", 2021-02-10T11:26:35.354 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addr": "172.21.2.102:6815/36089", 2021-02-10T11:26:35.354 INFO:teuthology.orchestra.run.gibba002.stdout: "state": [ 2021-02-10T11:26:35.355 INFO:teuthology.orchestra.run.gibba002.stdout: "exists", 2021-02-10T11:26:35.355 INFO:teuthology.orchestra.run.gibba002.stdout: "up" 2021-02-10T11:26:35.355 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:26:35.355 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:26:35.355 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:26:35.355 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 3, 2021-02-10T11:26:35.355 INFO:teuthology.orchestra.run.gibba002.stdout: "uuid": "cbc678f4-83a9-45e6-8830-2f7c232b1ea7", 2021-02-10T11:26:35.355 INFO:teuthology.orchestra.run.gibba002.stdout: "up": 1, 2021-02-10T11:26:35.356 INFO:teuthology.orchestra.run.gibba002.stdout: "in": 1, 2021-02-10T11:26:35.356 INFO:teuthology.orchestra.run.gibba002.stdout: "weight": 1, 2021-02-10T11:26:35.356 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_affinity": 1, 2021-02-10T11:26:35.356 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_begin": 0, 2021-02-10T11:26:35.356 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_end": 0, 2021-02-10T11:26:35.356 INFO:teuthology.orchestra.run.gibba002.stdout: "up_from": 13, 2021-02-10T11:26:35.356 INFO:teuthology.orchestra.run.gibba002.stdout: "up_thru": 46, 2021-02-10T11:26:35.357 INFO:teuthology.orchestra.run.gibba002.stdout: "down_at": 0, 2021-02-10T11:26:35.357 INFO:teuthology.orchestra.run.gibba002.stdout: "lost_at": 0, 2021-02-10T11:26:35.357 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addrs": { 2021-02-10T11:26:35.357 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:26:35.357 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:26:35.357 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:26:35.357 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6802", 2021-02-10T11:26:35.357 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36090 2021-02-10T11:26:35.357 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:26:35.357 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:26:35.357 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:26:35.357 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6803", 2021-02-10T11:26:35.357 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36090 2021-02-10T11:26:35.358 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:26:35.358 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:26:35.358 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:26:35.358 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addrs": { 2021-02-10T11:26:35.358 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:26:35.358 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:26:35.358 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:26:35.358 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6804", 2021-02-10T11:26:35.358 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36090 2021-02-10T11:26:35.358 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:26:35.358 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:26:35.359 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:26:35.359 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6805", 2021-02-10T11:26:35.359 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36090 2021-02-10T11:26:35.359 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:26:35.359 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:26:35.359 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:26:35.359 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addrs": { 2021-02-10T11:26:35.359 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:26:35.359 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:26:35.359 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:26:35.359 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6808", 2021-02-10T11:26:35.359 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36090 2021-02-10T11:26:35.360 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:26:35.360 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:26:35.360 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:26:35.360 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6809", 2021-02-10T11:26:35.360 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36090 2021-02-10T11:26:35.360 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:26:35.360 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:26:35.360 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:26:35.360 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addrs": { 2021-02-10T11:26:35.360 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:26:35.360 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:26:35.360 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:26:35.360 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6806", 2021-02-10T11:26:35.361 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36090 2021-02-10T11:26:35.361 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:26:35.361 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:26:35.361 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:26:35.361 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6807", 2021-02-10T11:26:35.361 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36090 2021-02-10T11:26:35.361 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:26:35.361 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:26:35.361 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:26:35.361 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addr": "172.21.2.102:6803/36090", 2021-02-10T11:26:35.361 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addr": "172.21.2.102:6805/36090", 2021-02-10T11:26:35.361 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addr": "172.21.2.102:6809/36090", 2021-02-10T11:26:35.362 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addr": "172.21.2.102:6807/36090", 2021-02-10T11:26:35.362 INFO:teuthology.orchestra.run.gibba002.stdout: "state": [ 2021-02-10T11:26:35.362 INFO:teuthology.orchestra.run.gibba002.stdout: "exists", 2021-02-10T11:26:35.362 INFO:teuthology.orchestra.run.gibba002.stdout: "up" 2021-02-10T11:26:35.362 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:26:35.362 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:26:35.362 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:26:35.362 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 4, 2021-02-10T11:26:35.362 INFO:teuthology.orchestra.run.gibba002.stdout: "uuid": "c01e4008-b258-4f10-afd2-9e7f37214351", 2021-02-10T11:26:35.362 INFO:teuthology.orchestra.run.gibba002.stdout: "up": 1, 2021-02-10T11:26:35.362 INFO:teuthology.orchestra.run.gibba002.stdout: "in": 1, 2021-02-10T11:26:35.362 INFO:teuthology.orchestra.run.gibba002.stdout: "weight": 1, 2021-02-10T11:26:35.362 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_affinity": 1, 2021-02-10T11:26:35.363 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_begin": 0, 2021-02-10T11:26:35.363 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_end": 0, 2021-02-10T11:26:35.363 INFO:teuthology.orchestra.run.gibba002.stdout: "up_from": 13, 2021-02-10T11:26:35.363 INFO:teuthology.orchestra.run.gibba002.stdout: "up_thru": 64, 2021-02-10T11:26:35.363 INFO:teuthology.orchestra.run.gibba002.stdout: "down_at": 0, 2021-02-10T11:26:35.363 INFO:teuthology.orchestra.run.gibba002.stdout: "lost_at": 0, 2021-02-10T11:26:35.363 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addrs": { 2021-02-10T11:26:35.363 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:26:35.363 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:26:35.363 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:26:35.363 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6808", 2021-02-10T11:26:35.363 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36368 2021-02-10T11:26:35.363 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:26:35.364 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:26:35.364 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:26:35.364 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6811", 2021-02-10T11:26:35.364 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36368 2021-02-10T11:26:35.364 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:26:35.364 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:26:35.364 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:26:35.364 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addrs": { 2021-02-10T11:26:35.364 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:26:35.364 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:26:35.364 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:26:35.364 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6814", 2021-02-10T11:26:35.364 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36368 2021-02-10T11:26:35.364 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:26:35.365 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:26:35.365 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:26:35.365 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6818", 2021-02-10T11:26:35.365 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36368 2021-02-10T11:26:35.367 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:26:35.367 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:26:35.367 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:26:35.367 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addrs": { 2021-02-10T11:26:35.367 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:26:35.367 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:26:35.367 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:26:35.367 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6826", 2021-02-10T11:26:35.367 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36368 2021-02-10T11:26:35.367 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:26:35.367 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:26:35.367 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:26:35.367 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6828", 2021-02-10T11:26:35.368 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36368 2021-02-10T11:26:35.368 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:26:35.368 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:26:35.368 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:26:35.368 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addrs": { 2021-02-10T11:26:35.368 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:26:35.368 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:26:35.368 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:26:35.368 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6820", 2021-02-10T11:26:35.368 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36368 2021-02-10T11:26:35.368 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:26:35.368 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:26:35.368 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:26:35.369 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6824", 2021-02-10T11:26:35.369 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36368 2021-02-10T11:26:35.369 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:26:35.369 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:26:35.369 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:26:35.369 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addr": "172.21.2.113:6811/36368", 2021-02-10T11:26:35.369 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addr": "172.21.2.113:6818/36368", 2021-02-10T11:26:35.369 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addr": "172.21.2.113:6828/36368", 2021-02-10T11:26:35.369 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addr": "172.21.2.113:6824/36368", 2021-02-10T11:26:35.369 INFO:teuthology.orchestra.run.gibba002.stdout: "state": [ 2021-02-10T11:26:35.369 INFO:teuthology.orchestra.run.gibba002.stdout: "exists", 2021-02-10T11:26:35.369 INFO:teuthology.orchestra.run.gibba002.stdout: "up" 2021-02-10T11:26:35.370 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:26:35.370 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:26:35.370 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:26:35.370 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 5, 2021-02-10T11:26:35.370 INFO:teuthology.orchestra.run.gibba002.stdout: "uuid": "551f6007-daf8-4a98-8ee0-c2b16bcccb5e", 2021-02-10T11:26:35.370 INFO:teuthology.orchestra.run.gibba002.stdout: "up": 1, 2021-02-10T11:26:35.370 INFO:teuthology.orchestra.run.gibba002.stdout: "in": 1, 2021-02-10T11:26:35.370 INFO:teuthology.orchestra.run.gibba002.stdout: "weight": 1, 2021-02-10T11:26:35.370 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_affinity": 1, 2021-02-10T11:26:35.370 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_begin": 0, 2021-02-10T11:26:35.370 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_end": 0, 2021-02-10T11:26:35.371 INFO:teuthology.orchestra.run.gibba002.stdout: "up_from": 13, 2021-02-10T11:26:35.371 INFO:teuthology.orchestra.run.gibba002.stdout: "up_thru": 64, 2021-02-10T11:26:35.371 INFO:teuthology.orchestra.run.gibba002.stdout: "down_at": 0, 2021-02-10T11:26:35.371 INFO:teuthology.orchestra.run.gibba002.stdout: "lost_at": 0, 2021-02-10T11:26:35.372 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addrs": { 2021-02-10T11:26:35.372 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:26:35.372 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:26:35.372 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:26:35.373 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6802", 2021-02-10T11:26:35.373 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36366 2021-02-10T11:26:35.373 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:26:35.373 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:26:35.373 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:26:35.373 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6805", 2021-02-10T11:26:35.374 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36366 2021-02-10T11:26:35.374 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:26:35.374 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:26:35.374 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:26:35.374 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addrs": { 2021-02-10T11:26:35.374 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:26:35.374 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:26:35.374 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:26:35.374 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6807", 2021-02-10T11:26:35.374 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36366 2021-02-10T11:26:35.375 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:26:35.375 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:26:35.375 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:26:35.375 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6810", 2021-02-10T11:26:35.375 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36366 2021-02-10T11:26:35.375 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:26:35.375 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:26:35.375 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:26:35.375 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addrs": { 2021-02-10T11:26:35.375 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:26:35.375 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:26:35.375 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:26:35.375 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6819", 2021-02-10T11:26:35.376 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36366 2021-02-10T11:26:35.376 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:26:35.376 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:26:35.376 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:26:35.376 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6822", 2021-02-10T11:26:35.376 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36366 2021-02-10T11:26:35.376 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:26:35.376 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:26:35.376 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:26:35.377 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addrs": { 2021-02-10T11:26:35.377 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:26:35.377 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:26:35.377 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:26:35.377 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6812", 2021-02-10T11:26:35.377 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36366 2021-02-10T11:26:35.377 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:26:35.377 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:26:35.377 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:26:35.377 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6815", 2021-02-10T11:26:35.377 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36366 2021-02-10T11:26:35.377 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:26:35.377 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:26:35.377 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:26:35.378 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addr": "172.21.2.113:6805/36366", 2021-02-10T11:26:35.378 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addr": "172.21.2.113:6810/36366", 2021-02-10T11:26:35.378 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addr": "172.21.2.113:6822/36366", 2021-02-10T11:26:35.378 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addr": "172.21.2.113:6815/36366", 2021-02-10T11:26:35.378 INFO:teuthology.orchestra.run.gibba002.stdout: "state": [ 2021-02-10T11:26:35.378 INFO:teuthology.orchestra.run.gibba002.stdout: "exists", 2021-02-10T11:26:35.378 INFO:teuthology.orchestra.run.gibba002.stdout: "up" 2021-02-10T11:26:35.378 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:26:35.378 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:26:35.378 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:26:35.378 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 6, 2021-02-10T11:26:35.378 INFO:teuthology.orchestra.run.gibba002.stdout: "uuid": "bf67c040-0e50-48be-b102-c3943bc9ebef", 2021-02-10T11:26:35.379 INFO:teuthology.orchestra.run.gibba002.stdout: "up": 1, 2021-02-10T11:26:35.379 INFO:teuthology.orchestra.run.gibba002.stdout: "in": 1, 2021-02-10T11:26:35.379 INFO:teuthology.orchestra.run.gibba002.stdout: "weight": 1, 2021-02-10T11:26:35.379 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_affinity": 1, 2021-02-10T11:26:35.379 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_begin": 0, 2021-02-10T11:26:35.379 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_end": 0, 2021-02-10T11:26:35.379 INFO:teuthology.orchestra.run.gibba002.stdout: "up_from": 13, 2021-02-10T11:26:35.379 INFO:teuthology.orchestra.run.gibba002.stdout: "up_thru": 64, 2021-02-10T11:26:35.379 INFO:teuthology.orchestra.run.gibba002.stdout: "down_at": 0, 2021-02-10T11:26:35.379 INFO:teuthology.orchestra.run.gibba002.stdout: "lost_at": 0, 2021-02-10T11:26:35.379 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addrs": { 2021-02-10T11:26:35.380 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:26:35.380 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:26:35.380 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:26:35.380 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6800", 2021-02-10T11:26:35.380 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36367 2021-02-10T11:26:35.380 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:26:35.380 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:26:35.380 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:26:35.380 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6801", 2021-02-10T11:26:35.380 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36367 2021-02-10T11:26:35.380 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:26:35.380 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:26:35.381 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:26:35.381 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addrs": { 2021-02-10T11:26:35.381 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:26:35.381 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:26:35.381 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:26:35.381 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6803", 2021-02-10T11:26:35.381 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36367 2021-02-10T11:26:35.381 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:26:35.381 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:26:35.381 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:26:35.381 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6804", 2021-02-10T11:26:35.381 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36367 2021-02-10T11:26:35.382 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:26:35.382 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:26:35.382 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:26:35.382 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addrs": { 2021-02-10T11:26:35.382 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:26:35.382 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:26:35.382 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:26:35.382 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6813", 2021-02-10T11:26:35.382 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36367 2021-02-10T11:26:35.382 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:26:35.382 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:26:35.382 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:26:35.383 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6816", 2021-02-10T11:26:35.383 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36367 2021-02-10T11:26:35.383 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:26:35.383 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:26:35.383 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:26:35.383 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addrs": { 2021-02-10T11:26:35.383 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:26:35.383 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:26:35.383 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:26:35.383 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6806", 2021-02-10T11:26:35.383 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36367 2021-02-10T11:26:35.383 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:26:35.383 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:26:35.384 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:26:35.384 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6809", 2021-02-10T11:26:35.384 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36367 2021-02-10T11:26:35.384 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:26:35.384 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:26:35.384 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:26:35.384 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addr": "172.21.2.113:6801/36367", 2021-02-10T11:26:35.384 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addr": "172.21.2.113:6804/36367", 2021-02-10T11:26:35.384 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addr": "172.21.2.113:6816/36367", 2021-02-10T11:26:35.384 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addr": "172.21.2.113:6809/36367", 2021-02-10T11:26:35.384 INFO:teuthology.orchestra.run.gibba002.stdout: "state": [ 2021-02-10T11:26:35.384 INFO:teuthology.orchestra.run.gibba002.stdout: "exists", 2021-02-10T11:26:35.384 INFO:teuthology.orchestra.run.gibba002.stdout: "up" 2021-02-10T11:26:35.385 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:26:35.385 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:26:35.385 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:26:35.385 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 7, 2021-02-10T11:26:35.385 INFO:teuthology.orchestra.run.gibba002.stdout: "uuid": "f5461f80-018d-428f-be89-2eb2bcbe7bb6", 2021-02-10T11:26:35.385 INFO:teuthology.orchestra.run.gibba002.stdout: "up": 1, 2021-02-10T11:26:35.385 INFO:teuthology.orchestra.run.gibba002.stdout: "in": 1, 2021-02-10T11:26:35.385 INFO:teuthology.orchestra.run.gibba002.stdout: "weight": 1, 2021-02-10T11:26:35.385 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_affinity": 1, 2021-02-10T11:26:35.385 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_begin": 0, 2021-02-10T11:26:35.385 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_end": 0, 2021-02-10T11:26:35.385 INFO:teuthology.orchestra.run.gibba002.stdout: "up_from": 13, 2021-02-10T11:26:35.385 INFO:teuthology.orchestra.run.gibba002.stdout: "up_thru": 64, 2021-02-10T11:26:35.386 INFO:teuthology.orchestra.run.gibba002.stdout: "down_at": 0, 2021-02-10T11:26:35.386 INFO:teuthology.orchestra.run.gibba002.stdout: "lost_at": 0, 2021-02-10T11:26:35.386 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addrs": { 2021-02-10T11:26:35.386 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:26:35.386 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:26:35.386 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:26:35.386 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6817", 2021-02-10T11:26:35.386 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36365 2021-02-10T11:26:35.386 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:26:35.386 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:26:35.386 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:26:35.386 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6821", 2021-02-10T11:26:35.386 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36365 2021-02-10T11:26:35.387 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:26:35.387 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:26:35.387 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:26:35.387 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addrs": { 2021-02-10T11:26:35.387 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:26:35.387 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:26:35.387 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:26:35.387 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6823", 2021-02-10T11:26:35.387 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36365 2021-02-10T11:26:35.387 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:26:35.387 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:26:35.387 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:26:35.387 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6825", 2021-02-10T11:26:35.388 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36365 2021-02-10T11:26:35.388 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:26:35.388 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:26:35.388 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:26:35.388 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addrs": { 2021-02-10T11:26:35.388 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:26:35.388 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:26:35.388 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:26:35.388 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6830", 2021-02-10T11:26:35.388 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36365 2021-02-10T11:26:35.388 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:26:35.388 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:26:35.388 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:26:35.388 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6831", 2021-02-10T11:26:35.389 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36365 2021-02-10T11:26:35.389 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:26:35.389 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:26:35.389 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:26:35.389 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addrs": { 2021-02-10T11:26:35.389 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:26:35.389 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:26:35.389 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:26:35.389 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6827", 2021-02-10T11:26:35.389 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36365 2021-02-10T11:26:35.389 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:26:35.389 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:26:35.389 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:26:35.390 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6829", 2021-02-10T11:26:35.390 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36365 2021-02-10T11:26:35.390 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:26:35.390 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:26:35.390 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:26:35.390 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addr": "172.21.2.113:6821/36365", 2021-02-10T11:26:35.390 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addr": "172.21.2.113:6825/36365", 2021-02-10T11:26:35.390 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addr": "172.21.2.113:6831/36365", 2021-02-10T11:26:35.390 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addr": "172.21.2.113:6829/36365", 2021-02-10T11:26:35.390 INFO:teuthology.orchestra.run.gibba002.stdout: "state": [ 2021-02-10T11:26:35.390 INFO:teuthology.orchestra.run.gibba002.stdout: "exists", 2021-02-10T11:26:35.390 INFO:teuthology.orchestra.run.gibba002.stdout: "up" 2021-02-10T11:26:35.390 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:26:35.391 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:26:35.391 INFO:teuthology.orchestra.run.gibba002.stdout: ], 2021-02-10T11:26:35.391 INFO:teuthology.orchestra.run.gibba002.stdout: "osd_xinfo": [ 2021-02-10T11:26:35.391 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:26:35.391 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 0, 2021-02-10T11:26:35.391 INFO:teuthology.orchestra.run.gibba002.stdout: "down_stamp": "0.000000", 2021-02-10T11:26:35.391 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_probability": 0, 2021-02-10T11:26:35.391 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_interval": 0, 2021-02-10T11:26:35.391 INFO:teuthology.orchestra.run.gibba002.stdout: "features": 4611087854035861503, 2021-02-10T11:26:35.391 INFO:teuthology.orchestra.run.gibba002.stdout: "old_weight": 0 2021-02-10T11:26:35.391 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:26:35.391 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:26:35.391 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 1, 2021-02-10T11:26:35.392 INFO:teuthology.orchestra.run.gibba002.stdout: "down_stamp": "0.000000", 2021-02-10T11:26:35.392 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_probability": 0, 2021-02-10T11:26:35.392 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_interval": 0, 2021-02-10T11:26:35.392 INFO:teuthology.orchestra.run.gibba002.stdout: "features": 4611087854035861503, 2021-02-10T11:26:35.392 INFO:teuthology.orchestra.run.gibba002.stdout: "old_weight": 0 2021-02-10T11:26:35.392 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:26:35.392 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:26:35.392 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 2, 2021-02-10T11:26:35.392 INFO:teuthology.orchestra.run.gibba002.stdout: "down_stamp": "0.000000", 2021-02-10T11:26:35.392 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_probability": 0, 2021-02-10T11:26:35.392 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_interval": 0, 2021-02-10T11:26:35.392 INFO:teuthology.orchestra.run.gibba002.stdout: "features": 4611087854035861503, 2021-02-10T11:26:35.392 INFO:teuthology.orchestra.run.gibba002.stdout: "old_weight": 0 2021-02-10T11:26:35.392 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:26:35.393 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:26:35.393 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 3, 2021-02-10T11:26:35.393 INFO:teuthology.orchestra.run.gibba002.stdout: "down_stamp": "0.000000", 2021-02-10T11:26:35.393 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_probability": 0, 2021-02-10T11:26:35.393 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_interval": 0, 2021-02-10T11:26:35.393 INFO:teuthology.orchestra.run.gibba002.stdout: "features": 4611087854035861503, 2021-02-10T11:26:35.393 INFO:teuthology.orchestra.run.gibba002.stdout: "old_weight": 0 2021-02-10T11:26:35.393 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:26:35.393 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:26:35.393 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 4, 2021-02-10T11:26:35.393 INFO:teuthology.orchestra.run.gibba002.stdout: "down_stamp": "0.000000", 2021-02-10T11:26:35.393 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_probability": 0, 2021-02-10T11:26:35.393 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_interval": 0, 2021-02-10T11:26:35.393 INFO:teuthology.orchestra.run.gibba002.stdout: "features": 4611087854035861503, 2021-02-10T11:26:35.394 INFO:teuthology.orchestra.run.gibba002.stdout: "old_weight": 0 2021-02-10T11:26:35.394 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:26:35.394 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:26:35.394 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 5, 2021-02-10T11:26:35.394 INFO:teuthology.orchestra.run.gibba002.stdout: "down_stamp": "0.000000", 2021-02-10T11:26:35.394 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_probability": 0, 2021-02-10T11:26:35.394 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_interval": 0, 2021-02-10T11:26:35.394 INFO:teuthology.orchestra.run.gibba002.stdout: "features": 4611087854035861503, 2021-02-10T11:26:35.394 INFO:teuthology.orchestra.run.gibba002.stdout: "old_weight": 0 2021-02-10T11:26:35.394 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:26:35.394 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:26:35.394 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 6, 2021-02-10T11:26:35.394 INFO:teuthology.orchestra.run.gibba002.stdout: "down_stamp": "0.000000", 2021-02-10T11:26:35.395 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_probability": 0, 2021-02-10T11:26:35.395 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_interval": 0, 2021-02-10T11:26:35.395 INFO:teuthology.orchestra.run.gibba002.stdout: "features": 4611087854035861503, 2021-02-10T11:26:35.395 INFO:teuthology.orchestra.run.gibba002.stdout: "old_weight": 0 2021-02-10T11:26:35.395 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:26:35.395 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:26:35.395 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 7, 2021-02-10T11:26:35.395 INFO:teuthology.orchestra.run.gibba002.stdout: "down_stamp": "0.000000", 2021-02-10T11:26:35.395 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_probability": 0, 2021-02-10T11:26:35.395 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_interval": 0, 2021-02-10T11:26:35.395 INFO:teuthology.orchestra.run.gibba002.stdout: "features": 4611087854035861503, 2021-02-10T11:26:35.395 INFO:teuthology.orchestra.run.gibba002.stdout: "old_weight": 0 2021-02-10T11:26:35.395 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:26:35.395 INFO:teuthology.orchestra.run.gibba002.stdout: ], 2021-02-10T11:26:35.396 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_upmap": [], 2021-02-10T11:26:35.396 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_upmap_items": [], 2021-02-10T11:26:35.396 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_temp": [], 2021-02-10T11:26:35.396 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_temp": [], 2021-02-10T11:26:35.396 INFO:teuthology.orchestra.run.gibba002.stdout: "blacklist": {}, 2021-02-10T11:26:35.396 INFO:teuthology.orchestra.run.gibba002.stdout: "erasure_code_profiles": { 2021-02-10T11:26:35.396 INFO:teuthology.orchestra.run.gibba002.stdout: "default": { 2021-02-10T11:26:35.396 INFO:teuthology.orchestra.run.gibba002.stdout: "crush-failure-domain": "osd", 2021-02-10T11:26:35.396 INFO:teuthology.orchestra.run.gibba002.stdout: "k": "2", 2021-02-10T11:26:35.396 INFO:teuthology.orchestra.run.gibba002.stdout: "m": "1", 2021-02-10T11:26:35.396 INFO:teuthology.orchestra.run.gibba002.stdout: "plugin": "jerasure", 2021-02-10T11:26:35.396 INFO:teuthology.orchestra.run.gibba002.stdout: "ruleset-failure-domain": "osd", 2021-02-10T11:26:35.396 INFO:teuthology.orchestra.run.gibba002.stdout: "technique": "reed_sol_van" 2021-02-10T11:26:35.397 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:26:35.397 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:26:35.397 INFO:teuthology.orchestra.run.gibba002.stdout: "removed_snaps_queue": [], 2021-02-10T11:26:35.397 INFO:teuthology.orchestra.run.gibba002.stdout: "new_removed_snaps": [], 2021-02-10T11:26:35.397 INFO:teuthology.orchestra.run.gibba002.stdout: "new_purged_snaps": [], 2021-02-10T11:26:35.397 INFO:teuthology.orchestra.run.gibba002.stdout: "crush_node_flags": {}, 2021-02-10T11:26:35.397 INFO:teuthology.orchestra.run.gibba002.stdout: "device_class_flags": {} 2021-02-10T11:26:35.397 INFO:teuthology.orchestra.run.gibba002.stdout:} 2021-02-10T11:26:35.398 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:35.320 7f83828db700 1 -- 172.21.2.102:0/3624204280 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f835c018700 msgr2=0x7f835c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:35.398 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:35.320 7f83828db700 1 --2- 172.21.2.102:0/3624204280 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f835c018700 0x7f835c01abc0 crc :-1 s=READY pgs=732 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:35.399 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:35.320 7f83828db700 1 -- 172.21.2.102:0/3624204280 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f837c093060 msgr2=0x7f837c19a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:35.399 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:35.320 7f83828db700 1 --2- 172.21.2.102:0/3624204280 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f837c093060 0x7f837c19a890 secure :-1 s=READY pgs=1614 cs=0 l=1 rev1=1 rx=0x7f836c00b560 tx=0x7f836c00b900).stop 2021-02-10T11:26:35.399 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:35.320 7f83828db700 1 -- 172.21.2.102:0/3624204280 shutdown_connections 2021-02-10T11:26:35.399 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:35.320 7f83828db700 1 --2- 172.21.2.102:0/3624204280 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f837c19b2b0 0x7f837c0b1fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:35.399 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:35.320 7f83828db700 1 --2- 172.21.2.102:0/3624204280 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f835c018700 0x7f835c01abc0 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:35.399 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:35.320 7f83828db700 1 --2- 172.21.2.102:0/3624204280 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f837c19aeb0 0x7f837c0b1990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:35.399 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:35.320 7f83828db700 1 --2- 172.21.2.102:0/3624204280 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f837c093060 0x7f837c19a890 unknown :-1 s=CLOSED pgs=1614 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:35.399 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:35.320 7f83828db700 1 -- 172.21.2.102:0/3624204280 >> 172.21.2.102:0/3624204280 conn(0x7f837c19f880 msgr2=0x7f837c199a80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:26:35.399 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:35.320 7f83828db700 1 -- 172.21.2.102:0/3624204280 shutdown_connections 2021-02-10T11:26:35.399 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:35.320 7f83828db700 1 -- 172.21.2.102:0/3624204280 wait complete. 2021-02-10T11:26:35.400 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:26:35.571 DEBUG:teuthology.orchestra.run.gibba002:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:26:35.574 DEBUG:teuthology.orchestra.run.gibba013:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:26:35.827 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:35.810 7fe0b4ead700 1 Processor -- start 2021-02-10T11:26:35.827 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:35.811 7fe0b4ead700 1 -- start start 2021-02-10T11:26:35.827 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:35.811 7fe0b4ead700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe0b00d1870 0x7fe0b01b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:35.827 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:35.811 7fe0b4ead700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fe0b01b0a10 0x7fe0b00cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:35.827 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:35.811 7fe0b4ead700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0b00f0b10 con 0x7fe0b00d1c70 2021-02-10T11:26:35.828 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:35.811 7fe0b4ead700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0b00fe230 con 0x7fe0b01b0a10 2021-02-10T11:26:35.828 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:35.811 7fe0b4ead700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0b00e5ed0 con 0x7fe0b00d1870 2021-02-10T11:26:35.828 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:35.811 7fe0af314700 1 --1- >> v1:172.21.2.113:6789/0 conn(0x7fe0b00d1c70 0x7fe0b00cf950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.113:6789/0 says I am v1:172.21.2.102:58318/0 (socket says 172.21.2.102:58318) 2021-02-10T11:26:35.828 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:35.811 7fe0af314700 1 -- 172.21.2.102:0/3922644320 learned_addr learned my addr 172.21.2.102:0/3922644320 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:26:35.828 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:35.811 7fe0aeb13700 1 --2- 172.21.2.102:0/3922644320 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe0b00d1870 0x7fe0b01b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:35.828 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:35.812 7fe0ae312700 1 --2- 172.21.2.102:0/3922644320 >> v2:172.21.2.113:3300/0 conn(0x7fe0b01b0a10 0x7fe0b00cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:35.828 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:35.812 7fe0adb11700 1 -- 172.21.2.102:0/3922644320 <== mon.1 v1:172.21.2.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 789843028 0 0) 0x7fe0a8001510 con 0x7fe0b00d1c70 2021-02-10T11:26:35.828 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:35.812 7fe0adb11700 1 -- 172.21.2.102:0/3922644320 --> v1:172.21.2.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe09c002fe0 con 0x7fe0b00d1c70 2021-02-10T11:26:35.828 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:35.812 7fe0ae312700 1 -- 172.21.2.102:0/3922644320 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe0b00d1870 msgr2=0x7fe0b01b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:26:35.828 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:35.812 7fe0ae312700 1 --2- 172.21.2.102:0/3922644320 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe0b00d1870 0x7fe0b01b03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:35.828 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:35.812 7fe0ae312700 1 -- 172.21.2.102:0/3922644320 >> v1:172.21.2.113:6789/0 conn(0x7fe0b00d1c70 legacy=0x7fe0b00cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:35.828 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:35.812 7fe0ae312700 1 -- 172.21.2.102:0/3922644320 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0b00f0650 con 0x7fe0b01b0a10 2021-02-10T11:26:35.828 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:35.812 7fe0aeb13700 1 --2- 172.21.2.102:0/3922644320 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe0b00d1870 0x7fe0b01b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:26:35.829 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:35.812 7fe0ae312700 1 --2- 172.21.2.102:0/3922644320 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe0b01b0a10 0x7fe0b00cf330 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 rx=0x7fe0a0006650 tx=0x7fe0a0009a80).ready entity=mon.1 client_cookie=5de2a01e1b734b38 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:35.829 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:35.812 7fe0adb11700 1 -- 172.21.2.102:0/3922644320 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe0a00009b0 con 0x7fe0b01b0a10 2021-02-10T11:26:35.829 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:35.812 7fe0adb11700 1 -- 172.21.2.102:0/3922644320 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0a000c0c0 con 0x7fe0b01b0a10 2021-02-10T11:26:35.829 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:35.812 7fe0adb11700 1 -- 172.21.2.102:0/3922644320 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe0a000c550 con 0x7fe0b01b0a10 2021-02-10T11:26:35.829 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:35.813 7fe0b4ead700 1 -- 172.21.2.102:0/3922644320 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe0b01b0a10 msgr2=0x7fe0b00cf330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:35.829 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:35.813 7fe0b4ead700 1 --2- 172.21.2.102:0/3922644320 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe0b01b0a10 0x7fe0b00cf330 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 rx=0x7fe0a0006650 tx=0x7fe0a0009a80).stop 2021-02-10T11:26:35.829 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:35.813 7fe0b4ead700 1 -- 172.21.2.102:0/3922644320 shutdown_connections 2021-02-10T11:26:35.829 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:35.813 7fe0b4ead700 1 --2- 172.21.2.102:0/3922644320 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe0b01b0a10 0x7fe0b00cf330 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:35.829 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:35.813 7fe0b4ead700 1 --2- 172.21.2.102:0/3922644320 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe0b00d1870 0x7fe0b01b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:35.829 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:35.813 7fe0b4ead700 1 -- 172.21.2.102:0/3922644320 >> 172.21.2.102:0/3922644320 conn(0x7fe0b00ce450 msgr2=0x7fe0b00d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:26:35.829 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:35.813 7fe0b4ead700 1 -- 172.21.2.102:0/3922644320 shutdown_connections 2021-02-10T11:26:35.829 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:35.813 7fe0b4ead700 1 -- 172.21.2.102:0/3922644320 wait complete. 2021-02-10T11:26:35.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:35.828 7fe0b4ead700 1 Processor -- start 2021-02-10T11:26:35.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:35.828 7fe0b4ead700 1 -- start start 2021-02-10T11:26:35.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:35.828 7fe0b4ead700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe0b00f0280 0x7fe0b00a2310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:35.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:35.828 7fe0b4ead700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe0b00f0680 0x7fe0b00a2930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:35.832 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:35.828 7fe0b4ead700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe0b00ec180 0x7fe0b00ec580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:35.832 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:35.828 7fe0b4ead700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0b00efd00 con 0x7fe0b00f0680 2021-02-10T11:26:35.832 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:35.828 7fe0b4ead700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0b00ecba0 con 0x7fe0b00ec180 2021-02-10T11:26:35.832 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:35.828 7fe0b4ead700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0b00ecd80 con 0x7fe0b00f0280 2021-02-10T11:26:35.832 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:35.828 7fe0ae312700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe0b00f0680 0x7fe0b00a2930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:35.832 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:35.828 7fe0ae312700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe0b00f0680 0x7fe0b00a2930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:58772/0 (socket says 172.21.2.102:58772) 2021-02-10T11:26:35.832 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:35.828 7fe0ae312700 1 -- 172.21.2.102:0/2983423278 learned_addr learned my addr 172.21.2.102:0/2983423278 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:26:35.832 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:35.829 7fe0af314700 1 --2- 172.21.2.102:0/2983423278 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe0b00ec180 0x7fe0b00ec580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:35.832 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:35.829 7fe0aeb13700 1 --2- 172.21.2.102:0/2983423278 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe0b00f0280 0x7fe0b00a2310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:35.832 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:35.829 7fe0ae312700 1 -- 172.21.2.102:0/2983423278 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe0b00f0280 msgr2=0x7fe0b00a2310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:35.832 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:35.829 7fe0ae312700 1 --2- 172.21.2.102:0/2983423278 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe0b00f0280 0x7fe0b00a2310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:35.832 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:35.829 7fe0ae312700 1 -- 172.21.2.102:0/2983423278 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe0b00ec180 msgr2=0x7fe0b00ec580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:35.832 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:35.829 7fe0ae312700 1 --2- 172.21.2.102:0/2983423278 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe0b00ec180 0x7fe0b00ec580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:35.832 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:35.829 7fe0ae312700 1 -- 172.21.2.102:0/2983423278 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0b00ecfa0 con 0x7fe0b00f0680 2021-02-10T11:26:35.833 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:35.829 7fe0aeb13700 1 --2- 172.21.2.102:0/2983423278 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe0b00f0280 0x7fe0b00a2310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:26:35.833 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:35.829 7fe0ae312700 1 --2- 172.21.2.102:0/2983423278 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe0b00f0680 0x7fe0b00a2930 secure :-1 s=READY pgs=1615 cs=0 l=1 rev1=1 rx=0x7fe0a0009630 tx=0x7fe0a000b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:35.833 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:35.829 7fe0a77fe700 1 -- 172.21.2.102:0/2983423278 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe0a00093a0 con 0x7fe0b00f0680 2021-02-10T11:26:35.833 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:35.829 7fe0a77fe700 1 -- 172.21.2.102:0/2983423278 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0a000bd70 con 0x7fe0b00f0680 2021-02-10T11:26:35.833 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:35.829 7fe0a77fe700 1 -- 172.21.2.102:0/2983423278 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe0a0005560 con 0x7fe0b00f0680 2021-02-10T11:26:35.833 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:35.829 7fe0b4ead700 1 -- 172.21.2.102:0/2983423278 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0b00ecfa0 con 0x7fe0b00f0680 2021-02-10T11:26:35.833 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:35.829 7fe0b4ead700 1 -- 172.21.2.102:0/2983423278 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0b00ed530 con 0x7fe0b00f0680 2021-02-10T11:26:35.833 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:35.830 7fe0a77fe700 1 -- 172.21.2.102:0/2983423278 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fe0a000a000 con 0x7fe0b00f0680 2021-02-10T11:26:35.833 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:35.830 7fe0a77fe700 1 --2- 172.21.2.102:0/2983423278 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe0940186c0 0x7fe09401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:35.833 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:35.830 7fe0a77fe700 1 -- 172.21.2.102:0/2983423278 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fe0a0032560 con 0x7fe0b00f0680 2021-02-10T11:26:35.833 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:35.830 7fe0aeb13700 1 --2- 172.21.2.102:0/2983423278 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe0940186c0 0x7fe09401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:35.833 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:35.830 7fe0aeb13700 1 --2- 172.21.2.102:0/2983423278 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe0940186c0 0x7fe09401abc0 crc :-1 s=READY pgs=733 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:35.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:35.861 7fe0b4ead700 1 -- 172.21.2.102:0/2983423278 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0b00ebc60 con 0x7fe0b00f0680 2021-02-10T11:26:35.863 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:35.864 7fe0a77fe700 1 -- 172.21.2.102:0/2983423278 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fe0a0029c30 con 0x7fe0b00f0680 2021-02-10T11:26:35.994 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:35.995 7fe0b4ead700 1 -- 172.21.2.102:0/2983423278 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe0b00ebc60 con 0x7fe0b00f0680 2021-02-10T11:26:35.995 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:35.995 7fe0a77fe700 1 -- 172.21.2.102:0/2983423278 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) v1 ==== 94+0+2101 (secure 0 0 0) 0x7fe0a0029c30 con 0x7fe0b00f0680 2021-02-10T11:26:35.995 INFO:teuthology.orchestra.run.gibba002.stderr:dumped fsmap epoch 49 2021-02-10T11:26:35.996 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:26:35.996 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":49,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8412,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.102:6835/3414701760","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6834","nonce":3414701760},{"type":"v1","addr":"172.21.2.102:6835","nonce":3414701760}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":44}],"filesystems":[{"mdsmap":{"epoch":48,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:25:31.513947","modified":"2021-02-10 11:25:33.524416","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5813},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5813":{"gid":5813,"name":"b","rank":0,"incarnation":45,"state":"up:active","state_seq":3,"addr":"172.21.2.113:6833/2951398038","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6832","nonce":2951398038},{"type":"v1","addr":"172.21.2.113:6833","nonce":2951398038}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":6}]} 2021-02-10T11:26:35.997 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:35.997 7fe0b4ead700 1 -- 172.21.2.102:0/2983423278 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe0940186c0 msgr2=0x7fe09401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:35.997 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:35.997 7fe0b4ead700 1 --2- 172.21.2.102:0/2983423278 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe0940186c0 0x7fe09401abc0 crc :-1 s=READY pgs=733 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:35.997 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:35.997 7fe0b4ead700 1 -- 172.21.2.102:0/2983423278 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe0b00f0680 msgr2=0x7fe0b00a2930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:35.998 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:35.997 7fe0b4ead700 1 --2- 172.21.2.102:0/2983423278 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe0b00f0680 0x7fe0b00a2930 secure :-1 s=READY pgs=1615 cs=0 l=1 rev1=1 rx=0x7fe0a0009630 tx=0x7fe0a000b980).stop 2021-02-10T11:26:35.998 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:35.997 7fe0b4ead700 1 -- 172.21.2.102:0/2983423278 shutdown_connections 2021-02-10T11:26:35.998 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:35.997 7fe0b4ead700 1 --2- 172.21.2.102:0/2983423278 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe0940186c0 0x7fe09401abc0 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:35.998 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:35.997 7fe0b4ead700 1 --2- 172.21.2.102:0/2983423278 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe0b00ec180 0x7fe0b00ec580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:35.998 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:35.997 7fe0b4ead700 1 --2- 172.21.2.102:0/2983423278 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe0b00f0680 0x7fe0b00a2930 unknown :-1 s=CLOSED pgs=1615 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:35.998 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:35.998 7fe0b4ead700 1 --2- 172.21.2.102:0/2983423278 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe0b00f0280 0x7fe0b00a2310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:35.998 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:35.998 7fe0b4ead700 1 -- 172.21.2.102:0/2983423278 >> 172.21.2.102:0/2983423278 conn(0x7fe0b00c2130 msgr2=0x7fe0b009e3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:26:35.998 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:35.998 7fe0b4ead700 1 -- 172.21.2.102:0/2983423278 shutdown_connections 2021-02-10T11:26:35.998 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:35.998 7fe0b4ead700 1 -- 172.21.2.102:0/2983423278 wait complete. 2021-02-10T11:26:36.006 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs fail cephfs 2021-02-10T11:26:36.437 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:36.417 7fc38568a700 1 Processor -- start 2021-02-10T11:26:36.437 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:36.417 7fc38568a700 1 -- start start 2021-02-10T11:26:36.437 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:36.418 7fc38568a700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fc3800d1860 0x7fc3801b03e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:36.437 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:36.418 7fc38568a700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc3801b0a00 0x7fc3800cf320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:36.438 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:36.418 7fc38568a700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3800f0b00 con 0x7fc3800d1c60 2021-02-10T11:26:36.438 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:36.418 7fc38568a700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3800e5ec0 con 0x7fc3800d1860 2021-02-10T11:26:36.438 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:36.418 7fc38568a700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3800f0640 con 0x7fc3801b0a00 2021-02-10T11:26:36.438 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:36.418 7fc37f2bf700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fc3800d1860 0x7fc3801b03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:36.438 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:36.418 7fc37f2bf700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fc3800d1860 0x7fc3801b03e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:58780/0 (socket says 172.21.2.102:58780) 2021-02-10T11:26:36.439 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:36.418 7fc37f2bf700 1 -- 172.21.2.102:0/3162440067 learned_addr learned my addr 172.21.2.102:0/3162440067 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:26:36.439 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:36.418 7fc37f2bf700 1 -- 172.21.2.102:0/3162440067 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc3801b0a00 msgr2=0x7fc3800cf320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:26:36.439 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:36.418 7fc37f2bf700 1 --2- 172.21.2.102:0/3162440067 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc3801b0a00 0x7fc3800cf320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:36.439 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:36.418 7fc37f2bf700 1 -- 172.21.2.102:0/3162440067 >> v1:172.21.2.113:6789/0 conn(0x7fc3800d1c60 legacy=0x7fc3800cf940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:26:36.439 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:36.418 7fc37f2bf700 1 -- 172.21.2.102:0/3162440067 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3800f0860 con 0x7fc3800d1860 2021-02-10T11:26:36.439 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:36.419 7fc37f2bf700 1 --2- 172.21.2.102:0/3162440067 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc3800d1860 0x7fc3801b03e0 secure :-1 s=READY pgs=1616 cs=0 l=1 rev1=1 rx=0x7fc370006700 tx=0x7fc370009b80).ready entity=mon.0 client_cookie=d241e83c00e73d81 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:36.440 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:36.419 7fc37eabe700 1 -- 172.21.2.102:0/3162440067 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc37000c0c0 con 0x7fc3800d1860 2021-02-10T11:26:36.440 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:36.419 7fc37eabe700 1 -- 172.21.2.102:0/3162440067 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc37000c440 con 0x7fc3800d1860 2021-02-10T11:26:36.440 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:36.419 7fc37eabe700 1 -- 172.21.2.102:0/3162440067 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc37000c920 con 0x7fc3800d1860 2021-02-10T11:26:36.440 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:36.419 7fc38568a700 1 -- 172.21.2.102:0/3162440067 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc3800d1860 msgr2=0x7fc3801b03e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:36.440 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:36.419 7fc38568a700 1 --2- 172.21.2.102:0/3162440067 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc3800d1860 0x7fc3801b03e0 secure :-1 s=READY pgs=1616 cs=0 l=1 rev1=1 rx=0x7fc370006700 tx=0x7fc370009b80).stop 2021-02-10T11:26:36.441 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:36.419 7fc38568a700 1 -- 172.21.2.102:0/3162440067 shutdown_connections 2021-02-10T11:26:36.441 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:36.419 7fc38568a700 1 --2- 172.21.2.102:0/3162440067 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc3801b0a00 0x7fc3800cf320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:36.441 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:36.419 7fc38568a700 1 --2- 172.21.2.102:0/3162440067 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc3800d1860 0x7fc3801b03e0 unknown :-1 s=CLOSED pgs=1616 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:36.441 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:36.419 7fc38568a700 1 -- 172.21.2.102:0/3162440067 >> 172.21.2.102:0/3162440067 conn(0x7fc3800ce440 msgr2=0x7fc3800d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:26:36.441 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:36.419 7fc38568a700 1 -- 172.21.2.102:0/3162440067 shutdown_connections 2021-02-10T11:26:36.441 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:36.419 7fc38568a700 1 -- 172.21.2.102:0/3162440067 wait complete. 2021-02-10T11:26:36.441 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:36.434 7fc38568a700 1 Processor -- start 2021-02-10T11:26:36.442 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:36.434 7fc38568a700 1 -- start start 2021-02-10T11:26:36.442 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:36.434 7fc38568a700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc3800acbb0 0x7fc3800ac030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:36.442 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:36.434 7fc38568a700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc3800ac650 0x7fc3800ad080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:36.442 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:36.434 7fc38568a700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc3800ad6a0 0x7fc3800f19b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:36.442 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:36.434 7fc38568a700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3800f1430 con 0x7fc3800acbb0 2021-02-10T11:26:36.443 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:36.434 7fc38568a700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3800adaa0 con 0x7fc3800ac650 2021-02-10T11:26:36.443 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:36.434 7fc38568a700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3800f1fd0 con 0x7fc3800ad6a0 2021-02-10T11:26:36.443 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:36.434 7fc37f2bf700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc3800acbb0 0x7fc3800ac030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:36.443 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:36.434 7fc37f2bf700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc3800acbb0 0x7fc3800ac030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:58788/0 (socket says 172.21.2.102:58788) 2021-02-10T11:26:36.443 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:36.434 7fc37f2bf700 1 -- 172.21.2.102:0/3260747790 learned_addr learned my addr 172.21.2.102:0/3260747790 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:26:36.443 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:36.435 7fc37fac0700 1 --2- 172.21.2.102:0/3260747790 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc3800ad6a0 0x7fc3800f19b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:36.443 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:36.435 7fc376abe700 1 --2- 172.21.2.102:0/3260747790 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc3800ac650 0x7fc3800ad080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:36.444 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:36.435 7fc37f2bf700 1 -- 172.21.2.102:0/3260747790 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc3800ad6a0 msgr2=0x7fc3800f19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:36.444 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:36.435 7fc37f2bf700 1 --2- 172.21.2.102:0/3260747790 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc3800ad6a0 0x7fc3800f19b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:36.444 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:36.435 7fc37f2bf700 1 -- 172.21.2.102:0/3260747790 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc3800ac650 msgr2=0x7fc3800ad080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:36.444 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:36.435 7fc37f2bf700 1 --2- 172.21.2.102:0/3260747790 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc3800ac650 0x7fc3800ad080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:36.444 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:36.435 7fc37f2bf700 1 -- 172.21.2.102:0/3260747790 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3800f21f0 con 0x7fc3800acbb0 2021-02-10T11:26:36.444 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:36.435 7fc376abe700 1 --2- 172.21.2.102:0/3260747790 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc3800ac650 0x7fc3800ad080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:26:36.445 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:36.435 7fc37f2bf700 1 --2- 172.21.2.102:0/3260747790 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc3800acbb0 0x7fc3800ac030 secure :-1 s=READY pgs=1617 cs=0 l=1 rev1=1 rx=0x7fc37000b580 tx=0x7fc37000b920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:36.445 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:36.435 7fc37caba700 1 -- 172.21.2.102:0/3260747790 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc37000bd10 con 0x7fc3800acbb0 2021-02-10T11:26:36.445 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:36.435 7fc37caba700 1 -- 172.21.2.102:0/3260747790 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc370002030 con 0x7fc3800acbb0 2021-02-10T11:26:36.445 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:36.435 7fc37caba700 1 -- 172.21.2.102:0/3260747790 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc370002480 con 0x7fc3800acbb0 2021-02-10T11:26:36.445 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:36.435 7fc38568a700 1 -- 172.21.2.102:0/3260747790 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3800f21f0 con 0x7fc3800acbb0 2021-02-10T11:26:36.446 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:36.435 7fc38568a700 1 -- 172.21.2.102:0/3260747790 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3800f21f0 con 0x7fc3800acbb0 2021-02-10T11:26:36.446 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:36.436 7fc37caba700 1 -- 172.21.2.102:0/3260747790 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fc370000c90 con 0x7fc3800acbb0 2021-02-10T11:26:36.446 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:36.436 7fc37caba700 1 --2- 172.21.2.102:0/3260747790 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc3640186d0 0x7fc36401abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:36.446 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:36.436 7fc37caba700 1 -- 172.21.2.102:0/3260747790 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fc3700327f0 con 0x7fc3800acbb0 2021-02-10T11:26:36.446 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:36.436 7fc376abe700 1 --2- 172.21.2.102:0/3260747790 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc3640186d0 0x7fc36401abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:36.446 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:36.436 7fc376abe700 1 --2- 172.21.2.102:0/3260747790 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc3640186d0 0x7fc36401abd0 crc :-1 s=READY pgs=734 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:36.468 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:36.468 7fc38568a700 1 -- 172.21.2.102:0/3260747790 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3800ee060 con 0x7fc3800acbb0 2021-02-10T11:26:36.470 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:36.471 7fc37caba700 1 -- 172.21.2.102:0/3260747790 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fc370000940 con 0x7fc3800acbb0 2021-02-10T11:26:36.599 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:36.599 7fc38568a700 1 -- 172.21.2.102:0/3260747790 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fc3800ce440 con 0x7fc3800acbb0 2021-02-10T11:26:37.377 INFO:tasks.ceph.mon.a.gibba002.stderr:2021-02-10 11:26:37.377 7f8dd2755700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-02-10T11:26:37.383 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:37.383 7fc37caba700 1 -- 172.21.2.102:0/3260747790 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v50) v1 ==== 161+0+0 (secure 0 0 0) 0x7fc370000c90 con 0x7fc3800acbb0 2021-02-10T11:26:37.418 INFO:teuthology.orchestra.run.gibba002.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2021-02-10T11:26:37.420 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:37.421 7fc38568a700 1 -- 172.21.2.102:0/3260747790 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc3640186d0 msgr2=0x7fc36401abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:37.420 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:37.421 7fc38568a700 1 --2- 172.21.2.102:0/3260747790 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc3640186d0 0x7fc36401abd0 crc :-1 s=READY pgs=734 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:37.421 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:37.421 7fc38568a700 1 -- 172.21.2.102:0/3260747790 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc3800acbb0 msgr2=0x7fc3800ac030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:37.421 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:37.421 7fc38568a700 1 --2- 172.21.2.102:0/3260747790 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc3800acbb0 0x7fc3800ac030 secure :-1 s=READY pgs=1617 cs=0 l=1 rev1=1 rx=0x7fc37000b580 tx=0x7fc37000b920).stop 2021-02-10T11:26:37.421 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:37.421 7fc38568a700 1 -- 172.21.2.102:0/3260747790 shutdown_connections 2021-02-10T11:26:37.421 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:37.421 7fc38568a700 1 --2- 172.21.2.102:0/3260747790 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc3800ad6a0 0x7fc3800f19b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:37.421 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:37.421 7fc38568a700 1 --2- 172.21.2.102:0/3260747790 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc3640186d0 0x7fc36401abd0 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:37.421 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:37.421 7fc38568a700 1 --2- 172.21.2.102:0/3260747790 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc3800ac650 0x7fc3800ad080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:37.421 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:37.421 7fc38568a700 1 --2- 172.21.2.102:0/3260747790 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc3800acbb0 0x7fc3800ac030 unknown :-1 s=CLOSED pgs=1617 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:37.421 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:37.421 7fc38568a700 1 -- 172.21.2.102:0/3260747790 >> 172.21.2.102:0/3260747790 conn(0x7fc3800bea70 msgr2=0x7fc380095700 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:26:37.422 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:37.421 7fc38568a700 1 -- 172.21.2.102:0/3260747790 shutdown_connections 2021-02-10T11:26:37.422 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:37.421 7fc38568a700 1 -- 172.21.2.102:0/3260747790 wait complete. 2021-02-10T11:26:37.429 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:26:37.433 INFO:tasks.ceph.mds.b.gibba013.stdout:starting mds.b at 2021-02-10T11:26:37.856 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:37.840 7f84fa5c9700 1 Processor -- start 2021-02-10T11:26:37.856 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:37.840 7f84fa5c9700 1 -- start start 2021-02-10T11:26:37.856 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:37.840 7f84fa5c9700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f84f40d1870 0x7f84f41b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:37.856 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:37.840 7f84fa5c9700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f84f41b0a10 0x7f84f40cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:37.856 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:37.840 7f84fa5c9700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84f40f0b10 con 0x7f84f40d1c70 2021-02-10T11:26:37.856 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:37.840 7f84fa5c9700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84f40fe230 con 0x7f84f41b0a10 2021-02-10T11:26:37.856 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:37.840 7f84fa5c9700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84f40e5ed0 con 0x7f84f40d1870 2021-02-10T11:26:37.856 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:37.841 7f84f8be6700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f84f40d1c70 0x7f84f40cf950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:51016/0 (socket says 172.21.2.102:51016) 2021-02-10T11:26:37.857 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:37.841 7f84f8be6700 1 -- 172.21.2.102:0/1818115615 learned_addr learned my addr 172.21.2.102:0/1818115615 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:26:37.857 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:37.841 7f84f37fe700 1 --2- 172.21.2.102:0/1818115615 >> v2:172.21.2.113:3300/0 conn(0x7f84f41b0a10 0x7f84f40cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:37.857 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:37.841 7f84f3fff700 1 --2- 172.21.2.102:0/1818115615 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f84f40d1870 0x7f84f41b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:37.857 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:37.841 7f84f2ffd700 1 -- 172.21.2.102:0/1818115615 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 230360731 0 0) 0x7f84ec001500 con 0x7f84f40d1c70 2021-02-10T11:26:37.857 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:37.841 7f84f2ffd700 1 -- 172.21.2.102:0/1818115615 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f84dc002fe0 con 0x7f84f40d1c70 2021-02-10T11:26:37.857 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:37.841 7f84f37fe700 1 -- 172.21.2.102:0/1818115615 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f84f40d1870 msgr2=0x7f84f41b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:26:37.857 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:37.841 7f84f37fe700 1 --2- 172.21.2.102:0/1818115615 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f84f40d1870 0x7f84f41b03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:37.857 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:37.841 7f84f37fe700 1 -- 172.21.2.102:0/1818115615 >> v1:172.21.2.102:6789/0 conn(0x7f84f40d1c70 legacy=0x7f84f40cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:37.857 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:37.841 7f84f37fe700 1 -- 172.21.2.102:0/1818115615 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84f40f0650 con 0x7f84f41b0a10 2021-02-10T11:26:37.857 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:37.841 7f84f37fe700 1 --2- 172.21.2.102:0/1818115615 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f84f41b0a10 0x7f84f40cf330 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 rx=0x7f84e8006650 tx=0x7f84e8009a80).ready entity=mon.1 client_cookie=adf0487a3c8e20df server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:37.857 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:37.841 7f84f2ffd700 1 -- 172.21.2.102:0/1818115615 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f84e800c0c0 con 0x7f84f41b0a10 2021-02-10T11:26:37.857 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:37.841 7f84f2ffd700 1 -- 172.21.2.102:0/1818115615 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84e800c340 con 0x7f84f41b0a10 2021-02-10T11:26:37.857 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:37.841 7f84f2ffd700 1 -- 172.21.2.102:0/1818115615 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f84e800c820 con 0x7f84f41b0a10 2021-02-10T11:26:37.858 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:37.841 7f84fa5c9700 1 -- 172.21.2.102:0/1818115615 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f84f41b0a10 msgr2=0x7f84f40cf330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:37.858 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:37.841 7f84fa5c9700 1 --2- 172.21.2.102:0/1818115615 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f84f41b0a10 0x7f84f40cf330 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 rx=0x7f84e8006650 tx=0x7f84e8009a80).stop 2021-02-10T11:26:37.858 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:37.842 7f84fa5c9700 1 -- 172.21.2.102:0/1818115615 shutdown_connections 2021-02-10T11:26:37.858 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:37.842 7f84fa5c9700 1 --2- 172.21.2.102:0/1818115615 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f84f41b0a10 0x7f84f40cf330 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:37.858 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:37.842 7f84fa5c9700 1 --2- 172.21.2.102:0/1818115615 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f84f40d1870 0x7f84f41b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:37.858 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:37.842 7f84fa5c9700 1 -- 172.21.2.102:0/1818115615 >> 172.21.2.102:0/1818115615 conn(0x7f84f40ce450 msgr2=0x7f84f40d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:26:37.858 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:37.842 7f84fa5c9700 1 -- 172.21.2.102:0/1818115615 shutdown_connections 2021-02-10T11:26:37.858 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:37.842 7f84fa5c9700 1 -- 172.21.2.102:0/1818115615 wait complete. 2021-02-10T11:26:37.860 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:37.856 7f84fa5c9700 1 Processor -- start 2021-02-10T11:26:37.860 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:37.857 7f84fa5c9700 1 -- start start 2021-02-10T11:26:37.860 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:37.857 7f84fa5c9700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f84f40beeb0 0x7f84f40bf2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:37.860 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:37.857 7f84fa5c9700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f84f418ef70 0x7f84f418f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:37.860 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:37.857 7f84fa5c9700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f84f418f990 0x7f84f40c01d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:37.860 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:37.857 7f84fa5c9700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84f40bfc50 con 0x7f84f418ef70 2021-02-10T11:26:37.860 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:37.857 7f84fa5c9700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84f418fd90 con 0x7f84f40beeb0 2021-02-10T11:26:37.860 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:37.857 7f84fa5c9700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84f40bf8d0 con 0x7f84f418f990 2021-02-10T11:26:37.860 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:37.857 7f84f37fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f84f418ef70 0x7f84f418f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:37.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:37.857 7f84f37fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f84f418ef70 0x7f84f418f370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:58798/0 (socket says 172.21.2.102:58798) 2021-02-10T11:26:37.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:37.857 7f84f37fe700 1 -- 172.21.2.102:0/3277627101 learned_addr learned my addr 172.21.2.102:0/3277627101 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:26:37.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:37.857 7f84f3fff700 1 --2- 172.21.2.102:0/3277627101 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f84f40beeb0 0x7f84f40bf2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:37.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:37.858 7f84f8be6700 1 --2- 172.21.2.102:0/3277627101 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f84f418f990 0x7f84f40c01d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:37.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:37.858 7f84f37fe700 1 -- 172.21.2.102:0/3277627101 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f84f418f990 msgr2=0x7f84f40c01d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:37.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:37.858 7f84f37fe700 1 --2- 172.21.2.102:0/3277627101 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f84f418f990 0x7f84f40c01d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:37.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:37.858 7f84f37fe700 1 -- 172.21.2.102:0/3277627101 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f84f40beeb0 msgr2=0x7f84f40bf2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:37.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:37.858 7f84f37fe700 1 --2- 172.21.2.102:0/3277627101 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f84f40beeb0 0x7f84f40bf2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:37.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:37.858 7f84f37fe700 1 -- 172.21.2.102:0/3277627101 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84f40c07f0 con 0x7f84f418ef70 2021-02-10T11:26:37.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:37.858 7f84f37fe700 1 --2- 172.21.2.102:0/3277627101 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f84f418ef70 0x7f84f418f370 secure :-1 s=READY pgs=1619 cs=0 l=1 rev1=1 rx=0x7f84e8009650 tx=0x7f84e800b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:37.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:37.858 7f84f0ff9700 1 -- 172.21.2.102:0/3277627101 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f84e800bd50 con 0x7f84f418ef70 2021-02-10T11:26:37.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:37.858 7f84f0ff9700 1 -- 172.21.2.102:0/3277627101 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84e8006030 con 0x7f84f418ef70 2021-02-10T11:26:37.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:37.858 7f84fa5c9700 1 -- 172.21.2.102:0/3277627101 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84f40c07f0 con 0x7f84f418ef70 2021-02-10T11:26:37.862 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:37.858 7f84fa5c9700 1 -- 172.21.2.102:0/3277627101 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84f40c07f0 con 0x7f84f418ef70 2021-02-10T11:26:37.862 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:37.858 7f84f0ff9700 1 -- 172.21.2.102:0/3277627101 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f84e8005560 con 0x7f84f418ef70 2021-02-10T11:26:37.862 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:37.859 7f84f0ff9700 1 -- 172.21.2.102:0/3277627101 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f84e8009300 con 0x7f84f418ef70 2021-02-10T11:26:37.862 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:37.859 7f84f0ff9700 1 --2- 172.21.2.102:0/3277627101 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f84d4018730 0x7f84d401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:37.862 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:37.859 7f84f0ff9700 1 -- 172.21.2.102:0/3277627101 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 5659+0+0 (secure 0 0 0) 0x7f84e80327e0 con 0x7f84f418ef70 2021-02-10T11:26:37.862 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:37.859 7f84f3fff700 1 --2- 172.21.2.102:0/3277627101 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f84d4018730 0x7f84d401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:37.862 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:37.859 7f84f3fff700 1 --2- 172.21.2.102:0/3277627101 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f84d4018730 0x7f84d401abc0 crc :-1 s=READY pgs=735 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:37.890 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:37.890 7f84fa5c9700 1 -- 172.21.2.102:0/3277627101 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84f40ebc60 con 0x7f84f418ef70 2021-02-10T11:26:37.892 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:37.893 7f84f0ff9700 1 -- 172.21.2.102:0/3277627101 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f84e80009c0 con 0x7f84f418ef70 2021-02-10T11:26:38.019 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:38.019 7f84fa5c9700 1 -- 172.21.2.102:0/3277627101 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f84f40ebc60 con 0x7f84f418ef70 2021-02-10T11:26:38.019 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:38.020 7f84f0ff9700 1 -- 172.21.2.102:0/3277627101 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) v1 ==== 94+0+1756 (secure 0 0 0) 0x7f84e800bd50 con 0x7f84f418ef70 2021-02-10T11:26:38.020 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:26:38.020 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":50,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8412,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.102:6835/3414701760","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6834","nonce":3414701760},{"type":"v1","addr":"172.21.2.102:6835","nonce":3414701760}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":44}],"filesystems":[{"mdsmap":{"epoch":50,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:25:31.513947","modified":"2021-02-10 11:26:37.378483","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":6}]} 2021-02-10T11:26:38.020 INFO:teuthology.orchestra.run.gibba002.stderr:dumped fsmap epoch 50 2021-02-10T11:26:38.021 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:38.021 7f84fa5c9700 1 -- 172.21.2.102:0/3277627101 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f84d4018730 msgr2=0x7f84d401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:38.022 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:38.021 7f84fa5c9700 1 --2- 172.21.2.102:0/3277627101 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f84d4018730 0x7f84d401abc0 crc :-1 s=READY pgs=735 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:38.022 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:38.022 7f84fa5c9700 1 -- 172.21.2.102:0/3277627101 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f84f418ef70 msgr2=0x7f84f418f370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:38.022 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:38.022 7f84fa5c9700 1 --2- 172.21.2.102:0/3277627101 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f84f418ef70 0x7f84f418f370 secure :-1 s=READY pgs=1619 cs=0 l=1 rev1=1 rx=0x7f84e8009650 tx=0x7f84e800b960).stop 2021-02-10T11:26:38.022 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:38.022 7f84fa5c9700 1 -- 172.21.2.102:0/3277627101 shutdown_connections 2021-02-10T11:26:38.022 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:38.022 7f84fa5c9700 1 --2- 172.21.2.102:0/3277627101 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f84f418f990 0x7f84f40c01d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:38.022 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:38.022 7f84fa5c9700 1 --2- 172.21.2.102:0/3277627101 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f84f418ef70 0x7f84f418f370 unknown :-1 s=CLOSED pgs=1619 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:38.022 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:38.022 7f84fa5c9700 1 --2- 172.21.2.102:0/3277627101 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f84d4018730 0x7f84d401abc0 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:38.022 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:38.022 7f84fa5c9700 1 --2- 172.21.2.102:0/3277627101 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f84f40beeb0 0x7f84f40bf2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:38.022 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:38.022 7f84fa5c9700 1 -- 172.21.2.102:0/3277627101 >> 172.21.2.102:0/3277627101 conn(0x7f84f40c2130 msgr2=0x7f84f409e3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:26:38.022 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:38.022 7f84fa5c9700 1 -- 172.21.2.102:0/3277627101 shutdown_connections 2021-02-10T11:26:38.022 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:38.022 7f84fa5c9700 1 -- 172.21.2.102:0/3277627101 wait complete. 2021-02-10T11:26:38.030 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2021-02-10T11:26:38.462 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:38.445 7f50c57e9700 1 Processor -- start 2021-02-10T11:26:38.462 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:38.445 7f50c57e9700 1 -- start start 2021-02-10T11:26:38.462 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:38.445 7f50c57e9700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f50c00d1870 0x7f50c01b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:38.462 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:38.445 7f50c57e9700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50c00f0b10 con 0x7f50c00f0650 2021-02-10T11:26:38.462 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:38.445 7f50c57e9700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50c00d0310 con 0x7f50c01b0a10 2021-02-10T11:26:38.462 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:38.445 7f50c57e9700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50c00d0530 con 0x7f50c00d1870 2021-02-10T11:26:38.462 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:38.445 7f50bf574700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f50c00d1870 0x7f50c01b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:38.462 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:38.445 7f50bf574700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f50c00d1870 0x7f50c01b03f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:58802/0 (socket says 172.21.2.102:58802) 2021-02-10T11:26:38.463 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:38.445 7f50bf574700 1 -- 172.21.2.102:0/1082324554 learned_addr learned my addr 172.21.2.102:0/1082324554 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:26:38.463 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:38.446 7f50bf574700 1 -- 172.21.2.102:0/1082324554 >> v1:172.21.2.113:6789/0 conn(0x7f50c01b0a10 legacy=0x7f50c00cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:26:38.463 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:38.446 7f50bf574700 1 -- 172.21.2.102:0/1082324554 >> v1:172.21.2.102:6789/0 conn(0x7f50c00f0650 legacy=0x7f50c00cfb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:38.463 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:38.446 7f50bf574700 1 -- 172.21.2.102:0/1082324554 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50c00d1c70 con 0x7f50c00d1870 2021-02-10T11:26:38.463 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:38.446 7f50bf574700 1 --2- 172.21.2.102:0/1082324554 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f50c00d1870 0x7f50c01b03f0 secure :-1 s=READY pgs=1621 cs=0 l=1 rev1=1 rx=0x7f50b0006700 tx=0x7f50b0009b00).ready entity=mon.0 client_cookie=e6aab70b89078a91 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:38.463 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:38.446 7f50be572700 1 -- 172.21.2.102:0/1082324554 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f50b000c0c0 con 0x7f50c00d1870 2021-02-10T11:26:38.463 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:38.446 7f50be572700 1 -- 172.21.2.102:0/1082324554 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50b000c340 con 0x7f50c00d1870 2021-02-10T11:26:38.463 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:38.446 7f50be572700 1 -- 172.21.2.102:0/1082324554 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f50b000c820 con 0x7f50c00d1870 2021-02-10T11:26:38.463 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:38.446 7f50c57e9700 1 -- 172.21.2.102:0/1082324554 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f50c00d1870 msgr2=0x7f50c01b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:38.463 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:38.446 7f50c57e9700 1 --2- 172.21.2.102:0/1082324554 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f50c00d1870 0x7f50c01b03f0 secure :-1 s=READY pgs=1621 cs=0 l=1 rev1=1 rx=0x7f50b0006700 tx=0x7f50b0009b00).stop 2021-02-10T11:26:38.463 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:38.447 7f50c57e9700 1 -- 172.21.2.102:0/1082324554 shutdown_connections 2021-02-10T11:26:38.463 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:38.447 7f50c57e9700 1 --2- 172.21.2.102:0/1082324554 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f50c00d1870 0x7f50c01b03f0 unknown :-1 s=CLOSED pgs=1621 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:38.463 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:38.447 7f50c57e9700 1 -- 172.21.2.102:0/1082324554 >> 172.21.2.102:0/1082324554 conn(0x7f50c00ce450 msgr2=0x7f50c00d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:26:38.464 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:38.447 7f50c57e9700 1 -- 172.21.2.102:0/1082324554 shutdown_connections 2021-02-10T11:26:38.464 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:38.447 7f50c57e9700 1 -- 172.21.2.102:0/1082324554 wait complete. 2021-02-10T11:26:38.464 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:38.462 7f50c57e9700 1 Processor -- start 2021-02-10T11:26:38.464 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:38.462 7f50c57e9700 1 -- start start 2021-02-10T11:26:38.465 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:38.462 7f50c57e9700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f50c00c33b0 0x7f50c009b280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:38.465 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:38.462 7f50c57e9700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f50c01b00b0 0x7f50c01b04b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:38.466 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:38.462 7f50c57e9700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f50c01b0ad0 0x7f50c00c1c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:38.466 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:38.462 7f50c57e9700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50c009ad00 con 0x7f50c01b0ad0 2021-02-10T11:26:38.466 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:38.462 7f50c57e9700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50c00bc950 con 0x7f50c01b00b0 2021-02-10T11:26:38.466 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:38.462 7f50c57e9700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50c00bcb30 con 0x7f50c00c33b0 2021-02-10T11:26:38.466 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:38.462 7f50bfd75700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f50c01b0ad0 0x7f50c00c1c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:38.466 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:38.462 7f50bfd75700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f50c01b0ad0 0x7f50c00c1c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:58810/0 (socket says 172.21.2.102:58810) 2021-02-10T11:26:38.466 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:38.462 7f50bfd75700 1 -- 172.21.2.102:0/1217062809 learned_addr learned my addr 172.21.2.102:0/1217062809 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:26:38.466 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:38.463 7f50bed73700 1 --2- 172.21.2.102:0/1217062809 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f50c01b00b0 0x7f50c01b04b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:38.466 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:38.463 7f50bfd75700 1 -- 172.21.2.102:0/1217062809 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f50c00c33b0 msgr2=0x7f50c009b280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:38.466 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:38.463 7f50bfd75700 1 --2- 172.21.2.102:0/1217062809 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f50c00c33b0 0x7f50c009b280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:38.466 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:38.463 7f50bfd75700 1 -- 172.21.2.102:0/1217062809 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f50c01b00b0 msgr2=0x7f50c01b04b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:38.466 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:38.463 7f50bfd75700 1 --2- 172.21.2.102:0/1217062809 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f50c01b00b0 0x7f50c01b04b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:38.467 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:38.463 7f50bfd75700 1 -- 172.21.2.102:0/1217062809 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50c00c22a0 con 0x7f50c01b0ad0 2021-02-10T11:26:38.467 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:38.463 7f50bed73700 1 --2- 172.21.2.102:0/1217062809 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f50c01b00b0 0x7f50c01b04b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:26:38.467 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:38.463 7f50bfd75700 1 --2- 172.21.2.102:0/1217062809 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f50c01b0ad0 0x7f50c00c1c50 secure :-1 s=READY pgs=1622 cs=0 l=1 rev1=1 rx=0x7f50b8009980 tx=0x7f50b800bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:38.467 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:38.463 7f50b7fff700 1 -- 172.21.2.102:0/1217062809 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f50b8005870 con 0x7f50c01b0ad0 2021-02-10T11:26:38.467 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:38.463 7f50b7fff700 1 -- 172.21.2.102:0/1217062809 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50b8005a70 con 0x7f50c01b0ad0 2021-02-10T11:26:38.467 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:38.463 7f50c57e9700 1 -- 172.21.2.102:0/1217062809 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50c00c22a0 con 0x7f50c01b0ad0 2021-02-10T11:26:38.467 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:38.463 7f50b7fff700 1 -- 172.21.2.102:0/1217062809 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f50b8006030 con 0x7f50c01b0ad0 2021-02-10T11:26:38.467 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:38.463 7f50c57e9700 1 -- 172.21.2.102:0/1217062809 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50c00c22a0 con 0x7f50c01b0ad0 2021-02-10T11:26:38.467 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:38.464 7f50b7fff700 1 -- 172.21.2.102:0/1217062809 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f50b8005c50 con 0x7f50c01b0ad0 2021-02-10T11:26:38.467 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:38.464 7f50b7fff700 1 --2- 172.21.2.102:0/1217062809 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f50a40186c0 0x7f50a401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:38.467 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:38.464 7f50b7fff700 1 -- 172.21.2.102:0/1217062809 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 5659+0+0 (secure 0 0 0) 0x7f50b8032970 con 0x7f50c01b0ad0 2021-02-10T11:26:38.467 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:38.464 7f50bf574700 1 --2- 172.21.2.102:0/1217062809 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f50a40186c0 0x7f50a401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:38.468 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:38.464 7f50bf574700 1 --2- 172.21.2.102:0/1217062809 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f50a40186c0 0x7f50a401abc0 crc :-1 s=READY pgs=736 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:38.495 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:38.495 7f50c57e9700 1 -- 172.21.2.102:0/1217062809 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50c00d1ee0 con 0x7f50c01b0ad0 2021-02-10T11:26:38.498 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:38.498 7f50b7fff700 1 -- 172.21.2.102:0/1217062809 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f50b800be10 con 0x7f50c01b0ad0 2021-02-10T11:26:38.625 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:38.625 7f50c57e9700 1 -- 172.21.2.102:0/1217062809 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f50c00d1ee0 con 0x7f50c01b0ad0 2021-02-10T11:26:39.391 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:39.391 7f50b7fff700 1 -- 172.21.2.102:0/1217062809 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v51) v1 ==== 104+0+0 (secure 0 0 0) 0x7f50b8005c50 con 0x7f50c01b0ad0 2021-02-10T11:26:39.396 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:39.396 7f50c57e9700 1 -- 172.21.2.102:0/1217062809 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f50a40186c0 msgr2=0x7f50a401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:39.396 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:39.396 7f50c57e9700 1 --2- 172.21.2.102:0/1217062809 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f50a40186c0 0x7f50a401abc0 crc :-1 s=READY pgs=736 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:39.396 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:39.396 7f50c57e9700 1 -- 172.21.2.102:0/1217062809 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f50c01b0ad0 msgr2=0x7f50c00c1c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:39.396 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:39.396 7f50c57e9700 1 --2- 172.21.2.102:0/1217062809 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f50c01b0ad0 0x7f50c00c1c50 secure :-1 s=READY pgs=1622 cs=0 l=1 rev1=1 rx=0x7f50b8009980 tx=0x7f50b800bcd0).stop 2021-02-10T11:26:39.397 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:39.396 7f50c57e9700 1 -- 172.21.2.102:0/1217062809 shutdown_connections 2021-02-10T11:26:39.397 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:39.396 7f50c57e9700 1 --2- 172.21.2.102:0/1217062809 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f50c01b0ad0 0x7f50c00c1c50 unknown :-1 s=CLOSED pgs=1622 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:39.397 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:39.396 7f50c57e9700 1 --2- 172.21.2.102:0/1217062809 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f50a40186c0 0x7f50a401abc0 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:39.397 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:39.396 7f50c57e9700 1 --2- 172.21.2.102:0/1217062809 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f50c01b00b0 0x7f50c01b04b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:39.397 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:39.396 7f50c57e9700 1 --2- 172.21.2.102:0/1217062809 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f50c00c33b0 0x7f50c009b280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:39.397 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:39.396 7f50c57e9700 1 -- 172.21.2.102:0/1217062809 >> 172.21.2.102:0/1217062809 conn(0x7f50c00afc60 msgr2=0x7f50c00eb260 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:26:39.397 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:39.397 7f50c57e9700 1 -- 172.21.2.102:0/1217062809 shutdown_connections 2021-02-10T11:26:39.397 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:39.397 7f50c57e9700 1 -- 172.21.2.102:0/1217062809 wait complete. 2021-02-10T11:26:39.405 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool delete cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2021-02-10T11:26:39.835 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:39.819 7fa4589e5700 1 Processor -- start 2021-02-10T11:26:39.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:39.819 7fa4589e5700 1 -- start start 2021-02-10T11:26:39.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:39.819 7fa4589e5700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fa4540d1880 0x7fa4541b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:39.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:39.819 7fa4589e5700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa4541b0af0 0x7fa454108950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:39.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:39.819 7fa4589e5700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4540fe1d0 con 0x7fa4540d1c80 2021-02-10T11:26:39.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:39.819 7fa4589e5700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa454109a70 con 0x7fa4540d1880 2021-02-10T11:26:39.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:39.819 7fa4589e5700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa454109760 con 0x7fa4541b0af0 2021-02-10T11:26:39.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:39.820 7fa452e1d700 1 --1- >> v1:172.21.2.113:6789/0 conn(0x7fa4540d1c80 0x7fa454108f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.113:6789/0 says I am v1:172.21.2.102:58362/0 (socket says 172.21.2.102:58362) 2021-02-10T11:26:39.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:39.820 7fa452e1d700 1 -- 172.21.2.102:0/3890141638 learned_addr learned my addr 172.21.2.102:0/3890141638 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:26:39.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:39.820 7fa451e1b700 1 --2- 172.21.2.102:0/3890141638 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa4541b0af0 0x7fa454108950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:39.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:39.820 7fa45261c700 1 --2- 172.21.2.102:0/3890141638 >> v2:172.21.2.113:3300/0 conn(0x7fa4540d1880 0x7fa4541b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:39.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:39.820 7fa45261c700 1 -- 172.21.2.102:0/3890141638 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa4541b0af0 msgr2=0x7fa454108950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:26:39.837 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:39.820 7fa45261c700 1 --2- 172.21.2.102:0/3890141638 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa4541b0af0 0x7fa454108950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:39.837 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:39.820 7fa45261c700 1 -- 172.21.2.102:0/3890141638 >> v1:172.21.2.113:6789/0 conn(0x7fa4540d1c80 legacy=0x7fa454108f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:39.837 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:39.820 7fa45261c700 1 -- 172.21.2.102:0/3890141638 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa454088070 con 0x7fa4540d1880 2021-02-10T11:26:39.837 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:39.820 7fa45261c700 1 --2- 172.21.2.102:0/3890141638 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa4540d1880 0x7fa4541b04d0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 rx=0x7fa43c006650 tx=0x7fa43c009a80).ready entity=mon.1 client_cookie=741dd9e9d7257afe server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:39.837 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:39.821 7fa45161a700 1 -- 172.21.2.102:0/3890141638 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa43c00c0c0 con 0x7fa4540d1880 2021-02-10T11:26:39.837 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:39.821 7fa45161a700 1 -- 172.21.2.102:0/3890141638 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa43c00c340 con 0x7fa4540d1880 2021-02-10T11:26:39.837 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:39.821 7fa45161a700 1 -- 172.21.2.102:0/3890141638 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa43c00c860 con 0x7fa4540d1880 2021-02-10T11:26:39.837 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:39.821 7fa4589e5700 1 -- 172.21.2.102:0/3890141638 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa4540d1880 msgr2=0x7fa4541b04d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:39.837 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:39.821 7fa4589e5700 1 --2- 172.21.2.102:0/3890141638 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa4540d1880 0x7fa4541b04d0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 rx=0x7fa43c006650 tx=0x7fa43c009a80).stop 2021-02-10T11:26:39.837 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:39.821 7fa4589e5700 1 -- 172.21.2.102:0/3890141638 shutdown_connections 2021-02-10T11:26:39.837 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:39.821 7fa4589e5700 1 --2- 172.21.2.102:0/3890141638 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa4541b0af0 0x7fa454108950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:39.837 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:39.821 7fa4589e5700 1 --2- 172.21.2.102:0/3890141638 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa4540d1880 0x7fa4541b04d0 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:39.837 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:39.821 7fa4589e5700 1 -- 172.21.2.102:0/3890141638 >> 172.21.2.102:0/3890141638 conn(0x7fa4540ce430 msgr2=0x7fa4540d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:26:39.837 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:39.821 7fa4589e5700 1 -- 172.21.2.102:0/3890141638 shutdown_connections 2021-02-10T11:26:39.838 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:39.821 7fa4589e5700 1 -- 172.21.2.102:0/3890141638 wait complete. 2021-02-10T11:26:39.838 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:39.836 7fa4589e5700 1 Processor -- start 2021-02-10T11:26:39.838 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:39.836 7fa4589e5700 1 -- start start 2021-02-10T11:26:39.839 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:39.836 7fa4589e5700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa4540aef00 0x7fa454099c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:39.839 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:39.836 7fa4589e5700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa45409a260 0x7fa45409a660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:39.839 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:39.836 7fa4589e5700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa45409ac80 0x7fa45419a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:39.839 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:39.836 7fa4589e5700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa45419bdd0 con 0x7fa4540aef00 2021-02-10T11:26:39.839 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:39.836 7fa4589e5700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa45419de70 con 0x7fa45409ac80 2021-02-10T11:26:39.839 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:39.836 7fa4589e5700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa45419a2d0 con 0x7fa45409a260 2021-02-10T11:26:39.840 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:39.836 7fa451e1b700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa45409a260 0x7fa45409a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:39.840 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:39.836 7fa45261c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa4540aef00 0x7fa454099c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:39.840 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:39.836 7fa451e1b700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa45409a260 0x7fa45409a660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.113:3301/0 says I am v2:172.21.2.102:36124/0 (socket says 172.21.2.102:36124) 2021-02-10T11:26:39.840 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:39.836 7fa451e1b700 1 -- 172.21.2.102:0/1595329150 learned_addr learned my addr 172.21.2.102:0/1595329150 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:26:39.840 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:39.836 7fa452e1d700 1 --2- 172.21.2.102:0/1595329150 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa45409ac80 0x7fa45419a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:39.840 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:39.837 7fa451e1b700 1 -- 172.21.2.102:0/1595329150 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa45409ac80 msgr2=0x7fa45419a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:39.840 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:39.837 7fa451e1b700 1 --2- 172.21.2.102:0/1595329150 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa45409ac80 0x7fa45419a850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:39.840 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:39.837 7fa451e1b700 1 -- 172.21.2.102:0/1595329150 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa4540aef00 msgr2=0x7fa454099c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:39.840 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:39.837 7fa451e1b700 1 --2- 172.21.2.102:0/1595329150 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa4540aef00 0x7fa454099c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:39.840 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:39.837 7fa451e1b700 1 -- 172.21.2.102:0/1595329150 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa454199f50 con 0x7fa45409a260 2021-02-10T11:26:39.840 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:39.837 7fa451e1b700 1 --2- 172.21.2.102:0/1595329150 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa45409a260 0x7fa45409a660 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7fa444006f60 tx=0x7fa44400b970).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:39.840 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:39.837 7fa44affd700 1 -- 172.21.2.102:0/1595329150 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa44400bd60 con 0x7fa45409a260 2021-02-10T11:26:39.840 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:39.837 7fa44affd700 1 -- 172.21.2.102:0/1595329150 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa444000c90 con 0x7fa45409a260 2021-02-10T11:26:39.841 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:39.837 7fa4589e5700 1 -- 172.21.2.102:0/1595329150 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa454199f50 con 0x7fa45409a260 2021-02-10T11:26:39.841 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:39.837 7fa44affd700 1 -- 172.21.2.102:0/1595329150 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa4440022b0 con 0x7fa45409a260 2021-02-10T11:26:39.841 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:39.837 7fa4589e5700 1 -- 172.21.2.102:0/1595329150 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa454199f50 con 0x7fa45409a260 2021-02-10T11:26:39.841 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:39.838 7fa44affd700 1 -- 172.21.2.102:0/1595329150 <== mon.2 v2:172.21.2.113:3301/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa444006b00 con 0x7fa45409a260 2021-02-10T11:26:39.841 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:39.838 7fa44affd700 1 --2- 172.21.2.102:0/1595329150 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa438018730 0x7fa43801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:39.841 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:39.838 7fa44affd700 1 -- 172.21.2.102:0/1595329150 <== mon.2 v2:172.21.2.113:3301/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 5659+0+0 (secure 0 0 0) 0x7fa444032660 con 0x7fa45409a260 2021-02-10T11:26:39.841 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:39.838 7fa45261c700 1 --2- 172.21.2.102:0/1595329150 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa438018730 0x7fa43801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:39.841 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:39.838 7fa45261c700 1 --2- 172.21.2.102:0/1595329150 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa438018730 0x7fa43801ac10 crc :-1 s=READY pgs=737 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:39.869 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:39.869 7fa4589e5700 1 -- 172.21.2.102:0/1595329150 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa454082a70 con 0x7fa45409a260 2021-02-10T11:26:39.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:39.872 7fa44affd700 1 -- 172.21.2.102:0/1595329150 <== mon.2 v2:172.21.2.113:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fa444000c90 con 0x7fa45409a260 2021-02-10T11:26:39.999 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:39.999 7fa4589e5700 1 -- 172.21.2.102:0/1595329150 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_command({"pool2": "cephfs_metadata", "prefix": "osd pool delete", "pool": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fa454199f50 con 0x7fa45409a260 2021-02-10T11:26:40.395 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:40.396 7fa44affd700 1 -- 172.21.2.102:0/1595329150 <== mon.2 v2:172.21.2.113:3301/0 7 ==== mon_command_ack([{"pool2": "cephfs_metadata", "prefix": "osd pool delete", "pool": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v68) v1 ==== 185+0+0 (secure 0 0 0) 0x7fa44401f4d0 con 0x7fa45409a260 2021-02-10T11:26:40.414 INFO:teuthology.orchestra.run.gibba002.stderr:pool 'cephfs_metadata' removed 2021-02-10T11:26:40.417 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:40.417 7fa4589e5700 1 -- 172.21.2.102:0/1595329150 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa438018730 msgr2=0x7fa43801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:40.417 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:40.417 7fa4589e5700 1 --2- 172.21.2.102:0/1595329150 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa438018730 0x7fa43801ac10 crc :-1 s=READY pgs=737 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:40.417 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:40.417 7fa4589e5700 1 -- 172.21.2.102:0/1595329150 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa45409a260 msgr2=0x7fa45409a660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:40.417 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:40.417 7fa4589e5700 1 --2- 172.21.2.102:0/1595329150 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa45409a260 0x7fa45409a660 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7fa444006f60 tx=0x7fa44400b970).stop 2021-02-10T11:26:40.417 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:40.417 7fa4589e5700 1 -- 172.21.2.102:0/1595329150 shutdown_connections 2021-02-10T11:26:40.417 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:40.417 7fa4589e5700 1 --2- 172.21.2.102:0/1595329150 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa438018730 0x7fa43801ac10 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:40.417 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:40.417 7fa4589e5700 1 --2- 172.21.2.102:0/1595329150 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa45409ac80 0x7fa45419a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:40.417 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:40.417 7fa4589e5700 1 --2- 172.21.2.102:0/1595329150 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa45409a260 0x7fa45409a660 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:40.417 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:40.417 7fa4589e5700 1 --2- 172.21.2.102:0/1595329150 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa4540aef00 0x7fa454099c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:40.417 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:40.417 7fa4589e5700 1 -- 172.21.2.102:0/1595329150 >> 172.21.2.102:0/1595329150 conn(0x7fa45418e810 msgr2=0x7fa45418c290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:26:40.417 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:40.417 7fa4589e5700 1 -- 172.21.2.102:0/1595329150 shutdown_connections 2021-02-10T11:26:40.418 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:40.417 7fa4589e5700 1 -- 172.21.2.102:0/1595329150 wait complete. 2021-02-10T11:26:40.425 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool delete cephfs_data cephfs_data --yes-i-really-really-mean-it 2021-02-10T11:26:40.856 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:40.840 7f9879852700 1 Processor -- start 2021-02-10T11:26:40.856 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:40.840 7f9879852700 1 -- start start 2021-02-10T11:26:40.856 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:40.841 7f9879852700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f98740d1880 0x7f98741b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:40.856 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:40.841 7f9879852700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f98741b0af0 0x7f9874108950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:40.856 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:40.841 7f9879852700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98740fe1d0 con 0x7f98740d1c80 2021-02-10T11:26:40.856 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:40.841 7f9879852700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9874109a70 con 0x7f98740d1880 2021-02-10T11:26:40.856 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:40.841 7f9879852700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9874109760 con 0x7f98741b0af0 2021-02-10T11:26:40.856 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:40.841 7f9873574700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f98740d1880 0x7f98741b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:40.856 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:40.841 7f9873574700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f98740d1880 0x7f98741b04d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:58828/0 (socket says 172.21.2.102:58828) 2021-02-10T11:26:40.856 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:40.841 7f9873574700 1 -- 172.21.2.102:0/2443986531 learned_addr learned my addr 172.21.2.102:0/2443986531 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:26:40.857 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:40.841 7f9873574700 1 -- 172.21.2.102:0/2443986531 >> v2:172.21.2.113:3300/0 conn(0x7f98741b0af0 msgr2=0x7f9874108950 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:26:40.857 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:40.841 7f9873574700 1 --2- 172.21.2.102:0/2443986531 >> v2:172.21.2.113:3300/0 conn(0x7f98741b0af0 0x7f9874108950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:40.857 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:40.841 7f9873574700 1 -- 172.21.2.102:0/2443986531 >> v1:172.21.2.102:6789/0 conn(0x7f98740d1c80 legacy=0x7f9874108f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:40.857 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:40.841 7f9873574700 1 -- 172.21.2.102:0/2443986531 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9874088070 con 0x7f98740d1880 2021-02-10T11:26:40.857 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:40.841 7f9873574700 1 --2- 172.21.2.102:0/2443986531 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f98740d1880 0x7f98741b04d0 secure :-1 s=READY pgs=1624 cs=0 l=1 rev1=1 rx=0x7f985c006700 tx=0x7f985c009b00).ready entity=mon.0 client_cookie=734e0ed44628c572 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:40.857 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:40.841 7f9872572700 1 -- 172.21.2.102:0/2443986531 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f985c00c0c0 con 0x7f98740d1880 2021-02-10T11:26:40.857 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:40.841 7f9872572700 1 -- 172.21.2.102:0/2443986531 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f985c00c340 con 0x7f98740d1880 2021-02-10T11:26:40.857 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:40.841 7f9872572700 1 -- 172.21.2.102:0/2443986531 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f985c00c820 con 0x7f98740d1880 2021-02-10T11:26:40.857 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:40.842 7f9879852700 1 -- 172.21.2.102:0/2443986531 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f98740d1880 msgr2=0x7f98741b04d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:40.857 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:40.842 7f9879852700 1 --2- 172.21.2.102:0/2443986531 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f98740d1880 0x7f98741b04d0 secure :-1 s=READY pgs=1624 cs=0 l=1 rev1=1 rx=0x7f985c006700 tx=0x7f985c009b00).stop 2021-02-10T11:26:40.857 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:40.842 7f9879852700 1 -- 172.21.2.102:0/2443986531 shutdown_connections 2021-02-10T11:26:40.857 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:40.842 7f9879852700 1 --2- 172.21.2.102:0/2443986531 >> v2:172.21.2.113:3300/0 conn(0x7f98741b0af0 0x7f9874108950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:40.857 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:40.842 7f9879852700 1 --2- 172.21.2.102:0/2443986531 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f98740d1880 0x7f98741b04d0 unknown :-1 s=CLOSED pgs=1624 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:40.858 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:40.842 7f9879852700 1 -- 172.21.2.102:0/2443986531 >> 172.21.2.102:0/2443986531 conn(0x7f98740ce430 msgr2=0x7f98740d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:26:40.858 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:40.842 7f9879852700 1 -- 172.21.2.102:0/2443986531 shutdown_connections 2021-02-10T11:26:40.858 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:40.842 7f9879852700 1 -- 172.21.2.102:0/2443986531 wait complete. 2021-02-10T11:26:40.859 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:40.856 7f9879852700 1 Processor -- start 2021-02-10T11:26:40.859 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:40.857 7f9879852700 1 -- start start 2021-02-10T11:26:40.859 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:40.857 7f9879852700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f987419e5b0 0x7f98741afc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:40.859 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:40.857 7f9879852700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f987419e9b0 0x7f98741b0dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:40.859 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:40.857 7f9879852700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f98741b0250 0x7f98741b13e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:40.860 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:40.857 7f9879852700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9874195bd0 con 0x7f98741b0250 2021-02-10T11:26:40.860 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:40.857 7f9879852700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98741b0840 con 0x7f987419e5b0 2021-02-10T11:26:40.860 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:40.857 7f9879852700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f987419eed0 con 0x7f987419e9b0 2021-02-10T11:26:40.860 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:40.857 7f9873d75700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f98741b0250 0x7f98741b13e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:40.860 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:40.857 7f9873d75700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f98741b0250 0x7f98741b13e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:58834/0 (socket says 172.21.2.102:58834) 2021-02-10T11:26:40.860 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:40.857 7f9873d75700 1 -- 172.21.2.102:0/3571146404 learned_addr learned my addr 172.21.2.102:0/3571146404 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:26:40.860 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:40.858 7f9873d75700 1 -- 172.21.2.102:0/3571146404 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f987419e9b0 msgr2=0x7f98741b0dc0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:26:40.860 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:40.858 7f9873d75700 1 --2- 172.21.2.102:0/3571146404 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f987419e9b0 0x7f98741b0dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:40.860 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:40.858 7f9873d75700 1 -- 172.21.2.102:0/3571146404 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f987419e5b0 msgr2=0x7f98741afc30 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:26:40.860 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:40.858 7f9873d75700 1 --2- 172.21.2.102:0/3571146404 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f987419e5b0 0x7f98741afc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:40.860 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:40.858 7f9873d75700 1 -- 172.21.2.102:0/3571146404 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f987419f0b0 con 0x7f98741b0250 2021-02-10T11:26:40.860 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:40.858 7f9873d75700 1 --2- 172.21.2.102:0/3571146404 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f98741b0250 0x7f98741b13e0 secure :-1 s=READY pgs=1625 cs=0 l=1 rev1=1 rx=0x7f986c009980 tx=0x7f986c00bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:40.860 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:40.858 7f986bfff700 1 -- 172.21.2.102:0/3571146404 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f986c005870 con 0x7f98741b0250 2021-02-10T11:26:40.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:40.858 7f986bfff700 1 -- 172.21.2.102:0/3571146404 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f986c005a70 con 0x7f98741b0250 2021-02-10T11:26:40.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:40.858 7f9879852700 1 -- 172.21.2.102:0/3571146404 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f987419f0b0 con 0x7f98741b0250 2021-02-10T11:26:40.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:40.858 7f9879852700 1 -- 172.21.2.102:0/3571146404 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98741b1cf0 con 0x7f98741b0250 2021-02-10T11:26:40.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:40.858 7f986bfff700 1 -- 172.21.2.102:0/3571146404 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f986c006030 con 0x7f98741b0250 2021-02-10T11:26:40.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:40.859 7f986bfff700 1 -- 172.21.2.102:0/3571146404 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f986c005c50 con 0x7f98741b0250 2021-02-10T11:26:40.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:40.859 7f986bfff700 1 --2- 172.21.2.102:0/3571146404 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f98580186c0 0x7f985801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:40.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:40.859 7f986bfff700 1 -- 172.21.2.102:0/3571146404 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 5221+0+0 (secure 0 0 0) 0x7f986c032a40 con 0x7f98741b0250 2021-02-10T11:26:40.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:40.859 7f9873574700 1 --2- 172.21.2.102:0/3571146404 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f98580186c0 0x7f985801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:40.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:40.859 7f9873574700 1 --2- 172.21.2.102:0/3571146404 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f98580186c0 0x7f985801abc0 crc :-1 s=READY pgs=738 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:40.890 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:40.890 7f9879852700 1 -- 172.21.2.102:0/3571146404 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f987419f0b0 con 0x7f98741b0250 2021-02-10T11:26:40.892 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:40.893 7f986bfff700 1 -- 172.21.2.102:0/3571146404 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f986c005c80 con 0x7f98741b0250 2021-02-10T11:26:41.023 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:41.023 7f9879852700 1 -- 172.21.2.102:0/3571146404 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"pool2": "cephfs_data", "prefix": "osd pool delete", "pool": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f987419f0b0 con 0x7f98741b0250 2021-02-10T11:26:41.396 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:41.397 7f986bfff700 1 -- 172.21.2.102:0/3571146404 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"pool2": "cephfs_data", "prefix": "osd pool delete", "pool": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v69) v1 ==== 173+0+0 (secure 0 0 0) 0x7f986c0062f0 con 0x7f98741b0250 2021-02-10T11:26:41.439 INFO:teuthology.orchestra.run.gibba002.stderr:pool 'cephfs_data' removed 2021-02-10T11:26:41.441 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:41.441 7f9879852700 1 -- 172.21.2.102:0/3571146404 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f98580186c0 msgr2=0x7f985801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:41.441 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:41.441 7f9879852700 1 --2- 172.21.2.102:0/3571146404 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f98580186c0 0x7f985801abc0 crc :-1 s=READY pgs=738 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:41.442 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:41.441 7f9879852700 1 -- 172.21.2.102:0/3571146404 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f98741b0250 msgr2=0x7f98741b13e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:41.442 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:41.441 7f9879852700 1 --2- 172.21.2.102:0/3571146404 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f98741b0250 0x7f98741b13e0 secure :-1 s=READY pgs=1625 cs=0 l=1 rev1=1 rx=0x7f986c009980 tx=0x7f986c00bcd0).stop 2021-02-10T11:26:41.442 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:41.442 7f9879852700 1 -- 172.21.2.102:0/3571146404 shutdown_connections 2021-02-10T11:26:41.442 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:41.442 7f9879852700 1 --2- 172.21.2.102:0/3571146404 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f98741b0250 0x7f98741b13e0 unknown :-1 s=CLOSED pgs=1625 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:41.442 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:41.442 7f9879852700 1 --2- 172.21.2.102:0/3571146404 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f987419e9b0 0x7f98741b0dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:41.442 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:41.442 7f9879852700 1 --2- 172.21.2.102:0/3571146404 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f98580186c0 0x7f985801abc0 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:41.442 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:41.442 7f9879852700 1 --2- 172.21.2.102:0/3571146404 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f987419e5b0 0x7f98741afc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:41.442 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:41.442 7f9879852700 1 -- 172.21.2.102:0/3571146404 >> 172.21.2.102:0/3571146404 conn(0x7f987419f940 msgr2=0x7f987418c290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:26:41.442 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:41.442 7f9879852700 1 -- 172.21.2.102:0/3571146404 shutdown_connections 2021-02-10T11:26:41.442 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:41.442 7f9879852700 1 -- 172.21.2.102:0/3571146404 wait complete. 2021-02-10T11:26:41.450 INFO:tasks.ceph.mds.a:Restarting daemon 2021-02-10T11:26:41.450 INFO:tasks.ceph.mds.a:Stopping old one... 2021-02-10T11:26:41.450 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2021-02-10T11:26:41.450 INFO:teuthology.orchestra.run:waiting for 300 2021-02-10T11:26:41.450 INFO:tasks.ceph.mds.b:Restarting daemon 2021-02-10T11:26:41.451 INFO:tasks.ceph.mds.b:Stopping old one... 2021-02-10T11:26:41.451 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2021-02-10T11:26:41.451 INFO:teuthology.orchestra.run:waiting for 300 2021-02-10T11:26:41.456 INFO:tasks.ceph.mds.a:Stopped 2021-02-10T11:26:41.456 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2021-02-10T11:26:41.457 INFO:tasks.ceph.mds.b:Stopped 2021-02-10T11:26:41.458 DEBUG:teuthology.orchestra.run.gibba013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2021-02-10T11:26:41.498 INFO:tasks.ceph.mds.a:Started 2021-02-10T11:26:41.499 DEBUG:teuthology.parallel:result is None 2021-02-10T11:26:41.500 INFO:tasks.ceph.mds.b:Started 2021-02-10T11:26:41.500 DEBUG:teuthology.parallel:result is None 2021-02-10T11:26:41.500 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd blacklist clear 2021-02-10T11:26:41.906 INFO:tasks.ceph.mds.b.gibba013.stdout:starting mds.b at 2021-02-10T11:26:41.906 INFO:tasks.ceph.mds.a.gibba002.stdout:starting mds.a at 2021-02-10T11:26:41.924 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:41.909 7f8302a13700 1 Processor -- start 2021-02-10T11:26:41.924 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:41.909 7f8302a13700 1 -- start start 2021-02-10T11:26:41.925 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:41.909 7f8302a13700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f82fc0d1870 0x7f82fc1b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:41.925 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:41.909 7f8302a13700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f82fc1b0a10 0x7f82fc0cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:41.925 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:41.909 7f8302a13700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f82fc0d1c70 0x7f82fc0cf950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:41.925 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:41.909 7f8302a13700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82fc0fe200 con 0x7f82fc0d1c70 2021-02-10T11:26:41.925 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:41.909 7f8302a13700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82fc0e5f10 con 0x7f82fc0d1870 2021-02-10T11:26:41.925 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:41.909 7f8302a13700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82fc0f0b50 con 0x7f82fc1b0a10 2021-02-10T11:26:41.925 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:41.909 7f8301030700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f82fc0d1c70 0x7f82fc0cf950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:41.925 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:41.909 7f8301030700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f82fc0d1c70 0x7f82fc0cf950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:58852/0 (socket says 172.21.2.102:58852) 2021-02-10T11:26:41.925 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:41.909 7f8301030700 1 -- 172.21.2.102:0/3030629864 learned_addr learned my addr 172.21.2.102:0/3030629864 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:26:41.925 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:41.909 7f830082f700 1 --2- 172.21.2.102:0/3030629864 >> v2:172.21.2.113:3300/0 conn(0x7f82fc0d1870 0x7f82fc1b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:41.925 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:41.909 7f82f3fff700 1 --2- 172.21.2.102:0/3030629864 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f82fc1b0a10 0x7f82fc0cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:41.925 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:41.909 7f8301030700 1 -- 172.21.2.102:0/3030629864 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f82fc1b0a10 msgr2=0x7f82fc0cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:26:41.926 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:41.909 7f8301030700 1 --2- 172.21.2.102:0/3030629864 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f82fc1b0a10 0x7f82fc0cf330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:41.926 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:41.909 7f8301030700 1 -- 172.21.2.102:0/3030629864 >> v2:172.21.2.113:3300/0 conn(0x7f82fc0d1870 msgr2=0x7f82fc1b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:26:41.926 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:41.910 7f8301030700 1 --2- 172.21.2.102:0/3030629864 >> v2:172.21.2.113:3300/0 conn(0x7f82fc0d1870 0x7f82fc1b03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:41.926 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:41.910 7f8301030700 1 -- 172.21.2.102:0/3030629864 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82fc0cff70 con 0x7f82fc0d1c70 2021-02-10T11:26:41.926 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:41.910 7f830082f700 1 --2- 172.21.2.102:0/3030629864 >> v2:172.21.2.113:3300/0 conn(0x7f82fc0d1870 0x7f82fc1b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:26:41.926 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:41.910 7f8301030700 1 --2- 172.21.2.102:0/3030629864 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f82fc0d1c70 0x7f82fc0cf950 secure :-1 s=READY pgs=1628 cs=0 l=1 rev1=1 rx=0x7f82f4000b10 tx=0x7f82f4009ac0).ready entity=mon.0 client_cookie=b3ae0030fa67e6a9 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:41.926 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:41.910 7f82f37fe700 1 -- 172.21.2.102:0/3030629864 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f82f400c0c0 con 0x7f82fc0d1c70 2021-02-10T11:26:41.926 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:41.910 7f82f37fe700 1 -- 172.21.2.102:0/3030629864 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82f400c440 con 0x7f82fc0d1c70 2021-02-10T11:26:41.926 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:41.910 7f8302a13700 1 -- 172.21.2.102:0/3030629864 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f82fc0d1c70 msgr2=0x7f82fc0cf950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:41.926 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:41.910 7f8302a13700 1 --2- 172.21.2.102:0/3030629864 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f82fc0d1c70 0x7f82fc0cf950 secure :-1 s=READY pgs=1628 cs=0 l=1 rev1=1 rx=0x7f82f4000b10 tx=0x7f82f4009ac0).stop 2021-02-10T11:26:41.926 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:41.910 7f82f37fe700 1 -- 172.21.2.102:0/3030629864 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f82f400c960 con 0x7f82fc0d1c70 2021-02-10T11:26:41.926 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:41.910 7f8302a13700 1 -- 172.21.2.102:0/3030629864 shutdown_connections 2021-02-10T11:26:41.926 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:41.910 7f8302a13700 1 --2- 172.21.2.102:0/3030629864 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f82fc1b0a10 0x7f82fc0cf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:41.927 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:41.910 7f8302a13700 1 --2- 172.21.2.102:0/3030629864 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f82fc0d1c70 0x7f82fc0cf950 unknown :-1 s=CLOSED pgs=1628 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:41.927 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:41.910 7f8302a13700 1 --2- 172.21.2.102:0/3030629864 >> v2:172.21.2.113:3300/0 conn(0x7f82fc0d1870 0x7f82fc1b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:41.927 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:41.910 7f8302a13700 1 -- 172.21.2.102:0/3030629864 >> 172.21.2.102:0/3030629864 conn(0x7f82fc0ce450 msgr2=0x7f82fc0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:26:41.927 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:41.910 7f8302a13700 1 -- 172.21.2.102:0/3030629864 shutdown_connections 2021-02-10T11:26:41.927 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:41.910 7f8302a13700 1 -- 172.21.2.102:0/3030629864 wait complete. 2021-02-10T11:26:41.928 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:41.925 7f8302a13700 1 Processor -- start 2021-02-10T11:26:41.929 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:41.925 7f8302a13700 1 -- start start 2021-02-10T11:26:41.929 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:41.926 7f8302a13700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f82fc189d00 0x7f82fc18a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:41.929 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:41.926 7f8302a13700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f82fc18a720 0x7f82fc18ab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:41.929 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:41.926 7f8302a13700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f82fc187e50 0x7f82fc188250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:41.929 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:41.926 7f8302a13700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82fc0c37e0 con 0x7f82fc18a720 2021-02-10T11:26:41.929 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:41.926 7f8302a13700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82fc0c39d0 con 0x7f82fc187e50 2021-02-10T11:26:41.929 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:41.926 7f8302a13700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82fc188870 con 0x7f82fc189d00 2021-02-10T11:26:41.929 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:41.926 7f82f3fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f82fc18a720 0x7f82fc18ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:41.929 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:41.926 7f82f3fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f82fc18a720 0x7f82fc18ab20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:58856/0 (socket says 172.21.2.102:58856) 2021-02-10T11:26:41.929 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:41.926 7f82f3fff700 1 -- 172.21.2.102:0/3381843872 learned_addr learned my addr 172.21.2.102:0/3381843872 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:26:41.929 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:41.926 7f8301030700 1 --2- 172.21.2.102:0/3381843872 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f82fc187e50 0x7f82fc188250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:41.929 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:41.926 7f830082f700 1 --2- 172.21.2.102:0/3381843872 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f82fc189d00 0x7f82fc18a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:41.930 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:41.926 7f82f3fff700 1 -- 172.21.2.102:0/3381843872 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f82fc189d00 msgr2=0x7f82fc18a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:41.930 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:41.926 7f82f3fff700 1 --2- 172.21.2.102:0/3381843872 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f82fc189d00 0x7f82fc18a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:41.930 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:41.926 7f82f3fff700 1 -- 172.21.2.102:0/3381843872 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f82fc187e50 msgr2=0x7f82fc188250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:41.930 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:41.926 7f82f3fff700 1 --2- 172.21.2.102:0/3381843872 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f82fc187e50 0x7f82fc188250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:41.930 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:41.926 7f82f3fff700 1 -- 172.21.2.102:0/3381843872 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82fc188a90 con 0x7f82fc18a720 2021-02-10T11:26:41.930 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:41.926 7f830082f700 1 --2- 172.21.2.102:0/3381843872 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f82fc189d00 0x7f82fc18a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:26:41.930 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:41.926 7f82f3fff700 1 --2- 172.21.2.102:0/3381843872 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f82fc18a720 0x7f82fc18ab20 secure :-1 s=READY pgs=1629 cs=0 l=1 rev1=1 rx=0x7f82f8009980 tx=0x7f82f800bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:41.930 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:41.926 7f82f17fa700 1 -- 172.21.2.102:0/3381843872 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f82f8006030 con 0x7f82fc18a720 2021-02-10T11:26:41.930 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:41.926 7f82f17fa700 1 -- 172.21.2.102:0/3381843872 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82f8006370 con 0x7f82fc18a720 2021-02-10T11:26:41.930 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:41.926 7f8302a13700 1 -- 172.21.2.102:0/3381843872 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82fc188a90 con 0x7f82fc18a720 2021-02-10T11:26:41.930 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:41.926 7f82f17fa700 1 -- 172.21.2.102:0/3381843872 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f82f8005660 con 0x7f82fc18a720 2021-02-10T11:26:41.930 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:41.926 7f8302a13700 1 -- 172.21.2.102:0/3381843872 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82fc188a90 con 0x7f82fc18a720 2021-02-10T11:26:41.931 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:41.927 7f82f17fa700 1 -- 172.21.2.102:0/3381843872 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f82f8006550 con 0x7f82fc18a720 2021-02-10T11:26:41.931 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:41.927 7f82f17fa700 1 --2- 172.21.2.102:0/3381843872 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f82dc018730 0x7f82dc01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:41.931 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:41.927 7f82f17fa700 1 -- 172.21.2.102:0/3381843872 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 4839+0+0 (secure 0 0 0) 0x7f82f8032740 con 0x7f82fc18a720 2021-02-10T11:26:41.931 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:41.927 7f830082f700 1 --2- 172.21.2.102:0/3381843872 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f82dc018730 0x7f82dc01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:41.931 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:41.927 7f830082f700 1 --2- 172.21.2.102:0/3381843872 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f82dc018730 0x7f82dc01abc0 crc :-1 s=READY pgs=739 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:41.958 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:41.958 7f8302a13700 1 -- 172.21.2.102:0/3381843872 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82fc0ebc50 con 0x7f82fc18a720 2021-02-10T11:26:41.960 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:41.961 7f82f17fa700 1 -- 172.21.2.102:0/3381843872 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f82f800be10 con 0x7f82fc18a720 2021-02-10T11:26:42.100 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:42.100 7f8302a13700 1 -- 172.21.2.102:0/3381843872 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd blacklist clear"} v 0) v1 -- 0x7f82fc0f0b10 con 0x7f82fc18a720 2021-02-10T11:26:42.443 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:42.443 7f82f17fa700 1 -- 172.21.2.102:0/3381843872 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd blacklist clear"}]=0 removed all blacklist entries v70) v1 ==== 97+0+0 (secure 0 0 0) 0x7f82f800be10 con 0x7f82fc18a720 2021-02-10T11:26:42.466 INFO:teuthology.orchestra.run.gibba002.stderr: removed all blacklist entries 2021-02-10T11:26:42.468 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:42.468 7f8302a13700 1 -- 172.21.2.102:0/3381843872 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f82dc018730 msgr2=0x7f82dc01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:42.468 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:42.468 7f8302a13700 1 --2- 172.21.2.102:0/3381843872 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f82dc018730 0x7f82dc01abc0 crc :-1 s=READY pgs=739 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:42.468 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:42.468 7f8302a13700 1 -- 172.21.2.102:0/3381843872 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f82fc18a720 msgr2=0x7f82fc18ab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:42.468 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:42.468 7f8302a13700 1 --2- 172.21.2.102:0/3381843872 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f82fc18a720 0x7f82fc18ab20 secure :-1 s=READY pgs=1629 cs=0 l=1 rev1=1 rx=0x7f82f8009980 tx=0x7f82f800bcd0).stop 2021-02-10T11:26:42.468 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:42.468 7f8302a13700 1 -- 172.21.2.102:0/3381843872 shutdown_connections 2021-02-10T11:26:42.468 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:42.468 7f8302a13700 1 --2- 172.21.2.102:0/3381843872 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f82dc018730 0x7f82dc01abc0 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:42.469 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:42.468 7f8302a13700 1 --2- 172.21.2.102:0/3381843872 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f82fc187e50 0x7f82fc188250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:42.469 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:42.468 7f8302a13700 1 --2- 172.21.2.102:0/3381843872 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f82fc18a720 0x7f82fc18ab20 unknown :-1 s=CLOSED pgs=1629 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:42.469 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:42.468 7f8302a13700 1 --2- 172.21.2.102:0/3381843872 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f82fc189d00 0x7f82fc18a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:42.469 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:42.468 7f8302a13700 1 -- 172.21.2.102:0/3381843872 >> 172.21.2.102:0/3381843872 conn(0x7f82fc0aed50 msgr2=0x7f82fc09c9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:26:42.469 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:42.468 7f8302a13700 1 -- 172.21.2.102:0/3381843872 shutdown_connections 2021-02-10T11:26:42.469 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:42.469 7f8302a13700 1 -- 172.21.2.102:0/3381843872 wait complete. 2021-02-10T11:26:42.476 INFO:tasks.cephfs.cephfs_test_case:['0', '1', '2', '3'] 2021-02-10T11:26:42.477 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth ls --format=json-pretty 2021-02-10T11:26:42.905 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:42.887 7fc206f1e700 1 Processor -- start 2021-02-10T11:26:42.905 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:42.887 7fc206f1e700 1 -- start start 2021-02-10T11:26:42.905 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:42.888 7fc206f1e700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc2000d1870 0x7fc2001b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:42.905 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:42.888 7fc206f1e700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fc2001b0a10 0x7fc2000cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:42.906 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:42.888 7fc206f1e700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2000f0b10 con 0x7fc2000d1c70 2021-02-10T11:26:42.906 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:42.888 7fc206f1e700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2000fe230 con 0x7fc2001b0a10 2021-02-10T11:26:42.906 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:42.888 7fc206f1e700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2000e5ed0 con 0x7fc2000d1870 2021-02-10T11:26:42.906 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:42.889 7fc204d3a700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc2000d1870 0x7fc2001b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:42.906 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:42.889 7fc204d3a700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc2000d1870 0x7fc2001b03f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.113:3301/0 says I am v2:172.21.2.102:36170/0 (socket says 172.21.2.102:36170) 2021-02-10T11:26:42.906 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:42.889 7fc204d3a700 1 -- 172.21.2.102:0/2403954814 learned_addr learned my addr 172.21.2.102:0/2403954814 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:26:42.906 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:42.889 7fc1fffff700 1 --2- 172.21.2.102:0/2403954814 >> v2:172.21.2.113:3300/0 conn(0x7fc2001b0a10 0x7fc2000cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:42.906 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:42.890 7fc204d3a700 1 -- 172.21.2.102:0/2403954814 >> v2:172.21.2.113:3300/0 conn(0x7fc2001b0a10 msgr2=0x7fc2000cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:26:42.906 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:42.890 7fc204d3a700 1 --2- 172.21.2.102:0/2403954814 >> v2:172.21.2.113:3300/0 conn(0x7fc2001b0a10 0x7fc2000cf330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:42.906 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:42.890 7fc204d3a700 1 -- 172.21.2.102:0/2403954814 >> v1:172.21.2.113:6789/0 conn(0x7fc2000d1c70 legacy=0x7fc2000cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:42.906 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:42.890 7fc204d3a700 1 -- 172.21.2.102:0/2403954814 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2000f0650 con 0x7fc2000d1870 2021-02-10T11:26:42.906 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:42.890 7fc204d3a700 1 --2- 172.21.2.102:0/2403954814 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc2000d1870 0x7fc2001b03f0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7fc1e8006700 tx=0x7fc1e8009b80).ready entity=mon.2 client_cookie=6ba673074afb8bb0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:42.907 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:42.890 7fc1ff7fe700 1 -- 172.21.2.102:0/2403954814 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc1e800c0c0 con 0x7fc2000d1870 2021-02-10T11:26:42.907 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:42.890 7fc1ff7fe700 1 -- 172.21.2.102:0/2403954814 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1e800c340 con 0x7fc2000d1870 2021-02-10T11:26:42.907 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:42.890 7fc1ff7fe700 1 -- 172.21.2.102:0/2403954814 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc1e800c860 con 0x7fc2000d1870 2021-02-10T11:26:42.907 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:42.891 7fc206f1e700 1 -- 172.21.2.102:0/2403954814 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc2000d1870 msgr2=0x7fc2001b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:42.907 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:42.891 7fc206f1e700 1 --2- 172.21.2.102:0/2403954814 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc2000d1870 0x7fc2001b03f0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7fc1e8006700 tx=0x7fc1e8009b80).stop 2021-02-10T11:26:42.907 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:42.891 7fc206f1e700 1 -- 172.21.2.102:0/2403954814 shutdown_connections 2021-02-10T11:26:42.907 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:42.891 7fc206f1e700 1 --2- 172.21.2.102:0/2403954814 >> v2:172.21.2.113:3300/0 conn(0x7fc2001b0a10 0x7fc2000cf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:42.907 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:42.891 7fc206f1e700 1 --2- 172.21.2.102:0/2403954814 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc2000d1870 0x7fc2001b03f0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:42.907 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:42.891 7fc206f1e700 1 -- 172.21.2.102:0/2403954814 >> 172.21.2.102:0/2403954814 conn(0x7fc2000ce450 msgr2=0x7fc2000d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:26:42.907 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:42.891 7fc206f1e700 1 -- 172.21.2.102:0/2403954814 shutdown_connections 2021-02-10T11:26:42.907 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:42.891 7fc206f1e700 1 -- 172.21.2.102:0/2403954814 wait complete. 2021-02-10T11:26:42.909 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:42.906 7fc206f1e700 1 Processor -- start 2021-02-10T11:26:42.909 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:42.906 7fc206f1e700 1 -- start start 2021-02-10T11:26:42.909 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:42.906 7fc206f1e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc2000ada90 0x7fc2000ade90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:42.909 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:42.906 7fc206f1e700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc200099d20 0x7fc20009a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:42.909 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:42.906 7fc206f1e700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc20009a740 0x7fc200097bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:42.909 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:42.906 7fc206f1e700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2000997a0 con 0x7fc2000ada90 2021-02-10T11:26:42.909 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:42.906 7fc206f1e700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc20018f0c0 con 0x7fc20009a740 2021-02-10T11:26:42.909 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:42.906 7fc206f1e700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc20009ab40 con 0x7fc200099d20 2021-02-10T11:26:42.909 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:42.906 7fc204d3a700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc2000ada90 0x7fc2000ade90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:42.910 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:42.906 7fc204d3a700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc2000ada90 0x7fc2000ade90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:58874/0 (socket says 172.21.2.102:58874) 2021-02-10T11:26:42.910 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:42.906 7fc204d3a700 1 -- 172.21.2.102:0/1090825510 learned_addr learned my addr 172.21.2.102:0/1090825510 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:26:42.910 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:42.907 7fc204d3a700 1 -- 172.21.2.102:0/1090825510 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc200099d20 msgr2=0x7fc20009a120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:42.910 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:42.907 7fc204d3a700 1 --2- 172.21.2.102:0/1090825510 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc200099d20 0x7fc20009a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:42.910 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:42.907 7fc204d3a700 1 -- 172.21.2.102:0/1090825510 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc20009a740 msgr2=0x7fc200097bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:42.910 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:42.907 7fc204d3a700 1 --2- 172.21.2.102:0/1090825510 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc20009a740 0x7fc200097bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:42.910 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:42.907 7fc204d3a700 1 -- 172.21.2.102:0/1090825510 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc20009ad60 con 0x7fc2000ada90 2021-02-10T11:26:42.910 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:42.907 7fc204d3a700 1 --2- 172.21.2.102:0/1090825510 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc2000ada90 0x7fc2000ade90 secure :-1 s=READY pgs=1630 cs=0 l=1 rev1=1 rx=0x7fc1e800b5e0 tx=0x7fc1e800b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:42.910 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:42.907 7fc1fd7fa700 1 -- 172.21.2.102:0/1090825510 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc1e800bd70 con 0x7fc2000ada90 2021-02-10T11:26:42.910 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:42.907 7fc1fd7fa700 1 -- 172.21.2.102:0/1090825510 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1e8002030 con 0x7fc2000ada90 2021-02-10T11:26:42.910 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:42.907 7fc1fd7fa700 1 -- 172.21.2.102:0/1090825510 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc1e8002550 con 0x7fc2000ada90 2021-02-10T11:26:42.910 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:42.907 7fc206f1e700 1 -- 172.21.2.102:0/1090825510 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc20009ad60 con 0x7fc2000ada90 2021-02-10T11:26:42.910 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:42.907 7fc206f1e700 1 -- 172.21.2.102:0/1090825510 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc20009ad60 con 0x7fc2000ada90 2021-02-10T11:26:42.911 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:42.908 7fc1fd7fa700 1 -- 172.21.2.102:0/1090825510 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fc1e8000c90 con 0x7fc2000ada90 2021-02-10T11:26:42.911 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:42.908 7fc1fd7fa700 1 --2- 172.21.2.102:0/1090825510 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc1e40186c0 0x7fc1e401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:42.911 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:42.908 7fc1fd7fa700 1 -- 172.21.2.102:0/1090825510 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 4753+0+0 (secure 0 0 0) 0x7fc1e8032270 con 0x7fc2000ada90 2021-02-10T11:26:42.911 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:42.908 7fc1fffff700 1 --2- 172.21.2.102:0/1090825510 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc1e40186c0 0x7fc1e401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:42.911 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:42.908 7fc1fffff700 1 --2- 172.21.2.102:0/1090825510 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc1e40186c0 0x7fc1e401abc0 crc :-1 s=READY pgs=740 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:42.938 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:42.938 7fc206f1e700 1 -- 172.21.2.102:0/1090825510 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2000ebc00 con 0x7fc2000ada90 2021-02-10T11:26:42.940 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:42.940 7fc1fd7fa700 1 -- 172.21.2.102:0/1090825510 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fc1e80166d0 con 0x7fc2000ada90 2021-02-10T11:26:43.071 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:43.071 7fc206f1e700 1 -- 172.21.2.102:0/1090825510 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fc2000ebc00 con 0x7fc2000ada90 2021-02-10T11:26:43.071 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:43.072 7fc1fd7fa700 1 -- 172.21.2.102:0/1090825510 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v42) v1 ==== 80+0+4513 (secure 0 0 0) 0x7fc1e80166d0 con 0x7fc2000ada90 2021-02-10T11:26:43.072 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:26:43.072 INFO:teuthology.orchestra.run.gibba002.stdout:{ 2021-02-10T11:26:43.072 INFO:teuthology.orchestra.run.gibba002.stdout: "auth_dump": [ 2021-02-10T11:26:43.072 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:26:43.072 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "mds.a", 2021-02-10T11:26:43.072 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAevSNgsgyEARAAvr0cUjjCoEHLxaZSZ2HtVA==", 2021-02-10T11:26:43.072 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:26:43.072 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow", 2021-02-10T11:26:43.073 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:26:43.073 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:26:43.073 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:26:43.073 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:26:43.073 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:26:43.073 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:26:43.073 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "mds.b", 2021-02-10T11:26:43.073 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAevSNgwOFMMRAALfb1fbX56ao1TU8ClnTW1A==", 2021-02-10T11:26:43.073 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:26:43.073 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow", 2021-02-10T11:26:43.073 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:26:43.073 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:26:43.073 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:26:43.074 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:26:43.074 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:26:43.074 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:26:43.074 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.0", 2021-02-10T11:26:43.074 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAmvSNgj+bMMBAAD3HUD6fLiMAg+++iNzLJKQ==", 2021-02-10T11:26:43.074 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:26:43.074 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:26:43.074 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:26:43.074 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:26:43.074 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:26:43.074 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:26:43.074 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:26:43.075 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.1", 2021-02-10T11:26:43.075 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQApvSNgnvc+MRAAAOxqlNwiyLHqNITOrYeFKQ==", 2021-02-10T11:26:43.075 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:26:43.075 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:26:43.075 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:26:43.075 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:26:43.075 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:26:43.075 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:26:43.075 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:26:43.075 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.2", 2021-02-10T11:26:43.075 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAsvSNgZ4SbMRAA76IOYWfbiAuBoM/4k81GxA==", 2021-02-10T11:26:43.075 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:26:43.076 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:26:43.076 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:26:43.076 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:26:43.076 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:26:43.076 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:26:43.076 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:26:43.076 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.3", 2021-02-10T11:26:43.076 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAvvSNgFk3rMRAAWCmKKyw0LDQ+UlAHZAITOw==", 2021-02-10T11:26:43.076 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:26:43.076 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:26:43.076 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:26:43.076 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:26:43.077 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:26:43.077 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:26:43.077 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:26:43.077 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.4", 2021-02-10T11:26:43.077 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQA4vSNgpKcNJBAApJMmvj5YxVHPDOmvmQONOg==", 2021-02-10T11:26:43.077 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:26:43.077 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:26:43.077 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:26:43.077 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:26:43.077 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:26:43.077 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:26:43.077 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:26:43.077 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.5", 2021-02-10T11:26:43.077 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQA7vSNgf2ujJBAAfkuYpyb3WTzc8V5kw+Wktg==", 2021-02-10T11:26:43.078 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:26:43.078 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:26:43.078 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:26:43.078 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:26:43.078 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:26:43.078 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:26:43.078 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:26:43.078 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.6", 2021-02-10T11:26:43.078 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQA+vSNgqw9uJRAAFMU0R+UxU9C3o/9byb3e0g==", 2021-02-10T11:26:43.078 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:26:43.078 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:26:43.078 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:26:43.078 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:26:43.079 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:26:43.079 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:26:43.079 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:26:43.079 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.7", 2021-02-10T11:26:43.079 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQBBvSNgvWSrJRAA2KaCHmLpzovRgYzLvyofTQ==", 2021-02-10T11:26:43.079 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:26:43.079 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:26:43.079 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:26:43.079 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:26:43.079 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:26:43.079 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:26:43.079 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:26:43.080 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "client.0", 2021-02-10T11:26:43.080 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAfvSNgZd9XIBAAvyrvl8fdKuHnJZfczfRoFw==", 2021-02-10T11:26:43.080 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:26:43.080 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow", 2021-02-10T11:26:43.080 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow r", 2021-02-10T11:26:43.080 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:26:43.080 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:26:43.080 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:26:43.080 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:26:43.080 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "client.1", 2021-02-10T11:26:43.080 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAfvSNgSyfXNhAAIduPnF7Vd/6HSkJ0MlcMSA==", 2021-02-10T11:26:43.080 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:26:43.080 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow", 2021-02-10T11:26:43.080 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow r", 2021-02-10T11:26:43.081 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:26:43.081 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:26:43.081 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:26:43.081 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:26:43.081 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "client.2", 2021-02-10T11:26:43.081 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAgvSNgiIG/ERAA9ZTamaTI7ZsPSnxBqjplCg==", 2021-02-10T11:26:43.081 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:26:43.081 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow", 2021-02-10T11:26:43.081 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow r", 2021-02-10T11:26:43.081 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:26:43.081 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:26:43.081 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:26:43.081 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:26:43.081 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "client.3", 2021-02-10T11:26:43.082 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAgvSNgOWpOKBAAlXsEB04Q/GvtL06PqRMLeA==", 2021-02-10T11:26:43.082 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:26:43.082 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow", 2021-02-10T11:26:43.082 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow r", 2021-02-10T11:26:43.082 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:26:43.082 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:26:43.082 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:26:43.082 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:26:43.082 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "client.admin", 2021-02-10T11:26:43.083 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAavSNgO3GQBhAAL+NVnYskQRKVWLgyDPY/xw==", 2021-02-10T11:26:43.083 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:26:43.083 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow *", 2021-02-10T11:26:43.083 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:26:43.083 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:26:43.083 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:26:43.083 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:26:43.083 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:26:43.083 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:26:43.083 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "mgr.x", 2021-02-10T11:26:43.083 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAdvSNgXZZgJhAA7dtBSGzH+DsBwnddZpOBJA==", 2021-02-10T11:26:43.084 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:26:43.084 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow *", 2021-02-10T11:26:43.084 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow profile mgr", 2021-02-10T11:26:43.084 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:26:43.084 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:26:43.084 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:26:43.084 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:26:43.084 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "mgr.y", 2021-02-10T11:26:43.084 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAdvSNgVqQFDRAAwo6YEdWZCAr5GdOWVL/Qvg==", 2021-02-10T11:26:43.084 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:26:43.084 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow *", 2021-02-10T11:26:43.084 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow profile mgr", 2021-02-10T11:26:43.085 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:26:43.085 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:26:43.085 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:26:43.085 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:26:43.085 INFO:teuthology.orchestra.run.gibba002.stdout:} 2021-02-10T11:26:43.086 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:43.073 7fc206f1e700 1 -- 172.21.2.102:0/1090825510 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc1e40186c0 msgr2=0x7fc1e401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:43.086 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:43.073 7fc206f1e700 1 --2- 172.21.2.102:0/1090825510 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc1e40186c0 0x7fc1e401abc0 crc :-1 s=READY pgs=740 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:43.087 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:43.073 7fc206f1e700 1 -- 172.21.2.102:0/1090825510 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc2000ada90 msgr2=0x7fc2000ade90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:43.087 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:43.073 7fc206f1e700 1 --2- 172.21.2.102:0/1090825510 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc2000ada90 0x7fc2000ade90 secure :-1 s=READY pgs=1630 cs=0 l=1 rev1=1 rx=0x7fc1e800b5e0 tx=0x7fc1e800b980).stop 2021-02-10T11:26:43.087 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:43.074 7fc206f1e700 1 -- 172.21.2.102:0/1090825510 shutdown_connections 2021-02-10T11:26:43.087 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:43.074 7fc206f1e700 1 --2- 172.21.2.102:0/1090825510 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc1e40186c0 0x7fc1e401abc0 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:43.087 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:43.074 7fc206f1e700 1 --2- 172.21.2.102:0/1090825510 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc20009a740 0x7fc200097bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:43.087 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:43.074 7fc206f1e700 1 --2- 172.21.2.102:0/1090825510 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc200099d20 0x7fc20009a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:43.087 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:43.074 7fc206f1e700 1 --2- 172.21.2.102:0/1090825510 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc2000ada90 0x7fc2000ade90 unknown :-1 s=CLOSED pgs=1630 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:43.087 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:43.074 7fc206f1e700 1 -- 172.21.2.102:0/1090825510 >> 172.21.2.102:0/1090825510 conn(0x7fc2000f0550 msgr2=0x7fc200191730 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:26:43.087 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:43.074 7fc206f1e700 1 -- 172.21.2.102:0/1090825510 shutdown_connections 2021-02-10T11:26:43.087 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:43.074 7fc206f1e700 1 -- 172.21.2.102:0/1090825510 wait complete. 2021-02-10T11:26:43.088 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:26:43.515 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:43.499 7f43b3746700 1 Processor -- start 2021-02-10T11:26:43.515 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:43.499 7f43b3746700 1 -- start start 2021-02-10T11:26:43.515 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:43.500 7f43b3746700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f43ac0d1c70 0x7f43ac0f20a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:43.515 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:43.500 7f43b3746700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43ac0f0b10 con 0x7f43ac0d1870 2021-02-10T11:26:43.515 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:43.500 7f43b3746700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43ac1b0be0 con 0x7f43ac0d0340 2021-02-10T11:26:43.516 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:43.500 7f43b3746700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43ac0e5ed0 con 0x7f43ac0d1c70 2021-02-10T11:26:43.516 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:43.500 7f43b1d63700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f43ac0d1c70 0x7f43ac0f20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:43.516 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:43.500 7f43b1d63700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f43ac0d1c70 0x7f43ac0f20a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:58878/0 (socket says 172.21.2.102:58878) 2021-02-10T11:26:43.516 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:43.500 7f43b1d63700 1 -- 172.21.2.102:0/4132651829 learned_addr learned my addr 172.21.2.102:0/4132651829 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:26:43.516 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:43.500 7f43b1562700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f43ac0d1870 0x7f43ac1b03f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:51102/0 (socket says 172.21.2.102:51102) 2021-02-10T11:26:43.516 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:43.500 7f43b1d63700 1 -- 172.21.2.102:0/4132651829 >> v1:172.21.2.113:6789/0 conn(0x7f43ac0d0340 legacy=0x7f43ac0ef640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:43.516 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:43.500 7f43b1d63700 1 -- 172.21.2.102:0/4132651829 >> v1:172.21.2.102:6789/0 conn(0x7f43ac0d1870 legacy=0x7f43ac1b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:43.516 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:43.500 7f43b1d63700 1 -- 172.21.2.102:0/4132651829 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43ac0efe30 con 0x7f43ac0d1c70 2021-02-10T11:26:43.516 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:43.500 7f43b1d63700 1 --2- 172.21.2.102:0/4132651829 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f43ac0d1c70 0x7f43ac0f20a0 secure :-1 s=READY pgs=1632 cs=0 l=1 rev1=1 rx=0x7f43a4006f00 tx=0x7f43a4009f10).ready entity=mon.0 client_cookie=f5559eb63afb120b server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:43.516 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:43.500 7f439ffff700 1 -- 172.21.2.102:0/4132651829 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f43a400b3b0 con 0x7f43ac0d1c70 2021-02-10T11:26:43.516 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:43.500 7f439ffff700 1 -- 172.21.2.102:0/4132651829 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43a400b630 con 0x7f43ac0d1c70 2021-02-10T11:26:43.516 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:43.500 7f439ffff700 1 -- 172.21.2.102:0/4132651829 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f43a400bb50 con 0x7f43ac0d1c70 2021-02-10T11:26:43.516 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:43.501 7f43b3746700 1 -- 172.21.2.102:0/4132651829 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f43ac0d1c70 msgr2=0x7f43ac0f20a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:43.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:43.501 7f43b3746700 1 --2- 172.21.2.102:0/4132651829 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f43ac0d1c70 0x7f43ac0f20a0 secure :-1 s=READY pgs=1632 cs=0 l=1 rev1=1 rx=0x7f43a4006f00 tx=0x7f43a4009f10).stop 2021-02-10T11:26:43.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:43.501 7f43b3746700 1 -- 172.21.2.102:0/4132651829 shutdown_connections 2021-02-10T11:26:43.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:43.501 7f43b3746700 1 --2- 172.21.2.102:0/4132651829 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f43ac0d1c70 0x7f43ac0f20a0 unknown :-1 s=CLOSED pgs=1632 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:43.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:43.501 7f43b3746700 1 -- 172.21.2.102:0/4132651829 >> 172.21.2.102:0/4132651829 conn(0x7f43ac0ce450 msgr2=0x7f43ac0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:26:43.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:43.501 7f43b3746700 1 -- 172.21.2.102:0/4132651829 shutdown_connections 2021-02-10T11:26:43.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:43.501 7f43b3746700 1 -- 172.21.2.102:0/4132651829 wait complete. 2021-02-10T11:26:43.518 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:43.516 7f43b3746700 1 Processor -- start 2021-02-10T11:26:43.518 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:43.516 7f43b3746700 1 -- start start 2021-02-10T11:26:43.519 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:43.516 7f43b3746700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f43ac0af040 0x7f43ac0af440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:43.519 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:43.516 7f43b3746700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f43ac0afa60 0x7f43ac0ada20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:43.519 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:43.516 7f43b3746700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f43ac0afe60 0x7f43ac0ae040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:43.519 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:43.516 7f43b3746700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43ac0923c0 con 0x7f43ac0af040 2021-02-10T11:26:43.519 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:43.516 7f43b3746700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43ac0b0410 con 0x7f43ac0afe60 2021-02-10T11:26:43.519 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:43.516 7f43b3746700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43ac091e40 con 0x7f43ac0afa60 2021-02-10T11:26:43.519 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:43.516 7f43b1562700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f43ac0af040 0x7f43ac0af440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:43.519 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:43.516 7f43b1562700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f43ac0af040 0x7f43ac0af440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:58882/0 (socket says 172.21.2.102:58882) 2021-02-10T11:26:43.519 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:43.516 7f43b1562700 1 -- 172.21.2.102:0/987369648 learned_addr learned my addr 172.21.2.102:0/987369648 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:26:43.519 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:43.517 7f43b1562700 1 -- 172.21.2.102:0/987369648 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f43ac0afa60 msgr2=0x7f43ac0ada20 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:26:43.519 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:43.517 7f43b1562700 1 --2- 172.21.2.102:0/987369648 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f43ac0afa60 0x7f43ac0ada20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:43.519 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:43.517 7f43b1562700 1 -- 172.21.2.102:0/987369648 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f43ac0afe60 msgr2=0x7f43ac0ae040 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:26:43.519 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:43.517 7f43b1562700 1 --2- 172.21.2.102:0/987369648 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f43ac0afe60 0x7f43ac0ae040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:43.520 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:43.517 7f43b1562700 1 -- 172.21.2.102:0/987369648 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43ac08f6d0 con 0x7f43ac0af040 2021-02-10T11:26:43.520 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:43.517 7f43b1562700 1 --2- 172.21.2.102:0/987369648 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f43ac0af040 0x7f43ac0af440 secure :-1 s=READY pgs=1633 cs=0 l=1 rev1=1 rx=0x7f43a0001c60 tx=0x7f43a000a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:43.520 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:43.517 7f439dffb700 1 -- 172.21.2.102:0/987369648 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f43a000acc0 con 0x7f43ac0af040 2021-02-10T11:26:43.520 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:43.517 7f439dffb700 1 -- 172.21.2.102:0/987369648 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43a0004140 con 0x7f43ac0af040 2021-02-10T11:26:43.520 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:43.517 7f439dffb700 1 -- 172.21.2.102:0/987369648 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f43a0004620 con 0x7f43ac0af040 2021-02-10T11:26:43.520 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:43.517 7f43b3746700 1 -- 172.21.2.102:0/987369648 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43ac08f6d0 con 0x7f43ac0af040 2021-02-10T11:26:43.520 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:43.517 7f43b3746700 1 -- 172.21.2.102:0/987369648 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43ac0ae950 con 0x7f43ac0af040 2021-02-10T11:26:43.520 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:43.518 7f439dffb700 1 -- 172.21.2.102:0/987369648 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f43a0004320 con 0x7f43ac0af040 2021-02-10T11:26:43.520 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:43.518 7f439dffb700 1 --2- 172.21.2.102:0/987369648 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f438c0186c0 0x7f438c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:43.520 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:43.518 7f439dffb700 1 -- 172.21.2.102:0/987369648 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 4753+0+0 (secure 0 0 0) 0x7f43a0032260 con 0x7f43ac0af040 2021-02-10T11:26:43.520 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:43.518 7f43b0d61700 1 --2- 172.21.2.102:0/987369648 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f438c0186c0 0x7f438c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:43.520 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:43.518 7f43b0d61700 1 --2- 172.21.2.102:0/987369648 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f438c0186c0 0x7f438c01abc0 crc :-1 s=READY pgs=741 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:43.549 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:43.549 7f43b3746700 1 -- 172.21.2.102:0/987369648 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43ac0aec00 con 0x7f43ac0af040 2021-02-10T11:26:43.552 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:43.552 7f439dffb700 1 -- 172.21.2.102:0/987369648 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f43a0000dd0 con 0x7f43ac0af040 2021-02-10T11:26:43.676 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:43.676 7f43b3746700 1 -- 172.21.2.102:0/987369648 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f43ac0aec00 con 0x7f43ac0af040 2021-02-10T11:26:43.677 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:43.677 7f439dffb700 1 -- 172.21.2.102:0/987369648 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v70) v1 ==== 74+0+11145 (secure 0 0 0) 0x7f43a0004300 con 0x7f43ac0af040 2021-02-10T11:26:43.677 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:26:43.677 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":70,"fsid":"68db30ea-dbce-4aef-a6df-f01c6fea0513","created":"2021-02-10 11:03:06.331557","modified":"2021-02-10 11:26:42.440489","last_up_change":"2021-02-10 11:03:32.690562","last_in_change":"2021-02-10 11:03:32.690562","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:03:37.322519","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"be691b4d-a039-408a-af80-9d53f02b6f1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6818","nonce":36088},{"type":"v1","addr":"172.21.2.102:6820","nonce":36088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6822","nonce":36088},{"type":"v1","addr":"172.21.2.102:6824","nonce":36088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6830","nonce":36088},{"type":"v1","addr":"172.21.2.102:6832","nonce":36088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6826","nonce":36088},{"type":"v1","addr":"172.21.2.102:6828","nonce":36088}]},"public_addr":"172.21.2.102:6820/36088","cluster_addr":"172.21.2.102:6824/36088","heartbeat_back_addr":"172.21.2.102:6832/36088","heartbeat_front_addr":"172.21.2.102:6828/36088","state":["exists","up"]},{"osd":1,"uuid":"64e3ccc8-9c69-4d7c-865e-af8e83822635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6819","nonce":36087},{"type":"v1","addr":"172.21.2.102:6821","nonce":36087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6823","nonce":36087},{"type":"v1","addr":"172.21.2.102:6825","nonce":36087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6831","nonce":36087},{"type":"v1","addr":"172.21.2.102:6833","nonce":36087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6827","nonce":36087},{"type":"v1","addr":"172.21.2.102:6829","nonce":36087}]},"public_addr":"172.21.2.102:6821/36087","cluster_addr":"172.21.2.102:6825/36087","heartbeat_back_addr":"172.21.2.102:6833/36087","heartbeat_front_addr":"172.21.2.102:6829/36087","state":["exists","up"]},{"osd":2,"uuid":"153700af-747f-481f-bdf4-d6072d3a842f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6810","nonce":36089},{"type":"v1","addr":"172.21.2.102:6811","nonce":36089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6812","nonce":36089},{"type":"v1","addr":"172.21.2.102:6813","nonce":36089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6816","nonce":36089},{"type":"v1","addr":"172.21.2.102:6817","nonce":36089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6814","nonce":36089},{"type":"v1","addr":"172.21.2.102:6815","nonce":36089}]},"public_addr":"172.21.2.102:6811/36089","cluster_addr":"172.21.2.102:6813/36089","heartbeat_back_addr":"172.21.2.102:6817/36089","heartbeat_front_addr":"172.21.2.102:6815/36089","state":["exists","up"]},{"osd":3,"uuid":"cbc678f4-83a9-45e6-8830-2f7c232b1ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6802","nonce":36090},{"type":"v1","addr":"172.21.2.102:6803","nonce":36090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6804","nonce":36090},{"type":"v1","addr":"172.21.2.102:6805","nonce":36090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6808","nonce":36090},{"type":"v1","addr":"172.21.2.102:6809","nonce":36090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6806","nonce":36090},{"type":"v1","addr":"172.21.2.102:6807","nonce":36090}]},"public_addr":"172.21.2.102:6803/36090","cluster_addr":"172.21.2.102:6805/36090","heartbeat_back_addr":"172.21.2.102:6809/36090","heartbeat_front_addr":"172.21.2.102:6807/36090","state":["exists","up"]},{"osd":4,"uuid":"c01e4008-b258-4f10-afd2-9e7f37214351","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6808","nonce":36368},{"type":"v1","addr":"172.21.2.113:6811","nonce":36368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6814","nonce":36368},{"type":"v1","addr":"172.21.2.113:6818","nonce":36368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6826","nonce":36368},{"type":"v1","addr":"172.21.2.113:6828","nonce":36368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6820","nonce":36368},{"type":"v1","addr":"172.21.2.113:6824","nonce":36368}]},"public_addr":"172.21.2.113:6811/36368","cluster_addr":"172.21.2.113:6818/36368","heartbeat_back_addr":"172.21.2.113:6828/36368","heartbeat_front_addr":"172.21.2.113:6824/36368","state":["exists","up"]},{"osd":5,"uuid":"551f6007-daf8-4a98-8ee0-c2b16bcccb5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6802","nonce":36366},{"type":"v1","addr":"172.21.2.113:6805","nonce":36366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6807","nonce":36366},{"type":"v1","addr":"172.21.2.113:6810","nonce":36366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6819","nonce":36366},{"type":"v1","addr":"172.21.2.113:6822","nonce":36366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6812","nonce":36366},{"type":"v1","addr":"172.21.2.113:6815","nonce":36366}]},"public_addr":"172.21.2.113:6805/36366","cluster_addr":"172.21.2.113:6810/36366","heartbeat_back_addr":"172.21.2.113:6822/36366","heartbeat_front_addr":"172.21.2.113:6815/36366","state":["exists","up"]},{"osd":6,"uuid":"bf67c040-0e50-48be-b102-c3943bc9ebef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6800","nonce":36367},{"type":"v1","addr":"172.21.2.113:6801","nonce":36367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6803","nonce":36367},{"type":"v1","addr":"172.21.2.113:6804","nonce":36367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6813","nonce":36367},{"type":"v1","addr":"172.21.2.113:6816","nonce":36367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6806","nonce":36367},{"type":"v1","addr":"172.21.2.113:6809","nonce":36367}]},"public_addr":"172.21.2.113:6801/36367","cluster_addr":"172.21.2.113:6804/36367","heartbeat_back_addr":"172.21.2.113:6816/36367","heartbeat_front_addr":"172.21.2.113:6809/36367","state":["exists","up"]},{"osd":7,"uuid":"f5461f80-018d-428f-be89-2eb2bcbe7bb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6817","nonce":36365},{"type":"v1","addr":"172.21.2.113:6821","nonce":36365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6823","nonce":36365},{"type":"v1","addr":"172.21.2.113:6825","nonce":36365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6830","nonce":36365},{"type":"v1","addr":"172.21.2.113:6831","nonce":36365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6827","nonce":36365},{"type":"v1","addr":"172.21.2.113:6829","nonce":36365}]},"public_addr":"172.21.2.113:6821/36365","cluster_addr":"172.21.2.113:6825/36365","heartbeat_back_addr":"172.21.2.113:6831/36365","heartbeat_front_addr":"172.21.2.113:6829/36365","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:26:43.678 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:43.679 7f43b3746700 1 -- 172.21.2.102:0/987369648 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f438c0186c0 msgr2=0x7f438c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:43.678 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:43.679 7f43b3746700 1 --2- 172.21.2.102:0/987369648 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f438c0186c0 0x7f438c01abc0 crc :-1 s=READY pgs=741 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:43.679 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:43.679 7f43b3746700 1 -- 172.21.2.102:0/987369648 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f43ac0af040 msgr2=0x7f43ac0af440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:43.679 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:43.679 7f43b3746700 1 --2- 172.21.2.102:0/987369648 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f43ac0af040 0x7f43ac0af440 secure :-1 s=READY pgs=1633 cs=0 l=1 rev1=1 rx=0x7f43a0001c60 tx=0x7f43a000a8d0).stop 2021-02-10T11:26:43.680 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:43.679 7f43b3746700 1 -- 172.21.2.102:0/987369648 shutdown_connections 2021-02-10T11:26:43.680 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:43.679 7f43b3746700 1 --2- 172.21.2.102:0/987369648 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f438c0186c0 0x7f438c01abc0 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:43.680 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:43.679 7f43b3746700 1 --2- 172.21.2.102:0/987369648 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f43ac0afe60 0x7f43ac0ae040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:43.680 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:43.679 7f43b3746700 1 --2- 172.21.2.102:0/987369648 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f43ac0afa60 0x7f43ac0ada20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:43.680 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:43.679 7f43b3746700 1 --2- 172.21.2.102:0/987369648 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f43ac0af040 0x7f43ac0af440 unknown :-1 s=CLOSED pgs=1633 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:43.680 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:43.679 7f43b3746700 1 -- 172.21.2.102:0/987369648 >> 172.21.2.102:0/987369648 conn(0x7f43ac0a0030 msgr2=0x7f43ac0f22b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:26:43.680 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:43.679 7f43b3746700 1 -- 172.21.2.102:0/987369648 shutdown_connections 2021-02-10T11:26:43.680 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:43.679 7f43b3746700 1 -- 172.21.2.102:0/987369648 wait complete. 2021-02-10T11:26:43.688 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:03:37.322519', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2021-02-10T11:26:43.688 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:26:44.124 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:44.103 7ff21029e700 1 Processor -- start 2021-02-10T11:26:44.124 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:44.103 7ff21029e700 1 -- start start 2021-02-10T11:26:44.125 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:44.104 7ff21029e700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff2080d1870 0x7ff2081b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:44.125 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:44.104 7ff21029e700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff208109a60 con 0x7ff2080cff70 2021-02-10T11:26:44.125 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:44.104 7ff21029e700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff208088070 con 0x7ff2081b0ae0 2021-02-10T11:26:44.125 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:44.104 7ff21029e700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2080e5ef0 con 0x7ff2080d1870 2021-02-10T11:26:44.125 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:44.104 7ff20e8bb700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7ff2080cff70 0x7ff208109130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:51114/0 (socket says 172.21.2.102:51114) 2021-02-10T11:26:44.125 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:44.104 7ff20e8bb700 1 -- 172.21.2.102:0/353413511 learned_addr learned my addr 172.21.2.102:0/353413511 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:26:44.126 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:44.104 7ff20d0b8700 1 -- 172.21.2.102:0/353413511 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2850109144 0 0) 0x7ff200001500 con 0x7ff2080cff70 2021-02-10T11:26:44.126 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:44.104 7ff20d0b8700 1 -- 172.21.2.102:0/353413511 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1f4002fe0 con 0x7ff2080cff70 2021-02-10T11:26:44.126 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:44.104 7ff20e0ba700 1 --2- 172.21.2.102:0/353413511 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff2080d1870 0x7ff2081b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:44.126 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:44.105 7ff20d0b8700 1 -- 172.21.2.102:0/353413511 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1079854980 0 0) 0x7ff200001460 con 0x7ff2080cff70 2021-02-10T11:26:44.126 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:44.105 7ff20d0b8700 1 -- 172.21.2.102:0/353413511 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff2080d1870 msgr2=0x7ff2081b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:26:44.127 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:44.105 7ff20d0b8700 1 --2- 172.21.2.102:0/353413511 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff2080d1870 0x7ff2081b04c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:44.127 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:44.105 7ff20d0b8700 1 -- 172.21.2.102:0/353413511 >> v1:172.21.2.113:6789/0 conn(0x7ff2081b0ae0 legacy=0x7ff208108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:44.127 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:44.105 7ff20d0b8700 1 -- 172.21.2.102:0/353413511 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2080d1c70 con 0x7ff2080cff70 2021-02-10T11:26:44.127 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:44.105 7ff20d0b8700 1 -- 172.21.2.102:0/353413511 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7ff200001980 con 0x7ff2080cff70 2021-02-10T11:26:44.127 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:44.105 7ff20d0b8700 1 -- 172.21.2.102:0/353413511 >> v1:172.21.2.102:6789/0 conn(0x7ff2080cff70 legacy=0x7ff208109130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:44.128 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:44.105 7ff20d0b8700 1 --2- 172.21.2.102:0/353413511 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff1f4003aa0 0x7ff1f4005eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:44.128 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:44.105 7ff20d0b8700 1 --2- 172.21.2.102:0/353413511 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff1f4006730 0x7ff1f4008c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:44.129 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:44.105 7ff20d0b8700 1 --2- 172.21.2.102:0/353413511 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff1f40094f0 0x7ff1f400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:44.129 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:44.105 7ff20d0b8700 1 -- 172.21.2.102:0/353413511 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1f400c2c0 con 0x7ff1f4006730 2021-02-10T11:26:44.129 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:44.105 7ff20d0b8700 1 -- 172.21.2.102:0/353413511 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1f400c4e0 con 0x7ff1f40094f0 2021-02-10T11:26:44.129 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:44.105 7ff20d0b8700 1 -- 172.21.2.102:0/353413511 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1f400c700 con 0x7ff1f4003aa0 2021-02-10T11:26:44.129 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:44.105 7ff20d8b9700 1 --2- 172.21.2.102:0/353413511 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff1f4006730 0x7ff1f4008c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:44.129 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:44.105 7ff20e8bb700 1 --2- 172.21.2.102:0/353413511 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff1f40094f0 0x7ff1f400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:44.129 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:44.105 7ff20e0ba700 1 --2- 172.21.2.102:0/353413511 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff1f4003aa0 0x7ff1f4005eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:44.129 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:44.105 7ff20d8b9700 1 -- 172.21.2.102:0/353413511 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff1f4003aa0 msgr2=0x7ff1f4005eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:26:44.129 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:44.105 7ff20d8b9700 1 --2- 172.21.2.102:0/353413511 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff1f4003aa0 0x7ff1f4005eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:44.129 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:44.105 7ff20d8b9700 1 -- 172.21.2.102:0/353413511 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff1f40094f0 msgr2=0x7ff1f400ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:26:44.130 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:44.105 7ff20d8b9700 1 --2- 172.21.2.102:0/353413511 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff1f40094f0 0x7ff1f400ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:44.130 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:44.105 7ff20d8b9700 1 -- 172.21.2.102:0/353413511 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1f4050940 con 0x7ff1f4006730 2021-02-10T11:26:44.130 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:44.105 7ff20e0ba700 1 --2- 172.21.2.102:0/353413511 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff1f4003aa0 0x7ff1f4005eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:26:44.130 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:44.105 7ff20d8b9700 1 --2- 172.21.2.102:0/353413511 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff1f4006730 0x7ff1f4008c50 secure :-1 s=READY pgs=1635 cs=0 l=1 rev1=1 rx=0x7ff204006ac0 tx=0x7ff204009af0).ready entity=mon.0 client_cookie=8e34b0ee1565e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:44.130 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:44.105 7ff20d0b8700 1 -- 172.21.2.102:0/353413511 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff204007500 con 0x7ff1f4006730 2021-02-10T11:26:44.130 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:44.105 7ff20d0b8700 1 -- 172.21.2.102:0/353413511 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2040066e0 con 0x7ff1f4006730 2021-02-10T11:26:44.130 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:44.105 7ff20d0b8700 1 -- 172.21.2.102:0/353413511 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff204007970 con 0x7ff1f4006730 2021-02-10T11:26:44.130 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:44.106 7ff21029e700 1 -- 172.21.2.102:0/353413511 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff1f4006730 msgr2=0x7ff1f4008c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:44.130 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:44.106 7ff21029e700 1 --2- 172.21.2.102:0/353413511 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff1f4006730 0x7ff1f4008c50 secure :-1 s=READY pgs=1635 cs=0 l=1 rev1=1 rx=0x7ff204006ac0 tx=0x7ff204009af0).stop 2021-02-10T11:26:44.130 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:44.106 7ff20e8bb700 1 -- 172.21.2.102:0/353413511 reap_dead start 2021-02-10T11:26:44.130 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:44.106 7ff21029e700 1 -- 172.21.2.102:0/353413511 shutdown_connections 2021-02-10T11:26:44.130 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:44.106 7ff21029e700 1 -- 172.21.2.102:0/353413511 >> 172.21.2.102:0/353413511 conn(0x7ff2080ce420 msgr2=0x7ff2080d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:26:44.131 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:44.106 7ff21029e700 1 -- 172.21.2.102:0/353413511 shutdown_connections 2021-02-10T11:26:44.131 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:44.106 7ff21029e700 1 -- 172.21.2.102:0/353413511 wait complete. 2021-02-10T11:26:44.131 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:44.121 7ff21029e700 1 Processor -- start 2021-02-10T11:26:44.131 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:44.121 7ff21029e700 1 -- start start 2021-02-10T11:26:44.131 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:44.121 7ff21029e700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff208085bc0 0x7ff2080863e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:44.131 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:44.121 7ff21029e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff2080954c0 0x7ff208096300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:44.131 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:44.121 7ff21029e700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff2080958c0 0x7ff208096920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:44.131 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:44.121 7ff21029e700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff208095d80 con 0x7ff2080954c0 2021-02-10T11:26:44.131 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:44.121 7ff21029e700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2080bfa60 con 0x7ff2080958c0 2021-02-10T11:26:44.131 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:44.121 7ff21029e700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff208085fc0 con 0x7ff208085bc0 2021-02-10T11:26:44.131 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:44.122 7ff20d8b9700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff2080954c0 0x7ff208096300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:44.131 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:44.122 7ff20d8b9700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff2080954c0 0x7ff208096300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:58898/0 (socket says 172.21.2.102:58898) 2021-02-10T11:26:44.131 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:44.122 7ff20d8b9700 1 -- 172.21.2.102:0/577154293 learned_addr learned my addr 172.21.2.102:0/577154293 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:26:44.132 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:44.122 7ff20d8b9700 1 -- 172.21.2.102:0/577154293 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff208085bc0 msgr2=0x7ff2080863e0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:26:44.132 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:44.122 7ff20d8b9700 1 --2- 172.21.2.102:0/577154293 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff208085bc0 0x7ff2080863e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:44.132 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:44.122 7ff20d8b9700 1 -- 172.21.2.102:0/577154293 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff2080958c0 msgr2=0x7ff208096920 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:26:44.132 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:44.122 7ff20d8b9700 1 --2- 172.21.2.102:0/577154293 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff2080958c0 0x7ff208096920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:44.132 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:44.122 7ff20d8b9700 1 -- 172.21.2.102:0/577154293 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff208086c10 con 0x7ff2080954c0 2021-02-10T11:26:44.132 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:44.122 7ff20d8b9700 1 --2- 172.21.2.102:0/577154293 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff2080954c0 0x7ff208096300 secure :-1 s=READY pgs=1636 cs=0 l=1 rev1=1 rx=0x7ff204007f00 tx=0x7ff20400c920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:44.132 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:44.122 7ff1feffd700 1 -- 172.21.2.102:0/577154293 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff204006280 con 0x7ff2080954c0 2021-02-10T11:26:44.132 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:44.122 7ff1feffd700 1 -- 172.21.2.102:0/577154293 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff204006500 con 0x7ff2080954c0 2021-02-10T11:26:44.132 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:44.122 7ff1feffd700 1 -- 172.21.2.102:0/577154293 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff204006bb0 con 0x7ff2080954c0 2021-02-10T11:26:44.132 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:44.122 7ff21029e700 1 -- 172.21.2.102:0/577154293 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff208086c10 con 0x7ff2080954c0 2021-02-10T11:26:44.132 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:44.122 7ff21029e700 1 -- 172.21.2.102:0/577154293 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff208097230 con 0x7ff2080954c0 2021-02-10T11:26:44.132 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:44.123 7ff1feffd700 1 -- 172.21.2.102:0/577154293 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ff204007230 con 0x7ff2080954c0 2021-02-10T11:26:44.132 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:44.123 7ff1feffd700 1 --2- 172.21.2.102:0/577154293 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff1f00186c0 0x7ff1f001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:44.133 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:44.123 7ff1feffd700 1 -- 172.21.2.102:0/577154293 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 4753+0+0 (secure 0 0 0) 0x7ff204032310 con 0x7ff2080954c0 2021-02-10T11:26:44.133 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:44.123 7ff20e0ba700 1 --2- 172.21.2.102:0/577154293 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff1f00186c0 0x7ff1f001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:44.133 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:44.124 7ff20e0ba700 1 --2- 172.21.2.102:0/577154293 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff1f00186c0 0x7ff1f001abc0 crc :-1 s=READY pgs=742 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:44.153 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:44.154 7ff21029e700 1 -- 172.21.2.102:0/577154293 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2080fe1c0 con 0x7ff2080954c0 2021-02-10T11:26:44.156 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:44.156 7ff1feffd700 1 -- 172.21.2.102:0/577154293 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ff204007020 con 0x7ff2080954c0 2021-02-10T11:26:44.285 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:44.285 7ff21029e700 1 -- 172.21.2.102:0/577154293 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7ff2080fe1c0 con 0x7ff2080954c0 2021-02-10T11:26:44.285 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:44.285 7ff1feffd700 1 -- 172.21.2.102:0/577154293 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v70) v1 ==== 92+0+10 (secure 0 0 0) 0x7ff204007210 con 0x7ff2080954c0 2021-02-10T11:26:44.286 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:26:44.287 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:44.288 7ff21029e700 1 -- 172.21.2.102:0/577154293 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff1f00186c0 msgr2=0x7ff1f001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:44.287 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:44.288 7ff21029e700 1 --2- 172.21.2.102:0/577154293 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff1f00186c0 0x7ff1f001abc0 crc :-1 s=READY pgs=742 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:44.287 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:44.288 7ff21029e700 1 -- 172.21.2.102:0/577154293 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff2080954c0 msgr2=0x7ff208096300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:44.287 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:44.288 7ff21029e700 1 --2- 172.21.2.102:0/577154293 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff2080954c0 0x7ff208096300 secure :-1 s=READY pgs=1636 cs=0 l=1 rev1=1 rx=0x7ff204007f00 tx=0x7ff20400c920).stop 2021-02-10T11:26:44.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:44.288 7ff21029e700 1 -- 172.21.2.102:0/577154293 shutdown_connections 2021-02-10T11:26:44.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:44.288 7ff21029e700 1 --2- 172.21.2.102:0/577154293 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff1f00186c0 0x7ff1f001abc0 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:44.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:44.288 7ff21029e700 1 --2- 172.21.2.102:0/577154293 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff2080958c0 0x7ff208096920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:44.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:44.288 7ff21029e700 1 --2- 172.21.2.102:0/577154293 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff2080954c0 0x7ff208096300 unknown :-1 s=CLOSED pgs=1636 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:44.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:44.288 7ff21029e700 1 --2- 172.21.2.102:0/577154293 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff208085bc0 0x7ff2080863e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:44.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:44.288 7ff21029e700 1 -- 172.21.2.102:0/577154293 >> 172.21.2.102:0/577154293 conn(0x7ff2080bef00 msgr2=0x7ff20809d1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:26:44.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:44.288 7ff21029e700 1 -- 172.21.2.102:0/577154293 shutdown_connections 2021-02-10T11:26:44.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:44.288 7ff21029e700 1 -- 172.21.2.102:0/577154293 wait complete. 2021-02-10T11:26:44.296 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --format=json-pretty osd lspools 2021-02-10T11:26:44.723 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:44.707 7fc2b6fc4700 1 Processor -- start 2021-02-10T11:26:44.723 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:44.707 7fc2b6fc4700 1 -- start start 2021-02-10T11:26:44.723 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:44.707 7fc2b6fc4700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fc2b00d0340 0x7fc2b00ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:44.723 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:44.707 7fc2b6fc4700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fc2b00d1c70 0x7fc2b00efc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:44.723 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:44.707 7fc2b6fc4700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2b00f0b10 con 0x7fc2b00d1870 2021-02-10T11:26:44.723 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:44.707 7fc2b6fc4700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2b00e5ed0 con 0x7fc2b00d1c70 2021-02-10T11:26:44.723 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:44.707 7fc2b6fc4700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2b01b0be0 con 0x7fc2b00d0340 2021-02-10T11:26:44.723 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:44.707 7fc2b55e1700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fc2b00d1c70 0x7fc2b00efc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:44.723 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:44.707 7fc2b55e1700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fc2b00d1c70 0x7fc2b00efc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:58908/0 (socket says 172.21.2.102:58908) 2021-02-10T11:26:44.724 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:44.707 7fc2b55e1700 1 -- 172.21.2.102:0/3927048937 learned_addr learned my addr 172.21.2.102:0/3927048937 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:26:44.724 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:44.707 7fc2b55e1700 1 -- 172.21.2.102:0/3927048937 >> v2:172.21.2.113:3300/0 conn(0x7fc2b00d0340 msgr2=0x7fc2b00ef640 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:26:44.724 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:44.707 7fc2b55e1700 1 --2- 172.21.2.102:0/3927048937 >> v2:172.21.2.113:3300/0 conn(0x7fc2b00d0340 0x7fc2b00ef640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:44.724 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:44.708 7fc2b55e1700 1 -- 172.21.2.102:0/3927048937 >> v1:172.21.2.102:6789/0 conn(0x7fc2b00d1870 legacy=0x7fc2b01b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:44.724 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:44.708 7fc2b55e1700 1 -- 172.21.2.102:0/3927048937 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2b00f20a0 con 0x7fc2b00d1c70 2021-02-10T11:26:44.724 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:44.708 7fc2b55e1700 1 --2- 172.21.2.102:0/3927048937 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc2b00d1c70 0x7fc2b00efc60 secure :-1 s=READY pgs=1638 cs=0 l=1 rev1=1 rx=0x7fc2a8006b80 tx=0x7fc2a8009b90).ready entity=mon.0 client_cookie=27ea161bd2fb22b0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:44.724 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:44.708 7fc2af7fe700 1 -- 172.21.2.102:0/3927048937 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc2a80022e0 con 0x7fc2b00d1c70 2021-02-10T11:26:44.724 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:44.708 7fc2af7fe700 1 -- 172.21.2.102:0/3927048937 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2a8002560 con 0x7fc2b00d1c70 2021-02-10T11:26:44.724 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:44.708 7fc2af7fe700 1 -- 172.21.2.102:0/3927048937 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc2a8002a40 con 0x7fc2b00d1c70 2021-02-10T11:26:44.724 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:44.708 7fc2b6fc4700 1 -- 172.21.2.102:0/3927048937 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc2b00d1c70 msgr2=0x7fc2b00efc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:44.724 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:44.708 7fc2b6fc4700 1 --2- 172.21.2.102:0/3927048937 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc2b00d1c70 0x7fc2b00efc60 secure :-1 s=READY pgs=1638 cs=0 l=1 rev1=1 rx=0x7fc2a8006b80 tx=0x7fc2a8009b90).stop 2021-02-10T11:26:44.724 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:44.709 7fc2b6fc4700 1 -- 172.21.2.102:0/3927048937 shutdown_connections 2021-02-10T11:26:44.724 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:44.709 7fc2b6fc4700 1 --2- 172.21.2.102:0/3927048937 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc2b00d1c70 0x7fc2b00efc60 unknown :-1 s=CLOSED pgs=1638 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:44.725 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:44.709 7fc2b6fc4700 1 --2- 172.21.2.102:0/3927048937 >> v2:172.21.2.113:3300/0 conn(0x7fc2b00d0340 0x7fc2b00ef640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:44.725 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:44.709 7fc2b6fc4700 1 -- 172.21.2.102:0/3927048937 >> 172.21.2.102:0/3927048937 conn(0x7fc2b00ce450 msgr2=0x7fc2b00d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:26:44.725 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:44.709 7fc2b6fc4700 1 -- 172.21.2.102:0/3927048937 shutdown_connections 2021-02-10T11:26:44.725 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:44.709 7fc2b6fc4700 1 -- 172.21.2.102:0/3927048937 wait complete. 2021-02-10T11:26:44.726 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:44.723 7fc2b6fc4700 1 Processor -- start 2021-02-10T11:26:44.727 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:44.723 7fc2b6fc4700 1 -- start start 2021-02-10T11:26:44.727 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:44.724 7fc2b6fc4700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc2b018f0a0 0x7fc2b0196ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:44.727 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:44.724 7fc2b6fc4700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc2b01970d0 0x7fc2b01974d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:44.727 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:44.724 7fc2b6fc4700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc2b0094650 0x7fc2b0094a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:44.727 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:44.724 7fc2b6fc4700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2b0095bd0 con 0x7fc2b018f0a0 2021-02-10T11:26:44.727 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:44.724 7fc2b6fc4700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2b00ad7c0 con 0x7fc2b01970d0 2021-02-10T11:26:44.727 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:44.724 7fc2b6fc4700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2b018ec70 con 0x7fc2b0094650 2021-02-10T11:26:44.727 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:44.724 7fc2b4de0700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc2b018f0a0 0x7fc2b0196ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:44.727 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:44.724 7fc2b4de0700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc2b018f0a0 0x7fc2b0196ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:58912/0 (socket says 172.21.2.102:58912) 2021-02-10T11:26:44.727 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:44.724 7fc2b4de0700 1 -- 172.21.2.102:0/2781262881 learned_addr learned my addr 172.21.2.102:0/2781262881 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:26:44.727 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:44.724 7fc2b55e1700 1 --2- 172.21.2.102:0/2781262881 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc2b0094650 0x7fc2b0094a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:44.727 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:44.724 7fc2affff700 1 --2- 172.21.2.102:0/2781262881 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc2b01970d0 0x7fc2b01974d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:44.728 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:44.725 7fc2b4de0700 1 -- 172.21.2.102:0/2781262881 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc2b0094650 msgr2=0x7fc2b0094a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:44.728 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:44.725 7fc2b4de0700 1 --2- 172.21.2.102:0/2781262881 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc2b0094650 0x7fc2b0094a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:44.728 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:44.725 7fc2b4de0700 1 -- 172.21.2.102:0/2781262881 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc2b01970d0 msgr2=0x7fc2b01974d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:44.728 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:44.725 7fc2b4de0700 1 --2- 172.21.2.102:0/2781262881 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc2b01970d0 0x7fc2b01974d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:44.728 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:44.725 7fc2b4de0700 1 -- 172.21.2.102:0/2781262881 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2b0095070 con 0x7fc2b018f0a0 2021-02-10T11:26:44.728 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:44.725 7fc2b4de0700 1 --2- 172.21.2.102:0/2781262881 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc2b018f0a0 0x7fc2b0196ab0 secure :-1 s=READY pgs=1639 cs=0 l=1 rev1=1 rx=0x7fc298006ed0 tx=0x7fc29800b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:44.728 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:44.725 7fc2ad7fa700 1 -- 172.21.2.102:0/2781262881 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc298006240 con 0x7fc2b018f0a0 2021-02-10T11:26:44.728 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:44.725 7fc2ad7fa700 1 -- 172.21.2.102:0/2781262881 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2980054b0 con 0x7fc2b018f0a0 2021-02-10T11:26:44.728 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:44.725 7fc2ad7fa700 1 -- 172.21.2.102:0/2781262881 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc298005990 con 0x7fc2b018f0a0 2021-02-10T11:26:44.728 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:44.725 7fc2b6fc4700 1 -- 172.21.2.102:0/2781262881 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2b0095070 con 0x7fc2b018f0a0 2021-02-10T11:26:44.728 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:44.725 7fc2b6fc4700 1 -- 172.21.2.102:0/2781262881 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2b0095600 con 0x7fc2b018f0a0 2021-02-10T11:26:44.728 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:44.726 7fc2ad7fa700 1 -- 172.21.2.102:0/2781262881 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fc298006030 con 0x7fc2b018f0a0 2021-02-10T11:26:44.728 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:44.726 7fc2ad7fa700 1 --2- 172.21.2.102:0/2781262881 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc2940186c0 0x7fc29401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:44.728 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:44.726 7fc2ad7fa700 1 -- 172.21.2.102:0/2781262881 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 4753+0+0 (secure 0 0 0) 0x7fc298032280 con 0x7fc2b018f0a0 2021-02-10T11:26:44.729 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:44.726 7fc2affff700 1 --2- 172.21.2.102:0/2781262881 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc2940186c0 0x7fc29401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:44.729 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:44.726 7fc2affff700 1 --2- 172.21.2.102:0/2781262881 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc2940186c0 0x7fc29401abc0 crc :-1 s=READY pgs=743 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:44.757 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:44.757 7fc2b6fc4700 1 -- 172.21.2.102:0/2781262881 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2b00ad790 con 0x7fc2b018f0a0 2021-02-10T11:26:44.760 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:44.760 7fc2ad7fa700 1 -- 172.21.2.102:0/2781262881 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fc298030de0 con 0x7fc2b018f0a0 2021-02-10T11:26:44.888 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:44.889 7fc2b6fc4700 1 -- 172.21.2.102:0/2781262881 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fc2b00ad790 con 0x7fc2b018f0a0 2021-02-10T11:26:44.889 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:44.889 7fc2ad7fa700 1 -- 172.21.2.102:0/2781262881 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v70) v1 ==== 84+0+64 (secure 0 0 0) 0x7fc298006030 con 0x7fc2b018f0a0 2021-02-10T11:26:44.889 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:26:44.889 INFO:teuthology.orchestra.run.gibba002.stdout:[ 2021-02-10T11:26:44.889 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:26:44.889 INFO:teuthology.orchestra.run.gibba002.stdout: "poolnum": 1, 2021-02-10T11:26:44.889 INFO:teuthology.orchestra.run.gibba002.stdout: "poolname": "rbd" 2021-02-10T11:26:44.890 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:26:44.890 INFO:teuthology.orchestra.run.gibba002.stdout:] 2021-02-10T11:26:44.890 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:44.891 7fc2b6fc4700 1 -- 172.21.2.102:0/2781262881 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc2940186c0 msgr2=0x7fc29401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:44.891 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:44.891 7fc2b6fc4700 1 --2- 172.21.2.102:0/2781262881 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc2940186c0 0x7fc29401abc0 crc :-1 s=READY pgs=743 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:44.891 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:44.891 7fc2b6fc4700 1 -- 172.21.2.102:0/2781262881 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc2b018f0a0 msgr2=0x7fc2b0196ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:44.891 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:44.891 7fc2b6fc4700 1 --2- 172.21.2.102:0/2781262881 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc2b018f0a0 0x7fc2b0196ab0 secure :-1 s=READY pgs=1639 cs=0 l=1 rev1=1 rx=0x7fc298006ed0 tx=0x7fc29800b960).stop 2021-02-10T11:26:44.891 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:44.891 7fc2b6fc4700 1 -- 172.21.2.102:0/2781262881 shutdown_connections 2021-02-10T11:26:44.891 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:44.891 7fc2b6fc4700 1 --2- 172.21.2.102:0/2781262881 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc2b0094650 0x7fc2b0094a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:44.891 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:44.891 7fc2b6fc4700 1 --2- 172.21.2.102:0/2781262881 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc2940186c0 0x7fc29401abc0 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:44.892 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:44.891 7fc2b6fc4700 1 --2- 172.21.2.102:0/2781262881 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc2b01970d0 0x7fc2b01974d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:44.892 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:44.891 7fc2b6fc4700 1 --2- 172.21.2.102:0/2781262881 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc2b018f0a0 0x7fc2b0196ab0 unknown :-1 s=CLOSED pgs=1639 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:44.892 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:44.891 7fc2b6fc4700 1 -- 172.21.2.102:0/2781262881 >> 172.21.2.102:0/2781262881 conn(0x7fc2b01a38c0 msgr2=0x7fc2b01a2f00 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:26:44.892 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:44.891 7fc2b6fc4700 1 -- 172.21.2.102:0/2781262881 shutdown_connections 2021-02-10T11:26:44.892 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:44.891 7fc2b6fc4700 1 -- 172.21.2.102:0/2781262881 wait complete. 2021-02-10T11:26:44.899 INFO:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2021-02-10T11:26:44.899 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool create cephfs_metadata 8 2021-02-10T11:26:45.331 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:45.315 7ff194409700 1 Processor -- start 2021-02-10T11:26:45.331 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:45.315 7ff194409700 1 -- start start 2021-02-10T11:26:45.331 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:45.315 7ff194409700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7ff18c0d1c70 0x7ff18c0cef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:45.332 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:45.316 7ff194409700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff18c0d0170 con 0x7ff18c0d1870 2021-02-10T11:26:45.332 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:45.316 7ff194409700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff18c0fe1c0 con 0x7ff18c0cf580 2021-02-10T11:26:45.332 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:45.316 7ff194409700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff18c1b0cb0 con 0x7ff18c0d1c70 2021-02-10T11:26:45.332 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:45.316 7ff192225700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7ff18c0d1870 0x7ff18c1b04c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:51142/0 (socket says 172.21.2.102:51142) 2021-02-10T11:26:45.332 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:45.316 7ff192225700 1 -- 172.21.2.102:0/2907258707 learned_addr learned my addr 172.21.2.102:0/2907258707 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:26:45.332 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:45.316 7ff191a24700 1 --2- 172.21.2.102:0/2907258707 >> v2:172.21.2.113:3300/0 conn(0x7ff18c0d1c70 0x7ff18c0cef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:45.332 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:45.316 7ff191223700 1 -- 172.21.2.102:0/2907258707 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3409277748 0 0) 0x7ff1880010b0 con 0x7ff18c0d1870 2021-02-10T11:26:45.332 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:45.316 7ff191a24700 1 -- 172.21.2.102:0/2907258707 >> v1:172.21.2.113:6789/0 conn(0x7ff18c0cf580 legacy=0x7ff18c0cf980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:45.332 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:45.316 7ff191a24700 1 -- 172.21.2.102:0/2907258707 >> v1:172.21.2.102:6789/0 conn(0x7ff18c0d1870 legacy=0x7ff18c1b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:45.332 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:45.316 7ff191a24700 1 -- 172.21.2.102:0/2907258707 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff18c109a60 con 0x7ff18c0d1c70 2021-02-10T11:26:45.332 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:45.317 7ff191a24700 1 --2- 172.21.2.102:0/2907258707 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff18c0d1c70 0x7ff18c0cef60 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 rx=0x7ff180006650 tx=0x7ff180009a80).ready entity=mon.1 client_cookie=e4d7faf32bdca01d server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:45.333 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:45.317 7ff191223700 1 -- 172.21.2.102:0/2907258707 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff1800009b0 con 0x7ff18c0d1c70 2021-02-10T11:26:45.333 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:45.317 7ff191223700 1 -- 172.21.2.102:0/2907258707 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff18000c0c0 con 0x7ff18c0d1c70 2021-02-10T11:26:45.333 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:45.317 7ff191223700 1 -- 172.21.2.102:0/2907258707 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff1800009b0 con 0x7ff18c0d1c70 2021-02-10T11:26:45.333 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:45.317 7ff194409700 1 -- 172.21.2.102:0/2907258707 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff18c0d1c70 msgr2=0x7ff18c0cef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:45.333 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:45.317 7ff194409700 1 --2- 172.21.2.102:0/2907258707 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff18c0d1c70 0x7ff18c0cef60 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 rx=0x7ff180006650 tx=0x7ff180009a80).stop 2021-02-10T11:26:45.333 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:45.317 7ff194409700 1 -- 172.21.2.102:0/2907258707 shutdown_connections 2021-02-10T11:26:45.333 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:45.317 7ff194409700 1 --2- 172.21.2.102:0/2907258707 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff18c0d1c70 0x7ff18c0cef60 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:45.333 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:45.317 7ff194409700 1 -- 172.21.2.102:0/2907258707 >> 172.21.2.102:0/2907258707 conn(0x7ff18c0ce420 msgr2=0x7ff18c0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:26:45.333 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:45.317 7ff194409700 1 -- 172.21.2.102:0/2907258707 shutdown_connections 2021-02-10T11:26:45.333 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:45.318 7ff194409700 1 -- 172.21.2.102:0/2907258707 wait complete. 2021-02-10T11:26:45.335 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:45.332 7ff194409700 1 Processor -- start 2021-02-10T11:26:45.335 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:45.332 7ff194409700 1 -- start start 2021-02-10T11:26:45.335 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:45.332 7ff194409700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff18c0c3820 0x7ff18c084cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:45.335 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:45.332 7ff194409700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff18c0a0290 0x7ff18c0852d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:45.335 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:45.332 7ff194409700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff18c086270 0x7ff18c086670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:45.335 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:45.332 7ff194409700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff18c0c0700 con 0x7ff18c086270 2021-02-10T11:26:45.335 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:45.332 7ff194409700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff18c0858f0 con 0x7ff18c0c3820 2021-02-10T11:26:45.335 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:45.332 7ff194409700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff18c085b10 con 0x7ff18c0a0290 2021-02-10T11:26:45.335 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:45.333 7ff192a26700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff18c086270 0x7ff18c086670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:45.335 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:45.333 7ff192a26700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff18c086270 0x7ff18c086670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:58928/0 (socket says 172.21.2.102:58928) 2021-02-10T11:26:45.336 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:45.333 7ff192a26700 1 -- 172.21.2.102:0/951749173 learned_addr learned my addr 172.21.2.102:0/951749173 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:26:45.336 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:45.333 7ff192a26700 1 -- 172.21.2.102:0/951749173 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff18c0a0290 msgr2=0x7ff18c0852d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:45.336 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:45.333 7ff192a26700 1 --2- 172.21.2.102:0/951749173 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff18c0a0290 0x7ff18c0852d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:45.336 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:45.333 7ff192a26700 1 -- 172.21.2.102:0/951749173 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff18c0c3820 msgr2=0x7ff18c084cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:45.336 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:45.333 7ff192a26700 1 --2- 172.21.2.102:0/951749173 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff18c0c3820 0x7ff18c084cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:45.336 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:45.333 7ff192a26700 1 -- 172.21.2.102:0/951749173 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff18c086c90 con 0x7ff18c086270 2021-02-10T11:26:45.336 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:45.333 7ff192a26700 1 --2- 172.21.2.102:0/951749173 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff18c086270 0x7ff18c086670 secure :-1 s=READY pgs=1641 cs=0 l=1 rev1=1 rx=0x7ff184009940 tx=0x7ff18400bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:45.336 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:45.333 7ff17effd700 1 -- 172.21.2.102:0/951749173 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff184000ce0 con 0x7ff18c086270 2021-02-10T11:26:45.336 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:45.333 7ff17effd700 1 -- 172.21.2.102:0/951749173 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff18400c2f0 con 0x7ff18c086270 2021-02-10T11:26:45.336 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:45.333 7ff194409700 1 -- 172.21.2.102:0/951749173 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff18c086c90 con 0x7ff18c086270 2021-02-10T11:26:45.336 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:45.334 7ff17effd700 1 -- 172.21.2.102:0/951749173 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff18400c750 con 0x7ff18c086270 2021-02-10T11:26:45.336 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:45.334 7ff194409700 1 -- 172.21.2.102:0/951749173 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff18c086c90 con 0x7ff18c086270 2021-02-10T11:26:45.336 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:45.334 7ff17effd700 1 -- 172.21.2.102:0/951749173 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ff184001220 con 0x7ff18c086270 2021-02-10T11:26:45.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:45.334 7ff17effd700 1 --2- 172.21.2.102:0/951749173 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff1740186c0 0x7ff17401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:45.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:45.334 7ff17effd700 1 -- 172.21.2.102:0/951749173 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 4753+0+0 (secure 0 0 0) 0x7ff1840324f0 con 0x7ff18c086270 2021-02-10T11:26:45.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:45.334 7ff192225700 1 --2- 172.21.2.102:0/951749173 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff1740186c0 0x7ff17401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:45.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:45.335 7ff192225700 1 --2- 172.21.2.102:0/951749173 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff1740186c0 0x7ff17401abc0 crc :-1 s=READY pgs=744 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:45.365 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:45.365 7ff194409700 1 -- 172.21.2.102:0/951749173 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff18c082a70 con 0x7ff18c086270 2021-02-10T11:26:45.367 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:45.368 7ff17effd700 1 -- 172.21.2.102:0/951749173 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ff184000f90 con 0x7ff18c086270 2021-02-10T11:26:45.496 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:45.497 7ff194409700 1 -- 172.21.2.102:0/951749173 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"} v 0) v1 -- 0x7ff18c082a70 con 0x7ff18c086270 2021-02-10T11:26:45.914 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:45.914 7ff17effd700 1 -- 172.21.2.102:0/951749173 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"}]=0 pool 'cephfs_metadata' created v71) v1 ==== 133+0+0 (secure 0 0 0) 0x7ff184001220 con 0x7ff18c086270 2021-02-10T11:26:45.962 INFO:teuthology.orchestra.run.gibba002.stderr:pool 'cephfs_metadata' created 2021-02-10T11:26:45.964 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:45.964 7ff194409700 1 -- 172.21.2.102:0/951749173 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff1740186c0 msgr2=0x7ff17401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:45.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:45.964 7ff194409700 1 --2- 172.21.2.102:0/951749173 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff1740186c0 0x7ff17401abc0 crc :-1 s=READY pgs=744 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:45.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:45.964 7ff194409700 1 -- 172.21.2.102:0/951749173 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff18c086270 msgr2=0x7ff18c086670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:45.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:45.964 7ff194409700 1 --2- 172.21.2.102:0/951749173 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff18c086270 0x7ff18c086670 secure :-1 s=READY pgs=1641 cs=0 l=1 rev1=1 rx=0x7ff184009940 tx=0x7ff18400bcf0).stop 2021-02-10T11:26:45.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:45.965 7ff194409700 1 -- 172.21.2.102:0/951749173 shutdown_connections 2021-02-10T11:26:45.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:45.965 7ff194409700 1 --2- 172.21.2.102:0/951749173 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff18c086270 0x7ff18c086670 unknown :-1 s=CLOSED pgs=1641 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:45.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:45.965 7ff194409700 1 --2- 172.21.2.102:0/951749173 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff18c0a0290 0x7ff18c0852d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:45.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:45.965 7ff194409700 1 --2- 172.21.2.102:0/951749173 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff1740186c0 0x7ff17401abc0 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:45.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:45.965 7ff194409700 1 --2- 172.21.2.102:0/951749173 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff18c0c3820 0x7ff18c084cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:45.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:45.965 7ff194409700 1 -- 172.21.2.102:0/951749173 >> 172.21.2.102:0/951749173 conn(0x7ff18c0872c0 msgr2=0x7ff18c095700 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:26:45.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:45.965 7ff194409700 1 -- 172.21.2.102:0/951749173 shutdown_connections 2021-02-10T11:26:45.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:45.965 7ff194409700 1 -- 172.21.2.102:0/951749173 wait complete. 2021-02-10T11:26:45.973 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool create cephfs_data 8 2021-02-10T11:26:46.405 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:46.388 7fad5d449700 1 Processor -- start 2021-02-10T11:26:46.405 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:46.388 7fad5d449700 1 -- start start 2021-02-10T11:26:46.405 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:46.389 7fad5d449700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fad5810aff0 0x7fad580cf750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:46.406 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:46.389 7fad5d449700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad580fe1c0 con 0x7fad580d1870 2021-02-10T11:26:46.406 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:46.389 7fad5d449700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad581b0cb0 con 0x7fad580d1c70 2021-02-10T11:26:46.406 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:46.389 7fad5d449700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad58109a60 con 0x7fad5810aff0 2021-02-10T11:26:46.406 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:46.389 7fad5707d700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fad580d1870 0x7fad581b04c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:51156/0 (socket says 172.21.2.102:51156) 2021-02-10T11:26:46.406 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:46.389 7fad5707d700 1 -- 172.21.2.102:0/3822534042 learned_addr learned my addr 172.21.2.102:0/3822534042 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:26:46.406 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:46.389 7fad5607b700 1 -- 172.21.2.102:0/3822534042 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 169170597 0 0) 0x7fad480010b0 con 0x7fad580d1870 2021-02-10T11:26:46.406 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:46.389 7fad5607b700 1 -- 172.21.2.102:0/3822534042 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad44002fe0 con 0x7fad580d1870 2021-02-10T11:26:46.406 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:46.389 7fad5787e700 1 --2- 172.21.2.102:0/3822534042 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fad5810aff0 0x7fad580cf750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:46.406 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:46.389 7fad5607b700 1 -- 172.21.2.102:0/3822534042 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 909506924 0 0) 0x7fad480010b0 con 0x7fad580d1870 2021-02-10T11:26:46.406 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:46.390 7fad5607b700 1 -- 172.21.2.102:0/3822534042 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fad5810aff0 msgr2=0x7fad580cf750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:26:46.406 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:46.390 7fad5607b700 1 --2- 172.21.2.102:0/3822534042 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fad5810aff0 0x7fad580cf750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:46.407 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:46.390 7fad5607b700 1 -- 172.21.2.102:0/3822534042 >> v1:172.21.2.113:6789/0 conn(0x7fad580d1c70 legacy=0x7fad580cef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:46.407 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:46.390 7fad5607b700 1 -- 172.21.2.102:0/3822534042 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad5810b3f0 con 0x7fad580d1870 2021-02-10T11:26:46.407 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:46.390 7fad5607b700 1 -- 172.21.2.102:0/3822534042 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7fad480015b0 con 0x7fad580d1870 2021-02-10T11:26:46.407 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:46.390 7fad5607b700 1 -- 172.21.2.102:0/3822534042 >> v1:172.21.2.102:6789/0 conn(0x7fad580d1870 legacy=0x7fad581b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:46.407 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:46.390 7fad5607b700 1 --2- 172.21.2.102:0/3822534042 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fad44003a60 0x7fad44005f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:46.407 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:46.390 7fad5607b700 1 --2- 172.21.2.102:0/3822534042 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fad440067d0 0x7fad44008cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:46.407 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:46.390 7fad5607b700 1 --2- 172.21.2.102:0/3822534042 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fad44009590 0x7fad4400bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:46.407 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:46.390 7fad5607b700 1 -- 172.21.2.102:0/3822534042 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad4400c2e0 con 0x7fad44009590 2021-02-10T11:26:46.407 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:46.390 7fad5607b700 1 -- 172.21.2.102:0/3822534042 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad4400c500 con 0x7fad440067d0 2021-02-10T11:26:46.407 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:46.390 7fad5607b700 1 -- 172.21.2.102:0/3822534042 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad4400c720 con 0x7fad44003a60 2021-02-10T11:26:46.407 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:46.390 7fad5787e700 1 --2- 172.21.2.102:0/3822534042 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fad44009590 0x7fad4400bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:46.407 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:46.390 7fad5687c700 1 --2- 172.21.2.102:0/3822534042 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fad440067d0 0x7fad44008cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:46.407 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:46.390 7fad5707d700 1 --2- 172.21.2.102:0/3822534042 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fad44003a60 0x7fad44005f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:46.407 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:46.390 7fad5787e700 1 -- 172.21.2.102:0/3822534042 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fad44003a60 msgr2=0x7fad44005f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:26:46.408 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:46.390 7fad5787e700 1 --2- 172.21.2.102:0/3822534042 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fad44003a60 0x7fad44005f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:46.408 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:46.390 7fad5787e700 1 -- 172.21.2.102:0/3822534042 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fad440067d0 msgr2=0x7fad44008cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:26:46.408 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:46.390 7fad5787e700 1 --2- 172.21.2.102:0/3822534042 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fad440067d0 0x7fad44008cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:46.408 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:46.390 7fad5787e700 1 -- 172.21.2.102:0/3822534042 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad44050960 con 0x7fad44009590 2021-02-10T11:26:46.408 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:46.390 7fad5707d700 1 --2- 172.21.2.102:0/3822534042 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fad44003a60 0x7fad44005f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:26:46.408 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:46.390 7fad5787e700 1 --2- 172.21.2.102:0/3822534042 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fad44009590 0x7fad4400bab0 secure :-1 s=READY pgs=1643 cs=0 l=1 rev1=1 rx=0x7fad50009b10 tx=0x7fad50009eb0).ready entity=mon.0 client_cookie=72474b0083fead1c server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:46.408 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:46.390 7fad5607b700 1 -- 172.21.2.102:0/3822534042 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fad500066f0 con 0x7fad44009590 2021-02-10T11:26:46.408 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:46.390 7fad5607b700 1 -- 172.21.2.102:0/3822534042 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad50003030 con 0x7fad44009590 2021-02-10T11:26:46.408 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:46.390 7fad5607b700 1 -- 172.21.2.102:0/3822534042 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fad5000b700 con 0x7fad44009590 2021-02-10T11:26:46.408 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:46.390 7fad5d449700 1 -- 172.21.2.102:0/3822534042 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fad44009590 msgr2=0x7fad4400bab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:46.408 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:46.390 7fad5d449700 1 --2- 172.21.2.102:0/3822534042 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fad44009590 0x7fad4400bab0 secure :-1 s=READY pgs=1643 cs=0 l=1 rev1=1 rx=0x7fad50009b10 tx=0x7fad50009eb0).stop 2021-02-10T11:26:46.408 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:46.390 7fad5787e700 1 -- 172.21.2.102:0/3822534042 reap_dead start 2021-02-10T11:26:46.408 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:46.391 7fad5d449700 1 -- 172.21.2.102:0/3822534042 shutdown_connections 2021-02-10T11:26:46.409 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:46.391 7fad5d449700 1 -- 172.21.2.102:0/3822534042 >> 172.21.2.102:0/3822534042 conn(0x7fad580ce420 msgr2=0x7fad580d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:26:46.409 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:46.391 7fad5d449700 1 -- 172.21.2.102:0/3822534042 shutdown_connections 2021-02-10T11:26:46.409 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:46.391 7fad5d449700 1 -- 172.21.2.102:0/3822534042 wait complete. 2021-02-10T11:26:46.410 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:46.406 7fad5d449700 1 Processor -- start 2021-02-10T11:26:46.410 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:46.406 7fad5d449700 1 -- start start 2021-02-10T11:26:46.410 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:46.406 7fad5d449700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fad580926d0 0x7fad580b6820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:46.410 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:46.406 7fad5d449700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fad580c3840 0x7fad58093270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:46.410 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:46.406 7fad5d449700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fad580c3c40 0x7fad58093890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:46.410 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:46.406 7fad5d449700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad58092ad0 con 0x7fad580c3c40 2021-02-10T11:26:46.411 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:46.406 7fad5d449700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad58092cf0 con 0x7fad580926d0 2021-02-10T11:26:46.411 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:46.406 7fad5d449700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad580b6f80 con 0x7fad580c3840 2021-02-10T11:26:46.411 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:46.407 7fad5787e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fad580c3c40 0x7fad58093890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:46.411 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:46.407 7fad5787e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fad580c3c40 0x7fad58093890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:58946/0 (socket says 172.21.2.102:58946) 2021-02-10T11:26:46.411 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:46.407 7fad5787e700 1 -- 172.21.2.102:0/2828681614 learned_addr learned my addr 172.21.2.102:0/2828681614 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:26:46.411 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:46.407 7fad5787e700 1 -- 172.21.2.102:0/2828681614 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fad580c3840 msgr2=0x7fad58093270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:46.411 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:46.407 7fad5787e700 1 --2- 172.21.2.102:0/2828681614 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fad580c3840 0x7fad58093270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:46.411 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:46.407 7fad5787e700 1 -- 172.21.2.102:0/2828681614 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fad580926d0 msgr2=0x7fad580b6820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:46.411 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:46.407 7fad5787e700 1 --2- 172.21.2.102:0/2828681614 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fad580926d0 0x7fad580b6820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:46.411 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:46.407 7fad5787e700 1 -- 172.21.2.102:0/2828681614 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad58093eb0 con 0x7fad580c3c40 2021-02-10T11:26:46.411 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:46.407 7fad5787e700 1 --2- 172.21.2.102:0/2828681614 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fad580c3c40 0x7fad58093890 secure :-1 s=READY pgs=1644 cs=0 l=1 rev1=1 rx=0x7fad50009ac0 tx=0x7fad5000bd20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:46.411 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:46.407 7fad4ffff700 1 -- 172.21.2.102:0/2828681614 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fad500093d0 con 0x7fad580c3c40 2021-02-10T11:26:46.411 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:46.407 7fad4ffff700 1 -- 172.21.2.102:0/2828681614 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad5000be60 con 0x7fad580c3c40 2021-02-10T11:26:46.411 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:46.407 7fad5d449700 1 -- 172.21.2.102:0/2828681614 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad58093eb0 con 0x7fad580c3c40 2021-02-10T11:26:46.411 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:46.407 7fad4ffff700 1 -- 172.21.2.102:0/2828681614 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fad5000c4e0 con 0x7fad580c3c40 2021-02-10T11:26:46.412 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:46.407 7fad5d449700 1 -- 172.21.2.102:0/2828681614 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad58093eb0 con 0x7fad580c3c40 2021-02-10T11:26:46.412 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:46.408 7fad4ffff700 1 -- 172.21.2.102:0/2828681614 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fad500056b0 con 0x7fad580c3c40 2021-02-10T11:26:46.412 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:46.408 7fad4ffff700 1 --2- 172.21.2.102:0/2828681614 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fad3c0186c0 0x7fad3c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:46.412 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:46.408 7fad4ffff700 1 -- 172.21.2.102:0/2828681614 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5107+0+0 (secure 0 0 0) 0x7fad500325a0 con 0x7fad580c3c40 2021-02-10T11:26:46.412 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:46.408 7fad5707d700 1 --2- 172.21.2.102:0/2828681614 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fad3c0186c0 0x7fad3c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:46.412 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:46.408 7fad5707d700 1 --2- 172.21.2.102:0/2828681614 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fad3c0186c0 0x7fad3c01abc0 crc :-1 s=READY pgs=745 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:46.438 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:46.438 7fad5d449700 1 -- 172.21.2.102:0/2828681614 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad58082a70 con 0x7fad580c3c40 2021-02-10T11:26:46.441 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:46.441 7fad4ffff700 1 -- 172.21.2.102:0/2828681614 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fad5002b020 con 0x7fad580c3c40 2021-02-10T11:26:46.569 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:46.570 7fad5d449700 1 -- 172.21.2.102:0/2828681614 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_data"} v 0) v1 -- 0x7fad58082a70 con 0x7fad580c3c40 2021-02-10T11:26:46.920 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:46.920 7fad4ffff700 1 -- 172.21.2.102:0/2828681614 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_data"}]=0 pool 'cephfs_data' created v73) v1 ==== 125+0+0 (secure 0 0 0) 0x7fad5002b020 con 0x7fad580c3c40 2021-02-10T11:26:46.934 INFO:teuthology.orchestra.run.gibba002.stderr:pool 'cephfs_data' created 2021-02-10T11:26:46.936 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:46.936 7fad5d449700 1 -- 172.21.2.102:0/2828681614 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fad3c0186c0 msgr2=0x7fad3c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:46.937 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:46.936 7fad5d449700 1 --2- 172.21.2.102:0/2828681614 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fad3c0186c0 0x7fad3c01abc0 crc :-1 s=READY pgs=745 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:46.937 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:46.936 7fad5d449700 1 -- 172.21.2.102:0/2828681614 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fad580c3c40 msgr2=0x7fad58093890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:46.937 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:46.936 7fad5d449700 1 --2- 172.21.2.102:0/2828681614 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fad580c3c40 0x7fad58093890 secure :-1 s=READY pgs=1644 cs=0 l=1 rev1=1 rx=0x7fad50009ac0 tx=0x7fad5000bd20).stop 2021-02-10T11:26:46.937 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:46.936 7fad5d449700 1 -- 172.21.2.102:0/2828681614 shutdown_connections 2021-02-10T11:26:46.937 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:46.936 7fad5d449700 1 --2- 172.21.2.102:0/2828681614 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fad580c3c40 0x7fad58093890 unknown :-1 s=CLOSED pgs=1644 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:46.937 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:46.936 7fad5d449700 1 --2- 172.21.2.102:0/2828681614 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fad580c3840 0x7fad58093270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:46.937 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:46.936 7fad5d449700 1 --2- 172.21.2.102:0/2828681614 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fad3c0186c0 0x7fad3c01abc0 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:46.937 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:46.936 7fad5d449700 1 --2- 172.21.2.102:0/2828681614 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fad580926d0 0x7fad580b6820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:46.937 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:46.936 7fad5d449700 1 -- 172.21.2.102:0/2828681614 >> 172.21.2.102:0/2828681614 conn(0x7fad5809ffd0 msgr2=0x7fad58087030 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:26:46.937 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:46.937 7fad5d449700 1 -- 172.21.2.102:0/2828681614 shutdown_connections 2021-02-10T11:26:46.937 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:46.937 7fad5d449700 1 -- 172.21.2.102:0/2828681614 wait complete. 2021-02-10T11:26:46.945 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2021-02-10T11:26:47.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:47.359 7fbd5776d700 1 Processor -- start 2021-02-10T11:26:47.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:47.359 7fbd5776d700 1 -- start start 2021-02-10T11:26:47.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:47.360 7fbd5776d700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbd500d1880 0x7fbd501b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:47.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:47.360 7fbd5776d700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fbd501b0af0 0x7fbd50108950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:47.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:47.360 7fbd5776d700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd50109a70 con 0x7fbd500d1c80 2021-02-10T11:26:47.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:47.360 7fbd5776d700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd500e5f00 con 0x7fbd501b0af0 2021-02-10T11:26:47.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:47.360 7fbd5776d700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd50109760 con 0x7fbd500d1880 2021-02-10T11:26:47.377 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:47.360 7fbd55589700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbd500d1880 0x7fbd501b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:47.377 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:47.360 7fbd54d88700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fbd501b0af0 0x7fbd50108950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:47.377 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:47.360 7fbd55d8a700 1 --1- >> v1:172.21.2.113:6789/0 conn(0x7fbd500d1c80 0x7fbd50108f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.113:6789/0 says I am v1:172.21.2.102:58502/0 (socket says 172.21.2.102:58502) 2021-02-10T11:26:47.377 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:47.360 7fbd55d8a700 1 -- 172.21.2.102:0/2437067378 learned_addr learned my addr 172.21.2.102:0/2437067378 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:26:47.377 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:47.361 7fbd47fff700 1 -- 172.21.2.102:0/2437067378 <== mon.1 v1:172.21.2.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 409913203 0 0) 0x7fbd48001440 con 0x7fbd500d1c80 2021-02-10T11:26:47.377 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:47.361 7fbd47fff700 1 -- 172.21.2.102:0/2437067378 --> v1:172.21.2.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd3c002fe0 con 0x7fbd500d1c80 2021-02-10T11:26:47.377 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:47.361 7fbd55589700 1 -- 172.21.2.102:0/2437067378 >> v2:172.21.2.113:3300/0 conn(0x7fbd501b0af0 msgr2=0x7fbd50108950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:26:47.377 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:47.361 7fbd55589700 1 --2- 172.21.2.102:0/2437067378 >> v2:172.21.2.113:3300/0 conn(0x7fbd501b0af0 0x7fbd50108950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:47.377 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:47.361 7fbd55589700 1 -- 172.21.2.102:0/2437067378 >> v1:172.21.2.113:6789/0 conn(0x7fbd500d1c80 legacy=0x7fbd50108f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:47.377 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:47.361 7fbd55589700 1 -- 172.21.2.102:0/2437067378 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd50088070 con 0x7fbd500d1880 2021-02-10T11:26:47.377 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:47.361 7fbd55589700 1 --2- 172.21.2.102:0/2437067378 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbd500d1880 0x7fbd501b04d0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7fbd4c006650 tx=0x7fbd4c009a80).ready entity=mon.2 client_cookie=15db77271f69be58 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:47.377 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:47.361 7fbd47fff700 1 -- 172.21.2.102:0/2437067378 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbd4c00c0c0 con 0x7fbd500d1880 2021-02-10T11:26:47.377 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:47.361 7fbd47fff700 1 -- 172.21.2.102:0/2437067378 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd4c00c340 con 0x7fbd500d1880 2021-02-10T11:26:47.378 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:47.361 7fbd47fff700 1 -- 172.21.2.102:0/2437067378 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbd4c00c860 con 0x7fbd500d1880 2021-02-10T11:26:47.378 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:47.361 7fbd5776d700 1 -- 172.21.2.102:0/2437067378 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbd500d1880 msgr2=0x7fbd501b04d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:47.378 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:47.361 7fbd5776d700 1 --2- 172.21.2.102:0/2437067378 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbd500d1880 0x7fbd501b04d0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7fbd4c006650 tx=0x7fbd4c009a80).stop 2021-02-10T11:26:47.378 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:47.362 7fbd5776d700 1 -- 172.21.2.102:0/2437067378 shutdown_connections 2021-02-10T11:26:47.378 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:47.362 7fbd5776d700 1 --2- 172.21.2.102:0/2437067378 >> v2:172.21.2.113:3300/0 conn(0x7fbd501b0af0 0x7fbd50108950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:47.378 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:47.362 7fbd5776d700 1 --2- 172.21.2.102:0/2437067378 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbd500d1880 0x7fbd501b04d0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:47.378 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:47.362 7fbd5776d700 1 -- 172.21.2.102:0/2437067378 >> 172.21.2.102:0/2437067378 conn(0x7fbd500ce430 msgr2=0x7fbd500d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:26:47.378 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:47.362 7fbd5776d700 1 -- 172.21.2.102:0/2437067378 shutdown_connections 2021-02-10T11:26:47.378 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:47.362 7fbd5776d700 1 -- 172.21.2.102:0/2437067378 wait complete. 2021-02-10T11:26:47.380 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:47.377 7fbd5776d700 1 Processor -- start 2021-02-10T11:26:47.380 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:47.377 7fbd5776d700 1 -- start start 2021-02-10T11:26:47.380 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:47.377 7fbd5776d700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fbd501a0580 0x7fbd50082c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:47.380 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:47.377 7fbd5776d700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbd500832a0 0x7fbd500836a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:47.380 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:47.377 7fbd5776d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbd500b66d0 0x7fbd500b6ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:47.380 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:47.377 7fbd5776d700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd500ba540 con 0x7fbd500b66d0 2021-02-10T11:26:47.380 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:47.377 7fbd5776d700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd500baac0 con 0x7fbd501a0580 2021-02-10T11:26:47.380 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:47.377 7fbd5776d700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd500b70f0 con 0x7fbd500832a0 2021-02-10T11:26:47.380 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:47.377 7fbd55d8a700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbd500b66d0 0x7fbd500b6ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:47.380 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:47.377 7fbd55d8a700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbd500b66d0 0x7fbd500b6ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:58962/0 (socket says 172.21.2.102:58962) 2021-02-10T11:26:47.380 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:47.377 7fbd55d8a700 1 -- 172.21.2.102:0/3884028283 learned_addr learned my addr 172.21.2.102:0/3884028283 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:26:47.381 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:47.378 7fbd55d8a700 1 -- 172.21.2.102:0/3884028283 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbd500832a0 msgr2=0x7fbd500836a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:47.381 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:47.378 7fbd55d8a700 1 --2- 172.21.2.102:0/3884028283 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbd500832a0 0x7fbd500836a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:47.381 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:47.378 7fbd55d8a700 1 -- 172.21.2.102:0/3884028283 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fbd501a0580 msgr2=0x7fbd50082c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:47.381 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:47.378 7fbd55d8a700 1 --2- 172.21.2.102:0/3884028283 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fbd501a0580 0x7fbd50082c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:47.381 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:47.378 7fbd55d8a700 1 -- 172.21.2.102:0/3884028283 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd500b7310 con 0x7fbd500b66d0 2021-02-10T11:26:47.381 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:47.378 7fbd55d8a700 1 --2- 172.21.2.102:0/3884028283 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbd500b66d0 0x7fbd500b6ad0 secure :-1 s=READY pgs=1645 cs=0 l=1 rev1=1 rx=0x7fbd4800a8f0 tx=0x7fbd4800ac90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:47.381 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:47.378 7fbd45ffb700 1 -- 172.21.2.102:0/3884028283 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbd480048b0 con 0x7fbd500b66d0 2021-02-10T11:26:47.381 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:47.378 7fbd45ffb700 1 -- 172.21.2.102:0/3884028283 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd480079c0 con 0x7fbd500b66d0 2021-02-10T11:26:47.381 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:47.378 7fbd45ffb700 1 -- 172.21.2.102:0/3884028283 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbd48004d20 con 0x7fbd500b66d0 2021-02-10T11:26:47.381 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:47.378 7fbd5776d700 1 -- 172.21.2.102:0/3884028283 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd500b7310 con 0x7fbd500b66d0 2021-02-10T11:26:47.381 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:47.378 7fbd5776d700 1 -- 172.21.2.102:0/3884028283 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd500b78a0 con 0x7fbd500b66d0 2021-02-10T11:26:47.381 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:47.379 7fbd45ffb700 1 -- 172.21.2.102:0/3884028283 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fbd480046a0 con 0x7fbd500b66d0 2021-02-10T11:26:47.381 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:47.379 7fbd45ffb700 1 --2- 172.21.2.102:0/3884028283 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fbd300186c0 0x7fbd3001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:47.381 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:47.379 7fbd45ffb700 1 -- 172.21.2.102:0/3884028283 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 5457+0+0 (secure 0 0 0) 0x7fbd480326d0 con 0x7fbd500b66d0 2021-02-10T11:26:47.382 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:47.379 7fbd55589700 1 --2- 172.21.2.102:0/3884028283 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fbd300186c0 0x7fbd3001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:47.382 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:47.379 7fbd55589700 1 --2- 172.21.2.102:0/3884028283 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fbd300186c0 0x7fbd3001abc0 crc :-1 s=READY pgs=748 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:47.410 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:47.410 7fbd5776d700 1 -- 172.21.2.102:0/3884028283 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd50082a50 con 0x7fbd500b66d0 2021-02-10T11:26:47.412 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:47.413 7fbd45ffb700 1 -- 172.21.2.102:0/3884028283 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fbd48000f40 con 0x7fbd500b66d0 2021-02-10T11:26:47.541 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:47.541 7fbd5776d700 1 -- 172.21.2.102:0/3884028283 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"data": "cephfs_data", "prefix": "fs new", "force": true, "fs_name": "cephfs", "metadata": "cephfs_metadata"} v 0) v1 -- 0x7fbd50109a70 con 0x7fbd500b66d0 2021-02-10T11:26:47.545 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:47.546 7fbd45ffb700 1 -- 172.21.2.102:0/3884028283 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"data": "cephfs_data", "prefix": "fs new", "force": true, "fs_name": "cephfs", "metadata": "cephfs_metadata"}]=0 new fs with metadata pool 14 and data pool 15 v54) v1 ==== 189+0+0 (secure 0 0 0) 0x7fbd480046a0 con 0x7fbd500b66d0 2021-02-10T11:26:47.550 INFO:teuthology.orchestra.run.gibba002.stderr:new fs with metadata pool 14 and data pool 15 2021-02-10T11:26:47.551 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:47.551 7fbd5776d700 1 -- 172.21.2.102:0/3884028283 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fbd300186c0 msgr2=0x7fbd3001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:47.551 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:47.551 7fbd5776d700 1 --2- 172.21.2.102:0/3884028283 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fbd300186c0 0x7fbd3001abc0 crc :-1 s=READY pgs=748 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:47.551 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:47.551 7fbd5776d700 1 -- 172.21.2.102:0/3884028283 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbd500b66d0 msgr2=0x7fbd500b6ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:47.551 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:47.551 7fbd5776d700 1 --2- 172.21.2.102:0/3884028283 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbd500b66d0 0x7fbd500b6ad0 secure :-1 s=READY pgs=1645 cs=0 l=1 rev1=1 rx=0x7fbd4800a8f0 tx=0x7fbd4800ac90).stop 2021-02-10T11:26:47.552 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:47.551 7fbd5776d700 1 -- 172.21.2.102:0/3884028283 shutdown_connections 2021-02-10T11:26:47.552 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:47.551 7fbd5776d700 1 --2- 172.21.2.102:0/3884028283 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbd500b66d0 0x7fbd500b6ad0 unknown :-1 s=CLOSED pgs=1645 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:47.552 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:47.551 7fbd5776d700 1 --2- 172.21.2.102:0/3884028283 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbd500832a0 0x7fbd500836a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:47.552 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:47.551 7fbd5776d700 1 --2- 172.21.2.102:0/3884028283 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fbd300186c0 0x7fbd3001abc0 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:47.552 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:47.551 7fbd5776d700 1 --2- 172.21.2.102:0/3884028283 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fbd501a0580 0x7fbd50082c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:47.552 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:47.551 7fbd5776d700 1 -- 172.21.2.102:0/3884028283 >> 172.21.2.102:0/3884028283 conn(0x7fbd500c20f0 msgr2=0x7fbd5009e410 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:26:47.552 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:47.551 7fbd5776d700 1 -- 172.21.2.102:0/3884028283 shutdown_connections 2021-02-10T11:26:47.552 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:47.551 7fbd5776d700 1 -- 172.21.2.102:0/3884028283 wait complete. 2021-02-10T11:26:47.560 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:26:47.985 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:47.969 7f8b5c325700 1 Processor -- start 2021-02-10T11:26:47.985 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:47.969 7f8b5c325700 1 -- start start 2021-02-10T11:26:47.985 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:47.969 7f8b5c325700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f8b540d1870 0x7f8b541b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:47.986 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:47.969 7f8b5c325700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f8b541b0a10 0x7f8b540cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:47.986 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:47.969 7f8b5c325700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8b540d1c70 0x7f8b540cf950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:47.986 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:47.969 7f8b5c325700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b540e5ed0 con 0x7f8b540d1870 2021-02-10T11:26:47.986 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:47.969 7f8b5c325700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b540f0b10 con 0x7f8b541b0a10 2021-02-10T11:26:47.986 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:47.969 7f8b5c325700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b540cff70 con 0x7f8b540d1c70 2021-02-10T11:26:47.986 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:47.970 7f8b5a141700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f8b540d1870 0x7f8b541b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:47.986 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:47.970 7f8b5a942700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8b540d1c70 0x7f8b540cf950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:47.986 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:47.970 7f8b5a141700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f8b540d1870 0x7f8b541b03f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:58966/0 (socket says 172.21.2.102:58966) 2021-02-10T11:26:47.986 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:47.970 7f8b5a141700 1 -- 172.21.2.102:0/2581322995 learned_addr learned my addr 172.21.2.102:0/2581322995 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:26:47.986 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:47.970 7f8b59940700 1 --2- 172.21.2.102:0/2581322995 >> v2:172.21.2.113:3300/0 conn(0x7f8b541b0a10 0x7f8b540cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:47.986 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:47.970 7f8b59940700 1 -- 172.21.2.102:0/2581322995 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8b540d1c70 msgr2=0x7f8b540cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:26:47.986 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:47.970 7f8b59940700 1 --2- 172.21.2.102:0/2581322995 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8b540d1c70 0x7f8b540cf950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:47.987 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:47.970 7f8b59940700 1 -- 172.21.2.102:0/2581322995 >> v2:172.21.2.102:3300/0 conn(0x7f8b540d1870 msgr2=0x7f8b541b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:26:47.987 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:47.970 7f8b59940700 1 --2- 172.21.2.102:0/2581322995 >> v2:172.21.2.102:3300/0 conn(0x7f8b540d1870 0x7f8b541b03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:47.987 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:47.970 7f8b59940700 1 -- 172.21.2.102:0/2581322995 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b540d0150 con 0x7f8b541b0a10 2021-02-10T11:26:47.987 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:47.970 7f8b5a141700 1 --2- 172.21.2.102:0/2581322995 >> v2:172.21.2.102:3300/0 conn(0x7f8b540d1870 0x7f8b541b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:26:47.987 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:47.970 7f8b59940700 1 --2- 172.21.2.102:0/2581322995 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8b541b0a10 0x7f8b540cf330 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 rx=0x7f8b44006650 tx=0x7f8b44009a80).ready entity=mon.1 client_cookie=e39b01539c7fd83d server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:47.987 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:47.970 7f8b5913f700 1 -- 172.21.2.102:0/2581322995 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8b4400c0c0 con 0x7f8b541b0a10 2021-02-10T11:26:47.987 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:47.970 7f8b5913f700 1 -- 172.21.2.102:0/2581322995 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b4400c440 con 0x7f8b541b0a10 2021-02-10T11:26:47.987 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:47.970 7f8b5913f700 1 -- 172.21.2.102:0/2581322995 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8b4400c960 con 0x7f8b541b0a10 2021-02-10T11:26:47.987 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:47.971 7f8b5c325700 1 -- 172.21.2.102:0/2581322995 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8b541b0a10 msgr2=0x7f8b540cf330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:47.987 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:47.971 7f8b5c325700 1 --2- 172.21.2.102:0/2581322995 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8b541b0a10 0x7f8b540cf330 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 rx=0x7f8b44006650 tx=0x7f8b44009a80).stop 2021-02-10T11:26:47.987 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:47.971 7f8b5c325700 1 -- 172.21.2.102:0/2581322995 shutdown_connections 2021-02-10T11:26:47.987 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:47.971 7f8b5c325700 1 --2- 172.21.2.102:0/2581322995 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8b540d1c70 0x7f8b540cf950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:47.987 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:47.971 7f8b5c325700 1 --2- 172.21.2.102:0/2581322995 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8b541b0a10 0x7f8b540cf330 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:47.987 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:47.971 7f8b5c325700 1 --2- 172.21.2.102:0/2581322995 >> v2:172.21.2.102:3300/0 conn(0x7f8b540d1870 0x7f8b541b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:47.988 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:47.971 7f8b5c325700 1 -- 172.21.2.102:0/2581322995 >> 172.21.2.102:0/2581322995 conn(0x7f8b540ce450 msgr2=0x7f8b540d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:26:47.988 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:47.971 7f8b5c325700 1 -- 172.21.2.102:0/2581322995 shutdown_connections 2021-02-10T11:26:47.988 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:47.971 7f8b5c325700 1 -- 172.21.2.102:0/2581322995 wait complete. 2021-02-10T11:26:47.989 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:47.986 7f8b5c325700 1 Processor -- start 2021-02-10T11:26:47.989 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:47.986 7f8b5c325700 1 -- start start 2021-02-10T11:26:47.989 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:47.986 7f8b5c325700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8b54195650 0x7f8b540b5560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:47.989 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:47.986 7f8b5c325700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8b540b5b80 0x7f8b541a2e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:47.989 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:47.986 7f8b5c325700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8b540acfa0 0x7f8b541a3440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:47.990 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:47.986 7f8b5c325700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b540b6010 con 0x7f8b540b5b80 2021-02-10T11:26:47.990 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:47.986 7f8b5c325700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b541a3a60 con 0x7f8b54195650 2021-02-10T11:26:47.990 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:47.986 7f8b5c325700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b541a3c40 con 0x7f8b540acfa0 2021-02-10T11:26:47.990 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:47.986 7f8b59940700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8b540b5b80 0x7f8b541a2e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:47.990 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:47.986 7f8b59940700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8b540b5b80 0x7f8b541a2e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:58970/0 (socket says 172.21.2.102:58970) 2021-02-10T11:26:47.990 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:47.986 7f8b59940700 1 -- 172.21.2.102:0/2227339737 learned_addr learned my addr 172.21.2.102:0/2227339737 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:26:47.990 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:47.987 7f8b5a141700 1 --2- 172.21.2.102:0/2227339737 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8b54195650 0x7f8b540b5560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:47.990 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:47.987 7f8b5a942700 1 --2- 172.21.2.102:0/2227339737 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8b540acfa0 0x7f8b541a3440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:47.990 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:47.987 7f8b59940700 1 -- 172.21.2.102:0/2227339737 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8b540acfa0 msgr2=0x7f8b541a3440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:47.990 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:47.987 7f8b59940700 1 --2- 172.21.2.102:0/2227339737 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8b540acfa0 0x7f8b541a3440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:47.990 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:47.987 7f8b59940700 1 -- 172.21.2.102:0/2227339737 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8b54195650 msgr2=0x7f8b540b5560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:47.990 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:47.987 7f8b59940700 1 --2- 172.21.2.102:0/2227339737 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8b54195650 0x7f8b540b5560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:47.990 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:47.987 7f8b59940700 1 -- 172.21.2.102:0/2227339737 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b540bcae0 con 0x7f8b540b5b80 2021-02-10T11:26:47.991 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:47.987 7f8b59940700 1 --2- 172.21.2.102:0/2227339737 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8b540b5b80 0x7f8b541a2e20 secure :-1 s=READY pgs=1646 cs=0 l=1 rev1=1 rx=0x7f8b4400b5e0 tx=0x7f8b4400b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:47.991 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:47.987 7f8b4affd700 1 -- 172.21.2.102:0/2227339737 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8b44000940 con 0x7f8b540b5b80 2021-02-10T11:26:47.991 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:47.987 7f8b4affd700 1 -- 172.21.2.102:0/2227339737 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b4400bd70 con 0x7f8b540b5b80 2021-02-10T11:26:47.991 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:47.987 7f8b5c325700 1 -- 172.21.2.102:0/2227339737 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b540bcae0 con 0x7f8b540b5b80 2021-02-10T11:26:47.991 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:47.987 7f8b4affd700 1 -- 172.21.2.102:0/2227339737 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8b44002270 con 0x7f8b540b5b80 2021-02-10T11:26:47.991 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:47.987 7f8b5c325700 1 -- 172.21.2.102:0/2227339737 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b540bcae0 con 0x7f8b540b5b80 2021-02-10T11:26:47.991 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:47.988 7f8b4affd700 1 -- 172.21.2.102:0/2227339737 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f8b44002020 con 0x7f8b540b5b80 2021-02-10T11:26:47.991 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:47.988 7f8b4affd700 1 --2- 172.21.2.102:0/2227339737 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8b3c018730 0x7f8b3c01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:47.991 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:47.988 7f8b4affd700 1 -- 172.21.2.102:0/2227339737 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f8b44032680 con 0x7f8b540b5b80 2021-02-10T11:26:47.991 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:47.988 7f8b5a141700 1 --2- 172.21.2.102:0/2227339737 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8b3c018730 0x7f8b3c01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:47.991 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:47.988 7f8b5a141700 1 --2- 172.21.2.102:0/2227339737 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8b3c018730 0x7f8b3c01ac10 crc :-1 s=READY pgs=749 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:48.018 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:48.019 7f8b5c325700 1 -- 172.21.2.102:0/2227339737 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b541a3c40 con 0x7f8b540b5b80 2021-02-10T11:26:48.021 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:48.021 7f8b4affd700 1 -- 172.21.2.102:0/2227339737 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f8b44000c90 con 0x7f8b540b5b80 2021-02-10T11:26:48.146 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:48.146 7f8b5c325700 1 -- 172.21.2.102:0/2227339737 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8b541a3c40 con 0x7f8b540b5b80 2021-02-10T11:26:48.146 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:48.146 7f8b4affd700 1 -- 172.21.2.102:0/2227339737 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v74) v1 ==== 74+0+14009 (secure 0 0 0) 0x7f8b44002000 con 0x7f8b540b5b80 2021-02-10T11:26:48.146 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:26:48.147 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":74,"fsid":"68db30ea-dbce-4aef-a6df-f01c6fea0513","created":"2021-02-10 11:03:06.331557","modified":"2021-02-10 11:26:47.543298","last_up_change":"2021-02-10 11:03:32.690562","last_in_change":"2021-02-10 11:03:32.690562","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:03:37.322519","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:26:45.515299","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"74","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":15,"pool_name":"cephfs_data","create_time":"2021-02-10 11:26:46.587944","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"74","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"be691b4d-a039-408a-af80-9d53f02b6f1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6818","nonce":36088},{"type":"v1","addr":"172.21.2.102:6820","nonce":36088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6822","nonce":36088},{"type":"v1","addr":"172.21.2.102:6824","nonce":36088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6830","nonce":36088},{"type":"v1","addr":"172.21.2.102:6832","nonce":36088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6826","nonce":36088},{"type":"v1","addr":"172.21.2.102:6828","nonce":36088}]},"public_addr":"172.21.2.102:6820/36088","cluster_addr":"172.21.2.102:6824/36088","heartbeat_back_addr":"172.21.2.102:6832/36088","heartbeat_front_addr":"172.21.2.102:6828/36088","state":["exists","up"]},{"osd":1,"uuid":"64e3ccc8-9c69-4d7c-865e-af8e83822635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6819","nonce":36087},{"type":"v1","addr":"172.21.2.102:6821","nonce":36087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6823","nonce":36087},{"type":"v1","addr":"172.21.2.102:6825","nonce":36087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6831","nonce":36087},{"type":"v1","addr":"172.21.2.102:6833","nonce":36087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6827","nonce":36087},{"type":"v1","addr":"172.21.2.102:6829","nonce":36087}]},"public_addr":"172.21.2.102:6821/36087","cluster_addr":"172.21.2.102:6825/36087","heartbeat_back_addr":"172.21.2.102:6833/36087","heartbeat_front_addr":"172.21.2.102:6829/36087","state":["exists","up"]},{"osd":2,"uuid":"153700af-747f-481f-bdf4-d6072d3a842f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6810","nonce":36089},{"type":"v1","addr":"172.21.2.102:6811","nonce":36089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6812","nonce":36089},{"type":"v1","addr":"172.21.2.102:6813","nonce":36089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6816","nonce":36089},{"type":"v1","addr":"172.21.2.102:6817","nonce":36089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6814","nonce":36089},{"type":"v1","addr":"172.21.2.102:6815","nonce":36089}]},"public_addr":"172.21.2.102:6811/36089","cluster_addr":"172.21.2.102:6813/36089","heartbeat_back_addr":"172.21.2.102:6817/36089","heartbeat_front_addr":"172.21.2.102:6815/36089","state":["exists","up"]},{"osd":3,"uuid":"cbc678f4-83a9-45e6-8830-2f7c232b1ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6802","nonce":36090},{"type":"v1","addr":"172.21.2.102:6803","nonce":36090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6804","nonce":36090},{"type":"v1","addr":"172.21.2.102:6805","nonce":36090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6808","nonce":36090},{"type":"v1","addr":"172.21.2.102:6809","nonce":36090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6806","nonce":36090},{"type":"v1","addr":"172.21.2.102:6807","nonce":36090}]},"public_addr":"172.21.2.102:6803/36090","cluster_addr":"172.21.2.102:6805/36090","heartbeat_back_addr":"172.21.2.102:6809/36090","heartbeat_front_addr":"172.21.2.102:6807/36090","state":["exists","up"]},{"osd":4,"uuid":"c01e4008-b258-4f10-afd2-9e7f37214351","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6808","nonce":36368},{"type":"v1","addr":"172.21.2.113:6811","nonce":36368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6814","nonce":36368},{"type":"v1","addr":"172.21.2.113:6818","nonce":36368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6826","nonce":36368},{"type":"v1","addr":"172.21.2.113:6828","nonce":36368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6820","nonce":36368},{"type":"v1","addr":"172.21.2.113:6824","nonce":36368}]},"public_addr":"172.21.2.113:6811/36368","cluster_addr":"172.21.2.113:6818/36368","heartbeat_back_addr":"172.21.2.113:6828/36368","heartbeat_front_addr":"172.21.2.113:6824/36368","state":["exists","up"]},{"osd":5,"uuid":"551f6007-daf8-4a98-8ee0-c2b16bcccb5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6802","nonce":36366},{"type":"v1","addr":"172.21.2.113:6805","nonce":36366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6807","nonce":36366},{"type":"v1","addr":"172.21.2.113:6810","nonce":36366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6819","nonce":36366},{"type":"v1","addr":"172.21.2.113:6822","nonce":36366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6812","nonce":36366},{"type":"v1","addr":"172.21.2.113:6815","nonce":36366}]},"public_addr":"172.21.2.113:6805/36366","cluster_addr":"172.21.2.113:6810/36366","heartbeat_back_addr":"172.21.2.113:6822/36366","heartbeat_front_addr":"172.21.2.113:6815/36366","state":["exists","up"]},{"osd":6,"uuid":"bf67c040-0e50-48be-b102-c3943bc9ebef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6800","nonce":36367},{"type":"v1","addr":"172.21.2.113:6801","nonce":36367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6803","nonce":36367},{"type":"v1","addr":"172.21.2.113:6804","nonce":36367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6813","nonce":36367},{"type":"v1","addr":"172.21.2.113:6816","nonce":36367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6806","nonce":36367},{"type":"v1","addr":"172.21.2.113:6809","nonce":36367}]},"public_addr":"172.21.2.113:6801/36367","cluster_addr":"172.21.2.113:6804/36367","heartbeat_back_addr":"172.21.2.113:6816/36367","heartbeat_front_addr":"172.21.2.113:6809/36367","state":["exists","up"]},{"osd":7,"uuid":"f5461f80-018d-428f-be89-2eb2bcbe7bb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6817","nonce":36365},{"type":"v1","addr":"172.21.2.113:6821","nonce":36365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6823","nonce":36365},{"type":"v1","addr":"172.21.2.113:6825","nonce":36365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6830","nonce":36365},{"type":"v1","addr":"172.21.2.113:6831","nonce":36365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6827","nonce":36365},{"type":"v1","addr":"172.21.2.113:6829","nonce":36365}]},"public_addr":"172.21.2.113:6821/36365","cluster_addr":"172.21.2.113:6825/36365","heartbeat_back_addr":"172.21.2.113:6831/36365","heartbeat_front_addr":"172.21.2.113:6829/36365","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:26:48.148 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:48.148 7f8b5c325700 1 -- 172.21.2.102:0/2227339737 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8b3c018730 msgr2=0x7f8b3c01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:48.148 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:48.148 7f8b5c325700 1 --2- 172.21.2.102:0/2227339737 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8b3c018730 0x7f8b3c01ac10 crc :-1 s=READY pgs=749 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:48.148 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:48.148 7f8b5c325700 1 -- 172.21.2.102:0/2227339737 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8b540b5b80 msgr2=0x7f8b541a2e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:48.148 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:48.148 7f8b5c325700 1 --2- 172.21.2.102:0/2227339737 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8b540b5b80 0x7f8b541a2e20 secure :-1 s=READY pgs=1646 cs=0 l=1 rev1=1 rx=0x7f8b4400b5e0 tx=0x7f8b4400b980).stop 2021-02-10T11:26:48.148 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:48.148 7f8b5c325700 1 -- 172.21.2.102:0/2227339737 shutdown_connections 2021-02-10T11:26:48.148 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:48.148 7f8b5c325700 1 --2- 172.21.2.102:0/2227339737 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8b540acfa0 0x7f8b541a3440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:48.148 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:48.148 7f8b5c325700 1 --2- 172.21.2.102:0/2227339737 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8b540b5b80 0x7f8b541a2e20 unknown :-1 s=CLOSED pgs=1646 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:48.148 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:48.148 7f8b5c325700 1 --2- 172.21.2.102:0/2227339737 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8b3c018730 0x7f8b3c01ac10 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:48.148 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:48.148 7f8b5c325700 1 --2- 172.21.2.102:0/2227339737 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8b54195650 0x7f8b540b5560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:48.148 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:48.148 7f8b5c325700 1 -- 172.21.2.102:0/2227339737 >> 172.21.2.102:0/2227339737 conn(0x7f8b540b3040 msgr2=0x7f8b540c3250 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:26:48.149 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:48.148 7f8b5c325700 1 -- 172.21.2.102:0/2227339737 shutdown_connections 2021-02-10T11:26:48.149 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:48.148 7f8b5c325700 1 -- 172.21.2.102:0/2227339737 wait complete. 2021-02-10T11:26:48.157 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:26:48.585 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:48.569 7f5103691700 1 Processor -- start 2021-02-10T11:26:48.585 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:48.569 7f5103691700 1 -- start start 2021-02-10T11:26:48.585 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:48.569 7f5103691700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f50fc0d1c70 0x7f50fc0f20a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:48.586 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:48.569 7f5103691700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50fc0f0b10 con 0x7f50fc0d1870 2021-02-10T11:26:48.586 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:48.569 7f5103691700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50fc1b0be0 con 0x7f50fc0d0340 2021-02-10T11:26:48.586 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:48.569 7f5103691700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50fc0e5ed0 con 0x7f50fc0d1c70 2021-02-10T11:26:48.586 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:48.569 7f51014ad700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f50fc0d1870 0x7f50fc1b03f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:51202/0 (socket says 172.21.2.102:51202) 2021-02-10T11:26:48.586 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:48.569 7f51014ad700 1 -- 172.21.2.102:0/3044816229 learned_addr learned my addr 172.21.2.102:0/3044816229 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:26:48.586 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:48.569 7f5101cae700 1 --2- 172.21.2.102:0/3044816229 >> v2:172.21.2.113:3300/0 conn(0x7f50fc0d1c70 0x7f50fc0f20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:48.586 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:48.570 7f50effff700 1 -- 172.21.2.102:0/3044816229 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 700938530 0 0) 0x7f50f8000ff0 con 0x7f50fc0d1870 2021-02-10T11:26:48.586 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:48.570 7f50effff700 1 -- 172.21.2.102:0/3044816229 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50e8002fe0 con 0x7f50fc0d1870 2021-02-10T11:26:48.586 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:48.570 7f5101cae700 1 -- 172.21.2.102:0/3044816229 >> v1:172.21.2.113:6789/0 conn(0x7f50fc0d0340 legacy=0x7f50fc0ef640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:48.586 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:48.570 7f5101cae700 1 -- 172.21.2.102:0/3044816229 >> v1:172.21.2.102:6789/0 conn(0x7f50fc0d1870 legacy=0x7f50fc1b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:48.587 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:48.570 7f5101cae700 1 -- 172.21.2.102:0/3044816229 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50fc0efe30 con 0x7f50fc0d1c70 2021-02-10T11:26:48.587 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:48.570 7f5101cae700 1 --2- 172.21.2.102:0/3044816229 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f50fc0d1c70 0x7f50fc0f20a0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 rx=0x7f50f4006e50 tx=0x7f50f4009e70).ready entity=mon.1 client_cookie=6124c63f66153719 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:48.587 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:48.570 7f50effff700 1 -- 172.21.2.102:0/3044816229 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f50f4006a20 con 0x7f50fc0d1c70 2021-02-10T11:26:48.587 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:48.570 7f50effff700 1 -- 172.21.2.102:0/3044816229 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50f4006790 con 0x7f50fc0d1c70 2021-02-10T11:26:48.587 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:48.570 7f50effff700 1 -- 172.21.2.102:0/3044816229 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f50f4000f90 con 0x7f50fc0d1c70 2021-02-10T11:26:48.587 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:48.570 7f5103691700 1 -- 172.21.2.102:0/3044816229 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f50fc0d1c70 msgr2=0x7f50fc0f20a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:48.587 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:48.570 7f5103691700 1 --2- 172.21.2.102:0/3044816229 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f50fc0d1c70 0x7f50fc0f20a0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 rx=0x7f50f4006e50 tx=0x7f50f4009e70).stop 2021-02-10T11:26:48.587 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:48.571 7f5103691700 1 -- 172.21.2.102:0/3044816229 shutdown_connections 2021-02-10T11:26:48.587 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:48.571 7f5103691700 1 --2- 172.21.2.102:0/3044816229 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f50fc0d1c70 0x7f50fc0f20a0 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:48.587 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:48.571 7f5103691700 1 -- 172.21.2.102:0/3044816229 >> 172.21.2.102:0/3044816229 conn(0x7f50fc0ce450 msgr2=0x7f50fc0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:26:48.587 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:48.571 7f5103691700 1 -- 172.21.2.102:0/3044816229 shutdown_connections 2021-02-10T11:26:48.587 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:48.571 7f5103691700 1 -- 172.21.2.102:0/3044816229 wait complete. 2021-02-10T11:26:48.589 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:48.586 7f5103691700 1 Processor -- start 2021-02-10T11:26:48.589 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:48.586 7f5103691700 1 -- start start 2021-02-10T11:26:48.589 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:48.586 7f5103691700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f50fc1947b0 0x7f50fc0a69a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:48.589 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:48.586 7f5103691700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f50fc0a6fc0 0x7f50fc1a15d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:48.589 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:48.586 7f5103691700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f50fc1a1bf0 0x7f50fc1a1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:48.589 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:48.586 7f5103691700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50fc0a73c0 con 0x7f50fc1947b0 2021-02-10T11:26:48.590 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:48.586 7f5103691700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50fc18f170 con 0x7f50fc1a1bf0 2021-02-10T11:26:48.590 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:48.586 7f5103691700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50fc194bb0 con 0x7f50fc0a6fc0 2021-02-10T11:26:48.590 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:48.587 7f51014ad700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f50fc1947b0 0x7f50fc0a69a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:48.590 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:48.587 7f51014ad700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f50fc1947b0 0x7f50fc0a69a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:58982/0 (socket says 172.21.2.102:58982) 2021-02-10T11:26:48.590 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:48.587 7f51014ad700 1 -- 172.21.2.102:0/2477695967 learned_addr learned my addr 172.21.2.102:0/2477695967 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:26:48.590 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:48.587 7f5101cae700 1 --2- 172.21.2.102:0/2477695967 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f50fc1a1bf0 0x7f50fc1a1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:48.590 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:48.587 7f5100cac700 1 --2- 172.21.2.102:0/2477695967 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f50fc0a6fc0 0x7f50fc1a15d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:48.590 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:48.587 7f51014ad700 1 -- 172.21.2.102:0/2477695967 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f50fc0a6fc0 msgr2=0x7f50fc1a15d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:48.590 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:48.587 7f51014ad700 1 --2- 172.21.2.102:0/2477695967 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f50fc0a6fc0 0x7f50fc1a15d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:48.590 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:48.587 7f51014ad700 1 -- 172.21.2.102:0/2477695967 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f50fc1a1bf0 msgr2=0x7f50fc1a1ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:48.590 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:48.587 7f51014ad700 1 --2- 172.21.2.102:0/2477695967 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f50fc1a1bf0 0x7f50fc1a1ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:48.590 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:48.587 7f51014ad700 1 -- 172.21.2.102:0/2477695967 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50fc18ec70 con 0x7f50fc1947b0 2021-02-10T11:26:48.590 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:48.587 7f51014ad700 1 --2- 172.21.2.102:0/2477695967 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f50fc1947b0 0x7f50fc0a69a0 secure :-1 s=READY pgs=1648 cs=0 l=1 rev1=1 rx=0x7f50f800a5d0 tx=0x7f50f800a970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:48.591 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:48.587 7f50edffb700 1 -- 172.21.2.102:0/2477695967 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f50f800ad60 con 0x7f50fc1947b0 2021-02-10T11:26:48.591 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:48.587 7f50edffb700 1 -- 172.21.2.102:0/2477695967 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50f8004350 con 0x7f50fc1947b0 2021-02-10T11:26:48.591 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:48.587 7f50edffb700 1 -- 172.21.2.102:0/2477695967 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f50f8004830 con 0x7f50fc1947b0 2021-02-10T11:26:48.591 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:48.587 7f5103691700 1 -- 172.21.2.102:0/2477695967 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50fc18ec70 con 0x7f50fc1947b0 2021-02-10T11:26:48.591 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:48.587 7f5103691700 1 -- 172.21.2.102:0/2477695967 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50fc1a2900 con 0x7f50fc1947b0 2021-02-10T11:26:48.591 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:48.588 7f50edffb700 1 -- 172.21.2.102:0/2477695967 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f50f8000d00 con 0x7f50fc1947b0 2021-02-10T11:26:48.591 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:48.588 7f50edffb700 1 --2- 172.21.2.102:0/2477695967 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f50dc0186c0 0x7f50dc01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:48.591 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:48.588 7f50edffb700 1 -- 172.21.2.102:0/2477695967 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f50f80323f0 con 0x7f50fc1947b0 2021-02-10T11:26:48.591 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:48.588 7f5100cac700 1 --2- 172.21.2.102:0/2477695967 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f50dc0186c0 0x7f50dc01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:48.591 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:48.588 7f5100cac700 1 --2- 172.21.2.102:0/2477695967 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f50dc0186c0 0x7f50dc01abc0 crc :-1 s=READY pgs=750 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:48.619 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:48.619 7f5103691700 1 -- 172.21.2.102:0/2477695967 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50fc0ee0e0 con 0x7f50fc1947b0 2021-02-10T11:26:48.621 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:48.622 7f50edffb700 1 -- 172.21.2.102:0/2477695967 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f50f8000940 con 0x7f50fc1947b0 2021-02-10T11:26:48.747 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:48.747 7f5103691700 1 -- 172.21.2.102:0/2477695967 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f50fc0ce450 con 0x7f50fc1947b0 2021-02-10T11:26:48.748 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:48.748 7f50edffb700 1 -- 172.21.2.102:0/2477695967 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v75) v1 ==== 74+0+13996 (secure 0 0 0) 0x7f50f8000d00 con 0x7f50fc1947b0 2021-02-10T11:26:48.748 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:26:48.749 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":75,"fsid":"68db30ea-dbce-4aef-a6df-f01c6fea0513","created":"2021-02-10 11:03:06.331557","modified":"2021-02-10 11:26:48.545650","last_up_change":"2021-02-10 11:03:32.690562","last_in_change":"2021-02-10 11:03:32.690562","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:03:37.322519","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:26:45.515299","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"74","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":15,"pool_name":"cephfs_data","create_time":"2021-02-10 11:26:46.587944","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"be691b4d-a039-408a-af80-9d53f02b6f1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6818","nonce":36088},{"type":"v1","addr":"172.21.2.102:6820","nonce":36088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6822","nonce":36088},{"type":"v1","addr":"172.21.2.102:6824","nonce":36088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6830","nonce":36088},{"type":"v1","addr":"172.21.2.102:6832","nonce":36088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6826","nonce":36088},{"type":"v1","addr":"172.21.2.102:6828","nonce":36088}]},"public_addr":"172.21.2.102:6820/36088","cluster_addr":"172.21.2.102:6824/36088","heartbeat_back_addr":"172.21.2.102:6832/36088","heartbeat_front_addr":"172.21.2.102:6828/36088","state":["exists","up"]},{"osd":1,"uuid":"64e3ccc8-9c69-4d7c-865e-af8e83822635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6819","nonce":36087},{"type":"v1","addr":"172.21.2.102:6821","nonce":36087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6823","nonce":36087},{"type":"v1","addr":"172.21.2.102:6825","nonce":36087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6831","nonce":36087},{"type":"v1","addr":"172.21.2.102:6833","nonce":36087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6827","nonce":36087},{"type":"v1","addr":"172.21.2.102:6829","nonce":36087}]},"public_addr":"172.21.2.102:6821/36087","cluster_addr":"172.21.2.102:6825/36087","heartbeat_back_addr":"172.21.2.102:6833/36087","heartbeat_front_addr":"172.21.2.102:6829/36087","state":["exists","up"]},{"osd":2,"uuid":"153700af-747f-481f-bdf4-d6072d3a842f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6810","nonce":36089},{"type":"v1","addr":"172.21.2.102:6811","nonce":36089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6812","nonce":36089},{"type":"v1","addr":"172.21.2.102:6813","nonce":36089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6816","nonce":36089},{"type":"v1","addr":"172.21.2.102:6817","nonce":36089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6814","nonce":36089},{"type":"v1","addr":"172.21.2.102:6815","nonce":36089}]},"public_addr":"172.21.2.102:6811/36089","cluster_addr":"172.21.2.102:6813/36089","heartbeat_back_addr":"172.21.2.102:6817/36089","heartbeat_front_addr":"172.21.2.102:6815/36089","state":["exists","up"]},{"osd":3,"uuid":"cbc678f4-83a9-45e6-8830-2f7c232b1ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6802","nonce":36090},{"type":"v1","addr":"172.21.2.102:6803","nonce":36090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6804","nonce":36090},{"type":"v1","addr":"172.21.2.102:6805","nonce":36090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6808","nonce":36090},{"type":"v1","addr":"172.21.2.102:6809","nonce":36090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6806","nonce":36090},{"type":"v1","addr":"172.21.2.102:6807","nonce":36090}]},"public_addr":"172.21.2.102:6803/36090","cluster_addr":"172.21.2.102:6805/36090","heartbeat_back_addr":"172.21.2.102:6809/36090","heartbeat_front_addr":"172.21.2.102:6807/36090","state":["exists","up"]},{"osd":4,"uuid":"c01e4008-b258-4f10-afd2-9e7f37214351","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6808","nonce":36368},{"type":"v1","addr":"172.21.2.113:6811","nonce":36368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6814","nonce":36368},{"type":"v1","addr":"172.21.2.113:6818","nonce":36368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6826","nonce":36368},{"type":"v1","addr":"172.21.2.113:6828","nonce":36368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6820","nonce":36368},{"type":"v1","addr":"172.21.2.113:6824","nonce":36368}]},"public_addr":"172.21.2.113:6811/36368","cluster_addr":"172.21.2.113:6818/36368","heartbeat_back_addr":"172.21.2.113:6828/36368","heartbeat_front_addr":"172.21.2.113:6824/36368","state":["exists","up"]},{"osd":5,"uuid":"551f6007-daf8-4a98-8ee0-c2b16bcccb5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6802","nonce":36366},{"type":"v1","addr":"172.21.2.113:6805","nonce":36366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6807","nonce":36366},{"type":"v1","addr":"172.21.2.113:6810","nonce":36366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6819","nonce":36366},{"type":"v1","addr":"172.21.2.113:6822","nonce":36366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6812","nonce":36366},{"type":"v1","addr":"172.21.2.113:6815","nonce":36366}]},"public_addr":"172.21.2.113:6805/36366","cluster_addr":"172.21.2.113:6810/36366","heartbeat_back_addr":"172.21.2.113:6822/36366","heartbeat_front_addr":"172.21.2.113:6815/36366","state":["exists","up"]},{"osd":6,"uuid":"bf67c040-0e50-48be-b102-c3943bc9ebef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6800","nonce":36367},{"type":"v1","addr":"172.21.2.113:6801","nonce":36367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6803","nonce":36367},{"type":"v1","addr":"172.21.2.113:6804","nonce":36367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6813","nonce":36367},{"type":"v1","addr":"172.21.2.113:6816","nonce":36367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6806","nonce":36367},{"type":"v1","addr":"172.21.2.113:6809","nonce":36367}]},"public_addr":"172.21.2.113:6801/36367","cluster_addr":"172.21.2.113:6804/36367","heartbeat_back_addr":"172.21.2.113:6816/36367","heartbeat_front_addr":"172.21.2.113:6809/36367","state":["exists","up"]},{"osd":7,"uuid":"f5461f80-018d-428f-be89-2eb2bcbe7bb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6817","nonce":36365},{"type":"v1","addr":"172.21.2.113:6821","nonce":36365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6823","nonce":36365},{"type":"v1","addr":"172.21.2.113:6825","nonce":36365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6830","nonce":36365},{"type":"v1","addr":"172.21.2.113:6831","nonce":36365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6827","nonce":36365},{"type":"v1","addr":"172.21.2.113:6829","nonce":36365}]},"public_addr":"172.21.2.113:6821/36365","cluster_addr":"172.21.2.113:6825/36365","heartbeat_back_addr":"172.21.2.113:6831/36365","heartbeat_front_addr":"172.21.2.113:6829/36365","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:26:48.750 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:48.750 7f5103691700 1 -- 172.21.2.102:0/2477695967 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f50dc0186c0 msgr2=0x7f50dc01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:48.750 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:48.750 7f5103691700 1 --2- 172.21.2.102:0/2477695967 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f50dc0186c0 0x7f50dc01abc0 crc :-1 s=READY pgs=750 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:48.750 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:48.750 7f5103691700 1 -- 172.21.2.102:0/2477695967 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f50fc1947b0 msgr2=0x7f50fc0a69a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:48.750 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:48.750 7f5103691700 1 --2- 172.21.2.102:0/2477695967 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f50fc1947b0 0x7f50fc0a69a0 secure :-1 s=READY pgs=1648 cs=0 l=1 rev1=1 rx=0x7f50f800a5d0 tx=0x7f50f800a970).stop 2021-02-10T11:26:48.750 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:48.750 7f5103691700 1 -- 172.21.2.102:0/2477695967 shutdown_connections 2021-02-10T11:26:48.751 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:48.750 7f5103691700 1 --2- 172.21.2.102:0/2477695967 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f50dc0186c0 0x7f50dc01abc0 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:48.751 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:48.750 7f5103691700 1 --2- 172.21.2.102:0/2477695967 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f50fc1a1bf0 0x7f50fc1a1ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:48.751 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:48.750 7f5103691700 1 --2- 172.21.2.102:0/2477695967 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f50fc0a6fc0 0x7f50fc1a15d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:48.751 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:48.750 7f5103691700 1 --2- 172.21.2.102:0/2477695967 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f50fc1947b0 0x7f50fc0a69a0 unknown :-1 s=CLOSED pgs=1648 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:48.751 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:48.750 7f5103691700 1 -- 172.21.2.102:0/2477695967 >> 172.21.2.102:0/2477695967 conn(0x7f50fc1a38c0 msgr2=0x7f50fc1a2f00 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:26:48.751 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:48.750 7f5103691700 1 -- 172.21.2.102:0/2477695967 shutdown_connections 2021-02-10T11:26:48.751 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:48.750 7f5103691700 1 -- 172.21.2.102:0/2477695967 wait complete. 2021-02-10T11:26:48.759 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2021-02-10T11:26:49.190 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:49.174 7f53b80e8700 1 Processor -- start 2021-02-10T11:26:49.190 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:49.174 7f53b80e8700 1 -- start start 2021-02-10T11:26:49.190 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:49.175 7f53b80e8700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f53b00d1870 0x7f53b01b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:49.191 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:49.175 7f53b80e8700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f53b00cff70 0x7f53b0109130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:49.191 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:49.175 7f53b80e8700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53b0109a60 con 0x7f53b01b0ae0 2021-02-10T11:26:49.191 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:49.175 7f53b80e8700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53b0109750 con 0x7f53b00cff70 2021-02-10T11:26:49.191 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:49.175 7f53b80e8700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53b0088070 con 0x7f53b00d1870 2021-02-10T11:26:49.191 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:49.175 7f53b6705700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f53b00cff70 0x7f53b0109130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:49.191 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:49.175 7f53b6705700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f53b00cff70 0x7f53b0109130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:58990/0 (socket says 172.21.2.102:58990) 2021-02-10T11:26:49.191 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:49.175 7f53b6705700 1 -- 172.21.2.102:0/2786087376 learned_addr learned my addr 172.21.2.102:0/2786087376 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:26:49.191 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:49.175 7f53b6705700 1 -- 172.21.2.102:0/2786087376 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f53b00d1870 msgr2=0x7f53b01b04c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:26:49.191 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:49.175 7f53b6705700 1 --2- 172.21.2.102:0/2786087376 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f53b00d1870 0x7f53b01b04c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:49.191 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:49.175 7f53b6705700 1 -- 172.21.2.102:0/2786087376 >> v1:172.21.2.113:6789/0 conn(0x7f53b01b0ae0 legacy=0x7f53b0108940 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:26:49.191 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:49.175 7f53b6705700 1 -- 172.21.2.102:0/2786087376 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53b0088260 con 0x7f53b00cff70 2021-02-10T11:26:49.192 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:49.175 7f53b6705700 1 --2- 172.21.2.102:0/2786087376 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f53b00cff70 0x7f53b0109130 secure :-1 s=READY pgs=1649 cs=0 l=1 rev1=1 rx=0x7f53a8006f00 tx=0x7f53a8009f10).ready entity=mon.0 client_cookie=93ba2592750b36ec server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:49.192 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:49.176 7f53b4f02700 1 -- 172.21.2.102:0/2786087376 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f53a800b3b0 con 0x7f53b00cff70 2021-02-10T11:26:49.192 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:49.176 7f53b4f02700 1 -- 172.21.2.102:0/2786087376 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53a800b630 con 0x7f53b00cff70 2021-02-10T11:26:49.192 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:49.176 7f53b4f02700 1 -- 172.21.2.102:0/2786087376 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f53a800bb10 con 0x7f53b00cff70 2021-02-10T11:26:49.192 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:49.176 7f53b80e8700 1 -- 172.21.2.102:0/2786087376 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f53b00cff70 msgr2=0x7f53b0109130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:49.192 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:49.176 7f53b80e8700 1 --2- 172.21.2.102:0/2786087376 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f53b00cff70 0x7f53b0109130 secure :-1 s=READY pgs=1649 cs=0 l=1 rev1=1 rx=0x7f53a8006f00 tx=0x7f53a8009f10).stop 2021-02-10T11:26:49.192 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:49.176 7f53b80e8700 1 -- 172.21.2.102:0/2786087376 shutdown_connections 2021-02-10T11:26:49.192 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:49.176 7f53b80e8700 1 --2- 172.21.2.102:0/2786087376 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f53b00cff70 0x7f53b0109130 unknown :-1 s=CLOSED pgs=1649 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:49.192 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:49.176 7f53b80e8700 1 --2- 172.21.2.102:0/2786087376 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f53b00d1870 0x7f53b01b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:49.192 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:49.176 7f53b80e8700 1 -- 172.21.2.102:0/2786087376 >> 172.21.2.102:0/2786087376 conn(0x7f53b00ce420 msgr2=0x7f53b00d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:26:49.192 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:49.177 7f53b80e8700 1 -- 172.21.2.102:0/2786087376 shutdown_connections 2021-02-10T11:26:49.192 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:49.177 7f53b80e8700 1 -- 172.21.2.102:0/2786087376 wait complete. 2021-02-10T11:26:49.194 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:49.191 7f53b80e8700 1 Processor -- start 2021-02-10T11:26:49.194 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:49.191 7f53b80e8700 1 -- start start 2021-02-10T11:26:49.194 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:49.191 7f53b80e8700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f53b018f9d0 0x7f53b0084cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:49.195 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:49.191 7f53b80e8700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f53b00c23e0 0x7f53b01a44b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:49.195 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:49.191 7f53b80e8700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f53b00c27e0 0x7f53b00859f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:49.195 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:49.191 7f53b80e8700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53b00854f0 con 0x7f53b00c27e0 2021-02-10T11:26:49.195 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:49.191 7f53b80e8700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53b018fdd0 con 0x7f53b00c23e0 2021-02-10T11:26:49.195 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:49.191 7f53b80e8700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53b01a4ad0 con 0x7f53b018f9d0 2021-02-10T11:26:49.195 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:49.191 7f53b6705700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f53b00c27e0 0x7f53b00859f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:49.196 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:49.191 7f53b6705700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f53b00c27e0 0x7f53b00859f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:58998/0 (socket says 172.21.2.102:58998) 2021-02-10T11:26:49.196 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:49.191 7f53b6705700 1 -- 172.21.2.102:0/2799966816 learned_addr learned my addr 172.21.2.102:0/2799966816 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:26:49.196 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:49.192 7f53b6705700 1 -- 172.21.2.102:0/2799966816 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f53b018f9d0 msgr2=0x7f53b0084cb0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:26:49.196 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:49.192 7f53b6705700 1 --2- 172.21.2.102:0/2799966816 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f53b018f9d0 0x7f53b0084cb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:49.196 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:49.192 7f53b6705700 1 -- 172.21.2.102:0/2799966816 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f53b00c23e0 msgr2=0x7f53b01a44b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:26:49.197 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:49.192 7f53b6705700 1 --2- 172.21.2.102:0/2799966816 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f53b00c23e0 0x7f53b01a44b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:49.197 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:49.192 7f53b6705700 1 -- 172.21.2.102:0/2799966816 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53b01a4cf0 con 0x7f53b00c27e0 2021-02-10T11:26:49.197 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:49.192 7f53b6705700 1 --2- 172.21.2.102:0/2799966816 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f53b00c27e0 0x7f53b00859f0 secure :-1 s=READY pgs=1650 cs=0 l=1 rev1=1 rx=0x7f53a8009390 tx=0x7f53a800bd40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:49.197 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:49.192 7f53a27fc700 1 -- 172.21.2.102:0/2799966816 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f53a8003030 con 0x7f53b00c27e0 2021-02-10T11:26:49.197 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:49.192 7f53a27fc700 1 -- 172.21.2.102:0/2799966816 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53a8006c80 con 0x7f53b00c27e0 2021-02-10T11:26:49.198 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:49.192 7f53a27fc700 1 -- 172.21.2.102:0/2799966816 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f53a8001480 con 0x7f53b00c27e0 2021-02-10T11:26:49.198 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:49.192 7f53b80e8700 1 -- 172.21.2.102:0/2799966816 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53b01a4cf0 con 0x7f53b00c27e0 2021-02-10T11:26:49.198 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:49.192 7f53b80e8700 1 -- 172.21.2.102:0/2799966816 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53b01a4cf0 con 0x7f53b00c27e0 2021-02-10T11:26:49.198 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:49.193 7f53a27fc700 1 -- 172.21.2.102:0/2799966816 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f53a8001270 con 0x7f53b00c27e0 2021-02-10T11:26:49.198 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:49.193 7f53a27fc700 1 --2- 172.21.2.102:0/2799966816 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5398018730 0x7f539801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:49.199 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:49.193 7f53a27fc700 1 -- 172.21.2.102:0/2799966816 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f53a80329f0 con 0x7f53b00c27e0 2021-02-10T11:26:49.199 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:49.193 7f53b5f04700 1 --2- 172.21.2.102:0/2799966816 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5398018730 0x7f539801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:49.199 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:49.193 7f53b5f04700 1 --2- 172.21.2.102:0/2799966816 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5398018730 0x7f539801ac10 crc :-1 s=READY pgs=751 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:49.225 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:49.225 7f53b80e8700 1 -- 172.21.2.102:0/2799966816 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53b0082a70 con 0x7f53b00c27e0 2021-02-10T11:26:49.227 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:49.228 7f53a27fc700 1 -- 172.21.2.102:0/2799966816 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f53a8003020 con 0x7f53b00c27e0 2021-02-10T11:26:49.364 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:49.364 7f53b80e8700 1 -- 172.21.2.102:0/2799966816 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"} v 0) v1 -- 0x7f53b0082a70 con 0x7f53b00c27e0 2021-02-10T11:26:49.553 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:49.553 7f53a27fc700 1 -- 172.21.2.102:0/2799966816 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"}]=0 v57) v1 ==== 118+0+0 (secure 0 0 0) 0x7f53a8006c80 con 0x7f53b00c27e0 2021-02-10T11:26:49.581 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:49.582 7f53b80e8700 1 -- 172.21.2.102:0/2799966816 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5398018730 msgr2=0x7f539801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:49.582 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:49.582 7f53b80e8700 1 --2- 172.21.2.102:0/2799966816 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5398018730 0x7f539801ac10 crc :-1 s=READY pgs=751 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:49.582 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:49.582 7f53b80e8700 1 -- 172.21.2.102:0/2799966816 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f53b00c27e0 msgr2=0x7f53b00859f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:49.582 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:49.582 7f53b80e8700 1 --2- 172.21.2.102:0/2799966816 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f53b00c27e0 0x7f53b00859f0 secure :-1 s=READY pgs=1650 cs=0 l=1 rev1=1 rx=0x7f53a8009390 tx=0x7f53a800bd40).stop 2021-02-10T11:26:49.582 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:49.582 7f53b80e8700 1 -- 172.21.2.102:0/2799966816 shutdown_connections 2021-02-10T11:26:49.582 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:49.582 7f53b80e8700 1 --2- 172.21.2.102:0/2799966816 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f53b00c27e0 0x7f53b00859f0 unknown :-1 s=CLOSED pgs=1650 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:49.582 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:49.582 7f53b80e8700 1 --2- 172.21.2.102:0/2799966816 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5398018730 0x7f539801ac10 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:49.582 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:49.582 7f53b80e8700 1 --2- 172.21.2.102:0/2799966816 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f53b00c23e0 0x7f53b01a44b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:49.582 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:49.582 7f53b80e8700 1 --2- 172.21.2.102:0/2799966816 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f53b018f9d0 0x7f53b0084cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:49.582 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:49.582 7f53b80e8700 1 -- 172.21.2.102:0/2799966816 >> 172.21.2.102:0/2799966816 conn(0x7f53b00b5dc0 msgr2=0x7f53b00c2d90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:26:49.583 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:49.582 7f53b80e8700 1 -- 172.21.2.102:0/2799966816 shutdown_connections 2021-02-10T11:26:49.583 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:49.582 7f53b80e8700 1 -- 172.21.2.102:0/2799966816 wait complete. 2021-02-10T11:26:49.590 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:26:50.016 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:50.000 7f006a666700 1 Processor -- start 2021-02-10T11:26:50.016 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:50.000 7f006a666700 1 -- start start 2021-02-10T11:26:50.016 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:50.000 7f006a666700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f00640d1870 0x7f00641b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:50.016 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:50.000 7f006a666700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f00641b0a10 0x7f00640cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:50.016 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:50.000 7f006a666700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00640f0b10 con 0x7f00640d1c70 2021-02-10T11:26:50.017 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:50.000 7f006a666700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00640e5ed0 con 0x7f00640d1870 2021-02-10T11:26:50.017 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:50.000 7f006a666700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00640f0650 con 0x7f00641b0a10 2021-02-10T11:26:50.017 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:50.000 7f0068c83700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f00640d1c70 0x7f00640cf950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:51228/0 (socket says 172.21.2.102:51228) 2021-02-10T11:26:50.017 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:50.000 7f0068c83700 1 -- 172.21.2.102:0/958030860 learned_addr learned my addr 172.21.2.102:0/958030860 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:26:50.017 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:50.001 7f0063fff700 1 --2- 172.21.2.102:0/958030860 >> v2:172.21.2.113:3300/0 conn(0x7f00640d1870 0x7f00641b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:50.017 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:50.001 7f00637fe700 1 --2- 172.21.2.102:0/958030860 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f00641b0a10 0x7f00640cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:50.017 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:50.001 7f0062ffd700 1 -- 172.21.2.102:0/958030860 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4242596640 0 0) 0x7f005c000d30 con 0x7f00640d1c70 2021-02-10T11:26:50.017 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:50.001 7f0062ffd700 1 -- 172.21.2.102:0/958030860 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f004c002fe0 con 0x7f00640d1c70 2021-02-10T11:26:50.017 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:50.001 7f00637fe700 1 -- 172.21.2.102:0/958030860 >> v2:172.21.2.113:3300/0 conn(0x7f00640d1870 msgr2=0x7f00641b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:26:50.017 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:50.001 7f00637fe700 1 --2- 172.21.2.102:0/958030860 >> v2:172.21.2.113:3300/0 conn(0x7f00640d1870 0x7f00641b03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:50.017 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:50.001 7f00637fe700 1 -- 172.21.2.102:0/958030860 >> v1:172.21.2.102:6789/0 conn(0x7f00640d1c70 legacy=0x7f00640cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:50.017 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:50.001 7f00637fe700 1 -- 172.21.2.102:0/958030860 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00640f0870 con 0x7f00641b0a10 2021-02-10T11:26:50.017 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:50.001 7f00637fe700 1 --2- 172.21.2.102:0/958030860 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f00641b0a10 0x7f00640cf330 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7f0058006650 tx=0x7f0058009a80).ready entity=mon.2 client_cookie=8aa2cfa4e353ce72 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:50.018 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:50.002 7f0062ffd700 1 -- 172.21.2.102:0/958030860 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f005800c0c0 con 0x7f00641b0a10 2021-02-10T11:26:50.018 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:50.002 7f0062ffd700 1 -- 172.21.2.102:0/958030860 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f005800c340 con 0x7f00641b0a10 2021-02-10T11:26:50.018 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:50.002 7f0062ffd700 1 -- 172.21.2.102:0/958030860 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f005800c860 con 0x7f00641b0a10 2021-02-10T11:26:50.018 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:50.002 7f006a666700 1 -- 172.21.2.102:0/958030860 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f00641b0a10 msgr2=0x7f00640cf330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:50.018 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:50.002 7f006a666700 1 --2- 172.21.2.102:0/958030860 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f00641b0a10 0x7f00640cf330 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7f0058006650 tx=0x7f0058009a80).stop 2021-02-10T11:26:50.018 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:50.002 7f006a666700 1 -- 172.21.2.102:0/958030860 shutdown_connections 2021-02-10T11:26:50.018 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:50.002 7f006a666700 1 --2- 172.21.2.102:0/958030860 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f00641b0a10 0x7f00640cf330 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:50.018 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:50.002 7f006a666700 1 --2- 172.21.2.102:0/958030860 >> v2:172.21.2.113:3300/0 conn(0x7f00640d1870 0x7f00641b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:50.018 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:50.002 7f006a666700 1 -- 172.21.2.102:0/958030860 >> 172.21.2.102:0/958030860 conn(0x7f00640ce450 msgr2=0x7f00640d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:26:50.018 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:50.002 7f006a666700 1 -- 172.21.2.102:0/958030860 shutdown_connections 2021-02-10T11:26:50.018 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:50.002 7f006a666700 1 -- 172.21.2.102:0/958030860 wait complete. 2021-02-10T11:26:50.020 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:50.017 7f006a666700 1 Processor -- start 2021-02-10T11:26:50.020 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:50.017 7f006a666700 1 -- start start 2021-02-10T11:26:50.020 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:50.017 7f006a666700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0064195650 0x7f00640b5560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:50.020 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:50.017 7f006a666700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f00640b5b80 0x7f00641a2e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:50.020 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:50.017 7f006a666700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f00640acfa0 0x7f00641a3440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:50.020 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:50.017 7f006a666700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00640b6010 con 0x7f00640acfa0 2021-02-10T11:26:50.020 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:50.017 7f006a666700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00641a3a60 con 0x7f0064195650 2021-02-10T11:26:50.020 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:50.017 7f006a666700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00641a3c40 con 0x7f00640b5b80 2021-02-10T11:26:50.020 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:50.017 7f0068c83700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f00640acfa0 0x7f00641a3440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:50.020 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:50.017 7f0068c83700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f00640acfa0 0x7f00641a3440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:59008/0 (socket says 172.21.2.102:59008) 2021-02-10T11:26:50.020 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:50.017 7f0068c83700 1 -- 172.21.2.102:0/3957032559 learned_addr learned my addr 172.21.2.102:0/3957032559 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:26:50.020 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:50.018 7f0068c83700 1 -- 172.21.2.102:0/3957032559 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f00640b5b80 msgr2=0x7f00641a2e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:50.021 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:50.018 7f0068c83700 1 --2- 172.21.2.102:0/3957032559 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f00640b5b80 0x7f00641a2e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:50.021 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:50.018 7f0068c83700 1 -- 172.21.2.102:0/3957032559 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0064195650 msgr2=0x7f00640b5560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:50.021 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:50.018 7f0068c83700 1 --2- 172.21.2.102:0/3957032559 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0064195650 0x7f00640b5560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:50.021 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:50.018 7f0068c83700 1 -- 172.21.2.102:0/3957032559 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00640bcae0 con 0x7f00640acfa0 2021-02-10T11:26:50.021 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:50.018 7f0068c83700 1 --2- 172.21.2.102:0/3957032559 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f00640acfa0 0x7f00641a3440 secure :-1 s=READY pgs=1652 cs=0 l=1 rev1=1 rx=0x7f005c00a940 tx=0x7f005c00ace0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:50.021 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:50.018 7f0060ff9700 1 -- 172.21.2.102:0/3957032559 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f005c000d30 con 0x7f00640acfa0 2021-02-10T11:26:50.021 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:50.018 7f0060ff9700 1 -- 172.21.2.102:0/3957032559 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f005c007d40 con 0x7f00640acfa0 2021-02-10T11:26:50.021 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:50.018 7f006a666700 1 -- 172.21.2.102:0/3957032559 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00640bcae0 con 0x7f00640acfa0 2021-02-10T11:26:50.021 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:50.018 7f0060ff9700 1 -- 172.21.2.102:0/3957032559 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f005c004bc0 con 0x7f00640acfa0 2021-02-10T11:26:50.021 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:50.018 7f006a666700 1 -- 172.21.2.102:0/3957032559 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00640bd070 con 0x7f00640acfa0 2021-02-10T11:26:50.021 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:50.019 7f0060ff9700 1 -- 172.21.2.102:0/3957032559 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f005c007020 con 0x7f00640acfa0 2021-02-10T11:26:50.021 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:50.019 7f0060ff9700 1 --2- 172.21.2.102:0/3957032559 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f00440186c0 0x7f004401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:50.021 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:50.019 7f0060ff9700 1 -- 172.21.2.102:0/3957032559 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f005c032780 con 0x7f00640acfa0 2021-02-10T11:26:50.022 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:50.019 7f0063fff700 1 --2- 172.21.2.102:0/3957032559 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f00440186c0 0x7f004401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:50.022 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:50.019 7f0063fff700 1 --2- 172.21.2.102:0/3957032559 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f00440186c0 0x7f004401abc0 crc :-1 s=READY pgs=752 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:50.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:50.050 7f006a666700 1 -- 172.21.2.102:0/3957032559 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00640f0ab0 con 0x7f00640acfa0 2021-02-10T11:26:50.052 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:50.053 7f0060ff9700 1 -- 172.21.2.102:0/3957032559 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f005c000d30 con 0x7f00640acfa0 2021-02-10T11:26:50.180 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:50.180 7f006a666700 1 -- 172.21.2.102:0/3957032559 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f00640f0ab0 con 0x7f00640acfa0 2021-02-10T11:26:50.181 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:50.181 7f0060ff9700 1 -- 172.21.2.102:0/3957032559 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 57 v57) v1 ==== 94+0+2098 (secure 0 0 0) 0x7f005c006ef0 con 0x7f00640acfa0 2021-02-10T11:26:50.181 INFO:teuthology.orchestra.run.gibba002.stderr:dumped fsmap epoch 57 2021-02-10T11:26:50.182 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:26:50.182 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":57,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8865,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.102:6835/980649634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6834","nonce":980649634},{"type":"v1","addr":"172.21.2.102:6835","nonce":980649634}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":53}],"filesystems":[{"mdsmap":{"epoch":57,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:26:47.543386","modified":"2021-02-10 11:26:49.552060","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6799},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6799":{"gid":6799,"name":"b","rank":0,"incarnation":54,"state":"up:active","state_seq":3,"addr":"172.21.2.113:6833/2002224946","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6832","nonce":2002224946},{"type":"v1","addr":"172.21.2.113:6833","nonce":2002224946}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2021-02-10T11:26:50.182 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:50.183 7f006a666700 1 -- 172.21.2.102:0/3957032559 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f00440186c0 msgr2=0x7f004401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:50.182 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:50.183 7f006a666700 1 --2- 172.21.2.102:0/3957032559 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f00440186c0 0x7f004401abc0 crc :-1 s=READY pgs=752 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:50.183 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:50.183 7f006a666700 1 -- 172.21.2.102:0/3957032559 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f00640acfa0 msgr2=0x7f00641a3440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:50.183 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:50.183 7f006a666700 1 --2- 172.21.2.102:0/3957032559 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f00640acfa0 0x7f00641a3440 secure :-1 s=READY pgs=1652 cs=0 l=1 rev1=1 rx=0x7f005c00a940 tx=0x7f005c00ace0).stop 2021-02-10T11:26:50.183 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:50.183 7f006a666700 1 -- 172.21.2.102:0/3957032559 shutdown_connections 2021-02-10T11:26:50.183 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:50.183 7f006a666700 1 --2- 172.21.2.102:0/3957032559 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f00640acfa0 0x7f00641a3440 unknown :-1 s=CLOSED pgs=1652 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:50.183 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:50.183 7f006a666700 1 --2- 172.21.2.102:0/3957032559 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f00640b5b80 0x7f00641a2e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:50.183 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:50.183 7f006a666700 1 --2- 172.21.2.102:0/3957032559 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f00440186c0 0x7f004401abc0 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:50.183 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:50.183 7f006a666700 1 --2- 172.21.2.102:0/3957032559 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0064195650 0x7f00640b5560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:50.183 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:50.183 7f006a666700 1 -- 172.21.2.102:0/3957032559 >> 172.21.2.102:0/3957032559 conn(0x7f00640b3040 msgr2=0x7f00640c3250 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:26:50.184 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:50.183 7f006a666700 1 -- 172.21.2.102:0/3957032559 shutdown_connections 2021-02-10T11:26:50.184 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:50.183 7f006a666700 1 -- 172.21.2.102:0/3957032559 wait complete. 2021-02-10T11:26:50.191 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:26:50.621 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:50.605 7f4cab0cb700 1 Processor -- start 2021-02-10T11:26:50.622 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:50.605 7f4cab0cb700 1 -- start start 2021-02-10T11:26:50.622 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:50.606 7f4cab0cb700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f4ca40d0340 0x7f4ca40ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:50.622 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:50.606 7f4cab0cb700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f4ca40d1c70 0x7f4ca40efc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:50.622 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:50.606 7f4cab0cb700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ca40f0b10 con 0x7f4ca40d1870 2021-02-10T11:26:50.622 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:50.606 7f4cab0cb700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ca40e5ed0 con 0x7f4ca40d1c70 2021-02-10T11:26:50.622 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:50.606 7f4cab0cb700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ca41b0be0 con 0x7f4ca40d0340 2021-02-10T11:26:50.622 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:50.606 7f4ca96e8700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f4ca40d1c70 0x7f4ca40efc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:50.622 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:50.606 7f4ca96e8700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f4ca40d1c70 0x7f4ca40efc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:59018/0 (socket says 172.21.2.102:59018) 2021-02-10T11:26:50.622 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:50.606 7f4ca96e8700 1 -- 172.21.2.102:0/3133312393 learned_addr learned my addr 172.21.2.102:0/3133312393 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:26:50.622 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:50.606 7f4ca3fff700 1 --2- 172.21.2.102:0/3133312393 >> v2:172.21.2.113:3300/0 conn(0x7f4ca40d0340 0x7f4ca40ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:50.622 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:50.606 7f4ca96e8700 1 -- 172.21.2.102:0/3133312393 >> v2:172.21.2.113:3300/0 conn(0x7f4ca40d0340 msgr2=0x7f4ca40ef640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:26:50.623 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:50.606 7f4ca96e8700 1 --2- 172.21.2.102:0/3133312393 >> v2:172.21.2.113:3300/0 conn(0x7f4ca40d0340 0x7f4ca40ef640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:50.623 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:50.606 7f4ca96e8700 1 -- 172.21.2.102:0/3133312393 >> v1:172.21.2.102:6789/0 conn(0x7f4ca40d1870 legacy=0x7f4ca41b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:50.623 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:50.606 7f4ca96e8700 1 -- 172.21.2.102:0/3133312393 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ca40f20a0 con 0x7f4ca40d1c70 2021-02-10T11:26:50.623 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:50.607 7f4ca96e8700 1 --2- 172.21.2.102:0/3133312393 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4ca40d1c70 0x7f4ca40efc60 secure :-1 s=READY pgs=1654 cs=0 l=1 rev1=1 rx=0x7f4c9c006f00 tx=0x7f4c9c009f10).ready entity=mon.0 client_cookie=ccec50092947204a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:50.623 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:50.607 7f4ca37fe700 1 -- 172.21.2.102:0/3133312393 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4c9c00b3b0 con 0x7f4ca40d1c70 2021-02-10T11:26:50.623 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:50.607 7f4ca37fe700 1 -- 172.21.2.102:0/3133312393 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c9c00b630 con 0x7f4ca40d1c70 2021-02-10T11:26:50.623 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:50.607 7f4ca37fe700 1 -- 172.21.2.102:0/3133312393 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4c9c00bb10 con 0x7f4ca40d1c70 2021-02-10T11:26:50.623 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:50.607 7f4cab0cb700 1 -- 172.21.2.102:0/3133312393 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4ca40d1c70 msgr2=0x7f4ca40efc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:50.623 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:50.607 7f4cab0cb700 1 --2- 172.21.2.102:0/3133312393 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4ca40d1c70 0x7f4ca40efc60 secure :-1 s=READY pgs=1654 cs=0 l=1 rev1=1 rx=0x7f4c9c006f00 tx=0x7f4c9c009f10).stop 2021-02-10T11:26:50.623 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:50.607 7f4cab0cb700 1 -- 172.21.2.102:0/3133312393 shutdown_connections 2021-02-10T11:26:50.623 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:50.607 7f4cab0cb700 1 --2- 172.21.2.102:0/3133312393 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4ca40d1c70 0x7f4ca40efc60 unknown :-1 s=CLOSED pgs=1654 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:50.623 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:50.607 7f4cab0cb700 1 --2- 172.21.2.102:0/3133312393 >> v2:172.21.2.113:3300/0 conn(0x7f4ca40d0340 0x7f4ca40ef640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:50.623 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:50.607 7f4cab0cb700 1 -- 172.21.2.102:0/3133312393 >> 172.21.2.102:0/3133312393 conn(0x7f4ca40ce450 msgr2=0x7f4ca40d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:26:50.624 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:50.607 7f4cab0cb700 1 -- 172.21.2.102:0/3133312393 shutdown_connections 2021-02-10T11:26:50.624 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:50.607 7f4cab0cb700 1 -- 172.21.2.102:0/3133312393 wait complete. 2021-02-10T11:26:50.625 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:50.622 7f4cab0cb700 1 Processor -- start 2021-02-10T11:26:50.625 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:50.622 7f4cab0cb700 1 -- start start 2021-02-10T11:26:50.625 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:50.623 7f4cab0cb700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4ca40970b0 0x7f4ca40ebe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:50.625 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:50.623 7f4cab0cb700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4ca4098810 0x7f4ca4098c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:50.626 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:50.623 7f4cab0cb700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4ca4099230 0x7f4ca40b18c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:50.626 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:50.623 7f4cab0cb700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ca40ec6d0 con 0x7f4ca4099230 2021-02-10T11:26:50.626 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:50.623 7f4cab0cb700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ca40974b0 con 0x7f4ca4098810 2021-02-10T11:26:50.626 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:50.623 7f4cab0cb700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ca40b1ee0 con 0x7f4ca40970b0 2021-02-10T11:26:50.626 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:50.623 7f4ca96e8700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4ca4099230 0x7f4ca40b18c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:50.626 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:50.623 7f4ca96e8700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4ca4099230 0x7f4ca40b18c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:59024/0 (socket says 172.21.2.102:59024) 2021-02-10T11:26:50.626 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:50.623 7f4ca96e8700 1 -- 172.21.2.102:0/2064200634 learned_addr learned my addr 172.21.2.102:0/2064200634 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:26:50.626 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:50.623 7f4ca3fff700 1 --2- 172.21.2.102:0/2064200634 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4ca4098810 0x7f4ca4098c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:50.626 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:50.623 7f4ca96e8700 1 -- 172.21.2.102:0/2064200634 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4ca40970b0 msgr2=0x7f4ca40ebe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:50.626 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:50.623 7f4ca96e8700 1 --2- 172.21.2.102:0/2064200634 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4ca40970b0 0x7f4ca40ebe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:50.626 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:50.623 7f4ca96e8700 1 -- 172.21.2.102:0/2064200634 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4ca4098810 msgr2=0x7f4ca4098c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:50.626 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:50.623 7f4ca96e8700 1 --2- 172.21.2.102:0/2064200634 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4ca4098810 0x7f4ca4098c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:50.627 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:50.623 7f4ca96e8700 1 -- 172.21.2.102:0/2064200634 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ca40b2100 con 0x7f4ca4099230 2021-02-10T11:26:50.627 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:50.623 7f4ca3fff700 1 --2- 172.21.2.102:0/2064200634 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4ca4098810 0x7f4ca4098c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:26:50.627 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:50.623 7f4ca96e8700 1 --2- 172.21.2.102:0/2064200634 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4ca4099230 0x7f4ca40b18c0 secure :-1 s=READY pgs=1655 cs=0 l=1 rev1=1 rx=0x7f4c9c00b9b0 tx=0x7f4c9c00bd50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:50.627 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:50.623 7f4ca17fa700 1 -- 172.21.2.102:0/2064200634 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4c9c003030 con 0x7f4ca4099230 2021-02-10T11:26:50.627 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:50.624 7f4ca17fa700 1 -- 172.21.2.102:0/2064200634 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c9c00c2f0 con 0x7f4ca4099230 2021-02-10T11:26:50.627 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:50.624 7f4ca17fa700 1 -- 172.21.2.102:0/2064200634 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4c9c00c7d0 con 0x7f4ca4099230 2021-02-10T11:26:50.627 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:50.624 7f4cab0cb700 1 -- 172.21.2.102:0/2064200634 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ca40b2100 con 0x7f4ca4099230 2021-02-10T11:26:50.627 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:50.624 7f4cab0cb700 1 -- 172.21.2.102:0/2064200634 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ca40b2100 con 0x7f4ca4099230 2021-02-10T11:26:50.627 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:50.624 7f4ca17fa700 1 -- 172.21.2.102:0/2064200634 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4c9c009450 con 0x7f4ca4099230 2021-02-10T11:26:50.627 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:50.624 7f4ca17fa700 1 --2- 172.21.2.102:0/2064200634 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4c880186c0 0x7f4c8801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:50.627 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:50.624 7f4ca17fa700 1 -- 172.21.2.102:0/2064200634 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f4c9c032960 con 0x7f4ca4099230 2021-02-10T11:26:50.627 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:50.624 7f4ca8ee7700 1 --2- 172.21.2.102:0/2064200634 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4c880186c0 0x7f4c8801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:50.627 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:50.625 7f4ca8ee7700 1 --2- 172.21.2.102:0/2064200634 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4c880186c0 0x7f4c8801abc0 crc :-1 s=READY pgs=753 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:50.655 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:50.656 7f4cab0cb700 1 -- 172.21.2.102:0/2064200634 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ca40ebc00 con 0x7f4ca4099230 2021-02-10T11:26:50.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:50.658 7f4ca17fa700 1 -- 172.21.2.102:0/2064200634 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f4c9c000f90 con 0x7f4ca4099230 2021-02-10T11:26:50.783 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:50.783 7f4cab0cb700 1 -- 172.21.2.102:0/2064200634 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4ca40f0b10 con 0x7f4ca4099230 2021-02-10T11:26:50.784 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:50.784 7f4ca17fa700 1 -- 172.21.2.102:0/2064200634 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v75) v1 ==== 74+0+13996 (secure 0 0 0) 0x7f4c9c009450 con 0x7f4ca4099230 2021-02-10T11:26:50.784 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:26:50.784 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":75,"fsid":"68db30ea-dbce-4aef-a6df-f01c6fea0513","created":"2021-02-10 11:03:06.331557","modified":"2021-02-10 11:26:48.545650","last_up_change":"2021-02-10 11:03:32.690562","last_in_change":"2021-02-10 11:03:32.690562","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:03:37.322519","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:26:45.515299","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"74","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":15,"pool_name":"cephfs_data","create_time":"2021-02-10 11:26:46.587944","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"be691b4d-a039-408a-af80-9d53f02b6f1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6818","nonce":36088},{"type":"v1","addr":"172.21.2.102:6820","nonce":36088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6822","nonce":36088},{"type":"v1","addr":"172.21.2.102:6824","nonce":36088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6830","nonce":36088},{"type":"v1","addr":"172.21.2.102:6832","nonce":36088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6826","nonce":36088},{"type":"v1","addr":"172.21.2.102:6828","nonce":36088}]},"public_addr":"172.21.2.102:6820/36088","cluster_addr":"172.21.2.102:6824/36088","heartbeat_back_addr":"172.21.2.102:6832/36088","heartbeat_front_addr":"172.21.2.102:6828/36088","state":["exists","up"]},{"osd":1,"uuid":"64e3ccc8-9c69-4d7c-865e-af8e83822635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6819","nonce":36087},{"type":"v1","addr":"172.21.2.102:6821","nonce":36087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6823","nonce":36087},{"type":"v1","addr":"172.21.2.102:6825","nonce":36087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6831","nonce":36087},{"type":"v1","addr":"172.21.2.102:6833","nonce":36087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6827","nonce":36087},{"type":"v1","addr":"172.21.2.102:6829","nonce":36087}]},"public_addr":"172.21.2.102:6821/36087","cluster_addr":"172.21.2.102:6825/36087","heartbeat_back_addr":"172.21.2.102:6833/36087","heartbeat_front_addr":"172.21.2.102:6829/36087","state":["exists","up"]},{"osd":2,"uuid":"153700af-747f-481f-bdf4-d6072d3a842f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6810","nonce":36089},{"type":"v1","addr":"172.21.2.102:6811","nonce":36089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6812","nonce":36089},{"type":"v1","addr":"172.21.2.102:6813","nonce":36089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6816","nonce":36089},{"type":"v1","addr":"172.21.2.102:6817","nonce":36089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6814","nonce":36089},{"type":"v1","addr":"172.21.2.102:6815","nonce":36089}]},"public_addr":"172.21.2.102:6811/36089","cluster_addr":"172.21.2.102:6813/36089","heartbeat_back_addr":"172.21.2.102:6817/36089","heartbeat_front_addr":"172.21.2.102:6815/36089","state":["exists","up"]},{"osd":3,"uuid":"cbc678f4-83a9-45e6-8830-2f7c232b1ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6802","nonce":36090},{"type":"v1","addr":"172.21.2.102:6803","nonce":36090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6804","nonce":36090},{"type":"v1","addr":"172.21.2.102:6805","nonce":36090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6808","nonce":36090},{"type":"v1","addr":"172.21.2.102:6809","nonce":36090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6806","nonce":36090},{"type":"v1","addr":"172.21.2.102:6807","nonce":36090}]},"public_addr":"172.21.2.102:6803/36090","cluster_addr":"172.21.2.102:6805/36090","heartbeat_back_addr":"172.21.2.102:6809/36090","heartbeat_front_addr":"172.21.2.102:6807/36090","state":["exists","up"]},{"osd":4,"uuid":"c01e4008-b258-4f10-afd2-9e7f37214351","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6808","nonce":36368},{"type":"v1","addr":"172.21.2.113:6811","nonce":36368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6814","nonce":36368},{"type":"v1","addr":"172.21.2.113:6818","nonce":36368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6826","nonce":36368},{"type":"v1","addr":"172.21.2.113:6828","nonce":36368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6820","nonce":36368},{"type":"v1","addr":"172.21.2.113:6824","nonce":36368}]},"public_addr":"172.21.2.113:6811/36368","cluster_addr":"172.21.2.113:6818/36368","heartbeat_back_addr":"172.21.2.113:6828/36368","heartbeat_front_addr":"172.21.2.113:6824/36368","state":["exists","up"]},{"osd":5,"uuid":"551f6007-daf8-4a98-8ee0-c2b16bcccb5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6802","nonce":36366},{"type":"v1","addr":"172.21.2.113:6805","nonce":36366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6807","nonce":36366},{"type":"v1","addr":"172.21.2.113:6810","nonce":36366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6819","nonce":36366},{"type":"v1","addr":"172.21.2.113:6822","nonce":36366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6812","nonce":36366},{"type":"v1","addr":"172.21.2.113:6815","nonce":36366}]},"public_addr":"172.21.2.113:6805/36366","cluster_addr":"172.21.2.113:6810/36366","heartbeat_back_addr":"172.21.2.113:6822/36366","heartbeat_front_addr":"172.21.2.113:6815/36366","state":["exists","up"]},{"osd":6,"uuid":"bf67c040-0e50-48be-b102-c3943bc9ebef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6800","nonce":36367},{"type":"v1","addr":"172.21.2.113:6801","nonce":36367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6803","nonce":36367},{"type":"v1","addr":"172.21.2.113:6804","nonce":36367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6813","nonce":36367},{"type":"v1","addr":"172.21.2.113:6816","nonce":36367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6806","nonce":36367},{"type":"v1","addr":"172.21.2.113:6809","nonce":36367}]},"public_addr":"172.21.2.113:6801/36367","cluster_addr":"172.21.2.113:6804/36367","heartbeat_back_addr":"172.21.2.113:6816/36367","heartbeat_front_addr":"172.21.2.113:6809/36367","state":["exists","up"]},{"osd":7,"uuid":"f5461f80-018d-428f-be89-2eb2bcbe7bb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6817","nonce":36365},{"type":"v1","addr":"172.21.2.113:6821","nonce":36365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6823","nonce":36365},{"type":"v1","addr":"172.21.2.113:6825","nonce":36365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6830","nonce":36365},{"type":"v1","addr":"172.21.2.113:6831","nonce":36365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6827","nonce":36365},{"type":"v1","addr":"172.21.2.113:6829","nonce":36365}]},"public_addr":"172.21.2.113:6821/36365","cluster_addr":"172.21.2.113:6825/36365","heartbeat_back_addr":"172.21.2.113:6831/36365","heartbeat_front_addr":"172.21.2.113:6829/36365","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:26:50.785 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:50.785 7f4cab0cb700 1 -- 172.21.2.102:0/2064200634 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4c880186c0 msgr2=0x7f4c8801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:50.785 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:50.785 7f4cab0cb700 1 --2- 172.21.2.102:0/2064200634 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4c880186c0 0x7f4c8801abc0 crc :-1 s=READY pgs=753 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:50.785 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:50.785 7f4cab0cb700 1 -- 172.21.2.102:0/2064200634 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4ca4099230 msgr2=0x7f4ca40b18c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:50.786 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:50.785 7f4cab0cb700 1 --2- 172.21.2.102:0/2064200634 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4ca4099230 0x7f4ca40b18c0 secure :-1 s=READY pgs=1655 cs=0 l=1 rev1=1 rx=0x7f4c9c00b9b0 tx=0x7f4c9c00bd50).stop 2021-02-10T11:26:50.786 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:50.786 7f4cab0cb700 1 -- 172.21.2.102:0/2064200634 shutdown_connections 2021-02-10T11:26:50.786 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:50.786 7f4cab0cb700 1 --2- 172.21.2.102:0/2064200634 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4ca4099230 0x7f4ca40b18c0 unknown :-1 s=CLOSED pgs=1655 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:50.786 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:50.786 7f4cab0cb700 1 --2- 172.21.2.102:0/2064200634 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4c880186c0 0x7f4c8801abc0 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:50.787 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:50.786 7f4cab0cb700 1 --2- 172.21.2.102:0/2064200634 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4ca4098810 0x7f4ca4098c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:50.787 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:50.786 7f4cab0cb700 1 --2- 172.21.2.102:0/2064200634 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4ca40970b0 0x7f4ca40ebe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:50.787 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:50.786 7f4cab0cb700 1 -- 172.21.2.102:0/2064200634 >> 172.21.2.102:0/2064200634 conn(0x7f4ca419a730 msgr2=0x7f4ca419e480 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:26:50.787 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:50.786 7f4cab0cb700 1 -- 172.21.2.102:0/2064200634 shutdown_connections 2021-02-10T11:26:50.787 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:50.786 7f4cab0cb700 1 -- 172.21.2.102:0/2064200634 wait complete. 2021-02-10T11:26:50.795 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:26:51.234 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:51.217 7fa4899a4700 1 Processor -- start 2021-02-10T11:26:51.234 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:51.218 7fa4899a4700 1 -- start start 2021-02-10T11:26:51.234 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:51.218 7fa4899a4700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fa4840d1880 0x7fa4841b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:51.234 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:51.218 7fa4899a4700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fa4841b0af0 0x7fa484108950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:51.234 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:51.218 7fa4899a4700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4840fe1d0 con 0x7fa4840d1c80 2021-02-10T11:26:51.234 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:51.218 7fa4899a4700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa484109a70 con 0x7fa4840d1880 2021-02-10T11:26:51.234 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:51.218 7fa4899a4700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa484109760 con 0x7fa4841b0af0 2021-02-10T11:26:51.235 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:51.218 7fa483574700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fa4840d1880 0x7fa4841b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:51.235 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:51.218 7fa483574700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fa4840d1880 0x7fa4841b04d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:59028/0 (socket says 172.21.2.102:59028) 2021-02-10T11:26:51.235 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:51.218 7fa483574700 1 -- 172.21.2.102:0/3675300764 learned_addr learned my addr 172.21.2.102:0/3675300764 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:26:51.235 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:51.219 7fa483574700 1 -- 172.21.2.102:0/3675300764 >> v2:172.21.2.113:3300/0 conn(0x7fa4841b0af0 msgr2=0x7fa484108950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:26:51.235 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:51.219 7fa483574700 1 --2- 172.21.2.102:0/3675300764 >> v2:172.21.2.113:3300/0 conn(0x7fa4841b0af0 0x7fa484108950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:51.235 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:51.219 7fa483574700 1 -- 172.21.2.102:0/3675300764 >> v1:172.21.2.113:6789/0 conn(0x7fa4840d1c80 legacy=0x7fa484108f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:26:51.235 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:51.219 7fa483574700 1 -- 172.21.2.102:0/3675300764 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa484088070 con 0x7fa4840d1880 2021-02-10T11:26:51.235 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:51.219 7fa483574700 1 --2- 172.21.2.102:0/3675300764 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa4840d1880 0x7fa4841b04d0 secure :-1 s=READY pgs=1656 cs=0 l=1 rev1=1 rx=0x7fa46c006700 tx=0x7fa46c009b00).ready entity=mon.0 client_cookie=c0c3ac5d59c6c019 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:51.235 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:51.219 7fa482572700 1 -- 172.21.2.102:0/3675300764 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa46c00c0c0 con 0x7fa4840d1880 2021-02-10T11:26:51.235 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:51.219 7fa482572700 1 -- 172.21.2.102:0/3675300764 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa46c00c400 con 0x7fa4840d1880 2021-02-10T11:26:51.235 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:51.219 7fa482572700 1 -- 172.21.2.102:0/3675300764 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa46c00c920 con 0x7fa4840d1880 2021-02-10T11:26:51.235 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:51.219 7fa4899a4700 1 -- 172.21.2.102:0/3675300764 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa4840d1880 msgr2=0x7fa4841b04d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:51.235 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:51.219 7fa4899a4700 1 --2- 172.21.2.102:0/3675300764 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa4840d1880 0x7fa4841b04d0 secure :-1 s=READY pgs=1656 cs=0 l=1 rev1=1 rx=0x7fa46c006700 tx=0x7fa46c009b00).stop 2021-02-10T11:26:51.236 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:51.220 7fa4899a4700 1 -- 172.21.2.102:0/3675300764 shutdown_connections 2021-02-10T11:26:51.236 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:51.220 7fa4899a4700 1 --2- 172.21.2.102:0/3675300764 >> v2:172.21.2.113:3300/0 conn(0x7fa4841b0af0 0x7fa484108950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:51.236 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:51.220 7fa4899a4700 1 --2- 172.21.2.102:0/3675300764 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa4840d1880 0x7fa4841b04d0 unknown :-1 s=CLOSED pgs=1656 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:51.236 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:51.220 7fa4899a4700 1 -- 172.21.2.102:0/3675300764 >> 172.21.2.102:0/3675300764 conn(0x7fa4840ce430 msgr2=0x7fa4840d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:26:51.236 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:51.220 7fa4899a4700 1 -- 172.21.2.102:0/3675300764 shutdown_connections 2021-02-10T11:26:51.236 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:51.220 7fa4899a4700 1 -- 172.21.2.102:0/3675300764 wait complete. 2021-02-10T11:26:51.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:51.234 7fa4899a4700 1 Processor -- start 2021-02-10T11:26:51.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:51.235 7fa4899a4700 1 -- start start 2021-02-10T11:26:51.238 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:51.235 7fa4899a4700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa484086810 0x7fa48419fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:51.238 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:51.235 7fa4899a4700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa4840b04d0 0x7fa484097450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:51.238 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:51.235 7fa4899a4700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa4840b08d0 0x7fa484097a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:51.238 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:51.235 7fa4899a4700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4841a02c0 con 0x7fa4840b04d0 2021-02-10T11:26:51.238 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:51.235 7fa4899a4700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa484098090 con 0x7fa484086810 2021-02-10T11:26:51.238 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:51.235 7fa4899a4700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4840982b0 con 0x7fa4840b08d0 2021-02-10T11:26:51.238 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:51.235 7fa482d73700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa4840b04d0 0x7fa484097450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:51.238 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:51.235 7fa482d73700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa4840b04d0 0x7fa484097450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:59032/0 (socket says 172.21.2.102:59032) 2021-02-10T11:26:51.238 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:51.235 7fa482d73700 1 -- 172.21.2.102:0/1410111907 learned_addr learned my addr 172.21.2.102:0/1410111907 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:26:51.238 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:51.235 7fa482d73700 1 -- 172.21.2.102:0/1410111907 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa4840b08d0 msgr2=0x7fa484097a70 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:26:51.238 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:51.235 7fa482d73700 1 --2- 172.21.2.102:0/1410111907 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa4840b08d0 0x7fa484097a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:51.238 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:51.235 7fa482d73700 1 -- 172.21.2.102:0/1410111907 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa484086810 msgr2=0x7fa48419fca0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:26:51.239 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:51.235 7fa482d73700 1 --2- 172.21.2.102:0/1410111907 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa484086810 0x7fa48419fca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:51.239 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:51.235 7fa482d73700 1 -- 172.21.2.102:0/1410111907 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4840984d0 con 0x7fa4840b04d0 2021-02-10T11:26:51.239 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:51.236 7fa482d73700 1 --2- 172.21.2.102:0/1410111907 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa4840b04d0 0x7fa484097450 secure :-1 s=READY pgs=1657 cs=0 l=1 rev1=1 rx=0x7fa474006f00 tx=0x7fa47400b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:51.239 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:51.236 7fa47bfff700 1 -- 172.21.2.102:0/1410111907 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa474006350 con 0x7fa4840b04d0 2021-02-10T11:26:51.239 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:51.236 7fa47bfff700 1 -- 172.21.2.102:0/1410111907 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa47400bd30 con 0x7fa4840b04d0 2021-02-10T11:26:51.239 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:51.236 7fa4899a4700 1 -- 172.21.2.102:0/1410111907 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4840984d0 con 0x7fa4840b04d0 2021-02-10T11:26:51.239 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:51.236 7fa4899a4700 1 -- 172.21.2.102:0/1410111907 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa484098b80 con 0x7fa4840b04d0 2021-02-10T11:26:51.239 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:51.236 7fa47bfff700 1 -- 172.21.2.102:0/1410111907 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa474002020 con 0x7fa4840b04d0 2021-02-10T11:26:51.239 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:51.236 7fa47bfff700 1 -- 172.21.2.102:0/1410111907 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa474005310 con 0x7fa4840b04d0 2021-02-10T11:26:51.239 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:51.236 7fa47bfff700 1 --2- 172.21.2.102:0/1410111907 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa4680186c0 0x7fa46801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:51.239 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:51.236 7fa47bfff700 1 -- 172.21.2.102:0/1410111907 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fa474032610 con 0x7fa4840b04d0 2021-02-10T11:26:51.239 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:51.237 7fa483574700 1 --2- 172.21.2.102:0/1410111907 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa4680186c0 0x7fa46801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:51.239 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:51.237 7fa483574700 1 --2- 172.21.2.102:0/1410111907 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa4680186c0 0x7fa46801abc0 crc :-1 s=READY pgs=754 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:51.268 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:51.268 7fa4899a4700 1 -- 172.21.2.102:0/1410111907 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4840fe1d0 con 0x7fa4840b04d0 2021-02-10T11:26:51.270 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:51.271 7fa47bfff700 1 -- 172.21.2.102:0/1410111907 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fa474000960 con 0x7fa4840b04d0 2021-02-10T11:26:51.399 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:51.400 7fa4899a4700 1 -- 172.21.2.102:0/1410111907 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7fa4840fe1d0 con 0x7fa4840b04d0 2021-02-10T11:26:51.403 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:51.403 7fa47bfff700 1 -- 172.21.2.102:0/1410111907 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.0 v43) v1 ==== 184+0+0 (secure 0 0 0) 0x7fa47401f4d0 con 0x7fa4840b04d0 2021-02-10T11:26:51.406 INFO:teuthology.orchestra.run.gibba002.stderr:updated caps for client.0 2021-02-10T11:26:51.408 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:51.408 7fa4899a4700 1 -- 172.21.2.102:0/1410111907 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa4680186c0 msgr2=0x7fa46801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:51.408 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:51.408 7fa4899a4700 1 --2- 172.21.2.102:0/1410111907 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa4680186c0 0x7fa46801abc0 crc :-1 s=READY pgs=754 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:51.409 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:51.408 7fa4899a4700 1 -- 172.21.2.102:0/1410111907 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa4840b04d0 msgr2=0x7fa484097450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:51.409 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:51.408 7fa4899a4700 1 --2- 172.21.2.102:0/1410111907 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa4840b04d0 0x7fa484097450 secure :-1 s=READY pgs=1657 cs=0 l=1 rev1=1 rx=0x7fa474006f00 tx=0x7fa47400b940).stop 2021-02-10T11:26:51.409 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:51.408 7fa4899a4700 1 -- 172.21.2.102:0/1410111907 shutdown_connections 2021-02-10T11:26:51.409 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:51.408 7fa4899a4700 1 --2- 172.21.2.102:0/1410111907 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa4840b08d0 0x7fa484097a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:51.409 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:51.408 7fa4899a4700 1 --2- 172.21.2.102:0/1410111907 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa4840b04d0 0x7fa484097450 unknown :-1 s=CLOSED pgs=1657 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:51.409 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:51.408 7fa4899a4700 1 --2- 172.21.2.102:0/1410111907 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa4680186c0 0x7fa46801abc0 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:51.409 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:51.408 7fa4899a4700 1 --2- 172.21.2.102:0/1410111907 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa484086810 0x7fa48419fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:51.409 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:51.408 7fa4899a4700 1 -- 172.21.2.102:0/1410111907 >> 172.21.2.102:0/1410111907 conn(0x7fa48419f880 msgr2=0x7fa48408f890 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:26:51.409 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:51.409 7fa4899a4700 1 -- 172.21.2.102:0/1410111907 shutdown_connections 2021-02-10T11:26:51.409 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:51.409 7fa4899a4700 1 -- 172.21.2.102:0/1410111907 wait complete. 2021-02-10T11:26:51.417 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.1 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:26:51.857 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:51.840 7fd20b750700 1 Processor -- start 2021-02-10T11:26:51.857 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:51.840 7fd20b750700 1 -- start start 2021-02-10T11:26:51.857 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:51.840 7fd20b750700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fd2040d1880 0x7fd2041b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:51.858 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:51.840 7fd20b750700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd2041b0af0 0x7fd204108950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:51.858 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:51.840 7fd20b750700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2040fe1d0 con 0x7fd2040d1c80 2021-02-10T11:26:51.858 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:51.840 7fd20b750700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd204109a70 con 0x7fd2040d1880 2021-02-10T11:26:51.858 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:51.840 7fd20b750700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd204109760 con 0x7fd2041b0af0 2021-02-10T11:26:51.858 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:51.841 7fd209d6d700 1 --1- >> v1:172.21.2.113:6789/0 conn(0x7fd2040d1c80 0x7fd204108f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.113:6789/0 says I am v1:172.21.2.102:58588/0 (socket says 172.21.2.102:58588) 2021-02-10T11:26:51.858 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:51.841 7fd209d6d700 1 -- 172.21.2.102:0/3773490348 learned_addr learned my addr 172.21.2.102:0/3773490348 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:26:51.858 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:51.841 7fd20956c700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fd2040d1880 0x7fd2041b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:51.858 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:51.841 7fd208d6b700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd2041b0af0 0x7fd204108950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:51.858 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:51.842 7fd1f7fff700 1 -- 172.21.2.102:0/3773490348 <== mon.1 v1:172.21.2.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4273893402 0 0) 0x7fd1fc001510 con 0x7fd2040d1c80 2021-02-10T11:26:51.858 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:51.842 7fd1f7fff700 1 -- 172.21.2.102:0/3773490348 --> v1:172.21.2.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd1f0002fe0 con 0x7fd2040d1c80 2021-02-10T11:26:51.858 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:51.842 7fd208d6b700 1 -- 172.21.2.102:0/3773490348 >> v2:172.21.2.113:3300/0 conn(0x7fd2040d1880 msgr2=0x7fd2041b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:26:51.858 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:51.842 7fd208d6b700 1 --2- 172.21.2.102:0/3773490348 >> v2:172.21.2.113:3300/0 conn(0x7fd2040d1880 0x7fd2041b04d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:51.859 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:51.842 7fd208d6b700 1 -- 172.21.2.102:0/3773490348 >> v1:172.21.2.113:6789/0 conn(0x7fd2040d1c80 legacy=0x7fd204108f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:51.859 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:51.842 7fd208d6b700 1 -- 172.21.2.102:0/3773490348 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd204088070 con 0x7fd2041b0af0 2021-02-10T11:26:51.859 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:51.842 7fd20956c700 1 --2- 172.21.2.102:0/3773490348 >> v2:172.21.2.113:3300/0 conn(0x7fd2040d1880 0x7fd2041b04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:26:51.859 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:51.842 7fd208d6b700 1 --2- 172.21.2.102:0/3773490348 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd2041b0af0 0x7fd204108950 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7fd200006650 tx=0x7fd200009a80).ready entity=mon.2 client_cookie=d29be463a6f257c8 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:51.859 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:51.843 7fd1f7fff700 1 -- 172.21.2.102:0/3773490348 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd2000009b0 con 0x7fd2041b0af0 2021-02-10T11:26:51.859 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:51.843 7fd1f7fff700 1 -- 172.21.2.102:0/3773490348 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd20000c0c0 con 0x7fd2041b0af0 2021-02-10T11:26:51.859 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:51.843 7fd1f7fff700 1 -- 172.21.2.102:0/3773490348 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd20000c5e0 con 0x7fd2041b0af0 2021-02-10T11:26:51.859 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:51.843 7fd20b750700 1 -- 172.21.2.102:0/3773490348 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd2041b0af0 msgr2=0x7fd204108950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:51.859 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:51.843 7fd20b750700 1 --2- 172.21.2.102:0/3773490348 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd2041b0af0 0x7fd204108950 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7fd200006650 tx=0x7fd200009a80).stop 2021-02-10T11:26:51.859 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:51.843 7fd20b750700 1 -- 172.21.2.102:0/3773490348 shutdown_connections 2021-02-10T11:26:51.859 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:51.843 7fd20b750700 1 --2- 172.21.2.102:0/3773490348 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd2041b0af0 0x7fd204108950 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:51.859 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:51.843 7fd20b750700 1 --2- 172.21.2.102:0/3773490348 >> v2:172.21.2.113:3300/0 conn(0x7fd2040d1880 0x7fd2041b04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:51.859 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:51.843 7fd20b750700 1 -- 172.21.2.102:0/3773490348 >> 172.21.2.102:0/3773490348 conn(0x7fd2040ce430 msgr2=0x7fd2040d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:26:51.859 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:51.844 7fd20b750700 1 -- 172.21.2.102:0/3773490348 shutdown_connections 2021-02-10T11:26:51.860 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:51.844 7fd20b750700 1 -- 172.21.2.102:0/3773490348 wait complete. 2021-02-10T11:26:51.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:51.858 7fd20b750700 1 Processor -- start 2021-02-10T11:26:51.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:51.858 7fd20b750700 1 -- start start 2021-02-10T11:26:51.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:51.858 7fd20b750700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd2040abd30 0x7fd204082bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:51.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:51.858 7fd20b750700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd2040831d0 0x7fd20408a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:51.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:51.858 7fd20b750700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd2040ac130 0x7fd20408aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:51.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:51.858 7fd20b750700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd204083700 con 0x7fd2040831d0 2021-02-10T11:26:51.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:51.858 7fd20b750700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd20408b0c0 con 0x7fd2040ac130 2021-02-10T11:26:51.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:51.858 7fd20b750700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd20408b2a0 con 0x7fd2040abd30 2021-02-10T11:26:51.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:51.859 7fd208d6b700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd2040831d0 0x7fd20408a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:51.862 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:51.859 7fd208d6b700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd2040831d0 0x7fd20408a480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:59046/0 (socket says 172.21.2.102:59046) 2021-02-10T11:26:51.862 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:51.859 7fd208d6b700 1 -- 172.21.2.102:0/2955097260 learned_addr learned my addr 172.21.2.102:0/2955097260 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:26:51.862 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:51.859 7fd208d6b700 1 -- 172.21.2.102:0/2955097260 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd2040abd30 msgr2=0x7fd204082bb0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:26:51.862 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:51.859 7fd208d6b700 1 --2- 172.21.2.102:0/2955097260 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd2040abd30 0x7fd204082bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:51.862 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:51.859 7fd208d6b700 1 -- 172.21.2.102:0/2955097260 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd2040ac130 msgr2=0x7fd20408aaa0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:26:51.862 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:51.859 7fd208d6b700 1 --2- 172.21.2.102:0/2955097260 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd2040ac130 0x7fd20408aaa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:51.862 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:51.859 7fd208d6b700 1 -- 172.21.2.102:0/2955097260 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd20408b5f0 con 0x7fd2040831d0 2021-02-10T11:26:51.862 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:51.859 7fd208d6b700 1 --2- 172.21.2.102:0/2955097260 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd2040831d0 0x7fd20408a480 secure :-1 s=READY pgs=1658 cs=0 l=1 rev1=1 rx=0x7fd200009630 tx=0x7fd20000b900).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:51.862 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:51.859 7fd1f5ffb700 1 -- 172.21.2.102:0/2955097260 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd2000093a0 con 0x7fd2040831d0 2021-02-10T11:26:51.862 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:51.859 7fd1f5ffb700 1 -- 172.21.2.102:0/2955097260 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd20000bcf0 con 0x7fd2040831d0 2021-02-10T11:26:51.863 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:51.859 7fd1f5ffb700 1 -- 172.21.2.102:0/2955097260 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd200005560 con 0x7fd2040831d0 2021-02-10T11:26:51.863 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:51.859 7fd20b750700 1 -- 172.21.2.102:0/2955097260 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd20408b5f0 con 0x7fd2040831d0 2021-02-10T11:26:51.863 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:51.859 7fd20b750700 1 -- 172.21.2.102:0/2955097260 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd20408bb50 con 0x7fd2040831d0 2021-02-10T11:26:51.863 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:51.860 7fd1f5ffb700 1 -- 172.21.2.102:0/2955097260 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd20000a000 con 0x7fd2040831d0 2021-02-10T11:26:51.863 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:51.860 7fd1f5ffb700 1 --2- 172.21.2.102:0/2955097260 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd1e40186c0 0x7fd1e401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:51.863 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:51.860 7fd1f5ffb700 1 -- 172.21.2.102:0/2955097260 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fd200032540 con 0x7fd2040831d0 2021-02-10T11:26:51.863 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:51.860 7fd20956c700 1 --2- 172.21.2.102:0/2955097260 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd1e40186c0 0x7fd1e401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:51.863 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:51.860 7fd20956c700 1 --2- 172.21.2.102:0/2955097260 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd1e40186c0 0x7fd1e401abc0 crc :-1 s=READY pgs=755 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:51.890 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:51.890 7fd20b750700 1 -- 172.21.2.102:0/2955097260 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd20408b2a0 con 0x7fd2040831d0 2021-02-10T11:26:51.892 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:51.893 7fd1f5ffb700 1 -- 172.21.2.102:0/2955097260 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd200029040 con 0x7fd2040831d0 2021-02-10T11:26:52.024 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:52.024 7fd20b750700 1 -- 172.21.2.102:0/2955097260 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7fd20408b2a0 con 0x7fd2040831d0 2021-02-10T11:26:52.028 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:52.028 7fd1f5ffb700 1 -- 172.21.2.102:0/2955097260 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.1 v44) v1 ==== 184+0+0 (secure 0 0 0) 0x7fd20000a000 con 0x7fd2040831d0 2021-02-10T11:26:52.031 INFO:teuthology.orchestra.run.gibba002.stderr:updated caps for client.1 2021-02-10T11:26:52.032 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:52.033 7fd20b750700 1 -- 172.21.2.102:0/2955097260 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd1e40186c0 msgr2=0x7fd1e401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:52.032 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:52.033 7fd20b750700 1 --2- 172.21.2.102:0/2955097260 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd1e40186c0 0x7fd1e401abc0 crc :-1 s=READY pgs=755 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:52.033 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:52.033 7fd20b750700 1 -- 172.21.2.102:0/2955097260 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd2040831d0 msgr2=0x7fd20408a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:52.033 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:52.033 7fd20b750700 1 --2- 172.21.2.102:0/2955097260 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd2040831d0 0x7fd20408a480 secure :-1 s=READY pgs=1658 cs=0 l=1 rev1=1 rx=0x7fd200009630 tx=0x7fd20000b900).stop 2021-02-10T11:26:52.033 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:52.033 7fd20b750700 1 -- 172.21.2.102:0/2955097260 shutdown_connections 2021-02-10T11:26:52.033 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:52.033 7fd20b750700 1 --2- 172.21.2.102:0/2955097260 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd1e40186c0 0x7fd1e401abc0 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:52.033 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:52.033 7fd20b750700 1 --2- 172.21.2.102:0/2955097260 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd2040ac130 0x7fd20408aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:52.033 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:52.033 7fd20b750700 1 --2- 172.21.2.102:0/2955097260 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd2040831d0 0x7fd20408a480 unknown :-1 s=CLOSED pgs=1658 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:52.033 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:52.033 7fd20b750700 1 --2- 172.21.2.102:0/2955097260 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd2040abd30 0x7fd204082bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:52.034 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:52.033 7fd20b750700 1 -- 172.21.2.102:0/2955097260 >> 172.21.2.102:0/2955097260 conn(0x7fd2040afc60 msgr2=0x7fd2040b50d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:26:52.034 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:52.033 7fd20b750700 1 -- 172.21.2.102:0/2955097260 shutdown_connections 2021-02-10T11:26:52.034 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:52.033 7fd20b750700 1 -- 172.21.2.102:0/2955097260 wait complete. 2021-02-10T11:26:52.041 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.2 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:26:52.473 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:52.455 7f8a76d54700 1 Processor -- start 2021-02-10T11:26:52.474 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:52.455 7f8a76d54700 1 -- start start 2021-02-10T11:26:52.474 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:52.455 7f8a76d54700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f8a700d1c60 0x7f8a700cef50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:52.474 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:52.455 7f8a76d54700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a700d0160 con 0x7f8a700d1860 2021-02-10T11:26:52.474 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:52.455 7f8a76d54700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a700fe1b0 con 0x7f8a700cf570 2021-02-10T11:26:52.474 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:52.455 7f8a76d54700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a701b0ca0 con 0x7f8a700d1c60 2021-02-10T11:26:52.474 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:52.455 7f8a74b70700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f8a700d1860 0x7f8a701b04b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:51276/0 (socket says 172.21.2.102:51276) 2021-02-10T11:26:52.474 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:52.455 7f8a74b70700 1 -- 172.21.2.102:0/1917075466 learned_addr learned my addr 172.21.2.102:0/1917075466 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:26:52.474 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:52.456 7f8a6f7fe700 1 -- 172.21.2.102:0/1917075466 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3325456228 0 0) 0x7f8a58001180 con 0x7f8a700d1860 2021-02-10T11:26:52.474 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:52.456 7f8a6f7fe700 1 -- 172.21.2.102:0/1917075466 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a5c002fe0 con 0x7f8a700d1860 2021-02-10T11:26:52.474 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:52.456 7f8a6ffff700 1 --2- 172.21.2.102:0/1917075466 >> v2:172.21.2.113:3300/0 conn(0x7f8a700d1c60 0x7f8a700cef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:52.474 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:52.456 7f8a6f7fe700 1 -- 172.21.2.102:0/1917075466 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2665528629 0 0) 0x7f8a58001180 con 0x7f8a700d1860 2021-02-10T11:26:52.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:52.456 7f8a6f7fe700 1 -- 172.21.2.102:0/1917075466 >> v2:172.21.2.113:3300/0 conn(0x7f8a700d1c60 msgr2=0x7f8a700cef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:26:52.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:52.456 7f8a6f7fe700 1 --2- 172.21.2.102:0/1917075466 >> v2:172.21.2.113:3300/0 conn(0x7f8a700d1c60 0x7f8a700cef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:52.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:52.456 7f8a6f7fe700 1 -- 172.21.2.102:0/1917075466 >> v1:172.21.2.113:6789/0 conn(0x7f8a700cf570 legacy=0x7f8a700cf970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:26:52.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:52.456 7f8a6f7fe700 1 -- 172.21.2.102:0/1917075466 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a70109a50 con 0x7f8a700d1860 2021-02-10T11:26:52.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:52.456 7f8a6f7fe700 1 -- 172.21.2.102:0/1917075466 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7f8a58001680 con 0x7f8a700d1860 2021-02-10T11:26:52.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:52.456 7f8a6f7fe700 1 -- 172.21.2.102:0/1917075466 >> v1:172.21.2.102:6789/0 conn(0x7f8a700d1860 legacy=0x7f8a701b04b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:52.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:52.456 7f8a6f7fe700 1 --2- 172.21.2.102:0/1917075466 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8a5c003aa0 0x7f8a5c005ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:52.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:52.456 7f8a6f7fe700 1 --2- 172.21.2.102:0/1917075466 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8a5c006790 0x7f8a5c008c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:52.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:52.456 7f8a6f7fe700 1 --2- 172.21.2.102:0/1917075466 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8a5c009590 0x7f8a5c00bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:52.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:52.456 7f8a6f7fe700 1 -- 172.21.2.102:0/1917075466 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a5c00c300 con 0x7f8a5c009590 2021-02-10T11:26:52.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:52.456 7f8a6f7fe700 1 -- 172.21.2.102:0/1917075466 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a5c00c520 con 0x7f8a5c006790 2021-02-10T11:26:52.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:52.456 7f8a6f7fe700 1 -- 172.21.2.102:0/1917075466 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a5c00c740 con 0x7f8a5c003aa0 2021-02-10T11:26:52.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:52.457 7f8a75371700 1 --2- 172.21.2.102:0/1917075466 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8a5c009590 0x7f8a5c00bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:52.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:52.457 7f8a6ffff700 1 --2- 172.21.2.102:0/1917075466 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8a5c006790 0x7f8a5c008c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:52.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:52.457 7f8a74b70700 1 --2- 172.21.2.102:0/1917075466 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8a5c003aa0 0x7f8a5c005ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:52.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:52.457 7f8a75371700 1 -- 172.21.2.102:0/1917075466 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8a5c003aa0 msgr2=0x7f8a5c005ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:26:52.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:52.457 7f8a75371700 1 --2- 172.21.2.102:0/1917075466 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8a5c003aa0 0x7f8a5c005ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:52.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:52.457 7f8a75371700 1 -- 172.21.2.102:0/1917075466 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8a5c006790 msgr2=0x7f8a5c008c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:26:52.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:52.457 7f8a75371700 1 --2- 172.21.2.102:0/1917075466 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8a5c006790 0x7f8a5c008c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:52.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:52.457 7f8a75371700 1 -- 172.21.2.102:0/1917075466 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a5c00c9e0 con 0x7f8a5c009590 2021-02-10T11:26:52.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:52.457 7f8a75371700 1 -- 172.21.2.102:0/1917075466 reap_dead start 2021-02-10T11:26:52.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:52.457 7f8a75371700 1 --2- 172.21.2.102:0/1917075466 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8a5c009590 0x7f8a5c00bab0 secure :-1 s=READY pgs=1660 cs=0 l=1 rev1=1 rx=0x7f8a68006e30 tx=0x7f8a6804de60).ready entity=mon.0 client_cookie=ac02228d504f2308 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:52.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:52.457 7f8a6f7fe700 1 -- 172.21.2.102:0/1917075466 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8a68003020 con 0x7f8a5c009590 2021-02-10T11:26:52.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:52.457 7f8a6f7fe700 1 -- 172.21.2.102:0/1917075466 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a68003220 con 0x7f8a5c009590 2021-02-10T11:26:52.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:52.458 7f8a6f7fe700 1 -- 172.21.2.102:0/1917075466 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8a680015f0 con 0x7f8a5c009590 2021-02-10T11:26:52.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:52.458 7f8a76d54700 1 -- 172.21.2.102:0/1917075466 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8a5c009590 msgr2=0x7f8a5c00bab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:52.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:52.458 7f8a76d54700 1 --2- 172.21.2.102:0/1917075466 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8a5c009590 0x7f8a5c00bab0 secure :-1 s=READY pgs=1660 cs=0 l=1 rev1=1 rx=0x7f8a68006e30 tx=0x7f8a6804de60).stop 2021-02-10T11:26:52.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:52.458 7f8a76d54700 1 -- 172.21.2.102:0/1917075466 shutdown_connections 2021-02-10T11:26:52.477 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:52.458 7f8a76d54700 1 --2- 172.21.2.102:0/1917075466 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8a5c009590 0x7f8a5c00bab0 unknown :-1 s=CLOSED pgs=1660 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:52.477 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:52.458 7f8a76d54700 1 -- 172.21.2.102:0/1917075466 >> 172.21.2.102:0/1917075466 conn(0x7f8a700ce410 msgr2=0x7f8a700d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:26:52.477 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:52.458 7f8a76d54700 1 -- 172.21.2.102:0/1917075466 shutdown_connections 2021-02-10T11:26:52.477 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:52.458 7f8a76d54700 1 -- 172.21.2.102:0/1917075466 wait complete. 2021-02-10T11:26:52.477 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:52.473 7f8a76d54700 1 Processor -- start 2021-02-10T11:26:52.477 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:52.473 7f8a76d54700 1 -- start start 2021-02-10T11:26:52.477 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:52.474 7f8a76d54700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8a700b5fb0 0x7f8a700a81f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:52.477 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:52.474 7f8a76d54700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8a700a8810 0x7f8a700a2540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:52.477 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:52.474 7f8a76d54700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8a700a8c10 0x7f8a700a2b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:52.477 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:52.474 7f8a76d54700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a700b63b0 con 0x7f8a700b5fb0 2021-02-10T11:26:52.477 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:52.474 7f8a76d54700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a700a9210 con 0x7f8a700a8810 2021-02-10T11:26:52.477 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:52.474 7f8a76d54700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a700b2650 con 0x7f8a700a8c10 2021-02-10T11:26:52.477 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:52.474 7f8a74b70700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8a700b5fb0 0x7f8a700a81f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:52.477 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:52.474 7f8a74b70700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8a700b5fb0 0x7f8a700a81f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:59064/0 (socket says 172.21.2.102:59064) 2021-02-10T11:26:52.478 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:52.474 7f8a74b70700 1 -- 172.21.2.102:0/1916215775 learned_addr learned my addr 172.21.2.102:0/1916215775 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:26:52.478 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:52.474 7f8a74b70700 1 -- 172.21.2.102:0/1916215775 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8a700a8c10 msgr2=0x7f8a700a2b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:52.479 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:52.474 7f8a74b70700 1 --2- 172.21.2.102:0/1916215775 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8a700a8c10 0x7f8a700a2b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:52.479 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:52.474 7f8a74b70700 1 -- 172.21.2.102:0/1916215775 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8a700a8810 msgr2=0x7f8a700a2540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:52.479 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:52.474 7f8a74b70700 1 --2- 172.21.2.102:0/1916215775 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8a700a8810 0x7f8a700a2540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:52.479 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:52.474 7f8a74b70700 1 -- 172.21.2.102:0/1916215775 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a700a3180 con 0x7f8a700b5fb0 2021-02-10T11:26:52.479 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:52.474 7f8a74b70700 1 --2- 172.21.2.102:0/1916215775 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8a700b5fb0 0x7f8a700a81f0 secure :-1 s=READY pgs=1661 cs=0 l=1 rev1=1 rx=0x7f8a58000b80 tx=0x7f8a5800a970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:52.479 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:52.475 7f8a6d7fa700 1 -- 172.21.2.102:0/1916215775 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8a58001c20 con 0x7f8a700b5fb0 2021-02-10T11:26:52.479 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:52.475 7f8a6d7fa700 1 -- 172.21.2.102:0/1916215775 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a5800ad60 con 0x7f8a700b5fb0 2021-02-10T11:26:52.479 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:52.475 7f8a6d7fa700 1 -- 172.21.2.102:0/1916215775 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8a580043a0 con 0x7f8a700b5fb0 2021-02-10T11:26:52.479 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:52.475 7f8a76d54700 1 -- 172.21.2.102:0/1916215775 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a700a3180 con 0x7f8a700b5fb0 2021-02-10T11:26:52.479 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:52.475 7f8a76d54700 1 -- 172.21.2.102:0/1916215775 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a700a3710 con 0x7f8a700b5fb0 2021-02-10T11:26:52.479 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:52.475 7f8a6d7fa700 1 -- 172.21.2.102:0/1916215775 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f8a580043a0 con 0x7f8a700b5fb0 2021-02-10T11:26:52.479 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:52.475 7f8a6d7fa700 1 --2- 172.21.2.102:0/1916215775 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8a540186c0 0x7f8a5401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:52.479 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:52.475 7f8a6d7fa700 1 -- 172.21.2.102:0/1916215775 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f8a580323c0 con 0x7f8a700b5fb0 2021-02-10T11:26:52.480 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:52.475 7f8a6ffff700 1 --2- 172.21.2.102:0/1916215775 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8a540186c0 0x7f8a5401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:52.480 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:52.476 7f8a6ffff700 1 --2- 172.21.2.102:0/1916215775 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8a540186c0 0x7f8a5401abc0 crc :-1 s=READY pgs=756 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:52.505 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:52.505 7f8a76d54700 1 -- 172.21.2.102:0/1916215775 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a700fe1b0 con 0x7f8a700b5fb0 2021-02-10T11:26:52.507 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:52.508 7f8a6d7fa700 1 -- 172.21.2.102:0/1916215775 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f8a58001c20 con 0x7f8a700b5fb0 2021-02-10T11:26:52.638 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:52.638 7f8a76d54700 1 -- 172.21.2.102:0/1916215775 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f8a700fe1b0 con 0x7f8a700b5fb0 2021-02-10T11:26:52.641 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:52.642 7f8a6d7fa700 1 -- 172.21.2.102:0/1916215775 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.2 v45) v1 ==== 184+0+0 (secure 0 0 0) 0x7f8a58004730 con 0x7f8a700b5fb0 2021-02-10T11:26:52.645 INFO:teuthology.orchestra.run.gibba002.stderr:updated caps for client.2 2021-02-10T11:26:52.646 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:52.647 7f8a76d54700 1 -- 172.21.2.102:0/1916215775 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8a540186c0 msgr2=0x7f8a5401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:52.647 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:52.647 7f8a76d54700 1 --2- 172.21.2.102:0/1916215775 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8a540186c0 0x7f8a5401abc0 crc :-1 s=READY pgs=756 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:52.647 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:52.647 7f8a76d54700 1 -- 172.21.2.102:0/1916215775 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8a700b5fb0 msgr2=0x7f8a700a81f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:52.647 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:52.647 7f8a76d54700 1 --2- 172.21.2.102:0/1916215775 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8a700b5fb0 0x7f8a700a81f0 secure :-1 s=READY pgs=1661 cs=0 l=1 rev1=1 rx=0x7f8a58000b80 tx=0x7f8a5800a970).stop 2021-02-10T11:26:52.647 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:52.647 7f8a76d54700 1 -- 172.21.2.102:0/1916215775 shutdown_connections 2021-02-10T11:26:52.647 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:52.647 7f8a76d54700 1 --2- 172.21.2.102:0/1916215775 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8a700a8c10 0x7f8a700a2b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:52.647 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:52.647 7f8a76d54700 1 --2- 172.21.2.102:0/1916215775 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8a540186c0 0x7f8a5401abc0 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:52.648 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:52.647 7f8a76d54700 1 --2- 172.21.2.102:0/1916215775 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8a700a8810 0x7f8a700a2540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:52.648 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:52.647 7f8a76d54700 1 --2- 172.21.2.102:0/1916215775 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8a700b5fb0 0x7f8a700a81f0 unknown :-1 s=CLOSED pgs=1661 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:52.648 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:52.647 7f8a76d54700 1 -- 172.21.2.102:0/1916215775 >> 172.21.2.102:0/1916215775 conn(0x7f8a7009e660 msgr2=0x7f8a700a9550 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:26:52.648 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:52.647 7f8a76d54700 1 -- 172.21.2.102:0/1916215775 shutdown_connections 2021-02-10T11:26:52.648 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:52.647 7f8a76d54700 1 -- 172.21.2.102:0/1916215775 wait complete. 2021-02-10T11:26:52.655 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.3 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:26:53.086 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:53.069 7f926b73c700 1 Processor -- start 2021-02-10T11:26:53.086 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:53.070 7f926b73c700 1 -- start start 2021-02-10T11:26:53.086 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:53.070 7f926b73c700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f92640d1860 0x7f92641b04b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:53.086 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:53.070 7f926b73c700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f92641b0ad0 0x7f9264108930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:53.086 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:53.070 7f926b73c700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92640fe1b0 con 0x7f92640d1c60 2021-02-10T11:26:53.087 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:53.070 7f926b73c700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9264109a50 con 0x7f92640d1860 2021-02-10T11:26:53.087 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:53.070 7f926b73c700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9264109740 con 0x7f92641b0ad0 2021-02-10T11:26:53.087 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:53.070 7f9269558700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f92640d1860 0x7f92641b04b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:53.087 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:53.070 7f9269558700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f92640d1860 0x7f92641b04b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:59072/0 (socket says 172.21.2.102:59072) 2021-02-10T11:26:53.087 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:53.070 7f9269558700 1 -- 172.21.2.102:0/3534968529 learned_addr learned my addr 172.21.2.102:0/3534968529 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:26:53.087 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:53.071 7f9269558700 1 -- 172.21.2.102:0/3534968529 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f92641b0ad0 msgr2=0x7f9264108930 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:26:53.087 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:53.071 7f9269558700 1 --2- 172.21.2.102:0/3534968529 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f92641b0ad0 0x7f9264108930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:53.087 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:53.071 7f9269558700 1 -- 172.21.2.102:0/3534968529 >> v1:172.21.2.113:6789/0 conn(0x7f92640d1c60 legacy=0x7f9264108f50 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:26:53.087 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:53.071 7f9269558700 1 -- 172.21.2.102:0/3534968529 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9264088070 con 0x7f92640d1860 2021-02-10T11:26:53.087 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:53.071 7f9269558700 1 --2- 172.21.2.102:0/3534968529 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f92640d1860 0x7f92641b04b0 secure :-1 s=READY pgs=1662 cs=0 l=1 rev1=1 rx=0x7f9260006700 tx=0x7f9260009b00).ready entity=mon.0 client_cookie=aaac3456f4926268 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:53.087 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:53.071 7f9257fff700 1 -- 172.21.2.102:0/3534968529 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f926000c0c0 con 0x7f92640d1860 2021-02-10T11:26:53.088 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:53.071 7f9257fff700 1 -- 172.21.2.102:0/3534968529 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f926000c340 con 0x7f92640d1860 2021-02-10T11:26:53.088 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:53.071 7f9257fff700 1 -- 172.21.2.102:0/3534968529 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f926000c820 con 0x7f92640d1860 2021-02-10T11:26:53.088 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:53.071 7f926b73c700 1 -- 172.21.2.102:0/3534968529 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f92640d1860 msgr2=0x7f92641b04b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:53.088 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:53.071 7f926b73c700 1 --2- 172.21.2.102:0/3534968529 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f92640d1860 0x7f92641b04b0 secure :-1 s=READY pgs=1662 cs=0 l=1 rev1=1 rx=0x7f9260006700 tx=0x7f9260009b00).stop 2021-02-10T11:26:53.088 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:53.072 7f926b73c700 1 -- 172.21.2.102:0/3534968529 shutdown_connections 2021-02-10T11:26:53.088 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:53.072 7f926b73c700 1 --2- 172.21.2.102:0/3534968529 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f92641b0ad0 0x7f9264108930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:53.088 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:53.072 7f926b73c700 1 --2- 172.21.2.102:0/3534968529 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f92640d1860 0x7f92641b04b0 unknown :-1 s=CLOSED pgs=1662 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:53.088 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:53.072 7f926b73c700 1 -- 172.21.2.102:0/3534968529 >> 172.21.2.102:0/3534968529 conn(0x7f92640ce410 msgr2=0x7f92640d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:26:53.088 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:53.072 7f926b73c700 1 -- 172.21.2.102:0/3534968529 shutdown_connections 2021-02-10T11:26:53.088 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:53.072 7f926b73c700 1 -- 172.21.2.102:0/3534968529 wait complete. 2021-02-10T11:26:53.090 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:53.087 7f926b73c700 1 Processor -- start 2021-02-10T11:26:53.090 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:53.087 7f926b73c700 1 -- start start 2021-02-10T11:26:53.090 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:53.087 7f926b73c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9264195500 0x7f92640b0410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:53.090 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:53.087 7f926b73c700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f92640b0a30 0x7f926418f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:53.090 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:53.087 7f926b73c700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f926418f990 0x7f9264190680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:53.090 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:53.087 7f926b73c700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9264195a60 con 0x7f9264195500 2021-02-10T11:26:53.090 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:53.087 7f926b73c700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92641a5970 con 0x7f926418f990 2021-02-10T11:26:53.090 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:53.087 7f926b73c700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f926418fd90 con 0x7f92640b0a30 2021-02-10T11:26:53.091 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:53.087 7f9269558700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9264195500 0x7f92640b0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:53.091 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:53.087 7f9269558700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9264195500 0x7f92640b0410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:59080/0 (socket says 172.21.2.102:59080) 2021-02-10T11:26:53.091 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:53.087 7f9269558700 1 -- 172.21.2.102:0/1606950079 learned_addr learned my addr 172.21.2.102:0/1606950079 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:26:53.091 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:53.088 7f9269558700 1 -- 172.21.2.102:0/1606950079 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f92640b0a30 msgr2=0x7f926418f370 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:26:53.091 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:53.088 7f9269558700 1 --2- 172.21.2.102:0/1606950079 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f92640b0a30 0x7f926418f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:53.091 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:53.088 7f9269558700 1 -- 172.21.2.102:0/1606950079 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f926418f990 msgr2=0x7f9264190680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:53.091 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:53.088 7f9269558700 1 --2- 172.21.2.102:0/1606950079 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f926418f990 0x7f9264190680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:53.091 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:53.088 7f9269558700 1 -- 172.21.2.102:0/1606950079 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9264190ca0 con 0x7f9264195500 2021-02-10T11:26:53.091 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:53.088 7f9269558700 1 --2- 172.21.2.102:0/1606950079 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9264195500 0x7f92640b0410 secure :-1 s=READY pgs=1663 cs=0 l=1 rev1=1 rx=0x7f9260006f20 tx=0x7f926000b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:53.091 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:53.088 7f9255ffb700 1 -- 172.21.2.102:0/1606950079 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9260000c90 con 0x7f9264195500 2021-02-10T11:26:53.091 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:53.088 7f9255ffb700 1 -- 172.21.2.102:0/1606950079 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9260009100 con 0x7f9264195500 2021-02-10T11:26:53.092 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:53.088 7f9255ffb700 1 -- 172.21.2.102:0/1606950079 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9260002220 con 0x7f9264195500 2021-02-10T11:26:53.092 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:53.088 7f926b73c700 1 -- 172.21.2.102:0/1606950079 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9264190ca0 con 0x7f9264195500 2021-02-10T11:26:53.092 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:53.088 7f926b73c700 1 -- 172.21.2.102:0/1606950079 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9264190ca0 con 0x7f9264195500 2021-02-10T11:26:53.092 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:53.089 7f9255ffb700 1 -- 172.21.2.102:0/1606950079 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f926000a000 con 0x7f9264195500 2021-02-10T11:26:53.092 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:53.089 7f9255ffb700 1 --2- 172.21.2.102:0/1606950079 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f92440186c0 0x7f924401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:53.092 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:53.089 7f9255ffb700 1 -- 172.21.2.102:0/1606950079 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f9260032780 con 0x7f9264195500 2021-02-10T11:26:53.092 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:53.089 7f9268d57700 1 --2- 172.21.2.102:0/1606950079 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f92440186c0 0x7f924401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:53.092 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:53.089 7f9268d57700 1 --2- 172.21.2.102:0/1606950079 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f92440186c0 0x7f924401abc0 crc :-1 s=READY pgs=757 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:53.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:53.120 7f926b73c700 1 -- 172.21.2.102:0/1606950079 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9264082980 con 0x7f9264195500 2021-02-10T11:26:53.123 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:53.123 7f9255ffb700 1 -- 172.21.2.102:0/1606950079 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f9260000c90 con 0x7f9264195500 2021-02-10T11:26:53.253 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:53.253 7f926b73c700 1 -- 172.21.2.102:0/1606950079 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.3", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f9264082980 con 0x7f9264195500 2021-02-10T11:26:53.257 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:53.257 7f9255ffb700 1 -- 172.21.2.102:0/1606950079 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.3", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.3 v46) v1 ==== 184+0+0 (secure 0 0 0) 0x7f926000a000 con 0x7f9264195500 2021-02-10T11:26:53.260 INFO:teuthology.orchestra.run.gibba002.stderr:updated caps for client.3 2021-02-10T11:26:53.262 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:53.262 7f926b73c700 1 -- 172.21.2.102:0/1606950079 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f92440186c0 msgr2=0x7f924401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:53.262 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:53.262 7f926b73c700 1 --2- 172.21.2.102:0/1606950079 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f92440186c0 0x7f924401abc0 crc :-1 s=READY pgs=757 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:53.263 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:53.262 7f926b73c700 1 -- 172.21.2.102:0/1606950079 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9264195500 msgr2=0x7f92640b0410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:53.263 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:53.262 7f926b73c700 1 --2- 172.21.2.102:0/1606950079 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9264195500 0x7f92640b0410 secure :-1 s=READY pgs=1663 cs=0 l=1 rev1=1 rx=0x7f9260006f20 tx=0x7f926000b990).stop 2021-02-10T11:26:53.263 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:53.262 7f926b73c700 1 -- 172.21.2.102:0/1606950079 shutdown_connections 2021-02-10T11:26:53.263 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:53.262 7f926b73c700 1 --2- 172.21.2.102:0/1606950079 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f92440186c0 0x7f924401abc0 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:53.263 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:53.262 7f926b73c700 1 --2- 172.21.2.102:0/1606950079 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f926418f990 0x7f9264190680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:53.263 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:53.262 7f926b73c700 1 --2- 172.21.2.102:0/1606950079 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f92640b0a30 0x7f926418f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:53.263 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:53.262 7f926b73c700 1 --2- 172.21.2.102:0/1606950079 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9264195500 0x7f92640b0410 unknown :-1 s=CLOSED pgs=1663 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:53.263 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:53.262 7f926b73c700 1 -- 172.21.2.102:0/1606950079 >> 172.21.2.102:0/1606950079 conn(0x7f926419f820 msgr2=0x7f92641a5cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:26:53.263 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:53.263 7f926b73c700 1 -- 172.21.2.102:0/1606950079 shutdown_connections 2021-02-10T11:26:53.263 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:53.263 7f926b73c700 1 -- 172.21.2.102:0/1606950079 wait complete. 2021-02-10T11:26:53.271 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:26:53.699 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:53.682 7efc3ac75700 1 Processor -- start 2021-02-10T11:26:53.699 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:53.682 7efc3ac75700 1 -- start start 2021-02-10T11:26:53.699 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:53.682 7efc3ac75700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7efc340d1c70 0x7efc340f20a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:53.699 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:53.682 7efc3ac75700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc340f0b10 con 0x7efc340d1870 2021-02-10T11:26:53.699 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:53.682 7efc3ac75700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc341b0be0 con 0x7efc340d0340 2021-02-10T11:26:53.699 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:53.682 7efc3ac75700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc340e5ed0 con 0x7efc340d1c70 2021-02-10T11:26:53.699 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:53.682 7efc38a91700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7efc340d1870 0x7efc341b03f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:51310/0 (socket says 172.21.2.102:51310) 2021-02-10T11:26:53.700 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:53.682 7efc38a91700 1 -- 172.21.2.102:0/3121089032 learned_addr learned my addr 172.21.2.102:0/3121089032 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:26:53.700 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:53.683 7efc337fe700 1 -- 172.21.2.102:0/3121089032 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4118571023 0 0) 0x7efc1c0010b0 con 0x7efc340d1870 2021-02-10T11:26:53.700 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:53.683 7efc337fe700 1 -- 172.21.2.102:0/3121089032 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc20002fe0 con 0x7efc340d1870 2021-02-10T11:26:53.700 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:53.683 7efc39292700 1 --2- 172.21.2.102:0/3121089032 >> v2:172.21.2.113:3300/0 conn(0x7efc340d1c70 0x7efc340f20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:53.700 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:53.683 7efc337fe700 1 -- 172.21.2.102:0/3121089032 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1573972307 0 0) 0x7efc1c0010b0 con 0x7efc340d1870 2021-02-10T11:26:53.700 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:53.683 7efc337fe700 1 -- 172.21.2.102:0/3121089032 >> v2:172.21.2.113:3300/0 conn(0x7efc340d1c70 msgr2=0x7efc340f20a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:26:53.700 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:53.683 7efc337fe700 1 --2- 172.21.2.102:0/3121089032 >> v2:172.21.2.113:3300/0 conn(0x7efc340d1c70 0x7efc340f20a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:53.700 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:53.683 7efc337fe700 1 -- 172.21.2.102:0/3121089032 >> v1:172.21.2.113:6789/0 conn(0x7efc340d0340 legacy=0x7efc340ef640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:53.700 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:53.683 7efc337fe700 1 -- 172.21.2.102:0/3121089032 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc340efe30 con 0x7efc340d1870 2021-02-10T11:26:53.700 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:53.683 7efc337fe700 1 -- 172.21.2.102:0/3121089032 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7efc1c0015b0 con 0x7efc340d1870 2021-02-10T11:26:53.700 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:53.683 7efc337fe700 1 -- 172.21.2.102:0/3121089032 >> v1:172.21.2.102:6789/0 conn(0x7efc340d1870 legacy=0x7efc341b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:53.700 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:53.683 7efc337fe700 1 --2- 172.21.2.102:0/3121089032 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efc20003ac0 0x7efc20005f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:53.700 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:53.683 7efc337fe700 1 --2- 172.21.2.102:0/3121089032 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7efc20006780 0x7efc20008ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:53.700 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:53.683 7efc337fe700 1 --2- 172.21.2.102:0/3121089032 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7efc200095a0 0x7efc2000baa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:53.701 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:53.683 7efc337fe700 1 -- 172.21.2.102:0/3121089032 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc2000c2f0 con 0x7efc20003ac0 2021-02-10T11:26:53.701 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:53.683 7efc337fe700 1 -- 172.21.2.102:0/3121089032 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc2000c510 con 0x7efc20006780 2021-02-10T11:26:53.701 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:53.683 7efc337fe700 1 -- 172.21.2.102:0/3121089032 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc2000c730 con 0x7efc200095a0 2021-02-10T11:26:53.701 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:53.683 7efc38a91700 1 --2- 172.21.2.102:0/3121089032 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efc20003ac0 0x7efc20005f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:53.701 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:53.683 7efc33fff700 1 --2- 172.21.2.102:0/3121089032 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7efc20006780 0x7efc20008ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:53.701 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:53.683 7efc39292700 1 --2- 172.21.2.102:0/3121089032 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7efc200095a0 0x7efc2000baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:53.701 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:53.683 7efc38a91700 1 -- 172.21.2.102:0/3121089032 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7efc200095a0 msgr2=0x7efc2000baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:26:53.701 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:53.683 7efc38a91700 1 --2- 172.21.2.102:0/3121089032 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7efc200095a0 0x7efc2000baa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:53.701 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:53.683 7efc38a91700 1 -- 172.21.2.102:0/3121089032 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7efc20006780 msgr2=0x7efc20008ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:26:53.701 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:53.683 7efc38a91700 1 --2- 172.21.2.102:0/3121089032 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7efc20006780 0x7efc20008ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:53.701 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:53.683 7efc38a91700 1 -- 172.21.2.102:0/3121089032 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc200509a0 con 0x7efc20003ac0 2021-02-10T11:26:53.701 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:53.683 7efc39292700 1 --2- 172.21.2.102:0/3121089032 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7efc200095a0 0x7efc2000baa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:26:53.701 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:53.683 7efc39292700 1 -- 172.21.2.102:0/3121089032 reap_dead start 2021-02-10T11:26:53.702 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:53.683 7efc38a91700 1 --2- 172.21.2.102:0/3121089032 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efc20003ac0 0x7efc20005f60 secure :-1 s=READY pgs=1665 cs=0 l=1 rev1=1 rx=0x7efc1c007990 tx=0x7efc1c007d30).ready entity=mon.0 client_cookie=f3e79a5104eddabc server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:53.702 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:53.684 7efc337fe700 1 -- 172.21.2.102:0/3121089032 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efc1c0009b0 con 0x7efc20003ac0 2021-02-10T11:26:53.702 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:53.684 7efc337fe700 1 -- 172.21.2.102:0/3121089032 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc1c00a310 con 0x7efc20003ac0 2021-02-10T11:26:53.702 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:53.684 7efc337fe700 1 -- 172.21.2.102:0/3121089032 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efc1c00a6e0 con 0x7efc20003ac0 2021-02-10T11:26:53.702 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:53.684 7efc3ac75700 1 -- 172.21.2.102:0/3121089032 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efc20003ac0 msgr2=0x7efc20005f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:53.702 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:53.684 7efc3ac75700 1 --2- 172.21.2.102:0/3121089032 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efc20003ac0 0x7efc20005f60 secure :-1 s=READY pgs=1665 cs=0 l=1 rev1=1 rx=0x7efc1c007990 tx=0x7efc1c007d30).stop 2021-02-10T11:26:53.702 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:53.684 7efc3ac75700 1 -- 172.21.2.102:0/3121089032 shutdown_connections 2021-02-10T11:26:53.702 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:53.684 7efc3ac75700 1 --2- 172.21.2.102:0/3121089032 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efc20003ac0 0x7efc20005f60 unknown :-1 s=CLOSED pgs=1665 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:53.702 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:53.684 7efc3ac75700 1 -- 172.21.2.102:0/3121089032 >> 172.21.2.102:0/3121089032 conn(0x7efc340ce450 msgr2=0x7efc340d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:26:53.702 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:53.684 7efc3ac75700 1 -- 172.21.2.102:0/3121089032 shutdown_connections 2021-02-10T11:26:53.702 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:53.684 7efc3ac75700 1 -- 172.21.2.102:0/3121089032 wait complete. 2021-02-10T11:26:53.702 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:53.699 7efc3ac75700 1 Processor -- start 2021-02-10T11:26:53.703 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:53.699 7efc3ac75700 1 -- start start 2021-02-10T11:26:53.704 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:53.700 7efc3ac75700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7efc341a4a60 0x7efc340c1c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:53.704 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:53.700 7efc3ac75700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efc340c2290 0x7efc340c2690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:53.704 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:53.700 7efc3ac75700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7efc340c2cb0 0x7efc340bfd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:53.704 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:53.700 7efc3ac75700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc340f05b0 con 0x7efc340c2290 2021-02-10T11:26:53.704 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:53.700 7efc3ac75700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc340f07d0 con 0x7efc341a4a60 2021-02-10T11:26:53.704 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:53.700 7efc3ac75700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc340c0340 con 0x7efc340c2cb0 2021-02-10T11:26:53.704 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:53.700 7efc33fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efc340c2290 0x7efc340c2690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:53.704 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:53.700 7efc33fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efc340c2290 0x7efc340c2690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:59098/0 (socket says 172.21.2.102:59098) 2021-02-10T11:26:53.704 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:53.700 7efc33fff700 1 -- 172.21.2.102:0/622049858 learned_addr learned my addr 172.21.2.102:0/622049858 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:26:53.704 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:53.700 7efc38a91700 1 --2- 172.21.2.102:0/622049858 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7efc341a4a60 0x7efc340c1c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:53.705 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:53.700 7efc39292700 1 --2- 172.21.2.102:0/622049858 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7efc340c2cb0 0x7efc340bfd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:53.705 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:53.700 7efc33fff700 1 -- 172.21.2.102:0/622049858 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7efc340c2cb0 msgr2=0x7efc340bfd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:53.705 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:53.700 7efc33fff700 1 --2- 172.21.2.102:0/622049858 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7efc340c2cb0 0x7efc340bfd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:53.705 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:53.700 7efc33fff700 1 -- 172.21.2.102:0/622049858 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7efc341a4a60 msgr2=0x7efc340c1c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:53.705 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:53.700 7efc33fff700 1 --2- 172.21.2.102:0/622049858 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7efc341a4a60 0x7efc340c1c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:53.705 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:53.700 7efc33fff700 1 -- 172.21.2.102:0/622049858 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc340c0560 con 0x7efc340c2290 2021-02-10T11:26:53.705 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:53.700 7efc38a91700 1 --2- 172.21.2.102:0/622049858 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7efc341a4a60 0x7efc340c1c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:26:53.705 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:53.700 7efc33fff700 1 --2- 172.21.2.102:0/622049858 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efc340c2290 0x7efc340c2690 secure :-1 s=READY pgs=1666 cs=0 l=1 rev1=1 rx=0x7efc2400b610 tx=0x7efc2400b9b0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:53.705 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:53.700 7efc317fa700 1 -- 172.21.2.102:0/622049858 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efc2400bda0 con 0x7efc340c2290 2021-02-10T11:26:53.705 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:53.700 7efc317fa700 1 -- 172.21.2.102:0/622049858 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc24000ca0 con 0x7efc340c2290 2021-02-10T11:26:53.705 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:53.701 7efc3ac75700 1 -- 172.21.2.102:0/622049858 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc340c0560 con 0x7efc340c2290 2021-02-10T11:26:53.705 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:53.701 7efc3ac75700 1 -- 172.21.2.102:0/622049858 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc340c0af0 con 0x7efc340c2290 2021-02-10T11:26:53.705 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:53.701 7efc317fa700 1 -- 172.21.2.102:0/622049858 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efc240022b0 con 0x7efc340c2290 2021-02-10T11:26:53.706 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:53.701 7efc317fa700 1 -- 172.21.2.102:0/622049858 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7efc240052e0 con 0x7efc340c2290 2021-02-10T11:26:53.706 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:53.701 7efc317fa700 1 --2- 172.21.2.102:0/622049858 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7efc180186c0 0x7efc1801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:53.706 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:53.701 7efc317fa700 1 -- 172.21.2.102:0/622049858 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5573+0+0 (secure 0 0 0) 0x7efc24032540 con 0x7efc340c2290 2021-02-10T11:26:53.706 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:53.701 7efc38a91700 1 --2- 172.21.2.102:0/622049858 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7efc180186c0 0x7efc1801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:53.706 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:53.702 7efc38a91700 1 --2- 172.21.2.102:0/622049858 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7efc180186c0 0x7efc1801abc0 crc :-1 s=READY pgs=758 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:53.731 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:53.732 7efc3ac75700 1 -- 172.21.2.102:0/622049858 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc340f0af0 con 0x7efc340c2290 2021-02-10T11:26:53.734 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:53.734 7efc317fa700 1 -- 172.21.2.102:0/622049858 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7efc2402d370 con 0x7efc340c2290 2021-02-10T11:26:53.862 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:53.862 7efc3ac75700 1 -- 172.21.2.102:0/622049858 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efc340f0af0 con 0x7efc340c2290 2021-02-10T11:26:53.862 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:53.863 7efc317fa700 1 -- 172.21.2.102:0/622049858 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 57 v57) v1 ==== 94+0+2098 (secure 0 0 0) 0x7efc2402d370 con 0x7efc340c2290 2021-02-10T11:26:53.863 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:26:53.863 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":57,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8865,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.102:6835/980649634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6834","nonce":980649634},{"type":"v1","addr":"172.21.2.102:6835","nonce":980649634}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":53}],"filesystems":[{"mdsmap":{"epoch":57,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:26:47.543386","modified":"2021-02-10 11:26:49.552060","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6799},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6799":{"gid":6799,"name":"b","rank":0,"incarnation":54,"state":"up:active","state_seq":3,"addr":"172.21.2.113:6833/2002224946","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6832","nonce":2002224946},{"type":"v1","addr":"172.21.2.113:6833","nonce":2002224946}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2021-02-10T11:26:53.863 INFO:teuthology.orchestra.run.gibba002.stderr:dumped fsmap epoch 57 2021-02-10T11:26:53.864 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:53.864 7efc3ac75700 1 -- 172.21.2.102:0/622049858 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7efc180186c0 msgr2=0x7efc1801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:53.864 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:53.864 7efc3ac75700 1 --2- 172.21.2.102:0/622049858 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7efc180186c0 0x7efc1801abc0 crc :-1 s=READY pgs=758 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:53.864 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:53.864 7efc3ac75700 1 -- 172.21.2.102:0/622049858 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efc340c2290 msgr2=0x7efc340c2690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:53.865 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:53.865 7efc3ac75700 1 --2- 172.21.2.102:0/622049858 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efc340c2290 0x7efc340c2690 secure :-1 s=READY pgs=1666 cs=0 l=1 rev1=1 rx=0x7efc2400b610 tx=0x7efc2400b9b0).stop 2021-02-10T11:26:53.865 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:53.865 7efc3ac75700 1 -- 172.21.2.102:0/622049858 shutdown_connections 2021-02-10T11:26:53.865 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:53.865 7efc3ac75700 1 --2- 172.21.2.102:0/622049858 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7efc340c2cb0 0x7efc340bfd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:53.865 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:53.865 7efc3ac75700 1 --2- 172.21.2.102:0/622049858 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efc340c2290 0x7efc340c2690 unknown :-1 s=CLOSED pgs=1666 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:53.866 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:53.865 7efc3ac75700 1 --2- 172.21.2.102:0/622049858 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7efc180186c0 0x7efc1801abc0 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:53.866 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:53.865 7efc3ac75700 1 --2- 172.21.2.102:0/622049858 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7efc341a4a60 0x7efc340c1c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:53.866 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:53.865 7efc3ac75700 1 -- 172.21.2.102:0/622049858 >> 172.21.2.102:0/622049858 conn(0x7efc340ebe30 msgr2=0x7efc340bf200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:26:53.866 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:53.865 7efc3ac75700 1 -- 172.21.2.102:0/622049858 shutdown_connections 2021-02-10T11:26:53.866 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:53.865 7efc3ac75700 1 -- 172.21.2.102:0/622049858 wait complete. 2021-02-10T11:26:54.075 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 57, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:26:47.543386', 'modified': '2021-02-10 11:26:49.552060', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6799}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6799': {'gid': 6799, 'name': 'b', 'rank': 0, 'incarnation': 54, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.113:6833/2002224946', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.113:6832', 'nonce': 2002224946}, {'type': 'v1', 'addr': '172.21.2.113:6833', 'nonce': 2002224946}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:26:54.075 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:26:54.075 DEBUG:teuthology.orchestra.run.gibba013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:26:54.475 INFO:teuthology.orchestra.run.gibba013.stdout:{ 2021-02-10T11:26:54.475 INFO:teuthology.orchestra.run.gibba013.stdout: "cluster_fsid": "68db30ea-dbce-4aef-a6df-f01c6fea0513", 2021-02-10T11:26:54.475 INFO:teuthology.orchestra.run.gibba013.stdout: "whoami": 0, 2021-02-10T11:26:54.475 INFO:teuthology.orchestra.run.gibba013.stdout: "id": 6799, 2021-02-10T11:26:54.475 INFO:teuthology.orchestra.run.gibba013.stdout: "want_state": "up:active", 2021-02-10T11:26:54.475 INFO:teuthology.orchestra.run.gibba013.stdout: "state": "up:active", 2021-02-10T11:26:54.476 INFO:teuthology.orchestra.run.gibba013.stdout: "rank_uptime": 6.9302587769999997, 2021-02-10T11:26:54.476 INFO:teuthology.orchestra.run.gibba013.stdout: "mdsmap_epoch": 57, 2021-02-10T11:26:54.476 INFO:teuthology.orchestra.run.gibba013.stdout: "osdmap_epoch": 75, 2021-02-10T11:26:54.476 INFO:teuthology.orchestra.run.gibba013.stdout: "osdmap_epoch_barrier": 75, 2021-02-10T11:26:54.476 INFO:teuthology.orchestra.run.gibba013.stdout: "uptime": 12.567969985 2021-02-10T11:26:54.476 INFO:teuthology.orchestra.run.gibba013.stdout:} 2021-02-10T11:26:54.479 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "68db30ea-dbce-4aef-a6df-f01c6fea0513", "whoami": 0, "id": 6799, "want_state": "up:active", "state": "up:active", "rank_uptime": 6.9302587769999997, "mdsmap_epoch": 57, "osdmap_epoch": 75, "osdmap_epoch_barrier": 75, "uptime": 12.567969985 } 2021-02-10T11:26:54.480 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:26:54.907 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:54.891 7f1ae9215700 1 Processor -- start 2021-02-10T11:26:54.907 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:54.891 7f1ae9215700 1 -- start start 2021-02-10T11:26:54.907 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:54.891 7f1ae9215700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f1ae40d1870 0x7f1ae41b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:54.907 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:54.891 7f1ae9215700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f1ae41b0a10 0x7f1ae40cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:54.908 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:54.891 7f1ae9215700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ae40f0b10 con 0x7f1ae40d1c70 2021-02-10T11:26:54.908 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:54.891 7f1ae9215700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ae40e5ed0 con 0x7f1ae40d1870 2021-02-10T11:26:54.908 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:54.891 7f1ae9215700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ae40f0650 con 0x7f1ae41b0a10 2021-02-10T11:26:54.908 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:54.892 7f1ae2d9d700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f1ae40d1870 0x7f1ae41b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:54.908 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:54.892 7f1ae2d9d700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f1ae40d1870 0x7f1ae41b03f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:59102/0 (socket says 172.21.2.102:59102) 2021-02-10T11:26:54.908 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:54.892 7f1ae359e700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f1ae40d1c70 0x7f1ae40cf950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:51330/0 (socket says 172.21.2.102:51330) 2021-02-10T11:26:54.908 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:54.892 7f1ae2d9d700 1 -- 172.21.2.102:0/4048213056 learned_addr learned my addr 172.21.2.102:0/4048213056 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:26:54.908 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:54.892 7f1ae2d9d700 1 -- 172.21.2.102:0/4048213056 >> v2:172.21.2.113:3300/0 conn(0x7f1ae41b0a10 msgr2=0x7f1ae40cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:26:54.908 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:54.892 7f1ae2d9d700 1 --2- 172.21.2.102:0/4048213056 >> v2:172.21.2.113:3300/0 conn(0x7f1ae41b0a10 0x7f1ae40cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:54.908 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:54.892 7f1ae2d9d700 1 -- 172.21.2.102:0/4048213056 >> v1:172.21.2.102:6789/0 conn(0x7f1ae40d1c70 legacy=0x7f1ae40cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:54.908 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:54.892 7f1ae1d9b700 1 -- 172.21.2.102:0/4048213056 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1258308943 0 0) 0x7f1adc001480 con 0x7f1ae40d1c70 2021-02-10T11:26:54.909 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:54.892 7f1ae2d9d700 1 -- 172.21.2.102:0/4048213056 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ae40f0840 con 0x7f1ae40d1870 2021-02-10T11:26:54.909 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:54.892 7f1ae2d9d700 1 --2- 172.21.2.102:0/4048213056 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1ae40d1870 0x7f1ae41b03f0 secure :-1 s=READY pgs=1668 cs=0 l=1 rev1=1 rx=0x7f1ad4006700 tx=0x7f1ad4009b00).ready entity=mon.0 client_cookie=e2093e9311b5097c server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:54.909 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:54.892 7f1ae1d9b700 1 -- 172.21.2.102:0/4048213056 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1ad400c0c0 con 0x7f1ae40d1870 2021-02-10T11:26:54.909 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:54.892 7f1ae1d9b700 1 -- 172.21.2.102:0/4048213056 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ad400c340 con 0x7f1ae40d1870 2021-02-10T11:26:54.909 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:54.892 7f1ae1d9b700 1 -- 172.21.2.102:0/4048213056 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1ad400c820 con 0x7f1ae40d1870 2021-02-10T11:26:54.909 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:54.893 7f1ae9215700 1 -- 172.21.2.102:0/4048213056 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1ae40d1870 msgr2=0x7f1ae41b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:54.909 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:54.893 7f1ae9215700 1 --2- 172.21.2.102:0/4048213056 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1ae40d1870 0x7f1ae41b03f0 secure :-1 s=READY pgs=1668 cs=0 l=1 rev1=1 rx=0x7f1ad4006700 tx=0x7f1ad4009b00).stop 2021-02-10T11:26:54.909 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:54.893 7f1ae9215700 1 -- 172.21.2.102:0/4048213056 shutdown_connections 2021-02-10T11:26:54.909 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:54.893 7f1ae9215700 1 --2- 172.21.2.102:0/4048213056 >> v2:172.21.2.113:3300/0 conn(0x7f1ae41b0a10 0x7f1ae40cf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:54.909 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:54.893 7f1ae9215700 1 --2- 172.21.2.102:0/4048213056 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1ae40d1870 0x7f1ae41b03f0 unknown :-1 s=CLOSED pgs=1668 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:54.909 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:54.893 7f1ae9215700 1 -- 172.21.2.102:0/4048213056 >> 172.21.2.102:0/4048213056 conn(0x7f1ae40ce450 msgr2=0x7f1ae40d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:26:54.909 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:54.893 7f1ae9215700 1 -- 172.21.2.102:0/4048213056 shutdown_connections 2021-02-10T11:26:54.910 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:54.893 7f1ae9215700 1 -- 172.21.2.102:0/4048213056 wait complete. 2021-02-10T11:26:54.911 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:54.908 7f1ae9215700 1 Processor -- start 2021-02-10T11:26:54.911 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:54.908 7f1ae9215700 1 -- start start 2021-02-10T11:26:54.911 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:54.908 7f1ae9215700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1ae41ac980 0x7f1ae41ace80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:54.912 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:54.908 7f1ae9215700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1ae40aa400 0x7f1ae41b72d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:54.912 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:54.908 7f1ae9215700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1ae40aa800 0x7f1ae41b78f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:54.912 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:54.908 7f1ae9215700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ae41b6d50 con 0x7f1ae40aa400 2021-02-10T11:26:54.912 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:54.908 7f1ae9215700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ae41ad4a0 con 0x7f1ae41ac980 2021-02-10T11:26:54.912 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:54.908 7f1ae9215700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ae418f0e0 con 0x7f1ae40aa800 2021-02-10T11:26:54.912 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:54.908 7f1ae259c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1ae40aa400 0x7f1ae41b72d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:54.912 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:54.908 7f1ae259c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1ae40aa400 0x7f1ae41b72d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:59108/0 (socket says 172.21.2.102:59108) 2021-02-10T11:26:54.912 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:54.908 7f1ae259c700 1 -- 172.21.2.102:0/866155181 learned_addr learned my addr 172.21.2.102:0/866155181 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:26:54.912 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:54.909 7f1ae259c700 1 -- 172.21.2.102:0/866155181 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1ae40aa800 msgr2=0x7f1ae41b78f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:54.912 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:54.909 7f1ae259c700 1 --2- 172.21.2.102:0/866155181 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1ae40aa800 0x7f1ae41b78f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:54.912 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:54.909 7f1ae259c700 1 -- 172.21.2.102:0/866155181 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1ae41ac980 msgr2=0x7f1ae41ace80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:54.913 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:54.909 7f1ae259c700 1 --2- 172.21.2.102:0/866155181 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1ae41ac980 0x7f1ae41ace80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:54.913 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:54.909 7f1ae259c700 1 -- 172.21.2.102:0/866155181 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ae41ac4c0 con 0x7f1ae40aa400 2021-02-10T11:26:54.913 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:54.909 7f1ae259c700 1 --2- 172.21.2.102:0/866155181 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1ae40aa400 0x7f1ae41b72d0 secure :-1 s=READY pgs=1669 cs=0 l=1 rev1=1 rx=0x7f1ad8006f00 tx=0x7f1ad800b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:54.913 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:54.909 7f1ad37fe700 1 -- 172.21.2.102:0/866155181 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1ad8006350 con 0x7f1ae40aa400 2021-02-10T11:26:54.913 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:54.909 7f1ad37fe700 1 -- 172.21.2.102:0/866155181 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ad8006b90 con 0x7f1ae40aa400 2021-02-10T11:26:54.913 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:54.909 7f1ae9215700 1 -- 172.21.2.102:0/866155181 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ae41ac4c0 con 0x7f1ae40aa400 2021-02-10T11:26:54.913 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:54.909 7f1ae9215700 1 -- 172.21.2.102:0/866155181 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ae41ac4c0 con 0x7f1ae40aa400 2021-02-10T11:26:54.913 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:54.909 7f1ad37fe700 1 -- 172.21.2.102:0/866155181 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1ad8000c90 con 0x7f1ae40aa400 2021-02-10T11:26:54.913 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:54.910 7f1ad37fe700 1 -- 172.21.2.102:0/866155181 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1ad8002240 con 0x7f1ae40aa400 2021-02-10T11:26:54.913 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:54.910 7f1ad37fe700 1 --2- 172.21.2.102:0/866155181 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1ac8018730 0x7f1ac801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:54.913 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:54.910 7f1ad37fe700 1 -- 172.21.2.102:0/866155181 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f1ad8032780 con 0x7f1ae40aa400 2021-02-10T11:26:54.913 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:54.910 7f1ae2d9d700 1 --2- 172.21.2.102:0/866155181 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1ac8018730 0x7f1ac801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:54.913 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:54.910 7f1ae2d9d700 1 --2- 172.21.2.102:0/866155181 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1ac8018730 0x7f1ac801ac10 crc :-1 s=READY pgs=759 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:54.941 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:54.941 7f1ae9215700 1 -- 172.21.2.102:0/866155181 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ae40ebc60 con 0x7f1ae40aa400 2021-02-10T11:26:54.943 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:54.944 7f1ad37fe700 1 -- 172.21.2.102:0/866155181 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f1ad8002030 con 0x7f1ae40aa400 2021-02-10T11:26:55.071 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:55.071 7f1ae9215700 1 -- 172.21.2.102:0/866155181 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1ae40f0b10 con 0x7f1ae40aa400 2021-02-10T11:26:55.072 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:55.072 7f1ad37fe700 1 -- 172.21.2.102:0/866155181 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v75) v1 ==== 74+0+13996 (secure 0 0 0) 0x7f1ad8006b90 con 0x7f1ae40aa400 2021-02-10T11:26:55.072 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:26:55.072 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":75,"fsid":"68db30ea-dbce-4aef-a6df-f01c6fea0513","created":"2021-02-10 11:03:06.331557","modified":"2021-02-10 11:26:48.545650","last_up_change":"2021-02-10 11:03:32.690562","last_in_change":"2021-02-10 11:03:32.690562","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:03:37.322519","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:26:45.515299","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"74","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":15,"pool_name":"cephfs_data","create_time":"2021-02-10 11:26:46.587944","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"be691b4d-a039-408a-af80-9d53f02b6f1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6818","nonce":36088},{"type":"v1","addr":"172.21.2.102:6820","nonce":36088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6822","nonce":36088},{"type":"v1","addr":"172.21.2.102:6824","nonce":36088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6830","nonce":36088},{"type":"v1","addr":"172.21.2.102:6832","nonce":36088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6826","nonce":36088},{"type":"v1","addr":"172.21.2.102:6828","nonce":36088}]},"public_addr":"172.21.2.102:6820/36088","cluster_addr":"172.21.2.102:6824/36088","heartbeat_back_addr":"172.21.2.102:6832/36088","heartbeat_front_addr":"172.21.2.102:6828/36088","state":["exists","up"]},{"osd":1,"uuid":"64e3ccc8-9c69-4d7c-865e-af8e83822635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6819","nonce":36087},{"type":"v1","addr":"172.21.2.102:6821","nonce":36087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6823","nonce":36087},{"type":"v1","addr":"172.21.2.102:6825","nonce":36087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6831","nonce":36087},{"type":"v1","addr":"172.21.2.102:6833","nonce":36087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6827","nonce":36087},{"type":"v1","addr":"172.21.2.102:6829","nonce":36087}]},"public_addr":"172.21.2.102:6821/36087","cluster_addr":"172.21.2.102:6825/36087","heartbeat_back_addr":"172.21.2.102:6833/36087","heartbeat_front_addr":"172.21.2.102:6829/36087","state":["exists","up"]},{"osd":2,"uuid":"153700af-747f-481f-bdf4-d6072d3a842f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6810","nonce":36089},{"type":"v1","addr":"172.21.2.102:6811","nonce":36089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6812","nonce":36089},{"type":"v1","addr":"172.21.2.102:6813","nonce":36089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6816","nonce":36089},{"type":"v1","addr":"172.21.2.102:6817","nonce":36089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6814","nonce":36089},{"type":"v1","addr":"172.21.2.102:6815","nonce":36089}]},"public_addr":"172.21.2.102:6811/36089","cluster_addr":"172.21.2.102:6813/36089","heartbeat_back_addr":"172.21.2.102:6817/36089","heartbeat_front_addr":"172.21.2.102:6815/36089","state":["exists","up"]},{"osd":3,"uuid":"cbc678f4-83a9-45e6-8830-2f7c232b1ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6802","nonce":36090},{"type":"v1","addr":"172.21.2.102:6803","nonce":36090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6804","nonce":36090},{"type":"v1","addr":"172.21.2.102:6805","nonce":36090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6808","nonce":36090},{"type":"v1","addr":"172.21.2.102:6809","nonce":36090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6806","nonce":36090},{"type":"v1","addr":"172.21.2.102:6807","nonce":36090}]},"public_addr":"172.21.2.102:6803/36090","cluster_addr":"172.21.2.102:6805/36090","heartbeat_back_addr":"172.21.2.102:6809/36090","heartbeat_front_addr":"172.21.2.102:6807/36090","state":["exists","up"]},{"osd":4,"uuid":"c01e4008-b258-4f10-afd2-9e7f37214351","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6808","nonce":36368},{"type":"v1","addr":"172.21.2.113:6811","nonce":36368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6814","nonce":36368},{"type":"v1","addr":"172.21.2.113:6818","nonce":36368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6826","nonce":36368},{"type":"v1","addr":"172.21.2.113:6828","nonce":36368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6820","nonce":36368},{"type":"v1","addr":"172.21.2.113:6824","nonce":36368}]},"public_addr":"172.21.2.113:6811/36368","cluster_addr":"172.21.2.113:6818/36368","heartbeat_back_addr":"172.21.2.113:6828/36368","heartbeat_front_addr":"172.21.2.113:6824/36368","state":["exists","up"]},{"osd":5,"uuid":"551f6007-daf8-4a98-8ee0-c2b16bcccb5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6802","nonce":36366},{"type":"v1","addr":"172.21.2.113:6805","nonce":36366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6807","nonce":36366},{"type":"v1","addr":"172.21.2.113:6810","nonce":36366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6819","nonce":36366},{"type":"v1","addr":"172.21.2.113:6822","nonce":36366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6812","nonce":36366},{"type":"v1","addr":"172.21.2.113:6815","nonce":36366}]},"public_addr":"172.21.2.113:6805/36366","cluster_addr":"172.21.2.113:6810/36366","heartbeat_back_addr":"172.21.2.113:6822/36366","heartbeat_front_addr":"172.21.2.113:6815/36366","state":["exists","up"]},{"osd":6,"uuid":"bf67c040-0e50-48be-b102-c3943bc9ebef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6800","nonce":36367},{"type":"v1","addr":"172.21.2.113:6801","nonce":36367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6803","nonce":36367},{"type":"v1","addr":"172.21.2.113:6804","nonce":36367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6813","nonce":36367},{"type":"v1","addr":"172.21.2.113:6816","nonce":36367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6806","nonce":36367},{"type":"v1","addr":"172.21.2.113:6809","nonce":36367}]},"public_addr":"172.21.2.113:6801/36367","cluster_addr":"172.21.2.113:6804/36367","heartbeat_back_addr":"172.21.2.113:6816/36367","heartbeat_front_addr":"172.21.2.113:6809/36367","state":["exists","up"]},{"osd":7,"uuid":"f5461f80-018d-428f-be89-2eb2bcbe7bb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6817","nonce":36365},{"type":"v1","addr":"172.21.2.113:6821","nonce":36365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6823","nonce":36365},{"type":"v1","addr":"172.21.2.113:6825","nonce":36365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6830","nonce":36365},{"type":"v1","addr":"172.21.2.113:6831","nonce":36365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6827","nonce":36365},{"type":"v1","addr":"172.21.2.113:6829","nonce":36365}]},"public_addr":"172.21.2.113:6821/36365","cluster_addr":"172.21.2.113:6825/36365","heartbeat_back_addr":"172.21.2.113:6831/36365","heartbeat_front_addr":"172.21.2.113:6829/36365","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:26:55.074 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:55.074 7f1ae9215700 1 -- 172.21.2.102:0/866155181 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1ac8018730 msgr2=0x7f1ac801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:55.074 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:55.074 7f1ae9215700 1 --2- 172.21.2.102:0/866155181 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1ac8018730 0x7f1ac801ac10 crc :-1 s=READY pgs=759 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:55.074 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:55.074 7f1ae9215700 1 -- 172.21.2.102:0/866155181 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1ae40aa400 msgr2=0x7f1ae41b72d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:55.074 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:55.074 7f1ae9215700 1 --2- 172.21.2.102:0/866155181 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1ae40aa400 0x7f1ae41b72d0 secure :-1 s=READY pgs=1669 cs=0 l=1 rev1=1 rx=0x7f1ad8006f00 tx=0x7f1ad800b940).stop 2021-02-10T11:26:55.074 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:55.074 7f1ae9215700 1 -- 172.21.2.102:0/866155181 shutdown_connections 2021-02-10T11:26:55.074 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:55.074 7f1ae9215700 1 --2- 172.21.2.102:0/866155181 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1ae40aa800 0x7f1ae41b78f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:55.074 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:55.074 7f1ae9215700 1 --2- 172.21.2.102:0/866155181 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1ae40aa400 0x7f1ae41b72d0 unknown :-1 s=CLOSED pgs=1669 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:55.074 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:55.074 7f1ae9215700 1 --2- 172.21.2.102:0/866155181 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1ac8018730 0x7f1ac801ac10 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:55.074 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:55.074 7f1ae9215700 1 --2- 172.21.2.102:0/866155181 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1ae41ac980 0x7f1ae41ace80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:55.075 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:55.074 7f1ae9215700 1 -- 172.21.2.102:0/866155181 >> 172.21.2.102:0/866155181 conn(0x7f1ae41ac040 msgr2=0x7f1ae4094980 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:26:55.075 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:55.074 7f1ae9215700 1 -- 172.21.2.102:0/866155181 shutdown_connections 2021-02-10T11:26:55.075 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:55.074 7f1ae9215700 1 -- 172.21.2.102:0/866155181 wait complete. 2021-02-10T11:26:55.083 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:03:37.322519', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 14, 'pool_name': 'cephfs_metadata', 'create_time': '2021-02-10 11:26:45.515299', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '74', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 15, 'pool_name': 'cephfs_data', 'create_time': '2021-02-10 11:26:46.587944', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '75', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-02-10T11:26:55.083 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:26:55.515 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:55.498 7f3e8c22a700 1 Processor -- start 2021-02-10T11:26:55.515 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:55.498 7f3e8c22a700 1 -- start start 2021-02-10T11:26:55.515 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:55.499 7f3e8c22a700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f3e840d1870 0x7f3e841b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:55.515 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:55.499 7f3e8c22a700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f3e840cff70 0x7f3e84109130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:55.516 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:55.499 7f3e8c22a700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e840fe1c0 con 0x7f3e841b0ae0 2021-02-10T11:26:55.516 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:55.499 7f3e8c22a700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e84109ad0 con 0x7f3e840cff70 2021-02-10T11:26:55.516 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:55.499 7f3e8c22a700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e84109750 con 0x7f3e840d1870 2021-02-10T11:26:55.516 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:55.499 7f3e8a847700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f3e840cff70 0x7f3e84109130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:55.516 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:55.499 7f3e8a847700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f3e840cff70 0x7f3e84109130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:59116/0 (socket says 172.21.2.102:59116) 2021-02-10T11:26:55.516 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:55.499 7f3e8a847700 1 -- 172.21.2.102:0/1920271829 learned_addr learned my addr 172.21.2.102:0/1920271829 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:26:55.516 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:55.499 7f3e8a046700 1 --2- 172.21.2.102:0/1920271829 >> v2:172.21.2.113:3300/0 conn(0x7f3e840d1870 0x7f3e841b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:55.516 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:55.499 7f3e8a847700 1 -- 172.21.2.102:0/1920271829 >> v2:172.21.2.113:3300/0 conn(0x7f3e840d1870 msgr2=0x7f3e841b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:26:55.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:55.499 7f3e8a847700 1 --2- 172.21.2.102:0/1920271829 >> v2:172.21.2.113:3300/0 conn(0x7f3e840d1870 0x7f3e841b04c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:55.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:55.499 7f3e8a847700 1 -- 172.21.2.102:0/1920271829 >> v1:172.21.2.102:6789/0 conn(0x7f3e841b0ae0 legacy=0x7f3e84108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:55.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:55.499 7f3e8a847700 1 -- 172.21.2.102:0/1920271829 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e84088070 con 0x7f3e840cff70 2021-02-10T11:26:55.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:55.499 7f3e8a847700 1 --2- 172.21.2.102:0/1920271829 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3e840cff70 0x7f3e84109130 secure :-1 s=READY pgs=1671 cs=0 l=1 rev1=1 rx=0x7f3e7c006f00 tx=0x7f3e7c009e90).ready entity=mon.0 client_cookie=7b0e64cfefc15a98 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:55.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:55.500 7f3e89044700 1 -- 172.21.2.102:0/1920271829 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3e7c00b300 con 0x7f3e840cff70 2021-02-10T11:26:55.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:55.500 7f3e89044700 1 -- 172.21.2.102:0/1920271829 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e7c00b300 con 0x7f3e840cff70 2021-02-10T11:26:55.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:55.500 7f3e89044700 1 -- 172.21.2.102:0/1920271829 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3e7c00b640 con 0x7f3e840cff70 2021-02-10T11:26:55.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:55.500 7f3e8c22a700 1 -- 172.21.2.102:0/1920271829 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3e840cff70 msgr2=0x7f3e84109130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:55.518 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:55.500 7f3e8c22a700 1 --2- 172.21.2.102:0/1920271829 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3e840cff70 0x7f3e84109130 secure :-1 s=READY pgs=1671 cs=0 l=1 rev1=1 rx=0x7f3e7c006f00 tx=0x7f3e7c009e90).stop 2021-02-10T11:26:55.518 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:55.500 7f3e8c22a700 1 -- 172.21.2.102:0/1920271829 shutdown_connections 2021-02-10T11:26:55.518 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:55.500 7f3e8c22a700 1 --2- 172.21.2.102:0/1920271829 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3e840cff70 0x7f3e84109130 unknown :-1 s=CLOSED pgs=1671 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:55.518 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:55.500 7f3e8c22a700 1 --2- 172.21.2.102:0/1920271829 >> v2:172.21.2.113:3300/0 conn(0x7f3e840d1870 0x7f3e841b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:55.518 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:55.500 7f3e8c22a700 1 -- 172.21.2.102:0/1920271829 >> 172.21.2.102:0/1920271829 conn(0x7f3e840ce420 msgr2=0x7f3e840d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:26:55.518 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:55.501 7f3e8c22a700 1 -- 172.21.2.102:0/1920271829 shutdown_connections 2021-02-10T11:26:55.518 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:55.501 7f3e8c22a700 1 -- 172.21.2.102:0/1920271829 wait complete. 2021-02-10T11:26:55.518 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:55.515 7f3e8c22a700 1 Processor -- start 2021-02-10T11:26:55.519 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:55.515 7f3e8c22a700 1 -- start start 2021-02-10T11:26:55.521 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:55.516 7f3e8c22a700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3e84191b10 0x7f3e840ae640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:55.521 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:55.516 7f3e8c22a700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3e84191f10 0x7f3e8419ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:55.521 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:55.516 7f3e8c22a700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3e841a05f0 0x7f3e84082ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:55.521 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:55.516 7f3e8c22a700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e84192440 con 0x7f3e84191b10 2021-02-10T11:26:55.521 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:55.516 7f3e8c22a700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e841916e0 con 0x7f3e841a05f0 2021-02-10T11:26:55.521 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:55.516 7f3e8c22a700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e84191900 con 0x7f3e84191f10 2021-02-10T11:26:55.522 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:55.516 7f3e8a046700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3e84191b10 0x7f3e840ae640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:55.522 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:55.516 7f3e8a046700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3e84191b10 0x7f3e840ae640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:59122/0 (socket says 172.21.2.102:59122) 2021-02-10T11:26:55.522 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:55.516 7f3e8a046700 1 -- 172.21.2.102:0/3060668551 learned_addr learned my addr 172.21.2.102:0/3060668551 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:26:55.522 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:55.516 7f3e8a046700 1 -- 172.21.2.102:0/3060668551 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3e84191f10 msgr2=0x7f3e8419ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:55.522 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:55.516 7f3e8a046700 1 --2- 172.21.2.102:0/3060668551 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3e84191f10 0x7f3e8419ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:55.527 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:55.516 7f3e8a046700 1 -- 172.21.2.102:0/3060668551 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3e841a05f0 msgr2=0x7f3e84082ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:55.527 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:55.516 7f3e8a046700 1 --2- 172.21.2.102:0/3060668551 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3e841a05f0 0x7f3e84082ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:55.527 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:55.516 7f3e8a046700 1 -- 172.21.2.102:0/3060668551 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e840834f0 con 0x7f3e84191b10 2021-02-10T11:26:55.527 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:55.516 7f3e8a046700 1 --2- 172.21.2.102:0/3060668551 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3e84191b10 0x7f3e840ae640 secure :-1 s=READY pgs=1672 cs=0 l=1 rev1=1 rx=0x7f3e7800b560 tx=0x7f3e7800b900).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:55.527 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:55.517 7f3e76ffd700 1 -- 172.21.2.102:0/3060668551 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3e7800bcf0 con 0x7f3e84191b10 2021-02-10T11:26:55.527 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:55.517 7f3e76ffd700 1 -- 172.21.2.102:0/3060668551 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e78000c90 con 0x7f3e84191b10 2021-02-10T11:26:55.528 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:55.517 7f3e8c22a700 1 -- 172.21.2.102:0/3060668551 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e840834f0 con 0x7f3e84191b10 2021-02-10T11:26:55.528 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:55.517 7f3e76ffd700 1 -- 172.21.2.102:0/3060668551 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3e78002220 con 0x7f3e84191b10 2021-02-10T11:26:55.528 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:55.517 7f3e8c22a700 1 -- 172.21.2.102:0/3060668551 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e84083a80 con 0x7f3e84191b10 2021-02-10T11:26:55.528 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:55.517 7f3e76ffd700 1 -- 172.21.2.102:0/3060668551 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f3e78006b00 con 0x7f3e84191b10 2021-02-10T11:26:55.529 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:55.517 7f3e76ffd700 1 --2- 172.21.2.102:0/3060668551 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3e6c018730 0x7f3e6c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:55.529 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:55.517 7f3e76ffd700 1 -- 172.21.2.102:0/3060668551 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f3e78032690 con 0x7f3e84191b10 2021-02-10T11:26:55.529 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:55.518 7f3e89845700 1 --2- 172.21.2.102:0/3060668551 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3e6c018730 0x7f3e6c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:55.529 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:55.518 7f3e89845700 1 --2- 172.21.2.102:0/3060668551 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3e6c018730 0x7f3e6c01abc0 crc :-1 s=READY pgs=760 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:55.548 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:55.548 7f3e8c22a700 1 -- 172.21.2.102:0/3060668551 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e84191900 con 0x7f3e84191b10 2021-02-10T11:26:55.550 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:55.552 7f3e76ffd700 1 -- 172.21.2.102:0/3060668551 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f3e780009b0 con 0x7f3e84191b10 2021-02-10T11:26:55.681 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:55.680 7f3e8c22a700 1 -- 172.21.2.102:0/3060668551 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f3e84191900 con 0x7f3e84191b10 2021-02-10T11:26:55.681 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:55.681 7f3e76ffd700 1 -- 172.21.2.102:0/3060668551 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v75) v1 ==== 92+0+10 (secure 0 0 0) 0x7f3e78006b00 con 0x7f3e84191b10 2021-02-10T11:26:55.683 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:26:55.684 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:55.683 7f3e8c22a700 1 -- 172.21.2.102:0/3060668551 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3e6c018730 msgr2=0x7f3e6c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:55.685 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:55.683 7f3e8c22a700 1 --2- 172.21.2.102:0/3060668551 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3e6c018730 0x7f3e6c01abc0 crc :-1 s=READY pgs=760 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:55.685 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:55.683 7f3e8c22a700 1 -- 172.21.2.102:0/3060668551 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3e84191b10 msgr2=0x7f3e840ae640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:55.685 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:55.683 7f3e8c22a700 1 --2- 172.21.2.102:0/3060668551 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3e84191b10 0x7f3e840ae640 secure :-1 s=READY pgs=1672 cs=0 l=1 rev1=1 rx=0x7f3e7800b560 tx=0x7f3e7800b900).stop 2021-02-10T11:26:55.685 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:55.683 7f3e8c22a700 1 -- 172.21.2.102:0/3060668551 shutdown_connections 2021-02-10T11:26:55.686 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:55.683 7f3e8c22a700 1 --2- 172.21.2.102:0/3060668551 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3e6c018730 0x7f3e6c01abc0 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:55.686 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:55.683 7f3e8c22a700 1 --2- 172.21.2.102:0/3060668551 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3e841a05f0 0x7f3e84082ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:55.686 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:55.683 7f3e8c22a700 1 --2- 172.21.2.102:0/3060668551 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3e84191f10 0x7f3e8419ffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:55.686 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:55.683 7f3e8c22a700 1 --2- 172.21.2.102:0/3060668551 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3e84191b10 0x7f3e840ae640 unknown :-1 s=CLOSED pgs=1672 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:55.686 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:55.683 7f3e8c22a700 1 -- 172.21.2.102:0/3060668551 >> 172.21.2.102:0/3060668551 conn(0x7f3e840a02d0 msgr2=0x7f3e8409f200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:26:55.687 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:55.683 7f3e8c22a700 1 -- 172.21.2.102:0/3060668551 shutdown_connections 2021-02-10T11:26:55.687 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:55.683 7f3e8c22a700 1 -- 172.21.2.102:0/3060668551 wait complete. 2021-02-10T11:26:55.692 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-02-10T11:26:56.126 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:56.110 7f7835b1b700 1 Processor -- start 2021-02-10T11:26:56.126 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:56.110 7f7835b1b700 1 -- start start 2021-02-10T11:26:56.126 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:56.110 7f7835b1b700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f78300d1c70 0x7f78300cef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:56.127 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:56.110 7f7835b1b700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f78300cf580 0x7f78300cf980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:56.127 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:56.110 7f7835b1b700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78300fe1c0 con 0x7f78300d1870 2021-02-10T11:26:56.127 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:56.110 7f7835b1b700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78301b0cb0 con 0x7f78300cf580 2021-02-10T11:26:56.127 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:56.110 7f7835b1b700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7830109a60 con 0x7f78300d1c70 2021-02-10T11:26:56.127 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:56.111 7f782ffff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f78300cf580 0x7f78300cf980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:56.127 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:56.111 7f782ffff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f78300cf580 0x7f78300cf980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:59132/0 (socket says 172.21.2.102:59132) 2021-02-10T11:26:56.127 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:56.111 7f782ffff700 1 -- 172.21.2.102:0/307140808 learned_addr learned my addr 172.21.2.102:0/307140808 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:26:56.127 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:56.111 7f782ffff700 1 -- 172.21.2.102:0/307140808 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f78300d1c70 msgr2=0x7f78300cef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:26:56.127 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:56.111 7f782ffff700 1 --2- 172.21.2.102:0/307140808 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f78300d1c70 0x7f78300cef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:56.127 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:56.111 7f782ffff700 1 -- 172.21.2.102:0/307140808 >> v1:172.21.2.113:6789/0 conn(0x7f78300d1870 legacy=0x7f78301b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:26:56.127 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:56.111 7f782ffff700 1 -- 172.21.2.102:0/307140808 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7830088070 con 0x7f78300cf580 2021-02-10T11:26:56.127 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:56.111 7f782ffff700 1 --2- 172.21.2.102:0/307140808 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f78300cf580 0x7f78300cf980 secure :-1 s=READY pgs=1673 cs=0 l=1 rev1=1 rx=0x7f7828006f00 tx=0x7f7828009f10).ready entity=mon.0 client_cookie=4bc61923c621c315 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:56.127 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:56.111 7f782e7fc700 1 -- 172.21.2.102:0/307140808 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f782800b3b0 con 0x7f78300cf580 2021-02-10T11:26:56.128 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:56.111 7f782e7fc700 1 -- 172.21.2.102:0/307140808 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f782800b730 con 0x7f78300cf580 2021-02-10T11:26:56.128 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:56.111 7f782e7fc700 1 -- 172.21.2.102:0/307140808 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f782800bc50 con 0x7f78300cf580 2021-02-10T11:26:56.128 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:56.111 7f7835b1b700 1 -- 172.21.2.102:0/307140808 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f78300cf580 msgr2=0x7f78300cf980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:56.128 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:56.111 7f7835b1b700 1 --2- 172.21.2.102:0/307140808 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f78300cf580 0x7f78300cf980 secure :-1 s=READY pgs=1673 cs=0 l=1 rev1=1 rx=0x7f7828006f00 tx=0x7f7828009f10).stop 2021-02-10T11:26:56.128 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:56.113 7f7835b1b700 1 -- 172.21.2.102:0/307140808 shutdown_connections 2021-02-10T11:26:56.128 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:56.113 7f7835b1b700 1 --2- 172.21.2.102:0/307140808 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f78300d1c70 0x7f78300cef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:56.128 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:56.113 7f7835b1b700 1 --2- 172.21.2.102:0/307140808 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f78300cf580 0x7f78300cf980 unknown :-1 s=CLOSED pgs=1673 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:56.128 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:56.113 7f7835b1b700 1 -- 172.21.2.102:0/307140808 >> 172.21.2.102:0/307140808 conn(0x7f78300ce420 msgr2=0x7f78300d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:26:56.128 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:56.113 7f7835b1b700 1 -- 172.21.2.102:0/307140808 shutdown_connections 2021-02-10T11:26:56.128 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:56.113 7f7835b1b700 1 -- 172.21.2.102:0/307140808 wait complete. 2021-02-10T11:26:56.129 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:56.126 7f7835b1b700 1 Processor -- start 2021-02-10T11:26:56.130 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:56.126 7f7835b1b700 1 -- start start 2021-02-10T11:26:56.131 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:56.127 7f7835b1b700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7830099f10 0x7f78300b0850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:56.131 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:56.127 7f7835b1b700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f783009a310 0x7f78300b0e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:56.131 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:56.127 7f7835b1b700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f78300b1490 0x7f78300ac790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:56.131 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:56.127 7f7835b1b700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78300abfa0 con 0x7f78300b1490 2021-02-10T11:26:56.131 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:56.127 7f7835b1b700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78300ac210 con 0x7f783009a310 2021-02-10T11:26:56.131 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:56.127 7f7835b1b700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78300b1890 con 0x7f7830099f10 2021-02-10T11:26:56.131 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:56.127 7f782ffff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f78300b1490 0x7f78300ac790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:56.132 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:56.127 7f782ffff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f78300b1490 0x7f78300ac790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:59134/0 (socket says 172.21.2.102:59134) 2021-02-10T11:26:56.132 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:56.127 7f782ffff700 1 -- 172.21.2.102:0/1453571971 learned_addr learned my addr 172.21.2.102:0/1453571971 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:26:56.132 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:56.127 7f782ffff700 1 -- 172.21.2.102:0/1453571971 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7830099f10 msgr2=0x7f78300b0850 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:26:56.132 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:56.127 7f782ffff700 1 --2- 172.21.2.102:0/1453571971 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7830099f10 0x7f78300b0850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:56.132 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:56.127 7f782ffff700 1 -- 172.21.2.102:0/1453571971 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f783009a310 msgr2=0x7f78300b0e70 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:26:56.132 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:56.127 7f782ffff700 1 --2- 172.21.2.102:0/1453571971 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f783009a310 0x7f78300b0e70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:56.132 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:56.127 7f782ffff700 1 -- 172.21.2.102:0/1453571971 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78300acdb0 con 0x7f78300b1490 2021-02-10T11:26:56.132 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:56.128 7f782ffff700 1 --2- 172.21.2.102:0/1453571971 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f78300b1490 0x7f78300ac790 secure :-1 s=READY pgs=1674 cs=0 l=1 rev1=1 rx=0x7f7828009390 tx=0x7f782800bd40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:56.132 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:56.128 7f7827fff700 1 -- 172.21.2.102:0/1453571971 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7828003030 con 0x7f78300b1490 2021-02-10T11:26:56.132 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:56.128 7f7827fff700 1 -- 172.21.2.102:0/1453571971 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78280068e0 con 0x7f78300b1490 2021-02-10T11:26:56.132 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:56.128 7f7827fff700 1 -- 172.21.2.102:0/1453571971 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f782800c4e0 con 0x7f78300b1490 2021-02-10T11:26:56.132 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:56.128 7f7835b1b700 1 -- 172.21.2.102:0/1453571971 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78300acdb0 con 0x7f78300b1490 2021-02-10T11:26:56.132 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:56.128 7f7835b1b700 1 -- 172.21.2.102:0/1453571971 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78300ad340 con 0x7f78300b1490 2021-02-10T11:26:56.133 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:56.128 7f7827fff700 1 -- 172.21.2.102:0/1453571971 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f7828001270 con 0x7f78300b1490 2021-02-10T11:26:56.133 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:56.128 7f7827fff700 1 --2- 172.21.2.102:0/1453571971 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f78140186c0 0x7f781401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:56.133 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:56.130 7f7827fff700 1 -- 172.21.2.102:0/1453571971 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f7828032760 con 0x7f78300b1490 2021-02-10T11:26:56.133 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:56.130 7f782f7fe700 1 --2- 172.21.2.102:0/1453571971 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f78140186c0 0x7f781401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:56.133 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:56.130 7f782f7fe700 1 --2- 172.21.2.102:0/1453571971 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f78140186c0 0x7f781401abc0 crc :-1 s=READY pgs=761 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:56.160 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:56.161 7f7835b1b700 1 -- 172.21.2.102:0/1453571971 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78300adbd0 con 0x7f78300b1490 2021-02-10T11:26:56.163 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:56.163 7f7827fff700 1 -- 172.21.2.102:0/1453571971 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f782802df80 con 0x7f78300b1490 2021-02-10T11:26:56.291 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:56.292 7f7835b1b700 1 -- 172.21.2.102:0/1453571971 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7f78300adbd0 con 0x7f78300b1490 2021-02-10T11:26:56.292 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:56.293 7f7827fff700 1 -- 172.21.2.102:0/1453571971 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v75) v1 ==== 104+0+10 (secure 0 0 0) 0x7f782802df80 con 0x7f78300b1490 2021-02-10T11:26:56.292 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:26:56.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:56.295 7f7835b1b700 1 -- 172.21.2.102:0/1453571971 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f78140186c0 msgr2=0x7f781401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:56.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:56.295 7f7835b1b700 1 --2- 172.21.2.102:0/1453571971 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f78140186c0 0x7f781401abc0 crc :-1 s=READY pgs=761 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:56.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:56.295 7f7835b1b700 1 -- 172.21.2.102:0/1453571971 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f78300b1490 msgr2=0x7f78300ac790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:56.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:56.295 7f7835b1b700 1 --2- 172.21.2.102:0/1453571971 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f78300b1490 0x7f78300ac790 secure :-1 s=READY pgs=1674 cs=0 l=1 rev1=1 rx=0x7f7828009390 tx=0x7f782800bd40).stop 2021-02-10T11:26:56.294 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:56.295 7f7835b1b700 1 -- 172.21.2.102:0/1453571971 shutdown_connections 2021-02-10T11:26:56.294 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:56.295 7f7835b1b700 1 --2- 172.21.2.102:0/1453571971 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f78300b1490 0x7f78300ac790 unknown :-1 s=CLOSED pgs=1674 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:56.294 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:56.295 7f7835b1b700 1 --2- 172.21.2.102:0/1453571971 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f78140186c0 0x7f781401abc0 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:56.294 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:56.295 7f7835b1b700 1 --2- 172.21.2.102:0/1453571971 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f783009a310 0x7f78300b0e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:56.294 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:56.295 7f7835b1b700 1 --2- 172.21.2.102:0/1453571971 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7830099f10 0x7f78300b0850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:56.295 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:56.295 7f7835b1b700 1 -- 172.21.2.102:0/1453571971 >> 172.21.2.102:0/1453571971 conn(0x7f783009b130 msgr2=0x7f78300a4da0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:26:56.295 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:56.295 7f7835b1b700 1 -- 172.21.2.102:0/1453571971 shutdown_connections 2021-02-10T11:26:56.295 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:56.295 7f7835b1b700 1 -- 172.21.2.102:0/1453571971 wait complete. 2021-02-10T11:26:56.302 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-02-10T11:26:56.734 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:56.719 7fdde6a68700 1 Processor -- start 2021-02-10T11:26:56.734 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:56.719 7fdde6a68700 1 -- start start 2021-02-10T11:26:56.734 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:56.719 7fdde6a68700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdde00d1870 0x7fdde01b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:56.734 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:56.719 7fdde6a68700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fdde00cff70 0x7fdde0109130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:56.734 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:56.719 7fdde6a68700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdde0109a60 con 0x7fdde01b0ae0 2021-02-10T11:26:56.734 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:56.719 7fdde6a68700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdde0109750 con 0x7fdde00cff70 2021-02-10T11:26:56.734 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:56.719 7fdde6a68700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdde0088070 con 0x7fdde00d1870 2021-02-10T11:26:56.734 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:56.719 7fdde5085700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fdde00cff70 0x7fdde0109130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:56.734 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:56.719 7fdddffff700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fdde01b0ae0 0x7fdde0108940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:51368/0 (socket says 172.21.2.102:51368) 2021-02-10T11:26:56.734 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:56.719 7fdddffff700 1 -- 172.21.2.102:0/3827172557 learned_addr learned my addr 172.21.2.102:0/3827172557 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:26:56.735 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:56.720 7fdde5085700 1 -- 172.21.2.102:0/3827172557 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdde00d1870 msgr2=0x7fdde01b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:26:56.735 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:56.720 7fdde5085700 1 --2- 172.21.2.102:0/3827172557 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdde00d1870 0x7fdde01b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:56.735 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:56.720 7fdde5085700 1 -- 172.21.2.102:0/3827172557 >> v1:172.21.2.102:6789/0 conn(0x7fdde01b0ae0 legacy=0x7fdde0108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:56.735 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:56.720 7fdde5085700 1 -- 172.21.2.102:0/3827172557 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdde0088260 con 0x7fdde00cff70 2021-02-10T11:26:56.735 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:56.720 7fdde5085700 1 --2- 172.21.2.102:0/3827172557 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdde00cff70 0x7fdde0109130 secure :-1 s=READY pgs=1676 cs=0 l=1 rev1=1 rx=0x7fddd8006ab0 tx=0x7fddd8009ac0).ready entity=mon.0 client_cookie=62c58bf223b8f783 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:56.735 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:56.720 7fdddf7fe700 1 -- 172.21.2.102:0/3827172557 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fddd80066d0 con 0x7fdde00cff70 2021-02-10T11:26:56.735 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:56.720 7fdddf7fe700 1 -- 172.21.2.102:0/3827172557 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddd80022e0 con 0x7fdde00cff70 2021-02-10T11:26:56.735 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:56.720 7fdddf7fe700 1 -- 172.21.2.102:0/3827172557 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fddd8002740 con 0x7fdde00cff70 2021-02-10T11:26:56.735 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:56.720 7fdde6a68700 1 -- 172.21.2.102:0/3827172557 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdde00cff70 msgr2=0x7fdde0109130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:56.735 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:56.720 7fdde6a68700 1 --2- 172.21.2.102:0/3827172557 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdde00cff70 0x7fdde0109130 secure :-1 s=READY pgs=1676 cs=0 l=1 rev1=1 rx=0x7fddd8006ab0 tx=0x7fddd8009ac0).stop 2021-02-10T11:26:56.735 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:56.721 7fdde6a68700 1 -- 172.21.2.102:0/3827172557 shutdown_connections 2021-02-10T11:26:56.735 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:56.721 7fdde6a68700 1 --2- 172.21.2.102:0/3827172557 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdde00cff70 0x7fdde0109130 unknown :-1 s=CLOSED pgs=1676 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:56.735 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:56.721 7fdde6a68700 1 --2- 172.21.2.102:0/3827172557 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdde00d1870 0x7fdde01b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:56.736 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:56.721 7fdde6a68700 1 -- 172.21.2.102:0/3827172557 >> 172.21.2.102:0/3827172557 conn(0x7fdde00ce420 msgr2=0x7fdde00d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:26:56.736 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:56.721 7fdde6a68700 1 -- 172.21.2.102:0/3827172557 shutdown_connections 2021-02-10T11:26:56.736 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:56.721 7fdde6a68700 1 -- 172.21.2.102:0/3827172557 wait complete. 2021-02-10T11:26:56.737 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:56.735 7fdde6a68700 1 Processor -- start 2021-02-10T11:26:56.737 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:56.735 7fdde6a68700 1 -- start start 2021-02-10T11:26:56.737 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:56.736 7fdde6a68700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdde00c3820 0x7fdde0084cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:56.737 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:56.736 7fdde6a68700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fdde00a0290 0x7fdde00852d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:56.738 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:56.736 7fdde6a68700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdde0086270 0x7fdde0086670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:56.738 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:56.736 7fdde6a68700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdde00c0700 con 0x7fdde0086270 2021-02-10T11:26:56.738 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:56.736 7fdde6a68700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdde00858f0 con 0x7fdde00a0290 2021-02-10T11:26:56.738 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:56.736 7fdde6a68700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdde0085b10 con 0x7fdde00c3820 2021-02-10T11:26:56.738 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:56.736 7fdde5085700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdde0086270 0x7fdde0086670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:56.738 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:56.736 7fdde5085700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdde0086270 0x7fdde0086670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:59150/0 (socket says 172.21.2.102:59150) 2021-02-10T11:26:56.738 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:56.736 7fdde5085700 1 -- 172.21.2.102:0/4210593318 learned_addr learned my addr 172.21.2.102:0/4210593318 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:26:56.738 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:56.736 7fdde5085700 1 -- 172.21.2.102:0/4210593318 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdde00c3820 msgr2=0x7fdde0084cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:56.738 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:56.736 7fdde5085700 1 --2- 172.21.2.102:0/4210593318 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdde00c3820 0x7fdde0084cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:56.738 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:56.736 7fdde5085700 1 -- 172.21.2.102:0/4210593318 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fdde00a0290 msgr2=0x7fdde00852d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:56.738 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:56.736 7fdde5085700 1 --2- 172.21.2.102:0/4210593318 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fdde00a0290 0x7fdde00852d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:56.738 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:56.736 7fdde5085700 1 -- 172.21.2.102:0/4210593318 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdde0086c90 con 0x7fdde0086270 2021-02-10T11:26:56.738 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:56.737 7fdde5085700 1 --2- 172.21.2.102:0/4210593318 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdde0086270 0x7fdde0086670 secure :-1 s=READY pgs=1677 cs=0 l=1 rev1=1 rx=0x7fddd8000e10 tx=0x7fddd8002880).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:56.739 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:56.737 7fdddd7fa700 1 -- 172.21.2.102:0/4210593318 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fddd800c2c0 con 0x7fdde0086270 2021-02-10T11:26:56.739 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:56.737 7fdddd7fa700 1 -- 172.21.2.102:0/4210593318 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddd800c540 con 0x7fdde0086270 2021-02-10T11:26:56.739 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:56.737 7fdde6a68700 1 -- 172.21.2.102:0/4210593318 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdde0086c90 con 0x7fdde0086270 2021-02-10T11:26:56.739 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:56.737 7fdddd7fa700 1 -- 172.21.2.102:0/4210593318 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fddd800c9d0 con 0x7fdde0086270 2021-02-10T11:26:56.739 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:56.737 7fdde6a68700 1 -- 172.21.2.102:0/4210593318 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdde0086c90 con 0x7fdde0086270 2021-02-10T11:26:56.739 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:56.737 7fdddd7fa700 1 -- 172.21.2.102:0/4210593318 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fddd8006fa0 con 0x7fdde0086270 2021-02-10T11:26:56.739 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:56.738 7fdddd7fa700 1 --2- 172.21.2.102:0/4210593318 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fddc4018730 0x7fddc401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:56.739 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:56.738 7fdddd7fa700 1 -- 172.21.2.102:0/4210593318 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fddd8032790 con 0x7fdde0086270 2021-02-10T11:26:56.739 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:56.738 7fdde4884700 1 --2- 172.21.2.102:0/4210593318 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fddc4018730 0x7fddc401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:56.739 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:56.738 7fdde4884700 1 --2- 172.21.2.102:0/4210593318 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fddc4018730 0x7fddc401ac10 crc :-1 s=READY pgs=762 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:56.768 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:56.769 7fdde6a68700 1 -- 172.21.2.102:0/4210593318 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdde0082a70 con 0x7fdde0086270 2021-02-10T11:26:56.771 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:56.772 7fdddd7fa700 1 -- 172.21.2.102:0/4210593318 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fddd8002c70 con 0x7fdde0086270 2021-02-10T11:26:56.899 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:56.900 7fdde6a68700 1 -- 172.21.2.102:0/4210593318 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7fdde0082a70 con 0x7fdde0086270 2021-02-10T11:26:56.899 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:56.901 7fdddd7fa700 1 -- 172.21.2.102:0/4210593318 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v75) v1 ==== 100+0+10 (secure 0 0 0) 0x7fddd800c540 con 0x7fdde0086270 2021-02-10T11:26:56.900 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:26:56.901 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:56.902 7fdde6a68700 1 -- 172.21.2.102:0/4210593318 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fddc4018730 msgr2=0x7fddc401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:56.901 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:56.902 7fdde6a68700 1 --2- 172.21.2.102:0/4210593318 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fddc4018730 0x7fddc401ac10 crc :-1 s=READY pgs=762 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:56.901 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:56.902 7fdde6a68700 1 -- 172.21.2.102:0/4210593318 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdde0086270 msgr2=0x7fdde0086670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:56.901 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:56.902 7fdde6a68700 1 --2- 172.21.2.102:0/4210593318 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdde0086270 0x7fdde0086670 secure :-1 s=READY pgs=1677 cs=0 l=1 rev1=1 rx=0x7fddd8000e10 tx=0x7fddd8002880).stop 2021-02-10T11:26:56.902 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:56.903 7fdde6a68700 1 -- 172.21.2.102:0/4210593318 shutdown_connections 2021-02-10T11:26:56.902 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:56.903 7fdde6a68700 1 --2- 172.21.2.102:0/4210593318 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdde0086270 0x7fdde0086670 unknown :-1 s=CLOSED pgs=1677 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:56.902 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:56.903 7fdde6a68700 1 --2- 172.21.2.102:0/4210593318 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fddc4018730 0x7fddc401ac10 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:56.902 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:56.903 7fdde6a68700 1 --2- 172.21.2.102:0/4210593318 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fdde00a0290 0x7fdde00852d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:56.902 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:56.903 7fdde6a68700 1 --2- 172.21.2.102:0/4210593318 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdde00c3820 0x7fdde0084cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:56.902 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:56.903 7fdde6a68700 1 -- 172.21.2.102:0/4210593318 >> 172.21.2.102:0/4210593318 conn(0x7fdde00872c0 msgr2=0x7fdde0095700 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:26:56.902 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:56.903 7fdde6a68700 1 -- 172.21.2.102:0/4210593318 shutdown_connections 2021-02-10T11:26:56.902 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:56.903 7fdde6a68700 1 -- 172.21.2.102:0/4210593318 wait complete. 2021-02-10T11:26:56.910 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-02-10T11:26:56.910 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:26:57.338 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:57.321 7f204c77f700 1 Processor -- start 2021-02-10T11:26:57.338 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:57.321 7f204c77f700 1 -- start start 2021-02-10T11:26:57.338 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:57.322 7f204c77f700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f20440d0340 0x7f20440ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:57.338 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:57.322 7f204c77f700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20440f0b10 con 0x7f20440d1870 2021-02-10T11:26:57.338 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:57.322 7f204c77f700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20441b0be0 con 0x7f20440d1c70 2021-02-10T11:26:57.338 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:57.322 7f204c77f700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20440e5ed0 con 0x7f20440d0340 2021-02-10T11:26:57.338 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:57.322 7f204a59b700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f20440d1870 0x7f20441b03f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:51378/0 (socket says 172.21.2.102:51378) 2021-02-10T11:26:57.338 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:57.322 7f204a59b700 1 -- 172.21.2.102:0/2180525657 learned_addr learned my addr 172.21.2.102:0/2180525657 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:26:57.338 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:57.322 7f2049599700 1 -- 172.21.2.102:0/2180525657 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3548886233 0 0) 0x7f20380009b0 con 0x7f20440d1870 2021-02-10T11:26:57.339 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:57.323 7f2049599700 1 -- 172.21.2.102:0/2180525657 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2030002fe0 con 0x7f20440d1870 2021-02-10T11:26:57.339 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:57.323 7f2049599700 1 -- 172.21.2.102:0/2180525657 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1295331674 0 0) 0x7f2038001010 con 0x7f20440d1870 2021-02-10T11:26:57.339 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:57.323 7f2049d9a700 1 --2- 172.21.2.102:0/2180525657 >> v2:172.21.2.113:3300/0 conn(0x7f20440d0340 0x7f20440ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:57.339 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:57.323 7f2049599700 1 -- 172.21.2.102:0/2180525657 >> v2:172.21.2.113:3300/0 conn(0x7f20440d0340 msgr2=0x7f20440ef640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:26:57.339 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:57.323 7f2049599700 1 --2- 172.21.2.102:0/2180525657 >> v2:172.21.2.113:3300/0 conn(0x7f20440d0340 0x7f20440ef640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:57.339 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:57.323 7f2049599700 1 -- 172.21.2.102:0/2180525657 >> v1:172.21.2.113:6789/0 conn(0x7f20440d1c70 legacy=0x7f20440efc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:26:57.339 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:57.323 7f2049599700 1 -- 172.21.2.102:0/2180525657 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20440f0450 con 0x7f20440d1870 2021-02-10T11:26:57.339 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:57.323 7f2049599700 1 -- 172.21.2.102:0/2180525657 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7f2038001590 con 0x7f20440d1870 2021-02-10T11:26:57.339 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:57.323 7f2049599700 1 -- 172.21.2.102:0/2180525657 >> v1:172.21.2.102:6789/0 conn(0x7f20440d1870 legacy=0x7f20441b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:57.339 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:57.323 7f2049599700 1 --2- 172.21.2.102:0/2180525657 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2030003ae0 0x7f2030006000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:57.339 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:57.323 7f2049599700 1 --2- 172.21.2.102:0/2180525657 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2030006820 0x7f2030008d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:57.339 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:57.323 7f2049599700 1 --2- 172.21.2.102:0/2180525657 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2030009640 0x7f203000bb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:57.339 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:57.323 7f2049599700 1 -- 172.21.2.102:0/2180525657 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f203000c390 con 0x7f2030006820 2021-02-10T11:26:57.339 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:57.323 7f2049599700 1 -- 172.21.2.102:0/2180525657 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f203000c5b0 con 0x7f2030009640 2021-02-10T11:26:57.340 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:57.323 7f2049599700 1 -- 172.21.2.102:0/2180525657 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f203000c7d0 con 0x7f2030003ae0 2021-02-10T11:26:57.340 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:57.323 7f2049d9a700 1 --2- 172.21.2.102:0/2180525657 >> v2:172.21.2.113:3300/0 conn(0x7f20440d0340 0x7f20440ef640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:26:57.340 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:57.323 7f204ad9c700 1 --2- 172.21.2.102:0/2180525657 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2030009640 0x7f203000bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:57.340 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:57.323 7f204a59b700 1 --2- 172.21.2.102:0/2180525657 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2030003ae0 0x7f2030006000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:57.340 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:57.323 7f2049d9a700 1 --2- 172.21.2.102:0/2180525657 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2030006820 0x7f2030008d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:57.340 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:57.323 7f2049d9a700 1 -- 172.21.2.102:0/2180525657 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2030003ae0 msgr2=0x7f2030006000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:26:57.340 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:57.323 7f2049d9a700 1 --2- 172.21.2.102:0/2180525657 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2030003ae0 0x7f2030006000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:57.340 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:57.323 7f2049d9a700 1 -- 172.21.2.102:0/2180525657 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2030009640 msgr2=0x7f203000bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:26:57.340 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:57.323 7f2049d9a700 1 --2- 172.21.2.102:0/2180525657 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2030009640 0x7f203000bb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:57.340 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:57.323 7f2049d9a700 1 -- 172.21.2.102:0/2180525657 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f203000ca70 con 0x7f2030006820 2021-02-10T11:26:57.340 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:57.323 7f204ad9c700 1 -- 172.21.2.102:0/2180525657 reap_dead start 2021-02-10T11:26:57.340 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:57.324 7f2049d9a700 1 --2- 172.21.2.102:0/2180525657 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2030006820 0x7f2030008d40 secure :-1 s=READY pgs=1679 cs=0 l=1 rev1=1 rx=0x7f2040009970 tx=0x7f2040009d10).ready entity=mon.0 client_cookie=f7b4a5ef8b2555f3 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:57.340 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:57.324 7f2049599700 1 -- 172.21.2.102:0/2180525657 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2040006960 con 0x7f2030006820 2021-02-10T11:26:57.341 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:57.324 7f2049599700 1 -- 172.21.2.102:0/2180525657 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20400065d0 con 0x7f2030006820 2021-02-10T11:26:57.341 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:57.324 7f2049599700 1 -- 172.21.2.102:0/2180525657 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2040002030 con 0x7f2030006820 2021-02-10T11:26:57.341 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:57.324 7f204c77f700 1 -- 172.21.2.102:0/2180525657 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2030006820 msgr2=0x7f2030008d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:57.341 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:57.324 7f204c77f700 1 --2- 172.21.2.102:0/2180525657 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2030006820 0x7f2030008d40 secure :-1 s=READY pgs=1679 cs=0 l=1 rev1=1 rx=0x7f2040009970 tx=0x7f2040009d10).stop 2021-02-10T11:26:57.341 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:57.324 7f204c77f700 1 -- 172.21.2.102:0/2180525657 shutdown_connections 2021-02-10T11:26:57.341 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:57.324 7f204c77f700 1 --2- 172.21.2.102:0/2180525657 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2030006820 0x7f2030008d40 unknown :-1 s=CLOSED pgs=1679 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:57.341 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:57.324 7f204c77f700 1 -- 172.21.2.102:0/2180525657 >> 172.21.2.102:0/2180525657 conn(0x7f20440ce450 msgr2=0x7f20440d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:26:57.341 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:57.324 7f204c77f700 1 -- 172.21.2.102:0/2180525657 shutdown_connections 2021-02-10T11:26:57.341 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:57.324 7f204c77f700 1 -- 172.21.2.102:0/2180525657 wait complete. 2021-02-10T11:26:57.341 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:57.339 7f204c77f700 1 Processor -- start 2021-02-10T11:26:57.341 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:57.339 7f204c77f700 1 -- start start 2021-02-10T11:26:57.342 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:57.339 7f204c77f700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2044195650 0x7f20440b5560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:57.343 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:57.339 7f204c77f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f20440b5b80 0x7f20441a2e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:57.343 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:57.339 7f204c77f700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f20440acfa0 0x7f20441a3440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:57.343 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:57.339 7f204c77f700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20440b6010 con 0x7f20440b5b80 2021-02-10T11:26:57.343 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:57.339 7f204c77f700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20441a3a60 con 0x7f2044195650 2021-02-10T11:26:57.343 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:57.339 7f204c77f700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20441a3c40 con 0x7f20440acfa0 2021-02-10T11:26:57.343 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:57.340 7f2049d9a700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f20440b5b80 0x7f20441a2e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:57.343 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:57.340 7f2049d9a700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f20440b5b80 0x7f20441a2e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:59170/0 (socket says 172.21.2.102:59170) 2021-02-10T11:26:57.343 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:57.340 7f2049d9a700 1 -- 172.21.2.102:0/2744358859 learned_addr learned my addr 172.21.2.102:0/2744358859 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:26:57.343 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:57.340 7f2049d9a700 1 -- 172.21.2.102:0/2744358859 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f20440acfa0 msgr2=0x7f20441a3440 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:26:57.343 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:57.340 7f2049d9a700 1 --2- 172.21.2.102:0/2744358859 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f20440acfa0 0x7f20441a3440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:57.343 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:57.340 7f2049d9a700 1 -- 172.21.2.102:0/2744358859 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2044195650 msgr2=0x7f20440b5560 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:26:57.343 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:57.340 7f2049d9a700 1 --2- 172.21.2.102:0/2744358859 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2044195650 0x7f20440b5560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:57.344 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:57.340 7f2049d9a700 1 -- 172.21.2.102:0/2744358859 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20440bcae0 con 0x7f20440b5b80 2021-02-10T11:26:57.344 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:57.340 7f2049d9a700 1 --2- 172.21.2.102:0/2744358859 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f20440b5b80 0x7f20441a2e20 secure :-1 s=READY pgs=1680 cs=0 l=1 rev1=1 rx=0x7f2040006c30 tx=0x7f204000b9e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:57.344 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:57.340 7f2036ffd700 1 -- 172.21.2.102:0/2744358859 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f204000bdd0 con 0x7f20440b5b80 2021-02-10T11:26:57.344 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:57.340 7f2036ffd700 1 -- 172.21.2.102:0/2744358859 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2040006510 con 0x7f20440b5b80 2021-02-10T11:26:57.344 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:57.340 7f2036ffd700 1 -- 172.21.2.102:0/2744358859 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f20400022b0 con 0x7f20440b5b80 2021-02-10T11:26:57.344 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:57.340 7f204c77f700 1 -- 172.21.2.102:0/2744358859 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20440bcae0 con 0x7f20440b5b80 2021-02-10T11:26:57.344 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:57.340 7f204c77f700 1 -- 172.21.2.102:0/2744358859 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20440bcae0 con 0x7f20440b5b80 2021-02-10T11:26:57.344 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:57.341 7f2036ffd700 1 -- 172.21.2.102:0/2744358859 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f2040009fd0 con 0x7f20440b5b80 2021-02-10T11:26:57.344 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:57.341 7f2036ffd700 1 --2- 172.21.2.102:0/2744358859 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f202c0186c0 0x7f202c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:57.344 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:57.341 7f2036ffd700 1 -- 172.21.2.102:0/2744358859 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f20400325d0 con 0x7f20440b5b80 2021-02-10T11:26:57.344 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:57.341 7f204a59b700 1 --2- 172.21.2.102:0/2744358859 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f202c0186c0 0x7f202c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:57.344 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:57.341 7f204a59b700 1 --2- 172.21.2.102:0/2744358859 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f202c0186c0 0x7f202c01abc0 crc :-1 s=READY pgs=763 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:57.371 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:57.372 7f204c77f700 1 -- 172.21.2.102:0/2744358859 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20441a3c40 con 0x7f20440b5b80 2021-02-10T11:26:57.373 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:57.375 7f2036ffd700 1 -- 172.21.2.102:0/2744358859 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f2040006510 con 0x7f20440b5b80 2021-02-10T11:26:57.501 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:57.502 7f204c77f700 1 -- 172.21.2.102:0/2744358859 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f20441a3c40 con 0x7f20440b5b80 2021-02-10T11:26:57.502 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:57.503 7f2036ffd700 1 -- 172.21.2.102:0/2744358859 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 57 v57) v1 ==== 94+0+2098 (secure 0 0 0) 0x7f204000bdd0 con 0x7f20440b5b80 2021-02-10T11:26:57.502 INFO:teuthology.orchestra.run.gibba002.stderr:dumped fsmap epoch 57 2021-02-10T11:26:57.502 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:26:57.502 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":57,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8865,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.102:6835/980649634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6834","nonce":980649634},{"type":"v1","addr":"172.21.2.102:6835","nonce":980649634}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":53}],"filesystems":[{"mdsmap":{"epoch":57,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:26:47.543386","modified":"2021-02-10 11:26:49.552060","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6799},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6799":{"gid":6799,"name":"b","rank":0,"incarnation":54,"state":"up:active","state_seq":3,"addr":"172.21.2.113:6833/2002224946","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6832","nonce":2002224946},{"type":"v1","addr":"172.21.2.113:6833","nonce":2002224946}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2021-02-10T11:26:57.503 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:57.505 7f204c77f700 1 -- 172.21.2.102:0/2744358859 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f202c0186c0 msgr2=0x7f202c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:57.504 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:57.505 7f204c77f700 1 --2- 172.21.2.102:0/2744358859 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f202c0186c0 0x7f202c01abc0 crc :-1 s=READY pgs=763 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:57.504 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:57.505 7f204c77f700 1 -- 172.21.2.102:0/2744358859 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f20440b5b80 msgr2=0x7f20441a2e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:57.504 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:57.505 7f204c77f700 1 --2- 172.21.2.102:0/2744358859 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f20440b5b80 0x7f20441a2e20 secure :-1 s=READY pgs=1680 cs=0 l=1 rev1=1 rx=0x7f2040006c30 tx=0x7f204000b9e0).stop 2021-02-10T11:26:57.504 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:57.505 7f204c77f700 1 -- 172.21.2.102:0/2744358859 shutdown_connections 2021-02-10T11:26:57.504 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:57.505 7f204c77f700 1 --2- 172.21.2.102:0/2744358859 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f20440acfa0 0x7f20441a3440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:57.504 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:57.505 7f204c77f700 1 --2- 172.21.2.102:0/2744358859 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f20440b5b80 0x7f20441a2e20 unknown :-1 s=CLOSED pgs=1680 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:57.504 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:57.505 7f204c77f700 1 --2- 172.21.2.102:0/2744358859 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f202c0186c0 0x7f202c01abc0 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:57.504 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:57.505 7f204c77f700 1 --2- 172.21.2.102:0/2744358859 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2044195650 0x7f20440b5560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:57.504 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:57.505 7f204c77f700 1 -- 172.21.2.102:0/2744358859 >> 172.21.2.102:0/2744358859 conn(0x7f20440b3040 msgr2=0x7f20440c3250 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:26:57.504 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:57.505 7f204c77f700 1 -- 172.21.2.102:0/2744358859 shutdown_connections 2021-02-10T11:26:57.504 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:26:57.505 7f204c77f700 1 -- 172.21.2.102:0/2744358859 wait complete. 2021-02-10T11:26:57.714 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 57, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:26:47.543386', 'modified': '2021-02-10 11:26:49.552060', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6799}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6799': {'gid': 6799, 'name': 'b', 'rank': 0, 'incarnation': 54, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.113:6833/2002224946', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.113:6832', 'nonce': 2002224946}, {'type': 'v1', 'addr': '172.21.2.113:6833', 'nonce': 2002224946}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:26:57.714 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:26:57.714 DEBUG:teuthology.orchestra.run.gibba013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:26:58.113 INFO:teuthology.orchestra.run.gibba013.stdout:{ 2021-02-10T11:26:58.113 INFO:teuthology.orchestra.run.gibba013.stdout: "cluster_fsid": "68db30ea-dbce-4aef-a6df-f01c6fea0513", 2021-02-10T11:26:58.113 INFO:teuthology.orchestra.run.gibba013.stdout: "whoami": 0, 2021-02-10T11:26:58.113 INFO:teuthology.orchestra.run.gibba013.stdout: "id": 6799, 2021-02-10T11:26:58.113 INFO:teuthology.orchestra.run.gibba013.stdout: "want_state": "up:active", 2021-02-10T11:26:58.113 INFO:teuthology.orchestra.run.gibba013.stdout: "state": "up:active", 2021-02-10T11:26:58.114 INFO:teuthology.orchestra.run.gibba013.stdout: "rank_uptime": 10.568335244, 2021-02-10T11:26:58.114 INFO:teuthology.orchestra.run.gibba013.stdout: "mdsmap_epoch": 57, 2021-02-10T11:26:58.114 INFO:teuthology.orchestra.run.gibba013.stdout: "osdmap_epoch": 75, 2021-02-10T11:26:58.114 INFO:teuthology.orchestra.run.gibba013.stdout: "osdmap_epoch_barrier": 75, 2021-02-10T11:26:58.114 INFO:teuthology.orchestra.run.gibba013.stdout: "uptime": 16.206049141000001 2021-02-10T11:26:58.114 INFO:teuthology.orchestra.run.gibba013.stdout:} 2021-02-10T11:26:58.117 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "68db30ea-dbce-4aef-a6df-f01c6fea0513", "whoami": 0, "id": 6799, "want_state": "up:active", "state": "up:active", "rank_uptime": 10.568335244, "mdsmap_epoch": 57, "osdmap_epoch": 75, "osdmap_epoch_barrier": 75, "uptime": 16.206049141000001 } 2021-02-10T11:26:58.117 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2021-02-10T11:26:58.117 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2021-02-10T11:26:58.117 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@gibba002.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2021-02-10T11:26:58.118 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:58.118 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec mkdir -p /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:26:58.454 DEBUG:teuthology.orchestra.run.gibba002:> sudo modprobe fuse 2021-02-10T11:26:58.799 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:26:58.800 DEBUG:teuthology.orchestra.run.gibba002:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:26:59.144 INFO:teuthology.orchestra.run.gibba002.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:26:59.144 INFO:teuthology.orchestra.run.gibba002.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:26:59.145 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:26:59.145 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:59.145 DEBUG:teuthology.orchestra.run.gibba002:> ls /sys/fs/fuse/connections 2021-02-10T11:26:59.480 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2021-02-10T11:26:59.480 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:26:59.483 DEBUG:teuthology.orchestra.run.gibba002:> sudo modprobe fuse 2021-02-10T11:26:59.825 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:26:59.826 DEBUG:teuthology.orchestra.run.gibba002:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:26:59.891 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba002.stderr:ceph-fuse[165230]: starting ceph client 2021-02-10T11:26:59.891 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba002.stderr:2021-02-10 11:26:59.892 7feaefe7af80 -1 init, newargv = 0x559a2a233200 newargc=9 2021-02-10T11:26:59.896 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba002.stderr:ceph-fuse[165230]: starting fuse 2021-02-10T11:27:00.169 INFO:teuthology.orchestra.run.gibba002.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:27:00.169 INFO:teuthology.orchestra.run.gibba002.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:27:00.170 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:27:00.170 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:00.170 DEBUG:teuthology.orchestra.run.gibba002:> ls /sys/fs/fuse/connections 2021-02-10T11:27:00.504 INFO:teuthology.orchestra.run.gibba002.stdout:43 2021-02-10T11:27:00.505 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2021-02-10T11:27:00.505 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:00.505 DEBUG:teuthology.orchestra.run.gibba002:> sudo python3 -c ' 2021-02-10T11:27:00.505 DEBUG:teuthology.orchestra.run.gibba002:> import glob 2021-02-10T11:27:00.505 DEBUG:teuthology.orchestra.run.gibba002:> import re 2021-02-10T11:27:00.505 DEBUG:teuthology.orchestra.run.gibba002:> import os 2021-02-10T11:27:00.506 DEBUG:teuthology.orchestra.run.gibba002:> import subprocess 2021-02-10T11:27:00.506 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:27:00.506 DEBUG:teuthology.orchestra.run.gibba002:> def find_socket(client_name): 2021-02-10T11:27:00.506 DEBUG:teuthology.orchestra.run.gibba002:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2021-02-10T11:27:00.506 DEBUG:teuthology.orchestra.run.gibba002:> files = glob.glob(asok_path) 2021-02-10T11:27:00.506 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:27:00.506 DEBUG:teuthology.orchestra.run.gibba002:> # Given a non-glob path, it better be there 2021-02-10T11:27:00.506 DEBUG:teuthology.orchestra.run.gibba002:> if "*" not in asok_path: 2021-02-10T11:27:00.506 DEBUG:teuthology.orchestra.run.gibba002:> assert(len(files) == 1) 2021-02-10T11:27:00.506 DEBUG:teuthology.orchestra.run.gibba002:> return files[0] 2021-02-10T11:27:00.506 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:27:00.506 DEBUG:teuthology.orchestra.run.gibba002:> for f in files: 2021-02-10T11:27:00.506 DEBUG:teuthology.orchestra.run.gibba002:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2021-02-10T11:27:00.506 DEBUG:teuthology.orchestra.run.gibba002:> if os.path.exists("/proc/{0}".format(pid)): 2021-02-10T11:27:00.507 DEBUG:teuthology.orchestra.run.gibba002:> return f 2021-02-10T11:27:00.507 DEBUG:teuthology.orchestra.run.gibba002:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2021-02-10T11:27:00.507 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:27:00.507 DEBUG:teuthology.orchestra.run.gibba002:> print(find_socket("client.0")) 2021-02-10T11:27:00.507 DEBUG:teuthology.orchestra.run.gibba002:> ' 2021-02-10T11:27:00.868 INFO:teuthology.orchestra.run.gibba002.stdout:/var/run/ceph/ceph-client.0.165230.asok 2021-02-10T11:27:00.870 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.165230.asok 2021-02-10T11:27:00.871 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:00.871 DEBUG:teuthology.orchestra.run.gibba002:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.165230.asok status 2021-02-10T11:27:01.260 INFO:teuthology.orchestra.run.gibba002.stdout:{ 2021-02-10T11:27:01.261 INFO:teuthology.orchestra.run.gibba002.stdout: "metadata": { 2021-02-10T11:27:01.261 INFO:teuthology.orchestra.run.gibba002.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:27:01.261 INFO:teuthology.orchestra.run.gibba002.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:27:01.261 INFO:teuthology.orchestra.run.gibba002.stdout: "entity_id": "0", 2021-02-10T11:27:01.261 INFO:teuthology.orchestra.run.gibba002.stdout: "hostname": "gibba002", 2021-02-10T11:27:01.261 INFO:teuthology.orchestra.run.gibba002.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2021-02-10T11:27:01.261 INFO:teuthology.orchestra.run.gibba002.stdout: "pid": "165230", 2021-02-10T11:27:01.261 INFO:teuthology.orchestra.run.gibba002.stdout: "root": "/" 2021-02-10T11:27:01.261 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:27:01.261 INFO:teuthology.orchestra.run.gibba002.stdout: "dentry_count": 0, 2021-02-10T11:27:01.262 INFO:teuthology.orchestra.run.gibba002.stdout: "dentry_pinned_count": 0, 2021-02-10T11:27:01.262 INFO:teuthology.orchestra.run.gibba002.stdout: "id": 9015, 2021-02-10T11:27:01.262 INFO:teuthology.orchestra.run.gibba002.stdout: "inst": { 2021-02-10T11:27:01.262 INFO:teuthology.orchestra.run.gibba002.stdout: "name": { 2021-02-10T11:27:01.262 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "client", 2021-02-10T11:27:01.262 INFO:teuthology.orchestra.run.gibba002.stdout: "num": 9015 2021-02-10T11:27:01.262 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:27:01.262 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": { 2021-02-10T11:27:01.262 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:27:01.262 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:0", 2021-02-10T11:27:01.262 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 1002640293 2021-02-10T11:27:01.263 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:27:01.263 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:27:01.263 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": { 2021-02-10T11:27:01.263 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:27:01.263 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:0", 2021-02-10T11:27:01.263 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 1002640293 2021-02-10T11:27:01.263 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:27:01.263 INFO:teuthology.orchestra.run.gibba002.stdout: "inst_str": "client.9015 172.21.2.102:0/1002640293", 2021-02-10T11:27:01.263 INFO:teuthology.orchestra.run.gibba002.stdout: "addr_str": "172.21.2.102:0/1002640293", 2021-02-10T11:27:01.263 INFO:teuthology.orchestra.run.gibba002.stdout: "inode_count": 1, 2021-02-10T11:27:01.263 INFO:teuthology.orchestra.run.gibba002.stdout: "mds_epoch": 57, 2021-02-10T11:27:01.263 INFO:teuthology.orchestra.run.gibba002.stdout: "osd_epoch": 75, 2021-02-10T11:27:01.264 INFO:teuthology.orchestra.run.gibba002.stdout: "osd_epoch_barrier": 0, 2021-02-10T11:27:01.264 INFO:teuthology.orchestra.run.gibba002.stdout: "blacklisted": false 2021-02-10T11:27:01.264 INFO:teuthology.orchestra.run.gibba002.stdout:} 2021-02-10T11:27:01.265 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:01.265 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:27:01.265 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:27:01.600 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:27:01.601 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:27:01.601 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:01.601 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:27:01.950 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:27:02.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:02.361 7fb4c8cd5700 1 Processor -- start 2021-02-10T11:27:02.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:02.361 7fb4c8cd5700 1 -- start start 2021-02-10T11:27:02.377 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:02.361 7fb4c8cd5700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fb4c40d1c70 0x7fb4c40f20a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:02.377 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:02.361 7fb4c8cd5700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4c40f0b10 con 0x7fb4c40d0340 2021-02-10T11:27:02.377 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:02.361 7fb4c8cd5700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4c41b0be0 con 0x7fb4c40d1870 2021-02-10T11:27:02.377 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:02.361 7fb4c8cd5700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4c40e5ed0 con 0x7fb4c40d1c70 2021-02-10T11:27:02.377 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:02.362 7fb4c22e7700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fb4c40d0340 0x7fb4c40ef640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:51410/0 (socket says 172.21.2.102:51410) 2021-02-10T11:27:02.377 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:02.362 7fb4c22e7700 1 -- 172.21.2.102:0/2302535793 learned_addr learned my addr 172.21.2.102:0/2302535793 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:27:02.377 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:02.362 7fb4c32e9700 1 --2- 172.21.2.102:0/2302535793 >> v2:172.21.2.102:3300/0 conn(0x7fb4c40d1c70 0x7fb4c40f20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:02.377 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:02.362 7fb4c32e9700 1 -- 172.21.2.102:0/2302535793 >> v1:172.21.2.113:6789/0 conn(0x7fb4c40d1870 legacy=0x7fb4c41b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:27:02.377 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:02.362 7fb4c32e9700 1 -- 172.21.2.102:0/2302535793 >> v1:172.21.2.102:6789/0 conn(0x7fb4c40d0340 legacy=0x7fb4c40ef640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:02.377 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:02.362 7fb4c32e9700 1 -- 172.21.2.102:0/2302535793 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4c40efe30 con 0x7fb4c40d1c70 2021-02-10T11:27:02.377 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:02.362 7fb4c32e9700 1 --2- 172.21.2.102:0/2302535793 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb4c40d1c70 0x7fb4c40f20a0 secure :-1 s=READY pgs=1684 cs=0 l=1 rev1=1 rx=0x7fb4bc006e30 tx=0x7fb4bc009de0).ready entity=mon.0 client_cookie=4443fa6304c8563d server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:02.377 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:02.363 7fb4c1ae6700 1 -- 172.21.2.102:0/2302535793 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb4bc00b300 con 0x7fb4c40d1c70 2021-02-10T11:27:02.378 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:02.363 7fb4c1ae6700 1 -- 172.21.2.102:0/2302535793 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4bc00b680 con 0x7fb4c40d1c70 2021-02-10T11:27:02.378 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:02.363 7fb4c1ae6700 1 -- 172.21.2.102:0/2302535793 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb4bc00bba0 con 0x7fb4c40d1c70 2021-02-10T11:27:02.378 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:02.363 7fb4c8cd5700 1 -- 172.21.2.102:0/2302535793 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb4c40d1c70 msgr2=0x7fb4c40f20a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:02.378 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:02.363 7fb4c8cd5700 1 --2- 172.21.2.102:0/2302535793 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb4c40d1c70 0x7fb4c40f20a0 secure :-1 s=READY pgs=1684 cs=0 l=1 rev1=1 rx=0x7fb4bc006e30 tx=0x7fb4bc009de0).stop 2021-02-10T11:27:02.378 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:02.363 7fb4c8cd5700 1 -- 172.21.2.102:0/2302535793 shutdown_connections 2021-02-10T11:27:02.378 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:02.363 7fb4c8cd5700 1 --2- 172.21.2.102:0/2302535793 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb4c40d1c70 0x7fb4c40f20a0 unknown :-1 s=CLOSED pgs=1684 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:02.378 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:02.363 7fb4c8cd5700 1 -- 172.21.2.102:0/2302535793 >> 172.21.2.102:0/2302535793 conn(0x7fb4c40ce450 msgr2=0x7fb4c40d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:02.378 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:02.364 7fb4c8cd5700 1 -- 172.21.2.102:0/2302535793 shutdown_connections 2021-02-10T11:27:02.378 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:02.364 7fb4c8cd5700 1 -- 172.21.2.102:0/2302535793 wait complete. 2021-02-10T11:27:02.380 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:02.378 7fb4c8cd5700 1 Processor -- start 2021-02-10T11:27:02.380 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:02.378 7fb4c8cd5700 1 -- start start 2021-02-10T11:27:02.380 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:02.378 7fb4c8cd5700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb4c4092010 0x7fb4c40c33c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:02.380 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:02.378 7fb4c8cd5700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb4c40c39e0 0x7fb4c40af040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:02.380 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:02.378 7fb4c8cd5700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb4c40af660 0x7fb4c40afa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:02.380 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:02.378 7fb4c8cd5700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4c40b0500 con 0x7fb4c4092010 2021-02-10T11:27:02.380 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:02.378 7fb4c8cd5700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4c40b0080 con 0x7fb4c40c39e0 2021-02-10T11:27:02.380 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:02.378 7fb4c8cd5700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4c40b02a0 con 0x7fb4c40af660 2021-02-10T11:27:02.380 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:02.379 7fb4c2ae8700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb4c4092010 0x7fb4c40c33c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:02.380 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:02.379 7fb4c2ae8700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb4c4092010 0x7fb4c40c33c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:59190/0 (socket says 172.21.2.102:59190) 2021-02-10T11:27:02.380 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:02.379 7fb4c2ae8700 1 -- 172.21.2.102:0/909894815 learned_addr learned my addr 172.21.2.102:0/909894815 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:27:02.381 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:02.379 7fb4c2ae8700 1 -- 172.21.2.102:0/909894815 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb4c40af660 msgr2=0x7fb4c40afa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:02.381 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:02.379 7fb4c2ae8700 1 --2- 172.21.2.102:0/909894815 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb4c40af660 0x7fb4c40afa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:02.381 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:02.379 7fb4c2ae8700 1 -- 172.21.2.102:0/909894815 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb4c40c39e0 msgr2=0x7fb4c40af040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:02.381 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:02.379 7fb4c2ae8700 1 --2- 172.21.2.102:0/909894815 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb4c40c39e0 0x7fb4c40af040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:02.381 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:02.379 7fb4c2ae8700 1 -- 172.21.2.102:0/909894815 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4c40b0a70 con 0x7fb4c4092010 2021-02-10T11:27:02.381 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:02.379 7fb4c2ae8700 1 --2- 172.21.2.102:0/909894815 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb4c4092010 0x7fb4c40c33c0 secure :-1 s=READY pgs=1685 cs=0 l=1 rev1=1 rx=0x7fb4b400b5b0 tx=0x7fb4b400b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:02.381 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:02.380 7fb4b37fe700 1 -- 172.21.2.102:0/909894815 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb4b400bd40 con 0x7fb4c4092010 2021-02-10T11:27:02.381 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:02.380 7fb4b37fe700 1 -- 172.21.2.102:0/909894815 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4b4000ce0 con 0x7fb4c4092010 2021-02-10T11:27:02.381 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:02.380 7fb4c8cd5700 1 -- 172.21.2.102:0/909894815 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4c40b0a70 con 0x7fb4c4092010 2021-02-10T11:27:02.381 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:02.380 7fb4c8cd5700 1 -- 172.21.2.102:0/909894815 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4c40b1000 con 0x7fb4c4092010 2021-02-10T11:27:02.381 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:02.380 7fb4b37fe700 1 -- 172.21.2.102:0/909894815 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb4b40054a0 con 0x7fb4c4092010 2021-02-10T11:27:02.381 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:02.380 7fb4b37fe700 1 -- 172.21.2.102:0/909894815 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fb4b4006030 con 0x7fb4c4092010 2021-02-10T11:27:02.381 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:02.380 7fb4b37fe700 1 --2- 172.21.2.102:0/909894815 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb4a8018730 0x7fb4a801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:02.381 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:02.381 7fb4b37fe700 1 -- 172.21.2.102:0/909894815 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fb4b40327c0 con 0x7fb4c4092010 2021-02-10T11:27:02.382 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:02.381 7fb4c22e7700 1 --2- 172.21.2.102:0/909894815 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb4a8018730 0x7fb4a801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:02.382 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:02.381 7fb4c22e7700 1 --2- 172.21.2.102:0/909894815 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb4a8018730 0x7fb4a801ac10 crc :-1 s=READY pgs=764 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:02.410 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:02.411 7fb4c8cd5700 1 -- 172.21.2.102:0/909894815 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4c40e7df0 con 0x7fb4c4092010 2021-02-10T11:27:02.413 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:02.414 7fb4b37fe700 1 -- 172.21.2.102:0/909894815 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fb4b4006240 con 0x7fb4c4092010 2021-02-10T11:27:02.536 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:02.537 7fb4c8cd5700 1 -- 172.21.2.102:0/909894815 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb4c40ce450 con 0x7fb4c4092010 2021-02-10T11:27:02.537 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:02.538 7fb4b37fe700 1 -- 172.21.2.102:0/909894815 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v75) v1 ==== 74+0+13996 (secure 0 0 0) 0x7fb4b4006030 con 0x7fb4c4092010 2021-02-10T11:27:02.537 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:27:02.537 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":75,"fsid":"68db30ea-dbce-4aef-a6df-f01c6fea0513","created":"2021-02-10 11:03:06.331557","modified":"2021-02-10 11:26:48.545650","last_up_change":"2021-02-10 11:03:32.690562","last_in_change":"2021-02-10 11:03:32.690562","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:03:37.322519","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:26:45.515299","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"74","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":15,"pool_name":"cephfs_data","create_time":"2021-02-10 11:26:46.587944","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"be691b4d-a039-408a-af80-9d53f02b6f1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6818","nonce":36088},{"type":"v1","addr":"172.21.2.102:6820","nonce":36088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6822","nonce":36088},{"type":"v1","addr":"172.21.2.102:6824","nonce":36088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6830","nonce":36088},{"type":"v1","addr":"172.21.2.102:6832","nonce":36088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6826","nonce":36088},{"type":"v1","addr":"172.21.2.102:6828","nonce":36088}]},"public_addr":"172.21.2.102:6820/36088","cluster_addr":"172.21.2.102:6824/36088","heartbeat_back_addr":"172.21.2.102:6832/36088","heartbeat_front_addr":"172.21.2.102:6828/36088","state":["exists","up"]},{"osd":1,"uuid":"64e3ccc8-9c69-4d7c-865e-af8e83822635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6819","nonce":36087},{"type":"v1","addr":"172.21.2.102:6821","nonce":36087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6823","nonce":36087},{"type":"v1","addr":"172.21.2.102:6825","nonce":36087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6831","nonce":36087},{"type":"v1","addr":"172.21.2.102:6833","nonce":36087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6827","nonce":36087},{"type":"v1","addr":"172.21.2.102:6829","nonce":36087}]},"public_addr":"172.21.2.102:6821/36087","cluster_addr":"172.21.2.102:6825/36087","heartbeat_back_addr":"172.21.2.102:6833/36087","heartbeat_front_addr":"172.21.2.102:6829/36087","state":["exists","up"]},{"osd":2,"uuid":"153700af-747f-481f-bdf4-d6072d3a842f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6810","nonce":36089},{"type":"v1","addr":"172.21.2.102:6811","nonce":36089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6812","nonce":36089},{"type":"v1","addr":"172.21.2.102:6813","nonce":36089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6816","nonce":36089},{"type":"v1","addr":"172.21.2.102:6817","nonce":36089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6814","nonce":36089},{"type":"v1","addr":"172.21.2.102:6815","nonce":36089}]},"public_addr":"172.21.2.102:6811/36089","cluster_addr":"172.21.2.102:6813/36089","heartbeat_back_addr":"172.21.2.102:6817/36089","heartbeat_front_addr":"172.21.2.102:6815/36089","state":["exists","up"]},{"osd":3,"uuid":"cbc678f4-83a9-45e6-8830-2f7c232b1ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6802","nonce":36090},{"type":"v1","addr":"172.21.2.102:6803","nonce":36090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6804","nonce":36090},{"type":"v1","addr":"172.21.2.102:6805","nonce":36090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6808","nonce":36090},{"type":"v1","addr":"172.21.2.102:6809","nonce":36090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6806","nonce":36090},{"type":"v1","addr":"172.21.2.102:6807","nonce":36090}]},"public_addr":"172.21.2.102:6803/36090","cluster_addr":"172.21.2.102:6805/36090","heartbeat_back_addr":"172.21.2.102:6809/36090","heartbeat_front_addr":"172.21.2.102:6807/36090","state":["exists","up"]},{"osd":4,"uuid":"c01e4008-b258-4f10-afd2-9e7f37214351","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6808","nonce":36368},{"type":"v1","addr":"172.21.2.113:6811","nonce":36368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6814","nonce":36368},{"type":"v1","addr":"172.21.2.113:6818","nonce":36368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6826","nonce":36368},{"type":"v1","addr":"172.21.2.113:6828","nonce":36368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6820","nonce":36368},{"type":"v1","addr":"172.21.2.113:6824","nonce":36368}]},"public_addr":"172.21.2.113:6811/36368","cluster_addr":"172.21.2.113:6818/36368","heartbeat_back_addr":"172.21.2.113:6828/36368","heartbeat_front_addr":"172.21.2.113:6824/36368","state":["exists","up"]},{"osd":5,"uuid":"551f6007-daf8-4a98-8ee0-c2b16bcccb5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6802","nonce":36366},{"type":"v1","addr":"172.21.2.113:6805","nonce":36366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6807","nonce":36366},{"type":"v1","addr":"172.21.2.113:6810","nonce":36366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6819","nonce":36366},{"type":"v1","addr":"172.21.2.113:6822","nonce":36366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6812","nonce":36366},{"type":"v1","addr":"172.21.2.113:6815","nonce":36366}]},"public_addr":"172.21.2.113:6805/36366","cluster_addr":"172.21.2.113:6810/36366","heartbeat_back_addr":"172.21.2.113:6822/36366","heartbeat_front_addr":"172.21.2.113:6815/36366","state":["exists","up"]},{"osd":6,"uuid":"bf67c040-0e50-48be-b102-c3943bc9ebef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6800","nonce":36367},{"type":"v1","addr":"172.21.2.113:6801","nonce":36367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6803","nonce":36367},{"type":"v1","addr":"172.21.2.113:6804","nonce":36367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6813","nonce":36367},{"type":"v1","addr":"172.21.2.113:6816","nonce":36367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6806","nonce":36367},{"type":"v1","addr":"172.21.2.113:6809","nonce":36367}]},"public_addr":"172.21.2.113:6801/36367","cluster_addr":"172.21.2.113:6804/36367","heartbeat_back_addr":"172.21.2.113:6816/36367","heartbeat_front_addr":"172.21.2.113:6809/36367","state":["exists","up"]},{"osd":7,"uuid":"f5461f80-018d-428f-be89-2eb2bcbe7bb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6817","nonce":36365},{"type":"v1","addr":"172.21.2.113:6821","nonce":36365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6823","nonce":36365},{"type":"v1","addr":"172.21.2.113:6825","nonce":36365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6830","nonce":36365},{"type":"v1","addr":"172.21.2.113:6831","nonce":36365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6827","nonce":36365},{"type":"v1","addr":"172.21.2.113:6829","nonce":36365}]},"public_addr":"172.21.2.113:6821/36365","cluster_addr":"172.21.2.113:6825/36365","heartbeat_back_addr":"172.21.2.113:6831/36365","heartbeat_front_addr":"172.21.2.113:6829/36365","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:27:02.538 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:02.540 7fb4c8cd5700 1 -- 172.21.2.102:0/909894815 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb4a8018730 msgr2=0x7fb4a801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:02.538 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:02.540 7fb4c8cd5700 1 --2- 172.21.2.102:0/909894815 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb4a8018730 0x7fb4a801ac10 crc :-1 s=READY pgs=764 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:02.539 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:02.540 7fb4c8cd5700 1 -- 172.21.2.102:0/909894815 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb4c4092010 msgr2=0x7fb4c40c33c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:02.539 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:02.540 7fb4c8cd5700 1 --2- 172.21.2.102:0/909894815 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb4c4092010 0x7fb4c40c33c0 secure :-1 s=READY pgs=1685 cs=0 l=1 rev1=1 rx=0x7fb4b400b5b0 tx=0x7fb4b400b950).stop 2021-02-10T11:27:02.539 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:02.540 7fb4c8cd5700 1 -- 172.21.2.102:0/909894815 shutdown_connections 2021-02-10T11:27:02.539 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:02.540 7fb4c8cd5700 1 --2- 172.21.2.102:0/909894815 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb4c40af660 0x7fb4c40afa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:02.539 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:02.540 7fb4c8cd5700 1 --2- 172.21.2.102:0/909894815 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb4a8018730 0x7fb4a801ac10 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:02.539 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:02.540 7fb4c8cd5700 1 --2- 172.21.2.102:0/909894815 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb4c40c39e0 0x7fb4c40af040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:02.539 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:02.540 7fb4c8cd5700 1 --2- 172.21.2.102:0/909894815 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb4c4092010 0x7fb4c40c33c0 unknown :-1 s=CLOSED pgs=1685 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:02.540 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:02.540 7fb4c8cd5700 1 -- 172.21.2.102:0/909894815 >> 172.21.2.102:0/909894815 conn(0x7fb4c40a0030 msgr2=0x7fb4c40f22b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:02.540 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:02.540 7fb4c8cd5700 1 -- 172.21.2.102:0/909894815 shutdown_connections 2021-02-10T11:27:02.540 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:02.540 7fb4c8cd5700 1 -- 172.21.2.102:0/909894815 wait complete. 2021-02-10T11:27:02.548 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:03:37.322519', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 14, 'pool_name': 'cephfs_metadata', 'create_time': '2021-02-10 11:26:45.515299', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '74', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 15, 'pool_name': 'cephfs_data', 'create_time': '2021-02-10 11:26:46.587944', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '75', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-02-10T11:27:02.549 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:27:02.979 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:02.963 7f5df0ac9700 1 Processor -- start 2021-02-10T11:27:02.979 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:02.963 7f5df0ac9700 1 -- start start 2021-02-10T11:27:02.979 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:02.964 7f5df0ac9700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f5dec0d1c70 0x7f5dec0cef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:02.979 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:02.964 7f5df0ac9700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f5dec0cf580 0x7f5dec0cf980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:02.979 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:02.964 7f5df0ac9700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5dec0fe1c0 con 0x7f5dec0d1870 2021-02-10T11:27:02.979 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:02.964 7f5df0ac9700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5dec1b0cb0 con 0x7f5dec0d1c70 2021-02-10T11:27:02.979 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:02.964 7f5df0ac9700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5dec109a60 con 0x7f5dec0cf580 2021-02-10T11:27:02.979 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:02.964 7f5dea05d700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f5dec0d1c70 0x7f5dec0cef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:02.980 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:02.964 7f5dea05d700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f5dec0d1c70 0x7f5dec0cef60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:59198/0 (socket says 172.21.2.102:59198) 2021-02-10T11:27:02.980 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:02.964 7f5dea05d700 1 -- 172.21.2.102:0/1320892131 learned_addr learned my addr 172.21.2.102:0/1320892131 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:27:02.980 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:02.964 7f5deb05f700 1 --2- 172.21.2.102:0/1320892131 >> v2:172.21.2.113:3300/0 conn(0x7f5dec0cf580 0x7f5dec0cf980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:02.980 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:02.964 7f5dea05d700 1 -- 172.21.2.102:0/1320892131 >> v2:172.21.2.113:3300/0 conn(0x7f5dec0cf580 msgr2=0x7f5dec0cf980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:27:02.980 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:02.964 7f5dea05d700 1 --2- 172.21.2.102:0/1320892131 >> v2:172.21.2.113:3300/0 conn(0x7f5dec0cf580 0x7f5dec0cf980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:02.980 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:02.964 7f5dea05d700 1 -- 172.21.2.102:0/1320892131 >> v1:172.21.2.113:6789/0 conn(0x7f5dec0d1870 legacy=0x7f5dec1b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:27:02.980 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:02.964 7f5dea05d700 1 -- 172.21.2.102:0/1320892131 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dec088070 con 0x7f5dec0d1c70 2021-02-10T11:27:02.980 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:02.965 7f5dea05d700 1 --2- 172.21.2.102:0/1320892131 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5dec0d1c70 0x7f5dec0cef60 secure :-1 s=READY pgs=1686 cs=0 l=1 rev1=1 rx=0x7f5ddc006700 tx=0x7f5ddc009b80).ready entity=mon.0 client_cookie=b9ba69b393fc3416 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:02.980 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:02.965 7f5de985c700 1 -- 172.21.2.102:0/1320892131 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5ddc00c0c0 con 0x7f5dec0d1c70 2021-02-10T11:27:02.980 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:02.965 7f5de985c700 1 -- 172.21.2.102:0/1320892131 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ddc00c440 con 0x7f5dec0d1c70 2021-02-10T11:27:02.980 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:02.965 7f5de985c700 1 -- 172.21.2.102:0/1320892131 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5ddc00c8a0 con 0x7f5dec0d1c70 2021-02-10T11:27:02.980 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:02.965 7f5df0ac9700 1 -- 172.21.2.102:0/1320892131 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5dec0d1c70 msgr2=0x7f5dec0cef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:02.980 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:02.965 7f5df0ac9700 1 --2- 172.21.2.102:0/1320892131 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5dec0d1c70 0x7f5dec0cef60 secure :-1 s=READY pgs=1686 cs=0 l=1 rev1=1 rx=0x7f5ddc006700 tx=0x7f5ddc009b80).stop 2021-02-10T11:27:02.981 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:02.966 7f5df0ac9700 1 -- 172.21.2.102:0/1320892131 shutdown_connections 2021-02-10T11:27:02.981 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:02.966 7f5df0ac9700 1 --2- 172.21.2.102:0/1320892131 >> v2:172.21.2.113:3300/0 conn(0x7f5dec0cf580 0x7f5dec0cf980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:02.981 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:02.966 7f5df0ac9700 1 --2- 172.21.2.102:0/1320892131 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5dec0d1c70 0x7f5dec0cef60 unknown :-1 s=CLOSED pgs=1686 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:02.981 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:02.966 7f5df0ac9700 1 -- 172.21.2.102:0/1320892131 >> 172.21.2.102:0/1320892131 conn(0x7f5dec0ce420 msgr2=0x7f5dec0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:02.981 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:02.966 7f5df0ac9700 1 -- 172.21.2.102:0/1320892131 shutdown_connections 2021-02-10T11:27:02.981 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:02.966 7f5df0ac9700 1 -- 172.21.2.102:0/1320892131 wait complete. 2021-02-10T11:27:02.982 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:02.980 7f5df0ac9700 1 Processor -- start 2021-02-10T11:27:02.982 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:02.980 7f5df0ac9700 1 -- start start 2021-02-10T11:27:02.982 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:02.981 7f5df0ac9700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5dec09bcc0 0x7f5dec09c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:02.983 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:02.981 7f5df0ac9700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5dec0a0820 0x7f5dec0a0c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:02.983 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:02.981 7f5df0ac9700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5dec0a1240 0x7f5dec0858d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:02.983 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:02.981 7f5df0ac9700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dec0a66b0 con 0x7f5dec0a1240 2021-02-10T11:27:02.983 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:02.981 7f5df0ac9700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dec1a49b0 con 0x7f5dec09bcc0 2021-02-10T11:27:02.983 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:02.981 7f5df0ac9700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5dec1a4b90 con 0x7f5dec0a0820 2021-02-10T11:27:02.983 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:02.981 7f5deb05f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5dec0a1240 0x7f5dec0858d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:02.983 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:02.981 7f5deb05f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5dec0a1240 0x7f5dec0858d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:59208/0 (socket says 172.21.2.102:59208) 2021-02-10T11:27:02.983 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:02.981 7f5deb05f700 1 -- 172.21.2.102:0/3689223145 learned_addr learned my addr 172.21.2.102:0/3689223145 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:27:02.983 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:02.981 7f5deb05f700 1 -- 172.21.2.102:0/3689223145 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5dec0a0820 msgr2=0x7f5dec0a0c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:02.983 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:02.981 7f5deb05f700 1 --2- 172.21.2.102:0/3689223145 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5dec0a0820 0x7f5dec0a0c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:02.983 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:02.981 7f5deb05f700 1 -- 172.21.2.102:0/3689223145 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5dec09bcc0 msgr2=0x7f5dec09c0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:02.983 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:02.981 7f5deb05f700 1 --2- 172.21.2.102:0/3689223145 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5dec09bcc0 0x7f5dec09c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:02.983 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:02.981 7f5deb05f700 1 -- 172.21.2.102:0/3689223145 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dec0a1640 con 0x7f5dec0a1240 2021-02-10T11:27:02.984 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:02.982 7f5deb05f700 1 --2- 172.21.2.102:0/3689223145 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5dec0a1240 0x7f5dec0858d0 secure :-1 s=READY pgs=1687 cs=0 l=1 rev1=1 rx=0x7f5de40099a0 tx=0x7f5de400bc70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:02.984 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:02.982 7f5de37fe700 1 -- 172.21.2.102:0/3689223145 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5de400bdb0 con 0x7f5dec0a1240 2021-02-10T11:27:02.984 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:02.982 7f5de37fe700 1 -- 172.21.2.102:0/3689223145 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5de4003030 con 0x7f5dec0a1240 2021-02-10T11:27:02.984 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:02.982 7f5de37fe700 1 -- 172.21.2.102:0/3689223145 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5de40013e0 con 0x7f5dec0a1240 2021-02-10T11:27:02.984 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:02.982 7f5df0ac9700 1 -- 172.21.2.102:0/3689223145 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5dec0a1640 con 0x7f5dec0a1240 2021-02-10T11:27:02.984 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:02.982 7f5df0ac9700 1 -- 172.21.2.102:0/3689223145 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5dec0861e0 con 0x7f5dec0a1240 2021-02-10T11:27:02.984 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:02.983 7f5de37fe700 1 -- 172.21.2.102:0/3689223145 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f5de40011f0 con 0x7f5dec0a1240 2021-02-10T11:27:02.984 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:02.983 7f5de37fe700 1 --2- 172.21.2.102:0/3689223145 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5dd00186c0 0x7f5dd001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:02.984 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:02.983 7f5de37fe700 1 -- 172.21.2.102:0/3689223145 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f5de4032900 con 0x7f5dec0a1240 2021-02-10T11:27:02.984 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:02.983 7f5dea85e700 1 --2- 172.21.2.102:0/3689223145 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5dd00186c0 0x7f5dd001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:02.984 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:02.983 7f5dea85e700 1 --2- 172.21.2.102:0/3689223145 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5dd00186c0 0x7f5dd001abc0 crc :-1 s=READY pgs=765 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:03.013 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:03.014 7f5df0ac9700 1 -- 172.21.2.102:0/3689223145 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5dec086a70 con 0x7f5dec0a1240 2021-02-10T11:27:03.016 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:03.017 7f5de37fe700 1 -- 172.21.2.102:0/3689223145 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f5de4000f90 con 0x7f5dec0a1240 2021-02-10T11:27:03.144 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:03.145 7f5df0ac9700 1 -- 172.21.2.102:0/3689223145 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f5dec086a70 con 0x7f5dec0a1240 2021-02-10T11:27:03.144 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:03.146 7f5de37fe700 1 -- 172.21.2.102:0/3689223145 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v75) v1 ==== 92+0+10 (secure 0 0 0) 0x7f5de401f6a0 con 0x7f5dec0a1240 2021-02-10T11:27:03.145 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:27:03.147 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:03.147 7f5df0ac9700 1 -- 172.21.2.102:0/3689223145 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5dd00186c0 msgr2=0x7f5dd001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:03.147 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:03.147 7f5df0ac9700 1 --2- 172.21.2.102:0/3689223145 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5dd00186c0 0x7f5dd001abc0 crc :-1 s=READY pgs=765 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:03.147 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:03.147 7f5df0ac9700 1 -- 172.21.2.102:0/3689223145 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5dec0a1240 msgr2=0x7f5dec0858d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:03.147 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:03.147 7f5df0ac9700 1 --2- 172.21.2.102:0/3689223145 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5dec0a1240 0x7f5dec0858d0 secure :-1 s=READY pgs=1687 cs=0 l=1 rev1=1 rx=0x7f5de40099a0 tx=0x7f5de400bc70).stop 2021-02-10T11:27:03.147 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:03.148 7f5df0ac9700 1 -- 172.21.2.102:0/3689223145 shutdown_connections 2021-02-10T11:27:03.147 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:03.148 7f5df0ac9700 1 --2- 172.21.2.102:0/3689223145 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5dec0a1240 0x7f5dec0858d0 unknown :-1 s=CLOSED pgs=1687 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:03.147 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:03.148 7f5df0ac9700 1 --2- 172.21.2.102:0/3689223145 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5dec0a0820 0x7f5dec0a0c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:03.147 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:03.148 7f5df0ac9700 1 --2- 172.21.2.102:0/3689223145 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5dd00186c0 0x7f5dd001abc0 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:03.147 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:03.148 7f5df0ac9700 1 --2- 172.21.2.102:0/3689223145 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5dec09bcc0 0x7f5dec09c0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:03.148 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:03.148 7f5df0ac9700 1 -- 172.21.2.102:0/3689223145 >> 172.21.2.102:0/3689223145 conn(0x7f5dec0bef00 msgr2=0x7f5dec09d1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:03.148 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:03.148 7f5df0ac9700 1 -- 172.21.2.102:0/3689223145 shutdown_connections 2021-02-10T11:27:03.148 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:03.148 7f5df0ac9700 1 -- 172.21.2.102:0/3689223145 wait complete. 2021-02-10T11:27:03.155 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-02-10T11:27:03.593 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:03.572 7f1a91f77700 1 Processor -- start 2021-02-10T11:27:03.593 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:03.572 7f1a91f77700 1 -- start start 2021-02-10T11:27:03.593 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:03.572 7f1a91f77700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f1a8c0d1870 0x7f1a8c1b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:03.593 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:03.572 7f1a91f77700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f1a8c1b0ae0 0x7f1a8c108940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:03.593 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:03.572 7f1a91f77700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a8c0fe1c0 con 0x7f1a8c0d1c70 2021-02-10T11:27:03.594 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:03.572 7f1a91f77700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a8c109a60 con 0x7f1a8c1b0ae0 2021-02-10T11:27:03.594 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:03.572 7f1a91f77700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a8c109750 con 0x7f1a8c0d1870 2021-02-10T11:27:03.594 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:03.573 7f1a8affd700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f1a8c1b0ae0 0x7f1a8c108940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:03.594 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:03.573 7f1a8affd700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f1a8c1b0ae0 0x7f1a8c108940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:59212/0 (socket says 172.21.2.102:59212) 2021-02-10T11:27:03.594 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:03.573 7f1a8affd700 1 -- 172.21.2.102:0/2764371048 learned_addr learned my addr 172.21.2.102:0/2764371048 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:27:03.594 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:03.573 7f1a8affd700 1 -- 172.21.2.102:0/2764371048 >> v2:172.21.2.113:3300/0 conn(0x7f1a8c0d1870 msgr2=0x7f1a8c1b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:27:03.595 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:03.573 7f1a8affd700 1 --2- 172.21.2.102:0/2764371048 >> v2:172.21.2.113:3300/0 conn(0x7f1a8c0d1870 0x7f1a8c1b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:03.595 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:03.573 7f1a8affd700 1 -- 172.21.2.102:0/2764371048 >> v1:172.21.2.113:6789/0 conn(0x7f1a8c0d1c70 legacy=0x7f1a8c108f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:27:03.595 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:03.573 7f1a8affd700 1 -- 172.21.2.102:0/2764371048 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a8c088070 con 0x7f1a8c1b0ae0 2021-02-10T11:27:03.595 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:03.573 7f1a8affd700 1 --2- 172.21.2.102:0/2764371048 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1a8c1b0ae0 0x7f1a8c108940 secure :-1 s=READY pgs=1688 cs=0 l=1 rev1=1 rx=0x7f1a80006700 tx=0x7f1a80009b80).ready entity=mon.0 client_cookie=3e62eebb28c3b7fe server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:03.595 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:03.574 7f1a8a7fc700 1 -- 172.21.2.102:0/2764371048 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1a8000c0c0 con 0x7f1a8c1b0ae0 2021-02-10T11:27:03.595 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:03.574 7f1a8a7fc700 1 -- 172.21.2.102:0/2764371048 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a8000c440 con 0x7f1a8c1b0ae0 2021-02-10T11:27:03.595 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:03.574 7f1a8a7fc700 1 -- 172.21.2.102:0/2764371048 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1a8000c960 con 0x7f1a8c1b0ae0 2021-02-10T11:27:03.595 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:03.574 7f1a91f77700 1 -- 172.21.2.102:0/2764371048 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1a8c1b0ae0 msgr2=0x7f1a8c108940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:03.595 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:03.574 7f1a91f77700 1 --2- 172.21.2.102:0/2764371048 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1a8c1b0ae0 0x7f1a8c108940 secure :-1 s=READY pgs=1688 cs=0 l=1 rev1=1 rx=0x7f1a80006700 tx=0x7f1a80009b80).stop 2021-02-10T11:27:03.596 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:03.574 7f1a91f77700 1 -- 172.21.2.102:0/2764371048 shutdown_connections 2021-02-10T11:27:03.596 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:03.574 7f1a91f77700 1 --2- 172.21.2.102:0/2764371048 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1a8c1b0ae0 0x7f1a8c108940 unknown :-1 s=CLOSED pgs=1688 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:03.596 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:03.574 7f1a91f77700 1 --2- 172.21.2.102:0/2764371048 >> v2:172.21.2.113:3300/0 conn(0x7f1a8c0d1870 0x7f1a8c1b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:03.596 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:03.574 7f1a91f77700 1 -- 172.21.2.102:0/2764371048 >> 172.21.2.102:0/2764371048 conn(0x7f1a8c0ce420 msgr2=0x7f1a8c0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:03.596 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:03.574 7f1a91f77700 1 -- 172.21.2.102:0/2764371048 shutdown_connections 2021-02-10T11:27:03.596 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:03.574 7f1a91f77700 1 -- 172.21.2.102:0/2764371048 wait complete. 2021-02-10T11:27:03.596 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:03.589 7f1a91f77700 1 Processor -- start 2021-02-10T11:27:03.596 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:03.589 7f1a91f77700 1 -- start start 2021-02-10T11:27:03.596 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:03.589 7f1a91f77700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1a8c19e420 0x7f1a8c08a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:03.596 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:03.589 7f1a91f77700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1a8c08afc0 0x7f1a8c08b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:03.596 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:03.589 7f1a91f77700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1a8c08b9e0 0x7f1a8c0b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:03.597 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:03.589 7f1a91f77700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a8c0b54d0 con 0x7f1a8c08afc0 2021-02-10T11:27:03.597 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:03.589 7f1a91f77700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a8c0b56f0 con 0x7f1a8c08b9e0 2021-02-10T11:27:03.597 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:03.589 7f1a91f77700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a8c0b5910 con 0x7f1a8c19e420 2021-02-10T11:27:03.597 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:03.590 7f1a8affd700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1a8c08afc0 0x7f1a8c08b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:03.597 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:03.590 7f1a8affd700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1a8c08afc0 0x7f1a8c08b3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:59218/0 (socket says 172.21.2.102:59218) 2021-02-10T11:27:03.597 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:03.590 7f1a8affd700 1 -- 172.21.2.102:0/1775628237 learned_addr learned my addr 172.21.2.102:0/1775628237 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:27:03.597 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:03.590 7f1a8bfff700 1 --2- 172.21.2.102:0/1775628237 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1a8c08b9e0 0x7f1a8c0b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:03.597 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:03.590 7f1a8affd700 1 -- 172.21.2.102:0/1775628237 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1a8c19e420 msgr2=0x7f1a8c08a9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:03.597 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:03.590 7f1a8affd700 1 --2- 172.21.2.102:0/1775628237 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1a8c19e420 0x7f1a8c08a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:03.597 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:03.590 7f1a8affd700 1 -- 172.21.2.102:0/1775628237 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1a8c08b9e0 msgr2=0x7f1a8c0b4eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:03.597 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:03.590 7f1a8affd700 1 --2- 172.21.2.102:0/1775628237 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1a8c08b9e0 0x7f1a8c0b4eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:03.598 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:03.590 7f1a8affd700 1 -- 172.21.2.102:0/1775628237 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a8c0b5b30 con 0x7f1a8c08afc0 2021-02-10T11:27:03.598 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:03.590 7f1a8bfff700 1 --2- 172.21.2.102:0/1775628237 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1a8c08b9e0 0x7f1a8c0b4eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:27:03.598 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:03.590 7f1a8affd700 1 --2- 172.21.2.102:0/1775628237 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1a8c08afc0 0x7f1a8c08b3c0 secure :-1 s=READY pgs=1689 cs=0 l=1 rev1=1 rx=0x7f1a8000b5e0 tx=0x7f1a8000b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:03.598 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:03.590 7f1a73fff700 1 -- 172.21.2.102:0/1775628237 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1a8000bd70 con 0x7f1a8c08afc0 2021-02-10T11:27:03.598 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:03.591 7f1a73fff700 1 -- 172.21.2.102:0/1775628237 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a80002030 con 0x7f1a8c08afc0 2021-02-10T11:27:03.598 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:03.591 7f1a73fff700 1 -- 172.21.2.102:0/1775628237 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1a80002550 con 0x7f1a8c08afc0 2021-02-10T11:27:03.598 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:03.591 7f1a91f77700 1 -- 172.21.2.102:0/1775628237 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a8c0b5b30 con 0x7f1a8c08afc0 2021-02-10T11:27:03.598 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:03.591 7f1a91f77700 1 -- 172.21.2.102:0/1775628237 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a8c0b60c0 con 0x7f1a8c08afc0 2021-02-10T11:27:03.598 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:03.591 7f1a73fff700 1 -- 172.21.2.102:0/1775628237 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1a80000c90 con 0x7f1a8c08afc0 2021-02-10T11:27:03.598 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:03.591 7f1a73fff700 1 --2- 172.21.2.102:0/1775628237 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1a6c0186d0 0x7f1a6c01abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:03.598 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:03.591 7f1a73fff700 1 -- 172.21.2.102:0/1775628237 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f1a80032690 con 0x7f1a8c08afc0 2021-02-10T11:27:03.598 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:03.591 7f1a8b7fe700 1 --2- 172.21.2.102:0/1775628237 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1a6c0186d0 0x7f1a6c01abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:03.598 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:03.592 7f1a8b7fe700 1 --2- 172.21.2.102:0/1775628237 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1a6c0186d0 0x7f1a6c01abd0 crc :-1 s=READY pgs=766 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:03.621 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:03.622 7f1a91f77700 1 -- 172.21.2.102:0/1775628237 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a8c08a470 con 0x7f1a8c08afc0 2021-02-10T11:27:03.623 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:03.625 7f1a73fff700 1 -- 172.21.2.102:0/1775628237 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f1a800009b0 con 0x7f1a8c08afc0 2021-02-10T11:27:03.750 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:03.751 7f1a91f77700 1 -- 172.21.2.102:0/1775628237 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7f1a8c08a470 con 0x7f1a8c08afc0 2021-02-10T11:27:03.750 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:03.751 7f1a73fff700 1 -- 172.21.2.102:0/1775628237 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v75) v1 ==== 104+0+10 (secure 0 0 0) 0x7f1a80000c90 con 0x7f1a8c08afc0 2021-02-10T11:27:03.750 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:27:03.752 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:03.753 7f1a91f77700 1 -- 172.21.2.102:0/1775628237 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1a6c0186d0 msgr2=0x7f1a6c01abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:03.752 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:03.753 7f1a91f77700 1 --2- 172.21.2.102:0/1775628237 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1a6c0186d0 0x7f1a6c01abd0 crc :-1 s=READY pgs=766 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:03.752 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:03.753 7f1a91f77700 1 -- 172.21.2.102:0/1775628237 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1a8c08afc0 msgr2=0x7f1a8c08b3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:03.752 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:03.753 7f1a91f77700 1 --2- 172.21.2.102:0/1775628237 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1a8c08afc0 0x7f1a8c08b3c0 secure :-1 s=READY pgs=1689 cs=0 l=1 rev1=1 rx=0x7f1a8000b5e0 tx=0x7f1a8000b980).stop 2021-02-10T11:27:03.752 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:03.753 7f1a91f77700 1 -- 172.21.2.102:0/1775628237 shutdown_connections 2021-02-10T11:27:03.752 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:03.753 7f1a91f77700 1 --2- 172.21.2.102:0/1775628237 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1a6c0186d0 0x7f1a6c01abd0 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:03.752 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:03.753 7f1a91f77700 1 --2- 172.21.2.102:0/1775628237 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1a8c08b9e0 0x7f1a8c0b4eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:03.753 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:03.753 7f1a91f77700 1 --2- 172.21.2.102:0/1775628237 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1a8c08afc0 0x7f1a8c08b3c0 unknown :-1 s=CLOSED pgs=1689 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:03.753 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:03.753 7f1a91f77700 1 --2- 172.21.2.102:0/1775628237 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1a8c19e420 0x7f1a8c08a9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:03.753 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:03.753 7f1a91f77700 1 -- 172.21.2.102:0/1775628237 >> 172.21.2.102:0/1775628237 conn(0x7f1a8c094dc0 msgr2=0x7f1a8c082df0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:03.753 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:03.754 7f1a91f77700 1 -- 172.21.2.102:0/1775628237 shutdown_connections 2021-02-10T11:27:03.753 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:03.754 7f1a91f77700 1 -- 172.21.2.102:0/1775628237 wait complete. 2021-02-10T11:27:03.761 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-02-10T11:27:04.193 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:04.177 7f1a60c92700 1 Processor -- start 2021-02-10T11:27:04.193 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:04.178 7f1a60c92700 1 -- start start 2021-02-10T11:27:04.194 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:04.178 7f1a60c92700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f1a5c0d1870 0x7f1a5c1b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:04.194 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:04.178 7f1a60c92700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f1a5c1b0ae0 0x7f1a5c108940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:04.194 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:04.178 7f1a60c92700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1a5c0d1c70 0x7f1a5c108f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:04.194 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:04.178 7f1a60c92700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a5c0fe1c0 con 0x7f1a5c0d1870 2021-02-10T11:27:04.194 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:04.178 7f1a60c92700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a5c109a60 con 0x7f1a5c1b0ae0 2021-02-10T11:27:04.194 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:04.178 7f1a60c92700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a5c088070 con 0x7f1a5c0d1c70 2021-02-10T11:27:04.194 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:04.178 7f1a5a85e700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f1a5c0d1870 0x7f1a5c1b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:04.194 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:04.178 7f1a5a85e700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f1a5c0d1870 0x7f1a5c1b04c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:59224/0 (socket says 172.21.2.102:59224) 2021-02-10T11:27:04.194 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:04.178 7f1a5a85e700 1 -- 172.21.2.102:0/2609672497 learned_addr learned my addr 172.21.2.102:0/2609672497 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:27:04.194 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:04.179 7f1a5a85e700 1 -- 172.21.2.102:0/2609672497 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1a5c0d1c70 msgr2=0x7f1a5c108f60 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:27:04.195 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:04.179 7f1a5a85e700 1 --2- 172.21.2.102:0/2609672497 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1a5c0d1c70 0x7f1a5c108f60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:04.195 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:04.179 7f1a5a85e700 1 -- 172.21.2.102:0/2609672497 >> v2:172.21.2.113:3300/0 conn(0x7f1a5c1b0ae0 msgr2=0x7f1a5c108940 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:27:04.195 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:04.179 7f1a5a85e700 1 --2- 172.21.2.102:0/2609672497 >> v2:172.21.2.113:3300/0 conn(0x7f1a5c1b0ae0 0x7f1a5c108940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:04.195 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:04.179 7f1a5a85e700 1 -- 172.21.2.102:0/2609672497 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a5c109580 con 0x7f1a5c0d1870 2021-02-10T11:27:04.195 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:04.179 7f1a5a85e700 1 --2- 172.21.2.102:0/2609672497 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1a5c0d1870 0x7f1a5c1b04c0 secure :-1 s=READY pgs=1690 cs=0 l=1 rev1=1 rx=0x7f1a4c006700 tx=0x7f1a4c009b80).ready entity=mon.0 client_cookie=761ead32d780eb26 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:04.195 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:04.179 7f1a5985c700 1 -- 172.21.2.102:0/2609672497 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1a4c00c0c0 con 0x7f1a5c0d1870 2021-02-10T11:27:04.195 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:04.179 7f1a5985c700 1 -- 172.21.2.102:0/2609672497 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a4c00c340 con 0x7f1a5c0d1870 2021-02-10T11:27:04.195 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:04.179 7f1a5985c700 1 -- 172.21.2.102:0/2609672497 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1a4c00c860 con 0x7f1a5c0d1870 2021-02-10T11:27:04.195 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:04.179 7f1a60c92700 1 -- 172.21.2.102:0/2609672497 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1a5c0d1870 msgr2=0x7f1a5c1b04c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:04.195 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:04.179 7f1a60c92700 1 --2- 172.21.2.102:0/2609672497 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1a5c0d1870 0x7f1a5c1b04c0 secure :-1 s=READY pgs=1690 cs=0 l=1 rev1=1 rx=0x7f1a4c006700 tx=0x7f1a4c009b80).stop 2021-02-10T11:27:04.195 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:04.180 7f1a60c92700 1 -- 172.21.2.102:0/2609672497 shutdown_connections 2021-02-10T11:27:04.196 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:04.180 7f1a60c92700 1 --2- 172.21.2.102:0/2609672497 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1a5c0d1c70 0x7f1a5c108f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:04.196 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:04.180 7f1a60c92700 1 --2- 172.21.2.102:0/2609672497 >> v2:172.21.2.113:3300/0 conn(0x7f1a5c1b0ae0 0x7f1a5c108940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:04.196 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:04.180 7f1a60c92700 1 --2- 172.21.2.102:0/2609672497 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1a5c0d1870 0x7f1a5c1b04c0 unknown :-1 s=CLOSED pgs=1690 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:04.196 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:04.180 7f1a60c92700 1 -- 172.21.2.102:0/2609672497 >> 172.21.2.102:0/2609672497 conn(0x7f1a5c0ce420 msgr2=0x7f1a5c0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:04.196 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:04.180 7f1a60c92700 1 -- 172.21.2.102:0/2609672497 shutdown_connections 2021-02-10T11:27:04.196 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:04.180 7f1a60c92700 1 -- 172.21.2.102:0/2609672497 wait complete. 2021-02-10T11:27:04.198 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:04.195 7f1a60c92700 1 Processor -- start 2021-02-10T11:27:04.198 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:04.195 7f1a60c92700 1 -- start start 2021-02-10T11:27:04.198 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:04.195 7f1a60c92700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1a5c0c3820 0x7f1a5c084cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:04.198 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:04.195 7f1a60c92700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1a5c0a0290 0x7f1a5c0852d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:04.198 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:04.195 7f1a60c92700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1a5c086270 0x7f1a5c086670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:04.198 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:04.195 7f1a60c92700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a5c0c0700 con 0x7f1a5c086270 2021-02-10T11:27:04.198 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:04.195 7f1a60c92700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a5c0858f0 con 0x7f1a5c0a0290 2021-02-10T11:27:04.198 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:04.195 7f1a60c92700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a5c085b10 con 0x7f1a5c0c3820 2021-02-10T11:27:04.198 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:04.196 7f1a5b05f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1a5c086270 0x7f1a5c086670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:04.199 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:04.196 7f1a5b05f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1a5c086270 0x7f1a5c086670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:59232/0 (socket says 172.21.2.102:59232) 2021-02-10T11:27:04.199 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:04.196 7f1a5b05f700 1 -- 172.21.2.102:0/876154518 learned_addr learned my addr 172.21.2.102:0/876154518 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:27:04.199 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:04.196 7f1a5a85e700 1 --2- 172.21.2.102:0/876154518 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1a5c0c3820 0x7f1a5c084cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:04.199 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:04.196 7f1a5a05d700 1 --2- 172.21.2.102:0/876154518 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1a5c0a0290 0x7f1a5c0852d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:04.199 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:04.196 7f1a5b05f700 1 -- 172.21.2.102:0/876154518 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1a5c0c3820 msgr2=0x7f1a5c084cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:04.199 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:04.196 7f1a5b05f700 1 --2- 172.21.2.102:0/876154518 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1a5c0c3820 0x7f1a5c084cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:04.199 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:04.196 7f1a5b05f700 1 -- 172.21.2.102:0/876154518 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1a5c0a0290 msgr2=0x7f1a5c0852d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:04.199 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:04.196 7f1a5b05f700 1 --2- 172.21.2.102:0/876154518 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1a5c0a0290 0x7f1a5c0852d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:04.199 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:04.196 7f1a5b05f700 1 -- 172.21.2.102:0/876154518 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a5c086c90 con 0x7f1a5c086270 2021-02-10T11:27:04.199 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:04.196 7f1a5b05f700 1 --2- 172.21.2.102:0/876154518 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1a5c086270 0x7f1a5c086670 secure :-1 s=READY pgs=1691 cs=0 l=1 rev1=1 rx=0x7f1a54009960 tx=0x7f1a5400bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:04.199 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:04.197 7f1a537fe700 1 -- 172.21.2.102:0/876154518 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1a54003030 con 0x7f1a5c086270 2021-02-10T11:27:04.200 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:04.197 7f1a537fe700 1 -- 172.21.2.102:0/876154518 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a54001310 con 0x7f1a5c086270 2021-02-10T11:27:04.200 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:04.197 7f1a60c92700 1 -- 172.21.2.102:0/876154518 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a5c086c90 con 0x7f1a5c086270 2021-02-10T11:27:04.200 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:04.197 7f1a537fe700 1 -- 172.21.2.102:0/876154518 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1a54001770 con 0x7f1a5c086270 2021-02-10T11:27:04.200 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:04.197 7f1a60c92700 1 -- 172.21.2.102:0/876154518 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a5c1a2f00 con 0x7f1a5c086270 2021-02-10T11:27:04.200 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:04.197 7f1a537fe700 1 -- 172.21.2.102:0/876154518 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1a54000d10 con 0x7f1a5c086270 2021-02-10T11:27:04.200 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:04.197 7f1a537fe700 1 --2- 172.21.2.102:0/876154518 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1a40018730 0x7f1a4001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:04.200 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:04.197 7f1a537fe700 1 -- 172.21.2.102:0/876154518 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f1a540329c0 con 0x7f1a5c086270 2021-02-10T11:27:04.200 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:04.197 7f1a5a85e700 1 --2- 172.21.2.102:0/876154518 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1a40018730 0x7f1a4001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:04.200 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:04.198 7f1a5a85e700 1 --2- 172.21.2.102:0/876154518 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1a40018730 0x7f1a4001ac10 crc :-1 s=READY pgs=767 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:04.227 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:04.228 7f1a60c92700 1 -- 172.21.2.102:0/876154518 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a5c082a70 con 0x7f1a5c086270 2021-02-10T11:27:04.229 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:04.230 7f1a537fe700 1 -- 172.21.2.102:0/876154518 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f1a5400be30 con 0x7f1a5c086270 2021-02-10T11:27:04.360 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:04.361 7f1a60c92700 1 -- 172.21.2.102:0/876154518 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7f1a5c082a70 con 0x7f1a5c086270 2021-02-10T11:27:04.360 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:04.362 7f1a537fe700 1 -- 172.21.2.102:0/876154518 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v75) v1 ==== 100+0+10 (secure 0 0 0) 0x7f1a54001310 con 0x7f1a5c086270 2021-02-10T11:27:04.361 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:27:04.363 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:04.364 7f1a60c92700 1 -- 172.21.2.102:0/876154518 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1a40018730 msgr2=0x7f1a4001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:04.363 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:04.364 7f1a60c92700 1 --2- 172.21.2.102:0/876154518 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1a40018730 0x7f1a4001ac10 crc :-1 s=READY pgs=767 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:04.363 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:04.364 7f1a60c92700 1 -- 172.21.2.102:0/876154518 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1a5c086270 msgr2=0x7f1a5c086670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:04.363 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:04.364 7f1a60c92700 1 --2- 172.21.2.102:0/876154518 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1a5c086270 0x7f1a5c086670 secure :-1 s=READY pgs=1691 cs=0 l=1 rev1=1 rx=0x7f1a54009960 tx=0x7f1a5400bcf0).stop 2021-02-10T11:27:04.363 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:04.364 7f1a60c92700 1 -- 172.21.2.102:0/876154518 shutdown_connections 2021-02-10T11:27:04.363 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:04.364 7f1a60c92700 1 --2- 172.21.2.102:0/876154518 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1a5c086270 0x7f1a5c086670 unknown :-1 s=CLOSED pgs=1691 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:04.363 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:04.364 7f1a60c92700 1 --2- 172.21.2.102:0/876154518 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1a40018730 0x7f1a4001ac10 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:04.363 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:04.364 7f1a60c92700 1 --2- 172.21.2.102:0/876154518 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1a5c0a0290 0x7f1a5c0852d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:04.363 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:04.364 7f1a60c92700 1 --2- 172.21.2.102:0/876154518 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1a5c0c3820 0x7f1a5c084cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:04.364 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:04.364 7f1a60c92700 1 -- 172.21.2.102:0/876154518 >> 172.21.2.102:0/876154518 conn(0x7f1a5c0872c0 msgr2=0x7f1a5c095700 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:04.364 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:04.364 7f1a60c92700 1 -- 172.21.2.102:0/876154518 shutdown_connections 2021-02-10T11:27:04.364 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:04.364 7f1a60c92700 1 -- 172.21.2.102:0/876154518 wait complete. 2021-02-10T11:27:04.371 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-02-10T11:27:04.371 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:27:04.799 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:04.782 7f8d357b8700 1 Processor -- start 2021-02-10T11:27:04.799 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:04.782 7f8d357b8700 1 -- start start 2021-02-10T11:27:04.799 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:04.783 7f8d357b8700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f8d300d0340 0x7f8d300ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:04.799 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:04.783 7f8d357b8700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d300f0b10 con 0x7f8d300d1c70 2021-02-10T11:27:04.799 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:04.783 7f8d357b8700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d301b0be0 con 0x7f8d300d1870 2021-02-10T11:27:04.799 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:04.783 7f8d357b8700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d300e5ed0 con 0x7f8d300d0340 2021-02-10T11:27:04.799 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:04.783 7f8d2fd75700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f8d300d1c70 0x7f8d300efc60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:51464/0 (socket says 172.21.2.102:51464) 2021-02-10T11:27:04.800 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:04.783 7f8d2fd75700 1 -- 172.21.2.102:0/1071677914 learned_addr learned my addr 172.21.2.102:0/1071677914 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:27:04.800 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:04.783 7f8d2e572700 1 -- 172.21.2.102:0/1071677914 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3049584800 0 0) 0x7f8d28001500 con 0x7f8d300d1c70 2021-02-10T11:27:04.800 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:04.784 7f8d2e572700 1 -- 172.21.2.102:0/1071677914 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d1c002fe0 con 0x7f8d300d1c70 2021-02-10T11:27:04.800 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:04.784 7f8d2ed73700 1 --2- 172.21.2.102:0/1071677914 >> v2:172.21.2.113:3300/0 conn(0x7f8d300d0340 0x7f8d300ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:04.800 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:04.784 7f8d2e572700 1 -- 172.21.2.102:0/1071677914 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 887977607 0 0) 0x7f8d28001500 con 0x7f8d300d1c70 2021-02-10T11:27:04.800 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:04.784 7f8d2e572700 1 -- 172.21.2.102:0/1071677914 >> v2:172.21.2.113:3300/0 conn(0x7f8d300d0340 msgr2=0x7f8d300ef640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:27:04.800 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:04.784 7f8d2e572700 1 --2- 172.21.2.102:0/1071677914 >> v2:172.21.2.113:3300/0 conn(0x7f8d300d0340 0x7f8d300ef640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:04.800 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:04.784 7f8d2e572700 1 -- 172.21.2.102:0/1071677914 >> v1:172.21.2.113:6789/0 conn(0x7f8d300d1870 legacy=0x7f8d301b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:04.800 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:04.784 7f8d2e572700 1 -- 172.21.2.102:0/1071677914 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d300f0450 con 0x7f8d300d1c70 2021-02-10T11:27:04.800 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:04.784 7f8d2e572700 1 -- 172.21.2.102:0/1071677914 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7f8d28001a00 con 0x7f8d300d1c70 2021-02-10T11:27:04.800 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:04.784 7f8d2ed73700 1 --2- 172.21.2.102:0/1071677914 >> v2:172.21.2.113:3300/0 conn(0x7f8d300d0340 0x7f8d300ef640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:27:04.800 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:04.784 7f8d2e572700 1 -- 172.21.2.102:0/1071677914 >> v1:172.21.2.102:6789/0 conn(0x7f8d300d1c70 legacy=0x7f8d300efc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:04.801 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:04.784 7f8d2e572700 1 --2- 172.21.2.102:0/1071677914 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8d1c003aa0 0x7f8d1c005f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:04.801 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:04.784 7f8d2e572700 1 --2- 172.21.2.102:0/1071677914 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8d1c006780 0x7f8d1c008ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:04.801 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:04.784 7f8d2e572700 1 --2- 172.21.2.102:0/1071677914 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8d1c0095a0 0x7f8d1c00baa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:04.801 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:04.784 7f8d2e572700 1 -- 172.21.2.102:0/1071677914 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d1c00c2f0 con 0x7f8d1c003aa0 2021-02-10T11:27:04.801 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:04.784 7f8d2e572700 1 -- 172.21.2.102:0/1071677914 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d1c00c510 con 0x7f8d1c0095a0 2021-02-10T11:27:04.801 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:04.784 7f8d2e572700 1 -- 172.21.2.102:0/1071677914 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d1c0506d0 con 0x7f8d1c006780 2021-02-10T11:27:04.801 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:04.784 7f8d2f574700 1 --2- 172.21.2.102:0/1071677914 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8d1c003aa0 0x7f8d1c005f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:04.801 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:04.784 7f8d2fd75700 1 --2- 172.21.2.102:0/1071677914 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8d1c0095a0 0x7f8d1c00baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:04.801 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:04.784 7f8d2ed73700 1 --2- 172.21.2.102:0/1071677914 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8d1c006780 0x7f8d1c008ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:04.801 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:04.785 7f8d2f574700 1 -- 172.21.2.102:0/1071677914 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8d1c006780 msgr2=0x7f8d1c008ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:27:04.801 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:04.785 7f8d2f574700 1 --2- 172.21.2.102:0/1071677914 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8d1c006780 0x7f8d1c008ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:04.801 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:04.785 7f8d2f574700 1 -- 172.21.2.102:0/1071677914 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8d1c0095a0 msgr2=0x7f8d1c00baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:27:04.802 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:04.785 7f8d2f574700 1 --2- 172.21.2.102:0/1071677914 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8d1c0095a0 0x7f8d1c00baa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:04.802 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:04.785 7f8d2f574700 1 -- 172.21.2.102:0/1071677914 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d1c050970 con 0x7f8d1c003aa0 2021-02-10T11:27:04.802 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:04.785 7f8d2fd75700 1 -- 172.21.2.102:0/1071677914 reap_dead start 2021-02-10T11:27:04.802 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:04.785 7f8d2f574700 1 --2- 172.21.2.102:0/1071677914 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8d1c003aa0 0x7f8d1c005f60 secure :-1 s=READY pgs=1693 cs=0 l=1 rev1=1 rx=0x7f8d20006ac0 tx=0x7f8d20009af0).ready entity=mon.0 client_cookie=bc4318fec1328503 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:04.802 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:04.785 7f8d2e572700 1 -- 172.21.2.102:0/1071677914 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8d20007280 con 0x7f8d1c003aa0 2021-02-10T11:27:04.802 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:04.785 7f8d2e572700 1 -- 172.21.2.102:0/1071677914 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d20007540 con 0x7f8d1c003aa0 2021-02-10T11:27:04.802 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:04.785 7f8d2e572700 1 -- 172.21.2.102:0/1071677914 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8d200079a0 con 0x7f8d1c003aa0 2021-02-10T11:27:04.802 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:04.785 7f8d357b8700 1 -- 172.21.2.102:0/1071677914 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8d1c003aa0 msgr2=0x7f8d1c005f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:04.802 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:04.785 7f8d357b8700 1 --2- 172.21.2.102:0/1071677914 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8d1c003aa0 0x7f8d1c005f60 secure :-1 s=READY pgs=1693 cs=0 l=1 rev1=1 rx=0x7f8d20006ac0 tx=0x7f8d20009af0).stop 2021-02-10T11:27:04.802 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:04.785 7f8d357b8700 1 -- 172.21.2.102:0/1071677914 shutdown_connections 2021-02-10T11:27:04.802 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:04.785 7f8d357b8700 1 --2- 172.21.2.102:0/1071677914 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8d1c003aa0 0x7f8d1c005f60 unknown :-1 s=CLOSED pgs=1693 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:04.802 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:04.785 7f8d357b8700 1 -- 172.21.2.102:0/1071677914 >> 172.21.2.102:0/1071677914 conn(0x7f8d300ce450 msgr2=0x7f8d300d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:04.803 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:04.785 7f8d357b8700 1 -- 172.21.2.102:0/1071677914 shutdown_connections 2021-02-10T11:27:04.803 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:04.785 7f8d357b8700 1 -- 172.21.2.102:0/1071677914 wait complete. 2021-02-10T11:27:04.803 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:04.800 7f8d357b8700 1 Processor -- start 2021-02-10T11:27:04.803 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:04.800 7f8d357b8700 1 -- start start 2021-02-10T11:27:04.804 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:04.801 7f8d357b8700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8d30195650 0x7f8d300b5560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:04.804 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:04.801 7f8d357b8700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8d300b5b80 0x7f8d301a2e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:04.805 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:04.801 7f8d357b8700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8d300acfa0 0x7f8d301a3440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:04.805 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:04.801 7f8d357b8700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d300b6010 con 0x7f8d300acfa0 2021-02-10T11:27:04.805 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:04.801 7f8d357b8700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d301a3a60 con 0x7f8d30195650 2021-02-10T11:27:04.805 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:04.801 7f8d357b8700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d301a3c40 con 0x7f8d300b5b80 2021-02-10T11:27:04.805 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:04.801 7f8d2fd75700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8d300acfa0 0x7f8d301a3440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:04.805 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:04.801 7f8d2fd75700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8d300acfa0 0x7f8d301a3440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:59254/0 (socket says 172.21.2.102:59254) 2021-02-10T11:27:04.805 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:04.801 7f8d2fd75700 1 -- 172.21.2.102:0/1496416458 learned_addr learned my addr 172.21.2.102:0/1496416458 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:27:04.805 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:04.801 7f8d2fd75700 1 -- 172.21.2.102:0/1496416458 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8d300b5b80 msgr2=0x7f8d301a2e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:04.805 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:04.801 7f8d2fd75700 1 --2- 172.21.2.102:0/1496416458 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8d300b5b80 0x7f8d301a2e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:04.805 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:04.801 7f8d2fd75700 1 -- 172.21.2.102:0/1496416458 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8d30195650 msgr2=0x7f8d300b5560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:04.805 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:04.801 7f8d2fd75700 1 --2- 172.21.2.102:0/1496416458 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8d30195650 0x7f8d300b5560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:04.805 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:04.801 7f8d2fd75700 1 -- 172.21.2.102:0/1496416458 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d300bcae0 con 0x7f8d300acfa0 2021-02-10T11:27:04.805 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:04.801 7f8d2fd75700 1 --2- 172.21.2.102:0/1496416458 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8d300acfa0 0x7f8d301a3440 secure :-1 s=READY pgs=1694 cs=0 l=1 rev1=1 rx=0x7f8d2800a920 tx=0x7f8d2800acc0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:04.806 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:04.802 7f8d27fff700 1 -- 172.21.2.102:0/1496416458 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8d280070d0 con 0x7f8d300acfa0 2021-02-10T11:27:04.806 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:04.802 7f8d27fff700 1 -- 172.21.2.102:0/1496416458 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d28007350 con 0x7f8d300acfa0 2021-02-10T11:27:04.806 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:04.802 7f8d27fff700 1 -- 172.21.2.102:0/1496416458 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8d28004930 con 0x7f8d300acfa0 2021-02-10T11:27:04.806 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:04.802 7f8d357b8700 1 -- 172.21.2.102:0/1496416458 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d300bcae0 con 0x7f8d300acfa0 2021-02-10T11:27:04.806 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:04.802 7f8d357b8700 1 -- 172.21.2.102:0/1496416458 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d300bd070 con 0x7f8d300acfa0 2021-02-10T11:27:04.806 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:04.802 7f8d27fff700 1 -- 172.21.2.102:0/1496416458 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f8d28000e10 con 0x7f8d300acfa0 2021-02-10T11:27:04.806 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:04.802 7f8d27fff700 1 --2- 172.21.2.102:0/1496416458 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8d140186c0 0x7f8d1401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:04.806 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:04.802 7f8d27fff700 1 -- 172.21.2.102:0/1496416458 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f8d280328c0 con 0x7f8d300acfa0 2021-02-10T11:27:04.806 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:04.803 7f8d2f574700 1 --2- 172.21.2.102:0/1496416458 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8d140186c0 0x7f8d1401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:04.806 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:04.803 7f8d2f574700 1 --2- 172.21.2.102:0/1496416458 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8d140186c0 0x7f8d1401abc0 crc :-1 s=READY pgs=768 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:04.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:04.833 7f8d357b8700 1 -- 172.21.2.102:0/1496416458 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d300f0ab0 con 0x7f8d300acfa0 2021-02-10T11:27:04.834 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:04.835 7f8d27fff700 1 -- 172.21.2.102:0/1496416458 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f8d28001270 con 0x7f8d300acfa0 2021-02-10T11:27:04.962 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:04.963 7f8d357b8700 1 -- 172.21.2.102:0/1496416458 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8d300f0ab0 con 0x7f8d300acfa0 2021-02-10T11:27:04.963 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:04.964 7f8d27fff700 1 -- 172.21.2.102:0/1496416458 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 57 v57) v1 ==== 94+0+2098 (secure 0 0 0) 0x7f8d28000da0 con 0x7f8d300acfa0 2021-02-10T11:27:04.963 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:27:04.963 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":57,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8865,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.102:6835/980649634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6834","nonce":980649634},{"type":"v1","addr":"172.21.2.102:6835","nonce":980649634}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":53}],"filesystems":[{"mdsmap":{"epoch":57,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:26:47.543386","modified":"2021-02-10 11:26:49.552060","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6799},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6799":{"gid":6799,"name":"b","rank":0,"incarnation":54,"state":"up:active","state_seq":3,"addr":"172.21.2.113:6833/2002224946","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6832","nonce":2002224946},{"type":"v1","addr":"172.21.2.113:6833","nonce":2002224946}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2021-02-10T11:27:04.963 INFO:teuthology.orchestra.run.gibba002.stderr:dumped fsmap epoch 57 2021-02-10T11:27:04.964 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:04.965 7f8d357b8700 1 -- 172.21.2.102:0/1496416458 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8d140186c0 msgr2=0x7f8d1401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:04.964 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:04.966 7f8d357b8700 1 --2- 172.21.2.102:0/1496416458 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8d140186c0 0x7f8d1401abc0 crc :-1 s=READY pgs=768 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:04.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:04.966 7f8d357b8700 1 -- 172.21.2.102:0/1496416458 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8d300acfa0 msgr2=0x7f8d301a3440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:04.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:04.966 7f8d357b8700 1 --2- 172.21.2.102:0/1496416458 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8d300acfa0 0x7f8d301a3440 secure :-1 s=READY pgs=1694 cs=0 l=1 rev1=1 rx=0x7f8d2800a920 tx=0x7f8d2800acc0).stop 2021-02-10T11:27:04.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:04.966 7f8d357b8700 1 -- 172.21.2.102:0/1496416458 shutdown_connections 2021-02-10T11:27:04.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:04.966 7f8d357b8700 1 --2- 172.21.2.102:0/1496416458 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8d300acfa0 0x7f8d301a3440 unknown :-1 s=CLOSED pgs=1694 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:04.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:04.966 7f8d357b8700 1 --2- 172.21.2.102:0/1496416458 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8d300b5b80 0x7f8d301a2e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:04.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:04.966 7f8d357b8700 1 --2- 172.21.2.102:0/1496416458 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8d140186c0 0x7f8d1401abc0 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:04.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:04.966 7f8d357b8700 1 --2- 172.21.2.102:0/1496416458 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8d30195650 0x7f8d300b5560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:04.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:04.966 7f8d357b8700 1 -- 172.21.2.102:0/1496416458 >> 172.21.2.102:0/1496416458 conn(0x7f8d300b3040 msgr2=0x7f8d300c3250 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:04.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:04.966 7f8d357b8700 1 -- 172.21.2.102:0/1496416458 shutdown_connections 2021-02-10T11:27:04.966 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:04.966 7f8d357b8700 1 -- 172.21.2.102:0/1496416458 wait complete. 2021-02-10T11:27:05.175 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 57, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:26:47.543386', 'modified': '2021-02-10 11:26:49.552060', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6799}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6799': {'gid': 6799, 'name': 'b', 'rank': 0, 'incarnation': 54, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.113:6833/2002224946', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.113:6832', 'nonce': 2002224946}, {'type': 'v1', 'addr': '172.21.2.113:6833', 'nonce': 2002224946}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:27:05.176 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:27:05.176 DEBUG:teuthology.orchestra.run.gibba013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:27:05.575 INFO:teuthology.orchestra.run.gibba013.stdout:{ 2021-02-10T11:27:05.575 INFO:teuthology.orchestra.run.gibba013.stdout: "cluster_fsid": "68db30ea-dbce-4aef-a6df-f01c6fea0513", 2021-02-10T11:27:05.575 INFO:teuthology.orchestra.run.gibba013.stdout: "whoami": 0, 2021-02-10T11:27:05.575 INFO:teuthology.orchestra.run.gibba013.stdout: "id": 6799, 2021-02-10T11:27:05.575 INFO:teuthology.orchestra.run.gibba013.stdout: "want_state": "up:active", 2021-02-10T11:27:05.575 INFO:teuthology.orchestra.run.gibba013.stdout: "state": "up:active", 2021-02-10T11:27:05.576 INFO:teuthology.orchestra.run.gibba013.stdout: "rank_uptime": 18.030033380999999, 2021-02-10T11:27:05.576 INFO:teuthology.orchestra.run.gibba013.stdout: "mdsmap_epoch": 57, 2021-02-10T11:27:05.576 INFO:teuthology.orchestra.run.gibba013.stdout: "osdmap_epoch": 75, 2021-02-10T11:27:05.576 INFO:teuthology.orchestra.run.gibba013.stdout: "osdmap_epoch_barrier": 75, 2021-02-10T11:27:05.576 INFO:teuthology.orchestra.run.gibba013.stdout: "uptime": 23.667758235000001 2021-02-10T11:27:05.576 INFO:teuthology.orchestra.run.gibba013.stdout:} 2021-02-10T11:27:05.579 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "68db30ea-dbce-4aef-a6df-f01c6fea0513", "whoami": 0, "id": 6799, "want_state": "up:active", "state": "up:active", "rank_uptime": 18.030033380999999, "mdsmap_epoch": 57, "osdmap_epoch": 75, "osdmap_epoch_barrier": 75, "uptime": 23.667758235000001 } 2021-02-10T11:27:05.579 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2021-02-10T11:27:05.579 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2021-02-10T11:27:05.579 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.1 at ubuntu@gibba002.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.1... 2021-02-10T11:27:05.579 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:05.580 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec mkdir -p /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:27:05.916 DEBUG:teuthology.orchestra.run.gibba002:> sudo modprobe fuse 2021-02-10T11:27:06.006 DEBUG:teuthology.orchestra.run.gibba002:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:27:06.010 DEBUG:teuthology.orchestra.run.gibba013:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:27:06.261 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:27:06.261 DEBUG:teuthology.orchestra.run.gibba002:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:27:06.605 INFO:teuthology.orchestra.run.gibba002.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:27:06.605 INFO:teuthology.orchestra.run.gibba002.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:27:06.606 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:27:06.606 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:06.606 DEBUG:teuthology.orchestra.run.gibba002:> ls /sys/fs/fuse/connections 2021-02-10T11:27:06.939 INFO:teuthology.orchestra.run.gibba002.stdout:43 2021-02-10T11:27:06.940 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [43] 2021-02-10T11:27:06.940 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.1 /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:27:06.943 DEBUG:teuthology.orchestra.run.gibba002:> sudo modprobe fuse 2021-02-10T11:27:07.288 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:27:07.288 DEBUG:teuthology.orchestra.run.gibba002:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:27:07.354 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba002.stderr:ceph-fuse[165899]: starting ceph client 2021-02-10T11:27:07.354 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba002.stderr:2021-02-10 11:27:07.355 7f6b6a095f80 -1 init, newargv = 0x55888eaa1d70 newargc=9 2021-02-10T11:27:07.360 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba002.stderr:ceph-fuse[165899]: starting fuse 2021-02-10T11:27:07.631 INFO:teuthology.orchestra.run.gibba002.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:27:07.631 INFO:teuthology.orchestra.run.gibba002.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:27:07.632 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:27:07.632 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:07.632 DEBUG:teuthology.orchestra.run.gibba002:> ls /sys/fs/fuse/connections 2021-02-10T11:27:07.967 INFO:teuthology.orchestra.run.gibba002.stdout:43 2021-02-10T11:27:07.967 INFO:teuthology.orchestra.run.gibba002.stdout:44 2021-02-10T11:27:07.967 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43, 44] 2021-02-10T11:27:07.967 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:07.967 DEBUG:teuthology.orchestra.run.gibba002:> sudo python3 -c ' 2021-02-10T11:27:07.967 DEBUG:teuthology.orchestra.run.gibba002:> import glob 2021-02-10T11:27:07.967 DEBUG:teuthology.orchestra.run.gibba002:> import re 2021-02-10T11:27:07.968 DEBUG:teuthology.orchestra.run.gibba002:> import os 2021-02-10T11:27:07.968 DEBUG:teuthology.orchestra.run.gibba002:> import subprocess 2021-02-10T11:27:07.968 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:27:07.968 DEBUG:teuthology.orchestra.run.gibba002:> def find_socket(client_name): 2021-02-10T11:27:07.968 DEBUG:teuthology.orchestra.run.gibba002:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2021-02-10T11:27:07.968 DEBUG:teuthology.orchestra.run.gibba002:> files = glob.glob(asok_path) 2021-02-10T11:27:07.968 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:27:07.968 DEBUG:teuthology.orchestra.run.gibba002:> # Given a non-glob path, it better be there 2021-02-10T11:27:07.968 DEBUG:teuthology.orchestra.run.gibba002:> if "*" not in asok_path: 2021-02-10T11:27:07.968 DEBUG:teuthology.orchestra.run.gibba002:> assert(len(files) == 1) 2021-02-10T11:27:07.968 DEBUG:teuthology.orchestra.run.gibba002:> return files[0] 2021-02-10T11:27:07.968 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:27:07.968 DEBUG:teuthology.orchestra.run.gibba002:> for f in files: 2021-02-10T11:27:07.968 DEBUG:teuthology.orchestra.run.gibba002:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2021-02-10T11:27:07.968 DEBUG:teuthology.orchestra.run.gibba002:> if os.path.exists("/proc/{0}".format(pid)): 2021-02-10T11:27:07.968 DEBUG:teuthology.orchestra.run.gibba002:> return f 2021-02-10T11:27:07.969 DEBUG:teuthology.orchestra.run.gibba002:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2021-02-10T11:27:07.969 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:27:07.969 DEBUG:teuthology.orchestra.run.gibba002:> print(find_socket("client.1")) 2021-02-10T11:27:07.969 DEBUG:teuthology.orchestra.run.gibba002:> ' 2021-02-10T11:27:08.331 INFO:teuthology.orchestra.run.gibba002.stdout:/var/run/ceph/ceph-client.1.165899.asok 2021-02-10T11:27:08.334 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.165899.asok 2021-02-10T11:27:08.334 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:08.334 DEBUG:teuthology.orchestra.run.gibba002:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.165899.asok status 2021-02-10T11:27:08.725 INFO:teuthology.orchestra.run.gibba002.stdout:{ 2021-02-10T11:27:08.725 INFO:teuthology.orchestra.run.gibba002.stdout: "metadata": { 2021-02-10T11:27:08.725 INFO:teuthology.orchestra.run.gibba002.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:27:08.725 INFO:teuthology.orchestra.run.gibba002.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:27:08.725 INFO:teuthology.orchestra.run.gibba002.stdout: "entity_id": "1", 2021-02-10T11:27:08.725 INFO:teuthology.orchestra.run.gibba002.stdout: "hostname": "gibba002", 2021-02-10T11:27:08.726 INFO:teuthology.orchestra.run.gibba002.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2021-02-10T11:27:08.726 INFO:teuthology.orchestra.run.gibba002.stdout: "pid": "165899", 2021-02-10T11:27:08.726 INFO:teuthology.orchestra.run.gibba002.stdout: "root": "/" 2021-02-10T11:27:08.726 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:27:08.726 INFO:teuthology.orchestra.run.gibba002.stdout: "dentry_count": 0, 2021-02-10T11:27:08.726 INFO:teuthology.orchestra.run.gibba002.stdout: "dentry_pinned_count": 0, 2021-02-10T11:27:08.726 INFO:teuthology.orchestra.run.gibba002.stdout: "id": 9051, 2021-02-10T11:27:08.726 INFO:teuthology.orchestra.run.gibba002.stdout: "inst": { 2021-02-10T11:27:08.726 INFO:teuthology.orchestra.run.gibba002.stdout: "name": { 2021-02-10T11:27:08.726 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "client", 2021-02-10T11:27:08.727 INFO:teuthology.orchestra.run.gibba002.stdout: "num": 9051 2021-02-10T11:27:08.727 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:27:08.727 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": { 2021-02-10T11:27:08.727 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:27:08.728 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:0", 2021-02-10T11:27:08.728 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 3245853741 2021-02-10T11:27:08.728 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:27:08.728 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:27:08.728 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": { 2021-02-10T11:27:08.728 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:27:08.728 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:0", 2021-02-10T11:27:08.728 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 3245853741 2021-02-10T11:27:08.728 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:27:08.728 INFO:teuthology.orchestra.run.gibba002.stdout: "inst_str": "client.9051 172.21.2.102:0/3245853741", 2021-02-10T11:27:08.728 INFO:teuthology.orchestra.run.gibba002.stdout: "addr_str": "172.21.2.102:0/3245853741", 2021-02-10T11:27:08.728 INFO:teuthology.orchestra.run.gibba002.stdout: "inode_count": 1, 2021-02-10T11:27:08.729 INFO:teuthology.orchestra.run.gibba002.stdout: "mds_epoch": 57, 2021-02-10T11:27:08.729 INFO:teuthology.orchestra.run.gibba002.stdout: "osd_epoch": 75, 2021-02-10T11:27:08.729 INFO:teuthology.orchestra.run.gibba002.stdout: "osd_epoch_barrier": 0, 2021-02-10T11:27:08.729 INFO:teuthology.orchestra.run.gibba002.stdout: "blacklisted": false 2021-02-10T11:27:08.729 INFO:teuthology.orchestra.run.gibba002.stdout:} 2021-02-10T11:27:08.730 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:08.730 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:27:08.730 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:27:09.065 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:27:09.065 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2021-02-10T11:27:09.066 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:09.066 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:27:09.413 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2021-02-10T11:27:09.844 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:09.829 7f19e894b700 1 Processor -- start 2021-02-10T11:27:09.844 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:09.829 7f19e894b700 1 -- start start 2021-02-10T11:27:09.844 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:09.830 7f19e894b700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f19e40d1880 0x7f19e41b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:09.844 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:09.830 7f19e894b700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f19e40cff80 0x7f19e4109140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:09.844 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:09.830 7f19e894b700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19e4109a70 con 0x7f19e41b0af0 2021-02-10T11:27:09.845 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:09.830 7f19e894b700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19e4109760 con 0x7f19e40cff80 2021-02-10T11:27:09.845 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:09.830 7f19e894b700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19e4088070 con 0x7f19e40d1880 2021-02-10T11:27:09.845 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:09.830 7f19e2e1d700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f19e40cff80 0x7f19e4109140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:09.845 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:09.830 7f19e2e1d700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f19e40cff80 0x7f19e4109140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:59270/0 (socket says 172.21.2.102:59270) 2021-02-10T11:27:09.845 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:09.830 7f19e2e1d700 1 -- 172.21.2.102:0/4041530313 learned_addr learned my addr 172.21.2.102:0/4041530313 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:27:09.845 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:09.830 7f19e2e1d700 1 -- 172.21.2.102:0/4041530313 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f19e40d1880 msgr2=0x7f19e41b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:27:09.845 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:09.830 7f19e2e1d700 1 --2- 172.21.2.102:0/4041530313 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f19e40d1880 0x7f19e41b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:09.845 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:09.830 7f19e2e1d700 1 -- 172.21.2.102:0/4041530313 >> v1:172.21.2.102:6789/0 conn(0x7f19e41b0af0 legacy=0x7f19e4108950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:09.845 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:09.830 7f19e2e1d700 1 -- 172.21.2.102:0/4041530313 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19e4088260 con 0x7f19e40cff80 2021-02-10T11:27:09.845 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:09.830 7f19e2e1d700 1 --2- 172.21.2.102:0/4041530313 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f19e40cff80 0x7f19e4109140 secure :-1 s=READY pgs=1697 cs=0 l=1 rev1=1 rx=0x7f19dc006f00 tx=0x7f19dc009e90).ready entity=mon.0 client_cookie=da100c7d3f0e172 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:09.845 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:09.830 7f19e161a700 1 -- 172.21.2.102:0/4041530313 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f19dc00b300 con 0x7f19e40cff80 2021-02-10T11:27:09.845 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:09.830 7f19e161a700 1 -- 172.21.2.102:0/4041530313 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19dc00b580 con 0x7f19e40cff80 2021-02-10T11:27:09.845 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:09.830 7f19e161a700 1 -- 172.21.2.102:0/4041530313 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f19dc00ba60 con 0x7f19e40cff80 2021-02-10T11:27:09.846 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:09.830 7f19e894b700 1 -- 172.21.2.102:0/4041530313 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f19e40cff80 msgr2=0x7f19e4109140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:09.846 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:09.830 7f19e894b700 1 --2- 172.21.2.102:0/4041530313 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f19e40cff80 0x7f19e4109140 secure :-1 s=READY pgs=1697 cs=0 l=1 rev1=1 rx=0x7f19dc006f00 tx=0x7f19dc009e90).stop 2021-02-10T11:27:09.846 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:09.831 7f19e894b700 1 -- 172.21.2.102:0/4041530313 shutdown_connections 2021-02-10T11:27:09.846 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:09.831 7f19e894b700 1 --2- 172.21.2.102:0/4041530313 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f19e40cff80 0x7f19e4109140 unknown :-1 s=CLOSED pgs=1697 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:09.846 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:09.831 7f19e894b700 1 --2- 172.21.2.102:0/4041530313 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f19e40d1880 0x7f19e41b04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:09.846 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:09.831 7f19e894b700 1 -- 172.21.2.102:0/4041530313 >> 172.21.2.102:0/4041530313 conn(0x7f19e40ce430 msgr2=0x7f19e40d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:09.846 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:09.831 7f19e894b700 1 -- 172.21.2.102:0/4041530313 shutdown_connections 2021-02-10T11:27:09.846 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:09.831 7f19e894b700 1 -- 172.21.2.102:0/4041530313 wait complete. 2021-02-10T11:27:09.847 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:09.846 7f19e894b700 1 Processor -- start 2021-02-10T11:27:09.847 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:09.846 7f19e894b700 1 -- start start 2021-02-10T11:27:09.847 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:09.846 7f19e894b700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f19e41b2980 0x7f19e41b2d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:09.848 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:09.846 7f19e894b700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f19e41a8ec0 0x7f19e41a92c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:09.848 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:09.846 7f19e894b700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f19e41a98e0 0x7f19e41aa620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:09.848 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:09.846 7f19e894b700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19e41aa120 con 0x7f19e41b2980 2021-02-10T11:27:09.848 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:09.846 7f19e894b700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19e41b33e0 con 0x7f19e41a8ec0 2021-02-10T11:27:09.848 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:09.846 7f19e894b700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19e41a9ce0 con 0x7f19e41a98e0 2021-02-10T11:27:09.848 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:09.846 7f19e261c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f19e41b2980 0x7f19e41b2d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:09.848 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:09.846 7f19e261c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f19e41b2980 0x7f19e41b2d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:59278/0 (socket says 172.21.2.102:59278) 2021-02-10T11:27:09.848 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:09.846 7f19e261c700 1 -- 172.21.2.102:0/4254763515 learned_addr learned my addr 172.21.2.102:0/4254763515 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:27:09.848 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:09.847 7f19e261c700 1 -- 172.21.2.102:0/4254763515 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f19e41a98e0 msgr2=0x7f19e41aa620 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:27:09.848 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:09.847 7f19e261c700 1 --2- 172.21.2.102:0/4254763515 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f19e41a98e0 0x7f19e41aa620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:09.848 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:09.847 7f19e261c700 1 -- 172.21.2.102:0/4254763515 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f19e41a8ec0 msgr2=0x7f19e41a92c0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:27:09.848 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:09.847 7f19e261c700 1 --2- 172.21.2.102:0/4254763515 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f19e41a8ec0 0x7f19e41a92c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:09.848 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:09.847 7f19e261c700 1 -- 172.21.2.102:0/4254763515 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19e41aac40 con 0x7f19e41b2980 2021-02-10T11:27:09.849 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:09.847 7f19e261c700 1 --2- 172.21.2.102:0/4254763515 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f19e41b2980 0x7f19e41b2d80 secure :-1 s=READY pgs=1698 cs=0 l=1 rev1=1 rx=0x7f19d400b530 tx=0x7f19d400b8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:09.849 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:09.847 7f19daffd700 1 -- 172.21.2.102:0/4254763515 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f19d400bcc0 con 0x7f19e41b2980 2021-02-10T11:27:09.849 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:09.847 7f19daffd700 1 -- 172.21.2.102:0/4254763515 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19d4000c90 con 0x7f19e41b2980 2021-02-10T11:27:09.849 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:09.847 7f19daffd700 1 -- 172.21.2.102:0/4254763515 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f19d40022a0 con 0x7f19e41b2980 2021-02-10T11:27:09.849 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:09.847 7f19e894b700 1 -- 172.21.2.102:0/4254763515 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19e41aac40 con 0x7f19e41b2980 2021-02-10T11:27:09.849 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:09.847 7f19e894b700 1 -- 172.21.2.102:0/4254763515 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19e41ab1d0 con 0x7f19e41b2980 2021-02-10T11:27:09.849 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:09.847 7f19daffd700 1 -- 172.21.2.102:0/4254763515 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f19d4005310 con 0x7f19e41b2980 2021-02-10T11:27:09.849 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:09.848 7f19daffd700 1 --2- 172.21.2.102:0/4254763515 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f19c80186c0 0x7f19c801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:09.849 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:09.848 7f19daffd700 1 -- 172.21.2.102:0/4254763515 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f19d4032530 con 0x7f19e41b2980 2021-02-10T11:27:09.849 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:09.848 7f19e1e1b700 1 --2- 172.21.2.102:0/4254763515 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f19c80186c0 0x7f19c801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:09.849 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:09.848 7f19e1e1b700 1 --2- 172.21.2.102:0/4254763515 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f19c80186c0 0x7f19c801abc0 crc :-1 s=READY pgs=769 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:09.878 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:09.880 7f19e894b700 1 -- 172.21.2.102:0/4254763515 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19e41aba60 con 0x7f19e41b2980 2021-02-10T11:27:09.881 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:09.882 7f19daffd700 1 -- 172.21.2.102:0/4254763515 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f19d4000960 con 0x7f19e41b2980 2021-02-10T11:27:10.018 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:10.019 7f19e894b700 1 -- 172.21.2.102:0/4254763515 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f19e41aba60 con 0x7f19e41b2980 2021-02-10T11:27:10.465 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:10.465 7f19daffd700 1 -- 172.21.2.102:0/4254763515 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v58) v1 ==== 136+0+0 (secure 0 0 0) 0x7f19d4005310 con 0x7f19e41b2980 2021-02-10T11:27:10.488 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:10.487 7f19e894b700 1 -- 172.21.2.102:0/4254763515 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f19c80186c0 msgr2=0x7f19c801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:10.488 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:10.487 7f19e894b700 1 --2- 172.21.2.102:0/4254763515 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f19c80186c0 0x7f19c801abc0 crc :-1 s=READY pgs=769 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:10.488 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:10.487 7f19e894b700 1 -- 172.21.2.102:0/4254763515 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f19e41b2980 msgr2=0x7f19e41b2d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:10.488 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:10.487 7f19e894b700 1 --2- 172.21.2.102:0/4254763515 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f19e41b2980 0x7f19e41b2d80 secure :-1 s=READY pgs=1698 cs=0 l=1 rev1=1 rx=0x7f19d400b530 tx=0x7f19d400b8d0).stop 2021-02-10T11:27:10.488 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:10.487 7f19e894b700 1 -- 172.21.2.102:0/4254763515 shutdown_connections 2021-02-10T11:27:10.488 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:10.487 7f19e894b700 1 --2- 172.21.2.102:0/4254763515 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f19e41a98e0 0x7f19e41aa620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:10.488 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:10.487 7f19e894b700 1 --2- 172.21.2.102:0/4254763515 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f19c80186c0 0x7f19c801abc0 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:10.488 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:10.487 7f19e894b700 1 --2- 172.21.2.102:0/4254763515 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f19e41a8ec0 0x7f19e41a92c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:10.488 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:10.487 7f19e894b700 1 --2- 172.21.2.102:0/4254763515 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f19e41b2980 0x7f19e41b2d80 unknown :-1 s=CLOSED pgs=1698 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:10.488 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:10.487 7f19e894b700 1 -- 172.21.2.102:0/4254763515 >> 172.21.2.102:0/4254763515 conn(0x7f19e419c670 msgr2=0x7f19e4083710 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:10.489 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:10.488 7f19e894b700 1 -- 172.21.2.102:0/4254763515 shutdown_connections 2021-02-10T11:27:10.489 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:10.488 7f19e894b700 1 -- 172.21.2.102:0/4254763515 wait complete. 2021-02-10T11:27:10.496 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs volume ls 2021-02-10T11:27:10.923 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:10.906 7f0f942c1700 1 Processor -- start 2021-02-10T11:27:10.923 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:10.906 7f0f942c1700 1 -- start start 2021-02-10T11:27:10.923 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:10.906 7f0f942c1700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f0f8c0d1870 0x7f0f8c1b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:10.924 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:10.906 7f0f942c1700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0f8c1b0a10 0x7f0f8c0cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:10.924 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:10.907 7f0f942c1700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f8c0f0b10 con 0x7f0f8c0d1c70 2021-02-10T11:27:10.924 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:10.907 7f0f942c1700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f8c0e5ed0 con 0x7f0f8c0d1870 2021-02-10T11:27:10.924 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:10.907 7f0f942c1700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f8c0f0650 con 0x7f0f8c1b0a10 2021-02-10T11:27:10.924 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:10.907 7f0f920dd700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f0f8c0d1870 0x7f0f8c1b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:10.924 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:10.907 7f0f920dd700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f0f8c0d1870 0x7f0f8c1b03f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.113:3300/0 says I am v2:172.21.2.102:34480/0 (socket says 172.21.2.102:34480) 2021-02-10T11:27:10.924 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:10.907 7f0f920dd700 1 -- 172.21.2.102:0/342598788 learned_addr learned my addr 172.21.2.102:0/342598788 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:27:10.924 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:10.907 7f0f918dc700 1 --2- 172.21.2.102:0/342598788 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0f8c1b0a10 0x7f0f8c0cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:10.924 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:10.908 7f0f920dd700 1 -- 172.21.2.102:0/342598788 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0f8c1b0a10 msgr2=0x7f0f8c0cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:27:10.924 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:10.908 7f0f920dd700 1 --2- 172.21.2.102:0/342598788 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0f8c1b0a10 0x7f0f8c0cf330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:10.924 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:10.908 7f0f920dd700 1 -- 172.21.2.102:0/342598788 >> v1:172.21.2.113:6789/0 conn(0x7f0f8c0d1c70 legacy=0x7f0f8c0cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:10.924 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:10.908 7f0f920dd700 1 -- 172.21.2.102:0/342598788 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f8c0f0870 con 0x7f0f8c0d1870 2021-02-10T11:27:10.924 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:10.908 7f0f918dc700 1 --2- 172.21.2.102:0/342598788 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0f8c1b0a10 0x7f0f8c0cf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:27:10.925 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:10.908 7f0f920dd700 1 --2- 172.21.2.102:0/342598788 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0f8c0d1870 0x7f0f8c1b03f0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 rx=0x7f0f84006a80 tx=0x7f0f84009f00).ready entity=mon.1 client_cookie=fe60742de358f65c server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:10.925 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:10.908 7f0f910db700 1 -- 172.21.2.102:0/342598788 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0f840066c0 con 0x7f0f8c0d1870 2021-02-10T11:27:10.925 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:10.908 7f0f910db700 1 -- 172.21.2.102:0/342598788 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f8400b5c0 con 0x7f0f8c0d1870 2021-02-10T11:27:10.925 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:10.908 7f0f910db700 1 -- 172.21.2.102:0/342598788 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0f8400ba10 con 0x7f0f8c0d1870 2021-02-10T11:27:10.925 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:10.909 7f0f942c1700 1 -- 172.21.2.102:0/342598788 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0f8c0d1870 msgr2=0x7f0f8c1b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:10.925 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:10.909 7f0f942c1700 1 --2- 172.21.2.102:0/342598788 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0f8c0d1870 0x7f0f8c1b03f0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 rx=0x7f0f84006a80 tx=0x7f0f84009f00).stop 2021-02-10T11:27:10.925 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:10.909 7f0f942c1700 1 -- 172.21.2.102:0/342598788 shutdown_connections 2021-02-10T11:27:10.925 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:10.909 7f0f942c1700 1 --2- 172.21.2.102:0/342598788 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0f8c1b0a10 0x7f0f8c0cf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:10.925 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:10.909 7f0f942c1700 1 --2- 172.21.2.102:0/342598788 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0f8c0d1870 0x7f0f8c1b03f0 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:10.925 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:10.909 7f0f942c1700 1 -- 172.21.2.102:0/342598788 >> 172.21.2.102:0/342598788 conn(0x7f0f8c0ce450 msgr2=0x7f0f8c0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:10.925 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:10.909 7f0f942c1700 1 -- 172.21.2.102:0/342598788 shutdown_connections 2021-02-10T11:27:10.925 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:10.909 7f0f942c1700 1 -- 172.21.2.102:0/342598788 wait complete. 2021-02-10T11:27:10.925 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:10.923 7f0f942c1700 1 Processor -- start 2021-02-10T11:27:10.926 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:10.923 7f0f942c1700 1 -- start start 2021-02-10T11:27:10.927 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:10.924 7f0f942c1700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0f8c1a0f90 0x7f0f8c1a1390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:10.927 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:10.924 7f0f942c1700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0f8c1a19b0 0x7f0f8c0902a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:10.927 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:10.924 7f0f942c1700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0f8c0908c0 0x7f0f8c090cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:10.928 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:10.924 7f0f942c1700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f8c091370 con 0x7f0f8c1a19b0 2021-02-10T11:27:10.928 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:10.924 7f0f942c1700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f8c08d9c0 con 0x7f0f8c0908c0 2021-02-10T11:27:10.928 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:10.924 7f0f942c1700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f8c08dbe0 con 0x7f0f8c1a0f90 2021-02-10T11:27:10.928 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:10.924 7f0f918dc700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0f8c1a19b0 0x7f0f8c0902a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:10.928 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:10.924 7f0f918dc700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0f8c1a19b0 0x7f0f8c0902a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:59286/0 (socket says 172.21.2.102:59286) 2021-02-10T11:27:10.928 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:10.924 7f0f918dc700 1 -- 172.21.2.102:0/598523753 learned_addr learned my addr 172.21.2.102:0/598523753 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:27:10.928 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:10.924 7f0f928de700 1 --2- 172.21.2.102:0/598523753 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0f8c0908c0 0x7f0f8c090cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:10.928 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:10.925 7f0f920dd700 1 --2- 172.21.2.102:0/598523753 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0f8c1a0f90 0x7f0f8c1a1390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:10.928 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:10.925 7f0f918dc700 1 -- 172.21.2.102:0/598523753 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0f8c1a0f90 msgr2=0x7f0f8c1a1390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:10.928 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:10.925 7f0f918dc700 1 --2- 172.21.2.102:0/598523753 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0f8c1a0f90 0x7f0f8c1a1390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:10.928 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:10.925 7f0f918dc700 1 -- 172.21.2.102:0/598523753 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0f8c0908c0 msgr2=0x7f0f8c090cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:10.928 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:10.925 7f0f918dc700 1 --2- 172.21.2.102:0/598523753 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0f8c0908c0 0x7f0f8c090cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:10.928 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:10.925 7f0f918dc700 1 -- 172.21.2.102:0/598523753 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f8c08de00 con 0x7f0f8c1a19b0 2021-02-10T11:27:10.929 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:10.925 7f0f920dd700 1 --2- 172.21.2.102:0/598523753 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0f8c1a0f90 0x7f0f8c1a1390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:27:10.929 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:10.925 7f0f928de700 1 --2- 172.21.2.102:0/598523753 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0f8c0908c0 0x7f0f8c090cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:27:10.929 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:10.925 7f0f918dc700 1 --2- 172.21.2.102:0/598523753 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0f8c1a19b0 0x7f0f8c0902a0 secure :-1 s=READY pgs=1699 cs=0 l=1 rev1=1 rx=0x7f0f80006f40 tx=0x7f0f8000b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:10.929 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:10.925 7f0f8affd700 1 -- 172.21.2.102:0/598523753 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0f800009b0 con 0x7f0f8c1a19b0 2021-02-10T11:27:10.929 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:10.925 7f0f8affd700 1 -- 172.21.2.102:0/598523753 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f8000bd80 con 0x7f0f8c1a19b0 2021-02-10T11:27:10.929 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:10.925 7f0f942c1700 1 -- 172.21.2.102:0/598523753 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f8c08de00 con 0x7f0f8c1a19b0 2021-02-10T11:27:10.929 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:10.925 7f0f942c1700 1 -- 172.21.2.102:0/598523753 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f8c0b8950 con 0x7f0f8c1a19b0 2021-02-10T11:27:10.929 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:10.925 7f0f8affd700 1 -- 172.21.2.102:0/598523753 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0f80002220 con 0x7f0f8c1a19b0 2021-02-10T11:27:10.929 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:10.926 7f0f8affd700 1 -- 172.21.2.102:0/598523753 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f0f80006b00 con 0x7f0f8c1a19b0 2021-02-10T11:27:10.929 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:10.926 7f0f8affd700 1 --2- 172.21.2.102:0/598523753 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0f6c018730 0x7f0f6c01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:10.929 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:10.926 7f0f8affd700 1 -- 172.21.2.102:0/598523753 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f0f800327a0 con 0x7f0f8c1a19b0 2021-02-10T11:27:10.929 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:10.926 7f0f920dd700 1 --2- 172.21.2.102:0/598523753 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0f6c018730 0x7f0f6c01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:10.929 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:10.926 7f0f920dd700 1 --2- 172.21.2.102:0/598523753 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0f6c018730 0x7f0f6c01ac10 crc :-1 s=READY pgs=770 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:10.956 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:10.956 7f0f942c1700 1 -- 172.21.2.102:0/598523753 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f8c0ebc50 con 0x7f0f8c1a19b0 2021-02-10T11:27:10.959 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:10.959 7f0f8affd700 1 -- 172.21.2.102:0/598523753 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f0f80000c90 con 0x7f0f8c1a19b0 2021-02-10T11:27:11.093 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:11.093 7f0f942c1700 1 -- 172.21.2.102:0/598523753 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs volume ls", "target": ["mgr", ""]}) v1 -- 0x7f0f8c0ebc50 con 0x7f0f6c018730 2021-02-10T11:27:11.094 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:11.094 7f0f8affd700 1 -- 172.21.2.102:0/598523753 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+41 (crc 0 0 0) 0x7f0f84000cc0 con 0x7f0f6c018730 2021-02-10T11:27:11.094 INFO:teuthology.orchestra.run.gibba002.stdout:[ 2021-02-10T11:27:11.094 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:27:11.094 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "cephfs" 2021-02-10T11:27:11.094 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:27:11.095 INFO:teuthology.orchestra.run.gibba002.stdout:] 2021-02-10T11:27:11.095 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:11.095 7f0f942c1700 1 -- 172.21.2.102:0/598523753 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0f6c018730 msgr2=0x7f0f6c01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:11.095 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:11.095 7f0f942c1700 1 --2- 172.21.2.102:0/598523753 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0f6c018730 0x7f0f6c01ac10 crc :-1 s=READY pgs=770 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:11.095 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:11.095 7f0f942c1700 1 -- 172.21.2.102:0/598523753 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0f8c1a19b0 msgr2=0x7f0f8c0902a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:11.095 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:11.095 7f0f942c1700 1 --2- 172.21.2.102:0/598523753 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0f8c1a19b0 0x7f0f8c0902a0 secure :-1 s=READY pgs=1699 cs=0 l=1 rev1=1 rx=0x7f0f80006f40 tx=0x7f0f8000b990).stop 2021-02-10T11:27:11.096 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:11.096 7f0f942c1700 1 -- 172.21.2.102:0/598523753 shutdown_connections 2021-02-10T11:27:11.096 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:11.096 7f0f942c1700 1 --2- 172.21.2.102:0/598523753 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0f6c018730 0x7f0f6c01ac10 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:11.096 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:11.096 7f0f942c1700 1 --2- 172.21.2.102:0/598523753 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0f8c0908c0 0x7f0f8c090cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:11.096 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:11.096 7f0f942c1700 1 --2- 172.21.2.102:0/598523753 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0f8c1a19b0 0x7f0f8c0902a0 unknown :-1 s=CLOSED pgs=1699 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:11.096 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:11.096 7f0f942c1700 1 --2- 172.21.2.102:0/598523753 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0f8c1a0f90 0x7f0f8c1a1390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:11.096 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:11.096 7f0f942c1700 1 -- 172.21.2.102:0/598523753 >> 172.21.2.102:0/598523753 conn(0x7f0f8c09bb00 msgr2=0x7f0f8c09bfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:11.096 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:11.096 7f0f942c1700 1 -- 172.21.2.102:0/598523753 shutdown_connections 2021-02-10T11:27:11.096 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:11.096 7f0f942c1700 1 -- 172.21.2.102:0/598523753 wait complete. 2021-02-10T11:27:11.105 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config set mon mon_allow_pool_delete True 2021-02-10T11:27:11.535 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:11.519 7f69569a4700 1 Processor -- start 2021-02-10T11:27:11.535 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:11.519 7f69569a4700 1 -- start start 2021-02-10T11:27:11.536 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:11.519 7f69569a4700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f69500d1870 0x7f69501b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:11.536 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:11.519 7f69569a4700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f69500cff70 0x7f6950109130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:11.536 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:11.519 7f69569a4700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69500fe1c0 con 0x7f69501b0ae0 2021-02-10T11:27:11.536 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:11.519 7f69569a4700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6950109ad0 con 0x7f69500cff70 2021-02-10T11:27:11.536 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:11.519 7f69569a4700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6950109750 con 0x7f69500d1870 2021-02-10T11:27:11.536 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:11.520 7f6954fc1700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f69500cff70 0x7f6950109130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:11.536 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:11.520 7f694f7fe700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f69501b0ae0 0x7f6950108940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:51518/0 (socket says 172.21.2.102:51518) 2021-02-10T11:27:11.536 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:11.520 7f694f7fe700 1 -- 172.21.2.102:0/3208330975 learned_addr learned my addr 172.21.2.102:0/3208330975 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:27:11.536 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:11.520 7f694ffff700 1 --2- 172.21.2.102:0/3208330975 >> v2:172.21.2.113:3300/0 conn(0x7f69500d1870 0x7f69501b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:11.536 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:11.520 7f6954fc1700 1 -- 172.21.2.102:0/3208330975 >> v2:172.21.2.113:3300/0 conn(0x7f69500d1870 msgr2=0x7f69501b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:27:11.537 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:11.520 7f6954fc1700 1 --2- 172.21.2.102:0/3208330975 >> v2:172.21.2.113:3300/0 conn(0x7f69500d1870 0x7f69501b04c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:11.537 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:11.520 7f6954fc1700 1 -- 172.21.2.102:0/3208330975 >> v1:172.21.2.102:6789/0 conn(0x7f69501b0ae0 legacy=0x7f6950108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:11.537 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:11.520 7f6954fc1700 1 -- 172.21.2.102:0/3208330975 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6950088070 con 0x7f69500cff70 2021-02-10T11:27:11.537 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:11.520 7f6954fc1700 1 --2- 172.21.2.102:0/3208330975 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f69500cff70 0x7f6950109130 secure :-1 s=READY pgs=1701 cs=0 l=1 rev1=1 rx=0x7f6948006e30 tx=0x7f6948009e60).ready entity=mon.0 client_cookie=c83bcf357c55a3d8 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:11.537 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:11.520 7f694effd700 1 -- 172.21.2.102:0/3208330975 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f694800b300 con 0x7f69500cff70 2021-02-10T11:27:11.537 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:11.520 7f694effd700 1 -- 172.21.2.102:0/3208330975 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6948000d30 con 0x7f69500cff70 2021-02-10T11:27:11.537 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:11.520 7f694effd700 1 -- 172.21.2.102:0/3208330975 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f694800b730 con 0x7f69500cff70 2021-02-10T11:27:11.537 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:11.520 7f69569a4700 1 -- 172.21.2.102:0/3208330975 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f69500cff70 msgr2=0x7f6950109130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:11.537 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:11.520 7f69569a4700 1 --2- 172.21.2.102:0/3208330975 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f69500cff70 0x7f6950109130 secure :-1 s=READY pgs=1701 cs=0 l=1 rev1=1 rx=0x7f6948006e30 tx=0x7f6948009e60).stop 2021-02-10T11:27:11.537 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:11.521 7f69569a4700 1 -- 172.21.2.102:0/3208330975 shutdown_connections 2021-02-10T11:27:11.537 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:11.521 7f69569a4700 1 --2- 172.21.2.102:0/3208330975 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f69500cff70 0x7f6950109130 unknown :-1 s=CLOSED pgs=1701 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:11.537 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:11.521 7f69569a4700 1 --2- 172.21.2.102:0/3208330975 >> v2:172.21.2.113:3300/0 conn(0x7f69500d1870 0x7f69501b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:11.537 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:11.521 7f69569a4700 1 -- 172.21.2.102:0/3208330975 >> 172.21.2.102:0/3208330975 conn(0x7f69500ce420 msgr2=0x7f69500d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:11.538 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:11.521 7f69569a4700 1 -- 172.21.2.102:0/3208330975 shutdown_connections 2021-02-10T11:27:11.538 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:11.521 7f69569a4700 1 -- 172.21.2.102:0/3208330975 wait complete. 2021-02-10T11:27:11.539 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:11.536 7f69569a4700 1 Processor -- start 2021-02-10T11:27:11.540 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:11.536 7f69569a4700 1 -- start start 2021-02-10T11:27:11.540 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:11.536 7f69569a4700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6950085bc0 0x7f69500863e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:11.540 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:11.536 7f69569a4700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f69500954c0 0x7f6950096300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:11.540 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:11.536 7f69569a4700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f69500958c0 0x7f6950096920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:11.540 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:11.536 7f69569a4700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6950095d80 con 0x7f69500954c0 2021-02-10T11:27:11.540 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:11.536 7f69569a4700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69500bfa60 con 0x7f6950085bc0 2021-02-10T11:27:11.540 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:11.536 7f69569a4700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6950085fc0 con 0x7f69500958c0 2021-02-10T11:27:11.540 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:11.536 7f694f7fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f69500954c0 0x7f6950096300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:11.540 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:11.536 7f6954fc1700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f69500958c0 0x7f6950096920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:11.540 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:11.536 7f694f7fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f69500954c0 0x7f6950096300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:59298/0 (socket says 172.21.2.102:59298) 2021-02-10T11:27:11.540 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:11.536 7f694f7fe700 1 -- 172.21.2.102:0/1409478654 learned_addr learned my addr 172.21.2.102:0/1409478654 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:27:11.540 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:11.537 7f694ffff700 1 --2- 172.21.2.102:0/1409478654 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6950085bc0 0x7f69500863e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:11.541 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:11.537 7f6954fc1700 1 -- 172.21.2.102:0/1409478654 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6950085bc0 msgr2=0x7f69500863e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:11.541 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:11.537 7f6954fc1700 1 --2- 172.21.2.102:0/1409478654 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6950085bc0 0x7f69500863e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:11.541 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:11.537 7f6954fc1700 1 -- 172.21.2.102:0/1409478654 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f69500954c0 msgr2=0x7f6950096300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:11.541 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:11.537 7f6954fc1700 1 --2- 172.21.2.102:0/1409478654 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f69500954c0 0x7f6950096300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:11.541 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:11.537 7f6954fc1700 1 -- 172.21.2.102:0/1409478654 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6950086c10 con 0x7f69500958c0 2021-02-10T11:27:11.541 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:11.537 7f694f7fe700 1 --2- 172.21.2.102:0/1409478654 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f69500954c0 0x7f6950096300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:27:11.541 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:11.537 7f6954fc1700 1 --2- 172.21.2.102:0/1409478654 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f69500958c0 0x7f6950096920 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f694800b8e0 tx=0x7f694800bc80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:11.541 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:11.537 7f694cff9700 1 -- 172.21.2.102:0/1409478654 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6948009380 con 0x7f69500958c0 2021-02-10T11:27:11.541 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:11.537 7f694cff9700 1 -- 172.21.2.102:0/1409478654 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69480096c0 con 0x7f69500958c0 2021-02-10T11:27:11.541 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:11.537 7f69569a4700 1 -- 172.21.2.102:0/1409478654 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6950086c10 con 0x7f69500958c0 2021-02-10T11:27:11.541 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:11.537 7f694cff9700 1 -- 172.21.2.102:0/1409478654 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f69480014f0 con 0x7f69500958c0 2021-02-10T11:27:11.541 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:11.537 7f69569a4700 1 -- 172.21.2.102:0/1409478654 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6950097230 con 0x7f69500958c0 2021-02-10T11:27:11.541 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:11.538 7f694cff9700 1 -- 172.21.2.102:0/1409478654 <== mon.2 v2:172.21.2.113:3301/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f6948009380 con 0x7f69500958c0 2021-02-10T11:27:11.541 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:11.538 7f694cff9700 1 --2- 172.21.2.102:0/1409478654 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f69300186c0 0x7f693001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:11.542 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:11.538 7f694cff9700 1 -- 172.21.2.102:0/1409478654 <== mon.2 v2:172.21.2.113:3301/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f6948032720 con 0x7f69500958c0 2021-02-10T11:27:11.542 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:11.538 7f694ffff700 1 --2- 172.21.2.102:0/1409478654 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f69300186c0 0x7f693001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:11.542 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:11.539 7f694ffff700 1 --2- 172.21.2.102:0/1409478654 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f69300186c0 0x7f693001abc0 crc :-1 s=READY pgs=771 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:11.569 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:11.569 7f69569a4700 1 -- 172.21.2.102:0/1409478654 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69500fe1c0 con 0x7f69500958c0 2021-02-10T11:27:11.572 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:11.572 7f694cff9700 1 -- 172.21.2.102:0/1409478654 <== mon.2 v2:172.21.2.113:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f6948001790 con 0x7f69500958c0 2021-02-10T11:27:11.724 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:11.724 7f69569a4700 1 -- 172.21.2.102:0/1409478654 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f69500fe1c0 con 0x7f69500958c0 2021-02-10T11:27:11.726 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:11.726 7f694cff9700 1 -- 172.21.2.102:0/1409478654 <== mon.2 v2:172.21.2.113:3301/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v20)=0 v20) v1 ==== 122+0+0 (secure 0 0 0) 0x7f6948009380 con 0x7f69500958c0 2021-02-10T11:27:11.730 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:11.729 7f69569a4700 1 -- 172.21.2.102:0/1409478654 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f69300186c0 msgr2=0x7f693001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:11.730 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:11.729 7f69569a4700 1 --2- 172.21.2.102:0/1409478654 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f69300186c0 0x7f693001abc0 crc :-1 s=READY pgs=771 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:11.730 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:11.729 7f69569a4700 1 -- 172.21.2.102:0/1409478654 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f69500958c0 msgr2=0x7f6950096920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:11.730 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:11.729 7f69569a4700 1 --2- 172.21.2.102:0/1409478654 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f69500958c0 0x7f6950096920 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f694800b8e0 tx=0x7f694800bc80).stop 2021-02-10T11:27:11.730 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:11.730 7f69569a4700 1 -- 172.21.2.102:0/1409478654 shutdown_connections 2021-02-10T11:27:11.730 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:11.730 7f69569a4700 1 --2- 172.21.2.102:0/1409478654 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f69500958c0 0x7f6950096920 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:11.730 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:11.730 7f69569a4700 1 --2- 172.21.2.102:0/1409478654 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f69500954c0 0x7f6950096300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:11.730 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:11.730 7f69569a4700 1 --2- 172.21.2.102:0/1409478654 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f69300186c0 0x7f693001abc0 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:11.730 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:11.730 7f69569a4700 1 --2- 172.21.2.102:0/1409478654 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6950085bc0 0x7f69500863e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:11.730 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:11.730 7f69569a4700 1 -- 172.21.2.102:0/1409478654 >> 172.21.2.102:0/1409478654 conn(0x7f69500bef00 msgr2=0x7f695009d1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:11.730 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:11.730 7f69569a4700 1 -- 172.21.2.102:0/1409478654 shutdown_connections 2021-02-10T11:27:11.730 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:11.730 7f69569a4700 1 -- 172.21.2.102:0/1409478654 wait complete. 2021-02-10T11:27:11.738 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000414869 2021-02-10T11:27:12.169 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:12.152 7f1d8eb4e700 1 Processor -- start 2021-02-10T11:27:12.169 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:12.152 7f1d8eb4e700 1 -- start start 2021-02-10T11:27:12.170 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:12.153 7f1d8eb4e700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f1d8810aff0 0x7f1d880cf750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:12.170 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:12.153 7f1d8eb4e700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d880fe1c0 con 0x7f1d880d1870 2021-02-10T11:27:12.170 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:12.153 7f1d8eb4e700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d881b0cb0 con 0x7f1d880d1c70 2021-02-10T11:27:12.170 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:12.153 7f1d8eb4e700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d88109a60 con 0x7f1d8810aff0 2021-02-10T11:27:12.170 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:12.153 7f1d8d16b700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f1d8810aff0 0x7f1d880cf750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:12.170 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:12.153 7f1d8d16b700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f1d8810aff0 0x7f1d880cf750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:59308/0 (socket says 172.21.2.102:59308) 2021-02-10T11:27:12.170 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:12.153 7f1d8d16b700 1 -- 172.21.2.102:0/912223097 learned_addr learned my addr 172.21.2.102:0/912223097 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:27:12.170 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:12.154 7f1d8d16b700 1 -- 172.21.2.102:0/912223097 >> v1:172.21.2.113:6789/0 conn(0x7f1d880d1c70 legacy=0x7f1d880cef60 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:27:12.170 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:12.154 7f1d8d16b700 1 -- 172.21.2.102:0/912223097 >> v1:172.21.2.102:6789/0 conn(0x7f1d880d1870 legacy=0x7f1d881b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:12.170 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:12.154 7f1d877fe700 1 -- 172.21.2.102:0/912223097 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2810685945 0 0) 0x7f1d70000ce0 con 0x7f1d880d1870 2021-02-10T11:27:12.170 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:12.154 7f1d8d16b700 1 -- 172.21.2.102:0/912223097 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d8810b3f0 con 0x7f1d8810aff0 2021-02-10T11:27:12.171 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:12.154 7f1d8d16b700 1 --2- 172.21.2.102:0/912223097 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1d8810aff0 0x7f1d880cf750 secure :-1 s=READY pgs=1703 cs=0 l=1 rev1=1 rx=0x7f1d80006f00 tx=0x7f1d80009e90).ready entity=mon.0 client_cookie=5018afcac9190a70 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:12.171 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:12.154 7f1d877fe700 1 -- 172.21.2.102:0/912223097 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1d8000b300 con 0x7f1d8810aff0 2021-02-10T11:27:12.171 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:12.154 7f1d877fe700 1 -- 172.21.2.102:0/912223097 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d8000b640 con 0x7f1d8810aff0 2021-02-10T11:27:12.171 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:12.154 7f1d877fe700 1 -- 172.21.2.102:0/912223097 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1d8000ba90 con 0x7f1d8810aff0 2021-02-10T11:27:12.171 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:12.154 7f1d8eb4e700 1 -- 172.21.2.102:0/912223097 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1d8810aff0 msgr2=0x7f1d880cf750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:12.171 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:12.154 7f1d8eb4e700 1 --2- 172.21.2.102:0/912223097 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1d8810aff0 0x7f1d880cf750 secure :-1 s=READY pgs=1703 cs=0 l=1 rev1=1 rx=0x7f1d80006f00 tx=0x7f1d80009e90).stop 2021-02-10T11:27:12.171 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:12.155 7f1d8eb4e700 1 -- 172.21.2.102:0/912223097 shutdown_connections 2021-02-10T11:27:12.171 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:12.155 7f1d8eb4e700 1 --2- 172.21.2.102:0/912223097 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1d8810aff0 0x7f1d880cf750 unknown :-1 s=CLOSED pgs=1703 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:12.171 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:12.155 7f1d8eb4e700 1 -- 172.21.2.102:0/912223097 >> 172.21.2.102:0/912223097 conn(0x7f1d880ce420 msgr2=0x7f1d880d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:12.171 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:12.155 7f1d8eb4e700 1 -- 172.21.2.102:0/912223097 shutdown_connections 2021-02-10T11:27:12.171 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:12.155 7f1d8eb4e700 1 -- 172.21.2.102:0/912223097 wait complete. 2021-02-10T11:27:12.173 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:12.170 7f1d8eb4e700 1 Processor -- start 2021-02-10T11:27:12.173 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:12.170 7f1d8eb4e700 1 -- start start 2021-02-10T11:27:12.173 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:12.170 7f1d8eb4e700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1d880c3820 0x7f1d880a7950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:12.173 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:12.170 7f1d8eb4e700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1d881a44c0 0x7f1d881a48c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:12.173 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:12.170 7f1d8eb4e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1d880863d0 0x7f1d880867d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:12.173 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:12.170 7f1d8eb4e700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d88085ed0 con 0x7f1d880863d0 2021-02-10T11:27:12.174 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:12.170 7f1d8eb4e700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d880c0740 con 0x7f1d880c3820 2021-02-10T11:27:12.174 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:12.170 7f1d8eb4e700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d88086df0 con 0x7f1d881a44c0 2021-02-10T11:27:12.174 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:12.171 7f1d8d16b700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1d880863d0 0x7f1d880867d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:12.174 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:12.171 7f1d8d16b700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1d880863d0 0x7f1d880867d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:59314/0 (socket says 172.21.2.102:59314) 2021-02-10T11:27:12.174 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:12.171 7f1d8d16b700 1 -- 172.21.2.102:0/3315632157 learned_addr learned my addr 172.21.2.102:0/3315632157 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:27:12.174 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:12.171 7f1d8d16b700 1 -- 172.21.2.102:0/3315632157 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1d881a44c0 msgr2=0x7f1d881a48c0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:27:12.174 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:12.171 7f1d8d16b700 1 --2- 172.21.2.102:0/3315632157 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1d881a44c0 0x7f1d881a48c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:12.174 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:12.171 7f1d8d16b700 1 -- 172.21.2.102:0/3315632157 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1d880c3820 msgr2=0x7f1d880a7950 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:27:12.174 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:12.171 7f1d8d16b700 1 --2- 172.21.2.102:0/3315632157 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1d880c3820 0x7f1d880a7950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:12.174 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:12.171 7f1d8d16b700 1 -- 172.21.2.102:0/3315632157 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d88087010 con 0x7f1d880863d0 2021-02-10T11:27:12.174 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:12.171 7f1d8d16b700 1 --2- 172.21.2.102:0/3315632157 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1d880863d0 0x7f1d880867d0 secure :-1 s=READY pgs=1704 cs=0 l=1 rev1=1 rx=0x7f1d8000b910 tx=0x7f1d8000bcb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:12.174 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:12.171 7f1d857fa700 1 -- 172.21.2.102:0/3315632157 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1d80000cc0 con 0x7f1d880863d0 2021-02-10T11:27:12.175 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:12.171 7f1d857fa700 1 -- 172.21.2.102:0/3315632157 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d8000bdf0 con 0x7f1d880863d0 2021-02-10T11:27:12.175 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:12.171 7f1d857fa700 1 -- 172.21.2.102:0/3315632157 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1d80005a10 con 0x7f1d880863d0 2021-02-10T11:27:12.175 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:12.171 7f1d8eb4e700 1 -- 172.21.2.102:0/3315632157 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d88087010 con 0x7f1d880863d0 2021-02-10T11:27:12.175 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:12.171 7f1d8eb4e700 1 -- 172.21.2.102:0/3315632157 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d88083000 con 0x7f1d880863d0 2021-02-10T11:27:12.175 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:12.172 7f1d857fa700 1 -- 172.21.2.102:0/3315632157 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1d80005820 con 0x7f1d880863d0 2021-02-10T11:27:12.175 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:12.172 7f1d857fa700 1 --2- 172.21.2.102:0/3315632157 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1d6c0186c0 0x7f1d6c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:12.175 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:12.172 7f1d857fa700 1 -- 172.21.2.102:0/3315632157 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f1d800328e0 con 0x7f1d880863d0 2021-02-10T11:27:12.175 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:12.172 7f1d8c96a700 1 --2- 172.21.2.102:0/3315632157 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1d6c0186c0 0x7f1d6c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:12.175 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:12.173 7f1d8c96a700 1 --2- 172.21.2.102:0/3315632157 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1d6c0186c0 0x7f1d6c01abc0 crc :-1 s=READY pgs=772 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:12.203 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:12.203 7f1d8eb4e700 1 -- 172.21.2.102:0/3315632157 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d880fe1c0 con 0x7f1d880863d0 2021-02-10T11:27:12.205 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:12.206 7f1d857fa700 1 -- 172.21.2.102:0/3315632157 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f1d80000cc0 con 0x7f1d880863d0 2021-02-10T11:27:12.331 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:12.331 7f1d8eb4e700 1 -- 172.21.2.102:0/3315632157 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000414869", "target": ["mgr", ""]}) v1 -- 0x7f1d88083000 con 0x7f1d6c0186c0 2021-02-10T11:27:12.334 INFO:tasks.ceph.mgr.y.gibba002.stderr:2021-02-10 11:27:12.335 7fd94bc53700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:27:12.335 INFO:tasks.ceph.mgr.y.gibba002.stderr:2021-02-10 11:27:12.335 7fd94bc53700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:27:12.335 INFO:tasks.ceph.mgr.y.gibba002.stderr:2021-02-10 11:27:12.335 7fd94bc53700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:27:12.335 INFO:tasks.ceph.mgr.y.gibba002.stderr:2021-02-10 11:27:12.335 7fd94bc53700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:27:12.335 INFO:tasks.ceph.mgr.y.gibba002.stderr:2021-02-10 11:27:12.335 7fd94bc53700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:27:12.367 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:12.367 7f1d857fa700 1 -- 172.21.2.102:0/3315632157 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f1d70000ce0 con 0x7f1d6c0186c0 2021-02-10T11:27:12.397 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:12.396 7f1d8eb4e700 1 -- 172.21.2.102:0/3315632157 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1d6c0186c0 msgr2=0x7f1d6c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:12.397 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:12.396 7f1d8eb4e700 1 --2- 172.21.2.102:0/3315632157 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1d6c0186c0 0x7f1d6c01abc0 crc :-1 s=READY pgs=772 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:12.397 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:12.396 7f1d8eb4e700 1 -- 172.21.2.102:0/3315632157 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1d880863d0 msgr2=0x7f1d880867d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:12.397 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:12.396 7f1d8eb4e700 1 --2- 172.21.2.102:0/3315632157 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1d880863d0 0x7f1d880867d0 secure :-1 s=READY pgs=1704 cs=0 l=1 rev1=1 rx=0x7f1d8000b910 tx=0x7f1d8000bcb0).stop 2021-02-10T11:27:12.397 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:12.396 7f1d8eb4e700 1 -- 172.21.2.102:0/3315632157 shutdown_connections 2021-02-10T11:27:12.397 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:12.396 7f1d8eb4e700 1 --2- 172.21.2.102:0/3315632157 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1d880863d0 0x7f1d880867d0 unknown :-1 s=CLOSED pgs=1704 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:12.397 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:12.396 7f1d8eb4e700 1 --2- 172.21.2.102:0/3315632157 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1d881a44c0 0x7f1d881a48c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:12.397 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:12.396 7f1d8eb4e700 1 --2- 172.21.2.102:0/3315632157 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1d6c0186c0 0x7f1d6c01abc0 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:12.397 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:12.396 7f1d8eb4e700 1 --2- 172.21.2.102:0/3315632157 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1d880c3820 0x7f1d880a7950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:12.397 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:12.397 7f1d8eb4e700 1 -- 172.21.2.102:0/3315632157 >> 172.21.2.102:0/3315632157 conn(0x7f1d880872c0 msgr2=0x7f1d88095700 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:12.398 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:12.397 7f1d8eb4e700 1 -- 172.21.2.102:0/3315632157 shutdown_connections 2021-02-10T11:27:12.398 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:12.397 7f1d8eb4e700 1 -- 172.21.2.102:0/3315632157 wait complete. 2021-02-10T11:27:12.405 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000414869 2021-02-10T11:27:12.834 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:12.818 7fa923e4e700 1 Processor -- start 2021-02-10T11:27:12.834 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:12.818 7fa923e4e700 1 -- start start 2021-02-10T11:27:12.834 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:12.819 7fa923e4e700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa91c0d1870 0x7fa91c1b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:12.834 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:12.819 7fa923e4e700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fa91c1b0ae0 0x7fa91c108940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:12.835 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:12.819 7fa923e4e700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fa91c0d1c70 0x7fa91c108f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:12.835 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:12.819 7fa923e4e700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa91c0e5ef0 con 0x7fa91c1b0ae0 2021-02-10T11:27:12.835 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:12.819 7fa923e4e700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa91c109a60 con 0x7fa91c0d1c70 2021-02-10T11:27:12.835 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:12.819 7fa923e4e700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa91c088070 con 0x7fa91c0d1870 2021-02-10T11:27:12.835 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:12.819 7fa921469700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fa91c1b0ae0 0x7fa91c108940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:12.835 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:12.819 7fa921469700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fa91c1b0ae0 0x7fa91c108940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:59334/0 (socket says 172.21.2.102:59334) 2021-02-10T11:27:12.835 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:12.819 7fa921469700 1 -- 172.21.2.102:0/2750723203 learned_addr learned my addr 172.21.2.102:0/2750723203 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:27:12.835 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:12.819 7fa92246b700 1 --2- 172.21.2.102:0/2750723203 >> v2:172.21.2.113:3300/0 conn(0x7fa91c0d1c70 0x7fa91c108f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:12.835 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:12.819 7fa921c6a700 1 --2- 172.21.2.102:0/2750723203 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa91c0d1870 0x7fa91c1b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:12.835 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:12.819 7fa921469700 1 -- 172.21.2.102:0/2750723203 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa91c0d1870 msgr2=0x7fa91c1b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:27:12.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:12.819 7fa921469700 1 --2- 172.21.2.102:0/2750723203 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa91c0d1870 0x7fa91c1b04c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:12.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:12.819 7fa921469700 1 -- 172.21.2.102:0/2750723203 >> v2:172.21.2.113:3300/0 conn(0x7fa91c0d1c70 msgr2=0x7fa91c108f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:27:12.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:12.819 7fa921469700 1 --2- 172.21.2.102:0/2750723203 >> v2:172.21.2.113:3300/0 conn(0x7fa91c0d1c70 0x7fa91c108f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:12.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:12.819 7fa921469700 1 -- 172.21.2.102:0/2750723203 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa91c109580 con 0x7fa91c1b0ae0 2021-02-10T11:27:12.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:12.819 7fa921469700 1 --2- 172.21.2.102:0/2750723203 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa91c1b0ae0 0x7fa91c108940 secure :-1 s=READY pgs=1707 cs=0 l=1 rev1=1 rx=0x7fa918006700 tx=0x7fa918009b80).ready entity=mon.0 client_cookie=1f02e58072477a1b server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:12.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:12.819 7fa920c68700 1 -- 172.21.2.102:0/2750723203 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa91800c0c0 con 0x7fa91c1b0ae0 2021-02-10T11:27:12.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:12.819 7fa920c68700 1 -- 172.21.2.102:0/2750723203 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa91800c340 con 0x7fa91c1b0ae0 2021-02-10T11:27:12.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:12.819 7fa920c68700 1 -- 172.21.2.102:0/2750723203 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa91800c820 con 0x7fa91c1b0ae0 2021-02-10T11:27:12.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:12.819 7fa923e4e700 1 -- 172.21.2.102:0/2750723203 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa91c1b0ae0 msgr2=0x7fa91c108940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:12.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:12.819 7fa923e4e700 1 --2- 172.21.2.102:0/2750723203 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa91c1b0ae0 0x7fa91c108940 secure :-1 s=READY pgs=1707 cs=0 l=1 rev1=1 rx=0x7fa918006700 tx=0x7fa918009b80).stop 2021-02-10T11:27:12.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:12.820 7fa923e4e700 1 -- 172.21.2.102:0/2750723203 shutdown_connections 2021-02-10T11:27:12.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:12.820 7fa923e4e700 1 --2- 172.21.2.102:0/2750723203 >> v2:172.21.2.113:3300/0 conn(0x7fa91c0d1c70 0x7fa91c108f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:12.837 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:12.820 7fa923e4e700 1 --2- 172.21.2.102:0/2750723203 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa91c1b0ae0 0x7fa91c108940 unknown :-1 s=CLOSED pgs=1707 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:12.837 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:12.820 7fa923e4e700 1 --2- 172.21.2.102:0/2750723203 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa91c0d1870 0x7fa91c1b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:12.837 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:12.820 7fa923e4e700 1 -- 172.21.2.102:0/2750723203 >> 172.21.2.102:0/2750723203 conn(0x7fa91c0ce420 msgr2=0x7fa91c0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:12.837 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:12.820 7fa923e4e700 1 -- 172.21.2.102:0/2750723203 shutdown_connections 2021-02-10T11:27:12.837 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:12.820 7fa923e4e700 1 -- 172.21.2.102:0/2750723203 wait complete. 2021-02-10T11:27:12.839 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:12.835 7fa923e4e700 1 Processor -- start 2021-02-10T11:27:12.839 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:12.835 7fa923e4e700 1 -- start start 2021-02-10T11:27:12.839 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:12.835 7fa923e4e700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa91c0abd60 0x7fa91c0ac160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:12.839 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:12.835 7fa923e4e700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa91c0ac780 0x7fa91c0acb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:12.839 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:12.835 7fa923e4e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa91c083910 0x7fa91c083d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:12.839 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:12.835 7fa923e4e700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa91c087180 con 0x7fa91c083910 2021-02-10T11:27:12.839 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:12.835 7fa923e4e700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa91c18fcd0 con 0x7fa91c0ac780 2021-02-10T11:27:12.839 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:12.835 7fa923e4e700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa91c084330 con 0x7fa91c0abd60 2021-02-10T11:27:12.839 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:12.835 7fa92246b700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa91c083910 0x7fa91c083d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:12.839 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:12.835 7fa92246b700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa91c083910 0x7fa91c083d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:59342/0 (socket says 172.21.2.102:59342) 2021-02-10T11:27:12.839 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:12.835 7fa92246b700 1 -- 172.21.2.102:0/1681332673 learned_addr learned my addr 172.21.2.102:0/1681332673 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:27:12.840 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:12.836 7fa92246b700 1 -- 172.21.2.102:0/1681332673 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa91c0abd60 msgr2=0x7fa91c0ac160 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:27:12.840 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:12.836 7fa92246b700 1 --2- 172.21.2.102:0/1681332673 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa91c0abd60 0x7fa91c0ac160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:12.840 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:12.836 7fa92246b700 1 -- 172.21.2.102:0/1681332673 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa91c0ac780 msgr2=0x7fa91c0acb80 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:27:12.840 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:12.836 7fa92246b700 1 --2- 172.21.2.102:0/1681332673 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa91c0ac780 0x7fa91c0acb80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:12.840 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:12.836 7fa92246b700 1 -- 172.21.2.102:0/1681332673 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa91c084550 con 0x7fa91c083910 2021-02-10T11:27:12.840 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:12.836 7fa92246b700 1 --2- 172.21.2.102:0/1681332673 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa91c083910 0x7fa91c083d10 secure :-1 s=READY pgs=1708 cs=0 l=1 rev1=1 rx=0x7fa9140099a0 tx=0x7fa91400bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:12.840 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:12.836 7fa90e7fc700 1 -- 172.21.2.102:0/1681332673 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa9140066f0 con 0x7fa91c083910 2021-02-10T11:27:12.840 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:12.836 7fa90e7fc700 1 -- 172.21.2.102:0/1681332673 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa914003030 con 0x7fa91c083910 2021-02-10T11:27:12.840 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:12.836 7fa90e7fc700 1 -- 172.21.2.102:0/1681332673 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa9140013e0 con 0x7fa91c083910 2021-02-10T11:27:12.840 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:12.836 7fa923e4e700 1 -- 172.21.2.102:0/1681332673 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa91c084550 con 0x7fa91c083910 2021-02-10T11:27:12.840 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:12.836 7fa923e4e700 1 -- 172.21.2.102:0/1681332673 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa91c084550 con 0x7fa91c083910 2021-02-10T11:27:12.841 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:12.837 7fa90e7fc700 1 -- 172.21.2.102:0/1681332673 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa9140011f0 con 0x7fa91c083910 2021-02-10T11:27:12.841 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:12.837 7fa90e7fc700 1 --2- 172.21.2.102:0/1681332673 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa904018730 0x7fa90401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:12.841 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:12.837 7fa90e7fc700 1 -- 172.21.2.102:0/1681332673 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fa9140329c0 con 0x7fa91c083910 2021-02-10T11:27:12.841 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:12.837 7fa921c6a700 1 --2- 172.21.2.102:0/1681332673 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa904018730 0x7fa90401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:12.841 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:12.837 7fa921c6a700 1 --2- 172.21.2.102:0/1681332673 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa904018730 0x7fa90401ac10 crc :-1 s=READY pgs=773 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:12.868 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:12.868 7fa923e4e700 1 -- 172.21.2.102:0/1681332673 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa91c082980 con 0x7fa91c083910 2021-02-10T11:27:12.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:12.871 7fa90e7fc700 1 -- 172.21.2.102:0/1681332673 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fa91400be30 con 0x7fa91c083910 2021-02-10T11:27:13.000 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:13.000 7fa923e4e700 1 -- 172.21.2.102:0/1681332673 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000414869", "target": ["mgr", ""]}) v1 -- 0x7fa91c109a60 con 0x7fa904018730 2021-02-10T11:27:13.002 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:13.003 7fa90e7fc700 1 -- 172.21.2.102:0/1681332673 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7fa910006060 con 0x7fa904018730 2021-02-10T11:27:13.003 INFO:teuthology.orchestra.run.gibba002.stdout:/volumes/_nogroup/subvolume_0000000000414869/ce232bc9-f3e4-4acb-b112-69faf3523a34 2021-02-10T11:27:13.005 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:13.005 7fa923e4e700 1 -- 172.21.2.102:0/1681332673 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa904018730 msgr2=0x7fa90401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:13.005 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:13.005 7fa923e4e700 1 --2- 172.21.2.102:0/1681332673 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa904018730 0x7fa90401ac10 crc :-1 s=READY pgs=773 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:13.005 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:13.005 7fa923e4e700 1 -- 172.21.2.102:0/1681332673 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa91c083910 msgr2=0x7fa91c083d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:13.005 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:13.005 7fa923e4e700 1 --2- 172.21.2.102:0/1681332673 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa91c083910 0x7fa91c083d10 secure :-1 s=READY pgs=1708 cs=0 l=1 rev1=1 rx=0x7fa9140099a0 tx=0x7fa91400bcf0).stop 2021-02-10T11:27:13.005 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:13.005 7fa923e4e700 1 -- 172.21.2.102:0/1681332673 shutdown_connections 2021-02-10T11:27:13.005 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:13.005 7fa923e4e700 1 --2- 172.21.2.102:0/1681332673 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa91c083910 0x7fa91c083d10 unknown :-1 s=CLOSED pgs=1708 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:13.005 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:13.005 7fa923e4e700 1 --2- 172.21.2.102:0/1681332673 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa904018730 0x7fa90401ac10 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:13.005 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:13.005 7fa923e4e700 1 --2- 172.21.2.102:0/1681332673 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa91c0ac780 0x7fa91c0acb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:13.005 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:13.005 7fa923e4e700 1 --2- 172.21.2.102:0/1681332673 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa91c0abd60 0x7fa91c0ac160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:13.005 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:13.005 7fa923e4e700 1 -- 172.21.2.102:0/1681332673 >> 172.21.2.102:0/1681332673 conn(0x7fa91c0a2b30 msgr2=0x7fa91c09ce60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:13.006 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:13.005 7fa923e4e700 1 -- 172.21.2.102:0/1681332673 shutdown_connections 2021-02-10T11:27:13.006 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:13.005 7fa923e4e700 1 -- 172.21.2.102:0/1681332673 wait complete. 2021-02-10T11:27:13.013 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000414869 with 1 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000414869/ce232bc9-f3e4-4acb-b112-69faf3523a34 2021-02-10T11:27:13.013 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:13.014 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:27:13.014 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:27:13.350 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:27:13.350 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:27:13.351 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000414869/ce232bc9-f3e4-4acb-b112-69faf3523a34/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:27:13.737 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:27:13.737 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:27:13.737 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0291243 s, 36.0 MB/s 2021-02-10T11:27:13.739 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume snapshot create cephfs subvolume_0000000000414869 snapshot_0000000000810191 2021-02-10T11:27:14.168 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:14.152 7f962e2db700 1 Processor -- start 2021-02-10T11:27:14.168 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:14.152 7f962e2db700 1 -- start start 2021-02-10T11:27:14.168 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:14.153 7f962e2db700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f96280d1880 0x7f96281b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:14.169 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:14.153 7f962e2db700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f96281b0af0 0x7f9628108950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:14.169 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:14.153 7f962e2db700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96280fe1d0 con 0x7f96280d1c80 2021-02-10T11:27:14.169 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:14.153 7f962e2db700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9628109a70 con 0x7f96280d1880 2021-02-10T11:27:14.169 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:14.153 7f962e2db700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9628109760 con 0x7f96281b0af0 2021-02-10T11:27:14.169 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:14.153 7f9627fff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f96280d1880 0x7f96281b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:14.169 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:14.153 7f96277fe700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f96281b0af0 0x7f9628108950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:14.169 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:14.153 7f96277fe700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f96281b0af0 0x7f9628108950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.113:3301/0 says I am v2:172.21.2.102:36652/0 (socket says 172.21.2.102:36652) 2021-02-10T11:27:14.169 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:14.153 7f9627fff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f96280d1880 0x7f96281b04d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:59348/0 (socket says 172.21.2.102:59348) 2021-02-10T11:27:14.169 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:14.153 7f96277fe700 1 -- 172.21.2.102:0/2158349017 learned_addr learned my addr 172.21.2.102:0/2158349017 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:27:14.169 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:14.153 7f96277fe700 1 -- 172.21.2.102:0/2158349017 >> v2:172.21.2.102:3300/0 conn(0x7f96280d1880 msgr2=0x7f96281b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:27:14.169 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:14.153 7f96277fe700 1 --2- 172.21.2.102:0/2158349017 >> v2:172.21.2.102:3300/0 conn(0x7f96280d1880 0x7f96281b04d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:14.169 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:14.153 7f96277fe700 1 -- 172.21.2.102:0/2158349017 >> v1:172.21.2.102:6789/0 conn(0x7f96280d1c80 legacy=0x7f9628108f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:14.170 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:14.153 7f9626ffd700 1 -- 172.21.2.102:0/2158349017 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 565501552 0 0) 0x7f9620000f90 con 0x7f96280d1c80 2021-02-10T11:27:14.170 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:14.153 7f96277fe700 1 -- 172.21.2.102:0/2158349017 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9628088070 con 0x7f96281b0af0 2021-02-10T11:27:14.170 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:14.154 7f96277fe700 1 --2- 172.21.2.102:0/2158349017 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f96281b0af0 0x7f9628108950 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7f9614006700 tx=0x7f9614009b80).ready entity=mon.2 client_cookie=26eb30a399e064fb server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:14.170 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:14.154 7f9626ffd700 1 -- 172.21.2.102:0/2158349017 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f961400c0c0 con 0x7f96281b0af0 2021-02-10T11:27:14.170 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:14.154 7f9626ffd700 1 -- 172.21.2.102:0/2158349017 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f961400c340 con 0x7f96281b0af0 2021-02-10T11:27:14.170 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:14.154 7f9626ffd700 1 -- 172.21.2.102:0/2158349017 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f961400c860 con 0x7f96281b0af0 2021-02-10T11:27:14.170 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:14.154 7f962e2db700 1 -- 172.21.2.102:0/2158349017 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f96281b0af0 msgr2=0x7f9628108950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:14.170 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:14.154 7f962e2db700 1 --2- 172.21.2.102:0/2158349017 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f96281b0af0 0x7f9628108950 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7f9614006700 tx=0x7f9614009b80).stop 2021-02-10T11:27:14.170 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:14.155 7f962e2db700 1 -- 172.21.2.102:0/2158349017 shutdown_connections 2021-02-10T11:27:14.170 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:14.155 7f962e2db700 1 --2- 172.21.2.102:0/2158349017 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f96281b0af0 0x7f9628108950 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:14.170 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:14.155 7f962e2db700 1 --2- 172.21.2.102:0/2158349017 >> v2:172.21.2.102:3300/0 conn(0x7f96280d1880 0x7f96281b04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:14.170 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:14.155 7f962e2db700 1 -- 172.21.2.102:0/2158349017 >> 172.21.2.102:0/2158349017 conn(0x7f96280ce430 msgr2=0x7f96280d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:14.170 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:14.155 7f962e2db700 1 -- 172.21.2.102:0/2158349017 shutdown_connections 2021-02-10T11:27:14.171 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:14.155 7f962e2db700 1 -- 172.21.2.102:0/2158349017 wait complete. 2021-02-10T11:27:14.172 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:14.169 7f962e2db700 1 Processor -- start 2021-02-10T11:27:14.172 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:14.169 7f962e2db700 1 -- start start 2021-02-10T11:27:14.172 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:14.169 7f962e2db700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9628188400 0x7f9628188800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:14.173 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:14.169 7f962e2db700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9628188e20 0x7f9628189220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:14.173 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:14.169 7f962e2db700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f962809cbc0 0x7f962809cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:14.173 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:14.169 7f962e2db700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9628187e80 con 0x7f9628188e20 2021-02-10T11:27:14.173 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:14.169 7f962e2db700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96280b27c0 con 0x7f962809cbc0 2021-02-10T11:27:14.173 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:14.169 7f962e2db700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96280b29a0 con 0x7f9628188400 2021-02-10T11:27:14.173 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:14.169 7f96277fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9628188e20 0x7f9628189220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:14.173 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:14.169 7f96277fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9628188e20 0x7f9628189220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:59356/0 (socket says 172.21.2.102:59356) 2021-02-10T11:27:14.173 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:14.170 7f96277fe700 1 -- 172.21.2.102:0/4055324452 learned_addr learned my addr 172.21.2.102:0/4055324452 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:27:14.173 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:14.170 7f9627fff700 1 --2- 172.21.2.102:0/4055324452 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9628188400 0x7f9628188800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:14.173 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:14.170 7f962c8f8700 1 --2- 172.21.2.102:0/4055324452 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f962809cbc0 0x7f962809cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:14.174 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:14.170 7f96277fe700 1 -- 172.21.2.102:0/4055324452 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9628188400 msgr2=0x7f9628188800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:14.174 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:14.170 7f96277fe700 1 --2- 172.21.2.102:0/4055324452 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9628188400 0x7f9628188800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:14.174 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:14.170 7f96277fe700 1 -- 172.21.2.102:0/4055324452 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f962809cbc0 msgr2=0x7f962809cfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:14.174 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:14.170 7f96277fe700 1 --2- 172.21.2.102:0/4055324452 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f962809cbc0 0x7f962809cfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:14.174 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:14.170 7f96277fe700 1 -- 172.21.2.102:0/4055324452 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f962809d5e0 con 0x7f9628188e20 2021-02-10T11:27:14.174 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:14.170 7f96277fe700 1 --2- 172.21.2.102:0/4055324452 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9628188e20 0x7f9628189220 secure :-1 s=READY pgs=1710 cs=0 l=1 rev1=1 rx=0x7f961400b5e0 tx=0x7f961400b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:14.174 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:14.170 7f9624ff9700 1 -- 172.21.2.102:0/4055324452 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f961400bd70 con 0x7f9628188e20 2021-02-10T11:27:14.174 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:14.170 7f9624ff9700 1 -- 172.21.2.102:0/4055324452 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9614002030 con 0x7f9628188e20 2021-02-10T11:27:14.174 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:14.170 7f9624ff9700 1 -- 172.21.2.102:0/4055324452 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9614002550 con 0x7f9628188e20 2021-02-10T11:27:14.174 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:14.170 7f962e2db700 1 -- 172.21.2.102:0/4055324452 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f962809d5e0 con 0x7f9628188e20 2021-02-10T11:27:14.174 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:14.170 7f962e2db700 1 -- 172.21.2.102:0/4055324452 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f962809d5e0 con 0x7f9628188e20 2021-02-10T11:27:14.174 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:14.171 7f9624ff9700 1 -- 172.21.2.102:0/4055324452 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f9614000c90 con 0x7f9628188e20 2021-02-10T11:27:14.175 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:14.171 7f9624ff9700 1 --2- 172.21.2.102:0/4055324452 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f96040186c0 0x7f960401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:14.175 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:14.171 7f9624ff9700 1 -- 172.21.2.102:0/4055324452 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f9614032590 con 0x7f9628188e20 2021-02-10T11:27:14.175 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:14.171 7f9627fff700 1 --2- 172.21.2.102:0/4055324452 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f96040186c0 0x7f960401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:14.175 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:14.171 7f9627fff700 1 --2- 172.21.2.102:0/4055324452 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f96040186c0 0x7f960401abc0 crc :-1 s=READY pgs=774 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:14.202 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:14.203 7f962e2db700 1 -- 172.21.2.102:0/4055324452 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f962809db70 con 0x7f9628188e20 2021-02-10T11:27:14.205 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:14.205 7f9624ff9700 1 -- 172.21.2.102:0/4055324452 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f9614030d40 con 0x7f9628188e20 2021-02-10T11:27:14.334 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:14.334 7f962e2db700 1 -- 172.21.2.102:0/4055324452 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume snapshot create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000414869", "target": ["mgr", ""], "snap_name": "snapshot_0000000000810191"}) v1 -- 0x7f9628082a70 con 0x7f96040186c0 2021-02-10T11:27:14.345 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:14.345 7f9624ff9700 1 -- 172.21.2.102:0/4055324452 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f9618006a70 con 0x7f96040186c0 2021-02-10T11:27:14.350 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:14.350 7f962e2db700 1 -- 172.21.2.102:0/4055324452 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f96040186c0 msgr2=0x7f960401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:14.350 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:14.350 7f962e2db700 1 --2- 172.21.2.102:0/4055324452 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f96040186c0 0x7f960401abc0 crc :-1 s=READY pgs=774 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:14.350 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:14.350 7f962e2db700 1 -- 172.21.2.102:0/4055324452 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9628188e20 msgr2=0x7f9628189220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:14.351 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:14.350 7f962e2db700 1 --2- 172.21.2.102:0/4055324452 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9628188e20 0x7f9628189220 secure :-1 s=READY pgs=1710 cs=0 l=1 rev1=1 rx=0x7f961400b5e0 tx=0x7f961400b980).stop 2021-02-10T11:27:14.351 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:14.351 7f962e2db700 1 -- 172.21.2.102:0/4055324452 shutdown_connections 2021-02-10T11:27:14.351 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:14.351 7f962e2db700 1 --2- 172.21.2.102:0/4055324452 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f96040186c0 0x7f960401abc0 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:14.351 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:14.351 7f962e2db700 1 --2- 172.21.2.102:0/4055324452 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f962809cbc0 0x7f962809cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:14.351 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:14.351 7f962e2db700 1 --2- 172.21.2.102:0/4055324452 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9628188e20 0x7f9628189220 unknown :-1 s=CLOSED pgs=1710 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:14.351 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:14.351 7f962e2db700 1 --2- 172.21.2.102:0/4055324452 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9628188400 0x7f9628188800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:14.352 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:14.351 7f962e2db700 1 -- 172.21.2.102:0/4055324452 >> 172.21.2.102:0/4055324452 conn(0x7f96280c2cd0 msgr2=0x7f96280b5950 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:14.352 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:14.351 7f962e2db700 1 -- 172.21.2.102:0/4055324452 shutdown_connections 2021-02-10T11:27:14.352 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:14.351 7f962e2db700 1 -- 172.21.2.102:0/4055324452 wait complete. 2021-02-10T11:27:14.359 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume snapshot clone cephfs subvolume_0000000000414869 snapshot_0000000000810191 clone_0000000000602982 2021-02-10T11:27:14.791 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:14.776 7f6f69e2e700 1 Processor -- start 2021-02-10T11:27:14.791 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:14.776 7f6f69e2e700 1 -- start start 2021-02-10T11:27:14.792 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:14.776 7f6f69e2e700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f6f640eb220 0x7f6f640eb620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:14.792 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:14.776 7f6f69e2e700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6f6410aad0 0x7f6f6410aed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:14.792 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:14.776 7f6f69e2e700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f6410b4f0 con 0x7f6f640dce80 2021-02-10T11:27:14.792 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:14.776 7f6f69e2e700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f6410a550 con 0x7f6f640eb220 2021-02-10T11:27:14.792 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:14.776 7f6f69e2e700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f640881d0 con 0x7f6f6410aad0 2021-02-10T11:27:14.792 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:14.776 7f6f62ffd700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f6f640eb220 0x7f6f640eb620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:14.792 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:14.776 7f6f62ffd700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f6f640eb220 0x7f6f640eb620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:59362/0 (socket says 172.21.2.102:59362) 2021-02-10T11:27:14.792 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:14.776 7f6f62ffd700 1 -- 172.21.2.102:0/1089482725 learned_addr learned my addr 172.21.2.102:0/1089482725 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:27:14.792 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:14.776 7f6f63fff700 1 --2- 172.21.2.102:0/1089482725 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6f6410aad0 0x7f6f6410aed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:14.792 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:14.776 7f6f62ffd700 1 -- 172.21.2.102:0/1089482725 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6f6410aad0 msgr2=0x7f6f6410aed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:27:14.792 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:14.776 7f6f62ffd700 1 --2- 172.21.2.102:0/1089482725 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6f6410aad0 0x7f6f6410aed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:14.792 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:14.776 7f6f62ffd700 1 -- 172.21.2.102:0/1089482725 >> v1:172.21.2.113:6789/0 conn(0x7f6f640dce80 legacy=0x7f6f640d14f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:14.792 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:14.776 7f6f62ffd700 1 -- 172.21.2.102:0/1089482725 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f640d3370 con 0x7f6f640eb220 2021-02-10T11:27:14.793 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:14.777 7f6f63fff700 1 --2- 172.21.2.102:0/1089482725 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6f6410aad0 0x7f6f6410aed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:27:14.793 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:14.777 7f6f62ffd700 1 --2- 172.21.2.102:0/1089482725 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6f640eb220 0x7f6f640eb620 secure :-1 s=READY pgs=1711 cs=0 l=1 rev1=1 rx=0x7f6f5c006a80 tx=0x7f6f5c009f00).ready entity=mon.0 client_cookie=a51354489bbbf89 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:14.793 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:14.777 7f6f627fc700 1 -- 172.21.2.102:0/1089482725 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6f5c00b3b0 con 0x7f6f640eb220 2021-02-10T11:27:14.793 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:14.777 7f6f627fc700 1 -- 172.21.2.102:0/1089482725 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f5c00b5b0 con 0x7f6f640eb220 2021-02-10T11:27:14.793 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:14.777 7f6f627fc700 1 -- 172.21.2.102:0/1089482725 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6f5c00ba00 con 0x7f6f640eb220 2021-02-10T11:27:14.793 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:14.777 7f6f69e2e700 1 -- 172.21.2.102:0/1089482725 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6f640eb220 msgr2=0x7f6f640eb620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:14.793 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:14.777 7f6f69e2e700 1 --2- 172.21.2.102:0/1089482725 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6f640eb220 0x7f6f640eb620 secure :-1 s=READY pgs=1711 cs=0 l=1 rev1=1 rx=0x7f6f5c006a80 tx=0x7f6f5c009f00).stop 2021-02-10T11:27:14.793 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:14.777 7f6f69e2e700 1 -- 172.21.2.102:0/1089482725 shutdown_connections 2021-02-10T11:27:14.793 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:14.777 7f6f69e2e700 1 --2- 172.21.2.102:0/1089482725 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6f6410aad0 0x7f6f6410aed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:14.793 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:14.777 7f6f69e2e700 1 --2- 172.21.2.102:0/1089482725 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6f640eb220 0x7f6f640eb620 unknown :-1 s=CLOSED pgs=1711 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:14.793 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:14.777 7f6f69e2e700 1 -- 172.21.2.102:0/1089482725 >> 172.21.2.102:0/1089482725 conn(0x7f6f640ea630 msgr2=0x7f6f640d0800 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:14.793 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:14.777 7f6f69e2e700 1 -- 172.21.2.102:0/1089482725 shutdown_connections 2021-02-10T11:27:14.793 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:14.777 7f6f69e2e700 1 -- 172.21.2.102:0/1089482725 wait complete. 2021-02-10T11:27:14.795 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:14.792 7f6f69e2e700 1 Processor -- start 2021-02-10T11:27:14.795 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:14.792 7f6f69e2e700 1 -- start start 2021-02-10T11:27:14.795 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:14.792 7f6f69e2e700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6f64094310 0x7f6f64094710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:14.795 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:14.792 7f6f69e2e700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6f6419a660 0x7f6f64095870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:14.795 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:14.792 7f6f69e2e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6f64094d30 0x7f6f64095e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:14.795 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:14.792 7f6f69e2e700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f64096df0 con 0x7f6f64094d30 2021-02-10T11:27:14.795 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:14.792 7f6f69e2e700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f640952f0 con 0x7f6f6419a660 2021-02-10T11:27:14.795 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:14.792 7f6f69e2e700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f641b4c80 con 0x7f6f64094310 2021-02-10T11:27:14.795 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:14.793 7f6f63fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6f64094d30 0x7f6f64095e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:14.796 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:14.793 7f6f62ffd700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6f6419a660 0x7f6f64095870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:14.796 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:14.793 7f6f63fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6f64094d30 0x7f6f64095e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:59372/0 (socket says 172.21.2.102:59372) 2021-02-10T11:27:14.796 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:14.793 7f6f63fff700 1 -- 172.21.2.102:0/3541415043 learned_addr learned my addr 172.21.2.102:0/3541415043 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:27:14.796 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:14.793 7f6f62ffd700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6f6419a660 0x7f6f64095870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.113:3300/0 says I am v2:172.21.2.102:34566/0 (socket says 172.21.2.102:34566) 2021-02-10T11:27:14.796 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:14.793 7f6f637fe700 1 --2- 172.21.2.102:0/3541415043 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6f64094310 0x7f6f64094710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:14.796 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:14.793 7f6f62ffd700 1 -- 172.21.2.102:0/3541415043 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6f64094310 msgr2=0x7f6f64094710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:14.796 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:14.793 7f6f62ffd700 1 --2- 172.21.2.102:0/3541415043 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6f64094310 0x7f6f64094710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:14.796 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:14.793 7f6f62ffd700 1 -- 172.21.2.102:0/3541415043 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6f64094d30 msgr2=0x7f6f64095e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:14.796 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:14.793 7f6f62ffd700 1 --2- 172.21.2.102:0/3541415043 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6f64094d30 0x7f6f64095e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:14.796 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:14.793 7f6f62ffd700 1 -- 172.21.2.102:0/3541415043 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f641b4e60 con 0x7f6f6419a660 2021-02-10T11:27:14.796 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:14.793 7f6f62ffd700 1 --2- 172.21.2.102:0/3541415043 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6f6419a660 0x7f6f64095870 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 rx=0x7f6f5c009f50 tx=0x7f6f5c00bc40).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:14.796 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:14.793 7f6f4bfff700 1 -- 172.21.2.102:0/3541415043 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6f5c009810 con 0x7f6f6419a660 2021-02-10T11:27:14.796 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:14.793 7f6f4bfff700 1 -- 172.21.2.102:0/3541415043 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f5c00c2f0 con 0x7f6f6419a660 2021-02-10T11:27:14.796 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:14.793 7f6f4bfff700 1 -- 172.21.2.102:0/3541415043 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6f5c00c7f0 con 0x7f6f6419a660 2021-02-10T11:27:14.797 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:14.793 7f6f69e2e700 1 -- 172.21.2.102:0/3541415043 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f641b4e60 con 0x7f6f6419a660 2021-02-10T11:27:14.797 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:14.793 7f6f69e2e700 1 -- 172.21.2.102:0/3541415043 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f641b4e60 con 0x7f6f6419a660 2021-02-10T11:27:14.797 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:14.794 7f6f4bfff700 1 -- 172.21.2.102:0/3541415043 <== mon.1 v2:172.21.2.113:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f6f5c001010 con 0x7f6f6419a660 2021-02-10T11:27:14.797 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:14.794 7f6f4bfff700 1 --2- 172.21.2.102:0/3541415043 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6f44018730 0x7f6f4401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:14.797 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:14.794 7f6f4bfff700 1 -- 172.21.2.102:0/3541415043 <== mon.1 v2:172.21.2.113:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f6f5c032ac0 con 0x7f6f6419a660 2021-02-10T11:27:14.797 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:14.794 7f6f637fe700 1 --2- 172.21.2.102:0/3541415043 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6f44018730 0x7f6f4401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:14.797 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:14.794 7f6f637fe700 1 --2- 172.21.2.102:0/3541415043 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6f44018730 0x7f6f4401ac10 crc :-1 s=READY pgs=775 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:14.825 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:14.825 7f6f69e2e700 1 -- 172.21.2.102:0/3541415043 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f640f93d0 con 0x7f6f6419a660 2021-02-10T11:27:14.828 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:14.828 7f6f4bfff700 1 -- 172.21.2.102:0/3541415043 <== mon.1 v2:172.21.2.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f6f5c00bd80 con 0x7f6f6419a660 2021-02-10T11:27:14.957 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:14.957 7f6f69e2e700 1 -- 172.21.2.102:0/3541415043 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "subvolume_0000000000414869", "target": ["mgr", ""], "snap_name": "snapshot_0000000000810191", "target_sub_name": "clone_0000000000602982"}) v1 -- 0x7f6f6410a550 con 0x7f6f44018730 2021-02-10T11:27:15.005 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:15.005 7f6f4bfff700 1 -- 172.21.2.102:0/3541415043 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f6f58006550 con 0x7f6f44018730 2021-02-10T11:27:15.022 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:15.022 7f6f69e2e700 1 -- 172.21.2.102:0/3541415043 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6f44018730 msgr2=0x7f6f4401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:15.023 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:15.022 7f6f69e2e700 1 --2- 172.21.2.102:0/3541415043 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6f44018730 0x7f6f4401ac10 crc :-1 s=READY pgs=775 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:15.023 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:15.022 7f6f69e2e700 1 -- 172.21.2.102:0/3541415043 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6f6419a660 msgr2=0x7f6f64095870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:15.023 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:15.022 7f6f69e2e700 1 --2- 172.21.2.102:0/3541415043 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6f6419a660 0x7f6f64095870 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 rx=0x7f6f5c009f50 tx=0x7f6f5c00bc40).stop 2021-02-10T11:27:15.023 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:15.023 7f6f69e2e700 1 -- 172.21.2.102:0/3541415043 shutdown_connections 2021-02-10T11:27:15.023 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:15.023 7f6f69e2e700 1 --2- 172.21.2.102:0/3541415043 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6f64094d30 0x7f6f64095e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:15.023 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:15.023 7f6f69e2e700 1 --2- 172.21.2.102:0/3541415043 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6f44018730 0x7f6f4401ac10 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:15.023 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:15.023 7f6f69e2e700 1 --2- 172.21.2.102:0/3541415043 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6f6419a660 0x7f6f64095870 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:15.023 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:15.023 7f6f69e2e700 1 --2- 172.21.2.102:0/3541415043 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6f64094310 0x7f6f64094710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:15.023 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:15.023 7f6f69e2e700 1 -- 172.21.2.102:0/3541415043 >> 172.21.2.102:0/3541415043 conn(0x7f6f641a8260 msgr2=0x7f6f641a7890 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:15.023 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:15.023 7f6f69e2e700 1 -- 172.21.2.102:0/3541415043 shutdown_connections 2021-02-10T11:27:15.023 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:15.023 7f6f69e2e700 1 -- 172.21.2.102:0/3541415043 wait complete. 2021-02-10T11:27:15.031 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs clone status cephfs clone_0000000000602982 2021-02-10T11:27:15.464 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:15.445 7fdc935d9700 1 Processor -- start 2021-02-10T11:27:15.464 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:15.445 7fdc935d9700 1 -- start start 2021-02-10T11:27:15.465 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:15.445 7fdc935d9700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fdc8c10aff0 0x7fdc8c0cf750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:15.465 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:15.445 7fdc935d9700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc8c0fe1c0 con 0x7fdc8c0d1c70 2021-02-10T11:27:15.465 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:15.445 7fdc935d9700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc8c1b0cb0 con 0x7fdc8c0d1870 2021-02-10T11:27:15.465 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:15.445 7fdc935d9700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc8c109a60 con 0x7fdc8c10aff0 2021-02-10T11:27:15.465 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:15.445 7fdc90bf4700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fdc8c0d1c70 0x7fdc8c0cef60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:51604/0 (socket says 172.21.2.102:51604) 2021-02-10T11:27:15.465 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:15.445 7fdc90bf4700 1 -- 172.21.2.102:0/746835916 learned_addr learned my addr 172.21.2.102:0/746835916 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:27:15.465 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:15.446 7fdc7ffff700 1 -- 172.21.2.102:0/746835916 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1059655662 0 0) 0x7fdc80000ce0 con 0x7fdc8c0d1c70 2021-02-10T11:27:15.465 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:15.446 7fdc7ffff700 1 -- 172.21.2.102:0/746835916 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc78002fe0 con 0x7fdc8c0d1c70 2021-02-10T11:27:15.465 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:15.446 7fdc91bf6700 1 --2- 172.21.2.102:0/746835916 >> v2:172.21.2.113:3300/0 conn(0x7fdc8c10aff0 0x7fdc8c0cf750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:15.465 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:15.446 7fdc7ffff700 1 -- 172.21.2.102:0/746835916 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 137913709 0 0) 0x7fdc800010e0 con 0x7fdc8c0d1c70 2021-02-10T11:27:15.465 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:15.446 7fdc7ffff700 1 -- 172.21.2.102:0/746835916 >> v2:172.21.2.113:3300/0 conn(0x7fdc8c10aff0 msgr2=0x7fdc8c0cf750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:27:15.465 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:15.446 7fdc7ffff700 1 --2- 172.21.2.102:0/746835916 >> v2:172.21.2.113:3300/0 conn(0x7fdc8c10aff0 0x7fdc8c0cf750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:15.465 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:15.446 7fdc7ffff700 1 -- 172.21.2.102:0/746835916 >> v1:172.21.2.113:6789/0 conn(0x7fdc8c0d1870 legacy=0x7fdc8c1b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:15.466 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:15.446 7fdc7ffff700 1 -- 172.21.2.102:0/746835916 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc8c10b3f0 con 0x7fdc8c0d1c70 2021-02-10T11:27:15.466 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:15.446 7fdc7ffff700 1 -- 172.21.2.102:0/746835916 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7fdc80001660 con 0x7fdc8c0d1c70 2021-02-10T11:27:15.466 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:15.446 7fdc7ffff700 1 -- 172.21.2.102:0/746835916 >> v1:172.21.2.102:6789/0 conn(0x7fdc8c0d1c70 legacy=0x7fdc8c0cef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:15.466 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:15.446 7fdc7ffff700 1 --2- 172.21.2.102:0/746835916 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdc78003a60 0x7fdc78005ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:15.466 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:15.446 7fdc7ffff700 1 --2- 172.21.2.102:0/746835916 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fdc78006790 0x7fdc78008cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:15.466 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:15.446 7fdc7ffff700 1 --2- 172.21.2.102:0/746835916 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdc78009590 0x7fdc7800bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:15.466 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:15.446 7fdc7ffff700 1 -- 172.21.2.102:0/746835916 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc7800c300 con 0x7fdc78009590 2021-02-10T11:27:15.466 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:15.446 7fdc7ffff700 1 -- 172.21.2.102:0/746835916 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc7800c520 con 0x7fdc78006790 2021-02-10T11:27:15.466 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:15.446 7fdc7ffff700 1 -- 172.21.2.102:0/746835916 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc7800c740 con 0x7fdc78003a60 2021-02-10T11:27:15.466 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:15.447 7fdc91bf6700 1 --2- 172.21.2.102:0/746835916 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdc78009590 0x7fdc7800bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:15.467 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:15.447 7fdc90bf4700 1 --2- 172.21.2.102:0/746835916 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fdc78006790 0x7fdc78008cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:15.467 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:15.447 7fdc913f5700 1 --2- 172.21.2.102:0/746835916 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdc78003a60 0x7fdc78005ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:15.467 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:15.447 7fdc91bf6700 1 -- 172.21.2.102:0/746835916 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdc78003a60 msgr2=0x7fdc78005ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:27:15.467 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:15.447 7fdc91bf6700 1 --2- 172.21.2.102:0/746835916 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdc78003a60 0x7fdc78005ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:15.467 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:15.447 7fdc91bf6700 1 -- 172.21.2.102:0/746835916 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fdc78006790 msgr2=0x7fdc78008cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:27:15.467 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:15.447 7fdc91bf6700 1 --2- 172.21.2.102:0/746835916 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fdc78006790 0x7fdc78008cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:15.467 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:15.447 7fdc91bf6700 1 -- 172.21.2.102:0/746835916 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc7800c9e0 con 0x7fdc78009590 2021-02-10T11:27:15.467 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:15.447 7fdc90bf4700 1 --2- 172.21.2.102:0/746835916 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fdc78006790 0x7fdc78008cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:27:15.467 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:15.447 7fdc91bf6700 1 -- 172.21.2.102:0/746835916 reap_dead start 2021-02-10T11:27:15.467 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:15.447 7fdc91bf6700 1 --2- 172.21.2.102:0/746835916 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdc78009590 0x7fdc7800bab0 secure :-1 s=READY pgs=1713 cs=0 l=1 rev1=1 rx=0x7fdc8404dc80 tx=0x7fdc8404f300).ready entity=mon.0 client_cookie=cea7f76030270637 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:15.467 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:15.447 7fdc7ffff700 1 -- 172.21.2.102:0/746835916 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdc8404f6f0 con 0x7fdc78009590 2021-02-10T11:27:15.467 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:15.447 7fdc7ffff700 1 -- 172.21.2.102:0/746835916 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc84003030 con 0x7fdc78009590 2021-02-10T11:27:15.468 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:15.447 7fdc7ffff700 1 -- 172.21.2.102:0/746835916 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdc8404fcb0 con 0x7fdc78009590 2021-02-10T11:27:15.468 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:15.447 7fdc935d9700 1 -- 172.21.2.102:0/746835916 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdc78009590 msgr2=0x7fdc7800bab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:15.468 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:15.447 7fdc935d9700 1 --2- 172.21.2.102:0/746835916 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdc78009590 0x7fdc7800bab0 secure :-1 s=READY pgs=1713 cs=0 l=1 rev1=1 rx=0x7fdc8404dc80 tx=0x7fdc8404f300).stop 2021-02-10T11:27:15.468 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:15.448 7fdc935d9700 1 -- 172.21.2.102:0/746835916 shutdown_connections 2021-02-10T11:27:15.468 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:15.448 7fdc935d9700 1 --2- 172.21.2.102:0/746835916 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdc78009590 0x7fdc7800bab0 unknown :-1 s=CLOSED pgs=1713 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:15.468 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:15.448 7fdc935d9700 1 -- 172.21.2.102:0/746835916 >> 172.21.2.102:0/746835916 conn(0x7fdc8c0ce420 msgr2=0x7fdc8c0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:15.468 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:15.448 7fdc935d9700 1 -- 172.21.2.102:0/746835916 shutdown_connections 2021-02-10T11:27:15.468 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:15.448 7fdc935d9700 1 -- 172.21.2.102:0/746835916 wait complete. 2021-02-10T11:27:15.468 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:15.463 7fdc935d9700 1 Processor -- start 2021-02-10T11:27:15.468 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:15.463 7fdc935d9700 1 -- start start 2021-02-10T11:27:15.468 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:15.463 7fdc935d9700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdc8c085bc0 0x7fdc8c0863e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:15.468 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:15.463 7fdc935d9700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdc8c0954c0 0x7fdc8c096300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:15.468 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:15.463 7fdc935d9700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fdc8c0958c0 0x7fdc8c096920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:15.469 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:15.463 7fdc935d9700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc8c095d80 con 0x7fdc8c085bc0 2021-02-10T11:27:15.469 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:15.463 7fdc935d9700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc8c0bfa60 con 0x7fdc8c0958c0 2021-02-10T11:27:15.469 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:15.463 7fdc935d9700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc8c085fc0 con 0x7fdc8c0954c0 2021-02-10T11:27:15.469 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:15.463 7fdc913f5700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdc8c085bc0 0x7fdc8c0863e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:15.469 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:15.463 7fdc913f5700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdc8c085bc0 0x7fdc8c0863e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:59394/0 (socket says 172.21.2.102:59394) 2021-02-10T11:27:15.469 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:15.463 7fdc913f5700 1 -- 172.21.2.102:0/1388918379 learned_addr learned my addr 172.21.2.102:0/1388918379 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:27:15.469 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:15.464 7fdc913f5700 1 -- 172.21.2.102:0/1388918379 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdc8c0954c0 msgr2=0x7fdc8c096300 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:27:15.469 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:15.464 7fdc913f5700 1 --2- 172.21.2.102:0/1388918379 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdc8c0954c0 0x7fdc8c096300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:15.469 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:15.464 7fdc913f5700 1 -- 172.21.2.102:0/1388918379 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fdc8c0958c0 msgr2=0x7fdc8c096920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:15.469 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:15.464 7fdc913f5700 1 --2- 172.21.2.102:0/1388918379 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fdc8c0958c0 0x7fdc8c096920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:15.469 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:15.464 7fdc913f5700 1 -- 172.21.2.102:0/1388918379 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc8c086c10 con 0x7fdc8c085bc0 2021-02-10T11:27:15.469 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:15.464 7fdc913f5700 1 --2- 172.21.2.102:0/1388918379 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdc8c085bc0 0x7fdc8c0863e0 secure :-1 s=READY pgs=1714 cs=0 l=1 rev1=1 rx=0x7fdc88006f10 tx=0x7fdc8800b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:15.469 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:15.464 7fdc7dffb700 1 -- 172.21.2.102:0/1388918379 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdc88000960 con 0x7fdc8c085bc0 2021-02-10T11:27:15.470 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:15.464 7fdc7dffb700 1 -- 172.21.2.102:0/1388918379 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc8800be10 con 0x7fdc8c085bc0 2021-02-10T11:27:15.470 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:15.464 7fdc935d9700 1 -- 172.21.2.102:0/1388918379 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc8c086c10 con 0x7fdc8c085bc0 2021-02-10T11:27:15.470 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:15.464 7fdc935d9700 1 -- 172.21.2.102:0/1388918379 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc8c097230 con 0x7fdc8c085bc0 2021-02-10T11:27:15.470 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:15.464 7fdc7dffb700 1 -- 172.21.2.102:0/1388918379 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdc880062e0 con 0x7fdc8c085bc0 2021-02-10T11:27:15.470 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:15.465 7fdc7dffb700 1 -- 172.21.2.102:0/1388918379 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fdc88006030 con 0x7fdc8c085bc0 2021-02-10T11:27:15.471 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:15.465 7fdc7dffb700 1 --2- 172.21.2.102:0/1388918379 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fdc6c0186d0 0x7fdc6c01abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:15.471 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:15.465 7fdc7dffb700 1 -- 172.21.2.102:0/1388918379 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fdc88032600 con 0x7fdc8c085bc0 2021-02-10T11:27:15.471 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:15.465 7fdc90bf4700 1 --2- 172.21.2.102:0/1388918379 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fdc6c0186d0 0x7fdc6c01abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:15.471 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:15.466 7fdc90bf4700 1 --2- 172.21.2.102:0/1388918379 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fdc6c0186d0 0x7fdc6c01abd0 crc :-1 s=READY pgs=776 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:15.495 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:15.496 7fdc935d9700 1 -- 172.21.2.102:0/1388918379 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc8c0fe1c0 con 0x7fdc8c085bc0 2021-02-10T11:27:15.498 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:15.498 7fdc7dffb700 1 -- 172.21.2.102:0/1388918379 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fdc88000940 con 0x7fdc8c085bc0 2021-02-10T11:27:15.636 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:15.636 7fdc935d9700 1 -- 172.21.2.102:0/1388918379 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "clone_0000000000602982", "target": ["mgr", ""]}) v1 -- 0x7fdc8c097230 con 0x7fdc6c0186d0 2021-02-10T11:27:15.639 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:15.639 7fdc7dffb700 1 -- 172.21.2.102:0/1388918379 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+46 (crc 0 0 0) 0x7fdc80001910 con 0x7fdc6c0186d0 2021-02-10T11:27:15.639 INFO:teuthology.orchestra.run.gibba002.stdout:{ 2021-02-10T11:27:15.640 INFO:teuthology.orchestra.run.gibba002.stdout: "status": { 2021-02-10T11:27:15.640 INFO:teuthology.orchestra.run.gibba002.stdout: "state": "complete" 2021-02-10T11:27:15.640 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:27:15.640 INFO:teuthology.orchestra.run.gibba002.stdout:} 2021-02-10T11:27:15.641 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:15.641 7fdc935d9700 1 -- 172.21.2.102:0/1388918379 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fdc6c0186d0 msgr2=0x7fdc6c01abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:15.641 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:15.641 7fdc935d9700 1 --2- 172.21.2.102:0/1388918379 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fdc6c0186d0 0x7fdc6c01abd0 crc :-1 s=READY pgs=776 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:15.641 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:15.641 7fdc935d9700 1 -- 172.21.2.102:0/1388918379 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdc8c085bc0 msgr2=0x7fdc8c0863e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:15.641 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:15.641 7fdc935d9700 1 --2- 172.21.2.102:0/1388918379 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdc8c085bc0 0x7fdc8c0863e0 secure :-1 s=READY pgs=1714 cs=0 l=1 rev1=1 rx=0x7fdc88006f10 tx=0x7fdc8800b990).stop 2021-02-10T11:27:15.642 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:15.641 7fdc935d9700 1 -- 172.21.2.102:0/1388918379 shutdown_connections 2021-02-10T11:27:15.642 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:15.641 7fdc935d9700 1 --2- 172.21.2.102:0/1388918379 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fdc6c0186d0 0x7fdc6c01abd0 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:15.642 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:15.641 7fdc935d9700 1 --2- 172.21.2.102:0/1388918379 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fdc8c0958c0 0x7fdc8c096920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:15.642 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:15.641 7fdc935d9700 1 --2- 172.21.2.102:0/1388918379 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdc8c0954c0 0x7fdc8c096300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:15.643 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:15.641 7fdc935d9700 1 --2- 172.21.2.102:0/1388918379 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdc8c085bc0 0x7fdc8c0863e0 unknown :-1 s=CLOSED pgs=1714 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:15.643 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:15.641 7fdc935d9700 1 -- 172.21.2.102:0/1388918379 >> 172.21.2.102:0/1388918379 conn(0x7fdc8c0bef00 msgr2=0x7fdc8c09d1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:15.643 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:15.641 7fdc935d9700 1 -- 172.21.2.102:0/1388918379 shutdown_connections 2021-02-10T11:27:15.643 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:15.641 7fdc935d9700 1 -- 172.21.2.102:0/1388918379 wait complete. 2021-02-10T11:27:15.649 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume snapshot rm cephfs subvolume_0000000000414869 snapshot_0000000000810191 2021-02-10T11:27:16.087 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:16.064 7f8e75a34700 1 Processor -- start 2021-02-10T11:27:16.087 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:16.064 7f8e75a34700 1 -- start start 2021-02-10T11:27:16.087 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:16.065 7f8e75a34700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f8e700d1870 0x7f8e701b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:16.087 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:16.065 7f8e75a34700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8e701b0ae0 0x7f8e70108940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:16.088 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:16.065 7f8e75a34700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f8e700d1c70 0x7f8e70108f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:16.088 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:16.065 7f8e75a34700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e700fe1c0 con 0x7f8e700d1870 2021-02-10T11:27:16.088 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:16.065 7f8e75a34700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e70109a60 con 0x7f8e700d1c70 2021-02-10T11:27:16.088 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:16.065 7f8e75a34700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e70088070 con 0x7f8e701b0ae0 2021-02-10T11:27:16.088 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:16.065 7f8e6f7fe700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f8e700d1870 0x7f8e701b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:16.088 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:16.065 7f8e6f7fe700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f8e700d1870 0x7f8e701b04c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:59400/0 (socket says 172.21.2.102:59400) 2021-02-10T11:27:16.090 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:16.065 7f8e6f7fe700 1 -- 172.21.2.102:0/556734220 learned_addr learned my addr 172.21.2.102:0/556734220 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:27:16.090 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:16.065 7f8e6ffff700 1 --2- 172.21.2.102:0/556734220 >> v2:172.21.2.113:3300/0 conn(0x7f8e700d1c70 0x7f8e70108f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:16.091 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:16.065 7f8e6effd700 1 --2- 172.21.2.102:0/556734220 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8e701b0ae0 0x7f8e70108940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:16.091 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:16.065 7f8e6f7fe700 1 -- 172.21.2.102:0/556734220 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8e701b0ae0 msgr2=0x7f8e70108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:27:16.091 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:16.065 7f8e6f7fe700 1 --2- 172.21.2.102:0/556734220 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8e701b0ae0 0x7f8e70108940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:16.091 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:16.066 7f8e6f7fe700 1 -- 172.21.2.102:0/556734220 >> v2:172.21.2.113:3300/0 conn(0x7f8e700d1c70 msgr2=0x7f8e70108f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:27:16.091 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:16.066 7f8e6f7fe700 1 --2- 172.21.2.102:0/556734220 >> v2:172.21.2.113:3300/0 conn(0x7f8e700d1c70 0x7f8e70108f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:16.091 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:16.066 7f8e6f7fe700 1 -- 172.21.2.102:0/556734220 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e70109580 con 0x7f8e700d1870 2021-02-10T11:27:16.091 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:16.066 7f8e6ffff700 1 --2- 172.21.2.102:0/556734220 >> v2:172.21.2.113:3300/0 conn(0x7f8e700d1c70 0x7f8e70108f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:27:16.091 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:16.066 7f8e6f7fe700 1 --2- 172.21.2.102:0/556734220 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8e700d1870 0x7f8e701b04c0 secure :-1 s=READY pgs=1715 cs=0 l=1 rev1=1 rx=0x7f8e60006700 tx=0x7f8e60009b00).ready entity=mon.0 client_cookie=a7ba06e8ad51b1f1 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:16.092 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:16.066 7f8e6e7fc700 1 -- 172.21.2.102:0/556734220 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8e6000c0c0 con 0x7f8e700d1870 2021-02-10T11:27:16.092 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:16.066 7f8e6e7fc700 1 -- 172.21.2.102:0/556734220 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e6000c400 con 0x7f8e700d1870 2021-02-10T11:27:16.092 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:16.066 7f8e6e7fc700 1 -- 172.21.2.102:0/556734220 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8e6000c920 con 0x7f8e700d1870 2021-02-10T11:27:16.092 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:16.066 7f8e75a34700 1 -- 172.21.2.102:0/556734220 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8e700d1870 msgr2=0x7f8e701b04c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:16.092 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:16.066 7f8e75a34700 1 --2- 172.21.2.102:0/556734220 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8e700d1870 0x7f8e701b04c0 secure :-1 s=READY pgs=1715 cs=0 l=1 rev1=1 rx=0x7f8e60006700 tx=0x7f8e60009b00).stop 2021-02-10T11:27:16.092 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:16.067 7f8e75a34700 1 -- 172.21.2.102:0/556734220 shutdown_connections 2021-02-10T11:27:16.092 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:16.067 7f8e75a34700 1 --2- 172.21.2.102:0/556734220 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8e701b0ae0 0x7f8e70108940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:16.092 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:16.067 7f8e75a34700 1 --2- 172.21.2.102:0/556734220 >> v2:172.21.2.113:3300/0 conn(0x7f8e700d1c70 0x7f8e70108f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:16.093 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:16.067 7f8e75a34700 1 --2- 172.21.2.102:0/556734220 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8e700d1870 0x7f8e701b04c0 unknown :-1 s=CLOSED pgs=1715 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:16.093 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:16.067 7f8e75a34700 1 -- 172.21.2.102:0/556734220 >> 172.21.2.102:0/556734220 conn(0x7f8e700ce420 msgr2=0x7f8e700d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:16.093 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:16.067 7f8e75a34700 1 -- 172.21.2.102:0/556734220 shutdown_connections 2021-02-10T11:27:16.093 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:16.067 7f8e75a34700 1 -- 172.21.2.102:0/556734220 wait complete. 2021-02-10T11:27:16.093 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:16.082 7f8e75a34700 1 Processor -- start 2021-02-10T11:27:16.093 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:16.082 7f8e75a34700 1 -- start start 2021-02-10T11:27:16.094 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:16.082 7f8e75a34700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8e700b8a80 0x7f8e70190740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:16.094 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:16.082 7f8e75a34700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8e701a76c0 0x7f8e701a7ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:16.094 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:16.082 7f8e75a34700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8e701a80e0 0x7f8e700834e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:16.094 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:16.082 7f8e75a34700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e701a84e0 con 0x7f8e700b8a80 2021-02-10T11:27:16.094 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:16.083 7f8e75a34700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e700910d0 con 0x7f8e701a76c0 2021-02-10T11:27:16.094 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:16.083 7f8e75a34700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e700b8e80 con 0x7f8e701a80e0 2021-02-10T11:27:16.095 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:16.083 7f8e6f7fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8e700b8a80 0x7f8e70190740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:16.095 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:16.083 7f8e6f7fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8e700b8a80 0x7f8e70190740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:59404/0 (socket says 172.21.2.102:59404) 2021-02-10T11:27:16.095 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:16.083 7f8e6f7fe700 1 -- 172.21.2.102:0/1218296935 learned_addr learned my addr 172.21.2.102:0/1218296935 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:27:16.100 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:16.083 7f8e6f7fe700 1 -- 172.21.2.102:0/1218296935 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8e701a80e0 msgr2=0x7f8e700834e0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:27:16.100 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:16.083 7f8e6f7fe700 1 --2- 172.21.2.102:0/1218296935 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8e701a80e0 0x7f8e700834e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:16.100 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:16.083 7f8e6f7fe700 1 -- 172.21.2.102:0/1218296935 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8e701a76c0 msgr2=0x7f8e701a7ac0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:27:16.100 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:16.083 7f8e6f7fe700 1 --2- 172.21.2.102:0/1218296935 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8e701a76c0 0x7f8e701a7ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:16.100 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:16.083 7f8e6f7fe700 1 -- 172.21.2.102:0/1218296935 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e70083b00 con 0x7f8e700b8a80 2021-02-10T11:27:16.100 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:16.083 7f8e6f7fe700 1 --2- 172.21.2.102:0/1218296935 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8e700b8a80 0x7f8e70190740 secure :-1 s=READY pgs=1716 cs=0 l=1 rev1=1 rx=0x7f8e6000b660 tx=0x7f8e6000ba00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:16.101 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:16.083 7f8e57fff700 1 -- 172.21.2.102:0/1218296935 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8e60009440 con 0x7f8e700b8a80 2021-02-10T11:27:16.101 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:16.083 7f8e57fff700 1 -- 172.21.2.102:0/1218296935 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e60006390 con 0x7f8e700b8a80 2021-02-10T11:27:16.102 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:16.083 7f8e57fff700 1 -- 172.21.2.102:0/1218296935 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8e60002030 con 0x7f8e700b8a80 2021-02-10T11:27:16.102 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:16.083 7f8e75a34700 1 -- 172.21.2.102:0/1218296935 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e70083b00 con 0x7f8e700b8a80 2021-02-10T11:27:16.103 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:16.083 7f8e75a34700 1 -- 172.21.2.102:0/1218296935 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e70084090 con 0x7f8e700b8a80 2021-02-10T11:27:16.104 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:16.084 7f8e57fff700 1 -- 172.21.2.102:0/1218296935 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f8e600093e0 con 0x7f8e700b8a80 2021-02-10T11:27:16.104 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:16.084 7f8e57fff700 1 --2- 172.21.2.102:0/1218296935 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8e50018730 0x7f8e5001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:16.105 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:16.084 7f8e57fff700 1 -- 172.21.2.102:0/1218296935 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f8e60032660 con 0x7f8e700b8a80 2021-02-10T11:27:16.105 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:16.084 7f8e6effd700 1 --2- 172.21.2.102:0/1218296935 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8e50018730 0x7f8e5001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:16.105 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:16.084 7f8e6effd700 1 --2- 172.21.2.102:0/1218296935 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8e50018730 0x7f8e5001abc0 crc :-1 s=READY pgs=777 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:16.115 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:16.114 7f8e75a34700 1 -- 172.21.2.102:0/1218296935 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e700825d0 con 0x7f8e700b8a80 2021-02-10T11:27:16.117 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:16.117 7f8e57fff700 1 -- 172.21.2.102:0/1218296935 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f8e6001f520 con 0x7f8e700b8a80 2021-02-10T11:27:16.245 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:16.245 7f8e75a34700 1 -- 172.21.2.102:0/1218296935 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume snapshot rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000414869", "target": ["mgr", ""], "snap_name": "snapshot_0000000000810191"}) v1 -- 0x7f8e701a84e0 con 0x7f8e50018730 2021-02-10T11:27:16.256 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:16.255 7f8e57fff700 1 -- 172.21.2.102:0/1218296935 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f8e64000c90 con 0x7f8e50018730 2021-02-10T11:27:16.263 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:16.261 7f8e75a34700 1 -- 172.21.2.102:0/1218296935 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8e50018730 msgr2=0x7f8e5001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:16.263 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:16.261 7f8e75a34700 1 --2- 172.21.2.102:0/1218296935 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8e50018730 0x7f8e5001abc0 crc :-1 s=READY pgs=777 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:16.263 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:16.261 7f8e75a34700 1 -- 172.21.2.102:0/1218296935 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8e700b8a80 msgr2=0x7f8e70190740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:16.263 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:16.261 7f8e75a34700 1 --2- 172.21.2.102:0/1218296935 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8e700b8a80 0x7f8e70190740 secure :-1 s=READY pgs=1716 cs=0 l=1 rev1=1 rx=0x7f8e6000b660 tx=0x7f8e6000ba00).stop 2021-02-10T11:27:16.264 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:16.261 7f8e75a34700 1 -- 172.21.2.102:0/1218296935 shutdown_connections 2021-02-10T11:27:16.264 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:16.261 7f8e75a34700 1 --2- 172.21.2.102:0/1218296935 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8e701a80e0 0x7f8e700834e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:16.264 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:16.261 7f8e75a34700 1 --2- 172.21.2.102:0/1218296935 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8e50018730 0x7f8e5001abc0 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:16.264 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:16.261 7f8e75a34700 1 --2- 172.21.2.102:0/1218296935 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8e701a76c0 0x7f8e701a7ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:16.264 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:16.261 7f8e75a34700 1 --2- 172.21.2.102:0/1218296935 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8e700b8a80 0x7f8e70190740 unknown :-1 s=CLOSED pgs=1716 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:16.264 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:16.261 7f8e75a34700 1 -- 172.21.2.102:0/1218296935 >> 172.21.2.102:0/1218296935 conn(0x7f8e700a2bc0 msgr2=0x7f8e700aaeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:16.265 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:16.262 7f8e75a34700 1 -- 172.21.2.102:0/1218296935 shutdown_connections 2021-02-10T11:27:16.265 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:16.262 7f8e75a34700 1 -- 172.21.2.102:0/1218296935 wait complete. 2021-02-10T11:27:16.271 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume info cephfs clone_0000000000602982 2021-02-10T11:27:16.704 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:16.687 7f893bd15700 1 Processor -- start 2021-02-10T11:27:16.705 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:16.687 7f893bd15700 1 -- start start 2021-02-10T11:27:16.705 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:16.688 7f893bd15700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f89340d1c70 0x7f89340cef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:16.705 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:16.688 7f893bd15700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f89340cf580 0x7f89340cf980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:16.705 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:16.688 7f893bd15700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89340fe1c0 con 0x7f89340d1870 2021-02-10T11:27:16.705 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:16.688 7f893bd15700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89341b0cb0 con 0x7f89340d1c70 2021-02-10T11:27:16.705 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:16.688 7f893bd15700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8934109a60 con 0x7f89340cf580 2021-02-10T11:27:16.705 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:16.688 7f8939330700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f89340d1c70 0x7f89340cef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:16.705 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:16.688 7f8939330700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f89340d1c70 0x7f89340cef60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:59412/0 (socket says 172.21.2.102:59412) 2021-02-10T11:27:16.706 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:16.688 7f8939330700 1 -- 172.21.2.102:0/4239737010 learned_addr learned my addr 172.21.2.102:0/4239737010 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:27:16.706 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:16.688 7f893a332700 1 --2- 172.21.2.102:0/4239737010 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f89340cf580 0x7f89340cf980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:16.706 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:16.688 7f8939330700 1 -- 172.21.2.102:0/4239737010 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f89340cf580 msgr2=0x7f89340cf980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:27:16.706 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:16.689 7f8939330700 1 --2- 172.21.2.102:0/4239737010 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f89340cf580 0x7f89340cf980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:16.706 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:16.689 7f8939330700 1 -- 172.21.2.102:0/4239737010 >> v1:172.21.2.113:6789/0 conn(0x7f89340d1870 legacy=0x7f89341b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:16.706 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:16.689 7f8939330700 1 -- 172.21.2.102:0/4239737010 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8934088070 con 0x7f89340d1c70 2021-02-10T11:27:16.706 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:16.689 7f8939330700 1 --2- 172.21.2.102:0/4239737010 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f89340d1c70 0x7f89340cef60 secure :-1 s=READY pgs=1717 cs=0 l=1 rev1=1 rx=0x7f8928006700 tx=0x7f8928009b80).ready entity=mon.0 client_cookie=82efb398023303ca server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:16.706 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:16.689 7f8938b2f700 1 -- 172.21.2.102:0/4239737010 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f892800c0c0 con 0x7f89340d1c70 2021-02-10T11:27:16.706 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:16.689 7f8938b2f700 1 -- 172.21.2.102:0/4239737010 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f892800c400 con 0x7f89340d1c70 2021-02-10T11:27:16.706 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:16.689 7f8938b2f700 1 -- 172.21.2.102:0/4239737010 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f892800c920 con 0x7f89340d1c70 2021-02-10T11:27:16.707 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:16.689 7f893bd15700 1 -- 172.21.2.102:0/4239737010 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f89340d1c70 msgr2=0x7f89340cef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:16.707 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:16.689 7f893bd15700 1 --2- 172.21.2.102:0/4239737010 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f89340d1c70 0x7f89340cef60 secure :-1 s=READY pgs=1717 cs=0 l=1 rev1=1 rx=0x7f8928006700 tx=0x7f8928009b80).stop 2021-02-10T11:27:16.707 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:16.690 7f893bd15700 1 -- 172.21.2.102:0/4239737010 shutdown_connections 2021-02-10T11:27:16.707 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:16.690 7f893bd15700 1 --2- 172.21.2.102:0/4239737010 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f89340cf580 0x7f89340cf980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:16.707 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:16.690 7f893bd15700 1 --2- 172.21.2.102:0/4239737010 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f89340d1c70 0x7f89340cef60 unknown :-1 s=CLOSED pgs=1717 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:16.707 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:16.690 7f893bd15700 1 -- 172.21.2.102:0/4239737010 >> 172.21.2.102:0/4239737010 conn(0x7f89340ce420 msgr2=0x7f89340d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:16.707 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:16.690 7f893bd15700 1 -- 172.21.2.102:0/4239737010 shutdown_connections 2021-02-10T11:27:16.707 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:16.690 7f893bd15700 1 -- 172.21.2.102:0/4239737010 wait complete. 2021-02-10T11:27:16.707 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:16.705 7f893bd15700 1 Processor -- start 2021-02-10T11:27:16.707 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:16.705 7f893bd15700 1 -- start start 2021-02-10T11:27:16.709 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:16.705 7f893bd15700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8934191b10 0x7f89340ae640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:16.709 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:16.705 7f893bd15700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8934191f10 0x7f893419ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:16.709 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:16.705 7f893bd15700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f89341a05f0 0x7f8934082ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:16.709 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:16.705 7f893bd15700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8934192440 con 0x7f8934191f10 2021-02-10T11:27:16.709 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:16.705 7f893bd15700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89341916e0 con 0x7f89341a05f0 2021-02-10T11:27:16.709 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:16.705 7f893bd15700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8934191900 con 0x7f8934191b10 2021-02-10T11:27:16.709 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:16.705 7f8939330700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8934191f10 0x7f893419ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:16.709 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:16.705 7f8939330700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8934191f10 0x7f893419ffd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:59418/0 (socket says 172.21.2.102:59418) 2021-02-10T11:27:16.709 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:16.705 7f8939330700 1 -- 172.21.2.102:0/421853503 learned_addr learned my addr 172.21.2.102:0/421853503 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:27:16.710 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:16.706 7f8939330700 1 -- 172.21.2.102:0/421853503 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8934191b10 msgr2=0x7f89340ae640 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:27:16.710 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:16.706 7f8939330700 1 --2- 172.21.2.102:0/421853503 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8934191b10 0x7f89340ae640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:16.710 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:16.706 7f8939330700 1 -- 172.21.2.102:0/421853503 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f89341a05f0 msgr2=0x7f8934082ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:16.710 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:16.706 7f8939330700 1 --2- 172.21.2.102:0/421853503 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f89341a05f0 0x7f8934082ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:16.710 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:16.706 7f8939330700 1 -- 172.21.2.102:0/421853503 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89340834f0 con 0x7f8934191f10 2021-02-10T11:27:16.710 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:16.706 7f8939330700 1 --2- 172.21.2.102:0/421853503 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8934191f10 0x7f893419ffd0 secure :-1 s=READY pgs=1718 cs=0 l=1 rev1=1 rx=0x7f8928006420 tx=0x7f892800b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:16.710 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:16.706 7f89267fc700 1 -- 172.21.2.102:0/421853503 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f89280090d0 con 0x7f8934191f10 2021-02-10T11:27:16.710 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:16.706 7f89267fc700 1 -- 172.21.2.102:0/421853503 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f892800bd50 con 0x7f8934191f10 2021-02-10T11:27:16.710 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:16.706 7f89267fc700 1 -- 172.21.2.102:0/421853503 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f89280054d0 con 0x7f8934191f10 2021-02-10T11:27:16.710 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:16.706 7f893bd15700 1 -- 172.21.2.102:0/421853503 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89340834f0 con 0x7f8934191f10 2021-02-10T11:27:16.710 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:16.706 7f893bd15700 1 -- 172.21.2.102:0/421853503 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89340834f0 con 0x7f8934191f10 2021-02-10T11:27:16.711 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:16.707 7f89267fc700 1 -- 172.21.2.102:0/421853503 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f8928009fd0 con 0x7f8934191f10 2021-02-10T11:27:16.711 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:16.707 7f89267fc700 1 --2- 172.21.2.102:0/421853503 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f891c0186c0 0x7f891c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:16.711 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:16.707 7f89267fc700 1 -- 172.21.2.102:0/421853503 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f89280323e0 con 0x7f8934191f10 2021-02-10T11:27:16.711 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:16.707 7f8939b31700 1 --2- 172.21.2.102:0/421853503 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f891c0186c0 0x7f891c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:16.711 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:16.708 7f8939b31700 1 --2- 172.21.2.102:0/421853503 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f891c0186c0 0x7f891c01abc0 crc :-1 s=READY pgs=778 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:16.738 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:16.738 7f893bd15700 1 -- 172.21.2.102:0/421853503 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8934191900 con 0x7f8934191f10 2021-02-10T11:27:16.740 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:16.740 7f89267fc700 1 -- 172.21.2.102:0/421853503 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f8928009030 con 0x7f8934191f10 2021-02-10T11:27:16.870 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:16.870 7f893bd15700 1 -- 172.21.2.102:0/421853503 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume info", "vol_name": "cephfs", "sub_name": "clone_0000000000602982", "target": ["mgr", ""]}) v1 -- 0x7f89341916e0 con 0x7f891c0186c0 2021-02-10T11:27:16.875 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:16.875 7f89267fc700 1 -- 172.21.2.102:0/421853503 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+724 (crc 0 0 0) 0x7f8930006e40 con 0x7f891c0186c0 2021-02-10T11:27:16.877 INFO:teuthology.orchestra.run.gibba002.stdout:{ 2021-02-10T11:27:16.877 INFO:teuthology.orchestra.run.gibba002.stdout: "atime": "2021-02-10 11:27:12", 2021-02-10T11:27:16.877 INFO:teuthology.orchestra.run.gibba002.stdout: "bytes_pcent": "undefined", 2021-02-10T11:27:16.877 INFO:teuthology.orchestra.run.gibba002.stdout: "bytes_quota": "infinite", 2021-02-10T11:27:16.877 INFO:teuthology.orchestra.run.gibba002.stdout: "bytes_used": 0, 2021-02-10T11:27:16.878 INFO:teuthology.orchestra.run.gibba002.stdout: "created_at": "2021-02-10 11:27:14", 2021-02-10T11:27:16.878 INFO:teuthology.orchestra.run.gibba002.stdout: "ctime": "2021-02-10 11:27:15", 2021-02-10T11:27:16.878 INFO:teuthology.orchestra.run.gibba002.stdout: "data_pool": "cephfs_data", 2021-02-10T11:27:16.878 INFO:teuthology.orchestra.run.gibba002.stdout: "features": [ 2021-02-10T11:27:16.878 INFO:teuthology.orchestra.run.gibba002.stdout: "snapshot-clone", 2021-02-10T11:27:16.878 INFO:teuthology.orchestra.run.gibba002.stdout: "snapshot-autoprotect", 2021-02-10T11:27:16.878 INFO:teuthology.orchestra.run.gibba002.stdout: "snapshot-retention" 2021-02-10T11:27:16.879 INFO:teuthology.orchestra.run.gibba002.stdout: ], 2021-02-10T11:27:16.879 INFO:teuthology.orchestra.run.gibba002.stdout: "gid": 0, 2021-02-10T11:27:16.879 INFO:teuthology.orchestra.run.gibba002.stdout: "mode": 16877, 2021-02-10T11:27:16.879 INFO:teuthology.orchestra.run.gibba002.stdout: "mon_addrs": [ 2021-02-10T11:27:16.879 INFO:teuthology.orchestra.run.gibba002.stdout: "172.21.2.102:6789", 2021-02-10T11:27:16.879 INFO:teuthology.orchestra.run.gibba002.stdout: "172.21.2.113:6789", 2021-02-10T11:27:16.879 INFO:teuthology.orchestra.run.gibba002.stdout: "172.21.2.113:6790" 2021-02-10T11:27:16.879 INFO:teuthology.orchestra.run.gibba002.stdout: ], 2021-02-10T11:27:16.879 INFO:teuthology.orchestra.run.gibba002.stdout: "mtime": "2021-02-10 11:27:13", 2021-02-10T11:27:16.879 INFO:teuthology.orchestra.run.gibba002.stdout: "path": "/volumes/_nogroup/clone_0000000000602982/d0e7dac0-74b6-49dc-8fb5-ed8745cea58e", 2021-02-10T11:27:16.880 INFO:teuthology.orchestra.run.gibba002.stdout: "pool_namespace": "", 2021-02-10T11:27:16.880 INFO:teuthology.orchestra.run.gibba002.stdout: "state": "complete", 2021-02-10T11:27:16.880 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "clone", 2021-02-10T11:27:16.880 INFO:teuthology.orchestra.run.gibba002.stdout: "uid": 0 2021-02-10T11:27:16.880 INFO:teuthology.orchestra.run.gibba002.stdout:} 2021-02-10T11:27:16.881 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:16.879 7f893bd15700 1 -- 172.21.2.102:0/421853503 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f891c0186c0 msgr2=0x7f891c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:16.881 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:16.879 7f893bd15700 1 --2- 172.21.2.102:0/421853503 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f891c0186c0 0x7f891c01abc0 crc :-1 s=READY pgs=778 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:16.881 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:16.879 7f893bd15700 1 -- 172.21.2.102:0/421853503 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8934191f10 msgr2=0x7f893419ffd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:16.881 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:16.879 7f893bd15700 1 --2- 172.21.2.102:0/421853503 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8934191f10 0x7f893419ffd0 secure :-1 s=READY pgs=1718 cs=0 l=1 rev1=1 rx=0x7f8928006420 tx=0x7f892800b960).stop 2021-02-10T11:27:16.881 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:16.879 7f893bd15700 1 -- 172.21.2.102:0/421853503 shutdown_connections 2021-02-10T11:27:16.881 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:16.879 7f893bd15700 1 --2- 172.21.2.102:0/421853503 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f891c0186c0 0x7f891c01abc0 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:16.881 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:16.879 7f893bd15700 1 --2- 172.21.2.102:0/421853503 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f89341a05f0 0x7f8934082ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:16.881 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:16.879 7f893bd15700 1 --2- 172.21.2.102:0/421853503 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8934191f10 0x7f893419ffd0 unknown :-1 s=CLOSED pgs=1718 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:16.882 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:16.879 7f893bd15700 1 --2- 172.21.2.102:0/421853503 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8934191b10 0x7f89340ae640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:16.882 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:16.879 7f893bd15700 1 -- 172.21.2.102:0/421853503 >> 172.21.2.102:0/421853503 conn(0x7f89340a02d0 msgr2=0x7f893409f200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:16.882 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:16.879 7f893bd15700 1 -- 172.21.2.102:0/421853503 shutdown_connections 2021-02-10T11:27:16.882 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:16.879 7f893bd15700 1 -- 172.21.2.102:0/421853503 wait complete. 2021-02-10T11:27:16.887 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000414869 2021-02-10T11:27:17.319 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:17.302 7fdbb73a8700 1 Processor -- start 2021-02-10T11:27:17.319 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:17.302 7fdbb73a8700 1 -- start start 2021-02-10T11:27:17.319 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:17.302 7fdbb73a8700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdbb00d1870 0x7fdbb01b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:17.320 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:17.302 7fdbb73a8700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fdbb01b0ae0 0x7fdbb0108940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:17.320 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:17.302 7fdbb73a8700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbb0109a60 con 0x7fdbb00d1c70 2021-02-10T11:27:17.320 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:17.303 7fdbb73a8700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdbb00e5ef0 con 0x7fdbb01b0ae0 2021-02-10T11:27:17.320 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:17.303 7fdbb73a8700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbb0109750 con 0x7fdbb00d1870 2021-02-10T11:27:17.320 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:17.303 7fdbb59c5700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fdbb00d1c70 0x7fdbb0108f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:51648/0 (socket says 172.21.2.102:51648) 2021-02-10T11:27:17.320 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:17.303 7fdbb59c5700 1 -- 172.21.2.102:0/2641926517 learned_addr learned my addr 172.21.2.102:0/2641926517 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:27:17.320 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:17.303 7fdba3fff700 1 -- 172.21.2.102:0/2641926517 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3259896402 0 0) 0x7fdba8001500 con 0x7fdbb00d1c70 2021-02-10T11:27:17.320 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:17.303 7fdba3fff700 1 -- 172.21.2.102:0/2641926517 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb9c002fe0 con 0x7fdbb00d1c70 2021-02-10T11:27:17.320 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:17.303 7fdbb49c3700 1 --2- 172.21.2.102:0/2641926517 >> v2:172.21.2.113:3300/0 conn(0x7fdbb01b0ae0 0x7fdbb0108940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:17.320 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:17.303 7fdbb51c4700 1 --2- 172.21.2.102:0/2641926517 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdbb00d1870 0x7fdbb01b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:17.320 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:17.303 7fdba3fff700 1 -- 172.21.2.102:0/2641926517 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1898804361 0 0) 0x7fdba8001500 con 0x7fdbb00d1c70 2021-02-10T11:27:17.320 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:17.304 7fdbb49c3700 1 -- 172.21.2.102:0/2641926517 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdbb00d1870 msgr2=0x7fdbb01b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:27:17.320 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:17.304 7fdbb49c3700 1 --2- 172.21.2.102:0/2641926517 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdbb00d1870 0x7fdbb01b04c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:17.321 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:17.304 7fdbb49c3700 1 -- 172.21.2.102:0/2641926517 >> v1:172.21.2.102:6789/0 conn(0x7fdbb00d1c70 legacy=0x7fdbb0108f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:17.321 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:17.304 7fdbb49c3700 1 -- 172.21.2.102:0/2641926517 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbb0088070 con 0x7fdbb01b0ae0 2021-02-10T11:27:17.321 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:17.304 7fdbb49c3700 1 --2- 172.21.2.102:0/2641926517 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fdbb01b0ae0 0x7fdbb0108940 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 rx=0x7fdbac006650 tx=0x7fdbac009a80).ready entity=mon.1 client_cookie=2cb95b3547412d7e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:17.321 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:17.304 7fdba3fff700 1 -- 172.21.2.102:0/2641926517 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdbac00c0c0 con 0x7fdbb01b0ae0 2021-02-10T11:27:17.321 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:17.304 7fdba3fff700 1 -- 172.21.2.102:0/2641926517 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbac00c300 con 0x7fdbb01b0ae0 2021-02-10T11:27:17.321 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:17.304 7fdba3fff700 1 -- 172.21.2.102:0/2641926517 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdbac00c7e0 con 0x7fdbb01b0ae0 2021-02-10T11:27:17.321 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:17.304 7fdbb73a8700 1 -- 172.21.2.102:0/2641926517 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fdbb01b0ae0 msgr2=0x7fdbb0108940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:17.321 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:17.304 7fdbb73a8700 1 --2- 172.21.2.102:0/2641926517 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fdbb01b0ae0 0x7fdbb0108940 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 rx=0x7fdbac006650 tx=0x7fdbac009a80).stop 2021-02-10T11:27:17.321 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:17.305 7fdbb73a8700 1 -- 172.21.2.102:0/2641926517 shutdown_connections 2021-02-10T11:27:17.321 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:17.305 7fdbb73a8700 1 --2- 172.21.2.102:0/2641926517 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fdbb01b0ae0 0x7fdbb0108940 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:17.321 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:17.305 7fdbb73a8700 1 --2- 172.21.2.102:0/2641926517 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdbb00d1870 0x7fdbb01b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:17.321 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:17.305 7fdbb73a8700 1 -- 172.21.2.102:0/2641926517 >> 172.21.2.102:0/2641926517 conn(0x7fdbb00ce420 msgr2=0x7fdbb00d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:17.321 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:17.305 7fdbb73a8700 1 -- 172.21.2.102:0/2641926517 shutdown_connections 2021-02-10T11:27:17.322 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:17.305 7fdbb73a8700 1 -- 172.21.2.102:0/2641926517 wait complete. 2021-02-10T11:27:17.323 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:17.320 7fdbb73a8700 1 Processor -- start 2021-02-10T11:27:17.323 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:17.320 7fdbb73a8700 1 -- start start 2021-02-10T11:27:17.323 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:17.320 7fdbb73a8700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdbb00a2580 0x7fdbb00a1cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:17.323 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:17.320 7fdbb73a8700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdbb0094c40 0x7fdbb01a75e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:17.324 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:17.320 7fdbb73a8700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fdbb0095040 0x7fdbb01a7c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:17.324 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:17.320 7fdbb73a8700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbb0188180 con 0x7fdbb00a2580 2021-02-10T11:27:17.324 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:17.320 7fdbb73a8700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbb0187e70 con 0x7fdbb0095040 2021-02-10T11:27:17.324 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:17.320 7fdbb73a8700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbb01a8220 con 0x7fdbb0094c40 2021-02-10T11:27:17.324 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:17.320 7fdbb51c4700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdbb00a2580 0x7fdbb00a1cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:17.324 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:17.320 7fdbb51c4700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdbb00a2580 0x7fdbb00a1cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:59430/0 (socket says 172.21.2.102:59430) 2021-02-10T11:27:17.324 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:17.320 7fdbb51c4700 1 -- 172.21.2.102:0/3675291348 learned_addr learned my addr 172.21.2.102:0/3675291348 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:27:17.324 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:17.321 7fdbb51c4700 1 -- 172.21.2.102:0/3675291348 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdbb0094c40 msgr2=0x7fdbb01a75e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:17.324 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:17.321 7fdbb51c4700 1 --2- 172.21.2.102:0/3675291348 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdbb0094c40 0x7fdbb01a75e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:17.324 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:17.321 7fdbb51c4700 1 -- 172.21.2.102:0/3675291348 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fdbb0095040 msgr2=0x7fdbb01a7c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:17.324 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:17.321 7fdbb51c4700 1 --2- 172.21.2.102:0/3675291348 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fdbb0095040 0x7fdbb01a7c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:17.324 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:17.321 7fdbb51c4700 1 -- 172.21.2.102:0/3675291348 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbb0188700 con 0x7fdbb00a2580 2021-02-10T11:27:17.324 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:17.321 7fdbb51c4700 1 --2- 172.21.2.102:0/3675291348 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdbb00a2580 0x7fdbb00a1cf0 secure :-1 s=READY pgs=1720 cs=0 l=1 rev1=1 rx=0x7fdba400b560 tx=0x7fdba400b900).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:17.325 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:17.321 7fdba1ffb700 1 -- 172.21.2.102:0/3675291348 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdba400bcf0 con 0x7fdbb00a2580 2021-02-10T11:27:17.325 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:17.321 7fdba1ffb700 1 -- 172.21.2.102:0/3675291348 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdba4000c90 con 0x7fdbb00a2580 2021-02-10T11:27:17.325 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:17.321 7fdba1ffb700 1 -- 172.21.2.102:0/3675291348 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdba40022b0 con 0x7fdbb00a2580 2021-02-10T11:27:17.325 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:17.321 7fdbb73a8700 1 -- 172.21.2.102:0/3675291348 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbb0188700 con 0x7fdbb00a2580 2021-02-10T11:27:17.325 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:17.321 7fdbb73a8700 1 -- 172.21.2.102:0/3675291348 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbb0188c90 con 0x7fdbb00a2580 2021-02-10T11:27:17.325 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:17.322 7fdba1ffb700 1 -- 172.21.2.102:0/3675291348 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fdba4006b00 con 0x7fdbb00a2580 2021-02-10T11:27:17.325 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:17.322 7fdba1ffb700 1 --2- 172.21.2.102:0/3675291348 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fdb900186c0 0x7fdb9001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:17.325 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:17.322 7fdba1ffb700 1 -- 172.21.2.102:0/3675291348 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fdba4032410 con 0x7fdbb00a2580 2021-02-10T11:27:17.325 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:17.322 7fdbb49c3700 1 --2- 172.21.2.102:0/3675291348 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fdb900186c0 0x7fdb9001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:17.325 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:17.322 7fdbb49c3700 1 --2- 172.21.2.102:0/3675291348 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fdb900186c0 0x7fdb9001abc0 crc :-1 s=READY pgs=779 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:17.352 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:17.352 7fdbb73a8700 1 -- 172.21.2.102:0/3675291348 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbb0082980 con 0x7fdbb00a2580 2021-02-10T11:27:17.354 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:17.355 7fdba1ffb700 1 -- 172.21.2.102:0/3675291348 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fdba4000940 con 0x7fdbb00a2580 2021-02-10T11:27:17.481 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:17.481 7fdbb73a8700 1 -- 172.21.2.102:0/3675291348 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000414869", "target": ["mgr", ""]}) v1 -- 0x7fdbb0188c90 con 0x7fdb900186c0 2021-02-10T11:27:17.492 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:17.493 7fdba1ffb700 1 -- 172.21.2.102:0/3675291348 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fdbac0067b0 con 0x7fdb900186c0 2021-02-10T11:27:17.497 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:17.497 7fdbb73a8700 1 -- 172.21.2.102:0/3675291348 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fdb900186c0 msgr2=0x7fdb9001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:17.497 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:17.497 7fdbb73a8700 1 --2- 172.21.2.102:0/3675291348 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fdb900186c0 0x7fdb9001abc0 crc :-1 s=READY pgs=779 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:17.497 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:17.497 7fdbb73a8700 1 -- 172.21.2.102:0/3675291348 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdbb00a2580 msgr2=0x7fdbb00a1cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:17.497 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:17.497 7fdbb73a8700 1 --2- 172.21.2.102:0/3675291348 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdbb00a2580 0x7fdbb00a1cf0 secure :-1 s=READY pgs=1720 cs=0 l=1 rev1=1 rx=0x7fdba400b560 tx=0x7fdba400b900).stop 2021-02-10T11:27:17.498 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:17.497 7fdbb73a8700 1 -- 172.21.2.102:0/3675291348 shutdown_connections 2021-02-10T11:27:17.498 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:17.497 7fdbb73a8700 1 --2- 172.21.2.102:0/3675291348 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fdb900186c0 0x7fdb9001abc0 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:17.498 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:17.497 7fdbb73a8700 1 --2- 172.21.2.102:0/3675291348 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fdbb0095040 0x7fdbb01a7c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:17.498 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:17.498 7fdbb73a8700 1 --2- 172.21.2.102:0/3675291348 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdbb0094c40 0x7fdbb01a75e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:17.498 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:17.498 7fdbb73a8700 1 --2- 172.21.2.102:0/3675291348 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdbb00a2580 0x7fdbb00a1cf0 unknown :-1 s=CLOSED pgs=1720 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:17.498 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:17.498 7fdbb73a8700 1 -- 172.21.2.102:0/3675291348 >> 172.21.2.102:0/3675291348 conn(0x7fdbb00a2b30 msgr2=0x7fdbb009ce60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:17.498 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:17.498 7fdbb73a8700 1 -- 172.21.2.102:0/3675291348 shutdown_connections 2021-02-10T11:27:17.498 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:17.498 7fdbb73a8700 1 -- 172.21.2.102:0/3675291348 wait complete. 2021-02-10T11:27:17.506 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs clone_0000000000602982 2021-02-10T11:27:17.939 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:17.922 7facf5696700 1 Processor -- start 2021-02-10T11:27:17.939 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:17.922 7facf5696700 1 -- start start 2021-02-10T11:27:17.939 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:17.922 7facf5696700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7facf00d1870 0x7facf01b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:17.939 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:17.922 7facf5696700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7facf01b0ae0 0x7facf0108940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:17.939 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:17.922 7facf5696700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7facf00d1c70 0x7facf0108f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:17.939 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:17.922 7facf5696700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7facf00fe1c0 con 0x7facf00d1870 2021-02-10T11:27:17.939 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:17.922 7facf5696700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7facf0109a60 con 0x7facf00d1c70 2021-02-10T11:27:17.939 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:17.922 7facf5696700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facf0088070 con 0x7facf01b0ae0 2021-02-10T11:27:17.940 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:17.923 7facef2bf700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7facf00d1870 0x7facf01b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:17.940 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:17.923 7facef2bf700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7facf00d1870 0x7facf01b04c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:59438/0 (socket says 172.21.2.102:59438) 2021-02-10T11:27:17.940 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:17.923 7facef2bf700 1 -- 172.21.2.102:0/2131722717 learned_addr learned my addr 172.21.2.102:0/2131722717 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:27:17.940 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:17.923 7facefac0700 1 --2- 172.21.2.102:0/2131722717 >> v2:172.21.2.113:3300/0 conn(0x7facf00d1c70 0x7facf0108f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:17.940 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:17.923 7faceeabe700 1 --2- 172.21.2.102:0/2131722717 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7facf01b0ae0 0x7facf0108940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:17.940 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:17.923 7facef2bf700 1 -- 172.21.2.102:0/2131722717 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7facf01b0ae0 msgr2=0x7facf0108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:27:17.940 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:17.923 7facef2bf700 1 --2- 172.21.2.102:0/2131722717 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7facf01b0ae0 0x7facf0108940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:17.940 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:17.923 7facef2bf700 1 -- 172.21.2.102:0/2131722717 >> v2:172.21.2.113:3300/0 conn(0x7facf00d1c70 msgr2=0x7facf0108f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:27:17.940 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:17.923 7facef2bf700 1 --2- 172.21.2.102:0/2131722717 >> v2:172.21.2.113:3300/0 conn(0x7facf00d1c70 0x7facf0108f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:17.940 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:17.923 7facef2bf700 1 -- 172.21.2.102:0/2131722717 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facf0109580 con 0x7facf00d1870 2021-02-10T11:27:17.940 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:17.923 7facef2bf700 1 --2- 172.21.2.102:0/2131722717 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7facf00d1870 0x7facf01b04c0 secure :-1 s=READY pgs=1721 cs=0 l=1 rev1=1 rx=0x7facd8006700 tx=0x7facd8009b00).ready entity=mon.0 client_cookie=8bc63c11d80cee1f server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:17.940 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:17.923 7facee2bd700 1 -- 172.21.2.102:0/2131722717 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7facd800c0c0 con 0x7facf00d1870 2021-02-10T11:27:17.940 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:17.923 7facee2bd700 1 -- 172.21.2.102:0/2131722717 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facd800c400 con 0x7facf00d1870 2021-02-10T11:27:17.941 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:17.923 7facee2bd700 1 -- 172.21.2.102:0/2131722717 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7facd800c8e0 con 0x7facf00d1870 2021-02-10T11:27:17.941 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:17.923 7facf5696700 1 -- 172.21.2.102:0/2131722717 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7facf00d1870 msgr2=0x7facf01b04c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:17.941 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:17.923 7facf5696700 1 --2- 172.21.2.102:0/2131722717 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7facf00d1870 0x7facf01b04c0 secure :-1 s=READY pgs=1721 cs=0 l=1 rev1=1 rx=0x7facd8006700 tx=0x7facd8009b00).stop 2021-02-10T11:27:17.941 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:17.924 7facf5696700 1 -- 172.21.2.102:0/2131722717 shutdown_connections 2021-02-10T11:27:17.941 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:17.924 7facf5696700 1 --2- 172.21.2.102:0/2131722717 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7facf01b0ae0 0x7facf0108940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:17.941 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:17.924 7facf5696700 1 --2- 172.21.2.102:0/2131722717 >> v2:172.21.2.113:3300/0 conn(0x7facf00d1c70 0x7facf0108f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:17.941 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:17.924 7facf5696700 1 --2- 172.21.2.102:0/2131722717 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7facf00d1870 0x7facf01b04c0 unknown :-1 s=CLOSED pgs=1721 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:17.941 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:17.924 7facf5696700 1 -- 172.21.2.102:0/2131722717 >> 172.21.2.102:0/2131722717 conn(0x7facf00ce420 msgr2=0x7facf00d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:17.941 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:17.924 7facf5696700 1 -- 172.21.2.102:0/2131722717 shutdown_connections 2021-02-10T11:27:17.941 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:17.924 7facf5696700 1 -- 172.21.2.102:0/2131722717 wait complete. 2021-02-10T11:27:17.941 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:17.939 7facf5696700 1 Processor -- start 2021-02-10T11:27:17.941 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:17.939 7facf5696700 1 -- start start 2021-02-10T11:27:17.943 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:17.939 7facf5696700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7facf0085bc0 0x7facf00863e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:17.943 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:17.939 7facf5696700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7facf00954c0 0x7facf0096300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:17.943 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:17.939 7facf5696700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7facf00958c0 0x7facf0096920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:17.943 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:17.939 7facf5696700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facf0095d80 con 0x7facf0085bc0 2021-02-10T11:27:17.943 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:17.939 7facf5696700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facf00bfa60 con 0x7facf00954c0 2021-02-10T11:27:17.943 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:17.939 7facf5696700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facf0085fc0 con 0x7facf00958c0 2021-02-10T11:27:17.943 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:17.939 7facef2bf700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7facf0085bc0 0x7facf00863e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:17.943 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:17.940 7facef2bf700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7facf0085bc0 0x7facf00863e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:59442/0 (socket says 172.21.2.102:59442) 2021-02-10T11:27:17.943 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:17.940 7facef2bf700 1 -- 172.21.2.102:0/3582539588 learned_addr learned my addr 172.21.2.102:0/3582539588 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:27:17.943 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:17.940 7facef2bf700 1 -- 172.21.2.102:0/3582539588 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7facf00958c0 msgr2=0x7facf0096920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:17.943 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:17.940 7facef2bf700 1 --2- 172.21.2.102:0/3582539588 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7facf00958c0 0x7facf0096920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:17.944 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:17.940 7facef2bf700 1 -- 172.21.2.102:0/3582539588 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7facf00954c0 msgr2=0x7facf0096300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:17.944 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:17.940 7facef2bf700 1 --2- 172.21.2.102:0/3582539588 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7facf00954c0 0x7facf0096300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:17.944 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:17.940 7facef2bf700 1 -- 172.21.2.102:0/3582539588 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facf0086c10 con 0x7facf0085bc0 2021-02-10T11:27:17.944 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:17.940 7facef2bf700 1 --2- 172.21.2.102:0/3582539588 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7facf0085bc0 0x7facf00863e0 secure :-1 s=READY pgs=1722 cs=0 l=1 rev1=1 rx=0x7facd800b660 tx=0x7facd800ba00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:17.944 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:17.940 7face7fff700 1 -- 172.21.2.102:0/3582539588 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7facd8009440 con 0x7facf0085bc0 2021-02-10T11:27:17.944 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:17.940 7face7fff700 1 -- 172.21.2.102:0/3582539588 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facd80090c0 con 0x7facf0085bc0 2021-02-10T11:27:17.944 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:17.940 7face7fff700 1 -- 172.21.2.102:0/3582539588 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7facd8005370 con 0x7facf0085bc0 2021-02-10T11:27:17.944 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:17.940 7facf5696700 1 -- 172.21.2.102:0/3582539588 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7facf0086c10 con 0x7facf0085bc0 2021-02-10T11:27:17.944 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:17.940 7facf5696700 1 -- 172.21.2.102:0/3582539588 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7facf0097230 con 0x7facf0085bc0 2021-02-10T11:27:17.944 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:17.941 7face7fff700 1 -- 172.21.2.102:0/3582539588 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7facd80093e0 con 0x7facf0085bc0 2021-02-10T11:27:17.944 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:17.941 7face7fff700 1 --2- 172.21.2.102:0/3582539588 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7facd40186c0 0x7facd401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:17.944 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:17.941 7face7fff700 1 -- 172.21.2.102:0/3582539588 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5573+0+0 (secure 0 0 0) 0x7facd80325b0 con 0x7facf0085bc0 2021-02-10T11:27:17.945 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:17.941 7faceeabe700 1 --2- 172.21.2.102:0/3582539588 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7facd40186c0 0x7facd401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:17.945 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:17.941 7faceeabe700 1 --2- 172.21.2.102:0/3582539588 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7facd40186c0 0x7facd401abc0 crc :-1 s=READY pgs=780 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:17.972 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:17.972 7facf5696700 1 -- 172.21.2.102:0/3582539588 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7facf00fe1c0 con 0x7facf0085bc0 2021-02-10T11:27:17.974 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:17.974 7face7fff700 1 -- 172.21.2.102:0/3582539588 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7facd8030d60 con 0x7facf0085bc0 2021-02-10T11:27:18.101 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:18.101 7facf5696700 1 -- 172.21.2.102:0/3582539588 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "clone_0000000000602982", "target": ["mgr", ""]}) v1 -- 0x7facf0097230 con 0x7facd40186c0 2021-02-10T11:27:18.109 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:18.109 7face7fff700 1 -- 172.21.2.102:0/3582539588 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7face0006e70 con 0x7facd40186c0 2021-02-10T11:27:18.119 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:18.118 7facf5696700 1 -- 172.21.2.102:0/3582539588 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7facd40186c0 msgr2=0x7facd401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:18.119 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:18.118 7facf5696700 1 --2- 172.21.2.102:0/3582539588 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7facd40186c0 0x7facd401abc0 crc :-1 s=READY pgs=780 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:18.119 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:18.118 7facf5696700 1 -- 172.21.2.102:0/3582539588 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7facf0085bc0 msgr2=0x7facf00863e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:18.119 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:18.118 7facf5696700 1 --2- 172.21.2.102:0/3582539588 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7facf0085bc0 0x7facf00863e0 secure :-1 s=READY pgs=1722 cs=0 l=1 rev1=1 rx=0x7facd800b660 tx=0x7facd800ba00).stop 2021-02-10T11:27:18.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:18.118 7facf5696700 1 -- 172.21.2.102:0/3582539588 shutdown_connections 2021-02-10T11:27:18.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:18.118 7facf5696700 1 --2- 172.21.2.102:0/3582539588 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7facf00958c0 0x7facf0096920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:18.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:18.118 7facf5696700 1 --2- 172.21.2.102:0/3582539588 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7facd40186c0 0x7facd401abc0 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:18.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:18.118 7facf5696700 1 --2- 172.21.2.102:0/3582539588 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7facf00954c0 0x7facf0096300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:18.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:18.118 7facf5696700 1 --2- 172.21.2.102:0/3582539588 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7facf0085bc0 0x7facf00863e0 unknown :-1 s=CLOSED pgs=1722 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:18.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:18.118 7facf5696700 1 -- 172.21.2.102:0/3582539588 >> 172.21.2.102:0/3582539588 conn(0x7facf00bef00 msgr2=0x7facf009d1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:18.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:18.118 7facf5696700 1 -- 172.21.2.102:0/3582539588 shutdown_connections 2021-02-10T11:27:18.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:18.119 7facf5696700 1 -- 172.21.2.102:0/3582539588 wait complete. 2021-02-10T11:27:18.127 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo getfattr --only-values -n ceph.dir.entries /home/ubuntu/cephtest/mnt.0/./volumes/_deleting 2021-02-10T11:27:18.477 INFO:teuthology.orchestra.run.gibba002.stderr:getfattr: Removing leading '/' from absolute path names 2021-02-10T11:27:18.478 INFO:teuthology.orchestra.run.gibba002.stdout:0 2021-02-10T11:27:18.478 DEBUG:tasks.cephfs.mount:Directory ./volumes/_deleting seen empty from 0 after 0s 2021-02-10T11:27:18.478 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2021-02-10T11:27:18.478 DEBUG:teuthology.orchestra.run.gibba002:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:27:18.823 DEBUG:teuthology.orchestra.run.gibba013:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:27:19.167 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2021-02-10T11:27:19.167 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:19.168 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:27:19.168 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:27:19.502 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:27:19.502 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:27:19.503 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@gibba002.front.sepia.ceph.com... 2021-02-10T11:27:19.503 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-02-10T11:27:19.503 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:27:19.853 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba002.stderr:ceph-fuse[165230]: fuse finished with error 0 and tester_r 0 2021-02-10T11:27:19.854 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:19.854 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:27:19.854 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:27:20.189 INFO:teuthology.orchestra.run.gibba002.stdout:ext2/ext3 2021-02-10T11:27:20.189 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2021-02-10T11:27:20.190 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:27:20.190 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:27:20.525 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:20.526 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:27:20.526 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:27:20.861 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:27:20.862 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2021-02-10T11:27:20.862 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@gibba002.front.sepia.ceph.com... 2021-02-10T11:27:20.862 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-02-10T11:27:20.862 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:27:21.212 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba002.stderr:ceph-fuse[165899]: fuse finished with error 0 and tester_r 0 2021-02-10T11:27:21.213 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:21.213 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:27:21.213 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:27:21.548 INFO:teuthology.orchestra.run.gibba002.stdout:ext2/ext3 2021-02-10T11:27:21.548 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2021-02-10T11:27:21.549 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:27:21.549 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:27:21.884 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:21.884 DEBUG:teuthology.orchestra.run.gibba013:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:27:21.884 DEBUG:teuthology.orchestra.run.gibba013:> ' -- /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:27:21.967 INFO:tasks.ceph.osd.6.gibba013.stderr:2021-02-10 11:27:21.965 7fa264002700 -1 osd.6 pg_epoch: 76 pg[15.5( v 75'8 (0'0,75'8] local-lis/les=73/74 n=2 ec=73/73 lis/c 73/73 les/c/f 74/74/0 73/73/73) [6,3] r=0 lpr=73 crt=75'8 lcod 75'7 mlcod 75'7 active+clean+snaptrim trimq=[2~2]] removing snap head 2021-02-10T11:27:22.219 INFO:teuthology.orchestra.run.gibba013.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.2’: No such file or directory 2021-02-10T11:27:22.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:27:22.220 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2021-02-10T11:27:22.220 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:27:22.220 DEBUG:teuthology.orchestra.run.gibba013:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:27:22.555 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:22.555 DEBUG:teuthology.orchestra.run.gibba013:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:27:22.555 DEBUG:teuthology.orchestra.run.gibba013:> ' -- /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:27:22.890 INFO:teuthology.orchestra.run.gibba013.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.3’: No such file or directory 2021-02-10T11:27:22.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:27:22.891 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.3 2021-02-10T11:27:22.891 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:27:22.891 DEBUG:teuthology.orchestra.run.gibba013:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:27:23.226 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config rm mon mon_allow_pool_delete 2021-02-10T11:27:23.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:23.641 7f7ad8802700 1 Processor -- start 2021-02-10T11:27:23.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:23.641 7f7ad8802700 1 -- start start 2021-02-10T11:27:23.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:23.642 7f7ad8802700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7ad40d1870 0x7f7ad41b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:23.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:23.642 7f7ad8802700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f7ad41b0a10 0x7f7ad40cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:23.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:23.642 7f7ad8802700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ad40f0b10 con 0x7f7ad40d1c70 2021-02-10T11:27:23.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:23.642 7f7ad8802700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ad40fe230 con 0x7f7ad41b0a10 2021-02-10T11:27:23.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:23.642 7f7ad8802700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ad40e5ed0 con 0x7f7ad40d1870 2021-02-10T11:27:23.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:23.642 7f7ad1e1b700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f7ad41b0a10 0x7f7ad40cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:23.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:23.642 7f7ad1e1b700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f7ad41b0a10 0x7f7ad40cf330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:59450/0 (socket says 172.21.2.102:59450) 2021-02-10T11:27:23.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:23.642 7f7ad1e1b700 1 -- 172.21.2.102:0/1599011593 learned_addr learned my addr 172.21.2.102:0/1599011593 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:27:23.659 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:23.642 7f7ad1e1b700 1 -- 172.21.2.102:0/1599011593 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7ad40d1870 msgr2=0x7f7ad41b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:27:23.659 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:23.642 7f7ad1e1b700 1 --2- 172.21.2.102:0/1599011593 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7ad40d1870 0x7f7ad41b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:23.659 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:23.642 7f7ad1e1b700 1 -- 172.21.2.102:0/1599011593 >> v1:172.21.2.113:6789/0 conn(0x7f7ad40d1c70 legacy=0x7f7ad40cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:27:23.659 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:23.642 7f7ad1e1b700 1 -- 172.21.2.102:0/1599011593 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ad40f0650 con 0x7f7ad41b0a10 2021-02-10T11:27:23.659 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:23.642 7f7ad1e1b700 1 --2- 172.21.2.102:0/1599011593 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7ad41b0a10 0x7f7ad40cf330 secure :-1 s=READY pgs=1723 cs=0 l=1 rev1=1 rx=0x7f7ac8006700 tx=0x7f7ac8009b80).ready entity=mon.0 client_cookie=a6a4d76805e33430 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:23.659 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:23.642 7f7ad161a700 1 -- 172.21.2.102:0/1599011593 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7ac800c0c0 con 0x7f7ad41b0a10 2021-02-10T11:27:23.659 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:23.643 7f7ad161a700 1 -- 172.21.2.102:0/1599011593 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ac800c400 con 0x7f7ad41b0a10 2021-02-10T11:27:23.659 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:23.643 7f7ad161a700 1 -- 172.21.2.102:0/1599011593 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7ac800c8e0 con 0x7f7ad41b0a10 2021-02-10T11:27:23.659 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:23.643 7f7ad8802700 1 -- 172.21.2.102:0/1599011593 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7ad41b0a10 msgr2=0x7f7ad40cf330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:23.659 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:23.643 7f7ad8802700 1 --2- 172.21.2.102:0/1599011593 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7ad41b0a10 0x7f7ad40cf330 secure :-1 s=READY pgs=1723 cs=0 l=1 rev1=1 rx=0x7f7ac8006700 tx=0x7f7ac8009b80).stop 2021-02-10T11:27:23.659 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:23.643 7f7ad8802700 1 -- 172.21.2.102:0/1599011593 shutdown_connections 2021-02-10T11:27:23.659 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:23.643 7f7ad8802700 1 --2- 172.21.2.102:0/1599011593 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7ad41b0a10 0x7f7ad40cf330 unknown :-1 s=CLOSED pgs=1723 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:23.659 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:23.643 7f7ad8802700 1 --2- 172.21.2.102:0/1599011593 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7ad40d1870 0x7f7ad41b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:23.660 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:23.643 7f7ad8802700 1 -- 172.21.2.102:0/1599011593 >> 172.21.2.102:0/1599011593 conn(0x7f7ad40ce450 msgr2=0x7f7ad40d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:23.660 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:23.644 7f7ad8802700 1 -- 172.21.2.102:0/1599011593 shutdown_connections 2021-02-10T11:27:23.660 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:23.644 7f7ad8802700 1 -- 172.21.2.102:0/1599011593 wait complete. 2021-02-10T11:27:23.661 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:23.658 7f7ad8802700 1 Processor -- start 2021-02-10T11:27:23.661 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:23.658 7f7ad8802700 1 -- start start 2021-02-10T11:27:23.661 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:23.658 7f7ad8802700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7ad40a2fb0 0x7f7ad40a33b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:23.661 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:23.658 7f7ad8802700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7ad40a39d0 0x7f7ad40a4860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:23.661 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:23.659 7f7ad8802700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7ad40a3dd0 0x7f7ad40a4e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:23.661 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:23.659 7f7ad8802700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ad40a2620 con 0x7f7ad40a2fb0 2021-02-10T11:27:23.661 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:23.659 7f7ad8802700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ad40a2840 con 0x7f7ad40a39d0 2021-02-10T11:27:23.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:23.659 7f7ad8802700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ad40a2b80 con 0x7f7ad40a3dd0 2021-02-10T11:27:23.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:23.659 7f7ad261c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7ad40a2fb0 0x7f7ad40a33b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:23.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:23.659 7f7ad261c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7ad40a2fb0 0x7f7ad40a33b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:59456/0 (socket says 172.21.2.102:59456) 2021-02-10T11:27:23.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:23.659 7f7ad261c700 1 -- 172.21.2.102:0/1407766534 learned_addr learned my addr 172.21.2.102:0/1407766534 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:27:23.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:23.659 7f7ad261c700 1 -- 172.21.2.102:0/1407766534 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7ad40a3dd0 msgr2=0x7f7ad40a4e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:23.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:23.659 7f7ad261c700 1 --2- 172.21.2.102:0/1407766534 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7ad40a3dd0 0x7f7ad40a4e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:23.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:23.659 7f7ad261c700 1 -- 172.21.2.102:0/1407766534 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7ad40a39d0 msgr2=0x7f7ad40a4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:23.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:23.659 7f7ad261c700 1 --2- 172.21.2.102:0/1407766534 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7ad40a39d0 0x7f7ad40a4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:23.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:23.659 7f7ad261c700 1 -- 172.21.2.102:0/1407766534 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ad40a54a0 con 0x7f7ad40a2fb0 2021-02-10T11:27:23.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:23.659 7f7ad261c700 1 --2- 172.21.2.102:0/1407766534 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7ad40a2fb0 0x7f7ad40a33b0 secure :-1 s=READY pgs=1724 cs=0 l=1 rev1=1 rx=0x7f7ac4006f00 tx=0x7f7ac400b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:23.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:23.659 7f7ac2ffd700 1 -- 172.21.2.102:0/1407766534 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7ac4006b90 con 0x7f7ad40a2fb0 2021-02-10T11:27:23.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:23.659 7f7ac2ffd700 1 -- 172.21.2.102:0/1407766534 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ac4000c90 con 0x7f7ad40a2fb0 2021-02-10T11:27:23.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:23.659 7f7ad8802700 1 -- 172.21.2.102:0/1407766534 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ad40a54a0 con 0x7f7ad40a2fb0 2021-02-10T11:27:23.663 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:23.660 7f7ac2ffd700 1 -- 172.21.2.102:0/1407766534 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7ac40022a0 con 0x7f7ad40a2fb0 2021-02-10T11:27:23.663 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:23.660 7f7ad8802700 1 -- 172.21.2.102:0/1407766534 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ad40a54a0 con 0x7f7ad40a2fb0 2021-02-10T11:27:23.663 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:23.660 7f7ac2ffd700 1 -- 172.21.2.102:0/1407766534 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f7ac4005310 con 0x7f7ad40a2fb0 2021-02-10T11:27:23.663 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:23.660 7f7ac2ffd700 1 --2- 172.21.2.102:0/1407766534 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7ab80186c0 0x7f7ab801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:23.663 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:23.660 7f7ac2ffd700 1 -- 172.21.2.102:0/1407766534 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 5645+0+0 (secure 0 0 0) 0x7f7ac4032490 con 0x7f7ad40a2fb0 2021-02-10T11:27:23.663 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:23.660 7f7ad1e1b700 1 --2- 172.21.2.102:0/1407766534 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7ab80186c0 0x7f7ab801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:23.663 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:23.661 7f7ad1e1b700 1 --2- 172.21.2.102:0/1407766534 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7ab80186c0 0x7f7ab801abc0 crc :-1 s=READY pgs=781 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:23.691 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:23.691 7f7ad8802700 1 -- 172.21.2.102:0/1407766534 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ad40ee060 con 0x7f7ad40a2fb0 2021-02-10T11:27:23.694 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:23.694 7f7ac2ffd700 1 -- 172.21.2.102:0/1407766534 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f7ac4030d80 con 0x7f7ad40a2fb0 2021-02-10T11:27:23.842 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:23.842 7f7ad8802700 1 -- 172.21.2.102:0/1407766534 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f7ad40a5d20 con 0x7f7ad40a2fb0 2021-02-10T11:27:23.846 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:23.846 7f7ac2ffd700 1 -- 172.21.2.102:0/1407766534 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v21) v1 ==== 104+0+0 (secure 0 0 0) 0x7f7ac4005310 con 0x7f7ad40a2fb0 2021-02-10T11:27:23.851 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:23.851 7f7ad8802700 1 -- 172.21.2.102:0/1407766534 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7ab80186c0 msgr2=0x7f7ab801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:23.851 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:23.851 7f7ad8802700 1 --2- 172.21.2.102:0/1407766534 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7ab80186c0 0x7f7ab801abc0 crc :-1 s=READY pgs=781 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:23.851 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:23.851 7f7ad8802700 1 -- 172.21.2.102:0/1407766534 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7ad40a2fb0 msgr2=0x7f7ad40a33b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:23.851 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:23.851 7f7ad8802700 1 --2- 172.21.2.102:0/1407766534 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7ad40a2fb0 0x7f7ad40a33b0 secure :-1 s=READY pgs=1724 cs=0 l=1 rev1=1 rx=0x7f7ac4006f00 tx=0x7f7ac400b940).stop 2021-02-10T11:27:23.852 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:23.852 7f7ad8802700 1 -- 172.21.2.102:0/1407766534 shutdown_connections 2021-02-10T11:27:23.852 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:23.852 7f7ad8802700 1 --2- 172.21.2.102:0/1407766534 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7ad40a3dd0 0x7f7ad40a4e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:23.852 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:23.852 7f7ad8802700 1 --2- 172.21.2.102:0/1407766534 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7ab80186c0 0x7f7ab801abc0 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:23.852 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:23.852 7f7ad8802700 1 --2- 172.21.2.102:0/1407766534 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7ad40a39d0 0x7f7ad40a4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:23.852 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:23.852 7f7ad8802700 1 --2- 172.21.2.102:0/1407766534 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7ad40a2fb0 0x7f7ad40a33b0 unknown :-1 s=CLOSED pgs=1724 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:23.852 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:23.852 7f7ad8802700 1 -- 172.21.2.102:0/1407766534 >> 172.21.2.102:0/1407766534 conn(0x7f7ad408f030 msgr2=0x7f7ad40b5e80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:23.852 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:23.852 7f7ad8802700 1 -- 172.21.2.102:0/1407766534 shutdown_connections 2021-02-10T11:27:23.853 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:23.852 7f7ad8802700 1 -- 172.21.2.102:0/1407766534 wait complete. 2021-02-10T11:27:23.860 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestVolumes.test_clone_subvolume_info' 2021-02-10T11:27:24.287 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:24.271 7f4e9a08f700 1 Processor -- start 2021-02-10T11:27:24.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:24.271 7f4e9a08f700 1 -- start start 2021-02-10T11:27:24.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:24.271 7f4e9a08f700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f4e940d0340 0x7f4e940ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:24.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:24.271 7f4e9a08f700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4e940d1c70 0x7f4e940efc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:24.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:24.271 7f4e9a08f700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e940f0b10 con 0x7f4e940d1870 2021-02-10T11:27:24.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:24.271 7f4e9a08f700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e940e5ed0 con 0x7f4e940d0340 2021-02-10T11:27:24.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:24.271 7f4e9a08f700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e940d2fe0 con 0x7f4e940d1c70 2021-02-10T11:27:24.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:24.272 7f4e92ffd700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f4e940d0340 0x7f4e940ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:24.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:24.272 7f4e92ffd700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f4e940d0340 0x7f4e940ef640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:59468/0 (socket says 172.21.2.102:59468) 2021-02-10T11:27:24.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:24.272 7f4e92ffd700 1 -- 172.21.2.102:0/1730475838 learned_addr learned my addr 172.21.2.102:0/1730475838 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:27:24.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:24.272 7f4e93fff700 1 --2- 172.21.2.102:0/1730475838 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4e940d1c70 0x7f4e940efc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:24.289 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:24.272 7f4e927fc700 1 -- 172.21.2.102:0/1730475838 <== mon.1 v1:172.21.2.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3175007472 0 0) 0x7f4e8c000fe0 con 0x7f4e940d1870 2021-02-10T11:27:24.289 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:24.272 7f4e927fc700 1 -- 172.21.2.102:0/1730475838 --> v1:172.21.2.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e7c002fe0 con 0x7f4e940d1870 2021-02-10T11:27:24.289 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:24.272 7f4e92ffd700 1 -- 172.21.2.102:0/1730475838 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4e940d1c70 msgr2=0x7f4e940efc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:27:24.289 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:24.272 7f4e92ffd700 1 --2- 172.21.2.102:0/1730475838 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4e940d1c70 0x7f4e940efc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:24.289 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:24.272 7f4e92ffd700 1 -- 172.21.2.102:0/1730475838 >> v1:172.21.2.113:6789/0 conn(0x7f4e940d1870 legacy=0x7f4e941b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:24.289 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:24.272 7f4e92ffd700 1 -- 172.21.2.102:0/1730475838 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e941b0be0 con 0x7f4e940d0340 2021-02-10T11:27:24.289 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:24.272 7f4e93fff700 1 --2- 172.21.2.102:0/1730475838 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4e940d1c70 0x7f4e940efc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:27:24.289 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:24.272 7f4e92ffd700 1 --2- 172.21.2.102:0/1730475838 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4e940d0340 0x7f4e940ef640 secure :-1 s=READY pgs=1725 cs=0 l=1 rev1=1 rx=0x7f4e88006700 tx=0x7f4e88009b80).ready entity=mon.0 client_cookie=fd4ec2fc068f26cf server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:24.289 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:24.272 7f4e927fc700 1 -- 172.21.2.102:0/1730475838 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4e8800c0c0 con 0x7f4e940d0340 2021-02-10T11:27:24.289 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:24.273 7f4e927fc700 1 -- 172.21.2.102:0/1730475838 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e8800c340 con 0x7f4e940d0340 2021-02-10T11:27:24.289 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:24.273 7f4e927fc700 1 -- 172.21.2.102:0/1730475838 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4e8800c820 con 0x7f4e940d0340 2021-02-10T11:27:24.289 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:24.273 7f4e9a08f700 1 -- 172.21.2.102:0/1730475838 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4e940d0340 msgr2=0x7f4e940ef640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:24.289 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:24.273 7f4e9a08f700 1 --2- 172.21.2.102:0/1730475838 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4e940d0340 0x7f4e940ef640 secure :-1 s=READY pgs=1725 cs=0 l=1 rev1=1 rx=0x7f4e88006700 tx=0x7f4e88009b80).stop 2021-02-10T11:27:24.290 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:24.273 7f4e9a08f700 1 -- 172.21.2.102:0/1730475838 shutdown_connections 2021-02-10T11:27:24.290 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:24.273 7f4e9a08f700 1 --2- 172.21.2.102:0/1730475838 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4e940d1c70 0x7f4e940efc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:24.290 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:24.273 7f4e9a08f700 1 --2- 172.21.2.102:0/1730475838 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4e940d0340 0x7f4e940ef640 unknown :-1 s=CLOSED pgs=1725 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:24.290 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:24.273 7f4e9a08f700 1 -- 172.21.2.102:0/1730475838 >> 172.21.2.102:0/1730475838 conn(0x7f4e940ce450 msgr2=0x7f4e940d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:24.290 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:24.273 7f4e9a08f700 1 -- 172.21.2.102:0/1730475838 shutdown_connections 2021-02-10T11:27:24.290 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:24.273 7f4e9a08f700 1 -- 172.21.2.102:0/1730475838 wait complete. 2021-02-10T11:27:24.291 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:24.288 7f4e9a08f700 1 Processor -- start 2021-02-10T11:27:24.291 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:24.288 7f4e9a08f700 1 -- start start 2021-02-10T11:27:24.292 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:24.288 7f4e9a08f700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4e94195650 0x7f4e940b5560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:24.292 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:24.288 7f4e9a08f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4e940b5b80 0x7f4e941a2e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:24.292 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:24.288 7f4e9a08f700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4e940acfa0 0x7f4e941a3440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:24.292 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:24.288 7f4e9a08f700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e940b6010 con 0x7f4e940b5b80 2021-02-10T11:27:24.292 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:24.288 7f4e9a08f700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e941a3a60 con 0x7f4e940acfa0 2021-02-10T11:27:24.292 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:24.288 7f4e9a08f700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e941a3c40 con 0x7f4e94195650 2021-02-10T11:27:24.292 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:24.289 7f4e92ffd700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4e940b5b80 0x7f4e941a2e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:24.292 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:24.289 7f4e92ffd700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4e940b5b80 0x7f4e941a2e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:59470/0 (socket says 172.21.2.102:59470) 2021-02-10T11:27:24.292 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:24.289 7f4e92ffd700 1 -- 172.21.2.102:0/3778129986 learned_addr learned my addr 172.21.2.102:0/3778129986 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:27:24.292 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:24.289 7f4e937fe700 1 --2- 172.21.2.102:0/3778129986 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4e94195650 0x7f4e940b5560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:24.292 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:24.289 7f4e93fff700 1 --2- 172.21.2.102:0/3778129986 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4e940acfa0 0x7f4e941a3440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:24.292 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:24.289 7f4e92ffd700 1 -- 172.21.2.102:0/3778129986 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4e94195650 msgr2=0x7f4e940b5560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:24.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:24.289 7f4e92ffd700 1 --2- 172.21.2.102:0/3778129986 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4e94195650 0x7f4e940b5560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:24.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:24.289 7f4e92ffd700 1 -- 172.21.2.102:0/3778129986 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4e940acfa0 msgr2=0x7f4e941a3440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:24.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:24.289 7f4e92ffd700 1 --2- 172.21.2.102:0/3778129986 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4e940acfa0 0x7f4e941a3440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:24.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:24.289 7f4e92ffd700 1 -- 172.21.2.102:0/3778129986 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e940bcae0 con 0x7f4e940b5b80 2021-02-10T11:27:24.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:24.289 7f4e92ffd700 1 --2- 172.21.2.102:0/3778129986 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4e940b5b80 0x7f4e941a2e20 secure :-1 s=READY pgs=1726 cs=0 l=1 rev1=1 rx=0x7f4e88009720 tx=0x7f4e8800b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:24.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:24.289 7f4e7bfff700 1 -- 172.21.2.102:0/3778129986 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4e880090c0 con 0x7f4e940b5b80 2021-02-10T11:27:24.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:24.289 7f4e9a08f700 1 -- 172.21.2.102:0/3778129986 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e940bcae0 con 0x7f4e940b5b80 2021-02-10T11:27:24.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:24.289 7f4e7bfff700 1 -- 172.21.2.102:0/3778129986 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e880090c0 con 0x7f4e940b5b80 2021-02-10T11:27:24.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:24.289 7f4e7bfff700 1 -- 172.21.2.102:0/3778129986 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4e88002030 con 0x7f4e940b5b80 2021-02-10T11:27:24.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:24.289 7f4e9a08f700 1 -- 172.21.2.102:0/3778129986 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e940bcae0 con 0x7f4e940b5b80 2021-02-10T11:27:24.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:24.290 7f4e7bfff700 1 -- 172.21.2.102:0/3778129986 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4e880052e0 con 0x7f4e940b5b80 2021-02-10T11:27:24.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:24.290 7f4e7bfff700 1 --2- 172.21.2.102:0/3778129986 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4e740186c0 0x7f4e7401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:24.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:24.290 7f4e7bfff700 1 -- 172.21.2.102:0/3778129986 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 5645+0+0 (secure 0 0 0) 0x7f4e88032660 con 0x7f4e940b5b80 2021-02-10T11:27:24.294 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:24.290 7f4e937fe700 1 --2- 172.21.2.102:0/3778129986 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4e740186c0 0x7f4e7401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:24.294 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:24.290 7f4e937fe700 1 --2- 172.21.2.102:0/3778129986 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4e740186c0 0x7f4e7401abc0 crc :-1 s=READY pgs=782 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:24.321 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:24.321 7f4e9a08f700 1 -- 172.21.2.102:0/3778129986 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e941a3c40 con 0x7f4e940b5b80 2021-02-10T11:27:24.323 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:24.323 7f4e7bfff700 1 -- 172.21.2.102:0/3778129986 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f4e88008fc0 con 0x7f4e940b5b80 2021-02-10T11:27:24.455 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:24.455 7f4e9a08f700 1 -- 172.21.2.102:0/3778129986 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestVolumes.test_clone_subvolume_info"]} v 0) v1 -- 0x7f4e941a3c40 con 0x7f4e940b5b80 2021-02-10T11:27:24.978 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:24.978 7f4e7bfff700 1 -- 172.21.2.102:0/3778129986 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestVolumes.test_clone_subvolume_info"]}]=0 v425) v1 ==== 142+0+0 (secure 0 0 0) 0x7f4e880052e0 con 0x7f4e940b5b80 2021-02-10T11:27:25.026 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:25.026 7f4e9a08f700 1 -- 172.21.2.102:0/3778129986 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4e740186c0 msgr2=0x7f4e7401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:25.026 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:25.026 7f4e9a08f700 1 --2- 172.21.2.102:0/3778129986 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4e740186c0 0x7f4e7401abc0 crc :-1 s=READY pgs=782 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:25.026 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:25.026 7f4e9a08f700 1 -- 172.21.2.102:0/3778129986 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4e940b5b80 msgr2=0x7f4e941a2e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:25.026 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:25.026 7f4e9a08f700 1 --2- 172.21.2.102:0/3778129986 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4e940b5b80 0x7f4e941a2e20 secure :-1 s=READY pgs=1726 cs=0 l=1 rev1=1 rx=0x7f4e88009720 tx=0x7f4e8800b960).stop 2021-02-10T11:27:25.027 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:25.026 7f4e9a08f700 1 -- 172.21.2.102:0/3778129986 shutdown_connections 2021-02-10T11:27:25.027 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:25.026 7f4e9a08f700 1 --2- 172.21.2.102:0/3778129986 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4e740186c0 0x7f4e7401abc0 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:25.027 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:25.026 7f4e9a08f700 1 --2- 172.21.2.102:0/3778129986 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4e940acfa0 0x7f4e941a3440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:25.027 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:25.026 7f4e9a08f700 1 --2- 172.21.2.102:0/3778129986 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4e940b5b80 0x7f4e941a2e20 unknown :-1 s=CLOSED pgs=1726 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:25.027 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:25.026 7f4e9a08f700 1 --2- 172.21.2.102:0/3778129986 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4e94195650 0x7f4e940b5560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:25.027 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:25.026 7f4e9a08f700 1 -- 172.21.2.102:0/3778129986 >> 172.21.2.102:0/3778129986 conn(0x7f4e940b3040 msgr2=0x7f4e940c3250 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:25.027 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:25.026 7f4e9a08f700 1 -- 172.21.2.102:0/3778129986 shutdown_connections 2021-02-10T11:27:25.027 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:25.027 7f4e9a08f700 1 -- 172.21.2.102:0/3778129986 wait complete. 2021-02-10T11:27:25.035 INFO:tasks.cephfs_test_runner:test_clone_subvolume_info (tasks.cephfs.test_volumes.TestVolumes) ... ok 2021-02-10T11:27:25.035 INFO:tasks.cephfs_test_runner:Starting test: test_connection_expiration (tasks.cephfs.test_volumes.TestVolumes) 2021-02-10T11:27:25.035 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestVolumes.test_connection_expiration' 2021-02-10T11:27:25.461 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:25.445 7f8b8c402700 1 Processor -- start 2021-02-10T11:27:25.462 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:25.445 7f8b8c402700 1 -- start start 2021-02-10T11:27:25.462 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:25.445 7f8b8c402700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f8b840d1c70 0x7f8b840f20a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:25.462 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:25.445 7f8b8c402700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b840f0b10 con 0x7f8b840d1870 2021-02-10T11:27:25.462 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:25.445 7f8b8c402700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b841b0be0 con 0x7f8b840d0340 2021-02-10T11:27:25.462 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:25.445 7f8b8c402700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b840e5ed0 con 0x7f8b840d1c70 2021-02-10T11:27:25.462 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:25.445 7f8b8aa1f700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f8b840d1c70 0x7f8b840f20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:25.462 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:25.446 7f8b8aa1f700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f8b840d1c70 0x7f8b840f20a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:59478/0 (socket says 172.21.2.102:59478) 2021-02-10T11:27:25.462 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:25.446 7f8b8aa1f700 1 -- 172.21.2.102:0/1730881900 learned_addr learned my addr 172.21.2.102:0/1730881900 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:27:25.462 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:25.446 7f8b8a21e700 1 --1- 172.21.2.102:0/1730881900 >> v1:172.21.2.102:6789/0 conn(0x7f8b840d1870 0x7f8b841b03f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:51702/0 (socket says 172.21.2.102:51702) 2021-02-10T11:27:25.462 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:25.446 7f8b8aa1f700 1 -- 172.21.2.102:0/1730881900 >> v1:172.21.2.113:6789/0 conn(0x7f8b840d0340 legacy=0x7f8b840ef640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:27:25.462 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:25.446 7f8b8aa1f700 1 -- 172.21.2.102:0/1730881900 >> v1:172.21.2.102:6789/0 conn(0x7f8b840d1870 legacy=0x7f8b841b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:25.463 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:25.446 7f8b8aa1f700 1 -- 172.21.2.102:0/1730881900 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b840efe30 con 0x7f8b840d1c70 2021-02-10T11:27:25.463 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:25.446 7f8b8aa1f700 1 --2- 172.21.2.102:0/1730881900 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8b840d1c70 0x7f8b840f20a0 secure :-1 s=READY pgs=1728 cs=0 l=1 rev1=1 rx=0x7f8b7c006f00 tx=0x7f8b7c009f10).ready entity=mon.0 client_cookie=a3c4c898898abac2 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:25.463 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:25.446 7f8b8921c700 1 -- 172.21.2.102:0/1730881900 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8b7c00b3b0 con 0x7f8b840d1c70 2021-02-10T11:27:25.463 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:25.447 7f8b8921c700 1 -- 172.21.2.102:0/1730881900 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b7c00b730 con 0x7f8b840d1c70 2021-02-10T11:27:25.463 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:25.447 7f8b8921c700 1 -- 172.21.2.102:0/1730881900 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8b7c00bc50 con 0x7f8b840d1c70 2021-02-10T11:27:25.463 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:25.447 7f8b8c402700 1 -- 172.21.2.102:0/1730881900 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8b840d1c70 msgr2=0x7f8b840f20a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:25.463 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:25.447 7f8b8c402700 1 --2- 172.21.2.102:0/1730881900 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8b840d1c70 0x7f8b840f20a0 secure :-1 s=READY pgs=1728 cs=0 l=1 rev1=1 rx=0x7f8b7c006f00 tx=0x7f8b7c009f10).stop 2021-02-10T11:27:25.463 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:25.447 7f8b8c402700 1 -- 172.21.2.102:0/1730881900 shutdown_connections 2021-02-10T11:27:25.463 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:25.447 7f8b8c402700 1 --2- 172.21.2.102:0/1730881900 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8b840d1c70 0x7f8b840f20a0 unknown :-1 s=CLOSED pgs=1728 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:25.463 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:25.447 7f8b8c402700 1 -- 172.21.2.102:0/1730881900 >> 172.21.2.102:0/1730881900 conn(0x7f8b840ce450 msgr2=0x7f8b840d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:25.463 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:25.447 7f8b8c402700 1 -- 172.21.2.102:0/1730881900 shutdown_connections 2021-02-10T11:27:25.464 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:25.447 7f8b8c402700 1 -- 172.21.2.102:0/1730881900 wait complete. 2021-02-10T11:27:25.465 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:25.462 7f8b8c402700 1 Processor -- start 2021-02-10T11:27:25.465 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:25.462 7f8b8c402700 1 -- start start 2021-02-10T11:27:25.465 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:25.462 7f8b8c402700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8b840955d0 0x7f8b840959d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:25.465 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:25.462 7f8b8c402700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8b84095ff0 0x7f8b840963f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:25.465 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:25.462 7f8b8c402700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8b84093440 0x7f8b84093840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:25.466 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:25.463 7f8b8c402700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b84095050 con 0x7f8b840955d0 2021-02-10T11:27:25.466 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:25.463 7f8b8c402700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b840b2510 con 0x7f8b84093440 2021-02-10T11:27:25.466 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:25.463 7f8b8c402700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b841be1f0 con 0x7f8b84095ff0 2021-02-10T11:27:25.466 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:25.463 7f8b8a21e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8b840955d0 0x7f8b840959d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:25.466 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:25.463 7f8b8a21e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8b840955d0 0x7f8b840959d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:59484/0 (socket says 172.21.2.102:59484) 2021-02-10T11:27:25.466 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:25.463 7f8b8a21e700 1 -- 172.21.2.102:0/4145939201 learned_addr learned my addr 172.21.2.102:0/4145939201 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:27:25.466 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:25.463 7f8b8a21e700 1 -- 172.21.2.102:0/4145939201 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8b84095ff0 msgr2=0x7f8b840963f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:25.466 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:25.463 7f8b8a21e700 1 --2- 172.21.2.102:0/4145939201 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8b84095ff0 0x7f8b840963f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:25.466 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:25.463 7f8b8a21e700 1 -- 172.21.2.102:0/4145939201 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8b84093440 msgr2=0x7f8b84093840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:25.466 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:25.463 7f8b8a21e700 1 --2- 172.21.2.102:0/4145939201 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8b84093440 0x7f8b84093840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:25.466 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:25.463 7f8b8a21e700 1 -- 172.21.2.102:0/4145939201 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b84093e60 con 0x7f8b840955d0 2021-02-10T11:27:25.467 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:25.463 7f8b8a21e700 1 --2- 172.21.2.102:0/4145939201 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8b840955d0 0x7f8b840959d0 secure :-1 s=READY pgs=1729 cs=0 l=1 rev1=1 rx=0x7f8b80001c60 tx=0x7f8b8000a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:25.467 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:25.464 7f8b7affd700 1 -- 172.21.2.102:0/4145939201 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8b8000acc0 con 0x7f8b840955d0 2021-02-10T11:27:25.467 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:25.464 7f8b7affd700 1 -- 172.21.2.102:0/4145939201 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b80004140 con 0x7f8b840955d0 2021-02-10T11:27:25.467 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:25.464 7f8b7affd700 1 -- 172.21.2.102:0/4145939201 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8b80004620 con 0x7f8b840955d0 2021-02-10T11:27:25.467 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:25.464 7f8b8c402700 1 -- 172.21.2.102:0/4145939201 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b84093e60 con 0x7f8b840955d0 2021-02-10T11:27:25.467 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:25.464 7f8b8c402700 1 -- 172.21.2.102:0/4145939201 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b840943f0 con 0x7f8b840955d0 2021-02-10T11:27:25.467 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:25.464 7f8b7affd700 1 -- 172.21.2.102:0/4145939201 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f8b80004320 con 0x7f8b840955d0 2021-02-10T11:27:25.467 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:25.464 7f8b7affd700 1 --2- 172.21.2.102:0/4145939201 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8b6c0186c0 0x7f8b6c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:25.467 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:25.464 7f8b7affd700 1 -- 172.21.2.102:0/4145939201 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 5645+0+0 (secure 0 0 0) 0x7f8b80032590 con 0x7f8b840955d0 2021-02-10T11:27:25.467 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:25.465 7f8b89a1d700 1 --2- 172.21.2.102:0/4145939201 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8b6c0186c0 0x7f8b6c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:25.467 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:25.465 7f8b89a1d700 1 --2- 172.21.2.102:0/4145939201 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8b6c0186c0 0x7f8b6c01abc0 crc :-1 s=READY pgs=783 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:25.495 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:25.495 7f8b8c402700 1 -- 172.21.2.102:0/4145939201 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b840e7df0 con 0x7f8b840955d0 2021-02-10T11:27:25.497 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:25.498 7f8b7affd700 1 -- 172.21.2.102:0/4145939201 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f8b80030d00 con 0x7f8b840955d0 2021-02-10T11:27:25.628 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:25.628 7f8b8c402700 1 -- 172.21.2.102:0/4145939201 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestVolumes.test_connection_expiration"]} v 0) v1 -- 0x7f8b840ce450 con 0x7f8b840955d0 2021-02-10T11:27:25.981 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:25.981 7f8b7affd700 1 -- 172.21.2.102:0/4145939201 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestVolumes.test_connection_expiration"]}]=0 v426) v1 ==== 146+0+0 (secure 0 0 0) 0x7f8b8001f4c0 con 0x7f8b840955d0 2021-02-10T11:27:25.998 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:25.997 7f8b8c402700 1 -- 172.21.2.102:0/4145939201 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8b6c0186c0 msgr2=0x7f8b6c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:25.998 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:25.997 7f8b8c402700 1 --2- 172.21.2.102:0/4145939201 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8b6c0186c0 0x7f8b6c01abc0 crc :-1 s=READY pgs=783 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:25.998 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:25.997 7f8b8c402700 1 -- 172.21.2.102:0/4145939201 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8b840955d0 msgr2=0x7f8b840959d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:25.998 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:25.997 7f8b8c402700 1 --2- 172.21.2.102:0/4145939201 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8b840955d0 0x7f8b840959d0 secure :-1 s=READY pgs=1729 cs=0 l=1 rev1=1 rx=0x7f8b80001c60 tx=0x7f8b8000a8d0).stop 2021-02-10T11:27:25.998 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:25.997 7f8b8c402700 1 -- 172.21.2.102:0/4145939201 shutdown_connections 2021-02-10T11:27:25.998 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:25.997 7f8b8c402700 1 --2- 172.21.2.102:0/4145939201 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8b6c0186c0 0x7f8b6c01abc0 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:25.998 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:25.997 7f8b8c402700 1 --2- 172.21.2.102:0/4145939201 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8b84093440 0x7f8b84093840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:25.999 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:25.997 7f8b8c402700 1 --2- 172.21.2.102:0/4145939201 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8b84095ff0 0x7f8b840963f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:25.999 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:25.997 7f8b8c402700 1 --2- 172.21.2.102:0/4145939201 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8b840955d0 0x7f8b840959d0 unknown :-1 s=CLOSED pgs=1729 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:25.999 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:25.997 7f8b8c402700 1 -- 172.21.2.102:0/4145939201 >> 172.21.2.102:0/4145939201 conn(0x7f8b840a0030 msgr2=0x7f8b840f22b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:25.999 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:25.998 7f8b8c402700 1 -- 172.21.2.102:0/4145939201 shutdown_connections 2021-02-10T11:27:25.999 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:25.998 7f8b8c402700 1 -- 172.21.2.102:0/4145939201 wait complete. 2021-02-10T11:27:26.006 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config set mon mon_allow_pool_delete True 2021-02-10T11:27:26.440 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:26.420 7f0ff331f700 1 Processor -- start 2021-02-10T11:27:26.440 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:26.420 7f0ff331f700 1 -- start start 2021-02-10T11:27:26.440 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:26.420 7f0ff331f700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f0fec0d1c70 0x7f0fec0cef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:26.440 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:26.420 7f0ff331f700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0fec0cf580 0x7f0fec0cf980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:26.440 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:26.420 7f0ff331f700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fec0fe1c0 con 0x7f0fec0d1870 2021-02-10T11:27:26.440 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:26.420 7f0ff331f700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0fec1b0cb0 con 0x7f0fec0d1c70 2021-02-10T11:27:26.440 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:26.420 7f0ff331f700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fec109a60 con 0x7f0fec0cf580 2021-02-10T11:27:26.441 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:26.420 7f0ff113b700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f0fec0d1870 0x7f0fec1b04c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:51716/0 (socket says 172.21.2.102:51716) 2021-02-10T11:27:26.441 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:26.420 7f0ff113b700 1 -- 172.21.2.102:0/4243749317 learned_addr learned my addr 172.21.2.102:0/4243749317 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:27:26.441 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:26.421 7f0ff093a700 1 --2- 172.21.2.102:0/4243749317 >> v2:172.21.2.113:3300/0 conn(0x7f0fec0d1c70 0x7f0fec0cef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:26.441 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:26.421 7f0ff193c700 1 --2- 172.21.2.102:0/4243749317 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0fec0cf580 0x7f0fec0cf980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:26.441 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:26.421 7f0fe3fff700 1 -- 172.21.2.102:0/4243749317 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 717687429 0 0) 0x7f0fdc0009b0 con 0x7f0fec0d1870 2021-02-10T11:27:26.441 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:26.421 7f0fe3fff700 1 -- 172.21.2.102:0/4243749317 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0fd8002fe0 con 0x7f0fec0d1870 2021-02-10T11:27:26.441 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:26.421 7f0ff193c700 1 -- 172.21.2.102:0/4243749317 >> v2:172.21.2.113:3300/0 conn(0x7f0fec0d1c70 msgr2=0x7f0fec0cef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:27:26.441 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:26.421 7f0ff193c700 1 --2- 172.21.2.102:0/4243749317 >> v2:172.21.2.113:3300/0 conn(0x7f0fec0d1c70 0x7f0fec0cef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:26.442 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:26.421 7f0ff193c700 1 -- 172.21.2.102:0/4243749317 >> v1:172.21.2.102:6789/0 conn(0x7f0fec0d1870 legacy=0x7f0fec1b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:26.442 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:26.421 7f0ff193c700 1 -- 172.21.2.102:0/4243749317 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fec088070 con 0x7f0fec0cf580 2021-02-10T11:27:26.442 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:26.421 7f0ff193c700 1 --2- 172.21.2.102:0/4243749317 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0fec0cf580 0x7f0fec0cf980 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f0fe4006e50 tx=0x7f0fe4009e70).ready entity=mon.2 client_cookie=35458fc7feafa1c1 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:26.442 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:26.421 7f0fe3fff700 1 -- 172.21.2.102:0/4243749317 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0fe400b510 con 0x7f0fec0cf580 2021-02-10T11:27:26.442 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:26.421 7f0fe3fff700 1 -- 172.21.2.102:0/4243749317 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fe4006790 con 0x7f0fec0cf580 2021-02-10T11:27:26.442 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:26.421 7f0fe3fff700 1 -- 172.21.2.102:0/4243749317 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0fe400b940 con 0x7f0fec0cf580 2021-02-10T11:27:26.442 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:26.422 7f0ff331f700 1 -- 172.21.2.102:0/4243749317 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0fec0cf580 msgr2=0x7f0fec0cf980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:26.442 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:26.422 7f0ff331f700 1 --2- 172.21.2.102:0/4243749317 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0fec0cf580 0x7f0fec0cf980 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f0fe4006e50 tx=0x7f0fe4009e70).stop 2021-02-10T11:27:26.442 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:26.422 7f0ff331f700 1 -- 172.21.2.102:0/4243749317 shutdown_connections 2021-02-10T11:27:26.443 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:26.422 7f0ff331f700 1 --2- 172.21.2.102:0/4243749317 >> v2:172.21.2.113:3300/0 conn(0x7f0fec0d1c70 0x7f0fec0cef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:26.443 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:26.422 7f0ff331f700 1 --2- 172.21.2.102:0/4243749317 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0fec0cf580 0x7f0fec0cf980 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:26.443 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:26.422 7f0ff331f700 1 -- 172.21.2.102:0/4243749317 >> 172.21.2.102:0/4243749317 conn(0x7f0fec0ce420 msgr2=0x7f0fec0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:26.443 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:26.422 7f0ff331f700 1 -- 172.21.2.102:0/4243749317 shutdown_connections 2021-02-10T11:27:26.443 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:26.423 7f0ff331f700 1 -- 172.21.2.102:0/4243749317 wait complete. 2021-02-10T11:27:26.443 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:26.437 7f0ff331f700 1 Processor -- start 2021-02-10T11:27:26.443 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:26.437 7f0ff331f700 1 -- start start 2021-02-10T11:27:26.443 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:26.438 7f0ff331f700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0fec085bc0 0x7f0fec0863e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:26.444 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:26.438 7f0ff331f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0fec0954c0 0x7f0fec096300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:26.444 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:26.438 7f0ff331f700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0fec0958c0 0x7f0fec096920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:26.444 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:26.438 7f0ff331f700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fec095d80 con 0x7f0fec0954c0 2021-02-10T11:27:26.444 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:26.438 7f0ff331f700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fec0bfa60 con 0x7f0fec085bc0 2021-02-10T11:27:26.444 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:26.438 7f0ff331f700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fec085fc0 con 0x7f0fec0958c0 2021-02-10T11:27:26.444 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:26.438 7f0ff093a700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0fec0954c0 0x7f0fec096300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:26.444 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:26.438 7f0ff093a700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0fec0954c0 0x7f0fec096300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:59500/0 (socket says 172.21.2.102:59500) 2021-02-10T11:27:26.444 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:26.438 7f0ff093a700 1 -- 172.21.2.102:0/3693502735 learned_addr learned my addr 172.21.2.102:0/3693502735 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:27:26.444 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:26.438 7f0ff093a700 1 -- 172.21.2.102:0/3693502735 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0fec0958c0 msgr2=0x7f0fec096920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:26.445 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:26.438 7f0ff093a700 1 --2- 172.21.2.102:0/3693502735 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0fec0958c0 0x7f0fec096920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:26.445 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:26.438 7f0ff093a700 1 -- 172.21.2.102:0/3693502735 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0fec085bc0 msgr2=0x7f0fec0863e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:26.445 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:26.438 7f0ff093a700 1 --2- 172.21.2.102:0/3693502735 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0fec085bc0 0x7f0fec0863e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:26.445 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:26.438 7f0ff093a700 1 -- 172.21.2.102:0/3693502735 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fec086c10 con 0x7f0fec0954c0 2021-02-10T11:27:26.445 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:26.438 7f0ff093a700 1 --2- 172.21.2.102:0/3693502735 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0fec0954c0 0x7f0fec096300 secure :-1 s=READY pgs=1731 cs=0 l=1 rev1=1 rx=0x7f0fe8006f60 tx=0x7f0fe800b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:26.445 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:26.438 7f0fe1ffb700 1 -- 172.21.2.102:0/3693502735 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0fe800bd60 con 0x7f0fec0954c0 2021-02-10T11:27:26.445 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:26.439 7f0fe1ffb700 1 -- 172.21.2.102:0/3693502735 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fe8000c90 con 0x7f0fec0954c0 2021-02-10T11:27:26.445 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:26.439 7f0ff331f700 1 -- 172.21.2.102:0/3693502735 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fec086c10 con 0x7f0fec0954c0 2021-02-10T11:27:26.445 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:26.439 7f0ff331f700 1 -- 172.21.2.102:0/3693502735 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fec097230 con 0x7f0fec0954c0 2021-02-10T11:27:26.446 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:26.439 7f0fe1ffb700 1 -- 172.21.2.102:0/3693502735 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0fe8002220 con 0x7f0fec0954c0 2021-02-10T11:27:26.446 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:26.439 7f0fe1ffb700 1 -- 172.21.2.102:0/3693502735 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f0fe80052e0 con 0x7f0fec0954c0 2021-02-10T11:27:26.446 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:26.439 7f0fe1ffb700 1 --2- 172.21.2.102:0/3693502735 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0fcc0186c0 0x7f0fcc01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:26.446 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:26.439 7f0fe1ffb700 1 -- 172.21.2.102:0/3693502735 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 5617+0+0 (secure 0 0 0) 0x7f0fe8032810 con 0x7f0fec0954c0 2021-02-10T11:27:26.446 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:26.439 7f0ff113b700 1 --2- 172.21.2.102:0/3693502735 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0fcc0186c0 0x7f0fcc01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:26.447 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:26.440 7f0ff113b700 1 --2- 172.21.2.102:0/3693502735 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0fcc0186c0 0x7f0fcc01abc0 crc :-1 s=READY pgs=784 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:26.471 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:26.470 7f0ff331f700 1 -- 172.21.2.102:0/3693502735 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fec0fe1c0 con 0x7f0fec0954c0 2021-02-10T11:27:26.473 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:26.473 7f0fe1ffb700 1 -- 172.21.2.102:0/3693502735 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f0fe80009b0 con 0x7f0fec0954c0 2021-02-10T11:27:26.622 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:26.622 7f0ff331f700 1 -- 172.21.2.102:0/3693502735 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f0fec0fe1c0 con 0x7f0fec0954c0 2021-02-10T11:27:26.626 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:26.626 7f0fe1ffb700 1 -- 172.21.2.102:0/3693502735 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v22)=0 v22) v1 ==== 122+0+0 (secure 0 0 0) 0x7f0fe80052e0 con 0x7f0fec0954c0 2021-02-10T11:27:26.631 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:26.631 7f0ff331f700 1 -- 172.21.2.102:0/3693502735 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0fcc0186c0 msgr2=0x7f0fcc01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:26.631 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:26.631 7f0ff331f700 1 --2- 172.21.2.102:0/3693502735 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0fcc0186c0 0x7f0fcc01abc0 crc :-1 s=READY pgs=784 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:26.631 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:26.631 7f0ff331f700 1 -- 172.21.2.102:0/3693502735 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0fec0954c0 msgr2=0x7f0fec096300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:26.631 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:26.631 7f0ff331f700 1 --2- 172.21.2.102:0/3693502735 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0fec0954c0 0x7f0fec096300 secure :-1 s=READY pgs=1731 cs=0 l=1 rev1=1 rx=0x7f0fe8006f60 tx=0x7f0fe800b970).stop 2021-02-10T11:27:26.632 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:26.631 7f0ff331f700 1 -- 172.21.2.102:0/3693502735 shutdown_connections 2021-02-10T11:27:26.632 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:26.631 7f0ff331f700 1 --2- 172.21.2.102:0/3693502735 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0fec0958c0 0x7f0fec096920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:26.632 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:26.631 7f0ff331f700 1 --2- 172.21.2.102:0/3693502735 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0fec0954c0 0x7f0fec096300 unknown :-1 s=CLOSED pgs=1731 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:26.632 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:26.631 7f0ff331f700 1 --2- 172.21.2.102:0/3693502735 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0fcc0186c0 0x7f0fcc01abc0 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:26.632 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:26.631 7f0ff331f700 1 --2- 172.21.2.102:0/3693502735 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0fec085bc0 0x7f0fec0863e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:26.632 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:26.631 7f0ff331f700 1 -- 172.21.2.102:0/3693502735 >> 172.21.2.102:0/3693502735 conn(0x7f0fec0bef00 msgr2=0x7f0fec09d1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:26.632 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:26.632 7f0ff331f700 1 -- 172.21.2.102:0/3693502735 shutdown_connections 2021-02-10T11:27:26.632 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:26.632 7f0ff331f700 1 -- 172.21.2.102:0/3693502735 wait complete. 2021-02-10T11:27:26.640 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2021-02-10T11:27:26.640 DEBUG:teuthology.orchestra.run.gibba002:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:27:26.993 DEBUG:teuthology.orchestra.run.gibba013:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:27:27.340 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2021-02-10T11:27:27.340 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:27.341 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:27:27.341 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:27:27.676 INFO:teuthology.orchestra.run.gibba002.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.0’: No such file or directory 2021-02-10T11:27:27.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:27:27.677 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.0 2021-02-10T11:27:27.677 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:27.677 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:27:27.677 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:27:28.011 INFO:teuthology.orchestra.run.gibba002.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.1’: No such file or directory 2021-02-10T11:27:28.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:27:28.012 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2021-02-10T11:27:28.012 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:28.012 DEBUG:teuthology.orchestra.run.gibba013:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:27:28.012 DEBUG:teuthology.orchestra.run.gibba013:> ' -- /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:27:28.347 INFO:teuthology.orchestra.run.gibba013.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.2’: No such file or directory 2021-02-10T11:27:28.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:27:28.348 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2021-02-10T11:27:28.348 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:28.348 DEBUG:teuthology.orchestra.run.gibba013:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:27:28.348 DEBUG:teuthology.orchestra.run.gibba013:> ' -- /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:27:28.683 INFO:teuthology.orchestra.run.gibba013.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.3’: No such file or directory 2021-02-10T11:27:28.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:27:28.683 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.3 2021-02-10T11:27:28.683 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json-pretty 2021-02-10T11:27:29.113 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:29.097 7f91af87a700 1 Processor -- start 2021-02-10T11:27:29.114 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:29.097 7f91af87a700 1 -- start start 2021-02-10T11:27:29.114 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:29.098 7f91af87a700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f91a80d1870 0x7f91a81b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:29.114 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:29.098 7f91af87a700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f91a81b0a10 0x7f91a80cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:29.114 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:29.098 7f91af87a700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f91a80d1c70 0x7f91a80cf950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:29.114 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:29.098 7f91af87a700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91a80e5ed0 con 0x7f91a81b0a10 2021-02-10T11:27:29.114 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:29.098 7f91af87a700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91a80f0b10 con 0x7f91a80d1870 2021-02-10T11:27:29.114 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:29.098 7f91af87a700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91a80cff70 con 0x7f91a80d1c70 2021-02-10T11:27:29.114 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:29.098 7f91ace95700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f91a81b0a10 0x7f91a80cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:29.114 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:29.098 7f91ace95700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f91a81b0a10 0x7f91a80cf330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:59506/0 (socket says 172.21.2.102:59506) 2021-02-10T11:27:29.114 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:29.098 7f91ace95700 1 -- 172.21.2.102:0/1491562068 learned_addr learned my addr 172.21.2.102:0/1491562068 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:27:29.115 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:29.098 7f91ad696700 1 --2- 172.21.2.102:0/1491562068 >> v2:172.21.2.113:3300/0 conn(0x7f91a80d1870 0x7f91a81b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:29.115 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:29.098 7f91ace95700 1 -- 172.21.2.102:0/1491562068 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f91a80d1c70 msgr2=0x7f91a80cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:27:29.115 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:29.098 7f91ade97700 1 --2- 172.21.2.102:0/1491562068 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f91a80d1c70 0x7f91a80cf950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:29.115 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:29.098 7f91ace95700 1 --2- 172.21.2.102:0/1491562068 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f91a80d1c70 0x7f91a80cf950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:29.115 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:29.098 7f91ace95700 1 -- 172.21.2.102:0/1491562068 >> v2:172.21.2.113:3300/0 conn(0x7f91a80d1870 msgr2=0x7f91a81b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:27:29.115 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:29.098 7f91ace95700 1 --2- 172.21.2.102:0/1491562068 >> v2:172.21.2.113:3300/0 conn(0x7f91a80d1870 0x7f91a81b03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:29.115 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:29.098 7f91ace95700 1 -- 172.21.2.102:0/1491562068 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91a80d0150 con 0x7f91a81b0a10 2021-02-10T11:27:29.115 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:29.098 7f91ade97700 1 --2- 172.21.2.102:0/1491562068 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f91a80d1c70 0x7f91a80cf950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:27:29.115 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:29.098 7f91ad696700 1 --2- 172.21.2.102:0/1491562068 >> v2:172.21.2.113:3300/0 conn(0x7f91a80d1870 0x7f91a81b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:27:29.115 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:29.098 7f91ace95700 1 --2- 172.21.2.102:0/1491562068 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f91a81b0a10 0x7f91a80cf330 secure :-1 s=READY pgs=1732 cs=0 l=1 rev1=1 rx=0x7f91a4006700 tx=0x7f91a4009b80).ready entity=mon.0 client_cookie=a8f7a56191396d31 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:29.115 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:29.098 7f919ffff700 1 -- 172.21.2.102:0/1491562068 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f91a4006340 con 0x7f91a81b0a10 2021-02-10T11:27:29.115 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:29.099 7f919ffff700 1 -- 172.21.2.102:0/1491562068 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91a400c0c0 con 0x7f91a81b0a10 2021-02-10T11:27:29.116 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:29.099 7f919ffff700 1 -- 172.21.2.102:0/1491562068 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f91a400c520 con 0x7f91a81b0a10 2021-02-10T11:27:29.116 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:29.099 7f91af87a700 1 -- 172.21.2.102:0/1491562068 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f91a81b0a10 msgr2=0x7f91a80cf330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:29.116 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:29.099 7f91af87a700 1 --2- 172.21.2.102:0/1491562068 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f91a81b0a10 0x7f91a80cf330 secure :-1 s=READY pgs=1732 cs=0 l=1 rev1=1 rx=0x7f91a4006700 tx=0x7f91a4009b80).stop 2021-02-10T11:27:29.116 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:29.099 7f91af87a700 1 -- 172.21.2.102:0/1491562068 shutdown_connections 2021-02-10T11:27:29.116 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:29.099 7f91af87a700 1 --2- 172.21.2.102:0/1491562068 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f91a80d1c70 0x7f91a80cf950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:29.116 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:29.099 7f91af87a700 1 --2- 172.21.2.102:0/1491562068 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f91a81b0a10 0x7f91a80cf330 unknown :-1 s=CLOSED pgs=1732 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:29.116 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:29.099 7f91af87a700 1 --2- 172.21.2.102:0/1491562068 >> v2:172.21.2.113:3300/0 conn(0x7f91a80d1870 0x7f91a81b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:29.116 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:29.099 7f91af87a700 1 -- 172.21.2.102:0/1491562068 >> 172.21.2.102:0/1491562068 conn(0x7f91a80ce450 msgr2=0x7f91a80d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:29.116 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:29.099 7f91af87a700 1 -- 172.21.2.102:0/1491562068 shutdown_connections 2021-02-10T11:27:29.116 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:29.099 7f91af87a700 1 -- 172.21.2.102:0/1491562068 wait complete. 2021-02-10T11:27:29.118 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:29.114 7f91af87a700 1 Processor -- start 2021-02-10T11:27:29.118 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:29.114 7f91af87a700 1 -- start start 2021-02-10T11:27:29.118 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:29.114 7f91af87a700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f91a81942b0 0x7f91a80b3550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:29.118 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:29.114 7f91af87a700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f91a8195070 0x7f91a8195470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:29.118 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:29.114 7f91af87a700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f91a80bb8e0 0x7f91a80bbce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:29.118 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:29.114 7f91af87a700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91a81946b0 con 0x7f91a80bb8e0 2021-02-10T11:27:29.118 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:29.114 7f91af87a700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91a80b3b70 con 0x7f91a8195070 2021-02-10T11:27:29.118 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:29.114 7f91af87a700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91a80b3d90 con 0x7f91a81942b0 2021-02-10T11:27:29.119 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:29.115 7f91ade97700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f91a80bb8e0 0x7f91a80bbce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:29.119 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:29.115 7f91ade97700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f91a80bb8e0 0x7f91a80bbce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:59516/0 (socket says 172.21.2.102:59516) 2021-02-10T11:27:29.119 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:29.115 7f91ade97700 1 -- 172.21.2.102:0/3025754720 learned_addr learned my addr 172.21.2.102:0/3025754720 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:27:29.119 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:29.115 7f91ade97700 1 -- 172.21.2.102:0/3025754720 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f91a81942b0 msgr2=0x7f91a80b3550 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:27:29.119 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:29.115 7f91ade97700 1 --2- 172.21.2.102:0/3025754720 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f91a81942b0 0x7f91a80b3550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:29.119 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:29.115 7f91ade97700 1 -- 172.21.2.102:0/3025754720 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f91a8195070 msgr2=0x7f91a8195470 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:27:29.119 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:29.115 7f91ade97700 1 --2- 172.21.2.102:0/3025754720 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f91a8195070 0x7f91a8195470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:29.119 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:29.115 7f91ade97700 1 -- 172.21.2.102:0/3025754720 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91a81a49e0 con 0x7f91a80bb8e0 2021-02-10T11:27:29.119 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:29.115 7f91ade97700 1 --2- 172.21.2.102:0/3025754720 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f91a80bb8e0 0x7f91a80bbce0 secure :-1 s=READY pgs=1733 cs=0 l=1 rev1=1 rx=0x7f9198006f10 tx=0x7f919800b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:29.119 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:29.115 7f919dffb700 1 -- 172.21.2.102:0/3025754720 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f919800bd80 con 0x7f91a80bb8e0 2021-02-10T11:27:29.119 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:29.115 7f919dffb700 1 -- 172.21.2.102:0/3025754720 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9198006410 con 0x7f91a80bb8e0 2021-02-10T11:27:29.119 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:29.115 7f91af87a700 1 -- 172.21.2.102:0/3025754720 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91a81a49e0 con 0x7f91a80bb8e0 2021-02-10T11:27:29.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:29.115 7f91af87a700 1 -- 172.21.2.102:0/3025754720 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91a80bc5f0 con 0x7f91a80bb8e0 2021-02-10T11:27:29.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:29.116 7f919dffb700 1 -- 172.21.2.102:0/3025754720 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9198002220 con 0x7f91a80bb8e0 2021-02-10T11:27:29.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:29.116 7f919dffb700 1 -- 172.21.2.102:0/3025754720 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f919800a030 con 0x7f91a80bb8e0 2021-02-10T11:27:29.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:29.116 7f919dffb700 1 --2- 172.21.2.102:0/3025754720 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f9188018730 0x7f918801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:29.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:29.116 7f919dffb700 1 -- 172.21.2.102:0/3025754720 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 5617+0+0 (secure 0 0 0) 0x7f9198032620 con 0x7f91a80bb8e0 2021-02-10T11:27:29.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:29.116 7f91ad696700 1 --2- 172.21.2.102:0/3025754720 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f9188018730 0x7f918801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:29.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:29.116 7f91ad696700 1 --2- 172.21.2.102:0/3025754720 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f9188018730 0x7f918801abc0 crc :-1 s=READY pgs=785 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:29.147 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:29.148 7f91af87a700 1 -- 172.21.2.102:0/3025754720 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91a80bc5f0 con 0x7f91a80bb8e0 2021-02-10T11:27:29.150 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:29.150 7f919dffb700 1 -- 172.21.2.102:0/3025754720 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f9198000c10 con 0x7f91a80bb8e0 2021-02-10T11:27:29.275 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:29.275 7f91af87a700 1 -- 172.21.2.102:0/3025754720 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7f91a80bc5f0 con 0x7f91a80bb8e0 2021-02-10T11:27:29.276 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:29.276 7f919dffb700 1 -- 172.21.2.102:0/3025754720 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v77) v1 ==== 81+0+30574 (secure 0 0 0) 0x7f919800a030 con 0x7f91a80bb8e0 2021-02-10T11:27:29.277 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:27:29.277 INFO:teuthology.orchestra.run.gibba002.stdout:{ 2021-02-10T11:27:29.277 INFO:teuthology.orchestra.run.gibba002.stdout: "epoch": 77, 2021-02-10T11:27:29.277 INFO:teuthology.orchestra.run.gibba002.stdout: "fsid": "68db30ea-dbce-4aef-a6df-f01c6fea0513", 2021-02-10T11:27:29.277 INFO:teuthology.orchestra.run.gibba002.stdout: "created": "2021-02-10 11:03:06.331557", 2021-02-10T11:27:29.277 INFO:teuthology.orchestra.run.gibba002.stdout: "modified": "2021-02-10 11:27:26.265425", 2021-02-10T11:27:29.277 INFO:teuthology.orchestra.run.gibba002.stdout: "last_up_change": "2021-02-10 11:03:32.690562", 2021-02-10T11:27:29.277 INFO:teuthology.orchestra.run.gibba002.stdout: "last_in_change": "2021-02-10 11:03:32.690562", 2021-02-10T11:27:29.277 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2021-02-10T11:27:29.277 INFO:teuthology.orchestra.run.gibba002.stdout: "flags_num": 5799936, 2021-02-10T11:27:29.278 INFO:teuthology.orchestra.run.gibba002.stdout: "flags_set": [ 2021-02-10T11:27:29.278 INFO:teuthology.orchestra.run.gibba002.stdout: "pglog_hardlimit", 2021-02-10T11:27:29.278 INFO:teuthology.orchestra.run.gibba002.stdout: "purged_snapdirs", 2021-02-10T11:27:29.278 INFO:teuthology.orchestra.run.gibba002.stdout: "recovery_deletes", 2021-02-10T11:27:29.278 INFO:teuthology.orchestra.run.gibba002.stdout: "sortbitwise" 2021-02-10T11:27:29.278 INFO:teuthology.orchestra.run.gibba002.stdout: ], 2021-02-10T11:27:29.278 INFO:teuthology.orchestra.run.gibba002.stdout: "crush_version": 4, 2021-02-10T11:27:29.278 INFO:teuthology.orchestra.run.gibba002.stdout: "full_ratio": 0.94999998807907104, 2021-02-10T11:27:29.278 INFO:teuthology.orchestra.run.gibba002.stdout: "backfillfull_ratio": 0.89999997615814209, 2021-02-10T11:27:29.278 INFO:teuthology.orchestra.run.gibba002.stdout: "nearfull_ratio": 0.85000002384185791, 2021-02-10T11:27:29.278 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_snapshot": "", 2021-02-10T11:27:29.278 INFO:teuthology.orchestra.run.gibba002.stdout: "pool_max": 15, 2021-02-10T11:27:29.278 INFO:teuthology.orchestra.run.gibba002.stdout: "max_osd": 8, 2021-02-10T11:27:29.279 INFO:teuthology.orchestra.run.gibba002.stdout: "require_min_compat_client": "jewel", 2021-02-10T11:27:29.279 INFO:teuthology.orchestra.run.gibba002.stdout: "min_compat_client": "jewel", 2021-02-10T11:27:29.279 INFO:teuthology.orchestra.run.gibba002.stdout: "require_osd_release": "nautilus", 2021-02-10T11:27:29.279 INFO:teuthology.orchestra.run.gibba002.stdout: "pools": [ 2021-02-10T11:27:29.279 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:27:29.279 INFO:teuthology.orchestra.run.gibba002.stdout: "pool": 1, 2021-02-10T11:27:29.279 INFO:teuthology.orchestra.run.gibba002.stdout: "pool_name": "rbd", 2021-02-10T11:27:29.279 INFO:teuthology.orchestra.run.gibba002.stdout: "create_time": "2021-02-10 11:03:37.322519", 2021-02-10T11:27:29.279 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 1, 2021-02-10T11:27:29.279 INFO:teuthology.orchestra.run.gibba002.stdout: "flags_names": "hashpspool", 2021-02-10T11:27:29.279 INFO:teuthology.orchestra.run.gibba002.stdout: "type": 1, 2021-02-10T11:27:29.279 INFO:teuthology.orchestra.run.gibba002.stdout: "size": 2, 2021-02-10T11:27:29.279 INFO:teuthology.orchestra.run.gibba002.stdout: "min_size": 1, 2021-02-10T11:27:29.280 INFO:teuthology.orchestra.run.gibba002.stdout: "crush_rule": 0, 2021-02-10T11:27:29.280 INFO:teuthology.orchestra.run.gibba002.stdout: "object_hash": 2, 2021-02-10T11:27:29.280 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:27:29.280 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num": 8, 2021-02-10T11:27:29.280 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_placement_num": 8, 2021-02-10T11:27:29.280 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_placement_num_target": 8, 2021-02-10T11:27:29.280 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num_target": 8, 2021-02-10T11:27:29.280 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num_pending": 8, 2021-02-10T11:27:29.280 INFO:teuthology.orchestra.run.gibba002.stdout: "last_pg_merge_meta": { 2021-02-10T11:27:29.280 INFO:teuthology.orchestra.run.gibba002.stdout: "source_pgid": "0.0", 2021-02-10T11:27:29.280 INFO:teuthology.orchestra.run.gibba002.stdout: "ready_epoch": 0, 2021-02-10T11:27:29.280 INFO:teuthology.orchestra.run.gibba002.stdout: "last_epoch_started": 0, 2021-02-10T11:27:29.280 INFO:teuthology.orchestra.run.gibba002.stdout: "last_epoch_clean": 0, 2021-02-10T11:27:29.280 INFO:teuthology.orchestra.run.gibba002.stdout: "source_version": "0'0", 2021-02-10T11:27:29.281 INFO:teuthology.orchestra.run.gibba002.stdout: "target_version": "0'0" 2021-02-10T11:27:29.281 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:27:29.281 INFO:teuthology.orchestra.run.gibba002.stdout: "last_change": "16", 2021-02-10T11:27:29.281 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend": "0", 2021-02-10T11:27:29.281 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:27:29.281 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:27:29.281 INFO:teuthology.orchestra.run.gibba002.stdout: "auid": 0, 2021-02-10T11:27:29.281 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:27:29.281 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_seq": 0, 2021-02-10T11:27:29.281 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_epoch": 0, 2021-02-10T11:27:29.281 INFO:teuthology.orchestra.run.gibba002.stdout: "pool_snaps": [], 2021-02-10T11:27:29.281 INFO:teuthology.orchestra.run.gibba002.stdout: "removed_snaps": "[]", 2021-02-10T11:27:29.281 INFO:teuthology.orchestra.run.gibba002.stdout: "quota_max_bytes": 0, 2021-02-10T11:27:29.281 INFO:teuthology.orchestra.run.gibba002.stdout: "quota_max_objects": 0, 2021-02-10T11:27:29.282 INFO:teuthology.orchestra.run.gibba002.stdout: "tiers": [], 2021-02-10T11:27:29.282 INFO:teuthology.orchestra.run.gibba002.stdout: "tier_of": -1, 2021-02-10T11:27:29.282 INFO:teuthology.orchestra.run.gibba002.stdout: "read_tier": -1, 2021-02-10T11:27:29.282 INFO:teuthology.orchestra.run.gibba002.stdout: "write_tier": -1, 2021-02-10T11:27:29.282 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_mode": "none", 2021-02-10T11:27:29.282 INFO:teuthology.orchestra.run.gibba002.stdout: "target_max_bytes": 0, 2021-02-10T11:27:29.282 INFO:teuthology.orchestra.run.gibba002.stdout: "target_max_objects": 0, 2021-02-10T11:27:29.282 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:27:29.282 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:27:29.282 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:27:29.282 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_min_flush_age": 0, 2021-02-10T11:27:29.282 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_min_evict_age": 0, 2021-02-10T11:27:29.282 INFO:teuthology.orchestra.run.gibba002.stdout: "erasure_code_profile": "", 2021-02-10T11:27:29.283 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_params": { 2021-02-10T11:27:29.283 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "none" 2021-02-10T11:27:29.283 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:27:29.283 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_period": 0, 2021-02-10T11:27:29.283 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_count": 0, 2021-02-10T11:27:29.283 INFO:teuthology.orchestra.run.gibba002.stdout: "use_gmt_hitset": true, 2021-02-10T11:27:29.283 INFO:teuthology.orchestra.run.gibba002.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:27:29.283 INFO:teuthology.orchestra.run.gibba002.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:27:29.283 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:27:29.283 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:27:29.283 INFO:teuthology.orchestra.run.gibba002.stdout: "grade_table": [], 2021-02-10T11:27:29.283 INFO:teuthology.orchestra.run.gibba002.stdout: "stripe_width": 0, 2021-02-10T11:27:29.283 INFO:teuthology.orchestra.run.gibba002.stdout: "expected_num_objects": 0, 2021-02-10T11:27:29.284 INFO:teuthology.orchestra.run.gibba002.stdout: "fast_read": false, 2021-02-10T11:27:29.284 INFO:teuthology.orchestra.run.gibba002.stdout: "options": {}, 2021-02-10T11:27:29.284 INFO:teuthology.orchestra.run.gibba002.stdout: "application_metadata": { 2021-02-10T11:27:29.284 INFO:teuthology.orchestra.run.gibba002.stdout: "rbd": {} 2021-02-10T11:27:29.284 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:27:29.284 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:27:29.284 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:27:29.284 INFO:teuthology.orchestra.run.gibba002.stdout: "pool": 14, 2021-02-10T11:27:29.284 INFO:teuthology.orchestra.run.gibba002.stdout: "pool_name": "cephfs_metadata", 2021-02-10T11:27:29.284 INFO:teuthology.orchestra.run.gibba002.stdout: "create_time": "2021-02-10 11:26:45.515299", 2021-02-10T11:27:29.284 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 1, 2021-02-10T11:27:29.284 INFO:teuthology.orchestra.run.gibba002.stdout: "flags_names": "hashpspool", 2021-02-10T11:27:29.284 INFO:teuthology.orchestra.run.gibba002.stdout: "type": 1, 2021-02-10T11:27:29.285 INFO:teuthology.orchestra.run.gibba002.stdout: "size": 2, 2021-02-10T11:27:29.285 INFO:teuthology.orchestra.run.gibba002.stdout: "min_size": 1, 2021-02-10T11:27:29.285 INFO:teuthology.orchestra.run.gibba002.stdout: "crush_rule": 0, 2021-02-10T11:27:29.285 INFO:teuthology.orchestra.run.gibba002.stdout: "object_hash": 2, 2021-02-10T11:27:29.285 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:27:29.285 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num": 8, 2021-02-10T11:27:29.285 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_placement_num": 8, 2021-02-10T11:27:29.285 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_placement_num_target": 8, 2021-02-10T11:27:29.285 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num_target": 8, 2021-02-10T11:27:29.285 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num_pending": 8, 2021-02-10T11:27:29.286 INFO:teuthology.orchestra.run.gibba002.stdout: "last_pg_merge_meta": { 2021-02-10T11:27:29.286 INFO:teuthology.orchestra.run.gibba002.stdout: "source_pgid": "0.0", 2021-02-10T11:27:29.286 INFO:teuthology.orchestra.run.gibba002.stdout: "ready_epoch": 0, 2021-02-10T11:27:29.286 INFO:teuthology.orchestra.run.gibba002.stdout: "last_epoch_started": 0, 2021-02-10T11:27:29.286 INFO:teuthology.orchestra.run.gibba002.stdout: "last_epoch_clean": 0, 2021-02-10T11:27:29.286 INFO:teuthology.orchestra.run.gibba002.stdout: "source_version": "0'0", 2021-02-10T11:27:29.286 INFO:teuthology.orchestra.run.gibba002.stdout: "target_version": "0'0" 2021-02-10T11:27:29.286 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:27:29.286 INFO:teuthology.orchestra.run.gibba002.stdout: "last_change": "74", 2021-02-10T11:27:29.286 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend": "0", 2021-02-10T11:27:29.286 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:27:29.286 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:27:29.287 INFO:teuthology.orchestra.run.gibba002.stdout: "auid": 0, 2021-02-10T11:27:29.287 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:27:29.287 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_seq": 0, 2021-02-10T11:27:29.287 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_epoch": 0, 2021-02-10T11:27:29.287 INFO:teuthology.orchestra.run.gibba002.stdout: "pool_snaps": [], 2021-02-10T11:27:29.287 INFO:teuthology.orchestra.run.gibba002.stdout: "removed_snaps": "[]", 2021-02-10T11:27:29.287 INFO:teuthology.orchestra.run.gibba002.stdout: "quota_max_bytes": 0, 2021-02-10T11:27:29.287 INFO:teuthology.orchestra.run.gibba002.stdout: "quota_max_objects": 0, 2021-02-10T11:27:29.287 INFO:teuthology.orchestra.run.gibba002.stdout: "tiers": [], 2021-02-10T11:27:29.287 INFO:teuthology.orchestra.run.gibba002.stdout: "tier_of": -1, 2021-02-10T11:27:29.287 INFO:teuthology.orchestra.run.gibba002.stdout: "read_tier": -1, 2021-02-10T11:27:29.287 INFO:teuthology.orchestra.run.gibba002.stdout: "write_tier": -1, 2021-02-10T11:27:29.287 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_mode": "none", 2021-02-10T11:27:29.287 INFO:teuthology.orchestra.run.gibba002.stdout: "target_max_bytes": 0, 2021-02-10T11:27:29.288 INFO:teuthology.orchestra.run.gibba002.stdout: "target_max_objects": 0, 2021-02-10T11:27:29.288 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:27:29.288 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:27:29.288 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:27:29.288 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_min_flush_age": 0, 2021-02-10T11:27:29.288 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_min_evict_age": 0, 2021-02-10T11:27:29.288 INFO:teuthology.orchestra.run.gibba002.stdout: "erasure_code_profile": "", 2021-02-10T11:27:29.288 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_params": { 2021-02-10T11:27:29.288 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "none" 2021-02-10T11:27:29.288 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:27:29.288 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_period": 0, 2021-02-10T11:27:29.288 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_count": 0, 2021-02-10T11:27:29.288 INFO:teuthology.orchestra.run.gibba002.stdout: "use_gmt_hitset": true, 2021-02-10T11:27:29.288 INFO:teuthology.orchestra.run.gibba002.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:27:29.288 INFO:teuthology.orchestra.run.gibba002.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:27:29.288 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:27:29.289 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:27:29.289 INFO:teuthology.orchestra.run.gibba002.stdout: "grade_table": [], 2021-02-10T11:27:29.289 INFO:teuthology.orchestra.run.gibba002.stdout: "stripe_width": 0, 2021-02-10T11:27:29.289 INFO:teuthology.orchestra.run.gibba002.stdout: "expected_num_objects": 0, 2021-02-10T11:27:29.289 INFO:teuthology.orchestra.run.gibba002.stdout: "fast_read": false, 2021-02-10T11:27:29.289 INFO:teuthology.orchestra.run.gibba002.stdout: "options": { 2021-02-10T11:27:29.289 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_autoscale_bias": 4, 2021-02-10T11:27:29.289 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num_min": 16, 2021-02-10T11:27:29.289 INFO:teuthology.orchestra.run.gibba002.stdout: "recovery_priority": 5 2021-02-10T11:27:29.289 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:27:29.289 INFO:teuthology.orchestra.run.gibba002.stdout: "application_metadata": { 2021-02-10T11:27:29.289 INFO:teuthology.orchestra.run.gibba002.stdout: "cephfs": { 2021-02-10T11:27:29.290 INFO:teuthology.orchestra.run.gibba002.stdout: "metadata": "cephfs" 2021-02-10T11:27:29.290 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:27:29.290 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:27:29.290 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:27:29.290 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:27:29.290 INFO:teuthology.orchestra.run.gibba002.stdout: "pool": 15, 2021-02-10T11:27:29.290 INFO:teuthology.orchestra.run.gibba002.stdout: "pool_name": "cephfs_data", 2021-02-10T11:27:29.290 INFO:teuthology.orchestra.run.gibba002.stdout: "create_time": "2021-02-10 11:26:46.587944", 2021-02-10T11:27:29.290 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 8193, 2021-02-10T11:27:29.290 INFO:teuthology.orchestra.run.gibba002.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2021-02-10T11:27:29.290 INFO:teuthology.orchestra.run.gibba002.stdout: "type": 1, 2021-02-10T11:27:29.290 INFO:teuthology.orchestra.run.gibba002.stdout: "size": 2, 2021-02-10T11:27:29.290 INFO:teuthology.orchestra.run.gibba002.stdout: "min_size": 1, 2021-02-10T11:27:29.290 INFO:teuthology.orchestra.run.gibba002.stdout: "crush_rule": 0, 2021-02-10T11:27:29.291 INFO:teuthology.orchestra.run.gibba002.stdout: "object_hash": 2, 2021-02-10T11:27:29.291 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:27:29.291 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num": 8, 2021-02-10T11:27:29.291 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_placement_num": 8, 2021-02-10T11:27:29.291 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_placement_num_target": 8, 2021-02-10T11:27:29.291 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num_target": 8, 2021-02-10T11:27:29.291 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num_pending": 8, 2021-02-10T11:27:29.291 INFO:teuthology.orchestra.run.gibba002.stdout: "last_pg_merge_meta": { 2021-02-10T11:27:29.291 INFO:teuthology.orchestra.run.gibba002.stdout: "source_pgid": "0.0", 2021-02-10T11:27:29.291 INFO:teuthology.orchestra.run.gibba002.stdout: "ready_epoch": 0, 2021-02-10T11:27:29.291 INFO:teuthology.orchestra.run.gibba002.stdout: "last_epoch_started": 0, 2021-02-10T11:27:29.291 INFO:teuthology.orchestra.run.gibba002.stdout: "last_epoch_clean": 0, 2021-02-10T11:27:29.291 INFO:teuthology.orchestra.run.gibba002.stdout: "source_version": "0'0", 2021-02-10T11:27:29.291 INFO:teuthology.orchestra.run.gibba002.stdout: "target_version": "0'0" 2021-02-10T11:27:29.292 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:27:29.292 INFO:teuthology.orchestra.run.gibba002.stdout: "last_change": "76", 2021-02-10T11:27:29.292 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend": "0", 2021-02-10T11:27:29.292 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:27:29.292 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:27:29.292 INFO:teuthology.orchestra.run.gibba002.stdout: "auid": 0, 2021-02-10T11:27:29.292 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:27:29.292 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_seq": 3, 2021-02-10T11:27:29.292 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_epoch": 76, 2021-02-10T11:27:29.292 INFO:teuthology.orchestra.run.gibba002.stdout: "pool_snaps": [], 2021-02-10T11:27:29.292 INFO:teuthology.orchestra.run.gibba002.stdout: "removed_snaps": "[2~2]", 2021-02-10T11:27:29.292 INFO:teuthology.orchestra.run.gibba002.stdout: "quota_max_bytes": 0, 2021-02-10T11:27:29.293 INFO:teuthology.orchestra.run.gibba002.stdout: "quota_max_objects": 0, 2021-02-10T11:27:29.293 INFO:teuthology.orchestra.run.gibba002.stdout: "tiers": [], 2021-02-10T11:27:29.293 INFO:teuthology.orchestra.run.gibba002.stdout: "tier_of": -1, 2021-02-10T11:27:29.293 INFO:teuthology.orchestra.run.gibba002.stdout: "read_tier": -1, 2021-02-10T11:27:29.293 INFO:teuthology.orchestra.run.gibba002.stdout: "write_tier": -1, 2021-02-10T11:27:29.293 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_mode": "none", 2021-02-10T11:27:29.293 INFO:teuthology.orchestra.run.gibba002.stdout: "target_max_bytes": 0, 2021-02-10T11:27:29.293 INFO:teuthology.orchestra.run.gibba002.stdout: "target_max_objects": 0, 2021-02-10T11:27:29.293 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:27:29.293 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:27:29.293 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:27:29.293 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_min_flush_age": 0, 2021-02-10T11:27:29.294 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_min_evict_age": 0, 2021-02-10T11:27:29.294 INFO:teuthology.orchestra.run.gibba002.stdout: "erasure_code_profile": "", 2021-02-10T11:27:29.294 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_params": { 2021-02-10T11:27:29.294 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "none" 2021-02-10T11:27:29.295 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:27:29.295 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_period": 0, 2021-02-10T11:27:29.295 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_count": 0, 2021-02-10T11:27:29.295 INFO:teuthology.orchestra.run.gibba002.stdout: "use_gmt_hitset": true, 2021-02-10T11:27:29.295 INFO:teuthology.orchestra.run.gibba002.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:27:29.295 INFO:teuthology.orchestra.run.gibba002.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:27:29.295 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:27:29.295 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:27:29.295 INFO:teuthology.orchestra.run.gibba002.stdout: "grade_table": [], 2021-02-10T11:27:29.295 INFO:teuthology.orchestra.run.gibba002.stdout: "stripe_width": 0, 2021-02-10T11:27:29.295 INFO:teuthology.orchestra.run.gibba002.stdout: "expected_num_objects": 0, 2021-02-10T11:27:29.295 INFO:teuthology.orchestra.run.gibba002.stdout: "fast_read": false, 2021-02-10T11:27:29.296 INFO:teuthology.orchestra.run.gibba002.stdout: "options": {}, 2021-02-10T11:27:29.296 INFO:teuthology.orchestra.run.gibba002.stdout: "application_metadata": { 2021-02-10T11:27:29.296 INFO:teuthology.orchestra.run.gibba002.stdout: "cephfs": { 2021-02-10T11:27:29.296 INFO:teuthology.orchestra.run.gibba002.stdout: "data": "cephfs" 2021-02-10T11:27:29.296 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:27:29.296 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:27:29.296 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:27:29.296 INFO:teuthology.orchestra.run.gibba002.stdout: ], 2021-02-10T11:27:29.296 INFO:teuthology.orchestra.run.gibba002.stdout: "osds": [ 2021-02-10T11:27:29.296 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:27:29.296 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 0, 2021-02-10T11:27:29.296 INFO:teuthology.orchestra.run.gibba002.stdout: "uuid": "be691b4d-a039-408a-af80-9d53f02b6f1f", 2021-02-10T11:27:29.297 INFO:teuthology.orchestra.run.gibba002.stdout: "up": 1, 2021-02-10T11:27:29.297 INFO:teuthology.orchestra.run.gibba002.stdout: "in": 1, 2021-02-10T11:27:29.297 INFO:teuthology.orchestra.run.gibba002.stdout: "weight": 1, 2021-02-10T11:27:29.297 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_affinity": 1, 2021-02-10T11:27:29.297 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_begin": 0, 2021-02-10T11:27:29.297 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_end": 0, 2021-02-10T11:27:29.297 INFO:teuthology.orchestra.run.gibba002.stdout: "up_from": 13, 2021-02-10T11:27:29.297 INFO:teuthology.orchestra.run.gibba002.stdout: "up_thru": 73, 2021-02-10T11:27:29.297 INFO:teuthology.orchestra.run.gibba002.stdout: "down_at": 0, 2021-02-10T11:27:29.297 INFO:teuthology.orchestra.run.gibba002.stdout: "lost_at": 0, 2021-02-10T11:27:29.297 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addrs": { 2021-02-10T11:27:29.297 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:27:29.297 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:27:29.297 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:27:29.298 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6818", 2021-02-10T11:27:29.298 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36088 2021-02-10T11:27:29.298 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:27:29.298 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:27:29.298 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:27:29.298 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6820", 2021-02-10T11:27:29.298 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36088 2021-02-10T11:27:29.298 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:27:29.298 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:27:29.298 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:27:29.298 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addrs": { 2021-02-10T11:27:29.298 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:27:29.298 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:27:29.299 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:27:29.299 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6822", 2021-02-10T11:27:29.299 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36088 2021-02-10T11:27:29.299 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:27:29.299 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:27:29.299 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:27:29.299 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6824", 2021-02-10T11:27:29.299 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36088 2021-02-10T11:27:29.299 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:27:29.299 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:27:29.299 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:27:29.299 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addrs": { 2021-02-10T11:27:29.299 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:27:29.299 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:27:29.300 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:27:29.300 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6830", 2021-02-10T11:27:29.300 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36088 2021-02-10T11:27:29.300 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:27:29.300 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:27:29.300 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:27:29.300 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6832", 2021-02-10T11:27:29.300 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36088 2021-02-10T11:27:29.300 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:27:29.300 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:27:29.300 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:27:29.300 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addrs": { 2021-02-10T11:27:29.300 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:27:29.301 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:27:29.301 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:27:29.301 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6826", 2021-02-10T11:27:29.301 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36088 2021-02-10T11:27:29.301 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:27:29.301 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:27:29.301 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:27:29.301 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6828", 2021-02-10T11:27:29.301 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36088 2021-02-10T11:27:29.301 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:27:29.301 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:27:29.301 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:27:29.301 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addr": "172.21.2.102:6820/36088", 2021-02-10T11:27:29.301 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addr": "172.21.2.102:6824/36088", 2021-02-10T11:27:29.302 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addr": "172.21.2.102:6832/36088", 2021-02-10T11:27:29.302 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addr": "172.21.2.102:6828/36088", 2021-02-10T11:27:29.302 INFO:teuthology.orchestra.run.gibba002.stdout: "state": [ 2021-02-10T11:27:29.302 INFO:teuthology.orchestra.run.gibba002.stdout: "exists", 2021-02-10T11:27:29.302 INFO:teuthology.orchestra.run.gibba002.stdout: "up" 2021-02-10T11:27:29.302 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:27:29.302 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:27:29.302 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:27:29.302 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 1, 2021-02-10T11:27:29.302 INFO:teuthology.orchestra.run.gibba002.stdout: "uuid": "64e3ccc8-9c69-4d7c-865e-af8e83822635", 2021-02-10T11:27:29.302 INFO:teuthology.orchestra.run.gibba002.stdout: "up": 1, 2021-02-10T11:27:29.302 INFO:teuthology.orchestra.run.gibba002.stdout: "in": 1, 2021-02-10T11:27:29.302 INFO:teuthology.orchestra.run.gibba002.stdout: "weight": 1, 2021-02-10T11:27:29.303 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_affinity": 1, 2021-02-10T11:27:29.303 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_begin": 0, 2021-02-10T11:27:29.303 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_end": 0, 2021-02-10T11:27:29.303 INFO:teuthology.orchestra.run.gibba002.stdout: "up_from": 13, 2021-02-10T11:27:29.303 INFO:teuthology.orchestra.run.gibba002.stdout: "up_thru": 73, 2021-02-10T11:27:29.303 INFO:teuthology.orchestra.run.gibba002.stdout: "down_at": 0, 2021-02-10T11:27:29.303 INFO:teuthology.orchestra.run.gibba002.stdout: "lost_at": 0, 2021-02-10T11:27:29.303 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addrs": { 2021-02-10T11:27:29.303 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:27:29.303 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:27:29.303 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:27:29.303 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6819", 2021-02-10T11:27:29.303 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36087 2021-02-10T11:27:29.304 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:27:29.304 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:27:29.304 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:27:29.304 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6821", 2021-02-10T11:27:29.304 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36087 2021-02-10T11:27:29.304 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:27:29.304 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:27:29.304 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:27:29.304 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addrs": { 2021-02-10T11:27:29.304 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:27:29.304 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:27:29.304 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:27:29.304 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6823", 2021-02-10T11:27:29.305 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36087 2021-02-10T11:27:29.305 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:27:29.305 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:27:29.305 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:27:29.305 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6825", 2021-02-10T11:27:29.305 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36087 2021-02-10T11:27:29.305 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:27:29.305 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:27:29.305 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:27:29.305 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addrs": { 2021-02-10T11:27:29.305 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:27:29.305 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:27:29.305 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:27:29.305 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6831", 2021-02-10T11:27:29.305 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36087 2021-02-10T11:27:29.306 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:27:29.306 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:27:29.306 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:27:29.306 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6833", 2021-02-10T11:27:29.306 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36087 2021-02-10T11:27:29.306 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:27:29.306 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:27:29.306 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:27:29.306 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addrs": { 2021-02-10T11:27:29.306 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:27:29.306 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:27:29.306 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:27:29.306 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6827", 2021-02-10T11:27:29.307 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36087 2021-02-10T11:27:29.307 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:27:29.307 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:27:29.307 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:27:29.307 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6829", 2021-02-10T11:27:29.307 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36087 2021-02-10T11:27:29.307 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:27:29.307 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:27:29.307 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:27:29.307 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addr": "172.21.2.102:6821/36087", 2021-02-10T11:27:29.307 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addr": "172.21.2.102:6825/36087", 2021-02-10T11:27:29.307 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addr": "172.21.2.102:6833/36087", 2021-02-10T11:27:29.307 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addr": "172.21.2.102:6829/36087", 2021-02-10T11:27:29.308 INFO:teuthology.orchestra.run.gibba002.stdout: "state": [ 2021-02-10T11:27:29.308 INFO:teuthology.orchestra.run.gibba002.stdout: "exists", 2021-02-10T11:27:29.308 INFO:teuthology.orchestra.run.gibba002.stdout: "up" 2021-02-10T11:27:29.308 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:27:29.308 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:27:29.308 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:27:29.308 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 2, 2021-02-10T11:27:29.308 INFO:teuthology.orchestra.run.gibba002.stdout: "uuid": "153700af-747f-481f-bdf4-d6072d3a842f", 2021-02-10T11:27:29.308 INFO:teuthology.orchestra.run.gibba002.stdout: "up": 1, 2021-02-10T11:27:29.308 INFO:teuthology.orchestra.run.gibba002.stdout: "in": 1, 2021-02-10T11:27:29.308 INFO:teuthology.orchestra.run.gibba002.stdout: "weight": 1, 2021-02-10T11:27:29.308 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_affinity": 1, 2021-02-10T11:27:29.308 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_begin": 0, 2021-02-10T11:27:29.308 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_end": 0, 2021-02-10T11:27:29.308 INFO:teuthology.orchestra.run.gibba002.stdout: "up_from": 13, 2021-02-10T11:27:29.309 INFO:teuthology.orchestra.run.gibba002.stdout: "up_thru": 73, 2021-02-10T11:27:29.309 INFO:teuthology.orchestra.run.gibba002.stdout: "down_at": 0, 2021-02-10T11:27:29.309 INFO:teuthology.orchestra.run.gibba002.stdout: "lost_at": 0, 2021-02-10T11:27:29.309 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addrs": { 2021-02-10T11:27:29.309 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:27:29.309 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:27:29.309 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:27:29.309 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6810", 2021-02-10T11:27:29.309 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36089 2021-02-10T11:27:29.309 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:27:29.309 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:27:29.309 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:27:29.309 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6811", 2021-02-10T11:27:29.310 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36089 2021-02-10T11:27:29.310 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:27:29.310 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:27:29.310 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:27:29.310 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addrs": { 2021-02-10T11:27:29.310 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:27:29.310 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:27:29.310 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:27:29.310 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6812", 2021-02-10T11:27:29.310 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36089 2021-02-10T11:27:29.310 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:27:29.310 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:27:29.310 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:27:29.311 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6813", 2021-02-10T11:27:29.311 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36089 2021-02-10T11:27:29.311 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:27:29.311 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:27:29.311 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:27:29.311 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addrs": { 2021-02-10T11:27:29.311 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:27:29.311 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:27:29.311 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:27:29.311 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6816", 2021-02-10T11:27:29.311 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36089 2021-02-10T11:27:29.311 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:27:29.311 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:27:29.312 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:27:29.312 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6817", 2021-02-10T11:27:29.312 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36089 2021-02-10T11:27:29.312 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:27:29.312 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:27:29.312 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:27:29.312 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addrs": { 2021-02-10T11:27:29.312 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:27:29.312 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:27:29.312 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:27:29.312 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6814", 2021-02-10T11:27:29.312 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36089 2021-02-10T11:27:29.312 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:27:29.313 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:27:29.313 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:27:29.313 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6815", 2021-02-10T11:27:29.313 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36089 2021-02-10T11:27:29.313 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:27:29.313 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:27:29.313 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:27:29.313 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addr": "172.21.2.102:6811/36089", 2021-02-10T11:27:29.313 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addr": "172.21.2.102:6813/36089", 2021-02-10T11:27:29.313 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addr": "172.21.2.102:6817/36089", 2021-02-10T11:27:29.313 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addr": "172.21.2.102:6815/36089", 2021-02-10T11:27:29.313 INFO:teuthology.orchestra.run.gibba002.stdout: "state": [ 2021-02-10T11:27:29.313 INFO:teuthology.orchestra.run.gibba002.stdout: "exists", 2021-02-10T11:27:29.314 INFO:teuthology.orchestra.run.gibba002.stdout: "up" 2021-02-10T11:27:29.314 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:27:29.314 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:27:29.314 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:27:29.314 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 3, 2021-02-10T11:27:29.314 INFO:teuthology.orchestra.run.gibba002.stdout: "uuid": "cbc678f4-83a9-45e6-8830-2f7c232b1ea7", 2021-02-10T11:27:29.314 INFO:teuthology.orchestra.run.gibba002.stdout: "up": 1, 2021-02-10T11:27:29.314 INFO:teuthology.orchestra.run.gibba002.stdout: "in": 1, 2021-02-10T11:27:29.314 INFO:teuthology.orchestra.run.gibba002.stdout: "weight": 1, 2021-02-10T11:27:29.314 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_affinity": 1, 2021-02-10T11:27:29.314 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_begin": 0, 2021-02-10T11:27:29.314 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_end": 0, 2021-02-10T11:27:29.315 INFO:teuthology.orchestra.run.gibba002.stdout: "up_from": 13, 2021-02-10T11:27:29.315 INFO:teuthology.orchestra.run.gibba002.stdout: "up_thru": 72, 2021-02-10T11:27:29.315 INFO:teuthology.orchestra.run.gibba002.stdout: "down_at": 0, 2021-02-10T11:27:29.315 INFO:teuthology.orchestra.run.gibba002.stdout: "lost_at": 0, 2021-02-10T11:27:29.315 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addrs": { 2021-02-10T11:27:29.315 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:27:29.315 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:27:29.315 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:27:29.315 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6802", 2021-02-10T11:27:29.315 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36090 2021-02-10T11:27:29.315 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:27:29.315 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:27:29.315 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:27:29.315 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6803", 2021-02-10T11:27:29.316 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36090 2021-02-10T11:27:29.316 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:27:29.316 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:27:29.316 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:27:29.316 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addrs": { 2021-02-10T11:27:29.316 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:27:29.316 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:27:29.316 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:27:29.316 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6804", 2021-02-10T11:27:29.316 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36090 2021-02-10T11:27:29.316 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:27:29.316 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:27:29.317 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:27:29.317 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6805", 2021-02-10T11:27:29.317 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36090 2021-02-10T11:27:29.317 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:27:29.317 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:27:29.317 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:27:29.317 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addrs": { 2021-02-10T11:27:29.317 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:27:29.317 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:27:29.317 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:27:29.317 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6808", 2021-02-10T11:27:29.317 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36090 2021-02-10T11:27:29.317 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:27:29.318 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:27:29.318 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:27:29.318 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6809", 2021-02-10T11:27:29.318 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36090 2021-02-10T11:27:29.318 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:27:29.318 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:27:29.318 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:27:29.318 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addrs": { 2021-02-10T11:27:29.318 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:27:29.318 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:27:29.318 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:27:29.318 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6806", 2021-02-10T11:27:29.318 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36090 2021-02-10T11:27:29.319 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:27:29.319 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:27:29.319 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:27:29.319 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6807", 2021-02-10T11:27:29.319 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36090 2021-02-10T11:27:29.319 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:27:29.319 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:27:29.319 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:27:29.319 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addr": "172.21.2.102:6803/36090", 2021-02-10T11:27:29.319 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addr": "172.21.2.102:6805/36090", 2021-02-10T11:27:29.319 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addr": "172.21.2.102:6809/36090", 2021-02-10T11:27:29.319 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addr": "172.21.2.102:6807/36090", 2021-02-10T11:27:29.319 INFO:teuthology.orchestra.run.gibba002.stdout: "state": [ 2021-02-10T11:27:29.320 INFO:teuthology.orchestra.run.gibba002.stdout: "exists", 2021-02-10T11:27:29.320 INFO:teuthology.orchestra.run.gibba002.stdout: "up" 2021-02-10T11:27:29.320 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:27:29.320 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:27:29.320 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:27:29.320 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 4, 2021-02-10T11:27:29.320 INFO:teuthology.orchestra.run.gibba002.stdout: "uuid": "c01e4008-b258-4f10-afd2-9e7f37214351", 2021-02-10T11:27:29.320 INFO:teuthology.orchestra.run.gibba002.stdout: "up": 1, 2021-02-10T11:27:29.320 INFO:teuthology.orchestra.run.gibba002.stdout: "in": 1, 2021-02-10T11:27:29.320 INFO:teuthology.orchestra.run.gibba002.stdout: "weight": 1, 2021-02-10T11:27:29.320 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_affinity": 1, 2021-02-10T11:27:29.320 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_begin": 0, 2021-02-10T11:27:29.320 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_end": 0, 2021-02-10T11:27:29.320 INFO:teuthology.orchestra.run.gibba002.stdout: "up_from": 13, 2021-02-10T11:27:29.321 INFO:teuthology.orchestra.run.gibba002.stdout: "up_thru": 73, 2021-02-10T11:27:29.321 INFO:teuthology.orchestra.run.gibba002.stdout: "down_at": 0, 2021-02-10T11:27:29.321 INFO:teuthology.orchestra.run.gibba002.stdout: "lost_at": 0, 2021-02-10T11:27:29.321 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addrs": { 2021-02-10T11:27:29.321 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:27:29.321 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:27:29.321 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:27:29.321 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6808", 2021-02-10T11:27:29.321 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36368 2021-02-10T11:27:29.321 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:27:29.321 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:27:29.321 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:27:29.321 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6811", 2021-02-10T11:27:29.321 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36368 2021-02-10T11:27:29.322 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:27:29.322 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:27:29.322 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:27:29.322 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addrs": { 2021-02-10T11:27:29.322 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:27:29.322 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:27:29.322 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:27:29.322 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6814", 2021-02-10T11:27:29.322 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36368 2021-02-10T11:27:29.322 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:27:29.322 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:27:29.322 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:27:29.322 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6818", 2021-02-10T11:27:29.323 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36368 2021-02-10T11:27:29.323 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:27:29.323 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:27:29.323 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:27:29.323 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addrs": { 2021-02-10T11:27:29.323 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:27:29.323 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:27:29.323 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:27:29.323 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6826", 2021-02-10T11:27:29.323 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36368 2021-02-10T11:27:29.323 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:27:29.323 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:27:29.323 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:27:29.324 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6828", 2021-02-10T11:27:29.324 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36368 2021-02-10T11:27:29.324 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:27:29.324 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:27:29.324 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:27:29.324 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addrs": { 2021-02-10T11:27:29.324 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:27:29.324 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:27:29.324 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:27:29.324 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6820", 2021-02-10T11:27:29.324 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36368 2021-02-10T11:27:29.324 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:27:29.324 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:27:29.325 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:27:29.325 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6824", 2021-02-10T11:27:29.325 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36368 2021-02-10T11:27:29.325 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:27:29.325 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:27:29.325 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:27:29.325 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addr": "172.21.2.113:6811/36368", 2021-02-10T11:27:29.325 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addr": "172.21.2.113:6818/36368", 2021-02-10T11:27:29.325 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addr": "172.21.2.113:6828/36368", 2021-02-10T11:27:29.325 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addr": "172.21.2.113:6824/36368", 2021-02-10T11:27:29.325 INFO:teuthology.orchestra.run.gibba002.stdout: "state": [ 2021-02-10T11:27:29.326 INFO:teuthology.orchestra.run.gibba002.stdout: "exists", 2021-02-10T11:27:29.326 INFO:teuthology.orchestra.run.gibba002.stdout: "up" 2021-02-10T11:27:29.326 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:27:29.326 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:27:29.326 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:27:29.326 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 5, 2021-02-10T11:27:29.326 INFO:teuthology.orchestra.run.gibba002.stdout: "uuid": "551f6007-daf8-4a98-8ee0-c2b16bcccb5e", 2021-02-10T11:27:29.326 INFO:teuthology.orchestra.run.gibba002.stdout: "up": 1, 2021-02-10T11:27:29.326 INFO:teuthology.orchestra.run.gibba002.stdout: "in": 1, 2021-02-10T11:27:29.326 INFO:teuthology.orchestra.run.gibba002.stdout: "weight": 1, 2021-02-10T11:27:29.327 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_affinity": 1, 2021-02-10T11:27:29.327 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_begin": 0, 2021-02-10T11:27:29.327 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_end": 0, 2021-02-10T11:27:29.327 INFO:teuthology.orchestra.run.gibba002.stdout: "up_from": 13, 2021-02-10T11:27:29.327 INFO:teuthology.orchestra.run.gibba002.stdout: "up_thru": 72, 2021-02-10T11:27:29.327 INFO:teuthology.orchestra.run.gibba002.stdout: "down_at": 0, 2021-02-10T11:27:29.327 INFO:teuthology.orchestra.run.gibba002.stdout: "lost_at": 0, 2021-02-10T11:27:29.327 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addrs": { 2021-02-10T11:27:29.327 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:27:29.327 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:27:29.327 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:27:29.327 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6802", 2021-02-10T11:27:29.328 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36366 2021-02-10T11:27:29.328 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:27:29.328 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:27:29.328 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:27:29.328 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6805", 2021-02-10T11:27:29.328 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36366 2021-02-10T11:27:29.328 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:27:29.328 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:27:29.328 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:27:29.328 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addrs": { 2021-02-10T11:27:29.328 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:27:29.328 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:27:29.328 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:27:29.328 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6807", 2021-02-10T11:27:29.329 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36366 2021-02-10T11:27:29.329 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:27:29.329 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:27:29.329 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:27:29.329 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6810", 2021-02-10T11:27:29.329 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36366 2021-02-10T11:27:29.329 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:27:29.329 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:27:29.329 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:27:29.329 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addrs": { 2021-02-10T11:27:29.329 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:27:29.329 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:27:29.329 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:27:29.330 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6819", 2021-02-10T11:27:29.330 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36366 2021-02-10T11:27:29.330 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:27:29.330 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:27:29.330 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:27:29.330 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6822", 2021-02-10T11:27:29.330 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36366 2021-02-10T11:27:29.330 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:27:29.330 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:27:29.330 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:27:29.330 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addrs": { 2021-02-10T11:27:29.330 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:27:29.330 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:27:29.330 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:27:29.331 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6812", 2021-02-10T11:27:29.331 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36366 2021-02-10T11:27:29.331 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:27:29.331 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:27:29.331 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:27:29.331 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6815", 2021-02-10T11:27:29.331 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36366 2021-02-10T11:27:29.331 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:27:29.331 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:27:29.331 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:27:29.331 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addr": "172.21.2.113:6805/36366", 2021-02-10T11:27:29.331 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addr": "172.21.2.113:6810/36366", 2021-02-10T11:27:29.331 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addr": "172.21.2.113:6822/36366", 2021-02-10T11:27:29.331 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addr": "172.21.2.113:6815/36366", 2021-02-10T11:27:29.332 INFO:teuthology.orchestra.run.gibba002.stdout: "state": [ 2021-02-10T11:27:29.332 INFO:teuthology.orchestra.run.gibba002.stdout: "exists", 2021-02-10T11:27:29.332 INFO:teuthology.orchestra.run.gibba002.stdout: "up" 2021-02-10T11:27:29.332 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:27:29.332 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:27:29.332 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:27:29.332 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 6, 2021-02-10T11:27:29.332 INFO:teuthology.orchestra.run.gibba002.stdout: "uuid": "bf67c040-0e50-48be-b102-c3943bc9ebef", 2021-02-10T11:27:29.332 INFO:teuthology.orchestra.run.gibba002.stdout: "up": 1, 2021-02-10T11:27:29.332 INFO:teuthology.orchestra.run.gibba002.stdout: "in": 1, 2021-02-10T11:27:29.332 INFO:teuthology.orchestra.run.gibba002.stdout: "weight": 1, 2021-02-10T11:27:29.332 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_affinity": 1, 2021-02-10T11:27:29.332 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_begin": 0, 2021-02-10T11:27:29.333 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_end": 0, 2021-02-10T11:27:29.333 INFO:teuthology.orchestra.run.gibba002.stdout: "up_from": 13, 2021-02-10T11:27:29.333 INFO:teuthology.orchestra.run.gibba002.stdout: "up_thru": 73, 2021-02-10T11:27:29.333 INFO:teuthology.orchestra.run.gibba002.stdout: "down_at": 0, 2021-02-10T11:27:29.333 INFO:teuthology.orchestra.run.gibba002.stdout: "lost_at": 0, 2021-02-10T11:27:29.333 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addrs": { 2021-02-10T11:27:29.333 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:27:29.333 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:27:29.333 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:27:29.333 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6800", 2021-02-10T11:27:29.333 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36367 2021-02-10T11:27:29.333 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:27:29.333 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:27:29.333 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:27:29.334 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6801", 2021-02-10T11:27:29.334 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36367 2021-02-10T11:27:29.334 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:27:29.334 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:27:29.334 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:27:29.334 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addrs": { 2021-02-10T11:27:29.334 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:27:29.334 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:27:29.334 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:27:29.334 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6803", 2021-02-10T11:27:29.334 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36367 2021-02-10T11:27:29.334 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:27:29.335 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:27:29.335 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:27:29.335 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6804", 2021-02-10T11:27:29.335 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36367 2021-02-10T11:27:29.335 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:27:29.335 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:27:29.335 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:27:29.335 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addrs": { 2021-02-10T11:27:29.335 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:27:29.335 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:27:29.335 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:27:29.335 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6813", 2021-02-10T11:27:29.335 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36367 2021-02-10T11:27:29.336 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:27:29.336 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:27:29.336 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:27:29.336 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6816", 2021-02-10T11:27:29.336 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36367 2021-02-10T11:27:29.336 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:27:29.336 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:27:29.336 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:27:29.336 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addrs": { 2021-02-10T11:27:29.337 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:27:29.337 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:27:29.337 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:27:29.337 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6806", 2021-02-10T11:27:29.337 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36367 2021-02-10T11:27:29.337 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:27:29.337 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:27:29.337 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:27:29.337 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6809", 2021-02-10T11:27:29.337 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36367 2021-02-10T11:27:29.337 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:27:29.337 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:27:29.338 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:27:29.338 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addr": "172.21.2.113:6801/36367", 2021-02-10T11:27:29.338 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addr": "172.21.2.113:6804/36367", 2021-02-10T11:27:29.338 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addr": "172.21.2.113:6816/36367", 2021-02-10T11:27:29.338 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addr": "172.21.2.113:6809/36367", 2021-02-10T11:27:29.338 INFO:teuthology.orchestra.run.gibba002.stdout: "state": [ 2021-02-10T11:27:29.338 INFO:teuthology.orchestra.run.gibba002.stdout: "exists", 2021-02-10T11:27:29.338 INFO:teuthology.orchestra.run.gibba002.stdout: "up" 2021-02-10T11:27:29.338 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:27:29.338 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:27:29.339 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:27:29.339 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 7, 2021-02-10T11:27:29.339 INFO:teuthology.orchestra.run.gibba002.stdout: "uuid": "f5461f80-018d-428f-be89-2eb2bcbe7bb6", 2021-02-10T11:27:29.339 INFO:teuthology.orchestra.run.gibba002.stdout: "up": 1, 2021-02-10T11:27:29.339 INFO:teuthology.orchestra.run.gibba002.stdout: "in": 1, 2021-02-10T11:27:29.339 INFO:teuthology.orchestra.run.gibba002.stdout: "weight": 1, 2021-02-10T11:27:29.339 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_affinity": 1, 2021-02-10T11:27:29.339 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_begin": 0, 2021-02-10T11:27:29.339 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_end": 0, 2021-02-10T11:27:29.339 INFO:teuthology.orchestra.run.gibba002.stdout: "up_from": 13, 2021-02-10T11:27:29.339 INFO:teuthology.orchestra.run.gibba002.stdout: "up_thru": 64, 2021-02-10T11:27:29.339 INFO:teuthology.orchestra.run.gibba002.stdout: "down_at": 0, 2021-02-10T11:27:29.339 INFO:teuthology.orchestra.run.gibba002.stdout: "lost_at": 0, 2021-02-10T11:27:29.340 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addrs": { 2021-02-10T11:27:29.340 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:27:29.340 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:27:29.340 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:27:29.340 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6817", 2021-02-10T11:27:29.340 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36365 2021-02-10T11:27:29.340 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:27:29.340 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:27:29.340 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:27:29.340 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6821", 2021-02-10T11:27:29.340 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36365 2021-02-10T11:27:29.340 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:27:29.340 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:27:29.340 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:27:29.341 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addrs": { 2021-02-10T11:27:29.341 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:27:29.341 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:27:29.341 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:27:29.341 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6823", 2021-02-10T11:27:29.341 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36365 2021-02-10T11:27:29.341 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:27:29.341 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:27:29.341 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:27:29.341 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6825", 2021-02-10T11:27:29.341 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36365 2021-02-10T11:27:29.341 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:27:29.341 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:27:29.342 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:27:29.342 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addrs": { 2021-02-10T11:27:29.342 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:27:29.342 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:27:29.342 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:27:29.342 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6830", 2021-02-10T11:27:29.342 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36365 2021-02-10T11:27:29.342 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:27:29.342 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:27:29.342 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:27:29.342 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6831", 2021-02-10T11:27:29.342 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36365 2021-02-10T11:27:29.342 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:27:29.343 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:27:29.343 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:27:29.343 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addrs": { 2021-02-10T11:27:29.343 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:27:29.343 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:27:29.343 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:27:29.343 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6827", 2021-02-10T11:27:29.343 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36365 2021-02-10T11:27:29.343 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:27:29.343 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:27:29.343 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:27:29.343 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6829", 2021-02-10T11:27:29.343 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36365 2021-02-10T11:27:29.343 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:27:29.344 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:27:29.344 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:27:29.344 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addr": "172.21.2.113:6821/36365", 2021-02-10T11:27:29.344 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addr": "172.21.2.113:6825/36365", 2021-02-10T11:27:29.344 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addr": "172.21.2.113:6831/36365", 2021-02-10T11:27:29.344 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addr": "172.21.2.113:6829/36365", 2021-02-10T11:27:29.344 INFO:teuthology.orchestra.run.gibba002.stdout: "state": [ 2021-02-10T11:27:29.344 INFO:teuthology.orchestra.run.gibba002.stdout: "exists", 2021-02-10T11:27:29.344 INFO:teuthology.orchestra.run.gibba002.stdout: "up" 2021-02-10T11:27:29.344 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:27:29.344 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:27:29.344 INFO:teuthology.orchestra.run.gibba002.stdout: ], 2021-02-10T11:27:29.344 INFO:teuthology.orchestra.run.gibba002.stdout: "osd_xinfo": [ 2021-02-10T11:27:29.345 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:27:29.345 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 0, 2021-02-10T11:27:29.345 INFO:teuthology.orchestra.run.gibba002.stdout: "down_stamp": "0.000000", 2021-02-10T11:27:29.345 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_probability": 0, 2021-02-10T11:27:29.345 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_interval": 0, 2021-02-10T11:27:29.345 INFO:teuthology.orchestra.run.gibba002.stdout: "features": 4611087854035861503, 2021-02-10T11:27:29.345 INFO:teuthology.orchestra.run.gibba002.stdout: "old_weight": 0 2021-02-10T11:27:29.345 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:27:29.345 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:27:29.345 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 1, 2021-02-10T11:27:29.345 INFO:teuthology.orchestra.run.gibba002.stdout: "down_stamp": "0.000000", 2021-02-10T11:27:29.345 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_probability": 0, 2021-02-10T11:27:29.345 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_interval": 0, 2021-02-10T11:27:29.345 INFO:teuthology.orchestra.run.gibba002.stdout: "features": 4611087854035861503, 2021-02-10T11:27:29.346 INFO:teuthology.orchestra.run.gibba002.stdout: "old_weight": 0 2021-02-10T11:27:29.346 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:27:29.346 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:27:29.346 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 2, 2021-02-10T11:27:29.346 INFO:teuthology.orchestra.run.gibba002.stdout: "down_stamp": "0.000000", 2021-02-10T11:27:29.346 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_probability": 0, 2021-02-10T11:27:29.346 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_interval": 0, 2021-02-10T11:27:29.346 INFO:teuthology.orchestra.run.gibba002.stdout: "features": 4611087854035861503, 2021-02-10T11:27:29.346 INFO:teuthology.orchestra.run.gibba002.stdout: "old_weight": 0 2021-02-10T11:27:29.346 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:27:29.346 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:27:29.346 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 3, 2021-02-10T11:27:29.346 INFO:teuthology.orchestra.run.gibba002.stdout: "down_stamp": "0.000000", 2021-02-10T11:27:29.347 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_probability": 0, 2021-02-10T11:27:29.347 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_interval": 0, 2021-02-10T11:27:29.347 INFO:teuthology.orchestra.run.gibba002.stdout: "features": 4611087854035861503, 2021-02-10T11:27:29.347 INFO:teuthology.orchestra.run.gibba002.stdout: "old_weight": 0 2021-02-10T11:27:29.347 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:27:29.347 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:27:29.347 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 4, 2021-02-10T11:27:29.347 INFO:teuthology.orchestra.run.gibba002.stdout: "down_stamp": "0.000000", 2021-02-10T11:27:29.347 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_probability": 0, 2021-02-10T11:27:29.347 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_interval": 0, 2021-02-10T11:27:29.347 INFO:teuthology.orchestra.run.gibba002.stdout: "features": 4611087854035861503, 2021-02-10T11:27:29.347 INFO:teuthology.orchestra.run.gibba002.stdout: "old_weight": 0 2021-02-10T11:27:29.347 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:27:29.348 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:27:29.348 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 5, 2021-02-10T11:27:29.348 INFO:teuthology.orchestra.run.gibba002.stdout: "down_stamp": "0.000000", 2021-02-10T11:27:29.348 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_probability": 0, 2021-02-10T11:27:29.348 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_interval": 0, 2021-02-10T11:27:29.348 INFO:teuthology.orchestra.run.gibba002.stdout: "features": 4611087854035861503, 2021-02-10T11:27:29.348 INFO:teuthology.orchestra.run.gibba002.stdout: "old_weight": 0 2021-02-10T11:27:29.348 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:27:29.348 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:27:29.348 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 6, 2021-02-10T11:27:29.348 INFO:teuthology.orchestra.run.gibba002.stdout: "down_stamp": "0.000000", 2021-02-10T11:27:29.348 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_probability": 0, 2021-02-10T11:27:29.348 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_interval": 0, 2021-02-10T11:27:29.349 INFO:teuthology.orchestra.run.gibba002.stdout: "features": 4611087854035861503, 2021-02-10T11:27:29.349 INFO:teuthology.orchestra.run.gibba002.stdout: "old_weight": 0 2021-02-10T11:27:29.349 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:27:29.349 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:27:29.349 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 7, 2021-02-10T11:27:29.349 INFO:teuthology.orchestra.run.gibba002.stdout: "down_stamp": "0.000000", 2021-02-10T11:27:29.349 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_probability": 0, 2021-02-10T11:27:29.349 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_interval": 0, 2021-02-10T11:27:29.349 INFO:teuthology.orchestra.run.gibba002.stdout: "features": 4611087854035861503, 2021-02-10T11:27:29.349 INFO:teuthology.orchestra.run.gibba002.stdout: "old_weight": 0 2021-02-10T11:27:29.349 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:27:29.349 INFO:teuthology.orchestra.run.gibba002.stdout: ], 2021-02-10T11:27:29.349 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_upmap": [], 2021-02-10T11:27:29.349 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_upmap_items": [], 2021-02-10T11:27:29.350 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_temp": [], 2021-02-10T11:27:29.350 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_temp": [], 2021-02-10T11:27:29.350 INFO:teuthology.orchestra.run.gibba002.stdout: "blacklist": {}, 2021-02-10T11:27:29.350 INFO:teuthology.orchestra.run.gibba002.stdout: "erasure_code_profiles": { 2021-02-10T11:27:29.350 INFO:teuthology.orchestra.run.gibba002.stdout: "default": { 2021-02-10T11:27:29.350 INFO:teuthology.orchestra.run.gibba002.stdout: "crush-failure-domain": "osd", 2021-02-10T11:27:29.350 INFO:teuthology.orchestra.run.gibba002.stdout: "k": "2", 2021-02-10T11:27:29.350 INFO:teuthology.orchestra.run.gibba002.stdout: "m": "1", 2021-02-10T11:27:29.350 INFO:teuthology.orchestra.run.gibba002.stdout: "plugin": "jerasure", 2021-02-10T11:27:29.350 INFO:teuthology.orchestra.run.gibba002.stdout: "ruleset-failure-domain": "osd", 2021-02-10T11:27:29.350 INFO:teuthology.orchestra.run.gibba002.stdout: "technique": "reed_sol_van" 2021-02-10T11:27:29.350 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:27:29.350 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:27:29.351 INFO:teuthology.orchestra.run.gibba002.stdout: "removed_snaps_queue": [], 2021-02-10T11:27:29.351 INFO:teuthology.orchestra.run.gibba002.stdout: "new_removed_snaps": [], 2021-02-10T11:27:29.351 INFO:teuthology.orchestra.run.gibba002.stdout: "new_purged_snaps": [ 2021-02-10T11:27:29.351 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:27:29.351 INFO:teuthology.orchestra.run.gibba002.stdout: "pool": 15, 2021-02-10T11:27:29.351 INFO:teuthology.orchestra.run.gibba002.stdout: "snaps": [ 2021-02-10T11:27:29.351 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:27:29.351 INFO:teuthology.orchestra.run.gibba002.stdout: "begin": 2, 2021-02-10T11:27:29.351 INFO:teuthology.orchestra.run.gibba002.stdout: "length": 2 2021-02-10T11:27:29.351 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:27:29.351 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:27:29.351 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:27:29.351 INFO:teuthology.orchestra.run.gibba002.stdout: ], 2021-02-10T11:27:29.351 INFO:teuthology.orchestra.run.gibba002.stdout: "crush_node_flags": {}, 2021-02-10T11:27:29.352 INFO:teuthology.orchestra.run.gibba002.stdout: "device_class_flags": {} 2021-02-10T11:27:29.352 INFO:teuthology.orchestra.run.gibba002.stdout:} 2021-02-10T11:27:29.353 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:29.277 7f91af87a700 1 -- 172.21.2.102:0/3025754720 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f9188018730 msgr2=0x7f918801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:29.353 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:29.277 7f91af87a700 1 --2- 172.21.2.102:0/3025754720 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f9188018730 0x7f918801abc0 crc :-1 s=READY pgs=785 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:29.353 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:29.278 7f91af87a700 1 -- 172.21.2.102:0/3025754720 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f91a80bb8e0 msgr2=0x7f91a80bbce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:29.353 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:29.278 7f91af87a700 1 --2- 172.21.2.102:0/3025754720 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f91a80bb8e0 0x7f91a80bbce0 secure :-1 s=READY pgs=1733 cs=0 l=1 rev1=1 rx=0x7f9198006f10 tx=0x7f919800b990).stop 2021-02-10T11:27:29.353 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:29.278 7f91af87a700 1 -- 172.21.2.102:0/3025754720 shutdown_connections 2021-02-10T11:27:29.353 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:29.278 7f91af87a700 1 --2- 172.21.2.102:0/3025754720 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f91a80bb8e0 0x7f91a80bbce0 unknown :-1 s=CLOSED pgs=1733 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:29.353 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:29.278 7f91af87a700 1 --2- 172.21.2.102:0/3025754720 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f9188018730 0x7f918801abc0 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:29.353 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:29.278 7f91af87a700 1 --2- 172.21.2.102:0/3025754720 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f91a8195070 0x7f91a8195470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:29.353 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:29.278 7f91af87a700 1 --2- 172.21.2.102:0/3025754720 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f91a81942b0 0x7f91a80b3550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:29.353 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:29.278 7f91af87a700 1 -- 172.21.2.102:0/3025754720 >> 172.21.2.102:0/3025754720 conn(0x7f91a80b3040 msgr2=0x7f91a80c3250 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:29.354 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:29.278 7f91af87a700 1 -- 172.21.2.102:0/3025754720 shutdown_connections 2021-02-10T11:27:29.354 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:29.278 7f91af87a700 1 -- 172.21.2.102:0/3025754720 wait complete. 2021-02-10T11:27:29.354 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:27:29.782 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:29.764 7fd9ab6bf700 1 Processor -- start 2021-02-10T11:27:29.782 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:29.764 7fd9ab6bf700 1 -- start start 2021-02-10T11:27:29.782 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:29.765 7fd9ab6bf700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fd9a41b0470 0x7fd9a41b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:29.782 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:29.765 7fd9ab6bf700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd9a40f20e0 0x7fd9a40f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:29.782 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:29.765 7fd9ab6bf700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9a40f0b50 con 0x7fd9a40ef610 2021-02-10T11:27:29.782 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:29.765 7fd9ab6bf700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9a40f0820 con 0x7fd9a41b0470 2021-02-10T11:27:29.782 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:29.765 7fd9ab6bf700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9a40f24e0 con 0x7fd9a40f20e0 2021-02-10T11:27:29.782 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:29.765 7fd9a94db700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fd9a41b0470 0x7fd9a41b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:29.783 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:29.765 7fd9a8cda700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fd9a40ef610 0x7fd9a40efa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:51744/0 (socket says 172.21.2.102:51744) 2021-02-10T11:27:29.783 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:29.765 7fd9a8cda700 1 -- 172.21.2.102:0/4004821604 learned_addr learned my addr 172.21.2.102:0/4004821604 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:27:29.783 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:29.765 7fd9a94db700 1 -- 172.21.2.102:0/4004821604 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd9a40f20e0 msgr2=0x7fd9a40f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:27:29.783 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:29.765 7fd9a94db700 1 --2- 172.21.2.102:0/4004821604 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd9a40f20e0 0x7fd9a40f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:29.783 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:29.765 7fd9a94db700 1 -- 172.21.2.102:0/4004821604 >> v1:172.21.2.102:6789/0 conn(0x7fd9a40ef610 legacy=0x7fd9a40efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:29.783 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:29.766 7fd9a94db700 1 -- 172.21.2.102:0/4004821604 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9a40e52e0 con 0x7fd9a41b0470 2021-02-10T11:27:29.783 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:29.766 7fd9a94db700 1 --2- 172.21.2.102:0/4004821604 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd9a41b0470 0x7fd9a41b0870 secure :-1 s=READY pgs=1735 cs=0 l=1 rev1=1 rx=0x7fd998006650 tx=0x7fd998009a80).ready entity=mon.0 client_cookie=d349c022d0a6dc56 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:29.783 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:29.766 7fd997fff700 1 -- 172.21.2.102:0/4004821604 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd99800c0c0 con 0x7fd9a41b0470 2021-02-10T11:27:29.783 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:29.766 7fd997fff700 1 -- 172.21.2.102:0/4004821604 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd99800c340 con 0x7fd9a41b0470 2021-02-10T11:27:29.783 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:29.766 7fd997fff700 1 -- 172.21.2.102:0/4004821604 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd99800c860 con 0x7fd9a41b0470 2021-02-10T11:27:29.783 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:29.766 7fd9ab6bf700 1 -- 172.21.2.102:0/4004821604 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd9a41b0470 msgr2=0x7fd9a41b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:29.783 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:29.766 7fd9ab6bf700 1 --2- 172.21.2.102:0/4004821604 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd9a41b0470 0x7fd9a41b0870 secure :-1 s=READY pgs=1735 cs=0 l=1 rev1=1 rx=0x7fd998006650 tx=0x7fd998009a80).stop 2021-02-10T11:27:29.783 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:29.767 7fd9ab6bf700 1 -- 172.21.2.102:0/4004821604 shutdown_connections 2021-02-10T11:27:29.784 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:29.767 7fd9ab6bf700 1 --2- 172.21.2.102:0/4004821604 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd9a40f20e0 0x7fd9a40f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:29.784 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:29.767 7fd9ab6bf700 1 --2- 172.21.2.102:0/4004821604 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd9a41b0470 0x7fd9a41b0870 unknown :-1 s=CLOSED pgs=1735 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:29.784 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:29.767 7fd9ab6bf700 1 -- 172.21.2.102:0/4004821604 >> 172.21.2.102:0/4004821604 conn(0x7fd9a40ce3e0 msgr2=0x7fd9a40e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:29.784 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:29.767 7fd9ab6bf700 1 -- 172.21.2.102:0/4004821604 shutdown_connections 2021-02-10T11:27:29.784 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:29.767 7fd9ab6bf700 1 -- 172.21.2.102:0/4004821604 wait complete. 2021-02-10T11:27:29.786 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:29.782 7fd9ab6bf700 1 Processor -- start 2021-02-10T11:27:29.786 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:29.782 7fd9ab6bf700 1 -- start start 2021-02-10T11:27:29.786 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:29.783 7fd9ab6bf700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd9a40b0970 0x7fd9a4092ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:29.786 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:29.783 7fd9ab6bf700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd9a40934f0 0x7fd9a4094dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:29.786 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:29.783 7fd9ab6bf700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd9a40938f0 0x7fd9a40953e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:29.786 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:29.783 7fd9ab6bf700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9a4195ad0 con 0x7fd9a40938f0 2021-02-10T11:27:29.786 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:29.783 7fd9ab6bf700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9a4095f40 con 0x7fd9a40b0970 2021-02-10T11:27:29.786 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:29.783 7fd9ab6bf700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9a4095a30 con 0x7fd9a40934f0 2021-02-10T11:27:29.786 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:29.783 7fd9a9cdc700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd9a40938f0 0x7fd9a40953e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:29.786 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:29.783 7fd9a9cdc700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd9a40938f0 0x7fd9a40953e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:59530/0 (socket says 172.21.2.102:59530) 2021-02-10T11:27:29.786 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:29.783 7fd9a9cdc700 1 -- 172.21.2.102:0/2388841973 learned_addr learned my addr 172.21.2.102:0/2388841973 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:27:29.786 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:29.783 7fd9a94db700 1 --2- 172.21.2.102:0/2388841973 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd9a40b0970 0x7fd9a4092ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:29.787 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:29.783 7fd9a8cda700 1 --2- 172.21.2.102:0/2388841973 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd9a40934f0 0x7fd9a4094dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:29.787 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:29.783 7fd9a9cdc700 1 -- 172.21.2.102:0/2388841973 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd9a40934f0 msgr2=0x7fd9a4094dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:29.787 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:29.783 7fd9a9cdc700 1 --2- 172.21.2.102:0/2388841973 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd9a40934f0 0x7fd9a4094dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:29.787 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:29.783 7fd9a9cdc700 1 -- 172.21.2.102:0/2388841973 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd9a40b0970 msgr2=0x7fd9a4092ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:29.787 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:29.783 7fd9a9cdc700 1 --2- 172.21.2.102:0/2388841973 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd9a40b0970 0x7fd9a4092ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:29.787 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:29.783 7fd9a9cdc700 1 -- 172.21.2.102:0/2388841973 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9a4095c50 con 0x7fd9a40938f0 2021-02-10T11:27:29.787 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:29.783 7fd9a9cdc700 1 --2- 172.21.2.102:0/2388841973 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd9a40938f0 0x7fd9a40953e0 secure :-1 s=READY pgs=1736 cs=0 l=1 rev1=1 rx=0x7fd99c009980 tx=0x7fd99c00bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:29.787 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:29.784 7fd995ffb700 1 -- 172.21.2.102:0/2388841973 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd99c005870 con 0x7fd9a40938f0 2021-02-10T11:27:29.787 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:29.784 7fd995ffb700 1 -- 172.21.2.102:0/2388841973 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd99c005b30 con 0x7fd9a40938f0 2021-02-10T11:27:29.787 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:29.784 7fd995ffb700 1 -- 172.21.2.102:0/2388841973 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd99c006030 con 0x7fd9a40938f0 2021-02-10T11:27:29.787 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:29.784 7fd9ab6bf700 1 -- 172.21.2.102:0/2388841973 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9a4095c50 con 0x7fd9a40938f0 2021-02-10T11:27:29.787 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:29.784 7fd9ab6bf700 1 -- 172.21.2.102:0/2388841973 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9a40967b0 con 0x7fd9a40938f0 2021-02-10T11:27:29.787 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:29.784 7fd995ffb700 1 -- 172.21.2.102:0/2388841973 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd99c005d10 con 0x7fd9a40938f0 2021-02-10T11:27:29.788 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:29.784 7fd995ffb700 1 --2- 172.21.2.102:0/2388841973 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd9840186c0 0x7fd98401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:29.788 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:29.784 7fd995ffb700 1 -- 172.21.2.102:0/2388841973 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 5617+0+0 (secure 0 0 0) 0x7fd99c032970 con 0x7fd9a40938f0 2021-02-10T11:27:29.788 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:29.785 7fd9a94db700 1 --2- 172.21.2.102:0/2388841973 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd9840186c0 0x7fd98401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:29.788 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:29.785 7fd9a94db700 1 --2- 172.21.2.102:0/2388841973 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd9840186c0 0x7fd98401abc0 crc :-1 s=READY pgs=786 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:29.815 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:29.815 7fd9ab6bf700 1 -- 172.21.2.102:0/2388841973 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9a40e52e0 con 0x7fd9a40938f0 2021-02-10T11:27:29.817 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:29.817 7fd995ffb700 1 -- 172.21.2.102:0/2388841973 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd99c023560 con 0x7fd9a40938f0 2021-02-10T11:27:29.945 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:29.945 7fd9ab6bf700 1 -- 172.21.2.102:0/2388841973 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd9a40e52e0 con 0x7fd9a40938f0 2021-02-10T11:27:29.946 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:29.945 7fd995ffb700 1 -- 172.21.2.102:0/2388841973 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 58 v58) v1 ==== 94+0+2098 (secure 0 0 0) 0x7fd99c006f20 con 0x7fd9a40938f0 2021-02-10T11:27:29.946 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:27:29.946 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":58,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8865,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.102:6835/980649634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6834","nonce":980649634},{"type":"v1","addr":"172.21.2.102:6835","nonce":980649634}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":53}],"filesystems":[{"mdsmap":{"epoch":57,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:26:47.543386","modified":"2021-02-10 11:26:49.552060","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6799},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6799":{"gid":6799,"name":"b","rank":0,"incarnation":54,"state":"up:active","state_seq":3,"addr":"172.21.2.113:6833/2002224946","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6832","nonce":2002224946},{"type":"v1","addr":"172.21.2.113:6833","nonce":2002224946}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2021-02-10T11:27:29.946 INFO:teuthology.orchestra.run.gibba002.stderr:dumped fsmap epoch 58 2021-02-10T11:27:29.947 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:29.947 7fd9ab6bf700 1 -- 172.21.2.102:0/2388841973 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd9840186c0 msgr2=0x7fd98401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:29.947 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:29.947 7fd9ab6bf700 1 --2- 172.21.2.102:0/2388841973 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd9840186c0 0x7fd98401abc0 crc :-1 s=READY pgs=786 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:29.947 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:29.947 7fd9ab6bf700 1 -- 172.21.2.102:0/2388841973 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd9a40938f0 msgr2=0x7fd9a40953e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:29.947 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:29.947 7fd9ab6bf700 1 --2- 172.21.2.102:0/2388841973 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd9a40938f0 0x7fd9a40953e0 secure :-1 s=READY pgs=1736 cs=0 l=1 rev1=1 rx=0x7fd99c009980 tx=0x7fd99c00bcd0).stop 2021-02-10T11:27:29.948 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:29.948 7fd9ab6bf700 1 -- 172.21.2.102:0/2388841973 shutdown_connections 2021-02-10T11:27:29.948 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:29.948 7fd9ab6bf700 1 --2- 172.21.2.102:0/2388841973 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd9a40938f0 0x7fd9a40953e0 unknown :-1 s=CLOSED pgs=1736 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:29.948 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:29.948 7fd9ab6bf700 1 --2- 172.21.2.102:0/2388841973 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd9a40934f0 0x7fd9a4094dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:29.948 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:29.948 7fd9ab6bf700 1 --2- 172.21.2.102:0/2388841973 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd9840186c0 0x7fd98401abc0 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:29.948 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:29.948 7fd9ab6bf700 1 --2- 172.21.2.102:0/2388841973 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd9a40b0970 0x7fd9a4092ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:29.948 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:29.948 7fd9ab6bf700 1 -- 172.21.2.102:0/2388841973 >> 172.21.2.102:0/2388841973 conn(0x7fd9a419f7a0 msgr2=0x7fd9a40940e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:29.949 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:29.948 7fd9ab6bf700 1 -- 172.21.2.102:0/2388841973 shutdown_connections 2021-02-10T11:27:29.949 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:29.948 7fd9ab6bf700 1 -- 172.21.2.102:0/2388841973 wait complete. 2021-02-10T11:27:29.956 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs fail cephfs 2021-02-10T11:27:30.384 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:30.367 7fb126a5f700 1 Processor -- start 2021-02-10T11:27:30.384 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:30.367 7fb126a5f700 1 -- start start 2021-02-10T11:27:30.384 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:30.368 7fb126a5f700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fb1200d1c60 0x7fb1200f2090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:30.384 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:30.368 7fb126a5f700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1200f0b00 con 0x7fb1200d1860 2021-02-10T11:27:30.384 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:30.368 7fb126a5f700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1201b0bd0 con 0x7fb1200d0330 2021-02-10T11:27:30.384 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:30.368 7fb126a5f700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1200e5ec0 con 0x7fb1200d1c60 2021-02-10T11:27:30.384 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:30.368 7fb11ffff700 1 --1- >> v1:172.21.2.113:6789/0 conn(0x7fb1200d0330 0x7fb1200ef630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.113:6789/0 says I am v1:172.21.2.102:59082/0 (socket says 172.21.2.102:59082) 2021-02-10T11:27:30.385 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:30.368 7fb11ffff700 1 -- 172.21.2.102:0/4215817600 learned_addr learned my addr 172.21.2.102:0/4215817600 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:27:30.385 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:30.368 7fb12507c700 1 --2- 172.21.2.102:0/4215817600 >> v2:172.21.2.102:3300/0 conn(0x7fb1200d1c60 0x7fb1200f2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:30.385 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:30.369 7fb11f7fe700 1 -- 172.21.2.102:0/4215817600 <== mon.1 v1:172.21.2.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 743040918 0 0) 0x7fb110001510 con 0x7fb1200d0330 2021-02-10T11:27:30.385 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:30.369 7fb11f7fe700 1 -- 172.21.2.102:0/4215817600 --> v1:172.21.2.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb10c002fe0 con 0x7fb1200d0330 2021-02-10T11:27:30.385 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:30.369 7fb11f7fe700 1 -- 172.21.2.102:0/4215817600 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1668353192 0 0) 0x7fb108000ce0 con 0x7fb1200d1860 2021-02-10T11:27:30.385 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:30.369 7fb11f7fe700 1 -- 172.21.2.102:0/4215817600 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb10c0042c0 con 0x7fb1200d1860 2021-02-10T11:27:30.385 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:30.369 7fb12507c700 1 -- 172.21.2.102:0/4215817600 >> v1:172.21.2.113:6789/0 conn(0x7fb1200d0330 legacy=0x7fb1200ef630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:30.385 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:30.369 7fb12507c700 1 -- 172.21.2.102:0/4215817600 >> v1:172.21.2.102:6789/0 conn(0x7fb1200d1860 legacy=0x7fb1201b03e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:30.385 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:30.369 7fb12507c700 1 -- 172.21.2.102:0/4215817600 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1200efe20 con 0x7fb1200d1c60 2021-02-10T11:27:30.385 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:30.369 7fb12507c700 1 --2- 172.21.2.102:0/4215817600 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb1200d1c60 0x7fb1200f2090 secure :-1 s=READY pgs=1738 cs=0 l=1 rev1=1 rx=0x7fb118006ab0 tx=0x7fb118009b40).ready entity=mon.0 client_cookie=a9c4baf5be8c66cb server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:30.385 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:30.369 7fb11f7fe700 1 -- 172.21.2.102:0/4215817600 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb1180022e0 con 0x7fb1200d1c60 2021-02-10T11:27:30.386 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:30.369 7fb11f7fe700 1 -- 172.21.2.102:0/4215817600 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb118002560 con 0x7fb1200d1c60 2021-02-10T11:27:30.386 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:30.369 7fb11f7fe700 1 -- 172.21.2.102:0/4215817600 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb1180029f0 con 0x7fb1200d1c60 2021-02-10T11:27:30.386 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:30.369 7fb126a5f700 1 -- 172.21.2.102:0/4215817600 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb1200d1c60 msgr2=0x7fb1200f2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:30.386 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:30.369 7fb126a5f700 1 --2- 172.21.2.102:0/4215817600 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb1200d1c60 0x7fb1200f2090 secure :-1 s=READY pgs=1738 cs=0 l=1 rev1=1 rx=0x7fb118006ab0 tx=0x7fb118009b40).stop 2021-02-10T11:27:30.386 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:30.370 7fb126a5f700 1 -- 172.21.2.102:0/4215817600 shutdown_connections 2021-02-10T11:27:30.386 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:30.370 7fb126a5f700 1 --2- 172.21.2.102:0/4215817600 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb1200d1c60 0x7fb1200f2090 unknown :-1 s=CLOSED pgs=1738 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:30.386 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:30.370 7fb126a5f700 1 -- 172.21.2.102:0/4215817600 >> 172.21.2.102:0/4215817600 conn(0x7fb1200ce440 msgr2=0x7fb1200d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:30.386 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:30.370 7fb126a5f700 1 -- 172.21.2.102:0/4215817600 shutdown_connections 2021-02-10T11:27:30.386 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:30.370 7fb126a5f700 1 -- 172.21.2.102:0/4215817600 wait complete. 2021-02-10T11:27:30.388 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:30.384 7fb126a5f700 1 Processor -- start 2021-02-10T11:27:30.388 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:30.384 7fb126a5f700 1 -- start start 2021-02-10T11:27:30.388 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:30.385 7fb126a5f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb120187e40 0x7fb12009f570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:30.388 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:30.385 7fb126a5f700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb120188240 0x7fb12009fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:30.388 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:30.385 7fb126a5f700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb12009e9b0 0x7fb1200b78e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:30.388 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:30.385 7fb126a5f700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb12009edb0 con 0x7fb120187e40 2021-02-10T11:27:30.389 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:30.385 7fb126a5f700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb12009eff0 con 0x7fb12009e9b0 2021-02-10T11:27:30.389 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:30.385 7fb126a5f700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1200c3730 con 0x7fb120188240 2021-02-10T11:27:30.389 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:30.385 7fb12487b700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb120187e40 0x7fb12009f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:30.389 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:30.385 7fb12487b700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb120187e40 0x7fb12009f570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:59538/0 (socket says 172.21.2.102:59538) 2021-02-10T11:27:30.389 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:30.385 7fb12487b700 1 -- 172.21.2.102:0/4120267643 learned_addr learned my addr 172.21.2.102:0/4120267643 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:27:30.389 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:30.385 7fb12487b700 1 -- 172.21.2.102:0/4120267643 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb120188240 msgr2=0x7fb12009fb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:30.389 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:30.385 7fb12487b700 1 --2- 172.21.2.102:0/4120267643 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb120188240 0x7fb12009fb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:30.389 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:30.385 7fb12487b700 1 -- 172.21.2.102:0/4120267643 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb12009e9b0 msgr2=0x7fb1200b78e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:30.389 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:30.385 7fb12487b700 1 --2- 172.21.2.102:0/4120267643 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb12009e9b0 0x7fb1200b78e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:30.389 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:30.385 7fb12487b700 1 -- 172.21.2.102:0/4120267643 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1200b7f00 con 0x7fb120187e40 2021-02-10T11:27:30.389 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:30.385 7fb12487b700 1 --2- 172.21.2.102:0/4120267643 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb120187e40 0x7fb12009f570 secure :-1 s=READY pgs=1739 cs=0 l=1 rev1=1 rx=0x7fb108006f20 tx=0x7fb10800b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:30.389 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:30.386 7fb11d7fa700 1 -- 172.21.2.102:0/4120267643 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb10800bd80 con 0x7fb120187e40 2021-02-10T11:27:30.390 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:30.386 7fb126a5f700 1 -- 172.21.2.102:0/4120267643 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1200b7f00 con 0x7fb120187e40 2021-02-10T11:27:30.390 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:30.386 7fb11d7fa700 1 -- 172.21.2.102:0/4120267643 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb10800bd80 con 0x7fb120187e40 2021-02-10T11:27:30.390 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:30.386 7fb126a5f700 1 -- 172.21.2.102:0/4120267643 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1201a76d0 con 0x7fb120187e40 2021-02-10T11:27:30.390 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:30.386 7fb11d7fa700 1 -- 172.21.2.102:0/4120267643 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb108001100 con 0x7fb120187e40 2021-02-10T11:27:30.390 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:30.386 7fb11d7fa700 1 -- 172.21.2.102:0/4120267643 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fb108000ce0 con 0x7fb120187e40 2021-02-10T11:27:30.390 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:30.386 7fb11d7fa700 1 --2- 172.21.2.102:0/4120267643 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb1040186c0 0x7fb10401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:30.390 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:30.386 7fb11d7fa700 1 -- 172.21.2.102:0/4120267643 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 5617+0+0 (secure 0 0 0) 0x7fb1080325f0 con 0x7fb120187e40 2021-02-10T11:27:30.390 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:30.386 7fb11ffff700 1 --2- 172.21.2.102:0/4120267643 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb1040186c0 0x7fb10401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:30.390 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:30.387 7fb11ffff700 1 --2- 172.21.2.102:0/4120267643 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb1040186c0 0x7fb10401abc0 crc :-1 s=READY pgs=787 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:30.418 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:30.418 7fb126a5f700 1 -- 172.21.2.102:0/4120267643 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb12009edb0 con 0x7fb120187e40 2021-02-10T11:27:30.420 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:30.420 7fb11d7fa700 1 -- 172.21.2.102:0/4120267643 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fb108030d70 con 0x7fb120187e40 2021-02-10T11:27:30.548 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:30.548 7fb126a5f700 1 -- 172.21.2.102:0/4120267643 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fb12009edb0 con 0x7fb120187e40 2021-02-10T11:27:31.336 INFO:tasks.ceph.mon.a.gibba002.stderr:2021-02-10 11:27:31.336 7f8dd2755700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-02-10T11:27:31.342 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:31.342 7fb11d7fa700 1 -- 172.21.2.102:0/4120267643 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v59) v1 ==== 161+0+0 (secure 0 0 0) 0x7fb10801f730 con 0x7fb120187e40 2021-02-10T11:27:31.368 INFO:teuthology.orchestra.run.gibba002.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2021-02-10T11:27:31.370 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:31.369 7fb126a5f700 1 -- 172.21.2.102:0/4120267643 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb1040186c0 msgr2=0x7fb10401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:31.370 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:31.369 7fb126a5f700 1 --2- 172.21.2.102:0/4120267643 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb1040186c0 0x7fb10401abc0 crc :-1 s=READY pgs=787 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:31.370 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:31.369 7fb126a5f700 1 -- 172.21.2.102:0/4120267643 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb120187e40 msgr2=0x7fb12009f570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:31.370 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:31.369 7fb126a5f700 1 --2- 172.21.2.102:0/4120267643 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb120187e40 0x7fb12009f570 secure :-1 s=READY pgs=1739 cs=0 l=1 rev1=1 rx=0x7fb108006f20 tx=0x7fb10800b990).stop 2021-02-10T11:27:31.370 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:31.370 7fb126a5f700 1 -- 172.21.2.102:0/4120267643 shutdown_connections 2021-02-10T11:27:31.370 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:31.370 7fb126a5f700 1 --2- 172.21.2.102:0/4120267643 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb1040186c0 0x7fb10401abc0 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:31.370 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:31.370 7fb126a5f700 1 --2- 172.21.2.102:0/4120267643 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb12009e9b0 0x7fb1200b78e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:31.370 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:31.370 7fb126a5f700 1 --2- 172.21.2.102:0/4120267643 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb120188240 0x7fb12009fb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:31.370 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:31.370 7fb126a5f700 1 --2- 172.21.2.102:0/4120267643 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb120187e40 0x7fb12009f570 unknown :-1 s=CLOSED pgs=1739 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:31.370 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:31.370 7fb126a5f700 1 -- 172.21.2.102:0/4120267643 >> 172.21.2.102:0/4120267643 conn(0x7fb1200a6b00 msgr2=0x7fb120095700 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:31.371 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:31.370 7fb126a5f700 1 -- 172.21.2.102:0/4120267643 shutdown_connections 2021-02-10T11:27:31.371 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:31.370 7fb126a5f700 1 -- 172.21.2.102:0/4120267643 wait complete. 2021-02-10T11:27:31.378 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:27:31.393 INFO:tasks.ceph.mds.b.gibba013.stdout:starting mds.b at 2021-02-10T11:27:31.808 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:31.790 7f21eee95700 1 Processor -- start 2021-02-10T11:27:32.043 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:31.790 7f21eee95700 1 -- start start 2021-02-10T11:27:32.043 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:31.790 7f21eee95700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f21e80d1870 0x7f21e81b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:32.043 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:31.790 7f21eee95700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21e80f0b10 con 0x7f21e81b0a10 2021-02-10T11:27:32.043 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:31.790 7f21eee95700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21e80d0310 con 0x7f21e80f0650 2021-02-10T11:27:32.043 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:31.790 7f21eee95700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21e80e5ed0 con 0x7f21e80d1870 2021-02-10T11:27:32.043 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:31.790 7f21e7fff700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f21e81b0a10 0x7f21e80cf330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:51770/0 (socket says 172.21.2.102:51770) 2021-02-10T11:27:32.043 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:31.791 7f21e7fff700 1 -- 172.21.2.102:0/3332447447 learned_addr learned my addr 172.21.2.102:0/3332447447 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:27:32.043 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:31.791 7f21e77fe700 1 -- 172.21.2.102:0/3332447447 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3166023379 0 0) 0x7f21d8001180 con 0x7f21e81b0a10 2021-02-10T11:27:32.043 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:31.791 7f21e77fe700 1 -- 172.21.2.102:0/3332447447 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f21d4002fe0 con 0x7f21e81b0a10 2021-02-10T11:27:32.043 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:31.791 7f21eccb1700 1 --2- 172.21.2.102:0/3332447447 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f21e80d1870 0x7f21e81b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:32.043 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:31.791 7f21e77fe700 1 -- 172.21.2.102:0/3332447447 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4024642161 0 0) 0x7f21d8000ce0 con 0x7f21e81b0a10 2021-02-10T11:27:32.044 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:31.791 7f21e77fe700 1 -- 172.21.2.102:0/3332447447 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f21e80d1870 msgr2=0x7f21e81b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:27:32.044 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:31.791 7f21e77fe700 1 --2- 172.21.2.102:0/3332447447 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f21e80d1870 0x7f21e81b03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:32.044 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:31.791 7f21e77fe700 1 -- 172.21.2.102:0/3332447447 >> v1:172.21.2.113:6789/0 conn(0x7f21e80f0650 legacy=0x7f21e80cfb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:32.044 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:31.791 7f21e77fe700 1 -- 172.21.2.102:0/3332447447 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21e80d0530 con 0x7f21e81b0a10 2021-02-10T11:27:32.044 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:31.791 7f21e77fe700 1 -- 172.21.2.102:0/3332447447 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7f21d80017f0 con 0x7f21e81b0a10 2021-02-10T11:27:32.044 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:31.791 7f21e77fe700 1 -- 172.21.2.102:0/3332447447 >> v1:172.21.2.102:6789/0 conn(0x7f21e81b0a10 legacy=0x7f21e80cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:32.044 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:31.791 7f21e77fe700 1 --2- 172.21.2.102:0/3332447447 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f21d4003ac0 0x7f21d4005f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:32.044 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:31.791 7f21e77fe700 1 --2- 172.21.2.102:0/3332447447 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f21d4006790 0x7f21d4008c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:32.044 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:31.791 7f21e77fe700 1 --2- 172.21.2.102:0/3332447447 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f21d4009590 0x7f21d400ba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:32.044 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:31.791 7f21e77fe700 1 -- 172.21.2.102:0/3332447447 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21d400c2d0 con 0x7f21d4003ac0 2021-02-10T11:27:32.044 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:31.791 7f21e77fe700 1 -- 172.21.2.102:0/3332447447 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21d400c4f0 con 0x7f21d4006790 2021-02-10T11:27:32.044 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:31.791 7f21e77fe700 1 -- 172.21.2.102:0/3332447447 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21d400c710 con 0x7f21d4009590 2021-02-10T11:27:32.044 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:31.791 7f21eccb1700 1 --2- 172.21.2.102:0/3332447447 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f21d4003ac0 0x7f21d4005f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:32.045 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:31.792 7f21e7fff700 1 --2- 172.21.2.102:0/3332447447 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f21d4006790 0x7f21d4008c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:32.045 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:31.792 7f21ed4b2700 1 --2- 172.21.2.102:0/3332447447 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f21d4009590 0x7f21d400ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:32.045 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:31.792 7f21eccb1700 1 -- 172.21.2.102:0/3332447447 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f21d4009590 msgr2=0x7f21d400ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:27:32.045 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:31.792 7f21eccb1700 1 --2- 172.21.2.102:0/3332447447 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f21d4009590 0x7f21d400ba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:32.045 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:31.792 7f21eccb1700 1 -- 172.21.2.102:0/3332447447 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f21d4006790 msgr2=0x7f21d4008c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:27:32.045 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:31.792 7f21eccb1700 1 --2- 172.21.2.102:0/3332447447 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f21d4006790 0x7f21d4008c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:32.045 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:31.792 7f21eccb1700 1 -- 172.21.2.102:0/3332447447 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21d400c9b0 con 0x7f21d4003ac0 2021-02-10T11:27:32.045 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:31.792 7f21eccb1700 1 --2- 172.21.2.102:0/3332447447 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f21d4003ac0 0x7f21d4005f10 secure :-1 s=READY pgs=1744 cs=0 l=1 rev1=1 rx=0x7f21d004d720 tx=0x7f21d004dac0).ready entity=mon.0 client_cookie=741bf92d609a0eff server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:32.045 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:31.792 7f21e77fe700 1 -- 172.21.2.102:0/3332447447 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f21d00066d0 con 0x7f21d4003ac0 2021-02-10T11:27:32.045 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:31.792 7f21e77fe700 1 -- 172.21.2.102:0/3332447447 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21d0002030 con 0x7f21d4003ac0 2021-02-10T11:27:32.045 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:31.792 7f21e77fe700 1 -- 172.21.2.102:0/3332447447 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f21d0002480 con 0x7f21d4003ac0 2021-02-10T11:27:32.045 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:31.792 7f21eee95700 1 -- 172.21.2.102:0/3332447447 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f21d4003ac0 msgr2=0x7f21d4005f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:32.045 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:31.792 7f21eee95700 1 --2- 172.21.2.102:0/3332447447 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f21d4003ac0 0x7f21d4005f10 secure :-1 s=READY pgs=1744 cs=0 l=1 rev1=1 rx=0x7f21d004d720 tx=0x7f21d004dac0).stop 2021-02-10T11:27:32.046 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:31.792 7f21ed4b2700 1 -- 172.21.2.102:0/3332447447 reap_dead start 2021-02-10T11:27:32.046 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:31.793 7f21eee95700 1 -- 172.21.2.102:0/3332447447 shutdown_connections 2021-02-10T11:27:32.046 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:31.793 7f21eee95700 1 -- 172.21.2.102:0/3332447447 >> 172.21.2.102:0/3332447447 conn(0x7f21e80ce450 msgr2=0x7f21e80d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:32.046 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:31.793 7f21eee95700 1 -- 172.21.2.102:0/3332447447 shutdown_connections 2021-02-10T11:27:32.046 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:31.793 7f21eee95700 1 -- 172.21.2.102:0/3332447447 wait complete. 2021-02-10T11:27:32.046 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:31.808 7f21eee95700 1 Processor -- start 2021-02-10T11:27:32.046 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:31.808 7f21eee95700 1 -- start start 2021-02-10T11:27:32.046 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:31.808 7f21eee95700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f21e80a2560 0x7f21e80b4fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:32.046 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:31.808 7f21eee95700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f21e80b55e0 0x7f21e80b59e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:32.046 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:31.808 7f21eee95700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f21e80ee200 0x7f21e80ee600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:32.046 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:31.808 7f21eee95700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21e818fda0 con 0x7f21e80ee200 2021-02-10T11:27:32.046 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:31.808 7f21eee95700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21e80a20c0 con 0x7f21e80a2560 2021-02-10T11:27:32.046 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:31.808 7f21eee95700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21e80b6000 con 0x7f21e80b55e0 2021-02-10T11:27:32.047 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:31.808 7f21ed4b2700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f21e80ee200 0x7f21e80ee600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:32.047 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:31.808 7f21ed4b2700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f21e80ee200 0x7f21e80ee600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:59558/0 (socket says 172.21.2.102:59558) 2021-02-10T11:27:32.047 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:31.808 7f21ed4b2700 1 -- 172.21.2.102:0/1238652183 learned_addr learned my addr 172.21.2.102:0/1238652183 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:27:32.047 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:31.809 7f21e7fff700 1 --2- 172.21.2.102:0/1238652183 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f21e80b55e0 0x7f21e80b59e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:32.047 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:31.809 7f21eccb1700 1 --2- 172.21.2.102:0/1238652183 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f21e80a2560 0x7f21e80b4fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:32.049 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:31.809 7f21ed4b2700 1 -- 172.21.2.102:0/1238652183 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f21e80b55e0 msgr2=0x7f21e80b59e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:32.049 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:31.809 7f21ed4b2700 1 --2- 172.21.2.102:0/1238652183 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f21e80b55e0 0x7f21e80b59e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:32.049 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:31.809 7f21ed4b2700 1 -- 172.21.2.102:0/1238652183 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f21e80a2560 msgr2=0x7f21e80b4fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:32.049 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:31.809 7f21ed4b2700 1 --2- 172.21.2.102:0/1238652183 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f21e80a2560 0x7f21e80b4fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:32.049 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:31.809 7f21ed4b2700 1 -- 172.21.2.102:0/1238652183 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21e80eec20 con 0x7f21e80ee200 2021-02-10T11:27:32.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:31.809 7f21ed4b2700 1 --2- 172.21.2.102:0/1238652183 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f21e80ee200 0x7f21e80ee600 secure :-1 s=READY pgs=1745 cs=0 l=1 rev1=1 rx=0x7f21e0006900 tx=0x7f21e000bd20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:32.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:31.809 7f21e57fa700 1 -- 172.21.2.102:0/1238652183 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f21e0006240 con 0x7f21e80ee200 2021-02-10T11:27:32.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:31.809 7f21e57fa700 1 -- 172.21.2.102:0/1238652183 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e000be60 con 0x7f21e80ee200 2021-02-10T11:27:32.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:31.809 7f21e57fa700 1 -- 172.21.2.102:0/1238652183 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f21e00064c0 con 0x7f21e80ee200 2021-02-10T11:27:32.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:31.809 7f21eee95700 1 -- 172.21.2.102:0/1238652183 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21e80eec20 con 0x7f21e80ee200 2021-02-10T11:27:32.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:31.809 7f21eee95700 1 -- 172.21.2.102:0/1238652183 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21e80ef1b0 con 0x7f21e80ee200 2021-02-10T11:27:32.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:31.810 7f21e57fa700 1 -- 172.21.2.102:0/1238652183 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f21e0006a30 con 0x7f21e80ee200 2021-02-10T11:27:32.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:31.810 7f21e57fa700 1 --2- 172.21.2.102:0/1238652183 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f21cc0186d0 0x7f21cc01abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:32.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:31.810 7f21e57fa700 1 -- 172.21.2.102:0/1238652183 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 5675+0+0 (secure 0 0 0) 0x7f21e0032a20 con 0x7f21e80ee200 2021-02-10T11:27:32.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:31.810 7f21eccb1700 1 --2- 172.21.2.102:0/1238652183 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f21cc0186d0 0x7f21cc01abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:32.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:31.810 7f21eccb1700 1 --2- 172.21.2.102:0/1238652183 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f21cc0186d0 0x7f21cc01abd0 crc :-1 s=READY pgs=788 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:32.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:31.840 7f21eee95700 1 -- 172.21.2.102:0/1238652183 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21e80ebcd0 con 0x7f21e80ee200 2021-02-10T11:27:32.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:31.843 7f21e57fa700 1 -- 172.21.2.102:0/1238652183 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f21e00056a0 con 0x7f21e80ee200 2021-02-10T11:27:32.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:31.972 7f21eee95700 1 -- 172.21.2.102:0/1238652183 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f21e80f0b10 con 0x7f21e80ee200 2021-02-10T11:27:32.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:31.972 7f21e57fa700 1 -- 172.21.2.102:0/1238652183 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 59 v59) v1 ==== 94+0+1753 (secure 0 0 0) 0x7f21e00069f0 con 0x7f21e80ee200 2021-02-10T11:27:32.051 INFO:teuthology.orchestra.run.gibba002.stderr:dumped fsmap epoch 59 2021-02-10T11:27:32.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:31.974 7f21eee95700 1 -- 172.21.2.102:0/1238652183 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f21cc0186d0 msgr2=0x7f21cc01abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:32.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:31.974 7f21eee95700 1 --2- 172.21.2.102:0/1238652183 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f21cc0186d0 0x7f21cc01abd0 crc :-1 s=READY pgs=788 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:32.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:31.974 7f21eee95700 1 -- 172.21.2.102:0/1238652183 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f21e80ee200 msgr2=0x7f21e80ee600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:32.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:31.974 7f21eee95700 1 --2- 172.21.2.102:0/1238652183 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f21e80ee200 0x7f21e80ee600 secure :-1 s=READY pgs=1745 cs=0 l=1 rev1=1 rx=0x7f21e0006900 tx=0x7f21e000bd20).stop 2021-02-10T11:27:32.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:31.974 7f21eee95700 1 -- 172.21.2.102:0/1238652183 shutdown_connections 2021-02-10T11:27:32.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:31.974 7f21eee95700 1 --2- 172.21.2.102:0/1238652183 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f21e80ee200 0x7f21e80ee600 unknown :-1 s=CLOSED pgs=1745 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:32.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:31.974 7f21eee95700 1 --2- 172.21.2.102:0/1238652183 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f21e80b55e0 0x7f21e80b59e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:32.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:31.974 7f21eee95700 1 --2- 172.21.2.102:0/1238652183 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f21cc0186d0 0x7f21cc01abd0 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:32.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:31.974 7f21eee95700 1 --2- 172.21.2.102:0/1238652183 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f21e80a2560 0x7f21e80b4fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:32.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:31.974 7f21eee95700 1 -- 172.21.2.102:0/1238652183 >> 172.21.2.102:0/1238652183 conn(0x7f21e80ac990 msgr2=0x7f21e8091b30 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:32.052 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:31.974 7f21eee95700 1 -- 172.21.2.102:0/1238652183 shutdown_connections 2021-02-10T11:27:32.052 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:31.975 7f21eee95700 1 -- 172.21.2.102:0/1238652183 wait complete. 2021-02-10T11:27:32.052 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:27:32.052 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":59,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8865,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.102:6835/980649634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6834","nonce":980649634},{"type":"v1","addr":"172.21.2.102:6835","nonce":980649634}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":53}],"filesystems":[{"mdsmap":{"epoch":59,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:26:47.543386","modified":"2021-02-10 11:27:31.337398","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":78,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2021-02-10T11:27:32.052 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2021-02-10T11:27:32.485 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:32.468 7fa6278a1700 1 Processor -- start 2021-02-10T11:27:32.486 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:32.468 7fa6278a1700 1 -- start start 2021-02-10T11:27:32.486 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:32.469 7fa6278a1700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa6200d0340 0x7fa6200ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:32.486 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:32.469 7fa6278a1700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fa6200d1c70 0x7fa6200efc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:32.486 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:32.469 7fa6278a1700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6200f0b10 con 0x7fa6200d1870 2021-02-10T11:27:32.486 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:32.469 7fa6278a1700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6200e5ed0 con 0x7fa6200d1c70 2021-02-10T11:27:32.486 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:32.469 7fa6278a1700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6201b0be0 con 0x7fa6200d0340 2021-02-10T11:27:32.486 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:32.469 7fa6256bd700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fa6200d1870 0x7fa6201b03f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:51790/0 (socket says 172.21.2.102:51790) 2021-02-10T11:27:32.486 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:32.469 7fa6256bd700 1 -- 172.21.2.102:0/321011995 learned_addr learned my addr 172.21.2.102:0/321011995 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:27:32.486 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:32.469 7fa625ebe700 1 --2- 172.21.2.102:0/321011995 >> v2:172.21.2.113:3300/0 conn(0x7fa6200d1c70 0x7fa6200efc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:32.486 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:32.469 7fa624ebc700 1 --2- 172.21.2.102:0/321011995 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa6200d0340 0x7fa6200ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:32.487 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:32.469 7fa613fff700 1 -- 172.21.2.102:0/321011995 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1813400443 0 0) 0x7fa614000ce0 con 0x7fa6200d1870 2021-02-10T11:27:32.487 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:32.469 7fa613fff700 1 -- 172.21.2.102:0/321011995 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa60c002fe0 con 0x7fa6200d1870 2021-02-10T11:27:32.487 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:32.469 7fa625ebe700 1 -- 172.21.2.102:0/321011995 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa6200d0340 msgr2=0x7fa6200ef640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:27:32.487 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:32.469 7fa625ebe700 1 --2- 172.21.2.102:0/321011995 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa6200d0340 0x7fa6200ef640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:32.487 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:32.469 7fa625ebe700 1 -- 172.21.2.102:0/321011995 >> v1:172.21.2.102:6789/0 conn(0x7fa6200d1870 legacy=0x7fa6201b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:32.487 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:32.469 7fa625ebe700 1 -- 172.21.2.102:0/321011995 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6200f20a0 con 0x7fa6200d1c70 2021-02-10T11:27:32.487 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:32.469 7fa624ebc700 1 --2- 172.21.2.102:0/321011995 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa6200d0340 0x7fa6200ef640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:27:32.487 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:32.470 7fa625ebe700 1 --2- 172.21.2.102:0/321011995 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa6200d1c70 0x7fa6200efc60 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 rx=0x7fa618006e50 tx=0x7fa618009e70).ready entity=mon.1 client_cookie=170e61e786421cb8 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:32.487 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:32.470 7fa613fff700 1 -- 172.21.2.102:0/321011995 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa61800b510 con 0x7fa6200d1c70 2021-02-10T11:27:32.487 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:32.470 7fa613fff700 1 -- 172.21.2.102:0/321011995 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa618006790 con 0x7fa6200d1c70 2021-02-10T11:27:32.487 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:32.470 7fa613fff700 1 -- 172.21.2.102:0/321011995 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa61800b940 con 0x7fa6200d1c70 2021-02-10T11:27:32.487 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:32.470 7fa6278a1700 1 -- 172.21.2.102:0/321011995 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa6200d1c70 msgr2=0x7fa6200efc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:32.487 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:32.470 7fa6278a1700 1 --2- 172.21.2.102:0/321011995 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa6200d1c70 0x7fa6200efc60 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 rx=0x7fa618006e50 tx=0x7fa618009e70).stop 2021-02-10T11:27:32.488 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:32.471 7fa6278a1700 1 -- 172.21.2.102:0/321011995 shutdown_connections 2021-02-10T11:27:32.488 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:32.471 7fa6278a1700 1 --2- 172.21.2.102:0/321011995 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa6200d1c70 0x7fa6200efc60 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:32.488 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:32.471 7fa6278a1700 1 --2- 172.21.2.102:0/321011995 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa6200d0340 0x7fa6200ef640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:32.488 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:32.471 7fa6278a1700 1 -- 172.21.2.102:0/321011995 >> 172.21.2.102:0/321011995 conn(0x7fa6200ce450 msgr2=0x7fa6200d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:32.488 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:32.471 7fa6278a1700 1 -- 172.21.2.102:0/321011995 shutdown_connections 2021-02-10T11:27:32.488 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:32.471 7fa6278a1700 1 -- 172.21.2.102:0/321011995 wait complete. 2021-02-10T11:27:32.489 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:32.486 7fa6278a1700 1 Processor -- start 2021-02-10T11:27:32.490 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:32.486 7fa6278a1700 1 -- start start 2021-02-10T11:27:32.490 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:32.486 7fa6278a1700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa6200bfa90 0x7fa62018a050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:32.490 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:32.486 7fa6278a1700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa6200bfe90 0x7fa62018a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:32.490 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:32.486 7fa6278a1700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa620192510 0x7fa620192910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:32.490 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:32.486 7fa6278a1700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa620189b50 con 0x7fa620192510 2021-02-10T11:27:32.490 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:32.486 7fa6278a1700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa620190d30 con 0x7fa6200bfe90 2021-02-10T11:27:32.490 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:32.486 7fa6278a1700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa620193d70 con 0x7fa6200bfa90 2021-02-10T11:27:32.490 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:32.487 7fa625ebe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa620192510 0x7fa620192910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:32.490 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:32.487 7fa625ebe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa620192510 0x7fa620192910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:59574/0 (socket says 172.21.2.102:59574) 2021-02-10T11:27:32.490 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:32.487 7fa6256bd700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa6200bfa90 0x7fa62018a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:32.490 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:32.487 7fa625ebe700 1 -- 172.21.2.102:0/2908535060 learned_addr learned my addr 172.21.2.102:0/2908535060 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:27:32.490 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:32.487 7fa624ebc700 1 --2- 172.21.2.102:0/2908535060 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa6200bfe90 0x7fa62018a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:32.491 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:32.487 7fa625ebe700 1 -- 172.21.2.102:0/2908535060 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa6200bfa90 msgr2=0x7fa62018a050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:32.491 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:32.487 7fa625ebe700 1 --2- 172.21.2.102:0/2908535060 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa6200bfa90 0x7fa62018a050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:32.491 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:32.487 7fa625ebe700 1 -- 172.21.2.102:0/2908535060 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa6200bfe90 msgr2=0x7fa62018a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:32.491 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:32.487 7fa625ebe700 1 --2- 172.21.2.102:0/2908535060 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa6200bfe90 0x7fa62018a670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:32.491 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:32.487 7fa625ebe700 1 -- 172.21.2.102:0/2908535060 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa620192f30 con 0x7fa620192510 2021-02-10T11:27:32.491 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:32.487 7fa625ebe700 1 --2- 172.21.2.102:0/2908535060 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa620192510 0x7fa620192910 secure :-1 s=READY pgs=1747 cs=0 l=1 rev1=1 rx=0x7fa618006c50 tx=0x7fa61800bd20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:32.491 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:32.488 7fa611ffb700 1 -- 172.21.2.102:0/2908535060 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa6180093d0 con 0x7fa620192510 2021-02-10T11:27:32.491 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:32.488 7fa611ffb700 1 -- 172.21.2.102:0/2908535060 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa61800be60 con 0x7fa620192510 2021-02-10T11:27:32.491 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:32.488 7fa611ffb700 1 -- 172.21.2.102:0/2908535060 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa61800c4e0 con 0x7fa620192510 2021-02-10T11:27:32.491 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:32.488 7fa6278a1700 1 -- 172.21.2.102:0/2908535060 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa620192f30 con 0x7fa620192510 2021-02-10T11:27:32.491 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:32.488 7fa6278a1700 1 -- 172.21.2.102:0/2908535060 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6201934c0 con 0x7fa620192510 2021-02-10T11:27:32.491 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:32.488 7fa611ffb700 1 -- 172.21.2.102:0/2908535060 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa6180093d0 con 0x7fa620192510 2021-02-10T11:27:32.491 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:32.488 7fa611ffb700 1 --2- 172.21.2.102:0/2908535060 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa6000186c0 0x7fa60001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:32.492 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:32.488 7fa611ffb700 1 -- 172.21.2.102:0/2908535060 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 5675+0+0 (secure 0 0 0) 0x7fa618032990 con 0x7fa620192510 2021-02-10T11:27:32.492 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:32.488 7fa6256bd700 1 --2- 172.21.2.102:0/2908535060 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa6000186c0 0x7fa60001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:32.492 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:32.489 7fa6256bd700 1 --2- 172.21.2.102:0/2908535060 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa6000186c0 0x7fa60001abc0 crc :-1 s=READY pgs=789 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:32.518 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:32.518 7fa6278a1700 1 -- 172.21.2.102:0/2908535060 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6200ebc50 con 0x7fa620192510 2021-02-10T11:27:32.521 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:32.521 7fa611ffb700 1 -- 172.21.2.102:0/2908535060 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fa618002ef0 con 0x7fa620192510 2021-02-10T11:27:32.650 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:32.650 7fa6278a1700 1 -- 172.21.2.102:0/2908535060 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fa6200f0b10 con 0x7fa620192510 2021-02-10T11:27:33.349 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:33.349 7fa611ffb700 1 -- 172.21.2.102:0/2908535060 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v60) v1 ==== 104+0+0 (secure 0 0 0) 0x7fa6180093d0 con 0x7fa620192510 2021-02-10T11:27:33.372 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:33.372 7fa6278a1700 1 -- 172.21.2.102:0/2908535060 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa6000186c0 msgr2=0x7fa60001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:33.372 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:33.372 7fa6278a1700 1 --2- 172.21.2.102:0/2908535060 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa6000186c0 0x7fa60001abc0 crc :-1 s=READY pgs=789 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:33.372 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:33.372 7fa6278a1700 1 -- 172.21.2.102:0/2908535060 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa620192510 msgr2=0x7fa620192910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:33.372 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:33.372 7fa6278a1700 1 --2- 172.21.2.102:0/2908535060 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa620192510 0x7fa620192910 secure :-1 s=READY pgs=1747 cs=0 l=1 rev1=1 rx=0x7fa618006c50 tx=0x7fa61800bd20).stop 2021-02-10T11:27:33.372 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:33.372 7fa6278a1700 1 -- 172.21.2.102:0/2908535060 shutdown_connections 2021-02-10T11:27:33.372 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:33.372 7fa6278a1700 1 --2- 172.21.2.102:0/2908535060 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa620192510 0x7fa620192910 unknown :-1 s=CLOSED pgs=1747 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:33.372 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:33.372 7fa6278a1700 1 --2- 172.21.2.102:0/2908535060 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa6000186c0 0x7fa60001abc0 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:33.373 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:33.372 7fa6278a1700 1 --2- 172.21.2.102:0/2908535060 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa6200bfe90 0x7fa62018a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:33.373 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:33.372 7fa6278a1700 1 --2- 172.21.2.102:0/2908535060 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa6200bfa90 0x7fa62018a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:33.373 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:33.372 7fa6278a1700 1 -- 172.21.2.102:0/2908535060 >> 172.21.2.102:0/2908535060 conn(0x7fa6200ed860 msgr2=0x7fa6200bf0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:33.373 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:33.372 7fa6278a1700 1 -- 172.21.2.102:0/2908535060 shutdown_connections 2021-02-10T11:27:33.373 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:33.372 7fa6278a1700 1 -- 172.21.2.102:0/2908535060 wait complete. 2021-02-10T11:27:33.380 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool delete cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2021-02-10T11:27:33.813 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:33.796 7fa685c5c700 1 Processor -- start 2021-02-10T11:27:33.813 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:33.796 7fa685c5c700 1 -- start start 2021-02-10T11:27:33.813 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:33.797 7fa685c5c700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa6800d1880 0x7fa6801b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:33.813 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:33.797 7fa685c5c700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fa6800cff80 0x7fa680109140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:33.813 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:33.797 7fa685c5c700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa680109a70 con 0x7fa6801b0af0 2021-02-10T11:27:33.813 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:33.797 7fa685c5c700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa680109760 con 0x7fa6800cff80 2021-02-10T11:27:33.813 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:33.797 7fa685c5c700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa680088070 con 0x7fa6800d1880 2021-02-10T11:27:33.813 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:33.797 7fa67effd700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fa6801b0af0 0x7fa680108950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:51802/0 (socket says 172.21.2.102:51802) 2021-02-10T11:27:33.814 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:33.797 7fa67effd700 1 -- 172.21.2.102:0/3954721137 learned_addr learned my addr 172.21.2.102:0/3954721137 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:27:33.814 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:33.797 7fa67f7fe700 1 --2- 172.21.2.102:0/3954721137 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa6800d1880 0x7fa6801b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:33.814 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:33.797 7fa67ffff700 1 --2- 172.21.2.102:0/3954721137 >> v2:172.21.2.113:3300/0 conn(0x7fa6800cff80 0x7fa680109140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:33.814 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:33.797 7fa67e7fc700 1 -- 172.21.2.102:0/3954721137 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2840570649 0 0) 0x7fa6700009b0 con 0x7fa6801b0af0 2021-02-10T11:27:33.814 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:33.797 7fa67e7fc700 1 -- 172.21.2.102:0/3954721137 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa66c002fe0 con 0x7fa6801b0af0 2021-02-10T11:27:33.814 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:33.797 7fa67ffff700 1 -- 172.21.2.102:0/3954721137 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa6800d1880 msgr2=0x7fa6801b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:27:33.814 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:33.797 7fa67ffff700 1 --2- 172.21.2.102:0/3954721137 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa6800d1880 0x7fa6801b04d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:33.814 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:33.797 7fa67ffff700 1 -- 172.21.2.102:0/3954721137 >> v1:172.21.2.102:6789/0 conn(0x7fa6801b0af0 legacy=0x7fa680108950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:33.814 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:33.798 7fa67ffff700 1 -- 172.21.2.102:0/3954721137 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa680088260 con 0x7fa6800cff80 2021-02-10T11:27:33.814 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:33.798 7fa67ffff700 1 --2- 172.21.2.102:0/3954721137 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa6800cff80 0x7fa680109140 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 rx=0x7fa678006e50 tx=0x7fa678009e70).ready entity=mon.1 client_cookie=aec06e9dfc3d941d server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:33.814 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:33.798 7fa67e7fc700 1 -- 172.21.2.102:0/3954721137 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa67800b510 con 0x7fa6800cff80 2021-02-10T11:27:33.815 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:33.798 7fa67e7fc700 1 -- 172.21.2.102:0/3954721137 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa678006790 con 0x7fa6800cff80 2021-02-10T11:27:33.815 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:33.798 7fa685c5c700 1 -- 172.21.2.102:0/3954721137 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa6800cff80 msgr2=0x7fa680109140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:33.815 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:33.798 7fa685c5c700 1 --2- 172.21.2.102:0/3954721137 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa6800cff80 0x7fa680109140 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 rx=0x7fa678006e50 tx=0x7fa678009e70).stop 2021-02-10T11:27:33.815 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:33.798 7fa67e7fc700 1 -- 172.21.2.102:0/3954721137 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa678000f90 con 0x7fa6800cff80 2021-02-10T11:27:33.815 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:33.798 7fa685c5c700 1 -- 172.21.2.102:0/3954721137 shutdown_connections 2021-02-10T11:27:33.815 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:33.798 7fa685c5c700 1 --2- 172.21.2.102:0/3954721137 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa6800cff80 0x7fa680109140 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:33.815 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:33.798 7fa685c5c700 1 --2- 172.21.2.102:0/3954721137 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa6800d1880 0x7fa6801b04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:33.815 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:33.798 7fa685c5c700 1 -- 172.21.2.102:0/3954721137 >> 172.21.2.102:0/3954721137 conn(0x7fa6800ce430 msgr2=0x7fa6800d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:33.815 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:33.799 7fa685c5c700 1 -- 172.21.2.102:0/3954721137 shutdown_connections 2021-02-10T11:27:33.815 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:33.799 7fa685c5c700 1 -- 172.21.2.102:0/3954721137 wait complete. 2021-02-10T11:27:33.817 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:33.813 7fa685c5c700 1 Processor -- start 2021-02-10T11:27:33.817 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:33.813 7fa685c5c700 1 -- start start 2021-02-10T11:27:33.817 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:33.814 7fa685c5c700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa68009a690 0x7fa68009aa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:33.817 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:33.814 7fa685c5c700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa6801b4160 0x7fa6801b4560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:33.817 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:33.814 7fa685c5c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa6801b7dd0 0x7fa6801b81d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:33.818 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:33.814 7fa685c5c700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6801b4b80 con 0x7fa6801b7dd0 2021-02-10T11:27:33.818 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:33.814 7fa685c5c700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6801b87f0 con 0x7fa68009a690 2021-02-10T11:27:33.818 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:33.814 7fa685c5c700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6801b8a10 con 0x7fa6801b4160 2021-02-10T11:27:33.818 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:33.814 7fa67f7fe700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa68009a690 0x7fa68009aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:33.818 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:33.814 7fa67f7fe700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa68009a690 0x7fa68009aa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.113:3300/0 says I am v2:172.21.2.102:34780/0 (socket says 172.21.2.102:34780) 2021-02-10T11:27:33.818 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:33.814 7fa67f7fe700 1 -- 172.21.2.102:0/865827819 learned_addr learned my addr 172.21.2.102:0/865827819 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:27:33.818 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:33.814 7fa67ffff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa6801b7dd0 0x7fa6801b81d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:33.818 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:33.814 7fa67effd700 1 --2- 172.21.2.102:0/865827819 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa6801b4160 0x7fa6801b4560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:33.818 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:33.814 7fa67ffff700 1 -- 172.21.2.102:0/865827819 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa6801b4160 msgr2=0x7fa6801b4560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:33.818 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:33.814 7fa67ffff700 1 --2- 172.21.2.102:0/865827819 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa6801b4160 0x7fa6801b4560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:33.818 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:33.814 7fa67ffff700 1 -- 172.21.2.102:0/865827819 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa68009a690 msgr2=0x7fa68009aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:33.819 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:33.814 7fa67ffff700 1 --2- 172.21.2.102:0/865827819 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa68009a690 0x7fa68009aa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:33.819 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:33.814 7fa67ffff700 1 -- 172.21.2.102:0/865827819 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6801b8c30 con 0x7fa6801b7dd0 2021-02-10T11:27:33.819 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:33.814 7fa67ffff700 1 --2- 172.21.2.102:0/865827819 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa6801b7dd0 0x7fa6801b81d0 secure :-1 s=READY pgs=1749 cs=0 l=1 rev1=1 rx=0x7fa678009950 tx=0x7fa67800bbf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:33.819 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:33.815 7fa677fff700 1 -- 172.21.2.102:0/865827819 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa6780082e0 con 0x7fa6801b7dd0 2021-02-10T11:27:33.819 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:33.815 7fa685c5c700 1 -- 172.21.2.102:0/865827819 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6801b8c30 con 0x7fa6801b7dd0 2021-02-10T11:27:33.819 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:33.815 7fa677fff700 1 -- 172.21.2.102:0/865827819 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6780082e0 con 0x7fa6801b7dd0 2021-02-10T11:27:33.819 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:33.815 7fa677fff700 1 -- 172.21.2.102:0/865827819 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa6780085e0 con 0x7fa6801b7dd0 2021-02-10T11:27:33.820 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:33.815 7fa685c5c700 1 -- 172.21.2.102:0/865827819 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6801b8c30 con 0x7fa6801b7dd0 2021-02-10T11:27:33.820 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:33.815 7fa677fff700 1 -- 172.21.2.102:0/865827819 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa6780093d0 con 0x7fa6801b7dd0 2021-02-10T11:27:33.820 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:33.815 7fa677fff700 1 --2- 172.21.2.102:0/865827819 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa6640186c0 0x7fa66401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:33.821 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:33.815 7fa677fff700 1 -- 172.21.2.102:0/865827819 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 5675+0+0 (secure 0 0 0) 0x7fa678032720 con 0x7fa6801b7dd0 2021-02-10T11:27:33.821 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:33.815 7fa67f7fe700 1 --2- 172.21.2.102:0/865827819 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa6640186c0 0x7fa66401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:33.821 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:33.816 7fa67f7fe700 1 --2- 172.21.2.102:0/865827819 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa6640186c0 0x7fa66401abc0 crc :-1 s=READY pgs=790 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:33.847 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:33.847 7fa685c5c700 1 -- 172.21.2.102:0/865827819 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa68009a160 con 0x7fa6801b7dd0 2021-02-10T11:27:33.849 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:33.849 7fa677fff700 1 -- 172.21.2.102:0/865827819 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fa6780080d0 con 0x7fa6801b7dd0 2021-02-10T11:27:33.976 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:33.976 7fa685c5c700 1 -- 172.21.2.102:0/865827819 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"pool2": "cephfs_metadata", "prefix": "osd pool delete", "pool": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fa68009a160 con 0x7fa6801b7dd0 2021-02-10T11:27:34.355 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:34.355 7fa677fff700 1 -- 172.21.2.102:0/865827819 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"pool2": "cephfs_metadata", "prefix": "osd pool delete", "pool": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v79) v1 ==== 185+0+0 (secure 0 0 0) 0x7fa67801f7a0 con 0x7fa6801b7dd0 2021-02-10T11:27:34.393 INFO:teuthology.orchestra.run.gibba002.stderr:pool 'cephfs_metadata' removed 2021-02-10T11:27:34.394 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:34.395 7fa685c5c700 1 -- 172.21.2.102:0/865827819 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa6640186c0 msgr2=0x7fa66401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:34.395 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:34.395 7fa685c5c700 1 --2- 172.21.2.102:0/865827819 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa6640186c0 0x7fa66401abc0 crc :-1 s=READY pgs=790 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:34.395 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:34.395 7fa685c5c700 1 -- 172.21.2.102:0/865827819 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa6801b7dd0 msgr2=0x7fa6801b81d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:34.395 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:34.395 7fa685c5c700 1 --2- 172.21.2.102:0/865827819 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa6801b7dd0 0x7fa6801b81d0 secure :-1 s=READY pgs=1749 cs=0 l=1 rev1=1 rx=0x7fa678009950 tx=0x7fa67800bbf0).stop 2021-02-10T11:27:34.395 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:34.395 7fa685c5c700 1 -- 172.21.2.102:0/865827819 shutdown_connections 2021-02-10T11:27:34.395 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:34.395 7fa685c5c700 1 --2- 172.21.2.102:0/865827819 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa6801b7dd0 0x7fa6801b81d0 unknown :-1 s=CLOSED pgs=1749 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:34.395 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:34.395 7fa685c5c700 1 --2- 172.21.2.102:0/865827819 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa6801b4160 0x7fa6801b4560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:34.395 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:34.395 7fa685c5c700 1 --2- 172.21.2.102:0/865827819 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa6640186c0 0x7fa66401abc0 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:34.395 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:34.395 7fa685c5c700 1 --2- 172.21.2.102:0/865827819 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa68009a690 0x7fa68009aa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:34.395 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:34.395 7fa685c5c700 1 -- 172.21.2.102:0/865827819 >> 172.21.2.102:0/865827819 conn(0x7fa68018e870 msgr2=0x7fa68018ecc0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:34.396 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:34.395 7fa685c5c700 1 -- 172.21.2.102:0/865827819 shutdown_connections 2021-02-10T11:27:34.396 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:34.395 7fa685c5c700 1 -- 172.21.2.102:0/865827819 wait complete. 2021-02-10T11:27:34.403 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool delete cephfs_data cephfs_data --yes-i-really-really-mean-it 2021-02-10T11:27:34.838 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:34.818 7f98d4e29700 1 Processor -- start 2021-02-10T11:27:34.838 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:34.818 7f98d4e29700 1 -- start start 2021-02-10T11:27:34.839 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:34.819 7f98d4e29700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f98d00d1870 0x7f98d01b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:34.839 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:34.819 7f98d4e29700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f98d00cff70 0x7f98d0109130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:34.839 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:34.819 7f98d4e29700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98d00fe1c0 con 0x7f98d01b0ae0 2021-02-10T11:27:34.839 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:34.819 7f98d4e29700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98d0109ad0 con 0x7f98d00d1870 2021-02-10T11:27:34.839 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:34.819 7f98d4e29700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98d0109750 con 0x7f98d00cff70 2021-02-10T11:27:34.839 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:34.819 7f98ceb13700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f98d00d1870 0x7f98d01b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:34.840 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:34.819 7f98ce312700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f98d01b0ae0 0x7f98d0108940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:51816/0 (socket says 172.21.2.102:51816) 2021-02-10T11:27:34.840 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:34.819 7f98ce312700 1 -- 172.21.2.102:0/1373335432 learned_addr learned my addr 172.21.2.102:0/1373335432 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:27:34.840 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:34.819 7f98cf314700 1 --2- 172.21.2.102:0/1373335432 >> v2:172.21.2.113:3300/0 conn(0x7f98d00cff70 0x7f98d0109130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:34.840 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:34.819 7f98ceb13700 1 -- 172.21.2.102:0/1373335432 >> v2:172.21.2.113:3300/0 conn(0x7f98d00cff70 msgr2=0x7f98d0109130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:27:34.840 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:34.819 7f98ceb13700 1 --2- 172.21.2.102:0/1373335432 >> v2:172.21.2.113:3300/0 conn(0x7f98d00cff70 0x7f98d0109130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:34.841 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:34.819 7f98ceb13700 1 -- 172.21.2.102:0/1373335432 >> v1:172.21.2.102:6789/0 conn(0x7f98d01b0ae0 legacy=0x7f98d0108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:34.841 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:34.819 7f98ceb13700 1 -- 172.21.2.102:0/1373335432 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98d0088070 con 0x7f98d00d1870 2021-02-10T11:27:34.841 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:34.819 7f98ceb13700 1 --2- 172.21.2.102:0/1373335432 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f98d00d1870 0x7f98d01b04c0 secure :-1 s=READY pgs=1751 cs=0 l=1 rev1=1 rx=0x7f98c0006650 tx=0x7f98c0009a80).ready entity=mon.0 client_cookie=808ddf66c2565689 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:34.841 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:34.819 7f98cdb11700 1 -- 172.21.2.102:0/1373335432 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f98c00009b0 con 0x7f98d00d1870 2021-02-10T11:27:34.841 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:34.819 7f98cdb11700 1 -- 172.21.2.102:0/1373335432 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98c000c0c0 con 0x7f98d00d1870 2021-02-10T11:27:34.842 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:34.819 7f98cdb11700 1 -- 172.21.2.102:0/1373335432 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f98c000c520 con 0x7f98d00d1870 2021-02-10T11:27:34.842 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:34.820 7f98d4e29700 1 -- 172.21.2.102:0/1373335432 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f98d00d1870 msgr2=0x7f98d01b04c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:34.842 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:34.820 7f98d4e29700 1 --2- 172.21.2.102:0/1373335432 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f98d00d1870 0x7f98d01b04c0 secure :-1 s=READY pgs=1751 cs=0 l=1 rev1=1 rx=0x7f98c0006650 tx=0x7f98c0009a80).stop 2021-02-10T11:27:34.842 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:34.820 7f98d4e29700 1 -- 172.21.2.102:0/1373335432 shutdown_connections 2021-02-10T11:27:34.842 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:34.820 7f98d4e29700 1 --2- 172.21.2.102:0/1373335432 >> v2:172.21.2.113:3300/0 conn(0x7f98d00cff70 0x7f98d0109130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:34.843 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:34.820 7f98d4e29700 1 --2- 172.21.2.102:0/1373335432 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f98d00d1870 0x7f98d01b04c0 unknown :-1 s=CLOSED pgs=1751 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:34.843 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:34.820 7f98d4e29700 1 -- 172.21.2.102:0/1373335432 >> 172.21.2.102:0/1373335432 conn(0x7f98d00ce420 msgr2=0x7f98d00d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:34.843 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:34.820 7f98d4e29700 1 -- 172.21.2.102:0/1373335432 shutdown_connections 2021-02-10T11:27:34.843 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:34.820 7f98d4e29700 1 -- 172.21.2.102:0/1373335432 wait complete. 2021-02-10T11:27:34.843 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:34.835 7f98d4e29700 1 Processor -- start 2021-02-10T11:27:34.843 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:34.835 7f98d4e29700 1 -- start start 2021-02-10T11:27:34.844 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:34.835 7f98d4e29700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f98d019fb30 0x7f98d019ff30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:34.844 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:34.835 7f98d4e29700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f98d009a0d0 0x7f98d009a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:34.844 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:34.835 7f98d4e29700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f98d009aaf0 0x7f98d01a0ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:34.844 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:34.835 7f98d4e29700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98d01a0920 con 0x7f98d009a0d0 2021-02-10T11:27:34.844 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:34.835 7f98d4e29700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98d009aef0 con 0x7f98d019fb30 2021-02-10T11:27:34.845 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:34.835 7f98d4e29700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98d01a0550 con 0x7f98d009aaf0 2021-02-10T11:27:34.845 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:34.836 7f98ce312700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f98d009a0d0 0x7f98d009a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:34.845 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:34.836 7f98ce312700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f98d009a0d0 0x7f98d009a4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:59598/0 (socket says 172.21.2.102:59598) 2021-02-10T11:27:34.845 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:34.836 7f98ce312700 1 -- 172.21.2.102:0/3466297900 learned_addr learned my addr 172.21.2.102:0/3466297900 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:27:34.845 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:34.836 7f98ce312700 1 -- 172.21.2.102:0/3466297900 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f98d009aaf0 msgr2=0x7f98d01a0ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:34.846 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:34.836 7f98ce312700 1 --2- 172.21.2.102:0/3466297900 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f98d009aaf0 0x7f98d01a0ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:34.846 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:34.836 7f98ce312700 1 -- 172.21.2.102:0/3466297900 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f98d019fb30 msgr2=0x7f98d019ff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:34.846 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:34.836 7f98ce312700 1 --2- 172.21.2.102:0/3466297900 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f98d019fb30 0x7f98d019ff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:34.846 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:34.836 7f98ce312700 1 -- 172.21.2.102:0/3466297900 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98d01a14c0 con 0x7f98d009a0d0 2021-02-10T11:27:34.846 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:34.836 7f98ce312700 1 --2- 172.21.2.102:0/3466297900 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f98d009a0d0 0x7f98d009a4d0 secure :-1 s=READY pgs=1752 cs=0 l=1 rev1=1 rx=0x7f98b800a920 tx=0x7f98b800acc0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:34.847 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:34.836 7f98c77fe700 1 -- 172.21.2.102:0/3466297900 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f98b80070d0 con 0x7f98d009a0d0 2021-02-10T11:27:34.847 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:34.836 7f98c77fe700 1 -- 172.21.2.102:0/3466297900 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98b8007390 con 0x7f98d009a0d0 2021-02-10T11:27:34.847 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:34.836 7f98c77fe700 1 -- 172.21.2.102:0/3466297900 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f98b8004740 con 0x7f98d009a0d0 2021-02-10T11:27:34.847 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:34.836 7f98d4e29700 1 -- 172.21.2.102:0/3466297900 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98d01a14c0 con 0x7f98d009a0d0 2021-02-10T11:27:34.847 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:34.837 7f98d4e29700 1 -- 172.21.2.102:0/3466297900 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98d01a14c0 con 0x7f98d009a0d0 2021-02-10T11:27:34.847 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:34.837 7f98c77fe700 1 -- 172.21.2.102:0/3466297900 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f98b800ae00 con 0x7f98d009a0d0 2021-02-10T11:27:34.848 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:34.837 7f98c77fe700 1 --2- 172.21.2.102:0/3466297900 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f98b40186c0 0x7f98b401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:34.848 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:34.837 7f98c77fe700 1 -- 172.21.2.102:0/3466297900 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 5237+0+0 (secure 0 0 0) 0x7f98b80329e0 con 0x7f98d009a0d0 2021-02-10T11:27:34.848 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:34.837 7f98ceb13700 1 --2- 172.21.2.102:0/3466297900 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f98b40186c0 0x7f98b401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:34.848 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:34.838 7f98ceb13700 1 --2- 172.21.2.102:0/3466297900 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f98b40186c0 0x7f98b401abc0 crc :-1 s=READY pgs=791 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:34.869 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:34.868 7f98d4e29700 1 -- 172.21.2.102:0/3466297900 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98d00825d0 con 0x7f98d009a0d0 2021-02-10T11:27:34.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:34.871 7f98c77fe700 1 -- 172.21.2.102:0/3466297900 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f98b8007b80 con 0x7f98d009a0d0 2021-02-10T11:27:34.999 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:34.999 7f98d4e29700 1 -- 172.21.2.102:0/3466297900 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"pool2": "cephfs_data", "prefix": "osd pool delete", "pool": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f98d00825d0 con 0x7f98d009a0d0 2021-02-10T11:27:35.361 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:35.361 7f98c77fe700 1 -- 172.21.2.102:0/3466297900 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"pool2": "cephfs_data", "prefix": "osd pool delete", "pool": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v80) v1 ==== 173+0+0 (secure 0 0 0) 0x7f98b800ae00 con 0x7f98d009a0d0 2021-02-10T11:27:35.364 INFO:teuthology.orchestra.run.gibba002.stderr:pool 'cephfs_data' removed 2021-02-10T11:27:35.367 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:35.366 7f98d4e29700 1 -- 172.21.2.102:0/3466297900 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f98b40186c0 msgr2=0x7f98b401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:35.367 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:35.366 7f98d4e29700 1 --2- 172.21.2.102:0/3466297900 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f98b40186c0 0x7f98b401abc0 crc :-1 s=READY pgs=791 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:35.367 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:35.366 7f98d4e29700 1 -- 172.21.2.102:0/3466297900 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f98d009a0d0 msgr2=0x7f98d009a4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:35.367 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:35.366 7f98d4e29700 1 --2- 172.21.2.102:0/3466297900 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f98d009a0d0 0x7f98d009a4d0 secure :-1 s=READY pgs=1752 cs=0 l=1 rev1=1 rx=0x7f98b800a920 tx=0x7f98b800acc0).stop 2021-02-10T11:27:35.367 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:35.366 7f98d4e29700 1 -- 172.21.2.102:0/3466297900 shutdown_connections 2021-02-10T11:27:35.367 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:35.366 7f98d4e29700 1 --2- 172.21.2.102:0/3466297900 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f98d009aaf0 0x7f98d01a0ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:35.367 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:35.366 7f98d4e29700 1 --2- 172.21.2.102:0/3466297900 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f98d009a0d0 0x7f98d009a4d0 unknown :-1 s=CLOSED pgs=1752 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:35.367 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:35.366 7f98d4e29700 1 --2- 172.21.2.102:0/3466297900 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f98b40186c0 0x7f98b401abc0 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:35.367 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:35.366 7f98d4e29700 1 --2- 172.21.2.102:0/3466297900 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f98d019fb30 0x7f98d019ff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:35.367 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:35.366 7f98d4e29700 1 -- 172.21.2.102:0/3466297900 >> 172.21.2.102:0/3466297900 conn(0x7f98d01b3560 msgr2=0x7f98d01b90e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:35.367 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:35.366 7f98d4e29700 1 -- 172.21.2.102:0/3466297900 shutdown_connections 2021-02-10T11:27:35.368 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:35.367 7f98d4e29700 1 -- 172.21.2.102:0/3466297900 wait complete. 2021-02-10T11:27:35.375 INFO:tasks.ceph.mds.a:Restarting daemon 2021-02-10T11:27:35.375 INFO:tasks.ceph.mds.a:Stopping old one... 2021-02-10T11:27:35.376 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2021-02-10T11:27:35.376 INFO:teuthology.orchestra.run:waiting for 300 2021-02-10T11:27:35.376 INFO:tasks.ceph.mds.b:Restarting daemon 2021-02-10T11:27:35.376 INFO:tasks.ceph.mds.b:Stopping old one... 2021-02-10T11:27:35.376 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2021-02-10T11:27:35.376 INFO:teuthology.orchestra.run:waiting for 300 2021-02-10T11:27:35.382 INFO:tasks.ceph.mds.a:Stopped 2021-02-10T11:27:35.382 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2021-02-10T11:27:35.384 INFO:tasks.ceph.mds.b:Stopped 2021-02-10T11:27:35.384 DEBUG:teuthology.orchestra.run.gibba013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2021-02-10T11:27:35.424 INFO:tasks.ceph.mds.a:Started 2021-02-10T11:27:35.425 DEBUG:teuthology.parallel:result is None 2021-02-10T11:27:35.425 INFO:tasks.ceph.mds.b:Started 2021-02-10T11:27:35.425 DEBUG:teuthology.parallel:result is None 2021-02-10T11:27:35.426 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd blacklist clear 2021-02-10T11:27:35.831 INFO:tasks.ceph.mds.b.gibba013.stdout:starting mds.b at 2021-02-10T11:27:35.832 INFO:tasks.ceph.mds.a.gibba002.stdout:starting mds.a at 2021-02-10T11:27:35.853 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:35.836 7f7676605700 1 Processor -- start 2021-02-10T11:27:35.853 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:35.836 7f7676605700 1 -- start start 2021-02-10T11:27:35.853 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:35.836 7f7676605700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f76700d1860 0x7f76701b03e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:35.854 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:35.836 7f7676605700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f76700f0640 0x7f76700cfb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:35.854 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:35.836 7f7676605700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76700f0b00 con 0x7f76701b0a00 2021-02-10T11:27:35.854 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:35.836 7f7676605700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76700e5ec0 con 0x7f76700d1860 2021-02-10T11:27:35.854 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:35.836 7f7676605700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76700d1c60 con 0x7f76700f0640 2021-02-10T11:27:35.854 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:35.836 7f7667fff700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f76701b0a00 0x7f76700cf320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:51844/0 (socket says 172.21.2.102:51844) 2021-02-10T11:27:35.854 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:35.836 7f7667fff700 1 -- 172.21.2.102:0/1677959634 learned_addr learned my addr 172.21.2.102:0/1677959634 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:27:35.854 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:35.836 7f7674c22700 1 --2- 172.21.2.102:0/1677959634 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f76700f0640 0x7f76700cfb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:35.854 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:35.837 7f766ffff700 1 --2- 172.21.2.102:0/1677959634 >> v2:172.21.2.113:3300/0 conn(0x7f76700d1860 0x7f76701b03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:35.854 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:35.837 7f766f7fe700 1 -- 172.21.2.102:0/1677959634 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 743449225 0 0) 0x7f7660000d30 con 0x7f76701b0a00 2021-02-10T11:27:35.854 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:35.837 7f766f7fe700 1 -- 172.21.2.102:0/1677959634 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f765c002fe0 con 0x7f76701b0a00 2021-02-10T11:27:35.854 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:35.837 7f7674c22700 1 -- 172.21.2.102:0/1677959634 >> v2:172.21.2.113:3300/0 conn(0x7f76700d1860 msgr2=0x7f76701b03e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:27:35.854 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:35.837 7f7674c22700 1 --2- 172.21.2.102:0/1677959634 >> v2:172.21.2.113:3300/0 conn(0x7f76700d1860 0x7f76701b03e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:35.855 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:35.837 7f7674c22700 1 -- 172.21.2.102:0/1677959634 >> v1:172.21.2.102:6789/0 conn(0x7f76701b0a00 legacy=0x7f76700cf320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:35.855 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:35.837 7f7674c22700 1 -- 172.21.2.102:0/1677959634 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76700d1e80 con 0x7f76700f0640 2021-02-10T11:27:35.855 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:35.837 7f7674c22700 1 --2- 172.21.2.102:0/1677959634 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f76700f0640 0x7f76700cfb10 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7f7668006ad0 tx=0x7f7668009ad0).ready entity=mon.2 client_cookie=c29f3d853e862f91 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:35.855 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:35.837 7f766f7fe700 1 -- 172.21.2.102:0/1677959634 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f76680022e0 con 0x7f76700f0640 2021-02-10T11:27:35.855 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:35.837 7f766f7fe700 1 -- 172.21.2.102:0/1677959634 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7668006410 con 0x7f76700f0640 2021-02-10T11:27:35.855 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:35.837 7f766f7fe700 1 -- 172.21.2.102:0/1677959634 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7668002620 con 0x7f76700f0640 2021-02-10T11:27:35.855 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:35.837 7f7676605700 1 -- 172.21.2.102:0/1677959634 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f76700f0640 msgr2=0x7f76700cfb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:35.855 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:35.837 7f7676605700 1 --2- 172.21.2.102:0/1677959634 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f76700f0640 0x7f76700cfb10 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7f7668006ad0 tx=0x7f7668009ad0).stop 2021-02-10T11:27:35.855 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:35.838 7f7676605700 1 -- 172.21.2.102:0/1677959634 shutdown_connections 2021-02-10T11:27:35.855 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:35.838 7f7676605700 1 --2- 172.21.2.102:0/1677959634 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f76700f0640 0x7f76700cfb10 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:35.855 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:35.838 7f7676605700 1 --2- 172.21.2.102:0/1677959634 >> v2:172.21.2.113:3300/0 conn(0x7f76700d1860 0x7f76701b03e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:35.855 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:35.838 7f7676605700 1 -- 172.21.2.102:0/1677959634 >> 172.21.2.102:0/1677959634 conn(0x7f76700ce440 msgr2=0x7f76700d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:35.855 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:35.838 7f7676605700 1 -- 172.21.2.102:0/1677959634 shutdown_connections 2021-02-10T11:27:35.855 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:35.838 7f7676605700 1 -- 172.21.2.102:0/1677959634 wait complete. 2021-02-10T11:27:35.856 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:35.853 7f7676605700 1 Processor -- start 2021-02-10T11:27:35.856 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:35.853 7f7676605700 1 -- start start 2021-02-10T11:27:35.856 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:35.853 7f7676605700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f76701a35c0 0x7f76701a39c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:35.856 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:35.853 7f7676605700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f767009d0e0 0x7f767009d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:35.856 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:35.853 7f7676605700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f767009db00 0x7f767009df00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:35.856 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:35.853 7f7676605700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f767009cb60 con 0x7f767009d0e0 2021-02-10T11:27:35.856 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:35.853 7f7676605700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76700c3730 con 0x7f767009db00 2021-02-10T11:27:35.856 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:35.853 7f7676605700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76700c3910 con 0x7f76701a35c0 2021-02-10T11:27:35.856 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:35.853 7f7667fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f767009d0e0 0x7f767009d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:35.856 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:35.853 7f7667fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f767009d0e0 0x7f767009d4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:59624/0 (socket says 172.21.2.102:59624) 2021-02-10T11:27:35.856 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:35.853 7f7667fff700 1 -- 172.21.2.102:0/3317325803 learned_addr learned my addr 172.21.2.102:0/3317325803 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:27:35.856 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:35.853 7f766ffff700 1 --2- 172.21.2.102:0/3317325803 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f76701a35c0 0x7f76701a39c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:35.856 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:35.853 7f7674c22700 1 --2- 172.21.2.102:0/3317325803 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f767009db00 0x7f767009df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:35.858 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:35.854 7f7667fff700 1 -- 172.21.2.102:0/3317325803 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f76701a35c0 msgr2=0x7f76701a39c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:35.858 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:35.854 7f7667fff700 1 --2- 172.21.2.102:0/3317325803 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f76701a35c0 0x7f76701a39c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:35.858 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:35.854 7f7667fff700 1 -- 172.21.2.102:0/3317325803 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f767009db00 msgr2=0x7f767009df00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:35.858 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:35.854 7f7667fff700 1 --2- 172.21.2.102:0/3317325803 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f767009db00 0x7f767009df00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:35.858 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:35.854 7f7667fff700 1 -- 172.21.2.102:0/3317325803 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7670093e60 con 0x7f767009d0e0 2021-02-10T11:27:35.858 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:35.854 7f7667fff700 1 --2- 172.21.2.102:0/3317325803 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f767009d0e0 0x7f767009d4e0 secure :-1 s=READY pgs=1756 cs=0 l=1 rev1=1 rx=0x7f766000a920 tx=0x7f766000acc0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:35.858 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:35.854 7f766d7fa700 1 -- 172.21.2.102:0/3317325803 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7660007020 con 0x7f767009d0e0 2021-02-10T11:27:35.858 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:35.854 7f766d7fa700 1 -- 172.21.2.102:0/3317325803 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76600072e0 con 0x7f767009d0e0 2021-02-10T11:27:35.859 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:35.854 7f7676605700 1 -- 172.21.2.102:0/3317325803 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7670093e60 con 0x7f767009d0e0 2021-02-10T11:27:35.859 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:35.854 7f7676605700 1 -- 172.21.2.102:0/3317325803 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76700943f0 con 0x7f767009d0e0 2021-02-10T11:27:35.859 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:35.854 7f766d7fa700 1 -- 172.21.2.102:0/3317325803 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7660004740 con 0x7f767009d0e0 2021-02-10T11:27:35.859 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:35.855 7f766d7fa700 1 -- 172.21.2.102:0/3317325803 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f766000ae00 con 0x7f767009d0e0 2021-02-10T11:27:35.859 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:35.855 7f766d7fa700 1 --2- 172.21.2.102:0/3317325803 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f76540186c0 0x7f765401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:35.859 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:35.855 7f766d7fa700 1 -- 172.21.2.102:0/3317325803 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 4839+0+0 (secure 0 0 0) 0x7f7660032640 con 0x7f767009d0e0 2021-02-10T11:27:35.860 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:35.855 7f766ffff700 1 --2- 172.21.2.102:0/3317325803 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f76540186c0 0x7f765401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:35.860 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:35.855 7f766ffff700 1 --2- 172.21.2.102:0/3317325803 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f76540186c0 0x7f765401abc0 crc :-1 s=READY pgs=792 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:35.885 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:35.885 7f7676605700 1 -- 172.21.2.102:0/3317325803 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76700ebbe0 con 0x7f767009d0e0 2021-02-10T11:27:35.887 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:35.887 7f766d7fa700 1 -- 172.21.2.102:0/3317325803 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f766002cfd0 con 0x7f767009d0e0 2021-02-10T11:27:36.025 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:36.025 7f7676605700 1 -- 172.21.2.102:0/3317325803 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd blacklist clear"} v 0) v1 -- 0x7f76700f0b00 con 0x7f767009d0e0 2021-02-10T11:27:36.366 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:36.366 7f766d7fa700 1 -- 172.21.2.102:0/3317325803 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd blacklist clear"}]=0 removed all blacklist entries v81) v1 ==== 97+0+0 (secure 0 0 0) 0x7f766002df90 con 0x7f767009d0e0 2021-02-10T11:27:36.383 DEBUG:teuthology.orchestra.run.gibba002:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:27:36.386 DEBUG:teuthology.orchestra.run.gibba013:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:27:36.391 INFO:teuthology.orchestra.run.gibba002.stderr: removed all blacklist entries 2021-02-10T11:27:36.393 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:36.393 7f7676605700 1 -- 172.21.2.102:0/3317325803 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f76540186c0 msgr2=0x7f765401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:36.393 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:36.393 7f7676605700 1 --2- 172.21.2.102:0/3317325803 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f76540186c0 0x7f765401abc0 crc :-1 s=READY pgs=792 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:36.394 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:36.393 7f7676605700 1 -- 172.21.2.102:0/3317325803 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f767009d0e0 msgr2=0x7f767009d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:36.394 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:36.393 7f7676605700 1 --2- 172.21.2.102:0/3317325803 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f767009d0e0 0x7f767009d4e0 secure :-1 s=READY pgs=1756 cs=0 l=1 rev1=1 rx=0x7f766000a920 tx=0x7f766000acc0).stop 2021-02-10T11:27:36.394 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:36.393 7f7676605700 1 -- 172.21.2.102:0/3317325803 shutdown_connections 2021-02-10T11:27:36.394 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:36.393 7f7676605700 1 --2- 172.21.2.102:0/3317325803 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f76540186c0 0x7f765401abc0 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:36.394 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:36.393 7f7676605700 1 --2- 172.21.2.102:0/3317325803 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f767009db00 0x7f767009df00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:36.394 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:36.393 7f7676605700 1 --2- 172.21.2.102:0/3317325803 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f767009d0e0 0x7f767009d4e0 unknown :-1 s=CLOSED pgs=1756 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:36.394 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:36.393 7f7676605700 1 --2- 172.21.2.102:0/3317325803 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f76701a35c0 0x7f76701a39c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:36.395 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:36.393 7f7676605700 1 -- 172.21.2.102:0/3317325803 >> 172.21.2.102:0/3317325803 conn(0x7f76700a6b00 msgr2=0x7f7670095700 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:36.395 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:36.393 7f7676605700 1 -- 172.21.2.102:0/3317325803 shutdown_connections 2021-02-10T11:27:36.395 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:36.393 7f7676605700 1 -- 172.21.2.102:0/3317325803 wait complete. 2021-02-10T11:27:36.401 INFO:tasks.cephfs.cephfs_test_case:['0', '1', '2', '3'] 2021-02-10T11:27:36.402 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth ls --format=json-pretty 2021-02-10T11:27:36.815 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:36.796 7fd12ee0a700 1 Processor -- start 2021-02-10T11:27:36.815 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:36.796 7fd12ee0a700 1 -- start start 2021-02-10T11:27:36.815 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:36.797 7fd12ee0a700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fd1280d1870 0x7fd1281b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:36.815 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:36.797 7fd12ee0a700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fd1280f0650 0x7fd1280cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:36.815 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:36.797 7fd12ee0a700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1280f0b10 con 0x7fd1281b0a10 2021-02-10T11:27:36.815 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:36.797 7fd12ee0a700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1280e5ed0 con 0x7fd1280f0650 2021-02-10T11:27:36.815 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:36.797 7fd12ee0a700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1280d1c70 con 0x7fd1280d1870 2021-02-10T11:27:36.815 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:36.797 7fd12d427700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fd1280f0650 0x7fd1280cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:36.816 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:36.797 7fd12d427700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fd1280f0650 0x7fd1280cfb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:59632/0 (socket says 172.21.2.102:59632) 2021-02-10T11:27:36.816 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:36.797 7fd12d427700 1 -- 172.21.2.102:0/2070699782 learned_addr learned my addr 172.21.2.102:0/2070699782 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:27:36.816 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:36.797 7fd12d427700 1 -- 172.21.2.102:0/2070699782 >> v2:172.21.2.113:3300/0 conn(0x7fd1280d1870 msgr2=0x7fd1281b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:27:36.816 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:36.797 7fd12d427700 1 --2- 172.21.2.102:0/2070699782 >> v2:172.21.2.113:3300/0 conn(0x7fd1280d1870 0x7fd1281b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:36.816 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:36.797 7fd12d427700 1 -- 172.21.2.102:0/2070699782 >> v1:172.21.2.113:6789/0 conn(0x7fd1281b0a10 legacy=0x7fd1280cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:27:36.816 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:36.797 7fd12d427700 1 -- 172.21.2.102:0/2070699782 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1280d1e90 con 0x7fd1280f0650 2021-02-10T11:27:36.816 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:36.797 7fd12d427700 1 --2- 172.21.2.102:0/2070699782 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd1280f0650 0x7fd1280cfb20 secure :-1 s=READY pgs=1757 cs=0 l=1 rev1=1 rx=0x7fd120006f00 tx=0x7fd120009f10).ready entity=mon.0 client_cookie=95ab89688ae896c6 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:36.816 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:36.797 7fd1277fe700 1 -- 172.21.2.102:0/2070699782 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd12000b3b0 con 0x7fd1280f0650 2021-02-10T11:27:36.816 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:36.798 7fd1277fe700 1 -- 172.21.2.102:0/2070699782 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd12000b630 con 0x7fd1280f0650 2021-02-10T11:27:36.816 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:36.798 7fd1277fe700 1 -- 172.21.2.102:0/2070699782 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd12000ba90 con 0x7fd1280f0650 2021-02-10T11:27:36.816 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:36.798 7fd12ee0a700 1 -- 172.21.2.102:0/2070699782 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd1280f0650 msgr2=0x7fd1280cfb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:36.817 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:36.798 7fd12ee0a700 1 --2- 172.21.2.102:0/2070699782 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd1280f0650 0x7fd1280cfb20 secure :-1 s=READY pgs=1757 cs=0 l=1 rev1=1 rx=0x7fd120006f00 tx=0x7fd120009f10).stop 2021-02-10T11:27:36.817 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:36.798 7fd12ee0a700 1 -- 172.21.2.102:0/2070699782 shutdown_connections 2021-02-10T11:27:36.817 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:36.798 7fd12ee0a700 1 --2- 172.21.2.102:0/2070699782 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd1280f0650 0x7fd1280cfb20 unknown :-1 s=CLOSED pgs=1757 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:36.817 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:36.798 7fd12ee0a700 1 --2- 172.21.2.102:0/2070699782 >> v2:172.21.2.113:3300/0 conn(0x7fd1280d1870 0x7fd1281b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:36.817 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:36.798 7fd12ee0a700 1 -- 172.21.2.102:0/2070699782 >> 172.21.2.102:0/2070699782 conn(0x7fd1280ce450 msgr2=0x7fd1280d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:36.817 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:36.798 7fd12ee0a700 1 -- 172.21.2.102:0/2070699782 shutdown_connections 2021-02-10T11:27:36.817 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:36.798 7fd12ee0a700 1 -- 172.21.2.102:0/2070699782 wait complete. 2021-02-10T11:27:36.817 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:36.813 7fd12ee0a700 1 Processor -- start 2021-02-10T11:27:36.817 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:36.813 7fd12ee0a700 1 -- start start 2021-02-10T11:27:36.817 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:36.814 7fd12ee0a700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd1280b0ad0 0x7fd128091f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:36.817 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:36.814 7fd12ee0a700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd1281a8850 0x7fd1281a8c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:36.817 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:36.814 7fd12ee0a700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd1281a6fa0 0x7fd1281a73a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:36.817 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:36.814 7fd12ee0a700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1281abfe0 con 0x7fd1280b0ad0 2021-02-10T11:27:36.817 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:36.814 7fd12ee0a700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1281a9c40 con 0x7fd1281a6fa0 2021-02-10T11:27:36.818 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:36.814 7fd12ee0a700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1281a9730 con 0x7fd1281a8850 2021-02-10T11:27:36.818 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:36.814 7fd12cc26700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd1280b0ad0 0x7fd128091f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:36.818 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:36.814 7fd12cc26700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd1280b0ad0 0x7fd128091f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:59634/0 (socket says 172.21.2.102:59634) 2021-02-10T11:27:36.818 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:36.814 7fd12cc26700 1 -- 172.21.2.102:0/3464353341 learned_addr learned my addr 172.21.2.102:0/3464353341 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:27:36.818 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:36.814 7fd12cc26700 1 -- 172.21.2.102:0/3464353341 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd1281a8850 msgr2=0x7fd1281a8c50 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:27:36.818 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:36.814 7fd12cc26700 1 --2- 172.21.2.102:0/3464353341 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd1281a8850 0x7fd1281a8c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:36.818 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:36.814 7fd12cc26700 1 -- 172.21.2.102:0/3464353341 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd1281a6fa0 msgr2=0x7fd1281a73a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:36.818 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:36.814 7fd12cc26700 1 --2- 172.21.2.102:0/3464353341 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd1281a6fa0 0x7fd1281a73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:36.818 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:36.814 7fd12cc26700 1 -- 172.21.2.102:0/3464353341 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1281a9270 con 0x7fd1280b0ad0 2021-02-10T11:27:36.818 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:36.814 7fd12cc26700 1 --2- 172.21.2.102:0/3464353341 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd1280b0ad0 0x7fd128091f30 secure :-1 s=READY pgs=1758 cs=0 l=1 rev1=1 rx=0x7fd110006f00 tx=0x7fd11000b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:36.818 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:36.815 7fd1257fa700 1 -- 172.21.2.102:0/3464353341 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd110006b90 con 0x7fd1280b0ad0 2021-02-10T11:27:36.819 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:36.815 7fd1257fa700 1 -- 172.21.2.102:0/3464353341 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd11000bd30 con 0x7fd1280b0ad0 2021-02-10T11:27:36.819 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:36.815 7fd1257fa700 1 -- 172.21.2.102:0/3464353341 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd1100020d0 con 0x7fd1280b0ad0 2021-02-10T11:27:36.819 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:36.815 7fd12ee0a700 1 -- 172.21.2.102:0/3464353341 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1281a9270 con 0x7fd1280b0ad0 2021-02-10T11:27:36.819 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:36.815 7fd12ee0a700 1 -- 172.21.2.102:0/3464353341 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1281a7cb0 con 0x7fd1280b0ad0 2021-02-10T11:27:36.820 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:36.815 7fd1257fa700 1 -- 172.21.2.102:0/3464353341 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd110007020 con 0x7fd1280b0ad0 2021-02-10T11:27:36.820 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:36.815 7fd1257fa700 1 --2- 172.21.2.102:0/3464353341 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd10c0186c0 0x7fd10c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:36.820 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:36.815 7fd1257fa700 1 -- 172.21.2.102:0/3464353341 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 4753+0+0 (secure 0 0 0) 0x7fd110032220 con 0x7fd1280b0ad0 2021-02-10T11:27:36.820 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:36.815 7fd127fff700 1 --2- 172.21.2.102:0/3464353341 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd10c0186c0 0x7fd10c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:36.820 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:36.816 7fd127fff700 1 --2- 172.21.2.102:0/3464353341 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd10c0186c0 0x7fd10c01abc0 crc :-1 s=READY pgs=793 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:36.846 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:36.846 7fd12ee0a700 1 -- 172.21.2.102:0/3464353341 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1280ce450 con 0x7fd1280b0ad0 2021-02-10T11:27:36.849 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:36.849 7fd1257fa700 1 -- 172.21.2.102:0/3464353341 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd11001c400 con 0x7fd1280b0ad0 2021-02-10T11:27:36.983 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:36.982 7fd12ee0a700 1 -- 172.21.2.102:0/3464353341 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fd1280ce450 con 0x7fd1280b0ad0 2021-02-10T11:27:36.984 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:36.982 7fd1257fa700 1 -- 172.21.2.102:0/3464353341 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v46) v1 ==== 80+0+4513 (secure 0 0 0) 0x7fd11001c400 con 0x7fd1280b0ad0 2021-02-10T11:27:36.984 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:27:36.985 INFO:teuthology.orchestra.run.gibba002.stdout:{ 2021-02-10T11:27:36.985 INFO:teuthology.orchestra.run.gibba002.stdout: "auth_dump": [ 2021-02-10T11:27:36.985 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:27:36.986 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "mds.a", 2021-02-10T11:27:36.986 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAevSNgsgyEARAAvr0cUjjCoEHLxaZSZ2HtVA==", 2021-02-10T11:27:36.986 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:27:36.987 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow", 2021-02-10T11:27:36.987 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:27:36.987 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:27:36.987 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:27:36.987 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:27:36.988 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:27:36.988 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:27:36.988 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "mds.b", 2021-02-10T11:27:36.988 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAevSNgwOFMMRAALfb1fbX56ao1TU8ClnTW1A==", 2021-02-10T11:27:36.988 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:27:36.988 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow", 2021-02-10T11:27:36.989 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:27:36.989 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:27:36.989 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:27:36.989 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:27:36.989 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:27:36.989 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:27:36.990 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.0", 2021-02-10T11:27:36.990 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAmvSNgj+bMMBAAD3HUD6fLiMAg+++iNzLJKQ==", 2021-02-10T11:27:36.990 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:27:36.990 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:27:36.991 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:27:36.991 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:27:36.991 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:27:36.991 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:27:36.991 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:27:36.992 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.1", 2021-02-10T11:27:36.992 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQApvSNgnvc+MRAAAOxqlNwiyLHqNITOrYeFKQ==", 2021-02-10T11:27:36.992 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:27:36.992 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:27:36.993 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:27:36.993 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:27:36.993 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:27:36.993 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:27:36.994 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:27:36.996 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.2", 2021-02-10T11:27:36.996 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAsvSNgZ4SbMRAA76IOYWfbiAuBoM/4k81GxA==", 2021-02-10T11:27:36.996 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:27:36.997 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:27:36.997 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:27:36.997 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:27:36.997 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:27:36.997 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:27:36.997 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:27:36.997 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.3", 2021-02-10T11:27:36.997 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAvvSNgFk3rMRAAWCmKKyw0LDQ+UlAHZAITOw==", 2021-02-10T11:27:36.998 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:27:36.998 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:27:36.998 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:27:36.998 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:27:36.998 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:27:36.998 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:27:36.998 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:27:36.998 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.4", 2021-02-10T11:27:36.999 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQA4vSNgpKcNJBAApJMmvj5YxVHPDOmvmQONOg==", 2021-02-10T11:27:36.999 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:27:36.999 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:27:36.999 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:27:36.999 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:27:36.999 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:27:36.999 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:27:36.999 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:27:37.000 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.5", 2021-02-10T11:27:37.000 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQA7vSNgf2ujJBAAfkuYpyb3WTzc8V5kw+Wktg==", 2021-02-10T11:27:37.000 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:27:37.000 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:27:37.000 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:27:37.000 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:27:37.001 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:27:37.001 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:27:37.001 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:27:37.001 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.6", 2021-02-10T11:27:37.001 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQA+vSNgqw9uJRAAFMU0R+UxU9C3o/9byb3e0g==", 2021-02-10T11:27:37.001 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:27:37.001 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:27:37.001 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:27:37.002 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:27:37.002 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:27:37.002 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:27:37.002 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:27:37.002 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.7", 2021-02-10T11:27:37.002 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQBBvSNgvWSrJRAA2KaCHmLpzovRgYzLvyofTQ==", 2021-02-10T11:27:37.002 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:27:37.003 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:27:37.003 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:27:37.003 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:27:37.003 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:27:37.003 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:27:37.003 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:27:37.003 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "client.0", 2021-02-10T11:27:37.004 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAfvSNgZd9XIBAAvyrvl8fdKuHnJZfczfRoFw==", 2021-02-10T11:27:37.004 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:27:37.004 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow", 2021-02-10T11:27:37.004 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow r", 2021-02-10T11:27:37.004 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:27:37.004 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:27:37.004 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:27:37.004 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:27:37.005 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "client.1", 2021-02-10T11:27:37.005 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAfvSNgSyfXNhAAIduPnF7Vd/6HSkJ0MlcMSA==", 2021-02-10T11:27:37.005 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:27:37.005 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow", 2021-02-10T11:27:37.005 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow r", 2021-02-10T11:27:37.005 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:27:37.005 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:27:37.006 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:27:37.006 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:27:37.006 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "client.2", 2021-02-10T11:27:37.006 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAgvSNgiIG/ERAA9ZTamaTI7ZsPSnxBqjplCg==", 2021-02-10T11:27:37.006 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:27:37.006 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow", 2021-02-10T11:27:37.006 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow r", 2021-02-10T11:27:37.007 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:27:37.007 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:27:37.007 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:27:37.007 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:27:37.007 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "client.3", 2021-02-10T11:27:37.007 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAgvSNgOWpOKBAAlXsEB04Q/GvtL06PqRMLeA==", 2021-02-10T11:27:37.007 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:27:37.008 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow", 2021-02-10T11:27:37.008 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow r", 2021-02-10T11:27:37.008 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:27:37.008 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:27:37.008 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:27:37.008 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:27:37.008 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "client.admin", 2021-02-10T11:27:37.008 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAavSNgO3GQBhAAL+NVnYskQRKVWLgyDPY/xw==", 2021-02-10T11:27:37.009 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:27:37.009 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow *", 2021-02-10T11:27:37.009 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:27:37.009 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:27:37.009 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:27:37.009 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:27:37.010 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:27:37.010 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:27:37.010 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "mgr.x", 2021-02-10T11:27:37.010 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAdvSNgXZZgJhAA7dtBSGzH+DsBwnddZpOBJA==", 2021-02-10T11:27:37.010 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:27:37.010 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow *", 2021-02-10T11:27:37.010 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow profile mgr", 2021-02-10T11:27:37.011 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:27:37.011 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:27:37.011 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:27:37.011 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:27:37.011 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "mgr.y", 2021-02-10T11:27:37.011 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAdvSNgVqQFDRAAwo6YEdWZCAr5GdOWVL/Qvg==", 2021-02-10T11:27:37.011 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:27:37.012 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow *", 2021-02-10T11:27:37.012 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow profile mgr", 2021-02-10T11:27:37.012 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:27:37.012 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:27:37.012 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:27:37.012 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:27:37.012 INFO:teuthology.orchestra.run.gibba002.stdout:} 2021-02-10T11:27:37.014 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:36.984 7fd12ee0a700 1 -- 172.21.2.102:0/3464353341 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd10c0186c0 msgr2=0x7fd10c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:37.014 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:36.984 7fd12ee0a700 1 --2- 172.21.2.102:0/3464353341 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd10c0186c0 0x7fd10c01abc0 crc :-1 s=READY pgs=793 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:37.014 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:36.984 7fd12ee0a700 1 -- 172.21.2.102:0/3464353341 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd1280b0ad0 msgr2=0x7fd128091f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:37.015 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:36.984 7fd12ee0a700 1 --2- 172.21.2.102:0/3464353341 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd1280b0ad0 0x7fd128091f30 secure :-1 s=READY pgs=1758 cs=0 l=1 rev1=1 rx=0x7fd110006f00 tx=0x7fd11000b940).stop 2021-02-10T11:27:37.015 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:36.985 7fd12ee0a700 1 -- 172.21.2.102:0/3464353341 shutdown_connections 2021-02-10T11:27:37.015 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:36.985 7fd12ee0a700 1 --2- 172.21.2.102:0/3464353341 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd10c0186c0 0x7fd10c01abc0 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:37.015 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:36.985 7fd12ee0a700 1 --2- 172.21.2.102:0/3464353341 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd1281a6fa0 0x7fd1281a73a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:37.015 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:36.985 7fd12ee0a700 1 --2- 172.21.2.102:0/3464353341 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd1281a8850 0x7fd1281a8c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:37.015 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:36.985 7fd12ee0a700 1 --2- 172.21.2.102:0/3464353341 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd1280b0ad0 0x7fd128091f30 unknown :-1 s=CLOSED pgs=1758 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:37.016 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:36.985 7fd12ee0a700 1 -- 172.21.2.102:0/3464353341 >> 172.21.2.102:0/3464353341 conn(0x7fd12819f8f0 msgr2=0x7fd1281a5f80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:37.016 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:36.985 7fd12ee0a700 1 -- 172.21.2.102:0/3464353341 shutdown_connections 2021-02-10T11:27:37.016 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:36.985 7fd12ee0a700 1 -- 172.21.2.102:0/3464353341 wait complete. 2021-02-10T11:27:37.017 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:27:37.452 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:37.435 7f905f36c700 1 Processor -- start 2021-02-10T11:27:37.452 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:37.435 7f905f36c700 1 -- start start 2021-02-10T11:27:37.452 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:37.435 7f905f36c700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f90580d1870 0x7f90581b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:37.452 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:37.435 7f905f36c700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90580f0b10 con 0x7f90581b0a10 2021-02-10T11:27:37.452 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:37.435 7f905f36c700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90580d0310 con 0x7f90580f0650 2021-02-10T11:27:37.452 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:37.435 7f905f36c700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90580e5ed0 con 0x7f90580d1870 2021-02-10T11:27:37.453 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:37.436 7f905c987700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f90581b0a10 0x7f90580cf330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:51866/0 (socket says 172.21.2.102:51866) 2021-02-10T11:27:37.453 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:37.436 7f905c987700 1 -- 172.21.2.102:0/2129796059 learned_addr learned my addr 172.21.2.102:0/2129796059 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:27:37.453 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:37.436 7f904bfff700 1 -- 172.21.2.102:0/2129796059 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2322639744 0 0) 0x7f90540010b0 con 0x7f90581b0a10 2021-02-10T11:27:37.453 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:37.436 7f904bfff700 1 -- 172.21.2.102:0/2129796059 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9044002fe0 con 0x7f90581b0a10 2021-02-10T11:27:37.453 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:37.436 7f905d188700 1 --2- 172.21.2.102:0/2129796059 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f90580d1870 0x7f90581b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:37.453 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:37.436 7f904bfff700 1 -- 172.21.2.102:0/2129796059 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 302681651 0 0) 0x7f90540010b0 con 0x7f90581b0a10 2021-02-10T11:27:37.453 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:37.436 7f904bfff700 1 -- 172.21.2.102:0/2129796059 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f90580d1870 msgr2=0x7f90581b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:27:37.453 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:37.436 7f904bfff700 1 --2- 172.21.2.102:0/2129796059 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f90580d1870 0x7f90581b03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:37.453 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:37.436 7f904bfff700 1 -- 172.21.2.102:0/2129796059 >> v1:172.21.2.113:6789/0 conn(0x7f90580f0650 legacy=0x7f90580cfb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:37.453 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:37.436 7f904bfff700 1 -- 172.21.2.102:0/2129796059 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90580d0530 con 0x7f90581b0a10 2021-02-10T11:27:37.453 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:37.436 7f904bfff700 1 -- 172.21.2.102:0/2129796059 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7f90540015b0 con 0x7f90581b0a10 2021-02-10T11:27:37.453 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:37.436 7f904bfff700 1 -- 172.21.2.102:0/2129796059 >> v1:172.21.2.102:6789/0 conn(0x7f90581b0a10 legacy=0x7f90580cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:37.453 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:37.436 7f904bfff700 1 --2- 172.21.2.102:0/2129796059 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9044003b10 0x7f9044005f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:37.454 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:37.436 7f904bfff700 1 --2- 172.21.2.102:0/2129796059 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9044006820 0x7f9044008c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:37.454 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:37.436 7f904bfff700 1 --2- 172.21.2.102:0/2129796059 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f9044009510 0x7f904400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:37.454 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:37.436 7f904bfff700 1 -- 172.21.2.102:0/2129796059 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f904400c2c0 con 0x7f9044003b10 2021-02-10T11:27:37.454 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:37.436 7f904bfff700 1 -- 172.21.2.102:0/2129796059 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f904400c4e0 con 0x7f9044009510 2021-02-10T11:27:37.454 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:37.436 7f904bfff700 1 -- 172.21.2.102:0/2129796059 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f904400c700 con 0x7f9044006820 2021-02-10T11:27:37.454 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:37.436 7f905d188700 1 --2- 172.21.2.102:0/2129796059 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9044003b10 0x7f9044005f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:37.454 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:37.437 7f905c987700 1 --2- 172.21.2.102:0/2129796059 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9044006820 0x7f9044008c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:37.454 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:37.437 7f905d188700 1 -- 172.21.2.102:0/2129796059 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9044006820 msgr2=0x7f9044008c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:27:37.454 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:37.437 7f905d188700 1 --2- 172.21.2.102:0/2129796059 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9044006820 0x7f9044008c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:37.454 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:37.437 7f905d188700 1 -- 172.21.2.102:0/2129796059 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f9044009510 msgr2=0x7f904400ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:27:37.454 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:37.437 7f905d188700 1 --2- 172.21.2.102:0/2129796059 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f9044009510 0x7f904400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:37.454 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:37.437 7f905d188700 1 -- 172.21.2.102:0/2129796059 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9044050970 con 0x7f9044003b10 2021-02-10T11:27:37.454 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:37.437 7f905c987700 1 --2- 172.21.2.102:0/2129796059 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9044006820 0x7f9044008c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:27:37.455 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:37.437 7f905d188700 1 --2- 172.21.2.102:0/2129796059 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9044003b10 0x7f9044005f80 secure :-1 s=READY pgs=1760 cs=0 l=1 rev1=1 rx=0x7f904c009310 tx=0x7f904c009a90).ready entity=mon.0 client_cookie=fcdeea5f2a63fd9 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:37.455 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:37.437 7f904bfff700 1 -- 172.21.2.102:0/2129796059 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f904c00c0c0 con 0x7f9044003b10 2021-02-10T11:27:37.455 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:37.437 7f904bfff700 1 -- 172.21.2.102:0/2129796059 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f904c00c2c0 con 0x7f9044003b10 2021-02-10T11:27:37.455 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:37.437 7f904bfff700 1 -- 172.21.2.102:0/2129796059 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f904c00c7e0 con 0x7f9044003b10 2021-02-10T11:27:37.455 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:37.437 7f905f36c700 1 -- 172.21.2.102:0/2129796059 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9044003b10 msgr2=0x7f9044005f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:37.455 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:37.437 7f905f36c700 1 --2- 172.21.2.102:0/2129796059 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9044003b10 0x7f9044005f80 secure :-1 s=READY pgs=1760 cs=0 l=1 rev1=1 rx=0x7f904c009310 tx=0x7f904c009a90).stop 2021-02-10T11:27:37.455 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:37.437 7f905d989700 1 -- 172.21.2.102:0/2129796059 reap_dead start 2021-02-10T11:27:37.455 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:37.437 7f905f36c700 1 -- 172.21.2.102:0/2129796059 shutdown_connections 2021-02-10T11:27:37.455 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:37.437 7f905f36c700 1 -- 172.21.2.102:0/2129796059 >> 172.21.2.102:0/2129796059 conn(0x7f90580ce450 msgr2=0x7f90580d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:37.455 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:37.438 7f905f36c700 1 -- 172.21.2.102:0/2129796059 shutdown_connections 2021-02-10T11:27:37.455 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:37.438 7f905f36c700 1 -- 172.21.2.102:0/2129796059 wait complete. 2021-02-10T11:27:37.455 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:37.452 7f905f36c700 1 Processor -- start 2021-02-10T11:27:37.458 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:37.452 7f905f36c700 1 -- start start 2021-02-10T11:27:37.458 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:37.453 7f905f36c700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f905809bb70 0x7f90580b4fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:37.459 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:37.453 7f905f36c700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f90580b55e0 0x7f90580c1370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:37.459 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:37.453 7f905f36c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f90580b59e0 0x7f90580c1990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:37.459 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:37.453 7f905f36c700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90580b6010 con 0x7f90580b59e0 2021-02-10T11:27:37.459 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:37.453 7f905f36c700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90580b6230 con 0x7f905809bb70 2021-02-10T11:27:37.459 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:37.453 7f905f36c700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f905809bf70 con 0x7f90580b55e0 2021-02-10T11:27:37.459 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:37.453 7f905d989700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f90580b59e0 0x7f90580c1990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:37.459 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:37.453 7f905d989700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f90580b59e0 0x7f90580c1990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:59658/0 (socket says 172.21.2.102:59658) 2021-02-10T11:27:37.459 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:37.453 7f905d989700 1 -- 172.21.2.102:0/174310702 learned_addr learned my addr 172.21.2.102:0/174310702 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:27:37.459 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:37.453 7f905d989700 1 -- 172.21.2.102:0/174310702 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f90580b55e0 msgr2=0x7f90580c1370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:37.459 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:37.453 7f905d989700 1 --2- 172.21.2.102:0/174310702 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f90580b55e0 0x7f90580c1370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:37.460 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:37.453 7f905d989700 1 -- 172.21.2.102:0/174310702 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f905809bb70 msgr2=0x7f90580b4fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:37.460 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:37.453 7f905d989700 1 --2- 172.21.2.102:0/174310702 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f905809bb70 0x7f90580b4fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:37.460 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:37.453 7f905d989700 1 -- 172.21.2.102:0/174310702 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90580c1fb0 con 0x7f90580b59e0 2021-02-10T11:27:37.460 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:37.453 7f905d989700 1 --2- 172.21.2.102:0/174310702 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f90580b59e0 0x7f90580c1990 secure :-1 s=READY pgs=1761 cs=0 l=1 rev1=1 rx=0x7f905000a920 tx=0x7f905000acc0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:37.460 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:37.454 7f9049ffb700 1 -- 172.21.2.102:0/174310702 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9050000ce0 con 0x7f90580b59e0 2021-02-10T11:27:37.460 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:37.454 7f9049ffb700 1 -- 172.21.2.102:0/174310702 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90500070b0 con 0x7f90580b59e0 2021-02-10T11:27:37.460 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:37.454 7f9049ffb700 1 -- 172.21.2.102:0/174310702 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f90500045a0 con 0x7f90580b59e0 2021-02-10T11:27:37.460 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:37.454 7f905f36c700 1 -- 172.21.2.102:0/174310702 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90580c1fb0 con 0x7f90580b59e0 2021-02-10T11:27:37.460 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:37.454 7f905f36c700 1 -- 172.21.2.102:0/174310702 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90580c1fb0 con 0x7f90580b59e0 2021-02-10T11:27:37.460 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:37.454 7f9049ffb700 1 -- 172.21.2.102:0/174310702 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f9050007290 con 0x7f90580b59e0 2021-02-10T11:27:37.460 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:37.454 7f9049ffb700 1 --2- 172.21.2.102:0/174310702 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f90380186c0 0x7f903801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:37.460 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:37.454 7f9049ffb700 1 -- 172.21.2.102:0/174310702 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 4753+0+0 (secure 0 0 0) 0x7f90500325b0 con 0x7f90580b59e0 2021-02-10T11:27:37.460 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:37.455 7f905d188700 1 --2- 172.21.2.102:0/174310702 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f90380186c0 0x7f903801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:37.461 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:37.455 7f905d188700 1 --2- 172.21.2.102:0/174310702 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f90380186c0 0x7f903801abc0 crc :-1 s=READY pgs=794 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:37.486 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:37.486 7f905f36c700 1 -- 172.21.2.102:0/174310702 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90580ebcd0 con 0x7f90580b59e0 2021-02-10T11:27:37.488 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:37.488 7f9049ffb700 1 -- 172.21.2.102:0/174310702 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f9050007c70 con 0x7f90580b59e0 2021-02-10T11:27:37.612 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:37.612 7f905f36c700 1 -- 172.21.2.102:0/174310702 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f90580ebcd0 con 0x7f90580b59e0 2021-02-10T11:27:37.613 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:37.613 7f9049ffb700 1 -- 172.21.2.102:0/174310702 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) v1 ==== 74+0+11145 (secure 0 0 0) 0x7f9050007270 con 0x7f90580b59e0 2021-02-10T11:27:37.614 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:27:37.614 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":81,"fsid":"68db30ea-dbce-4aef-a6df-f01c6fea0513","created":"2021-02-10 11:03:06.331557","modified":"2021-02-10 11:27:36.362021","last_up_change":"2021-02-10 11:03:32.690562","last_in_change":"2021-02-10 11:03:32.690562","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:03:37.322519","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"be691b4d-a039-408a-af80-9d53f02b6f1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6818","nonce":36088},{"type":"v1","addr":"172.21.2.102:6820","nonce":36088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6822","nonce":36088},{"type":"v1","addr":"172.21.2.102:6824","nonce":36088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6830","nonce":36088},{"type":"v1","addr":"172.21.2.102:6832","nonce":36088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6826","nonce":36088},{"type":"v1","addr":"172.21.2.102:6828","nonce":36088}]},"public_addr":"172.21.2.102:6820/36088","cluster_addr":"172.21.2.102:6824/36088","heartbeat_back_addr":"172.21.2.102:6832/36088","heartbeat_front_addr":"172.21.2.102:6828/36088","state":["exists","up"]},{"osd":1,"uuid":"64e3ccc8-9c69-4d7c-865e-af8e83822635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6819","nonce":36087},{"type":"v1","addr":"172.21.2.102:6821","nonce":36087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6823","nonce":36087},{"type":"v1","addr":"172.21.2.102:6825","nonce":36087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6831","nonce":36087},{"type":"v1","addr":"172.21.2.102:6833","nonce":36087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6827","nonce":36087},{"type":"v1","addr":"172.21.2.102:6829","nonce":36087}]},"public_addr":"172.21.2.102:6821/36087","cluster_addr":"172.21.2.102:6825/36087","heartbeat_back_addr":"172.21.2.102:6833/36087","heartbeat_front_addr":"172.21.2.102:6829/36087","state":["exists","up"]},{"osd":2,"uuid":"153700af-747f-481f-bdf4-d6072d3a842f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6810","nonce":36089},{"type":"v1","addr":"172.21.2.102:6811","nonce":36089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6812","nonce":36089},{"type":"v1","addr":"172.21.2.102:6813","nonce":36089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6816","nonce":36089},{"type":"v1","addr":"172.21.2.102:6817","nonce":36089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6814","nonce":36089},{"type":"v1","addr":"172.21.2.102:6815","nonce":36089}]},"public_addr":"172.21.2.102:6811/36089","cluster_addr":"172.21.2.102:6813/36089","heartbeat_back_addr":"172.21.2.102:6817/36089","heartbeat_front_addr":"172.21.2.102:6815/36089","state":["exists","up"]},{"osd":3,"uuid":"cbc678f4-83a9-45e6-8830-2f7c232b1ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6802","nonce":36090},{"type":"v1","addr":"172.21.2.102:6803","nonce":36090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6804","nonce":36090},{"type":"v1","addr":"172.21.2.102:6805","nonce":36090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6808","nonce":36090},{"type":"v1","addr":"172.21.2.102:6809","nonce":36090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6806","nonce":36090},{"type":"v1","addr":"172.21.2.102:6807","nonce":36090}]},"public_addr":"172.21.2.102:6803/36090","cluster_addr":"172.21.2.102:6805/36090","heartbeat_back_addr":"172.21.2.102:6809/36090","heartbeat_front_addr":"172.21.2.102:6807/36090","state":["exists","up"]},{"osd":4,"uuid":"c01e4008-b258-4f10-afd2-9e7f37214351","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6808","nonce":36368},{"type":"v1","addr":"172.21.2.113:6811","nonce":36368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6814","nonce":36368},{"type":"v1","addr":"172.21.2.113:6818","nonce":36368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6826","nonce":36368},{"type":"v1","addr":"172.21.2.113:6828","nonce":36368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6820","nonce":36368},{"type":"v1","addr":"172.21.2.113:6824","nonce":36368}]},"public_addr":"172.21.2.113:6811/36368","cluster_addr":"172.21.2.113:6818/36368","heartbeat_back_addr":"172.21.2.113:6828/36368","heartbeat_front_addr":"172.21.2.113:6824/36368","state":["exists","up"]},{"osd":5,"uuid":"551f6007-daf8-4a98-8ee0-c2b16bcccb5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6802","nonce":36366},{"type":"v1","addr":"172.21.2.113:6805","nonce":36366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6807","nonce":36366},{"type":"v1","addr":"172.21.2.113:6810","nonce":36366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6819","nonce":36366},{"type":"v1","addr":"172.21.2.113:6822","nonce":36366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6812","nonce":36366},{"type":"v1","addr":"172.21.2.113:6815","nonce":36366}]},"public_addr":"172.21.2.113:6805/36366","cluster_addr":"172.21.2.113:6810/36366","heartbeat_back_addr":"172.21.2.113:6822/36366","heartbeat_front_addr":"172.21.2.113:6815/36366","state":["exists","up"]},{"osd":6,"uuid":"bf67c040-0e50-48be-b102-c3943bc9ebef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6800","nonce":36367},{"type":"v1","addr":"172.21.2.113:6801","nonce":36367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6803","nonce":36367},{"type":"v1","addr":"172.21.2.113:6804","nonce":36367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6813","nonce":36367},{"type":"v1","addr":"172.21.2.113:6816","nonce":36367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6806","nonce":36367},{"type":"v1","addr":"172.21.2.113:6809","nonce":36367}]},"public_addr":"172.21.2.113:6801/36367","cluster_addr":"172.21.2.113:6804/36367","heartbeat_back_addr":"172.21.2.113:6816/36367","heartbeat_front_addr":"172.21.2.113:6809/36367","state":["exists","up"]},{"osd":7,"uuid":"f5461f80-018d-428f-be89-2eb2bcbe7bb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6817","nonce":36365},{"type":"v1","addr":"172.21.2.113:6821","nonce":36365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6823","nonce":36365},{"type":"v1","addr":"172.21.2.113:6825","nonce":36365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6830","nonce":36365},{"type":"v1","addr":"172.21.2.113:6831","nonce":36365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6827","nonce":36365},{"type":"v1","addr":"172.21.2.113:6829","nonce":36365}]},"public_addr":"172.21.2.113:6821/36365","cluster_addr":"172.21.2.113:6825/36365","heartbeat_back_addr":"172.21.2.113:6831/36365","heartbeat_front_addr":"172.21.2.113:6829/36365","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:27:37.615 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:37.614 7f905f36c700 1 -- 172.21.2.102:0/174310702 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f90380186c0 msgr2=0x7f903801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:37.615 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:37.614 7f905f36c700 1 --2- 172.21.2.102:0/174310702 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f90380186c0 0x7f903801abc0 crc :-1 s=READY pgs=794 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:37.615 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:37.614 7f905f36c700 1 -- 172.21.2.102:0/174310702 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f90580b59e0 msgr2=0x7f90580c1990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:37.615 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:37.614 7f905f36c700 1 --2- 172.21.2.102:0/174310702 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f90580b59e0 0x7f90580c1990 secure :-1 s=READY pgs=1761 cs=0 l=1 rev1=1 rx=0x7f905000a920 tx=0x7f905000acc0).stop 2021-02-10T11:27:37.616 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:37.615 7f905f36c700 1 -- 172.21.2.102:0/174310702 shutdown_connections 2021-02-10T11:27:37.616 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:37.615 7f905f36c700 1 --2- 172.21.2.102:0/174310702 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f90580b59e0 0x7f90580c1990 unknown :-1 s=CLOSED pgs=1761 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:37.616 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:37.615 7f905f36c700 1 --2- 172.21.2.102:0/174310702 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f90580b55e0 0x7f90580c1370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:37.617 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:37.615 7f905f36c700 1 --2- 172.21.2.102:0/174310702 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f90380186c0 0x7f903801abc0 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:37.617 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:37.615 7f905f36c700 1 --2- 172.21.2.102:0/174310702 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f905809bb70 0x7f90580b4fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:37.617 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:37.615 7f905f36c700 1 -- 172.21.2.102:0/174310702 >> 172.21.2.102:0/174310702 conn(0x7f90580ac990 msgr2=0x7f9058091b30 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:37.617 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:37.615 7f905f36c700 1 -- 172.21.2.102:0/174310702 shutdown_connections 2021-02-10T11:27:37.617 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:37.615 7f905f36c700 1 -- 172.21.2.102:0/174310702 wait complete. 2021-02-10T11:27:37.625 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:03:37.322519', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2021-02-10T11:27:37.626 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:27:38.062 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:38.044 7f7b74d04700 1 Processor -- start 2021-02-10T11:27:38.062 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:38.044 7f7b74d04700 1 -- start start 2021-02-10T11:27:38.062 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:38.045 7f7b74d04700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f7b700d1870 0x7f7b701b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:38.062 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:38.045 7f7b74d04700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7b701b0ae0 0x7f7b70108940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:38.062 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:38.045 7f7b74d04700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b700fe1c0 con 0x7f7b700d1c70 2021-02-10T11:27:38.063 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:38.045 7f7b74d04700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b70109a60 con 0x7f7b700d1870 2021-02-10T11:27:38.063 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:38.045 7f7b74d04700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b70109750 con 0x7f7b701b0ae0 2021-02-10T11:27:38.063 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:38.045 7f7b6f314700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f7b700d1c70 0x7f7b70108f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:51890/0 (socket says 172.21.2.102:51890) 2021-02-10T11:27:38.063 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:38.045 7f7b6f314700 1 -- 172.21.2.102:0/213837031 learned_addr learned my addr 172.21.2.102:0/213837031 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:27:38.063 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:38.045 7f7b6db11700 1 -- 172.21.2.102:0/213837031 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1400087092 0 0) 0x7f7b68001500 con 0x7f7b700d1c70 2021-02-10T11:27:38.063 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:38.045 7f7b6db11700 1 -- 172.21.2.102:0/213837031 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b5c002fe0 con 0x7f7b700d1c70 2021-02-10T11:27:38.063 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:38.045 7f7b6e312700 1 --2- 172.21.2.102:0/213837031 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7b701b0ae0 0x7f7b70108940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:38.063 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:38.045 7f7b6eb13700 1 --2- 172.21.2.102:0/213837031 >> v2:172.21.2.113:3300/0 conn(0x7f7b700d1870 0x7f7b701b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:38.063 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:38.046 7f7b6db11700 1 -- 172.21.2.102:0/213837031 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3920146466 0 0) 0x7f7b68001500 con 0x7f7b700d1c70 2021-02-10T11:27:38.063 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:38.046 7f7b6db11700 1 -- 172.21.2.102:0/213837031 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7b701b0ae0 msgr2=0x7f7b70108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:27:38.063 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:38.046 7f7b6db11700 1 --2- 172.21.2.102:0/213837031 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7b701b0ae0 0x7f7b70108940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:38.063 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:38.046 7f7b6db11700 1 -- 172.21.2.102:0/213837031 >> v2:172.21.2.113:3300/0 conn(0x7f7b700d1870 msgr2=0x7f7b701b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:27:38.064 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:38.046 7f7b6db11700 1 --2- 172.21.2.102:0/213837031 >> v2:172.21.2.113:3300/0 conn(0x7f7b700d1870 0x7f7b701b04c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:38.064 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:38.046 7f7b6db11700 1 -- 172.21.2.102:0/213837031 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b70088070 con 0x7f7b700d1c70 2021-02-10T11:27:38.064 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:38.046 7f7b6db11700 1 -- 172.21.2.102:0/213837031 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7f7b68001a00 con 0x7f7b700d1c70 2021-02-10T11:27:38.064 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:38.046 7f7b6db11700 1 -- 172.21.2.102:0/213837031 >> v1:172.21.2.102:6789/0 conn(0x7f7b700d1c70 legacy=0x7f7b70108f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:38.064 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:38.046 7f7b6db11700 1 --2- 172.21.2.102:0/213837031 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7b5c003a60 0x7f7b5c005f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:38.064 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:38.046 7f7b6db11700 1 --2- 172.21.2.102:0/213837031 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7b5c04a810 0x7f7b5c04cca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:38.064 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:38.046 7f7b6db11700 1 --2- 172.21.2.102:0/213837031 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7b5c04d5a0 0x7f7b5c04fa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:38.064 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:38.046 7f7b6db11700 1 -- 172.21.2.102:0/213837031 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b5c0502e0 con 0x7f7b5c003a60 2021-02-10T11:27:38.064 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:38.046 7f7b6db11700 1 -- 172.21.2.102:0/213837031 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b5c050500 con 0x7f7b5c04a810 2021-02-10T11:27:38.064 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:38.046 7f7b6db11700 1 -- 172.21.2.102:0/213837031 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b5c050720 con 0x7f7b5c04d5a0 2021-02-10T11:27:38.064 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:38.046 7f7b6eb13700 1 --2- 172.21.2.102:0/213837031 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7b5c003a60 0x7f7b5c005f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:38.064 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:38.046 7f7b6e312700 1 --2- 172.21.2.102:0/213837031 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7b5c04a810 0x7f7b5c04cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:38.065 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:38.046 7f7b6eb13700 1 -- 172.21.2.102:0/213837031 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7b5c04d5a0 msgr2=0x7f7b5c04fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:27:38.065 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:38.046 7f7b6eb13700 1 --2- 172.21.2.102:0/213837031 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7b5c04d5a0 0x7f7b5c04fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:38.065 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:38.046 7f7b6eb13700 1 -- 172.21.2.102:0/213837031 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7b5c04a810 msgr2=0x7f7b5c04cca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:27:38.065 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:38.046 7f7b6eb13700 1 --2- 172.21.2.102:0/213837031 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7b5c04a810 0x7f7b5c04cca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:38.065 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:38.046 7f7b6eb13700 1 -- 172.21.2.102:0/213837031 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b5c0509c0 con 0x7f7b5c003a60 2021-02-10T11:27:38.065 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:38.046 7f7b6e312700 1 --2- 172.21.2.102:0/213837031 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7b5c04a810 0x7f7b5c04cca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:27:38.065 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:38.046 7f7b6eb13700 1 --2- 172.21.2.102:0/213837031 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7b5c003a60 0x7f7b5c005f60 secure :-1 s=READY pgs=1763 cs=0 l=1 rev1=1 rx=0x7f7b5800c8f0 tx=0x7f7b5800cc90).ready entity=mon.0 client_cookie=9a541f99a87f5a5d server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:38.065 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:38.047 7f7b6db11700 1 -- 172.21.2.102:0/213837031 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7b5800cdd0 con 0x7f7b5c003a60 2021-02-10T11:27:38.065 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:38.047 7f7b6db11700 1 -- 172.21.2.102:0/213837031 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b58009310 con 0x7f7b5c003a60 2021-02-10T11:27:38.065 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:38.047 7f7b6db11700 1 -- 172.21.2.102:0/213837031 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7b580097f0 con 0x7f7b5c003a60 2021-02-10T11:27:38.065 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:38.047 7f7b74d04700 1 -- 172.21.2.102:0/213837031 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7b5c003a60 msgr2=0x7f7b5c005f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:38.066 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:38.047 7f7b74d04700 1 --2- 172.21.2.102:0/213837031 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7b5c003a60 0x7f7b5c005f60 secure :-1 s=READY pgs=1763 cs=0 l=1 rev1=1 rx=0x7f7b5800c8f0 tx=0x7f7b5800cc90).stop 2021-02-10T11:27:38.066 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:38.047 7f7b6f314700 1 -- 172.21.2.102:0/213837031 reap_dead start 2021-02-10T11:27:38.066 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:38.047 7f7b74d04700 1 -- 172.21.2.102:0/213837031 shutdown_connections 2021-02-10T11:27:38.066 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:38.047 7f7b74d04700 1 -- 172.21.2.102:0/213837031 >> 172.21.2.102:0/213837031 conn(0x7f7b700ce420 msgr2=0x7f7b700d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:38.066 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:38.047 7f7b74d04700 1 -- 172.21.2.102:0/213837031 shutdown_connections 2021-02-10T11:27:38.066 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:38.047 7f7b74d04700 1 -- 172.21.2.102:0/213837031 wait complete. 2021-02-10T11:27:38.066 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:38.062 7f7b74d04700 1 Processor -- start 2021-02-10T11:27:38.066 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:38.062 7f7b74d04700 1 -- start start 2021-02-10T11:27:38.068 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:38.062 7f7b74d04700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7b70085bc0 0x7f7b700863e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:38.068 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:38.062 7f7b74d04700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7b700954c0 0x7f7b70096300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:38.068 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:38.062 7f7b74d04700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7b700958c0 0x7f7b70096920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:38.068 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:38.062 7f7b74d04700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b70095d80 con 0x7f7b70085bc0 2021-02-10T11:27:38.068 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:38.062 7f7b74d04700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b700bfa60 con 0x7f7b700954c0 2021-02-10T11:27:38.068 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:38.062 7f7b74d04700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b70085fc0 con 0x7f7b700958c0 2021-02-10T11:27:38.068 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:38.062 7f7b6eb13700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7b70085bc0 0x7f7b700863e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:38.068 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:38.063 7f7b6eb13700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7b70085bc0 0x7f7b700863e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:59674/0 (socket says 172.21.2.102:59674) 2021-02-10T11:27:38.068 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:38.063 7f7b6eb13700 1 -- 172.21.2.102:0/2565756912 learned_addr learned my addr 172.21.2.102:0/2565756912 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:27:38.068 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:38.063 7f7b6eb13700 1 -- 172.21.2.102:0/2565756912 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7b700958c0 msgr2=0x7f7b70096920 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:27:38.068 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:38.063 7f7b6eb13700 1 --2- 172.21.2.102:0/2565756912 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7b700958c0 0x7f7b70096920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:38.068 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:38.063 7f7b6eb13700 1 -- 172.21.2.102:0/2565756912 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7b700954c0 msgr2=0x7f7b70096300 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:27:38.069 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:38.063 7f7b6eb13700 1 --2- 172.21.2.102:0/2565756912 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7b700954c0 0x7f7b70096300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:38.069 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:38.063 7f7b6eb13700 1 -- 172.21.2.102:0/2565756912 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b70086c10 con 0x7f7b70085bc0 2021-02-10T11:27:38.069 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:38.063 7f7b6eb13700 1 --2- 172.21.2.102:0/2565756912 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7b70085bc0 0x7f7b700863e0 secure :-1 s=READY pgs=1764 cs=0 l=1 rev1=1 rx=0x7f7b5800c2f0 tx=0x7f7b5800b300).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:38.069 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:38.063 7f7b677fe700 1 -- 172.21.2.102:0/2565756912 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7b5800c870 con 0x7f7b70085bc0 2021-02-10T11:27:38.069 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:38.063 7f7b677fe700 1 -- 172.21.2.102:0/2565756912 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b5800caf0 con 0x7f7b70085bc0 2021-02-10T11:27:38.069 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:38.063 7f7b677fe700 1 -- 172.21.2.102:0/2565756912 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7b5800b6f0 con 0x7f7b70085bc0 2021-02-10T11:27:38.069 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:38.063 7f7b74d04700 1 -- 172.21.2.102:0/2565756912 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b70086c10 con 0x7f7b70085bc0 2021-02-10T11:27:38.069 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:38.063 7f7b74d04700 1 -- 172.21.2.102:0/2565756912 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b70086c10 con 0x7f7b70085bc0 2021-02-10T11:27:38.069 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:38.064 7f7b677fe700 1 -- 172.21.2.102:0/2565756912 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f7b58000c90 con 0x7f7b70085bc0 2021-02-10T11:27:38.069 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:38.064 7f7b677fe700 1 --2- 172.21.2.102:0/2565756912 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7b540186c0 0x7f7b5401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:38.069 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:38.064 7f7b677fe700 1 -- 172.21.2.102:0/2565756912 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 4753+0+0 (secure 0 0 0) 0x7f7b58032310 con 0x7f7b70085bc0 2021-02-10T11:27:38.069 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:38.064 7f7b6e312700 1 --2- 172.21.2.102:0/2565756912 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7b540186c0 0x7f7b5401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:38.069 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:38.064 7f7b6e312700 1 --2- 172.21.2.102:0/2565756912 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7b540186c0 0x7f7b5401abc0 crc :-1 s=READY pgs=795 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:38.095 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:38.095 7f7b74d04700 1 -- 172.21.2.102:0/2565756912 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b700fe1c0 con 0x7f7b70085bc0 2021-02-10T11:27:38.097 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:38.097 7f7b677fe700 1 -- 172.21.2.102:0/2565756912 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f7b5802cfc0 con 0x7f7b70085bc0 2021-02-10T11:27:38.226 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:38.225 7f7b74d04700 1 -- 172.21.2.102:0/2565756912 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f7b700fe1c0 con 0x7f7b70085bc0 2021-02-10T11:27:38.226 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:38.226 7f7b677fe700 1 -- 172.21.2.102:0/2565756912 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v81) v1 ==== 92+0+10 (secure 0 0 0) 0x7f7b5802cfc0 con 0x7f7b70085bc0 2021-02-10T11:27:38.227 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:27:38.229 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:38.228 7f7b74d04700 1 -- 172.21.2.102:0/2565756912 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7b540186c0 msgr2=0x7f7b5401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:38.229 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:38.228 7f7b74d04700 1 --2- 172.21.2.102:0/2565756912 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7b540186c0 0x7f7b5401abc0 crc :-1 s=READY pgs=795 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:38.229 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:38.228 7f7b74d04700 1 -- 172.21.2.102:0/2565756912 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7b70085bc0 msgr2=0x7f7b700863e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:38.229 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:38.228 7f7b74d04700 1 --2- 172.21.2.102:0/2565756912 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7b70085bc0 0x7f7b700863e0 secure :-1 s=READY pgs=1764 cs=0 l=1 rev1=1 rx=0x7f7b5800c2f0 tx=0x7f7b5800b300).stop 2021-02-10T11:27:38.229 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:38.228 7f7b74d04700 1 -- 172.21.2.102:0/2565756912 shutdown_connections 2021-02-10T11:27:38.230 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:38.228 7f7b74d04700 1 --2- 172.21.2.102:0/2565756912 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7b700958c0 0x7f7b70096920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:38.230 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:38.228 7f7b74d04700 1 --2- 172.21.2.102:0/2565756912 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7b540186c0 0x7f7b5401abc0 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:38.230 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:38.228 7f7b74d04700 1 --2- 172.21.2.102:0/2565756912 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7b700954c0 0x7f7b70096300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:38.230 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:38.228 7f7b74d04700 1 --2- 172.21.2.102:0/2565756912 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7b70085bc0 0x7f7b700863e0 unknown :-1 s=CLOSED pgs=1764 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:38.230 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:38.228 7f7b74d04700 1 -- 172.21.2.102:0/2565756912 >> 172.21.2.102:0/2565756912 conn(0x7f7b700bef00 msgr2=0x7f7b7009d1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:38.231 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:38.228 7f7b74d04700 1 -- 172.21.2.102:0/2565756912 shutdown_connections 2021-02-10T11:27:38.231 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:38.228 7f7b74d04700 1 -- 172.21.2.102:0/2565756912 wait complete. 2021-02-10T11:27:38.237 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --format=json-pretty osd lspools 2021-02-10T11:27:38.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:38.648 7f35ccb2f700 1 Processor -- start 2021-02-10T11:27:38.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:38.648 7f35ccb2f700 1 -- start start 2021-02-10T11:27:38.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:38.649 7f35ccb2f700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f35c80d1870 0x7f35c81b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:38.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:38.649 7f35ccb2f700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35c80f0b10 con 0x7f35c81b0a10 2021-02-10T11:27:38.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:38.649 7f35ccb2f700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35c80d0310 con 0x7f35c80f0650 2021-02-10T11:27:38.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:38.649 7f35ccb2f700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35c80d0530 con 0x7f35c80d1870 2021-02-10T11:27:38.666 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:38.649 7f35c685e700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f35c80d1870 0x7f35c81b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:38.666 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:38.649 7f35c685e700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f35c80d1870 0x7f35c81b03f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:59682/0 (socket says 172.21.2.102:59682) 2021-02-10T11:27:38.666 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:38.649 7f35c685e700 1 -- 172.21.2.102:0/2067300637 learned_addr learned my addr 172.21.2.102:0/2067300637 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:27:38.666 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:38.649 7f35c685e700 1 -- 172.21.2.102:0/2067300637 >> v1:172.21.2.113:6789/0 conn(0x7f35c80f0650 legacy=0x7f35c80cfb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:27:38.666 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:38.650 7f35c685e700 1 -- 172.21.2.102:0/2067300637 >> v1:172.21.2.102:6789/0 conn(0x7f35c81b0a10 legacy=0x7f35c80cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:38.666 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:38.650 7f35c685e700 1 -- 172.21.2.102:0/2067300637 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35c80d1c70 con 0x7f35c80d1870 2021-02-10T11:27:38.666 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:38.650 7f35c685e700 1 --2- 172.21.2.102:0/2067300637 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f35c80d1870 0x7f35c81b03f0 secure :-1 s=READY pgs=1766 cs=0 l=1 rev1=1 rx=0x7f35b0006700 tx=0x7f35b0009b80).ready entity=mon.0 client_cookie=e87399d95928444c server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:38.666 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:38.650 7f35c585c700 1 -- 172.21.2.102:0/2067300637 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f35b000c0c0 con 0x7f35c80d1870 2021-02-10T11:27:38.666 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:38.650 7f35c585c700 1 -- 172.21.2.102:0/2067300637 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35b000c340 con 0x7f35c80d1870 2021-02-10T11:27:38.666 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:38.650 7f35c585c700 1 -- 172.21.2.102:0/2067300637 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f35b000c860 con 0x7f35c80d1870 2021-02-10T11:27:38.666 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:38.650 7f35ccb2f700 1 -- 172.21.2.102:0/2067300637 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f35c80d1870 msgr2=0x7f35c81b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:38.666 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:38.650 7f35ccb2f700 1 --2- 172.21.2.102:0/2067300637 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f35c80d1870 0x7f35c81b03f0 secure :-1 s=READY pgs=1766 cs=0 l=1 rev1=1 rx=0x7f35b0006700 tx=0x7f35b0009b80).stop 2021-02-10T11:27:38.667 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:38.651 7f35ccb2f700 1 -- 172.21.2.102:0/2067300637 shutdown_connections 2021-02-10T11:27:38.667 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:38.651 7f35ccb2f700 1 --2- 172.21.2.102:0/2067300637 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f35c80d1870 0x7f35c81b03f0 unknown :-1 s=CLOSED pgs=1766 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:38.667 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:38.651 7f35ccb2f700 1 -- 172.21.2.102:0/2067300637 >> 172.21.2.102:0/2067300637 conn(0x7f35c80ce450 msgr2=0x7f35c80d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:38.667 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:38.651 7f35ccb2f700 1 -- 172.21.2.102:0/2067300637 shutdown_connections 2021-02-10T11:27:38.667 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:38.651 7f35ccb2f700 1 -- 172.21.2.102:0/2067300637 wait complete. 2021-02-10T11:27:38.668 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:38.665 7f35ccb2f700 1 Processor -- start 2021-02-10T11:27:38.668 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:38.665 7f35ccb2f700 1 -- start start 2021-02-10T11:27:38.668 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:38.666 7f35ccb2f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f35c80a08f0 0x7f35c80a0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:38.669 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:38.666 7f35ccb2f700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f35c80b5230 0x7f35c80b5630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:38.669 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:38.666 7f35ccb2f700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f35c80b5c50 0x7f35c80a1b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:38.669 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:38.666 7f35ccb2f700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35c80a1600 con 0x7f35c80a08f0 2021-02-10T11:27:38.669 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:38.666 7f35ccb2f700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35c80b6050 con 0x7f35c80b5c50 2021-02-10T11:27:38.669 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:38.666 7f35ccb2f700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35c80b6240 con 0x7f35c80b5230 2021-02-10T11:27:38.669 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:38.666 7f35c685e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f35c80a08f0 0x7f35c80a0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:38.669 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:38.666 7f35c685e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f35c80a08f0 0x7f35c80a0cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:59688/0 (socket says 172.21.2.102:59688) 2021-02-10T11:27:38.669 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:38.666 7f35c685e700 1 -- 172.21.2.102:0/2825890743 learned_addr learned my addr 172.21.2.102:0/2825890743 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:27:38.669 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:38.666 7f35c685e700 1 -- 172.21.2.102:0/2825890743 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f35c80b5230 msgr2=0x7f35c80b5630 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:27:38.669 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:38.666 7f35c685e700 1 --2- 172.21.2.102:0/2825890743 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f35c80b5230 0x7f35c80b5630 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:38.669 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:38.666 7f35c685e700 1 -- 172.21.2.102:0/2825890743 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f35c80b5c50 msgr2=0x7f35c80a1b00 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:27:38.669 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:38.666 7f35c685e700 1 --2- 172.21.2.102:0/2825890743 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f35c80b5c50 0x7f35c80a1b00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:38.669 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:38.667 7f35c685e700 1 -- 172.21.2.102:0/2825890743 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35c80a2120 con 0x7f35c80a08f0 2021-02-10T11:27:38.670 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:38.667 7f35c685e700 1 --2- 172.21.2.102:0/2825890743 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f35c80a08f0 0x7f35c80a0cf0 secure :-1 s=READY pgs=1767 cs=0 l=1 rev1=1 rx=0x7f35b000b5e0 tx=0x7f35b000b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:38.670 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:38.667 7f35bf7fe700 1 -- 172.21.2.102:0/2825890743 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f35b000bd70 con 0x7f35c80a08f0 2021-02-10T11:27:38.670 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:38.667 7f35bf7fe700 1 -- 172.21.2.102:0/2825890743 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35b0002030 con 0x7f35c80a08f0 2021-02-10T11:27:38.670 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:38.667 7f35bf7fe700 1 -- 172.21.2.102:0/2825890743 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f35b0002550 con 0x7f35c80a08f0 2021-02-10T11:27:38.670 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:38.667 7f35ccb2f700 1 -- 172.21.2.102:0/2825890743 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35c80a2120 con 0x7f35c80a08f0 2021-02-10T11:27:38.670 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:38.667 7f35ccb2f700 1 -- 172.21.2.102:0/2825890743 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35c80a2120 con 0x7f35c80a08f0 2021-02-10T11:27:38.670 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:38.668 7f35bf7fe700 1 -- 172.21.2.102:0/2825890743 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f35b0000c90 con 0x7f35c80a08f0 2021-02-10T11:27:38.670 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:38.668 7f35bf7fe700 1 --2- 172.21.2.102:0/2825890743 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f35ac0186c0 0x7f35ac01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:38.670 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:38.668 7f35bf7fe700 1 -- 172.21.2.102:0/2825890743 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 4753+0+0 (secure 0 0 0) 0x7f35b00322c0 con 0x7f35c80a08f0 2021-02-10T11:27:38.670 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:38.668 7f35c605d700 1 --2- 172.21.2.102:0/2825890743 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f35ac0186c0 0x7f35ac01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:38.670 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:38.668 7f35c605d700 1 --2- 172.21.2.102:0/2825890743 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f35ac0186c0 0x7f35ac01abc0 crc :-1 s=READY pgs=796 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:38.699 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:38.699 7f35ccb2f700 1 -- 172.21.2.102:0/2825890743 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35c80e7de0 con 0x7f35c80a08f0 2021-02-10T11:27:38.701 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:38.701 7f35bf7fe700 1 -- 172.21.2.102:0/2825890743 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f35b002cf70 con 0x7f35c80a08f0 2021-02-10T11:27:38.826 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:38.825 7f35ccb2f700 1 -- 172.21.2.102:0/2825890743 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f35c80e7de0 con 0x7f35c80a08f0 2021-02-10T11:27:38.826 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:38.826 7f35bf7fe700 1 -- 172.21.2.102:0/2825890743 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v81) v1 ==== 84+0+64 (secure 0 0 0) 0x7f35b002cf70 con 0x7f35c80a08f0 2021-02-10T11:27:38.826 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:27:38.827 INFO:teuthology.orchestra.run.gibba002.stdout:[ 2021-02-10T11:27:38.827 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:27:38.827 INFO:teuthology.orchestra.run.gibba002.stdout: "poolnum": 1, 2021-02-10T11:27:38.827 INFO:teuthology.orchestra.run.gibba002.stdout: "poolname": "rbd" 2021-02-10T11:27:38.827 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:27:38.827 INFO:teuthology.orchestra.run.gibba002.stdout:] 2021-02-10T11:27:38.828 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:38.828 7f35ccb2f700 1 -- 172.21.2.102:0/2825890743 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f35ac0186c0 msgr2=0x7f35ac01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:38.828 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:38.828 7f35ccb2f700 1 --2- 172.21.2.102:0/2825890743 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f35ac0186c0 0x7f35ac01abc0 crc :-1 s=READY pgs=796 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:38.828 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:38.828 7f35ccb2f700 1 -- 172.21.2.102:0/2825890743 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f35c80a08f0 msgr2=0x7f35c80a0cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:38.828 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:38.828 7f35ccb2f700 1 --2- 172.21.2.102:0/2825890743 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f35c80a08f0 0x7f35c80a0cf0 secure :-1 s=READY pgs=1767 cs=0 l=1 rev1=1 rx=0x7f35b000b5e0 tx=0x7f35b000b980).stop 2021-02-10T11:27:38.828 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:38.828 7f35ccb2f700 1 -- 172.21.2.102:0/2825890743 shutdown_connections 2021-02-10T11:27:38.828 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:38.828 7f35ccb2f700 1 --2- 172.21.2.102:0/2825890743 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f35ac0186c0 0x7f35ac01abc0 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:38.828 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:38.828 7f35ccb2f700 1 --2- 172.21.2.102:0/2825890743 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f35c80b5c50 0x7f35c80a1b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:38.829 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:38.828 7f35ccb2f700 1 --2- 172.21.2.102:0/2825890743 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f35c80b5230 0x7f35c80b5630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:38.829 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:38.828 7f35ccb2f700 1 --2- 172.21.2.102:0/2825890743 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f35c80a08f0 0x7f35c80a0cf0 unknown :-1 s=CLOSED pgs=1767 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:38.829 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:38.828 7f35ccb2f700 1 -- 172.21.2.102:0/2825890743 >> 172.21.2.102:0/2825890743 conn(0x7f35c80a0030 msgr2=0x7f35c809b6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:38.829 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:38.828 7f35ccb2f700 1 -- 172.21.2.102:0/2825890743 shutdown_connections 2021-02-10T11:27:38.829 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:38.828 7f35ccb2f700 1 -- 172.21.2.102:0/2825890743 wait complete. 2021-02-10T11:27:38.837 INFO:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2021-02-10T11:27:38.837 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool create cephfs_metadata 8 2021-02-10T11:27:39.292 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:39.251 7f4516c1e700 1 Processor -- start 2021-02-10T11:27:39.292 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:39.251 7f4516c1e700 1 -- start start 2021-02-10T11:27:39.292 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:39.252 7f4516c1e700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f451010aff0 0x7f45100cf750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:39.292 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:39.252 7f4516c1e700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45100fe1c0 con 0x7f45100d1870 2021-02-10T11:27:39.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:39.252 7f4516c1e700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45101b0cb0 con 0x7f45100d1c70 2021-02-10T11:27:39.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:39.252 7f4516c1e700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4510109a60 con 0x7f451010aff0 2021-02-10T11:27:39.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:39.252 7f4514a3a700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f45100d1870 0x7f45101b04c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:51918/0 (socket says 172.21.2.102:51918) 2021-02-10T11:27:39.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:39.252 7f4514a3a700 1 -- 172.21.2.102:0/513457075 learned_addr learned my addr 172.21.2.102:0/513457075 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:27:39.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:39.252 7f451523b700 1 --2- 172.21.2.102:0/513457075 >> v2:172.21.2.113:3300/0 conn(0x7f451010aff0 0x7f45100cf750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:39.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:39.253 7f450f7fe700 1 -- 172.21.2.102:0/513457075 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3873221408 0 0) 0x7f44f8000ce0 con 0x7f45100d1870 2021-02-10T11:27:39.294 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:39.253 7f450f7fe700 1 -- 172.21.2.102:0/513457075 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44fc002fe0 con 0x7f45100d1870 2021-02-10T11:27:39.294 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:39.253 7f451523b700 1 -- 172.21.2.102:0/513457075 >> v1:172.21.2.113:6789/0 conn(0x7f45100d1c70 legacy=0x7f45100cef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:39.294 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:39.253 7f451523b700 1 -- 172.21.2.102:0/513457075 >> v1:172.21.2.102:6789/0 conn(0x7f45100d1870 legacy=0x7f45101b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:39.294 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:39.253 7f451523b700 1 -- 172.21.2.102:0/513457075 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f451010b3f0 con 0x7f451010aff0 2021-02-10T11:27:39.294 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:39.253 7f451523b700 1 --2- 172.21.2.102:0/513457075 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f451010aff0 0x7f45100cf750 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 rx=0x7f4508006ad0 tx=0x7f4508009ad0).ready entity=mon.1 client_cookie=48f067ef9aaeeb25 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:39.294 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:39.253 7f450f7fe700 1 -- 172.21.2.102:0/513457075 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f45080022e0 con 0x7f451010aff0 2021-02-10T11:27:39.295 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:39.253 7f450f7fe700 1 -- 172.21.2.102:0/513457075 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4508006410 con 0x7f451010aff0 2021-02-10T11:27:39.295 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:39.253 7f450f7fe700 1 -- 172.21.2.102:0/513457075 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4508002620 con 0x7f451010aff0 2021-02-10T11:27:39.295 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:39.253 7f4516c1e700 1 -- 172.21.2.102:0/513457075 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f451010aff0 msgr2=0x7f45100cf750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:39.295 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:39.253 7f4516c1e700 1 --2- 172.21.2.102:0/513457075 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f451010aff0 0x7f45100cf750 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 rx=0x7f4508006ad0 tx=0x7f4508009ad0).stop 2021-02-10T11:27:39.295 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:39.254 7f4516c1e700 1 -- 172.21.2.102:0/513457075 shutdown_connections 2021-02-10T11:27:39.296 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:39.254 7f4516c1e700 1 --2- 172.21.2.102:0/513457075 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f451010aff0 0x7f45100cf750 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:39.296 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:39.254 7f4516c1e700 1 -- 172.21.2.102:0/513457075 >> 172.21.2.102:0/513457075 conn(0x7f45100ce420 msgr2=0x7f45100d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:39.296 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:39.254 7f4516c1e700 1 -- 172.21.2.102:0/513457075 shutdown_connections 2021-02-10T11:27:39.296 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:39.254 7f4516c1e700 1 -- 172.21.2.102:0/513457075 wait complete. 2021-02-10T11:27:39.299 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:39.292 7f4516c1e700 1 Processor -- start 2021-02-10T11:27:39.299 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:39.292 7f4516c1e700 1 -- start start 2021-02-10T11:27:39.299 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:39.293 7f4516c1e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f451019e410 0x7f451008a420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:39.299 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:39.293 7f4516c1e700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4510083980 0x7f4510083d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:39.299 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:39.293 7f4516c1e700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f45100baa30 0x7f45100bae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:39.300 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:39.293 7f4516c1e700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45100ba4b0 con 0x7f451019e410 2021-02-10T11:27:39.300 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:39.293 7f4516c1e700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45100843a0 con 0x7f45100baa30 2021-02-10T11:27:39.300 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:39.293 7f4516c1e700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4510084710 con 0x7f4510083980 2021-02-10T11:27:39.300 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:39.293 7f4514a3a700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f451019e410 0x7f451008a420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:39.300 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:39.293 7f4514a3a700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f451019e410 0x7f451008a420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:59698/0 (socket says 172.21.2.102:59698) 2021-02-10T11:27:39.300 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:39.293 7f4514a3a700 1 -- 172.21.2.102:0/3680028473 learned_addr learned my addr 172.21.2.102:0/3680028473 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:27:39.300 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:39.293 7f4514a3a700 1 -- 172.21.2.102:0/3680028473 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4510083980 msgr2=0x7f4510083d80 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:27:39.301 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:39.293 7f4514a3a700 1 --2- 172.21.2.102:0/3680028473 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4510083980 0x7f4510083d80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:39.301 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:39.293 7f4514a3a700 1 -- 172.21.2.102:0/3680028473 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f45100baa30 msgr2=0x7f45100bae30 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:27:39.301 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:39.293 7f4514a3a700 1 --2- 172.21.2.102:0/3680028473 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f45100baa30 0x7f45100bae30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:39.301 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:39.293 7f4514a3a700 1 -- 172.21.2.102:0/3680028473 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45100bb450 con 0x7f451019e410 2021-02-10T11:27:39.301 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:39.293 7f4514a3a700 1 --2- 172.21.2.102:0/3680028473 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f451019e410 0x7f451008a420 secure :-1 s=READY pgs=1769 cs=0 l=1 rev1=1 rx=0x7f44f800b5c0 tx=0x7f44f800b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:39.301 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:39.294 7f450d7fa700 1 -- 172.21.2.102:0/3680028473 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f44f8006030 con 0x7f451019e410 2021-02-10T11:27:39.302 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:39.294 7f450d7fa700 1 -- 172.21.2.102:0/3680028473 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44f80043a0 con 0x7f451019e410 2021-02-10T11:27:39.302 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:39.294 7f450d7fa700 1 -- 172.21.2.102:0/3680028473 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f44f8004840 con 0x7f451019e410 2021-02-10T11:27:39.302 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:39.294 7f4516c1e700 1 -- 172.21.2.102:0/3680028473 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45100bb450 con 0x7f451019e410 2021-02-10T11:27:39.302 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:39.294 7f4516c1e700 1 -- 172.21.2.102:0/3680028473 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45100b9830 con 0x7f451019e410 2021-02-10T11:27:39.302 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:39.294 7f450d7fa700 1 -- 172.21.2.102:0/3680028473 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f44f8004580 con 0x7f451019e410 2021-02-10T11:27:39.303 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:39.294 7f450d7fa700 1 --2- 172.21.2.102:0/3680028473 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f44f40186c0 0x7f44f401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:39.303 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:39.294 7f450d7fa700 1 -- 172.21.2.102:0/3680028473 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 4753+0+0 (secure 0 0 0) 0x7f44f80320a0 con 0x7f451019e410 2021-02-10T11:27:39.303 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:39.295 7f450ffff700 1 --2- 172.21.2.102:0/3680028473 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f44f40186c0 0x7f44f401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:39.303 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:39.295 7f450ffff700 1 --2- 172.21.2.102:0/3680028473 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f44f40186c0 0x7f44f401abc0 crc :-1 s=READY pgs=797 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:39.304 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:39.302 7f4516c1e700 1 -- 172.21.2.102:0/3680028473 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4510082a70 con 0x7f451019e410 2021-02-10T11:27:39.304 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:39.305 7f450d7fa700 1 -- 172.21.2.102:0/3680028473 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f44f8030d00 con 0x7f451019e410 2021-02-10T11:27:39.434 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:39.434 7f4516c1e700 1 -- 172.21.2.102:0/3680028473 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"} v 0) v1 -- 0x7f4510082a70 con 0x7f451019e410 2021-02-10T11:27:39.842 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:39.842 7f450d7fa700 1 -- 172.21.2.102:0/3680028473 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"}]=0 pool 'cephfs_metadata' created v82) v1 ==== 133+0+0 (secure 0 0 0) 0x7f44f8004580 con 0x7f451019e410 2021-02-10T11:27:39.849 INFO:teuthology.orchestra.run.gibba002.stderr:pool 'cephfs_metadata' created 2021-02-10T11:27:39.851 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:39.851 7f4516c1e700 1 -- 172.21.2.102:0/3680028473 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f44f40186c0 msgr2=0x7f44f401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:39.851 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:39.851 7f4516c1e700 1 --2- 172.21.2.102:0/3680028473 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f44f40186c0 0x7f44f401abc0 crc :-1 s=READY pgs=797 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:39.851 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:39.851 7f4516c1e700 1 -- 172.21.2.102:0/3680028473 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f451019e410 msgr2=0x7f451008a420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:39.851 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:39.851 7f4516c1e700 1 --2- 172.21.2.102:0/3680028473 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f451019e410 0x7f451008a420 secure :-1 s=READY pgs=1769 cs=0 l=1 rev1=1 rx=0x7f44f800b5c0 tx=0x7f44f800b960).stop 2021-02-10T11:27:39.852 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:39.851 7f4516c1e700 1 -- 172.21.2.102:0/3680028473 shutdown_connections 2021-02-10T11:27:39.852 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:39.851 7f4516c1e700 1 --2- 172.21.2.102:0/3680028473 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f44f40186c0 0x7f44f401abc0 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:39.852 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:39.851 7f4516c1e700 1 --2- 172.21.2.102:0/3680028473 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f45100baa30 0x7f45100bae30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:39.852 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:39.851 7f4516c1e700 1 --2- 172.21.2.102:0/3680028473 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4510083980 0x7f4510083d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:39.852 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:39.851 7f4516c1e700 1 --2- 172.21.2.102:0/3680028473 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f451019e410 0x7f451008a420 unknown :-1 s=CLOSED pgs=1769 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:39.852 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:39.851 7f4516c1e700 1 -- 172.21.2.102:0/3680028473 >> 172.21.2.102:0/3680028473 conn(0x7f45100bc0c0 msgr2=0x7f4510082ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:39.852 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:39.851 7f4516c1e700 1 -- 172.21.2.102:0/3680028473 shutdown_connections 2021-02-10T11:27:39.852 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:39.851 7f4516c1e700 1 -- 172.21.2.102:0/3680028473 wait complete. 2021-02-10T11:27:39.860 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool create cephfs_data 8 2021-02-10T11:27:40.291 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:40.274 7fb24a3d9700 1 Processor -- start 2021-02-10T11:27:40.291 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:40.274 7fb24a3d9700 1 -- start start 2021-02-10T11:27:40.291 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:40.274 7fb24a3d9700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fb2440d1c70 0x7fb2440cef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:40.291 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:40.274 7fb24a3d9700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2440d0170 con 0x7fb2440d1870 2021-02-10T11:27:40.291 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:40.274 7fb24a3d9700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2440fe1c0 con 0x7fb2440cf580 2021-02-10T11:27:40.291 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:40.274 7fb24a3d9700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2441b0cb0 con 0x7fb2440d1c70 2021-02-10T11:27:40.291 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:40.275 7fb2437fe700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fb2440d1c70 0x7fb2440cef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:40.291 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:40.275 7fb2437fe700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fb2440d1c70 0x7fb2440cef60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:59710/0 (socket says 172.21.2.102:59710) 2021-02-10T11:27:40.292 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:40.275 7fb2437fe700 1 -- 172.21.2.102:0/3299083427 learned_addr learned my addr 172.21.2.102:0/3299083427 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:27:40.292 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:40.275 7fb2437fe700 1 -- 172.21.2.102:0/3299083427 >> v1:172.21.2.113:6789/0 conn(0x7fb2440cf580 legacy=0x7fb2440cf980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:27:40.292 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:40.275 7fb2437fe700 1 -- 172.21.2.102:0/3299083427 >> v1:172.21.2.102:6789/0 conn(0x7fb2440d1870 legacy=0x7fb2441b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:40.292 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:40.275 7fb2437fe700 1 -- 172.21.2.102:0/3299083427 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb244109a60 con 0x7fb2440d1c70 2021-02-10T11:27:40.292 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:40.275 7fb2437fe700 1 --2- 172.21.2.102:0/3299083427 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb2440d1c70 0x7fb2440cef60 secure :-1 s=READY pgs=1771 cs=0 l=1 rev1=1 rx=0x7fb238006700 tx=0x7fb238009b80).ready entity=mon.0 client_cookie=25eb152a41cbd6e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:40.292 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:40.275 7fb242ffd700 1 -- 172.21.2.102:0/3299083427 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb2380009b0 con 0x7fb2440d1c70 2021-02-10T11:27:40.292 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:40.275 7fb242ffd700 1 -- 172.21.2.102:0/3299083427 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb23800c0c0 con 0x7fb2440d1c70 2021-02-10T11:27:40.292 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:40.275 7fb242ffd700 1 -- 172.21.2.102:0/3299083427 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb23800c540 con 0x7fb2440d1c70 2021-02-10T11:27:40.292 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:40.275 7fb24a3d9700 1 -- 172.21.2.102:0/3299083427 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb2440d1c70 msgr2=0x7fb2440cef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:40.292 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:40.275 7fb24a3d9700 1 --2- 172.21.2.102:0/3299083427 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb2440d1c70 0x7fb2440cef60 secure :-1 s=READY pgs=1771 cs=0 l=1 rev1=1 rx=0x7fb238006700 tx=0x7fb238009b80).stop 2021-02-10T11:27:40.292 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:40.276 7fb24a3d9700 1 -- 172.21.2.102:0/3299083427 shutdown_connections 2021-02-10T11:27:40.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:40.276 7fb24a3d9700 1 --2- 172.21.2.102:0/3299083427 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb2440d1c70 0x7fb2440cef60 unknown :-1 s=CLOSED pgs=1771 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:40.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:40.276 7fb24a3d9700 1 -- 172.21.2.102:0/3299083427 >> 172.21.2.102:0/3299083427 conn(0x7fb2440ce420 msgr2=0x7fb2440d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:40.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:40.276 7fb24a3d9700 1 -- 172.21.2.102:0/3299083427 shutdown_connections 2021-02-10T11:27:40.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:40.276 7fb24a3d9700 1 -- 172.21.2.102:0/3299083427 wait complete. 2021-02-10T11:27:40.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:40.291 7fb24a3d9700 1 Processor -- start 2021-02-10T11:27:40.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:40.291 7fb24a3d9700 1 -- start start 2021-02-10T11:27:40.294 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:40.291 7fb24a3d9700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb2440c0740 0x7fb2441c5580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:40.295 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:40.291 7fb24a3d9700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb2441c5ba0 0x7fb2440a5c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:40.295 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:40.291 7fb24a3d9700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb2440b6e60 0x7fb2440a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:40.295 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:40.291 7fb24a3d9700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2440a6880 con 0x7fb2441c5ba0 2021-02-10T11:27:40.295 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:40.291 7fb24a3d9700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2440a6aa0 con 0x7fb2440b6e60 2021-02-10T11:27:40.295 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:40.291 7fb24a3d9700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2440a6cc0 con 0x7fb2440c0740 2021-02-10T11:27:40.295 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:40.291 7fb2437fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb2441c5ba0 0x7fb2440a5c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:40.295 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:40.291 7fb2437fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb2441c5ba0 0x7fb2440a5c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:59714/0 (socket says 172.21.2.102:59714) 2021-02-10T11:27:40.295 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:40.291 7fb2437fe700 1 -- 172.21.2.102:0/3735259011 learned_addr learned my addr 172.21.2.102:0/3735259011 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:27:40.295 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:40.292 7fb2437fe700 1 -- 172.21.2.102:0/3735259011 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb2440c0740 msgr2=0x7fb2441c5580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:40.295 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:40.292 7fb2437fe700 1 --2- 172.21.2.102:0/3735259011 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb2440c0740 0x7fb2441c5580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:40.295 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:40.292 7fb2437fe700 1 -- 172.21.2.102:0/3735259011 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb2440b6e60 msgr2=0x7fb2440a6260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:40.295 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:40.292 7fb2437fe700 1 --2- 172.21.2.102:0/3735259011 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb2440b6e60 0x7fb2440a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:40.296 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:40.292 7fb2437fe700 1 -- 172.21.2.102:0/3735259011 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2440a6ee0 con 0x7fb2441c5ba0 2021-02-10T11:27:40.296 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:40.292 7fb2437fe700 1 --2- 172.21.2.102:0/3735259011 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb2441c5ba0 0x7fb2440a5c40 secure :-1 s=READY pgs=1772 cs=0 l=1 rev1=1 rx=0x7fb238009240 tx=0x7fb23800b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:40.296 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:40.292 7fb240ff9700 1 -- 172.21.2.102:0/3735259011 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb238002030 con 0x7fb2441c5ba0 2021-02-10T11:27:40.296 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:40.292 7fb240ff9700 1 -- 172.21.2.102:0/3735259011 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb238006580 con 0x7fb2441c5ba0 2021-02-10T11:27:40.296 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:40.292 7fb240ff9700 1 -- 172.21.2.102:0/3735259011 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb238002630 con 0x7fb2441c5ba0 2021-02-10T11:27:40.296 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:40.292 7fb24a3d9700 1 -- 172.21.2.102:0/3735259011 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2440a6ee0 con 0x7fb2441c5ba0 2021-02-10T11:27:40.296 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:40.292 7fb24a3d9700 1 -- 172.21.2.102:0/3735259011 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2440a6ee0 con 0x7fb2441c5ba0 2021-02-10T11:27:40.296 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:40.293 7fb240ff9700 1 -- 172.21.2.102:0/3735259011 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fb238000c90 con 0x7fb2441c5ba0 2021-02-10T11:27:40.296 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:40.293 7fb240ff9700 1 --2- 172.21.2.102:0/3735259011 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb2240186c0 0x7fb22401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:40.296 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:40.293 7fb240ff9700 1 -- 172.21.2.102:0/3735259011 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5107+0+0 (secure 0 0 0) 0x7fb238032370 con 0x7fb2441c5ba0 2021-02-10T11:27:40.296 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:40.293 7fb243fff700 1 --2- 172.21.2.102:0/3735259011 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb2240186c0 0x7fb22401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:40.296 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:40.293 7fb243fff700 1 --2- 172.21.2.102:0/3735259011 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb2240186c0 0x7fb22401abc0 crc :-1 s=READY pgs=798 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:40.325 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:40.324 7fb24a3d9700 1 -- 172.21.2.102:0/3735259011 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2440807d0 con 0x7fb2441c5ba0 2021-02-10T11:27:40.327 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:40.327 7fb240ff9700 1 -- 172.21.2.102:0/3735259011 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fb23802fd70 con 0x7fb2441c5ba0 2021-02-10T11:27:40.457 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:40.457 7fb24a3d9700 1 -- 172.21.2.102:0/3735259011 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_data"} v 0) v1 -- 0x7fb244088070 con 0x7fb2441c5ba0 2021-02-10T11:27:40.846 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:40.846 7fb240ff9700 1 -- 172.21.2.102:0/3735259011 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_data"}]=0 pool 'cephfs_data' created v83) v1 ==== 125+0+0 (secure 0 0 0) 0x7fb23802fd70 con 0x7fb2441c5ba0 2021-02-10T11:27:40.872 INFO:teuthology.orchestra.run.gibba002.stderr:pool 'cephfs_data' created 2021-02-10T11:27:40.874 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:40.874 7fb24a3d9700 1 -- 172.21.2.102:0/3735259011 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb2240186c0 msgr2=0x7fb22401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:40.874 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:40.874 7fb24a3d9700 1 --2- 172.21.2.102:0/3735259011 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb2240186c0 0x7fb22401abc0 crc :-1 s=READY pgs=798 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:40.874 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:40.874 7fb24a3d9700 1 -- 172.21.2.102:0/3735259011 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb2441c5ba0 msgr2=0x7fb2440a5c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:40.874 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:40.874 7fb24a3d9700 1 --2- 172.21.2.102:0/3735259011 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb2441c5ba0 0x7fb2440a5c40 secure :-1 s=READY pgs=1772 cs=0 l=1 rev1=1 rx=0x7fb238009240 tx=0x7fb23800b960).stop 2021-02-10T11:27:40.875 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:40.874 7fb24a3d9700 1 -- 172.21.2.102:0/3735259011 shutdown_connections 2021-02-10T11:27:40.875 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:40.874 7fb24a3d9700 1 --2- 172.21.2.102:0/3735259011 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb2240186c0 0x7fb22401abc0 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:40.875 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:40.874 7fb24a3d9700 1 --2- 172.21.2.102:0/3735259011 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb2440b6e60 0x7fb2440a6260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:40.875 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:40.874 7fb24a3d9700 1 --2- 172.21.2.102:0/3735259011 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb2441c5ba0 0x7fb2440a5c40 unknown :-1 s=CLOSED pgs=1772 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:40.875 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:40.874 7fb24a3d9700 1 --2- 172.21.2.102:0/3735259011 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb2440c0740 0x7fb2441c5580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:40.875 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:40.874 7fb24a3d9700 1 -- 172.21.2.102:0/3735259011 >> 172.21.2.102:0/3735259011 conn(0x7fb2440c4120 msgr2=0x7fb2440ffd60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:40.875 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:40.874 7fb24a3d9700 1 -- 172.21.2.102:0/3735259011 shutdown_connections 2021-02-10T11:27:40.875 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:40.874 7fb24a3d9700 1 -- 172.21.2.102:0/3735259011 wait complete. 2021-02-10T11:27:40.883 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2021-02-10T11:27:41.314 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:41.298 7f1fd5087700 1 Processor -- start 2021-02-10T11:27:41.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:41.298 7f1fd5087700 1 -- start start 2021-02-10T11:27:41.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:41.298 7f1fd5087700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1fd00d1880 0x7f1fd01b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:41.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:41.298 7f1fd5087700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f1fd01b0af0 0x7f1fd0108950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:41.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:41.298 7f1fd5087700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f1fd00d1c80 0x7f1fd0108f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:41.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:41.298 7f1fd5087700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1fd00e5f00 con 0x7f1fd01b0af0 2021-02-10T11:27:41.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:41.298 7f1fd5087700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1fd0109a70 con 0x7f1fd00d1c80 2021-02-10T11:27:41.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:41.298 7f1fd5087700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fd0088070 con 0x7f1fd00d1880 2021-02-10T11:27:41.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:41.298 7f1fce59c700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f1fd01b0af0 0x7f1fd0108950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:41.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:41.298 7f1fce59c700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f1fd01b0af0 0x7f1fd0108950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:59724/0 (socket says 172.21.2.102:59724) 2021-02-10T11:27:41.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:41.298 7f1fce59c700 1 -- 172.21.2.102:0/1199848453 learned_addr learned my addr 172.21.2.102:0/1199848453 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:27:41.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:41.299 7f1fce59c700 1 -- 172.21.2.102:0/1199848453 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1fd00d1880 msgr2=0x7f1fd01b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:27:41.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:41.299 7f1fce59c700 1 --2- 172.21.2.102:0/1199848453 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1fd00d1880 0x7f1fd01b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:41.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:41.299 7f1fce59c700 1 -- 172.21.2.102:0/1199848453 >> v2:172.21.2.113:3300/0 conn(0x7f1fd00d1c80 msgr2=0x7f1fd0108f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:27:41.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:41.299 7f1fce59c700 1 --2- 172.21.2.102:0/1199848453 >> v2:172.21.2.113:3300/0 conn(0x7f1fd00d1c80 0x7f1fd0108f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:41.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:41.299 7f1fce59c700 1 -- 172.21.2.102:0/1199848453 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fd0109590 con 0x7f1fd01b0af0 2021-02-10T11:27:41.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:41.299 7f1fce59c700 1 --2- 172.21.2.102:0/1199848453 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1fd01b0af0 0x7f1fd0108950 secure :-1 s=READY pgs=1773 cs=0 l=1 rev1=1 rx=0x7f1fb8006700 tx=0x7f1fb8009b80).ready entity=mon.0 client_cookie=107afe3eb3a7cc1f server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:41.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:41.299 7f1fcdd9b700 1 -- 172.21.2.102:0/1199848453 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1fb800c0c0 con 0x7f1fd01b0af0 2021-02-10T11:27:41.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:41.299 7f1fcdd9b700 1 -- 172.21.2.102:0/1199848453 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fb800c440 con 0x7f1fd01b0af0 2021-02-10T11:27:41.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:41.299 7f1fcdd9b700 1 -- 172.21.2.102:0/1199848453 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1fb800c920 con 0x7f1fd01b0af0 2021-02-10T11:27:41.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:41.299 7f1fd5087700 1 -- 172.21.2.102:0/1199848453 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1fd01b0af0 msgr2=0x7f1fd0108950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:41.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:41.299 7f1fd5087700 1 --2- 172.21.2.102:0/1199848453 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1fd01b0af0 0x7f1fd0108950 secure :-1 s=READY pgs=1773 cs=0 l=1 rev1=1 rx=0x7f1fb8006700 tx=0x7f1fb8009b80).stop 2021-02-10T11:27:41.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:41.300 7f1fd5087700 1 -- 172.21.2.102:0/1199848453 shutdown_connections 2021-02-10T11:27:41.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:41.300 7f1fd5087700 1 --2- 172.21.2.102:0/1199848453 >> v2:172.21.2.113:3300/0 conn(0x7f1fd00d1c80 0x7f1fd0108f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:41.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:41.300 7f1fd5087700 1 --2- 172.21.2.102:0/1199848453 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1fd01b0af0 0x7f1fd0108950 unknown :-1 s=CLOSED pgs=1773 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:41.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:41.300 7f1fd5087700 1 --2- 172.21.2.102:0/1199848453 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1fd00d1880 0x7f1fd01b04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:41.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:41.300 7f1fd5087700 1 -- 172.21.2.102:0/1199848453 >> 172.21.2.102:0/1199848453 conn(0x7f1fd00ce430 msgr2=0x7f1fd00d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:41.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:41.300 7f1fd5087700 1 -- 172.21.2.102:0/1199848453 shutdown_connections 2021-02-10T11:27:41.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:41.300 7f1fd5087700 1 -- 172.21.2.102:0/1199848453 wait complete. 2021-02-10T11:27:41.319 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:41.315 7f1fd5087700 1 Processor -- start 2021-02-10T11:27:41.319 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:41.315 7f1fd5087700 1 -- start start 2021-02-10T11:27:41.319 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:41.315 7f1fd5087700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1fd0096710 0x7f1fd0096f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:41.319 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:41.315 7f1fd5087700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1fd019c360 0x7f1fd00ad520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:41.319 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:41.315 7f1fd5087700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1fd00adb40 0x7f1fd019cef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:41.319 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:41.315 7f1fd5087700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fd019c970 con 0x7f1fd00adb40 2021-02-10T11:27:41.319 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:41.315 7f1fd5087700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fd0096b10 con 0x7f1fd019c360 2021-02-10T11:27:41.319 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:41.315 7f1fd5087700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fd0082c60 con 0x7f1fd0096710 2021-02-10T11:27:41.319 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:41.315 7f1fcf59e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1fd00adb40 0x7f1fd019cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:41.319 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:41.315 7f1fcf59e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1fd00adb40 0x7f1fd019cef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:59732/0 (socket says 172.21.2.102:59732) 2021-02-10T11:27:41.320 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:41.315 7f1fcf59e700 1 -- 172.21.2.102:0/342166385 learned_addr learned my addr 172.21.2.102:0/342166385 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:27:41.320 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:41.316 7f1fced9d700 1 --2- 172.21.2.102:0/342166385 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1fd0096710 0x7f1fd0096f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:41.320 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:41.316 7f1fce59c700 1 --2- 172.21.2.102:0/342166385 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1fd019c360 0x7f1fd00ad520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:41.320 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:41.316 7f1fcf59e700 1 -- 172.21.2.102:0/342166385 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1fd0096710 msgr2=0x7f1fd0096f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:41.320 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:41.316 7f1fcf59e700 1 --2- 172.21.2.102:0/342166385 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1fd0096710 0x7f1fd0096f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:41.320 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:41.316 7f1fcf59e700 1 -- 172.21.2.102:0/342166385 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1fd019c360 msgr2=0x7f1fd00ad520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:41.320 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:41.316 7f1fcf59e700 1 --2- 172.21.2.102:0/342166385 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1fd019c360 0x7f1fd00ad520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:41.320 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:41.316 7f1fcf59e700 1 -- 172.21.2.102:0/342166385 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fd0097590 con 0x7f1fd00adb40 2021-02-10T11:27:41.320 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:41.316 7f1fcf59e700 1 --2- 172.21.2.102:0/342166385 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1fd00adb40 0x7f1fd019cef0 secure :-1 s=READY pgs=1774 cs=0 l=1 rev1=1 rx=0x7f1fc8009980 tx=0x7f1fc800bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:41.320 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:41.316 7f1fc77fe700 1 -- 172.21.2.102:0/342166385 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1fc8005870 con 0x7f1fd00adb40 2021-02-10T11:27:41.320 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:41.316 7f1fc77fe700 1 -- 172.21.2.102:0/342166385 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fc8005b30 con 0x7f1fd00adb40 2021-02-10T11:27:41.321 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:41.316 7f1fc77fe700 1 -- 172.21.2.102:0/342166385 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1fc8006030 con 0x7f1fd00adb40 2021-02-10T11:27:41.321 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:41.316 7f1fd5087700 1 -- 172.21.2.102:0/342166385 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fd0097590 con 0x7f1fd00adb40 2021-02-10T11:27:41.321 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:41.316 7f1fd5087700 1 -- 172.21.2.102:0/342166385 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fd0097590 con 0x7f1fd00adb40 2021-02-10T11:27:41.321 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:41.317 7f1fc77fe700 1 -- 172.21.2.102:0/342166385 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1fc8005d10 con 0x7f1fd00adb40 2021-02-10T11:27:41.321 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:41.317 7f1fc77fe700 1 --2- 172.21.2.102:0/342166385 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1fb40186c0 0x7f1fb401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:41.321 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:41.317 7f1fc77fe700 1 -- 172.21.2.102:0/342166385 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5457+0+0 (secure 0 0 0) 0x7f1fc8032730 con 0x7f1fd00adb40 2021-02-10T11:27:41.321 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:41.317 7f1fced9d700 1 --2- 172.21.2.102:0/342166385 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1fb40186c0 0x7f1fb401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:41.321 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:41.317 7f1fced9d700 1 --2- 172.21.2.102:0/342166385 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1fb40186c0 0x7f1fb401abc0 crc :-1 s=READY pgs=801 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:41.348 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:41.348 7f1fd5087700 1 -- 172.21.2.102:0/342166385 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fd0096b10 con 0x7f1fd00adb40 2021-02-10T11:27:41.350 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:41.350 7f1fc77fe700 1 -- 172.21.2.102:0/342166385 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f1fc802ff90 con 0x7f1fd00adb40 2021-02-10T11:27:41.477 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:41.477 7f1fd5087700 1 -- 172.21.2.102:0/342166385 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"data": "cephfs_data", "prefix": "fs new", "force": true, "fs_name": "cephfs", "metadata": "cephfs_metadata"} v 0) v1 -- 0x7f1fd0096b10 con 0x7f1fd00adb40 2021-02-10T11:27:41.481 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:41.482 7f1fc77fe700 1 -- 172.21.2.102:0/342166385 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"data": "cephfs_data", "prefix": "fs new", "force": true, "fs_name": "cephfs", "metadata": "cephfs_metadata"}]=0 new fs with metadata pool 16 and data pool 17 v62) v1 ==== 189+0+0 (secure 0 0 0) 0x7f1fc802ff90 con 0x7f1fd00adb40 2021-02-10T11:27:41.484 INFO:teuthology.orchestra.run.gibba002.stderr:new fs with metadata pool 16 and data pool 17 2021-02-10T11:27:41.487 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:41.486 7f1fd5087700 1 -- 172.21.2.102:0/342166385 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1fb40186c0 msgr2=0x7f1fb401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:41.487 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:41.486 7f1fd5087700 1 --2- 172.21.2.102:0/342166385 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1fb40186c0 0x7f1fb401abc0 crc :-1 s=READY pgs=801 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:41.487 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:41.486 7f1fd5087700 1 -- 172.21.2.102:0/342166385 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1fd00adb40 msgr2=0x7f1fd019cef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:41.487 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:41.486 7f1fd5087700 1 --2- 172.21.2.102:0/342166385 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1fd00adb40 0x7f1fd019cef0 secure :-1 s=READY pgs=1774 cs=0 l=1 rev1=1 rx=0x7f1fc8009980 tx=0x7f1fc800bcd0).stop 2021-02-10T11:27:41.487 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:41.486 7f1fd5087700 1 -- 172.21.2.102:0/342166385 shutdown_connections 2021-02-10T11:27:41.487 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:41.486 7f1fd5087700 1 --2- 172.21.2.102:0/342166385 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1fd00adb40 0x7f1fd019cef0 unknown :-1 s=CLOSED pgs=1774 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:41.487 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:41.486 7f1fd5087700 1 --2- 172.21.2.102:0/342166385 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1fb40186c0 0x7f1fb401abc0 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:41.487 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:41.486 7f1fd5087700 1 --2- 172.21.2.102:0/342166385 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1fd019c360 0x7f1fd00ad520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:41.487 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:41.486 7f1fd5087700 1 --2- 172.21.2.102:0/342166385 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1fd0096710 0x7f1fd0096f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:41.487 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:41.486 7f1fd5087700 1 -- 172.21.2.102:0/342166385 >> 172.21.2.102:0/342166385 conn(0x7f1fd019e990 msgr2=0x7f1fd01918e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:41.487 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:41.487 7f1fd5087700 1 -- 172.21.2.102:0/342166385 shutdown_connections 2021-02-10T11:27:41.487 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:41.487 7f1fd5087700 1 -- 172.21.2.102:0/342166385 wait complete. 2021-02-10T11:27:41.495 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:27:41.921 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:41.905 7fb26ed2b700 1 Processor -- start 2021-02-10T11:27:41.922 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:41.905 7fb26ed2b700 1 -- start start 2021-02-10T11:27:41.922 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:41.906 7fb26ed2b700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fb2680d1870 0x7fb2681b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:41.922 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:41.906 7fb26ed2b700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fb2681b0a10 0x7fb2680cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:41.922 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:41.906 7fb26ed2b700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb2680d1c70 0x7fb2680cf950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:41.922 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:41.906 7fb26ed2b700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2680e5ed0 con 0x7fb2680d1870 2021-02-10T11:27:41.922 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:41.906 7fb26ed2b700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2680f0b10 con 0x7fb2681b0a10 2021-02-10T11:27:41.922 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:41.906 7fb26ed2b700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2680cff70 con 0x7fb2680d1c70 2021-02-10T11:27:41.922 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:41.906 7fb26cb47700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fb2680d1870 0x7fb2681b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:41.923 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:41.906 7fb26cb47700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fb2680d1870 0x7fb2681b03f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:59738/0 (socket says 172.21.2.102:59738) 2021-02-10T11:27:41.923 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:41.906 7fb26cb47700 1 -- 172.21.2.102:0/732064273 learned_addr learned my addr 172.21.2.102:0/732064273 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:27:41.923 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:41.906 7fb26cb47700 1 -- 172.21.2.102:0/732064273 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb2680d1c70 msgr2=0x7fb2680cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:27:41.923 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:41.906 7fb26cb47700 1 --2- 172.21.2.102:0/732064273 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb2680d1c70 0x7fb2680cf950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:41.923 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:41.906 7fb26cb47700 1 -- 172.21.2.102:0/732064273 >> v2:172.21.2.113:3300/0 conn(0x7fb2681b0a10 msgr2=0x7fb2680cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:27:41.923 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:41.906 7fb26cb47700 1 --2- 172.21.2.102:0/732064273 >> v2:172.21.2.113:3300/0 conn(0x7fb2681b0a10 0x7fb2680cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:41.923 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:41.906 7fb26cb47700 1 -- 172.21.2.102:0/732064273 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2680d0150 con 0x7fb2680d1870 2021-02-10T11:27:41.923 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:41.907 7fb26cb47700 1 --2- 172.21.2.102:0/732064273 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb2680d1870 0x7fb2681b03f0 secure :-1 s=READY pgs=1775 cs=0 l=1 rev1=1 rx=0x7fb250006700 tx=0x7fb250009b80).ready entity=mon.0 client_cookie=663101ac9e07726e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:41.923 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:41.907 7fb2677fe700 1 -- 172.21.2.102:0/732064273 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb25000c0c0 con 0x7fb2680d1870 2021-02-10T11:27:41.923 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:41.907 7fb2677fe700 1 -- 172.21.2.102:0/732064273 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb25000c440 con 0x7fb2680d1870 2021-02-10T11:27:41.923 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:41.907 7fb2677fe700 1 -- 172.21.2.102:0/732064273 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb25000c960 con 0x7fb2680d1870 2021-02-10T11:27:41.924 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:41.907 7fb26ed2b700 1 -- 172.21.2.102:0/732064273 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb2680d1870 msgr2=0x7fb2681b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:41.924 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:41.907 7fb26ed2b700 1 --2- 172.21.2.102:0/732064273 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb2680d1870 0x7fb2681b03f0 secure :-1 s=READY pgs=1775 cs=0 l=1 rev1=1 rx=0x7fb250006700 tx=0x7fb250009b80).stop 2021-02-10T11:27:41.924 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:41.907 7fb26ed2b700 1 -- 172.21.2.102:0/732064273 shutdown_connections 2021-02-10T11:27:41.924 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:41.907 7fb26ed2b700 1 --2- 172.21.2.102:0/732064273 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb2680d1c70 0x7fb2680cf950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:41.924 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:41.907 7fb26ed2b700 1 --2- 172.21.2.102:0/732064273 >> v2:172.21.2.113:3300/0 conn(0x7fb2681b0a10 0x7fb2680cf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:41.924 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:41.907 7fb26ed2b700 1 --2- 172.21.2.102:0/732064273 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb2680d1870 0x7fb2681b03f0 unknown :-1 s=CLOSED pgs=1775 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:41.924 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:41.907 7fb26ed2b700 1 -- 172.21.2.102:0/732064273 >> 172.21.2.102:0/732064273 conn(0x7fb2680ce450 msgr2=0x7fb2680d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:41.924 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:41.907 7fb26ed2b700 1 -- 172.21.2.102:0/732064273 shutdown_connections 2021-02-10T11:27:41.924 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:41.907 7fb26ed2b700 1 -- 172.21.2.102:0/732064273 wait complete. 2021-02-10T11:27:41.926 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:41.922 7fb26ed2b700 1 Processor -- start 2021-02-10T11:27:41.926 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:41.922 7fb26ed2b700 1 -- start start 2021-02-10T11:27:41.926 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:41.922 7fb26ed2b700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb2681b8ad0 0x7fb268187e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:41.926 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:41.922 7fb26ed2b700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb2681b8ed0 0x7fb2680b1af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:41.926 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:41.922 7fb26ed2b700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb2680b2110 0x7fb2681b7a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:41.926 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:41.922 7fb26ed2b700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb268188470 con 0x7fb2680b2110 2021-02-10T11:27:41.927 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:41.922 7fb26ed2b700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb26818be10 con 0x7fb2681b8ad0 2021-02-10T11:27:41.927 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:41.922 7fb26ed2b700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb26818c030 con 0x7fb2681b8ed0 2021-02-10T11:27:41.927 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:41.923 7fb26d348700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb2680b2110 0x7fb2681b7a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:41.927 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:41.923 7fb26d348700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb2680b2110 0x7fb2681b7a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:59744/0 (socket says 172.21.2.102:59744) 2021-02-10T11:27:41.927 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:41.923 7fb26d348700 1 -- 172.21.2.102:0/669122719 learned_addr learned my addr 172.21.2.102:0/669122719 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:27:41.927 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:41.923 7fb26d348700 1 -- 172.21.2.102:0/669122719 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb2681b8ed0 msgr2=0x7fb2680b1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:41.927 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:41.923 7fb26d348700 1 --2- 172.21.2.102:0/669122719 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb2681b8ed0 0x7fb2680b1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:41.927 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:41.923 7fb26d348700 1 -- 172.21.2.102:0/669122719 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb2681b8ad0 msgr2=0x7fb268187e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:41.927 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:41.923 7fb26d348700 1 --2- 172.21.2.102:0/669122719 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb2681b8ad0 0x7fb268187e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:41.927 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:41.923 7fb26d348700 1 -- 172.21.2.102:0/669122719 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2680b2510 con 0x7fb2680b2110 2021-02-10T11:27:41.928 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:41.923 7fb26d348700 1 --2- 172.21.2.102:0/669122719 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb2680b2110 0x7fb2681b7a00 secure :-1 s=READY pgs=1776 cs=0 l=1 rev1=1 rx=0x7fb260006f10 tx=0x7fb26000b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:41.928 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:41.923 7fb2657fa700 1 -- 172.21.2.102:0/669122719 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb260006240 con 0x7fb2680b2110 2021-02-10T11:27:41.928 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:41.923 7fb2657fa700 1 -- 172.21.2.102:0/669122719 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb260000c10 con 0x7fb2680b2110 2021-02-10T11:27:41.928 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:41.923 7fb26ed2b700 1 -- 172.21.2.102:0/669122719 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2680b2510 con 0x7fb2680b2110 2021-02-10T11:27:41.928 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:41.923 7fb2657fa700 1 -- 172.21.2.102:0/669122719 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb260005770 con 0x7fb2680b2110 2021-02-10T11:27:41.928 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:41.923 7fb26ed2b700 1 -- 172.21.2.102:0/669122719 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2680b2510 con 0x7fb2680b2110 2021-02-10T11:27:41.928 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:41.924 7fb2657fa700 1 -- 172.21.2.102:0/669122719 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fb260006030 con 0x7fb2680b2110 2021-02-10T11:27:41.928 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:41.924 7fb2657fa700 1 --2- 172.21.2.102:0/669122719 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb24c0186d0 0x7fb24c01abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:41.928 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:41.924 7fb2657fa700 1 -- 172.21.2.102:0/669122719 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fb260032660 con 0x7fb2680b2110 2021-02-10T11:27:41.928 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:41.924 7fb26cb47700 1 --2- 172.21.2.102:0/669122719 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb24c0186d0 0x7fb24c01abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:41.928 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:41.924 7fb26cb47700 1 --2- 172.21.2.102:0/669122719 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb24c0186d0 0x7fb24c01abd0 crc :-1 s=READY pgs=802 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:41.956 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:41.956 7fb26ed2b700 1 -- 172.21.2.102:0/669122719 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2680ebc00 con 0x7fb2680b2110 2021-02-10T11:27:41.958 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:41.959 7fb2657fa700 1 -- 172.21.2.102:0/669122719 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fb260000c10 con 0x7fb2680b2110 2021-02-10T11:27:42.084 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:42.084 7fb26ed2b700 1 -- 172.21.2.102:0/669122719 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb2680ebc00 con 0x7fb2680b2110 2021-02-10T11:27:42.084 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:42.084 7fb2657fa700 1 -- 172.21.2.102:0/669122719 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v84) v1 ==== 74+0+14009 (secure 0 0 0) 0x7fb260006030 con 0x7fb2680b2110 2021-02-10T11:27:42.085 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:27:42.085 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":84,"fsid":"68db30ea-dbce-4aef-a6df-f01c6fea0513","created":"2021-02-10 11:03:06.331557","modified":"2021-02-10 11:27:41.479288","last_up_change":"2021-02-10 11:03:32.690562","last_in_change":"2021-02-10 11:03:32.690562","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:03:37.322519","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:27:39.452483","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":17,"pool_name":"cephfs_data","create_time":"2021-02-10 11:27:40.475662","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"be691b4d-a039-408a-af80-9d53f02b6f1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6818","nonce":36088},{"type":"v1","addr":"172.21.2.102:6820","nonce":36088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6822","nonce":36088},{"type":"v1","addr":"172.21.2.102:6824","nonce":36088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6830","nonce":36088},{"type":"v1","addr":"172.21.2.102:6832","nonce":36088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6826","nonce":36088},{"type":"v1","addr":"172.21.2.102:6828","nonce":36088}]},"public_addr":"172.21.2.102:6820/36088","cluster_addr":"172.21.2.102:6824/36088","heartbeat_back_addr":"172.21.2.102:6832/36088","heartbeat_front_addr":"172.21.2.102:6828/36088","state":["exists","up"]},{"osd":1,"uuid":"64e3ccc8-9c69-4d7c-865e-af8e83822635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6819","nonce":36087},{"type":"v1","addr":"172.21.2.102:6821","nonce":36087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6823","nonce":36087},{"type":"v1","addr":"172.21.2.102:6825","nonce":36087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6831","nonce":36087},{"type":"v1","addr":"172.21.2.102:6833","nonce":36087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6827","nonce":36087},{"type":"v1","addr":"172.21.2.102:6829","nonce":36087}]},"public_addr":"172.21.2.102:6821/36087","cluster_addr":"172.21.2.102:6825/36087","heartbeat_back_addr":"172.21.2.102:6833/36087","heartbeat_front_addr":"172.21.2.102:6829/36087","state":["exists","up"]},{"osd":2,"uuid":"153700af-747f-481f-bdf4-d6072d3a842f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6810","nonce":36089},{"type":"v1","addr":"172.21.2.102:6811","nonce":36089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6812","nonce":36089},{"type":"v1","addr":"172.21.2.102:6813","nonce":36089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6816","nonce":36089},{"type":"v1","addr":"172.21.2.102:6817","nonce":36089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6814","nonce":36089},{"type":"v1","addr":"172.21.2.102:6815","nonce":36089}]},"public_addr":"172.21.2.102:6811/36089","cluster_addr":"172.21.2.102:6813/36089","heartbeat_back_addr":"172.21.2.102:6817/36089","heartbeat_front_addr":"172.21.2.102:6815/36089","state":["exists","up"]},{"osd":3,"uuid":"cbc678f4-83a9-45e6-8830-2f7c232b1ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6802","nonce":36090},{"type":"v1","addr":"172.21.2.102:6803","nonce":36090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6804","nonce":36090},{"type":"v1","addr":"172.21.2.102:6805","nonce":36090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6808","nonce":36090},{"type":"v1","addr":"172.21.2.102:6809","nonce":36090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6806","nonce":36090},{"type":"v1","addr":"172.21.2.102:6807","nonce":36090}]},"public_addr":"172.21.2.102:6803/36090","cluster_addr":"172.21.2.102:6805/36090","heartbeat_back_addr":"172.21.2.102:6809/36090","heartbeat_front_addr":"172.21.2.102:6807/36090","state":["exists","up"]},{"osd":4,"uuid":"c01e4008-b258-4f10-afd2-9e7f37214351","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6808","nonce":36368},{"type":"v1","addr":"172.21.2.113:6811","nonce":36368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6814","nonce":36368},{"type":"v1","addr":"172.21.2.113:6818","nonce":36368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6826","nonce":36368},{"type":"v1","addr":"172.21.2.113:6828","nonce":36368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6820","nonce":36368},{"type":"v1","addr":"172.21.2.113:6824","nonce":36368}]},"public_addr":"172.21.2.113:6811/36368","cluster_addr":"172.21.2.113:6818/36368","heartbeat_back_addr":"172.21.2.113:6828/36368","heartbeat_front_addr":"172.21.2.113:6824/36368","state":["exists","up"]},{"osd":5,"uuid":"551f6007-daf8-4a98-8ee0-c2b16bcccb5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6802","nonce":36366},{"type":"v1","addr":"172.21.2.113:6805","nonce":36366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6807","nonce":36366},{"type":"v1","addr":"172.21.2.113:6810","nonce":36366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6819","nonce":36366},{"type":"v1","addr":"172.21.2.113:6822","nonce":36366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6812","nonce":36366},{"type":"v1","addr":"172.21.2.113:6815","nonce":36366}]},"public_addr":"172.21.2.113:6805/36366","cluster_addr":"172.21.2.113:6810/36366","heartbeat_back_addr":"172.21.2.113:6822/36366","heartbeat_front_addr":"172.21.2.113:6815/36366","state":["exists","up"]},{"osd":6,"uuid":"bf67c040-0e50-48be-b102-c3943bc9ebef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6800","nonce":36367},{"type":"v1","addr":"172.21.2.113:6801","nonce":36367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6803","nonce":36367},{"type":"v1","addr":"172.21.2.113:6804","nonce":36367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6813","nonce":36367},{"type":"v1","addr":"172.21.2.113:6816","nonce":36367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6806","nonce":36367},{"type":"v1","addr":"172.21.2.113:6809","nonce":36367}]},"public_addr":"172.21.2.113:6801/36367","cluster_addr":"172.21.2.113:6804/36367","heartbeat_back_addr":"172.21.2.113:6816/36367","heartbeat_front_addr":"172.21.2.113:6809/36367","state":["exists","up"]},{"osd":7,"uuid":"f5461f80-018d-428f-be89-2eb2bcbe7bb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6817","nonce":36365},{"type":"v1","addr":"172.21.2.113:6821","nonce":36365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6823","nonce":36365},{"type":"v1","addr":"172.21.2.113:6825","nonce":36365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6830","nonce":36365},{"type":"v1","addr":"172.21.2.113:6831","nonce":36365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6827","nonce":36365},{"type":"v1","addr":"172.21.2.113:6829","nonce":36365}]},"public_addr":"172.21.2.113:6821/36365","cluster_addr":"172.21.2.113:6825/36365","heartbeat_back_addr":"172.21.2.113:6831/36365","heartbeat_front_addr":"172.21.2.113:6829/36365","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:27:42.086 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:42.086 7fb26ed2b700 1 -- 172.21.2.102:0/669122719 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb24c0186d0 msgr2=0x7fb24c01abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:42.086 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:42.086 7fb26ed2b700 1 --2- 172.21.2.102:0/669122719 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb24c0186d0 0x7fb24c01abd0 crc :-1 s=READY pgs=802 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:42.086 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:42.086 7fb26ed2b700 1 -- 172.21.2.102:0/669122719 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb2680b2110 msgr2=0x7fb2681b7a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:42.086 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:42.086 7fb26ed2b700 1 --2- 172.21.2.102:0/669122719 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb2680b2110 0x7fb2681b7a00 secure :-1 s=READY pgs=1776 cs=0 l=1 rev1=1 rx=0x7fb260006f10 tx=0x7fb26000b990).stop 2021-02-10T11:27:42.087 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:42.086 7fb26ed2b700 1 -- 172.21.2.102:0/669122719 shutdown_connections 2021-02-10T11:27:42.087 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:42.086 7fb26ed2b700 1 --2- 172.21.2.102:0/669122719 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb2680b2110 0x7fb2681b7a00 unknown :-1 s=CLOSED pgs=1776 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:42.087 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:42.086 7fb26ed2b700 1 --2- 172.21.2.102:0/669122719 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb2681b8ed0 0x7fb2680b1af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:42.087 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:42.086 7fb26ed2b700 1 --2- 172.21.2.102:0/669122719 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb24c0186d0 0x7fb24c01abd0 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:42.087 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:42.086 7fb26ed2b700 1 --2- 172.21.2.102:0/669122719 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb2681b8ad0 0x7fb268187e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:42.087 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:42.086 7fb26ed2b700 1 -- 172.21.2.102:0/669122719 >> 172.21.2.102:0/669122719 conn(0x7fb2680ed810 msgr2=0x7fb268094980 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:42.087 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:42.086 7fb26ed2b700 1 -- 172.21.2.102:0/669122719 shutdown_connections 2021-02-10T11:27:42.087 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:42.086 7fb26ed2b700 1 -- 172.21.2.102:0/669122719 wait complete. 2021-02-10T11:27:42.095 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:27:42.522 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:42.506 7fbf2f9e6700 1 Processor -- start 2021-02-10T11:27:42.523 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:42.506 7fbf2f9e6700 1 -- start start 2021-02-10T11:27:42.523 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:42.506 7fbf2f9e6700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbf280d0340 0x7fbf280ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:42.523 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:42.506 7fbf2f9e6700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fbf280d1c70 0x7fbf280efc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:42.523 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:42.506 7fbf2f9e6700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf280f0b10 con 0x7fbf280d1870 2021-02-10T11:27:42.523 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:42.506 7fbf2f9e6700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf280e5ed0 con 0x7fbf280d1c70 2021-02-10T11:27:42.523 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:42.507 7fbf2f9e6700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf281b0be0 con 0x7fbf280d0340 2021-02-10T11:27:42.523 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:42.507 7fbf2d001700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbf280d0340 0x7fbf280ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:42.523 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:42.507 7fbf2d802700 1 --1- >> v1:172.21.2.113:6789/0 conn(0x7fbf280d1870 0x7fbf281b03f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.113:6789/0 says I am v1:172.21.2.102:59298/0 (socket says 172.21.2.102:59298) 2021-02-10T11:27:42.523 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:42.507 7fbf2d802700 1 -- 172.21.2.102:0/4130906751 learned_addr learned my addr 172.21.2.102:0/4130906751 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:27:42.523 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:42.507 7fbf2e003700 1 --2- 172.21.2.102:0/4130906751 >> v2:172.21.2.113:3300/0 conn(0x7fbf280d1c70 0x7fbf280efc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:42.523 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:42.507 7fbf2c800700 1 -- 172.21.2.102:0/4130906751 <== mon.1 v1:172.21.2.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1603643433 0 0) 0x7fbf24000c60 con 0x7fbf280d1870 2021-02-10T11:27:42.523 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:42.507 7fbf2d001700 1 -- 172.21.2.102:0/4130906751 >> v2:172.21.2.113:3300/0 conn(0x7fbf280d1c70 msgr2=0x7fbf280efc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:27:42.524 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:42.507 7fbf2d001700 1 --2- 172.21.2.102:0/4130906751 >> v2:172.21.2.113:3300/0 conn(0x7fbf280d1c70 0x7fbf280efc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:42.524 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:42.507 7fbf2d001700 1 -- 172.21.2.102:0/4130906751 >> v1:172.21.2.113:6789/0 conn(0x7fbf280d1870 legacy=0x7fbf281b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:42.524 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:42.507 7fbf2d001700 1 -- 172.21.2.102:0/4130906751 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf280f20a0 con 0x7fbf280d0340 2021-02-10T11:27:42.524 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:42.507 7fbf2e003700 1 --2- 172.21.2.102:0/4130906751 >> v2:172.21.2.113:3300/0 conn(0x7fbf280d1c70 0x7fbf280efc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:27:42.524 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:42.507 7fbf2d001700 1 --2- 172.21.2.102:0/4130906751 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbf280d0340 0x7fbf280ef640 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7fbf1c006650 tx=0x7fbf1c009a80).ready entity=mon.2 client_cookie=ecbaa16f6e927db1 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:42.524 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:42.508 7fbf2c800700 1 -- 172.21.2.102:0/4130906751 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbf1c0009b0 con 0x7fbf280d0340 2021-02-10T11:27:42.524 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:42.508 7fbf2c800700 1 -- 172.21.2.102:0/4130906751 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf1c00c0c0 con 0x7fbf280d0340 2021-02-10T11:27:42.524 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:42.508 7fbf2c800700 1 -- 172.21.2.102:0/4130906751 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbf1c00c520 con 0x7fbf280d0340 2021-02-10T11:27:42.524 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:42.508 7fbf2f9e6700 1 -- 172.21.2.102:0/4130906751 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbf280d0340 msgr2=0x7fbf280ef640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:42.524 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:42.508 7fbf2f9e6700 1 --2- 172.21.2.102:0/4130906751 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbf280d0340 0x7fbf280ef640 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7fbf1c006650 tx=0x7fbf1c009a80).stop 2021-02-10T11:27:42.524 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:42.508 7fbf2f9e6700 1 -- 172.21.2.102:0/4130906751 shutdown_connections 2021-02-10T11:27:42.524 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:42.508 7fbf2f9e6700 1 --2- 172.21.2.102:0/4130906751 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbf280d0340 0x7fbf280ef640 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:42.524 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:42.508 7fbf2f9e6700 1 --2- 172.21.2.102:0/4130906751 >> v2:172.21.2.113:3300/0 conn(0x7fbf280d1c70 0x7fbf280efc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:42.524 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:42.508 7fbf2f9e6700 1 -- 172.21.2.102:0/4130906751 >> 172.21.2.102:0/4130906751 conn(0x7fbf280ce450 msgr2=0x7fbf280d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:42.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:42.508 7fbf2f9e6700 1 -- 172.21.2.102:0/4130906751 shutdown_connections 2021-02-10T11:27:42.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:42.509 7fbf2f9e6700 1 -- 172.21.2.102:0/4130906751 wait complete. 2021-02-10T11:27:42.526 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:42.523 7fbf2f9e6700 1 Processor -- start 2021-02-10T11:27:42.526 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:42.523 7fbf2f9e6700 1 -- start start 2021-02-10T11:27:42.526 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:42.523 7fbf2f9e6700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbf28195650 0x7fbf280b5560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:42.526 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:42.523 7fbf2f9e6700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbf280b5b80 0x7fbf281a2e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:42.526 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:42.523 7fbf2f9e6700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fbf280acfa0 0x7fbf281a3440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:42.526 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:42.523 7fbf2f9e6700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf280b6010 con 0x7fbf28195650 2021-02-10T11:27:42.526 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:42.523 7fbf2f9e6700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf281a3a60 con 0x7fbf280acfa0 2021-02-10T11:27:42.527 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:42.523 7fbf2f9e6700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf281a3c40 con 0x7fbf280b5b80 2021-02-10T11:27:42.527 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:42.524 7fbf2d802700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbf28195650 0x7fbf280b5560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:42.527 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:42.524 7fbf2d001700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbf280b5b80 0x7fbf281a2e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:42.527 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:42.524 7fbf2d802700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbf28195650 0x7fbf280b5560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:59758/0 (socket says 172.21.2.102:59758) 2021-02-10T11:27:42.527 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:42.524 7fbf2d802700 1 -- 172.21.2.102:0/2604710973 learned_addr learned my addr 172.21.2.102:0/2604710973 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:27:42.527 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:42.524 7fbf2e003700 1 --2- 172.21.2.102:0/2604710973 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fbf280acfa0 0x7fbf281a3440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:42.527 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:42.524 7fbf2d001700 1 -- 172.21.2.102:0/2604710973 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fbf280acfa0 msgr2=0x7fbf281a3440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:42.527 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:42.524 7fbf2d001700 1 --2- 172.21.2.102:0/2604710973 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fbf280acfa0 0x7fbf281a3440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:42.527 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:42.524 7fbf2d001700 1 -- 172.21.2.102:0/2604710973 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbf28195650 msgr2=0x7fbf280b5560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:42.527 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:42.524 7fbf2d001700 1 --2- 172.21.2.102:0/2604710973 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbf28195650 0x7fbf280b5560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:42.527 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:42.524 7fbf2d001700 1 -- 172.21.2.102:0/2604710973 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf280bcae0 con 0x7fbf280b5b80 2021-02-10T11:27:42.527 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:42.524 7fbf2d001700 1 --2- 172.21.2.102:0/2604710973 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbf280b5b80 0x7fbf281a2e20 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7fbf1c00b530 tx=0x7fbf1c00b8d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:42.527 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:42.524 7fbf1a7fc700 1 -- 172.21.2.102:0/2604710973 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbf1c006740 con 0x7fbf280b5b80 2021-02-10T11:27:42.528 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:42.524 7fbf1a7fc700 1 -- 172.21.2.102:0/2604710973 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf1c006740 con 0x7fbf280b5b80 2021-02-10T11:27:42.528 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:42.524 7fbf1a7fc700 1 -- 172.21.2.102:0/2604710973 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbf1c00bcc0 con 0x7fbf280b5b80 2021-02-10T11:27:42.528 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:42.524 7fbf2f9e6700 1 -- 172.21.2.102:0/2604710973 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf280bcae0 con 0x7fbf280b5b80 2021-02-10T11:27:42.528 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:42.524 7fbf2f9e6700 1 -- 172.21.2.102:0/2604710973 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf280bcae0 con 0x7fbf280b5b80 2021-02-10T11:27:42.528 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:42.525 7fbf1a7fc700 1 -- 172.21.2.102:0/2604710973 <== mon.2 v2:172.21.2.113:3301/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fbf1c0052e0 con 0x7fbf280b5b80 2021-02-10T11:27:42.528 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:42.525 7fbf1a7fc700 1 --2- 172.21.2.102:0/2604710973 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fbf100186d0 0x7fbf1001abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:42.528 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:42.525 7fbf1a7fc700 1 -- 172.21.2.102:0/2604710973 <== mon.2 v2:172.21.2.113:3301/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fbf1c0326e0 con 0x7fbf280b5b80 2021-02-10T11:27:42.528 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:42.525 7fbf2d802700 1 --2- 172.21.2.102:0/2604710973 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fbf100186d0 0x7fbf1001abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:42.528 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:42.525 7fbf2d802700 1 --2- 172.21.2.102:0/2604710973 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fbf100186d0 0x7fbf1001abd0 crc :-1 s=READY pgs=803 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:42.557 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:42.556 7fbf2f9e6700 1 -- 172.21.2.102:0/2604710973 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf280f0ab0 con 0x7fbf280b5b80 2021-02-10T11:27:42.559 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:42.559 7fbf1a7fc700 1 -- 172.21.2.102:0/2604710973 <== mon.2 v2:172.21.2.113:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fbf1c006030 con 0x7fbf280b5b80 2021-02-10T11:27:42.686 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:42.686 7fbf2f9e6700 1 -- 172.21.2.102:0/2604710973 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbf280f0ab0 con 0x7fbf280b5b80 2021-02-10T11:27:42.687 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:42.687 7fbf1a7fc700 1 -- 172.21.2.102:0/2604710973 <== mon.2 v2:172.21.2.113:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) v1 ==== 74+0+13996 (secure 0 0 0) 0x7fbf1c0052e0 con 0x7fbf280b5b80 2021-02-10T11:27:42.688 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:27:42.688 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":85,"fsid":"68db30ea-dbce-4aef-a6df-f01c6fea0513","created":"2021-02-10 11:03:06.331557","modified":"2021-02-10 11:27:42.481451","last_up_change":"2021-02-10 11:03:32.690562","last_in_change":"2021-02-10 11:03:32.690562","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:03:37.322519","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:27:39.452483","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":17,"pool_name":"cephfs_data","create_time":"2021-02-10 11:27:40.475662","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"be691b4d-a039-408a-af80-9d53f02b6f1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6818","nonce":36088},{"type":"v1","addr":"172.21.2.102:6820","nonce":36088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6822","nonce":36088},{"type":"v1","addr":"172.21.2.102:6824","nonce":36088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6830","nonce":36088},{"type":"v1","addr":"172.21.2.102:6832","nonce":36088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6826","nonce":36088},{"type":"v1","addr":"172.21.2.102:6828","nonce":36088}]},"public_addr":"172.21.2.102:6820/36088","cluster_addr":"172.21.2.102:6824/36088","heartbeat_back_addr":"172.21.2.102:6832/36088","heartbeat_front_addr":"172.21.2.102:6828/36088","state":["exists","up"]},{"osd":1,"uuid":"64e3ccc8-9c69-4d7c-865e-af8e83822635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6819","nonce":36087},{"type":"v1","addr":"172.21.2.102:6821","nonce":36087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6823","nonce":36087},{"type":"v1","addr":"172.21.2.102:6825","nonce":36087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6831","nonce":36087},{"type":"v1","addr":"172.21.2.102:6833","nonce":36087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6827","nonce":36087},{"type":"v1","addr":"172.21.2.102:6829","nonce":36087}]},"public_addr":"172.21.2.102:6821/36087","cluster_addr":"172.21.2.102:6825/36087","heartbeat_back_addr":"172.21.2.102:6833/36087","heartbeat_front_addr":"172.21.2.102:6829/36087","state":["exists","up"]},{"osd":2,"uuid":"153700af-747f-481f-bdf4-d6072d3a842f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6810","nonce":36089},{"type":"v1","addr":"172.21.2.102:6811","nonce":36089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6812","nonce":36089},{"type":"v1","addr":"172.21.2.102:6813","nonce":36089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6816","nonce":36089},{"type":"v1","addr":"172.21.2.102:6817","nonce":36089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6814","nonce":36089},{"type":"v1","addr":"172.21.2.102:6815","nonce":36089}]},"public_addr":"172.21.2.102:6811/36089","cluster_addr":"172.21.2.102:6813/36089","heartbeat_back_addr":"172.21.2.102:6817/36089","heartbeat_front_addr":"172.21.2.102:6815/36089","state":["exists","up"]},{"osd":3,"uuid":"cbc678f4-83a9-45e6-8830-2f7c232b1ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6802","nonce":36090},{"type":"v1","addr":"172.21.2.102:6803","nonce":36090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6804","nonce":36090},{"type":"v1","addr":"172.21.2.102:6805","nonce":36090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6808","nonce":36090},{"type":"v1","addr":"172.21.2.102:6809","nonce":36090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6806","nonce":36090},{"type":"v1","addr":"172.21.2.102:6807","nonce":36090}]},"public_addr":"172.21.2.102:6803/36090","cluster_addr":"172.21.2.102:6805/36090","heartbeat_back_addr":"172.21.2.102:6809/36090","heartbeat_front_addr":"172.21.2.102:6807/36090","state":["exists","up"]},{"osd":4,"uuid":"c01e4008-b258-4f10-afd2-9e7f37214351","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6808","nonce":36368},{"type":"v1","addr":"172.21.2.113:6811","nonce":36368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6814","nonce":36368},{"type":"v1","addr":"172.21.2.113:6818","nonce":36368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6826","nonce":36368},{"type":"v1","addr":"172.21.2.113:6828","nonce":36368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6820","nonce":36368},{"type":"v1","addr":"172.21.2.113:6824","nonce":36368}]},"public_addr":"172.21.2.113:6811/36368","cluster_addr":"172.21.2.113:6818/36368","heartbeat_back_addr":"172.21.2.113:6828/36368","heartbeat_front_addr":"172.21.2.113:6824/36368","state":["exists","up"]},{"osd":5,"uuid":"551f6007-daf8-4a98-8ee0-c2b16bcccb5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6802","nonce":36366},{"type":"v1","addr":"172.21.2.113:6805","nonce":36366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6807","nonce":36366},{"type":"v1","addr":"172.21.2.113:6810","nonce":36366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6819","nonce":36366},{"type":"v1","addr":"172.21.2.113:6822","nonce":36366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6812","nonce":36366},{"type":"v1","addr":"172.21.2.113:6815","nonce":36366}]},"public_addr":"172.21.2.113:6805/36366","cluster_addr":"172.21.2.113:6810/36366","heartbeat_back_addr":"172.21.2.113:6822/36366","heartbeat_front_addr":"172.21.2.113:6815/36366","state":["exists","up"]},{"osd":6,"uuid":"bf67c040-0e50-48be-b102-c3943bc9ebef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6800","nonce":36367},{"type":"v1","addr":"172.21.2.113:6801","nonce":36367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6803","nonce":36367},{"type":"v1","addr":"172.21.2.113:6804","nonce":36367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6813","nonce":36367},{"type":"v1","addr":"172.21.2.113:6816","nonce":36367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6806","nonce":36367},{"type":"v1","addr":"172.21.2.113:6809","nonce":36367}]},"public_addr":"172.21.2.113:6801/36367","cluster_addr":"172.21.2.113:6804/36367","heartbeat_back_addr":"172.21.2.113:6816/36367","heartbeat_front_addr":"172.21.2.113:6809/36367","state":["exists","up"]},{"osd":7,"uuid":"f5461f80-018d-428f-be89-2eb2bcbe7bb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6817","nonce":36365},{"type":"v1","addr":"172.21.2.113:6821","nonce":36365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6823","nonce":36365},{"type":"v1","addr":"172.21.2.113:6825","nonce":36365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6830","nonce":36365},{"type":"v1","addr":"172.21.2.113:6831","nonce":36365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6827","nonce":36365},{"type":"v1","addr":"172.21.2.113:6829","nonce":36365}]},"public_addr":"172.21.2.113:6821/36365","cluster_addr":"172.21.2.113:6825/36365","heartbeat_back_addr":"172.21.2.113:6831/36365","heartbeat_front_addr":"172.21.2.113:6829/36365","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:27:42.689 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:42.689 7fbf2f9e6700 1 -- 172.21.2.102:0/2604710973 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fbf100186d0 msgr2=0x7fbf1001abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:42.689 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:42.689 7fbf2f9e6700 1 --2- 172.21.2.102:0/2604710973 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fbf100186d0 0x7fbf1001abd0 crc :-1 s=READY pgs=803 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:42.689 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:42.689 7fbf2f9e6700 1 -- 172.21.2.102:0/2604710973 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbf280b5b80 msgr2=0x7fbf281a2e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:42.689 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:42.689 7fbf2f9e6700 1 --2- 172.21.2.102:0/2604710973 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbf280b5b80 0x7fbf281a2e20 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7fbf1c00b530 tx=0x7fbf1c00b8d0).stop 2021-02-10T11:27:42.689 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:42.689 7fbf2f9e6700 1 -- 172.21.2.102:0/2604710973 shutdown_connections 2021-02-10T11:27:42.690 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:42.689 7fbf2f9e6700 1 --2- 172.21.2.102:0/2604710973 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fbf100186d0 0x7fbf1001abd0 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:42.690 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:42.689 7fbf2f9e6700 1 --2- 172.21.2.102:0/2604710973 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fbf280acfa0 0x7fbf281a3440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:42.690 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:42.689 7fbf2f9e6700 1 --2- 172.21.2.102:0/2604710973 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbf280b5b80 0x7fbf281a2e20 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:42.690 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:42.689 7fbf2f9e6700 1 --2- 172.21.2.102:0/2604710973 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbf28195650 0x7fbf280b5560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:42.690 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:42.689 7fbf2f9e6700 1 -- 172.21.2.102:0/2604710973 >> 172.21.2.102:0/2604710973 conn(0x7fbf280b3040 msgr2=0x7fbf280c3250 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:42.690 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:42.689 7fbf2f9e6700 1 -- 172.21.2.102:0/2604710973 shutdown_connections 2021-02-10T11:27:42.690 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:42.689 7fbf2f9e6700 1 -- 172.21.2.102:0/2604710973 wait complete. 2021-02-10T11:27:42.698 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2021-02-10T11:27:43.129 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:43.112 7f0cf440f700 1 Processor -- start 2021-02-10T11:27:43.129 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:43.112 7f0cf440f700 1 -- start start 2021-02-10T11:27:43.129 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:43.113 7f0cf440f700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f0cec0d1c60 0x7f0cec0cef50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:43.130 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:43.113 7f0cf440f700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0cec0cf570 0x7f0cec0cf970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:43.130 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:43.113 7f0cf440f700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cec0fe1b0 con 0x7f0cec0d1860 2021-02-10T11:27:43.130 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:43.113 7f0cf440f700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0cec1b0ca0 con 0x7f0cec0d1c60 2021-02-10T11:27:43.130 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:43.113 7f0cf440f700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cec109a50 con 0x7f0cec0cf570 2021-02-10T11:27:43.130 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:43.113 7f0cf1a2a700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f0cec0d1c60 0x7f0cec0cef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:43.130 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:43.113 7f0cf1a2a700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f0cec0d1c60 0x7f0cec0cef50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:59762/0 (socket says 172.21.2.102:59762) 2021-02-10T11:27:43.130 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:43.113 7f0cf1a2a700 1 -- 172.21.2.102:0/664724466 learned_addr learned my addr 172.21.2.102:0/664724466 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:27:43.130 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:43.113 7f0cf2a2c700 1 --2- 172.21.2.102:0/664724466 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0cec0cf570 0x7f0cec0cf970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:43.130 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:43.114 7f0cf1a2a700 1 -- 172.21.2.102:0/664724466 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0cec0cf570 msgr2=0x7f0cec0cf970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:27:43.130 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:43.114 7f0cf1a2a700 1 --2- 172.21.2.102:0/664724466 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0cec0cf570 0x7f0cec0cf970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:43.130 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:43.114 7f0cf1a2a700 1 -- 172.21.2.102:0/664724466 >> v1:172.21.2.113:6789/0 conn(0x7f0cec0d1860 legacy=0x7f0cec1b04b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:43.131 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:43.114 7f0cf1a2a700 1 -- 172.21.2.102:0/664724466 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cec088070 con 0x7f0cec0d1c60 2021-02-10T11:27:43.131 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:43.114 7f0cf1a2a700 1 --2- 172.21.2.102:0/664724466 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0cec0d1c60 0x7f0cec0cef50 secure :-1 s=READY pgs=1777 cs=0 l=1 rev1=1 rx=0x7f0ce8006700 tx=0x7f0ce8009b80).ready entity=mon.0 client_cookie=616bb7b2c8251d45 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:43.131 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:43.114 7f0cf1229700 1 -- 172.21.2.102:0/664724466 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0ce800c0c0 con 0x7f0cec0d1c60 2021-02-10T11:27:43.131 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:43.114 7f0cf1229700 1 -- 172.21.2.102:0/664724466 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ce800c440 con 0x7f0cec0d1c60 2021-02-10T11:27:43.131 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:43.114 7f0cf1229700 1 -- 172.21.2.102:0/664724466 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0ce800c960 con 0x7f0cec0d1c60 2021-02-10T11:27:43.131 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:43.114 7f0cf440f700 1 -- 172.21.2.102:0/664724466 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0cec0d1c60 msgr2=0x7f0cec0cef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:43.131 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:43.114 7f0cf440f700 1 --2- 172.21.2.102:0/664724466 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0cec0d1c60 0x7f0cec0cef50 secure :-1 s=READY pgs=1777 cs=0 l=1 rev1=1 rx=0x7f0ce8006700 tx=0x7f0ce8009b80).stop 2021-02-10T11:27:43.131 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:43.115 7f0cf440f700 1 -- 172.21.2.102:0/664724466 shutdown_connections 2021-02-10T11:27:43.131 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:43.115 7f0cf440f700 1 --2- 172.21.2.102:0/664724466 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0cec0cf570 0x7f0cec0cf970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:43.131 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:43.115 7f0cf440f700 1 --2- 172.21.2.102:0/664724466 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0cec0d1c60 0x7f0cec0cef50 unknown :-1 s=CLOSED pgs=1777 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:43.131 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:43.115 7f0cf440f700 1 -- 172.21.2.102:0/664724466 >> 172.21.2.102:0/664724466 conn(0x7f0cec0ce410 msgr2=0x7f0cec0d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:43.131 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:43.115 7f0cf440f700 1 -- 172.21.2.102:0/664724466 shutdown_connections 2021-02-10T11:27:43.131 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:43.115 7f0cf440f700 1 -- 172.21.2.102:0/664724466 wait complete. 2021-02-10T11:27:43.133 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:43.130 7f0cf440f700 1 Processor -- start 2021-02-10T11:27:43.133 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:43.130 7f0cf440f700 1 -- start start 2021-02-10T11:27:43.133 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:43.130 7f0cf440f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0cec085270 0x7f0cec085670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:43.133 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:43.130 7f0cf440f700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0cec085c90 0x7f0cec086090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:43.133 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:43.130 7f0cf440f700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0cec0c2740 0x7f0cec0c2b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:43.133 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:43.130 7f0cf440f700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cec084cf0 con 0x7f0cec085270 2021-02-10T11:27:43.133 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:43.130 7f0cf440f700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cec0be1b0 con 0x7f0cec085c90 2021-02-10T11:27:43.133 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:43.130 7f0cf440f700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cec0be390 con 0x7f0cec0c2740 2021-02-10T11:27:43.133 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:43.130 7f0cf222b700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0cec085270 0x7f0cec085670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:43.133 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:43.130 7f0cf222b700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0cec085270 0x7f0cec085670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:59768/0 (socket says 172.21.2.102:59768) 2021-02-10T11:27:43.134 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:43.130 7f0cf222b700 1 -- 172.21.2.102:0/3178237621 learned_addr learned my addr 172.21.2.102:0/3178237621 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:27:43.134 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:43.131 7f0cf222b700 1 -- 172.21.2.102:0/3178237621 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0cec0c2740 msgr2=0x7f0cec0c2b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:43.134 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:43.131 7f0cf222b700 1 --2- 172.21.2.102:0/3178237621 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0cec0c2740 0x7f0cec0c2b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:43.134 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:43.131 7f0cf222b700 1 -- 172.21.2.102:0/3178237621 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0cec085c90 msgr2=0x7f0cec086090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:43.134 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:43.131 7f0cf222b700 1 --2- 172.21.2.102:0/3178237621 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0cec085c90 0x7f0cec086090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:43.134 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:43.131 7f0cf222b700 1 -- 172.21.2.102:0/3178237621 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cec0c3160 con 0x7f0cec085270 2021-02-10T11:27:43.134 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:43.131 7f0cf222b700 1 --2- 172.21.2.102:0/3178237621 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0cec085270 0x7f0cec085670 secure :-1 s=READY pgs=1778 cs=0 l=1 rev1=1 rx=0x7f0ce0006ed0 tx=0x7f0ce000b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:43.134 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:43.131 7f0cdeffd700 1 -- 172.21.2.102:0/3178237621 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0ce0006240 con 0x7f0cec085270 2021-02-10T11:27:43.134 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:43.131 7f0cdeffd700 1 -- 172.21.2.102:0/3178237621 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ce00054b0 con 0x7f0cec085270 2021-02-10T11:27:43.134 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:43.131 7f0cdeffd700 1 -- 172.21.2.102:0/3178237621 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0ce00059d0 con 0x7f0cec085270 2021-02-10T11:27:43.134 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:43.131 7f0cf440f700 1 -- 172.21.2.102:0/3178237621 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0cec0c3160 con 0x7f0cec085270 2021-02-10T11:27:43.134 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:43.131 7f0cf440f700 1 -- 172.21.2.102:0/3178237621 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cec0c3160 con 0x7f0cec085270 2021-02-10T11:27:43.134 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:43.132 7f0cdeffd700 1 -- 172.21.2.102:0/3178237621 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f0ce0006030 con 0x7f0cec085270 2021-02-10T11:27:43.135 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:43.132 7f0cdeffd700 1 --2- 172.21.2.102:0/3178237621 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0cd40186c0 0x7f0cd401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:43.135 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:43.132 7f0cdeffd700 1 -- 172.21.2.102:0/3178237621 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f0ce00325e0 con 0x7f0cec085270 2021-02-10T11:27:43.135 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:43.132 7f0cf1a2a700 1 --2- 172.21.2.102:0/3178237621 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0cd40186c0 0x7f0cd401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:43.135 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:43.132 7f0cf1a2a700 1 --2- 172.21.2.102:0/3178237621 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0cd40186c0 0x7f0cd401abc0 crc :-1 s=READY pgs=804 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:43.163 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:43.163 7f0cf440f700 1 -- 172.21.2.102:0/3178237621 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0cec082a20 con 0x7f0cec085270 2021-02-10T11:27:43.166 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:43.166 7f0cdeffd700 1 -- 172.21.2.102:0/3178237621 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f0ce0000de0 con 0x7f0cec085270 2021-02-10T11:27:43.303 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:43.303 7f0cf440f700 1 -- 172.21.2.102:0/3178237621 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"} v 0) v1 -- 0x7f0cec082a20 con 0x7f0cec085270 2021-02-10T11:27:43.492 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:43.491 7f0cdeffd700 1 -- 172.21.2.102:0/3178237621 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"}]=0 v65) v1 ==== 118+0+0 (secure 0 0 0) 0x7f0ce0006030 con 0x7f0cec085270 2021-02-10T11:27:43.521 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:43.521 7f0cf440f700 1 -- 172.21.2.102:0/3178237621 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0cd40186c0 msgr2=0x7f0cd401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:43.521 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:43.521 7f0cf440f700 1 --2- 172.21.2.102:0/3178237621 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0cd40186c0 0x7f0cd401abc0 crc :-1 s=READY pgs=804 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:43.521 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:43.521 7f0cf440f700 1 -- 172.21.2.102:0/3178237621 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0cec085270 msgr2=0x7f0cec085670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:43.521 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:43.521 7f0cf440f700 1 --2- 172.21.2.102:0/3178237621 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0cec085270 0x7f0cec085670 secure :-1 s=READY pgs=1778 cs=0 l=1 rev1=1 rx=0x7f0ce0006ed0 tx=0x7f0ce000b960).stop 2021-02-10T11:27:43.521 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:43.521 7f0cf440f700 1 -- 172.21.2.102:0/3178237621 shutdown_connections 2021-02-10T11:27:43.522 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:43.521 7f0cf440f700 1 --2- 172.21.2.102:0/3178237621 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0cec0c2740 0x7f0cec0c2b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:43.522 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:43.521 7f0cf440f700 1 --2- 172.21.2.102:0/3178237621 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0cd40186c0 0x7f0cd401abc0 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:43.522 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:43.521 7f0cf440f700 1 --2- 172.21.2.102:0/3178237621 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0cec085c90 0x7f0cec086090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:43.522 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:43.521 7f0cf440f700 1 --2- 172.21.2.102:0/3178237621 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0cec085270 0x7f0cec085670 unknown :-1 s=CLOSED pgs=1778 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:43.522 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:43.521 7f0cf440f700 1 -- 172.21.2.102:0/3178237621 >> 172.21.2.102:0/3178237621 conn(0x7f0cec0b5f10 msgr2=0x7f0cec09f200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:43.522 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:43.521 7f0cf440f700 1 -- 172.21.2.102:0/3178237621 shutdown_connections 2021-02-10T11:27:43.522 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:43.521 7f0cf440f700 1 -- 172.21.2.102:0/3178237621 wait complete. 2021-02-10T11:27:43.530 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:27:43.959 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:43.941 7fd2d9f51700 1 Processor -- start 2021-02-10T11:27:43.959 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:43.941 7fd2d9f51700 1 -- start start 2021-02-10T11:27:43.959 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:43.941 7fd2d9f51700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd2d40d0340 0x7fd2d40ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:43.960 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:43.941 7fd2d9f51700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2d40f0b10 con 0x7fd2d40d1870 2021-02-10T11:27:43.960 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:43.941 7fd2d9f51700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2d41b0be0 con 0x7fd2d40d1c70 2021-02-10T11:27:43.960 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:43.941 7fd2d9f51700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2d40e5ed0 con 0x7fd2d40d0340 2021-02-10T11:27:43.960 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:43.942 7fd2d37fe700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fd2d40d1870 0x7fd2d41b03f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:52000/0 (socket says 172.21.2.102:52000) 2021-02-10T11:27:43.960 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:43.942 7fd2d37fe700 1 -- 172.21.2.102:0/3687491693 learned_addr learned my addr 172.21.2.102:0/3687491693 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:27:43.960 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:43.942 7fd2d27fc700 1 -- 172.21.2.102:0/3687491693 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1195848497 0 0) 0x7fd2bc001180 con 0x7fd2d40d1870 2021-02-10T11:27:43.960 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:43.942 7fd2d27fc700 1 -- 172.21.2.102:0/3687491693 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2c0002fe0 con 0x7fd2d40d1870 2021-02-10T11:27:43.960 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:43.942 7fd2d2ffd700 1 --2- 172.21.2.102:0/3687491693 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd2d40d0340 0x7fd2d40ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:43.960 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:43.942 7fd2d27fc700 1 -- 172.21.2.102:0/3687491693 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1598280512 0 0) 0x7fd2bc001180 con 0x7fd2d40d1870 2021-02-10T11:27:43.960 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:43.942 7fd2d27fc700 1 -- 172.21.2.102:0/3687491693 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd2d40d0340 msgr2=0x7fd2d40ef640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:27:43.960 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:43.942 7fd2d27fc700 1 --2- 172.21.2.102:0/3687491693 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd2d40d0340 0x7fd2d40ef640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:43.960 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:43.942 7fd2d27fc700 1 -- 172.21.2.102:0/3687491693 >> v1:172.21.2.113:6789/0 conn(0x7fd2d40d1c70 legacy=0x7fd2d40efc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:43.960 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:43.942 7fd2d27fc700 1 -- 172.21.2.102:0/3687491693 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2d40f0450 con 0x7fd2d40d1870 2021-02-10T11:27:43.961 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:43.943 7fd2d27fc700 1 -- 172.21.2.102:0/3687491693 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7fd2bc001680 con 0x7fd2d40d1870 2021-02-10T11:27:43.961 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:43.943 7fd2d27fc700 1 -- 172.21.2.102:0/3687491693 >> v1:172.21.2.102:6789/0 conn(0x7fd2d40d1870 legacy=0x7fd2d41b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:43.961 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:43.943 7fd2d27fc700 1 --2- 172.21.2.102:0/3687491693 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd2c0003aa0 0x7fd2c0005ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:43.961 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:43.943 7fd2d27fc700 1 --2- 172.21.2.102:0/3687491693 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd2c0006790 0x7fd2c0008c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:43.961 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:43.943 7fd2d27fc700 1 --2- 172.21.2.102:0/3687491693 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd2c0009500 0x7fd2c000ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:43.961 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:43.943 7fd2d27fc700 1 -- 172.21.2.102:0/3687491693 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2c000c2d0 con 0x7fd2c0003aa0 2021-02-10T11:27:43.961 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:43.943 7fd2d27fc700 1 -- 172.21.2.102:0/3687491693 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2c000c4f0 con 0x7fd2c0009500 2021-02-10T11:27:43.961 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:43.943 7fd2d27fc700 1 -- 172.21.2.102:0/3687491693 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2c000c710 con 0x7fd2c0006790 2021-02-10T11:27:43.961 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:43.943 7fd2d37fe700 1 --2- 172.21.2.102:0/3687491693 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd2c0003aa0 0x7fd2c0005ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:43.961 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:43.943 7fd2d2ffd700 1 --2- 172.21.2.102:0/3687491693 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd2c0006790 0x7fd2c0008c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:43.961 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:43.943 7fd2d3fff700 1 --2- 172.21.2.102:0/3687491693 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd2c0009500 0x7fd2c000ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:43.961 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:43.943 7fd2d37fe700 1 -- 172.21.2.102:0/3687491693 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd2c0006790 msgr2=0x7fd2c0008c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:27:43.961 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:43.943 7fd2d37fe700 1 --2- 172.21.2.102:0/3687491693 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd2c0006790 0x7fd2c0008c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:43.961 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:43.943 7fd2d37fe700 1 -- 172.21.2.102:0/3687491693 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd2c0009500 msgr2=0x7fd2c000ba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:27:43.962 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:43.943 7fd2d37fe700 1 --2- 172.21.2.102:0/3687491693 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd2c0009500 0x7fd2c000ba20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:43.962 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:43.943 7fd2d37fe700 1 -- 172.21.2.102:0/3687491693 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2c0050950 con 0x7fd2c0003aa0 2021-02-10T11:27:43.962 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:43.943 7fd2d37fe700 1 --2- 172.21.2.102:0/3687491693 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd2c0003aa0 0x7fd2c0005ef0 secure :-1 s=READY pgs=1780 cs=0 l=1 rev1=1 rx=0x7fd2bc0014e0 tx=0x7fd2bc007ca0).ready entity=mon.0 client_cookie=f6601c21e63156af server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:43.962 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:43.943 7fd2d27fc700 1 -- 172.21.2.102:0/3687491693 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd2bc0070c0 con 0x7fd2c0003aa0 2021-02-10T11:27:43.962 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:43.943 7fd2d27fc700 1 -- 172.21.2.102:0/3687491693 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2bc0043a0 con 0x7fd2c0003aa0 2021-02-10T11:27:43.962 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:43.943 7fd2d27fc700 1 -- 172.21.2.102:0/3687491693 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd2bc004810 con 0x7fd2c0003aa0 2021-02-10T11:27:43.962 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:43.943 7fd2d9f51700 1 -- 172.21.2.102:0/3687491693 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd2c0003aa0 msgr2=0x7fd2c0005ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:43.962 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:43.943 7fd2d9f51700 1 --2- 172.21.2.102:0/3687491693 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd2c0003aa0 0x7fd2c0005ef0 secure :-1 s=READY pgs=1780 cs=0 l=1 rev1=1 rx=0x7fd2bc0014e0 tx=0x7fd2bc007ca0).stop 2021-02-10T11:27:43.962 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:43.943 7fd2d3fff700 1 -- 172.21.2.102:0/3687491693 reap_dead start 2021-02-10T11:27:43.962 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:43.944 7fd2d9f51700 1 -- 172.21.2.102:0/3687491693 shutdown_connections 2021-02-10T11:27:43.962 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:43.944 7fd2d9f51700 1 -- 172.21.2.102:0/3687491693 >> 172.21.2.102:0/3687491693 conn(0x7fd2d40ce450 msgr2=0x7fd2d40d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:43.962 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:43.944 7fd2d9f51700 1 -- 172.21.2.102:0/3687491693 shutdown_connections 2021-02-10T11:27:43.963 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:43.944 7fd2d9f51700 1 -- 172.21.2.102:0/3687491693 wait complete. 2021-02-10T11:27:43.963 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:43.959 7fd2d9f51700 1 Processor -- start 2021-02-10T11:27:43.963 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:43.959 7fd2d9f51700 1 -- start start 2021-02-10T11:27:43.964 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:43.959 7fd2d9f51700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd2d40c33c0 0x7fd2d40ec230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:43.964 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:43.959 7fd2d9f51700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd2d40b5e30 0x7fd2d419ec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:43.964 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:43.959 7fd2d9f51700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd2d419f260 0x7fd2d40ed010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:43.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:43.959 7fd2d9f51700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2d40c3810 con 0x7fd2d419f260 2021-02-10T11:27:43.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:43.959 7fd2d9f51700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2d40eca90 con 0x7fd2d40c33c0 2021-02-10T11:27:43.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:43.959 7fd2d9f51700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2d40b2790 con 0x7fd2d40b5e30 2021-02-10T11:27:43.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:43.959 7fd2d3fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd2d419f260 0x7fd2d40ed010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:43.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:43.959 7fd2d3fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd2d419f260 0x7fd2d40ed010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:59788/0 (socket says 172.21.2.102:59788) 2021-02-10T11:27:43.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:43.959 7fd2d3fff700 1 -- 172.21.2.102:0/3106751105 learned_addr learned my addr 172.21.2.102:0/3106751105 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:27:43.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:43.960 7fd2d2ffd700 1 --2- 172.21.2.102:0/3106751105 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd2d40b5e30 0x7fd2d419ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:43.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:43.960 7fd2d37fe700 1 --2- 172.21.2.102:0/3106751105 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd2d40c33c0 0x7fd2d40ec230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:43.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:43.960 7fd2d3fff700 1 -- 172.21.2.102:0/3106751105 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd2d40b5e30 msgr2=0x7fd2d419ec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:43.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:43.960 7fd2d3fff700 1 --2- 172.21.2.102:0/3106751105 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd2d40b5e30 0x7fd2d419ec40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:43.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:43.960 7fd2d3fff700 1 -- 172.21.2.102:0/3106751105 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd2d40c33c0 msgr2=0x7fd2d40ec230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:43.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:43.960 7fd2d3fff700 1 --2- 172.21.2.102:0/3106751105 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd2d40c33c0 0x7fd2d40ec230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:43.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:43.960 7fd2d3fff700 1 -- 172.21.2.102:0/3106751105 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2d419f660 con 0x7fd2d419f260 2021-02-10T11:27:43.966 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:43.960 7fd2d3fff700 1 --2- 172.21.2.102:0/3106751105 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd2d419f260 0x7fd2d40ed010 secure :-1 s=READY pgs=1781 cs=0 l=1 rev1=1 rx=0x7fd2cc00b610 tx=0x7fd2cc00b9b0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:43.966 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:43.960 7fd2cbfff700 1 -- 172.21.2.102:0/3106751105 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd2cc00bda0 con 0x7fd2d419f260 2021-02-10T11:27:43.966 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:43.960 7fd2cbfff700 1 -- 172.21.2.102:0/3106751105 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2cc002030 con 0x7fd2d419f260 2021-02-10T11:27:43.966 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:43.960 7fd2d9f51700 1 -- 172.21.2.102:0/3106751105 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2d419f660 con 0x7fd2d419f260 2021-02-10T11:27:43.966 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:43.960 7fd2cbfff700 1 -- 172.21.2.102:0/3106751105 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd2cc0024d0 con 0x7fd2d419f260 2021-02-10T11:27:43.966 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:43.960 7fd2d9f51700 1 -- 172.21.2.102:0/3106751105 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2d419f660 con 0x7fd2d419f260 2021-02-10T11:27:43.966 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:43.961 7fd2cbfff700 1 -- 172.21.2.102:0/3106751105 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd2cc002420 con 0x7fd2d419f260 2021-02-10T11:27:43.966 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:43.961 7fd2cbfff700 1 --2- 172.21.2.102:0/3106751105 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd2b8018730 0x7fd2b801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:43.966 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:43.961 7fd2cbfff700 1 -- 172.21.2.102:0/3106751105 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fd2cc032630 con 0x7fd2d419f260 2021-02-10T11:27:43.966 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:43.961 7fd2d37fe700 1 --2- 172.21.2.102:0/3106751105 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd2b8018730 0x7fd2b801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:43.966 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:43.961 7fd2d37fe700 1 --2- 172.21.2.102:0/3106751105 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd2b8018730 0x7fd2b801ac10 crc :-1 s=READY pgs=805 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:43.991 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:43.991 7fd2d9f51700 1 -- 172.21.2.102:0/3106751105 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2d40ec850 con 0x7fd2d419f260 2021-02-10T11:27:43.994 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:43.994 7fd2cbfff700 1 -- 172.21.2.102:0/3106751105 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd2cc002030 con 0x7fd2d419f260 2021-02-10T11:27:44.121 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:44.121 7fd2d9f51700 1 -- 172.21.2.102:0/3106751105 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd2d40ec850 con 0x7fd2d419f260 2021-02-10T11:27:44.121 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:44.121 7fd2cbfff700 1 -- 172.21.2.102:0/3106751105 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 65 v65) v1 ==== 94+0+2101 (secure 0 0 0) 0x7fd2cc00bda0 con 0x7fd2d419f260 2021-02-10T11:27:44.122 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:27:44.122 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":65,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6110,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.102:6835/1471871278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6834","nonce":1471871278},{"type":"v1","addr":"172.21.2.102:6835","nonce":1471871278}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":61}],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:27:41.479375","modified":"2021-02-10 11:27:43.490605","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6107},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6107":{"gid":6107,"name":"b","rank":0,"incarnation":62,"state":"up:active","state_seq":3,"addr":"172.21.2.113:6833/4220238062","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6832","nonce":4220238062},{"type":"v1","addr":"172.21.2.113:6833","nonce":4220238062}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":8}]} 2021-02-10T11:27:44.122 INFO:teuthology.orchestra.run.gibba002.stderr:dumped fsmap epoch 65 2021-02-10T11:27:44.123 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:44.123 7fd2d9f51700 1 -- 172.21.2.102:0/3106751105 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd2b8018730 msgr2=0x7fd2b801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:44.123 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:44.123 7fd2d9f51700 1 --2- 172.21.2.102:0/3106751105 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd2b8018730 0x7fd2b801ac10 crc :-1 s=READY pgs=805 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:44.123 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:44.123 7fd2d9f51700 1 -- 172.21.2.102:0/3106751105 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd2d419f260 msgr2=0x7fd2d40ed010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:44.123 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:44.123 7fd2d9f51700 1 --2- 172.21.2.102:0/3106751105 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd2d419f260 0x7fd2d40ed010 secure :-1 s=READY pgs=1781 cs=0 l=1 rev1=1 rx=0x7fd2cc00b610 tx=0x7fd2cc00b9b0).stop 2021-02-10T11:27:44.124 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:44.123 7fd2d9f51700 1 -- 172.21.2.102:0/3106751105 shutdown_connections 2021-02-10T11:27:44.124 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:44.123 7fd2d9f51700 1 --2- 172.21.2.102:0/3106751105 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd2d419f260 0x7fd2d40ed010 unknown :-1 s=CLOSED pgs=1781 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:44.124 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:44.123 7fd2d9f51700 1 --2- 172.21.2.102:0/3106751105 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd2d40b5e30 0x7fd2d419ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:44.124 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:44.123 7fd2d9f51700 1 --2- 172.21.2.102:0/3106751105 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd2b8018730 0x7fd2b801ac10 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:44.124 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:44.123 7fd2d9f51700 1 --2- 172.21.2.102:0/3106751105 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd2d40c33c0 0x7fd2d40ec230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:44.124 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:44.123 7fd2d9f51700 1 -- 172.21.2.102:0/3106751105 >> 172.21.2.102:0/3106751105 conn(0x7fd2d40ebe30 msgr2=0x7fd2d408a880 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:44.124 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:44.123 7fd2d9f51700 1 -- 172.21.2.102:0/3106751105 shutdown_connections 2021-02-10T11:27:44.124 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:44.123 7fd2d9f51700 1 -- 172.21.2.102:0/3106751105 wait complete. 2021-02-10T11:27:44.132 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:27:44.562 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:44.545 7f9340eb5700 1 Processor -- start 2021-02-10T11:27:44.562 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:44.545 7f9340eb5700 1 -- start start 2021-02-10T11:27:44.562 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:44.546 7f9340eb5700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f933c0d1870 0x7f933c1b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:44.562 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:44.546 7f9340eb5700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f933c1b0a10 0x7f933c0cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:44.563 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:44.546 7f9340eb5700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f933c0f0b10 con 0x7f933c0d1c70 2021-02-10T11:27:44.563 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:44.546 7f9340eb5700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f933c0e5ed0 con 0x7f933c0d1870 2021-02-10T11:27:44.563 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:44.546 7f9340eb5700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f933c0f0650 con 0x7f933c1b0a10 2021-02-10T11:27:44.563 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:44.546 7f933ab13700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f933c0d1870 0x7f933c1b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:44.563 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:44.546 7f933ab13700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f933c0d1870 0x7f933c1b03f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:59796/0 (socket says 172.21.2.102:59796) 2021-02-10T11:27:44.563 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:44.546 7f933ab13700 1 -- 172.21.2.102:0/1263439600 learned_addr learned my addr 172.21.2.102:0/1263439600 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:27:44.563 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:44.547 7f933ab13700 1 -- 172.21.2.102:0/1263439600 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f933c1b0a10 msgr2=0x7f933c0cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:27:44.563 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:44.547 7f933ab13700 1 --2- 172.21.2.102:0/1263439600 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f933c1b0a10 0x7f933c0cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:44.563 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:44.547 7f933ab13700 1 -- 172.21.2.102:0/1263439600 >> v1:172.21.2.113:6789/0 conn(0x7f933c0d1c70 legacy=0x7f933c0cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:27:44.563 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:44.547 7f933ab13700 1 -- 172.21.2.102:0/1263439600 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f933c0f0870 con 0x7f933c0d1870 2021-02-10T11:27:44.563 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:44.547 7f933ab13700 1 --2- 172.21.2.102:0/1263439600 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f933c0d1870 0x7f933c1b03f0 secure :-1 s=READY pgs=1782 cs=0 l=1 rev1=1 rx=0x7f932c006700 tx=0x7f932c009b80).ready entity=mon.0 client_cookie=d484a1c5cd227fb7 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:44.563 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:44.547 7f9339b11700 1 -- 172.21.2.102:0/1263439600 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f932c00c0c0 con 0x7f933c0d1870 2021-02-10T11:27:44.563 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:44.547 7f9339b11700 1 -- 172.21.2.102:0/1263439600 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f932c00c340 con 0x7f933c0d1870 2021-02-10T11:27:44.564 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:44.547 7f9339b11700 1 -- 172.21.2.102:0/1263439600 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f932c00c820 con 0x7f933c0d1870 2021-02-10T11:27:44.564 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:44.547 7f9340eb5700 1 -- 172.21.2.102:0/1263439600 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f933c0d1870 msgr2=0x7f933c1b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:44.564 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:44.547 7f9340eb5700 1 --2- 172.21.2.102:0/1263439600 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f933c0d1870 0x7f933c1b03f0 secure :-1 s=READY pgs=1782 cs=0 l=1 rev1=1 rx=0x7f932c006700 tx=0x7f932c009b80).stop 2021-02-10T11:27:44.564 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:44.548 7f9340eb5700 1 -- 172.21.2.102:0/1263439600 shutdown_connections 2021-02-10T11:27:44.564 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:44.548 7f9340eb5700 1 --2- 172.21.2.102:0/1263439600 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f933c1b0a10 0x7f933c0cf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:44.564 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:44.548 7f9340eb5700 1 --2- 172.21.2.102:0/1263439600 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f933c0d1870 0x7f933c1b03f0 unknown :-1 s=CLOSED pgs=1782 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:44.564 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:44.548 7f9340eb5700 1 -- 172.21.2.102:0/1263439600 >> 172.21.2.102:0/1263439600 conn(0x7f933c0ce450 msgr2=0x7f933c0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:44.564 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:44.548 7f9340eb5700 1 -- 172.21.2.102:0/1263439600 shutdown_connections 2021-02-10T11:27:44.564 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:44.548 7f9340eb5700 1 -- 172.21.2.102:0/1263439600 wait complete. 2021-02-10T11:27:44.566 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:44.563 7f9340eb5700 1 Processor -- start 2021-02-10T11:27:44.566 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:44.563 7f9340eb5700 1 -- start start 2021-02-10T11:27:44.566 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:44.563 7f9340eb5700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f933c197460 0x7f933c18a540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:44.566 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:44.563 7f9340eb5700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f933c18ab60 0x7f933c0b77d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:44.566 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:44.563 7f9340eb5700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f933c0b7df0 0x7f933c0b81f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:44.566 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:44.563 7f9340eb5700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f933c1aa280 con 0x7f933c197460 2021-02-10T11:27:44.566 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:44.563 7f9340eb5700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f933c1a7940 con 0x7f933c0b7df0 2021-02-10T11:27:44.566 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:44.563 7f9340eb5700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f933c1a9e20 con 0x7f933c18ab60 2021-02-10T11:27:44.566 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:44.563 7f933ab13700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f933c197460 0x7f933c18a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:44.566 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:44.563 7f933ab13700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f933c197460 0x7f933c18a540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:59800/0 (socket says 172.21.2.102:59800) 2021-02-10T11:27:44.567 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:44.563 7f933ab13700 1 -- 172.21.2.102:0/2150627018 learned_addr learned my addr 172.21.2.102:0/2150627018 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:27:44.567 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:44.564 7f933ab13700 1 -- 172.21.2.102:0/2150627018 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f933c18ab60 msgr2=0x7f933c0b77d0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:27:44.567 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:44.564 7f933ab13700 1 --2- 172.21.2.102:0/2150627018 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f933c18ab60 0x7f933c0b77d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:44.567 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:44.564 7f933ab13700 1 -- 172.21.2.102:0/2150627018 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f933c0b7df0 msgr2=0x7f933c0b81f0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:27:44.567 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:44.564 7f933ab13700 1 --2- 172.21.2.102:0/2150627018 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f933c0b7df0 0x7f933c0b81f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:44.567 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:44.564 7f933ab13700 1 -- 172.21.2.102:0/2150627018 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f933c1aa780 con 0x7f933c197460 2021-02-10T11:27:44.567 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:44.564 7f933ab13700 1 --2- 172.21.2.102:0/2150627018 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f933c197460 0x7f933c18a540 secure :-1 s=READY pgs=1783 cs=0 l=1 rev1=1 rx=0x7f932c00b650 tx=0x7f932c00b9f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:44.567 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:44.564 7f932b7fe700 1 -- 172.21.2.102:0/2150627018 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f932c00bde0 con 0x7f933c197460 2021-02-10T11:27:44.567 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:44.564 7f932b7fe700 1 -- 172.21.2.102:0/2150627018 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f932c002030 con 0x7f933c197460 2021-02-10T11:27:44.567 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:44.564 7f932b7fe700 1 -- 172.21.2.102:0/2150627018 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f932c002550 con 0x7f933c197460 2021-02-10T11:27:44.567 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:44.564 7f9340eb5700 1 -- 172.21.2.102:0/2150627018 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f933c1aa780 con 0x7f933c197460 2021-02-10T11:27:44.567 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:44.564 7f9340eb5700 1 -- 172.21.2.102:0/2150627018 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f933c1aa780 con 0x7f933c197460 2021-02-10T11:27:44.567 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:44.565 7f932b7fe700 1 -- 172.21.2.102:0/2150627018 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f932c0064b0 con 0x7f933c197460 2021-02-10T11:27:44.568 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:44.565 7f932b7fe700 1 --2- 172.21.2.102:0/2150627018 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f93200186c0 0x7f932001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:44.568 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:44.565 7f932b7fe700 1 -- 172.21.2.102:0/2150627018 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f932c032590 con 0x7f933c197460 2021-02-10T11:27:44.568 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:44.565 7f933a312700 1 --2- 172.21.2.102:0/2150627018 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f93200186c0 0x7f932001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:44.568 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:44.566 7f933a312700 1 --2- 172.21.2.102:0/2150627018 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f93200186c0 0x7f932001abc0 crc :-1 s=READY pgs=806 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:44.596 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:44.596 7f9340eb5700 1 -- 172.21.2.102:0/2150627018 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f933c0ce450 con 0x7f933c197460 2021-02-10T11:27:44.599 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:44.599 7f932b7fe700 1 -- 172.21.2.102:0/2150627018 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f932c030820 con 0x7f933c197460 2021-02-10T11:27:44.724 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:44.724 7f9340eb5700 1 -- 172.21.2.102:0/2150627018 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f933c0ce450 con 0x7f933c197460 2021-02-10T11:27:44.725 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:44.725 7f932b7fe700 1 -- 172.21.2.102:0/2150627018 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) v1 ==== 74+0+13996 (secure 0 0 0) 0x7f932c030820 con 0x7f933c197460 2021-02-10T11:27:44.725 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:27:44.726 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":85,"fsid":"68db30ea-dbce-4aef-a6df-f01c6fea0513","created":"2021-02-10 11:03:06.331557","modified":"2021-02-10 11:27:42.481451","last_up_change":"2021-02-10 11:03:32.690562","last_in_change":"2021-02-10 11:03:32.690562","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:03:37.322519","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:27:39.452483","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":17,"pool_name":"cephfs_data","create_time":"2021-02-10 11:27:40.475662","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"be691b4d-a039-408a-af80-9d53f02b6f1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6818","nonce":36088},{"type":"v1","addr":"172.21.2.102:6820","nonce":36088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6822","nonce":36088},{"type":"v1","addr":"172.21.2.102:6824","nonce":36088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6830","nonce":36088},{"type":"v1","addr":"172.21.2.102:6832","nonce":36088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6826","nonce":36088},{"type":"v1","addr":"172.21.2.102:6828","nonce":36088}]},"public_addr":"172.21.2.102:6820/36088","cluster_addr":"172.21.2.102:6824/36088","heartbeat_back_addr":"172.21.2.102:6832/36088","heartbeat_front_addr":"172.21.2.102:6828/36088","state":["exists","up"]},{"osd":1,"uuid":"64e3ccc8-9c69-4d7c-865e-af8e83822635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6819","nonce":36087},{"type":"v1","addr":"172.21.2.102:6821","nonce":36087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6823","nonce":36087},{"type":"v1","addr":"172.21.2.102:6825","nonce":36087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6831","nonce":36087},{"type":"v1","addr":"172.21.2.102:6833","nonce":36087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6827","nonce":36087},{"type":"v1","addr":"172.21.2.102:6829","nonce":36087}]},"public_addr":"172.21.2.102:6821/36087","cluster_addr":"172.21.2.102:6825/36087","heartbeat_back_addr":"172.21.2.102:6833/36087","heartbeat_front_addr":"172.21.2.102:6829/36087","state":["exists","up"]},{"osd":2,"uuid":"153700af-747f-481f-bdf4-d6072d3a842f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6810","nonce":36089},{"type":"v1","addr":"172.21.2.102:6811","nonce":36089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6812","nonce":36089},{"type":"v1","addr":"172.21.2.102:6813","nonce":36089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6816","nonce":36089},{"type":"v1","addr":"172.21.2.102:6817","nonce":36089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6814","nonce":36089},{"type":"v1","addr":"172.21.2.102:6815","nonce":36089}]},"public_addr":"172.21.2.102:6811/36089","cluster_addr":"172.21.2.102:6813/36089","heartbeat_back_addr":"172.21.2.102:6817/36089","heartbeat_front_addr":"172.21.2.102:6815/36089","state":["exists","up"]},{"osd":3,"uuid":"cbc678f4-83a9-45e6-8830-2f7c232b1ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6802","nonce":36090},{"type":"v1","addr":"172.21.2.102:6803","nonce":36090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6804","nonce":36090},{"type":"v1","addr":"172.21.2.102:6805","nonce":36090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6808","nonce":36090},{"type":"v1","addr":"172.21.2.102:6809","nonce":36090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6806","nonce":36090},{"type":"v1","addr":"172.21.2.102:6807","nonce":36090}]},"public_addr":"172.21.2.102:6803/36090","cluster_addr":"172.21.2.102:6805/36090","heartbeat_back_addr":"172.21.2.102:6809/36090","heartbeat_front_addr":"172.21.2.102:6807/36090","state":["exists","up"]},{"osd":4,"uuid":"c01e4008-b258-4f10-afd2-9e7f37214351","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6808","nonce":36368},{"type":"v1","addr":"172.21.2.113:6811","nonce":36368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6814","nonce":36368},{"type":"v1","addr":"172.21.2.113:6818","nonce":36368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6826","nonce":36368},{"type":"v1","addr":"172.21.2.113:6828","nonce":36368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6820","nonce":36368},{"type":"v1","addr":"172.21.2.113:6824","nonce":36368}]},"public_addr":"172.21.2.113:6811/36368","cluster_addr":"172.21.2.113:6818/36368","heartbeat_back_addr":"172.21.2.113:6828/36368","heartbeat_front_addr":"172.21.2.113:6824/36368","state":["exists","up"]},{"osd":5,"uuid":"551f6007-daf8-4a98-8ee0-c2b16bcccb5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6802","nonce":36366},{"type":"v1","addr":"172.21.2.113:6805","nonce":36366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6807","nonce":36366},{"type":"v1","addr":"172.21.2.113:6810","nonce":36366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6819","nonce":36366},{"type":"v1","addr":"172.21.2.113:6822","nonce":36366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6812","nonce":36366},{"type":"v1","addr":"172.21.2.113:6815","nonce":36366}]},"public_addr":"172.21.2.113:6805/36366","cluster_addr":"172.21.2.113:6810/36366","heartbeat_back_addr":"172.21.2.113:6822/36366","heartbeat_front_addr":"172.21.2.113:6815/36366","state":["exists","up"]},{"osd":6,"uuid":"bf67c040-0e50-48be-b102-c3943bc9ebef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6800","nonce":36367},{"type":"v1","addr":"172.21.2.113:6801","nonce":36367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6803","nonce":36367},{"type":"v1","addr":"172.21.2.113:6804","nonce":36367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6813","nonce":36367},{"type":"v1","addr":"172.21.2.113:6816","nonce":36367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6806","nonce":36367},{"type":"v1","addr":"172.21.2.113:6809","nonce":36367}]},"public_addr":"172.21.2.113:6801/36367","cluster_addr":"172.21.2.113:6804/36367","heartbeat_back_addr":"172.21.2.113:6816/36367","heartbeat_front_addr":"172.21.2.113:6809/36367","state":["exists","up"]},{"osd":7,"uuid":"f5461f80-018d-428f-be89-2eb2bcbe7bb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6817","nonce":36365},{"type":"v1","addr":"172.21.2.113:6821","nonce":36365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6823","nonce":36365},{"type":"v1","addr":"172.21.2.113:6825","nonce":36365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6830","nonce":36365},{"type":"v1","addr":"172.21.2.113:6831","nonce":36365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6827","nonce":36365},{"type":"v1","addr":"172.21.2.113:6829","nonce":36365}]},"public_addr":"172.21.2.113:6821/36365","cluster_addr":"172.21.2.113:6825/36365","heartbeat_back_addr":"172.21.2.113:6831/36365","heartbeat_front_addr":"172.21.2.113:6829/36365","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:27:44.726 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:44.727 7f9340eb5700 1 -- 172.21.2.102:0/2150627018 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f93200186c0 msgr2=0x7f932001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:44.727 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:44.727 7f9340eb5700 1 --2- 172.21.2.102:0/2150627018 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f93200186c0 0x7f932001abc0 crc :-1 s=READY pgs=806 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:44.727 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:44.727 7f9340eb5700 1 -- 172.21.2.102:0/2150627018 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f933c197460 msgr2=0x7f933c18a540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:44.727 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:44.727 7f9340eb5700 1 --2- 172.21.2.102:0/2150627018 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f933c197460 0x7f933c18a540 secure :-1 s=READY pgs=1783 cs=0 l=1 rev1=1 rx=0x7f932c00b650 tx=0x7f932c00b9f0).stop 2021-02-10T11:27:44.727 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:44.727 7f9340eb5700 1 -- 172.21.2.102:0/2150627018 shutdown_connections 2021-02-10T11:27:44.728 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:44.727 7f9340eb5700 1 --2- 172.21.2.102:0/2150627018 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f93200186c0 0x7f932001abc0 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:44.728 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:44.727 7f9340eb5700 1 --2- 172.21.2.102:0/2150627018 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f933c0b7df0 0x7f933c0b81f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:44.728 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:44.727 7f9340eb5700 1 --2- 172.21.2.102:0/2150627018 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f933c18ab60 0x7f933c0b77d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:44.728 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:44.727 7f9340eb5700 1 --2- 172.21.2.102:0/2150627018 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f933c197460 0x7f933c18a540 unknown :-1 s=CLOSED pgs=1783 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:44.728 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:44.727 7f9340eb5700 1 -- 172.21.2.102:0/2150627018 >> 172.21.2.102:0/2150627018 conn(0x7f933c0efc70 msgr2=0x7f933c0ef410 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:44.728 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:44.727 7f9340eb5700 1 -- 172.21.2.102:0/2150627018 shutdown_connections 2021-02-10T11:27:44.728 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:44.727 7f9340eb5700 1 -- 172.21.2.102:0/2150627018 wait complete. 2021-02-10T11:27:44.736 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:27:45.175 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:45.158 7f115ff92700 1 Processor -- start 2021-02-10T11:27:45.175 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:45.158 7f115ff92700 1 -- start start 2021-02-10T11:27:45.175 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:45.159 7f115ff92700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f11580d1c80 0x7f11580cef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:45.175 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:45.159 7f115ff92700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f11580cf590 0x7f11580cf990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:45.176 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:45.159 7f115ff92700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11580fe1d0 con 0x7f11580d1880 2021-02-10T11:27:45.176 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:45.159 7f115ff92700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11581b0cc0 con 0x7f11580d1c80 2021-02-10T11:27:45.176 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:45.159 7f115ff92700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1158109a70 con 0x7f11580cf590 2021-02-10T11:27:45.176 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:45.159 7f115d5ad700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f11580d1c80 0x7f11580cef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:45.176 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:45.159 7f115d5ad700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f11580d1c80 0x7f11580cef70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.113:3300/0 says I am v2:172.21.2.102:35006/0 (socket says 172.21.2.102:35006) 2021-02-10T11:27:45.176 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:45.159 7f115d5ad700 1 -- 172.21.2.102:0/1533097796 learned_addr learned my addr 172.21.2.102:0/1533097796 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:27:45.176 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:45.159 7f115e5af700 1 --2- 172.21.2.102:0/1533097796 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f11580cf590 0x7f11580cf990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:45.176 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:45.160 7f115d5ad700 1 -- 172.21.2.102:0/1533097796 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f11580cf590 msgr2=0x7f11580cf990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:27:45.176 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:45.160 7f115d5ad700 1 --2- 172.21.2.102:0/1533097796 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f11580cf590 0x7f11580cf990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:45.176 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:45.160 7f115d5ad700 1 -- 172.21.2.102:0/1533097796 >> v1:172.21.2.113:6789/0 conn(0x7f11580d1880 legacy=0x7f11581b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:45.176 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:45.160 7f115cdac700 1 -- 172.21.2.102:0/1533097796 <== mon.1 v1:172.21.2.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2981260895 0 0) 0x7f114c001080 con 0x7f11580d1880 2021-02-10T11:27:45.177 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:45.160 7f115d5ad700 1 -- 172.21.2.102:0/1533097796 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1158088070 con 0x7f11580d1c80 2021-02-10T11:27:45.177 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:45.160 7f115d5ad700 1 --2- 172.21.2.102:0/1533097796 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f11580d1c80 0x7f11580cef70 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 rx=0x7f1154006700 tx=0x7f1154009b00).ready entity=mon.1 client_cookie=c52f68304209929 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:45.177 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:45.160 7f115cdac700 1 -- 172.21.2.102:0/1533097796 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f115400c0c0 con 0x7f11580d1c80 2021-02-10T11:27:45.177 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:45.160 7f115cdac700 1 -- 172.21.2.102:0/1533097796 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f115400c400 con 0x7f11580d1c80 2021-02-10T11:27:45.177 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:45.160 7f115cdac700 1 -- 172.21.2.102:0/1533097796 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f115400c920 con 0x7f11580d1c80 2021-02-10T11:27:45.177 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:45.161 7f115ff92700 1 -- 172.21.2.102:0/1533097796 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f11580d1c80 msgr2=0x7f11580cef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:45.177 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:45.161 7f115ff92700 1 --2- 172.21.2.102:0/1533097796 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f11580d1c80 0x7f11580cef70 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 rx=0x7f1154006700 tx=0x7f1154009b00).stop 2021-02-10T11:27:45.177 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:45.161 7f115ff92700 1 -- 172.21.2.102:0/1533097796 shutdown_connections 2021-02-10T11:27:45.177 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:45.161 7f115ff92700 1 --2- 172.21.2.102:0/1533097796 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f11580cf590 0x7f11580cf990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:45.177 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:45.161 7f115ff92700 1 --2- 172.21.2.102:0/1533097796 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f11580d1c80 0x7f11580cef70 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:45.177 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:45.161 7f115ff92700 1 -- 172.21.2.102:0/1533097796 >> 172.21.2.102:0/1533097796 conn(0x7f11580ce430 msgr2=0x7f11580d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:45.178 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:45.161 7f115ff92700 1 -- 172.21.2.102:0/1533097796 shutdown_connections 2021-02-10T11:27:45.178 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:45.161 7f115ff92700 1 -- 172.21.2.102:0/1533097796 wait complete. 2021-02-10T11:27:45.179 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:45.175 7f115ff92700 1 Processor -- start 2021-02-10T11:27:45.179 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:45.176 7f115ff92700 1 -- start start 2021-02-10T11:27:45.179 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:45.176 7f115ff92700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f115819a070 0x7f115808c780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:45.180 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:45.176 7f115ff92700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f115819a470 0x7f115808cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:45.180 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:45.176 7f115ff92700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1158095520 0x7f1158095920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:45.180 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:45.176 7f115ff92700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1158096900 con 0x7f1158095520 2021-02-10T11:27:45.180 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:45.176 7f115ff92700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f115808d4c0 con 0x7f115819a070 2021-02-10T11:27:45.180 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:45.176 7f115ff92700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f115819e9e0 con 0x7f115819a470 2021-02-10T11:27:45.180 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:45.176 7f115e5af700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1158095520 0x7f1158095920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:45.180 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:45.176 7f115e5af700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1158095520 0x7f1158095920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:59818/0 (socket says 172.21.2.102:59818) 2021-02-10T11:27:45.181 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:45.176 7f115e5af700 1 -- 172.21.2.102:0/1174721026 learned_addr learned my addr 172.21.2.102:0/1174721026 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:27:45.181 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:45.176 7f115e5af700 1 -- 172.21.2.102:0/1174721026 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f115819a470 msgr2=0x7f115808cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:45.181 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:45.176 7f115e5af700 1 --2- 172.21.2.102:0/1174721026 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f115819a470 0x7f115808cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:45.181 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:45.176 7f115e5af700 1 -- 172.21.2.102:0/1174721026 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f115819a070 msgr2=0x7f115808c780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:45.181 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:45.176 7f115e5af700 1 --2- 172.21.2.102:0/1174721026 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f115819a070 0x7f115808c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:45.181 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:45.176 7f115e5af700 1 -- 172.21.2.102:0/1174721026 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1158095f40 con 0x7f1158095520 2021-02-10T11:27:45.181 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:45.177 7f115e5af700 1 --2- 172.21.2.102:0/1174721026 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1158095520 0x7f1158095920 secure :-1 s=READY pgs=1784 cs=0 l=1 rev1=1 rx=0x7f115000b970 tx=0x7f115000bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:45.181 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:45.177 7f114a7fc700 1 -- 172.21.2.102:0/1174721026 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1150000f90 con 0x7f1158095520 2021-02-10T11:27:45.181 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:45.177 7f114a7fc700 1 -- 172.21.2.102:0/1174721026 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1150003030 con 0x7f1158095520 2021-02-10T11:27:45.181 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:45.177 7f115ff92700 1 -- 172.21.2.102:0/1174721026 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1158095f40 con 0x7f1158095520 2021-02-10T11:27:45.181 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:45.177 7f114a7fc700 1 -- 172.21.2.102:0/1174721026 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1150001400 con 0x7f1158095520 2021-02-10T11:27:45.182 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:45.177 7f115ff92700 1 -- 172.21.2.102:0/1174721026 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1158095f40 con 0x7f1158095520 2021-02-10T11:27:45.182 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:45.178 7f114a7fc700 1 -- 172.21.2.102:0/1174721026 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1150000eb0 con 0x7f1158095520 2021-02-10T11:27:45.182 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:45.178 7f114a7fc700 1 --2- 172.21.2.102:0/1174721026 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1140018730 0x7f114001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:45.182 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:45.178 7f114a7fc700 1 -- 172.21.2.102:0/1174721026 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f1150032b30 con 0x7f1158095520 2021-02-10T11:27:45.182 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:45.178 7f115ddae700 1 --2- 172.21.2.102:0/1174721026 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1140018730 0x7f114001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:45.182 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:45.178 7f115ddae700 1 --2- 172.21.2.102:0/1174721026 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1140018730 0x7f114001ac10 crc :-1 s=READY pgs=807 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:45.209 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:45.209 7f115ff92700 1 -- 172.21.2.102:0/1174721026 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1158082980 con 0x7f1158095520 2021-02-10T11:27:45.211 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:45.211 7f114a7fc700 1 -- 172.21.2.102:0/1174721026 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f1150006af0 con 0x7f1158095520 2021-02-10T11:27:45.344 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:45.344 7f115ff92700 1 -- 172.21.2.102:0/1174721026 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f1158082980 con 0x7f1158095520 2021-02-10T11:27:45.347 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:45.347 7f114a7fc700 1 -- 172.21.2.102:0/1174721026 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.0 v47) v1 ==== 184+0+0 (secure 0 0 0) 0x7f1150000eb0 con 0x7f1158095520 2021-02-10T11:27:45.351 INFO:teuthology.orchestra.run.gibba002.stderr:updated caps for client.0 2021-02-10T11:27:45.352 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:45.353 7f115ff92700 1 -- 172.21.2.102:0/1174721026 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1140018730 msgr2=0x7f114001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:45.353 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:45.353 7f115ff92700 1 --2- 172.21.2.102:0/1174721026 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1140018730 0x7f114001ac10 crc :-1 s=READY pgs=807 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:45.353 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:45.353 7f115ff92700 1 -- 172.21.2.102:0/1174721026 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1158095520 msgr2=0x7f1158095920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:45.353 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:45.353 7f115ff92700 1 --2- 172.21.2.102:0/1174721026 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1158095520 0x7f1158095920 secure :-1 s=READY pgs=1784 cs=0 l=1 rev1=1 rx=0x7f115000b970 tx=0x7f115000bd10).stop 2021-02-10T11:27:45.353 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:45.353 7f115ff92700 1 -- 172.21.2.102:0/1174721026 shutdown_connections 2021-02-10T11:27:45.353 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:45.353 7f115ff92700 1 --2- 172.21.2.102:0/1174721026 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1158095520 0x7f1158095920 unknown :-1 s=CLOSED pgs=1784 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:45.353 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:45.353 7f115ff92700 1 --2- 172.21.2.102:0/1174721026 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f115819a470 0x7f115808cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:45.354 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:45.353 7f115ff92700 1 --2- 172.21.2.102:0/1174721026 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1140018730 0x7f114001ac10 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:45.354 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:45.353 7f115ff92700 1 --2- 172.21.2.102:0/1174721026 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f115819a070 0x7f115808c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:45.354 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:45.353 7f115ff92700 1 -- 172.21.2.102:0/1174721026 >> 172.21.2.102:0/1174721026 conn(0x7f11581a6190 msgr2=0x7f11580949b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:45.354 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:45.353 7f115ff92700 1 -- 172.21.2.102:0/1174721026 shutdown_connections 2021-02-10T11:27:45.354 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:45.353 7f115ff92700 1 -- 172.21.2.102:0/1174721026 wait complete. 2021-02-10T11:27:45.361 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.1 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:27:45.801 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:45.785 7f1a794b6700 1 Processor -- start 2021-02-10T11:27:45.801 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:45.785 7f1a794b6700 1 -- start start 2021-02-10T11:27:45.801 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:45.785 7f1a794b6700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f1a740d1880 0x7f1a741b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:45.801 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:45.785 7f1a794b6700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1a741b0af0 0x7f1a74108950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:45.801 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:45.785 7f1a794b6700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f1a740d1c80 0x7f1a74108f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:45.801 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:45.785 7f1a794b6700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a740fe1d0 con 0x7f1a740d1c80 2021-02-10T11:27:45.801 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:45.785 7f1a794b6700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a74109a70 con 0x7f1a740d1880 2021-02-10T11:27:45.802 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:45.785 7f1a794b6700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a74088070 con 0x7f1a741b0af0 2021-02-10T11:27:45.802 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:45.786 7f1a73ac0700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f1a740d1c80 0x7f1a74108f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:45.802 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:45.786 7f1a73ac0700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f1a740d1c80 0x7f1a74108f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:59822/0 (socket says 172.21.2.102:59822) 2021-02-10T11:27:45.802 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:45.786 7f1a73ac0700 1 -- 172.21.2.102:0/1113462835 learned_addr learned my addr 172.21.2.102:0/1113462835 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:27:45.802 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:45.786 7f1a73ac0700 1 -- 172.21.2.102:0/1113462835 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1a741b0af0 msgr2=0x7f1a74108950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:27:45.802 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:45.786 7f1a73ac0700 1 --2- 172.21.2.102:0/1113462835 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1a741b0af0 0x7f1a74108950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:45.802 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:45.786 7f1a73ac0700 1 -- 172.21.2.102:0/1113462835 >> v2:172.21.2.113:3300/0 conn(0x7f1a740d1880 msgr2=0x7f1a741b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:27:45.802 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:45.786 7f1a73ac0700 1 --2- 172.21.2.102:0/1113462835 >> v2:172.21.2.113:3300/0 conn(0x7f1a740d1880 0x7f1a741b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:45.802 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:45.786 7f1a73ac0700 1 -- 172.21.2.102:0/1113462835 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a74109590 con 0x7f1a740d1c80 2021-02-10T11:27:45.802 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:45.786 7f1a73ac0700 1 --2- 172.21.2.102:0/1113462835 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1a740d1c80 0x7f1a74108f70 secure :-1 s=READY pgs=1785 cs=0 l=1 rev1=1 rx=0x7f1a6c006f00 tx=0x7f1a6c009f10).ready entity=mon.0 client_cookie=de27796b6a8e8ff0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:45.802 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:45.786 7f1a722bd700 1 -- 172.21.2.102:0/1113462835 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1a6c00b3b0 con 0x7f1a740d1c80 2021-02-10T11:27:45.803 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:45.786 7f1a722bd700 1 -- 172.21.2.102:0/1113462835 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a6c00b630 con 0x7f1a740d1c80 2021-02-10T11:27:45.803 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:45.786 7f1a722bd700 1 -- 172.21.2.102:0/1113462835 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1a6c00bb50 con 0x7f1a740d1c80 2021-02-10T11:27:45.803 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:45.786 7f1a794b6700 1 -- 172.21.2.102:0/1113462835 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1a740d1c80 msgr2=0x7f1a74108f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:45.803 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:45.786 7f1a794b6700 1 --2- 172.21.2.102:0/1113462835 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1a740d1c80 0x7f1a74108f70 secure :-1 s=READY pgs=1785 cs=0 l=1 rev1=1 rx=0x7f1a6c006f00 tx=0x7f1a6c009f10).stop 2021-02-10T11:27:45.803 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:45.787 7f1a794b6700 1 -- 172.21.2.102:0/1113462835 shutdown_connections 2021-02-10T11:27:45.803 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:45.787 7f1a794b6700 1 --2- 172.21.2.102:0/1113462835 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1a741b0af0 0x7f1a74108950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:45.803 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:45.787 7f1a794b6700 1 --2- 172.21.2.102:0/1113462835 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1a740d1c80 0x7f1a74108f70 unknown :-1 s=CLOSED pgs=1785 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:45.803 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:45.787 7f1a794b6700 1 --2- 172.21.2.102:0/1113462835 >> v2:172.21.2.113:3300/0 conn(0x7f1a740d1880 0x7f1a741b04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:45.803 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:45.787 7f1a794b6700 1 -- 172.21.2.102:0/1113462835 >> 172.21.2.102:0/1113462835 conn(0x7f1a740ce430 msgr2=0x7f1a740d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:45.803 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:45.787 7f1a794b6700 1 -- 172.21.2.102:0/1113462835 shutdown_connections 2021-02-10T11:27:45.803 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:45.787 7f1a794b6700 1 -- 172.21.2.102:0/1113462835 wait complete. 2021-02-10T11:27:45.805 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:45.801 7f1a794b6700 1 Processor -- start 2021-02-10T11:27:45.805 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:45.801 7f1a794b6700 1 -- start start 2021-02-10T11:27:45.805 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:45.802 7f1a794b6700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1a741b9280 0x7f1a7418f750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:45.805 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:45.802 7f1a794b6700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1a741b9680 0x7f1a7418fd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:45.805 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:45.802 7f1a794b6700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1a741acd60 0x7f1a741ad160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:45.805 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:45.802 7f1a794b6700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a74190390 con 0x7f1a741acd60 2021-02-10T11:27:45.805 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:45.802 7f1a794b6700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a741907c0 con 0x7f1a741b9280 2021-02-10T11:27:45.806 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:45.802 7f1a794b6700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a741ac3d0 con 0x7f1a741b9680 2021-02-10T11:27:45.806 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:45.802 7f1a73ac0700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1a741acd60 0x7f1a741ad160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:45.806 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:45.802 7f1a73ac0700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1a741acd60 0x7f1a741ad160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:59826/0 (socket says 172.21.2.102:59826) 2021-02-10T11:27:45.806 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:45.802 7f1a73ac0700 1 -- 172.21.2.102:0/2940980859 learned_addr learned my addr 172.21.2.102:0/2940980859 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:27:45.806 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:45.802 7f1a73ac0700 1 -- 172.21.2.102:0/2940980859 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1a741b9680 msgr2=0x7f1a7418fd70 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:27:45.806 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:45.802 7f1a73ac0700 1 --2- 172.21.2.102:0/2940980859 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1a741b9680 0x7f1a7418fd70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:45.806 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:45.802 7f1a73ac0700 1 -- 172.21.2.102:0/2940980859 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1a741b9280 msgr2=0x7f1a7418f750 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:27:45.806 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:45.802 7f1a73ac0700 1 --2- 172.21.2.102:0/2940980859 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1a741b9280 0x7f1a7418f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:45.806 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:45.803 7f1a73ac0700 1 -- 172.21.2.102:0/2940980859 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a741ad780 con 0x7f1a741acd60 2021-02-10T11:27:45.806 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:45.803 7f1a73ac0700 1 --2- 172.21.2.102:0/2940980859 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1a741acd60 0x7f1a741ad160 secure :-1 s=READY pgs=1786 cs=0 l=1 rev1=1 rx=0x7f1a6c00b8e0 tx=0x7f1a6c00bc80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:45.806 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:45.803 7f1a6bfff700 1 -- 172.21.2.102:0/2940980859 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1a6c009450 con 0x7f1a741acd60 2021-02-10T11:27:45.806 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:45.803 7f1a6bfff700 1 -- 172.21.2.102:0/2940980859 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a6c005820 con 0x7f1a741acd60 2021-02-10T11:27:45.806 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:45.803 7f1a6bfff700 1 -- 172.21.2.102:0/2940980859 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1a6c005c70 con 0x7f1a741acd60 2021-02-10T11:27:45.807 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:45.803 7f1a794b6700 1 -- 172.21.2.102:0/2940980859 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a741ad780 con 0x7f1a741acd60 2021-02-10T11:27:45.807 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:45.803 7f1a794b6700 1 -- 172.21.2.102:0/2940980859 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a741ad780 con 0x7f1a741acd60 2021-02-10T11:27:45.807 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:45.804 7f1a6bfff700 1 -- 172.21.2.102:0/2940980859 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1a6c005a00 con 0x7f1a741acd60 2021-02-10T11:27:45.807 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:45.804 7f1a6bfff700 1 --2- 172.21.2.102:0/2940980859 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1a580186c0 0x7f1a5801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:45.807 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:45.804 7f1a6bfff700 1 -- 172.21.2.102:0/2940980859 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f1a6c032770 con 0x7f1a741acd60 2021-02-10T11:27:45.807 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:45.804 7f1a732bf700 1 --2- 172.21.2.102:0/2940980859 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1a580186c0 0x7f1a5801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:45.807 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:45.804 7f1a732bf700 1 --2- 172.21.2.102:0/2940980859 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1a580186c0 0x7f1a5801abc0 crc :-1 s=READY pgs=808 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:45.835 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:45.835 7f1a794b6700 1 -- 172.21.2.102:0/2940980859 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a74190390 con 0x7f1a741acd60 2021-02-10T11:27:45.837 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:45.837 7f1a6bfff700 1 -- 172.21.2.102:0/2940980859 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f1a6c000f90 con 0x7f1a741acd60 2021-02-10T11:27:45.968 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:45.968 7f1a794b6700 1 -- 172.21.2.102:0/2940980859 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f1a74190390 con 0x7f1a741acd60 2021-02-10T11:27:45.972 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:45.972 7f1a6bfff700 1 -- 172.21.2.102:0/2940980859 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.1 v48) v1 ==== 184+0+0 (secure 0 0 0) 0x7f1a6c01f850 con 0x7f1a741acd60 2021-02-10T11:27:45.975 INFO:teuthology.orchestra.run.gibba002.stderr:updated caps for client.1 2021-02-10T11:27:45.977 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:45.977 7f1a794b6700 1 -- 172.21.2.102:0/2940980859 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1a580186c0 msgr2=0x7f1a5801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:45.977 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:45.977 7f1a794b6700 1 --2- 172.21.2.102:0/2940980859 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1a580186c0 0x7f1a5801abc0 crc :-1 s=READY pgs=808 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:45.977 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:45.977 7f1a794b6700 1 -- 172.21.2.102:0/2940980859 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1a741acd60 msgr2=0x7f1a741ad160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:45.978 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:45.977 7f1a794b6700 1 --2- 172.21.2.102:0/2940980859 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1a741acd60 0x7f1a741ad160 secure :-1 s=READY pgs=1786 cs=0 l=1 rev1=1 rx=0x7f1a6c00b8e0 tx=0x7f1a6c00bc80).stop 2021-02-10T11:27:45.978 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:45.977 7f1a794b6700 1 -- 172.21.2.102:0/2940980859 shutdown_connections 2021-02-10T11:27:45.978 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:45.977 7f1a794b6700 1 --2- 172.21.2.102:0/2940980859 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1a741acd60 0x7f1a741ad160 unknown :-1 s=CLOSED pgs=1786 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:45.978 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:45.977 7f1a794b6700 1 --2- 172.21.2.102:0/2940980859 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1a741b9680 0x7f1a7418fd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:45.978 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:45.977 7f1a794b6700 1 --2- 172.21.2.102:0/2940980859 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1a580186c0 0x7f1a5801abc0 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:45.978 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:45.977 7f1a794b6700 1 --2- 172.21.2.102:0/2940980859 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1a741b9280 0x7f1a7418f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:45.978 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:45.977 7f1a794b6700 1 -- 172.21.2.102:0/2940980859 >> 172.21.2.102:0/2940980859 conn(0x7f1a740845b0 msgr2=0x7f1a7418c290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:45.978 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:45.978 7f1a794b6700 1 -- 172.21.2.102:0/2940980859 shutdown_connections 2021-02-10T11:27:45.979 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:45.978 7f1a794b6700 1 -- 172.21.2.102:0/2940980859 wait complete. 2021-02-10T11:27:45.986 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.2 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:27:46.423 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:46.407 7f783587d700 1 Processor -- start 2021-02-10T11:27:46.424 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:46.408 7f783587d700 1 -- start start 2021-02-10T11:27:46.424 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:46.408 7f783587d700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f78300d1860 0x7f78301b04b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:46.424 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:46.408 7f783587d700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f78300cff60 0x7f7830109120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:46.424 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:46.408 7f783587d700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7830109a50 con 0x7f78301b0ad0 2021-02-10T11:27:46.424 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:46.408 7f783587d700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7830109740 con 0x7f78300cff60 2021-02-10T11:27:46.424 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:46.408 7f783587d700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7830088070 con 0x7f78300d1860 2021-02-10T11:27:46.424 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:46.408 7f782fd75700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f78300cff60 0x7f7830109120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:46.424 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:46.408 7f782fd75700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f78300cff60 0x7f7830109120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:59834/0 (socket says 172.21.2.102:59834) 2021-02-10T11:27:46.424 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:46.408 7f782fd75700 1 -- 172.21.2.102:0/4255763383 learned_addr learned my addr 172.21.2.102:0/4255763383 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:27:46.424 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:46.409 7f782fd75700 1 -- 172.21.2.102:0/4255763383 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f78300d1860 msgr2=0x7f78301b04b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:27:46.424 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:46.409 7f782fd75700 1 --2- 172.21.2.102:0/4255763383 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f78300d1860 0x7f78301b04b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:46.424 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:46.409 7f782fd75700 1 -- 172.21.2.102:0/4255763383 >> v1:172.21.2.113:6789/0 conn(0x7f78301b0ad0 legacy=0x7f7830108930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:27:46.425 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:46.409 7f782fd75700 1 -- 172.21.2.102:0/4255763383 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7830088260 con 0x7f78300cff60 2021-02-10T11:27:46.425 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:46.409 7f782fd75700 1 --2- 172.21.2.102:0/4255763383 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f78300cff60 0x7f7830109120 secure :-1 s=READY pgs=1787 cs=0 l=1 rev1=1 rx=0x7f7828006f00 tx=0x7f7828009f10).ready entity=mon.0 client_cookie=5eb09c01bee3864e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:46.425 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:46.409 7f782e572700 1 -- 172.21.2.102:0/4255763383 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f782800b3b0 con 0x7f78300cff60 2021-02-10T11:27:46.425 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:46.409 7f782e572700 1 -- 172.21.2.102:0/4255763383 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f782800b630 con 0x7f78300cff60 2021-02-10T11:27:46.425 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:46.409 7f782e572700 1 -- 172.21.2.102:0/4255763383 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f782800bb50 con 0x7f78300cff60 2021-02-10T11:27:46.425 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:46.409 7f783587d700 1 -- 172.21.2.102:0/4255763383 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f78300cff60 msgr2=0x7f7830109120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:46.425 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:46.409 7f783587d700 1 --2- 172.21.2.102:0/4255763383 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f78300cff60 0x7f7830109120 secure :-1 s=READY pgs=1787 cs=0 l=1 rev1=1 rx=0x7f7828006f00 tx=0x7f7828009f10).stop 2021-02-10T11:27:46.425 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:46.409 7f783587d700 1 -- 172.21.2.102:0/4255763383 shutdown_connections 2021-02-10T11:27:46.425 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:46.409 7f783587d700 1 --2- 172.21.2.102:0/4255763383 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f78300cff60 0x7f7830109120 unknown :-1 s=CLOSED pgs=1787 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:46.425 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:46.409 7f783587d700 1 --2- 172.21.2.102:0/4255763383 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f78300d1860 0x7f78301b04b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:46.425 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:46.409 7f783587d700 1 -- 172.21.2.102:0/4255763383 >> 172.21.2.102:0/4255763383 conn(0x7f78300ce410 msgr2=0x7f78300d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:46.425 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:46.409 7f783587d700 1 -- 172.21.2.102:0/4255763383 shutdown_connections 2021-02-10T11:27:46.425 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:46.410 7f783587d700 1 -- 172.21.2.102:0/4255763383 wait complete. 2021-02-10T11:27:46.427 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:46.424 7f783587d700 1 Processor -- start 2021-02-10T11:27:46.427 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:46.424 7f783587d700 1 -- start start 2021-02-10T11:27:46.427 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:46.424 7f783587d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f783018f2c0 0x7f783018f6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:46.427 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:46.424 7f783587d700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f783018fce0 0x7f78301900e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:46.427 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:46.424 7f783587d700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f783019ce80 0x7f783019d280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:46.427 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:46.424 7f783587d700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f783018ed40 con 0x7f783018f2c0 2021-02-10T11:27:46.427 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:46.424 7f783587d700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78300b1b00 con 0x7f783019ce80 2021-02-10T11:27:46.428 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:46.424 7f783587d700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f783019d8a0 con 0x7f783018fce0 2021-02-10T11:27:46.428 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:46.424 7f782f574700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f783018f2c0 0x7f783018f6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:46.428 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:46.425 7f782f574700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f783018f2c0 0x7f783018f6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:59840/0 (socket says 172.21.2.102:59840) 2021-02-10T11:27:46.428 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:46.425 7f782f574700 1 -- 172.21.2.102:0/279765002 learned_addr learned my addr 172.21.2.102:0/279765002 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:27:46.428 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:46.425 7f782f574700 1 -- 172.21.2.102:0/279765002 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f783018fce0 msgr2=0x7f78301900e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:46.428 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:46.425 7f782ed73700 1 --2- 172.21.2.102:0/279765002 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f783018fce0 0x7f78301900e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:46.428 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:46.425 7f782fd75700 1 --2- 172.21.2.102:0/279765002 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f783019ce80 0x7f783019d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:46.428 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:46.425 7f782f574700 1 --2- 172.21.2.102:0/279765002 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f783018fce0 0x7f78301900e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:46.428 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:46.425 7f782f574700 1 -- 172.21.2.102:0/279765002 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f783019ce80 msgr2=0x7f783019d280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:46.428 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:46.425 7f782f574700 1 --2- 172.21.2.102:0/279765002 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f783019ce80 0x7f783019d280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:46.428 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:46.425 7f782f574700 1 -- 172.21.2.102:0/279765002 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f783019dac0 con 0x7f783018f2c0 2021-02-10T11:27:46.428 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:46.425 7f782fd75700 1 --2- 172.21.2.102:0/279765002 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f783019ce80 0x7f783019d280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:27:46.428 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:46.425 7f782f574700 1 --2- 172.21.2.102:0/279765002 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f783018f2c0 0x7f783018f6c0 secure :-1 s=READY pgs=1788 cs=0 l=1 rev1=1 rx=0x7f781800b530 tx=0x7f781800b8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:46.429 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:46.425 7f7827fff700 1 -- 172.21.2.102:0/279765002 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7818006350 con 0x7f783018f2c0 2021-02-10T11:27:46.429 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:46.425 7f7827fff700 1 -- 172.21.2.102:0/279765002 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7818006c20 con 0x7f783018f2c0 2021-02-10T11:27:46.429 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:46.425 7f783587d700 1 -- 172.21.2.102:0/279765002 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f783019dac0 con 0x7f783018f2c0 2021-02-10T11:27:46.429 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:46.425 7f7827fff700 1 -- 172.21.2.102:0/279765002 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7818000c90 con 0x7f783018f2c0 2021-02-10T11:27:46.429 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:46.425 7f783587d700 1 -- 172.21.2.102:0/279765002 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f783019e050 con 0x7f783018f2c0 2021-02-10T11:27:46.429 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:46.426 7f7827fff700 1 -- 172.21.2.102:0/279765002 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f7818005310 con 0x7f783018f2c0 2021-02-10T11:27:46.429 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:46.426 7f7827fff700 1 --2- 172.21.2.102:0/279765002 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f78140186c0 0x7f781401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:46.429 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:46.426 7f7827fff700 1 -- 172.21.2.102:0/279765002 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f7818032600 con 0x7f783018f2c0 2021-02-10T11:27:46.429 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:46.426 7f782ed73700 1 --2- 172.21.2.102:0/279765002 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f78140186c0 0x7f781401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:46.429 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:46.427 7f782ed73700 1 --2- 172.21.2.102:0/279765002 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f78140186c0 0x7f781401abc0 crc :-1 s=READY pgs=809 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:46.458 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:46.458 7f783587d700 1 -- 172.21.2.102:0/279765002 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78300b1b00 con 0x7f783018f2c0 2021-02-10T11:27:46.460 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:46.460 7f7827fff700 1 -- 172.21.2.102:0/279765002 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f7818006350 con 0x7f783018f2c0 2021-02-10T11:27:46.591 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:46.591 7f783587d700 1 -- 172.21.2.102:0/279765002 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f78300b1b00 con 0x7f783018f2c0 2021-02-10T11:27:46.594 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:46.594 7f7827fff700 1 -- 172.21.2.102:0/279765002 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.2 v49) v1 ==== 184+0+0 (secure 0 0 0) 0x7f7818005310 con 0x7f783018f2c0 2021-02-10T11:27:46.598 INFO:teuthology.orchestra.run.gibba002.stderr:updated caps for client.2 2021-02-10T11:27:46.601 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:46.600 7f783587d700 1 -- 172.21.2.102:0/279765002 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f78140186c0 msgr2=0x7f781401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:46.601 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:46.600 7f783587d700 1 --2- 172.21.2.102:0/279765002 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f78140186c0 0x7f781401abc0 crc :-1 s=READY pgs=809 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:46.601 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:46.600 7f783587d700 1 -- 172.21.2.102:0/279765002 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f783018f2c0 msgr2=0x7f783018f6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:46.601 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:46.600 7f783587d700 1 --2- 172.21.2.102:0/279765002 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f783018f2c0 0x7f783018f6c0 secure :-1 s=READY pgs=1788 cs=0 l=1 rev1=1 rx=0x7f781800b530 tx=0x7f781800b8d0).stop 2021-02-10T11:27:46.601 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:46.600 7f783587d700 1 -- 172.21.2.102:0/279765002 shutdown_connections 2021-02-10T11:27:46.601 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:46.600 7f783587d700 1 --2- 172.21.2.102:0/279765002 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f78140186c0 0x7f781401abc0 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:46.602 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:46.600 7f783587d700 1 --2- 172.21.2.102:0/279765002 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f783019ce80 0x7f783019d280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:46.602 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:46.600 7f783587d700 1 --2- 172.21.2.102:0/279765002 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f783018fce0 0x7f78301900e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:46.602 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:46.600 7f783587d700 1 --2- 172.21.2.102:0/279765002 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f783018f2c0 0x7f783018f6c0 unknown :-1 s=CLOSED pgs=1788 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:46.602 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:46.600 7f783587d700 1 -- 172.21.2.102:0/279765002 >> 172.21.2.102:0/279765002 conn(0x7f783019f820 msgr2=0x7f7830085d20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:46.602 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:46.600 7f783587d700 1 -- 172.21.2.102:0/279765002 shutdown_connections 2021-02-10T11:27:46.603 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:46.601 7f783587d700 1 -- 172.21.2.102:0/279765002 wait complete. 2021-02-10T11:27:46.609 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.3 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:27:47.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:47.034 7fc43226a700 1 Processor -- start 2021-02-10T11:27:47.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:47.034 7fc43226a700 1 -- start start 2021-02-10T11:27:47.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:47.034 7fc43226a700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fc42c0d1880 0x7fc42c1b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:47.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:47.034 7fc43226a700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc42c1b0af0 0x7fc42c108950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:47.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:47.034 7fc43226a700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fc42c0d1c80 0x7fc42c108f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:47.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:47.034 7fc43226a700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc42c0fe1d0 con 0x7fc42c0d1c80 2021-02-10T11:27:47.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:47.034 7fc43226a700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc42c109a70 con 0x7fc42c0d1880 2021-02-10T11:27:47.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:47.034 7fc43226a700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc42c088070 con 0x7fc42c1b0af0 2021-02-10T11:27:47.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:47.034 7fc430887700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fc42c0d1c80 0x7fc42c108f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:47.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:47.034 7fc430887700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fc42c0d1c80 0x7fc42c108f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:59850/0 (socket says 172.21.2.102:59850) 2021-02-10T11:27:47.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:47.034 7fc430887700 1 -- 172.21.2.102:0/3309177033 learned_addr learned my addr 172.21.2.102:0/3309177033 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:27:47.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:47.035 7fc430887700 1 -- 172.21.2.102:0/3309177033 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc42c1b0af0 msgr2=0x7fc42c108950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:27:47.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:47.035 7fc430887700 1 --2- 172.21.2.102:0/3309177033 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc42c1b0af0 0x7fc42c108950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:47.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:47.035 7fc430887700 1 -- 172.21.2.102:0/3309177033 >> v2:172.21.2.113:3300/0 conn(0x7fc42c0d1880 msgr2=0x7fc42c1b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:27:47.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:47.035 7fc430887700 1 --2- 172.21.2.102:0/3309177033 >> v2:172.21.2.113:3300/0 conn(0x7fc42c0d1880 0x7fc42c1b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:47.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:47.035 7fc430887700 1 -- 172.21.2.102:0/3309177033 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc42c109590 con 0x7fc42c0d1c80 2021-02-10T11:27:47.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:47.035 7fc430887700 1 --2- 172.21.2.102:0/3309177033 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc42c0d1c80 0x7fc42c108f70 secure :-1 s=READY pgs=1789 cs=0 l=1 rev1=1 rx=0x7fc424006f00 tx=0x7fc424009f10).ready entity=mon.0 client_cookie=a90d603036191878 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:47.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:47.035 7fc42b7fe700 1 -- 172.21.2.102:0/3309177033 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc42400b3b0 con 0x7fc42c0d1c80 2021-02-10T11:27:47.052 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:47.035 7fc42b7fe700 1 -- 172.21.2.102:0/3309177033 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc42400b730 con 0x7fc42c0d1c80 2021-02-10T11:27:47.052 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:47.035 7fc42b7fe700 1 -- 172.21.2.102:0/3309177033 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc42400bc50 con 0x7fc42c0d1c80 2021-02-10T11:27:47.052 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:47.035 7fc43226a700 1 -- 172.21.2.102:0/3309177033 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc42c0d1c80 msgr2=0x7fc42c108f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:47.052 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:47.035 7fc43226a700 1 --2- 172.21.2.102:0/3309177033 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc42c0d1c80 0x7fc42c108f70 secure :-1 s=READY pgs=1789 cs=0 l=1 rev1=1 rx=0x7fc424006f00 tx=0x7fc424009f10).stop 2021-02-10T11:27:47.052 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:47.036 7fc43226a700 1 -- 172.21.2.102:0/3309177033 shutdown_connections 2021-02-10T11:27:47.052 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:47.036 7fc43226a700 1 --2- 172.21.2.102:0/3309177033 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc42c1b0af0 0x7fc42c108950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:47.052 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:47.036 7fc43226a700 1 --2- 172.21.2.102:0/3309177033 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc42c0d1c80 0x7fc42c108f70 unknown :-1 s=CLOSED pgs=1789 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:47.052 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:47.036 7fc43226a700 1 --2- 172.21.2.102:0/3309177033 >> v2:172.21.2.113:3300/0 conn(0x7fc42c0d1880 0x7fc42c1b04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:47.052 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:47.036 7fc43226a700 1 -- 172.21.2.102:0/3309177033 >> 172.21.2.102:0/3309177033 conn(0x7fc42c0ce430 msgr2=0x7fc42c0d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:47.052 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:47.036 7fc43226a700 1 -- 172.21.2.102:0/3309177033 shutdown_connections 2021-02-10T11:27:47.052 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:47.036 7fc43226a700 1 -- 172.21.2.102:0/3309177033 wait complete. 2021-02-10T11:27:47.054 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:47.050 7fc43226a700 1 Processor -- start 2021-02-10T11:27:47.054 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:47.050 7fc43226a700 1 -- start start 2021-02-10T11:27:47.054 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:47.051 7fc43226a700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc42c1b9470 0x7fc42c1ae6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:47.054 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:47.051 7fc43226a700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc42c1aecf0 0x7fc42c0957f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:47.054 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:47.051 7fc43226a700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc42c1af0f0 0x7fc42c095e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:47.054 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:47.051 7fc43226a700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc42c096d70 con 0x7fc42c1af0f0 2021-02-10T11:27:47.054 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:47.051 7fc43226a700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc42c0a7780 con 0x7fc42c1aecf0 2021-02-10T11:27:47.054 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:47.051 7fc43226a700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc42c1b9870 con 0x7fc42c1b9470 2021-02-10T11:27:47.054 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:47.051 7fc430887700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc42c1af0f0 0x7fc42c095e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:47.054 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:47.051 7fc430887700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc42c1af0f0 0x7fc42c095e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:59854/0 (socket says 172.21.2.102:59854) 2021-02-10T11:27:47.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:47.051 7fc430887700 1 -- 172.21.2.102:0/4147074065 learned_addr learned my addr 172.21.2.102:0/4147074065 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:27:47.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:47.051 7fc430887700 1 -- 172.21.2.102:0/4147074065 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc42c1b9470 msgr2=0x7fc42c1ae6d0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:27:47.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:47.051 7fc430887700 1 --2- 172.21.2.102:0/4147074065 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc42c1b9470 0x7fc42c1ae6d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:47.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:47.051 7fc430887700 1 -- 172.21.2.102:0/4147074065 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc42c1aecf0 msgr2=0x7fc42c0957f0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:27:47.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:47.051 7fc430887700 1 --2- 172.21.2.102:0/4147074065 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc42c1aecf0 0x7fc42c0957f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:47.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:47.051 7fc430887700 1 -- 172.21.2.102:0/4147074065 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc42c19e6b0 con 0x7fc42c1af0f0 2021-02-10T11:27:47.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:47.051 7fc430887700 1 --2- 172.21.2.102:0/4147074065 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc42c1af0f0 0x7fc42c095e10 secure :-1 s=READY pgs=1790 cs=0 l=1 rev1=1 rx=0x7fc42400b9b0 tx=0x7fc42400bd50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:47.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:47.051 7fc4297fa700 1 -- 172.21.2.102:0/4147074065 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc424005820 con 0x7fc42c1af0f0 2021-02-10T11:27:47.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:47.052 7fc4297fa700 1 -- 172.21.2.102:0/4147074065 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc424005aa0 con 0x7fc42c1af0f0 2021-02-10T11:27:47.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:47.052 7fc4297fa700 1 -- 172.21.2.102:0/4147074065 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc424006030 con 0x7fc42c1af0f0 2021-02-10T11:27:47.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:47.052 7fc43226a700 1 -- 172.21.2.102:0/4147074065 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc42c19e6b0 con 0x7fc42c1af0f0 2021-02-10T11:27:47.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:47.052 7fc43226a700 1 -- 172.21.2.102:0/4147074065 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc42c19e6b0 con 0x7fc42c1af0f0 2021-02-10T11:27:47.056 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:47.052 7fc4297fa700 1 -- 172.21.2.102:0/4147074065 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fc424006030 con 0x7fc42c1af0f0 2021-02-10T11:27:47.056 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:47.052 7fc4297fa700 1 --2- 172.21.2.102:0/4147074065 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc4100186c0 0x7fc41001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:47.056 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:47.052 7fc4297fa700 1 -- 172.21.2.102:0/4147074065 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fc4240328c0 con 0x7fc42c1af0f0 2021-02-10T11:27:47.056 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:47.052 7fc42bfff700 1 --2- 172.21.2.102:0/4147074065 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc4100186c0 0x7fc41001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:47.056 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:47.053 7fc42bfff700 1 --2- 172.21.2.102:0/4147074065 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc4100186c0 0x7fc41001abc0 crc :-1 s=READY pgs=810 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:47.084 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:47.084 7fc43226a700 1 -- 172.21.2.102:0/4147074065 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc42c0829a0 con 0x7fc42c1af0f0 2021-02-10T11:27:47.086 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:47.086 7fc4297fa700 1 -- 172.21.2.102:0/4147074065 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fc424000f90 con 0x7fc42c1af0f0 2021-02-10T11:27:47.217 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:47.217 7fc43226a700 1 -- 172.21.2.102:0/4147074065 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.3", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7fc42c0a7780 con 0x7fc42c1af0f0 2021-02-10T11:27:47.221 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:47.221 7fc4297fa700 1 -- 172.21.2.102:0/4147074065 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.3", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.3 v50) v1 ==== 184+0+0 (secure 0 0 0) 0x7fc42401f820 con 0x7fc42c1af0f0 2021-02-10T11:27:47.224 INFO:teuthology.orchestra.run.gibba002.stderr:updated caps for client.3 2021-02-10T11:27:47.226 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:47.226 7fc43226a700 1 -- 172.21.2.102:0/4147074065 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc4100186c0 msgr2=0x7fc41001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:47.226 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:47.226 7fc43226a700 1 --2- 172.21.2.102:0/4147074065 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc4100186c0 0x7fc41001abc0 crc :-1 s=READY pgs=810 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:47.226 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:47.226 7fc43226a700 1 -- 172.21.2.102:0/4147074065 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc42c1af0f0 msgr2=0x7fc42c095e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:47.226 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:47.226 7fc43226a700 1 --2- 172.21.2.102:0/4147074065 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc42c1af0f0 0x7fc42c095e10 secure :-1 s=READY pgs=1790 cs=0 l=1 rev1=1 rx=0x7fc42400b9b0 tx=0x7fc42400bd50).stop 2021-02-10T11:27:47.227 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:47.226 7fc43226a700 1 -- 172.21.2.102:0/4147074065 shutdown_connections 2021-02-10T11:27:47.227 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:47.226 7fc43226a700 1 --2- 172.21.2.102:0/4147074065 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc42c1af0f0 0x7fc42c095e10 unknown :-1 s=CLOSED pgs=1790 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:47.227 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:47.226 7fc43226a700 1 --2- 172.21.2.102:0/4147074065 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc4100186c0 0x7fc41001abc0 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:47.227 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:47.226 7fc43226a700 1 --2- 172.21.2.102:0/4147074065 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc42c1aecf0 0x7fc42c0957f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:47.227 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:47.226 7fc43226a700 1 --2- 172.21.2.102:0/4147074065 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc42c1b9470 0x7fc42c1ae6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:47.227 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:47.226 7fc43226a700 1 -- 172.21.2.102:0/4147074065 >> 172.21.2.102:0/4147074065 conn(0x7fc42c0845b0 msgr2=0x7fc42c18c290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:47.227 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:47.226 7fc43226a700 1 -- 172.21.2.102:0/4147074065 shutdown_connections 2021-02-10T11:27:47.228 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:47.226 7fc43226a700 1 -- 172.21.2.102:0/4147074065 wait complete. 2021-02-10T11:27:47.236 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:27:47.663 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:47.645 7fd63c696700 1 Processor -- start 2021-02-10T11:27:47.663 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:47.645 7fd63c696700 1 -- start start 2021-02-10T11:27:47.663 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:47.646 7fd63c696700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fd6340d1c70 0x7fd6340f20a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:47.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:47.646 7fd63c696700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6340f0b10 con 0x7fd6340d1870 2021-02-10T11:27:47.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:47.646 7fd63c696700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6341b0be0 con 0x7fd6340d0340 2021-02-10T11:27:47.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:47.646 7fd63c696700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6340e5ed0 con 0x7fd6340d1c70 2021-02-10T11:27:47.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:47.646 7fd63a4b2700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fd6340d1870 0x7fd6341b03f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:52084/0 (socket says 172.21.2.102:52084) 2021-02-10T11:27:47.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:47.646 7fd63a4b2700 1 -- 172.21.2.102:0/2056769929 learned_addr learned my addr 172.21.2.102:0/2056769929 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:27:47.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:47.646 7fd6394b0700 1 -- 172.21.2.102:0/2056769929 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3404418491 0 0) 0x7fd628000ce0 con 0x7fd6340d1870 2021-02-10T11:27:47.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:47.646 7fd6394b0700 1 -- 172.21.2.102:0/2056769929 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd620002fe0 con 0x7fd6340d1870 2021-02-10T11:27:47.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:47.647 7fd63acb3700 1 --2- 172.21.2.102:0/2056769929 >> v2:172.21.2.113:3300/0 conn(0x7fd6340d1c70 0x7fd6340f20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:47.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:47.647 7fd6394b0700 1 -- 172.21.2.102:0/2056769929 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3792279076 0 0) 0x7fd628001180 con 0x7fd6340d1870 2021-02-10T11:27:47.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:47.647 7fd6394b0700 1 -- 172.21.2.102:0/2056769929 >> v2:172.21.2.113:3300/0 conn(0x7fd6340d1c70 msgr2=0x7fd6340f20a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:27:47.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:47.647 7fd6394b0700 1 --2- 172.21.2.102:0/2056769929 >> v2:172.21.2.113:3300/0 conn(0x7fd6340d1c70 0x7fd6340f20a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:47.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:47.647 7fd6394b0700 1 -- 172.21.2.102:0/2056769929 >> v1:172.21.2.113:6789/0 conn(0x7fd6340d0340 legacy=0x7fd6340ef640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:47.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:47.647 7fd6394b0700 1 -- 172.21.2.102:0/2056769929 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6340efe30 con 0x7fd6340d1870 2021-02-10T11:27:47.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:47.647 7fd6394b0700 1 -- 172.21.2.102:0/2056769929 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7fd628001680 con 0x7fd6340d1870 2021-02-10T11:27:47.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:47.647 7fd6394b0700 1 -- 172.21.2.102:0/2056769929 >> v1:172.21.2.102:6789/0 conn(0x7fd6340d1870 legacy=0x7fd6341b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:47.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:47.647 7fd6394b0700 1 --2- 172.21.2.102:0/2056769929 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd620003b90 0x7fd620005fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:47.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:47.647 7fd6394b0700 1 --2- 172.21.2.102:0/2056769929 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd6200067b0 0x7fd620008c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:47.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:47.647 7fd6394b0700 1 --2- 172.21.2.102:0/2056769929 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd620009590 0x7fd62000bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:47.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:47.647 7fd6394b0700 1 -- 172.21.2.102:0/2056769929 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd62000c300 con 0x7fd620003b90 2021-02-10T11:27:47.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:47.647 7fd6394b0700 1 -- 172.21.2.102:0/2056769929 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd62000c520 con 0x7fd620009590 2021-02-10T11:27:47.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:47.647 7fd6394b0700 1 -- 172.21.2.102:0/2056769929 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd62000c740 con 0x7fd6200067b0 2021-02-10T11:27:47.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:47.647 7fd63a4b2700 1 --2- 172.21.2.102:0/2056769929 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd620003b90 0x7fd620005fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:47.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:47.647 7fd639cb1700 1 --2- 172.21.2.102:0/2056769929 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd6200067b0 0x7fd620008c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:47.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:47.647 7fd63acb3700 1 --2- 172.21.2.102:0/2056769929 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd620009590 0x7fd62000bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:47.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:47.647 7fd63a4b2700 1 -- 172.21.2.102:0/2056769929 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd6200067b0 msgr2=0x7fd620008c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:27:47.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:47.647 7fd63a4b2700 1 --2- 172.21.2.102:0/2056769929 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd6200067b0 0x7fd620008c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:47.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:47.647 7fd63a4b2700 1 -- 172.21.2.102:0/2056769929 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd620009590 msgr2=0x7fd62000bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:27:47.666 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:47.647 7fd63a4b2700 1 --2- 172.21.2.102:0/2056769929 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd620009590 0x7fd62000bab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:47.666 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:47.647 7fd63a4b2700 1 -- 172.21.2.102:0/2056769929 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6200509b0 con 0x7fd620003b90 2021-02-10T11:27:47.666 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:47.647 7fd63acb3700 1 --2- 172.21.2.102:0/2056769929 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd620009590 0x7fd62000bab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:27:47.666 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:47.647 7fd63acb3700 1 -- 172.21.2.102:0/2056769929 reap_dead start 2021-02-10T11:27:47.666 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:47.648 7fd63a4b2700 1 --2- 172.21.2.102:0/2056769929 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd620003b90 0x7fd620005fc0 secure :-1 s=READY pgs=1792 cs=0 l=1 rev1=1 rx=0x7fd628000b60 tx=0x7fd628007da0).ready entity=mon.0 client_cookie=f262e97fae17881e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:47.666 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:47.648 7fd6394b0700 1 -- 172.21.2.102:0/2056769929 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd628007810 con 0x7fd620003b90 2021-02-10T11:27:47.666 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:47.648 7fd6394b0700 1 -- 172.21.2.102:0/2056769929 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6280070c0 con 0x7fd620003b90 2021-02-10T11:27:47.666 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:47.648 7fd6394b0700 1 -- 172.21.2.102:0/2056769929 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd628004620 con 0x7fd620003b90 2021-02-10T11:27:47.666 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:47.648 7fd63c696700 1 -- 172.21.2.102:0/2056769929 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd620003b90 msgr2=0x7fd620005fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:47.666 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:47.648 7fd63c696700 1 --2- 172.21.2.102:0/2056769929 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd620003b90 0x7fd620005fc0 secure :-1 s=READY pgs=1792 cs=0 l=1 rev1=1 rx=0x7fd628000b60 tx=0x7fd628007da0).stop 2021-02-10T11:27:47.666 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:47.649 7fd63c696700 1 -- 172.21.2.102:0/2056769929 shutdown_connections 2021-02-10T11:27:47.666 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:47.649 7fd63c696700 1 --2- 172.21.2.102:0/2056769929 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd620003b90 0x7fd620005fc0 unknown :-1 s=CLOSED pgs=1792 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:47.666 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:47.649 7fd63c696700 1 -- 172.21.2.102:0/2056769929 >> 172.21.2.102:0/2056769929 conn(0x7fd6340ce450 msgr2=0x7fd6340d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:47.667 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:47.649 7fd63c696700 1 -- 172.21.2.102:0/2056769929 shutdown_connections 2021-02-10T11:27:47.667 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:47.649 7fd63c696700 1 -- 172.21.2.102:0/2056769929 wait complete. 2021-02-10T11:27:47.668 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:47.663 7fd63c696700 1 Processor -- start 2021-02-10T11:27:47.668 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:47.664 7fd63c696700 1 -- start start 2021-02-10T11:27:47.668 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:47.664 7fd63c696700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd6341942b0 0x7fd6340b3550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:47.668 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:47.664 7fd63c696700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd634195070 0x7fd634195470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:47.668 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:47.664 7fd63c696700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd6340bb8e0 0x7fd6340bbce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:47.668 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:47.664 7fd63c696700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6341946b0 con 0x7fd634195070 2021-02-10T11:27:47.668 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:47.664 7fd63c696700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6340b3b70 con 0x7fd6341942b0 2021-02-10T11:27:47.668 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:47.664 7fd63c696700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6340b3d90 con 0x7fd6340bb8e0 2021-02-10T11:27:47.669 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:47.664 7fd639cb1700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd634195070 0x7fd634195470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:47.669 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:47.664 7fd639cb1700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd634195070 0x7fd634195470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:59874/0 (socket says 172.21.2.102:59874) 2021-02-10T11:27:47.669 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:47.664 7fd639cb1700 1 -- 172.21.2.102:0/2801663235 learned_addr learned my addr 172.21.2.102:0/2801663235 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:27:47.669 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:47.664 7fd639cb1700 1 -- 172.21.2.102:0/2801663235 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd6340bb8e0 msgr2=0x7fd6340bbce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:47.669 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:47.664 7fd639cb1700 1 --2- 172.21.2.102:0/2801663235 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd6340bb8e0 0x7fd6340bbce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:47.669 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:47.664 7fd639cb1700 1 -- 172.21.2.102:0/2801663235 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd6341942b0 msgr2=0x7fd6340b3550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:47.669 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:47.665 7fd639cb1700 1 --2- 172.21.2.102:0/2801663235 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd6341942b0 0x7fd6340b3550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:47.669 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:47.665 7fd639cb1700 1 -- 172.21.2.102:0/2801663235 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6341a49e0 con 0x7fd634195070 2021-02-10T11:27:47.669 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:47.665 7fd639cb1700 1 --2- 172.21.2.102:0/2801663235 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd634195070 0x7fd634195470 secure :-1 s=READY pgs=1793 cs=0 l=1 rev1=1 rx=0x7fd63000b610 tx=0x7fd63000b9b0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:47.669 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:47.665 7fd626ffd700 1 -- 172.21.2.102:0/2801663235 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd630000960 con 0x7fd634195070 2021-02-10T11:27:47.669 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:47.665 7fd626ffd700 1 -- 172.21.2.102:0/2801663235 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd63000bda0 con 0x7fd634195070 2021-02-10T11:27:47.669 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:47.665 7fd626ffd700 1 -- 172.21.2.102:0/2801663235 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd630002030 con 0x7fd634195070 2021-02-10T11:27:47.669 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:47.665 7fd63c696700 1 -- 172.21.2.102:0/2801663235 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6341a49e0 con 0x7fd634195070 2021-02-10T11:27:47.670 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:47.665 7fd63c696700 1 -- 172.21.2.102:0/2801663235 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6340bc5f0 con 0x7fd634195070 2021-02-10T11:27:47.670 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:47.665 7fd626ffd700 1 -- 172.21.2.102:0/2801663235 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd630000ca0 con 0x7fd634195070 2021-02-10T11:27:47.670 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:47.666 7fd626ffd700 1 --2- 172.21.2.102:0/2801663235 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd61c0186c0 0x7fd61c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:47.670 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:47.666 7fd626ffd700 1 -- 172.21.2.102:0/2801663235 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fd630032430 con 0x7fd634195070 2021-02-10T11:27:47.670 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:47.666 7fd63a4b2700 1 --2- 172.21.2.102:0/2801663235 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd61c0186c0 0x7fd61c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:47.670 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:47.666 7fd63a4b2700 1 --2- 172.21.2.102:0/2801663235 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd61c0186c0 0x7fd61c01abc0 crc :-1 s=READY pgs=811 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:47.696 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:47.696 7fd63c696700 1 -- 172.21.2.102:0/2801663235 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6340bc5f0 con 0x7fd634195070 2021-02-10T11:27:47.698 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:47.698 7fd626ffd700 1 -- 172.21.2.102:0/2801663235 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd63002cc60 con 0x7fd634195070 2021-02-10T11:27:47.825 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:47.825 7fd63c696700 1 -- 172.21.2.102:0/2801663235 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd6340bc5f0 con 0x7fd634195070 2021-02-10T11:27:47.826 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:47.826 7fd626ffd700 1 -- 172.21.2.102:0/2801663235 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 65 v65) v1 ==== 94+0+2101 (secure 0 0 0) 0x7fd63002cf30 con 0x7fd634195070 2021-02-10T11:27:47.826 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:27:47.826 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":65,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6110,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.102:6835/1471871278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6834","nonce":1471871278},{"type":"v1","addr":"172.21.2.102:6835","nonce":1471871278}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":61}],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:27:41.479375","modified":"2021-02-10 11:27:43.490605","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6107},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6107":{"gid":6107,"name":"b","rank":0,"incarnation":62,"state":"up:active","state_seq":3,"addr":"172.21.2.113:6833/4220238062","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6832","nonce":4220238062},{"type":"v1","addr":"172.21.2.113:6833","nonce":4220238062}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":8}]} 2021-02-10T11:27:47.826 INFO:teuthology.orchestra.run.gibba002.stderr:dumped fsmap epoch 65 2021-02-10T11:27:47.828 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:47.827 7fd63c696700 1 -- 172.21.2.102:0/2801663235 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd61c0186c0 msgr2=0x7fd61c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:47.828 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:47.827 7fd63c696700 1 --2- 172.21.2.102:0/2801663235 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd61c0186c0 0x7fd61c01abc0 crc :-1 s=READY pgs=811 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:47.828 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:47.827 7fd63c696700 1 -- 172.21.2.102:0/2801663235 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd634195070 msgr2=0x7fd634195470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:47.828 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:47.827 7fd63c696700 1 --2- 172.21.2.102:0/2801663235 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd634195070 0x7fd634195470 secure :-1 s=READY pgs=1793 cs=0 l=1 rev1=1 rx=0x7fd63000b610 tx=0x7fd63000b9b0).stop 2021-02-10T11:27:47.828 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:47.828 7fd63c696700 1 -- 172.21.2.102:0/2801663235 shutdown_connections 2021-02-10T11:27:47.828 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:47.828 7fd63c696700 1 --2- 172.21.2.102:0/2801663235 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd6340bb8e0 0x7fd6340bbce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:47.828 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:47.828 7fd63c696700 1 --2- 172.21.2.102:0/2801663235 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd634195070 0x7fd634195470 unknown :-1 s=CLOSED pgs=1793 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:47.828 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:47.828 7fd63c696700 1 --2- 172.21.2.102:0/2801663235 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd61c0186c0 0x7fd61c01abc0 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:47.828 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:47.828 7fd63c696700 1 --2- 172.21.2.102:0/2801663235 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd6341942b0 0x7fd6340b3550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:47.828 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:47.828 7fd63c696700 1 -- 172.21.2.102:0/2801663235 >> 172.21.2.102:0/2801663235 conn(0x7fd6340b3040 msgr2=0x7fd6340c3250 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:47.829 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:47.828 7fd63c696700 1 -- 172.21.2.102:0/2801663235 shutdown_connections 2021-02-10T11:27:47.829 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:47.828 7fd63c696700 1 -- 172.21.2.102:0/2801663235 wait complete. 2021-02-10T11:27:48.037 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 65, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:27:41.479375', 'modified': '2021-02-10 11:27:43.490605', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6107}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6107': {'gid': 6107, 'name': 'b', 'rank': 0, 'incarnation': 62, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.113:6833/4220238062', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.113:6832', 'nonce': 4220238062}, {'type': 'v1', 'addr': '172.21.2.113:6833', 'nonce': 4220238062}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [17], 'metadata_pool': 16, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:27:48.038 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:27:48.038 DEBUG:teuthology.orchestra.run.gibba013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:27:48.437 INFO:teuthology.orchestra.run.gibba013.stdout:{ 2021-02-10T11:27:48.437 INFO:teuthology.orchestra.run.gibba013.stdout: "cluster_fsid": "68db30ea-dbce-4aef-a6df-f01c6fea0513", 2021-02-10T11:27:48.437 INFO:teuthology.orchestra.run.gibba013.stdout: "whoami": 0, 2021-02-10T11:27:48.437 INFO:teuthology.orchestra.run.gibba013.stdout: "id": 6107, 2021-02-10T11:27:48.438 INFO:teuthology.orchestra.run.gibba013.stdout: "want_state": "up:active", 2021-02-10T11:27:48.438 INFO:teuthology.orchestra.run.gibba013.stdout: "state": "up:active", 2021-02-10T11:27:48.438 INFO:teuthology.orchestra.run.gibba013.stdout: "rank_uptime": 6.9565991939999998, 2021-02-10T11:27:48.438 INFO:teuthology.orchestra.run.gibba013.stdout: "mdsmap_epoch": 65, 2021-02-10T11:27:48.438 INFO:teuthology.orchestra.run.gibba013.stdout: "osdmap_epoch": 85, 2021-02-10T11:27:48.438 INFO:teuthology.orchestra.run.gibba013.stdout: "osdmap_epoch_barrier": 85, 2021-02-10T11:27:48.438 INFO:teuthology.orchestra.run.gibba013.stdout: "uptime": 12.605416797 2021-02-10T11:27:48.438 INFO:teuthology.orchestra.run.gibba013.stdout:} 2021-02-10T11:27:48.441 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "68db30ea-dbce-4aef-a6df-f01c6fea0513", "whoami": 0, "id": 6107, "want_state": "up:active", "state": "up:active", "rank_uptime": 6.9565991939999998, "mdsmap_epoch": 65, "osdmap_epoch": 85, "osdmap_epoch_barrier": 85, "uptime": 12.605416797 } 2021-02-10T11:27:48.442 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:27:48.870 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:48.853 7f7acd893700 1 Processor -- start 2021-02-10T11:27:48.870 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:48.853 7f7acd893700 1 -- start start 2021-02-10T11:27:48.870 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:48.854 7f7acd893700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f7ac80d1c70 0x7f7ac80f20a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:48.870 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:48.854 7f7acd893700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ac80f0b10 con 0x7f7ac80d1870 2021-02-10T11:27:48.870 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:48.854 7f7acd893700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ac81b0be0 con 0x7f7ac80d0340 2021-02-10T11:27:48.870 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:48.854 7f7acd893700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ac80e5ed0 con 0x7f7ac80d1c70 2021-02-10T11:27:48.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:48.854 7f7ac7d75700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f7ac80d1c70 0x7f7ac80f20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:48.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:48.854 7f7ac7d75700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f7ac80d1c70 0x7f7ac80f20a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:59880/0 (socket says 172.21.2.102:59880) 2021-02-10T11:27:48.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:48.854 7f7ac7d75700 1 -- 172.21.2.102:0/3226588458 learned_addr learned my addr 172.21.2.102:0/3226588458 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:27:48.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:48.854 7f7ac7574700 1 --1- 172.21.2.102:0/3226588458 >> v1:172.21.2.102:6789/0 conn(0x7f7ac80d1870 0x7f7ac81b03f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:52104/0 (socket says 172.21.2.102:52104) 2021-02-10T11:27:48.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:48.854 7f7ac7d75700 1 -- 172.21.2.102:0/3226588458 >> v1:172.21.2.113:6789/0 conn(0x7f7ac80d0340 legacy=0x7f7ac80ef640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:27:48.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:48.854 7f7ac7d75700 1 -- 172.21.2.102:0/3226588458 >> v1:172.21.2.102:6789/0 conn(0x7f7ac80d1870 legacy=0x7f7ac81b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:48.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:48.854 7f7ac7d75700 1 -- 172.21.2.102:0/3226588458 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ac80efe30 con 0x7f7ac80d1c70 2021-02-10T11:27:48.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:48.854 7f7ac7d75700 1 --2- 172.21.2.102:0/3226588458 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7ac80d1c70 0x7f7ac80f20a0 secure :-1 s=READY pgs=1795 cs=0 l=1 rev1=1 rx=0x7f7ac0006f00 tx=0x7f7ac0009f10).ready entity=mon.0 client_cookie=89179f5cb762be67 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:48.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:48.855 7f7ac6572700 1 -- 172.21.2.102:0/3226588458 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7ac000b3b0 con 0x7f7ac80d1c70 2021-02-10T11:27:48.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:48.855 7f7ac6572700 1 -- 172.21.2.102:0/3226588458 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ac000b630 con 0x7f7ac80d1c70 2021-02-10T11:27:48.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:48.855 7f7ac6572700 1 -- 172.21.2.102:0/3226588458 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7ac000bb50 con 0x7f7ac80d1c70 2021-02-10T11:27:48.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:48.855 7f7acd893700 1 -- 172.21.2.102:0/3226588458 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7ac80d1c70 msgr2=0x7f7ac80f20a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:48.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:48.855 7f7acd893700 1 --2- 172.21.2.102:0/3226588458 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7ac80d1c70 0x7f7ac80f20a0 secure :-1 s=READY pgs=1795 cs=0 l=1 rev1=1 rx=0x7f7ac0006f00 tx=0x7f7ac0009f10).stop 2021-02-10T11:27:48.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:48.855 7f7acd893700 1 -- 172.21.2.102:0/3226588458 shutdown_connections 2021-02-10T11:27:48.872 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:48.855 7f7acd893700 1 --2- 172.21.2.102:0/3226588458 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7ac80d1c70 0x7f7ac80f20a0 unknown :-1 s=CLOSED pgs=1795 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:48.872 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:48.855 7f7acd893700 1 -- 172.21.2.102:0/3226588458 >> 172.21.2.102:0/3226588458 conn(0x7f7ac80ce450 msgr2=0x7f7ac80d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:48.872 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:48.855 7f7acd893700 1 -- 172.21.2.102:0/3226588458 shutdown_connections 2021-02-10T11:27:48.872 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:48.855 7f7acd893700 1 -- 172.21.2.102:0/3226588458 wait complete. 2021-02-10T11:27:48.873 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:48.870 7f7acd893700 1 Processor -- start 2021-02-10T11:27:48.873 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:48.870 7f7acd893700 1 -- start start 2021-02-10T11:27:48.873 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:48.871 7f7acd893700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7ac8092010 0x7f7ac80c33c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:48.873 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:48.871 7f7acd893700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7ac80c39e0 0x7f7ac80af040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:48.873 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:48.871 7f7acd893700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7ac80af660 0x7f7ac80afa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:48.874 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:48.871 7f7acd893700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ac80b0500 con 0x7f7ac8092010 2021-02-10T11:27:48.874 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:48.871 7f7acd893700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ac80b0080 con 0x7f7ac80af660 2021-02-10T11:27:48.874 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:48.871 7f7acd893700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ac80b02a0 con 0x7f7ac80c39e0 2021-02-10T11:27:48.874 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:48.871 7f7ac7574700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7ac8092010 0x7f7ac80c33c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:48.874 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:48.871 7f7ac7574700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7ac8092010 0x7f7ac80c33c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:59884/0 (socket says 172.21.2.102:59884) 2021-02-10T11:27:48.874 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:48.871 7f7ac7574700 1 -- 172.21.2.102:0/2284462198 learned_addr learned my addr 172.21.2.102:0/2284462198 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:27:48.874 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:48.871 7f7ac7574700 1 -- 172.21.2.102:0/2284462198 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7ac80c39e0 msgr2=0x7f7ac80af040 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:27:48.874 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:48.871 7f7ac7574700 1 --2- 172.21.2.102:0/2284462198 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7ac80c39e0 0x7f7ac80af040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:48.875 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:48.871 7f7ac7574700 1 -- 172.21.2.102:0/2284462198 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7ac80af660 msgr2=0x7f7ac80afa60 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:27:48.875 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:48.871 7f7ac7574700 1 --2- 172.21.2.102:0/2284462198 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7ac80af660 0x7f7ac80afa60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:48.875 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:48.871 7f7ac7574700 1 -- 172.21.2.102:0/2284462198 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ac80b0a70 con 0x7f7ac8092010 2021-02-10T11:27:48.875 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:48.872 7f7ac7574700 1 --2- 172.21.2.102:0/2284462198 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7ac8092010 0x7f7ac80c33c0 secure :-1 s=READY pgs=1796 cs=0 l=1 rev1=1 rx=0x7f7ab0001c10 tx=0x7f7ab000a8f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:48.875 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:48.872 7f7abffff700 1 -- 172.21.2.102:0/2284462198 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7ab00012c0 con 0x7f7ac8092010 2021-02-10T11:27:48.875 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:48.872 7f7abffff700 1 -- 172.21.2.102:0/2284462198 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ab000ace0 con 0x7f7ac8092010 2021-02-10T11:27:48.875 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:48.872 7f7abffff700 1 -- 172.21.2.102:0/2284462198 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7ab00045d0 con 0x7f7ac8092010 2021-02-10T11:27:48.875 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:48.872 7f7acd893700 1 -- 172.21.2.102:0/2284462198 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ac80b0a70 con 0x7f7ac8092010 2021-02-10T11:27:48.875 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:48.872 7f7acd893700 1 -- 172.21.2.102:0/2284462198 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ac80b0a70 con 0x7f7ac8092010 2021-02-10T11:27:48.875 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:48.873 7f7abffff700 1 -- 172.21.2.102:0/2284462198 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f7ab0004350 con 0x7f7ac8092010 2021-02-10T11:27:48.875 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:48.873 7f7abffff700 1 --2- 172.21.2.102:0/2284462198 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7aac0186c0 0x7f7aac01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:48.875 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:48.873 7f7abffff700 1 -- 172.21.2.102:0/2284462198 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f7ab0032580 con 0x7f7ac8092010 2021-02-10T11:27:48.875 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:48.873 7f7ac6d73700 1 --2- 172.21.2.102:0/2284462198 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7aac0186c0 0x7f7aac01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:48.876 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:48.873 7f7ac6d73700 1 --2- 172.21.2.102:0/2284462198 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7aac0186c0 0x7f7aac01abc0 crc :-1 s=READY pgs=812 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:48.904 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:48.904 7f7acd893700 1 -- 172.21.2.102:0/2284462198 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ac80e7df0 con 0x7f7ac8092010 2021-02-10T11:27:48.906 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:48.906 7f7abffff700 1 -- 172.21.2.102:0/2284462198 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f7ab0004140 con 0x7f7ac8092010 2021-02-10T11:27:49.032 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:49.032 7f7acd893700 1 -- 172.21.2.102:0/2284462198 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7ac80ce450 con 0x7f7ac8092010 2021-02-10T11:27:49.032 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:49.032 7f7abffff700 1 -- 172.21.2.102:0/2284462198 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) v1 ==== 74+0+13996 (secure 0 0 0) 0x7f7ab0004350 con 0x7f7ac8092010 2021-02-10T11:27:49.033 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:27:49.033 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":85,"fsid":"68db30ea-dbce-4aef-a6df-f01c6fea0513","created":"2021-02-10 11:03:06.331557","modified":"2021-02-10 11:27:42.481451","last_up_change":"2021-02-10 11:03:32.690562","last_in_change":"2021-02-10 11:03:32.690562","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:03:37.322519","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:27:39.452483","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":17,"pool_name":"cephfs_data","create_time":"2021-02-10 11:27:40.475662","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"be691b4d-a039-408a-af80-9d53f02b6f1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6818","nonce":36088},{"type":"v1","addr":"172.21.2.102:6820","nonce":36088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6822","nonce":36088},{"type":"v1","addr":"172.21.2.102:6824","nonce":36088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6830","nonce":36088},{"type":"v1","addr":"172.21.2.102:6832","nonce":36088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6826","nonce":36088},{"type":"v1","addr":"172.21.2.102:6828","nonce":36088}]},"public_addr":"172.21.2.102:6820/36088","cluster_addr":"172.21.2.102:6824/36088","heartbeat_back_addr":"172.21.2.102:6832/36088","heartbeat_front_addr":"172.21.2.102:6828/36088","state":["exists","up"]},{"osd":1,"uuid":"64e3ccc8-9c69-4d7c-865e-af8e83822635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6819","nonce":36087},{"type":"v1","addr":"172.21.2.102:6821","nonce":36087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6823","nonce":36087},{"type":"v1","addr":"172.21.2.102:6825","nonce":36087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6831","nonce":36087},{"type":"v1","addr":"172.21.2.102:6833","nonce":36087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6827","nonce":36087},{"type":"v1","addr":"172.21.2.102:6829","nonce":36087}]},"public_addr":"172.21.2.102:6821/36087","cluster_addr":"172.21.2.102:6825/36087","heartbeat_back_addr":"172.21.2.102:6833/36087","heartbeat_front_addr":"172.21.2.102:6829/36087","state":["exists","up"]},{"osd":2,"uuid":"153700af-747f-481f-bdf4-d6072d3a842f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6810","nonce":36089},{"type":"v1","addr":"172.21.2.102:6811","nonce":36089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6812","nonce":36089},{"type":"v1","addr":"172.21.2.102:6813","nonce":36089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6816","nonce":36089},{"type":"v1","addr":"172.21.2.102:6817","nonce":36089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6814","nonce":36089},{"type":"v1","addr":"172.21.2.102:6815","nonce":36089}]},"public_addr":"172.21.2.102:6811/36089","cluster_addr":"172.21.2.102:6813/36089","heartbeat_back_addr":"172.21.2.102:6817/36089","heartbeat_front_addr":"172.21.2.102:6815/36089","state":["exists","up"]},{"osd":3,"uuid":"cbc678f4-83a9-45e6-8830-2f7c232b1ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6802","nonce":36090},{"type":"v1","addr":"172.21.2.102:6803","nonce":36090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6804","nonce":36090},{"type":"v1","addr":"172.21.2.102:6805","nonce":36090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6808","nonce":36090},{"type":"v1","addr":"172.21.2.102:6809","nonce":36090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6806","nonce":36090},{"type":"v1","addr":"172.21.2.102:6807","nonce":36090}]},"public_addr":"172.21.2.102:6803/36090","cluster_addr":"172.21.2.102:6805/36090","heartbeat_back_addr":"172.21.2.102:6809/36090","heartbeat_front_addr":"172.21.2.102:6807/36090","state":["exists","up"]},{"osd":4,"uuid":"c01e4008-b258-4f10-afd2-9e7f37214351","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6808","nonce":36368},{"type":"v1","addr":"172.21.2.113:6811","nonce":36368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6814","nonce":36368},{"type":"v1","addr":"172.21.2.113:6818","nonce":36368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6826","nonce":36368},{"type":"v1","addr":"172.21.2.113:6828","nonce":36368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6820","nonce":36368},{"type":"v1","addr":"172.21.2.113:6824","nonce":36368}]},"public_addr":"172.21.2.113:6811/36368","cluster_addr":"172.21.2.113:6818/36368","heartbeat_back_addr":"172.21.2.113:6828/36368","heartbeat_front_addr":"172.21.2.113:6824/36368","state":["exists","up"]},{"osd":5,"uuid":"551f6007-daf8-4a98-8ee0-c2b16bcccb5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6802","nonce":36366},{"type":"v1","addr":"172.21.2.113:6805","nonce":36366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6807","nonce":36366},{"type":"v1","addr":"172.21.2.113:6810","nonce":36366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6819","nonce":36366},{"type":"v1","addr":"172.21.2.113:6822","nonce":36366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6812","nonce":36366},{"type":"v1","addr":"172.21.2.113:6815","nonce":36366}]},"public_addr":"172.21.2.113:6805/36366","cluster_addr":"172.21.2.113:6810/36366","heartbeat_back_addr":"172.21.2.113:6822/36366","heartbeat_front_addr":"172.21.2.113:6815/36366","state":["exists","up"]},{"osd":6,"uuid":"bf67c040-0e50-48be-b102-c3943bc9ebef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6800","nonce":36367},{"type":"v1","addr":"172.21.2.113:6801","nonce":36367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6803","nonce":36367},{"type":"v1","addr":"172.21.2.113:6804","nonce":36367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6813","nonce":36367},{"type":"v1","addr":"172.21.2.113:6816","nonce":36367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6806","nonce":36367},{"type":"v1","addr":"172.21.2.113:6809","nonce":36367}]},"public_addr":"172.21.2.113:6801/36367","cluster_addr":"172.21.2.113:6804/36367","heartbeat_back_addr":"172.21.2.113:6816/36367","heartbeat_front_addr":"172.21.2.113:6809/36367","state":["exists","up"]},{"osd":7,"uuid":"f5461f80-018d-428f-be89-2eb2bcbe7bb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6817","nonce":36365},{"type":"v1","addr":"172.21.2.113:6821","nonce":36365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6823","nonce":36365},{"type":"v1","addr":"172.21.2.113:6825","nonce":36365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6830","nonce":36365},{"type":"v1","addr":"172.21.2.113:6831","nonce":36365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6827","nonce":36365},{"type":"v1","addr":"172.21.2.113:6829","nonce":36365}]},"public_addr":"172.21.2.113:6821/36365","cluster_addr":"172.21.2.113:6825/36365","heartbeat_back_addr":"172.21.2.113:6831/36365","heartbeat_front_addr":"172.21.2.113:6829/36365","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:27:49.034 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:49.034 7f7acd893700 1 -- 172.21.2.102:0/2284462198 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7aac0186c0 msgr2=0x7f7aac01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:49.034 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:49.034 7f7acd893700 1 --2- 172.21.2.102:0/2284462198 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7aac0186c0 0x7f7aac01abc0 crc :-1 s=READY pgs=812 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:49.035 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:49.034 7f7acd893700 1 -- 172.21.2.102:0/2284462198 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7ac8092010 msgr2=0x7f7ac80c33c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:49.035 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:49.034 7f7acd893700 1 --2- 172.21.2.102:0/2284462198 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7ac8092010 0x7f7ac80c33c0 secure :-1 s=READY pgs=1796 cs=0 l=1 rev1=1 rx=0x7f7ab0001c10 tx=0x7f7ab000a8f0).stop 2021-02-10T11:27:49.035 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:49.034 7f7acd893700 1 -- 172.21.2.102:0/2284462198 shutdown_connections 2021-02-10T11:27:49.035 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:49.034 7f7acd893700 1 --2- 172.21.2.102:0/2284462198 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7aac0186c0 0x7f7aac01abc0 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:49.035 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:49.034 7f7acd893700 1 --2- 172.21.2.102:0/2284462198 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7ac80af660 0x7f7ac80afa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:49.035 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:49.034 7f7acd893700 1 --2- 172.21.2.102:0/2284462198 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7ac80c39e0 0x7f7ac80af040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:49.035 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:49.034 7f7acd893700 1 --2- 172.21.2.102:0/2284462198 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7ac8092010 0x7f7ac80c33c0 unknown :-1 s=CLOSED pgs=1796 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:49.035 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:49.034 7f7acd893700 1 -- 172.21.2.102:0/2284462198 >> 172.21.2.102:0/2284462198 conn(0x7f7ac80a0030 msgr2=0x7f7ac80f22b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:49.035 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:49.034 7f7acd893700 1 -- 172.21.2.102:0/2284462198 shutdown_connections 2021-02-10T11:27:49.035 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:49.035 7f7acd893700 1 -- 172.21.2.102:0/2284462198 wait complete. 2021-02-10T11:27:49.043 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:03:37.322519', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 16, 'pool_name': 'cephfs_metadata', 'create_time': '2021-02-10 11:27:39.452483', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 17, 'pool_name': 'cephfs_data', 'create_time': '2021-02-10 11:27:40.475662', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-02-10T11:27:49.043 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:27:49.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:49.458 7fe46f883700 1 Processor -- start 2021-02-10T11:27:49.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:49.458 7fe46f883700 1 -- start start 2021-02-10T11:27:49.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:49.459 7fe46f883700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe4680d1c70 0x7fe4680cef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:49.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:49.459 7fe46f883700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fe4680cf580 0x7fe4680cf980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:49.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:49.459 7fe46f883700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4680fe1c0 con 0x7fe4680d1870 2021-02-10T11:27:49.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:49.459 7fe46f883700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4681b0cb0 con 0x7fe4680cf580 2021-02-10T11:27:49.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:49.459 7fe46f883700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe468109a60 con 0x7fe4680d1c70 2021-02-10T11:27:49.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:49.459 7fe46d69f700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fe4680d1870 0x7fe4681b04c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:52116/0 (socket says 172.21.2.102:52116) 2021-02-10T11:27:49.477 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:49.459 7fe46d69f700 1 -- 172.21.2.102:0/1898358983 learned_addr learned my addr 172.21.2.102:0/1898358983 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:27:49.477 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:49.459 7fe46ce9e700 1 --2- 172.21.2.102:0/1898358983 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe4680d1c70 0x7fe4680cef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:49.477 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:49.459 7fe46dea0700 1 --2- 172.21.2.102:0/1898358983 >> v2:172.21.2.113:3300/0 conn(0x7fe4680cf580 0x7fe4680cf980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:49.477 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:49.460 7fe45bfff700 1 -- 172.21.2.102:0/1898358983 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 264239029 0 0) 0x7fe4640009b0 con 0x7fe4680d1870 2021-02-10T11:27:49.477 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:49.460 7fe45bfff700 1 -- 172.21.2.102:0/1898358983 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe454002fe0 con 0x7fe4680d1870 2021-02-10T11:27:49.477 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:49.460 7fe46dea0700 1 -- 172.21.2.102:0/1898358983 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe4680d1c70 msgr2=0x7fe4680cef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:27:49.477 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:49.460 7fe46dea0700 1 --2- 172.21.2.102:0/1898358983 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe4680d1c70 0x7fe4680cef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:49.477 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:49.460 7fe45bfff700 1 -- 172.21.2.102:0/1898358983 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 157351201 0 0) 0x7fe464000e20 con 0x7fe4680d1870 2021-02-10T11:27:49.477 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:49.460 7fe46dea0700 1 -- 172.21.2.102:0/1898358983 >> v1:172.21.2.102:6789/0 conn(0x7fe4680d1870 legacy=0x7fe4681b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:49.477 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:49.460 7fe46dea0700 1 -- 172.21.2.102:0/1898358983 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe468088070 con 0x7fe4680cf580 2021-02-10T11:27:49.477 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:49.460 7fe46dea0700 1 --2- 172.21.2.102:0/1898358983 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe4680cf580 0x7fe4680cf980 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 rx=0x7fe460006ad0 tx=0x7fe460009ad0).ready entity=mon.1 client_cookie=8331762a3bc5f841 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:49.477 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:49.460 7fe45bfff700 1 -- 172.21.2.102:0/1898358983 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe4600022e0 con 0x7fe4680cf580 2021-02-10T11:27:49.477 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:49.460 7fe45bfff700 1 -- 172.21.2.102:0/1898358983 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe460006410 con 0x7fe4680cf580 2021-02-10T11:27:49.478 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:49.460 7fe45bfff700 1 -- 172.21.2.102:0/1898358983 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe460002560 con 0x7fe4680cf580 2021-02-10T11:27:49.478 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:49.460 7fe46f883700 1 -- 172.21.2.102:0/1898358983 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe4680cf580 msgr2=0x7fe4680cf980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:49.478 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:49.460 7fe46f883700 1 --2- 172.21.2.102:0/1898358983 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe4680cf580 0x7fe4680cf980 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 rx=0x7fe460006ad0 tx=0x7fe460009ad0).stop 2021-02-10T11:27:49.478 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:49.461 7fe46f883700 1 -- 172.21.2.102:0/1898358983 shutdown_connections 2021-02-10T11:27:49.478 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:49.461 7fe46f883700 1 --2- 172.21.2.102:0/1898358983 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe4680cf580 0x7fe4680cf980 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:49.478 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:49.461 7fe46f883700 1 --2- 172.21.2.102:0/1898358983 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe4680d1c70 0x7fe4680cef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:49.478 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:49.461 7fe46f883700 1 -- 172.21.2.102:0/1898358983 >> 172.21.2.102:0/1898358983 conn(0x7fe4680ce420 msgr2=0x7fe4680d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:49.478 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:49.461 7fe46f883700 1 -- 172.21.2.102:0/1898358983 shutdown_connections 2021-02-10T11:27:49.478 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:49.461 7fe46f883700 1 -- 172.21.2.102:0/1898358983 wait complete. 2021-02-10T11:27:49.480 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:49.476 7fe46f883700 1 Processor -- start 2021-02-10T11:27:49.480 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:49.476 7fe46f883700 1 -- start start 2021-02-10T11:27:49.480 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:49.477 7fe46f883700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe4680b22c0 0x7fe4680b26c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:49.480 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:49.477 7fe46f883700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe4680a6df0 0x7fe4680a71f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:49.480 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:49.477 7fe46f883700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe4680a5cd0 0x7fe4680a60d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:49.480 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:49.477 7fe46f883700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4680c23f0 con 0x7fe4680a6df0 2021-02-10T11:27:49.480 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:49.477 7fe46f883700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4680a66f0 con 0x7fe4680a5cd0 2021-02-10T11:27:49.480 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:49.477 7fe46f883700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4680a6910 con 0x7fe4680b22c0 2021-02-10T11:27:49.480 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:49.477 7fe46ce9e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe4680a6df0 0x7fe4680a71f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:49.481 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:49.477 7fe46ce9e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe4680a6df0 0x7fe4680a71f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:59896/0 (socket says 172.21.2.102:59896) 2021-02-10T11:27:49.481 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:49.477 7fe46ce9e700 1 -- 172.21.2.102:0/3559391390 learned_addr learned my addr 172.21.2.102:0/3559391390 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:27:49.481 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:49.477 7fe46d69f700 1 --2- 172.21.2.102:0/3559391390 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe4680b22c0 0x7fe4680b26c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:49.481 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:49.477 7fe46dea0700 1 --2- 172.21.2.102:0/3559391390 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe4680a5cd0 0x7fe4680a60d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:49.481 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:49.477 7fe46ce9e700 1 -- 172.21.2.102:0/3559391390 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe4680b22c0 msgr2=0x7fe4680b26c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:49.481 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:49.477 7fe46ce9e700 1 --2- 172.21.2.102:0/3559391390 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe4680b22c0 0x7fe4680b26c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:49.481 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:49.477 7fe46ce9e700 1 -- 172.21.2.102:0/3559391390 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe4680a5cd0 msgr2=0x7fe4680a60d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:49.481 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:49.477 7fe46ce9e700 1 --2- 172.21.2.102:0/3559391390 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe4680a5cd0 0x7fe4680a60d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:49.481 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:49.478 7fe46ce9e700 1 -- 172.21.2.102:0/3559391390 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe468083980 con 0x7fe4680a6df0 2021-02-10T11:27:49.481 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:49.478 7fe46d69f700 1 --2- 172.21.2.102:0/3559391390 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe4680b22c0 0x7fe4680b26c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:27:49.481 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:49.478 7fe46ce9e700 1 --2- 172.21.2.102:0/3559391390 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe4680a6df0 0x7fe4680a71f0 secure :-1 s=READY pgs=1798 cs=0 l=1 rev1=1 rx=0x7fe45c0099e0 tx=0x7fe45c00bd40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:49.481 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:49.478 7fe459ffb700 1 -- 172.21.2.102:0/3559391390 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe45c00c2f0 con 0x7fe4680a6df0 2021-02-10T11:27:49.481 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:49.478 7fe459ffb700 1 -- 172.21.2.102:0/3559391390 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe45c00c630 con 0x7fe4680a6df0 2021-02-10T11:27:49.482 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:49.478 7fe459ffb700 1 -- 172.21.2.102:0/3559391390 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe45c00cb50 con 0x7fe4680a6df0 2021-02-10T11:27:49.482 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:49.478 7fe46f883700 1 -- 172.21.2.102:0/3559391390 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe468083980 con 0x7fe4680a6df0 2021-02-10T11:27:49.482 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:49.478 7fe46f883700 1 -- 172.21.2.102:0/3559391390 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe468083f10 con 0x7fe4680a6df0 2021-02-10T11:27:49.482 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:49.479 7fe459ffb700 1 -- 172.21.2.102:0/3559391390 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fe45c00c2f0 con 0x7fe4680a6df0 2021-02-10T11:27:49.482 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:49.479 7fe459ffb700 1 --2- 172.21.2.102:0/3559391390 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe4480186c0 0x7fe44801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:49.482 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:49.479 7fe459ffb700 1 -- 172.21.2.102:0/3559391390 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fe45c0328b0 con 0x7fe4680a6df0 2021-02-10T11:27:49.482 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:49.479 7fe46d69f700 1 --2- 172.21.2.102:0/3559391390 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe4480186c0 0x7fe44801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:49.482 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:49.479 7fe46d69f700 1 --2- 172.21.2.102:0/3559391390 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe4480186c0 0x7fe44801abc0 crc :-1 s=READY pgs=813 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:49.508 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:49.508 7fe46f883700 1 -- 172.21.2.102:0/3559391390 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe468082a70 con 0x7fe4680a6df0 2021-02-10T11:27:49.511 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:49.511 7fe459ffb700 1 -- 172.21.2.102:0/3559391390 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fe45c02cfc0 con 0x7fe4680a6df0 2021-02-10T11:27:49.641 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:49.641 7fe46f883700 1 -- 172.21.2.102:0/3559391390 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7fe468082a70 con 0x7fe4680a6df0 2021-02-10T11:27:49.641 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:49.641 7fe459ffb700 1 -- 172.21.2.102:0/3559391390 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v85) v1 ==== 92+0+10 (secure 0 0 0) 0x7fe45c02df50 con 0x7fe4680a6df0 2021-02-10T11:27:49.642 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:27:49.643 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:49.643 7fe46f883700 1 -- 172.21.2.102:0/3559391390 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe4480186c0 msgr2=0x7fe44801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:49.643 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:49.643 7fe46f883700 1 --2- 172.21.2.102:0/3559391390 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe4480186c0 0x7fe44801abc0 crc :-1 s=READY pgs=813 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:49.643 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:49.643 7fe46f883700 1 -- 172.21.2.102:0/3559391390 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe4680a6df0 msgr2=0x7fe4680a71f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:49.643 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:49.643 7fe46f883700 1 --2- 172.21.2.102:0/3559391390 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe4680a6df0 0x7fe4680a71f0 secure :-1 s=READY pgs=1798 cs=0 l=1 rev1=1 rx=0x7fe45c0099e0 tx=0x7fe45c00bd40).stop 2021-02-10T11:27:49.644 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:49.644 7fe46f883700 1 -- 172.21.2.102:0/3559391390 shutdown_connections 2021-02-10T11:27:49.644 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:49.644 7fe46f883700 1 --2- 172.21.2.102:0/3559391390 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe4480186c0 0x7fe44801abc0 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:49.644 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:49.644 7fe46f883700 1 --2- 172.21.2.102:0/3559391390 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe4680a5cd0 0x7fe4680a60d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:49.644 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:49.644 7fe46f883700 1 --2- 172.21.2.102:0/3559391390 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe4680a6df0 0x7fe4680a71f0 unknown :-1 s=CLOSED pgs=1798 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:49.644 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:49.644 7fe46f883700 1 --2- 172.21.2.102:0/3559391390 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe4680b22c0 0x7fe4680b26c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:49.644 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:49.644 7fe46f883700 1 -- 172.21.2.102:0/3559391390 >> 172.21.2.102:0/3559391390 conn(0x7fe46809ffd0 msgr2=0x7fe468082ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:49.644 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:49.644 7fe46f883700 1 -- 172.21.2.102:0/3559391390 shutdown_connections 2021-02-10T11:27:49.644 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:49.644 7fe46f883700 1 -- 172.21.2.102:0/3559391390 wait complete. 2021-02-10T11:27:49.652 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-02-10T11:27:50.083 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:50.066 7f89351e0700 1 Processor -- start 2021-02-10T11:27:50.083 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:50.066 7f89351e0700 1 -- start start 2021-02-10T11:27:50.083 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:50.066 7f89351e0700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f89300d1870 0x7f89301b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:50.083 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:50.066 7f89351e0700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f89301b0ae0 0x7f8930108940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:50.083 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:50.066 7f89351e0700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8930109a60 con 0x7f89300d1c70 2021-02-10T11:27:50.084 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:50.066 7f89351e0700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89300e5ef0 con 0x7f89301b0ae0 2021-02-10T11:27:50.084 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:50.066 7f89351e0700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8930109750 con 0x7f89300d1870 2021-02-10T11:27:50.084 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:50.066 7f892e59c700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f89301b0ae0 0x7f8930108940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:50.084 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:50.066 7f892e59c700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f89301b0ae0 0x7f8930108940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:59904/0 (socket says 172.21.2.102:59904) 2021-02-10T11:27:50.084 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:50.066 7f892e59c700 1 -- 172.21.2.102:0/2961888782 learned_addr learned my addr 172.21.2.102:0/2961888782 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:27:50.084 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:50.067 7f892e59c700 1 -- 172.21.2.102:0/2961888782 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f89300d1870 msgr2=0x7f89301b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:27:50.084 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:50.067 7f892e59c700 1 --2- 172.21.2.102:0/2961888782 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f89300d1870 0x7f89301b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:50.084 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:50.067 7f892e59c700 1 -- 172.21.2.102:0/2961888782 >> v1:172.21.2.113:6789/0 conn(0x7f89300d1c70 legacy=0x7f8930108f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:27:50.084 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:50.067 7f892e59c700 1 -- 172.21.2.102:0/2961888782 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8930088070 con 0x7f89301b0ae0 2021-02-10T11:27:50.084 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:50.067 7f892e59c700 1 --2- 172.21.2.102:0/2961888782 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f89301b0ae0 0x7f8930108940 secure :-1 s=READY pgs=1799 cs=0 l=1 rev1=1 rx=0x7f8924006700 tx=0x7f8924009b80).ready entity=mon.0 client_cookie=c0b3287ee185c637 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:50.084 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:50.067 7f892dd9b700 1 -- 172.21.2.102:0/2961888782 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f892400c0c0 con 0x7f89301b0ae0 2021-02-10T11:27:50.085 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:50.067 7f892dd9b700 1 -- 172.21.2.102:0/2961888782 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f892400c440 con 0x7f89301b0ae0 2021-02-10T11:27:50.085 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:50.067 7f892dd9b700 1 -- 172.21.2.102:0/2961888782 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f892400c960 con 0x7f89301b0ae0 2021-02-10T11:27:50.085 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:50.067 7f89351e0700 1 -- 172.21.2.102:0/2961888782 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f89301b0ae0 msgr2=0x7f8930108940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:50.085 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:50.067 7f89351e0700 1 --2- 172.21.2.102:0/2961888782 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f89301b0ae0 0x7f8930108940 secure :-1 s=READY pgs=1799 cs=0 l=1 rev1=1 rx=0x7f8924006700 tx=0x7f8924009b80).stop 2021-02-10T11:27:50.085 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:50.068 7f89351e0700 1 -- 172.21.2.102:0/2961888782 shutdown_connections 2021-02-10T11:27:50.085 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:50.068 7f89351e0700 1 --2- 172.21.2.102:0/2961888782 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f89301b0ae0 0x7f8930108940 unknown :-1 s=CLOSED pgs=1799 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:50.085 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:50.068 7f89351e0700 1 --2- 172.21.2.102:0/2961888782 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f89300d1870 0x7f89301b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:50.085 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:50.068 7f89351e0700 1 -- 172.21.2.102:0/2961888782 >> 172.21.2.102:0/2961888782 conn(0x7f89300ce420 msgr2=0x7f89300d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:50.085 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:50.068 7f89351e0700 1 -- 172.21.2.102:0/2961888782 shutdown_connections 2021-02-10T11:27:50.085 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:50.068 7f89351e0700 1 -- 172.21.2.102:0/2961888782 wait complete. 2021-02-10T11:27:50.085 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:50.083 7f89351e0700 1 Processor -- start 2021-02-10T11:27:50.085 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:50.083 7f89351e0700 1 -- start start 2021-02-10T11:27:50.087 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:50.083 7f89351e0700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f89300c2370 0x7f893018f870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:50.087 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:50.083 7f89351e0700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8930084cb0 0x7f89301a7470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:50.087 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:50.083 7f89351e0700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f89301a7a90 0x7f8930085700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:50.088 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:50.083 7f89351e0700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89300be840 con 0x7f89300c2370 2021-02-10T11:27:50.088 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:50.083 7f89351e0700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89300c29a0 con 0x7f8930084cb0 2021-02-10T11:27:50.088 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:50.083 7f89351e0700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89301a7e90 con 0x7f89301a7a90 2021-02-10T11:27:50.088 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:50.083 7f892ed9d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f89300c2370 0x7f893018f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:50.088 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:50.084 7f892ed9d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f89300c2370 0x7f893018f870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:59908/0 (socket says 172.21.2.102:59908) 2021-02-10T11:27:50.088 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:50.084 7f892ed9d700 1 -- 172.21.2.102:0/348211402 learned_addr learned my addr 172.21.2.102:0/348211402 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:27:50.088 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:50.084 7f892ed9d700 1 -- 172.21.2.102:0/348211402 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f89301a7a90 msgr2=0x7f8930085700 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:27:50.088 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:50.084 7f892ed9d700 1 --2- 172.21.2.102:0/348211402 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f89301a7a90 0x7f8930085700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:50.088 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:50.084 7f892ed9d700 1 -- 172.21.2.102:0/348211402 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8930084cb0 msgr2=0x7f89301a7470 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:27:50.088 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:50.084 7f892ed9d700 1 --2- 172.21.2.102:0/348211402 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8930084cb0 0x7f89301a7470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:50.088 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:50.084 7f892ed9d700 1 -- 172.21.2.102:0/348211402 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89301a80b0 con 0x7f89300c2370 2021-02-10T11:27:50.088 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:50.084 7f892ed9d700 1 --2- 172.21.2.102:0/348211402 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f89300c2370 0x7f893018f870 secure :-1 s=READY pgs=1800 cs=0 l=1 rev1=1 rx=0x7f8920006f00 tx=0x7f892000b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:50.089 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:50.084 7f891f7fe700 1 -- 172.21.2.102:0/348211402 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f89200009b0 con 0x7f89300c2370 2021-02-10T11:27:50.089 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:50.084 7f891f7fe700 1 -- 172.21.2.102:0/348211402 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89200063d0 con 0x7f89300c2370 2021-02-10T11:27:50.089 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:50.084 7f891f7fe700 1 -- 172.21.2.102:0/348211402 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8920000c90 con 0x7f89300c2370 2021-02-10T11:27:50.089 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:50.084 7f89351e0700 1 -- 172.21.2.102:0/348211402 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89301a80b0 con 0x7f89300c2370 2021-02-10T11:27:50.089 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:50.084 7f89351e0700 1 -- 172.21.2.102:0/348211402 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89301a80b0 con 0x7f89300c2370 2021-02-10T11:27:50.089 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:50.085 7f891f7fe700 1 -- 172.21.2.102:0/348211402 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f8920007000 con 0x7f89300c2370 2021-02-10T11:27:50.089 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:50.085 7f891f7fe700 1 --2- 172.21.2.102:0/348211402 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f89140186c0 0x7f891401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:50.089 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:50.085 7f891f7fe700 1 -- 172.21.2.102:0/348211402 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f8920032770 con 0x7f89300c2370 2021-02-10T11:27:50.089 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:50.085 7f892e59c700 1 --2- 172.21.2.102:0/348211402 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f89140186c0 0x7f891401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:50.089 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:50.085 7f892e59c700 1 --2- 172.21.2.102:0/348211402 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f89140186c0 0x7f891401abc0 crc :-1 s=READY pgs=814 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:50.117 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:50.116 7f89351e0700 1 -- 172.21.2.102:0/348211402 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89300be840 con 0x7f89300c2370 2021-02-10T11:27:50.119 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:50.119 7f891f7fe700 1 -- 172.21.2.102:0/348211402 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f8920006b90 con 0x7f89300c2370 2021-02-10T11:27:50.248 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:50.247 7f89351e0700 1 -- 172.21.2.102:0/348211402 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7f89300be840 con 0x7f89300c2370 2021-02-10T11:27:50.248 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:50.248 7f891f7fe700 1 -- 172.21.2.102:0/348211402 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v85) v1 ==== 104+0+10 (secure 0 0 0) 0x7f892000bd30 con 0x7f89300c2370 2021-02-10T11:27:50.249 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:27:50.250 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:50.250 7f89351e0700 1 -- 172.21.2.102:0/348211402 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f89140186c0 msgr2=0x7f891401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:50.250 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:50.250 7f89351e0700 1 --2- 172.21.2.102:0/348211402 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f89140186c0 0x7f891401abc0 crc :-1 s=READY pgs=814 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:50.250 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:50.250 7f89351e0700 1 -- 172.21.2.102:0/348211402 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f89300c2370 msgr2=0x7f893018f870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:50.250 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:50.250 7f89351e0700 1 --2- 172.21.2.102:0/348211402 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f89300c2370 0x7f893018f870 secure :-1 s=READY pgs=1800 cs=0 l=1 rev1=1 rx=0x7f8920006f00 tx=0x7f892000b940).stop 2021-02-10T11:27:50.251 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:50.250 7f89351e0700 1 -- 172.21.2.102:0/348211402 shutdown_connections 2021-02-10T11:27:50.251 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:50.250 7f89351e0700 1 --2- 172.21.2.102:0/348211402 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f89301a7a90 0x7f8930085700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:50.251 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:50.250 7f89351e0700 1 --2- 172.21.2.102:0/348211402 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f89140186c0 0x7f891401abc0 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:50.251 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:50.250 7f89351e0700 1 --2- 172.21.2.102:0/348211402 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8930084cb0 0x7f89301a7470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:50.251 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:50.250 7f89351e0700 1 --2- 172.21.2.102:0/348211402 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f89300c2370 0x7f893018f870 unknown :-1 s=CLOSED pgs=1800 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:50.251 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:50.250 7f89351e0700 1 -- 172.21.2.102:0/348211402 >> 172.21.2.102:0/348211402 conn(0x7f89300b5dc0 msgr2=0x7f89300c2d90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:50.251 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:50.250 7f89351e0700 1 -- 172.21.2.102:0/348211402 shutdown_connections 2021-02-10T11:27:50.251 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:50.250 7f89351e0700 1 -- 172.21.2.102:0/348211402 wait complete. 2021-02-10T11:27:50.259 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-02-10T11:27:50.689 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:50.672 7f9ebeef0700 1 Processor -- start 2021-02-10T11:27:50.689 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:50.672 7f9ebeef0700 1 -- start start 2021-02-10T11:27:50.689 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:50.673 7f9ebeef0700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f9eb810aff0 0x7f9eb80cf750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:50.689 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:50.673 7f9ebeef0700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9eb80fe1c0 con 0x7f9eb80d1870 2021-02-10T11:27:50.689 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:50.673 7f9ebeef0700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9eb81b0cb0 con 0x7f9eb80d1c70 2021-02-10T11:27:50.689 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:50.673 7f9ebeef0700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9eb8109a60 con 0x7f9eb810aff0 2021-02-10T11:27:50.689 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:50.673 7f9ebcd0c700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f9eb80d1870 0x7f9eb81b04c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:52140/0 (socket says 172.21.2.102:52140) 2021-02-10T11:27:50.690 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:50.673 7f9ebcd0c700 1 -- 172.21.2.102:0/132040382 learned_addr learned my addr 172.21.2.102:0/132040382 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:27:50.690 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:50.673 7f9ebd50d700 1 --2- 172.21.2.102:0/132040382 >> v2:172.21.2.113:3300/0 conn(0x7f9eb810aff0 0x7f9eb80cf750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:50.690 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:50.674 7f9eb77fe700 1 -- 172.21.2.102:0/132040382 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1302990646 0 0) 0x7f9ea00010b0 con 0x7f9eb80d1870 2021-02-10T11:27:50.690 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:50.674 7f9eb77fe700 1 -- 172.21.2.102:0/132040382 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ea4002fe0 con 0x7f9eb80d1870 2021-02-10T11:27:50.690 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:50.674 7f9eb77fe700 1 -- 172.21.2.102:0/132040382 <== mon.1 v1:172.21.2.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2797700020 0 0) 0x7f9ea8000ff0 con 0x7f9eb80d1c70 2021-02-10T11:27:50.690 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:50.674 7f9eb77fe700 1 -- 172.21.2.102:0/132040382 --> v1:172.21.2.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ea4004370 con 0x7f9eb80d1c70 2021-02-10T11:27:50.690 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:50.674 7f9ebd50d700 1 -- 172.21.2.102:0/132040382 >> v1:172.21.2.113:6789/0 conn(0x7f9eb80d1c70 legacy=0x7f9eb80cef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:50.690 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:50.674 7f9ebd50d700 1 -- 172.21.2.102:0/132040382 >> v1:172.21.2.102:6789/0 conn(0x7f9eb80d1870 legacy=0x7f9eb81b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:50.690 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:50.674 7f9ebd50d700 1 -- 172.21.2.102:0/132040382 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9eb810b3f0 con 0x7f9eb810aff0 2021-02-10T11:27:50.690 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:50.674 7f9ebd50d700 1 --2- 172.21.2.102:0/132040382 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f9eb810aff0 0x7f9eb80cf750 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 rx=0x7f9eb0006e50 tx=0x7f9eb0009e70).ready entity=mon.1 client_cookie=aa51be41100087a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:50.690 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:50.674 7f9eb77fe700 1 -- 172.21.2.102:0/132040382 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9eb0006a20 con 0x7f9eb810aff0 2021-02-10T11:27:50.691 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:50.674 7f9eb77fe700 1 -- 172.21.2.102:0/132040382 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9eb0006790 con 0x7f9eb810aff0 2021-02-10T11:27:50.691 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:50.674 7f9eb77fe700 1 -- 172.21.2.102:0/132040382 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9eb000b700 con 0x7f9eb810aff0 2021-02-10T11:27:50.691 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:50.674 7f9ebeef0700 1 -- 172.21.2.102:0/132040382 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f9eb810aff0 msgr2=0x7f9eb80cf750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:50.691 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:50.674 7f9ebeef0700 1 --2- 172.21.2.102:0/132040382 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f9eb810aff0 0x7f9eb80cf750 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 rx=0x7f9eb0006e50 tx=0x7f9eb0009e70).stop 2021-02-10T11:27:50.691 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:50.675 7f9ebeef0700 1 -- 172.21.2.102:0/132040382 shutdown_connections 2021-02-10T11:27:50.691 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:50.675 7f9ebeef0700 1 --2- 172.21.2.102:0/132040382 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f9eb810aff0 0x7f9eb80cf750 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:50.691 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:50.675 7f9ebeef0700 1 -- 172.21.2.102:0/132040382 >> 172.21.2.102:0/132040382 conn(0x7f9eb80ce420 msgr2=0x7f9eb80d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:50.691 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:50.675 7f9ebeef0700 1 -- 172.21.2.102:0/132040382 shutdown_connections 2021-02-10T11:27:50.691 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:50.675 7f9ebeef0700 1 -- 172.21.2.102:0/132040382 wait complete. 2021-02-10T11:27:50.693 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:50.689 7f9ebeef0700 1 Processor -- start 2021-02-10T11:27:50.693 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:50.689 7f9ebeef0700 1 -- start start 2021-02-10T11:27:50.694 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:50.690 7f9ebeef0700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9eb8191b10 0x7f9eb80ae640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:50.694 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:50.690 7f9ebeef0700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f9eb8191f10 0x7f9eb819ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:50.694 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:50.690 7f9ebeef0700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9eb81a05f0 0x7f9eb8082ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:50.694 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:50.690 7f9ebeef0700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9eb8192440 con 0x7f9eb8191b10 2021-02-10T11:27:50.694 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:50.690 7f9ebeef0700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9eb81916e0 con 0x7f9eb8191f10 2021-02-10T11:27:50.694 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:50.690 7f9ebeef0700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9eb8191900 con 0x7f9eb81a05f0 2021-02-10T11:27:50.694 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:50.690 7f9ebcd0c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9eb8191b10 0x7f9eb80ae640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:50.694 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:50.690 7f9ebcd0c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9eb8191b10 0x7f9eb80ae640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:59924/0 (socket says 172.21.2.102:59924) 2021-02-10T11:27:50.694 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:50.690 7f9ebcd0c700 1 -- 172.21.2.102:0/185212288 learned_addr learned my addr 172.21.2.102:0/185212288 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:27:50.694 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:50.690 7f9ebd50d700 1 --2- 172.21.2.102:0/185212288 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9eb81a05f0 0x7f9eb8082ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:50.694 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:50.690 7f9eb7fff700 1 --2- 172.21.2.102:0/185212288 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f9eb8191f10 0x7f9eb819ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:50.694 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:50.691 7f9ebcd0c700 1 -- 172.21.2.102:0/185212288 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9eb81a05f0 msgr2=0x7f9eb8082ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:50.695 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:50.691 7f9ebcd0c700 1 --2- 172.21.2.102:0/185212288 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9eb81a05f0 0x7f9eb8082ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:50.695 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:50.691 7f9ebcd0c700 1 -- 172.21.2.102:0/185212288 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f9eb8191f10 msgr2=0x7f9eb819ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:50.695 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:50.691 7f9ebcd0c700 1 --2- 172.21.2.102:0/185212288 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f9eb8191f10 0x7f9eb819ffd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:50.695 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:50.691 7f9ebcd0c700 1 -- 172.21.2.102:0/185212288 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9eb80834f0 con 0x7f9eb8191b10 2021-02-10T11:27:50.695 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:50.691 7f9eb7fff700 1 --2- 172.21.2.102:0/185212288 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f9eb8191f10 0x7f9eb819ffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:27:50.695 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:50.691 7f9ebcd0c700 1 --2- 172.21.2.102:0/185212288 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9eb8191b10 0x7f9eb80ae640 secure :-1 s=READY pgs=1802 cs=0 l=1 rev1=1 rx=0x7f9ea0001c60 tx=0x7f9ea000a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:50.695 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:50.691 7f9eb57fa700 1 -- 172.21.2.102:0/185212288 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9ea0000e00 con 0x7f9eb8191b10 2021-02-10T11:27:50.695 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:50.691 7f9eb57fa700 1 -- 172.21.2.102:0/185212288 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ea000acc0 con 0x7f9eb8191b10 2021-02-10T11:27:50.695 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:50.691 7f9ebeef0700 1 -- 172.21.2.102:0/185212288 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9eb80834f0 con 0x7f9eb8191b10 2021-02-10T11:27:50.695 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:50.691 7f9eb57fa700 1 -- 172.21.2.102:0/185212288 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9ea0004540 con 0x7f9eb8191b10 2021-02-10T11:27:50.695 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:50.691 7f9ebeef0700 1 -- 172.21.2.102:0/185212288 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9eb80834f0 con 0x7f9eb8191b10 2021-02-10T11:27:50.695 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:50.691 7f9eb57fa700 1 -- 172.21.2.102:0/185212288 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f9ea0004140 con 0x7f9eb8191b10 2021-02-10T11:27:50.696 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:50.691 7f9eb57fa700 1 --2- 172.21.2.102:0/185212288 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f9e9c0186d0 0x7f9e9c01abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:50.696 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:50.691 7f9eb57fa700 1 -- 172.21.2.102:0/185212288 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f9ea0032610 con 0x7f9eb8191b10 2021-02-10T11:27:50.696 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:50.692 7f9eb7fff700 1 --2- 172.21.2.102:0/185212288 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f9e9c0186d0 0x7f9e9c01abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:50.696 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:50.692 7f9eb7fff700 1 --2- 172.21.2.102:0/185212288 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f9e9c0186d0 0x7f9e9c01abd0 crc :-1 s=READY pgs=815 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:50.723 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:50.723 7f9ebeef0700 1 -- 172.21.2.102:0/185212288 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9eb8192440 con 0x7f9eb8191b10 2021-02-10T11:27:50.725 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:50.725 7f9eb57fa700 1 -- 172.21.2.102:0/185212288 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f9ea0000e00 con 0x7f9eb8191b10 2021-02-10T11:27:50.854 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:50.853 7f9ebeef0700 1 -- 172.21.2.102:0/185212288 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7f9eb8192440 con 0x7f9eb8191b10 2021-02-10T11:27:50.854 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:50.854 7f9eb57fa700 1 -- 172.21.2.102:0/185212288 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v85) v1 ==== 100+0+10 (secure 0 0 0) 0x7f9ea0004140 con 0x7f9eb8191b10 2021-02-10T11:27:50.854 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:27:50.856 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:50.856 7f9ebeef0700 1 -- 172.21.2.102:0/185212288 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f9e9c0186d0 msgr2=0x7f9e9c01abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:50.856 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:50.856 7f9ebeef0700 1 --2- 172.21.2.102:0/185212288 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f9e9c0186d0 0x7f9e9c01abd0 crc :-1 s=READY pgs=815 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:50.856 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:50.856 7f9ebeef0700 1 -- 172.21.2.102:0/185212288 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9eb8191b10 msgr2=0x7f9eb80ae640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:50.856 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:50.856 7f9ebeef0700 1 --2- 172.21.2.102:0/185212288 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9eb8191b10 0x7f9eb80ae640 secure :-1 s=READY pgs=1802 cs=0 l=1 rev1=1 rx=0x7f9ea0001c60 tx=0x7f9ea000a8d0).stop 2021-02-10T11:27:50.856 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:50.856 7f9ebeef0700 1 -- 172.21.2.102:0/185212288 shutdown_connections 2021-02-10T11:27:50.857 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:50.856 7f9ebeef0700 1 --2- 172.21.2.102:0/185212288 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9eb81a05f0 0x7f9eb8082ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:50.857 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:50.856 7f9ebeef0700 1 --2- 172.21.2.102:0/185212288 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f9e9c0186d0 0x7f9e9c01abd0 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:50.857 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:50.856 7f9ebeef0700 1 --2- 172.21.2.102:0/185212288 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f9eb8191f10 0x7f9eb819ffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:50.857 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:50.856 7f9ebeef0700 1 --2- 172.21.2.102:0/185212288 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9eb8191b10 0x7f9eb80ae640 unknown :-1 s=CLOSED pgs=1802 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:50.857 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:50.856 7f9ebeef0700 1 -- 172.21.2.102:0/185212288 >> 172.21.2.102:0/185212288 conn(0x7f9eb80a02d0 msgr2=0x7f9eb809f200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:50.857 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:50.856 7f9ebeef0700 1 -- 172.21.2.102:0/185212288 shutdown_connections 2021-02-10T11:27:50.857 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:50.856 7f9ebeef0700 1 -- 172.21.2.102:0/185212288 wait complete. 2021-02-10T11:27:50.865 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-02-10T11:27:50.865 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:27:51.291 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:51.275 7fc6fb8f6700 1 Processor -- start 2021-02-10T11:27:51.292 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:51.275 7fc6fb8f6700 1 -- start start 2021-02-10T11:27:51.292 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:51.275 7fc6fb8f6700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fc6f41b0870 0x7fc6f40f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:51.292 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:51.275 7fc6fb8f6700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fc6f40efe00 0x7fc6f40f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:51.292 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:51.275 7fc6fb8f6700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6f41b0c70 con 0x7fc6f41b0470 2021-02-10T11:27:51.292 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:51.275 7fc6fb8f6700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6f40e5ea0 con 0x7fc6f41b0870 2021-02-10T11:27:51.292 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:51.275 7fc6fb8f6700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6f40f0b50 con 0x7fc6f40efe00 2021-02-10T11:27:51.292 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:51.275 7fc6f9712700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fc6f41b0470 0x7fc6f40ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:52154/0 (socket says 172.21.2.102:52154) 2021-02-10T11:27:51.292 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:51.275 7fc6f9712700 1 -- 172.21.2.102:0/821411811 learned_addr learned my addr 172.21.2.102:0/821411811 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:27:51.292 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:51.275 7fc6f8f11700 1 --2- 172.21.2.102:0/821411811 >> v2:172.21.2.102:3300/0 conn(0x7fc6f41b0870 0x7fc6f40f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:51.292 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:51.275 7fc6f9f13700 1 --2- 172.21.2.102:0/821411811 >> v2:172.21.2.113:3300/0 conn(0x7fc6f40efe00 0x7fc6f40f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:51.292 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:51.276 7fc6ebfff700 1 -- 172.21.2.102:0/821411811 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2527462696 0 0) 0x7fc6f0000ce0 con 0x7fc6f41b0470 2021-02-10T11:27:51.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:51.276 7fc6ebfff700 1 -- 172.21.2.102:0/821411811 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6e0002fe0 con 0x7fc6f41b0470 2021-02-10T11:27:51.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:51.276 7fc6f9f13700 1 -- 172.21.2.102:0/821411811 >> v2:172.21.2.102:3300/0 conn(0x7fc6f41b0870 msgr2=0x7fc6f40f20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:27:51.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:51.276 7fc6f9f13700 1 --2- 172.21.2.102:0/821411811 >> v2:172.21.2.102:3300/0 conn(0x7fc6f41b0870 0x7fc6f40f20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:51.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:51.276 7fc6f9f13700 1 -- 172.21.2.102:0/821411811 >> v1:172.21.2.102:6789/0 conn(0x7fc6f41b0470 legacy=0x7fc6f40ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:51.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:51.276 7fc6f9f13700 1 -- 172.21.2.102:0/821411811 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6f40f0820 con 0x7fc6f40efe00 2021-02-10T11:27:51.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:51.276 7fc6f8f11700 1 --2- 172.21.2.102:0/821411811 >> v2:172.21.2.102:3300/0 conn(0x7fc6f41b0870 0x7fc6f40f20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:27:51.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:51.276 7fc6f9f13700 1 --2- 172.21.2.102:0/821411811 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc6f40efe00 0x7fc6f40f0200 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 rx=0x7fc6ec006e50 tx=0x7fc6ec009e70).ready entity=mon.1 client_cookie=289b408faeeac4ee server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:51.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:51.276 7fc6ebfff700 1 -- 172.21.2.102:0/821411811 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc6ec006a20 con 0x7fc6f40efe00 2021-02-10T11:27:51.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:51.276 7fc6ebfff700 1 -- 172.21.2.102:0/821411811 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6ec006790 con 0x7fc6f40efe00 2021-02-10T11:27:51.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:51.276 7fc6ebfff700 1 -- 172.21.2.102:0/821411811 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc6ec000f90 con 0x7fc6f40efe00 2021-02-10T11:27:51.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:51.276 7fc6fb8f6700 1 -- 172.21.2.102:0/821411811 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc6f40efe00 msgr2=0x7fc6f40f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:51.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:51.276 7fc6fb8f6700 1 --2- 172.21.2.102:0/821411811 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc6f40efe00 0x7fc6f40f0200 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 rx=0x7fc6ec006e50 tx=0x7fc6ec009e70).stop 2021-02-10T11:27:51.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:51.277 7fc6fb8f6700 1 -- 172.21.2.102:0/821411811 shutdown_connections 2021-02-10T11:27:51.294 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:51.277 7fc6fb8f6700 1 --2- 172.21.2.102:0/821411811 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc6f40efe00 0x7fc6f40f0200 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:51.294 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:51.277 7fc6fb8f6700 1 --2- 172.21.2.102:0/821411811 >> v2:172.21.2.102:3300/0 conn(0x7fc6f41b0870 0x7fc6f40f20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:51.294 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:51.277 7fc6fb8f6700 1 -- 172.21.2.102:0/821411811 >> 172.21.2.102:0/821411811 conn(0x7fc6f40ce3e0 msgr2=0x7fc6f40e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:51.294 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:51.277 7fc6fb8f6700 1 -- 172.21.2.102:0/821411811 shutdown_connections 2021-02-10T11:27:51.294 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:51.277 7fc6fb8f6700 1 -- 172.21.2.102:0/821411811 wait complete. 2021-02-10T11:27:51.294 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:51.292 7fc6fb8f6700 1 Processor -- start 2021-02-10T11:27:51.294 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:51.292 7fc6fb8f6700 1 -- start start 2021-02-10T11:27:51.295 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:51.292 7fc6fb8f6700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc6f41af8a0 0x7fc6f4198e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:51.295 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:51.292 7fc6fb8f6700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc6f41af050 0x7fc6f40aa3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:51.296 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:51.292 7fc6fb8f6700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc6f4197d00 0x7fc6f4198100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:51.296 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:51.292 7fc6fb8f6700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6f41988e0 con 0x7fc6f4197d00 2021-02-10T11:27:51.296 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:51.292 7fc6fb8f6700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6f41af450 con 0x7fc6f41af050 2021-02-10T11:27:51.296 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:51.292 7fc6fb8f6700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6f40aaa10 con 0x7fc6f41af8a0 2021-02-10T11:27:51.296 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:51.292 7fc6f9f13700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc6f4197d00 0x7fc6f4198100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:51.296 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:51.292 7fc6f9f13700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc6f4197d00 0x7fc6f4198100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:59940/0 (socket says 172.21.2.102:59940) 2021-02-10T11:27:51.296 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:51.292 7fc6f9f13700 1 -- 172.21.2.102:0/3284128998 learned_addr learned my addr 172.21.2.102:0/3284128998 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:27:51.296 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:51.293 7fc6f9712700 1 --2- 172.21.2.102:0/3284128998 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc6f41af8a0 0x7fc6f4198e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:51.296 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:51.293 7fc6f8f11700 1 --2- 172.21.2.102:0/3284128998 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc6f41af050 0x7fc6f40aa3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:51.296 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:51.293 7fc6f9f13700 1 -- 172.21.2.102:0/3284128998 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc6f41af8a0 msgr2=0x7fc6f4198e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:51.296 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:51.293 7fc6f9f13700 1 --2- 172.21.2.102:0/3284128998 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc6f41af8a0 0x7fc6f4198e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:51.296 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:51.293 7fc6f9f13700 1 -- 172.21.2.102:0/3284128998 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc6f41af050 msgr2=0x7fc6f40aa3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:51.296 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:51.293 7fc6f9f13700 1 --2- 172.21.2.102:0/3284128998 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc6f41af050 0x7fc6f40aa3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:51.297 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:51.293 7fc6f9f13700 1 -- 172.21.2.102:0/3284128998 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6f419fec0 con 0x7fc6f4197d00 2021-02-10T11:27:51.297 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:51.293 7fc6f8f11700 1 --2- 172.21.2.102:0/3284128998 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc6f41af050 0x7fc6f40aa3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:27:51.297 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:51.293 7fc6f9f13700 1 --2- 172.21.2.102:0/3284128998 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc6f4197d00 0x7fc6f4198100 secure :-1 s=READY pgs=1804 cs=0 l=1 rev1=1 rx=0x7fc6ec0099f0 tx=0x7fc6ec00bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:51.297 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:51.293 7fc6e9ffb700 1 -- 172.21.2.102:0/3284128998 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc6ec0093d0 con 0x7fc6f4197d00 2021-02-10T11:27:51.297 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:51.293 7fc6e9ffb700 1 -- 172.21.2.102:0/3284128998 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6ec0090c0 con 0x7fc6f4197d00 2021-02-10T11:27:51.297 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:51.293 7fc6e9ffb700 1 -- 172.21.2.102:0/3284128998 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc6ec001420 con 0x7fc6f4197d00 2021-02-10T11:27:51.297 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:51.293 7fc6fb8f6700 1 -- 172.21.2.102:0/3284128998 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6f419fec0 con 0x7fc6f4197d00 2021-02-10T11:27:51.297 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:51.293 7fc6fb8f6700 1 -- 172.21.2.102:0/3284128998 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6f419fec0 con 0x7fc6f4197d00 2021-02-10T11:27:51.297 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:51.294 7fc6e9ffb700 1 -- 172.21.2.102:0/3284128998 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fc6ec0056b0 con 0x7fc6f4197d00 2021-02-10T11:27:51.297 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:51.294 7fc6e9ffb700 1 --2- 172.21.2.102:0/3284128998 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc6d40186d0 0x7fc6d401abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:51.297 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:51.294 7fc6e9ffb700 1 -- 172.21.2.102:0/3284128998 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fc6ec0329a0 con 0x7fc6f4197d00 2021-02-10T11:27:51.297 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:51.294 7fc6f9712700 1 --2- 172.21.2.102:0/3284128998 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc6d40186d0 0x7fc6d401abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:51.298 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:51.294 7fc6f9712700 1 --2- 172.21.2.102:0/3284128998 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc6d40186d0 0x7fc6d401abd0 crc :-1 s=READY pgs=816 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:51.325 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:51.325 7fc6fb8f6700 1 -- 172.21.2.102:0/3284128998 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6f4105a20 con 0x7fc6f4197d00 2021-02-10T11:27:51.327 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:51.327 7fc6e9ffb700 1 -- 172.21.2.102:0/3284128998 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fc6ec0056b0 con 0x7fc6f4197d00 2021-02-10T11:27:51.458 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:51.457 7fc6fb8f6700 1 -- 172.21.2.102:0/3284128998 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc6f40f0b50 con 0x7fc6f4197d00 2021-02-10T11:27:51.458 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:51.458 7fc6e9ffb700 1 -- 172.21.2.102:0/3284128998 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 65 v65) v1 ==== 94+0+2101 (secure 0 0 0) 0x7fc6ec000df0 con 0x7fc6f4197d00 2021-02-10T11:27:51.459 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:27:51.459 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":65,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6110,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.102:6835/1471871278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6834","nonce":1471871278},{"type":"v1","addr":"172.21.2.102:6835","nonce":1471871278}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":61}],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:27:41.479375","modified":"2021-02-10 11:27:43.490605","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6107},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6107":{"gid":6107,"name":"b","rank":0,"incarnation":62,"state":"up:active","state_seq":3,"addr":"172.21.2.113:6833/4220238062","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6832","nonce":4220238062},{"type":"v1","addr":"172.21.2.113:6833","nonce":4220238062}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":8}]} 2021-02-10T11:27:51.459 INFO:teuthology.orchestra.run.gibba002.stderr:dumped fsmap epoch 65 2021-02-10T11:27:51.461 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:51.459 7fc6fb8f6700 1 -- 172.21.2.102:0/3284128998 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc6d40186d0 msgr2=0x7fc6d401abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:51.461 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:51.459 7fc6fb8f6700 1 --2- 172.21.2.102:0/3284128998 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc6d40186d0 0x7fc6d401abd0 crc :-1 s=READY pgs=816 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:51.461 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:51.459 7fc6fb8f6700 1 -- 172.21.2.102:0/3284128998 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc6f4197d00 msgr2=0x7fc6f4198100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:51.461 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:51.459 7fc6fb8f6700 1 --2- 172.21.2.102:0/3284128998 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc6f4197d00 0x7fc6f4198100 secure :-1 s=READY pgs=1804 cs=0 l=1 rev1=1 rx=0x7fc6ec0099f0 tx=0x7fc6ec00bd10).stop 2021-02-10T11:27:51.462 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:51.460 7fc6fb8f6700 1 -- 172.21.2.102:0/3284128998 shutdown_connections 2021-02-10T11:27:51.462 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:51.460 7fc6fb8f6700 1 --2- 172.21.2.102:0/3284128998 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc6f4197d00 0x7fc6f4198100 unknown :-1 s=CLOSED pgs=1804 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:51.462 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:51.460 7fc6fb8f6700 1 --2- 172.21.2.102:0/3284128998 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc6d40186d0 0x7fc6d401abd0 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:51.462 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:51.460 7fc6fb8f6700 1 --2- 172.21.2.102:0/3284128998 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc6f41af050 0x7fc6f40aa3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:51.462 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:51.460 7fc6fb8f6700 1 --2- 172.21.2.102:0/3284128998 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc6f41af8a0 0x7fc6f4198e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:51.463 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:51.460 7fc6fb8f6700 1 -- 172.21.2.102:0/3284128998 >> 172.21.2.102:0/3284128998 conn(0x7fc6f41a37a0 msgr2=0x7fc6f418c230 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:51.463 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:51.460 7fc6fb8f6700 1 -- 172.21.2.102:0/3284128998 shutdown_connections 2021-02-10T11:27:51.463 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:51.460 7fc6fb8f6700 1 -- 172.21.2.102:0/3284128998 wait complete. 2021-02-10T11:27:51.670 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 65, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:27:41.479375', 'modified': '2021-02-10 11:27:43.490605', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6107}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6107': {'gid': 6107, 'name': 'b', 'rank': 0, 'incarnation': 62, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.113:6833/4220238062', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.113:6832', 'nonce': 4220238062}, {'type': 'v1', 'addr': '172.21.2.113:6833', 'nonce': 4220238062}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [17], 'metadata_pool': 16, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:27:51.670 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:27:51.670 DEBUG:teuthology.orchestra.run.gibba013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:27:52.069 INFO:teuthology.orchestra.run.gibba013.stdout:{ 2021-02-10T11:27:52.069 INFO:teuthology.orchestra.run.gibba013.stdout: "cluster_fsid": "68db30ea-dbce-4aef-a6df-f01c6fea0513", 2021-02-10T11:27:52.069 INFO:teuthology.orchestra.run.gibba013.stdout: "whoami": 0, 2021-02-10T11:27:52.069 INFO:teuthology.orchestra.run.gibba013.stdout: "id": 6107, 2021-02-10T11:27:52.070 INFO:teuthology.orchestra.run.gibba013.stdout: "want_state": "up:active", 2021-02-10T11:27:52.070 INFO:teuthology.orchestra.run.gibba013.stdout: "state": "up:active", 2021-02-10T11:27:52.070 INFO:teuthology.orchestra.run.gibba013.stdout: "rank_uptime": 10.588669428999999, 2021-02-10T11:27:52.070 INFO:teuthology.orchestra.run.gibba013.stdout: "mdsmap_epoch": 65, 2021-02-10T11:27:52.070 INFO:teuthology.orchestra.run.gibba013.stdout: "osdmap_epoch": 85, 2021-02-10T11:27:52.070 INFO:teuthology.orchestra.run.gibba013.stdout: "osdmap_epoch_barrier": 85, 2021-02-10T11:27:52.070 INFO:teuthology.orchestra.run.gibba013.stdout: "uptime": 16.237487303000002 2021-02-10T11:27:52.070 INFO:teuthology.orchestra.run.gibba013.stdout:} 2021-02-10T11:27:52.073 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "68db30ea-dbce-4aef-a6df-f01c6fea0513", "whoami": 0, "id": 6107, "want_state": "up:active", "state": "up:active", "rank_uptime": 10.588669428999999, "mdsmap_epoch": 65, "osdmap_epoch": 85, "osdmap_epoch_barrier": 85, "uptime": 16.237487303000002 } 2021-02-10T11:27:52.074 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2021-02-10T11:27:52.074 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2021-02-10T11:27:52.074 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@gibba002.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2021-02-10T11:27:52.074 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:52.074 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec mkdir -p /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:27:52.409 DEBUG:teuthology.orchestra.run.gibba002:> sudo modprobe fuse 2021-02-10T11:27:52.754 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:27:52.754 DEBUG:teuthology.orchestra.run.gibba002:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:27:53.097 INFO:teuthology.orchestra.run.gibba002.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:27:53.097 INFO:teuthology.orchestra.run.gibba002.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:27:53.098 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:27:53.098 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:53.098 DEBUG:teuthology.orchestra.run.gibba002:> ls /sys/fs/fuse/connections 2021-02-10T11:27:53.433 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2021-02-10T11:27:53.434 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:27:53.436 DEBUG:teuthology.orchestra.run.gibba002:> sudo modprobe fuse 2021-02-10T11:27:53.780 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:27:53.781 DEBUG:teuthology.orchestra.run.gibba002:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:27:53.846 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba002.stderr:ceph-fuse[169377]: starting ceph client 2021-02-10T11:27:53.846 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba002.stderr:2021-02-10 11:27:53.846 7fc3ed524f80 -1 init, newargv = 0x564d84e3c980 newargc=9 2021-02-10T11:27:53.851 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba002.stderr:ceph-fuse[169377]: starting fuse 2021-02-10T11:27:54.124 INFO:teuthology.orchestra.run.gibba002.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:27:54.124 INFO:teuthology.orchestra.run.gibba002.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:27:54.124 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:27:54.125 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:54.125 DEBUG:teuthology.orchestra.run.gibba002:> ls /sys/fs/fuse/connections 2021-02-10T11:27:54.460 INFO:teuthology.orchestra.run.gibba002.stdout:43 2021-02-10T11:27:54.461 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2021-02-10T11:27:54.461 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:54.461 DEBUG:teuthology.orchestra.run.gibba002:> sudo python3 -c ' 2021-02-10T11:27:54.461 DEBUG:teuthology.orchestra.run.gibba002:> import glob 2021-02-10T11:27:54.461 DEBUG:teuthology.orchestra.run.gibba002:> import re 2021-02-10T11:27:54.461 DEBUG:teuthology.orchestra.run.gibba002:> import os 2021-02-10T11:27:54.461 DEBUG:teuthology.orchestra.run.gibba002:> import subprocess 2021-02-10T11:27:54.461 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:27:54.461 DEBUG:teuthology.orchestra.run.gibba002:> def find_socket(client_name): 2021-02-10T11:27:54.461 DEBUG:teuthology.orchestra.run.gibba002:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2021-02-10T11:27:54.462 DEBUG:teuthology.orchestra.run.gibba002:> files = glob.glob(asok_path) 2021-02-10T11:27:54.462 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:27:54.462 DEBUG:teuthology.orchestra.run.gibba002:> # Given a non-glob path, it better be there 2021-02-10T11:27:54.462 DEBUG:teuthology.orchestra.run.gibba002:> if "*" not in asok_path: 2021-02-10T11:27:54.462 DEBUG:teuthology.orchestra.run.gibba002:> assert(len(files) == 1) 2021-02-10T11:27:54.462 DEBUG:teuthology.orchestra.run.gibba002:> return files[0] 2021-02-10T11:27:54.462 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:27:54.462 DEBUG:teuthology.orchestra.run.gibba002:> for f in files: 2021-02-10T11:27:54.462 DEBUG:teuthology.orchestra.run.gibba002:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2021-02-10T11:27:54.462 DEBUG:teuthology.orchestra.run.gibba002:> if os.path.exists("/proc/{0}".format(pid)): 2021-02-10T11:27:54.462 DEBUG:teuthology.orchestra.run.gibba002:> return f 2021-02-10T11:27:54.462 DEBUG:teuthology.orchestra.run.gibba002:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2021-02-10T11:27:54.462 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:27:54.463 DEBUG:teuthology.orchestra.run.gibba002:> print(find_socket("client.0")) 2021-02-10T11:27:54.463 DEBUG:teuthology.orchestra.run.gibba002:> ' 2021-02-10T11:27:54.825 INFO:teuthology.orchestra.run.gibba002.stdout:/var/run/ceph/ceph-client.0.169377.asok 2021-02-10T11:27:54.827 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.169377.asok 2021-02-10T11:27:54.828 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:54.828 DEBUG:teuthology.orchestra.run.gibba002:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.169377.asok status 2021-02-10T11:27:55.218 INFO:teuthology.orchestra.run.gibba002.stdout:{ 2021-02-10T11:27:55.218 INFO:teuthology.orchestra.run.gibba002.stdout: "metadata": { 2021-02-10T11:27:55.218 INFO:teuthology.orchestra.run.gibba002.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:27:55.218 INFO:teuthology.orchestra.run.gibba002.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:27:55.218 INFO:teuthology.orchestra.run.gibba002.stdout: "entity_id": "0", 2021-02-10T11:27:55.218 INFO:teuthology.orchestra.run.gibba002.stdout: "hostname": "gibba002", 2021-02-10T11:27:55.218 INFO:teuthology.orchestra.run.gibba002.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2021-02-10T11:27:55.218 INFO:teuthology.orchestra.run.gibba002.stdout: "pid": "169377", 2021-02-10T11:27:55.219 INFO:teuthology.orchestra.run.gibba002.stdout: "root": "/" 2021-02-10T11:27:55.219 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:27:55.219 INFO:teuthology.orchestra.run.gibba002.stdout: "dentry_count": 0, 2021-02-10T11:27:55.219 INFO:teuthology.orchestra.run.gibba002.stdout: "dentry_pinned_count": 0, 2021-02-10T11:27:55.219 INFO:teuthology.orchestra.run.gibba002.stdout: "id": 9393, 2021-02-10T11:27:55.219 INFO:teuthology.orchestra.run.gibba002.stdout: "inst": { 2021-02-10T11:27:55.219 INFO:teuthology.orchestra.run.gibba002.stdout: "name": { 2021-02-10T11:27:55.219 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "client", 2021-02-10T11:27:55.219 INFO:teuthology.orchestra.run.gibba002.stdout: "num": 9393 2021-02-10T11:27:55.219 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:27:55.219 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": { 2021-02-10T11:27:55.219 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:27:55.219 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:0", 2021-02-10T11:27:55.220 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 3618432722 2021-02-10T11:27:55.220 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:27:55.220 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:27:55.220 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": { 2021-02-10T11:27:55.220 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:27:55.220 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:0", 2021-02-10T11:27:55.220 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 3618432722 2021-02-10T11:27:55.220 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:27:55.220 INFO:teuthology.orchestra.run.gibba002.stdout: "inst_str": "client.9393 172.21.2.102:0/3618432722", 2021-02-10T11:27:55.220 INFO:teuthology.orchestra.run.gibba002.stdout: "addr_str": "172.21.2.102:0/3618432722", 2021-02-10T11:27:55.220 INFO:teuthology.orchestra.run.gibba002.stdout: "inode_count": 1, 2021-02-10T11:27:55.220 INFO:teuthology.orchestra.run.gibba002.stdout: "mds_epoch": 65, 2021-02-10T11:27:55.220 INFO:teuthology.orchestra.run.gibba002.stdout: "osd_epoch": 85, 2021-02-10T11:27:55.220 INFO:teuthology.orchestra.run.gibba002.stdout: "osd_epoch_barrier": 0, 2021-02-10T11:27:55.220 INFO:teuthology.orchestra.run.gibba002.stdout: "blacklisted": false 2021-02-10T11:27:55.221 INFO:teuthology.orchestra.run.gibba002.stdout:} 2021-02-10T11:27:55.222 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:55.222 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:27:55.222 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:27:55.556 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:27:55.557 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:27:55.557 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:55.557 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:27:55.903 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:27:56.331 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:56.314 7fd1e333a700 1 Processor -- start 2021-02-10T11:27:56.331 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:56.314 7fd1e333a700 1 -- start start 2021-02-10T11:27:56.331 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:56.315 7fd1e333a700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fd1dc0d1870 0x7fd1dc1b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:56.331 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:56.315 7fd1e333a700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd1dc0f0650 0x7fd1dc0cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:56.331 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:56.315 7fd1e333a700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1dc0f0b10 con 0x7fd1dc1b0a10 2021-02-10T11:27:56.331 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:56.315 7fd1e333a700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1dc0e5ed0 con 0x7fd1dc0d1870 2021-02-10T11:27:56.331 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:56.315 7fd1e333a700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1dc0d1c70 con 0x7fd1dc0f0650 2021-02-10T11:27:56.332 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:56.315 7fd1e1156700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fd1dc0d1870 0x7fd1dc1b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:56.332 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:56.315 7fd1e0955700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fd1dc1b0a10 0x7fd1dc0cf330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:52182/0 (socket says 172.21.2.102:52182) 2021-02-10T11:27:56.332 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:56.315 7fd1e0955700 1 -- 172.21.2.102:0/1805803253 learned_addr learned my addr 172.21.2.102:0/1805803253 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:27:56.332 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:56.315 7fd1e1156700 1 -- 172.21.2.102:0/1805803253 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd1dc0f0650 msgr2=0x7fd1dc0cfb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:27:56.332 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:56.315 7fd1e1156700 1 --2- 172.21.2.102:0/1805803253 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd1dc0f0650 0x7fd1dc0cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:56.332 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:56.315 7fd1e1156700 1 -- 172.21.2.102:0/1805803253 >> v1:172.21.2.102:6789/0 conn(0x7fd1dc1b0a10 legacy=0x7fd1dc0cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:56.332 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:56.315 7fd1e1156700 1 -- 172.21.2.102:0/1805803253 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1dc0d1e90 con 0x7fd1dc0d1870 2021-02-10T11:27:56.332 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:56.315 7fd1e1156700 1 --2- 172.21.2.102:0/1805803253 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd1dc0d1870 0x7fd1dc1b03f0 secure :-1 s=READY pgs=1809 cs=0 l=1 rev1=1 rx=0x7fd1d0006650 tx=0x7fd1d0009a80).ready entity=mon.0 client_cookie=d7c3c7c98dd74a8e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:56.332 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:56.316 7fd1cffff700 1 -- 172.21.2.102:0/1805803253 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd1d0006390 con 0x7fd1dc0d1870 2021-02-10T11:27:56.332 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:56.316 7fd1cffff700 1 -- 172.21.2.102:0/1805803253 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1d00009b0 con 0x7fd1dc0d1870 2021-02-10T11:27:56.332 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:56.316 7fd1cffff700 1 -- 172.21.2.102:0/1805803253 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd1d000c2b0 con 0x7fd1dc0d1870 2021-02-10T11:27:56.332 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:56.316 7fd1e333a700 1 -- 172.21.2.102:0/1805803253 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd1dc0d1870 msgr2=0x7fd1dc1b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:56.332 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:56.316 7fd1e333a700 1 --2- 172.21.2.102:0/1805803253 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd1dc0d1870 0x7fd1dc1b03f0 secure :-1 s=READY pgs=1809 cs=0 l=1 rev1=1 rx=0x7fd1d0006650 tx=0x7fd1d0009a80).stop 2021-02-10T11:27:56.333 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:56.316 7fd1e333a700 1 -- 172.21.2.102:0/1805803253 shutdown_connections 2021-02-10T11:27:56.333 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:56.316 7fd1e333a700 1 --2- 172.21.2.102:0/1805803253 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd1dc0f0650 0x7fd1dc0cfb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:56.333 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:56.316 7fd1e333a700 1 --2- 172.21.2.102:0/1805803253 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd1dc0d1870 0x7fd1dc1b03f0 unknown :-1 s=CLOSED pgs=1809 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:56.333 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:56.316 7fd1e333a700 1 -- 172.21.2.102:0/1805803253 >> 172.21.2.102:0/1805803253 conn(0x7fd1dc0ce450 msgr2=0x7fd1dc0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:56.333 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:56.316 7fd1e333a700 1 -- 172.21.2.102:0/1805803253 shutdown_connections 2021-02-10T11:27:56.333 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:56.316 7fd1e333a700 1 -- 172.21.2.102:0/1805803253 wait complete. 2021-02-10T11:27:56.334 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:56.331 7fd1e333a700 1 Processor -- start 2021-02-10T11:27:56.334 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:56.331 7fd1e333a700 1 -- start start 2021-02-10T11:27:56.335 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:56.332 7fd1e333a700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd1dc0b0ad0 0x7fd1dc091f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:56.335 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:56.332 7fd1e333a700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd1dc1a8850 0x7fd1dc1a8c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:56.335 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:56.332 7fd1e333a700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd1dc1a6fa0 0x7fd1dc1a73a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:56.335 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:56.332 7fd1e333a700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1dc1abfe0 con 0x7fd1dc0b0ad0 2021-02-10T11:27:56.335 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:56.332 7fd1e333a700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1dc1a9c40 con 0x7fd1dc1a8850 2021-02-10T11:27:56.335 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:56.332 7fd1e333a700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1dc1a9730 con 0x7fd1dc1a6fa0 2021-02-10T11:27:56.335 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:56.332 7fd1e1156700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd1dc0b0ad0 0x7fd1dc091f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:56.335 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:56.332 7fd1e1156700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd1dc0b0ad0 0x7fd1dc091f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:59962/0 (socket says 172.21.2.102:59962) 2021-02-10T11:27:56.335 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:56.332 7fd1e1156700 1 -- 172.21.2.102:0/580405072 learned_addr learned my addr 172.21.2.102:0/580405072 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:27:56.335 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:56.332 7fd1e1156700 1 -- 172.21.2.102:0/580405072 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd1dc1a6fa0 msgr2=0x7fd1dc1a73a0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:27:56.335 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:56.332 7fd1e1156700 1 --2- 172.21.2.102:0/580405072 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd1dc1a6fa0 0x7fd1dc1a73a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:56.335 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:56.332 7fd1e1156700 1 -- 172.21.2.102:0/580405072 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd1dc1a8850 msgr2=0x7fd1dc1a8c50 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:27:56.336 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:56.332 7fd1e1156700 1 --2- 172.21.2.102:0/580405072 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd1dc1a8850 0x7fd1dc1a8c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:56.336 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:56.332 7fd1e1156700 1 -- 172.21.2.102:0/580405072 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1dc1a9270 con 0x7fd1dc0b0ad0 2021-02-10T11:27:56.336 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:56.332 7fd1e1156700 1 --2- 172.21.2.102:0/580405072 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd1dc0b0ad0 0x7fd1dc091f30 secure :-1 s=READY pgs=1810 cs=0 l=1 rev1=1 rx=0x7fd1d0006f70 tx=0x7fd1d000b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:56.336 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:56.332 7fd1cdffb700 1 -- 172.21.2.102:0/580405072 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd1d00064a0 con 0x7fd1dc0b0ad0 2021-02-10T11:27:56.336 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:56.333 7fd1cdffb700 1 -- 172.21.2.102:0/580405072 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1d000bde0 con 0x7fd1dc0b0ad0 2021-02-10T11:27:56.336 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:56.333 7fd1cdffb700 1 -- 172.21.2.102:0/580405072 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd1d00052e0 con 0x7fd1dc0b0ad0 2021-02-10T11:27:56.336 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:56.333 7fd1e333a700 1 -- 172.21.2.102:0/580405072 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1dc1a9270 con 0x7fd1dc0b0ad0 2021-02-10T11:27:56.336 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:56.333 7fd1e333a700 1 -- 172.21.2.102:0/580405072 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1dc1a7cb0 con 0x7fd1dc0b0ad0 2021-02-10T11:27:56.336 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:56.333 7fd1cdffb700 1 -- 172.21.2.102:0/580405072 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd1d0006030 con 0x7fd1dc0b0ad0 2021-02-10T11:27:56.336 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:56.333 7fd1cdffb700 1 --2- 172.21.2.102:0/580405072 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd1bc0186c0 0x7fd1bc01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:56.336 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:56.333 7fd1cdffb700 1 -- 172.21.2.102:0/580405072 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fd1d0032400 con 0x7fd1dc0b0ad0 2021-02-10T11:27:56.336 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:56.333 7fd1e0955700 1 --2- 172.21.2.102:0/580405072 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd1bc0186c0 0x7fd1bc01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:56.336 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:56.334 7fd1e0955700 1 --2- 172.21.2.102:0/580405072 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd1bc0186c0 0x7fd1bc01abc0 crc :-1 s=READY pgs=817 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:56.365 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:56.365 7fd1e333a700 1 -- 172.21.2.102:0/580405072 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1dc1a9710 con 0x7fd1dc0b0ad0 2021-02-10T11:27:56.368 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:56.368 7fd1cdffb700 1 -- 172.21.2.102:0/580405072 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd1d0000940 con 0x7fd1dc0b0ad0 2021-02-10T11:27:56.492 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:56.492 7fd1e333a700 1 -- 172.21.2.102:0/580405072 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd1dc1a9710 con 0x7fd1dc0b0ad0 2021-02-10T11:27:56.492 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:56.493 7fd1cdffb700 1 -- 172.21.2.102:0/580405072 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) v1 ==== 74+0+13996 (secure 0 0 0) 0x7fd1d0006030 con 0x7fd1dc0b0ad0 2021-02-10T11:27:56.493 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:27:56.493 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":85,"fsid":"68db30ea-dbce-4aef-a6df-f01c6fea0513","created":"2021-02-10 11:03:06.331557","modified":"2021-02-10 11:27:42.481451","last_up_change":"2021-02-10 11:03:32.690562","last_in_change":"2021-02-10 11:03:32.690562","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:03:37.322519","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:27:39.452483","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":17,"pool_name":"cephfs_data","create_time":"2021-02-10 11:27:40.475662","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"be691b4d-a039-408a-af80-9d53f02b6f1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6818","nonce":36088},{"type":"v1","addr":"172.21.2.102:6820","nonce":36088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6822","nonce":36088},{"type":"v1","addr":"172.21.2.102:6824","nonce":36088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6830","nonce":36088},{"type":"v1","addr":"172.21.2.102:6832","nonce":36088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6826","nonce":36088},{"type":"v1","addr":"172.21.2.102:6828","nonce":36088}]},"public_addr":"172.21.2.102:6820/36088","cluster_addr":"172.21.2.102:6824/36088","heartbeat_back_addr":"172.21.2.102:6832/36088","heartbeat_front_addr":"172.21.2.102:6828/36088","state":["exists","up"]},{"osd":1,"uuid":"64e3ccc8-9c69-4d7c-865e-af8e83822635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6819","nonce":36087},{"type":"v1","addr":"172.21.2.102:6821","nonce":36087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6823","nonce":36087},{"type":"v1","addr":"172.21.2.102:6825","nonce":36087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6831","nonce":36087},{"type":"v1","addr":"172.21.2.102:6833","nonce":36087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6827","nonce":36087},{"type":"v1","addr":"172.21.2.102:6829","nonce":36087}]},"public_addr":"172.21.2.102:6821/36087","cluster_addr":"172.21.2.102:6825/36087","heartbeat_back_addr":"172.21.2.102:6833/36087","heartbeat_front_addr":"172.21.2.102:6829/36087","state":["exists","up"]},{"osd":2,"uuid":"153700af-747f-481f-bdf4-d6072d3a842f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6810","nonce":36089},{"type":"v1","addr":"172.21.2.102:6811","nonce":36089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6812","nonce":36089},{"type":"v1","addr":"172.21.2.102:6813","nonce":36089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6816","nonce":36089},{"type":"v1","addr":"172.21.2.102:6817","nonce":36089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6814","nonce":36089},{"type":"v1","addr":"172.21.2.102:6815","nonce":36089}]},"public_addr":"172.21.2.102:6811/36089","cluster_addr":"172.21.2.102:6813/36089","heartbeat_back_addr":"172.21.2.102:6817/36089","heartbeat_front_addr":"172.21.2.102:6815/36089","state":["exists","up"]},{"osd":3,"uuid":"cbc678f4-83a9-45e6-8830-2f7c232b1ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6802","nonce":36090},{"type":"v1","addr":"172.21.2.102:6803","nonce":36090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6804","nonce":36090},{"type":"v1","addr":"172.21.2.102:6805","nonce":36090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6808","nonce":36090},{"type":"v1","addr":"172.21.2.102:6809","nonce":36090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6806","nonce":36090},{"type":"v1","addr":"172.21.2.102:6807","nonce":36090}]},"public_addr":"172.21.2.102:6803/36090","cluster_addr":"172.21.2.102:6805/36090","heartbeat_back_addr":"172.21.2.102:6809/36090","heartbeat_front_addr":"172.21.2.102:6807/36090","state":["exists","up"]},{"osd":4,"uuid":"c01e4008-b258-4f10-afd2-9e7f37214351","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6808","nonce":36368},{"type":"v1","addr":"172.21.2.113:6811","nonce":36368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6814","nonce":36368},{"type":"v1","addr":"172.21.2.113:6818","nonce":36368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6826","nonce":36368},{"type":"v1","addr":"172.21.2.113:6828","nonce":36368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6820","nonce":36368},{"type":"v1","addr":"172.21.2.113:6824","nonce":36368}]},"public_addr":"172.21.2.113:6811/36368","cluster_addr":"172.21.2.113:6818/36368","heartbeat_back_addr":"172.21.2.113:6828/36368","heartbeat_front_addr":"172.21.2.113:6824/36368","state":["exists","up"]},{"osd":5,"uuid":"551f6007-daf8-4a98-8ee0-c2b16bcccb5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6802","nonce":36366},{"type":"v1","addr":"172.21.2.113:6805","nonce":36366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6807","nonce":36366},{"type":"v1","addr":"172.21.2.113:6810","nonce":36366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6819","nonce":36366},{"type":"v1","addr":"172.21.2.113:6822","nonce":36366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6812","nonce":36366},{"type":"v1","addr":"172.21.2.113:6815","nonce":36366}]},"public_addr":"172.21.2.113:6805/36366","cluster_addr":"172.21.2.113:6810/36366","heartbeat_back_addr":"172.21.2.113:6822/36366","heartbeat_front_addr":"172.21.2.113:6815/36366","state":["exists","up"]},{"osd":6,"uuid":"bf67c040-0e50-48be-b102-c3943bc9ebef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6800","nonce":36367},{"type":"v1","addr":"172.21.2.113:6801","nonce":36367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6803","nonce":36367},{"type":"v1","addr":"172.21.2.113:6804","nonce":36367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6813","nonce":36367},{"type":"v1","addr":"172.21.2.113:6816","nonce":36367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6806","nonce":36367},{"type":"v1","addr":"172.21.2.113:6809","nonce":36367}]},"public_addr":"172.21.2.113:6801/36367","cluster_addr":"172.21.2.113:6804/36367","heartbeat_back_addr":"172.21.2.113:6816/36367","heartbeat_front_addr":"172.21.2.113:6809/36367","state":["exists","up"]},{"osd":7,"uuid":"f5461f80-018d-428f-be89-2eb2bcbe7bb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6817","nonce":36365},{"type":"v1","addr":"172.21.2.113:6821","nonce":36365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6823","nonce":36365},{"type":"v1","addr":"172.21.2.113:6825","nonce":36365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6830","nonce":36365},{"type":"v1","addr":"172.21.2.113:6831","nonce":36365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6827","nonce":36365},{"type":"v1","addr":"172.21.2.113:6829","nonce":36365}]},"public_addr":"172.21.2.113:6821/36365","cluster_addr":"172.21.2.113:6825/36365","heartbeat_back_addr":"172.21.2.113:6831/36365","heartbeat_front_addr":"172.21.2.113:6829/36365","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:27:56.494 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:56.494 7fd1e333a700 1 -- 172.21.2.102:0/580405072 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd1bc0186c0 msgr2=0x7fd1bc01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:56.494 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:56.494 7fd1e333a700 1 --2- 172.21.2.102:0/580405072 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd1bc0186c0 0x7fd1bc01abc0 crc :-1 s=READY pgs=817 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:56.494 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:56.494 7fd1e333a700 1 -- 172.21.2.102:0/580405072 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd1dc0b0ad0 msgr2=0x7fd1dc091f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:56.494 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:56.494 7fd1e333a700 1 --2- 172.21.2.102:0/580405072 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd1dc0b0ad0 0x7fd1dc091f30 secure :-1 s=READY pgs=1810 cs=0 l=1 rev1=1 rx=0x7fd1d0006f70 tx=0x7fd1d000b960).stop 2021-02-10T11:27:56.495 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:56.494 7fd1e333a700 1 -- 172.21.2.102:0/580405072 shutdown_connections 2021-02-10T11:27:56.495 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:56.494 7fd1e333a700 1 --2- 172.21.2.102:0/580405072 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd1dc1a6fa0 0x7fd1dc1a73a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:56.496 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:56.494 7fd1e333a700 1 --2- 172.21.2.102:0/580405072 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd1bc0186c0 0x7fd1bc01abc0 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:56.496 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:56.494 7fd1e333a700 1 --2- 172.21.2.102:0/580405072 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd1dc1a8850 0x7fd1dc1a8c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:56.496 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:56.494 7fd1e333a700 1 --2- 172.21.2.102:0/580405072 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd1dc0b0ad0 0x7fd1dc091f30 unknown :-1 s=CLOSED pgs=1810 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:56.496 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:56.494 7fd1e333a700 1 -- 172.21.2.102:0/580405072 >> 172.21.2.102:0/580405072 conn(0x7fd1dc19f8f0 msgr2=0x7fd1dc1a5f80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:56.496 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:56.494 7fd1e333a700 1 -- 172.21.2.102:0/580405072 shutdown_connections 2021-02-10T11:27:56.496 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:56.495 7fd1e333a700 1 -- 172.21.2.102:0/580405072 wait complete. 2021-02-10T11:27:56.503 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:03:37.322519', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 16, 'pool_name': 'cephfs_metadata', 'create_time': '2021-02-10 11:27:39.452483', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 17, 'pool_name': 'cephfs_data', 'create_time': '2021-02-10 11:27:40.475662', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-02-10T11:27:56.503 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:27:56.936 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:56.918 7f583784b700 1 Processor -- start 2021-02-10T11:27:56.936 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:56.918 7f583784b700 1 -- start start 2021-02-10T11:27:56.936 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:56.919 7f583784b700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f583010aff0 0x7f58300cf750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:56.936 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:56.919 7f583784b700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58300fe1c0 con 0x7f58300d1c70 2021-02-10T11:27:56.936 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:56.919 7f583784b700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58301b0cb0 con 0x7f58300d1870 2021-02-10T11:27:56.936 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:56.919 7f583784b700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5830109a60 con 0x7f583010aff0 2021-02-10T11:27:56.936 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:56.919 7f5834e66700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f58300d1c70 0x7f58300cef60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:52194/0 (socket says 172.21.2.102:52194) 2021-02-10T11:27:56.937 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:56.919 7f5834e66700 1 -- 172.21.2.102:0/724467012 learned_addr learned my addr 172.21.2.102:0/724467012 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:27:56.937 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:56.919 7f5823fff700 1 -- 172.21.2.102:0/724467012 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3231603477 0 0) 0x7f582c0010b0 con 0x7f58300d1c70 2021-02-10T11:27:56.937 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:56.919 7f5823fff700 1 -- 172.21.2.102:0/724467012 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f581c002fe0 con 0x7f58300d1c70 2021-02-10T11:27:56.937 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:56.919 7f5835e68700 1 --2- 172.21.2.102:0/724467012 >> v2:172.21.2.113:3300/0 conn(0x7f583010aff0 0x7f58300cf750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:56.937 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:56.920 7f5823fff700 1 -- 172.21.2.102:0/724467012 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1951642246 0 0) 0x7f582c0010b0 con 0x7f58300d1c70 2021-02-10T11:27:56.937 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:56.920 7f5823fff700 1 -- 172.21.2.102:0/724467012 >> v2:172.21.2.113:3300/0 conn(0x7f583010aff0 msgr2=0x7f58300cf750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:27:56.937 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:56.920 7f5823fff700 1 --2- 172.21.2.102:0/724467012 >> v2:172.21.2.113:3300/0 conn(0x7f583010aff0 0x7f58300cf750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:56.937 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:56.920 7f5823fff700 1 -- 172.21.2.102:0/724467012 >> v1:172.21.2.113:6789/0 conn(0x7f58300d1870 legacy=0x7f58301b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:56.937 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:56.920 7f5823fff700 1 -- 172.21.2.102:0/724467012 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f583010b3f0 con 0x7f58300d1c70 2021-02-10T11:27:56.937 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:56.920 7f5823fff700 1 -- 172.21.2.102:0/724467012 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7f582c0015b0 con 0x7f58300d1c70 2021-02-10T11:27:56.937 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:56.920 7f5823fff700 1 -- 172.21.2.102:0/724467012 >> v1:172.21.2.102:6789/0 conn(0x7f58300d1c70 legacy=0x7f58300cef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:56.937 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:56.920 7f5823fff700 1 --2- 172.21.2.102:0/724467012 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f581c003a60 0x7f581c005ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:56.938 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:56.920 7f5823fff700 1 --2- 172.21.2.102:0/724467012 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f581c006790 0x7f581c008c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:56.938 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:56.920 7f5823fff700 1 --2- 172.21.2.102:0/724467012 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f581c009590 0x7f581c00bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:56.938 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:56.920 7f5823fff700 1 -- 172.21.2.102:0/724467012 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f581c00c300 con 0x7f581c006790 2021-02-10T11:27:56.938 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:56.920 7f5823fff700 1 -- 172.21.2.102:0/724467012 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f581c00c520 con 0x7f581c009590 2021-02-10T11:27:56.938 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:56.920 7f5823fff700 1 -- 172.21.2.102:0/724467012 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f581c050750 con 0x7f581c003a60 2021-02-10T11:27:56.938 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:56.920 7f5834e66700 1 --2- 172.21.2.102:0/724467012 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f581c006790 0x7f581c008c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:56.938 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:56.920 7f5835e68700 1 --2- 172.21.2.102:0/724467012 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f581c009590 0x7f581c00bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:56.938 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:56.920 7f5835667700 1 --2- 172.21.2.102:0/724467012 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f581c003a60 0x7f581c005ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:56.938 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:56.920 7f5834e66700 1 -- 172.21.2.102:0/724467012 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f581c003a60 msgr2=0x7f581c005ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:27:56.938 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:56.920 7f5834e66700 1 --2- 172.21.2.102:0/724467012 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f581c003a60 0x7f581c005ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:56.938 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:56.920 7f5834e66700 1 -- 172.21.2.102:0/724467012 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f581c009590 msgr2=0x7f581c00bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:27:56.939 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:56.920 7f5834e66700 1 --2- 172.21.2.102:0/724467012 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f581c009590 0x7f581c00bab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:56.939 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:56.920 7f5834e66700 1 -- 172.21.2.102:0/724467012 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f581c0509b0 con 0x7f581c006790 2021-02-10T11:27:56.939 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:56.920 7f5835e68700 1 -- 172.21.2.102:0/724467012 reap_dead start 2021-02-10T11:27:56.939 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:56.921 7f5834e66700 1 --2- 172.21.2.102:0/724467012 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f581c006790 0x7f581c008c90 secure :-1 s=READY pgs=1812 cs=0 l=1 rev1=1 rx=0x7f582c001270 tx=0x7f582c007d70).ready entity=mon.0 client_cookie=7576149c60afcb78 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:56.939 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:56.921 7f5823fff700 1 -- 172.21.2.102:0/724467012 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f582c0009b0 con 0x7f581c006790 2021-02-10T11:27:56.939 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:56.921 7f5823fff700 1 -- 172.21.2.102:0/724467012 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f582c00a520 con 0x7f581c006790 2021-02-10T11:27:56.939 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:56.921 7f5823fff700 1 -- 172.21.2.102:0/724467012 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f582c00aa40 con 0x7f581c006790 2021-02-10T11:27:56.939 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:56.921 7f583784b700 1 -- 172.21.2.102:0/724467012 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f581c006790 msgr2=0x7f581c008c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:56.939 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:56.921 7f583784b700 1 --2- 172.21.2.102:0/724467012 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f581c006790 0x7f581c008c90 secure :-1 s=READY pgs=1812 cs=0 l=1 rev1=1 rx=0x7f582c001270 tx=0x7f582c007d70).stop 2021-02-10T11:27:56.939 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:56.921 7f583784b700 1 -- 172.21.2.102:0/724467012 shutdown_connections 2021-02-10T11:27:56.939 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:56.921 7f583784b700 1 --2- 172.21.2.102:0/724467012 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f581c006790 0x7f581c008c90 unknown :-1 s=CLOSED pgs=1812 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:56.939 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:56.921 7f583784b700 1 -- 172.21.2.102:0/724467012 >> 172.21.2.102:0/724467012 conn(0x7f58300ce420 msgr2=0x7f58300d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:56.939 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:56.922 7f583784b700 1 -- 172.21.2.102:0/724467012 shutdown_connections 2021-02-10T11:27:56.940 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:56.922 7f583784b700 1 -- 172.21.2.102:0/724467012 wait complete. 2021-02-10T11:27:56.941 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:56.936 7f583784b700 1 Processor -- start 2021-02-10T11:27:56.941 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:56.936 7f583784b700 1 -- start start 2021-02-10T11:27:56.941 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:56.937 7f583784b700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5830085bc0 0x7f58300863e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:56.942 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:56.937 7f583784b700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f58300954c0 0x7f5830096300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:56.942 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:56.937 7f583784b700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f58300958c0 0x7f5830096920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:56.942 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:56.937 7f583784b700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5830095d80 con 0x7f5830085bc0 2021-02-10T11:27:56.942 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:56.937 7f583784b700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58300bfa60 con 0x7f58300958c0 2021-02-10T11:27:56.942 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:56.937 7f583784b700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5830085fc0 con 0x7f58300954c0 2021-02-10T11:27:56.942 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:56.937 7f5835667700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5830085bc0 0x7f58300863e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:56.942 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:56.937 7f5835667700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5830085bc0 0x7f58300863e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:59984/0 (socket says 172.21.2.102:59984) 2021-02-10T11:27:56.942 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:56.937 7f5835667700 1 -- 172.21.2.102:0/1000469223 learned_addr learned my addr 172.21.2.102:0/1000469223 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:27:56.942 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:56.937 7f5835e68700 1 --2- 172.21.2.102:0/1000469223 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f58300958c0 0x7f5830096920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:56.942 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:56.937 7f5834e66700 1 --2- 172.21.2.102:0/1000469223 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f58300954c0 0x7f5830096300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:56.942 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:56.937 7f5835667700 1 -- 172.21.2.102:0/1000469223 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f58300954c0 msgr2=0x7f5830096300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:56.943 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:56.937 7f5835667700 1 --2- 172.21.2.102:0/1000469223 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f58300954c0 0x7f5830096300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:56.943 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:56.937 7f5835667700 1 -- 172.21.2.102:0/1000469223 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f58300958c0 msgr2=0x7f5830096920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:56.943 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:56.937 7f5835667700 1 --2- 172.21.2.102:0/1000469223 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f58300958c0 0x7f5830096920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:56.943 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:56.937 7f5835667700 1 -- 172.21.2.102:0/1000469223 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5830086c10 con 0x7f5830085bc0 2021-02-10T11:27:56.943 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:56.937 7f5834e66700 1 --2- 172.21.2.102:0/1000469223 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f58300954c0 0x7f5830096300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:27:56.943 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:56.937 7f5835667700 1 --2- 172.21.2.102:0/1000469223 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5830085bc0 0x7f58300863e0 secure :-1 s=READY pgs=1813 cs=0 l=1 rev1=1 rx=0x7f5824005bf0 tx=0x7f582400a980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:56.943 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:56.937 7f5821ffb700 1 -- 172.21.2.102:0/1000469223 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f58240054b0 con 0x7f5830085bc0 2021-02-10T11:27:56.943 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:56.937 7f5821ffb700 1 -- 172.21.2.102:0/1000469223 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f582400c0d0 con 0x7f5830085bc0 2021-02-10T11:27:56.943 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:56.937 7f583784b700 1 -- 172.21.2.102:0/1000469223 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5830086c10 con 0x7f5830085bc0 2021-02-10T11:27:56.943 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:56.937 7f5821ffb700 1 -- 172.21.2.102:0/1000469223 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f582400c5b0 con 0x7f5830085bc0 2021-02-10T11:27:56.943 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:56.937 7f583784b700 1 -- 172.21.2.102:0/1000469223 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5830086c10 con 0x7f5830085bc0 2021-02-10T11:27:56.943 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:56.938 7f5821ffb700 1 -- 172.21.2.102:0/1000469223 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f582400b000 con 0x7f5830085bc0 2021-02-10T11:27:56.943 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:56.938 7f5821ffb700 1 --2- 172.21.2.102:0/1000469223 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5810018730 0x7f581001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:56.943 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:56.938 7f5821ffb700 1 -- 172.21.2.102:0/1000469223 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f5824032620 con 0x7f5830085bc0 2021-02-10T11:27:56.944 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:56.938 7f5834e66700 1 --2- 172.21.2.102:0/1000469223 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5810018730 0x7f581001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:56.944 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:56.938 7f5834e66700 1 --2- 172.21.2.102:0/1000469223 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5810018730 0x7f581001abc0 crc :-1 s=READY pgs=818 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:56.968 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:56.969 7f583784b700 1 -- 172.21.2.102:0/1000469223 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58300fe1c0 con 0x7f5830085bc0 2021-02-10T11:27:56.972 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:56.971 7f5821ffb700 1 -- 172.21.2.102:0/1000469223 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f582400ad70 con 0x7f5830085bc0 2021-02-10T11:27:57.097 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:57.097 7f583784b700 1 -- 172.21.2.102:0/1000469223 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f58300fe1c0 con 0x7f5830085bc0 2021-02-10T11:27:57.098 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:57.098 7f5821ffb700 1 -- 172.21.2.102:0/1000469223 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v85) v1 ==== 92+0+10 (secure 0 0 0) 0x7f582400ad70 con 0x7f5830085bc0 2021-02-10T11:27:57.098 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:27:57.100 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:57.100 7f583784b700 1 -- 172.21.2.102:0/1000469223 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5810018730 msgr2=0x7f581001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:57.100 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:57.100 7f583784b700 1 --2- 172.21.2.102:0/1000469223 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5810018730 0x7f581001abc0 crc :-1 s=READY pgs=818 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:57.100 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:57.100 7f583784b700 1 -- 172.21.2.102:0/1000469223 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5830085bc0 msgr2=0x7f58300863e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:57.100 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:57.100 7f583784b700 1 --2- 172.21.2.102:0/1000469223 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5830085bc0 0x7f58300863e0 secure :-1 s=READY pgs=1813 cs=0 l=1 rev1=1 rx=0x7f5824005bf0 tx=0x7f582400a980).stop 2021-02-10T11:27:57.101 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:57.100 7f583784b700 1 -- 172.21.2.102:0/1000469223 shutdown_connections 2021-02-10T11:27:57.101 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:57.100 7f583784b700 1 --2- 172.21.2.102:0/1000469223 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5810018730 0x7f581001abc0 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:57.101 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:57.100 7f583784b700 1 --2- 172.21.2.102:0/1000469223 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f58300958c0 0x7f5830096920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:57.101 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:57.100 7f583784b700 1 --2- 172.21.2.102:0/1000469223 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f58300954c0 0x7f5830096300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:57.101 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:57.100 7f583784b700 1 --2- 172.21.2.102:0/1000469223 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5830085bc0 0x7f58300863e0 unknown :-1 s=CLOSED pgs=1813 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:57.101 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:57.100 7f583784b700 1 -- 172.21.2.102:0/1000469223 >> 172.21.2.102:0/1000469223 conn(0x7f58300bef00 msgr2=0x7f583009d1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:57.101 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:57.100 7f583784b700 1 -- 172.21.2.102:0/1000469223 shutdown_connections 2021-02-10T11:27:57.101 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:57.100 7f583784b700 1 -- 172.21.2.102:0/1000469223 wait complete. 2021-02-10T11:27:57.108 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-02-10T11:27:57.540 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:57.524 7f9527c64700 1 Processor -- start 2021-02-10T11:27:57.540 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:57.524 7f9527c64700 1 -- start start 2021-02-10T11:27:57.541 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:57.524 7f9527c64700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f95200d1870 0x7f95201b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:57.541 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:57.524 7f9527c64700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f95201b0ae0 0x7f9520108940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:57.541 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:57.524 7f9527c64700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f95200d1c70 0x7f9520108f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:57.541 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:57.524 7f9527c64700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95200fe1c0 con 0x7f95200d1870 2021-02-10T11:27:57.541 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:57.524 7f9527c64700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9520109a60 con 0x7f95200d1c70 2021-02-10T11:27:57.541 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:57.524 7f9527c64700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9520088070 con 0x7f95201b0ae0 2021-02-10T11:27:57.541 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:57.525 7f9525a80700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f95200d1870 0x7f95201b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:57.541 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:57.525 7f9525a80700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f95200d1870 0x7f95201b04c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:59990/0 (socket says 172.21.2.102:59990) 2021-02-10T11:27:57.541 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:57.525 7f9525a80700 1 -- 172.21.2.102:0/3385977060 learned_addr learned my addr 172.21.2.102:0/3385977060 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:27:57.541 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:57.525 7f9525a80700 1 -- 172.21.2.102:0/3385977060 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f95201b0ae0 msgr2=0x7f9520108940 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:27:57.541 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:57.525 7f9525a80700 1 --2- 172.21.2.102:0/3385977060 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f95201b0ae0 0x7f9520108940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:57.541 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:57.525 7f9525a80700 1 -- 172.21.2.102:0/3385977060 >> v2:172.21.2.113:3300/0 conn(0x7f95200d1c70 msgr2=0x7f9520108f60 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:27:57.541 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:57.525 7f9525a80700 1 --2- 172.21.2.102:0/3385977060 >> v2:172.21.2.113:3300/0 conn(0x7f95200d1c70 0x7f9520108f60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:57.542 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:57.525 7f9525a80700 1 -- 172.21.2.102:0/3385977060 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9520109580 con 0x7f95200d1870 2021-02-10T11:27:57.542 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:57.525 7f9525a80700 1 --2- 172.21.2.102:0/3385977060 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f95200d1870 0x7f95201b04c0 secure :-1 s=READY pgs=1814 cs=0 l=1 rev1=1 rx=0x7f9514006700 tx=0x7f9514009b00).ready entity=mon.0 client_cookie=15d68333179b8 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:57.542 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:57.525 7f9524a7e700 1 -- 172.21.2.102:0/3385977060 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f951400c0c0 con 0x7f95200d1870 2021-02-10T11:27:57.542 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:57.525 7f9524a7e700 1 -- 172.21.2.102:0/3385977060 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f951400c340 con 0x7f95200d1870 2021-02-10T11:27:57.542 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:57.525 7f9524a7e700 1 -- 172.21.2.102:0/3385977060 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f951400c820 con 0x7f95200d1870 2021-02-10T11:27:57.542 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:57.525 7f9527c64700 1 -- 172.21.2.102:0/3385977060 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f95200d1870 msgr2=0x7f95201b04c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:57.542 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:57.525 7f9527c64700 1 --2- 172.21.2.102:0/3385977060 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f95200d1870 0x7f95201b04c0 secure :-1 s=READY pgs=1814 cs=0 l=1 rev1=1 rx=0x7f9514006700 tx=0x7f9514009b00).stop 2021-02-10T11:27:57.542 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:57.527 7f9527c64700 1 -- 172.21.2.102:0/3385977060 shutdown_connections 2021-02-10T11:27:57.542 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:57.527 7f9527c64700 1 --2- 172.21.2.102:0/3385977060 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f95201b0ae0 0x7f9520108940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:57.542 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:57.527 7f9527c64700 1 --2- 172.21.2.102:0/3385977060 >> v2:172.21.2.113:3300/0 conn(0x7f95200d1c70 0x7f9520108f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:57.542 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:57.527 7f9527c64700 1 --2- 172.21.2.102:0/3385977060 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f95200d1870 0x7f95201b04c0 unknown :-1 s=CLOSED pgs=1814 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:57.542 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:57.527 7f9527c64700 1 -- 172.21.2.102:0/3385977060 >> 172.21.2.102:0/3385977060 conn(0x7f95200ce420 msgr2=0x7f95200d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:57.543 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:57.527 7f9527c64700 1 -- 172.21.2.102:0/3385977060 shutdown_connections 2021-02-10T11:27:57.543 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:57.527 7f9527c64700 1 -- 172.21.2.102:0/3385977060 wait complete. 2021-02-10T11:27:57.544 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:57.541 7f9527c64700 1 Processor -- start 2021-02-10T11:27:57.544 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:57.541 7f9527c64700 1 -- start start 2021-02-10T11:27:57.544 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:57.541 7f9527c64700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f95200c2370 0x7f952018f870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:57.544 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:57.541 7f9527c64700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f9520084cb0 0x7f95201a7470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:57.544 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:57.541 7f9527c64700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f95201a7a90 0x7f9520085700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:57.544 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:57.541 7f9527c64700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95200be840 con 0x7f95201a7a90 2021-02-10T11:27:57.544 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:57.541 7f9527c64700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95200c29a0 con 0x7f9520084cb0 2021-02-10T11:27:57.544 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:57.541 7f9527c64700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95201a7e90 con 0x7f95200c2370 2021-02-10T11:27:57.544 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:57.541 7f9526281700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f95201a7a90 0x7f9520085700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:57.545 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:57.541 7f9526281700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f95201a7a90 0x7f9520085700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:59998/0 (socket says 172.21.2.102:59998) 2021-02-10T11:27:57.545 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:57.541 7f9526281700 1 -- 172.21.2.102:0/1726516507 learned_addr learned my addr 172.21.2.102:0/1726516507 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:27:57.545 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:57.542 7f9526281700 1 -- 172.21.2.102:0/1726516507 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f95200c2370 msgr2=0x7f952018f870 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:27:57.545 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:57.542 7f9526281700 1 --2- 172.21.2.102:0/1726516507 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f95200c2370 0x7f952018f870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:57.545 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:57.542 7f9526281700 1 -- 172.21.2.102:0/1726516507 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f9520084cb0 msgr2=0x7f95201a7470 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:27:57.545 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:57.542 7f9526281700 1 --2- 172.21.2.102:0/1726516507 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f9520084cb0 0x7f95201a7470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:57.545 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:57.542 7f9526281700 1 -- 172.21.2.102:0/1726516507 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95201a80b0 con 0x7f95201a7a90 2021-02-10T11:27:57.545 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:57.542 7f9526281700 1 --2- 172.21.2.102:0/1726516507 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f95201a7a90 0x7f9520085700 secure :-1 s=READY pgs=1815 cs=0 l=1 rev1=1 rx=0x7f9518009960 tx=0x7f951800bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:57.545 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:57.542 7f95127fc700 1 -- 172.21.2.102:0/1726516507 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9518003030 con 0x7f95201a7a90 2021-02-10T11:27:57.545 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:57.542 7f95127fc700 1 -- 172.21.2.102:0/1726516507 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9518001310 con 0x7f95201a7a90 2021-02-10T11:27:57.545 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:57.542 7f95127fc700 1 -- 172.21.2.102:0/1726516507 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9518001830 con 0x7f95201a7a90 2021-02-10T11:27:57.545 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:57.542 7f9527c64700 1 -- 172.21.2.102:0/1726516507 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95201a80b0 con 0x7f95201a7a90 2021-02-10T11:27:57.545 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:57.542 7f9527c64700 1 -- 172.21.2.102:0/1726516507 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95201a80b0 con 0x7f95201a7a90 2021-02-10T11:27:57.546 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:57.543 7f95127fc700 1 -- 172.21.2.102:0/1726516507 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f9518000d10 con 0x7f95201a7a90 2021-02-10T11:27:57.546 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:57.543 7f95127fc700 1 --2- 172.21.2.102:0/1726516507 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f95080186c0 0x7f950801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:57.546 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:57.543 7f95127fc700 1 -- 172.21.2.102:0/1726516507 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f95180328e0 con 0x7f95201a7a90 2021-02-10T11:27:57.546 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:57.543 7f9525a80700 1 --2- 172.21.2.102:0/1726516507 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f95080186c0 0x7f950801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:57.546 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:57.543 7f9525a80700 1 --2- 172.21.2.102:0/1726516507 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f95080186c0 0x7f950801abc0 crc :-1 s=READY pgs=819 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:57.574 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:57.574 7f9527c64700 1 -- 172.21.2.102:0/1726516507 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9520082a70 con 0x7f95201a7a90 2021-02-10T11:27:57.577 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:57.577 7f95127fc700 1 -- 172.21.2.102:0/1726516507 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f951800be30 con 0x7f95201a7a90 2021-02-10T11:27:57.703 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:57.703 7f9527c64700 1 -- 172.21.2.102:0/1726516507 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7f9520082a70 con 0x7f95201a7a90 2021-02-10T11:27:57.704 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:57.703 7f95127fc700 1 -- 172.21.2.102:0/1726516507 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v85) v1 ==== 104+0+10 (secure 0 0 0) 0x7f9518000d10 con 0x7f95201a7a90 2021-02-10T11:27:57.704 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:27:57.705 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:57.705 7f9527c64700 1 -- 172.21.2.102:0/1726516507 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f95080186c0 msgr2=0x7f950801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:57.705 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:57.705 7f9527c64700 1 --2- 172.21.2.102:0/1726516507 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f95080186c0 0x7f950801abc0 crc :-1 s=READY pgs=819 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:57.706 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:57.705 7f9527c64700 1 -- 172.21.2.102:0/1726516507 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f95201a7a90 msgr2=0x7f9520085700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:57.706 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:57.705 7f9527c64700 1 --2- 172.21.2.102:0/1726516507 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f95201a7a90 0x7f9520085700 secure :-1 s=READY pgs=1815 cs=0 l=1 rev1=1 rx=0x7f9518009960 tx=0x7f951800bcf0).stop 2021-02-10T11:27:57.706 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:57.706 7f9527c64700 1 -- 172.21.2.102:0/1726516507 shutdown_connections 2021-02-10T11:27:57.706 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:57.706 7f9527c64700 1 --2- 172.21.2.102:0/1726516507 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f95201a7a90 0x7f9520085700 unknown :-1 s=CLOSED pgs=1815 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:57.707 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:57.706 7f9527c64700 1 --2- 172.21.2.102:0/1726516507 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f95080186c0 0x7f950801abc0 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:57.707 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:57.706 7f9527c64700 1 --2- 172.21.2.102:0/1726516507 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f9520084cb0 0x7f95201a7470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:57.707 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:57.706 7f9527c64700 1 --2- 172.21.2.102:0/1726516507 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f95200c2370 0x7f952018f870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:57.707 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:57.706 7f9527c64700 1 -- 172.21.2.102:0/1726516507 >> 172.21.2.102:0/1726516507 conn(0x7f95200b5dc0 msgr2=0x7f95200c2d90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:57.707 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:57.706 7f9527c64700 1 -- 172.21.2.102:0/1726516507 shutdown_connections 2021-02-10T11:27:57.707 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:57.706 7f9527c64700 1 -- 172.21.2.102:0/1726516507 wait complete. 2021-02-10T11:27:57.717 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-02-10T11:27:58.165 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:58.148 7f924e6be700 1 Processor -- start 2021-02-10T11:27:58.166 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:58.148 7f924e6be700 1 -- start start 2021-02-10T11:27:58.166 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:58.148 7f924e6be700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f92480d1c60 0x7f92480cef50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:58.166 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:58.148 7f924e6be700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92480d0160 con 0x7f92480d1860 2021-02-10T11:27:58.166 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:58.148 7f924e6be700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92480fe1b0 con 0x7f92480cf570 2021-02-10T11:27:58.166 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:58.148 7f924e6be700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92481b0ca0 con 0x7f92480d1c60 2021-02-10T11:27:58.166 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:58.148 7f9247fff700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f92480d1860 0x7f92481b04b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:52226/0 (socket says 172.21.2.102:52226) 2021-02-10T11:27:58.166 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:58.148 7f9247fff700 1 -- 172.21.2.102:0/2696420535 learned_addr learned my addr 172.21.2.102:0/2696420535 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:27:58.166 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:58.149 7f92477fe700 1 --2- 172.21.2.102:0/2696420535 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f92480d1c60 0x7f92480cef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:58.166 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:58.149 7f9246ffd700 1 -- 172.21.2.102:0/2696420535 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 345813186 0 0) 0x7f9238001180 con 0x7f92480d1860 2021-02-10T11:27:58.166 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:58.149 7f9246ffd700 1 -- 172.21.2.102:0/2696420535 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9230002fe0 con 0x7f92480d1860 2021-02-10T11:27:58.166 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:58.149 7f9246ffd700 1 -- 172.21.2.102:0/2696420535 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 733207630 0 0) 0x7f9238001180 con 0x7f92480d1860 2021-02-10T11:27:58.166 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:58.149 7f9246ffd700 1 -- 172.21.2.102:0/2696420535 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f92480d1c60 msgr2=0x7f92480cef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:27:58.167 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:58.149 7f9246ffd700 1 --2- 172.21.2.102:0/2696420535 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f92480d1c60 0x7f92480cef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:58.167 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:58.149 7f9246ffd700 1 -- 172.21.2.102:0/2696420535 >> v1:172.21.2.113:6789/0 conn(0x7f92480cf570 legacy=0x7f92480cf970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:58.167 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:58.149 7f9246ffd700 1 -- 172.21.2.102:0/2696420535 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9248109a50 con 0x7f92480d1860 2021-02-10T11:27:58.167 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:58.149 7f9246ffd700 1 -- 172.21.2.102:0/2696420535 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7f9238001680 con 0x7f92480d1860 2021-02-10T11:27:58.167 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:58.149 7f92477fe700 1 --2- 172.21.2.102:0/2696420535 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f92480d1c60 0x7f92480cef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:27:58.167 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:58.149 7f9246ffd700 1 -- 172.21.2.102:0/2696420535 >> v1:172.21.2.102:6789/0 conn(0x7f92480d1860 legacy=0x7f92481b04b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:58.167 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:58.149 7f9246ffd700 1 --2- 172.21.2.102:0/2696420535 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9230003a60 0x7f9230005f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:58.167 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:58.149 7f9246ffd700 1 --2- 172.21.2.102:0/2696420535 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f9230006800 0x7f9230008d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:58.167 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:58.149 7f9246ffd700 1 --2- 172.21.2.102:0/2696420535 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9230009620 0x7f923000bb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:58.167 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:58.149 7f9246ffd700 1 -- 172.21.2.102:0/2696420535 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f923000c2e0 con 0x7f9230003a60 2021-02-10T11:27:58.167 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:58.149 7f9246ffd700 1 -- 172.21.2.102:0/2696420535 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f923000c500 con 0x7f9230006800 2021-02-10T11:27:58.167 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:58.149 7f9246ffd700 1 -- 172.21.2.102:0/2696420535 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92300506c0 con 0x7f9230009620 2021-02-10T11:27:58.167 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:58.149 7f9247fff700 1 --2- 172.21.2.102:0/2696420535 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9230003a60 0x7f9230005f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:58.168 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:58.149 7f924ccdb700 1 --2- 172.21.2.102:0/2696420535 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9230009620 0x7f923000bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:58.168 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:58.150 7f92477fe700 1 --2- 172.21.2.102:0/2696420535 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f9230006800 0x7f9230008d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:58.168 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:58.150 7f9247fff700 1 -- 172.21.2.102:0/2696420535 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9230009620 msgr2=0x7f923000bb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:27:58.168 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:58.150 7f9247fff700 1 --2- 172.21.2.102:0/2696420535 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9230009620 0x7f923000bb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:58.168 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:58.150 7f9247fff700 1 -- 172.21.2.102:0/2696420535 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f9230006800 msgr2=0x7f9230008d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:27:58.168 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:58.150 7f9247fff700 1 --2- 172.21.2.102:0/2696420535 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f9230006800 0x7f9230008d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:58.168 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:58.150 7f9247fff700 1 -- 172.21.2.102:0/2696420535 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9230050960 con 0x7f9230003a60 2021-02-10T11:27:58.168 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:58.150 7f9247fff700 1 --2- 172.21.2.102:0/2696420535 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9230003a60 0x7f9230005f60 secure :-1 s=READY pgs=1817 cs=0 l=1 rev1=1 rx=0x7f9238000b60 tx=0x7f9238007d20).ready entity=mon.0 client_cookie=9d9710a95b33fab0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:58.168 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:58.150 7f9246ffd700 1 -- 172.21.2.102:0/2696420535 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f92380070c0 con 0x7f9230003a60 2021-02-10T11:27:58.168 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:58.150 7f9246ffd700 1 -- 172.21.2.102:0/2696420535 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92380043a0 con 0x7f9230003a60 2021-02-10T11:27:58.168 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:58.150 7f9246ffd700 1 -- 172.21.2.102:0/2696420535 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f92380048c0 con 0x7f9230003a60 2021-02-10T11:27:58.168 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:58.150 7f924e6be700 1 -- 172.21.2.102:0/2696420535 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9230003a60 msgr2=0x7f9230005f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:58.168 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:58.150 7f924e6be700 1 --2- 172.21.2.102:0/2696420535 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9230003a60 0x7f9230005f60 secure :-1 s=READY pgs=1817 cs=0 l=1 rev1=1 rx=0x7f9238000b60 tx=0x7f9238007d20).stop 2021-02-10T11:27:58.168 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:58.150 7f924ccdb700 1 -- 172.21.2.102:0/2696420535 reap_dead start 2021-02-10T11:27:58.169 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:58.151 7f924e6be700 1 -- 172.21.2.102:0/2696420535 shutdown_connections 2021-02-10T11:27:58.169 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:58.151 7f924e6be700 1 -- 172.21.2.102:0/2696420535 >> 172.21.2.102:0/2696420535 conn(0x7f92480ce410 msgr2=0x7f92480d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:58.169 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:58.151 7f924e6be700 1 -- 172.21.2.102:0/2696420535 shutdown_connections 2021-02-10T11:27:58.169 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:58.151 7f924e6be700 1 -- 172.21.2.102:0/2696420535 wait complete. 2021-02-10T11:27:58.169 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:58.165 7f924e6be700 1 Processor -- start 2021-02-10T11:27:58.169 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:58.165 7f924e6be700 1 -- start start 2021-02-10T11:27:58.171 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:58.166 7f924e6be700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f92480b8100 0x7f9248083440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:58.171 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:58.166 7f924e6be700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f92480b8500 0x7f9248083a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:58.171 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:58.166 7f924e6be700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f92480846f0 0x7f9248084af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:58.171 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:58.166 7f924e6be700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92481a1720 con 0x7f92480846f0 2021-02-10T11:27:58.171 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:58.166 7f924e6be700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9248189e70 con 0x7f92480b8100 2021-02-10T11:27:58.171 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:58.166 7f924e6be700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9248084170 con 0x7f92480b8500 2021-02-10T11:27:58.171 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:58.166 7f924ccdb700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f92480846f0 0x7f9248084af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:58.172 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:58.166 7f924ccdb700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f92480846f0 0x7f9248084af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:60018/0 (socket says 172.21.2.102:60018) 2021-02-10T11:27:58.172 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:58.166 7f924ccdb700 1 -- 172.21.2.102:0/3839483945 learned_addr learned my addr 172.21.2.102:0/3839483945 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:27:58.172 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:58.166 7f92477fe700 1 --2- 172.21.2.102:0/3839483945 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f92480b8500 0x7f9248083a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:58.172 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:58.166 7f924ccdb700 1 -- 172.21.2.102:0/3839483945 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f92480b8500 msgr2=0x7f9248083a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:58.172 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:58.166 7f924ccdb700 1 --2- 172.21.2.102:0/3839483945 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f92480b8500 0x7f9248083a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:58.172 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:58.166 7f924ccdb700 1 -- 172.21.2.102:0/3839483945 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f92480b8100 msgr2=0x7f9248083440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:58.172 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:58.166 7f924ccdb700 1 --2- 172.21.2.102:0/3839483945 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f92480b8100 0x7f9248083440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:58.172 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:58.166 7f924ccdb700 1 -- 172.21.2.102:0/3839483945 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9248085110 con 0x7f92480846f0 2021-02-10T11:27:58.172 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:58.167 7f924ccdb700 1 --2- 172.21.2.102:0/3839483945 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f92480846f0 0x7f9248084af0 secure :-1 s=READY pgs=1818 cs=0 l=1 rev1=1 rx=0x7f924000a8f0 tx=0x7f924000ac90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:58.172 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:58.167 7f9244ff9700 1 -- 172.21.2.102:0/3839483945 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f924000c300 con 0x7f92480846f0 2021-02-10T11:27:58.172 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:58.167 7f9244ff9700 1 -- 172.21.2.102:0/3839483945 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f924000c640 con 0x7f92480846f0 2021-02-10T11:27:58.172 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:58.167 7f9244ff9700 1 -- 172.21.2.102:0/3839483945 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f924000caa0 con 0x7f92480846f0 2021-02-10T11:27:58.172 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:58.167 7f924e6be700 1 -- 172.21.2.102:0/3839483945 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9248085110 con 0x7f92480846f0 2021-02-10T11:27:58.173 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:58.167 7f924e6be700 1 -- 172.21.2.102:0/3839483945 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9248085660 con 0x7f92480846f0 2021-02-10T11:27:58.173 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:58.167 7f9244ff9700 1 -- 172.21.2.102:0/3839483945 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f924000c300 con 0x7f92480846f0 2021-02-10T11:27:58.173 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:58.167 7f9244ff9700 1 --2- 172.21.2.102:0/3839483945 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f92280186c0 0x7f922801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:58.173 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:58.167 7f9244ff9700 1 -- 172.21.2.102:0/3839483945 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f92400327b0 con 0x7f92480846f0 2021-02-10T11:27:58.173 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:58.168 7f9247fff700 1 --2- 172.21.2.102:0/3839483945 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f92280186c0 0x7f922801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:58.173 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:58.168 7f9247fff700 1 --2- 172.21.2.102:0/3839483945 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f92280186c0 0x7f922801abc0 crc :-1 s=READY pgs=820 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:58.198 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:58.198 7f924e6be700 1 -- 172.21.2.102:0/3839483945 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92481a1720 con 0x7f92480846f0 2021-02-10T11:27:58.201 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:58.202 7f9244ff9700 1 -- 172.21.2.102:0/3839483945 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f924002f600 con 0x7f92480846f0 2021-02-10T11:27:58.330 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:58.330 7f924e6be700 1 -- 172.21.2.102:0/3839483945 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7f9248189e70 con 0x7f92480846f0 2021-02-10T11:27:58.330 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:58.330 7f9244ff9700 1 -- 172.21.2.102:0/3839483945 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v85) v1 ==== 100+0+10 (secure 0 0 0) 0x7f924002f600 con 0x7f92480846f0 2021-02-10T11:27:58.331 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:27:58.332 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:58.333 7f924e6be700 1 -- 172.21.2.102:0/3839483945 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f92280186c0 msgr2=0x7f922801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:58.332 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:58.333 7f924e6be700 1 --2- 172.21.2.102:0/3839483945 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f92280186c0 0x7f922801abc0 crc :-1 s=READY pgs=820 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:58.333 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:58.333 7f924e6be700 1 -- 172.21.2.102:0/3839483945 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f92480846f0 msgr2=0x7f9248084af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:58.333 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:58.333 7f924e6be700 1 --2- 172.21.2.102:0/3839483945 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f92480846f0 0x7f9248084af0 secure :-1 s=READY pgs=1818 cs=0 l=1 rev1=1 rx=0x7f924000a8f0 tx=0x7f924000ac90).stop 2021-02-10T11:27:58.333 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:58.333 7f924e6be700 1 -- 172.21.2.102:0/3839483945 shutdown_connections 2021-02-10T11:27:58.333 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:58.333 7f924e6be700 1 --2- 172.21.2.102:0/3839483945 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f92480846f0 0x7f9248084af0 unknown :-1 s=CLOSED pgs=1818 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:58.333 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:58.333 7f924e6be700 1 --2- 172.21.2.102:0/3839483945 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f92480b8500 0x7f9248083a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:58.333 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:58.333 7f924e6be700 1 --2- 172.21.2.102:0/3839483945 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f92280186c0 0x7f922801abc0 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:58.333 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:58.333 7f924e6be700 1 --2- 172.21.2.102:0/3839483945 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f92480b8100 0x7f9248083440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:58.333 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:58.333 7f924e6be700 1 -- 172.21.2.102:0/3839483945 >> 172.21.2.102:0/3839483945 conn(0x7f92480bee90 msgr2=0x7f924809f200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:58.333 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:58.333 7f924e6be700 1 -- 172.21.2.102:0/3839483945 shutdown_connections 2021-02-10T11:27:58.333 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:58.333 7f924e6be700 1 -- 172.21.2.102:0/3839483945 wait complete. 2021-02-10T11:27:58.341 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-02-10T11:27:58.341 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:27:58.770 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:58.753 7f67d13fc700 1 Processor -- start 2021-02-10T11:27:58.770 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:58.754 7f67d13fc700 1 -- start start 2021-02-10T11:27:58.770 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:58.754 7f67d13fc700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f67cc0d1870 0x7f67cc1b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:58.770 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:58.754 7f67d13fc700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f67cc1b0a10 0x7f67cc0cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:58.770 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:58.754 7f67d13fc700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f67cc0d1c70 0x7f67cc0cf950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:58.770 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:58.754 7f67d13fc700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67cc0fe200 con 0x7f67cc0d1c70 2021-02-10T11:27:58.770 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:58.754 7f67d13fc700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67cc0e5f10 con 0x7f67cc0d1870 2021-02-10T11:27:58.770 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:58.754 7f67d13fc700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67cc0f0b50 con 0x7f67cc1b0a10 2021-02-10T11:27:58.770 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:58.754 7f67cb87e700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f67cc0d1c70 0x7f67cc0cf950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:58.771 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:58.754 7f67cb87e700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f67cc0d1c70 0x7f67cc0cf950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:60022/0 (socket says 172.21.2.102:60022) 2021-02-10T11:27:58.771 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:58.754 7f67cb87e700 1 -- 172.21.2.102:0/3782197659 learned_addr learned my addr 172.21.2.102:0/3782197659 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:27:58.771 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:58.754 7f67cb87e700 1 -- 172.21.2.102:0/3782197659 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f67cc1b0a10 msgr2=0x7f67cc0cf330 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:27:58.771 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:58.754 7f67cb87e700 1 --2- 172.21.2.102:0/3782197659 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f67cc1b0a10 0x7f67cc0cf330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:58.771 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:58.755 7f67cb87e700 1 -- 172.21.2.102:0/3782197659 >> v2:172.21.2.113:3300/0 conn(0x7f67cc0d1870 msgr2=0x7f67cc1b03f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:27:58.771 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:58.755 7f67cb87e700 1 --2- 172.21.2.102:0/3782197659 >> v2:172.21.2.113:3300/0 conn(0x7f67cc0d1870 0x7f67cc1b03f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:58.771 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:58.755 7f67cb87e700 1 -- 172.21.2.102:0/3782197659 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67cc0cff70 con 0x7f67cc0d1c70 2021-02-10T11:27:58.771 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:58.755 7f67cb87e700 1 --2- 172.21.2.102:0/3782197659 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f67cc0d1c70 0x7f67cc0cf950 secure :-1 s=READY pgs=1819 cs=0 l=1 rev1=1 rx=0x7f67c4006f00 tx=0x7f67c4009f10).ready entity=mon.0 client_cookie=911f30c368fa836a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:58.771 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:58.755 7f67ca87c700 1 -- 172.21.2.102:0/3782197659 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f67c400b3b0 con 0x7f67cc0d1c70 2021-02-10T11:27:58.771 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:58.755 7f67ca87c700 1 -- 172.21.2.102:0/3782197659 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67c400b630 con 0x7f67cc0d1c70 2021-02-10T11:27:58.771 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:58.755 7f67ca87c700 1 -- 172.21.2.102:0/3782197659 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f67c400bb10 con 0x7f67cc0d1c70 2021-02-10T11:27:58.771 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:58.755 7f67d13fc700 1 -- 172.21.2.102:0/3782197659 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f67cc0d1c70 msgr2=0x7f67cc0cf950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:58.771 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:58.755 7f67d13fc700 1 --2- 172.21.2.102:0/3782197659 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f67cc0d1c70 0x7f67cc0cf950 secure :-1 s=READY pgs=1819 cs=0 l=1 rev1=1 rx=0x7f67c4006f00 tx=0x7f67c4009f10).stop 2021-02-10T11:27:58.772 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:58.756 7f67d13fc700 1 -- 172.21.2.102:0/3782197659 shutdown_connections 2021-02-10T11:27:58.772 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:58.756 7f67d13fc700 1 --2- 172.21.2.102:0/3782197659 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f67cc1b0a10 0x7f67cc0cf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:58.772 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:58.756 7f67d13fc700 1 --2- 172.21.2.102:0/3782197659 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f67cc0d1c70 0x7f67cc0cf950 unknown :-1 s=CLOSED pgs=1819 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:58.772 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:58.756 7f67d13fc700 1 --2- 172.21.2.102:0/3782197659 >> v2:172.21.2.113:3300/0 conn(0x7f67cc0d1870 0x7f67cc1b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:58.772 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:58.756 7f67d13fc700 1 -- 172.21.2.102:0/3782197659 >> 172.21.2.102:0/3782197659 conn(0x7f67cc0ce450 msgr2=0x7f67cc0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:58.772 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:58.756 7f67d13fc700 1 -- 172.21.2.102:0/3782197659 shutdown_connections 2021-02-10T11:27:58.772 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:58.756 7f67d13fc700 1 -- 172.21.2.102:0/3782197659 wait complete. 2021-02-10T11:27:58.774 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:58.770 7f67d13fc700 1 Processor -- start 2021-02-10T11:27:58.774 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:58.770 7f67d13fc700 1 -- start start 2021-02-10T11:27:58.774 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:58.771 7f67d13fc700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f67cc0ada20 0x7f67cc0ade20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:58.774 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:58.771 7f67d13fc700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f67cc1b7040 0x7f67cc1b7440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:58.774 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:58.771 7f67d13fc700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f67cc1b7a60 0x7f67cc1b7e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:58.774 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:58.771 7f67d13fc700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67cc1b6b40 con 0x7f67cc0ada20 2021-02-10T11:27:58.774 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:58.771 7f67d13fc700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67cc1acf70 con 0x7f67cc1b7040 2021-02-10T11:27:58.774 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:58.771 7f67d13fc700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67cc1b5200 con 0x7f67cc1b7a60 2021-02-10T11:27:58.774 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:58.771 7f67cb07d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f67cc0ada20 0x7f67cc0ade20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:58.774 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:58.771 7f67cb07d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f67cc0ada20 0x7f67cc0ade20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:60028/0 (socket says 172.21.2.102:60028) 2021-02-10T11:27:58.775 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:58.771 7f67cb07d700 1 -- 172.21.2.102:0/1528093810 learned_addr learned my addr 172.21.2.102:0/1528093810 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:27:58.775 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:58.771 7f67cb07d700 1 -- 172.21.2.102:0/1528093810 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f67cc1b7a60 msgr2=0x7f67cc1b7e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:58.775 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:58.771 7f67cb07d700 1 --2- 172.21.2.102:0/1528093810 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f67cc1b7a60 0x7f67cc1b7e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:58.775 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:58.771 7f67cb07d700 1 -- 172.21.2.102:0/1528093810 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f67cc1b7040 msgr2=0x7f67cc1b7440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:58.775 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:58.771 7f67cb07d700 1 --2- 172.21.2.102:0/1528093810 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f67cc1b7040 0x7f67cc1b7440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:58.775 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:58.771 7f67cb07d700 1 -- 172.21.2.102:0/1528093810 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67cc1b5420 con 0x7f67cc0ada20 2021-02-10T11:27:58.775 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:58.771 7f67cb07d700 1 --2- 172.21.2.102:0/1528093810 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f67cc0ada20 0x7f67cc0ade20 secure :-1 s=READY pgs=1820 cs=0 l=1 rev1=1 rx=0x7f67bc00b5c0 tx=0x7f67bc00b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:58.775 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:58.773 7f67c8878700 1 -- 172.21.2.102:0/1528093810 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f67bc00bd50 con 0x7f67cc0ada20 2021-02-10T11:27:58.775 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:58.773 7f67c8878700 1 -- 172.21.2.102:0/1528093810 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67bc006030 con 0x7f67cc0ada20 2021-02-10T11:27:58.775 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:58.773 7f67c8878700 1 -- 172.21.2.102:0/1528093810 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f67bc0054d0 con 0x7f67cc0ada20 2021-02-10T11:27:58.775 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:58.773 7f67d13fc700 1 -- 172.21.2.102:0/1528093810 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67cc1b5420 con 0x7f67cc0ada20 2021-02-10T11:27:58.775 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:58.773 7f67d13fc700 1 -- 172.21.2.102:0/1528093810 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67cc1b59b0 con 0x7f67cc0ada20 2021-02-10T11:27:58.776 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:58.774 7f67c8878700 1 -- 172.21.2.102:0/1528093810 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f67bc006210 con 0x7f67cc0ada20 2021-02-10T11:27:58.776 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:58.774 7f67c8878700 1 --2- 172.21.2.102:0/1528093810 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f67b00186c0 0x7f67b001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:58.776 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:58.774 7f67c8878700 1 -- 172.21.2.102:0/1528093810 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f67bc032400 con 0x7f67cc0ada20 2021-02-10T11:27:58.776 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:58.774 7f67c287c700 1 --2- 172.21.2.102:0/1528093810 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f67b00186c0 0x7f67b001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:58.776 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:58.774 7f67c287c700 1 --2- 172.21.2.102:0/1528093810 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f67b00186c0 0x7f67b001abc0 crc :-1 s=READY pgs=821 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:58.804 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:58.804 7f67d13fc700 1 -- 172.21.2.102:0/1528093810 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67cc0ee060 con 0x7f67cc0ada20 2021-02-10T11:27:58.806 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:58.807 7f67c8878700 1 -- 172.21.2.102:0/1528093810 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f67bc000960 con 0x7f67cc0ada20 2021-02-10T11:27:58.935 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:58.935 7f67d13fc700 1 -- 172.21.2.102:0/1528093810 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f67cc0ce450 con 0x7f67cc0ada20 2021-02-10T11:27:58.935 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:58.935 7f67c8878700 1 -- 172.21.2.102:0/1528093810 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 65 v65) v1 ==== 94+0+2101 (secure 0 0 0) 0x7f67bc00bd50 con 0x7f67cc0ada20 2021-02-10T11:27:58.936 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:27:58.936 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":65,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6110,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.102:6835/1471871278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6834","nonce":1471871278},{"type":"v1","addr":"172.21.2.102:6835","nonce":1471871278}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":61}],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:27:41.479375","modified":"2021-02-10 11:27:43.490605","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6107},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6107":{"gid":6107,"name":"b","rank":0,"incarnation":62,"state":"up:active","state_seq":3,"addr":"172.21.2.113:6833/4220238062","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6832","nonce":4220238062},{"type":"v1","addr":"172.21.2.113:6833","nonce":4220238062}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":8}]} 2021-02-10T11:27:58.936 INFO:teuthology.orchestra.run.gibba002.stderr:dumped fsmap epoch 65 2021-02-10T11:27:58.937 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:58.937 7f67d13fc700 1 -- 172.21.2.102:0/1528093810 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f67b00186c0 msgr2=0x7f67b001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:58.937 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:58.937 7f67d13fc700 1 --2- 172.21.2.102:0/1528093810 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f67b00186c0 0x7f67b001abc0 crc :-1 s=READY pgs=821 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:58.937 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:58.937 7f67d13fc700 1 -- 172.21.2.102:0/1528093810 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f67cc0ada20 msgr2=0x7f67cc0ade20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:58.937 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:58.937 7f67d13fc700 1 --2- 172.21.2.102:0/1528093810 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f67cc0ada20 0x7f67cc0ade20 secure :-1 s=READY pgs=1820 cs=0 l=1 rev1=1 rx=0x7f67bc00b5c0 tx=0x7f67bc00b960).stop 2021-02-10T11:27:58.938 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:58.937 7f67d13fc700 1 -- 172.21.2.102:0/1528093810 shutdown_connections 2021-02-10T11:27:58.938 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:58.937 7f67d13fc700 1 --2- 172.21.2.102:0/1528093810 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f67cc1b7a60 0x7f67cc1b7e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:58.938 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:58.937 7f67d13fc700 1 --2- 172.21.2.102:0/1528093810 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f67b00186c0 0x7f67b001abc0 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:58.938 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:58.937 7f67d13fc700 1 --2- 172.21.2.102:0/1528093810 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f67cc1b7040 0x7f67cc1b7440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:58.938 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:58.937 7f67d13fc700 1 --2- 172.21.2.102:0/1528093810 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f67cc0ada20 0x7f67cc0ade20 unknown :-1 s=CLOSED pgs=1820 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:58.938 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:58.937 7f67d13fc700 1 -- 172.21.2.102:0/1528093810 >> 172.21.2.102:0/1528093810 conn(0x7f67cc19c120 msgr2=0x7f67cc092e30 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:58.938 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:58.937 7f67d13fc700 1 -- 172.21.2.102:0/1528093810 shutdown_connections 2021-02-10T11:27:58.938 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:27:58.937 7f67d13fc700 1 -- 172.21.2.102:0/1528093810 wait complete. 2021-02-10T11:27:59.147 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 65, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:27:41.479375', 'modified': '2021-02-10 11:27:43.490605', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6107}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6107': {'gid': 6107, 'name': 'b', 'rank': 0, 'incarnation': 62, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.113:6833/4220238062', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.113:6832', 'nonce': 4220238062}, {'type': 'v1', 'addr': '172.21.2.113:6833', 'nonce': 4220238062}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [17], 'metadata_pool': 16, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:27:59.147 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:27:59.148 DEBUG:teuthology.orchestra.run.gibba013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:27:59.545 INFO:teuthology.orchestra.run.gibba013.stdout:{ 2021-02-10T11:27:59.545 INFO:teuthology.orchestra.run.gibba013.stdout: "cluster_fsid": "68db30ea-dbce-4aef-a6df-f01c6fea0513", 2021-02-10T11:27:59.545 INFO:teuthology.orchestra.run.gibba013.stdout: "whoami": 0, 2021-02-10T11:27:59.545 INFO:teuthology.orchestra.run.gibba013.stdout: "id": 6107, 2021-02-10T11:27:59.545 INFO:teuthology.orchestra.run.gibba013.stdout: "want_state": "up:active", 2021-02-10T11:27:59.546 INFO:teuthology.orchestra.run.gibba013.stdout: "state": "up:active", 2021-02-10T11:27:59.546 INFO:teuthology.orchestra.run.gibba013.stdout: "rank_uptime": 18.064597883000001, 2021-02-10T11:27:59.546 INFO:teuthology.orchestra.run.gibba013.stdout: "mdsmap_epoch": 65, 2021-02-10T11:27:59.546 INFO:teuthology.orchestra.run.gibba013.stdout: "osdmap_epoch": 85, 2021-02-10T11:27:59.546 INFO:teuthology.orchestra.run.gibba013.stdout: "osdmap_epoch_barrier": 85, 2021-02-10T11:27:59.546 INFO:teuthology.orchestra.run.gibba013.stdout: "uptime": 23.713415854000001 2021-02-10T11:27:59.546 INFO:teuthology.orchestra.run.gibba013.stdout:} 2021-02-10T11:27:59.549 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "68db30ea-dbce-4aef-a6df-f01c6fea0513", "whoami": 0, "id": 6107, "want_state": "up:active", "state": "up:active", "rank_uptime": 18.064597883000001, "mdsmap_epoch": 65, "osdmap_epoch": 85, "osdmap_epoch_barrier": 85, "uptime": 23.713415854000001 } 2021-02-10T11:27:59.549 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2021-02-10T11:27:59.549 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2021-02-10T11:27:59.550 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.1 at ubuntu@gibba002.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.1... 2021-02-10T11:27:59.550 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:59.550 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec mkdir -p /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:27:59.886 DEBUG:teuthology.orchestra.run.gibba002:> sudo modprobe fuse 2021-02-10T11:28:00.231 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:28:00.231 DEBUG:teuthology.orchestra.run.gibba002:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:28:00.575 INFO:teuthology.orchestra.run.gibba002.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:28:00.576 INFO:teuthology.orchestra.run.gibba002.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:28:00.576 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:28:00.576 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:28:00.577 DEBUG:teuthology.orchestra.run.gibba002:> ls /sys/fs/fuse/connections 2021-02-10T11:28:00.910 INFO:teuthology.orchestra.run.gibba002.stdout:43 2021-02-10T11:28:00.910 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [43] 2021-02-10T11:28:00.910 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.1 /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:28:00.913 DEBUG:teuthology.orchestra.run.gibba002:> sudo modprobe fuse 2021-02-10T11:28:01.259 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:28:01.259 DEBUG:teuthology.orchestra.run.gibba002:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:28:01.326 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba002.stderr:ceph-fuse[170017]: starting ceph client 2021-02-10T11:28:01.326 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba002.stderr:2021-02-10 11:28:01.327 7fa5fe5c3f80 -1 init, newargv = 0x55e55fbe51a0 newargc=9 2021-02-10T11:28:01.331 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba002.stderr:ceph-fuse[170017]: starting fuse 2021-02-10T11:28:01.602 INFO:teuthology.orchestra.run.gibba002.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:28:01.603 INFO:teuthology.orchestra.run.gibba002.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:28:01.603 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:28:01.604 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:28:01.604 DEBUG:teuthology.orchestra.run.gibba002:> ls /sys/fs/fuse/connections 2021-02-10T11:28:01.938 INFO:teuthology.orchestra.run.gibba002.stdout:43 2021-02-10T11:28:01.938 INFO:teuthology.orchestra.run.gibba002.stdout:44 2021-02-10T11:28:01.938 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43, 44] 2021-02-10T11:28:01.938 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:28:01.938 DEBUG:teuthology.orchestra.run.gibba002:> sudo python3 -c ' 2021-02-10T11:28:01.938 DEBUG:teuthology.orchestra.run.gibba002:> import glob 2021-02-10T11:28:01.938 DEBUG:teuthology.orchestra.run.gibba002:> import re 2021-02-10T11:28:01.939 DEBUG:teuthology.orchestra.run.gibba002:> import os 2021-02-10T11:28:01.939 DEBUG:teuthology.orchestra.run.gibba002:> import subprocess 2021-02-10T11:28:01.939 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:28:01.939 DEBUG:teuthology.orchestra.run.gibba002:> def find_socket(client_name): 2021-02-10T11:28:01.939 DEBUG:teuthology.orchestra.run.gibba002:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2021-02-10T11:28:01.939 DEBUG:teuthology.orchestra.run.gibba002:> files = glob.glob(asok_path) 2021-02-10T11:28:01.939 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:28:01.939 DEBUG:teuthology.orchestra.run.gibba002:> # Given a non-glob path, it better be there 2021-02-10T11:28:01.939 DEBUG:teuthology.orchestra.run.gibba002:> if "*" not in asok_path: 2021-02-10T11:28:01.939 DEBUG:teuthology.orchestra.run.gibba002:> assert(len(files) == 1) 2021-02-10T11:28:01.939 DEBUG:teuthology.orchestra.run.gibba002:> return files[0] 2021-02-10T11:28:01.939 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:28:01.939 DEBUG:teuthology.orchestra.run.gibba002:> for f in files: 2021-02-10T11:28:01.939 DEBUG:teuthology.orchestra.run.gibba002:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2021-02-10T11:28:01.940 DEBUG:teuthology.orchestra.run.gibba002:> if os.path.exists("/proc/{0}".format(pid)): 2021-02-10T11:28:01.940 DEBUG:teuthology.orchestra.run.gibba002:> return f 2021-02-10T11:28:01.940 DEBUG:teuthology.orchestra.run.gibba002:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2021-02-10T11:28:01.940 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:28:01.940 DEBUG:teuthology.orchestra.run.gibba002:> print(find_socket("client.1")) 2021-02-10T11:28:01.940 DEBUG:teuthology.orchestra.run.gibba002:> ' 2021-02-10T11:28:02.300 INFO:teuthology.orchestra.run.gibba002.stdout:/var/run/ceph/ceph-client.1.170017.asok 2021-02-10T11:28:02.303 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.170017.asok 2021-02-10T11:28:02.303 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:28:02.303 DEBUG:teuthology.orchestra.run.gibba002:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.170017.asok status 2021-02-10T11:28:02.692 INFO:teuthology.orchestra.run.gibba002.stdout:{ 2021-02-10T11:28:02.692 INFO:teuthology.orchestra.run.gibba002.stdout: "metadata": { 2021-02-10T11:28:02.692 INFO:teuthology.orchestra.run.gibba002.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:28:02.692 INFO:teuthology.orchestra.run.gibba002.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:28:02.692 INFO:teuthology.orchestra.run.gibba002.stdout: "entity_id": "1", 2021-02-10T11:28:02.692 INFO:teuthology.orchestra.run.gibba002.stdout: "hostname": "gibba002", 2021-02-10T11:28:02.692 INFO:teuthology.orchestra.run.gibba002.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2021-02-10T11:28:02.692 INFO:teuthology.orchestra.run.gibba002.stdout: "pid": "170017", 2021-02-10T11:28:02.692 INFO:teuthology.orchestra.run.gibba002.stdout: "root": "/" 2021-02-10T11:28:02.693 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:28:02.693 INFO:teuthology.orchestra.run.gibba002.stdout: "dentry_count": 0, 2021-02-10T11:28:02.693 INFO:teuthology.orchestra.run.gibba002.stdout: "dentry_pinned_count": 0, 2021-02-10T11:28:02.693 INFO:teuthology.orchestra.run.gibba002.stdout: "id": 9432, 2021-02-10T11:28:02.693 INFO:teuthology.orchestra.run.gibba002.stdout: "inst": { 2021-02-10T11:28:02.693 INFO:teuthology.orchestra.run.gibba002.stdout: "name": { 2021-02-10T11:28:02.693 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "client", 2021-02-10T11:28:02.693 INFO:teuthology.orchestra.run.gibba002.stdout: "num": 9432 2021-02-10T11:28:02.693 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:28:02.693 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": { 2021-02-10T11:28:02.693 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:28:02.693 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:0", 2021-02-10T11:28:02.693 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 4019902900 2021-02-10T11:28:02.693 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:28:02.694 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:28:02.694 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": { 2021-02-10T11:28:02.694 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:28:02.694 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:0", 2021-02-10T11:28:02.694 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 4019902900 2021-02-10T11:28:02.694 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:28:02.694 INFO:teuthology.orchestra.run.gibba002.stdout: "inst_str": "client.9432 172.21.2.102:0/4019902900", 2021-02-10T11:28:02.694 INFO:teuthology.orchestra.run.gibba002.stdout: "addr_str": "172.21.2.102:0/4019902900", 2021-02-10T11:28:02.694 INFO:teuthology.orchestra.run.gibba002.stdout: "inode_count": 1, 2021-02-10T11:28:02.694 INFO:teuthology.orchestra.run.gibba002.stdout: "mds_epoch": 65, 2021-02-10T11:28:02.694 INFO:teuthology.orchestra.run.gibba002.stdout: "osd_epoch": 85, 2021-02-10T11:28:02.694 INFO:teuthology.orchestra.run.gibba002.stdout: "osd_epoch_barrier": 0, 2021-02-10T11:28:02.694 INFO:teuthology.orchestra.run.gibba002.stdout: "blacklisted": false 2021-02-10T11:28:02.695 INFO:teuthology.orchestra.run.gibba002.stdout:} 2021-02-10T11:28:02.696 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:28:02.696 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:28:02.696 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:28:03.033 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:28:03.928 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2021-02-10T11:28:03.929 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:28:03.929 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:28:04.276 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2021-02-10T11:28:04.711 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:04.693 7f1458670700 1 Processor -- start 2021-02-10T11:28:04.711 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:04.693 7f1458670700 1 -- start start 2021-02-10T11:28:04.711 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:04.694 7f1458670700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f14500d1870 0x7f14501b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:04.711 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:04.694 7f1458670700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f14500cff70 0x7f1450109130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:04.711 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:04.694 7f1458670700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1450109a60 con 0x7f14501b0ae0 2021-02-10T11:28:04.711 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:04.694 7f1458670700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1450109750 con 0x7f14500d1870 2021-02-10T11:28:04.711 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:04.694 7f1458670700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1450088070 con 0x7f14500cff70 2021-02-10T11:28:04.711 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:04.694 7f1455c8b700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f14501b0ae0 0x7f1450108940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:52280/0 (socket says 172.21.2.102:52280) 2021-02-10T11:28:04.711 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:04.694 7f1455c8b700 1 -- 172.21.2.102:0/598540025 learned_addr learned my addr 172.21.2.102:0/598540025 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:28:04.711 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:04.694 7f145548a700 1 -- 172.21.2.102:0/598540025 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 399647343 0 0) 0x7f14440010b0 con 0x7f14501b0ae0 2021-02-10T11:28:04.712 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:04.694 7f145548a700 1 -- 172.21.2.102:0/598540025 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f143c002fe0 con 0x7f14501b0ae0 2021-02-10T11:28:04.712 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:04.694 7f1456c8d700 1 --2- 172.21.2.102:0/598540025 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f14500cff70 0x7f1450109130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:04.712 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:04.694 7f145648c700 1 --2- 172.21.2.102:0/598540025 >> v2:172.21.2.113:3300/0 conn(0x7f14500d1870 0x7f14501b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:04.712 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:04.695 7f145548a700 1 -- 172.21.2.102:0/598540025 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3088503285 0 0) 0x7f14440010b0 con 0x7f14501b0ae0 2021-02-10T11:28:04.712 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:04.695 7f145548a700 1 -- 172.21.2.102:0/598540025 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f14500cff70 msgr2=0x7f1450109130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:28:04.712 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:04.695 7f145548a700 1 --2- 172.21.2.102:0/598540025 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f14500cff70 0x7f1450109130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:04.712 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:04.695 7f145548a700 1 -- 172.21.2.102:0/598540025 >> v2:172.21.2.113:3300/0 conn(0x7f14500d1870 msgr2=0x7f14501b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:28:04.712 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:04.695 7f145548a700 1 --2- 172.21.2.102:0/598540025 >> v2:172.21.2.113:3300/0 conn(0x7f14500d1870 0x7f14501b04c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:04.712 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:04.695 7f145548a700 1 -- 172.21.2.102:0/598540025 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14500d1c70 con 0x7f14501b0ae0 2021-02-10T11:28:04.712 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:04.695 7f145548a700 1 -- 172.21.2.102:0/598540025 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7f14440015b0 con 0x7f14501b0ae0 2021-02-10T11:28:04.712 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:04.695 7f145548a700 1 -- 172.21.2.102:0/598540025 >> v1:172.21.2.102:6789/0 conn(0x7f14501b0ae0 legacy=0x7f1450108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:04.712 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:04.695 7f145548a700 1 --2- 172.21.2.102:0/598540025 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f143c003ae0 0x7f143c005f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:04.712 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:04.695 7f145548a700 1 --2- 172.21.2.102:0/598540025 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f143c0067f0 0x7f143c008d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:04.713 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:04.695 7f145548a700 1 --2- 172.21.2.102:0/598540025 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f143c009590 0x7f143c00bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:04.713 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:04.695 7f145548a700 1 -- 172.21.2.102:0/598540025 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f143c00c360 con 0x7f143c009590 2021-02-10T11:28:04.713 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:04.695 7f145548a700 1 -- 172.21.2.102:0/598540025 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f143c050550 con 0x7f143c0067f0 2021-02-10T11:28:04.713 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:04.695 7f145548a700 1 -- 172.21.2.102:0/598540025 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f143c050770 con 0x7f143c003ae0 2021-02-10T11:28:04.713 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:04.695 7f1456c8d700 1 --2- 172.21.2.102:0/598540025 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f14500cff70 0x7f1450109130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:28:04.713 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:04.695 7f1456c8d700 1 --2- 172.21.2.102:0/598540025 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f143c009590 0x7f143c00bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:04.713 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:04.695 7f1455c8b700 1 --2- 172.21.2.102:0/598540025 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f143c0067f0 0x7f143c008d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:04.713 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:04.695 7f145648c700 1 --2- 172.21.2.102:0/598540025 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f143c003ae0 0x7f143c005f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:04.713 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:04.695 7f1456c8d700 1 -- 172.21.2.102:0/598540025 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f143c003ae0 msgr2=0x7f143c005f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:28:04.713 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:04.695 7f1456c8d700 1 --2- 172.21.2.102:0/598540025 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f143c003ae0 0x7f143c005f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:04.713 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:04.695 7f1456c8d700 1 -- 172.21.2.102:0/598540025 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f143c0067f0 msgr2=0x7f143c008d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:28:04.713 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:04.695 7f1456c8d700 1 --2- 172.21.2.102:0/598540025 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f143c0067f0 0x7f143c008d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:04.713 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:04.695 7f1456c8d700 1 -- 172.21.2.102:0/598540025 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f143c050a10 con 0x7f143c009590 2021-02-10T11:28:04.714 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:04.696 7f1456c8d700 1 --2- 172.21.2.102:0/598540025 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f143c009590 0x7f143c00bab0 secure :-1 s=READY pgs=1825 cs=0 l=1 rev1=1 rx=0x7f1448009b30 tx=0x7f1448009ed0).ready entity=mon.0 client_cookie=eecb738722f1411a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:28:04.714 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:04.696 7f145548a700 1 -- 172.21.2.102:0/598540025 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1448006710 con 0x7f143c009590 2021-02-10T11:28:04.714 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:04.696 7f145548a700 1 -- 172.21.2.102:0/598540025 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1448003030 con 0x7f143c009590 2021-02-10T11:28:04.714 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:04.696 7f145548a700 1 -- 172.21.2.102:0/598540025 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f144800b7b0 con 0x7f143c009590 2021-02-10T11:28:04.714 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:04.696 7f1458670700 1 -- 172.21.2.102:0/598540025 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f143c009590 msgr2=0x7f143c00bab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:04.714 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:04.696 7f1458670700 1 --2- 172.21.2.102:0/598540025 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f143c009590 0x7f143c00bab0 secure :-1 s=READY pgs=1825 cs=0 l=1 rev1=1 rx=0x7f1448009b30 tx=0x7f1448009ed0).stop 2021-02-10T11:28:04.714 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:04.696 7f1456c8d700 1 -- 172.21.2.102:0/598540025 reap_dead start 2021-02-10T11:28:04.714 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:04.697 7f1458670700 1 -- 172.21.2.102:0/598540025 shutdown_connections 2021-02-10T11:28:04.714 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:04.697 7f1458670700 1 -- 172.21.2.102:0/598540025 >> 172.21.2.102:0/598540025 conn(0x7f14500ce420 msgr2=0x7f14500d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:28:04.714 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:04.697 7f1458670700 1 -- 172.21.2.102:0/598540025 shutdown_connections 2021-02-10T11:28:04.714 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:04.697 7f1458670700 1 -- 172.21.2.102:0/598540025 wait complete. 2021-02-10T11:28:04.714 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:04.712 7f1458670700 1 Processor -- start 2021-02-10T11:28:04.714 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:04.712 7f1458670700 1 -- start start 2021-02-10T11:28:04.716 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:04.712 7f1458670700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f14501a05a0 0x7f14500a8c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:04.716 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:04.712 7f1458670700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f14500b5090 0x7f14500b5490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:04.716 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:04.712 7f1458670700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f14500b5ab0 0x7f14500b68d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:04.716 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:04.712 7f1458670700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14500b6090 con 0x7f14500b5090 2021-02-10T11:28:04.716 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:04.712 7f1458670700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14500a9240 con 0x7f14500b5ab0 2021-02-10T11:28:04.716 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:04.712 7f1458670700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14500b2a30 con 0x7f14501a05a0 2021-02-10T11:28:04.717 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:04.712 7f1455c8b700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f14500b5090 0x7f14500b5490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:04.717 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:04.712 7f1455c8b700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f14500b5090 0x7f14500b5490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:60070/0 (socket says 172.21.2.102:60070) 2021-02-10T11:28:04.717 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:04.712 7f1455c8b700 1 -- 172.21.2.102:0/2464261095 learned_addr learned my addr 172.21.2.102:0/2464261095 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:28:04.717 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:04.712 7f1455c8b700 1 -- 172.21.2.102:0/2464261095 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f14501a05a0 msgr2=0x7f14500a8c20 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:28:04.717 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:04.713 7f1455c8b700 1 --2- 172.21.2.102:0/2464261095 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f14501a05a0 0x7f14500a8c20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:28:04.717 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:04.713 7f1455c8b700 1 -- 172.21.2.102:0/2464261095 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f14500b5ab0 msgr2=0x7f14500b68d0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:28:04.717 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:04.713 7f1455c8b700 1 --2- 172.21.2.102:0/2464261095 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f14500b5ab0 0x7f14500b68d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:28:04.717 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:04.713 7f1455c8b700 1 -- 172.21.2.102:0/2464261095 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14500b6ef0 con 0x7f14500b5090 2021-02-10T11:28:04.717 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:04.713 7f1455c8b700 1 --2- 172.21.2.102:0/2464261095 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f14500b5090 0x7f14500b5490 secure :-1 s=READY pgs=1826 cs=0 l=1 rev1=1 rx=0x7f1444001c00 tx=0x7f144400a8e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:28:04.717 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:04.713 7f1442ffd700 1 -- 172.21.2.102:0/2464261095 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f14440009b0 con 0x7f14500b5090 2021-02-10T11:28:04.717 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:04.713 7f1442ffd700 1 -- 172.21.2.102:0/2464261095 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f144400acd0 con 0x7f14500b5090 2021-02-10T11:28:04.717 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:04.713 7f1442ffd700 1 -- 172.21.2.102:0/2464261095 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1444004540 con 0x7f14500b5090 2021-02-10T11:28:04.717 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:04.713 7f1458670700 1 -- 172.21.2.102:0/2464261095 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14500b6ef0 con 0x7f14500b5090 2021-02-10T11:28:04.718 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:04.713 7f1458670700 1 -- 172.21.2.102:0/2464261095 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14500b7480 con 0x7f14500b5090 2021-02-10T11:28:04.718 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:04.714 7f1442ffd700 1 -- 172.21.2.102:0/2464261095 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1444004140 con 0x7f14500b5090 2021-02-10T11:28:04.718 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:04.714 7f1442ffd700 1 --2- 172.21.2.102:0/2464261095 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f14380186c0 0x7f143801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:04.718 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:04.714 7f1442ffd700 1 -- 172.21.2.102:0/2464261095 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f1444032520 con 0x7f14500b5090 2021-02-10T11:28:04.718 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:04.714 7f145648c700 1 --2- 172.21.2.102:0/2464261095 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f14380186c0 0x7f143801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:04.718 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:04.714 7f145648c700 1 --2- 172.21.2.102:0/2464261095 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f14380186c0 0x7f143801abc0 crc :-1 s=READY pgs=822 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:28:04.742 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:04.743 7f1458670700 1 -- 172.21.2.102:0/2464261095 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14500825c0 con 0x7f14500b5090 2021-02-10T11:28:04.745 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:04.746 7f1442ffd700 1 -- 172.21.2.102:0/2464261095 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f1444000ce0 con 0x7f14500b5090 2021-02-10T11:28:04.882 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:04.883 7f1458670700 1 -- 172.21.2.102:0/2464261095 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f14500825c0 con 0x7f14500b5090 2021-02-10T11:28:05.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:05.052 7f1442ffd700 1 -- 172.21.2.102:0/2464261095 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v66) v1 ==== 136+0+0 (secure 0 0 0) 0x7f144401f4e0 con 0x7f14500b5090 2021-02-10T11:28:05.101 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:05.101 7f1458670700 1 -- 172.21.2.102:0/2464261095 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f14380186c0 msgr2=0x7f143801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:05.101 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:05.101 7f1458670700 1 --2- 172.21.2.102:0/2464261095 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f14380186c0 0x7f143801abc0 crc :-1 s=READY pgs=822 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:05.101 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:05.101 7f1458670700 1 -- 172.21.2.102:0/2464261095 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f14500b5090 msgr2=0x7f14500b5490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:05.101 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:05.101 7f1458670700 1 --2- 172.21.2.102:0/2464261095 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f14500b5090 0x7f14500b5490 secure :-1 s=READY pgs=1826 cs=0 l=1 rev1=1 rx=0x7f1444001c00 tx=0x7f144400a8e0).stop 2021-02-10T11:28:05.101 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:05.101 7f1458670700 1 -- 172.21.2.102:0/2464261095 shutdown_connections 2021-02-10T11:28:05.101 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:05.101 7f1458670700 1 --2- 172.21.2.102:0/2464261095 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f14380186c0 0x7f143801abc0 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:05.101 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:05.101 7f1458670700 1 --2- 172.21.2.102:0/2464261095 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f14500b5ab0 0x7f14500b68d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:28:05.101 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:05.101 7f1458670700 1 --2- 172.21.2.102:0/2464261095 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f14500b5090 0x7f14500b5490 unknown :-1 s=CLOSED pgs=1826 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:05.101 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:05.101 7f1458670700 1 --2- 172.21.2.102:0/2464261095 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f14501a05a0 0x7f14500a8c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:28:05.101 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:05.101 7f1458670700 1 -- 172.21.2.102:0/2464261095 >> 172.21.2.102:0/2464261095 conn(0x7f14500bc130 msgr2=0x7f14500a9630 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:28:05.102 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:05.101 7f1458670700 1 -- 172.21.2.102:0/2464261095 shutdown_connections 2021-02-10T11:28:05.102 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:05.102 7f1458670700 1 -- 172.21.2.102:0/2464261095 wait complete. 2021-02-10T11:28:05.108 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs volume ls 2021-02-10T11:28:05.534 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:05.519 7f9fc17b8700 1 Processor -- start 2021-02-10T11:28:05.535 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:05.519 7f9fc17b8700 1 -- start start 2021-02-10T11:28:05.535 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:05.520 7f9fc17b8700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f9fbc0d0340 0x7f9fbc0ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:05.535 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:05.520 7f9fc17b8700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f9fbc0d1c70 0x7f9fbc0efc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:05.535 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:05.520 7f9fc17b8700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fbc0f0b10 con 0x7f9fbc0d1870 2021-02-10T11:28:05.535 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:05.520 7f9fc17b8700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9fbc0e5ed0 con 0x7f9fbc0d0340 2021-02-10T11:28:05.535 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:05.520 7f9fc17b8700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9fbc1b0be0 con 0x7f9fbc0d1c70 2021-02-10T11:28:05.535 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:05.520 7f9fbad73700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f9fbc0d0340 0x7f9fbc0ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:05.535 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:05.520 7f9fbb574700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f9fbc0d1870 0x7f9fbc1b03f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:52300/0 (socket says 172.21.2.102:52300) 2021-02-10T11:28:05.535 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:05.520 7f9fbb574700 1 -- 172.21.2.102:0/1008105602 learned_addr learned my addr 172.21.2.102:0/1008105602 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:28:05.535 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:05.520 7f9fbbd75700 1 --2- 172.21.2.102:0/1008105602 >> v2:172.21.2.113:3300/0 conn(0x7f9fbc0d1c70 0x7f9fbc0efc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:05.535 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:05.520 7f9fbad73700 1 -- 172.21.2.102:0/1008105602 >> v2:172.21.2.113:3300/0 conn(0x7f9fbc0d1c70 msgr2=0x7f9fbc0efc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:28:05.535 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:05.520 7f9fbad73700 1 --2- 172.21.2.102:0/1008105602 >> v2:172.21.2.113:3300/0 conn(0x7f9fbc0d1c70 0x7f9fbc0efc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:05.536 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:05.520 7f9fbad73700 1 -- 172.21.2.102:0/1008105602 >> v1:172.21.2.102:6789/0 conn(0x7f9fbc0d1870 legacy=0x7f9fbc1b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:05.536 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:05.520 7f9fbad73700 1 -- 172.21.2.102:0/1008105602 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fbc0f20a0 con 0x7f9fbc0d0340 2021-02-10T11:28:05.536 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:05.520 7f9fbad73700 1 --2- 172.21.2.102:0/1008105602 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9fbc0d0340 0x7f9fbc0ef640 secure :-1 s=READY pgs=1828 cs=0 l=1 rev1=1 rx=0x7f9fa4006650 tx=0x7f9fa4009a80).ready entity=mon.0 client_cookie=c1ae08198a76bb07 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:28:05.536 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:05.521 7f9fba572700 1 -- 172.21.2.102:0/1008105602 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9fa40009b0 con 0x7f9fbc0d0340 2021-02-10T11:28:05.536 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:05.521 7f9fba572700 1 -- 172.21.2.102:0/1008105602 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fa400c0c0 con 0x7f9fbc0d0340 2021-02-10T11:28:05.536 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:05.521 7f9fba572700 1 -- 172.21.2.102:0/1008105602 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9fa400c510 con 0x7f9fbc0d0340 2021-02-10T11:28:05.536 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:05.521 7f9fc17b8700 1 -- 172.21.2.102:0/1008105602 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9fbc0d0340 msgr2=0x7f9fbc0ef640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:05.536 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:05.521 7f9fc17b8700 1 --2- 172.21.2.102:0/1008105602 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9fbc0d0340 0x7f9fbc0ef640 secure :-1 s=READY pgs=1828 cs=0 l=1 rev1=1 rx=0x7f9fa4006650 tx=0x7f9fa4009a80).stop 2021-02-10T11:28:05.536 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:05.521 7f9fc17b8700 1 -- 172.21.2.102:0/1008105602 shutdown_connections 2021-02-10T11:28:05.536 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:05.521 7f9fc17b8700 1 --2- 172.21.2.102:0/1008105602 >> v2:172.21.2.113:3300/0 conn(0x7f9fbc0d1c70 0x7f9fbc0efc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:05.536 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:05.521 7f9fc17b8700 1 --2- 172.21.2.102:0/1008105602 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9fbc0d0340 0x7f9fbc0ef640 unknown :-1 s=CLOSED pgs=1828 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:05.536 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:05.521 7f9fc17b8700 1 -- 172.21.2.102:0/1008105602 >> 172.21.2.102:0/1008105602 conn(0x7f9fbc0ce450 msgr2=0x7f9fbc0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:28:05.536 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:05.521 7f9fc17b8700 1 -- 172.21.2.102:0/1008105602 shutdown_connections 2021-02-10T11:28:05.537 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:05.521 7f9fc17b8700 1 -- 172.21.2.102:0/1008105602 wait complete. 2021-02-10T11:28:05.538 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:05.536 7f9fc17b8700 1 Processor -- start 2021-02-10T11:28:05.538 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:05.536 7f9fc17b8700 1 -- start start 2021-02-10T11:28:05.538 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:05.536 7f9fc17b8700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9fbc19ffa0 0x7f9fbc0eda10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:05.538 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:05.536 7f9fc17b8700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f9fbc1a03a0 0x7f9fbc0ee030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:05.538 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:05.536 7f9fc17b8700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9fbc0ee650 0x7f9fbc0ebe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:05.538 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:05.536 7f9fc17b8700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fbc0ed490 con 0x7f9fbc19ffa0 2021-02-10T11:28:05.538 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:05.536 7f9fc17b8700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fbc0eea50 con 0x7f9fbc1a03a0 2021-02-10T11:28:05.539 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:05.536 7f9fc17b8700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9fbc0eec30 con 0x7f9fbc0ee650 2021-02-10T11:28:05.539 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:05.536 7f9fbb574700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9fbc19ffa0 0x7f9fbc0eda10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:05.539 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:05.536 7f9fbb574700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9fbc19ffa0 0x7f9fbc0eda10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:60082/0 (socket says 172.21.2.102:60082) 2021-02-10T11:28:05.539 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:05.536 7f9fbb574700 1 -- 172.21.2.102:0/2979159813 learned_addr learned my addr 172.21.2.102:0/2979159813 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:28:05.539 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:05.537 7f9fbb574700 1 -- 172.21.2.102:0/2979159813 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9fbc0ee650 msgr2=0x7f9fbc0ebe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:05.539 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:05.537 7f9fbb574700 1 --2- 172.21.2.102:0/2979159813 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9fbc0ee650 0x7f9fbc0ebe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:28:05.539 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:05.537 7f9fbb574700 1 -- 172.21.2.102:0/2979159813 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f9fbc1a03a0 msgr2=0x7f9fbc0ee030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:05.539 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:05.537 7f9fbb574700 1 --2- 172.21.2.102:0/2979159813 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f9fbc1a03a0 0x7f9fbc0ee030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:28:05.539 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:05.537 7f9fbb574700 1 -- 172.21.2.102:0/2979159813 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fbc0ec4a0 con 0x7f9fbc19ffa0 2021-02-10T11:28:05.539 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:05.537 7f9fbb574700 1 --2- 172.21.2.102:0/2979159813 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9fbc19ffa0 0x7f9fbc0eda10 secure :-1 s=READY pgs=1829 cs=0 l=1 rev1=1 rx=0x7f9fac001c60 tx=0x7f9fac00a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:28:05.539 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:05.537 7f9fb3fff700 1 -- 172.21.2.102:0/2979159813 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9fac00acc0 con 0x7f9fbc19ffa0 2021-02-10T11:28:05.539 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:05.537 7f9fb3fff700 1 -- 172.21.2.102:0/2979159813 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fac004140 con 0x7f9fbc19ffa0 2021-02-10T11:28:05.539 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:05.537 7f9fb3fff700 1 -- 172.21.2.102:0/2979159813 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9fac004660 con 0x7f9fbc19ffa0 2021-02-10T11:28:05.540 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:05.537 7f9fc17b8700 1 -- 172.21.2.102:0/2979159813 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9fbc0ec4a0 con 0x7f9fbc19ffa0 2021-02-10T11:28:05.540 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:05.537 7f9fc17b8700 1 -- 172.21.2.102:0/2979159813 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9fbc0eca30 con 0x7f9fbc19ffa0 2021-02-10T11:28:05.540 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:05.538 7f9fb3fff700 1 -- 172.21.2.102:0/2979159813 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f9fac004320 con 0x7f9fbc19ffa0 2021-02-10T11:28:05.540 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:05.538 7f9fb3fff700 1 --2- 172.21.2.102:0/2979159813 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f9fa00186c0 0x7f9fa001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:05.540 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:05.538 7f9fb3fff700 1 -- 172.21.2.102:0/2979159813 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f9fac0323d0 con 0x7f9fbc19ffa0 2021-02-10T11:28:05.540 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:05.538 7f9fbad73700 1 --2- 172.21.2.102:0/2979159813 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f9fa00186c0 0x7f9fa001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:05.540 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:05.538 7f9fbad73700 1 --2- 172.21.2.102:0/2979159813 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f9fa00186c0 0x7f9fa001abc0 crc :-1 s=READY pgs=823 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:28:05.569 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:05.570 7f9fc17b8700 1 -- 172.21.2.102:0/2979159813 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9fbc0ebc50 con 0x7f9fbc19ffa0 2021-02-10T11:28:05.571 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:05.572 7f9fb3fff700 1 -- 172.21.2.102:0/2979159813 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f9fac0010b0 con 0x7f9fbc19ffa0 2021-02-10T11:28:05.705 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:05.706 7f9fc17b8700 1 -- 172.21.2.102:0/2979159813 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs volume ls", "target": ["mgr", ""]}) v1 -- 0x7f9fbc0eca30 con 0x7f9fa00186c0 2021-02-10T11:28:05.705 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:05.706 7f9fb3fff700 1 -- 172.21.2.102:0/2979159813 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+41 (crc 0 0 0) 0x7f9fa4006040 con 0x7f9fa00186c0 2021-02-10T11:28:05.706 INFO:teuthology.orchestra.run.gibba002.stdout:[ 2021-02-10T11:28:05.706 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:28:05.706 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "cephfs" 2021-02-10T11:28:05.706 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:28:05.706 INFO:teuthology.orchestra.run.gibba002.stdout:] 2021-02-10T11:28:05.707 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:05.708 7f9fc17b8700 1 -- 172.21.2.102:0/2979159813 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f9fa00186c0 msgr2=0x7f9fa001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:05.707 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:05.708 7f9fc17b8700 1 --2- 172.21.2.102:0/2979159813 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f9fa00186c0 0x7f9fa001abc0 crc :-1 s=READY pgs=823 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:05.707 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:05.708 7f9fc17b8700 1 -- 172.21.2.102:0/2979159813 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9fbc19ffa0 msgr2=0x7f9fbc0eda10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:05.707 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:05.708 7f9fc17b8700 1 --2- 172.21.2.102:0/2979159813 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9fbc19ffa0 0x7f9fbc0eda10 secure :-1 s=READY pgs=1829 cs=0 l=1 rev1=1 rx=0x7f9fac001c60 tx=0x7f9fac00a8d0).stop 2021-02-10T11:28:05.707 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:05.708 7f9fc17b8700 1 -- 172.21.2.102:0/2979159813 shutdown_connections 2021-02-10T11:28:05.707 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:05.708 7f9fc17b8700 1 --2- 172.21.2.102:0/2979159813 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9fbc0ee650 0x7f9fbc0ebe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:28:05.707 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:05.708 7f9fc17b8700 1 --2- 172.21.2.102:0/2979159813 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f9fa00186c0 0x7f9fa001abc0 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:05.707 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:05.708 7f9fc17b8700 1 --2- 172.21.2.102:0/2979159813 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f9fbc1a03a0 0x7f9fbc0ee030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:28:05.708 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:05.708 7f9fc17b8700 1 --2- 172.21.2.102:0/2979159813 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9fbc19ffa0 0x7f9fbc0eda10 unknown :-1 s=CLOSED pgs=1829 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:05.708 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:05.708 7f9fc17b8700 1 -- 172.21.2.102:0/2979159813 >> 172.21.2.102:0/2979159813 conn(0x7f9fbc094d90 msgr2=0x7f9fbc08a420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:28:05.708 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:05.708 7f9fc17b8700 1 -- 172.21.2.102:0/2979159813 shutdown_connections 2021-02-10T11:28:05.708 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:05.708 7f9fc17b8700 1 -- 172.21.2.102:0/2979159813 wait complete. 2021-02-10T11:28:05.716 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config set mon mon_allow_pool_delete True 2021-02-10T11:28:06.148 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:06.132 7f3224047700 1 Processor -- start 2021-02-10T11:28:06.149 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:06.132 7f3224047700 1 -- start start 2021-02-10T11:28:06.149 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:06.133 7f3224047700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f321c0d1c70 0x7f321c0cef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:06.149 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:06.133 7f3224047700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f321c0d0170 con 0x7f321c0d1870 2021-02-10T11:28:06.149 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:06.133 7f3224047700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f321c0fe1c0 con 0x7f321c0cf580 2021-02-10T11:28:06.149 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:06.133 7f3224047700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f321c1b0cb0 con 0x7f321c0d1c70 2021-02-10T11:28:06.149 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:06.133 7f3221e63700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f321c0d1870 0x7f321c1b04c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:52314/0 (socket says 172.21.2.102:52314) 2021-02-10T11:28:06.149 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:06.133 7f3221e63700 1 -- 172.21.2.102:0/2353196042 learned_addr learned my addr 172.21.2.102:0/2353196042 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:28:06.149 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:06.133 7f3220e61700 1 -- 172.21.2.102:0/2353196042 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2488909352 0 0) 0x7f32180010b0 con 0x7f321c0d1870 2021-02-10T11:28:06.149 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:06.133 7f3220e61700 1 -- 172.21.2.102:0/2353196042 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3208002fe0 con 0x7f321c0d1870 2021-02-10T11:28:06.149 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:06.133 7f3221662700 1 --2- 172.21.2.102:0/2353196042 >> v2:172.21.2.113:3300/0 conn(0x7f321c0d1c70 0x7f321c0cef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:06.150 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:06.133 7f3220e61700 1 -- 172.21.2.102:0/2353196042 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2140839748 0 0) 0x7f32180010b0 con 0x7f321c0d1870 2021-02-10T11:28:06.150 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:06.133 7f3220e61700 1 -- 172.21.2.102:0/2353196042 >> v2:172.21.2.113:3300/0 conn(0x7f321c0d1c70 msgr2=0x7f321c0cef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:28:06.150 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:06.133 7f3220e61700 1 --2- 172.21.2.102:0/2353196042 >> v2:172.21.2.113:3300/0 conn(0x7f321c0d1c70 0x7f321c0cef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:06.150 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:06.133 7f3220e61700 1 -- 172.21.2.102:0/2353196042 >> v1:172.21.2.113:6789/0 conn(0x7f321c0cf580 legacy=0x7f321c0cf980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:06.150 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:06.134 7f3220e61700 1 -- 172.21.2.102:0/2353196042 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f321c109a60 con 0x7f321c0d1870 2021-02-10T11:28:06.150 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:06.134 7f3220e61700 1 -- 172.21.2.102:0/2353196042 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7f32180015b0 con 0x7f321c0d1870 2021-02-10T11:28:06.150 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:06.134 7f3221662700 1 --2- 172.21.2.102:0/2353196042 >> v2:172.21.2.113:3300/0 conn(0x7f321c0d1c70 0x7f321c0cef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:28:06.150 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:06.134 7f3220e61700 1 -- 172.21.2.102:0/2353196042 >> v1:172.21.2.102:6789/0 conn(0x7f321c0d1870 legacy=0x7f321c1b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:06.150 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:06.134 7f3220e61700 1 --2- 172.21.2.102:0/2353196042 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3208003a60 0x7f3208005ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:06.150 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:06.134 7f3220e61700 1 --2- 172.21.2.102:0/2353196042 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3208006790 0x7f3208008c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:06.150 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:06.134 7f3220e61700 1 --2- 172.21.2.102:0/2353196042 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3208009590 0x7f320800bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:06.150 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:06.134 7f3220e61700 1 -- 172.21.2.102:0/2353196042 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f320800c300 con 0x7f3208003a60 2021-02-10T11:28:06.150 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:06.134 7f3220e61700 1 -- 172.21.2.102:0/2353196042 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f320800c520 con 0x7f3208009590 2021-02-10T11:28:06.151 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:06.134 7f3220e61700 1 -- 172.21.2.102:0/2353196042 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3208050750 con 0x7f3208006790 2021-02-10T11:28:06.151 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:06.134 7f3221e63700 1 --2- 172.21.2.102:0/2353196042 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3208003a60 0x7f3208005ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:06.151 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:06.134 7f3221662700 1 --2- 172.21.2.102:0/2353196042 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3208006790 0x7f3208008c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:06.151 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:06.134 7f3222664700 1 --2- 172.21.2.102:0/2353196042 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3208009590 0x7f320800bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:06.151 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:06.134 7f3221e63700 1 -- 172.21.2.102:0/2353196042 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3208006790 msgr2=0x7f3208008c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:28:06.151 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:06.134 7f3221e63700 1 --2- 172.21.2.102:0/2353196042 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3208006790 0x7f3208008c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:06.151 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:06.134 7f3221e63700 1 -- 172.21.2.102:0/2353196042 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3208009590 msgr2=0x7f320800bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:28:06.151 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:06.134 7f3221e63700 1 --2- 172.21.2.102:0/2353196042 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3208009590 0x7f320800bab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:06.151 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:06.134 7f3221e63700 1 -- 172.21.2.102:0/2353196042 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32080509b0 con 0x7f3208003a60 2021-02-10T11:28:06.151 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:06.134 7f3222664700 1 --2- 172.21.2.102:0/2353196042 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3208009590 0x7f320800bab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:28:06.151 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:06.134 7f3222664700 1 -- 172.21.2.102:0/2353196042 reap_dead start 2021-02-10T11:28:06.151 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:06.134 7f3221e63700 1 --2- 172.21.2.102:0/2353196042 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3208003a60 0x7f3208005ef0 secure :-1 s=READY pgs=1831 cs=0 l=1 rev1=1 rx=0x7f3218000b60 tx=0x7f3218007d40).ready entity=mon.0 client_cookie=adf22381302c05e7 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:28:06.151 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:06.134 7f3220e61700 1 -- 172.21.2.102:0/2353196042 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3218001010 con 0x7f3208003a60 2021-02-10T11:28:06.151 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:06.134 7f3220e61700 1 -- 172.21.2.102:0/2353196042 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32180013d0 con 0x7f3208003a60 2021-02-10T11:28:06.152 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:06.134 7f3220e61700 1 -- 172.21.2.102:0/2353196042 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f321800a7a0 con 0x7f3208003a60 2021-02-10T11:28:06.152 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:06.134 7f3224047700 1 -- 172.21.2.102:0/2353196042 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3208003a60 msgr2=0x7f3208005ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:06.152 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:06.134 7f3224047700 1 --2- 172.21.2.102:0/2353196042 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3208003a60 0x7f3208005ef0 secure :-1 s=READY pgs=1831 cs=0 l=1 rev1=1 rx=0x7f3218000b60 tx=0x7f3218007d40).stop 2021-02-10T11:28:06.152 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:06.135 7f3224047700 1 -- 172.21.2.102:0/2353196042 shutdown_connections 2021-02-10T11:28:06.152 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:06.135 7f3224047700 1 --2- 172.21.2.102:0/2353196042 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3208003a60 0x7f3208005ef0 unknown :-1 s=CLOSED pgs=1831 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:06.152 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:06.135 7f3224047700 1 -- 172.21.2.102:0/2353196042 >> 172.21.2.102:0/2353196042 conn(0x7f321c0ce420 msgr2=0x7f321c0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:28:06.152 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:06.135 7f3224047700 1 -- 172.21.2.102:0/2353196042 shutdown_connections 2021-02-10T11:28:06.152 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:06.135 7f3224047700 1 -- 172.21.2.102:0/2353196042 wait complete. 2021-02-10T11:28:06.154 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:06.150 7f3224047700 1 Processor -- start 2021-02-10T11:28:06.154 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:06.150 7f3224047700 1 -- start start 2021-02-10T11:28:06.154 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:06.150 7f3224047700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f321c085bc0 0x7f321c0863e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:06.154 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:06.150 7f3224047700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f321c0954c0 0x7f321c096300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:06.154 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:06.150 7f3224047700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f321c0958c0 0x7f321c096920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:06.154 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:06.150 7f3224047700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f321c095d80 con 0x7f321c0954c0 2021-02-10T11:28:06.154 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:06.150 7f3224047700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f321c0bfa60 con 0x7f321c085bc0 2021-02-10T11:28:06.154 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:06.150 7f3224047700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f321c085fc0 con 0x7f321c0958c0 2021-02-10T11:28:06.154 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:06.150 7f3221662700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f321c0954c0 0x7f321c096300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:06.154 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:06.150 7f3221662700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f321c0954c0 0x7f321c096300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:60102/0 (socket says 172.21.2.102:60102) 2021-02-10T11:28:06.155 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:06.151 7f3221662700 1 -- 172.21.2.102:0/4230231431 learned_addr learned my addr 172.21.2.102:0/4230231431 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:28:06.155 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:06.151 7f3222664700 1 --2- 172.21.2.102:0/4230231431 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f321c0958c0 0x7f321c096920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:06.155 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:06.151 7f3221e63700 1 --2- 172.21.2.102:0/4230231431 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f321c085bc0 0x7f321c0863e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:06.155 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:06.151 7f3221662700 1 -- 172.21.2.102:0/4230231431 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f321c0958c0 msgr2=0x7f321c096920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:06.155 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:06.151 7f3221662700 1 --2- 172.21.2.102:0/4230231431 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f321c0958c0 0x7f321c096920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:06.155 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:06.151 7f3221662700 1 -- 172.21.2.102:0/4230231431 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f321c085bc0 msgr2=0x7f321c0863e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:06.155 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:06.151 7f3221662700 1 --2- 172.21.2.102:0/4230231431 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f321c085bc0 0x7f321c0863e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:06.155 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:06.151 7f3221662700 1 -- 172.21.2.102:0/4230231431 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f321c086c10 con 0x7f321c0954c0 2021-02-10T11:28:06.155 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:06.151 7f3222664700 1 --2- 172.21.2.102:0/4230231431 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f321c0958c0 0x7f321c096920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:28:06.155 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:06.151 7f3221e63700 1 --2- 172.21.2.102:0/4230231431 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f321c085bc0 0x7f321c0863e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:28:06.155 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:06.151 7f3221662700 1 --2- 172.21.2.102:0/4230231431 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f321c0954c0 0x7f321c096300 secure :-1 s=READY pgs=1832 cs=0 l=1 rev1=1 rx=0x7f3210006b80 tx=0x7f321000b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:28:06.155 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:06.151 7f320e7fc700 1 -- 172.21.2.102:0/4230231431 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f321000bd80 con 0x7f321c0954c0 2021-02-10T11:28:06.155 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:06.151 7f320e7fc700 1 -- 172.21.2.102:0/4230231431 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3210006740 con 0x7f321c0954c0 2021-02-10T11:28:06.156 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:06.151 7f3224047700 1 -- 172.21.2.102:0/4230231431 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f321c086c10 con 0x7f321c0954c0 2021-02-10T11:28:06.156 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:06.151 7f320e7fc700 1 -- 172.21.2.102:0/4230231431 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f32100022b0 con 0x7f321c0954c0 2021-02-10T11:28:06.156 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:06.152 7f3224047700 1 -- 172.21.2.102:0/4230231431 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f321c086c10 con 0x7f321c0954c0 2021-02-10T11:28:06.156 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:06.152 7f320e7fc700 1 -- 172.21.2.102:0/4230231431 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f321000a000 con 0x7f321c0954c0 2021-02-10T11:28:06.156 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:06.152 7f320e7fc700 1 --2- 172.21.2.102:0/4230231431 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3204018730 0x7f320401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:06.156 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:06.152 7f320e7fc700 1 -- 172.21.2.102:0/4230231431 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f3210032610 con 0x7f321c0954c0 2021-02-10T11:28:06.157 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:06.152 7f3221e63700 1 --2- 172.21.2.102:0/4230231431 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3204018730 0x7f320401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:06.158 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:06.153 7f3221e63700 1 --2- 172.21.2.102:0/4230231431 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3204018730 0x7f320401ac10 crc :-1 s=READY pgs=824 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:28:06.181 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:06.182 7f3224047700 1 -- 172.21.2.102:0/4230231431 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f321c0fe1c0 con 0x7f321c0954c0 2021-02-10T11:28:06.184 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:06.185 7f320e7fc700 1 -- 172.21.2.102:0/4230231431 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f3210000c90 con 0x7f321c0954c0 2021-02-10T11:28:06.334 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:06.335 7f3224047700 1 -- 172.21.2.102:0/4230231431 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f321c0fe1c0 con 0x7f321c0954c0 2021-02-10T11:28:06.334 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:06.335 7f320e7fc700 1 -- 172.21.2.102:0/4230231431 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v22)=0 v22) v1 ==== 122+0+0 (secure 0 0 0) 0x7f321000a000 con 0x7f321c0954c0 2021-02-10T11:28:06.336 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:06.337 7f3224047700 1 -- 172.21.2.102:0/4230231431 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3204018730 msgr2=0x7f320401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:06.336 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:06.337 7f3224047700 1 --2- 172.21.2.102:0/4230231431 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3204018730 0x7f320401ac10 crc :-1 s=READY pgs=824 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:06.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:06.337 7f3224047700 1 -- 172.21.2.102:0/4230231431 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f321c0954c0 msgr2=0x7f321c096300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:06.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:06.337 7f3224047700 1 --2- 172.21.2.102:0/4230231431 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f321c0954c0 0x7f321c096300 secure :-1 s=READY pgs=1832 cs=0 l=1 rev1=1 rx=0x7f3210006b80 tx=0x7f321000b990).stop 2021-02-10T11:28:06.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:06.337 7f3224047700 1 -- 172.21.2.102:0/4230231431 shutdown_connections 2021-02-10T11:28:06.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:06.337 7f3224047700 1 --2- 172.21.2.102:0/4230231431 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f321c0958c0 0x7f321c096920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:06.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:06.337 7f3224047700 1 --2- 172.21.2.102:0/4230231431 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f321c0954c0 0x7f321c096300 unknown :-1 s=CLOSED pgs=1832 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:06.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:06.337 7f3224047700 1 --2- 172.21.2.102:0/4230231431 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3204018730 0x7f320401ac10 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:06.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:06.337 7f3224047700 1 --2- 172.21.2.102:0/4230231431 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f321c085bc0 0x7f321c0863e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:06.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:06.337 7f3224047700 1 -- 172.21.2.102:0/4230231431 >> 172.21.2.102:0/4230231431 conn(0x7f321c0bef00 msgr2=0x7f321c09d1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:28:06.339 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:06.338 7f3224047700 1 -- 172.21.2.102:0/4230231431 shutdown_connections 2021-02-10T11:28:06.339 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:06.338 7f3224047700 1 -- 172.21.2.102:0/4230231431 wait complete. 2021-02-10T11:28:06.345 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:28:06.345 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:28:06.345 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:28:06.681 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:28:06.681 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:28:06.681 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@gibba002.front.sepia.ceph.com... 2021-02-10T11:28:06.682 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-02-10T11:28:06.682 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:28:06.760 DEBUG:teuthology.orchestra.run.gibba002:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:28:06.764 DEBUG:teuthology.orchestra.run.gibba013:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:28:07.034 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba002.stderr:ceph-fuse[169377]: fuse finished with error 0 and tester_r 0 2021-02-10T11:28:07.036 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:28:07.036 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:28:07.037 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:28:07.372 INFO:teuthology.orchestra.run.gibba002.stdout:ext2/ext3 2021-02-10T11:28:07.373 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2021-02-10T11:28:07.373 INFO:teuthology.orchestra.run:waiting for 900 2021-02-10T11:28:07.373 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:28:07.373 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:28:07.708 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:28:07.709 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:28:07.709 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:28:08.045 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:28:08.045 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2021-02-10T11:28:08.045 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@gibba002.front.sepia.ceph.com... 2021-02-10T11:28:08.046 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-02-10T11:28:08.046 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:28:08.398 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba002.stderr:ceph-fuse[170017]: fuse finished with error 0 and tester_r 0 2021-02-10T11:28:08.399 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:28:08.400 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:28:08.400 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:28:08.736 INFO:teuthology.orchestra.run.gibba002.stdout:ext2/ext3 2021-02-10T11:28:08.736 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2021-02-10T11:28:08.737 INFO:teuthology.orchestra.run:waiting for 900 2021-02-10T11:28:08.737 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:28:08.737 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:28:09.072 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:28:09.498 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:09.483 7f09526fb700 1 Processor -- start 2021-02-10T11:28:09.499 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:09.483 7f09526fb700 1 -- start start 2021-02-10T11:28:09.499 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:09.484 7f09526fb700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f094c1b0470 0x7f094c1b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:09.499 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:09.484 7f09526fb700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f094c0ef610 0x7f094c0efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:09.499 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:09.484 7f09526fb700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f094c0f0030 0x7f094c0f0430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:09.499 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:09.484 7f09526fb700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f094c0e5ea0 con 0x7f094c1b0470 2021-02-10T11:28:09.499 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:09.484 7f09526fb700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f094c0f0b50 con 0x7f094c0ef610 2021-02-10T11:28:09.499 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:09.484 7f09526fb700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f094c0e52e0 con 0x7f094c0f0030 2021-02-10T11:28:09.499 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:09.484 7f094bfff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f094c1b0470 0x7f094c1b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:09.499 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:09.484 7f094bfff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f094c1b0470 0x7f094c1b0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:60110/0 (socket says 172.21.2.102:60110) 2021-02-10T11:28:09.499 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:09.484 7f094bfff700 1 -- 172.21.2.102:0/1139315613 learned_addr learned my addr 172.21.2.102:0/1139315613 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:28:09.499 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:09.484 7f0950d18700 1 --2- 172.21.2.102:0/1139315613 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f094c0f0030 0x7f094c0f0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:09.499 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:09.484 7f094b7fe700 1 --2- 172.21.2.102:0/1139315613 >> v2:172.21.2.113:3300/0 conn(0x7f094c0ef610 0x7f094c0efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:09.500 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:09.484 7f094bfff700 1 -- 172.21.2.102:0/1139315613 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f094c0f0030 msgr2=0x7f094c0f0430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:28:09.500 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:09.484 7f094bfff700 1 --2- 172.21.2.102:0/1139315613 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f094c0f0030 0x7f094c0f0430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:09.500 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:09.484 7f094bfff700 1 -- 172.21.2.102:0/1139315613 >> v2:172.21.2.113:3300/0 conn(0x7f094c0ef610 msgr2=0x7f094c0efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:28:09.500 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:09.484 7f094bfff700 1 --2- 172.21.2.102:0/1139315613 >> v2:172.21.2.113:3300/0 conn(0x7f094c0ef610 0x7f094c0efa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:09.500 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:09.484 7f094bfff700 1 -- 172.21.2.102:0/1139315613 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f094c0f10d0 con 0x7f094c1b0470 2021-02-10T11:28:09.500 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:09.484 7f094b7fe700 1 --2- 172.21.2.102:0/1139315613 >> v2:172.21.2.113:3300/0 conn(0x7f094c0ef610 0x7f094c0efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:28:09.500 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:09.484 7f094bfff700 1 --2- 172.21.2.102:0/1139315613 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f094c1b0470 0x7f094c1b0870 secure :-1 s=READY pgs=1833 cs=0 l=1 rev1=1 rx=0x7f093c006700 tx=0x7f093c009b80).ready entity=mon.0 client_cookie=8db67479545a3487 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:28:09.500 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:09.484 7f094affd700 1 -- 172.21.2.102:0/1139315613 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f093c00c0c0 con 0x7f094c1b0470 2021-02-10T11:28:09.500 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:09.484 7f094affd700 1 -- 172.21.2.102:0/1139315613 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f093c00c340 con 0x7f094c1b0470 2021-02-10T11:28:09.500 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:09.484 7f094affd700 1 -- 172.21.2.102:0/1139315613 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f093c00c860 con 0x7f094c1b0470 2021-02-10T11:28:09.500 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:09.484 7f09526fb700 1 -- 172.21.2.102:0/1139315613 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f094c1b0470 msgr2=0x7f094c1b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:09.500 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:09.484 7f09526fb700 1 --2- 172.21.2.102:0/1139315613 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f094c1b0470 0x7f094c1b0870 secure :-1 s=READY pgs=1833 cs=0 l=1 rev1=1 rx=0x7f093c006700 tx=0x7f093c009b80).stop 2021-02-10T11:28:09.500 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:09.485 7f09526fb700 1 -- 172.21.2.102:0/1139315613 shutdown_connections 2021-02-10T11:28:09.501 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:09.485 7f09526fb700 1 --2- 172.21.2.102:0/1139315613 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f094c0f0030 0x7f094c0f0430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:09.501 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:09.485 7f09526fb700 1 --2- 172.21.2.102:0/1139315613 >> v2:172.21.2.113:3300/0 conn(0x7f094c0ef610 0x7f094c0efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:09.501 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:09.485 7f09526fb700 1 --2- 172.21.2.102:0/1139315613 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f094c1b0470 0x7f094c1b0870 unknown :-1 s=CLOSED pgs=1833 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:09.501 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:09.485 7f09526fb700 1 -- 172.21.2.102:0/1139315613 >> 172.21.2.102:0/1139315613 conn(0x7f094c0ce3e0 msgr2=0x7f094c0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:28:09.501 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:09.485 7f09526fb700 1 -- 172.21.2.102:0/1139315613 shutdown_connections 2021-02-10T11:28:09.501 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:09.485 7f09526fb700 1 -- 172.21.2.102:0/1139315613 wait complete. 2021-02-10T11:28:09.502 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:09.500 7f09526fb700 1 Processor -- start 2021-02-10T11:28:09.502 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:09.500 7f09526fb700 1 -- start start 2021-02-10T11:28:09.503 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:09.500 7f09526fb700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f094c0b11f0 0x7f094c189d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:09.503 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:09.500 7f09526fb700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f094c18a390 0x7f094c18a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:09.503 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:09.500 7f09526fb700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f094c18adb0 0x7f094c187e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:09.503 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:09.500 7f09526fb700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f094c0b15f0 con 0x7f094c0b11f0 2021-02-10T11:28:09.503 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:09.500 7f09526fb700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f094c188440 con 0x7f094c18adb0 2021-02-10T11:28:09.503 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:09.500 7f09526fb700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f094c188660 con 0x7f094c18a390 2021-02-10T11:28:09.503 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:09.500 7f094bfff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f094c0b11f0 0x7f094c189d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:09.503 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:09.500 7f094bfff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f094c0b11f0 0x7f094c189d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:60118/0 (socket says 172.21.2.102:60118) 2021-02-10T11:28:09.503 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:09.500 7f094bfff700 1 -- 172.21.2.102:0/90728799 learned_addr learned my addr 172.21.2.102:0/90728799 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:28:09.503 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:09.501 7f094bfff700 1 -- 172.21.2.102:0/90728799 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f094c18a390 msgr2=0x7f094c18a790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:09.503 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:09.501 7f094bfff700 1 --2- 172.21.2.102:0/90728799 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f094c18a390 0x7f094c18a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:28:09.503 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:09.501 7f094bfff700 1 -- 172.21.2.102:0/90728799 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f094c18adb0 msgr2=0x7f094c187e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:09.503 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:09.501 7f094bfff700 1 --2- 172.21.2.102:0/90728799 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f094c18adb0 0x7f094c187e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:28:09.504 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:09.501 7f094bfff700 1 -- 172.21.2.102:0/90728799 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f094c188880 con 0x7f094c0b11f0 2021-02-10T11:28:09.504 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:09.501 7f094bfff700 1 --2- 172.21.2.102:0/90728799 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f094c0b11f0 0x7f094c189d70 secure :-1 s=READY pgs=1834 cs=0 l=1 rev1=1 rx=0x7f093c00b5e0 tx=0x7f093c00b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:28:09.504 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:09.501 7f0948ff9700 1 -- 172.21.2.102:0/90728799 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f093c00bd70 con 0x7f094c0b11f0 2021-02-10T11:28:09.504 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:09.501 7f0948ff9700 1 -- 172.21.2.102:0/90728799 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f093c002030 con 0x7f094c0b11f0 2021-02-10T11:28:09.504 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:09.501 7f0948ff9700 1 -- 172.21.2.102:0/90728799 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f093c002590 con 0x7f094c0b11f0 2021-02-10T11:28:09.504 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:09.501 7f09526fb700 1 -- 172.21.2.102:0/90728799 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f094c188880 con 0x7f094c0b11f0 2021-02-10T11:28:09.504 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:09.501 7f09526fb700 1 -- 172.21.2.102:0/90728799 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f094c188880 con 0x7f094c0b11f0 2021-02-10T11:28:09.504 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:09.502 7f0948ff9700 1 -- 172.21.2.102:0/90728799 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f093c000c90 con 0x7f094c0b11f0 2021-02-10T11:28:09.504 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:09.502 7f0948ff9700 1 --2- 172.21.2.102:0/90728799 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f092c0186c0 0x7f092c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:09.504 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:09.502 7f0948ff9700 1 -- 172.21.2.102:0/90728799 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f093c0325f0 con 0x7f094c0b11f0 2021-02-10T11:28:09.504 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:09.502 7f094b7fe700 1 --2- 172.21.2.102:0/90728799 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f092c0186c0 0x7f092c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:09.504 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:09.502 7f094b7fe700 1 --2- 172.21.2.102:0/90728799 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f092c0186c0 0x7f092c01abc0 crc :-1 s=READY pgs=825 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:28:09.532 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:09.533 7f09526fb700 1 -- 172.21.2.102:0/90728799 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f094c189840 con 0x7f094c0b11f0 2021-02-10T11:28:09.535 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:09.536 7f0948ff9700 1 -- 172.21.2.102:0/90728799 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f093c01f570 con 0x7f094c0b11f0 2021-02-10T11:28:09.663 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:09.664 7f09526fb700 1 -- 172.21.2.102:0/90728799 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f094c189840 con 0x7f094c0b11f0 2021-02-10T11:28:09.663 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:09.664 7f0948ff9700 1 -- 172.21.2.102:0/90728799 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+2101 (secure 0 0 0) 0x7f093c00bd70 con 0x7f094c0b11f0 2021-02-10T11:28:09.664 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:28:09.664 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":66,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6110,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.102:6835/1471871278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6834","nonce":1471871278},{"type":"v1","addr":"172.21.2.102:6835","nonce":1471871278}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":61}],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:27:41.479375","modified":"2021-02-10 11:27:43.490605","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6107},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6107":{"gid":6107,"name":"b","rank":0,"incarnation":62,"state":"up:active","state_seq":3,"addr":"172.21.2.113:6833/4220238062","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6832","nonce":4220238062},{"type":"v1","addr":"172.21.2.113:6833","nonce":4220238062}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":8}]} 2021-02-10T11:28:09.664 INFO:teuthology.orchestra.run.gibba002.stderr:dumped fsmap epoch 66 2021-02-10T11:28:09.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:09.666 7f09526fb700 1 -- 172.21.2.102:0/90728799 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f092c0186c0 msgr2=0x7f092c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:09.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:09.666 7f09526fb700 1 --2- 172.21.2.102:0/90728799 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f092c0186c0 0x7f092c01abc0 crc :-1 s=READY pgs=825 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:09.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:09.666 7f09526fb700 1 -- 172.21.2.102:0/90728799 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f094c0b11f0 msgr2=0x7f094c189d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:09.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:09.666 7f09526fb700 1 --2- 172.21.2.102:0/90728799 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f094c0b11f0 0x7f094c189d70 secure :-1 s=READY pgs=1834 cs=0 l=1 rev1=1 rx=0x7f093c00b5e0 tx=0x7f093c00b980).stop 2021-02-10T11:28:09.666 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:09.666 7f09526fb700 1 -- 172.21.2.102:0/90728799 shutdown_connections 2021-02-10T11:28:09.666 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:09.666 7f09526fb700 1 --2- 172.21.2.102:0/90728799 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f092c0186c0 0x7f092c01abc0 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:09.666 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:09.666 7f09526fb700 1 --2- 172.21.2.102:0/90728799 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f094c18adb0 0x7f094c187e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:28:09.666 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:09.666 7f09526fb700 1 --2- 172.21.2.102:0/90728799 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f094c18a390 0x7f094c18a790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:28:09.666 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:09.666 7f09526fb700 1 --2- 172.21.2.102:0/90728799 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f094c0b11f0 0x7f094c189d70 unknown :-1 s=CLOSED pgs=1834 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:09.666 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:09.666 7f09526fb700 1 -- 172.21.2.102:0/90728799 >> 172.21.2.102:0/90728799 conn(0x7f094c0a78c0 msgr2=0x7f094c0b1f70 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:28:09.666 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:09.666 7f09526fb700 1 -- 172.21.2.102:0/90728799 shutdown_connections 2021-02-10T11:28:09.666 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:09.667 7f09526fb700 1 -- 172.21.2.102:0/90728799 wait complete. 2021-02-10T11:28:09.674 DEBUG:teuthology.orchestra.run.gibba013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok session ls 2021-02-10T11:28:10.071 INFO:teuthology.orchestra.run.gibba013.stdout:[] 2021-02-10T11:28:10.076 INFO:tasks.cephfs.filesystem:_json_asok output: [] 2021-02-10T11:28:10.076 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000713508 2021-02-10T11:28:10.508 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:10.492 7f42f853f700 1 Processor -- start 2021-02-10T11:28:10.509 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:10.492 7f42f853f700 1 -- start start 2021-02-10T11:28:10.509 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:10.492 7f42f853f700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f42f00d1c70 0x7f42f00cef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:10.509 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:10.492 7f42f853f700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f42f00cf580 0x7f42f00cf980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:10.509 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:10.492 7f42f853f700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42f00fe1c0 con 0x7f42f00d1870 2021-02-10T11:28:10.509 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:10.492 7f42f853f700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42f01b0cb0 con 0x7f42f00cf580 2021-02-10T11:28:10.509 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:10.492 7f42f853f700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42f0109a60 con 0x7f42f00d1c70 2021-02-10T11:28:10.509 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:10.492 7f42f635b700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f42f00d1870 0x7f42f01b04c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:52346/0 (socket says 172.21.2.102:52346) 2021-02-10T11:28:10.509 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:10.492 7f42f635b700 1 -- 172.21.2.102:0/1390132132 learned_addr learned my addr 172.21.2.102:0/1390132132 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:28:10.509 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:10.492 7f42f5b5a700 1 --2- 172.21.2.102:0/1390132132 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f42f00d1c70 0x7f42f00cef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:10.509 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:10.493 7f42f6b5c700 1 --2- 172.21.2.102:0/1390132132 >> v2:172.21.2.113:3300/0 conn(0x7f42f00cf580 0x7f42f00cf980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:10.509 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:10.493 7f42f5359700 1 -- 172.21.2.102:0/1390132132 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2125925707 0 0) 0x7f42e4001180 con 0x7f42f00d1870 2021-02-10T11:28:10.509 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:10.493 7f42f5359700 1 -- 172.21.2.102:0/1390132132 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42dc002fe0 con 0x7f42f00d1870 2021-02-10T11:28:10.510 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:10.493 7f42f5b5a700 1 -- 172.21.2.102:0/1390132132 >> v2:172.21.2.113:3300/0 conn(0x7f42f00cf580 msgr2=0x7f42f00cf980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:28:10.510 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:10.493 7f42f5b5a700 1 --2- 172.21.2.102:0/1390132132 >> v2:172.21.2.113:3300/0 conn(0x7f42f00cf580 0x7f42f00cf980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:10.510 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:10.493 7f42f5b5a700 1 -- 172.21.2.102:0/1390132132 >> v1:172.21.2.102:6789/0 conn(0x7f42f00d1870 legacy=0x7f42f01b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:10.510 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:10.493 7f42f5359700 1 -- 172.21.2.102:0/1390132132 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1661760742 0 0) 0x7f42e4001180 con 0x7f42f00d1870 2021-02-10T11:28:10.510 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:10.493 7f42f5b5a700 1 -- 172.21.2.102:0/1390132132 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42f0088070 con 0x7f42f00d1c70 2021-02-10T11:28:10.510 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:10.493 7f42f5b5a700 1 --2- 172.21.2.102:0/1390132132 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f42f00d1c70 0x7f42f00cef60 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7f42ec006650 tx=0x7f42ec009a80).ready entity=mon.2 client_cookie=1ea3b726220c05b6 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:28:10.510 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:10.493 7f42f5359700 1 -- 172.21.2.102:0/1390132132 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f42ec00c0c0 con 0x7f42f00d1c70 2021-02-10T11:28:10.510 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:10.493 7f42f5359700 1 -- 172.21.2.102:0/1390132132 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42ec00c340 con 0x7f42f00d1c70 2021-02-10T11:28:10.510 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:10.493 7f42f5359700 1 -- 172.21.2.102:0/1390132132 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f42ec00c860 con 0x7f42f00d1c70 2021-02-10T11:28:10.510 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:10.493 7f42f853f700 1 -- 172.21.2.102:0/1390132132 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f42f00d1c70 msgr2=0x7f42f00cef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:10.510 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:10.493 7f42f853f700 1 --2- 172.21.2.102:0/1390132132 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f42f00d1c70 0x7f42f00cef60 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7f42ec006650 tx=0x7f42ec009a80).stop 2021-02-10T11:28:10.510 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:10.494 7f42f853f700 1 -- 172.21.2.102:0/1390132132 shutdown_connections 2021-02-10T11:28:10.510 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:10.494 7f42f853f700 1 --2- 172.21.2.102:0/1390132132 >> v2:172.21.2.113:3300/0 conn(0x7f42f00cf580 0x7f42f00cf980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:10.510 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:10.494 7f42f853f700 1 --2- 172.21.2.102:0/1390132132 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f42f00d1c70 0x7f42f00cef60 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:10.511 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:10.494 7f42f853f700 1 -- 172.21.2.102:0/1390132132 >> 172.21.2.102:0/1390132132 conn(0x7f42f00ce420 msgr2=0x7f42f00d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:28:10.511 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:10.494 7f42f853f700 1 -- 172.21.2.102:0/1390132132 shutdown_connections 2021-02-10T11:28:10.511 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:10.494 7f42f853f700 1 -- 172.21.2.102:0/1390132132 wait complete. 2021-02-10T11:28:10.511 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:10.509 7f42f853f700 1 Processor -- start 2021-02-10T11:28:10.511 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:10.509 7f42f853f700 1 -- start start 2021-02-10T11:28:10.511 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:10.509 7f42f853f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f42f009bcc0 0x7f42f009c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:10.511 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:10.509 7f42f853f700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f42f00a0820 0x7f42f00a0c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:10.511 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:10.509 7f42f853f700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f42f00a1240 0x7f42f00858d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:10.511 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:10.509 7f42f853f700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42f00a66b0 con 0x7f42f009bcc0 2021-02-10T11:28:10.511 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:10.509 7f42f853f700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42f01a49b0 con 0x7f42f00a1240 2021-02-10T11:28:10.511 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:10.509 7f42f853f700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42f01a4b90 con 0x7f42f00a0820 2021-02-10T11:28:10.511 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:10.509 7f42f635b700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f42f009bcc0 0x7f42f009c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:10.511 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:10.509 7f42f635b700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f42f009bcc0 0x7f42f009c0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:60128/0 (socket says 172.21.2.102:60128) 2021-02-10T11:28:10.512 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:10.509 7f42f635b700 1 -- 172.21.2.102:0/4208845744 learned_addr learned my addr 172.21.2.102:0/4208845744 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:28:10.513 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:10.510 7f42f635b700 1 -- 172.21.2.102:0/4208845744 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f42f00a0820 msgr2=0x7f42f00a0c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:10.513 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:10.510 7f42f635b700 1 --2- 172.21.2.102:0/4208845744 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f42f00a0820 0x7f42f00a0c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:28:10.513 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:10.510 7f42f635b700 1 -- 172.21.2.102:0/4208845744 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f42f00a1240 msgr2=0x7f42f00858d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:10.514 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:10.510 7f42f635b700 1 --2- 172.21.2.102:0/4208845744 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f42f00a1240 0x7f42f00858d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:28:10.514 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:10.510 7f42f635b700 1 -- 172.21.2.102:0/4208845744 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42f00a1640 con 0x7f42f009bcc0 2021-02-10T11:28:10.514 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:10.510 7f42f635b700 1 --2- 172.21.2.102:0/4208845744 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f42f009bcc0 0x7f42f009c0c0 secure :-1 s=READY pgs=1836 cs=0 l=1 rev1=1 rx=0x7f42e4001ec0 tx=0x7f42e400a930).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:28:10.514 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:10.510 7f42e2ffd700 1 -- 172.21.2.102:0/4208845744 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f42e40070d0 con 0x7f42f009bcc0 2021-02-10T11:28:10.514 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:10.510 7f42e2ffd700 1 -- 172.21.2.102:0/4208845744 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42e4001180 con 0x7f42f009bcc0 2021-02-10T11:28:10.514 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:10.510 7f42e2ffd700 1 -- 172.21.2.102:0/4208845744 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f42e4004830 con 0x7f42f009bcc0 2021-02-10T11:28:10.514 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:10.510 7f42f853f700 1 -- 172.21.2.102:0/4208845744 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42f00a1640 con 0x7f42f009bcc0 2021-02-10T11:28:10.514 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:10.510 7f42f853f700 1 -- 172.21.2.102:0/4208845744 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42f00861e0 con 0x7f42f009bcc0 2021-02-10T11:28:10.514 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:10.511 7f42e2ffd700 1 -- 172.21.2.102:0/4208845744 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f42e40047c0 con 0x7f42f009bcc0 2021-02-10T11:28:10.514 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:10.511 7f42e2ffd700 1 --2- 172.21.2.102:0/4208845744 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f42d80186c0 0x7f42d801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:10.514 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:10.511 7f42e2ffd700 1 -- 172.21.2.102:0/4208845744 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f42e4032420 con 0x7f42f009bcc0 2021-02-10T11:28:10.514 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:10.511 7f42f5b5a700 1 --2- 172.21.2.102:0/4208845744 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f42d80186c0 0x7f42d801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:10.514 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:10.511 7f42f5b5a700 1 --2- 172.21.2.102:0/4208845744 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f42d80186c0 0x7f42d801abc0 crc :-1 s=READY pgs=826 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:28:10.541 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:10.542 7f42f853f700 1 -- 172.21.2.102:0/4208845744 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42f0086a70 con 0x7f42f009bcc0 2021-02-10T11:28:10.544 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:10.545 7f42e2ffd700 1 -- 172.21.2.102:0/4208845744 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f42e40045b0 con 0x7f42f009bcc0 2021-02-10T11:28:10.670 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:10.671 7f42f853f700 1 -- 172.21.2.102:0/4208845744 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000713508", "target": ["mgr", ""]}) v1 -- 0x7f42f00861e0 con 0x7f42d80186c0 2021-02-10T11:28:10.674 INFO:tasks.ceph.mgr.y.gibba002.stderr:2021-02-10 11:28:10.675 7fd94bc53700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:28:10.674 INFO:tasks.ceph.mgr.y.gibba002.stderr:2021-02-10 11:28:10.675 7fd94bc53700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:28:10.675 INFO:tasks.ceph.mgr.y.gibba002.stderr:2021-02-10 11:28:10.675 7fd94bc53700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:28:10.675 INFO:tasks.ceph.mgr.y.gibba002.stderr:2021-02-10 11:28:10.675 7fd94bc53700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:28:10.675 INFO:tasks.ceph.mgr.y.gibba002.stderr:2021-02-10 11:28:10.675 7fd94bc53700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:28:10.706 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:10.707 7f42e2ffd700 1 -- 172.21.2.102:0/4208845744 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f42ec000940 con 0x7f42d80186c0 2021-02-10T11:28:10.736 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:10.736 7f42f853f700 1 -- 172.21.2.102:0/4208845744 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f42d80186c0 msgr2=0x7f42d801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:10.736 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:10.736 7f42f853f700 1 --2- 172.21.2.102:0/4208845744 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f42d80186c0 0x7f42d801abc0 crc :-1 s=READY pgs=826 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:10.736 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:10.736 7f42f853f700 1 -- 172.21.2.102:0/4208845744 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f42f009bcc0 msgr2=0x7f42f009c0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:10.736 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:10.736 7f42f853f700 1 --2- 172.21.2.102:0/4208845744 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f42f009bcc0 0x7f42f009c0c0 secure :-1 s=READY pgs=1836 cs=0 l=1 rev1=1 rx=0x7f42e4001ec0 tx=0x7f42e400a930).stop 2021-02-10T11:28:10.736 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:10.736 7f42f853f700 1 -- 172.21.2.102:0/4208845744 shutdown_connections 2021-02-10T11:28:10.736 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:10.736 7f42f853f700 1 --2- 172.21.2.102:0/4208845744 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f42d80186c0 0x7f42d801abc0 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:10.736 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:10.736 7f42f853f700 1 --2- 172.21.2.102:0/4208845744 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f42f00a1240 0x7f42f00858d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:28:10.736 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:10.736 7f42f853f700 1 --2- 172.21.2.102:0/4208845744 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f42f00a0820 0x7f42f00a0c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:28:10.736 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:10.736 7f42f853f700 1 --2- 172.21.2.102:0/4208845744 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f42f009bcc0 0x7f42f009c0c0 unknown :-1 s=CLOSED pgs=1836 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:10.736 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:10.736 7f42f853f700 1 -- 172.21.2.102:0/4208845744 >> 172.21.2.102:0/4208845744 conn(0x7f42f00bef00 msgr2=0x7f42f009d1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:28:10.736 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:10.737 7f42f853f700 1 -- 172.21.2.102:0/4208845744 shutdown_connections 2021-02-10T11:28:10.737 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:10.737 7f42f853f700 1 -- 172.21.2.102:0/4208845744 wait complete. 2021-02-10T11:28:10.744 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:28:11.170 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:11.154 7fae06a4b700 1 Processor -- start 2021-02-10T11:28:11.170 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:11.154 7fae06a4b700 1 -- start start 2021-02-10T11:28:11.170 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:11.155 7fae06a4b700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fae000d0340 0x7fae000ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:11.170 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:11.155 7fae06a4b700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fae000d1c70 0x7fae000efc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:11.170 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:11.155 7fae06a4b700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae000f0b10 con 0x7fae000d1870 2021-02-10T11:28:11.170 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:11.155 7fae06a4b700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae000e5ed0 con 0x7fae000d0340 2021-02-10T11:28:11.170 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:11.155 7fae06a4b700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae001b0be0 con 0x7fae000d1c70 2021-02-10T11:28:11.170 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:11.155 7fadfffff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fae000d0340 0x7fae000ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:11.170 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:11.155 7fadfffff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fae000d0340 0x7fae000ef640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:60154/0 (socket says 172.21.2.102:60154) 2021-02-10T11:28:11.171 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:11.155 7fadfffff700 1 -- 172.21.2.102:0/2972981277 learned_addr learned my addr 172.21.2.102:0/2972981277 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:28:11.171 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:11.155 7fadfffff700 1 -- 172.21.2.102:0/2972981277 >> v2:172.21.2.113:3300/0 conn(0x7fae000d1c70 msgr2=0x7fae000efc60 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:28:11.171 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:11.155 7fadfffff700 1 --2- 172.21.2.102:0/2972981277 >> v2:172.21.2.113:3300/0 conn(0x7fae000d1c70 0x7fae000efc60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:28:11.171 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:11.155 7fadfffff700 1 -- 172.21.2.102:0/2972981277 >> v1:172.21.2.102:6789/0 conn(0x7fae000d1870 legacy=0x7fae001b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:11.171 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:11.155 7fadfffff700 1 -- 172.21.2.102:0/2972981277 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae000f20a0 con 0x7fae000d0340 2021-02-10T11:28:11.171 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:11.156 7fadfffff700 1 --2- 172.21.2.102:0/2972981277 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fae000d0340 0x7fae000ef640 secure :-1 s=READY pgs=1840 cs=0 l=1 rev1=1 rx=0x7fadf0006a80 tx=0x7fadf0009e80).ready entity=mon.0 client_cookie=3aeb23d85b50e5d1 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:28:11.172 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:11.156 7fadff7fe700 1 -- 172.21.2.102:0/2972981277 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fadf000b510 con 0x7fae000d0340 2021-02-10T11:28:11.172 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:11.156 7fadff7fe700 1 -- 172.21.2.102:0/2972981277 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadf000b7d0 con 0x7fae000d0340 2021-02-10T11:28:11.172 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:11.156 7fadff7fe700 1 -- 172.21.2.102:0/2972981277 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fadf000bcb0 con 0x7fae000d0340 2021-02-10T11:28:11.172 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:11.156 7fae06a4b700 1 -- 172.21.2.102:0/2972981277 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fae000d0340 msgr2=0x7fae000ef640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:11.172 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:11.156 7fae06a4b700 1 --2- 172.21.2.102:0/2972981277 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fae000d0340 0x7fae000ef640 secure :-1 s=READY pgs=1840 cs=0 l=1 rev1=1 rx=0x7fadf0006a80 tx=0x7fadf0009e80).stop 2021-02-10T11:28:11.172 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:11.156 7fae06a4b700 1 -- 172.21.2.102:0/2972981277 shutdown_connections 2021-02-10T11:28:11.173 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:11.156 7fae06a4b700 1 --2- 172.21.2.102:0/2972981277 >> v2:172.21.2.113:3300/0 conn(0x7fae000d1c70 0x7fae000efc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:28:11.173 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:11.156 7fae06a4b700 1 --2- 172.21.2.102:0/2972981277 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fae000d0340 0x7fae000ef640 unknown :-1 s=CLOSED pgs=1840 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:11.173 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:11.156 7fae06a4b700 1 -- 172.21.2.102:0/2972981277 >> 172.21.2.102:0/2972981277 conn(0x7fae000ce450 msgr2=0x7fae000d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:28:11.173 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:11.157 7fae06a4b700 1 -- 172.21.2.102:0/2972981277 shutdown_connections 2021-02-10T11:28:11.173 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:11.157 7fae06a4b700 1 -- 172.21.2.102:0/2972981277 wait complete. 2021-02-10T11:28:11.176 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:11.171 7fae06a4b700 1 Processor -- start 2021-02-10T11:28:11.176 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:11.171 7fae06a4b700 1 -- start start 2021-02-10T11:28:11.176 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:11.172 7fae06a4b700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fae0018b9d0 0x7fae0019d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:11.176 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:11.172 7fae06a4b700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fae0018bdd0 0x7fae000f22b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:11.176 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:11.172 7fae06a4b700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fae000f28d0 0x7fae00091580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:11.176 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:11.172 7fae06a4b700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae0019d6f0 con 0x7fae0018bdd0 2021-02-10T11:28:11.177 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:11.172 7fae06a4b700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae001afb10 con 0x7fae000f28d0 2021-02-10T11:28:11.177 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:11.172 7fae06a4b700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae001affb0 con 0x7fae0018b9d0 2021-02-10T11:28:11.177 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:11.172 7fadfffff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fae0018bdd0 0x7fae000f22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:11.178 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:11.172 7fadfffff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fae0018bdd0 0x7fae000f22b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:60162/0 (socket says 172.21.2.102:60162) 2021-02-10T11:28:11.178 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:11.172 7fadfffff700 1 -- 172.21.2.102:0/2477668696 learned_addr learned my addr 172.21.2.102:0/2477668696 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:28:11.178 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:11.172 7fadfffff700 1 -- 172.21.2.102:0/2477668696 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fae0018b9d0 msgr2=0x7fae0019d0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:11.178 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:11.172 7fadfffff700 1 --2- 172.21.2.102:0/2477668696 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fae0018b9d0 0x7fae0019d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:28:11.178 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:11.172 7fadfffff700 1 -- 172.21.2.102:0/2477668696 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fae000f28d0 msgr2=0x7fae00091580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:11.178 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:11.172 7fadfffff700 1 --2- 172.21.2.102:0/2477668696 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fae000f28d0 0x7fae00091580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:28:11.178 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:11.172 7fadfffff700 1 -- 172.21.2.102:0/2477668696 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae00091ba0 con 0x7fae0018bdd0 2021-02-10T11:28:11.178 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:11.172 7fadfffff700 1 --2- 172.21.2.102:0/2477668696 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fae0018bdd0 0x7fae000f22b0 secure :-1 s=READY pgs=1841 cs=0 l=1 rev1=1 rx=0x7fadf000b910 tx=0x7fadf000bcb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:28:11.178 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:11.173 7fadfd7fa700 1 -- 172.21.2.102:0/2477668696 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fadf0006030 con 0x7fae0018bdd0 2021-02-10T11:28:11.178 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:11.173 7fadfd7fa700 1 -- 172.21.2.102:0/2477668696 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadf00062b0 con 0x7fae0018bdd0 2021-02-10T11:28:11.179 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:11.173 7fadfd7fa700 1 -- 172.21.2.102:0/2477668696 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fadf0005660 con 0x7fae0018bdd0 2021-02-10T11:28:11.179 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:11.173 7fae06a4b700 1 -- 172.21.2.102:0/2477668696 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae00091ba0 con 0x7fae0018bdd0 2021-02-10T11:28:11.179 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:11.173 7fae06a4b700 1 -- 172.21.2.102:0/2477668696 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae00092130 con 0x7fae0018bdd0 2021-02-10T11:28:11.179 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:11.173 7fadfd7fa700 1 -- 172.21.2.102:0/2477668696 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fadf0005660 con 0x7fae0018bdd0 2021-02-10T11:28:11.179 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:11.173 7fadfd7fa700 1 --2- 172.21.2.102:0/2477668696 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fade401cad0 0x7fade401eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:11.179 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:11.173 7fadfd7fa700 1 -- 172.21.2.102:0/2477668696 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fadf0032940 con 0x7fae0018bdd0 2021-02-10T11:28:11.179 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:11.174 7fae04867700 1 --2- 172.21.2.102:0/2477668696 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fade401cad0 0x7fade401eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:11.179 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:11.174 7fae04867700 1 --2- 172.21.2.102:0/2477668696 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fade401cad0 0x7fade401eff0 crc :-1 s=READY pgs=827 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:28:11.204 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:11.205 7fae06a4b700 1 -- 172.21.2.102:0/2477668696 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae000ee0e0 con 0x7fae0018bdd0 2021-02-10T11:28:11.206 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:11.207 7fadfd7fa700 1 -- 172.21.2.102:0/2477668696 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fadf0022c10 con 0x7fae0018bdd0 2021-02-10T11:28:11.334 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:11.335 7fae06a4b700 1 -- 172.21.2.102:0/2477668696 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fae000ce450 con 0x7fae0018bdd0 2021-02-10T11:28:11.335 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:11.336 7fadfd7fa700 1 -- 172.21.2.102:0/2477668696 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+2101 (secure 0 0 0) 0x7fadf0022c10 con 0x7fae0018bdd0 2021-02-10T11:28:11.335 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:28:11.335 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":66,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6110,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.102:6835/1471871278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6834","nonce":1471871278},{"type":"v1","addr":"172.21.2.102:6835","nonce":1471871278}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":61}],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:27:41.479375","modified":"2021-02-10 11:27:43.490605","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6107},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6107":{"gid":6107,"name":"b","rank":0,"incarnation":62,"state":"up:active","state_seq":3,"addr":"172.21.2.113:6833/4220238062","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6832","nonce":4220238062},{"type":"v1","addr":"172.21.2.113:6833","nonce":4220238062}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":8}]} 2021-02-10T11:28:11.336 INFO:teuthology.orchestra.run.gibba002.stderr:dumped fsmap epoch 66 2021-02-10T11:28:11.336 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:11.337 7fae06a4b700 1 -- 172.21.2.102:0/2477668696 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fade401cad0 msgr2=0x7fade401eff0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:11.336 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:11.337 7fae06a4b700 1 --2- 172.21.2.102:0/2477668696 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fade401cad0 0x7fade401eff0 crc :-1 s=READY pgs=827 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:11.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:11.337 7fae06a4b700 1 -- 172.21.2.102:0/2477668696 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fae0018bdd0 msgr2=0x7fae000f22b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:11.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:11.337 7fae06a4b700 1 --2- 172.21.2.102:0/2477668696 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fae0018bdd0 0x7fae000f22b0 secure :-1 s=READY pgs=1841 cs=0 l=1 rev1=1 rx=0x7fadf000b910 tx=0x7fadf000bcb0).stop 2021-02-10T11:28:11.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:11.338 7fae06a4b700 1 -- 172.21.2.102:0/2477668696 shutdown_connections 2021-02-10T11:28:11.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:11.338 7fae06a4b700 1 --2- 172.21.2.102:0/2477668696 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fade401cad0 0x7fade401eff0 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:11.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:11.338 7fae06a4b700 1 --2- 172.21.2.102:0/2477668696 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fae000f28d0 0x7fae00091580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:28:11.338 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:11.338 7fae06a4b700 1 --2- 172.21.2.102:0/2477668696 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fae0018bdd0 0x7fae000f22b0 unknown :-1 s=CLOSED pgs=1841 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:11.338 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:11.338 7fae06a4b700 1 --2- 172.21.2.102:0/2477668696 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fae0018b9d0 0x7fae0019d0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:28:11.338 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:11.338 7fae06a4b700 1 -- 172.21.2.102:0/2477668696 >> 172.21.2.102:0/2477668696 conn(0x7fae001a05c0 msgr2=0x7fae001b2f80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:28:11.338 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:11.338 7fae06a4b700 1 -- 172.21.2.102:0/2477668696 shutdown_connections 2021-02-10T11:28:11.338 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:11.338 7fae06a4b700 1 -- 172.21.2.102:0/2477668696 wait complete. 2021-02-10T11:28:11.346 DEBUG:teuthology.orchestra.run.gibba013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok session ls 2021-02-10T11:28:11.743 INFO:teuthology.orchestra.run.gibba013.stdout:[ 2021-02-10T11:28:11.744 INFO:teuthology.orchestra.run.gibba013.stdout: { 2021-02-10T11:28:11.744 INFO:teuthology.orchestra.run.gibba013.stdout: "id": 9471, 2021-02-10T11:28:11.744 INFO:teuthology.orchestra.run.gibba013.stdout: "entity": { 2021-02-10T11:28:11.744 INFO:teuthology.orchestra.run.gibba013.stdout: "name": { 2021-02-10T11:28:11.744 INFO:teuthology.orchestra.run.gibba013.stdout: "type": "client", 2021-02-10T11:28:11.744 INFO:teuthology.orchestra.run.gibba013.stdout: "num": 9471 2021-02-10T11:28:11.744 INFO:teuthology.orchestra.run.gibba013.stdout: }, 2021-02-10T11:28:11.744 INFO:teuthology.orchestra.run.gibba013.stdout: "addr": { 2021-02-10T11:28:11.744 INFO:teuthology.orchestra.run.gibba013.stdout: "type": "any", 2021-02-10T11:28:11.744 INFO:teuthology.orchestra.run.gibba013.stdout: "addr": "172.21.2.102:0", 2021-02-10T11:28:11.744 INFO:teuthology.orchestra.run.gibba013.stdout: "nonce": 190431005 2021-02-10T11:28:11.745 INFO:teuthology.orchestra.run.gibba013.stdout: } 2021-02-10T11:28:11.745 INFO:teuthology.orchestra.run.gibba013.stdout: }, 2021-02-10T11:28:11.745 INFO:teuthology.orchestra.run.gibba013.stdout: "state": "open", 2021-02-10T11:28:11.745 INFO:teuthology.orchestra.run.gibba013.stdout: "num_leases": 1, 2021-02-10T11:28:11.745 INFO:teuthology.orchestra.run.gibba013.stdout: "num_caps": 7, 2021-02-10T11:28:11.745 INFO:teuthology.orchestra.run.gibba013.stdout: "request_load_avg": 13, 2021-02-10T11:28:11.745 INFO:teuthology.orchestra.run.gibba013.stdout: "uptime": 1.0660000549999999, 2021-02-10T11:28:11.745 INFO:teuthology.orchestra.run.gibba013.stdout: "requests_in_flight": 0, 2021-02-10T11:28:11.745 INFO:teuthology.orchestra.run.gibba013.stdout: "completed_requests": 1, 2021-02-10T11:28:11.745 INFO:teuthology.orchestra.run.gibba013.stdout: "reconnecting": false, 2021-02-10T11:28:11.745 INFO:teuthology.orchestra.run.gibba013.stdout: "recall_caps": { 2021-02-10T11:28:11.745 INFO:teuthology.orchestra.run.gibba013.stdout: "value": 0, 2021-02-10T11:28:11.745 INFO:teuthology.orchestra.run.gibba013.stdout: "halflife": 60 2021-02-10T11:28:11.746 INFO:teuthology.orchestra.run.gibba013.stdout: }, 2021-02-10T11:28:11.746 INFO:teuthology.orchestra.run.gibba013.stdout: "release_caps": { 2021-02-10T11:28:11.746 INFO:teuthology.orchestra.run.gibba013.stdout: "value": 0, 2021-02-10T11:28:11.746 INFO:teuthology.orchestra.run.gibba013.stdout: "halflife": 60 2021-02-10T11:28:11.746 INFO:teuthology.orchestra.run.gibba013.stdout: }, 2021-02-10T11:28:11.746 INFO:teuthology.orchestra.run.gibba013.stdout: "recall_caps_throttle": { 2021-02-10T11:28:11.746 INFO:teuthology.orchestra.run.gibba013.stdout: "value": 0, 2021-02-10T11:28:11.746 INFO:teuthology.orchestra.run.gibba013.stdout: "halflife": 2.5 2021-02-10T11:28:11.746 INFO:teuthology.orchestra.run.gibba013.stdout: }, 2021-02-10T11:28:11.746 INFO:teuthology.orchestra.run.gibba013.stdout: "recall_caps_throttle2o": { 2021-02-10T11:28:11.746 INFO:teuthology.orchestra.run.gibba013.stdout: "value": 0, 2021-02-10T11:28:11.746 INFO:teuthology.orchestra.run.gibba013.stdout: "halflife": 0.5 2021-02-10T11:28:11.746 INFO:teuthology.orchestra.run.gibba013.stdout: }, 2021-02-10T11:28:11.747 INFO:teuthology.orchestra.run.gibba013.stdout: "session_cache_liveness": { 2021-02-10T11:28:11.747 INFO:teuthology.orchestra.run.gibba013.stdout: "value": 19.951287274048916, 2021-02-10T11:28:11.747 INFO:teuthology.orchestra.run.gibba013.stdout: "halflife": 300 2021-02-10T11:28:11.747 INFO:teuthology.orchestra.run.gibba013.stdout: }, 2021-02-10T11:28:11.747 INFO:teuthology.orchestra.run.gibba013.stdout: "cap_acquisition": { 2021-02-10T11:28:11.747 INFO:teuthology.orchestra.run.gibba013.stdout: "value": 0, 2021-02-10T11:28:11.747 INFO:teuthology.orchestra.run.gibba013.stdout: "halflife": 10 2021-02-10T11:28:11.747 INFO:teuthology.orchestra.run.gibba013.stdout: }, 2021-02-10T11:28:11.747 INFO:teuthology.orchestra.run.gibba013.stdout: "inst": "client.9471 172.21.2.102:0/190431005", 2021-02-10T11:28:11.747 INFO:teuthology.orchestra.run.gibba013.stdout: "completed_requests": [ 2021-02-10T11:28:11.747 INFO:teuthology.orchestra.run.gibba013.stdout: { 2021-02-10T11:28:11.747 INFO:teuthology.orchestra.run.gibba013.stdout: "tid": 14, 2021-02-10T11:28:11.747 INFO:teuthology.orchestra.run.gibba013.stdout: "created_ino": "0x10000000005" 2021-02-10T11:28:11.748 INFO:teuthology.orchestra.run.gibba013.stdout: } 2021-02-10T11:28:11.748 INFO:teuthology.orchestra.run.gibba013.stdout: ], 2021-02-10T11:28:11.748 INFO:teuthology.orchestra.run.gibba013.stdout: "prealloc_inos": [ 2021-02-10T11:28:11.748 INFO:teuthology.orchestra.run.gibba013.stdout: { 2021-02-10T11:28:11.748 INFO:teuthology.orchestra.run.gibba013.stdout: "start": 1099511627782, 2021-02-10T11:28:11.748 INFO:teuthology.orchestra.run.gibba013.stdout: "length": 995 2021-02-10T11:28:11.748 INFO:teuthology.orchestra.run.gibba013.stdout: } 2021-02-10T11:28:11.748 INFO:teuthology.orchestra.run.gibba013.stdout: ], 2021-02-10T11:28:11.748 INFO:teuthology.orchestra.run.gibba013.stdout: "used_inos": [], 2021-02-10T11:28:11.748 INFO:teuthology.orchestra.run.gibba013.stdout: "client_metadata": { 2021-02-10T11:28:11.748 INFO:teuthology.orchestra.run.gibba013.stdout: "features": "0x0000000000001fff", 2021-02-10T11:28:11.748 INFO:teuthology.orchestra.run.gibba013.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:28:11.748 INFO:teuthology.orchestra.run.gibba013.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:28:11.748 INFO:teuthology.orchestra.run.gibba013.stdout: "entity_id": "y", 2021-02-10T11:28:11.749 INFO:teuthology.orchestra.run.gibba013.stdout: "hostname": "gibba002", 2021-02-10T11:28:11.749 INFO:teuthology.orchestra.run.gibba013.stdout: "pid": "35161", 2021-02-10T11:28:11.749 INFO:teuthology.orchestra.run.gibba013.stdout: "root": "/" 2021-02-10T11:28:11.749 INFO:teuthology.orchestra.run.gibba013.stdout: } 2021-02-10T11:28:11.749 INFO:teuthology.orchestra.run.gibba013.stdout: } 2021-02-10T11:28:11.749 INFO:teuthology.orchestra.run.gibba013.stdout:] 2021-02-10T11:28:11.750 INFO:tasks.cephfs.filesystem:_json_asok output: [ { "id": 9471, "entity": { "name": { "type": "client", "num": 9471 }, "addr": { "type": "any", "addr": "172.21.2.102:0", "nonce": 190431005 } }, "state": "open", "num_leases": 1, "num_caps": 7, "request_load_avg": 13, "uptime": 1.0660000549999999, "requests_in_flight": 0, "completed_requests": 1, "reconnecting": false, "recall_caps": { "value": 0, "halflife": 60 }, "release_caps": { "value": 0, "halflife": 60 }, "recall_caps_throttle": { "value": 0, "halflife": 2.5 }, "recall_caps_throttle2o": { "value": 0, "halflife": 0.5 }, "session_cache_liveness": { "value": 19.951287274048916, "halflife": 300 }, "cap_acquisition": { "value": 0, "halflife": 10 }, "inst": "client.9471 172.21.2.102:0/190431005", "completed_requests": [ { "tid": 14, "created_ino": "0x10000000005" } ], "prealloc_inos": [ { "start": 1099511627782, "length": 995 } ], "used_inos": [], "client_metadata": { "features": "0x0000000000001fff", "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", "entity_id": "y", "hostname": "gibba002", "pid": "35161", "root": "/" } } ] 2021-02-10T11:28:11.750 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:28:12.180 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:12.164 7f5ed19d9700 1 Processor -- start 2021-02-10T11:28:12.180 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:12.164 7f5ed19d9700 1 -- start start 2021-02-10T11:28:12.181 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:12.164 7f5ed19d9700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f5ecc0d1870 0x7f5ecc1b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:12.181 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:12.164 7f5ed19d9700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5ecc0f0650 0x7f5ecc0cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:12.181 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:12.164 7f5ed19d9700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ecc0f0b10 con 0x7f5ecc1b0a10 2021-02-10T11:28:12.181 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:12.164 7f5ed19d9700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ecc0e5ed0 con 0x7f5ecc0d1870 2021-02-10T11:28:12.181 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:12.164 7f5ed19d9700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ecc0d1c70 con 0x7f5ecc0f0650 2021-02-10T11:28:12.181 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:12.164 7f5ecad73700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f5ecc1b0a10 0x7f5ecc0cf330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:52390/0 (socket says 172.21.2.102:52390) 2021-02-10T11:28:12.181 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:12.164 7f5ecad73700 1 -- 172.21.2.102:0/2101704528 learned_addr learned my addr 172.21.2.102:0/2101704528 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:28:12.181 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:12.165 7f5ecbd75700 1 --2- 172.21.2.102:0/2101704528 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5ecc0f0650 0x7f5ecc0cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:12.181 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:12.165 7f5ecb574700 1 --2- 172.21.2.102:0/2101704528 >> v2:172.21.2.113:3300/0 conn(0x7f5ecc0d1870 0x7f5ecc1b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:12.181 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:12.165 7f5eca572700 1 -- 172.21.2.102:0/2101704528 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 669839032 0 0) 0x7f5eb40009b0 con 0x7f5ecc1b0a10 2021-02-10T11:28:12.181 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:12.165 7f5eca572700 1 -- 172.21.2.102:0/2101704528 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5eb8002fe0 con 0x7f5ecc1b0a10 2021-02-10T11:28:12.181 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:12.165 7f5eca572700 1 -- 172.21.2.102:0/2101704528 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1424927871 0 0) 0x7f5eb4000e20 con 0x7f5ecc1b0a10 2021-02-10T11:28:12.181 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:12.165 7f5eca572700 1 -- 172.21.2.102:0/2101704528 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5ecc0f0650 msgr2=0x7f5ecc0cfb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:28:12.182 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:12.165 7f5eca572700 1 --2- 172.21.2.102:0/2101704528 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5ecc0f0650 0x7f5ecc0cfb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:12.182 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:12.165 7f5eca572700 1 -- 172.21.2.102:0/2101704528 >> v2:172.21.2.113:3300/0 conn(0x7f5ecc0d1870 msgr2=0x7f5ecc1b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:28:12.182 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:12.165 7f5eca572700 1 --2- 172.21.2.102:0/2101704528 >> v2:172.21.2.113:3300/0 conn(0x7f5ecc0d1870 0x7f5ecc1b03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:12.182 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:12.165 7f5eca572700 1 -- 172.21.2.102:0/2101704528 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ecc0d1e90 con 0x7f5ecc1b0a10 2021-02-10T11:28:12.182 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:12.165 7f5eca572700 1 -- 172.21.2.102:0/2101704528 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7f5eb4001530 con 0x7f5ecc1b0a10 2021-02-10T11:28:12.182 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:12.165 7f5eca572700 1 -- 172.21.2.102:0/2101704528 >> v1:172.21.2.102:6789/0 conn(0x7f5ecc1b0a10 legacy=0x7f5ecc0cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:12.182 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:12.165 7f5eca572700 1 --2- 172.21.2.102:0/2101704528 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5eb8003a70 0x7f5eb8005ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:12.182 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:12.165 7f5eca572700 1 --2- 172.21.2.102:0/2101704528 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5eb804a750 0x7f5eb804cc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:12.182 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:12.165 7f5eca572700 1 --2- 172.21.2.102:0/2101704528 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5eb804d510 0x7f5eb804fa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:12.182 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:12.165 7f5eca572700 1 -- 172.21.2.102:0/2101704528 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5eb80502e0 con 0x7f5eb804a750 2021-02-10T11:28:12.182 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:12.165 7f5eca572700 1 -- 172.21.2.102:0/2101704528 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5eb8050500 con 0x7f5eb804d510 2021-02-10T11:28:12.182 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:12.165 7f5eca572700 1 -- 172.21.2.102:0/2101704528 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5eb8050720 con 0x7f5eb8003a70 2021-02-10T11:28:12.182 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:12.165 7f5ecad73700 1 --2- 172.21.2.102:0/2101704528 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5eb804a750 0x7f5eb804cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:12.183 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:12.165 7f5ecb574700 1 --2- 172.21.2.102:0/2101704528 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5eb8003a70 0x7f5eb8005ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:12.183 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:12.165 7f5ecad73700 1 -- 172.21.2.102:0/2101704528 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5eb8003a70 msgr2=0x7f5eb8005ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:28:12.183 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:12.165 7f5ecad73700 1 --2- 172.21.2.102:0/2101704528 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5eb8003a70 0x7f5eb8005ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:12.183 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:12.165 7f5ecad73700 1 -- 172.21.2.102:0/2101704528 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5eb804d510 msgr2=0x7f5eb804fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:28:12.183 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:12.165 7f5ecad73700 1 --2- 172.21.2.102:0/2101704528 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5eb804d510 0x7f5eb804fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:28:12.183 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:12.165 7f5ecad73700 1 -- 172.21.2.102:0/2101704528 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5eb80509c0 con 0x7f5eb804a750 2021-02-10T11:28:12.183 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:12.165 7f5ecad73700 1 --2- 172.21.2.102:0/2101704528 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5eb804a750 0x7f5eb804cc70 secure :-1 s=READY pgs=1843 cs=0 l=1 rev1=1 rx=0x7f5eb4007560 tx=0x7f5eb4007d10).ready entity=mon.0 client_cookie=3007c1f9da81b4d5 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:28:12.183 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:12.166 7f5eca572700 1 -- 172.21.2.102:0/2101704528 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5eb4001750 con 0x7f5eb804a750 2021-02-10T11:28:12.183 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:12.166 7f5eca572700 1 -- 172.21.2.102:0/2101704528 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5eb400a310 con 0x7f5eb804a750 2021-02-10T11:28:12.183 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:12.166 7f5eca572700 1 -- 172.21.2.102:0/2101704528 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5eb400a830 con 0x7f5eb804a750 2021-02-10T11:28:12.183 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:12.166 7f5ed19d9700 1 -- 172.21.2.102:0/2101704528 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5eb804a750 msgr2=0x7f5eb804cc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:12.183 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:12.166 7f5ed19d9700 1 --2- 172.21.2.102:0/2101704528 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5eb804a750 0x7f5eb804cc70 secure :-1 s=READY pgs=1843 cs=0 l=1 rev1=1 rx=0x7f5eb4007560 tx=0x7f5eb4007d10).stop 2021-02-10T11:28:12.183 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:12.166 7f5ecbd75700 1 -- 172.21.2.102:0/2101704528 reap_dead start 2021-02-10T11:28:12.184 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:12.166 7f5ed19d9700 1 -- 172.21.2.102:0/2101704528 shutdown_connections 2021-02-10T11:28:12.184 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:12.166 7f5ed19d9700 1 -- 172.21.2.102:0/2101704528 >> 172.21.2.102:0/2101704528 conn(0x7f5ecc0ce450 msgr2=0x7f5ecc0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:28:12.184 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:12.166 7f5ed19d9700 1 -- 172.21.2.102:0/2101704528 shutdown_connections 2021-02-10T11:28:12.184 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:12.166 7f5ed19d9700 1 -- 172.21.2.102:0/2101704528 wait complete. 2021-02-10T11:28:12.184 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:12.181 7f5ed19d9700 1 Processor -- start 2021-02-10T11:28:12.184 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:12.181 7f5ed19d9700 1 -- start start 2021-02-10T11:28:12.186 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:12.182 7f5ed19d9700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5ecc0f04f0 0x7f5ecc1a7a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:12.186 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:12.182 7f5ed19d9700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5ecc1a8090 0x7f5ecc0b0830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:12.186 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:12.182 7f5ed19d9700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5ecc0b0e50 0x7f5ecc0b1250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:12.186 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:12.182 7f5ed19d9700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ecc0b02b0 con 0x7f5ecc1a8090 2021-02-10T11:28:12.186 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:12.182 7f5ed19d9700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ecc0b1870 con 0x7f5ecc0f04f0 2021-02-10T11:28:12.186 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:12.182 7f5ed19d9700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ecc0b1a50 con 0x7f5ecc0b0e50 2021-02-10T11:28:12.186 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:12.182 7f5ecad73700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5ecc1a8090 0x7f5ecc0b0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:12.186 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:12.182 7f5ecad73700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5ecc1a8090 0x7f5ecc0b0830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:60178/0 (socket says 172.21.2.102:60178) 2021-02-10T11:28:12.186 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:12.182 7f5ecad73700 1 -- 172.21.2.102:0/987253484 learned_addr learned my addr 172.21.2.102:0/987253484 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:28:12.186 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:12.182 7f5ecad73700 1 -- 172.21.2.102:0/987253484 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5ecc0b0e50 msgr2=0x7f5ecc0b1250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:12.186 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:12.182 7f5ecad73700 1 --2- 172.21.2.102:0/987253484 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5ecc0b0e50 0x7f5ecc0b1250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:28:12.187 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:12.182 7f5ecad73700 1 -- 172.21.2.102:0/987253484 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5ecc0f04f0 msgr2=0x7f5ecc1a7a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:12.187 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:12.182 7f5ecad73700 1 --2- 172.21.2.102:0/987253484 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5ecc0f04f0 0x7f5ecc1a7a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:28:12.187 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:12.182 7f5ecad73700 1 -- 172.21.2.102:0/987253484 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ecc0aec30 con 0x7f5ecc1a8090 2021-02-10T11:28:12.187 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:12.182 7f5ecad73700 1 --2- 172.21.2.102:0/987253484 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5ecc1a8090 0x7f5ecc0b0830 secure :-1 s=READY pgs=1844 cs=0 l=1 rev1=1 rx=0x7f5eb400a520 tx=0x7f5eb400a8c0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:28:12.187 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:12.182 7f5ec3fff700 1 -- 172.21.2.102:0/987253484 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5eb400acb0 con 0x7f5ecc1a8090 2021-02-10T11:28:12.187 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:12.182 7f5ec3fff700 1 -- 172.21.2.102:0/987253484 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5eb4004140 con 0x7f5ecc1a8090 2021-02-10T11:28:12.187 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:12.182 7f5ec3fff700 1 -- 172.21.2.102:0/987253484 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5eb4004590 con 0x7f5ecc1a8090 2021-02-10T11:28:12.187 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:12.183 7f5ed19d9700 1 -- 172.21.2.102:0/987253484 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ecc0aec30 con 0x7f5ecc1a8090 2021-02-10T11:28:12.187 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:12.183 7f5ed19d9700 1 -- 172.21.2.102:0/987253484 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ecc0aec30 con 0x7f5ecc1a8090 2021-02-10T11:28:12.187 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:12.183 7f5ec3fff700 1 -- 172.21.2.102:0/987253484 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f5eb40010c0 con 0x7f5ecc1a8090 2021-02-10T11:28:12.187 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:12.183 7f5ec3fff700 1 --2- 172.21.2.102:0/987253484 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5eb0018700 0x7f5eb001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:12.187 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:12.183 7f5ec3fff700 1 -- 172.21.2.102:0/987253484 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f5eb40327b0 con 0x7f5ecc1a8090 2021-02-10T11:28:12.187 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:12.183 7f5ecb574700 1 --2- 172.21.2.102:0/987253484 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5eb0018700 0x7f5eb001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:12.187 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:12.184 7f5ecb574700 1 --2- 172.21.2.102:0/987253484 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5eb0018700 0x7f5eb001abc0 crc :-1 s=READY pgs=828 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:28:12.213 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:12.214 7f5ed19d9700 1 -- 172.21.2.102:0/987253484 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ecc0f0b10 con 0x7f5ecc1a8090 2021-02-10T11:28:12.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:12.216 7f5ec3fff700 1 -- 172.21.2.102:0/987253484 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f5eb4000d90 con 0x7f5ecc1a8090 2021-02-10T11:28:12.344 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:12.345 7f5ed19d9700 1 -- 172.21.2.102:0/987253484 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5ecc0f0b10 con 0x7f5ecc1a8090 2021-02-10T11:28:12.345 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:12.346 7f5ec3fff700 1 -- 172.21.2.102:0/987253484 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+2101 (secure 0 0 0) 0x7f5eb40010c0 con 0x7f5ecc1a8090 2021-02-10T11:28:12.345 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:28:12.345 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":66,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6110,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.102:6835/1471871278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6834","nonce":1471871278},{"type":"v1","addr":"172.21.2.102:6835","nonce":1471871278}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":61}],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:27:41.479375","modified":"2021-02-10 11:27:43.490605","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6107},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6107":{"gid":6107,"name":"b","rank":0,"incarnation":62,"state":"up:active","state_seq":3,"addr":"172.21.2.113:6833/4220238062","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6832","nonce":4220238062},{"type":"v1","addr":"172.21.2.113:6833","nonce":4220238062}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":8}]} 2021-02-10T11:28:12.346 INFO:teuthology.orchestra.run.gibba002.stderr:dumped fsmap epoch 66 2021-02-10T11:28:12.347 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:12.347 7f5ed19d9700 1 -- 172.21.2.102:0/987253484 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5eb0018700 msgr2=0x7f5eb001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:12.347 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:12.347 7f5ed19d9700 1 --2- 172.21.2.102:0/987253484 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5eb0018700 0x7f5eb001abc0 crc :-1 s=READY pgs=828 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:12.347 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:12.347 7f5ed19d9700 1 -- 172.21.2.102:0/987253484 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5ecc1a8090 msgr2=0x7f5ecc0b0830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:12.347 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:12.347 7f5ed19d9700 1 --2- 172.21.2.102:0/987253484 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5ecc1a8090 0x7f5ecc0b0830 secure :-1 s=READY pgs=1844 cs=0 l=1 rev1=1 rx=0x7f5eb400a520 tx=0x7f5eb400a8c0).stop 2021-02-10T11:28:12.347 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:12.347 7f5ed19d9700 1 -- 172.21.2.102:0/987253484 shutdown_connections 2021-02-10T11:28:12.347 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:12.347 7f5ed19d9700 1 --2- 172.21.2.102:0/987253484 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5ecc0b0e50 0x7f5ecc0b1250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:28:12.347 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:12.347 7f5ed19d9700 1 --2- 172.21.2.102:0/987253484 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5ecc1a8090 0x7f5ecc0b0830 unknown :-1 s=CLOSED pgs=1844 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:12.347 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:12.347 7f5ed19d9700 1 --2- 172.21.2.102:0/987253484 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5eb0018700 0x7f5eb001abc0 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:12.347 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:12.347 7f5ed19d9700 1 --2- 172.21.2.102:0/987253484 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5ecc0f04f0 0x7f5ecc1a7a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:28:12.347 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:12.347 7f5ed19d9700 1 -- 172.21.2.102:0/987253484 >> 172.21.2.102:0/987253484 conn(0x7f5ecc09eb60 msgr2=0x7f5ecc190180 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:28:12.348 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:12.348 7f5ed19d9700 1 -- 172.21.2.102:0/987253484 shutdown_connections 2021-02-10T11:28:12.348 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:12.348 7f5ed19d9700 1 -- 172.21.2.102:0/987253484 wait complete. 2021-02-10T11:28:12.356 DEBUG:teuthology.orchestra.run.gibba013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok session ls 2021-02-10T11:28:12.751 INFO:teuthology.orchestra.run.gibba013.stdout:[ 2021-02-10T11:28:12.752 INFO:teuthology.orchestra.run.gibba013.stdout: { 2021-02-10T11:28:12.752 INFO:teuthology.orchestra.run.gibba013.stdout: "id": 9471, 2021-02-10T11:28:12.752 INFO:teuthology.orchestra.run.gibba013.stdout: "entity": { 2021-02-10T11:28:12.752 INFO:teuthology.orchestra.run.gibba013.stdout: "name": { 2021-02-10T11:28:12.752 INFO:teuthology.orchestra.run.gibba013.stdout: "type": "client", 2021-02-10T11:28:12.752 INFO:teuthology.orchestra.run.gibba013.stdout: "num": 9471 2021-02-10T11:28:12.752 INFO:teuthology.orchestra.run.gibba013.stdout: }, 2021-02-10T11:28:12.752 INFO:teuthology.orchestra.run.gibba013.stdout: "addr": { 2021-02-10T11:28:12.752 INFO:teuthology.orchestra.run.gibba013.stdout: "type": "any", 2021-02-10T11:28:12.752 INFO:teuthology.orchestra.run.gibba013.stdout: "addr": "172.21.2.102:0", 2021-02-10T11:28:12.752 INFO:teuthology.orchestra.run.gibba013.stdout: "nonce": 190431005 2021-02-10T11:28:12.753 INFO:teuthology.orchestra.run.gibba013.stdout: } 2021-02-10T11:28:12.753 INFO:teuthology.orchestra.run.gibba013.stdout: }, 2021-02-10T11:28:12.753 INFO:teuthology.orchestra.run.gibba013.stdout: "state": "open", 2021-02-10T11:28:12.753 INFO:teuthology.orchestra.run.gibba013.stdout: "num_leases": 1, 2021-02-10T11:28:12.753 INFO:teuthology.orchestra.run.gibba013.stdout: "num_caps": 7, 2021-02-10T11:28:12.753 INFO:teuthology.orchestra.run.gibba013.stdout: "request_load_avg": 13, 2021-02-10T11:28:12.753 INFO:teuthology.orchestra.run.gibba013.stdout: "uptime": 2.0740001060000002, 2021-02-10T11:28:12.753 INFO:teuthology.orchestra.run.gibba013.stdout: "requests_in_flight": 0, 2021-02-10T11:28:12.753 INFO:teuthology.orchestra.run.gibba013.stdout: "completed_requests": 1, 2021-02-10T11:28:12.753 INFO:teuthology.orchestra.run.gibba013.stdout: "reconnecting": false, 2021-02-10T11:28:12.753 INFO:teuthology.orchestra.run.gibba013.stdout: "recall_caps": { 2021-02-10T11:28:12.753 INFO:teuthology.orchestra.run.gibba013.stdout: "value": 0, 2021-02-10T11:28:12.753 INFO:teuthology.orchestra.run.gibba013.stdout: "halflife": 60 2021-02-10T11:28:12.754 INFO:teuthology.orchestra.run.gibba013.stdout: }, 2021-02-10T11:28:12.754 INFO:teuthology.orchestra.run.gibba013.stdout: "release_caps": { 2021-02-10T11:28:12.754 INFO:teuthology.orchestra.run.gibba013.stdout: "value": 0, 2021-02-10T11:28:12.754 INFO:teuthology.orchestra.run.gibba013.stdout: "halflife": 60 2021-02-10T11:28:12.754 INFO:teuthology.orchestra.run.gibba013.stdout: }, 2021-02-10T11:28:12.754 INFO:teuthology.orchestra.run.gibba013.stdout: "recall_caps_throttle": { 2021-02-10T11:28:12.754 INFO:teuthology.orchestra.run.gibba013.stdout: "value": 0, 2021-02-10T11:28:12.754 INFO:teuthology.orchestra.run.gibba013.stdout: "halflife": 2.5 2021-02-10T11:28:12.754 INFO:teuthology.orchestra.run.gibba013.stdout: }, 2021-02-10T11:28:12.754 INFO:teuthology.orchestra.run.gibba013.stdout: "recall_caps_throttle2o": { 2021-02-10T11:28:12.754 INFO:teuthology.orchestra.run.gibba013.stdout: "value": 0, 2021-02-10T11:28:12.754 INFO:teuthology.orchestra.run.gibba013.stdout: "halflife": 0.5 2021-02-10T11:28:12.754 INFO:teuthology.orchestra.run.gibba013.stdout: }, 2021-02-10T11:28:12.754 INFO:teuthology.orchestra.run.gibba013.stdout: "session_cache_liveness": { 2021-02-10T11:28:12.755 INFO:teuthology.orchestra.run.gibba013.stdout: "value": 19.904875298997425, 2021-02-10T11:28:12.755 INFO:teuthology.orchestra.run.gibba013.stdout: "halflife": 300 2021-02-10T11:28:12.755 INFO:teuthology.orchestra.run.gibba013.stdout: }, 2021-02-10T11:28:12.755 INFO:teuthology.orchestra.run.gibba013.stdout: "cap_acquisition": { 2021-02-10T11:28:12.755 INFO:teuthology.orchestra.run.gibba013.stdout: "value": 0, 2021-02-10T11:28:12.755 INFO:teuthology.orchestra.run.gibba013.stdout: "halflife": 10 2021-02-10T11:28:12.755 INFO:teuthology.orchestra.run.gibba013.stdout: }, 2021-02-10T11:28:12.755 INFO:teuthology.orchestra.run.gibba013.stdout: "inst": "client.9471 172.21.2.102:0/190431005", 2021-02-10T11:28:12.755 INFO:teuthology.orchestra.run.gibba013.stdout: "completed_requests": [ 2021-02-10T11:28:12.755 INFO:teuthology.orchestra.run.gibba013.stdout: { 2021-02-10T11:28:12.755 INFO:teuthology.orchestra.run.gibba013.stdout: "tid": 14, 2021-02-10T11:28:12.755 INFO:teuthology.orchestra.run.gibba013.stdout: "created_ino": "0x10000000005" 2021-02-10T11:28:12.755 INFO:teuthology.orchestra.run.gibba013.stdout: } 2021-02-10T11:28:12.755 INFO:teuthology.orchestra.run.gibba013.stdout: ], 2021-02-10T11:28:12.756 INFO:teuthology.orchestra.run.gibba013.stdout: "prealloc_inos": [ 2021-02-10T11:28:12.756 INFO:teuthology.orchestra.run.gibba013.stdout: { 2021-02-10T11:28:12.756 INFO:teuthology.orchestra.run.gibba013.stdout: "start": 1099511627782, 2021-02-10T11:28:12.756 INFO:teuthology.orchestra.run.gibba013.stdout: "length": 995 2021-02-10T11:28:12.756 INFO:teuthology.orchestra.run.gibba013.stdout: } 2021-02-10T11:28:12.756 INFO:teuthology.orchestra.run.gibba013.stdout: ], 2021-02-10T11:28:12.756 INFO:teuthology.orchestra.run.gibba013.stdout: "used_inos": [], 2021-02-10T11:28:12.756 INFO:teuthology.orchestra.run.gibba013.stdout: "client_metadata": { 2021-02-10T11:28:12.756 INFO:teuthology.orchestra.run.gibba013.stdout: "features": "0x0000000000001fff", 2021-02-10T11:28:12.756 INFO:teuthology.orchestra.run.gibba013.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:28:12.756 INFO:teuthology.orchestra.run.gibba013.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:28:12.756 INFO:teuthology.orchestra.run.gibba013.stdout: "entity_id": "y", 2021-02-10T11:28:12.756 INFO:teuthology.orchestra.run.gibba013.stdout: "hostname": "gibba002", 2021-02-10T11:28:12.757 INFO:teuthology.orchestra.run.gibba013.stdout: "pid": "35161", 2021-02-10T11:28:12.757 INFO:teuthology.orchestra.run.gibba013.stdout: "root": "/" 2021-02-10T11:28:12.757 INFO:teuthology.orchestra.run.gibba013.stdout: } 2021-02-10T11:28:12.757 INFO:teuthology.orchestra.run.gibba013.stdout: } 2021-02-10T11:28:12.757 INFO:teuthology.orchestra.run.gibba013.stdout:] 2021-02-10T11:28:12.758 INFO:tasks.cephfs.filesystem:_json_asok output: [ { "id": 9471, "entity": { "name": { "type": "client", "num": 9471 }, "addr": { "type": "any", "addr": "172.21.2.102:0", "nonce": 190431005 } }, "state": "open", "num_leases": 1, "num_caps": 7, "request_load_avg": 13, "uptime": 2.0740001060000002, "requests_in_flight": 0, "completed_requests": 1, "reconnecting": false, "recall_caps": { "value": 0, "halflife": 60 }, "release_caps": { "value": 0, "halflife": 60 }, "recall_caps_throttle": { "value": 0, "halflife": 2.5 }, "recall_caps_throttle2o": { "value": 0, "halflife": 0.5 }, "session_cache_liveness": { "value": 19.904875298997425, "halflife": 300 }, "cap_acquisition": { "value": 0, "halflife": 10 }, "inst": "client.9471 172.21.2.102:0/190431005", "completed_requests": [ { "tid": 14, "created_ino": "0x10000000005" } ], "prealloc_inos": [ { "start": 1099511627782, "length": 995 } ], "used_inos": [], "client_metadata": { "features": "0x0000000000001fff", "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", "entity_id": "y", "hostname": "gibba002", "pid": "35161", "root": "/" } } ] 2021-02-10T11:28:12.758 DEBUG:tasks.ceph_test_case:wait_until_true: waiting... 2021-02-10T11:28:17.759 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:28:18.188 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:18.171 7f95b028e700 1 Processor -- start 2021-02-10T11:28:18.188 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:18.171 7f95b028e700 1 -- start start 2021-02-10T11:28:18.188 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:18.172 7f95b028e700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f95a80d1c70 0x7f95a80f20a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:18.188 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:18.172 7f95b028e700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95a80f0b10 con 0x7f95a80d0340 2021-02-10T11:28:18.188 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:18.172 7f95b028e700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95a81b0be0 con 0x7f95a80d1870 2021-02-10T11:28:18.188 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:18.172 7f95b028e700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95a80e5ed0 con 0x7f95a80d1c70 2021-02-10T11:28:18.188 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:18.172 7f95ae8ab700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f95a80d1c70 0x7f95a80f20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:18.189 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:18.172 7f95ae8ab700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f95a80d1c70 0x7f95a80f20a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:60184/0 (socket says 172.21.2.102:60184) 2021-02-10T11:28:18.189 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:18.172 7f95ae8ab700 1 -- 172.21.2.102:0/4244170907 learned_addr learned my addr 172.21.2.102:0/4244170907 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:28:18.189 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:18.173 7f95ae8ab700 1 -- 172.21.2.102:0/4244170907 >> v1:172.21.2.113:6789/0 conn(0x7f95a80d1870 legacy=0x7f95a81b03f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:28:18.189 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:18.173 7f95ae8ab700 1 -- 172.21.2.102:0/4244170907 >> v1:172.21.2.102:6789/0 conn(0x7f95a80d0340 legacy=0x7f95a80ef640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:18.189 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:18.173 7f95ae8ab700 1 -- 172.21.2.102:0/4244170907 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95a80efe30 con 0x7f95a80d1c70 2021-02-10T11:28:18.189 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:18.173 7f95ae8ab700 1 --2- 172.21.2.102:0/4244170907 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f95a80d1c70 0x7f95a80f20a0 secure :-1 s=READY pgs=1846 cs=0 l=1 rev1=1 rx=0x7f95a0006f00 tx=0x7f95a0009e90).ready entity=mon.0 client_cookie=c303830c2b165dfe server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:28:18.189 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:18.173 7f95ad0a8700 1 -- 172.21.2.102:0/4244170907 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f95a000b300 con 0x7f95a80d1c70 2021-02-10T11:28:18.189 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:18.173 7f95ad0a8700 1 -- 172.21.2.102:0/4244170907 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95a000b580 con 0x7f95a80d1c70 2021-02-10T11:28:18.189 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:18.173 7f95ad0a8700 1 -- 172.21.2.102:0/4244170907 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f95a000b9d0 con 0x7f95a80d1c70 2021-02-10T11:28:18.189 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:18.173 7f95b028e700 1 -- 172.21.2.102:0/4244170907 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f95a80d1c70 msgr2=0x7f95a80f20a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:18.189 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:18.173 7f95b028e700 1 --2- 172.21.2.102:0/4244170907 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f95a80d1c70 0x7f95a80f20a0 secure :-1 s=READY pgs=1846 cs=0 l=1 rev1=1 rx=0x7f95a0006f00 tx=0x7f95a0009e90).stop 2021-02-10T11:28:18.189 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:18.174 7f95b028e700 1 -- 172.21.2.102:0/4244170907 shutdown_connections 2021-02-10T11:28:18.189 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:18.174 7f95b028e700 1 --2- 172.21.2.102:0/4244170907 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f95a80d1c70 0x7f95a80f20a0 unknown :-1 s=CLOSED pgs=1846 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:18.190 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:18.174 7f95b028e700 1 -- 172.21.2.102:0/4244170907 >> 172.21.2.102:0/4244170907 conn(0x7f95a80ce450 msgr2=0x7f95a80d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:28:18.190 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:18.174 7f95b028e700 1 -- 172.21.2.102:0/4244170907 shutdown_connections 2021-02-10T11:28:18.190 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:18.174 7f95b028e700 1 -- 172.21.2.102:0/4244170907 wait complete. 2021-02-10T11:28:18.191 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:18.188 7f95b028e700 1 Processor -- start 2021-02-10T11:28:18.191 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:18.188 7f95b028e700 1 -- start start 2021-02-10T11:28:18.191 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:18.188 7f95b028e700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f95a80a35b0 0x7f95a80a2900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:18.191 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:18.189 7f95b028e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f95a81c6350 0x7f95a80b17a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:18.191 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:18.189 7f95b028e700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f95a81c6750 0x7f95a80b1dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:18.191 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:18.189 7f95b028e700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95a80a3030 con 0x7f95a81c6350 2021-02-10T11:28:18.192 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:18.189 7f95b028e700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95a80f2c00 con 0x7f95a81c6750 2021-02-10T11:28:18.192 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:18.189 7f95b028e700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95a809bfc0 con 0x7f95a80a35b0 2021-02-10T11:28:18.192 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:18.189 7f95ad8a9700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f95a81c6350 0x7f95a80b17a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:18.192 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:18.189 7f95ad8a9700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f95a81c6350 0x7f95a80b17a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:60188/0 (socket says 172.21.2.102:60188) 2021-02-10T11:28:18.192 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:18.189 7f95ad8a9700 1 -- 172.21.2.102:0/3657625887 learned_addr learned my addr 172.21.2.102:0/3657625887 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:28:18.192 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:18.189 7f95ad8a9700 1 -- 172.21.2.102:0/3657625887 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f95a80a35b0 msgr2=0x7f95a80a2900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:18.192 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:18.189 7f95ad8a9700 1 --2- 172.21.2.102:0/3657625887 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f95a80a35b0 0x7f95a80a2900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:28:18.192 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:18.189 7f95ad8a9700 1 -- 172.21.2.102:0/3657625887 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f95a81c6750 msgr2=0x7f95a80b1dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:18.192 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:18.189 7f95ad8a9700 1 --2- 172.21.2.102:0/3657625887 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f95a81c6750 0x7f95a80b1dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:28:18.192 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:18.189 7f95ad8a9700 1 -- 172.21.2.102:0/3657625887 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95a809c1e0 con 0x7f95a81c6350 2021-02-10T11:28:18.192 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:18.189 7f95ad8a9700 1 --2- 172.21.2.102:0/3657625887 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f95a81c6350 0x7f95a80b17a0 secure :-1 s=READY pgs=1847 cs=0 l=1 rev1=1 rx=0x7f95a4006f00 tx=0x7f95a400b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:28:18.192 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:18.189 7f959affd700 1 -- 172.21.2.102:0/3657625887 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f95a4007070 con 0x7f95a81c6350 2021-02-10T11:28:18.192 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:18.189 7f959affd700 1 -- 172.21.2.102:0/3657625887 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95a4007330 con 0x7f95a81c6350 2021-02-10T11:28:18.193 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:18.189 7f959affd700 1 -- 172.21.2.102:0/3657625887 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f95a4007850 con 0x7f95a81c6350 2021-02-10T11:28:18.193 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:18.189 7f95b028e700 1 -- 172.21.2.102:0/3657625887 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95a809c1e0 con 0x7f95a81c6350 2021-02-10T11:28:18.193 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:18.190 7f95b028e700 1 -- 172.21.2.102:0/3657625887 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95a809c1e0 con 0x7f95a81c6350 2021-02-10T11:28:18.193 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:18.190 7f959affd700 1 -- 172.21.2.102:0/3657625887 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f95a4007000 con 0x7f95a81c6350 2021-02-10T11:28:18.193 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:18.190 7f959affd700 1 --2- 172.21.2.102:0/3657625887 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f9590018730 0x7f959001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:18.193 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:18.190 7f959affd700 1 -- 172.21.2.102:0/3657625887 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f95a4032660 con 0x7f95a81c6350 2021-02-10T11:28:18.193 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:18.190 7f95ae0aa700 1 --2- 172.21.2.102:0/3657625887 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f9590018730 0x7f959001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:18.193 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:18.191 7f95ae0aa700 1 --2- 172.21.2.102:0/3657625887 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f9590018730 0x7f959001abc0 crc :-1 s=READY pgs=829 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:28:18.222 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:18.222 7f95b028e700 1 -- 172.21.2.102:0/3657625887 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95a80e7df0 con 0x7f95a81c6350 2021-02-10T11:28:18.224 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:18.224 7f959affd700 1 -- 172.21.2.102:0/3657625887 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f95a4006b90 con 0x7f95a81c6350 2021-02-10T11:28:18.354 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:18.354 7f95b028e700 1 -- 172.21.2.102:0/3657625887 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f95a80ce450 con 0x7f95a81c6350 2021-02-10T11:28:18.354 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:18.354 7f959affd700 1 -- 172.21.2.102:0/3657625887 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+2101 (secure 0 0 0) 0x7f95a4007000 con 0x7f95a81c6350 2021-02-10T11:28:18.355 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:28:18.355 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":66,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6110,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.102:6835/1471871278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6834","nonce":1471871278},{"type":"v1","addr":"172.21.2.102:6835","nonce":1471871278}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":61}],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:27:41.479375","modified":"2021-02-10 11:27:43.490605","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6107},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6107":{"gid":6107,"name":"b","rank":0,"incarnation":62,"state":"up:active","state_seq":3,"addr":"172.21.2.113:6833/4220238062","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6832","nonce":4220238062},{"type":"v1","addr":"172.21.2.113:6833","nonce":4220238062}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":8}]} 2021-02-10T11:28:18.355 INFO:teuthology.orchestra.run.gibba002.stderr:dumped fsmap epoch 66 2021-02-10T11:28:18.357 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:18.356 7f95b028e700 1 -- 172.21.2.102:0/3657625887 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f9590018730 msgr2=0x7f959001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:18.357 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:18.356 7f95b028e700 1 --2- 172.21.2.102:0/3657625887 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f9590018730 0x7f959001abc0 crc :-1 s=READY pgs=829 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:18.357 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:18.356 7f95b028e700 1 -- 172.21.2.102:0/3657625887 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f95a81c6350 msgr2=0x7f95a80b17a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:18.357 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:18.356 7f95b028e700 1 --2- 172.21.2.102:0/3657625887 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f95a81c6350 0x7f95a80b17a0 secure :-1 s=READY pgs=1847 cs=0 l=1 rev1=1 rx=0x7f95a4006f00 tx=0x7f95a400b940).stop 2021-02-10T11:28:18.357 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:18.356 7f95b028e700 1 -- 172.21.2.102:0/3657625887 shutdown_connections 2021-02-10T11:28:18.357 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:18.356 7f95b028e700 1 --2- 172.21.2.102:0/3657625887 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f9590018730 0x7f959001abc0 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:18.357 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:18.356 7f95b028e700 1 --2- 172.21.2.102:0/3657625887 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f95a81c6750 0x7f95a80b1dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:28:18.357 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:18.356 7f95b028e700 1 --2- 172.21.2.102:0/3657625887 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f95a81c6350 0x7f95a80b17a0 unknown :-1 s=CLOSED pgs=1847 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:18.357 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:18.356 7f95b028e700 1 --2- 172.21.2.102:0/3657625887 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f95a80a35b0 0x7f95a80a2900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:28:18.357 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:18.356 7f95b028e700 1 -- 172.21.2.102:0/3657625887 >> 172.21.2.102:0/3657625887 conn(0x7f95a80a0030 msgr2=0x7f95a80ec020 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:28:18.357 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:18.356 7f95b028e700 1 -- 172.21.2.102:0/3657625887 shutdown_connections 2021-02-10T11:28:18.358 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:18.356 7f95b028e700 1 -- 172.21.2.102:0/3657625887 wait complete. 2021-02-10T11:28:18.365 DEBUG:teuthology.orchestra.run.gibba013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok session ls 2021-02-10T11:28:18.762 INFO:teuthology.orchestra.run.gibba013.stdout:[ 2021-02-10T11:28:18.763 INFO:teuthology.orchestra.run.gibba013.stdout: { 2021-02-10T11:28:18.763 INFO:teuthology.orchestra.run.gibba013.stdout: "id": 9471, 2021-02-10T11:28:18.763 INFO:teuthology.orchestra.run.gibba013.stdout: "entity": { 2021-02-10T11:28:18.763 INFO:teuthology.orchestra.run.gibba013.stdout: "name": { 2021-02-10T11:28:18.763 INFO:teuthology.orchestra.run.gibba013.stdout: "type": "client", 2021-02-10T11:28:18.763 INFO:teuthology.orchestra.run.gibba013.stdout: "num": 9471 2021-02-10T11:28:18.763 INFO:teuthology.orchestra.run.gibba013.stdout: }, 2021-02-10T11:28:18.763 INFO:teuthology.orchestra.run.gibba013.stdout: "addr": { 2021-02-10T11:28:18.763 INFO:teuthology.orchestra.run.gibba013.stdout: "type": "any", 2021-02-10T11:28:18.763 INFO:teuthology.orchestra.run.gibba013.stdout: "addr": "172.21.2.102:0", 2021-02-10T11:28:18.763 INFO:teuthology.orchestra.run.gibba013.stdout: "nonce": 190431005 2021-02-10T11:28:18.764 INFO:teuthology.orchestra.run.gibba013.stdout: } 2021-02-10T11:28:18.764 INFO:teuthology.orchestra.run.gibba013.stdout: }, 2021-02-10T11:28:18.764 INFO:teuthology.orchestra.run.gibba013.stdout: "state": "open", 2021-02-10T11:28:18.764 INFO:teuthology.orchestra.run.gibba013.stdout: "num_leases": 1, 2021-02-10T11:28:18.764 INFO:teuthology.orchestra.run.gibba013.stdout: "num_caps": 7, 2021-02-10T11:28:18.764 INFO:teuthology.orchestra.run.gibba013.stdout: "request_load_avg": 12, 2021-02-10T11:28:18.764 INFO:teuthology.orchestra.run.gibba013.stdout: "uptime": 8.0850003959999999, 2021-02-10T11:28:18.764 INFO:teuthology.orchestra.run.gibba013.stdout: "requests_in_flight": 0, 2021-02-10T11:28:18.764 INFO:teuthology.orchestra.run.gibba013.stdout: "completed_requests": 1, 2021-02-10T11:28:18.764 INFO:teuthology.orchestra.run.gibba013.stdout: "reconnecting": false, 2021-02-10T11:28:18.764 INFO:teuthology.orchestra.run.gibba013.stdout: "recall_caps": { 2021-02-10T11:28:18.764 INFO:teuthology.orchestra.run.gibba013.stdout: "value": 0, 2021-02-10T11:28:18.765 INFO:teuthology.orchestra.run.gibba013.stdout: "halflife": 60 2021-02-10T11:28:18.765 INFO:teuthology.orchestra.run.gibba013.stdout: }, 2021-02-10T11:28:18.765 INFO:teuthology.orchestra.run.gibba013.stdout: "release_caps": { 2021-02-10T11:28:18.765 INFO:teuthology.orchestra.run.gibba013.stdout: "value": 0, 2021-02-10T11:28:18.765 INFO:teuthology.orchestra.run.gibba013.stdout: "halflife": 60 2021-02-10T11:28:18.765 INFO:teuthology.orchestra.run.gibba013.stdout: }, 2021-02-10T11:28:18.765 INFO:teuthology.orchestra.run.gibba013.stdout: "recall_caps_throttle": { 2021-02-10T11:28:18.765 INFO:teuthology.orchestra.run.gibba013.stdout: "value": 0, 2021-02-10T11:28:18.765 INFO:teuthology.orchestra.run.gibba013.stdout: "halflife": 2.5 2021-02-10T11:28:18.765 INFO:teuthology.orchestra.run.gibba013.stdout: }, 2021-02-10T11:28:18.765 INFO:teuthology.orchestra.run.gibba013.stdout: "recall_caps_throttle2o": { 2021-02-10T11:28:18.765 INFO:teuthology.orchestra.run.gibba013.stdout: "value": 0, 2021-02-10T11:28:18.766 INFO:teuthology.orchestra.run.gibba013.stdout: "halflife": 0.5 2021-02-10T11:28:18.766 INFO:teuthology.orchestra.run.gibba013.stdout: }, 2021-02-10T11:28:18.766 INFO:teuthology.orchestra.run.gibba013.stdout: "session_cache_liveness": { 2021-02-10T11:28:18.766 INFO:teuthology.orchestra.run.gibba013.stdout: "value": 20.625536577546239, 2021-02-10T11:28:18.766 INFO:teuthology.orchestra.run.gibba013.stdout: "halflife": 300 2021-02-10T11:28:18.766 INFO:teuthology.orchestra.run.gibba013.stdout: }, 2021-02-10T11:28:18.766 INFO:teuthology.orchestra.run.gibba013.stdout: "cap_acquisition": { 2021-02-10T11:28:18.766 INFO:teuthology.orchestra.run.gibba013.stdout: "value": 0, 2021-02-10T11:28:18.766 INFO:teuthology.orchestra.run.gibba013.stdout: "halflife": 10 2021-02-10T11:28:18.766 INFO:teuthology.orchestra.run.gibba013.stdout: }, 2021-02-10T11:28:18.766 INFO:teuthology.orchestra.run.gibba013.stdout: "inst": "client.9471 172.21.2.102:0/190431005", 2021-02-10T11:28:18.766 INFO:teuthology.orchestra.run.gibba013.stdout: "completed_requests": [ 2021-02-10T11:28:18.766 INFO:teuthology.orchestra.run.gibba013.stdout: { 2021-02-10T11:28:18.767 INFO:teuthology.orchestra.run.gibba013.stdout: "tid": 14, 2021-02-10T11:28:18.767 INFO:teuthology.orchestra.run.gibba013.stdout: "created_ino": "0x10000000005" 2021-02-10T11:28:18.767 INFO:teuthology.orchestra.run.gibba013.stdout: } 2021-02-10T11:28:18.767 INFO:teuthology.orchestra.run.gibba013.stdout: ], 2021-02-10T11:28:18.767 INFO:teuthology.orchestra.run.gibba013.stdout: "prealloc_inos": [ 2021-02-10T11:28:18.767 INFO:teuthology.orchestra.run.gibba013.stdout: { 2021-02-10T11:28:18.767 INFO:teuthology.orchestra.run.gibba013.stdout: "start": 1099511627782, 2021-02-10T11:28:18.767 INFO:teuthology.orchestra.run.gibba013.stdout: "length": 995 2021-02-10T11:28:18.767 INFO:teuthology.orchestra.run.gibba013.stdout: } 2021-02-10T11:28:18.767 INFO:teuthology.orchestra.run.gibba013.stdout: ], 2021-02-10T11:28:18.767 INFO:teuthology.orchestra.run.gibba013.stdout: "used_inos": [], 2021-02-10T11:28:18.767 INFO:teuthology.orchestra.run.gibba013.stdout: "client_metadata": { 2021-02-10T11:28:18.767 INFO:teuthology.orchestra.run.gibba013.stdout: "features": "0x0000000000001fff", 2021-02-10T11:28:18.768 INFO:teuthology.orchestra.run.gibba013.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:28:18.768 INFO:teuthology.orchestra.run.gibba013.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:28:18.768 INFO:teuthology.orchestra.run.gibba013.stdout: "entity_id": "y", 2021-02-10T11:28:18.768 INFO:teuthology.orchestra.run.gibba013.stdout: "hostname": "gibba002", 2021-02-10T11:28:18.768 INFO:teuthology.orchestra.run.gibba013.stdout: "pid": "35161", 2021-02-10T11:28:18.768 INFO:teuthology.orchestra.run.gibba013.stdout: "root": "/" 2021-02-10T11:28:18.768 INFO:teuthology.orchestra.run.gibba013.stdout: } 2021-02-10T11:28:18.768 INFO:teuthology.orchestra.run.gibba013.stdout: } 2021-02-10T11:28:18.768 INFO:teuthology.orchestra.run.gibba013.stdout:] 2021-02-10T11:28:18.770 INFO:tasks.cephfs.filesystem:_json_asok output: [ { "id": 9471, "entity": { "name": { "type": "client", "num": 9471 }, "addr": { "type": "any", "addr": "172.21.2.102:0", "nonce": 190431005 } }, "state": "open", "num_leases": 1, "num_caps": 7, "request_load_avg": 12, "uptime": 8.0850003959999999, "requests_in_flight": 0, "completed_requests": 1, "reconnecting": false, "recall_caps": { "value": 0, "halflife": 60 }, "release_caps": { "value": 0, "halflife": 60 }, "recall_caps_throttle": { "value": 0, "halflife": 2.5 }, "recall_caps_throttle2o": { "value": 0, "halflife": 0.5 }, "session_cache_liveness": { "value": 20.625536577546239, "halflife": 300 }, "cap_acquisition": { "value": 0, "halflife": 10 }, "inst": "client.9471 172.21.2.102:0/190431005", "completed_requests": [ { "tid": 14, "created_ino": "0x10000000005" } ], "prealloc_inos": [ { "start": 1099511627782, "length": 995 } ], "used_inos": [], "client_metadata": { "features": "0x0000000000001fff", "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", "entity_id": "y", "hostname": "gibba002", "pid": "35161", "root": "/" } } ] 2021-02-10T11:28:18.770 DEBUG:tasks.ceph_test_case:wait_until_true: waiting... 2021-02-10T11:28:23.771 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:28:24.200 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:24.182 7fe78cff9700 1 Processor -- start 2021-02-10T11:28:24.201 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:24.182 7fe78cff9700 1 -- start start 2021-02-10T11:28:24.201 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:24.183 7fe78cff9700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe7881b0870 0x7fe7880f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:24.201 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:24.183 7fe78cff9700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7881b0c70 con 0x7fe7880efe00 2021-02-10T11:28:24.201 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:24.183 7fe78cff9700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7880f0b50 con 0x7fe7881b0470 2021-02-10T11:28:24.201 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:24.183 7fe78cff9700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7880e5ea0 con 0x7fe7881b0870 2021-02-10T11:28:24.201 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:24.183 7fe78759e700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fe7880efe00 0x7fe7880f0200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:52422/0 (socket says 172.21.2.102:52422) 2021-02-10T11:28:24.201 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:24.183 7fe78759e700 1 -- 172.21.2.102:0/1622833369 learned_addr learned my addr 172.21.2.102:0/1622833369 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:28:24.201 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:24.184 7fe785d9b700 1 -- 172.21.2.102:0/1622833369 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 796226912 0 0) 0x7fe780001500 con 0x7fe7880efe00 2021-02-10T11:28:24.201 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:24.184 7fe785d9b700 1 -- 172.21.2.102:0/1622833369 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe774002fe0 con 0x7fe7880efe00 2021-02-10T11:28:24.201 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:24.184 7fe785d9b700 1 -- 172.21.2.102:0/1622833369 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3444848877 0 0) 0x7fe780001460 con 0x7fe7880efe00 2021-02-10T11:28:24.201 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:24.184 7fe785d9b700 1 -- 172.21.2.102:0/1622833369 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe7881b0870 msgr2=0x7fe7880f20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:28:24.201 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:24.184 7fe785d9b700 1 --2- 172.21.2.102:0/1622833369 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe7881b0870 0x7fe7880f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:28:24.201 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:24.184 7fe785d9b700 1 -- 172.21.2.102:0/1622833369 >> v1:172.21.2.113:6789/0 conn(0x7fe7881b0470 legacy=0x7fe7880ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:28:24.202 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:24.184 7fe785d9b700 1 -- 172.21.2.102:0/1622833369 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7880e52e0 con 0x7fe7880efe00 2021-02-10T11:28:24.202 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:24.184 7fe785d9b700 1 -- 172.21.2.102:0/1622833369 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7fe780001980 con 0x7fe7880efe00 2021-02-10T11:28:24.202 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:24.184 7fe785d9b700 1 -- 172.21.2.102:0/1622833369 >> v1:172.21.2.102:6789/0 conn(0x7fe7880efe00 legacy=0x7fe7880f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:24.202 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:24.184 7fe785d9b700 1 --2- 172.21.2.102:0/1622833369 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe774003a40 0x7fe774005eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:24.202 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:24.184 7fe785d9b700 1 --2- 172.21.2.102:0/1622833369 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe774006750 0x7fe774008c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:24.202 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:24.184 7fe785d9b700 1 --2- 172.21.2.102:0/1622833369 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe7740094f0 0x7fe77400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:24.202 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:24.184 7fe785d9b700 1 -- 172.21.2.102:0/1622833369 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe77400c2c0 con 0x7fe7740094f0 2021-02-10T11:28:24.202 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:24.184 7fe785d9b700 1 -- 172.21.2.102:0/1622833369 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe77400c4e0 con 0x7fe774006750 2021-02-10T11:28:24.202 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:24.184 7fe785d9b700 1 -- 172.21.2.102:0/1622833369 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe77400c700 con 0x7fe774003a40 2021-02-10T11:28:24.202 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:24.184 7fe78759e700 1 --2- 172.21.2.102:0/1622833369 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe7740094f0 0x7fe77400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:24.202 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:24.184 7fe78659c700 1 --2- 172.21.2.102:0/1622833369 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe774006750 0x7fe774008c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:24.202 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:24.184 7fe786d9d700 1 --2- 172.21.2.102:0/1622833369 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe774003a40 0x7fe774005eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:24.202 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:24.184 7fe78759e700 1 -- 172.21.2.102:0/1622833369 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe774003a40 msgr2=0x7fe774005eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:28:24.203 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:24.184 7fe78759e700 1 --2- 172.21.2.102:0/1622833369 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe774003a40 0x7fe774005eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:24.203 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:24.184 7fe78759e700 1 -- 172.21.2.102:0/1622833369 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe774006750 msgr2=0x7fe774008c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:28:24.203 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:24.184 7fe78759e700 1 --2- 172.21.2.102:0/1622833369 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe774006750 0x7fe774008c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:24.203 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:24.184 7fe78759e700 1 -- 172.21.2.102:0/1622833369 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe77400c9a0 con 0x7fe7740094f0 2021-02-10T11:28:24.203 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:24.185 7fe78759e700 1 --2- 172.21.2.102:0/1622833369 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe7740094f0 0x7fe77400ba10 secure :-1 s=READY pgs=1849 cs=0 l=1 rev1=1 rx=0x7fe780001c20 tx=0x7fe780050310).ready entity=mon.0 client_cookie=a61bff70d7d11f98 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:28:24.203 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:24.185 7fe785d9b700 1 -- 172.21.2.102:0/1622833369 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe780007020 con 0x7fe7740094f0 2021-02-10T11:28:24.203 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:24.185 7fe785d9b700 1 -- 172.21.2.102:0/1622833369 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe780007320 con 0x7fe7740094f0 2021-02-10T11:28:24.203 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:24.185 7fe785d9b700 1 -- 172.21.2.102:0/1622833369 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe780004720 con 0x7fe7740094f0 2021-02-10T11:28:24.203 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:24.185 7fe78cff9700 1 -- 172.21.2.102:0/1622833369 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe7740094f0 msgr2=0x7fe77400ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:24.203 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:24.185 7fe78cff9700 1 --2- 172.21.2.102:0/1622833369 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe7740094f0 0x7fe77400ba10 secure :-1 s=READY pgs=1849 cs=0 l=1 rev1=1 rx=0x7fe780001c20 tx=0x7fe780050310).stop 2021-02-10T11:28:24.203 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:24.185 7fe78759e700 1 -- 172.21.2.102:0/1622833369 reap_dead start 2021-02-10T11:28:24.203 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:24.186 7fe78cff9700 1 -- 172.21.2.102:0/1622833369 shutdown_connections 2021-02-10T11:28:24.203 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:24.186 7fe78cff9700 1 -- 172.21.2.102:0/1622833369 >> 172.21.2.102:0/1622833369 conn(0x7fe7880ce3e0 msgr2=0x7fe7880e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:28:24.204 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:24.186 7fe78cff9700 1 -- 172.21.2.102:0/1622833369 shutdown_connections 2021-02-10T11:28:24.204 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:24.186 7fe78cff9700 1 -- 172.21.2.102:0/1622833369 wait complete. 2021-02-10T11:28:24.205 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:24.201 7fe78cff9700 1 Processor -- start 2021-02-10T11:28:24.205 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:24.201 7fe78cff9700 1 -- start start 2021-02-10T11:28:24.205 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:24.201 7fe78cff9700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe7880938f0 0x7fe7881a4310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:24.205 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:24.201 7fe78cff9700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe788093cf0 0x7fe788092690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:24.205 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:24.201 7fe78cff9700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe788092cb0 0x7fe7880930b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:24.205 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:24.201 7fe78cff9700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7880b8300 con 0x7fe7880938f0 2021-02-10T11:28:24.205 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:24.201 7fe78cff9700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7880b7f40 con 0x7fe788093cf0 2021-02-10T11:28:24.206 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:24.201 7fe78cff9700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7881a4930 con 0x7fe788092cb0 2021-02-10T11:28:24.206 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:24.201 7fe786d9d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe7880938f0 0x7fe7881a4310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:24.206 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:24.201 7fe786d9d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe7880938f0 0x7fe7881a4310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:60206/0 (socket says 172.21.2.102:60206) 2021-02-10T11:28:24.206 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:24.201 7fe786d9d700 1 -- 172.21.2.102:0/1828900910 learned_addr learned my addr 172.21.2.102:0/1828900910 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:28:24.206 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:24.201 7fe786d9d700 1 -- 172.21.2.102:0/1828900910 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe788092cb0 msgr2=0x7fe7880930b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:28:24.206 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:24.201 7fe786d9d700 1 --2- 172.21.2.102:0/1828900910 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe788092cb0 0x7fe7880930b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:28:24.206 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:24.201 7fe786d9d700 1 -- 172.21.2.102:0/1828900910 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe788093cf0 msgr2=0x7fe788092690 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:28:24.206 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:24.201 7fe786d9d700 1 --2- 172.21.2.102:0/1828900910 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe788093cf0 0x7fe788092690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:28:24.206 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:24.201 7fe786d9d700 1 -- 172.21.2.102:0/1828900910 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7880ad530 con 0x7fe7880938f0 2021-02-10T11:28:24.206 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:24.202 7fe786d9d700 1 --2- 172.21.2.102:0/1828900910 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe7880938f0 0x7fe7881a4310 secure :-1 s=READY pgs=1850 cs=0 l=1 rev1=1 rx=0x7fe770006990 tx=0x7fe77000b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:28:24.206 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:24.202 7fe77f7fe700 1 -- 172.21.2.102:0/1828900910 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe770006a80 con 0x7fe7880938f0 2021-02-10T11:28:24.206 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:24.202 7fe77f7fe700 1 -- 172.21.2.102:0/1828900910 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe770000c60 con 0x7fe7880938f0 2021-02-10T11:28:24.206 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:24.202 7fe78cff9700 1 -- 172.21.2.102:0/1828900910 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7880ad530 con 0x7fe7880938f0 2021-02-10T11:28:24.207 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:24.202 7fe78cff9700 1 -- 172.21.2.102:0/1828900910 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7880adac0 con 0x7fe7880938f0 2021-02-10T11:28:24.207 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:24.202 7fe77f7fe700 1 -- 172.21.2.102:0/1828900910 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe7700022c0 con 0x7fe7880938f0 2021-02-10T11:28:24.207 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:24.203 7fe77f7fe700 1 -- 172.21.2.102:0/1828900910 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fe770006a80 con 0x7fe7880938f0 2021-02-10T11:28:24.207 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:24.203 7fe77f7fe700 1 --2- 172.21.2.102:0/1828900910 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe76c018730 0x7fe76c01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:24.207 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:24.203 7fe77f7fe700 1 -- 172.21.2.102:0/1828900910 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fe7700326b0 con 0x7fe7880938f0 2021-02-10T11:28:24.207 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:24.203 7fe78659c700 1 --2- 172.21.2.102:0/1828900910 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe76c018730 0x7fe76c01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:24.207 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:24.203 7fe78659c700 1 --2- 172.21.2.102:0/1828900910 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe76c018730 0x7fe76c01ac10 crc :-1 s=READY pgs=830 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:28:24.233 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:24.233 7fe78cff9700 1 -- 172.21.2.102:0/1828900910 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe788105a80 con 0x7fe7880938f0 2021-02-10T11:28:24.236 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:24.236 7fe77f7fe700 1 -- 172.21.2.102:0/1828900910 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fe770002560 con 0x7fe7880938f0 2021-02-10T11:28:24.364 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:24.364 7fe78cff9700 1 -- 172.21.2.102:0/1828900910 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe7880f0b50 con 0x7fe7880938f0 2021-02-10T11:28:24.365 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:24.364 7fe77f7fe700 1 -- 172.21.2.102:0/1828900910 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+2101 (secure 0 0 0) 0x7fe770006a80 con 0x7fe7880938f0 2021-02-10T11:28:24.365 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:28:24.365 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":66,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6110,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.102:6835/1471871278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6834","nonce":1471871278},{"type":"v1","addr":"172.21.2.102:6835","nonce":1471871278}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":61}],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:27:41.479375","modified":"2021-02-10 11:27:43.490605","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6107},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6107":{"gid":6107,"name":"b","rank":0,"incarnation":62,"state":"up:active","state_seq":3,"addr":"172.21.2.113:6833/4220238062","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6832","nonce":4220238062},{"type":"v1","addr":"172.21.2.113:6833","nonce":4220238062}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":8}]} 2021-02-10T11:28:24.366 INFO:teuthology.orchestra.run.gibba002.stderr:dumped fsmap epoch 66 2021-02-10T11:28:24.367 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:24.366 7fe78cff9700 1 -- 172.21.2.102:0/1828900910 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe76c018730 msgr2=0x7fe76c01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:24.367 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:24.366 7fe78cff9700 1 --2- 172.21.2.102:0/1828900910 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe76c018730 0x7fe76c01ac10 crc :-1 s=READY pgs=830 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:24.367 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:24.366 7fe78cff9700 1 -- 172.21.2.102:0/1828900910 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe7880938f0 msgr2=0x7fe7881a4310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:24.367 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:24.366 7fe78cff9700 1 --2- 172.21.2.102:0/1828900910 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe7880938f0 0x7fe7881a4310 secure :-1 s=READY pgs=1850 cs=0 l=1 rev1=1 rx=0x7fe770006990 tx=0x7fe77000b990).stop 2021-02-10T11:28:24.367 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:24.366 7fe78cff9700 1 -- 172.21.2.102:0/1828900910 shutdown_connections 2021-02-10T11:28:24.367 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:24.367 7fe78cff9700 1 --2- 172.21.2.102:0/1828900910 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe788092cb0 0x7fe7880930b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:28:24.367 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:24.367 7fe78cff9700 1 --2- 172.21.2.102:0/1828900910 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe76c018730 0x7fe76c01ac10 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:24.367 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:24.367 7fe78cff9700 1 --2- 172.21.2.102:0/1828900910 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe788093cf0 0x7fe788092690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:28:24.367 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:24.367 7fe78cff9700 1 --2- 172.21.2.102:0/1828900910 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe7880938f0 0x7fe7881a4310 unknown :-1 s=CLOSED pgs=1850 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:24.368 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:24.367 7fe78cff9700 1 -- 172.21.2.102:0/1828900910 >> 172.21.2.102:0/1828900910 conn(0x7fe7880a1dc0 msgr2=0x7fe78808e860 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:28:24.368 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:24.367 7fe78cff9700 1 -- 172.21.2.102:0/1828900910 shutdown_connections 2021-02-10T11:28:24.368 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:24.367 7fe78cff9700 1 -- 172.21.2.102:0/1828900910 wait complete. 2021-02-10T11:28:24.377 DEBUG:teuthology.orchestra.run.gibba013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok session ls 2021-02-10T11:28:24.775 INFO:teuthology.orchestra.run.gibba013.stdout:[ 2021-02-10T11:28:24.775 INFO:teuthology.orchestra.run.gibba013.stdout: { 2021-02-10T11:28:24.775 INFO:teuthology.orchestra.run.gibba013.stdout: "id": 9471, 2021-02-10T11:28:24.775 INFO:teuthology.orchestra.run.gibba013.stdout: "entity": { 2021-02-10T11:28:24.775 INFO:teuthology.orchestra.run.gibba013.stdout: "name": { 2021-02-10T11:28:24.775 INFO:teuthology.orchestra.run.gibba013.stdout: "type": "client", 2021-02-10T11:28:24.775 INFO:teuthology.orchestra.run.gibba013.stdout: "num": 9471 2021-02-10T11:28:24.775 INFO:teuthology.orchestra.run.gibba013.stdout: }, 2021-02-10T11:28:24.776 INFO:teuthology.orchestra.run.gibba013.stdout: "addr": { 2021-02-10T11:28:24.776 INFO:teuthology.orchestra.run.gibba013.stdout: "type": "any", 2021-02-10T11:28:24.776 INFO:teuthology.orchestra.run.gibba013.stdout: "addr": "172.21.2.102:0", 2021-02-10T11:28:24.776 INFO:teuthology.orchestra.run.gibba013.stdout: "nonce": 190431005 2021-02-10T11:28:24.776 INFO:teuthology.orchestra.run.gibba013.stdout: } 2021-02-10T11:28:24.776 INFO:teuthology.orchestra.run.gibba013.stdout: }, 2021-02-10T11:28:24.776 INFO:teuthology.orchestra.run.gibba013.stdout: "state": "open", 2021-02-10T11:28:24.776 INFO:teuthology.orchestra.run.gibba013.stdout: "num_leases": 1, 2021-02-10T11:28:24.776 INFO:teuthology.orchestra.run.gibba013.stdout: "num_caps": 7, 2021-02-10T11:28:24.776 INFO:teuthology.orchestra.run.gibba013.stdout: "request_load_avg": 11, 2021-02-10T11:28:24.776 INFO:teuthology.orchestra.run.gibba013.stdout: "uptime": 14.098000668999999, 2021-02-10T11:28:24.776 INFO:teuthology.orchestra.run.gibba013.stdout: "requests_in_flight": 0, 2021-02-10T11:28:24.776 INFO:teuthology.orchestra.run.gibba013.stdout: "completed_requests": 1, 2021-02-10T11:28:24.777 INFO:teuthology.orchestra.run.gibba013.stdout: "reconnecting": false, 2021-02-10T11:28:24.777 INFO:teuthology.orchestra.run.gibba013.stdout: "recall_caps": { 2021-02-10T11:28:24.777 INFO:teuthology.orchestra.run.gibba013.stdout: "value": 0, 2021-02-10T11:28:24.777 INFO:teuthology.orchestra.run.gibba013.stdout: "halflife": 60 2021-02-10T11:28:24.777 INFO:teuthology.orchestra.run.gibba013.stdout: }, 2021-02-10T11:28:24.777 INFO:teuthology.orchestra.run.gibba013.stdout: "release_caps": { 2021-02-10T11:28:24.777 INFO:teuthology.orchestra.run.gibba013.stdout: "value": 0, 2021-02-10T11:28:24.777 INFO:teuthology.orchestra.run.gibba013.stdout: "halflife": 60 2021-02-10T11:28:24.777 INFO:teuthology.orchestra.run.gibba013.stdout: }, 2021-02-10T11:28:24.777 INFO:teuthology.orchestra.run.gibba013.stdout: "recall_caps_throttle": { 2021-02-10T11:28:24.777 INFO:teuthology.orchestra.run.gibba013.stdout: "value": 0, 2021-02-10T11:28:24.777 INFO:teuthology.orchestra.run.gibba013.stdout: "halflife": 2.5 2021-02-10T11:28:24.777 INFO:teuthology.orchestra.run.gibba013.stdout: }, 2021-02-10T11:28:24.778 INFO:teuthology.orchestra.run.gibba013.stdout: "recall_caps_throttle2o": { 2021-02-10T11:28:24.778 INFO:teuthology.orchestra.run.gibba013.stdout: "value": 0, 2021-02-10T11:28:24.778 INFO:teuthology.orchestra.run.gibba013.stdout: "halflife": 0.5 2021-02-10T11:28:24.778 INFO:teuthology.orchestra.run.gibba013.stdout: }, 2021-02-10T11:28:24.778 INFO:teuthology.orchestra.run.gibba013.stdout: "session_cache_liveness": { 2021-02-10T11:28:24.778 INFO:teuthology.orchestra.run.gibba013.stdout: "value": 20.340967729501621, 2021-02-10T11:28:24.778 INFO:teuthology.orchestra.run.gibba013.stdout: "halflife": 300 2021-02-10T11:28:24.778 INFO:teuthology.orchestra.run.gibba013.stdout: }, 2021-02-10T11:28:24.778 INFO:teuthology.orchestra.run.gibba013.stdout: "cap_acquisition": { 2021-02-10T11:28:24.778 INFO:teuthology.orchestra.run.gibba013.stdout: "value": 0, 2021-02-10T11:28:24.779 INFO:teuthology.orchestra.run.gibba013.stdout: "halflife": 10 2021-02-10T11:28:24.779 INFO:teuthology.orchestra.run.gibba013.stdout: }, 2021-02-10T11:28:24.779 INFO:teuthology.orchestra.run.gibba013.stdout: "inst": "client.9471 172.21.2.102:0/190431005", 2021-02-10T11:28:24.779 INFO:teuthology.orchestra.run.gibba013.stdout: "completed_requests": [ 2021-02-10T11:28:24.779 INFO:teuthology.orchestra.run.gibba013.stdout: { 2021-02-10T11:28:24.779 INFO:teuthology.orchestra.run.gibba013.stdout: "tid": 14, 2021-02-10T11:28:24.779 INFO:teuthology.orchestra.run.gibba013.stdout: "created_ino": "0x10000000005" 2021-02-10T11:28:24.779 INFO:teuthology.orchestra.run.gibba013.stdout: } 2021-02-10T11:28:24.779 INFO:teuthology.orchestra.run.gibba013.stdout: ], 2021-02-10T11:28:24.779 INFO:teuthology.orchestra.run.gibba013.stdout: "prealloc_inos": [ 2021-02-10T11:28:24.779 INFO:teuthology.orchestra.run.gibba013.stdout: { 2021-02-10T11:28:24.779 INFO:teuthology.orchestra.run.gibba013.stdout: "start": 1099511627782, 2021-02-10T11:28:24.779 INFO:teuthology.orchestra.run.gibba013.stdout: "length": 995 2021-02-10T11:28:24.780 INFO:teuthology.orchestra.run.gibba013.stdout: } 2021-02-10T11:28:24.780 INFO:teuthology.orchestra.run.gibba013.stdout: ], 2021-02-10T11:28:24.780 INFO:teuthology.orchestra.run.gibba013.stdout: "used_inos": [], 2021-02-10T11:28:24.780 INFO:teuthology.orchestra.run.gibba013.stdout: "client_metadata": { 2021-02-10T11:28:24.780 INFO:teuthology.orchestra.run.gibba013.stdout: "features": "0x0000000000001fff", 2021-02-10T11:28:24.780 INFO:teuthology.orchestra.run.gibba013.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:28:24.780 INFO:teuthology.orchestra.run.gibba013.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:28:24.780 INFO:teuthology.orchestra.run.gibba013.stdout: "entity_id": "y", 2021-02-10T11:28:24.780 INFO:teuthology.orchestra.run.gibba013.stdout: "hostname": "gibba002", 2021-02-10T11:28:24.780 INFO:teuthology.orchestra.run.gibba013.stdout: "pid": "35161", 2021-02-10T11:28:24.780 INFO:teuthology.orchestra.run.gibba013.stdout: "root": "/" 2021-02-10T11:28:24.781 INFO:teuthology.orchestra.run.gibba013.stdout: } 2021-02-10T11:28:24.781 INFO:teuthology.orchestra.run.gibba013.stdout: } 2021-02-10T11:28:24.781 INFO:teuthology.orchestra.run.gibba013.stdout:] 2021-02-10T11:28:24.782 INFO:tasks.cephfs.filesystem:_json_asok output: [ { "id": 9471, "entity": { "name": { "type": "client", "num": 9471 }, "addr": { "type": "any", "addr": "172.21.2.102:0", "nonce": 190431005 } }, "state": "open", "num_leases": 1, "num_caps": 7, "request_load_avg": 11, "uptime": 14.098000668999999, "requests_in_flight": 0, "completed_requests": 1, "reconnecting": false, "recall_caps": { "value": 0, "halflife": 60 }, "release_caps": { "value": 0, "halflife": 60 }, "recall_caps_throttle": { "value": 0, "halflife": 2.5 }, "recall_caps_throttle2o": { "value": 0, "halflife": 0.5 }, "session_cache_liveness": { "value": 20.340967729501621, "halflife": 300 }, "cap_acquisition": { "value": 0, "halflife": 10 }, "inst": "client.9471 172.21.2.102:0/190431005", "completed_requests": [ { "tid": 14, "created_ino": "0x10000000005" } ], "prealloc_inos": [ { "start": 1099511627782, "length": 995 } ], "used_inos": [], "client_metadata": { "features": "0x0000000000001fff", "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", "entity_id": "y", "hostname": "gibba002", "pid": "35161", "root": "/" } } ] 2021-02-10T11:28:24.782 DEBUG:tasks.ceph_test_case:wait_until_true: waiting... 2021-02-10T11:28:29.782 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:28:30.210 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:30.193 7f85f3d34700 1 Processor -- start 2021-02-10T11:28:30.210 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:30.193 7f85f3d34700 1 -- start start 2021-02-10T11:28:30.210 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:30.194 7f85f3d34700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f85ec0d0340 0x7f85ec0ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:30.210 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:30.194 7f85f3d34700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f85ec0d1c70 0x7f85ec0efc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:30.210 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:30.194 7f85f3d34700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85ec0f0b10 con 0x7f85ec0d1870 2021-02-10T11:28:30.210 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:30.194 7f85f3d34700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85ec0e5ed0 con 0x7f85ec0d1c70 2021-02-10T11:28:30.210 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:30.194 7f85f3d34700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85ec1b0be0 con 0x7f85ec0d0340 2021-02-10T11:28:30.210 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:30.194 7f85f2351700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f85ec0d1c70 0x7f85ec0efc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:30.210 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:30.194 7f85f2351700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f85ec0d1c70 0x7f85ec0efc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:60218/0 (socket says 172.21.2.102:60218) 2021-02-10T11:28:30.211 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:30.194 7f85f1b50700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f85ec0d1870 0x7f85ec1b03f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:52438/0 (socket says 172.21.2.102:52438) 2021-02-10T11:28:30.211 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:30.194 7f85f2351700 1 -- 172.21.2.102:0/1412389754 learned_addr learned my addr 172.21.2.102:0/1412389754 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:28:30.211 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:30.194 7f85f2351700 1 -- 172.21.2.102:0/1412389754 >> v2:172.21.2.113:3300/0 conn(0x7f85ec0d0340 msgr2=0x7f85ec0ef640 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:28:30.211 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:30.194 7f85f2351700 1 --2- 172.21.2.102:0/1412389754 >> v2:172.21.2.113:3300/0 conn(0x7f85ec0d0340 0x7f85ec0ef640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:28:30.211 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:30.194 7f85f2351700 1 -- 172.21.2.102:0/1412389754 >> v1:172.21.2.102:6789/0 conn(0x7f85ec0d1870 legacy=0x7f85ec1b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:30.211 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:30.194 7f85f2351700 1 -- 172.21.2.102:0/1412389754 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85ec0f20a0 con 0x7f85ec0d1c70 2021-02-10T11:28:30.211 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:30.195 7f85f2351700 1 --2- 172.21.2.102:0/1412389754 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f85ec0d1c70 0x7f85ec0efc60 secure :-1 s=READY pgs=1852 cs=0 l=1 rev1=1 rx=0x7f85e4006f00 tx=0x7f85e4009f10).ready entity=mon.0 client_cookie=793671a465e5cba1 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:28:30.211 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:30.195 7f85f0b4e700 1 -- 172.21.2.102:0/1412389754 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f85e400b3b0 con 0x7f85ec0d1c70 2021-02-10T11:28:30.211 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:30.195 7f85f0b4e700 1 -- 172.21.2.102:0/1412389754 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85e400b630 con 0x7f85ec0d1c70 2021-02-10T11:28:30.211 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:30.195 7f85f0b4e700 1 -- 172.21.2.102:0/1412389754 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f85e400bb50 con 0x7f85ec0d1c70 2021-02-10T11:28:30.211 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:30.195 7f85f3d34700 1 -- 172.21.2.102:0/1412389754 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f85ec0d1c70 msgr2=0x7f85ec0efc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:30.211 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:30.195 7f85f3d34700 1 --2- 172.21.2.102:0/1412389754 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f85ec0d1c70 0x7f85ec0efc60 secure :-1 s=READY pgs=1852 cs=0 l=1 rev1=1 rx=0x7f85e4006f00 tx=0x7f85e4009f10).stop 2021-02-10T11:28:30.211 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:30.195 7f85f3d34700 1 -- 172.21.2.102:0/1412389754 shutdown_connections 2021-02-10T11:28:30.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:30.195 7f85f3d34700 1 --2- 172.21.2.102:0/1412389754 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f85ec0d1c70 0x7f85ec0efc60 unknown :-1 s=CLOSED pgs=1852 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:30.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:30.195 7f85f3d34700 1 --2- 172.21.2.102:0/1412389754 >> v2:172.21.2.113:3300/0 conn(0x7f85ec0d0340 0x7f85ec0ef640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:28:30.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:30.195 7f85f3d34700 1 -- 172.21.2.102:0/1412389754 >> 172.21.2.102:0/1412389754 conn(0x7f85ec0ce450 msgr2=0x7f85ec0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:28:30.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:30.196 7f85f3d34700 1 -- 172.21.2.102:0/1412389754 shutdown_connections 2021-02-10T11:28:30.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:30.196 7f85f3d34700 1 -- 172.21.2.102:0/1412389754 wait complete. 2021-02-10T11:28:30.213 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:30.210 7f85f3d34700 1 Processor -- start 2021-02-10T11:28:30.214 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:30.210 7f85f3d34700 1 -- start start 2021-02-10T11:28:30.214 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:30.210 7f85f3d34700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f85ec1b8ad0 0x7f85ec187e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:30.214 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:30.210 7f85f3d34700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f85ec1b8ed0 0x7f85ec0b1af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:30.214 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:30.210 7f85f3d34700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f85ec0b2110 0x7f85ec1b7a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:30.214 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:30.210 7f85f3d34700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85ec188470 con 0x7f85ec1b8ed0 2021-02-10T11:28:30.214 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:30.210 7f85f3d34700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85ec18be10 con 0x7f85ec1b8ad0 2021-02-10T11:28:30.214 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:30.210 7f85f3d34700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85ec18c030 con 0x7f85ec0b2110 2021-02-10T11:28:30.214 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:30.211 7f85f134f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f85ec1b8ed0 0x7f85ec0b1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:30.214 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:30.211 7f85f134f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f85ec1b8ed0 0x7f85ec0b1af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:60224/0 (socket says 172.21.2.102:60224) 2021-02-10T11:28:30.214 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:30.211 7f85f134f700 1 -- 172.21.2.102:0/3869881269 learned_addr learned my addr 172.21.2.102:0/3869881269 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:28:30.214 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:30.211 7f85f2351700 1 --2- 172.21.2.102:0/3869881269 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f85ec0b2110 0x7f85ec1b7a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:30.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:30.211 7f85f1b50700 1 --2- 172.21.2.102:0/3869881269 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f85ec1b8ad0 0x7f85ec187e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:30.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:30.211 7f85f134f700 1 -- 172.21.2.102:0/3869881269 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f85ec0b2110 msgr2=0x7f85ec1b7a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:30.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:30.211 7f85f134f700 1 --2- 172.21.2.102:0/3869881269 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f85ec0b2110 0x7f85ec1b7a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:30.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:30.211 7f85f134f700 1 -- 172.21.2.102:0/3869881269 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f85ec1b8ad0 msgr2=0x7f85ec187e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:30.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:30.211 7f85f134f700 1 --2- 172.21.2.102:0/3869881269 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f85ec1b8ad0 0x7f85ec187e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:30.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:30.211 7f85f134f700 1 -- 172.21.2.102:0/3869881269 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85ec0b2510 con 0x7f85ec1b8ed0 2021-02-10T11:28:30.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:30.211 7f85f134f700 1 --2- 172.21.2.102:0/3869881269 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f85ec1b8ed0 0x7f85ec0b1af0 secure :-1 s=READY pgs=1853 cs=0 l=1 rev1=1 rx=0x7f85e800b560 tx=0x7f85e800b900).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:28:30.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:30.211 7f85de7fc700 1 -- 172.21.2.102:0/3869881269 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f85e800bcf0 con 0x7f85ec1b8ed0 2021-02-10T11:28:30.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:30.211 7f85de7fc700 1 -- 172.21.2.102:0/3869881269 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85e8006030 con 0x7f85ec1b8ed0 2021-02-10T11:28:30.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:30.211 7f85f3d34700 1 -- 172.21.2.102:0/3869881269 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85ec0b2510 con 0x7f85ec1b8ed0 2021-02-10T11:28:30.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:30.211 7f85de7fc700 1 -- 172.21.2.102:0/3869881269 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f85e80054d0 con 0x7f85ec1b8ed0 2021-02-10T11:28:30.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:30.212 7f85f3d34700 1 -- 172.21.2.102:0/3869881269 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85ec0b2510 con 0x7f85ec1b8ed0 2021-02-10T11:28:30.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:30.212 7f85de7fc700 1 -- 172.21.2.102:0/3869881269 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f85e8006210 con 0x7f85ec1b8ed0 2021-02-10T11:28:30.216 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:30.212 7f85de7fc700 1 --2- 172.21.2.102:0/3869881269 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f85d4018730 0x7f85d401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:30.216 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:30.212 7f85de7fc700 1 -- 172.21.2.102:0/3869881269 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f85e8032790 con 0x7f85ec1b8ed0 2021-02-10T11:28:30.216 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:30.212 7f85f1b50700 1 --2- 172.21.2.102:0/3869881269 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f85d4018730 0x7f85d401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:30.216 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:30.213 7f85f1b50700 1 --2- 172.21.2.102:0/3869881269 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f85d4018730 0x7f85d401ac10 crc :-1 s=READY pgs=831 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:28:30.244 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:30.243 7f85f3d34700 1 -- 172.21.2.102:0/3869881269 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85ec0ebc00 con 0x7f85ec1b8ed0 2021-02-10T11:28:30.246 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:30.246 7f85de7fc700 1 -- 172.21.2.102:0/3869881269 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f85e8006b80 con 0x7f85ec1b8ed0 2021-02-10T11:28:30.374 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:30.374 7f85f3d34700 1 -- 172.21.2.102:0/3869881269 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f85ec0ebc00 con 0x7f85ec1b8ed0 2021-02-10T11:28:30.375 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:30.374 7f85de7fc700 1 -- 172.21.2.102:0/3869881269 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+2101 (secure 0 0 0) 0x7f85e800bcf0 con 0x7f85ec1b8ed0 2021-02-10T11:28:30.375 INFO:teuthology.orchestra.run.gibba002.stderr:dumped fsmap epoch 66 2021-02-10T11:28:30.375 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:28:30.375 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":66,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6110,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.102:6835/1471871278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6834","nonce":1471871278},{"type":"v1","addr":"172.21.2.102:6835","nonce":1471871278}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":61}],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:27:41.479375","modified":"2021-02-10 11:27:43.490605","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6107},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6107":{"gid":6107,"name":"b","rank":0,"incarnation":62,"state":"up:active","state_seq":3,"addr":"172.21.2.113:6833/4220238062","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6832","nonce":4220238062},{"type":"v1","addr":"172.21.2.113:6833","nonce":4220238062}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":8}]} 2021-02-10T11:28:30.377 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:30.376 7f85f3d34700 1 -- 172.21.2.102:0/3869881269 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f85d4018730 msgr2=0x7f85d401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:30.377 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:30.376 7f85f3d34700 1 --2- 172.21.2.102:0/3869881269 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f85d4018730 0x7f85d401ac10 crc :-1 s=READY pgs=831 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:30.377 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:30.376 7f85f3d34700 1 -- 172.21.2.102:0/3869881269 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f85ec1b8ed0 msgr2=0x7f85ec0b1af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:30.377 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:30.376 7f85f3d34700 1 --2- 172.21.2.102:0/3869881269 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f85ec1b8ed0 0x7f85ec0b1af0 secure :-1 s=READY pgs=1853 cs=0 l=1 rev1=1 rx=0x7f85e800b560 tx=0x7f85e800b900).stop 2021-02-10T11:28:30.377 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:30.377 7f85f3d34700 1 -- 172.21.2.102:0/3869881269 shutdown_connections 2021-02-10T11:28:30.377 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:30.377 7f85f3d34700 1 --2- 172.21.2.102:0/3869881269 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f85ec0b2110 0x7f85ec1b7a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:30.377 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:30.377 7f85f3d34700 1 --2- 172.21.2.102:0/3869881269 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f85ec1b8ed0 0x7f85ec0b1af0 unknown :-1 s=CLOSED pgs=1853 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:30.377 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:30.377 7f85f3d34700 1 --2- 172.21.2.102:0/3869881269 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f85d4018730 0x7f85d401ac10 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:30.377 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:30.377 7f85f3d34700 1 --2- 172.21.2.102:0/3869881269 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f85ec1b8ad0 0x7f85ec187e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:30.377 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:30.377 7f85f3d34700 1 -- 172.21.2.102:0/3869881269 >> 172.21.2.102:0/3869881269 conn(0x7f85ec0ed810 msgr2=0x7f85ec094980 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:28:30.378 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:30.377 7f85f3d34700 1 -- 172.21.2.102:0/3869881269 shutdown_connections 2021-02-10T11:28:30.378 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:30.377 7f85f3d34700 1 -- 172.21.2.102:0/3869881269 wait complete. 2021-02-10T11:28:30.386 DEBUG:teuthology.orchestra.run.gibba013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok session ls 2021-02-10T11:28:30.782 INFO:teuthology.orchestra.run.gibba013.stdout:[ 2021-02-10T11:28:30.782 INFO:teuthology.orchestra.run.gibba013.stdout: { 2021-02-10T11:28:30.782 INFO:teuthology.orchestra.run.gibba013.stdout: "id": 9471, 2021-02-10T11:28:30.782 INFO:teuthology.orchestra.run.gibba013.stdout: "entity": { 2021-02-10T11:28:30.782 INFO:teuthology.orchestra.run.gibba013.stdout: "name": { 2021-02-10T11:28:30.782 INFO:teuthology.orchestra.run.gibba013.stdout: "type": "client", 2021-02-10T11:28:30.782 INFO:teuthology.orchestra.run.gibba013.stdout: "num": 9471 2021-02-10T11:28:30.782 INFO:teuthology.orchestra.run.gibba013.stdout: }, 2021-02-10T11:28:30.782 INFO:teuthology.orchestra.run.gibba013.stdout: "addr": { 2021-02-10T11:28:30.782 INFO:teuthology.orchestra.run.gibba013.stdout: "type": "any", 2021-02-10T11:28:30.783 INFO:teuthology.orchestra.run.gibba013.stdout: "addr": "172.21.2.102:0", 2021-02-10T11:28:30.783 INFO:teuthology.orchestra.run.gibba013.stdout: "nonce": 190431005 2021-02-10T11:28:30.783 INFO:teuthology.orchestra.run.gibba013.stdout: } 2021-02-10T11:28:30.783 INFO:teuthology.orchestra.run.gibba013.stdout: }, 2021-02-10T11:28:30.783 INFO:teuthology.orchestra.run.gibba013.stdout: "state": "open", 2021-02-10T11:28:30.783 INFO:teuthology.orchestra.run.gibba013.stdout: "num_leases": 1, 2021-02-10T11:28:30.783 INFO:teuthology.orchestra.run.gibba013.stdout: "num_caps": 7, 2021-02-10T11:28:30.783 INFO:teuthology.orchestra.run.gibba013.stdout: "request_load_avg": 11, 2021-02-10T11:28:30.783 INFO:teuthology.orchestra.run.gibba013.stdout: "uptime": 20.104000923000001, 2021-02-10T11:28:30.783 INFO:teuthology.orchestra.run.gibba013.stdout: "requests_in_flight": 0, 2021-02-10T11:28:30.783 INFO:teuthology.orchestra.run.gibba013.stdout: "completed_requests": 1, 2021-02-10T11:28:30.783 INFO:teuthology.orchestra.run.gibba013.stdout: "reconnecting": false, 2021-02-10T11:28:30.783 INFO:teuthology.orchestra.run.gibba013.stdout: "recall_caps": { 2021-02-10T11:28:30.783 INFO:teuthology.orchestra.run.gibba013.stdout: "value": 0, 2021-02-10T11:28:30.784 INFO:teuthology.orchestra.run.gibba013.stdout: "halflife": 60 2021-02-10T11:28:30.784 INFO:teuthology.orchestra.run.gibba013.stdout: }, 2021-02-10T11:28:30.784 INFO:teuthology.orchestra.run.gibba013.stdout: "release_caps": { 2021-02-10T11:28:30.784 INFO:teuthology.orchestra.run.gibba013.stdout: "value": 0, 2021-02-10T11:28:30.784 INFO:teuthology.orchestra.run.gibba013.stdout: "halflife": 60 2021-02-10T11:28:30.784 INFO:teuthology.orchestra.run.gibba013.stdout: }, 2021-02-10T11:28:30.784 INFO:teuthology.orchestra.run.gibba013.stdout: "recall_caps_throttle": { 2021-02-10T11:28:30.784 INFO:teuthology.orchestra.run.gibba013.stdout: "value": 0, 2021-02-10T11:28:30.784 INFO:teuthology.orchestra.run.gibba013.stdout: "halflife": 2.5 2021-02-10T11:28:30.784 INFO:teuthology.orchestra.run.gibba013.stdout: }, 2021-02-10T11:28:30.784 INFO:teuthology.orchestra.run.gibba013.stdout: "recall_caps_throttle2o": { 2021-02-10T11:28:30.784 INFO:teuthology.orchestra.run.gibba013.stdout: "value": 0, 2021-02-10T11:28:30.784 INFO:teuthology.orchestra.run.gibba013.stdout: "halflife": 0.5 2021-02-10T11:28:30.785 INFO:teuthology.orchestra.run.gibba013.stdout: }, 2021-02-10T11:28:30.785 INFO:teuthology.orchestra.run.gibba013.stdout: "session_cache_liveness": { 2021-02-10T11:28:30.785 INFO:teuthology.orchestra.run.gibba013.stdout: "value": 20.060649502524978, 2021-02-10T11:28:30.785 INFO:teuthology.orchestra.run.gibba013.stdout: "halflife": 300 2021-02-10T11:28:30.785 INFO:teuthology.orchestra.run.gibba013.stdout: }, 2021-02-10T11:28:30.785 INFO:teuthology.orchestra.run.gibba013.stdout: "cap_acquisition": { 2021-02-10T11:28:30.785 INFO:teuthology.orchestra.run.gibba013.stdout: "value": 0, 2021-02-10T11:28:30.785 INFO:teuthology.orchestra.run.gibba013.stdout: "halflife": 10 2021-02-10T11:28:30.785 INFO:teuthology.orchestra.run.gibba013.stdout: }, 2021-02-10T11:28:30.785 INFO:teuthology.orchestra.run.gibba013.stdout: "inst": "client.9471 172.21.2.102:0/190431005", 2021-02-10T11:28:30.785 INFO:teuthology.orchestra.run.gibba013.stdout: "completed_requests": [ 2021-02-10T11:28:30.785 INFO:teuthology.orchestra.run.gibba013.stdout: { 2021-02-10T11:28:30.785 INFO:teuthology.orchestra.run.gibba013.stdout: "tid": 14, 2021-02-10T11:28:30.786 INFO:teuthology.orchestra.run.gibba013.stdout: "created_ino": "0x10000000005" 2021-02-10T11:28:30.786 INFO:teuthology.orchestra.run.gibba013.stdout: } 2021-02-10T11:28:30.786 INFO:teuthology.orchestra.run.gibba013.stdout: ], 2021-02-10T11:28:30.786 INFO:teuthology.orchestra.run.gibba013.stdout: "prealloc_inos": [ 2021-02-10T11:28:30.786 INFO:teuthology.orchestra.run.gibba013.stdout: { 2021-02-10T11:28:30.786 INFO:teuthology.orchestra.run.gibba013.stdout: "start": 1099511627782, 2021-02-10T11:28:30.786 INFO:teuthology.orchestra.run.gibba013.stdout: "length": 995 2021-02-10T11:28:30.786 INFO:teuthology.orchestra.run.gibba013.stdout: } 2021-02-10T11:28:30.786 INFO:teuthology.orchestra.run.gibba013.stdout: ], 2021-02-10T11:28:30.786 INFO:teuthology.orchestra.run.gibba013.stdout: "used_inos": [], 2021-02-10T11:28:30.786 INFO:teuthology.orchestra.run.gibba013.stdout: "client_metadata": { 2021-02-10T11:28:30.786 INFO:teuthology.orchestra.run.gibba013.stdout: "features": "0x0000000000001fff", 2021-02-10T11:28:30.787 INFO:teuthology.orchestra.run.gibba013.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:28:30.787 INFO:teuthology.orchestra.run.gibba013.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:28:30.787 INFO:teuthology.orchestra.run.gibba013.stdout: "entity_id": "y", 2021-02-10T11:28:30.787 INFO:teuthology.orchestra.run.gibba013.stdout: "hostname": "gibba002", 2021-02-10T11:28:30.787 INFO:teuthology.orchestra.run.gibba013.stdout: "pid": "35161", 2021-02-10T11:28:30.787 INFO:teuthology.orchestra.run.gibba013.stdout: "root": "/" 2021-02-10T11:28:30.787 INFO:teuthology.orchestra.run.gibba013.stdout: } 2021-02-10T11:28:30.787 INFO:teuthology.orchestra.run.gibba013.stdout: } 2021-02-10T11:28:30.787 INFO:teuthology.orchestra.run.gibba013.stdout:] 2021-02-10T11:28:30.788 INFO:tasks.cephfs.filesystem:_json_asok output: [ { "id": 9471, "entity": { "name": { "type": "client", "num": 9471 }, "addr": { "type": "any", "addr": "172.21.2.102:0", "nonce": 190431005 } }, "state": "open", "num_leases": 1, "num_caps": 7, "request_load_avg": 11, "uptime": 20.104000923000001, "requests_in_flight": 0, "completed_requests": 1, "reconnecting": false, "recall_caps": { "value": 0, "halflife": 60 }, "release_caps": { "value": 0, "halflife": 60 }, "recall_caps_throttle": { "value": 0, "halflife": 2.5 }, "recall_caps_throttle2o": { "value": 0, "halflife": 0.5 }, "session_cache_liveness": { "value": 20.060649502524978, "halflife": 300 }, "cap_acquisition": { "value": 0, "halflife": 10 }, "inst": "client.9471 172.21.2.102:0/190431005", "completed_requests": [ { "tid": 14, "created_ino": "0x10000000005" } ], "prealloc_inos": [ { "start": 1099511627782, "length": 995 } ], "used_inos": [], "client_metadata": { "features": "0x0000000000001fff", "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", "entity_id": "y", "hostname": "gibba002", "pid": "35161", "root": "/" } } ] 2021-02-10T11:28:30.788 DEBUG:tasks.ceph_test_case:wait_until_true: waiting... 2021-02-10T11:28:35.790 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:28:36.217 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:36.200 7f2e5eae7700 1 Processor -- start 2021-02-10T11:28:36.217 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:36.200 7f2e5eae7700 1 -- start start 2021-02-10T11:28:36.217 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:36.201 7f2e5eae7700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f2e580d1870 0x7f2e581b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:36.218 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:36.201 7f2e5eae7700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f2e580f0650 0x7f2e580cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:36.218 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:36.201 7f2e5eae7700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e580f0b10 con 0x7f2e581b0a10 2021-02-10T11:28:36.218 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:36.201 7f2e5eae7700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e580e5ed0 con 0x7f2e580d1870 2021-02-10T11:28:36.218 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:36.201 7f2e5eae7700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e580d1c70 con 0x7f2e580f0650 2021-02-10T11:28:36.218 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:36.201 7f2e5c903700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f2e580d1870 0x7f2e581b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:36.218 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:36.201 7f2e5c903700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f2e580d1870 0x7f2e581b03f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:60230/0 (socket says 172.21.2.102:60230) 2021-02-10T11:28:36.218 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:36.201 7f2e5c903700 1 -- 172.21.2.102:0/3787489718 learned_addr learned my addr 172.21.2.102:0/3787489718 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:28:36.218 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:36.201 7f2e5d104700 1 --2- 172.21.2.102:0/3787489718 >> v2:172.21.2.113:3300/0 conn(0x7f2e580f0650 0x7f2e580cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:36.218 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:36.201 7f2e577fe700 1 -- 172.21.2.102:0/3787489718 <== mon.1 v1:172.21.2.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3613156727 0 0) 0x7f2e48000fe0 con 0x7f2e581b0a10 2021-02-10T11:28:36.218 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:36.201 7f2e577fe700 1 -- 172.21.2.102:0/3787489718 --> v1:172.21.2.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e44002fe0 con 0x7f2e581b0a10 2021-02-10T11:28:36.218 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:36.202 7f2e5d104700 1 -- 172.21.2.102:0/3787489718 >> v2:172.21.2.102:3300/0 conn(0x7f2e580d1870 msgr2=0x7f2e581b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:28:36.218 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:36.202 7f2e5d104700 1 --2- 172.21.2.102:0/3787489718 >> v2:172.21.2.102:3300/0 conn(0x7f2e580d1870 0x7f2e581b03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:36.219 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:36.202 7f2e5d104700 1 -- 172.21.2.102:0/3787489718 >> v1:172.21.2.113:6789/0 conn(0x7f2e581b0a10 legacy=0x7f2e580cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:36.219 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:36.202 7f2e5d104700 1 -- 172.21.2.102:0/3787489718 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e580d1e90 con 0x7f2e580f0650 2021-02-10T11:28:36.219 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:36.202 7f2e5d104700 1 --2- 172.21.2.102:0/3787489718 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2e580f0650 0x7f2e580cfb20 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 rx=0x7f2e50006ad0 tx=0x7f2e50009ad0).ready entity=mon.1 client_cookie=85f46008c5ca70e7 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:28:36.219 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:36.202 7f2e577fe700 1 -- 172.21.2.102:0/3787489718 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2e500022e0 con 0x7f2e580f0650 2021-02-10T11:28:36.219 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:36.202 7f2e577fe700 1 -- 172.21.2.102:0/3787489718 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e50006410 con 0x7f2e580f0650 2021-02-10T11:28:36.219 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:36.202 7f2e577fe700 1 -- 172.21.2.102:0/3787489718 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2e50002620 con 0x7f2e580f0650 2021-02-10T11:28:36.219 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:36.202 7f2e5eae7700 1 -- 172.21.2.102:0/3787489718 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2e580f0650 msgr2=0x7f2e580cfb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:36.219 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:36.202 7f2e5eae7700 1 --2- 172.21.2.102:0/3787489718 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2e580f0650 0x7f2e580cfb20 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 rx=0x7f2e50006ad0 tx=0x7f2e50009ad0).stop 2021-02-10T11:28:36.219 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:36.202 7f2e5eae7700 1 -- 172.21.2.102:0/3787489718 shutdown_connections 2021-02-10T11:28:36.219 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:36.202 7f2e5eae7700 1 --2- 172.21.2.102:0/3787489718 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2e580f0650 0x7f2e580cfb20 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:36.219 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:36.202 7f2e5eae7700 1 --2- 172.21.2.102:0/3787489718 >> v2:172.21.2.102:3300/0 conn(0x7f2e580d1870 0x7f2e581b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:36.219 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:36.203 7f2e5eae7700 1 -- 172.21.2.102:0/3787489718 >> 172.21.2.102:0/3787489718 conn(0x7f2e580ce450 msgr2=0x7f2e580d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:28:36.220 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:36.203 7f2e5eae7700 1 -- 172.21.2.102:0/3787489718 shutdown_connections 2021-02-10T11:28:36.220 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:36.203 7f2e5eae7700 1 -- 172.21.2.102:0/3787489718 wait complete. 2021-02-10T11:28:36.220 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:36.217 7f2e5eae7700 1 Processor -- start 2021-02-10T11:28:36.221 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:36.217 7f2e5eae7700 1 -- start start 2021-02-10T11:28:36.221 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:36.218 7f2e5eae7700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2e58195650 0x7f2e580b5560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:36.221 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:36.218 7f2e5eae7700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2e580b5b80 0x7f2e581a2e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:36.221 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:36.218 7f2e5eae7700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2e580acfa0 0x7f2e581a3440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:36.221 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:36.218 7f2e5eae7700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e580b6010 con 0x7f2e58195650 2021-02-10T11:28:36.221 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:36.218 7f2e5eae7700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e581a3a60 con 0x7f2e580acfa0 2021-02-10T11:28:36.221 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:36.218 7f2e5eae7700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e581a3c40 con 0x7f2e580b5b80 2021-02-10T11:28:36.222 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:36.218 7f2e5c903700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2e58195650 0x7f2e580b5560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:36.222 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:36.218 7f2e5c903700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2e58195650 0x7f2e580b5560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:60236/0 (socket says 172.21.2.102:60236) 2021-02-10T11:28:36.222 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:36.218 7f2e5c903700 1 -- 172.21.2.102:0/1267187983 learned_addr learned my addr 172.21.2.102:0/1267187983 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:28:36.222 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:36.218 7f2e57fff700 1 --2- 172.21.2.102:0/1267187983 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2e580b5b80 0x7f2e581a2e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:36.222 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:36.218 7f2e5d104700 1 --2- 172.21.2.102:0/1267187983 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2e580acfa0 0x7f2e581a3440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:36.222 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:36.218 7f2e5c903700 1 -- 172.21.2.102:0/1267187983 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2e580b5b80 msgr2=0x7f2e581a2e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:36.222 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:36.218 7f2e5c903700 1 --2- 172.21.2.102:0/1267187983 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2e580b5b80 0x7f2e581a2e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:36.222 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:36.218 7f2e5c903700 1 -- 172.21.2.102:0/1267187983 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2e580acfa0 msgr2=0x7f2e581a3440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:36.222 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:36.218 7f2e5c903700 1 --2- 172.21.2.102:0/1267187983 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2e580acfa0 0x7f2e581a3440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:36.222 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:36.218 7f2e5c903700 1 -- 172.21.2.102:0/1267187983 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e580bcae0 con 0x7f2e58195650 2021-02-10T11:28:36.222 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:36.218 7f2e5c903700 1 --2- 172.21.2.102:0/1267187983 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2e58195650 0x7f2e580b5560 secure :-1 s=READY pgs=1854 cs=0 l=1 rev1=1 rx=0x7f2e4000b5e0 tx=0x7f2e4000b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:28:36.222 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:36.219 7f2e557fa700 1 -- 172.21.2.102:0/1267187983 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2e40006340 con 0x7f2e58195650 2021-02-10T11:28:36.223 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:36.219 7f2e557fa700 1 -- 172.21.2.102:0/1267187983 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e4000bd70 con 0x7f2e58195650 2021-02-10T11:28:36.223 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:36.219 7f2e5eae7700 1 -- 172.21.2.102:0/1267187983 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e580bcae0 con 0x7f2e58195650 2021-02-10T11:28:36.223 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:36.219 7f2e5eae7700 1 -- 172.21.2.102:0/1267187983 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e580bcae0 con 0x7f2e58195650 2021-02-10T11:28:36.223 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:36.219 7f2e557fa700 1 -- 172.21.2.102:0/1267187983 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2e400022b0 con 0x7f2e58195650 2021-02-10T11:28:36.223 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:36.219 7f2e557fa700 1 -- 172.21.2.102:0/1267187983 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f2e40000c90 con 0x7f2e58195650 2021-02-10T11:28:36.223 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:36.219 7f2e557fa700 1 --2- 172.21.2.102:0/1267187983 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2e3c018730 0x7f2e3c01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:36.223 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:36.219 7f2e557fa700 1 -- 172.21.2.102:0/1267187983 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f2e40032650 con 0x7f2e58195650 2021-02-10T11:28:36.223 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:36.219 7f2e57fff700 1 --2- 172.21.2.102:0/1267187983 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2e3c018730 0x7f2e3c01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:36.223 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:36.220 7f2e57fff700 1 --2- 172.21.2.102:0/1267187983 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2e3c018730 0x7f2e3c01ac10 crc :-1 s=READY pgs=832 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:28:36.251 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:36.250 7f2e5eae7700 1 -- 172.21.2.102:0/1267187983 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e580b6010 con 0x7f2e58195650 2021-02-10T11:28:36.253 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:36.253 7f2e557fa700 1 -- 172.21.2.102:0/1267187983 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f2e40006340 con 0x7f2e58195650 2021-02-10T11:28:36.385 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:36.384 7f2e5eae7700 1 -- 172.21.2.102:0/1267187983 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2e580b6010 con 0x7f2e58195650 2021-02-10T11:28:36.385 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:36.384 7f2e557fa700 1 -- 172.21.2.102:0/1267187983 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+2101 (secure 0 0 0) 0x7f2e4000bd70 con 0x7f2e58195650 2021-02-10T11:28:36.386 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:28:36.386 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":66,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6110,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.102:6835/1471871278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6834","nonce":1471871278},{"type":"v1","addr":"172.21.2.102:6835","nonce":1471871278}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":61}],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:27:41.479375","modified":"2021-02-10 11:27:43.490605","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6107},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6107":{"gid":6107,"name":"b","rank":0,"incarnation":62,"state":"up:active","state_seq":3,"addr":"172.21.2.113:6833/4220238062","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6832","nonce":4220238062},{"type":"v1","addr":"172.21.2.113:6833","nonce":4220238062}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":8}]} 2021-02-10T11:28:36.386 INFO:teuthology.orchestra.run.gibba002.stderr:dumped fsmap epoch 66 2021-02-10T11:28:36.386 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:36.386 7f2e5eae7700 1 -- 172.21.2.102:0/1267187983 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2e3c018730 msgr2=0x7f2e3c01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:36.387 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:36.386 7f2e5eae7700 1 --2- 172.21.2.102:0/1267187983 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2e3c018730 0x7f2e3c01ac10 crc :-1 s=READY pgs=832 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:36.387 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:36.386 7f2e5eae7700 1 -- 172.21.2.102:0/1267187983 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2e58195650 msgr2=0x7f2e580b5560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:36.387 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:36.386 7f2e5eae7700 1 --2- 172.21.2.102:0/1267187983 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2e58195650 0x7f2e580b5560 secure :-1 s=READY pgs=1854 cs=0 l=1 rev1=1 rx=0x7f2e4000b5e0 tx=0x7f2e4000b980).stop 2021-02-10T11:28:36.387 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:36.387 7f2e5eae7700 1 -- 172.21.2.102:0/1267187983 shutdown_connections 2021-02-10T11:28:36.387 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:36.387 7f2e5eae7700 1 --2- 172.21.2.102:0/1267187983 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2e3c018730 0x7f2e3c01ac10 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:36.387 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:36.387 7f2e5eae7700 1 --2- 172.21.2.102:0/1267187983 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2e580acfa0 0x7f2e581a3440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:36.387 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:36.387 7f2e5eae7700 1 --2- 172.21.2.102:0/1267187983 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2e580b5b80 0x7f2e581a2e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:36.388 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:36.387 7f2e5eae7700 1 --2- 172.21.2.102:0/1267187983 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2e58195650 0x7f2e580b5560 unknown :-1 s=CLOSED pgs=1854 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:36.388 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:36.387 7f2e5eae7700 1 -- 172.21.2.102:0/1267187983 >> 172.21.2.102:0/1267187983 conn(0x7f2e580b3040 msgr2=0x7f2e580c3250 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:28:36.388 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:36.387 7f2e5eae7700 1 -- 172.21.2.102:0/1267187983 shutdown_connections 2021-02-10T11:28:36.388 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:36.387 7f2e5eae7700 1 -- 172.21.2.102:0/1267187983 wait complete. 2021-02-10T11:28:36.396 DEBUG:teuthology.orchestra.run.gibba013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok session ls 2021-02-10T11:28:36.793 INFO:teuthology.orchestra.run.gibba013.stdout:[ 2021-02-10T11:28:36.793 INFO:teuthology.orchestra.run.gibba013.stdout: { 2021-02-10T11:28:36.793 INFO:teuthology.orchestra.run.gibba013.stdout: "id": 9471, 2021-02-10T11:28:36.793 INFO:teuthology.orchestra.run.gibba013.stdout: "entity": { 2021-02-10T11:28:36.793 INFO:teuthology.orchestra.run.gibba013.stdout: "name": { 2021-02-10T11:28:36.794 INFO:teuthology.orchestra.run.gibba013.stdout: "type": "client", 2021-02-10T11:28:36.794 INFO:teuthology.orchestra.run.gibba013.stdout: "num": 9471 2021-02-10T11:28:36.794 INFO:teuthology.orchestra.run.gibba013.stdout: }, 2021-02-10T11:28:36.794 INFO:teuthology.orchestra.run.gibba013.stdout: "addr": { 2021-02-10T11:28:36.794 INFO:teuthology.orchestra.run.gibba013.stdout: "type": "any", 2021-02-10T11:28:36.794 INFO:teuthology.orchestra.run.gibba013.stdout: "addr": "172.21.2.102:0", 2021-02-10T11:28:36.794 INFO:teuthology.orchestra.run.gibba013.stdout: "nonce": 190431005 2021-02-10T11:28:36.794 INFO:teuthology.orchestra.run.gibba013.stdout: } 2021-02-10T11:28:36.795 INFO:teuthology.orchestra.run.gibba013.stdout: }, 2021-02-10T11:28:36.795 INFO:teuthology.orchestra.run.gibba013.stdout: "state": "open", 2021-02-10T11:28:36.795 INFO:teuthology.orchestra.run.gibba013.stdout: "num_leases": 1, 2021-02-10T11:28:36.795 INFO:teuthology.orchestra.run.gibba013.stdout: "num_caps": 7, 2021-02-10T11:28:36.795 INFO:teuthology.orchestra.run.gibba013.stdout: "request_load_avg": 10, 2021-02-10T11:28:36.795 INFO:teuthology.orchestra.run.gibba013.stdout: "uptime": 26.115001160999999, 2021-02-10T11:28:36.795 INFO:teuthology.orchestra.run.gibba013.stdout: "requests_in_flight": 0, 2021-02-10T11:28:36.795 INFO:teuthology.orchestra.run.gibba013.stdout: "completed_requests": 1, 2021-02-10T11:28:36.795 INFO:teuthology.orchestra.run.gibba013.stdout: "reconnecting": false, 2021-02-10T11:28:36.795 INFO:teuthology.orchestra.run.gibba013.stdout: "recall_caps": { 2021-02-10T11:28:36.795 INFO:teuthology.orchestra.run.gibba013.stdout: "value": 0, 2021-02-10T11:28:36.796 INFO:teuthology.orchestra.run.gibba013.stdout: "halflife": 60 2021-02-10T11:28:36.796 INFO:teuthology.orchestra.run.gibba013.stdout: }, 2021-02-10T11:28:36.796 INFO:teuthology.orchestra.run.gibba013.stdout: "release_caps": { 2021-02-10T11:28:36.796 INFO:teuthology.orchestra.run.gibba013.stdout: "value": 0, 2021-02-10T11:28:36.796 INFO:teuthology.orchestra.run.gibba013.stdout: "halflife": 60 2021-02-10T11:28:36.796 INFO:teuthology.orchestra.run.gibba013.stdout: }, 2021-02-10T11:28:36.796 INFO:teuthology.orchestra.run.gibba013.stdout: "recall_caps_throttle": { 2021-02-10T11:28:36.796 INFO:teuthology.orchestra.run.gibba013.stdout: "value": 0, 2021-02-10T11:28:36.796 INFO:teuthology.orchestra.run.gibba013.stdout: "halflife": 2.5 2021-02-10T11:28:36.796 INFO:teuthology.orchestra.run.gibba013.stdout: }, 2021-02-10T11:28:36.796 INFO:teuthology.orchestra.run.gibba013.stdout: "recall_caps_throttle2o": { 2021-02-10T11:28:36.797 INFO:teuthology.orchestra.run.gibba013.stdout: "value": 0, 2021-02-10T11:28:36.797 INFO:teuthology.orchestra.run.gibba013.stdout: "halflife": 0.5 2021-02-10T11:28:36.797 INFO:teuthology.orchestra.run.gibba013.stdout: }, 2021-02-10T11:28:36.797 INFO:teuthology.orchestra.run.gibba013.stdout: "session_cache_liveness": { 2021-02-10T11:28:36.797 INFO:teuthology.orchestra.run.gibba013.stdout: "value": 19.783965778162472, 2021-02-10T11:28:36.797 INFO:teuthology.orchestra.run.gibba013.stdout: "halflife": 300 2021-02-10T11:28:36.797 INFO:teuthology.orchestra.run.gibba013.stdout: }, 2021-02-10T11:28:36.797 INFO:teuthology.orchestra.run.gibba013.stdout: "cap_acquisition": { 2021-02-10T11:28:36.797 INFO:teuthology.orchestra.run.gibba013.stdout: "value": 0, 2021-02-10T11:28:36.797 INFO:teuthology.orchestra.run.gibba013.stdout: "halflife": 10 2021-02-10T11:28:36.797 INFO:teuthology.orchestra.run.gibba013.stdout: }, 2021-02-10T11:28:36.798 INFO:teuthology.orchestra.run.gibba013.stdout: "inst": "client.9471 172.21.2.102:0/190431005", 2021-02-10T11:28:36.798 INFO:teuthology.orchestra.run.gibba013.stdout: "completed_requests": [ 2021-02-10T11:28:36.798 INFO:teuthology.orchestra.run.gibba013.stdout: { 2021-02-10T11:28:36.798 INFO:teuthology.orchestra.run.gibba013.stdout: "tid": 14, 2021-02-10T11:28:36.798 INFO:teuthology.orchestra.run.gibba013.stdout: "created_ino": "0x10000000005" 2021-02-10T11:28:36.798 INFO:teuthology.orchestra.run.gibba013.stdout: } 2021-02-10T11:28:36.798 INFO:teuthology.orchestra.run.gibba013.stdout: ], 2021-02-10T11:28:36.798 INFO:teuthology.orchestra.run.gibba013.stdout: "prealloc_inos": [ 2021-02-10T11:28:36.798 INFO:teuthology.orchestra.run.gibba013.stdout: { 2021-02-10T11:28:36.799 INFO:teuthology.orchestra.run.gibba013.stdout: "start": 1099511627782, 2021-02-10T11:28:36.799 INFO:teuthology.orchestra.run.gibba013.stdout: "length": 995 2021-02-10T11:28:36.799 INFO:teuthology.orchestra.run.gibba013.stdout: } 2021-02-10T11:28:36.799 INFO:teuthology.orchestra.run.gibba013.stdout: ], 2021-02-10T11:28:36.799 INFO:teuthology.orchestra.run.gibba013.stdout: "used_inos": [], 2021-02-10T11:28:36.799 INFO:teuthology.orchestra.run.gibba013.stdout: "client_metadata": { 2021-02-10T11:28:36.799 INFO:teuthology.orchestra.run.gibba013.stdout: "features": "0x0000000000001fff", 2021-02-10T11:28:36.799 INFO:teuthology.orchestra.run.gibba013.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:28:36.799 INFO:teuthology.orchestra.run.gibba013.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:28:36.799 INFO:teuthology.orchestra.run.gibba013.stdout: "entity_id": "y", 2021-02-10T11:28:36.799 INFO:teuthology.orchestra.run.gibba013.stdout: "hostname": "gibba002", 2021-02-10T11:28:36.800 INFO:teuthology.orchestra.run.gibba013.stdout: "pid": "35161", 2021-02-10T11:28:36.800 INFO:teuthology.orchestra.run.gibba013.stdout: "root": "/" 2021-02-10T11:28:36.800 INFO:teuthology.orchestra.run.gibba013.stdout: } 2021-02-10T11:28:36.800 INFO:teuthology.orchestra.run.gibba013.stdout: } 2021-02-10T11:28:36.800 INFO:teuthology.orchestra.run.gibba013.stdout:] 2021-02-10T11:28:36.801 INFO:tasks.cephfs.filesystem:_json_asok output: [ { "id": 9471, "entity": { "name": { "type": "client", "num": 9471 }, "addr": { "type": "any", "addr": "172.21.2.102:0", "nonce": 190431005 } }, "state": "open", "num_leases": 1, "num_caps": 7, "request_load_avg": 10, "uptime": 26.115001160999999, "requests_in_flight": 0, "completed_requests": 1, "reconnecting": false, "recall_caps": { "value": 0, "halflife": 60 }, "release_caps": { "value": 0, "halflife": 60 }, "recall_caps_throttle": { "value": 0, "halflife": 2.5 }, "recall_caps_throttle2o": { "value": 0, "halflife": 0.5 }, "session_cache_liveness": { "value": 19.783965778162472, "halflife": 300 }, "cap_acquisition": { "value": 0, "halflife": 10 }, "inst": "client.9471 172.21.2.102:0/190431005", "completed_requests": [ { "tid": 14, "created_ino": "0x10000000005" } ], "prealloc_inos": [ { "start": 1099511627782, "length": 995 } ], "used_inos": [], "client_metadata": { "features": "0x0000000000001fff", "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", "entity_id": "y", "hostname": "gibba002", "pid": "35161", "root": "/" } } ] 2021-02-10T11:28:36.801 DEBUG:tasks.ceph_test_case:wait_until_true: waiting... 2021-02-10T11:28:37.138 DEBUG:teuthology.orchestra.run.gibba002:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:28:37.142 DEBUG:teuthology.orchestra.run.gibba013:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:28:41.803 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:28:42.231 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:42.213 7fe7cb7d2700 1 Processor -- start 2021-02-10T11:28:42.231 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:42.213 7fe7cb7d2700 1 -- start start 2021-02-10T11:28:42.231 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:42.214 7fe7cb7d2700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe7c40d0340 0x7fe7c40ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:42.231 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:42.214 7fe7cb7d2700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7c40f0b10 con 0x7fe7c40d1c70 2021-02-10T11:28:42.231 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:42.214 7fe7cb7d2700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7c41b0be0 con 0x7fe7c40d1870 2021-02-10T11:28:42.232 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:42.214 7fe7cb7d2700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7c40e5ed0 con 0x7fe7c40d0340 2021-02-10T11:28:42.232 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:42.214 7fe7c9def700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fe7c40d1c70 0x7fe7c40efc60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:52470/0 (socket says 172.21.2.102:52470) 2021-02-10T11:28:42.232 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:42.214 7fe7c9def700 1 -- 172.21.2.102:0/1507804693 learned_addr learned my addr 172.21.2.102:0/1507804693 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:28:42.232 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:42.214 7fe7c8ded700 1 --2- 172.21.2.102:0/1507804693 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe7c40d0340 0x7fe7c40ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:42.232 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:42.214 7fe7b7fff700 1 -- 172.21.2.102:0/1507804693 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 992643048 0 0) 0x7fe7bc001500 con 0x7fe7c40d1c70 2021-02-10T11:28:42.232 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:42.214 7fe7b7fff700 1 -- 172.21.2.102:0/1507804693 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7b0002fe0 con 0x7fe7c40d1c70 2021-02-10T11:28:42.232 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:42.215 7fe7b7fff700 1 -- 172.21.2.102:0/1507804693 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2044827239 0 0) 0x7fe7bc001460 con 0x7fe7c40d1c70 2021-02-10T11:28:42.232 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:42.215 7fe7b7fff700 1 -- 172.21.2.102:0/1507804693 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe7c40d0340 msgr2=0x7fe7c40ef640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:28:42.232 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:42.215 7fe7b7fff700 1 --2- 172.21.2.102:0/1507804693 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe7c40d0340 0x7fe7c40ef640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:42.232 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:42.215 7fe7b7fff700 1 -- 172.21.2.102:0/1507804693 >> v1:172.21.2.113:6789/0 conn(0x7fe7c40d1870 legacy=0x7fe7c41b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:42.232 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:42.215 7fe7b7fff700 1 -- 172.21.2.102:0/1507804693 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7c40f0450 con 0x7fe7c40d1c70 2021-02-10T11:28:42.232 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:42.215 7fe7b7fff700 1 -- 172.21.2.102:0/1507804693 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7fe7bc001980 con 0x7fe7c40d1c70 2021-02-10T11:28:42.232 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:42.215 7fe7b7fff700 1 -- 172.21.2.102:0/1507804693 >> v1:172.21.2.102:6789/0 conn(0x7fe7c40d1c70 legacy=0x7fe7c40efc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:42.233 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:42.215 7fe7b7fff700 1 --2- 172.21.2.102:0/1507804693 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe7b0003af0 0x7fe7b0005f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:42.233 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:42.215 7fe7b7fff700 1 --2- 172.21.2.102:0/1507804693 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe7b00067a0 0x7fe7b0008c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:42.233 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:42.215 7fe7b7fff700 1 --2- 172.21.2.102:0/1507804693 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe7b00094f0 0x7fe7b000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:42.233 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:42.215 7fe7b7fff700 1 -- 172.21.2.102:0/1507804693 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7b000c2c0 con 0x7fe7b00094f0 2021-02-10T11:28:42.233 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:42.215 7fe7b7fff700 1 -- 172.21.2.102:0/1507804693 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7b000c4e0 con 0x7fe7b0003af0 2021-02-10T11:28:42.233 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:42.215 7fe7b7fff700 1 -- 172.21.2.102:0/1507804693 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7b000c700 con 0x7fe7b00067a0 2021-02-10T11:28:42.233 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:42.215 7fe7c9def700 1 --2- 172.21.2.102:0/1507804693 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe7b00094f0 0x7fe7b000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:42.233 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:42.215 7fe7c95ee700 1 --2- 172.21.2.102:0/1507804693 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe7b0003af0 0x7fe7b0005f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:42.233 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:42.215 7fe7c9def700 1 -- 172.21.2.102:0/1507804693 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe7b00067a0 msgr2=0x7fe7b0008c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:28:42.233 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:42.215 7fe7c9def700 1 --2- 172.21.2.102:0/1507804693 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe7b00067a0 0x7fe7b0008c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:28:42.233 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:42.215 7fe7c9def700 1 -- 172.21.2.102:0/1507804693 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe7b0003af0 msgr2=0x7fe7b0005f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:28:42.233 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:42.215 7fe7c9def700 1 --2- 172.21.2.102:0/1507804693 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe7b0003af0 0x7fe7b0005f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:42.234 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:42.215 7fe7c9def700 1 -- 172.21.2.102:0/1507804693 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7b0050970 con 0x7fe7b00094f0 2021-02-10T11:28:42.234 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:42.215 7fe7c9def700 1 --2- 172.21.2.102:0/1507804693 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe7b00094f0 0x7fe7b000ba10 secure :-1 s=READY pgs=1856 cs=0 l=1 rev1=1 rx=0x7fe7bc001af0 tx=0x7fe7bc00a310).ready entity=mon.0 client_cookie=80d9237c5bef058d server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:28:42.234 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:42.215 7fe7b7fff700 1 -- 172.21.2.102:0/1507804693 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe7bc0070c0 con 0x7fe7b00094f0 2021-02-10T11:28:42.234 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:42.216 7fe7b7fff700 1 -- 172.21.2.102:0/1507804693 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7bc000e60 con 0x7fe7b00094f0 2021-02-10T11:28:42.234 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:42.216 7fe7b7fff700 1 -- 172.21.2.102:0/1507804693 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe7bc00a910 con 0x7fe7b00094f0 2021-02-10T11:28:42.234 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:42.216 7fe7cb7d2700 1 -- 172.21.2.102:0/1507804693 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe7b00094f0 msgr2=0x7fe7b000ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:42.234 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:42.216 7fe7cb7d2700 1 --2- 172.21.2.102:0/1507804693 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe7b00094f0 0x7fe7b000ba10 secure :-1 s=READY pgs=1856 cs=0 l=1 rev1=1 rx=0x7fe7bc001af0 tx=0x7fe7bc00a310).stop 2021-02-10T11:28:42.234 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:42.216 7fe7c9def700 1 -- 172.21.2.102:0/1507804693 reap_dead start 2021-02-10T11:28:42.234 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:42.216 7fe7cb7d2700 1 -- 172.21.2.102:0/1507804693 shutdown_connections 2021-02-10T11:28:42.234 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:42.216 7fe7cb7d2700 1 -- 172.21.2.102:0/1507804693 >> 172.21.2.102:0/1507804693 conn(0x7fe7c40ce450 msgr2=0x7fe7c40d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:28:42.234 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:42.216 7fe7cb7d2700 1 -- 172.21.2.102:0/1507804693 shutdown_connections 2021-02-10T11:28:42.234 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:42.216 7fe7cb7d2700 1 -- 172.21.2.102:0/1507804693 wait complete. 2021-02-10T11:28:42.235 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:42.231 7fe7cb7d2700 1 Processor -- start 2021-02-10T11:28:42.235 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:42.231 7fe7cb7d2700 1 -- start start 2021-02-10T11:28:42.236 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:42.231 7fe7cb7d2700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe7c4195650 0x7fe7c40b5560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:42.236 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:42.231 7fe7cb7d2700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe7c40b5b80 0x7fe7c41a2e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:42.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:42.231 7fe7cb7d2700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe7c40acfa0 0x7fe7c41a3440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:42.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:42.231 7fe7cb7d2700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7c40b6010 con 0x7fe7c40acfa0 2021-02-10T11:28:42.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:42.231 7fe7cb7d2700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7c41a3a60 con 0x7fe7c4195650 2021-02-10T11:28:42.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:42.231 7fe7cb7d2700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7c41a3c40 con 0x7fe7c40b5b80 2021-02-10T11:28:42.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:42.232 7fe7c9def700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe7c40acfa0 0x7fe7c41a3440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:42.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:42.232 7fe7c9def700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe7c40acfa0 0x7fe7c41a3440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:60258/0 (socket says 172.21.2.102:60258) 2021-02-10T11:28:42.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:42.232 7fe7c9def700 1 -- 172.21.2.102:0/3328147866 learned_addr learned my addr 172.21.2.102:0/3328147866 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:28:42.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:42.232 7fe7c9def700 1 -- 172.21.2.102:0/3328147866 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe7c40b5b80 msgr2=0x7fe7c41a2e20 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:28:42.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:42.232 7fe7c9def700 1 --2- 172.21.2.102:0/3328147866 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe7c40b5b80 0x7fe7c41a2e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:28:42.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:42.232 7fe7c9def700 1 -- 172.21.2.102:0/3328147866 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe7c4195650 msgr2=0x7fe7c40b5560 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:28:42.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:42.232 7fe7c9def700 1 --2- 172.21.2.102:0/3328147866 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe7c4195650 0x7fe7c40b5560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:28:42.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:42.232 7fe7c9def700 1 -- 172.21.2.102:0/3328147866 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7c40bcae0 con 0x7fe7c40acfa0 2021-02-10T11:28:42.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:42.232 7fe7c9def700 1 --2- 172.21.2.102:0/3328147866 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe7c40acfa0 0x7fe7c41a3440 secure :-1 s=READY pgs=1857 cs=0 l=1 rev1=1 rx=0x7fe7bc007a00 tx=0x7fe7bc00aca0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:28:42.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:42.232 7fe7b5ffb700 1 -- 172.21.2.102:0/3328147866 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe7bc000e60 con 0x7fe7c40acfa0 2021-02-10T11:28:42.238 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:42.232 7fe7b5ffb700 1 -- 172.21.2.102:0/3328147866 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7bc001930 con 0x7fe7c40acfa0 2021-02-10T11:28:42.238 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:42.232 7fe7cb7d2700 1 -- 172.21.2.102:0/3328147866 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7c40bcae0 con 0x7fe7c40acfa0 2021-02-10T11:28:42.238 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:42.232 7fe7b5ffb700 1 -- 172.21.2.102:0/3328147866 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe7bc007300 con 0x7fe7c40acfa0 2021-02-10T11:28:42.238 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:42.232 7fe7cb7d2700 1 -- 172.21.2.102:0/3328147866 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7c40bd070 con 0x7fe7c40acfa0 2021-02-10T11:28:42.238 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:42.233 7fe7b5ffb700 1 -- 172.21.2.102:0/3328147866 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fe7bc004720 con 0x7fe7c40acfa0 2021-02-10T11:28:42.238 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:42.233 7fe7b5ffb700 1 --2- 172.21.2.102:0/3328147866 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe7a4018730 0x7fe7a401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:42.238 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:42.233 7fe7b5ffb700 1 -- 172.21.2.102:0/3328147866 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fe7bc032b20 con 0x7fe7c40acfa0 2021-02-10T11:28:42.238 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:42.233 7fe7c95ee700 1 --2- 172.21.2.102:0/3328147866 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe7a4018730 0x7fe7a401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:42.238 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:42.233 7fe7c95ee700 1 --2- 172.21.2.102:0/3328147866 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe7a4018730 0x7fe7a401abc0 crc :-1 s=READY pgs=833 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:28:42.264 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:42.264 7fe7cb7d2700 1 -- 172.21.2.102:0/3328147866 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7c40f0ab0 con 0x7fe7c40acfa0 2021-02-10T11:28:42.267 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:42.266 7fe7b5ffb700 1 -- 172.21.2.102:0/3328147866 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fe7bc000e60 con 0x7fe7c40acfa0 2021-02-10T11:28:42.394 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:42.394 7fe7cb7d2700 1 -- 172.21.2.102:0/3328147866 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe7c40f0ab0 con 0x7fe7c40acfa0 2021-02-10T11:28:42.395 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:42.395 7fe7b5ffb700 1 -- 172.21.2.102:0/3328147866 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+2101 (secure 0 0 0) 0x7fe7bc004720 con 0x7fe7c40acfa0 2021-02-10T11:28:42.395 INFO:teuthology.orchestra.run.gibba002.stderr:dumped fsmap epoch 66 2021-02-10T11:28:42.396 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:28:42.396 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":66,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6110,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.102:6835/1471871278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6834","nonce":1471871278},{"type":"v1","addr":"172.21.2.102:6835","nonce":1471871278}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":61}],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:27:41.479375","modified":"2021-02-10 11:27:43.490605","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6107},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6107":{"gid":6107,"name":"b","rank":0,"incarnation":62,"state":"up:active","state_seq":3,"addr":"172.21.2.113:6833/4220238062","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6832","nonce":4220238062},{"type":"v1","addr":"172.21.2.113:6833","nonce":4220238062}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":8}]} 2021-02-10T11:28:42.396 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:42.396 7fe7cb7d2700 1 -- 172.21.2.102:0/3328147866 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe7a4018730 msgr2=0x7fe7a401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:42.397 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:42.396 7fe7cb7d2700 1 --2- 172.21.2.102:0/3328147866 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe7a4018730 0x7fe7a401abc0 crc :-1 s=READY pgs=833 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:42.397 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:42.396 7fe7cb7d2700 1 -- 172.21.2.102:0/3328147866 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe7c40acfa0 msgr2=0x7fe7c41a3440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:42.397 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:42.396 7fe7cb7d2700 1 --2- 172.21.2.102:0/3328147866 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe7c40acfa0 0x7fe7c41a3440 secure :-1 s=READY pgs=1857 cs=0 l=1 rev1=1 rx=0x7fe7bc007a00 tx=0x7fe7bc00aca0).stop 2021-02-10T11:28:42.397 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:42.397 7fe7cb7d2700 1 -- 172.21.2.102:0/3328147866 shutdown_connections 2021-02-10T11:28:42.397 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:42.397 7fe7cb7d2700 1 --2- 172.21.2.102:0/3328147866 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe7c40acfa0 0x7fe7c41a3440 unknown :-1 s=CLOSED pgs=1857 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:42.398 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:42.397 7fe7cb7d2700 1 --2- 172.21.2.102:0/3328147866 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe7c40b5b80 0x7fe7c41a2e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:28:42.398 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:42.397 7fe7cb7d2700 1 --2- 172.21.2.102:0/3328147866 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe7a4018730 0x7fe7a401abc0 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:42.398 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:42.397 7fe7cb7d2700 1 --2- 172.21.2.102:0/3328147866 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe7c4195650 0x7fe7c40b5560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:28:42.398 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:42.397 7fe7cb7d2700 1 -- 172.21.2.102:0/3328147866 >> 172.21.2.102:0/3328147866 conn(0x7fe7c40b3040 msgr2=0x7fe7c40c3250 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:28:42.398 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:42.397 7fe7cb7d2700 1 -- 172.21.2.102:0/3328147866 shutdown_connections 2021-02-10T11:28:42.398 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:42.397 7fe7cb7d2700 1 -- 172.21.2.102:0/3328147866 wait complete. 2021-02-10T11:28:42.406 DEBUG:teuthology.orchestra.run.gibba013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok session ls 2021-02-10T11:28:42.802 INFO:teuthology.orchestra.run.gibba013.stdout:[ 2021-02-10T11:28:42.802 INFO:teuthology.orchestra.run.gibba013.stdout: { 2021-02-10T11:28:42.803 INFO:teuthology.orchestra.run.gibba013.stdout: "id": 9471, 2021-02-10T11:28:42.803 INFO:teuthology.orchestra.run.gibba013.stdout: "entity": { 2021-02-10T11:28:42.803 INFO:teuthology.orchestra.run.gibba013.stdout: "name": { 2021-02-10T11:28:42.803 INFO:teuthology.orchestra.run.gibba013.stdout: "type": "client", 2021-02-10T11:28:42.803 INFO:teuthology.orchestra.run.gibba013.stdout: "num": 9471 2021-02-10T11:28:42.803 INFO:teuthology.orchestra.run.gibba013.stdout: }, 2021-02-10T11:28:42.803 INFO:teuthology.orchestra.run.gibba013.stdout: "addr": { 2021-02-10T11:28:42.803 INFO:teuthology.orchestra.run.gibba013.stdout: "type": "any", 2021-02-10T11:28:42.803 INFO:teuthology.orchestra.run.gibba013.stdout: "addr": "172.21.2.102:0", 2021-02-10T11:28:42.803 INFO:teuthology.orchestra.run.gibba013.stdout: "nonce": 190431005 2021-02-10T11:28:42.803 INFO:teuthology.orchestra.run.gibba013.stdout: } 2021-02-10T11:28:42.804 INFO:teuthology.orchestra.run.gibba013.stdout: }, 2021-02-10T11:28:42.804 INFO:teuthology.orchestra.run.gibba013.stdout: "state": "open", 2021-02-10T11:28:42.804 INFO:teuthology.orchestra.run.gibba013.stdout: "num_leases": 0, 2021-02-10T11:28:42.804 INFO:teuthology.orchestra.run.gibba013.stdout: "num_caps": 7, 2021-02-10T11:28:42.804 INFO:teuthology.orchestra.run.gibba013.stdout: "request_load_avg": 9, 2021-02-10T11:28:42.804 INFO:teuthology.orchestra.run.gibba013.stdout: "uptime": 32.125001382000001, 2021-02-10T11:28:42.804 INFO:teuthology.orchestra.run.gibba013.stdout: "requests_in_flight": 0, 2021-02-10T11:28:42.804 INFO:teuthology.orchestra.run.gibba013.stdout: "completed_requests": 1, 2021-02-10T11:28:42.804 INFO:teuthology.orchestra.run.gibba013.stdout: "reconnecting": false, 2021-02-10T11:28:42.804 INFO:teuthology.orchestra.run.gibba013.stdout: "recall_caps": { 2021-02-10T11:28:42.804 INFO:teuthology.orchestra.run.gibba013.stdout: "value": 0, 2021-02-10T11:28:42.804 INFO:teuthology.orchestra.run.gibba013.stdout: "halflife": 60 2021-02-10T11:28:42.805 INFO:teuthology.orchestra.run.gibba013.stdout: }, 2021-02-10T11:28:42.805 INFO:teuthology.orchestra.run.gibba013.stdout: "release_caps": { 2021-02-10T11:28:42.805 INFO:teuthology.orchestra.run.gibba013.stdout: "value": 0, 2021-02-10T11:28:42.805 INFO:teuthology.orchestra.run.gibba013.stdout: "halflife": 60 2021-02-10T11:28:42.805 INFO:teuthology.orchestra.run.gibba013.stdout: }, 2021-02-10T11:28:42.805 INFO:teuthology.orchestra.run.gibba013.stdout: "recall_caps_throttle": { 2021-02-10T11:28:42.805 INFO:teuthology.orchestra.run.gibba013.stdout: "value": 0, 2021-02-10T11:28:42.805 INFO:teuthology.orchestra.run.gibba013.stdout: "halflife": 2.5 2021-02-10T11:28:42.805 INFO:teuthology.orchestra.run.gibba013.stdout: }, 2021-02-10T11:28:42.805 INFO:teuthology.orchestra.run.gibba013.stdout: "recall_caps_throttle2o": { 2021-02-10T11:28:42.805 INFO:teuthology.orchestra.run.gibba013.stdout: "value": 0, 2021-02-10T11:28:42.805 INFO:teuthology.orchestra.run.gibba013.stdout: "halflife": 0.5 2021-02-10T11:28:42.805 INFO:teuthology.orchestra.run.gibba013.stdout: }, 2021-02-10T11:28:42.806 INFO:teuthology.orchestra.run.gibba013.stdout: "session_cache_liveness": { 2021-02-10T11:28:42.806 INFO:teuthology.orchestra.run.gibba013.stdout: "value": 19.511143256699079, 2021-02-10T11:28:42.806 INFO:teuthology.orchestra.run.gibba013.stdout: "halflife": 300 2021-02-10T11:28:42.806 INFO:teuthology.orchestra.run.gibba013.stdout: }, 2021-02-10T11:28:42.806 INFO:teuthology.orchestra.run.gibba013.stdout: "cap_acquisition": { 2021-02-10T11:28:42.806 INFO:teuthology.orchestra.run.gibba013.stdout: "value": 0, 2021-02-10T11:28:42.806 INFO:teuthology.orchestra.run.gibba013.stdout: "halflife": 10 2021-02-10T11:28:42.806 INFO:teuthology.orchestra.run.gibba013.stdout: }, 2021-02-10T11:28:42.806 INFO:teuthology.orchestra.run.gibba013.stdout: "inst": "client.9471 172.21.2.102:0/190431005", 2021-02-10T11:28:42.806 INFO:teuthology.orchestra.run.gibba013.stdout: "completed_requests": [ 2021-02-10T11:28:42.806 INFO:teuthology.orchestra.run.gibba013.stdout: { 2021-02-10T11:28:42.807 INFO:teuthology.orchestra.run.gibba013.stdout: "tid": 14, 2021-02-10T11:28:42.807 INFO:teuthology.orchestra.run.gibba013.stdout: "created_ino": "0x10000000005" 2021-02-10T11:28:42.807 INFO:teuthology.orchestra.run.gibba013.stdout: } 2021-02-10T11:28:42.807 INFO:teuthology.orchestra.run.gibba013.stdout: ], 2021-02-10T11:28:42.807 INFO:teuthology.orchestra.run.gibba013.stdout: "prealloc_inos": [ 2021-02-10T11:28:42.807 INFO:teuthology.orchestra.run.gibba013.stdout: { 2021-02-10T11:28:42.807 INFO:teuthology.orchestra.run.gibba013.stdout: "start": 1099511627782, 2021-02-10T11:28:42.807 INFO:teuthology.orchestra.run.gibba013.stdout: "length": 995 2021-02-10T11:28:42.807 INFO:teuthology.orchestra.run.gibba013.stdout: } 2021-02-10T11:28:42.807 INFO:teuthology.orchestra.run.gibba013.stdout: ], 2021-02-10T11:28:42.807 INFO:teuthology.orchestra.run.gibba013.stdout: "used_inos": [], 2021-02-10T11:28:42.807 INFO:teuthology.orchestra.run.gibba013.stdout: "client_metadata": { 2021-02-10T11:28:42.808 INFO:teuthology.orchestra.run.gibba013.stdout: "features": "0x0000000000001fff", 2021-02-10T11:28:42.808 INFO:teuthology.orchestra.run.gibba013.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:28:42.808 INFO:teuthology.orchestra.run.gibba013.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:28:42.808 INFO:teuthology.orchestra.run.gibba013.stdout: "entity_id": "y", 2021-02-10T11:28:42.808 INFO:teuthology.orchestra.run.gibba013.stdout: "hostname": "gibba002", 2021-02-10T11:28:42.808 INFO:teuthology.orchestra.run.gibba013.stdout: "pid": "35161", 2021-02-10T11:28:42.808 INFO:teuthology.orchestra.run.gibba013.stdout: "root": "/" 2021-02-10T11:28:42.808 INFO:teuthology.orchestra.run.gibba013.stdout: } 2021-02-10T11:28:42.808 INFO:teuthology.orchestra.run.gibba013.stdout: } 2021-02-10T11:28:42.808 INFO:teuthology.orchestra.run.gibba013.stdout:] 2021-02-10T11:28:42.809 INFO:tasks.cephfs.filesystem:_json_asok output: [ { "id": 9471, "entity": { "name": { "type": "client", "num": 9471 }, "addr": { "type": "any", "addr": "172.21.2.102:0", "nonce": 190431005 } }, "state": "open", "num_leases": 0, "num_caps": 7, "request_load_avg": 9, "uptime": 32.125001382000001, "requests_in_flight": 0, "completed_requests": 1, "reconnecting": false, "recall_caps": { "value": 0, "halflife": 60 }, "release_caps": { "value": 0, "halflife": 60 }, "recall_caps_throttle": { "value": 0, "halflife": 2.5 }, "recall_caps_throttle2o": { "value": 0, "halflife": 0.5 }, "session_cache_liveness": { "value": 19.511143256699079, "halflife": 300 }, "cap_acquisition": { "value": 0, "halflife": 10 }, "inst": "client.9471 172.21.2.102:0/190431005", "completed_requests": [ { "tid": 14, "created_ino": "0x10000000005" } ], "prealloc_inos": [ { "start": 1099511627782, "length": 995 } ], "used_inos": [], "client_metadata": { "features": "0x0000000000001fff", "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", "entity_id": "y", "hostname": "gibba002", "pid": "35161", "root": "/" } } ] 2021-02-10T11:28:42.809 DEBUG:tasks.ceph_test_case:wait_until_true: waiting... 2021-02-10T11:28:47.811 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:28:48.243 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:48.223 7ff782b38700 1 Processor -- start 2021-02-10T11:28:48.243 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:48.223 7ff782b38700 1 -- start start 2021-02-10T11:28:48.243 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:48.223 7ff782b38700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7ff77c0d1870 0x7ff77c1b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:48.243 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:48.223 7ff782b38700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff77c0f0b10 con 0x7ff77c0f0650 2021-02-10T11:28:48.243 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:48.223 7ff782b38700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff77c0d0310 con 0x7ff77c1b0a10 2021-02-10T11:28:48.243 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:48.223 7ff782b38700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff77c0d0530 con 0x7ff77c0d1870 2021-02-10T11:28:48.243 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:48.223 7ff781155700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7ff77c0f0650 0x7ff77c0cfb20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:52488/0 (socket says 172.21.2.102:52488) 2021-02-10T11:28:48.243 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:48.223 7ff781155700 1 -- 172.21.2.102:0/4197258455 learned_addr learned my addr 172.21.2.102:0/4197258455 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:28:48.244 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:48.223 7ff780954700 1 --2- 172.21.2.102:0/4197258455 >> v2:172.21.2.113:3300/0 conn(0x7ff77c0d1870 0x7ff77c1b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:48.244 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:48.224 7ff77b7fe700 1 -- 172.21.2.102:0/4197258455 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3138549556 0 0) 0x7ff774001500 con 0x7ff77c0f0650 2021-02-10T11:28:48.244 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:48.224 7ff77b7fe700 1 -- 172.21.2.102:0/4197258455 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff768002fe0 con 0x7ff77c0f0650 2021-02-10T11:28:48.244 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:48.224 7ff780954700 1 -- 172.21.2.102:0/4197258455 >> v1:172.21.2.113:6789/0 conn(0x7ff77c1b0a10 legacy=0x7ff77c0cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:48.244 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:48.224 7ff780954700 1 -- 172.21.2.102:0/4197258455 >> v1:172.21.2.102:6789/0 conn(0x7ff77c0f0650 legacy=0x7ff77c0cfb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:48.244 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:48.224 7ff780954700 1 -- 172.21.2.102:0/4197258455 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff77c0d1c70 con 0x7ff77c0d1870 2021-02-10T11:28:48.244 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:48.224 7ff780954700 1 --2- 172.21.2.102:0/4197258455 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff77c0d1870 0x7ff77c1b03f0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 rx=0x7ff764006650 tx=0x7ff764009a80).ready entity=mon.1 client_cookie=dca510f53f67fa48 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:28:48.244 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:48.224 7ff77b7fe700 1 -- 172.21.2.102:0/4197258455 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff76400c0c0 con 0x7ff77c0d1870 2021-02-10T11:28:48.244 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:48.224 7ff77b7fe700 1 -- 172.21.2.102:0/4197258455 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff76400c340 con 0x7ff77c0d1870 2021-02-10T11:28:48.244 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:48.224 7ff77b7fe700 1 -- 172.21.2.102:0/4197258455 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff76400c7a0 con 0x7ff77c0d1870 2021-02-10T11:28:48.244 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:48.224 7ff782b38700 1 -- 172.21.2.102:0/4197258455 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff77c0d1870 msgr2=0x7ff77c1b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:48.244 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:48.224 7ff782b38700 1 --2- 172.21.2.102:0/4197258455 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff77c0d1870 0x7ff77c1b03f0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 rx=0x7ff764006650 tx=0x7ff764009a80).stop 2021-02-10T11:28:48.244 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:48.225 7ff782b38700 1 -- 172.21.2.102:0/4197258455 shutdown_connections 2021-02-10T11:28:48.245 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:48.225 7ff782b38700 1 --2- 172.21.2.102:0/4197258455 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff77c0d1870 0x7ff77c1b03f0 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:48.245 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:48.225 7ff782b38700 1 -- 172.21.2.102:0/4197258455 >> 172.21.2.102:0/4197258455 conn(0x7ff77c0ce450 msgr2=0x7ff77c0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:28:48.245 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:48.225 7ff782b38700 1 -- 172.21.2.102:0/4197258455 shutdown_connections 2021-02-10T11:28:48.245 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:48.225 7ff782b38700 1 -- 172.21.2.102:0/4197258455 wait complete. 2021-02-10T11:28:48.245 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:48.240 7ff782b38700 1 Processor -- start 2021-02-10T11:28:48.245 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:48.240 7ff782b38700 1 -- start start 2021-02-10T11:28:48.245 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:48.240 7ff782b38700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff77c099d50 0x7ff77c1b31b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:48.245 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:48.240 7ff782b38700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff77c187e50 0x7ff77c188250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:48.245 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:48.240 7ff782b38700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff77c1b12a0 0x7ff77c1b16a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:48.245 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:48.240 7ff782b38700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff77c188870 con 0x7ff77c1b12a0 2021-02-10T11:28:48.245 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:48.240 7ff782b38700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff77c1b37d0 con 0x7ff77c099d50 2021-02-10T11:28:48.245 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:48.240 7ff782b38700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff77c1b1cc0 con 0x7ff77c187e50 2021-02-10T11:28:48.245 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:48.240 7ff781155700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff77c1b12a0 0x7ff77c1b16a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:48.245 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:48.240 7ff781155700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff77c1b12a0 0x7ff77c1b16a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:60266/0 (socket says 172.21.2.102:60266) 2021-02-10T11:28:48.246 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:48.240 7ff781155700 1 -- 172.21.2.102:0/4159386171 learned_addr learned my addr 172.21.2.102:0/4159386171 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:28:48.246 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:48.240 7ff77bfff700 1 --2- 172.21.2.102:0/4159386171 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff77c187e50 0x7ff77c188250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:48.246 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:48.240 7ff780954700 1 --2- 172.21.2.102:0/4159386171 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff77c099d50 0x7ff77c1b31b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:48.246 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:48.241 7ff781155700 1 -- 172.21.2.102:0/4159386171 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff77c187e50 msgr2=0x7ff77c188250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:48.246 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:48.241 7ff781155700 1 --2- 172.21.2.102:0/4159386171 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff77c187e50 0x7ff77c188250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:48.246 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:48.241 7ff781155700 1 -- 172.21.2.102:0/4159386171 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff77c099d50 msgr2=0x7ff77c1b31b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:48.246 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:48.241 7ff781155700 1 --2- 172.21.2.102:0/4159386171 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff77c099d50 0x7ff77c1b31b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:48.246 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:48.241 7ff781155700 1 -- 172.21.2.102:0/4159386171 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff77c1b1ee0 con 0x7ff77c1b12a0 2021-02-10T11:28:48.246 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:48.241 7ff781155700 1 --2- 172.21.2.102:0/4159386171 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff77c1b12a0 0x7ff77c1b16a0 secure :-1 s=READY pgs=1859 cs=0 l=1 rev1=1 rx=0x7ff774001c10 tx=0x7ff77400aca0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:28:48.246 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:48.241 7ff7797fa700 1 -- 172.21.2.102:0/4159386171 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff774007ce0 con 0x7ff77c1b12a0 2021-02-10T11:28:48.246 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:48.241 7ff7797fa700 1 -- 172.21.2.102:0/4159386171 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7740070d0 con 0x7ff77c1b12a0 2021-02-10T11:28:48.246 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:48.241 7ff782b38700 1 -- 172.21.2.102:0/4159386171 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff77c1b1ee0 con 0x7ff77c1b12a0 2021-02-10T11:28:48.247 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:48.241 7ff782b38700 1 -- 172.21.2.102:0/4159386171 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff77c1b2470 con 0x7ff77c1b12a0 2021-02-10T11:28:48.247 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:48.241 7ff7797fa700 1 -- 172.21.2.102:0/4159386171 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff774004930 con 0x7ff77c1b12a0 2021-02-10T11:28:48.247 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:48.242 7ff7797fa700 1 -- 172.21.2.102:0/4159386171 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ff774004720 con 0x7ff77c1b12a0 2021-02-10T11:28:48.247 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:48.242 7ff7797fa700 1 --2- 172.21.2.102:0/4159386171 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff760018730 0x7ff76001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:48.247 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:48.242 7ff7797fa700 1 -- 172.21.2.102:0/4159386171 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 5573+0+0 (secure 0 0 0) 0x7ff774032b50 con 0x7ff77c1b12a0 2021-02-10T11:28:48.247 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:48.242 7ff780954700 1 --2- 172.21.2.102:0/4159386171 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff760018730 0x7ff76001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:48.247 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:48.242 7ff780954700 1 --2- 172.21.2.102:0/4159386171 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff760018730 0x7ff76001abc0 crc :-1 s=READY pgs=834 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:28:48.273 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:48.273 7ff782b38700 1 -- 172.21.2.102:0/4159386171 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff77c09a1e0 con 0x7ff77c1b12a0 2021-02-10T11:28:48.275 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:48.275 7ff7797fa700 1 -- 172.21.2.102:0/4159386171 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ff774007540 con 0x7ff77c1b12a0 2021-02-10T11:28:48.404 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:48.403 7ff782b38700 1 -- 172.21.2.102:0/4159386171 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff77c09a1e0 con 0x7ff77c1b12a0 2021-02-10T11:28:48.404 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:48.404 7ff7797fa700 1 -- 172.21.2.102:0/4159386171 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+2101 (secure 0 0 0) 0x7ff774004720 con 0x7ff77c1b12a0 2021-02-10T11:28:48.405 INFO:teuthology.orchestra.run.gibba002.stderr:dumped fsmap epoch 66 2021-02-10T11:28:48.405 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:28:48.405 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":66,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6110,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.102:6835/1471871278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6834","nonce":1471871278},{"type":"v1","addr":"172.21.2.102:6835","nonce":1471871278}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":61}],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:27:41.479375","modified":"2021-02-10 11:27:43.490605","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6107},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6107":{"gid":6107,"name":"b","rank":0,"incarnation":62,"state":"up:active","state_seq":3,"addr":"172.21.2.113:6833/4220238062","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6832","nonce":4220238062},{"type":"v1","addr":"172.21.2.113:6833","nonce":4220238062}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":8}]} 2021-02-10T11:28:48.406 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:48.405 7ff782b38700 1 -- 172.21.2.102:0/4159386171 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff760018730 msgr2=0x7ff76001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:48.406 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:48.405 7ff782b38700 1 --2- 172.21.2.102:0/4159386171 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff760018730 0x7ff76001abc0 crc :-1 s=READY pgs=834 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:48.406 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:48.405 7ff782b38700 1 -- 172.21.2.102:0/4159386171 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff77c1b12a0 msgr2=0x7ff77c1b16a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:48.406 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:48.405 7ff782b38700 1 --2- 172.21.2.102:0/4159386171 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff77c1b12a0 0x7ff77c1b16a0 secure :-1 s=READY pgs=1859 cs=0 l=1 rev1=1 rx=0x7ff774001c10 tx=0x7ff77400aca0).stop 2021-02-10T11:28:48.406 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:48.406 7ff782b38700 1 -- 172.21.2.102:0/4159386171 shutdown_connections 2021-02-10T11:28:48.407 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:48.406 7ff782b38700 1 --2- 172.21.2.102:0/4159386171 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff77c1b12a0 0x7ff77c1b16a0 unknown :-1 s=CLOSED pgs=1859 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:48.407 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:48.406 7ff782b38700 1 --2- 172.21.2.102:0/4159386171 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff77c187e50 0x7ff77c188250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:48.407 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:48.406 7ff782b38700 1 --2- 172.21.2.102:0/4159386171 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff760018730 0x7ff76001abc0 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:48.407 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:48.406 7ff782b38700 1 --2- 172.21.2.102:0/4159386171 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff77c099d50 0x7ff77c1b31b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:48.407 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:48.406 7ff782b38700 1 -- 172.21.2.102:0/4159386171 >> 172.21.2.102:0/4159386171 conn(0x7ff77c19f8c0 msgr2=0x7ff77c1a5fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:28:48.407 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:48.406 7ff782b38700 1 -- 172.21.2.102:0/4159386171 shutdown_connections 2021-02-10T11:28:48.407 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:48.406 7ff782b38700 1 -- 172.21.2.102:0/4159386171 wait complete. 2021-02-10T11:28:48.415 DEBUG:teuthology.orchestra.run.gibba013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok session ls 2021-02-10T11:28:48.812 INFO:teuthology.orchestra.run.gibba013.stdout:[ 2021-02-10T11:28:48.813 INFO:teuthology.orchestra.run.gibba013.stdout: { 2021-02-10T11:28:48.813 INFO:teuthology.orchestra.run.gibba013.stdout: "id": 9471, 2021-02-10T11:28:48.813 INFO:teuthology.orchestra.run.gibba013.stdout: "entity": { 2021-02-10T11:28:48.813 INFO:teuthology.orchestra.run.gibba013.stdout: "name": { 2021-02-10T11:28:48.813 INFO:teuthology.orchestra.run.gibba013.stdout: "type": "client", 2021-02-10T11:28:48.813 INFO:teuthology.orchestra.run.gibba013.stdout: "num": 9471 2021-02-10T11:28:48.814 INFO:teuthology.orchestra.run.gibba013.stdout: }, 2021-02-10T11:28:48.814 INFO:teuthology.orchestra.run.gibba013.stdout: "addr": { 2021-02-10T11:28:48.814 INFO:teuthology.orchestra.run.gibba013.stdout: "type": "any", 2021-02-10T11:28:48.814 INFO:teuthology.orchestra.run.gibba013.stdout: "addr": "172.21.2.102:0", 2021-02-10T11:28:48.814 INFO:teuthology.orchestra.run.gibba013.stdout: "nonce": 190431005 2021-02-10T11:28:48.814 INFO:teuthology.orchestra.run.gibba013.stdout: } 2021-02-10T11:28:48.814 INFO:teuthology.orchestra.run.gibba013.stdout: }, 2021-02-10T11:28:48.814 INFO:teuthology.orchestra.run.gibba013.stdout: "state": "open", 2021-02-10T11:28:48.815 INFO:teuthology.orchestra.run.gibba013.stdout: "num_leases": 0, 2021-02-10T11:28:48.815 INFO:teuthology.orchestra.run.gibba013.stdout: "num_caps": 7, 2021-02-10T11:28:48.815 INFO:teuthology.orchestra.run.gibba013.stdout: "request_load_avg": 9, 2021-02-10T11:28:48.815 INFO:teuthology.orchestra.run.gibba013.stdout: "uptime": 38.135001586999998, 2021-02-10T11:28:48.815 INFO:teuthology.orchestra.run.gibba013.stdout: "requests_in_flight": 0, 2021-02-10T11:28:48.815 INFO:teuthology.orchestra.run.gibba013.stdout: "completed_requests": 1, 2021-02-10T11:28:48.815 INFO:teuthology.orchestra.run.gibba013.stdout: "reconnecting": false, 2021-02-10T11:28:48.815 INFO:teuthology.orchestra.run.gibba013.stdout: "recall_caps": { 2021-02-10T11:28:48.815 INFO:teuthology.orchestra.run.gibba013.stdout: "value": 0, 2021-02-10T11:28:48.816 INFO:teuthology.orchestra.run.gibba013.stdout: "halflife": 60 2021-02-10T11:28:48.816 INFO:teuthology.orchestra.run.gibba013.stdout: }, 2021-02-10T11:28:48.816 INFO:teuthology.orchestra.run.gibba013.stdout: "release_caps": { 2021-02-10T11:28:48.816 INFO:teuthology.orchestra.run.gibba013.stdout: "value": 0, 2021-02-10T11:28:48.816 INFO:teuthology.orchestra.run.gibba013.stdout: "halflife": 60 2021-02-10T11:28:48.816 INFO:teuthology.orchestra.run.gibba013.stdout: }, 2021-02-10T11:28:48.816 INFO:teuthology.orchestra.run.gibba013.stdout: "recall_caps_throttle": { 2021-02-10T11:28:48.816 INFO:teuthology.orchestra.run.gibba013.stdout: "value": 0, 2021-02-10T11:28:48.816 INFO:teuthology.orchestra.run.gibba013.stdout: "halflife": 2.5 2021-02-10T11:28:48.817 INFO:teuthology.orchestra.run.gibba013.stdout: }, 2021-02-10T11:28:48.817 INFO:teuthology.orchestra.run.gibba013.stdout: "recall_caps_throttle2o": { 2021-02-10T11:28:48.817 INFO:teuthology.orchestra.run.gibba013.stdout: "value": 0, 2021-02-10T11:28:48.817 INFO:teuthology.orchestra.run.gibba013.stdout: "halflife": 0.5 2021-02-10T11:28:48.817 INFO:teuthology.orchestra.run.gibba013.stdout: }, 2021-02-10T11:28:48.817 INFO:teuthology.orchestra.run.gibba013.stdout: "session_cache_liveness": { 2021-02-10T11:28:48.817 INFO:teuthology.orchestra.run.gibba013.stdout: "value": 19.242082981042479, 2021-02-10T11:28:48.818 INFO:teuthology.orchestra.run.gibba013.stdout: "halflife": 300 2021-02-10T11:28:48.818 INFO:teuthology.orchestra.run.gibba013.stdout: }, 2021-02-10T11:28:48.818 INFO:teuthology.orchestra.run.gibba013.stdout: "cap_acquisition": { 2021-02-10T11:28:48.818 INFO:teuthology.orchestra.run.gibba013.stdout: "value": 0, 2021-02-10T11:28:48.819 INFO:teuthology.orchestra.run.gibba013.stdout: "halflife": 10 2021-02-10T11:28:48.819 INFO:teuthology.orchestra.run.gibba013.stdout: }, 2021-02-10T11:28:48.819 INFO:teuthology.orchestra.run.gibba013.stdout: "inst": "client.9471 172.21.2.102:0/190431005", 2021-02-10T11:28:48.819 INFO:teuthology.orchestra.run.gibba013.stdout: "completed_requests": [ 2021-02-10T11:28:48.819 INFO:teuthology.orchestra.run.gibba013.stdout: { 2021-02-10T11:28:48.819 INFO:teuthology.orchestra.run.gibba013.stdout: "tid": 14, 2021-02-10T11:28:48.819 INFO:teuthology.orchestra.run.gibba013.stdout: "created_ino": "0x10000000005" 2021-02-10T11:28:48.819 INFO:teuthology.orchestra.run.gibba013.stdout: } 2021-02-10T11:28:48.820 INFO:teuthology.orchestra.run.gibba013.stdout: ], 2021-02-10T11:28:48.820 INFO:teuthology.orchestra.run.gibba013.stdout: "prealloc_inos": [ 2021-02-10T11:28:48.820 INFO:teuthology.orchestra.run.gibba013.stdout: { 2021-02-10T11:28:48.820 INFO:teuthology.orchestra.run.gibba013.stdout: "start": 1099511627782, 2021-02-10T11:28:48.820 INFO:teuthology.orchestra.run.gibba013.stdout: "length": 995 2021-02-10T11:28:48.820 INFO:teuthology.orchestra.run.gibba013.stdout: } 2021-02-10T11:28:48.820 INFO:teuthology.orchestra.run.gibba013.stdout: ], 2021-02-10T11:28:48.820 INFO:teuthology.orchestra.run.gibba013.stdout: "used_inos": [], 2021-02-10T11:28:48.820 INFO:teuthology.orchestra.run.gibba013.stdout: "client_metadata": { 2021-02-10T11:28:48.820 INFO:teuthology.orchestra.run.gibba013.stdout: "features": "0x0000000000001fff", 2021-02-10T11:28:48.821 INFO:teuthology.orchestra.run.gibba013.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:28:48.821 INFO:teuthology.orchestra.run.gibba013.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:28:48.821 INFO:teuthology.orchestra.run.gibba013.stdout: "entity_id": "y", 2021-02-10T11:28:48.821 INFO:teuthology.orchestra.run.gibba013.stdout: "hostname": "gibba002", 2021-02-10T11:28:48.821 INFO:teuthology.orchestra.run.gibba013.stdout: "pid": "35161", 2021-02-10T11:28:48.821 INFO:teuthology.orchestra.run.gibba013.stdout: "root": "/" 2021-02-10T11:28:48.821 INFO:teuthology.orchestra.run.gibba013.stdout: } 2021-02-10T11:28:48.821 INFO:teuthology.orchestra.run.gibba013.stdout: } 2021-02-10T11:28:48.821 INFO:teuthology.orchestra.run.gibba013.stdout:] 2021-02-10T11:28:48.823 INFO:tasks.cephfs.filesystem:_json_asok output: [ { "id": 9471, "entity": { "name": { "type": "client", "num": 9471 }, "addr": { "type": "any", "addr": "172.21.2.102:0", "nonce": 190431005 } }, "state": "open", "num_leases": 0, "num_caps": 7, "request_load_avg": 9, "uptime": 38.135001586999998, "requests_in_flight": 0, "completed_requests": 1, "reconnecting": false, "recall_caps": { "value": 0, "halflife": 60 }, "release_caps": { "value": 0, "halflife": 60 }, "recall_caps_throttle": { "value": 0, "halflife": 2.5 }, "recall_caps_throttle2o": { "value": 0, "halflife": 0.5 }, "session_cache_liveness": { "value": 19.242082981042479, "halflife": 300 }, "cap_acquisition": { "value": 0, "halflife": 10 }, "inst": "client.9471 172.21.2.102:0/190431005", "completed_requests": [ { "tid": 14, "created_ino": "0x10000000005" } ], "prealloc_inos": [ { "start": 1099511627782, "length": 995 } ], "used_inos": [], "client_metadata": { "features": "0x0000000000001fff", "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", "entity_id": "y", "hostname": "gibba002", "pid": "35161", "root": "/" } } ] 2021-02-10T11:28:48.823 DEBUG:tasks.ceph_test_case:wait_until_true: waiting... 2021-02-10T11:28:53.824 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:28:54.251 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:54.233 7f71848cc700 1 Processor -- start 2021-02-10T11:28:54.251 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:54.233 7f71848cc700 1 -- start start 2021-02-10T11:28:54.251 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:54.234 7f71848cc700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f71801b0870 0x7f71800f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:54.251 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:54.234 7f71848cc700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f71800efe00 0x7f71800f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:54.251 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:54.234 7f71848cc700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71801b0c70 con 0x7f71801b0470 2021-02-10T11:28:54.251 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:54.234 7f71848cc700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71800f0b50 con 0x7f71800efe00 2021-02-10T11:28:54.251 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:54.234 7f71848cc700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71800f0820 con 0x7f71801b0870 2021-02-10T11:28:54.251 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:54.234 7f717de1b700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f71801b0870 0x7f71800f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:54.252 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:54.234 7f717e61c700 1 --1- >> v1:172.21.2.113:6789/0 conn(0x7f71801b0470 0x7f71800ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.113:6789/0 says I am v1:172.21.2.102:59822/0 (socket says 172.21.2.102:59822) 2021-02-10T11:28:54.252 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:54.235 7f717e61c700 1 -- 172.21.2.102:0/4196866840 learned_addr learned my addr 172.21.2.102:0/4196866840 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:28:54.252 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:54.235 7f717ee1d700 1 --2- 172.21.2.102:0/4196866840 >> v2:172.21.2.113:3300/0 conn(0x7f71800efe00 0x7f71800f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:54.252 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:54.235 7f717de1b700 1 -- 172.21.2.102:0/4196866840 >> v2:172.21.2.113:3300/0 conn(0x7f71800efe00 msgr2=0x7f71800f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:28:54.252 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:54.235 7f717de1b700 1 --2- 172.21.2.102:0/4196866840 >> v2:172.21.2.113:3300/0 conn(0x7f71800efe00 0x7f71800f0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:54.252 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:54.235 7f717de1b700 1 -- 172.21.2.102:0/4196866840 >> v1:172.21.2.113:6789/0 conn(0x7f71801b0470 legacy=0x7f71800ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:54.252 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:54.235 7f717de1b700 1 -- 172.21.2.102:0/4196866840 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71800e52e0 con 0x7f71801b0870 2021-02-10T11:28:54.252 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:54.235 7f717de1b700 1 --2- 172.21.2.102:0/4196866840 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f71801b0870 0x7f71800f20e0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f7170006650 tx=0x7f7170009a80).ready entity=mon.2 client_cookie=887d30860406f048 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:28:54.252 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:54.235 7f717d61a700 1 -- 172.21.2.102:0/4196866840 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f717000c0c0 con 0x7f71801b0870 2021-02-10T11:28:54.252 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:54.235 7f717d61a700 1 -- 172.21.2.102:0/4196866840 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f717000c340 con 0x7f71801b0870 2021-02-10T11:28:54.252 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:54.235 7f717d61a700 1 -- 172.21.2.102:0/4196866840 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f717000c860 con 0x7f71801b0870 2021-02-10T11:28:54.252 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:54.236 7f71848cc700 1 -- 172.21.2.102:0/4196866840 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f71801b0870 msgr2=0x7f71800f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:54.252 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:54.236 7f71848cc700 1 --2- 172.21.2.102:0/4196866840 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f71801b0870 0x7f71800f20e0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f7170006650 tx=0x7f7170009a80).stop 2021-02-10T11:28:54.253 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:54.236 7f71848cc700 1 -- 172.21.2.102:0/4196866840 shutdown_connections 2021-02-10T11:28:54.253 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:54.236 7f71848cc700 1 --2- 172.21.2.102:0/4196866840 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f71801b0870 0x7f71800f20e0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:54.253 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:54.236 7f71848cc700 1 --2- 172.21.2.102:0/4196866840 >> v2:172.21.2.113:3300/0 conn(0x7f71800efe00 0x7f71800f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:54.253 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:54.236 7f71848cc700 1 -- 172.21.2.102:0/4196866840 >> 172.21.2.102:0/4196866840 conn(0x7f71800ce3e0 msgr2=0x7f71800e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:28:54.253 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:54.236 7f71848cc700 1 -- 172.21.2.102:0/4196866840 shutdown_connections 2021-02-10T11:28:54.253 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:54.236 7f71848cc700 1 -- 172.21.2.102:0/4196866840 wait complete. 2021-02-10T11:28:54.254 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:54.251 7f71848cc700 1 Processor -- start 2021-02-10T11:28:54.254 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:54.251 7f71848cc700 1 -- start start 2021-02-10T11:28:54.254 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:54.251 7f71848cc700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f71801af8a0 0x7f7180198e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:54.255 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:54.251 7f71848cc700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f71801af050 0x7f71800aa3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:54.255 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:54.251 7f71848cc700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7180197d00 0x7f7180198100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:54.255 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:54.251 7f71848cc700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71801988e0 con 0x7f71801af8a0 2021-02-10T11:28:54.255 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:54.251 7f71848cc700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71801af450 con 0x7f7180197d00 2021-02-10T11:28:54.255 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:54.251 7f71848cc700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71800aaa10 con 0x7f71801af050 2021-02-10T11:28:54.255 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:54.252 7f717e61c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f71801af8a0 0x7f7180198e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:54.255 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:54.252 7f717e61c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f71801af8a0 0x7f7180198e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:60280/0 (socket says 172.21.2.102:60280) 2021-02-10T11:28:54.255 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:54.252 7f717e61c700 1 -- 172.21.2.102:0/3395246814 learned_addr learned my addr 172.21.2.102:0/3395246814 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:28:54.255 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:54.252 7f717e61c700 1 -- 172.21.2.102:0/3395246814 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f71801af050 msgr2=0x7f71800aa3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:54.255 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:54.252 7f717e61c700 1 --2- 172.21.2.102:0/3395246814 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f71801af050 0x7f71800aa3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:28:54.255 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:54.252 7f717e61c700 1 -- 172.21.2.102:0/3395246814 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7180197d00 msgr2=0x7f7180198100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:54.255 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:54.252 7f717e61c700 1 --2- 172.21.2.102:0/3395246814 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7180197d00 0x7f7180198100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:28:54.255 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:54.252 7f717e61c700 1 -- 172.21.2.102:0/3395246814 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f718019fec0 con 0x7f71801af8a0 2021-02-10T11:28:54.256 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:54.252 7f717e61c700 1 --2- 172.21.2.102:0/3395246814 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f71801af8a0 0x7f7180198e60 secure :-1 s=READY pgs=1860 cs=0 l=1 rev1=1 rx=0x7f7168007fa0 tx=0x7f716800a950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:28:54.256 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:54.252 7f7176ffd700 1 -- 172.21.2.102:0/3395246814 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f71680070d0 con 0x7f71801af8a0 2021-02-10T11:28:54.256 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:54.252 7f7176ffd700 1 -- 172.21.2.102:0/3395246814 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f716800ad40 con 0x7f71801af8a0 2021-02-10T11:28:54.256 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:54.252 7f71848cc700 1 -- 172.21.2.102:0/3395246814 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f718019fec0 con 0x7f71801af8a0 2021-02-10T11:28:54.256 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:54.252 7f7176ffd700 1 -- 172.21.2.102:0/3395246814 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7168004510 con 0x7f71801af8a0 2021-02-10T11:28:54.256 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:54.253 7f71848cc700 1 -- 172.21.2.102:0/3395246814 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f718019fec0 con 0x7f71801af8a0 2021-02-10T11:28:54.256 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:54.253 7f7176ffd700 1 -- 172.21.2.102:0/3395246814 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f7168000e20 con 0x7f71801af8a0 2021-02-10T11:28:54.256 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:54.253 7f7176ffd700 1 --2- 172.21.2.102:0/3395246814 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f71640186c0 0x7f716401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:54.256 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:54.253 7f7176ffd700 1 -- 172.21.2.102:0/3395246814 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f71680325b0 con 0x7f71801af8a0 2021-02-10T11:28:54.256 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:54.253 7f717de1b700 1 --2- 172.21.2.102:0/3395246814 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f71640186c0 0x7f716401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:54.256 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:54.254 7f717de1b700 1 --2- 172.21.2.102:0/3395246814 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f71640186c0 0x7f716401abc0 crc :-1 s=READY pgs=835 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:28:54.284 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:54.283 7f71848cc700 1 -- 172.21.2.102:0/3395246814 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7180105a20 con 0x7f71801af8a0 2021-02-10T11:28:54.286 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:54.286 7f7176ffd700 1 -- 172.21.2.102:0/3395246814 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f716802df10 con 0x7f71801af8a0 2021-02-10T11:28:54.414 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:54.414 7f71848cc700 1 -- 172.21.2.102:0/3395246814 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f71800f0b50 con 0x7f71801af8a0 2021-02-10T11:28:54.414 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:54.414 7f7176ffd700 1 -- 172.21.2.102:0/3395246814 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+2101 (secure 0 0 0) 0x7f716802df10 con 0x7f71801af8a0 2021-02-10T11:28:54.415 INFO:teuthology.orchestra.run.gibba002.stderr:dumped fsmap epoch 66 2021-02-10T11:28:54.415 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:28:54.415 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":66,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6110,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.102:6835/1471871278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6834","nonce":1471871278},{"type":"v1","addr":"172.21.2.102:6835","nonce":1471871278}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":61}],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:27:41.479375","modified":"2021-02-10 11:27:43.490605","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6107},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6107":{"gid":6107,"name":"b","rank":0,"incarnation":62,"state":"up:active","state_seq":3,"addr":"172.21.2.113:6833/4220238062","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6832","nonce":4220238062},{"type":"v1","addr":"172.21.2.113:6833","nonce":4220238062}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":8}]} 2021-02-10T11:28:54.417 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:54.416 7f71848cc700 1 -- 172.21.2.102:0/3395246814 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f71640186c0 msgr2=0x7f716401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:54.417 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:54.416 7f71848cc700 1 --2- 172.21.2.102:0/3395246814 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f71640186c0 0x7f716401abc0 crc :-1 s=READY pgs=835 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:54.417 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:54.416 7f71848cc700 1 -- 172.21.2.102:0/3395246814 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f71801af8a0 msgr2=0x7f7180198e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:54.417 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:54.416 7f71848cc700 1 --2- 172.21.2.102:0/3395246814 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f71801af8a0 0x7f7180198e60 secure :-1 s=READY pgs=1860 cs=0 l=1 rev1=1 rx=0x7f7168007fa0 tx=0x7f716800a950).stop 2021-02-10T11:28:54.417 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:54.416 7f71848cc700 1 -- 172.21.2.102:0/3395246814 shutdown_connections 2021-02-10T11:28:54.417 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:54.416 7f71848cc700 1 --2- 172.21.2.102:0/3395246814 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f71640186c0 0x7f716401abc0 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:54.417 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:54.416 7f71848cc700 1 --2- 172.21.2.102:0/3395246814 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7180197d00 0x7f7180198100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:28:54.417 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:54.416 7f71848cc700 1 --2- 172.21.2.102:0/3395246814 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f71801af050 0x7f71800aa3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:28:54.417 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:54.416 7f71848cc700 1 --2- 172.21.2.102:0/3395246814 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f71801af8a0 0x7f7180198e60 unknown :-1 s=CLOSED pgs=1860 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:54.417 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:54.416 7f71848cc700 1 -- 172.21.2.102:0/3395246814 >> 172.21.2.102:0/3395246814 conn(0x7f71801a37a0 msgr2=0x7f718018c230 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:28:54.417 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:54.417 7f71848cc700 1 -- 172.21.2.102:0/3395246814 shutdown_connections 2021-02-10T11:28:54.418 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:28:54.417 7f71848cc700 1 -- 172.21.2.102:0/3395246814 wait complete. 2021-02-10T11:28:54.426 DEBUG:teuthology.orchestra.run.gibba013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok session ls 2021-02-10T11:28:54.822 INFO:teuthology.orchestra.run.gibba013.stdout:[ 2021-02-10T11:28:54.823 INFO:teuthology.orchestra.run.gibba013.stdout: { 2021-02-10T11:28:54.823 INFO:teuthology.orchestra.run.gibba013.stdout: "id": 9471, 2021-02-10T11:28:54.823 INFO:teuthology.orchestra.run.gibba013.stdout: "entity": { 2021-02-10T11:28:54.823 INFO:teuthology.orchestra.run.gibba013.stdout: "name": { 2021-02-10T11:28:54.823 INFO:teuthology.orchestra.run.gibba013.stdout: "type": "client", 2021-02-10T11:28:54.824 INFO:teuthology.orchestra.run.gibba013.stdout: "num": 9471 2021-02-10T11:28:54.824 INFO:teuthology.orchestra.run.gibba013.stdout: }, 2021-02-10T11:28:54.824 INFO:teuthology.orchestra.run.gibba013.stdout: "addr": { 2021-02-10T11:28:54.824 INFO:teuthology.orchestra.run.gibba013.stdout: "type": "any", 2021-02-10T11:28:54.824 INFO:teuthology.orchestra.run.gibba013.stdout: "addr": "172.21.2.102:0", 2021-02-10T11:28:54.824 INFO:teuthology.orchestra.run.gibba013.stdout: "nonce": 190431005 2021-02-10T11:28:54.825 INFO:teuthology.orchestra.run.gibba013.stdout: } 2021-02-10T11:28:54.825 INFO:teuthology.orchestra.run.gibba013.stdout: }, 2021-02-10T11:28:54.825 INFO:teuthology.orchestra.run.gibba013.stdout: "state": "open", 2021-02-10T11:28:54.825 INFO:teuthology.orchestra.run.gibba013.stdout: "num_leases": 0, 2021-02-10T11:28:54.825 INFO:teuthology.orchestra.run.gibba013.stdout: "num_caps": 7, 2021-02-10T11:28:54.825 INFO:teuthology.orchestra.run.gibba013.stdout: "request_load_avg": 8, 2021-02-10T11:28:54.825 INFO:teuthology.orchestra.run.gibba013.stdout: "uptime": 44.145001776000001, 2021-02-10T11:28:54.825 INFO:teuthology.orchestra.run.gibba013.stdout: "requests_in_flight": 0, 2021-02-10T11:28:54.826 INFO:teuthology.orchestra.run.gibba013.stdout: "completed_requests": 1, 2021-02-10T11:28:54.826 INFO:teuthology.orchestra.run.gibba013.stdout: "reconnecting": false, 2021-02-10T11:28:54.826 INFO:teuthology.orchestra.run.gibba013.stdout: "recall_caps": { 2021-02-10T11:28:54.826 INFO:teuthology.orchestra.run.gibba013.stdout: "value": 0, 2021-02-10T11:28:54.826 INFO:teuthology.orchestra.run.gibba013.stdout: "halflife": 60 2021-02-10T11:28:54.826 INFO:teuthology.orchestra.run.gibba013.stdout: }, 2021-02-10T11:28:54.827 INFO:teuthology.orchestra.run.gibba013.stdout: "release_caps": { 2021-02-10T11:28:54.827 INFO:teuthology.orchestra.run.gibba013.stdout: "value": 0, 2021-02-10T11:28:54.827 INFO:teuthology.orchestra.run.gibba013.stdout: "halflife": 60 2021-02-10T11:28:54.827 INFO:teuthology.orchestra.run.gibba013.stdout: }, 2021-02-10T11:28:54.827 INFO:teuthology.orchestra.run.gibba013.stdout: "recall_caps_throttle": { 2021-02-10T11:28:54.827 INFO:teuthology.orchestra.run.gibba013.stdout: "value": 0, 2021-02-10T11:28:54.827 INFO:teuthology.orchestra.run.gibba013.stdout: "halflife": 2.5 2021-02-10T11:28:54.828 INFO:teuthology.orchestra.run.gibba013.stdout: }, 2021-02-10T11:28:54.828 INFO:teuthology.orchestra.run.gibba013.stdout: "recall_caps_throttle2o": { 2021-02-10T11:28:54.828 INFO:teuthology.orchestra.run.gibba013.stdout: "value": 0, 2021-02-10T11:28:54.828 INFO:teuthology.orchestra.run.gibba013.stdout: "halflife": 0.5 2021-02-10T11:28:54.828 INFO:teuthology.orchestra.run.gibba013.stdout: }, 2021-02-10T11:28:54.828 INFO:teuthology.orchestra.run.gibba013.stdout: "session_cache_liveness": { 2021-02-10T11:28:54.828 INFO:teuthology.orchestra.run.gibba013.stdout: "value": 18.976733069492767, 2021-02-10T11:28:54.829 INFO:teuthology.orchestra.run.gibba013.stdout: "halflife": 300 2021-02-10T11:28:54.829 INFO:teuthology.orchestra.run.gibba013.stdout: }, 2021-02-10T11:28:54.829 INFO:teuthology.orchestra.run.gibba013.stdout: "cap_acquisition": { 2021-02-10T11:28:54.830 INFO:teuthology.orchestra.run.gibba013.stdout: "value": 0, 2021-02-10T11:28:54.830 INFO:teuthology.orchestra.run.gibba013.stdout: "halflife": 10 2021-02-10T11:28:54.830 INFO:teuthology.orchestra.run.gibba013.stdout: }, 2021-02-10T11:28:54.830 INFO:teuthology.orchestra.run.gibba013.stdout: "inst": "client.9471 172.21.2.102:0/190431005", 2021-02-10T11:28:54.830 INFO:teuthology.orchestra.run.gibba013.stdout: "completed_requests": [ 2021-02-10T11:28:54.830 INFO:teuthology.orchestra.run.gibba013.stdout: { 2021-02-10T11:28:54.830 INFO:teuthology.orchestra.run.gibba013.stdout: "tid": 14, 2021-02-10T11:28:54.831 INFO:teuthology.orchestra.run.gibba013.stdout: "created_ino": "0x10000000005" 2021-02-10T11:28:54.831 INFO:teuthology.orchestra.run.gibba013.stdout: } 2021-02-10T11:28:54.831 INFO:teuthology.orchestra.run.gibba013.stdout: ], 2021-02-10T11:28:54.831 INFO:teuthology.orchestra.run.gibba013.stdout: "prealloc_inos": [ 2021-02-10T11:28:54.831 INFO:teuthology.orchestra.run.gibba013.stdout: { 2021-02-10T11:28:54.831 INFO:teuthology.orchestra.run.gibba013.stdout: "start": 1099511627782, 2021-02-10T11:28:54.831 INFO:teuthology.orchestra.run.gibba013.stdout: "length": 995 2021-02-10T11:28:54.831 INFO:teuthology.orchestra.run.gibba013.stdout: } 2021-02-10T11:28:54.832 INFO:teuthology.orchestra.run.gibba013.stdout: ], 2021-02-10T11:28:54.832 INFO:teuthology.orchestra.run.gibba013.stdout: "used_inos": [], 2021-02-10T11:28:54.832 INFO:teuthology.orchestra.run.gibba013.stdout: "client_metadata": { 2021-02-10T11:28:54.832 INFO:teuthology.orchestra.run.gibba013.stdout: "features": "0x0000000000001fff", 2021-02-10T11:28:54.832 INFO:teuthology.orchestra.run.gibba013.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:28:54.832 INFO:teuthology.orchestra.run.gibba013.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:28:54.832 INFO:teuthology.orchestra.run.gibba013.stdout: "entity_id": "y", 2021-02-10T11:28:54.833 INFO:teuthology.orchestra.run.gibba013.stdout: "hostname": "gibba002", 2021-02-10T11:28:54.833 INFO:teuthology.orchestra.run.gibba013.stdout: "pid": "35161", 2021-02-10T11:28:54.833 INFO:teuthology.orchestra.run.gibba013.stdout: "root": "/" 2021-02-10T11:28:54.833 INFO:teuthology.orchestra.run.gibba013.stdout: } 2021-02-10T11:28:54.833 INFO:teuthology.orchestra.run.gibba013.stdout: } 2021-02-10T11:28:54.833 INFO:teuthology.orchestra.run.gibba013.stdout:] 2021-02-10T11:28:54.835 INFO:tasks.cephfs.filesystem:_json_asok output: [ { "id": 9471, "entity": { "name": { "type": "client", "num": 9471 }, "addr": { "type": "any", "addr": "172.21.2.102:0", "nonce": 190431005 } }, "state": "open", "num_leases": 0, "num_caps": 7, "request_load_avg": 8, "uptime": 44.145001776000001, "requests_in_flight": 0, "completed_requests": 1, "reconnecting": false, "recall_caps": { "value": 0, "halflife": 60 }, "release_caps": { "value": 0, "halflife": 60 }, "recall_caps_throttle": { "value": 0, "halflife": 2.5 }, "recall_caps_throttle2o": { "value": 0, "halflife": 0.5 }, "session_cache_liveness": { "value": 18.976733069492767, "halflife": 300 }, "cap_acquisition": { "value": 0, "halflife": 10 }, "inst": "client.9471 172.21.2.102:0/190431005", "completed_requests": [ { "tid": 14, "created_ino": "0x10000000005" } ], "prealloc_inos": [ { "start": 1099511627782, "length": 995 } ], "used_inos": [], "client_metadata": { "features": "0x0000000000001fff", "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", "entity_id": "y", "hostname": "gibba002", "pid": "35161", "root": "/" } } ] 2021-02-10T11:28:54.835 DEBUG:tasks.ceph_test_case:wait_until_true: waiting... 2021-02-10T11:28:59.836 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:29:00.267 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:00.248 7f45a0d0c700 1 Processor -- start 2021-02-10T11:29:00.267 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:00.248 7f45a0d0c700 1 -- start start 2021-02-10T11:29:00.267 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:00.249 7f45a0d0c700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f459c0f05f0 0x7f459c0f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:00.267 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:00.249 7f45a0d0c700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f459c1b0c70 con 0x7f459c1b0470 2021-02-10T11:29:00.267 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:00.249 7f45a0d0c700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f459c0f0b50 con 0x7f459c1b0870 2021-02-10T11:29:00.267 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:00.249 7f45a0d0c700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f459c0e5ea0 con 0x7f459c0f05f0 2021-02-10T11:29:00.267 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:00.249 7f459ab13700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f459c1b0470 0x7f459c0ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:52512/0 (socket says 172.21.2.102:52512) 2021-02-10T11:29:00.267 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:00.249 7f459ab13700 1 -- 172.21.2.102:0/2717631204 learned_addr learned my addr 172.21.2.102:0/2717631204 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:29:00.267 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:00.250 7f4599b11700 1 -- 172.21.2.102:0/2717631204 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2753963821 0 0) 0x7f458c0010b0 con 0x7f459c1b0470 2021-02-10T11:29:00.268 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:00.250 7f4599b11700 1 -- 172.21.2.102:0/2717631204 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4584002fe0 con 0x7f459c1b0470 2021-02-10T11:29:00.268 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:00.250 7f459b314700 1 --2- 172.21.2.102:0/2717631204 >> v2:172.21.2.113:3300/0 conn(0x7f459c0f05f0 0x7f459c0f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:00.268 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:00.250 7f4599b11700 1 -- 172.21.2.102:0/2717631204 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3601505622 0 0) 0x7f458c0010b0 con 0x7f459c1b0470 2021-02-10T11:29:00.268 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:00.250 7f4599b11700 1 -- 172.21.2.102:0/2717631204 >> v2:172.21.2.113:3300/0 conn(0x7f459c0f05f0 msgr2=0x7f459c0f20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:29:00.268 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:00.250 7f4599b11700 1 --2- 172.21.2.102:0/2717631204 >> v2:172.21.2.113:3300/0 conn(0x7f459c0f05f0 0x7f459c0f20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:00.268 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:00.250 7f4599b11700 1 -- 172.21.2.102:0/2717631204 >> v1:172.21.2.113:6789/0 conn(0x7f459c1b0870 legacy=0x7f459c0efe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:00.268 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:00.250 7f4599b11700 1 -- 172.21.2.102:0/2717631204 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f459c0e52e0 con 0x7f459c1b0470 2021-02-10T11:29:00.268 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:00.250 7f4599b11700 1 -- 172.21.2.102:0/2717631204 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7f458c0015b0 con 0x7f459c1b0470 2021-02-10T11:29:00.268 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:00.250 7f459b314700 1 --2- 172.21.2.102:0/2717631204 >> v2:172.21.2.113:3300/0 conn(0x7f459c0f05f0 0x7f459c0f20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:29:00.268 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:00.250 7f4599b11700 1 -- 172.21.2.102:0/2717631204 >> v1:172.21.2.102:6789/0 conn(0x7f459c1b0470 legacy=0x7f459c0ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:00.268 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:00.250 7f4599b11700 1 --2- 172.21.2.102:0/2717631204 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4584003a60 0x7f4584005f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:00.268 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:00.250 7f4599b11700 1 --2- 172.21.2.102:0/2717631204 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4584006780 0x7f4584008ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:00.268 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:00.250 7f4599b11700 1 --2- 172.21.2.102:0/2717631204 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f45840095a0 0x7f458400bac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:00.269 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:00.250 7f4599b11700 1 -- 172.21.2.102:0/2717631204 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f458400c2f0 con 0x7f45840095a0 2021-02-10T11:29:00.269 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:00.250 7f4599b11700 1 -- 172.21.2.102:0/2717631204 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f458400c510 con 0x7f4584006780 2021-02-10T11:29:00.269 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:00.250 7f4599b11700 1 -- 172.21.2.102:0/2717631204 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4584050740 con 0x7f4584003a60 2021-02-10T11:29:00.269 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:00.250 7f459b314700 1 --2- 172.21.2.102:0/2717631204 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f45840095a0 0x7f458400bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:00.269 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:00.251 7f459a312700 1 --2- 172.21.2.102:0/2717631204 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4584006780 0x7f4584008ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:00.269 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:00.251 7f459ab13700 1 --2- 172.21.2.102:0/2717631204 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4584003a60 0x7f4584005f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:00.269 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:00.251 7f459b314700 1 -- 172.21.2.102:0/2717631204 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4584003a60 msgr2=0x7f4584005f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:29:00.269 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:00.251 7f459b314700 1 --2- 172.21.2.102:0/2717631204 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4584003a60 0x7f4584005f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:00.269 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:00.251 7f459b314700 1 -- 172.21.2.102:0/2717631204 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4584006780 msgr2=0x7f4584008ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:29:00.269 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:00.251 7f459b314700 1 --2- 172.21.2.102:0/2717631204 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4584006780 0x7f4584008ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:00.269 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:00.251 7f459b314700 1 -- 172.21.2.102:0/2717631204 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4584050970 con 0x7f45840095a0 2021-02-10T11:29:00.269 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:00.251 7f459b314700 1 -- 172.21.2.102:0/2717631204 reap_dead start 2021-02-10T11:29:00.269 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:00.251 7f459b314700 1 --2- 172.21.2.102:0/2717631204 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f45840095a0 0x7f458400bac0 secure :-1 s=READY pgs=1862 cs=0 l=1 rev1=1 rx=0x7f4594009ec0 tx=0x7f459400c370).ready entity=mon.0 client_cookie=369619a5168393fc server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:29:00.269 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:00.251 7f4599b11700 1 -- 172.21.2.102:0/2717631204 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4594007070 con 0x7f45840095a0 2021-02-10T11:29:00.270 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:00.251 7f4599b11700 1 -- 172.21.2.102:0/2717631204 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4594006b00 con 0x7f45840095a0 2021-02-10T11:29:00.270 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:00.251 7f4599b11700 1 -- 172.21.2.102:0/2717631204 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4594007670 con 0x7f45840095a0 2021-02-10T11:29:00.270 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:00.251 7f45a0d0c700 1 -- 172.21.2.102:0/2717631204 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f45840095a0 msgr2=0x7f458400bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:00.270 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:00.251 7f45a0d0c700 1 --2- 172.21.2.102:0/2717631204 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f45840095a0 0x7f458400bac0 secure :-1 s=READY pgs=1862 cs=0 l=1 rev1=1 rx=0x7f4594009ec0 tx=0x7f459400c370).stop 2021-02-10T11:29:00.270 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:00.252 7f45a0d0c700 1 -- 172.21.2.102:0/2717631204 shutdown_connections 2021-02-10T11:29:00.270 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:00.252 7f45a0d0c700 1 --2- 172.21.2.102:0/2717631204 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f45840095a0 0x7f458400bac0 unknown :-1 s=CLOSED pgs=1862 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:00.270 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:00.252 7f45a0d0c700 1 -- 172.21.2.102:0/2717631204 >> 172.21.2.102:0/2717631204 conn(0x7f459c0ce3e0 msgr2=0x7f459c0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:29:00.270 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:00.252 7f45a0d0c700 1 -- 172.21.2.102:0/2717631204 shutdown_connections 2021-02-10T11:29:00.270 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:00.252 7f45a0d0c700 1 -- 172.21.2.102:0/2717631204 wait complete. 2021-02-10T11:29:00.272 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:00.267 7f45a0d0c700 1 Processor -- start 2021-02-10T11:29:00.272 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:00.267 7f45a0d0c700 1 -- start start 2021-02-10T11:29:00.272 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:00.267 7f45a0d0c700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f459c108ce0 0x7f459c09da50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:00.272 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:00.267 7f45a0d0c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f459c1090e0 0x7f459c09e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:00.272 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:00.267 7f45a0d0c700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f459c09eee0 0x7f459c09f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:00.272 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:00.267 7f45a0d0c700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f459c1094e0 con 0x7f459c1090e0 2021-02-10T11:29:00.272 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:00.267 7f45a0d0c700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f459c09e960 con 0x7f459c108ce0 2021-02-10T11:29:00.272 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:00.267 7f45a0d0c700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f459c09e690 con 0x7f459c09eee0 2021-02-10T11:29:00.272 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:00.267 7f459a312700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f459c1090e0 0x7f459c09e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:00.273 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:00.267 7f459a312700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f459c1090e0 0x7f459c09e070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:60298/0 (socket says 172.21.2.102:60298) 2021-02-10T11:29:00.273 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:00.267 7f459a312700 1 -- 172.21.2.102:0/989675358 learned_addr learned my addr 172.21.2.102:0/989675358 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:29:00.273 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:00.268 7f459a312700 1 -- 172.21.2.102:0/989675358 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f459c09eee0 msgr2=0x7f459c09f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:00.273 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:00.268 7f459a312700 1 --2- 172.21.2.102:0/989675358 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f459c09eee0 0x7f459c09f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:00.273 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:00.268 7f459a312700 1 -- 172.21.2.102:0/989675358 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f459c108ce0 msgr2=0x7f459c09da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:00.273 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:00.268 7f459a312700 1 --2- 172.21.2.102:0/989675358 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f459c108ce0 0x7f459c09da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:00.273 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:00.268 7f459a312700 1 -- 172.21.2.102:0/989675358 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f459c09f900 con 0x7f459c1090e0 2021-02-10T11:29:00.273 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:00.268 7f459a312700 1 --2- 172.21.2.102:0/989675358 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f459c1090e0 0x7f459c09e070 secure :-1 s=READY pgs=1863 cs=0 l=1 rev1=1 rx=0x7f4590006f10 tx=0x7f459000b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:29:00.273 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:00.268 7f458b7fe700 1 -- 172.21.2.102:0/989675358 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f459000bd80 con 0x7f459c1090e0 2021-02-10T11:29:00.273 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:00.268 7f458b7fe700 1 -- 172.21.2.102:0/989675358 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45900060c0 con 0x7f459c1090e0 2021-02-10T11:29:00.273 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:00.268 7f458b7fe700 1 -- 172.21.2.102:0/989675358 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f45900052e0 con 0x7f459c1090e0 2021-02-10T11:29:00.273 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:00.268 7f45a0d0c700 1 -- 172.21.2.102:0/989675358 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f459c09f900 con 0x7f459c1090e0 2021-02-10T11:29:00.273 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:00.268 7f45a0d0c700 1 -- 172.21.2.102:0/989675358 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f459c09f900 con 0x7f459c1090e0 2021-02-10T11:29:00.274 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:00.269 7f458b7fe700 1 -- 172.21.2.102:0/989675358 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f45900062a0 con 0x7f459c1090e0 2021-02-10T11:29:00.274 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:00.269 7f458b7fe700 1 --2- 172.21.2.102:0/989675358 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f45800186c0 0x7f458001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:00.274 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:00.269 7f458b7fe700 1 -- 172.21.2.102:0/989675358 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f4590032800 con 0x7f459c1090e0 2021-02-10T11:29:00.274 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:00.269 7f459ab13700 1 --2- 172.21.2.102:0/989675358 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f45800186c0 0x7f458001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:00.274 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:00.269 7f459ab13700 1 --2- 172.21.2.102:0/989675358 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f45800186c0 0x7f458001abc0 crc :-1 s=READY pgs=836 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:29:00.299 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:00.299 7f45a0d0c700 1 -- 172.21.2.102:0/989675358 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f459c0f0b50 con 0x7f459c1090e0 2021-02-10T11:29:00.302 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:00.302 7f458b7fe700 1 -- 172.21.2.102:0/989675358 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f4590006390 con 0x7f459c1090e0 2021-02-10T11:29:00.431 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:00.430 7f45a0d0c700 1 -- 172.21.2.102:0/989675358 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f459c0f0b50 con 0x7f459c1090e0 2021-02-10T11:29:00.431 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:00.431 7f458b7fe700 1 -- 172.21.2.102:0/989675358 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+2101 (secure 0 0 0) 0x7f459000bd80 con 0x7f459c1090e0 2021-02-10T11:29:00.432 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:29:00.432 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":66,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6110,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.102:6835/1471871278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6834","nonce":1471871278},{"type":"v1","addr":"172.21.2.102:6835","nonce":1471871278}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":61}],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:27:41.479375","modified":"2021-02-10 11:27:43.490605","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6107},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6107":{"gid":6107,"name":"b","rank":0,"incarnation":62,"state":"up:active","state_seq":3,"addr":"172.21.2.113:6833/4220238062","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6832","nonce":4220238062},{"type":"v1","addr":"172.21.2.113:6833","nonce":4220238062}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":8}]} 2021-02-10T11:29:00.432 INFO:teuthology.orchestra.run.gibba002.stderr:dumped fsmap epoch 66 2021-02-10T11:29:00.433 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:00.433 7f45a0d0c700 1 -- 172.21.2.102:0/989675358 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f45800186c0 msgr2=0x7f458001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:00.433 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:00.433 7f45a0d0c700 1 --2- 172.21.2.102:0/989675358 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f45800186c0 0x7f458001abc0 crc :-1 s=READY pgs=836 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:00.433 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:00.433 7f45a0d0c700 1 -- 172.21.2.102:0/989675358 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f459c1090e0 msgr2=0x7f459c09e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:00.433 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:00.433 7f45a0d0c700 1 --2- 172.21.2.102:0/989675358 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f459c1090e0 0x7f459c09e070 secure :-1 s=READY pgs=1863 cs=0 l=1 rev1=1 rx=0x7f4590006f10 tx=0x7f459000b990).stop 2021-02-10T11:29:00.434 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:00.433 7f45a0d0c700 1 -- 172.21.2.102:0/989675358 shutdown_connections 2021-02-10T11:29:00.434 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:00.433 7f45a0d0c700 1 --2- 172.21.2.102:0/989675358 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f459c09eee0 0x7f459c09f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:00.434 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:00.433 7f45a0d0c700 1 --2- 172.21.2.102:0/989675358 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f459c1090e0 0x7f459c09e070 unknown :-1 s=CLOSED pgs=1863 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:00.434 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:00.433 7f45a0d0c700 1 --2- 172.21.2.102:0/989675358 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f45800186c0 0x7f458001abc0 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:00.434 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:00.433 7f45a0d0c700 1 --2- 172.21.2.102:0/989675358 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f459c108ce0 0x7f459c09da50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:00.434 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:00.433 7f45a0d0c700 1 -- 172.21.2.102:0/989675358 >> 172.21.2.102:0/989675358 conn(0x7f459c0bc6c0 msgr2=0x7f459c0b1f70 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:29:00.434 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:00.433 7f45a0d0c700 1 -- 172.21.2.102:0/989675358 shutdown_connections 2021-02-10T11:29:00.434 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:00.433 7f45a0d0c700 1 -- 172.21.2.102:0/989675358 wait complete. 2021-02-10T11:29:00.442 DEBUG:teuthology.orchestra.run.gibba013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok session ls 2021-02-10T11:29:00.839 INFO:teuthology.orchestra.run.gibba013.stdout:[ 2021-02-10T11:29:00.839 INFO:teuthology.orchestra.run.gibba013.stdout: { 2021-02-10T11:29:00.839 INFO:teuthology.orchestra.run.gibba013.stdout: "id": 9471, 2021-02-10T11:29:00.839 INFO:teuthology.orchestra.run.gibba013.stdout: "entity": { 2021-02-10T11:29:00.839 INFO:teuthology.orchestra.run.gibba013.stdout: "name": { 2021-02-10T11:29:00.839 INFO:teuthology.orchestra.run.gibba013.stdout: "type": "client", 2021-02-10T11:29:00.839 INFO:teuthology.orchestra.run.gibba013.stdout: "num": 9471 2021-02-10T11:29:00.839 INFO:teuthology.orchestra.run.gibba013.stdout: }, 2021-02-10T11:29:00.839 INFO:teuthology.orchestra.run.gibba013.stdout: "addr": { 2021-02-10T11:29:00.839 INFO:teuthology.orchestra.run.gibba013.stdout: "type": "any", 2021-02-10T11:29:00.839 INFO:teuthology.orchestra.run.gibba013.stdout: "addr": "172.21.2.102:0", 2021-02-10T11:29:00.840 INFO:teuthology.orchestra.run.gibba013.stdout: "nonce": 190431005 2021-02-10T11:29:00.840 INFO:teuthology.orchestra.run.gibba013.stdout: } 2021-02-10T11:29:00.840 INFO:teuthology.orchestra.run.gibba013.stdout: }, 2021-02-10T11:29:00.840 INFO:teuthology.orchestra.run.gibba013.stdout: "state": "open", 2021-02-10T11:29:00.840 INFO:teuthology.orchestra.run.gibba013.stdout: "num_leases": 0, 2021-02-10T11:29:00.840 INFO:teuthology.orchestra.run.gibba013.stdout: "num_caps": 7, 2021-02-10T11:29:00.840 INFO:teuthology.orchestra.run.gibba013.stdout: "request_load_avg": 7, 2021-02-10T11:29:00.840 INFO:teuthology.orchestra.run.gibba013.stdout: "uptime": 50.161001949999999, 2021-02-10T11:29:00.840 INFO:teuthology.orchestra.run.gibba013.stdout: "requests_in_flight": 0, 2021-02-10T11:29:00.840 INFO:teuthology.orchestra.run.gibba013.stdout: "completed_requests": 1, 2021-02-10T11:29:00.840 INFO:teuthology.orchestra.run.gibba013.stdout: "reconnecting": false, 2021-02-10T11:29:00.840 INFO:teuthology.orchestra.run.gibba013.stdout: "recall_caps": { 2021-02-10T11:29:00.840 INFO:teuthology.orchestra.run.gibba013.stdout: "value": 0, 2021-02-10T11:29:00.840 INFO:teuthology.orchestra.run.gibba013.stdout: "halflife": 60 2021-02-10T11:29:00.841 INFO:teuthology.orchestra.run.gibba013.stdout: }, 2021-02-10T11:29:00.841 INFO:teuthology.orchestra.run.gibba013.stdout: "release_caps": { 2021-02-10T11:29:00.841 INFO:teuthology.orchestra.run.gibba013.stdout: "value": 0, 2021-02-10T11:29:00.841 INFO:teuthology.orchestra.run.gibba013.stdout: "halflife": 60 2021-02-10T11:29:00.841 INFO:teuthology.orchestra.run.gibba013.stdout: }, 2021-02-10T11:29:00.841 INFO:teuthology.orchestra.run.gibba013.stdout: "recall_caps_throttle": { 2021-02-10T11:29:00.841 INFO:teuthology.orchestra.run.gibba013.stdout: "value": 0, 2021-02-10T11:29:00.841 INFO:teuthology.orchestra.run.gibba013.stdout: "halflife": 2.5 2021-02-10T11:29:00.841 INFO:teuthology.orchestra.run.gibba013.stdout: }, 2021-02-10T11:29:00.841 INFO:teuthology.orchestra.run.gibba013.stdout: "recall_caps_throttle2o": { 2021-02-10T11:29:00.841 INFO:teuthology.orchestra.run.gibba013.stdout: "value": 0, 2021-02-10T11:29:00.841 INFO:teuthology.orchestra.run.gibba013.stdout: "halflife": 0.5 2021-02-10T11:29:00.841 INFO:teuthology.orchestra.run.gibba013.stdout: }, 2021-02-10T11:29:00.842 INFO:teuthology.orchestra.run.gibba013.stdout: "session_cache_liveness": { 2021-02-10T11:29:00.842 INFO:teuthology.orchestra.run.gibba013.stdout: "value": 18.714782911981601, 2021-02-10T11:29:00.842 INFO:teuthology.orchestra.run.gibba013.stdout: "halflife": 300 2021-02-10T11:29:00.842 INFO:teuthology.orchestra.run.gibba013.stdout: }, 2021-02-10T11:29:00.842 INFO:teuthology.orchestra.run.gibba013.stdout: "cap_acquisition": { 2021-02-10T11:29:00.842 INFO:teuthology.orchestra.run.gibba013.stdout: "value": 0, 2021-02-10T11:29:00.842 INFO:teuthology.orchestra.run.gibba013.stdout: "halflife": 10 2021-02-10T11:29:00.842 INFO:teuthology.orchestra.run.gibba013.stdout: }, 2021-02-10T11:29:00.842 INFO:teuthology.orchestra.run.gibba013.stdout: "inst": "client.9471 172.21.2.102:0/190431005", 2021-02-10T11:29:00.842 INFO:teuthology.orchestra.run.gibba013.stdout: "completed_requests": [ 2021-02-10T11:29:00.842 INFO:teuthology.orchestra.run.gibba013.stdout: { 2021-02-10T11:29:00.842 INFO:teuthology.orchestra.run.gibba013.stdout: "tid": 14, 2021-02-10T11:29:00.842 INFO:teuthology.orchestra.run.gibba013.stdout: "created_ino": "0x10000000005" 2021-02-10T11:29:00.843 INFO:teuthology.orchestra.run.gibba013.stdout: } 2021-02-10T11:29:00.843 INFO:teuthology.orchestra.run.gibba013.stdout: ], 2021-02-10T11:29:00.843 INFO:teuthology.orchestra.run.gibba013.stdout: "prealloc_inos": [ 2021-02-10T11:29:00.843 INFO:teuthology.orchestra.run.gibba013.stdout: { 2021-02-10T11:29:00.843 INFO:teuthology.orchestra.run.gibba013.stdout: "start": 1099511627782, 2021-02-10T11:29:00.843 INFO:teuthology.orchestra.run.gibba013.stdout: "length": 995 2021-02-10T11:29:00.843 INFO:teuthology.orchestra.run.gibba013.stdout: } 2021-02-10T11:29:00.843 INFO:teuthology.orchestra.run.gibba013.stdout: ], 2021-02-10T11:29:00.843 INFO:teuthology.orchestra.run.gibba013.stdout: "used_inos": [], 2021-02-10T11:29:00.843 INFO:teuthology.orchestra.run.gibba013.stdout: "client_metadata": { 2021-02-10T11:29:00.843 INFO:teuthology.orchestra.run.gibba013.stdout: "features": "0x0000000000001fff", 2021-02-10T11:29:00.843 INFO:teuthology.orchestra.run.gibba013.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:29:00.843 INFO:teuthology.orchestra.run.gibba013.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:29:00.843 INFO:teuthology.orchestra.run.gibba013.stdout: "entity_id": "y", 2021-02-10T11:29:00.844 INFO:teuthology.orchestra.run.gibba013.stdout: "hostname": "gibba002", 2021-02-10T11:29:00.844 INFO:teuthology.orchestra.run.gibba013.stdout: "pid": "35161", 2021-02-10T11:29:00.844 INFO:teuthology.orchestra.run.gibba013.stdout: "root": "/" 2021-02-10T11:29:00.844 INFO:teuthology.orchestra.run.gibba013.stdout: } 2021-02-10T11:29:00.844 INFO:teuthology.orchestra.run.gibba013.stdout: } 2021-02-10T11:29:00.844 INFO:teuthology.orchestra.run.gibba013.stdout:] 2021-02-10T11:29:00.845 INFO:tasks.cephfs.filesystem:_json_asok output: [ { "id": 9471, "entity": { "name": { "type": "client", "num": 9471 }, "addr": { "type": "any", "addr": "172.21.2.102:0", "nonce": 190431005 } }, "state": "open", "num_leases": 0, "num_caps": 7, "request_load_avg": 7, "uptime": 50.161001949999999, "requests_in_flight": 0, "completed_requests": 1, "reconnecting": false, "recall_caps": { "value": 0, "halflife": 60 }, "release_caps": { "value": 0, "halflife": 60 }, "recall_caps_throttle": { "value": 0, "halflife": 2.5 }, "recall_caps_throttle2o": { "value": 0, "halflife": 0.5 }, "session_cache_liveness": { "value": 18.714782911981601, "halflife": 300 }, "cap_acquisition": { "value": 0, "halflife": 10 }, "inst": "client.9471 172.21.2.102:0/190431005", "completed_requests": [ { "tid": 14, "created_ino": "0x10000000005" } ], "prealloc_inos": [ { "start": 1099511627782, "length": 995 } ], "used_inos": [], "client_metadata": { "features": "0x0000000000001fff", "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", "entity_id": "y", "hostname": "gibba002", "pid": "35161", "root": "/" } } ] 2021-02-10T11:29:00.845 DEBUG:tasks.ceph_test_case:wait_until_true: waiting... 2021-02-10T11:29:05.846 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:29:06.276 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:06.257 7f39ecf5c700 1 Processor -- start 2021-02-10T11:29:06.276 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:06.257 7f39ecf5c700 1 -- start start 2021-02-10T11:29:06.277 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:06.258 7f39ecf5c700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f39e80d1870 0x7f39e81b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:06.277 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:06.258 7f39ecf5c700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f39e80f0650 0x7f39e80cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:06.277 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:06.258 7f39ecf5c700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39e80f0b10 con 0x7f39e81b0a10 2021-02-10T11:29:06.277 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:06.258 7f39ecf5c700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39e80fe230 con 0x7f39e80f0650 2021-02-10T11:29:06.277 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:06.258 7f39ecf5c700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39e80e5ed0 con 0x7f39e80d1870 2021-02-10T11:29:06.277 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:06.258 7f39e6312700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f39e81b0a10 0x7f39e80cf330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:52530/0 (socket says 172.21.2.102:52530) 2021-02-10T11:29:06.277 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:06.258 7f39e6312700 1 -- 172.21.2.102:0/4137185712 learned_addr learned my addr 172.21.2.102:0/4137185712 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:29:06.277 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:06.259 7f39e5b11700 1 -- 172.21.2.102:0/4137185712 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 444530780 0 0) 0x7f39dc0010b0 con 0x7f39e81b0a10 2021-02-10T11:29:06.277 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:06.259 7f39e5b11700 1 -- 172.21.2.102:0/4137185712 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39d0002fe0 con 0x7f39e81b0a10 2021-02-10T11:29:06.277 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:06.259 7f39e7314700 1 --2- 172.21.2.102:0/4137185712 >> v2:172.21.2.113:3300/0 conn(0x7f39e80f0650 0x7f39e80cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:06.277 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:06.259 7f39e6b13700 1 --2- 172.21.2.102:0/4137185712 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f39e80d1870 0x7f39e81b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:06.277 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:06.259 7f39e5b11700 1 -- 172.21.2.102:0/4137185712 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3725604034 0 0) 0x7f39dc0010b0 con 0x7f39e81b0a10 2021-02-10T11:29:06.278 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:06.259 7f39e5b11700 1 -- 172.21.2.102:0/4137185712 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f39e80d1870 msgr2=0x7f39e81b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:29:06.278 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:06.259 7f39e5b11700 1 --2- 172.21.2.102:0/4137185712 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f39e80d1870 0x7f39e81b03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:06.278 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:06.259 7f39e5b11700 1 -- 172.21.2.102:0/4137185712 >> v2:172.21.2.113:3300/0 conn(0x7f39e80f0650 msgr2=0x7f39e80cfb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:29:06.278 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:06.259 7f39e5b11700 1 --2- 172.21.2.102:0/4137185712 >> v2:172.21.2.113:3300/0 conn(0x7f39e80f0650 0x7f39e80cfb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:06.278 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:06.259 7f39e5b11700 1 -- 172.21.2.102:0/4137185712 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39e80f20a0 con 0x7f39e81b0a10 2021-02-10T11:29:06.278 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:06.259 7f39e5b11700 1 -- 172.21.2.102:0/4137185712 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7f39dc0015b0 con 0x7f39e81b0a10 2021-02-10T11:29:06.278 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:06.260 7f39e5b11700 1 -- 172.21.2.102:0/4137185712 >> v1:172.21.2.102:6789/0 conn(0x7f39e81b0a10 legacy=0x7f39e80cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:06.278 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:06.260 7f39e5b11700 1 --2- 172.21.2.102:0/4137185712 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f39d0003a70 0x7f39d0005f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:06.279 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:06.260 7f39e5b11700 1 --2- 172.21.2.102:0/4137185712 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f39d004a7b0 0x7f39d004ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:06.279 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:06.260 7f39e5b11700 1 --2- 172.21.2.102:0/4137185712 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f39d004d5b0 0x7f39d004faa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:06.279 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:06.260 7f39e5b11700 1 -- 172.21.2.102:0/4137185712 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39d00502f0 con 0x7f39d0003a70 2021-02-10T11:29:06.279 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:06.260 7f39e5b11700 1 -- 172.21.2.102:0/4137185712 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39d0050510 con 0x7f39d004a7b0 2021-02-10T11:29:06.279 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:06.260 7f39e5b11700 1 -- 172.21.2.102:0/4137185712 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39d0050730 con 0x7f39d004d5b0 2021-02-10T11:29:06.279 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:06.260 7f39e6b13700 1 --2- 172.21.2.102:0/4137185712 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f39d0003a70 0x7f39d0005f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:06.279 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:06.260 7f39e6312700 1 --2- 172.21.2.102:0/4137185712 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f39d004a7b0 0x7f39d004ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:06.280 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:06.260 7f39e7314700 1 --2- 172.21.2.102:0/4137185712 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f39d004d5b0 0x7f39d004faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:06.280 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:06.260 7f39e6b13700 1 -- 172.21.2.102:0/4137185712 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f39d004d5b0 msgr2=0x7f39d004faa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:29:06.280 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:06.260 7f39e6b13700 1 --2- 172.21.2.102:0/4137185712 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f39d004d5b0 0x7f39d004faa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:06.280 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:06.260 7f39e6b13700 1 -- 172.21.2.102:0/4137185712 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f39d004a7b0 msgr2=0x7f39d004ccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:29:06.280 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:06.260 7f39e6b13700 1 --2- 172.21.2.102:0/4137185712 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f39d004a7b0 0x7f39d004ccb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:06.280 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:06.260 7f39e6b13700 1 -- 172.21.2.102:0/4137185712 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39d00509d0 con 0x7f39d0003a70 2021-02-10T11:29:06.281 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:06.260 7f39e6312700 1 --2- 172.21.2.102:0/4137185712 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f39d004a7b0 0x7f39d004ccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:29:06.281 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:06.260 7f39e6b13700 1 --2- 172.21.2.102:0/4137185712 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f39d0003a70 0x7f39d0005f00 secure :-1 s=READY pgs=1865 cs=0 l=1 rev1=1 rx=0x7f39d8006ac0 tx=0x7f39d8009ab0).ready entity=mon.0 client_cookie=5778cda39a277c15 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:29:06.281 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:06.260 7f39e5b11700 1 -- 172.21.2.102:0/4137185712 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f39d800c0c0 con 0x7f39d0003a70 2021-02-10T11:29:06.281 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:06.260 7f39e5b11700 1 -- 172.21.2.102:0/4137185712 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39d800c0c0 con 0x7f39d0003a70 2021-02-10T11:29:06.281 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:06.260 7f39e5b11700 1 -- 172.21.2.102:0/4137185712 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f39d800c340 con 0x7f39d0003a70 2021-02-10T11:29:06.281 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:06.260 7f39ecf5c700 1 -- 172.21.2.102:0/4137185712 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f39d0003a70 msgr2=0x7f39d0005f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:06.281 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:06.260 7f39ecf5c700 1 --2- 172.21.2.102:0/4137185712 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f39d0003a70 0x7f39d0005f00 secure :-1 s=READY pgs=1865 cs=0 l=1 rev1=1 rx=0x7f39d8006ac0 tx=0x7f39d8009ab0).stop 2021-02-10T11:29:06.282 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:06.261 7f39e7314700 1 -- 172.21.2.102:0/4137185712 reap_dead start 2021-02-10T11:29:06.282 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:06.261 7f39ecf5c700 1 -- 172.21.2.102:0/4137185712 shutdown_connections 2021-02-10T11:29:06.282 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:06.261 7f39ecf5c700 1 -- 172.21.2.102:0/4137185712 >> 172.21.2.102:0/4137185712 conn(0x7f39e80ce450 msgr2=0x7f39e80d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:29:06.282 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:06.261 7f39ecf5c700 1 -- 172.21.2.102:0/4137185712 shutdown_connections 2021-02-10T11:29:06.282 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:06.261 7f39ecf5c700 1 -- 172.21.2.102:0/4137185712 wait complete. 2021-02-10T11:29:06.282 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:06.276 7f39ecf5c700 1 Processor -- start 2021-02-10T11:29:06.282 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:06.276 7f39ecf5c700 1 -- start start 2021-02-10T11:29:06.284 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:06.276 7f39ecf5c700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f39e8195650 0x7f39e80b5560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:06.284 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:06.276 7f39ecf5c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f39e80b5b80 0x7f39e81a2e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:06.284 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:06.276 7f39ecf5c700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f39e80acfa0 0x7f39e81a3440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:06.285 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:06.276 7f39ecf5c700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39e80b6010 con 0x7f39e80b5b80 2021-02-10T11:29:06.285 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:06.276 7f39ecf5c700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39e81a3a60 con 0x7f39e80acfa0 2021-02-10T11:29:06.285 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:06.276 7f39ecf5c700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39e81a3c40 con 0x7f39e8195650 2021-02-10T11:29:06.285 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:06.277 7f39e6312700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f39e80b5b80 0x7f39e81a2e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:06.285 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:06.277 7f39e6312700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f39e80b5b80 0x7f39e81a2e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:60316/0 (socket says 172.21.2.102:60316) 2021-02-10T11:29:06.285 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:06.277 7f39e6312700 1 -- 172.21.2.102:0/1876957958 learned_addr learned my addr 172.21.2.102:0/1876957958 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:29:06.285 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:06.277 7f39e6b13700 1 --2- 172.21.2.102:0/1876957958 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f39e8195650 0x7f39e80b5560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:06.285 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:06.277 7f39e7314700 1 --2- 172.21.2.102:0/1876957958 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f39e80acfa0 0x7f39e81a3440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:06.285 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:06.277 7f39e6312700 1 -- 172.21.2.102:0/1876957958 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f39e8195650 msgr2=0x7f39e80b5560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:06.285 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:06.277 7f39e6312700 1 --2- 172.21.2.102:0/1876957958 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f39e8195650 0x7f39e80b5560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:06.285 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:06.277 7f39e6312700 1 -- 172.21.2.102:0/1876957958 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f39e80acfa0 msgr2=0x7f39e81a3440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:06.285 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:06.277 7f39e6312700 1 --2- 172.21.2.102:0/1876957958 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f39e80acfa0 0x7f39e81a3440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:06.285 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:06.277 7f39e6312700 1 -- 172.21.2.102:0/1876957958 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39e80bcae0 con 0x7f39e80b5b80 2021-02-10T11:29:06.286 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:06.277 7f39e7314700 1 --2- 172.21.2.102:0/1876957958 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f39e80acfa0 0x7f39e81a3440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:29:06.286 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:06.277 7f39e6312700 1 --2- 172.21.2.102:0/1876957958 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f39e80b5b80 0x7f39e81a2e20 secure :-1 s=READY pgs=1866 cs=0 l=1 rev1=1 rx=0x7f39dc001bf0 tx=0x7f39dc00a900).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:29:06.286 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:06.277 7f39d77fe700 1 -- 172.21.2.102:0/1876957958 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f39dc00acf0 con 0x7f39e80b5b80 2021-02-10T11:29:06.286 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:06.277 7f39d77fe700 1 -- 172.21.2.102:0/1876957958 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39dc004350 con 0x7f39e80b5b80 2021-02-10T11:29:06.286 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:06.277 7f39ecf5c700 1 -- 172.21.2.102:0/1876957958 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39e80bcae0 con 0x7f39e80b5b80 2021-02-10T11:29:06.286 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:06.277 7f39ecf5c700 1 -- 172.21.2.102:0/1876957958 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39e80bd070 con 0x7f39e80b5b80 2021-02-10T11:29:06.286 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:06.278 7f39d77fe700 1 -- 172.21.2.102:0/1876957958 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f39dc004870 con 0x7f39e80b5b80 2021-02-10T11:29:06.286 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:06.278 7f39d77fe700 1 -- 172.21.2.102:0/1876957958 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f39dc004530 con 0x7f39e80b5b80 2021-02-10T11:29:06.286 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:06.278 7f39d77fe700 1 --2- 172.21.2.102:0/1876957958 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f39cc018730 0x7f39cc01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:06.286 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:06.278 7f39d77fe700 1 -- 172.21.2.102:0/1876957958 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f39dc0327b0 con 0x7f39e80b5b80 2021-02-10T11:29:06.286 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:06.278 7f39e6b13700 1 --2- 172.21.2.102:0/1876957958 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f39cc018730 0x7f39cc01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:06.287 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:06.278 7f39e6b13700 1 --2- 172.21.2.102:0/1876957958 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f39cc018730 0x7f39cc01ac10 crc :-1 s=READY pgs=837 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:29:06.308 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:06.308 7f39ecf5c700 1 -- 172.21.2.102:0/1876957958 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39e80f0ab0 con 0x7f39e80b5b80 2021-02-10T11:29:06.311 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:06.311 7f39d77fe700 1 -- 172.21.2.102:0/1876957958 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f39dc004560 con 0x7f39e80b5b80 2021-02-10T11:29:06.439 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:06.439 7f39ecf5c700 1 -- 172.21.2.102:0/1876957958 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f39e80f0ab0 con 0x7f39e80b5b80 2021-02-10T11:29:06.440 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:06.440 7f39d77fe700 1 -- 172.21.2.102:0/1876957958 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+2101 (secure 0 0 0) 0x7f39dc004350 con 0x7f39e80b5b80 2021-02-10T11:29:06.440 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:29:06.441 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":66,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6110,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.102:6835/1471871278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6834","nonce":1471871278},{"type":"v1","addr":"172.21.2.102:6835","nonce":1471871278}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":61}],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:27:41.479375","modified":"2021-02-10 11:27:43.490605","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6107},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6107":{"gid":6107,"name":"b","rank":0,"incarnation":62,"state":"up:active","state_seq":3,"addr":"172.21.2.113:6833/4220238062","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6832","nonce":4220238062},{"type":"v1","addr":"172.21.2.113:6833","nonce":4220238062}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":8}]} 2021-02-10T11:29:06.441 INFO:teuthology.orchestra.run.gibba002.stderr:dumped fsmap epoch 66 2021-02-10T11:29:06.442 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:06.442 7f39ecf5c700 1 -- 172.21.2.102:0/1876957958 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f39cc018730 msgr2=0x7f39cc01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:06.442 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:06.442 7f39ecf5c700 1 --2- 172.21.2.102:0/1876957958 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f39cc018730 0x7f39cc01ac10 crc :-1 s=READY pgs=837 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:06.442 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:06.442 7f39ecf5c700 1 -- 172.21.2.102:0/1876957958 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f39e80b5b80 msgr2=0x7f39e81a2e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:06.442 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:06.442 7f39ecf5c700 1 --2- 172.21.2.102:0/1876957958 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f39e80b5b80 0x7f39e81a2e20 secure :-1 s=READY pgs=1866 cs=0 l=1 rev1=1 rx=0x7f39dc001bf0 tx=0x7f39dc00a900).stop 2021-02-10T11:29:06.442 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:06.442 7f39ecf5c700 1 -- 172.21.2.102:0/1876957958 shutdown_connections 2021-02-10T11:29:06.443 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:06.442 7f39ecf5c700 1 --2- 172.21.2.102:0/1876957958 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f39cc018730 0x7f39cc01ac10 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:06.443 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:06.442 7f39ecf5c700 1 --2- 172.21.2.102:0/1876957958 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f39e80acfa0 0x7f39e81a3440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:06.443 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:06.442 7f39ecf5c700 1 --2- 172.21.2.102:0/1876957958 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f39e80b5b80 0x7f39e81a2e20 unknown :-1 s=CLOSED pgs=1866 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:06.443 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:06.442 7f39ecf5c700 1 --2- 172.21.2.102:0/1876957958 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f39e8195650 0x7f39e80b5560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:06.443 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:06.442 7f39ecf5c700 1 -- 172.21.2.102:0/1876957958 >> 172.21.2.102:0/1876957958 conn(0x7f39e80b3040 msgr2=0x7f39e80c3250 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:29:06.443 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:06.442 7f39ecf5c700 1 -- 172.21.2.102:0/1876957958 shutdown_connections 2021-02-10T11:29:06.443 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:06.442 7f39ecf5c700 1 -- 172.21.2.102:0/1876957958 wait complete. 2021-02-10T11:29:06.451 DEBUG:teuthology.orchestra.run.gibba013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok session ls 2021-02-10T11:29:06.848 INFO:teuthology.orchestra.run.gibba013.stdout:[ 2021-02-10T11:29:06.848 INFO:teuthology.orchestra.run.gibba013.stdout: { 2021-02-10T11:29:06.848 INFO:teuthology.orchestra.run.gibba013.stdout: "id": 9471, 2021-02-10T11:29:06.849 INFO:teuthology.orchestra.run.gibba013.stdout: "entity": { 2021-02-10T11:29:06.849 INFO:teuthology.orchestra.run.gibba013.stdout: "name": { 2021-02-10T11:29:06.849 INFO:teuthology.orchestra.run.gibba013.stdout: "type": "client", 2021-02-10T11:29:06.849 INFO:teuthology.orchestra.run.gibba013.stdout: "num": 9471 2021-02-10T11:29:06.849 INFO:teuthology.orchestra.run.gibba013.stdout: }, 2021-02-10T11:29:06.849 INFO:teuthology.orchestra.run.gibba013.stdout: "addr": { 2021-02-10T11:29:06.849 INFO:teuthology.orchestra.run.gibba013.stdout: "type": "any", 2021-02-10T11:29:06.850 INFO:teuthology.orchestra.run.gibba013.stdout: "addr": "172.21.2.102:0", 2021-02-10T11:29:06.850 INFO:teuthology.orchestra.run.gibba013.stdout: "nonce": 190431005 2021-02-10T11:29:06.850 INFO:teuthology.orchestra.run.gibba013.stdout: } 2021-02-10T11:29:06.850 INFO:teuthology.orchestra.run.gibba013.stdout: }, 2021-02-10T11:29:06.850 INFO:teuthology.orchestra.run.gibba013.stdout: "state": "open", 2021-02-10T11:29:06.850 INFO:teuthology.orchestra.run.gibba013.stdout: "num_leases": 0, 2021-02-10T11:29:06.851 INFO:teuthology.orchestra.run.gibba013.stdout: "num_caps": 7, 2021-02-10T11:29:06.851 INFO:teuthology.orchestra.run.gibba013.stdout: "request_load_avg": 7, 2021-02-10T11:29:06.851 INFO:teuthology.orchestra.run.gibba013.stdout: "uptime": 56.171002108000003, 2021-02-10T11:29:06.851 INFO:teuthology.orchestra.run.gibba013.stdout: "requests_in_flight": 0, 2021-02-10T11:29:06.851 INFO:teuthology.orchestra.run.gibba013.stdout: "completed_requests": 1, 2021-02-10T11:29:06.851 INFO:teuthology.orchestra.run.gibba013.stdout: "reconnecting": false, 2021-02-10T11:29:06.851 INFO:teuthology.orchestra.run.gibba013.stdout: "recall_caps": { 2021-02-10T11:29:06.851 INFO:teuthology.orchestra.run.gibba013.stdout: "value": 0, 2021-02-10T11:29:06.851 INFO:teuthology.orchestra.run.gibba013.stdout: "halflife": 60 2021-02-10T11:29:06.851 INFO:teuthology.orchestra.run.gibba013.stdout: }, 2021-02-10T11:29:06.851 INFO:teuthology.orchestra.run.gibba013.stdout: "release_caps": { 2021-02-10T11:29:06.852 INFO:teuthology.orchestra.run.gibba013.stdout: "value": 0, 2021-02-10T11:29:06.852 INFO:teuthology.orchestra.run.gibba013.stdout: "halflife": 60 2021-02-10T11:29:06.852 INFO:teuthology.orchestra.run.gibba013.stdout: }, 2021-02-10T11:29:06.852 INFO:teuthology.orchestra.run.gibba013.stdout: "recall_caps_throttle": { 2021-02-10T11:29:06.852 INFO:teuthology.orchestra.run.gibba013.stdout: "value": 0, 2021-02-10T11:29:06.852 INFO:teuthology.orchestra.run.gibba013.stdout: "halflife": 2.5 2021-02-10T11:29:06.852 INFO:teuthology.orchestra.run.gibba013.stdout: }, 2021-02-10T11:29:06.852 INFO:teuthology.orchestra.run.gibba013.stdout: "recall_caps_throttle2o": { 2021-02-10T11:29:06.852 INFO:teuthology.orchestra.run.gibba013.stdout: "value": 0, 2021-02-10T11:29:06.852 INFO:teuthology.orchestra.run.gibba013.stdout: "halflife": 0.5 2021-02-10T11:29:06.852 INFO:teuthology.orchestra.run.gibba013.stdout: }, 2021-02-10T11:29:06.852 INFO:teuthology.orchestra.run.gibba013.stdout: "session_cache_liveness": { 2021-02-10T11:29:06.852 INFO:teuthology.orchestra.run.gibba013.stdout: "value": 18.456704513202137, 2021-02-10T11:29:06.852 INFO:teuthology.orchestra.run.gibba013.stdout: "halflife": 300 2021-02-10T11:29:06.853 INFO:teuthology.orchestra.run.gibba013.stdout: }, 2021-02-10T11:29:06.853 INFO:teuthology.orchestra.run.gibba013.stdout: "cap_acquisition": { 2021-02-10T11:29:06.853 INFO:teuthology.orchestra.run.gibba013.stdout: "value": 0, 2021-02-10T11:29:06.853 INFO:teuthology.orchestra.run.gibba013.stdout: "halflife": 10 2021-02-10T11:29:06.853 INFO:teuthology.orchestra.run.gibba013.stdout: }, 2021-02-10T11:29:06.853 INFO:teuthology.orchestra.run.gibba013.stdout: "inst": "client.9471 172.21.2.102:0/190431005", 2021-02-10T11:29:06.853 INFO:teuthology.orchestra.run.gibba013.stdout: "completed_requests": [ 2021-02-10T11:29:06.853 INFO:teuthology.orchestra.run.gibba013.stdout: { 2021-02-10T11:29:06.853 INFO:teuthology.orchestra.run.gibba013.stdout: "tid": 14, 2021-02-10T11:29:06.853 INFO:teuthology.orchestra.run.gibba013.stdout: "created_ino": "0x10000000005" 2021-02-10T11:29:06.853 INFO:teuthology.orchestra.run.gibba013.stdout: } 2021-02-10T11:29:06.853 INFO:teuthology.orchestra.run.gibba013.stdout: ], 2021-02-10T11:29:06.854 INFO:teuthology.orchestra.run.gibba013.stdout: "prealloc_inos": [ 2021-02-10T11:29:06.854 INFO:teuthology.orchestra.run.gibba013.stdout: { 2021-02-10T11:29:06.854 INFO:teuthology.orchestra.run.gibba013.stdout: "start": 1099511627782, 2021-02-10T11:29:06.854 INFO:teuthology.orchestra.run.gibba013.stdout: "length": 995 2021-02-10T11:29:06.854 INFO:teuthology.orchestra.run.gibba013.stdout: } 2021-02-10T11:29:06.855 INFO:teuthology.orchestra.run.gibba013.stdout: ], 2021-02-10T11:29:06.855 INFO:teuthology.orchestra.run.gibba013.stdout: "used_inos": [], 2021-02-10T11:29:06.855 INFO:teuthology.orchestra.run.gibba013.stdout: "client_metadata": { 2021-02-10T11:29:06.855 INFO:teuthology.orchestra.run.gibba013.stdout: "features": "0x0000000000001fff", 2021-02-10T11:29:06.855 INFO:teuthology.orchestra.run.gibba013.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:29:06.855 INFO:teuthology.orchestra.run.gibba013.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:29:06.855 INFO:teuthology.orchestra.run.gibba013.stdout: "entity_id": "y", 2021-02-10T11:29:06.855 INFO:teuthology.orchestra.run.gibba013.stdout: "hostname": "gibba002", 2021-02-10T11:29:06.855 INFO:teuthology.orchestra.run.gibba013.stdout: "pid": "35161", 2021-02-10T11:29:06.855 INFO:teuthology.orchestra.run.gibba013.stdout: "root": "/" 2021-02-10T11:29:06.855 INFO:teuthology.orchestra.run.gibba013.stdout: } 2021-02-10T11:29:06.855 INFO:teuthology.orchestra.run.gibba013.stdout: } 2021-02-10T11:29:06.855 INFO:teuthology.orchestra.run.gibba013.stdout:] 2021-02-10T11:29:06.857 INFO:tasks.cephfs.filesystem:_json_asok output: [ { "id": 9471, "entity": { "name": { "type": "client", "num": 9471 }, "addr": { "type": "any", "addr": "172.21.2.102:0", "nonce": 190431005 } }, "state": "open", "num_leases": 0, "num_caps": 7, "request_load_avg": 7, "uptime": 56.171002108000003, "requests_in_flight": 0, "completed_requests": 1, "reconnecting": false, "recall_caps": { "value": 0, "halflife": 60 }, "release_caps": { "value": 0, "halflife": 60 }, "recall_caps_throttle": { "value": 0, "halflife": 2.5 }, "recall_caps_throttle2o": { "value": 0, "halflife": 0.5 }, "session_cache_liveness": { "value": 18.456704513202137, "halflife": 300 }, "cap_acquisition": { "value": 0, "halflife": 10 }, "inst": "client.9471 172.21.2.102:0/190431005", "completed_requests": [ { "tid": 14, "created_ino": "0x10000000005" } ], "prealloc_inos": [ { "start": 1099511627782, "length": 995 } ], "used_inos": [], "client_metadata": { "features": "0x0000000000001fff", "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", "entity_id": "y", "hostname": "gibba002", "pid": "35161", "root": "/" } } ] 2021-02-10T11:29:06.857 DEBUG:tasks.ceph_test_case:wait_until_true: waiting... 2021-02-10T11:29:07.547 DEBUG:teuthology.orchestra.run.gibba002:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:29:07.551 DEBUG:teuthology.orchestra.run.gibba013:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:29:11.858 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:29:12.285 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:12.270 7effb6623700 1 Processor -- start 2021-02-10T11:29:12.286 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:12.270 7effb6623700 1 -- start start 2021-02-10T11:29:12.286 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:12.270 7effb6623700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7effb00d1870 0x7effb01b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:12.286 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:12.270 7effb6623700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7effb01b0a10 0x7effb00cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:12.286 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:12.270 7effb6623700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7effb00d1c70 0x7effb00cf950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:12.286 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:12.270 7effb6623700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7effb00e5ed0 con 0x7effb01b0a10 2021-02-10T11:29:12.286 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:12.271 7effb6623700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7effb00f0b10 con 0x7effb00d1870 2021-02-10T11:29:12.286 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:12.271 7effb6623700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effb00cff70 con 0x7effb00d1c70 2021-02-10T11:29:12.286 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:12.271 7effaf7fe700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7effb01b0a10 0x7effb00cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:12.286 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:12.271 7effaf7fe700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7effb01b0a10 0x7effb00cf330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:60324/0 (socket says 172.21.2.102:60324) 2021-02-10T11:29:12.286 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:12.271 7effaf7fe700 1 -- 172.21.2.102:0/3700387825 learned_addr learned my addr 172.21.2.102:0/3700387825 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:29:12.287 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:12.271 7effaf7fe700 1 -- 172.21.2.102:0/3700387825 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7effb00d1c70 msgr2=0x7effb00cf950 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:29:12.287 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:12.271 7effaf7fe700 1 --2- 172.21.2.102:0/3700387825 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7effb00d1c70 0x7effb00cf950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:12.287 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:12.271 7effaf7fe700 1 -- 172.21.2.102:0/3700387825 >> v2:172.21.2.113:3300/0 conn(0x7effb00d1870 msgr2=0x7effb01b03f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:29:12.287 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:12.271 7effaf7fe700 1 --2- 172.21.2.102:0/3700387825 >> v2:172.21.2.113:3300/0 conn(0x7effb00d1870 0x7effb01b03f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:12.287 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:12.271 7effaf7fe700 1 -- 172.21.2.102:0/3700387825 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effb00d0150 con 0x7effb01b0a10 2021-02-10T11:29:12.287 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:12.271 7effaf7fe700 1 --2- 172.21.2.102:0/3700387825 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7effb01b0a10 0x7effb00cf330 secure :-1 s=READY pgs=1867 cs=0 l=1 rev1=1 rx=0x7effa4006700 tx=0x7effa4009b80).ready entity=mon.0 client_cookie=44f67cc6111ab0a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:29:12.287 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:12.271 7effaeffd700 1 -- 172.21.2.102:0/3700387825 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7effa400c0c0 con 0x7effb01b0a10 2021-02-10T11:29:12.287 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:12.271 7effaeffd700 1 -- 172.21.2.102:0/3700387825 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effa400c440 con 0x7effb01b0a10 2021-02-10T11:29:12.287 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:12.271 7effaeffd700 1 -- 172.21.2.102:0/3700387825 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7effa400c8a0 con 0x7effb01b0a10 2021-02-10T11:29:12.287 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:12.272 7effb6623700 1 -- 172.21.2.102:0/3700387825 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7effb01b0a10 msgr2=0x7effb00cf330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:12.287 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:12.272 7effb6623700 1 --2- 172.21.2.102:0/3700387825 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7effb01b0a10 0x7effb00cf330 secure :-1 s=READY pgs=1867 cs=0 l=1 rev1=1 rx=0x7effa4006700 tx=0x7effa4009b80).stop 2021-02-10T11:29:12.287 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:12.272 7effb6623700 1 -- 172.21.2.102:0/3700387825 shutdown_connections 2021-02-10T11:29:12.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:12.272 7effb6623700 1 --2- 172.21.2.102:0/3700387825 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7effb00d1c70 0x7effb00cf950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:12.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:12.272 7effb6623700 1 --2- 172.21.2.102:0/3700387825 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7effb01b0a10 0x7effb00cf330 unknown :-1 s=CLOSED pgs=1867 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:12.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:12.272 7effb6623700 1 --2- 172.21.2.102:0/3700387825 >> v2:172.21.2.113:3300/0 conn(0x7effb00d1870 0x7effb01b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:12.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:12.272 7effb6623700 1 -- 172.21.2.102:0/3700387825 >> 172.21.2.102:0/3700387825 conn(0x7effb00ce450 msgr2=0x7effb00d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:29:12.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:12.272 7effb6623700 1 -- 172.21.2.102:0/3700387825 shutdown_connections 2021-02-10T11:29:12.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:12.272 7effb6623700 1 -- 172.21.2.102:0/3700387825 wait complete. 2021-02-10T11:29:12.289 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:12.286 7effb6623700 1 Processor -- start 2021-02-10T11:29:12.290 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:12.287 7effb6623700 1 -- start start 2021-02-10T11:29:12.290 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:12.287 7effb6623700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7effb01b7cf0 0x7effb019cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:12.290 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:12.287 7effb6623700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7effb01b80f0 0x7effb01b84f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:12.290 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:12.287 7effb6623700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7effb019da20 0x7effb019de20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:12.290 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:12.287 7effb6623700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effb00a77c0 con 0x7effb019da20 2021-02-10T11:29:12.290 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:12.287 7effb6623700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effb019d4a0 con 0x7effb01b80f0 2021-02-10T11:29:12.290 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:12.287 7effb6623700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effb00954f0 con 0x7effb01b7cf0 2021-02-10T11:29:12.290 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:12.287 7effb4c40700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7effb019da20 0x7effb019de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:12.290 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:12.287 7effb4c40700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7effb019da20 0x7effb019de20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:60330/0 (socket says 172.21.2.102:60330) 2021-02-10T11:29:12.290 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:12.287 7effb4c40700 1 -- 172.21.2.102:0/476894487 learned_addr learned my addr 172.21.2.102:0/476894487 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:29:12.290 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:12.287 7effaf7fe700 1 --2- 172.21.2.102:0/476894487 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7effb01b80f0 0x7effb01b84f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:12.290 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:12.287 7effaffff700 1 --2- 172.21.2.102:0/476894487 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7effb01b7cf0 0x7effb019cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:12.291 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:12.287 7effb4c40700 1 -- 172.21.2.102:0/476894487 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7effb01b7cf0 msgr2=0x7effb019cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:12.291 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:12.287 7effb4c40700 1 --2- 172.21.2.102:0/476894487 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7effb01b7cf0 0x7effb019cdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:12.291 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:12.287 7effb4c40700 1 -- 172.21.2.102:0/476894487 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7effb01b80f0 msgr2=0x7effb01b84f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:12.291 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:12.287 7effb4c40700 1 --2- 172.21.2.102:0/476894487 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7effb01b80f0 0x7effb01b84f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:12.291 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:12.287 7effb4c40700 1 -- 172.21.2.102:0/476894487 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effb01b8d60 con 0x7effb019da20 2021-02-10T11:29:12.291 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:12.287 7effaffff700 1 --2- 172.21.2.102:0/476894487 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7effb01b7cf0 0x7effb019cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:29:12.291 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:12.288 7effb4c40700 1 --2- 172.21.2.102:0/476894487 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7effb019da20 0x7effb019de20 secure :-1 s=READY pgs=1868 cs=0 l=1 rev1=1 rx=0x7effa000b5c0 tx=0x7effa000b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:29:12.291 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:12.288 7effacff9700 1 -- 172.21.2.102:0/476894487 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7effa000bd50 con 0x7effb019da20 2021-02-10T11:29:12.291 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:12.288 7effacff9700 1 -- 172.21.2.102:0/476894487 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effa0006030 con 0x7effb019da20 2021-02-10T11:29:12.291 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:12.288 7effb6623700 1 -- 172.21.2.102:0/476894487 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effb01b8d60 con 0x7effb019da20 2021-02-10T11:29:12.291 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:12.288 7effacff9700 1 -- 172.21.2.102:0/476894487 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7effa00054d0 con 0x7effb019da20 2021-02-10T11:29:12.291 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:12.288 7effb6623700 1 -- 172.21.2.102:0/476894487 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effb01b8d60 con 0x7effb019da20 2021-02-10T11:29:12.291 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:12.288 7effacff9700 1 -- 172.21.2.102:0/476894487 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7effa0006210 con 0x7effb019da20 2021-02-10T11:29:12.291 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:12.289 7effacff9700 1 --2- 172.21.2.102:0/476894487 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7eff90018730 0x7eff9001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:12.292 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:12.289 7effacff9700 1 -- 172.21.2.102:0/476894487 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 5573+0+0 (secure 0 0 0) 0x7effa0032640 con 0x7effb019da20 2021-02-10T11:29:12.292 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:12.289 7effaffff700 1 --2- 172.21.2.102:0/476894487 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7eff90018730 0x7eff9001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:12.292 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:12.289 7effaffff700 1 --2- 172.21.2.102:0/476894487 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7eff90018730 0x7eff9001ac10 crc :-1 s=READY pgs=838 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:29:12.320 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:12.320 7effb6623700 1 -- 172.21.2.102:0/476894487 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effb00ebc00 con 0x7effb019da20 2021-02-10T11:29:12.322 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:12.323 7effacff9700 1 -- 172.21.2.102:0/476894487 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7effa0006b80 con 0x7effb019da20 2021-02-10T11:29:12.450 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:12.450 7effb6623700 1 -- 172.21.2.102:0/476894487 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7effb00f0b10 con 0x7effb019da20 2021-02-10T11:29:12.450 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:12.451 7effacff9700 1 -- 172.21.2.102:0/476894487 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+2101 (secure 0 0 0) 0x7effa000bd50 con 0x7effb019da20 2021-02-10T11:29:12.451 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:29:12.451 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":66,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6110,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.102:6835/1471871278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6834","nonce":1471871278},{"type":"v1","addr":"172.21.2.102:6835","nonce":1471871278}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":61}],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:27:41.479375","modified":"2021-02-10 11:27:43.490605","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6107},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6107":{"gid":6107,"name":"b","rank":0,"incarnation":62,"state":"up:active","state_seq":3,"addr":"172.21.2.113:6833/4220238062","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6832","nonce":4220238062},{"type":"v1","addr":"172.21.2.113:6833","nonce":4220238062}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":8}]} 2021-02-10T11:29:12.451 INFO:teuthology.orchestra.run.gibba002.stderr:dumped fsmap epoch 66 2021-02-10T11:29:12.452 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:12.453 7effb6623700 1 -- 172.21.2.102:0/476894487 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7eff90018730 msgr2=0x7eff9001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:12.452 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:12.453 7effb6623700 1 --2- 172.21.2.102:0/476894487 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7eff90018730 0x7eff9001ac10 crc :-1 s=READY pgs=838 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:12.452 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:12.453 7effb6623700 1 -- 172.21.2.102:0/476894487 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7effb019da20 msgr2=0x7effb019de20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:12.452 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:12.453 7effb6623700 1 --2- 172.21.2.102:0/476894487 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7effb019da20 0x7effb019de20 secure :-1 s=READY pgs=1868 cs=0 l=1 rev1=1 rx=0x7effa000b5c0 tx=0x7effa000b960).stop 2021-02-10T11:29:12.453 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:12.453 7effb6623700 1 -- 172.21.2.102:0/476894487 shutdown_connections 2021-02-10T11:29:12.453 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:12.453 7effb6623700 1 --2- 172.21.2.102:0/476894487 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7effb019da20 0x7effb019de20 unknown :-1 s=CLOSED pgs=1868 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:12.453 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:12.453 7effb6623700 1 --2- 172.21.2.102:0/476894487 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7eff90018730 0x7eff9001ac10 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:12.453 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:12.453 7effb6623700 1 --2- 172.21.2.102:0/476894487 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7effb01b80f0 0x7effb01b84f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:12.453 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:12.453 7effb6623700 1 --2- 172.21.2.102:0/476894487 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7effb01b7cf0 0x7effb019cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:12.453 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:12.453 7effb6623700 1 -- 172.21.2.102:0/476894487 >> 172.21.2.102:0/476894487 conn(0x7effb00ed810 msgr2=0x7effb0094980 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:29:12.453 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:12.453 7effb6623700 1 -- 172.21.2.102:0/476894487 shutdown_connections 2021-02-10T11:29:12.453 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:12.453 7effb6623700 1 -- 172.21.2.102:0/476894487 wait complete. 2021-02-10T11:29:12.461 DEBUG:teuthology.orchestra.run.gibba013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok session ls 2021-02-10T11:29:12.859 INFO:teuthology.orchestra.run.gibba013.stdout:[ 2021-02-10T11:29:12.859 INFO:teuthology.orchestra.run.gibba013.stdout: { 2021-02-10T11:29:12.859 INFO:teuthology.orchestra.run.gibba013.stdout: "id": 9471, 2021-02-10T11:29:12.859 INFO:teuthology.orchestra.run.gibba013.stdout: "entity": { 2021-02-10T11:29:12.859 INFO:teuthology.orchestra.run.gibba013.stdout: "name": { 2021-02-10T11:29:12.859 INFO:teuthology.orchestra.run.gibba013.stdout: "type": "client", 2021-02-10T11:29:12.860 INFO:teuthology.orchestra.run.gibba013.stdout: "num": 9471 2021-02-10T11:29:12.860 INFO:teuthology.orchestra.run.gibba013.stdout: }, 2021-02-10T11:29:12.860 INFO:teuthology.orchestra.run.gibba013.stdout: "addr": { 2021-02-10T11:29:12.860 INFO:teuthology.orchestra.run.gibba013.stdout: "type": "any", 2021-02-10T11:29:12.860 INFO:teuthology.orchestra.run.gibba013.stdout: "addr": "172.21.2.102:0", 2021-02-10T11:29:12.860 INFO:teuthology.orchestra.run.gibba013.stdout: "nonce": 190431005 2021-02-10T11:29:12.860 INFO:teuthology.orchestra.run.gibba013.stdout: } 2021-02-10T11:29:12.860 INFO:teuthology.orchestra.run.gibba013.stdout: }, 2021-02-10T11:29:12.860 INFO:teuthology.orchestra.run.gibba013.stdout: "state": "open", 2021-02-10T11:29:12.860 INFO:teuthology.orchestra.run.gibba013.stdout: "num_leases": 0, 2021-02-10T11:29:12.860 INFO:teuthology.orchestra.run.gibba013.stdout: "num_caps": 7, 2021-02-10T11:29:12.860 INFO:teuthology.orchestra.run.gibba013.stdout: "request_load_avg": 6, 2021-02-10T11:29:12.860 INFO:teuthology.orchestra.run.gibba013.stdout: "uptime": 62.182002251, 2021-02-10T11:29:12.861 INFO:teuthology.orchestra.run.gibba013.stdout: "requests_in_flight": 0, 2021-02-10T11:29:12.861 INFO:teuthology.orchestra.run.gibba013.stdout: "completed_requests": 1, 2021-02-10T11:29:12.861 INFO:teuthology.orchestra.run.gibba013.stdout: "reconnecting": false, 2021-02-10T11:29:12.861 INFO:teuthology.orchestra.run.gibba013.stdout: "recall_caps": { 2021-02-10T11:29:12.861 INFO:teuthology.orchestra.run.gibba013.stdout: "value": 0, 2021-02-10T11:29:12.861 INFO:teuthology.orchestra.run.gibba013.stdout: "halflife": 60 2021-02-10T11:29:12.861 INFO:teuthology.orchestra.run.gibba013.stdout: }, 2021-02-10T11:29:12.861 INFO:teuthology.orchestra.run.gibba013.stdout: "release_caps": { 2021-02-10T11:29:12.861 INFO:teuthology.orchestra.run.gibba013.stdout: "value": 0, 2021-02-10T11:29:12.861 INFO:teuthology.orchestra.run.gibba013.stdout: "halflife": 60 2021-02-10T11:29:12.861 INFO:teuthology.orchestra.run.gibba013.stdout: }, 2021-02-10T11:29:12.861 INFO:teuthology.orchestra.run.gibba013.stdout: "recall_caps_throttle": { 2021-02-10T11:29:12.861 INFO:teuthology.orchestra.run.gibba013.stdout: "value": 0, 2021-02-10T11:29:12.862 INFO:teuthology.orchestra.run.gibba013.stdout: "halflife": 2.5 2021-02-10T11:29:12.862 INFO:teuthology.orchestra.run.gibba013.stdout: }, 2021-02-10T11:29:12.862 INFO:teuthology.orchestra.run.gibba013.stdout: "recall_caps_throttle2o": { 2021-02-10T11:29:12.862 INFO:teuthology.orchestra.run.gibba013.stdout: "value": 0, 2021-02-10T11:29:12.862 INFO:teuthology.orchestra.run.gibba013.stdout: "halflife": 0.5 2021-02-10T11:29:12.862 INFO:teuthology.orchestra.run.gibba013.stdout: }, 2021-02-10T11:29:12.862 INFO:teuthology.orchestra.run.gibba013.stdout: "session_cache_liveness": { 2021-02-10T11:29:12.862 INFO:teuthology.orchestra.run.gibba013.stdout: "value": 18.202142981513951, 2021-02-10T11:29:12.862 INFO:teuthology.orchestra.run.gibba013.stdout: "halflife": 300 2021-02-10T11:29:12.862 INFO:teuthology.orchestra.run.gibba013.stdout: }, 2021-02-10T11:29:12.862 INFO:teuthology.orchestra.run.gibba013.stdout: "cap_acquisition": { 2021-02-10T11:29:12.862 INFO:teuthology.orchestra.run.gibba013.stdout: "value": 0, 2021-02-10T11:29:12.862 INFO:teuthology.orchestra.run.gibba013.stdout: "halflife": 10 2021-02-10T11:29:12.863 INFO:teuthology.orchestra.run.gibba013.stdout: }, 2021-02-10T11:29:12.863 INFO:teuthology.orchestra.run.gibba013.stdout: "inst": "client.9471 172.21.2.102:0/190431005", 2021-02-10T11:29:12.863 INFO:teuthology.orchestra.run.gibba013.stdout: "completed_requests": [ 2021-02-10T11:29:12.863 INFO:teuthology.orchestra.run.gibba013.stdout: { 2021-02-10T11:29:12.863 INFO:teuthology.orchestra.run.gibba013.stdout: "tid": 14, 2021-02-10T11:29:12.863 INFO:teuthology.orchestra.run.gibba013.stdout: "created_ino": "0x10000000005" 2021-02-10T11:29:12.863 INFO:teuthology.orchestra.run.gibba013.stdout: } 2021-02-10T11:29:12.863 INFO:teuthology.orchestra.run.gibba013.stdout: ], 2021-02-10T11:29:12.863 INFO:teuthology.orchestra.run.gibba013.stdout: "prealloc_inos": [ 2021-02-10T11:29:12.863 INFO:teuthology.orchestra.run.gibba013.stdout: { 2021-02-10T11:29:12.863 INFO:teuthology.orchestra.run.gibba013.stdout: "start": 1099511627782, 2021-02-10T11:29:12.863 INFO:teuthology.orchestra.run.gibba013.stdout: "length": 995 2021-02-10T11:29:12.863 INFO:teuthology.orchestra.run.gibba013.stdout: } 2021-02-10T11:29:12.864 INFO:teuthology.orchestra.run.gibba013.stdout: ], 2021-02-10T11:29:12.864 INFO:teuthology.orchestra.run.gibba013.stdout: "used_inos": [], 2021-02-10T11:29:12.864 INFO:teuthology.orchestra.run.gibba013.stdout: "client_metadata": { 2021-02-10T11:29:12.864 INFO:teuthology.orchestra.run.gibba013.stdout: "features": "0x0000000000001fff", 2021-02-10T11:29:12.864 INFO:teuthology.orchestra.run.gibba013.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:29:12.864 INFO:teuthology.orchestra.run.gibba013.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:29:12.864 INFO:teuthology.orchestra.run.gibba013.stdout: "entity_id": "y", 2021-02-10T11:29:12.864 INFO:teuthology.orchestra.run.gibba013.stdout: "hostname": "gibba002", 2021-02-10T11:29:12.864 INFO:teuthology.orchestra.run.gibba013.stdout: "pid": "35161", 2021-02-10T11:29:12.864 INFO:teuthology.orchestra.run.gibba013.stdout: "root": "/" 2021-02-10T11:29:12.864 INFO:teuthology.orchestra.run.gibba013.stdout: } 2021-02-10T11:29:12.864 INFO:teuthology.orchestra.run.gibba013.stdout: } 2021-02-10T11:29:12.864 INFO:teuthology.orchestra.run.gibba013.stdout:] 2021-02-10T11:29:12.865 INFO:tasks.cephfs.filesystem:_json_asok output: [ { "id": 9471, "entity": { "name": { "type": "client", "num": 9471 }, "addr": { "type": "any", "addr": "172.21.2.102:0", "nonce": 190431005 } }, "state": "open", "num_leases": 0, "num_caps": 7, "request_load_avg": 6, "uptime": 62.182002251, "requests_in_flight": 0, "completed_requests": 1, "reconnecting": false, "recall_caps": { "value": 0, "halflife": 60 }, "release_caps": { "value": 0, "halflife": 60 }, "recall_caps_throttle": { "value": 0, "halflife": 2.5 }, "recall_caps_throttle2o": { "value": 0, "halflife": 0.5 }, "session_cache_liveness": { "value": 18.202142981513951, "halflife": 300 }, "cap_acquisition": { "value": 0, "halflife": 10 }, "inst": "client.9471 172.21.2.102:0/190431005", "completed_requests": [ { "tid": 14, "created_ino": "0x10000000005" } ], "prealloc_inos": [ { "start": 1099511627782, "length": 995 } ], "used_inos": [], "client_metadata": { "features": "0x0000000000001fff", "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", "entity_id": "y", "hostname": "gibba002", "pid": "35161", "root": "/" } } ] 2021-02-10T11:29:12.865 DEBUG:tasks.ceph_test_case:wait_until_true: waiting... 2021-02-10T11:29:17.867 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:29:18.295 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:18.279 7f448512b700 1 Processor -- start 2021-02-10T11:29:18.295 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:18.279 7f448512b700 1 -- start start 2021-02-10T11:29:18.295 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:18.280 7f448512b700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f44800d1870 0x7f44801b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:18.295 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:18.280 7f448512b700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44800f0b10 con 0x7f44801b0a10 2021-02-10T11:29:18.295 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:18.280 7f448512b700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44800d0310 con 0x7f44800f0650 2021-02-10T11:29:18.295 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:18.280 7f448512b700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44800d0530 con 0x7f44800d1870 2021-02-10T11:29:18.295 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:18.280 7f447e59c700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f44801b0a10 0x7f44800cf330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:52562/0 (socket says 172.21.2.102:52562) 2021-02-10T11:29:18.295 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:18.280 7f447ed9d700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f44800d1870 0x7f44801b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:18.295 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:18.280 7f447e59c700 1 -- 172.21.2.102:0/3415889785 learned_addr learned my addr 172.21.2.102:0/3415889785 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:29:18.295 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:18.280 7f447ed9d700 1 -- 172.21.2.102:0/3415889785 >> v1:172.21.2.113:6789/0 conn(0x7f44800f0650 legacy=0x7f44800cfb20 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:29:18.296 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:18.280 7f447ed9d700 1 -- 172.21.2.102:0/3415889785 >> v1:172.21.2.102:6789/0 conn(0x7f44801b0a10 legacy=0x7f44800cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:18.296 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:18.280 7f447ed9d700 1 -- 172.21.2.102:0/3415889785 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44800d1c70 con 0x7f44800d1870 2021-02-10T11:29:18.296 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:18.280 7f447dd9b700 1 -- 172.21.2.102:0/3415889785 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 877019898 0 0) 0x7f44700010b0 con 0x7f44801b0a10 2021-02-10T11:29:18.296 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:18.280 7f447ed9d700 1 --2- 172.21.2.102:0/3415889785 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f44800d1870 0x7f44801b03f0 secure :-1 s=READY pgs=1870 cs=0 l=1 rev1=1 rx=0x7f4468006650 tx=0x7f4468009a00).ready entity=mon.0 client_cookie=23d104fdcff9d1d0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:29:18.296 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:18.280 7f447dd9b700 1 -- 172.21.2.102:0/3415889785 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f44680009b0 con 0x7f44800d1870 2021-02-10T11:29:18.296 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:18.280 7f447dd9b700 1 -- 172.21.2.102:0/3415889785 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f446800c0c0 con 0x7f44800d1870 2021-02-10T11:29:18.296 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:18.280 7f447dd9b700 1 -- 172.21.2.102:0/3415889785 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f446800c510 con 0x7f44800d1870 2021-02-10T11:29:18.296 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:18.280 7f448512b700 1 -- 172.21.2.102:0/3415889785 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f44800d1870 msgr2=0x7f44801b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:18.296 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:18.280 7f448512b700 1 --2- 172.21.2.102:0/3415889785 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f44800d1870 0x7f44801b03f0 secure :-1 s=READY pgs=1870 cs=0 l=1 rev1=1 rx=0x7f4468006650 tx=0x7f4468009a00).stop 2021-02-10T11:29:18.296 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:18.281 7f448512b700 1 -- 172.21.2.102:0/3415889785 shutdown_connections 2021-02-10T11:29:18.296 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:18.281 7f448512b700 1 --2- 172.21.2.102:0/3415889785 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f44800d1870 0x7f44801b03f0 unknown :-1 s=CLOSED pgs=1870 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:18.296 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:18.281 7f448512b700 1 -- 172.21.2.102:0/3415889785 >> 172.21.2.102:0/3415889785 conn(0x7f44800ce450 msgr2=0x7f44800d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:29:18.296 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:18.281 7f448512b700 1 -- 172.21.2.102:0/3415889785 shutdown_connections 2021-02-10T11:29:18.297 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:18.281 7f448512b700 1 -- 172.21.2.102:0/3415889785 wait complete. 2021-02-10T11:29:18.298 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:18.296 7f448512b700 1 Processor -- start 2021-02-10T11:29:18.298 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:18.296 7f448512b700 1 -- start start 2021-02-10T11:29:18.298 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:18.296 7f448512b700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f44800b0ad0 0x7f4480091f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:18.298 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:18.296 7f448512b700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f44801a8850 0x7f44801a8c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:18.298 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:18.296 7f448512b700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f44801a6fa0 0x7f44801a73a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:18.298 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:18.296 7f448512b700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44801abfe0 con 0x7f44801a8850 2021-02-10T11:29:18.298 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:18.296 7f448512b700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44801a9c40 con 0x7f44801a6fa0 2021-02-10T11:29:18.299 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:18.296 7f448512b700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44801a9730 con 0x7f44800b0ad0 2021-02-10T11:29:18.299 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:18.296 7f447e59c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f44801a8850 0x7f44801a8c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:18.299 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:18.296 7f447e59c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f44801a8850 0x7f44801a8c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:60344/0 (socket says 172.21.2.102:60344) 2021-02-10T11:29:18.299 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:18.296 7f447e59c700 1 -- 172.21.2.102:0/4280717039 learned_addr learned my addr 172.21.2.102:0/4280717039 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:29:18.299 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:18.296 7f447e59c700 1 -- 172.21.2.102:0/4280717039 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f44800b0ad0 msgr2=0x7f4480091f30 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:29:18.299 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:18.296 7f447e59c700 1 --2- 172.21.2.102:0/4280717039 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f44800b0ad0 0x7f4480091f30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:18.299 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:18.296 7f447e59c700 1 -- 172.21.2.102:0/4280717039 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f44801a6fa0 msgr2=0x7f44801a73a0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:29:18.299 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:18.296 7f447e59c700 1 --2- 172.21.2.102:0/4280717039 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f44801a6fa0 0x7f44801a73a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:18.299 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:18.296 7f447e59c700 1 -- 172.21.2.102:0/4280717039 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44801a9270 con 0x7f44801a8850 2021-02-10T11:29:18.299 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:18.297 7f447e59c700 1 --2- 172.21.2.102:0/4280717039 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f44801a8850 0x7f44801a8c50 secure :-1 s=READY pgs=1871 cs=0 l=1 rev1=1 rx=0x7f4470001c10 tx=0x7f447000a8f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:29:18.299 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:18.297 7f44777fe700 1 -- 172.21.2.102:0/4280717039 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f44700009b0 con 0x7f44801a8850 2021-02-10T11:29:18.299 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:18.297 7f44777fe700 1 -- 172.21.2.102:0/4280717039 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44700012c0 con 0x7f44801a8850 2021-02-10T11:29:18.299 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:18.297 7f44777fe700 1 -- 172.21.2.102:0/4280717039 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4470004350 con 0x7f44801a8850 2021-02-10T11:29:18.300 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:18.297 7f448512b700 1 -- 172.21.2.102:0/4280717039 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44801a9270 con 0x7f44801a8850 2021-02-10T11:29:18.300 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:18.297 7f448512b700 1 -- 172.21.2.102:0/4280717039 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44801a7cb0 con 0x7f44801a8850 2021-02-10T11:29:18.300 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:18.298 7f44777fe700 1 -- 172.21.2.102:0/4280717039 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4470004140 con 0x7f44801a8850 2021-02-10T11:29:18.300 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:18.298 7f44777fe700 1 --2- 172.21.2.102:0/4280717039 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f44640186c0 0x7f446401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:18.300 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:18.298 7f44777fe700 1 -- 172.21.2.102:0/4280717039 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f44700323d0 con 0x7f44801a8850 2021-02-10T11:29:18.300 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:18.298 7f447ed9d700 1 --2- 172.21.2.102:0/4280717039 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f44640186c0 0x7f446401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:18.300 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:18.298 7f447ed9d700 1 --2- 172.21.2.102:0/4280717039 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f44640186c0 0x7f446401abc0 crc :-1 s=READY pgs=839 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:29:18.329 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:18.330 7f448512b700 1 -- 172.21.2.102:0/4280717039 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44801abfe0 con 0x7f44801a8850 2021-02-10T11:29:18.331 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:18.332 7f44777fe700 1 -- 172.21.2.102:0/4280717039 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f4470030d00 con 0x7f44801a8850 2021-02-10T11:29:18.461 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:18.461 7f448512b700 1 -- 172.21.2.102:0/4280717039 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f44801abfe0 con 0x7f44801a8850 2021-02-10T11:29:18.461 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:18.461 7f44777fe700 1 -- 172.21.2.102:0/4280717039 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+2101 (secure 0 0 0) 0x7f4470004140 con 0x7f44801a8850 2021-02-10T11:29:18.462 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:29:18.462 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":66,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6110,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.102:6835/1471871278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6834","nonce":1471871278},{"type":"v1","addr":"172.21.2.102:6835","nonce":1471871278}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":61}],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:27:41.479375","modified":"2021-02-10 11:27:43.490605","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6107},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6107":{"gid":6107,"name":"b","rank":0,"incarnation":62,"state":"up:active","state_seq":3,"addr":"172.21.2.113:6833/4220238062","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6832","nonce":4220238062},{"type":"v1","addr":"172.21.2.113:6833","nonce":4220238062}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":8}]} 2021-02-10T11:29:18.462 INFO:teuthology.orchestra.run.gibba002.stderr:dumped fsmap epoch 66 2021-02-10T11:29:18.463 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:18.463 7f448512b700 1 -- 172.21.2.102:0/4280717039 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f44640186c0 msgr2=0x7f446401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:18.463 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:18.463 7f448512b700 1 --2- 172.21.2.102:0/4280717039 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f44640186c0 0x7f446401abc0 crc :-1 s=READY pgs=839 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:18.463 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:18.463 7f448512b700 1 -- 172.21.2.102:0/4280717039 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f44801a8850 msgr2=0x7f44801a8c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:18.463 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:18.463 7f448512b700 1 --2- 172.21.2.102:0/4280717039 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f44801a8850 0x7f44801a8c50 secure :-1 s=READY pgs=1871 cs=0 l=1 rev1=1 rx=0x7f4470001c10 tx=0x7f447000a8f0).stop 2021-02-10T11:29:18.463 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:18.464 7f448512b700 1 -- 172.21.2.102:0/4280717039 shutdown_connections 2021-02-10T11:29:18.464 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:18.464 7f448512b700 1 --2- 172.21.2.102:0/4280717039 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f44640186c0 0x7f446401abc0 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:18.464 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:18.464 7f448512b700 1 --2- 172.21.2.102:0/4280717039 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f44801a6fa0 0x7f44801a73a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:18.464 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:18.464 7f448512b700 1 --2- 172.21.2.102:0/4280717039 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f44801a8850 0x7f44801a8c50 unknown :-1 s=CLOSED pgs=1871 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:18.464 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:18.464 7f448512b700 1 --2- 172.21.2.102:0/4280717039 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f44800b0ad0 0x7f4480091f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:18.464 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:18.464 7f448512b700 1 -- 172.21.2.102:0/4280717039 >> 172.21.2.102:0/4280717039 conn(0x7f448019f8f0 msgr2=0x7f44801a5f80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:29:18.464 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:18.464 7f448512b700 1 -- 172.21.2.102:0/4280717039 shutdown_connections 2021-02-10T11:29:18.464 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:18.464 7f448512b700 1 -- 172.21.2.102:0/4280717039 wait complete. 2021-02-10T11:29:18.472 DEBUG:teuthology.orchestra.run.gibba013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok session ls 2021-02-10T11:29:18.867 INFO:teuthology.orchestra.run.gibba013.stdout:[ 2021-02-10T11:29:18.868 INFO:teuthology.orchestra.run.gibba013.stdout: { 2021-02-10T11:29:18.868 INFO:teuthology.orchestra.run.gibba013.stdout: "id": 9471, 2021-02-10T11:29:18.868 INFO:teuthology.orchestra.run.gibba013.stdout: "entity": { 2021-02-10T11:29:18.868 INFO:teuthology.orchestra.run.gibba013.stdout: "name": { 2021-02-10T11:29:18.868 INFO:teuthology.orchestra.run.gibba013.stdout: "type": "client", 2021-02-10T11:29:18.868 INFO:teuthology.orchestra.run.gibba013.stdout: "num": 9471 2021-02-10T11:29:18.868 INFO:teuthology.orchestra.run.gibba013.stdout: }, 2021-02-10T11:29:18.868 INFO:teuthology.orchestra.run.gibba013.stdout: "addr": { 2021-02-10T11:29:18.868 INFO:teuthology.orchestra.run.gibba013.stdout: "type": "any", 2021-02-10T11:29:18.868 INFO:teuthology.orchestra.run.gibba013.stdout: "addr": "172.21.2.102:0", 2021-02-10T11:29:18.868 INFO:teuthology.orchestra.run.gibba013.stdout: "nonce": 190431005 2021-02-10T11:29:18.868 INFO:teuthology.orchestra.run.gibba013.stdout: } 2021-02-10T11:29:18.869 INFO:teuthology.orchestra.run.gibba013.stdout: }, 2021-02-10T11:29:18.869 INFO:teuthology.orchestra.run.gibba013.stdout: "state": "open", 2021-02-10T11:29:18.869 INFO:teuthology.orchestra.run.gibba013.stdout: "num_leases": 0, 2021-02-10T11:29:18.869 INFO:teuthology.orchestra.run.gibba013.stdout: "num_caps": 7, 2021-02-10T11:29:18.869 INFO:teuthology.orchestra.run.gibba013.stdout: "request_load_avg": 6, 2021-02-10T11:29:18.869 INFO:teuthology.orchestra.run.gibba013.stdout: "uptime": 68.190002380999999, 2021-02-10T11:29:18.869 INFO:teuthology.orchestra.run.gibba013.stdout: "requests_in_flight": 0, 2021-02-10T11:29:18.869 INFO:teuthology.orchestra.run.gibba013.stdout: "completed_requests": 1, 2021-02-10T11:29:18.869 INFO:teuthology.orchestra.run.gibba013.stdout: "reconnecting": false, 2021-02-10T11:29:18.869 INFO:teuthology.orchestra.run.gibba013.stdout: "recall_caps": { 2021-02-10T11:29:18.869 INFO:teuthology.orchestra.run.gibba013.stdout: "value": 0, 2021-02-10T11:29:18.869 INFO:teuthology.orchestra.run.gibba013.stdout: "halflife": 60 2021-02-10T11:29:18.869 INFO:teuthology.orchestra.run.gibba013.stdout: }, 2021-02-10T11:29:18.870 INFO:teuthology.orchestra.run.gibba013.stdout: "release_caps": { 2021-02-10T11:29:18.870 INFO:teuthology.orchestra.run.gibba013.stdout: "value": 0, 2021-02-10T11:29:18.870 INFO:teuthology.orchestra.run.gibba013.stdout: "halflife": 60 2021-02-10T11:29:18.870 INFO:teuthology.orchestra.run.gibba013.stdout: }, 2021-02-10T11:29:18.870 INFO:teuthology.orchestra.run.gibba013.stdout: "recall_caps_throttle": { 2021-02-10T11:29:18.870 INFO:teuthology.orchestra.run.gibba013.stdout: "value": 0, 2021-02-10T11:29:18.870 INFO:teuthology.orchestra.run.gibba013.stdout: "halflife": 2.5 2021-02-10T11:29:18.870 INFO:teuthology.orchestra.run.gibba013.stdout: }, 2021-02-10T11:29:18.870 INFO:teuthology.orchestra.run.gibba013.stdout: "recall_caps_throttle2o": { 2021-02-10T11:29:18.870 INFO:teuthology.orchestra.run.gibba013.stdout: "value": 0, 2021-02-10T11:29:18.870 INFO:teuthology.orchestra.run.gibba013.stdout: "halflife": 0.5 2021-02-10T11:29:18.870 INFO:teuthology.orchestra.run.gibba013.stdout: }, 2021-02-10T11:29:18.870 INFO:teuthology.orchestra.run.gibba013.stdout: "session_cache_liveness": { 2021-02-10T11:29:18.870 INFO:teuthology.orchestra.run.gibba013.stdout: "value": 17.951216882831538, 2021-02-10T11:29:18.871 INFO:teuthology.orchestra.run.gibba013.stdout: "halflife": 300 2021-02-10T11:29:18.871 INFO:teuthology.orchestra.run.gibba013.stdout: }, 2021-02-10T11:29:18.871 INFO:teuthology.orchestra.run.gibba013.stdout: "cap_acquisition": { 2021-02-10T11:29:18.871 INFO:teuthology.orchestra.run.gibba013.stdout: "value": 0, 2021-02-10T11:29:18.871 INFO:teuthology.orchestra.run.gibba013.stdout: "halflife": 10 2021-02-10T11:29:18.871 INFO:teuthology.orchestra.run.gibba013.stdout: }, 2021-02-10T11:29:18.871 INFO:teuthology.orchestra.run.gibba013.stdout: "inst": "client.9471 172.21.2.102:0/190431005", 2021-02-10T11:29:18.871 INFO:teuthology.orchestra.run.gibba013.stdout: "completed_requests": [ 2021-02-10T11:29:18.871 INFO:teuthology.orchestra.run.gibba013.stdout: { 2021-02-10T11:29:18.871 INFO:teuthology.orchestra.run.gibba013.stdout: "tid": 14, 2021-02-10T11:29:18.871 INFO:teuthology.orchestra.run.gibba013.stdout: "created_ino": "0x10000000005" 2021-02-10T11:29:18.871 INFO:teuthology.orchestra.run.gibba013.stdout: } 2021-02-10T11:29:18.871 INFO:teuthology.orchestra.run.gibba013.stdout: ], 2021-02-10T11:29:18.872 INFO:teuthology.orchestra.run.gibba013.stdout: "prealloc_inos": [ 2021-02-10T11:29:18.872 INFO:teuthology.orchestra.run.gibba013.stdout: { 2021-02-10T11:29:18.872 INFO:teuthology.orchestra.run.gibba013.stdout: "start": 1099511627782, 2021-02-10T11:29:18.872 INFO:teuthology.orchestra.run.gibba013.stdout: "length": 995 2021-02-10T11:29:18.872 INFO:teuthology.orchestra.run.gibba013.stdout: } 2021-02-10T11:29:18.872 INFO:teuthology.orchestra.run.gibba013.stdout: ], 2021-02-10T11:29:18.872 INFO:teuthology.orchestra.run.gibba013.stdout: "used_inos": [], 2021-02-10T11:29:18.872 INFO:teuthology.orchestra.run.gibba013.stdout: "client_metadata": { 2021-02-10T11:29:18.872 INFO:teuthology.orchestra.run.gibba013.stdout: "features": "0x0000000000001fff", 2021-02-10T11:29:18.872 INFO:teuthology.orchestra.run.gibba013.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:29:18.872 INFO:teuthology.orchestra.run.gibba013.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:29:18.872 INFO:teuthology.orchestra.run.gibba013.stdout: "entity_id": "y", 2021-02-10T11:29:18.872 INFO:teuthology.orchestra.run.gibba013.stdout: "hostname": "gibba002", 2021-02-10T11:29:18.873 INFO:teuthology.orchestra.run.gibba013.stdout: "pid": "35161", 2021-02-10T11:29:18.873 INFO:teuthology.orchestra.run.gibba013.stdout: "root": "/" 2021-02-10T11:29:18.873 INFO:teuthology.orchestra.run.gibba013.stdout: } 2021-02-10T11:29:18.873 INFO:teuthology.orchestra.run.gibba013.stdout: } 2021-02-10T11:29:18.873 INFO:teuthology.orchestra.run.gibba013.stdout:] 2021-02-10T11:29:18.874 INFO:tasks.cephfs.filesystem:_json_asok output: [ { "id": 9471, "entity": { "name": { "type": "client", "num": 9471 }, "addr": { "type": "any", "addr": "172.21.2.102:0", "nonce": 190431005 } }, "state": "open", "num_leases": 0, "num_caps": 7, "request_load_avg": 6, "uptime": 68.190002380999999, "requests_in_flight": 0, "completed_requests": 1, "reconnecting": false, "recall_caps": { "value": 0, "halflife": 60 }, "release_caps": { "value": 0, "halflife": 60 }, "recall_caps_throttle": { "value": 0, "halflife": 2.5 }, "recall_caps_throttle2o": { "value": 0, "halflife": 0.5 }, "session_cache_liveness": { "value": 17.951216882831538, "halflife": 300 }, "cap_acquisition": { "value": 0, "halflife": 10 }, "inst": "client.9471 172.21.2.102:0/190431005", "completed_requests": [ { "tid": 14, "created_ino": "0x10000000005" } ], "prealloc_inos": [ { "start": 1099511627782, "length": 995 } ], "used_inos": [], "client_metadata": { "features": "0x0000000000001fff", "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", "entity_id": "y", "hostname": "gibba002", "pid": "35161", "root": "/" } } ] 2021-02-10T11:29:18.874 DEBUG:tasks.ceph_test_case:wait_until_true: waiting... 2021-02-10T11:29:23.875 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:29:24.303 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:24.288 7f9d9f216700 1 Processor -- start 2021-02-10T11:29:24.304 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:24.288 7f9d9f216700 1 -- start start 2021-02-10T11:29:24.304 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:24.289 7f9d9f216700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f9d980d1870 0x7f9d981b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:24.304 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:24.289 7f9d9f216700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9d981b0a10 0x7f9d980cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:24.304 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:24.289 7f9d9f216700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f9d980d1c70 0x7f9d980cf950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:24.304 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:24.289 7f9d9f216700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d980fe200 con 0x7f9d980d1870 2021-02-10T11:29:24.304 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:24.289 7f9d9f216700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d980e5f10 con 0x7f9d980d1c70 2021-02-10T11:29:24.304 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:24.289 7f9d9f216700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d980f0b50 con 0x7f9d981b0a10 2021-02-10T11:29:24.304 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:24.289 7f9d9d032700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f9d980d1870 0x7f9d981b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:24.304 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:24.289 7f9d9d032700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f9d980d1870 0x7f9d981b03f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:60350/0 (socket says 172.21.2.102:60350) 2021-02-10T11:29:24.304 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:24.289 7f9d9d032700 1 -- 172.21.2.102:0/3172347355 learned_addr learned my addr 172.21.2.102:0/3172347355 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:29:24.305 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:24.289 7f9d9d032700 1 -- 172.21.2.102:0/3172347355 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9d981b0a10 msgr2=0x7f9d980cf330 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:29:24.305 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:24.289 7f9d9d032700 1 --2- 172.21.2.102:0/3172347355 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9d981b0a10 0x7f9d980cf330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:24.305 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:24.289 7f9d9d032700 1 -- 172.21.2.102:0/3172347355 >> v2:172.21.2.113:3300/0 conn(0x7f9d980d1c70 msgr2=0x7f9d980cf950 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:29:24.305 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:24.289 7f9d9d032700 1 --2- 172.21.2.102:0/3172347355 >> v2:172.21.2.113:3300/0 conn(0x7f9d980d1c70 0x7f9d980cf950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:24.305 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:24.289 7f9d9d032700 1 -- 172.21.2.102:0/3172347355 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d980cff70 con 0x7f9d980d1870 2021-02-10T11:29:24.305 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:24.289 7f9d9d032700 1 --2- 172.21.2.102:0/3172347355 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9d980d1870 0x7f9d981b03f0 secure :-1 s=READY pgs=1872 cs=0 l=1 rev1=1 rx=0x7f9d8c006700 tx=0x7f9d8c009b00).ready entity=mon.0 client_cookie=1130a2e45b6a56d6 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:29:24.305 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:24.289 7f9d8bfff700 1 -- 172.21.2.102:0/3172347355 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9d8c00c0c0 con 0x7f9d980d1870 2021-02-10T11:29:24.305 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:24.289 7f9d8bfff700 1 -- 172.21.2.102:0/3172347355 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d8c00c340 con 0x7f9d980d1870 2021-02-10T11:29:24.305 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:24.289 7f9d8bfff700 1 -- 172.21.2.102:0/3172347355 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9d8c00c820 con 0x7f9d980d1870 2021-02-10T11:29:24.305 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:24.289 7f9d9f216700 1 -- 172.21.2.102:0/3172347355 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9d980d1870 msgr2=0x7f9d981b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:24.305 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:24.289 7f9d9f216700 1 --2- 172.21.2.102:0/3172347355 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9d980d1870 0x7f9d981b03f0 secure :-1 s=READY pgs=1872 cs=0 l=1 rev1=1 rx=0x7f9d8c006700 tx=0x7f9d8c009b00).stop 2021-02-10T11:29:24.305 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:24.290 7f9d9f216700 1 -- 172.21.2.102:0/3172347355 shutdown_connections 2021-02-10T11:29:24.305 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:24.290 7f9d9f216700 1 --2- 172.21.2.102:0/3172347355 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9d981b0a10 0x7f9d980cf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:24.306 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:24.290 7f9d9f216700 1 --2- 172.21.2.102:0/3172347355 >> v2:172.21.2.113:3300/0 conn(0x7f9d980d1c70 0x7f9d980cf950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:24.306 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:24.290 7f9d9f216700 1 --2- 172.21.2.102:0/3172347355 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9d980d1870 0x7f9d981b03f0 unknown :-1 s=CLOSED pgs=1872 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:24.306 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:24.290 7f9d9f216700 1 -- 172.21.2.102:0/3172347355 >> 172.21.2.102:0/3172347355 conn(0x7f9d980ce450 msgr2=0x7f9d980d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:29:24.306 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:24.290 7f9d9f216700 1 -- 172.21.2.102:0/3172347355 shutdown_connections 2021-02-10T11:29:24.306 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:24.290 7f9d9f216700 1 -- 172.21.2.102:0/3172347355 wait complete. 2021-02-10T11:29:24.307 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:24.305 7f9d9f216700 1 Processor -- start 2021-02-10T11:29:24.307 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:24.305 7f9d9f216700 1 -- start start 2021-02-10T11:29:24.307 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:24.305 7f9d9f216700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f9d9818e930 0x7f9d980aa4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:24.307 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:24.305 7f9d9f216700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9d980aaaf0 0x7f9d98093700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:24.308 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:24.305 7f9d9f216700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9d9818ed30 0x7f9d98093d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:24.308 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:24.305 7f9d9f216700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d980944e0 con 0x7f9d980aaaf0 2021-02-10T11:29:24.308 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:24.305 7f9d9f216700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d9808a590 con 0x7f9d9818e930 2021-02-10T11:29:24.308 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:24.305 7f9d9f216700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d9808a770 con 0x7f9d9818ed30 2021-02-10T11:29:24.308 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:24.305 7f9d9c831700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9d980aaaf0 0x7f9d98093700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:24.308 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:24.305 7f9d9c831700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9d980aaaf0 0x7f9d98093700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:60354/0 (socket says 172.21.2.102:60354) 2021-02-10T11:29:24.308 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:24.305 7f9d9c831700 1 -- 172.21.2.102:0/3674048045 learned_addr learned my addr 172.21.2.102:0/3674048045 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:29:24.308 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:24.306 7f9d9c831700 1 -- 172.21.2.102:0/3674048045 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9d9818ed30 msgr2=0x7f9d98093d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:24.308 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:24.306 7f9d9c831700 1 --2- 172.21.2.102:0/3674048045 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9d9818ed30 0x7f9d98093d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:24.308 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:24.306 7f9d9c831700 1 -- 172.21.2.102:0/3674048045 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f9d9818e930 msgr2=0x7f9d980aa4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:24.308 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:24.306 7f9d9c831700 1 --2- 172.21.2.102:0/3674048045 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f9d9818e930 0x7f9d980aa4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:24.309 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:24.306 7f9d9c831700 1 -- 172.21.2.102:0/3674048045 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d9808a990 con 0x7f9d980aaaf0 2021-02-10T11:29:24.309 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:24.306 7f9d9c831700 1 --2- 172.21.2.102:0/3674048045 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9d980aaaf0 0x7f9d98093700 secure :-1 s=READY pgs=1873 cs=0 l=1 rev1=1 rx=0x7f9d9400b5c0 tx=0x7f9d9400b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:29:24.309 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:24.306 7f9d89ffb700 1 -- 172.21.2.102:0/3674048045 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9d9400bd50 con 0x7f9d980aaaf0 2021-02-10T11:29:24.309 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:24.306 7f9d89ffb700 1 -- 172.21.2.102:0/3674048045 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d94006030 con 0x7f9d980aaaf0 2021-02-10T11:29:24.309 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:24.306 7f9d89ffb700 1 -- 172.21.2.102:0/3674048045 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9d940054d0 con 0x7f9d980aaaf0 2021-02-10T11:29:24.309 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:24.306 7f9d9f216700 1 -- 172.21.2.102:0/3674048045 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d9808a990 con 0x7f9d980aaaf0 2021-02-10T11:29:24.310 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:24.306 7f9d9f216700 1 -- 172.21.2.102:0/3674048045 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d9808a990 con 0x7f9d980aaaf0 2021-02-10T11:29:24.310 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:24.307 7f9d89ffb700 1 -- 172.21.2.102:0/3674048045 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f9d94006210 con 0x7f9d980aaaf0 2021-02-10T11:29:24.310 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:24.307 7f9d89ffb700 1 --2- 172.21.2.102:0/3674048045 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f9d780186c0 0x7f9d7801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:24.310 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:24.307 7f9d89ffb700 1 -- 172.21.2.102:0/3674048045 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f9d940323f0 con 0x7f9d980aaaf0 2021-02-10T11:29:24.310 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:24.307 7f9d9d032700 1 --2- 172.21.2.102:0/3674048045 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f9d780186c0 0x7f9d7801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:24.310 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:24.307 7f9d9d032700 1 --2- 172.21.2.102:0/3674048045 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f9d780186c0 0x7f9d7801abc0 crc :-1 s=READY pgs=840 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:29:24.338 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:24.339 7f9d9f216700 1 -- 172.21.2.102:0/3674048045 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d9808a530 con 0x7f9d980aaaf0 2021-02-10T11:29:24.340 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:24.341 7f9d89ffb700 1 -- 172.21.2.102:0/3674048045 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f9d9402ded0 con 0x7f9d980aaaf0 2021-02-10T11:29:24.469 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:24.470 7f9d9f216700 1 -- 172.21.2.102:0/3674048045 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9d9808a530 con 0x7f9d980aaaf0 2021-02-10T11:29:24.469 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:24.470 7f9d89ffb700 1 -- 172.21.2.102:0/3674048045 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+2101 (secure 0 0 0) 0x7f9d94000c10 con 0x7f9d980aaaf0 2021-02-10T11:29:24.470 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:29:24.470 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":66,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6110,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.102:6835/1471871278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6834","nonce":1471871278},{"type":"v1","addr":"172.21.2.102:6835","nonce":1471871278}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":61}],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:27:41.479375","modified":"2021-02-10 11:27:43.490605","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6107},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6107":{"gid":6107,"name":"b","rank":0,"incarnation":62,"state":"up:active","state_seq":3,"addr":"172.21.2.113:6833/4220238062","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6832","nonce":4220238062},{"type":"v1","addr":"172.21.2.113:6833","nonce":4220238062}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":8}]} 2021-02-10T11:29:24.470 INFO:teuthology.orchestra.run.gibba002.stderr:dumped fsmap epoch 66 2021-02-10T11:29:24.471 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:24.472 7f9d9f216700 1 -- 172.21.2.102:0/3674048045 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f9d780186c0 msgr2=0x7f9d7801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:24.471 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:24.472 7f9d9f216700 1 --2- 172.21.2.102:0/3674048045 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f9d780186c0 0x7f9d7801abc0 crc :-1 s=READY pgs=840 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:24.471 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:24.472 7f9d9f216700 1 -- 172.21.2.102:0/3674048045 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9d980aaaf0 msgr2=0x7f9d98093700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:24.471 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:24.472 7f9d9f216700 1 --2- 172.21.2.102:0/3674048045 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9d980aaaf0 0x7f9d98093700 secure :-1 s=READY pgs=1873 cs=0 l=1 rev1=1 rx=0x7f9d9400b5c0 tx=0x7f9d9400b960).stop 2021-02-10T11:29:24.472 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:24.472 7f9d9f216700 1 -- 172.21.2.102:0/3674048045 shutdown_connections 2021-02-10T11:29:24.472 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:24.472 7f9d9f216700 1 --2- 172.21.2.102:0/3674048045 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9d9818ed30 0x7f9d98093d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:24.472 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:24.472 7f9d9f216700 1 --2- 172.21.2.102:0/3674048045 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9d980aaaf0 0x7f9d98093700 unknown :-1 s=CLOSED pgs=1873 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:24.472 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:24.472 7f9d9f216700 1 --2- 172.21.2.102:0/3674048045 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f9d780186c0 0x7f9d7801abc0 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:24.472 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:24.472 7f9d9f216700 1 --2- 172.21.2.102:0/3674048045 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f9d9818e930 0x7f9d980aa4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:24.472 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:24.472 7f9d9f216700 1 -- 172.21.2.102:0/3674048045 >> 172.21.2.102:0/3674048045 conn(0x7f9d9819f8f0 msgr2=0x7f9d9818c260 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:29:24.472 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:24.472 7f9d9f216700 1 -- 172.21.2.102:0/3674048045 shutdown_connections 2021-02-10T11:29:24.472 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:24.472 7f9d9f216700 1 -- 172.21.2.102:0/3674048045 wait complete. 2021-02-10T11:29:24.480 DEBUG:teuthology.orchestra.run.gibba013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok session ls 2021-02-10T11:29:24.877 INFO:teuthology.orchestra.run.gibba013.stdout:[ 2021-02-10T11:29:24.878 INFO:teuthology.orchestra.run.gibba013.stdout: { 2021-02-10T11:29:24.878 INFO:teuthology.orchestra.run.gibba013.stdout: "id": 9471, 2021-02-10T11:29:24.878 INFO:teuthology.orchestra.run.gibba013.stdout: "entity": { 2021-02-10T11:29:24.878 INFO:teuthology.orchestra.run.gibba013.stdout: "name": { 2021-02-10T11:29:24.878 INFO:teuthology.orchestra.run.gibba013.stdout: "type": "client", 2021-02-10T11:29:24.878 INFO:teuthology.orchestra.run.gibba013.stdout: "num": 9471 2021-02-10T11:29:24.878 INFO:teuthology.orchestra.run.gibba013.stdout: }, 2021-02-10T11:29:24.878 INFO:teuthology.orchestra.run.gibba013.stdout: "addr": { 2021-02-10T11:29:24.878 INFO:teuthology.orchestra.run.gibba013.stdout: "type": "any", 2021-02-10T11:29:24.878 INFO:teuthology.orchestra.run.gibba013.stdout: "addr": "172.21.2.102:0", 2021-02-10T11:29:24.878 INFO:teuthology.orchestra.run.gibba013.stdout: "nonce": 190431005 2021-02-10T11:29:24.879 INFO:teuthology.orchestra.run.gibba013.stdout: } 2021-02-10T11:29:24.879 INFO:teuthology.orchestra.run.gibba013.stdout: }, 2021-02-10T11:29:24.879 INFO:teuthology.orchestra.run.gibba013.stdout: "state": "open", 2021-02-10T11:29:24.879 INFO:teuthology.orchestra.run.gibba013.stdout: "num_leases": 0, 2021-02-10T11:29:24.879 INFO:teuthology.orchestra.run.gibba013.stdout: "num_caps": 7, 2021-02-10T11:29:24.879 INFO:teuthology.orchestra.run.gibba013.stdout: "request_load_avg": 5, 2021-02-10T11:29:24.879 INFO:teuthology.orchestra.run.gibba013.stdout: "uptime": 74.200002495000007, 2021-02-10T11:29:24.879 INFO:teuthology.orchestra.run.gibba013.stdout: "requests_in_flight": 0, 2021-02-10T11:29:24.879 INFO:teuthology.orchestra.run.gibba013.stdout: "completed_requests": 1, 2021-02-10T11:29:24.879 INFO:teuthology.orchestra.run.gibba013.stdout: "reconnecting": false, 2021-02-10T11:29:24.879 INFO:teuthology.orchestra.run.gibba013.stdout: "recall_caps": { 2021-02-10T11:29:24.879 INFO:teuthology.orchestra.run.gibba013.stdout: "value": 0, 2021-02-10T11:29:24.879 INFO:teuthology.orchestra.run.gibba013.stdout: "halflife": 60 2021-02-10T11:29:24.880 INFO:teuthology.orchestra.run.gibba013.stdout: }, 2021-02-10T11:29:24.880 INFO:teuthology.orchestra.run.gibba013.stdout: "release_caps": { 2021-02-10T11:29:24.880 INFO:teuthology.orchestra.run.gibba013.stdout: "value": 0, 2021-02-10T11:29:24.880 INFO:teuthology.orchestra.run.gibba013.stdout: "halflife": 60 2021-02-10T11:29:24.880 INFO:teuthology.orchestra.run.gibba013.stdout: }, 2021-02-10T11:29:24.880 INFO:teuthology.orchestra.run.gibba013.stdout: "recall_caps_throttle": { 2021-02-10T11:29:24.880 INFO:teuthology.orchestra.run.gibba013.stdout: "value": 0, 2021-02-10T11:29:24.880 INFO:teuthology.orchestra.run.gibba013.stdout: "halflife": 2.5 2021-02-10T11:29:24.880 INFO:teuthology.orchestra.run.gibba013.stdout: }, 2021-02-10T11:29:24.880 INFO:teuthology.orchestra.run.gibba013.stdout: "recall_caps_throttle2o": { 2021-02-10T11:29:24.880 INFO:teuthology.orchestra.run.gibba013.stdout: "value": 0, 2021-02-10T11:29:24.880 INFO:teuthology.orchestra.run.gibba013.stdout: "halflife": 0.5 2021-02-10T11:29:24.880 INFO:teuthology.orchestra.run.gibba013.stdout: }, 2021-02-10T11:29:24.880 INFO:teuthology.orchestra.run.gibba013.stdout: "session_cache_liveness": { 2021-02-10T11:29:24.881 INFO:teuthology.orchestra.run.gibba013.stdout: "value": 17.703668124324942, 2021-02-10T11:29:24.881 INFO:teuthology.orchestra.run.gibba013.stdout: "halflife": 300 2021-02-10T11:29:24.881 INFO:teuthology.orchestra.run.gibba013.stdout: }, 2021-02-10T11:29:24.881 INFO:teuthology.orchestra.run.gibba013.stdout: "cap_acquisition": { 2021-02-10T11:29:24.881 INFO:teuthology.orchestra.run.gibba013.stdout: "value": 0, 2021-02-10T11:29:24.881 INFO:teuthology.orchestra.run.gibba013.stdout: "halflife": 10 2021-02-10T11:29:24.881 INFO:teuthology.orchestra.run.gibba013.stdout: }, 2021-02-10T11:29:24.881 INFO:teuthology.orchestra.run.gibba013.stdout: "inst": "client.9471 172.21.2.102:0/190431005", 2021-02-10T11:29:24.881 INFO:teuthology.orchestra.run.gibba013.stdout: "completed_requests": [ 2021-02-10T11:29:24.881 INFO:teuthology.orchestra.run.gibba013.stdout: { 2021-02-10T11:29:24.881 INFO:teuthology.orchestra.run.gibba013.stdout: "tid": 14, 2021-02-10T11:29:24.881 INFO:teuthology.orchestra.run.gibba013.stdout: "created_ino": "0x10000000005" 2021-02-10T11:29:24.881 INFO:teuthology.orchestra.run.gibba013.stdout: } 2021-02-10T11:29:24.882 INFO:teuthology.orchestra.run.gibba013.stdout: ], 2021-02-10T11:29:24.882 INFO:teuthology.orchestra.run.gibba013.stdout: "prealloc_inos": [ 2021-02-10T11:29:24.882 INFO:teuthology.orchestra.run.gibba013.stdout: { 2021-02-10T11:29:24.882 INFO:teuthology.orchestra.run.gibba013.stdout: "start": 1099511627782, 2021-02-10T11:29:24.882 INFO:teuthology.orchestra.run.gibba013.stdout: "length": 995 2021-02-10T11:29:24.882 INFO:teuthology.orchestra.run.gibba013.stdout: } 2021-02-10T11:29:24.882 INFO:teuthology.orchestra.run.gibba013.stdout: ], 2021-02-10T11:29:24.882 INFO:teuthology.orchestra.run.gibba013.stdout: "used_inos": [], 2021-02-10T11:29:24.882 INFO:teuthology.orchestra.run.gibba013.stdout: "client_metadata": { 2021-02-10T11:29:24.882 INFO:teuthology.orchestra.run.gibba013.stdout: "features": "0x0000000000001fff", 2021-02-10T11:29:24.882 INFO:teuthology.orchestra.run.gibba013.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:29:24.882 INFO:teuthology.orchestra.run.gibba013.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:29:24.882 INFO:teuthology.orchestra.run.gibba013.stdout: "entity_id": "y", 2021-02-10T11:29:24.883 INFO:teuthology.orchestra.run.gibba013.stdout: "hostname": "gibba002", 2021-02-10T11:29:24.883 INFO:teuthology.orchestra.run.gibba013.stdout: "pid": "35161", 2021-02-10T11:29:24.883 INFO:teuthology.orchestra.run.gibba013.stdout: "root": "/" 2021-02-10T11:29:24.883 INFO:teuthology.orchestra.run.gibba013.stdout: } 2021-02-10T11:29:24.883 INFO:teuthology.orchestra.run.gibba013.stdout: } 2021-02-10T11:29:24.883 INFO:teuthology.orchestra.run.gibba013.stdout:] 2021-02-10T11:29:24.884 INFO:tasks.cephfs.filesystem:_json_asok output: [ { "id": 9471, "entity": { "name": { "type": "client", "num": 9471 }, "addr": { "type": "any", "addr": "172.21.2.102:0", "nonce": 190431005 } }, "state": "open", "num_leases": 0, "num_caps": 7, "request_load_avg": 5, "uptime": 74.200002495000007, "requests_in_flight": 0, "completed_requests": 1, "reconnecting": false, "recall_caps": { "value": 0, "halflife": 60 }, "release_caps": { "value": 0, "halflife": 60 }, "recall_caps_throttle": { "value": 0, "halflife": 2.5 }, "recall_caps_throttle2o": { "value": 0, "halflife": 0.5 }, "session_cache_liveness": { "value": 17.703668124324942, "halflife": 300 }, "cap_acquisition": { "value": 0, "halflife": 10 }, "inst": "client.9471 172.21.2.102:0/190431005", "completed_requests": [ { "tid": 14, "created_ino": "0x10000000005" } ], "prealloc_inos": [ { "start": 1099511627782, "length": 995 } ], "used_inos": [], "client_metadata": { "features": "0x0000000000001fff", "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", "entity_id": "y", "hostname": "gibba002", "pid": "35161", "root": "/" } } ] 2021-02-10T11:29:24.884 DEBUG:tasks.ceph_test_case:wait_until_true: waiting... 2021-02-10T11:29:29.885 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:29:30.314 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:30.297 7fad51a19700 1 Processor -- start 2021-02-10T11:29:30.314 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:30.297 7fad51a19700 1 -- start start 2021-02-10T11:29:30.314 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:30.298 7fad51a19700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fad4c0d0340 0x7fad4c0ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:30.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:30.298 7fad51a19700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fad4c0d1c70 0x7fad4c0efc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:30.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:30.298 7fad51a19700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad4c0f0b10 con 0x7fad4c0d1870 2021-02-10T11:29:30.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:30.298 7fad51a19700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad4c0e5ed0 con 0x7fad4c0d0340 2021-02-10T11:29:30.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:30.298 7fad51a19700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad4c1b0be0 con 0x7fad4c0d1c70 2021-02-10T11:29:30.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:30.298 7fad4affd700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fad4c0d0340 0x7fad4c0ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:30.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:30.298 7fad4affd700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fad4c0d0340 0x7fad4c0ef640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:60360/0 (socket says 172.21.2.102:60360) 2021-02-10T11:29:30.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:30.298 7fad4affd700 1 -- 172.21.2.102:0/525994280 learned_addr learned my addr 172.21.2.102:0/525994280 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:29:30.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:30.298 7fad4affd700 1 -- 172.21.2.102:0/525994280 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fad4c0d1c70 msgr2=0x7fad4c0efc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:29:30.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:30.298 7fad4affd700 1 --2- 172.21.2.102:0/525994280 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fad4c0d1c70 0x7fad4c0efc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:30.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:30.299 7fad4affd700 1 -- 172.21.2.102:0/525994280 >> v1:172.21.2.113:6789/0 conn(0x7fad4c0d1870 legacy=0x7fad4c1b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:29:30.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:30.299 7fad4affd700 1 -- 172.21.2.102:0/525994280 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad4c0f20a0 con 0x7fad4c0d0340 2021-02-10T11:29:30.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:30.299 7fad4affd700 1 --2- 172.21.2.102:0/525994280 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fad4c0d0340 0x7fad4c0ef640 secure :-1 s=READY pgs=1874 cs=0 l=1 rev1=1 rx=0x7fad40006700 tx=0x7fad40009b80).ready entity=mon.0 client_cookie=4500d67f60024b16 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:29:30.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:30.299 7fad4a7fc700 1 -- 172.21.2.102:0/525994280 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fad4000c0c0 con 0x7fad4c0d0340 2021-02-10T11:29:30.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:30.299 7fad4a7fc700 1 -- 172.21.2.102:0/525994280 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad4000c340 con 0x7fad4c0d0340 2021-02-10T11:29:30.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:30.299 7fad4a7fc700 1 -- 172.21.2.102:0/525994280 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fad4000c860 con 0x7fad4c0d0340 2021-02-10T11:29:30.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:30.299 7fad51a19700 1 -- 172.21.2.102:0/525994280 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fad4c0d0340 msgr2=0x7fad4c0ef640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:30.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:30.299 7fad51a19700 1 --2- 172.21.2.102:0/525994280 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fad4c0d0340 0x7fad4c0ef640 secure :-1 s=READY pgs=1874 cs=0 l=1 rev1=1 rx=0x7fad40006700 tx=0x7fad40009b80).stop 2021-02-10T11:29:30.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:30.300 7fad51a19700 1 -- 172.21.2.102:0/525994280 shutdown_connections 2021-02-10T11:29:30.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:30.300 7fad51a19700 1 --2- 172.21.2.102:0/525994280 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fad4c0d1c70 0x7fad4c0efc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:30.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:30.300 7fad51a19700 1 --2- 172.21.2.102:0/525994280 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fad4c0d0340 0x7fad4c0ef640 unknown :-1 s=CLOSED pgs=1874 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:30.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:30.300 7fad51a19700 1 -- 172.21.2.102:0/525994280 >> 172.21.2.102:0/525994280 conn(0x7fad4c0ce450 msgr2=0x7fad4c0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:29:30.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:30.300 7fad51a19700 1 -- 172.21.2.102:0/525994280 shutdown_connections 2021-02-10T11:29:30.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:30.300 7fad51a19700 1 -- 172.21.2.102:0/525994280 wait complete. 2021-02-10T11:29:30.318 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:30.314 7fad51a19700 1 Processor -- start 2021-02-10T11:29:30.318 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:30.314 7fad51a19700 1 -- start start 2021-02-10T11:29:30.318 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:30.315 7fad51a19700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fad4c0b4f30 0x7fad4c18b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:30.318 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:30.315 7fad51a19700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fad4c093f40 0x7fad4c094340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:30.318 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:30.315 7fad51a19700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fad4c094960 0x7fad4c187e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:30.318 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:30.315 7fad51a19700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad4c094e90 con 0x7fad4c094960 2021-02-10T11:29:30.318 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:30.315 7fad51a19700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad4c18bff0 con 0x7fad4c093f40 2021-02-10T11:29:30.318 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:30.315 7fad51a19700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad4c08f780 con 0x7fad4c0b4f30 2021-02-10T11:29:30.318 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:30.315 7fad4bfff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fad4c094960 0x7fad4c187e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:30.318 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:30.315 7fad4bfff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fad4c094960 0x7fad4c187e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:60366/0 (socket says 172.21.2.102:60366) 2021-02-10T11:29:30.319 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:30.315 7fad4bfff700 1 -- 172.21.2.102:0/1265459152 learned_addr learned my addr 172.21.2.102:0/1265459152 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:29:30.319 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:30.315 7fad4bfff700 1 -- 172.21.2.102:0/1265459152 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fad4c0b4f30 msgr2=0x7fad4c18b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:30.319 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:30.315 7fad4bfff700 1 --2- 172.21.2.102:0/1265459152 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fad4c0b4f30 0x7fad4c18b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:30.319 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:30.315 7fad4bfff700 1 -- 172.21.2.102:0/1265459152 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fad4c093f40 msgr2=0x7fad4c094340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:30.319 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:30.315 7fad4bfff700 1 --2- 172.21.2.102:0/1265459152 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fad4c093f40 0x7fad4c094340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:30.319 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:30.315 7fad4bfff700 1 -- 172.21.2.102:0/1265459152 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad4c188470 con 0x7fad4c094960 2021-02-10T11:29:30.319 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:30.316 7fad4bfff700 1 --2- 172.21.2.102:0/1265459152 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fad4c094960 0x7fad4c187e50 secure :-1 s=READY pgs=1875 cs=0 l=1 rev1=1 rx=0x7fad44009980 tx=0x7fad4400bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:29:30.319 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:30.316 7fad33fff700 1 -- 172.21.2.102:0/1265459152 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fad44005870 con 0x7fad4c094960 2021-02-10T11:29:30.319 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:30.316 7fad33fff700 1 -- 172.21.2.102:0/1265459152 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad44005af0 con 0x7fad4c094960 2021-02-10T11:29:30.319 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:30.316 7fad51a19700 1 -- 172.21.2.102:0/1265459152 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad4c188470 con 0x7fad4c094960 2021-02-10T11:29:30.319 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:30.316 7fad51a19700 1 -- 172.21.2.102:0/1265459152 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad4c188a00 con 0x7fad4c094960 2021-02-10T11:29:30.319 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:30.316 7fad33fff700 1 -- 172.21.2.102:0/1265459152 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fad44006030 con 0x7fad4c094960 2021-02-10T11:29:30.319 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:30.317 7fad33fff700 1 -- 172.21.2.102:0/1265459152 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fad44005cd0 con 0x7fad4c094960 2021-02-10T11:29:30.320 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:30.317 7fad33fff700 1 --2- 172.21.2.102:0/1265459152 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fad2c018730 0x7fad2c01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:30.320 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:30.317 7fad33fff700 1 -- 172.21.2.102:0/1265459152 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fad44032b70 con 0x7fad4c094960 2021-02-10T11:29:30.320 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:30.317 7fad4b7fe700 1 --2- 172.21.2.102:0/1265459152 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fad2c018730 0x7fad2c01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:30.320 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:30.317 7fad4b7fe700 1 --2- 172.21.2.102:0/1265459152 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fad2c018730 0x7fad2c01ac10 crc :-1 s=READY pgs=841 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:29:30.348 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:30.348 7fad51a19700 1 -- 172.21.2.102:0/1265459152 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad4c0ce450 con 0x7fad4c094960 2021-02-10T11:29:30.351 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:30.350 7fad33fff700 1 -- 172.21.2.102:0/1265459152 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fad44005d00 con 0x7fad4c094960 2021-02-10T11:29:30.479 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:30.478 7fad51a19700 1 -- 172.21.2.102:0/1265459152 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fad4c0ce450 con 0x7fad4c094960 2021-02-10T11:29:30.479 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:30.479 7fad33fff700 1 -- 172.21.2.102:0/1265459152 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+2101 (secure 0 0 0) 0x7fad44006f20 con 0x7fad4c094960 2021-02-10T11:29:30.480 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:29:30.480 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":66,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6110,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.102:6835/1471871278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6834","nonce":1471871278},{"type":"v1","addr":"172.21.2.102:6835","nonce":1471871278}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":61}],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:27:41.479375","modified":"2021-02-10 11:27:43.490605","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6107},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6107":{"gid":6107,"name":"b","rank":0,"incarnation":62,"state":"up:active","state_seq":3,"addr":"172.21.2.113:6833/4220238062","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6832","nonce":4220238062},{"type":"v1","addr":"172.21.2.113:6833","nonce":4220238062}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":8}]} 2021-02-10T11:29:30.480 INFO:teuthology.orchestra.run.gibba002.stderr:dumped fsmap epoch 66 2021-02-10T11:29:30.481 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:30.480 7fad51a19700 1 -- 172.21.2.102:0/1265459152 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fad2c018730 msgr2=0x7fad2c01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:30.481 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:30.480 7fad51a19700 1 --2- 172.21.2.102:0/1265459152 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fad2c018730 0x7fad2c01ac10 crc :-1 s=READY pgs=841 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:30.481 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:30.480 7fad51a19700 1 -- 172.21.2.102:0/1265459152 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fad4c094960 msgr2=0x7fad4c187e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:30.481 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:30.480 7fad51a19700 1 --2- 172.21.2.102:0/1265459152 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fad4c094960 0x7fad4c187e50 secure :-1 s=READY pgs=1875 cs=0 l=1 rev1=1 rx=0x7fad44009980 tx=0x7fad4400bcd0).stop 2021-02-10T11:29:30.481 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:30.481 7fad51a19700 1 -- 172.21.2.102:0/1265459152 shutdown_connections 2021-02-10T11:29:30.482 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:30.481 7fad51a19700 1 --2- 172.21.2.102:0/1265459152 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fad4c094960 0x7fad4c187e50 unknown :-1 s=CLOSED pgs=1875 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:30.482 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:30.481 7fad51a19700 1 --2- 172.21.2.102:0/1265459152 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fad2c018730 0x7fad2c01ac10 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:30.482 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:30.481 7fad51a19700 1 --2- 172.21.2.102:0/1265459152 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fad4c093f40 0x7fad4c094340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:30.482 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:30.481 7fad51a19700 1 --2- 172.21.2.102:0/1265459152 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fad4c0b4f30 0x7fad4c18b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:30.482 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:30.481 7fad51a19700 1 -- 172.21.2.102:0/1265459152 >> 172.21.2.102:0/1265459152 conn(0x7fad4c19f8f0 msgr2=0x7fad4c1a5f80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:29:30.482 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:30.481 7fad51a19700 1 -- 172.21.2.102:0/1265459152 shutdown_connections 2021-02-10T11:29:30.482 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:30.481 7fad51a19700 1 -- 172.21.2.102:0/1265459152 wait complete. 2021-02-10T11:29:30.490 DEBUG:teuthology.orchestra.run.gibba013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok session ls 2021-02-10T11:29:30.888 INFO:teuthology.orchestra.run.gibba013.stdout:[ 2021-02-10T11:29:30.888 INFO:teuthology.orchestra.run.gibba013.stdout: { 2021-02-10T11:29:30.888 INFO:teuthology.orchestra.run.gibba013.stdout: "id": 9471, 2021-02-10T11:29:30.888 INFO:teuthology.orchestra.run.gibba013.stdout: "entity": { 2021-02-10T11:29:30.888 INFO:teuthology.orchestra.run.gibba013.stdout: "name": { 2021-02-10T11:29:30.888 INFO:teuthology.orchestra.run.gibba013.stdout: "type": "client", 2021-02-10T11:29:30.889 INFO:teuthology.orchestra.run.gibba013.stdout: "num": 9471 2021-02-10T11:29:30.889 INFO:teuthology.orchestra.run.gibba013.stdout: }, 2021-02-10T11:29:30.889 INFO:teuthology.orchestra.run.gibba013.stdout: "addr": { 2021-02-10T11:29:30.889 INFO:teuthology.orchestra.run.gibba013.stdout: "type": "any", 2021-02-10T11:29:30.889 INFO:teuthology.orchestra.run.gibba013.stdout: "addr": "172.21.2.102:0", 2021-02-10T11:29:30.889 INFO:teuthology.orchestra.run.gibba013.stdout: "nonce": 190431005 2021-02-10T11:29:30.889 INFO:teuthology.orchestra.run.gibba013.stdout: } 2021-02-10T11:29:30.889 INFO:teuthology.orchestra.run.gibba013.stdout: }, 2021-02-10T11:29:30.889 INFO:teuthology.orchestra.run.gibba013.stdout: "state": "open", 2021-02-10T11:29:30.889 INFO:teuthology.orchestra.run.gibba013.stdout: "num_leases": 0, 2021-02-10T11:29:30.889 INFO:teuthology.orchestra.run.gibba013.stdout: "num_caps": 7, 2021-02-10T11:29:30.889 INFO:teuthology.orchestra.run.gibba013.stdout: "request_load_avg": 5, 2021-02-10T11:29:30.889 INFO:teuthology.orchestra.run.gibba013.stdout: "uptime": 80.211002597000004, 2021-02-10T11:29:30.890 INFO:teuthology.orchestra.run.gibba013.stdout: "requests_in_flight": 0, 2021-02-10T11:29:30.890 INFO:teuthology.orchestra.run.gibba013.stdout: "completed_requests": 1, 2021-02-10T11:29:30.890 INFO:teuthology.orchestra.run.gibba013.stdout: "reconnecting": false, 2021-02-10T11:29:30.890 INFO:teuthology.orchestra.run.gibba013.stdout: "recall_caps": { 2021-02-10T11:29:30.890 INFO:teuthology.orchestra.run.gibba013.stdout: "value": 0, 2021-02-10T11:29:30.890 INFO:teuthology.orchestra.run.gibba013.stdout: "halflife": 60 2021-02-10T11:29:30.890 INFO:teuthology.orchestra.run.gibba013.stdout: }, 2021-02-10T11:29:30.890 INFO:teuthology.orchestra.run.gibba013.stdout: "release_caps": { 2021-02-10T11:29:30.890 INFO:teuthology.orchestra.run.gibba013.stdout: "value": 0, 2021-02-10T11:29:30.890 INFO:teuthology.orchestra.run.gibba013.stdout: "halflife": 60 2021-02-10T11:29:30.890 INFO:teuthology.orchestra.run.gibba013.stdout: }, 2021-02-10T11:29:30.890 INFO:teuthology.orchestra.run.gibba013.stdout: "recall_caps_throttle": { 2021-02-10T11:29:30.890 INFO:teuthology.orchestra.run.gibba013.stdout: "value": 0, 2021-02-10T11:29:30.890 INFO:teuthology.orchestra.run.gibba013.stdout: "halflife": 2.5 2021-02-10T11:29:30.891 INFO:teuthology.orchestra.run.gibba013.stdout: }, 2021-02-10T11:29:30.891 INFO:teuthology.orchestra.run.gibba013.stdout: "recall_caps_throttle2o": { 2021-02-10T11:29:30.891 INFO:teuthology.orchestra.run.gibba013.stdout: "value": 0, 2021-02-10T11:29:30.891 INFO:teuthology.orchestra.run.gibba013.stdout: "halflife": 0.5 2021-02-10T11:29:30.891 INFO:teuthology.orchestra.run.gibba013.stdout: }, 2021-02-10T11:29:30.891 INFO:teuthology.orchestra.run.gibba013.stdout: "session_cache_liveness": { 2021-02-10T11:29:30.891 INFO:teuthology.orchestra.run.gibba013.stdout: "value": 17.45949274401924, 2021-02-10T11:29:30.891 INFO:teuthology.orchestra.run.gibba013.stdout: "halflife": 300 2021-02-10T11:29:30.891 INFO:teuthology.orchestra.run.gibba013.stdout: }, 2021-02-10T11:29:30.891 INFO:teuthology.orchestra.run.gibba013.stdout: "cap_acquisition": { 2021-02-10T11:29:30.891 INFO:teuthology.orchestra.run.gibba013.stdout: "value": 0, 2021-02-10T11:29:30.891 INFO:teuthology.orchestra.run.gibba013.stdout: "halflife": 10 2021-02-10T11:29:30.891 INFO:teuthology.orchestra.run.gibba013.stdout: }, 2021-02-10T11:29:30.891 INFO:teuthology.orchestra.run.gibba013.stdout: "inst": "client.9471 172.21.2.102:0/190431005", 2021-02-10T11:29:30.892 INFO:teuthology.orchestra.run.gibba013.stdout: "completed_requests": [ 2021-02-10T11:29:30.892 INFO:teuthology.orchestra.run.gibba013.stdout: { 2021-02-10T11:29:30.892 INFO:teuthology.orchestra.run.gibba013.stdout: "tid": 14, 2021-02-10T11:29:30.892 INFO:teuthology.orchestra.run.gibba013.stdout: "created_ino": "0x10000000005" 2021-02-10T11:29:30.892 INFO:teuthology.orchestra.run.gibba013.stdout: } 2021-02-10T11:29:30.892 INFO:teuthology.orchestra.run.gibba013.stdout: ], 2021-02-10T11:29:30.892 INFO:teuthology.orchestra.run.gibba013.stdout: "prealloc_inos": [ 2021-02-10T11:29:30.892 INFO:teuthology.orchestra.run.gibba013.stdout: { 2021-02-10T11:29:30.892 INFO:teuthology.orchestra.run.gibba013.stdout: "start": 1099511627782, 2021-02-10T11:29:30.892 INFO:teuthology.orchestra.run.gibba013.stdout: "length": 995 2021-02-10T11:29:30.892 INFO:teuthology.orchestra.run.gibba013.stdout: } 2021-02-10T11:29:30.892 INFO:teuthology.orchestra.run.gibba013.stdout: ], 2021-02-10T11:29:30.892 INFO:teuthology.orchestra.run.gibba013.stdout: "used_inos": [], 2021-02-10T11:29:30.892 INFO:teuthology.orchestra.run.gibba013.stdout: "client_metadata": { 2021-02-10T11:29:30.893 INFO:teuthology.orchestra.run.gibba013.stdout: "features": "0x0000000000001fff", 2021-02-10T11:29:30.893 INFO:teuthology.orchestra.run.gibba013.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:29:30.893 INFO:teuthology.orchestra.run.gibba013.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:29:30.893 INFO:teuthology.orchestra.run.gibba013.stdout: "entity_id": "y", 2021-02-10T11:29:30.893 INFO:teuthology.orchestra.run.gibba013.stdout: "hostname": "gibba002", 2021-02-10T11:29:30.893 INFO:teuthology.orchestra.run.gibba013.stdout: "pid": "35161", 2021-02-10T11:29:30.893 INFO:teuthology.orchestra.run.gibba013.stdout: "root": "/" 2021-02-10T11:29:30.893 INFO:teuthology.orchestra.run.gibba013.stdout: } 2021-02-10T11:29:30.893 INFO:teuthology.orchestra.run.gibba013.stdout: } 2021-02-10T11:29:30.893 INFO:teuthology.orchestra.run.gibba013.stdout:] 2021-02-10T11:29:30.894 INFO:tasks.cephfs.filesystem:_json_asok output: [ { "id": 9471, "entity": { "name": { "type": "client", "num": 9471 }, "addr": { "type": "any", "addr": "172.21.2.102:0", "nonce": 190431005 } }, "state": "open", "num_leases": 0, "num_caps": 7, "request_load_avg": 5, "uptime": 80.211002597000004, "requests_in_flight": 0, "completed_requests": 1, "reconnecting": false, "recall_caps": { "value": 0, "halflife": 60 }, "release_caps": { "value": 0, "halflife": 60 }, "recall_caps_throttle": { "value": 0, "halflife": 2.5 }, "recall_caps_throttle2o": { "value": 0, "halflife": 0.5 }, "session_cache_liveness": { "value": 17.45949274401924, "halflife": 300 }, "cap_acquisition": { "value": 0, "halflife": 10 }, "inst": "client.9471 172.21.2.102:0/190431005", "completed_requests": [ { "tid": 14, "created_ino": "0x10000000005" } ], "prealloc_inos": [ { "start": 1099511627782, "length": 995 } ], "used_inos": [], "client_metadata": { "features": "0x0000000000001fff", "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", "entity_id": "y", "hostname": "gibba002", "pid": "35161", "root": "/" } } ] 2021-02-10T11:29:30.894 DEBUG:tasks.ceph_test_case:wait_until_true: waiting... 2021-02-10T11:29:35.896 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:29:36.323 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:36.307 7f1d362dc700 1 Processor -- start 2021-02-10T11:29:36.324 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:36.307 7f1d362dc700 1 -- start start 2021-02-10T11:29:36.324 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:36.308 7f1d362dc700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f1d300d1870 0x7f1d301b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:36.324 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:36.308 7f1d362dc700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f1d301b0a10 0x7f1d300cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:36.324 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:36.308 7f1d362dc700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1d300d1c70 0x7f1d300cf950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:36.324 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:36.308 7f1d362dc700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d300e5ed0 con 0x7f1d301b0a10 2021-02-10T11:29:36.324 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:36.308 7f1d362dc700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d300f0b10 con 0x7f1d300d1870 2021-02-10T11:29:36.324 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:36.308 7f1d362dc700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d300cff70 con 0x7f1d300d1c70 2021-02-10T11:29:36.324 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:36.308 7f1d2f7fe700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f1d301b0a10 0x7f1d300cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:36.324 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:36.308 7f1d2f7fe700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f1d301b0a10 0x7f1d300cf330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:60372/0 (socket says 172.21.2.102:60372) 2021-02-10T11:29:36.324 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:36.308 7f1d2f7fe700 1 -- 172.21.2.102:0/2896225257 learned_addr learned my addr 172.21.2.102:0/2896225257 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:29:36.324 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:36.308 7f1d2f7fe700 1 -- 172.21.2.102:0/2896225257 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1d300d1c70 msgr2=0x7f1d300cf950 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:29:36.325 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:36.308 7f1d2f7fe700 1 --2- 172.21.2.102:0/2896225257 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1d300d1c70 0x7f1d300cf950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:36.325 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:36.308 7f1d2f7fe700 1 -- 172.21.2.102:0/2896225257 >> v2:172.21.2.113:3300/0 conn(0x7f1d300d1870 msgr2=0x7f1d301b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:29:36.325 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:36.308 7f1d2f7fe700 1 --2- 172.21.2.102:0/2896225257 >> v2:172.21.2.113:3300/0 conn(0x7f1d300d1870 0x7f1d301b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:36.325 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:36.308 7f1d2f7fe700 1 -- 172.21.2.102:0/2896225257 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d300d0150 con 0x7f1d301b0a10 2021-02-10T11:29:36.325 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:36.308 7f1d2f7fe700 1 --2- 172.21.2.102:0/2896225257 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1d301b0a10 0x7f1d300cf330 secure :-1 s=READY pgs=1876 cs=0 l=1 rev1=1 rx=0x7f1d24006700 tx=0x7f1d24009b80).ready entity=mon.0 client_cookie=4ee97084440f69b2 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:29:36.325 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:36.308 7f1d2effd700 1 -- 172.21.2.102:0/2896225257 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1d2400c0c0 con 0x7f1d301b0a10 2021-02-10T11:29:36.325 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:36.309 7f1d2effd700 1 -- 172.21.2.102:0/2896225257 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d2400c340 con 0x7f1d301b0a10 2021-02-10T11:29:36.325 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:36.309 7f1d2effd700 1 -- 172.21.2.102:0/2896225257 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1d2400c860 con 0x7f1d301b0a10 2021-02-10T11:29:36.325 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:36.309 7f1d362dc700 1 -- 172.21.2.102:0/2896225257 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1d301b0a10 msgr2=0x7f1d300cf330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:36.325 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:36.309 7f1d362dc700 1 --2- 172.21.2.102:0/2896225257 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1d301b0a10 0x7f1d300cf330 secure :-1 s=READY pgs=1876 cs=0 l=1 rev1=1 rx=0x7f1d24006700 tx=0x7f1d24009b80).stop 2021-02-10T11:29:36.325 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:36.309 7f1d362dc700 1 -- 172.21.2.102:0/2896225257 shutdown_connections 2021-02-10T11:29:36.325 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:36.309 7f1d362dc700 1 --2- 172.21.2.102:0/2896225257 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1d300d1c70 0x7f1d300cf950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:36.325 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:36.309 7f1d362dc700 1 --2- 172.21.2.102:0/2896225257 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1d301b0a10 0x7f1d300cf330 unknown :-1 s=CLOSED pgs=1876 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:36.326 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:36.309 7f1d362dc700 1 --2- 172.21.2.102:0/2896225257 >> v2:172.21.2.113:3300/0 conn(0x7f1d300d1870 0x7f1d301b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:36.326 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:36.309 7f1d362dc700 1 -- 172.21.2.102:0/2896225257 >> 172.21.2.102:0/2896225257 conn(0x7f1d300ce450 msgr2=0x7f1d300d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:29:36.326 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:36.309 7f1d362dc700 1 -- 172.21.2.102:0/2896225257 shutdown_connections 2021-02-10T11:29:36.326 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:36.309 7f1d362dc700 1 -- 172.21.2.102:0/2896225257 wait complete. 2021-02-10T11:29:36.327 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:36.324 7f1d362dc700 1 Processor -- start 2021-02-10T11:29:36.327 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:36.324 7f1d362dc700 1 -- start start 2021-02-10T11:29:36.327 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:36.324 7f1d362dc700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1d301b7cf0 0x7f1d3019cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:36.328 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:36.324 7f1d362dc700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1d301b80f0 0x7f1d301b84f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:36.328 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:36.324 7f1d362dc700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1d3019da20 0x7f1d3019de20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:36.328 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:36.324 7f1d362dc700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d300a77c0 con 0x7f1d3019da20 2021-02-10T11:29:36.328 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:36.324 7f1d362dc700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d3019d4a0 con 0x7f1d301b7cf0 2021-02-10T11:29:36.328 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:36.324 7f1d362dc700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d300954f0 con 0x7f1d301b80f0 2021-02-10T11:29:36.328 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:36.324 7f1d348f9700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1d3019da20 0x7f1d3019de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:36.328 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:36.324 7f1d348f9700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1d3019da20 0x7f1d3019de20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:60380/0 (socket says 172.21.2.102:60380) 2021-02-10T11:29:36.328 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:36.324 7f1d348f9700 1 -- 172.21.2.102:0/3001353311 learned_addr learned my addr 172.21.2.102:0/3001353311 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:29:36.328 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:36.324 7f1d2f7fe700 1 --2- 172.21.2.102:0/3001353311 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1d301b80f0 0x7f1d301b84f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:36.328 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:36.324 7f1d2ffff700 1 --2- 172.21.2.102:0/3001353311 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1d301b7cf0 0x7f1d3019cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:36.328 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:36.325 7f1d348f9700 1 -- 172.21.2.102:0/3001353311 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1d301b80f0 msgr2=0x7f1d301b84f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:36.328 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:36.325 7f1d348f9700 1 --2- 172.21.2.102:0/3001353311 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1d301b80f0 0x7f1d301b84f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:36.329 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:36.325 7f1d348f9700 1 -- 172.21.2.102:0/3001353311 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1d301b7cf0 msgr2=0x7f1d3019cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:36.329 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:36.325 7f1d348f9700 1 --2- 172.21.2.102:0/3001353311 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1d301b7cf0 0x7f1d3019cdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:36.329 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:36.325 7f1d348f9700 1 -- 172.21.2.102:0/3001353311 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d301b8d60 con 0x7f1d3019da20 2021-02-10T11:29:36.329 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:36.325 7f1d2f7fe700 1 --2- 172.21.2.102:0/3001353311 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1d301b80f0 0x7f1d301b84f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:29:36.329 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:36.325 7f1d348f9700 1 --2- 172.21.2.102:0/3001353311 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1d3019da20 0x7f1d3019de20 secure :-1 s=READY pgs=1877 cs=0 l=1 rev1=1 rx=0x7f1d28009980 tx=0x7f1d2800bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:29:36.329 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:36.325 7f1d2cff9700 1 -- 172.21.2.102:0/3001353311 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1d28005870 con 0x7f1d3019da20 2021-02-10T11:29:36.329 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:36.325 7f1d2cff9700 1 -- 172.21.2.102:0/3001353311 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d28005b30 con 0x7f1d3019da20 2021-02-10T11:29:36.329 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:36.325 7f1d362dc700 1 -- 172.21.2.102:0/3001353311 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d301b8d60 con 0x7f1d3019da20 2021-02-10T11:29:36.329 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:36.325 7f1d2cff9700 1 -- 172.21.2.102:0/3001353311 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1d28006030 con 0x7f1d3019da20 2021-02-10T11:29:36.329 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:36.325 7f1d362dc700 1 -- 172.21.2.102:0/3001353311 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d3019e730 con 0x7f1d3019da20 2021-02-10T11:29:36.329 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:36.326 7f1d2cff9700 1 -- 172.21.2.102:0/3001353311 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1d28005d10 con 0x7f1d3019da20 2021-02-10T11:29:36.329 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:36.326 7f1d2cff9700 1 --2- 172.21.2.102:0/3001353311 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1d100186c0 0x7f1d1001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:36.329 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:36.326 7f1d2cff9700 1 -- 172.21.2.102:0/3001353311 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f1d28032810 con 0x7f1d3019da20 2021-02-10T11:29:36.330 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:36.326 7f1d2ffff700 1 --2- 172.21.2.102:0/3001353311 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1d100186c0 0x7f1d1001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:36.330 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:36.326 7f1d2ffff700 1 --2- 172.21.2.102:0/3001353311 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1d100186c0 0x7f1d1001abc0 crc :-1 s=READY pgs=842 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:29:36.358 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:36.357 7f1d362dc700 1 -- 172.21.2.102:0/3001353311 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d300ebc00 con 0x7f1d3019da20 2021-02-10T11:29:36.360 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:36.360 7f1d2cff9700 1 -- 172.21.2.102:0/3001353311 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f1d2802ded0 con 0x7f1d3019da20 2021-02-10T11:29:36.489 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:36.488 7f1d362dc700 1 -- 172.21.2.102:0/3001353311 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1d300f0b10 con 0x7f1d3019da20 2021-02-10T11:29:36.489 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:36.489 7f1d2cff9700 1 -- 172.21.2.102:0/3001353311 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+2101 (secure 0 0 0) 0x7f1d2802ded0 con 0x7f1d3019da20 2021-02-10T11:29:36.490 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:29:36.490 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":66,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6110,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.102:6835/1471871278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6834","nonce":1471871278},{"type":"v1","addr":"172.21.2.102:6835","nonce":1471871278}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":61}],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:27:41.479375","modified":"2021-02-10 11:27:43.490605","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6107},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6107":{"gid":6107,"name":"b","rank":0,"incarnation":62,"state":"up:active","state_seq":3,"addr":"172.21.2.113:6833/4220238062","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6832","nonce":4220238062},{"type":"v1","addr":"172.21.2.113:6833","nonce":4220238062}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":8}]} 2021-02-10T11:29:36.490 INFO:teuthology.orchestra.run.gibba002.stderr:dumped fsmap epoch 66 2021-02-10T11:29:36.491 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:36.490 7f1d362dc700 1 -- 172.21.2.102:0/3001353311 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1d100186c0 msgr2=0x7f1d1001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:36.491 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:36.490 7f1d362dc700 1 --2- 172.21.2.102:0/3001353311 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1d100186c0 0x7f1d1001abc0 crc :-1 s=READY pgs=842 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:36.491 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:36.490 7f1d362dc700 1 -- 172.21.2.102:0/3001353311 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1d3019da20 msgr2=0x7f1d3019de20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:36.491 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:36.490 7f1d362dc700 1 --2- 172.21.2.102:0/3001353311 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1d3019da20 0x7f1d3019de20 secure :-1 s=READY pgs=1877 cs=0 l=1 rev1=1 rx=0x7f1d28009980 tx=0x7f1d2800bcd0).stop 2021-02-10T11:29:36.491 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:36.491 7f1d362dc700 1 -- 172.21.2.102:0/3001353311 shutdown_connections 2021-02-10T11:29:36.491 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:36.491 7f1d362dc700 1 --2- 172.21.2.102:0/3001353311 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1d3019da20 0x7f1d3019de20 unknown :-1 s=CLOSED pgs=1877 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:36.491 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:36.491 7f1d362dc700 1 --2- 172.21.2.102:0/3001353311 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1d301b80f0 0x7f1d301b84f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:36.491 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:36.491 7f1d362dc700 1 --2- 172.21.2.102:0/3001353311 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1d100186c0 0x7f1d1001abc0 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:36.491 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:36.491 7f1d362dc700 1 --2- 172.21.2.102:0/3001353311 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1d301b7cf0 0x7f1d3019cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:36.491 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:36.491 7f1d362dc700 1 -- 172.21.2.102:0/3001353311 >> 172.21.2.102:0/3001353311 conn(0x7f1d300ed810 msgr2=0x7f1d30094980 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:29:36.492 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:36.491 7f1d362dc700 1 -- 172.21.2.102:0/3001353311 shutdown_connections 2021-02-10T11:29:36.492 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:36.491 7f1d362dc700 1 -- 172.21.2.102:0/3001353311 wait complete. 2021-02-10T11:29:36.500 DEBUG:teuthology.orchestra.run.gibba013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok session ls 2021-02-10T11:29:36.897 INFO:teuthology.orchestra.run.gibba013.stdout:[ 2021-02-10T11:29:36.897 INFO:teuthology.orchestra.run.gibba013.stdout: { 2021-02-10T11:29:36.897 INFO:teuthology.orchestra.run.gibba013.stdout: "id": 9471, 2021-02-10T11:29:36.897 INFO:teuthology.orchestra.run.gibba013.stdout: "entity": { 2021-02-10T11:29:36.897 INFO:teuthology.orchestra.run.gibba013.stdout: "name": { 2021-02-10T11:29:36.897 INFO:teuthology.orchestra.run.gibba013.stdout: "type": "client", 2021-02-10T11:29:36.897 INFO:teuthology.orchestra.run.gibba013.stdout: "num": 9471 2021-02-10T11:29:36.897 INFO:teuthology.orchestra.run.gibba013.stdout: }, 2021-02-10T11:29:36.898 INFO:teuthology.orchestra.run.gibba013.stdout: "addr": { 2021-02-10T11:29:36.898 INFO:teuthology.orchestra.run.gibba013.stdout: "type": "any", 2021-02-10T11:29:36.898 INFO:teuthology.orchestra.run.gibba013.stdout: "addr": "172.21.2.102:0", 2021-02-10T11:29:36.898 INFO:teuthology.orchestra.run.gibba013.stdout: "nonce": 190431005 2021-02-10T11:29:36.898 INFO:teuthology.orchestra.run.gibba013.stdout: } 2021-02-10T11:29:36.898 INFO:teuthology.orchestra.run.gibba013.stdout: }, 2021-02-10T11:29:36.898 INFO:teuthology.orchestra.run.gibba013.stdout: "state": "open", 2021-02-10T11:29:36.898 INFO:teuthology.orchestra.run.gibba013.stdout: "num_leases": 0, 2021-02-10T11:29:36.898 INFO:teuthology.orchestra.run.gibba013.stdout: "num_caps": 7, 2021-02-10T11:29:36.898 INFO:teuthology.orchestra.run.gibba013.stdout: "request_load_avg": 5, 2021-02-10T11:29:36.898 INFO:teuthology.orchestra.run.gibba013.stdout: "uptime": 86.220002683999994, 2021-02-10T11:29:36.898 INFO:teuthology.orchestra.run.gibba013.stdout: "requests_in_flight": 0, 2021-02-10T11:29:36.899 INFO:teuthology.orchestra.run.gibba013.stdout: "completed_requests": 1, 2021-02-10T11:29:36.899 INFO:teuthology.orchestra.run.gibba013.stdout: "reconnecting": false, 2021-02-10T11:29:36.899 INFO:teuthology.orchestra.run.gibba013.stdout: "recall_caps": { 2021-02-10T11:29:36.899 INFO:teuthology.orchestra.run.gibba013.stdout: "value": 0, 2021-02-10T11:29:36.899 INFO:teuthology.orchestra.run.gibba013.stdout: "halflife": 60 2021-02-10T11:29:36.899 INFO:teuthology.orchestra.run.gibba013.stdout: }, 2021-02-10T11:29:36.899 INFO:teuthology.orchestra.run.gibba013.stdout: "release_caps": { 2021-02-10T11:29:36.899 INFO:teuthology.orchestra.run.gibba013.stdout: "value": 0, 2021-02-10T11:29:36.899 INFO:teuthology.orchestra.run.gibba013.stdout: "halflife": 60 2021-02-10T11:29:36.899 INFO:teuthology.orchestra.run.gibba013.stdout: }, 2021-02-10T11:29:36.899 INFO:teuthology.orchestra.run.gibba013.stdout: "recall_caps_throttle": { 2021-02-10T11:29:36.899 INFO:teuthology.orchestra.run.gibba013.stdout: "value": 0, 2021-02-10T11:29:36.899 INFO:teuthology.orchestra.run.gibba013.stdout: "halflife": 2.5 2021-02-10T11:29:36.900 INFO:teuthology.orchestra.run.gibba013.stdout: }, 2021-02-10T11:29:36.900 INFO:teuthology.orchestra.run.gibba013.stdout: "recall_caps_throttle2o": { 2021-02-10T11:29:36.900 INFO:teuthology.orchestra.run.gibba013.stdout: "value": 0, 2021-02-10T11:29:36.900 INFO:teuthology.orchestra.run.gibba013.stdout: "halflife": 0.5 2021-02-10T11:29:36.900 INFO:teuthology.orchestra.run.gibba013.stdout: }, 2021-02-10T11:29:36.900 INFO:teuthology.orchestra.run.gibba013.stdout: "session_cache_liveness": { 2021-02-10T11:29:36.900 INFO:teuthology.orchestra.run.gibba013.stdout: "value": 17.218764686686363, 2021-02-10T11:29:36.900 INFO:teuthology.orchestra.run.gibba013.stdout: "halflife": 300 2021-02-10T11:29:36.900 INFO:teuthology.orchestra.run.gibba013.stdout: }, 2021-02-10T11:29:36.900 INFO:teuthology.orchestra.run.gibba013.stdout: "cap_acquisition": { 2021-02-10T11:29:36.900 INFO:teuthology.orchestra.run.gibba013.stdout: "value": 0, 2021-02-10T11:29:36.900 INFO:teuthology.orchestra.run.gibba013.stdout: "halflife": 10 2021-02-10T11:29:36.900 INFO:teuthology.orchestra.run.gibba013.stdout: }, 2021-02-10T11:29:36.900 INFO:teuthology.orchestra.run.gibba013.stdout: "inst": "client.9471 172.21.2.102:0/190431005", 2021-02-10T11:29:36.901 INFO:teuthology.orchestra.run.gibba013.stdout: "completed_requests": [ 2021-02-10T11:29:36.901 INFO:teuthology.orchestra.run.gibba013.stdout: { 2021-02-10T11:29:36.901 INFO:teuthology.orchestra.run.gibba013.stdout: "tid": 14, 2021-02-10T11:29:36.901 INFO:teuthology.orchestra.run.gibba013.stdout: "created_ino": "0x10000000005" 2021-02-10T11:29:36.901 INFO:teuthology.orchestra.run.gibba013.stdout: } 2021-02-10T11:29:36.901 INFO:teuthology.orchestra.run.gibba013.stdout: ], 2021-02-10T11:29:36.901 INFO:teuthology.orchestra.run.gibba013.stdout: "prealloc_inos": [ 2021-02-10T11:29:36.901 INFO:teuthology.orchestra.run.gibba013.stdout: { 2021-02-10T11:29:36.901 INFO:teuthology.orchestra.run.gibba013.stdout: "start": 1099511627782, 2021-02-10T11:29:36.901 INFO:teuthology.orchestra.run.gibba013.stdout: "length": 995 2021-02-10T11:29:36.901 INFO:teuthology.orchestra.run.gibba013.stdout: } 2021-02-10T11:29:36.901 INFO:teuthology.orchestra.run.gibba013.stdout: ], 2021-02-10T11:29:36.901 INFO:teuthology.orchestra.run.gibba013.stdout: "used_inos": [], 2021-02-10T11:29:36.902 INFO:teuthology.orchestra.run.gibba013.stdout: "client_metadata": { 2021-02-10T11:29:36.902 INFO:teuthology.orchestra.run.gibba013.stdout: "features": "0x0000000000001fff", 2021-02-10T11:29:36.902 INFO:teuthology.orchestra.run.gibba013.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:29:36.902 INFO:teuthology.orchestra.run.gibba013.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:29:36.902 INFO:teuthology.orchestra.run.gibba013.stdout: "entity_id": "y", 2021-02-10T11:29:36.902 INFO:teuthology.orchestra.run.gibba013.stdout: "hostname": "gibba002", 2021-02-10T11:29:36.902 INFO:teuthology.orchestra.run.gibba013.stdout: "pid": "35161", 2021-02-10T11:29:36.902 INFO:teuthology.orchestra.run.gibba013.stdout: "root": "/" 2021-02-10T11:29:36.902 INFO:teuthology.orchestra.run.gibba013.stdout: } 2021-02-10T11:29:36.902 INFO:teuthology.orchestra.run.gibba013.stdout: } 2021-02-10T11:29:36.902 INFO:teuthology.orchestra.run.gibba013.stdout:] 2021-02-10T11:29:36.903 INFO:tasks.cephfs.filesystem:_json_asok output: [ { "id": 9471, "entity": { "name": { "type": "client", "num": 9471 }, "addr": { "type": "any", "addr": "172.21.2.102:0", "nonce": 190431005 } }, "state": "open", "num_leases": 0, "num_caps": 7, "request_load_avg": 5, "uptime": 86.220002683999994, "requests_in_flight": 0, "completed_requests": 1, "reconnecting": false, "recall_caps": { "value": 0, "halflife": 60 }, "release_caps": { "value": 0, "halflife": 60 }, "recall_caps_throttle": { "value": 0, "halflife": 2.5 }, "recall_caps_throttle2o": { "value": 0, "halflife": 0.5 }, "session_cache_liveness": { "value": 17.218764686686363, "halflife": 300 }, "cap_acquisition": { "value": 0, "halflife": 10 }, "inst": "client.9471 172.21.2.102:0/190431005", "completed_requests": [ { "tid": 14, "created_ino": "0x10000000005" } ], "prealloc_inos": [ { "start": 1099511627782, "length": 995 } ], "used_inos": [], "client_metadata": { "features": "0x0000000000001fff", "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", "entity_id": "y", "hostname": "gibba002", "pid": "35161", "root": "/" } } ] 2021-02-10T11:29:36.903 DEBUG:tasks.ceph_test_case:wait_until_true: waiting... 2021-02-10T11:29:37.927 DEBUG:teuthology.orchestra.run.gibba002:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:29:37.930 DEBUG:teuthology.orchestra.run.gibba013:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:29:41.905 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:29:42.332 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:42.315 7fe1ba863700 1 Processor -- start 2021-02-10T11:29:42.332 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:42.315 7fe1ba863700 1 -- start start 2021-02-10T11:29:42.333 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:42.315 7fe1ba863700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fe1b40d1870 0x7fe1b41b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:42.333 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:42.316 7fe1ba863700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe1b40f0650 0x7fe1b40cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:42.333 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:42.316 7fe1ba863700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1b40f0b10 con 0x7fe1b41b0a10 2021-02-10T11:29:42.333 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:42.316 7fe1ba863700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1b40e5ed0 con 0x7fe1b40d1870 2021-02-10T11:29:42.333 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:42.316 7fe1ba863700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1b40d1c70 con 0x7fe1b40f0650 2021-02-10T11:29:42.333 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:42.316 7fe1b37fe700 1 --1- >> v1:172.21.2.113:6789/0 conn(0x7fe1b41b0a10 0x7fe1b40cf330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.113:6789/0 says I am v1:172.21.2.102:59934/0 (socket says 172.21.2.102:59934) 2021-02-10T11:29:42.333 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:42.316 7fe1b37fe700 1 -- 172.21.2.102:0/1778051614 learned_addr learned my addr 172.21.2.102:0/1778051614 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:29:42.333 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:42.316 7fe1b8e80700 1 --2- 172.21.2.102:0/1778051614 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe1b40f0650 0x7fe1b40cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:42.333 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:42.316 7fe1b3fff700 1 --2- 172.21.2.102:0/1778051614 >> v2:172.21.2.113:3300/0 conn(0x7fe1b40d1870 0x7fe1b41b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:42.333 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:42.317 7fe1b2ffd700 1 -- 172.21.2.102:0/1778051614 <== mon.1 v1:172.21.2.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3113599337 0 0) 0x7fe1a80010b0 con 0x7fe1b41b0a10 2021-02-10T11:29:42.333 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:42.317 7fe1b2ffd700 1 -- 172.21.2.102:0/1778051614 --> v1:172.21.2.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe19c002fe0 con 0x7fe1b41b0a10 2021-02-10T11:29:42.333 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:42.317 7fe1b8e80700 1 -- 172.21.2.102:0/1778051614 >> v2:172.21.2.113:3300/0 conn(0x7fe1b40d1870 msgr2=0x7fe1b41b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:29:42.334 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:42.317 7fe1b8e80700 1 --2- 172.21.2.102:0/1778051614 >> v2:172.21.2.113:3300/0 conn(0x7fe1b40d1870 0x7fe1b41b03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:42.334 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:42.317 7fe1b8e80700 1 -- 172.21.2.102:0/1778051614 >> v1:172.21.2.113:6789/0 conn(0x7fe1b41b0a10 legacy=0x7fe1b40cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:42.334 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:42.317 7fe1b8e80700 1 -- 172.21.2.102:0/1778051614 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1b40d1e90 con 0x7fe1b40f0650 2021-02-10T11:29:42.334 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:42.317 7fe1b3fff700 1 --2- 172.21.2.102:0/1778051614 >> v2:172.21.2.113:3300/0 conn(0x7fe1b40d1870 0x7fe1b41b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:29:42.334 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:42.317 7fe1b8e80700 1 --2- 172.21.2.102:0/1778051614 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe1b40f0650 0x7fe1b40cfb20 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7fe1ac006e50 tx=0x7fe1ac009e70).ready entity=mon.2 client_cookie=6eaf5d113b5253b1 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:29:42.334 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:42.317 7fe1b2ffd700 1 -- 172.21.2.102:0/1778051614 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe1ac000f90 con 0x7fe1b40f0650 2021-02-10T11:29:42.334 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:42.317 7fe1b2ffd700 1 -- 172.21.2.102:0/1778051614 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1ac0066f0 con 0x7fe1b40f0650 2021-02-10T11:29:42.334 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:42.318 7fe1ba863700 1 -- 172.21.2.102:0/1778051614 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe1b40f0650 msgr2=0x7fe1b40cfb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:42.334 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:42.318 7fe1ba863700 1 --2- 172.21.2.102:0/1778051614 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe1b40f0650 0x7fe1b40cfb20 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7fe1ac006e50 tx=0x7fe1ac009e70).stop 2021-02-10T11:29:42.334 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:42.317 7fe1b2ffd700 1 -- 172.21.2.102:0/1778051614 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe1ac00b700 con 0x7fe1b40f0650 2021-02-10T11:29:42.334 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:42.318 7fe1ba863700 1 -- 172.21.2.102:0/1778051614 shutdown_connections 2021-02-10T11:29:42.334 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:42.318 7fe1ba863700 1 --2- 172.21.2.102:0/1778051614 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe1b40f0650 0x7fe1b40cfb20 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:42.335 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:42.318 7fe1ba863700 1 --2- 172.21.2.102:0/1778051614 >> v2:172.21.2.113:3300/0 conn(0x7fe1b40d1870 0x7fe1b41b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:42.335 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:42.318 7fe1ba863700 1 -- 172.21.2.102:0/1778051614 >> 172.21.2.102:0/1778051614 conn(0x7fe1b40ce450 msgr2=0x7fe1b40d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:29:42.335 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:42.318 7fe1ba863700 1 -- 172.21.2.102:0/1778051614 shutdown_connections 2021-02-10T11:29:42.335 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:42.318 7fe1ba863700 1 -- 172.21.2.102:0/1778051614 wait complete. 2021-02-10T11:29:42.336 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:42.332 7fe1ba863700 1 Processor -- start 2021-02-10T11:29:42.336 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:42.332 7fe1ba863700 1 -- start start 2021-02-10T11:29:42.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:42.333 7fe1ba863700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe1b4093f10 0x7fe1b4094310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:42.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:42.333 7fe1ba863700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe1b40ae9e0 0x7fe1b40aede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:42.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:42.333 7fe1ba863700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe1b4092d90 0x7fe1b4093190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:42.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:42.333 7fe1ba863700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1b419ebc0 con 0x7fe1b4093f10 2021-02-10T11:29:42.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:42.333 7fe1ba863700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1b41b8760 con 0x7fe1b40ae9e0 2021-02-10T11:29:42.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:42.333 7fe1ba863700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1b40937b0 con 0x7fe1b4092d90 2021-02-10T11:29:42.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:42.333 7fe1b3fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe1b4093f10 0x7fe1b4094310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:42.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:42.333 7fe1b3fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe1b4093f10 0x7fe1b4094310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:60390/0 (socket says 172.21.2.102:60390) 2021-02-10T11:29:42.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:42.333 7fe1b3fff700 1 -- 172.21.2.102:0/2513469080 learned_addr learned my addr 172.21.2.102:0/2513469080 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:29:42.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:42.333 7fe1b3fff700 1 -- 172.21.2.102:0/2513469080 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe1b4092d90 msgr2=0x7fe1b4093190 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:29:42.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:42.333 7fe1b3fff700 1 --2- 172.21.2.102:0/2513469080 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe1b4092d90 0x7fe1b4093190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:42.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:42.333 7fe1b3fff700 1 -- 172.21.2.102:0/2513469080 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe1b40ae9e0 msgr2=0x7fe1b40aede0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:29:42.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:42.333 7fe1b3fff700 1 --2- 172.21.2.102:0/2513469080 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe1b40ae9e0 0x7fe1b40aede0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:42.338 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:42.333 7fe1b3fff700 1 -- 172.21.2.102:0/2513469080 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1b40939d0 con 0x7fe1b4093f10 2021-02-10T11:29:42.338 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:42.333 7fe1b3fff700 1 --2- 172.21.2.102:0/2513469080 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe1b4093f10 0x7fe1b4094310 secure :-1 s=READY pgs=1878 cs=0 l=1 rev1=1 rx=0x7fe1a4006f40 tx=0x7fe1a400b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:29:42.338 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:42.334 7fe1b0ff9700 1 -- 172.21.2.102:0/2513469080 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe1a400bd80 con 0x7fe1b4093f10 2021-02-10T11:29:42.338 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:42.334 7fe1b0ff9700 1 -- 172.21.2.102:0/2513469080 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1a4000c90 con 0x7fe1b4093f10 2021-02-10T11:29:42.338 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:42.334 7fe1b0ff9700 1 -- 172.21.2.102:0/2513469080 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe1a4002220 con 0x7fe1b4093f10 2021-02-10T11:29:42.338 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:42.334 7fe1ba863700 1 -- 172.21.2.102:0/2513469080 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1b40939d0 con 0x7fe1b4093f10 2021-02-10T11:29:42.338 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:42.334 7fe1ba863700 1 -- 172.21.2.102:0/2513469080 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1b40939d0 con 0x7fe1b4093f10 2021-02-10T11:29:42.338 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:42.334 7fe1b0ff9700 1 -- 172.21.2.102:0/2513469080 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fe1a400a000 con 0x7fe1b4093f10 2021-02-10T11:29:42.338 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:42.334 7fe1b0ff9700 1 --2- 172.21.2.102:0/2513469080 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe1940186c0 0x7fe19401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:42.338 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:42.334 7fe1b0ff9700 1 -- 172.21.2.102:0/2513469080 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fe1a4032570 con 0x7fe1b4093f10 2021-02-10T11:29:42.338 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:42.334 7fe1b37fe700 1 --2- 172.21.2.102:0/2513469080 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe1940186c0 0x7fe19401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:42.338 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:42.335 7fe1b37fe700 1 --2- 172.21.2.102:0/2513469080 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe1940186c0 0x7fe19401abc0 crc :-1 s=READY pgs=843 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:29:42.366 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:42.365 7fe1ba863700 1 -- 172.21.2.102:0/2513469080 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1b40ebc60 con 0x7fe1b4093f10 2021-02-10T11:29:42.368 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:42.368 7fe1b0ff9700 1 -- 172.21.2.102:0/2513469080 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fe1a402df00 con 0x7fe1b4093f10 2021-02-10T11:29:42.495 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:42.495 7fe1ba863700 1 -- 172.21.2.102:0/2513469080 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe1b40f0b10 con 0x7fe1b4093f10 2021-02-10T11:29:42.496 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:42.495 7fe1b0ff9700 1 -- 172.21.2.102:0/2513469080 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+2101 (secure 0 0 0) 0x7fe1a402df00 con 0x7fe1b4093f10 2021-02-10T11:29:42.496 INFO:teuthology.orchestra.run.gibba002.stderr:dumped fsmap epoch 66 2021-02-10T11:29:42.496 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:29:42.496 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":66,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6110,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.102:6835/1471871278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6834","nonce":1471871278},{"type":"v1","addr":"172.21.2.102:6835","nonce":1471871278}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":61}],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:27:41.479375","modified":"2021-02-10 11:27:43.490605","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6107},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6107":{"gid":6107,"name":"b","rank":0,"incarnation":62,"state":"up:active","state_seq":3,"addr":"172.21.2.113:6833/4220238062","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6832","nonce":4220238062},{"type":"v1","addr":"172.21.2.113:6833","nonce":4220238062}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":8}]} 2021-02-10T11:29:42.497 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:42.497 7fe1ba863700 1 -- 172.21.2.102:0/2513469080 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe1940186c0 msgr2=0x7fe19401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:42.498 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:42.497 7fe1ba863700 1 --2- 172.21.2.102:0/2513469080 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe1940186c0 0x7fe19401abc0 crc :-1 s=READY pgs=843 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:42.498 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:42.497 7fe1ba863700 1 -- 172.21.2.102:0/2513469080 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe1b4093f10 msgr2=0x7fe1b4094310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:42.498 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:42.497 7fe1ba863700 1 --2- 172.21.2.102:0/2513469080 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe1b4093f10 0x7fe1b4094310 secure :-1 s=READY pgs=1878 cs=0 l=1 rev1=1 rx=0x7fe1a4006f40 tx=0x7fe1a400b990).stop 2021-02-10T11:29:42.498 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:42.497 7fe1ba863700 1 -- 172.21.2.102:0/2513469080 shutdown_connections 2021-02-10T11:29:42.498 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:42.497 7fe1ba863700 1 --2- 172.21.2.102:0/2513469080 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe1b4092d90 0x7fe1b4093190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:42.498 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:42.497 7fe1ba863700 1 --2- 172.21.2.102:0/2513469080 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe1940186c0 0x7fe19401abc0 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:42.498 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:42.497 7fe1ba863700 1 --2- 172.21.2.102:0/2513469080 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe1b40ae9e0 0x7fe1b40aede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:42.499 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:42.497 7fe1ba863700 1 --2- 172.21.2.102:0/2513469080 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe1b4093f10 0x7fe1b4094310 unknown :-1 s=CLOSED pgs=1878 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:42.499 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:42.497 7fe1ba863700 1 -- 172.21.2.102:0/2513469080 >> 172.21.2.102:0/2513469080 conn(0x7fe1b419f8a0 msgr2=0x7fe1b4094980 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:29:42.499 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:42.498 7fe1ba863700 1 -- 172.21.2.102:0/2513469080 shutdown_connections 2021-02-10T11:29:42.499 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:42.498 7fe1ba863700 1 -- 172.21.2.102:0/2513469080 wait complete. 2021-02-10T11:29:42.506 DEBUG:teuthology.orchestra.run.gibba013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok session ls 2021-02-10T11:29:42.904 INFO:teuthology.orchestra.run.gibba013.stdout:[] 2021-02-10T11:29:42.908 INFO:tasks.cephfs.filesystem:_json_asok output: [] 2021-02-10T11:29:42.909 DEBUG:tasks.ceph_test_case:wait_until_true: success in 75s 2021-02-10T11:29:42.909 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2021-02-10T11:29:42.909 DEBUG:teuthology.orchestra.run.gibba002:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:29:43.255 DEBUG:teuthology.orchestra.run.gibba013:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:29:43.599 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2021-02-10T11:29:43.600 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:29:43.600 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:29:43.600 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:29:43.934 INFO:teuthology.orchestra.run.gibba002.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.0’: No such file or directory 2021-02-10T11:29:43.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:29:43.934 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.0 2021-02-10T11:29:43.934 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:29:43.935 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:29:44.268 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:29:44.269 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:29:44.269 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:29:44.604 INFO:teuthology.orchestra.run.gibba002.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.1’: No such file or directory 2021-02-10T11:29:44.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:29:44.604 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2021-02-10T11:29:44.604 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:29:44.605 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:29:44.940 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:29:44.940 DEBUG:teuthology.orchestra.run.gibba013:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:29:44.940 DEBUG:teuthology.orchestra.run.gibba013:> ' -- /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:29:45.275 INFO:teuthology.orchestra.run.gibba013.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.2’: No such file or directory 2021-02-10T11:29:45.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:29:45.275 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2021-02-10T11:29:45.276 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:29:45.276 DEBUG:teuthology.orchestra.run.gibba013:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:29:45.611 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:29:45.612 DEBUG:teuthology.orchestra.run.gibba013:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:29:45.612 DEBUG:teuthology.orchestra.run.gibba013:> ' -- /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:29:45.947 INFO:teuthology.orchestra.run.gibba013.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.3’: No such file or directory 2021-02-10T11:29:45.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:29:45.948 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.3 2021-02-10T11:29:45.948 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:29:45.948 DEBUG:teuthology.orchestra.run.gibba013:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:29:46.283 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config rm mon mon_allow_pool_delete 2021-02-10T11:29:46.713 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:46.696 7f4dd3127700 1 Processor -- start 2021-02-10T11:29:46.713 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:46.696 7f4dd3127700 1 -- start start 2021-02-10T11:29:46.713 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:46.696 7f4dd3127700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f4dcc0d0330 0x7f4dcc0ef630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:46.713 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:46.696 7f4dd3127700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f4dcc0d1c60 0x7f4dcc0efc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:46.713 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:46.696 7f4dd3127700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4dcc0f0b00 con 0x7f4dcc0d1860 2021-02-10T11:29:46.713 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:46.696 7f4dd3127700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4dcc0e5ec0 con 0x7f4dcc0d0330 2021-02-10T11:29:46.713 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:46.696 7f4dd3127700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4dcc1b0bd0 con 0x7f4dcc0d1c60 2021-02-10T11:29:46.713 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:46.697 7f4dcbfff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f4dcc0d0330 0x7f4dcc0ef630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:46.713 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:46.697 7f4dcbfff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f4dcc0d0330 0x7f4dcc0ef630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:60398/0 (socket says 172.21.2.102:60398) 2021-02-10T11:29:46.714 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:46.697 7f4dcbfff700 1 -- 172.21.2.102:0/494305101 learned_addr learned my addr 172.21.2.102:0/494305101 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:29:46.714 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:46.697 7f4dcbfff700 1 -- 172.21.2.102:0/494305101 >> v2:172.21.2.113:3300/0 conn(0x7f4dcc0d1c60 msgr2=0x7f4dcc0efc50 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:29:46.714 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:46.697 7f4dcbfff700 1 --2- 172.21.2.102:0/494305101 >> v2:172.21.2.113:3300/0 conn(0x7f4dcc0d1c60 0x7f4dcc0efc50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:46.714 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:46.697 7f4dcbfff700 1 -- 172.21.2.102:0/494305101 >> v1:172.21.2.102:6789/0 conn(0x7f4dcc0d1860 legacy=0x7f4dcc1b03e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:46.714 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:46.697 7f4dcbfff700 1 -- 172.21.2.102:0/494305101 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dcc0f2090 con 0x7f4dcc0d0330 2021-02-10T11:29:46.714 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:46.697 7f4dcbfff700 1 --2- 172.21.2.102:0/494305101 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4dcc0d0330 0x7f4dcc0ef630 secure :-1 s=READY pgs=1880 cs=0 l=1 rev1=1 rx=0x7f4dc0006700 tx=0x7f4dc0009b80).ready entity=mon.0 client_cookie=5182085919dd4eee server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:29:46.714 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:46.697 7f4dcb7fe700 1 -- 172.21.2.102:0/494305101 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4dc000c0c0 con 0x7f4dcc0d0330 2021-02-10T11:29:46.714 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:46.697 7f4dcb7fe700 1 -- 172.21.2.102:0/494305101 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4dc000c440 con 0x7f4dcc0d0330 2021-02-10T11:29:46.714 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:46.697 7f4dcb7fe700 1 -- 172.21.2.102:0/494305101 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4dc000c960 con 0x7f4dcc0d0330 2021-02-10T11:29:46.714 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:46.697 7f4dd3127700 1 -- 172.21.2.102:0/494305101 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4dcc0d0330 msgr2=0x7f4dcc0ef630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:46.714 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:46.697 7f4dd3127700 1 --2- 172.21.2.102:0/494305101 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4dcc0d0330 0x7f4dcc0ef630 secure :-1 s=READY pgs=1880 cs=0 l=1 rev1=1 rx=0x7f4dc0006700 tx=0x7f4dc0009b80).stop 2021-02-10T11:29:46.714 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:46.698 7f4dd3127700 1 -- 172.21.2.102:0/494305101 shutdown_connections 2021-02-10T11:29:46.715 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:46.698 7f4dd3127700 1 --2- 172.21.2.102:0/494305101 >> v2:172.21.2.113:3300/0 conn(0x7f4dcc0d1c60 0x7f4dcc0efc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:46.715 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:46.698 7f4dd3127700 1 --2- 172.21.2.102:0/494305101 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4dcc0d0330 0x7f4dcc0ef630 unknown :-1 s=CLOSED pgs=1880 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:46.715 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:46.698 7f4dd3127700 1 -- 172.21.2.102:0/494305101 >> 172.21.2.102:0/494305101 conn(0x7f4dcc0ce440 msgr2=0x7f4dcc0d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:29:46.715 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:46.698 7f4dd3127700 1 -- 172.21.2.102:0/494305101 shutdown_connections 2021-02-10T11:29:46.715 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:46.698 7f4dd3127700 1 -- 172.21.2.102:0/494305101 wait complete. 2021-02-10T11:29:46.716 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:46.713 7f4dd3127700 1 Processor -- start 2021-02-10T11:29:46.716 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:46.713 7f4dd3127700 1 -- start start 2021-02-10T11:29:46.716 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:46.713 7f4dd3127700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4dcc09f360 0x7f4dcc0b5000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:46.716 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:46.713 7f4dd3127700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4dcc08d7a0 0x7f4dcc08dba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:46.716 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:46.713 7f4dd3127700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4dcc08c430 0x7f4dcc08c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:46.717 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:46.713 7f4dd3127700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dcc0b5620 con 0x7f4dcc08c430 2021-02-10T11:29:46.717 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:46.713 7f4dd3127700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dcc08e1c0 con 0x7f4dcc08d7a0 2021-02-10T11:29:46.717 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:46.713 7f4dd3127700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4dcc09f870 con 0x7f4dcc09f360 2021-02-10T11:29:46.717 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:46.713 7f4dd1744700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4dcc08c430 0x7f4dcc08c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:46.717 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:46.713 7f4dd1744700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4dcc08c430 0x7f4dcc08c830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:60406/0 (socket says 172.21.2.102:60406) 2021-02-10T11:29:46.717 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:46.713 7f4dd1744700 1 -- 172.21.2.102:0/2798827345 learned_addr learned my addr 172.21.2.102:0/2798827345 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:29:46.717 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:46.714 7f4dd1744700 1 -- 172.21.2.102:0/2798827345 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4dcc09f360 msgr2=0x7f4dcc0b5000 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:29:46.717 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:46.714 7f4dd1744700 1 --2- 172.21.2.102:0/2798827345 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4dcc09f360 0x7f4dcc0b5000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:46.717 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:46.714 7f4dd1744700 1 -- 172.21.2.102:0/2798827345 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4dcc08d7a0 msgr2=0x7f4dcc08dba0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:29:46.717 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:46.714 7f4dd1744700 1 --2- 172.21.2.102:0/2798827345 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4dcc08d7a0 0x7f4dcc08dba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:46.717 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:46.714 7f4dd1744700 1 -- 172.21.2.102:0/2798827345 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dcc08ce50 con 0x7f4dcc08c430 2021-02-10T11:29:46.717 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:46.714 7f4dd1744700 1 --2- 172.21.2.102:0/2798827345 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4dcc08c430 0x7f4dcc08c830 secure :-1 s=READY pgs=1881 cs=0 l=1 rev1=1 rx=0x7f4dbc00b5c0 tx=0x7f4dbc00b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:29:46.718 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:46.714 7f4dc97fa700 1 -- 172.21.2.102:0/2798827345 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4dbc00bd50 con 0x7f4dcc08c430 2021-02-10T11:29:46.718 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:46.714 7f4dc97fa700 1 -- 172.21.2.102:0/2798827345 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4dbc006030 con 0x7f4dcc08c430 2021-02-10T11:29:46.718 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:46.714 7f4dc97fa700 1 -- 172.21.2.102:0/2798827345 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4dbc0054d0 con 0x7f4dcc08c430 2021-02-10T11:29:46.718 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:46.714 7f4dd3127700 1 -- 172.21.2.102:0/2798827345 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4dcc08ce50 con 0x7f4dcc08c430 2021-02-10T11:29:46.718 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:46.714 7f4dd3127700 1 -- 172.21.2.102:0/2798827345 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4dcc08ce50 con 0x7f4dcc08c430 2021-02-10T11:29:46.718 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:46.715 7f4dc97fa700 1 -- 172.21.2.102:0/2798827345 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4dbc006210 con 0x7f4dcc08c430 2021-02-10T11:29:46.718 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:46.715 7f4dc97fa700 1 --2- 172.21.2.102:0/2798827345 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4dac0186c0 0x7f4dac01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:46.718 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:46.715 7f4dc97fa700 1 -- 172.21.2.102:0/2798827345 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f4dbc032800 con 0x7f4dcc08c430 2021-02-10T11:29:46.718 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:46.715 7f4dd0f43700 1 --2- 172.21.2.102:0/2798827345 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4dac0186c0 0x7f4dac01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:46.718 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:46.715 7f4dd0f43700 1 --2- 172.21.2.102:0/2798827345 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4dac0186c0 0x7f4dac01abc0 crc :-1 s=READY pgs=844 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:29:46.747 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:46.746 7f4dd3127700 1 -- 172.21.2.102:0/2798827345 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4dcc08d270 con 0x7f4dcc08c430 2021-02-10T11:29:46.749 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:46.749 7f4dc97fa700 1 -- 172.21.2.102:0/2798827345 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f4dbc006b80 con 0x7f4dcc08c430 2021-02-10T11:29:46.897 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:46.897 7f4dd3127700 1 -- 172.21.2.102:0/2798827345 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f4dcc08d270 con 0x7f4dcc08c430 2021-02-10T11:29:46.901 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:46.901 7f4dc97fa700 1 -- 172.21.2.102:0/2798827345 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v23) v1 ==== 104+0+0 (secure 0 0 0) 0x7f4dbc006030 con 0x7f4dcc08c430 2021-02-10T11:29:46.906 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:46.905 7f4dd3127700 1 -- 172.21.2.102:0/2798827345 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4dac0186c0 msgr2=0x7f4dac01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:46.906 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:46.905 7f4dd3127700 1 --2- 172.21.2.102:0/2798827345 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4dac0186c0 0x7f4dac01abc0 crc :-1 s=READY pgs=844 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:46.906 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:46.905 7f4dd3127700 1 -- 172.21.2.102:0/2798827345 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4dcc08c430 msgr2=0x7f4dcc08c830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:46.906 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:46.905 7f4dd3127700 1 --2- 172.21.2.102:0/2798827345 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4dcc08c430 0x7f4dcc08c830 secure :-1 s=READY pgs=1881 cs=0 l=1 rev1=1 rx=0x7f4dbc00b5c0 tx=0x7f4dbc00b960).stop 2021-02-10T11:29:46.907 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:46.906 7f4dd3127700 1 -- 172.21.2.102:0/2798827345 shutdown_connections 2021-02-10T11:29:46.907 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:46.906 7f4dd3127700 1 --2- 172.21.2.102:0/2798827345 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4dcc08c430 0x7f4dcc08c830 unknown :-1 s=CLOSED pgs=1881 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:46.907 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:46.906 7f4dd3127700 1 --2- 172.21.2.102:0/2798827345 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4dac0186c0 0x7f4dac01abc0 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:46.907 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:46.906 7f4dd3127700 1 --2- 172.21.2.102:0/2798827345 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4dcc08d7a0 0x7f4dcc08dba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:46.907 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:46.906 7f4dd3127700 1 --2- 172.21.2.102:0/2798827345 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4dcc09f360 0x7f4dcc0b5000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:46.907 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:46.906 7f4dd3127700 1 -- 172.21.2.102:0/2798827345 >> 172.21.2.102:0/2798827345 conn(0x7f4dcc0a67e0 msgr2=0x7f4dcc094e60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:29:46.907 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:46.906 7f4dd3127700 1 -- 172.21.2.102:0/2798827345 shutdown_connections 2021-02-10T11:29:46.907 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:46.906 7f4dd3127700 1 -- 172.21.2.102:0/2798827345 wait complete. 2021-02-10T11:29:46.915 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestVolumes.test_connection_expiration' 2021-02-10T11:29:47.341 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:47.324 7fd78515a700 1 Processor -- start 2021-02-10T11:29:47.341 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:47.324 7fd78515a700 1 -- start start 2021-02-10T11:29:47.341 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:47.325 7fd78515a700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fd7800d0340 0x7fd7800ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:47.341 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:47.325 7fd78515a700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fd7800d1c70 0x7fd7800efc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:47.341 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:47.325 7fd78515a700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7800f0b10 con 0x7fd7800d1870 2021-02-10T11:29:47.341 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:47.325 7fd78515a700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7800e5ed0 con 0x7fd7800d1c70 2021-02-10T11:29:47.341 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:47.325 7fd78515a700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7801b0be0 con 0x7fd7800d0340 2021-02-10T11:29:47.341 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:47.325 7fd77f59e700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fd7800d1c70 0x7fd7800efc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:47.341 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:47.325 7fd77f59e700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fd7800d1c70 0x7fd7800efc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:60414/0 (socket says 172.21.2.102:60414) 2021-02-10T11:29:47.341 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:47.325 7fd77f59e700 1 -- 172.21.2.102:0/2117803022 learned_addr learned my addr 172.21.2.102:0/2117803022 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:29:47.341 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:47.325 7fd77f59e700 1 -- 172.21.2.102:0/2117803022 >> v2:172.21.2.113:3300/0 conn(0x7fd7800d0340 msgr2=0x7fd7800ef640 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:29:47.342 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:47.325 7fd77f59e700 1 --2- 172.21.2.102:0/2117803022 >> v2:172.21.2.113:3300/0 conn(0x7fd7800d0340 0x7fd7800ef640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:47.342 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:47.325 7fd77f59e700 1 -- 172.21.2.102:0/2117803022 >> v1:172.21.2.113:6789/0 conn(0x7fd7800d1870 legacy=0x7fd7801b03f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:29:47.342 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:47.325 7fd77f59e700 1 -- 172.21.2.102:0/2117803022 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7800f20a0 con 0x7fd7800d1c70 2021-02-10T11:29:47.342 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:47.325 7fd77f59e700 1 --2- 172.21.2.102:0/2117803022 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd7800d1c70 0x7fd7800efc60 secure :-1 s=READY pgs=1882 cs=0 l=1 rev1=1 rx=0x7fd778006f00 tx=0x7fd778009f10).ready entity=mon.0 client_cookie=7629c499896c6 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:29:47.342 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:47.325 7fd77dd9b700 1 -- 172.21.2.102:0/2117803022 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd77800b3b0 con 0x7fd7800d1c70 2021-02-10T11:29:47.342 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:47.326 7fd77dd9b700 1 -- 172.21.2.102:0/2117803022 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd77800b630 con 0x7fd7800d1c70 2021-02-10T11:29:47.342 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:47.326 7fd77dd9b700 1 -- 172.21.2.102:0/2117803022 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd77800bb10 con 0x7fd7800d1c70 2021-02-10T11:29:47.342 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:47.326 7fd78515a700 1 -- 172.21.2.102:0/2117803022 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd7800d1c70 msgr2=0x7fd7800efc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:47.342 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:47.326 7fd78515a700 1 --2- 172.21.2.102:0/2117803022 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd7800d1c70 0x7fd7800efc60 secure :-1 s=READY pgs=1882 cs=0 l=1 rev1=1 rx=0x7fd778006f00 tx=0x7fd778009f10).stop 2021-02-10T11:29:47.342 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:47.326 7fd78515a700 1 -- 172.21.2.102:0/2117803022 shutdown_connections 2021-02-10T11:29:47.342 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:47.326 7fd78515a700 1 --2- 172.21.2.102:0/2117803022 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd7800d1c70 0x7fd7800efc60 unknown :-1 s=CLOSED pgs=1882 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:47.342 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:47.326 7fd78515a700 1 --2- 172.21.2.102:0/2117803022 >> v2:172.21.2.113:3300/0 conn(0x7fd7800d0340 0x7fd7800ef640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:47.343 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:47.326 7fd78515a700 1 -- 172.21.2.102:0/2117803022 >> 172.21.2.102:0/2117803022 conn(0x7fd7800ce450 msgr2=0x7fd7800d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:29:47.343 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:47.326 7fd78515a700 1 -- 172.21.2.102:0/2117803022 shutdown_connections 2021-02-10T11:29:47.343 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:47.326 7fd78515a700 1 -- 172.21.2.102:0/2117803022 wait complete. 2021-02-10T11:29:47.344 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:47.340 7fd78515a700 1 Processor -- start 2021-02-10T11:29:47.344 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:47.341 7fd78515a700 1 -- start start 2021-02-10T11:29:47.344 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:47.341 7fd78515a700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd7800b04f0 0x7fd7801aed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:47.344 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:47.341 7fd78515a700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd7800b08f0 0x7fd7801af390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:47.344 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:47.341 7fd78515a700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd78009e740 0x7fd78009eb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:47.344 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:47.341 7fd78515a700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7801ac290 con 0x7fd7800b08f0 2021-02-10T11:29:47.344 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:47.341 7fd78515a700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7801ac5d0 con 0x7fd7800b04f0 2021-02-10T11:29:47.344 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:47.341 7fd78515a700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7801af9b0 con 0x7fd78009e740 2021-02-10T11:29:47.345 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:47.341 7fd77e59c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd7800b08f0 0x7fd7801af390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:47.345 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:47.341 7fd77e59c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd7800b08f0 0x7fd7801af390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:60420/0 (socket says 172.21.2.102:60420) 2021-02-10T11:29:47.345 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:47.341 7fd77e59c700 1 -- 172.21.2.102:0/2051581575 learned_addr learned my addr 172.21.2.102:0/2051581575 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:29:47.345 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:47.341 7fd77f59e700 1 --2- 172.21.2.102:0/2051581575 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd78009e740 0x7fd78009eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:47.345 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:47.341 7fd77ed9d700 1 --2- 172.21.2.102:0/2051581575 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd7800b04f0 0x7fd7801aed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:47.345 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:47.342 7fd77e59c700 1 -- 172.21.2.102:0/2051581575 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd78009e740 msgr2=0x7fd78009eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:47.345 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:47.342 7fd77e59c700 1 --2- 172.21.2.102:0/2051581575 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd78009e740 0x7fd78009eb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:47.345 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:47.342 7fd77e59c700 1 -- 172.21.2.102:0/2051581575 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd7800b04f0 msgr2=0x7fd7801aed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:47.345 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:47.342 7fd77e59c700 1 --2- 172.21.2.102:0/2051581575 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd7800b04f0 0x7fd7801aed70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:47.345 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:47.342 7fd77e59c700 1 -- 172.21.2.102:0/2051581575 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd780190960 con 0x7fd7800b08f0 2021-02-10T11:29:47.345 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:47.342 7fd77e59c700 1 --2- 172.21.2.102:0/2051581575 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd7800b08f0 0x7fd7801af390 secure :-1 s=READY pgs=1883 cs=0 l=1 rev1=1 rx=0x7fd77000b5c0 tx=0x7fd77000b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:29:47.345 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:47.342 7fd7777fe700 1 -- 172.21.2.102:0/2051581575 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd77000bd50 con 0x7fd7800b08f0 2021-02-10T11:29:47.345 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:47.342 7fd7777fe700 1 -- 172.21.2.102:0/2051581575 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd770006030 con 0x7fd7800b08f0 2021-02-10T11:29:47.346 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:47.342 7fd78515a700 1 -- 172.21.2.102:0/2051581575 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd780190960 con 0x7fd7800b08f0 2021-02-10T11:29:47.346 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:47.342 7fd7777fe700 1 -- 172.21.2.102:0/2051581575 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd7700054d0 con 0x7fd7800b08f0 2021-02-10T11:29:47.346 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:47.342 7fd78515a700 1 -- 172.21.2.102:0/2051581575 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd780190960 con 0x7fd7800b08f0 2021-02-10T11:29:47.346 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:47.343 7fd7777fe700 1 -- 172.21.2.102:0/2051581575 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd770006210 con 0x7fd7800b08f0 2021-02-10T11:29:47.346 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:47.343 7fd7777fe700 1 --2- 172.21.2.102:0/2051581575 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd764018730 0x7fd76401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:47.346 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:47.343 7fd7777fe700 1 -- 172.21.2.102:0/2051581575 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fd770032640 con 0x7fd7800b08f0 2021-02-10T11:29:47.346 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:47.343 7fd77ed9d700 1 --2- 172.21.2.102:0/2051581575 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd764018730 0x7fd76401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:47.346 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:47.343 7fd77ed9d700 1 --2- 172.21.2.102:0/2051581575 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd764018730 0x7fd76401ac10 crc :-1 s=READY pgs=845 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:29:47.375 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:47.375 7fd78515a700 1 -- 172.21.2.102:0/2051581575 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7800edd90 con 0x7fd7800b08f0 2021-02-10T11:29:47.378 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:47.377 7fd7777fe700 1 -- 172.21.2.102:0/2051581575 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd770006b80 con 0x7fd7800b08f0 2021-02-10T11:29:47.510 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:47.509 7fd78515a700 1 -- 172.21.2.102:0/2051581575 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestVolumes.test_connection_expiration"]} v 0) v1 -- 0x7fd7800edd90 con 0x7fd7800b08f0 2021-02-10T11:29:48.081 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:48.080 7fd7777fe700 1 -- 172.21.2.102:0/2051581575 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestVolumes.test_connection_expiration"]}]=0 v486) v1 ==== 143+0+0 (secure 0 0 0) 0x7fd770006030 con 0x7fd7800b08f0 2021-02-10T11:29:48.129 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:48.128 7fd78515a700 1 -- 172.21.2.102:0/2051581575 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd764018730 msgr2=0x7fd76401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:48.129 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:48.128 7fd78515a700 1 --2- 172.21.2.102:0/2051581575 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd764018730 0x7fd76401ac10 crc :-1 s=READY pgs=845 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:48.129 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:48.128 7fd78515a700 1 -- 172.21.2.102:0/2051581575 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd7800b08f0 msgr2=0x7fd7801af390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:48.129 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:48.128 7fd78515a700 1 --2- 172.21.2.102:0/2051581575 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd7800b08f0 0x7fd7801af390 secure :-1 s=READY pgs=1883 cs=0 l=1 rev1=1 rx=0x7fd77000b5c0 tx=0x7fd77000b960).stop 2021-02-10T11:29:48.129 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:48.128 7fd78515a700 1 -- 172.21.2.102:0/2051581575 shutdown_connections 2021-02-10T11:29:48.130 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:48.128 7fd78515a700 1 --2- 172.21.2.102:0/2051581575 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd78009e740 0x7fd78009eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:48.130 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:48.128 7fd78515a700 1 --2- 172.21.2.102:0/2051581575 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd7800b08f0 0x7fd7801af390 unknown :-1 s=CLOSED pgs=1883 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:48.130 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:48.128 7fd78515a700 1 --2- 172.21.2.102:0/2051581575 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd764018730 0x7fd76401ac10 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:48.130 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:48.128 7fd78515a700 1 --2- 172.21.2.102:0/2051581575 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd7800b04f0 0x7fd7801aed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:48.130 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:48.128 7fd78515a700 1 -- 172.21.2.102:0/2051581575 >> 172.21.2.102:0/2051581575 conn(0x7fd78019e650 msgr2=0x7fd7801b35c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:29:48.130 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:48.128 7fd78515a700 1 -- 172.21.2.102:0/2051581575 shutdown_connections 2021-02-10T11:29:48.130 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:48.128 7fd78515a700 1 -- 172.21.2.102:0/2051581575 wait complete. 2021-02-10T11:29:48.137 INFO:tasks.cephfs_test_runner:test_connection_expiration (tasks.cephfs.test_volumes.TestVolumes) ... ok 2021-02-10T11:29:48.137 INFO:tasks.cephfs_test_runner:Starting test: test_deauthorize_auth_id_after_out_of_band_update (tasks.cephfs.test_volumes.TestVolumes) 2021-02-10T11:29:48.137 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestVolumes.test_deauthorize_auth_id_after_out_of_band_update' 2021-02-10T11:29:48.564 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:48.548 7f7f2b66b700 1 Processor -- start 2021-02-10T11:29:48.564 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:48.548 7f7f2b66b700 1 -- start start 2021-02-10T11:29:48.565 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:48.548 7f7f2b66b700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7f240d1870 0x7f7f241b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:48.565 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:48.548 7f7f2b66b700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f7f240f0650 0x7f7f240cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:48.565 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:48.548 7f7f2b66b700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f240f0b10 con 0x7f7f241b0a10 2021-02-10T11:29:48.565 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:48.548 7f7f2b66b700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f240fe230 con 0x7f7f240f0650 2021-02-10T11:29:48.565 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:48.548 7f7f2b66b700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f240e5ed0 con 0x7f7f240d1870 2021-02-10T11:29:48.565 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:48.548 7f7f29c88700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f7f240f0650 0x7f7f240cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:48.565 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:48.548 7f7f29c88700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f7f240f0650 0x7f7f240cfb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:60424/0 (socket says 172.21.2.102:60424) 2021-02-10T11:29:48.565 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:48.548 7f7f29c88700 1 -- 172.21.2.102:0/2514396997 learned_addr learned my addr 172.21.2.102:0/2514396997 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:29:48.565 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:48.549 7f7f29c88700 1 -- 172.21.2.102:0/2514396997 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7f240d1870 msgr2=0x7f7f241b03f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:29:48.565 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:48.549 7f7f29c88700 1 --2- 172.21.2.102:0/2514396997 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7f240d1870 0x7f7f241b03f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:48.565 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:48.549 7f7f29c88700 1 -- 172.21.2.102:0/2514396997 >> v1:172.21.2.113:6789/0 conn(0x7f7f241b0a10 legacy=0x7f7f240cf330 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:29:48.565 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:48.549 7f7f29c88700 1 -- 172.21.2.102:0/2514396997 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f240f20a0 con 0x7f7f240f0650 2021-02-10T11:29:48.565 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:48.549 7f7f29c88700 1 --2- 172.21.2.102:0/2514396997 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7f240f0650 0x7f7f240cfb20 secure :-1 s=READY pgs=1884 cs=0 l=1 rev1=1 rx=0x7f7f1c006f00 tx=0x7f7f1c009f10).ready entity=mon.0 client_cookie=9dfca232f2096957 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:29:48.566 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:48.549 7f7f1bfff700 1 -- 172.21.2.102:0/2514396997 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7f1c00b3b0 con 0x7f7f240f0650 2021-02-10T11:29:48.566 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:48.549 7f7f1bfff700 1 -- 172.21.2.102:0/2514396997 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f1c00b730 con 0x7f7f240f0650 2021-02-10T11:29:48.566 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:48.549 7f7f1bfff700 1 -- 172.21.2.102:0/2514396997 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7f1c00bc10 con 0x7f7f240f0650 2021-02-10T11:29:48.566 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:48.549 7f7f2b66b700 1 -- 172.21.2.102:0/2514396997 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7f240f0650 msgr2=0x7f7f240cfb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:48.566 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:48.549 7f7f2b66b700 1 --2- 172.21.2.102:0/2514396997 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7f240f0650 0x7f7f240cfb20 secure :-1 s=READY pgs=1884 cs=0 l=1 rev1=1 rx=0x7f7f1c006f00 tx=0x7f7f1c009f10).stop 2021-02-10T11:29:48.566 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:48.550 7f7f2b66b700 1 -- 172.21.2.102:0/2514396997 shutdown_connections 2021-02-10T11:29:48.566 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:48.550 7f7f2b66b700 1 --2- 172.21.2.102:0/2514396997 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7f240f0650 0x7f7f240cfb20 unknown :-1 s=CLOSED pgs=1884 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:48.566 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:48.550 7f7f2b66b700 1 --2- 172.21.2.102:0/2514396997 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7f240d1870 0x7f7f241b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:48.566 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:48.550 7f7f2b66b700 1 -- 172.21.2.102:0/2514396997 >> 172.21.2.102:0/2514396997 conn(0x7f7f240ce450 msgr2=0x7f7f240d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:29:48.566 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:48.550 7f7f2b66b700 1 -- 172.21.2.102:0/2514396997 shutdown_connections 2021-02-10T11:29:48.566 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:48.550 7f7f2b66b700 1 -- 172.21.2.102:0/2514396997 wait complete. 2021-02-10T11:29:48.568 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:48.564 7f7f2b66b700 1 Processor -- start 2021-02-10T11:29:48.568 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:48.565 7f7f2b66b700 1 -- start start 2021-02-10T11:29:48.568 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:48.565 7f7f2b66b700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7f240b4f30 0x7f7f2418b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:48.568 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:48.565 7f7f2b66b700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7f24093f40 0x7f7f24094340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:48.568 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:48.565 7f7f2b66b700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7f24094960 0x7f7f24187e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:48.568 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:48.565 7f7f2b66b700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f24094e90 con 0x7f7f24094960 2021-02-10T11:29:48.568 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:48.565 7f7f2b66b700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f2418bff0 con 0x7f7f24093f40 2021-02-10T11:29:48.568 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:48.565 7f7f2b66b700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f2408f780 con 0x7f7f240b4f30 2021-02-10T11:29:48.569 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:48.565 7f7f29c88700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7f24094960 0x7f7f24187e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:48.569 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:48.565 7f7f29c88700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7f24094960 0x7f7f24187e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:60432/0 (socket says 172.21.2.102:60432) 2021-02-10T11:29:48.569 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:48.565 7f7f29c88700 1 -- 172.21.2.102:0/3721053265 learned_addr learned my addr 172.21.2.102:0/3721053265 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:29:48.569 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:48.566 7f7f29c88700 1 -- 172.21.2.102:0/3721053265 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7f240b4f30 msgr2=0x7f7f2418b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:48.569 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:48.566 7f7f29487700 1 --2- 172.21.2.102:0/3721053265 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7f240b4f30 0x7f7f2418b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:48.569 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:48.566 7f7f29c88700 1 --2- 172.21.2.102:0/3721053265 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7f240b4f30 0x7f7f2418b9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:48.569 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:48.566 7f7f29c88700 1 -- 172.21.2.102:0/3721053265 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7f24093f40 msgr2=0x7f7f24094340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:48.569 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:48.566 7f7f29c88700 1 --2- 172.21.2.102:0/3721053265 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7f24093f40 0x7f7f24094340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:48.569 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:48.566 7f7f29c88700 1 -- 172.21.2.102:0/3721053265 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f24188470 con 0x7f7f24094960 2021-02-10T11:29:48.569 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:48.566 7f7f29c88700 1 --2- 172.21.2.102:0/3721053265 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7f24094960 0x7f7f24187e50 secure :-1 s=READY pgs=1885 cs=0 l=1 rev1=1 rx=0x7f7f1c009390 tx=0x7f7f1c00bd40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:29:48.569 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:48.566 7f7f19ffb700 1 -- 172.21.2.102:0/3721053265 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7f1c003030 con 0x7f7f24094960 2021-02-10T11:29:48.569 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:48.566 7f7f19ffb700 1 -- 172.21.2.102:0/3721053265 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f1c0068e0 con 0x7f7f24094960 2021-02-10T11:29:48.569 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:48.566 7f7f2b66b700 1 -- 172.21.2.102:0/3721053265 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f24188470 con 0x7f7f24094960 2021-02-10T11:29:48.569 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:48.566 7f7f2b66b700 1 -- 172.21.2.102:0/3721053265 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f24188a00 con 0x7f7f24094960 2021-02-10T11:29:48.570 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:48.566 7f7f19ffb700 1 -- 172.21.2.102:0/3721053265 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7f1c00c570 con 0x7f7f24094960 2021-02-10T11:29:48.570 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:48.567 7f7f19ffb700 1 -- 172.21.2.102:0/3721053265 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f7f1c001270 con 0x7f7f24094960 2021-02-10T11:29:48.570 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:48.567 7f7f19ffb700 1 --2- 172.21.2.102:0/3721053265 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7f040186c0 0x7f7f0401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:48.570 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:48.567 7f7f19ffb700 1 -- 172.21.2.102:0/3721053265 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f7f1c032b40 con 0x7f7f24094960 2021-02-10T11:29:48.570 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:48.567 7f7f29487700 1 --2- 172.21.2.102:0/3721053265 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7f040186c0 0x7f7f0401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:48.570 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:48.568 7f7f29487700 1 --2- 172.21.2.102:0/3721053265 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7f040186c0 0x7f7f0401abc0 crc :-1 s=READY pgs=846 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:29:48.599 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:48.598 7f7f2b66b700 1 -- 172.21.2.102:0/3721053265 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f2408f760 con 0x7f7f24094960 2021-02-10T11:29:48.601 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:48.601 7f7f19ffb700 1 -- 172.21.2.102:0/3721053265 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f7f1c009450 con 0x7f7f24094960 2021-02-10T11:29:48.733 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:48.733 7f7f2b66b700 1 -- 172.21.2.102:0/3721053265 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestVolumes.test_deauthorize_auth_id_after_out_of_band_update"]} v 0) v1 -- 0x7f7f2408f760 con 0x7f7f24094960 2021-02-10T11:29:49.084 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:49.083 7f7f19ffb700 1 -- 172.21.2.102:0/3721053265 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestVolumes.test_deauthorize_auth_id_after_out_of_band_update"]}]=0 v487) v1 ==== 169+0+0 (secure 0 0 0) 0x7f7f1c0068e0 con 0x7f7f24094960 2021-02-10T11:29:49.101 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:49.101 7f7f2b66b700 1 -- 172.21.2.102:0/3721053265 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7f040186c0 msgr2=0x7f7f0401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:49.101 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:49.101 7f7f2b66b700 1 --2- 172.21.2.102:0/3721053265 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7f040186c0 0x7f7f0401abc0 crc :-1 s=READY pgs=846 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:49.101 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:49.101 7f7f2b66b700 1 -- 172.21.2.102:0/3721053265 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7f24094960 msgr2=0x7f7f24187e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:49.101 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:49.101 7f7f2b66b700 1 --2- 172.21.2.102:0/3721053265 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7f24094960 0x7f7f24187e50 secure :-1 s=READY pgs=1885 cs=0 l=1 rev1=1 rx=0x7f7f1c009390 tx=0x7f7f1c00bd40).stop 2021-02-10T11:29:49.102 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:49.101 7f7f2b66b700 1 -- 172.21.2.102:0/3721053265 shutdown_connections 2021-02-10T11:29:49.102 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:49.101 7f7f2b66b700 1 --2- 172.21.2.102:0/3721053265 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7f24094960 0x7f7f24187e50 unknown :-1 s=CLOSED pgs=1885 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:49.102 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:49.101 7f7f2b66b700 1 --2- 172.21.2.102:0/3721053265 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7f040186c0 0x7f7f0401abc0 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:49.102 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:49.101 7f7f2b66b700 1 --2- 172.21.2.102:0/3721053265 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7f24093f40 0x7f7f24094340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:49.102 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:49.101 7f7f2b66b700 1 --2- 172.21.2.102:0/3721053265 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7f240b4f30 0x7f7f2418b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:49.102 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:49.101 7f7f2b66b700 1 -- 172.21.2.102:0/3721053265 >> 172.21.2.102:0/3721053265 conn(0x7f7f2419f8f0 msgr2=0x7f7f241a5f80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:29:49.102 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:49.101 7f7f2b66b700 1 -- 172.21.2.102:0/3721053265 shutdown_connections 2021-02-10T11:29:49.102 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:49.101 7f7f2b66b700 1 -- 172.21.2.102:0/3721053265 wait complete. 2021-02-10T11:29:49.109 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config set mon mon_allow_pool_delete True 2021-02-10T11:29:49.543 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:49.524 7f46507d6700 1 Processor -- start 2021-02-10T11:29:49.543 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:49.524 7f46507d6700 1 -- start start 2021-02-10T11:29:49.543 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:49.525 7f46507d6700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f464810aff0 0x7f46480cf750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:49.544 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:49.525 7f46507d6700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46480fe1c0 con 0x7f46480d1c70 2021-02-10T11:29:49.544 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:49.525 7f46507d6700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46481b0cb0 con 0x7f46480d1870 2021-02-10T11:29:49.544 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:49.525 7f46507d6700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4648109a60 con 0x7f464810aff0 2021-02-10T11:29:49.544 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:49.525 7f464edf3700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f464810aff0 0x7f46480cf750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:49.544 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:49.525 7f464edf3700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f464810aff0 0x7f46480cf750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:60440/0 (socket says 172.21.2.102:60440) 2021-02-10T11:29:49.544 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:49.525 7f464edf3700 1 -- 172.21.2.102:0/2576773059 learned_addr learned my addr 172.21.2.102:0/2576773059 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:29:49.544 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:49.525 7f464edf3700 1 -- 172.21.2.102:0/2576773059 >> v1:172.21.2.113:6789/0 conn(0x7f46480d1870 legacy=0x7f46481b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:49.544 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:49.525 7f464edf3700 1 -- 172.21.2.102:0/2576773059 >> v1:172.21.2.102:6789/0 conn(0x7f46480d1c70 legacy=0x7f46480cef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:49.544 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:49.525 7f464d5f0700 1 -- 172.21.2.102:0/2576773059 <== mon.1 v1:172.21.2.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3542278823 0 0) 0x7f463c000ff0 con 0x7f46480d1870 2021-02-10T11:29:49.544 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:49.526 7f464edf3700 1 -- 172.21.2.102:0/2576773059 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f464810b3f0 con 0x7f464810aff0 2021-02-10T11:29:49.544 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:49.526 7f464edf3700 1 --2- 172.21.2.102:0/2576773059 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f464810aff0 0x7f46480cf750 secure :-1 s=READY pgs=1887 cs=0 l=1 rev1=1 rx=0x7f4640006f00 tx=0x7f4640009e90).ready entity=mon.0 client_cookie=9cbfde147e1d1bc1 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:29:49.544 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:49.526 7f464d5f0700 1 -- 172.21.2.102:0/2576773059 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f464000b300 con 0x7f464810aff0 2021-02-10T11:29:49.544 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:49.526 7f464d5f0700 1 -- 172.21.2.102:0/2576773059 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f464000b300 con 0x7f464810aff0 2021-02-10T11:29:49.545 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:49.526 7f464d5f0700 1 -- 172.21.2.102:0/2576773059 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f464000b580 con 0x7f464810aff0 2021-02-10T11:29:49.545 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:49.526 7f46507d6700 1 -- 172.21.2.102:0/2576773059 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f464810aff0 msgr2=0x7f46480cf750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:49.545 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:49.526 7f46507d6700 1 --2- 172.21.2.102:0/2576773059 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f464810aff0 0x7f46480cf750 secure :-1 s=READY pgs=1887 cs=0 l=1 rev1=1 rx=0x7f4640006f00 tx=0x7f4640009e90).stop 2021-02-10T11:29:49.545 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:49.526 7f46507d6700 1 -- 172.21.2.102:0/2576773059 shutdown_connections 2021-02-10T11:29:49.545 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:49.526 7f46507d6700 1 --2- 172.21.2.102:0/2576773059 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f464810aff0 0x7f46480cf750 unknown :-1 s=CLOSED pgs=1887 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:49.545 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:49.527 7f46507d6700 1 -- 172.21.2.102:0/2576773059 >> 172.21.2.102:0/2576773059 conn(0x7f46480ce420 msgr2=0x7f46480d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:29:49.545 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:49.527 7f46507d6700 1 -- 172.21.2.102:0/2576773059 shutdown_connections 2021-02-10T11:29:49.545 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:49.527 7f46507d6700 1 -- 172.21.2.102:0/2576773059 wait complete. 2021-02-10T11:29:49.545 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:49.541 7f46507d6700 1 Processor -- start 2021-02-10T11:29:49.545 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:49.541 7f46507d6700 1 -- start start 2021-02-10T11:29:49.545 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:49.542 7f46507d6700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f46480c3820 0x7f46480a7950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:49.545 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:49.542 7f46507d6700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f46481a44c0 0x7f46481a48c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:49.545 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:49.542 7f46507d6700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f46480863d0 0x7f46480867d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:49.546 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:49.542 7f46507d6700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4648085ed0 con 0x7f46481a44c0 2021-02-10T11:29:49.546 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:49.542 7f46507d6700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46480c0740 con 0x7f46480863d0 2021-02-10T11:29:49.546 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:49.542 7f46507d6700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4648086df0 con 0x7f46480c3820 2021-02-10T11:29:49.546 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:49.542 7f464ddf1700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f46481a44c0 0x7f46481a48c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:49.546 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:49.542 7f464ddf1700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f46481a44c0 0x7f46481a48c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:60442/0 (socket says 172.21.2.102:60442) 2021-02-10T11:29:49.546 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:49.542 7f464ddf1700 1 -- 172.21.2.102:0/434565649 learned_addr learned my addr 172.21.2.102:0/434565649 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:29:49.546 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:49.542 7f464e5f2700 1 --2- 172.21.2.102:0/434565649 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f46480c3820 0x7f46480a7950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:49.546 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:49.542 7f464edf3700 1 --2- 172.21.2.102:0/434565649 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f46480863d0 0x7f46480867d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:49.546 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:49.542 7f464ddf1700 1 -- 172.21.2.102:0/434565649 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f46480c3820 msgr2=0x7f46480a7950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:49.546 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:49.542 7f464ddf1700 1 --2- 172.21.2.102:0/434565649 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f46480c3820 0x7f46480a7950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:49.546 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:49.542 7f464ddf1700 1 -- 172.21.2.102:0/434565649 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f46480863d0 msgr2=0x7f46480867d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:49.546 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:49.542 7f464ddf1700 1 --2- 172.21.2.102:0/434565649 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f46480863d0 0x7f46480867d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:49.547 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:49.542 7f464ddf1700 1 -- 172.21.2.102:0/434565649 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4648087010 con 0x7f46481a44c0 2021-02-10T11:29:49.547 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:49.542 7f464edf3700 1 --2- 172.21.2.102:0/434565649 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f46480863d0 0x7f46480867d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:29:49.547 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:49.542 7f464ddf1700 1 --2- 172.21.2.102:0/434565649 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f46481a44c0 0x7f46481a48c0 secure :-1 s=READY pgs=1888 cs=0 l=1 rev1=1 rx=0x7f464400a550 tx=0x7f464400a8f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:29:49.547 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:49.542 7f463affd700 1 -- 172.21.2.102:0/434565649 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f464400b070 con 0x7f46481a44c0 2021-02-10T11:29:49.547 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:49.543 7f463affd700 1 -- 172.21.2.102:0/434565649 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f464400b3f0 con 0x7f46481a44c0 2021-02-10T11:29:49.547 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:49.543 7f463affd700 1 -- 172.21.2.102:0/434565649 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f464400b8d0 con 0x7f46481a44c0 2021-02-10T11:29:49.547 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:49.543 7f46507d6700 1 -- 172.21.2.102:0/434565649 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4648087010 con 0x7f46481a44c0 2021-02-10T11:29:49.547 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:49.543 7f46507d6700 1 -- 172.21.2.102:0/434565649 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4648083000 con 0x7f46481a44c0 2021-02-10T11:29:49.548 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:49.543 7f463affd700 1 -- 172.21.2.102:0/434565649 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4644000d40 con 0x7f46481a44c0 2021-02-10T11:29:49.548 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:49.543 7f463affd700 1 --2- 172.21.2.102:0/434565649 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f46300186c0 0x7f463001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:49.548 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:49.543 7f463affd700 1 -- 172.21.2.102:0/434565649 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f46440323c0 con 0x7f46481a44c0 2021-02-10T11:29:49.548 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:49.543 7f464e5f2700 1 --2- 172.21.2.102:0/434565649 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f46300186c0 0x7f463001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:49.548 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:49.544 7f464e5f2700 1 --2- 172.21.2.102:0/434565649 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f46300186c0 0x7f463001abc0 crc :-1 s=READY pgs=847 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:29:49.575 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:49.575 7f46507d6700 1 -- 172.21.2.102:0/434565649 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46480fe1c0 con 0x7f46481a44c0 2021-02-10T11:29:49.577 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:49.577 7f463affd700 1 -- 172.21.2.102:0/434565649 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f46440009b0 con 0x7f46481a44c0 2021-02-10T11:29:49.727 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:49.726 7f46507d6700 1 -- 172.21.2.102:0/434565649 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f46480fe1c0 con 0x7f46481a44c0 2021-02-10T11:29:49.731 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:49.730 7f463affd700 1 -- 172.21.2.102:0/434565649 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v24)=0 v24) v1 ==== 122+0+0 (secure 0 0 0) 0x7f4644000d40 con 0x7f46481a44c0 2021-02-10T11:29:49.736 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:49.735 7f46507d6700 1 -- 172.21.2.102:0/434565649 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f46300186c0 msgr2=0x7f463001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:49.736 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:49.735 7f46507d6700 1 --2- 172.21.2.102:0/434565649 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f46300186c0 0x7f463001abc0 crc :-1 s=READY pgs=847 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:49.736 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:49.735 7f46507d6700 1 -- 172.21.2.102:0/434565649 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f46481a44c0 msgr2=0x7f46481a48c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:49.736 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:49.735 7f46507d6700 1 --2- 172.21.2.102:0/434565649 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f46481a44c0 0x7f46481a48c0 secure :-1 s=READY pgs=1888 cs=0 l=1 rev1=1 rx=0x7f464400a550 tx=0x7f464400a8f0).stop 2021-02-10T11:29:49.736 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:49.735 7f46507d6700 1 -- 172.21.2.102:0/434565649 shutdown_connections 2021-02-10T11:29:49.736 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:49.735 7f46507d6700 1 --2- 172.21.2.102:0/434565649 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f46300186c0 0x7f463001abc0 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:49.736 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:49.735 7f46507d6700 1 --2- 172.21.2.102:0/434565649 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f46480863d0 0x7f46480867d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:49.736 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:49.735 7f46507d6700 1 --2- 172.21.2.102:0/434565649 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f46481a44c0 0x7f46481a48c0 unknown :-1 s=CLOSED pgs=1888 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:49.737 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:49.735 7f46507d6700 1 --2- 172.21.2.102:0/434565649 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f46480c3820 0x7f46480a7950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:49.737 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:49.735 7f46507d6700 1 -- 172.21.2.102:0/434565649 >> 172.21.2.102:0/434565649 conn(0x7f46480872c0 msgr2=0x7f4648095700 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:29:49.737 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:49.735 7f46507d6700 1 -- 172.21.2.102:0/434565649 shutdown_connections 2021-02-10T11:29:49.737 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:49.735 7f46507d6700 1 -- 172.21.2.102:0/434565649 wait complete. 2021-02-10T11:29:49.744 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2021-02-10T11:29:49.744 DEBUG:teuthology.orchestra.run.gibba002:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:29:50.091 DEBUG:teuthology.orchestra.run.gibba013:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:29:50.437 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2021-02-10T11:29:50.438 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:29:50.438 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:29:50.438 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:29:50.773 INFO:teuthology.orchestra.run.gibba002.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.0’: No such file or directory 2021-02-10T11:29:50.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:29:50.773 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.0 2021-02-10T11:29:50.773 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:29:50.773 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:29:50.773 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:29:51.108 INFO:teuthology.orchestra.run.gibba002.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.1’: No such file or directory 2021-02-10T11:29:51.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:29:51.108 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2021-02-10T11:29:51.108 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:29:51.108 DEBUG:teuthology.orchestra.run.gibba013:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:29:51.109 DEBUG:teuthology.orchestra.run.gibba013:> ' -- /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:29:51.444 INFO:teuthology.orchestra.run.gibba013.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.2’: No such file or directory 2021-02-10T11:29:51.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:29:51.445 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2021-02-10T11:29:51.445 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:29:51.445 DEBUG:teuthology.orchestra.run.gibba013:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:29:51.446 DEBUG:teuthology.orchestra.run.gibba013:> ' -- /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:29:51.779 INFO:teuthology.orchestra.run.gibba013.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.3’: No such file or directory 2021-02-10T11:29:51.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:29:51.779 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.3 2021-02-10T11:29:51.780 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json-pretty 2021-02-10T11:29:52.206 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:52.190 7fbaa888d700 1 Processor -- start 2021-02-10T11:29:52.207 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:52.190 7fbaa888d700 1 -- start start 2021-02-10T11:29:52.207 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:52.190 7fbaa888d700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fbaa40d1870 0x7fbaa41b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:52.207 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:52.190 7fbaa888d700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fbaa41b0a10 0x7fbaa40cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:52.207 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:52.190 7fbaa888d700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbaa40f0b10 con 0x7fbaa40d1c70 2021-02-10T11:29:52.207 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:52.190 7fbaa888d700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbaa40e5ed0 con 0x7fbaa41b0a10 2021-02-10T11:29:52.207 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:52.190 7fbaa888d700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbaa40f0650 con 0x7fbaa40d1870 2021-02-10T11:29:52.207 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:52.190 7fbaa1e1b700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fbaa41b0a10 0x7fbaa40cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:52.207 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:52.190 7fbaa1e1b700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fbaa41b0a10 0x7fbaa40cf330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:60450/0 (socket says 172.21.2.102:60450) 2021-02-10T11:29:52.207 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:52.190 7fbaa1e1b700 1 -- 172.21.2.102:0/1968010592 learned_addr learned my addr 172.21.2.102:0/1968010592 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:29:52.207 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:52.191 7fbaa1e1b700 1 -- 172.21.2.102:0/1968010592 >> v2:172.21.2.113:3300/0 conn(0x7fbaa40d1870 msgr2=0x7fbaa41b03f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:29:52.207 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:52.191 7fbaa1e1b700 1 --2- 172.21.2.102:0/1968010592 >> v2:172.21.2.113:3300/0 conn(0x7fbaa40d1870 0x7fbaa41b03f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:52.208 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:52.191 7fbaa1e1b700 1 -- 172.21.2.102:0/1968010592 >> v1:172.21.2.113:6789/0 conn(0x7fbaa40d1c70 legacy=0x7fbaa40cf950 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:29:52.208 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:52.191 7fbaa1e1b700 1 -- 172.21.2.102:0/1968010592 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbaa40f0840 con 0x7fbaa41b0a10 2021-02-10T11:29:52.208 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:52.191 7fbaa1e1b700 1 --2- 172.21.2.102:0/1968010592 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbaa41b0a10 0x7fbaa40cf330 secure :-1 s=READY pgs=1889 cs=0 l=1 rev1=1 rx=0x7fba8c006700 tx=0x7fba8c009b00).ready entity=mon.0 client_cookie=825c912f0d0f55d9 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:29:52.208 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:52.191 7fbaa161a700 1 -- 172.21.2.102:0/1968010592 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fba8c00c0c0 con 0x7fbaa41b0a10 2021-02-10T11:29:52.208 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:52.191 7fbaa161a700 1 -- 172.21.2.102:0/1968010592 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba8c00c400 con 0x7fbaa41b0a10 2021-02-10T11:29:52.208 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:52.191 7fbaa161a700 1 -- 172.21.2.102:0/1968010592 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fba8c00c920 con 0x7fbaa41b0a10 2021-02-10T11:29:52.208 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:52.191 7fbaa888d700 1 -- 172.21.2.102:0/1968010592 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbaa41b0a10 msgr2=0x7fbaa40cf330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:52.208 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:52.191 7fbaa888d700 1 --2- 172.21.2.102:0/1968010592 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbaa41b0a10 0x7fbaa40cf330 secure :-1 s=READY pgs=1889 cs=0 l=1 rev1=1 rx=0x7fba8c006700 tx=0x7fba8c009b00).stop 2021-02-10T11:29:52.208 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:52.192 7fbaa888d700 1 -- 172.21.2.102:0/1968010592 shutdown_connections 2021-02-10T11:29:52.208 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:52.192 7fbaa888d700 1 --2- 172.21.2.102:0/1968010592 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbaa41b0a10 0x7fbaa40cf330 unknown :-1 s=CLOSED pgs=1889 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:52.208 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:52.192 7fbaa888d700 1 --2- 172.21.2.102:0/1968010592 >> v2:172.21.2.113:3300/0 conn(0x7fbaa40d1870 0x7fbaa41b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:52.208 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:52.192 7fbaa888d700 1 -- 172.21.2.102:0/1968010592 >> 172.21.2.102:0/1968010592 conn(0x7fbaa40ce450 msgr2=0x7fbaa40d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:29:52.208 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:52.192 7fbaa888d700 1 -- 172.21.2.102:0/1968010592 shutdown_connections 2021-02-10T11:29:52.209 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:52.192 7fbaa888d700 1 -- 172.21.2.102:0/1968010592 wait complete. 2021-02-10T11:29:52.211 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:52.206 7fbaa888d700 1 Processor -- start 2021-02-10T11:29:52.211 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:52.207 7fbaa888d700 1 -- start start 2021-02-10T11:29:52.211 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:52.207 7fbaa888d700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbaa4195000 0x7fbaa40b7750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:52.211 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:52.207 7fbaa888d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbaa40b7d70 0x7fbaa40b8170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:52.211 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:52.207 7fbaa888d700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fbaa41a2770 0x7fbaa41a2b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:52.211 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:52.207 7fbaa888d700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbaa41a3c20 con 0x7fbaa40b7d70 2021-02-10T11:29:52.211 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:52.207 7fbaa888d700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbaa41a3190 con 0x7fbaa41a2770 2021-02-10T11:29:52.211 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:52.207 7fbaa888d700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbaa41a33b0 con 0x7fbaa4195000 2021-02-10T11:29:52.211 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:52.207 7fbaa1e1b700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbaa40b7d70 0x7fbaa40b8170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:52.211 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:52.207 7fbaa1e1b700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbaa40b7d70 0x7fbaa40b8170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:60458/0 (socket says 172.21.2.102:60458) 2021-02-10T11:29:52.211 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:52.207 7fbaa1e1b700 1 -- 172.21.2.102:0/2543870723 learned_addr learned my addr 172.21.2.102:0/2543870723 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:29:52.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:52.207 7fbaa261c700 1 --2- 172.21.2.102:0/2543870723 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbaa4195000 0x7fbaa40b7750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:52.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:52.207 7fbaa1e1b700 1 -- 172.21.2.102:0/2543870723 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbaa4195000 msgr2=0x7fbaa40b7750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:52.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:52.207 7fbaa1e1b700 1 --2- 172.21.2.102:0/2543870723 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbaa4195000 0x7fbaa40b7750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:52.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:52.207 7fbaa2e1d700 1 --2- 172.21.2.102:0/2543870723 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fbaa41a2770 0x7fbaa41a2b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:52.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:52.207 7fbaa1e1b700 1 -- 172.21.2.102:0/2543870723 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fbaa41a2770 msgr2=0x7fbaa41a2b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:52.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:52.208 7fbaa1e1b700 1 --2- 172.21.2.102:0/2543870723 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fbaa41a2770 0x7fbaa41a2b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:52.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:52.208 7fbaa1e1b700 1 -- 172.21.2.102:0/2543870723 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbaa41a35d0 con 0x7fbaa40b7d70 2021-02-10T11:29:52.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:52.208 7fbaa261c700 1 --2- 172.21.2.102:0/2543870723 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbaa4195000 0x7fbaa40b7750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:29:52.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:52.208 7fbaa2e1d700 1 --2- 172.21.2.102:0/2543870723 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fbaa41a2770 0x7fbaa41a2b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:29:52.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:52.208 7fbaa1e1b700 1 --2- 172.21.2.102:0/2543870723 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbaa40b7d70 0x7fbaa40b8170 secure :-1 s=READY pgs=1890 cs=0 l=1 rev1=1 rx=0x7fba8c00b660 tx=0x7fba8c00ba00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:29:52.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:52.208 7fba9affd700 1 -- 172.21.2.102:0/2543870723 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fba8c009440 con 0x7fbaa40b7d70 2021-02-10T11:29:52.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:52.208 7fba9affd700 1 -- 172.21.2.102:0/2543870723 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba8c006390 con 0x7fbaa40b7d70 2021-02-10T11:29:52.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:52.208 7fba9affd700 1 -- 172.21.2.102:0/2543870723 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fba8c005560 con 0x7fbaa40b7d70 2021-02-10T11:29:52.213 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:52.208 7fbaa888d700 1 -- 172.21.2.102:0/2543870723 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbaa41a35d0 con 0x7fbaa40b7d70 2021-02-10T11:29:52.213 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:52.208 7fbaa888d700 1 -- 172.21.2.102:0/2543870723 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbaa40933b0 con 0x7fbaa40b7d70 2021-02-10T11:29:52.213 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:52.209 7fba9affd700 1 -- 172.21.2.102:0/2543870723 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fba8c0093e0 con 0x7fbaa40b7d70 2021-02-10T11:29:52.213 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:52.209 7fba9affd700 1 --2- 172.21.2.102:0/2543870723 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fba880186c0 0x7fba8801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:52.213 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:52.209 7fba9affd700 1 -- 172.21.2.102:0/2543870723 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fba8c032660 con 0x7fbaa40b7d70 2021-02-10T11:29:52.213 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:52.209 7fbaa261c700 1 --2- 172.21.2.102:0/2543870723 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fba880186c0 0x7fba8801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:52.213 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:52.209 7fbaa261c700 1 --2- 172.21.2.102:0/2543870723 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fba880186c0 0x7fba8801abc0 crc :-1 s=READY pgs=848 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:29:52.240 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:52.240 7fbaa888d700 1 -- 172.21.2.102:0/2543870723 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbaa40ee030 con 0x7fbaa40b7d70 2021-02-10T11:29:52.243 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:52.243 7fba9affd700 1 -- 172.21.2.102:0/2543870723 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fba8c01f520 con 0x7fbaa40b7d70 2021-02-10T11:29:52.367 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:52.367 7fbaa888d700 1 -- 172.21.2.102:0/2543870723 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7fbaa40ce450 con 0x7fbaa40b7d70 2021-02-10T11:29:52.368 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:52.367 7fba9affd700 1 -- 172.21.2.102:0/2543870723 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v85) v1 ==== 81+0+30363 (secure 0 0 0) 0x7fba8c006390 con 0x7fbaa40b7d70 2021-02-10T11:29:52.371 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:29:52.371 INFO:teuthology.orchestra.run.gibba002.stdout:{ 2021-02-10T11:29:52.371 INFO:teuthology.orchestra.run.gibba002.stdout: "epoch": 85, 2021-02-10T11:29:52.371 INFO:teuthology.orchestra.run.gibba002.stdout: "fsid": "68db30ea-dbce-4aef-a6df-f01c6fea0513", 2021-02-10T11:29:52.371 INFO:teuthology.orchestra.run.gibba002.stdout: "created": "2021-02-10 11:03:06.331557", 2021-02-10T11:29:52.371 INFO:teuthology.orchestra.run.gibba002.stdout: "modified": "2021-02-10 11:27:42.481451", 2021-02-10T11:29:52.371 INFO:teuthology.orchestra.run.gibba002.stdout: "last_up_change": "2021-02-10 11:03:32.690562", 2021-02-10T11:29:52.371 INFO:teuthology.orchestra.run.gibba002.stdout: "last_in_change": "2021-02-10 11:03:32.690562", 2021-02-10T11:29:52.371 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2021-02-10T11:29:52.371 INFO:teuthology.orchestra.run.gibba002.stdout: "flags_num": 5799936, 2021-02-10T11:29:52.371 INFO:teuthology.orchestra.run.gibba002.stdout: "flags_set": [ 2021-02-10T11:29:52.372 INFO:teuthology.orchestra.run.gibba002.stdout: "pglog_hardlimit", 2021-02-10T11:29:52.372 INFO:teuthology.orchestra.run.gibba002.stdout: "purged_snapdirs", 2021-02-10T11:29:52.372 INFO:teuthology.orchestra.run.gibba002.stdout: "recovery_deletes", 2021-02-10T11:29:52.372 INFO:teuthology.orchestra.run.gibba002.stdout: "sortbitwise" 2021-02-10T11:29:52.372 INFO:teuthology.orchestra.run.gibba002.stdout: ], 2021-02-10T11:29:52.372 INFO:teuthology.orchestra.run.gibba002.stdout: "crush_version": 4, 2021-02-10T11:29:52.372 INFO:teuthology.orchestra.run.gibba002.stdout: "full_ratio": 0.94999998807907104, 2021-02-10T11:29:52.372 INFO:teuthology.orchestra.run.gibba002.stdout: "backfillfull_ratio": 0.89999997615814209, 2021-02-10T11:29:52.372 INFO:teuthology.orchestra.run.gibba002.stdout: "nearfull_ratio": 0.85000002384185791, 2021-02-10T11:29:52.372 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_snapshot": "", 2021-02-10T11:29:52.372 INFO:teuthology.orchestra.run.gibba002.stdout: "pool_max": 17, 2021-02-10T11:29:52.372 INFO:teuthology.orchestra.run.gibba002.stdout: "max_osd": 8, 2021-02-10T11:29:52.372 INFO:teuthology.orchestra.run.gibba002.stdout: "require_min_compat_client": "jewel", 2021-02-10T11:29:52.373 INFO:teuthology.orchestra.run.gibba002.stdout: "min_compat_client": "jewel", 2021-02-10T11:29:52.373 INFO:teuthology.orchestra.run.gibba002.stdout: "require_osd_release": "nautilus", 2021-02-10T11:29:52.373 INFO:teuthology.orchestra.run.gibba002.stdout: "pools": [ 2021-02-10T11:29:52.373 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:29:52.373 INFO:teuthology.orchestra.run.gibba002.stdout: "pool": 1, 2021-02-10T11:29:52.373 INFO:teuthology.orchestra.run.gibba002.stdout: "pool_name": "rbd", 2021-02-10T11:29:52.373 INFO:teuthology.orchestra.run.gibba002.stdout: "create_time": "2021-02-10 11:03:37.322519", 2021-02-10T11:29:52.373 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 1, 2021-02-10T11:29:52.373 INFO:teuthology.orchestra.run.gibba002.stdout: "flags_names": "hashpspool", 2021-02-10T11:29:52.373 INFO:teuthology.orchestra.run.gibba002.stdout: "type": 1, 2021-02-10T11:29:52.373 INFO:teuthology.orchestra.run.gibba002.stdout: "size": 2, 2021-02-10T11:29:52.373 INFO:teuthology.orchestra.run.gibba002.stdout: "min_size": 1, 2021-02-10T11:29:52.373 INFO:teuthology.orchestra.run.gibba002.stdout: "crush_rule": 0, 2021-02-10T11:29:52.374 INFO:teuthology.orchestra.run.gibba002.stdout: "object_hash": 2, 2021-02-10T11:29:52.374 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:29:52.374 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num": 8, 2021-02-10T11:29:52.374 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_placement_num": 8, 2021-02-10T11:29:52.374 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_placement_num_target": 8, 2021-02-10T11:29:52.374 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num_target": 8, 2021-02-10T11:29:52.374 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num_pending": 8, 2021-02-10T11:29:52.374 INFO:teuthology.orchestra.run.gibba002.stdout: "last_pg_merge_meta": { 2021-02-10T11:29:52.374 INFO:teuthology.orchestra.run.gibba002.stdout: "source_pgid": "0.0", 2021-02-10T11:29:52.374 INFO:teuthology.orchestra.run.gibba002.stdout: "ready_epoch": 0, 2021-02-10T11:29:52.374 INFO:teuthology.orchestra.run.gibba002.stdout: "last_epoch_started": 0, 2021-02-10T11:29:52.374 INFO:teuthology.orchestra.run.gibba002.stdout: "last_epoch_clean": 0, 2021-02-10T11:29:52.374 INFO:teuthology.orchestra.run.gibba002.stdout: "source_version": "0'0", 2021-02-10T11:29:52.375 INFO:teuthology.orchestra.run.gibba002.stdout: "target_version": "0'0" 2021-02-10T11:29:52.375 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:29:52.375 INFO:teuthology.orchestra.run.gibba002.stdout: "last_change": "16", 2021-02-10T11:29:52.375 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend": "0", 2021-02-10T11:29:52.375 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:29:52.375 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:29:52.375 INFO:teuthology.orchestra.run.gibba002.stdout: "auid": 0, 2021-02-10T11:29:52.375 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:29:52.375 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_seq": 0, 2021-02-10T11:29:52.375 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_epoch": 0, 2021-02-10T11:29:52.375 INFO:teuthology.orchestra.run.gibba002.stdout: "pool_snaps": [], 2021-02-10T11:29:52.375 INFO:teuthology.orchestra.run.gibba002.stdout: "removed_snaps": "[]", 2021-02-10T11:29:52.376 INFO:teuthology.orchestra.run.gibba002.stdout: "quota_max_bytes": 0, 2021-02-10T11:29:52.376 INFO:teuthology.orchestra.run.gibba002.stdout: "quota_max_objects": 0, 2021-02-10T11:29:52.376 INFO:teuthology.orchestra.run.gibba002.stdout: "tiers": [], 2021-02-10T11:29:52.376 INFO:teuthology.orchestra.run.gibba002.stdout: "tier_of": -1, 2021-02-10T11:29:52.376 INFO:teuthology.orchestra.run.gibba002.stdout: "read_tier": -1, 2021-02-10T11:29:52.376 INFO:teuthology.orchestra.run.gibba002.stdout: "write_tier": -1, 2021-02-10T11:29:52.376 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_mode": "none", 2021-02-10T11:29:52.376 INFO:teuthology.orchestra.run.gibba002.stdout: "target_max_bytes": 0, 2021-02-10T11:29:52.376 INFO:teuthology.orchestra.run.gibba002.stdout: "target_max_objects": 0, 2021-02-10T11:29:52.376 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:29:52.376 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:29:52.376 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:29:52.376 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_min_flush_age": 0, 2021-02-10T11:29:52.377 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_min_evict_age": 0, 2021-02-10T11:29:52.377 INFO:teuthology.orchestra.run.gibba002.stdout: "erasure_code_profile": "", 2021-02-10T11:29:52.377 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_params": { 2021-02-10T11:29:52.377 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "none" 2021-02-10T11:29:52.377 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:29:52.377 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_period": 0, 2021-02-10T11:29:52.377 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_count": 0, 2021-02-10T11:29:52.377 INFO:teuthology.orchestra.run.gibba002.stdout: "use_gmt_hitset": true, 2021-02-10T11:29:52.377 INFO:teuthology.orchestra.run.gibba002.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:29:52.377 INFO:teuthology.orchestra.run.gibba002.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:29:52.377 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:29:52.377 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:29:52.377 INFO:teuthology.orchestra.run.gibba002.stdout: "grade_table": [], 2021-02-10T11:29:52.377 INFO:teuthology.orchestra.run.gibba002.stdout: "stripe_width": 0, 2021-02-10T11:29:52.378 INFO:teuthology.orchestra.run.gibba002.stdout: "expected_num_objects": 0, 2021-02-10T11:29:52.378 INFO:teuthology.orchestra.run.gibba002.stdout: "fast_read": false, 2021-02-10T11:29:52.378 INFO:teuthology.orchestra.run.gibba002.stdout: "options": {}, 2021-02-10T11:29:52.378 INFO:teuthology.orchestra.run.gibba002.stdout: "application_metadata": { 2021-02-10T11:29:52.378 INFO:teuthology.orchestra.run.gibba002.stdout: "rbd": {} 2021-02-10T11:29:52.378 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:29:52.378 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:29:52.378 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:29:52.378 INFO:teuthology.orchestra.run.gibba002.stdout: "pool": 16, 2021-02-10T11:29:52.378 INFO:teuthology.orchestra.run.gibba002.stdout: "pool_name": "cephfs_metadata", 2021-02-10T11:29:52.378 INFO:teuthology.orchestra.run.gibba002.stdout: "create_time": "2021-02-10 11:27:39.452483", 2021-02-10T11:29:52.378 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 1, 2021-02-10T11:29:52.378 INFO:teuthology.orchestra.run.gibba002.stdout: "flags_names": "hashpspool", 2021-02-10T11:29:52.379 INFO:teuthology.orchestra.run.gibba002.stdout: "type": 1, 2021-02-10T11:29:52.379 INFO:teuthology.orchestra.run.gibba002.stdout: "size": 2, 2021-02-10T11:29:52.379 INFO:teuthology.orchestra.run.gibba002.stdout: "min_size": 1, 2021-02-10T11:29:52.379 INFO:teuthology.orchestra.run.gibba002.stdout: "crush_rule": 0, 2021-02-10T11:29:52.379 INFO:teuthology.orchestra.run.gibba002.stdout: "object_hash": 2, 2021-02-10T11:29:52.379 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:29:52.379 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num": 8, 2021-02-10T11:29:52.379 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_placement_num": 8, 2021-02-10T11:29:52.379 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_placement_num_target": 8, 2021-02-10T11:29:52.379 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num_target": 8, 2021-02-10T11:29:52.379 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num_pending": 8, 2021-02-10T11:29:52.379 INFO:teuthology.orchestra.run.gibba002.stdout: "last_pg_merge_meta": { 2021-02-10T11:29:52.379 INFO:teuthology.orchestra.run.gibba002.stdout: "source_pgid": "0.0", 2021-02-10T11:29:52.380 INFO:teuthology.orchestra.run.gibba002.stdout: "ready_epoch": 0, 2021-02-10T11:29:52.380 INFO:teuthology.orchestra.run.gibba002.stdout: "last_epoch_started": 0, 2021-02-10T11:29:52.380 INFO:teuthology.orchestra.run.gibba002.stdout: "last_epoch_clean": 0, 2021-02-10T11:29:52.380 INFO:teuthology.orchestra.run.gibba002.stdout: "source_version": "0'0", 2021-02-10T11:29:52.380 INFO:teuthology.orchestra.run.gibba002.stdout: "target_version": "0'0" 2021-02-10T11:29:52.380 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:29:52.380 INFO:teuthology.orchestra.run.gibba002.stdout: "last_change": "84", 2021-02-10T11:29:52.380 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend": "0", 2021-02-10T11:29:52.380 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:29:52.380 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:29:52.380 INFO:teuthology.orchestra.run.gibba002.stdout: "auid": 0, 2021-02-10T11:29:52.380 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:29:52.380 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_seq": 0, 2021-02-10T11:29:52.380 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_epoch": 0, 2021-02-10T11:29:52.381 INFO:teuthology.orchestra.run.gibba002.stdout: "pool_snaps": [], 2021-02-10T11:29:52.381 INFO:teuthology.orchestra.run.gibba002.stdout: "removed_snaps": "[]", 2021-02-10T11:29:52.381 INFO:teuthology.orchestra.run.gibba002.stdout: "quota_max_bytes": 0, 2021-02-10T11:29:52.381 INFO:teuthology.orchestra.run.gibba002.stdout: "quota_max_objects": 0, 2021-02-10T11:29:52.381 INFO:teuthology.orchestra.run.gibba002.stdout: "tiers": [], 2021-02-10T11:29:52.381 INFO:teuthology.orchestra.run.gibba002.stdout: "tier_of": -1, 2021-02-10T11:29:52.381 INFO:teuthology.orchestra.run.gibba002.stdout: "read_tier": -1, 2021-02-10T11:29:52.381 INFO:teuthology.orchestra.run.gibba002.stdout: "write_tier": -1, 2021-02-10T11:29:52.381 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_mode": "none", 2021-02-10T11:29:52.381 INFO:teuthology.orchestra.run.gibba002.stdout: "target_max_bytes": 0, 2021-02-10T11:29:52.381 INFO:teuthology.orchestra.run.gibba002.stdout: "target_max_objects": 0, 2021-02-10T11:29:52.381 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:29:52.381 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:29:52.382 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:29:52.382 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_min_flush_age": 0, 2021-02-10T11:29:52.382 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_min_evict_age": 0, 2021-02-10T11:29:52.382 INFO:teuthology.orchestra.run.gibba002.stdout: "erasure_code_profile": "", 2021-02-10T11:29:52.382 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_params": { 2021-02-10T11:29:52.382 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "none" 2021-02-10T11:29:52.382 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:29:52.382 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_period": 0, 2021-02-10T11:29:52.382 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_count": 0, 2021-02-10T11:29:52.382 INFO:teuthology.orchestra.run.gibba002.stdout: "use_gmt_hitset": true, 2021-02-10T11:29:52.382 INFO:teuthology.orchestra.run.gibba002.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:29:52.382 INFO:teuthology.orchestra.run.gibba002.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:29:52.382 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:29:52.383 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:29:52.383 INFO:teuthology.orchestra.run.gibba002.stdout: "grade_table": [], 2021-02-10T11:29:52.383 INFO:teuthology.orchestra.run.gibba002.stdout: "stripe_width": 0, 2021-02-10T11:29:52.383 INFO:teuthology.orchestra.run.gibba002.stdout: "expected_num_objects": 0, 2021-02-10T11:29:52.383 INFO:teuthology.orchestra.run.gibba002.stdout: "fast_read": false, 2021-02-10T11:29:52.383 INFO:teuthology.orchestra.run.gibba002.stdout: "options": { 2021-02-10T11:29:52.383 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_autoscale_bias": 4, 2021-02-10T11:29:52.383 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num_min": 16, 2021-02-10T11:29:52.383 INFO:teuthology.orchestra.run.gibba002.stdout: "recovery_priority": 5 2021-02-10T11:29:52.383 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:29:52.383 INFO:teuthology.orchestra.run.gibba002.stdout: "application_metadata": { 2021-02-10T11:29:52.383 INFO:teuthology.orchestra.run.gibba002.stdout: "cephfs": { 2021-02-10T11:29:52.383 INFO:teuthology.orchestra.run.gibba002.stdout: "metadata": "cephfs" 2021-02-10T11:29:52.383 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:29:52.384 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:29:52.384 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:29:52.384 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:29:52.384 INFO:teuthology.orchestra.run.gibba002.stdout: "pool": 17, 2021-02-10T11:29:52.384 INFO:teuthology.orchestra.run.gibba002.stdout: "pool_name": "cephfs_data", 2021-02-10T11:29:52.384 INFO:teuthology.orchestra.run.gibba002.stdout: "create_time": "2021-02-10 11:27:40.475662", 2021-02-10T11:29:52.384 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 1, 2021-02-10T11:29:52.384 INFO:teuthology.orchestra.run.gibba002.stdout: "flags_names": "hashpspool", 2021-02-10T11:29:52.384 INFO:teuthology.orchestra.run.gibba002.stdout: "type": 1, 2021-02-10T11:29:52.384 INFO:teuthology.orchestra.run.gibba002.stdout: "size": 2, 2021-02-10T11:29:52.384 INFO:teuthology.orchestra.run.gibba002.stdout: "min_size": 1, 2021-02-10T11:29:52.384 INFO:teuthology.orchestra.run.gibba002.stdout: "crush_rule": 0, 2021-02-10T11:29:52.384 INFO:teuthology.orchestra.run.gibba002.stdout: "object_hash": 2, 2021-02-10T11:29:52.385 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:29:52.385 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num": 8, 2021-02-10T11:29:52.385 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_placement_num": 8, 2021-02-10T11:29:52.385 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_placement_num_target": 8, 2021-02-10T11:29:52.385 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num_target": 8, 2021-02-10T11:29:52.385 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num_pending": 8, 2021-02-10T11:29:52.385 INFO:teuthology.orchestra.run.gibba002.stdout: "last_pg_merge_meta": { 2021-02-10T11:29:52.385 INFO:teuthology.orchestra.run.gibba002.stdout: "source_pgid": "0.0", 2021-02-10T11:29:52.385 INFO:teuthology.orchestra.run.gibba002.stdout: "ready_epoch": 0, 2021-02-10T11:29:52.385 INFO:teuthology.orchestra.run.gibba002.stdout: "last_epoch_started": 0, 2021-02-10T11:29:52.385 INFO:teuthology.orchestra.run.gibba002.stdout: "last_epoch_clean": 0, 2021-02-10T11:29:52.385 INFO:teuthology.orchestra.run.gibba002.stdout: "source_version": "0'0", 2021-02-10T11:29:52.385 INFO:teuthology.orchestra.run.gibba002.stdout: "target_version": "0'0" 2021-02-10T11:29:52.385 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:29:52.386 INFO:teuthology.orchestra.run.gibba002.stdout: "last_change": "85", 2021-02-10T11:29:52.386 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend": "0", 2021-02-10T11:29:52.386 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:29:52.386 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:29:52.386 INFO:teuthology.orchestra.run.gibba002.stdout: "auid": 0, 2021-02-10T11:29:52.386 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:29:52.386 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_seq": 0, 2021-02-10T11:29:52.386 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_epoch": 0, 2021-02-10T11:29:52.386 INFO:teuthology.orchestra.run.gibba002.stdout: "pool_snaps": [], 2021-02-10T11:29:52.386 INFO:teuthology.orchestra.run.gibba002.stdout: "removed_snaps": "[]", 2021-02-10T11:29:52.386 INFO:teuthology.orchestra.run.gibba002.stdout: "quota_max_bytes": 0, 2021-02-10T11:29:52.386 INFO:teuthology.orchestra.run.gibba002.stdout: "quota_max_objects": 0, 2021-02-10T11:29:52.386 INFO:teuthology.orchestra.run.gibba002.stdout: "tiers": [], 2021-02-10T11:29:52.387 INFO:teuthology.orchestra.run.gibba002.stdout: "tier_of": -1, 2021-02-10T11:29:52.387 INFO:teuthology.orchestra.run.gibba002.stdout: "read_tier": -1, 2021-02-10T11:29:52.387 INFO:teuthology.orchestra.run.gibba002.stdout: "write_tier": -1, 2021-02-10T11:29:52.387 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_mode": "none", 2021-02-10T11:29:52.387 INFO:teuthology.orchestra.run.gibba002.stdout: "target_max_bytes": 0, 2021-02-10T11:29:52.387 INFO:teuthology.orchestra.run.gibba002.stdout: "target_max_objects": 0, 2021-02-10T11:29:52.387 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:29:52.387 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:29:52.387 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:29:52.387 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_min_flush_age": 0, 2021-02-10T11:29:52.387 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_min_evict_age": 0, 2021-02-10T11:29:52.387 INFO:teuthology.orchestra.run.gibba002.stdout: "erasure_code_profile": "", 2021-02-10T11:29:52.387 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_params": { 2021-02-10T11:29:52.388 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "none" 2021-02-10T11:29:52.388 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:29:52.388 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_period": 0, 2021-02-10T11:29:52.388 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_count": 0, 2021-02-10T11:29:52.388 INFO:teuthology.orchestra.run.gibba002.stdout: "use_gmt_hitset": true, 2021-02-10T11:29:52.388 INFO:teuthology.orchestra.run.gibba002.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:29:52.388 INFO:teuthology.orchestra.run.gibba002.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:29:52.388 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:29:52.388 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:29:52.388 INFO:teuthology.orchestra.run.gibba002.stdout: "grade_table": [], 2021-02-10T11:29:52.388 INFO:teuthology.orchestra.run.gibba002.stdout: "stripe_width": 0, 2021-02-10T11:29:52.388 INFO:teuthology.orchestra.run.gibba002.stdout: "expected_num_objects": 0, 2021-02-10T11:29:52.388 INFO:teuthology.orchestra.run.gibba002.stdout: "fast_read": false, 2021-02-10T11:29:52.388 INFO:teuthology.orchestra.run.gibba002.stdout: "options": {}, 2021-02-10T11:29:52.389 INFO:teuthology.orchestra.run.gibba002.stdout: "application_metadata": { 2021-02-10T11:29:52.389 INFO:teuthology.orchestra.run.gibba002.stdout: "cephfs": { 2021-02-10T11:29:52.389 INFO:teuthology.orchestra.run.gibba002.stdout: "data": "cephfs" 2021-02-10T11:29:52.389 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:29:52.389 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:29:52.389 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:29:52.389 INFO:teuthology.orchestra.run.gibba002.stdout: ], 2021-02-10T11:29:52.389 INFO:teuthology.orchestra.run.gibba002.stdout: "osds": [ 2021-02-10T11:29:52.389 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:29:52.389 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 0, 2021-02-10T11:29:52.389 INFO:teuthology.orchestra.run.gibba002.stdout: "uuid": "be691b4d-a039-408a-af80-9d53f02b6f1f", 2021-02-10T11:29:52.389 INFO:teuthology.orchestra.run.gibba002.stdout: "up": 1, 2021-02-10T11:29:52.389 INFO:teuthology.orchestra.run.gibba002.stdout: "in": 1, 2021-02-10T11:29:52.390 INFO:teuthology.orchestra.run.gibba002.stdout: "weight": 1, 2021-02-10T11:29:52.390 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_affinity": 1, 2021-02-10T11:29:52.390 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_begin": 0, 2021-02-10T11:29:52.390 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_end": 0, 2021-02-10T11:29:52.390 INFO:teuthology.orchestra.run.gibba002.stdout: "up_from": 13, 2021-02-10T11:29:52.390 INFO:teuthology.orchestra.run.gibba002.stdout: "up_thru": 83, 2021-02-10T11:29:52.390 INFO:teuthology.orchestra.run.gibba002.stdout: "down_at": 0, 2021-02-10T11:29:52.390 INFO:teuthology.orchestra.run.gibba002.stdout: "lost_at": 0, 2021-02-10T11:29:52.390 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addrs": { 2021-02-10T11:29:52.390 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:29:52.390 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:29:52.390 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:29:52.390 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6818", 2021-02-10T11:29:52.391 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36088 2021-02-10T11:29:52.391 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:29:52.391 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:29:52.391 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:29:52.391 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6820", 2021-02-10T11:29:52.391 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36088 2021-02-10T11:29:52.391 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:29:52.391 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:29:52.391 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:29:52.391 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addrs": { 2021-02-10T11:29:52.391 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:29:52.391 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:29:52.391 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:29:52.392 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6822", 2021-02-10T11:29:52.392 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36088 2021-02-10T11:29:52.392 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:29:52.392 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:29:52.392 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:29:52.392 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6824", 2021-02-10T11:29:52.392 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36088 2021-02-10T11:29:52.392 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:29:52.392 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:29:52.392 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:29:52.392 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addrs": { 2021-02-10T11:29:52.392 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:29:52.392 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:29:52.393 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:29:52.393 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6830", 2021-02-10T11:29:52.393 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36088 2021-02-10T11:29:52.393 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:29:52.393 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:29:52.393 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:29:52.393 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6832", 2021-02-10T11:29:52.393 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36088 2021-02-10T11:29:52.393 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:29:52.393 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:29:52.393 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:29:52.393 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addrs": { 2021-02-10T11:29:52.393 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:29:52.394 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:29:52.394 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:29:52.394 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6826", 2021-02-10T11:29:52.394 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36088 2021-02-10T11:29:52.394 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:29:52.394 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:29:52.394 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:29:52.394 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6828", 2021-02-10T11:29:52.394 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36088 2021-02-10T11:29:52.394 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:29:52.394 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:29:52.394 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:29:52.394 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addr": "172.21.2.102:6820/36088", 2021-02-10T11:29:52.394 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addr": "172.21.2.102:6824/36088", 2021-02-10T11:29:52.395 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addr": "172.21.2.102:6832/36088", 2021-02-10T11:29:52.395 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addr": "172.21.2.102:6828/36088", 2021-02-10T11:29:52.395 INFO:teuthology.orchestra.run.gibba002.stdout: "state": [ 2021-02-10T11:29:52.395 INFO:teuthology.orchestra.run.gibba002.stdout: "exists", 2021-02-10T11:29:52.395 INFO:teuthology.orchestra.run.gibba002.stdout: "up" 2021-02-10T11:29:52.395 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:29:52.395 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:29:52.395 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:29:52.395 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 1, 2021-02-10T11:29:52.395 INFO:teuthology.orchestra.run.gibba002.stdout: "uuid": "64e3ccc8-9c69-4d7c-865e-af8e83822635", 2021-02-10T11:29:52.395 INFO:teuthology.orchestra.run.gibba002.stdout: "up": 1, 2021-02-10T11:29:52.395 INFO:teuthology.orchestra.run.gibba002.stdout: "in": 1, 2021-02-10T11:29:52.395 INFO:teuthology.orchestra.run.gibba002.stdout: "weight": 1, 2021-02-10T11:29:52.396 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_affinity": 1, 2021-02-10T11:29:52.396 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_begin": 0, 2021-02-10T11:29:52.396 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_end": 0, 2021-02-10T11:29:52.396 INFO:teuthology.orchestra.run.gibba002.stdout: "up_from": 13, 2021-02-10T11:29:52.396 INFO:teuthology.orchestra.run.gibba002.stdout: "up_thru": 73, 2021-02-10T11:29:52.396 INFO:teuthology.orchestra.run.gibba002.stdout: "down_at": 0, 2021-02-10T11:29:52.396 INFO:teuthology.orchestra.run.gibba002.stdout: "lost_at": 0, 2021-02-10T11:29:52.396 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addrs": { 2021-02-10T11:29:52.396 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:29:52.396 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:29:52.396 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:29:52.396 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6819", 2021-02-10T11:29:52.396 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36087 2021-02-10T11:29:52.397 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:29:52.397 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:29:52.397 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:29:52.397 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6821", 2021-02-10T11:29:52.397 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36087 2021-02-10T11:29:52.397 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:29:52.397 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:29:52.397 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:29:52.397 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addrs": { 2021-02-10T11:29:52.397 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:29:52.397 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:29:52.397 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:29:52.397 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6823", 2021-02-10T11:29:52.397 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36087 2021-02-10T11:29:52.398 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:29:52.398 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:29:52.398 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:29:52.398 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6825", 2021-02-10T11:29:52.398 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36087 2021-02-10T11:29:52.398 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:29:52.398 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:29:52.398 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:29:52.398 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addrs": { 2021-02-10T11:29:52.398 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:29:52.398 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:29:52.398 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:29:52.398 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6831", 2021-02-10T11:29:52.399 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36087 2021-02-10T11:29:52.399 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:29:52.399 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:29:52.399 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:29:52.399 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6833", 2021-02-10T11:29:52.399 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36087 2021-02-10T11:29:52.399 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:29:52.399 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:29:52.399 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:29:52.399 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addrs": { 2021-02-10T11:29:52.399 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:29:52.399 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:29:52.400 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:29:52.400 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6827", 2021-02-10T11:29:52.400 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36087 2021-02-10T11:29:52.400 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:29:52.400 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:29:52.400 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:29:52.400 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6829", 2021-02-10T11:29:52.401 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36087 2021-02-10T11:29:52.401 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:29:52.401 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:29:52.401 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:29:52.401 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addr": "172.21.2.102:6821/36087", 2021-02-10T11:29:52.401 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addr": "172.21.2.102:6825/36087", 2021-02-10T11:29:52.401 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addr": "172.21.2.102:6833/36087", 2021-02-10T11:29:52.401 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addr": "172.21.2.102:6829/36087", 2021-02-10T11:29:52.401 INFO:teuthology.orchestra.run.gibba002.stdout: "state": [ 2021-02-10T11:29:52.401 INFO:teuthology.orchestra.run.gibba002.stdout: "exists", 2021-02-10T11:29:52.401 INFO:teuthology.orchestra.run.gibba002.stdout: "up" 2021-02-10T11:29:52.402 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:29:52.402 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:29:52.402 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:29:52.402 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 2, 2021-02-10T11:29:52.402 INFO:teuthology.orchestra.run.gibba002.stdout: "uuid": "153700af-747f-481f-bdf4-d6072d3a842f", 2021-02-10T11:29:52.402 INFO:teuthology.orchestra.run.gibba002.stdout: "up": 1, 2021-02-10T11:29:52.402 INFO:teuthology.orchestra.run.gibba002.stdout: "in": 1, 2021-02-10T11:29:52.402 INFO:teuthology.orchestra.run.gibba002.stdout: "weight": 1, 2021-02-10T11:29:52.402 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_affinity": 1, 2021-02-10T11:29:52.402 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_begin": 0, 2021-02-10T11:29:52.402 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_end": 0, 2021-02-10T11:29:52.402 INFO:teuthology.orchestra.run.gibba002.stdout: "up_from": 13, 2021-02-10T11:29:52.402 INFO:teuthology.orchestra.run.gibba002.stdout: "up_thru": 82, 2021-02-10T11:29:52.402 INFO:teuthology.orchestra.run.gibba002.stdout: "down_at": 0, 2021-02-10T11:29:52.403 INFO:teuthology.orchestra.run.gibba002.stdout: "lost_at": 0, 2021-02-10T11:29:52.403 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addrs": { 2021-02-10T11:29:52.403 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:29:52.403 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:29:52.403 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:29:52.403 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6810", 2021-02-10T11:29:52.403 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36089 2021-02-10T11:29:52.403 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:29:52.403 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:29:52.403 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:29:52.403 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6811", 2021-02-10T11:29:52.403 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36089 2021-02-10T11:29:52.403 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:29:52.404 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:29:52.404 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:29:52.404 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addrs": { 2021-02-10T11:29:52.404 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:29:52.404 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:29:52.404 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:29:52.404 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6812", 2021-02-10T11:29:52.404 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36089 2021-02-10T11:29:52.404 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:29:52.404 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:29:52.404 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:29:52.404 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6813", 2021-02-10T11:29:52.404 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36089 2021-02-10T11:29:52.404 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:29:52.405 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:29:52.405 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:29:52.405 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addrs": { 2021-02-10T11:29:52.405 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:29:52.405 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:29:52.405 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:29:52.405 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6816", 2021-02-10T11:29:52.405 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36089 2021-02-10T11:29:52.405 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:29:52.405 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:29:52.405 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:29:52.405 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6817", 2021-02-10T11:29:52.405 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36089 2021-02-10T11:29:52.406 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:29:52.406 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:29:52.406 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:29:52.406 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addrs": { 2021-02-10T11:29:52.406 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:29:52.406 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:29:52.406 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:29:52.406 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6814", 2021-02-10T11:29:52.406 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36089 2021-02-10T11:29:52.406 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:29:52.406 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:29:52.406 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:29:52.406 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6815", 2021-02-10T11:29:52.407 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36089 2021-02-10T11:29:52.407 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:29:52.407 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:29:52.407 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:29:52.407 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addr": "172.21.2.102:6811/36089", 2021-02-10T11:29:52.407 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addr": "172.21.2.102:6813/36089", 2021-02-10T11:29:52.407 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addr": "172.21.2.102:6817/36089", 2021-02-10T11:29:52.407 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addr": "172.21.2.102:6815/36089", 2021-02-10T11:29:52.407 INFO:teuthology.orchestra.run.gibba002.stdout: "state": [ 2021-02-10T11:29:52.407 INFO:teuthology.orchestra.run.gibba002.stdout: "exists", 2021-02-10T11:29:52.407 INFO:teuthology.orchestra.run.gibba002.stdout: "up" 2021-02-10T11:29:52.407 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:29:52.407 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:29:52.407 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:29:52.408 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 3, 2021-02-10T11:29:52.408 INFO:teuthology.orchestra.run.gibba002.stdout: "uuid": "cbc678f4-83a9-45e6-8830-2f7c232b1ea7", 2021-02-10T11:29:52.408 INFO:teuthology.orchestra.run.gibba002.stdout: "up": 1, 2021-02-10T11:29:52.408 INFO:teuthology.orchestra.run.gibba002.stdout: "in": 1, 2021-02-10T11:29:52.408 INFO:teuthology.orchestra.run.gibba002.stdout: "weight": 1, 2021-02-10T11:29:52.408 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_affinity": 1, 2021-02-10T11:29:52.408 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_begin": 0, 2021-02-10T11:29:52.408 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_end": 0, 2021-02-10T11:29:52.408 INFO:teuthology.orchestra.run.gibba002.stdout: "up_from": 13, 2021-02-10T11:29:52.408 INFO:teuthology.orchestra.run.gibba002.stdout: "up_thru": 82, 2021-02-10T11:29:52.408 INFO:teuthology.orchestra.run.gibba002.stdout: "down_at": 0, 2021-02-10T11:29:52.408 INFO:teuthology.orchestra.run.gibba002.stdout: "lost_at": 0, 2021-02-10T11:29:52.409 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addrs": { 2021-02-10T11:29:52.409 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:29:52.409 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:29:52.409 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:29:52.409 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6802", 2021-02-10T11:29:52.409 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36090 2021-02-10T11:29:52.409 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:29:52.409 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:29:52.409 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:29:52.409 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6803", 2021-02-10T11:29:52.409 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36090 2021-02-10T11:29:52.411 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:29:52.411 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:29:52.411 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:29:52.411 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addrs": { 2021-02-10T11:29:52.411 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:29:52.411 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:29:52.411 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:29:52.411 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6804", 2021-02-10T11:29:52.411 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36090 2021-02-10T11:29:52.411 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:29:52.412 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:29:52.412 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:29:52.412 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6805", 2021-02-10T11:29:52.412 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36090 2021-02-10T11:29:52.412 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:29:52.412 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:29:52.412 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:29:52.412 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addrs": { 2021-02-10T11:29:52.412 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:29:52.412 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:29:52.412 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:29:52.412 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6808", 2021-02-10T11:29:52.412 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36090 2021-02-10T11:29:52.413 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:29:52.413 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:29:52.413 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:29:52.413 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6809", 2021-02-10T11:29:52.413 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36090 2021-02-10T11:29:52.413 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:29:52.413 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:29:52.413 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:29:52.413 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addrs": { 2021-02-10T11:29:52.413 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:29:52.413 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:29:52.413 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:29:52.413 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6806", 2021-02-10T11:29:52.414 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36090 2021-02-10T11:29:52.414 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:29:52.414 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:29:52.414 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:29:52.414 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6807", 2021-02-10T11:29:52.414 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36090 2021-02-10T11:29:52.414 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:29:52.414 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:29:52.414 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:29:52.414 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addr": "172.21.2.102:6803/36090", 2021-02-10T11:29:52.414 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addr": "172.21.2.102:6805/36090", 2021-02-10T11:29:52.414 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addr": "172.21.2.102:6809/36090", 2021-02-10T11:29:52.414 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addr": "172.21.2.102:6807/36090", 2021-02-10T11:29:52.415 INFO:teuthology.orchestra.run.gibba002.stdout: "state": [ 2021-02-10T11:29:52.415 INFO:teuthology.orchestra.run.gibba002.stdout: "exists", 2021-02-10T11:29:52.415 INFO:teuthology.orchestra.run.gibba002.stdout: "up" 2021-02-10T11:29:52.415 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:29:52.415 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:29:52.415 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:29:52.415 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 4, 2021-02-10T11:29:52.415 INFO:teuthology.orchestra.run.gibba002.stdout: "uuid": "c01e4008-b258-4f10-afd2-9e7f37214351", 2021-02-10T11:29:52.415 INFO:teuthology.orchestra.run.gibba002.stdout: "up": 1, 2021-02-10T11:29:52.415 INFO:teuthology.orchestra.run.gibba002.stdout: "in": 1, 2021-02-10T11:29:52.415 INFO:teuthology.orchestra.run.gibba002.stdout: "weight": 1, 2021-02-10T11:29:52.415 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_affinity": 1, 2021-02-10T11:29:52.415 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_begin": 0, 2021-02-10T11:29:52.415 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_end": 0, 2021-02-10T11:29:52.416 INFO:teuthology.orchestra.run.gibba002.stdout: "up_from": 13, 2021-02-10T11:29:52.416 INFO:teuthology.orchestra.run.gibba002.stdout: "up_thru": 83, 2021-02-10T11:29:52.416 INFO:teuthology.orchestra.run.gibba002.stdout: "down_at": 0, 2021-02-10T11:29:52.416 INFO:teuthology.orchestra.run.gibba002.stdout: "lost_at": 0, 2021-02-10T11:29:52.416 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addrs": { 2021-02-10T11:29:52.416 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:29:52.416 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:29:52.416 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:29:52.416 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6808", 2021-02-10T11:29:52.416 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36368 2021-02-10T11:29:52.416 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:29:52.416 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:29:52.416 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:29:52.417 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6811", 2021-02-10T11:29:52.417 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36368 2021-02-10T11:29:52.417 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:29:52.417 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:29:52.417 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:29:52.417 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addrs": { 2021-02-10T11:29:52.417 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:29:52.417 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:29:52.417 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:29:52.417 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6814", 2021-02-10T11:29:52.417 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36368 2021-02-10T11:29:52.417 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:29:52.417 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:29:52.417 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:29:52.418 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6818", 2021-02-10T11:29:52.418 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36368 2021-02-10T11:29:52.418 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:29:52.418 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:29:52.418 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:29:52.418 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addrs": { 2021-02-10T11:29:52.418 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:29:52.418 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:29:52.418 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:29:52.418 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6826", 2021-02-10T11:29:52.418 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36368 2021-02-10T11:29:52.418 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:29:52.418 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:29:52.419 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:29:52.419 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6828", 2021-02-10T11:29:52.419 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36368 2021-02-10T11:29:52.419 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:29:52.419 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:29:52.419 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:29:52.419 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addrs": { 2021-02-10T11:29:52.419 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:29:52.419 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:29:52.419 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:29:52.419 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6820", 2021-02-10T11:29:52.419 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36368 2021-02-10T11:29:52.419 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:29:52.419 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:29:52.420 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:29:52.420 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6824", 2021-02-10T11:29:52.420 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36368 2021-02-10T11:29:52.420 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:29:52.420 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:29:52.420 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:29:52.420 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addr": "172.21.2.113:6811/36368", 2021-02-10T11:29:52.420 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addr": "172.21.2.113:6818/36368", 2021-02-10T11:29:52.420 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addr": "172.21.2.113:6828/36368", 2021-02-10T11:29:52.420 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addr": "172.21.2.113:6824/36368", 2021-02-10T11:29:52.420 INFO:teuthology.orchestra.run.gibba002.stdout: "state": [ 2021-02-10T11:29:52.420 INFO:teuthology.orchestra.run.gibba002.stdout: "exists", 2021-02-10T11:29:52.420 INFO:teuthology.orchestra.run.gibba002.stdout: "up" 2021-02-10T11:29:52.421 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:29:52.421 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:29:52.421 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:29:52.421 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 5, 2021-02-10T11:29:52.421 INFO:teuthology.orchestra.run.gibba002.stdout: "uuid": "551f6007-daf8-4a98-8ee0-c2b16bcccb5e", 2021-02-10T11:29:52.421 INFO:teuthology.orchestra.run.gibba002.stdout: "up": 1, 2021-02-10T11:29:52.421 INFO:teuthology.orchestra.run.gibba002.stdout: "in": 1, 2021-02-10T11:29:52.421 INFO:teuthology.orchestra.run.gibba002.stdout: "weight": 1, 2021-02-10T11:29:52.421 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_affinity": 1, 2021-02-10T11:29:52.421 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_begin": 0, 2021-02-10T11:29:52.421 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_end": 0, 2021-02-10T11:29:52.421 INFO:teuthology.orchestra.run.gibba002.stdout: "up_from": 13, 2021-02-10T11:29:52.421 INFO:teuthology.orchestra.run.gibba002.stdout: "up_thru": 82, 2021-02-10T11:29:52.421 INFO:teuthology.orchestra.run.gibba002.stdout: "down_at": 0, 2021-02-10T11:29:52.422 INFO:teuthology.orchestra.run.gibba002.stdout: "lost_at": 0, 2021-02-10T11:29:52.422 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addrs": { 2021-02-10T11:29:52.422 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:29:52.422 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:29:52.422 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:29:52.422 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6802", 2021-02-10T11:29:52.422 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36366 2021-02-10T11:29:52.422 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:29:52.422 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:29:52.422 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:29:52.422 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6805", 2021-02-10T11:29:52.422 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36366 2021-02-10T11:29:52.422 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:29:52.423 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:29:52.423 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:29:52.423 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addrs": { 2021-02-10T11:29:52.423 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:29:52.423 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:29:52.423 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:29:52.423 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6807", 2021-02-10T11:29:52.423 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36366 2021-02-10T11:29:52.423 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:29:52.423 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:29:52.423 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:29:52.423 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6810", 2021-02-10T11:29:52.423 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36366 2021-02-10T11:29:52.424 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:29:52.424 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:29:52.424 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:29:52.424 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addrs": { 2021-02-10T11:29:52.424 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:29:52.424 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:29:52.424 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:29:52.424 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6819", 2021-02-10T11:29:52.424 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36366 2021-02-10T11:29:52.424 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:29:52.424 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:29:52.424 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:29:52.424 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6822", 2021-02-10T11:29:52.424 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36366 2021-02-10T11:29:52.425 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:29:52.425 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:29:52.425 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:29:52.425 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addrs": { 2021-02-10T11:29:52.425 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:29:52.425 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:29:52.425 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:29:52.425 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6812", 2021-02-10T11:29:52.425 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36366 2021-02-10T11:29:52.425 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:29:52.425 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:29:52.425 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:29:52.425 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6815", 2021-02-10T11:29:52.426 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36366 2021-02-10T11:29:52.426 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:29:52.426 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:29:52.426 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:29:52.426 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addr": "172.21.2.113:6805/36366", 2021-02-10T11:29:52.426 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addr": "172.21.2.113:6810/36366", 2021-02-10T11:29:52.426 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addr": "172.21.2.113:6822/36366", 2021-02-10T11:29:52.426 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addr": "172.21.2.113:6815/36366", 2021-02-10T11:29:52.426 INFO:teuthology.orchestra.run.gibba002.stdout: "state": [ 2021-02-10T11:29:52.426 INFO:teuthology.orchestra.run.gibba002.stdout: "exists", 2021-02-10T11:29:52.426 INFO:teuthology.orchestra.run.gibba002.stdout: "up" 2021-02-10T11:29:52.426 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:29:52.426 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:29:52.427 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:29:52.427 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 6, 2021-02-10T11:29:52.427 INFO:teuthology.orchestra.run.gibba002.stdout: "uuid": "bf67c040-0e50-48be-b102-c3943bc9ebef", 2021-02-10T11:29:52.427 INFO:teuthology.orchestra.run.gibba002.stdout: "up": 1, 2021-02-10T11:29:52.427 INFO:teuthology.orchestra.run.gibba002.stdout: "in": 1, 2021-02-10T11:29:52.427 INFO:teuthology.orchestra.run.gibba002.stdout: "weight": 1, 2021-02-10T11:29:52.427 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_affinity": 1, 2021-02-10T11:29:52.427 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_begin": 0, 2021-02-10T11:29:52.427 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_end": 0, 2021-02-10T11:29:52.427 INFO:teuthology.orchestra.run.gibba002.stdout: "up_from": 13, 2021-02-10T11:29:52.427 INFO:teuthology.orchestra.run.gibba002.stdout: "up_thru": 83, 2021-02-10T11:29:52.427 INFO:teuthology.orchestra.run.gibba002.stdout: "down_at": 0, 2021-02-10T11:29:52.427 INFO:teuthology.orchestra.run.gibba002.stdout: "lost_at": 0, 2021-02-10T11:29:52.427 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addrs": { 2021-02-10T11:29:52.428 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:29:52.428 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:29:52.428 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:29:52.428 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6800", 2021-02-10T11:29:52.428 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36367 2021-02-10T11:29:52.428 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:29:52.428 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:29:52.428 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:29:52.428 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6801", 2021-02-10T11:29:52.428 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36367 2021-02-10T11:29:52.428 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:29:52.428 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:29:52.428 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:29:52.429 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addrs": { 2021-02-10T11:29:52.429 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:29:52.429 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:29:52.429 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:29:52.429 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6803", 2021-02-10T11:29:52.429 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36367 2021-02-10T11:29:52.429 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:29:52.429 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:29:52.429 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:29:52.429 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6804", 2021-02-10T11:29:52.429 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36367 2021-02-10T11:29:52.429 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:29:52.429 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:29:52.429 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:29:52.430 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addrs": { 2021-02-10T11:29:52.430 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:29:52.430 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:29:52.430 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:29:52.430 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6813", 2021-02-10T11:29:52.430 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36367 2021-02-10T11:29:52.430 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:29:52.430 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:29:52.430 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:29:52.430 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6816", 2021-02-10T11:29:52.430 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36367 2021-02-10T11:29:52.430 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:29:52.430 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:29:52.431 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:29:52.431 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addrs": { 2021-02-10T11:29:52.431 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:29:52.431 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:29:52.431 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:29:52.431 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6806", 2021-02-10T11:29:52.431 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36367 2021-02-10T11:29:52.431 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:29:52.431 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:29:52.431 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:29:52.431 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6809", 2021-02-10T11:29:52.431 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36367 2021-02-10T11:29:52.431 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:29:52.432 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:29:52.432 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:29:52.432 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addr": "172.21.2.113:6801/36367", 2021-02-10T11:29:52.432 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addr": "172.21.2.113:6804/36367", 2021-02-10T11:29:52.432 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addr": "172.21.2.113:6816/36367", 2021-02-10T11:29:52.432 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addr": "172.21.2.113:6809/36367", 2021-02-10T11:29:52.432 INFO:teuthology.orchestra.run.gibba002.stdout: "state": [ 2021-02-10T11:29:52.432 INFO:teuthology.orchestra.run.gibba002.stdout: "exists", 2021-02-10T11:29:52.432 INFO:teuthology.orchestra.run.gibba002.stdout: "up" 2021-02-10T11:29:52.432 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:29:52.432 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:29:52.432 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:29:52.432 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 7, 2021-02-10T11:29:52.433 INFO:teuthology.orchestra.run.gibba002.stdout: "uuid": "f5461f80-018d-428f-be89-2eb2bcbe7bb6", 2021-02-10T11:29:52.433 INFO:teuthology.orchestra.run.gibba002.stdout: "up": 1, 2021-02-10T11:29:52.433 INFO:teuthology.orchestra.run.gibba002.stdout: "in": 1, 2021-02-10T11:29:52.433 INFO:teuthology.orchestra.run.gibba002.stdout: "weight": 1, 2021-02-10T11:29:52.433 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_affinity": 1, 2021-02-10T11:29:52.433 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_begin": 0, 2021-02-10T11:29:52.433 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_end": 0, 2021-02-10T11:29:52.433 INFO:teuthology.orchestra.run.gibba002.stdout: "up_from": 13, 2021-02-10T11:29:52.433 INFO:teuthology.orchestra.run.gibba002.stdout: "up_thru": 83, 2021-02-10T11:29:52.433 INFO:teuthology.orchestra.run.gibba002.stdout: "down_at": 0, 2021-02-10T11:29:52.433 INFO:teuthology.orchestra.run.gibba002.stdout: "lost_at": 0, 2021-02-10T11:29:52.433 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addrs": { 2021-02-10T11:29:52.433 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:29:52.433 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:29:52.434 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:29:52.434 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6817", 2021-02-10T11:29:52.434 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36365 2021-02-10T11:29:52.434 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:29:52.434 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:29:52.434 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:29:52.434 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6821", 2021-02-10T11:29:52.434 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36365 2021-02-10T11:29:52.434 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:29:52.434 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:29:52.435 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:29:52.435 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addrs": { 2021-02-10T11:29:52.435 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:29:52.435 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:29:52.435 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:29:52.435 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6823", 2021-02-10T11:29:52.435 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36365 2021-02-10T11:29:52.435 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:29:52.435 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:29:52.435 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:29:52.435 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6825", 2021-02-10T11:29:52.435 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36365 2021-02-10T11:29:52.436 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:29:52.436 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:29:52.436 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:29:52.436 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addrs": { 2021-02-10T11:29:52.436 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:29:52.436 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:29:52.436 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:29:52.436 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6830", 2021-02-10T11:29:52.436 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36365 2021-02-10T11:29:52.436 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:29:52.436 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:29:52.436 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:29:52.437 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6831", 2021-02-10T11:29:52.437 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36365 2021-02-10T11:29:52.437 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:29:52.437 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:29:52.437 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:29:52.437 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addrs": { 2021-02-10T11:29:52.437 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:29:52.437 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:29:52.437 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:29:52.437 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6827", 2021-02-10T11:29:52.438 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36365 2021-02-10T11:29:52.438 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:29:52.438 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:29:52.438 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:29:52.438 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6829", 2021-02-10T11:29:52.438 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36365 2021-02-10T11:29:52.438 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:29:52.438 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:29:52.438 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:29:52.438 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addr": "172.21.2.113:6821/36365", 2021-02-10T11:29:52.438 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addr": "172.21.2.113:6825/36365", 2021-02-10T11:29:52.438 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addr": "172.21.2.113:6831/36365", 2021-02-10T11:29:52.438 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addr": "172.21.2.113:6829/36365", 2021-02-10T11:29:52.439 INFO:teuthology.orchestra.run.gibba002.stdout: "state": [ 2021-02-10T11:29:52.439 INFO:teuthology.orchestra.run.gibba002.stdout: "exists", 2021-02-10T11:29:52.439 INFO:teuthology.orchestra.run.gibba002.stdout: "up" 2021-02-10T11:29:52.439 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:29:52.439 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:29:52.439 INFO:teuthology.orchestra.run.gibba002.stdout: ], 2021-02-10T11:29:52.439 INFO:teuthology.orchestra.run.gibba002.stdout: "osd_xinfo": [ 2021-02-10T11:29:52.439 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:29:52.439 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 0, 2021-02-10T11:29:52.439 INFO:teuthology.orchestra.run.gibba002.stdout: "down_stamp": "0.000000", 2021-02-10T11:29:52.439 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_probability": 0, 2021-02-10T11:29:52.439 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_interval": 0, 2021-02-10T11:29:52.439 INFO:teuthology.orchestra.run.gibba002.stdout: "features": 4611087854035861503, 2021-02-10T11:29:52.439 INFO:teuthology.orchestra.run.gibba002.stdout: "old_weight": 0 2021-02-10T11:29:52.440 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:29:52.440 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:29:52.440 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 1, 2021-02-10T11:29:52.440 INFO:teuthology.orchestra.run.gibba002.stdout: "down_stamp": "0.000000", 2021-02-10T11:29:52.440 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_probability": 0, 2021-02-10T11:29:52.440 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_interval": 0, 2021-02-10T11:29:52.440 INFO:teuthology.orchestra.run.gibba002.stdout: "features": 4611087854035861503, 2021-02-10T11:29:52.440 INFO:teuthology.orchestra.run.gibba002.stdout: "old_weight": 0 2021-02-10T11:29:52.440 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:29:52.440 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:29:52.440 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 2, 2021-02-10T11:29:52.440 INFO:teuthology.orchestra.run.gibba002.stdout: "down_stamp": "0.000000", 2021-02-10T11:29:52.440 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_probability": 0, 2021-02-10T11:29:52.441 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_interval": 0, 2021-02-10T11:29:52.441 INFO:teuthology.orchestra.run.gibba002.stdout: "features": 4611087854035861503, 2021-02-10T11:29:52.441 INFO:teuthology.orchestra.run.gibba002.stdout: "old_weight": 0 2021-02-10T11:29:52.441 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:29:52.441 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:29:52.441 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 3, 2021-02-10T11:29:52.441 INFO:teuthology.orchestra.run.gibba002.stdout: "down_stamp": "0.000000", 2021-02-10T11:29:52.441 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_probability": 0, 2021-02-10T11:29:52.441 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_interval": 0, 2021-02-10T11:29:52.441 INFO:teuthology.orchestra.run.gibba002.stdout: "features": 4611087854035861503, 2021-02-10T11:29:52.441 INFO:teuthology.orchestra.run.gibba002.stdout: "old_weight": 0 2021-02-10T11:29:52.441 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:29:52.441 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:29:52.442 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 4, 2021-02-10T11:29:52.442 INFO:teuthology.orchestra.run.gibba002.stdout: "down_stamp": "0.000000", 2021-02-10T11:29:52.442 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_probability": 0, 2021-02-10T11:29:52.442 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_interval": 0, 2021-02-10T11:29:52.442 INFO:teuthology.orchestra.run.gibba002.stdout: "features": 4611087854035861503, 2021-02-10T11:29:52.442 INFO:teuthology.orchestra.run.gibba002.stdout: "old_weight": 0 2021-02-10T11:29:52.442 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:29:52.442 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:29:52.442 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 5, 2021-02-10T11:29:52.442 INFO:teuthology.orchestra.run.gibba002.stdout: "down_stamp": "0.000000", 2021-02-10T11:29:52.442 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_probability": 0, 2021-02-10T11:29:52.442 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_interval": 0, 2021-02-10T11:29:52.442 INFO:teuthology.orchestra.run.gibba002.stdout: "features": 4611087854035861503, 2021-02-10T11:29:52.442 INFO:teuthology.orchestra.run.gibba002.stdout: "old_weight": 0 2021-02-10T11:29:52.443 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:29:52.443 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:29:52.443 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 6, 2021-02-10T11:29:52.443 INFO:teuthology.orchestra.run.gibba002.stdout: "down_stamp": "0.000000", 2021-02-10T11:29:52.443 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_probability": 0, 2021-02-10T11:29:52.443 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_interval": 0, 2021-02-10T11:29:52.443 INFO:teuthology.orchestra.run.gibba002.stdout: "features": 4611087854035861503, 2021-02-10T11:29:52.443 INFO:teuthology.orchestra.run.gibba002.stdout: "old_weight": 0 2021-02-10T11:29:52.443 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:29:52.443 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:29:52.443 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 7, 2021-02-10T11:29:52.443 INFO:teuthology.orchestra.run.gibba002.stdout: "down_stamp": "0.000000", 2021-02-10T11:29:52.443 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_probability": 0, 2021-02-10T11:29:52.444 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_interval": 0, 2021-02-10T11:29:52.444 INFO:teuthology.orchestra.run.gibba002.stdout: "features": 4611087854035861503, 2021-02-10T11:29:52.444 INFO:teuthology.orchestra.run.gibba002.stdout: "old_weight": 0 2021-02-10T11:29:52.444 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:29:52.444 INFO:teuthology.orchestra.run.gibba002.stdout: ], 2021-02-10T11:29:52.444 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_upmap": [], 2021-02-10T11:29:52.444 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_upmap_items": [], 2021-02-10T11:29:52.444 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_temp": [], 2021-02-10T11:29:52.444 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_temp": [], 2021-02-10T11:29:52.444 INFO:teuthology.orchestra.run.gibba002.stdout: "blacklist": {}, 2021-02-10T11:29:52.444 INFO:teuthology.orchestra.run.gibba002.stdout: "erasure_code_profiles": { 2021-02-10T11:29:52.444 INFO:teuthology.orchestra.run.gibba002.stdout: "default": { 2021-02-10T11:29:52.444 INFO:teuthology.orchestra.run.gibba002.stdout: "crush-failure-domain": "osd", 2021-02-10T11:29:52.444 INFO:teuthology.orchestra.run.gibba002.stdout: "k": "2", 2021-02-10T11:29:52.445 INFO:teuthology.orchestra.run.gibba002.stdout: "m": "1", 2021-02-10T11:29:52.445 INFO:teuthology.orchestra.run.gibba002.stdout: "plugin": "jerasure", 2021-02-10T11:29:52.445 INFO:teuthology.orchestra.run.gibba002.stdout: "ruleset-failure-domain": "osd", 2021-02-10T11:29:52.445 INFO:teuthology.orchestra.run.gibba002.stdout: "technique": "reed_sol_van" 2021-02-10T11:29:52.445 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:29:52.445 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:29:52.445 INFO:teuthology.orchestra.run.gibba002.stdout: "removed_snaps_queue": [], 2021-02-10T11:29:52.445 INFO:teuthology.orchestra.run.gibba002.stdout: "new_removed_snaps": [], 2021-02-10T11:29:52.445 INFO:teuthology.orchestra.run.gibba002.stdout: "new_purged_snaps": [], 2021-02-10T11:29:52.445 INFO:teuthology.orchestra.run.gibba002.stdout: "crush_node_flags": {}, 2021-02-10T11:29:52.445 INFO:teuthology.orchestra.run.gibba002.stdout: "device_class_flags": {} 2021-02-10T11:29:52.445 INFO:teuthology.orchestra.run.gibba002.stdout:} 2021-02-10T11:29:52.446 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:52.369 7fbaa888d700 1 -- 172.21.2.102:0/2543870723 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fba880186c0 msgr2=0x7fba8801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:52.446 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:52.369 7fbaa888d700 1 --2- 172.21.2.102:0/2543870723 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fba880186c0 0x7fba8801abc0 crc :-1 s=READY pgs=848 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:52.446 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:52.369 7fbaa888d700 1 -- 172.21.2.102:0/2543870723 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbaa40b7d70 msgr2=0x7fbaa40b8170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:52.446 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:52.369 7fbaa888d700 1 --2- 172.21.2.102:0/2543870723 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbaa40b7d70 0x7fbaa40b8170 secure :-1 s=READY pgs=1890 cs=0 l=1 rev1=1 rx=0x7fba8c00b660 tx=0x7fba8c00ba00).stop 2021-02-10T11:29:52.446 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:52.369 7fbaa888d700 1 -- 172.21.2.102:0/2543870723 shutdown_connections 2021-02-10T11:29:52.446 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:52.369 7fbaa888d700 1 --2- 172.21.2.102:0/2543870723 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fba880186c0 0x7fba8801abc0 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:52.446 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:52.369 7fbaa888d700 1 --2- 172.21.2.102:0/2543870723 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fbaa41a2770 0x7fbaa41a2b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:52.446 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:52.369 7fbaa888d700 1 --2- 172.21.2.102:0/2543870723 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbaa40b7d70 0x7fbaa40b8170 unknown :-1 s=CLOSED pgs=1890 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:52.446 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:52.369 7fbaa888d700 1 --2- 172.21.2.102:0/2543870723 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbaa4195000 0x7fbaa40b7750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:52.446 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:52.369 7fbaa888d700 1 -- 172.21.2.102:0/2543870723 >> 172.21.2.102:0/2543870723 conn(0x7fbaa419c5c0 msgr2=0x7fbaa40ee260 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:29:52.446 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:52.369 7fbaa888d700 1 -- 172.21.2.102:0/2543870723 shutdown_connections 2021-02-10T11:29:52.446 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:52.369 7fbaa888d700 1 -- 172.21.2.102:0/2543870723 wait complete. 2021-02-10T11:29:52.447 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:29:52.890 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:52.872 7f78af315700 1 Processor -- start 2021-02-10T11:29:52.891 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:52.872 7f78af315700 1 -- start start 2021-02-10T11:29:52.891 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:52.873 7f78af315700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f78a80d0340 0x7f78a80ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:52.891 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:52.873 7f78af315700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78a80f0b10 con 0x7f78a80d1c70 2021-02-10T11:29:52.891 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:52.873 7f78af315700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78a81b0be0 con 0x7f78a80d1870 2021-02-10T11:29:52.891 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:52.873 7f78af315700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78a80e5ed0 con 0x7f78a80d0340 2021-02-10T11:29:52.891 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:52.873 7f78ad932700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f78a80d1c70 0x7f78a80efc60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:52690/0 (socket says 172.21.2.102:52690) 2021-02-10T11:29:52.891 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:52.873 7f78ad932700 1 -- 172.21.2.102:0/361901855 learned_addr learned my addr 172.21.2.102:0/361901855 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:29:52.891 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:52.873 7f789ffff700 1 -- 172.21.2.102:0/361901855 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4277605511 0 0) 0x7f78a0001500 con 0x7f78a80d1c70 2021-02-10T11:29:52.891 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:52.873 7f789ffff700 1 -- 172.21.2.102:0/361901855 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7894002fe0 con 0x7f78a80d1c70 2021-02-10T11:29:52.891 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:52.874 7f789ffff700 1 -- 172.21.2.102:0/361901855 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2052731795 0 0) 0x7f78a0001460 con 0x7f78a80d1c70 2021-02-10T11:29:52.891 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:52.874 7f789ffff700 1 -- 172.21.2.102:0/361901855 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f78a80d0340 msgr2=0x7f78a80ef640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:29:52.891 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:52.874 7f789ffff700 1 --2- 172.21.2.102:0/361901855 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f78a80d0340 0x7f78a80ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:52.891 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:52.874 7f789ffff700 1 -- 172.21.2.102:0/361901855 >> v1:172.21.2.113:6789/0 conn(0x7f78a80d1870 legacy=0x7f78a81b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:29:52.892 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:52.874 7f789ffff700 1 -- 172.21.2.102:0/361901855 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78a80f0450 con 0x7f78a80d1c70 2021-02-10T11:29:52.892 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:52.874 7f789ffff700 1 -- 172.21.2.102:0/361901855 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7f78a0001980 con 0x7f78a80d1c70 2021-02-10T11:29:52.892 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:52.874 7f789ffff700 1 -- 172.21.2.102:0/361901855 >> v1:172.21.2.102:6789/0 conn(0x7f78a80d1c70 legacy=0x7f78a80efc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:52.892 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:52.874 7f789ffff700 1 --2- 172.21.2.102:0/361901855 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7894003b10 0x7f7894005f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:52.892 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:52.874 7f789ffff700 1 --2- 172.21.2.102:0/361901855 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f78940067a0 0x7f7894008c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:52.892 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:52.874 7f789ffff700 1 --2- 172.21.2.102:0/361901855 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7894009520 0x7f789400ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:52.892 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:52.874 7f789ffff700 1 -- 172.21.2.102:0/361901855 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f789400c2d0 con 0x7f78940067a0 2021-02-10T11:29:52.892 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:52.874 7f789ffff700 1 -- 172.21.2.102:0/361901855 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f789400c4f0 con 0x7f7894009520 2021-02-10T11:29:52.892 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:52.874 7f789ffff700 1 -- 172.21.2.102:0/361901855 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f789400c710 con 0x7f7894003b10 2021-02-10T11:29:52.892 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:52.874 7f78ac930700 1 --2- 172.21.2.102:0/361901855 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f78940067a0 0x7f7894008c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:52.892 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:52.874 7f78ad131700 1 --2- 172.21.2.102:0/361901855 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7894003b10 0x7f7894005f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:52.892 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:52.874 7f78ad932700 1 --2- 172.21.2.102:0/361901855 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7894009520 0x7f789400ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:52.892 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:52.874 7f78ac930700 1 -- 172.21.2.102:0/361901855 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7894003b10 msgr2=0x7f7894005f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:29:52.893 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:52.874 7f78ac930700 1 --2- 172.21.2.102:0/361901855 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7894003b10 0x7f7894005f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:52.893 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:52.874 7f78ac930700 1 -- 172.21.2.102:0/361901855 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7894009520 msgr2=0x7f789400ba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:29:52.893 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:52.874 7f78ac930700 1 --2- 172.21.2.102:0/361901855 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7894009520 0x7f789400ba20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:52.893 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:52.874 7f78ac930700 1 -- 172.21.2.102:0/361901855 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f789400c9b0 con 0x7f78940067a0 2021-02-10T11:29:52.893 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:52.874 7f78ad932700 1 --2- 172.21.2.102:0/361901855 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7894009520 0x7f789400ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:29:52.893 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:52.874 7f78ac930700 1 --2- 172.21.2.102:0/361901855 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f78940067a0 0x7f7894008c80 secure :-1 s=READY pgs=1892 cs=0 l=1 rev1=1 rx=0x7f7898006ab0 tx=0x7f789804fbc0).ready entity=mon.0 client_cookie=809f474a2f87c8de server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:29:52.893 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:52.874 7f789ffff700 1 -- 172.21.2.102:0/361901855 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7898006350 con 0x7f78940067a0 2021-02-10T11:29:52.893 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:52.875 7f789ffff700 1 -- 172.21.2.102:0/361901855 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7898002080 con 0x7f78940067a0 2021-02-10T11:29:52.893 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:52.875 7f789ffff700 1 -- 172.21.2.102:0/361901855 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f78980024e0 con 0x7f78940067a0 2021-02-10T11:29:52.893 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:52.875 7f78af315700 1 -- 172.21.2.102:0/361901855 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f78940067a0 msgr2=0x7f7894008c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:52.893 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:52.875 7f78af315700 1 --2- 172.21.2.102:0/361901855 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f78940067a0 0x7f7894008c80 secure :-1 s=READY pgs=1892 cs=0 l=1 rev1=1 rx=0x7f7898006ab0 tx=0x7f789804fbc0).stop 2021-02-10T11:29:52.893 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:52.875 7f78ad932700 1 -- 172.21.2.102:0/361901855 reap_dead start 2021-02-10T11:29:52.893 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:52.875 7f78af315700 1 -- 172.21.2.102:0/361901855 shutdown_connections 2021-02-10T11:29:52.894 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:52.875 7f78af315700 1 -- 172.21.2.102:0/361901855 >> 172.21.2.102:0/361901855 conn(0x7f78a80ce450 msgr2=0x7f78a80d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:29:52.894 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:52.875 7f78af315700 1 -- 172.21.2.102:0/361901855 shutdown_connections 2021-02-10T11:29:52.894 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:52.875 7f78af315700 1 -- 172.21.2.102:0/361901855 wait complete. 2021-02-10T11:29:52.894 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:52.890 7f78af315700 1 Processor -- start 2021-02-10T11:29:52.894 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:52.890 7f78af315700 1 -- start start 2021-02-10T11:29:52.895 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:52.890 7f78af315700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f78a80f0280 0x7f78a80a2310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:52.895 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:52.890 7f78af315700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f78a80f0680 0x7f78a80a2930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:52.895 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:52.890 7f78af315700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f78a80ec180 0x7f78a80ec580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:52.895 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:52.890 7f78af315700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78a80efd00 con 0x7f78a80f0680 2021-02-10T11:29:52.895 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:52.890 7f78af315700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78a80ecba0 con 0x7f78a80ec180 2021-02-10T11:29:52.896 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:52.890 7f78af315700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78a80ecd80 con 0x7f78a80f0280 2021-02-10T11:29:52.896 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:52.891 7f78ac930700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f78a80f0680 0x7f78a80a2930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:52.896 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:52.891 7f78ac930700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f78a80f0680 0x7f78a80a2930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:60476/0 (socket says 172.21.2.102:60476) 2021-02-10T11:29:52.896 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:52.891 7f78ac930700 1 -- 172.21.2.102:0/513344273 learned_addr learned my addr 172.21.2.102:0/513344273 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:29:52.896 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:52.891 7f78ac930700 1 -- 172.21.2.102:0/513344273 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f78a80f0280 msgr2=0x7f78a80a2310 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:29:52.896 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:52.891 7f78ac930700 1 --2- 172.21.2.102:0/513344273 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f78a80f0280 0x7f78a80a2310 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:52.896 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:52.891 7f78ac930700 1 -- 172.21.2.102:0/513344273 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f78a80ec180 msgr2=0x7f78a80ec580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:29:52.896 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:52.891 7f78ac930700 1 --2- 172.21.2.102:0/513344273 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f78a80ec180 0x7f78a80ec580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:52.896 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:52.891 7f78ac930700 1 -- 172.21.2.102:0/513344273 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78a80ecfa0 con 0x7f78a80f0680 2021-02-10T11:29:52.896 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:52.891 7f78ac930700 1 --2- 172.21.2.102:0/513344273 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f78a80f0680 0x7f78a80a2930 secure :-1 s=READY pgs=1893 cs=0 l=1 rev1=1 rx=0x7f7898006550 tx=0x7f7898009a30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:29:52.896 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:52.891 7f789dffb700 1 -- 172.21.2.102:0/513344273 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7898050020 con 0x7f78a80f0680 2021-02-10T11:29:52.896 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:52.891 7f789dffb700 1 -- 172.21.2.102:0/513344273 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7898050320 con 0x7f78a80f0680 2021-02-10T11:29:52.896 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:52.891 7f78af315700 1 -- 172.21.2.102:0/513344273 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78a80ecfa0 con 0x7f78a80f0680 2021-02-10T11:29:52.897 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:52.891 7f789dffb700 1 -- 172.21.2.102:0/513344273 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f789804f5c0 con 0x7f78a80f0680 2021-02-10T11:29:52.897 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:52.891 7f78af315700 1 -- 172.21.2.102:0/513344273 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78a80ecfa0 con 0x7f78a80f0680 2021-02-10T11:29:52.897 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:52.892 7f789dffb700 1 -- 172.21.2.102:0/513344273 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f789804f3b0 con 0x7f78a80f0680 2021-02-10T11:29:52.897 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:52.892 7f789dffb700 1 --2- 172.21.2.102:0/513344273 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7888018730 0x7f788801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:52.897 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:52.892 7f789dffb700 1 -- 172.21.2.102:0/513344273 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f78980767e0 con 0x7f78a80f0680 2021-02-10T11:29:52.897 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:52.892 7f78ad131700 1 --2- 172.21.2.102:0/513344273 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7888018730 0x7f788801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:52.897 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:52.892 7f78ad131700 1 --2- 172.21.2.102:0/513344273 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7888018730 0x7f788801ac10 crc :-1 s=READY pgs=849 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:29:52.923 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:52.923 7f78af315700 1 -- 172.21.2.102:0/513344273 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78a80ebc60 con 0x7f78a80f0680 2021-02-10T11:29:52.926 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:52.925 7f789dffb700 1 -- 172.21.2.102:0/513344273 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f7898009e20 con 0x7f78a80f0680 2021-02-10T11:29:53.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:53.054 7f78af315700 1 -- 172.21.2.102:0/513344273 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f78a80ebc60 con 0x7f78a80f0680 2021-02-10T11:29:53.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:53.055 7f789dffb700 1 -- 172.21.2.102:0/513344273 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+2101 (secure 0 0 0) 0x7f7898050320 con 0x7f78a80f0680 2021-02-10T11:29:53.056 INFO:teuthology.orchestra.run.gibba002.stderr:dumped fsmap epoch 66 2021-02-10T11:29:53.056 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:29:53.056 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":66,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6110,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.102:6835/1471871278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6834","nonce":1471871278},{"type":"v1","addr":"172.21.2.102:6835","nonce":1471871278}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":61}],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:27:41.479375","modified":"2021-02-10 11:27:43.490605","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6107},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6107":{"gid":6107,"name":"b","rank":0,"incarnation":62,"state":"up:active","state_seq":3,"addr":"172.21.2.113:6833/4220238062","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6832","nonce":4220238062},{"type":"v1","addr":"172.21.2.113:6833","nonce":4220238062}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":8}]} 2021-02-10T11:29:53.058 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:53.057 7f78af315700 1 -- 172.21.2.102:0/513344273 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7888018730 msgr2=0x7f788801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:53.058 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:53.057 7f78af315700 1 --2- 172.21.2.102:0/513344273 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7888018730 0x7f788801ac10 crc :-1 s=READY pgs=849 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:53.058 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:53.057 7f78af315700 1 -- 172.21.2.102:0/513344273 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f78a80f0680 msgr2=0x7f78a80a2930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:53.058 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:53.057 7f78af315700 1 --2- 172.21.2.102:0/513344273 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f78a80f0680 0x7f78a80a2930 secure :-1 s=READY pgs=1893 cs=0 l=1 rev1=1 rx=0x7f7898006550 tx=0x7f7898009a30).stop 2021-02-10T11:29:53.058 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:53.057 7f78af315700 1 -- 172.21.2.102:0/513344273 shutdown_connections 2021-02-10T11:29:53.058 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:53.057 7f78af315700 1 --2- 172.21.2.102:0/513344273 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7888018730 0x7f788801ac10 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:53.059 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:53.057 7f78af315700 1 --2- 172.21.2.102:0/513344273 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f78a80ec180 0x7f78a80ec580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:53.059 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:53.057 7f78af315700 1 --2- 172.21.2.102:0/513344273 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f78a80f0680 0x7f78a80a2930 unknown :-1 s=CLOSED pgs=1893 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:53.059 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:53.057 7f78af315700 1 --2- 172.21.2.102:0/513344273 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f78a80f0280 0x7f78a80a2310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:53.059 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:53.057 7f78af315700 1 -- 172.21.2.102:0/513344273 >> 172.21.2.102:0/513344273 conn(0x7f78a80c2130 msgr2=0x7f78a809e3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:29:53.059 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:53.057 7f78af315700 1 -- 172.21.2.102:0/513344273 shutdown_connections 2021-02-10T11:29:53.059 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:53.057 7f78af315700 1 -- 172.21.2.102:0/513344273 wait complete. 2021-02-10T11:29:53.066 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs fail cephfs 2021-02-10T11:29:53.497 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:53.480 7fcc36796700 1 Processor -- start 2021-02-10T11:29:53.498 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:53.480 7fcc36796700 1 -- start start 2021-02-10T11:29:53.498 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:53.481 7fcc36796700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fcc300d1870 0x7fcc301b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:53.498 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:53.481 7fcc36796700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcc301b0a10 0x7fcc300cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:53.498 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:53.481 7fcc36796700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc300f0b10 con 0x7fcc300d1c70 2021-02-10T11:29:53.498 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:53.481 7fcc36796700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc300e5ed0 con 0x7fcc300d1870 2021-02-10T11:29:53.498 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:53.481 7fcc36796700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc300f0650 con 0x7fcc301b0a10 2021-02-10T11:29:53.498 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:53.481 7fcc2ffff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fcc300d1870 0x7fcc301b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:53.498 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:53.481 7fcc2ffff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fcc300d1870 0x7fcc301b03f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:60482/0 (socket says 172.21.2.102:60482) 2021-02-10T11:29:53.498 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:53.481 7fcc34db3700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fcc300d1c70 0x7fcc300cf950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:52706/0 (socket says 172.21.2.102:52706) 2021-02-10T11:29:53.498 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:53.481 7fcc2ffff700 1 -- 172.21.2.102:0/3267477198 learned_addr learned my addr 172.21.2.102:0/3267477198 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:29:53.498 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:53.481 7fcc2ffff700 1 -- 172.21.2.102:0/3267477198 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcc301b0a10 msgr2=0x7fcc300cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:29:53.498 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:53.481 7fcc2f7fe700 1 --2- 172.21.2.102:0/3267477198 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcc301b0a10 0x7fcc300cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:53.499 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:53.481 7fcc2ffff700 1 --2- 172.21.2.102:0/3267477198 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcc301b0a10 0x7fcc300cf330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:53.499 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:53.481 7fcc2effd700 1 -- 172.21.2.102:0/3267477198 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3267723312 0 0) 0x7fcc28001480 con 0x7fcc300d1c70 2021-02-10T11:29:53.499 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:53.481 7fcc2ffff700 1 -- 172.21.2.102:0/3267477198 >> v1:172.21.2.102:6789/0 conn(0x7fcc300d1c70 legacy=0x7fcc300cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:53.499 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:53.481 7fcc2ffff700 1 -- 172.21.2.102:0/3267477198 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc300f0870 con 0x7fcc300d1870 2021-02-10T11:29:53.499 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:53.481 7fcc2f7fe700 1 --2- 172.21.2.102:0/3267477198 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcc301b0a10 0x7fcc300cf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:29:53.499 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:53.481 7fcc2ffff700 1 --2- 172.21.2.102:0/3267477198 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcc300d1870 0x7fcc301b03f0 secure :-1 s=READY pgs=1895 cs=0 l=1 rev1=1 rx=0x7fcc20006700 tx=0x7fcc20009b80).ready entity=mon.0 client_cookie=208701e5fcffa180 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:29:53.499 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:53.482 7fcc2effd700 1 -- 172.21.2.102:0/3267477198 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcc2000c0c0 con 0x7fcc300d1870 2021-02-10T11:29:53.499 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:53.482 7fcc2effd700 1 -- 172.21.2.102:0/3267477198 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc2000c440 con 0x7fcc300d1870 2021-02-10T11:29:53.499 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:53.482 7fcc2effd700 1 -- 172.21.2.102:0/3267477198 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcc2000c960 con 0x7fcc300d1870 2021-02-10T11:29:53.499 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:53.482 7fcc36796700 1 -- 172.21.2.102:0/3267477198 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcc300d1870 msgr2=0x7fcc301b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:53.499 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:53.482 7fcc36796700 1 --2- 172.21.2.102:0/3267477198 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcc300d1870 0x7fcc301b03f0 secure :-1 s=READY pgs=1895 cs=0 l=1 rev1=1 rx=0x7fcc20006700 tx=0x7fcc20009b80).stop 2021-02-10T11:29:53.499 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:53.482 7fcc36796700 1 -- 172.21.2.102:0/3267477198 shutdown_connections 2021-02-10T11:29:53.499 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:53.482 7fcc36796700 1 --2- 172.21.2.102:0/3267477198 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcc301b0a10 0x7fcc300cf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:53.499 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:53.482 7fcc36796700 1 --2- 172.21.2.102:0/3267477198 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcc300d1870 0x7fcc301b03f0 unknown :-1 s=CLOSED pgs=1895 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:53.500 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:53.482 7fcc36796700 1 -- 172.21.2.102:0/3267477198 >> 172.21.2.102:0/3267477198 conn(0x7fcc300ce450 msgr2=0x7fcc300d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:29:53.500 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:53.482 7fcc36796700 1 -- 172.21.2.102:0/3267477198 shutdown_connections 2021-02-10T11:29:53.500 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:53.482 7fcc36796700 1 -- 172.21.2.102:0/3267477198 wait complete. 2021-02-10T11:29:53.500 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:53.497 7fcc36796700 1 Processor -- start 2021-02-10T11:29:53.500 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:53.497 7fcc36796700 1 -- start start 2021-02-10T11:29:53.501 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:53.497 7fcc36796700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fcc301a06a0 0x7fcc3009f380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:53.501 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:53.497 7fcc36796700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcc3009f9a0 0x7fcc3008b560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:53.501 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:53.497 7fcc36796700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcc3008bb80 0x7fcc3008bf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:53.501 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:53.497 7fcc36796700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc3008c780 con 0x7fcc3009f9a0 2021-02-10T11:29:53.501 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:53.497 7fcc36796700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc3009fda0 con 0x7fcc301a06a0 2021-02-10T11:29:53.501 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:53.497 7fcc36796700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc3008cd00 con 0x7fcc3008bb80 2021-02-10T11:29:53.501 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:53.498 7fcc2f7fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcc3009f9a0 0x7fcc3008b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:53.502 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:53.498 7fcc2f7fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcc3009f9a0 0x7fcc3008b560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:60488/0 (socket says 172.21.2.102:60488) 2021-02-10T11:29:53.502 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:53.498 7fcc2f7fe700 1 -- 172.21.2.102:0/3194196695 learned_addr learned my addr 172.21.2.102:0/3194196695 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:29:53.502 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:53.498 7fcc2f7fe700 1 -- 172.21.2.102:0/3194196695 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcc3008bb80 msgr2=0x7fcc3008bf80 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:29:53.502 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:53.498 7fcc2f7fe700 1 --2- 172.21.2.102:0/3194196695 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcc3008bb80 0x7fcc3008bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:53.502 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:53.498 7fcc2f7fe700 1 -- 172.21.2.102:0/3194196695 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fcc301a06a0 msgr2=0x7fcc3009f380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:53.502 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:53.498 7fcc2f7fe700 1 --2- 172.21.2.102:0/3194196695 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fcc301a06a0 0x7fcc3009f380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:53.502 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:53.498 7fcc2f7fe700 1 -- 172.21.2.102:0/3194196695 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc3008cf20 con 0x7fcc3009f9a0 2021-02-10T11:29:53.502 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:53.498 7fcc2f7fe700 1 --2- 172.21.2.102:0/3194196695 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcc3009f9a0 0x7fcc3008b560 secure :-1 s=READY pgs=1896 cs=0 l=1 rev1=1 rx=0x7fcc24006f20 tx=0x7fcc2400b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:29:53.502 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:53.498 7fcc2cff9700 1 -- 172.21.2.102:0/3194196695 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcc2400bd80 con 0x7fcc3009f9a0 2021-02-10T11:29:53.502 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:53.499 7fcc2cff9700 1 -- 172.21.2.102:0/3194196695 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc24000c90 con 0x7fcc3009f9a0 2021-02-10T11:29:53.502 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:53.499 7fcc2cff9700 1 -- 172.21.2.102:0/3194196695 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcc240022b0 con 0x7fcc3009f9a0 2021-02-10T11:29:53.502 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:53.499 7fcc36796700 1 -- 172.21.2.102:0/3194196695 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc3008cf20 con 0x7fcc3009f9a0 2021-02-10T11:29:53.502 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:53.499 7fcc36796700 1 -- 172.21.2.102:0/3194196695 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc3008cf20 con 0x7fcc3009f9a0 2021-02-10T11:29:53.503 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:53.499 7fcc2cff9700 1 -- 172.21.2.102:0/3194196695 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fcc2400a000 con 0x7fcc3009f9a0 2021-02-10T11:29:53.503 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:53.499 7fcc2cff9700 1 --2- 172.21.2.102:0/3194196695 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcc10018730 0x7fcc1001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:53.503 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:53.499 7fcc2cff9700 1 -- 172.21.2.102:0/3194196695 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fcc24032680 con 0x7fcc3009f9a0 2021-02-10T11:29:53.503 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:53.499 7fcc2ffff700 1 --2- 172.21.2.102:0/3194196695 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcc10018730 0x7fcc1001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:53.503 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:53.499 7fcc2ffff700 1 --2- 172.21.2.102:0/3194196695 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcc10018730 0x7fcc1001abc0 crc :-1 s=READY pgs=850 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:29:53.531 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:53.530 7fcc36796700 1 -- 172.21.2.102:0/3194196695 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc300ebc50 con 0x7fcc3009f9a0 2021-02-10T11:29:53.533 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:53.533 7fcc2cff9700 1 -- 172.21.2.102:0/3194196695 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fcc24006bb0 con 0x7fcc3009f9a0 2021-02-10T11:29:53.660 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:53.659 7fcc36796700 1 -- 172.21.2.102:0/3194196695 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fcc300f0b10 con 0x7fcc3009f9a0 2021-02-10T11:29:54.076 INFO:tasks.ceph.mon.a.gibba002.stderr:2021-02-10 11:29:54.076 7f8dd2755700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-02-10T11:29:54.083 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:54.083 7fcc2cff9700 1 -- 172.21.2.102:0/3194196695 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v67) v1 ==== 161+0+0 (secure 0 0 0) 0x7fcc2400a000 con 0x7fcc3009f9a0 2021-02-10T11:29:54.125 INFO:teuthology.orchestra.run.gibba002.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2021-02-10T11:29:54.127 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:54.127 7fcc36796700 1 -- 172.21.2.102:0/3194196695 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcc10018730 msgr2=0x7fcc1001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:54.127 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:54.127 7fcc36796700 1 --2- 172.21.2.102:0/3194196695 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcc10018730 0x7fcc1001abc0 crc :-1 s=READY pgs=850 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:54.127 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:54.127 7fcc36796700 1 -- 172.21.2.102:0/3194196695 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcc3009f9a0 msgr2=0x7fcc3008b560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:54.127 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:54.127 7fcc36796700 1 --2- 172.21.2.102:0/3194196695 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcc3009f9a0 0x7fcc3008b560 secure :-1 s=READY pgs=1896 cs=0 l=1 rev1=1 rx=0x7fcc24006f20 tx=0x7fcc2400b990).stop 2021-02-10T11:29:54.128 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:54.127 7fcc36796700 1 -- 172.21.2.102:0/3194196695 shutdown_connections 2021-02-10T11:29:54.128 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:54.127 7fcc36796700 1 --2- 172.21.2.102:0/3194196695 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcc3008bb80 0x7fcc3008bf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:54.128 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:54.127 7fcc36796700 1 --2- 172.21.2.102:0/3194196695 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcc3009f9a0 0x7fcc3008b560 unknown :-1 s=CLOSED pgs=1896 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:54.128 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:54.127 7fcc36796700 1 --2- 172.21.2.102:0/3194196695 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcc10018730 0x7fcc1001abc0 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:54.128 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:54.127 7fcc36796700 1 --2- 172.21.2.102:0/3194196695 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fcc301a06a0 0x7fcc3009f380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:54.129 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:54.127 7fcc36796700 1 -- 172.21.2.102:0/3194196695 >> 172.21.2.102:0/3194196695 conn(0x7fcc300b83b0 msgr2=0x7fcc300a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:29:54.129 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:54.127 7fcc36796700 1 -- 172.21.2.102:0/3194196695 shutdown_connections 2021-02-10T11:29:54.129 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:54.127 7fcc36796700 1 -- 172.21.2.102:0/3194196695 wait complete. 2021-02-10T11:29:54.131 INFO:tasks.ceph.mds.b.gibba013.stdout:starting mds.b at 2021-02-10T11:29:54.135 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:29:54.565 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:54.548 7f9b880b4700 1 Processor -- start 2021-02-10T11:29:54.565 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:54.548 7f9b880b4700 1 -- start start 2021-02-10T11:29:54.565 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:54.548 7f9b880b4700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9b800d0340 0x7f9b800ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:54.565 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:54.548 7f9b880b4700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f9b800d1c70 0x7f9b800efc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:54.566 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:54.548 7f9b880b4700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b800f0b10 con 0x7f9b800d1870 2021-02-10T11:29:54.566 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:54.548 7f9b880b4700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b800e5ed0 con 0x7f9b800d1c70 2021-02-10T11:29:54.566 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:54.548 7f9b880b4700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b801b0be0 con 0x7f9b800d0340 2021-02-10T11:29:54.566 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:54.549 7f9b866d1700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f9b800d1c70 0x7f9b800efc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:54.566 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:54.549 7f9b866d1700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f9b800d1c70 0x7f9b800efc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:60498/0 (socket says 172.21.2.102:60498) 2021-02-10T11:29:54.566 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:54.549 7f9b866d1700 1 -- 172.21.2.102:0/1208516709 learned_addr learned my addr 172.21.2.102:0/1208516709 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:29:54.566 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:54.549 7f9b866d1700 1 -- 172.21.2.102:0/1208516709 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9b800d0340 msgr2=0x7f9b800ef640 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:29:54.566 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:54.549 7f9b866d1700 1 --2- 172.21.2.102:0/1208516709 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9b800d0340 0x7f9b800ef640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:54.566 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:54.549 7f9b866d1700 1 -- 172.21.2.102:0/1208516709 >> v1:172.21.2.113:6789/0 conn(0x7f9b800d1870 legacy=0x7f9b801b03f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:29:54.566 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:54.549 7f9b866d1700 1 -- 172.21.2.102:0/1208516709 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b800f20a0 con 0x7f9b800d1c70 2021-02-10T11:29:54.566 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:54.549 7f9b866d1700 1 --2- 172.21.2.102:0/1208516709 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9b800d1c70 0x7f9b800efc60 secure :-1 s=READY pgs=1898 cs=0 l=1 rev1=1 rx=0x7f9b78006f00 tx=0x7f9b78009f10).ready entity=mon.0 client_cookie=e5b2bc165bed5e1b server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:29:54.567 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:54.549 7f9b84ece700 1 -- 172.21.2.102:0/1208516709 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9b7800b3b0 con 0x7f9b800d1c70 2021-02-10T11:29:54.567 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:54.549 7f9b84ece700 1 -- 172.21.2.102:0/1208516709 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b7800b630 con 0x7f9b800d1c70 2021-02-10T11:29:54.567 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:54.549 7f9b84ece700 1 -- 172.21.2.102:0/1208516709 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9b7800bb50 con 0x7f9b800d1c70 2021-02-10T11:29:54.567 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:54.549 7f9b880b4700 1 -- 172.21.2.102:0/1208516709 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9b800d1c70 msgr2=0x7f9b800efc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:54.567 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:54.549 7f9b880b4700 1 --2- 172.21.2.102:0/1208516709 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9b800d1c70 0x7f9b800efc60 secure :-1 s=READY pgs=1898 cs=0 l=1 rev1=1 rx=0x7f9b78006f00 tx=0x7f9b78009f10).stop 2021-02-10T11:29:54.567 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:54.550 7f9b880b4700 1 -- 172.21.2.102:0/1208516709 shutdown_connections 2021-02-10T11:29:54.567 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:54.550 7f9b880b4700 1 --2- 172.21.2.102:0/1208516709 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9b800d0340 0x7f9b800ef640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:54.567 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:54.550 7f9b880b4700 1 --2- 172.21.2.102:0/1208516709 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9b800d1c70 0x7f9b800efc60 unknown :-1 s=CLOSED pgs=1898 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:54.567 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:54.550 7f9b880b4700 1 -- 172.21.2.102:0/1208516709 >> 172.21.2.102:0/1208516709 conn(0x7f9b800ce450 msgr2=0x7f9b800d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:29:54.567 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:54.550 7f9b880b4700 1 -- 172.21.2.102:0/1208516709 shutdown_connections 2021-02-10T11:29:54.567 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:54.550 7f9b880b4700 1 -- 172.21.2.102:0/1208516709 wait complete. 2021-02-10T11:29:54.569 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:54.565 7f9b880b4700 1 Processor -- start 2021-02-10T11:29:54.569 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:54.565 7f9b880b4700 1 -- start start 2021-02-10T11:29:54.570 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:54.565 7f9b880b4700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9b80199690 0x7f9b800af0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:54.570 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:54.565 7f9b880b4700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f9b800af710 0x7f9b801aca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:54.570 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:54.565 7f9b880b4700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9b800afb10 0x7f9b801ad060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:54.570 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:54.565 7f9b880b4700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b8009a590 con 0x7f9b80199690 2021-02-10T11:29:54.570 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:54.565 7f9b880b4700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b8009a7b0 con 0x7f9b800af710 2021-02-10T11:29:54.570 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:54.565 7f9b880b4700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b801991e0 con 0x7f9b800afb10 2021-02-10T11:29:54.570 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:54.566 7f9b85ed0700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9b80199690 0x7f9b800af0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:54.570 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:54.566 7f9b85ed0700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9b80199690 0x7f9b800af0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:60500/0 (socket says 172.21.2.102:60500) 2021-02-10T11:29:54.570 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:54.566 7f9b85ed0700 1 -- 172.21.2.102:0/3277963903 learned_addr learned my addr 172.21.2.102:0/3277963903 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:29:54.570 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:54.566 7f9b866d1700 1 --2- 172.21.2.102:0/3277963903 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9b800afb10 0x7f9b801ad060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:54.570 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:54.566 7f9b856cf700 1 --2- 172.21.2.102:0/3277963903 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f9b800af710 0x7f9b801aca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:54.570 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:54.566 7f9b85ed0700 1 -- 172.21.2.102:0/3277963903 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9b800afb10 msgr2=0x7f9b801ad060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:54.571 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:54.566 7f9b85ed0700 1 --2- 172.21.2.102:0/3277963903 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9b800afb10 0x7f9b801ad060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:54.571 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:54.566 7f9b85ed0700 1 -- 172.21.2.102:0/3277963903 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f9b800af710 msgr2=0x7f9b801aca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:54.571 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:54.566 7f9b85ed0700 1 --2- 172.21.2.102:0/3277963903 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f9b800af710 0x7f9b801aca40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:54.571 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:54.566 7f9b85ed0700 1 -- 172.21.2.102:0/3277963903 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b801ad680 con 0x7f9b80199690 2021-02-10T11:29:54.571 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:54.566 7f9b866d1700 1 --2- 172.21.2.102:0/3277963903 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9b800afb10 0x7f9b801ad060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:29:54.571 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:54.566 7f9b856cf700 1 --2- 172.21.2.102:0/3277963903 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f9b800af710 0x7f9b801aca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:29:54.571 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:54.566 7f9b85ed0700 1 --2- 172.21.2.102:0/3277963903 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9b80199690 0x7f9b800af0f0 secure :-1 s=READY pgs=1899 cs=0 l=1 rev1=1 rx=0x7f9b7c00b5b0 tx=0x7f9b7c00b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:29:54.571 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:54.566 7f9b767fc700 1 -- 172.21.2.102:0/3277963903 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9b7c00bd40 con 0x7f9b80199690 2021-02-10T11:29:54.571 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:54.567 7f9b880b4700 1 -- 172.21.2.102:0/3277963903 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b801ad680 con 0x7f9b80199690 2021-02-10T11:29:54.571 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:54.567 7f9b767fc700 1 -- 172.21.2.102:0/3277963903 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b7c006020 con 0x7f9b80199690 2021-02-10T11:29:54.571 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:54.567 7f9b767fc700 1 -- 172.21.2.102:0/3277963903 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9b7c0054a0 con 0x7f9b80199690 2021-02-10T11:29:54.572 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:54.567 7f9b880b4700 1 -- 172.21.2.102:0/3277963903 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b801ad680 con 0x7f9b80199690 2021-02-10T11:29:54.572 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:54.567 7f9b767fc700 1 -- 172.21.2.102:0/3277963903 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f9b7c006200 con 0x7f9b80199690 2021-02-10T11:29:54.572 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:54.567 7f9b767fc700 1 --2- 172.21.2.102:0/3277963903 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f9b68018700 0x7f9b6801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:54.572 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:54.567 7f9b767fc700 1 -- 172.21.2.102:0/3277963903 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 5659+0+0 (secure 0 0 0) 0x7f9b7c032800 con 0x7f9b80199690 2021-02-10T11:29:54.572 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:54.567 7f9b856cf700 1 --2- 172.21.2.102:0/3277963903 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f9b68018700 0x7f9b6801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:54.572 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:54.568 7f9b856cf700 1 --2- 172.21.2.102:0/3277963903 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f9b68018700 0x7f9b6801abc0 crc :-1 s=READY pgs=851 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:29:54.599 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:54.598 7f9b880b4700 1 -- 172.21.2.102:0/3277963903 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b801adc80 con 0x7f9b80199690 2021-02-10T11:29:54.601 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:54.601 7f9b767fc700 1 -- 172.21.2.102:0/3277963903 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f9b7c0009b0 con 0x7f9b80199690 2021-02-10T11:29:54.729 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:54.728 7f9b880b4700 1 -- 172.21.2.102:0/3277963903 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9b801adc80 con 0x7f9b80199690 2021-02-10T11:29:54.729 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:54.729 7f9b767fc700 1 -- 172.21.2.102:0/3277963903 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+1756 (secure 0 0 0) 0x7f9b7c006020 con 0x7f9b80199690 2021-02-10T11:29:54.730 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:29:54.730 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":67,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6110,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.102:6835/1471871278","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6834","nonce":1471871278},{"type":"v1","addr":"172.21.2.102:6835","nonce":1471871278}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":61}],"filesystems":[{"mdsmap":{"epoch":67,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:27:41.479375","modified":"2021-02-10 11:29:54.077352","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":86,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":8}]} 2021-02-10T11:29:54.730 INFO:teuthology.orchestra.run.gibba002.stderr:dumped fsmap epoch 67 2021-02-10T11:29:54.731 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:54.731 7f9b880b4700 1 -- 172.21.2.102:0/3277963903 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f9b68018700 msgr2=0x7f9b6801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:54.731 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:54.731 7f9b880b4700 1 --2- 172.21.2.102:0/3277963903 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f9b68018700 0x7f9b6801abc0 crc :-1 s=READY pgs=851 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:54.732 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:54.731 7f9b880b4700 1 -- 172.21.2.102:0/3277963903 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9b80199690 msgr2=0x7f9b800af0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:54.732 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:54.731 7f9b880b4700 1 --2- 172.21.2.102:0/3277963903 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9b80199690 0x7f9b800af0f0 secure :-1 s=READY pgs=1899 cs=0 l=1 rev1=1 rx=0x7f9b7c00b5b0 tx=0x7f9b7c00b950).stop 2021-02-10T11:29:54.732 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:54.731 7f9b880b4700 1 -- 172.21.2.102:0/3277963903 shutdown_connections 2021-02-10T11:29:54.732 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:54.731 7f9b880b4700 1 --2- 172.21.2.102:0/3277963903 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9b800afb10 0x7f9b801ad060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:54.732 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:54.731 7f9b880b4700 1 --2- 172.21.2.102:0/3277963903 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f9b68018700 0x7f9b6801abc0 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:54.732 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:54.731 7f9b880b4700 1 --2- 172.21.2.102:0/3277963903 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f9b800af710 0x7f9b801aca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:54.732 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:54.731 7f9b880b4700 1 --2- 172.21.2.102:0/3277963903 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9b80199690 0x7f9b800af0f0 unknown :-1 s=CLOSED pgs=1899 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:54.732 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:54.731 7f9b880b4700 1 -- 172.21.2.102:0/3277963903 >> 172.21.2.102:0/3277963903 conn(0x7f9b801a63c0 msgr2=0x7f9b80094040 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:29:54.732 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:54.731 7f9b880b4700 1 -- 172.21.2.102:0/3277963903 shutdown_connections 2021-02-10T11:29:54.732 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:54.731 7f9b880b4700 1 -- 172.21.2.102:0/3277963903 wait complete. 2021-02-10T11:29:54.740 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2021-02-10T11:29:55.173 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:55.156 7efcc4900700 1 Processor -- start 2021-02-10T11:29:55.174 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:55.156 7efcc4900700 1 -- start start 2021-02-10T11:29:55.174 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:55.156 7efcc4900700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7efcc00d0340 0x7efcc00ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:55.174 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:55.156 7efcc4900700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7efcc00d1c70 0x7efcc00efc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:55.174 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:55.156 7efcc4900700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcc00f0b10 con 0x7efcc00d1870 2021-02-10T11:29:55.174 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:55.156 7efcc4900700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efcc00e5ed0 con 0x7efcc00d1c70 2021-02-10T11:29:55.174 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:55.156 7efcc4900700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efcc01b0be0 con 0x7efcc00d0340 2021-02-10T11:29:55.174 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:55.157 7efcbee1d700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7efcc00d1c70 0x7efcc00efc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:55.174 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:55.157 7efcbee1d700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7efcc00d1c70 0x7efcc00efc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:60508/0 (socket says 172.21.2.102:60508) 2021-02-10T11:29:55.174 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:55.157 7efcbee1d700 1 -- 172.21.2.102:0/1703195951 learned_addr learned my addr 172.21.2.102:0/1703195951 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:29:55.175 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:55.157 7efcbee1d700 1 -- 172.21.2.102:0/1703195951 >> v2:172.21.2.113:3300/0 conn(0x7efcc00d0340 msgr2=0x7efcc00ef640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:29:55.175 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:55.157 7efcbee1d700 1 --2- 172.21.2.102:0/1703195951 >> v2:172.21.2.113:3300/0 conn(0x7efcc00d0340 0x7efcc00ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:55.175 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:55.157 7efcbee1d700 1 -- 172.21.2.102:0/1703195951 >> v1:172.21.2.113:6789/0 conn(0x7efcc00d1870 legacy=0x7efcc01b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:29:55.175 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:55.157 7efcbee1d700 1 -- 172.21.2.102:0/1703195951 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcc00f20a0 con 0x7efcc00d1c70 2021-02-10T11:29:55.175 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:55.157 7efcbee1d700 1 --2- 172.21.2.102:0/1703195951 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efcc00d1c70 0x7efcc00efc60 secure :-1 s=READY pgs=1900 cs=0 l=1 rev1=1 rx=0x7efcb8006f00 tx=0x7efcb8009e90).ready entity=mon.0 client_cookie=ea74ec8b1adb43cf server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:29:55.175 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:55.158 7efcbd61a700 1 -- 172.21.2.102:0/1703195951 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efcb800b300 con 0x7efcc00d1c70 2021-02-10T11:29:55.175 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:55.158 7efcbd61a700 1 -- 172.21.2.102:0/1703195951 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcb800b640 con 0x7efcc00d1c70 2021-02-10T11:29:55.175 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:55.158 7efcbd61a700 1 -- 172.21.2.102:0/1703195951 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efcb800baa0 con 0x7efcc00d1c70 2021-02-10T11:29:55.175 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:55.158 7efcc4900700 1 -- 172.21.2.102:0/1703195951 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efcc00d1c70 msgr2=0x7efcc00efc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:55.175 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:55.158 7efcc4900700 1 --2- 172.21.2.102:0/1703195951 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efcc00d1c70 0x7efcc00efc60 secure :-1 s=READY pgs=1900 cs=0 l=1 rev1=1 rx=0x7efcb8006f00 tx=0x7efcb8009e90).stop 2021-02-10T11:29:55.175 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:55.158 7efcc4900700 1 -- 172.21.2.102:0/1703195951 shutdown_connections 2021-02-10T11:29:55.176 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:55.158 7efcc4900700 1 --2- 172.21.2.102:0/1703195951 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efcc00d1c70 0x7efcc00efc60 unknown :-1 s=CLOSED pgs=1900 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:55.176 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:55.158 7efcc4900700 1 --2- 172.21.2.102:0/1703195951 >> v2:172.21.2.113:3300/0 conn(0x7efcc00d0340 0x7efcc00ef640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:55.176 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:55.158 7efcc4900700 1 -- 172.21.2.102:0/1703195951 >> 172.21.2.102:0/1703195951 conn(0x7efcc00ce450 msgr2=0x7efcc00d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:29:55.176 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:55.159 7efcc4900700 1 -- 172.21.2.102:0/1703195951 shutdown_connections 2021-02-10T11:29:55.176 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:55.159 7efcc4900700 1 -- 172.21.2.102:0/1703195951 wait complete. 2021-02-10T11:29:55.176 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:55.173 7efcc4900700 1 Processor -- start 2021-02-10T11:29:55.176 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:55.173 7efcc4900700 1 -- start start 2021-02-10T11:29:55.177 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:55.173 7efcc4900700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efcc009bf50 0x7efcc009c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:55.178 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:55.173 7efcc4900700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7efcc009c350 0x7efcc00ac5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:55.178 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:55.173 7efcc4900700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7efcc009cfc0 0x7efcc00acbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:55.178 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:55.173 7efcc4900700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcc00ad210 con 0x7efcc009bf50 2021-02-10T11:29:55.178 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:55.173 7efcc4900700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcc00ad400 con 0x7efcc009cfc0 2021-02-10T11:29:55.178 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:55.173 7efcc4900700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcc00ad620 con 0x7efcc009c350 2021-02-10T11:29:55.178 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:55.173 7efcbe61c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efcc009bf50 0x7efcc009c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:55.178 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:55.174 7efcbe61c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efcc009bf50 0x7efcc009c9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:60512/0 (socket says 172.21.2.102:60512) 2021-02-10T11:29:55.178 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:55.174 7efcbe61c700 1 -- 172.21.2.102:0/1729627110 learned_addr learned my addr 172.21.2.102:0/1729627110 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:29:55.178 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:55.174 7efcbe61c700 1 -- 172.21.2.102:0/1729627110 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7efcc009c350 msgr2=0x7efcc00ac5d0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:29:55.179 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:55.174 7efcbe61c700 1 --2- 172.21.2.102:0/1729627110 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7efcc009c350 0x7efcc00ac5d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:55.179 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:55.174 7efcbe61c700 1 -- 172.21.2.102:0/1729627110 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7efcc009cfc0 msgr2=0x7efcc00acbf0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:29:55.179 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:55.174 7efcbe61c700 1 --2- 172.21.2.102:0/1729627110 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7efcc009cfc0 0x7efcc00acbf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:55.179 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:55.174 7efcbe61c700 1 -- 172.21.2.102:0/1729627110 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcc00ada80 con 0x7efcc009bf50 2021-02-10T11:29:55.179 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:55.174 7efcbe61c700 1 --2- 172.21.2.102:0/1729627110 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efcc009bf50 0x7efcc009c9a0 secure :-1 s=READY pgs=1901 cs=0 l=1 rev1=1 rx=0x7efcb000b530 tx=0x7efcb000b8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:29:55.179 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:55.174 7efcaeffd700 1 -- 172.21.2.102:0/1729627110 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efcb000bcc0 con 0x7efcc009bf50 2021-02-10T11:29:55.179 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:55.174 7efcaeffd700 1 -- 172.21.2.102:0/1729627110 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcb0006020 con 0x7efcc009bf50 2021-02-10T11:29:55.179 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:55.174 7efcaeffd700 1 -- 172.21.2.102:0/1729627110 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efcb00054a0 con 0x7efcc009bf50 2021-02-10T11:29:55.179 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:55.174 7efcc4900700 1 -- 172.21.2.102:0/1729627110 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efcc00ada80 con 0x7efcc009bf50 2021-02-10T11:29:55.179 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:55.174 7efcc4900700 1 -- 172.21.2.102:0/1729627110 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efcc00ae010 con 0x7efcc009bf50 2021-02-10T11:29:55.179 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:55.175 7efcaeffd700 1 -- 172.21.2.102:0/1729627110 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7efcb0006200 con 0x7efcc009bf50 2021-02-10T11:29:55.179 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:55.175 7efcaeffd700 1 --2- 172.21.2.102:0/1729627110 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7efca40186c0 0x7efca401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:55.180 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:55.175 7efcaeffd700 1 -- 172.21.2.102:0/1729627110 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 5659+0+0 (secure 0 0 0) 0x7efcb00325c0 con 0x7efcc009bf50 2021-02-10T11:29:55.180 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:55.175 7efcbde1b700 1 --2- 172.21.2.102:0/1729627110 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7efca40186c0 0x7efca401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:55.180 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:55.175 7efcbde1b700 1 --2- 172.21.2.102:0/1729627110 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7efca40186c0 0x7efca401abc0 crc :-1 s=READY pgs=852 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:29:55.207 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:55.206 7efcc4900700 1 -- 172.21.2.102:0/1729627110 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efcc00ce450 con 0x7efcc009bf50 2021-02-10T11:29:55.209 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:55.209 7efcaeffd700 1 -- 172.21.2.102:0/1729627110 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7efcb0000c10 con 0x7efcc009bf50 2021-02-10T11:29:55.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:55.337 7efcc4900700 1 -- 172.21.2.102:0/1729627110 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7efcc00ce450 con 0x7efcc009bf50 2021-02-10T11:29:56.086 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:56.086 7efcaeffd700 1 -- 172.21.2.102:0/1729627110 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v68) v1 ==== 104+0+0 (secure 0 0 0) 0x7efcb0006200 con 0x7efcc009bf50 2021-02-10T11:29:56.107 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:56.106 7efcc4900700 1 -- 172.21.2.102:0/1729627110 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7efca40186c0 msgr2=0x7efca401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:56.107 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:56.106 7efcc4900700 1 --2- 172.21.2.102:0/1729627110 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7efca40186c0 0x7efca401abc0 crc :-1 s=READY pgs=852 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:56.107 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:56.106 7efcc4900700 1 -- 172.21.2.102:0/1729627110 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efcc009bf50 msgr2=0x7efcc009c9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:56.107 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:56.106 7efcc4900700 1 --2- 172.21.2.102:0/1729627110 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efcc009bf50 0x7efcc009c9a0 secure :-1 s=READY pgs=1901 cs=0 l=1 rev1=1 rx=0x7efcb000b530 tx=0x7efcb000b8d0).stop 2021-02-10T11:29:56.107 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:56.106 7efcc4900700 1 -- 172.21.2.102:0/1729627110 shutdown_connections 2021-02-10T11:29:56.107 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:56.106 7efcc4900700 1 --2- 172.21.2.102:0/1729627110 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7efca40186c0 0x7efca401abc0 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:56.107 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:56.106 7efcc4900700 1 --2- 172.21.2.102:0/1729627110 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7efcc009cfc0 0x7efcc00acbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:56.107 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:56.106 7efcc4900700 1 --2- 172.21.2.102:0/1729627110 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7efcc009c350 0x7efcc00ac5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:56.107 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:56.106 7efcc4900700 1 --2- 172.21.2.102:0/1729627110 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efcc009bf50 0x7efcc009c9a0 unknown :-1 s=CLOSED pgs=1901 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:56.107 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:56.106 7efcc4900700 1 -- 172.21.2.102:0/1729627110 >> 172.21.2.102:0/1729627110 conn(0x7efcc00a6620 msgr2=0x7efcc00b17a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:29:56.108 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:56.107 7efcc4900700 1 -- 172.21.2.102:0/1729627110 shutdown_connections 2021-02-10T11:29:56.108 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:56.107 7efcc4900700 1 -- 172.21.2.102:0/1729627110 wait complete. 2021-02-10T11:29:56.115 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool delete cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2021-02-10T11:29:56.547 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:56.529 7fd5acb6d700 1 Processor -- start 2021-02-10T11:29:56.547 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:56.529 7fd5acb6d700 1 -- start start 2021-02-10T11:29:56.547 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:56.530 7fd5acb6d700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fd5a80d1870 0x7fd5a81b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:56.547 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:56.530 7fd5acb6d700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fd5a81b0ae0 0x7fd5a8108940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:56.547 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:56.530 7fd5acb6d700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5a80fe1c0 con 0x7fd5a80d1c70 2021-02-10T11:29:56.547 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:56.530 7fd5acb6d700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5a8109a60 con 0x7fd5a80d1870 2021-02-10T11:29:56.547 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:56.530 7fd5acb6d700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5a8109750 con 0x7fd5a81b0ae0 2021-02-10T11:29:56.547 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:56.530 7fd5a685e700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fd5a80d1870 0x7fd5a81b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:56.547 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:56.530 7fd5a685e700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fd5a80d1870 0x7fd5a81b04c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:60520/0 (socket says 172.21.2.102:60520) 2021-02-10T11:29:56.548 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:56.530 7fd5a685e700 1 -- 172.21.2.102:0/771045206 learned_addr learned my addr 172.21.2.102:0/771045206 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:29:56.548 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:56.530 7fd5a605d700 1 --2- 172.21.2.102:0/771045206 >> v2:172.21.2.113:3300/0 conn(0x7fd5a81b0ae0 0x7fd5a8108940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:56.548 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:56.531 7fd5a685e700 1 -- 172.21.2.102:0/771045206 >> v2:172.21.2.113:3300/0 conn(0x7fd5a81b0ae0 msgr2=0x7fd5a8108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:29:56.548 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:56.531 7fd5a685e700 1 --2- 172.21.2.102:0/771045206 >> v2:172.21.2.113:3300/0 conn(0x7fd5a81b0ae0 0x7fd5a8108940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:56.548 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:56.531 7fd5a685e700 1 -- 172.21.2.102:0/771045206 >> v1:172.21.2.102:6789/0 conn(0x7fd5a80d1c70 legacy=0x7fd5a8108f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:56.548 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:56.531 7fd5a585c700 1 -- 172.21.2.102:0/771045206 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3195987056 0 0) 0x7fd5a0000f90 con 0x7fd5a80d1c70 2021-02-10T11:29:56.548 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:56.531 7fd5a685e700 1 -- 172.21.2.102:0/771045206 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5a8088070 con 0x7fd5a80d1870 2021-02-10T11:29:56.548 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:56.531 7fd5a605d700 1 --2- 172.21.2.102:0/771045206 >> v2:172.21.2.113:3300/0 conn(0x7fd5a81b0ae0 0x7fd5a8108940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:29:56.548 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:56.531 7fd5a685e700 1 --2- 172.21.2.102:0/771045206 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd5a80d1870 0x7fd5a81b04c0 secure :-1 s=READY pgs=1903 cs=0 l=1 rev1=1 rx=0x7fd590006700 tx=0x7fd590009b80).ready entity=mon.0 client_cookie=24c3329b8352dc75 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:29:56.548 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:56.531 7fd5a585c700 1 -- 172.21.2.102:0/771045206 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd59000c0c0 con 0x7fd5a80d1870 2021-02-10T11:29:56.548 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:56.531 7fd5a585c700 1 -- 172.21.2.102:0/771045206 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd59000c340 con 0x7fd5a80d1870 2021-02-10T11:29:56.548 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:56.531 7fd5a585c700 1 -- 172.21.2.102:0/771045206 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd59000c860 con 0x7fd5a80d1870 2021-02-10T11:29:56.548 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:56.531 7fd5acb6d700 1 -- 172.21.2.102:0/771045206 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd5a80d1870 msgr2=0x7fd5a81b04c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:56.549 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:56.531 7fd5acb6d700 1 --2- 172.21.2.102:0/771045206 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd5a80d1870 0x7fd5a81b04c0 secure :-1 s=READY pgs=1903 cs=0 l=1 rev1=1 rx=0x7fd590006700 tx=0x7fd590009b80).stop 2021-02-10T11:29:56.549 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:56.532 7fd5acb6d700 1 -- 172.21.2.102:0/771045206 shutdown_connections 2021-02-10T11:29:56.549 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:56.532 7fd5acb6d700 1 --2- 172.21.2.102:0/771045206 >> v2:172.21.2.113:3300/0 conn(0x7fd5a81b0ae0 0x7fd5a8108940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:56.549 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:56.532 7fd5acb6d700 1 --2- 172.21.2.102:0/771045206 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd5a80d1870 0x7fd5a81b04c0 unknown :-1 s=CLOSED pgs=1903 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:56.549 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:56.532 7fd5acb6d700 1 -- 172.21.2.102:0/771045206 >> 172.21.2.102:0/771045206 conn(0x7fd5a80ce420 msgr2=0x7fd5a80d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:29:56.549 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:56.532 7fd5acb6d700 1 -- 172.21.2.102:0/771045206 shutdown_connections 2021-02-10T11:29:56.549 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:56.532 7fd5acb6d700 1 -- 172.21.2.102:0/771045206 wait complete. 2021-02-10T11:29:56.550 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:56.547 7fd5acb6d700 1 Processor -- start 2021-02-10T11:29:56.550 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:56.547 7fd5acb6d700 1 -- start start 2021-02-10T11:29:56.551 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:56.547 7fd5acb6d700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd5a819e5c0 0x7fd5a8099fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:56.551 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:56.547 7fd5acb6d700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd5a809a5d0 0x7fd5a809a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:56.551 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:56.547 7fd5acb6d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd5a80982a0 0x7fd5a80986a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:56.551 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:56.547 7fd5acb6d700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5a819feb0 con 0x7fd5a80982a0 2021-02-10T11:29:56.551 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:56.547 7fd5acb6d700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5a8099a30 con 0x7fd5a809a5d0 2021-02-10T11:29:56.551 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:56.547 7fd5acb6d700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5a819fb30 con 0x7fd5a819e5c0 2021-02-10T11:29:56.551 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:56.547 7fd5a705f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd5a80982a0 0x7fd5a80986a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:56.551 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:56.547 7fd5a705f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd5a80982a0 0x7fd5a80986a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:60524/0 (socket says 172.21.2.102:60524) 2021-02-10T11:29:56.551 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:56.547 7fd5a705f700 1 -- 172.21.2.102:0/2132117445 learned_addr learned my addr 172.21.2.102:0/2132117445 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:29:56.551 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:56.548 7fd5a705f700 1 -- 172.21.2.102:0/2132117445 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd5a819e5c0 msgr2=0x7fd5a8099fb0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:29:56.551 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:56.548 7fd5a705f700 1 --2- 172.21.2.102:0/2132117445 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd5a819e5c0 0x7fd5a8099fb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:56.551 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:56.548 7fd5a705f700 1 -- 172.21.2.102:0/2132117445 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd5a809a5d0 msgr2=0x7fd5a809a9d0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:29:56.551 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:56.548 7fd5a705f700 1 --2- 172.21.2.102:0/2132117445 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd5a809a5d0 0x7fd5a809a9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:56.552 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:56.548 7fd5a705f700 1 -- 172.21.2.102:0/2132117445 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5a819e9c0 con 0x7fd5a80982a0 2021-02-10T11:29:56.552 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:56.548 7fd5a705f700 1 --2- 172.21.2.102:0/2132117445 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd5a80982a0 0x7fd5a80986a0 secure :-1 s=READY pgs=1904 cs=0 l=1 rev1=1 rx=0x7fd5a0009980 tx=0x7fd5a000bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:29:56.552 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:56.548 7fd59f7fe700 1 -- 172.21.2.102:0/2132117445 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd5a0005af0 con 0x7fd5a80982a0 2021-02-10T11:29:56.552 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:56.548 7fd59f7fe700 1 -- 172.21.2.102:0/2132117445 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5a0005d70 con 0x7fd5a80982a0 2021-02-10T11:29:56.552 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:56.548 7fd59f7fe700 1 -- 172.21.2.102:0/2132117445 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd5a00062b0 con 0x7fd5a80982a0 2021-02-10T11:29:56.552 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:56.548 7fd5acb6d700 1 -- 172.21.2.102:0/2132117445 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5a819e9c0 con 0x7fd5a80982a0 2021-02-10T11:29:56.552 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:56.548 7fd5acb6d700 1 -- 172.21.2.102:0/2132117445 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5a819e9c0 con 0x7fd5a80982a0 2021-02-10T11:29:56.552 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:56.549 7fd59f7fe700 1 -- 172.21.2.102:0/2132117445 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd5a0006030 con 0x7fd5a80982a0 2021-02-10T11:29:56.552 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:56.549 7fd59f7fe700 1 --2- 172.21.2.102:0/2132117445 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd58c0186c0 0x7fd58c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:56.552 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:56.549 7fd59f7fe700 1 -- 172.21.2.102:0/2132117445 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 5659+0+0 (secure 0 0 0) 0x7fd5a00329c0 con 0x7fd5a80982a0 2021-02-10T11:29:56.552 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:56.549 7fd5a685e700 1 --2- 172.21.2.102:0/2132117445 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd58c0186c0 0x7fd58c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:56.552 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:56.549 7fd5a685e700 1 --2- 172.21.2.102:0/2132117445 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd58c0186c0 0x7fd58c01abc0 crc :-1 s=READY pgs=853 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:29:56.580 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:56.580 7fd5acb6d700 1 -- 172.21.2.102:0/2132117445 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5a80825d0 con 0x7fd5a80982a0 2021-02-10T11:29:56.583 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:56.583 7fd59f7fe700 1 -- 172.21.2.102:0/2132117445 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd5a000be10 con 0x7fd5a80982a0 2021-02-10T11:29:56.711 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:56.710 7fd5acb6d700 1 -- 172.21.2.102:0/2132117445 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"pool2": "cephfs_metadata", "prefix": "osd pool delete", "pool": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fd5a80825d0 con 0x7fd5a80982a0 2021-02-10T11:29:57.091 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:57.090 7fd59f7fe700 1 -- 172.21.2.102:0/2132117445 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"pool2": "cephfs_metadata", "prefix": "osd pool delete", "pool": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v87) v1 ==== 185+0+0 (secure 0 0 0) 0x7fd5a0006030 con 0x7fd5a80982a0 2021-02-10T11:29:57.127 INFO:teuthology.orchestra.run.gibba002.stderr:pool 'cephfs_metadata' removed 2021-02-10T11:29:57.129 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:57.129 7fd5acb6d700 1 -- 172.21.2.102:0/2132117445 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd58c0186c0 msgr2=0x7fd58c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:57.129 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:57.129 7fd5acb6d700 1 --2- 172.21.2.102:0/2132117445 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd58c0186c0 0x7fd58c01abc0 crc :-1 s=READY pgs=853 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:57.130 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:57.129 7fd5acb6d700 1 -- 172.21.2.102:0/2132117445 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd5a80982a0 msgr2=0x7fd5a80986a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:57.130 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:57.129 7fd5acb6d700 1 --2- 172.21.2.102:0/2132117445 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd5a80982a0 0x7fd5a80986a0 secure :-1 s=READY pgs=1904 cs=0 l=1 rev1=1 rx=0x7fd5a0009980 tx=0x7fd5a000bcd0).stop 2021-02-10T11:29:57.130 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:57.129 7fd5acb6d700 1 -- 172.21.2.102:0/2132117445 shutdown_connections 2021-02-10T11:29:57.130 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:57.129 7fd5acb6d700 1 --2- 172.21.2.102:0/2132117445 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd5a80982a0 0x7fd5a80986a0 unknown :-1 s=CLOSED pgs=1904 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:57.130 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:57.129 7fd5acb6d700 1 --2- 172.21.2.102:0/2132117445 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd58c0186c0 0x7fd58c01abc0 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:57.130 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:57.129 7fd5acb6d700 1 --2- 172.21.2.102:0/2132117445 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd5a809a5d0 0x7fd5a809a9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:57.130 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:57.129 7fd5acb6d700 1 --2- 172.21.2.102:0/2132117445 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd5a819e5c0 0x7fd5a8099fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:57.130 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:57.129 7fd5acb6d700 1 -- 172.21.2.102:0/2132117445 >> 172.21.2.102:0/2132117445 conn(0x7fd5a81b3560 msgr2=0x7fd5a81b90e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:29:57.130 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:57.129 7fd5acb6d700 1 -- 172.21.2.102:0/2132117445 shutdown_connections 2021-02-10T11:29:57.130 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:57.129 7fd5acb6d700 1 -- 172.21.2.102:0/2132117445 wait complete. 2021-02-10T11:29:57.138 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool delete cephfs_data cephfs_data --yes-i-really-really-mean-it 2021-02-10T11:29:57.571 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:57.553 7faf9a0c8700 1 Processor -- start 2021-02-10T11:29:57.571 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:57.553 7faf9a0c8700 1 -- start start 2021-02-10T11:29:57.571 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:57.554 7faf9a0c8700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7faf940d1880 0x7faf941b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:57.571 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:57.554 7faf9a0c8700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7faf940cff80 0x7faf94109140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:57.571 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:57.554 7faf9a0c8700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf94109a70 con 0x7faf941b0af0 2021-02-10T11:29:57.571 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:57.554 7faf9a0c8700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf94109760 con 0x7faf940cff80 2021-02-10T11:29:57.572 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:57.554 7faf9a0c8700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf94088070 con 0x7faf940d1880 2021-02-10T11:29:57.572 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:57.554 7faf92ffd700 1 --1- >> v1:172.21.2.113:6789/0 conn(0x7faf941b0af0 0x7faf94108950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.113:6789/0 says I am v1:172.21.2.102:60080/0 (socket says 172.21.2.102:60080) 2021-02-10T11:29:57.572 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:57.554 7faf92ffd700 1 -- 172.21.2.102:0/691620462 learned_addr learned my addr 172.21.2.102:0/691620462 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:29:57.572 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:57.555 7faf937fe700 1 --2- 172.21.2.102:0/691620462 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7faf940d1880 0x7faf941b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:57.572 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:57.555 7faf93fff700 1 --2- 172.21.2.102:0/691620462 >> v2:172.21.2.113:3300/0 conn(0x7faf940cff80 0x7faf94109140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:57.572 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:57.555 7faf927fc700 1 -- 172.21.2.102:0/691620462 <== mon.1 v1:172.21.2.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3417849827 0 0) 0x7faf880010b0 con 0x7faf941b0af0 2021-02-10T11:29:57.572 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:57.555 7faf927fc700 1 -- 172.21.2.102:0/691620462 --> v1:172.21.2.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf7c002fe0 con 0x7faf941b0af0 2021-02-10T11:29:57.572 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:57.555 7faf937fe700 1 -- 172.21.2.102:0/691620462 >> v2:172.21.2.113:3300/0 conn(0x7faf940cff80 msgr2=0x7faf94109140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:29:57.572 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:57.555 7faf937fe700 1 --2- 172.21.2.102:0/691620462 >> v2:172.21.2.113:3300/0 conn(0x7faf940cff80 0x7faf94109140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:57.572 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:57.555 7faf937fe700 1 -- 172.21.2.102:0/691620462 >> v1:172.21.2.113:6789/0 conn(0x7faf941b0af0 legacy=0x7faf94108950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:57.572 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:57.555 7faf937fe700 1 -- 172.21.2.102:0/691620462 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf94088260 con 0x7faf940d1880 2021-02-10T11:29:57.572 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:57.555 7faf937fe700 1 --2- 172.21.2.102:0/691620462 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7faf940d1880 0x7faf941b04d0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7faf84006650 tx=0x7faf84009a80).ready entity=mon.2 client_cookie=b116a3404617f4fc server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:29:57.572 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:57.556 7faf927fc700 1 -- 172.21.2.102:0/691620462 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faf8400c0c0 con 0x7faf940d1880 2021-02-10T11:29:57.573 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:57.556 7faf927fc700 1 -- 172.21.2.102:0/691620462 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf8400c340 con 0x7faf940d1880 2021-02-10T11:29:57.573 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:57.556 7faf927fc700 1 -- 172.21.2.102:0/691620462 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faf8400c860 con 0x7faf940d1880 2021-02-10T11:29:57.573 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:57.556 7faf9a0c8700 1 -- 172.21.2.102:0/691620462 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7faf940d1880 msgr2=0x7faf941b04d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:57.573 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:57.556 7faf9a0c8700 1 --2- 172.21.2.102:0/691620462 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7faf940d1880 0x7faf941b04d0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7faf84006650 tx=0x7faf84009a80).stop 2021-02-10T11:29:57.573 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:57.556 7faf9a0c8700 1 -- 172.21.2.102:0/691620462 shutdown_connections 2021-02-10T11:29:57.573 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:57.556 7faf9a0c8700 1 --2- 172.21.2.102:0/691620462 >> v2:172.21.2.113:3300/0 conn(0x7faf940cff80 0x7faf94109140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:57.573 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:57.556 7faf9a0c8700 1 --2- 172.21.2.102:0/691620462 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7faf940d1880 0x7faf941b04d0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:57.573 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:57.556 7faf9a0c8700 1 -- 172.21.2.102:0/691620462 >> 172.21.2.102:0/691620462 conn(0x7faf940ce430 msgr2=0x7faf940d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:29:57.573 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:57.557 7faf9a0c8700 1 -- 172.21.2.102:0/691620462 shutdown_connections 2021-02-10T11:29:57.573 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:57.557 7faf9a0c8700 1 -- 172.21.2.102:0/691620462 wait complete. 2021-02-10T11:29:57.575 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:57.571 7faf9a0c8700 1 Processor -- start 2021-02-10T11:29:57.575 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:57.571 7faf9a0c8700 1 -- start start 2021-02-10T11:29:57.575 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:57.571 7faf9a0c8700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7faf941a0580 0x7faf94082c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:57.575 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:57.571 7faf9a0c8700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7faf940832a0 0x7faf940836a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:57.575 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:57.571 7faf9a0c8700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7faf940b66d0 0x7faf940b6ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:57.575 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:57.571 7faf9a0c8700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf940ba540 con 0x7faf940b66d0 2021-02-10T11:29:57.575 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:57.571 7faf9a0c8700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf940baac0 con 0x7faf941a0580 2021-02-10T11:29:57.575 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:57.571 7faf9a0c8700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf940b70f0 con 0x7faf940832a0 2021-02-10T11:29:57.576 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:57.572 7faf93fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7faf940b66d0 0x7faf940b6ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:57.576 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:57.572 7faf93fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7faf940b66d0 0x7faf940b6ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:60540/0 (socket says 172.21.2.102:60540) 2021-02-10T11:29:57.576 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:57.572 7faf93fff700 1 -- 172.21.2.102:0/2680405610 learned_addr learned my addr 172.21.2.102:0/2680405610 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:29:57.576 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:57.572 7faf93fff700 1 -- 172.21.2.102:0/2680405610 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7faf940832a0 msgr2=0x7faf940836a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:57.576 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:57.572 7faf93fff700 1 --2- 172.21.2.102:0/2680405610 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7faf940832a0 0x7faf940836a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:57.576 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:57.572 7faf93fff700 1 -- 172.21.2.102:0/2680405610 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7faf941a0580 msgr2=0x7faf94082c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:57.576 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:57.572 7faf93fff700 1 --2- 172.21.2.102:0/2680405610 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7faf941a0580 0x7faf94082c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:57.576 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:57.572 7faf93fff700 1 -- 172.21.2.102:0/2680405610 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf940b7310 con 0x7faf940b66d0 2021-02-10T11:29:57.576 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:57.572 7faf93fff700 1 --2- 172.21.2.102:0/2680405610 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7faf940b66d0 0x7faf940b6ad0 secure :-1 s=READY pgs=1905 cs=0 l=1 rev1=1 rx=0x7faf8c00b970 tx=0x7faf8c00bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:29:57.576 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:57.572 7faf7bfff700 1 -- 172.21.2.102:0/2680405610 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faf8c000f90 con 0x7faf940b66d0 2021-02-10T11:29:57.576 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:57.572 7faf7bfff700 1 -- 172.21.2.102:0/2680405610 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf8c003030 con 0x7faf940b66d0 2021-02-10T11:29:57.576 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:57.572 7faf9a0c8700 1 -- 172.21.2.102:0/2680405610 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf940b7310 con 0x7faf940b66d0 2021-02-10T11:29:57.576 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:57.572 7faf7bfff700 1 -- 172.21.2.102:0/2680405610 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faf8c001470 con 0x7faf940b66d0 2021-02-10T11:29:57.577 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:57.572 7faf9a0c8700 1 -- 172.21.2.102:0/2680405610 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf940b7310 con 0x7faf940b66d0 2021-02-10T11:29:57.577 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:57.573 7faf7bfff700 1 -- 172.21.2.102:0/2680405610 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7faf8c000e70 con 0x7faf940b66d0 2021-02-10T11:29:57.577 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:57.573 7faf7bfff700 1 --2- 172.21.2.102:0/2680405610 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7faf74018730 0x7faf7401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:57.577 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:57.573 7faf7bfff700 1 -- 172.21.2.102:0/2680405610 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 5221+0+0 (secure 0 0 0) 0x7faf8c032870 con 0x7faf940b66d0 2021-02-10T11:29:57.577 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:57.573 7faf937fe700 1 --2- 172.21.2.102:0/2680405610 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7faf74018730 0x7faf7401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:57.577 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:57.573 7faf937fe700 1 --2- 172.21.2.102:0/2680405610 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7faf74018730 0x7faf7401ac10 crc :-1 s=READY pgs=854 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:29:57.604 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:57.604 7faf9a0c8700 1 -- 172.21.2.102:0/2680405610 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf94082a50 con 0x7faf940b66d0 2021-02-10T11:29:57.606 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:57.606 7faf7bfff700 1 -- 172.21.2.102:0/2680405610 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7faf8c0066f0 con 0x7faf940b66d0 2021-02-10T11:29:57.733 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:57.732 7faf9a0c8700 1 -- 172.21.2.102:0/2680405610 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"pool2": "cephfs_data", "prefix": "osd pool delete", "pool": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7faf94082a50 con 0x7faf940b66d0 2021-02-10T11:29:58.098 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:58.097 7faf7bfff700 1 -- 172.21.2.102:0/2680405610 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"pool2": "cephfs_data", "prefix": "osd pool delete", "pool": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v88) v1 ==== 173+0+0 (secure 0 0 0) 0x7faf8c0066f0 con 0x7faf940b66d0 2021-02-10T11:29:58.100 INFO:teuthology.orchestra.run.gibba002.stderr:pool 'cephfs_data' removed 2021-02-10T11:29:58.101 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:58.101 7faf9a0c8700 1 -- 172.21.2.102:0/2680405610 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7faf74018730 msgr2=0x7faf7401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:58.102 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:58.101 7faf9a0c8700 1 --2- 172.21.2.102:0/2680405610 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7faf74018730 0x7faf7401ac10 crc :-1 s=READY pgs=854 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:58.102 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:58.101 7faf9a0c8700 1 -- 172.21.2.102:0/2680405610 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7faf940b66d0 msgr2=0x7faf940b6ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:58.102 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:58.101 7faf9a0c8700 1 --2- 172.21.2.102:0/2680405610 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7faf940b66d0 0x7faf940b6ad0 secure :-1 s=READY pgs=1905 cs=0 l=1 rev1=1 rx=0x7faf8c00b970 tx=0x7faf8c00bd10).stop 2021-02-10T11:29:58.102 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:58.101 7faf9a0c8700 1 -- 172.21.2.102:0/2680405610 shutdown_connections 2021-02-10T11:29:58.102 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:58.101 7faf9a0c8700 1 --2- 172.21.2.102:0/2680405610 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7faf940b66d0 0x7faf940b6ad0 unknown :-1 s=CLOSED pgs=1905 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:58.102 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:58.101 7faf9a0c8700 1 --2- 172.21.2.102:0/2680405610 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7faf940832a0 0x7faf940836a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:58.102 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:58.101 7faf9a0c8700 1 --2- 172.21.2.102:0/2680405610 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7faf74018730 0x7faf7401ac10 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:58.103 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:58.101 7faf9a0c8700 1 --2- 172.21.2.102:0/2680405610 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7faf941a0580 0x7faf94082c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:58.103 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:58.101 7faf9a0c8700 1 -- 172.21.2.102:0/2680405610 >> 172.21.2.102:0/2680405610 conn(0x7faf940c20f0 msgr2=0x7faf9409e410 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:29:58.103 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:58.101 7faf9a0c8700 1 -- 172.21.2.102:0/2680405610 shutdown_connections 2021-02-10T11:29:58.103 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:58.102 7faf9a0c8700 1 -- 172.21.2.102:0/2680405610 wait complete. 2021-02-10T11:29:58.110 INFO:tasks.ceph.mds.a:Restarting daemon 2021-02-10T11:29:58.111 INFO:tasks.ceph.mds.a:Stopping old one... 2021-02-10T11:29:58.111 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2021-02-10T11:29:58.111 INFO:teuthology.orchestra.run:waiting for 300 2021-02-10T11:29:58.111 INFO:tasks.ceph.mds.b:Restarting daemon 2021-02-10T11:29:58.111 INFO:tasks.ceph.mds.b:Stopping old one... 2021-02-10T11:29:58.112 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2021-02-10T11:29:58.112 INFO:teuthology.orchestra.run:waiting for 300 2021-02-10T11:29:58.117 INFO:tasks.ceph.mds.a:Stopped 2021-02-10T11:29:58.117 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2021-02-10T11:29:58.118 INFO:tasks.ceph.mds.b:Stopped 2021-02-10T11:29:58.118 DEBUG:teuthology.orchestra.run.gibba013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2021-02-10T11:29:58.160 INFO:tasks.ceph.mds.a:Started 2021-02-10T11:29:58.160 DEBUG:teuthology.parallel:result is None 2021-02-10T11:29:58.161 INFO:tasks.ceph.mds.b:Started 2021-02-10T11:29:58.161 DEBUG:teuthology.parallel:result is None 2021-02-10T11:29:58.161 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd blacklist clear 2021-02-10T11:29:58.567 INFO:tasks.ceph.mds.b.gibba013.stdout:starting mds.b at 2021-02-10T11:29:58.572 INFO:tasks.ceph.mds.a.gibba002.stdout:starting mds.a at 2021-02-10T11:29:58.595 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:58.577 7f466c79a700 1 Processor -- start 2021-02-10T11:29:58.595 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:58.577 7f466c79a700 1 -- start start 2021-02-10T11:29:58.595 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:58.578 7f466c79a700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f46640d1870 0x7f46641b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:58.595 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:58.578 7f466c79a700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f46640f0650 0x7f46640cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:58.595 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:58.578 7f466c79a700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46640f0b10 con 0x7f46641b0a10 2021-02-10T11:29:58.595 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:58.578 7f466c79a700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46640fe230 con 0x7f46640f0650 2021-02-10T11:29:58.595 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:58.578 7f466c79a700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46640e5ed0 con 0x7f46640d1870 2021-02-10T11:29:58.595 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:58.578 7f4669db5700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f46641b0a10 0x7f46640cf330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:52780/0 (socket says 172.21.2.102:52780) 2021-02-10T11:29:58.596 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:58.578 7f4669db5700 1 -- 172.21.2.102:0/1907862679 learned_addr learned my addr 172.21.2.102:0/1907862679 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:29:58.596 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:58.578 7f466adb7700 1 --2- 172.21.2.102:0/1907862679 >> v2:172.21.2.113:3300/0 conn(0x7f46640f0650 0x7f46640cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:58.596 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:58.578 7f466a5b6700 1 --2- 172.21.2.102:0/1907862679 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f46640d1870 0x7f46641b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:58.596 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:58.579 7f466adb7700 1 -- 172.21.2.102:0/1907862679 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f46640d1870 msgr2=0x7f46641b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:29:58.596 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:58.579 7f46695b4700 1 -- 172.21.2.102:0/1907862679 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1904376058 0 0) 0x7f46540010b0 con 0x7f46641b0a10 2021-02-10T11:29:58.596 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:58.579 7f466adb7700 1 --2- 172.21.2.102:0/1907862679 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f46640d1870 0x7f46641b03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:58.596 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:58.579 7f466adb7700 1 -- 172.21.2.102:0/1907862679 >> v1:172.21.2.102:6789/0 conn(0x7f46641b0a10 legacy=0x7f46640cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:58.596 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:58.579 7f466adb7700 1 -- 172.21.2.102:0/1907862679 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46640d2fe0 con 0x7f46640f0650 2021-02-10T11:29:58.596 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:58.579 7f466adb7700 1 --2- 172.21.2.102:0/1907862679 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f46640f0650 0x7f46640cfb20 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 rx=0x7f4660006ad0 tx=0x7f4660009a50).ready entity=mon.1 client_cookie=e684d6849c2f8e64 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:29:58.596 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:58.579 7f46695b4700 1 -- 172.21.2.102:0/1907862679 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f46600066a0 con 0x7f46640f0650 2021-02-10T11:29:58.596 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:58.579 7f46695b4700 1 -- 172.21.2.102:0/1907862679 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4660006410 con 0x7f46640f0650 2021-02-10T11:29:58.597 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:58.579 7f46695b4700 1 -- 172.21.2.102:0/1907862679 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4660002560 con 0x7f46640f0650 2021-02-10T11:29:58.597 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:58.579 7f466c79a700 1 -- 172.21.2.102:0/1907862679 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f46640f0650 msgr2=0x7f46640cfb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:58.597 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:58.579 7f466c79a700 1 --2- 172.21.2.102:0/1907862679 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f46640f0650 0x7f46640cfb20 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 rx=0x7f4660006ad0 tx=0x7f4660009a50).stop 2021-02-10T11:29:58.597 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:58.580 7f466c79a700 1 -- 172.21.2.102:0/1907862679 shutdown_connections 2021-02-10T11:29:58.597 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:58.580 7f466c79a700 1 --2- 172.21.2.102:0/1907862679 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f46640f0650 0x7f46640cfb20 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:58.597 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:58.580 7f466c79a700 1 --2- 172.21.2.102:0/1907862679 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f46640d1870 0x7f46641b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:58.597 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:58.580 7f466c79a700 1 -- 172.21.2.102:0/1907862679 >> 172.21.2.102:0/1907862679 conn(0x7f46640ce450 msgr2=0x7f46640d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:29:58.597 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:58.580 7f466c79a700 1 -- 172.21.2.102:0/1907862679 shutdown_connections 2021-02-10T11:29:58.597 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:58.580 7f466c79a700 1 -- 172.21.2.102:0/1907862679 wait complete. 2021-02-10T11:29:58.599 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:58.595 7f466c79a700 1 Processor -- start 2021-02-10T11:29:58.599 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:58.595 7f466c79a700 1 -- start start 2021-02-10T11:29:58.599 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:58.595 7f466c79a700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f46640b7510 0x7f46640b7910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:58.599 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:58.595 7f466c79a700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f466409d530 0x7f466409d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:58.599 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:58.595 7f466c79a700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f466409df50 0x7f466409eca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:58.599 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:58.595 7f466c79a700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46640b8140 con 0x7f466409df50 2021-02-10T11:29:58.599 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:58.595 7f466c79a700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f466409e750 con 0x7f46640b7510 2021-02-10T11:29:58.599 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:58.595 7f466c79a700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f466409e350 con 0x7f466409d530 2021-02-10T11:29:58.599 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:58.596 7f466adb7700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f466409df50 0x7f466409eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:58.600 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:58.596 7f466adb7700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f466409df50 0x7f466409eca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:60562/0 (socket says 172.21.2.102:60562) 2021-02-10T11:29:58.600 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:58.596 7f466adb7700 1 -- 172.21.2.102:0/1628788203 learned_addr learned my addr 172.21.2.102:0/1628788203 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:29:58.600 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:58.596 7f466a5b6700 1 --2- 172.21.2.102:0/1628788203 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f46640b7510 0x7f46640b7910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:58.600 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:58.596 7f4669db5700 1 --2- 172.21.2.102:0/1628788203 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f466409d530 0x7f466409d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:58.600 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:58.596 7f466adb7700 1 -- 172.21.2.102:0/1628788203 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f466409d530 msgr2=0x7f466409d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:58.600 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:58.596 7f466adb7700 1 --2- 172.21.2.102:0/1628788203 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f466409d530 0x7f466409d930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:58.600 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:58.596 7f466adb7700 1 -- 172.21.2.102:0/1628788203 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f46640b7510 msgr2=0x7f46640b7910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:58.600 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:58.596 7f466adb7700 1 --2- 172.21.2.102:0/1628788203 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f46640b7510 0x7f46640b7910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:58.600 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:58.596 7f466adb7700 1 -- 172.21.2.102:0/1628788203 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f466409f2c0 con 0x7f466409df50 2021-02-10T11:29:58.600 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:58.596 7f4669db5700 1 --2- 172.21.2.102:0/1628788203 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f466409d530 0x7f466409d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:29:58.600 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:58.596 7f466adb7700 1 --2- 172.21.2.102:0/1628788203 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f466409df50 0x7f466409eca0 secure :-1 s=READY pgs=1910 cs=0 l=1 rev1=1 rx=0x7f46600066d0 tx=0x7f466000b9a0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:29:58.600 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:58.596 7f465affd700 1 -- 172.21.2.102:0/1628788203 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4660000db0 con 0x7f466409df50 2021-02-10T11:29:58.601 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:58.596 7f465affd700 1 -- 172.21.2.102:0/1628788203 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4660002070 con 0x7f466409df50 2021-02-10T11:29:58.601 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:58.596 7f466c79a700 1 -- 172.21.2.102:0/1628788203 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f466409f2c0 con 0x7f466409df50 2021-02-10T11:29:58.601 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:58.596 7f466c79a700 1 -- 172.21.2.102:0/1628788203 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f466409f850 con 0x7f466409df50 2021-02-10T11:29:58.601 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:58.596 7f465affd700 1 -- 172.21.2.102:0/1628788203 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4660002590 con 0x7f466409df50 2021-02-10T11:29:58.601 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:58.597 7f465affd700 1 -- 172.21.2.102:0/1628788203 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4660000a70 con 0x7f466409df50 2021-02-10T11:29:58.601 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:58.597 7f465affd700 1 --2- 172.21.2.102:0/1628788203 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f464c018730 0x7f464c01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:58.601 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:58.597 7f465affd700 1 -- 172.21.2.102:0/1628788203 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 4839+0+0 (secure 0 0 0) 0x7f4660032500 con 0x7f466409df50 2021-02-10T11:29:58.601 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:58.597 7f466a5b6700 1 --2- 172.21.2.102:0/1628788203 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f464c018730 0x7f464c01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:58.601 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:58.597 7f466a5b6700 1 --2- 172.21.2.102:0/1628788203 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f464c018730 0x7f464c01ac10 crc :-1 s=READY pgs=855 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:29:58.628 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:58.628 7f466c79a700 1 -- 172.21.2.102:0/1628788203 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46640ebc50 con 0x7f466409df50 2021-02-10T11:29:58.631 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:58.630 7f465affd700 1 -- 172.21.2.102:0/1628788203 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f4660000a70 con 0x7f466409df50 2021-02-10T11:29:58.773 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:58.772 7f466c79a700 1 -- 172.21.2.102:0/1628788203 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd blacklist clear"} v 0) v1 -- 0x7f46640f0b10 con 0x7f466409df50 2021-02-10T11:29:59.100 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:59.100 7f465affd700 1 -- 172.21.2.102:0/1628788203 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd blacklist clear"}]=0 removed all blacklist entries v89) v1 ==== 97+0+0 (secure 0 0 0) 0x7f4660002850 con 0x7f466409df50 2021-02-10T11:29:59.138 INFO:teuthology.orchestra.run.gibba002.stderr: removed all blacklist entries 2021-02-10T11:29:59.140 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:59.140 7f466c79a700 1 -- 172.21.2.102:0/1628788203 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f464c018730 msgr2=0x7f464c01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:59.141 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:59.140 7f466c79a700 1 --2- 172.21.2.102:0/1628788203 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f464c018730 0x7f464c01ac10 crc :-1 s=READY pgs=855 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:59.141 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:59.140 7f466c79a700 1 -- 172.21.2.102:0/1628788203 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f466409df50 msgr2=0x7f466409eca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:59.141 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:59.140 7f466c79a700 1 --2- 172.21.2.102:0/1628788203 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f466409df50 0x7f466409eca0 secure :-1 s=READY pgs=1910 cs=0 l=1 rev1=1 rx=0x7f46600066d0 tx=0x7f466000b9a0).stop 2021-02-10T11:29:59.141 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:59.140 7f466c79a700 1 -- 172.21.2.102:0/1628788203 shutdown_connections 2021-02-10T11:29:59.141 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:59.140 7f466c79a700 1 --2- 172.21.2.102:0/1628788203 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f466409df50 0x7f466409eca0 unknown :-1 s=CLOSED pgs=1910 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:59.142 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:59.140 7f466c79a700 1 --2- 172.21.2.102:0/1628788203 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f466409d530 0x7f466409d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:59.142 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:59.140 7f466c79a700 1 --2- 172.21.2.102:0/1628788203 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f464c018730 0x7f464c01ac10 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:59.142 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:59.140 7f466c79a700 1 --2- 172.21.2.102:0/1628788203 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f46640b7510 0x7f46640b7910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:59.142 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:59.140 7f466c79a700 1 -- 172.21.2.102:0/1628788203 >> 172.21.2.102:0/1628788203 conn(0x7f46640b83b0 msgr2=0x7f46640a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:29:59.142 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:59.140 7f466c79a700 1 -- 172.21.2.102:0/1628788203 shutdown_connections 2021-02-10T11:29:59.142 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:59.140 7f466c79a700 1 -- 172.21.2.102:0/1628788203 wait complete. 2021-02-10T11:29:59.149 INFO:tasks.cephfs.cephfs_test_case:['0', '1', '2', '3'] 2021-02-10T11:29:59.149 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth ls --format=json-pretty 2021-02-10T11:29:59.578 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:59.560 7fd3ce942700 1 Processor -- start 2021-02-10T11:29:59.579 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:59.560 7fd3ce942700 1 -- start start 2021-02-10T11:29:59.579 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:59.560 7fd3ce942700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fd3c80d1870 0x7fd3c81b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:59.579 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:59.560 7fd3ce942700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd3c80f0650 0x7fd3c80cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:59.579 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:59.560 7fd3ce942700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3c80f0b10 con 0x7fd3c81b0a10 2021-02-10T11:29:59.579 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:59.560 7fd3ce942700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3c80e5ed0 con 0x7fd3c80d1870 2021-02-10T11:29:59.579 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:59.560 7fd3ce942700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3c80d1c70 con 0x7fd3c80f0650 2021-02-10T11:29:59.579 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:59.562 7fd3c7fff700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fd3c80d1870 0x7fd3c81b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:59.579 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:59.562 7fd3ccf5f700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd3c80f0650 0x7fd3c80cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:59.579 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:59.562 7fd3c77fe700 1 --1- >> v1:172.21.2.113:6789/0 conn(0x7fd3c81b0a10 0x7fd3c80cf330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.113:6789/0 says I am v1:172.21.2.102:60116/0 (socket says 172.21.2.102:60116) 2021-02-10T11:29:59.579 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:59.562 7fd3c77fe700 1 -- 172.21.2.102:0/80635636 learned_addr learned my addr 172.21.2.102:0/80635636 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:29:59.579 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:59.562 7fd3c7fff700 1 -- 172.21.2.102:0/80635636 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd3c80f0650 msgr2=0x7fd3c80cfb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:29:59.580 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:59.562 7fd3c7fff700 1 --2- 172.21.2.102:0/80635636 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd3c80f0650 0x7fd3c80cfb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:59.580 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:59.562 7fd3c7fff700 1 -- 172.21.2.102:0/80635636 >> v1:172.21.2.113:6789/0 conn(0x7fd3c81b0a10 legacy=0x7fd3c80cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:59.580 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:59.562 7fd3c7fff700 1 -- 172.21.2.102:0/80635636 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3c80d1e90 con 0x7fd3c80d1870 2021-02-10T11:29:59.580 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:59.562 7fd3ccf5f700 1 --2- 172.21.2.102:0/80635636 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd3c80f0650 0x7fd3c80cfb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:29:59.580 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:59.563 7fd3c7fff700 1 --2- 172.21.2.102:0/80635636 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd3c80d1870 0x7fd3c81b03f0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 rx=0x7fd3b8006650 tx=0x7fd3b8009a80).ready entity=mon.1 client_cookie=9c51ce340c2ca2eb server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:29:59.580 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:59.563 7fd3c6ffd700 1 -- 172.21.2.102:0/80635636 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd3b80009b0 con 0x7fd3c80d1870 2021-02-10T11:29:59.580 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:59.563 7fd3c6ffd700 1 -- 172.21.2.102:0/80635636 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3b8006740 con 0x7fd3c80d1870 2021-02-10T11:29:59.580 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:59.563 7fd3c6ffd700 1 -- 172.21.2.102:0/80635636 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd3b800c0c0 con 0x7fd3c80d1870 2021-02-10T11:29:59.580 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:59.563 7fd3ce942700 1 -- 172.21.2.102:0/80635636 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd3c80d1870 msgr2=0x7fd3c81b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:59.580 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:59.563 7fd3ce942700 1 --2- 172.21.2.102:0/80635636 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd3c80d1870 0x7fd3c81b03f0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 rx=0x7fd3b8006650 tx=0x7fd3b8009a80).stop 2021-02-10T11:29:59.580 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:59.564 7fd3ce942700 1 -- 172.21.2.102:0/80635636 shutdown_connections 2021-02-10T11:29:59.580 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:59.564 7fd3ce942700 1 --2- 172.21.2.102:0/80635636 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd3c80f0650 0x7fd3c80cfb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:59.580 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:59.564 7fd3ce942700 1 --2- 172.21.2.102:0/80635636 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd3c80d1870 0x7fd3c81b03f0 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:59.581 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:59.564 7fd3ce942700 1 -- 172.21.2.102:0/80635636 >> 172.21.2.102:0/80635636 conn(0x7fd3c80ce450 msgr2=0x7fd3c80d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:29:59.581 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:59.564 7fd3ce942700 1 -- 172.21.2.102:0/80635636 shutdown_connections 2021-02-10T11:29:59.581 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:59.564 7fd3ce942700 1 -- 172.21.2.102:0/80635636 wait complete. 2021-02-10T11:29:59.582 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:59.578 7fd3ce942700 1 Processor -- start 2021-02-10T11:29:59.583 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:59.579 7fd3ce942700 1 -- start start 2021-02-10T11:29:59.583 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:59.579 7fd3ce942700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd3c818d870 0x7fd3c80b19b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:59.583 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:59.579 7fd3ce942700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd3c80b1fd0 0x7fd3c8099ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:59.583 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:59.579 7fd3ce942700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd3c80b23d0 0x7fd3c809a510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:59.583 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:59.579 7fd3ce942700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3c818dc70 con 0x7fd3c818d870 2021-02-10T11:29:59.583 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:59.579 7fd3ce942700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3c8199e60 con 0x7fd3c80b23d0 2021-02-10T11:29:59.583 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:59.579 7fd3ce942700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3c809ab30 con 0x7fd3c80b1fd0 2021-02-10T11:29:59.583 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:59.579 7fd3c7fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd3c818d870 0x7fd3c80b19b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:59.583 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:59.579 7fd3c7fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd3c818d870 0x7fd3c80b19b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:60572/0 (socket says 172.21.2.102:60572) 2021-02-10T11:29:59.583 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:59.579 7fd3c7fff700 1 -- 172.21.2.102:0/304313272 learned_addr learned my addr 172.21.2.102:0/304313272 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:29:59.583 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:59.580 7fd3ccf5f700 1 --2- 172.21.2.102:0/304313272 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd3c80b23d0 0x7fd3c809a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:59.583 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:59.580 7fd3c7fff700 1 -- 172.21.2.102:0/304313272 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd3c80b1fd0 msgr2=0x7fd3c8099ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:59.583 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:59.580 7fd3c7fff700 1 --2- 172.21.2.102:0/304313272 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd3c80b1fd0 0x7fd3c8099ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:59.584 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:59.580 7fd3c7fff700 1 -- 172.21.2.102:0/304313272 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd3c80b23d0 msgr2=0x7fd3c809a510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:59.584 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:59.580 7fd3c7fff700 1 --2- 172.21.2.102:0/304313272 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd3c80b23d0 0x7fd3c809a510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:59.584 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:59.580 7fd3c7fff700 1 -- 172.21.2.102:0/304313272 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3c809ad50 con 0x7fd3c818d870 2021-02-10T11:29:59.584 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:59.580 7fd3ccf5f700 1 --2- 172.21.2.102:0/304313272 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd3c80b23d0 0x7fd3c809a510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:29:59.584 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:59.580 7fd3c7fff700 1 --2- 172.21.2.102:0/304313272 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd3c818d870 0x7fd3c80b19b0 secure :-1 s=READY pgs=1911 cs=0 l=1 rev1=1 rx=0x7fd3b8009630 tx=0x7fd3b800b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:29:59.584 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:59.580 7fd3c4ff9700 1 -- 172.21.2.102:0/304313272 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd3b80093a0 con 0x7fd3c818d870 2021-02-10T11:29:59.584 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:59.580 7fd3c4ff9700 1 -- 172.21.2.102:0/304313272 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3b800bd70 con 0x7fd3c818d870 2021-02-10T11:29:59.584 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:59.580 7fd3c4ff9700 1 -- 172.21.2.102:0/304313272 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd3b8005560 con 0x7fd3c818d870 2021-02-10T11:29:59.584 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:59.580 7fd3ce942700 1 -- 172.21.2.102:0/304313272 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3c809ad50 con 0x7fd3c818d870 2021-02-10T11:29:59.584 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:59.580 7fd3ce942700 1 -- 172.21.2.102:0/304313272 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3c818e3f0 con 0x7fd3c818d870 2021-02-10T11:29:59.584 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:59.581 7fd3c4ff9700 1 -- 172.21.2.102:0/304313272 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd3b800a000 con 0x7fd3c818d870 2021-02-10T11:29:59.584 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:59.581 7fd3c4ff9700 1 --2- 172.21.2.102:0/304313272 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd3a80186c0 0x7fd3a801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:59.584 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:59.581 7fd3c4ff9700 1 -- 172.21.2.102:0/304313272 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 4753+0+0 (secure 0 0 0) 0x7fd3b80320c0 con 0x7fd3c818d870 2021-02-10T11:29:59.584 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:59.581 7fd3c77fe700 1 --2- 172.21.2.102:0/304313272 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd3a80186c0 0x7fd3a801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:59.585 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:59.581 7fd3c77fe700 1 --2- 172.21.2.102:0/304313272 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd3a80186c0 0x7fd3a801abc0 crc :-1 s=READY pgs=856 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:29:59.611 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:59.610 7fd3ce942700 1 -- 172.21.2.102:0/304313272 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3c80999c0 con 0x7fd3c818d870 2021-02-10T11:29:59.613 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:59.613 7fd3c4ff9700 1 -- 172.21.2.102:0/304313272 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd3b8025fa0 con 0x7fd3c818d870 2021-02-10T11:29:59.747 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:59.747 7fd3ce942700 1 -- 172.21.2.102:0/304313272 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fd3c80999c0 con 0x7fd3c818d870 2021-02-10T11:29:59.747 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:59.747 7fd3c4ff9700 1 -- 172.21.2.102:0/304313272 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v50) v1 ==== 80+0+4513 (secure 0 0 0) 0x7fd3b8025fa0 con 0x7fd3c818d870 2021-02-10T11:29:59.748 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:29:59.748 INFO:teuthology.orchestra.run.gibba002.stdout:{ 2021-02-10T11:29:59.748 INFO:teuthology.orchestra.run.gibba002.stdout: "auth_dump": [ 2021-02-10T11:29:59.748 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:29:59.748 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "mds.a", 2021-02-10T11:29:59.748 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAevSNgsgyEARAAvr0cUjjCoEHLxaZSZ2HtVA==", 2021-02-10T11:29:59.749 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:29:59.749 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow", 2021-02-10T11:29:59.749 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:29:59.749 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:29:59.749 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:29:59.749 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:29:59.749 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:29:59.749 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:29:59.749 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "mds.b", 2021-02-10T11:29:59.749 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAevSNgwOFMMRAALfb1fbX56ao1TU8ClnTW1A==", 2021-02-10T11:29:59.749 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:29:59.749 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow", 2021-02-10T11:29:59.749 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:29:59.750 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:29:59.750 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:29:59.750 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:29:59.750 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:29:59.750 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:29:59.750 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.0", 2021-02-10T11:29:59.750 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAmvSNgj+bMMBAAD3HUD6fLiMAg+++iNzLJKQ==", 2021-02-10T11:29:59.750 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:29:59.750 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:29:59.750 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:29:59.750 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:29:59.750 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:29:59.750 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:29:59.751 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:29:59.751 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.1", 2021-02-10T11:29:59.751 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQApvSNgnvc+MRAAAOxqlNwiyLHqNITOrYeFKQ==", 2021-02-10T11:29:59.751 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:29:59.751 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:29:59.751 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:29:59.751 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:29:59.751 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:29:59.751 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:29:59.751 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:29:59.751 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.2", 2021-02-10T11:29:59.751 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAsvSNgZ4SbMRAA76IOYWfbiAuBoM/4k81GxA==", 2021-02-10T11:29:59.751 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:29:59.751 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:29:59.752 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:29:59.752 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:29:59.752 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:29:59.752 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:29:59.752 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:29:59.752 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.3", 2021-02-10T11:29:59.752 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAvvSNgFk3rMRAAWCmKKyw0LDQ+UlAHZAITOw==", 2021-02-10T11:29:59.752 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:29:59.752 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:29:59.752 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:29:59.752 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:29:59.752 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:29:59.752 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:29:59.752 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:29:59.753 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.4", 2021-02-10T11:29:59.753 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQA4vSNgpKcNJBAApJMmvj5YxVHPDOmvmQONOg==", 2021-02-10T11:29:59.753 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:29:59.753 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:29:59.753 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:29:59.753 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:29:59.753 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:29:59.753 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:29:59.753 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:29:59.753 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.5", 2021-02-10T11:29:59.753 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQA7vSNgf2ujJBAAfkuYpyb3WTzc8V5kw+Wktg==", 2021-02-10T11:29:59.753 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:29:59.753 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:29:59.754 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:29:59.754 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:29:59.754 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:29:59.754 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:29:59.754 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:29:59.754 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.6", 2021-02-10T11:29:59.754 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQA+vSNgqw9uJRAAFMU0R+UxU9C3o/9byb3e0g==", 2021-02-10T11:29:59.754 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:29:59.754 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:29:59.754 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:29:59.754 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:29:59.754 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:29:59.754 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:29:59.754 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:29:59.755 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.7", 2021-02-10T11:29:59.755 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQBBvSNgvWSrJRAA2KaCHmLpzovRgYzLvyofTQ==", 2021-02-10T11:29:59.755 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:29:59.755 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:29:59.755 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:29:59.755 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:29:59.755 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:29:59.755 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:29:59.755 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:29:59.755 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "client.0", 2021-02-10T11:29:59.755 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAfvSNgZd9XIBAAvyrvl8fdKuHnJZfczfRoFw==", 2021-02-10T11:29:59.755 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:29:59.755 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow", 2021-02-10T11:29:59.756 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow r", 2021-02-10T11:29:59.756 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:29:59.756 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:29:59.756 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:29:59.756 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:29:59.756 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "client.1", 2021-02-10T11:29:59.756 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAfvSNgSyfXNhAAIduPnF7Vd/6HSkJ0MlcMSA==", 2021-02-10T11:29:59.756 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:29:59.756 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow", 2021-02-10T11:29:59.756 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow r", 2021-02-10T11:29:59.756 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:29:59.756 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:29:59.756 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:29:59.756 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:29:59.757 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "client.2", 2021-02-10T11:29:59.757 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAgvSNgiIG/ERAA9ZTamaTI7ZsPSnxBqjplCg==", 2021-02-10T11:29:59.757 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:29:59.757 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow", 2021-02-10T11:29:59.757 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow r", 2021-02-10T11:29:59.757 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:29:59.757 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:29:59.757 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:29:59.757 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:29:59.757 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "client.3", 2021-02-10T11:29:59.757 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAgvSNgOWpOKBAAlXsEB04Q/GvtL06PqRMLeA==", 2021-02-10T11:29:59.757 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:29:59.757 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow", 2021-02-10T11:29:59.757 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow r", 2021-02-10T11:29:59.758 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:29:59.758 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:29:59.758 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:29:59.758 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:29:59.758 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "client.admin", 2021-02-10T11:29:59.758 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAavSNgO3GQBhAAL+NVnYskQRKVWLgyDPY/xw==", 2021-02-10T11:29:59.758 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:29:59.758 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow *", 2021-02-10T11:29:59.758 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:29:59.758 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:29:59.758 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:29:59.758 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:29:59.758 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:29:59.759 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:29:59.759 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "mgr.x", 2021-02-10T11:29:59.759 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAdvSNgXZZgJhAA7dtBSGzH+DsBwnddZpOBJA==", 2021-02-10T11:29:59.759 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:29:59.759 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow *", 2021-02-10T11:29:59.759 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow profile mgr", 2021-02-10T11:29:59.759 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:29:59.759 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:29:59.759 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:29:59.759 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:29:59.759 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "mgr.y", 2021-02-10T11:29:59.759 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAdvSNgVqQFDRAAwo6YEdWZCAr5GdOWVL/Qvg==", 2021-02-10T11:29:59.759 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:29:59.759 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow *", 2021-02-10T11:29:59.760 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow profile mgr", 2021-02-10T11:29:59.760 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:29:59.760 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:29:59.760 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:29:59.760 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:29:59.760 INFO:teuthology.orchestra.run.gibba002.stdout:} 2021-02-10T11:29:59.761 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:59.749 7fd3ce942700 1 -- 172.21.2.102:0/304313272 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd3a80186c0 msgr2=0x7fd3a801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:59.761 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:59.749 7fd3ce942700 1 --2- 172.21.2.102:0/304313272 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd3a80186c0 0x7fd3a801abc0 crc :-1 s=READY pgs=856 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:59.761 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:59.749 7fd3ce942700 1 -- 172.21.2.102:0/304313272 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd3c818d870 msgr2=0x7fd3c80b19b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:59.761 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:59.749 7fd3ce942700 1 --2- 172.21.2.102:0/304313272 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd3c818d870 0x7fd3c80b19b0 secure :-1 s=READY pgs=1911 cs=0 l=1 rev1=1 rx=0x7fd3b8009630 tx=0x7fd3b800b980).stop 2021-02-10T11:29:59.761 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:59.749 7fd3ce942700 1 -- 172.21.2.102:0/304313272 shutdown_connections 2021-02-10T11:29:59.761 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:59.749 7fd3ce942700 1 --2- 172.21.2.102:0/304313272 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd3a80186c0 0x7fd3a801abc0 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:59.761 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:59.749 7fd3ce942700 1 --2- 172.21.2.102:0/304313272 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd3c80b23d0 0x7fd3c809a510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:59.761 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:59.749 7fd3ce942700 1 --2- 172.21.2.102:0/304313272 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd3c80b1fd0 0x7fd3c8099ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:59.762 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:59.749 7fd3ce942700 1 --2- 172.21.2.102:0/304313272 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd3c818d870 0x7fd3c80b19b0 unknown :-1 s=CLOSED pgs=1911 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:59.762 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:59.749 7fd3ce942700 1 -- 172.21.2.102:0/304313272 >> 172.21.2.102:0/304313272 conn(0x7fd3c80f0550 msgr2=0x7fd3c8191730 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:29:59.762 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:59.750 7fd3ce942700 1 -- 172.21.2.102:0/304313272 shutdown_connections 2021-02-10T11:29:59.762 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:29:59.750 7fd3ce942700 1 -- 172.21.2.102:0/304313272 wait complete. 2021-02-10T11:29:59.762 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:30:00.193 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:00.175 7f58e961d700 1 Processor -- start 2021-02-10T11:30:00.193 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:00.175 7f58e961d700 1 -- start start 2021-02-10T11:30:00.193 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:00.175 7f58e961d700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f58e40d0340 0x7f58e40ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:00.193 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:00.175 7f58e961d700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f58e40d1c70 0x7f58e40efc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:00.193 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:00.175 7f58e961d700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58e40f0b10 con 0x7f58e40d1870 2021-02-10T11:30:00.193 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:00.175 7f58e961d700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58e40e5ed0 con 0x7f58e40d0340 2021-02-10T11:30:00.193 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:00.175 7f58e961d700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58e41b0be0 con 0x7f58e40d1c70 2021-02-10T11:30:00.193 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:00.175 7f58daabe700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f58e40d0340 0x7f58e40ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:00.193 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:00.175 7f58daabe700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f58e40d0340 0x7f58e40ef640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:60580/0 (socket says 172.21.2.102:60580) 2021-02-10T11:30:00.193 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:00.175 7f58daabe700 1 -- 172.21.2.102:0/890662326 learned_addr learned my addr 172.21.2.102:0/890662326 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:30:00.194 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:00.176 7f58e3ac0700 1 --2- 172.21.2.102:0/890662326 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f58e40d1c70 0x7f58e40efc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:00.194 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:00.176 7f58daabe700 1 -- 172.21.2.102:0/890662326 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f58e40d1c70 msgr2=0x7f58e40efc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:00.194 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:00.176 7f58daabe700 1 --2- 172.21.2.102:0/890662326 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f58e40d1c70 0x7f58e40efc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:00.194 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:00.176 7f58daabe700 1 -- 172.21.2.102:0/890662326 >> v1:172.21.2.113:6789/0 conn(0x7f58e40d1870 legacy=0x7f58e41b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:00.194 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:00.176 7f58daabe700 1 -- 172.21.2.102:0/890662326 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58e40f20a0 con 0x7f58e40d0340 2021-02-10T11:30:00.194 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:00.176 7f58daabe700 1 --2- 172.21.2.102:0/890662326 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f58e40d0340 0x7f58e40ef640 secure :-1 s=READY pgs=1912 cs=0 l=1 rev1=1 rx=0x7f58cc006700 tx=0x7f58cc009b80).ready entity=mon.0 client_cookie=f1dab487539d4fe3 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:00.194 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:00.176 7f58e2abe700 1 -- 172.21.2.102:0/890662326 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f58cc00c0c0 con 0x7f58e40d0340 2021-02-10T11:30:00.194 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:00.176 7f58e2abe700 1 -- 172.21.2.102:0/890662326 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58cc00c440 con 0x7f58e40d0340 2021-02-10T11:30:00.194 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:00.176 7f58e2abe700 1 -- 172.21.2.102:0/890662326 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f58cc00c8a0 con 0x7f58e40d0340 2021-02-10T11:30:00.194 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:00.176 7f58e961d700 1 -- 172.21.2.102:0/890662326 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f58e40d0340 msgr2=0x7f58e40ef640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:00.195 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:00.176 7f58e961d700 1 --2- 172.21.2.102:0/890662326 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f58e40d0340 0x7f58e40ef640 secure :-1 s=READY pgs=1912 cs=0 l=1 rev1=1 rx=0x7f58cc006700 tx=0x7f58cc009b80).stop 2021-02-10T11:30:00.195 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:00.177 7f58e961d700 1 -- 172.21.2.102:0/890662326 shutdown_connections 2021-02-10T11:30:00.195 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:00.177 7f58e961d700 1 --2- 172.21.2.102:0/890662326 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f58e40d1c70 0x7f58e40efc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:00.195 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:00.177 7f58e961d700 1 --2- 172.21.2.102:0/890662326 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f58e40d0340 0x7f58e40ef640 unknown :-1 s=CLOSED pgs=1912 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:00.195 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:00.177 7f58e961d700 1 -- 172.21.2.102:0/890662326 >> 172.21.2.102:0/890662326 conn(0x7f58e40ce450 msgr2=0x7f58e40d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:00.195 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:00.177 7f58e961d700 1 -- 172.21.2.102:0/890662326 shutdown_connections 2021-02-10T11:30:00.195 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:00.177 7f58e961d700 1 -- 172.21.2.102:0/890662326 wait complete. 2021-02-10T11:30:00.195 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:00.192 7f58e961d700 1 Processor -- start 2021-02-10T11:30:00.196 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:00.192 7f58e961d700 1 -- start start 2021-02-10T11:30:00.197 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:00.193 7f58e961d700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f58e40f0280 0x7f58e40a2310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:00.197 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:00.193 7f58e961d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f58e40f0680 0x7f58e40a2930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:00.197 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:00.193 7f58e961d700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f58e40ec180 0x7f58e40ec580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:00.197 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:00.193 7f58e961d700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58e40efd00 con 0x7f58e40f0680 2021-02-10T11:30:00.197 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:00.193 7f58e961d700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58e40ecba0 con 0x7f58e40f0280 2021-02-10T11:30:00.197 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:00.193 7f58e961d700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58e40ecd80 con 0x7f58e40ec180 2021-02-10T11:30:00.197 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:00.193 7f58daabe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f58e40f0680 0x7f58e40a2930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:00.197 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:00.193 7f58e32bf700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f58e40f0280 0x7f58e40a2310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:00.197 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:00.193 7f58daabe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f58e40f0680 0x7f58e40a2930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:60588/0 (socket says 172.21.2.102:60588) 2021-02-10T11:30:00.197 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:00.193 7f58e32bf700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f58e40f0280 0x7f58e40a2310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.113:3300/0 says I am v2:172.21.2.102:35784/0 (socket says 172.21.2.102:35784) 2021-02-10T11:30:00.197 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:00.193 7f58daabe700 1 -- 172.21.2.102:0/1207840334 learned_addr learned my addr 172.21.2.102:0/1207840334 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:30:00.197 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:00.193 7f58e3ac0700 1 --2- 172.21.2.102:0/1207840334 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f58e40ec180 0x7f58e40ec580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:00.198 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:00.193 7f58daabe700 1 -- 172.21.2.102:0/1207840334 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f58e40ec180 msgr2=0x7f58e40ec580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:00.198 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:00.193 7f58daabe700 1 --2- 172.21.2.102:0/1207840334 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f58e40ec180 0x7f58e40ec580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:00.198 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:00.193 7f58daabe700 1 -- 172.21.2.102:0/1207840334 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f58e40f0280 msgr2=0x7f58e40a2310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:00.198 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:00.193 7f58daabe700 1 --2- 172.21.2.102:0/1207840334 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f58e40f0280 0x7f58e40a2310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:00.198 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:00.193 7f58daabe700 1 -- 172.21.2.102:0/1207840334 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58e40ecfa0 con 0x7f58e40f0680 2021-02-10T11:30:00.198 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:00.193 7f58e32bf700 1 --2- 172.21.2.102:0/1207840334 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f58e40f0280 0x7f58e40a2310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:30:00.198 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:00.193 7f58daabe700 1 --2- 172.21.2.102:0/1207840334 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f58e40f0680 0x7f58e40a2930 secure :-1 s=READY pgs=1913 cs=0 l=1 rev1=1 rx=0x7f58cc00b580 tx=0x7f58cc00b920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:00.198 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:00.194 7f58e0aba700 1 -- 172.21.2.102:0/1207840334 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f58cc00bd10 con 0x7f58e40f0680 2021-02-10T11:30:00.198 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:00.194 7f58e0aba700 1 -- 172.21.2.102:0/1207840334 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58cc002030 con 0x7f58e40f0680 2021-02-10T11:30:00.198 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:00.194 7f58e0aba700 1 -- 172.21.2.102:0/1207840334 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f58cc002400 con 0x7f58e40f0680 2021-02-10T11:30:00.198 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:00.194 7f58e961d700 1 -- 172.21.2.102:0/1207840334 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58e40ecfa0 con 0x7f58e40f0680 2021-02-10T11:30:00.198 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:00.194 7f58e961d700 1 -- 172.21.2.102:0/1207840334 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58e40ed530 con 0x7f58e40f0680 2021-02-10T11:30:00.198 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:00.194 7f58e0aba700 1 -- 172.21.2.102:0/1207840334 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f58cc000c90 con 0x7f58e40f0680 2021-02-10T11:30:00.199 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:00.194 7f58e0aba700 1 --2- 172.21.2.102:0/1207840334 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f58bc0186c0 0x7f58bc01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:00.199 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:00.194 7f58e0aba700 1 -- 172.21.2.102:0/1207840334 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 4753+0+0 (secure 0 0 0) 0x7f58cc0322a0 con 0x7f58e40f0680 2021-02-10T11:30:00.199 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:00.194 7f58e32bf700 1 --2- 172.21.2.102:0/1207840334 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f58bc0186c0 0x7f58bc01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:00.199 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:00.195 7f58e32bf700 1 --2- 172.21.2.102:0/1207840334 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f58bc0186c0 0x7f58bc01abc0 crc :-1 s=READY pgs=857 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:00.224 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:00.224 7f58e961d700 1 -- 172.21.2.102:0/1207840334 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58e40ebc60 con 0x7f58e40f0680 2021-02-10T11:30:00.227 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:00.227 7f58e0aba700 1 -- 172.21.2.102:0/1207840334 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f58cc030d80 con 0x7f58e40f0680 2021-02-10T11:30:00.357 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:00.354 7f58e961d700 1 -- 172.21.2.102:0/1207840334 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f58e40ebc60 con 0x7f58e40f0680 2021-02-10T11:30:00.357 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:00.355 7f58e0aba700 1 -- 172.21.2.102:0/1207840334 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+11145 (secure 0 0 0) 0x7f58cc000c90 con 0x7f58e40f0680 2021-02-10T11:30:00.357 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:30:00.357 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":89,"fsid":"68db30ea-dbce-4aef-a6df-f01c6fea0513","created":"2021-02-10 11:03:06.331557","modified":"2021-02-10 11:29:59.097465","last_up_change":"2021-02-10 11:03:32.690562","last_in_change":"2021-02-10 11:03:32.690562","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:03:37.322519","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"be691b4d-a039-408a-af80-9d53f02b6f1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6818","nonce":36088},{"type":"v1","addr":"172.21.2.102:6820","nonce":36088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6822","nonce":36088},{"type":"v1","addr":"172.21.2.102:6824","nonce":36088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6830","nonce":36088},{"type":"v1","addr":"172.21.2.102:6832","nonce":36088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6826","nonce":36088},{"type":"v1","addr":"172.21.2.102:6828","nonce":36088}]},"public_addr":"172.21.2.102:6820/36088","cluster_addr":"172.21.2.102:6824/36088","heartbeat_back_addr":"172.21.2.102:6832/36088","heartbeat_front_addr":"172.21.2.102:6828/36088","state":["exists","up"]},{"osd":1,"uuid":"64e3ccc8-9c69-4d7c-865e-af8e83822635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6819","nonce":36087},{"type":"v1","addr":"172.21.2.102:6821","nonce":36087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6823","nonce":36087},{"type":"v1","addr":"172.21.2.102:6825","nonce":36087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6831","nonce":36087},{"type":"v1","addr":"172.21.2.102:6833","nonce":36087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6827","nonce":36087},{"type":"v1","addr":"172.21.2.102:6829","nonce":36087}]},"public_addr":"172.21.2.102:6821/36087","cluster_addr":"172.21.2.102:6825/36087","heartbeat_back_addr":"172.21.2.102:6833/36087","heartbeat_front_addr":"172.21.2.102:6829/36087","state":["exists","up"]},{"osd":2,"uuid":"153700af-747f-481f-bdf4-d6072d3a842f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6810","nonce":36089},{"type":"v1","addr":"172.21.2.102:6811","nonce":36089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6812","nonce":36089},{"type":"v1","addr":"172.21.2.102:6813","nonce":36089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6816","nonce":36089},{"type":"v1","addr":"172.21.2.102:6817","nonce":36089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6814","nonce":36089},{"type":"v1","addr":"172.21.2.102:6815","nonce":36089}]},"public_addr":"172.21.2.102:6811/36089","cluster_addr":"172.21.2.102:6813/36089","heartbeat_back_addr":"172.21.2.102:6817/36089","heartbeat_front_addr":"172.21.2.102:6815/36089","state":["exists","up"]},{"osd":3,"uuid":"cbc678f4-83a9-45e6-8830-2f7c232b1ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6802","nonce":36090},{"type":"v1","addr":"172.21.2.102:6803","nonce":36090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6804","nonce":36090},{"type":"v1","addr":"172.21.2.102:6805","nonce":36090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6808","nonce":36090},{"type":"v1","addr":"172.21.2.102:6809","nonce":36090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6806","nonce":36090},{"type":"v1","addr":"172.21.2.102:6807","nonce":36090}]},"public_addr":"172.21.2.102:6803/36090","cluster_addr":"172.21.2.102:6805/36090","heartbeat_back_addr":"172.21.2.102:6809/36090","heartbeat_front_addr":"172.21.2.102:6807/36090","state":["exists","up"]},{"osd":4,"uuid":"c01e4008-b258-4f10-afd2-9e7f37214351","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6808","nonce":36368},{"type":"v1","addr":"172.21.2.113:6811","nonce":36368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6814","nonce":36368},{"type":"v1","addr":"172.21.2.113:6818","nonce":36368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6826","nonce":36368},{"type":"v1","addr":"172.21.2.113:6828","nonce":36368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6820","nonce":36368},{"type":"v1","addr":"172.21.2.113:6824","nonce":36368}]},"public_addr":"172.21.2.113:6811/36368","cluster_addr":"172.21.2.113:6818/36368","heartbeat_back_addr":"172.21.2.113:6828/36368","heartbeat_front_addr":"172.21.2.113:6824/36368","state":["exists","up"]},{"osd":5,"uuid":"551f6007-daf8-4a98-8ee0-c2b16bcccb5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6802","nonce":36366},{"type":"v1","addr":"172.21.2.113:6805","nonce":36366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6807","nonce":36366},{"type":"v1","addr":"172.21.2.113:6810","nonce":36366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6819","nonce":36366},{"type":"v1","addr":"172.21.2.113:6822","nonce":36366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6812","nonce":36366},{"type":"v1","addr":"172.21.2.113:6815","nonce":36366}]},"public_addr":"172.21.2.113:6805/36366","cluster_addr":"172.21.2.113:6810/36366","heartbeat_back_addr":"172.21.2.113:6822/36366","heartbeat_front_addr":"172.21.2.113:6815/36366","state":["exists","up"]},{"osd":6,"uuid":"bf67c040-0e50-48be-b102-c3943bc9ebef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6800","nonce":36367},{"type":"v1","addr":"172.21.2.113:6801","nonce":36367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6803","nonce":36367},{"type":"v1","addr":"172.21.2.113:6804","nonce":36367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6813","nonce":36367},{"type":"v1","addr":"172.21.2.113:6816","nonce":36367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6806","nonce":36367},{"type":"v1","addr":"172.21.2.113:6809","nonce":36367}]},"public_addr":"172.21.2.113:6801/36367","cluster_addr":"172.21.2.113:6804/36367","heartbeat_back_addr":"172.21.2.113:6816/36367","heartbeat_front_addr":"172.21.2.113:6809/36367","state":["exists","up"]},{"osd":7,"uuid":"f5461f80-018d-428f-be89-2eb2bcbe7bb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6817","nonce":36365},{"type":"v1","addr":"172.21.2.113:6821","nonce":36365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6823","nonce":36365},{"type":"v1","addr":"172.21.2.113:6825","nonce":36365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6830","nonce":36365},{"type":"v1","addr":"172.21.2.113:6831","nonce":36365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6827","nonce":36365},{"type":"v1","addr":"172.21.2.113:6829","nonce":36365}]},"public_addr":"172.21.2.113:6821/36365","cluster_addr":"172.21.2.113:6825/36365","heartbeat_back_addr":"172.21.2.113:6831/36365","heartbeat_front_addr":"172.21.2.113:6829/36365","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:30:00.358 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:00.357 7f58e961d700 1 -- 172.21.2.102:0/1207840334 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f58bc0186c0 msgr2=0x7f58bc01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:00.358 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:00.357 7f58e961d700 1 --2- 172.21.2.102:0/1207840334 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f58bc0186c0 0x7f58bc01abc0 crc :-1 s=READY pgs=857 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:00.358 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:00.357 7f58e961d700 1 -- 172.21.2.102:0/1207840334 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f58e40f0680 msgr2=0x7f58e40a2930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:00.358 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:00.357 7f58e961d700 1 --2- 172.21.2.102:0/1207840334 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f58e40f0680 0x7f58e40a2930 secure :-1 s=READY pgs=1913 cs=0 l=1 rev1=1 rx=0x7f58cc00b580 tx=0x7f58cc00b920).stop 2021-02-10T11:30:00.358 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:00.357 7f58e961d700 1 -- 172.21.2.102:0/1207840334 shutdown_connections 2021-02-10T11:30:00.359 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:00.357 7f58e961d700 1 --2- 172.21.2.102:0/1207840334 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f58e40ec180 0x7f58e40ec580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:00.359 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:00.357 7f58e961d700 1 --2- 172.21.2.102:0/1207840334 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f58e40f0680 0x7f58e40a2930 unknown :-1 s=CLOSED pgs=1913 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:00.359 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:00.357 7f58e961d700 1 --2- 172.21.2.102:0/1207840334 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f58bc0186c0 0x7f58bc01abc0 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:00.359 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:00.357 7f58e961d700 1 --2- 172.21.2.102:0/1207840334 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f58e40f0280 0x7f58e40a2310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:00.359 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:00.357 7f58e961d700 1 -- 172.21.2.102:0/1207840334 >> 172.21.2.102:0/1207840334 conn(0x7f58e40c2130 msgr2=0x7f58e409e3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:00.359 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:00.357 7f58e961d700 1 -- 172.21.2.102:0/1207840334 shutdown_connections 2021-02-10T11:30:00.359 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:00.357 7f58e961d700 1 -- 172.21.2.102:0/1207840334 wait complete. 2021-02-10T11:30:00.366 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:03:37.322519', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2021-02-10T11:30:00.366 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:30:00.799 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:00.780 7ff66a662700 1 Processor -- start 2021-02-10T11:30:00.799 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:00.781 7ff66a662700 1 -- start start 2021-02-10T11:30:00.799 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:00.781 7ff66a662700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7ff6640d1860 0x7ff6641b04b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:00.799 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:00.781 7ff66a662700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff6640cff60 0x7ff664109120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:00.799 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:00.781 7ff66a662700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff664109a50 con 0x7ff6641b0ad0 2021-02-10T11:30:00.799 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:00.781 7ff66a662700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff664109740 con 0x7ff6640d1860 2021-02-10T11:30:00.799 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:00.781 7ff66a662700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff664088070 con 0x7ff6640cff60 2021-02-10T11:30:00.800 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:00.781 7ff6637fe700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7ff6641b0ad0 0x7ff664108930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:52822/0 (socket says 172.21.2.102:52822) 2021-02-10T11:30:00.800 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:00.781 7ff6637fe700 1 -- 172.21.2.102:0/1181259905 learned_addr learned my addr 172.21.2.102:0/1181259905 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:30:00.800 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:00.782 7ff662ffd700 1 -- 172.21.2.102:0/1181259905 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2698867066 0 0) 0x7ff6580009b0 con 0x7ff6641b0ad0 2021-02-10T11:30:00.800 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:00.782 7ff662ffd700 1 -- 172.21.2.102:0/1181259905 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff64c002fe0 con 0x7ff6641b0ad0 2021-02-10T11:30:00.800 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:00.782 7ff663fff700 1 --2- 172.21.2.102:0/1181259905 >> v2:172.21.2.113:3300/0 conn(0x7ff6640d1860 0x7ff6641b04b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:00.800 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:00.782 7ff668c7f700 1 --2- 172.21.2.102:0/1181259905 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff6640cff60 0x7ff664109120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:00.800 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:00.782 7ff662ffd700 1 -- 172.21.2.102:0/1181259905 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2256607613 0 0) 0x7ff658001010 con 0x7ff6641b0ad0 2021-02-10T11:30:00.800 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:00.782 7ff662ffd700 1 -- 172.21.2.102:0/1181259905 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff6640cff60 msgr2=0x7ff664109120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:00.800 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:00.782 7ff662ffd700 1 --2- 172.21.2.102:0/1181259905 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff6640cff60 0x7ff664109120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:00.800 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:00.782 7ff662ffd700 1 -- 172.21.2.102:0/1181259905 >> v2:172.21.2.113:3300/0 conn(0x7ff6640d1860 msgr2=0x7ff6641b04b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:00.800 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:00.782 7ff662ffd700 1 --2- 172.21.2.102:0/1181259905 >> v2:172.21.2.113:3300/0 conn(0x7ff6640d1860 0x7ff6641b04b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:00.800 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:00.782 7ff662ffd700 1 -- 172.21.2.102:0/1181259905 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6640d1c60 con 0x7ff6641b0ad0 2021-02-10T11:30:00.801 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:00.782 7ff662ffd700 1 -- 172.21.2.102:0/1181259905 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7ff658001590 con 0x7ff6641b0ad0 2021-02-10T11:30:00.801 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:00.782 7ff662ffd700 1 -- 172.21.2.102:0/1181259905 >> v1:172.21.2.102:6789/0 conn(0x7ff6641b0ad0 legacy=0x7ff664108930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:00.801 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:00.782 7ff662ffd700 1 --2- 172.21.2.102:0/1181259905 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff64c003b90 0x7ff64c005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:00.801 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:00.782 7ff662ffd700 1 --2- 172.21.2.102:0/1181259905 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff64c006820 0x7ff64c008d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:00.801 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:00.782 7ff662ffd700 1 --2- 172.21.2.102:0/1181259905 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff64c0095a0 0x7ff64c00bac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:00.801 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:00.782 7ff662ffd700 1 -- 172.21.2.102:0/1181259905 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff64c050380 con 0x7ff64c006820 2021-02-10T11:30:00.801 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:00.782 7ff662ffd700 1 -- 172.21.2.102:0/1181259905 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff64c050560 con 0x7ff64c0095a0 2021-02-10T11:30:00.801 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:00.782 7ff662ffd700 1 -- 172.21.2.102:0/1181259905 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff64c050780 con 0x7ff64c003b90 2021-02-10T11:30:00.801 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:00.782 7ff663fff700 1 --2- 172.21.2.102:0/1181259905 >> v2:172.21.2.113:3300/0 conn(0x7ff6640d1860 0x7ff6641b04b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:30:00.801 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:00.782 7ff6637fe700 1 --2- 172.21.2.102:0/1181259905 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff64c006820 0x7ff64c008d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:00.801 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:00.782 7ff668c7f700 1 --2- 172.21.2.102:0/1181259905 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff64c0095a0 0x7ff64c00bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:00.801 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:00.782 7ff663fff700 1 --2- 172.21.2.102:0/1181259905 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff64c003b90 0x7ff64c005fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:00.802 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:00.783 7ff6637fe700 1 -- 172.21.2.102:0/1181259905 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff64c003b90 msgr2=0x7ff64c005fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:00.802 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:00.783 7ff6637fe700 1 --2- 172.21.2.102:0/1181259905 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff64c003b90 0x7ff64c005fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:00.802 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:00.783 7ff6637fe700 1 -- 172.21.2.102:0/1181259905 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff64c0095a0 msgr2=0x7ff64c00bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:00.802 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:00.783 7ff6637fe700 1 --2- 172.21.2.102:0/1181259905 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff64c0095a0 0x7ff64c00bac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:00.802 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:00.783 7ff6637fe700 1 -- 172.21.2.102:0/1181259905 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff64c050a20 con 0x7ff64c006820 2021-02-10T11:30:00.802 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:00.783 7ff668c7f700 1 --2- 172.21.2.102:0/1181259905 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff64c0095a0 0x7ff64c00bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:30:00.802 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:00.783 7ff6637fe700 1 --2- 172.21.2.102:0/1181259905 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff64c006820 0x7ff64c008d20 secure :-1 s=READY pgs=1915 cs=0 l=1 rev1=1 rx=0x7ff6580017d0 tx=0x7ff658007d30).ready entity=mon.0 client_cookie=acce3bbe7c327f03 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:00.802 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:00.783 7ff662ffd700 1 -- 172.21.2.102:0/1181259905 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff65800a310 con 0x7ff64c006820 2021-02-10T11:30:00.802 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:00.783 7ff662ffd700 1 -- 172.21.2.102:0/1181259905 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff65800a310 con 0x7ff64c006820 2021-02-10T11:30:00.802 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:00.783 7ff662ffd700 1 -- 172.21.2.102:0/1181259905 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff65800a690 con 0x7ff64c006820 2021-02-10T11:30:00.802 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:00.783 7ff66a662700 1 -- 172.21.2.102:0/1181259905 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff64c006820 msgr2=0x7ff64c008d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:00.803 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:00.783 7ff66a662700 1 --2- 172.21.2.102:0/1181259905 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff64c006820 0x7ff64c008d20 secure :-1 s=READY pgs=1915 cs=0 l=1 rev1=1 rx=0x7ff6580017d0 tx=0x7ff658007d30).stop 2021-02-10T11:30:00.803 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:00.783 7ff668c7f700 1 -- 172.21.2.102:0/1181259905 reap_dead start 2021-02-10T11:30:00.803 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:00.784 7ff66a662700 1 -- 172.21.2.102:0/1181259905 shutdown_connections 2021-02-10T11:30:00.803 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:00.784 7ff66a662700 1 -- 172.21.2.102:0/1181259905 >> 172.21.2.102:0/1181259905 conn(0x7ff6640ce410 msgr2=0x7ff6640d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:00.803 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:00.784 7ff66a662700 1 -- 172.21.2.102:0/1181259905 shutdown_connections 2021-02-10T11:30:00.803 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:00.784 7ff66a662700 1 -- 172.21.2.102:0/1181259905 wait complete. 2021-02-10T11:30:00.803 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:00.798 7ff66a662700 1 Processor -- start 2021-02-10T11:30:00.803 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:00.799 7ff66a662700 1 -- start start 2021-02-10T11:30:00.804 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:00.799 7ff66a662700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff6640adf50 0x7ff6640ae350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:00.804 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:00.799 7ff66a662700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff6640ae970 0x7ff6640aed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:00.804 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:00.799 7ff66a662700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff6640982c0 0x7ff6640986c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:00.804 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:00.799 7ff66a662700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff66409f450 con 0x7ff6640982c0 2021-02-10T11:30:00.805 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:00.799 7ff66a662700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff664098ce0 con 0x7ff6640adf50 2021-02-10T11:30:00.805 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:00.799 7ff66a662700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff664098ed0 con 0x7ff6640ae970 2021-02-10T11:30:00.805 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:00.799 7ff668c7f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff6640982c0 0x7ff6640986c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:00.805 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:00.799 7ff668c7f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff6640982c0 0x7ff6640986c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:60610/0 (socket says 172.21.2.102:60610) 2021-02-10T11:30:00.805 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:00.799 7ff668c7f700 1 -- 172.21.2.102:0/3344804832 learned_addr learned my addr 172.21.2.102:0/3344804832 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:30:00.805 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:00.800 7ff668c7f700 1 -- 172.21.2.102:0/3344804832 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff6640ae970 msgr2=0x7ff6640aed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:00.805 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:00.800 7ff668c7f700 1 --2- 172.21.2.102:0/3344804832 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff6640ae970 0x7ff6640aed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:00.805 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:00.800 7ff668c7f700 1 -- 172.21.2.102:0/3344804832 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff6640adf50 msgr2=0x7ff6640ae350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:00.805 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:00.800 7ff668c7f700 1 --2- 172.21.2.102:0/3344804832 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff6640adf50 0x7ff6640ae350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:00.805 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:00.800 7ff668c7f700 1 -- 172.21.2.102:0/3344804832 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6640990f0 con 0x7ff6640982c0 2021-02-10T11:30:00.805 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:00.800 7ff668c7f700 1 --2- 172.21.2.102:0/3344804832 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff6640982c0 0x7ff6640986c0 secure :-1 s=READY pgs=1916 cs=0 l=1 rev1=1 rx=0x7ff65c0099e0 tx=0x7ff65c00bd20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:00.805 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:00.800 7ff660ff9700 1 -- 172.21.2.102:0/3344804832 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff65c001400 con 0x7ff6640982c0 2021-02-10T11:30:00.805 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:00.800 7ff660ff9700 1 -- 172.21.2.102:0/3344804832 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff65c00be60 con 0x7ff6640982c0 2021-02-10T11:30:00.806 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:00.800 7ff660ff9700 1 -- 172.21.2.102:0/3344804832 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff65c001900 con 0x7ff6640982c0 2021-02-10T11:30:00.806 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:00.800 7ff66a662700 1 -- 172.21.2.102:0/3344804832 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6640990f0 con 0x7ff6640982c0 2021-02-10T11:30:00.806 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:00.800 7ff66a662700 1 -- 172.21.2.102:0/3344804832 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff664099680 con 0x7ff6640982c0 2021-02-10T11:30:00.806 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:00.801 7ff660ff9700 1 -- 172.21.2.102:0/3344804832 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ff65c000df0 con 0x7ff6640982c0 2021-02-10T11:30:00.806 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:00.801 7ff660ff9700 1 --2- 172.21.2.102:0/3344804832 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff6440186c0 0x7ff64401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:00.806 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:00.801 7ff660ff9700 1 -- 172.21.2.102:0/3344804832 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 4753+0+0 (secure 0 0 0) 0x7ff65c0324c0 con 0x7ff6640982c0 2021-02-10T11:30:00.806 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:00.801 7ff663fff700 1 --2- 172.21.2.102:0/3344804832 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff6440186c0 0x7ff64401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:00.806 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:00.801 7ff663fff700 1 --2- 172.21.2.102:0/3344804832 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff6440186c0 0x7ff64401abc0 crc :-1 s=READY pgs=858 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:00.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:00.831 7ff66a662700 1 -- 172.21.2.102:0/3344804832 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff664082a20 con 0x7ff6640982c0 2021-02-10T11:30:00.833 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:00.833 7ff660ff9700 1 -- 172.21.2.102:0/3344804832 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ff65c001bc0 con 0x7ff6640982c0 2021-02-10T11:30:00.960 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:00.960 7ff66a662700 1 -- 172.21.2.102:0/3344804832 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7ff664082a20 con 0x7ff6640982c0 2021-02-10T11:30:00.961 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:00.960 7ff660ff9700 1 -- 172.21.2.102:0/3344804832 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v89) v1 ==== 92+0+10 (secure 0 0 0) 0x7ff65c000df0 con 0x7ff6640982c0 2021-02-10T11:30:00.961 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:30:00.963 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:00.962 7ff66a662700 1 -- 172.21.2.102:0/3344804832 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff6440186c0 msgr2=0x7ff64401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:00.963 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:00.962 7ff66a662700 1 --2- 172.21.2.102:0/3344804832 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff6440186c0 0x7ff64401abc0 crc :-1 s=READY pgs=858 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:00.963 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:00.962 7ff66a662700 1 -- 172.21.2.102:0/3344804832 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff6640982c0 msgr2=0x7ff6640986c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:00.963 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:00.962 7ff66a662700 1 --2- 172.21.2.102:0/3344804832 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff6640982c0 0x7ff6640986c0 secure :-1 s=READY pgs=1916 cs=0 l=1 rev1=1 rx=0x7ff65c0099e0 tx=0x7ff65c00bd20).stop 2021-02-10T11:30:00.963 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:00.962 7ff66a662700 1 -- 172.21.2.102:0/3344804832 shutdown_connections 2021-02-10T11:30:00.963 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:00.962 7ff66a662700 1 --2- 172.21.2.102:0/3344804832 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff6640982c0 0x7ff6640986c0 unknown :-1 s=CLOSED pgs=1916 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:00.963 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:00.962 7ff66a662700 1 --2- 172.21.2.102:0/3344804832 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff6640ae970 0x7ff6640aed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:00.963 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:00.962 7ff66a662700 1 --2- 172.21.2.102:0/3344804832 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff6440186c0 0x7ff64401abc0 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:00.963 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:00.962 7ff66a662700 1 --2- 172.21.2.102:0/3344804832 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff6640adf50 0x7ff6640ae350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:00.964 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:00.962 7ff66a662700 1 -- 172.21.2.102:0/3344804832 >> 172.21.2.102:0/3344804832 conn(0x7ff6640bee90 msgr2=0x7ff6640bb880 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:00.964 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:00.963 7ff66a662700 1 -- 172.21.2.102:0/3344804832 shutdown_connections 2021-02-10T11:30:00.964 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:00.963 7ff66a662700 1 -- 172.21.2.102:0/3344804832 wait complete. 2021-02-10T11:30:00.971 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --format=json-pretty osd lspools 2021-02-10T11:30:01.400 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:01.382 7f07003b7700 1 Processor -- start 2021-02-10T11:30:01.400 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:01.382 7f07003b7700 1 -- start start 2021-02-10T11:30:01.400 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:01.383 7f07003b7700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f06f80d1c70 0x7f06f80f20a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:01.400 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:01.383 7f07003b7700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06f80f0b10 con 0x7f06f80d0340 2021-02-10T11:30:01.401 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:01.383 7f07003b7700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06f81b0be0 con 0x7f06f80d1870 2021-02-10T11:30:01.401 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:01.383 7f07003b7700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06f80e5ed0 con 0x7f06f80d1c70 2021-02-10T11:30:01.401 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:01.383 7f06fe1d3700 1 --1- >> v1:172.21.2.113:6789/0 conn(0x7f06f80d1870 0x7f06f81b03f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.113:6789/0 says I am v1:172.21.2.102:60162/0 (socket says 172.21.2.102:60162) 2021-02-10T11:30:01.401 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:01.383 7f06fe1d3700 1 -- 172.21.2.102:0/667961710 learned_addr learned my addr 172.21.2.102:0/667961710 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:30:01.401 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:01.383 7f06fe9d4700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f06f80d1c70 0x7f06f80f20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:01.401 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:01.383 7f06fd9d2700 1 --1- 172.21.2.102:0/667961710 >> v1:172.21.2.102:6789/0 conn(0x7f06f80d0340 0x7f06f80ef640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:52838/0 (socket says 172.21.2.102:52838) 2021-02-10T11:30:01.401 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:01.384 7f06fe9d4700 1 -- 172.21.2.102:0/667961710 >> v1:172.21.2.113:6789/0 conn(0x7f06f80d1870 legacy=0x7f06f81b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:01.401 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:01.384 7f06fe9d4700 1 -- 172.21.2.102:0/667961710 >> v1:172.21.2.102:6789/0 conn(0x7f06f80d0340 legacy=0x7f06f80ef640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:01.401 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:01.384 7f06fe9d4700 1 -- 172.21.2.102:0/667961710 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06f80d2fe0 con 0x7f06f80d1c70 2021-02-10T11:30:01.401 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:01.384 7f06fe9d4700 1 --2- 172.21.2.102:0/667961710 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f06f80d1c70 0x7f06f80f20a0 secure :-1 s=READY pgs=1918 cs=0 l=1 rev1=1 rx=0x7f06f0006e30 tx=0x7f06f0009e60).ready entity=mon.0 client_cookie=4233173847a7a8b0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:01.401 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:01.384 7f06fd1d1700 1 -- 172.21.2.102:0/667961710 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f06f0006a50 con 0x7f06f80d1c70 2021-02-10T11:30:01.402 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:01.384 7f06fd1d1700 1 -- 172.21.2.102:0/667961710 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06f000b300 con 0x7f06f80d1c70 2021-02-10T11:30:01.402 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:01.384 7f06fd1d1700 1 -- 172.21.2.102:0/667961710 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f06f000b820 con 0x7f06f80d1c70 2021-02-10T11:30:01.402 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:01.384 7f07003b7700 1 -- 172.21.2.102:0/667961710 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f06f80d1c70 msgr2=0x7f06f80f20a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:01.402 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:01.384 7f07003b7700 1 --2- 172.21.2.102:0/667961710 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f06f80d1c70 0x7f06f80f20a0 secure :-1 s=READY pgs=1918 cs=0 l=1 rev1=1 rx=0x7f06f0006e30 tx=0x7f06f0009e60).stop 2021-02-10T11:30:01.402 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:01.385 7f07003b7700 1 -- 172.21.2.102:0/667961710 shutdown_connections 2021-02-10T11:30:01.402 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:01.385 7f07003b7700 1 --2- 172.21.2.102:0/667961710 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f06f80d1c70 0x7f06f80f20a0 unknown :-1 s=CLOSED pgs=1918 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:01.402 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:01.385 7f07003b7700 1 -- 172.21.2.102:0/667961710 >> 172.21.2.102:0/667961710 conn(0x7f06f80ce450 msgr2=0x7f06f80d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:01.402 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:01.385 7f07003b7700 1 -- 172.21.2.102:0/667961710 shutdown_connections 2021-02-10T11:30:01.402 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:01.385 7f07003b7700 1 -- 172.21.2.102:0/667961710 wait complete. 2021-02-10T11:30:01.402 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:01.400 7f07003b7700 1 Processor -- start 2021-02-10T11:30:01.402 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:01.400 7f07003b7700 1 -- start start 2021-02-10T11:30:01.404 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:01.400 7f07003b7700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f06f8092010 0x7f06f80c33c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:01.404 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:01.400 7f07003b7700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f06f80c39e0 0x7f06f80af040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:01.404 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:01.400 7f07003b7700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f06f80af660 0x7f06f80afa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:01.404 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:01.400 7f07003b7700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06f80b0500 con 0x7f06f80c39e0 2021-02-10T11:30:01.405 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:01.400 7f07003b7700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06f80b0080 con 0x7f06f8092010 2021-02-10T11:30:01.405 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:01.400 7f07003b7700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06f80b02a0 con 0x7f06f80af660 2021-02-10T11:30:01.405 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:01.400 7f06fd9d2700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f06f80c39e0 0x7f06f80af040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:01.405 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:01.400 7f06fd9d2700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f06f80c39e0 0x7f06f80af040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:60620/0 (socket says 172.21.2.102:60620) 2021-02-10T11:30:01.405 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:01.400 7f06fe1d3700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f06f8092010 0x7f06f80c33c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:01.406 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:01.400 7f06fd9d2700 1 -- 172.21.2.102:0/3360518603 learned_addr learned my addr 172.21.2.102:0/3360518603 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:30:01.406 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:01.400 7f06fe9d4700 1 --2- 172.21.2.102:0/3360518603 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f06f80af660 0x7f06f80afa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:01.406 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:01.401 7f06fd9d2700 1 -- 172.21.2.102:0/3360518603 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f06f80af660 msgr2=0x7f06f80afa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:01.406 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:01.401 7f06fd9d2700 1 --2- 172.21.2.102:0/3360518603 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f06f80af660 0x7f06f80afa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:01.406 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:01.401 7f06fd9d2700 1 -- 172.21.2.102:0/3360518603 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f06f8092010 msgr2=0x7f06f80c33c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:01.406 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:01.401 7f06fd9d2700 1 --2- 172.21.2.102:0/3360518603 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f06f8092010 0x7f06f80c33c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:01.407 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:01.401 7f06fd9d2700 1 -- 172.21.2.102:0/3360518603 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06f80b0a70 con 0x7f06f80c39e0 2021-02-10T11:30:01.407 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:01.401 7f06fd9d2700 1 --2- 172.21.2.102:0/3360518603 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f06f80c39e0 0x7f06f80af040 secure :-1 s=READY pgs=1919 cs=0 l=1 rev1=1 rx=0x7f06e800a5a0 tx=0x7f06e800a940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:01.407 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:01.401 7f06eeffd700 1 -- 172.21.2.102:0/3360518603 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f06e8000c90 con 0x7f06f80c39e0 2021-02-10T11:30:01.407 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:01.401 7f06eeffd700 1 -- 172.21.2.102:0/3360518603 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06e8000ed0 con 0x7f06f80c39e0 2021-02-10T11:30:01.407 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:01.401 7f06eeffd700 1 -- 172.21.2.102:0/3360518603 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f06e8004590 con 0x7f06f80c39e0 2021-02-10T11:30:01.408 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:01.401 7f07003b7700 1 -- 172.21.2.102:0/3360518603 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06f80b0a70 con 0x7f06f80c39e0 2021-02-10T11:30:01.408 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:01.401 7f07003b7700 1 -- 172.21.2.102:0/3360518603 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06f80b1000 con 0x7f06f80c39e0 2021-02-10T11:30:01.408 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:01.402 7f06eeffd700 1 -- 172.21.2.102:0/3360518603 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f06e80043a0 con 0x7f06f80c39e0 2021-02-10T11:30:01.408 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:01.402 7f06eeffd700 1 --2- 172.21.2.102:0/3360518603 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f06e00186c0 0x7f06e001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:01.408 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:01.402 7f06eeffd700 1 -- 172.21.2.102:0/3360518603 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 4753+0+0 (secure 0 0 0) 0x7f06e8032200 con 0x7f06f80c39e0 2021-02-10T11:30:01.408 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:01.402 7f06fe1d3700 1 --2- 172.21.2.102:0/3360518603 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f06e00186c0 0x7f06e001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:01.409 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:01.402 7f06fe1d3700 1 --2- 172.21.2.102:0/3360518603 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f06e00186c0 0x7f06e001abc0 crc :-1 s=READY pgs=859 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:01.433 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:01.432 7f07003b7700 1 -- 172.21.2.102:0/3360518603 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06f80e7df0 con 0x7f06f80c39e0 2021-02-10T11:30:01.435 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:01.435 7f06eeffd700 1 -- 172.21.2.102:0/3360518603 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f06e8030ce0 con 0x7f06f80c39e0 2021-02-10T11:30:01.559 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:01.559 7f07003b7700 1 -- 172.21.2.102:0/3360518603 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f06f80ce450 con 0x7f06f80c39e0 2021-02-10T11:30:01.560 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:01.559 7f06eeffd700 1 -- 172.21.2.102:0/3360518603 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v89) v1 ==== 84+0+64 (secure 0 0 0) 0x7f06e8030ce0 con 0x7f06f80c39e0 2021-02-10T11:30:01.560 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:30:01.560 INFO:teuthology.orchestra.run.gibba002.stdout:[ 2021-02-10T11:30:01.560 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:30:01.560 INFO:teuthology.orchestra.run.gibba002.stdout: "poolnum": 1, 2021-02-10T11:30:01.561 INFO:teuthology.orchestra.run.gibba002.stdout: "poolname": "rbd" 2021-02-10T11:30:01.561 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:30:01.561 INFO:teuthology.orchestra.run.gibba002.stdout:] 2021-02-10T11:30:01.562 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:01.561 7f07003b7700 1 -- 172.21.2.102:0/3360518603 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f06e00186c0 msgr2=0x7f06e001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:01.562 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:01.561 7f07003b7700 1 --2- 172.21.2.102:0/3360518603 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f06e00186c0 0x7f06e001abc0 crc :-1 s=READY pgs=859 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:01.562 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:01.561 7f07003b7700 1 -- 172.21.2.102:0/3360518603 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f06f80c39e0 msgr2=0x7f06f80af040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:01.562 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:01.561 7f07003b7700 1 --2- 172.21.2.102:0/3360518603 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f06f80c39e0 0x7f06f80af040 secure :-1 s=READY pgs=1919 cs=0 l=1 rev1=1 rx=0x7f06e800a5a0 tx=0x7f06e800a940).stop 2021-02-10T11:30:01.562 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:01.561 7f07003b7700 1 -- 172.21.2.102:0/3360518603 shutdown_connections 2021-02-10T11:30:01.563 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:01.561 7f07003b7700 1 --2- 172.21.2.102:0/3360518603 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f06f80af660 0x7f06f80afa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:01.563 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:01.561 7f07003b7700 1 --2- 172.21.2.102:0/3360518603 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f06f80c39e0 0x7f06f80af040 unknown :-1 s=CLOSED pgs=1919 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:01.563 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:01.561 7f07003b7700 1 --2- 172.21.2.102:0/3360518603 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f06e00186c0 0x7f06e001abc0 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:01.563 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:01.561 7f07003b7700 1 --2- 172.21.2.102:0/3360518603 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f06f8092010 0x7f06f80c33c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:01.563 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:01.561 7f07003b7700 1 -- 172.21.2.102:0/3360518603 >> 172.21.2.102:0/3360518603 conn(0x7f06f80a0030 msgr2=0x7f06f80f22b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:01.563 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:01.562 7f07003b7700 1 -- 172.21.2.102:0/3360518603 shutdown_connections 2021-02-10T11:30:01.563 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:01.562 7f07003b7700 1 -- 172.21.2.102:0/3360518603 wait complete. 2021-02-10T11:30:01.570 INFO:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2021-02-10T11:30:01.571 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool create cephfs_metadata 8 2021-02-10T11:30:02.006 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:01.988 7eff0f462700 1 Processor -- start 2021-02-10T11:30:02.006 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:01.988 7eff0f462700 1 -- start start 2021-02-10T11:30:02.006 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:01.988 7eff0f462700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7eff080d1860 0x7eff081b04b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:02.006 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:01.988 7eff0f462700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7eff081b0ad0 0x7eff08108930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:02.006 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:01.988 7eff0f462700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff080fe1b0 con 0x7eff080d1c60 2021-02-10T11:30:02.007 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:01.988 7eff0f462700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff08109a50 con 0x7eff080d1860 2021-02-10T11:30:02.007 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:01.988 7eff0f462700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff08109740 con 0x7eff081b0ad0 2021-02-10T11:30:02.007 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:01.989 7eff0d27e700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7eff080d1860 0x7eff081b04b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:02.007 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:01.989 7eff0d27e700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7eff080d1860 0x7eff081b04b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.113:3300/0 says I am v2:172.21.2.102:35824/0 (socket says 172.21.2.102:35824) 2021-02-10T11:30:02.007 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:01.989 7eff0d27e700 1 -- 172.21.2.102:0/721128517 learned_addr learned my addr 172.21.2.102:0/721128517 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:30:02.007 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:01.989 7eff0ca7d700 1 --2- 172.21.2.102:0/721128517 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7eff081b0ad0 0x7eff08108930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:02.007 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:01.990 7eff0d27e700 1 -- 172.21.2.102:0/721128517 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7eff081b0ad0 msgr2=0x7eff08108930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:02.007 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:01.990 7eff0d27e700 1 --2- 172.21.2.102:0/721128517 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7eff081b0ad0 0x7eff08108930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:02.007 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:01.990 7eff0d27e700 1 -- 172.21.2.102:0/721128517 >> v1:172.21.2.113:6789/0 conn(0x7eff080d1c60 legacy=0x7eff08108f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:02.007 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:01.990 7eff0d27e700 1 -- 172.21.2.102:0/721128517 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff08088070 con 0x7eff080d1860 2021-02-10T11:30:02.007 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:01.990 7eff0d27e700 1 --2- 172.21.2.102:0/721128517 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7eff080d1860 0x7eff081b04b0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 rx=0x7eff04006700 tx=0x7eff04009b80).ready entity=mon.1 client_cookie=3984e31aa0a1351b server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:02.008 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:01.990 7efefbfff700 1 -- 172.21.2.102:0/721128517 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7eff0400c0c0 con 0x7eff080d1860 2021-02-10T11:30:02.008 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:01.990 7efefbfff700 1 -- 172.21.2.102:0/721128517 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff0400c3c0 con 0x7eff080d1860 2021-02-10T11:30:02.008 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:01.990 7efefbfff700 1 -- 172.21.2.102:0/721128517 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7eff0400c900 con 0x7eff080d1860 2021-02-10T11:30:02.008 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:01.990 7eff0f462700 1 -- 172.21.2.102:0/721128517 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7eff080d1860 msgr2=0x7eff081b04b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:02.008 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:01.990 7eff0f462700 1 --2- 172.21.2.102:0/721128517 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7eff080d1860 0x7eff081b04b0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 rx=0x7eff04006700 tx=0x7eff04009b80).stop 2021-02-10T11:30:02.008 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:01.991 7eff0f462700 1 -- 172.21.2.102:0/721128517 shutdown_connections 2021-02-10T11:30:02.008 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:01.991 7eff0f462700 1 --2- 172.21.2.102:0/721128517 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7eff081b0ad0 0x7eff08108930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:02.008 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:01.991 7eff0f462700 1 --2- 172.21.2.102:0/721128517 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7eff080d1860 0x7eff081b04b0 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:02.008 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:01.991 7eff0f462700 1 -- 172.21.2.102:0/721128517 >> 172.21.2.102:0/721128517 conn(0x7eff080ce410 msgr2=0x7eff080d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:02.008 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:01.991 7eff0f462700 1 -- 172.21.2.102:0/721128517 shutdown_connections 2021-02-10T11:30:02.008 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:01.991 7eff0f462700 1 -- 172.21.2.102:0/721128517 wait complete. 2021-02-10T11:30:02.008 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:02.005 7eff0f462700 1 Processor -- start 2021-02-10T11:30:02.008 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:02.006 7eff0f462700 1 -- start start 2021-02-10T11:30:02.010 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:02.006 7eff0f462700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7eff080c2a30 0x7eff080c2e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:02.010 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:02.006 7eff0f462700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7eff080a7950 0x7eff080a7d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:02.010 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:02.006 7eff0f462700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7eff081a71f0 0x7eff081a75f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:02.010 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:02.006 7eff0f462700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff080a8440 con 0x7eff080a7950 2021-02-10T11:30:02.010 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:02.006 7eff0f462700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff080c3790 con 0x7eff081a71f0 2021-02-10T11:30:02.010 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:02.006 7eff0f462700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff081a7c10 con 0x7eff080c2a30 2021-02-10T11:30:02.011 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:02.006 7eff0ca7d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7eff080a7950 0x7eff080a7d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:02.011 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:02.006 7eff0ca7d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7eff080a7950 0x7eff080a7d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:60636/0 (socket says 172.21.2.102:60636) 2021-02-10T11:30:02.011 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:02.006 7eff0ca7d700 1 -- 172.21.2.102:0/3455177305 learned_addr learned my addr 172.21.2.102:0/3455177305 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:30:02.011 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:02.006 7eff0da7f700 1 --2- 172.21.2.102:0/3455177305 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7eff081a71f0 0x7eff081a75f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:02.011 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:02.006 7eff0d27e700 1 --2- 172.21.2.102:0/3455177305 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7eff080c2a30 0x7eff080c2e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:02.011 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:02.007 7eff0ca7d700 1 -- 172.21.2.102:0/3455177305 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7eff080c2a30 msgr2=0x7eff080c2e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:02.011 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:02.007 7eff0ca7d700 1 --2- 172.21.2.102:0/3455177305 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7eff080c2a30 0x7eff080c2e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:02.011 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:02.007 7eff0ca7d700 1 -- 172.21.2.102:0/3455177305 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7eff081a71f0 msgr2=0x7eff081a75f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:02.011 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:02.007 7eff0ca7d700 1 --2- 172.21.2.102:0/3455177305 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7eff081a71f0 0x7eff081a75f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:02.011 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:02.007 7eff0ca7d700 1 -- 172.21.2.102:0/3455177305 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff081a7e30 con 0x7eff080a7950 2021-02-10T11:30:02.011 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:02.007 7eff0ca7d700 1 --2- 172.21.2.102:0/3455177305 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7eff080a7950 0x7eff080a7d50 secure :-1 s=READY pgs=1920 cs=0 l=1 rev1=1 rx=0x7efefc006f60 tx=0x7efefc00b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:02.011 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:02.007 7efef9ffb700 1 -- 172.21.2.102:0/3455177305 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efefc0009b0 con 0x7eff080a7950 2021-02-10T11:30:02.011 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:02.007 7efef9ffb700 1 -- 172.21.2.102:0/3455177305 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efefc00bd60 con 0x7eff080a7950 2021-02-10T11:30:02.012 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:02.007 7eff0f462700 1 -- 172.21.2.102:0/3455177305 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff081a7e30 con 0x7eff080a7950 2021-02-10T11:30:02.012 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:02.007 7efef9ffb700 1 -- 172.21.2.102:0/3455177305 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efefc002030 con 0x7eff080a7950 2021-02-10T11:30:02.012 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:02.007 7eff0f462700 1 -- 172.21.2.102:0/3455177305 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff080a8f80 con 0x7eff080a7950 2021-02-10T11:30:02.012 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:02.008 7efef9ffb700 1 -- 172.21.2.102:0/3455177305 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7efefc0052e0 con 0x7eff080a7950 2021-02-10T11:30:02.012 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:02.008 7efef9ffb700 1 --2- 172.21.2.102:0/3455177305 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7efee80186c0 0x7efee801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:02.012 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:02.008 7efef9ffb700 1 -- 172.21.2.102:0/3455177305 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 4753+0+0 (secure 0 0 0) 0x7efefc032110 con 0x7eff080a7950 2021-02-10T11:30:02.012 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:02.008 7eff0d27e700 1 --2- 172.21.2.102:0/3455177305 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7efee80186c0 0x7efee801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:02.012 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:02.008 7eff0d27e700 1 --2- 172.21.2.102:0/3455177305 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7efee80186c0 0x7efee801abc0 crc :-1 s=READY pgs=860 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:02.038 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:02.038 7eff0f462700 1 -- 172.21.2.102:0/3455177305 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff08082a20 con 0x7eff080a7950 2021-02-10T11:30:02.041 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:02.040 7efef9ffb700 1 -- 172.21.2.102:0/3455177305 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7efefc02df20 con 0x7eff080a7950 2021-02-10T11:30:02.169 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:02.168 7eff0f462700 1 -- 172.21.2.102:0/3455177305 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"} v 0) v1 -- 0x7eff08082a20 con 0x7eff080a7950 2021-02-10T11:30:02.580 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:02.579 7efef9ffb700 1 -- 172.21.2.102:0/3455177305 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"}]=0 pool 'cephfs_metadata' created v90) v1 ==== 133+0+0 (secure 0 0 0) 0x7efefc02df20 con 0x7eff080a7950 2021-02-10T11:30:02.584 INFO:teuthology.orchestra.run.gibba002.stderr:pool 'cephfs_metadata' created 2021-02-10T11:30:02.586 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:02.585 7eff0f462700 1 -- 172.21.2.102:0/3455177305 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7efee80186c0 msgr2=0x7efee801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:02.586 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:02.585 7eff0f462700 1 --2- 172.21.2.102:0/3455177305 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7efee80186c0 0x7efee801abc0 crc :-1 s=READY pgs=860 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:02.586 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:02.585 7eff0f462700 1 -- 172.21.2.102:0/3455177305 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7eff080a7950 msgr2=0x7eff080a7d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:02.586 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:02.585 7eff0f462700 1 --2- 172.21.2.102:0/3455177305 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7eff080a7950 0x7eff080a7d50 secure :-1 s=READY pgs=1920 cs=0 l=1 rev1=1 rx=0x7efefc006f60 tx=0x7efefc00b970).stop 2021-02-10T11:30:02.586 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:02.586 7eff0f462700 1 -- 172.21.2.102:0/3455177305 shutdown_connections 2021-02-10T11:30:02.586 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:02.586 7eff0f462700 1 --2- 172.21.2.102:0/3455177305 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7efee80186c0 0x7efee801abc0 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:02.587 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:02.586 7eff0f462700 1 --2- 172.21.2.102:0/3455177305 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7eff081a71f0 0x7eff081a75f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:02.587 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:02.586 7eff0f462700 1 --2- 172.21.2.102:0/3455177305 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7eff080a7950 0x7eff080a7d50 unknown :-1 s=CLOSED pgs=1920 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:02.587 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:02.586 7eff0f462700 1 --2- 172.21.2.102:0/3455177305 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7eff080c2a30 0x7eff080c2e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:02.587 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:02.586 7eff0f462700 1 -- 172.21.2.102:0/3455177305 >> 172.21.2.102:0/3455177305 conn(0x7eff080b5f10 msgr2=0x7eff0809f200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:02.587 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:02.586 7eff0f462700 1 -- 172.21.2.102:0/3455177305 shutdown_connections 2021-02-10T11:30:02.587 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:02.586 7eff0f462700 1 -- 172.21.2.102:0/3455177305 wait complete. 2021-02-10T11:30:02.595 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool create cephfs_data 8 2021-02-10T11:30:03.032 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:03.015 7f7a06c7c700 1 Processor -- start 2021-02-10T11:30:03.033 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:03.015 7f7a06c7c700 1 -- start start 2021-02-10T11:30:03.033 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:03.015 7f7a06c7c700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f7a000d1c70 0x7f7a000cef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:03.033 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:03.015 7f7a06c7c700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a000d0170 con 0x7f7a000cf580 2021-02-10T11:30:03.033 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:03.015 7f7a06c7c700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a000fe1c0 con 0x7f7a000d1870 2021-02-10T11:30:03.033 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:03.015 7f7a06c7c700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a001b0cb0 con 0x7f7a000d1c70 2021-02-10T11:30:03.033 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:03.016 7f7a05299700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f7a000cf580 0x7f7a000cf980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:52868/0 (socket says 172.21.2.102:52868) 2021-02-10T11:30:03.033 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:03.016 7f7a05299700 1 -- 172.21.2.102:0/1872163543 learned_addr learned my addr 172.21.2.102:0/1872163543 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:30:03.033 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:03.016 7f79fffff700 1 --2- 172.21.2.102:0/1872163543 >> v2:172.21.2.102:3300/0 conn(0x7f7a000d1c70 0x7f7a000cef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:03.033 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:03.016 7f79fffff700 1 -- 172.21.2.102:0/1872163543 >> v1:172.21.2.113:6789/0 conn(0x7f7a000d1870 legacy=0x7f7a001b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:03.034 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:03.016 7f79fffff700 1 -- 172.21.2.102:0/1872163543 >> v1:172.21.2.102:6789/0 conn(0x7f7a000cf580 legacy=0x7f7a000cf980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:03.034 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:03.016 7f79fffff700 1 -- 172.21.2.102:0/1872163543 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a00109a60 con 0x7f7a000d1c70 2021-02-10T11:30:03.034 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:03.016 7f79fffff700 1 --2- 172.21.2.102:0/1872163543 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7a000d1c70 0x7f7a000cef60 secure :-1 s=READY pgs=1922 cs=0 l=1 rev1=1 rx=0x7f79f0006650 tx=0x7f79f0009a80).ready entity=mon.0 client_cookie=da5505659e5a31ee server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:03.034 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:03.016 7f79ff7fe700 1 -- 172.21.2.102:0/1872163543 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f79f00009b0 con 0x7f7a000d1c70 2021-02-10T11:30:03.034 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:03.016 7f79ff7fe700 1 -- 172.21.2.102:0/1872163543 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79f000c0c0 con 0x7f7a000d1c70 2021-02-10T11:30:03.034 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:03.016 7f79ff7fe700 1 -- 172.21.2.102:0/1872163543 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f79f000c520 con 0x7f7a000d1c70 2021-02-10T11:30:03.034 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:03.016 7f7a06c7c700 1 -- 172.21.2.102:0/1872163543 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7a000d1c70 msgr2=0x7f7a000cef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:03.034 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:03.016 7f7a06c7c700 1 --2- 172.21.2.102:0/1872163543 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7a000d1c70 0x7f7a000cef60 secure :-1 s=READY pgs=1922 cs=0 l=1 rev1=1 rx=0x7f79f0006650 tx=0x7f79f0009a80).stop 2021-02-10T11:30:03.034 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:03.017 7f7a06c7c700 1 -- 172.21.2.102:0/1872163543 shutdown_connections 2021-02-10T11:30:03.034 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:03.017 7f7a06c7c700 1 --2- 172.21.2.102:0/1872163543 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7a000d1c70 0x7f7a000cef60 unknown :-1 s=CLOSED pgs=1922 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:03.034 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:03.017 7f7a06c7c700 1 -- 172.21.2.102:0/1872163543 >> 172.21.2.102:0/1872163543 conn(0x7f7a000ce420 msgr2=0x7f7a000d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:03.035 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:03.017 7f7a06c7c700 1 -- 172.21.2.102:0/1872163543 shutdown_connections 2021-02-10T11:30:03.035 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:03.017 7f7a06c7c700 1 -- 172.21.2.102:0/1872163543 wait complete. 2021-02-10T11:30:03.035 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:03.032 7f7a06c7c700 1 Processor -- start 2021-02-10T11:30:03.035 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:03.032 7f7a06c7c700 1 -- start start 2021-02-10T11:30:03.036 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:03.032 7f7a06c7c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7a000a25d0 0x7f7a000a29d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:03.036 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:03.032 7f7a06c7c700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7a000a2ff0 0x7f7a000a33f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:03.036 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:03.033 7f7a06c7c700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7a000b5650 0x7f7a000b5a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:03.036 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:03.033 7f7a06c7c700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a001c3780 con 0x7f7a000a25d0 2021-02-10T11:30:03.036 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:03.033 7f7a06c7c700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a000a20c0 con 0x7f7a000b5650 2021-02-10T11:30:03.036 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:03.033 7f7a06c7c700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a000b60a0 con 0x7f7a000a2ff0 2021-02-10T11:30:03.036 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:03.033 7f7a04a98700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7a000a25d0 0x7f7a000a29d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:03.037 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:03.033 7f7a04a98700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7a000a25d0 0x7f7a000a29d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:60648/0 (socket says 172.21.2.102:60648) 2021-02-10T11:30:03.037 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:03.033 7f7a04a98700 1 -- 172.21.2.102:0/3385890256 learned_addr learned my addr 172.21.2.102:0/3385890256 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:30:03.037 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:03.033 7f7a05299700 1 --2- 172.21.2.102:0/3385890256 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7a000b5650 0x7f7a000b5a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:03.037 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:03.033 7f79fffff700 1 --2- 172.21.2.102:0/3385890256 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7a000a2ff0 0x7f7a000a33f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:03.037 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:03.033 7f7a04a98700 1 -- 172.21.2.102:0/3385890256 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7a000a2ff0 msgr2=0x7f7a000a33f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:03.037 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:03.033 7f7a04a98700 1 --2- 172.21.2.102:0/3385890256 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7a000a2ff0 0x7f7a000a33f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:03.037 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:03.033 7f7a04a98700 1 -- 172.21.2.102:0/3385890256 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7a000b5650 msgr2=0x7f7a000b5a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:03.037 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:03.033 7f7a04a98700 1 --2- 172.21.2.102:0/3385890256 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7a000b5650 0x7f7a000b5a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:03.037 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:03.033 7f7a04a98700 1 -- 172.21.2.102:0/3385890256 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a000b62c0 con 0x7f7a000a25d0 2021-02-10T11:30:03.037 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:03.033 7f7a04a98700 1 --2- 172.21.2.102:0/3385890256 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7a000a25d0 0x7f7a000a29d0 secure :-1 s=READY pgs=1923 cs=0 l=1 rev1=1 rx=0x7f79e800b5b0 tx=0x7f79e800b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:03.037 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:03.034 7f79fd7fa700 1 -- 172.21.2.102:0/3385890256 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f79e800bd40 con 0x7f7a000a25d0 2021-02-10T11:30:03.037 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:03.034 7f79fd7fa700 1 -- 172.21.2.102:0/3385890256 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79e8006240 con 0x7f7a000a25d0 2021-02-10T11:30:03.037 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:03.034 7f79fd7fa700 1 -- 172.21.2.102:0/3385890256 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f79e8005720 con 0x7f7a000a25d0 2021-02-10T11:30:03.038 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:03.034 7f7a06c7c700 1 -- 172.21.2.102:0/3385890256 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a000b62c0 con 0x7f7a000a25d0 2021-02-10T11:30:03.038 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:03.034 7f7a06c7c700 1 -- 172.21.2.102:0/3385890256 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a000b6850 con 0x7f7a000a25d0 2021-02-10T11:30:03.038 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:03.034 7f79fd7fa700 1 -- 172.21.2.102:0/3385890256 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f79e80054a0 con 0x7f7a000a25d0 2021-02-10T11:30:03.038 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:03.034 7f79fd7fa700 1 --2- 172.21.2.102:0/3385890256 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f79e40186c0 0x7f79e401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:03.038 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:03.034 7f79fd7fa700 1 -- 172.21.2.102:0/3385890256 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 5107+0+0 (secure 0 0 0) 0x7f79e80323c0 con 0x7f7a000a25d0 2021-02-10T11:30:03.038 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:03.034 7f79fffff700 1 --2- 172.21.2.102:0/3385890256 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f79e40186c0 0x7f79e401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:03.038 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:03.035 7f79fffff700 1 --2- 172.21.2.102:0/3385890256 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f79e40186c0 0x7f79e401abc0 crc :-1 s=READY pgs=861 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:03.065 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:03.065 7f7a06c7c700 1 -- 172.21.2.102:0/3385890256 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a000807d0 con 0x7f7a000a25d0 2021-02-10T11:30:03.068 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:03.068 7f79fd7fa700 1 -- 172.21.2.102:0/3385890256 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f79e8000c60 con 0x7f7a000a25d0 2021-02-10T11:30:03.201 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:03.200 7f7a06c7c700 1 -- 172.21.2.102:0/3385890256 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_data"} v 0) v1 -- 0x7f7a00088070 con 0x7f7a000a25d0 2021-02-10T11:30:03.583 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:03.583 7f79fd7fa700 1 -- 172.21.2.102:0/3385890256 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_data"}]=0 pool 'cephfs_data' created v91) v1 ==== 125+0+0 (secure 0 0 0) 0x7f79e80054a0 con 0x7f7a000a25d0 2021-02-10T11:30:03.616 INFO:teuthology.orchestra.run.gibba002.stderr:pool 'cephfs_data' created 2021-02-10T11:30:03.618 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:03.618 7f7a06c7c700 1 -- 172.21.2.102:0/3385890256 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f79e40186c0 msgr2=0x7f79e401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:03.618 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:03.618 7f7a06c7c700 1 --2- 172.21.2.102:0/3385890256 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f79e40186c0 0x7f79e401abc0 crc :-1 s=READY pgs=861 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:03.618 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:03.618 7f7a06c7c700 1 -- 172.21.2.102:0/3385890256 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7a000a25d0 msgr2=0x7f7a000a29d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:03.618 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:03.618 7f7a06c7c700 1 --2- 172.21.2.102:0/3385890256 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7a000a25d0 0x7f7a000a29d0 secure :-1 s=READY pgs=1923 cs=0 l=1 rev1=1 rx=0x7f79e800b5b0 tx=0x7f79e800b950).stop 2021-02-10T11:30:03.619 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:03.618 7f7a06c7c700 1 -- 172.21.2.102:0/3385890256 shutdown_connections 2021-02-10T11:30:03.619 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:03.618 7f7a06c7c700 1 --2- 172.21.2.102:0/3385890256 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f79e40186c0 0x7f79e401abc0 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:03.619 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:03.618 7f7a06c7c700 1 --2- 172.21.2.102:0/3385890256 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7a000b5650 0x7f7a000b5a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:03.619 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:03.618 7f7a06c7c700 1 --2- 172.21.2.102:0/3385890256 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7a000a2ff0 0x7f7a000a33f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:03.619 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:03.618 7f7a06c7c700 1 --2- 172.21.2.102:0/3385890256 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7a000a25d0 0x7f7a000a29d0 unknown :-1 s=CLOSED pgs=1923 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:03.619 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:03.618 7f7a06c7c700 1 -- 172.21.2.102:0/3385890256 >> 172.21.2.102:0/3385890256 conn(0x7f7a000c4120 msgr2=0x7f7a000ffd60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:03.619 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:03.618 7f7a06c7c700 1 -- 172.21.2.102:0/3385890256 shutdown_connections 2021-02-10T11:30:03.620 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:03.618 7f7a06c7c700 1 -- 172.21.2.102:0/3385890256 wait complete. 2021-02-10T11:30:03.627 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2021-02-10T11:30:04.060 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:04.043 7fc8b71e2700 1 Processor -- start 2021-02-10T11:30:04.060 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:04.043 7fc8b71e2700 1 -- start start 2021-02-10T11:30:04.060 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:04.043 7fc8b71e2700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fc8b00d1880 0x7fc8b01b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:04.060 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:04.043 7fc8b71e2700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8b00fe1d0 con 0x7fc8b01b0af0 2021-02-10T11:30:04.060 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:04.043 7fc8b71e2700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8b0109a70 con 0x7fc8b00cff80 2021-02-10T11:30:04.060 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:04.043 7fc8b71e2700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8b0088070 con 0x7fc8b00d1880 2021-02-10T11:30:04.060 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:04.043 7fc8affff700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fc8b01b0af0 0x7fc8b0108950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:52880/0 (socket says 172.21.2.102:52880) 2021-02-10T11:30:04.060 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:04.043 7fc8affff700 1 -- 172.21.2.102:0/4077247097 learned_addr learned my addr 172.21.2.102:0/4077247097 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:30:04.061 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:04.043 7fc8b4ffe700 1 --2- 172.21.2.102:0/4077247097 >> v2:172.21.2.102:3300/0 conn(0x7fc8b00d1880 0x7fc8b01b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:04.061 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:04.044 7fc8af7fe700 1 -- 172.21.2.102:0/4077247097 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2517669838 0 0) 0x7fc8a0000d30 con 0x7fc8b01b0af0 2021-02-10T11:30:04.061 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:04.044 7fc8af7fe700 1 -- 172.21.2.102:0/4077247097 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc89c002fe0 con 0x7fc8b01b0af0 2021-02-10T11:30:04.061 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:04.044 7fc8b4ffe700 1 -- 172.21.2.102:0/4077247097 >> v1:172.21.2.113:6789/0 conn(0x7fc8b00cff80 legacy=0x7fc8b0109140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:04.061 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:04.044 7fc8b4ffe700 1 -- 172.21.2.102:0/4077247097 >> v1:172.21.2.102:6789/0 conn(0x7fc8b01b0af0 legacy=0x7fc8b0108950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:04.061 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:04.044 7fc8b4ffe700 1 -- 172.21.2.102:0/4077247097 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8b0088260 con 0x7fc8b00d1880 2021-02-10T11:30:04.061 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:04.044 7fc8b4ffe700 1 --2- 172.21.2.102:0/4077247097 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc8b00d1880 0x7fc8b01b04d0 secure :-1 s=READY pgs=1925 cs=0 l=1 rev1=1 rx=0x7fc898006650 tx=0x7fc898009a80).ready entity=mon.0 client_cookie=dd5a2d12a1253409 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:04.061 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:04.044 7fc8af7fe700 1 -- 172.21.2.102:0/4077247097 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc89800c0c0 con 0x7fc8b00d1880 2021-02-10T11:30:04.061 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:04.044 7fc8af7fe700 1 -- 172.21.2.102:0/4077247097 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc89800c340 con 0x7fc8b00d1880 2021-02-10T11:30:04.061 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:04.044 7fc8af7fe700 1 -- 172.21.2.102:0/4077247097 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc89800c7a0 con 0x7fc8b00d1880 2021-02-10T11:30:04.061 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:04.044 7fc8b71e2700 1 -- 172.21.2.102:0/4077247097 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc8b00d1880 msgr2=0x7fc8b01b04d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:04.061 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:04.044 7fc8b71e2700 1 --2- 172.21.2.102:0/4077247097 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc8b00d1880 0x7fc8b01b04d0 secure :-1 s=READY pgs=1925 cs=0 l=1 rev1=1 rx=0x7fc898006650 tx=0x7fc898009a80).stop 2021-02-10T11:30:04.062 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:04.045 7fc8b71e2700 1 -- 172.21.2.102:0/4077247097 shutdown_connections 2021-02-10T11:30:04.062 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:04.045 7fc8b71e2700 1 --2- 172.21.2.102:0/4077247097 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc8b00d1880 0x7fc8b01b04d0 unknown :-1 s=CLOSED pgs=1925 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:04.062 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:04.045 7fc8b71e2700 1 -- 172.21.2.102:0/4077247097 >> 172.21.2.102:0/4077247097 conn(0x7fc8b00ce430 msgr2=0x7fc8b00d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:04.062 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:04.045 7fc8b71e2700 1 -- 172.21.2.102:0/4077247097 shutdown_connections 2021-02-10T11:30:04.062 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:04.045 7fc8b71e2700 1 -- 172.21.2.102:0/4077247097 wait complete. 2021-02-10T11:30:04.064 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:04.060 7fc8b71e2700 1 Processor -- start 2021-02-10T11:30:04.064 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:04.060 7fc8b71e2700 1 -- start start 2021-02-10T11:30:04.064 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:04.060 7fc8b71e2700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc8b01add90 0x7fc8b01b0630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:04.064 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:04.060 7fc8b71e2700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc8b019b180 0x7fc8b00b7aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:04.064 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:04.060 7fc8b71e2700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc8b019b580 0x7fc8b00b80c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:04.064 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:04.060 7fc8b71e2700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8b01ae190 con 0x7fc8b019b180 2021-02-10T11:30:04.064 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:04.060 7fc8b71e2700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8b00b86e0 con 0x7fc8b019b580 2021-02-10T11:30:04.064 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:04.060 7fc8b71e2700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8b00b8900 con 0x7fc8b01add90 2021-02-10T11:30:04.064 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:04.061 7fc8affff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc8b019b180 0x7fc8b00b7aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:04.064 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:04.061 7fc8affff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc8b019b180 0x7fc8b00b7aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:60660/0 (socket says 172.21.2.102:60660) 2021-02-10T11:30:04.065 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:04.061 7fc8affff700 1 -- 172.21.2.102:0/2808401258 learned_addr learned my addr 172.21.2.102:0/2808401258 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:30:04.065 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:04.061 7fc8b57ff700 1 --2- 172.21.2.102:0/2808401258 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc8b019b580 0x7fc8b00b80c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:04.065 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:04.061 7fc8b4ffe700 1 --2- 172.21.2.102:0/2808401258 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc8b01add90 0x7fc8b01b0630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:04.065 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:04.061 7fc8affff700 1 -- 172.21.2.102:0/2808401258 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc8b01add90 msgr2=0x7fc8b01b0630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:04.065 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:04.061 7fc8affff700 1 --2- 172.21.2.102:0/2808401258 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc8b01add90 0x7fc8b01b0630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:04.065 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:04.061 7fc8affff700 1 -- 172.21.2.102:0/2808401258 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc8b019b580 msgr2=0x7fc8b00b80c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:04.065 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:04.061 7fc8affff700 1 --2- 172.21.2.102:0/2808401258 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc8b019b580 0x7fc8b00b80c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:04.065 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:04.061 7fc8affff700 1 -- 172.21.2.102:0/2808401258 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8b019c040 con 0x7fc8b019b180 2021-02-10T11:30:04.065 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:04.061 7fc8b57ff700 1 --2- 172.21.2.102:0/2808401258 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc8b019b580 0x7fc8b00b80c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:30:04.065 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:04.061 7fc8b4ffe700 1 --2- 172.21.2.102:0/2808401258 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc8b01add90 0x7fc8b01b0630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:30:04.065 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:04.061 7fc8affff700 1 --2- 172.21.2.102:0/2808401258 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc8b019b180 0x7fc8b00b7aa0 secure :-1 s=READY pgs=1926 cs=0 l=1 rev1=1 rx=0x7fc8a000a920 tx=0x7fc8a000acc0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:04.065 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:04.061 7fc8ad7fa700 1 -- 172.21.2.102:0/2808401258 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc8a00070d0 con 0x7fc8b019b180 2021-02-10T11:30:04.065 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:04.061 7fc8ad7fa700 1 -- 172.21.2.102:0/2808401258 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8a00073d0 con 0x7fc8b019b180 2021-02-10T11:30:04.065 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:04.061 7fc8b71e2700 1 -- 172.21.2.102:0/2808401258 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8b019c040 con 0x7fc8b019b180 2021-02-10T11:30:04.066 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:04.061 7fc8ad7fa700 1 -- 172.21.2.102:0/2808401258 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc8a0004740 con 0x7fc8b019b180 2021-02-10T11:30:04.066 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:04.062 7fc8b71e2700 1 -- 172.21.2.102:0/2808401258 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8b019c040 con 0x7fc8b019b180 2021-02-10T11:30:04.066 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:04.062 7fc8ad7fa700 1 -- 172.21.2.102:0/2808401258 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fc8a00044c0 con 0x7fc8b019b180 2021-02-10T11:30:04.066 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:04.062 7fc8ad7fa700 1 --2- 172.21.2.102:0/2808401258 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc8940186c0 0x7fc89401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:04.066 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:04.062 7fc8ad7fa700 1 -- 172.21.2.102:0/2808401258 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 5457+0+0 (secure 0 0 0) 0x7fc8a0032ad0 con 0x7fc8b019b180 2021-02-10T11:30:04.066 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:04.062 7fc8b4ffe700 1 --2- 172.21.2.102:0/2808401258 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc8940186c0 0x7fc89401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:04.066 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:04.063 7fc8b4ffe700 1 --2- 172.21.2.102:0/2808401258 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc8940186c0 0x7fc89401abc0 crc :-1 s=READY pgs=864 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:04.094 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:04.093 7fc8b71e2700 1 -- 172.21.2.102:0/2808401258 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8b00fe1d0 con 0x7fc8b019b180 2021-02-10T11:30:04.098 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:04.096 7fc8ad7fa700 1 -- 172.21.2.102:0/2808401258 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fc8a00073b0 con 0x7fc8b019b180 2021-02-10T11:30:04.224 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:04.224 7fc8b71e2700 1 -- 172.21.2.102:0/2808401258 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"data": "cephfs_data", "prefix": "fs new", "force": true, "fs_name": "cephfs", "metadata": "cephfs_metadata"} v 0) v1 -- 0x7fc8b00fe1d0 con 0x7fc8b019b180 2021-02-10T11:30:04.229 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:04.228 7fc8ad7fa700 1 -- 172.21.2.102:0/2808401258 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"data": "cephfs_data", "prefix": "fs new", "force": true, "fs_name": "cephfs", "metadata": "cephfs_metadata"}]=0 new fs with metadata pool 18 and data pool 19 v70) v1 ==== 189+0+0 (secure 0 0 0) 0x7fc8a00073b0 con 0x7fc8b019b180 2021-02-10T11:30:04.234 INFO:teuthology.orchestra.run.gibba002.stderr:new fs with metadata pool 18 and data pool 19 2021-02-10T11:30:04.234 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:04.233 7fc8b71e2700 1 -- 172.21.2.102:0/2808401258 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc8940186c0 msgr2=0x7fc89401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:04.234 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:04.233 7fc8b71e2700 1 --2- 172.21.2.102:0/2808401258 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc8940186c0 0x7fc89401abc0 crc :-1 s=READY pgs=864 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:04.234 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:04.233 7fc8b71e2700 1 -- 172.21.2.102:0/2808401258 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc8b019b180 msgr2=0x7fc8b00b7aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:04.234 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:04.233 7fc8b71e2700 1 --2- 172.21.2.102:0/2808401258 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc8b019b180 0x7fc8b00b7aa0 secure :-1 s=READY pgs=1926 cs=0 l=1 rev1=1 rx=0x7fc8a000a920 tx=0x7fc8a000acc0).stop 2021-02-10T11:30:04.234 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:04.233 7fc8b71e2700 1 -- 172.21.2.102:0/2808401258 shutdown_connections 2021-02-10T11:30:04.234 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:04.233 7fc8b71e2700 1 --2- 172.21.2.102:0/2808401258 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc8940186c0 0x7fc89401abc0 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:04.234 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:04.233 7fc8b71e2700 1 --2- 172.21.2.102:0/2808401258 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc8b019b580 0x7fc8b00b80c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:04.235 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:04.233 7fc8b71e2700 1 --2- 172.21.2.102:0/2808401258 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc8b019b180 0x7fc8b00b7aa0 secure :-1 s=CLOSED pgs=1926 cs=0 l=1 rev1=1 rx=0x7fc8a000a920 tx=0x7fc8a000acc0).stop 2021-02-10T11:30:04.235 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:04.233 7fc8b71e2700 1 --2- 172.21.2.102:0/2808401258 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc8b01add90 0x7fc8b01b0630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:04.235 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:04.233 7fc8b71e2700 1 -- 172.21.2.102:0/2808401258 >> 172.21.2.102:0/2808401258 conn(0x7fc8b019f940 msgr2=0x7fc8b01ab890 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:04.236 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:04.236 7fc8b71e2700 1 -- 172.21.2.102:0/2808401258 shutdown_connections 2021-02-10T11:30:04.236 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:04.236 7fc8b71e2700 1 -- 172.21.2.102:0/2808401258 wait complete. 2021-02-10T11:30:04.247 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:30:04.676 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:04.658 7f3d1484c700 1 Processor -- start 2021-02-10T11:30:04.676 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:04.659 7f3d1484c700 1 -- start start 2021-02-10T11:30:04.676 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:04.659 7f3d1484c700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f3d100d0340 0x7f3d100ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:04.676 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:04.659 7f3d1484c700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d100f0b10 con 0x7f3d100d1870 2021-02-10T11:30:04.676 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:04.659 7f3d1484c700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d101b0be0 con 0x7f3d100d1c70 2021-02-10T11:30:04.677 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:04.659 7f3d1484c700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d100e5ed0 con 0x7f3d100d0340 2021-02-10T11:30:04.677 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:04.659 7f3d0de1b700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f3d100d0340 0x7f3d100ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:04.677 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:04.659 7f3d0e61c700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f3d100d1870 0x7f3d101b03f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:52890/0 (socket says 172.21.2.102:52890) 2021-02-10T11:30:04.677 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:04.659 7f3d0e61c700 1 -- 172.21.2.102:0/911402127 learned_addr learned my addr 172.21.2.102:0/911402127 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:30:04.677 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:04.660 7f3d0de1b700 1 -- 172.21.2.102:0/911402127 >> v1:172.21.2.113:6789/0 conn(0x7f3d100d1c70 legacy=0x7f3d100efc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:04.677 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:04.660 7f3d0d61a700 1 -- 172.21.2.102:0/911402127 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2182319302 0 0) 0x7f3d00001180 con 0x7f3d100d1870 2021-02-10T11:30:04.677 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:04.660 7f3d0de1b700 1 -- 172.21.2.102:0/911402127 >> v1:172.21.2.102:6789/0 conn(0x7f3d100d1870 legacy=0x7f3d101b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:04.677 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:04.660 7f3d0de1b700 1 -- 172.21.2.102:0/911402127 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d100f0450 con 0x7f3d100d0340 2021-02-10T11:30:04.677 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:04.660 7f3d0de1b700 1 --2- 172.21.2.102:0/911402127 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3d100d0340 0x7f3d100ef640 secure :-1 s=READY pgs=1928 cs=0 l=1 rev1=1 rx=0x7f3d04006650 tx=0x7f3d04009a80).ready entity=mon.0 client_cookie=86f4f3f0dba4eafe server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:04.677 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:04.660 7f3d0d61a700 1 -- 172.21.2.102:0/911402127 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3d040009b0 con 0x7f3d100d0340 2021-02-10T11:30:04.677 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:04.660 7f3d0d61a700 1 -- 172.21.2.102:0/911402127 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d0400c0c0 con 0x7f3d100d0340 2021-02-10T11:30:04.677 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:04.660 7f3d0d61a700 1 -- 172.21.2.102:0/911402127 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3d0400c5e0 con 0x7f3d100d0340 2021-02-10T11:30:04.677 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:04.661 7f3d1484c700 1 -- 172.21.2.102:0/911402127 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3d100d0340 msgr2=0x7f3d100ef640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:04.678 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:04.661 7f3d1484c700 1 --2- 172.21.2.102:0/911402127 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3d100d0340 0x7f3d100ef640 secure :-1 s=READY pgs=1928 cs=0 l=1 rev1=1 rx=0x7f3d04006650 tx=0x7f3d04009a80).stop 2021-02-10T11:30:04.678 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:04.661 7f3d1484c700 1 -- 172.21.2.102:0/911402127 shutdown_connections 2021-02-10T11:30:04.678 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:04.661 7f3d1484c700 1 --2- 172.21.2.102:0/911402127 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3d100d0340 0x7f3d100ef640 unknown :-1 s=CLOSED pgs=1928 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:04.678 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:04.661 7f3d1484c700 1 -- 172.21.2.102:0/911402127 >> 172.21.2.102:0/911402127 conn(0x7f3d100ce450 msgr2=0x7f3d100d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:04.678 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:04.661 7f3d1484c700 1 -- 172.21.2.102:0/911402127 shutdown_connections 2021-02-10T11:30:04.678 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:04.661 7f3d1484c700 1 -- 172.21.2.102:0/911402127 wait complete. 2021-02-10T11:30:04.680 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:04.676 7f3d1484c700 1 Processor -- start 2021-02-10T11:30:04.680 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:04.676 7f3d1484c700 1 -- start start 2021-02-10T11:30:04.680 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:04.676 7f3d1484c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3d100b4e00 0x7f3d100a0fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:04.680 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:04.676 7f3d1484c700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3d100a3a60 0x7f3d100a3e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:04.680 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:04.676 7f3d1484c700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3d10094750 0x7f3d10094b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:04.680 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:04.676 7f3d1484c700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d100a4550 con 0x7f3d100b4e00 2021-02-10T11:30:04.680 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:04.676 7f3d1484c700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d1018bf30 con 0x7f3d100a3a60 2021-02-10T11:30:04.680 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:04.676 7f3d1484c700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d1018ba60 con 0x7f3d10094750 2021-02-10T11:30:04.680 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:04.677 7f3d0e61c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3d100b4e00 0x7f3d100a0fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:04.680 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:04.677 7f3d0e61c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3d100b4e00 0x7f3d100a0fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:60674/0 (socket says 172.21.2.102:60674) 2021-02-10T11:30:04.680 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:04.677 7f3d0e61c700 1 -- 172.21.2.102:0/665425615 learned_addr learned my addr 172.21.2.102:0/665425615 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:30:04.681 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:04.677 7f3d0e61c700 1 -- 172.21.2.102:0/665425615 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3d10094750 msgr2=0x7f3d10094b50 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:30:04.681 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:04.677 7f3d0e61c700 1 --2- 172.21.2.102:0/665425615 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3d10094750 0x7f3d10094b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:04.681 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:04.677 7f3d0e61c700 1 -- 172.21.2.102:0/665425615 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3d100a3a60 msgr2=0x7f3d100a3e60 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:30:04.681 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:04.677 7f3d0e61c700 1 --2- 172.21.2.102:0/665425615 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3d100a3a60 0x7f3d100a3e60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:04.681 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:04.677 7f3d0e61c700 1 -- 172.21.2.102:0/665425615 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d100a15c0 con 0x7f3d100b4e00 2021-02-10T11:30:04.681 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:04.677 7f3d0e61c700 1 --2- 172.21.2.102:0/665425615 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3d100b4e00 0x7f3d100a0fa0 secure :-1 s=READY pgs=1929 cs=0 l=1 rev1=1 rx=0x7f3d0000a570 tx=0x7f3d0000a910).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:04.681 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:04.677 7f3cfeffd700 1 -- 172.21.2.102:0/665425615 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3d000070d0 con 0x7f3d100b4e00 2021-02-10T11:30:04.681 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:04.677 7f3cfeffd700 1 -- 172.21.2.102:0/665425615 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d0000ad00 con 0x7f3d100b4e00 2021-02-10T11:30:04.681 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:04.677 7f3d1484c700 1 -- 172.21.2.102:0/665425615 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d100a15c0 con 0x7f3d100b4e00 2021-02-10T11:30:04.681 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:04.677 7f3d1484c700 1 -- 172.21.2.102:0/665425615 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d10095460 con 0x7f3d100b4e00 2021-02-10T11:30:04.681 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:04.677 7f3cfeffd700 1 -- 172.21.2.102:0/665425615 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3d00004620 con 0x7f3d100b4e00 2021-02-10T11:30:04.681 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:04.678 7f3cfeffd700 1 -- 172.21.2.102:0/665425615 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f3d000043a0 con 0x7f3d100b4e00 2021-02-10T11:30:04.681 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:04.678 7f3cfeffd700 1 --2- 172.21.2.102:0/665425615 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3cf4018730 0x7f3cf401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:04.682 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:04.678 7f3cfeffd700 1 -- 172.21.2.102:0/665425615 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f3d000327f0 con 0x7f3d100b4e00 2021-02-10T11:30:04.682 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:04.678 7f3d0de1b700 1 --2- 172.21.2.102:0/665425615 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3cf4018730 0x7f3cf401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:04.682 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:04.678 7f3d0de1b700 1 --2- 172.21.2.102:0/665425615 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3cf4018730 0x7f3cf401ac10 crc :-1 s=READY pgs=865 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:04.710 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:04.709 7f3d1484c700 1 -- 172.21.2.102:0/665425615 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d100ee0e0 con 0x7f3d100b4e00 2021-02-10T11:30:04.712 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:04.712 7f3cfeffd700 1 -- 172.21.2.102:0/665425615 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f3d00001160 con 0x7f3d100b4e00 2021-02-10T11:30:04.837 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:04.837 7f3d1484c700 1 -- 172.21.2.102:0/665425615 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3d100ee0e0 con 0x7f3d100b4e00 2021-02-10T11:30:04.838 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:04.838 7f3cfeffd700 1 -- 172.21.2.102:0/665425615 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v92) v1 ==== 74+0+14009 (secure 0 0 0) 0x7f3d000043a0 con 0x7f3d100b4e00 2021-02-10T11:30:04.838 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:30:04.839 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":92,"fsid":"68db30ea-dbce-4aef-a6df-f01c6fea0513","created":"2021-02-10 11:03:06.331557","modified":"2021-02-10 11:30:04.225915","last_up_change":"2021-02-10 11:03:32.690562","last_in_change":"2021-02-10 11:03:32.690562","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:03:37.322519","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:30:02.187273","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":19,"pool_name":"cephfs_data","create_time":"2021-02-10 11:30:03.219690","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"be691b4d-a039-408a-af80-9d53f02b6f1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6818","nonce":36088},{"type":"v1","addr":"172.21.2.102:6820","nonce":36088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6822","nonce":36088},{"type":"v1","addr":"172.21.2.102:6824","nonce":36088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6830","nonce":36088},{"type":"v1","addr":"172.21.2.102:6832","nonce":36088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6826","nonce":36088},{"type":"v1","addr":"172.21.2.102:6828","nonce":36088}]},"public_addr":"172.21.2.102:6820/36088","cluster_addr":"172.21.2.102:6824/36088","heartbeat_back_addr":"172.21.2.102:6832/36088","heartbeat_front_addr":"172.21.2.102:6828/36088","state":["exists","up"]},{"osd":1,"uuid":"64e3ccc8-9c69-4d7c-865e-af8e83822635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6819","nonce":36087},{"type":"v1","addr":"172.21.2.102:6821","nonce":36087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6823","nonce":36087},{"type":"v1","addr":"172.21.2.102:6825","nonce":36087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6831","nonce":36087},{"type":"v1","addr":"172.21.2.102:6833","nonce":36087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6827","nonce":36087},{"type":"v1","addr":"172.21.2.102:6829","nonce":36087}]},"public_addr":"172.21.2.102:6821/36087","cluster_addr":"172.21.2.102:6825/36087","heartbeat_back_addr":"172.21.2.102:6833/36087","heartbeat_front_addr":"172.21.2.102:6829/36087","state":["exists","up"]},{"osd":2,"uuid":"153700af-747f-481f-bdf4-d6072d3a842f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6810","nonce":36089},{"type":"v1","addr":"172.21.2.102:6811","nonce":36089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6812","nonce":36089},{"type":"v1","addr":"172.21.2.102:6813","nonce":36089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6816","nonce":36089},{"type":"v1","addr":"172.21.2.102:6817","nonce":36089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6814","nonce":36089},{"type":"v1","addr":"172.21.2.102:6815","nonce":36089}]},"public_addr":"172.21.2.102:6811/36089","cluster_addr":"172.21.2.102:6813/36089","heartbeat_back_addr":"172.21.2.102:6817/36089","heartbeat_front_addr":"172.21.2.102:6815/36089","state":["exists","up"]},{"osd":3,"uuid":"cbc678f4-83a9-45e6-8830-2f7c232b1ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6802","nonce":36090},{"type":"v1","addr":"172.21.2.102:6803","nonce":36090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6804","nonce":36090},{"type":"v1","addr":"172.21.2.102:6805","nonce":36090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6808","nonce":36090},{"type":"v1","addr":"172.21.2.102:6809","nonce":36090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6806","nonce":36090},{"type":"v1","addr":"172.21.2.102:6807","nonce":36090}]},"public_addr":"172.21.2.102:6803/36090","cluster_addr":"172.21.2.102:6805/36090","heartbeat_back_addr":"172.21.2.102:6809/36090","heartbeat_front_addr":"172.21.2.102:6807/36090","state":["exists","up"]},{"osd":4,"uuid":"c01e4008-b258-4f10-afd2-9e7f37214351","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6808","nonce":36368},{"type":"v1","addr":"172.21.2.113:6811","nonce":36368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6814","nonce":36368},{"type":"v1","addr":"172.21.2.113:6818","nonce":36368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6826","nonce":36368},{"type":"v1","addr":"172.21.2.113:6828","nonce":36368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6820","nonce":36368},{"type":"v1","addr":"172.21.2.113:6824","nonce":36368}]},"public_addr":"172.21.2.113:6811/36368","cluster_addr":"172.21.2.113:6818/36368","heartbeat_back_addr":"172.21.2.113:6828/36368","heartbeat_front_addr":"172.21.2.113:6824/36368","state":["exists","up"]},{"osd":5,"uuid":"551f6007-daf8-4a98-8ee0-c2b16bcccb5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6802","nonce":36366},{"type":"v1","addr":"172.21.2.113:6805","nonce":36366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6807","nonce":36366},{"type":"v1","addr":"172.21.2.113:6810","nonce":36366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6819","nonce":36366},{"type":"v1","addr":"172.21.2.113:6822","nonce":36366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6812","nonce":36366},{"type":"v1","addr":"172.21.2.113:6815","nonce":36366}]},"public_addr":"172.21.2.113:6805/36366","cluster_addr":"172.21.2.113:6810/36366","heartbeat_back_addr":"172.21.2.113:6822/36366","heartbeat_front_addr":"172.21.2.113:6815/36366","state":["exists","up"]},{"osd":6,"uuid":"bf67c040-0e50-48be-b102-c3943bc9ebef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6800","nonce":36367},{"type":"v1","addr":"172.21.2.113:6801","nonce":36367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6803","nonce":36367},{"type":"v1","addr":"172.21.2.113:6804","nonce":36367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6813","nonce":36367},{"type":"v1","addr":"172.21.2.113:6816","nonce":36367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6806","nonce":36367},{"type":"v1","addr":"172.21.2.113:6809","nonce":36367}]},"public_addr":"172.21.2.113:6801/36367","cluster_addr":"172.21.2.113:6804/36367","heartbeat_back_addr":"172.21.2.113:6816/36367","heartbeat_front_addr":"172.21.2.113:6809/36367","state":["exists","up"]},{"osd":7,"uuid":"f5461f80-018d-428f-be89-2eb2bcbe7bb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6817","nonce":36365},{"type":"v1","addr":"172.21.2.113:6821","nonce":36365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6823","nonce":36365},{"type":"v1","addr":"172.21.2.113:6825","nonce":36365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6830","nonce":36365},{"type":"v1","addr":"172.21.2.113:6831","nonce":36365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6827","nonce":36365},{"type":"v1","addr":"172.21.2.113:6829","nonce":36365}]},"public_addr":"172.21.2.113:6821/36365","cluster_addr":"172.21.2.113:6825/36365","heartbeat_back_addr":"172.21.2.113:6831/36365","heartbeat_front_addr":"172.21.2.113:6829/36365","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:30:04.840 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:04.839 7f3d1484c700 1 -- 172.21.2.102:0/665425615 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3cf4018730 msgr2=0x7f3cf401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:04.840 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:04.839 7f3d1484c700 1 --2- 172.21.2.102:0/665425615 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3cf4018730 0x7f3cf401ac10 crc :-1 s=READY pgs=865 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:04.840 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:04.839 7f3d1484c700 1 -- 172.21.2.102:0/665425615 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3d100b4e00 msgr2=0x7f3d100a0fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:04.840 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:04.839 7f3d1484c700 1 --2- 172.21.2.102:0/665425615 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3d100b4e00 0x7f3d100a0fa0 secure :-1 s=READY pgs=1929 cs=0 l=1 rev1=1 rx=0x7f3d0000a570 tx=0x7f3d0000a910).stop 2021-02-10T11:30:04.840 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:04.839 7f3d1484c700 1 -- 172.21.2.102:0/665425615 shutdown_connections 2021-02-10T11:30:04.840 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:04.839 7f3d1484c700 1 --2- 172.21.2.102:0/665425615 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3d10094750 0x7f3d10094b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:04.841 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:04.839 7f3d1484c700 1 --2- 172.21.2.102:0/665425615 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3cf4018730 0x7f3cf401ac10 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:04.841 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:04.839 7f3d1484c700 1 --2- 172.21.2.102:0/665425615 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3d100a3a60 0x7f3d100a3e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:04.841 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:04.839 7f3d1484c700 1 --2- 172.21.2.102:0/665425615 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3d100b4e00 0x7f3d100a0fa0 unknown :-1 s=CLOSED pgs=1929 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:04.841 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:04.839 7f3d1484c700 1 -- 172.21.2.102:0/665425615 >> 172.21.2.102:0/665425615 conn(0x7f3d1018cab0 msgr2=0x7f3d10092c60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:04.841 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:04.840 7f3d1484c700 1 -- 172.21.2.102:0/665425615 shutdown_connections 2021-02-10T11:30:04.841 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:04.840 7f3d1484c700 1 -- 172.21.2.102:0/665425615 wait complete. 2021-02-10T11:30:04.849 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:30:05.278 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:05.261 7f63680ec700 1 Processor -- start 2021-02-10T11:30:05.278 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:05.261 7f63680ec700 1 -- start start 2021-02-10T11:30:05.278 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:05.262 7f63680ec700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f63600d1870 0x7f63601b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:05.278 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:05.262 7f63680ec700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63600f0b10 con 0x7f63600f0650 2021-02-10T11:30:05.279 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:05.262 7f63680ec700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63600d0310 con 0x7f63601b0a10 2021-02-10T11:30:05.279 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:05.262 7f63680ec700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63600e5ed0 con 0x7f63600d1870 2021-02-10T11:30:05.279 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:05.262 7f6366709700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f63600f0650 0x7f63600cfb20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:52908/0 (socket says 172.21.2.102:52908) 2021-02-10T11:30:05.279 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:05.262 7f6366709700 1 -- 172.21.2.102:0/4209418092 learned_addr learned my addr 172.21.2.102:0/4209418092 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:30:05.279 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:05.262 7f6365f08700 1 --2- 172.21.2.102:0/4209418092 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f63600d1870 0x7f63601b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:05.279 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:05.262 7f6365f08700 1 -- 172.21.2.102:0/4209418092 >> v1:172.21.2.113:6789/0 conn(0x7f63601b0a10 legacy=0x7f63600cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:05.279 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:05.262 7f6365f08700 1 -- 172.21.2.102:0/4209418092 >> v1:172.21.2.102:6789/0 conn(0x7f63600f0650 legacy=0x7f63600cfb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:05.279 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:05.262 7f6365f08700 1 -- 172.21.2.102:0/4209418092 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63600d0530 con 0x7f63600d1870 2021-02-10T11:30:05.279 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:05.262 7f6365f08700 1 --2- 172.21.2.102:0/4209418092 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f63600d1870 0x7f63601b03f0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f6350006650 tx=0x7f6350009a00).ready entity=mon.2 client_cookie=99e6c913959d838b server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:05.279 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:05.262 7f6364f06700 1 -- 172.21.2.102:0/4209418092 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6350009df0 con 0x7f63600d1870 2021-02-10T11:30:05.279 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:05.263 7f6364f06700 1 -- 172.21.2.102:0/4209418092 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63500009b0 con 0x7f63600d1870 2021-02-10T11:30:05.280 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:05.263 7f6364f06700 1 -- 172.21.2.102:0/4209418092 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f635000c2b0 con 0x7f63600d1870 2021-02-10T11:30:05.280 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:05.263 7f63680ec700 1 -- 172.21.2.102:0/4209418092 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f63600d1870 msgr2=0x7f63601b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:05.280 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:05.263 7f63680ec700 1 --2- 172.21.2.102:0/4209418092 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f63600d1870 0x7f63601b03f0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f6350006650 tx=0x7f6350009a00).stop 2021-02-10T11:30:05.280 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:05.263 7f63680ec700 1 -- 172.21.2.102:0/4209418092 shutdown_connections 2021-02-10T11:30:05.280 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:05.263 7f63680ec700 1 --2- 172.21.2.102:0/4209418092 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f63600d1870 0x7f63601b03f0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:05.280 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:05.263 7f63680ec700 1 -- 172.21.2.102:0/4209418092 >> 172.21.2.102:0/4209418092 conn(0x7f63600ce450 msgr2=0x7f63600d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:05.280 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:05.263 7f63680ec700 1 -- 172.21.2.102:0/4209418092 shutdown_connections 2021-02-10T11:30:05.280 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:05.263 7f63680ec700 1 -- 172.21.2.102:0/4209418092 wait complete. 2021-02-10T11:30:05.281 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:05.278 7f63680ec700 1 Processor -- start 2021-02-10T11:30:05.282 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:05.278 7f63680ec700 1 -- start start 2021-02-10T11:30:05.282 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:05.279 7f63680ec700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f63600af040 0x7f63600af440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:05.282 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:05.279 7f63680ec700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f63600afa60 0x7f63600ada20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:05.282 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:05.279 7f63680ec700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f63600afe60 0x7f63600ae040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:05.282 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:05.279 7f63680ec700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63600923c0 con 0x7f63600afe60 2021-02-10T11:30:05.282 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:05.279 7f63680ec700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63600b0410 con 0x7f63600afa60 2021-02-10T11:30:05.282 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:05.279 7f63680ec700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6360091e40 con 0x7f63600af040 2021-02-10T11:30:05.282 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:05.279 7f6366709700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f63600afe60 0x7f63600ae040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:05.282 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:05.279 7f6366709700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f63600afe60 0x7f63600ae040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:60686/0 (socket says 172.21.2.102:60686) 2021-02-10T11:30:05.282 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:05.279 7f6366709700 1 -- 172.21.2.102:0/4286669035 learned_addr learned my addr 172.21.2.102:0/4286669035 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:30:05.282 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:05.279 7f6365f08700 1 --2- 172.21.2.102:0/4286669035 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f63600af040 0x7f63600af440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:05.282 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:05.279 7f6365707700 1 --2- 172.21.2.102:0/4286669035 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f63600afa60 0x7f63600ada20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:05.283 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:05.279 7f6366709700 1 -- 172.21.2.102:0/4286669035 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f63600af040 msgr2=0x7f63600af440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:05.283 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:05.279 7f6366709700 1 --2- 172.21.2.102:0/4286669035 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f63600af040 0x7f63600af440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:05.283 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:05.279 7f6366709700 1 -- 172.21.2.102:0/4286669035 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f63600afa60 msgr2=0x7f63600ada20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:05.283 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:05.279 7f6366709700 1 --2- 172.21.2.102:0/4286669035 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f63600afa60 0x7f63600ada20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:05.283 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:05.279 7f6366709700 1 -- 172.21.2.102:0/4286669035 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f636008f6d0 con 0x7f63600afe60 2021-02-10T11:30:05.283 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:05.279 7f6366709700 1 --2- 172.21.2.102:0/4286669035 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f63600afe60 0x7f63600ae040 secure :-1 s=READY pgs=1931 cs=0 l=1 rev1=1 rx=0x7f635800a940 tx=0x7f635800ace0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:05.283 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:05.279 7f63567fc700 1 -- 172.21.2.102:0/4286669035 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6358001270 con 0x7f63600afe60 2021-02-10T11:30:05.283 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:05.279 7f63567fc700 1 -- 172.21.2.102:0/4286669035 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6358007880 con 0x7f63600afe60 2021-02-10T11:30:05.283 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:05.279 7f63680ec700 1 -- 172.21.2.102:0/4286669035 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f636008f6d0 con 0x7f63600afe60 2021-02-10T11:30:05.283 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:05.279 7f63567fc700 1 -- 172.21.2.102:0/4286669035 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f63580073d0 con 0x7f63600afe60 2021-02-10T11:30:05.283 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:05.280 7f63680ec700 1 -- 172.21.2.102:0/4286669035 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f636008f6d0 con 0x7f63600afe60 2021-02-10T11:30:05.283 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:05.280 7f63567fc700 1 -- 172.21.2.102:0/4286669035 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f6358004720 con 0x7f63600afe60 2021-02-10T11:30:05.283 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:05.280 7f63567fc700 1 --2- 172.21.2.102:0/4286669035 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f63480186c0 0x7f634801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:05.283 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:05.280 7f63567fc700 1 -- 172.21.2.102:0/4286669035 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f6358032980 con 0x7f63600afe60 2021-02-10T11:30:05.284 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:05.280 7f6365f08700 1 --2- 172.21.2.102:0/4286669035 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f63480186c0 0x7f634801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:05.284 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:05.281 7f6365f08700 1 --2- 172.21.2.102:0/4286669035 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f63480186c0 0x7f634801abc0 crc :-1 s=READY pgs=866 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:05.312 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:05.311 7f63680ec700 1 -- 172.21.2.102:0/4286669035 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63600e7df0 con 0x7f63600afe60 2021-02-10T11:30:05.314 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:05.314 7f63567fc700 1 -- 172.21.2.102:0/4286669035 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f6358004ee0 con 0x7f63600afe60 2021-02-10T11:30:05.441 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:05.441 7f63680ec700 1 -- 172.21.2.102:0/4286669035 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f63600ce450 con 0x7f63600afe60 2021-02-10T11:30:05.442 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:05.442 7f63567fc700 1 -- 172.21.2.102:0/4286669035 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) v1 ==== 74+0+13996 (secure 0 0 0) 0x7f6358001270 con 0x7f63600afe60 2021-02-10T11:30:05.442 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:30:05.443 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":93,"fsid":"68db30ea-dbce-4aef-a6df-f01c6fea0513","created":"2021-02-10 11:03:06.331557","modified":"2021-02-10 11:30:05.228248","last_up_change":"2021-02-10 11:03:32.690562","last_in_change":"2021-02-10 11:03:32.690562","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:03:37.322519","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:30:02.187273","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":19,"pool_name":"cephfs_data","create_time":"2021-02-10 11:30:03.219690","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"93","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"be691b4d-a039-408a-af80-9d53f02b6f1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6818","nonce":36088},{"type":"v1","addr":"172.21.2.102:6820","nonce":36088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6822","nonce":36088},{"type":"v1","addr":"172.21.2.102:6824","nonce":36088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6830","nonce":36088},{"type":"v1","addr":"172.21.2.102:6832","nonce":36088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6826","nonce":36088},{"type":"v1","addr":"172.21.2.102:6828","nonce":36088}]},"public_addr":"172.21.2.102:6820/36088","cluster_addr":"172.21.2.102:6824/36088","heartbeat_back_addr":"172.21.2.102:6832/36088","heartbeat_front_addr":"172.21.2.102:6828/36088","state":["exists","up"]},{"osd":1,"uuid":"64e3ccc8-9c69-4d7c-865e-af8e83822635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6819","nonce":36087},{"type":"v1","addr":"172.21.2.102:6821","nonce":36087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6823","nonce":36087},{"type":"v1","addr":"172.21.2.102:6825","nonce":36087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6831","nonce":36087},{"type":"v1","addr":"172.21.2.102:6833","nonce":36087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6827","nonce":36087},{"type":"v1","addr":"172.21.2.102:6829","nonce":36087}]},"public_addr":"172.21.2.102:6821/36087","cluster_addr":"172.21.2.102:6825/36087","heartbeat_back_addr":"172.21.2.102:6833/36087","heartbeat_front_addr":"172.21.2.102:6829/36087","state":["exists","up"]},{"osd":2,"uuid":"153700af-747f-481f-bdf4-d6072d3a842f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6810","nonce":36089},{"type":"v1","addr":"172.21.2.102:6811","nonce":36089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6812","nonce":36089},{"type":"v1","addr":"172.21.2.102:6813","nonce":36089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6816","nonce":36089},{"type":"v1","addr":"172.21.2.102:6817","nonce":36089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6814","nonce":36089},{"type":"v1","addr":"172.21.2.102:6815","nonce":36089}]},"public_addr":"172.21.2.102:6811/36089","cluster_addr":"172.21.2.102:6813/36089","heartbeat_back_addr":"172.21.2.102:6817/36089","heartbeat_front_addr":"172.21.2.102:6815/36089","state":["exists","up"]},{"osd":3,"uuid":"cbc678f4-83a9-45e6-8830-2f7c232b1ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6802","nonce":36090},{"type":"v1","addr":"172.21.2.102:6803","nonce":36090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6804","nonce":36090},{"type":"v1","addr":"172.21.2.102:6805","nonce":36090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6808","nonce":36090},{"type":"v1","addr":"172.21.2.102:6809","nonce":36090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6806","nonce":36090},{"type":"v1","addr":"172.21.2.102:6807","nonce":36090}]},"public_addr":"172.21.2.102:6803/36090","cluster_addr":"172.21.2.102:6805/36090","heartbeat_back_addr":"172.21.2.102:6809/36090","heartbeat_front_addr":"172.21.2.102:6807/36090","state":["exists","up"]},{"osd":4,"uuid":"c01e4008-b258-4f10-afd2-9e7f37214351","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6808","nonce":36368},{"type":"v1","addr":"172.21.2.113:6811","nonce":36368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6814","nonce":36368},{"type":"v1","addr":"172.21.2.113:6818","nonce":36368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6826","nonce":36368},{"type":"v1","addr":"172.21.2.113:6828","nonce":36368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6820","nonce":36368},{"type":"v1","addr":"172.21.2.113:6824","nonce":36368}]},"public_addr":"172.21.2.113:6811/36368","cluster_addr":"172.21.2.113:6818/36368","heartbeat_back_addr":"172.21.2.113:6828/36368","heartbeat_front_addr":"172.21.2.113:6824/36368","state":["exists","up"]},{"osd":5,"uuid":"551f6007-daf8-4a98-8ee0-c2b16bcccb5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6802","nonce":36366},{"type":"v1","addr":"172.21.2.113:6805","nonce":36366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6807","nonce":36366},{"type":"v1","addr":"172.21.2.113:6810","nonce":36366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6819","nonce":36366},{"type":"v1","addr":"172.21.2.113:6822","nonce":36366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6812","nonce":36366},{"type":"v1","addr":"172.21.2.113:6815","nonce":36366}]},"public_addr":"172.21.2.113:6805/36366","cluster_addr":"172.21.2.113:6810/36366","heartbeat_back_addr":"172.21.2.113:6822/36366","heartbeat_front_addr":"172.21.2.113:6815/36366","state":["exists","up"]},{"osd":6,"uuid":"bf67c040-0e50-48be-b102-c3943bc9ebef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6800","nonce":36367},{"type":"v1","addr":"172.21.2.113:6801","nonce":36367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6803","nonce":36367},{"type":"v1","addr":"172.21.2.113:6804","nonce":36367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6813","nonce":36367},{"type":"v1","addr":"172.21.2.113:6816","nonce":36367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6806","nonce":36367},{"type":"v1","addr":"172.21.2.113:6809","nonce":36367}]},"public_addr":"172.21.2.113:6801/36367","cluster_addr":"172.21.2.113:6804/36367","heartbeat_back_addr":"172.21.2.113:6816/36367","heartbeat_front_addr":"172.21.2.113:6809/36367","state":["exists","up"]},{"osd":7,"uuid":"f5461f80-018d-428f-be89-2eb2bcbe7bb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6817","nonce":36365},{"type":"v1","addr":"172.21.2.113:6821","nonce":36365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6823","nonce":36365},{"type":"v1","addr":"172.21.2.113:6825","nonce":36365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6830","nonce":36365},{"type":"v1","addr":"172.21.2.113:6831","nonce":36365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6827","nonce":36365},{"type":"v1","addr":"172.21.2.113:6829","nonce":36365}]},"public_addr":"172.21.2.113:6821/36365","cluster_addr":"172.21.2.113:6825/36365","heartbeat_back_addr":"172.21.2.113:6831/36365","heartbeat_front_addr":"172.21.2.113:6829/36365","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:30:05.444 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:05.443 7f63680ec700 1 -- 172.21.2.102:0/4286669035 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f63480186c0 msgr2=0x7f634801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:05.444 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:05.443 7f63680ec700 1 --2- 172.21.2.102:0/4286669035 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f63480186c0 0x7f634801abc0 crc :-1 s=READY pgs=866 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:05.444 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:05.443 7f63680ec700 1 -- 172.21.2.102:0/4286669035 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f63600afe60 msgr2=0x7f63600ae040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:05.444 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:05.443 7f63680ec700 1 --2- 172.21.2.102:0/4286669035 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f63600afe60 0x7f63600ae040 secure :-1 s=READY pgs=1931 cs=0 l=1 rev1=1 rx=0x7f635800a940 tx=0x7f635800ace0).stop 2021-02-10T11:30:05.444 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:05.443 7f63680ec700 1 -- 172.21.2.102:0/4286669035 shutdown_connections 2021-02-10T11:30:05.444 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:05.443 7f63680ec700 1 --2- 172.21.2.102:0/4286669035 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f63600afe60 0x7f63600ae040 unknown :-1 s=CLOSED pgs=1931 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:05.444 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:05.443 7f63680ec700 1 --2- 172.21.2.102:0/4286669035 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f63480186c0 0x7f634801abc0 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:05.445 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:05.443 7f63680ec700 1 --2- 172.21.2.102:0/4286669035 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f63600afa60 0x7f63600ada20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:05.445 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:05.443 7f63680ec700 1 --2- 172.21.2.102:0/4286669035 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f63600af040 0x7f63600af440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:05.445 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:05.443 7f63680ec700 1 -- 172.21.2.102:0/4286669035 >> 172.21.2.102:0/4286669035 conn(0x7f63600a0030 msgr2=0x7f63600f22b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:05.445 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:05.444 7f63680ec700 1 -- 172.21.2.102:0/4286669035 shutdown_connections 2021-02-10T11:30:05.445 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:05.444 7f63680ec700 1 -- 172.21.2.102:0/4286669035 wait complete. 2021-02-10T11:30:05.453 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2021-02-10T11:30:05.885 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:05.866 7f3ad8ba4700 1 Processor -- start 2021-02-10T11:30:05.885 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:05.867 7f3ad8ba4700 1 -- start start 2021-02-10T11:30:05.885 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:05.867 7f3ad8ba4700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f3ad40d1c70 0x7f3ad40cef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:05.885 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:05.867 7f3ad8ba4700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3ad40cf580 0x7f3ad40cf980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:05.885 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:05.867 7f3ad8ba4700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ad40fe1c0 con 0x7f3ad40d1870 2021-02-10T11:30:05.886 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:05.867 7f3ad8ba4700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ad41b0cb0 con 0x7f3ad40d1c70 2021-02-10T11:30:05.886 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:05.867 7f3ad8ba4700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ad4109a60 con 0x7f3ad40cf580 2021-02-10T11:30:05.886 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:05.867 7f3ad285e700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f3ad40d1870 0x7f3ad41b04c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:52918/0 (socket says 172.21.2.102:52918) 2021-02-10T11:30:05.886 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:05.867 7f3ad285e700 1 -- 172.21.2.102:0/3967358241 learned_addr learned my addr 172.21.2.102:0/3967358241 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:30:05.886 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:05.868 7f3ad305f700 1 --2- 172.21.2.102:0/3967358241 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3ad40cf580 0x7f3ad40cf980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:05.886 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:05.868 7f3ad185c700 1 -- 172.21.2.102:0/3967358241 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4016407147 0 0) 0x7f3ac40009b0 con 0x7f3ad40d1870 2021-02-10T11:30:05.886 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:05.868 7f3ad205d700 1 --2- 172.21.2.102:0/3967358241 >> v2:172.21.2.113:3300/0 conn(0x7f3ad40d1c70 0x7f3ad40cef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:05.886 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:05.868 7f3ad185c700 1 -- 172.21.2.102:0/3967358241 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3abc002fe0 con 0x7f3ad40d1870 2021-02-10T11:30:05.886 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:05.868 7f3ad185c700 1 -- 172.21.2.102:0/3967358241 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2124882511 0 0) 0x7f3ac4001010 con 0x7f3ad40d1870 2021-02-10T11:30:05.886 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:05.868 7f3ad185c700 1 -- 172.21.2.102:0/3967358241 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3ad40cf580 msgr2=0x7f3ad40cf980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:05.886 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:05.868 7f3ad185c700 1 --2- 172.21.2.102:0/3967358241 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3ad40cf580 0x7f3ad40cf980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:05.886 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:05.868 7f3ad185c700 1 -- 172.21.2.102:0/3967358241 >> v2:172.21.2.113:3300/0 conn(0x7f3ad40d1c70 msgr2=0x7f3ad40cef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:05.887 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:05.868 7f3ad185c700 1 --2- 172.21.2.102:0/3967358241 >> v2:172.21.2.113:3300/0 conn(0x7f3ad40d1c70 0x7f3ad40cef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:05.887 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:05.868 7f3ad185c700 1 -- 172.21.2.102:0/3967358241 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ad4088070 con 0x7f3ad40d1870 2021-02-10T11:30:05.887 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:05.868 7f3ad185c700 1 -- 172.21.2.102:0/3967358241 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7f3ac4001590 con 0x7f3ad40d1870 2021-02-10T11:30:05.887 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:05.869 7f3ad185c700 1 -- 172.21.2.102:0/3967358241 >> v1:172.21.2.102:6789/0 conn(0x7f3ad40d1870 legacy=0x7f3ad41b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:05.887 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:05.869 7f3ad185c700 1 --2- 172.21.2.102:0/3967358241 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3abc003a60 0x7f3abc005ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:05.887 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:05.869 7f3ad185c700 1 --2- 172.21.2.102:0/3967358241 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3abc04a780 0x7f3abc04cca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:05.887 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:05.869 7f3ad185c700 1 --2- 172.21.2.102:0/3967358241 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3abc04d5a0 0x7f3abc04fa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:05.887 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:05.869 7f3ad185c700 1 -- 172.21.2.102:0/3967358241 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3abc0502e0 con 0x7f3abc04a780 2021-02-10T11:30:05.887 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:05.869 7f3ad185c700 1 -- 172.21.2.102:0/3967358241 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3abc050500 con 0x7f3abc003a60 2021-02-10T11:30:05.887 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:05.869 7f3ad185c700 1 -- 172.21.2.102:0/3967358241 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3abc050720 con 0x7f3abc04d5a0 2021-02-10T11:30:05.887 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:05.869 7f3ad205d700 1 --2- 172.21.2.102:0/3967358241 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3abc04a780 0x7f3abc04cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:05.887 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:05.869 7f3ad285e700 1 --2- 172.21.2.102:0/3967358241 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3abc003a60 0x7f3abc005ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:05.887 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:05.869 7f3ad305f700 1 --2- 172.21.2.102:0/3967358241 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3abc04d5a0 0x7f3abc04fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:05.888 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:05.869 7f3ad205d700 1 -- 172.21.2.102:0/3967358241 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3abc04d5a0 msgr2=0x7f3abc04fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:05.888 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:05.869 7f3ad205d700 1 --2- 172.21.2.102:0/3967358241 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3abc04d5a0 0x7f3abc04fa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:05.888 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:05.869 7f3ad205d700 1 -- 172.21.2.102:0/3967358241 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3abc003a60 msgr2=0x7f3abc005ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:05.888 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:05.869 7f3ad205d700 1 --2- 172.21.2.102:0/3967358241 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3abc003a60 0x7f3abc005ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:05.888 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:05.869 7f3ad205d700 1 -- 172.21.2.102:0/3967358241 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3abc0509c0 con 0x7f3abc04a780 2021-02-10T11:30:05.888 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:05.869 7f3ad285e700 1 --2- 172.21.2.102:0/3967358241 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3abc003a60 0x7f3abc005ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:30:05.888 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:05.869 7f3ad205d700 1 --2- 172.21.2.102:0/3967358241 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3abc04a780 0x7f3abc04cca0 secure :-1 s=READY pgs=1933 cs=0 l=1 rev1=1 rx=0x7f3ac800c870 tx=0x7f3ac800cc10).ready entity=mon.0 client_cookie=b654351b4b8734aa server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:05.888 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:05.869 7f3ad185c700 1 -- 172.21.2.102:0/3967358241 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3ac80090d0 con 0x7f3abc04a780 2021-02-10T11:30:05.888 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:05.869 7f3ad185c700 1 -- 172.21.2.102:0/3967358241 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ac8009410 con 0x7f3abc04a780 2021-02-10T11:30:05.888 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:05.869 7f3ad185c700 1 -- 172.21.2.102:0/3967358241 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3ac80098f0 con 0x7f3abc04a780 2021-02-10T11:30:05.888 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:05.869 7f3ad8ba4700 1 -- 172.21.2.102:0/3967358241 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3abc04a780 msgr2=0x7f3abc04cca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:05.888 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:05.869 7f3ad8ba4700 1 --2- 172.21.2.102:0/3967358241 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3abc04a780 0x7f3abc04cca0 secure :-1 s=READY pgs=1933 cs=0 l=1 rev1=1 rx=0x7f3ac800c870 tx=0x7f3ac800cc10).stop 2021-02-10T11:30:05.888 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:05.869 7f3ad305f700 1 -- 172.21.2.102:0/3967358241 reap_dead start 2021-02-10T11:30:05.889 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:05.870 7f3ad8ba4700 1 -- 172.21.2.102:0/3967358241 shutdown_connections 2021-02-10T11:30:05.889 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:05.870 7f3ad8ba4700 1 -- 172.21.2.102:0/3967358241 >> 172.21.2.102:0/3967358241 conn(0x7f3ad40ce420 msgr2=0x7f3ad40d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:05.889 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:05.870 7f3ad8ba4700 1 -- 172.21.2.102:0/3967358241 shutdown_connections 2021-02-10T11:30:05.889 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:05.870 7f3ad8ba4700 1 -- 172.21.2.102:0/3967358241 wait complete. 2021-02-10T11:30:05.889 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:05.885 7f3ad8ba4700 1 Processor -- start 2021-02-10T11:30:05.889 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:05.885 7f3ad8ba4700 1 -- start start 2021-02-10T11:30:05.890 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:05.885 7f3ad8ba4700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3ad4085bc0 0x7f3ad40863e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:05.890 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:05.885 7f3ad8ba4700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3ad40954c0 0x7f3ad4096300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:05.890 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:05.885 7f3ad8ba4700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3ad40958c0 0x7f3ad4096920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:05.891 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:05.885 7f3ad8ba4700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ad4095d80 con 0x7f3ad40958c0 2021-02-10T11:30:05.891 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:05.885 7f3ad8ba4700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ad40bfa60 con 0x7f3ad40954c0 2021-02-10T11:30:05.891 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:05.885 7f3ad8ba4700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ad4085fc0 con 0x7f3ad4085bc0 2021-02-10T11:30:05.891 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:05.885 7f3ad305f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3ad40958c0 0x7f3ad4096920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:05.891 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:05.885 7f3ad305f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3ad40958c0 0x7f3ad4096920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:60704/0 (socket says 172.21.2.102:60704) 2021-02-10T11:30:05.891 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:05.885 7f3ad305f700 1 -- 172.21.2.102:0/4003425193 learned_addr learned my addr 172.21.2.102:0/4003425193 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:30:05.891 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:05.885 7f3ad305f700 1 -- 172.21.2.102:0/4003425193 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3ad4085bc0 msgr2=0x7f3ad40863e0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:30:05.891 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:05.885 7f3ad305f700 1 --2- 172.21.2.102:0/4003425193 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3ad4085bc0 0x7f3ad40863e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:05.891 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:05.885 7f3ad305f700 1 -- 172.21.2.102:0/4003425193 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3ad40954c0 msgr2=0x7f3ad4096300 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:30:05.891 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:05.885 7f3ad305f700 1 --2- 172.21.2.102:0/4003425193 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3ad40954c0 0x7f3ad4096300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:05.891 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:05.885 7f3ad305f700 1 -- 172.21.2.102:0/4003425193 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ad4086c10 con 0x7f3ad40958c0 2021-02-10T11:30:05.891 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:05.886 7f3ad305f700 1 --2- 172.21.2.102:0/4003425193 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3ad40958c0 0x7f3ad4096920 secure :-1 s=READY pgs=1934 cs=0 l=1 rev1=1 rx=0x7f3acc00a940 tx=0x7f3acc00ace0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:05.891 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:05.886 7f3ac37fe700 1 -- 172.21.2.102:0/4003425193 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3acc0066f0 con 0x7f3ad40958c0 2021-02-10T11:30:05.892 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:05.886 7f3ac37fe700 1 -- 172.21.2.102:0/4003425193 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3acc003030 con 0x7f3ad40958c0 2021-02-10T11:30:05.892 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:05.886 7f3ad8ba4700 1 -- 172.21.2.102:0/4003425193 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ad4086c10 con 0x7f3ad40958c0 2021-02-10T11:30:05.892 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:05.886 7f3ad8ba4700 1 -- 172.21.2.102:0/4003425193 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ad4086c10 con 0x7f3ad40958c0 2021-02-10T11:30:05.892 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:05.886 7f3ac37fe700 1 -- 172.21.2.102:0/4003425193 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3acc0015f0 con 0x7f3ad40958c0 2021-02-10T11:30:05.892 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:05.886 7f3ac37fe700 1 -- 172.21.2.102:0/4003425193 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f3acc001400 con 0x7f3ad40958c0 2021-02-10T11:30:05.892 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:05.887 7f3ac37fe700 1 --2- 172.21.2.102:0/4003425193 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3ab8018730 0x7f3ab801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:05.892 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:05.887 7f3ac37fe700 1 -- 172.21.2.102:0/4003425193 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f3acc032b40 con 0x7f3ad40958c0 2021-02-10T11:30:05.892 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:05.887 7f3ad285e700 1 --2- 172.21.2.102:0/4003425193 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3ab8018730 0x7f3ab801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:05.892 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:05.887 7f3ad285e700 1 --2- 172.21.2.102:0/4003425193 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3ab8018730 0x7f3ab801ac10 crc :-1 s=READY pgs=867 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:05.917 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:05.917 7f3ad8ba4700 1 -- 172.21.2.102:0/4003425193 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ad40fe1c0 con 0x7f3ad40958c0 2021-02-10T11:30:05.920 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:05.919 7f3ac37fe700 1 -- 172.21.2.102:0/4003425193 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f3acc003010 con 0x7f3ad40958c0 2021-02-10T11:30:06.057 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:06.056 7f3ad8ba4700 1 -- 172.21.2.102:0/4003425193 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"} v 0) v1 -- 0x7f3ad40fe1c0 con 0x7f3ad40958c0 2021-02-10T11:30:06.235 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:06.234 7f3ac37fe700 1 -- 172.21.2.102:0/4003425193 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"}]=0 v73) v1 ==== 118+0+0 (secure 0 0 0) 0x7f3acc003010 con 0x7f3ad40958c0 2021-02-10T11:30:06.275 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:06.274 7f3ad8ba4700 1 -- 172.21.2.102:0/4003425193 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3ab8018730 msgr2=0x7f3ab801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:06.275 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:06.274 7f3ad8ba4700 1 --2- 172.21.2.102:0/4003425193 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3ab8018730 0x7f3ab801ac10 crc :-1 s=READY pgs=867 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:06.275 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:06.274 7f3ad8ba4700 1 -- 172.21.2.102:0/4003425193 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3ad40958c0 msgr2=0x7f3ad4096920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:06.275 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:06.274 7f3ad8ba4700 1 --2- 172.21.2.102:0/4003425193 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3ad40958c0 0x7f3ad4096920 secure :-1 s=READY pgs=1934 cs=0 l=1 rev1=1 rx=0x7f3acc00a940 tx=0x7f3acc00ace0).stop 2021-02-10T11:30:06.275 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:06.274 7f3ad8ba4700 1 -- 172.21.2.102:0/4003425193 shutdown_connections 2021-02-10T11:30:06.276 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:06.274 7f3ad8ba4700 1 --2- 172.21.2.102:0/4003425193 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3ad40958c0 0x7f3ad4096920 unknown :-1 s=CLOSED pgs=1934 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:06.276 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:06.274 7f3ad8ba4700 1 --2- 172.21.2.102:0/4003425193 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3ab8018730 0x7f3ab801ac10 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:06.276 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:06.274 7f3ad8ba4700 1 --2- 172.21.2.102:0/4003425193 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3ad40954c0 0x7f3ad4096300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:06.276 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:06.274 7f3ad8ba4700 1 --2- 172.21.2.102:0/4003425193 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3ad4085bc0 0x7f3ad40863e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:06.276 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:06.274 7f3ad8ba4700 1 -- 172.21.2.102:0/4003425193 >> 172.21.2.102:0/4003425193 conn(0x7f3ad40bef00 msgr2=0x7f3ad409d1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:06.276 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:06.274 7f3ad8ba4700 1 -- 172.21.2.102:0/4003425193 shutdown_connections 2021-02-10T11:30:06.276 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:06.274 7f3ad8ba4700 1 -- 172.21.2.102:0/4003425193 wait complete. 2021-02-10T11:30:06.283 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:30:06.713 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:06.694 7f63d4621700 1 Processor -- start 2021-02-10T11:30:06.713 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:06.694 7f63d4621700 1 -- start start 2021-02-10T11:30:06.713 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:06.695 7f63d4621700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f63cc0d1870 0x7f63cc1b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:06.713 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:06.695 7f63d4621700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63cc0f0b10 con 0x7f63cc0f0650 2021-02-10T11:30:06.713 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:06.695 7f63d4621700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63cc0d0310 con 0x7f63cc1b0a10 2021-02-10T11:30:06.713 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:06.695 7f63d4621700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63cc0d0530 con 0x7f63cc0d1870 2021-02-10T11:30:06.713 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:06.695 7f63d2c3e700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f63cc0f0650 0x7f63cc0cfb20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:52936/0 (socket says 172.21.2.102:52936) 2021-02-10T11:30:06.713 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:06.695 7f63d2c3e700 1 -- 172.21.2.102:0/2670218072 learned_addr learned my addr 172.21.2.102:0/2670218072 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:30:06.714 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:06.695 7f63d143b700 1 -- 172.21.2.102:0/2670218072 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2951937808 0 0) 0x7f63c4001500 con 0x7f63cc0f0650 2021-02-10T11:30:06.714 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:06.695 7f63d143b700 1 -- 172.21.2.102:0/2670218072 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63b8002fe0 con 0x7f63cc0f0650 2021-02-10T11:30:06.714 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:06.695 7f63d243d700 1 --2- 172.21.2.102:0/2670218072 >> v2:172.21.2.113:3300/0 conn(0x7f63cc0d1870 0x7f63cc1b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:06.714 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:06.695 7f63d143b700 1 -- 172.21.2.102:0/2670218072 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3833470258 0 0) 0x7f63c4001460 con 0x7f63cc0f0650 2021-02-10T11:30:06.714 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:06.695 7f63d143b700 1 -- 172.21.2.102:0/2670218072 >> v2:172.21.2.113:3300/0 conn(0x7f63cc0d1870 msgr2=0x7f63cc1b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:06.714 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:06.695 7f63d143b700 1 --2- 172.21.2.102:0/2670218072 >> v2:172.21.2.113:3300/0 conn(0x7f63cc0d1870 0x7f63cc1b03f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:06.714 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:06.695 7f63d143b700 1 -- 172.21.2.102:0/2670218072 >> v1:172.21.2.113:6789/0 conn(0x7f63cc1b0a10 legacy=0x7f63cc0cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:06.714 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:06.695 7f63d143b700 1 -- 172.21.2.102:0/2670218072 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63cc0d1c70 con 0x7f63cc0f0650 2021-02-10T11:30:06.714 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:06.695 7f63d143b700 1 -- 172.21.2.102:0/2670218072 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7f63c4001980 con 0x7f63cc0f0650 2021-02-10T11:30:06.714 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:06.695 7f63d143b700 1 -- 172.21.2.102:0/2670218072 >> v1:172.21.2.102:6789/0 conn(0x7f63cc0f0650 legacy=0x7f63cc0cfb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:06.714 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:06.695 7f63d143b700 1 --2- 172.21.2.102:0/2670218072 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f63b8003b20 0x7f63b8005f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:06.714 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:06.695 7f63d143b700 1 --2- 172.21.2.102:0/2670218072 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f63b80067b0 0x7f63b8008c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:06.714 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:06.696 7f63d143b700 1 --2- 172.21.2.102:0/2670218072 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f63b8009590 0x7f63b800bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:06.715 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:06.696 7f63d143b700 1 -- 172.21.2.102:0/2670218072 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63b800c300 con 0x7f63b80067b0 2021-02-10T11:30:06.715 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:06.696 7f63d143b700 1 -- 172.21.2.102:0/2670218072 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63b800c520 con 0x7f63b8003b20 2021-02-10T11:30:06.715 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:06.696 7f63d143b700 1 -- 172.21.2.102:0/2670218072 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63b800c740 con 0x7f63b8009590 2021-02-10T11:30:06.715 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:06.696 7f63d243d700 1 --2- 172.21.2.102:0/2670218072 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f63b8003b20 0x7f63b8005f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:06.715 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:06.696 7f63d2c3e700 1 --2- 172.21.2.102:0/2670218072 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f63b8009590 0x7f63b800bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:06.715 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:06.696 7f63d1c3c700 1 --2- 172.21.2.102:0/2670218072 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f63b80067b0 0x7f63b8008c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:06.715 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:06.696 7f63d243d700 1 -- 172.21.2.102:0/2670218072 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f63b8009590 msgr2=0x7f63b800bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:06.715 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:06.696 7f63d243d700 1 --2- 172.21.2.102:0/2670218072 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f63b8009590 0x7f63b800bab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:06.715 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:06.696 7f63d243d700 1 -- 172.21.2.102:0/2670218072 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f63b80067b0 msgr2=0x7f63b8008c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:06.715 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:06.696 7f63d243d700 1 --2- 172.21.2.102:0/2670218072 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f63b80067b0 0x7f63b8008c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:06.715 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:06.696 7f63d2c3e700 1 -- 172.21.2.102:0/2670218072 reap_dead start 2021-02-10T11:30:06.715 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:06.696 7f63d243d700 1 -- 172.21.2.102:0/2670218072 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63b800c9e0 con 0x7f63b8003b20 2021-02-10T11:30:06.715 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:06.696 7f63d243d700 1 --2- 172.21.2.102:0/2670218072 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f63b8003b20 0x7f63b8005f30 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 rx=0x7f63c804d2f0 tx=0x7f63c8006460).ready entity=mon.1 client_cookie=c5990a256e717a84 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:06.716 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:06.696 7f63d143b700 1 -- 172.21.2.102:0/2670218072 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f63c8006880 con 0x7f63b8003b20 2021-02-10T11:30:06.716 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:06.696 7f63d143b700 1 -- 172.21.2.102:0/2670218072 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63c8000c90 con 0x7f63b8003b20 2021-02-10T11:30:06.716 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:06.696 7f63d143b700 1 -- 172.21.2.102:0/2670218072 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f63c80022b0 con 0x7f63b8003b20 2021-02-10T11:30:06.716 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:06.696 7f63d4621700 1 -- 172.21.2.102:0/2670218072 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f63b8003b20 msgr2=0x7f63b8005f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:06.716 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:06.696 7f63d4621700 1 --2- 172.21.2.102:0/2670218072 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f63b8003b20 0x7f63b8005f30 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 rx=0x7f63c804d2f0 tx=0x7f63c8006460).stop 2021-02-10T11:30:06.716 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:06.697 7f63d4621700 1 -- 172.21.2.102:0/2670218072 shutdown_connections 2021-02-10T11:30:06.716 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:06.697 7f63d4621700 1 --2- 172.21.2.102:0/2670218072 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f63b8003b20 0x7f63b8005f30 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:06.716 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:06.697 7f63d4621700 1 -- 172.21.2.102:0/2670218072 >> 172.21.2.102:0/2670218072 conn(0x7f63cc0ce450 msgr2=0x7f63cc0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:06.716 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:06.697 7f63d4621700 1 -- 172.21.2.102:0/2670218072 shutdown_connections 2021-02-10T11:30:06.716 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:06.697 7f63d4621700 1 -- 172.21.2.102:0/2670218072 wait complete. 2021-02-10T11:30:06.716 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:06.712 7f63d4621700 1 Processor -- start 2021-02-10T11:30:06.716 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:06.712 7f63d4621700 1 -- start start 2021-02-10T11:30:06.716 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:06.712 7f63d4621700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f63cc195650 0x7f63cc0b5560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:06.717 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:06.712 7f63d4621700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f63cc0b5b80 0x7f63cc1a2e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:06.717 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:06.712 7f63d4621700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f63cc0acfa0 0x7f63cc1a3440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:06.717 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:06.712 7f63d4621700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63cc0b6010 con 0x7f63cc0acfa0 2021-02-10T11:30:06.717 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:06.712 7f63d4621700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63cc1a3a60 con 0x7f63cc195650 2021-02-10T11:30:06.717 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:06.712 7f63d4621700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63cc1a3c40 con 0x7f63cc0b5b80 2021-02-10T11:30:06.717 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:06.712 7f63d2c3e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f63cc0acfa0 0x7f63cc1a3440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:06.717 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:06.712 7f63d2c3e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f63cc0acfa0 0x7f63cc1a3440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:60724/0 (socket says 172.21.2.102:60724) 2021-02-10T11:30:06.717 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:06.712 7f63d2c3e700 1 -- 172.21.2.102:0/2390900473 learned_addr learned my addr 172.21.2.102:0/2390900473 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:30:06.718 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:06.713 7f63d1c3c700 1 --2- 172.21.2.102:0/2390900473 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f63cc0b5b80 0x7f63cc1a2e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:06.718 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:06.713 7f63d243d700 1 --2- 172.21.2.102:0/2390900473 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f63cc195650 0x7f63cc0b5560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:06.719 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:06.713 7f63d2c3e700 1 -- 172.21.2.102:0/2390900473 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f63cc0b5b80 msgr2=0x7f63cc1a2e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:06.719 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:06.713 7f63d2c3e700 1 --2- 172.21.2.102:0/2390900473 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f63cc0b5b80 0x7f63cc1a2e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:06.719 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:06.713 7f63d2c3e700 1 -- 172.21.2.102:0/2390900473 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f63cc195650 msgr2=0x7f63cc0b5560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:06.719 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:06.713 7f63d2c3e700 1 --2- 172.21.2.102:0/2390900473 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f63cc195650 0x7f63cc0b5560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:06.719 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:06.713 7f63d2c3e700 1 -- 172.21.2.102:0/2390900473 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63cc0bcae0 con 0x7f63cc0acfa0 2021-02-10T11:30:06.719 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:06.713 7f63d2c3e700 1 --2- 172.21.2.102:0/2390900473 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f63cc0acfa0 0x7f63cc1a3440 secure :-1 s=READY pgs=1936 cs=0 l=1 rev1=1 rx=0x7f63c4007a00 tx=0x7f63c400acb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:06.720 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:06.713 7f63beffd700 1 -- 172.21.2.102:0/2390900473 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f63c4000eb0 con 0x7f63cc0acfa0 2021-02-10T11:30:06.720 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:06.713 7f63beffd700 1 -- 172.21.2.102:0/2390900473 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63c40070d0 con 0x7f63cc0acfa0 2021-02-10T11:30:06.720 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:06.713 7f63beffd700 1 -- 172.21.2.102:0/2390900473 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f63c4004930 con 0x7f63cc0acfa0 2021-02-10T11:30:06.720 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:06.713 7f63d4621700 1 -- 172.21.2.102:0/2390900473 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63cc0bcae0 con 0x7f63cc0acfa0 2021-02-10T11:30:06.720 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:06.713 7f63d4621700 1 -- 172.21.2.102:0/2390900473 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63cc0bd070 con 0x7f63cc0acfa0 2021-02-10T11:30:06.720 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:06.714 7f63beffd700 1 -- 172.21.2.102:0/2390900473 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f63c4001c60 con 0x7f63cc0acfa0 2021-02-10T11:30:06.720 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:06.714 7f63beffd700 1 --2- 172.21.2.102:0/2390900473 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f63b40186c0 0x7f63b401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:06.720 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:06.714 7f63beffd700 1 -- 172.21.2.102:0/2390900473 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f63c40329d0 con 0x7f63cc0acfa0 2021-02-10T11:30:06.720 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:06.714 7f63d243d700 1 --2- 172.21.2.102:0/2390900473 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f63b40186c0 0x7f63b401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:06.720 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:06.715 7f63d243d700 1 --2- 172.21.2.102:0/2390900473 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f63b40186c0 0x7f63b401abc0 crc :-1 s=READY pgs=868 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:06.745 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:06.744 7f63d4621700 1 -- 172.21.2.102:0/2390900473 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63cc0f0ab0 con 0x7f63cc0acfa0 2021-02-10T11:30:06.747 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:06.747 7f63beffd700 1 -- 172.21.2.102:0/2390900473 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f63c4000eb0 con 0x7f63cc0acfa0 2021-02-10T11:30:06.876 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:06.875 7f63d4621700 1 -- 172.21.2.102:0/2390900473 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f63cc0f0ab0 con 0x7f63cc0acfa0 2021-02-10T11:30:06.876 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:06.876 7f63beffd700 1 -- 172.21.2.102:0/2390900473 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) v1 ==== 94+0+2098 (secure 0 0 0) 0x7f63c4001c60 con 0x7f63cc0acfa0 2021-02-10T11:30:06.877 INFO:teuthology.orchestra.run.gibba002.stderr:dumped fsmap epoch 73 2021-02-10T11:30:06.877 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:30:06.877 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":73,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9672,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.102:6835/2071608955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6834","nonce":2071608955},{"type":"v1","addr":"172.21.2.102:6835","nonce":2071608955}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":69}],"filesystems":[{"mdsmap":{"epoch":73,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:30:04.225997","modified":"2021-02-10 11:30:06.232546","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7186},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7186":{"gid":7186,"name":"b","rank":0,"incarnation":70,"state":"up:active","state_seq":3,"addr":"172.21.2.113:6833/675492649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6832","nonce":675492649},{"type":"v1","addr":"172.21.2.113:6833","nonce":675492649}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":9}]} 2021-02-10T11:30:06.879 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:06.878 7f63d4621700 1 -- 172.21.2.102:0/2390900473 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f63b40186c0 msgr2=0x7f63b401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:06.879 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:06.878 7f63d4621700 1 --2- 172.21.2.102:0/2390900473 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f63b40186c0 0x7f63b401abc0 crc :-1 s=READY pgs=868 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:06.879 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:06.878 7f63d4621700 1 -- 172.21.2.102:0/2390900473 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f63cc0acfa0 msgr2=0x7f63cc1a3440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:06.879 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:06.878 7f63d4621700 1 --2- 172.21.2.102:0/2390900473 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f63cc0acfa0 0x7f63cc1a3440 secure :-1 s=READY pgs=1936 cs=0 l=1 rev1=1 rx=0x7f63c4007a00 tx=0x7f63c400acb0).stop 2021-02-10T11:30:06.879 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:06.878 7f63d4621700 1 -- 172.21.2.102:0/2390900473 shutdown_connections 2021-02-10T11:30:06.879 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:06.878 7f63d4621700 1 --2- 172.21.2.102:0/2390900473 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f63cc0acfa0 0x7f63cc1a3440 unknown :-1 s=CLOSED pgs=1936 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:06.880 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:06.878 7f63d4621700 1 --2- 172.21.2.102:0/2390900473 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f63cc0b5b80 0x7f63cc1a2e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:06.880 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:06.878 7f63d4621700 1 --2- 172.21.2.102:0/2390900473 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f63b40186c0 0x7f63b401abc0 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:06.880 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:06.878 7f63d4621700 1 --2- 172.21.2.102:0/2390900473 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f63cc195650 0x7f63cc0b5560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:06.880 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:06.878 7f63d4621700 1 -- 172.21.2.102:0/2390900473 >> 172.21.2.102:0/2390900473 conn(0x7f63cc0b3040 msgr2=0x7f63cc0c3250 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:06.880 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:06.879 7f63d4621700 1 -- 172.21.2.102:0/2390900473 shutdown_connections 2021-02-10T11:30:06.880 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:06.879 7f63d4621700 1 -- 172.21.2.102:0/2390900473 wait complete. 2021-02-10T11:30:06.887 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:30:07.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:07.298 7f5b6fbc2700 1 Processor -- start 2021-02-10T11:30:07.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:07.298 7f5b6fbc2700 1 -- start start 2021-02-10T11:30:07.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:07.299 7f5b6fbc2700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f5b681b0470 0x7f5b681b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:07.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:07.299 7f5b6fbc2700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f5b680f20e0 0x7f5b680f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:07.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:07.299 7f5b6fbc2700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b680f0b50 con 0x7f5b680ef610 2021-02-10T11:30:07.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:07.299 7f5b6fbc2700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b680f0820 con 0x7f5b680f20e0 2021-02-10T11:30:07.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:07.299 7f5b6fbc2700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b680f24e0 con 0x7f5b681b0470 2021-02-10T11:30:07.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:07.299 7f5b6e1df700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f5b680f20e0 0x7f5b680f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:07.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:07.299 7f5b6e1df700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f5b680f20e0 0x7f5b680f0200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:60734/0 (socket says 172.21.2.102:60734) 2021-02-10T11:30:07.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:07.299 7f5b6e1df700 1 -- 172.21.2.102:0/1967709689 learned_addr learned my addr 172.21.2.102:0/1967709689 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:30:07.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:07.299 7f5b6d9de700 1 --2- 172.21.2.102:0/1967709689 >> v2:172.21.2.113:3300/0 conn(0x7f5b681b0470 0x7f5b681b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:07.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:07.300 7f5b6e1df700 1 -- 172.21.2.102:0/1967709689 >> v2:172.21.2.113:3300/0 conn(0x7f5b681b0470 msgr2=0x7f5b681b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:07.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:07.300 7f5b6e1df700 1 --2- 172.21.2.102:0/1967709689 >> v2:172.21.2.113:3300/0 conn(0x7f5b681b0470 0x7f5b681b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:07.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:07.300 7f5b6e1df700 1 -- 172.21.2.102:0/1967709689 >> v1:172.21.2.113:6789/0 conn(0x7f5b680ef610 legacy=0x7f5b680efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:07.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:07.300 7f5b6e1df700 1 -- 172.21.2.102:0/1967709689 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b680e52e0 con 0x7f5b680f20e0 2021-02-10T11:30:07.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:07.300 7f5b6e1df700 1 --2- 172.21.2.102:0/1967709689 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5b680f20e0 0x7f5b680f0200 secure :-1 s=READY pgs=1937 cs=0 l=1 rev1=1 rx=0x7f5b60006f00 tx=0x7f5b60009f10).ready entity=mon.0 client_cookie=a452862eba874aaf server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:07.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:07.300 7f5b6c9dc700 1 -- 172.21.2.102:0/1967709689 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5b6000b3b0 con 0x7f5b680f20e0 2021-02-10T11:30:07.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:07.300 7f5b6c9dc700 1 -- 172.21.2.102:0/1967709689 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b6000b730 con 0x7f5b680f20e0 2021-02-10T11:30:07.318 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:07.300 7f5b6c9dc700 1 -- 172.21.2.102:0/1967709689 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5b6000bc50 con 0x7f5b680f20e0 2021-02-10T11:30:07.318 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:07.300 7f5b6fbc2700 1 -- 172.21.2.102:0/1967709689 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5b680f20e0 msgr2=0x7f5b680f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:07.318 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:07.300 7f5b6fbc2700 1 --2- 172.21.2.102:0/1967709689 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5b680f20e0 0x7f5b680f0200 secure :-1 s=READY pgs=1937 cs=0 l=1 rev1=1 rx=0x7f5b60006f00 tx=0x7f5b60009f10).stop 2021-02-10T11:30:07.318 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:07.301 7f5b6fbc2700 1 -- 172.21.2.102:0/1967709689 shutdown_connections 2021-02-10T11:30:07.318 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:07.301 7f5b6fbc2700 1 --2- 172.21.2.102:0/1967709689 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5b680f20e0 0x7f5b680f0200 unknown :-1 s=CLOSED pgs=1937 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:07.318 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:07.301 7f5b6fbc2700 1 --2- 172.21.2.102:0/1967709689 >> v2:172.21.2.113:3300/0 conn(0x7f5b681b0470 0x7f5b681b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:07.318 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:07.301 7f5b6fbc2700 1 -- 172.21.2.102:0/1967709689 >> 172.21.2.102:0/1967709689 conn(0x7f5b680ce3e0 msgr2=0x7f5b680e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:07.318 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:07.301 7f5b6fbc2700 1 -- 172.21.2.102:0/1967709689 shutdown_connections 2021-02-10T11:30:07.318 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:07.301 7f5b6fbc2700 1 -- 172.21.2.102:0/1967709689 wait complete. 2021-02-10T11:30:07.320 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:07.316 7f5b6fbc2700 1 Processor -- start 2021-02-10T11:30:07.320 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:07.316 7f5b6fbc2700 1 -- start start 2021-02-10T11:30:07.320 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:07.316 7f5b6fbc2700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5b6808f9d0 0x7f5b6808fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:07.320 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:07.316 7f5b6fbc2700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5b680903f0 0x7f5b680907f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:07.320 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:07.316 7f5b6fbc2700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5b6808a970 0x7f5b6808ad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:07.320 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:07.316 7f5b6fbc2700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b6808b390 con 0x7f5b6808a970 2021-02-10T11:30:07.320 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:07.316 7f5b6fbc2700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b6808b5b0 con 0x7f5b680903f0 2021-02-10T11:30:07.320 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:07.316 7f5b6fbc2700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b6808b7d0 con 0x7f5b6808f9d0 2021-02-10T11:30:07.320 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:07.316 7f5b6e1df700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5b6808a970 0x7f5b6808ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:07.320 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:07.316 7f5b6e1df700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5b6808a970 0x7f5b6808ad70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:60736/0 (socket says 172.21.2.102:60736) 2021-02-10T11:30:07.321 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:07.316 7f5b6e1df700 1 -- 172.21.2.102:0/545238324 learned_addr learned my addr 172.21.2.102:0/545238324 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:30:07.321 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:07.317 7f5b6e1df700 1 -- 172.21.2.102:0/545238324 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5b6808f9d0 msgr2=0x7f5b6808fdd0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:30:07.321 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:07.317 7f5b6e1df700 1 --2- 172.21.2.102:0/545238324 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5b6808f9d0 0x7f5b6808fdd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:07.321 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:07.317 7f5b6e1df700 1 -- 172.21.2.102:0/545238324 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5b680903f0 msgr2=0x7f5b680907f0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:30:07.321 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:07.317 7f5b6e1df700 1 --2- 172.21.2.102:0/545238324 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5b680903f0 0x7f5b680907f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:07.321 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:07.317 7f5b6e1df700 1 -- 172.21.2.102:0/545238324 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b6808b9f0 con 0x7f5b6808a970 2021-02-10T11:30:07.321 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:07.317 7f5b6e1df700 1 --2- 172.21.2.102:0/545238324 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5b6808a970 0x7f5b6808ad70 secure :-1 s=READY pgs=1938 cs=0 l=1 rev1=1 rx=0x7f5b60009390 tx=0x7f5b6000bd40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:07.321 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:07.317 7f5b5a7fc700 1 -- 172.21.2.102:0/545238324 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5b60003030 con 0x7f5b6808a970 2021-02-10T11:30:07.321 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:07.317 7f5b5a7fc700 1 -- 172.21.2.102:0/545238324 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b600068e0 con 0x7f5b6808a970 2021-02-10T11:30:07.321 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:07.317 7f5b5a7fc700 1 -- 172.21.2.102:0/545238324 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5b60001700 con 0x7f5b6808a970 2021-02-10T11:30:07.321 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:07.317 7f5b6fbc2700 1 -- 172.21.2.102:0/545238324 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b6808b9f0 con 0x7f5b6808a970 2021-02-10T11:30:07.321 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:07.317 7f5b6fbc2700 1 -- 172.21.2.102:0/545238324 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b6808b9f0 con 0x7f5b6808a970 2021-02-10T11:30:07.322 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:07.318 7f5b5a7fc700 1 -- 172.21.2.102:0/545238324 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f5b60001270 con 0x7f5b6808a970 2021-02-10T11:30:07.322 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:07.318 7f5b5a7fc700 1 --2- 172.21.2.102:0/545238324 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5b500186c0 0x7f5b5001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:07.322 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:07.318 7f5b5a7fc700 1 -- 172.21.2.102:0/545238324 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f5b60032910 con 0x7f5b6808a970 2021-02-10T11:30:07.322 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:07.318 7f5b6d9de700 1 --2- 172.21.2.102:0/545238324 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5b500186c0 0x7f5b5001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:07.322 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:07.318 7f5b6d9de700 1 --2- 172.21.2.102:0/545238324 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5b500186c0 0x7f5b5001abc0 crc :-1 s=READY pgs=869 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:07.349 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:07.349 7f5b6fbc2700 1 -- 172.21.2.102:0/545238324 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b68090ef0 con 0x7f5b6808a970 2021-02-10T11:30:07.351 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:07.351 7f5b5a7fc700 1 -- 172.21.2.102:0/545238324 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f5b60000cc0 con 0x7f5b6808a970 2021-02-10T11:30:07.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:07.475 7f5b6fbc2700 1 -- 172.21.2.102:0/545238324 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5b68090ef0 con 0x7f5b6808a970 2021-02-10T11:30:07.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:07.476 7f5b5a7fc700 1 -- 172.21.2.102:0/545238324 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) v1 ==== 74+0+13996 (secure 0 0 0) 0x7f5b60001270 con 0x7f5b6808a970 2021-02-10T11:30:07.477 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:30:07.477 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":93,"fsid":"68db30ea-dbce-4aef-a6df-f01c6fea0513","created":"2021-02-10 11:03:06.331557","modified":"2021-02-10 11:30:05.228248","last_up_change":"2021-02-10 11:03:32.690562","last_in_change":"2021-02-10 11:03:32.690562","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:03:37.322519","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:30:02.187273","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":19,"pool_name":"cephfs_data","create_time":"2021-02-10 11:30:03.219690","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"93","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"be691b4d-a039-408a-af80-9d53f02b6f1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6818","nonce":36088},{"type":"v1","addr":"172.21.2.102:6820","nonce":36088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6822","nonce":36088},{"type":"v1","addr":"172.21.2.102:6824","nonce":36088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6830","nonce":36088},{"type":"v1","addr":"172.21.2.102:6832","nonce":36088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6826","nonce":36088},{"type":"v1","addr":"172.21.2.102:6828","nonce":36088}]},"public_addr":"172.21.2.102:6820/36088","cluster_addr":"172.21.2.102:6824/36088","heartbeat_back_addr":"172.21.2.102:6832/36088","heartbeat_front_addr":"172.21.2.102:6828/36088","state":["exists","up"]},{"osd":1,"uuid":"64e3ccc8-9c69-4d7c-865e-af8e83822635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6819","nonce":36087},{"type":"v1","addr":"172.21.2.102:6821","nonce":36087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6823","nonce":36087},{"type":"v1","addr":"172.21.2.102:6825","nonce":36087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6831","nonce":36087},{"type":"v1","addr":"172.21.2.102:6833","nonce":36087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6827","nonce":36087},{"type":"v1","addr":"172.21.2.102:6829","nonce":36087}]},"public_addr":"172.21.2.102:6821/36087","cluster_addr":"172.21.2.102:6825/36087","heartbeat_back_addr":"172.21.2.102:6833/36087","heartbeat_front_addr":"172.21.2.102:6829/36087","state":["exists","up"]},{"osd":2,"uuid":"153700af-747f-481f-bdf4-d6072d3a842f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6810","nonce":36089},{"type":"v1","addr":"172.21.2.102:6811","nonce":36089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6812","nonce":36089},{"type":"v1","addr":"172.21.2.102:6813","nonce":36089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6816","nonce":36089},{"type":"v1","addr":"172.21.2.102:6817","nonce":36089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6814","nonce":36089},{"type":"v1","addr":"172.21.2.102:6815","nonce":36089}]},"public_addr":"172.21.2.102:6811/36089","cluster_addr":"172.21.2.102:6813/36089","heartbeat_back_addr":"172.21.2.102:6817/36089","heartbeat_front_addr":"172.21.2.102:6815/36089","state":["exists","up"]},{"osd":3,"uuid":"cbc678f4-83a9-45e6-8830-2f7c232b1ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6802","nonce":36090},{"type":"v1","addr":"172.21.2.102:6803","nonce":36090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6804","nonce":36090},{"type":"v1","addr":"172.21.2.102:6805","nonce":36090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6808","nonce":36090},{"type":"v1","addr":"172.21.2.102:6809","nonce":36090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6806","nonce":36090},{"type":"v1","addr":"172.21.2.102:6807","nonce":36090}]},"public_addr":"172.21.2.102:6803/36090","cluster_addr":"172.21.2.102:6805/36090","heartbeat_back_addr":"172.21.2.102:6809/36090","heartbeat_front_addr":"172.21.2.102:6807/36090","state":["exists","up"]},{"osd":4,"uuid":"c01e4008-b258-4f10-afd2-9e7f37214351","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6808","nonce":36368},{"type":"v1","addr":"172.21.2.113:6811","nonce":36368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6814","nonce":36368},{"type":"v1","addr":"172.21.2.113:6818","nonce":36368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6826","nonce":36368},{"type":"v1","addr":"172.21.2.113:6828","nonce":36368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6820","nonce":36368},{"type":"v1","addr":"172.21.2.113:6824","nonce":36368}]},"public_addr":"172.21.2.113:6811/36368","cluster_addr":"172.21.2.113:6818/36368","heartbeat_back_addr":"172.21.2.113:6828/36368","heartbeat_front_addr":"172.21.2.113:6824/36368","state":["exists","up"]},{"osd":5,"uuid":"551f6007-daf8-4a98-8ee0-c2b16bcccb5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6802","nonce":36366},{"type":"v1","addr":"172.21.2.113:6805","nonce":36366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6807","nonce":36366},{"type":"v1","addr":"172.21.2.113:6810","nonce":36366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6819","nonce":36366},{"type":"v1","addr":"172.21.2.113:6822","nonce":36366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6812","nonce":36366},{"type":"v1","addr":"172.21.2.113:6815","nonce":36366}]},"public_addr":"172.21.2.113:6805/36366","cluster_addr":"172.21.2.113:6810/36366","heartbeat_back_addr":"172.21.2.113:6822/36366","heartbeat_front_addr":"172.21.2.113:6815/36366","state":["exists","up"]},{"osd":6,"uuid":"bf67c040-0e50-48be-b102-c3943bc9ebef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6800","nonce":36367},{"type":"v1","addr":"172.21.2.113:6801","nonce":36367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6803","nonce":36367},{"type":"v1","addr":"172.21.2.113:6804","nonce":36367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6813","nonce":36367},{"type":"v1","addr":"172.21.2.113:6816","nonce":36367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6806","nonce":36367},{"type":"v1","addr":"172.21.2.113:6809","nonce":36367}]},"public_addr":"172.21.2.113:6801/36367","cluster_addr":"172.21.2.113:6804/36367","heartbeat_back_addr":"172.21.2.113:6816/36367","heartbeat_front_addr":"172.21.2.113:6809/36367","state":["exists","up"]},{"osd":7,"uuid":"f5461f80-018d-428f-be89-2eb2bcbe7bb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6817","nonce":36365},{"type":"v1","addr":"172.21.2.113:6821","nonce":36365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6823","nonce":36365},{"type":"v1","addr":"172.21.2.113:6825","nonce":36365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6830","nonce":36365},{"type":"v1","addr":"172.21.2.113:6831","nonce":36365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6827","nonce":36365},{"type":"v1","addr":"172.21.2.113:6829","nonce":36365}]},"public_addr":"172.21.2.113:6821/36365","cluster_addr":"172.21.2.113:6825/36365","heartbeat_back_addr":"172.21.2.113:6831/36365","heartbeat_front_addr":"172.21.2.113:6829/36365","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:30:07.478 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:07.478 7f5b6fbc2700 1 -- 172.21.2.102:0/545238324 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5b500186c0 msgr2=0x7f5b5001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:07.478 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:07.478 7f5b6fbc2700 1 --2- 172.21.2.102:0/545238324 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5b500186c0 0x7f5b5001abc0 crc :-1 s=READY pgs=869 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:07.479 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:07.478 7f5b6fbc2700 1 -- 172.21.2.102:0/545238324 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5b6808a970 msgr2=0x7f5b6808ad70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:07.479 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:07.478 7f5b6fbc2700 1 --2- 172.21.2.102:0/545238324 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5b6808a970 0x7f5b6808ad70 secure :-1 s=READY pgs=1938 cs=0 l=1 rev1=1 rx=0x7f5b60009390 tx=0x7f5b6000bd40).stop 2021-02-10T11:30:07.479 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:07.478 7f5b6fbc2700 1 -- 172.21.2.102:0/545238324 shutdown_connections 2021-02-10T11:30:07.479 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:07.478 7f5b6fbc2700 1 --2- 172.21.2.102:0/545238324 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5b6808a970 0x7f5b6808ad70 unknown :-1 s=CLOSED pgs=1938 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:07.479 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:07.478 7f5b6fbc2700 1 --2- 172.21.2.102:0/545238324 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5b500186c0 0x7f5b5001abc0 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:07.479 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:07.478 7f5b6fbc2700 1 --2- 172.21.2.102:0/545238324 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5b680903f0 0x7f5b680907f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:07.479 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:07.478 7f5b6fbc2700 1 --2- 172.21.2.102:0/545238324 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5b6808f9d0 0x7f5b6808fdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:07.479 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:07.478 7f5b6fbc2700 1 -- 172.21.2.102:0/545238324 >> 172.21.2.102:0/545238324 conn(0x7f5b6819f7c0 msgr2=0x7f5b681b6b60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:07.479 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:07.478 7f5b6fbc2700 1 -- 172.21.2.102:0/545238324 shutdown_connections 2021-02-10T11:30:07.479 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:07.478 7f5b6fbc2700 1 -- 172.21.2.102:0/545238324 wait complete. 2021-02-10T11:30:07.487 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:30:07.929 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:07.911 7f376a244700 1 Processor -- start 2021-02-10T11:30:07.930 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:07.911 7f376a244700 1 -- start start 2021-02-10T11:30:07.930 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:07.911 7f376a244700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f37640d1880 0x7f37641b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:07.930 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:07.911 7f376a244700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f37641b0af0 0x7f3764108950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:07.930 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:07.911 7f376a244700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3764109a70 con 0x7f37640d1c80 2021-02-10T11:30:07.930 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:07.911 7f376a244700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37640e5f00 con 0x7f37641b0af0 2021-02-10T11:30:07.930 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:07.911 7f376a244700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3764109760 con 0x7f37640d1880 2021-02-10T11:30:07.930 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:07.912 7f3768861700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f37640d1c80 0x7f3764108f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:52970/0 (socket says 172.21.2.102:52970) 2021-02-10T11:30:07.930 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:07.912 7f3768861700 1 -- 172.21.2.102:0/2166650278 learned_addr learned my addr 172.21.2.102:0/2166650278 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:30:07.930 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:07.912 7f3762ffd700 1 -- 172.21.2.102:0/2166650278 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3635426955 0 0) 0x7f375c001180 con 0x7f37640d1c80 2021-02-10T11:30:07.930 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:07.912 7f3762ffd700 1 -- 172.21.2.102:0/2166650278 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3750002fe0 con 0x7f37640d1c80 2021-02-10T11:30:07.930 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:07.912 7f3763fff700 1 --2- 172.21.2.102:0/2166650278 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f37640d1880 0x7f37641b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:07.931 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:07.912 7f37637fe700 1 --2- 172.21.2.102:0/2166650278 >> v2:172.21.2.113:3300/0 conn(0x7f37641b0af0 0x7f3764108950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:07.931 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:07.912 7f3762ffd700 1 -- 172.21.2.102:0/2166650278 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3318800477 0 0) 0x7f375c001180 con 0x7f37640d1c80 2021-02-10T11:30:07.931 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:07.912 7f3762ffd700 1 -- 172.21.2.102:0/2166650278 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f37640d1880 msgr2=0x7f37641b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:07.931 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:07.912 7f3762ffd700 1 --2- 172.21.2.102:0/2166650278 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f37640d1880 0x7f37641b04d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:07.931 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:07.912 7f3762ffd700 1 -- 172.21.2.102:0/2166650278 >> v2:172.21.2.113:3300/0 conn(0x7f37641b0af0 msgr2=0x7f3764108950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:07.931 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:07.912 7f3762ffd700 1 --2- 172.21.2.102:0/2166650278 >> v2:172.21.2.113:3300/0 conn(0x7f37641b0af0 0x7f3764108950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:07.931 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:07.912 7f3762ffd700 1 -- 172.21.2.102:0/2166650278 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3764088070 con 0x7f37640d1c80 2021-02-10T11:30:07.931 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:07.912 7f3762ffd700 1 -- 172.21.2.102:0/2166650278 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7f375c001680 con 0x7f37640d1c80 2021-02-10T11:30:07.931 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:07.912 7f3762ffd700 1 -- 172.21.2.102:0/2166650278 >> v1:172.21.2.102:6789/0 conn(0x7f37640d1c80 legacy=0x7f3764108f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:07.931 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:07.912 7f3762ffd700 1 --2- 172.21.2.102:0/2166650278 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3750003b90 0x7f37500060b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:07.931 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:07.912 7f3762ffd700 1 --2- 172.21.2.102:0/2166650278 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f37500068d0 0x7f3750008dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:07.931 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:07.912 7f3762ffd700 1 --2- 172.21.2.102:0/2166650278 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3750009660 0x7f375000bb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:07.931 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:07.912 7f3762ffd700 1 -- 172.21.2.102:0/2166650278 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f375000c410 con 0x7f3750003b90 2021-02-10T11:30:07.932 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:07.912 7f3762ffd700 1 -- 172.21.2.102:0/2166650278 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f375000c630 con 0x7f3750009660 2021-02-10T11:30:07.932 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:07.912 7f3762ffd700 1 -- 172.21.2.102:0/2166650278 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f375000c850 con 0x7f37500068d0 2021-02-10T11:30:07.932 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:07.912 7f3763fff700 1 --2- 172.21.2.102:0/2166650278 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f37640d1880 0x7f37641b04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:30:07.932 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:07.912 7f37637fe700 1 --2- 172.21.2.102:0/2166650278 >> v2:172.21.2.113:3300/0 conn(0x7f37641b0af0 0x7f3764108950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:30:07.932 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:07.913 7f3763fff700 1 --2- 172.21.2.102:0/2166650278 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3750003b90 0x7f37500060b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:07.932 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:07.913 7f3768861700 1 --2- 172.21.2.102:0/2166650278 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3750009660 0x7f375000bb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:07.932 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:07.913 7f37637fe700 1 --2- 172.21.2.102:0/2166650278 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f37500068d0 0x7f3750008dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:07.932 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:07.913 7f3763fff700 1 -- 172.21.2.102:0/2166650278 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f37500068d0 msgr2=0x7f3750008dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:07.932 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:07.913 7f3763fff700 1 --2- 172.21.2.102:0/2166650278 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f37500068d0 0x7f3750008dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:07.932 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:07.913 7f3763fff700 1 -- 172.21.2.102:0/2166650278 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3750009660 msgr2=0x7f375000bb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:07.932 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:07.913 7f3763fff700 1 --2- 172.21.2.102:0/2166650278 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3750009660 0x7f375000bb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:07.932 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:07.913 7f3763fff700 1 -- 172.21.2.102:0/2166650278 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f375000caf0 con 0x7f3750003b90 2021-02-10T11:30:07.932 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:07.913 7f37637fe700 1 --2- 172.21.2.102:0/2166650278 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f37500068d0 0x7f3750008dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:30:07.933 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:07.913 7f3763fff700 1 --2- 172.21.2.102:0/2166650278 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3750003b90 0x7f37500060b0 secure :-1 s=READY pgs=1940 cs=0 l=1 rev1=1 rx=0x7f3754003180 tx=0x7f375404edf0).ready entity=mon.0 client_cookie=84db96c4c5a2761f server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:07.933 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:07.913 7f3762ffd700 1 -- 172.21.2.102:0/2166650278 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3754006030 con 0x7f3750003b90 2021-02-10T11:30:07.933 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:07.913 7f3762ffd700 1 -- 172.21.2.102:0/2166650278 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3754006950 con 0x7f3750003b90 2021-02-10T11:30:07.933 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:07.913 7f3762ffd700 1 -- 172.21.2.102:0/2166650278 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f37540062b0 con 0x7f3750003b90 2021-02-10T11:30:07.933 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:07.913 7f376a244700 1 -- 172.21.2.102:0/2166650278 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3750003b90 msgr2=0x7f37500060b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:07.933 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:07.913 7f376a244700 1 --2- 172.21.2.102:0/2166650278 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3750003b90 0x7f37500060b0 secure :-1 s=READY pgs=1940 cs=0 l=1 rev1=1 rx=0x7f3754003180 tx=0x7f375404edf0).stop 2021-02-10T11:30:07.933 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:07.913 7f3768861700 1 -- 172.21.2.102:0/2166650278 reap_dead start 2021-02-10T11:30:07.933 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:07.914 7f376a244700 1 -- 172.21.2.102:0/2166650278 shutdown_connections 2021-02-10T11:30:07.933 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:07.914 7f376a244700 1 -- 172.21.2.102:0/2166650278 >> 172.21.2.102:0/2166650278 conn(0x7f37640ce430 msgr2=0x7f37640d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:07.933 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:07.914 7f376a244700 1 -- 172.21.2.102:0/2166650278 shutdown_connections 2021-02-10T11:30:07.933 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:07.914 7f376a244700 1 -- 172.21.2.102:0/2166650278 wait complete. 2021-02-10T11:30:07.933 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:07.929 7f376a244700 1 Processor -- start 2021-02-10T11:30:07.933 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:07.929 7f376a244700 1 -- start start 2021-02-10T11:30:07.935 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:07.929 7f376a244700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f37641a0580 0x7f3764086fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:07.935 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:07.929 7f376a244700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f37640b66b0 0x7f37640b6ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:07.935 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:07.929 7f376a244700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3764082bb0 0x7f3764082fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:07.935 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:07.929 7f376a244700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37640875d0 con 0x7f3764082bb0 2021-02-10T11:30:07.935 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:07.929 7f376a244700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37640835d0 con 0x7f37641a0580 2021-02-10T11:30:07.935 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:07.929 7f376a244700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37640837f0 con 0x7f37640b66b0 2021-02-10T11:30:07.935 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:07.929 7f3768861700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3764082bb0 0x7f3764082fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:07.935 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:07.929 7f3768861700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3764082bb0 0x7f3764082fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:60758/0 (socket says 172.21.2.102:60758) 2021-02-10T11:30:07.935 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:07.929 7f3768861700 1 -- 172.21.2.102:0/1747351192 learned_addr learned my addr 172.21.2.102:0/1747351192 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:30:07.935 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:07.930 7f3768861700 1 -- 172.21.2.102:0/1747351192 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f37640b66b0 msgr2=0x7f37640b6ab0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:30:07.936 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:07.930 7f3768861700 1 --2- 172.21.2.102:0/1747351192 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f37640b66b0 0x7f37640b6ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:07.936 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:07.930 7f3768861700 1 -- 172.21.2.102:0/1747351192 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f37641a0580 msgr2=0x7f3764086fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:07.936 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:07.930 7f3768861700 1 --2- 172.21.2.102:0/1747351192 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f37641a0580 0x7f3764086fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:07.936 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:07.930 7f3768861700 1 -- 172.21.2.102:0/1747351192 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3764083a10 con 0x7f3764082bb0 2021-02-10T11:30:07.936 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:07.930 7f3768861700 1 --2- 172.21.2.102:0/1747351192 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3764082bb0 0x7f3764082fb0 secure :-1 s=READY pgs=1941 cs=0 l=1 rev1=1 rx=0x7f375c00b690 tx=0x7f375c00ba30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:07.936 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:07.930 7f3760ff9700 1 -- 172.21.2.102:0/1747351192 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f375c007b80 con 0x7f3764082bb0 2021-02-10T11:30:07.936 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:07.930 7f3760ff9700 1 -- 172.21.2.102:0/1747351192 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f375c00be20 con 0x7f3764082bb0 2021-02-10T11:30:07.936 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:07.930 7f376a244700 1 -- 172.21.2.102:0/1747351192 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3764083a10 con 0x7f3764082bb0 2021-02-10T11:30:07.936 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:07.930 7f376a244700 1 -- 172.21.2.102:0/1747351192 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3764083fa0 con 0x7f3764082bb0 2021-02-10T11:30:07.936 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:07.930 7f3760ff9700 1 -- 172.21.2.102:0/1747351192 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f375c004590 con 0x7f3764082bb0 2021-02-10T11:30:07.936 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:07.931 7f3760ff9700 1 -- 172.21.2.102:0/1747351192 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f375c0043a0 con 0x7f3764082bb0 2021-02-10T11:30:07.936 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:07.931 7f3760ff9700 1 --2- 172.21.2.102:0/1747351192 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f37480186d0 0x7f374801abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:07.936 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:07.931 7f3760ff9700 1 -- 172.21.2.102:0/1747351192 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f375c0327e0 con 0x7f3764082bb0 2021-02-10T11:30:07.937 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:07.931 7f3763fff700 1 --2- 172.21.2.102:0/1747351192 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f37480186d0 0x7f374801abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:07.937 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:07.931 7f3763fff700 1 --2- 172.21.2.102:0/1747351192 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f37480186d0 0x7f374801abd0 crc :-1 s=READY pgs=870 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:07.961 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:07.961 7f376a244700 1 -- 172.21.2.102:0/1747351192 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3764082980 con 0x7f3764082bb0 2021-02-10T11:30:07.964 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:07.963 7f3760ff9700 1 -- 172.21.2.102:0/1747351192 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f375c006030 con 0x7f3764082bb0 2021-02-10T11:30:08.096 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:08.096 7f376a244700 1 -- 172.21.2.102:0/1747351192 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f3764082980 con 0x7f3764082bb0 2021-02-10T11:30:08.100 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:08.100 7f3760ff9700 1 -- 172.21.2.102:0/1747351192 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.0 v51) v1 ==== 184+0+0 (secure 0 0 0) 0x7f375c0043a0 con 0x7f3764082bb0 2021-02-10T11:30:08.103 INFO:teuthology.orchestra.run.gibba002.stderr:updated caps for client.0 2021-02-10T11:30:08.105 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:08.105 7f376a244700 1 -- 172.21.2.102:0/1747351192 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f37480186d0 msgr2=0x7f374801abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:08.105 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:08.105 7f376a244700 1 --2- 172.21.2.102:0/1747351192 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f37480186d0 0x7f374801abd0 crc :-1 s=READY pgs=870 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:08.105 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:08.105 7f376a244700 1 -- 172.21.2.102:0/1747351192 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3764082bb0 msgr2=0x7f3764082fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:08.105 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:08.105 7f376a244700 1 --2- 172.21.2.102:0/1747351192 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3764082bb0 0x7f3764082fb0 secure :-1 s=READY pgs=1941 cs=0 l=1 rev1=1 rx=0x7f375c00b690 tx=0x7f375c00ba30).stop 2021-02-10T11:30:08.106 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:08.105 7f376a244700 1 -- 172.21.2.102:0/1747351192 shutdown_connections 2021-02-10T11:30:08.106 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:08.105 7f376a244700 1 --2- 172.21.2.102:0/1747351192 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3764082bb0 0x7f3764082fb0 unknown :-1 s=CLOSED pgs=1941 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:08.106 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:08.105 7f376a244700 1 --2- 172.21.2.102:0/1747351192 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f37640b66b0 0x7f37640b6ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:08.106 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:08.105 7f376a244700 1 --2- 172.21.2.102:0/1747351192 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f37480186d0 0x7f374801abd0 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:08.106 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:08.105 7f376a244700 1 --2- 172.21.2.102:0/1747351192 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f37641a0580 0x7f3764086fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:08.106 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:08.105 7f376a244700 1 -- 172.21.2.102:0/1747351192 >> 172.21.2.102:0/1747351192 conn(0x7f37640afc60 msgr2=0x7f37640b50d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:08.106 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:08.105 7f376a244700 1 -- 172.21.2.102:0/1747351192 shutdown_connections 2021-02-10T11:30:08.106 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:08.105 7f376a244700 1 -- 172.21.2.102:0/1747351192 wait complete. 2021-02-10T11:30:08.114 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.1 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:30:08.303 DEBUG:teuthology.orchestra.run.gibba002:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:30:08.307 DEBUG:teuthology.orchestra.run.gibba013:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:30:08.553 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:08.535 7fa15048e700 1 Processor -- start 2021-02-10T11:30:08.553 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:08.535 7fa15048e700 1 -- start start 2021-02-10T11:30:08.553 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:08.536 7fa15048e700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa14810b000 0x7fa1480cf760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:08.553 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:08.536 7fa15048e700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1480fe1d0 con 0x7fa1480d1c80 2021-02-10T11:30:08.553 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:08.536 7fa15048e700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1481b0cc0 con 0x7fa1480d1880 2021-02-10T11:30:08.554 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:08.536 7fa15048e700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa148109a70 con 0x7fa14810b000 2021-02-10T11:30:08.554 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:08.536 7fa14daa9700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fa1480d1c80 0x7fa1480cef70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:52988/0 (socket says 172.21.2.102:52988) 2021-02-10T11:30:08.554 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:08.536 7fa14daa9700 1 -- 172.21.2.102:0/4264155237 learned_addr learned my addr 172.21.2.102:0/4264155237 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:30:08.554 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:08.536 7fa14eaab700 1 --2- 172.21.2.102:0/4264155237 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa14810b000 0x7fa1480cf760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:08.554 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:08.536 7fa14d2a8700 1 -- 172.21.2.102:0/4264155237 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 134526541 0 0) 0x7fa144001180 con 0x7fa1480d1c80 2021-02-10T11:30:08.554 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:08.536 7fa14d2a8700 1 -- 172.21.2.102:0/4264155237 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa134002fe0 con 0x7fa1480d1c80 2021-02-10T11:30:08.554 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:08.537 7fa14d2a8700 1 -- 172.21.2.102:0/4264155237 <== mon.1 v1:172.21.2.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1431266435 0 0) 0x7fa13c001080 con 0x7fa1480d1880 2021-02-10T11:30:08.554 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:08.537 7fa14d2a8700 1 -- 172.21.2.102:0/4264155237 --> v1:172.21.2.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa134004370 con 0x7fa1480d1880 2021-02-10T11:30:08.554 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:08.537 7fa14eaab700 1 -- 172.21.2.102:0/4264155237 >> v1:172.21.2.113:6789/0 conn(0x7fa1480d1880 legacy=0x7fa1481b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:08.554 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:08.537 7fa14eaab700 1 -- 172.21.2.102:0/4264155237 >> v1:172.21.2.102:6789/0 conn(0x7fa1480d1c80 legacy=0x7fa1480cef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:08.554 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:08.537 7fa14eaab700 1 -- 172.21.2.102:0/4264155237 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa14810b400 con 0x7fa14810b000 2021-02-10T11:30:08.554 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:08.537 7fa14eaab700 1 --2- 172.21.2.102:0/4264155237 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa14810b000 0x7fa1480cf760 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7fa140006e50 tx=0x7fa140009e70).ready entity=mon.2 client_cookie=37dfe3893827dedc server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:08.555 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:08.537 7fa14d2a8700 1 -- 172.21.2.102:0/4264155237 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa14000b510 con 0x7fa14810b000 2021-02-10T11:30:08.555 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:08.537 7fa14d2a8700 1 -- 172.21.2.102:0/4264155237 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa140006790 con 0x7fa14810b000 2021-02-10T11:30:08.555 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:08.537 7fa14d2a8700 1 -- 172.21.2.102:0/4264155237 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa14000b940 con 0x7fa14810b000 2021-02-10T11:30:08.555 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:08.537 7fa15048e700 1 -- 172.21.2.102:0/4264155237 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa14810b000 msgr2=0x7fa1480cf760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:08.555 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:08.537 7fa15048e700 1 --2- 172.21.2.102:0/4264155237 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa14810b000 0x7fa1480cf760 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7fa140006e50 tx=0x7fa140009e70).stop 2021-02-10T11:30:08.555 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:08.538 7fa15048e700 1 -- 172.21.2.102:0/4264155237 shutdown_connections 2021-02-10T11:30:08.555 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:08.538 7fa15048e700 1 --2- 172.21.2.102:0/4264155237 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa14810b000 0x7fa1480cf760 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:08.555 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:08.538 7fa15048e700 1 -- 172.21.2.102:0/4264155237 >> 172.21.2.102:0/4264155237 conn(0x7fa1480ce430 msgr2=0x7fa1480d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:08.555 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:08.538 7fa15048e700 1 -- 172.21.2.102:0/4264155237 shutdown_connections 2021-02-10T11:30:08.555 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:08.538 7fa15048e700 1 -- 172.21.2.102:0/4264155237 wait complete. 2021-02-10T11:30:08.557 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:08.553 7fa15048e700 1 Processor -- start 2021-02-10T11:30:08.557 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:08.553 7fa15048e700 1 -- start start 2021-02-10T11:30:08.557 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:08.553 7fa15048e700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa148095600 0x7fa148095a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:08.557 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:08.553 7fa15048e700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa148096020 0x7fa148096420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:08.557 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:08.553 7fa15048e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa148092a10 0x7fa148092e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:08.557 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:08.553 7fa15048e700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa148094100 con 0x7fa148092a10 2021-02-10T11:30:08.557 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:08.553 7fa15048e700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa14819e9e0 con 0x7fa148096020 2021-02-10T11:30:08.557 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:08.553 7fa15048e700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa148093430 con 0x7fa148095600 2021-02-10T11:30:08.557 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:08.554 7fa14eaab700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa148092a10 0x7fa148092e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:08.557 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:08.554 7fa14eaab700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa148092a10 0x7fa148092e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:60772/0 (socket says 172.21.2.102:60772) 2021-02-10T11:30:08.557 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:08.554 7fa14eaab700 1 -- 172.21.2.102:0/1499331624 learned_addr learned my addr 172.21.2.102:0/1499331624 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:30:08.558 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:08.554 7fa14eaab700 1 -- 172.21.2.102:0/1499331624 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa148095600 msgr2=0x7fa148095a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:08.558 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:08.554 7fa14eaab700 1 --2- 172.21.2.102:0/1499331624 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa148095600 0x7fa148095a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:08.558 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:08.554 7fa14eaab700 1 -- 172.21.2.102:0/1499331624 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa148096020 msgr2=0x7fa148096420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:08.558 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:08.554 7fa14eaab700 1 --2- 172.21.2.102:0/1499331624 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa148096020 0x7fa148096420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:08.558 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:08.554 7fa14eaab700 1 -- 172.21.2.102:0/1499331624 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa148093650 con 0x7fa148092a10 2021-02-10T11:30:08.558 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:08.554 7fa14eaab700 1 --2- 172.21.2.102:0/1499331624 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa148092a10 0x7fa148092e10 secure :-1 s=READY pgs=1943 cs=0 l=1 rev1=1 rx=0x7fa14000b970 tx=0x7fa14000bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:08.558 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:08.554 7fa13affd700 1 -- 172.21.2.102:0/1499331624 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa1400093d0 con 0x7fa148092a10 2021-02-10T11:30:08.558 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:08.554 7fa13affd700 1 -- 172.21.2.102:0/1499331624 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa140003030 con 0x7fa148092a10 2021-02-10T11:30:08.558 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:08.554 7fa13affd700 1 -- 172.21.2.102:0/1499331624 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa140001420 con 0x7fa148092a10 2021-02-10T11:30:08.558 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:08.554 7fa15048e700 1 -- 172.21.2.102:0/1499331624 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa148093650 con 0x7fa148092a10 2021-02-10T11:30:08.558 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:08.554 7fa15048e700 1 -- 172.21.2.102:0/1499331624 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa148093be0 con 0x7fa148092a10 2021-02-10T11:30:08.558 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:08.555 7fa13affd700 1 -- 172.21.2.102:0/1499331624 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa1400056b0 con 0x7fa148092a10 2021-02-10T11:30:08.558 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:08.555 7fa13affd700 1 --2- 172.21.2.102:0/1499331624 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa1300186c0 0x7fa13001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:08.559 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:08.555 7fa13affd700 1 -- 172.21.2.102:0/1499331624 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fa1400328d0 con 0x7fa148092a10 2021-02-10T11:30:08.559 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:08.555 7fa14e2aa700 1 --2- 172.21.2.102:0/1499331624 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa1300186c0 0x7fa13001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:08.559 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:08.555 7fa14e2aa700 1 --2- 172.21.2.102:0/1499331624 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa1300186c0 0x7fa13001abc0 crc :-1 s=READY pgs=871 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:08.586 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:08.586 7fa15048e700 1 -- 172.21.2.102:0/1499331624 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa148082980 con 0x7fa148092a10 2021-02-10T11:30:08.588 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:08.588 7fa13affd700 1 -- 172.21.2.102:0/1499331624 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fa1400093d0 con 0x7fa148092a10 2021-02-10T11:30:08.722 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:08.721 7fa15048e700 1 -- 172.21.2.102:0/1499331624 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7fa148094080 con 0x7fa148092a10 2021-02-10T11:30:08.725 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:08.725 7fa13affd700 1 -- 172.21.2.102:0/1499331624 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.1 v52) v1 ==== 184+0+0 (secure 0 0 0) 0x7fa14001f870 con 0x7fa148092a10 2021-02-10T11:30:08.729 INFO:teuthology.orchestra.run.gibba002.stderr:updated caps for client.1 2021-02-10T11:30:08.730 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:08.730 7fa15048e700 1 -- 172.21.2.102:0/1499331624 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa1300186c0 msgr2=0x7fa13001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:08.731 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:08.730 7fa15048e700 1 --2- 172.21.2.102:0/1499331624 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa1300186c0 0x7fa13001abc0 crc :-1 s=READY pgs=871 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:08.731 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:08.730 7fa15048e700 1 -- 172.21.2.102:0/1499331624 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa148092a10 msgr2=0x7fa148092e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:08.731 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:08.730 7fa15048e700 1 --2- 172.21.2.102:0/1499331624 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa148092a10 0x7fa148092e10 secure :-1 s=READY pgs=1943 cs=0 l=1 rev1=1 rx=0x7fa14000b970 tx=0x7fa14000bd10).stop 2021-02-10T11:30:08.731 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:08.730 7fa15048e700 1 -- 172.21.2.102:0/1499331624 shutdown_connections 2021-02-10T11:30:08.731 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:08.730 7fa15048e700 1 --2- 172.21.2.102:0/1499331624 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa148092a10 0x7fa148092e10 unknown :-1 s=CLOSED pgs=1943 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:08.732 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:08.730 7fa15048e700 1 --2- 172.21.2.102:0/1499331624 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa1300186c0 0x7fa13001abc0 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:08.732 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:08.730 7fa15048e700 1 --2- 172.21.2.102:0/1499331624 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa148096020 0x7fa148096420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:08.732 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:08.730 7fa15048e700 1 --2- 172.21.2.102:0/1499331624 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa148095600 0x7fa148095a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:08.732 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:08.730 7fa15048e700 1 -- 172.21.2.102:0/1499331624 >> 172.21.2.102:0/1499331624 conn(0x7fa1481a6190 msgr2=0x7fa1480949b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:08.732 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:08.731 7fa15048e700 1 -- 172.21.2.102:0/1499331624 shutdown_connections 2021-02-10T11:30:08.732 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:08.731 7fa15048e700 1 -- 172.21.2.102:0/1499331624 wait complete. 2021-02-10T11:30:08.741 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.2 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:30:09.171 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:09.154 7f7e2f1a7700 1 Processor -- start 2021-02-10T11:30:09.172 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:09.154 7f7e2f1a7700 1 -- start start 2021-02-10T11:30:09.172 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:09.154 7f7e2f1a7700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f7e280d1880 0x7f7e281b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:09.172 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:09.154 7f7e2f1a7700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7e280cff80 0x7f7e28109140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:09.172 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:09.154 7f7e2f1a7700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e28109a70 con 0x7f7e281b0af0 2021-02-10T11:30:09.172 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:09.154 7f7e2f1a7700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e28109760 con 0x7f7e280d1880 2021-02-10T11:30:09.172 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:09.154 7f7e2f1a7700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e28088070 con 0x7f7e280cff80 2021-02-10T11:30:09.172 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:09.155 7f7e2cfc3700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f7e280d1880 0x7f7e281b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:09.172 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:09.155 7f7e2cfc3700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f7e280d1880 0x7f7e281b04d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.113:3300/0 says I am v2:172.21.2.102:35974/0 (socket says 172.21.2.102:35974) 2021-02-10T11:30:09.172 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:09.155 7f7e2cfc3700 1 -- 172.21.2.102:0/115654849 learned_addr learned my addr 172.21.2.102:0/115654849 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:30:09.172 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:09.155 7f7e2d7c4700 1 --2- 172.21.2.102:0/115654849 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7e280cff80 0x7f7e28109140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:09.173 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:09.155 7f7e2cfc3700 1 -- 172.21.2.102:0/115654849 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7e280cff80 msgr2=0x7f7e28109140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:09.173 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:09.155 7f7e2cfc3700 1 --2- 172.21.2.102:0/115654849 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7e280cff80 0x7f7e28109140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:09.173 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:09.155 7f7e2cfc3700 1 -- 172.21.2.102:0/115654849 >> v1:172.21.2.113:6789/0 conn(0x7f7e281b0af0 legacy=0x7f7e28108950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:09.173 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:09.155 7f7e277fe700 1 -- 172.21.2.102:0/115654849 <== mon.1 v1:172.21.2.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 552087343 0 0) 0x7f7e18000ff0 con 0x7f7e281b0af0 2021-02-10T11:30:09.173 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:09.155 7f7e2cfc3700 1 -- 172.21.2.102:0/115654849 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e280d1c80 con 0x7f7e280d1880 2021-02-10T11:30:09.173 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:09.156 7f7e2d7c4700 1 --2- 172.21.2.102:0/115654849 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7e280cff80 0x7f7e28109140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:30:09.173 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:09.156 7f7e2cfc3700 1 --2- 172.21.2.102:0/115654849 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7e280d1880 0x7f7e281b04d0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 rx=0x7f7e10006700 tx=0x7f7e10009b80).ready entity=mon.1 client_cookie=459244bff5776d76 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:09.173 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:09.156 7f7e277fe700 1 -- 172.21.2.102:0/115654849 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7e1000c0c0 con 0x7f7e280d1880 2021-02-10T11:30:09.173 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:09.156 7f7e277fe700 1 -- 172.21.2.102:0/115654849 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e1000c3c0 con 0x7f7e280d1880 2021-02-10T11:30:09.173 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:09.156 7f7e277fe700 1 -- 172.21.2.102:0/115654849 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7e1000c8e0 con 0x7f7e280d1880 2021-02-10T11:30:09.173 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:09.156 7f7e2f1a7700 1 -- 172.21.2.102:0/115654849 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7e280d1880 msgr2=0x7f7e281b04d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:09.173 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:09.156 7f7e2f1a7700 1 --2- 172.21.2.102:0/115654849 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7e280d1880 0x7f7e281b04d0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 rx=0x7f7e10006700 tx=0x7f7e10009b80).stop 2021-02-10T11:30:09.173 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:09.157 7f7e2f1a7700 1 -- 172.21.2.102:0/115654849 shutdown_connections 2021-02-10T11:30:09.174 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:09.157 7f7e2f1a7700 1 --2- 172.21.2.102:0/115654849 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7e280cff80 0x7f7e28109140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:09.174 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:09.157 7f7e2f1a7700 1 --2- 172.21.2.102:0/115654849 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7e280d1880 0x7f7e281b04d0 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:09.174 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:09.157 7f7e2f1a7700 1 -- 172.21.2.102:0/115654849 >> 172.21.2.102:0/115654849 conn(0x7f7e280ce430 msgr2=0x7f7e280d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:09.174 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:09.157 7f7e2f1a7700 1 -- 172.21.2.102:0/115654849 shutdown_connections 2021-02-10T11:30:09.174 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:09.157 7f7e2f1a7700 1 -- 172.21.2.102:0/115654849 wait complete. 2021-02-10T11:30:09.175 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:09.171 7f7e2f1a7700 1 Processor -- start 2021-02-10T11:30:09.175 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:09.172 7f7e2f1a7700 1 -- start start 2021-02-10T11:30:09.175 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:09.172 7f7e2f1a7700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7e280abd30 0x7f7e28082bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:09.176 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:09.172 7f7e2f1a7700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7e280831d0 0x7f7e2808a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:09.176 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:09.172 7f7e2f1a7700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7e280ac130 0x7f7e2808aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:09.176 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:09.172 7f7e2f1a7700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e28083700 con 0x7f7e280831d0 2021-02-10T11:30:09.176 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:09.172 7f7e2f1a7700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e2808b0c0 con 0x7f7e280abd30 2021-02-10T11:30:09.176 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:09.172 7f7e2f1a7700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e2808b2a0 con 0x7f7e280ac130 2021-02-10T11:30:09.176 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:09.172 7f7e27fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7e280831d0 0x7f7e2808a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:09.176 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:09.172 7f7e27fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7e280831d0 0x7f7e2808a480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:60786/0 (socket says 172.21.2.102:60786) 2021-02-10T11:30:09.176 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:09.172 7f7e27fff700 1 -- 172.21.2.102:0/1854541981 learned_addr learned my addr 172.21.2.102:0/1854541981 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:30:09.176 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:09.173 7f7e27fff700 1 -- 172.21.2.102:0/1854541981 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7e280ac130 msgr2=0x7f7e2808aaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:09.176 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:09.173 7f7e27fff700 1 --2- 172.21.2.102:0/1854541981 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7e280ac130 0x7f7e2808aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:09.176 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:09.173 7f7e27fff700 1 -- 172.21.2.102:0/1854541981 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7e280abd30 msgr2=0x7f7e28082bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:09.176 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:09.173 7f7e27fff700 1 --2- 172.21.2.102:0/1854541981 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7e280abd30 0x7f7e28082bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:09.176 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:09.173 7f7e27fff700 1 -- 172.21.2.102:0/1854541981 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e2808b5f0 con 0x7f7e280831d0 2021-02-10T11:30:09.177 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:09.173 7f7e27fff700 1 --2- 172.21.2.102:0/1854541981 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7e280831d0 0x7f7e2808a480 secure :-1 s=READY pgs=1944 cs=0 l=1 rev1=1 rx=0x7f7e18006ed0 tx=0x7f7e1800b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:09.177 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:09.173 7f7e257fa700 1 -- 172.21.2.102:0/1854541981 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7e1800bd50 con 0x7f7e280831d0 2021-02-10T11:30:09.177 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:09.173 7f7e257fa700 1 -- 172.21.2.102:0/1854541981 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e18006020 con 0x7f7e280831d0 2021-02-10T11:30:09.177 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:09.173 7f7e257fa700 1 -- 172.21.2.102:0/1854541981 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7e180054b0 con 0x7f7e280831d0 2021-02-10T11:30:09.177 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:09.173 7f7e2f1a7700 1 -- 172.21.2.102:0/1854541981 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e2808b5f0 con 0x7f7e280831d0 2021-02-10T11:30:09.177 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:09.173 7f7e2f1a7700 1 -- 172.21.2.102:0/1854541981 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e2808bb50 con 0x7f7e280831d0 2021-02-10T11:30:09.177 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:09.174 7f7e257fa700 1 -- 172.21.2.102:0/1854541981 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f7e18006200 con 0x7f7e280831d0 2021-02-10T11:30:09.177 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:09.174 7f7e257fa700 1 --2- 172.21.2.102:0/1854541981 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7e0c0186c0 0x7f7e0c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:09.177 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:09.174 7f7e257fa700 1 -- 172.21.2.102:0/1854541981 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f7e18032540 con 0x7f7e280831d0 2021-02-10T11:30:09.177 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:09.174 7f7e2cfc3700 1 --2- 172.21.2.102:0/1854541981 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7e0c0186c0 0x7f7e0c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:09.177 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:09.174 7f7e2cfc3700 1 --2- 172.21.2.102:0/1854541981 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7e0c0186c0 0x7f7e0c01abc0 crc :-1 s=READY pgs=872 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:09.205 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:09.204 7f7e2f1a7700 1 -- 172.21.2.102:0/1854541981 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e28082980 con 0x7f7e280831d0 2021-02-10T11:30:09.207 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:09.207 7f7e257fa700 1 -- 172.21.2.102:0/1854541981 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f7e18005770 con 0x7f7e280831d0 2021-02-10T11:30:09.340 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:09.339 7f7e2f1a7700 1 -- 172.21.2.102:0/1854541981 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f7e28082980 con 0x7f7e280831d0 2021-02-10T11:30:09.342 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:09.342 7f7e257fa700 1 -- 172.21.2.102:0/1854541981 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.2 v53) v1 ==== 184+0+0 (secure 0 0 0) 0x7f7e18006200 con 0x7f7e280831d0 2021-02-10T11:30:09.343 INFO:teuthology.orchestra.run.gibba002.stderr:updated caps for client.2 2021-02-10T11:30:09.344 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:09.344 7f7e2f1a7700 1 -- 172.21.2.102:0/1854541981 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7e0c0186c0 msgr2=0x7f7e0c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:09.344 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:09.344 7f7e2f1a7700 1 --2- 172.21.2.102:0/1854541981 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7e0c0186c0 0x7f7e0c01abc0 crc :-1 s=READY pgs=872 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:09.344 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:09.344 7f7e2f1a7700 1 -- 172.21.2.102:0/1854541981 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7e280831d0 msgr2=0x7f7e2808a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:09.344 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:09.344 7f7e2f1a7700 1 --2- 172.21.2.102:0/1854541981 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7e280831d0 0x7f7e2808a480 secure :-1 s=READY pgs=1944 cs=0 l=1 rev1=1 rx=0x7f7e18006ed0 tx=0x7f7e1800b960).stop 2021-02-10T11:30:09.345 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:09.344 7f7e2f1a7700 1 -- 172.21.2.102:0/1854541981 shutdown_connections 2021-02-10T11:30:09.345 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:09.344 7f7e2f1a7700 1 --2- 172.21.2.102:0/1854541981 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7e280ac130 0x7f7e2808aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:09.345 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:09.344 7f7e2f1a7700 1 --2- 172.21.2.102:0/1854541981 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7e280831d0 0x7f7e2808a480 unknown :-1 s=CLOSED pgs=1944 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:09.345 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:09.344 7f7e2f1a7700 1 --2- 172.21.2.102:0/1854541981 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7e0c0186c0 0x7f7e0c01abc0 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:09.345 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:09.344 7f7e2f1a7700 1 --2- 172.21.2.102:0/1854541981 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7e280abd30 0x7f7e28082bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:09.345 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:09.344 7f7e2f1a7700 1 -- 172.21.2.102:0/1854541981 >> 172.21.2.102:0/1854541981 conn(0x7f7e280afc60 msgr2=0x7f7e280b50d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:09.345 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:09.344 7f7e2f1a7700 1 -- 172.21.2.102:0/1854541981 shutdown_connections 2021-02-10T11:30:09.345 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:09.344 7f7e2f1a7700 1 -- 172.21.2.102:0/1854541981 wait complete. 2021-02-10T11:30:09.353 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.3 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:30:09.785 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:09.766 7f35f7883700 1 Processor -- start 2021-02-10T11:30:09.785 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:09.766 7f35f7883700 1 -- start start 2021-02-10T11:30:09.785 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:09.766 7f35f7883700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f35f00d1860 0x7f35f01b04b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:09.785 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:09.766 7f35f7883700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f35f01b0ad0 0x7f35f0108930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:09.785 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:09.766 7f35f7883700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35f0109a50 con 0x7f35f00d1c60 2021-02-10T11:30:09.785 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:09.766 7f35f7883700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35f00e5ee0 con 0x7f35f01b0ad0 2021-02-10T11:30:09.785 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:09.766 7f35f7883700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35f0109740 con 0x7f35f00d1860 2021-02-10T11:30:09.785 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:09.768 7f35f4e9e700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f35f01b0ad0 0x7f35f0108930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:09.785 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:09.768 7f35f569f700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f35f00d1860 0x7f35f01b04b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:09.785 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:09.768 7f35f5ea0700 1 --1- >> v1:172.21.2.113:6789/0 conn(0x7f35f00d1c60 0x7f35f0108f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.113:6789/0 says I am v1:172.21.2.102:60338/0 (socket says 172.21.2.102:60338) 2021-02-10T11:30:09.786 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:09.768 7f35f4e9e700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f35f01b0ad0 0x7f35f0108930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.113:3300/0 says I am v2:172.21.2.102:35988/0 (socket says 172.21.2.102:35988) 2021-02-10T11:30:09.786 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:09.768 7f35f569f700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f35f00d1860 0x7f35f01b04b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.113:3301/0 says I am v2:172.21.2.102:38094/0 (socket says 172.21.2.102:38094) 2021-02-10T11:30:09.786 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:09.768 7f35f5ea0700 1 -- 172.21.2.102:0/1760325436 learned_addr learned my addr 172.21.2.102:0/1760325436 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:30:09.786 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:09.768 7f35f4e9e700 1 -- 172.21.2.102:0/1760325436 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f35f00d1860 msgr2=0x7f35f01b04b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:09.786 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:09.768 7f35f4e9e700 1 --2- 172.21.2.102:0/1760325436 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f35f00d1860 0x7f35f01b04b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:09.786 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:09.768 7f35f4e9e700 1 -- 172.21.2.102:0/1760325436 >> v1:172.21.2.113:6789/0 conn(0x7f35f00d1c60 legacy=0x7f35f0108f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:09.786 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:09.768 7f35f4e9e700 1 -- 172.21.2.102:0/1760325436 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35f0088070 con 0x7f35f01b0ad0 2021-02-10T11:30:09.786 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:09.768 7f35f569f700 1 --2- 172.21.2.102:0/1760325436 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f35f00d1860 0x7f35f01b04b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:30:09.786 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:09.769 7f35f4e9e700 1 --2- 172.21.2.102:0/1760325436 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f35f01b0ad0 0x7f35f0108930 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 rx=0x7f35e40069d0 tx=0x7f35e4009e40).ready entity=mon.1 client_cookie=2f710a990f1a0f30 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:09.786 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:09.769 7f35e3fff700 1 -- 172.21.2.102:0/1760325436 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f35e400b510 con 0x7f35f01b0ad0 2021-02-10T11:30:09.786 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:09.769 7f35e3fff700 1 -- 172.21.2.102:0/1760325436 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35e400b890 con 0x7f35f01b0ad0 2021-02-10T11:30:09.786 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:09.769 7f35e3fff700 1 -- 172.21.2.102:0/1760325436 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f35e400bdb0 con 0x7f35f01b0ad0 2021-02-10T11:30:09.786 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:09.769 7f35f7883700 1 -- 172.21.2.102:0/1760325436 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f35f01b0ad0 msgr2=0x7f35f0108930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:09.787 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:09.769 7f35f7883700 1 --2- 172.21.2.102:0/1760325436 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f35f01b0ad0 0x7f35f0108930 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 rx=0x7f35e40069d0 tx=0x7f35e4009e40).stop 2021-02-10T11:30:09.787 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:09.770 7f35f7883700 1 -- 172.21.2.102:0/1760325436 shutdown_connections 2021-02-10T11:30:09.787 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:09.770 7f35f7883700 1 --2- 172.21.2.102:0/1760325436 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f35f01b0ad0 0x7f35f0108930 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:09.787 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:09.770 7f35f7883700 1 --2- 172.21.2.102:0/1760325436 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f35f00d1860 0x7f35f01b04b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:09.787 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:09.770 7f35f7883700 1 -- 172.21.2.102:0/1760325436 >> 172.21.2.102:0/1760325436 conn(0x7f35f00ce410 msgr2=0x7f35f00d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:09.787 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:09.770 7f35f7883700 1 -- 172.21.2.102:0/1760325436 shutdown_connections 2021-02-10T11:30:09.787 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:09.770 7f35f7883700 1 -- 172.21.2.102:0/1760325436 wait complete. 2021-02-10T11:30:09.789 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:09.785 7f35f7883700 1 Processor -- start 2021-02-10T11:30:09.789 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:09.785 7f35f7883700 1 -- start start 2021-02-10T11:30:09.789 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:09.785 7f35f7883700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f35f01a3760 0x7f35f01a3b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:09.789 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:09.785 7f35f7883700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f35f00c0d10 0x7f35f00c1110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:09.789 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:09.785 7f35f7883700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f35f00c1730 0x7f35f00c1b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:09.789 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:09.785 7f35f7883700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35f00c0810 con 0x7f35f00c0d10 2021-02-10T11:30:09.789 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:09.785 7f35f7883700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35f00c2150 con 0x7f35f00c1730 2021-02-10T11:30:09.789 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:09.785 7f35f7883700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35f0097a80 con 0x7f35f01a3760 2021-02-10T11:30:09.789 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:09.785 7f35f4e9e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f35f00c0d10 0x7f35f00c1110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:09.789 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:09.785 7f35f4e9e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f35f00c0d10 0x7f35f00c1110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:60796/0 (socket says 172.21.2.102:60796) 2021-02-10T11:30:09.789 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:09.785 7f35f4e9e700 1 -- 172.21.2.102:0/1986924455 learned_addr learned my addr 172.21.2.102:0/1986924455 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:30:09.790 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:09.786 7f35f4e9e700 1 -- 172.21.2.102:0/1986924455 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f35f01a3760 msgr2=0x7f35f01a3b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:09.790 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:09.786 7f35f4e9e700 1 --2- 172.21.2.102:0/1986924455 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f35f01a3760 0x7f35f01a3b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:09.790 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:09.786 7f35f4e9e700 1 -- 172.21.2.102:0/1986924455 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f35f00c1730 msgr2=0x7f35f00c1b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:09.790 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:09.786 7f35f4e9e700 1 --2- 172.21.2.102:0/1986924455 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f35f00c1730 0x7f35f00c1b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:09.790 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:09.786 7f35f4e9e700 1 -- 172.21.2.102:0/1986924455 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35f0097ca0 con 0x7f35f00c0d10 2021-02-10T11:30:09.790 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:09.786 7f35f4e9e700 1 --2- 172.21.2.102:0/1986924455 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f35f00c0d10 0x7f35f00c1110 secure :-1 s=READY pgs=1945 cs=0 l=1 rev1=1 rx=0x7f35e4009f40 tx=0x7f35e400bd50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:09.790 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:09.786 7f35e1ffb700 1 -- 172.21.2.102:0/1986924455 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f35e4006240 con 0x7f35f00c0d10 2021-02-10T11:30:09.790 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:09.786 7f35e1ffb700 1 -- 172.21.2.102:0/1986924455 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35e40065c0 con 0x7f35f00c0d10 2021-02-10T11:30:09.790 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:09.786 7f35e1ffb700 1 -- 172.21.2.102:0/1986924455 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f35e40058e0 con 0x7f35f00c0d10 2021-02-10T11:30:09.790 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:09.786 7f35f7883700 1 -- 172.21.2.102:0/1986924455 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35f0097ca0 con 0x7f35f00c0d10 2021-02-10T11:30:09.790 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:09.786 7f35f7883700 1 -- 172.21.2.102:0/1986924455 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35f0097ca0 con 0x7f35f00c0d10 2021-02-10T11:30:09.790 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:09.787 7f35e1ffb700 1 -- 172.21.2.102:0/1986924455 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f35e4005660 con 0x7f35f00c0d10 2021-02-10T11:30:09.790 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:09.787 7f35e1ffb700 1 --2- 172.21.2.102:0/1986924455 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f35d00186c0 0x7f35d001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:09.791 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:09.787 7f35e1ffb700 1 -- 172.21.2.102:0/1986924455 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f35e40328e0 con 0x7f35f00c0d10 2021-02-10T11:30:09.791 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:09.787 7f35f569f700 1 --2- 172.21.2.102:0/1986924455 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f35d00186c0 0x7f35d001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:09.791 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:09.787 7f35f569f700 1 --2- 172.21.2.102:0/1986924455 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f35d00186c0 0x7f35d001abc0 crc :-1 s=READY pgs=873 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:09.817 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:09.817 7f35f7883700 1 -- 172.21.2.102:0/1986924455 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35f0082980 con 0x7f35f00c0d10 2021-02-10T11:30:09.819 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:09.819 7f35e1ffb700 1 -- 172.21.2.102:0/1986924455 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f35e402ff50 con 0x7f35f00c0d10 2021-02-10T11:30:09.949 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:09.948 7f35f7883700 1 -- 172.21.2.102:0/1986924455 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.3", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f35f0109a50 con 0x7f35f00c0d10 2021-02-10T11:30:09.953 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:09.952 7f35e1ffb700 1 -- 172.21.2.102:0/1986924455 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.3", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.3 v54) v1 ==== 184+0+0 (secure 0 0 0) 0x7f35e4005660 con 0x7f35f00c0d10 2021-02-10T11:30:09.956 INFO:teuthology.orchestra.run.gibba002.stderr:updated caps for client.3 2021-02-10T11:30:09.958 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:09.957 7f35f7883700 1 -- 172.21.2.102:0/1986924455 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f35d00186c0 msgr2=0x7f35d001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:09.958 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:09.957 7f35f7883700 1 --2- 172.21.2.102:0/1986924455 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f35d00186c0 0x7f35d001abc0 crc :-1 s=READY pgs=873 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:09.958 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:09.957 7f35f7883700 1 -- 172.21.2.102:0/1986924455 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f35f00c0d10 msgr2=0x7f35f00c1110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:09.958 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:09.957 7f35f7883700 1 --2- 172.21.2.102:0/1986924455 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f35f00c0d10 0x7f35f00c1110 secure :-1 s=READY pgs=1945 cs=0 l=1 rev1=1 rx=0x7f35e4009f40 tx=0x7f35e400bd50).stop 2021-02-10T11:30:09.959 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:09.958 7f35f7883700 1 -- 172.21.2.102:0/1986924455 shutdown_connections 2021-02-10T11:30:09.959 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:09.958 7f35f7883700 1 --2- 172.21.2.102:0/1986924455 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f35d00186c0 0x7f35d001abc0 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:09.959 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:09.958 7f35f7883700 1 --2- 172.21.2.102:0/1986924455 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f35f00c1730 0x7f35f00c1b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:09.959 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:09.958 7f35f7883700 1 --2- 172.21.2.102:0/1986924455 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f35f00c0d10 0x7f35f00c1110 unknown :-1 s=CLOSED pgs=1945 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:09.959 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:09.958 7f35f7883700 1 --2- 172.21.2.102:0/1986924455 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f35f01a3760 0x7f35f01a3b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:09.959 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:09.958 7f35f7883700 1 -- 172.21.2.102:0/1986924455 >> 172.21.2.102:0/1986924455 conn(0x7f35f00a9510 msgr2=0x7f35f009e360 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:09.959 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:09.958 7f35f7883700 1 -- 172.21.2.102:0/1986924455 shutdown_connections 2021-02-10T11:30:09.959 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:09.958 7f35f7883700 1 -- 172.21.2.102:0/1986924455 wait complete. 2021-02-10T11:30:09.967 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:30:10.393 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:10.376 7f5551def700 1 Processor -- start 2021-02-10T11:30:10.393 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:10.376 7f5551def700 1 -- start start 2021-02-10T11:30:10.393 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:10.377 7f5551def700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f554c1b0470 0x7f554c1b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:10.393 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:10.377 7f5551def700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f554c0f20e0 0x7f554c0f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:10.393 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:10.377 7f5551def700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f554c0f0b50 con 0x7f554c0ef610 2021-02-10T11:30:10.393 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:10.377 7f5551def700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f554c0f0820 con 0x7f554c1b0470 2021-02-10T11:30:10.394 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:10.377 7f5551def700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f554c0f24e0 con 0x7f554c0f20e0 2021-02-10T11:30:10.394 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:10.377 7f554affd700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f554c0ef610 0x7f554c0efa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:53028/0 (socket says 172.21.2.102:53028) 2021-02-10T11:30:10.394 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:10.377 7f554affd700 1 -- 172.21.2.102:0/191450503 learned_addr learned my addr 172.21.2.102:0/191450503 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:30:10.394 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:10.377 7f554b7fe700 1 --2- 172.21.2.102:0/191450503 >> v2:172.21.2.102:3300/0 conn(0x7f554c1b0470 0x7f554c1b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:10.394 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:10.377 7f554bfff700 1 --2- 172.21.2.102:0/191450503 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f554c0f20e0 0x7f554c0f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:10.394 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:10.377 7f554b7fe700 1 -- 172.21.2.102:0/191450503 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f554c0f20e0 msgr2=0x7f554c0f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:10.394 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:10.377 7f554b7fe700 1 --2- 172.21.2.102:0/191450503 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f554c0f20e0 0x7f554c0f0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:10.394 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:10.377 7f554b7fe700 1 -- 172.21.2.102:0/191450503 >> v1:172.21.2.102:6789/0 conn(0x7f554c0ef610 legacy=0x7f554c0efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:10.394 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:10.377 7f554b7fe700 1 -- 172.21.2.102:0/191450503 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f554c0e52e0 con 0x7f554c1b0470 2021-02-10T11:30:10.394 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:10.378 7f554b7fe700 1 --2- 172.21.2.102:0/191450503 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f554c1b0470 0x7f554c1b0870 secure :-1 s=READY pgs=1947 cs=0 l=1 rev1=1 rx=0x7f5534006650 tx=0x7f5534009a00).ready entity=mon.0 client_cookie=b97a548736a0911d server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:10.394 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:10.378 7f554a7fc700 1 -- 172.21.2.102:0/191450503 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5534009df0 con 0x7f554c1b0470 2021-02-10T11:30:10.394 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:10.378 7f554a7fc700 1 -- 172.21.2.102:0/191450503 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55340009b0 con 0x7f554c1b0470 2021-02-10T11:30:10.395 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:10.378 7f554a7fc700 1 -- 172.21.2.102:0/191450503 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f553400c2b0 con 0x7f554c1b0470 2021-02-10T11:30:10.395 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:10.378 7f5551def700 1 -- 172.21.2.102:0/191450503 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f554c1b0470 msgr2=0x7f554c1b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:10.395 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:10.378 7f5551def700 1 --2- 172.21.2.102:0/191450503 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f554c1b0470 0x7f554c1b0870 secure :-1 s=READY pgs=1947 cs=0 l=1 rev1=1 rx=0x7f5534006650 tx=0x7f5534009a00).stop 2021-02-10T11:30:10.395 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:10.378 7f5551def700 1 -- 172.21.2.102:0/191450503 shutdown_connections 2021-02-10T11:30:10.395 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:10.378 7f5551def700 1 --2- 172.21.2.102:0/191450503 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f554c0f20e0 0x7f554c0f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:10.395 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:10.378 7f5551def700 1 --2- 172.21.2.102:0/191450503 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f554c1b0470 0x7f554c1b0870 unknown :-1 s=CLOSED pgs=1947 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:10.395 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:10.378 7f5551def700 1 -- 172.21.2.102:0/191450503 >> 172.21.2.102:0/191450503 conn(0x7f554c0ce3e0 msgr2=0x7f554c0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:10.395 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:10.378 7f5551def700 1 -- 172.21.2.102:0/191450503 shutdown_connections 2021-02-10T11:30:10.395 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:10.378 7f5551def700 1 -- 172.21.2.102:0/191450503 wait complete. 2021-02-10T11:30:10.397 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:10.393 7f5551def700 1 Processor -- start 2021-02-10T11:30:10.397 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:10.393 7f5551def700 1 -- start start 2021-02-10T11:30:10.397 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:10.393 7f5551def700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f554c09ec00 0x7f554c0bfc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:10.397 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:10.393 7f5551def700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f554c0a1060 0x7f554c09e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:10.397 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:10.393 7f5551def700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f554c0c0940 0x7f554c0c0d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:10.397 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:10.393 7f5551def700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f554c0c03c0 con 0x7f554c0a1060 2021-02-10T11:30:10.397 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:10.393 7f5551def700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f554c09e6b0 con 0x7f554c09ec00 2021-02-10T11:30:10.398 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:10.393 7f5551def700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f554c09e890 con 0x7f554c0c0940 2021-02-10T11:30:10.398 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:10.394 7f554affd700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f554c0a1060 0x7f554c09e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:10.398 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:10.394 7f554affd700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f554c0a1060 0x7f554c09e090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:60814/0 (socket says 172.21.2.102:60814) 2021-02-10T11:30:10.398 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:10.394 7f554affd700 1 -- 172.21.2.102:0/1057647576 learned_addr learned my addr 172.21.2.102:0/1057647576 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:30:10.398 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:10.394 7f554bfff700 1 --2- 172.21.2.102:0/1057647576 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f554c0c0940 0x7f554c0c0d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:10.398 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:10.394 7f554b7fe700 1 --2- 172.21.2.102:0/1057647576 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f554c09ec00 0x7f554c0bfc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:10.398 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:10.394 7f554affd700 1 -- 172.21.2.102:0/1057647576 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f554c0c0940 msgr2=0x7f554c0c0d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:10.398 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:10.394 7f554affd700 1 --2- 172.21.2.102:0/1057647576 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f554c0c0940 0x7f554c0c0d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:10.398 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:10.394 7f554affd700 1 -- 172.21.2.102:0/1057647576 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f554c09ec00 msgr2=0x7f554c0bfc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:10.398 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:10.394 7f554affd700 1 --2- 172.21.2.102:0/1057647576 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f554c09ec00 0x7f554c0bfc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:10.398 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:10.394 7f554affd700 1 -- 172.21.2.102:0/1057647576 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f554c0c1360 con 0x7f554c0a1060 2021-02-10T11:30:10.399 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:10.394 7f554b7fe700 1 --2- 172.21.2.102:0/1057647576 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f554c09ec00 0x7f554c0bfc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:30:10.399 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:10.394 7f554affd700 1 --2- 172.21.2.102:0/1057647576 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f554c0a1060 0x7f554c09e090 secure :-1 s=READY pgs=1948 cs=0 l=1 rev1=1 rx=0x7f553c001c60 tx=0x7f553c00a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:10.399 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:10.394 7f5543fff700 1 -- 172.21.2.102:0/1057647576 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f553c00acc0 con 0x7f554c0a1060 2021-02-10T11:30:10.399 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:10.394 7f5543fff700 1 -- 172.21.2.102:0/1057647576 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f553c004140 con 0x7f554c0a1060 2021-02-10T11:30:10.399 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:10.395 7f5551def700 1 -- 172.21.2.102:0/1057647576 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f554c0c1360 con 0x7f554c0a1060 2021-02-10T11:30:10.399 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:10.395 7f5543fff700 1 -- 172.21.2.102:0/1057647576 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f553c004620 con 0x7f554c0a1060 2021-02-10T11:30:10.400 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:10.395 7f5551def700 1 -- 172.21.2.102:0/1057647576 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f554c0c18f0 con 0x7f554c0a1060 2021-02-10T11:30:10.400 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:10.395 7f5543fff700 1 -- 172.21.2.102:0/1057647576 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f553c000db0 con 0x7f554c0a1060 2021-02-10T11:30:10.400 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:10.395 7f5543fff700 1 --2- 172.21.2.102:0/1057647576 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5530018730 0x7f553001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:10.400 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:10.395 7f5543fff700 1 -- 172.21.2.102:0/1057647576 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f553c0325f0 con 0x7f554c0a1060 2021-02-10T11:30:10.400 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:10.395 7f554b7fe700 1 --2- 172.21.2.102:0/1057647576 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5530018730 0x7f553001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:10.400 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:10.396 7f554b7fe700 1 --2- 172.21.2.102:0/1057647576 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5530018730 0x7f553001abc0 crc :-1 s=READY pgs=874 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:10.427 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:10.427 7f5551def700 1 -- 172.21.2.102:0/1057647576 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f554c105ab0 con 0x7f554c0a1060 2021-02-10T11:30:10.430 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:10.429 7f5543fff700 1 -- 172.21.2.102:0/1057647576 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f553c0009b0 con 0x7f554c0a1060 2021-02-10T11:30:10.558 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:10.557 7f5551def700 1 -- 172.21.2.102:0/1057647576 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f554c0f0b50 con 0x7f554c0a1060 2021-02-10T11:30:10.558 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:10.558 7f5543fff700 1 -- 172.21.2.102:0/1057647576 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) v1 ==== 94+0+2098 (secure 0 0 0) 0x7f553c000db0 con 0x7f554c0a1060 2021-02-10T11:30:10.559 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:30:10.559 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":73,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9672,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.102:6835/2071608955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6834","nonce":2071608955},{"type":"v1","addr":"172.21.2.102:6835","nonce":2071608955}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":69}],"filesystems":[{"mdsmap":{"epoch":73,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:30:04.225997","modified":"2021-02-10 11:30:06.232546","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7186},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7186":{"gid":7186,"name":"b","rank":0,"incarnation":70,"state":"up:active","state_seq":3,"addr":"172.21.2.113:6833/675492649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6832","nonce":675492649},{"type":"v1","addr":"172.21.2.113:6833","nonce":675492649}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":9}]} 2021-02-10T11:30:10.559 INFO:teuthology.orchestra.run.gibba002.stderr:dumped fsmap epoch 73 2021-02-10T11:30:10.560 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:10.560 7f5551def700 1 -- 172.21.2.102:0/1057647576 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5530018730 msgr2=0x7f553001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:10.560 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:10.560 7f5551def700 1 --2- 172.21.2.102:0/1057647576 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5530018730 0x7f553001abc0 crc :-1 s=READY pgs=874 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:10.560 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:10.560 7f5551def700 1 -- 172.21.2.102:0/1057647576 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f554c0a1060 msgr2=0x7f554c09e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:10.560 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:10.560 7f5551def700 1 --2- 172.21.2.102:0/1057647576 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f554c0a1060 0x7f554c09e090 secure :-1 s=READY pgs=1948 cs=0 l=1 rev1=1 rx=0x7f553c001c60 tx=0x7f553c00a8d0).stop 2021-02-10T11:30:10.561 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:10.560 7f5551def700 1 -- 172.21.2.102:0/1057647576 shutdown_connections 2021-02-10T11:30:10.561 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:10.560 7f5551def700 1 --2- 172.21.2.102:0/1057647576 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f554c0c0940 0x7f554c0c0d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:10.561 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:10.560 7f5551def700 1 --2- 172.21.2.102:0/1057647576 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f554c0a1060 0x7f554c09e090 unknown :-1 s=CLOSED pgs=1948 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:10.561 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:10.560 7f5551def700 1 --2- 172.21.2.102:0/1057647576 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5530018730 0x7f553001abc0 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:10.561 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:10.560 7f5551def700 1 --2- 172.21.2.102:0/1057647576 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f554c09ec00 0x7f554c0bfc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:10.561 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:10.560 7f5551def700 1 -- 172.21.2.102:0/1057647576 >> 172.21.2.102:0/1057647576 conn(0x7f554c0b9290 msgr2=0x7f554c09c110 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:10.561 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:10.560 7f5551def700 1 -- 172.21.2.102:0/1057647576 shutdown_connections 2021-02-10T11:30:10.562 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:10.560 7f5551def700 1 -- 172.21.2.102:0/1057647576 wait complete. 2021-02-10T11:30:10.771 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 73, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:30:04.225997', 'modified': '2021-02-10 11:30:06.232546', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7186}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7186': {'gid': 7186, 'name': 'b', 'rank': 0, 'incarnation': 70, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.113:6833/675492649', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.113:6832', 'nonce': 675492649}, {'type': 'v1', 'addr': '172.21.2.113:6833', 'nonce': 675492649}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:30:10.771 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:30:10.771 DEBUG:teuthology.orchestra.run.gibba013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:30:11.170 INFO:teuthology.orchestra.run.gibba013.stdout:{ 2021-02-10T11:30:11.171 INFO:teuthology.orchestra.run.gibba013.stdout: "cluster_fsid": "68db30ea-dbce-4aef-a6df-f01c6fea0513", 2021-02-10T11:30:11.171 INFO:teuthology.orchestra.run.gibba013.stdout: "whoami": 0, 2021-02-10T11:30:11.171 INFO:teuthology.orchestra.run.gibba013.stdout: "id": 7186, 2021-02-10T11:30:11.171 INFO:teuthology.orchestra.run.gibba013.stdout: "want_state": "up:active", 2021-02-10T11:30:11.171 INFO:teuthology.orchestra.run.gibba013.stdout: "state": "up:active", 2021-02-10T11:30:11.171 INFO:teuthology.orchestra.run.gibba013.stdout: "rank_uptime": 6.942592469, 2021-02-10T11:30:11.171 INFO:teuthology.orchestra.run.gibba013.stdout: "mdsmap_epoch": 73, 2021-02-10T11:30:11.171 INFO:teuthology.orchestra.run.gibba013.stdout: "osdmap_epoch": 93, 2021-02-10T11:30:11.171 INFO:teuthology.orchestra.run.gibba013.stdout: "osdmap_epoch_barrier": 93, 2021-02-10T11:30:11.171 INFO:teuthology.orchestra.run.gibba013.stdout: "uptime": 12.602137225 2021-02-10T11:30:11.171 INFO:teuthology.orchestra.run.gibba013.stdout:} 2021-02-10T11:30:11.174 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "68db30ea-dbce-4aef-a6df-f01c6fea0513", "whoami": 0, "id": 7186, "want_state": "up:active", "state": "up:active", "rank_uptime": 6.942592469, "mdsmap_epoch": 73, "osdmap_epoch": 93, "osdmap_epoch_barrier": 93, "uptime": 12.602137225 } 2021-02-10T11:30:11.175 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:30:11.601 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:11.585 7f03db45a700 1 Processor -- start 2021-02-10T11:30:11.602 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:11.585 7f03db45a700 1 -- start start 2021-02-10T11:30:11.602 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:11.585 7f03db45a700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f03d40d1870 0x7f03d41b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:11.602 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:11.585 7f03db45a700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f03d40f0650 0x7f03d40cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:11.602 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:11.585 7f03db45a700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03d40f0b10 con 0x7f03d41b0a10 2021-02-10T11:30:11.602 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:11.585 7f03db45a700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03d40fe230 con 0x7f03d40f0650 2021-02-10T11:30:11.602 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:11.585 7f03db45a700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03d40e5ed0 con 0x7f03d40d1870 2021-02-10T11:30:11.602 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:11.586 7f03d9a77700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f03d40f0650 0x7f03d40cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:11.602 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:11.586 7f03d9a77700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f03d40f0650 0x7f03d40cfb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:60818/0 (socket says 172.21.2.102:60818) 2021-02-10T11:30:11.602 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:11.586 7f03d9a77700 1 -- 172.21.2.102:0/108550007 learned_addr learned my addr 172.21.2.102:0/108550007 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:30:11.602 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:11.586 7f03d9a77700 1 -- 172.21.2.102:0/108550007 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f03d40d1870 msgr2=0x7f03d41b03f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:30:11.602 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:11.586 7f03d9a77700 1 --2- 172.21.2.102:0/108550007 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f03d40d1870 0x7f03d41b03f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:11.603 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:11.586 7f03d9a77700 1 -- 172.21.2.102:0/108550007 >> v1:172.21.2.113:6789/0 conn(0x7f03d41b0a10 legacy=0x7f03d40cf330 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:30:11.603 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:11.586 7f03d9a77700 1 -- 172.21.2.102:0/108550007 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03d40f20a0 con 0x7f03d40f0650 2021-02-10T11:30:11.603 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:11.586 7f03d9a77700 1 --2- 172.21.2.102:0/108550007 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f03d40f0650 0x7f03d40cfb20 secure :-1 s=READY pgs=1949 cs=0 l=1 rev1=1 rx=0x7f03cc006f00 tx=0x7f03cc009f10).ready entity=mon.0 client_cookie=e5cc4b2e778a44cb server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:11.603 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:11.586 7f03cbfff700 1 -- 172.21.2.102:0/108550007 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f03cc00b3b0 con 0x7f03d40f0650 2021-02-10T11:30:11.603 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:11.586 7f03cbfff700 1 -- 172.21.2.102:0/108550007 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03cc00b630 con 0x7f03d40f0650 2021-02-10T11:30:11.603 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:11.586 7f03cbfff700 1 -- 172.21.2.102:0/108550007 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f03cc00bb50 con 0x7f03d40f0650 2021-02-10T11:30:11.603 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:11.586 7f03db45a700 1 -- 172.21.2.102:0/108550007 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f03d40f0650 msgr2=0x7f03d40cfb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:11.603 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:11.586 7f03db45a700 1 --2- 172.21.2.102:0/108550007 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f03d40f0650 0x7f03d40cfb20 secure :-1 s=READY pgs=1949 cs=0 l=1 rev1=1 rx=0x7f03cc006f00 tx=0x7f03cc009f10).stop 2021-02-10T11:30:11.603 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:11.587 7f03db45a700 1 -- 172.21.2.102:0/108550007 shutdown_connections 2021-02-10T11:30:11.603 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:11.587 7f03db45a700 1 --2- 172.21.2.102:0/108550007 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f03d40f0650 0x7f03d40cfb20 unknown :-1 s=CLOSED pgs=1949 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:11.603 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:11.587 7f03db45a700 1 --2- 172.21.2.102:0/108550007 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f03d40d1870 0x7f03d41b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:11.603 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:11.587 7f03db45a700 1 -- 172.21.2.102:0/108550007 >> 172.21.2.102:0/108550007 conn(0x7f03d40ce450 msgr2=0x7f03d40d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:11.603 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:11.587 7f03db45a700 1 -- 172.21.2.102:0/108550007 shutdown_connections 2021-02-10T11:30:11.604 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:11.587 7f03db45a700 1 -- 172.21.2.102:0/108550007 wait complete. 2021-02-10T11:30:11.605 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:11.601 7f03db45a700 1 Processor -- start 2021-02-10T11:30:11.605 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:11.601 7f03db45a700 1 -- start start 2021-02-10T11:30:11.605 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:11.602 7f03db45a700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f03d40b0ad0 0x7f03d4091f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:11.605 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:11.602 7f03db45a700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f03d41a8850 0x7f03d41a8c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:11.605 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:11.602 7f03db45a700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f03d41a6fa0 0x7f03d41a73a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:11.605 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:11.602 7f03db45a700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03d41abfe0 con 0x7f03d41a8850 2021-02-10T11:30:11.606 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:11.602 7f03db45a700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03d41a9c40 con 0x7f03d40b0ad0 2021-02-10T11:30:11.606 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:11.602 7f03db45a700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03d41a9730 con 0x7f03d41a6fa0 2021-02-10T11:30:11.606 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:11.602 7f03d8a75700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f03d41a8850 0x7f03d41a8c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:11.606 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:11.602 7f03d8a75700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f03d41a8850 0x7f03d41a8c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:60826/0 (socket says 172.21.2.102:60826) 2021-02-10T11:30:11.606 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:11.602 7f03d8a75700 1 -- 172.21.2.102:0/1030526249 learned_addr learned my addr 172.21.2.102:0/1030526249 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:30:11.606 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:11.602 7f03d8a75700 1 -- 172.21.2.102:0/1030526249 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f03d41a6fa0 msgr2=0x7f03d41a73a0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:30:11.606 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:11.602 7f03d8a75700 1 --2- 172.21.2.102:0/1030526249 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f03d41a6fa0 0x7f03d41a73a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:11.606 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:11.602 7f03d8a75700 1 -- 172.21.2.102:0/1030526249 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f03d40b0ad0 msgr2=0x7f03d4091f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:11.606 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:11.602 7f03d8a75700 1 --2- 172.21.2.102:0/1030526249 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f03d40b0ad0 0x7f03d4091f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:11.606 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:11.602 7f03d8a75700 1 -- 172.21.2.102:0/1030526249 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03d41a9270 con 0x7f03d41a8850 2021-02-10T11:30:11.606 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:11.603 7f03d8a75700 1 --2- 172.21.2.102:0/1030526249 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f03d41a8850 0x7f03d41a8c50 secure :-1 s=READY pgs=1950 cs=0 l=1 rev1=1 rx=0x7f03d000b5b0 tx=0x7f03d000b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:11.606 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:11.603 7f03c9ffb700 1 -- 172.21.2.102:0/1030526249 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f03d000bd40 con 0x7f03d41a8850 2021-02-10T11:30:11.606 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:11.603 7f03c9ffb700 1 -- 172.21.2.102:0/1030526249 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03d0006240 con 0x7f03d41a8850 2021-02-10T11:30:11.607 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:11.603 7f03c9ffb700 1 -- 172.21.2.102:0/1030526249 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f03d0005720 con 0x7f03d41a8850 2021-02-10T11:30:11.607 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:11.603 7f03db45a700 1 -- 172.21.2.102:0/1030526249 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03d41a9270 con 0x7f03d41a8850 2021-02-10T11:30:11.607 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:11.603 7f03db45a700 1 -- 172.21.2.102:0/1030526249 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03d41a9270 con 0x7f03d41a8850 2021-02-10T11:30:11.607 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:11.604 7f03c9ffb700 1 -- 172.21.2.102:0/1030526249 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f03d00054a0 con 0x7f03d41a8850 2021-02-10T11:30:11.607 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:11.604 7f03c9ffb700 1 --2- 172.21.2.102:0/1030526249 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f03b40186c0 0x7f03b401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:11.607 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:11.604 7f03c9ffb700 1 -- 172.21.2.102:0/1030526249 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f03d0032430 con 0x7f03d41a8850 2021-02-10T11:30:11.607 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:11.604 7f03d9276700 1 --2- 172.21.2.102:0/1030526249 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f03b40186c0 0x7f03b401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:11.607 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:11.604 7f03d9276700 1 --2- 172.21.2.102:0/1030526249 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f03b40186c0 0x7f03b401abc0 crc :-1 s=READY pgs=875 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:11.636 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:11.635 7f03db45a700 1 -- 172.21.2.102:0/1030526249 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03d41abfe0 con 0x7f03d41a8850 2021-02-10T11:30:11.638 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:11.638 7f03c9ffb700 1 -- 172.21.2.102:0/1030526249 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f03d0000c60 con 0x7f03d41a8850 2021-02-10T11:30:11.764 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:11.764 7f03db45a700 1 -- 172.21.2.102:0/1030526249 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f03d41abfe0 con 0x7f03d41a8850 2021-02-10T11:30:11.765 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:11.764 7f03c9ffb700 1 -- 172.21.2.102:0/1030526249 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) v1 ==== 74+0+13996 (secure 0 0 0) 0x7f03d00054a0 con 0x7f03d41a8850 2021-02-10T11:30:11.765 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:30:11.765 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":93,"fsid":"68db30ea-dbce-4aef-a6df-f01c6fea0513","created":"2021-02-10 11:03:06.331557","modified":"2021-02-10 11:30:05.228248","last_up_change":"2021-02-10 11:03:32.690562","last_in_change":"2021-02-10 11:03:32.690562","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:03:37.322519","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:30:02.187273","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":19,"pool_name":"cephfs_data","create_time":"2021-02-10 11:30:03.219690","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"93","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"be691b4d-a039-408a-af80-9d53f02b6f1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6818","nonce":36088},{"type":"v1","addr":"172.21.2.102:6820","nonce":36088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6822","nonce":36088},{"type":"v1","addr":"172.21.2.102:6824","nonce":36088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6830","nonce":36088},{"type":"v1","addr":"172.21.2.102:6832","nonce":36088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6826","nonce":36088},{"type":"v1","addr":"172.21.2.102:6828","nonce":36088}]},"public_addr":"172.21.2.102:6820/36088","cluster_addr":"172.21.2.102:6824/36088","heartbeat_back_addr":"172.21.2.102:6832/36088","heartbeat_front_addr":"172.21.2.102:6828/36088","state":["exists","up"]},{"osd":1,"uuid":"64e3ccc8-9c69-4d7c-865e-af8e83822635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6819","nonce":36087},{"type":"v1","addr":"172.21.2.102:6821","nonce":36087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6823","nonce":36087},{"type":"v1","addr":"172.21.2.102:6825","nonce":36087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6831","nonce":36087},{"type":"v1","addr":"172.21.2.102:6833","nonce":36087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6827","nonce":36087},{"type":"v1","addr":"172.21.2.102:6829","nonce":36087}]},"public_addr":"172.21.2.102:6821/36087","cluster_addr":"172.21.2.102:6825/36087","heartbeat_back_addr":"172.21.2.102:6833/36087","heartbeat_front_addr":"172.21.2.102:6829/36087","state":["exists","up"]},{"osd":2,"uuid":"153700af-747f-481f-bdf4-d6072d3a842f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6810","nonce":36089},{"type":"v1","addr":"172.21.2.102:6811","nonce":36089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6812","nonce":36089},{"type":"v1","addr":"172.21.2.102:6813","nonce":36089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6816","nonce":36089},{"type":"v1","addr":"172.21.2.102:6817","nonce":36089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6814","nonce":36089},{"type":"v1","addr":"172.21.2.102:6815","nonce":36089}]},"public_addr":"172.21.2.102:6811/36089","cluster_addr":"172.21.2.102:6813/36089","heartbeat_back_addr":"172.21.2.102:6817/36089","heartbeat_front_addr":"172.21.2.102:6815/36089","state":["exists","up"]},{"osd":3,"uuid":"cbc678f4-83a9-45e6-8830-2f7c232b1ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6802","nonce":36090},{"type":"v1","addr":"172.21.2.102:6803","nonce":36090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6804","nonce":36090},{"type":"v1","addr":"172.21.2.102:6805","nonce":36090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6808","nonce":36090},{"type":"v1","addr":"172.21.2.102:6809","nonce":36090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6806","nonce":36090},{"type":"v1","addr":"172.21.2.102:6807","nonce":36090}]},"public_addr":"172.21.2.102:6803/36090","cluster_addr":"172.21.2.102:6805/36090","heartbeat_back_addr":"172.21.2.102:6809/36090","heartbeat_front_addr":"172.21.2.102:6807/36090","state":["exists","up"]},{"osd":4,"uuid":"c01e4008-b258-4f10-afd2-9e7f37214351","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6808","nonce":36368},{"type":"v1","addr":"172.21.2.113:6811","nonce":36368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6814","nonce":36368},{"type":"v1","addr":"172.21.2.113:6818","nonce":36368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6826","nonce":36368},{"type":"v1","addr":"172.21.2.113:6828","nonce":36368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6820","nonce":36368},{"type":"v1","addr":"172.21.2.113:6824","nonce":36368}]},"public_addr":"172.21.2.113:6811/36368","cluster_addr":"172.21.2.113:6818/36368","heartbeat_back_addr":"172.21.2.113:6828/36368","heartbeat_front_addr":"172.21.2.113:6824/36368","state":["exists","up"]},{"osd":5,"uuid":"551f6007-daf8-4a98-8ee0-c2b16bcccb5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6802","nonce":36366},{"type":"v1","addr":"172.21.2.113:6805","nonce":36366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6807","nonce":36366},{"type":"v1","addr":"172.21.2.113:6810","nonce":36366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6819","nonce":36366},{"type":"v1","addr":"172.21.2.113:6822","nonce":36366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6812","nonce":36366},{"type":"v1","addr":"172.21.2.113:6815","nonce":36366}]},"public_addr":"172.21.2.113:6805/36366","cluster_addr":"172.21.2.113:6810/36366","heartbeat_back_addr":"172.21.2.113:6822/36366","heartbeat_front_addr":"172.21.2.113:6815/36366","state":["exists","up"]},{"osd":6,"uuid":"bf67c040-0e50-48be-b102-c3943bc9ebef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6800","nonce":36367},{"type":"v1","addr":"172.21.2.113:6801","nonce":36367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6803","nonce":36367},{"type":"v1","addr":"172.21.2.113:6804","nonce":36367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6813","nonce":36367},{"type":"v1","addr":"172.21.2.113:6816","nonce":36367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6806","nonce":36367},{"type":"v1","addr":"172.21.2.113:6809","nonce":36367}]},"public_addr":"172.21.2.113:6801/36367","cluster_addr":"172.21.2.113:6804/36367","heartbeat_back_addr":"172.21.2.113:6816/36367","heartbeat_front_addr":"172.21.2.113:6809/36367","state":["exists","up"]},{"osd":7,"uuid":"f5461f80-018d-428f-be89-2eb2bcbe7bb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6817","nonce":36365},{"type":"v1","addr":"172.21.2.113:6821","nonce":36365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6823","nonce":36365},{"type":"v1","addr":"172.21.2.113:6825","nonce":36365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6830","nonce":36365},{"type":"v1","addr":"172.21.2.113:6831","nonce":36365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6827","nonce":36365},{"type":"v1","addr":"172.21.2.113:6829","nonce":36365}]},"public_addr":"172.21.2.113:6821/36365","cluster_addr":"172.21.2.113:6825/36365","heartbeat_back_addr":"172.21.2.113:6831/36365","heartbeat_front_addr":"172.21.2.113:6829/36365","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:30:11.766 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:11.766 7f03db45a700 1 -- 172.21.2.102:0/1030526249 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f03b40186c0 msgr2=0x7f03b401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:11.766 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:11.766 7f03db45a700 1 --2- 172.21.2.102:0/1030526249 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f03b40186c0 0x7f03b401abc0 crc :-1 s=READY pgs=875 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:11.766 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:11.766 7f03db45a700 1 -- 172.21.2.102:0/1030526249 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f03d41a8850 msgr2=0x7f03d41a8c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:11.766 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:11.766 7f03db45a700 1 --2- 172.21.2.102:0/1030526249 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f03d41a8850 0x7f03d41a8c50 secure :-1 s=READY pgs=1950 cs=0 l=1 rev1=1 rx=0x7f03d000b5b0 tx=0x7f03d000b950).stop 2021-02-10T11:30:11.767 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:11.766 7f03db45a700 1 -- 172.21.2.102:0/1030526249 shutdown_connections 2021-02-10T11:30:11.767 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:11.766 7f03db45a700 1 --2- 172.21.2.102:0/1030526249 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f03d41a6fa0 0x7f03d41a73a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:11.767 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:11.766 7f03db45a700 1 --2- 172.21.2.102:0/1030526249 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f03d41a8850 0x7f03d41a8c50 unknown :-1 s=CLOSED pgs=1950 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:11.767 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:11.766 7f03db45a700 1 --2- 172.21.2.102:0/1030526249 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f03b40186c0 0x7f03b401abc0 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:11.767 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:11.766 7f03db45a700 1 --2- 172.21.2.102:0/1030526249 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f03d40b0ad0 0x7f03d4091f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:11.768 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:11.766 7f03db45a700 1 -- 172.21.2.102:0/1030526249 >> 172.21.2.102:0/1030526249 conn(0x7f03d419f8f0 msgr2=0x7f03d41a5f80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:11.768 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:11.766 7f03db45a700 1 -- 172.21.2.102:0/1030526249 shutdown_connections 2021-02-10T11:30:11.768 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:11.766 7f03db45a700 1 -- 172.21.2.102:0/1030526249 wait complete. 2021-02-10T11:30:11.776 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:03:37.322519', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 18, 'pool_name': 'cephfs_metadata', 'create_time': '2021-02-10 11:30:02.187273', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '92', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 19, 'pool_name': 'cephfs_data', 'create_time': '2021-02-10 11:30:03.219690', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '93', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-02-10T11:30:11.776 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:30:12.208 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:12.191 7f7d471f5700 1 Processor -- start 2021-02-10T11:30:12.208 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:12.191 7f7d471f5700 1 -- start start 2021-02-10T11:30:12.208 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:12.191 7f7d471f5700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f7d400d1870 0x7f7d401b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:12.208 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:12.191 7f7d471f5700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f7d401b0ae0 0x7f7d40108940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:12.208 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:12.191 7f7d471f5700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d400fe1c0 con 0x7f7d400d1c70 2021-02-10T11:30:12.209 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:12.191 7f7d471f5700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d40109a60 con 0x7f7d401b0ae0 2021-02-10T11:30:12.209 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:12.191 7f7d471f5700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d40109750 con 0x7f7d400d1870 2021-02-10T11:30:12.209 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:12.192 7f7d44810700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f7d401b0ae0 0x7f7d40108940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:12.209 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:12.192 7f7d44810700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f7d401b0ae0 0x7f7d40108940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:60832/0 (socket says 172.21.2.102:60832) 2021-02-10T11:30:12.209 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:12.192 7f7d44810700 1 -- 172.21.2.102:0/459586813 learned_addr learned my addr 172.21.2.102:0/459586813 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:30:12.209 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:12.192 7f7d44810700 1 -- 172.21.2.102:0/459586813 >> v2:172.21.2.113:3300/0 conn(0x7f7d400d1870 msgr2=0x7f7d401b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:12.209 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:12.192 7f7d44810700 1 --2- 172.21.2.102:0/459586813 >> v2:172.21.2.113:3300/0 conn(0x7f7d400d1870 0x7f7d401b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:12.209 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:12.192 7f7d44810700 1 -- 172.21.2.102:0/459586813 >> v1:172.21.2.113:6789/0 conn(0x7f7d400d1c70 legacy=0x7f7d40108f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:12.209 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:12.192 7f7d44810700 1 -- 172.21.2.102:0/459586813 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d40088070 con 0x7f7d401b0ae0 2021-02-10T11:30:12.209 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:12.192 7f7d44810700 1 --2- 172.21.2.102:0/459586813 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7d401b0ae0 0x7f7d40108940 secure :-1 s=READY pgs=1951 cs=0 l=1 rev1=1 rx=0x7f7d3c006700 tx=0x7f7d3c009b00).ready entity=mon.0 client_cookie=fb3988e40fb211bc server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:12.209 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:12.192 7f7d33fff700 1 -- 172.21.2.102:0/459586813 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7d3c00c0c0 con 0x7f7d401b0ae0 2021-02-10T11:30:12.210 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:12.192 7f7d33fff700 1 -- 172.21.2.102:0/459586813 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d3c00c340 con 0x7f7d401b0ae0 2021-02-10T11:30:12.210 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:12.192 7f7d33fff700 1 -- 172.21.2.102:0/459586813 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7d3c00c820 con 0x7f7d401b0ae0 2021-02-10T11:30:12.210 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:12.192 7f7d471f5700 1 -- 172.21.2.102:0/459586813 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7d401b0ae0 msgr2=0x7f7d40108940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:12.210 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:12.192 7f7d471f5700 1 --2- 172.21.2.102:0/459586813 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7d401b0ae0 0x7f7d40108940 secure :-1 s=READY pgs=1951 cs=0 l=1 rev1=1 rx=0x7f7d3c006700 tx=0x7f7d3c009b00).stop 2021-02-10T11:30:12.210 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:12.193 7f7d471f5700 1 -- 172.21.2.102:0/459586813 shutdown_connections 2021-02-10T11:30:12.210 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:12.193 7f7d471f5700 1 --2- 172.21.2.102:0/459586813 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7d401b0ae0 0x7f7d40108940 unknown :-1 s=CLOSED pgs=1951 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:12.210 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:12.193 7f7d471f5700 1 --2- 172.21.2.102:0/459586813 >> v2:172.21.2.113:3300/0 conn(0x7f7d400d1870 0x7f7d401b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:12.210 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:12.193 7f7d471f5700 1 -- 172.21.2.102:0/459586813 >> 172.21.2.102:0/459586813 conn(0x7f7d400ce420 msgr2=0x7f7d400d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:12.210 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:12.193 7f7d471f5700 1 -- 172.21.2.102:0/459586813 shutdown_connections 2021-02-10T11:30:12.210 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:12.193 7f7d471f5700 1 -- 172.21.2.102:0/459586813 wait complete. 2021-02-10T11:30:12.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:12.208 7f7d471f5700 1 Processor -- start 2021-02-10T11:30:12.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:12.208 7f7d471f5700 1 -- start start 2021-02-10T11:30:12.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:12.208 7f7d471f5700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7d400c3820 0x7f7d40084cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:12.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:12.208 7f7d471f5700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7d400a0290 0x7f7d400852d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:12.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:12.208 7f7d471f5700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7d40086270 0x7f7d40086670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:12.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:12.208 7f7d471f5700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d400c0700 con 0x7f7d400a0290 2021-02-10T11:30:12.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:12.208 7f7d471f5700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d400858f0 con 0x7f7d400c3820 2021-02-10T11:30:12.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:12.208 7f7d471f5700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d40085b10 con 0x7f7d40086270 2021-02-10T11:30:12.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:12.208 7f7d44810700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7d400a0290 0x7f7d400852d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:12.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:12.209 7f7d44810700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7d400a0290 0x7f7d400852d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:60838/0 (socket says 172.21.2.102:60838) 2021-02-10T11:30:12.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:12.209 7f7d44810700 1 -- 172.21.2.102:0/2914994879 learned_addr learned my addr 172.21.2.102:0/2914994879 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:30:12.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:12.209 7f7d44810700 1 -- 172.21.2.102:0/2914994879 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7d40086270 msgr2=0x7f7d40086670 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:30:12.213 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:12.209 7f7d44810700 1 --2- 172.21.2.102:0/2914994879 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7d40086270 0x7f7d40086670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:12.213 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:12.209 7f7d44810700 1 -- 172.21.2.102:0/2914994879 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7d400c3820 msgr2=0x7f7d40084cb0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:30:12.213 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:12.209 7f7d44810700 1 --2- 172.21.2.102:0/2914994879 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7d400c3820 0x7f7d40084cb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:12.213 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:12.209 7f7d44810700 1 -- 172.21.2.102:0/2914994879 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d40086c90 con 0x7f7d400a0290 2021-02-10T11:30:12.213 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:12.209 7f7d44810700 1 --2- 172.21.2.102:0/2914994879 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7d400a0290 0x7f7d400852d0 secure :-1 s=READY pgs=1952 cs=0 l=1 rev1=1 rx=0x7f7d3c006f20 tx=0x7f7d3c00b9b0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:12.213 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:12.209 7f7d31ffb700 1 -- 172.21.2.102:0/2914994879 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7d3c009100 con 0x7f7d400a0290 2021-02-10T11:30:12.213 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:12.209 7f7d31ffb700 1 -- 172.21.2.102:0/2914994879 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d3c00bda0 con 0x7f7d400a0290 2021-02-10T11:30:12.213 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:12.209 7f7d31ffb700 1 -- 172.21.2.102:0/2914994879 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7d3c0054d0 con 0x7f7d400a0290 2021-02-10T11:30:12.213 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:12.209 7f7d471f5700 1 -- 172.21.2.102:0/2914994879 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d40086c90 con 0x7f7d400a0290 2021-02-10T11:30:12.213 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:12.209 7f7d471f5700 1 -- 172.21.2.102:0/2914994879 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d401a2f00 con 0x7f7d400a0290 2021-02-10T11:30:12.213 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:12.210 7f7d31ffb700 1 -- 172.21.2.102:0/2914994879 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f7d3c006030 con 0x7f7d400a0290 2021-02-10T11:30:12.213 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:12.210 7f7d31ffb700 1 --2- 172.21.2.102:0/2914994879 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7d200186c0 0x7f7d2001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:12.213 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:12.210 7f7d31ffb700 1 -- 172.21.2.102:0/2914994879 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f7d3c032540 con 0x7f7d400a0290 2021-02-10T11:30:12.214 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:12.210 7f7d45011700 1 --2- 172.21.2.102:0/2914994879 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7d200186c0 0x7f7d2001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:12.214 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:12.210 7f7d45011700 1 --2- 172.21.2.102:0/2914994879 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7d200186c0 0x7f7d2001abc0 crc :-1 s=READY pgs=876 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:12.242 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:12.241 7f7d471f5700 1 -- 172.21.2.102:0/2914994879 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d40082a70 con 0x7f7d400a0290 2021-02-10T11:30:12.244 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:12.244 7f7d31ffb700 1 -- 172.21.2.102:0/2914994879 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f7d3c0064c0 con 0x7f7d400a0290 2021-02-10T11:30:12.374 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:12.373 7f7d471f5700 1 -- 172.21.2.102:0/2914994879 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f7d40082a70 con 0x7f7d400a0290 2021-02-10T11:30:12.374 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:12.374 7f7d31ffb700 1 -- 172.21.2.102:0/2914994879 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v93) v1 ==== 92+0+10 (secure 0 0 0) 0x7f7d3c0064c0 con 0x7f7d400a0290 2021-02-10T11:30:12.375 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:30:12.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:12.376 7f7d471f5700 1 -- 172.21.2.102:0/2914994879 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7d200186c0 msgr2=0x7f7d2001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:12.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:12.376 7f7d471f5700 1 --2- 172.21.2.102:0/2914994879 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7d200186c0 0x7f7d2001abc0 crc :-1 s=READY pgs=876 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:12.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:12.376 7f7d471f5700 1 -- 172.21.2.102:0/2914994879 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7d400a0290 msgr2=0x7f7d400852d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:12.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:12.376 7f7d471f5700 1 --2- 172.21.2.102:0/2914994879 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7d400a0290 0x7f7d400852d0 secure :-1 s=READY pgs=1952 cs=0 l=1 rev1=1 rx=0x7f7d3c006f20 tx=0x7f7d3c00b9b0).stop 2021-02-10T11:30:12.377 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:12.376 7f7d471f5700 1 -- 172.21.2.102:0/2914994879 shutdown_connections 2021-02-10T11:30:12.377 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:12.376 7f7d471f5700 1 --2- 172.21.2.102:0/2914994879 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7d40086270 0x7f7d40086670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:12.377 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:12.376 7f7d471f5700 1 --2- 172.21.2.102:0/2914994879 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7d400a0290 0x7f7d400852d0 unknown :-1 s=CLOSED pgs=1952 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:12.377 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:12.376 7f7d471f5700 1 --2- 172.21.2.102:0/2914994879 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7d200186c0 0x7f7d2001abc0 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:12.377 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:12.376 7f7d471f5700 1 --2- 172.21.2.102:0/2914994879 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7d400c3820 0x7f7d40084cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:12.377 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:12.376 7f7d471f5700 1 -- 172.21.2.102:0/2914994879 >> 172.21.2.102:0/2914994879 conn(0x7f7d400872c0 msgr2=0x7f7d40095700 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:12.377 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:12.376 7f7d471f5700 1 -- 172.21.2.102:0/2914994879 shutdown_connections 2021-02-10T11:30:12.377 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:12.376 7f7d471f5700 1 -- 172.21.2.102:0/2914994879 wait complete. 2021-02-10T11:30:12.385 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-02-10T11:30:12.818 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:12.801 7fd479a4a700 1 Processor -- start 2021-02-10T11:30:12.819 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:12.801 7fd479a4a700 1 -- start start 2021-02-10T11:30:12.819 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:12.801 7fd479a4a700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fd4740d1c60 0x7fd4740cef50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:12.819 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:12.801 7fd479a4a700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fd4740cf570 0x7fd4740cf970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:12.819 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:12.801 7fd479a4a700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4740fe1b0 con 0x7fd4740d1860 2021-02-10T11:30:12.819 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:12.801 7fd479a4a700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4741b0ca0 con 0x7fd4740d1c60 2021-02-10T11:30:12.819 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:12.801 7fd479a4a700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd474109a50 con 0x7fd4740cf570 2021-02-10T11:30:12.819 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:12.801 7fd472ffd700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fd4740d1c60 0x7fd4740cef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:12.820 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:12.801 7fd472ffd700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fd4740d1c60 0x7fd4740cef50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:60846/0 (socket says 172.21.2.102:60846) 2021-02-10T11:30:12.820 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:12.801 7fd472ffd700 1 -- 172.21.2.102:0/3715773009 learned_addr learned my addr 172.21.2.102:0/3715773009 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:30:12.820 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:12.802 7fd472ffd700 1 -- 172.21.2.102:0/3715773009 >> v2:172.21.2.113:3300/0 conn(0x7fd4740cf570 msgr2=0x7fd4740cf970 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:30:12.820 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:12.802 7fd472ffd700 1 --2- 172.21.2.102:0/3715773009 >> v2:172.21.2.113:3300/0 conn(0x7fd4740cf570 0x7fd4740cf970 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:12.820 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:12.802 7fd472ffd700 1 -- 172.21.2.102:0/3715773009 >> v1:172.21.2.113:6789/0 conn(0x7fd4740d1860 legacy=0x7fd4741b04b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:30:12.820 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:12.802 7fd472ffd700 1 -- 172.21.2.102:0/3715773009 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd474088070 con 0x7fd4740d1c60 2021-02-10T11:30:12.820 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:12.802 7fd472ffd700 1 --2- 172.21.2.102:0/3715773009 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd4740d1c60 0x7fd4740cef50 secure :-1 s=READY pgs=1953 cs=0 l=1 rev1=1 rx=0x7fd45c006700 tx=0x7fd45c009b80).ready entity=mon.0 client_cookie=bedd51c26cd2fab1 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:12.820 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:12.802 7fd4727fc700 1 -- 172.21.2.102:0/3715773009 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd45c00c0c0 con 0x7fd4740d1c60 2021-02-10T11:30:12.820 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:12.802 7fd4727fc700 1 -- 172.21.2.102:0/3715773009 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd45c00c380 con 0x7fd4740d1c60 2021-02-10T11:30:12.821 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:12.802 7fd4727fc700 1 -- 172.21.2.102:0/3715773009 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd45c00c8a0 con 0x7fd4740d1c60 2021-02-10T11:30:12.821 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:12.802 7fd479a4a700 1 -- 172.21.2.102:0/3715773009 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd4740d1c60 msgr2=0x7fd4740cef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:12.821 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:12.802 7fd479a4a700 1 --2- 172.21.2.102:0/3715773009 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd4740d1c60 0x7fd4740cef50 secure :-1 s=READY pgs=1953 cs=0 l=1 rev1=1 rx=0x7fd45c006700 tx=0x7fd45c009b80).stop 2021-02-10T11:30:12.821 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:12.803 7fd479a4a700 1 -- 172.21.2.102:0/3715773009 shutdown_connections 2021-02-10T11:30:12.821 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:12.803 7fd479a4a700 1 --2- 172.21.2.102:0/3715773009 >> v2:172.21.2.113:3300/0 conn(0x7fd4740cf570 0x7fd4740cf970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:12.821 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:12.803 7fd479a4a700 1 --2- 172.21.2.102:0/3715773009 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd4740d1c60 0x7fd4740cef50 unknown :-1 s=CLOSED pgs=1953 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:12.821 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:12.803 7fd479a4a700 1 -- 172.21.2.102:0/3715773009 >> 172.21.2.102:0/3715773009 conn(0x7fd4740ce410 msgr2=0x7fd4740d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:12.822 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:12.803 7fd479a4a700 1 -- 172.21.2.102:0/3715773009 shutdown_connections 2021-02-10T11:30:12.822 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:12.803 7fd479a4a700 1 -- 172.21.2.102:0/3715773009 wait complete. 2021-02-10T11:30:12.822 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:12.818 7fd479a4a700 1 Processor -- start 2021-02-10T11:30:12.822 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:12.818 7fd479a4a700 1 -- start start 2021-02-10T11:30:12.823 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:12.818 7fd479a4a700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd4741a05e0 0x7fd47418b970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:12.824 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:12.818 7fd479a4a700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd4740b8780 0x7fd4740bfa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:12.824 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:12.818 7fd479a4a700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd4740c0030 0x7fd474189e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:12.824 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:12.818 7fd479a4a700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4740b82a0 con 0x7fd4741a05e0 2021-02-10T11:30:12.824 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:12.818 7fd479a4a700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4740c0430 con 0x7fd4740c0030 2021-02-10T11:30:12.824 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:12.818 7fd479a4a700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4740c0650 con 0x7fd4740b8780 2021-02-10T11:30:12.824 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:12.818 7fd4737fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd4741a05e0 0x7fd47418b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:12.824 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:12.818 7fd4737fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd4741a05e0 0x7fd47418b970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:60852/0 (socket says 172.21.2.102:60852) 2021-02-10T11:30:12.824 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:12.818 7fd4737fe700 1 -- 172.21.2.102:0/2866833263 learned_addr learned my addr 172.21.2.102:0/2866833263 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:30:12.824 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:12.819 7fd4737fe700 1 -- 172.21.2.102:0/2866833263 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd4740b8780 msgr2=0x7fd4740bfa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:12.824 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:12.819 7fd4737fe700 1 --2- 172.21.2.102:0/2866833263 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd4740b8780 0x7fd4740bfa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:12.824 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:12.819 7fd4737fe700 1 -- 172.21.2.102:0/2866833263 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd4740c0030 msgr2=0x7fd474189e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:12.824 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:12.819 7fd4737fe700 1 --2- 172.21.2.102:0/2866833263 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd4740c0030 0x7fd474189e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:12.824 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:12.819 7fd4737fe700 1 -- 172.21.2.102:0/2866833263 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd47418a490 con 0x7fd4741a05e0 2021-02-10T11:30:12.825 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:12.819 7fd4737fe700 1 --2- 172.21.2.102:0/2866833263 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd4741a05e0 0x7fd47418b970 secure :-1 s=READY pgs=1954 cs=0 l=1 rev1=1 rx=0x7fd46400b530 tx=0x7fd46400b8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:12.825 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:12.819 7fd46bfff700 1 -- 172.21.2.102:0/2866833263 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd46400bcc0 con 0x7fd4741a05e0 2021-02-10T11:30:12.825 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:12.819 7fd46bfff700 1 -- 172.21.2.102:0/2866833263 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd464006020 con 0x7fd4741a05e0 2021-02-10T11:30:12.825 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:12.819 7fd46bfff700 1 -- 172.21.2.102:0/2866833263 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd4640054a0 con 0x7fd4741a05e0 2021-02-10T11:30:12.825 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:12.819 7fd479a4a700 1 -- 172.21.2.102:0/2866833263 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd47418a490 con 0x7fd4741a05e0 2021-02-10T11:30:12.825 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:12.819 7fd479a4a700 1 -- 172.21.2.102:0/2866833263 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd47418a490 con 0x7fd4741a05e0 2021-02-10T11:30:12.825 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:12.820 7fd46bfff700 1 -- 172.21.2.102:0/2866833263 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd464006200 con 0x7fd4741a05e0 2021-02-10T11:30:12.825 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:12.820 7fd46bfff700 1 --2- 172.21.2.102:0/2866833263 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd4580186c0 0x7fd45801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:12.825 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:12.820 7fd46bfff700 1 -- 172.21.2.102:0/2866833263 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fd4640325d0 con 0x7fd4741a05e0 2021-02-10T11:30:12.825 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:12.820 7fd472ffd700 1 --2- 172.21.2.102:0/2866833263 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd4580186c0 0x7fd45801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:12.825 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:12.820 7fd472ffd700 1 --2- 172.21.2.102:0/2866833263 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd4580186c0 0x7fd45801abc0 crc :-1 s=READY pgs=877 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:12.851 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:12.851 7fd479a4a700 1 -- 172.21.2.102:0/2866833263 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd474082a20 con 0x7fd4741a05e0 2021-02-10T11:30:12.854 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:12.853 7fd46bfff700 1 -- 172.21.2.102:0/2866833263 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd464030d80 con 0x7fd4741a05e0 2021-02-10T11:30:12.982 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:12.982 7fd479a4a700 1 -- 172.21.2.102:0/2866833263 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7fd474082a20 con 0x7fd4741a05e0 2021-02-10T11:30:12.982 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:12.982 7fd46bfff700 1 -- 172.21.2.102:0/2866833263 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v93) v1 ==== 104+0+10 (secure 0 0 0) 0x7fd46400bcc0 con 0x7fd4741a05e0 2021-02-10T11:30:12.983 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:30:12.984 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:12.984 7fd479a4a700 1 -- 172.21.2.102:0/2866833263 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd4580186c0 msgr2=0x7fd45801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:12.985 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:12.984 7fd479a4a700 1 --2- 172.21.2.102:0/2866833263 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd4580186c0 0x7fd45801abc0 crc :-1 s=READY pgs=877 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:12.985 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:12.984 7fd479a4a700 1 -- 172.21.2.102:0/2866833263 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd4741a05e0 msgr2=0x7fd47418b970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:12.985 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:12.984 7fd479a4a700 1 --2- 172.21.2.102:0/2866833263 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd4741a05e0 0x7fd47418b970 secure :-1 s=READY pgs=1954 cs=0 l=1 rev1=1 rx=0x7fd46400b530 tx=0x7fd46400b8d0).stop 2021-02-10T11:30:12.985 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:12.984 7fd479a4a700 1 -- 172.21.2.102:0/2866833263 shutdown_connections 2021-02-10T11:30:12.985 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:12.984 7fd479a4a700 1 --2- 172.21.2.102:0/2866833263 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd4580186c0 0x7fd45801abc0 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:12.985 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:12.984 7fd479a4a700 1 --2- 172.21.2.102:0/2866833263 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd4740c0030 0x7fd474189e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:12.985 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:12.984 7fd479a4a700 1 --2- 172.21.2.102:0/2866833263 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd4740b8780 0x7fd4740bfa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:12.985 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:12.984 7fd479a4a700 1 --2- 172.21.2.102:0/2866833263 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd4741a05e0 0x7fd47418b970 unknown :-1 s=CLOSED pgs=1954 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:12.985 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:12.984 7fd479a4a700 1 -- 172.21.2.102:0/2866833263 >> 172.21.2.102:0/2866833263 conn(0x7fd4740a6a80 msgr2=0x7fd4740ae110 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:12.985 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:12.984 7fd479a4a700 1 -- 172.21.2.102:0/2866833263 shutdown_connections 2021-02-10T11:30:12.985 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:12.984 7fd479a4a700 1 -- 172.21.2.102:0/2866833263 wait complete. 2021-02-10T11:30:12.994 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-02-10T11:30:13.430 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:13.409 7f8e979f8700 1 Processor -- start 2021-02-10T11:30:13.430 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:13.409 7f8e979f8700 1 -- start start 2021-02-10T11:30:13.430 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:13.409 7f8e979f8700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f8e900d1870 0x7f8e901b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:13.430 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:13.409 7f8e979f8700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e900fe1c0 con 0x7f8e900cff70 2021-02-10T11:30:13.431 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:13.409 7f8e979f8700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e90109a60 con 0x7f8e901b0ae0 2021-02-10T11:30:13.431 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:13.409 7f8e979f8700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e90088070 con 0x7f8e900d1870 2021-02-10T11:30:13.431 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:13.410 7f8e96015700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f8e900cff70 0x7f8e90109130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:53084/0 (socket says 172.21.2.102:53084) 2021-02-10T11:30:13.431 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:13.410 7f8e96015700 1 -- 172.21.2.102:0/1228637066 learned_addr learned my addr 172.21.2.102:0/1228637066 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:30:13.431 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:13.410 7f8e94812700 1 -- 172.21.2.102:0/1228637066 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 70923905 0 0) 0x7f8e88001500 con 0x7f8e900cff70 2021-02-10T11:30:13.431 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:13.410 7f8e94812700 1 -- 172.21.2.102:0/1228637066 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e7c002fe0 con 0x7f8e900cff70 2021-02-10T11:30:13.431 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:13.410 7f8e95814700 1 --2- 172.21.2.102:0/1228637066 >> v2:172.21.2.113:3300/0 conn(0x7f8e900d1870 0x7f8e901b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:13.431 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:13.410 7f8e94812700 1 -- 172.21.2.102:0/1228637066 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2212354617 0 0) 0x7f8e88001460 con 0x7f8e900cff70 2021-02-10T11:30:13.431 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:13.410 7f8e94812700 1 -- 172.21.2.102:0/1228637066 >> v2:172.21.2.113:3300/0 conn(0x7f8e900d1870 msgr2=0x7f8e901b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:13.431 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:13.410 7f8e94812700 1 --2- 172.21.2.102:0/1228637066 >> v2:172.21.2.113:3300/0 conn(0x7f8e900d1870 0x7f8e901b04c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:13.431 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:13.410 7f8e94812700 1 -- 172.21.2.102:0/1228637066 >> v1:172.21.2.113:6789/0 conn(0x7f8e901b0ae0 legacy=0x7f8e90108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:13.431 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:13.410 7f8e94812700 1 -- 172.21.2.102:0/1228637066 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e90088260 con 0x7f8e900cff70 2021-02-10T11:30:13.431 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:13.410 7f8e94812700 1 -- 172.21.2.102:0/1228637066 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7f8e88001980 con 0x7f8e900cff70 2021-02-10T11:30:13.431 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:13.410 7f8e94812700 1 -- 172.21.2.102:0/1228637066 >> v1:172.21.2.102:6789/0 conn(0x7f8e900cff70 legacy=0x7f8e90109130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:13.432 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:13.410 7f8e94812700 1 --2- 172.21.2.102:0/1228637066 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8e7c003ac0 0x7f8e7c005f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:13.432 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:13.410 7f8e94812700 1 --2- 172.21.2.102:0/1228637066 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8e7c006780 0x7f8e7c008ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:13.432 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:13.410 7f8e94812700 1 --2- 172.21.2.102:0/1228637066 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8e7c0095a0 0x7f8e7c00baa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:13.432 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:13.410 7f8e94812700 1 -- 172.21.2.102:0/1228637066 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e7c00c2f0 con 0x7f8e7c006780 2021-02-10T11:30:13.432 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:13.410 7f8e94812700 1 -- 172.21.2.102:0/1228637066 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e7c00c510 con 0x7f8e7c0095a0 2021-02-10T11:30:13.432 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:13.410 7f8e94812700 1 -- 172.21.2.102:0/1228637066 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e7c050740 con 0x7f8e7c003ac0 2021-02-10T11:30:13.432 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:13.411 7f8e95013700 1 --2- 172.21.2.102:0/1228637066 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8e7c006780 0x7f8e7c008ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:13.432 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:13.411 7f8e96015700 1 --2- 172.21.2.102:0/1228637066 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8e7c0095a0 0x7f8e7c00baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:13.432 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:13.411 7f8e95814700 1 --2- 172.21.2.102:0/1228637066 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8e7c003ac0 0x7f8e7c005f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:13.432 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:13.411 7f8e95013700 1 -- 172.21.2.102:0/1228637066 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8e7c003ac0 msgr2=0x7f8e7c005f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:13.432 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:13.411 7f8e95013700 1 --2- 172.21.2.102:0/1228637066 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8e7c003ac0 0x7f8e7c005f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:13.432 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:13.411 7f8e95013700 1 -- 172.21.2.102:0/1228637066 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8e7c0095a0 msgr2=0x7f8e7c00baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:13.433 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:13.411 7f8e95013700 1 --2- 172.21.2.102:0/1228637066 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8e7c0095a0 0x7f8e7c00baa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:13.433 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:13.411 7f8e95013700 1 -- 172.21.2.102:0/1228637066 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e7c050970 con 0x7f8e7c006780 2021-02-10T11:30:13.433 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:13.411 7f8e96015700 1 -- 172.21.2.102:0/1228637066 reap_dead start 2021-02-10T11:30:13.433 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:13.411 7f8e95814700 1 --2- 172.21.2.102:0/1228637066 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8e7c003ac0 0x7f8e7c005f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:30:13.433 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:13.411 7f8e95013700 1 --2- 172.21.2.102:0/1228637066 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8e7c006780 0x7f8e7c008ca0 secure :-1 s=READY pgs=1956 cs=0 l=1 rev1=1 rx=0x7f8e8c006ac0 tx=0x7f8e8c009af0).ready entity=mon.0 client_cookie=34c8a22a6cfb593a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:13.433 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:13.411 7f8e94812700 1 -- 172.21.2.102:0/1228637066 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8e8c007280 con 0x7f8e7c006780 2021-02-10T11:30:13.433 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:13.411 7f8e94812700 1 -- 172.21.2.102:0/1228637066 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e8c007600 con 0x7f8e7c006780 2021-02-10T11:30:13.433 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:13.411 7f8e94812700 1 -- 172.21.2.102:0/1228637066 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8e8c007a50 con 0x7f8e7c006780 2021-02-10T11:30:13.433 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:13.411 7f8e979f8700 1 -- 172.21.2.102:0/1228637066 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8e7c006780 msgr2=0x7f8e7c008ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:13.433 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:13.411 7f8e979f8700 1 --2- 172.21.2.102:0/1228637066 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8e7c006780 0x7f8e7c008ca0 secure :-1 s=READY pgs=1956 cs=0 l=1 rev1=1 rx=0x7f8e8c006ac0 tx=0x7f8e8c009af0).stop 2021-02-10T11:30:13.433 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:13.412 7f8e979f8700 1 -- 172.21.2.102:0/1228637066 shutdown_connections 2021-02-10T11:30:13.433 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:13.412 7f8e979f8700 1 --2- 172.21.2.102:0/1228637066 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8e7c006780 0x7f8e7c008ca0 unknown :-1 s=CLOSED pgs=1956 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:13.433 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:13.412 7f8e979f8700 1 -- 172.21.2.102:0/1228637066 >> 172.21.2.102:0/1228637066 conn(0x7f8e900ce420 msgr2=0x7f8e900d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:13.434 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:13.412 7f8e979f8700 1 -- 172.21.2.102:0/1228637066 shutdown_connections 2021-02-10T11:30:13.434 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:13.412 7f8e979f8700 1 -- 172.21.2.102:0/1228637066 wait complete. 2021-02-10T11:30:13.434 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:13.427 7f8e979f8700 1 Processor -- start 2021-02-10T11:30:13.434 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:13.427 7f8e979f8700 1 -- start start 2021-02-10T11:30:13.434 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:13.427 7f8e979f8700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8e9009bcc0 0x7f8e9009c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:13.434 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:13.427 7f8e979f8700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8e900a0820 0x7f8e900a0c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:13.434 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:13.427 7f8e979f8700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8e900a1240 0x7f8e900858d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:13.434 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:13.427 7f8e979f8700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e900a66b0 con 0x7f8e900a0820 2021-02-10T11:30:13.434 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:13.427 7f8e979f8700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e901a49b0 con 0x7f8e900a1240 2021-02-10T11:30:13.434 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:13.427 7f8e979f8700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e901a4b90 con 0x7f8e9009bcc0 2021-02-10T11:30:13.434 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:13.427 7f8e95013700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8e900a0820 0x7f8e900a0c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:13.434 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:13.427 7f8e95013700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8e900a0820 0x7f8e900a0c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:60874/0 (socket says 172.21.2.102:60874) 2021-02-10T11:30:13.434 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:13.427 7f8e95013700 1 -- 172.21.2.102:0/2920760260 learned_addr learned my addr 172.21.2.102:0/2920760260 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:30:13.435 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:13.428 7f8e95013700 1 -- 172.21.2.102:0/2920760260 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8e9009bcc0 msgr2=0x7f8e9009c0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:13.435 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:13.428 7f8e95013700 1 --2- 172.21.2.102:0/2920760260 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8e9009bcc0 0x7f8e9009c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:13.435 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:13.428 7f8e95013700 1 -- 172.21.2.102:0/2920760260 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8e900a1240 msgr2=0x7f8e900858d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:13.435 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:13.428 7f8e95013700 1 --2- 172.21.2.102:0/2920760260 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8e900a1240 0x7f8e900858d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:13.435 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:13.428 7f8e95013700 1 -- 172.21.2.102:0/2920760260 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e900a1640 con 0x7f8e900a0820 2021-02-10T11:30:13.435 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:13.428 7f8e95013700 1 --2- 172.21.2.102:0/2920760260 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8e900a0820 0x7f8e900a0c20 secure :-1 s=READY pgs=1957 cs=0 l=1 rev1=1 rx=0x7f8e8c00be80 tx=0x7f8e8c00d8e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:13.435 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:13.428 7f8e827fc700 1 -- 172.21.2.102:0/2920760260 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8e8c0009b0 con 0x7f8e900a0820 2021-02-10T11:30:13.435 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:13.428 7f8e827fc700 1 -- 172.21.2.102:0/2920760260 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e8c000d00 con 0x7f8e900a0820 2021-02-10T11:30:13.435 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:13.428 7f8e827fc700 1 -- 172.21.2.102:0/2920760260 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8e8c00b0d0 con 0x7f8e900a0820 2021-02-10T11:30:13.435 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:13.428 7f8e979f8700 1 -- 172.21.2.102:0/2920760260 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e900a1640 con 0x7f8e900a0820 2021-02-10T11:30:13.435 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:13.429 7f8e979f8700 1 -- 172.21.2.102:0/2920760260 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e900a1640 con 0x7f8e900a0820 2021-02-10T11:30:13.435 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:13.429 7f8e827fc700 1 -- 172.21.2.102:0/2920760260 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f8e8c001020 con 0x7f8e900a0820 2021-02-10T11:30:13.435 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:13.429 7f8e827fc700 1 --2- 172.21.2.102:0/2920760260 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8e780186c0 0x7f8e7801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:13.436 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:13.429 7f8e827fc700 1 -- 172.21.2.102:0/2920760260 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f8e8c032560 con 0x7f8e900a0820 2021-02-10T11:30:13.436 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:13.429 7f8e95814700 1 --2- 172.21.2.102:0/2920760260 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8e780186c0 0x7f8e7801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:13.436 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:13.430 7f8e95814700 1 --2- 172.21.2.102:0/2920760260 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8e780186c0 0x7f8e7801abc0 crc :-1 s=READY pgs=878 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:13.460 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:13.460 7f8e979f8700 1 -- 172.21.2.102:0/2920760260 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e900fe1c0 con 0x7f8e900a0820 2021-02-10T11:30:13.462 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:13.462 7f8e827fc700 1 -- 172.21.2.102:0/2920760260 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f8e8c0009b0 con 0x7f8e900a0820 2021-02-10T11:30:13.590 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:13.589 7f8e979f8700 1 -- 172.21.2.102:0/2920760260 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7f8e900fe1c0 con 0x7f8e900a0820 2021-02-10T11:30:13.590 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:13.590 7f8e827fc700 1 -- 172.21.2.102:0/2920760260 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v93) v1 ==== 100+0+10 (secure 0 0 0) 0x7f8e8c00dcd0 con 0x7f8e900a0820 2021-02-10T11:30:13.591 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:30:13.592 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:13.591 7f8e979f8700 1 -- 172.21.2.102:0/2920760260 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8e780186c0 msgr2=0x7f8e7801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:13.592 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:13.591 7f8e979f8700 1 --2- 172.21.2.102:0/2920760260 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8e780186c0 0x7f8e7801abc0 crc :-1 s=READY pgs=878 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:13.592 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:13.591 7f8e979f8700 1 -- 172.21.2.102:0/2920760260 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8e900a0820 msgr2=0x7f8e900a0c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:13.593 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:13.591 7f8e979f8700 1 --2- 172.21.2.102:0/2920760260 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8e900a0820 0x7f8e900a0c20 secure :-1 s=READY pgs=1957 cs=0 l=1 rev1=1 rx=0x7f8e8c00be80 tx=0x7f8e8c00d8e0).stop 2021-02-10T11:30:13.593 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:13.592 7f8e979f8700 1 -- 172.21.2.102:0/2920760260 shutdown_connections 2021-02-10T11:30:13.593 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:13.592 7f8e979f8700 1 --2- 172.21.2.102:0/2920760260 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8e780186c0 0x7f8e7801abc0 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:13.593 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:13.592 7f8e979f8700 1 --2- 172.21.2.102:0/2920760260 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8e900a1240 0x7f8e900858d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:13.593 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:13.592 7f8e979f8700 1 --2- 172.21.2.102:0/2920760260 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8e900a0820 0x7f8e900a0c20 unknown :-1 s=CLOSED pgs=1957 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:13.593 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:13.592 7f8e979f8700 1 --2- 172.21.2.102:0/2920760260 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8e9009bcc0 0x7f8e9009c0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:13.593 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:13.592 7f8e979f8700 1 -- 172.21.2.102:0/2920760260 >> 172.21.2.102:0/2920760260 conn(0x7f8e900bef00 msgr2=0x7f8e9009d1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:13.593 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:13.592 7f8e979f8700 1 -- 172.21.2.102:0/2920760260 shutdown_connections 2021-02-10T11:30:13.593 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:13.592 7f8e979f8700 1 -- 172.21.2.102:0/2920760260 wait complete. 2021-02-10T11:30:13.601 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-02-10T11:30:13.601 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:30:14.028 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:14.011 7f8e7c711700 1 Processor -- start 2021-02-10T11:30:14.028 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:14.011 7f8e7c711700 1 -- start start 2021-02-10T11:30:14.028 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:14.012 7f8e7c711700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f8e740d1870 0x7f8e741b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:14.028 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:14.012 7f8e7c711700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8e741b0a10 0x7f8e740cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:14.029 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:14.012 7f8e7c711700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f8e740d1c70 0x7f8e740cf950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:14.029 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:14.012 7f8e7c711700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e740fe200 con 0x7f8e740d1c70 2021-02-10T11:30:14.029 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:14.012 7f8e7c711700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e740e5f10 con 0x7f8e740d1870 2021-02-10T11:30:14.029 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:14.012 7f8e7c711700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e740f0b50 con 0x7f8e741b0a10 2021-02-10T11:30:14.029 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:14.012 7f8e7ad2e700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f8e740d1c70 0x7f8e740cf950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:14.029 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:14.012 7f8e7ad2e700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f8e740d1c70 0x7f8e740cf950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:60878/0 (socket says 172.21.2.102:60878) 2021-02-10T11:30:14.029 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:14.012 7f8e7ad2e700 1 -- 172.21.2.102:0/167195029 learned_addr learned my addr 172.21.2.102:0/167195029 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:30:14.029 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:14.012 7f8e7a52d700 1 --2- 172.21.2.102:0/167195029 >> v2:172.21.2.113:3300/0 conn(0x7f8e740d1870 0x7f8e741b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:14.029 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:14.012 7f8e79d2c700 1 --2- 172.21.2.102:0/167195029 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8e741b0a10 0x7f8e740cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:14.029 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:14.012 7f8e7ad2e700 1 -- 172.21.2.102:0/167195029 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8e741b0a10 msgr2=0x7f8e740cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:14.030 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:14.012 7f8e7ad2e700 1 --2- 172.21.2.102:0/167195029 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8e741b0a10 0x7f8e740cf330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:14.030 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:14.012 7f8e7ad2e700 1 -- 172.21.2.102:0/167195029 >> v2:172.21.2.113:3300/0 conn(0x7f8e740d1870 msgr2=0x7f8e741b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:14.030 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:14.012 7f8e7ad2e700 1 --2- 172.21.2.102:0/167195029 >> v2:172.21.2.113:3300/0 conn(0x7f8e740d1870 0x7f8e741b03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:14.030 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:14.012 7f8e7ad2e700 1 -- 172.21.2.102:0/167195029 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e740cff70 con 0x7f8e740d1c70 2021-02-10T11:30:14.030 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:14.012 7f8e7a52d700 1 --2- 172.21.2.102:0/167195029 >> v2:172.21.2.113:3300/0 conn(0x7f8e740d1870 0x7f8e741b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:30:14.030 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:14.012 7f8e79d2c700 1 --2- 172.21.2.102:0/167195029 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8e741b0a10 0x7f8e740cf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:30:14.030 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:14.012 7f8e7ad2e700 1 --2- 172.21.2.102:0/167195029 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8e740d1c70 0x7f8e740cf950 secure :-1 s=READY pgs=1958 cs=0 l=1 rev1=1 rx=0x7f8e6c006f00 tx=0x7f8e6c009f10).ready entity=mon.0 client_cookie=17492ea72e078c75 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:14.030 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:14.013 7f8e7952b700 1 -- 172.21.2.102:0/167195029 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8e6c00b3b0 con 0x7f8e740d1c70 2021-02-10T11:30:14.030 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:14.013 7f8e7952b700 1 -- 172.21.2.102:0/167195029 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e6c00b6b0 con 0x7f8e740d1c70 2021-02-10T11:30:14.031 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:14.013 7f8e7952b700 1 -- 172.21.2.102:0/167195029 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8e6c00bbd0 con 0x7f8e740d1c70 2021-02-10T11:30:14.031 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:14.013 7f8e7c711700 1 -- 172.21.2.102:0/167195029 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8e740d1c70 msgr2=0x7f8e740cf950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:14.031 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:14.013 7f8e7c711700 1 --2- 172.21.2.102:0/167195029 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8e740d1c70 0x7f8e740cf950 secure :-1 s=READY pgs=1958 cs=0 l=1 rev1=1 rx=0x7f8e6c006f00 tx=0x7f8e6c009f10).stop 2021-02-10T11:30:14.031 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:14.013 7f8e7c711700 1 -- 172.21.2.102:0/167195029 shutdown_connections 2021-02-10T11:30:14.031 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:14.013 7f8e7c711700 1 --2- 172.21.2.102:0/167195029 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8e741b0a10 0x7f8e740cf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:14.031 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:14.013 7f8e7c711700 1 --2- 172.21.2.102:0/167195029 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8e740d1c70 0x7f8e740cf950 unknown :-1 s=CLOSED pgs=1958 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:14.031 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:14.013 7f8e7c711700 1 --2- 172.21.2.102:0/167195029 >> v2:172.21.2.113:3300/0 conn(0x7f8e740d1870 0x7f8e741b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:14.031 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:14.013 7f8e7c711700 1 -- 172.21.2.102:0/167195029 >> 172.21.2.102:0/167195029 conn(0x7f8e740ce450 msgr2=0x7f8e740d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:14.031 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:14.013 7f8e7c711700 1 -- 172.21.2.102:0/167195029 shutdown_connections 2021-02-10T11:30:14.031 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:14.014 7f8e7c711700 1 -- 172.21.2.102:0/167195029 wait complete. 2021-02-10T11:30:14.033 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:14.028 7f8e7c711700 1 Processor -- start 2021-02-10T11:30:14.033 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:14.028 7f8e7c711700 1 -- start start 2021-02-10T11:30:14.033 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:14.028 7f8e7c711700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8e740f0280 0x7f8e740a2310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:14.034 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:14.028 7f8e7c711700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8e740f0680 0x7f8e740a2930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:14.034 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:14.028 7f8e7c711700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8e740ec180 0x7f8e740ec580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:14.034 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:14.028 7f8e7c711700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e740efd00 con 0x7f8e740ec180 2021-02-10T11:30:14.034 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:14.028 7f8e7c711700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e740ecba0 con 0x7f8e740f0680 2021-02-10T11:30:14.034 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:14.028 7f8e7c711700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e740ecd80 con 0x7f8e740f0280 2021-02-10T11:30:14.034 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:14.029 7f8e7ad2e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8e740ec180 0x7f8e740ec580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:14.034 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:14.029 7f8e7ad2e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8e740ec180 0x7f8e740ec580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:60886/0 (socket says 172.21.2.102:60886) 2021-02-10T11:30:14.034 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:14.029 7f8e7ad2e700 1 -- 172.21.2.102:0/2675211181 learned_addr learned my addr 172.21.2.102:0/2675211181 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:30:14.034 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:14.029 7f8e7ad2e700 1 -- 172.21.2.102:0/2675211181 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8e740f0280 msgr2=0x7f8e740a2310 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:30:14.034 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:14.029 7f8e7ad2e700 1 --2- 172.21.2.102:0/2675211181 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8e740f0280 0x7f8e740a2310 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:14.035 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:14.029 7f8e7ad2e700 1 -- 172.21.2.102:0/2675211181 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8e740f0680 msgr2=0x7f8e740a2930 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:30:14.035 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:14.029 7f8e7ad2e700 1 --2- 172.21.2.102:0/2675211181 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8e740f0680 0x7f8e740a2930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:14.035 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:14.029 7f8e7ad2e700 1 -- 172.21.2.102:0/2675211181 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e740ecfa0 con 0x7f8e740ec180 2021-02-10T11:30:14.035 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:14.029 7f8e7ad2e700 1 --2- 172.21.2.102:0/2675211181 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8e740ec180 0x7f8e740ec580 secure :-1 s=READY pgs=1959 cs=0 l=1 rev1=1 rx=0x7f8e6c00b970 tx=0x7f8e6c00bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:14.035 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:14.029 7f8e6affd700 1 -- 172.21.2.102:0/2675211181 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8e6c000f90 con 0x7f8e740ec180 2021-02-10T11:30:14.035 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:14.029 7f8e6affd700 1 -- 172.21.2.102:0/2675211181 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e6c000cc0 con 0x7f8e740ec180 2021-02-10T11:30:14.035 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:14.030 7f8e7c711700 1 -- 172.21.2.102:0/2675211181 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e740ecfa0 con 0x7f8e740ec180 2021-02-10T11:30:14.035 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:14.030 7f8e6affd700 1 -- 172.21.2.102:0/2675211181 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8e6c0014a0 con 0x7f8e740ec180 2021-02-10T11:30:14.035 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:14.030 7f8e7c711700 1 -- 172.21.2.102:0/2675211181 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e740ed530 con 0x7f8e740ec180 2021-02-10T11:30:14.035 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:14.030 7f8e6affd700 1 -- 172.21.2.102:0/2675211181 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f8e6c005820 con 0x7f8e740ec180 2021-02-10T11:30:14.035 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:14.030 7f8e6affd700 1 --2- 172.21.2.102:0/2675211181 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8e5c018730 0x7f8e5c01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:14.036 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:14.030 7f8e6affd700 1 -- 172.21.2.102:0/2675211181 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f8e6c032b30 con 0x7f8e740ec180 2021-02-10T11:30:14.036 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:14.030 7f8e7a52d700 1 --2- 172.21.2.102:0/2675211181 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8e5c018730 0x7f8e5c01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:14.036 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:14.031 7f8e7a52d700 1 --2- 172.21.2.102:0/2675211181 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8e5c018730 0x7f8e5c01ac10 crc :-1 s=READY pgs=879 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:14.062 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:14.061 7f8e7c711700 1 -- 172.21.2.102:0/2675211181 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e740ebc60 con 0x7f8e740ec180 2021-02-10T11:30:14.064 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:14.064 7f8e6affd700 1 -- 172.21.2.102:0/2675211181 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f8e6c000f90 con 0x7f8e740ec180 2021-02-10T11:30:14.192 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:14.192 7f8e7c711700 1 -- 172.21.2.102:0/2675211181 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8e740f0b10 con 0x7f8e740ec180 2021-02-10T11:30:14.193 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:14.192 7f8e6affd700 1 -- 172.21.2.102:0/2675211181 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) v1 ==== 94+0+2098 (secure 0 0 0) 0x7f8e6c005820 con 0x7f8e740ec180 2021-02-10T11:30:14.193 INFO:teuthology.orchestra.run.gibba002.stderr:dumped fsmap epoch 73 2021-02-10T11:30:14.193 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:30:14.194 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":73,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9672,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.102:6835/2071608955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6834","nonce":2071608955},{"type":"v1","addr":"172.21.2.102:6835","nonce":2071608955}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":69}],"filesystems":[{"mdsmap":{"epoch":73,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:30:04.225997","modified":"2021-02-10 11:30:06.232546","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7186},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7186":{"gid":7186,"name":"b","rank":0,"incarnation":70,"state":"up:active","state_seq":3,"addr":"172.21.2.113:6833/675492649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6832","nonce":675492649},{"type":"v1","addr":"172.21.2.113:6833","nonce":675492649}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":9}]} 2021-02-10T11:30:14.195 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:14.194 7f8e7c711700 1 -- 172.21.2.102:0/2675211181 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8e5c018730 msgr2=0x7f8e5c01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:14.195 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:14.194 7f8e7c711700 1 --2- 172.21.2.102:0/2675211181 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8e5c018730 0x7f8e5c01ac10 crc :-1 s=READY pgs=879 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:14.195 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:14.194 7f8e7c711700 1 -- 172.21.2.102:0/2675211181 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8e740ec180 msgr2=0x7f8e740ec580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:14.195 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:14.194 7f8e7c711700 1 --2- 172.21.2.102:0/2675211181 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8e740ec180 0x7f8e740ec580 secure :-1 s=READY pgs=1959 cs=0 l=1 rev1=1 rx=0x7f8e6c00b970 tx=0x7f8e6c00bd10).stop 2021-02-10T11:30:14.195 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:14.194 7f8e7c711700 1 -- 172.21.2.102:0/2675211181 shutdown_connections 2021-02-10T11:30:14.195 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:14.194 7f8e7c711700 1 --2- 172.21.2.102:0/2675211181 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8e740ec180 0x7f8e740ec580 unknown :-1 s=CLOSED pgs=1959 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:14.195 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:14.194 7f8e7c711700 1 --2- 172.21.2.102:0/2675211181 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8e5c018730 0x7f8e5c01ac10 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:14.196 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:14.194 7f8e7c711700 1 --2- 172.21.2.102:0/2675211181 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8e740f0680 0x7f8e740a2930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:14.196 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:14.194 7f8e7c711700 1 --2- 172.21.2.102:0/2675211181 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8e740f0280 0x7f8e740a2310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:14.196 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:14.194 7f8e7c711700 1 -- 172.21.2.102:0/2675211181 >> 172.21.2.102:0/2675211181 conn(0x7f8e740c2130 msgr2=0x7f8e7409e3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:14.196 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:14.194 7f8e7c711700 1 -- 172.21.2.102:0/2675211181 shutdown_connections 2021-02-10T11:30:14.196 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:14.194 7f8e7c711700 1 -- 172.21.2.102:0/2675211181 wait complete. 2021-02-10T11:30:14.404 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 73, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:30:04.225997', 'modified': '2021-02-10 11:30:06.232546', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7186}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7186': {'gid': 7186, 'name': 'b', 'rank': 0, 'incarnation': 70, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.113:6833/675492649', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.113:6832', 'nonce': 675492649}, {'type': 'v1', 'addr': '172.21.2.113:6833', 'nonce': 675492649}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:30:14.405 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:30:14.405 DEBUG:teuthology.orchestra.run.gibba013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:30:14.803 INFO:teuthology.orchestra.run.gibba013.stdout:{ 2021-02-10T11:30:14.804 INFO:teuthology.orchestra.run.gibba013.stdout: "cluster_fsid": "68db30ea-dbce-4aef-a6df-f01c6fea0513", 2021-02-10T11:30:14.804 INFO:teuthology.orchestra.run.gibba013.stdout: "whoami": 0, 2021-02-10T11:30:14.804 INFO:teuthology.orchestra.run.gibba013.stdout: "id": 7186, 2021-02-10T11:30:14.804 INFO:teuthology.orchestra.run.gibba013.stdout: "want_state": "up:active", 2021-02-10T11:30:14.804 INFO:teuthology.orchestra.run.gibba013.stdout: "state": "up:active", 2021-02-10T11:30:14.804 INFO:teuthology.orchestra.run.gibba013.stdout: "rank_uptime": 10.575677520999999, 2021-02-10T11:30:14.804 INFO:teuthology.orchestra.run.gibba013.stdout: "mdsmap_epoch": 73, 2021-02-10T11:30:14.804 INFO:teuthology.orchestra.run.gibba013.stdout: "osdmap_epoch": 93, 2021-02-10T11:30:14.804 INFO:teuthology.orchestra.run.gibba013.stdout: "osdmap_epoch_barrier": 93, 2021-02-10T11:30:14.804 INFO:teuthology.orchestra.run.gibba013.stdout: "uptime": 16.235222018000002 2021-02-10T11:30:14.804 INFO:teuthology.orchestra.run.gibba013.stdout:} 2021-02-10T11:30:14.807 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "68db30ea-dbce-4aef-a6df-f01c6fea0513", "whoami": 0, "id": 7186, "want_state": "up:active", "state": "up:active", "rank_uptime": 10.575677520999999, "mdsmap_epoch": 73, "osdmap_epoch": 93, "osdmap_epoch_barrier": 93, "uptime": 16.235222018000002 } 2021-02-10T11:30:14.808 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2021-02-10T11:30:14.808 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2021-02-10T11:30:14.808 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@gibba002.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2021-02-10T11:30:14.808 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:30:14.808 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec mkdir -p /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:30:15.143 DEBUG:teuthology.orchestra.run.gibba002:> sudo modprobe fuse 2021-02-10T11:30:15.489 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:30:15.489 DEBUG:teuthology.orchestra.run.gibba002:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:30:15.833 INFO:teuthology.orchestra.run.gibba002.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:30:15.834 INFO:teuthology.orchestra.run.gibba002.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:30:15.834 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:30:15.835 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:30:15.835 DEBUG:teuthology.orchestra.run.gibba002:> ls /sys/fs/fuse/connections 2021-02-10T11:30:16.170 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2021-02-10T11:30:16.170 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:30:16.173 DEBUG:teuthology.orchestra.run.gibba002:> sudo modprobe fuse 2021-02-10T11:30:16.517 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:30:16.518 DEBUG:teuthology.orchestra.run.gibba002:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:30:16.585 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba002.stderr:2021-02-10 11:30:16.586 7f409a948f80 -1 init, newargv = 0x563918eb91a0 newargc=9 2021-02-10T11:30:16.585 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba002.stderr:ceph-fuse[174201]: starting ceph client 2021-02-10T11:30:16.591 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba002.stderr:ceph-fuse[174201]: starting fuse 2021-02-10T11:30:16.860 INFO:teuthology.orchestra.run.gibba002.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:30:16.860 INFO:teuthology.orchestra.run.gibba002.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:30:16.861 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:30:16.861 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:30:16.861 DEBUG:teuthology.orchestra.run.gibba002:> ls /sys/fs/fuse/connections 2021-02-10T11:30:17.197 INFO:teuthology.orchestra.run.gibba002.stdout:43 2021-02-10T11:30:17.198 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2021-02-10T11:30:17.198 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:30:17.199 DEBUG:teuthology.orchestra.run.gibba002:> sudo python3 -c ' 2021-02-10T11:30:17.199 DEBUG:teuthology.orchestra.run.gibba002:> import glob 2021-02-10T11:30:17.199 DEBUG:teuthology.orchestra.run.gibba002:> import re 2021-02-10T11:30:17.199 DEBUG:teuthology.orchestra.run.gibba002:> import os 2021-02-10T11:30:17.199 DEBUG:teuthology.orchestra.run.gibba002:> import subprocess 2021-02-10T11:30:17.199 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:30:17.200 DEBUG:teuthology.orchestra.run.gibba002:> def find_socket(client_name): 2021-02-10T11:30:17.200 DEBUG:teuthology.orchestra.run.gibba002:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2021-02-10T11:30:17.200 DEBUG:teuthology.orchestra.run.gibba002:> files = glob.glob(asok_path) 2021-02-10T11:30:17.200 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:30:17.200 DEBUG:teuthology.orchestra.run.gibba002:> # Given a non-glob path, it better be there 2021-02-10T11:30:17.201 DEBUG:teuthology.orchestra.run.gibba002:> if "*" not in asok_path: 2021-02-10T11:30:17.201 DEBUG:teuthology.orchestra.run.gibba002:> assert(len(files) == 1) 2021-02-10T11:30:17.201 DEBUG:teuthology.orchestra.run.gibba002:> return files[0] 2021-02-10T11:30:17.201 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:30:17.201 DEBUG:teuthology.orchestra.run.gibba002:> for f in files: 2021-02-10T11:30:17.201 DEBUG:teuthology.orchestra.run.gibba002:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2021-02-10T11:30:17.202 DEBUG:teuthology.orchestra.run.gibba002:> if os.path.exists("/proc/{0}".format(pid)): 2021-02-10T11:30:17.202 DEBUG:teuthology.orchestra.run.gibba002:> return f 2021-02-10T11:30:17.202 DEBUG:teuthology.orchestra.run.gibba002:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2021-02-10T11:30:17.202 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:30:17.202 DEBUG:teuthology.orchestra.run.gibba002:> print(find_socket("client.0")) 2021-02-10T11:30:17.203 DEBUG:teuthology.orchestra.run.gibba002:> ' 2021-02-10T11:30:17.563 INFO:teuthology.orchestra.run.gibba002.stdout:/var/run/ceph/ceph-client.0.174201.asok 2021-02-10T11:30:17.566 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.174201.asok 2021-02-10T11:30:17.566 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:30:17.566 DEBUG:teuthology.orchestra.run.gibba002:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.174201.asok status 2021-02-10T11:30:17.958 INFO:teuthology.orchestra.run.gibba002.stdout:{ 2021-02-10T11:30:17.959 INFO:teuthology.orchestra.run.gibba002.stdout: "metadata": { 2021-02-10T11:30:17.959 INFO:teuthology.orchestra.run.gibba002.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:30:17.959 INFO:teuthology.orchestra.run.gibba002.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:30:17.959 INFO:teuthology.orchestra.run.gibba002.stdout: "entity_id": "0", 2021-02-10T11:30:17.959 INFO:teuthology.orchestra.run.gibba002.stdout: "hostname": "gibba002", 2021-02-10T11:30:17.959 INFO:teuthology.orchestra.run.gibba002.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2021-02-10T11:30:17.959 INFO:teuthology.orchestra.run.gibba002.stdout: "pid": "174201", 2021-02-10T11:30:17.959 INFO:teuthology.orchestra.run.gibba002.stdout: "root": "/" 2021-02-10T11:30:17.959 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:30:17.959 INFO:teuthology.orchestra.run.gibba002.stdout: "dentry_count": 0, 2021-02-10T11:30:17.959 INFO:teuthology.orchestra.run.gibba002.stdout: "dentry_pinned_count": 0, 2021-02-10T11:30:17.960 INFO:teuthology.orchestra.run.gibba002.stdout: "id": 9816, 2021-02-10T11:30:17.960 INFO:teuthology.orchestra.run.gibba002.stdout: "inst": { 2021-02-10T11:30:17.960 INFO:teuthology.orchestra.run.gibba002.stdout: "name": { 2021-02-10T11:30:17.960 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "client", 2021-02-10T11:30:17.960 INFO:teuthology.orchestra.run.gibba002.stdout: "num": 9816 2021-02-10T11:30:17.960 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:30:17.960 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": { 2021-02-10T11:30:17.960 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:30:17.960 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:0", 2021-02-10T11:30:17.960 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 6454636 2021-02-10T11:30:17.960 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:30:17.960 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:30:17.960 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": { 2021-02-10T11:30:17.960 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:30:17.961 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:0", 2021-02-10T11:30:17.961 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 6454636 2021-02-10T11:30:17.961 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:30:17.961 INFO:teuthology.orchestra.run.gibba002.stdout: "inst_str": "client.9816 172.21.2.102:0/6454636", 2021-02-10T11:30:17.961 INFO:teuthology.orchestra.run.gibba002.stdout: "addr_str": "172.21.2.102:0/6454636", 2021-02-10T11:30:17.961 INFO:teuthology.orchestra.run.gibba002.stdout: "inode_count": 1, 2021-02-10T11:30:17.961 INFO:teuthology.orchestra.run.gibba002.stdout: "mds_epoch": 73, 2021-02-10T11:30:17.961 INFO:teuthology.orchestra.run.gibba002.stdout: "osd_epoch": 93, 2021-02-10T11:30:17.961 INFO:teuthology.orchestra.run.gibba002.stdout: "osd_epoch_barrier": 0, 2021-02-10T11:30:17.961 INFO:teuthology.orchestra.run.gibba002.stdout: "blacklisted": false 2021-02-10T11:30:17.961 INFO:teuthology.orchestra.run.gibba002.stdout:} 2021-02-10T11:30:17.963 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:30:17.963 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:30:17.963 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:30:18.299 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:30:18.473 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:30:18.473 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:30:18.473 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:30:18.821 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:30:19.248 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:19.233 7f68c9c2c700 1 Processor -- start 2021-02-10T11:30:19.248 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:19.233 7f68c9c2c700 1 -- start start 2021-02-10T11:30:19.248 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:19.233 7f68c9c2c700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f68c40d1870 0x7f68c41b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:19.248 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:19.233 7f68c9c2c700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f68c41b0a10 0x7f68c40cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:19.248 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:19.233 7f68c9c2c700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68c40f0b10 con 0x7f68c40d1c70 2021-02-10T11:30:19.248 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:19.233 7f68c9c2c700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68c40fe230 con 0x7f68c41b0a10 2021-02-10T11:30:19.248 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:19.233 7f68c9c2c700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68c40e5ed0 con 0x7f68c40d1870 2021-02-10T11:30:19.248 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:19.233 7f68c3fff700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f68c40d1c70 0x7f68c40cf950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:53128/0 (socket says 172.21.2.102:53128) 2021-02-10T11:30:19.248 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:19.233 7f68c3fff700 1 -- 172.21.2.102:0/3778418087 learned_addr learned my addr 172.21.2.102:0/3778418087 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:30:19.248 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:19.233 7f68baffd700 1 --2- 172.21.2.102:0/3778418087 >> v2:172.21.2.102:3300/0 conn(0x7f68c41b0a10 0x7f68c40cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:19.249 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:19.233 7f68c2ffd700 1 -- 172.21.2.102:0/3778418087 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1300655538 0 0) 0x7f68bc001180 con 0x7f68c40d1c70 2021-02-10T11:30:19.249 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:19.233 7f68c2ffd700 1 -- 172.21.2.102:0/3778418087 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68b0002fe0 con 0x7f68c40d1c70 2021-02-10T11:30:19.249 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:19.233 7f68baffd700 1 -- 172.21.2.102:0/3778418087 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f68c40d1870 msgr2=0x7f68c41b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:19.249 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:19.233 7f68baffd700 1 --2- 172.21.2.102:0/3778418087 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f68c40d1870 0x7f68c41b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:19.249 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:19.233 7f68baffd700 1 -- 172.21.2.102:0/3778418087 >> v1:172.21.2.102:6789/0 conn(0x7f68c40d1c70 legacy=0x7f68c40cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:19.249 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:19.233 7f68baffd700 1 -- 172.21.2.102:0/3778418087 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68c40f0650 con 0x7f68c41b0a10 2021-02-10T11:30:19.249 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:19.233 7f68baffd700 1 --2- 172.21.2.102:0/3778418087 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f68c41b0a10 0x7f68c40cf330 secure :-1 s=READY pgs=1962 cs=0 l=1 rev1=1 rx=0x7f68ac006650 tx=0x7f68ac009a80).ready entity=mon.0 client_cookie=723e78f27df4cb07 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:19.249 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:19.234 7f68c2ffd700 1 -- 172.21.2.102:0/3778418087 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f68ac0009b0 con 0x7f68c41b0a10 2021-02-10T11:30:19.249 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:19.234 7f68c2ffd700 1 -- 172.21.2.102:0/3778418087 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68ac00c0c0 con 0x7f68c41b0a10 2021-02-10T11:30:19.249 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:19.234 7f68c2ffd700 1 -- 172.21.2.102:0/3778418087 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f68ac00c5e0 con 0x7f68c41b0a10 2021-02-10T11:30:19.249 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:19.234 7f68c9c2c700 1 -- 172.21.2.102:0/3778418087 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f68c41b0a10 msgr2=0x7f68c40cf330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:19.249 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:19.234 7f68c9c2c700 1 --2- 172.21.2.102:0/3778418087 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f68c41b0a10 0x7f68c40cf330 secure :-1 s=READY pgs=1962 cs=0 l=1 rev1=1 rx=0x7f68ac006650 tx=0x7f68ac009a80).stop 2021-02-10T11:30:19.249 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:19.234 7f68c9c2c700 1 -- 172.21.2.102:0/3778418087 shutdown_connections 2021-02-10T11:30:19.249 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:19.234 7f68c9c2c700 1 --2- 172.21.2.102:0/3778418087 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f68c41b0a10 0x7f68c40cf330 unknown :-1 s=CLOSED pgs=1962 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:19.250 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:19.234 7f68c9c2c700 1 --2- 172.21.2.102:0/3778418087 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f68c40d1870 0x7f68c41b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:19.250 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:19.234 7f68c9c2c700 1 -- 172.21.2.102:0/3778418087 >> 172.21.2.102:0/3778418087 conn(0x7f68c40ce450 msgr2=0x7f68c40d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:19.250 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:19.234 7f68c9c2c700 1 -- 172.21.2.102:0/3778418087 shutdown_connections 2021-02-10T11:30:19.250 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:19.234 7f68c9c2c700 1 -- 172.21.2.102:0/3778418087 wait complete. 2021-02-10T11:30:19.251 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:19.249 7f68c9c2c700 1 Processor -- start 2021-02-10T11:30:19.251 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:19.249 7f68c9c2c700 1 -- start start 2021-02-10T11:30:19.251 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:19.249 7f68c9c2c700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f68c40ada90 0x7f68c40ade90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:19.252 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:19.249 7f68c9c2c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f68c4099d20 0x7f68c409a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:19.252 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:19.249 7f68c9c2c700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f68c409a740 0x7f68c4097bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:19.252 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:19.249 7f68c9c2c700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68c40997a0 con 0x7f68c4099d20 2021-02-10T11:30:19.252 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:19.249 7f68c9c2c700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68c418f0c0 con 0x7f68c409a740 2021-02-10T11:30:19.252 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:19.249 7f68c9c2c700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68c409ab40 con 0x7f68c40ada90 2021-02-10T11:30:19.252 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:19.249 7f68baffd700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f68c4099d20 0x7f68c409a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:19.252 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:19.249 7f68baffd700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f68c4099d20 0x7f68c409a120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:60912/0 (socket says 172.21.2.102:60912) 2021-02-10T11:30:19.252 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:19.249 7f68baffd700 1 -- 172.21.2.102:0/1607091124 learned_addr learned my addr 172.21.2.102:0/1607091124 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:30:19.252 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:19.249 7f68baffd700 1 -- 172.21.2.102:0/1607091124 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f68c40ada90 msgr2=0x7f68c40ade90 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:30:19.252 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:19.249 7f68baffd700 1 --2- 172.21.2.102:0/1607091124 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f68c40ada90 0x7f68c40ade90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:19.252 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:19.249 7f68baffd700 1 -- 172.21.2.102:0/1607091124 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f68c409a740 msgr2=0x7f68c4097bc0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:30:19.252 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:19.249 7f68baffd700 1 --2- 172.21.2.102:0/1607091124 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f68c409a740 0x7f68c4097bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:19.252 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:19.249 7f68baffd700 1 -- 172.21.2.102:0/1607091124 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68c409ad60 con 0x7f68c4099d20 2021-02-10T11:30:19.253 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:19.250 7f68baffd700 1 --2- 172.21.2.102:0/1607091124 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f68c4099d20 0x7f68c409a120 secure :-1 s=READY pgs=1963 cs=0 l=1 rev1=1 rx=0x7f68ac009630 tx=0x7f68ac00b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:19.253 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:19.250 7f68c0ff9700 1 -- 172.21.2.102:0/1607091124 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f68ac0093a0 con 0x7f68c4099d20 2021-02-10T11:30:19.253 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:19.250 7f68c0ff9700 1 -- 172.21.2.102:0/1607091124 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68ac00bd70 con 0x7f68c4099d20 2021-02-10T11:30:19.253 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:19.250 7f68c9c2c700 1 -- 172.21.2.102:0/1607091124 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68c409ad60 con 0x7f68c4099d20 2021-02-10T11:30:19.253 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:19.250 7f68c0ff9700 1 -- 172.21.2.102:0/1607091124 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f68ac005560 con 0x7f68c4099d20 2021-02-10T11:30:19.253 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:19.250 7f68c9c2c700 1 -- 172.21.2.102:0/1607091124 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68c409ad60 con 0x7f68c4099d20 2021-02-10T11:30:19.253 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:19.250 7f68c0ff9700 1 -- 172.21.2.102:0/1607091124 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f68ac005740 con 0x7f68c4099d20 2021-02-10T11:30:19.253 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:19.250 7f68c0ff9700 1 --2- 172.21.2.102:0/1607091124 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f68a8018730 0x7f68a801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:19.253 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:19.250 7f68c0ff9700 1 -- 172.21.2.102:0/1607091124 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f68ac032650 con 0x7f68c4099d20 2021-02-10T11:30:19.253 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:19.250 7f68c37fe700 1 --2- 172.21.2.102:0/1607091124 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f68a8018730 0x7f68a801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:19.253 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:19.251 7f68c37fe700 1 --2- 172.21.2.102:0/1607091124 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f68a8018730 0x7f68a801ac10 crc :-1 s=READY pgs=880 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:19.282 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:19.283 7f68c9c2c700 1 -- 172.21.2.102:0/1607091124 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68c40ebc00 con 0x7f68c4099d20 2021-02-10T11:30:19.285 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:19.285 7f68c0ff9700 1 -- 172.21.2.102:0/1607091124 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f68ac005530 con 0x7f68c4099d20 2021-02-10T11:30:19.410 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:19.410 7f68c9c2c700 1 -- 172.21.2.102:0/1607091124 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f68c40ebc00 con 0x7f68c4099d20 2021-02-10T11:30:19.410 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:19.411 7f68c0ff9700 1 -- 172.21.2.102:0/1607091124 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) v1 ==== 74+0+13996 (secure 0 0 0) 0x7f68ac005740 con 0x7f68c4099d20 2021-02-10T11:30:19.411 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:30:19.411 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":93,"fsid":"68db30ea-dbce-4aef-a6df-f01c6fea0513","created":"2021-02-10 11:03:06.331557","modified":"2021-02-10 11:30:05.228248","last_up_change":"2021-02-10 11:03:32.690562","last_in_change":"2021-02-10 11:03:32.690562","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:03:37.322519","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:30:02.187273","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":19,"pool_name":"cephfs_data","create_time":"2021-02-10 11:30:03.219690","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"93","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"be691b4d-a039-408a-af80-9d53f02b6f1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6818","nonce":36088},{"type":"v1","addr":"172.21.2.102:6820","nonce":36088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6822","nonce":36088},{"type":"v1","addr":"172.21.2.102:6824","nonce":36088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6830","nonce":36088},{"type":"v1","addr":"172.21.2.102:6832","nonce":36088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6826","nonce":36088},{"type":"v1","addr":"172.21.2.102:6828","nonce":36088}]},"public_addr":"172.21.2.102:6820/36088","cluster_addr":"172.21.2.102:6824/36088","heartbeat_back_addr":"172.21.2.102:6832/36088","heartbeat_front_addr":"172.21.2.102:6828/36088","state":["exists","up"]},{"osd":1,"uuid":"64e3ccc8-9c69-4d7c-865e-af8e83822635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6819","nonce":36087},{"type":"v1","addr":"172.21.2.102:6821","nonce":36087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6823","nonce":36087},{"type":"v1","addr":"172.21.2.102:6825","nonce":36087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6831","nonce":36087},{"type":"v1","addr":"172.21.2.102:6833","nonce":36087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6827","nonce":36087},{"type":"v1","addr":"172.21.2.102:6829","nonce":36087}]},"public_addr":"172.21.2.102:6821/36087","cluster_addr":"172.21.2.102:6825/36087","heartbeat_back_addr":"172.21.2.102:6833/36087","heartbeat_front_addr":"172.21.2.102:6829/36087","state":["exists","up"]},{"osd":2,"uuid":"153700af-747f-481f-bdf4-d6072d3a842f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6810","nonce":36089},{"type":"v1","addr":"172.21.2.102:6811","nonce":36089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6812","nonce":36089},{"type":"v1","addr":"172.21.2.102:6813","nonce":36089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6816","nonce":36089},{"type":"v1","addr":"172.21.2.102:6817","nonce":36089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6814","nonce":36089},{"type":"v1","addr":"172.21.2.102:6815","nonce":36089}]},"public_addr":"172.21.2.102:6811/36089","cluster_addr":"172.21.2.102:6813/36089","heartbeat_back_addr":"172.21.2.102:6817/36089","heartbeat_front_addr":"172.21.2.102:6815/36089","state":["exists","up"]},{"osd":3,"uuid":"cbc678f4-83a9-45e6-8830-2f7c232b1ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6802","nonce":36090},{"type":"v1","addr":"172.21.2.102:6803","nonce":36090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6804","nonce":36090},{"type":"v1","addr":"172.21.2.102:6805","nonce":36090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6808","nonce":36090},{"type":"v1","addr":"172.21.2.102:6809","nonce":36090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6806","nonce":36090},{"type":"v1","addr":"172.21.2.102:6807","nonce":36090}]},"public_addr":"172.21.2.102:6803/36090","cluster_addr":"172.21.2.102:6805/36090","heartbeat_back_addr":"172.21.2.102:6809/36090","heartbeat_front_addr":"172.21.2.102:6807/36090","state":["exists","up"]},{"osd":4,"uuid":"c01e4008-b258-4f10-afd2-9e7f37214351","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6808","nonce":36368},{"type":"v1","addr":"172.21.2.113:6811","nonce":36368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6814","nonce":36368},{"type":"v1","addr":"172.21.2.113:6818","nonce":36368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6826","nonce":36368},{"type":"v1","addr":"172.21.2.113:6828","nonce":36368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6820","nonce":36368},{"type":"v1","addr":"172.21.2.113:6824","nonce":36368}]},"public_addr":"172.21.2.113:6811/36368","cluster_addr":"172.21.2.113:6818/36368","heartbeat_back_addr":"172.21.2.113:6828/36368","heartbeat_front_addr":"172.21.2.113:6824/36368","state":["exists","up"]},{"osd":5,"uuid":"551f6007-daf8-4a98-8ee0-c2b16bcccb5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6802","nonce":36366},{"type":"v1","addr":"172.21.2.113:6805","nonce":36366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6807","nonce":36366},{"type":"v1","addr":"172.21.2.113:6810","nonce":36366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6819","nonce":36366},{"type":"v1","addr":"172.21.2.113:6822","nonce":36366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6812","nonce":36366},{"type":"v1","addr":"172.21.2.113:6815","nonce":36366}]},"public_addr":"172.21.2.113:6805/36366","cluster_addr":"172.21.2.113:6810/36366","heartbeat_back_addr":"172.21.2.113:6822/36366","heartbeat_front_addr":"172.21.2.113:6815/36366","state":["exists","up"]},{"osd":6,"uuid":"bf67c040-0e50-48be-b102-c3943bc9ebef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6800","nonce":36367},{"type":"v1","addr":"172.21.2.113:6801","nonce":36367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6803","nonce":36367},{"type":"v1","addr":"172.21.2.113:6804","nonce":36367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6813","nonce":36367},{"type":"v1","addr":"172.21.2.113:6816","nonce":36367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6806","nonce":36367},{"type":"v1","addr":"172.21.2.113:6809","nonce":36367}]},"public_addr":"172.21.2.113:6801/36367","cluster_addr":"172.21.2.113:6804/36367","heartbeat_back_addr":"172.21.2.113:6816/36367","heartbeat_front_addr":"172.21.2.113:6809/36367","state":["exists","up"]},{"osd":7,"uuid":"f5461f80-018d-428f-be89-2eb2bcbe7bb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6817","nonce":36365},{"type":"v1","addr":"172.21.2.113:6821","nonce":36365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6823","nonce":36365},{"type":"v1","addr":"172.21.2.113:6825","nonce":36365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6830","nonce":36365},{"type":"v1","addr":"172.21.2.113:6831","nonce":36365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6827","nonce":36365},{"type":"v1","addr":"172.21.2.113:6829","nonce":36365}]},"public_addr":"172.21.2.113:6821/36365","cluster_addr":"172.21.2.113:6825/36365","heartbeat_back_addr":"172.21.2.113:6831/36365","heartbeat_front_addr":"172.21.2.113:6829/36365","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:30:19.412 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:19.412 7f68c9c2c700 1 -- 172.21.2.102:0/1607091124 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f68a8018730 msgr2=0x7f68a801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:19.412 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:19.412 7f68c9c2c700 1 --2- 172.21.2.102:0/1607091124 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f68a8018730 0x7f68a801ac10 crc :-1 s=READY pgs=880 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:19.412 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:19.412 7f68c9c2c700 1 -- 172.21.2.102:0/1607091124 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f68c4099d20 msgr2=0x7f68c409a120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:19.412 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:19.412 7f68c9c2c700 1 --2- 172.21.2.102:0/1607091124 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f68c4099d20 0x7f68c409a120 secure :-1 s=READY pgs=1963 cs=0 l=1 rev1=1 rx=0x7f68ac009630 tx=0x7f68ac00b980).stop 2021-02-10T11:30:19.412 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:19.413 7f68c9c2c700 1 -- 172.21.2.102:0/1607091124 shutdown_connections 2021-02-10T11:30:19.413 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:19.413 7f68c9c2c700 1 --2- 172.21.2.102:0/1607091124 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f68a8018730 0x7f68a801ac10 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:19.413 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:19.413 7f68c9c2c700 1 --2- 172.21.2.102:0/1607091124 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f68c409a740 0x7f68c4097bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:19.413 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:19.413 7f68c9c2c700 1 --2- 172.21.2.102:0/1607091124 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f68c4099d20 0x7f68c409a120 unknown :-1 s=CLOSED pgs=1963 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:19.413 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:19.413 7f68c9c2c700 1 --2- 172.21.2.102:0/1607091124 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f68c40ada90 0x7f68c40ade90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:19.413 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:19.413 7f68c9c2c700 1 -- 172.21.2.102:0/1607091124 >> 172.21.2.102:0/1607091124 conn(0x7f68c40f0550 msgr2=0x7f68c4191730 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:19.413 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:19.413 7f68c9c2c700 1 -- 172.21.2.102:0/1607091124 shutdown_connections 2021-02-10T11:30:19.413 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:19.413 7f68c9c2c700 1 -- 172.21.2.102:0/1607091124 wait complete. 2021-02-10T11:30:19.421 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:03:37.322519', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 18, 'pool_name': 'cephfs_metadata', 'create_time': '2021-02-10 11:30:02.187273', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '92', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 19, 'pool_name': 'cephfs_data', 'create_time': '2021-02-10 11:30:03.219690', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '93', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-02-10T11:30:19.421 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:30:19.852 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:19.836 7fe862493700 1 Processor -- start 2021-02-10T11:30:19.852 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:19.836 7fe862493700 1 -- start start 2021-02-10T11:30:19.852 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:19.837 7fe862493700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe85c0d1c70 0x7fe85c0cef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:19.852 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:19.837 7fe862493700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe85c0d0170 con 0x7fe85c0d1870 2021-02-10T11:30:19.852 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:19.837 7fe862493700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe85c0fe1c0 con 0x7fe85c0cf580 2021-02-10T11:30:19.852 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:19.837 7fe862493700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe85c1b0cb0 con 0x7fe85c0d1c70 2021-02-10T11:30:19.853 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:19.837 7fe85bfff700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fe85c0d1870 0x7fe85c1b04c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:53140/0 (socket says 172.21.2.102:53140) 2021-02-10T11:30:19.853 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:19.837 7fe85bfff700 1 -- 172.21.2.102:0/3807907146 learned_addr learned my addr 172.21.2.102:0/3807907146 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:30:19.853 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:19.837 7fe85b7fe700 1 --2- 172.21.2.102:0/3807907146 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe85c0d1c70 0x7fe85c0cef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:19.853 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:19.837 7fe85affd700 1 -- 172.21.2.102:0/3807907146 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 389938448 0 0) 0x7fe84c0010b0 con 0x7fe85c0d1870 2021-02-10T11:30:19.853 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:19.837 7fe85affd700 1 -- 172.21.2.102:0/3807907146 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe844002fe0 con 0x7fe85c0d1870 2021-02-10T11:30:19.853 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:19.837 7fe85b7fe700 1 -- 172.21.2.102:0/3807907146 >> v1:172.21.2.113:6789/0 conn(0x7fe85c0cf580 legacy=0x7fe85c0cf980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:19.853 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:19.837 7fe85b7fe700 1 -- 172.21.2.102:0/3807907146 >> v1:172.21.2.102:6789/0 conn(0x7fe85c0d1870 legacy=0x7fe85c1b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:19.853 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:19.837 7fe85affd700 1 -- 172.21.2.102:0/3807907146 <== mon.1 v1:172.21.2.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2172148468 0 0) 0x7fe854000f90 con 0x7fe85c0cf580 2021-02-10T11:30:19.853 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:19.837 7fe85b7fe700 1 -- 172.21.2.102:0/3807907146 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe85c109a60 con 0x7fe85c0d1c70 2021-02-10T11:30:19.853 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:19.838 7fe85b7fe700 1 --2- 172.21.2.102:0/3807907146 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe85c0d1c70 0x7fe85c0cef60 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7fe850006650 tx=0x7fe850009a80).ready entity=mon.2 client_cookie=67ea1225d0f4222c server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:19.853 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:19.838 7fe85affd700 1 -- 172.21.2.102:0/3807907146 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe8500009b0 con 0x7fe85c0d1c70 2021-02-10T11:30:19.853 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:19.838 7fe85affd700 1 -- 172.21.2.102:0/3807907146 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe85000c0c0 con 0x7fe85c0d1c70 2021-02-10T11:30:19.854 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:19.838 7fe85affd700 1 -- 172.21.2.102:0/3807907146 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe85000c520 con 0x7fe85c0d1c70 2021-02-10T11:30:19.854 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:19.838 7fe862493700 1 -- 172.21.2.102:0/3807907146 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe85c0d1c70 msgr2=0x7fe85c0cef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:19.854 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:19.838 7fe862493700 1 --2- 172.21.2.102:0/3807907146 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe85c0d1c70 0x7fe85c0cef60 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7fe850006650 tx=0x7fe850009a80).stop 2021-02-10T11:30:19.854 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:19.838 7fe862493700 1 -- 172.21.2.102:0/3807907146 shutdown_connections 2021-02-10T11:30:19.854 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:19.838 7fe862493700 1 --2- 172.21.2.102:0/3807907146 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe85c0d1c70 0x7fe85c0cef60 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:19.854 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:19.838 7fe862493700 1 -- 172.21.2.102:0/3807907146 >> 172.21.2.102:0/3807907146 conn(0x7fe85c0ce420 msgr2=0x7fe85c0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:19.854 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:19.839 7fe862493700 1 -- 172.21.2.102:0/3807907146 shutdown_connections 2021-02-10T11:30:19.854 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:19.839 7fe862493700 1 -- 172.21.2.102:0/3807907146 wait complete. 2021-02-10T11:30:19.855 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:19.853 7fe862493700 1 Processor -- start 2021-02-10T11:30:19.856 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:19.853 7fe862493700 1 -- start start 2021-02-10T11:30:19.856 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:19.853 7fe862493700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe85c0c3820 0x7fe85c0a7950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:19.856 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:19.853 7fe862493700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe85c1a44c0 0x7fe85c1a48c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:19.856 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:19.853 7fe862493700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe85c0863d0 0x7fe85c0867d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:19.856 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:19.853 7fe862493700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe85c085ed0 con 0x7fe85c1a44c0 2021-02-10T11:30:19.856 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:19.853 7fe862493700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe85c0c0740 con 0x7fe85c0c3820 2021-02-10T11:30:19.856 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:19.853 7fe862493700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe85c086df0 con 0x7fe85c0863d0 2021-02-10T11:30:19.856 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:19.853 7fe85b7fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe85c1a44c0 0x7fe85c1a48c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:19.856 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:19.853 7fe85b7fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe85c1a44c0 0x7fe85c1a48c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:60926/0 (socket says 172.21.2.102:60926) 2021-02-10T11:30:19.856 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:19.853 7fe85b7fe700 1 -- 172.21.2.102:0/1830545729 learned_addr learned my addr 172.21.2.102:0/1830545729 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:30:19.856 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:19.854 7fe85b7fe700 1 -- 172.21.2.102:0/1830545729 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe85c0863d0 msgr2=0x7fe85c0867d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:19.856 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:19.854 7fe85b7fe700 1 --2- 172.21.2.102:0/1830545729 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe85c0863d0 0x7fe85c0867d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:19.856 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:19.854 7fe85b7fe700 1 -- 172.21.2.102:0/1830545729 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe85c0c3820 msgr2=0x7fe85c0a7950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:19.857 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:19.854 7fe85b7fe700 1 --2- 172.21.2.102:0/1830545729 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe85c0c3820 0x7fe85c0a7950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:19.857 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:19.854 7fe85b7fe700 1 -- 172.21.2.102:0/1830545729 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe85c087010 con 0x7fe85c1a44c0 2021-02-10T11:30:19.857 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:19.854 7fe85b7fe700 1 --2- 172.21.2.102:0/1830545729 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe85c1a44c0 0x7fe85c1a48c0 secure :-1 s=READY pgs=1965 cs=0 l=1 rev1=1 rx=0x7fe850009630 tx=0x7fe85000b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:19.857 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:19.854 7fe858ff9700 1 -- 172.21.2.102:0/1830545729 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe8500093a0 con 0x7fe85c1a44c0 2021-02-10T11:30:19.857 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:19.854 7fe858ff9700 1 -- 172.21.2.102:0/1830545729 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe85000bd70 con 0x7fe85c1a44c0 2021-02-10T11:30:19.857 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:19.854 7fe858ff9700 1 -- 172.21.2.102:0/1830545729 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe850005560 con 0x7fe85c1a44c0 2021-02-10T11:30:19.857 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:19.854 7fe862493700 1 -- 172.21.2.102:0/1830545729 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe85c087010 con 0x7fe85c1a44c0 2021-02-10T11:30:19.857 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:19.854 7fe862493700 1 -- 172.21.2.102:0/1830545729 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe85c083000 con 0x7fe85c1a44c0 2021-02-10T11:30:19.857 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:19.855 7fe858ff9700 1 -- 172.21.2.102:0/1830545729 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fe8500052e0 con 0x7fe85c1a44c0 2021-02-10T11:30:19.857 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:19.855 7fe858ff9700 1 --2- 172.21.2.102:0/1830545729 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe83c0186c0 0x7fe83c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:19.857 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:19.855 7fe858ff9700 1 -- 172.21.2.102:0/1830545729 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fe850032400 con 0x7fe85c1a44c0 2021-02-10T11:30:19.857 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:19.855 7fe85bfff700 1 --2- 172.21.2.102:0/1830545729 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe83c0186c0 0x7fe83c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:19.857 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:19.855 7fe85bfff700 1 --2- 172.21.2.102:0/1830545729 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe83c0186c0 0x7fe83c01abc0 crc :-1 s=READY pgs=881 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:19.886 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:19.886 7fe862493700 1 -- 172.21.2.102:0/1830545729 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe85c0fe1c0 con 0x7fe85c1a44c0 2021-02-10T11:30:19.888 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:19.889 7fe858ff9700 1 -- 172.21.2.102:0/1830545729 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fe85002dee0 con 0x7fe85c1a44c0 2021-02-10T11:30:20.015 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:20.016 7fe862493700 1 -- 172.21.2.102:0/1830545729 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7fe85c0fe1c0 con 0x7fe85c1a44c0 2021-02-10T11:30:20.016 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:20.016 7fe858ff9700 1 -- 172.21.2.102:0/1830545729 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v93) v1 ==== 92+0+10 (secure 0 0 0) 0x7fe85002daf0 con 0x7fe85c1a44c0 2021-02-10T11:30:20.016 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:30:20.018 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:20.018 7fe862493700 1 -- 172.21.2.102:0/1830545729 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe83c0186c0 msgr2=0x7fe83c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:20.018 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:20.018 7fe862493700 1 --2- 172.21.2.102:0/1830545729 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe83c0186c0 0x7fe83c01abc0 crc :-1 s=READY pgs=881 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:20.018 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:20.018 7fe862493700 1 -- 172.21.2.102:0/1830545729 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe85c1a44c0 msgr2=0x7fe85c1a48c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:20.018 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:20.018 7fe862493700 1 --2- 172.21.2.102:0/1830545729 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe85c1a44c0 0x7fe85c1a48c0 secure :-1 s=READY pgs=1965 cs=0 l=1 rev1=1 rx=0x7fe850009630 tx=0x7fe85000b980).stop 2021-02-10T11:30:20.018 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:20.018 7fe862493700 1 -- 172.21.2.102:0/1830545729 shutdown_connections 2021-02-10T11:30:20.019 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:20.018 7fe862493700 1 --2- 172.21.2.102:0/1830545729 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe85c0863d0 0x7fe85c0867d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:20.019 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:20.018 7fe862493700 1 --2- 172.21.2.102:0/1830545729 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe85c1a44c0 0x7fe85c1a48c0 unknown :-1 s=CLOSED pgs=1965 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:20.019 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:20.018 7fe862493700 1 --2- 172.21.2.102:0/1830545729 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe83c0186c0 0x7fe83c01abc0 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:20.019 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:20.018 7fe862493700 1 --2- 172.21.2.102:0/1830545729 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe85c0c3820 0x7fe85c0a7950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:20.019 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:20.018 7fe862493700 1 -- 172.21.2.102:0/1830545729 >> 172.21.2.102:0/1830545729 conn(0x7fe85c0872c0 msgr2=0x7fe85c095700 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:20.019 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:20.019 7fe862493700 1 -- 172.21.2.102:0/1830545729 shutdown_connections 2021-02-10T11:30:20.019 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:20.019 7fe862493700 1 -- 172.21.2.102:0/1830545729 wait complete. 2021-02-10T11:30:20.027 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-02-10T11:30:20.462 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:20.444 7fb538ab3700 1 Processor -- start 2021-02-10T11:30:20.462 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:20.444 7fb538ab3700 1 -- start start 2021-02-10T11:30:20.462 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:20.445 7fb538ab3700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb5340d1c60 0x7fb5340cef50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:20.462 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:20.445 7fb538ab3700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fb5340cf570 0x7fb5340cf970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:20.462 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:20.445 7fb538ab3700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5340fe1b0 con 0x7fb5340d1860 2021-02-10T11:30:20.462 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:20.445 7fb538ab3700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5341b0ca0 con 0x7fb5340cf570 2021-02-10T11:30:20.463 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:20.445 7fb538ab3700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb534109a50 con 0x7fb5340d1c60 2021-02-10T11:30:20.463 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:20.445 7fb53285e700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fb5340d1860 0x7fb5341b04b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:53154/0 (socket says 172.21.2.102:53154) 2021-02-10T11:30:20.463 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:20.445 7fb53285e700 1 -- 172.21.2.102:0/1361524824 learned_addr learned my addr 172.21.2.102:0/1361524824 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:30:20.463 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:20.445 7fb53185c700 1 -- 172.21.2.102:0/1361524824 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4023561769 0 0) 0x7fb524000ce0 con 0x7fb5340d1860 2021-02-10T11:30:20.463 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:20.445 7fb53185c700 1 -- 172.21.2.102:0/1361524824 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb520002fe0 con 0x7fb5340d1860 2021-02-10T11:30:20.463 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:20.446 7fb53305f700 1 --2- 172.21.2.102:0/1361524824 >> v2:172.21.2.113:3300/0 conn(0x7fb5340cf570 0x7fb5340cf970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:20.463 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:20.446 7fb53205d700 1 --2- 172.21.2.102:0/1361524824 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb5340d1c60 0x7fb5340cef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:20.463 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:20.446 7fb53185c700 1 -- 172.21.2.102:0/1361524824 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1590133155 0 0) 0x7fb5240010e0 con 0x7fb5340d1860 2021-02-10T11:30:20.463 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:20.446 7fb53185c700 1 -- 172.21.2.102:0/1361524824 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb5340d1c60 msgr2=0x7fb5340cef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:20.463 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:20.446 7fb53185c700 1 --2- 172.21.2.102:0/1361524824 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb5340d1c60 0x7fb5340cef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:20.463 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:20.446 7fb53185c700 1 -- 172.21.2.102:0/1361524824 >> v2:172.21.2.113:3300/0 conn(0x7fb5340cf570 msgr2=0x7fb5340cf970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:20.463 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:20.446 7fb53185c700 1 --2- 172.21.2.102:0/1361524824 >> v2:172.21.2.113:3300/0 conn(0x7fb5340cf570 0x7fb5340cf970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:20.463 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:20.446 7fb53185c700 1 -- 172.21.2.102:0/1361524824 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb534088070 con 0x7fb5340d1860 2021-02-10T11:30:20.463 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:20.446 7fb53185c700 1 -- 172.21.2.102:0/1361524824 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7fb524001600 con 0x7fb5340d1860 2021-02-10T11:30:20.464 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:20.446 7fb53185c700 1 -- 172.21.2.102:0/1361524824 >> v1:172.21.2.102:6789/0 conn(0x7fb5340d1860 legacy=0x7fb5341b04b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:20.464 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:20.446 7fb53185c700 1 --2- 172.21.2.102:0/1361524824 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb520003b10 0x7fb520005fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:20.464 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:20.446 7fb53185c700 1 --2- 172.21.2.102:0/1361524824 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb520006880 0x7fb520008da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:20.464 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:20.446 7fb53185c700 1 --2- 172.21.2.102:0/1361524824 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb520009640 0x7fb52000bb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:20.464 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:20.446 7fb53185c700 1 -- 172.21.2.102:0/1361524824 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb52000c410 con 0x7fb520009640 2021-02-10T11:30:20.464 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:20.446 7fb53185c700 1 -- 172.21.2.102:0/1361524824 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb52000c630 con 0x7fb520006880 2021-02-10T11:30:20.464 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:20.446 7fb53185c700 1 -- 172.21.2.102:0/1361524824 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5200507f0 con 0x7fb520003b10 2021-02-10T11:30:20.464 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:20.446 7fb53205d700 1 --2- 172.21.2.102:0/1361524824 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb5340d1c60 0x7fb5340cef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:30:20.464 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:20.446 7fb53305f700 1 --2- 172.21.2.102:0/1361524824 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb520009640 0x7fb52000bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:20.464 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:20.446 7fb53285e700 1 --2- 172.21.2.102:0/1361524824 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb520003b10 0x7fb520005fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:20.464 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:20.446 7fb53205d700 1 --2- 172.21.2.102:0/1361524824 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb520006880 0x7fb520008da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:20.464 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:20.446 7fb53305f700 1 -- 172.21.2.102:0/1361524824 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb520003b10 msgr2=0x7fb520005fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:20.464 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:20.446 7fb53305f700 1 --2- 172.21.2.102:0/1361524824 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb520003b10 0x7fb520005fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:20.465 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:20.446 7fb53305f700 1 -- 172.21.2.102:0/1361524824 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb520006880 msgr2=0x7fb520008da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:20.465 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:20.446 7fb53305f700 1 --2- 172.21.2.102:0/1361524824 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb520006880 0x7fb520008da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:20.465 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:20.446 7fb53305f700 1 -- 172.21.2.102:0/1361524824 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb520050a50 con 0x7fb520009640 2021-02-10T11:30:20.465 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:20.446 7fb53285e700 1 --2- 172.21.2.102:0/1361524824 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb520003b10 0x7fb520005fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:30:20.465 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:20.447 7fb53305f700 1 --2- 172.21.2.102:0/1361524824 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb520009640 0x7fb52000bb60 secure :-1 s=READY pgs=1967 cs=0 l=1 rev1=1 rx=0x7fb52c0090f0 tx=0x7fb52c00b050).ready entity=mon.0 client_cookie=5d03a5bf40e04ddd server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:20.465 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:20.447 7fb53185c700 1 -- 172.21.2.102:0/1361524824 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb52c000f90 con 0x7fb520009640 2021-02-10T11:30:20.465 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:20.447 7fb53185c700 1 -- 172.21.2.102:0/1361524824 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb52c006240 con 0x7fb520009640 2021-02-10T11:30:20.465 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:20.447 7fb53185c700 1 -- 172.21.2.102:0/1361524824 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb52c0056b0 con 0x7fb520009640 2021-02-10T11:30:20.465 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:20.447 7fb538ab3700 1 -- 172.21.2.102:0/1361524824 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb520009640 msgr2=0x7fb52000bb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:20.465 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:20.447 7fb538ab3700 1 --2- 172.21.2.102:0/1361524824 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb520009640 0x7fb52000bb60 secure :-1 s=READY pgs=1967 cs=0 l=1 rev1=1 rx=0x7fb52c0090f0 tx=0x7fb52c00b050).stop 2021-02-10T11:30:20.465 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:20.447 7fb53305f700 1 -- 172.21.2.102:0/1361524824 reap_dead start 2021-02-10T11:30:20.465 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:20.448 7fb538ab3700 1 -- 172.21.2.102:0/1361524824 shutdown_connections 2021-02-10T11:30:20.466 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:20.448 7fb538ab3700 1 -- 172.21.2.102:0/1361524824 >> 172.21.2.102:0/1361524824 conn(0x7fb5340ce410 msgr2=0x7fb5340d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:20.466 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:20.448 7fb538ab3700 1 -- 172.21.2.102:0/1361524824 shutdown_connections 2021-02-10T11:30:20.466 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:20.448 7fb538ab3700 1 -- 172.21.2.102:0/1361524824 wait complete. 2021-02-10T11:30:20.467 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:20.463 7fb538ab3700 1 Processor -- start 2021-02-10T11:30:20.467 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:20.463 7fb538ab3700 1 -- start start 2021-02-10T11:30:20.467 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:20.463 7fb538ab3700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb5340adf50 0x7fb5340ae350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:20.467 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:20.463 7fb538ab3700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb5340ae970 0x7fb5340aed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:20.467 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:20.463 7fb538ab3700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb5340982c0 0x7fb5340986c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:20.468 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:20.463 7fb538ab3700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb53409f450 con 0x7fb5340ae970 2021-02-10T11:30:20.468 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:20.463 7fb538ab3700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb534098ce0 con 0x7fb5340982c0 2021-02-10T11:30:20.468 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:20.463 7fb538ab3700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb534098ed0 con 0x7fb5340adf50 2021-02-10T11:30:20.468 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:20.463 7fb53205d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb5340ae970 0x7fb5340aed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:20.468 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:20.463 7fb53205d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb5340ae970 0x7fb5340aed70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:60944/0 (socket says 172.21.2.102:60944) 2021-02-10T11:30:20.468 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:20.463 7fb53205d700 1 -- 172.21.2.102:0/55445141 learned_addr learned my addr 172.21.2.102:0/55445141 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:30:20.468 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:20.464 7fb53305f700 1 --2- 172.21.2.102:0/55445141 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb5340982c0 0x7fb5340986c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:20.468 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:20.464 7fb53285e700 1 --2- 172.21.2.102:0/55445141 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb5340adf50 0x7fb5340ae350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:20.468 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:20.464 7fb53205d700 1 -- 172.21.2.102:0/55445141 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb5340adf50 msgr2=0x7fb5340ae350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:20.468 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:20.464 7fb53205d700 1 --2- 172.21.2.102:0/55445141 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb5340adf50 0x7fb5340ae350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:20.468 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:20.464 7fb53205d700 1 -- 172.21.2.102:0/55445141 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb5340982c0 msgr2=0x7fb5340986c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:20.468 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:20.464 7fb53205d700 1 --2- 172.21.2.102:0/55445141 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb5340982c0 0x7fb5340986c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:20.468 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:20.464 7fb53205d700 1 -- 172.21.2.102:0/55445141 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5340990f0 con 0x7fb5340ae970 2021-02-10T11:30:20.469 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:20.464 7fb53305f700 1 --2- 172.21.2.102:0/55445141 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb5340982c0 0x7fb5340986c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:30:20.469 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:20.464 7fb53285e700 1 --2- 172.21.2.102:0/55445141 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb5340adf50 0x7fb5340ae350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:30:20.469 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:20.464 7fb53205d700 1 --2- 172.21.2.102:0/55445141 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb5340ae970 0x7fb5340aed70 secure :-1 s=READY pgs=1968 cs=0 l=1 rev1=1 rx=0x7fb51c006f50 tx=0x7fb51c00b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:20.469 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:20.464 7fb52b7fe700 1 -- 172.21.2.102:0/55445141 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb51c00bd50 con 0x7fb5340ae970 2021-02-10T11:30:20.469 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:20.464 7fb52b7fe700 1 -- 172.21.2.102:0/55445141 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb51c000c90 con 0x7fb5340ae970 2021-02-10T11:30:20.469 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:20.464 7fb52b7fe700 1 -- 172.21.2.102:0/55445141 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb51c002300 con 0x7fb5340ae970 2021-02-10T11:30:20.469 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:20.464 7fb538ab3700 1 -- 172.21.2.102:0/55445141 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5340990f0 con 0x7fb5340ae970 2021-02-10T11:30:20.469 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:20.464 7fb538ab3700 1 -- 172.21.2.102:0/55445141 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb534099680 con 0x7fb5340ae970 2021-02-10T11:30:20.469 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:20.465 7fb52b7fe700 1 -- 172.21.2.102:0/55445141 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fb51c002020 con 0x7fb5340ae970 2021-02-10T11:30:20.469 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:20.465 7fb52b7fe700 1 --2- 172.21.2.102:0/55445141 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb5180186c0 0x7fb51801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:20.469 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:20.465 7fb52b7fe700 1 -- 172.21.2.102:0/55445141 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fb51c032580 con 0x7fb5340ae970 2021-02-10T11:30:20.469 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:20.465 7fb53285e700 1 --2- 172.21.2.102:0/55445141 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb5180186c0 0x7fb51801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:20.469 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:20.466 7fb53285e700 1 --2- 172.21.2.102:0/55445141 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb5180186c0 0x7fb51801abc0 crc :-1 s=READY pgs=882 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:20.494 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:20.495 7fb538ab3700 1 -- 172.21.2.102:0/55445141 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb534082a20 con 0x7fb5340ae970 2021-02-10T11:30:20.497 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:20.497 7fb52b7fe700 1 -- 172.21.2.102:0/55445141 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fb51c025fb0 con 0x7fb5340ae970 2021-02-10T11:30:20.626 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:20.626 7fb538ab3700 1 -- 172.21.2.102:0/55445141 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7fb53409f3d0 con 0x7fb5340ae970 2021-02-10T11:30:20.626 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:20.627 7fb52b7fe700 1 -- 172.21.2.102:0/55445141 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v93) v1 ==== 104+0+10 (secure 0 0 0) 0x7fb51c025fb0 con 0x7fb5340ae970 2021-02-10T11:30:20.627 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:30:20.629 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:20.629 7fb538ab3700 1 -- 172.21.2.102:0/55445141 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb5180186c0 msgr2=0x7fb51801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:20.629 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:20.629 7fb538ab3700 1 --2- 172.21.2.102:0/55445141 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb5180186c0 0x7fb51801abc0 crc :-1 s=READY pgs=882 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:20.629 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:20.629 7fb538ab3700 1 -- 172.21.2.102:0/55445141 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb5340ae970 msgr2=0x7fb5340aed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:20.629 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:20.629 7fb538ab3700 1 --2- 172.21.2.102:0/55445141 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb5340ae970 0x7fb5340aed70 secure :-1 s=READY pgs=1968 cs=0 l=1 rev1=1 rx=0x7fb51c006f50 tx=0x7fb51c00b960).stop 2021-02-10T11:30:20.629 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:20.629 7fb538ab3700 1 -- 172.21.2.102:0/55445141 shutdown_connections 2021-02-10T11:30:20.629 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:20.629 7fb538ab3700 1 --2- 172.21.2.102:0/55445141 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb5180186c0 0x7fb51801abc0 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:20.629 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:20.629 7fb538ab3700 1 --2- 172.21.2.102:0/55445141 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb5340982c0 0x7fb5340986c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:20.629 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:20.629 7fb538ab3700 1 --2- 172.21.2.102:0/55445141 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb5340ae970 0x7fb5340aed70 unknown :-1 s=CLOSED pgs=1968 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:20.629 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:20.629 7fb538ab3700 1 --2- 172.21.2.102:0/55445141 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb5340adf50 0x7fb5340ae350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:20.629 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:20.629 7fb538ab3700 1 -- 172.21.2.102:0/55445141 >> 172.21.2.102:0/55445141 conn(0x7fb5340bee90 msgr2=0x7fb5340bb880 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:20.629 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:20.629 7fb538ab3700 1 -- 172.21.2.102:0/55445141 shutdown_connections 2021-02-10T11:30:20.629 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:20.629 7fb538ab3700 1 -- 172.21.2.102:0/55445141 wait complete. 2021-02-10T11:30:20.637 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-02-10T11:30:21.067 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:21.051 7f3dc639e700 1 Processor -- start 2021-02-10T11:30:21.067 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:21.051 7f3dc639e700 1 -- start start 2021-02-10T11:30:21.067 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:21.052 7f3dc639e700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f3dc00d1870 0x7f3dc01b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:21.067 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:21.052 7f3dc639e700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f3dc00cff70 0x7f3dc0109130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:21.067 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:21.052 7f3dc639e700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3dc00fe1c0 con 0x7f3dc01b0ae0 2021-02-10T11:30:21.067 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:21.052 7f3dc639e700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3dc0109ad0 con 0x7f3dc00cff70 2021-02-10T11:30:21.067 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:21.052 7f3dc639e700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3dc0109750 con 0x7f3dc00d1870 2021-02-10T11:30:21.068 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:21.052 7f3dc49bb700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f3dc00cff70 0x7f3dc0109130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:21.068 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:21.052 7f3dc49bb700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f3dc00cff70 0x7f3dc0109130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:60950/0 (socket says 172.21.2.102:60950) 2021-02-10T11:30:21.068 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:21.052 7f3dc49bb700 1 -- 172.21.2.102:0/2744483698 learned_addr learned my addr 172.21.2.102:0/2744483698 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:30:21.068 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:21.052 7f3dc49bb700 1 -- 172.21.2.102:0/2744483698 >> v2:172.21.2.113:3300/0 conn(0x7f3dc00d1870 msgr2=0x7f3dc01b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:21.068 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:21.052 7f3dc49bb700 1 --2- 172.21.2.102:0/2744483698 >> v2:172.21.2.113:3300/0 conn(0x7f3dc00d1870 0x7f3dc01b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:21.068 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:21.052 7f3dc49bb700 1 -- 172.21.2.102:0/2744483698 >> v1:172.21.2.113:6789/0 conn(0x7f3dc01b0ae0 legacy=0x7f3dc0108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:21.068 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:21.052 7f3dc49bb700 1 -- 172.21.2.102:0/2744483698 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3dc0088070 con 0x7f3dc00cff70 2021-02-10T11:30:21.068 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:21.053 7f3dc49bb700 1 --2- 172.21.2.102:0/2744483698 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3dc00cff70 0x7f3dc0109130 secure :-1 s=READY pgs=1969 cs=0 l=1 rev1=1 rx=0x7f3db8006f00 tx=0x7f3db8009f10).ready entity=mon.0 client_cookie=bc9e72ad1a7ebd52 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:21.068 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:21.053 7f3dbeffd700 1 -- 172.21.2.102:0/2744483698 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3db800b3b0 con 0x7f3dc00cff70 2021-02-10T11:30:21.068 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:21.053 7f3dbeffd700 1 -- 172.21.2.102:0/2744483698 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3db800b630 con 0x7f3dc00cff70 2021-02-10T11:30:21.068 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:21.053 7f3dbeffd700 1 -- 172.21.2.102:0/2744483698 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3db800bb50 con 0x7f3dc00cff70 2021-02-10T11:30:21.068 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:21.053 7f3dc639e700 1 -- 172.21.2.102:0/2744483698 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3dc00cff70 msgr2=0x7f3dc0109130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:21.069 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:21.053 7f3dc639e700 1 --2- 172.21.2.102:0/2744483698 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3dc00cff70 0x7f3dc0109130 secure :-1 s=READY pgs=1969 cs=0 l=1 rev1=1 rx=0x7f3db8006f00 tx=0x7f3db8009f10).stop 2021-02-10T11:30:21.069 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:21.053 7f3dc639e700 1 -- 172.21.2.102:0/2744483698 shutdown_connections 2021-02-10T11:30:21.069 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:21.053 7f3dc639e700 1 --2- 172.21.2.102:0/2744483698 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3dc00cff70 0x7f3dc0109130 unknown :-1 s=CLOSED pgs=1969 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:21.069 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:21.053 7f3dc639e700 1 --2- 172.21.2.102:0/2744483698 >> v2:172.21.2.113:3300/0 conn(0x7f3dc00d1870 0x7f3dc01b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:21.069 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:21.053 7f3dc639e700 1 -- 172.21.2.102:0/2744483698 >> 172.21.2.102:0/2744483698 conn(0x7f3dc00ce420 msgr2=0x7f3dc00d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:21.069 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:21.053 7f3dc639e700 1 -- 172.21.2.102:0/2744483698 shutdown_connections 2021-02-10T11:30:21.069 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:21.053 7f3dc639e700 1 -- 172.21.2.102:0/2744483698 wait complete. 2021-02-10T11:30:21.070 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:21.068 7f3dc639e700 1 Processor -- start 2021-02-10T11:30:21.071 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:21.068 7f3dc639e700 1 -- start start 2021-02-10T11:30:21.071 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:21.068 7f3dc639e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3dc00c3820 0x7f3dc0084cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:21.071 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:21.068 7f3dc639e700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3dc00a0290 0x7f3dc00852d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:21.071 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:21.068 7f3dc639e700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3dc0086270 0x7f3dc0086670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:21.071 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:21.068 7f3dc639e700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dc00c0700 con 0x7f3dc00c3820 2021-02-10T11:30:21.071 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:21.068 7f3dc639e700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dc00858f0 con 0x7f3dc00a0290 2021-02-10T11:30:21.071 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:21.068 7f3dc639e700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3dc0085b10 con 0x7f3dc0086270 2021-02-10T11:30:21.071 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:21.069 7f3dbffff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3dc00c3820 0x7f3dc0084cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:21.071 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:21.069 7f3dbffff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3dc00c3820 0x7f3dc0084cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:60956/0 (socket says 172.21.2.102:60956) 2021-02-10T11:30:21.071 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:21.069 7f3dbffff700 1 -- 172.21.2.102:0/2856696666 learned_addr learned my addr 172.21.2.102:0/2856696666 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:30:21.071 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:21.069 7f3dbffff700 1 -- 172.21.2.102:0/2856696666 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3dc0086270 msgr2=0x7f3dc0086670 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:30:21.071 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:21.069 7f3dbffff700 1 --2- 172.21.2.102:0/2856696666 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3dc0086270 0x7f3dc0086670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:21.071 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:21.069 7f3dbffff700 1 -- 172.21.2.102:0/2856696666 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3dc00a0290 msgr2=0x7f3dc00852d0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:30:21.071 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:21.069 7f3dbffff700 1 --2- 172.21.2.102:0/2856696666 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3dc00a0290 0x7f3dc00852d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:21.072 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:21.069 7f3dbffff700 1 -- 172.21.2.102:0/2856696666 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3dc0086c90 con 0x7f3dc00c3820 2021-02-10T11:30:21.072 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:21.069 7f3dbffff700 1 --2- 172.21.2.102:0/2856696666 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3dc00c3820 0x7f3dc0084cb0 secure :-1 s=READY pgs=1970 cs=0 l=1 rev1=1 rx=0x7f3db0006f00 tx=0x7f3db000b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:21.072 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:21.069 7f3dbcff9700 1 -- 172.21.2.102:0/2856696666 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3db000bd30 con 0x7f3dc00c3820 2021-02-10T11:30:21.072 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:21.069 7f3dbcff9700 1 -- 172.21.2.102:0/2856696666 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3db0000c90 con 0x7f3dc00c3820 2021-02-10T11:30:21.072 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:21.069 7f3dbcff9700 1 -- 172.21.2.102:0/2856696666 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3db00022a0 con 0x7f3dc00c3820 2021-02-10T11:30:21.072 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:21.069 7f3dc639e700 1 -- 172.21.2.102:0/2856696666 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3dc0086c90 con 0x7f3dc00c3820 2021-02-10T11:30:21.072 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:21.070 7f3dc639e700 1 -- 172.21.2.102:0/2856696666 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3dc01a2f00 con 0x7f3dc00c3820 2021-02-10T11:30:21.072 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:21.070 7f3dbcff9700 1 -- 172.21.2.102:0/2856696666 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f3db0005310 con 0x7f3dc00c3820 2021-02-10T11:30:21.072 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:21.070 7f3dbcff9700 1 --2- 172.21.2.102:0/2856696666 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3da00186d0 0x7f3da001abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:21.072 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:21.070 7f3dbcff9700 1 -- 172.21.2.102:0/2856696666 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f3db0032770 con 0x7f3dc00c3820 2021-02-10T11:30:21.072 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:21.070 7f3dbf7fe700 1 --2- 172.21.2.102:0/2856696666 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3da00186d0 0x7f3da001abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:21.072 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:21.071 7f3dbf7fe700 1 --2- 172.21.2.102:0/2856696666 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3da00186d0 0x7f3da001abd0 crc :-1 s=READY pgs=883 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:21.101 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:21.102 7f3dc639e700 1 -- 172.21.2.102:0/2856696666 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3dc0082a70 con 0x7f3dc00c3820 2021-02-10T11:30:21.104 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:21.104 7f3dbcff9700 1 -- 172.21.2.102:0/2856696666 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f3db0000960 con 0x7f3dc00c3820 2021-02-10T11:30:21.233 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:21.233 7f3dc639e700 1 -- 172.21.2.102:0/2856696666 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7f3dc0082a70 con 0x7f3dc00c3820 2021-02-10T11:30:21.233 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:21.233 7f3dbcff9700 1 -- 172.21.2.102:0/2856696666 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v93) v1 ==== 100+0+10 (secure 0 0 0) 0x7f3db000bd30 con 0x7f3dc00c3820 2021-02-10T11:30:21.233 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:30:21.235 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:21.235 7f3dc639e700 1 -- 172.21.2.102:0/2856696666 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3da00186d0 msgr2=0x7f3da001abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:21.235 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:21.235 7f3dc639e700 1 --2- 172.21.2.102:0/2856696666 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3da00186d0 0x7f3da001abd0 crc :-1 s=READY pgs=883 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:21.235 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:21.235 7f3dc639e700 1 -- 172.21.2.102:0/2856696666 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3dc00c3820 msgr2=0x7f3dc0084cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:21.236 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:21.235 7f3dc639e700 1 --2- 172.21.2.102:0/2856696666 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3dc00c3820 0x7f3dc0084cb0 secure :-1 s=READY pgs=1970 cs=0 l=1 rev1=1 rx=0x7f3db0006f00 tx=0x7f3db000b940).stop 2021-02-10T11:30:21.236 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:21.235 7f3dc639e700 1 -- 172.21.2.102:0/2856696666 shutdown_connections 2021-02-10T11:30:21.236 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:21.235 7f3dc639e700 1 --2- 172.21.2.102:0/2856696666 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3dc0086270 0x7f3dc0086670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:21.236 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:21.235 7f3dc639e700 1 --2- 172.21.2.102:0/2856696666 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3da00186d0 0x7f3da001abd0 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:21.236 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:21.235 7f3dc639e700 1 --2- 172.21.2.102:0/2856696666 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3dc00a0290 0x7f3dc00852d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:21.236 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:21.235 7f3dc639e700 1 --2- 172.21.2.102:0/2856696666 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3dc00c3820 0x7f3dc0084cb0 unknown :-1 s=CLOSED pgs=1970 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:21.236 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:21.235 7f3dc639e700 1 -- 172.21.2.102:0/2856696666 >> 172.21.2.102:0/2856696666 conn(0x7f3dc00872c0 msgr2=0x7f3dc0095700 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:21.236 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:21.236 7f3dc639e700 1 -- 172.21.2.102:0/2856696666 shutdown_connections 2021-02-10T11:30:21.236 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:21.236 7f3dc639e700 1 -- 172.21.2.102:0/2856696666 wait complete. 2021-02-10T11:30:21.244 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-02-10T11:30:21.244 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:30:21.671 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:21.654 7f92c9755700 1 Processor -- start 2021-02-10T11:30:21.671 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:21.654 7f92c9755700 1 -- start start 2021-02-10T11:30:21.671 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:21.654 7f92c9755700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f92c40f05f0 0x7f92c40f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:21.671 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:21.655 7f92c9755700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92c41b0c70 con 0x7f92c41b0870 2021-02-10T11:30:21.671 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:21.655 7f92c9755700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92c40f0b50 con 0x7f92c41b0470 2021-02-10T11:30:21.671 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:21.655 7f92c9755700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92c40e5ea0 con 0x7f92c40f05f0 2021-02-10T11:30:21.671 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:21.655 7f92c2d48700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f92c41b0870 0x7f92c40efe00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:53188/0 (socket says 172.21.2.102:53188) 2021-02-10T11:30:21.672 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:21.655 7f92c2d48700 1 -- 172.21.2.102:0/157940931 learned_addr learned my addr 172.21.2.102:0/157940931 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:30:21.672 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:21.655 7f92c2547700 1 -- 172.21.2.102:0/157940931 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4071281080 0 0) 0x7f92b0001180 con 0x7f92c41b0870 2021-02-10T11:30:21.672 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:21.655 7f92c2547700 1 -- 172.21.2.102:0/157940931 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92a8002fe0 con 0x7f92c41b0870 2021-02-10T11:30:21.672 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:21.655 7f92c3d4a700 1 --2- 172.21.2.102:0/157940931 >> v2:172.21.2.113:3300/0 conn(0x7f92c40f05f0 0x7f92c40f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:21.672 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:21.656 7f92c2547700 1 -- 172.21.2.102:0/157940931 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 254374615 0 0) 0x7f92b0001180 con 0x7f92c41b0870 2021-02-10T11:30:21.672 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:21.656 7f92c2547700 1 -- 172.21.2.102:0/157940931 >> v2:172.21.2.113:3300/0 conn(0x7f92c40f05f0 msgr2=0x7f92c40f20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:21.672 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:21.656 7f92c2547700 1 --2- 172.21.2.102:0/157940931 >> v2:172.21.2.113:3300/0 conn(0x7f92c40f05f0 0x7f92c40f20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:21.672 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:21.656 7f92c2547700 1 -- 172.21.2.102:0/157940931 >> v1:172.21.2.113:6789/0 conn(0x7f92c41b0470 legacy=0x7f92c40ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:21.672 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:21.656 7f92c2547700 1 -- 172.21.2.102:0/157940931 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92c40e52e0 con 0x7f92c41b0870 2021-02-10T11:30:21.672 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:21.656 7f92c2547700 1 -- 172.21.2.102:0/157940931 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7f92b0001680 con 0x7f92c41b0870 2021-02-10T11:30:21.672 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:21.656 7f92c2547700 1 -- 172.21.2.102:0/157940931 >> v1:172.21.2.102:6789/0 conn(0x7f92c41b0870 legacy=0x7f92c40efe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:21.672 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:21.656 7f92c2547700 1 --2- 172.21.2.102:0/157940931 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f92a8003a60 0x7f92a8005f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:21.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:21.656 7f92c2547700 1 --2- 172.21.2.102:0/157940931 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f92a8006780 0x7f92a8008ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:21.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:21.656 7f92c2547700 1 --2- 172.21.2.102:0/157940931 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f92a80095a0 0x7f92a800baa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:21.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:21.656 7f92c2547700 1 -- 172.21.2.102:0/157940931 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92a800c2f0 con 0x7f92a8003a60 2021-02-10T11:30:21.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:21.656 7f92c2547700 1 -- 172.21.2.102:0/157940931 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92a800c510 con 0x7f92a8006780 2021-02-10T11:30:21.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:21.656 7f92c2547700 1 -- 172.21.2.102:0/157940931 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92a800c730 con 0x7f92a80095a0 2021-02-10T11:30:21.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:21.656 7f92c3549700 1 --2- 172.21.2.102:0/157940931 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f92a8003a60 0x7f92a8005f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:21.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:21.656 7f92c2d48700 1 --2- 172.21.2.102:0/157940931 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f92a8006780 0x7f92a8008ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:21.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:21.656 7f92c3d4a700 1 --2- 172.21.2.102:0/157940931 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f92a80095a0 0x7f92a800baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:21.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:21.656 7f92c3549700 1 -- 172.21.2.102:0/157940931 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f92a80095a0 msgr2=0x7f92a800baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:21.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:21.656 7f92c3549700 1 --2- 172.21.2.102:0/157940931 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f92a80095a0 0x7f92a800baa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:21.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:21.656 7f92c3549700 1 -- 172.21.2.102:0/157940931 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f92a8006780 msgr2=0x7f92a8008ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:21.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:21.656 7f92c3549700 1 --2- 172.21.2.102:0/157940931 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f92a8006780 0x7f92a8008ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:21.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:21.656 7f92c3549700 1 -- 172.21.2.102:0/157940931 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92a80509a0 con 0x7f92a8003a60 2021-02-10T11:30:21.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:21.656 7f92c3d4a700 1 -- 172.21.2.102:0/157940931 reap_dead start 2021-02-10T11:30:21.674 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:21.656 7f92c2d48700 1 --2- 172.21.2.102:0/157940931 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f92a8006780 0x7f92a8008ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:30:21.674 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:21.656 7f92c3549700 1 --2- 172.21.2.102:0/157940931 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f92a8003a60 0x7f92a8005f60 secure :-1 s=READY pgs=1972 cs=0 l=1 rev1=1 rx=0x7f92b4006ab0 tx=0x7f92b4009ac0).ready entity=mon.0 client_cookie=32e33d187df3ea14 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:21.674 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:21.656 7f92c2547700 1 -- 172.21.2.102:0/157940931 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f92b40066a0 con 0x7f92a8003a60 2021-02-10T11:30:21.674 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:21.656 7f92c2547700 1 -- 172.21.2.102:0/157940931 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92b40022e0 con 0x7f92a8003a60 2021-02-10T11:30:21.674 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:21.656 7f92c2547700 1 -- 172.21.2.102:0/157940931 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f92b4000960 con 0x7f92a8003a60 2021-02-10T11:30:21.674 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:21.656 7f92c9755700 1 -- 172.21.2.102:0/157940931 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f92a8003a60 msgr2=0x7f92a8005f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:21.674 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:21.656 7f92c9755700 1 --2- 172.21.2.102:0/157940931 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f92a8003a60 0x7f92a8005f60 secure :-1 s=READY pgs=1972 cs=0 l=1 rev1=1 rx=0x7f92b4006ab0 tx=0x7f92b4009ac0).stop 2021-02-10T11:30:21.674 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:21.657 7f92c9755700 1 -- 172.21.2.102:0/157940931 shutdown_connections 2021-02-10T11:30:21.674 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:21.657 7f92c9755700 1 --2- 172.21.2.102:0/157940931 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f92a8003a60 0x7f92a8005f60 unknown :-1 s=CLOSED pgs=1972 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:21.674 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:21.657 7f92c9755700 1 -- 172.21.2.102:0/157940931 >> 172.21.2.102:0/157940931 conn(0x7f92c40ce3e0 msgr2=0x7f92c40e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:21.674 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:21.657 7f92c9755700 1 -- 172.21.2.102:0/157940931 shutdown_connections 2021-02-10T11:30:21.675 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:21.657 7f92c9755700 1 -- 172.21.2.102:0/157940931 wait complete. 2021-02-10T11:30:21.676 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:21.672 7f92c9755700 1 Processor -- start 2021-02-10T11:30:21.676 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:21.672 7f92c9755700 1 -- start start 2021-02-10T11:30:21.676 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:21.672 7f92c9755700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f92c4108ce0 0x7f92c409da50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:21.676 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:21.672 7f92c9755700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f92c41090e0 0x7f92c409e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:21.676 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:21.672 7f92c9755700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f92c409eee0 0x7f92c409f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:21.676 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:21.672 7f92c9755700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92c41094e0 con 0x7f92c409eee0 2021-02-10T11:30:21.677 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:21.672 7f92c9755700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92c409e960 con 0x7f92c41090e0 2021-02-10T11:30:21.677 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:21.672 7f92c9755700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92c409e690 con 0x7f92c4108ce0 2021-02-10T11:30:21.677 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:21.673 7f92c3d4a700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f92c409eee0 0x7f92c409f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:21.677 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:21.673 7f92c3d4a700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f92c409eee0 0x7f92c409f2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:60980/0 (socket says 172.21.2.102:60980) 2021-02-10T11:30:21.677 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:21.673 7f92c3d4a700 1 -- 172.21.2.102:0/1073710248 learned_addr learned my addr 172.21.2.102:0/1073710248 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:30:21.677 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:21.673 7f92c3d4a700 1 -- 172.21.2.102:0/1073710248 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f92c4108ce0 msgr2=0x7f92c409da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:21.677 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:21.673 7f92c3d4a700 1 --2- 172.21.2.102:0/1073710248 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f92c4108ce0 0x7f92c409da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:21.677 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:21.673 7f92c3d4a700 1 -- 172.21.2.102:0/1073710248 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f92c41090e0 msgr2=0x7f92c409e070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:21.677 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:21.673 7f92c2d48700 1 --2- 172.21.2.102:0/1073710248 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f92c41090e0 0x7f92c409e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:21.677 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:21.673 7f92c3d4a700 1 --2- 172.21.2.102:0/1073710248 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f92c41090e0 0x7f92c409e070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:21.677 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:21.673 7f92c3d4a700 1 -- 172.21.2.102:0/1073710248 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92c409f900 con 0x7f92c409eee0 2021-02-10T11:30:21.677 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:21.673 7f92c2d48700 1 --2- 172.21.2.102:0/1073710248 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f92c41090e0 0x7f92c409e070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:30:21.677 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:21.673 7f92c3d4a700 1 --2- 172.21.2.102:0/1073710248 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f92c409eee0 0x7f92c409f2e0 secure :-1 s=READY pgs=1973 cs=0 l=1 rev1=1 rx=0x7f92bc00b910 tx=0x7f92bc00bcb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:21.678 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:21.673 7f92bbfff700 1 -- 172.21.2.102:0/1073710248 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f92bc00c2f0 con 0x7f92c409eee0 2021-02-10T11:30:21.678 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:21.673 7f92bbfff700 1 -- 172.21.2.102:0/1073710248 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92bc00c5b0 con 0x7f92c409eee0 2021-02-10T11:30:21.678 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:21.673 7f92c9755700 1 -- 172.21.2.102:0/1073710248 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92c409f900 con 0x7f92c409eee0 2021-02-10T11:30:21.678 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:21.673 7f92bbfff700 1 -- 172.21.2.102:0/1073710248 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f92bc00ca00 con 0x7f92c409eee0 2021-02-10T11:30:21.678 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:21.673 7f92c9755700 1 -- 172.21.2.102:0/1073710248 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92c409fe90 con 0x7f92c409eee0 2021-02-10T11:30:21.678 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:21.674 7f92bbfff700 1 -- 172.21.2.102:0/1073710248 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f92bc006a70 con 0x7f92c409eee0 2021-02-10T11:30:21.678 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:21.674 7f92bbfff700 1 --2- 172.21.2.102:0/1073710248 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f92a00186c0 0x7f92a001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:21.678 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:21.674 7f92bbfff700 1 -- 172.21.2.102:0/1073710248 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f92bc032920 con 0x7f92c409eee0 2021-02-10T11:30:21.678 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:21.674 7f92c3549700 1 --2- 172.21.2.102:0/1073710248 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f92a00186c0 0x7f92a001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:21.678 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:21.674 7f92c3549700 1 --2- 172.21.2.102:0/1073710248 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f92a00186c0 0x7f92a001abc0 crc :-1 s=READY pgs=884 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:21.704 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:21.704 7f92c9755700 1 -- 172.21.2.102:0/1073710248 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92c409e960 con 0x7f92c409eee0 2021-02-10T11:30:21.706 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:21.707 7f92bbfff700 1 -- 172.21.2.102:0/1073710248 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f92bc0011f0 con 0x7f92c409eee0 2021-02-10T11:30:21.833 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:21.833 7f92c9755700 1 -- 172.21.2.102:0/1073710248 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f92c409e960 con 0x7f92c409eee0 2021-02-10T11:30:21.833 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:21.834 7f92bbfff700 1 -- 172.21.2.102:0/1073710248 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) v1 ==== 94+0+2098 (secure 0 0 0) 0x7f92bc006a70 con 0x7f92c409eee0 2021-02-10T11:30:21.834 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:30:21.834 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":73,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9672,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.102:6835/2071608955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6834","nonce":2071608955},{"type":"v1","addr":"172.21.2.102:6835","nonce":2071608955}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":69}],"filesystems":[{"mdsmap":{"epoch":73,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:30:04.225997","modified":"2021-02-10 11:30:06.232546","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7186},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7186":{"gid":7186,"name":"b","rank":0,"incarnation":70,"state":"up:active","state_seq":3,"addr":"172.21.2.113:6833/675492649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6832","nonce":675492649},{"type":"v1","addr":"172.21.2.113:6833","nonce":675492649}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":9}]} 2021-02-10T11:30:21.834 INFO:teuthology.orchestra.run.gibba002.stderr:dumped fsmap epoch 73 2021-02-10T11:30:21.835 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:21.836 7f92c9755700 1 -- 172.21.2.102:0/1073710248 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f92a00186c0 msgr2=0x7f92a001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:21.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:21.836 7f92c9755700 1 --2- 172.21.2.102:0/1073710248 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f92a00186c0 0x7f92a001abc0 crc :-1 s=READY pgs=884 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:21.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:21.836 7f92c9755700 1 -- 172.21.2.102:0/1073710248 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f92c409eee0 msgr2=0x7f92c409f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:21.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:21.836 7f92c9755700 1 --2- 172.21.2.102:0/1073710248 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f92c409eee0 0x7f92c409f2e0 secure :-1 s=READY pgs=1973 cs=0 l=1 rev1=1 rx=0x7f92bc00b910 tx=0x7f92bc00bcb0).stop 2021-02-10T11:30:21.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:21.836 7f92c9755700 1 -- 172.21.2.102:0/1073710248 shutdown_connections 2021-02-10T11:30:21.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:21.836 7f92c9755700 1 --2- 172.21.2.102:0/1073710248 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f92c409eee0 0x7f92c409f2e0 unknown :-1 s=CLOSED pgs=1973 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:21.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:21.836 7f92c9755700 1 --2- 172.21.2.102:0/1073710248 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f92a00186c0 0x7f92a001abc0 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:21.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:21.836 7f92c9755700 1 --2- 172.21.2.102:0/1073710248 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f92c41090e0 0x7f92c409e070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:21.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:21.836 7f92c9755700 1 --2- 172.21.2.102:0/1073710248 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f92c4108ce0 0x7f92c409da50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:21.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:21.836 7f92c9755700 1 -- 172.21.2.102:0/1073710248 >> 172.21.2.102:0/1073710248 conn(0x7f92c40bc6c0 msgr2=0x7f92c40b1f70 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:21.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:21.836 7f92c9755700 1 -- 172.21.2.102:0/1073710248 shutdown_connections 2021-02-10T11:30:21.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:21.836 7f92c9755700 1 -- 172.21.2.102:0/1073710248 wait complete. 2021-02-10T11:30:22.046 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 73, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:30:04.225997', 'modified': '2021-02-10 11:30:06.232546', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7186}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7186': {'gid': 7186, 'name': 'b', 'rank': 0, 'incarnation': 70, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.113:6833/675492649', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.113:6832', 'nonce': 675492649}, {'type': 'v1', 'addr': '172.21.2.113:6833', 'nonce': 675492649}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:30:22.046 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:30:22.046 DEBUG:teuthology.orchestra.run.gibba013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:30:22.444 INFO:teuthology.orchestra.run.gibba013.stdout:{ 2021-02-10T11:30:22.444 INFO:teuthology.orchestra.run.gibba013.stdout: "cluster_fsid": "68db30ea-dbce-4aef-a6df-f01c6fea0513", 2021-02-10T11:30:22.444 INFO:teuthology.orchestra.run.gibba013.stdout: "whoami": 0, 2021-02-10T11:30:22.444 INFO:teuthology.orchestra.run.gibba013.stdout: "id": 7186, 2021-02-10T11:30:22.444 INFO:teuthology.orchestra.run.gibba013.stdout: "want_state": "up:active", 2021-02-10T11:30:22.444 INFO:teuthology.orchestra.run.gibba013.stdout: "state": "up:active", 2021-02-10T11:30:22.444 INFO:teuthology.orchestra.run.gibba013.stdout: "rank_uptime": 18.215965185999998, 2021-02-10T11:30:22.445 INFO:teuthology.orchestra.run.gibba013.stdout: "mdsmap_epoch": 73, 2021-02-10T11:30:22.445 INFO:teuthology.orchestra.run.gibba013.stdout: "osdmap_epoch": 93, 2021-02-10T11:30:22.445 INFO:teuthology.orchestra.run.gibba013.stdout: "osdmap_epoch_barrier": 93, 2021-02-10T11:30:22.445 INFO:teuthology.orchestra.run.gibba013.stdout: "uptime": 23.875510808000001 2021-02-10T11:30:22.445 INFO:teuthology.orchestra.run.gibba013.stdout:} 2021-02-10T11:30:22.448 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "68db30ea-dbce-4aef-a6df-f01c6fea0513", "whoami": 0, "id": 7186, "want_state": "up:active", "state": "up:active", "rank_uptime": 18.215965185999998, "mdsmap_epoch": 73, "osdmap_epoch": 93, "osdmap_epoch_barrier": 93, "uptime": 23.875510808000001 } 2021-02-10T11:30:22.448 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2021-02-10T11:30:22.448 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2021-02-10T11:30:22.448 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.1 at ubuntu@gibba002.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.1... 2021-02-10T11:30:22.448 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:30:22.449 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec mkdir -p /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:30:22.785 DEBUG:teuthology.orchestra.run.gibba002:> sudo modprobe fuse 2021-02-10T11:30:23.130 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:30:23.131 DEBUG:teuthology.orchestra.run.gibba002:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:30:23.475 INFO:teuthology.orchestra.run.gibba002.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:30:23.475 INFO:teuthology.orchestra.run.gibba002.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:30:23.476 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:30:23.476 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:30:23.476 DEBUG:teuthology.orchestra.run.gibba002:> ls /sys/fs/fuse/connections 2021-02-10T11:30:23.826 INFO:teuthology.orchestra.run.gibba002.stdout:43 2021-02-10T11:30:23.826 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [43] 2021-02-10T11:30:23.827 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.1 /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:30:23.832 DEBUG:teuthology.orchestra.run.gibba002:> sudo modprobe fuse 2021-02-10T11:30:24.176 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:30:24.177 DEBUG:teuthology.orchestra.run.gibba002:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:30:24.242 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba002.stderr:ceph-fuse[174841]: starting ceph client 2021-02-10T11:30:24.242 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba002.stderr:2021-02-10 11:30:24.242 7f5ade3f3f80 -1 init, newargv = 0x5600d0bcd4d0 newargc=9 2021-02-10T11:30:24.247 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba002.stderr:ceph-fuse[174841]: starting fuse 2021-02-10T11:30:24.522 INFO:teuthology.orchestra.run.gibba002.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:30:24.522 INFO:teuthology.orchestra.run.gibba002.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:30:24.523 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:30:24.523 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:30:24.523 DEBUG:teuthology.orchestra.run.gibba002:> ls /sys/fs/fuse/connections 2021-02-10T11:30:24.858 INFO:teuthology.orchestra.run.gibba002.stdout:43 2021-02-10T11:30:24.859 INFO:teuthology.orchestra.run.gibba002.stdout:44 2021-02-10T11:30:24.859 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43, 44] 2021-02-10T11:30:24.860 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:30:24.860 DEBUG:teuthology.orchestra.run.gibba002:> sudo python3 -c ' 2021-02-10T11:30:24.860 DEBUG:teuthology.orchestra.run.gibba002:> import glob 2021-02-10T11:30:24.860 DEBUG:teuthology.orchestra.run.gibba002:> import re 2021-02-10T11:30:24.860 DEBUG:teuthology.orchestra.run.gibba002:> import os 2021-02-10T11:30:24.860 DEBUG:teuthology.orchestra.run.gibba002:> import subprocess 2021-02-10T11:30:24.860 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:30:24.860 DEBUG:teuthology.orchestra.run.gibba002:> def find_socket(client_name): 2021-02-10T11:30:24.860 DEBUG:teuthology.orchestra.run.gibba002:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2021-02-10T11:30:24.860 DEBUG:teuthology.orchestra.run.gibba002:> files = glob.glob(asok_path) 2021-02-10T11:30:24.860 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:30:24.861 DEBUG:teuthology.orchestra.run.gibba002:> # Given a non-glob path, it better be there 2021-02-10T11:30:24.861 DEBUG:teuthology.orchestra.run.gibba002:> if "*" not in asok_path: 2021-02-10T11:30:24.861 DEBUG:teuthology.orchestra.run.gibba002:> assert(len(files) == 1) 2021-02-10T11:30:24.861 DEBUG:teuthology.orchestra.run.gibba002:> return files[0] 2021-02-10T11:30:24.861 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:30:24.862 DEBUG:teuthology.orchestra.run.gibba002:> for f in files: 2021-02-10T11:30:24.862 DEBUG:teuthology.orchestra.run.gibba002:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2021-02-10T11:30:24.862 DEBUG:teuthology.orchestra.run.gibba002:> if os.path.exists("/proc/{0}".format(pid)): 2021-02-10T11:30:24.862 DEBUG:teuthology.orchestra.run.gibba002:> return f 2021-02-10T11:30:24.863 DEBUG:teuthology.orchestra.run.gibba002:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2021-02-10T11:30:24.863 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:30:24.863 DEBUG:teuthology.orchestra.run.gibba002:> print(find_socket("client.1")) 2021-02-10T11:30:24.863 DEBUG:teuthology.orchestra.run.gibba002:> ' 2021-02-10T11:30:25.224 INFO:teuthology.orchestra.run.gibba002.stdout:/var/run/ceph/ceph-client.1.174841.asok 2021-02-10T11:30:25.227 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.174841.asok 2021-02-10T11:30:25.227 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:30:25.227 DEBUG:teuthology.orchestra.run.gibba002:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.174841.asok status 2021-02-10T11:30:25.618 INFO:teuthology.orchestra.run.gibba002.stdout:{ 2021-02-10T11:30:25.618 INFO:teuthology.orchestra.run.gibba002.stdout: "metadata": { 2021-02-10T11:30:25.618 INFO:teuthology.orchestra.run.gibba002.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:30:25.618 INFO:teuthology.orchestra.run.gibba002.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:30:25.618 INFO:teuthology.orchestra.run.gibba002.stdout: "entity_id": "1", 2021-02-10T11:30:25.619 INFO:teuthology.orchestra.run.gibba002.stdout: "hostname": "gibba002", 2021-02-10T11:30:25.619 INFO:teuthology.orchestra.run.gibba002.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2021-02-10T11:30:25.619 INFO:teuthology.orchestra.run.gibba002.stdout: "pid": "174841", 2021-02-10T11:30:25.619 INFO:teuthology.orchestra.run.gibba002.stdout: "root": "/" 2021-02-10T11:30:25.619 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:30:25.619 INFO:teuthology.orchestra.run.gibba002.stdout: "dentry_count": 0, 2021-02-10T11:30:25.619 INFO:teuthology.orchestra.run.gibba002.stdout: "dentry_pinned_count": 0, 2021-02-10T11:30:25.619 INFO:teuthology.orchestra.run.gibba002.stdout: "id": 9858, 2021-02-10T11:30:25.619 INFO:teuthology.orchestra.run.gibba002.stdout: "inst": { 2021-02-10T11:30:25.619 INFO:teuthology.orchestra.run.gibba002.stdout: "name": { 2021-02-10T11:30:25.619 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "client", 2021-02-10T11:30:25.619 INFO:teuthology.orchestra.run.gibba002.stdout: "num": 9858 2021-02-10T11:30:25.620 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:30:25.620 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": { 2021-02-10T11:30:25.620 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:30:25.620 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:0", 2021-02-10T11:30:25.620 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 719939331 2021-02-10T11:30:25.620 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:30:25.620 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:30:25.620 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": { 2021-02-10T11:30:25.620 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:30:25.620 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:0", 2021-02-10T11:30:25.620 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 719939331 2021-02-10T11:30:25.620 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:30:25.621 INFO:teuthology.orchestra.run.gibba002.stdout: "inst_str": "client.9858 172.21.2.102:0/719939331", 2021-02-10T11:30:25.621 INFO:teuthology.orchestra.run.gibba002.stdout: "addr_str": "172.21.2.102:0/719939331", 2021-02-10T11:30:25.621 INFO:teuthology.orchestra.run.gibba002.stdout: "inode_count": 1, 2021-02-10T11:30:25.621 INFO:teuthology.orchestra.run.gibba002.stdout: "mds_epoch": 73, 2021-02-10T11:30:25.621 INFO:teuthology.orchestra.run.gibba002.stdout: "osd_epoch": 93, 2021-02-10T11:30:25.621 INFO:teuthology.orchestra.run.gibba002.stdout: "osd_epoch_barrier": 0, 2021-02-10T11:30:25.621 INFO:teuthology.orchestra.run.gibba002.stdout: "blacklisted": false 2021-02-10T11:30:25.621 INFO:teuthology.orchestra.run.gibba002.stdout:} 2021-02-10T11:30:25.622 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:30:25.622 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:30:25.622 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:30:25.958 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:30:25.959 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2021-02-10T11:30:25.959 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:30:25.959 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:30:26.305 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2021-02-10T11:30:26.741 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:26.722 7f9a87b84700 1 Processor -- start 2021-02-10T11:30:26.742 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:26.722 7f9a87b84700 1 -- start start 2021-02-10T11:30:26.742 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:26.722 7f9a87b84700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f9a800d1880 0x7f9a801b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:26.742 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:26.722 7f9a87b84700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a800fe1d0 con 0x7f9a800cff80 2021-02-10T11:30:26.742 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:26.722 7f9a87b84700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a80109a70 con 0x7f9a801b0af0 2021-02-10T11:30:26.743 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:26.722 7f9a87b84700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a80088070 con 0x7f9a800d1880 2021-02-10T11:30:26.743 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:26.723 7f9a861a1700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f9a800cff80 0x7f9a80109140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:53226/0 (socket says 172.21.2.102:53226) 2021-02-10T11:30:26.743 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:26.723 7f9a861a1700 1 -- 172.21.2.102:0/1081164497 learned_addr learned my addr 172.21.2.102:0/1081164497 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:30:26.743 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:26.723 7f9a859a0700 1 --2- 172.21.2.102:0/1081164497 >> v2:172.21.2.102:3300/0 conn(0x7f9a800d1880 0x7f9a801b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:26.743 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:26.723 7f9a8499e700 1 -- 172.21.2.102:0/1081164497 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 667323857 0 0) 0x7f9a78000f90 con 0x7f9a800cff80 2021-02-10T11:30:26.744 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:26.723 7f9a8499e700 1 -- 172.21.2.102:0/1081164497 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a6c002fe0 con 0x7f9a800cff80 2021-02-10T11:30:26.744 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:26.723 7f9a859a0700 1 -- 172.21.2.102:0/1081164497 >> v1:172.21.2.113:6789/0 conn(0x7f9a801b0af0 legacy=0x7f9a80108950 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:30:26.744 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:26.723 7f9a859a0700 1 -- 172.21.2.102:0/1081164497 >> v1:172.21.2.102:6789/0 conn(0x7f9a800cff80 legacy=0x7f9a80109140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:26.744 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:26.723 7f9a859a0700 1 -- 172.21.2.102:0/1081164497 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a80088260 con 0x7f9a800d1880 2021-02-10T11:30:26.744 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:26.723 7f9a859a0700 1 --2- 172.21.2.102:0/1081164497 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9a800d1880 0x7f9a801b04d0 secure :-1 s=READY pgs=1978 cs=0 l=1 rev1=1 rx=0x7f9a7c006650 tx=0x7f9a7c009a80).ready entity=mon.0 client_cookie=e7b5ed3e93014f61 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:26.744 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:26.723 7f9a8499e700 1 -- 172.21.2.102:0/1081164497 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9a7c0009b0 con 0x7f9a800d1880 2021-02-10T11:30:26.745 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:26.723 7f9a8499e700 1 -- 172.21.2.102:0/1081164497 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a7c00c0c0 con 0x7f9a800d1880 2021-02-10T11:30:26.745 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:26.724 7f9a8499e700 1 -- 172.21.2.102:0/1081164497 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9a7c00c5e0 con 0x7f9a800d1880 2021-02-10T11:30:26.745 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:26.724 7f9a87b84700 1 -- 172.21.2.102:0/1081164497 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9a800d1880 msgr2=0x7f9a801b04d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:26.745 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:26.724 7f9a87b84700 1 --2- 172.21.2.102:0/1081164497 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9a800d1880 0x7f9a801b04d0 secure :-1 s=READY pgs=1978 cs=0 l=1 rev1=1 rx=0x7f9a7c006650 tx=0x7f9a7c009a80).stop 2021-02-10T11:30:26.745 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:26.724 7f9a87b84700 1 -- 172.21.2.102:0/1081164497 shutdown_connections 2021-02-10T11:30:26.746 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:26.724 7f9a87b84700 1 --2- 172.21.2.102:0/1081164497 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9a800d1880 0x7f9a801b04d0 unknown :-1 s=CLOSED pgs=1978 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:26.746 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:26.724 7f9a87b84700 1 -- 172.21.2.102:0/1081164497 >> 172.21.2.102:0/1081164497 conn(0x7f9a800ce430 msgr2=0x7f9a800d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:26.746 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:26.724 7f9a87b84700 1 -- 172.21.2.102:0/1081164497 shutdown_connections 2021-02-10T11:30:26.746 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:26.724 7f9a87b84700 1 -- 172.21.2.102:0/1081164497 wait complete. 2021-02-10T11:30:26.746 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:26.739 7f9a87b84700 1 Processor -- start 2021-02-10T11:30:26.746 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:26.739 7f9a87b84700 1 -- start start 2021-02-10T11:30:26.747 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:26.739 7f9a87b84700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9a80187e80 0x7f9a80188280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:26.747 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:26.739 7f9a87b84700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f9a801888a0 0x7f9a80189490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:26.747 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:26.739 7f9a87b84700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9a80189ab0 0x7f9a80189eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:26.747 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:26.739 7f9a87b84700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a80188ca0 con 0x7f9a80187e80 2021-02-10T11:30:26.747 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:26.739 7f9a87b84700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a80188f10 con 0x7f9a801888a0 2021-02-10T11:30:26.748 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:26.739 7f9a87b84700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a80099520 con 0x7f9a80189ab0 2021-02-10T11:30:26.748 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:26.740 7f9a859a0700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9a80187e80 0x7f9a80188280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:26.748 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:26.740 7f9a859a0700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9a80187e80 0x7f9a80188280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:32772/0 (socket says 172.21.2.102:32772) 2021-02-10T11:30:26.748 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:26.740 7f9a859a0700 1 -- 172.21.2.102:0/1267671183 learned_addr learned my addr 172.21.2.102:0/1267671183 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:30:26.748 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:26.740 7f9a859a0700 1 -- 172.21.2.102:0/1267671183 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9a80189ab0 msgr2=0x7f9a80189eb0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:30:26.749 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:26.740 7f9a859a0700 1 --2- 172.21.2.102:0/1267671183 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9a80189ab0 0x7f9a80189eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:26.749 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:26.740 7f9a859a0700 1 -- 172.21.2.102:0/1267671183 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f9a801888a0 msgr2=0x7f9a80189490 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:30:26.749 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:26.740 7f9a859a0700 1 --2- 172.21.2.102:0/1267671183 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f9a801888a0 0x7f9a80189490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:26.749 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:26.740 7f9a859a0700 1 -- 172.21.2.102:0/1267671183 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a800999c0 con 0x7f9a80187e80 2021-02-10T11:30:26.749 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:26.740 7f9a859a0700 1 --2- 172.21.2.102:0/1267671183 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9a80187e80 0x7f9a80188280 secure :-1 s=READY pgs=1979 cs=0 l=1 rev1=1 rx=0x7f9a7c00a640 tx=0x7f9a7c00c960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:26.750 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:26.740 7f9a767fc700 1 -- 172.21.2.102:0/1267671183 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9a7c002030 con 0x7f9a80187e80 2021-02-10T11:30:26.750 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:26.740 7f9a767fc700 1 -- 172.21.2.102:0/1267671183 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a7c002270 con 0x7f9a80187e80 2021-02-10T11:30:26.750 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:26.740 7f9a87b84700 1 -- 172.21.2.102:0/1267671183 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a800999c0 con 0x7f9a80187e80 2021-02-10T11:30:26.750 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:26.740 7f9a767fc700 1 -- 172.21.2.102:0/1267671183 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9a7c002790 con 0x7f9a80187e80 2021-02-10T11:30:26.750 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:26.741 7f9a87b84700 1 -- 172.21.2.102:0/1267671183 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a800999c0 con 0x7f9a80187e80 2021-02-10T11:30:26.751 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:26.741 7f9a767fc700 1 -- 172.21.2.102:0/1267671183 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f9a7c000c90 con 0x7f9a80187e80 2021-02-10T11:30:26.751 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:26.741 7f9a767fc700 1 --2- 172.21.2.102:0/1267671183 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f9a68018730 0x7f9a6801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:26.751 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:26.741 7f9a767fc700 1 -- 172.21.2.102:0/1267671183 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f9a7c032810 con 0x7f9a80187e80 2021-02-10T11:30:26.751 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:26.741 7f9a8519f700 1 --2- 172.21.2.102:0/1267671183 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f9a68018730 0x7f9a6801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:26.751 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:26.742 7f9a8519f700 1 --2- 172.21.2.102:0/1267671183 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f9a68018730 0x7f9a6801ac10 crc :-1 s=READY pgs=885 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:26.772 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:26.772 7f9a87b84700 1 -- 172.21.2.102:0/1267671183 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a80195bd0 con 0x7f9a80187e80 2021-02-10T11:30:26.775 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:26.775 7f9a767fc700 1 -- 172.21.2.102:0/1267671183 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f9a7c01f550 con 0x7f9a80187e80 2021-02-10T11:30:26.911 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:26.911 7f9a87b84700 1 -- 172.21.2.102:0/1267671183 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f9a80195bd0 con 0x7f9a80187e80 2021-02-10T11:30:27.087 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:27.088 7f9a767fc700 1 -- 172.21.2.102:0/1267671183 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v74) v1 ==== 136+0+0 (secure 0 0 0) 0x7f9a7c000c90 con 0x7f9a80187e80 2021-02-10T11:30:27.129 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:27.128 7f9a87b84700 1 -- 172.21.2.102:0/1267671183 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f9a68018730 msgr2=0x7f9a6801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:27.129 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:27.128 7f9a87b84700 1 --2- 172.21.2.102:0/1267671183 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f9a68018730 0x7f9a6801ac10 crc :-1 s=READY pgs=885 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:27.129 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:27.128 7f9a87b84700 1 -- 172.21.2.102:0/1267671183 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9a80187e80 msgr2=0x7f9a80188280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:27.129 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:27.128 7f9a87b84700 1 --2- 172.21.2.102:0/1267671183 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9a80187e80 0x7f9a80188280 secure :-1 s=READY pgs=1979 cs=0 l=1 rev1=1 rx=0x7f9a7c00a640 tx=0x7f9a7c00c960).stop 2021-02-10T11:30:27.129 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:27.129 7f9a87b84700 1 -- 172.21.2.102:0/1267671183 shutdown_connections 2021-02-10T11:30:27.129 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:27.129 7f9a87b84700 1 --2- 172.21.2.102:0/1267671183 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9a80189ab0 0x7f9a80189eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:27.129 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:27.129 7f9a87b84700 1 --2- 172.21.2.102:0/1267671183 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f9a68018730 0x7f9a6801ac10 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:27.129 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:27.129 7f9a87b84700 1 --2- 172.21.2.102:0/1267671183 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f9a801888a0 0x7f9a80189490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:27.129 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:27.129 7f9a87b84700 1 --2- 172.21.2.102:0/1267671183 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9a80187e80 0x7f9a80188280 unknown :-1 s=CLOSED pgs=1979 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:27.129 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:27.129 7f9a87b84700 1 -- 172.21.2.102:0/1267671183 >> 172.21.2.102:0/1267671183 conn(0x7f9a8019f940 msgr2=0x7f9a8018c290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:27.129 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:27.129 7f9a87b84700 1 -- 172.21.2.102:0/1267671183 shutdown_connections 2021-02-10T11:30:27.130 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:27.129 7f9a87b84700 1 -- 172.21.2.102:0/1267671183 wait complete. 2021-02-10T11:30:27.137 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs volume ls 2021-02-10T11:30:27.564 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:27.548 7f55d315d700 1 Processor -- start 2021-02-10T11:30:27.564 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:27.548 7f55d315d700 1 -- start start 2021-02-10T11:30:27.564 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:27.548 7f55d315d700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f55cc0d0330 0x7f55cc0ef630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:27.564 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:27.548 7f55d315d700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55cc0f0b00 con 0x7f55cc0d1c60 2021-02-10T11:30:27.564 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:27.548 7f55d315d700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55cc1b0bd0 con 0x7f55cc0d1860 2021-02-10T11:30:27.565 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:27.548 7f55d315d700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55cc0e5ec0 con 0x7f55cc0d0330 2021-02-10T11:30:27.565 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:27.549 7f55d177a700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f55cc0d1c60 0x7f55cc0efc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:53236/0 (socket says 172.21.2.102:53236) 2021-02-10T11:30:27.565 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:27.549 7f55d177a700 1 -- 172.21.2.102:0/2541213127 learned_addr learned my addr 172.21.2.102:0/2541213127 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:30:27.565 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:27.549 7f55cbfff700 1 --2- 172.21.2.102:0/2541213127 >> v2:172.21.2.102:3300/0 conn(0x7f55cc0d0330 0x7f55cc0ef630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:27.565 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:27.549 7f55cbfff700 1 -- 172.21.2.102:0/2541213127 >> v1:172.21.2.113:6789/0 conn(0x7f55cc0d1860 legacy=0x7f55cc1b03e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:27.565 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:27.549 7f55cbfff700 1 -- 172.21.2.102:0/2541213127 >> v1:172.21.2.102:6789/0 conn(0x7f55cc0d1c60 legacy=0x7f55cc0efc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:27.565 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:27.549 7f55cbfff700 1 -- 172.21.2.102:0/2541213127 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55cc0f0440 con 0x7f55cc0d0330 2021-02-10T11:30:27.565 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:27.549 7f55cbfff700 1 --2- 172.21.2.102:0/2541213127 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f55cc0d0330 0x7f55cc0ef630 secure :-1 s=READY pgs=1981 cs=0 l=1 rev1=1 rx=0x7f55bc0069d0 tx=0x7f55bc009e40).ready entity=mon.0 client_cookie=b13aa51cbd74545e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:27.565 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:27.550 7f55cb7fe700 1 -- 172.21.2.102:0/2541213127 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f55bc00b510 con 0x7f55cc0d0330 2021-02-10T11:30:27.565 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:27.550 7f55cb7fe700 1 -- 172.21.2.102:0/2541213127 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55bc00b890 con 0x7f55cc0d0330 2021-02-10T11:30:27.565 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:27.550 7f55cb7fe700 1 -- 172.21.2.102:0/2541213127 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f55bc00bdb0 con 0x7f55cc0d0330 2021-02-10T11:30:27.566 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:27.550 7f55d315d700 1 -- 172.21.2.102:0/2541213127 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f55cc0d0330 msgr2=0x7f55cc0ef630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:27.566 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:27.550 7f55d315d700 1 --2- 172.21.2.102:0/2541213127 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f55cc0d0330 0x7f55cc0ef630 secure :-1 s=READY pgs=1981 cs=0 l=1 rev1=1 rx=0x7f55bc0069d0 tx=0x7f55bc009e40).stop 2021-02-10T11:30:27.566 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:27.550 7f55d315d700 1 -- 172.21.2.102:0/2541213127 shutdown_connections 2021-02-10T11:30:27.566 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:27.550 7f55d315d700 1 --2- 172.21.2.102:0/2541213127 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f55cc0d0330 0x7f55cc0ef630 unknown :-1 s=CLOSED pgs=1981 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:27.566 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:27.550 7f55d315d700 1 -- 172.21.2.102:0/2541213127 >> 172.21.2.102:0/2541213127 conn(0x7f55cc0ce440 msgr2=0x7f55cc0d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:27.566 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:27.550 7f55d315d700 1 -- 172.21.2.102:0/2541213127 shutdown_connections 2021-02-10T11:30:27.566 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:27.550 7f55d315d700 1 -- 172.21.2.102:0/2541213127 wait complete. 2021-02-10T11:30:27.568 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:27.565 7f55d315d700 1 Processor -- start 2021-02-10T11:30:27.568 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:27.565 7f55d315d700 1 -- start start 2021-02-10T11:30:27.568 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:27.565 7f55d315d700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f55cc08e5f0 0x7f55cc1b00a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:27.568 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:27.565 7f55d315d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f55cc1b06c0 0x7f55cc1b0ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:27.568 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:27.566 7f55d315d700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f55cc08f600 0x7f55cc08fa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:27.568 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:27.566 7f55d315d700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55cc08e9f0 con 0x7f55cc1b06c0 2021-02-10T11:30:27.568 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:27.566 7f55d315d700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55cc08ed00 con 0x7f55cc08e5f0 2021-02-10T11:30:27.568 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:27.566 7f55d315d700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55cc1b10e0 con 0x7f55cc08f600 2021-02-10T11:30:27.568 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:27.566 7f55cbfff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f55cc1b06c0 0x7f55cc1b0ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:27.568 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:27.566 7f55cbfff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f55cc1b06c0 0x7f55cc1b0ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:32786/0 (socket says 172.21.2.102:32786) 2021-02-10T11:30:27.568 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:27.566 7f55cbfff700 1 -- 172.21.2.102:0/127669969 learned_addr learned my addr 172.21.2.102:0/127669969 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:30:27.569 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:27.566 7f55cbfff700 1 -- 172.21.2.102:0/127669969 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f55cc08f600 msgr2=0x7f55cc08fa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:27.569 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:27.566 7f55cbfff700 1 --2- 172.21.2.102:0/127669969 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f55cc08f600 0x7f55cc08fa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:27.569 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:27.566 7f55cbfff700 1 -- 172.21.2.102:0/127669969 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f55cc08e5f0 msgr2=0x7f55cc1b00a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:27.569 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:27.566 7f55cbfff700 1 --2- 172.21.2.102:0/127669969 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f55cc08e5f0 0x7f55cc1b00a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:27.569 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:27.566 7f55cbfff700 1 -- 172.21.2.102:0/127669969 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55cc090020 con 0x7f55cc1b06c0 2021-02-10T11:30:27.569 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:27.566 7f55cbfff700 1 --2- 172.21.2.102:0/127669969 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f55cc1b06c0 0x7f55cc1b0ac0 secure :-1 s=READY pgs=1982 cs=0 l=1 rev1=1 rx=0x7f55bc009f40 tx=0x7f55bc00bd50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:27.569 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:27.567 7f55c97fa700 1 -- 172.21.2.102:0/127669969 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f55bc006240 con 0x7f55cc1b06c0 2021-02-10T11:30:27.569 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:27.567 7f55c97fa700 1 -- 172.21.2.102:0/127669969 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55bc0065c0 con 0x7f55cc1b06c0 2021-02-10T11:30:27.569 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:27.567 7f55c97fa700 1 -- 172.21.2.102:0/127669969 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f55bc0058e0 con 0x7f55cc1b06c0 2021-02-10T11:30:27.569 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:27.567 7f55d315d700 1 -- 172.21.2.102:0/127669969 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55cc090020 con 0x7f55cc1b06c0 2021-02-10T11:30:27.569 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:27.567 7f55d315d700 1 -- 172.21.2.102:0/127669969 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55cc0905b0 con 0x7f55cc1b06c0 2021-02-10T11:30:27.570 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:27.567 7f55c97fa700 1 -- 172.21.2.102:0/127669969 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f55bc005660 con 0x7f55cc1b06c0 2021-02-10T11:30:27.570 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:27.567 7f55c97fa700 1 --2- 172.21.2.102:0/127669969 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f55b00186c0 0x7f55b001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:27.570 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:27.567 7f55c97fa700 1 -- 172.21.2.102:0/127669969 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f55bc0328d0 con 0x7f55cc1b06c0 2021-02-10T11:30:27.570 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:27.568 7f55d0f79700 1 --2- 172.21.2.102:0/127669969 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f55b00186c0 0x7f55b001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:27.570 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:27.568 7f55d0f79700 1 --2- 172.21.2.102:0/127669969 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f55b00186c0 0x7f55b001abc0 crc :-1 s=READY pgs=886 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:27.598 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:27.598 7f55d315d700 1 -- 172.21.2.102:0/127669969 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55cc0ff1b0 con 0x7f55cc1b06c0 2021-02-10T11:30:27.600 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:27.601 7f55c97fa700 1 -- 172.21.2.102:0/127669969 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f55bc000d30 con 0x7f55cc1b06c0 2021-02-10T11:30:27.734 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:27.734 7f55d315d700 1 -- 172.21.2.102:0/127669969 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs volume ls", "target": ["mgr", ""]}) v1 -- 0x7f55cc0905b0 con 0x7f55b00186c0 2021-02-10T11:30:27.735 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:27.735 7f55c97fa700 1 -- 172.21.2.102:0/127669969 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+41 (crc 0 0 0) 0x7f55b4006e40 con 0x7f55b00186c0 2021-02-10T11:30:27.735 INFO:teuthology.orchestra.run.gibba002.stdout:[ 2021-02-10T11:30:27.735 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:30:27.735 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "cephfs" 2021-02-10T11:30:27.735 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:30:27.735 INFO:teuthology.orchestra.run.gibba002.stdout:] 2021-02-10T11:30:27.736 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:27.736 7f55d315d700 1 -- 172.21.2.102:0/127669969 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f55b00186c0 msgr2=0x7f55b001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:27.736 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:27.736 7f55d315d700 1 --2- 172.21.2.102:0/127669969 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f55b00186c0 0x7f55b001abc0 crc :-1 s=READY pgs=886 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:27.736 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:27.737 7f55d315d700 1 -- 172.21.2.102:0/127669969 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f55cc1b06c0 msgr2=0x7f55cc1b0ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:27.736 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:27.737 7f55d315d700 1 --2- 172.21.2.102:0/127669969 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f55cc1b06c0 0x7f55cc1b0ac0 secure :-1 s=READY pgs=1982 cs=0 l=1 rev1=1 rx=0x7f55bc009f40 tx=0x7f55bc00bd50).stop 2021-02-10T11:30:27.737 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:27.737 7f55d315d700 1 -- 172.21.2.102:0/127669969 shutdown_connections 2021-02-10T11:30:27.737 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:27.737 7f55d315d700 1 --2- 172.21.2.102:0/127669969 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f55cc08f600 0x7f55cc08fa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:27.737 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:27.737 7f55d315d700 1 --2- 172.21.2.102:0/127669969 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f55cc1b06c0 0x7f55cc1b0ac0 unknown :-1 s=CLOSED pgs=1982 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:27.737 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:27.737 7f55d315d700 1 --2- 172.21.2.102:0/127669969 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f55b00186c0 0x7f55b001abc0 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:27.737 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:27.737 7f55d315d700 1 --2- 172.21.2.102:0/127669969 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f55cc08e5f0 0x7f55cc1b00a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:27.737 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:27.737 7f55d315d700 1 -- 172.21.2.102:0/127669969 >> 172.21.2.102:0/127669969 conn(0x7f55cc0b8980 msgr2=0x7f55cc0b34a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:27.737 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:27.737 7f55d315d700 1 -- 172.21.2.102:0/127669969 shutdown_connections 2021-02-10T11:30:27.737 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:27.737 7f55d315d700 1 -- 172.21.2.102:0/127669969 wait complete. 2021-02-10T11:30:27.745 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config set mon mon_allow_pool_delete True 2021-02-10T11:30:28.177 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:28.161 7f0147510700 1 Processor -- start 2021-02-10T11:30:28.177 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:28.161 7f0147510700 1 -- start start 2021-02-10T11:30:28.177 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:28.161 7f0147510700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f01400d1c70 0x7f01400cef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:28.177 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:28.161 7f0147510700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f01400cf580 0x7f01400cf980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:28.177 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:28.161 7f0147510700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01400fe1c0 con 0x7f01400d1870 2021-02-10T11:30:28.177 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:28.161 7f0147510700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01401b0cb0 con 0x7f01400cf580 2021-02-10T11:30:28.177 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:28.161 7f0147510700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0140109a60 con 0x7f01400d1c70 2021-02-10T11:30:28.177 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:28.162 7f0145b2d700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f01400cf580 0x7f01400cf980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:28.178 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:28.162 7f0145b2d700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f01400cf580 0x7f01400cf980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:32792/0 (socket says 172.21.2.102:32792) 2021-02-10T11:30:28.178 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:28.162 7f0145b2d700 1 -- 172.21.2.102:0/1345459505 learned_addr learned my addr 172.21.2.102:0/1345459505 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:30:28.178 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:28.162 7f0145b2d700 1 -- 172.21.2.102:0/1345459505 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f01400d1c70 msgr2=0x7f01400cef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:28.178 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:28.162 7f0145b2d700 1 --2- 172.21.2.102:0/1345459505 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f01400d1c70 0x7f01400cef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:28.178 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:28.162 7f0145b2d700 1 -- 172.21.2.102:0/1345459505 >> v1:172.21.2.113:6789/0 conn(0x7f01400d1870 legacy=0x7f01401b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:28.178 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:28.162 7f0145b2d700 1 -- 172.21.2.102:0/1345459505 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0140088070 con 0x7f01400cf580 2021-02-10T11:30:28.178 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:28.162 7f0145b2d700 1 --2- 172.21.2.102:0/1345459505 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f01400cf580 0x7f01400cf980 secure :-1 s=READY pgs=1983 cs=0 l=1 rev1=1 rx=0x7f0138006f00 tx=0x7f0138009f10).ready entity=mon.0 client_cookie=6973f41ed171e0ba server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:28.178 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:28.162 7f0133fff700 1 -- 172.21.2.102:0/1345459505 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f013800b3b0 con 0x7f01400cf580 2021-02-10T11:30:28.178 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:28.162 7f0133fff700 1 -- 172.21.2.102:0/1345459505 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f013800b730 con 0x7f01400cf580 2021-02-10T11:30:28.178 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:28.162 7f0133fff700 1 -- 172.21.2.102:0/1345459505 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f013800bc50 con 0x7f01400cf580 2021-02-10T11:30:28.178 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:28.162 7f0147510700 1 -- 172.21.2.102:0/1345459505 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f01400cf580 msgr2=0x7f01400cf980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:28.178 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:28.162 7f0147510700 1 --2- 172.21.2.102:0/1345459505 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f01400cf580 0x7f01400cf980 secure :-1 s=READY pgs=1983 cs=0 l=1 rev1=1 rx=0x7f0138006f00 tx=0x7f0138009f10).stop 2021-02-10T11:30:28.178 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:28.163 7f0147510700 1 -- 172.21.2.102:0/1345459505 shutdown_connections 2021-02-10T11:30:28.179 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:28.163 7f0147510700 1 --2- 172.21.2.102:0/1345459505 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f01400d1c70 0x7f01400cef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:28.179 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:28.163 7f0147510700 1 --2- 172.21.2.102:0/1345459505 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f01400cf580 0x7f01400cf980 unknown :-1 s=CLOSED pgs=1983 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:28.179 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:28.163 7f0147510700 1 -- 172.21.2.102:0/1345459505 >> 172.21.2.102:0/1345459505 conn(0x7f01400ce420 msgr2=0x7f01400d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:28.179 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:28.163 7f0147510700 1 -- 172.21.2.102:0/1345459505 shutdown_connections 2021-02-10T11:30:28.179 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:28.163 7f0147510700 1 -- 172.21.2.102:0/1345459505 wait complete. 2021-02-10T11:30:28.180 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:28.178 7f0147510700 1 Processor -- start 2021-02-10T11:30:28.180 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:28.178 7f0147510700 1 -- start start 2021-02-10T11:30:28.180 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:28.178 7f0147510700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f01400c3820 0x7f01400a7950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:28.181 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:28.178 7f0147510700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f01401a44c0 0x7f01401a48c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:28.181 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:28.178 7f0147510700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f01400863d0 0x7f01400867d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:28.181 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:28.178 7f0147510700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0140085ed0 con 0x7f01400c3820 2021-02-10T11:30:28.181 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:28.178 7f0147510700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01400c0740 con 0x7f01401a44c0 2021-02-10T11:30:28.181 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:28.178 7f0147510700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0140086df0 con 0x7f01400863d0 2021-02-10T11:30:28.181 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:28.178 7f014532c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f01400c3820 0x7f01400a7950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:28.181 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:28.178 7f014532c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f01400c3820 0x7f01400a7950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:32802/0 (socket says 172.21.2.102:32802) 2021-02-10T11:30:28.181 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:28.178 7f014532c700 1 -- 172.21.2.102:0/2619590107 learned_addr learned my addr 172.21.2.102:0/2619590107 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:30:28.181 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:28.178 7f0144b2b700 1 --2- 172.21.2.102:0/2619590107 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f01401a44c0 0x7f01401a48c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:28.181 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:28.178 7f0145b2d700 1 --2- 172.21.2.102:0/2619590107 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f01400863d0 0x7f01400867d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:28.181 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:28.179 7f014532c700 1 -- 172.21.2.102:0/2619590107 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f01400863d0 msgr2=0x7f01400867d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:28.181 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:28.179 7f014532c700 1 --2- 172.21.2.102:0/2619590107 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f01400863d0 0x7f01400867d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:28.182 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:28.179 7f014532c700 1 -- 172.21.2.102:0/2619590107 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f01401a44c0 msgr2=0x7f01401a48c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:28.182 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:28.179 7f014532c700 1 --2- 172.21.2.102:0/2619590107 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f01401a44c0 0x7f01401a48c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:28.182 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:28.179 7f014532c700 1 -- 172.21.2.102:0/2619590107 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0140087010 con 0x7f01400c3820 2021-02-10T11:30:28.182 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:28.179 7f014532c700 1 --2- 172.21.2.102:0/2619590107 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f01400c3820 0x7f01400a7950 secure :-1 s=READY pgs=1984 cs=0 l=1 rev1=1 rx=0x7f013400b5b0 tx=0x7f013400b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:28.182 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:28.179 7f0131ffb700 1 -- 172.21.2.102:0/2619590107 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f013400bd40 con 0x7f01400c3820 2021-02-10T11:30:28.182 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:28.179 7f0131ffb700 1 -- 172.21.2.102:0/2619590107 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0134000e20 con 0x7f01400c3820 2021-02-10T11:30:28.182 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:28.179 7f0131ffb700 1 -- 172.21.2.102:0/2619590107 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f01340054a0 con 0x7f01400c3820 2021-02-10T11:30:28.182 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:28.179 7f0147510700 1 -- 172.21.2.102:0/2619590107 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0140087010 con 0x7f01400c3820 2021-02-10T11:30:28.182 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:28.179 7f0147510700 1 -- 172.21.2.102:0/2619590107 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0140083000 con 0x7f01400c3820 2021-02-10T11:30:28.182 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:28.180 7f0131ffb700 1 -- 172.21.2.102:0/2619590107 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f0134006030 con 0x7f01400c3820 2021-02-10T11:30:28.182 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:28.180 7f0131ffb700 1 --2- 172.21.2.102:0/2619590107 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f01200186c0 0x7f012001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:28.183 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:28.180 7f0131ffb700 1 -- 172.21.2.102:0/2619590107 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f0134032720 con 0x7f01400c3820 2021-02-10T11:30:28.183 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:28.180 7f0144b2b700 1 --2- 172.21.2.102:0/2619590107 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f01200186c0 0x7f012001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:28.183 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:28.180 7f0144b2b700 1 --2- 172.21.2.102:0/2619590107 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f01200186c0 0x7f012001abc0 crc :-1 s=READY pgs=887 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:28.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:28.211 7f0147510700 1 -- 172.21.2.102:0/2619590107 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0140085e50 con 0x7f01400c3820 2021-02-10T11:30:28.214 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:28.214 7f0131ffb700 1 -- 172.21.2.102:0/2619590107 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f01340009b0 con 0x7f01400c3820 2021-02-10T11:30:28.367 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:28.367 7f0147510700 1 -- 172.21.2.102:0/2619590107 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f0140085e50 con 0x7f01400c3820 2021-02-10T11:30:28.367 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:28.368 7f0131ffb700 1 -- 172.21.2.102:0/2619590107 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v24)=0 v24) v1 ==== 122+0+0 (secure 0 0 0) 0x7f0134000ce0 con 0x7f01400c3820 2021-02-10T11:30:28.369 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:28.370 7f0147510700 1 -- 172.21.2.102:0/2619590107 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f01200186c0 msgr2=0x7f012001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:28.370 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:28.370 7f0147510700 1 --2- 172.21.2.102:0/2619590107 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f01200186c0 0x7f012001abc0 crc :-1 s=READY pgs=887 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:28.370 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:28.370 7f0147510700 1 -- 172.21.2.102:0/2619590107 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f01400c3820 msgr2=0x7f01400a7950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:28.370 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:28.370 7f0147510700 1 --2- 172.21.2.102:0/2619590107 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f01400c3820 0x7f01400a7950 secure :-1 s=READY pgs=1984 cs=0 l=1 rev1=1 rx=0x7f013400b5b0 tx=0x7f013400b950).stop 2021-02-10T11:30:28.371 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:28.370 7f0147510700 1 -- 172.21.2.102:0/2619590107 shutdown_connections 2021-02-10T11:30:28.371 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:28.370 7f0147510700 1 --2- 172.21.2.102:0/2619590107 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f01400863d0 0x7f01400867d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:28.371 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:28.370 7f0147510700 1 --2- 172.21.2.102:0/2619590107 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f01200186c0 0x7f012001abc0 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:28.371 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:28.370 7f0147510700 1 --2- 172.21.2.102:0/2619590107 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f01401a44c0 0x7f01401a48c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:28.371 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:28.370 7f0147510700 1 --2- 172.21.2.102:0/2619590107 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f01400c3820 0x7f01400a7950 unknown :-1 s=CLOSED pgs=1984 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:28.372 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:28.370 7f0147510700 1 -- 172.21.2.102:0/2619590107 >> 172.21.2.102:0/2619590107 conn(0x7f01400872c0 msgr2=0x7f0140095700 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:28.372 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:28.370 7f0147510700 1 -- 172.21.2.102:0/2619590107 shutdown_connections 2021-02-10T11:30:28.372 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:28.370 7f0147510700 1 -- 172.21.2.102:0/2619590107 wait complete. 2021-02-10T11:30:28.379 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolumegroup create cephfs group_0000000000973488 2021-02-10T11:30:28.809 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:28.793 7f8f3abff700 1 Processor -- start 2021-02-10T11:30:28.809 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:28.793 7f8f3abff700 1 -- start start 2021-02-10T11:30:28.809 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:28.794 7f8f3abff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f8f340d1c70 0x7f8f340cef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:28.810 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:28.794 7f8f3abff700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8f340cf580 0x7f8f340cf980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:28.810 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:28.794 7f8f3abff700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f340fe1c0 con 0x7f8f340d1870 2021-02-10T11:30:28.810 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:28.794 7f8f3abff700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f341b0cb0 con 0x7f8f340d1c70 2021-02-10T11:30:28.810 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:28.794 7f8f3abff700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f34109a60 con 0x7f8f340cf580 2021-02-10T11:30:28.810 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:28.794 7f8f33fff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f8f340d1c70 0x7f8f340cef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:28.810 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:28.794 7f8f38a1b700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f8f340d1870 0x7f8f341b04c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:53262/0 (socket says 172.21.2.102:53262) 2021-02-10T11:30:28.810 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:28.794 7f8f38a1b700 1 -- 172.21.2.102:0/3107221443 learned_addr learned my addr 172.21.2.102:0/3107221443 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:30:28.810 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:28.794 7f8f33fff700 1 -- 172.21.2.102:0/3107221443 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8f340cf580 msgr2=0x7f8f340cf980 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:30:28.810 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:28.794 7f8f33fff700 1 --2- 172.21.2.102:0/3107221443 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8f340cf580 0x7f8f340cf980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:28.810 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:28.794 7f8f33fff700 1 -- 172.21.2.102:0/3107221443 >> v1:172.21.2.102:6789/0 conn(0x7f8f340d1870 legacy=0x7f8f341b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:28.810 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:28.794 7f8f33fff700 1 -- 172.21.2.102:0/3107221443 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f34088070 con 0x7f8f340d1c70 2021-02-10T11:30:28.810 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:28.795 7f8f33fff700 1 --2- 172.21.2.102:0/3107221443 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8f340d1c70 0x7f8f340cef60 secure :-1 s=READY pgs=1986 cs=0 l=1 rev1=1 rx=0x7f8f24006650 tx=0x7f8f24009a80).ready entity=mon.0 client_cookie=3f1e69014e86e2fa server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:28.811 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:28.795 7f8f337fe700 1 -- 172.21.2.102:0/3107221443 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8f2400c0c0 con 0x7f8f340d1c70 2021-02-10T11:30:28.811 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:28.795 7f8f337fe700 1 -- 172.21.2.102:0/3107221443 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f2400c3c0 con 0x7f8f340d1c70 2021-02-10T11:30:28.811 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:28.795 7f8f337fe700 1 -- 172.21.2.102:0/3107221443 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8f2400c8e0 con 0x7f8f340d1c70 2021-02-10T11:30:28.811 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:28.795 7f8f3abff700 1 -- 172.21.2.102:0/3107221443 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8f340d1c70 msgr2=0x7f8f340cef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:28.811 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:28.795 7f8f3abff700 1 --2- 172.21.2.102:0/3107221443 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8f340d1c70 0x7f8f340cef60 secure :-1 s=READY pgs=1986 cs=0 l=1 rev1=1 rx=0x7f8f24006650 tx=0x7f8f24009a80).stop 2021-02-10T11:30:28.811 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:28.795 7f8f3abff700 1 -- 172.21.2.102:0/3107221443 shutdown_connections 2021-02-10T11:30:28.811 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:28.795 7f8f3abff700 1 --2- 172.21.2.102:0/3107221443 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8f340d1c70 0x7f8f340cef60 unknown :-1 s=CLOSED pgs=1986 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:28.811 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:28.795 7f8f3abff700 1 --2- 172.21.2.102:0/3107221443 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8f340cf580 0x7f8f340cf980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:28.811 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:28.795 7f8f3abff700 1 -- 172.21.2.102:0/3107221443 >> 172.21.2.102:0/3107221443 conn(0x7f8f340ce420 msgr2=0x7f8f340d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:28.811 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:28.795 7f8f3abff700 1 -- 172.21.2.102:0/3107221443 shutdown_connections 2021-02-10T11:30:28.811 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:28.795 7f8f3abff700 1 -- 172.21.2.102:0/3107221443 wait complete. 2021-02-10T11:30:28.813 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:28.810 7f8f3abff700 1 Processor -- start 2021-02-10T11:30:28.813 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:28.810 7f8f3abff700 1 -- start start 2021-02-10T11:30:28.813 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:28.810 7f8f3abff700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8f340c2370 0x7f8f3418f870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:28.813 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:28.810 7f8f3abff700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8f34084cb0 0x7f8f341a7470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:28.813 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:28.810 7f8f3abff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8f341a7a90 0x7f8f34085700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:28.813 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:28.810 7f8f3abff700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f340be840 con 0x7f8f341a7a90 2021-02-10T11:30:28.813 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:28.810 7f8f3abff700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f340c29a0 con 0x7f8f340c2370 2021-02-10T11:30:28.813 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:28.810 7f8f3abff700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f341a7e90 con 0x7f8f34084cb0 2021-02-10T11:30:28.813 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:28.811 7f8f3921c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8f341a7a90 0x7f8f34085700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:28.813 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:28.811 7f8f3921c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8f341a7a90 0x7f8f34085700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:32814/0 (socket says 172.21.2.102:32814) 2021-02-10T11:30:28.813 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:28.811 7f8f3921c700 1 -- 172.21.2.102:0/479866981 learned_addr learned my addr 172.21.2.102:0/479866981 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:30:28.813 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:28.811 7f8f3921c700 1 -- 172.21.2.102:0/479866981 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8f34084cb0 msgr2=0x7f8f341a7470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:28.814 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:28.811 7f8f3921c700 1 --2- 172.21.2.102:0/479866981 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8f34084cb0 0x7f8f341a7470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:28.814 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:28.811 7f8f3921c700 1 -- 172.21.2.102:0/479866981 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8f340c2370 msgr2=0x7f8f3418f870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:28.814 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:28.811 7f8f3921c700 1 --2- 172.21.2.102:0/479866981 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8f340c2370 0x7f8f3418f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:28.814 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:28.811 7f8f3921c700 1 -- 172.21.2.102:0/479866981 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f341a80b0 con 0x7f8f341a7a90 2021-02-10T11:30:28.814 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:28.811 7f8f3921c700 1 --2- 172.21.2.102:0/479866981 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8f341a7a90 0x7f8f34085700 secure :-1 s=READY pgs=1987 cs=0 l=1 rev1=1 rx=0x7f8f2c009fc0 tx=0x7f8f2c00bc40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:28.814 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:28.812 7f8f317fa700 1 -- 172.21.2.102:0/479866981 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8f2c003020 con 0x7f8f341a7a90 2021-02-10T11:30:28.814 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:28.812 7f8f317fa700 1 -- 172.21.2.102:0/479866981 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f2c005850 con 0x7f8f341a7a90 2021-02-10T11:30:28.814 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:28.812 7f8f317fa700 1 -- 172.21.2.102:0/479866981 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8f2c005d70 con 0x7f8f341a7a90 2021-02-10T11:30:28.814 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:28.812 7f8f3abff700 1 -- 172.21.2.102:0/479866981 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f341a80b0 con 0x7f8f341a7a90 2021-02-10T11:30:28.814 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:28.812 7f8f3abff700 1 -- 172.21.2.102:0/479866981 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f341a80b0 con 0x7f8f341a7a90 2021-02-10T11:30:28.814 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:28.812 7f8f317fa700 1 -- 172.21.2.102:0/479866981 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f8f2c000d90 con 0x7f8f341a7a90 2021-02-10T11:30:28.814 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:28.812 7f8f317fa700 1 --2- 172.21.2.102:0/479866981 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8f180186c0 0x7f8f1801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:28.814 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:28.812 7f8f317fa700 1 -- 172.21.2.102:0/479866981 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f8f2c032b10 con 0x7f8f341a7a90 2021-02-10T11:30:28.815 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:28.812 7f8f38a1b700 1 --2- 172.21.2.102:0/479866981 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8f180186c0 0x7f8f1801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:28.815 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:28.813 7f8f38a1b700 1 --2- 172.21.2.102:0/479866981 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8f180186c0 0x7f8f1801abc0 crc :-1 s=READY pgs=888 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:28.843 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:28.844 7f8f3abff700 1 -- 172.21.2.102:0/479866981 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f340be840 con 0x7f8f341a7a90 2021-02-10T11:30:28.846 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:28.846 7f8f317fa700 1 -- 172.21.2.102:0/479866981 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f8f2c00bd80 con 0x7f8f341a7a90 2021-02-10T11:30:28.977 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:28.976 7f8f3abff700 1 -- 172.21.2.102:0/479866981 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "target": ["mgr", ""], "group_name": "group_0000000000973488"}) v1 -- 0x7f8f340be840 con 0x7f8f180186c0 2021-02-10T11:30:28.980 INFO:tasks.ceph.mgr.y.gibba002.stderr:2021-02-10 11:30:28.979 7fd94bc53700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:30:28.980 INFO:tasks.ceph.mgr.y.gibba002.stderr:2021-02-10 11:30:28.979 7fd94bc53700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:30:28.980 INFO:tasks.ceph.mgr.y.gibba002.stderr:2021-02-10 11:30:28.979 7fd94bc53700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:30:28.980 INFO:tasks.ceph.mgr.y.gibba002.stderr:2021-02-10 11:30:28.979 7fd94bc53700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:30:28.980 INFO:tasks.ceph.mgr.y.gibba002.stderr:2021-02-10 11:30:28.979 7fd94bc53700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:30:28.995 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:28.994 7f8f317fa700 1 -- 172.21.2.102:0/479866981 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f8f1c000e90 con 0x7f8f180186c0 2021-02-10T11:30:29.010 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:29.009 7f8f3abff700 1 -- 172.21.2.102:0/479866981 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8f180186c0 msgr2=0x7f8f1801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:29.010 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:29.009 7f8f3abff700 1 --2- 172.21.2.102:0/479866981 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8f180186c0 0x7f8f1801abc0 crc :-1 s=READY pgs=888 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:29.010 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:29.009 7f8f3abff700 1 -- 172.21.2.102:0/479866981 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8f341a7a90 msgr2=0x7f8f34085700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:29.010 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:29.009 7f8f3abff700 1 --2- 172.21.2.102:0/479866981 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8f341a7a90 0x7f8f34085700 secure :-1 s=READY pgs=1987 cs=0 l=1 rev1=1 rx=0x7f8f2c009fc0 tx=0x7f8f2c00bc40).stop 2021-02-10T11:30:29.011 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:29.009 7f8f3abff700 1 -- 172.21.2.102:0/479866981 shutdown_connections 2021-02-10T11:30:29.011 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:29.009 7f8f3abff700 1 --2- 172.21.2.102:0/479866981 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8f341a7a90 0x7f8f34085700 unknown :-1 s=CLOSED pgs=1987 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:29.011 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:29.009 7f8f3abff700 1 --2- 172.21.2.102:0/479866981 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8f34084cb0 0x7f8f341a7470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:29.011 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:29.009 7f8f3abff700 1 --2- 172.21.2.102:0/479866981 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8f180186c0 0x7f8f1801abc0 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:29.011 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:29.009 7f8f3abff700 1 --2- 172.21.2.102:0/479866981 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8f340c2370 0x7f8f3418f870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:29.011 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:29.009 7f8f3abff700 1 -- 172.21.2.102:0/479866981 >> 172.21.2.102:0/479866981 conn(0x7f8f340b5dc0 msgr2=0x7f8f340c2d90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:29.011 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:29.010 7f8f3abff700 1 -- 172.21.2.102:0/479866981 shutdown_connections 2021-02-10T11:30:29.011 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:29.010 7f8f3abff700 1 -- 172.21.2.102:0/479866981 wait complete. 2021-02-10T11:30:29.018 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000592823 --group_name group_0000000000973488 2021-02-10T11:30:29.449 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:29.432 7efe0dba7700 1 Processor -- start 2021-02-10T11:30:29.449 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:29.432 7efe0dba7700 1 -- start start 2021-02-10T11:30:29.449 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:29.432 7efe0dba7700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7efe080d1880 0x7efe081b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:29.449 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:29.432 7efe0dba7700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7efe081b0af0 0x7efe08108950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:29.449 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:29.432 7efe0dba7700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe08109a70 con 0x7efe080d1c80 2021-02-10T11:30:29.449 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:29.432 7efe0dba7700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe080e5f00 con 0x7efe081b0af0 2021-02-10T11:30:29.450 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:29.432 7efe0dba7700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe08109760 con 0x7efe080d1880 2021-02-10T11:30:29.450 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:29.432 7efe06ffd700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7efe081b0af0 0x7efe08108950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:29.450 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:29.432 7efe07fff700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7efe080d1c80 0x7efe08108f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:53288/0 (socket says 172.21.2.102:53288) 2021-02-10T11:30:29.450 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:29.433 7efe07fff700 1 -- 172.21.2.102:0/3500145480 learned_addr learned my addr 172.21.2.102:0/3500145480 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:30:29.450 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:29.433 7efe06ffd700 1 -- 172.21.2.102:0/3500145480 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7efe080d1880 msgr2=0x7efe081b04d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:30:29.450 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:29.433 7efe06ffd700 1 --2- 172.21.2.102:0/3500145480 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7efe080d1880 0x7efe081b04d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:29.450 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:29.433 7efe06ffd700 1 -- 172.21.2.102:0/3500145480 >> v1:172.21.2.102:6789/0 conn(0x7efe080d1c80 legacy=0x7efe08108f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:29.450 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:29.433 7efe06ffd700 1 -- 172.21.2.102:0/3500145480 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe08088070 con 0x7efe081b0af0 2021-02-10T11:30:29.450 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:29.433 7efe06ffd700 1 --2- 172.21.2.102:0/3500145480 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efe081b0af0 0x7efe08108950 secure :-1 s=READY pgs=1991 cs=0 l=1 rev1=1 rx=0x7efdfc006650 tx=0x7efdfc009a00).ready entity=mon.0 client_cookie=8fd7c85affe8da65 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:29.450 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:29.433 7efe067fc700 1 -- 172.21.2.102:0/3500145480 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efdfc0009b0 con 0x7efe081b0af0 2021-02-10T11:30:29.450 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:29.433 7efe067fc700 1 -- 172.21.2.102:0/3500145480 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdfc00c0c0 con 0x7efe081b0af0 2021-02-10T11:30:29.450 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:29.433 7efe067fc700 1 -- 172.21.2.102:0/3500145480 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efdfc00c5a0 con 0x7efe081b0af0 2021-02-10T11:30:29.451 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:29.433 7efe0dba7700 1 -- 172.21.2.102:0/3500145480 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efe081b0af0 msgr2=0x7efe08108950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:29.451 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:29.433 7efe0dba7700 1 --2- 172.21.2.102:0/3500145480 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efe081b0af0 0x7efe08108950 secure :-1 s=READY pgs=1991 cs=0 l=1 rev1=1 rx=0x7efdfc006650 tx=0x7efdfc009a00).stop 2021-02-10T11:30:29.451 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:29.434 7efe0dba7700 1 -- 172.21.2.102:0/3500145480 shutdown_connections 2021-02-10T11:30:29.451 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:29.434 7efe0dba7700 1 --2- 172.21.2.102:0/3500145480 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efe081b0af0 0x7efe08108950 unknown :-1 s=CLOSED pgs=1991 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:29.451 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:29.434 7efe0dba7700 1 --2- 172.21.2.102:0/3500145480 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7efe080d1880 0x7efe081b04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:29.451 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:29.434 7efe0dba7700 1 -- 172.21.2.102:0/3500145480 >> 172.21.2.102:0/3500145480 conn(0x7efe080ce430 msgr2=0x7efe080d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:29.451 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:29.434 7efe0dba7700 1 -- 172.21.2.102:0/3500145480 shutdown_connections 2021-02-10T11:30:29.451 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:29.434 7efe0dba7700 1 -- 172.21.2.102:0/3500145480 wait complete. 2021-02-10T11:30:29.453 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:29.449 7efe0dba7700 1 Processor -- start 2021-02-10T11:30:29.453 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:29.449 7efe0dba7700 1 -- start start 2021-02-10T11:30:29.453 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:29.449 7efe0dba7700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7efe0819e5b0 0x7efe081afc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:29.453 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:29.449 7efe0dba7700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7efe0819e9b0 0x7efe081b0dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:29.453 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:29.449 7efe0dba7700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efe081b0250 0x7efe081b13e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:29.453 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:29.449 7efe0dba7700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe08195bd0 con 0x7efe081b0250 2021-02-10T11:30:29.453 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:29.449 7efe0dba7700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe081b0840 con 0x7efe0819e5b0 2021-02-10T11:30:29.453 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:29.449 7efe0dba7700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe0819eed0 con 0x7efe0819e9b0 2021-02-10T11:30:29.453 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:29.449 7efe07fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efe081b0250 0x7efe081b13e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:29.453 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:29.449 7efe07fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efe081b0250 0x7efe081b13e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:32838/0 (socket says 172.21.2.102:32838) 2021-02-10T11:30:29.453 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:29.449 7efe07fff700 1 -- 172.21.2.102:0/1885647981 learned_addr learned my addr 172.21.2.102:0/1885647981 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:30:29.454 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:29.450 7efe077fe700 1 --2- 172.21.2.102:0/1885647981 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7efe0819e5b0 0x7efe081afc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:29.454 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:29.450 7efe06ffd700 1 --2- 172.21.2.102:0/1885647981 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7efe0819e9b0 0x7efe081b0dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:29.454 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:29.450 7efe07fff700 1 -- 172.21.2.102:0/1885647981 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7efe0819e9b0 msgr2=0x7efe081b0dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:29.454 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:29.450 7efe07fff700 1 --2- 172.21.2.102:0/1885647981 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7efe0819e9b0 0x7efe081b0dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:29.454 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:29.450 7efe07fff700 1 -- 172.21.2.102:0/1885647981 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7efe0819e5b0 msgr2=0x7efe081afc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:29.454 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:29.450 7efe07fff700 1 --2- 172.21.2.102:0/1885647981 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7efe0819e5b0 0x7efe081afc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:29.454 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:29.450 7efe07fff700 1 -- 172.21.2.102:0/1885647981 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe0819f0b0 con 0x7efe081b0250 2021-02-10T11:30:29.454 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:29.450 7efe07fff700 1 --2- 172.21.2.102:0/1885647981 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efe081b0250 0x7efe081b13e0 secure :-1 s=READY pgs=1992 cs=0 l=1 rev1=1 rx=0x7efe0000a940 tx=0x7efe0000ace0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:29.454 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:29.450 7efdeffff700 1 -- 172.21.2.102:0/1885647981 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efe00001270 con 0x7efe081b0250 2021-02-10T11:30:29.454 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:29.450 7efdeffff700 1 -- 172.21.2.102:0/1885647981 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe000070b0 con 0x7efe081b0250 2021-02-10T11:30:29.454 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:29.450 7efe0dba7700 1 -- 172.21.2.102:0/1885647981 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe0819f0b0 con 0x7efe081b0250 2021-02-10T11:30:29.454 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:29.450 7efe0dba7700 1 -- 172.21.2.102:0/1885647981 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe081b1cf0 con 0x7efe081b0250 2021-02-10T11:30:29.455 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:29.450 7efdeffff700 1 -- 172.21.2.102:0/1885647981 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efe00007540 con 0x7efe081b0250 2021-02-10T11:30:29.455 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:29.451 7efdeffff700 1 -- 172.21.2.102:0/1885647981 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7efe00007290 con 0x7efe081b0250 2021-02-10T11:30:29.455 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:29.451 7efdeffff700 1 --2- 172.21.2.102:0/1885647981 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7efde8018730 0x7efde801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:29.455 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:29.451 7efdeffff700 1 -- 172.21.2.102:0/1885647981 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5573+0+0 (secure 0 0 0) 0x7efe00032b20 con 0x7efe081b0250 2021-02-10T11:30:29.455 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:29.451 7efe077fe700 1 --2- 172.21.2.102:0/1885647981 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7efde8018730 0x7efde801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:29.455 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:29.451 7efe077fe700 1 --2- 172.21.2.102:0/1885647981 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7efde8018730 0x7efde801ac10 crc :-1 s=READY pgs=889 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:29.482 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:29.482 7efe0dba7700 1 -- 172.21.2.102:0/1885647981 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe081b0840 con 0x7efe081b0250 2021-02-10T11:30:29.485 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:29.484 7efdeffff700 1 -- 172.21.2.102:0/1885647981 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7efe00001270 con 0x7efe081b0250 2021-02-10T11:30:29.609 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:29.608 7efe0dba7700 1 -- 172.21.2.102:0/1885647981 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000592823", "target": ["mgr", ""], "group_name": "group_0000000000973488"}) v1 -- 0x7efe081b1cf0 con 0x7efde8018730 2021-02-10T11:30:29.634 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:29.633 7efdeffff700 1 -- 172.21.2.102:0/1885647981 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7efdf8006e30 con 0x7efde8018730 2021-02-10T11:30:29.642 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:29.641 7efe0dba7700 1 -- 172.21.2.102:0/1885647981 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7efde8018730 msgr2=0x7efde801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:29.642 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:29.641 7efe0dba7700 1 --2- 172.21.2.102:0/1885647981 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7efde8018730 0x7efde801ac10 crc :-1 s=READY pgs=889 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:29.643 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:29.641 7efe0dba7700 1 -- 172.21.2.102:0/1885647981 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efe081b0250 msgr2=0x7efe081b13e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:29.643 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:29.641 7efe0dba7700 1 --2- 172.21.2.102:0/1885647981 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efe081b0250 0x7efe081b13e0 secure :-1 s=READY pgs=1992 cs=0 l=1 rev1=1 rx=0x7efe0000a940 tx=0x7efe0000ace0).stop 2021-02-10T11:30:29.643 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:29.641 7efe0dba7700 1 -- 172.21.2.102:0/1885647981 shutdown_connections 2021-02-10T11:30:29.643 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:29.641 7efe0dba7700 1 --2- 172.21.2.102:0/1885647981 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efe081b0250 0x7efe081b13e0 unknown :-1 s=CLOSED pgs=1992 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:29.643 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:29.641 7efe0dba7700 1 --2- 172.21.2.102:0/1885647981 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7efe0819e9b0 0x7efe081b0dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:29.643 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:29.641 7efe0dba7700 1 --2- 172.21.2.102:0/1885647981 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7efde8018730 0x7efde801ac10 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:29.643 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:29.641 7efe0dba7700 1 --2- 172.21.2.102:0/1885647981 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7efe0819e5b0 0x7efe081afc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:29.643 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:29.641 7efe0dba7700 1 -- 172.21.2.102:0/1885647981 >> 172.21.2.102:0/1885647981 conn(0x7efe0819f940 msgr2=0x7efe0818c290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:29.643 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:29.642 7efe0dba7700 1 -- 172.21.2.102:0/1885647981 shutdown_connections 2021-02-10T11:30:29.643 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:29.642 7efe0dba7700 1 -- 172.21.2.102:0/1885647981 wait complete. 2021-02-10T11:30:29.651 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume authorize cephfs subvolume_0000000000592823 guest1 --group_name group_0000000000973488 --tenant_id tenant1 2021-02-10T11:30:30.093 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:30.075 7f64830b7700 1 Processor -- start 2021-02-10T11:30:30.094 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:30.076 7f64830b7700 1 -- start start 2021-02-10T11:30:30.094 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:30.076 7f64830b7700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f647c0e53a0 0x7f647c1b0550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:30.094 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:30.076 7f64830b7700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f647c1b0b70 0x7f647c1089d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:30.094 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:30.076 7f64830b7700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f647c109ac0 con 0x7f647c108ff0 2021-02-10T11:30:30.094 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:30.076 7f64830b7700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f647c0e5f80 con 0x7f647c1b0b70 2021-02-10T11:30:30.094 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:30.076 7f64830b7700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f647c0e57a0 con 0x7f647c0e53a0 2021-02-10T11:30:30.094 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:30.076 7f64816d4700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f647c108ff0 0x7f647c10a040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:53306/0 (socket says 172.21.2.102:53306) 2021-02-10T11:30:30.094 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:30.076 7f64816d4700 1 -- 172.21.2.102:0/2494380026 learned_addr learned my addr 172.21.2.102:0/2494380026 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:30:30.094 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:30.077 7f647b7fe700 1 -- 172.21.2.102:0/2494380026 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1677548025 0 0) 0x7f646c001180 con 0x7f647c108ff0 2021-02-10T11:30:30.094 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:30.077 7f647b7fe700 1 -- 172.21.2.102:0/2494380026 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6468002fe0 con 0x7f647c108ff0 2021-02-10T11:30:30.095 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:30.077 7f6480ed3700 1 --2- 172.21.2.102:0/2494380026 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f647c0e53a0 0x7f647c1b0550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:30.095 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:30.077 7f647bfff700 1 --2- 172.21.2.102:0/2494380026 >> v2:172.21.2.113:3300/0 conn(0x7f647c1b0b70 0x7f647c1089d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:30.095 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:30.077 7f647b7fe700 1 -- 172.21.2.102:0/2494380026 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 465886474 0 0) 0x7f646c0010e0 con 0x7f647c108ff0 2021-02-10T11:30:30.095 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:30.077 7f647b7fe700 1 -- 172.21.2.102:0/2494380026 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f647c0e53a0 msgr2=0x7f647c1b0550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:30.095 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:30.077 7f647b7fe700 1 --2- 172.21.2.102:0/2494380026 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f647c0e53a0 0x7f647c1b0550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:30.095 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:30.077 7f647b7fe700 1 -- 172.21.2.102:0/2494380026 >> v2:172.21.2.113:3300/0 conn(0x7f647c1b0b70 msgr2=0x7f647c1089d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:30.095 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:30.077 7f647b7fe700 1 --2- 172.21.2.102:0/2494380026 >> v2:172.21.2.113:3300/0 conn(0x7f647c1b0b70 0x7f647c1089d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:30.095 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:30.077 7f647b7fe700 1 -- 172.21.2.102:0/2494380026 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f647c088070 con 0x7f647c108ff0 2021-02-10T11:30:30.095 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:30.077 7f647b7fe700 1 -- 172.21.2.102:0/2494380026 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7f646c001600 con 0x7f647c108ff0 2021-02-10T11:30:30.095 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:30.077 7f647b7fe700 1 -- 172.21.2.102:0/2494380026 >> v1:172.21.2.102:6789/0 conn(0x7f647c108ff0 legacy=0x7f647c10a040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:30.095 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:30.077 7f647b7fe700 1 --2- 172.21.2.102:0/2494380026 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6468003a60 0x7f6468005f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:30.095 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:30.077 7f647b7fe700 1 --2- 172.21.2.102:0/2494380026 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f646804a7d0 0x7f646804cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:30.096 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:30.077 7f647b7fe700 1 --2- 172.21.2.102:0/2494380026 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f646804d590 0x7f646804fa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:30.096 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:30.077 7f647b7fe700 1 -- 172.21.2.102:0/2494380026 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64680502d0 con 0x7f6468003a60 2021-02-10T11:30:30.096 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:30.077 7f647b7fe700 1 -- 172.21.2.102:0/2494380026 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64680504f0 con 0x7f646804a7d0 2021-02-10T11:30:30.096 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:30.077 7f647b7fe700 1 -- 172.21.2.102:0/2494380026 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6468050710 con 0x7f646804d590 2021-02-10T11:30:30.096 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:30.077 7f6480ed3700 1 --2- 172.21.2.102:0/2494380026 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6468003a60 0x7f6468005f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:30.096 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:30.077 7f6480ed3700 1 -- 172.21.2.102:0/2494380026 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f646804d590 msgr2=0x7f646804fa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:30.096 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:30.077 7f6480ed3700 1 --2- 172.21.2.102:0/2494380026 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f646804d590 0x7f646804fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:30.096 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:30.077 7f6480ed3700 1 -- 172.21.2.102:0/2494380026 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f646804a7d0 msgr2=0x7f646804cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:30.096 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:30.077 7f6480ed3700 1 --2- 172.21.2.102:0/2494380026 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f646804a7d0 0x7f646804cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:30.096 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:30.077 7f6480ed3700 1 -- 172.21.2.102:0/2494380026 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64680509b0 con 0x7f6468003a60 2021-02-10T11:30:30.096 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:30.077 7f6480ed3700 1 --2- 172.21.2.102:0/2494380026 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6468003a60 0x7f6468005f60 secure :-1 s=READY pgs=1994 cs=0 l=1 rev1=1 rx=0x7f6464006ac0 tx=0x7f6464009ab0).ready entity=mon.0 client_cookie=5cd8c69bc44ca2ee server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:30.096 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:30.077 7f647b7fe700 1 -- 172.21.2.102:0/2494380026 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f646400c0c0 con 0x7f6468003a60 2021-02-10T11:30:30.097 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:30.078 7f647b7fe700 1 -- 172.21.2.102:0/2494380026 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f646400c380 con 0x7f6468003a60 2021-02-10T11:30:30.097 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:30.078 7f647b7fe700 1 -- 172.21.2.102:0/2494380026 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f646400c7e0 con 0x7f6468003a60 2021-02-10T11:30:30.097 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:30.078 7f64830b7700 1 -- 172.21.2.102:0/2494380026 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6468003a60 msgr2=0x7f6468005f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:30.097 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:30.078 7f64830b7700 1 --2- 172.21.2.102:0/2494380026 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6468003a60 0x7f6468005f60 secure :-1 s=READY pgs=1994 cs=0 l=1 rev1=1 rx=0x7f6464006ac0 tx=0x7f6464009ab0).stop 2021-02-10T11:30:30.097 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:30.078 7f64816d4700 1 -- 172.21.2.102:0/2494380026 reap_dead start 2021-02-10T11:30:30.097 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:30.078 7f64830b7700 1 -- 172.21.2.102:0/2494380026 shutdown_connections 2021-02-10T11:30:30.097 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:30.078 7f64830b7700 1 -- 172.21.2.102:0/2494380026 >> 172.21.2.102:0/2494380026 conn(0x7f647c0ce4b0 msgr2=0x7f647c0e45d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:30.097 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:30.078 7f64830b7700 1 -- 172.21.2.102:0/2494380026 shutdown_connections 2021-02-10T11:30:30.097 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:30.078 7f64830b7700 1 -- 172.21.2.102:0/2494380026 wait complete. 2021-02-10T11:30:30.097 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:30.093 7f64830b7700 1 Processor -- start 2021-02-10T11:30:30.097 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:30.093 7f64830b7700 1 -- start start 2021-02-10T11:30:30.099 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:30.093 7f64830b7700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f647c0b6710 0x7f647c0b73c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:30.099 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:30.093 7f64830b7700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f647c0b79e0 0x7f647c0b7de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:30.099 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:30.093 7f64830b7700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f647c0b8400 0x7f647c09c990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:30.099 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:30.093 7f64830b7700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f647c0b8880 con 0x7f647c0b79e0 2021-02-10T11:30:30.099 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:30.093 7f64830b7700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f647c0b6b10 con 0x7f647c0b6710 2021-02-10T11:30:30.099 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:30.093 7f64830b7700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f647c106230 con 0x7f647c0b8400 2021-02-10T11:30:30.099 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:30.094 7f647bfff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f647c0b79e0 0x7f647c0b7de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:30.099 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:30.094 7f647bfff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f647c0b79e0 0x7f647c0b7de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:32860/0 (socket says 172.21.2.102:32860) 2021-02-10T11:30:30.099 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:30.094 7f647bfff700 1 -- 172.21.2.102:0/3578850521 learned_addr learned my addr 172.21.2.102:0/3578850521 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:30:30.100 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:30.094 7f6480ed3700 1 --2- 172.21.2.102:0/3578850521 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f647c0b6710 0x7f647c0b73c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:30.100 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:30.094 7f64816d4700 1 --2- 172.21.2.102:0/3578850521 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f647c0b8400 0x7f647c09c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:30.100 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:30.094 7f647bfff700 1 -- 172.21.2.102:0/3578850521 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f647c0b8400 msgr2=0x7f647c09c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:30.100 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:30.094 7f647bfff700 1 --2- 172.21.2.102:0/3578850521 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f647c0b8400 0x7f647c09c990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:30.100 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:30.094 7f647bfff700 1 -- 172.21.2.102:0/3578850521 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f647c0b6710 msgr2=0x7f647c0b73c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:30.100 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:30.094 7f647bfff700 1 --2- 172.21.2.102:0/3578850521 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f647c0b6710 0x7f647c0b73c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:30.100 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:30.094 7f647bfff700 1 -- 172.21.2.102:0/3578850521 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f647c09cfb0 con 0x7f647c0b79e0 2021-02-10T11:30:30.100 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:30.094 7f647bfff700 1 --2- 172.21.2.102:0/3578850521 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f647c0b79e0 0x7f647c0b7de0 secure :-1 s=READY pgs=1995 cs=0 l=1 rev1=1 rx=0x7f647400b970 tx=0x7f647400bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:30.100 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:30.095 7f64797fa700 1 -- 172.21.2.102:0/3578850521 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6474000d30 con 0x7f647c0b79e0 2021-02-10T11:30:30.100 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:30.095 7f64797fa700 1 -- 172.21.2.102:0/3578850521 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6474006740 con 0x7f647c0b79e0 2021-02-10T11:30:30.100 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:30.095 7f64797fa700 1 -- 172.21.2.102:0/3578850521 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f64740064b0 con 0x7f647c0b79e0 2021-02-10T11:30:30.100 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:30.095 7f64830b7700 1 -- 172.21.2.102:0/3578850521 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f647c09cfb0 con 0x7f647c0b79e0 2021-02-10T11:30:30.101 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:30.095 7f64830b7700 1 -- 172.21.2.102:0/3578850521 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f647c09d540 con 0x7f647c0b79e0 2021-02-10T11:30:30.101 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:30.095 7f64797fa700 1 -- 172.21.2.102:0/3578850521 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f6474005660 con 0x7f647c0b79e0 2021-02-10T11:30:30.101 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:30.095 7f64797fa700 1 --2- 172.21.2.102:0/3578850521 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f64600186c0 0x7f646001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:30.101 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:30.095 7f64797fa700 1 -- 172.21.2.102:0/3578850521 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f64740328f0 con 0x7f647c0b79e0 2021-02-10T11:30:30.101 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:30.095 7f6480ed3700 1 --2- 172.21.2.102:0/3578850521 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f64600186c0 0x7f646001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:30.101 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:30.096 7f6480ed3700 1 --2- 172.21.2.102:0/3578850521 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f64600186c0 0x7f646001abc0 crc :-1 s=READY pgs=890 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:30.126 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:30.125 7f64830b7700 1 -- 172.21.2.102:0/3578850521 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f647c103750 con 0x7f647c0b79e0 2021-02-10T11:30:30.128 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:30.128 7f64797fa700 1 -- 172.21.2.102:0/3578850521 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f6474021f90 con 0x7f647c0b79e0 2021-02-10T11:30:30.254 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:30.253 7f64830b7700 1 -- 172.21.2.102:0/3578850521 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume authorize", "vol_name": "cephfs", "sub_name": "subvolume_0000000000592823", "target": ["mgr", ""], "tenant_id": "tenant1", "auth_id": "guest1", "group_name": "group_0000000000973488"}) v1 -- 0x7f647c106230 con 0x7f64600186c0 2021-02-10T11:30:30.259 INFO:tasks.ceph.mgr.y.gibba002.stderr:Creating meta for ID guest1 with tenant tenant1 2021-02-10T11:30:30.285 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:30.285 7f64797fa700 1 -- 172.21.2.102:0/3578850521 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+41 (crc 0 0 0) 0x7f6464006400 con 0x7f64600186c0 2021-02-10T11:30:30.286 INFO:teuthology.orchestra.run.gibba002.stdout:AQDWwyNgDrQfEBAA0jPM0K1bUzptthOlYECpWA== 2021-02-10T11:30:30.287 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:30.286 7f64830b7700 1 -- 172.21.2.102:0/3578850521 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f64600186c0 msgr2=0x7f646001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:30.287 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:30.286 7f64830b7700 1 --2- 172.21.2.102:0/3578850521 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f64600186c0 0x7f646001abc0 crc :-1 s=READY pgs=890 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:30.287 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:30.286 7f64830b7700 1 -- 172.21.2.102:0/3578850521 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f647c0b79e0 msgr2=0x7f647c0b7de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:30.287 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:30.286 7f64830b7700 1 --2- 172.21.2.102:0/3578850521 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f647c0b79e0 0x7f647c0b7de0 secure :-1 s=READY pgs=1995 cs=0 l=1 rev1=1 rx=0x7f647400b970 tx=0x7f647400bd10).stop 2021-02-10T11:30:30.287 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:30.286 7f64830b7700 1 -- 172.21.2.102:0/3578850521 shutdown_connections 2021-02-10T11:30:30.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:30.286 7f64830b7700 1 --2- 172.21.2.102:0/3578850521 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f647c0b8400 0x7f647c09c990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:30.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:30.286 7f64830b7700 1 --2- 172.21.2.102:0/3578850521 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f647c0b79e0 0x7f647c0b7de0 unknown :-1 s=CLOSED pgs=1995 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:30.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:30.286 7f64830b7700 1 --2- 172.21.2.102:0/3578850521 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f64600186c0 0x7f646001abc0 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:30.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:30.286 7f64830b7700 1 --2- 172.21.2.102:0/3578850521 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f647c0b6710 0x7f647c0b73c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:30.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:30.286 7f64830b7700 1 -- 172.21.2.102:0/3578850521 >> 172.21.2.102:0/3578850521 conn(0x7f647c09e480 msgr2=0x7f647c107f30 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:30.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:30.287 7f64830b7700 1 -- 172.21.2.102:0/3578850521 shutdown_connections 2021-02-10T11:30:30.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:30.287 7f64830b7700 1 -- 172.21.2.102:0/3578850521 wait complete. 2021-02-10T11:30:30.295 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000592823 --group_name group_0000000000973488 2021-02-10T11:30:30.728 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:30.710 7fadc5c19700 1 Processor -- start 2021-02-10T11:30:30.728 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:30.710 7fadc5c19700 1 -- start start 2021-02-10T11:30:30.728 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:30.710 7fadc5c19700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fadc00d1880 0x7fadc01b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:30.728 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:30.711 7fadc5c19700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fadc01b0af0 0x7fadc0108950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:30.729 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:30.711 7fadc5c19700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fadc00fe1d0 con 0x7fadc00d1c80 2021-02-10T11:30:30.729 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:30.711 7fadc5c19700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fadc0109a70 con 0x7fadc00d1880 2021-02-10T11:30:30.729 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:30.711 7fadc5c19700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fadc0109760 con 0x7fadc01b0af0 2021-02-10T11:30:30.729 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:30.711 7fadbffff700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fadc00d1c80 0x7fadc0108f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:53324/0 (socket says 172.21.2.102:53324) 2021-02-10T11:30:30.729 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:30.711 7fadbffff700 1 -- 172.21.2.102:0/1569720995 learned_addr learned my addr 172.21.2.102:0/1569720995 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:30:30.729 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:30.711 7fadbf7fe700 1 --2- 172.21.2.102:0/1569720995 >> v2:172.21.2.102:3300/0 conn(0x7fadc00d1880 0x7fadc01b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:30.730 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:30.711 7fadbe7fc700 1 -- 172.21.2.102:0/1569720995 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3031251051 0 0) 0x7fadb8001500 con 0x7fadc00d1c80 2021-02-10T11:30:30.730 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:30.711 7fadbe7fc700 1 -- 172.21.2.102:0/1569720995 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fadac002fe0 con 0x7fadc00d1c80 2021-02-10T11:30:30.730 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:30.711 7fadbeffd700 1 --2- 172.21.2.102:0/1569720995 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fadc01b0af0 0x7fadc0108950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:30.730 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:30.711 7fadbf7fe700 1 -- 172.21.2.102:0/1569720995 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fadc01b0af0 msgr2=0x7fadc0108950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:30.730 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:30.711 7fadbf7fe700 1 --2- 172.21.2.102:0/1569720995 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fadc01b0af0 0x7fadc0108950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:30.731 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:30.711 7fadbf7fe700 1 -- 172.21.2.102:0/1569720995 >> v1:172.21.2.102:6789/0 conn(0x7fadc00d1c80 legacy=0x7fadc0108f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:30.731 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:30.711 7fadbf7fe700 1 -- 172.21.2.102:0/1569720995 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadc0088070 con 0x7fadc00d1880 2021-02-10T11:30:30.731 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:30.711 7fadbeffd700 1 --2- 172.21.2.102:0/1569720995 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fadc01b0af0 0x7fadc0108950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:30:30.731 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:30.711 7fadbf7fe700 1 --2- 172.21.2.102:0/1569720995 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fadc00d1880 0x7fadc01b04d0 secure :-1 s=READY pgs=1997 cs=0 l=1 rev1=1 rx=0x7fadb0006650 tx=0x7fadb0009a80).ready entity=mon.0 client_cookie=dd3057c52ce6e2f7 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:30.731 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:30.712 7fadbe7fc700 1 -- 172.21.2.102:0/1569720995 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fadb000c0c0 con 0x7fadc00d1880 2021-02-10T11:30:30.731 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:30.712 7fadbe7fc700 1 -- 172.21.2.102:0/1569720995 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadb000c340 con 0x7fadc00d1880 2021-02-10T11:30:30.732 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:30.712 7fadbe7fc700 1 -- 172.21.2.102:0/1569720995 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fadb000c860 con 0x7fadc00d1880 2021-02-10T11:30:30.732 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:30.712 7fadc5c19700 1 -- 172.21.2.102:0/1569720995 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fadc00d1880 msgr2=0x7fadc01b04d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:30.732 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:30.712 7fadc5c19700 1 --2- 172.21.2.102:0/1569720995 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fadc00d1880 0x7fadc01b04d0 secure :-1 s=READY pgs=1997 cs=0 l=1 rev1=1 rx=0x7fadb0006650 tx=0x7fadb0009a80).stop 2021-02-10T11:30:30.732 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:30.712 7fadc5c19700 1 -- 172.21.2.102:0/1569720995 shutdown_connections 2021-02-10T11:30:30.733 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:30.712 7fadc5c19700 1 --2- 172.21.2.102:0/1569720995 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fadc01b0af0 0x7fadc0108950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:30.733 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:30.712 7fadc5c19700 1 --2- 172.21.2.102:0/1569720995 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fadc00d1880 0x7fadc01b04d0 unknown :-1 s=CLOSED pgs=1997 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:30.733 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:30.712 7fadc5c19700 1 -- 172.21.2.102:0/1569720995 >> 172.21.2.102:0/1569720995 conn(0x7fadc00ce430 msgr2=0x7fadc00d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:30.733 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:30.712 7fadc5c19700 1 -- 172.21.2.102:0/1569720995 shutdown_connections 2021-02-10T11:30:30.733 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:30.712 7fadc5c19700 1 -- 172.21.2.102:0/1569720995 wait complete. 2021-02-10T11:30:30.733 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:30.727 7fadc5c19700 1 Processor -- start 2021-02-10T11:30:30.734 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:30.727 7fadc5c19700 1 -- start start 2021-02-10T11:30:30.736 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:30.727 7fadc5c19700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fadc00ad520 0x7fadc00ad920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:30.737 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:30.727 7fadc5c19700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fadc00adf40 0x7fadc0097c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:30.737 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:30.727 7fadc5c19700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fadc0098250 0x7fadc0098650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:30.737 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:30.727 7fadc5c19700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadc0098c70 con 0x7fadc0098250 2021-02-10T11:30:30.737 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:30.727 7fadc5c19700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadc00910f0 con 0x7fadc00ad520 2021-02-10T11:30:30.737 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:30.727 7fadc5c19700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fadc0091310 con 0x7fadc00adf40 2021-02-10T11:30:30.737 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:30.728 7fadbeffd700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fadc00adf40 0x7fadc0097c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:30.737 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:30.728 7fadbeffd700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fadc00adf40 0x7fadc0097c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.113:3301/0 says I am v2:172.21.2.102:38410/0 (socket says 172.21.2.102:38410) 2021-02-10T11:30:30.738 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:30.728 7fadbffff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fadc0098250 0x7fadc0098650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:30.738 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:30.728 7fadbeffd700 1 -- 172.21.2.102:0/1298816658 learned_addr learned my addr 172.21.2.102:0/1298816658 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:30:30.738 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:30.728 7fadbf7fe700 1 --2- 172.21.2.102:0/1298816658 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fadc00ad520 0x7fadc00ad920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:30.738 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:30.728 7fadbeffd700 1 -- 172.21.2.102:0/1298816658 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fadc00ad520 msgr2=0x7fadc00ad920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:30.738 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:30.728 7fadbeffd700 1 --2- 172.21.2.102:0/1298816658 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fadc00ad520 0x7fadc00ad920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:30.738 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:30.728 7fadbeffd700 1 -- 172.21.2.102:0/1298816658 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fadc0098250 msgr2=0x7fadc0098650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:30.738 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:30.728 7fadbeffd700 1 --2- 172.21.2.102:0/1298816658 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fadc0098250 0x7fadc0098650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:30.738 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:30.728 7fadbeffd700 1 -- 172.21.2.102:0/1298816658 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadc019a3b0 con 0x7fadc00adf40 2021-02-10T11:30:30.739 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:30.728 7fadbffff700 1 --2- 172.21.2.102:0/1298816658 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fadc0098250 0x7fadc0098650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:30:30.739 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:30.728 7fadbf7fe700 1 --2- 172.21.2.102:0/1298816658 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fadc00ad520 0x7fadc00ad920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:30:30.739 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:30.728 7fadbeffd700 1 --2- 172.21.2.102:0/1298816658 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fadc00adf40 0x7fadc0097c30 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7fada8006f20 tx=0x7fada800b990).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:30.739 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:30.729 7fadb7fff700 1 -- 172.21.2.102:0/1298816658 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fada80009b0 con 0x7fadc00adf40 2021-02-10T11:30:30.739 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:30.729 7fadb7fff700 1 -- 172.21.2.102:0/1298816658 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fada800bd80 con 0x7fadc00adf40 2021-02-10T11:30:30.739 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:30.729 7fadc5c19700 1 -- 172.21.2.102:0/1298816658 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fadc019a3b0 con 0x7fadc00adf40 2021-02-10T11:30:30.739 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:30.729 7fadc5c19700 1 -- 172.21.2.102:0/1298816658 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fadc019a940 con 0x7fadc00adf40 2021-02-10T11:30:30.739 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:30.729 7fadb7fff700 1 -- 172.21.2.102:0/1298816658 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fada80009b0 con 0x7fadc00adf40 2021-02-10T11:30:30.739 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:30.729 7fadb7fff700 1 -- 172.21.2.102:0/1298816658 <== mon.2 v2:172.21.2.113:3301/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fada8002240 con 0x7fadc00adf40 2021-02-10T11:30:30.740 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:30.729 7fadb7fff700 1 --2- 172.21.2.102:0/1298816658 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fada40186c0 0x7fada401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:30.740 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:30.729 7fadb7fff700 1 -- 172.21.2.102:0/1298816658 <== mon.2 v2:172.21.2.113:3301/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fada80327b0 con 0x7fadc00adf40 2021-02-10T11:30:30.740 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:30.729 7fadbf7fe700 1 --2- 172.21.2.102:0/1298816658 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fada40186c0 0x7fada401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:30.740 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:30.730 7fadbf7fe700 1 --2- 172.21.2.102:0/1298816658 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fada40186c0 0x7fada401abc0 crc :-1 s=READY pgs=891 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:30.761 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:30.761 7fadc5c19700 1 -- 172.21.2.102:0/1298816658 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fadc0082a70 con 0x7fadc00adf40 2021-02-10T11:30:30.765 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:30.764 7fadb7fff700 1 -- 172.21.2.102:0/1298816658 <== mon.2 v2:172.21.2.113:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fada8002030 con 0x7fadc00adf40 2021-02-10T11:30:30.897 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:30.896 7fadc5c19700 1 -- 172.21.2.102:0/1298816658 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000592823", "target": ["mgr", ""], "group_name": "group_0000000000973488"}) v1 -- 0x7fadc00ae340 con 0x7fada40186c0 2021-02-10T11:30:30.900 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:30.899 7fadb7fff700 1 -- 172.21.2.102:0/1298816658 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+96 (crc 0 0 0) 0x7fadb00090f0 con 0x7fada40186c0 2021-02-10T11:30:30.903 INFO:teuthology.orchestra.run.gibba002.stdout:/volumes/group_0000000000973488/subvolume_0000000000592823/2b1cc9aa-7744-4e93-96bb-90fae43a8dd2 2021-02-10T11:30:30.904 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:30.904 7fadc5c19700 1 -- 172.21.2.102:0/1298816658 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fada40186c0 msgr2=0x7fada401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:30.905 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:30.904 7fadc5c19700 1 --2- 172.21.2.102:0/1298816658 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fada40186c0 0x7fada401abc0 crc :-1 s=READY pgs=891 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:30.905 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:30.904 7fadc5c19700 1 -- 172.21.2.102:0/1298816658 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fadc00adf40 msgr2=0x7fadc0097c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:30.905 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:30.904 7fadc5c19700 1 --2- 172.21.2.102:0/1298816658 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fadc00adf40 0x7fadc0097c30 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7fada8006f20 tx=0x7fada800b990).stop 2021-02-10T11:30:30.905 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:30.904 7fadc5c19700 1 -- 172.21.2.102:0/1298816658 shutdown_connections 2021-02-10T11:30:30.905 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:30.904 7fadc5c19700 1 --2- 172.21.2.102:0/1298816658 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fadc0098250 0x7fadc0098650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:30.906 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:30.904 7fadc5c19700 1 --2- 172.21.2.102:0/1298816658 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fadc00adf40 0x7fadc0097c30 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:30.906 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:30.904 7fadc5c19700 1 --2- 172.21.2.102:0/1298816658 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fada40186c0 0x7fada401abc0 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:30.906 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:30.904 7fadc5c19700 1 --2- 172.21.2.102:0/1298816658 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fadc00ad520 0x7fadc00ad920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:30.906 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:30.904 7fadc5c19700 1 -- 172.21.2.102:0/1298816658 >> 172.21.2.102:0/1298816658 conn(0x7fadc018e810 msgr2=0x7fadc018c290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:30.906 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:30.905 7fadc5c19700 1 -- 172.21.2.102:0/1298816658 shutdown_connections 2021-02-10T11:30:30.906 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:30.905 7fadc5c19700 1 -- 172.21.2.102:0/1298816658 wait complete. 2021-02-10T11:30:30.913 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.guest1 mds 'allow rw path=/volumes/group_0000000000973488, allow rw path=/volumes/group_0000000000973488/subvolume_0000000000592823/2b1cc9aa-7744-4e93-96bb-90fae43a8dd2' osd 'allow rw pool=cephfs_data' mon 'allow r' mgr 'allow *' 2021-02-10T11:30:31.355 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:31.336 7f932df59700 1 Processor -- start 2021-02-10T11:30:31.355 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:31.336 7f932df59700 1 -- start start 2021-02-10T11:30:31.355 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:31.337 7f932df59700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f932810e130 0x7f932810fce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:31.355 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:31.337 7f932df59700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f932810e750 con 0x7f93280e5350 2021-02-10T11:30:31.355 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:31.337 7f932df59700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93281b0be0 con 0x7f93280e5750 2021-02-10T11:30:31.355 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:31.337 7f932df59700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93281b0e00 con 0x7f932810e130 2021-02-10T11:30:31.355 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:31.337 7f93277fe700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f93280e5350 0x7f932810d150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:53336/0 (socket says 172.21.2.102:53336) 2021-02-10T11:30:31.355 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:31.337 7f93277fe700 1 -- 172.21.2.102:0/2069850612 learned_addr learned my addr 172.21.2.102:0/2069850612 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:30:31.355 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:31.337 7f93267fc700 1 -- 172.21.2.102:0/2069850612 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 992995109 0 0) 0x7f93100010b0 con 0x7f93280e5350 2021-02-10T11:30:31.356 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:31.337 7f93267fc700 1 -- 172.21.2.102:0/2069850612 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9314002fe0 con 0x7f93280e5350 2021-02-10T11:30:31.356 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:31.338 7f9327fff700 1 --2- 172.21.2.102:0/2069850612 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f932810e130 0x7f932810fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:31.356 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:31.338 7f93267fc700 1 -- 172.21.2.102:0/2069850612 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2599899209 0 0) 0x7f93100010b0 con 0x7f93280e5350 2021-02-10T11:30:31.356 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:31.338 7f93267fc700 1 -- 172.21.2.102:0/2069850612 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f932810e130 msgr2=0x7f932810fce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:31.356 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:31.338 7f93267fc700 1 --2- 172.21.2.102:0/2069850612 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f932810e130 0x7f932810fce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:31.356 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:31.338 7f93267fc700 1 -- 172.21.2.102:0/2069850612 >> v1:172.21.2.113:6789/0 conn(0x7f93280e5750 legacy=0x7f932810d940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:31.356 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:31.338 7f93267fc700 1 -- 172.21.2.102:0/2069850612 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93280e61d0 con 0x7f93280e5350 2021-02-10T11:30:31.356 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:31.338 7f93267fc700 1 -- 172.21.2.102:0/2069850612 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7f93100015b0 con 0x7f93280e5350 2021-02-10T11:30:31.356 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:31.338 7f93267fc700 1 -- 172.21.2.102:0/2069850612 >> v1:172.21.2.102:6789/0 conn(0x7f93280e5350 legacy=0x7f932810d150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:31.356 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:31.338 7f93267fc700 1 --2- 172.21.2.102:0/2069850612 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9314003b70 0x7f9314006000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:31.356 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:31.338 7f93267fc700 1 --2- 172.21.2.102:0/2069850612 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f9314006820 0x7f9314008d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:31.356 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:31.338 7f93267fc700 1 --2- 172.21.2.102:0/2069850612 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f93140095c0 0x7f931400bae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:31.356 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:31.338 7f93267fc700 1 -- 172.21.2.102:0/2069850612 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f931400c2f0 con 0x7f93140095c0 2021-02-10T11:30:31.357 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:31.338 7f93267fc700 1 -- 172.21.2.102:0/2069850612 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f931400c510 con 0x7f9314006820 2021-02-10T11:30:31.357 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:31.338 7f93267fc700 1 -- 172.21.2.102:0/2069850612 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f931400c730 con 0x7f9314003b70 2021-02-10T11:30:31.357 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:31.338 7f9326ffd700 1 --2- 172.21.2.102:0/2069850612 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f9314006820 0x7f9314008d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:31.357 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:31.338 7f93277fe700 1 --2- 172.21.2.102:0/2069850612 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9314003b70 0x7f9314006000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:31.357 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:31.338 7f9327fff700 1 --2- 172.21.2.102:0/2069850612 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f93140095c0 0x7f931400bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:31.357 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:31.338 7f9327fff700 1 -- 172.21.2.102:0/2069850612 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9314003b70 msgr2=0x7f9314006000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:31.357 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:31.338 7f9327fff700 1 --2- 172.21.2.102:0/2069850612 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9314003b70 0x7f9314006000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:31.357 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:31.338 7f9327fff700 1 -- 172.21.2.102:0/2069850612 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f9314006820 msgr2=0x7f9314008d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:31.357 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:31.338 7f9327fff700 1 --2- 172.21.2.102:0/2069850612 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f9314006820 0x7f9314008d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:31.357 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:31.338 7f9327fff700 1 -- 172.21.2.102:0/2069850612 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93140509a0 con 0x7f93140095c0 2021-02-10T11:30:31.357 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:31.338 7f9327fff700 1 --2- 172.21.2.102:0/2069850612 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f93140095c0 0x7f931400bae0 secure :-1 s=READY pgs=1999 cs=0 l=1 rev1=1 rx=0x7f9320006fc0 tx=0x7f9320009e70).ready entity=mon.0 client_cookie=ed015b7c3438ab16 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:31.357 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:31.339 7f93267fc700 1 -- 172.21.2.102:0/2069850612 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9320006a50 con 0x7f93140095c0 2021-02-10T11:30:31.357 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:31.339 7f93267fc700 1 -- 172.21.2.102:0/2069850612 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f932000b300 con 0x7f93140095c0 2021-02-10T11:30:31.358 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:31.339 7f93267fc700 1 -- 172.21.2.102:0/2069850612 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f932000b7a0 con 0x7f93140095c0 2021-02-10T11:30:31.358 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:31.339 7f932df59700 1 -- 172.21.2.102:0/2069850612 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f93140095c0 msgr2=0x7f931400bae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:31.358 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:31.339 7f932df59700 1 --2- 172.21.2.102:0/2069850612 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f93140095c0 0x7f931400bae0 secure :-1 s=READY pgs=1999 cs=0 l=1 rev1=1 rx=0x7f9320006fc0 tx=0x7f9320009e70).stop 2021-02-10T11:30:31.358 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:31.339 7f9327fff700 1 -- 172.21.2.102:0/2069850612 reap_dead start 2021-02-10T11:30:31.358 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:31.340 7f932df59700 1 -- 172.21.2.102:0/2069850612 shutdown_connections 2021-02-10T11:30:31.358 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:31.340 7f932df59700 1 -- 172.21.2.102:0/2069850612 >> 172.21.2.102:0/2069850612 conn(0x7f93280cea00 msgr2=0x7f93280e4840 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:31.358 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:31.340 7f932df59700 1 -- 172.21.2.102:0/2069850612 shutdown_connections 2021-02-10T11:30:31.358 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:31.340 7f932df59700 1 -- 172.21.2.102:0/2069850612 wait complete. 2021-02-10T11:30:31.358 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:31.354 7f932df59700 1 Processor -- start 2021-02-10T11:30:31.360 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:31.354 7f932df59700 1 -- start start 2021-02-10T11:30:31.360 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:31.355 7f932df59700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f93280a3fd0 0x7f9328094fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:31.360 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:31.355 7f932df59700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f93281aaa30 0x7f93281aae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:31.360 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:31.355 7f932df59700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f93280939f0 0x7f9328093df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:31.360 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:31.355 7f932df59700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9328094a40 con 0x7f93280a3fd0 2021-02-10T11:30:31.360 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:31.355 7f932df59700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93280a43d0 con 0x7f93280939f0 2021-02-10T11:30:31.360 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:31.355 7f932df59700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93281ab450 con 0x7f93281aaa30 2021-02-10T11:30:31.360 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:31.355 7f93277fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f93280a3fd0 0x7f9328094fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:31.360 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:31.355 7f93277fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f93280a3fd0 0x7f9328094fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:32892/0 (socket says 172.21.2.102:32892) 2021-02-10T11:30:31.361 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:31.355 7f93277fe700 1 -- 172.21.2.102:0/2666674074 learned_addr learned my addr 172.21.2.102:0/2666674074 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:30:31.361 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:31.355 7f9327fff700 1 --2- 172.21.2.102:0/2666674074 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f93280939f0 0x7f9328093df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:31.361 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:31.355 7f9326ffd700 1 --2- 172.21.2.102:0/2666674074 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f93281aaa30 0x7f93281aae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:31.361 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:31.355 7f93277fe700 1 -- 172.21.2.102:0/2666674074 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f93281aaa30 msgr2=0x7f93281aae30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:31.361 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:31.355 7f93277fe700 1 --2- 172.21.2.102:0/2666674074 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f93281aaa30 0x7f93281aae30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:31.361 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:31.355 7f93277fe700 1 -- 172.21.2.102:0/2666674074 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f93280939f0 msgr2=0x7f9328093df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:31.361 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:31.355 7f93277fe700 1 --2- 172.21.2.102:0/2666674074 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f93280939f0 0x7f9328093df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:31.361 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:31.355 7f93277fe700 1 -- 172.21.2.102:0/2666674074 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f932819a040 con 0x7f93280a3fd0 2021-02-10T11:30:31.361 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:31.355 7f9326ffd700 1 --2- 172.21.2.102:0/2666674074 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f93281aaa30 0x7f93281aae30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:30:31.361 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:31.356 7f93277fe700 1 --2- 172.21.2.102:0/2666674074 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f93280a3fd0 0x7f9328094fc0 secure :-1 s=READY pgs=2000 cs=0 l=1 rev1=1 rx=0x7f9310001660 tx=0x7f931000a930).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:31.361 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:31.356 7f931ffff700 1 -- 172.21.2.102:0/2666674074 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f931000ad20 con 0x7f93280a3fd0 2021-02-10T11:30:31.361 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:31.356 7f931ffff700 1 -- 172.21.2.102:0/2666674074 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93100016a0 con 0x7f93280a3fd0 2021-02-10T11:30:31.361 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:31.356 7f932df59700 1 -- 172.21.2.102:0/2666674074 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f932819a040 con 0x7f93280a3fd0 2021-02-10T11:30:31.362 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:31.356 7f932df59700 1 -- 172.21.2.102:0/2666674074 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f932819a040 con 0x7f93280a3fd0 2021-02-10T11:30:31.362 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:31.356 7f931ffff700 1 -- 172.21.2.102:0/2666674074 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9310004330 con 0x7f93280a3fd0 2021-02-10T11:30:31.362 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:31.357 7f931ffff700 1 -- 172.21.2.102:0/2666674074 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f9310004140 con 0x7f93280a3fd0 2021-02-10T11:30:31.362 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:31.357 7f931ffff700 1 --2- 172.21.2.102:0/2666674074 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f930c018730 0x7f930c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:31.362 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:31.357 7f931ffff700 1 -- 172.21.2.102:0/2666674074 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f93100327d0 con 0x7f93280a3fd0 2021-02-10T11:30:31.362 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:31.357 7f9326ffd700 1 --2- 172.21.2.102:0/2666674074 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f930c018730 0x7f930c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:31.362 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:31.357 7f9326ffd700 1 --2- 172.21.2.102:0/2666674074 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f930c018730 0x7f930c01abc0 crc :-1 s=READY pgs=892 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:31.387 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:31.386 7f932df59700 1 -- 172.21.2.102:0/2666674074 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93281ab450 con 0x7f93280a3fd0 2021-02-10T11:30:31.389 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:31.389 7f931ffff700 1 -- 172.21.2.102:0/2666674074 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f9310001220 con 0x7f93280a3fd0 2021-02-10T11:30:31.521 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:31.521 7f932df59700 1 -- 172.21.2.102:0/2666674074 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.guest1", "caps": ["mds", "allow rw path=/volumes/group_0000000000973488, allow rw path=/volumes/group_0000000000973488/subvolume_0000000000592823/2b1cc9aa-7744-4e93-96bb-90fae43a8dd2", "osd", "allow rw pool=cephfs_data", "mon", "allow r", "mgr", "allow *"]} v 0) v1 -- 0x7f932810e750 con 0x7f93280a3fd0 2021-02-10T11:30:31.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:31.524 7f931ffff700 1 -- 172.21.2.102:0/2666674074 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.guest1", "caps": ["mds", "allow rw path=/volumes/group_0000000000973488, allow rw path=/volumes/group_0000000000973488/subvolume_0000000000592823/2b1cc9aa-7744-4e93-96bb-90fae43a8dd2", "osd", "allow rw pool=cephfs_data", "mon", "allow r", "mgr", "allow *"]}]=0 updated caps for client.guest1 v56) v1 ==== 363+0+0 (secure 0 0 0) 0x7f9310004140 con 0x7f93280a3fd0 2021-02-10T11:30:31.528 INFO:teuthology.orchestra.run.gibba002.stderr:updated caps for client.guest1 2021-02-10T11:30:31.530 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:31.529 7f932df59700 1 -- 172.21.2.102:0/2666674074 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f930c018730 msgr2=0x7f930c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:31.530 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:31.529 7f932df59700 1 --2- 172.21.2.102:0/2666674074 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f930c018730 0x7f930c01abc0 crc :-1 s=READY pgs=892 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:31.530 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:31.529 7f932df59700 1 -- 172.21.2.102:0/2666674074 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f93280a3fd0 msgr2=0x7f9328094fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:31.530 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:31.529 7f932df59700 1 --2- 172.21.2.102:0/2666674074 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f93280a3fd0 0x7f9328094fc0 secure :-1 s=READY pgs=2000 cs=0 l=1 rev1=1 rx=0x7f9310001660 tx=0x7f931000a930).stop 2021-02-10T11:30:31.530 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:31.529 7f932df59700 1 -- 172.21.2.102:0/2666674074 shutdown_connections 2021-02-10T11:30:31.530 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:31.529 7f932df59700 1 --2- 172.21.2.102:0/2666674074 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f930c018730 0x7f930c01abc0 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:31.530 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:31.529 7f932df59700 1 --2- 172.21.2.102:0/2666674074 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f93280939f0 0x7f9328093df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:31.530 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:31.529 7f932df59700 1 --2- 172.21.2.102:0/2666674074 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f93281aaa30 0x7f93281aae30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:31.531 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:31.529 7f932df59700 1 --2- 172.21.2.102:0/2666674074 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f93280a3fd0 0x7f9328094fc0 unknown :-1 s=CLOSED pgs=2000 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:31.531 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:31.529 7f932df59700 1 -- 172.21.2.102:0/2666674074 >> 172.21.2.102:0/2666674074 conn(0x7f932819a480 msgr2=0x7f932819a900 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:31.531 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:31.530 7f932df59700 1 -- 172.21.2.102:0/2666674074 shutdown_connections 2021-02-10T11:30:31.531 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:31.530 7f932df59700 1 -- 172.21.2.102:0/2666674074 wait complete. 2021-02-10T11:30:31.538 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume deauthorize cephfs subvolume_0000000000592823 guest1 --group_name group_0000000000973488 2021-02-10T11:30:31.971 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:31.953 7f31a2955700 1 Processor -- start 2021-02-10T11:30:31.971 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:31.953 7f31a2955700 1 -- start start 2021-02-10T11:30:31.971 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:31.954 7f31a2955700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f319c0dce80 0x7f319c0d14f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:31.971 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:31.954 7f31a2955700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f319c10b2c0 0x7f319c0eb620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:31.971 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:31.954 7f31a2955700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f319c10a550 con 0x7f319c0eb220 2021-02-10T11:30:31.971 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:31.954 7f31a2955700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f319c088240 con 0x7f319c0dce80 2021-02-10T11:30:31.971 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:31.954 7f31a2955700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f319c0d1f30 con 0x7f319c10b2c0 2021-02-10T11:30:31.972 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:31.955 7f319bfff700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f319c0dce80 0x7f319c0d14f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:31.972 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:31.955 7f319bfff700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f319c0dce80 0x7f319c0d14f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.113:3300/0 says I am v2:172.21.2.102:36330/0 (socket says 172.21.2.102:36330) 2021-02-10T11:30:31.972 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:31.955 7f31a0f72700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f319c10b2c0 0x7f319c0eb620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:31.972 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:31.955 7f319b7fe700 1 --1- >> v1:172.21.2.113:6789/0 conn(0x7f319c0eb220 0x7f319c10aad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.113:6789/0 says I am v1:172.21.2.102:60680/0 (socket says 172.21.2.102:60680) 2021-02-10T11:30:31.972 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:31.955 7f319bfff700 1 -- 172.21.2.102:0/2006957394 learned_addr learned my addr 172.21.2.102:0/2006957394 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:30:31.972 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:31.955 7f31a0f72700 1 -- 172.21.2.102:0/2006957394 >> v2:172.21.2.113:3300/0 conn(0x7f319c0dce80 msgr2=0x7f319c0d14f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:31.972 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:31.955 7f31a0f72700 1 --2- 172.21.2.102:0/2006957394 >> v2:172.21.2.113:3300/0 conn(0x7f319c0dce80 0x7f319c0d14f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:31.972 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:31.955 7f31a0f72700 1 -- 172.21.2.102:0/2006957394 >> v1:172.21.2.113:6789/0 conn(0x7f319c0eb220 legacy=0x7f319c10aad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:31.972 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:31.955 7f31a0f72700 1 -- 172.21.2.102:0/2006957394 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f319c0dd280 con 0x7f319c10b2c0 2021-02-10T11:30:31.972 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:31.955 7f31a0f72700 1 --2- 172.21.2.102:0/2006957394 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f319c10b2c0 0x7f319c0eb620 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f3194006e50 tx=0x7f3194009e70).ready entity=mon.2 client_cookie=eb5cd9490c0aeda9 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:31.972 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:31.956 7f319affd700 1 -- 172.21.2.102:0/2006957394 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3194006a20 con 0x7f319c10b2c0 2021-02-10T11:30:31.972 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:31.956 7f319affd700 1 -- 172.21.2.102:0/2006957394 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3194006790 con 0x7f319c10b2c0 2021-02-10T11:30:31.972 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:31.956 7f31a2955700 1 -- 172.21.2.102:0/2006957394 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f319c10b2c0 msgr2=0x7f319c0eb620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:31.973 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:31.956 7f31a2955700 1 --2- 172.21.2.102:0/2006957394 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f319c10b2c0 0x7f319c0eb620 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f3194006e50 tx=0x7f3194009e70).stop 2021-02-10T11:30:31.973 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:31.956 7f31a2955700 1 -- 172.21.2.102:0/2006957394 shutdown_connections 2021-02-10T11:30:31.973 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:31.956 7f31a2955700 1 --2- 172.21.2.102:0/2006957394 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f319c10b2c0 0x7f319c0eb620 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:31.973 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:31.956 7f31a2955700 1 --2- 172.21.2.102:0/2006957394 >> v2:172.21.2.113:3300/0 conn(0x7f319c0dce80 0x7f319c0d14f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:31.973 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:31.956 7f31a2955700 1 -- 172.21.2.102:0/2006957394 >> 172.21.2.102:0/2006957394 conn(0x7f319c0ea630 msgr2=0x7f319c0d0800 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:31.973 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:31.956 7f31a2955700 1 -- 172.21.2.102:0/2006957394 shutdown_connections 2021-02-10T11:30:31.973 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:31.956 7f31a2955700 1 -- 172.21.2.102:0/2006957394 wait complete. 2021-02-10T11:30:31.974 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:31.971 7f31a2955700 1 Processor -- start 2021-02-10T11:30:31.975 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:31.971 7f31a2955700 1 -- start start 2021-02-10T11:30:31.975 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:31.971 7f31a2955700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f319c0bec90 0x7f319c0915f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:31.975 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:31.971 7f31a2955700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f319c1a2f10 0x7f319c1a3310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:31.975 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:31.971 7f31a2955700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f319c1a3930 0x7f319c095ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:31.975 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:31.971 7f31a2955700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f319c10a350 con 0x7f319c0bec90 2021-02-10T11:30:31.975 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:31.971 7f31a2955700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f319c091e90 con 0x7f319c1a3930 2021-02-10T11:30:31.975 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:31.971 7f31a2955700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f319c0964e0 con 0x7f319c1a2f10 2021-02-10T11:30:31.975 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:31.971 7f319bfff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f319c0bec90 0x7f319c0915f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:31.975 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:31.971 7f319bfff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f319c0bec90 0x7f319c0915f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:32904/0 (socket says 172.21.2.102:32904) 2021-02-10T11:30:31.975 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:31.971 7f319bfff700 1 -- 172.21.2.102:0/3895242429 learned_addr learned my addr 172.21.2.102:0/3895242429 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:30:31.975 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:31.972 7f31a0f72700 1 --2- 172.21.2.102:0/3895242429 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f319c1a3930 0x7f319c095ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:31.975 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:31.972 7f319b7fe700 1 --2- 172.21.2.102:0/3895242429 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f319c1a2f10 0x7f319c1a3310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:31.975 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:31.972 7f319bfff700 1 -- 172.21.2.102:0/3895242429 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f319c1a2f10 msgr2=0x7f319c1a3310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:31.976 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:31.972 7f319bfff700 1 --2- 172.21.2.102:0/3895242429 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f319c1a2f10 0x7f319c1a3310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:31.976 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:31.972 7f319bfff700 1 -- 172.21.2.102:0/3895242429 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f319c1a3930 msgr2=0x7f319c095ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:31.976 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:31.972 7f319bfff700 1 --2- 172.21.2.102:0/3895242429 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f319c1a3930 0x7f319c095ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:31.976 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:31.972 7f319bfff700 1 -- 172.21.2.102:0/3895242429 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f319c0966c0 con 0x7f319c0bec90 2021-02-10T11:30:31.976 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:31.972 7f319bfff700 1 --2- 172.21.2.102:0/3895242429 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f319c0bec90 0x7f319c0915f0 secure :-1 s=READY pgs=2001 cs=0 l=1 rev1=1 rx=0x7f318c00b5e0 tx=0x7f318c00b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:31.976 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:31.972 7f3198ff9700 1 -- 172.21.2.102:0/3895242429 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f318c006340 con 0x7f319c0bec90 2021-02-10T11:30:31.976 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:31.972 7f3198ff9700 1 -- 172.21.2.102:0/3895242429 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f318c00bd70 con 0x7f319c0bec90 2021-02-10T11:30:31.976 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:31.972 7f31a2955700 1 -- 172.21.2.102:0/3895242429 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f319c0966c0 con 0x7f319c0bec90 2021-02-10T11:30:31.976 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:31.972 7f3198ff9700 1 -- 172.21.2.102:0/3895242429 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f318c0022b0 con 0x7f319c0bec90 2021-02-10T11:30:31.976 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:31.972 7f31a2955700 1 -- 172.21.2.102:0/3895242429 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f319c096c50 con 0x7f319c0bec90 2021-02-10T11:30:31.976 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:31.973 7f3198ff9700 1 -- 172.21.2.102:0/3895242429 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f318c000c90 con 0x7f319c0bec90 2021-02-10T11:30:31.976 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:31.973 7f3198ff9700 1 --2- 172.21.2.102:0/3895242429 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f317c0186c0 0x7f317c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:31.976 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:31.973 7f3198ff9700 1 -- 172.21.2.102:0/3895242429 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f318c032420 con 0x7f319c0bec90 2021-02-10T11:30:31.977 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:31.973 7f319b7fe700 1 --2- 172.21.2.102:0/3895242429 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f317c0186c0 0x7f317c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:31.977 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:31.973 7f319b7fe700 1 --2- 172.21.2.102:0/3895242429 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f317c0186c0 0x7f317c01abc0 crc :-1 s=READY pgs=893 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:32.005 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:32.004 7f31a2955700 1 -- 172.21.2.102:0/3895242429 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f319c091c60 con 0x7f319c0bec90 2021-02-10T11:30:32.007 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:32.007 7f3198ff9700 1 -- 172.21.2.102:0/3895242429 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f318c000960 con 0x7f319c0bec90 2021-02-10T11:30:32.133 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:32.132 7f31a2955700 1 -- 172.21.2.102:0/3895242429 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume deauthorize", "vol_name": "cephfs", "sub_name": "subvolume_0000000000592823", "target": ["mgr", ""], "auth_id": "guest1", "group_name": "group_0000000000973488"}) v1 -- 0x7f319c091e90 con 0x7f317c0186c0 2021-02-10T11:30:32.167 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:32.166 7f3198ff9700 1 -- 172.21.2.102:0/3895242429 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f31900010b0 con 0x7f317c0186c0 2021-02-10T11:30:32.198 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:32.197 7f31a2955700 1 -- 172.21.2.102:0/3895242429 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f317c0186c0 msgr2=0x7f317c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:32.198 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:32.197 7f31a2955700 1 --2- 172.21.2.102:0/3895242429 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f317c0186c0 0x7f317c01abc0 crc :-1 s=READY pgs=893 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:32.199 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:32.197 7f31a2955700 1 -- 172.21.2.102:0/3895242429 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f319c0bec90 msgr2=0x7f319c0915f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:32.199 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:32.197 7f31a2955700 1 --2- 172.21.2.102:0/3895242429 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f319c0bec90 0x7f319c0915f0 secure :-1 s=READY pgs=2001 cs=0 l=1 rev1=1 rx=0x7f318c00b5e0 tx=0x7f318c00b980).stop 2021-02-10T11:30:32.199 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:32.197 7f31a2955700 1 -- 172.21.2.102:0/3895242429 shutdown_connections 2021-02-10T11:30:32.199 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:32.197 7f31a2955700 1 --2- 172.21.2.102:0/3895242429 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f317c0186c0 0x7f317c01abc0 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:32.199 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:32.197 7f31a2955700 1 --2- 172.21.2.102:0/3895242429 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f319c1a3930 0x7f319c095ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:32.199 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:32.197 7f31a2955700 1 --2- 172.21.2.102:0/3895242429 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f319c1a2f10 0x7f319c1a3310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:32.199 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:32.197 7f31a2955700 1 --2- 172.21.2.102:0/3895242429 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f319c0bec90 0x7f319c0915f0 unknown :-1 s=CLOSED pgs=2001 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:32.199 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:32.197 7f31a2955700 1 -- 172.21.2.102:0/3895242429 >> 172.21.2.102:0/3895242429 conn(0x7f319c1083c0 msgr2=0x7f319c094d50 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:32.199 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:32.197 7f31a2955700 1 -- 172.21.2.102:0/3895242429 shutdown_connections 2021-02-10T11:30:32.199 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:32.197 7f31a2955700 1 -- 172.21.2.102:0/3895242429 wait complete. 2021-02-10T11:30:32.207 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth get client.guest1 --format=json-pretty 2021-02-10T11:30:32.634 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:32.617 7f80100c7700 1 Processor -- start 2021-02-10T11:30:32.635 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:32.617 7f80100c7700 1 -- start start 2021-02-10T11:30:32.635 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:32.618 7f80100c7700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f80080d0340 0x7f80080ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:32.635 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:32.618 7f80100c7700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f80080d1c70 0x7f80080efc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:32.635 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:32.618 7f80100c7700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80080f0b10 con 0x7f80080d1870 2021-02-10T11:30:32.635 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:32.618 7f80100c7700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80080e5ed0 con 0x7f80080d1c70 2021-02-10T11:30:32.635 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:32.618 7f80100c7700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80081b0be0 con 0x7f80080d0340 2021-02-10T11:30:32.635 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:32.618 7f800e6e4700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f80080d1c70 0x7f80080efc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:32.635 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:32.618 7f800dee3700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f80080d1870 0x7f80081b03f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:53368/0 (socket says 172.21.2.102:53368) 2021-02-10T11:30:32.635 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:32.618 7f800e6e4700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f80080d1c70 0x7f80080efc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:32916/0 (socket says 172.21.2.102:32916) 2021-02-10T11:30:32.635 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:32.618 7f800dee3700 1 -- 172.21.2.102:0/853001179 learned_addr learned my addr 172.21.2.102:0/853001179 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:30:32.635 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:32.618 7f800d6e2700 1 --2- 172.21.2.102:0/853001179 >> v2:172.21.2.113:3300/0 conn(0x7f80080d0340 0x7f80080ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:32.636 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:32.618 7f800e6e4700 1 -- 172.21.2.102:0/853001179 >> v2:172.21.2.113:3300/0 conn(0x7f80080d0340 msgr2=0x7f80080ef640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:32.636 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:32.618 7f800e6e4700 1 --2- 172.21.2.102:0/853001179 >> v2:172.21.2.113:3300/0 conn(0x7f80080d0340 0x7f80080ef640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:32.636 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:32.618 7f800e6e4700 1 -- 172.21.2.102:0/853001179 >> v1:172.21.2.102:6789/0 conn(0x7f80080d1870 legacy=0x7f80081b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:32.636 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:32.618 7f800e6e4700 1 -- 172.21.2.102:0/853001179 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80080f20a0 con 0x7f80080d1c70 2021-02-10T11:30:32.636 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:32.619 7f800e6e4700 1 --2- 172.21.2.102:0/853001179 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f80080d1c70 0x7f80080efc60 secure :-1 s=READY pgs=2003 cs=0 l=1 rev1=1 rx=0x7f8000006e30 tx=0x7f8000009e60).ready entity=mon.0 client_cookie=7b783a2ab6869853 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:32.636 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:32.619 7f800cee1700 1 -- 172.21.2.102:0/853001179 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8000003030 con 0x7f80080d1c70 2021-02-10T11:30:32.636 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:32.619 7f800cee1700 1 -- 172.21.2.102:0/853001179 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f800000b300 con 0x7f80080d1c70 2021-02-10T11:30:32.636 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:32.619 7f800cee1700 1 -- 172.21.2.102:0/853001179 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f800000b7e0 con 0x7f80080d1c70 2021-02-10T11:30:32.636 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:32.619 7f80100c7700 1 -- 172.21.2.102:0/853001179 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f80080d1c70 msgr2=0x7f80080efc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:32.636 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:32.619 7f80100c7700 1 --2- 172.21.2.102:0/853001179 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f80080d1c70 0x7f80080efc60 secure :-1 s=READY pgs=2003 cs=0 l=1 rev1=1 rx=0x7f8000006e30 tx=0x7f8000009e60).stop 2021-02-10T11:30:32.636 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:32.620 7f80100c7700 1 -- 172.21.2.102:0/853001179 shutdown_connections 2021-02-10T11:30:32.636 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:32.620 7f80100c7700 1 --2- 172.21.2.102:0/853001179 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f80080d1c70 0x7f80080efc60 unknown :-1 s=CLOSED pgs=2003 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:32.636 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:32.620 7f80100c7700 1 --2- 172.21.2.102:0/853001179 >> v2:172.21.2.113:3300/0 conn(0x7f80080d0340 0x7f80080ef640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:32.637 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:32.620 7f80100c7700 1 -- 172.21.2.102:0/853001179 >> 172.21.2.102:0/853001179 conn(0x7f80080ce450 msgr2=0x7f80080d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:32.637 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:32.620 7f80100c7700 1 -- 172.21.2.102:0/853001179 shutdown_connections 2021-02-10T11:30:32.637 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:32.620 7f80100c7700 1 -- 172.21.2.102:0/853001179 wait complete. 2021-02-10T11:30:32.638 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:32.634 7f80100c7700 1 Processor -- start 2021-02-10T11:30:32.638 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:32.634 7f80100c7700 1 -- start start 2021-02-10T11:30:32.638 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:32.635 7f80100c7700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8008189d00 0x7f800818a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:32.638 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:32.635 7f80100c7700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f800818a720 0x7f800818ab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:32.638 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:32.635 7f80100c7700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8008187e50 0x7f8008188250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:32.638 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:32.635 7f80100c7700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80080c37e0 con 0x7f800818a720 2021-02-10T11:30:32.638 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:32.635 7f80100c7700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80080c39d0 con 0x7f8008187e50 2021-02-10T11:30:32.639 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:32.635 7f80100c7700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8008188870 con 0x7f8008189d00 2021-02-10T11:30:32.639 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:32.635 7f800d6e2700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f800818a720 0x7f800818ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:32.639 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:32.635 7f800d6e2700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f800818a720 0x7f800818ab20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:32922/0 (socket says 172.21.2.102:32922) 2021-02-10T11:30:32.639 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:32.635 7f800d6e2700 1 -- 172.21.2.102:0/2437361718 learned_addr learned my addr 172.21.2.102:0/2437361718 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:30:32.639 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:32.635 7f800d6e2700 1 -- 172.21.2.102:0/2437361718 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8008189d00 msgr2=0x7f800818a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:32.639 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:32.635 7f800d6e2700 1 --2- 172.21.2.102:0/2437361718 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8008189d00 0x7f800818a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:32.639 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:32.635 7f800d6e2700 1 -- 172.21.2.102:0/2437361718 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8008187e50 msgr2=0x7f8008188250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:32.639 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:32.635 7f800d6e2700 1 --2- 172.21.2.102:0/2437361718 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8008187e50 0x7f8008188250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:32.640 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:32.635 7f800d6e2700 1 -- 172.21.2.102:0/2437361718 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8008188a90 con 0x7f800818a720 2021-02-10T11:30:32.640 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:32.636 7f800d6e2700 1 --2- 172.21.2.102:0/2437361718 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f800818a720 0x7f800818ab20 secure :-1 s=READY pgs=2004 cs=0 l=1 rev1=1 rx=0x7f8004006f00 tx=0x7f800400b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:32.640 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:32.636 7f7ffa7fc700 1 -- 172.21.2.102:0/2437361718 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f800400bd30 con 0x7f800818a720 2021-02-10T11:30:32.640 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:32.636 7f7ffa7fc700 1 -- 172.21.2.102:0/2437361718 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8004002030 con 0x7f800818a720 2021-02-10T11:30:32.640 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:32.636 7f80100c7700 1 -- 172.21.2.102:0/2437361718 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8008188a90 con 0x7f800818a720 2021-02-10T11:30:32.640 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:32.636 7f7ffa7fc700 1 -- 172.21.2.102:0/2437361718 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8004002590 con 0x7f800818a720 2021-02-10T11:30:32.640 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:32.636 7f80100c7700 1 -- 172.21.2.102:0/2437361718 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8008188a90 con 0x7f800818a720 2021-02-10T11:30:32.640 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:32.636 7f7ffa7fc700 1 -- 172.21.2.102:0/2437361718 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f8004002210 con 0x7f800818a720 2021-02-10T11:30:32.641 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:32.636 7f7ffa7fc700 1 --2- 172.21.2.102:0/2437361718 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7ff0018730 0x7f7ff001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:32.641 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:32.636 7f7ffa7fc700 1 -- 172.21.2.102:0/2437361718 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f8004002210 con 0x7f800818a720 2021-02-10T11:30:32.641 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:32.636 7f800dee3700 1 --2- 172.21.2.102:0/2437361718 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7ff0018730 0x7f7ff001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:32.641 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:32.637 7f800dee3700 1 --2- 172.21.2.102:0/2437361718 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7ff0018730 0x7f7ff001ac10 crc :-1 s=READY pgs=894 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:32.668 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:32.667 7f80100c7700 1 -- 172.21.2.102:0/2437361718 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80080ebc50 con 0x7f800818a720 2021-02-10T11:30:32.671 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:32.670 7f7ffa7fc700 1 -- 172.21.2.102:0/2437361718 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f8004002030 con 0x7f800818a720 2021-02-10T11:30:32.804 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:32.803 7f80100c7700 1 -- 172.21.2.102:0/2437361718 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "auth get", "entity": "client.guest1", "format": "json-pretty"} v 0) v1 -- 0x7f80080f0b10 con 0x7f800818a720 2021-02-10T11:30:32.805 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:32.804 7f7ffa7fc700 1 -- 172.21.2.102:0/2437361718 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "auth get", "entity": "client.guest1", "format": "json-pretty"}]=0 exported keyring for client.guest1 v57) v1 ==== 142+0+235 (secure 0 0 0) 0x7f80040009b0 con 0x7f800818a720 2021-02-10T11:30:32.805 INFO:teuthology.orchestra.run.gibba002.stderr:exported keyring for client.guest1 2021-02-10T11:30:32.805 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:30:32.805 INFO:teuthology.orchestra.run.gibba002.stdout:[ 2021-02-10T11:30:32.806 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:30:32.806 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "client.guest1", 2021-02-10T11:30:32.806 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQDWwyNgDrQfEBAA0jPM0K1bUzptthOlYECpWA==", 2021-02-10T11:30:32.806 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:30:32.806 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow rw path=/volumes/group_0000000000973488", 2021-02-10T11:30:32.806 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *" 2021-02-10T11:30:32.806 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:30:32.806 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:30:32.806 INFO:teuthology.orchestra.run.gibba002.stdout:] 2021-02-10T11:30:32.807 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:32.806 7f80100c7700 1 -- 172.21.2.102:0/2437361718 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7ff0018730 msgr2=0x7f7ff001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:32.807 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:32.806 7f80100c7700 1 --2- 172.21.2.102:0/2437361718 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7ff0018730 0x7f7ff001ac10 crc :-1 s=READY pgs=894 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:32.807 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:32.806 7f80100c7700 1 -- 172.21.2.102:0/2437361718 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f800818a720 msgr2=0x7f800818ab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:32.807 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:32.806 7f80100c7700 1 --2- 172.21.2.102:0/2437361718 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f800818a720 0x7f800818ab20 secure :-1 s=READY pgs=2004 cs=0 l=1 rev1=1 rx=0x7f8004006f00 tx=0x7f800400b940).stop 2021-02-10T11:30:32.807 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:32.806 7f80100c7700 1 -- 172.21.2.102:0/2437361718 shutdown_connections 2021-02-10T11:30:32.807 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:32.806 7f80100c7700 1 --2- 172.21.2.102:0/2437361718 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7ff0018730 0x7f7ff001ac10 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:32.807 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:32.806 7f80100c7700 1 --2- 172.21.2.102:0/2437361718 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8008187e50 0x7f8008188250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:32.807 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:32.806 7f80100c7700 1 --2- 172.21.2.102:0/2437361718 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f800818a720 0x7f800818ab20 unknown :-1 s=CLOSED pgs=2004 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:32.808 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:32.806 7f80100c7700 1 --2- 172.21.2.102:0/2437361718 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8008189d00 0x7f800818a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:32.808 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:32.806 7f80100c7700 1 -- 172.21.2.102:0/2437361718 >> 172.21.2.102:0/2437361718 conn(0x7f80080aed50 msgr2=0x7f800809c9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:32.808 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:32.806 7f80100c7700 1 -- 172.21.2.102:0/2437361718 shutdown_connections 2021-02-10T11:30:32.808 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:32.806 7f80100c7700 1 -- 172.21.2.102:0/2437361718 wait complete. 2021-02-10T11:30:32.815 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth rm client.guest1 2021-02-10T11:30:33.244 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:33.226 7f0f696a3700 1 Processor -- start 2021-02-10T11:30:33.244 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:33.226 7f0f696a3700 1 -- start start 2021-02-10T11:30:33.245 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:33.227 7f0f696a3700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0f640d1870 0x7f0f641b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:33.245 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:33.227 7f0f696a3700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f0f640f0650 0x7f0f640cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:33.245 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:33.227 7f0f696a3700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f640f0b10 con 0x7f0f641b0a10 2021-02-10T11:30:33.245 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:33.227 7f0f696a3700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f640fe230 con 0x7f0f640f0650 2021-02-10T11:30:33.245 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:33.227 7f0f696a3700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f640e5ed0 con 0x7f0f640d1870 2021-02-10T11:30:33.245 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:33.227 7f0f62abe700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f0f641b0a10 0x7f0f640cf330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:53382/0 (socket says 172.21.2.102:53382) 2021-02-10T11:30:33.245 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:33.227 7f0f62abe700 1 -- 172.21.2.102:0/473148192 learned_addr learned my addr 172.21.2.102:0/473148192 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:30:33.245 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:33.227 7f0f622bd700 1 -- 172.21.2.102:0/473148192 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2942587032 0 0) 0x7f0f580010b0 con 0x7f0f641b0a10 2021-02-10T11:30:33.245 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:33.227 7f0f622bd700 1 -- 172.21.2.102:0/473148192 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f4c002fe0 con 0x7f0f641b0a10 2021-02-10T11:30:33.245 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:33.227 7f0f63ac0700 1 --2- 172.21.2.102:0/473148192 >> v2:172.21.2.113:3300/0 conn(0x7f0f640f0650 0x7f0f640cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:33.245 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:33.227 7f0f632bf700 1 --2- 172.21.2.102:0/473148192 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0f640d1870 0x7f0f641b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:33.245 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:33.228 7f0f622bd700 1 -- 172.21.2.102:0/473148192 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3210718357 0 0) 0x7f0f580010b0 con 0x7f0f641b0a10 2021-02-10T11:30:33.246 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:33.228 7f0f622bd700 1 -- 172.21.2.102:0/473148192 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0f640d1870 msgr2=0x7f0f641b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:33.246 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:33.228 7f0f622bd700 1 --2- 172.21.2.102:0/473148192 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0f640d1870 0x7f0f641b03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:33.246 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:33.228 7f0f622bd700 1 -- 172.21.2.102:0/473148192 >> v2:172.21.2.113:3300/0 conn(0x7f0f640f0650 msgr2=0x7f0f640cfb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:33.246 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:33.228 7f0f622bd700 1 --2- 172.21.2.102:0/473148192 >> v2:172.21.2.113:3300/0 conn(0x7f0f640f0650 0x7f0f640cfb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:33.246 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:33.228 7f0f622bd700 1 -- 172.21.2.102:0/473148192 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f640f20a0 con 0x7f0f641b0a10 2021-02-10T11:30:33.246 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:33.228 7f0f622bd700 1 -- 172.21.2.102:0/473148192 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7f0f580015b0 con 0x7f0f641b0a10 2021-02-10T11:30:33.246 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:33.228 7f0f622bd700 1 -- 172.21.2.102:0/473148192 >> v1:172.21.2.102:6789/0 conn(0x7f0f641b0a10 legacy=0x7f0f640cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:33.246 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:33.228 7f0f622bd700 1 --2- 172.21.2.102:0/473148192 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0f4c003b10 0x7f0f4c005fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:33.246 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:33.228 7f0f622bd700 1 --2- 172.21.2.102:0/473148192 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0f4c04a7c0 0x7f0f4c04cc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:33.246 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:33.228 7f0f622bd700 1 --2- 172.21.2.102:0/473148192 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0f4c04d500 0x7f0f4c04fa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:33.247 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:33.228 7f0f622bd700 1 -- 172.21.2.102:0/473148192 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f4c0502d0 con 0x7f0f4c04a7c0 2021-02-10T11:30:33.247 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:33.228 7f0f622bd700 1 -- 172.21.2.102:0/473148192 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f4c0504f0 con 0x7f0f4c04d500 2021-02-10T11:30:33.247 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:33.228 7f0f622bd700 1 -- 172.21.2.102:0/473148192 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f4c050710 con 0x7f0f4c003b10 2021-02-10T11:30:33.247 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:33.228 7f0f632bf700 1 --2- 172.21.2.102:0/473148192 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0f4c003b10 0x7f0f4c005fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:33.247 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:33.228 7f0f62abe700 1 --2- 172.21.2.102:0/473148192 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0f4c04a7c0 0x7f0f4c04cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:33.247 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:33.228 7f0f632bf700 1 -- 172.21.2.102:0/473148192 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0f4c04d500 msgr2=0x7f0f4c04fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:33.247 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:33.228 7f0f632bf700 1 --2- 172.21.2.102:0/473148192 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0f4c04d500 0x7f0f4c04fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:33.247 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:33.228 7f0f632bf700 1 -- 172.21.2.102:0/473148192 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0f4c04a7c0 msgr2=0x7f0f4c04cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:33.247 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:33.228 7f0f632bf700 1 --2- 172.21.2.102:0/473148192 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0f4c04a7c0 0x7f0f4c04cc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:33.247 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:33.228 7f0f632bf700 1 -- 172.21.2.102:0/473148192 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f4c0509b0 con 0x7f0f4c003b10 2021-02-10T11:30:33.247 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:33.228 7f0f632bf700 1 --2- 172.21.2.102:0/473148192 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0f4c003b10 0x7f0f4c005fe0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7f0f54009310 tx=0x7f0f54009a90).ready entity=mon.2 client_cookie=b320a5f37c5e57d1 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:33.247 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:33.228 7f0f622bd700 1 -- 172.21.2.102:0/473148192 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0f5400c0c0 con 0x7f0f4c003b10 2021-02-10T11:30:33.247 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:33.228 7f0f622bd700 1 -- 172.21.2.102:0/473148192 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f54006400 con 0x7f0f4c003b10 2021-02-10T11:30:33.248 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:33.228 7f0f622bd700 1 -- 172.21.2.102:0/473148192 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0f5400c0c0 con 0x7f0f4c003b10 2021-02-10T11:30:33.248 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:33.228 7f0f696a3700 1 -- 172.21.2.102:0/473148192 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0f4c003b10 msgr2=0x7f0f4c005fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:33.248 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:33.228 7f0f696a3700 1 --2- 172.21.2.102:0/473148192 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0f4c003b10 0x7f0f4c005fe0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7f0f54009310 tx=0x7f0f54009a90).stop 2021-02-10T11:30:33.248 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:33.228 7f0f63ac0700 1 -- 172.21.2.102:0/473148192 reap_dead start 2021-02-10T11:30:33.248 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:33.229 7f0f696a3700 1 -- 172.21.2.102:0/473148192 shutdown_connections 2021-02-10T11:30:33.248 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:33.229 7f0f696a3700 1 -- 172.21.2.102:0/473148192 >> 172.21.2.102:0/473148192 conn(0x7f0f640ce450 msgr2=0x7f0f640d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:33.248 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:33.229 7f0f696a3700 1 -- 172.21.2.102:0/473148192 shutdown_connections 2021-02-10T11:30:33.248 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:33.229 7f0f696a3700 1 -- 172.21.2.102:0/473148192 wait complete. 2021-02-10T11:30:33.248 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:33.244 7f0f696a3700 1 Processor -- start 2021-02-10T11:30:33.248 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:33.244 7f0f696a3700 1 -- start start 2021-02-10T11:30:33.249 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:33.244 7f0f696a3700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0f641a49b0 0x7f0f64094a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:33.249 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:33.244 7f0f696a3700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0f64095080 0x7f0f640c0370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:33.250 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:33.244 7f0f696a3700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0f640c0990 0x7f0f640c0d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:33.250 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:33.244 7f0f696a3700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f640c13b0 con 0x7f0f640c0990 2021-02-10T11:30:33.250 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:33.244 7f0f696a3700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f640c15d0 con 0x7f0f64095080 2021-02-10T11:30:33.250 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:33.244 7f0f696a3700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f640c1d80 con 0x7f0f641a49b0 2021-02-10T11:30:33.250 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:33.244 7f0f63ac0700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0f640c0990 0x7f0f640c0d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:33.250 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:33.244 7f0f63ac0700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0f640c0990 0x7f0f640c0d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:32938/0 (socket says 172.21.2.102:32938) 2021-02-10T11:30:33.250 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:33.244 7f0f63ac0700 1 -- 172.21.2.102:0/2757649972 learned_addr learned my addr 172.21.2.102:0/2757649972 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:30:33.250 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:33.245 7f0f63ac0700 1 -- 172.21.2.102:0/2757649972 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0f641a49b0 msgr2=0x7f0f64094a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:33.250 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:33.245 7f0f63ac0700 1 --2- 172.21.2.102:0/2757649972 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0f641a49b0 0x7f0f64094a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:33.250 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:33.245 7f0f63ac0700 1 -- 172.21.2.102:0/2757649972 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0f64095080 msgr2=0x7f0f640c0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:33.250 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:33.245 7f0f63ac0700 1 --2- 172.21.2.102:0/2757649972 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0f64095080 0x7f0f640c0370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:33.250 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:33.245 7f0f63ac0700 1 -- 172.21.2.102:0/2757649972 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f640c1fa0 con 0x7f0f640c0990 2021-02-10T11:30:33.250 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:33.245 7f0f63ac0700 1 --2- 172.21.2.102:0/2757649972 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0f640c0990 0x7f0f640c0d90 secure :-1 s=READY pgs=2006 cs=0 l=1 rev1=1 rx=0x7f0f5c0099a0 tx=0x7f0f5c00bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:33.251 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:33.245 7f0f4bfff700 1 -- 172.21.2.102:0/2757649972 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0f5c0066f0 con 0x7f0f640c0990 2021-02-10T11:30:33.251 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:33.245 7f0f4bfff700 1 -- 172.21.2.102:0/2757649972 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f5c00c2f0 con 0x7f0f640c0990 2021-02-10T11:30:33.251 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:33.245 7f0f4bfff700 1 -- 172.21.2.102:0/2757649972 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0f5c00c790 con 0x7f0f640c0990 2021-02-10T11:30:33.251 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:33.245 7f0f696a3700 1 -- 172.21.2.102:0/2757649972 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f640c1fa0 con 0x7f0f640c0990 2021-02-10T11:30:33.251 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:33.245 7f0f696a3700 1 -- 172.21.2.102:0/2757649972 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f640c1fa0 con 0x7f0f640c0990 2021-02-10T11:30:33.251 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:33.246 7f0f4bfff700 1 -- 172.21.2.102:0/2757649972 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f0f5c0011f0 con 0x7f0f640c0990 2021-02-10T11:30:33.251 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:33.246 7f0f4bfff700 1 --2- 172.21.2.102:0/2757649972 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0f440186c0 0x7f0f4401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:33.251 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:33.246 7f0f4bfff700 1 -- 172.21.2.102:0/2757649972 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f0f5c032760 con 0x7f0f640c0990 2021-02-10T11:30:33.251 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:33.246 7f0f632bf700 1 --2- 172.21.2.102:0/2757649972 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0f440186c0 0x7f0f4401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:33.251 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:33.246 7f0f632bf700 1 --2- 172.21.2.102:0/2757649972 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0f440186c0 0x7f0f4401abc0 crc :-1 s=READY pgs=895 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:33.277 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:33.277 7f0f696a3700 1 -- 172.21.2.102:0/2757649972 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f640ebc60 con 0x7f0f640c0990 2021-02-10T11:30:33.280 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:33.279 7f0f4bfff700 1 -- 172.21.2.102:0/2757649972 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f0f5c025ef0 con 0x7f0f640c0990 2021-02-10T11:30:33.407 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:33.406 7f0f696a3700 1 -- 172.21.2.102:0/2757649972 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "auth rm", "entity": "client.guest1"} v 0) v1 -- 0x7f0f640f0b10 con 0x7f0f640c0990 2021-02-10T11:30:33.410 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:33.410 7f0f4bfff700 1 -- 172.21.2.102:0/2757649972 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "auth rm", "entity": "client.guest1"}]=0 updated v58) v1 ==== 89+0+0 (secure 0 0 0) 0x7f0f5c025ef0 con 0x7f0f640c0990 2021-02-10T11:30:33.414 INFO:teuthology.orchestra.run.gibba002.stderr:updated 2021-02-10T11:30:33.416 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:33.415 7f0f696a3700 1 -- 172.21.2.102:0/2757649972 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0f440186c0 msgr2=0x7f0f4401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:33.416 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:33.415 7f0f696a3700 1 --2- 172.21.2.102:0/2757649972 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0f440186c0 0x7f0f4401abc0 crc :-1 s=READY pgs=895 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:33.416 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:33.415 7f0f696a3700 1 -- 172.21.2.102:0/2757649972 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0f640c0990 msgr2=0x7f0f640c0d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:33.416 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:33.415 7f0f696a3700 1 --2- 172.21.2.102:0/2757649972 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0f640c0990 0x7f0f640c0d90 secure :-1 s=READY pgs=2006 cs=0 l=1 rev1=1 rx=0x7f0f5c0099a0 tx=0x7f0f5c00bcf0).stop 2021-02-10T11:30:33.417 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:33.415 7f0f696a3700 1 -- 172.21.2.102:0/2757649972 shutdown_connections 2021-02-10T11:30:33.417 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:33.415 7f0f696a3700 1 --2- 172.21.2.102:0/2757649972 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0f640c0990 0x7f0f640c0d90 unknown :-1 s=CLOSED pgs=2006 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:33.417 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:33.415 7f0f696a3700 1 --2- 172.21.2.102:0/2757649972 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0f440186c0 0x7f0f4401abc0 unknown :-1 s=CLOSED pgs=895 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:33.417 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:33.415 7f0f696a3700 1 --2- 172.21.2.102:0/2757649972 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0f64095080 0x7f0f640c0370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:33.417 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:33.415 7f0f696a3700 1 --2- 172.21.2.102:0/2757649972 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0f641a49b0 0x7f0f64094a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:33.417 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:33.415 7f0f696a3700 1 -- 172.21.2.102:0/2757649972 >> 172.21.2.102:0/2757649972 conn(0x7f0f640c3e80 msgr2=0x7f0f640f00d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:33.417 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:33.416 7f0f696a3700 1 -- 172.21.2.102:0/2757649972 shutdown_connections 2021-02-10T11:30:33.417 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:33.416 7f0f696a3700 1 -- 172.21.2.102:0/2757649972 wait complete. 2021-02-10T11:30:33.424 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000592823 --group_name group_0000000000973488 2021-02-10T11:30:33.856 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:33.838 7f01ef4b3700 1 Processor -- start 2021-02-10T11:30:33.856 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:33.839 7f01ef4b3700 1 -- start start 2021-02-10T11:30:33.856 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:33.839 7f01ef4b3700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f01e810b000 0x7f01e80cf760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:33.856 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:33.839 7f01ef4b3700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01e80fe1d0 con 0x7f01e80d1880 2021-02-10T11:30:33.856 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:33.839 7f01ef4b3700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01e81b0cc0 con 0x7f01e80d1c80 2021-02-10T11:30:33.856 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:33.839 7f01ef4b3700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01e8109a70 con 0x7f01e810b000 2021-02-10T11:30:33.856 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:33.839 7f01edad0700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f01e810b000 0x7f01e80cf760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:33.856 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:33.839 7f01ed2cf700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f01e80d1880 0x7f01e81b04d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:53402/0 (socket says 172.21.2.102:53402) 2021-02-10T11:30:33.857 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:33.839 7f01edad0700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f01e810b000 0x7f01e80cf760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:32950/0 (socket says 172.21.2.102:32950) 2021-02-10T11:30:33.857 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:33.839 7f01edad0700 1 -- 172.21.2.102:0/3975346696 learned_addr learned my addr 172.21.2.102:0/3975346696 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:30:33.857 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:33.840 7f01edad0700 1 -- 172.21.2.102:0/3975346696 >> v1:172.21.2.113:6789/0 conn(0x7f01e80d1c80 legacy=0x7f01e80cef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:33.857 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:33.840 7f01edad0700 1 -- 172.21.2.102:0/3975346696 >> v1:172.21.2.102:6789/0 conn(0x7f01e80d1880 legacy=0x7f01e81b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:33.857 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:33.840 7f01edad0700 1 -- 172.21.2.102:0/3975346696 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01e810b400 con 0x7f01e810b000 2021-02-10T11:30:33.857 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:33.840 7f01edad0700 1 --2- 172.21.2.102:0/3975346696 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f01e810b000 0x7f01e80cf760 secure :-1 s=READY pgs=2008 cs=0 l=1 rev1=1 rx=0x7f01e0006f00 tx=0x7f01e0009f10).ready entity=mon.0 client_cookie=17a51ef054905ced server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:33.857 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:33.840 7f01dbfff700 1 -- 172.21.2.102:0/3975346696 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f01e000b3b0 con 0x7f01e810b000 2021-02-10T11:30:33.857 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:33.840 7f01dbfff700 1 -- 172.21.2.102:0/3975346696 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01e000b630 con 0x7f01e810b000 2021-02-10T11:30:33.857 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:33.840 7f01dbfff700 1 -- 172.21.2.102:0/3975346696 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f01e000bb50 con 0x7f01e810b000 2021-02-10T11:30:33.857 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:33.840 7f01ef4b3700 1 -- 172.21.2.102:0/3975346696 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f01e810b000 msgr2=0x7f01e80cf760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:33.857 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:33.840 7f01ef4b3700 1 --2- 172.21.2.102:0/3975346696 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f01e810b000 0x7f01e80cf760 secure :-1 s=READY pgs=2008 cs=0 l=1 rev1=1 rx=0x7f01e0006f00 tx=0x7f01e0009f10).stop 2021-02-10T11:30:33.857 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:33.840 7f01ef4b3700 1 -- 172.21.2.102:0/3975346696 shutdown_connections 2021-02-10T11:30:33.857 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:33.840 7f01ef4b3700 1 --2- 172.21.2.102:0/3975346696 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f01e810b000 0x7f01e80cf760 unknown :-1 s=CLOSED pgs=2008 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:33.858 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:33.840 7f01ef4b3700 1 -- 172.21.2.102:0/3975346696 >> 172.21.2.102:0/3975346696 conn(0x7f01e80ce430 msgr2=0x7f01e80d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:33.858 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:33.841 7f01ef4b3700 1 -- 172.21.2.102:0/3975346696 shutdown_connections 2021-02-10T11:30:33.858 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:33.841 7f01ef4b3700 1 -- 172.21.2.102:0/3975346696 wait complete. 2021-02-10T11:30:33.858 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:33.855 7f01ef4b3700 1 Processor -- start 2021-02-10T11:30:33.858 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:33.855 7f01ef4b3700 1 -- start start 2021-02-10T11:30:33.859 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:33.856 7f01ef4b3700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f01e818f590 0x7f01e8095190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:33.859 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:33.856 7f01ef4b3700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f01e80957b0 0x7f01e8095bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:33.859 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:33.856 7f01ef4b3700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f01e80961d0 0x7f01e8096c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:33.859 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:33.856 7f01ef4b3700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01e80966c0 con 0x7f01e818f590 2021-02-10T11:30:33.859 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:33.856 7f01ef4b3700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01e818f990 con 0x7f01e80961d0 2021-02-10T11:30:33.859 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:33.856 7f01ef4b3700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01e8097260 con 0x7f01e80957b0 2021-02-10T11:30:33.860 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:33.856 7f01ed2cf700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f01e818f590 0x7f01e8095190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:33.860 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:33.856 7f01ed2cf700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f01e818f590 0x7f01e8095190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:32952/0 (socket says 172.21.2.102:32952) 2021-02-10T11:30:33.860 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:33.856 7f01ed2cf700 1 -- 172.21.2.102:0/1672467902 learned_addr learned my addr 172.21.2.102:0/1672467902 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:30:33.860 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:33.856 7f01edad0700 1 --2- 172.21.2.102:0/1672467902 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f01e80961d0 0x7f01e8096c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:33.860 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:33.856 7f01ecace700 1 --2- 172.21.2.102:0/1672467902 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f01e80957b0 0x7f01e8095bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:33.860 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:33.856 7f01ed2cf700 1 -- 172.21.2.102:0/1672467902 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f01e80957b0 msgr2=0x7f01e8095bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:33.860 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:33.856 7f01ed2cf700 1 --2- 172.21.2.102:0/1672467902 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f01e80957b0 0x7f01e8095bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:33.860 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:33.856 7f01ed2cf700 1 -- 172.21.2.102:0/1672467902 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f01e80961d0 msgr2=0x7f01e8096c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:33.860 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:33.856 7f01ed2cf700 1 --2- 172.21.2.102:0/1672467902 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f01e80961d0 0x7f01e8096c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:33.860 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:33.856 7f01ed2cf700 1 -- 172.21.2.102:0/1672467902 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01e8097480 con 0x7f01e818f590 2021-02-10T11:30:33.860 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:33.856 7f01ecace700 1 --2- 172.21.2.102:0/1672467902 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f01e80957b0 0x7f01e8095bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:30:33.860 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:33.857 7f01ed2cf700 1 --2- 172.21.2.102:0/1672467902 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f01e818f590 0x7f01e8095190 secure :-1 s=READY pgs=2009 cs=0 l=1 rev1=1 rx=0x7f01dc00a590 tx=0x7f01dc00a930).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:33.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:33.857 7f01d9ffb700 1 -- 172.21.2.102:0/1672467902 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f01dc000ef0 con 0x7f01e818f590 2021-02-10T11:30:33.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:33.857 7f01d9ffb700 1 -- 172.21.2.102:0/1672467902 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01dc00ad20 con 0x7f01e818f590 2021-02-10T11:30:33.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:33.857 7f01ef4b3700 1 -- 172.21.2.102:0/1672467902 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01e8097480 con 0x7f01e818f590 2021-02-10T11:30:33.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:33.857 7f01d9ffb700 1 -- 172.21.2.102:0/1672467902 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f01dc004620 con 0x7f01e818f590 2021-02-10T11:30:33.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:33.857 7f01ef4b3700 1 -- 172.21.2.102:0/1672467902 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01e8097a10 con 0x7f01e818f590 2021-02-10T11:30:33.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:33.858 7f01d9ffb700 1 -- 172.21.2.102:0/1672467902 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f01dc0043a0 con 0x7f01e818f590 2021-02-10T11:30:33.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:33.858 7f01d9ffb700 1 --2- 172.21.2.102:0/1672467902 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f01c80186c0 0x7f01c801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:33.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:33.858 7f01d9ffb700 1 -- 172.21.2.102:0/1672467902 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f01dc032410 con 0x7f01e818f590 2021-02-10T11:30:33.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:33.858 7f01ecace700 1 --2- 172.21.2.102:0/1672467902 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f01c80186c0 0x7f01c801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:33.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:33.858 7f01ecace700 1 --2- 172.21.2.102:0/1672467902 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f01c80186c0 0x7f01c801abc0 crc :-1 s=READY pgs=896 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:33.890 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:33.889 7f01ef4b3700 1 -- 172.21.2.102:0/1672467902 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01e8097fb0 con 0x7f01e818f590 2021-02-10T11:30:33.892 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:33.891 7f01d9ffb700 1 -- 172.21.2.102:0/1672467902 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f01dc02fdc0 con 0x7f01e818f590 2021-02-10T11:30:34.018 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:34.018 7f01ef4b3700 1 -- 172.21.2.102:0/1672467902 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000592823", "target": ["mgr", ""], "group_name": "group_0000000000973488"}) v1 -- 0x7f01e8097a10 con 0x7f01c80186c0 2021-02-10T11:30:34.027 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:34.027 7f01d9ffb700 1 -- 172.21.2.102:0/1672467902 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f01e4006bc0 con 0x7f01c80186c0 2021-02-10T11:30:34.035 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:34.035 7f01ef4b3700 1 -- 172.21.2.102:0/1672467902 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f01c80186c0 msgr2=0x7f01c801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:34.035 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:34.035 7f01ef4b3700 1 --2- 172.21.2.102:0/1672467902 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f01c80186c0 0x7f01c801abc0 crc :-1 s=READY pgs=896 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:34.035 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:34.035 7f01ef4b3700 1 -- 172.21.2.102:0/1672467902 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f01e818f590 msgr2=0x7f01e8095190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:34.036 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:34.035 7f01ef4b3700 1 --2- 172.21.2.102:0/1672467902 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f01e818f590 0x7f01e8095190 secure :-1 s=READY pgs=2009 cs=0 l=1 rev1=1 rx=0x7f01dc00a590 tx=0x7f01dc00a930).stop 2021-02-10T11:30:34.036 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:34.035 7f01ef4b3700 1 -- 172.21.2.102:0/1672467902 shutdown_connections 2021-02-10T11:30:34.036 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:34.035 7f01ef4b3700 1 --2- 172.21.2.102:0/1672467902 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f01c80186c0 0x7f01c801abc0 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:34.036 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:34.035 7f01ef4b3700 1 --2- 172.21.2.102:0/1672467902 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f01e80961d0 0x7f01e8096c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:34.036 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:34.035 7f01ef4b3700 1 --2- 172.21.2.102:0/1672467902 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f01e80957b0 0x7f01e8095bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:34.036 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:34.035 7f01ef4b3700 1 --2- 172.21.2.102:0/1672467902 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f01e818f590 0x7f01e8095190 unknown :-1 s=CLOSED pgs=2009 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:34.037 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:34.035 7f01ef4b3700 1 -- 172.21.2.102:0/1672467902 >> 172.21.2.102:0/1672467902 conn(0x7f01e819d0a0 msgr2=0x7f01e8189e90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:34.037 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:34.035 7f01ef4b3700 1 -- 172.21.2.102:0/1672467902 shutdown_connections 2021-02-10T11:30:34.037 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:34.035 7f01ef4b3700 1 -- 172.21.2.102:0/1672467902 wait complete. 2021-02-10T11:30:34.045 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolumegroup rm cephfs group_0000000000973488 2021-02-10T11:30:34.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:34.458 7fb5ca230700 1 Processor -- start 2021-02-10T11:30:34.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:34.458 7fb5ca230700 1 -- start start 2021-02-10T11:30:34.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:34.459 7fb5ca230700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb5c40d1870 0x7fb5c41b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:34.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:34.459 7fb5ca230700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fb5c40cff70 0x7fb5c4109130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:34.477 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:34.459 7fb5ca230700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5c4109a60 con 0x7fb5c41b0ae0 2021-02-10T11:30:34.477 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:34.459 7fb5ca230700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5c4109750 con 0x7fb5c40cff70 2021-02-10T11:30:34.477 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:34.459 7fb5ca230700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5c4088070 con 0x7fb5c40d1870 2021-02-10T11:30:34.477 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:34.459 7fb5c884d700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fb5c40cff70 0x7fb5c4109130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:34.477 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:34.459 7fb5c37fe700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fb5c41b0ae0 0x7fb5c4108940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:53416/0 (socket says 172.21.2.102:53416) 2021-02-10T11:30:34.477 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:34.459 7fb5c884d700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fb5c40cff70 0x7fb5c4109130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:32960/0 (socket says 172.21.2.102:32960) 2021-02-10T11:30:34.477 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:34.459 7fb5c37fe700 1 -- 172.21.2.102:0/3800947860 learned_addr learned my addr 172.21.2.102:0/3800947860 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:30:34.477 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:34.460 7fb5c884d700 1 -- 172.21.2.102:0/3800947860 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb5c40d1870 msgr2=0x7fb5c41b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:34.477 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:34.459 7fb5c2ffd700 1 -- 172.21.2.102:0/3800947860 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 933284469 0 0) 0x7fb5b80010b0 con 0x7fb5c41b0ae0 2021-02-10T11:30:34.477 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:34.460 7fb5c3fff700 1 --2- 172.21.2.102:0/3800947860 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb5c40d1870 0x7fb5c41b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:34.477 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:34.460 7fb5c884d700 1 --2- 172.21.2.102:0/3800947860 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb5c40d1870 0x7fb5c41b04c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:34.477 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:34.460 7fb5c884d700 1 -- 172.21.2.102:0/3800947860 >> v1:172.21.2.102:6789/0 conn(0x7fb5c41b0ae0 legacy=0x7fb5c4108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:34.477 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:34.460 7fb5c884d700 1 -- 172.21.2.102:0/3800947860 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5c4088260 con 0x7fb5c40cff70 2021-02-10T11:30:34.478 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:34.460 7fb5c3fff700 1 --2- 172.21.2.102:0/3800947860 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb5c40d1870 0x7fb5c41b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:30:34.478 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:34.460 7fb5c884d700 1 --2- 172.21.2.102:0/3800947860 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb5c40cff70 0x7fb5c4109130 secure :-1 s=READY pgs=2011 cs=0 l=1 rev1=1 rx=0x7fb5bc006e30 tx=0x7fb5bc009e60).ready entity=mon.0 client_cookie=1708d56e2dc596b9 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:34.478 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:34.460 7fb5c2ffd700 1 -- 172.21.2.102:0/3800947860 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb5bc00b300 con 0x7fb5c40cff70 2021-02-10T11:30:34.478 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:34.460 7fb5c2ffd700 1 -- 172.21.2.102:0/3800947860 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5bc000d30 con 0x7fb5c40cff70 2021-02-10T11:30:34.478 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:34.460 7fb5c2ffd700 1 -- 172.21.2.102:0/3800947860 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb5bc00b830 con 0x7fb5c40cff70 2021-02-10T11:30:34.478 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:34.460 7fb5ca230700 1 -- 172.21.2.102:0/3800947860 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb5c40cff70 msgr2=0x7fb5c4109130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:34.478 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:34.460 7fb5ca230700 1 --2- 172.21.2.102:0/3800947860 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb5c40cff70 0x7fb5c4109130 secure :-1 s=READY pgs=2011 cs=0 l=1 rev1=1 rx=0x7fb5bc006e30 tx=0x7fb5bc009e60).stop 2021-02-10T11:30:34.478 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:34.461 7fb5ca230700 1 -- 172.21.2.102:0/3800947860 shutdown_connections 2021-02-10T11:30:34.478 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:34.461 7fb5ca230700 1 --2- 172.21.2.102:0/3800947860 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb5c40cff70 0x7fb5c4109130 unknown :-1 s=CLOSED pgs=2011 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:34.478 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:34.461 7fb5ca230700 1 --2- 172.21.2.102:0/3800947860 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb5c40d1870 0x7fb5c41b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:34.478 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:34.461 7fb5ca230700 1 -- 172.21.2.102:0/3800947860 >> 172.21.2.102:0/3800947860 conn(0x7fb5c40ce420 msgr2=0x7fb5c40d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:34.478 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:34.461 7fb5ca230700 1 -- 172.21.2.102:0/3800947860 shutdown_connections 2021-02-10T11:30:34.478 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:34.461 7fb5ca230700 1 -- 172.21.2.102:0/3800947860 wait complete. 2021-02-10T11:30:34.479 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:34.475 7fb5ca230700 1 Processor -- start 2021-02-10T11:30:34.479 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:34.476 7fb5ca230700 1 -- start start 2021-02-10T11:30:34.480 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:34.476 7fb5ca230700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb5c4082bb0 0x7fb5c41917d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:34.480 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:34.476 7fb5ca230700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb5c4192060 0x7fb5c41a43f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:34.480 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:34.476 7fb5ca230700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb5c4192530 0x7fb5c4192930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:34.480 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:34.476 7fb5ca230700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5c41a4ae0 con 0x7fb5c4192060 2021-02-10T11:30:34.480 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:34.476 7fb5ca230700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5c4192f50 con 0x7fb5c4082bb0 2021-02-10T11:30:34.480 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:34.476 7fb5ca230700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5c4193170 con 0x7fb5c4192530 2021-02-10T11:30:34.480 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:34.476 7fb5c37fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb5c4192060 0x7fb5c41a43f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:34.480 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:34.476 7fb5c37fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb5c4192060 0x7fb5c41a43f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:32966/0 (socket says 172.21.2.102:32966) 2021-02-10T11:30:34.480 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:34.476 7fb5c37fe700 1 -- 172.21.2.102:0/2437407012 learned_addr learned my addr 172.21.2.102:0/2437407012 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:30:34.480 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:34.476 7fb5c37fe700 1 -- 172.21.2.102:0/2437407012 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb5c4192530 msgr2=0x7fb5c4192930 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:30:34.481 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:34.476 7fb5c37fe700 1 --2- 172.21.2.102:0/2437407012 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb5c4192530 0x7fb5c4192930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:34.481 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:34.476 7fb5c37fe700 1 -- 172.21.2.102:0/2437407012 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb5c4082bb0 msgr2=0x7fb5c41917d0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:30:34.481 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:34.476 7fb5c37fe700 1 --2- 172.21.2.102:0/2437407012 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb5c4082bb0 0x7fb5c41917d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:34.481 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:34.476 7fb5c37fe700 1 -- 172.21.2.102:0/2437407012 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5c40a1840 con 0x7fb5c4192060 2021-02-10T11:30:34.481 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:34.477 7fb5c37fe700 1 --2- 172.21.2.102:0/2437407012 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb5c4192060 0x7fb5c41a43f0 secure :-1 s=READY pgs=2012 cs=0 l=1 rev1=1 rx=0x7fb5b8001c10 tx=0x7fb5b800a8f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:34.481 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:34.477 7fb5c0ff9700 1 -- 172.21.2.102:0/2437407012 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb5b80009b0 con 0x7fb5c4192060 2021-02-10T11:30:34.481 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:34.477 7fb5c0ff9700 1 -- 172.21.2.102:0/2437407012 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5b80012c0 con 0x7fb5c4192060 2021-02-10T11:30:34.481 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:34.477 7fb5ca230700 1 -- 172.21.2.102:0/2437407012 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5c40a1840 con 0x7fb5c4192060 2021-02-10T11:30:34.481 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:34.477 7fb5c0ff9700 1 -- 172.21.2.102:0/2437407012 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb5b8004350 con 0x7fb5c4192060 2021-02-10T11:30:34.481 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:34.477 7fb5ca230700 1 -- 172.21.2.102:0/2437407012 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5c40a1dd0 con 0x7fb5c4192060 2021-02-10T11:30:34.481 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:34.478 7fb5c0ff9700 1 -- 172.21.2.102:0/2437407012 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fb5b8004140 con 0x7fb5c4192060 2021-02-10T11:30:34.481 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:34.478 7fb5c0ff9700 1 --2- 172.21.2.102:0/2437407012 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb5a40186c0 0x7fb5a401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:34.481 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:34.478 7fb5c0ff9700 1 -- 172.21.2.102:0/2437407012 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fb5b80323d0 con 0x7fb5c4192060 2021-02-10T11:30:34.481 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:34.478 7fb5c3fff700 1 --2- 172.21.2.102:0/2437407012 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb5a40186c0 0x7fb5a401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:34.482 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:34.478 7fb5c3fff700 1 --2- 172.21.2.102:0/2437407012 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb5a40186c0 0x7fb5a401abc0 crc :-1 s=READY pgs=897 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:34.509 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:34.508 7fb5ca230700 1 -- 172.21.2.102:0/2437407012 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5c4082980 con 0x7fb5c4192060 2021-02-10T11:30:34.512 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:34.511 7fb5c0ff9700 1 -- 172.21.2.102:0/2437407012 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fb5b80009b0 con 0x7fb5c4192060 2021-02-10T11:30:34.639 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:34.638 7fb5ca230700 1 -- 172.21.2.102:0/2437407012 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolumegroup rm", "vol_name": "cephfs", "target": ["mgr", ""], "group_name": "group_0000000000973488"}) v1 -- 0x7fb5c40a1dd0 con 0x7fb5a40186c0 2021-02-10T11:30:34.643 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:34.642 7fb5c0ff9700 1 -- 172.21.2.102:0/2437407012 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fb5b40009b0 con 0x7fb5a40186c0 2021-02-10T11:30:34.648 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:34.647 7fb5ca230700 1 -- 172.21.2.102:0/2437407012 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb5a40186c0 msgr2=0x7fb5a401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:34.648 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:34.647 7fb5ca230700 1 --2- 172.21.2.102:0/2437407012 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb5a40186c0 0x7fb5a401abc0 crc :-1 s=READY pgs=897 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:34.648 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:34.647 7fb5ca230700 1 -- 172.21.2.102:0/2437407012 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb5c4192060 msgr2=0x7fb5c41a43f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:34.648 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:34.647 7fb5ca230700 1 --2- 172.21.2.102:0/2437407012 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb5c4192060 0x7fb5c41a43f0 secure :-1 s=READY pgs=2012 cs=0 l=1 rev1=1 rx=0x7fb5b8001c10 tx=0x7fb5b800a8f0).stop 2021-02-10T11:30:34.649 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:34.647 7fb5ca230700 1 -- 172.21.2.102:0/2437407012 shutdown_connections 2021-02-10T11:30:34.649 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:34.647 7fb5ca230700 1 --2- 172.21.2.102:0/2437407012 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb5c4192530 0x7fb5c4192930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:34.649 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:34.647 7fb5ca230700 1 --2- 172.21.2.102:0/2437407012 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb5c4192060 0x7fb5c41a43f0 unknown :-1 s=CLOSED pgs=2012 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:34.649 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:34.647 7fb5ca230700 1 --2- 172.21.2.102:0/2437407012 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb5a40186c0 0x7fb5a401abc0 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:34.649 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:34.647 7fb5ca230700 1 --2- 172.21.2.102:0/2437407012 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb5c4082bb0 0x7fb5c41917d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:34.649 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:34.647 7fb5ca230700 1 -- 172.21.2.102:0/2437407012 >> 172.21.2.102:0/2437407012 conn(0x7fb5c40c3920 msgr2=0x7fb5c40a6ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:34.649 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:34.648 7fb5ca230700 1 -- 172.21.2.102:0/2437407012 shutdown_connections 2021-02-10T11:30:34.649 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:34.648 7fb5ca230700 1 -- 172.21.2.102:0/2437407012 wait complete. 2021-02-10T11:30:34.656 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2021-02-10T11:30:34.657 DEBUG:teuthology.orchestra.run.gibba002:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:30:35.002 DEBUG:teuthology.orchestra.run.gibba013:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:30:35.346 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2021-02-10T11:30:35.347 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:30:35.347 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:30:35.347 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:30:35.682 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:30:35.682 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:30:35.683 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@gibba002.front.sepia.ceph.com... 2021-02-10T11:30:35.683 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-02-10T11:30:35.683 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:30:36.044 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba002.stderr:ceph-fuse[174201]: fuse finished with error 0 and tester_r 0 2021-02-10T11:30:36.045 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:30:36.045 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:30:36.045 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:30:36.379 INFO:teuthology.orchestra.run.gibba002.stdout:ext2/ext3 2021-02-10T11:30:36.380 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2021-02-10T11:30:36.380 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:30:36.380 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:30:36.715 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:30:36.715 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:30:36.715 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:30:37.050 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:30:37.051 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2021-02-10T11:30:37.051 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@gibba002.front.sepia.ceph.com... 2021-02-10T11:30:37.051 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-02-10T11:30:37.051 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:30:37.405 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba002.stderr:ceph-fuse[174841]: fuse finished with error 0 and tester_r 0 2021-02-10T11:30:37.406 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:30:37.406 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:30:37.406 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:30:37.741 INFO:teuthology.orchestra.run.gibba002.stdout:ext2/ext3 2021-02-10T11:30:37.741 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2021-02-10T11:30:37.741 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:30:37.742 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:30:38.078 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:30:38.078 DEBUG:teuthology.orchestra.run.gibba013:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:30:38.078 DEBUG:teuthology.orchestra.run.gibba013:> ' -- /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:30:38.414 INFO:teuthology.orchestra.run.gibba013.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.2’: No such file or directory 2021-02-10T11:30:38.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:30:38.414 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2021-02-10T11:30:38.414 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:30:38.415 DEBUG:teuthology.orchestra.run.gibba013:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:30:38.674 DEBUG:teuthology.orchestra.run.gibba002:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:30:38.677 DEBUG:teuthology.orchestra.run.gibba013:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:30:38.749 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:30:38.749 DEBUG:teuthology.orchestra.run.gibba013:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:30:38.749 DEBUG:teuthology.orchestra.run.gibba013:> ' -- /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:30:39.012 INFO:teuthology.orchestra.run.gibba013.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.3’: No such file or directory 2021-02-10T11:30:39.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:30:39.013 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.3 2021-02-10T11:30:39.013 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:30:39.013 DEBUG:teuthology.orchestra.run.gibba013:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:30:39.348 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config rm mon mon_allow_pool_delete 2021-02-10T11:30:39.780 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:39.761 7f95ae11b700 1 Processor -- start 2021-02-10T11:30:39.780 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:39.761 7f95ae11b700 1 -- start start 2021-02-10T11:30:39.780 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:39.762 7f95ae11b700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f95a80d1870 0x7f95a81b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:39.780 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:39.762 7f95ae11b700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f95a81b0a10 0x7f95a80cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:39.780 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:39.762 7f95ae11b700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95a80f0b10 con 0x7f95a80d1c70 2021-02-10T11:30:39.780 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:39.762 7f95ae11b700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95a80fe230 con 0x7f95a81b0a10 2021-02-10T11:30:39.780 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:39.762 7f95ae11b700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95a80e5ed0 con 0x7f95a80d1870 2021-02-10T11:30:39.780 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:39.763 7f95a77fe700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f95a80d1870 0x7f95a81b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:39.780 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:39.763 7f95a6ffd700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f95a81b0a10 0x7f95a80cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:39.781 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:39.763 7f95a6ffd700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f95a81b0a10 0x7f95a80cf330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.113:3300/0 says I am v2:172.21.2.102:36402/0 (socket says 172.21.2.102:36402) 2021-02-10T11:30:39.781 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:39.763 7f95a77fe700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f95a80d1870 0x7f95a81b03f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.113:3301/0 says I am v2:172.21.2.102:38508/0 (socket says 172.21.2.102:38508) 2021-02-10T11:30:39.781 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:39.763 7f95a6ffd700 1 -- 172.21.2.102:0/3261225871 learned_addr learned my addr 172.21.2.102:0/3261225871 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:30:39.781 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:39.763 7f95a77fe700 1 -- 172.21.2.102:0/3261225871 >> v2:172.21.2.113:3300/0 conn(0x7f95a81b0a10 msgr2=0x7f95a80cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:39.781 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:39.763 7f95a77fe700 1 --2- 172.21.2.102:0/3261225871 >> v2:172.21.2.113:3300/0 conn(0x7f95a81b0a10 0x7f95a80cf330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:39.781 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:39.763 7f95a77fe700 1 -- 172.21.2.102:0/3261225871 >> v1:172.21.2.113:6789/0 conn(0x7f95a80d1c70 legacy=0x7f95a80cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:39.781 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:39.763 7f95a77fe700 1 -- 172.21.2.102:0/3261225871 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95a80f0650 con 0x7f95a80d1870 2021-02-10T11:30:39.781 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:39.763 7f95a67fc700 1 -- 172.21.2.102:0/3261225871 <== mon.1 v1:172.21.2.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4186020849 0 0) 0x7f95a0000f90 con 0x7f95a80d1c70 2021-02-10T11:30:39.781 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:39.763 7f95a6ffd700 1 --2- 172.21.2.102:0/3261225871 >> v2:172.21.2.113:3300/0 conn(0x7f95a81b0a10 0x7f95a80cf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:30:39.781 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:39.764 7f95a77fe700 1 --2- 172.21.2.102:0/3261225871 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f95a80d1870 0x7f95a81b03f0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7f9598006650 tx=0x7f9598009a80).ready entity=mon.2 client_cookie=2e5c199e87bc362b server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:39.781 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:39.764 7f95a67fc700 1 -- 172.21.2.102:0/3261225871 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f959800c0c0 con 0x7f95a80d1870 2021-02-10T11:30:39.781 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:39.764 7f95a67fc700 1 -- 172.21.2.102:0/3261225871 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f959800c340 con 0x7f95a80d1870 2021-02-10T11:30:39.781 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:39.764 7f95a67fc700 1 -- 172.21.2.102:0/3261225871 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f959800c860 con 0x7f95a80d1870 2021-02-10T11:30:39.782 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:39.764 7f95ae11b700 1 -- 172.21.2.102:0/3261225871 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f95a80d1870 msgr2=0x7f95a81b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:39.782 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:39.764 7f95ae11b700 1 --2- 172.21.2.102:0/3261225871 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f95a80d1870 0x7f95a81b03f0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7f9598006650 tx=0x7f9598009a80).stop 2021-02-10T11:30:39.782 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:39.765 7f95ae11b700 1 -- 172.21.2.102:0/3261225871 shutdown_connections 2021-02-10T11:30:39.782 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:39.765 7f95ae11b700 1 --2- 172.21.2.102:0/3261225871 >> v2:172.21.2.113:3300/0 conn(0x7f95a81b0a10 0x7f95a80cf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:39.782 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:39.765 7f95ae11b700 1 --2- 172.21.2.102:0/3261225871 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f95a80d1870 0x7f95a81b03f0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:39.782 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:39.765 7f95ae11b700 1 -- 172.21.2.102:0/3261225871 >> 172.21.2.102:0/3261225871 conn(0x7f95a80ce450 msgr2=0x7f95a80d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:39.782 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:39.765 7f95ae11b700 1 -- 172.21.2.102:0/3261225871 shutdown_connections 2021-02-10T11:30:39.782 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:39.765 7f95ae11b700 1 -- 172.21.2.102:0/3261225871 wait complete. 2021-02-10T11:30:39.782 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:39.779 7f95ae11b700 1 Processor -- start 2021-02-10T11:30:39.784 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:39.779 7f95ae11b700 1 -- start start 2021-02-10T11:30:39.784 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:39.780 7f95ae11b700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f95a8094310 0x7f95a80ebe70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:39.784 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:39.780 7f95ae11b700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f95a8094710 0x7f95a80bca30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:39.784 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:39.780 7f95ae11b700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f95a80ec490 0x7f95a80bd050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:39.784 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:39.780 7f95ae11b700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95a8093d90 con 0x7f95a8094310 2021-02-10T11:30:39.784 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:39.780 7f95ae11b700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95a80bd670 con 0x7f95a8094710 2021-02-10T11:30:39.784 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:39.780 7f95ae11b700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95a80939f0 con 0x7f95a80ec490 2021-02-10T11:30:39.784 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:39.780 7f95a77fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f95a8094310 0x7f95a80ebe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:39.784 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:39.780 7f95a77fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f95a8094310 0x7f95a80ebe70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:32978/0 (socket says 172.21.2.102:32978) 2021-02-10T11:30:39.785 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:39.780 7f95a77fe700 1 -- 172.21.2.102:0/4167125452 learned_addr learned my addr 172.21.2.102:0/4167125452 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:30:39.785 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:39.780 7f95a77fe700 1 -- 172.21.2.102:0/4167125452 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f95a80ec490 msgr2=0x7f95a80bd050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:39.785 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:39.780 7f95a77fe700 1 --2- 172.21.2.102:0/4167125452 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f95a80ec490 0x7f95a80bd050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:39.785 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:39.780 7f95a77fe700 1 -- 172.21.2.102:0/4167125452 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f95a8094710 msgr2=0x7f95a80bca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:39.785 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:39.780 7f95a77fe700 1 --2- 172.21.2.102:0/4167125452 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f95a8094710 0x7f95a80bca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:39.785 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:39.780 7f95a77fe700 1 -- 172.21.2.102:0/4167125452 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95a80bda80 con 0x7f95a8094310 2021-02-10T11:30:39.785 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:39.780 7f95a77fe700 1 --2- 172.21.2.102:0/4167125452 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f95a8094310 0x7f95a80ebe70 secure :-1 s=READY pgs=2013 cs=0 l=1 rev1=1 rx=0x7f9598009630 tx=0x7f959800b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:39.785 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:39.781 7f958ffff700 1 -- 172.21.2.102:0/4167125452 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f95980093a0 con 0x7f95a8094310 2021-02-10T11:30:39.785 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:39.781 7f958ffff700 1 -- 172.21.2.102:0/4167125452 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f959800bd70 con 0x7f95a8094310 2021-02-10T11:30:39.785 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:39.781 7f95ae11b700 1 -- 172.21.2.102:0/4167125452 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95a80bda80 con 0x7f95a8094310 2021-02-10T11:30:39.785 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:39.781 7f95ae11b700 1 -- 172.21.2.102:0/4167125452 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95a81a76d0 con 0x7f95a8094310 2021-02-10T11:30:39.785 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:39.781 7f958ffff700 1 -- 172.21.2.102:0/4167125452 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f95980054d0 con 0x7f95a8094310 2021-02-10T11:30:39.786 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:39.781 7f958ffff700 1 -- 172.21.2.102:0/4167125452 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f959800a000 con 0x7f95a8094310 2021-02-10T11:30:39.786 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:39.782 7f958ffff700 1 --2- 172.21.2.102:0/4167125452 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f95880186c0 0x7f958801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:39.786 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:39.782 7f958ffff700 1 -- 172.21.2.102:0/4167125452 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f9598032430 con 0x7f95a8094310 2021-02-10T11:30:39.786 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:39.782 7f95a6ffd700 1 --2- 172.21.2.102:0/4167125452 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f95880186c0 0x7f958801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:39.786 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:39.782 7f95a6ffd700 1 --2- 172.21.2.102:0/4167125452 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f95880186c0 0x7f958801abc0 crc :-1 s=READY pgs=898 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:39.812 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:39.812 7f95ae11b700 1 -- 172.21.2.102:0/4167125452 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95a80ebc40 con 0x7f95a8094310 2021-02-10T11:30:39.815 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:39.814 7f958ffff700 1 -- 172.21.2.102:0/4167125452 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f9598009340 con 0x7f95a8094310 2021-02-10T11:30:39.966 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:39.965 7f95ae11b700 1 -- 172.21.2.102:0/4167125452 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f95a80f0b10 con 0x7f95a8094310 2021-02-10T11:30:39.969 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:39.968 7f958ffff700 1 -- 172.21.2.102:0/4167125452 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v25) v1 ==== 104+0+0 (secure 0 0 0) 0x7f959800a000 con 0x7f95a8094310 2021-02-10T11:30:39.975 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:39.974 7f95ae11b700 1 -- 172.21.2.102:0/4167125452 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f95880186c0 msgr2=0x7f958801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:39.975 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:39.974 7f95ae11b700 1 --2- 172.21.2.102:0/4167125452 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f95880186c0 0x7f958801abc0 crc :-1 s=READY pgs=898 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:39.975 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:39.974 7f95ae11b700 1 -- 172.21.2.102:0/4167125452 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f95a8094310 msgr2=0x7f95a80ebe70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:39.975 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:39.974 7f95ae11b700 1 --2- 172.21.2.102:0/4167125452 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f95a8094310 0x7f95a80ebe70 secure :-1 s=READY pgs=2013 cs=0 l=1 rev1=1 rx=0x7f9598009630 tx=0x7f959800b980).stop 2021-02-10T11:30:39.975 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:39.974 7f95ae11b700 1 -- 172.21.2.102:0/4167125452 shutdown_connections 2021-02-10T11:30:39.976 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:39.974 7f95ae11b700 1 --2- 172.21.2.102:0/4167125452 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f95a80ec490 0x7f95a80bd050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:39.976 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:39.974 7f95ae11b700 1 --2- 172.21.2.102:0/4167125452 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f95880186c0 0x7f958801abc0 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:39.976 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:39.974 7f95ae11b700 1 --2- 172.21.2.102:0/4167125452 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f95a8094710 0x7f95a80bca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:39.976 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:39.974 7f95ae11b700 1 --2- 172.21.2.102:0/4167125452 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f95a8094310 0x7f95a80ebe70 unknown :-1 s=CLOSED pgs=2013 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:39.976 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:39.974 7f95ae11b700 1 -- 172.21.2.102:0/4167125452 >> 172.21.2.102:0/4167125452 conn(0x7f95a808fa10 msgr2=0x7f95a80bfa60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:39.976 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:39.974 7f95ae11b700 1 -- 172.21.2.102:0/4167125452 shutdown_connections 2021-02-10T11:30:39.976 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:39.975 7f95ae11b700 1 -- 172.21.2.102:0/4167125452 wait complete. 2021-02-10T11:30:39.984 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestVolumes.test_deauthorize_auth_id_after_out_of_band_update' 2021-02-10T11:30:40.411 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:40.394 7f89116cd700 1 Processor -- start 2021-02-10T11:30:40.411 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:40.394 7f89116cd700 1 -- start start 2021-02-10T11:30:40.411 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:40.394 7f89116cd700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f890c0d0340 0x7f890c0ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:40.411 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:40.394 7f89116cd700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f890c0d1c70 0x7f890c0efc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:40.412 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:40.394 7f89116cd700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f890c0f0b10 con 0x7f890c0d1870 2021-02-10T11:30:40.412 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:40.394 7f89116cd700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f890c0e5ed0 con 0x7f890c0d0340 2021-02-10T11:30:40.412 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:40.395 7f89116cd700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f890c1b0be0 con 0x7f890c0d1c70 2021-02-10T11:30:40.412 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:40.395 7f890aabe700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f890c0d0340 0x7f890c0ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:40.412 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:40.395 7f890aabe700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f890c0d0340 0x7f890c0ef640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:32984/0 (socket says 172.21.2.102:32984) 2021-02-10T11:30:40.412 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:40.395 7f890aabe700 1 -- 172.21.2.102:0/2852901949 learned_addr learned my addr 172.21.2.102:0/2852901949 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:30:40.412 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:40.395 7f890aabe700 1 -- 172.21.2.102:0/2852901949 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f890c0d1c70 msgr2=0x7f890c0efc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:40.412 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:40.395 7f890aabe700 1 --2- 172.21.2.102:0/2852901949 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f890c0d1c70 0x7f890c0efc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:40.412 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:40.395 7f890aabe700 1 -- 172.21.2.102:0/2852901949 >> v1:172.21.2.113:6789/0 conn(0x7f890c0d1870 legacy=0x7f890c1b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:40.412 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:40.395 7f890aabe700 1 -- 172.21.2.102:0/2852901949 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f890c0f20a0 con 0x7f890c0d0340 2021-02-10T11:30:40.412 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:40.395 7f890aabe700 1 --2- 172.21.2.102:0/2852901949 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f890c0d0340 0x7f890c0ef640 secure :-1 s=READY pgs=2014 cs=0 l=1 rev1=1 rx=0x7f88f4006700 tx=0x7f88f4009b80).ready entity=mon.0 client_cookie=b3d7cd0bea512f22 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:40.412 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:40.395 7f890a2bd700 1 -- 172.21.2.102:0/2852901949 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f88f400c0c0 con 0x7f890c0d0340 2021-02-10T11:30:40.413 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:40.395 7f890a2bd700 1 -- 172.21.2.102:0/2852901949 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88f400c440 con 0x7f890c0d0340 2021-02-10T11:30:40.413 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:40.395 7f890a2bd700 1 -- 172.21.2.102:0/2852901949 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f88f400c8a0 con 0x7f890c0d0340 2021-02-10T11:30:40.413 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:40.396 7f89116cd700 1 -- 172.21.2.102:0/2852901949 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f890c0d0340 msgr2=0x7f890c0ef640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:40.413 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:40.396 7f89116cd700 1 --2- 172.21.2.102:0/2852901949 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f890c0d0340 0x7f890c0ef640 secure :-1 s=READY pgs=2014 cs=0 l=1 rev1=1 rx=0x7f88f4006700 tx=0x7f88f4009b80).stop 2021-02-10T11:30:40.413 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:40.396 7f89116cd700 1 -- 172.21.2.102:0/2852901949 shutdown_connections 2021-02-10T11:30:40.413 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:40.396 7f89116cd700 1 --2- 172.21.2.102:0/2852901949 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f890c0d1c70 0x7f890c0efc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:40.413 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:40.396 7f89116cd700 1 --2- 172.21.2.102:0/2852901949 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f890c0d0340 0x7f890c0ef640 unknown :-1 s=CLOSED pgs=2014 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:40.413 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:40.396 7f89116cd700 1 -- 172.21.2.102:0/2852901949 >> 172.21.2.102:0/2852901949 conn(0x7f890c0ce450 msgr2=0x7f890c0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:40.413 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:40.396 7f89116cd700 1 -- 172.21.2.102:0/2852901949 shutdown_connections 2021-02-10T11:30:40.413 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:40.396 7f89116cd700 1 -- 172.21.2.102:0/2852901949 wait complete. 2021-02-10T11:30:40.415 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:40.411 7f89116cd700 1 Processor -- start 2021-02-10T11:30:40.415 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:40.411 7f89116cd700 1 -- start start 2021-02-10T11:30:40.415 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:40.411 7f89116cd700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f890c0b0ad0 0x7f890c091f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:40.415 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:40.411 7f89116cd700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f890c1a8850 0x7f890c1a8c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:40.415 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:40.411 7f89116cd700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f890c1a6fa0 0x7f890c1a73a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:40.415 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:40.411 7f89116cd700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f890c1abfe0 con 0x7f890c1a6fa0 2021-02-10T11:30:40.415 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:40.411 7f89116cd700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f890c1a9c40 con 0x7f890c0b0ad0 2021-02-10T11:30:40.415 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:40.411 7f89116cd700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f890c1a9730 con 0x7f890c1a8850 2021-02-10T11:30:40.415 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:40.411 7f890bac0700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f890c1a6fa0 0x7f890c1a73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:40.415 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:40.412 7f890bac0700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f890c1a6fa0 0x7f890c1a73a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:32990/0 (socket says 172.21.2.102:32990) 2021-02-10T11:30:40.416 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:40.412 7f890bac0700 1 -- 172.21.2.102:0/3932797013 learned_addr learned my addr 172.21.2.102:0/3932797013 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:30:40.416 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:40.412 7f890bac0700 1 -- 172.21.2.102:0/3932797013 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f890c1a8850 msgr2=0x7f890c1a8c50 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:30:40.416 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:40.412 7f890bac0700 1 --2- 172.21.2.102:0/3932797013 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f890c1a8850 0x7f890c1a8c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:40.416 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:40.412 7f890bac0700 1 -- 172.21.2.102:0/3932797013 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f890c0b0ad0 msgr2=0x7f890c091f30 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:30:40.416 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:40.412 7f890bac0700 1 --2- 172.21.2.102:0/3932797013 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f890c0b0ad0 0x7f890c091f30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:40.416 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:40.412 7f890bac0700 1 -- 172.21.2.102:0/3932797013 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f890c1a9270 con 0x7f890c1a6fa0 2021-02-10T11:30:40.416 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:40.412 7f890bac0700 1 --2- 172.21.2.102:0/3932797013 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f890c1a6fa0 0x7f890c1a73a0 secure :-1 s=READY pgs=2015 cs=0 l=1 rev1=1 rx=0x7f8904009980 tx=0x7f890400bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:40.416 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:40.412 7f8903fff700 1 -- 172.21.2.102:0/3932797013 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8904005870 con 0x7f890c1a6fa0 2021-02-10T11:30:40.416 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:40.412 7f8903fff700 1 -- 172.21.2.102:0/3932797013 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8904005b30 con 0x7f890c1a6fa0 2021-02-10T11:30:40.416 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:40.412 7f8903fff700 1 -- 172.21.2.102:0/3932797013 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8904006030 con 0x7f890c1a6fa0 2021-02-10T11:30:40.416 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:40.412 7f89116cd700 1 -- 172.21.2.102:0/3932797013 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f890c1a9270 con 0x7f890c1a6fa0 2021-02-10T11:30:40.416 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:40.412 7f89116cd700 1 -- 172.21.2.102:0/3932797013 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f890c1a7cb0 con 0x7f890c1a6fa0 2021-02-10T11:30:40.416 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:40.413 7f8903fff700 1 -- 172.21.2.102:0/3932797013 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f8904005d10 con 0x7f890c1a6fa0 2021-02-10T11:30:40.416 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:40.413 7f8903fff700 1 --2- 172.21.2.102:0/3932797013 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f88f00186c0 0x7f88f001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:40.417 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:40.413 7f8903fff700 1 -- 172.21.2.102:0/3932797013 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f8904032940 con 0x7f890c1a6fa0 2021-02-10T11:30:40.417 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:40.413 7f890b2bf700 1 --2- 172.21.2.102:0/3932797013 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f88f00186c0 0x7f88f001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:40.417 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:40.413 7f890b2bf700 1 --2- 172.21.2.102:0/3932797013 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f88f00186c0 0x7f88f001abc0 crc :-1 s=READY pgs=899 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:40.445 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:40.444 7f89116cd700 1 -- 172.21.2.102:0/3932797013 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f890c1a9710 con 0x7f890c1a6fa0 2021-02-10T11:30:40.447 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:40.447 7f8903fff700 1 -- 172.21.2.102:0/3932797013 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f89040062b0 con 0x7f890c1a6fa0 2021-02-10T11:30:40.580 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:40.579 7f89116cd700 1 -- 172.21.2.102:0/3932797013 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestVolumes.test_deauthorize_auth_id_after_out_of_band_update"]} v 0) v1 -- 0x7f890c1a9710 con 0x7f890c1a6fa0 2021-02-10T11:30:41.472 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:41.472 7f8903fff700 1 -- 172.21.2.102:0/3932797013 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestVolumes.test_deauthorize_auth_id_after_out_of_band_update"]}]=0 v524) v1 ==== 166+0+0 (secure 0 0 0) 0x7f8904006f20 con 0x7f890c1a6fa0 2021-02-10T11:30:41.503 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:41.502 7f89116cd700 1 -- 172.21.2.102:0/3932797013 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f88f00186c0 msgr2=0x7f88f001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:41.504 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:41.502 7f89116cd700 1 --2- 172.21.2.102:0/3932797013 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f88f00186c0 0x7f88f001abc0 crc :-1 s=READY pgs=899 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:41.504 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:41.502 7f89116cd700 1 -- 172.21.2.102:0/3932797013 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f890c1a6fa0 msgr2=0x7f890c1a73a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:41.504 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:41.502 7f89116cd700 1 --2- 172.21.2.102:0/3932797013 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f890c1a6fa0 0x7f890c1a73a0 secure :-1 s=READY pgs=2015 cs=0 l=1 rev1=1 rx=0x7f8904009980 tx=0x7f890400bcd0).stop 2021-02-10T11:30:41.504 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:41.502 7f89116cd700 1 -- 172.21.2.102:0/3932797013 shutdown_connections 2021-02-10T11:30:41.504 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:41.502 7f89116cd700 1 --2- 172.21.2.102:0/3932797013 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f890c1a6fa0 0x7f890c1a73a0 unknown :-1 s=CLOSED pgs=2015 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:41.504 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:41.502 7f89116cd700 1 --2- 172.21.2.102:0/3932797013 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f890c1a8850 0x7f890c1a8c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:41.504 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:41.502 7f89116cd700 1 --2- 172.21.2.102:0/3932797013 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f88f00186c0 0x7f88f001abc0 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:41.504 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:41.502 7f89116cd700 1 --2- 172.21.2.102:0/3932797013 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f890c0b0ad0 0x7f890c091f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:41.504 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:41.502 7f89116cd700 1 -- 172.21.2.102:0/3932797013 >> 172.21.2.102:0/3932797013 conn(0x7f890c19f8f0 msgr2=0x7f890c1a5f80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:41.504 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:41.503 7f89116cd700 1 -- 172.21.2.102:0/3932797013 shutdown_connections 2021-02-10T11:30:41.504 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:41.503 7f89116cd700 1 -- 172.21.2.102:0/3932797013 wait complete. 2021-02-10T11:30:41.512 INFO:tasks.cephfs_test_runner:test_deauthorize_auth_id_after_out_of_band_update (tasks.cephfs.test_volumes.TestVolumes) ... ok 2021-02-10T11:30:41.512 INFO:tasks.cephfs_test_runner:Starting test: test_default_uid_gid_subvolume (tasks.cephfs.test_volumes.TestVolumes) 2021-02-10T11:30:41.512 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestVolumes.test_default_uid_gid_subvolume' 2021-02-10T11:30:41.939 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:41.922 7f1c494b9700 1 Processor -- start 2021-02-10T11:30:41.939 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:41.922 7f1c494b9700 1 -- start start 2021-02-10T11:30:41.939 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:41.923 7f1c494b9700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1c440d0340 0x7f1c440ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:41.939 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:41.923 7f1c494b9700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f1c440d1c70 0x7f1c440efc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:41.940 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:41.923 7f1c494b9700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c440f0b10 con 0x7f1c440d1870 2021-02-10T11:30:41.940 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:41.923 7f1c494b9700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c440e5ed0 con 0x7f1c440d1c70 2021-02-10T11:30:41.940 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:41.923 7f1c494b9700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c441b0be0 con 0x7f1c440d0340 2021-02-10T11:30:41.940 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:41.923 7f1c43ac0700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f1c440d1c70 0x7f1c440efc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:41.940 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:41.923 7f1c43ac0700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f1c440d1c70 0x7f1c440efc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:33000/0 (socket says 172.21.2.102:33000) 2021-02-10T11:30:41.940 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:41.923 7f1c43ac0700 1 -- 172.21.2.102:0/2761718212 learned_addr learned my addr 172.21.2.102:0/2761718212 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:30:41.940 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:41.923 7f1c43ac0700 1 -- 172.21.2.102:0/2761718212 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1c440d0340 msgr2=0x7f1c440ef640 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:30:41.940 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:41.923 7f1c43ac0700 1 --2- 172.21.2.102:0/2761718212 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1c440d0340 0x7f1c440ef640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:41.940 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:41.923 7f1c43ac0700 1 -- 172.21.2.102:0/2761718212 >> v1:172.21.2.113:6789/0 conn(0x7f1c440d1870 legacy=0x7f1c441b03f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:30:41.940 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:41.923 7f1c43ac0700 1 -- 172.21.2.102:0/2761718212 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c440f20a0 con 0x7f1c440d1c70 2021-02-10T11:30:41.940 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:41.923 7f1c43ac0700 1 --2- 172.21.2.102:0/2761718212 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1c440d1c70 0x7f1c440efc60 secure :-1 s=READY pgs=2016 cs=0 l=1 rev1=1 rx=0x7f1c3c006f00 tx=0x7f1c3c009f10).ready entity=mon.0 client_cookie=b6423b29936e1ea3 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:41.940 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:41.923 7f1c422bd700 1 -- 172.21.2.102:0/2761718212 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1c3c00b3b0 con 0x7f1c440d1c70 2021-02-10T11:30:41.940 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:41.923 7f1c422bd700 1 -- 172.21.2.102:0/2761718212 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c3c00b630 con 0x7f1c440d1c70 2021-02-10T11:30:41.941 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:41.923 7f1c422bd700 1 -- 172.21.2.102:0/2761718212 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1c3c00bb50 con 0x7f1c440d1c70 2021-02-10T11:30:41.941 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:41.923 7f1c494b9700 1 -- 172.21.2.102:0/2761718212 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1c440d1c70 msgr2=0x7f1c440efc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:41.941 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:41.923 7f1c494b9700 1 --2- 172.21.2.102:0/2761718212 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1c440d1c70 0x7f1c440efc60 secure :-1 s=READY pgs=2016 cs=0 l=1 rev1=1 rx=0x7f1c3c006f00 tx=0x7f1c3c009f10).stop 2021-02-10T11:30:41.941 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:41.924 7f1c494b9700 1 -- 172.21.2.102:0/2761718212 shutdown_connections 2021-02-10T11:30:41.941 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:41.924 7f1c494b9700 1 --2- 172.21.2.102:0/2761718212 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1c440d0340 0x7f1c440ef640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:41.941 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:41.924 7f1c494b9700 1 --2- 172.21.2.102:0/2761718212 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1c440d1c70 0x7f1c440efc60 unknown :-1 s=CLOSED pgs=2016 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:41.941 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:41.924 7f1c494b9700 1 -- 172.21.2.102:0/2761718212 >> 172.21.2.102:0/2761718212 conn(0x7f1c440ce450 msgr2=0x7f1c440d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:41.941 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:41.924 7f1c494b9700 1 -- 172.21.2.102:0/2761718212 shutdown_connections 2021-02-10T11:30:41.941 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:41.924 7f1c494b9700 1 -- 172.21.2.102:0/2761718212 wait complete. 2021-02-10T11:30:41.941 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:41.938 7f1c494b9700 1 Processor -- start 2021-02-10T11:30:41.943 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:41.939 7f1c494b9700 1 -- start start 2021-02-10T11:30:41.943 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:41.939 7f1c494b9700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1c441a8cd0 0x7f1c44093740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:41.943 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:41.939 7f1c494b9700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1c4418ecc0 0x7f1c441949e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:41.943 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:41.939 7f1c494b9700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1c4418f0c0 0x7f1c44195000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:41.943 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:41.939 7f1c494b9700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c44093f60 con 0x7f1c4418ecc0 2021-02-10T11:30:41.943 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:41.939 7f1c494b9700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c441a8830 con 0x7f1c4418f0c0 2021-02-10T11:30:41.943 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:41.939 7f1c494b9700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c441a8a10 con 0x7f1c441a8cd0 2021-02-10T11:30:41.943 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:41.939 7f1c42abe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1c4418ecc0 0x7f1c441949e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:41.943 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:41.939 7f1c42abe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1c4418ecc0 0x7f1c441949e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:33002/0 (socket says 172.21.2.102:33002) 2021-02-10T11:30:41.943 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:41.939 7f1c42abe700 1 -- 172.21.2.102:0/120412091 learned_addr learned my addr 172.21.2.102:0/120412091 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:30:41.944 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:41.939 7f1c42abe700 1 -- 172.21.2.102:0/120412091 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1c441a8cd0 msgr2=0x7f1c44093740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:41.944 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:41.939 7f1c42abe700 1 --2- 172.21.2.102:0/120412091 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1c441a8cd0 0x7f1c44093740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:41.944 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:41.939 7f1c42abe700 1 -- 172.21.2.102:0/120412091 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1c4418f0c0 msgr2=0x7f1c44195000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:41.944 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:41.940 7f1c42abe700 1 --2- 172.21.2.102:0/120412091 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1c4418f0c0 0x7f1c44195000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:41.944 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:41.940 7f1c42abe700 1 -- 172.21.2.102:0/120412091 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c441a90d0 con 0x7f1c4418ecc0 2021-02-10T11:30:41.944 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:41.940 7f1c42abe700 1 --2- 172.21.2.102:0/120412091 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1c4418ecc0 0x7f1c441949e0 secure :-1 s=READY pgs=2017 cs=0 l=1 rev1=1 rx=0x7f1c2c00b560 tx=0x7f1c2c00b900).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:41.944 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:41.940 7f1c3bfff700 1 -- 172.21.2.102:0/120412091 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1c2c000c40 con 0x7f1c4418ecc0 2021-02-10T11:30:41.944 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:41.940 7f1c3bfff700 1 -- 172.21.2.102:0/120412091 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c2c00bcf0 con 0x7f1c4418ecc0 2021-02-10T11:30:41.944 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:41.940 7f1c3bfff700 1 -- 172.21.2.102:0/120412091 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1c2c0054d0 con 0x7f1c4418ecc0 2021-02-10T11:30:41.944 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:41.940 7f1c494b9700 1 -- 172.21.2.102:0/120412091 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c441a90d0 con 0x7f1c4418ecc0 2021-02-10T11:30:41.944 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:41.940 7f1c494b9700 1 -- 172.21.2.102:0/120412091 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c441a90d0 con 0x7f1c4418ecc0 2021-02-10T11:30:41.944 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:41.941 7f1c3bfff700 1 -- 172.21.2.102:0/120412091 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1c2c006030 con 0x7f1c4418ecc0 2021-02-10T11:30:41.945 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:41.941 7f1c3bfff700 1 --2- 172.21.2.102:0/120412091 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1c280186c0 0x7f1c2801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:41.945 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:41.941 7f1c3bfff700 1 -- 172.21.2.102:0/120412091 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f1c2c0327b0 con 0x7f1c4418ecc0 2021-02-10T11:30:41.945 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:41.941 7f1c432bf700 1 --2- 172.21.2.102:0/120412091 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1c280186c0 0x7f1c2801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:41.945 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:41.941 7f1c432bf700 1 --2- 172.21.2.102:0/120412091 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1c280186c0 0x7f1c2801abc0 crc :-1 s=READY pgs=900 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:41.973 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:41.972 7f1c494b9700 1 -- 172.21.2.102:0/120412091 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c440f0b10 con 0x7f1c4418ecc0 2021-02-10T11:30:41.976 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:41.975 7f1c3bfff700 1 -- 172.21.2.102:0/120412091 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f1c2c000c40 con 0x7f1c4418ecc0 2021-02-10T11:30:42.108 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:42.107 7f1c494b9700 1 -- 172.21.2.102:0/120412091 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestVolumes.test_default_uid_gid_subvolume"]} v 0) v1 -- 0x7f1c440f0b10 con 0x7f1c4418ecc0 2021-02-10T11:30:42.477 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:42.476 7f1c3bfff700 1 -- 172.21.2.102:0/120412091 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestVolumes.test_default_uid_gid_subvolume"]}]=0 v525) v1 ==== 150+0+0 (secure 0 0 0) 0x7f1c2c01f4f0 con 0x7f1c4418ecc0 2021-02-10T11:30:42.526 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:42.524 7f1c494b9700 1 -- 172.21.2.102:0/120412091 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1c280186c0 msgr2=0x7f1c2801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:42.526 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:42.524 7f1c494b9700 1 --2- 172.21.2.102:0/120412091 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1c280186c0 0x7f1c2801abc0 crc :-1 s=READY pgs=900 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:42.526 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:42.524 7f1c494b9700 1 -- 172.21.2.102:0/120412091 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1c4418ecc0 msgr2=0x7f1c441949e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:42.526 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:42.524 7f1c494b9700 1 --2- 172.21.2.102:0/120412091 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1c4418ecc0 0x7f1c441949e0 secure :-1 s=READY pgs=2017 cs=0 l=1 rev1=1 rx=0x7f1c2c00b560 tx=0x7f1c2c00b900).stop 2021-02-10T11:30:42.527 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:42.524 7f1c494b9700 1 -- 172.21.2.102:0/120412091 shutdown_connections 2021-02-10T11:30:42.527 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:42.524 7f1c494b9700 1 --2- 172.21.2.102:0/120412091 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1c280186c0 0x7f1c2801abc0 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:42.527 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:42.524 7f1c494b9700 1 --2- 172.21.2.102:0/120412091 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1c4418f0c0 0x7f1c44195000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:42.527 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:42.524 7f1c494b9700 1 --2- 172.21.2.102:0/120412091 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1c4418ecc0 0x7f1c441949e0 unknown :-1 s=CLOSED pgs=2017 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:42.527 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:42.525 7f1c494b9700 1 --2- 172.21.2.102:0/120412091 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1c441a8cd0 0x7f1c44093740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:42.527 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:42.525 7f1c494b9700 1 -- 172.21.2.102:0/120412091 >> 172.21.2.102:0/120412091 conn(0x7f1c441a38b0 msgr2=0x7f1c441a2f20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:42.527 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:42.525 7f1c494b9700 1 -- 172.21.2.102:0/120412091 shutdown_connections 2021-02-10T11:30:42.527 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:42.525 7f1c494b9700 1 -- 172.21.2.102:0/120412091 wait complete. 2021-02-10T11:30:42.534 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config set mon mon_allow_pool_delete True 2021-02-10T11:30:42.967 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:42.948 7fbbbf03b700 1 Processor -- start 2021-02-10T11:30:42.967 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:42.948 7fbbbf03b700 1 -- start start 2021-02-10T11:30:42.967 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:42.949 7fbbbf03b700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fbbb810aff0 0x7fbbb80cf750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:42.967 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:42.949 7fbbbf03b700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbb80fe1c0 con 0x7fbbb80d1870 2021-02-10T11:30:42.967 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:42.949 7fbbbf03b700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbb81b0cb0 con 0x7fbbb80d1c70 2021-02-10T11:30:42.968 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:42.949 7fbbbf03b700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbbb8109a60 con 0x7fbbb810aff0 2021-02-10T11:30:42.968 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:42.949 7fbbbce57700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fbbb80d1870 0x7fbbb81b04c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:53466/0 (socket says 172.21.2.102:53466) 2021-02-10T11:30:42.968 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:42.949 7fbbbce57700 1 -- 172.21.2.102:0/4096979860 learned_addr learned my addr 172.21.2.102:0/4096979860 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:30:42.968 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:42.949 7fbbb77fe700 1 -- 172.21.2.102:0/4096979860 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3443105301 0 0) 0x7fbba0001180 con 0x7fbbb80d1870 2021-02-10T11:30:42.968 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:42.949 7fbbb77fe700 1 -- 172.21.2.102:0/4096979860 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbba4002fe0 con 0x7fbbb80d1870 2021-02-10T11:30:42.968 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:42.949 7fbbbd658700 1 --2- 172.21.2.102:0/4096979860 >> v2:172.21.2.113:3300/0 conn(0x7fbbb810aff0 0x7fbbb80cf750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:42.968 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:42.950 7fbbb77fe700 1 -- 172.21.2.102:0/4096979860 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1536253709 0 0) 0x7fbba00010e0 con 0x7fbbb80d1870 2021-02-10T11:30:42.968 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:42.950 7fbbb77fe700 1 -- 172.21.2.102:0/4096979860 >> v2:172.21.2.113:3300/0 conn(0x7fbbb810aff0 msgr2=0x7fbbb80cf750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:42.968 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:42.950 7fbbb77fe700 1 --2- 172.21.2.102:0/4096979860 >> v2:172.21.2.113:3300/0 conn(0x7fbbb810aff0 0x7fbbb80cf750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:42.968 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:42.950 7fbbb77fe700 1 -- 172.21.2.102:0/4096979860 >> v1:172.21.2.113:6789/0 conn(0x7fbbb80d1c70 legacy=0x7fbbb80cef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:42.969 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:42.950 7fbbb77fe700 1 -- 172.21.2.102:0/4096979860 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbb810b3f0 con 0x7fbbb80d1870 2021-02-10T11:30:42.969 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:42.950 7fbbb77fe700 1 -- 172.21.2.102:0/4096979860 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7fbba0001600 con 0x7fbbb80d1870 2021-02-10T11:30:42.969 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:42.950 7fbbb77fe700 1 -- 172.21.2.102:0/4096979860 >> v1:172.21.2.102:6789/0 conn(0x7fbbb80d1870 legacy=0x7fbbb81b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:42.969 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:42.950 7fbbb77fe700 1 --2- 172.21.2.102:0/4096979860 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbba4003ac0 0x7fbba4005ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:42.969 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:42.950 7fbbb77fe700 1 --2- 172.21.2.102:0/4096979860 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbba4006770 0x7fbba4008c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:42.969 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:42.950 7fbbb77fe700 1 --2- 172.21.2.102:0/4096979860 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fbba4009590 0x7fbba400bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:42.969 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:42.950 7fbbb77fe700 1 -- 172.21.2.102:0/4096979860 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbba400c300 con 0x7fbba4003ac0 2021-02-10T11:30:42.969 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:42.950 7fbbb77fe700 1 -- 172.21.2.102:0/4096979860 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbba400c520 con 0x7fbba4009590 2021-02-10T11:30:42.969 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:42.950 7fbbb77fe700 1 -- 172.21.2.102:0/4096979860 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbba400c740 con 0x7fbba4006770 2021-02-10T11:30:42.969 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:42.950 7fbbbce57700 1 --2- 172.21.2.102:0/4096979860 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbba4003ac0 0x7fbba4005ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:42.969 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:42.950 7fbbbd658700 1 --2- 172.21.2.102:0/4096979860 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fbba4009590 0x7fbba400bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:42.970 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:42.950 7fbbb7fff700 1 --2- 172.21.2.102:0/4096979860 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbba4006770 0x7fbba4008c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:42.970 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:42.950 7fbbbce57700 1 -- 172.21.2.102:0/4096979860 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbba4006770 msgr2=0x7fbba4008c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:42.970 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:42.950 7fbbbce57700 1 --2- 172.21.2.102:0/4096979860 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbba4006770 0x7fbba4008c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:42.970 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:42.950 7fbbbce57700 1 -- 172.21.2.102:0/4096979860 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fbba4009590 msgr2=0x7fbba400bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:42.970 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:42.950 7fbbbce57700 1 --2- 172.21.2.102:0/4096979860 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fbba4009590 0x7fbba400bab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:42.970 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:42.950 7fbbbce57700 1 -- 172.21.2.102:0/4096979860 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbba40509b0 con 0x7fbba4003ac0 2021-02-10T11:30:42.970 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:42.950 7fbbbd658700 1 --2- 172.21.2.102:0/4096979860 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fbba4009590 0x7fbba400bab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:30:42.970 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:42.950 7fbbbd658700 1 -- 172.21.2.102:0/4096979860 reap_dead start 2021-02-10T11:30:42.970 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:42.950 7fbbb7fff700 1 --2- 172.21.2.102:0/4096979860 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbba4006770 0x7fbba4008c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:30:42.970 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:42.950 7fbbbce57700 1 --2- 172.21.2.102:0/4096979860 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbba4003ac0 0x7fbba4005ef0 secure :-1 s=READY pgs=2019 cs=0 l=1 rev1=1 rx=0x7fbba0000b60 tx=0x7fbba0007d10).ready entity=mon.0 client_cookie=bb793a83b9ff43a6 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:42.970 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:42.950 7fbbb77fe700 1 -- 172.21.2.102:0/4096979860 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbba00015e0 con 0x7fbba4003ac0 2021-02-10T11:30:42.970 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:42.950 7fbbb77fe700 1 -- 172.21.2.102:0/4096979860 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbba00070c0 con 0x7fbba4003ac0 2021-02-10T11:30:42.971 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:42.950 7fbbb77fe700 1 -- 172.21.2.102:0/4096979860 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbba0004620 con 0x7fbba4003ac0 2021-02-10T11:30:42.971 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:42.950 7fbbbf03b700 1 -- 172.21.2.102:0/4096979860 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbba4003ac0 msgr2=0x7fbba4005ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:42.971 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:42.950 7fbbbf03b700 1 --2- 172.21.2.102:0/4096979860 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbba4003ac0 0x7fbba4005ef0 secure :-1 s=READY pgs=2019 cs=0 l=1 rev1=1 rx=0x7fbba0000b60 tx=0x7fbba0007d10).stop 2021-02-10T11:30:42.971 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:42.951 7fbbbf03b700 1 -- 172.21.2.102:0/4096979860 shutdown_connections 2021-02-10T11:30:42.971 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:42.951 7fbbbf03b700 1 --2- 172.21.2.102:0/4096979860 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbba4003ac0 0x7fbba4005ef0 unknown :-1 s=CLOSED pgs=2019 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:42.971 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:42.951 7fbbbf03b700 1 -- 172.21.2.102:0/4096979860 >> 172.21.2.102:0/4096979860 conn(0x7fbbb80ce420 msgr2=0x7fbbb80d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:42.971 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:42.951 7fbbbf03b700 1 -- 172.21.2.102:0/4096979860 shutdown_connections 2021-02-10T11:30:42.971 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:42.951 7fbbbf03b700 1 -- 172.21.2.102:0/4096979860 wait complete. 2021-02-10T11:30:42.971 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:42.966 7fbbbf03b700 1 Processor -- start 2021-02-10T11:30:42.971 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:42.966 7fbbbf03b700 1 -- start start 2021-02-10T11:30:42.973 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:42.966 7fbbbf03b700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbbb8085bc0 0x7fbbb80863e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:42.973 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:42.966 7fbbbf03b700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbbb80954c0 0x7fbbb8096300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:42.973 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:42.966 7fbbbf03b700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fbbb80958c0 0x7fbbb8096920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:42.973 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:42.966 7fbbbf03b700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbb8095d80 con 0x7fbbb8085bc0 2021-02-10T11:30:42.973 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:42.966 7fbbbf03b700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbb80bfa60 con 0x7fbbb80958c0 2021-02-10T11:30:42.973 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:42.966 7fbbbf03b700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbbb8085fc0 con 0x7fbbb80954c0 2021-02-10T11:30:42.974 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:42.967 7fbbbce57700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbbb8085bc0 0x7fbbb80863e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:42.974 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:42.967 7fbbbce57700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbbb8085bc0 0x7fbbb80863e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:33026/0 (socket says 172.21.2.102:33026) 2021-02-10T11:30:42.974 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:42.967 7fbbbce57700 1 -- 172.21.2.102:0/1900643727 learned_addr learned my addr 172.21.2.102:0/1900643727 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:30:42.974 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:42.967 7fbbbd658700 1 --2- 172.21.2.102:0/1900643727 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fbbb80958c0 0x7fbbb8096920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:42.974 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:42.967 7fbbb7fff700 1 --2- 172.21.2.102:0/1900643727 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbbb80954c0 0x7fbbb8096300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:42.974 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:42.967 7fbbbd658700 1 -- 172.21.2.102:0/1900643727 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbbb80954c0 msgr2=0x7fbbb8096300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:42.974 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:42.967 7fbbbd658700 1 --2- 172.21.2.102:0/1900643727 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbbb80954c0 0x7fbbb8096300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:42.974 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:42.967 7fbbbd658700 1 -- 172.21.2.102:0/1900643727 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbbb8085bc0 msgr2=0x7fbbb80863e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:42.974 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:42.967 7fbbbd658700 1 --2- 172.21.2.102:0/1900643727 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbbb8085bc0 0x7fbbb80863e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:42.974 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:42.967 7fbbbd658700 1 -- 172.21.2.102:0/1900643727 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbb8086c10 con 0x7fbbb80958c0 2021-02-10T11:30:42.975 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:42.967 7fbbbd658700 1 --2- 172.21.2.102:0/1900643727 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fbbb80958c0 0x7fbbb8096920 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 rx=0x7fbbb0007de0 tx=0x7fbbb000ac30).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:42.975 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:42.967 7fbbb57fa700 1 -- 172.21.2.102:0/1900643727 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbbb000ad70 con 0x7fbbb80958c0 2021-02-10T11:30:42.975 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:42.968 7fbbb57fa700 1 -- 172.21.2.102:0/1900643727 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbb0007e20 con 0x7fbbb80958c0 2021-02-10T11:30:42.975 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:42.968 7fbbb57fa700 1 -- 172.21.2.102:0/1900643727 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbbb0000f90 con 0x7fbbb80958c0 2021-02-10T11:30:42.975 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:42.968 7fbbbf03b700 1 -- 172.21.2.102:0/1900643727 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbbb8086c10 con 0x7fbbb80958c0 2021-02-10T11:30:42.975 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:42.968 7fbbbf03b700 1 -- 172.21.2.102:0/1900643727 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbb8086c10 con 0x7fbbb80958c0 2021-02-10T11:30:42.975 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:42.968 7fbbb57fa700 1 -- 172.21.2.102:0/1900643727 <== mon.1 v2:172.21.2.113:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fbbb00011f0 con 0x7fbbb80958c0 2021-02-10T11:30:42.975 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:42.968 7fbbb57fa700 1 --2- 172.21.2.102:0/1900643727 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fbb9c018730 0x7fbb9c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:42.975 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:42.968 7fbbb57fa700 1 -- 172.21.2.102:0/1900643727 <== mon.1 v2:172.21.2.113:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fbbb00328d0 con 0x7fbbb80958c0 2021-02-10T11:30:42.975 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:42.968 7fbbbce57700 1 --2- 172.21.2.102:0/1900643727 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fbb9c018730 0x7fbb9c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:42.975 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:42.969 7fbbbce57700 1 --2- 172.21.2.102:0/1900643727 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fbb9c018730 0x7fbb9c01abc0 crc :-1 s=READY pgs=901 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:42.999 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:42.998 7fbbbf03b700 1 -- 172.21.2.102:0/1900643727 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbbb80fe1c0 con 0x7fbbb80958c0 2021-02-10T11:30:43.002 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:43.001 7fbbb57fa700 1 -- 172.21.2.102:0/1900643727 <== mon.1 v2:172.21.2.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fbbb0001ef0 con 0x7fbbb80958c0 2021-02-10T11:30:43.150 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:43.149 7fbbbf03b700 1 -- 172.21.2.102:0/1900643727 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fbbb80fe1c0 con 0x7fbbb80958c0 2021-02-10T11:30:43.156 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:43.155 7fbbb57fa700 1 -- 172.21.2.102:0/1900643727 <== mon.1 v2:172.21.2.113:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v26)=0 v26) v1 ==== 122+0+0 (secure 0 0 0) 0x7fbbb00011f0 con 0x7fbbb80958c0 2021-02-10T11:30:43.159 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:43.158 7fbbbf03b700 1 -- 172.21.2.102:0/1900643727 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fbb9c018730 msgr2=0x7fbb9c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:43.159 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:43.158 7fbbbf03b700 1 --2- 172.21.2.102:0/1900643727 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fbb9c018730 0x7fbb9c01abc0 crc :-1 s=READY pgs=901 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:43.159 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:43.158 7fbbbf03b700 1 -- 172.21.2.102:0/1900643727 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fbbb80958c0 msgr2=0x7fbbb8096920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:43.159 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:43.158 7fbbbf03b700 1 --2- 172.21.2.102:0/1900643727 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fbbb80958c0 0x7fbbb8096920 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 rx=0x7fbbb0007de0 tx=0x7fbbb000ac30).stop 2021-02-10T11:30:43.160 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:43.159 7fbbbf03b700 1 -- 172.21.2.102:0/1900643727 shutdown_connections 2021-02-10T11:30:43.160 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:43.159 7fbbbf03b700 1 --2- 172.21.2.102:0/1900643727 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fbb9c018730 0x7fbb9c01abc0 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:43.160 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:43.159 7fbbbf03b700 1 --2- 172.21.2.102:0/1900643727 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fbbb80958c0 0x7fbbb8096920 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:43.160 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:43.159 7fbbbf03b700 1 --2- 172.21.2.102:0/1900643727 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbbb80954c0 0x7fbbb8096300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:43.160 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:43.159 7fbbbf03b700 1 --2- 172.21.2.102:0/1900643727 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbbb8085bc0 0x7fbbb80863e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:43.160 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:43.159 7fbbbf03b700 1 -- 172.21.2.102:0/1900643727 >> 172.21.2.102:0/1900643727 conn(0x7fbbb80bef00 msgr2=0x7fbbb809d1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:43.160 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:43.159 7fbbbf03b700 1 -- 172.21.2.102:0/1900643727 shutdown_connections 2021-02-10T11:30:43.160 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:43.159 7fbbbf03b700 1 -- 172.21.2.102:0/1900643727 wait complete. 2021-02-10T11:30:43.168 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2021-02-10T11:30:43.168 DEBUG:teuthology.orchestra.run.gibba002:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:30:43.513 DEBUG:teuthology.orchestra.run.gibba013:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:30:43.859 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2021-02-10T11:30:43.859 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:30:43.859 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:30:43.859 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:30:44.194 INFO:teuthology.orchestra.run.gibba002.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.0’: No such file or directory 2021-02-10T11:30:44.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:30:44.194 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.0 2021-02-10T11:30:44.195 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:30:44.195 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:30:44.195 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:30:44.530 INFO:teuthology.orchestra.run.gibba002.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.1’: No such file or directory 2021-02-10T11:30:44.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:30:44.531 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2021-02-10T11:30:44.531 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:30:44.531 DEBUG:teuthology.orchestra.run.gibba013:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:30:44.531 DEBUG:teuthology.orchestra.run.gibba013:> ' -- /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:30:44.865 INFO:teuthology.orchestra.run.gibba013.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.2’: No such file or directory 2021-02-10T11:30:44.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:30:44.866 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2021-02-10T11:30:44.866 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:30:44.866 DEBUG:teuthology.orchestra.run.gibba013:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:30:44.866 DEBUG:teuthology.orchestra.run.gibba013:> ' -- /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:30:45.201 INFO:teuthology.orchestra.run.gibba013.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.3’: No such file or directory 2021-02-10T11:30:45.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:30:45.202 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.3 2021-02-10T11:30:45.202 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json-pretty 2021-02-10T11:30:45.629 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:45.613 7fb3c8bd7700 1 Processor -- start 2021-02-10T11:30:45.630 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:45.613 7fb3c8bd7700 1 -- start start 2021-02-10T11:30:45.630 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:45.613 7fb3c8bd7700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fb3c40d0340 0x7fb3c40ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:45.630 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:45.613 7fb3c8bd7700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb3c40d1c70 0x7fb3c40efc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:45.630 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:45.613 7fb3c8bd7700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3c40f0b10 con 0x7fb3c40d1870 2021-02-10T11:30:45.630 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:45.613 7fb3c8bd7700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3c40e5ed0 con 0x7fb3c40d0340 2021-02-10T11:30:45.630 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:45.613 7fb3c8bd7700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3c41b0be0 con 0x7fb3c40d1c70 2021-02-10T11:30:45.630 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:45.613 7fb3c205d700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fb3c40d0340 0x7fb3c40ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:45.630 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:45.613 7fb3c205d700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fb3c40d0340 0x7fb3c40ef640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:33032/0 (socket says 172.21.2.102:33032) 2021-02-10T11:30:45.630 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:45.613 7fb3c205d700 1 -- 172.21.2.102:0/3217665910 learned_addr learned my addr 172.21.2.102:0/3217665910 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:30:45.630 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:45.614 7fb3c205d700 1 -- 172.21.2.102:0/3217665910 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb3c40d1c70 msgr2=0x7fb3c40efc60 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:30:45.630 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:45.614 7fb3c205d700 1 --2- 172.21.2.102:0/3217665910 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb3c40d1c70 0x7fb3c40efc60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:45.630 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:45.614 7fb3c205d700 1 -- 172.21.2.102:0/3217665910 >> v1:172.21.2.102:6789/0 conn(0x7fb3c40d1870 legacy=0x7fb3c41b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:45.630 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:45.614 7fb3c205d700 1 -- 172.21.2.102:0/3217665910 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3c40f20a0 con 0x7fb3c40d0340 2021-02-10T11:30:45.631 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:45.614 7fb3c205d700 1 --2- 172.21.2.102:0/3217665910 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb3c40d0340 0x7fb3c40ef640 secure :-1 s=READY pgs=2021 cs=0 l=1 rev1=1 rx=0x7fb3ac006700 tx=0x7fb3ac009b80).ready entity=mon.0 client_cookie=dfb60a8cd208a9b8 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:45.631 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:45.614 7fb3c185c700 1 -- 172.21.2.102:0/3217665910 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb3ac00c0c0 con 0x7fb3c40d0340 2021-02-10T11:30:45.631 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:45.614 7fb3c185c700 1 -- 172.21.2.102:0/3217665910 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3ac00c340 con 0x7fb3c40d0340 2021-02-10T11:30:45.631 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:45.614 7fb3c185c700 1 -- 172.21.2.102:0/3217665910 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb3ac00c820 con 0x7fb3c40d0340 2021-02-10T11:30:45.631 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:45.614 7fb3c8bd7700 1 -- 172.21.2.102:0/3217665910 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb3c40d0340 msgr2=0x7fb3c40ef640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:45.631 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:45.614 7fb3c8bd7700 1 --2- 172.21.2.102:0/3217665910 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb3c40d0340 0x7fb3c40ef640 secure :-1 s=READY pgs=2021 cs=0 l=1 rev1=1 rx=0x7fb3ac006700 tx=0x7fb3ac009b80).stop 2021-02-10T11:30:45.631 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:45.615 7fb3c8bd7700 1 -- 172.21.2.102:0/3217665910 shutdown_connections 2021-02-10T11:30:45.631 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:45.615 7fb3c8bd7700 1 --2- 172.21.2.102:0/3217665910 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb3c40d0340 0x7fb3c40ef640 unknown :-1 s=CLOSED pgs=2021 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:45.631 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:45.615 7fb3c8bd7700 1 --2- 172.21.2.102:0/3217665910 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb3c40d1c70 0x7fb3c40efc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:45.631 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:45.615 7fb3c8bd7700 1 -- 172.21.2.102:0/3217665910 >> 172.21.2.102:0/3217665910 conn(0x7fb3c40ce450 msgr2=0x7fb3c40d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:45.631 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:45.615 7fb3c8bd7700 1 -- 172.21.2.102:0/3217665910 shutdown_connections 2021-02-10T11:30:45.631 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:45.615 7fb3c8bd7700 1 -- 172.21.2.102:0/3217665910 wait complete. 2021-02-10T11:30:45.633 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:45.629 7fb3c8bd7700 1 Processor -- start 2021-02-10T11:30:45.633 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:45.629 7fb3c8bd7700 1 -- start start 2021-02-10T11:30:45.633 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:45.629 7fb3c8bd7700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb3c40b4f30 0x7fb3c418b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:45.633 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:45.629 7fb3c8bd7700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb3c4093f40 0x7fb3c4094340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:45.633 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:45.629 7fb3c8bd7700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb3c4094960 0x7fb3c4187e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:45.633 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:45.629 7fb3c8bd7700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3c4094e90 con 0x7fb3c4094960 2021-02-10T11:30:45.633 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:45.629 7fb3c8bd7700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3c418bff0 con 0x7fb3c4093f40 2021-02-10T11:30:45.633 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:45.629 7fb3c8bd7700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3c408f780 con 0x7fb3c40b4f30 2021-02-10T11:30:45.633 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:45.630 7fb3c305f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb3c4094960 0x7fb3c4187e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:45.633 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:45.630 7fb3c305f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb3c4094960 0x7fb3c4187e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:33036/0 (socket says 172.21.2.102:33036) 2021-02-10T11:30:45.634 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:45.630 7fb3c305f700 1 -- 172.21.2.102:0/3853670057 learned_addr learned my addr 172.21.2.102:0/3853670057 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:30:45.634 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:45.630 7fb3c305f700 1 -- 172.21.2.102:0/3853670057 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb3c40b4f30 msgr2=0x7fb3c418b9d0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:30:45.634 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:45.630 7fb3c305f700 1 --2- 172.21.2.102:0/3853670057 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb3c40b4f30 0x7fb3c418b9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:45.634 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:45.630 7fb3c305f700 1 -- 172.21.2.102:0/3853670057 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb3c4093f40 msgr2=0x7fb3c4094340 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:30:45.634 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:45.630 7fb3c305f700 1 --2- 172.21.2.102:0/3853670057 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb3c4093f40 0x7fb3c4094340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:45.634 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:45.630 7fb3c305f700 1 -- 172.21.2.102:0/3853670057 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3c4188470 con 0x7fb3c4094960 2021-02-10T11:30:45.634 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:45.630 7fb3c305f700 1 --2- 172.21.2.102:0/3853670057 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb3c4094960 0x7fb3c4187e50 secure :-1 s=READY pgs=2022 cs=0 l=1 rev1=1 rx=0x7fb3bc009960 tx=0x7fb3bc00bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:45.634 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:45.630 7fb3bb7fe700 1 -- 172.21.2.102:0/3853670057 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb3bc003030 con 0x7fb3c4094960 2021-02-10T11:30:45.634 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:45.630 7fb3bb7fe700 1 -- 172.21.2.102:0/3853670057 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3bc001310 con 0x7fb3c4094960 2021-02-10T11:30:45.634 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:45.630 7fb3c8bd7700 1 -- 172.21.2.102:0/3853670057 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3c4188470 con 0x7fb3c4094960 2021-02-10T11:30:45.634 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:45.630 7fb3c8bd7700 1 -- 172.21.2.102:0/3853670057 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3c4188470 con 0x7fb3c4094960 2021-02-10T11:30:45.634 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:45.630 7fb3bb7fe700 1 -- 172.21.2.102:0/3853670057 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb3bc001830 con 0x7fb3c4094960 2021-02-10T11:30:45.634 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:45.631 7fb3bb7fe700 1 -- 172.21.2.102:0/3853670057 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fb3bc000d10 con 0x7fb3c4094960 2021-02-10T11:30:45.635 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:45.631 7fb3bb7fe700 1 --2- 172.21.2.102:0/3853670057 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb3a8018730 0x7fb3a801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:45.635 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:45.631 7fb3bb7fe700 1 -- 172.21.2.102:0/3853670057 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fb3bc032ba0 con 0x7fb3c4094960 2021-02-10T11:30:45.635 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:45.631 7fb3c285e700 1 --2- 172.21.2.102:0/3853670057 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb3a8018730 0x7fb3a801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:45.635 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:45.631 7fb3c285e700 1 --2- 172.21.2.102:0/3853670057 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb3a8018730 0x7fb3a801ac10 crc :-1 s=READY pgs=902 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:45.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:45.663 7fb3c8bd7700 1 -- 172.21.2.102:0/3853670057 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3c418bff0 con 0x7fb3c4094960 2021-02-10T11:30:45.666 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:45.666 7fb3bb7fe700 1 -- 172.21.2.102:0/3853670057 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fb3bc00be30 con 0x7fb3c4094960 2021-02-10T11:30:45.791 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:45.790 7fb3c8bd7700 1 -- 172.21.2.102:0/3853670057 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7fb3c418bff0 con 0x7fb3c4094960 2021-02-10T11:30:45.791 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:45.791 7fb3bb7fe700 1 -- 172.21.2.102:0/3853670057 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v93) v1 ==== 81+0+30363 (secure 0 0 0) 0x7fb3bc000d10 con 0x7fb3c4094960 2021-02-10T11:30:45.792 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:30:45.792 INFO:teuthology.orchestra.run.gibba002.stdout:{ 2021-02-10T11:30:45.792 INFO:teuthology.orchestra.run.gibba002.stdout: "epoch": 93, 2021-02-10T11:30:45.792 INFO:teuthology.orchestra.run.gibba002.stdout: "fsid": "68db30ea-dbce-4aef-a6df-f01c6fea0513", 2021-02-10T11:30:45.793 INFO:teuthology.orchestra.run.gibba002.stdout: "created": "2021-02-10 11:03:06.331557", 2021-02-10T11:30:45.793 INFO:teuthology.orchestra.run.gibba002.stdout: "modified": "2021-02-10 11:30:05.228248", 2021-02-10T11:30:45.793 INFO:teuthology.orchestra.run.gibba002.stdout: "last_up_change": "2021-02-10 11:03:32.690562", 2021-02-10T11:30:45.793 INFO:teuthology.orchestra.run.gibba002.stdout: "last_in_change": "2021-02-10 11:03:32.690562", 2021-02-10T11:30:45.793 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2021-02-10T11:30:45.793 INFO:teuthology.orchestra.run.gibba002.stdout: "flags_num": 5799936, 2021-02-10T11:30:45.793 INFO:teuthology.orchestra.run.gibba002.stdout: "flags_set": [ 2021-02-10T11:30:45.793 INFO:teuthology.orchestra.run.gibba002.stdout: "pglog_hardlimit", 2021-02-10T11:30:45.793 INFO:teuthology.orchestra.run.gibba002.stdout: "purged_snapdirs", 2021-02-10T11:30:45.793 INFO:teuthology.orchestra.run.gibba002.stdout: "recovery_deletes", 2021-02-10T11:30:45.793 INFO:teuthology.orchestra.run.gibba002.stdout: "sortbitwise" 2021-02-10T11:30:45.793 INFO:teuthology.orchestra.run.gibba002.stdout: ], 2021-02-10T11:30:45.793 INFO:teuthology.orchestra.run.gibba002.stdout: "crush_version": 4, 2021-02-10T11:30:45.794 INFO:teuthology.orchestra.run.gibba002.stdout: "full_ratio": 0.94999998807907104, 2021-02-10T11:30:45.794 INFO:teuthology.orchestra.run.gibba002.stdout: "backfillfull_ratio": 0.89999997615814209, 2021-02-10T11:30:45.794 INFO:teuthology.orchestra.run.gibba002.stdout: "nearfull_ratio": 0.85000002384185791, 2021-02-10T11:30:45.794 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_snapshot": "", 2021-02-10T11:30:45.794 INFO:teuthology.orchestra.run.gibba002.stdout: "pool_max": 19, 2021-02-10T11:30:45.794 INFO:teuthology.orchestra.run.gibba002.stdout: "max_osd": 8, 2021-02-10T11:30:45.794 INFO:teuthology.orchestra.run.gibba002.stdout: "require_min_compat_client": "jewel", 2021-02-10T11:30:45.794 INFO:teuthology.orchestra.run.gibba002.stdout: "min_compat_client": "jewel", 2021-02-10T11:30:45.794 INFO:teuthology.orchestra.run.gibba002.stdout: "require_osd_release": "nautilus", 2021-02-10T11:30:45.794 INFO:teuthology.orchestra.run.gibba002.stdout: "pools": [ 2021-02-10T11:30:45.794 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:30:45.794 INFO:teuthology.orchestra.run.gibba002.stdout: "pool": 1, 2021-02-10T11:30:45.794 INFO:teuthology.orchestra.run.gibba002.stdout: "pool_name": "rbd", 2021-02-10T11:30:45.795 INFO:teuthology.orchestra.run.gibba002.stdout: "create_time": "2021-02-10 11:03:37.322519", 2021-02-10T11:30:45.795 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 1, 2021-02-10T11:30:45.795 INFO:teuthology.orchestra.run.gibba002.stdout: "flags_names": "hashpspool", 2021-02-10T11:30:45.795 INFO:teuthology.orchestra.run.gibba002.stdout: "type": 1, 2021-02-10T11:30:45.795 INFO:teuthology.orchestra.run.gibba002.stdout: "size": 2, 2021-02-10T11:30:45.795 INFO:teuthology.orchestra.run.gibba002.stdout: "min_size": 1, 2021-02-10T11:30:45.795 INFO:teuthology.orchestra.run.gibba002.stdout: "crush_rule": 0, 2021-02-10T11:30:45.795 INFO:teuthology.orchestra.run.gibba002.stdout: "object_hash": 2, 2021-02-10T11:30:45.795 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:30:45.795 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num": 8, 2021-02-10T11:30:45.795 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_placement_num": 8, 2021-02-10T11:30:45.795 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_placement_num_target": 8, 2021-02-10T11:30:45.795 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num_target": 8, 2021-02-10T11:30:45.795 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num_pending": 8, 2021-02-10T11:30:45.796 INFO:teuthology.orchestra.run.gibba002.stdout: "last_pg_merge_meta": { 2021-02-10T11:30:45.796 INFO:teuthology.orchestra.run.gibba002.stdout: "source_pgid": "0.0", 2021-02-10T11:30:45.796 INFO:teuthology.orchestra.run.gibba002.stdout: "ready_epoch": 0, 2021-02-10T11:30:45.796 INFO:teuthology.orchestra.run.gibba002.stdout: "last_epoch_started": 0, 2021-02-10T11:30:45.796 INFO:teuthology.orchestra.run.gibba002.stdout: "last_epoch_clean": 0, 2021-02-10T11:30:45.796 INFO:teuthology.orchestra.run.gibba002.stdout: "source_version": "0'0", 2021-02-10T11:30:45.796 INFO:teuthology.orchestra.run.gibba002.stdout: "target_version": "0'0" 2021-02-10T11:30:45.796 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:30:45.796 INFO:teuthology.orchestra.run.gibba002.stdout: "last_change": "16", 2021-02-10T11:30:45.796 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend": "0", 2021-02-10T11:30:45.796 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:30:45.796 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:30:45.796 INFO:teuthology.orchestra.run.gibba002.stdout: "auid": 0, 2021-02-10T11:30:45.796 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:30:45.797 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_seq": 0, 2021-02-10T11:30:45.797 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_epoch": 0, 2021-02-10T11:30:45.797 INFO:teuthology.orchestra.run.gibba002.stdout: "pool_snaps": [], 2021-02-10T11:30:45.797 INFO:teuthology.orchestra.run.gibba002.stdout: "removed_snaps": "[]", 2021-02-10T11:30:45.797 INFO:teuthology.orchestra.run.gibba002.stdout: "quota_max_bytes": 0, 2021-02-10T11:30:45.797 INFO:teuthology.orchestra.run.gibba002.stdout: "quota_max_objects": 0, 2021-02-10T11:30:45.797 INFO:teuthology.orchestra.run.gibba002.stdout: "tiers": [], 2021-02-10T11:30:45.797 INFO:teuthology.orchestra.run.gibba002.stdout: "tier_of": -1, 2021-02-10T11:30:45.797 INFO:teuthology.orchestra.run.gibba002.stdout: "read_tier": -1, 2021-02-10T11:30:45.797 INFO:teuthology.orchestra.run.gibba002.stdout: "write_tier": -1, 2021-02-10T11:30:45.797 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_mode": "none", 2021-02-10T11:30:45.797 INFO:teuthology.orchestra.run.gibba002.stdout: "target_max_bytes": 0, 2021-02-10T11:30:45.797 INFO:teuthology.orchestra.run.gibba002.stdout: "target_max_objects": 0, 2021-02-10T11:30:45.798 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:30:45.798 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:30:45.798 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:30:45.798 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_min_flush_age": 0, 2021-02-10T11:30:45.798 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_min_evict_age": 0, 2021-02-10T11:30:45.798 INFO:teuthology.orchestra.run.gibba002.stdout: "erasure_code_profile": "", 2021-02-10T11:30:45.798 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_params": { 2021-02-10T11:30:45.798 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "none" 2021-02-10T11:30:45.798 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:30:45.798 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_period": 0, 2021-02-10T11:30:45.798 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_count": 0, 2021-02-10T11:30:45.798 INFO:teuthology.orchestra.run.gibba002.stdout: "use_gmt_hitset": true, 2021-02-10T11:30:45.798 INFO:teuthology.orchestra.run.gibba002.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:30:45.799 INFO:teuthology.orchestra.run.gibba002.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:30:45.799 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:30:45.799 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:30:45.799 INFO:teuthology.orchestra.run.gibba002.stdout: "grade_table": [], 2021-02-10T11:30:45.799 INFO:teuthology.orchestra.run.gibba002.stdout: "stripe_width": 0, 2021-02-10T11:30:45.799 INFO:teuthology.orchestra.run.gibba002.stdout: "expected_num_objects": 0, 2021-02-10T11:30:45.799 INFO:teuthology.orchestra.run.gibba002.stdout: "fast_read": false, 2021-02-10T11:30:45.799 INFO:teuthology.orchestra.run.gibba002.stdout: "options": {}, 2021-02-10T11:30:45.799 INFO:teuthology.orchestra.run.gibba002.stdout: "application_metadata": { 2021-02-10T11:30:45.799 INFO:teuthology.orchestra.run.gibba002.stdout: "rbd": {} 2021-02-10T11:30:45.799 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:30:45.799 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:30:45.799 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:30:45.799 INFO:teuthology.orchestra.run.gibba002.stdout: "pool": 18, 2021-02-10T11:30:45.800 INFO:teuthology.orchestra.run.gibba002.stdout: "pool_name": "cephfs_metadata", 2021-02-10T11:30:45.800 INFO:teuthology.orchestra.run.gibba002.stdout: "create_time": "2021-02-10 11:30:02.187273", 2021-02-10T11:30:45.800 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 1, 2021-02-10T11:30:45.800 INFO:teuthology.orchestra.run.gibba002.stdout: "flags_names": "hashpspool", 2021-02-10T11:30:45.800 INFO:teuthology.orchestra.run.gibba002.stdout: "type": 1, 2021-02-10T11:30:45.800 INFO:teuthology.orchestra.run.gibba002.stdout: "size": 2, 2021-02-10T11:30:45.800 INFO:teuthology.orchestra.run.gibba002.stdout: "min_size": 1, 2021-02-10T11:30:45.800 INFO:teuthology.orchestra.run.gibba002.stdout: "crush_rule": 0, 2021-02-10T11:30:45.800 INFO:teuthology.orchestra.run.gibba002.stdout: "object_hash": 2, 2021-02-10T11:30:45.800 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:30:45.800 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num": 8, 2021-02-10T11:30:45.800 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_placement_num": 8, 2021-02-10T11:30:45.800 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_placement_num_target": 8, 2021-02-10T11:30:45.802 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num_target": 8, 2021-02-10T11:30:45.802 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num_pending": 8, 2021-02-10T11:30:45.802 INFO:teuthology.orchestra.run.gibba002.stdout: "last_pg_merge_meta": { 2021-02-10T11:30:45.802 INFO:teuthology.orchestra.run.gibba002.stdout: "source_pgid": "0.0", 2021-02-10T11:30:45.802 INFO:teuthology.orchestra.run.gibba002.stdout: "ready_epoch": 0, 2021-02-10T11:30:45.802 INFO:teuthology.orchestra.run.gibba002.stdout: "last_epoch_started": 0, 2021-02-10T11:30:45.802 INFO:teuthology.orchestra.run.gibba002.stdout: "last_epoch_clean": 0, 2021-02-10T11:30:45.802 INFO:teuthology.orchestra.run.gibba002.stdout: "source_version": "0'0", 2021-02-10T11:30:45.802 INFO:teuthology.orchestra.run.gibba002.stdout: "target_version": "0'0" 2021-02-10T11:30:45.803 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:30:45.803 INFO:teuthology.orchestra.run.gibba002.stdout: "last_change": "92", 2021-02-10T11:30:45.803 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend": "0", 2021-02-10T11:30:45.803 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:30:45.803 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:30:45.803 INFO:teuthology.orchestra.run.gibba002.stdout: "auid": 0, 2021-02-10T11:30:45.803 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:30:45.803 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_seq": 0, 2021-02-10T11:30:45.803 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_epoch": 0, 2021-02-10T11:30:45.803 INFO:teuthology.orchestra.run.gibba002.stdout: "pool_snaps": [], 2021-02-10T11:30:45.803 INFO:teuthology.orchestra.run.gibba002.stdout: "removed_snaps": "[]", 2021-02-10T11:30:45.803 INFO:teuthology.orchestra.run.gibba002.stdout: "quota_max_bytes": 0, 2021-02-10T11:30:45.803 INFO:teuthology.orchestra.run.gibba002.stdout: "quota_max_objects": 0, 2021-02-10T11:30:45.804 INFO:teuthology.orchestra.run.gibba002.stdout: "tiers": [], 2021-02-10T11:30:45.804 INFO:teuthology.orchestra.run.gibba002.stdout: "tier_of": -1, 2021-02-10T11:30:45.804 INFO:teuthology.orchestra.run.gibba002.stdout: "read_tier": -1, 2021-02-10T11:30:45.804 INFO:teuthology.orchestra.run.gibba002.stdout: "write_tier": -1, 2021-02-10T11:30:45.804 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_mode": "none", 2021-02-10T11:30:45.804 INFO:teuthology.orchestra.run.gibba002.stdout: "target_max_bytes": 0, 2021-02-10T11:30:45.804 INFO:teuthology.orchestra.run.gibba002.stdout: "target_max_objects": 0, 2021-02-10T11:30:45.804 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:30:45.804 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:30:45.804 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:30:45.804 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_min_flush_age": 0, 2021-02-10T11:30:45.804 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_min_evict_age": 0, 2021-02-10T11:30:45.804 INFO:teuthology.orchestra.run.gibba002.stdout: "erasure_code_profile": "", 2021-02-10T11:30:45.804 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_params": { 2021-02-10T11:30:45.805 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "none" 2021-02-10T11:30:45.805 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:30:45.805 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_period": 0, 2021-02-10T11:30:45.805 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_count": 0, 2021-02-10T11:30:45.805 INFO:teuthology.orchestra.run.gibba002.stdout: "use_gmt_hitset": true, 2021-02-10T11:30:45.805 INFO:teuthology.orchestra.run.gibba002.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:30:45.805 INFO:teuthology.orchestra.run.gibba002.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:30:45.805 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:30:45.805 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:30:45.805 INFO:teuthology.orchestra.run.gibba002.stdout: "grade_table": [], 2021-02-10T11:30:45.805 INFO:teuthology.orchestra.run.gibba002.stdout: "stripe_width": 0, 2021-02-10T11:30:45.805 INFO:teuthology.orchestra.run.gibba002.stdout: "expected_num_objects": 0, 2021-02-10T11:30:45.805 INFO:teuthology.orchestra.run.gibba002.stdout: "fast_read": false, 2021-02-10T11:30:45.806 INFO:teuthology.orchestra.run.gibba002.stdout: "options": { 2021-02-10T11:30:45.806 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_autoscale_bias": 4, 2021-02-10T11:30:45.806 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num_min": 16, 2021-02-10T11:30:45.806 INFO:teuthology.orchestra.run.gibba002.stdout: "recovery_priority": 5 2021-02-10T11:30:45.806 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:30:45.806 INFO:teuthology.orchestra.run.gibba002.stdout: "application_metadata": { 2021-02-10T11:30:45.806 INFO:teuthology.orchestra.run.gibba002.stdout: "cephfs": { 2021-02-10T11:30:45.806 INFO:teuthology.orchestra.run.gibba002.stdout: "metadata": "cephfs" 2021-02-10T11:30:45.806 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:30:45.806 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:30:45.807 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:30:45.807 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:30:45.807 INFO:teuthology.orchestra.run.gibba002.stdout: "pool": 19, 2021-02-10T11:30:45.807 INFO:teuthology.orchestra.run.gibba002.stdout: "pool_name": "cephfs_data", 2021-02-10T11:30:45.807 INFO:teuthology.orchestra.run.gibba002.stdout: "create_time": "2021-02-10 11:30:03.219690", 2021-02-10T11:30:45.807 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 1, 2021-02-10T11:30:45.807 INFO:teuthology.orchestra.run.gibba002.stdout: "flags_names": "hashpspool", 2021-02-10T11:30:45.807 INFO:teuthology.orchestra.run.gibba002.stdout: "type": 1, 2021-02-10T11:30:45.807 INFO:teuthology.orchestra.run.gibba002.stdout: "size": 2, 2021-02-10T11:30:45.807 INFO:teuthology.orchestra.run.gibba002.stdout: "min_size": 1, 2021-02-10T11:30:45.807 INFO:teuthology.orchestra.run.gibba002.stdout: "crush_rule": 0, 2021-02-10T11:30:45.807 INFO:teuthology.orchestra.run.gibba002.stdout: "object_hash": 2, 2021-02-10T11:30:45.808 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:30:45.808 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num": 8, 2021-02-10T11:30:45.808 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_placement_num": 8, 2021-02-10T11:30:45.808 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_placement_num_target": 8, 2021-02-10T11:30:45.808 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num_target": 8, 2021-02-10T11:30:45.808 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num_pending": 8, 2021-02-10T11:30:45.808 INFO:teuthology.orchestra.run.gibba002.stdout: "last_pg_merge_meta": { 2021-02-10T11:30:45.808 INFO:teuthology.orchestra.run.gibba002.stdout: "source_pgid": "0.0", 2021-02-10T11:30:45.808 INFO:teuthology.orchestra.run.gibba002.stdout: "ready_epoch": 0, 2021-02-10T11:30:45.808 INFO:teuthology.orchestra.run.gibba002.stdout: "last_epoch_started": 0, 2021-02-10T11:30:45.808 INFO:teuthology.orchestra.run.gibba002.stdout: "last_epoch_clean": 0, 2021-02-10T11:30:45.808 INFO:teuthology.orchestra.run.gibba002.stdout: "source_version": "0'0", 2021-02-10T11:30:45.808 INFO:teuthology.orchestra.run.gibba002.stdout: "target_version": "0'0" 2021-02-10T11:30:45.809 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:30:45.809 INFO:teuthology.orchestra.run.gibba002.stdout: "last_change": "93", 2021-02-10T11:30:45.809 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend": "0", 2021-02-10T11:30:45.809 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:30:45.809 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:30:45.809 INFO:teuthology.orchestra.run.gibba002.stdout: "auid": 0, 2021-02-10T11:30:45.809 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:30:45.809 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_seq": 0, 2021-02-10T11:30:45.809 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_epoch": 0, 2021-02-10T11:30:45.809 INFO:teuthology.orchestra.run.gibba002.stdout: "pool_snaps": [], 2021-02-10T11:30:45.809 INFO:teuthology.orchestra.run.gibba002.stdout: "removed_snaps": "[]", 2021-02-10T11:30:45.809 INFO:teuthology.orchestra.run.gibba002.stdout: "quota_max_bytes": 0, 2021-02-10T11:30:45.809 INFO:teuthology.orchestra.run.gibba002.stdout: "quota_max_objects": 0, 2021-02-10T11:30:45.809 INFO:teuthology.orchestra.run.gibba002.stdout: "tiers": [], 2021-02-10T11:30:45.810 INFO:teuthology.orchestra.run.gibba002.stdout: "tier_of": -1, 2021-02-10T11:30:45.810 INFO:teuthology.orchestra.run.gibba002.stdout: "read_tier": -1, 2021-02-10T11:30:45.810 INFO:teuthology.orchestra.run.gibba002.stdout: "write_tier": -1, 2021-02-10T11:30:45.810 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_mode": "none", 2021-02-10T11:30:45.810 INFO:teuthology.orchestra.run.gibba002.stdout: "target_max_bytes": 0, 2021-02-10T11:30:45.810 INFO:teuthology.orchestra.run.gibba002.stdout: "target_max_objects": 0, 2021-02-10T11:30:45.810 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:30:45.810 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:30:45.810 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:30:45.810 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_min_flush_age": 0, 2021-02-10T11:30:45.810 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_min_evict_age": 0, 2021-02-10T11:30:45.810 INFO:teuthology.orchestra.run.gibba002.stdout: "erasure_code_profile": "", 2021-02-10T11:30:45.810 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_params": { 2021-02-10T11:30:45.810 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "none" 2021-02-10T11:30:45.811 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:30:45.811 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_period": 0, 2021-02-10T11:30:45.811 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_count": 0, 2021-02-10T11:30:45.811 INFO:teuthology.orchestra.run.gibba002.stdout: "use_gmt_hitset": true, 2021-02-10T11:30:45.811 INFO:teuthology.orchestra.run.gibba002.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:30:45.811 INFO:teuthology.orchestra.run.gibba002.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:30:45.811 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:30:45.811 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:30:45.811 INFO:teuthology.orchestra.run.gibba002.stdout: "grade_table": [], 2021-02-10T11:30:45.811 INFO:teuthology.orchestra.run.gibba002.stdout: "stripe_width": 0, 2021-02-10T11:30:45.811 INFO:teuthology.orchestra.run.gibba002.stdout: "expected_num_objects": 0, 2021-02-10T11:30:45.811 INFO:teuthology.orchestra.run.gibba002.stdout: "fast_read": false, 2021-02-10T11:30:45.811 INFO:teuthology.orchestra.run.gibba002.stdout: "options": {}, 2021-02-10T11:30:45.811 INFO:teuthology.orchestra.run.gibba002.stdout: "application_metadata": { 2021-02-10T11:30:45.812 INFO:teuthology.orchestra.run.gibba002.stdout: "cephfs": { 2021-02-10T11:30:45.812 INFO:teuthology.orchestra.run.gibba002.stdout: "data": "cephfs" 2021-02-10T11:30:45.812 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:30:45.812 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:30:45.812 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:30:45.812 INFO:teuthology.orchestra.run.gibba002.stdout: ], 2021-02-10T11:30:45.812 INFO:teuthology.orchestra.run.gibba002.stdout: "osds": [ 2021-02-10T11:30:45.812 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:30:45.812 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 0, 2021-02-10T11:30:45.812 INFO:teuthology.orchestra.run.gibba002.stdout: "uuid": "be691b4d-a039-408a-af80-9d53f02b6f1f", 2021-02-10T11:30:45.812 INFO:teuthology.orchestra.run.gibba002.stdout: "up": 1, 2021-02-10T11:30:45.812 INFO:teuthology.orchestra.run.gibba002.stdout: "in": 1, 2021-02-10T11:30:45.812 INFO:teuthology.orchestra.run.gibba002.stdout: "weight": 1, 2021-02-10T11:30:45.813 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_affinity": 1, 2021-02-10T11:30:45.813 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_begin": 0, 2021-02-10T11:30:45.813 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_end": 0, 2021-02-10T11:30:45.813 INFO:teuthology.orchestra.run.gibba002.stdout: "up_from": 13, 2021-02-10T11:30:45.813 INFO:teuthology.orchestra.run.gibba002.stdout: "up_thru": 90, 2021-02-10T11:30:45.813 INFO:teuthology.orchestra.run.gibba002.stdout: "down_at": 0, 2021-02-10T11:30:45.813 INFO:teuthology.orchestra.run.gibba002.stdout: "lost_at": 0, 2021-02-10T11:30:45.813 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addrs": { 2021-02-10T11:30:45.813 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:30:45.813 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:30:45.813 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:30:45.813 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6818", 2021-02-10T11:30:45.813 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36088 2021-02-10T11:30:45.814 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:30:45.814 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:30:45.814 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:30:45.814 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6820", 2021-02-10T11:30:45.814 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36088 2021-02-10T11:30:45.814 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:30:45.814 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:30:45.814 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:30:45.814 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addrs": { 2021-02-10T11:30:45.814 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:30:45.814 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:30:45.814 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:30:45.815 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6822", 2021-02-10T11:30:45.815 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36088 2021-02-10T11:30:45.815 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:30:45.815 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:30:45.815 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:30:45.815 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6824", 2021-02-10T11:30:45.815 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36088 2021-02-10T11:30:45.815 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:30:45.815 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:30:45.815 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:30:45.815 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addrs": { 2021-02-10T11:30:45.815 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:30:45.815 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:30:45.816 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:30:45.816 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6830", 2021-02-10T11:30:45.816 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36088 2021-02-10T11:30:45.816 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:30:45.816 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:30:45.816 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:30:45.816 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6832", 2021-02-10T11:30:45.816 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36088 2021-02-10T11:30:45.816 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:30:45.816 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:30:45.817 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:30:45.817 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addrs": { 2021-02-10T11:30:45.817 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:30:45.817 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:30:45.817 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:30:45.817 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6826", 2021-02-10T11:30:45.817 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36088 2021-02-10T11:30:45.817 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:30:45.817 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:30:45.818 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:30:45.819 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6828", 2021-02-10T11:30:45.819 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36088 2021-02-10T11:30:45.819 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:30:45.819 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:30:45.819 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:30:45.819 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addr": "172.21.2.102:6820/36088", 2021-02-10T11:30:45.819 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addr": "172.21.2.102:6824/36088", 2021-02-10T11:30:45.819 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addr": "172.21.2.102:6832/36088", 2021-02-10T11:30:45.819 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addr": "172.21.2.102:6828/36088", 2021-02-10T11:30:45.819 INFO:teuthology.orchestra.run.gibba002.stdout: "state": [ 2021-02-10T11:30:45.819 INFO:teuthology.orchestra.run.gibba002.stdout: "exists", 2021-02-10T11:30:45.820 INFO:teuthology.orchestra.run.gibba002.stdout: "up" 2021-02-10T11:30:45.820 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:30:45.820 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:30:45.820 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:30:45.820 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 1, 2021-02-10T11:30:45.820 INFO:teuthology.orchestra.run.gibba002.stdout: "uuid": "64e3ccc8-9c69-4d7c-865e-af8e83822635", 2021-02-10T11:30:45.820 INFO:teuthology.orchestra.run.gibba002.stdout: "up": 1, 2021-02-10T11:30:45.820 INFO:teuthology.orchestra.run.gibba002.stdout: "in": 1, 2021-02-10T11:30:45.820 INFO:teuthology.orchestra.run.gibba002.stdout: "weight": 1, 2021-02-10T11:30:45.822 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_affinity": 1, 2021-02-10T11:30:45.822 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_begin": 0, 2021-02-10T11:30:45.823 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_end": 0, 2021-02-10T11:30:45.823 INFO:teuthology.orchestra.run.gibba002.stdout: "up_from": 13, 2021-02-10T11:30:45.823 INFO:teuthology.orchestra.run.gibba002.stdout: "up_thru": 91, 2021-02-10T11:30:45.823 INFO:teuthology.orchestra.run.gibba002.stdout: "down_at": 0, 2021-02-10T11:30:45.823 INFO:teuthology.orchestra.run.gibba002.stdout: "lost_at": 0, 2021-02-10T11:30:45.823 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addrs": { 2021-02-10T11:30:45.823 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:30:45.823 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:30:45.823 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:30:45.823 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6819", 2021-02-10T11:30:45.823 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36087 2021-02-10T11:30:45.824 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:30:45.824 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:30:45.824 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:30:45.824 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6821", 2021-02-10T11:30:45.824 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36087 2021-02-10T11:30:45.824 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:30:45.824 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:30:45.824 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:30:45.825 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addrs": { 2021-02-10T11:30:45.825 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:30:45.825 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:30:45.825 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:30:45.825 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6823", 2021-02-10T11:30:45.825 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36087 2021-02-10T11:30:45.825 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:30:45.825 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:30:45.826 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:30:45.826 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6825", 2021-02-10T11:30:45.826 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36087 2021-02-10T11:30:45.826 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:30:45.826 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:30:45.826 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:30:45.826 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addrs": { 2021-02-10T11:30:45.826 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:30:45.826 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:30:45.826 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:30:45.826 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6831", 2021-02-10T11:30:45.827 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36087 2021-02-10T11:30:45.827 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:30:45.827 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:30:45.827 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:30:45.827 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6833", 2021-02-10T11:30:45.827 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36087 2021-02-10T11:30:45.827 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:30:45.827 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:30:45.827 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:30:45.827 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addrs": { 2021-02-10T11:30:45.828 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:30:45.828 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:30:45.828 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:30:45.828 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6827", 2021-02-10T11:30:45.828 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36087 2021-02-10T11:30:45.828 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:30:45.828 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:30:45.828 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:30:45.828 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6829", 2021-02-10T11:30:45.829 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36087 2021-02-10T11:30:45.829 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:30:45.829 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:30:45.829 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:30:45.829 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addr": "172.21.2.102:6821/36087", 2021-02-10T11:30:45.830 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addr": "172.21.2.102:6825/36087", 2021-02-10T11:30:45.830 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addr": "172.21.2.102:6833/36087", 2021-02-10T11:30:45.831 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addr": "172.21.2.102:6829/36087", 2021-02-10T11:30:45.831 INFO:teuthology.orchestra.run.gibba002.stdout: "state": [ 2021-02-10T11:30:45.831 INFO:teuthology.orchestra.run.gibba002.stdout: "exists", 2021-02-10T11:30:45.831 INFO:teuthology.orchestra.run.gibba002.stdout: "up" 2021-02-10T11:30:45.831 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:30:45.832 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:30:45.832 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:30:45.832 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 2, 2021-02-10T11:30:45.832 INFO:teuthology.orchestra.run.gibba002.stdout: "uuid": "153700af-747f-481f-bdf4-d6072d3a842f", 2021-02-10T11:30:45.832 INFO:teuthology.orchestra.run.gibba002.stdout: "up": 1, 2021-02-10T11:30:45.832 INFO:teuthology.orchestra.run.gibba002.stdout: "in": 1, 2021-02-10T11:30:45.832 INFO:teuthology.orchestra.run.gibba002.stdout: "weight": 1, 2021-02-10T11:30:45.832 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_affinity": 1, 2021-02-10T11:30:45.832 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_begin": 0, 2021-02-10T11:30:45.832 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_end": 0, 2021-02-10T11:30:45.833 INFO:teuthology.orchestra.run.gibba002.stdout: "up_from": 13, 2021-02-10T11:30:45.833 INFO:teuthology.orchestra.run.gibba002.stdout: "up_thru": 91, 2021-02-10T11:30:45.833 INFO:teuthology.orchestra.run.gibba002.stdout: "down_at": 0, 2021-02-10T11:30:45.833 INFO:teuthology.orchestra.run.gibba002.stdout: "lost_at": 0, 2021-02-10T11:30:45.833 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addrs": { 2021-02-10T11:30:45.833 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:30:45.833 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:30:45.833 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:30:45.833 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6810", 2021-02-10T11:30:45.833 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36089 2021-02-10T11:30:45.834 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:30:45.834 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:30:45.834 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:30:45.834 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6811", 2021-02-10T11:30:45.834 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36089 2021-02-10T11:30:45.834 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:30:45.834 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:30:45.834 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:30:45.834 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addrs": { 2021-02-10T11:30:45.834 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:30:45.834 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:30:45.835 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:30:45.835 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6812", 2021-02-10T11:30:45.835 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36089 2021-02-10T11:30:45.835 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:30:45.835 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:30:45.835 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:30:45.835 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6813", 2021-02-10T11:30:45.835 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36089 2021-02-10T11:30:45.835 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:30:45.835 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:30:45.835 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:30:45.836 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addrs": { 2021-02-10T11:30:45.836 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:30:45.836 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:30:45.836 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:30:45.836 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6816", 2021-02-10T11:30:45.836 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36089 2021-02-10T11:30:45.836 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:30:45.837 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:30:45.837 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:30:45.837 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6817", 2021-02-10T11:30:45.837 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36089 2021-02-10T11:30:45.837 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:30:45.837 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:30:45.837 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:30:45.837 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addrs": { 2021-02-10T11:30:45.837 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:30:45.837 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:30:45.838 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:30:45.838 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6814", 2021-02-10T11:30:45.838 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36089 2021-02-10T11:30:45.838 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:30:45.838 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:30:45.838 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:30:45.838 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6815", 2021-02-10T11:30:45.838 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36089 2021-02-10T11:30:45.838 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:30:45.838 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:30:45.838 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:30:45.839 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addr": "172.21.2.102:6811/36089", 2021-02-10T11:30:45.839 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addr": "172.21.2.102:6813/36089", 2021-02-10T11:30:45.839 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addr": "172.21.2.102:6817/36089", 2021-02-10T11:30:45.839 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addr": "172.21.2.102:6815/36089", 2021-02-10T11:30:45.839 INFO:teuthology.orchestra.run.gibba002.stdout: "state": [ 2021-02-10T11:30:45.839 INFO:teuthology.orchestra.run.gibba002.stdout: "exists", 2021-02-10T11:30:45.839 INFO:teuthology.orchestra.run.gibba002.stdout: "up" 2021-02-10T11:30:45.839 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:30:45.840 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:30:45.840 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:30:45.840 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 3, 2021-02-10T11:30:45.840 INFO:teuthology.orchestra.run.gibba002.stdout: "uuid": "cbc678f4-83a9-45e6-8830-2f7c232b1ea7", 2021-02-10T11:30:45.840 INFO:teuthology.orchestra.run.gibba002.stdout: "up": 1, 2021-02-10T11:30:45.840 INFO:teuthology.orchestra.run.gibba002.stdout: "in": 1, 2021-02-10T11:30:45.840 INFO:teuthology.orchestra.run.gibba002.stdout: "weight": 1, 2021-02-10T11:30:45.840 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_affinity": 1, 2021-02-10T11:30:45.840 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_begin": 0, 2021-02-10T11:30:45.840 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_end": 0, 2021-02-10T11:30:45.840 INFO:teuthology.orchestra.run.gibba002.stdout: "up_from": 13, 2021-02-10T11:30:45.840 INFO:teuthology.orchestra.run.gibba002.stdout: "up_thru": 91, 2021-02-10T11:30:45.840 INFO:teuthology.orchestra.run.gibba002.stdout: "down_at": 0, 2021-02-10T11:30:45.841 INFO:teuthology.orchestra.run.gibba002.stdout: "lost_at": 0, 2021-02-10T11:30:45.841 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addrs": { 2021-02-10T11:30:45.841 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:30:45.841 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:30:45.841 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:30:45.841 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6802", 2021-02-10T11:30:45.841 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36090 2021-02-10T11:30:45.841 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:30:45.841 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:30:45.841 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:30:45.841 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6803", 2021-02-10T11:30:45.841 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36090 2021-02-10T11:30:45.841 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:30:45.842 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:30:45.842 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:30:45.842 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addrs": { 2021-02-10T11:30:45.842 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:30:45.842 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:30:45.842 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:30:45.842 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6804", 2021-02-10T11:30:45.842 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36090 2021-02-10T11:30:45.842 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:30:45.842 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:30:45.842 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:30:45.842 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6805", 2021-02-10T11:30:45.843 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36090 2021-02-10T11:30:45.843 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:30:45.843 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:30:45.843 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:30:45.843 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addrs": { 2021-02-10T11:30:45.843 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:30:45.843 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:30:45.843 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:30:45.843 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6808", 2021-02-10T11:30:45.843 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36090 2021-02-10T11:30:45.843 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:30:45.843 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:30:45.843 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:30:45.844 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6809", 2021-02-10T11:30:45.844 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36090 2021-02-10T11:30:45.844 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:30:45.844 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:30:45.844 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:30:45.844 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addrs": { 2021-02-10T11:30:45.844 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:30:45.844 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:30:45.844 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:30:45.844 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6806", 2021-02-10T11:30:45.844 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36090 2021-02-10T11:30:45.844 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:30:45.844 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:30:45.845 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:30:45.845 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6807", 2021-02-10T11:30:45.845 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36090 2021-02-10T11:30:45.845 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:30:45.845 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:30:45.845 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:30:45.845 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addr": "172.21.2.102:6803/36090", 2021-02-10T11:30:45.845 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addr": "172.21.2.102:6805/36090", 2021-02-10T11:30:45.845 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addr": "172.21.2.102:6809/36090", 2021-02-10T11:30:45.845 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addr": "172.21.2.102:6807/36090", 2021-02-10T11:30:45.845 INFO:teuthology.orchestra.run.gibba002.stdout: "state": [ 2021-02-10T11:30:45.845 INFO:teuthology.orchestra.run.gibba002.stdout: "exists", 2021-02-10T11:30:45.845 INFO:teuthology.orchestra.run.gibba002.stdout: "up" 2021-02-10T11:30:45.846 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:30:45.846 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:30:45.846 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:30:45.846 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 4, 2021-02-10T11:30:45.846 INFO:teuthology.orchestra.run.gibba002.stdout: "uuid": "c01e4008-b258-4f10-afd2-9e7f37214351", 2021-02-10T11:30:45.846 INFO:teuthology.orchestra.run.gibba002.stdout: "up": 1, 2021-02-10T11:30:45.846 INFO:teuthology.orchestra.run.gibba002.stdout: "in": 1, 2021-02-10T11:30:45.846 INFO:teuthology.orchestra.run.gibba002.stdout: "weight": 1, 2021-02-10T11:30:45.846 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_affinity": 1, 2021-02-10T11:30:45.846 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_begin": 0, 2021-02-10T11:30:45.846 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_end": 0, 2021-02-10T11:30:45.846 INFO:teuthology.orchestra.run.gibba002.stdout: "up_from": 13, 2021-02-10T11:30:45.847 INFO:teuthology.orchestra.run.gibba002.stdout: "up_thru": 91, 2021-02-10T11:30:45.847 INFO:teuthology.orchestra.run.gibba002.stdout: "down_at": 0, 2021-02-10T11:30:45.847 INFO:teuthology.orchestra.run.gibba002.stdout: "lost_at": 0, 2021-02-10T11:30:45.847 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addrs": { 2021-02-10T11:30:45.847 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:30:45.847 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:30:45.847 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:30:45.847 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6808", 2021-02-10T11:30:45.847 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36368 2021-02-10T11:30:45.847 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:30:45.847 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:30:45.847 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:30:45.847 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6811", 2021-02-10T11:30:45.847 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36368 2021-02-10T11:30:45.848 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:30:45.848 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:30:45.848 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:30:45.848 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addrs": { 2021-02-10T11:30:45.848 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:30:45.848 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:30:45.848 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:30:45.848 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6814", 2021-02-10T11:30:45.848 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36368 2021-02-10T11:30:45.848 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:30:45.848 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:30:45.848 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:30:45.848 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6818", 2021-02-10T11:30:45.849 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36368 2021-02-10T11:30:45.849 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:30:45.849 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:30:45.849 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:30:45.849 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addrs": { 2021-02-10T11:30:45.849 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:30:45.849 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:30:45.849 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:30:45.849 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6826", 2021-02-10T11:30:45.849 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36368 2021-02-10T11:30:45.849 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:30:45.849 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:30:45.849 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:30:45.849 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6828", 2021-02-10T11:30:45.850 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36368 2021-02-10T11:30:45.850 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:30:45.850 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:30:45.850 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:30:45.850 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addrs": { 2021-02-10T11:30:45.850 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:30:45.850 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:30:45.850 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:30:45.850 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6820", 2021-02-10T11:30:45.850 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36368 2021-02-10T11:30:45.850 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:30:45.850 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:30:45.851 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:30:45.851 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6824", 2021-02-10T11:30:45.851 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36368 2021-02-10T11:30:45.851 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:30:45.851 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:30:45.851 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:30:45.851 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addr": "172.21.2.113:6811/36368", 2021-02-10T11:30:45.851 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addr": "172.21.2.113:6818/36368", 2021-02-10T11:30:45.851 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addr": "172.21.2.113:6828/36368", 2021-02-10T11:30:45.851 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addr": "172.21.2.113:6824/36368", 2021-02-10T11:30:45.851 INFO:teuthology.orchestra.run.gibba002.stdout: "state": [ 2021-02-10T11:30:45.851 INFO:teuthology.orchestra.run.gibba002.stdout: "exists", 2021-02-10T11:30:45.851 INFO:teuthology.orchestra.run.gibba002.stdout: "up" 2021-02-10T11:30:45.852 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:30:45.852 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:30:45.852 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:30:45.852 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 5, 2021-02-10T11:30:45.852 INFO:teuthology.orchestra.run.gibba002.stdout: "uuid": "551f6007-daf8-4a98-8ee0-c2b16bcccb5e", 2021-02-10T11:30:45.852 INFO:teuthology.orchestra.run.gibba002.stdout: "up": 1, 2021-02-10T11:30:45.852 INFO:teuthology.orchestra.run.gibba002.stdout: "in": 1, 2021-02-10T11:30:45.852 INFO:teuthology.orchestra.run.gibba002.stdout: "weight": 1, 2021-02-10T11:30:45.852 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_affinity": 1, 2021-02-10T11:30:45.852 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_begin": 0, 2021-02-10T11:30:45.852 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_end": 0, 2021-02-10T11:30:45.852 INFO:teuthology.orchestra.run.gibba002.stdout: "up_from": 13, 2021-02-10T11:30:45.852 INFO:teuthology.orchestra.run.gibba002.stdout: "up_thru": 91, 2021-02-10T11:30:45.852 INFO:teuthology.orchestra.run.gibba002.stdout: "down_at": 0, 2021-02-10T11:30:45.853 INFO:teuthology.orchestra.run.gibba002.stdout: "lost_at": 0, 2021-02-10T11:30:45.853 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addrs": { 2021-02-10T11:30:45.853 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:30:45.853 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:30:45.853 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:30:45.853 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6802", 2021-02-10T11:30:45.853 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36366 2021-02-10T11:30:45.853 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:30:45.853 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:30:45.853 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:30:45.853 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6805", 2021-02-10T11:30:45.853 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36366 2021-02-10T11:30:45.853 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:30:45.854 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:30:45.854 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:30:45.854 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addrs": { 2021-02-10T11:30:45.854 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:30:45.854 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:30:45.854 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:30:45.854 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6807", 2021-02-10T11:30:45.854 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36366 2021-02-10T11:30:45.854 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:30:45.854 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:30:45.854 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:30:45.854 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6810", 2021-02-10T11:30:45.854 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36366 2021-02-10T11:30:45.855 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:30:45.855 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:30:45.855 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:30:45.855 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addrs": { 2021-02-10T11:30:45.855 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:30:45.855 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:30:45.855 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:30:45.855 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6819", 2021-02-10T11:30:45.855 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36366 2021-02-10T11:30:45.855 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:30:45.855 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:30:45.855 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:30:45.855 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6822", 2021-02-10T11:30:45.856 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36366 2021-02-10T11:30:45.856 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:30:45.856 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:30:45.856 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:30:45.856 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addrs": { 2021-02-10T11:30:45.856 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:30:45.856 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:30:45.856 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:30:45.856 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6812", 2021-02-10T11:30:45.856 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36366 2021-02-10T11:30:45.856 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:30:45.856 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:30:45.856 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:30:45.857 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6815", 2021-02-10T11:30:45.857 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36366 2021-02-10T11:30:45.857 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:30:45.857 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:30:45.857 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:30:45.857 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addr": "172.21.2.113:6805/36366", 2021-02-10T11:30:45.857 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addr": "172.21.2.113:6810/36366", 2021-02-10T11:30:45.857 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addr": "172.21.2.113:6822/36366", 2021-02-10T11:30:45.857 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addr": "172.21.2.113:6815/36366", 2021-02-10T11:30:45.857 INFO:teuthology.orchestra.run.gibba002.stdout: "state": [ 2021-02-10T11:30:45.857 INFO:teuthology.orchestra.run.gibba002.stdout: "exists", 2021-02-10T11:30:45.857 INFO:teuthology.orchestra.run.gibba002.stdout: "up" 2021-02-10T11:30:45.857 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:30:45.857 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:30:45.858 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:30:45.858 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 6, 2021-02-10T11:30:45.858 INFO:teuthology.orchestra.run.gibba002.stdout: "uuid": "bf67c040-0e50-48be-b102-c3943bc9ebef", 2021-02-10T11:30:45.858 INFO:teuthology.orchestra.run.gibba002.stdout: "up": 1, 2021-02-10T11:30:45.858 INFO:teuthology.orchestra.run.gibba002.stdout: "in": 1, 2021-02-10T11:30:45.858 INFO:teuthology.orchestra.run.gibba002.stdout: "weight": 1, 2021-02-10T11:30:45.858 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_affinity": 1, 2021-02-10T11:30:45.858 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_begin": 0, 2021-02-10T11:30:45.858 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_end": 0, 2021-02-10T11:30:45.858 INFO:teuthology.orchestra.run.gibba002.stdout: "up_from": 13, 2021-02-10T11:30:45.858 INFO:teuthology.orchestra.run.gibba002.stdout: "up_thru": 83, 2021-02-10T11:30:45.858 INFO:teuthology.orchestra.run.gibba002.stdout: "down_at": 0, 2021-02-10T11:30:45.858 INFO:teuthology.orchestra.run.gibba002.stdout: "lost_at": 0, 2021-02-10T11:30:45.859 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addrs": { 2021-02-10T11:30:45.859 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:30:45.859 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:30:45.859 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:30:45.859 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6800", 2021-02-10T11:30:45.859 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36367 2021-02-10T11:30:45.859 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:30:45.859 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:30:45.859 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:30:45.859 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6801", 2021-02-10T11:30:45.859 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36367 2021-02-10T11:30:45.859 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:30:45.859 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:30:45.860 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:30:45.860 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addrs": { 2021-02-10T11:30:45.860 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:30:45.860 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:30:45.860 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:30:45.860 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6803", 2021-02-10T11:30:45.860 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36367 2021-02-10T11:30:45.860 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:30:45.860 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:30:45.860 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:30:45.860 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6804", 2021-02-10T11:30:45.860 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36367 2021-02-10T11:30:45.860 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:30:45.861 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:30:45.861 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:30:45.861 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addrs": { 2021-02-10T11:30:45.861 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:30:45.861 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:30:45.861 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:30:45.861 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6813", 2021-02-10T11:30:45.861 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36367 2021-02-10T11:30:45.861 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:30:45.861 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:30:45.861 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:30:45.861 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6816", 2021-02-10T11:30:45.861 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36367 2021-02-10T11:30:45.862 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:30:45.862 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:30:45.862 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:30:45.862 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addrs": { 2021-02-10T11:30:45.862 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:30:45.862 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:30:45.862 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:30:45.862 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6806", 2021-02-10T11:30:45.862 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36367 2021-02-10T11:30:45.862 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:30:45.862 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:30:45.862 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:30:45.863 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6809", 2021-02-10T11:30:45.863 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36367 2021-02-10T11:30:45.863 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:30:45.863 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:30:45.863 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:30:45.863 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addr": "172.21.2.113:6801/36367", 2021-02-10T11:30:45.863 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addr": "172.21.2.113:6804/36367", 2021-02-10T11:30:45.863 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addr": "172.21.2.113:6816/36367", 2021-02-10T11:30:45.863 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addr": "172.21.2.113:6809/36367", 2021-02-10T11:30:45.863 INFO:teuthology.orchestra.run.gibba002.stdout: "state": [ 2021-02-10T11:30:45.863 INFO:teuthology.orchestra.run.gibba002.stdout: "exists", 2021-02-10T11:30:45.863 INFO:teuthology.orchestra.run.gibba002.stdout: "up" 2021-02-10T11:30:45.863 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:30:45.863 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:30:45.864 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:30:45.864 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 7, 2021-02-10T11:30:45.864 INFO:teuthology.orchestra.run.gibba002.stdout: "uuid": "f5461f80-018d-428f-be89-2eb2bcbe7bb6", 2021-02-10T11:30:45.864 INFO:teuthology.orchestra.run.gibba002.stdout: "up": 1, 2021-02-10T11:30:45.864 INFO:teuthology.orchestra.run.gibba002.stdout: "in": 1, 2021-02-10T11:30:45.864 INFO:teuthology.orchestra.run.gibba002.stdout: "weight": 1, 2021-02-10T11:30:45.864 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_affinity": 1, 2021-02-10T11:30:45.864 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_begin": 0, 2021-02-10T11:30:45.864 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_end": 0, 2021-02-10T11:30:45.864 INFO:teuthology.orchestra.run.gibba002.stdout: "up_from": 13, 2021-02-10T11:30:45.864 INFO:teuthology.orchestra.run.gibba002.stdout: "up_thru": 91, 2021-02-10T11:30:45.864 INFO:teuthology.orchestra.run.gibba002.stdout: "down_at": 0, 2021-02-10T11:30:45.864 INFO:teuthology.orchestra.run.gibba002.stdout: "lost_at": 0, 2021-02-10T11:30:45.865 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addrs": { 2021-02-10T11:30:45.865 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:30:45.865 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:30:45.865 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:30:45.865 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6817", 2021-02-10T11:30:45.865 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36365 2021-02-10T11:30:45.865 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:30:45.865 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:30:45.865 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:30:45.865 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6821", 2021-02-10T11:30:45.865 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36365 2021-02-10T11:30:45.865 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:30:45.865 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:30:45.865 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:30:45.866 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addrs": { 2021-02-10T11:30:45.866 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:30:45.866 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:30:45.866 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:30:45.866 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6823", 2021-02-10T11:30:45.866 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36365 2021-02-10T11:30:45.866 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:30:45.866 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:30:45.866 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:30:45.866 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6825", 2021-02-10T11:30:45.866 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36365 2021-02-10T11:30:45.866 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:30:45.867 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:30:45.867 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:30:45.867 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addrs": { 2021-02-10T11:30:45.867 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:30:45.867 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:30:45.867 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:30:45.867 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6830", 2021-02-10T11:30:45.867 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36365 2021-02-10T11:30:45.867 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:30:45.867 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:30:45.867 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:30:45.867 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6831", 2021-02-10T11:30:45.867 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36365 2021-02-10T11:30:45.867 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:30:45.868 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:30:45.868 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:30:45.868 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addrs": { 2021-02-10T11:30:45.868 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:30:45.868 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:30:45.868 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:30:45.868 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6827", 2021-02-10T11:30:45.868 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36365 2021-02-10T11:30:45.868 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:30:45.868 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:30:45.868 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:30:45.868 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6829", 2021-02-10T11:30:45.868 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36365 2021-02-10T11:30:45.869 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:30:45.869 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:30:45.869 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:30:45.869 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addr": "172.21.2.113:6821/36365", 2021-02-10T11:30:45.869 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addr": "172.21.2.113:6825/36365", 2021-02-10T11:30:45.869 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addr": "172.21.2.113:6831/36365", 2021-02-10T11:30:45.869 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addr": "172.21.2.113:6829/36365", 2021-02-10T11:30:45.869 INFO:teuthology.orchestra.run.gibba002.stdout: "state": [ 2021-02-10T11:30:45.869 INFO:teuthology.orchestra.run.gibba002.stdout: "exists", 2021-02-10T11:30:45.869 INFO:teuthology.orchestra.run.gibba002.stdout: "up" 2021-02-10T11:30:45.869 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:30:45.869 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:30:45.869 INFO:teuthology.orchestra.run.gibba002.stdout: ], 2021-02-10T11:30:45.869 INFO:teuthology.orchestra.run.gibba002.stdout: "osd_xinfo": [ 2021-02-10T11:30:45.870 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:30:45.870 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 0, 2021-02-10T11:30:45.870 INFO:teuthology.orchestra.run.gibba002.stdout: "down_stamp": "0.000000", 2021-02-10T11:30:45.870 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_probability": 0, 2021-02-10T11:30:45.870 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_interval": 0, 2021-02-10T11:30:45.870 INFO:teuthology.orchestra.run.gibba002.stdout: "features": 4611087854035861503, 2021-02-10T11:30:45.870 INFO:teuthology.orchestra.run.gibba002.stdout: "old_weight": 0 2021-02-10T11:30:45.870 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:30:45.870 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:30:45.870 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 1, 2021-02-10T11:30:45.870 INFO:teuthology.orchestra.run.gibba002.stdout: "down_stamp": "0.000000", 2021-02-10T11:30:45.870 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_probability": 0, 2021-02-10T11:30:45.870 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_interval": 0, 2021-02-10T11:30:45.871 INFO:teuthology.orchestra.run.gibba002.stdout: "features": 4611087854035861503, 2021-02-10T11:30:45.871 INFO:teuthology.orchestra.run.gibba002.stdout: "old_weight": 0 2021-02-10T11:30:45.871 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:30:45.871 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:30:45.871 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 2, 2021-02-10T11:30:45.871 INFO:teuthology.orchestra.run.gibba002.stdout: "down_stamp": "0.000000", 2021-02-10T11:30:45.871 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_probability": 0, 2021-02-10T11:30:45.871 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_interval": 0, 2021-02-10T11:30:45.871 INFO:teuthology.orchestra.run.gibba002.stdout: "features": 4611087854035861503, 2021-02-10T11:30:45.871 INFO:teuthology.orchestra.run.gibba002.stdout: "old_weight": 0 2021-02-10T11:30:45.871 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:30:45.871 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:30:45.871 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 3, 2021-02-10T11:30:45.872 INFO:teuthology.orchestra.run.gibba002.stdout: "down_stamp": "0.000000", 2021-02-10T11:30:45.872 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_probability": 0, 2021-02-10T11:30:45.872 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_interval": 0, 2021-02-10T11:30:45.872 INFO:teuthology.orchestra.run.gibba002.stdout: "features": 4611087854035861503, 2021-02-10T11:30:45.872 INFO:teuthology.orchestra.run.gibba002.stdout: "old_weight": 0 2021-02-10T11:30:45.872 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:30:45.872 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:30:45.872 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 4, 2021-02-10T11:30:45.872 INFO:teuthology.orchestra.run.gibba002.stdout: "down_stamp": "0.000000", 2021-02-10T11:30:45.872 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_probability": 0, 2021-02-10T11:30:45.872 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_interval": 0, 2021-02-10T11:30:45.872 INFO:teuthology.orchestra.run.gibba002.stdout: "features": 4611087854035861503, 2021-02-10T11:30:45.872 INFO:teuthology.orchestra.run.gibba002.stdout: "old_weight": 0 2021-02-10T11:30:45.873 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:30:45.873 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:30:45.873 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 5, 2021-02-10T11:30:45.873 INFO:teuthology.orchestra.run.gibba002.stdout: "down_stamp": "0.000000", 2021-02-10T11:30:45.873 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_probability": 0, 2021-02-10T11:30:45.873 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_interval": 0, 2021-02-10T11:30:45.873 INFO:teuthology.orchestra.run.gibba002.stdout: "features": 4611087854035861503, 2021-02-10T11:30:45.873 INFO:teuthology.orchestra.run.gibba002.stdout: "old_weight": 0 2021-02-10T11:30:45.873 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:30:45.873 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:30:45.873 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 6, 2021-02-10T11:30:45.873 INFO:teuthology.orchestra.run.gibba002.stdout: "down_stamp": "0.000000", 2021-02-10T11:30:45.873 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_probability": 0, 2021-02-10T11:30:45.873 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_interval": 0, 2021-02-10T11:30:45.874 INFO:teuthology.orchestra.run.gibba002.stdout: "features": 4611087854035861503, 2021-02-10T11:30:45.874 INFO:teuthology.orchestra.run.gibba002.stdout: "old_weight": 0 2021-02-10T11:30:45.874 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:30:45.874 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:30:45.874 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 7, 2021-02-10T11:30:45.874 INFO:teuthology.orchestra.run.gibba002.stdout: "down_stamp": "0.000000", 2021-02-10T11:30:45.874 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_probability": 0, 2021-02-10T11:30:45.874 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_interval": 0, 2021-02-10T11:30:45.874 INFO:teuthology.orchestra.run.gibba002.stdout: "features": 4611087854035861503, 2021-02-10T11:30:45.874 INFO:teuthology.orchestra.run.gibba002.stdout: "old_weight": 0 2021-02-10T11:30:45.874 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:30:45.874 INFO:teuthology.orchestra.run.gibba002.stdout: ], 2021-02-10T11:30:45.874 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_upmap": [], 2021-02-10T11:30:45.875 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_upmap_items": [], 2021-02-10T11:30:45.875 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_temp": [], 2021-02-10T11:30:45.875 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_temp": [], 2021-02-10T11:30:45.875 INFO:teuthology.orchestra.run.gibba002.stdout: "blacklist": {}, 2021-02-10T11:30:45.875 INFO:teuthology.orchestra.run.gibba002.stdout: "erasure_code_profiles": { 2021-02-10T11:30:45.875 INFO:teuthology.orchestra.run.gibba002.stdout: "default": { 2021-02-10T11:30:45.875 INFO:teuthology.orchestra.run.gibba002.stdout: "crush-failure-domain": "osd", 2021-02-10T11:30:45.875 INFO:teuthology.orchestra.run.gibba002.stdout: "k": "2", 2021-02-10T11:30:45.875 INFO:teuthology.orchestra.run.gibba002.stdout: "m": "1", 2021-02-10T11:30:45.875 INFO:teuthology.orchestra.run.gibba002.stdout: "plugin": "jerasure", 2021-02-10T11:30:45.875 INFO:teuthology.orchestra.run.gibba002.stdout: "ruleset-failure-domain": "osd", 2021-02-10T11:30:45.875 INFO:teuthology.orchestra.run.gibba002.stdout: "technique": "reed_sol_van" 2021-02-10T11:30:45.875 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:30:45.876 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:30:45.876 INFO:teuthology.orchestra.run.gibba002.stdout: "removed_snaps_queue": [], 2021-02-10T11:30:45.876 INFO:teuthology.orchestra.run.gibba002.stdout: "new_removed_snaps": [], 2021-02-10T11:30:45.876 INFO:teuthology.orchestra.run.gibba002.stdout: "new_purged_snaps": [], 2021-02-10T11:30:45.876 INFO:teuthology.orchestra.run.gibba002.stdout: "crush_node_flags": {}, 2021-02-10T11:30:45.876 INFO:teuthology.orchestra.run.gibba002.stdout: "device_class_flags": {} 2021-02-10T11:30:45.876 INFO:teuthology.orchestra.run.gibba002.stdout:} 2021-02-10T11:30:45.877 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:45.792 7fb3c8bd7700 1 -- 172.21.2.102:0/3853670057 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb3a8018730 msgr2=0x7fb3a801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:45.877 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:45.792 7fb3c8bd7700 1 --2- 172.21.2.102:0/3853670057 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb3a8018730 0x7fb3a801ac10 crc :-1 s=READY pgs=902 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:45.878 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:45.792 7fb3c8bd7700 1 -- 172.21.2.102:0/3853670057 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb3c4094960 msgr2=0x7fb3c4187e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:45.878 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:45.792 7fb3c8bd7700 1 --2- 172.21.2.102:0/3853670057 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb3c4094960 0x7fb3c4187e50 secure :-1 s=READY pgs=2022 cs=0 l=1 rev1=1 rx=0x7fb3bc009960 tx=0x7fb3bc00bcf0).stop 2021-02-10T11:30:45.878 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:45.792 7fb3c8bd7700 1 -- 172.21.2.102:0/3853670057 shutdown_connections 2021-02-10T11:30:45.878 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:45.792 7fb3c8bd7700 1 --2- 172.21.2.102:0/3853670057 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb3c4094960 0x7fb3c4187e50 unknown :-1 s=CLOSED pgs=2022 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:45.878 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:45.792 7fb3c8bd7700 1 --2- 172.21.2.102:0/3853670057 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb3a8018730 0x7fb3a801ac10 unknown :-1 s=CLOSED pgs=902 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:45.878 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:45.792 7fb3c8bd7700 1 --2- 172.21.2.102:0/3853670057 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb3c4093f40 0x7fb3c4094340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:45.878 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:45.792 7fb3c8bd7700 1 --2- 172.21.2.102:0/3853670057 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb3c40b4f30 0x7fb3c418b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:45.878 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:45.792 7fb3c8bd7700 1 -- 172.21.2.102:0/3853670057 >> 172.21.2.102:0/3853670057 conn(0x7fb3c419f8f0 msgr2=0x7fb3c41a5f80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:45.878 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:45.793 7fb3c8bd7700 1 -- 172.21.2.102:0/3853670057 shutdown_connections 2021-02-10T11:30:45.878 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:45.793 7fb3c8bd7700 1 -- 172.21.2.102:0/3853670057 wait complete. 2021-02-10T11:30:45.879 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:30:46.305 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:46.288 7f03a33b1700 1 Processor -- start 2021-02-10T11:30:46.306 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:46.288 7f03a33b1700 1 -- start start 2021-02-10T11:30:46.306 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:46.289 7f03a33b1700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f039c0d1870 0x7f039c1b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:46.306 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:46.289 7f03a33b1700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f039c0f0b10 con 0x7f039c1b0a10 2021-02-10T11:30:46.306 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:46.289 7f03a33b1700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f039c0d0310 con 0x7f039c0f0650 2021-02-10T11:30:46.306 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:46.289 7f03a33b1700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f039c0d0530 con 0x7f039c0d1870 2021-02-10T11:30:46.306 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:46.289 7f03a11cd700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f039c0d1870 0x7f039c1b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:46.306 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:46.289 7f03a09cc700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f039c1b0a10 0x7f039c0cf330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:53500/0 (socket says 172.21.2.102:53500) 2021-02-10T11:30:46.306 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:46.289 7f03a09cc700 1 -- 172.21.2.102:0/574758610 learned_addr learned my addr 172.21.2.102:0/574758610 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:30:46.306 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:46.289 7f03a11cd700 1 -- 172.21.2.102:0/574758610 >> v1:172.21.2.113:6789/0 conn(0x7f039c0f0650 legacy=0x7f039c0cfb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:46.306 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:46.289 7f03a11cd700 1 -- 172.21.2.102:0/574758610 >> v1:172.21.2.102:6789/0 conn(0x7f039c1b0a10 legacy=0x7f039c0cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:46.306 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:46.289 7f03a11cd700 1 -- 172.21.2.102:0/574758610 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f039c0d1c70 con 0x7f039c0d1870 2021-02-10T11:30:46.306 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:46.289 7f03a11cd700 1 --2- 172.21.2.102:0/574758610 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f039c0d1870 0x7f039c1b03f0 secure :-1 s=READY pgs=2024 cs=0 l=1 rev1=1 rx=0x7f0390006650 tx=0x7f0390009a80).ready entity=mon.0 client_cookie=73a43697f5694ecd server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:46.307 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:46.289 7f038ffff700 1 -- 172.21.2.102:0/574758610 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f039000c0c0 con 0x7f039c0d1870 2021-02-10T11:30:46.307 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:46.290 7f038ffff700 1 -- 172.21.2.102:0/574758610 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f039000c340 con 0x7f039c0d1870 2021-02-10T11:30:46.307 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:46.290 7f038ffff700 1 -- 172.21.2.102:0/574758610 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f039000c820 con 0x7f039c0d1870 2021-02-10T11:30:46.307 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:46.290 7f03a33b1700 1 -- 172.21.2.102:0/574758610 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f039c0d1870 msgr2=0x7f039c1b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:46.307 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:46.290 7f03a33b1700 1 --2- 172.21.2.102:0/574758610 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f039c0d1870 0x7f039c1b03f0 secure :-1 s=READY pgs=2024 cs=0 l=1 rev1=1 rx=0x7f0390006650 tx=0x7f0390009a80).stop 2021-02-10T11:30:46.307 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:46.290 7f03a33b1700 1 -- 172.21.2.102:0/574758610 shutdown_connections 2021-02-10T11:30:46.307 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:46.290 7f03a33b1700 1 --2- 172.21.2.102:0/574758610 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f039c0d1870 0x7f039c1b03f0 unknown :-1 s=CLOSED pgs=2024 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:46.307 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:46.290 7f03a33b1700 1 -- 172.21.2.102:0/574758610 >> 172.21.2.102:0/574758610 conn(0x7f039c0ce450 msgr2=0x7f039c0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:46.307 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:46.290 7f03a33b1700 1 -- 172.21.2.102:0/574758610 shutdown_connections 2021-02-10T11:30:46.307 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:46.290 7f03a33b1700 1 -- 172.21.2.102:0/574758610 wait complete. 2021-02-10T11:30:46.309 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:46.305 7f03a33b1700 1 Processor -- start 2021-02-10T11:30:46.309 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:46.305 7f03a33b1700 1 -- start start 2021-02-10T11:30:46.309 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:46.305 7f03a33b1700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f039c094020 0x7f039c093730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:46.310 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:46.305 7f03a33b1700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f039c09ea70 0x7f039c09ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:46.310 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:46.306 7f03a33b1700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f039c1b0770 0x7f039c1b0b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:46.310 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:46.306 7f03a33b1700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f039c09f4e0 con 0x7f039c09ea70 2021-02-10T11:30:46.310 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:46.306 7f03a33b1700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f039c0ab6b0 con 0x7f039c1b0770 2021-02-10T11:30:46.310 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:46.306 7f03a33b1700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f039c1b1190 con 0x7f039c094020 2021-02-10T11:30:46.310 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:46.306 7f03a09cc700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f039c09ea70 0x7f039c09ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:46.310 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:46.306 7f03a09cc700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f039c09ea70 0x7f039c09ee70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:33050/0 (socket says 172.21.2.102:33050) 2021-02-10T11:30:46.310 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:46.306 7f03a09cc700 1 -- 172.21.2.102:0/294991726 learned_addr learned my addr 172.21.2.102:0/294991726 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:30:46.310 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:46.306 7f03a09cc700 1 -- 172.21.2.102:0/294991726 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f039c094020 msgr2=0x7f039c093730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:46.310 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:46.306 7f03a09cc700 1 --2- 172.21.2.102:0/294991726 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f039c094020 0x7f039c093730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:46.310 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:46.306 7f03a09cc700 1 -- 172.21.2.102:0/294991726 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f039c1b0770 msgr2=0x7f039c1b0b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:46.310 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:46.306 7f03a09cc700 1 --2- 172.21.2.102:0/294991726 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f039c1b0770 0x7f039c1b0b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:46.310 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:46.306 7f03a09cc700 1 -- 172.21.2.102:0/294991726 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f039c1b13b0 con 0x7f039c09ea70 2021-02-10T11:30:46.311 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:46.306 7f03a09cc700 1 --2- 172.21.2.102:0/294991726 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f039c09ea70 0x7f039c09ee70 secure :-1 s=READY pgs=2025 cs=0 l=1 rev1=1 rx=0x7f039800a4c0 tx=0x7f039800a860).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:46.311 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:46.306 7f038dffb700 1 -- 172.21.2.102:0/294991726 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f039800ac50 con 0x7f039c09ea70 2021-02-10T11:30:46.311 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:46.306 7f038dffb700 1 -- 172.21.2.102:0/294991726 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03980012a0 con 0x7f039c09ea70 2021-02-10T11:30:46.311 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:46.306 7f038dffb700 1 -- 172.21.2.102:0/294991726 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0398004330 con 0x7f039c09ea70 2021-02-10T11:30:46.311 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:46.306 7f03a33b1700 1 -- 172.21.2.102:0/294991726 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f039c1b13b0 con 0x7f039c09ea70 2021-02-10T11:30:46.311 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:46.307 7f03a33b1700 1 -- 172.21.2.102:0/294991726 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f039c1b13b0 con 0x7f039c09ea70 2021-02-10T11:30:46.311 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:46.307 7f038dffb700 1 -- 172.21.2.102:0/294991726 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f0398004140 con 0x7f039c09ea70 2021-02-10T11:30:46.311 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:46.307 7f038dffb700 1 --2- 172.21.2.102:0/294991726 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f037c018730 0x7f037c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:46.311 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:46.307 7f038dffb700 1 -- 172.21.2.102:0/294991726 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f0398032610 con 0x7f039c09ea70 2021-02-10T11:30:46.311 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:46.307 7f03a11cd700 1 --2- 172.21.2.102:0/294991726 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f037c018730 0x7f037c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:46.311 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:46.307 7f03a11cd700 1 --2- 172.21.2.102:0/294991726 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f037c018730 0x7f037c01abc0 crc :-1 s=READY pgs=903 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:46.339 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:46.339 7f03a33b1700 1 -- 172.21.2.102:0/294991726 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f039c0e7e20 con 0x7f039c09ea70 2021-02-10T11:30:46.342 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:46.341 7f038dffb700 1 -- 172.21.2.102:0/294991726 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f03980009b0 con 0x7f039c09ea70 2021-02-10T11:30:46.469 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:46.469 7f03a33b1700 1 -- 172.21.2.102:0/294991726 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f039c0d1ee0 con 0x7f039c09ea70 2021-02-10T11:30:46.470 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:46.469 7f038dffb700 1 -- 172.21.2.102:0/294991726 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 74 v74) v1 ==== 94+0+2098 (secure 0 0 0) 0x7f0398001280 con 0x7f039c09ea70 2021-02-10T11:30:46.470 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:30:46.471 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":74,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9672,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.102:6835/2071608955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6834","nonce":2071608955},{"type":"v1","addr":"172.21.2.102:6835","nonce":2071608955}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":69}],"filesystems":[{"mdsmap":{"epoch":73,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:30:04.225997","modified":"2021-02-10 11:30:06.232546","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7186},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7186":{"gid":7186,"name":"b","rank":0,"incarnation":70,"state":"up:active","state_seq":3,"addr":"172.21.2.113:6833/675492649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6832","nonce":675492649},{"type":"v1","addr":"172.21.2.113:6833","nonce":675492649}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":9}]} 2021-02-10T11:30:46.471 INFO:teuthology.orchestra.run.gibba002.stderr:dumped fsmap epoch 74 2021-02-10T11:30:46.472 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:46.471 7f03a33b1700 1 -- 172.21.2.102:0/294991726 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f037c018730 msgr2=0x7f037c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:46.472 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:46.471 7f03a33b1700 1 --2- 172.21.2.102:0/294991726 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f037c018730 0x7f037c01abc0 crc :-1 s=READY pgs=903 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:46.472 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:46.471 7f03a33b1700 1 -- 172.21.2.102:0/294991726 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f039c09ea70 msgr2=0x7f039c09ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:46.472 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:46.471 7f03a33b1700 1 --2- 172.21.2.102:0/294991726 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f039c09ea70 0x7f039c09ee70 secure :-1 s=READY pgs=2025 cs=0 l=1 rev1=1 rx=0x7f039800a4c0 tx=0x7f039800a860).stop 2021-02-10T11:30:46.472 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:46.471 7f03a33b1700 1 -- 172.21.2.102:0/294991726 shutdown_connections 2021-02-10T11:30:46.473 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:46.471 7f03a33b1700 1 --2- 172.21.2.102:0/294991726 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f037c018730 0x7f037c01abc0 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:46.473 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:46.471 7f03a33b1700 1 --2- 172.21.2.102:0/294991726 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f039c1b0770 0x7f039c1b0b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:46.473 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:46.471 7f03a33b1700 1 --2- 172.21.2.102:0/294991726 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f039c09ea70 0x7f039c09ee70 unknown :-1 s=CLOSED pgs=2025 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:46.473 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:46.471 7f03a33b1700 1 --2- 172.21.2.102:0/294991726 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f039c094020 0x7f039c093730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:46.473 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:46.471 7f03a33b1700 1 -- 172.21.2.102:0/294991726 >> 172.21.2.102:0/294991726 conn(0x7f039c0a0030 msgr2=0x7f039c0ea250 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:46.473 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:46.471 7f03a33b1700 1 -- 172.21.2.102:0/294991726 shutdown_connections 2021-02-10T11:30:46.473 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:46.471 7f03a33b1700 1 -- 172.21.2.102:0/294991726 wait complete. 2021-02-10T11:30:46.480 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs fail cephfs 2021-02-10T11:30:46.909 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:46.892 7f9ab6b6c700 1 Processor -- start 2021-02-10T11:30:46.910 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:46.893 7f9ab6b6c700 1 -- start start 2021-02-10T11:30:46.910 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:46.893 7f9ab6b6c700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f9ab00d1870 0x7f9ab01b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:46.910 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:46.893 7f9ab6b6c700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f9ab00f0650 0x7f9ab00cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:46.910 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:46.893 7f9ab6b6c700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ab00f0b10 con 0x7f9ab01b0a10 2021-02-10T11:30:46.910 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:46.893 7f9ab6b6c700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ab00e5ed0 con 0x7f9ab00f0650 2021-02-10T11:30:46.910 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:46.893 7f9ab6b6c700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ab00d1c70 con 0x7f9ab00d1870 2021-02-10T11:30:46.910 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:46.893 7f9aaffff700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f9ab01b0a10 0x7f9ab00cf330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:53512/0 (socket says 172.21.2.102:53512) 2021-02-10T11:30:46.910 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:46.893 7f9aaffff700 1 -- 172.21.2.102:0/3098176766 learned_addr learned my addr 172.21.2.102:0/3098176766 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:30:46.910 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:46.893 7f9ab5189700 1 --2- 172.21.2.102:0/3098176766 >> v2:172.21.2.102:3300/0 conn(0x7f9ab00f0650 0x7f9ab00cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:46.910 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:46.893 7f9ab5189700 1 -- 172.21.2.102:0/3098176766 >> v2:172.21.2.113:3300/0 conn(0x7f9ab00d1870 msgr2=0x7f9ab01b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:46.910 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:46.893 7f9ab5189700 1 --2- 172.21.2.102:0/3098176766 >> v2:172.21.2.113:3300/0 conn(0x7f9ab00d1870 0x7f9ab01b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:46.911 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:46.893 7f9ab5189700 1 -- 172.21.2.102:0/3098176766 >> v1:172.21.2.102:6789/0 conn(0x7f9ab01b0a10 legacy=0x7f9ab00cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:46.911 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:46.893 7f9ab5189700 1 -- 172.21.2.102:0/3098176766 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ab00d1e90 con 0x7f9ab00f0650 2021-02-10T11:30:46.911 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:46.894 7f9ab5189700 1 --2- 172.21.2.102:0/3098176766 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9ab00f0650 0x7f9ab00cfb20 secure :-1 s=READY pgs=2027 cs=0 l=1 rev1=1 rx=0x7f9aa8006ab0 tx=0x7f9aa8009b40).ready entity=mon.0 client_cookie=636db2b9e02c445a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:46.911 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:46.894 7f9aaf7fe700 1 -- 172.21.2.102:0/3098176766 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9aa80022e0 con 0x7f9ab00f0650 2021-02-10T11:30:46.911 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:46.894 7f9aaf7fe700 1 -- 172.21.2.102:0/3098176766 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9aa8002560 con 0x7f9ab00f0650 2021-02-10T11:30:46.911 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:46.894 7f9aaf7fe700 1 -- 172.21.2.102:0/3098176766 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9aa80029f0 con 0x7f9ab00f0650 2021-02-10T11:30:46.911 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:46.894 7f9ab6b6c700 1 -- 172.21.2.102:0/3098176766 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9ab00f0650 msgr2=0x7f9ab00cfb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:46.911 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:46.894 7f9ab6b6c700 1 --2- 172.21.2.102:0/3098176766 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9ab00f0650 0x7f9ab00cfb20 secure :-1 s=READY pgs=2027 cs=0 l=1 rev1=1 rx=0x7f9aa8006ab0 tx=0x7f9aa8009b40).stop 2021-02-10T11:30:46.911 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:46.894 7f9ab6b6c700 1 -- 172.21.2.102:0/3098176766 shutdown_connections 2021-02-10T11:30:46.911 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:46.894 7f9ab6b6c700 1 --2- 172.21.2.102:0/3098176766 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9ab00f0650 0x7f9ab00cfb20 unknown :-1 s=CLOSED pgs=2027 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:46.911 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:46.894 7f9ab6b6c700 1 --2- 172.21.2.102:0/3098176766 >> v2:172.21.2.113:3300/0 conn(0x7f9ab00d1870 0x7f9ab01b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:46.911 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:46.894 7f9ab6b6c700 1 -- 172.21.2.102:0/3098176766 >> 172.21.2.102:0/3098176766 conn(0x7f9ab00ce450 msgr2=0x7f9ab00d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:46.911 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:46.895 7f9ab6b6c700 1 -- 172.21.2.102:0/3098176766 shutdown_connections 2021-02-10T11:30:46.912 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:46.895 7f9ab6b6c700 1 -- 172.21.2.102:0/3098176766 wait complete. 2021-02-10T11:30:46.914 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:46.909 7f9ab6b6c700 1 Processor -- start 2021-02-10T11:30:46.914 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:46.909 7f9ab6b6c700 1 -- start start 2021-02-10T11:30:46.914 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:46.910 7f9ab6b6c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9ab00a5ef0 0x7f9ab0188f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:46.914 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:46.910 7f9ab6b6c700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9ab00a62f0 0x7f9ab0189570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:46.914 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:46.910 7f9ab6b6c700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f9ab0189b90 0x7f9ab008cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:46.914 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:46.910 7f9ab6b6c700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ab008c360 con 0x7f9ab00a5ef0 2021-02-10T11:30:46.914 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:46.910 7f9ab6b6c700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ab0189f90 con 0x7f9ab0189b90 2021-02-10T11:30:46.914 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:46.910 7f9ab6b6c700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ab008d1b0 con 0x7f9ab00a62f0 2021-02-10T11:30:46.914 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:46.910 7f9ab4988700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9ab00a5ef0 0x7f9ab0188f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:46.914 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:46.910 7f9ab4988700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9ab00a5ef0 0x7f9ab0188f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:33066/0 (socket says 172.21.2.102:33066) 2021-02-10T11:30:46.914 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:46.910 7f9ab4988700 1 -- 172.21.2.102:0/3056582624 learned_addr learned my addr 172.21.2.102:0/3056582624 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:30:46.914 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:46.910 7f9ab5189700 1 --2- 172.21.2.102:0/3056582624 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f9ab0189b90 0x7f9ab008cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:46.915 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:46.910 7f9aaffff700 1 --2- 172.21.2.102:0/3056582624 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9ab00a62f0 0x7f9ab0189570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:46.915 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:46.910 7f9ab4988700 1 -- 172.21.2.102:0/3056582624 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9ab00a62f0 msgr2=0x7f9ab0189570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:46.915 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:46.910 7f9ab4988700 1 --2- 172.21.2.102:0/3056582624 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9ab00a62f0 0x7f9ab0189570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:46.915 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:46.910 7f9ab4988700 1 -- 172.21.2.102:0/3056582624 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f9ab0189b90 msgr2=0x7f9ab008cb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:46.915 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:46.910 7f9ab4988700 1 --2- 172.21.2.102:0/3056582624 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f9ab0189b90 0x7f9ab008cb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:46.915 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:46.910 7f9ab4988700 1 -- 172.21.2.102:0/3056582624 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ab008d3d0 con 0x7f9ab00a5ef0 2021-02-10T11:30:46.915 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:46.911 7f9ab4988700 1 --2- 172.21.2.102:0/3056582624 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9ab00a5ef0 0x7f9ab0188f50 secure :-1 s=READY pgs=2028 cs=0 l=1 rev1=1 rx=0x7f9a9800b530 tx=0x7f9a9800b8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:46.915 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:46.911 7f9aad7fa700 1 -- 172.21.2.102:0/3056582624 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9a9800bcc0 con 0x7f9ab00a5ef0 2021-02-10T11:30:46.915 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:46.911 7f9aad7fa700 1 -- 172.21.2.102:0/3056582624 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a98000c90 con 0x7f9ab00a5ef0 2021-02-10T11:30:46.915 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:46.911 7f9aad7fa700 1 -- 172.21.2.102:0/3056582624 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9a980022a0 con 0x7f9ab00a5ef0 2021-02-10T11:30:46.915 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:46.911 7f9ab6b6c700 1 -- 172.21.2.102:0/3056582624 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ab008d3d0 con 0x7f9ab00a5ef0 2021-02-10T11:30:46.915 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:46.911 7f9ab6b6c700 1 -- 172.21.2.102:0/3056582624 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ab008d3d0 con 0x7f9ab00a5ef0 2021-02-10T11:30:46.915 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:46.912 7f9aad7fa700 1 -- 172.21.2.102:0/3056582624 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f9a98005310 con 0x7f9ab00a5ef0 2021-02-10T11:30:46.916 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:46.912 7f9aad7fa700 1 --2- 172.21.2.102:0/3056582624 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f9a940186c0 0x7f9a9401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:46.916 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:46.912 7f9aad7fa700 1 -- 172.21.2.102:0/3056582624 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f9a98032530 con 0x7f9ab00a5ef0 2021-02-10T11:30:46.916 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:46.912 7f9aaffff700 1 --2- 172.21.2.102:0/3056582624 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f9a940186c0 0x7f9a9401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:46.916 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:46.912 7f9aaffff700 1 --2- 172.21.2.102:0/3056582624 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f9a940186c0 0x7f9a9401abc0 crc :-1 s=READY pgs=904 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:46.944 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:46.943 7f9ab6b6c700 1 -- 172.21.2.102:0/3056582624 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ab00ee0b0 con 0x7f9ab00a5ef0 2021-02-10T11:30:46.946 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:46.946 7f9aad7fa700 1 -- 172.21.2.102:0/3056582624 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f9a98006b90 con 0x7f9ab00a5ef0 2021-02-10T11:30:47.075 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:47.074 7f9ab6b6c700 1 -- 172.21.2.102:0/3056582624 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f9ab00ce450 con 0x7f9ab00a5ef0 2021-02-10T11:30:47.850 INFO:tasks.ceph.mon.a.gibba002.stderr:2021-02-10 11:30:47.849 7f8dd2755700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-02-10T11:30:47.940 INFO:tasks.ceph.mds.b.gibba013.stdout:starting mds.b at 2021-02-10T11:30:47.941 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:47.854 7f9aad7fa700 1 -- 172.21.2.102:0/3056582624 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v75) v1 ==== 161+0+0 (secure 0 0 0) 0x7f9a98005310 con 0x7f9ab00a5ef0 2021-02-10T11:30:47.941 INFO:teuthology.orchestra.run.gibba002.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2021-02-10T11:30:47.941 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:47.893 7f9ab6b6c700 1 -- 172.21.2.102:0/3056582624 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f9a940186c0 msgr2=0x7f9a9401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:47.941 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:47.893 7f9ab6b6c700 1 --2- 172.21.2.102:0/3056582624 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f9a940186c0 0x7f9a9401abc0 crc :-1 s=READY pgs=904 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:47.941 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:47.893 7f9ab6b6c700 1 -- 172.21.2.102:0/3056582624 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9ab00a5ef0 msgr2=0x7f9ab0188f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:47.941 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:47.893 7f9ab6b6c700 1 --2- 172.21.2.102:0/3056582624 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9ab00a5ef0 0x7f9ab0188f50 secure :-1 s=READY pgs=2028 cs=0 l=1 rev1=1 rx=0x7f9a9800b530 tx=0x7f9a9800b8d0).stop 2021-02-10T11:30:47.941 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:47.893 7f9ab6b6c700 1 -- 172.21.2.102:0/3056582624 shutdown_connections 2021-02-10T11:30:47.941 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:47.893 7f9ab6b6c700 1 --2- 172.21.2.102:0/3056582624 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f9a940186c0 0x7f9a9401abc0 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:47.941 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:47.893 7f9ab6b6c700 1 --2- 172.21.2.102:0/3056582624 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f9ab0189b90 0x7f9ab008cb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:47.941 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:47.893 7f9ab6b6c700 1 --2- 172.21.2.102:0/3056582624 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9ab00a62f0 0x7f9ab0189570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:47.941 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:47.893 7f9ab6b6c700 1 --2- 172.21.2.102:0/3056582624 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9ab00a5ef0 0x7f9ab0188f50 unknown :-1 s=CLOSED pgs=2028 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:47.942 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:47.893 7f9ab6b6c700 1 -- 172.21.2.102:0/3056582624 >> 172.21.2.102:0/3056582624 conn(0x7f9ab00a6eb0 msgr2=0x7f9ab0098ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:47.942 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:47.894 7f9ab6b6c700 1 -- 172.21.2.102:0/3056582624 shutdown_connections 2021-02-10T11:30:47.942 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:47.894 7f9ab6b6c700 1 -- 172.21.2.102:0/3056582624 wait complete. 2021-02-10T11:30:47.942 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:30:48.371 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:48.354 7f23f7c69700 1 Processor -- start 2021-02-10T11:30:48.371 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:48.354 7f23f7c69700 1 -- start start 2021-02-10T11:30:48.371 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:48.354 7f23f7c69700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f23f00d1870 0x7f23f01b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:48.371 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:48.354 7f23f7c69700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23f00f0b10 con 0x7f23f01b0a10 2021-02-10T11:30:48.372 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:48.354 7f23f7c69700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23f00d0310 con 0x7f23f00f0650 2021-02-10T11:30:48.372 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:48.354 7f23f7c69700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23f00d0530 con 0x7f23f00d1870 2021-02-10T11:30:48.372 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:48.354 7f23f5a85700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f23f00d1870 0x7f23f01b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:48.372 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:48.354 7f23f5284700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f23f01b0a10 0x7f23f00cf330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:53526/0 (socket says 172.21.2.102:53526) 2021-02-10T11:30:48.372 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:48.354 7f23f5284700 1 -- 172.21.2.102:0/3670950959 learned_addr learned my addr 172.21.2.102:0/3670950959 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:30:48.372 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:48.355 7f23f5a85700 1 -- 172.21.2.102:0/3670950959 >> v1:172.21.2.113:6789/0 conn(0x7f23f00f0650 legacy=0x7f23f00cfb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:48.372 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:48.355 7f23f5a85700 1 -- 172.21.2.102:0/3670950959 >> v1:172.21.2.102:6789/0 conn(0x7f23f01b0a10 legacy=0x7f23f00cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:48.372 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:48.355 7f23f5a85700 1 -- 172.21.2.102:0/3670950959 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23f00d1c70 con 0x7f23f00d1870 2021-02-10T11:30:48.372 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:48.355 7f23f5a85700 1 --2- 172.21.2.102:0/3670950959 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f23f00d1870 0x7f23f01b03f0 secure :-1 s=READY pgs=2031 cs=0 l=1 rev1=1 rx=0x7f23e40069d0 tx=0x7f23e4009e40).ready entity=mon.0 client_cookie=5846a51e2aebb777 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:48.372 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:48.355 7f23f4a83700 1 -- 172.21.2.102:0/3670950959 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f23e400b510 con 0x7f23f00d1870 2021-02-10T11:30:48.372 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:48.355 7f23f4a83700 1 -- 172.21.2.102:0/3670950959 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23e400b790 con 0x7f23f00d1870 2021-02-10T11:30:48.372 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:48.355 7f23f4a83700 1 -- 172.21.2.102:0/3670950959 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f23e400bcb0 con 0x7f23f00d1870 2021-02-10T11:30:48.373 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:48.355 7f23f7c69700 1 -- 172.21.2.102:0/3670950959 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f23f00d1870 msgr2=0x7f23f01b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:48.373 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:48.355 7f23f7c69700 1 --2- 172.21.2.102:0/3670950959 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f23f00d1870 0x7f23f01b03f0 secure :-1 s=READY pgs=2031 cs=0 l=1 rev1=1 rx=0x7f23e40069d0 tx=0x7f23e4009e40).stop 2021-02-10T11:30:48.373 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:48.356 7f23f7c69700 1 -- 172.21.2.102:0/3670950959 shutdown_connections 2021-02-10T11:30:48.373 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:48.356 7f23f7c69700 1 --2- 172.21.2.102:0/3670950959 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f23f00d1870 0x7f23f01b03f0 unknown :-1 s=CLOSED pgs=2031 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:48.373 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:48.356 7f23f7c69700 1 -- 172.21.2.102:0/3670950959 >> 172.21.2.102:0/3670950959 conn(0x7f23f00ce450 msgr2=0x7f23f00d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:48.373 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:48.356 7f23f7c69700 1 -- 172.21.2.102:0/3670950959 shutdown_connections 2021-02-10T11:30:48.373 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:48.356 7f23f7c69700 1 -- 172.21.2.102:0/3670950959 wait complete. 2021-02-10T11:30:48.375 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:48.371 7f23f7c69700 1 Processor -- start 2021-02-10T11:30:48.375 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:48.371 7f23f7c69700 1 -- start start 2021-02-10T11:30:48.375 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:48.371 7f23f7c69700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f23f00ae6b0 0x7f23f00a8bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:48.375 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:48.371 7f23f7c69700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f23f00adbd0 0x7f23f00a9a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:48.375 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:48.371 7f23f7c69700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f23f00adfd0 0x7f23f00aa090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:48.375 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:48.371 7f23f7c69700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23f01bf350 con 0x7f23f00adfd0 2021-02-10T11:30:48.375 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:48.371 7f23f7c69700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23f00a9560 con 0x7f23f00adbd0 2021-02-10T11:30:48.375 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:48.371 7f23f7c69700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23f00a9210 con 0x7f23f00ae6b0 2021-02-10T11:30:48.375 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:48.372 7f23f6286700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f23f00adfd0 0x7f23f00aa090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:48.375 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:48.372 7f23f6286700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f23f00adfd0 0x7f23f00aa090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:33080/0 (socket says 172.21.2.102:33080) 2021-02-10T11:30:48.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:48.372 7f23f6286700 1 -- 172.21.2.102:0/3920666631 learned_addr learned my addr 172.21.2.102:0/3920666631 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:30:48.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:48.372 7f23f6286700 1 -- 172.21.2.102:0/3920666631 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f23f00ae6b0 msgr2=0x7f23f00a8bc0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:30:48.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:48.372 7f23f6286700 1 --2- 172.21.2.102:0/3920666631 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f23f00ae6b0 0x7f23f00a8bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:48.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:48.372 7f23f6286700 1 -- 172.21.2.102:0/3920666631 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f23f00adbd0 msgr2=0x7f23f00a9a70 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:30:48.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:48.372 7f23f6286700 1 --2- 172.21.2.102:0/3920666631 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f23f00adbd0 0x7f23f00a9a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:48.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:48.372 7f23f6286700 1 -- 172.21.2.102:0/3920666631 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23f00aeab0 con 0x7f23f00adfd0 2021-02-10T11:30:48.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:48.372 7f23f6286700 1 --2- 172.21.2.102:0/3920666631 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f23f00adfd0 0x7f23f00aa090 secure :-1 s=READY pgs=2032 cs=0 l=1 rev1=1 rx=0x7f23e8006ed0 tx=0x7f23e800b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:48.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:48.372 7f23e27fc700 1 -- 172.21.2.102:0/3920666631 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f23e800bd50 con 0x7f23f00adfd0 2021-02-10T11:30:48.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:48.372 7f23e27fc700 1 -- 172.21.2.102:0/3920666631 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23e8006020 con 0x7f23f00adfd0 2021-02-10T11:30:48.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:48.372 7f23f7c69700 1 -- 172.21.2.102:0/3920666631 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23f00aeab0 con 0x7f23f00adfd0 2021-02-10T11:30:48.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:48.372 7f23e27fc700 1 -- 172.21.2.102:0/3920666631 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f23e80054b0 con 0x7f23f00adfd0 2021-02-10T11:30:48.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:48.372 7f23f7c69700 1 -- 172.21.2.102:0/3920666631 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23f00aeab0 con 0x7f23f00adfd0 2021-02-10T11:30:48.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:48.373 7f23e27fc700 1 -- 172.21.2.102:0/3920666631 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f23e8006200 con 0x7f23f00adfd0 2021-02-10T11:30:48.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:48.373 7f23e27fc700 1 --2- 172.21.2.102:0/3920666631 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f23d8018730 0x7f23d801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:48.377 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:48.373 7f23e27fc700 1 -- 172.21.2.102:0/3920666631 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5659+0+0 (secure 0 0 0) 0x7f23e80326b0 con 0x7f23f00adfd0 2021-02-10T11:30:48.377 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:48.373 7f23f5a85700 1 --2- 172.21.2.102:0/3920666631 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f23d8018730 0x7f23d801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:48.377 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:48.373 7f23f5a85700 1 --2- 172.21.2.102:0/3920666631 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f23d8018730 0x7f23d801ac10 crc :-1 s=READY pgs=905 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:48.405 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:48.404 7f23f7c69700 1 -- 172.21.2.102:0/3920666631 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23f00e7de0 con 0x7f23f00adfd0 2021-02-10T11:30:48.408 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:48.407 7f23e27fc700 1 -- 172.21.2.102:0/3920666631 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f23e8006230 con 0x7f23f00adfd0 2021-02-10T11:30:48.535 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:48.534 7f23f7c69700 1 -- 172.21.2.102:0/3920666631 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f23f00e7de0 con 0x7f23f00adfd0 2021-02-10T11:30:48.535 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:48.535 7f23e27fc700 1 -- 172.21.2.102:0/3920666631 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 75 v75) v1 ==== 94+0+1756 (secure 0 0 0) 0x7f23e8006000 con 0x7f23f00adfd0 2021-02-10T11:30:48.536 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:30:48.536 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":75,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9672,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.102:6835/2071608955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6834","nonce":2071608955},{"type":"v1","addr":"172.21.2.102:6835","nonce":2071608955}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":69}],"filesystems":[{"mdsmap":{"epoch":75,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:30:04.225997","modified":"2021-02-10 11:30:47.850933","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":94,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":9}]} 2021-02-10T11:30:48.536 INFO:teuthology.orchestra.run.gibba002.stderr:dumped fsmap epoch 75 2021-02-10T11:30:48.537 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:48.537 7f23f7c69700 1 -- 172.21.2.102:0/3920666631 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f23d8018730 msgr2=0x7f23d801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:48.537 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:48.537 7f23f7c69700 1 --2- 172.21.2.102:0/3920666631 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f23d8018730 0x7f23d801ac10 crc :-1 s=READY pgs=905 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:48.537 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:48.537 7f23f7c69700 1 -- 172.21.2.102:0/3920666631 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f23f00adfd0 msgr2=0x7f23f00aa090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:48.537 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:48.537 7f23f7c69700 1 --2- 172.21.2.102:0/3920666631 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f23f00adfd0 0x7f23f00aa090 secure :-1 s=READY pgs=2032 cs=0 l=1 rev1=1 rx=0x7f23e8006ed0 tx=0x7f23e800b960).stop 2021-02-10T11:30:48.538 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:48.537 7f23f7c69700 1 -- 172.21.2.102:0/3920666631 shutdown_connections 2021-02-10T11:30:48.538 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:48.537 7f23f7c69700 1 --2- 172.21.2.102:0/3920666631 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f23f00adfd0 0x7f23f00aa090 unknown :-1 s=CLOSED pgs=2032 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:48.538 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:48.537 7f23f7c69700 1 --2- 172.21.2.102:0/3920666631 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f23d8018730 0x7f23d801ac10 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:48.538 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:48.537 7f23f7c69700 1 --2- 172.21.2.102:0/3920666631 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f23f00adbd0 0x7f23f00a9a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:48.538 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:48.537 7f23f7c69700 1 --2- 172.21.2.102:0/3920666631 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f23f00ae6b0 0x7f23f00a8bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:48.538 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:48.537 7f23f7c69700 1 -- 172.21.2.102:0/3920666631 >> 172.21.2.102:0/3920666631 conn(0x7f23f00a0030 msgr2=0x7f23f00eb1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:48.538 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:48.537 7f23f7c69700 1 -- 172.21.2.102:0/3920666631 shutdown_connections 2021-02-10T11:30:48.539 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:48.537 7f23f7c69700 1 -- 172.21.2.102:0/3920666631 wait complete. 2021-02-10T11:30:48.546 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2021-02-10T11:30:48.979 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:48.960 7f47d253a700 1 Processor -- start 2021-02-10T11:30:48.979 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:48.960 7f47d253a700 1 -- start start 2021-02-10T11:30:48.979 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:48.961 7f47d253a700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f47cc0d1c70 0x7f47cc0f20a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:48.979 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:48.961 7f47d253a700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47cc0f0b10 con 0x7f47cc0d1870 2021-02-10T11:30:48.979 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:48.961 7f47d253a700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47cc1b0be0 con 0x7f47cc0d0340 2021-02-10T11:30:48.979 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:48.961 7f47d253a700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47cc0e5ed0 con 0x7f47cc0d1c70 2021-02-10T11:30:48.979 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:48.961 7f47cbfff700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f47cc0d1870 0x7f47cc1b03f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:53540/0 (socket says 172.21.2.102:53540) 2021-02-10T11:30:48.979 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:48.961 7f47cbfff700 1 -- 172.21.2.102:0/526879904 learned_addr learned my addr 172.21.2.102:0/526879904 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:30:48.979 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:48.961 7f47caffd700 1 -- 172.21.2.102:0/526879904 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1981172874 0 0) 0x7f47bc000ce0 con 0x7f47cc0d1870 2021-02-10T11:30:48.979 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:48.961 7f47caffd700 1 -- 172.21.2.102:0/526879904 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f47b4002fe0 con 0x7f47cc0d1870 2021-02-10T11:30:48.980 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:48.962 7f47d0b57700 1 --2- 172.21.2.102:0/526879904 >> v2:172.21.2.113:3300/0 conn(0x7f47cc0d1c70 0x7f47cc0f20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:48.980 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:48.962 7f47caffd700 1 -- 172.21.2.102:0/526879904 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1524531777 0 0) 0x7f47bc0010e0 con 0x7f47cc0d1870 2021-02-10T11:30:48.980 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:48.962 7f47caffd700 1 -- 172.21.2.102:0/526879904 >> v2:172.21.2.113:3300/0 conn(0x7f47cc0d1c70 msgr2=0x7f47cc0f20a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:48.980 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:48.962 7f47caffd700 1 --2- 172.21.2.102:0/526879904 >> v2:172.21.2.113:3300/0 conn(0x7f47cc0d1c70 0x7f47cc0f20a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:48.980 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:48.962 7f47caffd700 1 -- 172.21.2.102:0/526879904 >> v1:172.21.2.113:6789/0 conn(0x7f47cc0d0340 legacy=0x7f47cc0ef640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:48.980 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:48.962 7f47caffd700 1 -- 172.21.2.102:0/526879904 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47cc0efe30 con 0x7f47cc0d1870 2021-02-10T11:30:48.980 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:48.962 7f47caffd700 1 -- 172.21.2.102:0/526879904 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7f47bc001660 con 0x7f47cc0d1870 2021-02-10T11:30:48.980 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:48.962 7f47caffd700 1 -- 172.21.2.102:0/526879904 >> v1:172.21.2.102:6789/0 conn(0x7f47cc0d1870 legacy=0x7f47cc1b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:48.980 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:48.962 7f47caffd700 1 --2- 172.21.2.102:0/526879904 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f47b4003b70 0x7f47b4005f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:48.980 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:48.962 7f47caffd700 1 --2- 172.21.2.102:0/526879904 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f47b4006820 0x7f47b4008d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:48.980 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:48.962 7f47caffd700 1 --2- 172.21.2.102:0/526879904 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f47b4009640 0x7f47b400bb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:48.980 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:48.962 7f47caffd700 1 -- 172.21.2.102:0/526879904 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47b400c390 con 0x7f47b4006820 2021-02-10T11:30:48.980 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:48.962 7f47caffd700 1 -- 172.21.2.102:0/526879904 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47b400c5b0 con 0x7f47b4003b70 2021-02-10T11:30:48.980 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:48.962 7f47caffd700 1 -- 172.21.2.102:0/526879904 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47b400c7d0 con 0x7f47b4009640 2021-02-10T11:30:48.981 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:48.962 7f47d0b57700 1 --2- 172.21.2.102:0/526879904 >> v2:172.21.2.113:3300/0 conn(0x7f47cc0d1c70 0x7f47cc0f20a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:30:48.981 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:48.962 7f47cb7fe700 1 --2- 172.21.2.102:0/526879904 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f47b4006820 0x7f47b4008d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:48.981 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:48.962 7f47cbfff700 1 --2- 172.21.2.102:0/526879904 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f47b4003b70 0x7f47b4005f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:48.981 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:48.962 7f47cb7fe700 1 -- 172.21.2.102:0/526879904 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f47b4009640 msgr2=0x7f47b400bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:48.981 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:48.962 7f47cb7fe700 1 --2- 172.21.2.102:0/526879904 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f47b4009640 0x7f47b400bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:48.981 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:48.962 7f47cb7fe700 1 -- 172.21.2.102:0/526879904 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f47b4003b70 msgr2=0x7f47b4005f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:48.981 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:48.962 7f47cb7fe700 1 --2- 172.21.2.102:0/526879904 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f47b4003b70 0x7f47b4005f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:48.981 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:48.962 7f47cb7fe700 1 -- 172.21.2.102:0/526879904 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47b400ca70 con 0x7f47b4006820 2021-02-10T11:30:48.981 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:48.962 7f47d0b57700 1 -- 172.21.2.102:0/526879904 reap_dead start 2021-02-10T11:30:48.981 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:48.963 7f47cb7fe700 1 --2- 172.21.2.102:0/526879904 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f47b4006820 0x7f47b4008d40 secure :-1 s=READY pgs=2034 cs=0 l=1 rev1=1 rx=0x7f47c0006ab0 tx=0x7f47c0009b40).ready entity=mon.0 client_cookie=79eaeafe7429ee5e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:48.981 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:48.963 7f47caffd700 1 -- 172.21.2.102:0/526879904 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f47c00066a0 con 0x7f47b4006820 2021-02-10T11:30:48.981 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:48.963 7f47caffd700 1 -- 172.21.2.102:0/526879904 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47c0000e20 con 0x7f47b4006820 2021-02-10T11:30:48.981 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:48.963 7f47caffd700 1 -- 172.21.2.102:0/526879904 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f47c000b4d0 con 0x7f47b4006820 2021-02-10T11:30:48.982 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:48.963 7f47d253a700 1 -- 172.21.2.102:0/526879904 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f47b4006820 msgr2=0x7f47b4008d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:48.982 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:48.963 7f47d253a700 1 --2- 172.21.2.102:0/526879904 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f47b4006820 0x7f47b4008d40 secure :-1 s=READY pgs=2034 cs=0 l=1 rev1=1 rx=0x7f47c0006ab0 tx=0x7f47c0009b40).stop 2021-02-10T11:30:48.982 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:48.963 7f47d253a700 1 -- 172.21.2.102:0/526879904 shutdown_connections 2021-02-10T11:30:48.982 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:48.963 7f47d253a700 1 --2- 172.21.2.102:0/526879904 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f47b4006820 0x7f47b4008d40 unknown :-1 s=CLOSED pgs=2034 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:48.982 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:48.963 7f47d253a700 1 -- 172.21.2.102:0/526879904 >> 172.21.2.102:0/526879904 conn(0x7f47cc0ce450 msgr2=0x7f47cc0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:48.982 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:48.963 7f47d253a700 1 -- 172.21.2.102:0/526879904 shutdown_connections 2021-02-10T11:30:48.982 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:48.963 7f47d253a700 1 -- 172.21.2.102:0/526879904 wait complete. 2021-02-10T11:30:48.982 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:48.978 7f47d253a700 1 Processor -- start 2021-02-10T11:30:48.982 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:48.978 7f47d253a700 1 -- start start 2021-02-10T11:30:48.983 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:48.978 7f47d253a700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f47cc0ebe70 0x7f47cc0ec270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:48.983 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:48.978 7f47d253a700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f47cc093830 0x7f47cc093c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:48.983 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:48.978 7f47d253a700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f47cc094250 0x7f47cc094650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:48.984 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:48.978 7f47d253a700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47cc0ec940 con 0x7f47cc093830 2021-02-10T11:30:48.984 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:48.978 7f47d253a700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47cc0ece40 con 0x7f47cc094250 2021-02-10T11:30:48.984 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:48.978 7f47d253a700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47cc0ed060 con 0x7f47cc0ebe70 2021-02-10T11:30:48.984 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:48.978 7f47cb7fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f47cc093830 0x7f47cc093c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:48.984 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:48.978 7f47cb7fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f47cc093830 0x7f47cc093c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:33094/0 (socket says 172.21.2.102:33094) 2021-02-10T11:30:48.984 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:48.978 7f47cb7fe700 1 -- 172.21.2.102:0/1808141645 learned_addr learned my addr 172.21.2.102:0/1808141645 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:30:48.984 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:48.979 7f47cb7fe700 1 -- 172.21.2.102:0/1808141645 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f47cc0ebe70 msgr2=0x7f47cc0ec270 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:30:48.984 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:48.979 7f47cb7fe700 1 --2- 172.21.2.102:0/1808141645 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f47cc0ebe70 0x7f47cc0ec270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:48.984 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:48.979 7f47cb7fe700 1 -- 172.21.2.102:0/1808141645 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f47cc094250 msgr2=0x7f47cc094650 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:30:48.984 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:48.979 7f47cb7fe700 1 --2- 172.21.2.102:0/1808141645 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f47cc094250 0x7f47cc094650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:48.984 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:48.979 7f47cb7fe700 1 -- 172.21.2.102:0/1808141645 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47cc0ed280 con 0x7f47cc093830 2021-02-10T11:30:48.984 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:48.979 7f47cb7fe700 1 --2- 172.21.2.102:0/1808141645 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f47cc093830 0x7f47cc093c30 secure :-1 s=READY pgs=2035 cs=0 l=1 rev1=1 rx=0x7f47c000b630 tx=0x7f47c000b9d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:48.984 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:48.979 7f47c8ff9700 1 -- 172.21.2.102:0/1808141645 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f47c00093e0 con 0x7f47cc093830 2021-02-10T11:30:48.984 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:48.979 7f47c8ff9700 1 -- 172.21.2.102:0/1808141645 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47c0002030 con 0x7f47cc093830 2021-02-10T11:30:48.985 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:48.979 7f47c8ff9700 1 -- 172.21.2.102:0/1808141645 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f47c0002360 con 0x7f47cc093830 2021-02-10T11:30:48.985 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:48.979 7f47d253a700 1 -- 172.21.2.102:0/1808141645 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47cc0ed280 con 0x7f47cc093830 2021-02-10T11:30:48.985 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:48.979 7f47d253a700 1 -- 172.21.2.102:0/1808141645 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47cc0ed810 con 0x7f47cc093830 2021-02-10T11:30:48.985 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:48.980 7f47c8ff9700 1 -- 172.21.2.102:0/1808141645 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f47c0002210 con 0x7f47cc093830 2021-02-10T11:30:48.985 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:48.980 7f47c8ff9700 1 --2- 172.21.2.102:0/1808141645 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f47ac0186c0 0x7f47ac01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:48.985 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:48.980 7f47c8ff9700 1 -- 172.21.2.102:0/1808141645 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5659+0+0 (secure 0 0 0) 0x7f47c0032470 con 0x7f47cc093830 2021-02-10T11:30:48.985 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:48.980 7f47cbfff700 1 --2- 172.21.2.102:0/1808141645 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f47ac0186c0 0x7f47ac01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:48.985 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:48.980 7f47cbfff700 1 --2- 172.21.2.102:0/1808141645 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f47ac0186c0 0x7f47ac01abc0 crc :-1 s=READY pgs=906 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:49.012 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:49.011 7f47d253a700 1 -- 172.21.2.102:0/1808141645 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47cc0ebc40 con 0x7f47cc093830 2021-02-10T11:30:49.014 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:49.013 7f47c8ff9700 1 -- 172.21.2.102:0/1808141645 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f47c002a660 con 0x7f47cc093830 2021-02-10T11:30:49.141 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:49.140 7f47d253a700 1 -- 172.21.2.102:0/1808141645 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f47cc0f0b10 con 0x7f47cc093830 2021-02-10T11:30:49.865 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:49.864 7f47c8ff9700 1 -- 172.21.2.102:0/1808141645 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v76) v1 ==== 104+0+0 (secure 0 0 0) 0x7f47c002a660 con 0x7f47cc093830 2021-02-10T11:30:49.912 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:49.910 7f47d253a700 1 -- 172.21.2.102:0/1808141645 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f47ac0186c0 msgr2=0x7f47ac01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:49.912 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:49.910 7f47d253a700 1 --2- 172.21.2.102:0/1808141645 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f47ac0186c0 0x7f47ac01abc0 crc :-1 s=READY pgs=906 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:49.912 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:49.910 7f47d253a700 1 -- 172.21.2.102:0/1808141645 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f47cc093830 msgr2=0x7f47cc093c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:49.912 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:49.910 7f47d253a700 1 --2- 172.21.2.102:0/1808141645 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f47cc093830 0x7f47cc093c30 secure :-1 s=READY pgs=2035 cs=0 l=1 rev1=1 rx=0x7f47c000b630 tx=0x7f47c000b9d0).stop 2021-02-10T11:30:49.912 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:49.910 7f47d253a700 1 -- 172.21.2.102:0/1808141645 shutdown_connections 2021-02-10T11:30:49.912 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:49.910 7f47d253a700 1 --2- 172.21.2.102:0/1808141645 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f47ac0186c0 0x7f47ac01abc0 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:49.912 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:49.910 7f47d253a700 1 --2- 172.21.2.102:0/1808141645 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f47cc094250 0x7f47cc094650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:49.912 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:49.910 7f47d253a700 1 --2- 172.21.2.102:0/1808141645 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f47cc093830 0x7f47cc093c30 unknown :-1 s=CLOSED pgs=2035 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:49.912 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:49.910 7f47d253a700 1 --2- 172.21.2.102:0/1808141645 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f47cc0ebe70 0x7f47cc0ec270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:49.912 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:49.910 7f47d253a700 1 -- 172.21.2.102:0/1808141645 >> 172.21.2.102:0/1808141645 conn(0x7f47cc08fa10 msgr2=0x7f47cc0bfa60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:49.913 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:49.911 7f47d253a700 1 -- 172.21.2.102:0/1808141645 shutdown_connections 2021-02-10T11:30:49.913 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:49.911 7f47d253a700 1 -- 172.21.2.102:0/1808141645 wait complete. 2021-02-10T11:30:49.920 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool delete cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2021-02-10T11:30:50.354 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:50.336 7f4b4cfb5700 1 Processor -- start 2021-02-10T11:30:50.354 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:50.336 7f4b4cfb5700 1 -- start start 2021-02-10T11:30:50.354 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:50.337 7f4b4cfb5700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4b480d1880 0x7f4b481b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:50.354 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:50.337 7f4b4cfb5700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f4b480cff80 0x7f4b48109140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:50.355 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:50.337 7f4b4cfb5700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b48109a70 con 0x7f4b481b0af0 2021-02-10T11:30:50.355 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:50.337 7f4b4cfb5700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b48109760 con 0x7f4b480cff80 2021-02-10T11:30:50.355 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:50.337 7f4b4cfb5700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b48088070 con 0x7f4b480d1880 2021-02-10T11:30:50.355 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:50.337 7f4b4759e700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f4b480cff80 0x7f4b48109140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:50.355 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:50.337 7f4b46d9d700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4b480d1880 0x7f4b481b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:50.355 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:50.337 7f4b4759e700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f4b480cff80 0x7f4b48109140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.113:3300/0 says I am v2:172.21.2.102:36536/0 (socket says 172.21.2.102:36536) 2021-02-10T11:30:50.355 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:50.337 7f4b4759e700 1 -- 172.21.2.102:0/273499520 learned_addr learned my addr 172.21.2.102:0/273499520 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:30:50.355 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:50.337 7f4b4659c700 1 --1- 172.21.2.102:0/273499520 >> v1:172.21.2.113:6789/0 conn(0x7f4b481b0af0 0x7f4b48108950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.113:6789/0 says I am v1:172.21.2.102:60884/0 (socket says 172.21.2.102:60884) 2021-02-10T11:30:50.355 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:50.338 7f4b46d9d700 1 -- 172.21.2.102:0/273499520 >> v2:172.21.2.113:3300/0 conn(0x7f4b480cff80 msgr2=0x7f4b48109140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:50.355 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:50.338 7f4b46d9d700 1 --2- 172.21.2.102:0/273499520 >> v2:172.21.2.113:3300/0 conn(0x7f4b480cff80 0x7f4b48109140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:50.355 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:50.338 7f4b46d9d700 1 -- 172.21.2.102:0/273499520 >> v1:172.21.2.113:6789/0 conn(0x7f4b481b0af0 legacy=0x7f4b48108950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:50.355 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:50.338 7f4b46d9d700 1 -- 172.21.2.102:0/273499520 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b48088260 con 0x7f4b480d1880 2021-02-10T11:30:50.355 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:50.338 7f4b46d9d700 1 --2- 172.21.2.102:0/273499520 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4b480d1880 0x7f4b481b04d0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7f4b30006650 tx=0x7f4b30009a80).ready entity=mon.2 client_cookie=7ef10dff8bd9d675 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:50.356 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:50.338 7f4b45d9b700 1 -- 172.21.2.102:0/273499520 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4b3000c0c0 con 0x7f4b480d1880 2021-02-10T11:30:50.356 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:50.338 7f4b45d9b700 1 -- 172.21.2.102:0/273499520 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b3000c340 con 0x7f4b480d1880 2021-02-10T11:30:50.356 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:50.338 7f4b45d9b700 1 -- 172.21.2.102:0/273499520 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4b3000c860 con 0x7f4b480d1880 2021-02-10T11:30:50.356 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:50.338 7f4b4cfb5700 1 -- 172.21.2.102:0/273499520 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4b480d1880 msgr2=0x7f4b481b04d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:50.356 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:50.338 7f4b4cfb5700 1 --2- 172.21.2.102:0/273499520 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4b480d1880 0x7f4b481b04d0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7f4b30006650 tx=0x7f4b30009a80).stop 2021-02-10T11:30:50.356 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:50.339 7f4b4cfb5700 1 -- 172.21.2.102:0/273499520 shutdown_connections 2021-02-10T11:30:50.356 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:50.339 7f4b4cfb5700 1 --2- 172.21.2.102:0/273499520 >> v2:172.21.2.113:3300/0 conn(0x7f4b480cff80 0x7f4b48109140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:50.356 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:50.339 7f4b4cfb5700 1 --2- 172.21.2.102:0/273499520 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4b480d1880 0x7f4b481b04d0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:50.356 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:50.339 7f4b4cfb5700 1 -- 172.21.2.102:0/273499520 >> 172.21.2.102:0/273499520 conn(0x7f4b480ce430 msgr2=0x7f4b480d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:50.356 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:50.339 7f4b4cfb5700 1 -- 172.21.2.102:0/273499520 shutdown_connections 2021-02-10T11:30:50.356 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:50.339 7f4b4cfb5700 1 -- 172.21.2.102:0/273499520 wait complete. 2021-02-10T11:30:50.358 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:50.354 7f4b4cfb5700 1 Processor -- start 2021-02-10T11:30:50.358 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:50.354 7f4b4cfb5700 1 -- start start 2021-02-10T11:30:50.358 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:50.354 7f4b4cfb5700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4b481b9150 0x7f4b48197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:50.358 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:50.354 7f4b4cfb5700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4b481985f0 0x7f4b481b8060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:50.358 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:50.354 7f4b4cfb5700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4b481b8680 0x7f4b481ae590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:50.358 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:50.354 7f4b4cfb5700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b48187fe0 con 0x7f4b481b9150 2021-02-10T11:30:50.358 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:50.354 7f4b4cfb5700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b481b8a80 con 0x7f4b481985f0 2021-02-10T11:30:50.358 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:50.354 7f4b4cfb5700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b481b8ca0 con 0x7f4b481b8680 2021-02-10T11:30:50.358 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:50.354 7f4b46d9d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4b481b9150 0x7f4b48197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:50.359 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:50.354 7f4b46d9d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4b481b9150 0x7f4b48197fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:33110/0 (socket says 172.21.2.102:33110) 2021-02-10T11:30:50.359 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:50.354 7f4b46d9d700 1 -- 172.21.2.102:0/2674074566 learned_addr learned my addr 172.21.2.102:0/2674074566 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:30:50.359 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:50.355 7f4b4659c700 1 --2- 172.21.2.102:0/2674074566 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4b481985f0 0x7f4b481b8060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:50.359 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:50.355 7f4b46d9d700 1 -- 172.21.2.102:0/2674074566 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4b481b8680 msgr2=0x7f4b481ae590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:50.359 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:50.355 7f4b46d9d700 1 --2- 172.21.2.102:0/2674074566 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4b481b8680 0x7f4b481ae590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:50.359 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:50.355 7f4b46d9d700 1 -- 172.21.2.102:0/2674074566 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4b481985f0 msgr2=0x7f4b481b8060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:50.359 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:50.355 7f4b46d9d700 1 --2- 172.21.2.102:0/2674074566 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4b481985f0 0x7f4b481b8060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:50.359 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:50.355 7f4b46d9d700 1 -- 172.21.2.102:0/2674074566 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b481aebb0 con 0x7f4b481b9150 2021-02-10T11:30:50.359 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:50.355 7f4b4659c700 1 --2- 172.21.2.102:0/2674074566 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4b481985f0 0x7f4b481b8060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:30:50.359 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:50.355 7f4b46d9d700 1 --2- 172.21.2.102:0/2674074566 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4b481b9150 0x7f4b48197fd0 secure :-1 s=READY pgs=2036 cs=0 l=1 rev1=1 rx=0x7f4b30009630 tx=0x7f4b3000b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:50.359 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:50.355 7f4b3f7fe700 1 -- 172.21.2.102:0/2674074566 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4b300093a0 con 0x7f4b481b9150 2021-02-10T11:30:50.359 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:50.355 7f4b3f7fe700 1 -- 172.21.2.102:0/2674074566 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b3000bd70 con 0x7f4b481b9150 2021-02-10T11:30:50.359 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:50.355 7f4b4cfb5700 1 -- 172.21.2.102:0/2674074566 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b481aebb0 con 0x7f4b481b9150 2021-02-10T11:30:50.360 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:50.355 7f4b4cfb5700 1 -- 172.21.2.102:0/2674074566 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b481af140 con 0x7f4b481b9150 2021-02-10T11:30:50.360 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:50.355 7f4b3f7fe700 1 -- 172.21.2.102:0/2674074566 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4b300054d0 con 0x7f4b481b9150 2021-02-10T11:30:50.360 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:50.356 7f4b3f7fe700 1 -- 172.21.2.102:0/2674074566 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4b3000a000 con 0x7f4b481b9150 2021-02-10T11:30:50.360 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:50.356 7f4b3f7fe700 1 --2- 172.21.2.102:0/2674074566 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4b2c018730 0x7f4b2c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:50.360 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:50.356 7f4b3f7fe700 1 -- 172.21.2.102:0/2674074566 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5659+0+0 (secure 0 0 0) 0x7f4b30032820 con 0x7f4b481b9150 2021-02-10T11:30:50.360 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:50.356 7f4b4659c700 1 --2- 172.21.2.102:0/2674074566 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4b2c018730 0x7f4b2c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:50.360 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:50.356 7f4b4659c700 1 --2- 172.21.2.102:0/2674074566 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4b2c018730 0x7f4b2c01abc0 crc :-1 s=READY pgs=907 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:50.388 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:50.387 7f4b4cfb5700 1 -- 172.21.2.102:0/2674074566 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b481af6e0 con 0x7f4b481b9150 2021-02-10T11:30:50.390 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:50.389 7f4b3f7fe700 1 -- 172.21.2.102:0/2674074566 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f4b30009340 con 0x7f4b481b9150 2021-02-10T11:30:50.518 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:50.517 7f4b4cfb5700 1 -- 172.21.2.102:0/2674074566 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"pool2": "cephfs_metadata", "prefix": "osd pool delete", "pool": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f4b481af6e0 con 0x7f4b481b9150 2021-02-10T11:30:50.870 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:50.869 7f4b3f7fe700 1 -- 172.21.2.102:0/2674074566 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"pool2": "cephfs_metadata", "prefix": "osd pool delete", "pool": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v95) v1 ==== 185+0+0 (secure 0 0 0) 0x7f4b3000a330 con 0x7f4b481b9150 2021-02-10T11:30:50.883 INFO:teuthology.orchestra.run.gibba002.stderr:pool 'cephfs_metadata' removed 2021-02-10T11:30:50.886 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:50.884 7f4b4cfb5700 1 -- 172.21.2.102:0/2674074566 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4b2c018730 msgr2=0x7f4b2c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:50.886 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:50.884 7f4b4cfb5700 1 --2- 172.21.2.102:0/2674074566 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4b2c018730 0x7f4b2c01abc0 crc :-1 s=READY pgs=907 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:50.886 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:50.884 7f4b4cfb5700 1 -- 172.21.2.102:0/2674074566 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4b481b9150 msgr2=0x7f4b48197fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:50.886 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:50.884 7f4b4cfb5700 1 --2- 172.21.2.102:0/2674074566 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4b481b9150 0x7f4b48197fd0 secure :-1 s=READY pgs=2036 cs=0 l=1 rev1=1 rx=0x7f4b30009630 tx=0x7f4b3000b980).stop 2021-02-10T11:30:50.886 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:50.884 7f4b4cfb5700 1 -- 172.21.2.102:0/2674074566 shutdown_connections 2021-02-10T11:30:50.886 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:50.884 7f4b4cfb5700 1 --2- 172.21.2.102:0/2674074566 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4b481b8680 0x7f4b481ae590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:50.886 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:50.884 7f4b4cfb5700 1 --2- 172.21.2.102:0/2674074566 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4b2c018730 0x7f4b2c01abc0 unknown :-1 s=CLOSED pgs=907 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:50.886 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:50.884 7f4b4cfb5700 1 --2- 172.21.2.102:0/2674074566 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4b481985f0 0x7f4b481b8060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:50.886 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:50.884 7f4b4cfb5700 1 --2- 172.21.2.102:0/2674074566 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4b481b9150 0x7f4b48197fd0 unknown :-1 s=CLOSED pgs=2036 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:50.887 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:50.884 7f4b4cfb5700 1 -- 172.21.2.102:0/2674074566 >> 172.21.2.102:0/2674074566 conn(0x7f4b4819f910 msgr2=0x7f4b4818c290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:50.887 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:50.884 7f4b4cfb5700 1 -- 172.21.2.102:0/2674074566 shutdown_connections 2021-02-10T11:30:50.887 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:50.884 7f4b4cfb5700 1 -- 172.21.2.102:0/2674074566 wait complete. 2021-02-10T11:30:50.893 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool delete cephfs_data cephfs_data --yes-i-really-really-mean-it 2021-02-10T11:30:51.325 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:51.306 7f07fcc00700 1 Processor -- start 2021-02-10T11:30:51.325 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:51.306 7f07fcc00700 1 -- start start 2021-02-10T11:30:51.325 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:51.307 7f07fcc00700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f07f80d1880 0x7f07f81b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:51.325 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:51.307 7f07fcc00700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f07f81b0af0 0x7f07f8108950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:51.325 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:51.307 7f07fcc00700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07f80fe1d0 con 0x7f07f80d1c80 2021-02-10T11:30:51.325 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:51.307 7f07fcc00700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07f8109a70 con 0x7f07f80d1880 2021-02-10T11:30:51.325 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:51.307 7f07fcc00700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07f8109760 con 0x7f07f81b0af0 2021-02-10T11:30:51.325 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:51.307 7f07f705f700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f07f80d1c80 0x7f07f8108f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:53576/0 (socket says 172.21.2.102:53576) 2021-02-10T11:30:51.326 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:51.307 7f07f705f700 1 -- 172.21.2.102:0/3121096839 learned_addr learned my addr 172.21.2.102:0/3121096839 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:30:51.326 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:51.307 7f07f585c700 1 -- 172.21.2.102:0/3121096839 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3434805615 0 0) 0x7f07f0001500 con 0x7f07f80d1c80 2021-02-10T11:30:51.326 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:51.307 7f07f585c700 1 -- 172.21.2.102:0/3121096839 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f07e4002fe0 con 0x7f07f80d1c80 2021-02-10T11:30:51.326 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:51.307 7f07f685e700 1 --2- 172.21.2.102:0/3121096839 >> v2:172.21.2.113:3300/0 conn(0x7f07f80d1880 0x7f07f81b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:51.326 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:51.307 7f07f605d700 1 --2- 172.21.2.102:0/3121096839 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f07f81b0af0 0x7f07f8108950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:51.326 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:51.308 7f07f585c700 1 -- 172.21.2.102:0/3121096839 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2236138970 0 0) 0x7f07f0001270 con 0x7f07f80d1c80 2021-02-10T11:30:51.326 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:51.308 7f07f585c700 1 -- 172.21.2.102:0/3121096839 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f07f81b0af0 msgr2=0x7f07f8108950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:51.326 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:51.308 7f07f585c700 1 --2- 172.21.2.102:0/3121096839 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f07f81b0af0 0x7f07f8108950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:51.326 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:51.308 7f07f585c700 1 -- 172.21.2.102:0/3121096839 >> v2:172.21.2.113:3300/0 conn(0x7f07f80d1880 msgr2=0x7f07f81b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:51.326 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:51.308 7f07f585c700 1 --2- 172.21.2.102:0/3121096839 >> v2:172.21.2.113:3300/0 conn(0x7f07f80d1880 0x7f07f81b04d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:51.326 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:51.308 7f07f585c700 1 -- 172.21.2.102:0/3121096839 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07f8088070 con 0x7f07f80d1c80 2021-02-10T11:30:51.326 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:51.308 7f07f585c700 1 -- 172.21.2.102:0/3121096839 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7f07f0001b50 con 0x7f07f80d1c80 2021-02-10T11:30:51.327 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:51.308 7f07f585c700 1 -- 172.21.2.102:0/3121096839 >> v1:172.21.2.102:6789/0 conn(0x7f07f80d1c80 legacy=0x7f07f8108f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:51.327 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:51.308 7f07f585c700 1 --2- 172.21.2.102:0/3121096839 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f07e4003b90 0x7f07e4005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:51.327 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:51.308 7f07f585c700 1 --2- 172.21.2.102:0/3121096839 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f07e404a800 0x7f07e404cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:51.327 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:51.308 7f07f585c700 1 --2- 172.21.2.102:0/3121096839 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f07e404d530 0x7f07e404fa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:51.327 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:51.308 7f07f585c700 1 -- 172.21.2.102:0/3121096839 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07e4050300 con 0x7f07e404d530 2021-02-10T11:30:51.327 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:51.308 7f07f585c700 1 -- 172.21.2.102:0/3121096839 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07e4050520 con 0x7f07e4003b90 2021-02-10T11:30:51.327 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:51.308 7f07f585c700 1 -- 172.21.2.102:0/3121096839 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07e4050740 con 0x7f07e404a800 2021-02-10T11:30:51.327 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:51.308 7f07f685e700 1 --2- 172.21.2.102:0/3121096839 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f07e4003b90 0x7f07e4005fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:51.327 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:51.308 7f07f705f700 1 --2- 172.21.2.102:0/3121096839 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f07e404d530 0x7f07e404fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:51.327 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:51.308 7f07f605d700 1 --2- 172.21.2.102:0/3121096839 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f07e404a800 0x7f07e404cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:51.327 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:51.308 7f07f705f700 1 -- 172.21.2.102:0/3121096839 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f07e404a800 msgr2=0x7f07e404cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:51.327 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:51.308 7f07f705f700 1 --2- 172.21.2.102:0/3121096839 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f07e404a800 0x7f07e404cc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:51.327 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:51.308 7f07f705f700 1 -- 172.21.2.102:0/3121096839 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f07e4003b90 msgr2=0x7f07e4005fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:51.327 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:51.308 7f07f705f700 1 --2- 172.21.2.102:0/3121096839 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f07e4003b90 0x7f07e4005fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:51.328 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:51.308 7f07f705f700 1 -- 172.21.2.102:0/3121096839 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07e40509e0 con 0x7f07e404d530 2021-02-10T11:30:51.328 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:51.308 7f07f705f700 1 --2- 172.21.2.102:0/3121096839 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f07e404d530 0x7f07e404fa50 secure :-1 s=READY pgs=2038 cs=0 l=1 rev1=1 rx=0x7f07f0007be0 tx=0x7f07f000a060).ready entity=mon.0 client_cookie=4d8f74cbd43d7d20 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:51.328 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:51.308 7f07f585c700 1 -- 172.21.2.102:0/3121096839 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f07f0002040 con 0x7f07e404d530 2021-02-10T11:30:51.328 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:51.308 7f07f585c700 1 -- 172.21.2.102:0/3121096839 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07f00070c0 con 0x7f07e404d530 2021-02-10T11:30:51.328 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:51.308 7f07f585c700 1 -- 172.21.2.102:0/3121096839 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f07f0007510 con 0x7f07e404d530 2021-02-10T11:30:51.328 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:51.308 7f07fcc00700 1 -- 172.21.2.102:0/3121096839 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f07e404d530 msgr2=0x7f07e404fa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:51.328 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:51.308 7f07fcc00700 1 --2- 172.21.2.102:0/3121096839 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f07e404d530 0x7f07e404fa50 secure :-1 s=READY pgs=2038 cs=0 l=1 rev1=1 rx=0x7f07f0007be0 tx=0x7f07f000a060).stop 2021-02-10T11:30:51.328 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:51.308 7f07f705f700 1 -- 172.21.2.102:0/3121096839 reap_dead start 2021-02-10T11:30:51.328 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:51.309 7f07fcc00700 1 -- 172.21.2.102:0/3121096839 shutdown_connections 2021-02-10T11:30:51.328 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:51.309 7f07fcc00700 1 -- 172.21.2.102:0/3121096839 >> 172.21.2.102:0/3121096839 conn(0x7f07f80ce430 msgr2=0x7f07f80d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:51.328 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:51.309 7f07fcc00700 1 -- 172.21.2.102:0/3121096839 shutdown_connections 2021-02-10T11:30:51.328 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:51.309 7f07fcc00700 1 -- 172.21.2.102:0/3121096839 wait complete. 2021-02-10T11:30:51.328 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:51.324 7f07fcc00700 1 Processor -- start 2021-02-10T11:30:51.329 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:51.324 7f07fcc00700 1 -- start start 2021-02-10T11:30:51.330 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:51.324 7f07fcc00700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f07f8188400 0x7f07f8188800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:51.330 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:51.324 7f07fcc00700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f07f8188e20 0x7f07f8189220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:51.330 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:51.324 7f07fcc00700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f07f809cbc0 0x7f07f809cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:51.330 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:51.324 7f07fcc00700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07f8187e80 con 0x7f07f8188e20 2021-02-10T11:30:51.330 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:51.324 7f07fcc00700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07f80b27c0 con 0x7f07f809cbc0 2021-02-10T11:30:51.331 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:51.324 7f07fcc00700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07f80b29a0 con 0x7f07f8188400 2021-02-10T11:30:51.331 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:51.325 7f07f605d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f07f8188e20 0x7f07f8189220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:51.331 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:51.325 7f07f605d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f07f8188e20 0x7f07f8189220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:33132/0 (socket says 172.21.2.102:33132) 2021-02-10T11:30:51.331 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:51.325 7f07f605d700 1 -- 172.21.2.102:0/216601248 learned_addr learned my addr 172.21.2.102:0/216601248 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:30:51.331 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:51.325 7f07f605d700 1 -- 172.21.2.102:0/216601248 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f07f8188400 msgr2=0x7f07f8188800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:51.331 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:51.325 7f07f605d700 1 --2- 172.21.2.102:0/216601248 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f07f8188400 0x7f07f8188800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:51.331 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:51.325 7f07f605d700 1 -- 172.21.2.102:0/216601248 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f07f809cbc0 msgr2=0x7f07f809cfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:51.331 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:51.325 7f07f605d700 1 --2- 172.21.2.102:0/216601248 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f07f809cbc0 0x7f07f809cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:51.331 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:51.325 7f07f605d700 1 -- 172.21.2.102:0/216601248 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07f809d5e0 con 0x7f07f8188e20 2021-02-10T11:30:51.331 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:51.325 7f07f605d700 1 --2- 172.21.2.102:0/216601248 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f07f8188e20 0x7f07f8189220 secure :-1 s=READY pgs=2039 cs=0 l=1 rev1=1 rx=0x7f07e8006f60 tx=0x7f07e800b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:51.331 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:51.325 7f07ef7fe700 1 -- 172.21.2.102:0/216601248 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f07e800bd60 con 0x7f07f8188e20 2021-02-10T11:30:51.331 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:51.325 7f07ef7fe700 1 -- 172.21.2.102:0/216601248 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07e8000c90 con 0x7f07f8188e20 2021-02-10T11:30:51.332 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:51.325 7f07ef7fe700 1 -- 172.21.2.102:0/216601248 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f07e8002220 con 0x7f07f8188e20 2021-02-10T11:30:51.332 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:51.325 7f07fcc00700 1 -- 172.21.2.102:0/216601248 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07f809d5e0 con 0x7f07f8188e20 2021-02-10T11:30:51.332 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:51.325 7f07fcc00700 1 -- 172.21.2.102:0/216601248 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07f809db70 con 0x7f07f8188e20 2021-02-10T11:30:51.332 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:51.326 7f07ef7fe700 1 -- 172.21.2.102:0/216601248 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f07e8002020 con 0x7f07f8188e20 2021-02-10T11:30:51.332 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:51.326 7f07ef7fe700 1 --2- 172.21.2.102:0/216601248 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f07dc0186c0 0x7f07dc01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:51.332 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:51.326 7f07ef7fe700 1 -- 172.21.2.102:0/216601248 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 5221+0+0 (secure 0 0 0) 0x7f07e80323f0 con 0x7f07f8188e20 2021-02-10T11:30:51.332 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:51.326 7f07f685e700 1 --2- 172.21.2.102:0/216601248 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f07dc0186c0 0x7f07dc01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:51.332 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:51.326 7f07f685e700 1 --2- 172.21.2.102:0/216601248 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f07dc0186c0 0x7f07dc01abc0 crc :-1 s=READY pgs=908 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:51.357 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:51.356 7f07fcc00700 1 -- 172.21.2.102:0/216601248 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07f8082a70 con 0x7f07f8188e20 2021-02-10T11:30:51.360 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:51.359 7f07ef7fe700 1 -- 172.21.2.102:0/216601248 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f07e80268c0 con 0x7f07f8188e20 2021-02-10T11:30:51.486 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:51.485 7f07fcc00700 1 -- 172.21.2.102:0/216601248 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"pool2": "cephfs_data", "prefix": "osd pool delete", "pool": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f07f8082a70 con 0x7f07f8188e20 2021-02-10T11:30:51.874 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:51.872 7f07ef7fe700 1 -- 172.21.2.102:0/216601248 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"pool2": "cephfs_data", "prefix": "osd pool delete", "pool": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v96) v1 ==== 173+0+0 (secure 0 0 0) 0x7f07e80268c0 con 0x7f07f8188e20 2021-02-10T11:30:51.904 INFO:teuthology.orchestra.run.gibba002.stderr:pool 'cephfs_data' removed 2021-02-10T11:30:51.905 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:51.904 7f07fcc00700 1 -- 172.21.2.102:0/216601248 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f07dc0186c0 msgr2=0x7f07dc01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:51.905 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:51.904 7f07fcc00700 1 --2- 172.21.2.102:0/216601248 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f07dc0186c0 0x7f07dc01abc0 crc :-1 s=READY pgs=908 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:51.905 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:51.904 7f07fcc00700 1 -- 172.21.2.102:0/216601248 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f07f8188e20 msgr2=0x7f07f8189220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:51.906 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:51.904 7f07fcc00700 1 --2- 172.21.2.102:0/216601248 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f07f8188e20 0x7f07f8189220 secure :-1 s=READY pgs=2039 cs=0 l=1 rev1=1 rx=0x7f07e8006f60 tx=0x7f07e800b970).stop 2021-02-10T11:30:51.906 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:51.904 7f07fcc00700 1 -- 172.21.2.102:0/216601248 shutdown_connections 2021-02-10T11:30:51.906 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:51.904 7f07fcc00700 1 --2- 172.21.2.102:0/216601248 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f07dc0186c0 0x7f07dc01abc0 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:51.906 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:51.904 7f07fcc00700 1 --2- 172.21.2.102:0/216601248 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f07f809cbc0 0x7f07f809cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:51.906 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:51.904 7f07fcc00700 1 --2- 172.21.2.102:0/216601248 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f07f8188e20 0x7f07f8189220 unknown :-1 s=CLOSED pgs=2039 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:51.906 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:51.904 7f07fcc00700 1 --2- 172.21.2.102:0/216601248 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f07f8188400 0x7f07f8188800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:51.906 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:51.904 7f07fcc00700 1 -- 172.21.2.102:0/216601248 >> 172.21.2.102:0/216601248 conn(0x7f07f80c2cd0 msgr2=0x7f07f80b5950 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:51.906 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:51.905 7f07fcc00700 1 -- 172.21.2.102:0/216601248 shutdown_connections 2021-02-10T11:30:51.906 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:51.905 7f07fcc00700 1 -- 172.21.2.102:0/216601248 wait complete. 2021-02-10T11:30:51.914 INFO:tasks.ceph.mds.a:Restarting daemon 2021-02-10T11:30:51.915 INFO:tasks.ceph.mds.a:Stopping old one... 2021-02-10T11:30:51.915 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2021-02-10T11:30:51.915 INFO:teuthology.orchestra.run:waiting for 300 2021-02-10T11:30:51.915 INFO:tasks.ceph.mds.b:Restarting daemon 2021-02-10T11:30:51.915 INFO:tasks.ceph.mds.b:Stopping old one... 2021-02-10T11:30:51.915 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2021-02-10T11:30:51.915 INFO:teuthology.orchestra.run:waiting for 300 2021-02-10T11:30:51.929 INFO:tasks.ceph.mds.a:Stopped 2021-02-10T11:30:51.929 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2021-02-10T11:30:51.930 INFO:tasks.ceph.mds.b:Stopped 2021-02-10T11:30:51.931 DEBUG:teuthology.orchestra.run.gibba013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2021-02-10T11:30:51.987 INFO:tasks.ceph.mds.a:Started 2021-02-10T11:30:51.987 DEBUG:teuthology.parallel:result is None 2021-02-10T11:30:52.004 INFO:tasks.ceph.mds.b:Started 2021-02-10T11:30:52.005 DEBUG:teuthology.parallel:result is None 2021-02-10T11:30:52.005 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd blacklist clear 2021-02-10T11:30:52.394 INFO:tasks.ceph.mds.a.gibba002.stdout:starting mds.a at 2021-02-10T11:30:52.409 INFO:tasks.ceph.mds.b.gibba013.stdout:starting mds.b at 2021-02-10T11:30:52.417 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:52.400 7fcd5d8a9700 1 Processor -- start 2021-02-10T11:30:52.417 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:52.400 7fcd5d8a9700 1 -- start start 2021-02-10T11:30:52.417 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:52.400 7fcd5d8a9700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fcd580d0340 0x7fcd580ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:52.417 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:52.400 7fcd5d8a9700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd580f0b10 con 0x7fcd580d1c70 2021-02-10T11:30:52.417 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:52.400 7fcd5d8a9700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd581b0be0 con 0x7fcd580d1870 2021-02-10T11:30:52.417 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:52.400 7fcd5d8a9700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd580e5ed0 con 0x7fcd580d0340 2021-02-10T11:30:52.417 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:52.401 7fcd56d73700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fcd580d0340 0x7fcd580ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:52.418 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:52.401 7fcd56d73700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fcd580d0340 0x7fcd580ef640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:33144/0 (socket says 172.21.2.102:33144) 2021-02-10T11:30:52.418 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:52.401 7fcd56d73700 1 -- 172.21.2.102:0/1891179990 learned_addr learned my addr 172.21.2.102:0/1891179990 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:30:52.418 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:52.401 7fcd56d73700 1 -- 172.21.2.102:0/1891179990 >> v1:172.21.2.113:6789/0 conn(0x7fcd580d1870 legacy=0x7fcd581b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:52.418 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:52.401 7fcd56d73700 1 -- 172.21.2.102:0/1891179990 >> v1:172.21.2.102:6789/0 conn(0x7fcd580d1c70 legacy=0x7fcd580efc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:52.418 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:52.401 7fcd56d73700 1 -- 172.21.2.102:0/1891179990 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd580f0450 con 0x7fcd580d0340 2021-02-10T11:30:52.418 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:52.401 7fcd56d73700 1 --2- 172.21.2.102:0/1891179990 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcd580d0340 0x7fcd580ef640 secure :-1 s=READY pgs=2045 cs=0 l=1 rev1=1 rx=0x7fcd40006700 tx=0x7fcd40009b00).ready entity=mon.0 client_cookie=6a7c899a026b49fd server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:52.418 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:52.401 7fcd56572700 1 -- 172.21.2.102:0/1891179990 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcd4000c0c0 con 0x7fcd580d0340 2021-02-10T11:30:52.418 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:52.401 7fcd56572700 1 -- 172.21.2.102:0/1891179990 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd4000c400 con 0x7fcd580d0340 2021-02-10T11:30:52.418 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:52.401 7fcd56572700 1 -- 172.21.2.102:0/1891179990 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcd4000c8e0 con 0x7fcd580d0340 2021-02-10T11:30:52.418 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:52.401 7fcd5d8a9700 1 -- 172.21.2.102:0/1891179990 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcd580d0340 msgr2=0x7fcd580ef640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:52.418 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:52.401 7fcd5d8a9700 1 --2- 172.21.2.102:0/1891179990 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcd580d0340 0x7fcd580ef640 secure :-1 s=READY pgs=2045 cs=0 l=1 rev1=1 rx=0x7fcd40006700 tx=0x7fcd40009b00).stop 2021-02-10T11:30:52.418 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:52.402 7fcd5d8a9700 1 -- 172.21.2.102:0/1891179990 shutdown_connections 2021-02-10T11:30:52.418 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:52.402 7fcd5d8a9700 1 --2- 172.21.2.102:0/1891179990 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcd580d0340 0x7fcd580ef640 unknown :-1 s=CLOSED pgs=2045 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:52.419 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:52.402 7fcd5d8a9700 1 -- 172.21.2.102:0/1891179990 >> 172.21.2.102:0/1891179990 conn(0x7fcd580ce450 msgr2=0x7fcd580d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:52.419 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:52.402 7fcd5d8a9700 1 -- 172.21.2.102:0/1891179990 shutdown_connections 2021-02-10T11:30:52.419 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:52.402 7fcd5d8a9700 1 -- 172.21.2.102:0/1891179990 wait complete. 2021-02-10T11:30:52.419 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:52.417 7fcd5d8a9700 1 Processor -- start 2021-02-10T11:30:52.419 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:52.417 7fcd5d8a9700 1 -- start start 2021-02-10T11:30:52.420 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:52.417 7fcd5d8a9700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcd580c3210 0x7fcd581b4a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:52.420 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:52.417 7fcd5d8a9700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcd580c3610 0x7fcd581b5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:52.420 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:52.417 7fcd5d8a9700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fcd58099b30 0x7fcd58099f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:52.420 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:52.417 7fcd5d8a9700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd581b5a70 con 0x7fcd580c3210 2021-02-10T11:30:52.420 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:52.417 7fcd5d8a9700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd581b5670 con 0x7fcd58099b30 2021-02-10T11:30:52.420 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:52.417 7fcd5d8a9700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd581b5850 con 0x7fcd580c3610 2021-02-10T11:30:52.420 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:52.417 7fcd57574700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcd580c3210 0x7fcd581b4a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:52.420 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:52.417 7fcd57574700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcd580c3210 0x7fcd581b4a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:33150/0 (socket says 172.21.2.102:33150) 2021-02-10T11:30:52.420 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:52.417 7fcd57574700 1 -- 172.21.2.102:0/2446236814 learned_addr learned my addr 172.21.2.102:0/2446236814 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:30:52.422 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:52.418 7fcd57d75700 1 --2- 172.21.2.102:0/2446236814 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fcd58099b30 0x7fcd58099f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:52.422 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:52.418 7fcd56d73700 1 --2- 172.21.2.102:0/2446236814 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcd580c3610 0x7fcd581b5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:52.422 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:52.418 7fcd57574700 1 -- 172.21.2.102:0/2446236814 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcd580c3610 msgr2=0x7fcd581b5050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:52.423 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:52.418 7fcd57574700 1 --2- 172.21.2.102:0/2446236814 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcd580c3610 0x7fcd581b5050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:52.423 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:52.418 7fcd57574700 1 -- 172.21.2.102:0/2446236814 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fcd58099b30 msgr2=0x7fcd58099f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:52.423 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:52.418 7fcd57574700 1 --2- 172.21.2.102:0/2446236814 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fcd58099b30 0x7fcd58099f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:52.423 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:52.418 7fcd57574700 1 -- 172.21.2.102:0/2446236814 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd5809a550 con 0x7fcd580c3210 2021-02-10T11:30:52.423 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:52.418 7fcd57574700 1 --2- 172.21.2.102:0/2446236814 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcd580c3210 0x7fcd581b4a30 secure :-1 s=READY pgs=2046 cs=0 l=1 rev1=1 rx=0x7fcd48006ed0 tx=0x7fcd4800b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:52.423 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:52.418 7fcd4ffff700 1 -- 172.21.2.102:0/2446236814 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcd48006020 con 0x7fcd580c3210 2021-02-10T11:30:52.423 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:52.418 7fcd4ffff700 1 -- 172.21.2.102:0/2446236814 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd480009b0 con 0x7fcd580c3210 2021-02-10T11:30:52.423 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:52.418 7fcd4ffff700 1 -- 172.21.2.102:0/2446236814 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcd480054b0 con 0x7fcd580c3210 2021-02-10T11:30:52.423 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:52.418 7fcd5d8a9700 1 -- 172.21.2.102:0/2446236814 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd5809a550 con 0x7fcd580c3210 2021-02-10T11:30:52.423 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:52.418 7fcd5d8a9700 1 -- 172.21.2.102:0/2446236814 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd5809aab0 con 0x7fcd580c3210 2021-02-10T11:30:52.423 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:52.419 7fcd4ffff700 1 -- 172.21.2.102:0/2446236814 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fcd48006020 con 0x7fcd580c3210 2021-02-10T11:30:52.423 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:52.419 7fcd4ffff700 1 --2- 172.21.2.102:0/2446236814 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcd3c018730 0x7fcd3c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:52.423 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:52.419 7fcd4ffff700 1 -- 172.21.2.102:0/2446236814 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 4839+0+0 (secure 0 0 0) 0x7fcd48032350 con 0x7fcd580c3210 2021-02-10T11:30:52.424 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:52.419 7fcd56d73700 1 --2- 172.21.2.102:0/2446236814 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcd3c018730 0x7fcd3c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:52.424 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:52.419 7fcd56d73700 1 --2- 172.21.2.102:0/2446236814 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcd3c018730 0x7fcd3c01abc0 crc :-1 s=READY pgs=909 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:52.450 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:52.449 7fcd5d8a9700 1 -- 172.21.2.102:0/2446236814 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd580ffa00 con 0x7fcd580c3210 2021-02-10T11:30:52.452 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:52.452 7fcd4ffff700 1 -- 172.21.2.102:0/2446236814 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fcd4801f4f0 con 0x7fcd580c3210 2021-02-10T11:30:52.592 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:52.591 7fcd5d8a9700 1 -- 172.21.2.102:0/2446236814 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd blacklist clear"} v 0) v1 -- 0x7fcd580f06f0 con 0x7fcd580c3210 2021-02-10T11:30:52.918 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:52.917 7fcd4ffff700 1 -- 172.21.2.102:0/2446236814 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd blacklist clear"}]=0 removed all blacklist entries v97) v1 ==== 97+0+0 (secure 0 0 0) 0x7fcd48006020 con 0x7fcd580c3210 2021-02-10T11:30:52.959 INFO:teuthology.orchestra.run.gibba002.stderr: removed all blacklist entries 2021-02-10T11:30:52.961 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:52.959 7fcd5d8a9700 1 -- 172.21.2.102:0/2446236814 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcd3c018730 msgr2=0x7fcd3c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:52.961 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:52.959 7fcd5d8a9700 1 --2- 172.21.2.102:0/2446236814 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcd3c018730 0x7fcd3c01abc0 crc :-1 s=READY pgs=909 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:52.962 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:52.959 7fcd5d8a9700 1 -- 172.21.2.102:0/2446236814 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcd580c3210 msgr2=0x7fcd581b4a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:52.962 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:52.959 7fcd5d8a9700 1 --2- 172.21.2.102:0/2446236814 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcd580c3210 0x7fcd581b4a30 secure :-1 s=READY pgs=2046 cs=0 l=1 rev1=1 rx=0x7fcd48006ed0 tx=0x7fcd4800b960).stop 2021-02-10T11:30:52.962 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:52.959 7fcd5d8a9700 1 -- 172.21.2.102:0/2446236814 shutdown_connections 2021-02-10T11:30:52.962 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:52.959 7fcd5d8a9700 1 --2- 172.21.2.102:0/2446236814 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcd3c018730 0x7fcd3c01abc0 unknown :-1 s=CLOSED pgs=909 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:52.962 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:52.959 7fcd5d8a9700 1 --2- 172.21.2.102:0/2446236814 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fcd58099b30 0x7fcd58099f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:52.963 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:52.959 7fcd5d8a9700 1 --2- 172.21.2.102:0/2446236814 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcd580c3610 0x7fcd581b5050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:52.963 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:52.959 7fcd5d8a9700 1 --2- 172.21.2.102:0/2446236814 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcd580c3210 0x7fcd581b4a30 unknown :-1 s=CLOSED pgs=2046 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:52.963 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:52.959 7fcd5d8a9700 1 -- 172.21.2.102:0/2446236814 >> 172.21.2.102:0/2446236814 conn(0x7fcd58109a20 msgr2=0x7fcd581b00b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:52.963 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:52.960 7fcd5d8a9700 1 -- 172.21.2.102:0/2446236814 shutdown_connections 2021-02-10T11:30:52.963 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:52.960 7fcd5d8a9700 1 -- 172.21.2.102:0/2446236814 wait complete. 2021-02-10T11:30:52.972 INFO:tasks.cephfs.cephfs_test_case:['0', '1', '2', '3'] 2021-02-10T11:30:52.974 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth ls --format=json-pretty 2021-02-10T11:30:53.408 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:53.391 7fd189960700 1 Processor -- start 2021-02-10T11:30:53.408 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:53.391 7fd189960700 1 -- start start 2021-02-10T11:30:53.409 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:53.391 7fd189960700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fd1840d1870 0x7fd1841b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:53.409 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:53.391 7fd189960700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fd1841b0a10 0x7fd1840cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:53.409 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:53.391 7fd189960700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1840f0b10 con 0x7fd1840d1c70 2021-02-10T11:30:53.409 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:53.391 7fd189960700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1840e5ed0 con 0x7fd1840d1870 2021-02-10T11:30:53.409 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:53.391 7fd189960700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1840f0650 con 0x7fd1841b0a10 2021-02-10T11:30:53.409 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:53.391 7fd183574700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fd1840d1870 0x7fd1841b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:53.409 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:53.391 7fd183574700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fd1840d1870 0x7fd1841b03f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:33158/0 (socket says 172.21.2.102:33158) 2021-02-10T11:30:53.409 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:53.391 7fd183574700 1 -- 172.21.2.102:0/2780431498 learned_addr learned my addr 172.21.2.102:0/2780431498 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:30:53.409 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:53.392 7fd183574700 1 -- 172.21.2.102:0/2780431498 >> v2:172.21.2.113:3300/0 conn(0x7fd1841b0a10 msgr2=0x7fd1840cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:53.409 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:53.392 7fd183574700 1 --2- 172.21.2.102:0/2780431498 >> v2:172.21.2.113:3300/0 conn(0x7fd1841b0a10 0x7fd1840cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:53.409 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:53.392 7fd183574700 1 -- 172.21.2.102:0/2780431498 >> v1:172.21.2.102:6789/0 conn(0x7fd1840d1c70 legacy=0x7fd1840cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:53.409 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:53.392 7fd183574700 1 -- 172.21.2.102:0/2780431498 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1840f0840 con 0x7fd1840d1870 2021-02-10T11:30:53.410 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:53.392 7fd183574700 1 --2- 172.21.2.102:0/2780431498 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd1840d1870 0x7fd1841b03f0 secure :-1 s=READY pgs=2048 cs=0 l=1 rev1=1 rx=0x7fd16c006700 tx=0x7fd16c009b80).ready entity=mon.0 client_cookie=1cbc142d8cb17cc7 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:53.410 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:53.392 7fd182572700 1 -- 172.21.2.102:0/2780431498 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd16c00c0c0 con 0x7fd1840d1870 2021-02-10T11:30:53.410 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:53.392 7fd182572700 1 -- 172.21.2.102:0/2780431498 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd16c00c440 con 0x7fd1840d1870 2021-02-10T11:30:53.410 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:53.392 7fd182572700 1 -- 172.21.2.102:0/2780431498 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd16c00c960 con 0x7fd1840d1870 2021-02-10T11:30:53.410 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:53.392 7fd189960700 1 -- 172.21.2.102:0/2780431498 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd1840d1870 msgr2=0x7fd1841b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:53.410 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:53.392 7fd189960700 1 --2- 172.21.2.102:0/2780431498 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd1840d1870 0x7fd1841b03f0 secure :-1 s=READY pgs=2048 cs=0 l=1 rev1=1 rx=0x7fd16c006700 tx=0x7fd16c009b80).stop 2021-02-10T11:30:53.415 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:53.393 7fd189960700 1 -- 172.21.2.102:0/2780431498 shutdown_connections 2021-02-10T11:30:53.415 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:53.393 7fd189960700 1 --2- 172.21.2.102:0/2780431498 >> v2:172.21.2.113:3300/0 conn(0x7fd1841b0a10 0x7fd1840cf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:53.415 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:53.393 7fd189960700 1 --2- 172.21.2.102:0/2780431498 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd1840d1870 0x7fd1841b03f0 unknown :-1 s=CLOSED pgs=2048 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:53.415 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:53.393 7fd189960700 1 -- 172.21.2.102:0/2780431498 >> 172.21.2.102:0/2780431498 conn(0x7fd1840ce450 msgr2=0x7fd1840d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:53.415 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:53.393 7fd189960700 1 -- 172.21.2.102:0/2780431498 shutdown_connections 2021-02-10T11:30:53.415 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:53.393 7fd189960700 1 -- 172.21.2.102:0/2780431498 wait complete. 2021-02-10T11:30:53.417 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:53.408 7fd189960700 1 Processor -- start 2021-02-10T11:30:53.418 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:53.408 7fd189960700 1 -- start start 2021-02-10T11:30:53.418 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:53.408 7fd189960700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd1840b0ad0 0x7fd184091f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:53.418 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:53.408 7fd189960700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd1841a8850 0x7fd1841a8c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:53.418 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:53.408 7fd189960700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd1841a6fa0 0x7fd1841a73a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:53.418 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:53.408 7fd189960700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1841abfe0 con 0x7fd1841a8850 2021-02-10T11:30:53.418 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:53.408 7fd189960700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1841a9c40 con 0x7fd1841a6fa0 2021-02-10T11:30:53.418 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:53.408 7fd189960700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1841a9730 con 0x7fd1840b0ad0 2021-02-10T11:30:53.418 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:53.409 7fd182d73700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd1841a8850 0x7fd1841a8c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:53.418 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:53.409 7fd182d73700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd1841a8850 0x7fd1841a8c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:33166/0 (socket says 172.21.2.102:33166) 2021-02-10T11:30:53.418 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:53.409 7fd182d73700 1 -- 172.21.2.102:0/1236629202 learned_addr learned my addr 172.21.2.102:0/1236629202 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:30:53.419 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:53.409 7fd182d73700 1 -- 172.21.2.102:0/1236629202 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd1840b0ad0 msgr2=0x7fd184091f30 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:30:53.419 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:53.409 7fd182d73700 1 --2- 172.21.2.102:0/1236629202 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd1840b0ad0 0x7fd184091f30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:53.419 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:53.409 7fd182d73700 1 -- 172.21.2.102:0/1236629202 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd1841a6fa0 msgr2=0x7fd1841a73a0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:30:53.419 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:53.409 7fd182d73700 1 --2- 172.21.2.102:0/1236629202 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd1841a6fa0 0x7fd1841a73a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:53.419 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:53.409 7fd182d73700 1 -- 172.21.2.102:0/1236629202 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1841a9270 con 0x7fd1841a8850 2021-02-10T11:30:53.419 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:53.409 7fd182d73700 1 --2- 172.21.2.102:0/1236629202 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd1841a8850 0x7fd1841a8c50 secure :-1 s=READY pgs=2049 cs=0 l=1 rev1=1 rx=0x7fd174006f00 tx=0x7fd17400b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:53.419 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:53.409 7fd17bfff700 1 -- 172.21.2.102:0/1236629202 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd17400bd30 con 0x7fd1841a8850 2021-02-10T11:30:53.419 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:53.409 7fd17bfff700 1 -- 172.21.2.102:0/1236629202 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd174000c90 con 0x7fd1841a8850 2021-02-10T11:30:53.419 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:53.409 7fd189960700 1 -- 172.21.2.102:0/1236629202 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1841a9270 con 0x7fd1841a8850 2021-02-10T11:30:53.419 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:53.409 7fd17bfff700 1 -- 172.21.2.102:0/1236629202 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd1740022a0 con 0x7fd1841a8850 2021-02-10T11:30:53.419 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:53.409 7fd189960700 1 -- 172.21.2.102:0/1236629202 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1841a7cb0 con 0x7fd1841a8850 2021-02-10T11:30:53.419 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:53.410 7fd17bfff700 1 -- 172.21.2.102:0/1236629202 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd174005310 con 0x7fd1841a8850 2021-02-10T11:30:53.419 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:53.410 7fd17bfff700 1 --2- 172.21.2.102:0/1236629202 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd1680186d0 0x7fd16801abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:53.420 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:53.410 7fd17bfff700 1 -- 172.21.2.102:0/1236629202 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 4753+0+0 (secure 0 0 0) 0x7fd1740324a0 con 0x7fd1841a8850 2021-02-10T11:30:53.420 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:53.410 7fd183574700 1 --2- 172.21.2.102:0/1236629202 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd1680186d0 0x7fd16801abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:53.420 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:53.410 7fd183574700 1 --2- 172.21.2.102:0/1236629202 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd1680186d0 0x7fd16801abd0 crc :-1 s=READY pgs=910 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:53.442 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:53.441 7fd189960700 1 -- 172.21.2.102:0/1236629202 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1841a9710 con 0x7fd1841a8850 2021-02-10T11:30:53.444 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:53.443 7fd17bfff700 1 -- 172.21.2.102:0/1236629202 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd174000940 con 0x7fd1841a8850 2021-02-10T11:30:53.576 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:53.575 7fd189960700 1 -- 172.21.2.102:0/1236629202 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fd1841a9710 con 0x7fd1841a8850 2021-02-10T11:30:53.577 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:53.576 7fd17bfff700 1 -- 172.21.2.102:0/1236629202 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v58) v1 ==== 80+0+4513 (secure 0 0 0) 0x7fd17400bd30 con 0x7fd1841a8850 2021-02-10T11:30:53.577 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:30:53.577 INFO:teuthology.orchestra.run.gibba002.stdout:{ 2021-02-10T11:30:53.578 INFO:teuthology.orchestra.run.gibba002.stdout: "auth_dump": [ 2021-02-10T11:30:53.578 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:30:53.578 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "mds.a", 2021-02-10T11:30:53.578 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAevSNgsgyEARAAvr0cUjjCoEHLxaZSZ2HtVA==", 2021-02-10T11:30:53.578 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:30:53.578 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow", 2021-02-10T11:30:53.578 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:30:53.578 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:30:53.578 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:30:53.578 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:30:53.578 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:30:53.578 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:30:53.578 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "mds.b", 2021-02-10T11:30:53.579 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAevSNgwOFMMRAALfb1fbX56ao1TU8ClnTW1A==", 2021-02-10T11:30:53.579 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:30:53.579 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow", 2021-02-10T11:30:53.579 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:30:53.579 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:30:53.579 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:30:53.579 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:30:53.579 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:30:53.579 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:30:53.579 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.0", 2021-02-10T11:30:53.579 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAmvSNgj+bMMBAAD3HUD6fLiMAg+++iNzLJKQ==", 2021-02-10T11:30:53.579 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:30:53.579 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:30:53.580 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:30:53.580 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:30:53.580 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:30:53.580 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:30:53.580 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:30:53.580 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.1", 2021-02-10T11:30:53.580 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQApvSNgnvc+MRAAAOxqlNwiyLHqNITOrYeFKQ==", 2021-02-10T11:30:53.580 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:30:53.580 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:30:53.580 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:30:53.580 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:30:53.580 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:30:53.580 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:30:53.581 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:30:53.581 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.2", 2021-02-10T11:30:53.581 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAsvSNgZ4SbMRAA76IOYWfbiAuBoM/4k81GxA==", 2021-02-10T11:30:53.581 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:30:53.581 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:30:53.581 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:30:53.581 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:30:53.581 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:30:53.581 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:30:53.581 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:30:53.581 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.3", 2021-02-10T11:30:53.581 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAvvSNgFk3rMRAAWCmKKyw0LDQ+UlAHZAITOw==", 2021-02-10T11:30:53.581 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:30:53.582 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:30:53.582 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:30:53.582 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:30:53.582 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:30:53.582 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:30:53.582 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:30:53.582 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.4", 2021-02-10T11:30:53.582 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQA4vSNgpKcNJBAApJMmvj5YxVHPDOmvmQONOg==", 2021-02-10T11:30:53.582 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:30:53.582 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:30:53.582 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:30:53.582 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:30:53.582 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:30:53.583 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:30:53.583 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:30:53.583 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.5", 2021-02-10T11:30:53.583 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQA7vSNgf2ujJBAAfkuYpyb3WTzc8V5kw+Wktg==", 2021-02-10T11:30:53.583 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:30:53.583 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:30:53.583 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:30:53.583 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:30:53.583 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:30:53.583 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:30:53.583 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:30:53.583 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.6", 2021-02-10T11:30:53.583 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQA+vSNgqw9uJRAAFMU0R+UxU9C3o/9byb3e0g==", 2021-02-10T11:30:53.584 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:30:53.584 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:30:53.584 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:30:53.584 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:30:53.584 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:30:53.584 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:30:53.584 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:30:53.584 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.7", 2021-02-10T11:30:53.584 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQBBvSNgvWSrJRAA2KaCHmLpzovRgYzLvyofTQ==", 2021-02-10T11:30:53.584 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:30:53.584 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:30:53.584 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:30:53.584 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:30:53.585 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:30:53.585 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:30:53.585 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:30:53.585 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "client.0", 2021-02-10T11:30:53.585 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAfvSNgZd9XIBAAvyrvl8fdKuHnJZfczfRoFw==", 2021-02-10T11:30:53.585 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:30:53.585 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow", 2021-02-10T11:30:53.585 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow r", 2021-02-10T11:30:53.585 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:30:53.585 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:30:53.585 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:30:53.585 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:30:53.586 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "client.1", 2021-02-10T11:30:53.586 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAfvSNgSyfXNhAAIduPnF7Vd/6HSkJ0MlcMSA==", 2021-02-10T11:30:53.586 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:30:53.586 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow", 2021-02-10T11:30:53.586 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow r", 2021-02-10T11:30:53.586 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:30:53.586 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:30:53.586 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:30:53.586 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:30:53.586 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "client.2", 2021-02-10T11:30:53.586 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAgvSNgiIG/ERAA9ZTamaTI7ZsPSnxBqjplCg==", 2021-02-10T11:30:53.586 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:30:53.586 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow", 2021-02-10T11:30:53.587 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow r", 2021-02-10T11:30:53.587 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:30:53.587 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:30:53.587 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:30:53.587 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:30:53.587 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "client.3", 2021-02-10T11:30:53.587 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAgvSNgOWpOKBAAlXsEB04Q/GvtL06PqRMLeA==", 2021-02-10T11:30:53.587 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:30:53.587 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow", 2021-02-10T11:30:53.587 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow r", 2021-02-10T11:30:53.587 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:30:53.587 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:30:53.587 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:30:53.588 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:30:53.588 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "client.admin", 2021-02-10T11:30:53.588 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAavSNgO3GQBhAAL+NVnYskQRKVWLgyDPY/xw==", 2021-02-10T11:30:53.588 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:30:53.588 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow *", 2021-02-10T11:30:53.588 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:30:53.588 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:30:53.588 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:30:53.588 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:30:53.588 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:30:53.588 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:30:53.588 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "mgr.x", 2021-02-10T11:30:53.588 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAdvSNgXZZgJhAA7dtBSGzH+DsBwnddZpOBJA==", 2021-02-10T11:30:53.589 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:30:53.589 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow *", 2021-02-10T11:30:53.589 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow profile mgr", 2021-02-10T11:30:53.589 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:30:53.589 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:30:53.589 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:30:53.589 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:30:53.589 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "mgr.y", 2021-02-10T11:30:53.589 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAdvSNgVqQFDRAAwo6YEdWZCAr5GdOWVL/Qvg==", 2021-02-10T11:30:53.589 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:30:53.589 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow *", 2021-02-10T11:30:53.589 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow profile mgr", 2021-02-10T11:30:53.589 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:30:53.589 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:30:53.590 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:30:53.590 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:30:53.590 INFO:teuthology.orchestra.run.gibba002.stdout:} 2021-02-10T11:30:53.591 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:53.578 7fd189960700 1 -- 172.21.2.102:0/1236629202 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd1680186d0 msgr2=0x7fd16801abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:53.591 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:53.578 7fd189960700 1 --2- 172.21.2.102:0/1236629202 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd1680186d0 0x7fd16801abd0 crc :-1 s=READY pgs=910 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:53.591 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:53.578 7fd189960700 1 -- 172.21.2.102:0/1236629202 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd1841a8850 msgr2=0x7fd1841a8c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:53.591 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:53.578 7fd189960700 1 --2- 172.21.2.102:0/1236629202 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd1841a8850 0x7fd1841a8c50 secure :-1 s=READY pgs=2049 cs=0 l=1 rev1=1 rx=0x7fd174006f00 tx=0x7fd17400b940).stop 2021-02-10T11:30:53.591 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:53.578 7fd189960700 1 -- 172.21.2.102:0/1236629202 shutdown_connections 2021-02-10T11:30:53.591 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:53.578 7fd189960700 1 --2- 172.21.2.102:0/1236629202 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd1680186d0 0x7fd16801abd0 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:53.591 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:53.578 7fd189960700 1 --2- 172.21.2.102:0/1236629202 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd1841a6fa0 0x7fd1841a73a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:53.591 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:53.578 7fd189960700 1 --2- 172.21.2.102:0/1236629202 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd1841a8850 0x7fd1841a8c50 unknown :-1 s=CLOSED pgs=2049 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:53.592 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:53.578 7fd189960700 1 --2- 172.21.2.102:0/1236629202 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd1840b0ad0 0x7fd184091f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:53.592 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:53.578 7fd189960700 1 -- 172.21.2.102:0/1236629202 >> 172.21.2.102:0/1236629202 conn(0x7fd18419f8f0 msgr2=0x7fd1841a5f80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:53.592 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:53.578 7fd189960700 1 -- 172.21.2.102:0/1236629202 shutdown_connections 2021-02-10T11:30:53.592 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:53.578 7fd189960700 1 -- 172.21.2.102:0/1236629202 wait complete. 2021-02-10T11:30:53.592 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:30:54.025 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:54.006 7f047dcc1700 1 Processor -- start 2021-02-10T11:30:54.025 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:54.007 7f047dcc1700 1 -- start start 2021-02-10T11:30:54.025 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:54.007 7f047dcc1700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f04780d1870 0x7f04781b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:54.025 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:54.007 7f047dcc1700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04780f0b10 con 0x7f04780f0650 2021-02-10T11:30:54.025 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:54.007 7f047dcc1700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04780d0310 con 0x7f04781b0a10 2021-02-10T11:30:54.025 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:54.007 7f047dcc1700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04780d0530 con 0x7f04780d1870 2021-02-10T11:30:54.025 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:54.007 7f0477fff700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f04780f0650 0x7f04780cfb20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:53626/0 (socket says 172.21.2.102:53626) 2021-02-10T11:30:54.025 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:54.007 7f0477fff700 1 -- 172.21.2.102:0/758265573 learned_addr learned my addr 172.21.2.102:0/758265573 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:30:54.026 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:54.008 7f04767fc700 1 -- 172.21.2.102:0/758265573 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4285592718 0 0) 0x7f0470001500 con 0x7f04780f0650 2021-02-10T11:30:54.026 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:54.008 7f04767fc700 1 -- 172.21.2.102:0/758265573 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0464002fe0 con 0x7f04780f0650 2021-02-10T11:30:54.026 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:54.008 7f04777fe700 1 --2- 172.21.2.102:0/758265573 >> v2:172.21.2.113:3300/0 conn(0x7f04780d1870 0x7f04781b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:54.026 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:54.008 7f04767fc700 1 -- 172.21.2.102:0/758265573 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2399575183 0 0) 0x7f0470001460 con 0x7f04780f0650 2021-02-10T11:30:54.026 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:54.008 7f04767fc700 1 -- 172.21.2.102:0/758265573 >> v2:172.21.2.113:3300/0 conn(0x7f04780d1870 msgr2=0x7f04781b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:54.026 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:54.008 7f04767fc700 1 --2- 172.21.2.102:0/758265573 >> v2:172.21.2.113:3300/0 conn(0x7f04780d1870 0x7f04781b03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:54.026 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:54.008 7f04767fc700 1 -- 172.21.2.102:0/758265573 >> v1:172.21.2.113:6789/0 conn(0x7f04781b0a10 legacy=0x7f04780cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:54.026 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:54.008 7f04767fc700 1 -- 172.21.2.102:0/758265573 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04780d1c70 con 0x7f04780f0650 2021-02-10T11:30:54.026 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:54.008 7f04767fc700 1 -- 172.21.2.102:0/758265573 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7f0470001980 con 0x7f04780f0650 2021-02-10T11:30:54.026 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:54.008 7f04767fc700 1 -- 172.21.2.102:0/758265573 >> v1:172.21.2.102:6789/0 conn(0x7f04780f0650 legacy=0x7f04780cfb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:54.026 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:54.008 7f04767fc700 1 --2- 172.21.2.102:0/758265573 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0464003b90 0x7f0464006020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:54.026 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:54.008 7f04767fc700 1 --2- 172.21.2.102:0/758265573 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0464006840 0x7f0464008d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:54.026 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:54.008 7f04767fc700 1 --2- 172.21.2.102:0/758265573 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0464009640 0x7f046400bb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:54.027 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:54.008 7f04767fc700 1 -- 172.21.2.102:0/758265573 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f046400c390 con 0x7f0464003b90 2021-02-10T11:30:54.027 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:54.008 7f04767fc700 1 -- 172.21.2.102:0/758265573 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f046400c5b0 con 0x7f0464006840 2021-02-10T11:30:54.027 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:54.008 7f04767fc700 1 -- 172.21.2.102:0/758265573 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f046400c7d0 con 0x7f0464009640 2021-02-10T11:30:54.027 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:54.008 7f04777fe700 1 --2- 172.21.2.102:0/758265573 >> v2:172.21.2.113:3300/0 conn(0x7f04780d1870 0x7f04781b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:30:54.027 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:54.008 7f04777fe700 1 --2- 172.21.2.102:0/758265573 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0464003b90 0x7f0464006020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:54.027 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:54.008 7f0476ffd700 1 --2- 172.21.2.102:0/758265573 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0464006840 0x7f0464008d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:54.027 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:54.008 7f0477fff700 1 --2- 172.21.2.102:0/758265573 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0464009640 0x7f046400bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:54.027 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:54.009 7f04777fe700 1 -- 172.21.2.102:0/758265573 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0464009640 msgr2=0x7f046400bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:54.027 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:54.009 7f04777fe700 1 --2- 172.21.2.102:0/758265573 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0464009640 0x7f046400bb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:54.027 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:54.009 7f04777fe700 1 -- 172.21.2.102:0/758265573 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0464006840 msgr2=0x7f0464008d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:54.027 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:54.009 7f04777fe700 1 --2- 172.21.2.102:0/758265573 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0464006840 0x7f0464008d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:54.027 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:54.009 7f04777fe700 1 -- 172.21.2.102:0/758265573 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f046400ca70 con 0x7f0464003b90 2021-02-10T11:30:54.027 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:54.009 7f0477fff700 1 -- 172.21.2.102:0/758265573 reap_dead start 2021-02-10T11:30:54.028 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:54.009 7f04777fe700 1 --2- 172.21.2.102:0/758265573 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0464003b90 0x7f0464006020 secure :-1 s=READY pgs=2051 cs=0 l=1 rev1=1 rx=0x7f046804d9c0 tx=0x7f046804dd60).ready entity=mon.0 client_cookie=fc5d1a561767b581 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:54.028 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:54.009 7f04767fc700 1 -- 172.21.2.102:0/758265573 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0468000c90 con 0x7f0464003b90 2021-02-10T11:30:54.028 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:54.009 7f04767fc700 1 -- 172.21.2.102:0/758265573 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04680067d0 con 0x7f0464003b90 2021-02-10T11:30:54.028 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:54.009 7f04767fc700 1 -- 172.21.2.102:0/758265573 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0468002220 con 0x7f0464003b90 2021-02-10T11:30:54.028 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:54.009 7f047dcc1700 1 -- 172.21.2.102:0/758265573 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0464003b90 msgr2=0x7f0464006020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:54.028 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:54.009 7f047dcc1700 1 --2- 172.21.2.102:0/758265573 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0464003b90 0x7f0464006020 secure :-1 s=READY pgs=2051 cs=0 l=1 rev1=1 rx=0x7f046804d9c0 tx=0x7f046804dd60).stop 2021-02-10T11:30:54.028 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:54.009 7f047dcc1700 1 -- 172.21.2.102:0/758265573 shutdown_connections 2021-02-10T11:30:54.028 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:54.009 7f047dcc1700 1 --2- 172.21.2.102:0/758265573 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0464003b90 0x7f0464006020 unknown :-1 s=CLOSED pgs=2051 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:54.028 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:54.009 7f047dcc1700 1 -- 172.21.2.102:0/758265573 >> 172.21.2.102:0/758265573 conn(0x7f04780ce450 msgr2=0x7f04780d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:54.028 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:54.009 7f047dcc1700 1 -- 172.21.2.102:0/758265573 shutdown_connections 2021-02-10T11:30:54.028 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:54.009 7f047dcc1700 1 -- 172.21.2.102:0/758265573 wait complete. 2021-02-10T11:30:54.028 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:54.024 7f047dcc1700 1 Processor -- start 2021-02-10T11:30:54.028 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:54.024 7f047dcc1700 1 -- start start 2021-02-10T11:30:54.030 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:54.024 7f047dcc1700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f04781942b0 0x7f04780b3550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:54.030 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:54.024 7f047dcc1700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0478195070 0x7f0478195470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:54.030 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:54.024 7f047dcc1700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f04780bb8e0 0x7f04780bbce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:54.030 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:54.024 7f047dcc1700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04781946b0 con 0x7f04780bb8e0 2021-02-10T11:30:54.030 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:54.024 7f047dcc1700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04780b3b70 con 0x7f0478195070 2021-02-10T11:30:54.030 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:54.024 7f047dcc1700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04780b3d90 con 0x7f04781942b0 2021-02-10T11:30:54.030 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:54.025 7f0477fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f04780bb8e0 0x7f04780bbce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:54.030 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:54.025 7f0477fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f04780bb8e0 0x7f04780bbce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:33184/0 (socket says 172.21.2.102:33184) 2021-02-10T11:30:54.030 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:54.025 7f0477fff700 1 -- 172.21.2.102:0/1907243257 learned_addr learned my addr 172.21.2.102:0/1907243257 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:30:54.030 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:54.025 7f0477fff700 1 -- 172.21.2.102:0/1907243257 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f04781942b0 msgr2=0x7f04780b3550 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:30:54.030 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:54.025 7f0477fff700 1 --2- 172.21.2.102:0/1907243257 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f04781942b0 0x7f04780b3550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:54.030 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:54.025 7f0477fff700 1 -- 172.21.2.102:0/1907243257 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0478195070 msgr2=0x7f0478195470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:54.031 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:54.025 7f0477fff700 1 --2- 172.21.2.102:0/1907243257 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0478195070 0x7f0478195470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:54.031 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:54.025 7f0477fff700 1 -- 172.21.2.102:0/1907243257 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04781a49e0 con 0x7f04780bb8e0 2021-02-10T11:30:54.031 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:54.025 7f0477fff700 1 --2- 172.21.2.102:0/1907243257 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f04780bb8e0 0x7f04780bbce0 secure :-1 s=READY pgs=2052 cs=0 l=1 rev1=1 rx=0x7f047000a940 tx=0x7f047000ace0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:54.031 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:54.025 7f046ffff700 1 -- 172.21.2.102:0/1907243257 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0470007020 con 0x7f04780bb8e0 2021-02-10T11:30:54.031 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:54.025 7f046ffff700 1 -- 172.21.2.102:0/1907243257 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0470007220 con 0x7f04780bb8e0 2021-02-10T11:30:54.032 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:54.025 7f047dcc1700 1 -- 172.21.2.102:0/1907243257 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04781a49e0 con 0x7f04780bb8e0 2021-02-10T11:30:54.032 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:54.025 7f047dcc1700 1 -- 172.21.2.102:0/1907243257 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04780bc5f0 con 0x7f04780bb8e0 2021-02-10T11:30:54.032 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:54.025 7f046ffff700 1 -- 172.21.2.102:0/1907243257 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0470004720 con 0x7f04780bb8e0 2021-02-10T11:30:54.032 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:54.026 7f046ffff700 1 -- 172.21.2.102:0/1907243257 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f0470007400 con 0x7f04780bb8e0 2021-02-10T11:30:54.032 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:54.026 7f046ffff700 1 --2- 172.21.2.102:0/1907243257 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f045c0186d0 0x7f045c01abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:54.032 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:54.026 7f046ffff700 1 -- 172.21.2.102:0/1907243257 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 4753+0+0 (secure 0 0 0) 0x7f0470032680 con 0x7f04780bb8e0 2021-02-10T11:30:54.032 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:54.026 7f04777fe700 1 --2- 172.21.2.102:0/1907243257 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f045c0186d0 0x7f045c01abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:54.032 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:54.027 7f04777fe700 1 --2- 172.21.2.102:0/1907243257 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f045c0186d0 0x7f045c01abd0 crc :-1 s=READY pgs=911 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:54.058 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:54.057 7f047dcc1700 1 -- 172.21.2.102:0/1907243257 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04780bc5f0 con 0x7f04780bb8e0 2021-02-10T11:30:54.060 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:54.060 7f046ffff700 1 -- 172.21.2.102:0/1907243257 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f04700079c0 con 0x7f04780bb8e0 2021-02-10T11:30:54.185 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:54.184 7f047dcc1700 1 -- 172.21.2.102:0/1907243257 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f04780bc5f0 con 0x7f04780bb8e0 2021-02-10T11:30:54.185 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:54.184 7f046ffff700 1 -- 172.21.2.102:0/1907243257 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v97) v1 ==== 74+0+11145 (secure 0 0 0) 0x7f0470007200 con 0x7f04780bb8e0 2021-02-10T11:30:54.186 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:30:54.186 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":97,"fsid":"68db30ea-dbce-4aef-a6df-f01c6fea0513","created":"2021-02-10 11:03:06.331557","modified":"2021-02-10 11:30:52.913126","last_up_change":"2021-02-10 11:03:32.690562","last_in_change":"2021-02-10 11:03:32.690562","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:03:37.322519","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"be691b4d-a039-408a-af80-9d53f02b6f1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6818","nonce":36088},{"type":"v1","addr":"172.21.2.102:6820","nonce":36088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6822","nonce":36088},{"type":"v1","addr":"172.21.2.102:6824","nonce":36088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6830","nonce":36088},{"type":"v1","addr":"172.21.2.102:6832","nonce":36088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6826","nonce":36088},{"type":"v1","addr":"172.21.2.102:6828","nonce":36088}]},"public_addr":"172.21.2.102:6820/36088","cluster_addr":"172.21.2.102:6824/36088","heartbeat_back_addr":"172.21.2.102:6832/36088","heartbeat_front_addr":"172.21.2.102:6828/36088","state":["exists","up"]},{"osd":1,"uuid":"64e3ccc8-9c69-4d7c-865e-af8e83822635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6819","nonce":36087},{"type":"v1","addr":"172.21.2.102:6821","nonce":36087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6823","nonce":36087},{"type":"v1","addr":"172.21.2.102:6825","nonce":36087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6831","nonce":36087},{"type":"v1","addr":"172.21.2.102:6833","nonce":36087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6827","nonce":36087},{"type":"v1","addr":"172.21.2.102:6829","nonce":36087}]},"public_addr":"172.21.2.102:6821/36087","cluster_addr":"172.21.2.102:6825/36087","heartbeat_back_addr":"172.21.2.102:6833/36087","heartbeat_front_addr":"172.21.2.102:6829/36087","state":["exists","up"]},{"osd":2,"uuid":"153700af-747f-481f-bdf4-d6072d3a842f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6810","nonce":36089},{"type":"v1","addr":"172.21.2.102:6811","nonce":36089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6812","nonce":36089},{"type":"v1","addr":"172.21.2.102:6813","nonce":36089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6816","nonce":36089},{"type":"v1","addr":"172.21.2.102:6817","nonce":36089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6814","nonce":36089},{"type":"v1","addr":"172.21.2.102:6815","nonce":36089}]},"public_addr":"172.21.2.102:6811/36089","cluster_addr":"172.21.2.102:6813/36089","heartbeat_back_addr":"172.21.2.102:6817/36089","heartbeat_front_addr":"172.21.2.102:6815/36089","state":["exists","up"]},{"osd":3,"uuid":"cbc678f4-83a9-45e6-8830-2f7c232b1ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6802","nonce":36090},{"type":"v1","addr":"172.21.2.102:6803","nonce":36090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6804","nonce":36090},{"type":"v1","addr":"172.21.2.102:6805","nonce":36090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6808","nonce":36090},{"type":"v1","addr":"172.21.2.102:6809","nonce":36090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6806","nonce":36090},{"type":"v1","addr":"172.21.2.102:6807","nonce":36090}]},"public_addr":"172.21.2.102:6803/36090","cluster_addr":"172.21.2.102:6805/36090","heartbeat_back_addr":"172.21.2.102:6809/36090","heartbeat_front_addr":"172.21.2.102:6807/36090","state":["exists","up"]},{"osd":4,"uuid":"c01e4008-b258-4f10-afd2-9e7f37214351","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6808","nonce":36368},{"type":"v1","addr":"172.21.2.113:6811","nonce":36368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6814","nonce":36368},{"type":"v1","addr":"172.21.2.113:6818","nonce":36368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6826","nonce":36368},{"type":"v1","addr":"172.21.2.113:6828","nonce":36368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6820","nonce":36368},{"type":"v1","addr":"172.21.2.113:6824","nonce":36368}]},"public_addr":"172.21.2.113:6811/36368","cluster_addr":"172.21.2.113:6818/36368","heartbeat_back_addr":"172.21.2.113:6828/36368","heartbeat_front_addr":"172.21.2.113:6824/36368","state":["exists","up"]},{"osd":5,"uuid":"551f6007-daf8-4a98-8ee0-c2b16bcccb5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6802","nonce":36366},{"type":"v1","addr":"172.21.2.113:6805","nonce":36366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6807","nonce":36366},{"type":"v1","addr":"172.21.2.113:6810","nonce":36366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6819","nonce":36366},{"type":"v1","addr":"172.21.2.113:6822","nonce":36366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6812","nonce":36366},{"type":"v1","addr":"172.21.2.113:6815","nonce":36366}]},"public_addr":"172.21.2.113:6805/36366","cluster_addr":"172.21.2.113:6810/36366","heartbeat_back_addr":"172.21.2.113:6822/36366","heartbeat_front_addr":"172.21.2.113:6815/36366","state":["exists","up"]},{"osd":6,"uuid":"bf67c040-0e50-48be-b102-c3943bc9ebef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6800","nonce":36367},{"type":"v1","addr":"172.21.2.113:6801","nonce":36367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6803","nonce":36367},{"type":"v1","addr":"172.21.2.113:6804","nonce":36367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6813","nonce":36367},{"type":"v1","addr":"172.21.2.113:6816","nonce":36367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6806","nonce":36367},{"type":"v1","addr":"172.21.2.113:6809","nonce":36367}]},"public_addr":"172.21.2.113:6801/36367","cluster_addr":"172.21.2.113:6804/36367","heartbeat_back_addr":"172.21.2.113:6816/36367","heartbeat_front_addr":"172.21.2.113:6809/36367","state":["exists","up"]},{"osd":7,"uuid":"f5461f80-018d-428f-be89-2eb2bcbe7bb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6817","nonce":36365},{"type":"v1","addr":"172.21.2.113:6821","nonce":36365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6823","nonce":36365},{"type":"v1","addr":"172.21.2.113:6825","nonce":36365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6830","nonce":36365},{"type":"v1","addr":"172.21.2.113:6831","nonce":36365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6827","nonce":36365},{"type":"v1","addr":"172.21.2.113:6829","nonce":36365}]},"public_addr":"172.21.2.113:6821/36365","cluster_addr":"172.21.2.113:6825/36365","heartbeat_back_addr":"172.21.2.113:6831/36365","heartbeat_front_addr":"172.21.2.113:6829/36365","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:30:54.187 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:54.186 7f047dcc1700 1 -- 172.21.2.102:0/1907243257 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f045c0186d0 msgr2=0x7f045c01abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:54.187 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:54.186 7f047dcc1700 1 --2- 172.21.2.102:0/1907243257 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f045c0186d0 0x7f045c01abd0 crc :-1 s=READY pgs=911 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:54.187 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:54.186 7f047dcc1700 1 -- 172.21.2.102:0/1907243257 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f04780bb8e0 msgr2=0x7f04780bbce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:54.187 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:54.186 7f047dcc1700 1 --2- 172.21.2.102:0/1907243257 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f04780bb8e0 0x7f04780bbce0 secure :-1 s=READY pgs=2052 cs=0 l=1 rev1=1 rx=0x7f047000a940 tx=0x7f047000ace0).stop 2021-02-10T11:30:54.188 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:54.186 7f047dcc1700 1 -- 172.21.2.102:0/1907243257 shutdown_connections 2021-02-10T11:30:54.188 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:54.186 7f047dcc1700 1 --2- 172.21.2.102:0/1907243257 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f04780bb8e0 0x7f04780bbce0 unknown :-1 s=CLOSED pgs=2052 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:54.188 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:54.186 7f047dcc1700 1 --2- 172.21.2.102:0/1907243257 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f045c0186d0 0x7f045c01abd0 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:54.188 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:54.186 7f047dcc1700 1 --2- 172.21.2.102:0/1907243257 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0478195070 0x7f0478195470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:54.188 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:54.186 7f047dcc1700 1 --2- 172.21.2.102:0/1907243257 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f04781942b0 0x7f04780b3550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:54.188 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:54.186 7f047dcc1700 1 -- 172.21.2.102:0/1907243257 >> 172.21.2.102:0/1907243257 conn(0x7f04780b3040 msgr2=0x7f04780c3250 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:54.188 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:54.187 7f047dcc1700 1 -- 172.21.2.102:0/1907243257 shutdown_connections 2021-02-10T11:30:54.188 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:54.187 7f047dcc1700 1 -- 172.21.2.102:0/1907243257 wait complete. 2021-02-10T11:30:54.196 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:03:37.322519', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2021-02-10T11:30:54.196 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:30:54.628 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:54.611 7fcc0adf0700 1 Processor -- start 2021-02-10T11:30:54.628 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:54.611 7fcc0adf0700 1 -- start start 2021-02-10T11:30:54.628 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:54.611 7fcc0adf0700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fcc040d1870 0x7fcc041b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:54.628 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:54.611 7fcc0adf0700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc040fe1c0 con 0x7fcc041b0ae0 2021-02-10T11:30:54.629 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:54.611 7fcc0adf0700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc04109a60 con 0x7fcc040cff70 2021-02-10T11:30:54.629 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:54.611 7fcc0adf0700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc04088070 con 0x7fcc040d1870 2021-02-10T11:30:54.629 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:54.611 7fcc08c0c700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fcc040d1870 0x7fcc041b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:54.629 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:54.612 7fcc08c0c700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fcc040d1870 0x7fcc041b04c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:33190/0 (socket says 172.21.2.102:33190) 2021-02-10T11:30:54.629 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:54.612 7fcc08c0c700 1 -- 172.21.2.102:0/2183237321 learned_addr learned my addr 172.21.2.102:0/2183237321 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:30:54.629 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:54.612 7fcc08c0c700 1 -- 172.21.2.102:0/2183237321 >> v1:172.21.2.113:6789/0 conn(0x7fcc040cff70 legacy=0x7fcc04109130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:54.629 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:54.612 7fcc08c0c700 1 -- 172.21.2.102:0/2183237321 >> v1:172.21.2.102:6789/0 conn(0x7fcc041b0ae0 legacy=0x7fcc04108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:54.629 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:54.612 7fcc08c0c700 1 -- 172.21.2.102:0/2183237321 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc04088260 con 0x7fcc040d1870 2021-02-10T11:30:54.629 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:54.612 7fcc08c0c700 1 --2- 172.21.2.102:0/2183237321 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcc040d1870 0x7fcc041b04c0 secure :-1 s=READY pgs=2054 cs=0 l=1 rev1=1 rx=0x7fcbec006700 tx=0x7fcbec009b80).ready entity=mon.0 client_cookie=9328bf4e63aedaaf server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:54.629 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:54.612 7fcc037fe700 1 -- 172.21.2.102:0/2183237321 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcbec00c0c0 con 0x7fcc040d1870 2021-02-10T11:30:54.629 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:54.612 7fcc037fe700 1 -- 172.21.2.102:0/2183237321 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbec00c440 con 0x7fcc040d1870 2021-02-10T11:30:54.629 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:54.612 7fcc037fe700 1 -- 172.21.2.102:0/2183237321 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcbec00c960 con 0x7fcc040d1870 2021-02-10T11:30:54.630 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:54.613 7fcc0adf0700 1 -- 172.21.2.102:0/2183237321 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcc040d1870 msgr2=0x7fcc041b04c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:54.630 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:54.613 7fcc0adf0700 1 --2- 172.21.2.102:0/2183237321 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcc040d1870 0x7fcc041b04c0 secure :-1 s=READY pgs=2054 cs=0 l=1 rev1=1 rx=0x7fcbec006700 tx=0x7fcbec009b80).stop 2021-02-10T11:30:54.630 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:54.613 7fcc0adf0700 1 -- 172.21.2.102:0/2183237321 shutdown_connections 2021-02-10T11:30:54.630 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:54.613 7fcc0adf0700 1 --2- 172.21.2.102:0/2183237321 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcc040d1870 0x7fcc041b04c0 unknown :-1 s=CLOSED pgs=2054 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:54.630 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:54.613 7fcc0adf0700 1 -- 172.21.2.102:0/2183237321 >> 172.21.2.102:0/2183237321 conn(0x7fcc040ce420 msgr2=0x7fcc040d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:54.630 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:54.613 7fcc0adf0700 1 -- 172.21.2.102:0/2183237321 shutdown_connections 2021-02-10T11:30:54.630 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:54.613 7fcc0adf0700 1 -- 172.21.2.102:0/2183237321 wait complete. 2021-02-10T11:30:54.632 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:54.628 7fcc0adf0700 1 Processor -- start 2021-02-10T11:30:54.632 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:54.628 7fcc0adf0700 1 -- start start 2021-02-10T11:30:54.632 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:54.628 7fcc0adf0700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcc0408a650 0x7fcc041c1f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:54.632 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:54.628 7fcc0adf0700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fcc04091e10 0x7fcc04092210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:54.632 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:54.628 7fcc0adf0700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcc041076a0 0x7fcc04107aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:54.632 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:54.628 7fcc0adf0700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc04092830 con 0x7fcc041076a0 2021-02-10T11:30:54.632 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:54.628 7fcc0adf0700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc041003e0 con 0x7fcc04091e10 2021-02-10T11:30:54.632 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:54.628 7fcc0adf0700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc041c2540 con 0x7fcc0408a650 2021-02-10T11:30:54.632 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:54.628 7fcc0940d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcc041076a0 0x7fcc04107aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:54.632 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:54.628 7fcc0940d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcc041076a0 0x7fcc04107aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:33192/0 (socket says 172.21.2.102:33192) 2021-02-10T11:30:54.632 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:54.629 7fcc0940d700 1 -- 172.21.2.102:0/1731672748 learned_addr learned my addr 172.21.2.102:0/1731672748 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:30:54.633 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:54.629 7fcc0940d700 1 -- 172.21.2.102:0/1731672748 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcc0408a650 msgr2=0x7fcc041c1f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:54.633 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:54.629 7fcc0940d700 1 --2- 172.21.2.102:0/1731672748 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcc0408a650 0x7fcc041c1f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:54.633 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:54.629 7fcc0940d700 1 -- 172.21.2.102:0/1731672748 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fcc04091e10 msgr2=0x7fcc04092210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:54.633 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:54.629 7fcc0940d700 1 --2- 172.21.2.102:0/1731672748 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fcc04091e10 0x7fcc04092210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:54.633 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:54.629 7fcc0940d700 1 -- 172.21.2.102:0/1731672748 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc0408aa50 con 0x7fcc041076a0 2021-02-10T11:30:54.633 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:54.629 7fcc0940d700 1 --2- 172.21.2.102:0/1731672748 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcc041076a0 0x7fcc04107aa0 secure :-1 s=READY pgs=2055 cs=0 l=1 rev1=1 rx=0x7fcbfc009940 tx=0x7fcbfc00bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:54.633 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:54.629 7fcc017fa700 1 -- 172.21.2.102:0/1731672748 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcbfc000f90 con 0x7fcc041076a0 2021-02-10T11:30:54.633 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:54.629 7fcc017fa700 1 -- 172.21.2.102:0/1731672748 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbfc00c2f0 con 0x7fcc041076a0 2021-02-10T11:30:54.633 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:54.629 7fcc0adf0700 1 -- 172.21.2.102:0/1731672748 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc0408aa50 con 0x7fcc041076a0 2021-02-10T11:30:54.633 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:54.629 7fcc0adf0700 1 -- 172.21.2.102:0/1731672748 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc041083b0 con 0x7fcc041076a0 2021-02-10T11:30:54.633 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:54.629 7fcc017fa700 1 -- 172.21.2.102:0/1731672748 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcbfc00c750 con 0x7fcc041076a0 2021-02-10T11:30:54.633 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:54.630 7fcc017fa700 1 -- 172.21.2.102:0/1731672748 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fcbfc001220 con 0x7fcc041076a0 2021-02-10T11:30:54.633 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:54.630 7fcc017fa700 1 --2- 172.21.2.102:0/1731672748 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcbe80186c0 0x7fcbe801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:54.634 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:54.630 7fcc017fa700 1 -- 172.21.2.102:0/1731672748 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 4753+0+0 (secure 0 0 0) 0x7fcbfc032850 con 0x7fcc041076a0 2021-02-10T11:30:54.634 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:54.630 7fcc08c0c700 1 --2- 172.21.2.102:0/1731672748 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcbe80186c0 0x7fcbe801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:54.634 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:54.631 7fcc08c0c700 1 --2- 172.21.2.102:0/1731672748 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcbe80186c0 0x7fcbe801abc0 crc :-1 s=READY pgs=912 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:54.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:54.661 7fcc0adf0700 1 -- 172.21.2.102:0/1731672748 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc040918e0 con 0x7fcc041076a0 2021-02-10T11:30:54.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:54.664 7fcc017fa700 1 -- 172.21.2.102:0/1731672748 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fcbfc002ef0 con 0x7fcc041076a0 2021-02-10T11:30:54.792 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:54.791 7fcc0adf0700 1 -- 172.21.2.102:0/1731672748 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7fcc040918e0 con 0x7fcc041076a0 2021-02-10T11:30:54.792 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:54.791 7fcc017fa700 1 -- 172.21.2.102:0/1731672748 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v97) v1 ==== 92+0+10 (secure 0 0 0) 0x7fcbfc000f40 con 0x7fcc041076a0 2021-02-10T11:30:54.792 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:30:54.794 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:54.793 7fcc0adf0700 1 -- 172.21.2.102:0/1731672748 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcbe80186c0 msgr2=0x7fcbe801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:54.794 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:54.793 7fcc0adf0700 1 --2- 172.21.2.102:0/1731672748 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcbe80186c0 0x7fcbe801abc0 crc :-1 s=READY pgs=912 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:54.794 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:54.793 7fcc0adf0700 1 -- 172.21.2.102:0/1731672748 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcc041076a0 msgr2=0x7fcc04107aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:54.794 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:54.793 7fcc0adf0700 1 --2- 172.21.2.102:0/1731672748 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcc041076a0 0x7fcc04107aa0 secure :-1 s=READY pgs=2055 cs=0 l=1 rev1=1 rx=0x7fcbfc009940 tx=0x7fcbfc00bcf0).stop 2021-02-10T11:30:54.795 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:54.793 7fcc0adf0700 1 -- 172.21.2.102:0/1731672748 shutdown_connections 2021-02-10T11:30:54.795 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:54.793 7fcc0adf0700 1 --2- 172.21.2.102:0/1731672748 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcc041076a0 0x7fcc04107aa0 unknown :-1 s=CLOSED pgs=2055 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:54.795 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:54.793 7fcc0adf0700 1 --2- 172.21.2.102:0/1731672748 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcbe80186c0 0x7fcbe801abc0 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:54.795 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:54.793 7fcc0adf0700 1 --2- 172.21.2.102:0/1731672748 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fcc04091e10 0x7fcc04092210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:54.795 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:54.793 7fcc0adf0700 1 --2- 172.21.2.102:0/1731672748 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcc0408a650 0x7fcc041c1f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:54.795 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:54.793 7fcc0adf0700 1 -- 172.21.2.102:0/1731672748 >> 172.21.2.102:0/1731672748 conn(0x7fcc040bcff0 msgr2=0x7fcc0408e9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:54.795 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:54.794 7fcc0adf0700 1 -- 172.21.2.102:0/1731672748 shutdown_connections 2021-02-10T11:30:54.795 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:54.794 7fcc0adf0700 1 -- 172.21.2.102:0/1731672748 wait complete. 2021-02-10T11:30:54.804 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --format=json-pretty osd lspools 2021-02-10T11:30:55.230 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:55.213 7f46fde9b700 1 Processor -- start 2021-02-10T11:30:55.230 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:55.213 7f46fde9b700 1 -- start start 2021-02-10T11:30:55.230 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:55.214 7f46fde9b700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f46f80d1870 0x7f46f81b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:55.230 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:55.214 7f46fde9b700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f46f81b0a10 0x7f46f80cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:55.230 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:55.214 7f46fde9b700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f46f80d1c70 0x7f46f80cf950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:55.231 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:55.214 7f46fde9b700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46f80fe230 con 0x7f46f80d1c70 2021-02-10T11:30:55.231 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:55.214 7f46fde9b700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46f80e5ed0 con 0x7f46f81b0a10 2021-02-10T11:30:55.231 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:55.214 7f46fde9b700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46f80f0b10 con 0x7f46f80d1870 2021-02-10T11:30:55.231 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:55.214 7f46f7fff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f46f80d1c70 0x7f46f80cf950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:55.231 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:55.214 7f46f7fff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f46f80d1c70 0x7f46f80cf950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:33204/0 (socket says 172.21.2.102:33204) 2021-02-10T11:30:55.231 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:55.214 7f46f7fff700 1 -- 172.21.2.102:0/3051458581 learned_addr learned my addr 172.21.2.102:0/3051458581 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:30:55.231 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:55.214 7f46f7fff700 1 -- 172.21.2.102:0/3051458581 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f46f80d1870 msgr2=0x7f46f81b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:55.231 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:55.214 7f46f7fff700 1 --2- 172.21.2.102:0/3051458581 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f46f80d1870 0x7f46f81b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:55.231 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:55.214 7f46f7fff700 1 -- 172.21.2.102:0/3051458581 >> v2:172.21.2.113:3300/0 conn(0x7f46f81b0a10 msgr2=0x7f46f80cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:55.231 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:55.214 7f46f7fff700 1 --2- 172.21.2.102:0/3051458581 >> v2:172.21.2.113:3300/0 conn(0x7f46f81b0a10 0x7f46f80cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:55.231 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:55.214 7f46f7fff700 1 -- 172.21.2.102:0/3051458581 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46f80cff70 con 0x7f46f80d1c70 2021-02-10T11:30:55.231 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:55.214 7f46f7fff700 1 --2- 172.21.2.102:0/3051458581 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f46f80d1c70 0x7f46f80cf950 secure :-1 s=READY pgs=2056 cs=0 l=1 rev1=1 rx=0x7f46e8006b80 tx=0x7f46e8009b90).ready entity=mon.0 client_cookie=bc7b855f30f73aa2 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:55.232 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:55.215 7f46f67fc700 1 -- 172.21.2.102:0/3051458581 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f46e80022e0 con 0x7f46f80d1c70 2021-02-10T11:30:55.232 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:55.215 7f46f67fc700 1 -- 172.21.2.102:0/3051458581 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46e8002560 con 0x7f46f80d1c70 2021-02-10T11:30:55.232 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:55.215 7f46f67fc700 1 -- 172.21.2.102:0/3051458581 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f46e8002a80 con 0x7f46f80d1c70 2021-02-10T11:30:55.232 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:55.215 7f46fde9b700 1 -- 172.21.2.102:0/3051458581 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f46f80d1c70 msgr2=0x7f46f80cf950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:55.232 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:55.215 7f46fde9b700 1 --2- 172.21.2.102:0/3051458581 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f46f80d1c70 0x7f46f80cf950 secure :-1 s=READY pgs=2056 cs=0 l=1 rev1=1 rx=0x7f46e8006b80 tx=0x7f46e8009b90).stop 2021-02-10T11:30:55.232 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:55.215 7f46fde9b700 1 -- 172.21.2.102:0/3051458581 shutdown_connections 2021-02-10T11:30:55.232 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:55.215 7f46fde9b700 1 --2- 172.21.2.102:0/3051458581 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f46f80d1c70 0x7f46f80cf950 unknown :-1 s=CLOSED pgs=2056 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:55.232 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:55.215 7f46fde9b700 1 --2- 172.21.2.102:0/3051458581 >> v2:172.21.2.113:3300/0 conn(0x7f46f81b0a10 0x7f46f80cf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:55.232 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:55.215 7f46fde9b700 1 --2- 172.21.2.102:0/3051458581 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f46f80d1870 0x7f46f81b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:55.232 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:55.215 7f46fde9b700 1 -- 172.21.2.102:0/3051458581 >> 172.21.2.102:0/3051458581 conn(0x7f46f80ce450 msgr2=0x7f46f80d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:55.232 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:55.215 7f46fde9b700 1 -- 172.21.2.102:0/3051458581 shutdown_connections 2021-02-10T11:30:55.232 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:55.215 7f46fde9b700 1 -- 172.21.2.102:0/3051458581 wait complete. 2021-02-10T11:30:55.234 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:55.230 7f46fde9b700 1 Processor -- start 2021-02-10T11:30:55.234 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:55.230 7f46fde9b700 1 -- start start 2021-02-10T11:30:55.234 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:55.230 7f46fde9b700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f46f80aaaa0 0x7f46f81a8f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:55.234 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:55.230 7f46fde9b700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f46f81b2f50 0x7f46f819ad30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:55.234 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:55.230 7f46fde9b700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f46f81b3350 0x7f46f819b350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:55.234 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:55.230 7f46fde9b700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46f81a9710 con 0x7f46f80aaaa0 2021-02-10T11:30:55.234 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:55.230 7f46fde9b700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46f818b220 con 0x7f46f81b2f50 2021-02-10T11:30:55.235 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:55.230 7f46fde9b700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46f818b440 con 0x7f46f81b3350 2021-02-10T11:30:55.235 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:55.230 7f46f77fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f46f80aaaa0 0x7f46f81a8f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:55.235 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:55.230 7f46f77fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f46f80aaaa0 0x7f46f81a8f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:33206/0 (socket says 172.21.2.102:33206) 2021-02-10T11:30:55.235 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:55.230 7f46f77fe700 1 -- 172.21.2.102:0/4186831519 learned_addr learned my addr 172.21.2.102:0/4186831519 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:30:55.235 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:55.231 7f46f77fe700 1 -- 172.21.2.102:0/4186831519 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f46f81b3350 msgr2=0x7f46f819b350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:55.235 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:55.231 7f46f77fe700 1 --2- 172.21.2.102:0/4186831519 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f46f81b3350 0x7f46f819b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:55.235 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:55.231 7f46f77fe700 1 -- 172.21.2.102:0/4186831519 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f46f81b2f50 msgr2=0x7f46f819ad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:55.235 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:55.231 7f46f77fe700 1 --2- 172.21.2.102:0/4186831519 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f46f81b2f50 0x7f46f819ad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:55.235 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:55.231 7f46f77fe700 1 -- 172.21.2.102:0/4186831519 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46f818b660 con 0x7f46f80aaaa0 2021-02-10T11:30:55.235 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:55.231 7f46f77fe700 1 --2- 172.21.2.102:0/4186831519 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f46f80aaaa0 0x7f46f81a8f50 secure :-1 s=READY pgs=2057 cs=0 l=1 rev1=1 rx=0x7f46ec006f00 tx=0x7f46ec00b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:55.235 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:55.231 7f46dffff700 1 -- 172.21.2.102:0/4186831519 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f46ec00bd30 con 0x7f46f80aaaa0 2021-02-10T11:30:55.235 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:55.231 7f46dffff700 1 -- 172.21.2.102:0/4186831519 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46ec000c90 con 0x7f46f80aaaa0 2021-02-10T11:30:55.235 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:55.231 7f46dffff700 1 -- 172.21.2.102:0/4186831519 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f46ec0022a0 con 0x7f46f80aaaa0 2021-02-10T11:30:55.236 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:55.231 7f46fde9b700 1 -- 172.21.2.102:0/4186831519 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46f818b660 con 0x7f46f80aaaa0 2021-02-10T11:30:55.236 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:55.231 7f46fde9b700 1 -- 172.21.2.102:0/4186831519 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46f818b660 con 0x7f46f80aaaa0 2021-02-10T11:30:55.236 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:55.231 7f46dffff700 1 -- 172.21.2.102:0/4186831519 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f46ec005310 con 0x7f46f80aaaa0 2021-02-10T11:30:55.236 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:55.231 7f46dffff700 1 --2- 172.21.2.102:0/4186831519 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f46d80186c0 0x7f46d801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:55.236 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:55.231 7f46dffff700 1 -- 172.21.2.102:0/4186831519 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 4753+0+0 (secure 0 0 0) 0x7f46ec032220 con 0x7f46f80aaaa0 2021-02-10T11:30:55.236 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:55.232 7f46f6ffd700 1 --2- 172.21.2.102:0/4186831519 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f46d80186c0 0x7f46d801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:55.236 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:55.232 7f46f6ffd700 1 --2- 172.21.2.102:0/4186831519 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f46d80186c0 0x7f46d801abc0 crc :-1 s=READY pgs=913 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:55.265 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:55.264 7f46fde9b700 1 -- 172.21.2.102:0/4186831519 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46f80ebc60 con 0x7f46f80aaaa0 2021-02-10T11:30:55.267 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:55.267 7f46dffff700 1 -- 172.21.2.102:0/4186831519 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f46ec006b90 con 0x7f46f80aaaa0 2021-02-10T11:30:55.391 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:55.390 7f46fde9b700 1 -- 172.21.2.102:0/4186831519 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f46f80f0b10 con 0x7f46f80aaaa0 2021-02-10T11:30:55.392 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:55.391 7f46dffff700 1 -- 172.21.2.102:0/4186831519 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v97) v1 ==== 84+0+64 (secure 0 0 0) 0x7f46ec00bd30 con 0x7f46f80aaaa0 2021-02-10T11:30:55.392 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:30:55.392 INFO:teuthology.orchestra.run.gibba002.stdout:[ 2021-02-10T11:30:55.392 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:30:55.392 INFO:teuthology.orchestra.run.gibba002.stdout: "poolnum": 1, 2021-02-10T11:30:55.392 INFO:teuthology.orchestra.run.gibba002.stdout: "poolname": "rbd" 2021-02-10T11:30:55.392 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:30:55.392 INFO:teuthology.orchestra.run.gibba002.stdout:] 2021-02-10T11:30:55.394 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:55.393 7f46fde9b700 1 -- 172.21.2.102:0/4186831519 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f46d80186c0 msgr2=0x7f46d801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:55.394 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:55.393 7f46fde9b700 1 --2- 172.21.2.102:0/4186831519 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f46d80186c0 0x7f46d801abc0 crc :-1 s=READY pgs=913 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:55.394 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:55.393 7f46fde9b700 1 -- 172.21.2.102:0/4186831519 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f46f80aaaa0 msgr2=0x7f46f81a8f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:55.394 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:55.393 7f46fde9b700 1 --2- 172.21.2.102:0/4186831519 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f46f80aaaa0 0x7f46f81a8f50 secure :-1 s=READY pgs=2057 cs=0 l=1 rev1=1 rx=0x7f46ec006f00 tx=0x7f46ec00b940).stop 2021-02-10T11:30:55.394 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:55.393 7f46fde9b700 1 -- 172.21.2.102:0/4186831519 shutdown_connections 2021-02-10T11:30:55.394 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:55.393 7f46fde9b700 1 --2- 172.21.2.102:0/4186831519 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f46f81b3350 0x7f46f819b350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:55.394 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:55.393 7f46fde9b700 1 --2- 172.21.2.102:0/4186831519 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f46d80186c0 0x7f46d801abc0 unknown :-1 s=CLOSED pgs=913 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:55.394 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:55.393 7f46fde9b700 1 --2- 172.21.2.102:0/4186831519 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f46f81b2f50 0x7f46f819ad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:55.394 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:55.393 7f46fde9b700 1 --2- 172.21.2.102:0/4186831519 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f46f80aaaa0 0x7f46f81a8f50 unknown :-1 s=CLOSED pgs=2057 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:55.395 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:55.393 7f46fde9b700 1 -- 172.21.2.102:0/4186831519 >> 172.21.2.102:0/4186831519 conn(0x7f46f8196190 msgr2=0x7f46f80ebe90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:55.395 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:55.393 7f46fde9b700 1 -- 172.21.2.102:0/4186831519 shutdown_connections 2021-02-10T11:30:55.395 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:55.393 7f46fde9b700 1 -- 172.21.2.102:0/4186831519 wait complete. 2021-02-10T11:30:55.402 INFO:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2021-02-10T11:30:55.402 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool create cephfs_metadata 8 2021-02-10T11:30:55.835 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:55.817 7fcbf959d700 1 Processor -- start 2021-02-10T11:30:55.835 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:55.817 7fcbf959d700 1 -- start start 2021-02-10T11:30:55.835 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:55.817 7fcbf959d700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fcbf40d1c70 0x7fcbf40cef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:55.835 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:55.817 7fcbf959d700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbf40d0170 con 0x7fcbf40d1870 2021-02-10T11:30:55.835 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:55.817 7fcbf959d700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbf40fe1c0 con 0x7fcbf40cf580 2021-02-10T11:30:55.835 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:55.817 7fcbf959d700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcbf41b0cb0 con 0x7fcbf40d1c70 2021-02-10T11:30:55.835 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:55.818 7fcbf2abe700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fcbf40d1c70 0x7fcbf40cef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:55.835 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:55.818 7fcbf32bf700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fcbf40d1870 0x7fcbf41b04c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:53670/0 (socket says 172.21.2.102:53670) 2021-02-10T11:30:55.835 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:55.818 7fcbf32bf700 1 -- 172.21.2.102:0/980620606 learned_addr learned my addr 172.21.2.102:0/980620606 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:30:55.835 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:55.818 7fcbf2abe700 1 -- 172.21.2.102:0/980620606 >> v1:172.21.2.113:6789/0 conn(0x7fcbf40cf580 legacy=0x7fcbf40cf980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:55.835 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:55.818 7fcbf2abe700 1 -- 172.21.2.102:0/980620606 >> v1:172.21.2.102:6789/0 conn(0x7fcbf40d1870 legacy=0x7fcbf41b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:55.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:55.818 7fcbf2abe700 1 -- 172.21.2.102:0/980620606 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbf4109a60 con 0x7fcbf40d1c70 2021-02-10T11:30:55.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:55.818 7fcbf2abe700 1 --2- 172.21.2.102:0/980620606 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcbf40d1c70 0x7fcbf40cef60 secure :-1 s=READY pgs=2059 cs=0 l=1 rev1=1 rx=0x7fcbdc006650 tx=0x7fcbdc009a80).ready entity=mon.0 client_cookie=97ded356ec910379 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:55.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:55.818 7fcbf22bd700 1 -- 172.21.2.102:0/980620606 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcbdc0009b0 con 0x7fcbf40d1c70 2021-02-10T11:30:55.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:55.818 7fcbf22bd700 1 -- 172.21.2.102:0/980620606 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbdc00c0c0 con 0x7fcbf40d1c70 2021-02-10T11:30:55.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:55.818 7fcbf22bd700 1 -- 172.21.2.102:0/980620606 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcbdc00c5e0 con 0x7fcbf40d1c70 2021-02-10T11:30:55.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:55.818 7fcbf959d700 1 -- 172.21.2.102:0/980620606 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcbf40d1c70 msgr2=0x7fcbf40cef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:55.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:55.818 7fcbf959d700 1 --2- 172.21.2.102:0/980620606 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcbf40d1c70 0x7fcbf40cef60 secure :-1 s=READY pgs=2059 cs=0 l=1 rev1=1 rx=0x7fcbdc006650 tx=0x7fcbdc009a80).stop 2021-02-10T11:30:55.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:55.819 7fcbf959d700 1 -- 172.21.2.102:0/980620606 shutdown_connections 2021-02-10T11:30:55.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:55.819 7fcbf959d700 1 --2- 172.21.2.102:0/980620606 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcbf40d1c70 0x7fcbf40cef60 unknown :-1 s=CLOSED pgs=2059 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:55.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:55.819 7fcbf959d700 1 -- 172.21.2.102:0/980620606 >> 172.21.2.102:0/980620606 conn(0x7fcbf40ce420 msgr2=0x7fcbf40d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:55.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:55.819 7fcbf959d700 1 -- 172.21.2.102:0/980620606 shutdown_connections 2021-02-10T11:30:55.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:55.819 7fcbf959d700 1 -- 172.21.2.102:0/980620606 wait complete. 2021-02-10T11:30:55.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:55.834 7fcbf959d700 1 Processor -- start 2021-02-10T11:30:55.837 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:55.834 7fcbf959d700 1 -- start start 2021-02-10T11:30:55.838 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:55.834 7fcbf959d700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcbf40a25d0 0x7fcbf40a29d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:55.838 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:55.834 7fcbf959d700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fcbf40a2ff0 0x7fcbf40a33f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:55.838 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:55.834 7fcbf959d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcbf40b5650 0x7fcbf40b5a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:55.838 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:55.834 7fcbf959d700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbf41c3780 con 0x7fcbf40b5650 2021-02-10T11:30:55.838 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:55.834 7fcbf959d700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbf40a20c0 con 0x7fcbf40a2ff0 2021-02-10T11:30:55.838 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:55.834 7fcbf959d700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbf40b60a0 con 0x7fcbf40a25d0 2021-02-10T11:30:55.838 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:55.835 7fcbf3ac0700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcbf40b5650 0x7fcbf40b5a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:55.838 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:55.835 7fcbf3ac0700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcbf40b5650 0x7fcbf40b5a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:33222/0 (socket says 172.21.2.102:33222) 2021-02-10T11:30:55.838 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:55.835 7fcbf3ac0700 1 -- 172.21.2.102:0/2940429580 learned_addr learned my addr 172.21.2.102:0/2940429580 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:30:55.838 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:55.835 7fcbf3ac0700 1 -- 172.21.2.102:0/2940429580 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcbf40a25d0 msgr2=0x7fcbf40a29d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:55.839 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:55.835 7fcbf3ac0700 1 --2- 172.21.2.102:0/2940429580 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcbf40a25d0 0x7fcbf40a29d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:55.839 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:55.835 7fcbf3ac0700 1 -- 172.21.2.102:0/2940429580 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fcbf40a2ff0 msgr2=0x7fcbf40a33f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:55.839 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:55.835 7fcbf3ac0700 1 --2- 172.21.2.102:0/2940429580 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fcbf40a2ff0 0x7fcbf40a33f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:55.839 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:55.835 7fcbf3ac0700 1 -- 172.21.2.102:0/2940429580 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbf40b62c0 con 0x7fcbf40b5650 2021-02-10T11:30:55.839 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:55.835 7fcbf3ac0700 1 --2- 172.21.2.102:0/2940429580 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcbf40b5650 0x7fcbf40b5a50 secure :-1 s=READY pgs=2060 cs=0 l=1 rev1=1 rx=0x7fcbec009940 tx=0x7fcbec00bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:55.839 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:55.835 7fcbebfff700 1 -- 172.21.2.102:0/2940429580 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcbec000f90 con 0x7fcbf40b5650 2021-02-10T11:30:55.839 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:55.835 7fcbebfff700 1 -- 172.21.2.102:0/2940429580 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbec000d30 con 0x7fcbf40b5650 2021-02-10T11:30:55.839 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:55.835 7fcbebfff700 1 -- 172.21.2.102:0/2940429580 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcbec00c570 con 0x7fcbf40b5650 2021-02-10T11:30:55.839 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:55.835 7fcbf959d700 1 -- 172.21.2.102:0/2940429580 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbf40b62c0 con 0x7fcbf40b5650 2021-02-10T11:30:55.839 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:55.835 7fcbf959d700 1 -- 172.21.2.102:0/2940429580 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbf40b6850 con 0x7fcbf40b5650 2021-02-10T11:30:55.839 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:55.836 7fcbebfff700 1 -- 172.21.2.102:0/2940429580 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fcbec001220 con 0x7fcbf40b5650 2021-02-10T11:30:55.839 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:55.836 7fcbebfff700 1 --2- 172.21.2.102:0/2940429580 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcbd80186c0 0x7fcbd801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:55.839 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:55.836 7fcbebfff700 1 -- 172.21.2.102:0/2940429580 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 4753+0+0 (secure 0 0 0) 0x7fcbec0325e0 con 0x7fcbf40b5650 2021-02-10T11:30:55.839 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:55.836 7fcbf32bf700 1 --2- 172.21.2.102:0/2940429580 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcbd80186c0 0x7fcbd801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:55.840 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:55.836 7fcbf32bf700 1 --2- 172.21.2.102:0/2940429580 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcbd80186c0 0x7fcbd801abc0 crc :-1 s=READY pgs=914 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:55.868 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:55.867 7fcbf959d700 1 -- 172.21.2.102:0/2940429580 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcbf40807d0 con 0x7fcbf40b5650 2021-02-10T11:30:55.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:55.870 7fcbebfff700 1 -- 172.21.2.102:0/2940429580 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fcbec000f40 con 0x7fcbf40b5650 2021-02-10T11:30:55.998 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:55.997 7fcbf959d700 1 -- 172.21.2.102:0/2940429580 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"} v 0) v1 -- 0x7fcbf4088070 con 0x7fcbf40b5650 2021-02-10T11:30:56.403 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:56.402 7fcbebfff700 1 -- 172.21.2.102:0/2940429580 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"}]=0 pool 'cephfs_metadata' created v98) v1 ==== 133+0+0 (secure 0 0 0) 0x7fcbec01f870 con 0x7fcbf40b5650 2021-02-10T11:30:56.414 INFO:teuthology.orchestra.run.gibba002.stderr:pool 'cephfs_metadata' created 2021-02-10T11:30:56.416 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:56.415 7fcbf959d700 1 -- 172.21.2.102:0/2940429580 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcbd80186c0 msgr2=0x7fcbd801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:56.416 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:56.415 7fcbf959d700 1 --2- 172.21.2.102:0/2940429580 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcbd80186c0 0x7fcbd801abc0 crc :-1 s=READY pgs=914 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:56.416 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:56.415 7fcbf959d700 1 -- 172.21.2.102:0/2940429580 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcbf40b5650 msgr2=0x7fcbf40b5a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:56.416 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:56.415 7fcbf959d700 1 --2- 172.21.2.102:0/2940429580 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcbf40b5650 0x7fcbf40b5a50 secure :-1 s=READY pgs=2060 cs=0 l=1 rev1=1 rx=0x7fcbec009940 tx=0x7fcbec00bcf0).stop 2021-02-10T11:30:56.416 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:56.415 7fcbf959d700 1 -- 172.21.2.102:0/2940429580 shutdown_connections 2021-02-10T11:30:56.416 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:56.415 7fcbf959d700 1 --2- 172.21.2.102:0/2940429580 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcbf40b5650 0x7fcbf40b5a50 unknown :-1 s=CLOSED pgs=2060 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:56.417 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:56.415 7fcbf959d700 1 --2- 172.21.2.102:0/2940429580 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcbd80186c0 0x7fcbd801abc0 unknown :-1 s=CLOSED pgs=914 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:56.417 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:56.415 7fcbf959d700 1 --2- 172.21.2.102:0/2940429580 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fcbf40a2ff0 0x7fcbf40a33f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:56.417 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:56.415 7fcbf959d700 1 --2- 172.21.2.102:0/2940429580 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcbf40a25d0 0x7fcbf40a29d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:56.417 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:56.415 7fcbf959d700 1 -- 172.21.2.102:0/2940429580 >> 172.21.2.102:0/2940429580 conn(0x7fcbf40c4120 msgr2=0x7fcbf40ffd60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:56.417 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:56.415 7fcbf959d700 1 -- 172.21.2.102:0/2940429580 shutdown_connections 2021-02-10T11:30:56.417 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:56.415 7fcbf959d700 1 -- 172.21.2.102:0/2940429580 wait complete. 2021-02-10T11:30:56.424 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool create cephfs_data 8 2021-02-10T11:30:56.858 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:56.840 7f5960a7a700 1 Processor -- start 2021-02-10T11:30:56.858 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:56.840 7f5960a7a700 1 -- start start 2021-02-10T11:30:56.858 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:56.841 7f5960a7a700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f595c0d1c70 0x7f595c0cef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:56.858 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:56.841 7f5960a7a700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f595c0cf580 0x7f595c0cf980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:56.858 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:56.841 7f5960a7a700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f595c0fe1c0 con 0x7f595c0d1870 2021-02-10T11:30:56.859 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:56.841 7f5960a7a700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f595c1b0cb0 con 0x7f595c0cf580 2021-02-10T11:30:56.859 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:56.841 7f5960a7a700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f595c109a60 con 0x7f595c0d1c70 2021-02-10T11:30:56.859 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:56.841 7f595b05f700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f595c0cf580 0x7f595c0cf980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:56.859 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:56.841 7f595b05f700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f595c0cf580 0x7f595c0cf980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:33228/0 (socket says 172.21.2.102:33228) 2021-02-10T11:30:56.859 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:56.841 7f595b05f700 1 -- 172.21.2.102:0/3490241958 learned_addr learned my addr 172.21.2.102:0/3490241958 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:30:56.859 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:56.842 7f595b05f700 1 -- 172.21.2.102:0/3490241958 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f595c0d1c70 msgr2=0x7f595c0cef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:56.859 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:56.842 7f595b05f700 1 --2- 172.21.2.102:0/3490241958 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f595c0d1c70 0x7f595c0cef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:56.859 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:56.842 7f595b05f700 1 -- 172.21.2.102:0/3490241958 >> v1:172.21.2.113:6789/0 conn(0x7f595c0d1870 legacy=0x7f595c1b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:56.859 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:56.842 7f595b05f700 1 -- 172.21.2.102:0/3490241958 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f595c088070 con 0x7f595c0cf580 2021-02-10T11:30:56.859 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:56.842 7f595b05f700 1 --2- 172.21.2.102:0/3490241958 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f595c0cf580 0x7f595c0cf980 secure :-1 s=READY pgs=2061 cs=0 l=1 rev1=1 rx=0x7f5954006f00 tx=0x7f5954009f10).ready entity=mon.0 client_cookie=8a981c61e680b409 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:56.859 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:56.842 7f595985c700 1 -- 172.21.2.102:0/3490241958 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f595400b3b0 con 0x7f595c0cf580 2021-02-10T11:30:56.860 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:56.842 7f595985c700 1 -- 172.21.2.102:0/3490241958 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f595400b730 con 0x7f595c0cf580 2021-02-10T11:30:56.860 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:56.842 7f595985c700 1 -- 172.21.2.102:0/3490241958 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f595400bb90 con 0x7f595c0cf580 2021-02-10T11:30:56.860 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:56.842 7f5960a7a700 1 -- 172.21.2.102:0/3490241958 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f595c0cf580 msgr2=0x7f595c0cf980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:56.860 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:56.842 7f5960a7a700 1 --2- 172.21.2.102:0/3490241958 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f595c0cf580 0x7f595c0cf980 secure :-1 s=READY pgs=2061 cs=0 l=1 rev1=1 rx=0x7f5954006f00 tx=0x7f5954009f10).stop 2021-02-10T11:30:56.860 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:56.843 7f5960a7a700 1 -- 172.21.2.102:0/3490241958 shutdown_connections 2021-02-10T11:30:56.860 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:56.843 7f5960a7a700 1 --2- 172.21.2.102:0/3490241958 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f595c0d1c70 0x7f595c0cef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:56.860 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:56.843 7f5960a7a700 1 --2- 172.21.2.102:0/3490241958 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f595c0cf580 0x7f595c0cf980 unknown :-1 s=CLOSED pgs=2061 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:56.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:56.843 7f5960a7a700 1 -- 172.21.2.102:0/3490241958 >> 172.21.2.102:0/3490241958 conn(0x7f595c0ce420 msgr2=0x7f595c0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:56.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:56.843 7f5960a7a700 1 -- 172.21.2.102:0/3490241958 shutdown_connections 2021-02-10T11:30:56.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:56.843 7f5960a7a700 1 -- 172.21.2.102:0/3490241958 wait complete. 2021-02-10T11:30:56.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:56.857 7f5960a7a700 1 Processor -- start 2021-02-10T11:30:56.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:56.857 7f5960a7a700 1 -- start start 2021-02-10T11:30:56.864 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:56.858 7f5960a7a700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f595c0c3820 0x7f595c084cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:56.864 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:56.858 7f5960a7a700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f595c0a0290 0x7f595c0852d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:56.864 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:56.858 7f5960a7a700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f595c086270 0x7f595c086670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:56.865 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:56.858 7f5960a7a700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f595c0c0700 con 0x7f595c0c3820 2021-02-10T11:30:56.865 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:56.858 7f5960a7a700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f595c0858f0 con 0x7f595c0a0290 2021-02-10T11:30:56.865 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:56.858 7f5960a7a700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f595c085b10 con 0x7f595c086270 2021-02-10T11:30:56.865 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:56.858 7f595a85e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f595c0c3820 0x7f595c084cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:56.865 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:56.858 7f595a85e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f595c0c3820 0x7f595c084cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:33232/0 (socket says 172.21.2.102:33232) 2021-02-10T11:30:56.865 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:56.858 7f595a85e700 1 -- 172.21.2.102:0/1936074815 learned_addr learned my addr 172.21.2.102:0/1936074815 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:30:56.865 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:56.858 7f595a85e700 1 -- 172.21.2.102:0/1936074815 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f595c086270 msgr2=0x7f595c086670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:56.866 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:56.858 7f595a85e700 1 --2- 172.21.2.102:0/1936074815 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f595c086270 0x7f595c086670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:56.866 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:56.858 7f595a85e700 1 -- 172.21.2.102:0/1936074815 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f595c0a0290 msgr2=0x7f595c0852d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:56.866 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:56.858 7f595a85e700 1 --2- 172.21.2.102:0/1936074815 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f595c0a0290 0x7f595c0852d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:56.866 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:56.858 7f595a85e700 1 -- 172.21.2.102:0/1936074815 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f595c086c90 con 0x7f595c0c3820 2021-02-10T11:30:56.866 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:56.859 7f595a85e700 1 --2- 172.21.2.102:0/1936074815 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f595c0c3820 0x7f595c084cb0 secure :-1 s=READY pgs=2062 cs=0 l=1 rev1=1 rx=0x7f594c00b5b0 tx=0x7f594c00b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:56.866 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:56.859 7f594b7fe700 1 -- 172.21.2.102:0/1936074815 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f594c00bd40 con 0x7f595c0c3820 2021-02-10T11:30:56.866 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:56.859 7f594b7fe700 1 -- 172.21.2.102:0/1936074815 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f594c006240 con 0x7f595c0c3820 2021-02-10T11:30:56.866 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:56.859 7f594b7fe700 1 -- 172.21.2.102:0/1936074815 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f594c005720 con 0x7f595c0c3820 2021-02-10T11:30:56.866 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:56.859 7f5960a7a700 1 -- 172.21.2.102:0/1936074815 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f595c086c90 con 0x7f595c0c3820 2021-02-10T11:30:56.866 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:56.859 7f5960a7a700 1 -- 172.21.2.102:0/1936074815 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f595c086c90 con 0x7f595c0c3820 2021-02-10T11:30:56.866 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:56.860 7f594b7fe700 1 -- 172.21.2.102:0/1936074815 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f594c0054a0 con 0x7f595c0c3820 2021-02-10T11:30:56.866 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:56.860 7f594b7fe700 1 --2- 172.21.2.102:0/1936074815 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f59400186c0 0x7f594001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:56.866 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:56.860 7f594b7fe700 1 -- 172.21.2.102:0/1936074815 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5107+0+0 (secure 0 0 0) 0x7f594c0323d0 con 0x7f595c0c3820 2021-02-10T11:30:56.867 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:56.860 7f595a05d700 1 --2- 172.21.2.102:0/1936074815 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f59400186c0 0x7f594001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:56.867 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:56.860 7f595a05d700 1 --2- 172.21.2.102:0/1936074815 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f59400186c0 0x7f594001abc0 crc :-1 s=READY pgs=915 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:56.892 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:56.891 7f5960a7a700 1 -- 172.21.2.102:0/1936074815 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f595c082a70 con 0x7f595c0c3820 2021-02-10T11:30:56.895 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:56.894 7f594b7fe700 1 -- 172.21.2.102:0/1936074815 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f594c006030 con 0x7f595c0c3820 2021-02-10T11:30:57.025 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:57.024 7f5960a7a700 1 -- 172.21.2.102:0/1936074815 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_data"} v 0) v1 -- 0x7f595c082a70 con 0x7f595c0c3820 2021-02-10T11:30:57.422 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:57.421 7f594b7fe700 1 -- 172.21.2.102:0/1936074815 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_data"}]=0 pool 'cephfs_data' created v100) v1 ==== 125+0+0 (secure 0 0 0) 0x7f594c0054a0 con 0x7f595c0c3820 2021-02-10T11:30:57.440 INFO:teuthology.orchestra.run.gibba002.stderr:pool 'cephfs_data' created 2021-02-10T11:30:57.442 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:57.441 7f5960a7a700 1 -- 172.21.2.102:0/1936074815 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f59400186c0 msgr2=0x7f594001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:57.442 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:57.441 7f5960a7a700 1 --2- 172.21.2.102:0/1936074815 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f59400186c0 0x7f594001abc0 crc :-1 s=READY pgs=915 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:57.443 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:57.441 7f5960a7a700 1 -- 172.21.2.102:0/1936074815 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f595c0c3820 msgr2=0x7f595c084cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:57.443 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:57.441 7f5960a7a700 1 --2- 172.21.2.102:0/1936074815 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f595c0c3820 0x7f595c084cb0 secure :-1 s=READY pgs=2062 cs=0 l=1 rev1=1 rx=0x7f594c00b5b0 tx=0x7f594c00b950).stop 2021-02-10T11:30:57.443 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:57.441 7f5960a7a700 1 -- 172.21.2.102:0/1936074815 shutdown_connections 2021-02-10T11:30:57.443 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:57.441 7f5960a7a700 1 --2- 172.21.2.102:0/1936074815 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f595c086270 0x7f595c086670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:57.443 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:57.441 7f5960a7a700 1 --2- 172.21.2.102:0/1936074815 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f59400186c0 0x7f594001abc0 unknown :-1 s=CLOSED pgs=915 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:57.443 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:57.441 7f5960a7a700 1 --2- 172.21.2.102:0/1936074815 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f595c0a0290 0x7f595c0852d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:57.443 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:57.441 7f5960a7a700 1 --2- 172.21.2.102:0/1936074815 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f595c0c3820 0x7f595c084cb0 unknown :-1 s=CLOSED pgs=2062 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:57.443 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:57.441 7f5960a7a700 1 -- 172.21.2.102:0/1936074815 >> 172.21.2.102:0/1936074815 conn(0x7f595c0872c0 msgr2=0x7f595c095700 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:57.443 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:57.441 7f5960a7a700 1 -- 172.21.2.102:0/1936074815 shutdown_connections 2021-02-10T11:30:57.443 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:57.441 7f5960a7a700 1 -- 172.21.2.102:0/1936074815 wait complete. 2021-02-10T11:30:57.451 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2021-02-10T11:30:57.883 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:57.864 7fcb355b2700 1 Processor -- start 2021-02-10T11:30:57.884 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:57.864 7fcb355b2700 1 -- start start 2021-02-10T11:30:57.884 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:57.865 7fcb355b2700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fcb300d1c80 0x7fcb300cef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:57.884 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:57.865 7fcb355b2700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcb300cf590 0x7fcb300cf990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:57.884 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:57.865 7fcb355b2700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb300fe1d0 con 0x7fcb300d1880 2021-02-10T11:30:57.884 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:57.865 7fcb355b2700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb301b0cc0 con 0x7fcb300d1c80 2021-02-10T11:30:57.884 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:57.865 7fcb355b2700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb30109a70 con 0x7fcb300cf590 2021-02-10T11:30:57.884 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:57.866 7fcb2f2bf700 1 --1- >> v1:172.21.2.113:6789/0 conn(0x7fcb300d1880 0x7fcb301b04d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.113:6789/0 says I am v1:172.21.2.102:32790/0 (socket says 172.21.2.102:32790) 2021-02-10T11:30:57.884 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:57.866 7fcb2f2bf700 1 -- 172.21.2.102:0/1386623016 learned_addr learned my addr 172.21.2.102:0/1386623016 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:30:57.884 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:57.866 7fcb2fac0700 1 --2- 172.21.2.102:0/1386623016 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcb300cf590 0x7fcb300cf990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:57.884 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:57.866 7fcb2eabe700 1 --2- 172.21.2.102:0/1386623016 >> v2:172.21.2.113:3300/0 conn(0x7fcb300d1c80 0x7fcb300cef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:57.885 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:57.867 7fcb2eabe700 1 -- 172.21.2.102:0/1386623016 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcb300cf590 msgr2=0x7fcb300cf990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:57.885 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:57.867 7fcb2eabe700 1 --2- 172.21.2.102:0/1386623016 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcb300cf590 0x7fcb300cf990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:57.885 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:57.867 7fcb2eabe700 1 -- 172.21.2.102:0/1386623016 >> v1:172.21.2.113:6789/0 conn(0x7fcb300d1880 legacy=0x7fcb301b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:57.885 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:57.867 7fcb2eabe700 1 -- 172.21.2.102:0/1386623016 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb30088070 con 0x7fcb300d1c80 2021-02-10T11:30:57.885 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:57.867 7fcb2fac0700 1 --2- 172.21.2.102:0/1386623016 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcb300cf590 0x7fcb300cf990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:30:57.885 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:57.867 7fcb2eabe700 1 --2- 172.21.2.102:0/1386623016 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fcb300d1c80 0x7fcb300cef70 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 rx=0x7fcb24006650 tx=0x7fcb24009a80).ready entity=mon.1 client_cookie=4496da8c85725f11 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:57.885 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:57.868 7fcb2e2bd700 1 -- 172.21.2.102:0/1386623016 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcb240009b0 con 0x7fcb300d1c80 2021-02-10T11:30:57.885 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:57.868 7fcb2e2bd700 1 -- 172.21.2.102:0/1386623016 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb2400c0c0 con 0x7fcb300d1c80 2021-02-10T11:30:57.885 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:57.868 7fcb2e2bd700 1 -- 172.21.2.102:0/1386623016 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcb2400c5e0 con 0x7fcb300d1c80 2021-02-10T11:30:57.885 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:57.868 7fcb355b2700 1 -- 172.21.2.102:0/1386623016 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fcb300d1c80 msgr2=0x7fcb300cef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:57.885 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:57.868 7fcb355b2700 1 --2- 172.21.2.102:0/1386623016 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fcb300d1c80 0x7fcb300cef70 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 rx=0x7fcb24006650 tx=0x7fcb24009a80).stop 2021-02-10T11:30:57.885 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:57.869 7fcb355b2700 1 -- 172.21.2.102:0/1386623016 shutdown_connections 2021-02-10T11:30:57.885 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:57.869 7fcb355b2700 1 --2- 172.21.2.102:0/1386623016 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcb300cf590 0x7fcb300cf990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:57.886 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:57.869 7fcb355b2700 1 --2- 172.21.2.102:0/1386623016 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fcb300d1c80 0x7fcb300cef70 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:57.886 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:57.869 7fcb355b2700 1 -- 172.21.2.102:0/1386623016 >> 172.21.2.102:0/1386623016 conn(0x7fcb300ce430 msgr2=0x7fcb300d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:57.886 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:57.869 7fcb355b2700 1 -- 172.21.2.102:0/1386623016 shutdown_connections 2021-02-10T11:30:57.886 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:57.869 7fcb355b2700 1 -- 172.21.2.102:0/1386623016 wait complete. 2021-02-10T11:30:57.887 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:57.883 7fcb355b2700 1 Processor -- start 2021-02-10T11:30:57.887 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:57.883 7fcb355b2700 1 -- start start 2021-02-10T11:30:57.888 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:57.884 7fcb355b2700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcb300aef00 0x7fcb30099c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:57.888 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:57.884 7fcb355b2700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fcb3009a260 0x7fcb3009a660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:57.888 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:57.884 7fcb355b2700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcb3009ac80 0x7fcb3019a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:57.888 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:57.884 7fcb355b2700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb3019bdd0 con 0x7fcb300aef00 2021-02-10T11:30:57.888 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:57.884 7fcb355b2700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb3019de70 con 0x7fcb3009a260 2021-02-10T11:30:57.888 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:57.884 7fcb355b2700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb3019a2d0 con 0x7fcb3009ac80 2021-02-10T11:30:57.888 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:57.884 7fcb2f2bf700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcb300aef00 0x7fcb30099c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:57.888 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:57.884 7fcb2f2bf700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcb300aef00 0x7fcb30099c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:33250/0 (socket says 172.21.2.102:33250) 2021-02-10T11:30:57.888 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:57.884 7fcb2f2bf700 1 -- 172.21.2.102:0/454569671 learned_addr learned my addr 172.21.2.102:0/454569671 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:30:57.888 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:57.884 7fcb2f2bf700 1 -- 172.21.2.102:0/454569671 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcb3009ac80 msgr2=0x7fcb3019a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:57.888 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:57.884 7fcb2f2bf700 1 --2- 172.21.2.102:0/454569671 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcb3009ac80 0x7fcb3019a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:57.888 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:57.884 7fcb2f2bf700 1 -- 172.21.2.102:0/454569671 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fcb3009a260 msgr2=0x7fcb3009a660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:57.888 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:57.884 7fcb2f2bf700 1 --2- 172.21.2.102:0/454569671 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fcb3009a260 0x7fcb3009a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:57.889 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:57.884 7fcb2f2bf700 1 -- 172.21.2.102:0/454569671 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb30199f50 con 0x7fcb300aef00 2021-02-10T11:30:57.889 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:57.884 7fcb2f2bf700 1 --2- 172.21.2.102:0/454569671 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcb300aef00 0x7fcb30099c40 secure :-1 s=READY pgs=2063 cs=0 l=1 rev1=1 rx=0x7fcb2000a580 tx=0x7fcb2000a920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:57.889 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:57.885 7fcb17fff700 1 -- 172.21.2.102:0/454569671 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcb2000ad10 con 0x7fcb300aef00 2021-02-10T11:30:57.889 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:57.885 7fcb17fff700 1 -- 172.21.2.102:0/454569671 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb20004320 con 0x7fcb300aef00 2021-02-10T11:30:57.889 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:57.885 7fcb17fff700 1 -- 172.21.2.102:0/454569671 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcb200047b0 con 0x7fcb300aef00 2021-02-10T11:30:57.889 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:57.885 7fcb355b2700 1 -- 172.21.2.102:0/454569671 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb30199f50 con 0x7fcb300aef00 2021-02-10T11:30:57.889 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:57.885 7fcb355b2700 1 -- 172.21.2.102:0/454569671 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb30199f50 con 0x7fcb300aef00 2021-02-10T11:30:57.889 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:57.885 7fcb17fff700 1 -- 172.21.2.102:0/454569671 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fcb20004500 con 0x7fcb300aef00 2021-02-10T11:30:57.889 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:57.885 7fcb17fff700 1 --2- 172.21.2.102:0/454569671 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcb100186c0 0x7fcb1001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:57.889 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:57.886 7fcb17fff700 1 -- 172.21.2.102:0/454569671 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5457+0+0 (secure 0 0 0) 0x7fcb200324f0 con 0x7fcb300aef00 2021-02-10T11:30:57.889 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:57.886 7fcb2eabe700 1 --2- 172.21.2.102:0/454569671 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcb100186c0 0x7fcb1001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:57.889 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:57.886 7fcb2eabe700 1 --2- 172.21.2.102:0/454569671 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcb100186c0 0x7fcb1001abc0 crc :-1 s=READY pgs=918 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:57.916 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:57.915 7fcb355b2700 1 -- 172.21.2.102:0/454569671 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb30082a70 con 0x7fcb300aef00 2021-02-10T11:30:57.918 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:57.917 7fcb17fff700 1 -- 172.21.2.102:0/454569671 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fcb20000eb0 con 0x7fcb300aef00 2021-02-10T11:30:58.046 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:58.045 7fcb355b2700 1 -- 172.21.2.102:0/454569671 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"data": "cephfs_data", "prefix": "fs new", "force": true, "fs_name": "cephfs", "metadata": "cephfs_metadata"} v 0) v1 -- 0x7fcb30199f50 con 0x7fcb300aef00 2021-02-10T11:30:58.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:58.049 7fcb17fff700 1 -- 172.21.2.102:0/454569671 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"data": "cephfs_data", "prefix": "fs new", "force": true, "fs_name": "cephfs", "metadata": "cephfs_metadata"}]=0 new fs with metadata pool 20 and data pool 21 v79) v1 ==== 189+0+0 (secure 0 0 0) 0x7fcb20004500 con 0x7fcb300aef00 2021-02-10T11:30:58.055 INFO:teuthology.orchestra.run.gibba002.stderr:new fs with metadata pool 20 and data pool 21 2021-02-10T11:30:58.056 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:58.055 7fcb355b2700 1 -- 172.21.2.102:0/454569671 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcb100186c0 msgr2=0x7fcb1001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:58.057 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:58.055 7fcb355b2700 1 --2- 172.21.2.102:0/454569671 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcb100186c0 0x7fcb1001abc0 crc :-1 s=READY pgs=918 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:58.057 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:58.055 7fcb355b2700 1 -- 172.21.2.102:0/454569671 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcb300aef00 msgr2=0x7fcb30099c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:58.057 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:58.055 7fcb355b2700 1 --2- 172.21.2.102:0/454569671 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcb300aef00 0x7fcb30099c40 secure :-1 s=READY pgs=2063 cs=0 l=1 rev1=1 rx=0x7fcb2000a580 tx=0x7fcb2000a920).stop 2021-02-10T11:30:58.057 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:58.055 7fcb355b2700 1 -- 172.21.2.102:0/454569671 shutdown_connections 2021-02-10T11:30:58.057 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:58.055 7fcb355b2700 1 --2- 172.21.2.102:0/454569671 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcb3009ac80 0x7fcb3019a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:58.057 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:58.055 7fcb355b2700 1 --2- 172.21.2.102:0/454569671 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcb100186c0 0x7fcb1001abc0 unknown :-1 s=CLOSED pgs=918 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:58.058 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:58.055 7fcb355b2700 1 --2- 172.21.2.102:0/454569671 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fcb3009a260 0x7fcb3009a660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:58.058 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:58.055 7fcb355b2700 1 --2- 172.21.2.102:0/454569671 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcb300aef00 0x7fcb30099c40 unknown :-1 s=CLOSED pgs=2063 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:58.058 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:58.055 7fcb355b2700 1 -- 172.21.2.102:0/454569671 >> 172.21.2.102:0/454569671 conn(0x7fcb3018e810 msgr2=0x7fcb3018c290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:58.058 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:58.055 7fcb355b2700 1 -- 172.21.2.102:0/454569671 shutdown_connections 2021-02-10T11:30:58.058 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:58.055 7fcb355b2700 1 -- 172.21.2.102:0/454569671 wait complete. 2021-02-10T11:30:58.065 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:30:58.493 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:58.475 7faf46903700 1 Processor -- start 2021-02-10T11:30:58.494 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:58.475 7faf46903700 1 -- start start 2021-02-10T11:30:58.494 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:58.476 7faf46903700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7faf401b0870 0x7faf400f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:58.494 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:58.476 7faf46903700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf401b0c70 con 0x7faf401b0470 2021-02-10T11:30:58.494 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:58.476 7faf46903700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf400f0b50 con 0x7faf400efe00 2021-02-10T11:30:58.494 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:58.476 7faf46903700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf400e5ea0 con 0x7faf401b0870 2021-02-10T11:30:58.494 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:58.476 7faf3ffff700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7faf401b0470 0x7faf400ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:53712/0 (socket says 172.21.2.102:53712) 2021-02-10T11:30:58.494 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:58.476 7faf3ffff700 1 -- 172.21.2.102:0/4154462708 learned_addr learned my addr 172.21.2.102:0/4154462708 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:30:58.494 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:58.476 7faf3effd700 1 -- 172.21.2.102:0/4154462708 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2448667317 0 0) 0x7faf30000ce0 con 0x7faf401b0470 2021-02-10T11:30:58.494 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:58.476 7faf3effd700 1 -- 172.21.2.102:0/4154462708 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf28002fe0 con 0x7faf401b0470 2021-02-10T11:30:58.494 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:58.476 7faf3effd700 1 -- 172.21.2.102:0/4154462708 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1725146022 0 0) 0x7faf300010e0 con 0x7faf401b0470 2021-02-10T11:30:58.494 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:58.476 7faf3effd700 1 -- 172.21.2.102:0/4154462708 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7faf401b0870 msgr2=0x7faf400f20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:58.495 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:58.476 7faf3effd700 1 --2- 172.21.2.102:0/4154462708 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7faf401b0870 0x7faf400f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:58.495 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:58.476 7faf3effd700 1 -- 172.21.2.102:0/4154462708 >> v1:172.21.2.113:6789/0 conn(0x7faf400efe00 legacy=0x7faf400f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:58.495 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:58.476 7faf3effd700 1 -- 172.21.2.102:0/4154462708 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf400e52e0 con 0x7faf401b0470 2021-02-10T11:30:58.495 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:58.476 7faf3effd700 1 -- 172.21.2.102:0/4154462708 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7faf30001660 con 0x7faf401b0470 2021-02-10T11:30:58.495 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:58.476 7faf3effd700 1 -- 172.21.2.102:0/4154462708 >> v1:172.21.2.102:6789/0 conn(0x7faf401b0470 legacy=0x7faf400ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:58.495 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:58.476 7faf3effd700 1 --2- 172.21.2.102:0/4154462708 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7faf28003b10 0x7faf28005f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:58.495 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:58.476 7faf3effd700 1 --2- 172.21.2.102:0/4154462708 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7faf280067a0 0x7faf28008c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:58.495 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:58.476 7faf3effd700 1 --2- 172.21.2.102:0/4154462708 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7faf280094f0 0x7faf2800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:58.495 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:58.476 7faf3effd700 1 -- 172.21.2.102:0/4154462708 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf2800c2c0 con 0x7faf280094f0 2021-02-10T11:30:58.495 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:58.476 7faf3effd700 1 -- 172.21.2.102:0/4154462708 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf2800c4e0 con 0x7faf28003b10 2021-02-10T11:30:58.495 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:58.476 7faf3effd700 1 -- 172.21.2.102:0/4154462708 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf2800c700 con 0x7faf280067a0 2021-02-10T11:30:58.495 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:58.477 7faf44f20700 1 --2- 172.21.2.102:0/4154462708 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7faf280094f0 0x7faf2800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:58.495 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:58.477 7faf3ffff700 1 --2- 172.21.2.102:0/4154462708 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7faf28003b10 0x7faf28005f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:58.496 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:58.477 7faf3f7fe700 1 --2- 172.21.2.102:0/4154462708 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7faf280067a0 0x7faf28008c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:58.496 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:58.477 7faf44f20700 1 -- 172.21.2.102:0/4154462708 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7faf280067a0 msgr2=0x7faf28008c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:58.496 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:58.477 7faf44f20700 1 --2- 172.21.2.102:0/4154462708 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7faf280067a0 0x7faf28008c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:58.496 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:58.477 7faf44f20700 1 -- 172.21.2.102:0/4154462708 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7faf28003b10 msgr2=0x7faf28005f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:58.496 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:58.477 7faf44f20700 1 --2- 172.21.2.102:0/4154462708 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7faf28003b10 0x7faf28005f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:58.496 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:58.477 7faf44f20700 1 -- 172.21.2.102:0/4154462708 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf2800c9a0 con 0x7faf280094f0 2021-02-10T11:30:58.496 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:58.477 7faf44f20700 1 --2- 172.21.2.102:0/4154462708 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7faf280094f0 0x7faf2800ba10 secure :-1 s=READY pgs=2065 cs=0 l=1 rev1=1 rx=0x7faf38006e30 tx=0x7faf3804fed0).ready entity=mon.0 client_cookie=3b1e9fd942a74150 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:58.496 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:58.477 7faf3effd700 1 -- 172.21.2.102:0/4154462708 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faf38006a70 con 0x7faf280094f0 2021-02-10T11:30:58.496 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:58.477 7faf3effd700 1 -- 172.21.2.102:0/4154462708 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf380093c0 con 0x7faf280094f0 2021-02-10T11:30:58.496 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:58.477 7faf3effd700 1 -- 172.21.2.102:0/4154462708 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faf38009850 con 0x7faf280094f0 2021-02-10T11:30:58.496 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:58.477 7faf46903700 1 -- 172.21.2.102:0/4154462708 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7faf280094f0 msgr2=0x7faf2800ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:58.496 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:58.477 7faf46903700 1 --2- 172.21.2.102:0/4154462708 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7faf280094f0 0x7faf2800ba10 secure :-1 s=READY pgs=2065 cs=0 l=1 rev1=1 rx=0x7faf38006e30 tx=0x7faf3804fed0).stop 2021-02-10T11:30:58.496 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:58.477 7faf44f20700 1 -- 172.21.2.102:0/4154462708 reap_dead start 2021-02-10T11:30:58.497 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:58.478 7faf46903700 1 -- 172.21.2.102:0/4154462708 shutdown_connections 2021-02-10T11:30:58.497 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:58.478 7faf46903700 1 -- 172.21.2.102:0/4154462708 >> 172.21.2.102:0/4154462708 conn(0x7faf400ce3e0 msgr2=0x7faf400e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:58.497 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:58.478 7faf46903700 1 -- 172.21.2.102:0/4154462708 shutdown_connections 2021-02-10T11:30:58.497 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:58.478 7faf46903700 1 -- 172.21.2.102:0/4154462708 wait complete. 2021-02-10T11:30:58.497 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:58.493 7faf46903700 1 Processor -- start 2021-02-10T11:30:58.497 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:58.493 7faf46903700 1 -- start start 2021-02-10T11:30:58.498 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:58.493 7faf46903700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7faf40108ce0 0x7faf4009da50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:58.498 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:58.493 7faf46903700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7faf401090e0 0x7faf4009e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:58.499 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:58.493 7faf46903700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7faf4009eee0 0x7faf4009f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:58.499 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:58.493 7faf46903700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf401094e0 con 0x7faf40108ce0 2021-02-10T11:30:58.499 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:58.493 7faf46903700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf4009e960 con 0x7faf4009eee0 2021-02-10T11:30:58.499 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:58.493 7faf46903700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf4009e690 con 0x7faf401090e0 2021-02-10T11:30:58.499 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:58.493 7faf3ffff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7faf40108ce0 0x7faf4009da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:58.499 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:58.493 7faf3ffff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7faf40108ce0 0x7faf4009da50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:33270/0 (socket says 172.21.2.102:33270) 2021-02-10T11:30:58.499 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:58.493 7faf3ffff700 1 -- 172.21.2.102:0/4157324539 learned_addr learned my addr 172.21.2.102:0/4157324539 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:30:58.499 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:58.494 7faf3ffff700 1 -- 172.21.2.102:0/4157324539 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7faf401090e0 msgr2=0x7faf4009e070 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:30:58.499 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:58.494 7faf3ffff700 1 --2- 172.21.2.102:0/4157324539 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7faf401090e0 0x7faf4009e070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:58.499 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:58.494 7faf3ffff700 1 -- 172.21.2.102:0/4157324539 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7faf4009eee0 msgr2=0x7faf4009f2e0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:30:58.499 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:58.494 7faf3ffff700 1 --2- 172.21.2.102:0/4157324539 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7faf4009eee0 0x7faf4009f2e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:58.499 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:58.494 7faf3ffff700 1 -- 172.21.2.102:0/4157324539 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf4009f900 con 0x7faf40108ce0 2021-02-10T11:30:58.499 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:58.494 7faf3ffff700 1 --2- 172.21.2.102:0/4157324539 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7faf40108ce0 0x7faf4009da50 secure :-1 s=READY pgs=2066 cs=0 l=1 rev1=1 rx=0x7faf30006f00 tx=0x7faf3000b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:58.500 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:58.494 7faf3cff9700 1 -- 172.21.2.102:0/4157324539 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faf30006020 con 0x7faf40108ce0 2021-02-10T11:30:58.500 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:58.494 7faf3cff9700 1 -- 172.21.2.102:0/4157324539 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf300063a0 con 0x7faf40108ce0 2021-02-10T11:30:58.500 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:58.494 7faf3cff9700 1 -- 172.21.2.102:0/4157324539 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faf30004590 con 0x7faf40108ce0 2021-02-10T11:30:58.500 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:58.494 7faf46903700 1 -- 172.21.2.102:0/4157324539 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf4009f900 con 0x7faf40108ce0 2021-02-10T11:30:58.500 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:58.494 7faf46903700 1 -- 172.21.2.102:0/4157324539 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf4009fe90 con 0x7faf40108ce0 2021-02-10T11:30:58.500 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:58.495 7faf3cff9700 1 -- 172.21.2.102:0/4157324539 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7faf30009fb0 con 0x7faf40108ce0 2021-02-10T11:30:58.500 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:58.495 7faf3cff9700 1 --2- 172.21.2.102:0/4157324539 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7faf200186c0 0x7faf2001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:58.500 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:58.495 7faf3cff9700 1 -- 172.21.2.102:0/4157324539 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 5573+0+0 (secure 0 0 0) 0x7faf30032570 con 0x7faf40108ce0 2021-02-10T11:30:58.500 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:58.495 7faf3f7fe700 1 --2- 172.21.2.102:0/4157324539 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7faf200186c0 0x7faf2001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:58.500 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:58.495 7faf3f7fe700 1 --2- 172.21.2.102:0/4157324539 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7faf200186c0 0x7faf2001abc0 crc :-1 s=READY pgs=919 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:58.526 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:58.525 7faf46903700 1 -- 172.21.2.102:0/4157324539 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf4009e960 con 0x7faf40108ce0 2021-02-10T11:30:58.529 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:58.528 7faf3cff9700 1 -- 172.21.2.102:0/4157324539 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7faf3000bd30 con 0x7faf40108ce0 2021-02-10T11:30:58.653 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:58.652 7faf46903700 1 -- 172.21.2.102:0/4157324539 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7faf4009e960 con 0x7faf40108ce0 2021-02-10T11:30:58.654 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:58.653 7faf3cff9700 1 -- 172.21.2.102:0/4157324539 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v101) v1 ==== 74+0+14016 (secure 0 0 0) 0x7faf30009fb0 con 0x7faf40108ce0 2021-02-10T11:30:58.654 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:30:58.655 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":101,"fsid":"68db30ea-dbce-4aef-a6df-f01c6fea0513","created":"2021-02-10 11:03:06.331557","modified":"2021-02-10 11:30:58.046970","last_up_change":"2021-02-10 11:03:32.690562","last_in_change":"2021-02-10 11:03:32.690562","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:03:37.322519","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:30:56.015905","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"101","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":21,"pool_name":"cephfs_data","create_time":"2021-02-10 11:30:57.042942","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"101","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"be691b4d-a039-408a-af80-9d53f02b6f1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6818","nonce":36088},{"type":"v1","addr":"172.21.2.102:6820","nonce":36088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6822","nonce":36088},{"type":"v1","addr":"172.21.2.102:6824","nonce":36088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6830","nonce":36088},{"type":"v1","addr":"172.21.2.102:6832","nonce":36088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6826","nonce":36088},{"type":"v1","addr":"172.21.2.102:6828","nonce":36088}]},"public_addr":"172.21.2.102:6820/36088","cluster_addr":"172.21.2.102:6824/36088","heartbeat_back_addr":"172.21.2.102:6832/36088","heartbeat_front_addr":"172.21.2.102:6828/36088","state":["exists","up"]},{"osd":1,"uuid":"64e3ccc8-9c69-4d7c-865e-af8e83822635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6819","nonce":36087},{"type":"v1","addr":"172.21.2.102:6821","nonce":36087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6823","nonce":36087},{"type":"v1","addr":"172.21.2.102:6825","nonce":36087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6831","nonce":36087},{"type":"v1","addr":"172.21.2.102:6833","nonce":36087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6827","nonce":36087},{"type":"v1","addr":"172.21.2.102:6829","nonce":36087}]},"public_addr":"172.21.2.102:6821/36087","cluster_addr":"172.21.2.102:6825/36087","heartbeat_back_addr":"172.21.2.102:6833/36087","heartbeat_front_addr":"172.21.2.102:6829/36087","state":["exists","up"]},{"osd":2,"uuid":"153700af-747f-481f-bdf4-d6072d3a842f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6810","nonce":36089},{"type":"v1","addr":"172.21.2.102:6811","nonce":36089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6812","nonce":36089},{"type":"v1","addr":"172.21.2.102:6813","nonce":36089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6816","nonce":36089},{"type":"v1","addr":"172.21.2.102:6817","nonce":36089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6814","nonce":36089},{"type":"v1","addr":"172.21.2.102:6815","nonce":36089}]},"public_addr":"172.21.2.102:6811/36089","cluster_addr":"172.21.2.102:6813/36089","heartbeat_back_addr":"172.21.2.102:6817/36089","heartbeat_front_addr":"172.21.2.102:6815/36089","state":["exists","up"]},{"osd":3,"uuid":"cbc678f4-83a9-45e6-8830-2f7c232b1ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6802","nonce":36090},{"type":"v1","addr":"172.21.2.102:6803","nonce":36090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6804","nonce":36090},{"type":"v1","addr":"172.21.2.102:6805","nonce":36090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6808","nonce":36090},{"type":"v1","addr":"172.21.2.102:6809","nonce":36090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6806","nonce":36090},{"type":"v1","addr":"172.21.2.102:6807","nonce":36090}]},"public_addr":"172.21.2.102:6803/36090","cluster_addr":"172.21.2.102:6805/36090","heartbeat_back_addr":"172.21.2.102:6809/36090","heartbeat_front_addr":"172.21.2.102:6807/36090","state":["exists","up"]},{"osd":4,"uuid":"c01e4008-b258-4f10-afd2-9e7f37214351","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6808","nonce":36368},{"type":"v1","addr":"172.21.2.113:6811","nonce":36368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6814","nonce":36368},{"type":"v1","addr":"172.21.2.113:6818","nonce":36368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6826","nonce":36368},{"type":"v1","addr":"172.21.2.113:6828","nonce":36368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6820","nonce":36368},{"type":"v1","addr":"172.21.2.113:6824","nonce":36368}]},"public_addr":"172.21.2.113:6811/36368","cluster_addr":"172.21.2.113:6818/36368","heartbeat_back_addr":"172.21.2.113:6828/36368","heartbeat_front_addr":"172.21.2.113:6824/36368","state":["exists","up"]},{"osd":5,"uuid":"551f6007-daf8-4a98-8ee0-c2b16bcccb5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6802","nonce":36366},{"type":"v1","addr":"172.21.2.113:6805","nonce":36366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6807","nonce":36366},{"type":"v1","addr":"172.21.2.113:6810","nonce":36366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6819","nonce":36366},{"type":"v1","addr":"172.21.2.113:6822","nonce":36366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6812","nonce":36366},{"type":"v1","addr":"172.21.2.113:6815","nonce":36366}]},"public_addr":"172.21.2.113:6805/36366","cluster_addr":"172.21.2.113:6810/36366","heartbeat_back_addr":"172.21.2.113:6822/36366","heartbeat_front_addr":"172.21.2.113:6815/36366","state":["exists","up"]},{"osd":6,"uuid":"bf67c040-0e50-48be-b102-c3943bc9ebef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6800","nonce":36367},{"type":"v1","addr":"172.21.2.113:6801","nonce":36367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6803","nonce":36367},{"type":"v1","addr":"172.21.2.113:6804","nonce":36367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6813","nonce":36367},{"type":"v1","addr":"172.21.2.113:6816","nonce":36367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6806","nonce":36367},{"type":"v1","addr":"172.21.2.113:6809","nonce":36367}]},"public_addr":"172.21.2.113:6801/36367","cluster_addr":"172.21.2.113:6804/36367","heartbeat_back_addr":"172.21.2.113:6816/36367","heartbeat_front_addr":"172.21.2.113:6809/36367","state":["exists","up"]},{"osd":7,"uuid":"f5461f80-018d-428f-be89-2eb2bcbe7bb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6817","nonce":36365},{"type":"v1","addr":"172.21.2.113:6821","nonce":36365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6823","nonce":36365},{"type":"v1","addr":"172.21.2.113:6825","nonce":36365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6830","nonce":36365},{"type":"v1","addr":"172.21.2.113:6831","nonce":36365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6827","nonce":36365},{"type":"v1","addr":"172.21.2.113:6829","nonce":36365}]},"public_addr":"172.21.2.113:6821/36365","cluster_addr":"172.21.2.113:6825/36365","heartbeat_back_addr":"172.21.2.113:6831/36365","heartbeat_front_addr":"172.21.2.113:6829/36365","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:30:58.655 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:58.655 7faf46903700 1 -- 172.21.2.102:0/4157324539 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7faf200186c0 msgr2=0x7faf2001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:58.656 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:58.655 7faf46903700 1 --2- 172.21.2.102:0/4157324539 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7faf200186c0 0x7faf2001abc0 crc :-1 s=READY pgs=919 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:58.656 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:58.655 7faf46903700 1 -- 172.21.2.102:0/4157324539 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7faf40108ce0 msgr2=0x7faf4009da50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:58.656 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:58.655 7faf46903700 1 --2- 172.21.2.102:0/4157324539 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7faf40108ce0 0x7faf4009da50 secure :-1 s=READY pgs=2066 cs=0 l=1 rev1=1 rx=0x7faf30006f00 tx=0x7faf3000b940).stop 2021-02-10T11:30:58.656 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:58.655 7faf46903700 1 -- 172.21.2.102:0/4157324539 shutdown_connections 2021-02-10T11:30:58.656 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:58.655 7faf46903700 1 --2- 172.21.2.102:0/4157324539 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7faf200186c0 0x7faf2001abc0 unknown :-1 s=CLOSED pgs=919 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:58.656 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:58.655 7faf46903700 1 --2- 172.21.2.102:0/4157324539 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7faf4009eee0 0x7faf4009f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:58.656 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:58.655 7faf46903700 1 --2- 172.21.2.102:0/4157324539 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7faf401090e0 0x7faf4009e070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:58.656 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:58.655 7faf46903700 1 --2- 172.21.2.102:0/4157324539 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7faf40108ce0 0x7faf4009da50 unknown :-1 s=CLOSED pgs=2066 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:58.656 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:58.655 7faf46903700 1 -- 172.21.2.102:0/4157324539 >> 172.21.2.102:0/4157324539 conn(0x7faf400bc6c0 msgr2=0x7faf400b1f70 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:58.657 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:58.655 7faf46903700 1 -- 172.21.2.102:0/4157324539 shutdown_connections 2021-02-10T11:30:58.657 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:58.655 7faf46903700 1 -- 172.21.2.102:0/4157324539 wait complete. 2021-02-10T11:30:58.665 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:30:59.090 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:59.073 7f6c432a8700 1 Processor -- start 2021-02-10T11:30:59.090 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:59.073 7f6c432a8700 1 -- start start 2021-02-10T11:30:59.090 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:59.073 7f6c432a8700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f6c3c0d1870 0x7f6c3c1b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:59.091 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:59.073 7f6c432a8700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f6c3c0f0650 0x7f6c3c0cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:59.091 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:59.073 7f6c432a8700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c3c0f0b10 con 0x7f6c3c1b0a10 2021-02-10T11:30:59.091 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:59.073 7f6c432a8700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c3c0e5ed0 con 0x7f6c3c0f0650 2021-02-10T11:30:59.091 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:59.073 7f6c432a8700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c3c0d1c70 con 0x7f6c3c0d1870 2021-02-10T11:30:59.091 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:59.074 7f6c418c5700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f6c3c0f0650 0x7f6c3c0cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:59.091 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:59.074 7f6c410c4700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f6c3c0d1870 0x7f6c3c1b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:59.091 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:59.074 7f6c408c3700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f6c3c1b0a10 0x7f6c3c0cf330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:53736/0 (socket says 172.21.2.102:53736) 2021-02-10T11:30:59.091 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:59.074 7f6c408c3700 1 -- 172.21.2.102:0/3867753713 learned_addr learned my addr 172.21.2.102:0/3867753713 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:30:59.091 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:59.074 7f6c418c5700 1 -- 172.21.2.102:0/3867753713 >> v2:172.21.2.113:3300/0 conn(0x7f6c3c0d1870 msgr2=0x7f6c3c1b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:59.091 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:59.074 7f6c418c5700 1 --2- 172.21.2.102:0/3867753713 >> v2:172.21.2.113:3300/0 conn(0x7f6c3c0d1870 0x7f6c3c1b03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:59.091 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:59.074 7f6c418c5700 1 -- 172.21.2.102:0/3867753713 >> v1:172.21.2.102:6789/0 conn(0x7f6c3c1b0a10 legacy=0x7f6c3c0cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:59.091 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:59.074 7f6c418c5700 1 -- 172.21.2.102:0/3867753713 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c3c0d1e90 con 0x7f6c3c0f0650 2021-02-10T11:30:59.092 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:59.074 7f6c410c4700 1 --2- 172.21.2.102:0/3867753713 >> v2:172.21.2.113:3300/0 conn(0x7f6c3c0d1870 0x7f6c3c1b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:30:59.092 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:59.074 7f6c418c5700 1 --2- 172.21.2.102:0/3867753713 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6c3c0f0650 0x7f6c3c0cfb20 secure :-1 s=READY pgs=2068 cs=0 l=1 rev1=1 rx=0x7f6c34006e30 tx=0x7f6c34009e60).ready entity=mon.0 client_cookie=67bce187a4033a52 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:59.092 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:59.074 7f6c33fff700 1 -- 172.21.2.102:0/3867753713 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6c34006a50 con 0x7f6c3c0f0650 2021-02-10T11:30:59.092 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:59.074 7f6c33fff700 1 -- 172.21.2.102:0/3867753713 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c3400b300 con 0x7f6c3c0f0650 2021-02-10T11:30:59.092 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:59.074 7f6c33fff700 1 -- 172.21.2.102:0/3867753713 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6c3400b820 con 0x7f6c3c0f0650 2021-02-10T11:30:59.092 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:59.074 7f6c432a8700 1 -- 172.21.2.102:0/3867753713 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6c3c0f0650 msgr2=0x7f6c3c0cfb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:59.092 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:59.074 7f6c432a8700 1 --2- 172.21.2.102:0/3867753713 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6c3c0f0650 0x7f6c3c0cfb20 secure :-1 s=READY pgs=2068 cs=0 l=1 rev1=1 rx=0x7f6c34006e30 tx=0x7f6c34009e60).stop 2021-02-10T11:30:59.092 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:59.075 7f6c432a8700 1 -- 172.21.2.102:0/3867753713 shutdown_connections 2021-02-10T11:30:59.092 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:59.075 7f6c432a8700 1 --2- 172.21.2.102:0/3867753713 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6c3c0f0650 0x7f6c3c0cfb20 unknown :-1 s=CLOSED pgs=2068 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:59.092 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:59.075 7f6c432a8700 1 --2- 172.21.2.102:0/3867753713 >> v2:172.21.2.113:3300/0 conn(0x7f6c3c0d1870 0x7f6c3c1b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:59.092 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:59.075 7f6c432a8700 1 -- 172.21.2.102:0/3867753713 >> 172.21.2.102:0/3867753713 conn(0x7f6c3c0ce450 msgr2=0x7f6c3c0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:59.092 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:59.075 7f6c432a8700 1 -- 172.21.2.102:0/3867753713 shutdown_connections 2021-02-10T11:30:59.092 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:59.075 7f6c432a8700 1 -- 172.21.2.102:0/3867753713 wait complete. 2021-02-10T11:30:59.094 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:59.090 7f6c432a8700 1 Processor -- start 2021-02-10T11:30:59.094 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:59.090 7f6c432a8700 1 -- start start 2021-02-10T11:30:59.094 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:59.090 7f6c432a8700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6c3c1942b0 0x7f6c3c0b3550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:59.094 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:59.090 7f6c432a8700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6c3c195070 0x7f6c3c195470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:59.094 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:59.090 7f6c432a8700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6c3c0bb8e0 0x7f6c3c0bbce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:59.095 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:59.090 7f6c432a8700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c3c1946b0 con 0x7f6c3c0bb8e0 2021-02-10T11:30:59.095 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:59.090 7f6c432a8700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c3c0b3b70 con 0x7f6c3c195070 2021-02-10T11:30:59.095 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:59.090 7f6c432a8700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c3c0b3d90 con 0x7f6c3c1942b0 2021-02-10T11:30:59.095 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:59.090 7f6c418c5700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6c3c0bb8e0 0x7f6c3c0bbce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:59.095 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:59.090 7f6c418c5700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6c3c0bb8e0 0x7f6c3c0bbce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:33282/0 (socket says 172.21.2.102:33282) 2021-02-10T11:30:59.095 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:59.090 7f6c418c5700 1 -- 172.21.2.102:0/2562299796 learned_addr learned my addr 172.21.2.102:0/2562299796 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:30:59.095 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:59.090 7f6c408c3700 1 --2- 172.21.2.102:0/2562299796 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6c3c195070 0x7f6c3c195470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:59.095 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:59.091 7f6c410c4700 1 --2- 172.21.2.102:0/2562299796 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6c3c1942b0 0x7f6c3c0b3550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:59.095 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:59.091 7f6c418c5700 1 -- 172.21.2.102:0/2562299796 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6c3c1942b0 msgr2=0x7f6c3c0b3550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:59.095 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:59.091 7f6c418c5700 1 --2- 172.21.2.102:0/2562299796 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6c3c1942b0 0x7f6c3c0b3550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:59.095 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:59.091 7f6c418c5700 1 -- 172.21.2.102:0/2562299796 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6c3c195070 msgr2=0x7f6c3c195470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:59.095 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:59.091 7f6c418c5700 1 --2- 172.21.2.102:0/2562299796 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6c3c195070 0x7f6c3c195470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:59.095 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:59.091 7f6c418c5700 1 -- 172.21.2.102:0/2562299796 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c3c1a49e0 con 0x7f6c3c0bb8e0 2021-02-10T11:30:59.096 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:59.091 7f6c408c3700 1 --2- 172.21.2.102:0/2562299796 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6c3c195070 0x7f6c3c195470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:30:59.096 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:59.091 7f6c418c5700 1 --2- 172.21.2.102:0/2562299796 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6c3c0bb8e0 0x7f6c3c0bbce0 secure :-1 s=READY pgs=2069 cs=0 l=1 rev1=1 rx=0x7f6c3400b910 tx=0x7f6c3400bcb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:59.096 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:59.091 7f6c31ffb700 1 -- 172.21.2.102:0/2562299796 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6c3400bdf0 con 0x7f6c3c0bb8e0 2021-02-10T11:30:59.096 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:59.091 7f6c31ffb700 1 -- 172.21.2.102:0/2562299796 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c34005820 con 0x7f6c3c0bb8e0 2021-02-10T11:30:59.096 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:59.091 7f6c31ffb700 1 -- 172.21.2.102:0/2562299796 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6c34005c70 con 0x7f6c3c0bb8e0 2021-02-10T11:30:59.096 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:59.091 7f6c432a8700 1 -- 172.21.2.102:0/2562299796 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c3c1a49e0 con 0x7f6c3c0bb8e0 2021-02-10T11:30:59.096 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:59.091 7f6c432a8700 1 -- 172.21.2.102:0/2562299796 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c3c1a49e0 con 0x7f6c3c0bb8e0 2021-02-10T11:30:59.096 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:59.091 7f6c31ffb700 1 -- 172.21.2.102:0/2562299796 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f6c34005c10 con 0x7f6c3c0bb8e0 2021-02-10T11:30:59.096 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:59.091 7f6c31ffb700 1 --2- 172.21.2.102:0/2562299796 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6c1c0186c0 0x7f6c1c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:59.096 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:59.091 7f6c31ffb700 1 -- 172.21.2.102:0/2562299796 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f6c34032920 con 0x7f6c3c0bb8e0 2021-02-10T11:30:59.096 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:59.091 7f6c410c4700 1 --2- 172.21.2.102:0/2562299796 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6c1c0186c0 0x7f6c1c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:59.096 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:59.092 7f6c410c4700 1 --2- 172.21.2.102:0/2562299796 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6c1c0186c0 0x7f6c1c01abc0 crc :-1 s=READY pgs=920 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:59.124 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:59.123 7f6c432a8700 1 -- 172.21.2.102:0/2562299796 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c3c0bc5f0 con 0x7f6c3c0bb8e0 2021-02-10T11:30:59.126 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:59.126 7f6c31ffb700 1 -- 172.21.2.102:0/2562299796 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f6c340308c0 con 0x7f6c3c0bb8e0 2021-02-10T11:30:59.254 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:59.253 7f6c432a8700 1 -- 172.21.2.102:0/2562299796 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6c3c0bc5f0 con 0x7f6c3c0bb8e0 2021-02-10T11:30:59.254 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:59.254 7f6c31ffb700 1 -- 172.21.2.102:0/2562299796 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v102) v1 ==== 74+0+14018 (secure 0 0 0) 0x7f6c340308c0 con 0x7f6c3c0bb8e0 2021-02-10T11:30:59.255 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:30:59.255 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":102,"fsid":"68db30ea-dbce-4aef-a6df-f01c6fea0513","created":"2021-02-10 11:03:06.331557","modified":"2021-02-10 11:30:59.048777","last_up_change":"2021-02-10 11:03:32.690562","last_in_change":"2021-02-10 11:03:32.690562","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:03:37.322519","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:30:56.015905","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"101","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":21,"pool_name":"cephfs_data","create_time":"2021-02-10 11:30:57.042942","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"101","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"be691b4d-a039-408a-af80-9d53f02b6f1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6818","nonce":36088},{"type":"v1","addr":"172.21.2.102:6820","nonce":36088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6822","nonce":36088},{"type":"v1","addr":"172.21.2.102:6824","nonce":36088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6830","nonce":36088},{"type":"v1","addr":"172.21.2.102:6832","nonce":36088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6826","nonce":36088},{"type":"v1","addr":"172.21.2.102:6828","nonce":36088}]},"public_addr":"172.21.2.102:6820/36088","cluster_addr":"172.21.2.102:6824/36088","heartbeat_back_addr":"172.21.2.102:6832/36088","heartbeat_front_addr":"172.21.2.102:6828/36088","state":["exists","up"]},{"osd":1,"uuid":"64e3ccc8-9c69-4d7c-865e-af8e83822635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6819","nonce":36087},{"type":"v1","addr":"172.21.2.102:6821","nonce":36087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6823","nonce":36087},{"type":"v1","addr":"172.21.2.102:6825","nonce":36087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6831","nonce":36087},{"type":"v1","addr":"172.21.2.102:6833","nonce":36087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6827","nonce":36087},{"type":"v1","addr":"172.21.2.102:6829","nonce":36087}]},"public_addr":"172.21.2.102:6821/36087","cluster_addr":"172.21.2.102:6825/36087","heartbeat_back_addr":"172.21.2.102:6833/36087","heartbeat_front_addr":"172.21.2.102:6829/36087","state":["exists","up"]},{"osd":2,"uuid":"153700af-747f-481f-bdf4-d6072d3a842f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6810","nonce":36089},{"type":"v1","addr":"172.21.2.102:6811","nonce":36089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6812","nonce":36089},{"type":"v1","addr":"172.21.2.102:6813","nonce":36089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6816","nonce":36089},{"type":"v1","addr":"172.21.2.102:6817","nonce":36089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6814","nonce":36089},{"type":"v1","addr":"172.21.2.102:6815","nonce":36089}]},"public_addr":"172.21.2.102:6811/36089","cluster_addr":"172.21.2.102:6813/36089","heartbeat_back_addr":"172.21.2.102:6817/36089","heartbeat_front_addr":"172.21.2.102:6815/36089","state":["exists","up"]},{"osd":3,"uuid":"cbc678f4-83a9-45e6-8830-2f7c232b1ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6802","nonce":36090},{"type":"v1","addr":"172.21.2.102:6803","nonce":36090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6804","nonce":36090},{"type":"v1","addr":"172.21.2.102:6805","nonce":36090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6808","nonce":36090},{"type":"v1","addr":"172.21.2.102:6809","nonce":36090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6806","nonce":36090},{"type":"v1","addr":"172.21.2.102:6807","nonce":36090}]},"public_addr":"172.21.2.102:6803/36090","cluster_addr":"172.21.2.102:6805/36090","heartbeat_back_addr":"172.21.2.102:6809/36090","heartbeat_front_addr":"172.21.2.102:6807/36090","state":["exists","up"]},{"osd":4,"uuid":"c01e4008-b258-4f10-afd2-9e7f37214351","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6808","nonce":36368},{"type":"v1","addr":"172.21.2.113:6811","nonce":36368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6814","nonce":36368},{"type":"v1","addr":"172.21.2.113:6818","nonce":36368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6826","nonce":36368},{"type":"v1","addr":"172.21.2.113:6828","nonce":36368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6820","nonce":36368},{"type":"v1","addr":"172.21.2.113:6824","nonce":36368}]},"public_addr":"172.21.2.113:6811/36368","cluster_addr":"172.21.2.113:6818/36368","heartbeat_back_addr":"172.21.2.113:6828/36368","heartbeat_front_addr":"172.21.2.113:6824/36368","state":["exists","up"]},{"osd":5,"uuid":"551f6007-daf8-4a98-8ee0-c2b16bcccb5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6802","nonce":36366},{"type":"v1","addr":"172.21.2.113:6805","nonce":36366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6807","nonce":36366},{"type":"v1","addr":"172.21.2.113:6810","nonce":36366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6819","nonce":36366},{"type":"v1","addr":"172.21.2.113:6822","nonce":36366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6812","nonce":36366},{"type":"v1","addr":"172.21.2.113:6815","nonce":36366}]},"public_addr":"172.21.2.113:6805/36366","cluster_addr":"172.21.2.113:6810/36366","heartbeat_back_addr":"172.21.2.113:6822/36366","heartbeat_front_addr":"172.21.2.113:6815/36366","state":["exists","up"]},{"osd":6,"uuid":"bf67c040-0e50-48be-b102-c3943bc9ebef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6800","nonce":36367},{"type":"v1","addr":"172.21.2.113:6801","nonce":36367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6803","nonce":36367},{"type":"v1","addr":"172.21.2.113:6804","nonce":36367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6813","nonce":36367},{"type":"v1","addr":"172.21.2.113:6816","nonce":36367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6806","nonce":36367},{"type":"v1","addr":"172.21.2.113:6809","nonce":36367}]},"public_addr":"172.21.2.113:6801/36367","cluster_addr":"172.21.2.113:6804/36367","heartbeat_back_addr":"172.21.2.113:6816/36367","heartbeat_front_addr":"172.21.2.113:6809/36367","state":["exists","up"]},{"osd":7,"uuid":"f5461f80-018d-428f-be89-2eb2bcbe7bb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6817","nonce":36365},{"type":"v1","addr":"172.21.2.113:6821","nonce":36365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6823","nonce":36365},{"type":"v1","addr":"172.21.2.113:6825","nonce":36365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6830","nonce":36365},{"type":"v1","addr":"172.21.2.113:6831","nonce":36365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6827","nonce":36365},{"type":"v1","addr":"172.21.2.113:6829","nonce":36365}]},"public_addr":"172.21.2.113:6821/36365","cluster_addr":"172.21.2.113:6825/36365","heartbeat_back_addr":"172.21.2.113:6831/36365","heartbeat_front_addr":"172.21.2.113:6829/36365","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:30:59.256 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:59.255 7f6c432a8700 1 -- 172.21.2.102:0/2562299796 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6c1c0186c0 msgr2=0x7f6c1c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:59.256 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:59.255 7f6c432a8700 1 --2- 172.21.2.102:0/2562299796 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6c1c0186c0 0x7f6c1c01abc0 crc :-1 s=READY pgs=920 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:59.256 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:59.255 7f6c432a8700 1 -- 172.21.2.102:0/2562299796 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6c3c0bb8e0 msgr2=0x7f6c3c0bbce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:59.256 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:59.255 7f6c432a8700 1 --2- 172.21.2.102:0/2562299796 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6c3c0bb8e0 0x7f6c3c0bbce0 secure :-1 s=READY pgs=2069 cs=0 l=1 rev1=1 rx=0x7f6c3400b910 tx=0x7f6c3400bcb0).stop 2021-02-10T11:30:59.257 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:59.255 7f6c432a8700 1 -- 172.21.2.102:0/2562299796 shutdown_connections 2021-02-10T11:30:59.257 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:59.255 7f6c432a8700 1 --2- 172.21.2.102:0/2562299796 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6c3c0bb8e0 0x7f6c3c0bbce0 unknown :-1 s=CLOSED pgs=2069 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:59.257 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:59.255 7f6c432a8700 1 --2- 172.21.2.102:0/2562299796 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6c1c0186c0 0x7f6c1c01abc0 unknown :-1 s=CLOSED pgs=920 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:59.257 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:59.255 7f6c432a8700 1 --2- 172.21.2.102:0/2562299796 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6c3c195070 0x7f6c3c195470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:59.257 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:59.255 7f6c432a8700 1 --2- 172.21.2.102:0/2562299796 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6c3c1942b0 0x7f6c3c0b3550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:59.257 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:59.255 7f6c432a8700 1 -- 172.21.2.102:0/2562299796 >> 172.21.2.102:0/2562299796 conn(0x7f6c3c0b3040 msgr2=0x7f6c3c0c3250 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:59.257 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:59.255 7f6c432a8700 1 -- 172.21.2.102:0/2562299796 shutdown_connections 2021-02-10T11:30:59.257 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:59.256 7f6c432a8700 1 -- 172.21.2.102:0/2562299796 wait complete. 2021-02-10T11:30:59.265 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2021-02-10T11:30:59.696 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:59.679 7fb5b6ba1700 1 Processor -- start 2021-02-10T11:30:59.697 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:59.679 7fb5b6ba1700 1 -- start start 2021-02-10T11:30:59.697 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:59.679 7fb5b6ba1700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb5b00d1c70 0x7fb5b00cef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:59.697 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:59.679 7fb5b6ba1700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fb5b00cf580 0x7fb5b00cf980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:59.697 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:59.679 7fb5b6ba1700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5b00fe1c0 con 0x7fb5b00d1870 2021-02-10T11:30:59.697 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:59.679 7fb5b6ba1700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5b01b0cb0 con 0x7fb5b00cf580 2021-02-10T11:30:59.697 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:59.679 7fb5b6ba1700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5b0109a60 con 0x7fb5b00d1c70 2021-02-10T11:30:59.697 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:59.679 7fb5b51be700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fb5b00cf580 0x7fb5b00cf980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:59.697 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:59.679 7fb5b51be700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fb5b00cf580 0x7fb5b00cf980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:33292/0 (socket says 172.21.2.102:33292) 2021-02-10T11:30:59.697 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:59.679 7fb5b51be700 1 -- 172.21.2.102:0/1226602958 learned_addr learned my addr 172.21.2.102:0/1226602958 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:30:59.697 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:59.680 7fb5b51be700 1 -- 172.21.2.102:0/1226602958 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb5b00d1c70 msgr2=0x7fb5b00cef60 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:30:59.697 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:59.680 7fb5b51be700 1 --2- 172.21.2.102:0/1226602958 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb5b00d1c70 0x7fb5b00cef60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:59.698 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:59.680 7fb5b51be700 1 -- 172.21.2.102:0/1226602958 >> v1:172.21.2.102:6789/0 conn(0x7fb5b00d1870 legacy=0x7fb5b01b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:59.698 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:59.680 7fb5b51be700 1 -- 172.21.2.102:0/1226602958 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5b0088070 con 0x7fb5b00cf580 2021-02-10T11:30:59.698 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:59.680 7fb5b51be700 1 --2- 172.21.2.102:0/1226602958 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb5b00cf580 0x7fb5b00cf980 secure :-1 s=READY pgs=2071 cs=0 l=1 rev1=1 rx=0x7fb5a8006f00 tx=0x7fb5a8009e90).ready entity=mon.0 client_cookie=cb5d5df3901423fa server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:59.698 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:59.680 7fb5af7fe700 1 -- 172.21.2.102:0/1226602958 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb5a800b300 con 0x7fb5b00cf580 2021-02-10T11:30:59.698 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:59.680 7fb5af7fe700 1 -- 172.21.2.102:0/1226602958 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5a800b640 con 0x7fb5b00cf580 2021-02-10T11:30:59.698 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:59.680 7fb5af7fe700 1 -- 172.21.2.102:0/1226602958 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb5a800baa0 con 0x7fb5b00cf580 2021-02-10T11:30:59.698 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:59.680 7fb5b6ba1700 1 -- 172.21.2.102:0/1226602958 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb5b00cf580 msgr2=0x7fb5b00cf980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:59.698 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:59.680 7fb5b6ba1700 1 --2- 172.21.2.102:0/1226602958 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb5b00cf580 0x7fb5b00cf980 secure :-1 s=READY pgs=2071 cs=0 l=1 rev1=1 rx=0x7fb5a8006f00 tx=0x7fb5a8009e90).stop 2021-02-10T11:30:59.698 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:59.681 7fb5b6ba1700 1 -- 172.21.2.102:0/1226602958 shutdown_connections 2021-02-10T11:30:59.698 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:59.681 7fb5b6ba1700 1 --2- 172.21.2.102:0/1226602958 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb5b00cf580 0x7fb5b00cf980 unknown :-1 s=CLOSED pgs=2071 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:59.698 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:59.681 7fb5b6ba1700 1 --2- 172.21.2.102:0/1226602958 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb5b00d1c70 0x7fb5b00cef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:59.698 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:59.681 7fb5b6ba1700 1 -- 172.21.2.102:0/1226602958 >> 172.21.2.102:0/1226602958 conn(0x7fb5b00ce420 msgr2=0x7fb5b00d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:59.698 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:59.681 7fb5b6ba1700 1 -- 172.21.2.102:0/1226602958 shutdown_connections 2021-02-10T11:30:59.699 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:59.681 7fb5b6ba1700 1 -- 172.21.2.102:0/1226602958 wait complete. 2021-02-10T11:30:59.699 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:59.696 7fb5b6ba1700 1 Processor -- start 2021-02-10T11:30:59.699 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:59.696 7fb5b6ba1700 1 -- start start 2021-02-10T11:30:59.700 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:59.696 7fb5b6ba1700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb5b00c3820 0x7fb5b0084cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:59.700 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:59.696 7fb5b6ba1700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb5b00a0290 0x7fb5b00852d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:59.700 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:59.696 7fb5b6ba1700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb5b0086270 0x7fb5b0086670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:59.700 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:59.696 7fb5b6ba1700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5b00c0700 con 0x7fb5b0086270 2021-02-10T11:30:59.700 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:59.696 7fb5b6ba1700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5b00858f0 con 0x7fb5b00c3820 2021-02-10T11:30:59.700 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:59.696 7fb5b6ba1700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5b0085b10 con 0x7fb5b00a0290 2021-02-10T11:30:59.700 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:59.696 7fb5b51be700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb5b0086270 0x7fb5b0086670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:59.700 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:59.696 7fb5b51be700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb5b0086270 0x7fb5b0086670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:33298/0 (socket says 172.21.2.102:33298) 2021-02-10T11:30:59.701 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:59.696 7fb5b51be700 1 -- 172.21.2.102:0/3664672910 learned_addr learned my addr 172.21.2.102:0/3664672910 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:30:59.701 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:59.697 7fb5b51be700 1 -- 172.21.2.102:0/3664672910 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb5b00a0290 msgr2=0x7fb5b00852d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:59.701 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:59.697 7fb5b51be700 1 --2- 172.21.2.102:0/3664672910 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb5b00a0290 0x7fb5b00852d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:59.701 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:59.697 7fb5b51be700 1 -- 172.21.2.102:0/3664672910 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb5b00c3820 msgr2=0x7fb5b0084cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:59.701 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:59.697 7fb5b51be700 1 --2- 172.21.2.102:0/3664672910 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb5b00c3820 0x7fb5b0084cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:59.701 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:59.697 7fb5b51be700 1 -- 172.21.2.102:0/3664672910 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5b0086c90 con 0x7fb5b0086270 2021-02-10T11:30:59.701 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:59.697 7fb5b51be700 1 --2- 172.21.2.102:0/3664672910 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb5b0086270 0x7fb5b0086670 secure :-1 s=READY pgs=2072 cs=0 l=1 rev1=1 rx=0x7fb5a800b950 tx=0x7fb5a800bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:59.701 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:59.697 7fb5ad7fa700 1 -- 172.21.2.102:0/3664672910 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb5a8009450 con 0x7fb5b0086270 2021-02-10T11:30:59.701 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:59.697 7fb5ad7fa700 1 -- 172.21.2.102:0/3664672910 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5a80058b0 con 0x7fb5b0086270 2021-02-10T11:30:59.701 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:59.697 7fb5ad7fa700 1 -- 172.21.2.102:0/3664672910 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb5a8005cc0 con 0x7fb5b0086270 2021-02-10T11:30:59.701 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:59.697 7fb5b6ba1700 1 -- 172.21.2.102:0/3664672910 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5b0086c90 con 0x7fb5b0086270 2021-02-10T11:30:59.701 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:59.697 7fb5b6ba1700 1 -- 172.21.2.102:0/3664672910 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5b01a2f00 con 0x7fb5b0086270 2021-02-10T11:30:59.701 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:59.697 7fb5ad7fa700 1 -- 172.21.2.102:0/3664672910 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fb5a8005a90 con 0x7fb5b0086270 2021-02-10T11:30:59.702 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:59.698 7fb5ad7fa700 1 --2- 172.21.2.102:0/3664672910 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb5940186c0 0x7fb59401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:59.702 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:59.698 7fb5ad7fa700 1 -- 172.21.2.102:0/3664672910 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fb5a80328c0 con 0x7fb5b0086270 2021-02-10T11:30:59.702 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:59.698 7fb5b49bd700 1 --2- 172.21.2.102:0/3664672910 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb5940186c0 0x7fb59401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:59.702 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:59.698 7fb5b49bd700 1 --2- 172.21.2.102:0/3664672910 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb5940186c0 0x7fb59401abc0 crc :-1 s=READY pgs=921 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:59.729 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:59.729 7fb5b6ba1700 1 -- 172.21.2.102:0/3664672910 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5b0082a70 con 0x7fb5b0086270 2021-02-10T11:30:59.733 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:59.731 7fb5ad7fa700 1 -- 172.21.2.102:0/3664672910 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fb5a8030b20 con 0x7fb5b0086270 2021-02-10T11:30:59.870 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:30:59.869 7fb5b6ba1700 1 -- 172.21.2.102:0/3664672910 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"} v 0) v1 -- 0x7fb5b0082a70 con 0x7fb5b0086270 2021-02-10T11:31:00.060 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:00.059 7fb5ad7fa700 1 -- 172.21.2.102:0/3664672910 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"}]=0 v82) v1 ==== 118+0+0 (secure 0 0 0) 0x7fb5a8030b20 con 0x7fb5b0086270 2021-02-10T11:31:00.087 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:00.086 7fb5b6ba1700 1 -- 172.21.2.102:0/3664672910 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb5940186c0 msgr2=0x7fb59401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:00.087 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:00.086 7fb5b6ba1700 1 --2- 172.21.2.102:0/3664672910 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb5940186c0 0x7fb59401abc0 crc :-1 s=READY pgs=921 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:00.087 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:00.086 7fb5b6ba1700 1 -- 172.21.2.102:0/3664672910 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb5b0086270 msgr2=0x7fb5b0086670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:00.088 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:00.086 7fb5b6ba1700 1 --2- 172.21.2.102:0/3664672910 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb5b0086270 0x7fb5b0086670 secure :-1 s=READY pgs=2072 cs=0 l=1 rev1=1 rx=0x7fb5a800b950 tx=0x7fb5a800bcf0).stop 2021-02-10T11:31:00.088 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:00.087 7fb5b6ba1700 1 -- 172.21.2.102:0/3664672910 shutdown_connections 2021-02-10T11:31:00.088 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:00.087 7fb5b6ba1700 1 --2- 172.21.2.102:0/3664672910 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb5b0086270 0x7fb5b0086670 unknown :-1 s=CLOSED pgs=2072 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:00.088 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:00.087 7fb5b6ba1700 1 --2- 172.21.2.102:0/3664672910 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb5b00a0290 0x7fb5b00852d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:00.088 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:00.087 7fb5b6ba1700 1 --2- 172.21.2.102:0/3664672910 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb5940186c0 0x7fb59401abc0 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:00.088 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:00.087 7fb5b6ba1700 1 --2- 172.21.2.102:0/3664672910 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb5b00c3820 0x7fb5b0084cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:00.088 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:00.087 7fb5b6ba1700 1 -- 172.21.2.102:0/3664672910 >> 172.21.2.102:0/3664672910 conn(0x7fb5b00872c0 msgr2=0x7fb5b0095700 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:00.089 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:00.087 7fb5b6ba1700 1 -- 172.21.2.102:0/3664672910 shutdown_connections 2021-02-10T11:31:00.089 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:00.087 7fb5b6ba1700 1 -- 172.21.2.102:0/3664672910 wait complete. 2021-02-10T11:31:00.097 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:31:00.524 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:00.507 7f89feccc700 1 Processor -- start 2021-02-10T11:31:00.524 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:00.507 7f89feccc700 1 -- start start 2021-02-10T11:31:00.524 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:00.508 7f89feccc700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f89f80d1870 0x7f89f81b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:00.524 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:00.508 7f89feccc700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f89f81b0a10 0x7f89f80cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:00.524 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:00.508 7f89feccc700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f89f80d1c70 0x7f89f80cf950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:00.524 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:00.508 7f89feccc700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89f80e5ed0 con 0x7f89f81b0a10 2021-02-10T11:31:00.524 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:00.508 7f89feccc700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89f80f0b10 con 0x7f89f80d1870 2021-02-10T11:31:00.524 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:00.508 7f89feccc700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89f80cff70 con 0x7f89f80d1c70 2021-02-10T11:31:00.524 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:00.508 7f89f7fff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f89f81b0a10 0x7f89f80cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:00.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:00.508 7f89f7fff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f89f81b0a10 0x7f89f80cf330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:33304/0 (socket says 172.21.2.102:33304) 2021-02-10T11:31:00.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:00.508 7f89f7fff700 1 -- 172.21.2.102:0/71371001 learned_addr learned my addr 172.21.2.102:0/71371001 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:31:00.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:00.508 7f89f7fff700 1 -- 172.21.2.102:0/71371001 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f89f80d1c70 msgr2=0x7f89f80cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:00.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:00.508 7f89f7fff700 1 --2- 172.21.2.102:0/71371001 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f89f80d1c70 0x7f89f80cf950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:00.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:00.508 7f89f7fff700 1 -- 172.21.2.102:0/71371001 >> v2:172.21.2.113:3300/0 conn(0x7f89f80d1870 msgr2=0x7f89f81b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:00.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:00.508 7f89f7fff700 1 --2- 172.21.2.102:0/71371001 >> v2:172.21.2.113:3300/0 conn(0x7f89f80d1870 0x7f89f81b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:00.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:00.508 7f89f7fff700 1 -- 172.21.2.102:0/71371001 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89f80d0150 con 0x7f89f81b0a10 2021-02-10T11:31:00.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:00.508 7f89f7fff700 1 --2- 172.21.2.102:0/71371001 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f89f81b0a10 0x7f89f80cf330 secure :-1 s=READY pgs=2073 cs=0 l=1 rev1=1 rx=0x7f89e8006700 tx=0x7f89e8009b80).ready entity=mon.0 client_cookie=38ce7acc9486e22e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:00.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:00.508 7f89f77fe700 1 -- 172.21.2.102:0/71371001 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f89e800c0c0 con 0x7f89f81b0a10 2021-02-10T11:31:00.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:00.508 7f89f77fe700 1 -- 172.21.2.102:0/71371001 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89e800c440 con 0x7f89f81b0a10 2021-02-10T11:31:00.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:00.508 7f89f77fe700 1 -- 172.21.2.102:0/71371001 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f89e800c920 con 0x7f89f81b0a10 2021-02-10T11:31:00.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:00.508 7f89feccc700 1 -- 172.21.2.102:0/71371001 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f89f81b0a10 msgr2=0x7f89f80cf330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:00.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:00.509 7f89feccc700 1 --2- 172.21.2.102:0/71371001 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f89f81b0a10 0x7f89f80cf330 secure :-1 s=READY pgs=2073 cs=0 l=1 rev1=1 rx=0x7f89e8006700 tx=0x7f89e8009b80).stop 2021-02-10T11:31:00.526 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:00.509 7f89feccc700 1 -- 172.21.2.102:0/71371001 shutdown_connections 2021-02-10T11:31:00.526 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:00.509 7f89feccc700 1 --2- 172.21.2.102:0/71371001 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f89f80d1c70 0x7f89f80cf950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:00.526 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:00.509 7f89feccc700 1 --2- 172.21.2.102:0/71371001 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f89f81b0a10 0x7f89f80cf330 unknown :-1 s=CLOSED pgs=2073 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:00.526 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:00.509 7f89feccc700 1 --2- 172.21.2.102:0/71371001 >> v2:172.21.2.113:3300/0 conn(0x7f89f80d1870 0x7f89f81b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:00.526 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:00.509 7f89feccc700 1 -- 172.21.2.102:0/71371001 >> 172.21.2.102:0/71371001 conn(0x7f89f80ce450 msgr2=0x7f89f80d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:00.526 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:00.509 7f89feccc700 1 -- 172.21.2.102:0/71371001 shutdown_connections 2021-02-10T11:31:00.526 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:00.509 7f89feccc700 1 -- 172.21.2.102:0/71371001 wait complete. 2021-02-10T11:31:00.527 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:00.523 7f89feccc700 1 Processor -- start 2021-02-10T11:31:00.528 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:00.523 7f89feccc700 1 -- start start 2021-02-10T11:31:00.528 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:00.524 7f89feccc700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f89f81952e0 0x7f89f818ed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:00.528 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:00.524 7f89feccc700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f89f81b8bc0 0x7f89f81ac4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:00.528 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:00.524 7f89feccc700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f89f81acae0 0x7f89f8194240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:00.528 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:00.524 7f89feccc700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89f81b8640 con 0x7f89f81b8bc0 2021-02-10T11:31:00.528 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:00.524 7f89feccc700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89f819e8b0 con 0x7f89f81acae0 2021-02-10T11:31:00.528 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:00.524 7f89feccc700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89f819ea90 con 0x7f89f81952e0 2021-02-10T11:31:00.528 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:00.524 7f89f7fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f89f81b8bc0 0x7f89f81ac4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:00.528 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:00.524 7f89f7fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f89f81b8bc0 0x7f89f81ac4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:33312/0 (socket says 172.21.2.102:33312) 2021-02-10T11:31:00.528 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:00.524 7f89f7fff700 1 -- 172.21.2.102:0/1395722257 learned_addr learned my addr 172.21.2.102:0/1395722257 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:31:00.528 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:00.524 7f89f7fff700 1 -- 172.21.2.102:0/1395722257 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f89f81952e0 msgr2=0x7f89f818ed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:00.528 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:00.524 7f89f7fff700 1 --2- 172.21.2.102:0/1395722257 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f89f81952e0 0x7f89f818ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:00.528 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:00.524 7f89f7fff700 1 -- 172.21.2.102:0/1395722257 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f89f81acae0 msgr2=0x7f89f8194240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:00.529 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:00.524 7f89f7fff700 1 --2- 172.21.2.102:0/1395722257 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f89f81acae0 0x7f89f8194240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:00.529 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:00.524 7f89f7fff700 1 -- 172.21.2.102:0/1395722257 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89f81acee0 con 0x7f89f81b8bc0 2021-02-10T11:31:00.529 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:00.525 7f89f7fff700 1 --2- 172.21.2.102:0/1395722257 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f89f81b8bc0 0x7f89f81ac4c0 secure :-1 s=READY pgs=2074 cs=0 l=1 rev1=1 rx=0x7f89e800b5e0 tx=0x7f89e800b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:00.529 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:00.525 7f89f57fa700 1 -- 172.21.2.102:0/1395722257 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f89e800bd70 con 0x7f89f81b8bc0 2021-02-10T11:31:00.529 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:00.525 7f89f57fa700 1 -- 172.21.2.102:0/1395722257 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89e8002030 con 0x7f89f81b8bc0 2021-02-10T11:31:00.529 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:00.525 7f89f57fa700 1 -- 172.21.2.102:0/1395722257 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f89e8002550 con 0x7f89f81b8bc0 2021-02-10T11:31:00.529 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:00.525 7f89feccc700 1 -- 172.21.2.102:0/1395722257 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89f81acee0 con 0x7f89f81b8bc0 2021-02-10T11:31:00.529 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:00.525 7f89feccc700 1 -- 172.21.2.102:0/1395722257 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89f8194b50 con 0x7f89f81b8bc0 2021-02-10T11:31:00.529 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:00.526 7f89f57fa700 1 -- 172.21.2.102:0/1395722257 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f89e8000c90 con 0x7f89f81b8bc0 2021-02-10T11:31:00.529 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:00.526 7f89f57fa700 1 --2- 172.21.2.102:0/1395722257 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f89dc0186c0 0x7f89dc01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:00.529 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:00.526 7f89f57fa700 1 -- 172.21.2.102:0/1395722257 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f89e80325f0 con 0x7f89f81b8bc0 2021-02-10T11:31:00.529 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:00.526 7f89fcae8700 1 --2- 172.21.2.102:0/1395722257 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f89dc0186c0 0x7f89dc01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:00.529 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:00.526 7f89fcae8700 1 --2- 172.21.2.102:0/1395722257 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f89dc0186c0 0x7f89dc01abc0 crc :-1 s=READY pgs=922 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:00.559 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:00.558 7f89feccc700 1 -- 172.21.2.102:0/1395722257 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89f819ea90 con 0x7f89f81b8bc0 2021-02-10T11:31:00.561 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:00.560 7f89f57fa700 1 -- 172.21.2.102:0/1395722257 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f89e80009b0 con 0x7f89f81b8bc0 2021-02-10T11:31:00.689 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:00.689 7f89feccc700 1 -- 172.21.2.102:0/1395722257 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f89f819ea90 con 0x7f89f81b8bc0 2021-02-10T11:31:00.690 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:00.689 7f89f57fa700 1 -- 172.21.2.102:0/1395722257 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+2101 (secure 0 0 0) 0x7f89e8000c90 con 0x7f89f81b8bc0 2021-02-10T11:31:00.690 INFO:teuthology.orchestra.run.gibba002.stderr:dumped fsmap epoch 82 2021-02-10T11:31:00.691 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:31:00.691 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":82,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10056,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.102:6835/711305601","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6834","nonce":711305601},{"type":"v1","addr":"172.21.2.102:6835","nonce":711305601}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":78}],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:30:58.047057","modified":"2021-02-10 11:31:00.055485","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6353},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6353":{"gid":6353,"name":"b","rank":0,"incarnation":79,"state":"up:active","state_seq":3,"addr":"172.21.2.113:6833/1091618344","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6832","nonce":1091618344},{"type":"v1","addr":"172.21.2.113:6833","nonce":1091618344}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":10}]} 2021-02-10T11:31:00.691 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:00.691 7f89feccc700 1 -- 172.21.2.102:0/1395722257 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f89dc0186c0 msgr2=0x7f89dc01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:00.692 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:00.691 7f89feccc700 1 --2- 172.21.2.102:0/1395722257 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f89dc0186c0 0x7f89dc01abc0 crc :-1 s=READY pgs=922 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:00.692 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:00.691 7f89feccc700 1 -- 172.21.2.102:0/1395722257 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f89f81b8bc0 msgr2=0x7f89f81ac4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:00.692 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:00.691 7f89feccc700 1 --2- 172.21.2.102:0/1395722257 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f89f81b8bc0 0x7f89f81ac4c0 secure :-1 s=READY pgs=2074 cs=0 l=1 rev1=1 rx=0x7f89e800b5e0 tx=0x7f89e800b980).stop 2021-02-10T11:31:00.692 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:00.691 7f89feccc700 1 -- 172.21.2.102:0/1395722257 shutdown_connections 2021-02-10T11:31:00.692 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:00.691 7f89feccc700 1 --2- 172.21.2.102:0/1395722257 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f89dc0186c0 0x7f89dc01abc0 unknown :-1 s=CLOSED pgs=922 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:00.693 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:00.691 7f89feccc700 1 --2- 172.21.2.102:0/1395722257 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f89f81acae0 0x7f89f8194240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:00.693 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:00.691 7f89feccc700 1 --2- 172.21.2.102:0/1395722257 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f89f81b8bc0 0x7f89f81ac4c0 unknown :-1 s=CLOSED pgs=2074 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:00.693 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:00.691 7f89feccc700 1 --2- 172.21.2.102:0/1395722257 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f89f81952e0 0x7f89f818ed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:00.693 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:00.691 7f89feccc700 1 -- 172.21.2.102:0/1395722257 >> 172.21.2.102:0/1395722257 conn(0x7f89f81ac040 msgr2=0x7f89f8094980 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:00.693 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:00.691 7f89feccc700 1 -- 172.21.2.102:0/1395722257 shutdown_connections 2021-02-10T11:31:00.693 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:00.691 7f89feccc700 1 -- 172.21.2.102:0/1395722257 wait complete. 2021-02-10T11:31:00.700 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:31:01.130 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:01.112 7fdfee5de700 1 Processor -- start 2021-02-10T11:31:01.130 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:01.113 7fdfee5de700 1 -- start start 2021-02-10T11:31:01.130 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:01.113 7fdfee5de700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdfe80d1c70 0x7fdfe80f20a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:01.130 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:01.113 7fdfee5de700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfe80f0b10 con 0x7fdfe80d0340 2021-02-10T11:31:01.130 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:01.113 7fdfee5de700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfe81b0be0 con 0x7fdfe80d1870 2021-02-10T11:31:01.130 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:01.113 7fdfee5de700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdfe80e5ed0 con 0x7fdfe80d1c70 2021-02-10T11:31:01.131 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:01.113 7fdfe77fe700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fdfe80d0340 0x7fdfe80ef640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:53776/0 (socket says 172.21.2.102:53776) 2021-02-10T11:31:01.131 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:01.113 7fdfe77fe700 1 -- 172.21.2.102:0/1794013606 learned_addr learned my addr 172.21.2.102:0/1794013606 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:31:01.131 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:01.113 7fdfecbfb700 1 --2- 172.21.2.102:0/1794013606 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdfe80d1c70 0x7fdfe80f20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:01.131 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:01.114 7fdfe6ffd700 1 -- 172.21.2.102:0/1794013606 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3728543406 0 0) 0x7fdfdc0009b0 con 0x7fdfe80d0340 2021-02-10T11:31:01.131 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:01.114 7fdfe6ffd700 1 -- 172.21.2.102:0/1794013606 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdfd0002fe0 con 0x7fdfe80d0340 2021-02-10T11:31:01.131 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:01.114 7fdfecbfb700 1 -- 172.21.2.102:0/1794013606 >> v1:172.21.2.113:6789/0 conn(0x7fdfe80d1870 legacy=0x7fdfe81b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:01.131 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:01.114 7fdfe6ffd700 1 -- 172.21.2.102:0/1794013606 <== mon.1 v1:172.21.2.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2564137244 0 0) 0x7fdfd8000c60 con 0x7fdfe80d1870 2021-02-10T11:31:01.131 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:01.114 7fdfecbfb700 1 -- 172.21.2.102:0/1794013606 >> v1:172.21.2.102:6789/0 conn(0x7fdfe80d0340 legacy=0x7fdfe80ef640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:01.131 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:01.114 7fdfecbfb700 1 -- 172.21.2.102:0/1794013606 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfe80efe30 con 0x7fdfe80d1c70 2021-02-10T11:31:01.131 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:01.114 7fdfecbfb700 1 --2- 172.21.2.102:0/1794013606 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdfe80d1c70 0x7fdfe80f20a0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7fdfe0006e50 tx=0x7fdfe0009e70).ready entity=mon.2 client_cookie=891bc7a959358a08 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:01.132 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:01.114 7fdfe6ffd700 1 -- 172.21.2.102:0/1794013606 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdfe0006a20 con 0x7fdfe80d1c70 2021-02-10T11:31:01.132 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:01.114 7fdfe6ffd700 1 -- 172.21.2.102:0/1794013606 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfe0006790 con 0x7fdfe80d1c70 2021-02-10T11:31:01.132 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:01.114 7fdfe6ffd700 1 -- 172.21.2.102:0/1794013606 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdfe000b700 con 0x7fdfe80d1c70 2021-02-10T11:31:01.132 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:01.114 7fdfee5de700 1 -- 172.21.2.102:0/1794013606 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdfe80d1c70 msgr2=0x7fdfe80f20a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:01.132 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:01.114 7fdfee5de700 1 --2- 172.21.2.102:0/1794013606 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdfe80d1c70 0x7fdfe80f20a0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7fdfe0006e50 tx=0x7fdfe0009e70).stop 2021-02-10T11:31:01.132 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:01.115 7fdfee5de700 1 -- 172.21.2.102:0/1794013606 shutdown_connections 2021-02-10T11:31:01.132 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:01.115 7fdfee5de700 1 --2- 172.21.2.102:0/1794013606 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdfe80d1c70 0x7fdfe80f20a0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:01.132 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:01.115 7fdfee5de700 1 -- 172.21.2.102:0/1794013606 >> 172.21.2.102:0/1794013606 conn(0x7fdfe80ce450 msgr2=0x7fdfe80d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:01.132 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:01.115 7fdfee5de700 1 -- 172.21.2.102:0/1794013606 shutdown_connections 2021-02-10T11:31:01.132 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:01.115 7fdfee5de700 1 -- 172.21.2.102:0/1794013606 wait complete. 2021-02-10T11:31:01.134 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:01.129 7fdfee5de700 1 Processor -- start 2021-02-10T11:31:01.134 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:01.130 7fdfee5de700 1 -- start start 2021-02-10T11:31:01.134 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:01.130 7fdfee5de700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdfe81b8ad0 0x7fdfe8187e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:01.134 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:01.130 7fdfee5de700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdfe81b8ed0 0x7fdfe80b1af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:01.134 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:01.130 7fdfee5de700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fdfe80b2110 0x7fdfe81b7a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:01.134 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:01.130 7fdfee5de700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfe8188470 con 0x7fdfe81b8ad0 2021-02-10T11:31:01.134 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:01.130 7fdfee5de700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfe818be10 con 0x7fdfe80b2110 2021-02-10T11:31:01.134 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:01.130 7fdfee5de700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdfe818c030 con 0x7fdfe81b8ed0 2021-02-10T11:31:01.134 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:01.130 7fdfe7fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdfe81b8ad0 0x7fdfe8187e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:01.135 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:01.130 7fdfe7fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdfe81b8ad0 0x7fdfe8187e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:33322/0 (socket says 172.21.2.102:33322) 2021-02-10T11:31:01.135 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:01.130 7fdfe7fff700 1 -- 172.21.2.102:0/2588638823 learned_addr learned my addr 172.21.2.102:0/2588638823 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:31:01.135 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:01.131 7fdfe7fff700 1 -- 172.21.2.102:0/2588638823 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdfe81b8ed0 msgr2=0x7fdfe80b1af0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:31:01.135 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:01.131 7fdfe7fff700 1 --2- 172.21.2.102:0/2588638823 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdfe81b8ed0 0x7fdfe80b1af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:01.135 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:01.131 7fdfe7fff700 1 -- 172.21.2.102:0/2588638823 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fdfe80b2110 msgr2=0x7fdfe81b7a00 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:31:01.135 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:01.131 7fdfe7fff700 1 --2- 172.21.2.102:0/2588638823 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fdfe80b2110 0x7fdfe81b7a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:01.135 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:01.131 7fdfe7fff700 1 -- 172.21.2.102:0/2588638823 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfe80b2510 con 0x7fdfe81b8ad0 2021-02-10T11:31:01.135 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:01.131 7fdfe7fff700 1 --2- 172.21.2.102:0/2588638823 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdfe81b8ad0 0x7fdfe8187e50 secure :-1 s=READY pgs=2076 cs=0 l=1 rev1=1 rx=0x7fdfd8006f30 tx=0x7fdfd800b8e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:01.135 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:01.131 7fdfe4ff9700 1 -- 172.21.2.102:0/2588638823 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdfd8007070 con 0x7fdfe81b8ad0 2021-02-10T11:31:01.135 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:01.131 7fdfe4ff9700 1 -- 172.21.2.102:0/2588638823 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfd80073b0 con 0x7fdfe81b8ad0 2021-02-10T11:31:01.135 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:01.131 7fdfe4ff9700 1 -- 172.21.2.102:0/2588638823 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdfd80078d0 con 0x7fdfe81b8ad0 2021-02-10T11:31:01.136 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:01.131 7fdfee5de700 1 -- 172.21.2.102:0/2588638823 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfe80b2510 con 0x7fdfe81b8ad0 2021-02-10T11:31:01.136 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:01.131 7fdfee5de700 1 -- 172.21.2.102:0/2588638823 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfe81b8020 con 0x7fdfe81b8ad0 2021-02-10T11:31:01.136 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:01.132 7fdfe4ff9700 1 -- 172.21.2.102:0/2588638823 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fdfd8007000 con 0x7fdfe81b8ad0 2021-02-10T11:31:01.136 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:01.132 7fdfe4ff9700 1 --2- 172.21.2.102:0/2588638823 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fdfc80186c0 0x7fdfc801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:01.136 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:01.132 7fdfe4ff9700 1 -- 172.21.2.102:0/2588638823 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fdfd8032590 con 0x7fdfe81b8ad0 2021-02-10T11:31:01.136 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:01.132 7fdfe77fe700 1 --2- 172.21.2.102:0/2588638823 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fdfc80186c0 0x7fdfc801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:01.136 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:01.132 7fdfe77fe700 1 --2- 172.21.2.102:0/2588638823 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fdfc80186c0 0x7fdfc801abc0 crc :-1 s=READY pgs=923 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:01.164 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:01.163 7fdfee5de700 1 -- 172.21.2.102:0/2588638823 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdfe81b88b0 con 0x7fdfe81b8ad0 2021-02-10T11:31:01.166 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:01.165 7fdfe4ff9700 1 -- 172.21.2.102:0/2588638823 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fdfd8030d40 con 0x7fdfe81b8ad0 2021-02-10T11:31:01.290 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:01.289 7fdfee5de700 1 -- 172.21.2.102:0/2588638823 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdfe81b88b0 con 0x7fdfe81b8ad0 2021-02-10T11:31:01.291 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:01.290 7fdfe4ff9700 1 -- 172.21.2.102:0/2588638823 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v103) v1 ==== 74+0+14005 (secure 0 0 0) 0x7fdfd8007000 con 0x7fdfe81b8ad0 2021-02-10T11:31:01.291 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:31:01.292 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":103,"fsid":"68db30ea-dbce-4aef-a6df-f01c6fea0513","created":"2021-02-10 11:03:06.331557","modified":"2021-02-10 11:31:00.055039","last_up_change":"2021-02-10 11:03:32.690562","last_in_change":"2021-02-10 11:03:32.690562","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:03:37.322519","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:30:56.015905","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"101","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":21,"pool_name":"cephfs_data","create_time":"2021-02-10 11:30:57.042942","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"103","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"be691b4d-a039-408a-af80-9d53f02b6f1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6818","nonce":36088},{"type":"v1","addr":"172.21.2.102:6820","nonce":36088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6822","nonce":36088},{"type":"v1","addr":"172.21.2.102:6824","nonce":36088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6830","nonce":36088},{"type":"v1","addr":"172.21.2.102:6832","nonce":36088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6826","nonce":36088},{"type":"v1","addr":"172.21.2.102:6828","nonce":36088}]},"public_addr":"172.21.2.102:6820/36088","cluster_addr":"172.21.2.102:6824/36088","heartbeat_back_addr":"172.21.2.102:6832/36088","heartbeat_front_addr":"172.21.2.102:6828/36088","state":["exists","up"]},{"osd":1,"uuid":"64e3ccc8-9c69-4d7c-865e-af8e83822635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6819","nonce":36087},{"type":"v1","addr":"172.21.2.102:6821","nonce":36087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6823","nonce":36087},{"type":"v1","addr":"172.21.2.102:6825","nonce":36087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6831","nonce":36087},{"type":"v1","addr":"172.21.2.102:6833","nonce":36087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6827","nonce":36087},{"type":"v1","addr":"172.21.2.102:6829","nonce":36087}]},"public_addr":"172.21.2.102:6821/36087","cluster_addr":"172.21.2.102:6825/36087","heartbeat_back_addr":"172.21.2.102:6833/36087","heartbeat_front_addr":"172.21.2.102:6829/36087","state":["exists","up"]},{"osd":2,"uuid":"153700af-747f-481f-bdf4-d6072d3a842f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6810","nonce":36089},{"type":"v1","addr":"172.21.2.102:6811","nonce":36089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6812","nonce":36089},{"type":"v1","addr":"172.21.2.102:6813","nonce":36089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6816","nonce":36089},{"type":"v1","addr":"172.21.2.102:6817","nonce":36089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6814","nonce":36089},{"type":"v1","addr":"172.21.2.102:6815","nonce":36089}]},"public_addr":"172.21.2.102:6811/36089","cluster_addr":"172.21.2.102:6813/36089","heartbeat_back_addr":"172.21.2.102:6817/36089","heartbeat_front_addr":"172.21.2.102:6815/36089","state":["exists","up"]},{"osd":3,"uuid":"cbc678f4-83a9-45e6-8830-2f7c232b1ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6802","nonce":36090},{"type":"v1","addr":"172.21.2.102:6803","nonce":36090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6804","nonce":36090},{"type":"v1","addr":"172.21.2.102:6805","nonce":36090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6808","nonce":36090},{"type":"v1","addr":"172.21.2.102:6809","nonce":36090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6806","nonce":36090},{"type":"v1","addr":"172.21.2.102:6807","nonce":36090}]},"public_addr":"172.21.2.102:6803/36090","cluster_addr":"172.21.2.102:6805/36090","heartbeat_back_addr":"172.21.2.102:6809/36090","heartbeat_front_addr":"172.21.2.102:6807/36090","state":["exists","up"]},{"osd":4,"uuid":"c01e4008-b258-4f10-afd2-9e7f37214351","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6808","nonce":36368},{"type":"v1","addr":"172.21.2.113:6811","nonce":36368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6814","nonce":36368},{"type":"v1","addr":"172.21.2.113:6818","nonce":36368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6826","nonce":36368},{"type":"v1","addr":"172.21.2.113:6828","nonce":36368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6820","nonce":36368},{"type":"v1","addr":"172.21.2.113:6824","nonce":36368}]},"public_addr":"172.21.2.113:6811/36368","cluster_addr":"172.21.2.113:6818/36368","heartbeat_back_addr":"172.21.2.113:6828/36368","heartbeat_front_addr":"172.21.2.113:6824/36368","state":["exists","up"]},{"osd":5,"uuid":"551f6007-daf8-4a98-8ee0-c2b16bcccb5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6802","nonce":36366},{"type":"v1","addr":"172.21.2.113:6805","nonce":36366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6807","nonce":36366},{"type":"v1","addr":"172.21.2.113:6810","nonce":36366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6819","nonce":36366},{"type":"v1","addr":"172.21.2.113:6822","nonce":36366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6812","nonce":36366},{"type":"v1","addr":"172.21.2.113:6815","nonce":36366}]},"public_addr":"172.21.2.113:6805/36366","cluster_addr":"172.21.2.113:6810/36366","heartbeat_back_addr":"172.21.2.113:6822/36366","heartbeat_front_addr":"172.21.2.113:6815/36366","state":["exists","up"]},{"osd":6,"uuid":"bf67c040-0e50-48be-b102-c3943bc9ebef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6800","nonce":36367},{"type":"v1","addr":"172.21.2.113:6801","nonce":36367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6803","nonce":36367},{"type":"v1","addr":"172.21.2.113:6804","nonce":36367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6813","nonce":36367},{"type":"v1","addr":"172.21.2.113:6816","nonce":36367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6806","nonce":36367},{"type":"v1","addr":"172.21.2.113:6809","nonce":36367}]},"public_addr":"172.21.2.113:6801/36367","cluster_addr":"172.21.2.113:6804/36367","heartbeat_back_addr":"172.21.2.113:6816/36367","heartbeat_front_addr":"172.21.2.113:6809/36367","state":["exists","up"]},{"osd":7,"uuid":"f5461f80-018d-428f-be89-2eb2bcbe7bb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6817","nonce":36365},{"type":"v1","addr":"172.21.2.113:6821","nonce":36365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6823","nonce":36365},{"type":"v1","addr":"172.21.2.113:6825","nonce":36365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6830","nonce":36365},{"type":"v1","addr":"172.21.2.113:6831","nonce":36365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6827","nonce":36365},{"type":"v1","addr":"172.21.2.113:6829","nonce":36365}]},"public_addr":"172.21.2.113:6821/36365","cluster_addr":"172.21.2.113:6825/36365","heartbeat_back_addr":"172.21.2.113:6831/36365","heartbeat_front_addr":"172.21.2.113:6829/36365","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:31:01.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:01.292 7fdfee5de700 1 -- 172.21.2.102:0/2588638823 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fdfc80186c0 msgr2=0x7fdfc801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:01.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:01.292 7fdfee5de700 1 --2- 172.21.2.102:0/2588638823 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fdfc80186c0 0x7fdfc801abc0 crc :-1 s=READY pgs=923 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:01.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:01.292 7fdfee5de700 1 -- 172.21.2.102:0/2588638823 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdfe81b8ad0 msgr2=0x7fdfe8187e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:01.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:01.292 7fdfee5de700 1 --2- 172.21.2.102:0/2588638823 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdfe81b8ad0 0x7fdfe8187e50 secure :-1 s=READY pgs=2076 cs=0 l=1 rev1=1 rx=0x7fdfd8006f30 tx=0x7fdfd800b8e0).stop 2021-02-10T11:31:01.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:01.292 7fdfee5de700 1 -- 172.21.2.102:0/2588638823 shutdown_connections 2021-02-10T11:31:01.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:01.292 7fdfee5de700 1 --2- 172.21.2.102:0/2588638823 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fdfc80186c0 0x7fdfc801abc0 unknown :-1 s=CLOSED pgs=923 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:01.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:01.292 7fdfee5de700 1 --2- 172.21.2.102:0/2588638823 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fdfe80b2110 0x7fdfe81b7a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:01.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:01.292 7fdfee5de700 1 --2- 172.21.2.102:0/2588638823 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdfe81b8ed0 0x7fdfe80b1af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:01.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:01.292 7fdfee5de700 1 --2- 172.21.2.102:0/2588638823 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdfe81b8ad0 0x7fdfe8187e50 unknown :-1 s=CLOSED pgs=2076 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:01.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:01.292 7fdfee5de700 1 -- 172.21.2.102:0/2588638823 >> 172.21.2.102:0/2588638823 conn(0x7fdfe80ed810 msgr2=0x7fdfe8094980 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:01.294 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:01.292 7fdfee5de700 1 -- 172.21.2.102:0/2588638823 shutdown_connections 2021-02-10T11:31:01.294 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:01.292 7fdfee5de700 1 -- 172.21.2.102:0/2588638823 wait complete. 2021-02-10T11:31:01.302 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:31:01.742 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:01.725 7f47d94da700 1 Processor -- start 2021-02-10T11:31:01.742 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:01.725 7f47d94da700 1 -- start start 2021-02-10T11:31:01.742 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:01.725 7f47d94da700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f47d40d1880 0x7f47d41b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:01.742 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:01.725 7f47d94da700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f47d41b0af0 0x7f47d4108950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:01.743 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:01.725 7f47d94da700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f47d40d1c80 0x7f47d4108f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:01.743 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:01.726 7f47d94da700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47d40e5f00 con 0x7f47d41b0af0 2021-02-10T11:31:01.743 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:01.726 7f47d94da700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47d4109a70 con 0x7f47d40d1c80 2021-02-10T11:31:01.743 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:01.726 7f47d94da700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47d4088070 con 0x7f47d40d1880 2021-02-10T11:31:01.743 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:01.726 7f47d2abe700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f47d41b0af0 0x7f47d4108950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:01.743 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:01.726 7f47d2abe700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f47d41b0af0 0x7f47d4108950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:33330/0 (socket says 172.21.2.102:33330) 2021-02-10T11:31:01.743 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:01.726 7f47d2abe700 1 -- 172.21.2.102:0/4175225231 learned_addr learned my addr 172.21.2.102:0/4175225231 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:31:01.743 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:01.726 7f47d2abe700 1 -- 172.21.2.102:0/4175225231 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f47d40d1880 msgr2=0x7f47d41b04d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:31:01.743 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:01.726 7f47d2abe700 1 --2- 172.21.2.102:0/4175225231 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f47d40d1880 0x7f47d41b04d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:01.743 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:01.726 7f47d2abe700 1 -- 172.21.2.102:0/4175225231 >> v2:172.21.2.113:3300/0 conn(0x7f47d40d1c80 msgr2=0x7f47d4108f70 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:31:01.743 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:01.726 7f47d2abe700 1 --2- 172.21.2.102:0/4175225231 >> v2:172.21.2.113:3300/0 conn(0x7f47d40d1c80 0x7f47d4108f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:01.743 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:01.726 7f47d2abe700 1 -- 172.21.2.102:0/4175225231 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47d4109590 con 0x7f47d41b0af0 2021-02-10T11:31:01.743 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:01.726 7f47d2abe700 1 --2- 172.21.2.102:0/4175225231 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f47d41b0af0 0x7f47d4108950 secure :-1 s=READY pgs=2077 cs=0 l=1 rev1=1 rx=0x7f47bc006700 tx=0x7f47bc009b00).ready entity=mon.0 client_cookie=a948fbbcc7ecf4d2 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:01.744 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:01.726 7f47d22bd700 1 -- 172.21.2.102:0/4175225231 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f47bc00c0c0 con 0x7f47d41b0af0 2021-02-10T11:31:01.744 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:01.726 7f47d22bd700 1 -- 172.21.2.102:0/4175225231 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47bc00c340 con 0x7f47d41b0af0 2021-02-10T11:31:01.744 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:01.726 7f47d22bd700 1 -- 172.21.2.102:0/4175225231 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f47bc00c820 con 0x7f47d41b0af0 2021-02-10T11:31:01.744 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:01.727 7f47d94da700 1 -- 172.21.2.102:0/4175225231 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f47d41b0af0 msgr2=0x7f47d4108950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:01.744 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:01.727 7f47d94da700 1 --2- 172.21.2.102:0/4175225231 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f47d41b0af0 0x7f47d4108950 secure :-1 s=READY pgs=2077 cs=0 l=1 rev1=1 rx=0x7f47bc006700 tx=0x7f47bc009b00).stop 2021-02-10T11:31:01.744 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:01.727 7f47d94da700 1 -- 172.21.2.102:0/4175225231 shutdown_connections 2021-02-10T11:31:01.744 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:01.727 7f47d94da700 1 --2- 172.21.2.102:0/4175225231 >> v2:172.21.2.113:3300/0 conn(0x7f47d40d1c80 0x7f47d4108f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:01.744 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:01.727 7f47d94da700 1 --2- 172.21.2.102:0/4175225231 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f47d41b0af0 0x7f47d4108950 unknown :-1 s=CLOSED pgs=2077 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:01.744 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:01.727 7f47d94da700 1 --2- 172.21.2.102:0/4175225231 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f47d40d1880 0x7f47d41b04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:01.744 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:01.727 7f47d94da700 1 -- 172.21.2.102:0/4175225231 >> 172.21.2.102:0/4175225231 conn(0x7f47d40ce430 msgr2=0x7f47d40d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:01.744 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:01.727 7f47d94da700 1 -- 172.21.2.102:0/4175225231 shutdown_connections 2021-02-10T11:31:01.744 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:01.727 7f47d94da700 1 -- 172.21.2.102:0/4175225231 wait complete. 2021-02-10T11:31:01.746 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:01.741 7f47d94da700 1 Processor -- start 2021-02-10T11:31:01.746 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:01.742 7f47d94da700 1 -- start start 2021-02-10T11:31:01.746 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:01.742 7f47d94da700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f47d40aaa70 0x7f47d4198e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:01.746 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:01.742 7f47d94da700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f47d4199430 0x7f47d40946c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:01.746 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:01.742 7f47d94da700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f47d4094ce0 0x7f47d4196aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:01.746 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:01.742 7f47d94da700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47d4095550 con 0x7f47d40aaa70 2021-02-10T11:31:01.746 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:01.742 7f47d94da700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47d40950e0 con 0x7f47d4094ce0 2021-02-10T11:31:01.746 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:01.742 7f47d94da700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47d4095300 con 0x7f47d4199430 2021-02-10T11:31:01.746 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:01.742 7f47d32bf700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f47d40aaa70 0x7f47d4198e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:01.747 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:01.742 7f47d32bf700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f47d40aaa70 0x7f47d4198e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:33334/0 (socket says 172.21.2.102:33334) 2021-02-10T11:31:01.747 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:01.742 7f47d32bf700 1 -- 172.21.2.102:0/79239545 learned_addr learned my addr 172.21.2.102:0/79239545 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:31:01.747 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:01.743 7f47d32bf700 1 -- 172.21.2.102:0/79239545 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f47d4199430 msgr2=0x7f47d40946c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:01.747 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:01.743 7f47d32bf700 1 --2- 172.21.2.102:0/79239545 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f47d4199430 0x7f47d40946c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:01.747 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:01.743 7f47d32bf700 1 -- 172.21.2.102:0/79239545 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f47d4094ce0 msgr2=0x7f47d4196aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:01.747 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:01.743 7f47d32bf700 1 --2- 172.21.2.102:0/79239545 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f47d4094ce0 0x7f47d4196aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:01.747 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:01.743 7f47d32bf700 1 -- 172.21.2.102:0/79239545 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47d41970c0 con 0x7f47d40aaa70 2021-02-10T11:31:01.747 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:01.743 7f47d32bf700 1 --2- 172.21.2.102:0/79239545 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f47d40aaa70 0x7f47d4198e10 secure :-1 s=READY pgs=2078 cs=0 l=1 rev1=1 rx=0x7f47c400b5c0 tx=0x7f47c400b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:01.747 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:01.743 7f47cbfff700 1 -- 172.21.2.102:0/79239545 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f47c400bd50 con 0x7f47d40aaa70 2021-02-10T11:31:01.747 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:01.743 7f47cbfff700 1 -- 172.21.2.102:0/79239545 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47c4006030 con 0x7f47d40aaa70 2021-02-10T11:31:01.747 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:01.743 7f47cbfff700 1 -- 172.21.2.102:0/79239545 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f47c40054d0 con 0x7f47d40aaa70 2021-02-10T11:31:01.747 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:01.743 7f47d94da700 1 -- 172.21.2.102:0/79239545 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47d41970c0 con 0x7f47d40aaa70 2021-02-10T11:31:01.747 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:01.743 7f47d94da700 1 -- 172.21.2.102:0/79239545 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47d4197650 con 0x7f47d40aaa70 2021-02-10T11:31:01.748 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:01.744 7f47cbfff700 1 -- 172.21.2.102:0/79239545 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f47c4006210 con 0x7f47d40aaa70 2021-02-10T11:31:01.748 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:01.744 7f47cbfff700 1 --2- 172.21.2.102:0/79239545 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f47b80186c0 0x7f47b801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:01.748 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:01.744 7f47cbfff700 1 -- 172.21.2.102:0/79239545 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f47c4032400 con 0x7f47d40aaa70 2021-02-10T11:31:01.748 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:01.744 7f47d2abe700 1 --2- 172.21.2.102:0/79239545 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f47b80186c0 0x7f47b801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:01.748 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:01.744 7f47d2abe700 1 --2- 172.21.2.102:0/79239545 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f47b80186c0 0x7f47b801abc0 crc :-1 s=READY pgs=924 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:01.777 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:01.776 7f47d94da700 1 -- 172.21.2.102:0/79239545 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47d4197900 con 0x7f47d40aaa70 2021-02-10T11:31:01.779 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:01.778 7f47cbfff700 1 -- 172.21.2.102:0/79239545 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f47c4000960 con 0x7f47d40aaa70 2021-02-10T11:31:01.908 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:01.907 7f47d94da700 1 -- 172.21.2.102:0/79239545 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f47d4197900 con 0x7f47d40aaa70 2021-02-10T11:31:01.912 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:01.911 7f47cbfff700 1 -- 172.21.2.102:0/79239545 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.0 v59) v1 ==== 184+0+0 (secure 0 0 0) 0x7f47c4006210 con 0x7f47d40aaa70 2021-02-10T11:31:01.915 INFO:teuthology.orchestra.run.gibba002.stderr:updated caps for client.0 2021-02-10T11:31:01.917 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:01.916 7f47d94da700 1 -- 172.21.2.102:0/79239545 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f47b80186c0 msgr2=0x7f47b801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:01.917 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:01.916 7f47d94da700 1 --2- 172.21.2.102:0/79239545 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f47b80186c0 0x7f47b801abc0 crc :-1 s=READY pgs=924 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:01.917 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:01.916 7f47d94da700 1 -- 172.21.2.102:0/79239545 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f47d40aaa70 msgr2=0x7f47d4198e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:01.917 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:01.916 7f47d94da700 1 --2- 172.21.2.102:0/79239545 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f47d40aaa70 0x7f47d4198e10 secure :-1 s=READY pgs=2078 cs=0 l=1 rev1=1 rx=0x7f47c400b5c0 tx=0x7f47c400b960).stop 2021-02-10T11:31:01.917 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:01.916 7f47d94da700 1 -- 172.21.2.102:0/79239545 shutdown_connections 2021-02-10T11:31:01.917 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:01.916 7f47d94da700 1 --2- 172.21.2.102:0/79239545 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f47b80186c0 0x7f47b801abc0 unknown :-1 s=CLOSED pgs=924 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:01.917 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:01.916 7f47d94da700 1 --2- 172.21.2.102:0/79239545 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f47d4094ce0 0x7f47d4196aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:01.917 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:01.916 7f47d94da700 1 --2- 172.21.2.102:0/79239545 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f47d4199430 0x7f47d40946c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:01.917 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:01.916 7f47d94da700 1 --2- 172.21.2.102:0/79239545 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f47d40aaa70 0x7f47d4198e10 unknown :-1 s=CLOSED pgs=2078 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:01.917 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:01.916 7f47d94da700 1 -- 172.21.2.102:0/79239545 >> 172.21.2.102:0/79239545 conn(0x7f47d41b9600 msgr2=0x7f47d418c290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:01.918 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:01.916 7f47d94da700 1 -- 172.21.2.102:0/79239545 shutdown_connections 2021-02-10T11:31:01.918 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:01.916 7f47d94da700 1 -- 172.21.2.102:0/79239545 wait complete. 2021-02-10T11:31:01.925 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.1 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:31:02.358 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:02.340 7fc07f275700 1 Processor -- start 2021-02-10T11:31:02.358 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:02.340 7fc07f275700 1 -- start start 2021-02-10T11:31:02.358 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:02.340 7fc07f275700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fc0780d1c60 0x7fc0780cef50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:02.358 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:02.340 7fc07f275700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fc0780cf570 0x7fc0780cf970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:02.358 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:02.340 7fc07f275700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0780fe1b0 con 0x7fc0780d1860 2021-02-10T11:31:02.358 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:02.340 7fc07f275700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0781b0ca0 con 0x7fc0780d1c60 2021-02-10T11:31:02.358 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:02.340 7fc07f275700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc078109a50 con 0x7fc0780cf570 2021-02-10T11:31:02.358 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:02.341 7fc07c890700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fc0780d1c60 0x7fc0780cef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:02.358 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:02.341 7fc07c890700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fc0780d1c60 0x7fc0780cef50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:33342/0 (socket says 172.21.2.102:33342) 2021-02-10T11:31:02.358 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:02.341 7fc07c890700 1 -- 172.21.2.102:0/3136661927 learned_addr learned my addr 172.21.2.102:0/3136661927 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:31:02.359 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:02.341 7fc07d091700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fc0780d1860 0x7fc0781b04b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:53800/0 (socket says 172.21.2.102:53800) 2021-02-10T11:31:02.359 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:02.341 7fc07c890700 1 -- 172.21.2.102:0/3136661927 >> v2:172.21.2.113:3300/0 conn(0x7fc0780cf570 msgr2=0x7fc0780cf970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:02.359 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:02.341 7fc07c890700 1 --2- 172.21.2.102:0/3136661927 >> v2:172.21.2.113:3300/0 conn(0x7fc0780cf570 0x7fc0780cf970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:02.359 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:02.341 7fc07c890700 1 -- 172.21.2.102:0/3136661927 >> v1:172.21.2.102:6789/0 conn(0x7fc0780d1860 legacy=0x7fc0781b04b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:02.359 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:02.341 7fc07c890700 1 -- 172.21.2.102:0/3136661927 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc078088070 con 0x7fc0780d1c60 2021-02-10T11:31:02.359 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:02.341 7fc07c890700 1 --2- 172.21.2.102:0/3136661927 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc0780d1c60 0x7fc0780cef50 secure :-1 s=READY pgs=2080 cs=0 l=1 rev1=1 rx=0x7fc068006700 tx=0x7fc068009b80).ready entity=mon.0 client_cookie=5ea94ba6fea7baf8 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:02.359 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:02.342 7fc06ffff700 1 -- 172.21.2.102:0/3136661927 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc06800c0c0 con 0x7fc0780d1c60 2021-02-10T11:31:02.359 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:02.342 7fc06ffff700 1 -- 172.21.2.102:0/3136661927 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc06800c400 con 0x7fc0780d1c60 2021-02-10T11:31:02.359 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:02.342 7fc06ffff700 1 -- 172.21.2.102:0/3136661927 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc06800c860 con 0x7fc0780d1c60 2021-02-10T11:31:02.359 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:02.342 7fc07f275700 1 -- 172.21.2.102:0/3136661927 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc0780d1c60 msgr2=0x7fc0780cef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:02.359 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:02.342 7fc07f275700 1 --2- 172.21.2.102:0/3136661927 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc0780d1c60 0x7fc0780cef50 secure :-1 s=READY pgs=2080 cs=0 l=1 rev1=1 rx=0x7fc068006700 tx=0x7fc068009b80).stop 2021-02-10T11:31:02.359 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:02.342 7fc07f275700 1 -- 172.21.2.102:0/3136661927 shutdown_connections 2021-02-10T11:31:02.360 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:02.342 7fc07f275700 1 --2- 172.21.2.102:0/3136661927 >> v2:172.21.2.113:3300/0 conn(0x7fc0780cf570 0x7fc0780cf970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:02.360 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:02.342 7fc07f275700 1 --2- 172.21.2.102:0/3136661927 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc0780d1c60 0x7fc0780cef50 unknown :-1 s=CLOSED pgs=2080 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:02.360 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:02.342 7fc07f275700 1 -- 172.21.2.102:0/3136661927 >> 172.21.2.102:0/3136661927 conn(0x7fc0780ce410 msgr2=0x7fc0780d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:02.360 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:02.343 7fc07f275700 1 -- 172.21.2.102:0/3136661927 shutdown_connections 2021-02-10T11:31:02.360 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:02.343 7fc07f275700 1 -- 172.21.2.102:0/3136661927 wait complete. 2021-02-10T11:31:02.361 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:02.357 7fc07f275700 1 Processor -- start 2021-02-10T11:31:02.362 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:02.357 7fc07f275700 1 -- start start 2021-02-10T11:31:02.362 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:02.358 7fc07f275700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc078093cc0 0x7fc07819d700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:02.362 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:02.358 7fc07f275700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc07819e5b0 0x7fc0780aa420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:02.362 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:02.358 7fc07f275700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc0780aaa40 0x7fc078095660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:02.362 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:02.358 7fc07f275700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc07819dd20 con 0x7fc07819e5b0 2021-02-10T11:31:02.362 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:02.358 7fc07f275700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0780940c0 con 0x7fc0780aaa40 2021-02-10T11:31:02.362 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:02.358 7fc07f275700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0780943f0 con 0x7fc078093cc0 2021-02-10T11:31:02.362 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:02.358 7fc07c890700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc07819e5b0 0x7fc0780aa420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:02.362 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:02.358 7fc07c890700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc07819e5b0 0x7fc0780aa420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:33352/0 (socket says 172.21.2.102:33352) 2021-02-10T11:31:02.362 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:02.358 7fc07c890700 1 -- 172.21.2.102:0/3614819692 learned_addr learned my addr 172.21.2.102:0/3614819692 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:31:02.362 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:02.358 7fc07c890700 1 -- 172.21.2.102:0/3614819692 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc078093cc0 msgr2=0x7fc07819d700 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:31:02.362 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:02.358 7fc07c890700 1 --2- 172.21.2.102:0/3614819692 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc078093cc0 0x7fc07819d700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:02.363 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:02.358 7fc07c890700 1 -- 172.21.2.102:0/3614819692 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc0780aaa40 msgr2=0x7fc078095660 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:31:02.363 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:02.358 7fc07c890700 1 --2- 172.21.2.102:0/3614819692 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc0780aaa40 0x7fc078095660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:02.363 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:02.358 7fc07c890700 1 -- 172.21.2.102:0/3614819692 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0781b9520 con 0x7fc07819e5b0 2021-02-10T11:31:02.363 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:02.359 7fc07c890700 1 --2- 172.21.2.102:0/3614819692 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc07819e5b0 0x7fc0780aa420 secure :-1 s=READY pgs=2081 cs=0 l=1 rev1=1 rx=0x7fc06800b580 tx=0x7fc06800b920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:02.363 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:02.359 7fc06dffb700 1 -- 172.21.2.102:0/3614819692 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc06800bd10 con 0x7fc07819e5b0 2021-02-10T11:31:02.363 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:02.359 7fc06dffb700 1 -- 172.21.2.102:0/3614819692 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc068002030 con 0x7fc07819e5b0 2021-02-10T11:31:02.363 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:02.359 7fc06dffb700 1 -- 172.21.2.102:0/3614819692 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc068002550 con 0x7fc07819e5b0 2021-02-10T11:31:02.363 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:02.359 7fc07f275700 1 -- 172.21.2.102:0/3614819692 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0781b9520 con 0x7fc07819e5b0 2021-02-10T11:31:02.363 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:02.359 7fc07f275700 1 -- 172.21.2.102:0/3614819692 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0781b9520 con 0x7fc07819e5b0 2021-02-10T11:31:02.363 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:02.359 7fc06dffb700 1 -- 172.21.2.102:0/3614819692 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fc068000c90 con 0x7fc07819e5b0 2021-02-10T11:31:02.363 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:02.360 7fc06dffb700 1 --2- 172.21.2.102:0/3614819692 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc0580186c0 0x7fc05801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:02.363 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:02.360 7fc06dffb700 1 -- 172.21.2.102:0/3614819692 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fc068032830 con 0x7fc07819e5b0 2021-02-10T11:31:02.364 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:02.360 7fc07d091700 1 --2- 172.21.2.102:0/3614819692 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc0580186c0 0x7fc05801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:02.364 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:02.360 7fc07d091700 1 --2- 172.21.2.102:0/3614819692 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc0580186c0 0x7fc05801abc0 crc :-1 s=READY pgs=925 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:02.391 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:02.390 7fc07f275700 1 -- 172.21.2.102:0/3614819692 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc07819eca0 con 0x7fc07819e5b0 2021-02-10T11:31:02.394 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:02.393 7fc06dffb700 1 -- 172.21.2.102:0/3614819692 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fc06801f570 con 0x7fc07819e5b0 2021-02-10T11:31:02.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:02.524 7fc07f275700 1 -- 172.21.2.102:0/3614819692 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7fc07819eca0 con 0x7fc07819e5b0 2021-02-10T11:31:02.528 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:02.527 7fc06dffb700 1 -- 172.21.2.102:0/3614819692 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.1 v60) v1 ==== 184+0+0 (secure 0 0 0) 0x7fc06801f780 con 0x7fc07819e5b0 2021-02-10T11:31:02.532 INFO:teuthology.orchestra.run.gibba002.stderr:updated caps for client.1 2021-02-10T11:31:02.533 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:02.532 7fc07f275700 1 -- 172.21.2.102:0/3614819692 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc0580186c0 msgr2=0x7fc05801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:02.533 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:02.532 7fc07f275700 1 --2- 172.21.2.102:0/3614819692 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc0580186c0 0x7fc05801abc0 crc :-1 s=READY pgs=925 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:02.533 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:02.532 7fc07f275700 1 -- 172.21.2.102:0/3614819692 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc07819e5b0 msgr2=0x7fc0780aa420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:02.534 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:02.532 7fc07f275700 1 --2- 172.21.2.102:0/3614819692 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc07819e5b0 0x7fc0780aa420 secure :-1 s=READY pgs=2081 cs=0 l=1 rev1=1 rx=0x7fc06800b580 tx=0x7fc06800b920).stop 2021-02-10T11:31:02.534 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:02.533 7fc07f275700 1 -- 172.21.2.102:0/3614819692 shutdown_connections 2021-02-10T11:31:02.534 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:02.533 7fc07f275700 1 --2- 172.21.2.102:0/3614819692 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc0580186c0 0x7fc05801abc0 unknown :-1 s=CLOSED pgs=925 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:02.534 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:02.533 7fc07f275700 1 --2- 172.21.2.102:0/3614819692 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc0780aaa40 0x7fc078095660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:02.535 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:02.533 7fc07f275700 1 --2- 172.21.2.102:0/3614819692 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc07819e5b0 0x7fc0780aa420 unknown :-1 s=CLOSED pgs=2081 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:02.535 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:02.533 7fc07f275700 1 --2- 172.21.2.102:0/3614819692 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc078093cc0 0x7fc07819d700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:02.535 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:02.533 7fc07f275700 1 -- 172.21.2.102:0/3614819692 >> 172.21.2.102:0/3614819692 conn(0x7fc07819f820 msgr2=0x7fc07818c270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:02.535 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:02.533 7fc07f275700 1 -- 172.21.2.102:0/3614819692 shutdown_connections 2021-02-10T11:31:02.535 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:02.533 7fc07f275700 1 -- 172.21.2.102:0/3614819692 wait complete. 2021-02-10T11:31:02.542 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.2 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:31:02.983 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:02.964 7f341d0b2700 1 Processor -- start 2021-02-10T11:31:02.983 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:02.964 7f341d0b2700 1 -- start start 2021-02-10T11:31:02.984 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:02.965 7f341d0b2700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f341810b000 0x7f34180cf760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:02.984 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:02.965 7f341d0b2700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34180fe1d0 con 0x7f34180d1c80 2021-02-10T11:31:02.984 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:02.965 7f341d0b2700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34181b0cc0 con 0x7f34180d1880 2021-02-10T11:31:02.984 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:02.965 7f341d0b2700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3418109a70 con 0x7f341810b000 2021-02-10T11:31:02.984 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:02.965 7f341659c700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f34180d1c80 0x7f34180cef70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:53814/0 (socket says 172.21.2.102:53814) 2021-02-10T11:31:02.984 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:02.965 7f341659c700 1 -- 172.21.2.102:0/615462406 learned_addr learned my addr 172.21.2.102:0/615462406 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:31:02.984 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:02.965 7f3415d9b700 1 -- 172.21.2.102:0/615462406 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2139184794 0 0) 0x7f340c000ce0 con 0x7f34180d1c80 2021-02-10T11:31:02.984 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:02.966 7f3415d9b700 1 -- 172.21.2.102:0/615462406 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3400002fe0 con 0x7f34180d1c80 2021-02-10T11:31:02.984 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:02.966 7f341759e700 1 --2- 172.21.2.102:0/615462406 >> v2:172.21.2.113:3300/0 conn(0x7f341810b000 0x7f34180cf760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:02.984 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:02.966 7f3415d9b700 1 -- 172.21.2.102:0/615462406 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1561518933 0 0) 0x7f340c001180 con 0x7f34180d1c80 2021-02-10T11:31:02.984 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:02.966 7f3415d9b700 1 -- 172.21.2.102:0/615462406 >> v2:172.21.2.113:3300/0 conn(0x7f341810b000 msgr2=0x7f34180cf760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:02.985 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:02.966 7f3415d9b700 1 --2- 172.21.2.102:0/615462406 >> v2:172.21.2.113:3300/0 conn(0x7f341810b000 0x7f34180cf760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:02.985 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:02.966 7f3415d9b700 1 -- 172.21.2.102:0/615462406 >> v1:172.21.2.113:6789/0 conn(0x7f34180d1880 legacy=0x7f34181b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:02.985 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:02.966 7f3415d9b700 1 -- 172.21.2.102:0/615462406 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f341810b400 con 0x7f34180d1c80 2021-02-10T11:31:02.985 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:02.966 7f3415d9b700 1 -- 172.21.2.102:0/615462406 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7f340c001680 con 0x7f34180d1c80 2021-02-10T11:31:02.985 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:02.966 7f3415d9b700 1 -- 172.21.2.102:0/615462406 >> v1:172.21.2.102:6789/0 conn(0x7f34180d1c80 legacy=0x7f34180cef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:02.985 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:02.966 7f3415d9b700 1 --2- 172.21.2.102:0/615462406 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3400003aa0 0x7f3400005ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:02.985 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:02.966 7f3415d9b700 1 --2- 172.21.2.102:0/615462406 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3400006790 0x7f3400008c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:02.985 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:02.966 7f3415d9b700 1 --2- 172.21.2.102:0/615462406 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3400009590 0x7f340000bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:02.985 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:02.966 7f3415d9b700 1 -- 172.21.2.102:0/615462406 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f340000c300 con 0x7f3400003aa0 2021-02-10T11:31:02.985 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:02.966 7f3415d9b700 1 -- 172.21.2.102:0/615462406 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f340000c520 con 0x7f3400006790 2021-02-10T11:31:02.985 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:02.966 7f3415d9b700 1 -- 172.21.2.102:0/615462406 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f340000c740 con 0x7f3400009590 2021-02-10T11:31:02.985 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:02.966 7f3416d9d700 1 --2- 172.21.2.102:0/615462406 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3400003aa0 0x7f3400005ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:02.985 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:02.966 7f341659c700 1 --2- 172.21.2.102:0/615462406 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3400006790 0x7f3400008c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:02.986 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:02.966 7f341759e700 1 --2- 172.21.2.102:0/615462406 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3400009590 0x7f340000bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:02.986 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:02.966 7f3416d9d700 1 -- 172.21.2.102:0/615462406 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3400009590 msgr2=0x7f340000bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:02.986 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:02.966 7f3416d9d700 1 --2- 172.21.2.102:0/615462406 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3400009590 0x7f340000bab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:02.986 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:02.966 7f3416d9d700 1 -- 172.21.2.102:0/615462406 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3400006790 msgr2=0x7f3400008c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:02.986 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:02.966 7f3416d9d700 1 --2- 172.21.2.102:0/615462406 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3400006790 0x7f3400008c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:02.986 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:02.966 7f3416d9d700 1 -- 172.21.2.102:0/615462406 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3400050980 con 0x7f3400003aa0 2021-02-10T11:31:02.986 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:02.967 7f341759e700 1 --2- 172.21.2.102:0/615462406 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3400009590 0x7f340000bab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:31:02.986 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:02.967 7f341759e700 1 -- 172.21.2.102:0/615462406 reap_dead start 2021-02-10T11:31:02.986 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:02.967 7f341659c700 1 --2- 172.21.2.102:0/615462406 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3400006790 0x7f3400008c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:31:02.986 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:02.967 7f3416d9d700 1 --2- 172.21.2.102:0/615462406 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3400003aa0 0x7f3400005ef0 secure :-1 s=READY pgs=2083 cs=0 l=1 rev1=1 rx=0x7f3408000e30 tx=0x7f3408009aa0).ready entity=mon.0 client_cookie=c2d0d8012eea4b9c server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:02.986 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:02.967 7f3415d9b700 1 -- 172.21.2.102:0/615462406 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f34080066e0 con 0x7f3400003aa0 2021-02-10T11:31:02.986 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:02.967 7f3415d9b700 1 -- 172.21.2.102:0/615462406 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f340800c0c0 con 0x7f3400003aa0 2021-02-10T11:31:02.987 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:02.967 7f3415d9b700 1 -- 172.21.2.102:0/615462406 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f340800c560 con 0x7f3400003aa0 2021-02-10T11:31:02.987 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:02.967 7f341d0b2700 1 -- 172.21.2.102:0/615462406 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3400003aa0 msgr2=0x7f3400005ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:02.987 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:02.967 7f341d0b2700 1 --2- 172.21.2.102:0/615462406 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3400003aa0 0x7f3400005ef0 secure :-1 s=READY pgs=2083 cs=0 l=1 rev1=1 rx=0x7f3408000e30 tx=0x7f3408009aa0).stop 2021-02-10T11:31:02.987 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:02.968 7f341d0b2700 1 -- 172.21.2.102:0/615462406 shutdown_connections 2021-02-10T11:31:02.987 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:02.968 7f341d0b2700 1 --2- 172.21.2.102:0/615462406 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3400003aa0 0x7f3400005ef0 unknown :-1 s=CLOSED pgs=2083 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:02.987 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:02.968 7f341d0b2700 1 -- 172.21.2.102:0/615462406 >> 172.21.2.102:0/615462406 conn(0x7f34180ce430 msgr2=0x7f34180d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:02.987 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:02.968 7f341d0b2700 1 -- 172.21.2.102:0/615462406 shutdown_connections 2021-02-10T11:31:02.987 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:02.968 7f341d0b2700 1 -- 172.21.2.102:0/615462406 wait complete. 2021-02-10T11:31:02.987 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:02.983 7f341d0b2700 1 Processor -- start 2021-02-10T11:31:02.987 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:02.983 7f341d0b2700 1 -- start start 2021-02-10T11:31:02.988 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:02.983 7f341d0b2700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f34180abd30 0x7f3418082bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:02.989 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:02.983 7f341d0b2700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f34180831d0 0x7f341808a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:02.989 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:02.983 7f341d0b2700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f34180ac130 0x7f341808aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:02.989 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:02.983 7f341d0b2700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3418083700 con 0x7f34180831d0 2021-02-10T11:31:02.989 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:02.983 7f341d0b2700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f341808b0c0 con 0x7f34180ac130 2021-02-10T11:31:02.989 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:02.983 7f341d0b2700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f341808b2a0 con 0x7f34180abd30 2021-02-10T11:31:02.989 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:02.983 7f341659c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f34180831d0 0x7f341808a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:02.989 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:02.983 7f341659c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f34180831d0 0x7f341808a480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:33370/0 (socket says 172.21.2.102:33370) 2021-02-10T11:31:02.989 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:02.983 7f341659c700 1 -- 172.21.2.102:0/13069253 learned_addr learned my addr 172.21.2.102:0/13069253 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:31:02.989 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:02.983 7f341659c700 1 -- 172.21.2.102:0/13069253 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f34180abd30 msgr2=0x7f3418082bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:02.989 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:02.983 7f341659c700 1 --2- 172.21.2.102:0/13069253 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f34180abd30 0x7f3418082bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:02.989 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:02.983 7f341659c700 1 -- 172.21.2.102:0/13069253 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f34180ac130 msgr2=0x7f341808aaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:02.989 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:02.983 7f341659c700 1 --2- 172.21.2.102:0/13069253 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f34180ac130 0x7f341808aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:02.989 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:02.983 7f341659c700 1 -- 172.21.2.102:0/13069253 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f341808b5f0 con 0x7f34180831d0 2021-02-10T11:31:02.990 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:02.984 7f341659c700 1 --2- 172.21.2.102:0/13069253 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f34180831d0 0x7f341808a480 secure :-1 s=READY pgs=2084 cs=0 l=1 rev1=1 rx=0x7f340c00a520 tx=0x7f340c00a8c0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:02.990 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:02.984 7f34077fe700 1 -- 172.21.2.102:0/13069253 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f340c000d10 con 0x7f34180831d0 2021-02-10T11:31:02.990 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:02.984 7f34077fe700 1 -- 172.21.2.102:0/13069253 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f340c0043a0 con 0x7f34180831d0 2021-02-10T11:31:02.990 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:02.984 7f341d0b2700 1 -- 172.21.2.102:0/13069253 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f341808b5f0 con 0x7f34180831d0 2021-02-10T11:31:02.990 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:02.984 7f34077fe700 1 -- 172.21.2.102:0/13069253 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f340c004800 con 0x7f34180831d0 2021-02-10T11:31:02.990 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:02.984 7f341d0b2700 1 -- 172.21.2.102:0/13069253 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f341808b5f0 con 0x7f34180831d0 2021-02-10T11:31:02.990 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:02.984 7f34077fe700 1 -- 172.21.2.102:0/13069253 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f340c000c90 con 0x7f34180831d0 2021-02-10T11:31:02.990 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:02.984 7f34077fe700 1 --2- 172.21.2.102:0/13069253 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f33fc018730 0x7f33fc01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:02.990 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:02.984 7f34077fe700 1 -- 172.21.2.102:0/13069253 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f340c032630 con 0x7f34180831d0 2021-02-10T11:31:02.990 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:02.984 7f3416d9d700 1 --2- 172.21.2.102:0/13069253 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f33fc018730 0x7f33fc01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:02.990 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:02.985 7f3416d9d700 1 --2- 172.21.2.102:0/13069253 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f33fc018730 0x7f33fc01abc0 crc :-1 s=READY pgs=926 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:03.016 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:03.015 7f341d0b2700 1 -- 172.21.2.102:0/13069253 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3418082980 con 0x7f34180831d0 2021-02-10T11:31:03.018 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:03.017 7f34077fe700 1 -- 172.21.2.102:0/13069253 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f340c004f00 con 0x7f34180831d0 2021-02-10T11:31:03.150 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:03.149 7f341d0b2700 1 -- 172.21.2.102:0/13069253 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f3418082980 con 0x7f34180831d0 2021-02-10T11:31:03.154 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:03.153 7f34077fe700 1 -- 172.21.2.102:0/13069253 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.2 v61) v1 ==== 184+0+0 (secure 0 0 0) 0x7f340c007000 con 0x7f34180831d0 2021-02-10T11:31:03.158 INFO:teuthology.orchestra.run.gibba002.stderr:updated caps for client.2 2021-02-10T11:31:03.159 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:03.158 7f341d0b2700 1 -- 172.21.2.102:0/13069253 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f33fc018730 msgr2=0x7f33fc01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:03.159 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:03.158 7f341d0b2700 1 --2- 172.21.2.102:0/13069253 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f33fc018730 0x7f33fc01abc0 crc :-1 s=READY pgs=926 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:03.159 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:03.158 7f341d0b2700 1 -- 172.21.2.102:0/13069253 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f34180831d0 msgr2=0x7f341808a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:03.159 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:03.158 7f341d0b2700 1 --2- 172.21.2.102:0/13069253 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f34180831d0 0x7f341808a480 secure :-1 s=READY pgs=2084 cs=0 l=1 rev1=1 rx=0x7f340c00a520 tx=0x7f340c00a8c0).stop 2021-02-10T11:31:03.160 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:03.158 7f341d0b2700 1 -- 172.21.2.102:0/13069253 shutdown_connections 2021-02-10T11:31:03.160 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:03.158 7f341d0b2700 1 --2- 172.21.2.102:0/13069253 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f33fc018730 0x7f33fc01abc0 unknown :-1 s=CLOSED pgs=926 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:03.160 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:03.158 7f341d0b2700 1 --2- 172.21.2.102:0/13069253 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f34180ac130 0x7f341808aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:03.160 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:03.158 7f341d0b2700 1 --2- 172.21.2.102:0/13069253 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f34180831d0 0x7f341808a480 unknown :-1 s=CLOSED pgs=2084 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:03.160 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:03.158 7f341d0b2700 1 --2- 172.21.2.102:0/13069253 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f34180abd30 0x7f3418082bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:03.160 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:03.158 7f341d0b2700 1 -- 172.21.2.102:0/13069253 >> 172.21.2.102:0/13069253 conn(0x7f34180afc60 msgr2=0x7f34180b50d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:03.160 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:03.159 7f341d0b2700 1 -- 172.21.2.102:0/13069253 shutdown_connections 2021-02-10T11:31:03.160 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:03.159 7f341d0b2700 1 -- 172.21.2.102:0/13069253 wait complete. 2021-02-10T11:31:03.168 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.3 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:31:03.608 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:03.590 7faba60e6700 1 Processor -- start 2021-02-10T11:31:03.608 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:03.590 7faba60e6700 1 -- start start 2021-02-10T11:31:03.608 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:03.591 7faba60e6700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7faba00d1860 0x7faba01b04b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:03.608 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:03.591 7faba60e6700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7faba01b0ad0 0x7faba0108930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:03.608 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:03.591 7faba60e6700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7faba00d1c60 0x7faba0108f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:03.608 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:03.591 7faba60e6700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faba00fe1b0 con 0x7faba00d1c60 2021-02-10T11:31:03.608 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:03.591 7faba60e6700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faba0109a50 con 0x7faba00d1860 2021-02-10T11:31:03.608 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:03.591 7faba60e6700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faba0088070 con 0x7faba01b0ad0 2021-02-10T11:31:03.608 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:03.591 7fab9ffff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7faba00d1c60 0x7faba0108f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:03.608 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:03.591 7fab9ffff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7faba00d1c60 0x7faba0108f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:33376/0 (socket says 172.21.2.102:33376) 2021-02-10T11:31:03.608 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:03.591 7fab9ffff700 1 -- 172.21.2.102:0/589153338 learned_addr learned my addr 172.21.2.102:0/589153338 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:31:03.609 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:03.591 7fab9ffff700 1 -- 172.21.2.102:0/589153338 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7faba01b0ad0 msgr2=0x7faba0108930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:03.609 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:03.591 7fab9ffff700 1 --2- 172.21.2.102:0/589153338 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7faba01b0ad0 0x7faba0108930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:03.609 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:03.591 7fab9ffff700 1 -- 172.21.2.102:0/589153338 >> v2:172.21.2.113:3300/0 conn(0x7faba00d1860 msgr2=0x7faba01b04b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:03.609 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:03.591 7fab9ffff700 1 --2- 172.21.2.102:0/589153338 >> v2:172.21.2.113:3300/0 conn(0x7faba00d1860 0x7faba01b04b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:03.609 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:03.591 7fab9ffff700 1 -- 172.21.2.102:0/589153338 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faba0109570 con 0x7faba00d1c60 2021-02-10T11:31:03.609 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:03.591 7fab9ffff700 1 --2- 172.21.2.102:0/589153338 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7faba00d1c60 0x7faba0108f50 secure :-1 s=READY pgs=2085 cs=0 l=1 rev1=1 rx=0x7fab98006f00 tx=0x7fab98009f10).ready entity=mon.0 client_cookie=66ac2ddc493e8727 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:03.609 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:03.592 7fab9e7fc700 1 -- 172.21.2.102:0/589153338 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fab9800b3b0 con 0x7faba00d1c60 2021-02-10T11:31:03.609 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:03.592 7fab9e7fc700 1 -- 172.21.2.102:0/589153338 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab9800b730 con 0x7faba00d1c60 2021-02-10T11:31:03.609 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:03.592 7fab9e7fc700 1 -- 172.21.2.102:0/589153338 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fab9800bc50 con 0x7faba00d1c60 2021-02-10T11:31:03.609 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:03.592 7faba60e6700 1 -- 172.21.2.102:0/589153338 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7faba00d1c60 msgr2=0x7faba0108f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:03.609 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:03.592 7faba60e6700 1 --2- 172.21.2.102:0/589153338 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7faba00d1c60 0x7faba0108f50 secure :-1 s=READY pgs=2085 cs=0 l=1 rev1=1 rx=0x7fab98006f00 tx=0x7fab98009f10).stop 2021-02-10T11:31:03.609 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:03.592 7faba60e6700 1 -- 172.21.2.102:0/589153338 shutdown_connections 2021-02-10T11:31:03.609 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:03.592 7faba60e6700 1 --2- 172.21.2.102:0/589153338 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7faba01b0ad0 0x7faba0108930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:03.610 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:03.592 7faba60e6700 1 --2- 172.21.2.102:0/589153338 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7faba00d1c60 0x7faba0108f50 unknown :-1 s=CLOSED pgs=2085 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:03.610 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:03.592 7faba60e6700 1 --2- 172.21.2.102:0/589153338 >> v2:172.21.2.113:3300/0 conn(0x7faba00d1860 0x7faba01b04b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:03.610 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:03.592 7faba60e6700 1 -- 172.21.2.102:0/589153338 >> 172.21.2.102:0/589153338 conn(0x7faba00ce410 msgr2=0x7faba00d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:03.610 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:03.592 7faba60e6700 1 -- 172.21.2.102:0/589153338 shutdown_connections 2021-02-10T11:31:03.610 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:03.592 7faba60e6700 1 -- 172.21.2.102:0/589153338 wait complete. 2021-02-10T11:31:03.610 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:03.607 7faba60e6700 1 Processor -- start 2021-02-10T11:31:03.610 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:03.607 7faba60e6700 1 -- start start 2021-02-10T11:31:03.611 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:03.607 7faba60e6700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7faba00ae860 0x7faba00aec60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:03.611 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:03.607 7faba60e6700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7faba008d8c0 0x7faba008dcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:03.612 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:03.607 7faba60e6700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7faba008e2e0 0x7faba008ef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:03.612 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:03.607 7faba60e6700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faba008e760 con 0x7faba00ae860 2021-02-10T11:31:03.612 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:03.607 7faba60e6700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faba008b3e0 con 0x7faba008d8c0 2021-02-10T11:31:03.612 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:03.607 7faba60e6700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faba0192c40 con 0x7faba008e2e0 2021-02-10T11:31:03.612 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:03.607 7fab9f7fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7faba00ae860 0x7faba00aec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:03.612 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:03.607 7fab9f7fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7faba00ae860 0x7faba00aec60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:33384/0 (socket says 172.21.2.102:33384) 2021-02-10T11:31:03.612 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:03.607 7fab9f7fe700 1 -- 172.21.2.102:0/3228769242 learned_addr learned my addr 172.21.2.102:0/3228769242 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:31:03.612 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:03.608 7fab9f7fe700 1 -- 172.21.2.102:0/3228769242 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7faba008e2e0 msgr2=0x7faba008ef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:03.612 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:03.608 7fab9f7fe700 1 --2- 172.21.2.102:0/3228769242 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7faba008e2e0 0x7faba008ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:03.612 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:03.608 7fab9f7fe700 1 -- 172.21.2.102:0/3228769242 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7faba008d8c0 msgr2=0x7faba008dcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:03.612 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:03.608 7fab9f7fe700 1 --2- 172.21.2.102:0/3228769242 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7faba008d8c0 0x7faba008dcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:03.612 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:03.608 7fab9f7fe700 1 -- 172.21.2.102:0/3228769242 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faba008f530 con 0x7faba00ae860 2021-02-10T11:31:03.613 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:03.608 7fab9f7fe700 1 --2- 172.21.2.102:0/3228769242 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7faba00ae860 0x7faba00aec60 secure :-1 s=READY pgs=2086 cs=0 l=1 rev1=1 rx=0x7fab90006f00 tx=0x7fab9000b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:03.613 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:03.608 7fab97fff700 1 -- 172.21.2.102:0/3228769242 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fab9000bd30 con 0x7faba00ae860 2021-02-10T11:31:03.613 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:03.608 7fab97fff700 1 -- 172.21.2.102:0/3228769242 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab90006c20 con 0x7faba00ae860 2021-02-10T11:31:03.613 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:03.608 7fab97fff700 1 -- 172.21.2.102:0/3228769242 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fab90002020 con 0x7faba00ae860 2021-02-10T11:31:03.613 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:03.608 7faba60e6700 1 -- 172.21.2.102:0/3228769242 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faba008f530 con 0x7faba00ae860 2021-02-10T11:31:03.613 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:03.608 7faba60e6700 1 -- 172.21.2.102:0/3228769242 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faba008fac0 con 0x7faba00ae860 2021-02-10T11:31:03.613 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:03.609 7fab97fff700 1 -- 172.21.2.102:0/3228769242 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fab90005310 con 0x7faba00ae860 2021-02-10T11:31:03.613 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:03.609 7fab97fff700 1 --2- 172.21.2.102:0/3228769242 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fab840186c0 0x7fab8401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:03.613 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:03.609 7fab97fff700 1 -- 172.21.2.102:0/3228769242 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fab90032550 con 0x7faba00ae860 2021-02-10T11:31:03.613 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:03.609 7fab9effd700 1 --2- 172.21.2.102:0/3228769242 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fab840186c0 0x7fab8401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:03.613 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:03.609 7fab9effd700 1 --2- 172.21.2.102:0/3228769242 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fab840186c0 0x7fab8401abc0 crc :-1 s=READY pgs=927 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:03.641 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:03.641 7faba60e6700 1 -- 172.21.2.102:0/3228769242 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faba0082980 con 0x7faba00ae860 2021-02-10T11:31:03.644 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:03.643 7fab97fff700 1 -- 172.21.2.102:0/3228769242 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fab900009b0 con 0x7faba00ae860 2021-02-10T11:31:03.774 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:03.773 7faba60e6700 1 -- 172.21.2.102:0/3228769242 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.3", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7faba00fe1b0 con 0x7faba00ae860 2021-02-10T11:31:03.777 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:03.776 7fab97fff700 1 -- 172.21.2.102:0/3228769242 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.3", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.3 v62) v1 ==== 184+0+0 (secure 0 0 0) 0x7fab90005310 con 0x7faba00ae860 2021-02-10T11:31:03.781 INFO:teuthology.orchestra.run.gibba002.stderr:updated caps for client.3 2021-02-10T11:31:03.782 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:03.781 7faba60e6700 1 -- 172.21.2.102:0/3228769242 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fab840186c0 msgr2=0x7fab8401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:03.783 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:03.781 7faba60e6700 1 --2- 172.21.2.102:0/3228769242 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fab840186c0 0x7fab8401abc0 crc :-1 s=READY pgs=927 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:03.783 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:03.781 7faba60e6700 1 -- 172.21.2.102:0/3228769242 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7faba00ae860 msgr2=0x7faba00aec60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:03.783 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:03.781 7faba60e6700 1 --2- 172.21.2.102:0/3228769242 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7faba00ae860 0x7faba00aec60 secure :-1 s=READY pgs=2086 cs=0 l=1 rev1=1 rx=0x7fab90006f00 tx=0x7fab9000b940).stop 2021-02-10T11:31:03.783 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:03.781 7faba60e6700 1 -- 172.21.2.102:0/3228769242 shutdown_connections 2021-02-10T11:31:03.783 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:03.781 7faba60e6700 1 --2- 172.21.2.102:0/3228769242 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7faba008e2e0 0x7faba008ef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:03.783 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:03.781 7faba60e6700 1 --2- 172.21.2.102:0/3228769242 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fab840186c0 0x7fab8401abc0 unknown :-1 s=CLOSED pgs=927 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:03.783 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:03.781 7faba60e6700 1 --2- 172.21.2.102:0/3228769242 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7faba008d8c0 0x7faba008dcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:03.783 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:03.781 7faba60e6700 1 --2- 172.21.2.102:0/3228769242 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7faba00ae860 0x7faba00aec60 unknown :-1 s=CLOSED pgs=2086 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:03.783 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:03.781 7faba60e6700 1 -- 172.21.2.102:0/3228769242 >> 172.21.2.102:0/3228769242 conn(0x7faba01abb80 msgr2=0x7faba0082df0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:03.783 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:03.782 7faba60e6700 1 -- 172.21.2.102:0/3228769242 shutdown_connections 2021-02-10T11:31:03.783 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:03.782 7faba60e6700 1 -- 172.21.2.102:0/3228769242 wait complete. 2021-02-10T11:31:03.791 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:31:04.219 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:04.202 7f59c998c700 1 Processor -- start 2021-02-10T11:31:04.219 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:04.202 7f59c998c700 1 -- start start 2021-02-10T11:31:04.219 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:04.202 7f59c998c700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f59c40d1870 0x7f59c41b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:04.219 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:04.202 7f59c998c700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f59c40f0650 0x7f59c40cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:04.219 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:04.202 7f59c998c700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59c40f0b10 con 0x7f59c41b0a10 2021-02-10T11:31:04.220 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:04.202 7f59c998c700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59c40e5ed0 con 0x7f59c40d1870 2021-02-10T11:31:04.220 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:04.202 7f59c998c700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59c40d1c70 con 0x7f59c40f0650 2021-02-10T11:31:04.220 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:04.203 7f59c3574700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f59c40d1870 0x7f59c41b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:04.220 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:04.203 7f59c3574700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f59c40d1870 0x7f59c41b03f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:33388/0 (socket says 172.21.2.102:33388) 2021-02-10T11:31:04.220 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:04.203 7f59c3574700 1 -- 172.21.2.102:0/3067832947 learned_addr learned my addr 172.21.2.102:0/3067832947 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:31:04.220 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:04.203 7f59c3574700 1 -- 172.21.2.102:0/3067832947 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f59c40f0650 msgr2=0x7f59c40cfb20 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:31:04.220 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:04.203 7f59c3574700 1 --2- 172.21.2.102:0/3067832947 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f59c40f0650 0x7f59c40cfb20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:04.220 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:04.203 7f59c3574700 1 -- 172.21.2.102:0/3067832947 >> v1:172.21.2.113:6789/0 conn(0x7f59c41b0a10 legacy=0x7f59c40cf330 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:31:04.220 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:04.203 7f59c3574700 1 -- 172.21.2.102:0/3067832947 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59c40d1e90 con 0x7f59c40d1870 2021-02-10T11:31:04.220 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:04.203 7f59c3574700 1 --2- 172.21.2.102:0/3067832947 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f59c40d1870 0x7f59c41b03f0 secure :-1 s=READY pgs=2087 cs=0 l=1 rev1=1 rx=0x7f59ac006700 tx=0x7f59ac009b80).ready entity=mon.0 client_cookie=5b795ba98afd9fec server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:04.220 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:04.203 7f59c2572700 1 -- 172.21.2.102:0/3067832947 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f59ac0009b0 con 0x7f59c40d1870 2021-02-10T11:31:04.220 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:04.203 7f59c2572700 1 -- 172.21.2.102:0/3067832947 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59ac006390 con 0x7f59c40d1870 2021-02-10T11:31:04.221 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:04.203 7f59c2572700 1 -- 172.21.2.102:0/3067832947 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f59ac00c0c0 con 0x7f59c40d1870 2021-02-10T11:31:04.221 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:04.203 7f59c998c700 1 -- 172.21.2.102:0/3067832947 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f59c40d1870 msgr2=0x7f59c41b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:04.221 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:04.203 7f59c998c700 1 --2- 172.21.2.102:0/3067832947 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f59c40d1870 0x7f59c41b03f0 secure :-1 s=READY pgs=2087 cs=0 l=1 rev1=1 rx=0x7f59ac006700 tx=0x7f59ac009b80).stop 2021-02-10T11:31:04.221 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:04.204 7f59c998c700 1 -- 172.21.2.102:0/3067832947 shutdown_connections 2021-02-10T11:31:04.221 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:04.204 7f59c998c700 1 --2- 172.21.2.102:0/3067832947 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f59c40f0650 0x7f59c40cfb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:04.221 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:04.204 7f59c998c700 1 --2- 172.21.2.102:0/3067832947 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f59c40d1870 0x7f59c41b03f0 unknown :-1 s=CLOSED pgs=2087 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:04.221 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:04.204 7f59c998c700 1 -- 172.21.2.102:0/3067832947 >> 172.21.2.102:0/3067832947 conn(0x7f59c40ce450 msgr2=0x7f59c40d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:04.221 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:04.204 7f59c998c700 1 -- 172.21.2.102:0/3067832947 shutdown_connections 2021-02-10T11:31:04.221 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:04.204 7f59c998c700 1 -- 172.21.2.102:0/3067832947 wait complete. 2021-02-10T11:31:04.223 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:04.219 7f59c998c700 1 Processor -- start 2021-02-10T11:31:04.223 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:04.219 7f59c998c700 1 -- start start 2021-02-10T11:31:04.223 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:04.219 7f59c998c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f59c40b4f30 0x7f59c418b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:04.223 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:04.219 7f59c998c700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f59c4093f40 0x7f59c4094340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:04.223 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:04.219 7f59c998c700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f59c4094960 0x7f59c4187e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:04.223 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:04.219 7f59c998c700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59c4094e90 con 0x7f59c40b4f30 2021-02-10T11:31:04.223 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:04.219 7f59c998c700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59c418bff0 con 0x7f59c4094960 2021-02-10T11:31:04.223 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:04.219 7f59c998c700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59c408f780 con 0x7f59c4093f40 2021-02-10T11:31:04.223 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:04.219 7f59c3574700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f59c40b4f30 0x7f59c418b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:04.224 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:04.219 7f59c3574700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f59c40b4f30 0x7f59c418b9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:33396/0 (socket says 172.21.2.102:33396) 2021-02-10T11:31:04.224 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:04.219 7f59c3574700 1 -- 172.21.2.102:0/148683275 learned_addr learned my addr 172.21.2.102:0/148683275 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:31:04.224 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:04.219 7f59c3d75700 1 --2- 172.21.2.102:0/148683275 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f59c4094960 0x7f59c4187e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:04.224 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:04.220 7f59c2d73700 1 --2- 172.21.2.102:0/148683275 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f59c4093f40 0x7f59c4094340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:04.224 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:04.220 7f59c3574700 1 -- 172.21.2.102:0/148683275 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f59c4093f40 msgr2=0x7f59c4094340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:04.224 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:04.220 7f59c3574700 1 --2- 172.21.2.102:0/148683275 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f59c4093f40 0x7f59c4094340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:04.224 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:04.220 7f59c3574700 1 -- 172.21.2.102:0/148683275 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f59c4094960 msgr2=0x7f59c4187e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:04.224 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:04.220 7f59c3574700 1 --2- 172.21.2.102:0/148683275 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f59c4094960 0x7f59c4187e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:04.224 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:04.220 7f59c3574700 1 -- 172.21.2.102:0/148683275 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59c4188470 con 0x7f59c40b4f30 2021-02-10T11:31:04.224 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:04.220 7f59c2d73700 1 --2- 172.21.2.102:0/148683275 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f59c4093f40 0x7f59c4094340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:31:04.224 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:04.220 7f59c3d75700 1 --2- 172.21.2.102:0/148683275 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f59c4094960 0x7f59c4187e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:31:04.224 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:04.220 7f59c3574700 1 --2- 172.21.2.102:0/148683275 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f59c40b4f30 0x7f59c418b9d0 secure :-1 s=READY pgs=2088 cs=0 l=1 rev1=1 rx=0x7f59ac0066a0 tx=0x7f59ac00b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:04.225 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:04.220 7f59bbfff700 1 -- 172.21.2.102:0/148683275 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f59ac00bd80 con 0x7f59c40b4f30 2021-02-10T11:31:04.225 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:04.220 7f59bbfff700 1 -- 172.21.2.102:0/148683275 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59ac0052e0 con 0x7f59c40b4f30 2021-02-10T11:31:04.225 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:04.220 7f59bbfff700 1 -- 172.21.2.102:0/148683275 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f59ac005800 con 0x7f59c40b4f30 2021-02-10T11:31:04.225 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:04.220 7f59c998c700 1 -- 172.21.2.102:0/148683275 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59c4188470 con 0x7f59c40b4f30 2021-02-10T11:31:04.225 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:04.220 7f59c998c700 1 -- 172.21.2.102:0/148683275 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59c4188a00 con 0x7f59c40b4f30 2021-02-10T11:31:04.225 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:04.221 7f59bbfff700 1 -- 172.21.2.102:0/148683275 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f59ac006030 con 0x7f59c40b4f30 2021-02-10T11:31:04.225 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:04.221 7f59bbfff700 1 --2- 172.21.2.102:0/148683275 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f59a80186c0 0x7f59a801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:04.225 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:04.221 7f59bbfff700 1 -- 172.21.2.102:0/148683275 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f59ac0325a0 con 0x7f59c40b4f30 2021-02-10T11:31:04.225 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:04.221 7f59c2d73700 1 --2- 172.21.2.102:0/148683275 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f59a80186c0 0x7f59a801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:04.225 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:04.221 7f59c2d73700 1 --2- 172.21.2.102:0/148683275 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f59a80186c0 0x7f59a801abc0 crc :-1 s=READY pgs=928 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:04.253 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:04.252 7f59c998c700 1 -- 172.21.2.102:0/148683275 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59c40ce450 con 0x7f59c40b4f30 2021-02-10T11:31:04.255 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:04.255 7f59bbfff700 1 -- 172.21.2.102:0/148683275 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f59ac030d50 con 0x7f59c40b4f30 2021-02-10T11:31:04.384 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:04.383 7f59c998c700 1 -- 172.21.2.102:0/148683275 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f59c40ce450 con 0x7f59c40b4f30 2021-02-10T11:31:04.385 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:04.384 7f59bbfff700 1 -- 172.21.2.102:0/148683275 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+2101 (secure 0 0 0) 0x7f59ac00bd80 con 0x7f59c40b4f30 2021-02-10T11:31:04.385 INFO:teuthology.orchestra.run.gibba002.stderr:dumped fsmap epoch 82 2021-02-10T11:31:04.386 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:31:04.386 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":82,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10056,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.102:6835/711305601","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6834","nonce":711305601},{"type":"v1","addr":"172.21.2.102:6835","nonce":711305601}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":78}],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:30:58.047057","modified":"2021-02-10 11:31:00.055485","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6353},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6353":{"gid":6353,"name":"b","rank":0,"incarnation":79,"state":"up:active","state_seq":3,"addr":"172.21.2.113:6833/1091618344","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6832","nonce":1091618344},{"type":"v1","addr":"172.21.2.113:6833","nonce":1091618344}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":10}]} 2021-02-10T11:31:04.387 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:04.386 7f59c998c700 1 -- 172.21.2.102:0/148683275 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f59a80186c0 msgr2=0x7f59a801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:04.387 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:04.386 7f59c998c700 1 --2- 172.21.2.102:0/148683275 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f59a80186c0 0x7f59a801abc0 crc :-1 s=READY pgs=928 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:04.387 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:04.386 7f59c998c700 1 -- 172.21.2.102:0/148683275 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f59c40b4f30 msgr2=0x7f59c418b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:04.387 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:04.386 7f59c998c700 1 --2- 172.21.2.102:0/148683275 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f59c40b4f30 0x7f59c418b9d0 secure :-1 s=READY pgs=2088 cs=0 l=1 rev1=1 rx=0x7f59ac0066a0 tx=0x7f59ac00b990).stop 2021-02-10T11:31:04.387 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:04.386 7f59c998c700 1 -- 172.21.2.102:0/148683275 shutdown_connections 2021-02-10T11:31:04.387 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:04.386 7f59c998c700 1 --2- 172.21.2.102:0/148683275 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f59a80186c0 0x7f59a801abc0 unknown :-1 s=CLOSED pgs=928 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:04.387 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:04.386 7f59c998c700 1 --2- 172.21.2.102:0/148683275 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f59c4094960 0x7f59c4187e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:04.387 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:04.386 7f59c998c700 1 --2- 172.21.2.102:0/148683275 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f59c4093f40 0x7f59c4094340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:04.387 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:04.386 7f59c998c700 1 --2- 172.21.2.102:0/148683275 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f59c40b4f30 0x7f59c418b9d0 unknown :-1 s=CLOSED pgs=2088 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:04.387 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:04.386 7f59c998c700 1 -- 172.21.2.102:0/148683275 >> 172.21.2.102:0/148683275 conn(0x7f59c419f8f0 msgr2=0x7f59c41a5f80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:04.388 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:04.386 7f59c998c700 1 -- 172.21.2.102:0/148683275 shutdown_connections 2021-02-10T11:31:04.388 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:04.386 7f59c998c700 1 -- 172.21.2.102:0/148683275 wait complete. 2021-02-10T11:31:04.597 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 82, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:30:58.047057', 'modified': '2021-02-10 11:31:00.055485', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6353}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6353': {'gid': 6353, 'name': 'b', 'rank': 0, 'incarnation': 79, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.113:6833/1091618344', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.113:6832', 'nonce': 1091618344}, {'type': 'v1', 'addr': '172.21.2.113:6833', 'nonce': 1091618344}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:31:04.597 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:31:04.597 DEBUG:teuthology.orchestra.run.gibba013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:31:04.997 INFO:teuthology.orchestra.run.gibba013.stdout:{ 2021-02-10T11:31:04.997 INFO:teuthology.orchestra.run.gibba013.stdout: "cluster_fsid": "68db30ea-dbce-4aef-a6df-f01c6fea0513", 2021-02-10T11:31:04.997 INFO:teuthology.orchestra.run.gibba013.stdout: "whoami": 0, 2021-02-10T11:31:04.997 INFO:teuthology.orchestra.run.gibba013.stdout: "id": 6353, 2021-02-10T11:31:04.997 INFO:teuthology.orchestra.run.gibba013.stdout: "want_state": "up:active", 2021-02-10T11:31:04.997 INFO:teuthology.orchestra.run.gibba013.stdout: "state": "up:active", 2021-02-10T11:31:04.997 INFO:teuthology.orchestra.run.gibba013.stdout: "rank_uptime": 6.9474685029999996, 2021-02-10T11:31:04.997 INFO:teuthology.orchestra.run.gibba013.stdout: "mdsmap_epoch": 82, 2021-02-10T11:31:04.997 INFO:teuthology.orchestra.run.gibba013.stdout: "osdmap_epoch": 103, 2021-02-10T11:31:04.997 INFO:teuthology.orchestra.run.gibba013.stdout: "osdmap_epoch_barrier": 103, 2021-02-10T11:31:04.997 INFO:teuthology.orchestra.run.gibba013.stdout: "uptime": 12.586164409 2021-02-10T11:31:04.998 INFO:teuthology.orchestra.run.gibba013.stdout:} 2021-02-10T11:31:05.001 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "68db30ea-dbce-4aef-a6df-f01c6fea0513", "whoami": 0, "id": 6353, "want_state": "up:active", "state": "up:active", "rank_uptime": 6.9474685029999996, "mdsmap_epoch": 82, "osdmap_epoch": 103, "osdmap_epoch_barrier": 103, "uptime": 12.586164409 } 2021-02-10T11:31:05.001 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:31:05.429 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:05.410 7f6bcb08d700 1 Processor -- start 2021-02-10T11:31:05.429 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:05.410 7f6bcb08d700 1 -- start start 2021-02-10T11:31:05.429 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:05.411 7f6bcb08d700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f6bc40d1c70 0x7f6bc40f20a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:05.429 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:05.411 7f6bcb08d700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bc40f0b10 con 0x7f6bc40d0340 2021-02-10T11:31:05.429 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:05.411 7f6bcb08d700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bc41b0be0 con 0x7f6bc40d1870 2021-02-10T11:31:05.429 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:05.411 7f6bcb08d700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6bc40e5ed0 con 0x7f6bc40d1c70 2021-02-10T11:31:05.429 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:05.411 7f6bc3fff700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f6bc40d0340 0x7f6bc40ef640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:53856/0 (socket says 172.21.2.102:53856) 2021-02-10T11:31:05.429 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:05.411 7f6bc3fff700 1 -- 172.21.2.102:0/1359870822 learned_addr learned my addr 172.21.2.102:0/1359870822 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:31:05.430 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:05.412 7f6bc37fe700 1 -- 172.21.2.102:0/1359870822 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3054227816 0 0) 0x7f6bb40010b0 con 0x7f6bc40d0340 2021-02-10T11:31:05.430 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:05.412 7f6bc37fe700 1 -- 172.21.2.102:0/1359870822 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6bb0002fe0 con 0x7f6bc40d0340 2021-02-10T11:31:05.430 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:05.412 7f6bc96aa700 1 --2- 172.21.2.102:0/1359870822 >> v2:172.21.2.113:3300/0 conn(0x7f6bc40d1c70 0x7f6bc40f20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:05.430 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:05.412 7f6bc37fe700 1 -- 172.21.2.102:0/1359870822 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3239206619 0 0) 0x7f6bb40010b0 con 0x7f6bc40d0340 2021-02-10T11:31:05.430 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:05.412 7f6bc37fe700 1 -- 172.21.2.102:0/1359870822 >> v2:172.21.2.113:3300/0 conn(0x7f6bc40d1c70 msgr2=0x7f6bc40f20a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:05.430 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:05.412 7f6bc37fe700 1 --2- 172.21.2.102:0/1359870822 >> v2:172.21.2.113:3300/0 conn(0x7f6bc40d1c70 0x7f6bc40f20a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:05.430 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:05.412 7f6bc37fe700 1 -- 172.21.2.102:0/1359870822 >> v1:172.21.2.113:6789/0 conn(0x7f6bc40d1870 legacy=0x7f6bc41b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:05.431 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:05.412 7f6bc37fe700 1 -- 172.21.2.102:0/1359870822 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bc40efe30 con 0x7f6bc40d0340 2021-02-10T11:31:05.431 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:05.412 7f6bc37fe700 1 -- 172.21.2.102:0/1359870822 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7f6bb40015b0 con 0x7f6bc40d0340 2021-02-10T11:31:05.431 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:05.412 7f6bc37fe700 1 -- 172.21.2.102:0/1359870822 >> v1:172.21.2.102:6789/0 conn(0x7f6bc40d0340 legacy=0x7f6bc40ef640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:05.431 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:05.412 7f6bc37fe700 1 --2- 172.21.2.102:0/1359870822 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6bb0003a60 0x7f6bb0005f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:05.431 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:05.412 7f6bc37fe700 1 --2- 172.21.2.102:0/1359870822 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6bb0006800 0x7f6bb0008d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:05.431 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:05.412 7f6bc37fe700 1 --2- 172.21.2.102:0/1359870822 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6bb0009620 0x7f6bb000bb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:05.431 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:05.412 7f6bc37fe700 1 -- 172.21.2.102:0/1359870822 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bb000c390 con 0x7f6bb0003a60 2021-02-10T11:31:05.431 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:05.412 7f6bc37fe700 1 -- 172.21.2.102:0/1359870822 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bb000c5b0 con 0x7f6bb0009620 2021-02-10T11:31:05.431 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:05.412 7f6bc37fe700 1 -- 172.21.2.102:0/1359870822 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bb000c7d0 con 0x7f6bb0006800 2021-02-10T11:31:05.431 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:05.412 7f6bc8ea9700 1 --2- 172.21.2.102:0/1359870822 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6bb0003a60 0x7f6bb0005f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:05.431 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:05.412 7f6bc96aa700 1 --2- 172.21.2.102:0/1359870822 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6bb0009620 0x7f6bb000bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:05.431 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:05.412 7f6bc3fff700 1 --2- 172.21.2.102:0/1359870822 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6bb0006800 0x7f6bb0008d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:05.431 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:05.412 7f6bc8ea9700 1 -- 172.21.2.102:0/1359870822 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6bb0006800 msgr2=0x7f6bb0008d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:05.432 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:05.412 7f6bc8ea9700 1 --2- 172.21.2.102:0/1359870822 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6bb0006800 0x7f6bb0008d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:05.432 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:05.412 7f6bc8ea9700 1 -- 172.21.2.102:0/1359870822 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6bb0009620 msgr2=0x7f6bb000bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:05.432 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:05.412 7f6bc8ea9700 1 --2- 172.21.2.102:0/1359870822 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6bb0009620 0x7f6bb000bb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:05.432 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:05.412 7f6bc8ea9700 1 -- 172.21.2.102:0/1359870822 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bb0050a40 con 0x7f6bb0003a60 2021-02-10T11:31:05.432 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:05.412 7f6bc96aa700 1 --2- 172.21.2.102:0/1359870822 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6bb0009620 0x7f6bb000bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:31:05.432 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:05.412 7f6bc96aa700 1 -- 172.21.2.102:0/1359870822 reap_dead start 2021-02-10T11:31:05.432 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:05.412 7f6bc3fff700 1 --2- 172.21.2.102:0/1359870822 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6bb0006800 0x7f6bb0008d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:31:05.432 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:05.413 7f6bc8ea9700 1 --2- 172.21.2.102:0/1359870822 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6bb0003a60 0x7f6bb0005f60 secure :-1 s=READY pgs=2090 cs=0 l=1 rev1=1 rx=0x7f6bac006ac0 tx=0x7f6bac009af0).ready entity=mon.0 client_cookie=74c19a2bd1dcd4ba server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:05.432 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:05.413 7f6bc37fe700 1 -- 172.21.2.102:0/1359870822 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6bac007470 con 0x7f6bb0003a60 2021-02-10T11:31:05.432 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:05.413 7f6bc37fe700 1 -- 172.21.2.102:0/1359870822 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bac0066b0 con 0x7f6bb0003a60 2021-02-10T11:31:05.432 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:05.413 7f6bc37fe700 1 -- 172.21.2.102:0/1359870822 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6bac0079a0 con 0x7f6bb0003a60 2021-02-10T11:31:05.432 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:05.413 7f6bcb08d700 1 -- 172.21.2.102:0/1359870822 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6bb0003a60 msgr2=0x7f6bb0005f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:05.432 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:05.413 7f6bcb08d700 1 --2- 172.21.2.102:0/1359870822 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6bb0003a60 0x7f6bb0005f60 secure :-1 s=READY pgs=2090 cs=0 l=1 rev1=1 rx=0x7f6bac006ac0 tx=0x7f6bac009af0).stop 2021-02-10T11:31:05.433 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:05.413 7f6bcb08d700 1 -- 172.21.2.102:0/1359870822 shutdown_connections 2021-02-10T11:31:05.433 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:05.413 7f6bcb08d700 1 --2- 172.21.2.102:0/1359870822 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6bb0003a60 0x7f6bb0005f60 unknown :-1 s=CLOSED pgs=2090 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:05.433 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:05.413 7f6bcb08d700 1 -- 172.21.2.102:0/1359870822 >> 172.21.2.102:0/1359870822 conn(0x7f6bc40ce450 msgr2=0x7f6bc40d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:05.433 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:05.413 7f6bcb08d700 1 -- 172.21.2.102:0/1359870822 shutdown_connections 2021-02-10T11:31:05.433 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:05.413 7f6bcb08d700 1 -- 172.21.2.102:0/1359870822 wait complete. 2021-02-10T11:31:05.433 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:05.428 7f6bcb08d700 1 Processor -- start 2021-02-10T11:31:05.434 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:05.428 7f6bcb08d700 1 -- start start 2021-02-10T11:31:05.434 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:05.429 7f6bcb08d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6bc4195650 0x7f6bc40b5560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:05.434 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:05.429 7f6bcb08d700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6bc40b5b80 0x7f6bc41a2e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:05.434 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:05.429 7f6bcb08d700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6bc40acfa0 0x7f6bc41a3440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:05.434 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:05.429 7f6bcb08d700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bc40b6010 con 0x7f6bc4195650 2021-02-10T11:31:05.435 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:05.429 7f6bcb08d700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bc41a3a60 con 0x7f6bc40b5b80 2021-02-10T11:31:05.435 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:05.429 7f6bcb08d700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bc41a3c40 con 0x7f6bc40acfa0 2021-02-10T11:31:05.435 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:05.429 7f6bc8ea9700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6bc4195650 0x7f6bc40b5560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:05.435 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:05.429 7f6bc8ea9700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6bc4195650 0x7f6bc40b5560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:33410/0 (socket says 172.21.2.102:33410) 2021-02-10T11:31:05.435 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:05.429 7f6bc8ea9700 1 -- 172.21.2.102:0/2243865849 learned_addr learned my addr 172.21.2.102:0/2243865849 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:31:05.435 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:05.429 7f6bc8ea9700 1 -- 172.21.2.102:0/2243865849 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6bc40acfa0 msgr2=0x7f6bc41a3440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:05.435 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:05.429 7f6bc8ea9700 1 --2- 172.21.2.102:0/2243865849 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6bc40acfa0 0x7f6bc41a3440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:05.435 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:05.429 7f6bc8ea9700 1 -- 172.21.2.102:0/2243865849 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6bc40b5b80 msgr2=0x7f6bc41a2e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:05.435 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:05.429 7f6bc8ea9700 1 --2- 172.21.2.102:0/2243865849 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6bc40b5b80 0x7f6bc41a2e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:05.435 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:05.429 7f6bc8ea9700 1 -- 172.21.2.102:0/2243865849 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bc40bcae0 con 0x7f6bc4195650 2021-02-10T11:31:05.435 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:05.429 7f6bc8ea9700 1 --2- 172.21.2.102:0/2243865849 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6bc4195650 0x7f6bc40b5560 secure :-1 s=READY pgs=2091 cs=0 l=1 rev1=1 rx=0x7f6bac00b6b0 tx=0x7f6bac00d910).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:05.435 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:05.430 7f6bc17fa700 1 -- 172.21.2.102:0/2243865849 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6bac00b0d0 con 0x7f6bc4195650 2021-02-10T11:31:05.435 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:05.430 7f6bc17fa700 1 -- 172.21.2.102:0/2243865849 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bac00b350 con 0x7f6bc4195650 2021-02-10T11:31:05.435 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:05.430 7f6bc17fa700 1 -- 172.21.2.102:0/2243865849 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6bac009890 con 0x7f6bc4195650 2021-02-10T11:31:05.436 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:05.430 7f6bcb08d700 1 -- 172.21.2.102:0/2243865849 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6bc40bcae0 con 0x7f6bc4195650 2021-02-10T11:31:05.436 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:05.430 7f6bcb08d700 1 -- 172.21.2.102:0/2243865849 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bc40bcae0 con 0x7f6bc4195650 2021-02-10T11:31:05.436 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:05.430 7f6bc17fa700 1 -- 172.21.2.102:0/2243865849 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f6bac009860 con 0x7f6bc4195650 2021-02-10T11:31:05.436 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:05.430 7f6bc17fa700 1 --2- 172.21.2.102:0/2243865849 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6ba80186d0 0x7f6ba801abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:05.436 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:05.430 7f6bc17fa700 1 -- 172.21.2.102:0/2243865849 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f6bac032680 con 0x7f6bc4195650 2021-02-10T11:31:05.436 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:05.430 7f6bc3fff700 1 --2- 172.21.2.102:0/2243865849 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6ba80186d0 0x7f6ba801abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:05.436 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:05.431 7f6bc3fff700 1 --2- 172.21.2.102:0/2243865849 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6ba80186d0 0x7f6ba801abd0 crc :-1 s=READY pgs=929 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:05.462 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:05.461 7f6bcb08d700 1 -- 172.21.2.102:0/2243865849 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6bc40b6010 con 0x7f6bc4195650 2021-02-10T11:31:05.464 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:05.463 7f6bc17fa700 1 -- 172.21.2.102:0/2243865849 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f6bac0091a0 con 0x7f6bc4195650 2021-02-10T11:31:05.588 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:05.587 7f6bcb08d700 1 -- 172.21.2.102:0/2243865849 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6bc40b6010 con 0x7f6bc4195650 2021-02-10T11:31:05.589 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:05.588 7f6bc17fa700 1 -- 172.21.2.102:0/2243865849 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v103) v1 ==== 74+0+14005 (secure 0 0 0) 0x7f6bac009ef0 con 0x7f6bc4195650 2021-02-10T11:31:05.590 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:31:05.590 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":103,"fsid":"68db30ea-dbce-4aef-a6df-f01c6fea0513","created":"2021-02-10 11:03:06.331557","modified":"2021-02-10 11:31:00.055039","last_up_change":"2021-02-10 11:03:32.690562","last_in_change":"2021-02-10 11:03:32.690562","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:03:37.322519","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:30:56.015905","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"101","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":21,"pool_name":"cephfs_data","create_time":"2021-02-10 11:30:57.042942","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"103","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"be691b4d-a039-408a-af80-9d53f02b6f1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6818","nonce":36088},{"type":"v1","addr":"172.21.2.102:6820","nonce":36088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6822","nonce":36088},{"type":"v1","addr":"172.21.2.102:6824","nonce":36088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6830","nonce":36088},{"type":"v1","addr":"172.21.2.102:6832","nonce":36088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6826","nonce":36088},{"type":"v1","addr":"172.21.2.102:6828","nonce":36088}]},"public_addr":"172.21.2.102:6820/36088","cluster_addr":"172.21.2.102:6824/36088","heartbeat_back_addr":"172.21.2.102:6832/36088","heartbeat_front_addr":"172.21.2.102:6828/36088","state":["exists","up"]},{"osd":1,"uuid":"64e3ccc8-9c69-4d7c-865e-af8e83822635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6819","nonce":36087},{"type":"v1","addr":"172.21.2.102:6821","nonce":36087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6823","nonce":36087},{"type":"v1","addr":"172.21.2.102:6825","nonce":36087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6831","nonce":36087},{"type":"v1","addr":"172.21.2.102:6833","nonce":36087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6827","nonce":36087},{"type":"v1","addr":"172.21.2.102:6829","nonce":36087}]},"public_addr":"172.21.2.102:6821/36087","cluster_addr":"172.21.2.102:6825/36087","heartbeat_back_addr":"172.21.2.102:6833/36087","heartbeat_front_addr":"172.21.2.102:6829/36087","state":["exists","up"]},{"osd":2,"uuid":"153700af-747f-481f-bdf4-d6072d3a842f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6810","nonce":36089},{"type":"v1","addr":"172.21.2.102:6811","nonce":36089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6812","nonce":36089},{"type":"v1","addr":"172.21.2.102:6813","nonce":36089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6816","nonce":36089},{"type":"v1","addr":"172.21.2.102:6817","nonce":36089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6814","nonce":36089},{"type":"v1","addr":"172.21.2.102:6815","nonce":36089}]},"public_addr":"172.21.2.102:6811/36089","cluster_addr":"172.21.2.102:6813/36089","heartbeat_back_addr":"172.21.2.102:6817/36089","heartbeat_front_addr":"172.21.2.102:6815/36089","state":["exists","up"]},{"osd":3,"uuid":"cbc678f4-83a9-45e6-8830-2f7c232b1ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6802","nonce":36090},{"type":"v1","addr":"172.21.2.102:6803","nonce":36090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6804","nonce":36090},{"type":"v1","addr":"172.21.2.102:6805","nonce":36090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6808","nonce":36090},{"type":"v1","addr":"172.21.2.102:6809","nonce":36090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6806","nonce":36090},{"type":"v1","addr":"172.21.2.102:6807","nonce":36090}]},"public_addr":"172.21.2.102:6803/36090","cluster_addr":"172.21.2.102:6805/36090","heartbeat_back_addr":"172.21.2.102:6809/36090","heartbeat_front_addr":"172.21.2.102:6807/36090","state":["exists","up"]},{"osd":4,"uuid":"c01e4008-b258-4f10-afd2-9e7f37214351","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6808","nonce":36368},{"type":"v1","addr":"172.21.2.113:6811","nonce":36368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6814","nonce":36368},{"type":"v1","addr":"172.21.2.113:6818","nonce":36368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6826","nonce":36368},{"type":"v1","addr":"172.21.2.113:6828","nonce":36368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6820","nonce":36368},{"type":"v1","addr":"172.21.2.113:6824","nonce":36368}]},"public_addr":"172.21.2.113:6811/36368","cluster_addr":"172.21.2.113:6818/36368","heartbeat_back_addr":"172.21.2.113:6828/36368","heartbeat_front_addr":"172.21.2.113:6824/36368","state":["exists","up"]},{"osd":5,"uuid":"551f6007-daf8-4a98-8ee0-c2b16bcccb5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6802","nonce":36366},{"type":"v1","addr":"172.21.2.113:6805","nonce":36366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6807","nonce":36366},{"type":"v1","addr":"172.21.2.113:6810","nonce":36366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6819","nonce":36366},{"type":"v1","addr":"172.21.2.113:6822","nonce":36366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6812","nonce":36366},{"type":"v1","addr":"172.21.2.113:6815","nonce":36366}]},"public_addr":"172.21.2.113:6805/36366","cluster_addr":"172.21.2.113:6810/36366","heartbeat_back_addr":"172.21.2.113:6822/36366","heartbeat_front_addr":"172.21.2.113:6815/36366","state":["exists","up"]},{"osd":6,"uuid":"bf67c040-0e50-48be-b102-c3943bc9ebef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6800","nonce":36367},{"type":"v1","addr":"172.21.2.113:6801","nonce":36367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6803","nonce":36367},{"type":"v1","addr":"172.21.2.113:6804","nonce":36367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6813","nonce":36367},{"type":"v1","addr":"172.21.2.113:6816","nonce":36367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6806","nonce":36367},{"type":"v1","addr":"172.21.2.113:6809","nonce":36367}]},"public_addr":"172.21.2.113:6801/36367","cluster_addr":"172.21.2.113:6804/36367","heartbeat_back_addr":"172.21.2.113:6816/36367","heartbeat_front_addr":"172.21.2.113:6809/36367","state":["exists","up"]},{"osd":7,"uuid":"f5461f80-018d-428f-be89-2eb2bcbe7bb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6817","nonce":36365},{"type":"v1","addr":"172.21.2.113:6821","nonce":36365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6823","nonce":36365},{"type":"v1","addr":"172.21.2.113:6825","nonce":36365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6830","nonce":36365},{"type":"v1","addr":"172.21.2.113:6831","nonce":36365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6827","nonce":36365},{"type":"v1","addr":"172.21.2.113:6829","nonce":36365}]},"public_addr":"172.21.2.113:6821/36365","cluster_addr":"172.21.2.113:6825/36365","heartbeat_back_addr":"172.21.2.113:6831/36365","heartbeat_front_addr":"172.21.2.113:6829/36365","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:31:05.590 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:05.590 7f6bcb08d700 1 -- 172.21.2.102:0/2243865849 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6ba80186d0 msgr2=0x7f6ba801abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:05.591 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:05.590 7f6bcb08d700 1 --2- 172.21.2.102:0/2243865849 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6ba80186d0 0x7f6ba801abd0 crc :-1 s=READY pgs=929 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:05.591 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:05.590 7f6bcb08d700 1 -- 172.21.2.102:0/2243865849 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6bc4195650 msgr2=0x7f6bc40b5560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:05.591 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:05.590 7f6bcb08d700 1 --2- 172.21.2.102:0/2243865849 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6bc4195650 0x7f6bc40b5560 secure :-1 s=READY pgs=2091 cs=0 l=1 rev1=1 rx=0x7f6bac00b6b0 tx=0x7f6bac00d910).stop 2021-02-10T11:31:05.591 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:05.590 7f6bcb08d700 1 -- 172.21.2.102:0/2243865849 shutdown_connections 2021-02-10T11:31:05.591 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:05.590 7f6bcb08d700 1 --2- 172.21.2.102:0/2243865849 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6bc40acfa0 0x7f6bc41a3440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:05.592 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:05.590 7f6bcb08d700 1 --2- 172.21.2.102:0/2243865849 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6ba80186d0 0x7f6ba801abd0 unknown :-1 s=CLOSED pgs=929 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:05.592 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:05.590 7f6bcb08d700 1 --2- 172.21.2.102:0/2243865849 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6bc40b5b80 0x7f6bc41a2e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:05.592 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:05.590 7f6bcb08d700 1 --2- 172.21.2.102:0/2243865849 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6bc4195650 0x7f6bc40b5560 unknown :-1 s=CLOSED pgs=2091 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:05.592 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:05.590 7f6bcb08d700 1 -- 172.21.2.102:0/2243865849 >> 172.21.2.102:0/2243865849 conn(0x7f6bc40b3040 msgr2=0x7f6bc40c3250 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:05.592 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:05.590 7f6bcb08d700 1 -- 172.21.2.102:0/2243865849 shutdown_connections 2021-02-10T11:31:05.592 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:05.590 7f6bcb08d700 1 -- 172.21.2.102:0/2243865849 wait complete. 2021-02-10T11:31:05.600 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:03:37.322519', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 20, 'pool_name': 'cephfs_metadata', 'create_time': '2021-02-10 11:30:56.015905', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '101', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 21, 'pool_name': 'cephfs_data', 'create_time': '2021-02-10 11:30:57.042942', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '103', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-02-10T11:31:05.601 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:31:06.034 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:06.016 7f701ef31700 1 Processor -- start 2021-02-10T11:31:06.035 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:06.016 7f701ef31700 1 -- start start 2021-02-10T11:31:06.035 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:06.016 7f701ef31700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f70180d1870 0x7f70181b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:06.035 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:06.016 7f701ef31700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f70181b0ae0 0x7f7018108940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:06.035 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:06.016 7f701ef31700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7018109a60 con 0x7f70180d1c70 2021-02-10T11:31:06.035 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:06.016 7f701ef31700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70180e5ef0 con 0x7f70181b0ae0 2021-02-10T11:31:06.035 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:06.016 7f701ef31700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7018109750 con 0x7f70180d1870 2021-02-10T11:31:06.035 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:06.016 7f701d54e700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f70180d1c70 0x7f7018108f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:53874/0 (socket says 172.21.2.102:53874) 2021-02-10T11:31:06.035 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:06.016 7f701d54e700 1 -- 172.21.2.102:0/442388261 learned_addr learned my addr 172.21.2.102:0/442388261 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:31:06.035 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:06.017 7f70177fe700 1 -- 172.21.2.102:0/442388261 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1475687783 0 0) 0x7f7010001180 con 0x7f70180d1c70 2021-02-10T11:31:06.035 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:06.017 7f70177fe700 1 -- 172.21.2.102:0/442388261 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7004002fe0 con 0x7f70180d1c70 2021-02-10T11:31:06.035 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:06.017 7f7017fff700 1 --2- 172.21.2.102:0/442388261 >> v2:172.21.2.113:3300/0 conn(0x7f70181b0ae0 0x7f7018108940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:06.036 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:06.017 7f701cd4d700 1 --2- 172.21.2.102:0/442388261 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f70180d1870 0x7f70181b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:06.036 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:06.017 7f70177fe700 1 -- 172.21.2.102:0/442388261 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3812940660 0 0) 0x7f70100010e0 con 0x7f70180d1c70 2021-02-10T11:31:06.036 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:06.017 7f70177fe700 1 -- 172.21.2.102:0/442388261 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f70180d1870 msgr2=0x7f70181b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:06.036 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:06.017 7f70177fe700 1 --2- 172.21.2.102:0/442388261 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f70180d1870 0x7f70181b04c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:06.036 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:06.017 7f70177fe700 1 -- 172.21.2.102:0/442388261 >> v2:172.21.2.113:3300/0 conn(0x7f70181b0ae0 msgr2=0x7f7018108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:06.036 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:06.017 7f70177fe700 1 --2- 172.21.2.102:0/442388261 >> v2:172.21.2.113:3300/0 conn(0x7f70181b0ae0 0x7f7018108940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:06.036 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:06.017 7f70177fe700 1 -- 172.21.2.102:0/442388261 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7018088070 con 0x7f70180d1c70 2021-02-10T11:31:06.036 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:06.017 7f70177fe700 1 -- 172.21.2.102:0/442388261 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7f7010001600 con 0x7f70180d1c70 2021-02-10T11:31:06.036 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:06.017 7f70177fe700 1 -- 172.21.2.102:0/442388261 >> v1:172.21.2.102:6789/0 conn(0x7f70180d1c70 legacy=0x7f7018108f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:06.036 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:06.017 7f70177fe700 1 --2- 172.21.2.102:0/442388261 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7004003b90 0x7f7004005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:06.036 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:06.017 7f70177fe700 1 --2- 172.21.2.102:0/442388261 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7004006820 0x7f7004008d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:06.036 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:06.017 7f70177fe700 1 --2- 172.21.2.102:0/442388261 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f70040095a0 0x7f700400bac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:06.036 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:06.017 7f70177fe700 1 -- 172.21.2.102:0/442388261 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f700400c370 con 0x7f7004006820 2021-02-10T11:31:06.037 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:06.017 7f70177fe700 1 -- 172.21.2.102:0/442388261 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7004050560 con 0x7f70040095a0 2021-02-10T11:31:06.037 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:06.017 7f70177fe700 1 -- 172.21.2.102:0/442388261 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7004050780 con 0x7f7004003b90 2021-02-10T11:31:06.037 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:06.017 7f7017fff700 1 --2- 172.21.2.102:0/442388261 >> v2:172.21.2.113:3300/0 conn(0x7f70181b0ae0 0x7f7018108940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:31:06.037 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:06.017 7f701cd4d700 1 --2- 172.21.2.102:0/442388261 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7004003b90 0x7f7004005fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:06.037 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:06.017 7f7017fff700 1 --2- 172.21.2.102:0/442388261 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7004006820 0x7f7004008d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:06.037 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:06.017 7f701d54e700 1 --2- 172.21.2.102:0/442388261 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f70040095a0 0x7f700400bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:06.037 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:06.018 7f7017fff700 1 -- 172.21.2.102:0/442388261 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7004003b90 msgr2=0x7f7004005fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:06.037 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:06.018 7f7017fff700 1 --2- 172.21.2.102:0/442388261 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7004003b90 0x7f7004005fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:06.037 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:06.018 7f7017fff700 1 -- 172.21.2.102:0/442388261 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f70040095a0 msgr2=0x7f700400bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:06.037 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:06.018 7f7017fff700 1 --2- 172.21.2.102:0/442388261 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f70040095a0 0x7f700400bac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:06.037 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:06.018 7f7017fff700 1 -- 172.21.2.102:0/442388261 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7004050a20 con 0x7f7004006820 2021-02-10T11:31:06.037 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:06.018 7f701d54e700 1 --2- 172.21.2.102:0/442388261 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f70040095a0 0x7f700400bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:31:06.038 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:06.018 7f7017fff700 1 --2- 172.21.2.102:0/442388261 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7004006820 0x7f7004008d00 secure :-1 s=READY pgs=2093 cs=0 l=1 rev1=1 rx=0x7f7008006fa0 tx=0x7f700800b050).ready entity=mon.0 client_cookie=57d4c7fcc5d9a6d3 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:06.038 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:06.018 7f70177fe700 1 -- 172.21.2.102:0/442388261 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f700800b650 con 0x7f7004006820 2021-02-10T11:31:06.038 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:06.018 7f70177fe700 1 -- 172.21.2.102:0/442388261 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f700800b8d0 con 0x7f7004006820 2021-02-10T11:31:06.038 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:06.018 7f70177fe700 1 -- 172.21.2.102:0/442388261 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f700800bce0 con 0x7f7004006820 2021-02-10T11:31:06.038 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:06.018 7f701ef31700 1 -- 172.21.2.102:0/442388261 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7004006820 msgr2=0x7f7004008d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:06.038 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:06.018 7f701ef31700 1 --2- 172.21.2.102:0/442388261 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7004006820 0x7f7004008d00 secure :-1 s=READY pgs=2093 cs=0 l=1 rev1=1 rx=0x7f7008006fa0 tx=0x7f700800b050).stop 2021-02-10T11:31:06.038 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:06.018 7f701d54e700 1 -- 172.21.2.102:0/442388261 reap_dead start 2021-02-10T11:31:06.038 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:06.019 7f701ef31700 1 -- 172.21.2.102:0/442388261 shutdown_connections 2021-02-10T11:31:06.038 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:06.019 7f701ef31700 1 -- 172.21.2.102:0/442388261 >> 172.21.2.102:0/442388261 conn(0x7f70180ce420 msgr2=0x7f70180d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:06.038 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:06.019 7f701ef31700 1 -- 172.21.2.102:0/442388261 shutdown_connections 2021-02-10T11:31:06.038 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:06.019 7f701ef31700 1 -- 172.21.2.102:0/442388261 wait complete. 2021-02-10T11:31:06.038 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:06.034 7f701ef31700 1 Processor -- start 2021-02-10T11:31:06.038 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:06.034 7f701ef31700 1 -- start start 2021-02-10T11:31:06.040 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:06.034 7f701ef31700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7018085bc0 0x7f70180863e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:06.040 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:06.034 7f701ef31700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f70180954c0 0x7f7018096300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:06.040 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:06.034 7f701ef31700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f70180958c0 0x7f7018096920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:06.040 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:06.034 7f701ef31700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7018095d80 con 0x7f70180954c0 2021-02-10T11:31:06.040 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:06.034 7f701ef31700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70180bfa60 con 0x7f7018085bc0 2021-02-10T11:31:06.040 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:06.034 7f701ef31700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7018085fc0 con 0x7f70180958c0 2021-02-10T11:31:06.040 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:06.034 7f7017fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f70180954c0 0x7f7018096300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:06.040 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:06.034 7f7017fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f70180954c0 0x7f7018096300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:33428/0 (socket says 172.21.2.102:33428) 2021-02-10T11:31:06.040 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:06.034 7f7017fff700 1 -- 172.21.2.102:0/2574764611 learned_addr learned my addr 172.21.2.102:0/2574764611 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:31:06.041 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:06.035 7f7017fff700 1 -- 172.21.2.102:0/2574764611 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f70180958c0 msgr2=0x7f7018096920 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:31:06.041 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:06.035 7f7017fff700 1 --2- 172.21.2.102:0/2574764611 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f70180958c0 0x7f7018096920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:06.041 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:06.035 7f7017fff700 1 -- 172.21.2.102:0/2574764611 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7018085bc0 msgr2=0x7f70180863e0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:31:06.041 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:06.035 7f7017fff700 1 --2- 172.21.2.102:0/2574764611 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7018085bc0 0x7f70180863e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:06.041 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:06.035 7f7017fff700 1 -- 172.21.2.102:0/2574764611 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7018086c10 con 0x7f70180954c0 2021-02-10T11:31:06.041 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:06.035 7f7017fff700 1 --2- 172.21.2.102:0/2574764611 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f70180954c0 0x7f7018096300 secure :-1 s=READY pgs=2094 cs=0 l=1 rev1=1 rx=0x7f70080096a0 tx=0x7f700800bd00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:06.041 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:06.035 7f70157fa700 1 -- 172.21.2.102:0/2574764611 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7008006cb0 con 0x7f70180954c0 2021-02-10T11:31:06.041 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:06.035 7f70157fa700 1 -- 172.21.2.102:0/2574764611 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7008006030 con 0x7f70180954c0 2021-02-10T11:31:06.041 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:06.035 7f70157fa700 1 -- 172.21.2.102:0/2574764611 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f70080064e0 con 0x7f70180954c0 2021-02-10T11:31:06.041 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:06.035 7f701ef31700 1 -- 172.21.2.102:0/2574764611 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7018086c10 con 0x7f70180954c0 2021-02-10T11:31:06.041 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:06.035 7f701ef31700 1 -- 172.21.2.102:0/2574764611 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7018086c10 con 0x7f70180954c0 2021-02-10T11:31:06.041 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:06.036 7f70157fa700 1 -- 172.21.2.102:0/2574764611 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f7008006030 con 0x7f70180954c0 2021-02-10T11:31:06.041 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:06.036 7f70157fa700 1 --2- 172.21.2.102:0/2574764611 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6ffc0186c0 0x7f6ffc01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:06.042 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:06.036 7f70157fa700 1 -- 172.21.2.102:0/2574764611 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f7008032870 con 0x7f70180954c0 2021-02-10T11:31:06.042 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:06.036 7f701cd4d700 1 --2- 172.21.2.102:0/2574764611 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6ffc0186c0 0x7f6ffc01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:06.042 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:06.037 7f701cd4d700 1 --2- 172.21.2.102:0/2574764611 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6ffc0186c0 0x7f6ffc01abc0 crc :-1 s=READY pgs=930 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:06.067 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:06.066 7f701ef31700 1 -- 172.21.2.102:0/2574764611 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70180fe1c0 con 0x7f70180954c0 2021-02-10T11:31:06.070 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:06.069 7f70157fa700 1 -- 172.21.2.102:0/2574764611 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f70080067e0 con 0x7f70180954c0 2021-02-10T11:31:06.197 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:06.196 7f701ef31700 1 -- 172.21.2.102:0/2574764611 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f70180fe1c0 con 0x7f70180954c0 2021-02-10T11:31:06.197 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:06.196 7f70157fa700 1 -- 172.21.2.102:0/2574764611 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v103) v1 ==== 92+0+10 (secure 0 0 0) 0x7f7008006030 con 0x7f70180954c0 2021-02-10T11:31:06.198 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:31:06.199 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:06.198 7f701ef31700 1 -- 172.21.2.102:0/2574764611 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6ffc0186c0 msgr2=0x7f6ffc01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:06.199 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:06.198 7f701ef31700 1 --2- 172.21.2.102:0/2574764611 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6ffc0186c0 0x7f6ffc01abc0 crc :-1 s=READY pgs=930 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:06.199 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:06.198 7f701ef31700 1 -- 172.21.2.102:0/2574764611 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f70180954c0 msgr2=0x7f7018096300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:06.199 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:06.198 7f701ef31700 1 --2- 172.21.2.102:0/2574764611 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f70180954c0 0x7f7018096300 secure :-1 s=READY pgs=2094 cs=0 l=1 rev1=1 rx=0x7f70080096a0 tx=0x7f700800bd00).stop 2021-02-10T11:31:06.200 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:06.199 7f701ef31700 1 -- 172.21.2.102:0/2574764611 shutdown_connections 2021-02-10T11:31:06.200 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:06.199 7f701ef31700 1 --2- 172.21.2.102:0/2574764611 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f70180958c0 0x7f7018096920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:06.200 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:06.199 7f701ef31700 1 --2- 172.21.2.102:0/2574764611 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f70180954c0 0x7f7018096300 unknown :-1 s=CLOSED pgs=2094 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:06.200 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:06.199 7f701ef31700 1 --2- 172.21.2.102:0/2574764611 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6ffc0186c0 0x7f6ffc01abc0 unknown :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:06.200 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:06.199 7f701ef31700 1 --2- 172.21.2.102:0/2574764611 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7018085bc0 0x7f70180863e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:06.200 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:06.199 7f701ef31700 1 -- 172.21.2.102:0/2574764611 >> 172.21.2.102:0/2574764611 conn(0x7f70180bef00 msgr2=0x7f701809d1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:06.200 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:06.199 7f701ef31700 1 -- 172.21.2.102:0/2574764611 shutdown_connections 2021-02-10T11:31:06.200 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:06.199 7f701ef31700 1 -- 172.21.2.102:0/2574764611 wait complete. 2021-02-10T11:31:06.208 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-02-10T11:31:06.639 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:06.622 7f242e2f6700 1 Processor -- start 2021-02-10T11:31:06.639 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:06.622 7f242e2f6700 1 -- start start 2021-02-10T11:31:06.639 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:06.622 7f242e2f6700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f24280d1870 0x7f24281b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:06.640 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:06.622 7f242e2f6700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f24280cff70 0x7f2428109130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:06.640 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:06.622 7f242e2f6700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2428109a60 con 0x7f24281b0ae0 2021-02-10T11:31:06.640 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:06.622 7f242e2f6700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2428109750 con 0x7f24280cff70 2021-02-10T11:31:06.640 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:06.622 7f242e2f6700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2428088070 con 0x7f24280d1870 2021-02-10T11:31:06.640 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:06.623 7f242c913700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f24280cff70 0x7f2428109130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:06.640 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:06.623 7f242c913700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f24280cff70 0x7f2428109130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:33436/0 (socket says 172.21.2.102:33436) 2021-02-10T11:31:06.640 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:06.623 7f242c913700 1 -- 172.21.2.102:0/2630006954 learned_addr learned my addr 172.21.2.102:0/2630006954 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:31:06.640 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:06.623 7f242c913700 1 -- 172.21.2.102:0/2630006954 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f24280d1870 msgr2=0x7f24281b04c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:31:06.640 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:06.623 7f242c913700 1 --2- 172.21.2.102:0/2630006954 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f24280d1870 0x7f24281b04c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:06.640 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:06.623 7f242c913700 1 -- 172.21.2.102:0/2630006954 >> v1:172.21.2.113:6789/0 conn(0x7f24281b0ae0 legacy=0x7f2428108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:06.640 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:06.623 7f242c913700 1 -- 172.21.2.102:0/2630006954 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2428088260 con 0x7f24280cff70 2021-02-10T11:31:06.640 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:06.623 7f242c913700 1 --2- 172.21.2.102:0/2630006954 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f24280cff70 0x7f2428109130 secure :-1 s=READY pgs=2095 cs=0 l=1 rev1=1 rx=0x7f2420006f00 tx=0x7f2420009f10).ready entity=mon.0 client_cookie=e85acab71ffdfe5f server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:06.640 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:06.623 7f2426ffd700 1 -- 172.21.2.102:0/2630006954 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f242000b3b0 con 0x7f24280cff70 2021-02-10T11:31:06.641 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:06.623 7f2426ffd700 1 -- 172.21.2.102:0/2630006954 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f242000b670 con 0x7f24280cff70 2021-02-10T11:31:06.641 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:06.623 7f2426ffd700 1 -- 172.21.2.102:0/2630006954 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f242000bb90 con 0x7f24280cff70 2021-02-10T11:31:06.641 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:06.623 7f242e2f6700 1 -- 172.21.2.102:0/2630006954 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f24280cff70 msgr2=0x7f2428109130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:06.641 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:06.623 7f242e2f6700 1 --2- 172.21.2.102:0/2630006954 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f24280cff70 0x7f2428109130 secure :-1 s=READY pgs=2095 cs=0 l=1 rev1=1 rx=0x7f2420006f00 tx=0x7f2420009f10).stop 2021-02-10T11:31:06.641 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:06.624 7f242e2f6700 1 -- 172.21.2.102:0/2630006954 shutdown_connections 2021-02-10T11:31:06.641 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:06.624 7f242e2f6700 1 --2- 172.21.2.102:0/2630006954 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f24280cff70 0x7f2428109130 unknown :-1 s=CLOSED pgs=2095 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:06.641 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:06.624 7f242e2f6700 1 --2- 172.21.2.102:0/2630006954 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f24280d1870 0x7f24281b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:06.641 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:06.624 7f242e2f6700 1 -- 172.21.2.102:0/2630006954 >> 172.21.2.102:0/2630006954 conn(0x7f24280ce420 msgr2=0x7f24280d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:06.641 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:06.624 7f242e2f6700 1 -- 172.21.2.102:0/2630006954 shutdown_connections 2021-02-10T11:31:06.641 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:06.624 7f242e2f6700 1 -- 172.21.2.102:0/2630006954 wait complete. 2021-02-10T11:31:06.643 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:06.639 7f242e2f6700 1 Processor -- start 2021-02-10T11:31:06.643 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:06.639 7f242e2f6700 1 -- start start 2021-02-10T11:31:06.643 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:06.639 7f242e2f6700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f24280c2370 0x7f242818f870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:06.644 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:06.639 7f242e2f6700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2428084cb0 0x7f24281a7470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:06.644 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:06.639 7f242e2f6700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f24281a7a90 0x7f2428085700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:06.644 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:06.639 7f242e2f6700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24280be840 con 0x7f24280c2370 2021-02-10T11:31:06.644 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:06.639 7f242e2f6700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24280c29a0 con 0x7f2428084cb0 2021-02-10T11:31:06.644 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:06.639 7f242e2f6700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24281a7e90 con 0x7f24281a7a90 2021-02-10T11:31:06.644 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:06.639 7f2427fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f24280c2370 0x7f242818f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:06.644 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:06.639 7f2427fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f24280c2370 0x7f242818f870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:33444/0 (socket says 172.21.2.102:33444) 2021-02-10T11:31:06.644 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:06.639 7f2427fff700 1 -- 172.21.2.102:0/1161121629 learned_addr learned my addr 172.21.2.102:0/1161121629 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:31:06.644 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:06.640 7f242c913700 1 --2- 172.21.2.102:0/1161121629 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f24281a7a90 0x7f2428085700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:06.644 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:06.640 7f24277fe700 1 --2- 172.21.2.102:0/1161121629 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2428084cb0 0x7f24281a7470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:06.644 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:06.640 7f2427fff700 1 -- 172.21.2.102:0/1161121629 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f24281a7a90 msgr2=0x7f2428085700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:06.644 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:06.640 7f2427fff700 1 --2- 172.21.2.102:0/1161121629 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f24281a7a90 0x7f2428085700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:06.644 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:06.640 7f2427fff700 1 -- 172.21.2.102:0/1161121629 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2428084cb0 msgr2=0x7f24281a7470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:06.645 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:06.640 7f2427fff700 1 --2- 172.21.2.102:0/1161121629 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2428084cb0 0x7f24281a7470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:06.645 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:06.640 7f2427fff700 1 -- 172.21.2.102:0/1161121629 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24281a80b0 con 0x7f24280c2370 2021-02-10T11:31:06.645 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:06.640 7f2427fff700 1 --2- 172.21.2.102:0/1161121629 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f24280c2370 0x7f242818f870 secure :-1 s=READY pgs=2096 cs=0 l=1 rev1=1 rx=0x7f241800b560 tx=0x7f241800b900).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:06.645 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:06.640 7f2424ff9700 1 -- 172.21.2.102:0/1161121629 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f241800bcf0 con 0x7f24280c2370 2021-02-10T11:31:06.645 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:06.640 7f2424ff9700 1 -- 172.21.2.102:0/1161121629 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2418006030 con 0x7f24280c2370 2021-02-10T11:31:06.645 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:06.640 7f242e2f6700 1 -- 172.21.2.102:0/1161121629 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24281a80b0 con 0x7f24280c2370 2021-02-10T11:31:06.645 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:06.640 7f242e2f6700 1 -- 172.21.2.102:0/1161121629 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2428086010 con 0x7f24280c2370 2021-02-10T11:31:06.645 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:06.640 7f2424ff9700 1 -- 172.21.2.102:0/1161121629 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f24180054d0 con 0x7f24280c2370 2021-02-10T11:31:06.645 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:06.641 7f2424ff9700 1 -- 172.21.2.102:0/1161121629 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f2418006210 con 0x7f24280c2370 2021-02-10T11:31:06.645 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:06.641 7f2424ff9700 1 --2- 172.21.2.102:0/1161121629 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2408018730 0x7f240801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:06.645 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:06.641 7f2424ff9700 1 -- 172.21.2.102:0/1161121629 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f24180327e0 con 0x7f24280c2370 2021-02-10T11:31:06.645 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:06.641 7f24277fe700 1 --2- 172.21.2.102:0/1161121629 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2408018730 0x7f240801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:06.645 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:06.641 7f24277fe700 1 --2- 172.21.2.102:0/1161121629 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2408018730 0x7f240801abc0 crc :-1 s=READY pgs=931 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:06.674 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:06.673 7f242e2f6700 1 -- 172.21.2.102:0/1161121629 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2428082a70 con 0x7f24280c2370 2021-02-10T11:31:06.677 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:06.675 7f2424ff9700 1 -- 172.21.2.102:0/1161121629 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f2418000940 con 0x7f24280c2370 2021-02-10T11:31:06.805 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:06.804 7f242e2f6700 1 -- 172.21.2.102:0/1161121629 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7f2428082a70 con 0x7f24280c2370 2021-02-10T11:31:06.805 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:06.804 7f2424ff9700 1 -- 172.21.2.102:0/1161121629 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v103) v1 ==== 104+0+10 (secure 0 0 0) 0x7f2418006030 con 0x7f24280c2370 2021-02-10T11:31:06.806 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:31:06.807 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:06.806 7f242e2f6700 1 -- 172.21.2.102:0/1161121629 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2408018730 msgr2=0x7f240801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:06.808 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:06.806 7f242e2f6700 1 --2- 172.21.2.102:0/1161121629 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2408018730 0x7f240801abc0 crc :-1 s=READY pgs=931 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:06.808 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:06.806 7f242e2f6700 1 -- 172.21.2.102:0/1161121629 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f24280c2370 msgr2=0x7f242818f870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:06.808 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:06.806 7f242e2f6700 1 --2- 172.21.2.102:0/1161121629 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f24280c2370 0x7f242818f870 secure :-1 s=READY pgs=2096 cs=0 l=1 rev1=1 rx=0x7f241800b560 tx=0x7f241800b900).stop 2021-02-10T11:31:06.808 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:06.807 7f242e2f6700 1 -- 172.21.2.102:0/1161121629 shutdown_connections 2021-02-10T11:31:06.808 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:06.807 7f242e2f6700 1 --2- 172.21.2.102:0/1161121629 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f24281a7a90 0x7f2428085700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:06.808 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:06.807 7f242e2f6700 1 --2- 172.21.2.102:0/1161121629 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2408018730 0x7f240801abc0 unknown :-1 s=CLOSED pgs=931 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:06.808 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:06.807 7f242e2f6700 1 --2- 172.21.2.102:0/1161121629 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2428084cb0 0x7f24281a7470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:06.808 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:06.807 7f242e2f6700 1 --2- 172.21.2.102:0/1161121629 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f24280c2370 0x7f242818f870 unknown :-1 s=CLOSED pgs=2096 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:06.808 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:06.807 7f242e2f6700 1 -- 172.21.2.102:0/1161121629 >> 172.21.2.102:0/1161121629 conn(0x7f24280b5dc0 msgr2=0x7f24280c2d90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:06.808 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:06.807 7f242e2f6700 1 -- 172.21.2.102:0/1161121629 shutdown_connections 2021-02-10T11:31:06.809 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:06.807 7f242e2f6700 1 -- 172.21.2.102:0/1161121629 wait complete. 2021-02-10T11:31:06.816 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-02-10T11:31:07.247 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:07.230 7f0769a0f700 1 Processor -- start 2021-02-10T11:31:07.247 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:07.230 7f0769a0f700 1 -- start start 2021-02-10T11:31:07.247 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:07.230 7f0769a0f700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f076410aff0 0x7f07640cf750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:07.247 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:07.230 7f0769a0f700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07640fe1c0 con 0x7f07640d1c70 2021-02-10T11:31:07.247 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:07.230 7f0769a0f700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07641b0cb0 con 0x7f07640d1870 2021-02-10T11:31:07.247 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:07.230 7f0769a0f700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0764109a60 con 0x7f076410aff0 2021-02-10T11:31:07.247 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:07.230 7f0763fff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f076410aff0 0x7f07640cf750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:07.248 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:07.230 7f0763fff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f076410aff0 0x7f07640cf750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:33452/0 (socket says 172.21.2.102:33452) 2021-02-10T11:31:07.248 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:07.230 7f0763fff700 1 -- 172.21.2.102:0/2234314216 learned_addr learned my addr 172.21.2.102:0/2234314216 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:31:07.248 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:07.231 7f0763fff700 1 -- 172.21.2.102:0/2234314216 >> v1:172.21.2.113:6789/0 conn(0x7f07640d1870 legacy=0x7f07641b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:07.248 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:07.231 7f0763fff700 1 -- 172.21.2.102:0/2234314216 >> v1:172.21.2.102:6789/0 conn(0x7f07640d1c70 legacy=0x7f07640cef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:07.248 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:07.231 7f0763fff700 1 -- 172.21.2.102:0/2234314216 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f076410b3f0 con 0x7f076410aff0 2021-02-10T11:31:07.248 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:07.231 7f0763fff700 1 --2- 172.21.2.102:0/2234314216 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f076410aff0 0x7f07640cf750 secure :-1 s=READY pgs=2098 cs=0 l=1 rev1=1 rx=0x7f075c006f00 tx=0x7f075c009f10).ready entity=mon.0 client_cookie=52dce038fa975712 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:07.248 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:07.231 7f07627fc700 1 -- 172.21.2.102:0/2234314216 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f075c00b3b0 con 0x7f076410aff0 2021-02-10T11:31:07.248 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:07.231 7f07627fc700 1 -- 172.21.2.102:0/2234314216 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f075c00b730 con 0x7f076410aff0 2021-02-10T11:31:07.248 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:07.231 7f07627fc700 1 -- 172.21.2.102:0/2234314216 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f075c00bc50 con 0x7f076410aff0 2021-02-10T11:31:07.248 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:07.231 7f0769a0f700 1 -- 172.21.2.102:0/2234314216 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f076410aff0 msgr2=0x7f07640cf750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:07.248 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:07.231 7f0769a0f700 1 --2- 172.21.2.102:0/2234314216 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f076410aff0 0x7f07640cf750 secure :-1 s=READY pgs=2098 cs=0 l=1 rev1=1 rx=0x7f075c006f00 tx=0x7f075c009f10).stop 2021-02-10T11:31:07.248 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:07.232 7f0769a0f700 1 -- 172.21.2.102:0/2234314216 shutdown_connections 2021-02-10T11:31:07.249 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:07.232 7f0769a0f700 1 --2- 172.21.2.102:0/2234314216 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f076410aff0 0x7f07640cf750 unknown :-1 s=CLOSED pgs=2098 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:07.249 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:07.232 7f0769a0f700 1 -- 172.21.2.102:0/2234314216 >> 172.21.2.102:0/2234314216 conn(0x7f07640ce420 msgr2=0x7f07640d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:07.249 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:07.232 7f0769a0f700 1 -- 172.21.2.102:0/2234314216 shutdown_connections 2021-02-10T11:31:07.249 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:07.232 7f0769a0f700 1 -- 172.21.2.102:0/2234314216 wait complete. 2021-02-10T11:31:07.250 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:07.246 7f0769a0f700 1 Processor -- start 2021-02-10T11:31:07.250 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:07.246 7f0769a0f700 1 -- start start 2021-02-10T11:31:07.250 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:07.247 7f0769a0f700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f07641b8c10 0x7f07640ff210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:07.250 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:07.247 7f0769a0f700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f07640a00d0 0x7f07640a04d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:07.250 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:07.247 7f0769a0f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f07641c94e0 0x7f07641c98e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:07.251 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:07.247 7f0769a0f700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07640ff830 con 0x7f07641c94e0 2021-02-10T11:31:07.251 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:07.247 7f0769a0f700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07641c9f00 con 0x7f07641b8c10 2021-02-10T11:31:07.251 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:07.247 7f0769a0f700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07641ca120 con 0x7f07640a00d0 2021-02-10T11:31:07.251 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:07.247 7f0763fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f07641c94e0 0x7f07641c98e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:07.251 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:07.247 7f0763fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f07641c94e0 0x7f07641c98e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:33458/0 (socket says 172.21.2.102:33458) 2021-02-10T11:31:07.251 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:07.247 7f0763fff700 1 -- 172.21.2.102:0/1503670680 learned_addr learned my addr 172.21.2.102:0/1503670680 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:31:07.251 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:07.247 7f0763fff700 1 -- 172.21.2.102:0/1503670680 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f07640a00d0 msgr2=0x7f07640a04d0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:31:07.251 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:07.247 7f0763fff700 1 --2- 172.21.2.102:0/1503670680 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f07640a00d0 0x7f07640a04d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:07.251 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:07.247 7f0763fff700 1 -- 172.21.2.102:0/1503670680 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f07641b8c10 msgr2=0x7f07640ff210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:07.251 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:07.247 7f0763fff700 1 --2- 172.21.2.102:0/1503670680 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f07641b8c10 0x7f07640ff210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:07.251 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:07.247 7f0763fff700 1 -- 172.21.2.102:0/1503670680 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07640aaf70 con 0x7f07641c94e0 2021-02-10T11:31:07.251 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:07.247 7f0763fff700 1 --2- 172.21.2.102:0/1503670680 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f07641c94e0 0x7f07641c98e0 secure :-1 s=READY pgs=2099 cs=0 l=1 rev1=1 rx=0x7f075c009390 tx=0x7f075c00bd40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:07.251 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:07.248 7f075bfff700 1 -- 172.21.2.102:0/1503670680 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f075c003030 con 0x7f07641c94e0 2021-02-10T11:31:07.252 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:07.248 7f075bfff700 1 -- 172.21.2.102:0/1503670680 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f075c0068e0 con 0x7f07641c94e0 2021-02-10T11:31:07.252 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:07.248 7f0769a0f700 1 -- 172.21.2.102:0/1503670680 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07640aaf70 con 0x7f07641c94e0 2021-02-10T11:31:07.252 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:07.248 7f0769a0f700 1 -- 172.21.2.102:0/1503670680 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07640ab500 con 0x7f07641c94e0 2021-02-10T11:31:07.252 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:07.248 7f075bfff700 1 -- 172.21.2.102:0/1503670680 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f075c00c4e0 con 0x7f07641c94e0 2021-02-10T11:31:07.252 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:07.248 7f075bfff700 1 -- 172.21.2.102:0/1503670680 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f075c001270 con 0x7f07641c94e0 2021-02-10T11:31:07.252 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:07.248 7f075bfff700 1 --2- 172.21.2.102:0/1503670680 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f07480186c0 0x7f074801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:07.252 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:07.248 7f075bfff700 1 -- 172.21.2.102:0/1503670680 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f075c032b60 con 0x7f07641c94e0 2021-02-10T11:31:07.252 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:07.248 7f07637fe700 1 --2- 172.21.2.102:0/1503670680 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f07480186c0 0x7f074801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:07.252 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:07.249 7f07637fe700 1 --2- 172.21.2.102:0/1503670680 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f07480186c0 0x7f074801abc0 crc :-1 s=READY pgs=932 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:07.281 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:07.280 7f0769a0f700 1 -- 172.21.2.102:0/1503670680 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07640aaa40 con 0x7f07641c94e0 2021-02-10T11:31:07.283 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:07.282 7f075bfff700 1 -- 172.21.2.102:0/1503670680 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f075c009430 con 0x7f07641c94e0 2021-02-10T11:31:07.408 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:07.408 7f0769a0f700 1 -- 172.21.2.102:0/1503670680 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7f07640aaa40 con 0x7f07641c94e0 2021-02-10T11:31:07.409 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:07.408 7f075bfff700 1 -- 172.21.2.102:0/1503670680 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v103) v1 ==== 100+0+10 (secure 0 0 0) 0x7f075c01f8a0 con 0x7f07641c94e0 2021-02-10T11:31:07.409 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:31:07.410 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:07.410 7f0769a0f700 1 -- 172.21.2.102:0/1503670680 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f07480186c0 msgr2=0x7f074801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:07.411 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:07.410 7f0769a0f700 1 --2- 172.21.2.102:0/1503670680 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f07480186c0 0x7f074801abc0 crc :-1 s=READY pgs=932 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:07.411 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:07.410 7f0769a0f700 1 -- 172.21.2.102:0/1503670680 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f07641c94e0 msgr2=0x7f07641c98e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:07.411 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:07.410 7f0769a0f700 1 --2- 172.21.2.102:0/1503670680 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f07641c94e0 0x7f07641c98e0 secure :-1 s=READY pgs=2099 cs=0 l=1 rev1=1 rx=0x7f075c009390 tx=0x7f075c00bd40).stop 2021-02-10T11:31:07.411 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:07.410 7f0769a0f700 1 -- 172.21.2.102:0/1503670680 shutdown_connections 2021-02-10T11:31:07.411 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:07.410 7f0769a0f700 1 --2- 172.21.2.102:0/1503670680 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f07641c94e0 0x7f07641c98e0 unknown :-1 s=CLOSED pgs=2099 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:07.412 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:07.410 7f0769a0f700 1 --2- 172.21.2.102:0/1503670680 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f07640a00d0 0x7f07640a04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:07.412 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:07.410 7f0769a0f700 1 --2- 172.21.2.102:0/1503670680 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f07480186c0 0x7f074801abc0 unknown :-1 s=CLOSED pgs=932 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:07.412 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:07.410 7f0769a0f700 1 --2- 172.21.2.102:0/1503670680 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f07641b8c10 0x7f07640ff210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:07.412 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:07.410 7f0769a0f700 1 -- 172.21.2.102:0/1503670680 >> 172.21.2.102:0/1503670680 conn(0x7f07640b89c0 msgr2=0x7f07640bea50 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:07.412 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:07.410 7f0769a0f700 1 -- 172.21.2.102:0/1503670680 shutdown_connections 2021-02-10T11:31:07.412 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:07.410 7f0769a0f700 1 -- 172.21.2.102:0/1503670680 wait complete. 2021-02-10T11:31:07.419 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-02-10T11:31:07.420 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:31:07.849 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:07.830 7f20925ce700 1 Processor -- start 2021-02-10T11:31:07.849 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:07.830 7f20925ce700 1 -- start start 2021-02-10T11:31:07.849 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:07.831 7f20925ce700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f208c0d1c70 0x7f208c0f20a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:07.849 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:07.831 7f20925ce700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f208c0f0b10 con 0x7f208c0d0340 2021-02-10T11:31:07.849 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:07.831 7f20925ce700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f208c1b0be0 con 0x7f208c0d1870 2021-02-10T11:31:07.849 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:07.831 7f20925ce700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f208c0e5ed0 con 0x7f208c0d1c70 2021-02-10T11:31:07.849 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:07.831 7f208b7fe700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f208c0d0340 0x7f208c0ef640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:53918/0 (socket says 172.21.2.102:53918) 2021-02-10T11:31:07.850 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:07.831 7f208b7fe700 1 -- 172.21.2.102:0/1152151798 learned_addr learned my addr 172.21.2.102:0/1152151798 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:31:07.850 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:07.831 7f2090beb700 1 --2- 172.21.2.102:0/1152151798 >> v2:172.21.2.113:3300/0 conn(0x7f208c0d1c70 0x7f208c0f20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:07.850 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:07.831 7f208affd700 1 -- 172.21.2.102:0/1152151798 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 351319368 0 0) 0x7f20800009b0 con 0x7f208c0d0340 2021-02-10T11:31:07.850 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:07.831 7f208affd700 1 -- 172.21.2.102:0/1152151798 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2074002fe0 con 0x7f208c0d0340 2021-02-10T11:31:07.850 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:07.832 7f208affd700 1 -- 172.21.2.102:0/1152151798 <== mon.1 v1:172.21.2.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 883844364 0 0) 0x7f207c000ff0 con 0x7f208c0d1870 2021-02-10T11:31:07.850 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:07.832 7f208affd700 1 -- 172.21.2.102:0/1152151798 --> v1:172.21.2.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2074003210 con 0x7f208c0d1870 2021-02-10T11:31:07.850 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:07.832 7f208affd700 1 -- 172.21.2.102:0/1152151798 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2494015160 0 0) 0x7f20800010b0 con 0x7f208c0d0340 2021-02-10T11:31:07.850 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:07.832 7f208affd700 1 -- 172.21.2.102:0/1152151798 >> v2:172.21.2.113:3300/0 conn(0x7f208c0d1c70 msgr2=0x7f208c0f20a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:07.850 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:07.832 7f208affd700 1 --2- 172.21.2.102:0/1152151798 >> v2:172.21.2.113:3300/0 conn(0x7f208c0d1c70 0x7f208c0f20a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:07.850 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:07.832 7f208affd700 1 -- 172.21.2.102:0/1152151798 >> v1:172.21.2.113:6789/0 conn(0x7f208c0d1870 legacy=0x7f208c1b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:07.850 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:07.832 7f208affd700 1 -- 172.21.2.102:0/1152151798 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f208c0efe30 con 0x7f208c0d0340 2021-02-10T11:31:07.851 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:07.832 7f208affd700 1 -- 172.21.2.102:0/1152151798 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7f20800015b0 con 0x7f208c0d0340 2021-02-10T11:31:07.851 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:07.832 7f208affd700 1 -- 172.21.2.102:0/1152151798 >> v1:172.21.2.102:6789/0 conn(0x7f208c0d0340 legacy=0x7f208c0ef640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:07.851 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:07.832 7f208affd700 1 --2- 172.21.2.102:0/1152151798 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2074003b90 0x7f2074005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:07.851 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:07.832 7f208affd700 1 --2- 172.21.2.102:0/1152151798 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f20740067a0 0x7f2074008ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:07.851 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:07.832 7f208affd700 1 --2- 172.21.2.102:0/1152151798 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f20740095a0 0x7f207400bac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:07.851 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:07.832 7f208affd700 1 -- 172.21.2.102:0/1152151798 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f207400c2f0 con 0x7f20740095a0 2021-02-10T11:31:07.851 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:07.832 7f208affd700 1 -- 172.21.2.102:0/1152151798 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20740504e0 con 0x7f20740067a0 2021-02-10T11:31:07.851 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:07.832 7f208affd700 1 -- 172.21.2.102:0/1152151798 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2074050700 con 0x7f2074003b90 2021-02-10T11:31:07.851 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:07.832 7f2090beb700 1 --2- 172.21.2.102:0/1152151798 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f20740095a0 0x7f207400bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:07.851 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:07.832 7f208b7fe700 1 --2- 172.21.2.102:0/1152151798 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f20740067a0 0x7f2074008ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:07.851 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:07.832 7f208bfff700 1 --2- 172.21.2.102:0/1152151798 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2074003b90 0x7f2074005fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:07.851 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:07.832 7f208b7fe700 1 -- 172.21.2.102:0/1152151798 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2074003b90 msgr2=0x7f2074005fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:07.851 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:07.832 7f208b7fe700 1 --2- 172.21.2.102:0/1152151798 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2074003b90 0x7f2074005fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:07.852 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:07.832 7f208b7fe700 1 -- 172.21.2.102:0/1152151798 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f20740095a0 msgr2=0x7f207400bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:07.852 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:07.832 7f208b7fe700 1 --2- 172.21.2.102:0/1152151798 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f20740095a0 0x7f207400bac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:07.852 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:07.832 7f208b7fe700 1 -- 172.21.2.102:0/1152151798 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20740509a0 con 0x7f20740067a0 2021-02-10T11:31:07.852 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:07.832 7f2090beb700 1 --2- 172.21.2.102:0/1152151798 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f20740095a0 0x7f207400bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:31:07.852 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:07.832 7f2090beb700 1 -- 172.21.2.102:0/1152151798 reap_dead start 2021-02-10T11:31:07.852 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:07.833 7f208b7fe700 1 --2- 172.21.2.102:0/1152151798 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f20740067a0 0x7f2074008ca0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 rx=0x7f2080001360 tx=0x7f2080007cf0).ready entity=mon.1 client_cookie=c64e250fcf9100e1 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:07.852 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:07.833 7f208affd700 1 -- 172.21.2.102:0/1152151798 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f208000a520 con 0x7f20740067a0 2021-02-10T11:31:07.852 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:07.833 7f208affd700 1 -- 172.21.2.102:0/1152151798 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f208000a7a0 con 0x7f20740067a0 2021-02-10T11:31:07.852 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:07.833 7f208affd700 1 -- 172.21.2.102:0/1152151798 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f208000acc0 con 0x7f20740067a0 2021-02-10T11:31:07.852 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:07.833 7f20925ce700 1 -- 172.21.2.102:0/1152151798 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f20740067a0 msgr2=0x7f2074008ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:07.852 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:07.833 7f20925ce700 1 --2- 172.21.2.102:0/1152151798 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f20740067a0 0x7f2074008ca0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 rx=0x7f2080001360 tx=0x7f2080007cf0).stop 2021-02-10T11:31:07.852 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:07.833 7f20925ce700 1 -- 172.21.2.102:0/1152151798 shutdown_connections 2021-02-10T11:31:07.853 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:07.833 7f20925ce700 1 --2- 172.21.2.102:0/1152151798 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f20740067a0 0x7f2074008ca0 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:07.853 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:07.833 7f20925ce700 1 -- 172.21.2.102:0/1152151798 >> 172.21.2.102:0/1152151798 conn(0x7f208c0ce450 msgr2=0x7f208c0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:07.853 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:07.833 7f20925ce700 1 -- 172.21.2.102:0/1152151798 shutdown_connections 2021-02-10T11:31:07.853 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:07.833 7f20925ce700 1 -- 172.21.2.102:0/1152151798 wait complete. 2021-02-10T11:31:07.853 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:07.848 7f20925ce700 1 Processor -- start 2021-02-10T11:31:07.853 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:07.848 7f20925ce700 1 -- start start 2021-02-10T11:31:07.854 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:07.848 7f20925ce700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f208c195650 0x7f208c0b5560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:07.854 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:07.848 7f20925ce700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f208c0b5b80 0x7f208c1a2e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:07.854 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:07.848 7f20925ce700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f208c0acfa0 0x7f208c1a3440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:07.855 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:07.848 7f20925ce700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f208c0b6010 con 0x7f208c0acfa0 2021-02-10T11:31:07.855 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:07.848 7f20925ce700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f208c1a3a60 con 0x7f208c0b5b80 2021-02-10T11:31:07.855 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:07.848 7f20925ce700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f208c1a3c40 con 0x7f208c195650 2021-02-10T11:31:07.855 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:07.849 7f2090beb700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f208c0acfa0 0x7f208c1a3440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:07.855 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:07.849 7f2090beb700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f208c0acfa0 0x7f208c1a3440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:33478/0 (socket says 172.21.2.102:33478) 2021-02-10T11:31:07.855 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:07.849 7f2090beb700 1 -- 172.21.2.102:0/2124556292 learned_addr learned my addr 172.21.2.102:0/2124556292 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:31:07.855 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:07.849 7f2090beb700 1 -- 172.21.2.102:0/2124556292 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f208c195650 msgr2=0x7f208c0b5560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:07.855 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:07.849 7f2090beb700 1 --2- 172.21.2.102:0/2124556292 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f208c195650 0x7f208c0b5560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:07.855 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:07.849 7f2090beb700 1 -- 172.21.2.102:0/2124556292 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f208c0b5b80 msgr2=0x7f208c1a2e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:07.855 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:07.849 7f2090beb700 1 --2- 172.21.2.102:0/2124556292 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f208c0b5b80 0x7f208c1a2e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:07.855 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:07.849 7f2090beb700 1 -- 172.21.2.102:0/2124556292 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f208c0bcae0 con 0x7f208c0acfa0 2021-02-10T11:31:07.855 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:07.849 7f2090beb700 1 --2- 172.21.2.102:0/2124556292 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f208c0acfa0 0x7f208c1a3440 secure :-1 s=READY pgs=2101 cs=0 l=1 rev1=1 rx=0x7f208400b970 tx=0x7f208400bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:07.855 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:07.849 7f2088ff9700 1 -- 172.21.2.102:0/2124556292 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2084000f90 con 0x7f208c0acfa0 2021-02-10T11:31:07.855 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:07.849 7f2088ff9700 1 -- 172.21.2.102:0/2124556292 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20840011f0 con 0x7f208c0acfa0 2021-02-10T11:31:07.856 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:07.849 7f2088ff9700 1 -- 172.21.2.102:0/2124556292 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2084001690 con 0x7f208c0acfa0 2021-02-10T11:31:07.856 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:07.849 7f20925ce700 1 -- 172.21.2.102:0/2124556292 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f208c0bcae0 con 0x7f208c0acfa0 2021-02-10T11:31:07.856 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:07.850 7f20925ce700 1 -- 172.21.2.102:0/2124556292 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f208c0bd070 con 0x7f208c0acfa0 2021-02-10T11:31:07.856 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:07.850 7f2088ff9700 1 -- 172.21.2.102:0/2124556292 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f20840013d0 con 0x7f208c0acfa0 2021-02-10T11:31:07.856 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:07.850 7f2088ff9700 1 --2- 172.21.2.102:0/2124556292 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f206c0186c0 0x7f206c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:07.856 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:07.850 7f2088ff9700 1 -- 172.21.2.102:0/2124556292 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f2084032980 con 0x7f208c0acfa0 2021-02-10T11:31:07.856 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:07.850 7f208bfff700 1 --2- 172.21.2.102:0/2124556292 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f206c0186c0 0x7f206c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:07.856 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:07.850 7f208bfff700 1 --2- 172.21.2.102:0/2124556292 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f206c0186c0 0x7f206c01abc0 crc :-1 s=READY pgs=933 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:07.881 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:07.880 7f20925ce700 1 -- 172.21.2.102:0/2124556292 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f208c0bca80 con 0x7f208c0acfa0 2021-02-10T11:31:07.883 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:07.883 7f2088ff9700 1 -- 172.21.2.102:0/2124556292 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f208402cfd0 con 0x7f208c0acfa0 2021-02-10T11:31:08.010 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:08.009 7f20925ce700 1 -- 172.21.2.102:0/2124556292 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f208c0bca80 con 0x7f208c0acfa0 2021-02-10T11:31:08.011 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:08.010 7f2088ff9700 1 -- 172.21.2.102:0/2124556292 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+2101 (secure 0 0 0) 0x7f208402cfd0 con 0x7f208c0acfa0 2021-02-10T11:31:08.011 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:31:08.011 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":82,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10056,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.102:6835/711305601","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6834","nonce":711305601},{"type":"v1","addr":"172.21.2.102:6835","nonce":711305601}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":78}],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:30:58.047057","modified":"2021-02-10 11:31:00.055485","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6353},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6353":{"gid":6353,"name":"b","rank":0,"incarnation":79,"state":"up:active","state_seq":3,"addr":"172.21.2.113:6833/1091618344","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6832","nonce":1091618344},{"type":"v1","addr":"172.21.2.113:6833","nonce":1091618344}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":10}]} 2021-02-10T11:31:08.011 INFO:teuthology.orchestra.run.gibba002.stderr:dumped fsmap epoch 82 2021-02-10T11:31:08.012 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:08.011 7f20925ce700 1 -- 172.21.2.102:0/2124556292 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f206c0186c0 msgr2=0x7f206c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:08.013 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:08.011 7f20925ce700 1 --2- 172.21.2.102:0/2124556292 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f206c0186c0 0x7f206c01abc0 crc :-1 s=READY pgs=933 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:08.013 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:08.011 7f20925ce700 1 -- 172.21.2.102:0/2124556292 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f208c0acfa0 msgr2=0x7f208c1a3440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:08.013 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:08.011 7f20925ce700 1 --2- 172.21.2.102:0/2124556292 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f208c0acfa0 0x7f208c1a3440 secure :-1 s=READY pgs=2101 cs=0 l=1 rev1=1 rx=0x7f208400b970 tx=0x7f208400bd10).stop 2021-02-10T11:31:08.013 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:08.011 7f20925ce700 1 -- 172.21.2.102:0/2124556292 shutdown_connections 2021-02-10T11:31:08.013 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:08.011 7f20925ce700 1 --2- 172.21.2.102:0/2124556292 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f208c0acfa0 0x7f208c1a3440 unknown :-1 s=CLOSED pgs=2101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:08.013 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:08.012 7f20925ce700 1 --2- 172.21.2.102:0/2124556292 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f206c0186c0 0x7f206c01abc0 unknown :-1 s=CLOSED pgs=933 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:08.013 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:08.012 7f20925ce700 1 --2- 172.21.2.102:0/2124556292 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f208c0b5b80 0x7f208c1a2e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:08.013 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:08.012 7f20925ce700 1 --2- 172.21.2.102:0/2124556292 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f208c195650 0x7f208c0b5560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:08.013 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:08.012 7f20925ce700 1 -- 172.21.2.102:0/2124556292 >> 172.21.2.102:0/2124556292 conn(0x7f208c0b3040 msgr2=0x7f208c0c3250 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:08.013 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:08.012 7f20925ce700 1 -- 172.21.2.102:0/2124556292 shutdown_connections 2021-02-10T11:31:08.013 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:08.012 7f20925ce700 1 -- 172.21.2.102:0/2124556292 wait complete. 2021-02-10T11:31:08.223 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 82, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:30:58.047057', 'modified': '2021-02-10 11:31:00.055485', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6353}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6353': {'gid': 6353, 'name': 'b', 'rank': 0, 'incarnation': 79, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.113:6833/1091618344', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.113:6832', 'nonce': 1091618344}, {'type': 'v1', 'addr': '172.21.2.113:6833', 'nonce': 1091618344}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:31:08.223 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:31:08.223 DEBUG:teuthology.orchestra.run.gibba013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:31:08.622 INFO:teuthology.orchestra.run.gibba013.stdout:{ 2021-02-10T11:31:08.622 INFO:teuthology.orchestra.run.gibba013.stdout: "cluster_fsid": "68db30ea-dbce-4aef-a6df-f01c6fea0513", 2021-02-10T11:31:08.623 INFO:teuthology.orchestra.run.gibba013.stdout: "whoami": 0, 2021-02-10T11:31:08.623 INFO:teuthology.orchestra.run.gibba013.stdout: "id": 6353, 2021-02-10T11:31:08.623 INFO:teuthology.orchestra.run.gibba013.stdout: "want_state": "up:active", 2021-02-10T11:31:08.623 INFO:teuthology.orchestra.run.gibba013.stdout: "state": "up:active", 2021-02-10T11:31:08.623 INFO:teuthology.orchestra.run.gibba013.stdout: "rank_uptime": 10.573235309999999, 2021-02-10T11:31:08.623 INFO:teuthology.orchestra.run.gibba013.stdout: "mdsmap_epoch": 82, 2021-02-10T11:31:08.623 INFO:teuthology.orchestra.run.gibba013.stdout: "osdmap_epoch": 103, 2021-02-10T11:31:08.623 INFO:teuthology.orchestra.run.gibba013.stdout: "osdmap_epoch_barrier": 103, 2021-02-10T11:31:08.623 INFO:teuthology.orchestra.run.gibba013.stdout: "uptime": 16.211932714 2021-02-10T11:31:08.623 INFO:teuthology.orchestra.run.gibba013.stdout:} 2021-02-10T11:31:08.626 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "68db30ea-dbce-4aef-a6df-f01c6fea0513", "whoami": 0, "id": 6353, "want_state": "up:active", "state": "up:active", "rank_uptime": 10.573235309999999, "mdsmap_epoch": 82, "osdmap_epoch": 103, "osdmap_epoch_barrier": 103, "uptime": 16.211932714 } 2021-02-10T11:31:08.627 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2021-02-10T11:31:08.627 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2021-02-10T11:31:08.627 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@gibba002.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2021-02-10T11:31:08.627 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:31:08.627 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec mkdir -p /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:31:08.962 DEBUG:teuthology.orchestra.run.gibba002:> sudo modprobe fuse 2021-02-10T11:31:09.049 DEBUG:teuthology.orchestra.run.gibba002:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:31:09.053 DEBUG:teuthology.orchestra.run.gibba013:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:31:09.308 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:31:09.309 DEBUG:teuthology.orchestra.run.gibba002:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:31:09.652 INFO:teuthology.orchestra.run.gibba002.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:31:09.652 INFO:teuthology.orchestra.run.gibba002.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:31:09.653 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:31:09.654 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:31:09.654 DEBUG:teuthology.orchestra.run.gibba002:> ls /sys/fs/fuse/connections 2021-02-10T11:31:09.990 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2021-02-10T11:31:09.990 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:31:09.993 DEBUG:teuthology.orchestra.run.gibba002:> sudo modprobe fuse 2021-02-10T11:31:10.337 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:31:10.337 DEBUG:teuthology.orchestra.run.gibba002:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:31:10.403 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba002.stderr:ceph-fuse[178321]: starting ceph client 2021-02-10T11:31:10.403 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba002.stderr:2021-02-10 11:31:10.402 7fc9d9327f80 -1 init, newargv = 0x557a54979210 newargc=9 2021-02-10T11:31:10.409 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba002.stderr:ceph-fuse[178321]: starting fuse 2021-02-10T11:31:10.681 INFO:teuthology.orchestra.run.gibba002.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:31:10.681 INFO:teuthology.orchestra.run.gibba002.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:31:10.682 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:31:10.682 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:31:10.682 DEBUG:teuthology.orchestra.run.gibba002:> ls /sys/fs/fuse/connections 2021-02-10T11:31:11.017 INFO:teuthology.orchestra.run.gibba002.stdout:43 2021-02-10T11:31:11.017 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2021-02-10T11:31:11.018 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:31:11.018 DEBUG:teuthology.orchestra.run.gibba002:> sudo python3 -c ' 2021-02-10T11:31:11.018 DEBUG:teuthology.orchestra.run.gibba002:> import glob 2021-02-10T11:31:11.018 DEBUG:teuthology.orchestra.run.gibba002:> import re 2021-02-10T11:31:11.018 DEBUG:teuthology.orchestra.run.gibba002:> import os 2021-02-10T11:31:11.018 DEBUG:teuthology.orchestra.run.gibba002:> import subprocess 2021-02-10T11:31:11.018 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:31:11.018 DEBUG:teuthology.orchestra.run.gibba002:> def find_socket(client_name): 2021-02-10T11:31:11.018 DEBUG:teuthology.orchestra.run.gibba002:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2021-02-10T11:31:11.018 DEBUG:teuthology.orchestra.run.gibba002:> files = glob.glob(asok_path) 2021-02-10T11:31:11.018 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:31:11.019 DEBUG:teuthology.orchestra.run.gibba002:> # Given a non-glob path, it better be there 2021-02-10T11:31:11.019 DEBUG:teuthology.orchestra.run.gibba002:> if "*" not in asok_path: 2021-02-10T11:31:11.019 DEBUG:teuthology.orchestra.run.gibba002:> assert(len(files) == 1) 2021-02-10T11:31:11.019 DEBUG:teuthology.orchestra.run.gibba002:> return files[0] 2021-02-10T11:31:11.019 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:31:11.019 DEBUG:teuthology.orchestra.run.gibba002:> for f in files: 2021-02-10T11:31:11.019 DEBUG:teuthology.orchestra.run.gibba002:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2021-02-10T11:31:11.019 DEBUG:teuthology.orchestra.run.gibba002:> if os.path.exists("/proc/{0}".format(pid)): 2021-02-10T11:31:11.019 DEBUG:teuthology.orchestra.run.gibba002:> return f 2021-02-10T11:31:11.019 DEBUG:teuthology.orchestra.run.gibba002:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2021-02-10T11:31:11.019 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:31:11.019 DEBUG:teuthology.orchestra.run.gibba002:> print(find_socket("client.0")) 2021-02-10T11:31:11.019 DEBUG:teuthology.orchestra.run.gibba002:> ' 2021-02-10T11:31:11.381 INFO:teuthology.orchestra.run.gibba002.stdout:/var/run/ceph/ceph-client.0.178321.asok 2021-02-10T11:31:11.384 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.178321.asok 2021-02-10T11:31:11.384 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:31:11.384 DEBUG:teuthology.orchestra.run.gibba002:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.178321.asok status 2021-02-10T11:31:11.774 INFO:teuthology.orchestra.run.gibba002.stdout:{ 2021-02-10T11:31:11.774 INFO:teuthology.orchestra.run.gibba002.stdout: "metadata": { 2021-02-10T11:31:11.775 INFO:teuthology.orchestra.run.gibba002.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:31:11.775 INFO:teuthology.orchestra.run.gibba002.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:31:11.775 INFO:teuthology.orchestra.run.gibba002.stdout: "entity_id": "0", 2021-02-10T11:31:11.775 INFO:teuthology.orchestra.run.gibba002.stdout: "hostname": "gibba002", 2021-02-10T11:31:11.775 INFO:teuthology.orchestra.run.gibba002.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2021-02-10T11:31:11.776 INFO:teuthology.orchestra.run.gibba002.stdout: "pid": "178321", 2021-02-10T11:31:11.776 INFO:teuthology.orchestra.run.gibba002.stdout: "root": "/" 2021-02-10T11:31:11.776 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:31:11.776 INFO:teuthology.orchestra.run.gibba002.stdout: "dentry_count": 0, 2021-02-10T11:31:11.776 INFO:teuthology.orchestra.run.gibba002.stdout: "dentry_pinned_count": 0, 2021-02-10T11:31:11.776 INFO:teuthology.orchestra.run.gibba002.stdout: "id": 10224, 2021-02-10T11:31:11.777 INFO:teuthology.orchestra.run.gibba002.stdout: "inst": { 2021-02-10T11:31:11.777 INFO:teuthology.orchestra.run.gibba002.stdout: "name": { 2021-02-10T11:31:11.777 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "client", 2021-02-10T11:31:11.777 INFO:teuthology.orchestra.run.gibba002.stdout: "num": 10224 2021-02-10T11:31:11.777 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:31:11.778 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": { 2021-02-10T11:31:11.778 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:31:11.778 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:0", 2021-02-10T11:31:11.778 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 2895861929 2021-02-10T11:31:11.778 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:31:11.778 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:31:11.779 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": { 2021-02-10T11:31:11.779 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:31:11.779 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:0", 2021-02-10T11:31:11.779 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 2895861929 2021-02-10T11:31:11.779 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:31:11.780 INFO:teuthology.orchestra.run.gibba002.stdout: "inst_str": "client.10224 172.21.2.102:0/2895861929", 2021-02-10T11:31:11.780 INFO:teuthology.orchestra.run.gibba002.stdout: "addr_str": "172.21.2.102:0/2895861929", 2021-02-10T11:31:11.780 INFO:teuthology.orchestra.run.gibba002.stdout: "inode_count": 1, 2021-02-10T11:31:11.780 INFO:teuthology.orchestra.run.gibba002.stdout: "mds_epoch": 82, 2021-02-10T11:31:11.780 INFO:teuthology.orchestra.run.gibba002.stdout: "osd_epoch": 103, 2021-02-10T11:31:11.780 INFO:teuthology.orchestra.run.gibba002.stdout: "osd_epoch_barrier": 0, 2021-02-10T11:31:11.781 INFO:teuthology.orchestra.run.gibba002.stdout: "blacklisted": false 2021-02-10T11:31:11.781 INFO:teuthology.orchestra.run.gibba002.stdout:} 2021-02-10T11:31:11.782 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:31:11.783 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:31:11.783 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:31:12.118 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:31:12.119 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:31:12.119 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:31:12.119 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:31:12.465 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:31:12.893 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:12.876 7f7f905b9700 1 Processor -- start 2021-02-10T11:31:12.893 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:12.876 7f7f905b9700 1 -- start start 2021-02-10T11:31:12.893 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:12.877 7f7f905b9700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f7f881b0470 0x7f7f881b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:12.893 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:12.877 7f7f905b9700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f7f880ef610 0x7f7f880efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:12.893 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:12.877 7f7f905b9700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f880f0b50 con 0x7f7f880f0030 2021-02-10T11:31:12.893 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:12.877 7f7f905b9700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f880e52e0 con 0x7f7f881b0470 2021-02-10T11:31:12.894 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:12.877 7f7f905b9700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f880e5490 con 0x7f7f880ef610 2021-02-10T11:31:12.894 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:12.877 7f7f8ebd6700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f7f880f0030 0x7f7f880f10d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:53948/0 (socket says 172.21.2.102:53948) 2021-02-10T11:31:12.894 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:12.877 7f7f8ebd6700 1 -- 172.21.2.102:0/3724002111 learned_addr learned my addr 172.21.2.102:0/3724002111 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:31:12.894 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:12.877 7f7f8e3d5700 1 --2- 172.21.2.102:0/3724002111 >> v2:172.21.2.102:3300/0 conn(0x7f7f881b0470 0x7f7f881b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:12.894 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:12.877 7f7f8dbd4700 1 --2- 172.21.2.102:0/3724002111 >> v2:172.21.2.113:3300/0 conn(0x7f7f880ef610 0x7f7f880efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:12.894 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:12.877 7f7f8d3d3700 1 -- 172.21.2.102:0/3724002111 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 457632107 0 0) 0x7f7f80001500 con 0x7f7f880f0030 2021-02-10T11:31:12.894 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:12.877 7f7f8d3d3700 1 -- 172.21.2.102:0/3724002111 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f74002fe0 con 0x7f7f880f0030 2021-02-10T11:31:12.894 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:12.877 7f7f8e3d5700 1 -- 172.21.2.102:0/3724002111 >> v2:172.21.2.113:3300/0 conn(0x7f7f880ef610 msgr2=0x7f7f880efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:12.894 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:12.877 7f7f8e3d5700 1 --2- 172.21.2.102:0/3724002111 >> v2:172.21.2.113:3300/0 conn(0x7f7f880ef610 0x7f7f880efa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:12.894 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:12.877 7f7f8e3d5700 1 -- 172.21.2.102:0/3724002111 >> v1:172.21.2.102:6789/0 conn(0x7f7f880f0030 legacy=0x7f7f880f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:12.894 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:12.877 7f7f8e3d5700 1 -- 172.21.2.102:0/3724002111 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f880f0430 con 0x7f7f881b0470 2021-02-10T11:31:12.894 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:12.878 7f7f8e3d5700 1 --2- 172.21.2.102:0/3724002111 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7f881b0470 0x7f7f881b0870 secure :-1 s=READY pgs=2105 cs=0 l=1 rev1=1 rx=0x7f7f7c006650 tx=0x7f7f7c009a80).ready entity=mon.0 client_cookie=40e1bd91c54389b5 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:12.895 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:12.878 7f7f8d3d3700 1 -- 172.21.2.102:0/3724002111 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7f7c00c0c0 con 0x7f7f881b0470 2021-02-10T11:31:12.895 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:12.878 7f7f8d3d3700 1 -- 172.21.2.102:0/3724002111 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f7c00c340 con 0x7f7f881b0470 2021-02-10T11:31:12.895 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:12.878 7f7f8d3d3700 1 -- 172.21.2.102:0/3724002111 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7f7c00c860 con 0x7f7f881b0470 2021-02-10T11:31:12.895 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:12.878 7f7f905b9700 1 -- 172.21.2.102:0/3724002111 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7f881b0470 msgr2=0x7f7f881b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:12.895 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:12.878 7f7f905b9700 1 --2- 172.21.2.102:0/3724002111 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7f881b0470 0x7f7f881b0870 secure :-1 s=READY pgs=2105 cs=0 l=1 rev1=1 rx=0x7f7f7c006650 tx=0x7f7f7c009a80).stop 2021-02-10T11:31:12.895 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:12.879 7f7f905b9700 1 -- 172.21.2.102:0/3724002111 shutdown_connections 2021-02-10T11:31:12.895 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:12.879 7f7f905b9700 1 --2- 172.21.2.102:0/3724002111 >> v2:172.21.2.113:3300/0 conn(0x7f7f880ef610 0x7f7f880efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:12.895 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:12.879 7f7f905b9700 1 --2- 172.21.2.102:0/3724002111 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7f881b0470 0x7f7f881b0870 unknown :-1 s=CLOSED pgs=2105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:12.895 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:12.879 7f7f905b9700 1 -- 172.21.2.102:0/3724002111 >> 172.21.2.102:0/3724002111 conn(0x7f7f880ce3e0 msgr2=0x7f7f880e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:12.895 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:12.879 7f7f905b9700 1 -- 172.21.2.102:0/3724002111 shutdown_connections 2021-02-10T11:31:12.895 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:12.879 7f7f905b9700 1 -- 172.21.2.102:0/3724002111 wait complete. 2021-02-10T11:31:12.897 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:12.893 7f7f905b9700 1 Processor -- start 2021-02-10T11:31:12.897 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:12.893 7f7f905b9700 1 -- start start 2021-02-10T11:31:12.897 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:12.894 7f7f905b9700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7f8809eab0 0x7f7f8818a210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:12.897 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:12.894 7f7f905b9700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7f8809df40 0x7f7f8818ae20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:12.897 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:12.894 7f7f905b9700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7f8818b440 0x7f7f8818b840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:12.897 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:12.894 7f7f905b9700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f8818c3a0 con 0x7f7f8818b440 2021-02-10T11:31:12.897 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:12.894 7f7f905b9700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f8818a8d0 con 0x7f7f8809df40 2021-02-10T11:31:12.897 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:12.894 7f7f905b9700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f8809e340 con 0x7f7f8809eab0 2021-02-10T11:31:12.898 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:12.894 7f7f8ebd6700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7f8818b440 0x7f7f8818b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:12.898 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:12.894 7f7f8ebd6700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7f8818b440 0x7f7f8818b840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:33500/0 (socket says 172.21.2.102:33500) 2021-02-10T11:31:12.898 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:12.894 7f7f8ebd6700 1 -- 172.21.2.102:0/1819893044 learned_addr learned my addr 172.21.2.102:0/1819893044 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:31:12.898 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:12.894 7f7f8e3d5700 1 --2- 172.21.2.102:0/1819893044 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7f8809eab0 0x7f7f8818a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:12.898 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:12.894 7f7f8dbd4700 1 --2- 172.21.2.102:0/1819893044 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7f8809df40 0x7f7f8818ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:12.898 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:12.894 7f7f8ebd6700 1 -- 172.21.2.102:0/1819893044 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7f8809eab0 msgr2=0x7f7f8818a210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:12.898 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:12.894 7f7f8ebd6700 1 --2- 172.21.2.102:0/1819893044 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7f8809eab0 0x7f7f8818a210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:12.898 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:12.894 7f7f8ebd6700 1 -- 172.21.2.102:0/1819893044 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7f8809df40 msgr2=0x7f7f8818ae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:12.898 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:12.894 7f7f8ebd6700 1 --2- 172.21.2.102:0/1819893044 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7f8809df40 0x7f7f8818ae20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:12.898 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:12.894 7f7f8ebd6700 1 -- 172.21.2.102:0/1819893044 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f8809e5f0 con 0x7f7f8818b440 2021-02-10T11:31:12.898 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:12.894 7f7f8e3d5700 1 --2- 172.21.2.102:0/1819893044 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7f8809eab0 0x7f7f8818a210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:31:12.898 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:12.894 7f7f8ebd6700 1 --2- 172.21.2.102:0/1819893044 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7f8818b440 0x7f7f8818b840 secure :-1 s=READY pgs=2106 cs=0 l=1 rev1=1 rx=0x7f7f80009980 tx=0x7f7f8000bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:12.898 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:12.895 7f7f7affd700 1 -- 172.21.2.102:0/1819893044 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7f8000be10 con 0x7f7f8818b440 2021-02-10T11:31:12.899 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:12.895 7f7f7affd700 1 -- 172.21.2.102:0/1819893044 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f80006030 con 0x7f7f8818b440 2021-02-10T11:31:12.899 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:12.895 7f7f7affd700 1 -- 172.21.2.102:0/1819893044 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7f80004720 con 0x7f7f8818b440 2021-02-10T11:31:12.899 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:12.895 7f7f905b9700 1 -- 172.21.2.102:0/1819893044 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f8809e5f0 con 0x7f7f8818b440 2021-02-10T11:31:12.899 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:12.895 7f7f905b9700 1 -- 172.21.2.102:0/1819893044 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f8818be60 con 0x7f7f8818b440 2021-02-10T11:31:12.899 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:12.895 7f7f7affd700 1 -- 172.21.2.102:0/1819893044 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f7f80000d30 con 0x7f7f8818b440 2021-02-10T11:31:12.899 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:12.895 7f7f7affd700 1 --2- 172.21.2.102:0/1819893044 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7f700186c0 0x7f7f7001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:12.899 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:12.895 7f7f7affd700 1 -- 172.21.2.102:0/1819893044 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f7f80032890 con 0x7f7f8818b440 2021-02-10T11:31:12.899 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:12.896 7f7f8e3d5700 1 --2- 172.21.2.102:0/1819893044 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7f700186c0 0x7f7f7001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:12.899 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:12.896 7f7f8e3d5700 1 --2- 172.21.2.102:0/1819893044 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7f700186c0 0x7f7f7001abc0 crc :-1 s=READY pgs=934 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:12.927 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:12.927 7f7f905b9700 1 -- 172.21.2.102:0/1819893044 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f880f0b50 con 0x7f7f8818b440 2021-02-10T11:31:12.929 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:12.930 7f7f7affd700 1 -- 172.21.2.102:0/1819893044 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f7f80024a10 con 0x7f7f8818b440 2021-02-10T11:31:13.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:13.055 7f7f905b9700 1 -- 172.21.2.102:0/1819893044 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7f880f0b50 con 0x7f7f8818b440 2021-02-10T11:31:13.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:13.055 7f7f7affd700 1 -- 172.21.2.102:0/1819893044 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v103) v1 ==== 74+0+14005 (secure 0 0 0) 0x7f7f80024a10 con 0x7f7f8818b440 2021-02-10T11:31:13.056 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:31:13.056 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":103,"fsid":"68db30ea-dbce-4aef-a6df-f01c6fea0513","created":"2021-02-10 11:03:06.331557","modified":"2021-02-10 11:31:00.055039","last_up_change":"2021-02-10 11:03:32.690562","last_in_change":"2021-02-10 11:03:32.690562","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:03:37.322519","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:30:56.015905","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"101","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":21,"pool_name":"cephfs_data","create_time":"2021-02-10 11:30:57.042942","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"103","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"be691b4d-a039-408a-af80-9d53f02b6f1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6818","nonce":36088},{"type":"v1","addr":"172.21.2.102:6820","nonce":36088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6822","nonce":36088},{"type":"v1","addr":"172.21.2.102:6824","nonce":36088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6830","nonce":36088},{"type":"v1","addr":"172.21.2.102:6832","nonce":36088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6826","nonce":36088},{"type":"v1","addr":"172.21.2.102:6828","nonce":36088}]},"public_addr":"172.21.2.102:6820/36088","cluster_addr":"172.21.2.102:6824/36088","heartbeat_back_addr":"172.21.2.102:6832/36088","heartbeat_front_addr":"172.21.2.102:6828/36088","state":["exists","up"]},{"osd":1,"uuid":"64e3ccc8-9c69-4d7c-865e-af8e83822635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6819","nonce":36087},{"type":"v1","addr":"172.21.2.102:6821","nonce":36087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6823","nonce":36087},{"type":"v1","addr":"172.21.2.102:6825","nonce":36087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6831","nonce":36087},{"type":"v1","addr":"172.21.2.102:6833","nonce":36087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6827","nonce":36087},{"type":"v1","addr":"172.21.2.102:6829","nonce":36087}]},"public_addr":"172.21.2.102:6821/36087","cluster_addr":"172.21.2.102:6825/36087","heartbeat_back_addr":"172.21.2.102:6833/36087","heartbeat_front_addr":"172.21.2.102:6829/36087","state":["exists","up"]},{"osd":2,"uuid":"153700af-747f-481f-bdf4-d6072d3a842f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6810","nonce":36089},{"type":"v1","addr":"172.21.2.102:6811","nonce":36089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6812","nonce":36089},{"type":"v1","addr":"172.21.2.102:6813","nonce":36089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6816","nonce":36089},{"type":"v1","addr":"172.21.2.102:6817","nonce":36089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6814","nonce":36089},{"type":"v1","addr":"172.21.2.102:6815","nonce":36089}]},"public_addr":"172.21.2.102:6811/36089","cluster_addr":"172.21.2.102:6813/36089","heartbeat_back_addr":"172.21.2.102:6817/36089","heartbeat_front_addr":"172.21.2.102:6815/36089","state":["exists","up"]},{"osd":3,"uuid":"cbc678f4-83a9-45e6-8830-2f7c232b1ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6802","nonce":36090},{"type":"v1","addr":"172.21.2.102:6803","nonce":36090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6804","nonce":36090},{"type":"v1","addr":"172.21.2.102:6805","nonce":36090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6808","nonce":36090},{"type":"v1","addr":"172.21.2.102:6809","nonce":36090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6806","nonce":36090},{"type":"v1","addr":"172.21.2.102:6807","nonce":36090}]},"public_addr":"172.21.2.102:6803/36090","cluster_addr":"172.21.2.102:6805/36090","heartbeat_back_addr":"172.21.2.102:6809/36090","heartbeat_front_addr":"172.21.2.102:6807/36090","state":["exists","up"]},{"osd":4,"uuid":"c01e4008-b258-4f10-afd2-9e7f37214351","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6808","nonce":36368},{"type":"v1","addr":"172.21.2.113:6811","nonce":36368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6814","nonce":36368},{"type":"v1","addr":"172.21.2.113:6818","nonce":36368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6826","nonce":36368},{"type":"v1","addr":"172.21.2.113:6828","nonce":36368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6820","nonce":36368},{"type":"v1","addr":"172.21.2.113:6824","nonce":36368}]},"public_addr":"172.21.2.113:6811/36368","cluster_addr":"172.21.2.113:6818/36368","heartbeat_back_addr":"172.21.2.113:6828/36368","heartbeat_front_addr":"172.21.2.113:6824/36368","state":["exists","up"]},{"osd":5,"uuid":"551f6007-daf8-4a98-8ee0-c2b16bcccb5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6802","nonce":36366},{"type":"v1","addr":"172.21.2.113:6805","nonce":36366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6807","nonce":36366},{"type":"v1","addr":"172.21.2.113:6810","nonce":36366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6819","nonce":36366},{"type":"v1","addr":"172.21.2.113:6822","nonce":36366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6812","nonce":36366},{"type":"v1","addr":"172.21.2.113:6815","nonce":36366}]},"public_addr":"172.21.2.113:6805/36366","cluster_addr":"172.21.2.113:6810/36366","heartbeat_back_addr":"172.21.2.113:6822/36366","heartbeat_front_addr":"172.21.2.113:6815/36366","state":["exists","up"]},{"osd":6,"uuid":"bf67c040-0e50-48be-b102-c3943bc9ebef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6800","nonce":36367},{"type":"v1","addr":"172.21.2.113:6801","nonce":36367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6803","nonce":36367},{"type":"v1","addr":"172.21.2.113:6804","nonce":36367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6813","nonce":36367},{"type":"v1","addr":"172.21.2.113:6816","nonce":36367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6806","nonce":36367},{"type":"v1","addr":"172.21.2.113:6809","nonce":36367}]},"public_addr":"172.21.2.113:6801/36367","cluster_addr":"172.21.2.113:6804/36367","heartbeat_back_addr":"172.21.2.113:6816/36367","heartbeat_front_addr":"172.21.2.113:6809/36367","state":["exists","up"]},{"osd":7,"uuid":"f5461f80-018d-428f-be89-2eb2bcbe7bb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6817","nonce":36365},{"type":"v1","addr":"172.21.2.113:6821","nonce":36365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6823","nonce":36365},{"type":"v1","addr":"172.21.2.113:6825","nonce":36365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6830","nonce":36365},{"type":"v1","addr":"172.21.2.113:6831","nonce":36365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6827","nonce":36365},{"type":"v1","addr":"172.21.2.113:6829","nonce":36365}]},"public_addr":"172.21.2.113:6821/36365","cluster_addr":"172.21.2.113:6825/36365","heartbeat_back_addr":"172.21.2.113:6831/36365","heartbeat_front_addr":"172.21.2.113:6829/36365","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:31:13.057 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:13.057 7f7f905b9700 1 -- 172.21.2.102:0/1819893044 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7f700186c0 msgr2=0x7f7f7001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:13.058 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:13.057 7f7f905b9700 1 --2- 172.21.2.102:0/1819893044 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7f700186c0 0x7f7f7001abc0 crc :-1 s=READY pgs=934 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:13.058 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:13.057 7f7f905b9700 1 -- 172.21.2.102:0/1819893044 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7f8818b440 msgr2=0x7f7f8818b840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:13.058 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:13.057 7f7f905b9700 1 --2- 172.21.2.102:0/1819893044 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7f8818b440 0x7f7f8818b840 secure :-1 s=READY pgs=2106 cs=0 l=1 rev1=1 rx=0x7f7f80009980 tx=0x7f7f8000bcd0).stop 2021-02-10T11:31:13.058 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:13.057 7f7f905b9700 1 -- 172.21.2.102:0/1819893044 shutdown_connections 2021-02-10T11:31:13.058 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:13.057 7f7f905b9700 1 --2- 172.21.2.102:0/1819893044 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7f8818b440 0x7f7f8818b840 unknown :-1 s=CLOSED pgs=2106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:13.058 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:13.057 7f7f905b9700 1 --2- 172.21.2.102:0/1819893044 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7f700186c0 0x7f7f7001abc0 unknown :-1 s=CLOSED pgs=934 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:13.058 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:13.057 7f7f905b9700 1 --2- 172.21.2.102:0/1819893044 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7f8809df40 0x7f7f8818ae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:13.058 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:13.057 7f7f905b9700 1 --2- 172.21.2.102:0/1819893044 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7f8809eab0 0x7f7f8818a210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:13.058 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:13.057 7f7f905b9700 1 -- 172.21.2.102:0/1819893044 >> 172.21.2.102:0/1819893044 conn(0x7f7f8810a530 msgr2=0x7f7f880b0110 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:13.058 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:13.057 7f7f905b9700 1 -- 172.21.2.102:0/1819893044 shutdown_connections 2021-02-10T11:31:13.059 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:13.058 7f7f905b9700 1 -- 172.21.2.102:0/1819893044 wait complete. 2021-02-10T11:31:13.067 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:03:37.322519', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 20, 'pool_name': 'cephfs_metadata', 'create_time': '2021-02-10 11:30:56.015905', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '101', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 21, 'pool_name': 'cephfs_data', 'create_time': '2021-02-10 11:30:57.042942', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '103', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-02-10T11:31:13.067 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:31:13.501 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:13.485 7ffa8a60a700 1 Processor -- start 2021-02-10T11:31:13.501 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:13.485 7ffa8a60a700 1 -- start start 2021-02-10T11:31:13.501 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:13.485 7ffa8a60a700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7ffa840d1870 0x7ffa841b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:13.501 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:13.485 7ffa8a60a700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7ffa840cff70 0x7ffa84109130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:13.502 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:13.485 7ffa8a60a700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa840fe1c0 con 0x7ffa841b0ae0 2021-02-10T11:31:13.502 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:13.485 7ffa8a60a700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa84109ad0 con 0x7ffa840cff70 2021-02-10T11:31:13.502 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:13.485 7ffa8a60a700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa84109750 con 0x7ffa840d1870 2021-02-10T11:31:13.502 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:13.485 7ffa837fe700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7ffa841b0ae0 0x7ffa84108940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:53960/0 (socket says 172.21.2.102:53960) 2021-02-10T11:31:13.502 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:13.485 7ffa837fe700 1 -- 172.21.2.102:0/2861474132 learned_addr learned my addr 172.21.2.102:0/2861474132 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:31:13.502 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:13.485 7ffa88c27700 1 --2- 172.21.2.102:0/2861474132 >> v2:172.21.2.102:3300/0 conn(0x7ffa840cff70 0x7ffa84109130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:13.502 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:13.486 7ffa82ffd700 1 -- 172.21.2.102:0/2861474132 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4158951732 0 0) 0x7ffa780010b0 con 0x7ffa841b0ae0 2021-02-10T11:31:13.502 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:13.486 7ffa88c27700 1 -- 172.21.2.102:0/2861474132 >> v2:172.21.2.113:3300/0 conn(0x7ffa840d1870 msgr2=0x7ffa841b04c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:31:13.502 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:13.486 7ffa88c27700 1 --2- 172.21.2.102:0/2861474132 >> v2:172.21.2.113:3300/0 conn(0x7ffa840d1870 0x7ffa841b04c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:13.502 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:13.486 7ffa88c27700 1 -- 172.21.2.102:0/2861474132 >> v1:172.21.2.102:6789/0 conn(0x7ffa841b0ae0 legacy=0x7ffa84108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:13.502 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:13.486 7ffa88c27700 1 -- 172.21.2.102:0/2861474132 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa84088070 con 0x7ffa840cff70 2021-02-10T11:31:13.502 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:13.486 7ffa88c27700 1 --2- 172.21.2.102:0/2861474132 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ffa840cff70 0x7ffa84109130 secure :-1 s=READY pgs=2108 cs=0 l=1 rev1=1 rx=0x7ffa7c006e30 tx=0x7ffa7c009e60).ready entity=mon.0 client_cookie=acad21860236010c server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:13.503 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:13.486 7ffa82ffd700 1 -- 172.21.2.102:0/2861474132 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ffa7c006a50 con 0x7ffa840cff70 2021-02-10T11:31:13.503 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:13.486 7ffa82ffd700 1 -- 172.21.2.102:0/2861474132 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa7c00b300 con 0x7ffa840cff70 2021-02-10T11:31:13.503 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:13.486 7ffa82ffd700 1 -- 172.21.2.102:0/2861474132 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ffa7c00b750 con 0x7ffa840cff70 2021-02-10T11:31:13.503 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:13.486 7ffa8a60a700 1 -- 172.21.2.102:0/2861474132 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ffa840cff70 msgr2=0x7ffa84109130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:13.503 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:13.486 7ffa8a60a700 1 --2- 172.21.2.102:0/2861474132 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ffa840cff70 0x7ffa84109130 secure :-1 s=READY pgs=2108 cs=0 l=1 rev1=1 rx=0x7ffa7c006e30 tx=0x7ffa7c009e60).stop 2021-02-10T11:31:13.503 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:13.487 7ffa8a60a700 1 -- 172.21.2.102:0/2861474132 shutdown_connections 2021-02-10T11:31:13.503 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:13.487 7ffa8a60a700 1 --2- 172.21.2.102:0/2861474132 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ffa840cff70 0x7ffa84109130 unknown :-1 s=CLOSED pgs=2108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:13.503 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:13.487 7ffa8a60a700 1 --2- 172.21.2.102:0/2861474132 >> v2:172.21.2.113:3300/0 conn(0x7ffa840d1870 0x7ffa841b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:13.503 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:13.487 7ffa8a60a700 1 -- 172.21.2.102:0/2861474132 >> 172.21.2.102:0/2861474132 conn(0x7ffa840ce420 msgr2=0x7ffa840d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:13.503 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:13.487 7ffa8a60a700 1 -- 172.21.2.102:0/2861474132 shutdown_connections 2021-02-10T11:31:13.503 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:13.487 7ffa8a60a700 1 -- 172.21.2.102:0/2861474132 wait complete. 2021-02-10T11:31:13.505 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:13.502 7ffa8a60a700 1 Processor -- start 2021-02-10T11:31:13.505 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:13.502 7ffa8a60a700 1 -- start start 2021-02-10T11:31:13.505 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:13.502 7ffa8a60a700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ffa840c3820 0x7ffa840a7950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:13.505 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:13.502 7ffa8a60a700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ffa841a44c0 0x7ffa841a48c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:13.505 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:13.502 7ffa8a60a700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ffa840863d0 0x7ffa840867d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:13.505 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:13.502 7ffa8a60a700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa84085ed0 con 0x7ffa840863d0 2021-02-10T11:31:13.505 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:13.502 7ffa8a60a700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa840c0740 con 0x7ffa840c3820 2021-02-10T11:31:13.505 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:13.502 7ffa8a60a700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa84086df0 con 0x7ffa841a44c0 2021-02-10T11:31:13.505 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:13.502 7ffa88c27700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ffa840863d0 0x7ffa840867d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:13.505 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:13.502 7ffa88c27700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ffa840863d0 0x7ffa840867d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:33512/0 (socket says 172.21.2.102:33512) 2021-02-10T11:31:13.505 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:13.502 7ffa88c27700 1 -- 172.21.2.102:0/2471655961 learned_addr learned my addr 172.21.2.102:0/2471655961 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:31:13.505 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:13.503 7ffa88c27700 1 -- 172.21.2.102:0/2471655961 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ffa841a44c0 msgr2=0x7ffa841a48c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:13.506 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:13.503 7ffa88c27700 1 --2- 172.21.2.102:0/2471655961 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ffa841a44c0 0x7ffa841a48c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:13.506 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:13.503 7ffa88c27700 1 -- 172.21.2.102:0/2471655961 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ffa840c3820 msgr2=0x7ffa840a7950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:13.506 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:13.503 7ffa88c27700 1 --2- 172.21.2.102:0/2471655961 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ffa840c3820 0x7ffa840a7950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:13.506 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:13.503 7ffa88c27700 1 -- 172.21.2.102:0/2471655961 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa84087010 con 0x7ffa840863d0 2021-02-10T11:31:13.506 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:13.503 7ffa88c27700 1 --2- 172.21.2.102:0/2471655961 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ffa840863d0 0x7ffa840867d0 secure :-1 s=READY pgs=2109 cs=0 l=1 rev1=1 rx=0x7ffa7c00b970 tx=0x7ffa7c00bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:13.506 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:13.503 7ffa80ff9700 1 -- 172.21.2.102:0/2471655961 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ffa7c009380 con 0x7ffa840863d0 2021-02-10T11:31:13.506 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:13.503 7ffa80ff9700 1 -- 172.21.2.102:0/2471655961 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa7c005820 con 0x7ffa840863d0 2021-02-10T11:31:13.506 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:13.503 7ffa8a60a700 1 -- 172.21.2.102:0/2471655961 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa84087010 con 0x7ffa840863d0 2021-02-10T11:31:13.506 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:13.503 7ffa8a60a700 1 -- 172.21.2.102:0/2471655961 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa84083000 con 0x7ffa840863d0 2021-02-10T11:31:13.507 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:13.503 7ffa80ff9700 1 -- 172.21.2.102:0/2471655961 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ffa7c005d00 con 0x7ffa840863d0 2021-02-10T11:31:13.507 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:13.504 7ffa80ff9700 1 -- 172.21.2.102:0/2471655961 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ffa7c005a00 con 0x7ffa840863d0 2021-02-10T11:31:13.507 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:13.504 7ffa80ff9700 1 --2- 172.21.2.102:0/2471655961 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ffa640186c0 0x7ffa6401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:13.507 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:13.504 7ffa80ff9700 1 -- 172.21.2.102:0/2471655961 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 5573+0+0 (secure 0 0 0) 0x7ffa7c032930 con 0x7ffa840863d0 2021-02-10T11:31:13.507 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:13.504 7ffa83fff700 1 --2- 172.21.2.102:0/2471655961 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ffa640186c0 0x7ffa6401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:13.507 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:13.504 7ffa83fff700 1 --2- 172.21.2.102:0/2471655961 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ffa640186c0 0x7ffa6401abc0 crc :-1 s=READY pgs=935 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:13.535 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:13.535 7ffa8a60a700 1 -- 172.21.2.102:0/2471655961 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa84085e50 con 0x7ffa840863d0 2021-02-10T11:31:13.537 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:13.538 7ffa80ff9700 1 -- 172.21.2.102:0/2471655961 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ffa7c009380 con 0x7ffa840863d0 2021-02-10T11:31:13.666 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:13.666 7ffa8a60a700 1 -- 172.21.2.102:0/2471655961 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7ffa840fe1c0 con 0x7ffa840863d0 2021-02-10T11:31:13.666 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:13.666 7ffa80ff9700 1 -- 172.21.2.102:0/2471655961 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v103) v1 ==== 92+0+10 (secure 0 0 0) 0x7ffa7c005a00 con 0x7ffa840863d0 2021-02-10T11:31:13.667 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:31:13.668 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:13.668 7ffa8a60a700 1 -- 172.21.2.102:0/2471655961 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ffa640186c0 msgr2=0x7ffa6401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:13.668 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:13.668 7ffa8a60a700 1 --2- 172.21.2.102:0/2471655961 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ffa640186c0 0x7ffa6401abc0 crc :-1 s=READY pgs=935 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:13.668 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:13.668 7ffa8a60a700 1 -- 172.21.2.102:0/2471655961 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ffa840863d0 msgr2=0x7ffa840867d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:13.669 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:13.668 7ffa8a60a700 1 --2- 172.21.2.102:0/2471655961 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ffa840863d0 0x7ffa840867d0 secure :-1 s=READY pgs=2109 cs=0 l=1 rev1=1 rx=0x7ffa7c00b970 tx=0x7ffa7c00bd10).stop 2021-02-10T11:31:13.669 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:13.668 7ffa8a60a700 1 -- 172.21.2.102:0/2471655961 shutdown_connections 2021-02-10T11:31:13.669 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:13.668 7ffa8a60a700 1 --2- 172.21.2.102:0/2471655961 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ffa840863d0 0x7ffa840867d0 unknown :-1 s=CLOSED pgs=2109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:13.669 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:13.668 7ffa8a60a700 1 --2- 172.21.2.102:0/2471655961 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ffa841a44c0 0x7ffa841a48c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:13.669 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:13.668 7ffa8a60a700 1 --2- 172.21.2.102:0/2471655961 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ffa640186c0 0x7ffa6401abc0 unknown :-1 s=CLOSED pgs=935 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:13.669 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:13.668 7ffa8a60a700 1 --2- 172.21.2.102:0/2471655961 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ffa840c3820 0x7ffa840a7950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:13.669 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:13.668 7ffa8a60a700 1 -- 172.21.2.102:0/2471655961 >> 172.21.2.102:0/2471655961 conn(0x7ffa840872c0 msgr2=0x7ffa84095700 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:13.670 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:13.669 7ffa8a60a700 1 -- 172.21.2.102:0/2471655961 shutdown_connections 2021-02-10T11:31:13.670 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:13.669 7ffa8a60a700 1 -- 172.21.2.102:0/2471655961 wait complete. 2021-02-10T11:31:13.677 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-02-10T11:31:14.109 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:14.093 7f3485a9d700 1 Processor -- start 2021-02-10T11:31:14.110 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:14.093 7f3485a9d700 1 -- start start 2021-02-10T11:31:14.110 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:14.093 7f3485a9d700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f34800d1870 0x7f34801b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:14.110 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:14.093 7f3485a9d700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f34801b0ae0 0x7f3480108940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:14.110 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:14.093 7f3485a9d700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34800fe1c0 con 0x7f34800d1c70 2021-02-10T11:31:14.110 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:14.093 7f3485a9d700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3480109a60 con 0x7f34800d1870 2021-02-10T11:31:14.110 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:14.093 7f3485a9d700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3480109750 con 0x7f34801b0ae0 2021-02-10T11:31:14.110 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:14.093 7f347ffff700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f34800d1c70 0x7f3480108f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:53976/0 (socket says 172.21.2.102:53976) 2021-02-10T11:31:14.110 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:14.093 7f347ffff700 1 -- 172.21.2.102:0/3355903251 learned_addr learned my addr 172.21.2.102:0/3355903251 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:31:14.110 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:14.093 7f347f7fe700 1 --2- 172.21.2.102:0/3355903251 >> v2:172.21.2.102:3300/0 conn(0x7f34800d1870 0x7f34801b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:14.110 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:14.094 7f347effd700 1 --2- 172.21.2.102:0/3355903251 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f34801b0ae0 0x7f3480108940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:14.111 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:14.094 7f347e7fc700 1 -- 172.21.2.102:0/3355903251 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2428884458 0 0) 0x7f3478001500 con 0x7f34800d1c70 2021-02-10T11:31:14.111 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:14.094 7f347f7fe700 1 -- 172.21.2.102:0/3355903251 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f34801b0ae0 msgr2=0x7f3480108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:14.111 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:14.094 7f347f7fe700 1 --2- 172.21.2.102:0/3355903251 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f34801b0ae0 0x7f3480108940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:14.111 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:14.094 7f347f7fe700 1 -- 172.21.2.102:0/3355903251 >> v1:172.21.2.102:6789/0 conn(0x7f34800d1c70 legacy=0x7f3480108f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:14.111 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:14.094 7f347f7fe700 1 -- 172.21.2.102:0/3355903251 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3480088070 con 0x7f34800d1870 2021-02-10T11:31:14.111 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:14.094 7f347f7fe700 1 --2- 172.21.2.102:0/3355903251 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f34800d1870 0x7f34801b04c0 secure :-1 s=READY pgs=2111 cs=0 l=1 rev1=1 rx=0x7f3470006650 tx=0x7f3470009a80).ready entity=mon.0 client_cookie=1bf9cc64e9cd228b server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:14.111 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:14.094 7f347e7fc700 1 -- 172.21.2.102:0/3355903251 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f34700009b0 con 0x7f34800d1870 2021-02-10T11:31:14.111 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:14.094 7f347e7fc700 1 -- 172.21.2.102:0/3355903251 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f347000c0c0 con 0x7f34800d1870 2021-02-10T11:31:14.111 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:14.094 7f347e7fc700 1 -- 172.21.2.102:0/3355903251 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f347000c510 con 0x7f34800d1870 2021-02-10T11:31:14.111 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:14.094 7f3485a9d700 1 -- 172.21.2.102:0/3355903251 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f34800d1870 msgr2=0x7f34801b04c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:14.111 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:14.094 7f3485a9d700 1 --2- 172.21.2.102:0/3355903251 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f34800d1870 0x7f34801b04c0 secure :-1 s=READY pgs=2111 cs=0 l=1 rev1=1 rx=0x7f3470006650 tx=0x7f3470009a80).stop 2021-02-10T11:31:14.111 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:14.095 7f3485a9d700 1 -- 172.21.2.102:0/3355903251 shutdown_connections 2021-02-10T11:31:14.111 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:14.095 7f3485a9d700 1 --2- 172.21.2.102:0/3355903251 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f34801b0ae0 0x7f3480108940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:14.112 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:14.095 7f3485a9d700 1 --2- 172.21.2.102:0/3355903251 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f34800d1870 0x7f34801b04c0 unknown :-1 s=CLOSED pgs=2111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:14.112 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:14.095 7f3485a9d700 1 -- 172.21.2.102:0/3355903251 >> 172.21.2.102:0/3355903251 conn(0x7f34800ce420 msgr2=0x7f34800d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:14.112 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:14.095 7f3485a9d700 1 -- 172.21.2.102:0/3355903251 shutdown_connections 2021-02-10T11:31:14.112 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:14.095 7f3485a9d700 1 -- 172.21.2.102:0/3355903251 wait complete. 2021-02-10T11:31:14.114 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:14.110 7f3485a9d700 1 Processor -- start 2021-02-10T11:31:14.114 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:14.110 7f3485a9d700 1 -- start start 2021-02-10T11:31:14.114 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:14.110 7f3485a9d700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f34800a7950 0x7f34800b8df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:14.114 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:14.110 7f3485a9d700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f34801a7490 0x7f34801a7890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:14.114 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:14.111 7f3485a9d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f34801a7eb0 0x7f34800ac630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:14.114 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:14.111 7f3485a9d700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34800a7fb0 con 0x7f34801a7eb0 2021-02-10T11:31:14.114 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:14.111 7f3485a9d700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34800acc50 con 0x7f34801a7490 2021-02-10T11:31:14.114 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:14.111 7f3485a9d700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34800ace30 con 0x7f34800a7950 2021-02-10T11:31:14.114 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:14.111 7f347ffff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f34801a7eb0 0x7f34800ac630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:14.114 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:14.111 7f347ffff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f34801a7eb0 0x7f34800ac630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:33522/0 (socket says 172.21.2.102:33522) 2021-02-10T11:31:14.114 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:14.111 7f347ffff700 1 -- 172.21.2.102:0/4209923717 learned_addr learned my addr 172.21.2.102:0/4209923717 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:31:14.114 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:14.111 7f347f7fe700 1 --2- 172.21.2.102:0/4209923717 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f34800a7950 0x7f34800b8df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:14.115 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:14.111 7f347effd700 1 --2- 172.21.2.102:0/4209923717 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f34801a7490 0x7f34801a7890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:14.115 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:14.111 7f347ffff700 1 -- 172.21.2.102:0/4209923717 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f34800a7950 msgr2=0x7f34800b8df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:14.115 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:14.111 7f347ffff700 1 --2- 172.21.2.102:0/4209923717 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f34800a7950 0x7f34800b8df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:14.115 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:14.111 7f347ffff700 1 -- 172.21.2.102:0/4209923717 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f34801a7490 msgr2=0x7f34801a7890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:14.115 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:14.111 7f347ffff700 1 --2- 172.21.2.102:0/4209923717 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f34801a7490 0x7f34801a7890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:14.115 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:14.111 7f347ffff700 1 -- 172.21.2.102:0/4209923717 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34800ad050 con 0x7f34801a7eb0 2021-02-10T11:31:14.115 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:14.111 7f347f7fe700 1 --2- 172.21.2.102:0/4209923717 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f34800a7950 0x7f34800b8df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:31:14.115 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:14.111 7f347effd700 1 --2- 172.21.2.102:0/4209923717 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f34801a7490 0x7f34801a7890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:31:14.115 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:14.111 7f347ffff700 1 --2- 172.21.2.102:0/4209923717 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f34801a7eb0 0x7f34800ac630 secure :-1 s=READY pgs=2112 cs=0 l=1 rev1=1 rx=0x7f3478001c10 tx=0x7f347800ac60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:14.115 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:14.112 7f3477fff700 1 -- 172.21.2.102:0/4209923717 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3478007020 con 0x7f34801a7eb0 2021-02-10T11:31:14.115 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:14.112 7f3477fff700 1 -- 172.21.2.102:0/4209923717 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3478007320 con 0x7f34801a7eb0 2021-02-10T11:31:14.115 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:14.112 7f3485a9d700 1 -- 172.21.2.102:0/4209923717 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34800ad050 con 0x7f34801a7eb0 2021-02-10T11:31:14.115 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:14.112 7f3477fff700 1 -- 172.21.2.102:0/4209923717 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3478004720 con 0x7f34801a7eb0 2021-02-10T11:31:14.116 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:14.112 7f3485a9d700 1 -- 172.21.2.102:0/4209923717 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34800a8530 con 0x7f34801a7eb0 2021-02-10T11:31:14.116 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:14.112 7f3477fff700 1 -- 172.21.2.102:0/4209923717 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f347800ada0 con 0x7f34801a7eb0 2021-02-10T11:31:14.116 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:14.112 7f3477fff700 1 --2- 172.21.2.102:0/4209923717 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f34640186c0 0x7f346401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:14.116 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:14.112 7f3477fff700 1 -- 172.21.2.102:0/4209923717 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f3478032780 con 0x7f34801a7eb0 2021-02-10T11:31:14.116 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:14.112 7f347f7fe700 1 --2- 172.21.2.102:0/4209923717 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f34640186c0 0x7f346401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:14.116 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:14.112 7f347f7fe700 1 --2- 172.21.2.102:0/4209923717 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f34640186c0 0x7f346401abc0 crc :-1 s=READY pgs=936 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:14.143 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:14.143 7f3485a9d700 1 -- 172.21.2.102:0/4209923717 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3480082a70 con 0x7f34801a7eb0 2021-02-10T11:31:14.145 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:14.145 7f3477fff700 1 -- 172.21.2.102:0/4209923717 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f3478000f90 con 0x7f34801a7eb0 2021-02-10T11:31:14.276 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:14.276 7f3485a9d700 1 -- 172.21.2.102:0/4209923717 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7f34800a84f0 con 0x7f34801a7eb0 2021-02-10T11:31:14.276 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:14.276 7f3477fff700 1 -- 172.21.2.102:0/4209923717 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v103) v1 ==== 104+0+10 (secure 0 0 0) 0x7f3478000f90 con 0x7f34801a7eb0 2021-02-10T11:31:14.276 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:31:14.278 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:14.278 7f3485a9d700 1 -- 172.21.2.102:0/4209923717 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f34640186c0 msgr2=0x7f346401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:14.278 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:14.278 7f3485a9d700 1 --2- 172.21.2.102:0/4209923717 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f34640186c0 0x7f346401abc0 crc :-1 s=READY pgs=936 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:14.278 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:14.278 7f3485a9d700 1 -- 172.21.2.102:0/4209923717 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f34801a7eb0 msgr2=0x7f34800ac630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:14.278 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:14.278 7f3485a9d700 1 --2- 172.21.2.102:0/4209923717 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f34801a7eb0 0x7f34800ac630 secure :-1 s=READY pgs=2112 cs=0 l=1 rev1=1 rx=0x7f3478001c10 tx=0x7f347800ac60).stop 2021-02-10T11:31:14.279 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:14.278 7f3485a9d700 1 -- 172.21.2.102:0/4209923717 shutdown_connections 2021-02-10T11:31:14.279 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:14.278 7f3485a9d700 1 --2- 172.21.2.102:0/4209923717 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f34801a7eb0 0x7f34800ac630 unknown :-1 s=CLOSED pgs=2112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:14.279 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:14.278 7f3485a9d700 1 --2- 172.21.2.102:0/4209923717 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f34640186c0 0x7f346401abc0 unknown :-1 s=CLOSED pgs=936 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:14.279 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:14.278 7f3485a9d700 1 --2- 172.21.2.102:0/4209923717 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f34801a7490 0x7f34801a7890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:14.279 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:14.278 7f3485a9d700 1 --2- 172.21.2.102:0/4209923717 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f34800a7950 0x7f34800b8df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:14.279 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:14.278 7f3485a9d700 1 -- 172.21.2.102:0/4209923717 >> 172.21.2.102:0/4209923717 conn(0x7f34800a02d0 msgr2=0x7f348009f200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:14.279 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:14.278 7f3485a9d700 1 -- 172.21.2.102:0/4209923717 shutdown_connections 2021-02-10T11:31:14.279 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:14.279 7f3485a9d700 1 -- 172.21.2.102:0/4209923717 wait complete. 2021-02-10T11:31:14.286 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-02-10T11:31:14.718 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:14.701 7f283fffd700 1 Processor -- start 2021-02-10T11:31:14.718 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:14.701 7f283fffd700 1 -- start start 2021-02-10T11:31:14.718 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:14.702 7f283fffd700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f28380d1870 0x7f28381b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:14.718 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:14.702 7f283fffd700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f28381b0ae0 0x7f2838108940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:14.718 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:14.702 7f283fffd700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28380fe1c0 con 0x7f28380d1c70 2021-02-10T11:31:14.718 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:14.702 7f283fffd700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2838109a60 con 0x7f28381b0ae0 2021-02-10T11:31:14.718 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:14.702 7f283fffd700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2838109750 con 0x7f28380d1870 2021-02-10T11:31:14.718 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:14.702 7f283d618700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f28381b0ae0 0x7f2838108940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:14.718 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:14.702 7f283d618700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f28381b0ae0 0x7f2838108940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:33530/0 (socket says 172.21.2.102:33530) 2021-02-10T11:31:14.719 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:14.702 7f283d618700 1 -- 172.21.2.102:0/103269854 learned_addr learned my addr 172.21.2.102:0/103269854 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:31:14.719 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:14.702 7f283d618700 1 -- 172.21.2.102:0/103269854 >> v2:172.21.2.113:3300/0 conn(0x7f28380d1870 msgr2=0x7f28381b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:14.719 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:14.702 7f283d618700 1 --2- 172.21.2.102:0/103269854 >> v2:172.21.2.113:3300/0 conn(0x7f28380d1870 0x7f28381b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:14.719 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:14.702 7f283d618700 1 -- 172.21.2.102:0/103269854 >> v1:172.21.2.113:6789/0 conn(0x7f28380d1c70 legacy=0x7f2838108f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:14.719 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:14.702 7f283d618700 1 -- 172.21.2.102:0/103269854 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2838088070 con 0x7f28381b0ae0 2021-02-10T11:31:14.719 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:14.702 7f283d618700 1 --2- 172.21.2.102:0/103269854 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f28381b0ae0 0x7f2838108940 secure :-1 s=READY pgs=2113 cs=0 l=1 rev1=1 rx=0x7f282c006700 tx=0x7f282c009b80).ready entity=mon.0 client_cookie=ec525c437508fa8e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:14.719 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:14.703 7f283ce17700 1 -- 172.21.2.102:0/103269854 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f282c00c0c0 con 0x7f28381b0ae0 2021-02-10T11:31:14.719 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:14.703 7f283ce17700 1 -- 172.21.2.102:0/103269854 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f282c00c340 con 0x7f28381b0ae0 2021-02-10T11:31:14.719 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:14.703 7f283ce17700 1 -- 172.21.2.102:0/103269854 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f282c00c860 con 0x7f28381b0ae0 2021-02-10T11:31:14.719 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:14.703 7f283fffd700 1 -- 172.21.2.102:0/103269854 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f28381b0ae0 msgr2=0x7f2838108940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:14.719 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:14.703 7f283fffd700 1 --2- 172.21.2.102:0/103269854 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f28381b0ae0 0x7f2838108940 secure :-1 s=READY pgs=2113 cs=0 l=1 rev1=1 rx=0x7f282c006700 tx=0x7f282c009b80).stop 2021-02-10T11:31:14.719 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:14.703 7f283fffd700 1 -- 172.21.2.102:0/103269854 shutdown_connections 2021-02-10T11:31:14.719 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:14.703 7f283fffd700 1 --2- 172.21.2.102:0/103269854 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f28381b0ae0 0x7f2838108940 unknown :-1 s=CLOSED pgs=2113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:14.720 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:14.703 7f283fffd700 1 --2- 172.21.2.102:0/103269854 >> v2:172.21.2.113:3300/0 conn(0x7f28380d1870 0x7f28381b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:14.720 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:14.703 7f283fffd700 1 -- 172.21.2.102:0/103269854 >> 172.21.2.102:0/103269854 conn(0x7f28380ce420 msgr2=0x7f28380d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:14.720 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:14.704 7f283fffd700 1 -- 172.21.2.102:0/103269854 shutdown_connections 2021-02-10T11:31:14.720 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:14.704 7f283fffd700 1 -- 172.21.2.102:0/103269854 wait complete. 2021-02-10T11:31:14.721 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:14.718 7f283fffd700 1 Processor -- start 2021-02-10T11:31:14.721 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:14.718 7f283fffd700 1 -- start start 2021-02-10T11:31:14.721 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:14.719 7f283fffd700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f28380c3820 0x7f2838084cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:14.722 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:14.719 7f283fffd700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f28380a0290 0x7f28380852d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:14.722 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:14.719 7f283fffd700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2838086270 0x7f2838086670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:14.722 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:14.719 7f283fffd700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28380c0700 con 0x7f28380c3820 2021-02-10T11:31:14.722 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:14.719 7f283fffd700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28380858f0 con 0x7f2838086270 2021-02-10T11:31:14.722 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:14.719 7f283fffd700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2838085b10 con 0x7f28380a0290 2021-02-10T11:31:14.722 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:14.719 7f283de19700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f28380c3820 0x7f2838084cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:14.722 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:14.719 7f283de19700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f28380c3820 0x7f2838084cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:33534/0 (socket says 172.21.2.102:33534) 2021-02-10T11:31:14.723 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:14.719 7f283de19700 1 -- 172.21.2.102:0/1082881819 learned_addr learned my addr 172.21.2.102:0/1082881819 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:31:14.723 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:14.719 7f283e61a700 1 --2- 172.21.2.102:0/1082881819 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2838086270 0x7f2838086670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:14.723 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:14.719 7f283d618700 1 --2- 172.21.2.102:0/1082881819 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f28380a0290 0x7f28380852d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:14.723 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:14.719 7f283de19700 1 -- 172.21.2.102:0/1082881819 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f28380a0290 msgr2=0x7f28380852d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:14.723 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:14.719 7f283de19700 1 --2- 172.21.2.102:0/1082881819 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f28380a0290 0x7f28380852d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:14.723 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:14.719 7f283de19700 1 -- 172.21.2.102:0/1082881819 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2838086270 msgr2=0x7f2838086670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:14.723 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:14.719 7f283de19700 1 --2- 172.21.2.102:0/1082881819 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2838086270 0x7f2838086670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:14.723 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:14.719 7f283de19700 1 -- 172.21.2.102:0/1082881819 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2838086c90 con 0x7f28380c3820 2021-02-10T11:31:14.723 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:14.719 7f283e61a700 1 --2- 172.21.2.102:0/1082881819 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2838086270 0x7f2838086670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:31:14.723 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:14.719 7f283d618700 1 --2- 172.21.2.102:0/1082881819 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f28380a0290 0x7f28380852d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:31:14.723 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:14.719 7f283de19700 1 --2- 172.21.2.102:0/1082881819 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f28380c3820 0x7f2838084cb0 secure :-1 s=READY pgs=2114 cs=0 l=1 rev1=1 rx=0x7f2834006f00 tx=0x7f283400b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:14.723 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:14.719 7f282a7fc700 1 -- 172.21.2.102:0/1082881819 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f28340009b0 con 0x7f28380c3820 2021-02-10T11:31:14.723 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:14.720 7f282a7fc700 1 -- 172.21.2.102:0/1082881819 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2834006350 con 0x7f28380c3820 2021-02-10T11:31:14.724 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:14.720 7f283fffd700 1 -- 172.21.2.102:0/1082881819 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2838086c90 con 0x7f28380c3820 2021-02-10T11:31:14.724 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:14.720 7f282a7fc700 1 -- 172.21.2.102:0/1082881819 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2834000c90 con 0x7f28380c3820 2021-02-10T11:31:14.724 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:14.720 7f283fffd700 1 -- 172.21.2.102:0/1082881819 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2838086c90 con 0x7f28380c3820 2021-02-10T11:31:14.724 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:14.720 7f282a7fc700 1 -- 172.21.2.102:0/1082881819 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f2834007000 con 0x7f28380c3820 2021-02-10T11:31:14.724 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:14.720 7f282a7fc700 1 --2- 172.21.2.102:0/1082881819 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2820018730 0x7f282001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:14.724 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:14.720 7f282a7fc700 1 -- 172.21.2.102:0/1082881819 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f28340327b0 con 0x7f28380c3820 2021-02-10T11:31:14.724 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:14.720 7f283d618700 1 --2- 172.21.2.102:0/1082881819 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2820018730 0x7f282001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:14.724 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:14.721 7f283d618700 1 --2- 172.21.2.102:0/1082881819 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2820018730 0x7f282001abc0 crc :-1 s=READY pgs=937 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:14.751 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:14.751 7f283fffd700 1 -- 172.21.2.102:0/1082881819 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2838082a70 con 0x7f28380c3820 2021-02-10T11:31:14.754 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:14.754 7f282a7fc700 1 -- 172.21.2.102:0/1082881819 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f2834006b90 con 0x7f28380c3820 2021-02-10T11:31:14.884 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:14.884 7f283fffd700 1 -- 172.21.2.102:0/1082881819 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7f2838082a70 con 0x7f28380c3820 2021-02-10T11:31:14.885 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:14.884 7f282a7fc700 1 -- 172.21.2.102:0/1082881819 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v103) v1 ==== 100+0+10 (secure 0 0 0) 0x7f283400bd30 con 0x7f28380c3820 2021-02-10T11:31:14.885 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:31:14.887 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:14.886 7f283fffd700 1 -- 172.21.2.102:0/1082881819 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2820018730 msgr2=0x7f282001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:14.887 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:14.886 7f283fffd700 1 --2- 172.21.2.102:0/1082881819 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2820018730 0x7f282001abc0 crc :-1 s=READY pgs=937 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:14.887 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:14.886 7f283fffd700 1 -- 172.21.2.102:0/1082881819 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f28380c3820 msgr2=0x7f2838084cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:14.887 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:14.886 7f283fffd700 1 --2- 172.21.2.102:0/1082881819 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f28380c3820 0x7f2838084cb0 secure :-1 s=READY pgs=2114 cs=0 l=1 rev1=1 rx=0x7f2834006f00 tx=0x7f283400b940).stop 2021-02-10T11:31:14.887 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:14.887 7f283fffd700 1 -- 172.21.2.102:0/1082881819 shutdown_connections 2021-02-10T11:31:14.887 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:14.887 7f283fffd700 1 --2- 172.21.2.102:0/1082881819 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2820018730 0x7f282001abc0 unknown :-1 s=CLOSED pgs=937 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:14.887 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:14.887 7f283fffd700 1 --2- 172.21.2.102:0/1082881819 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2838086270 0x7f2838086670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:14.887 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:14.887 7f283fffd700 1 --2- 172.21.2.102:0/1082881819 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f28380a0290 0x7f28380852d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:14.887 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:14.887 7f283fffd700 1 --2- 172.21.2.102:0/1082881819 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f28380c3820 0x7f2838084cb0 unknown :-1 s=CLOSED pgs=2114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:14.887 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:14.887 7f283fffd700 1 -- 172.21.2.102:0/1082881819 >> 172.21.2.102:0/1082881819 conn(0x7f28380872c0 msgr2=0x7f2838095700 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:14.887 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:14.887 7f283fffd700 1 -- 172.21.2.102:0/1082881819 shutdown_connections 2021-02-10T11:31:14.887 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:14.887 7f283fffd700 1 -- 172.21.2.102:0/1082881819 wait complete. 2021-02-10T11:31:14.895 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-02-10T11:31:14.895 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:31:15.324 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:15.308 7f600ef72700 1 Processor -- start 2021-02-10T11:31:15.324 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:15.308 7f600ef72700 1 -- start start 2021-02-10T11:31:15.325 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:15.308 7f600ef72700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f60081b0870 0x7f60080f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:15.325 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:15.308 7f600ef72700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f60080efe00 0x7f60080f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:15.325 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:15.308 7f600ef72700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60081b0c70 con 0x7f60081b0470 2021-02-10T11:31:15.325 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:15.308 7f600ef72700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60080e5ea0 con 0x7f60080efe00 2021-02-10T11:31:15.325 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:15.308 7f600ef72700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60080f0b50 con 0x7f60081b0870 2021-02-10T11:31:15.325 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:15.309 7f600d58f700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f60080efe00 0x7f60080f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:15.325 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:15.309 7f600d58f700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f60080efe00 0x7f60080f0200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:33546/0 (socket says 172.21.2.102:33546) 2021-02-10T11:31:15.325 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:15.309 7f600d58f700 1 -- 172.21.2.102:0/3537816188 learned_addr learned my addr 172.21.2.102:0/3537816188 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:31:15.325 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:15.309 7f600d58f700 1 -- 172.21.2.102:0/3537816188 >> v2:172.21.2.113:3300/0 conn(0x7f60081b0870 msgr2=0x7f60080f20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:15.325 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:15.309 7f600d58f700 1 --2- 172.21.2.102:0/3537816188 >> v2:172.21.2.113:3300/0 conn(0x7f60081b0870 0x7f60080f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:15.325 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:15.309 7f600d58f700 1 -- 172.21.2.102:0/3537816188 >> v1:172.21.2.113:6789/0 conn(0x7f60081b0470 legacy=0x7f60080ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:15.325 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:15.309 7f600d58f700 1 -- 172.21.2.102:0/3537816188 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60080f0820 con 0x7f60080efe00 2021-02-10T11:31:15.325 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:15.309 7f600d58f700 1 --2- 172.21.2.102:0/3537816188 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f60080efe00 0x7f60080f0200 secure :-1 s=READY pgs=2115 cs=0 l=1 rev1=1 rx=0x7f6000006f00 tx=0x7f6000009f10).ready entity=mon.0 client_cookie=20c0f8d0371b0105 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:15.326 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:15.309 7f60077fe700 1 -- 172.21.2.102:0/3537816188 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f600000b3b0 con 0x7f60080efe00 2021-02-10T11:31:15.326 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:15.309 7f60077fe700 1 -- 172.21.2.102:0/3537816188 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f600000b630 con 0x7f60080efe00 2021-02-10T11:31:15.326 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:15.309 7f60077fe700 1 -- 172.21.2.102:0/3537816188 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f600000bb50 con 0x7f60080efe00 2021-02-10T11:31:15.326 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:15.309 7f600ef72700 1 -- 172.21.2.102:0/3537816188 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f60080efe00 msgr2=0x7f60080f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:15.326 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:15.309 7f600ef72700 1 --2- 172.21.2.102:0/3537816188 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f60080efe00 0x7f60080f0200 secure :-1 s=READY pgs=2115 cs=0 l=1 rev1=1 rx=0x7f6000006f00 tx=0x7f6000009f10).stop 2021-02-10T11:31:15.326 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:15.310 7f600ef72700 1 -- 172.21.2.102:0/3537816188 shutdown_connections 2021-02-10T11:31:15.326 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:15.310 7f600ef72700 1 --2- 172.21.2.102:0/3537816188 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f60080efe00 0x7f60080f0200 unknown :-1 s=CLOSED pgs=2115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:15.326 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:15.310 7f600ef72700 1 --2- 172.21.2.102:0/3537816188 >> v2:172.21.2.113:3300/0 conn(0x7f60081b0870 0x7f60080f20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:15.326 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:15.310 7f600ef72700 1 -- 172.21.2.102:0/3537816188 >> 172.21.2.102:0/3537816188 conn(0x7f60080ce3e0 msgr2=0x7f60080e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:15.326 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:15.310 7f600ef72700 1 -- 172.21.2.102:0/3537816188 shutdown_connections 2021-02-10T11:31:15.326 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:15.310 7f600ef72700 1 -- 172.21.2.102:0/3537816188 wait complete. 2021-02-10T11:31:15.328 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:15.325 7f600ef72700 1 Processor -- start 2021-02-10T11:31:15.328 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:15.325 7f600ef72700 1 -- start start 2021-02-10T11:31:15.328 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:15.325 7f600ef72700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6008092e40 0x7f6008093240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:15.328 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:15.325 7f600ef72700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6008093860 0x7f60081a83d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:15.328 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:15.325 7f600ef72700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f60081a89f0 0x7f60081a8df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:15.328 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:15.325 7f600ef72700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6008195ad0 con 0x7f60081a89f0 2021-02-10T11:31:15.328 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:15.325 7f600ef72700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60081a9450 con 0x7f6008093860 2021-02-10T11:31:15.328 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:15.325 7f600ef72700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60080b1b00 con 0x7f6008092e40 2021-02-10T11:31:15.328 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:15.325 7f600d58f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f60081a89f0 0x7f60081a8df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:15.328 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:15.325 7f600d58f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f60081a89f0 0x7f60081a8df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:33552/0 (socket says 172.21.2.102:33552) 2021-02-10T11:31:15.329 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:15.325 7f600d58f700 1 -- 172.21.2.102:0/2719711619 learned_addr learned my addr 172.21.2.102:0/2719711619 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:31:15.329 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:15.326 7f600d58f700 1 -- 172.21.2.102:0/2719711619 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6008092e40 msgr2=0x7f6008093240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:15.329 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:15.326 7f600d58f700 1 --2- 172.21.2.102:0/2719711619 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6008092e40 0x7f6008093240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:15.329 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:15.326 7f600d58f700 1 -- 172.21.2.102:0/2719711619 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6008093860 msgr2=0x7f60081a83d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:15.329 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:15.326 7f600d58f700 1 --2- 172.21.2.102:0/2719711619 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6008093860 0x7f60081a83d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:15.329 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:15.326 7f600d58f700 1 -- 172.21.2.102:0/2719711619 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60081a99d0 con 0x7f60081a89f0 2021-02-10T11:31:15.329 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:15.326 7f600d58f700 1 --2- 172.21.2.102:0/2719711619 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f60081a89f0 0x7f60081a8df0 secure :-1 s=READY pgs=2116 cs=0 l=1 rev1=1 rx=0x7f600000b9b0 tx=0x7f600000bd50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:15.329 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:15.326 7f60057fa700 1 -- 172.21.2.102:0/2719711619 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6000005820 con 0x7f60081a89f0 2021-02-10T11:31:15.329 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:15.326 7f60057fa700 1 -- 172.21.2.102:0/2719711619 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6000005aa0 con 0x7f60081a89f0 2021-02-10T11:31:15.329 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:15.326 7f600ef72700 1 -- 172.21.2.102:0/2719711619 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60081a99d0 con 0x7f60081a89f0 2021-02-10T11:31:15.329 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:15.326 7f600ef72700 1 -- 172.21.2.102:0/2719711619 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60081a99d0 con 0x7f60081a89f0 2021-02-10T11:31:15.329 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:15.326 7f60057fa700 1 -- 172.21.2.102:0/2719711619 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6000006030 con 0x7f60081a89f0 2021-02-10T11:31:15.329 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:15.327 7f60057fa700 1 -- 172.21.2.102:0/2719711619 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f6000006030 con 0x7f60081a89f0 2021-02-10T11:31:15.329 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:15.327 7f60057fa700 1 --2- 172.21.2.102:0/2719711619 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5fe8018730 0x7f5fe801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:15.330 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:15.327 7f60057fa700 1 -- 172.21.2.102:0/2719711619 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f6000032b40 con 0x7f60081a89f0 2021-02-10T11:31:15.330 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:15.327 7f600cd8e700 1 --2- 172.21.2.102:0/2719711619 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5fe8018730 0x7f5fe801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:15.330 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:15.327 7f600cd8e700 1 --2- 172.21.2.102:0/2719711619 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5fe8018730 0x7f5fe801ac10 crc :-1 s=READY pgs=938 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:15.358 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:15.358 7f600ef72700 1 -- 172.21.2.102:0/2719711619 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60080b1b00 con 0x7f60081a89f0 2021-02-10T11:31:15.361 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:15.361 7f60057fa700 1 -- 172.21.2.102:0/2719711619 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f6000000f90 con 0x7f60081a89f0 2021-02-10T11:31:15.488 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:15.488 7f600ef72700 1 -- 172.21.2.102:0/2719711619 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f60080b1b00 con 0x7f60081a89f0 2021-02-10T11:31:15.488 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:15.488 7f60057fa700 1 -- 172.21.2.102:0/2719711619 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+2101 (secure 0 0 0) 0x7f6000006030 con 0x7f60081a89f0 2021-02-10T11:31:15.489 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:31:15.489 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":82,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10056,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.102:6835/711305601","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6834","nonce":711305601},{"type":"v1","addr":"172.21.2.102:6835","nonce":711305601}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":78}],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:30:58.047057","modified":"2021-02-10 11:31:00.055485","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6353},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6353":{"gid":6353,"name":"b","rank":0,"incarnation":79,"state":"up:active","state_seq":3,"addr":"172.21.2.113:6833/1091618344","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6832","nonce":1091618344},{"type":"v1","addr":"172.21.2.113:6833","nonce":1091618344}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":10}]} 2021-02-10T11:31:15.489 INFO:teuthology.orchestra.run.gibba002.stderr:dumped fsmap epoch 82 2021-02-10T11:31:15.490 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:15.490 7f600ef72700 1 -- 172.21.2.102:0/2719711619 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5fe8018730 msgr2=0x7f5fe801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:15.490 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:15.490 7f600ef72700 1 --2- 172.21.2.102:0/2719711619 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5fe8018730 0x7f5fe801ac10 crc :-1 s=READY pgs=938 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:15.490 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:15.490 7f600ef72700 1 -- 172.21.2.102:0/2719711619 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f60081a89f0 msgr2=0x7f60081a8df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:15.490 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:15.490 7f600ef72700 1 --2- 172.21.2.102:0/2719711619 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f60081a89f0 0x7f60081a8df0 secure :-1 s=READY pgs=2116 cs=0 l=1 rev1=1 rx=0x7f600000b9b0 tx=0x7f600000bd50).stop 2021-02-10T11:31:15.491 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:15.491 7f600ef72700 1 -- 172.21.2.102:0/2719711619 shutdown_connections 2021-02-10T11:31:15.491 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:15.491 7f600ef72700 1 --2- 172.21.2.102:0/2719711619 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f60081a89f0 0x7f60081a8df0 unknown :-1 s=CLOSED pgs=2116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:15.491 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:15.491 7f600ef72700 1 --2- 172.21.2.102:0/2719711619 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5fe8018730 0x7f5fe801ac10 unknown :-1 s=CLOSED pgs=938 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:15.491 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:15.491 7f600ef72700 1 --2- 172.21.2.102:0/2719711619 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6008093860 0x7f60081a83d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:15.491 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:15.491 7f600ef72700 1 --2- 172.21.2.102:0/2719711619 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6008092e40 0x7f6008093240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:15.491 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:15.491 7f600ef72700 1 -- 172.21.2.102:0/2719711619 >> 172.21.2.102:0/2719711619 conn(0x7f600819f7a0 msgr2=0x7f60080940e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:15.491 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:15.491 7f600ef72700 1 -- 172.21.2.102:0/2719711619 shutdown_connections 2021-02-10T11:31:15.492 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:15.491 7f600ef72700 1 -- 172.21.2.102:0/2719711619 wait complete. 2021-02-10T11:31:15.701 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 82, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:30:58.047057', 'modified': '2021-02-10 11:31:00.055485', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6353}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6353': {'gid': 6353, 'name': 'b', 'rank': 0, 'incarnation': 79, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.113:6833/1091618344', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.113:6832', 'nonce': 1091618344}, {'type': 'v1', 'addr': '172.21.2.113:6833', 'nonce': 1091618344}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:31:15.701 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:31:15.701 DEBUG:teuthology.orchestra.run.gibba013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:31:16.099 INFO:teuthology.orchestra.run.gibba013.stdout:{ 2021-02-10T11:31:16.099 INFO:teuthology.orchestra.run.gibba013.stdout: "cluster_fsid": "68db30ea-dbce-4aef-a6df-f01c6fea0513", 2021-02-10T11:31:16.099 INFO:teuthology.orchestra.run.gibba013.stdout: "whoami": 0, 2021-02-10T11:31:16.100 INFO:teuthology.orchestra.run.gibba013.stdout: "id": 6353, 2021-02-10T11:31:16.100 INFO:teuthology.orchestra.run.gibba013.stdout: "want_state": "up:active", 2021-02-10T11:31:16.100 INFO:teuthology.orchestra.run.gibba013.stdout: "state": "up:active", 2021-02-10T11:31:16.100 INFO:teuthology.orchestra.run.gibba013.stdout: "rank_uptime": 18.050189177, 2021-02-10T11:31:16.100 INFO:teuthology.orchestra.run.gibba013.stdout: "mdsmap_epoch": 82, 2021-02-10T11:31:16.100 INFO:teuthology.orchestra.run.gibba013.stdout: "osdmap_epoch": 103, 2021-02-10T11:31:16.100 INFO:teuthology.orchestra.run.gibba013.stdout: "osdmap_epoch_barrier": 103, 2021-02-10T11:31:16.100 INFO:teuthology.orchestra.run.gibba013.stdout: "uptime": 23.688886318000002 2021-02-10T11:31:16.100 INFO:teuthology.orchestra.run.gibba013.stdout:} 2021-02-10T11:31:16.104 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "68db30ea-dbce-4aef-a6df-f01c6fea0513", "whoami": 0, "id": 6353, "want_state": "up:active", "state": "up:active", "rank_uptime": 18.050189177, "mdsmap_epoch": 82, "osdmap_epoch": 103, "osdmap_epoch_barrier": 103, "uptime": 23.688886318000002 } 2021-02-10T11:31:16.105 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2021-02-10T11:31:16.105 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2021-02-10T11:31:16.105 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.1 at ubuntu@gibba002.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.1... 2021-02-10T11:31:16.105 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:31:16.105 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec mkdir -p /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:31:16.440 DEBUG:teuthology.orchestra.run.gibba002:> sudo modprobe fuse 2021-02-10T11:31:16.784 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:31:16.785 DEBUG:teuthology.orchestra.run.gibba002:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:31:17.127 INFO:teuthology.orchestra.run.gibba002.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:31:17.127 INFO:teuthology.orchestra.run.gibba002.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:31:17.128 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:31:17.128 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:31:17.128 DEBUG:teuthology.orchestra.run.gibba002:> ls /sys/fs/fuse/connections 2021-02-10T11:31:17.464 INFO:teuthology.orchestra.run.gibba002.stdout:43 2021-02-10T11:31:17.464 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [43] 2021-02-10T11:31:17.464 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.1 /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:31:17.467 DEBUG:teuthology.orchestra.run.gibba002:> sudo modprobe fuse 2021-02-10T11:31:17.812 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:31:17.812 DEBUG:teuthology.orchestra.run.gibba002:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:31:17.879 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba002.stderr:2021-02-10 11:31:17.879 7f5db1551f80 -1 init, newargv = 0x55e7a40811c0 newargc=9 2021-02-10T11:31:17.880 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba002.stderr:ceph-fuse[178961]: starting ceph client 2021-02-10T11:31:17.884 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba002.stderr:ceph-fuse[178961]: starting fuse 2021-02-10T11:31:18.155 INFO:teuthology.orchestra.run.gibba002.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:31:18.155 INFO:teuthology.orchestra.run.gibba002.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:31:18.156 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:31:18.157 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:31:18.157 DEBUG:teuthology.orchestra.run.gibba002:> ls /sys/fs/fuse/connections 2021-02-10T11:31:18.493 INFO:teuthology.orchestra.run.gibba002.stdout:43 2021-02-10T11:31:18.493 INFO:teuthology.orchestra.run.gibba002.stdout:44 2021-02-10T11:31:18.493 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43, 44] 2021-02-10T11:31:18.493 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:31:18.494 DEBUG:teuthology.orchestra.run.gibba002:> sudo python3 -c ' 2021-02-10T11:31:18.494 DEBUG:teuthology.orchestra.run.gibba002:> import glob 2021-02-10T11:31:18.494 DEBUG:teuthology.orchestra.run.gibba002:> import re 2021-02-10T11:31:18.494 DEBUG:teuthology.orchestra.run.gibba002:> import os 2021-02-10T11:31:18.494 DEBUG:teuthology.orchestra.run.gibba002:> import subprocess 2021-02-10T11:31:18.494 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:31:18.495 DEBUG:teuthology.orchestra.run.gibba002:> def find_socket(client_name): 2021-02-10T11:31:18.495 DEBUG:teuthology.orchestra.run.gibba002:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2021-02-10T11:31:18.495 DEBUG:teuthology.orchestra.run.gibba002:> files = glob.glob(asok_path) 2021-02-10T11:31:18.495 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:31:18.495 DEBUG:teuthology.orchestra.run.gibba002:> # Given a non-glob path, it better be there 2021-02-10T11:31:18.495 DEBUG:teuthology.orchestra.run.gibba002:> if "*" not in asok_path: 2021-02-10T11:31:18.495 DEBUG:teuthology.orchestra.run.gibba002:> assert(len(files) == 1) 2021-02-10T11:31:18.496 DEBUG:teuthology.orchestra.run.gibba002:> return files[0] 2021-02-10T11:31:18.496 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:31:18.496 DEBUG:teuthology.orchestra.run.gibba002:> for f in files: 2021-02-10T11:31:18.496 DEBUG:teuthology.orchestra.run.gibba002:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2021-02-10T11:31:18.496 DEBUG:teuthology.orchestra.run.gibba002:> if os.path.exists("/proc/{0}".format(pid)): 2021-02-10T11:31:18.496 DEBUG:teuthology.orchestra.run.gibba002:> return f 2021-02-10T11:31:18.496 DEBUG:teuthology.orchestra.run.gibba002:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2021-02-10T11:31:18.496 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:31:18.496 DEBUG:teuthology.orchestra.run.gibba002:> print(find_socket("client.1")) 2021-02-10T11:31:18.496 DEBUG:teuthology.orchestra.run.gibba002:> ' 2021-02-10T11:31:18.858 INFO:teuthology.orchestra.run.gibba002.stdout:/var/run/ceph/ceph-client.1.178961.asok 2021-02-10T11:31:18.860 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.178961.asok 2021-02-10T11:31:18.861 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:31:18.861 DEBUG:teuthology.orchestra.run.gibba002:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.178961.asok status 2021-02-10T11:31:19.250 INFO:teuthology.orchestra.run.gibba002.stdout:{ 2021-02-10T11:31:19.250 INFO:teuthology.orchestra.run.gibba002.stdout: "metadata": { 2021-02-10T11:31:19.251 INFO:teuthology.orchestra.run.gibba002.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:31:19.251 INFO:teuthology.orchestra.run.gibba002.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:31:19.251 INFO:teuthology.orchestra.run.gibba002.stdout: "entity_id": "1", 2021-02-10T11:31:19.251 INFO:teuthology.orchestra.run.gibba002.stdout: "hostname": "gibba002", 2021-02-10T11:31:19.251 INFO:teuthology.orchestra.run.gibba002.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2021-02-10T11:31:19.251 INFO:teuthology.orchestra.run.gibba002.stdout: "pid": "178961", 2021-02-10T11:31:19.251 INFO:teuthology.orchestra.run.gibba002.stdout: "root": "/" 2021-02-10T11:31:19.251 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:31:19.251 INFO:teuthology.orchestra.run.gibba002.stdout: "dentry_count": 0, 2021-02-10T11:31:19.251 INFO:teuthology.orchestra.run.gibba002.stdout: "dentry_pinned_count": 0, 2021-02-10T11:31:19.251 INFO:teuthology.orchestra.run.gibba002.stdout: "id": 10269, 2021-02-10T11:31:19.252 INFO:teuthology.orchestra.run.gibba002.stdout: "inst": { 2021-02-10T11:31:19.252 INFO:teuthology.orchestra.run.gibba002.stdout: "name": { 2021-02-10T11:31:19.252 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "client", 2021-02-10T11:31:19.252 INFO:teuthology.orchestra.run.gibba002.stdout: "num": 10269 2021-02-10T11:31:19.252 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:31:19.252 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": { 2021-02-10T11:31:19.252 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:31:19.252 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:0", 2021-02-10T11:31:19.252 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 3393425805 2021-02-10T11:31:19.252 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:31:19.252 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:31:19.253 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": { 2021-02-10T11:31:19.253 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:31:19.253 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:0", 2021-02-10T11:31:19.253 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 3393425805 2021-02-10T11:31:19.253 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:31:19.253 INFO:teuthology.orchestra.run.gibba002.stdout: "inst_str": "client.10269 172.21.2.102:0/3393425805", 2021-02-10T11:31:19.253 INFO:teuthology.orchestra.run.gibba002.stdout: "addr_str": "172.21.2.102:0/3393425805", 2021-02-10T11:31:19.253 INFO:teuthology.orchestra.run.gibba002.stdout: "inode_count": 1, 2021-02-10T11:31:19.253 INFO:teuthology.orchestra.run.gibba002.stdout: "mds_epoch": 82, 2021-02-10T11:31:19.253 INFO:teuthology.orchestra.run.gibba002.stdout: "osd_epoch": 103, 2021-02-10T11:31:19.253 INFO:teuthology.orchestra.run.gibba002.stdout: "osd_epoch_barrier": 0, 2021-02-10T11:31:19.254 INFO:teuthology.orchestra.run.gibba002.stdout: "blacklisted": false 2021-02-10T11:31:19.254 INFO:teuthology.orchestra.run.gibba002.stdout:} 2021-02-10T11:31:19.255 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:31:19.255 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:31:19.255 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:31:19.594 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:31:19.595 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2021-02-10T11:31:19.595 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:31:19.595 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:31:19.940 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2021-02-10T11:31:20.372 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:20.355 7f3e5c2f0700 1 Processor -- start 2021-02-10T11:31:20.373 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:20.355 7f3e5c2f0700 1 -- start start 2021-02-10T11:31:20.373 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:20.355 7f3e5c2f0700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3e540d1c80 0x7f3e540cef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:20.373 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:20.355 7f3e5c2f0700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e540d0180 con 0x7f3e540cf590 2021-02-10T11:31:20.373 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:20.355 7f3e5c2f0700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e540fe1d0 con 0x7f3e540d1880 2021-02-10T11:31:20.373 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:20.355 7f3e5c2f0700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e541b0cc0 con 0x7f3e540d1c80 2021-02-10T11:31:20.373 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:20.356 7f3e5a90d700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f3e540cf590 0x7f3e540cf990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:54032/0 (socket says 172.21.2.102:54032) 2021-02-10T11:31:20.373 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:20.356 7f3e5a90d700 1 -- 172.21.2.102:0/734397397 learned_addr learned my addr 172.21.2.102:0/734397397 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:31:20.373 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:20.356 7f3e5910a700 1 -- 172.21.2.102:0/734397397 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2707540994 0 0) 0x7f3e4c001500 con 0x7f3e540cf590 2021-02-10T11:31:20.373 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:20.356 7f3e5910a700 1 -- 172.21.2.102:0/734397397 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e40002fe0 con 0x7f3e540cf590 2021-02-10T11:31:20.374 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:20.356 7f3e5990b700 1 --2- 172.21.2.102:0/734397397 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3e540d1c80 0x7f3e540cef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:20.374 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:20.356 7f3e5910a700 1 -- 172.21.2.102:0/734397397 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3597590995 0 0) 0x7f3e4c001460 con 0x7f3e540cf590 2021-02-10T11:31:20.374 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:20.356 7f3e5910a700 1 -- 172.21.2.102:0/734397397 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3e540d1c80 msgr2=0x7f3e540cef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:20.374 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:20.356 7f3e5910a700 1 --2- 172.21.2.102:0/734397397 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3e540d1c80 0x7f3e540cef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:20.374 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:20.356 7f3e5910a700 1 -- 172.21.2.102:0/734397397 >> v1:172.21.2.113:6789/0 conn(0x7f3e540d1880 legacy=0x7f3e541b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:20.374 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:20.356 7f3e5910a700 1 -- 172.21.2.102:0/734397397 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e54109a70 con 0x7f3e540cf590 2021-02-10T11:31:20.374 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:20.356 7f3e5910a700 1 -- 172.21.2.102:0/734397397 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7f3e4c001980 con 0x7f3e540cf590 2021-02-10T11:31:20.374 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:20.357 7f3e5910a700 1 -- 172.21.2.102:0/734397397 >> v1:172.21.2.102:6789/0 conn(0x7f3e540cf590 legacy=0x7f3e540cf990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:20.374 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:20.357 7f3e5910a700 1 --2- 172.21.2.102:0/734397397 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3e40003ac0 0x7f3e40005ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:20.374 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:20.357 7f3e5910a700 1 --2- 172.21.2.102:0/734397397 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3e40006770 0x7f3e40008c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:20.374 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:20.357 7f3e5910a700 1 --2- 172.21.2.102:0/734397397 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3e40009500 0x7f3e4000ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:20.375 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:20.357 7f3e5910a700 1 -- 172.21.2.102:0/734397397 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e4000c2d0 con 0x7f3e40003ac0 2021-02-10T11:31:20.375 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:20.357 7f3e5910a700 1 -- 172.21.2.102:0/734397397 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e4000c4f0 con 0x7f3e40009500 2021-02-10T11:31:20.375 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:20.357 7f3e5910a700 1 -- 172.21.2.102:0/734397397 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e40050720 con 0x7f3e40006770 2021-02-10T11:31:20.375 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:20.357 7f3e5a10c700 1 --2- 172.21.2.102:0/734397397 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3e40003ac0 0x7f3e40005ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:20.375 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:20.357 7f3e5990b700 1 --2- 172.21.2.102:0/734397397 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3e40006770 0x7f3e40008c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:20.375 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:20.357 7f3e5a90d700 1 --2- 172.21.2.102:0/734397397 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3e40009500 0x7f3e4000ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:20.375 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:20.357 7f3e5a10c700 1 -- 172.21.2.102:0/734397397 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3e40006770 msgr2=0x7f3e40008c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:20.375 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:20.357 7f3e5a10c700 1 --2- 172.21.2.102:0/734397397 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3e40006770 0x7f3e40008c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:20.375 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:20.357 7f3e5a10c700 1 -- 172.21.2.102:0/734397397 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3e40009500 msgr2=0x7f3e4000ba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:20.375 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:20.357 7f3e5a10c700 1 --2- 172.21.2.102:0/734397397 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3e40009500 0x7f3e4000ba20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:20.375 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:20.357 7f3e5a10c700 1 -- 172.21.2.102:0/734397397 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e40050980 con 0x7f3e40003ac0 2021-02-10T11:31:20.375 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:20.357 7f3e5a90d700 1 --2- 172.21.2.102:0/734397397 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3e40009500 0x7f3e4000ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:31:20.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:20.357 7f3e5990b700 1 --2- 172.21.2.102:0/734397397 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3e40006770 0x7f3e40008c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:31:20.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:20.357 7f3e5a10c700 1 --2- 172.21.2.102:0/734397397 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3e40003ac0 0x7f3e40005ed0 secure :-1 s=READY pgs=2121 cs=0 l=1 rev1=1 rx=0x7f3e50006ab0 tx=0x7f3e50009b40).ready entity=mon.0 client_cookie=7a4d3ddd60fe0039 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:20.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:20.357 7f3e5910a700 1 -- 172.21.2.102:0/734397397 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3e5000b2e0 con 0x7f3e40003ac0 2021-02-10T11:31:20.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:20.357 7f3e5910a700 1 -- 172.21.2.102:0/734397397 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e5000b560 con 0x7f3e40003ac0 2021-02-10T11:31:20.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:20.357 7f3e5910a700 1 -- 172.21.2.102:0/734397397 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3e5000ba40 con 0x7f3e40003ac0 2021-02-10T11:31:20.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:20.357 7f3e5c2f0700 1 -- 172.21.2.102:0/734397397 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3e40003ac0 msgr2=0x7f3e40005ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:20.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:20.357 7f3e5c2f0700 1 --2- 172.21.2.102:0/734397397 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3e40003ac0 0x7f3e40005ed0 secure :-1 s=READY pgs=2121 cs=0 l=1 rev1=1 rx=0x7f3e50006ab0 tx=0x7f3e50009b40).stop 2021-02-10T11:31:20.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:20.357 7f3e5a90d700 1 -- 172.21.2.102:0/734397397 reap_dead start 2021-02-10T11:31:20.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:20.358 7f3e5c2f0700 1 -- 172.21.2.102:0/734397397 shutdown_connections 2021-02-10T11:31:20.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:20.358 7f3e5c2f0700 1 -- 172.21.2.102:0/734397397 >> 172.21.2.102:0/734397397 conn(0x7f3e540ce430 msgr2=0x7f3e540d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:20.377 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:20.358 7f3e5c2f0700 1 -- 172.21.2.102:0/734397397 shutdown_connections 2021-02-10T11:31:20.377 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:20.358 7f3e5c2f0700 1 -- 172.21.2.102:0/734397397 wait complete. 2021-02-10T11:31:20.378 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:20.373 7f3e5c2f0700 1 Processor -- start 2021-02-10T11:31:20.378 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:20.373 7f3e5c2f0700 1 -- start start 2021-02-10T11:31:20.378 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:20.373 7f3e5c2f0700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3e541a3570 0x7f3e541a3970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:20.378 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:20.373 7f3e5c2f0700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3e5409c9f0 0x7f3e5409cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:20.378 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:20.373 7f3e5c2f0700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3e5409d410 0x7f3e54098170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:20.379 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:20.373 7f3e5c2f0700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e5409c470 con 0x7f3e5409d410 2021-02-10T11:31:20.379 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:20.373 7f3e5c2f0700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e540b2840 con 0x7f3e541a3570 2021-02-10T11:31:20.379 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:20.373 7f3e5c2f0700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e540b2a20 con 0x7f3e5409c9f0 2021-02-10T11:31:20.379 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:20.373 7f3e5a90d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3e5409d410 0x7f3e54098170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:20.379 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:20.373 7f3e5a90d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3e5409d410 0x7f3e54098170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:33586/0 (socket says 172.21.2.102:33586) 2021-02-10T11:31:20.379 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:20.373 7f3e5a90d700 1 -- 172.21.2.102:0/2054088601 learned_addr learned my addr 172.21.2.102:0/2054088601 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:31:20.379 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:20.374 7f3e5a90d700 1 -- 172.21.2.102:0/2054088601 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3e5409c9f0 msgr2=0x7f3e5409cdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:20.379 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:20.374 7f3e5a90d700 1 --2- 172.21.2.102:0/2054088601 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3e5409c9f0 0x7f3e5409cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:20.380 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:20.374 7f3e5a90d700 1 -- 172.21.2.102:0/2054088601 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3e541a3570 msgr2=0x7f3e541a3970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:20.380 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:20.374 7f3e5a90d700 1 --2- 172.21.2.102:0/2054088601 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3e541a3570 0x7f3e541a3970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:20.380 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:20.374 7f3e5a90d700 1 -- 172.21.2.102:0/2054088601 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e5409d880 con 0x7f3e5409d410 2021-02-10T11:31:20.380 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:20.374 7f3e5a90d700 1 --2- 172.21.2.102:0/2054088601 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3e5409d410 0x7f3e54098170 secure :-1 s=READY pgs=2122 cs=0 l=1 rev1=1 rx=0x7f3e4c007f60 tx=0x7f3e4c00ac70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:20.380 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:20.374 7f3e4affd700 1 -- 172.21.2.102:0/2054088601 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3e4c000da0 con 0x7f3e5409d410 2021-02-10T11:31:20.380 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:20.374 7f3e4affd700 1 -- 172.21.2.102:0/2054088601 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e4c004930 con 0x7f3e5409d410 2021-02-10T11:31:20.380 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:20.374 7f3e5c2f0700 1 -- 172.21.2.102:0/2054088601 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e5409d880 con 0x7f3e5409d410 2021-02-10T11:31:20.380 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:20.374 7f3e4affd700 1 -- 172.21.2.102:0/2054088601 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3e4c0070d0 con 0x7f3e5409d410 2021-02-10T11:31:20.380 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:20.374 7f3e5c2f0700 1 -- 172.21.2.102:0/2054088601 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e54098a80 con 0x7f3e5409d410 2021-02-10T11:31:20.380 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:20.375 7f3e4affd700 1 -- 172.21.2.102:0/2054088601 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f3e4c004b10 con 0x7f3e5409d410 2021-02-10T11:31:20.380 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:20.375 7f3e4affd700 1 --2- 172.21.2.102:0/2054088601 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3e3c018730 0x7f3e3c01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:20.381 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:20.375 7f3e4affd700 1 -- 172.21.2.102:0/2054088601 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f3e4c032990 con 0x7f3e5409d410 2021-02-10T11:31:20.381 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:20.375 7f3e5a10c700 1 --2- 172.21.2.102:0/2054088601 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3e3c018730 0x7f3e3c01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:20.381 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:20.375 7f3e5a10c700 1 --2- 172.21.2.102:0/2054088601 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3e3c018730 0x7f3e3c01ac10 crc :-1 s=READY pgs=939 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:20.406 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:20.405 7f3e5c2f0700 1 -- 172.21.2.102:0/2054088601 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e54082a70 con 0x7f3e5409d410 2021-02-10T11:31:20.408 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:20.408 7f3e4affd700 1 -- 172.21.2.102:0/2054088601 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f3e4c001470 con 0x7f3e5409d410 2021-02-10T11:31:20.544 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:20.544 7f3e5c2f0700 1 -- 172.21.2.102:0/2054088601 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f3e54082a70 con 0x7f3e5409d410 2021-02-10T11:31:21.479 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:21.479 7f3e4affd700 1 -- 172.21.2.102:0/2054088601 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v83) v1 ==== 136+0+0 (secure 0 0 0) 0x7f3e4c000f00 con 0x7f3e5409d410 2021-02-10T11:31:21.516 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:21.515 7f3e5c2f0700 1 -- 172.21.2.102:0/2054088601 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3e3c018730 msgr2=0x7f3e3c01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:21.516 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:21.515 7f3e5c2f0700 1 --2- 172.21.2.102:0/2054088601 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3e3c018730 0x7f3e3c01ac10 crc :-1 s=READY pgs=939 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:21.516 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:21.515 7f3e5c2f0700 1 -- 172.21.2.102:0/2054088601 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3e5409d410 msgr2=0x7f3e54098170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:21.516 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:21.515 7f3e5c2f0700 1 --2- 172.21.2.102:0/2054088601 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3e5409d410 0x7f3e54098170 secure :-1 s=READY pgs=2122 cs=0 l=1 rev1=1 rx=0x7f3e4c007f60 tx=0x7f3e4c00ac70).stop 2021-02-10T11:31:21.516 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:21.516 7f3e5c2f0700 1 -- 172.21.2.102:0/2054088601 shutdown_connections 2021-02-10T11:31:21.516 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:21.516 7f3e5c2f0700 1 --2- 172.21.2.102:0/2054088601 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3e5409d410 0x7f3e54098170 unknown :-1 s=CLOSED pgs=2122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:21.516 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:21.516 7f3e5c2f0700 1 --2- 172.21.2.102:0/2054088601 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3e5409c9f0 0x7f3e5409cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:21.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:21.516 7f3e5c2f0700 1 --2- 172.21.2.102:0/2054088601 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3e3c018730 0x7f3e3c01ac10 unknown :-1 s=CLOSED pgs=939 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:21.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:21.516 7f3e5c2f0700 1 --2- 172.21.2.102:0/2054088601 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3e541a3570 0x7f3e541a3970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:21.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:21.516 7f3e5c2f0700 1 -- 172.21.2.102:0/2054088601 >> 172.21.2.102:0/2054088601 conn(0x7f3e540c2cd0 msgr2=0x7f3e540b5950 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:21.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:21.516 7f3e5c2f0700 1 -- 172.21.2.102:0/2054088601 shutdown_connections 2021-02-10T11:31:21.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:21.516 7f3e5c2f0700 1 -- 172.21.2.102:0/2054088601 wait complete. 2021-02-10T11:31:21.524 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs volume ls 2021-02-10T11:31:21.951 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:21.933 7ff474249700 1 Processor -- start 2021-02-10T11:31:21.951 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:21.933 7ff474249700 1 -- start start 2021-02-10T11:31:21.951 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:21.934 7ff474249700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7ff46c0d1870 0x7ff46c1b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:21.951 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:21.934 7ff474249700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff46c1b0a10 0x7ff46c0cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:21.951 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:21.934 7ff474249700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff46c0f0b10 con 0x7ff46c0d1c70 2021-02-10T11:31:21.951 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:21.934 7ff474249700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff46c0e5ed0 con 0x7ff46c0d1870 2021-02-10T11:31:21.952 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:21.934 7ff474249700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff46c0f0650 con 0x7ff46c1b0a10 2021-02-10T11:31:21.952 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:21.934 7ff472065700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7ff46c0d1870 0x7ff46c1b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:21.952 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:21.934 7ff472866700 1 --1- >> v1:172.21.2.113:6789/0 conn(0x7ff46c0d1c70 0x7ff46c0cf950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.113:6789/0 says I am v1:172.21.2.102:33140/0 (socket says 172.21.2.102:33140) 2021-02-10T11:31:21.952 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:21.935 7ff472866700 1 -- 172.21.2.102:0/4027439340 learned_addr learned my addr 172.21.2.102:0/4027439340 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:31:21.952 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:21.934 7ff471864700 1 --2- 172.21.2.102:0/4027439340 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff46c1b0a10 0x7ff46c0cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:21.952 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:21.935 7ff471063700 1 -- 172.21.2.102:0/4027439340 <== mon.1 v1:172.21.2.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 792221452 0 0) 0x7ff464001510 con 0x7ff46c0d1c70 2021-02-10T11:31:21.952 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:21.935 7ff471063700 1 -- 172.21.2.102:0/4027439340 --> v1:172.21.2.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff458002fe0 con 0x7ff46c0d1c70 2021-02-10T11:31:21.952 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:21.935 7ff471864700 1 -- 172.21.2.102:0/4027439340 >> v2:172.21.2.113:3300/0 conn(0x7ff46c0d1870 msgr2=0x7ff46c1b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:21.952 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:21.935 7ff471864700 1 --2- 172.21.2.102:0/4027439340 >> v2:172.21.2.113:3300/0 conn(0x7ff46c0d1870 0x7ff46c1b03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:21.952 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:21.935 7ff471864700 1 -- 172.21.2.102:0/4027439340 >> v1:172.21.2.113:6789/0 conn(0x7ff46c0d1c70 legacy=0x7ff46c0cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:21.952 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:21.935 7ff471864700 1 -- 172.21.2.102:0/4027439340 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff46c0f0870 con 0x7ff46c1b0a10 2021-02-10T11:31:21.952 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:21.935 7ff472065700 1 --2- 172.21.2.102:0/4027439340 >> v2:172.21.2.113:3300/0 conn(0x7ff46c0d1870 0x7ff46c1b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:31:21.952 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:21.935 7ff471864700 1 --2- 172.21.2.102:0/4027439340 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff46c1b0a10 0x7ff46c0cf330 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7ff460006650 tx=0x7ff460009a80).ready entity=mon.2 client_cookie=9bb92cca4574fd50 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:21.953 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:21.936 7ff471063700 1 -- 172.21.2.102:0/4027439340 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff460006390 con 0x7ff46c1b0a10 2021-02-10T11:31:21.953 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:21.936 7ff471063700 1 -- 172.21.2.102:0/4027439340 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4600009b0 con 0x7ff46c1b0a10 2021-02-10T11:31:21.953 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:21.936 7ff471063700 1 -- 172.21.2.102:0/4027439340 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff46000c2b0 con 0x7ff46c1b0a10 2021-02-10T11:31:21.953 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:21.936 7ff474249700 1 -- 172.21.2.102:0/4027439340 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff46c1b0a10 msgr2=0x7ff46c0cf330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:21.953 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:21.936 7ff474249700 1 --2- 172.21.2.102:0/4027439340 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff46c1b0a10 0x7ff46c0cf330 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7ff460006650 tx=0x7ff460009a80).stop 2021-02-10T11:31:21.953 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:21.937 7ff474249700 1 -- 172.21.2.102:0/4027439340 shutdown_connections 2021-02-10T11:31:21.953 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:21.937 7ff474249700 1 --2- 172.21.2.102:0/4027439340 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff46c1b0a10 0x7ff46c0cf330 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:21.953 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:21.937 7ff474249700 1 --2- 172.21.2.102:0/4027439340 >> v2:172.21.2.113:3300/0 conn(0x7ff46c0d1870 0x7ff46c1b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:21.953 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:21.937 7ff474249700 1 -- 172.21.2.102:0/4027439340 >> 172.21.2.102:0/4027439340 conn(0x7ff46c0ce450 msgr2=0x7ff46c0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:21.953 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:21.937 7ff474249700 1 -- 172.21.2.102:0/4027439340 shutdown_connections 2021-02-10T11:31:21.953 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:21.937 7ff474249700 1 -- 172.21.2.102:0/4027439340 wait complete. 2021-02-10T11:31:21.955 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:21.951 7ff474249700 1 Processor -- start 2021-02-10T11:31:21.955 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:21.952 7ff474249700 1 -- start start 2021-02-10T11:31:21.955 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:21.952 7ff474249700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff46c19ffa0 0x7ff46c0eda10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:21.955 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:21.952 7ff474249700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff46c1a03a0 0x7ff46c0ee030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:21.955 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:21.952 7ff474249700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff46c0ee650 0x7ff46c0ebe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:21.955 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:21.952 7ff474249700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff46c0ed490 con 0x7ff46c19ffa0 2021-02-10T11:31:21.956 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:21.952 7ff474249700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff46c0eea50 con 0x7ff46c0ee650 2021-02-10T11:31:21.956 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:21.952 7ff474249700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff46c0eec30 con 0x7ff46c1a03a0 2021-02-10T11:31:21.956 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:21.952 7ff472065700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff46c19ffa0 0x7ff46c0eda10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:21.956 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:21.952 7ff472065700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff46c19ffa0 0x7ff46c0eda10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:33598/0 (socket says 172.21.2.102:33598) 2021-02-10T11:31:21.956 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:21.952 7ff472065700 1 -- 172.21.2.102:0/3983230005 learned_addr learned my addr 172.21.2.102:0/3983230005 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:31:21.956 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:21.953 7ff472866700 1 --2- 172.21.2.102:0/3983230005 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff46c0ee650 0x7ff46c0ebe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:21.956 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:21.953 7ff472065700 1 -- 172.21.2.102:0/3983230005 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff46c1a03a0 msgr2=0x7ff46c0ee030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:21.956 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:21.953 7ff471864700 1 --2- 172.21.2.102:0/3983230005 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff46c1a03a0 0x7ff46c0ee030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:21.956 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:21.953 7ff472065700 1 --2- 172.21.2.102:0/3983230005 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff46c1a03a0 0x7ff46c0ee030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:21.956 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:21.953 7ff472065700 1 -- 172.21.2.102:0/3983230005 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff46c0ee650 msgr2=0x7ff46c0ebe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:21.956 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:21.953 7ff472065700 1 --2- 172.21.2.102:0/3983230005 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff46c0ee650 0x7ff46c0ebe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:21.956 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:21.953 7ff472065700 1 -- 172.21.2.102:0/3983230005 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff46c0ec4a0 con 0x7ff46c19ffa0 2021-02-10T11:31:21.956 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:21.953 7ff472866700 1 --2- 172.21.2.102:0/3983230005 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff46c0ee650 0x7ff46c0ebe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:31:21.957 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:21.953 7ff471864700 1 --2- 172.21.2.102:0/3983230005 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff46c1a03a0 0x7ff46c0ee030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:31:21.957 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:21.953 7ff472065700 1 --2- 172.21.2.102:0/3983230005 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff46c19ffa0 0x7ff46c0eda10 secure :-1 s=READY pgs=2123 cs=0 l=1 rev1=1 rx=0x7ff468006f40 tx=0x7ff46800b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:21.957 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:21.953 7ff45effd700 1 -- 172.21.2.102:0/3983230005 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff46800bd80 con 0x7ff46c19ffa0 2021-02-10T11:31:21.957 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:21.953 7ff45effd700 1 -- 172.21.2.102:0/3983230005 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff468002030 con 0x7ff46c19ffa0 2021-02-10T11:31:21.957 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:21.953 7ff45effd700 1 -- 172.21.2.102:0/3983230005 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff468002510 con 0x7ff46c19ffa0 2021-02-10T11:31:21.957 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:21.953 7ff474249700 1 -- 172.21.2.102:0/3983230005 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff46c0ec4a0 con 0x7ff46c19ffa0 2021-02-10T11:31:21.957 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:21.953 7ff474249700 1 -- 172.21.2.102:0/3983230005 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff46c0eca30 con 0x7ff46c19ffa0 2021-02-10T11:31:21.957 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:21.954 7ff45effd700 1 -- 172.21.2.102:0/3983230005 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ff468000c90 con 0x7ff46c19ffa0 2021-02-10T11:31:21.957 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:21.954 7ff45effd700 1 --2- 172.21.2.102:0/3983230005 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff4540186d0 0x7ff45401abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:21.957 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:21.954 7ff45effd700 1 -- 172.21.2.102:0/3983230005 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 5573+0+0 (secure 0 0 0) 0x7ff468032800 con 0x7ff46c19ffa0 2021-02-10T11:31:21.957 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:21.954 7ff471864700 1 --2- 172.21.2.102:0/3983230005 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff4540186d0 0x7ff45401abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:21.957 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:21.954 7ff471864700 1 --2- 172.21.2.102:0/3983230005 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff4540186d0 0x7ff45401abd0 crc :-1 s=READY pgs=940 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:21.984 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:21.984 7ff474249700 1 -- 172.21.2.102:0/3983230005 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff46c0eee50 con 0x7ff46c19ffa0 2021-02-10T11:31:21.986 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:21.986 7ff45effd700 1 -- 172.21.2.102:0/3983230005 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ff468000940 con 0x7ff46c19ffa0 2021-02-10T11:31:22.119 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:22.119 7ff474249700 1 -- 172.21.2.102:0/3983230005 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs volume ls", "target": ["mgr", ""]}) v1 -- 0x7ff46c0eca30 con 0x7ff4540186d0 2021-02-10T11:31:22.119 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:22.119 7ff45effd700 1 -- 172.21.2.102:0/3983230005 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+41 (crc 0 0 0) 0x7ff4600009b0 con 0x7ff4540186d0 2021-02-10T11:31:22.119 INFO:teuthology.orchestra.run.gibba002.stdout:[ 2021-02-10T11:31:22.120 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:31:22.120 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "cephfs" 2021-02-10T11:31:22.120 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:31:22.120 INFO:teuthology.orchestra.run.gibba002.stdout:] 2021-02-10T11:31:22.121 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:22.121 7ff474249700 1 -- 172.21.2.102:0/3983230005 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff4540186d0 msgr2=0x7ff45401abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:22.121 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:22.121 7ff474249700 1 --2- 172.21.2.102:0/3983230005 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff4540186d0 0x7ff45401abd0 crc :-1 s=READY pgs=940 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:22.121 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:22.121 7ff474249700 1 -- 172.21.2.102:0/3983230005 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff46c19ffa0 msgr2=0x7ff46c0eda10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:22.121 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:22.121 7ff474249700 1 --2- 172.21.2.102:0/3983230005 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff46c19ffa0 0x7ff46c0eda10 secure :-1 s=READY pgs=2123 cs=0 l=1 rev1=1 rx=0x7ff468006f40 tx=0x7ff46800b990).stop 2021-02-10T11:31:22.121 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:22.121 7ff474249700 1 -- 172.21.2.102:0/3983230005 shutdown_connections 2021-02-10T11:31:22.122 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:22.121 7ff474249700 1 --2- 172.21.2.102:0/3983230005 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff4540186d0 0x7ff45401abd0 unknown :-1 s=CLOSED pgs=940 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:22.122 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:22.121 7ff474249700 1 --2- 172.21.2.102:0/3983230005 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff46c0ee650 0x7ff46c0ebe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:22.122 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:22.121 7ff474249700 1 --2- 172.21.2.102:0/3983230005 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff46c1a03a0 0x7ff46c0ee030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:22.122 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:22.121 7ff474249700 1 --2- 172.21.2.102:0/3983230005 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff46c19ffa0 0x7ff46c0eda10 unknown :-1 s=CLOSED pgs=2123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:22.122 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:22.121 7ff474249700 1 -- 172.21.2.102:0/3983230005 >> 172.21.2.102:0/3983230005 conn(0x7ff46c094d90 msgr2=0x7ff46c08a420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:22.122 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:22.121 7ff474249700 1 -- 172.21.2.102:0/3983230005 shutdown_connections 2021-02-10T11:31:22.122 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:22.121 7ff474249700 1 -- 172.21.2.102:0/3983230005 wait complete. 2021-02-10T11:31:22.129 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config set mon mon_allow_pool_delete True 2021-02-10T11:31:22.561 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:22.544 7f196a6f8700 1 Processor -- start 2021-02-10T11:31:22.561 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:22.544 7f196a6f8700 1 -- start start 2021-02-10T11:31:22.561 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:22.545 7f196a6f8700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f19640d1870 0x7f19641b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:22.562 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:22.545 7f196a6f8700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f19641b0ae0 0x7f1964108940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:22.562 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:22.545 7f196a6f8700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1964109a60 con 0x7f19640d1c70 2021-02-10T11:31:22.562 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:22.545 7f196a6f8700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19640e5ef0 con 0x7f19641b0ae0 2021-02-10T11:31:22.562 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:22.545 7f196a6f8700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1964109750 con 0x7f19640d1870 2021-02-10T11:31:22.562 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:22.545 7f1968d15700 1 --1- >> v1:172.21.2.113:6789/0 conn(0x7f19640d1c70 0x7f1964108f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.113:6789/0 says I am v1:172.21.2.102:33156/0 (socket says 172.21.2.102:33156) 2021-02-10T11:31:22.562 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:22.545 7f1968d15700 1 -- 172.21.2.102:0/1233251954 learned_addr learned my addr 172.21.2.102:0/1233251954 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:31:22.562 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:22.545 7f19637fe700 1 --2- 172.21.2.102:0/1233251954 >> v2:172.21.2.113:3300/0 conn(0x7f19641b0ae0 0x7f1964108940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:22.562 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:22.545 7f1963fff700 1 --2- 172.21.2.102:0/1233251954 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f19640d1870 0x7f19641b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:22.562 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:22.546 7f1963fff700 1 -- 172.21.2.102:0/1233251954 >> v2:172.21.2.113:3300/0 conn(0x7f19641b0ae0 msgr2=0x7f1964108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:22.562 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:22.546 7f1963fff700 1 --2- 172.21.2.102:0/1233251954 >> v2:172.21.2.113:3300/0 conn(0x7f19641b0ae0 0x7f1964108940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:22.562 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:22.546 7f1963fff700 1 -- 172.21.2.102:0/1233251954 >> v1:172.21.2.113:6789/0 conn(0x7f19640d1c70 legacy=0x7f1964108f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:22.562 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:22.546 7f1963fff700 1 -- 172.21.2.102:0/1233251954 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1964088070 con 0x7f19640d1870 2021-02-10T11:31:22.563 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:22.546 7f19637fe700 1 --2- 172.21.2.102:0/1233251954 >> v2:172.21.2.113:3300/0 conn(0x7f19641b0ae0 0x7f1964108940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:31:22.563 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:22.546 7f1963fff700 1 --2- 172.21.2.102:0/1233251954 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f19640d1870 0x7f19641b04c0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7f1954006650 tx=0x7f1954009a80).ready entity=mon.2 client_cookie=513354013cd466d3 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:22.563 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:22.546 7f1962ffd700 1 -- 172.21.2.102:0/1233251954 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f195400c0c0 con 0x7f19640d1870 2021-02-10T11:31:22.563 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:22.546 7f1962ffd700 1 -- 172.21.2.102:0/1233251954 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f195400c340 con 0x7f19640d1870 2021-02-10T11:31:22.563 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:22.546 7f1962ffd700 1 -- 172.21.2.102:0/1233251954 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f195400c860 con 0x7f19640d1870 2021-02-10T11:31:22.563 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:22.546 7f196a6f8700 1 -- 172.21.2.102:0/1233251954 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f19640d1870 msgr2=0x7f19641b04c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:22.563 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:22.546 7f196a6f8700 1 --2- 172.21.2.102:0/1233251954 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f19640d1870 0x7f19641b04c0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7f1954006650 tx=0x7f1954009a80).stop 2021-02-10T11:31:22.563 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:22.547 7f196a6f8700 1 -- 172.21.2.102:0/1233251954 shutdown_connections 2021-02-10T11:31:22.563 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:22.547 7f196a6f8700 1 --2- 172.21.2.102:0/1233251954 >> v2:172.21.2.113:3300/0 conn(0x7f19641b0ae0 0x7f1964108940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:22.563 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:22.547 7f196a6f8700 1 --2- 172.21.2.102:0/1233251954 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f19640d1870 0x7f19641b04c0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:22.563 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:22.547 7f196a6f8700 1 -- 172.21.2.102:0/1233251954 >> 172.21.2.102:0/1233251954 conn(0x7f19640ce420 msgr2=0x7f19640d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:22.563 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:22.547 7f196a6f8700 1 -- 172.21.2.102:0/1233251954 shutdown_connections 2021-02-10T11:31:22.563 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:22.547 7f196a6f8700 1 -- 172.21.2.102:0/1233251954 wait complete. 2021-02-10T11:31:22.564 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:22.561 7f196a6f8700 1 Processor -- start 2021-02-10T11:31:22.565 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:22.561 7f196a6f8700 1 -- start start 2021-02-10T11:31:22.566 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:22.562 7f196a6f8700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f19640abd60 0x7f19640ac160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:22.566 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:22.562 7f196a6f8700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f19640ac780 0x7f19640acb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:22.566 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:22.562 7f196a6f8700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1964083910 0x7f1964083d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:22.566 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:22.562 7f196a6f8700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1964087180 con 0x7f1964083910 2021-02-10T11:31:22.566 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:22.562 7f196a6f8700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f196418fcd0 con 0x7f19640abd60 2021-02-10T11:31:22.566 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:22.562 7f196a6f8700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1964084330 con 0x7f19640ac780 2021-02-10T11:31:22.566 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:22.562 7f1968d15700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1964083910 0x7f1964083d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:22.566 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:22.562 7f1968d15700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1964083910 0x7f1964083d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:33614/0 (socket says 172.21.2.102:33614) 2021-02-10T11:31:22.566 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:22.562 7f1968d15700 1 -- 172.21.2.102:0/1156664012 learned_addr learned my addr 172.21.2.102:0/1156664012 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:31:22.567 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:22.562 7f1968d15700 1 -- 172.21.2.102:0/1156664012 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f19640ac780 msgr2=0x7f19640acb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:22.567 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:22.562 7f1968d15700 1 --2- 172.21.2.102:0/1156664012 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f19640ac780 0x7f19640acb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:22.567 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:22.562 7f1968d15700 1 -- 172.21.2.102:0/1156664012 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f19640abd60 msgr2=0x7f19640ac160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:22.567 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:22.562 7f1968d15700 1 --2- 172.21.2.102:0/1156664012 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f19640abd60 0x7f19640ac160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:22.567 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:22.562 7f1968d15700 1 -- 172.21.2.102:0/1156664012 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1964084550 con 0x7f1964083910 2021-02-10T11:31:22.567 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:22.563 7f1968d15700 1 --2- 172.21.2.102:0/1156664012 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1964083910 0x7f1964083d10 secure :-1 s=READY pgs=2124 cs=0 l=1 rev1=1 rx=0x7f195c00a900 tx=0x7f195c00aca0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:22.567 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:22.563 7f1960ff9700 1 -- 172.21.2.102:0/1156664012 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f195c0070d0 con 0x7f1964083910 2021-02-10T11:31:22.567 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:22.563 7f1960ff9700 1 -- 172.21.2.102:0/1156664012 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f195c0073d0 con 0x7f1964083910 2021-02-10T11:31:22.567 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:22.563 7f1960ff9700 1 -- 172.21.2.102:0/1156664012 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f195c004920 con 0x7f1964083910 2021-02-10T11:31:22.567 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:22.563 7f196a6f8700 1 -- 172.21.2.102:0/1156664012 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1964084550 con 0x7f1964083910 2021-02-10T11:31:22.567 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:22.563 7f196a6f8700 1 -- 172.21.2.102:0/1156664012 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1964084ae0 con 0x7f1964083910 2021-02-10T11:31:22.567 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:22.564 7f1960ff9700 1 -- 172.21.2.102:0/1156664012 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f195c0046a0 con 0x7f1964083910 2021-02-10T11:31:22.567 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:22.564 7f1960ff9700 1 --2- 172.21.2.102:0/1156664012 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f19440186c0 0x7f194401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:22.568 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:22.564 7f1960ff9700 1 -- 172.21.2.102:0/1156664012 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f195c032810 con 0x7f1964083910 2021-02-10T11:31:22.568 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:22.564 7f1963fff700 1 --2- 172.21.2.102:0/1156664012 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f19440186c0 0x7f194401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:22.568 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:22.564 7f1963fff700 1 --2- 172.21.2.102:0/1156664012 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f19440186c0 0x7f194401abc0 crc :-1 s=READY pgs=941 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:22.594 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:22.594 7f196a6f8700 1 -- 172.21.2.102:0/1156664012 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1964084d90 con 0x7f1964083910 2021-02-10T11:31:22.597 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:22.597 7f1960ff9700 1 -- 172.21.2.102:0/1156664012 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f195c02fdf0 con 0x7f1964083910 2021-02-10T11:31:22.745 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:22.745 7f196a6f8700 1 -- 172.21.2.102:0/1156664012 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f1964084d90 con 0x7f1964083910 2021-02-10T11:31:22.820 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:22.745 7f1960ff9700 1 -- 172.21.2.102:0/1156664012 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v26)=0 v26) v1 ==== 122+0+0 (secure 0 0 0) 0x7f195c02fdf0 con 0x7f1964083910 2021-02-10T11:31:22.820 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:22.747 7f196a6f8700 1 -- 172.21.2.102:0/1156664012 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f19440186c0 msgr2=0x7f194401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:22.820 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:22.747 7f196a6f8700 1 --2- 172.21.2.102:0/1156664012 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f19440186c0 0x7f194401abc0 crc :-1 s=READY pgs=941 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:22.820 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:22.747 7f196a6f8700 1 -- 172.21.2.102:0/1156664012 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1964083910 msgr2=0x7f1964083d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:22.820 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:22.747 7f196a6f8700 1 --2- 172.21.2.102:0/1156664012 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1964083910 0x7f1964083d10 secure :-1 s=READY pgs=2124 cs=0 l=1 rev1=1 rx=0x7f195c00a900 tx=0x7f195c00aca0).stop 2021-02-10T11:31:22.820 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:22.747 7f196a6f8700 1 -- 172.21.2.102:0/1156664012 shutdown_connections 2021-02-10T11:31:22.820 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:22.747 7f196a6f8700 1 --2- 172.21.2.102:0/1156664012 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1964083910 0x7f1964083d10 unknown :-1 s=CLOSED pgs=2124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:22.820 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:22.747 7f196a6f8700 1 --2- 172.21.2.102:0/1156664012 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f19640ac780 0x7f19640acb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:22.821 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:22.747 7f196a6f8700 1 --2- 172.21.2.102:0/1156664012 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f19440186c0 0x7f194401abc0 unknown :-1 s=CLOSED pgs=941 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:22.821 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:22.747 7f196a6f8700 1 --2- 172.21.2.102:0/1156664012 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f19640abd60 0x7f19640ac160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:22.821 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:22.747 7f196a6f8700 1 -- 172.21.2.102:0/1156664012 >> 172.21.2.102:0/1156664012 conn(0x7f19640a2b30 msgr2=0x7f196409ce60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:22.821 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:22.747 7f196a6f8700 1 -- 172.21.2.102:0/1156664012 shutdown_connections 2021-02-10T11:31:22.821 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:22.747 7f196a6f8700 1 -- 172.21.2.102:0/1156664012 wait complete. 2021-02-10T11:31:22.821 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000946240 2021-02-10T11:31:23.276 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:23.256 7f6052688700 1 Processor -- start 2021-02-10T11:31:23.276 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:23.256 7f6052688700 1 -- start start 2021-02-10T11:31:23.276 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:23.256 7f6052688700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f604c0d1870 0x7f604c1b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:23.276 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:23.256 7f6052688700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f604c0cff70 0x7f604c109130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:23.276 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:23.256 7f6052688700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f604c109a60 con 0x7f604c1b0ae0 2021-02-10T11:31:23.276 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:23.256 7f6052688700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f604c109750 con 0x7f604c0cff70 2021-02-10T11:31:23.277 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:23.256 7f6052688700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f604c088070 con 0x7f604c0d1870 2021-02-10T11:31:23.277 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:23.257 7f604b7fe700 1 --1- >> v1:172.21.2.113:6789/0 conn(0x7f604c1b0ae0 0x7f604c108940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.113:6789/0 says I am v1:172.21.2.102:33166/0 (socket says 172.21.2.102:33166) 2021-02-10T11:31:23.277 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:23.257 7f6050ca5700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f604c0cff70 0x7f604c109130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:23.277 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:23.257 7f604b7fe700 1 -- 172.21.2.102:0/597234434 learned_addr learned my addr 172.21.2.102:0/597234434 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:31:23.277 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:23.257 7f604bfff700 1 --2- 172.21.2.102:0/597234434 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f604c0d1870 0x7f604c1b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:23.277 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:23.258 7f604bfff700 1 -- 172.21.2.102:0/597234434 >> v2:172.21.2.113:3300/0 conn(0x7f604c0cff70 msgr2=0x7f604c109130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:23.277 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:23.258 7f604bfff700 1 --2- 172.21.2.102:0/597234434 >> v2:172.21.2.113:3300/0 conn(0x7f604c0cff70 0x7f604c109130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:23.277 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:23.258 7f604bfff700 1 -- 172.21.2.102:0/597234434 >> v1:172.21.2.113:6789/0 conn(0x7f604c1b0ae0 legacy=0x7f604c108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:23.277 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:23.258 7f604bfff700 1 -- 172.21.2.102:0/597234434 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f604c088260 con 0x7f604c0d1870 2021-02-10T11:31:23.277 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:23.258 7f604bfff700 1 --2- 172.21.2.102:0/597234434 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f604c0d1870 0x7f604c1b04c0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f603c006650 tx=0x7f603c009a80).ready entity=mon.2 client_cookie=7f0ba3072dcbc13b server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:23.277 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:23.258 7f604affd700 1 -- 172.21.2.102:0/597234434 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f603c00c0c0 con 0x7f604c0d1870 2021-02-10T11:31:23.277 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:23.258 7f604affd700 1 -- 172.21.2.102:0/597234434 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f603c00c340 con 0x7f604c0d1870 2021-02-10T11:31:23.278 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:23.258 7f604affd700 1 -- 172.21.2.102:0/597234434 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f603c00c860 con 0x7f604c0d1870 2021-02-10T11:31:23.278 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:23.258 7f6052688700 1 -- 172.21.2.102:0/597234434 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f604c0d1870 msgr2=0x7f604c1b04c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:23.278 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:23.258 7f6052688700 1 --2- 172.21.2.102:0/597234434 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f604c0d1870 0x7f604c1b04c0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f603c006650 tx=0x7f603c009a80).stop 2021-02-10T11:31:23.278 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:23.259 7f6052688700 1 -- 172.21.2.102:0/597234434 shutdown_connections 2021-02-10T11:31:23.278 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:23.259 7f6052688700 1 --2- 172.21.2.102:0/597234434 >> v2:172.21.2.113:3300/0 conn(0x7f604c0cff70 0x7f604c109130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:23.278 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:23.259 7f6052688700 1 --2- 172.21.2.102:0/597234434 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f604c0d1870 0x7f604c1b04c0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:23.278 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:23.259 7f6052688700 1 -- 172.21.2.102:0/597234434 >> 172.21.2.102:0/597234434 conn(0x7f604c0ce420 msgr2=0x7f604c0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:23.278 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:23.259 7f6052688700 1 -- 172.21.2.102:0/597234434 shutdown_connections 2021-02-10T11:31:23.278 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:23.259 7f6052688700 1 -- 172.21.2.102:0/597234434 wait complete. 2021-02-10T11:31:23.278 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:23.274 7f6052688700 1 Processor -- start 2021-02-10T11:31:23.278 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:23.274 7f6052688700 1 -- start start 2021-02-10T11:31:23.278 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:23.274 7f6052688700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f604c0a2580 0x7f604c0a1cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:23.278 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:23.274 7f6052688700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f604c094c40 0x7f604c1a75e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:23.279 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:23.274 7f6052688700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f604c095040 0x7f604c1a7c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:23.279 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:23.274 7f6052688700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f604c188180 con 0x7f604c094c40 2021-02-10T11:31:23.279 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:23.274 7f6052688700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f604c187e70 con 0x7f604c095040 2021-02-10T11:31:23.279 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:23.274 7f6052688700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f604c1a8220 con 0x7f604c0a2580 2021-02-10T11:31:23.279 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:23.274 7f604b7fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f604c094c40 0x7f604c1a75e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:23.279 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:23.274 7f604b7fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f604c094c40 0x7f604c1a75e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:33628/0 (socket says 172.21.2.102:33628) 2021-02-10T11:31:23.279 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:23.274 7f604b7fe700 1 -- 172.21.2.102:0/674748598 learned_addr learned my addr 172.21.2.102:0/674748598 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:31:23.280 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:23.275 7f604b7fe700 1 -- 172.21.2.102:0/674748598 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f604c0a2580 msgr2=0x7f604c0a1cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:23.280 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:23.275 7f604b7fe700 1 --2- 172.21.2.102:0/674748598 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f604c0a2580 0x7f604c0a1cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:23.280 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:23.275 7f604b7fe700 1 -- 172.21.2.102:0/674748598 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f604c095040 msgr2=0x7f604c1a7c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:23.280 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:23.275 7f604b7fe700 1 --2- 172.21.2.102:0/674748598 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f604c095040 0x7f604c1a7c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:23.280 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:23.275 7f604b7fe700 1 -- 172.21.2.102:0/674748598 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f604c188700 con 0x7f604c094c40 2021-02-10T11:31:23.280 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:23.275 7f604b7fe700 1 --2- 172.21.2.102:0/674748598 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f604c094c40 0x7f604c1a75e0 secure :-1 s=READY pgs=2125 cs=0 l=1 rev1=1 rx=0x7f6040001c60 tx=0x7f604000a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:23.280 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:23.275 7f6048ff9700 1 -- 172.21.2.102:0/674748598 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f604000acc0 con 0x7f604c094c40 2021-02-10T11:31:23.281 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:23.275 7f6048ff9700 1 -- 172.21.2.102:0/674748598 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6040004140 con 0x7f604c094c40 2021-02-10T11:31:23.281 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:23.275 7f6052688700 1 -- 172.21.2.102:0/674748598 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f604c188700 con 0x7f604c094c40 2021-02-10T11:31:23.281 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:23.275 7f6052688700 1 -- 172.21.2.102:0/674748598 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f604c188c90 con 0x7f604c094c40 2021-02-10T11:31:23.281 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:23.275 7f6048ff9700 1 -- 172.21.2.102:0/674748598 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6040004660 con 0x7f604c094c40 2021-02-10T11:31:23.281 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:23.276 7f6048ff9700 1 -- 172.21.2.102:0/674748598 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f6040004320 con 0x7f604c094c40 2021-02-10T11:31:23.281 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:23.276 7f6048ff9700 1 --2- 172.21.2.102:0/674748598 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f602c018730 0x7f602c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:23.281 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:23.276 7f6048ff9700 1 -- 172.21.2.102:0/674748598 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f6040032770 con 0x7f604c094c40 2021-02-10T11:31:23.281 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:23.276 7f604bfff700 1 --2- 172.21.2.102:0/674748598 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f602c018730 0x7f602c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:23.281 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:23.276 7f604bfff700 1 --2- 172.21.2.102:0/674748598 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f602c018730 0x7f602c01abc0 crc :-1 s=READY pgs=942 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:23.307 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:23.306 7f6052688700 1 -- 172.21.2.102:0/674748598 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f604c082980 con 0x7f604c094c40 2021-02-10T11:31:23.309 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:23.309 7f6048ff9700 1 -- 172.21.2.102:0/674748598 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f6040000dd0 con 0x7f604c094c40 2021-02-10T11:31:23.436 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:23.435 7f6052688700 1 -- 172.21.2.102:0/674748598 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000946240", "target": ["mgr", ""]}) v1 -- 0x7f604c188c90 con 0x7f602c018730 2021-02-10T11:31:23.438 INFO:tasks.ceph.mgr.y.gibba002.stderr:2021-02-10 11:31:23.438 7fd94bc53700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:31:23.438 INFO:tasks.ceph.mgr.y.gibba002.stderr:2021-02-10 11:31:23.438 7fd94bc53700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:31:23.439 INFO:tasks.ceph.mgr.y.gibba002.stderr:2021-02-10 11:31:23.438 7fd94bc53700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:31:23.439 INFO:tasks.ceph.mgr.y.gibba002.stderr:2021-02-10 11:31:23.438 7fd94bc53700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:31:23.439 INFO:tasks.ceph.mgr.y.gibba002.stderr:2021-02-10 11:31:23.438 7fd94bc53700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:31:23.470 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:23.469 7f6048ff9700 1 -- 172.21.2.102:0/674748598 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f603c000940 con 0x7f602c018730 2021-02-10T11:31:23.501 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:23.500 7f6052688700 1 -- 172.21.2.102:0/674748598 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f602c018730 msgr2=0x7f602c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:23.501 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:23.500 7f6052688700 1 --2- 172.21.2.102:0/674748598 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f602c018730 0x7f602c01abc0 crc :-1 s=READY pgs=942 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:23.502 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:23.500 7f6052688700 1 -- 172.21.2.102:0/674748598 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f604c094c40 msgr2=0x7f604c1a75e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:23.502 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:23.500 7f6052688700 1 --2- 172.21.2.102:0/674748598 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f604c094c40 0x7f604c1a75e0 secure :-1 s=READY pgs=2125 cs=0 l=1 rev1=1 rx=0x7f6040001c60 tx=0x7f604000a8d0).stop 2021-02-10T11:31:23.502 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:23.500 7f6052688700 1 -- 172.21.2.102:0/674748598 shutdown_connections 2021-02-10T11:31:23.502 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:23.500 7f6052688700 1 --2- 172.21.2.102:0/674748598 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f602c018730 0x7f602c01abc0 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:23.502 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:23.500 7f6052688700 1 --2- 172.21.2.102:0/674748598 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f604c095040 0x7f604c1a7c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:23.502 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:23.500 7f6052688700 1 --2- 172.21.2.102:0/674748598 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f604c094c40 0x7f604c1a75e0 unknown :-1 s=CLOSED pgs=2125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:23.503 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:23.500 7f6052688700 1 --2- 172.21.2.102:0/674748598 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f604c0a2580 0x7f604c0a1cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:23.503 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:23.500 7f6052688700 1 -- 172.21.2.102:0/674748598 >> 172.21.2.102:0/674748598 conn(0x7f604c0a2b30 msgr2=0x7f604c09ce60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:23.503 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:23.500 7f6052688700 1 -- 172.21.2.102:0/674748598 shutdown_connections 2021-02-10T11:31:23.503 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:23.500 7f6052688700 1 -- 172.21.2.102:0/674748598 wait complete. 2021-02-10T11:31:23.508 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000946240 2021-02-10T11:31:23.944 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:23.927 7fba56864700 1 Processor -- start 2021-02-10T11:31:23.944 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:23.927 7fba56864700 1 -- start start 2021-02-10T11:31:23.945 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:23.927 7fba56864700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fba500d1870 0x7fba501b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:23.945 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:23.927 7fba56864700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fba501b0ae0 0x7fba50108940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:23.945 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:23.927 7fba56864700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba500fe1c0 con 0x7fba500d1c70 2021-02-10T11:31:23.945 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:23.927 7fba56864700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba50109a60 con 0x7fba500d1870 2021-02-10T11:31:23.945 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:23.927 7fba56864700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba50109750 con 0x7fba501b0ae0 2021-02-10T11:31:23.945 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:23.927 7fba4ffff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fba500d1870 0x7fba501b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:23.945 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:23.927 7fba4ffff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fba500d1870 0x7fba501b04c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:33646/0 (socket says 172.21.2.102:33646) 2021-02-10T11:31:23.945 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:23.927 7fba4ffff700 1 -- 172.21.2.102:0/81125798 learned_addr learned my addr 172.21.2.102:0/81125798 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:31:23.945 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:23.928 7fba4ffff700 1 -- 172.21.2.102:0/81125798 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fba501b0ae0 msgr2=0x7fba50108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:23.945 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:23.928 7fba4ffff700 1 --2- 172.21.2.102:0/81125798 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fba501b0ae0 0x7fba50108940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:23.945 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:23.928 7fba4ffff700 1 -- 172.21.2.102:0/81125798 >> v1:172.21.2.113:6789/0 conn(0x7fba500d1c70 legacy=0x7fba50108f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:23.945 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:23.928 7fba4ffff700 1 -- 172.21.2.102:0/81125798 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba50088070 con 0x7fba500d1870 2021-02-10T11:31:23.946 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:23.928 7fba4ffff700 1 --2- 172.21.2.102:0/81125798 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fba500d1870 0x7fba501b04c0 secure :-1 s=READY pgs=2128 cs=0 l=1 rev1=1 rx=0x7fba40006700 tx=0x7fba40009b80).ready entity=mon.0 client_cookie=693fef68a3ff61e9 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:23.946 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:23.928 7fba4effd700 1 -- 172.21.2.102:0/81125798 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fba4000c0c0 con 0x7fba500d1870 2021-02-10T11:31:23.946 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:23.928 7fba4effd700 1 -- 172.21.2.102:0/81125798 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba4000c440 con 0x7fba500d1870 2021-02-10T11:31:23.946 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:23.928 7fba4effd700 1 -- 172.21.2.102:0/81125798 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fba4000c960 con 0x7fba500d1870 2021-02-10T11:31:23.946 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:23.928 7fba56864700 1 -- 172.21.2.102:0/81125798 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fba500d1870 msgr2=0x7fba501b04c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:23.946 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:23.928 7fba56864700 1 --2- 172.21.2.102:0/81125798 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fba500d1870 0x7fba501b04c0 secure :-1 s=READY pgs=2128 cs=0 l=1 rev1=1 rx=0x7fba40006700 tx=0x7fba40009b80).stop 2021-02-10T11:31:23.946 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:23.929 7fba56864700 1 -- 172.21.2.102:0/81125798 shutdown_connections 2021-02-10T11:31:23.946 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:23.929 7fba56864700 1 --2- 172.21.2.102:0/81125798 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fba501b0ae0 0x7fba50108940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:23.946 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:23.929 7fba56864700 1 --2- 172.21.2.102:0/81125798 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fba500d1870 0x7fba501b04c0 unknown :-1 s=CLOSED pgs=2128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:23.946 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:23.929 7fba56864700 1 -- 172.21.2.102:0/81125798 >> 172.21.2.102:0/81125798 conn(0x7fba500ce420 msgr2=0x7fba500d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:23.946 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:23.929 7fba56864700 1 -- 172.21.2.102:0/81125798 shutdown_connections 2021-02-10T11:31:23.946 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:23.929 7fba56864700 1 -- 172.21.2.102:0/81125798 wait complete. 2021-02-10T11:31:23.946 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:23.944 7fba56864700 1 Processor -- start 2021-02-10T11:31:23.947 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:23.944 7fba56864700 1 -- start start 2021-02-10T11:31:23.947 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:23.944 7fba56864700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fba500c3820 0x7fba50084cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:23.947 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:23.944 7fba56864700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fba500a0290 0x7fba500852d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:23.947 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:23.944 7fba56864700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fba50086270 0x7fba50086670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:23.947 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:23.944 7fba56864700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba500c0700 con 0x7fba50086270 2021-02-10T11:31:23.947 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:23.944 7fba56864700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba500858f0 con 0x7fba500c3820 2021-02-10T11:31:23.947 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:23.944 7fba56864700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba50085b10 con 0x7fba500a0290 2021-02-10T11:31:23.947 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:23.944 7fba54e81700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fba50086270 0x7fba50086670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:23.947 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:23.944 7fba54e81700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fba50086270 0x7fba50086670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:33654/0 (socket says 172.21.2.102:33654) 2021-02-10T11:31:23.947 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:23.944 7fba54e81700 1 -- 172.21.2.102:0/3477810045 learned_addr learned my addr 172.21.2.102:0/3477810045 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:31:23.948 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:23.945 7fba54e81700 1 -- 172.21.2.102:0/3477810045 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fba500a0290 msgr2=0x7fba500852d0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:31:23.948 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:23.945 7fba54e81700 1 --2- 172.21.2.102:0/3477810045 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fba500a0290 0x7fba500852d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:23.948 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:23.945 7fba54e81700 1 -- 172.21.2.102:0/3477810045 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fba500c3820 msgr2=0x7fba50084cb0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:31:23.948 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:23.945 7fba54e81700 1 --2- 172.21.2.102:0/3477810045 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fba500c3820 0x7fba50084cb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:23.948 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:23.945 7fba54e81700 1 -- 172.21.2.102:0/3477810045 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba50086c90 con 0x7fba50086270 2021-02-10T11:31:23.949 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:23.945 7fba54e81700 1 --2- 172.21.2.102:0/3477810045 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fba50086270 0x7fba50086670 secure :-1 s=READY pgs=2129 cs=0 l=1 rev1=1 rx=0x7fba48009940 tx=0x7fba4800bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:23.949 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:23.945 7fba4cff9700 1 -- 172.21.2.102:0/3477810045 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fba48000d30 con 0x7fba50086270 2021-02-10T11:31:23.949 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:23.945 7fba4cff9700 1 -- 172.21.2.102:0/3477810045 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba4800c2f0 con 0x7fba50086270 2021-02-10T11:31:23.949 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:23.945 7fba56864700 1 -- 172.21.2.102:0/3477810045 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba50086c90 con 0x7fba50086270 2021-02-10T11:31:23.949 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:23.945 7fba56864700 1 -- 172.21.2.102:0/3477810045 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba501a2f00 con 0x7fba50086270 2021-02-10T11:31:23.949 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:23.945 7fba4cff9700 1 -- 172.21.2.102:0/3477810045 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fba4800c7d0 con 0x7fba50086270 2021-02-10T11:31:23.949 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:23.946 7fba4cff9700 1 -- 172.21.2.102:0/3477810045 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fba48001220 con 0x7fba50086270 2021-02-10T11:31:23.949 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:23.946 7fba4cff9700 1 --2- 172.21.2.102:0/3477810045 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fba300186c0 0x7fba3001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:23.949 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:23.946 7fba4cff9700 1 -- 172.21.2.102:0/3477810045 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fba48032950 con 0x7fba50086270 2021-02-10T11:31:23.952 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:23.952 7fba4ffff700 1 --2- 172.21.2.102:0/3477810045 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fba300186c0 0x7fba3001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:23.952 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:23.952 7fba4ffff700 1 --2- 172.21.2.102:0/3477810045 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fba300186c0 0x7fba3001abc0 crc :-1 s=READY pgs=943 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:23.977 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:23.977 7fba56864700 1 -- 172.21.2.102:0/3477810045 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba50082a70 con 0x7fba50086270 2021-02-10T11:31:23.980 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:23.980 7fba4cff9700 1 -- 172.21.2.102:0/3477810045 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fba4802cfa0 con 0x7fba50086270 2021-02-10T11:31:24.110 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:24.110 7fba56864700 1 -- 172.21.2.102:0/3477810045 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000946240", "target": ["mgr", ""]}) v1 -- 0x7fba500c0700 con 0x7fba300186c0 2021-02-10T11:31:24.112 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:24.113 7fba4cff9700 1 -- 172.21.2.102:0/3477810045 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7fba40009100 con 0x7fba300186c0 2021-02-10T11:31:24.113 INFO:teuthology.orchestra.run.gibba002.stdout:/volumes/_nogroup/subvolume_0000000000946240/fa19afc2-e8e1-4fe9-b7dc-49ae048bfc6f 2021-02-10T11:31:24.115 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:24.115 7fba56864700 1 -- 172.21.2.102:0/3477810045 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fba300186c0 msgr2=0x7fba3001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:24.115 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:24.115 7fba56864700 1 --2- 172.21.2.102:0/3477810045 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fba300186c0 0x7fba3001abc0 crc :-1 s=READY pgs=943 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:24.115 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:24.115 7fba56864700 1 -- 172.21.2.102:0/3477810045 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fba50086270 msgr2=0x7fba50086670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:24.115 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:24.115 7fba56864700 1 --2- 172.21.2.102:0/3477810045 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fba50086270 0x7fba50086670 secure :-1 s=READY pgs=2129 cs=0 l=1 rev1=1 rx=0x7fba48009940 tx=0x7fba4800bcf0).stop 2021-02-10T11:31:24.115 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:24.115 7fba56864700 1 -- 172.21.2.102:0/3477810045 shutdown_connections 2021-02-10T11:31:24.115 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:24.115 7fba56864700 1 --2- 172.21.2.102:0/3477810045 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fba50086270 0x7fba50086670 unknown :-1 s=CLOSED pgs=2129 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:24.115 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:24.115 7fba56864700 1 --2- 172.21.2.102:0/3477810045 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fba500a0290 0x7fba500852d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:24.115 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:24.115 7fba56864700 1 --2- 172.21.2.102:0/3477810045 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fba300186c0 0x7fba3001abc0 unknown :-1 s=CLOSED pgs=943 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:24.115 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:24.115 7fba56864700 1 --2- 172.21.2.102:0/3477810045 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fba500c3820 0x7fba50084cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:24.115 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:24.115 7fba56864700 1 -- 172.21.2.102:0/3477810045 >> 172.21.2.102:0/3477810045 conn(0x7fba500872c0 msgr2=0x7fba50095700 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:24.116 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:24.115 7fba56864700 1 -- 172.21.2.102:0/3477810045 shutdown_connections 2021-02-10T11:31:24.116 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:24.115 7fba56864700 1 -- 172.21.2.102:0/3477810045 wait complete. 2021-02-10T11:31:24.124 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2021-02-10T11:31:24.124 DEBUG:teuthology.orchestra.run.gibba002:> import os 2021-02-10T11:31:24.124 DEBUG:teuthology.orchestra.run.gibba002:> import stat 2021-02-10T11:31:24.124 DEBUG:teuthology.orchestra.run.gibba002:> import json 2021-02-10T11:31:24.124 DEBUG:teuthology.orchestra.run.gibba002:> import sys 2021-02-10T11:31:24.124 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:31:24.124 DEBUG:teuthology.orchestra.run.gibba002:> try: 2021-02-10T11:31:24.124 DEBUG:teuthology.orchestra.run.gibba002:> s = os.stat("/home/ubuntu/cephtest/mnt.0/volumes/_nogroup/subvolume_0000000000946240/fa19afc2-e8e1-4fe9-b7dc-49ae048bfc6f") 2021-02-10T11:31:24.124 DEBUG:teuthology.orchestra.run.gibba002:> except OSError as e: 2021-02-10T11:31:24.124 DEBUG:teuthology.orchestra.run.gibba002:> sys.exit(e.errno) 2021-02-10T11:31:24.124 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:31:24.125 DEBUG:teuthology.orchestra.run.gibba002:> attrs = ["st_mode", "st_ino", "st_dev", "st_nlink", "st_uid", "st_gid", "st_size", "st_atime", "st_mtime", "st_ctime"] 2021-02-10T11:31:24.125 DEBUG:teuthology.orchestra.run.gibba002:> print(json.dumps( 2021-02-10T11:31:24.125 DEBUG:teuthology.orchestra.run.gibba002:> dict([(a, getattr(s, a)) for a in attrs]), 2021-02-10T11:31:24.125 DEBUG:teuthology.orchestra.run.gibba002:> indent=2)) 2021-02-10T11:31:24.125 DEBUG:teuthology.orchestra.run.gibba002:> ' 2021-02-10T11:31:24.546 INFO:teuthology.orchestra.run.gibba002.stdout:{ 2021-02-10T11:31:24.546 INFO:teuthology.orchestra.run.gibba002.stdout: "st_mode": 16877, 2021-02-10T11:31:24.546 INFO:teuthology.orchestra.run.gibba002.stdout: "st_ino": 1099511627779, 2021-02-10T11:31:24.546 INFO:teuthology.orchestra.run.gibba002.stdout: "st_dev": 43, 2021-02-10T11:31:24.546 INFO:teuthology.orchestra.run.gibba002.stdout: "st_nlink": 2, 2021-02-10T11:31:24.546 INFO:teuthology.orchestra.run.gibba002.stdout: "st_uid": 0, 2021-02-10T11:31:24.546 INFO:teuthology.orchestra.run.gibba002.stdout: "st_gid": 0, 2021-02-10T11:31:24.546 INFO:teuthology.orchestra.run.gibba002.stdout: "st_size": 0, 2021-02-10T11:31:24.546 INFO:teuthology.orchestra.run.gibba002.stdout: "st_atime": 1612956683.4505992, 2021-02-10T11:31:24.547 INFO:teuthology.orchestra.run.gibba002.stdout: "st_mtime": 1612956683.4505992, 2021-02-10T11:31:24.547 INFO:teuthology.orchestra.run.gibba002.stdout: "st_ctime": 1612956683.45969 2021-02-10T11:31:24.547 INFO:teuthology.orchestra.run.gibba002.stdout:} 2021-02-10T11:31:24.710 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000946240 2021-02-10T11:31:25.183 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:25.165 7fc769af7700 1 Processor -- start 2021-02-10T11:31:25.184 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:25.165 7fc769af7700 1 -- start start 2021-02-10T11:31:25.184 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:25.166 7fc769af7700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc7640d1860 0x7fc7641b04b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:25.184 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:25.166 7fc769af7700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc764109a50 con 0x7fc7641b0ad0 2021-02-10T11:31:25.184 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:25.166 7fc769af7700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc764088070 con 0x7fc7640cff60 2021-02-10T11:31:25.184 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:25.166 7fc769af7700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7640e5ee0 con 0x7fc7640d1860 2021-02-10T11:31:25.184 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:25.166 7fc762ffd700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fc7641b0ad0 0x7fc764108930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:54116/0 (socket says 172.21.2.102:54116) 2021-02-10T11:31:25.184 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:25.166 7fc762ffd700 1 -- 172.21.2.102:0/4245960732 learned_addr learned my addr 172.21.2.102:0/4245960732 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:31:25.184 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:25.166 7fc7637fe700 1 --2- 172.21.2.102:0/4245960732 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc7640d1860 0x7fc7641b04b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:25.184 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:25.166 7fc7627fc700 1 -- 172.21.2.102:0/4245960732 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1705062098 0 0) 0x7fc7580010b0 con 0x7fc7641b0ad0 2021-02-10T11:31:25.184 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:25.166 7fc7637fe700 1 -- 172.21.2.102:0/4245960732 >> v1:172.21.2.113:6789/0 conn(0x7fc7640cff60 legacy=0x7fc764109120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:25.184 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:25.167 7fc7637fe700 1 -- 172.21.2.102:0/4245960732 >> v1:172.21.2.102:6789/0 conn(0x7fc7641b0ad0 legacy=0x7fc764108930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:25.185 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:25.167 7fc7637fe700 1 -- 172.21.2.102:0/4245960732 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7640d1c60 con 0x7fc7640d1860 2021-02-10T11:31:25.185 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:25.167 7fc7637fe700 1 --2- 172.21.2.102:0/4245960732 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc7640d1860 0x7fc7641b04b0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7fc75c0069d0 tx=0x7fc75c009e40).ready entity=mon.2 client_cookie=52065dd777ddc187 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:25.185 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:25.167 7fc7627fc700 1 -- 172.21.2.102:0/4245960732 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc75c00b510 con 0x7fc7640d1860 2021-02-10T11:31:25.185 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:25.167 7fc7627fc700 1 -- 172.21.2.102:0/4245960732 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc75c00b890 con 0x7fc7640d1860 2021-02-10T11:31:25.185 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:25.167 7fc7627fc700 1 -- 172.21.2.102:0/4245960732 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc75c00bd70 con 0x7fc7640d1860 2021-02-10T11:31:25.185 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:25.167 7fc769af7700 1 -- 172.21.2.102:0/4245960732 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc7640d1860 msgr2=0x7fc7641b04b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:25.185 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:25.167 7fc769af7700 1 --2- 172.21.2.102:0/4245960732 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc7640d1860 0x7fc7641b04b0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7fc75c0069d0 tx=0x7fc75c009e40).stop 2021-02-10T11:31:25.185 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:25.168 7fc769af7700 1 -- 172.21.2.102:0/4245960732 shutdown_connections 2021-02-10T11:31:25.185 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:25.168 7fc769af7700 1 --2- 172.21.2.102:0/4245960732 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc7640d1860 0x7fc7641b04b0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:25.185 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:25.168 7fc769af7700 1 -- 172.21.2.102:0/4245960732 >> 172.21.2.102:0/4245960732 conn(0x7fc7640ce410 msgr2=0x7fc7640d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:25.185 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:25.168 7fc769af7700 1 -- 172.21.2.102:0/4245960732 shutdown_connections 2021-02-10T11:31:25.185 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:25.168 7fc769af7700 1 -- 172.21.2.102:0/4245960732 wait complete. 2021-02-10T11:31:25.185 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:25.182 7fc769af7700 1 Processor -- start 2021-02-10T11:31:25.186 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:25.183 7fc769af7700 1 -- start start 2021-02-10T11:31:25.186 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:25.183 7fc769af7700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc7640bfa10 0x7fc7640b82a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:25.186 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:25.183 7fc769af7700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc7641a4a70 0x7fc764094c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:25.186 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:25.183 7fc769af7700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc7640c0410 0x7fc7640c0810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:25.186 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:25.183 7fc769af7700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7640c3a50 con 0x7fc7641a4a70 2021-02-10T11:31:25.186 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:25.183 7fc769af7700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7640b88c0 con 0x7fc7640bfa10 2021-02-10T11:31:25.186 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:25.183 7fc769af7700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc764095260 con 0x7fc7640c0410 2021-02-10T11:31:25.186 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:25.183 7fc762ffd700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc7641a4a70 0x7fc764094c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:25.186 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:25.183 7fc762ffd700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc7641a4a70 0x7fc764094c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:33666/0 (socket says 172.21.2.102:33666) 2021-02-10T11:31:25.186 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:25.183 7fc762ffd700 1 -- 172.21.2.102:0/4049030070 learned_addr learned my addr 172.21.2.102:0/4049030070 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:31:25.186 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:25.183 7fc762ffd700 1 -- 172.21.2.102:0/4049030070 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc7640c0410 msgr2=0x7fc7640c0810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:25.186 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:25.183 7fc762ffd700 1 --2- 172.21.2.102:0/4049030070 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc7640c0410 0x7fc7640c0810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:25.186 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:25.183 7fc762ffd700 1 -- 172.21.2.102:0/4049030070 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc7640bfa10 msgr2=0x7fc7640b82a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:25.187 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:25.183 7fc762ffd700 1 --2- 172.21.2.102:0/4049030070 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc7640bfa10 0x7fc7640b82a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:25.187 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:25.183 7fc762ffd700 1 -- 172.21.2.102:0/4049030070 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7640c3540 con 0x7fc7641a4a70 2021-02-10T11:31:25.187 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:25.184 7fc762ffd700 1 --2- 172.21.2.102:0/4049030070 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc7641a4a70 0x7fc764094c40 secure :-1 s=READY pgs=2131 cs=0 l=1 rev1=1 rx=0x7fc758001c10 tx=0x7fc75800a8f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:25.187 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:25.184 7fc74bfff700 1 -- 172.21.2.102:0/4049030070 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc7580009b0 con 0x7fc7641a4a70 2021-02-10T11:31:25.188 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:25.184 7fc74bfff700 1 -- 172.21.2.102:0/4049030070 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7580012c0 con 0x7fc7641a4a70 2021-02-10T11:31:25.188 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:25.184 7fc74bfff700 1 -- 172.21.2.102:0/4049030070 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc758004350 con 0x7fc7641a4a70 2021-02-10T11:31:25.188 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:25.184 7fc769af7700 1 -- 172.21.2.102:0/4049030070 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7640c3540 con 0x7fc7641a4a70 2021-02-10T11:31:25.188 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:25.184 7fc769af7700 1 -- 172.21.2.102:0/4049030070 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7640c1120 con 0x7fc7641a4a70 2021-02-10T11:31:25.188 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:25.184 7fc74bfff700 1 -- 172.21.2.102:0/4049030070 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fc758004140 con 0x7fc7641a4a70 2021-02-10T11:31:25.188 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:25.184 7fc74bfff700 1 --2- 172.21.2.102:0/4049030070 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc7440186c0 0x7fc74401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:25.188 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:25.184 7fc74bfff700 1 -- 172.21.2.102:0/4049030070 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fc758032500 con 0x7fc7641a4a70 2021-02-10T11:31:25.188 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:25.185 7fc7637fe700 1 --2- 172.21.2.102:0/4049030070 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc7440186c0 0x7fc74401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:25.188 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:25.185 7fc7637fe700 1 --2- 172.21.2.102:0/4049030070 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc7440186c0 0x7fc74401abc0 crc :-1 s=READY pgs=944 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:25.216 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:25.216 7fc769af7700 1 -- 172.21.2.102:0/4049030070 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7640bfee0 con 0x7fc7641a4a70 2021-02-10T11:31:25.218 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:25.218 7fc74bfff700 1 -- 172.21.2.102:0/4049030070 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fc7580009b0 con 0x7fc7641a4a70 2021-02-10T11:31:25.346 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:25.346 7fc769af7700 1 -- 172.21.2.102:0/4049030070 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000946240", "target": ["mgr", ""]}) v1 -- 0x7fc7640c1120 con 0x7fc7440186c0 2021-02-10T11:31:25.357 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:25.357 7fc74bfff700 1 -- 172.21.2.102:0/4049030070 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fc75c000cc0 con 0x7fc7440186c0 2021-02-10T11:31:25.363 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:25.363 7fc769af7700 1 -- 172.21.2.102:0/4049030070 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc7440186c0 msgr2=0x7fc74401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:25.364 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:25.363 7fc769af7700 1 --2- 172.21.2.102:0/4049030070 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc7440186c0 0x7fc74401abc0 crc :-1 s=READY pgs=944 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:25.364 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:25.363 7fc769af7700 1 -- 172.21.2.102:0/4049030070 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc7641a4a70 msgr2=0x7fc764094c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:25.364 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:25.363 7fc769af7700 1 --2- 172.21.2.102:0/4049030070 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc7641a4a70 0x7fc764094c40 secure :-1 s=READY pgs=2131 cs=0 l=1 rev1=1 rx=0x7fc758001c10 tx=0x7fc75800a8f0).stop 2021-02-10T11:31:25.364 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:25.363 7fc769af7700 1 -- 172.21.2.102:0/4049030070 shutdown_connections 2021-02-10T11:31:25.364 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:25.363 7fc769af7700 1 --2- 172.21.2.102:0/4049030070 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc7640c0410 0x7fc7640c0810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:25.364 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:25.363 7fc769af7700 1 --2- 172.21.2.102:0/4049030070 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc7641a4a70 0x7fc764094c40 unknown :-1 s=CLOSED pgs=2131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:25.364 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:25.363 7fc769af7700 1 --2- 172.21.2.102:0/4049030070 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc7440186c0 0x7fc74401abc0 unknown :-1 s=CLOSED pgs=944 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:25.364 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:25.363 7fc769af7700 1 --2- 172.21.2.102:0/4049030070 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc7640bfa10 0x7fc7640b82a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:25.364 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:25.363 7fc769af7700 1 -- 172.21.2.102:0/4049030070 >> 172.21.2.102:0/4049030070 conn(0x7fc7640a6a80 msgr2=0x7fc7640ae110 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:25.364 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:25.363 7fc769af7700 1 -- 172.21.2.102:0/4049030070 shutdown_connections 2021-02-10T11:31:25.364 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:25.363 7fc769af7700 1 -- 172.21.2.102:0/4049030070 wait complete. 2021-02-10T11:31:25.372 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo getfattr --only-values -n ceph.dir.entries /home/ubuntu/cephtest/mnt.0/./volumes/_deleting 2021-02-10T11:31:25.721 INFO:teuthology.orchestra.run.gibba002.stderr:getfattr: Removing leading '/' from absolute path names 2021-02-10T11:31:25.722 INFO:teuthology.orchestra.run.gibba002.stdout:0 2021-02-10T11:31:25.722 DEBUG:tasks.cephfs.mount:Directory ./volumes/_deleting seen empty from 0 after 0s 2021-02-10T11:31:25.722 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2021-02-10T11:31:25.722 DEBUG:teuthology.orchestra.run.gibba002:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:31:26.070 DEBUG:teuthology.orchestra.run.gibba013:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:31:26.414 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2021-02-10T11:31:26.414 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:31:26.414 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:31:26.414 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:31:26.751 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:31:26.751 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:31:26.751 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@gibba002.front.sepia.ceph.com... 2021-02-10T11:31:26.752 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-02-10T11:31:26.752 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:31:27.109 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba002.stderr:ceph-fuse[178321]: fuse finished with error 0 and tester_r 0 2021-02-10T11:31:27.111 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:31:27.111 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:31:27.111 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:31:27.446 INFO:teuthology.orchestra.run.gibba002.stdout:ext2/ext3 2021-02-10T11:31:27.446 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2021-02-10T11:31:27.446 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:31:27.447 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:31:27.783 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:31:27.783 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:31:27.783 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:31:28.119 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:31:28.119 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2021-02-10T11:31:28.119 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@gibba002.front.sepia.ceph.com... 2021-02-10T11:31:28.119 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-02-10T11:31:28.120 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:31:28.480 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba002.stderr:ceph-fuse[178961]: fuse finished with error 0 and tester_r 0 2021-02-10T11:31:28.481 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:31:28.481 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:31:28.481 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:31:28.816 INFO:teuthology.orchestra.run.gibba002.stdout:ext2/ext3 2021-02-10T11:31:28.816 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2021-02-10T11:31:28.817 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:31:28.817 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:31:29.153 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:31:29.153 DEBUG:teuthology.orchestra.run.gibba013:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:31:29.153 DEBUG:teuthology.orchestra.run.gibba013:> ' -- /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:31:29.488 INFO:teuthology.orchestra.run.gibba013.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.2’: No such file or directory 2021-02-10T11:31:29.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:31:29.488 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2021-02-10T11:31:29.488 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:31:29.489 DEBUG:teuthology.orchestra.run.gibba013:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:31:29.823 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:31:29.823 DEBUG:teuthology.orchestra.run.gibba013:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:31:29.823 DEBUG:teuthology.orchestra.run.gibba013:> ' -- /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:31:30.159 INFO:teuthology.orchestra.run.gibba013.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.3’: No such file or directory 2021-02-10T11:31:30.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:31:30.159 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.3 2021-02-10T11:31:30.160 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:31:30.160 DEBUG:teuthology.orchestra.run.gibba013:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:31:30.494 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config rm mon mon_allow_pool_delete 2021-02-10T11:31:30.925 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:30.908 7efc9237f700 1 Processor -- start 2021-02-10T11:31:30.926 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:30.908 7efc9237f700 1 -- start start 2021-02-10T11:31:30.926 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:30.908 7efc9237f700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7efc8c0d1870 0x7efc8c1b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:30.926 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:30.908 7efc9237f700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7efc8c1b0a10 0x7efc8c0cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:30.926 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:30.908 7efc9237f700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc8c0f0b10 con 0x7efc8c0d1c70 2021-02-10T11:31:30.926 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:30.908 7efc9237f700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc8c0e5ed0 con 0x7efc8c1b0a10 2021-02-10T11:31:30.926 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:30.908 7efc9237f700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc8c0f0650 con 0x7efc8c0d1870 2021-02-10T11:31:30.926 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:30.908 7efc8b7fe700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7efc8c1b0a10 0x7efc8c0cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:30.926 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:30.908 7efc9099c700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7efc8c0d1c70 0x7efc8c0cf950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:54126/0 (socket says 172.21.2.102:54126) 2021-02-10T11:31:30.926 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:30.908 7efc9099c700 1 -- 172.21.2.102:0/2404036093 learned_addr learned my addr 172.21.2.102:0/2404036093 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:31:30.926 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:30.909 7efc8b7fe700 1 -- 172.21.2.102:0/2404036093 >> v2:172.21.2.113:3300/0 conn(0x7efc8c0d1870 msgr2=0x7efc8c1b03f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:31:30.927 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:30.909 7efc8b7fe700 1 --2- 172.21.2.102:0/2404036093 >> v2:172.21.2.113:3300/0 conn(0x7efc8c0d1870 0x7efc8c1b03f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:30.927 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:30.909 7efc8b7fe700 1 -- 172.21.2.102:0/2404036093 >> v1:172.21.2.102:6789/0 conn(0x7efc8c0d1c70 legacy=0x7efc8c0cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:30.927 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:30.909 7efc8b7fe700 1 -- 172.21.2.102:0/2404036093 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc8c0f0840 con 0x7efc8c1b0a10 2021-02-10T11:31:30.927 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:30.909 7efc8b7fe700 1 --2- 172.21.2.102:0/2404036093 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efc8c1b0a10 0x7efc8c0cf330 secure :-1 s=READY pgs=2133 cs=0 l=1 rev1=1 rx=0x7efc80006650 tx=0x7efc80009a80).ready entity=mon.0 client_cookie=26b67e4893604577 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:30.927 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:30.909 7efc8affd700 1 -- 172.21.2.102:0/2404036093 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efc800009b0 con 0x7efc8c1b0a10 2021-02-10T11:31:30.927 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:30.909 7efc8affd700 1 -- 172.21.2.102:0/2404036093 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc8000c0c0 con 0x7efc8c1b0a10 2021-02-10T11:31:30.927 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:30.909 7efc8affd700 1 -- 172.21.2.102:0/2404036093 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efc8000c5e0 con 0x7efc8c1b0a10 2021-02-10T11:31:30.927 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:30.909 7efc9237f700 1 -- 172.21.2.102:0/2404036093 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efc8c1b0a10 msgr2=0x7efc8c0cf330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:30.927 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:30.909 7efc9237f700 1 --2- 172.21.2.102:0/2404036093 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efc8c1b0a10 0x7efc8c0cf330 secure :-1 s=READY pgs=2133 cs=0 l=1 rev1=1 rx=0x7efc80006650 tx=0x7efc80009a80).stop 2021-02-10T11:31:30.927 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:30.910 7efc9237f700 1 -- 172.21.2.102:0/2404036093 shutdown_connections 2021-02-10T11:31:30.927 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:30.910 7efc9237f700 1 --2- 172.21.2.102:0/2404036093 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efc8c1b0a10 0x7efc8c0cf330 unknown :-1 s=CLOSED pgs=2133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:30.928 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:30.910 7efc9237f700 1 --2- 172.21.2.102:0/2404036093 >> v2:172.21.2.113:3300/0 conn(0x7efc8c0d1870 0x7efc8c1b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:30.928 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:30.910 7efc9237f700 1 -- 172.21.2.102:0/2404036093 >> 172.21.2.102:0/2404036093 conn(0x7efc8c0ce450 msgr2=0x7efc8c0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:30.928 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:30.910 7efc9237f700 1 -- 172.21.2.102:0/2404036093 shutdown_connections 2021-02-10T11:31:30.928 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:30.910 7efc9237f700 1 -- 172.21.2.102:0/2404036093 wait complete. 2021-02-10T11:31:30.928 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:30.924 7efc9237f700 1 Processor -- start 2021-02-10T11:31:30.928 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:30.925 7efc9237f700 1 -- start start 2021-02-10T11:31:30.929 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:30.925 7efc9237f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efc8c0a2fb0 0x7efc8c0a33b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:30.930 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:30.925 7efc9237f700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7efc8c0a39d0 0x7efc8c0a4860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:30.930 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:30.925 7efc9237f700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7efc8c0a3dd0 0x7efc8c0a4e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:30.930 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:30.925 7efc9237f700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc8c0a2620 con 0x7efc8c0a2fb0 2021-02-10T11:31:30.930 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:30.925 7efc9237f700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc8c0a2840 con 0x7efc8c0a3dd0 2021-02-10T11:31:30.930 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:30.925 7efc9237f700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc8c0a2b80 con 0x7efc8c0a39d0 2021-02-10T11:31:30.930 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:30.925 7efc8bfff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efc8c0a2fb0 0x7efc8c0a33b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:30.930 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:30.925 7efc8bfff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efc8c0a2fb0 0x7efc8c0a33b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:33676/0 (socket says 172.21.2.102:33676) 2021-02-10T11:31:30.930 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:30.925 7efc8bfff700 1 -- 172.21.2.102:0/613480318 learned_addr learned my addr 172.21.2.102:0/613480318 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:31:30.930 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:30.925 7efc8bfff700 1 -- 172.21.2.102:0/613480318 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7efc8c0a39d0 msgr2=0x7efc8c0a4860 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:31:30.930 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:30.925 7efc8bfff700 1 --2- 172.21.2.102:0/613480318 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7efc8c0a39d0 0x7efc8c0a4860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:30.930 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:30.925 7efc8bfff700 1 -- 172.21.2.102:0/613480318 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7efc8c0a3dd0 msgr2=0x7efc8c0a4e80 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:31:30.931 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:30.925 7efc8bfff700 1 --2- 172.21.2.102:0/613480318 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7efc8c0a3dd0 0x7efc8c0a4e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:30.931 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:30.925 7efc8bfff700 1 -- 172.21.2.102:0/613480318 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc8c0a54a0 con 0x7efc8c0a2fb0 2021-02-10T11:31:30.931 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:30.926 7efc8bfff700 1 --2- 172.21.2.102:0/613480318 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efc8c0a2fb0 0x7efc8c0a33b0 secure :-1 s=READY pgs=2134 cs=0 l=1 rev1=1 rx=0x7efc7c00b5c0 tx=0x7efc7c00b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:30.931 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:30.926 7efc88ff9700 1 -- 172.21.2.102:0/613480318 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efc7c00bd50 con 0x7efc8c0a2fb0 2021-02-10T11:31:30.931 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:30.926 7efc88ff9700 1 -- 172.21.2.102:0/613480318 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc7c006030 con 0x7efc8c0a2fb0 2021-02-10T11:31:30.931 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:30.926 7efc88ff9700 1 -- 172.21.2.102:0/613480318 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efc7c0054d0 con 0x7efc8c0a2fb0 2021-02-10T11:31:30.931 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:30.926 7efc9237f700 1 -- 172.21.2.102:0/613480318 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc8c0a54a0 con 0x7efc8c0a2fb0 2021-02-10T11:31:30.931 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:30.926 7efc9237f700 1 -- 172.21.2.102:0/613480318 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc8c0a54a0 con 0x7efc8c0a2fb0 2021-02-10T11:31:30.931 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:30.926 7efc88ff9700 1 -- 172.21.2.102:0/613480318 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7efc7c006210 con 0x7efc8c0a2fb0 2021-02-10T11:31:30.931 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:30.926 7efc88ff9700 1 --2- 172.21.2.102:0/613480318 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7efc6c0186d0 0x7efc6c01abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:30.931 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:30.926 7efc88ff9700 1 -- 172.21.2.102:0/613480318 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 5573+0+0 (secure 0 0 0) 0x7efc7c032650 con 0x7efc8c0a2fb0 2021-02-10T11:31:30.931 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:30.926 7efc8b7fe700 1 --2- 172.21.2.102:0/613480318 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7efc6c0186d0 0x7efc6c01abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:30.932 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:30.927 7efc8b7fe700 1 --2- 172.21.2.102:0/613480318 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7efc6c0186d0 0x7efc6c01abd0 crc :-1 s=READY pgs=945 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:30.959 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:30.958 7efc9237f700 1 -- 172.21.2.102:0/613480318 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc8c0ee060 con 0x7efc8c0a2fb0 2021-02-10T11:31:30.962 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:30.961 7efc88ff9700 1 -- 172.21.2.102:0/613480318 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7efc7c006b80 con 0x7efc8c0a2fb0 2021-02-10T11:31:31.111 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:31.110 7efc9237f700 1 -- 172.21.2.102:0/613480318 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7efc8c0ce450 con 0x7efc8c0a2fb0 2021-02-10T11:31:31.115 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:31.114 7efc88ff9700 1 -- 172.21.2.102:0/613480318 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v27) v1 ==== 104+0+0 (secure 0 0 0) 0x7efc7c006030 con 0x7efc8c0a2fb0 2021-02-10T11:31:31.119 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:31.118 7efc9237f700 1 -- 172.21.2.102:0/613480318 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7efc6c0186d0 msgr2=0x7efc6c01abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:31.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:31.118 7efc9237f700 1 --2- 172.21.2.102:0/613480318 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7efc6c0186d0 0x7efc6c01abd0 crc :-1 s=READY pgs=945 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:31.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:31.118 7efc9237f700 1 -- 172.21.2.102:0/613480318 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efc8c0a2fb0 msgr2=0x7efc8c0a33b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:31.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:31.118 7efc9237f700 1 --2- 172.21.2.102:0/613480318 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efc8c0a2fb0 0x7efc8c0a33b0 secure :-1 s=READY pgs=2134 cs=0 l=1 rev1=1 rx=0x7efc7c00b5c0 tx=0x7efc7c00b960).stop 2021-02-10T11:31:31.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:31.118 7efc9237f700 1 -- 172.21.2.102:0/613480318 shutdown_connections 2021-02-10T11:31:31.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:31.118 7efc9237f700 1 --2- 172.21.2.102:0/613480318 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7efc6c0186d0 0x7efc6c01abd0 unknown :-1 s=CLOSED pgs=945 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:31.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:31.118 7efc9237f700 1 --2- 172.21.2.102:0/613480318 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7efc8c0a3dd0 0x7efc8c0a4e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:31.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:31.118 7efc9237f700 1 --2- 172.21.2.102:0/613480318 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7efc8c0a39d0 0x7efc8c0a4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:31.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:31.118 7efc9237f700 1 --2- 172.21.2.102:0/613480318 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efc8c0a2fb0 0x7efc8c0a33b0 unknown :-1 s=CLOSED pgs=2134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:31.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:31.118 7efc9237f700 1 -- 172.21.2.102:0/613480318 >> 172.21.2.102:0/613480318 conn(0x7efc8c08f030 msgr2=0x7efc8c0b5e80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:31.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:31.119 7efc9237f700 1 -- 172.21.2.102:0/613480318 shutdown_connections 2021-02-10T11:31:31.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:31.119 7efc9237f700 1 -- 172.21.2.102:0/613480318 wait complete. 2021-02-10T11:31:31.128 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestVolumes.test_default_uid_gid_subvolume' 2021-02-10T11:31:31.558 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:31.540 7f8dbec85700 1 Processor -- start 2021-02-10T11:31:31.558 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:31.540 7f8dbec85700 1 -- start start 2021-02-10T11:31:31.558 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:31.540 7f8dbec85700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f8db80d1870 0x7f8db81b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:31.558 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:31.540 7f8dbec85700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f8db81b0a10 0x7f8db80cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:31.558 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:31.540 7f8dbec85700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8db80d1c70 0x7f8db80cf950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:31.558 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:31.540 7f8dbec85700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8db80e5ed0 con 0x7f8db80d1870 2021-02-10T11:31:31.558 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:31.540 7f8dbec85700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8db80f0b10 con 0x7f8db81b0a10 2021-02-10T11:31:31.558 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:31.540 7f8dbec85700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8db80cff70 con 0x7f8db80d1c70 2021-02-10T11:31:31.559 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:31.540 7f8dbcaa1700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f8db80d1870 0x7f8db81b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:31.559 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:31.540 7f8dbcaa1700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f8db80d1870 0x7f8db81b03f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:33682/0 (socket says 172.21.2.102:33682) 2021-02-10T11:31:31.559 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:31.540 7f8dbcaa1700 1 -- 172.21.2.102:0/1555588894 learned_addr learned my addr 172.21.2.102:0/1555588894 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:31:31.559 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:31.540 7f8db7fff700 1 --2- 172.21.2.102:0/1555588894 >> v2:172.21.2.113:3300/0 conn(0x7f8db81b0a10 0x7f8db80cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:31.559 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:31.540 7f8dbd2a2700 1 --2- 172.21.2.102:0/1555588894 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8db80d1c70 0x7f8db80cf950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:31.559 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:31.541 7f8dbcaa1700 1 -- 172.21.2.102:0/1555588894 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8db80d1c70 msgr2=0x7f8db80cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:31.559 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:31.541 7f8dbcaa1700 1 --2- 172.21.2.102:0/1555588894 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8db80d1c70 0x7f8db80cf950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:31.559 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:31.541 7f8dbcaa1700 1 -- 172.21.2.102:0/1555588894 >> v2:172.21.2.113:3300/0 conn(0x7f8db81b0a10 msgr2=0x7f8db80cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:31.559 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:31.541 7f8dbcaa1700 1 --2- 172.21.2.102:0/1555588894 >> v2:172.21.2.113:3300/0 conn(0x7f8db81b0a10 0x7f8db80cf330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:31.560 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:31.541 7f8dbcaa1700 1 -- 172.21.2.102:0/1555588894 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8db80d0150 con 0x7f8db80d1870 2021-02-10T11:31:31.560 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:31.541 7f8dbcaa1700 1 --2- 172.21.2.102:0/1555588894 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8db80d1870 0x7f8db81b03f0 secure :-1 s=READY pgs=2135 cs=0 l=1 rev1=1 rx=0x7f8da8006700 tx=0x7f8da8009b80).ready entity=mon.0 client_cookie=7c5b32d69d627ef4 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:31.560 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:31.541 7f8db77fe700 1 -- 172.21.2.102:0/1555588894 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8da800c0c0 con 0x7f8db80d1870 2021-02-10T11:31:31.560 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:31.541 7f8db77fe700 1 -- 172.21.2.102:0/1555588894 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8da800c440 con 0x7f8db80d1870 2021-02-10T11:31:31.560 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:31.541 7f8db77fe700 1 -- 172.21.2.102:0/1555588894 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8da800c920 con 0x7f8db80d1870 2021-02-10T11:31:31.560 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:31.541 7f8dbec85700 1 -- 172.21.2.102:0/1555588894 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8db80d1870 msgr2=0x7f8db81b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:31.560 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:31.541 7f8dbec85700 1 --2- 172.21.2.102:0/1555588894 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8db80d1870 0x7f8db81b03f0 secure :-1 s=READY pgs=2135 cs=0 l=1 rev1=1 rx=0x7f8da8006700 tx=0x7f8da8009b80).stop 2021-02-10T11:31:31.560 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:31.542 7f8dbec85700 1 -- 172.21.2.102:0/1555588894 shutdown_connections 2021-02-10T11:31:31.560 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:31.542 7f8dbec85700 1 --2- 172.21.2.102:0/1555588894 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8db80d1c70 0x7f8db80cf950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:31.561 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:31.542 7f8dbec85700 1 --2- 172.21.2.102:0/1555588894 >> v2:172.21.2.113:3300/0 conn(0x7f8db81b0a10 0x7f8db80cf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:31.561 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:31.542 7f8dbec85700 1 --2- 172.21.2.102:0/1555588894 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8db80d1870 0x7f8db81b03f0 unknown :-1 s=CLOSED pgs=2135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:31.561 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:31.542 7f8dbec85700 1 -- 172.21.2.102:0/1555588894 >> 172.21.2.102:0/1555588894 conn(0x7f8db80ce450 msgr2=0x7f8db80d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:31.561 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:31.542 7f8dbec85700 1 -- 172.21.2.102:0/1555588894 shutdown_connections 2021-02-10T11:31:31.561 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:31.542 7f8dbec85700 1 -- 172.21.2.102:0/1555588894 wait complete. 2021-02-10T11:31:31.561 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:31.556 7f8dbec85700 1 Processor -- start 2021-02-10T11:31:31.561 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:31.557 7f8dbec85700 1 -- start start 2021-02-10T11:31:31.562 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:31.557 7f8dbec85700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8db80ada90 0x7f8db80ade90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:31.563 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:31.557 7f8dbec85700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8db8099d20 0x7f8db809a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:31.563 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:31.557 7f8dbec85700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8db809a740 0x7f8db8097bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:31.563 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:31.557 7f8dbec85700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8db80997a0 con 0x7f8db80ada90 2021-02-10T11:31:31.563 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:31.557 7f8dbec85700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8db818f0c0 con 0x7f8db8099d20 2021-02-10T11:31:31.563 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:31.557 7f8dbec85700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8db809ab40 con 0x7f8db809a740 2021-02-10T11:31:31.563 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:31.557 7f8dbcaa1700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8db80ada90 0x7f8db80ade90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:31.563 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:31.557 7f8dbcaa1700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8db80ada90 0x7f8db80ade90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:33688/0 (socket says 172.21.2.102:33688) 2021-02-10T11:31:31.563 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:31.557 7f8dbcaa1700 1 -- 172.21.2.102:0/1879930559 learned_addr learned my addr 172.21.2.102:0/1879930559 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:31:31.563 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:31.558 7f8dbcaa1700 1 -- 172.21.2.102:0/1879930559 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8db809a740 msgr2=0x7f8db8097bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:31.563 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:31.558 7f8dbcaa1700 1 --2- 172.21.2.102:0/1879930559 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8db809a740 0x7f8db8097bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:31.563 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:31.558 7f8dbcaa1700 1 -- 172.21.2.102:0/1879930559 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8db8099d20 msgr2=0x7f8db809a120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:31.564 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:31.558 7f8dbcaa1700 1 --2- 172.21.2.102:0/1879930559 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8db8099d20 0x7f8db809a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:31.564 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:31.558 7f8dbcaa1700 1 -- 172.21.2.102:0/1879930559 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8db809ad60 con 0x7f8db80ada90 2021-02-10T11:31:31.564 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:31.558 7f8dbcaa1700 1 --2- 172.21.2.102:0/1879930559 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8db80ada90 0x7f8db80ade90 secure :-1 s=READY pgs=2136 cs=0 l=1 rev1=1 rx=0x7f8da800b5e0 tx=0x7f8da800b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:31.564 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:31.558 7f8db57fa700 1 -- 172.21.2.102:0/1879930559 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8da800bd70 con 0x7f8db80ada90 2021-02-10T11:31:31.564 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:31.558 7f8db57fa700 1 -- 172.21.2.102:0/1879930559 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8da8002030 con 0x7f8db80ada90 2021-02-10T11:31:31.564 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:31.558 7f8db57fa700 1 -- 172.21.2.102:0/1879930559 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8da8002550 con 0x7f8db80ada90 2021-02-10T11:31:31.564 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:31.558 7f8dbec85700 1 -- 172.21.2.102:0/1879930559 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8db809ad60 con 0x7f8db80ada90 2021-02-10T11:31:31.564 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:31.558 7f8dbec85700 1 -- 172.21.2.102:0/1879930559 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8db809ad60 con 0x7f8db80ada90 2021-02-10T11:31:31.564 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:31.559 7f8db57fa700 1 -- 172.21.2.102:0/1879930559 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f8da8000c90 con 0x7f8db80ada90 2021-02-10T11:31:31.564 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:31.559 7f8db57fa700 1 --2- 172.21.2.102:0/1879930559 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8d980186c0 0x7f8d9801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:31.564 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:31.559 7f8db57fa700 1 -- 172.21.2.102:0/1879930559 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f8da80325f0 con 0x7f8db80ada90 2021-02-10T11:31:31.564 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:31.559 7f8db7fff700 1 --2- 172.21.2.102:0/1879930559 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8d980186c0 0x7f8d9801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:31.565 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:31.559 7f8db7fff700 1 --2- 172.21.2.102:0/1879930559 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8d980186c0 0x7f8d9801abc0 crc :-1 s=READY pgs=946 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:31.591 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:31.590 7f8dbec85700 1 -- 172.21.2.102:0/1879930559 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8db80ebc00 con 0x7f8db80ada90 2021-02-10T11:31:31.594 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:31.593 7f8db57fa700 1 -- 172.21.2.102:0/1879930559 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f8da8030da0 con 0x7f8db80ada90 2021-02-10T11:31:31.727 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:31.725 7f8dbec85700 1 -- 172.21.2.102:0/1879930559 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestVolumes.test_default_uid_gid_subvolume"]} v 0) v1 -- 0x7f8db80ebc00 con 0x7f8db80ada90 2021-02-10T11:31:32.480 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:32.479 7f8db57fa700 1 -- 172.21.2.102:0/1879930559 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestVolumes.test_default_uid_gid_subvolume"]}]=0 v560) v1 ==== 147+0+0 (secure 0 0 0) 0x7f8da8000c90 con 0x7f8db80ada90 2021-02-10T11:31:32.498 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:32.497 7f8dbec85700 1 -- 172.21.2.102:0/1879930559 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8d980186c0 msgr2=0x7f8d9801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:32.498 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:32.497 7f8dbec85700 1 --2- 172.21.2.102:0/1879930559 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8d980186c0 0x7f8d9801abc0 crc :-1 s=READY pgs=946 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:32.498 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:32.497 7f8dbec85700 1 -- 172.21.2.102:0/1879930559 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8db80ada90 msgr2=0x7f8db80ade90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:32.498 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:32.497 7f8dbec85700 1 --2- 172.21.2.102:0/1879930559 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8db80ada90 0x7f8db80ade90 secure :-1 s=READY pgs=2136 cs=0 l=1 rev1=1 rx=0x7f8da800b5e0 tx=0x7f8da800b980).stop 2021-02-10T11:31:32.499 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:32.497 7f8dbec85700 1 -- 172.21.2.102:0/1879930559 shutdown_connections 2021-02-10T11:31:32.499 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:32.497 7f8dbec85700 1 --2- 172.21.2.102:0/1879930559 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8db809a740 0x7f8db8097bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:32.499 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:32.497 7f8dbec85700 1 --2- 172.21.2.102:0/1879930559 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8d980186c0 0x7f8d9801abc0 unknown :-1 s=CLOSED pgs=946 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:32.499 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:32.497 7f8dbec85700 1 --2- 172.21.2.102:0/1879930559 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8db8099d20 0x7f8db809a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:32.499 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:32.497 7f8dbec85700 1 --2- 172.21.2.102:0/1879930559 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8db80ada90 0x7f8db80ade90 unknown :-1 s=CLOSED pgs=2136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:32.499 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:32.497 7f8dbec85700 1 -- 172.21.2.102:0/1879930559 >> 172.21.2.102:0/1879930559 conn(0x7f8db80f0550 msgr2=0x7f8db8191730 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:32.499 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:32.497 7f8dbec85700 1 -- 172.21.2.102:0/1879930559 shutdown_connections 2021-02-10T11:31:32.499 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:32.498 7f8dbec85700 1 -- 172.21.2.102:0/1879930559 wait complete. 2021-02-10T11:31:32.507 INFO:tasks.cephfs_test_runner:test_default_uid_gid_subvolume (tasks.cephfs.test_volumes.TestVolumes) ... ok 2021-02-10T11:31:32.507 INFO:tasks.cephfs_test_runner:Starting test: test_default_uid_gid_subvolume_group (tasks.cephfs.test_volumes.TestVolumes) 2021-02-10T11:31:32.508 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestVolumes.test_default_uid_gid_subvolume_group' 2021-02-10T11:31:32.936 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:32.918 7faddf7e7700 1 Processor -- start 2021-02-10T11:31:32.936 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:32.918 7faddf7e7700 1 -- start start 2021-02-10T11:31:32.936 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:32.919 7faddf7e7700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fadd80d1870 0x7fadd81b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:32.936 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:32.919 7faddf7e7700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fadd80f0650 0x7fadd80cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:32.936 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:32.919 7faddf7e7700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fadd80f0b10 con 0x7fadd81b0a10 2021-02-10T11:31:32.936 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:32.919 7faddf7e7700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fadd80e5ed0 con 0x7fadd80d1870 2021-02-10T11:31:32.937 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:32.919 7faddf7e7700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fadd80d1c70 con 0x7fadd80f0650 2021-02-10T11:31:32.937 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:32.919 7faddd603700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fadd80d1870 0x7fadd81b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:32.937 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:32.919 7faddce02700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fadd81b0a10 0x7fadd80cf330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:54150/0 (socket says 172.21.2.102:54150) 2021-02-10T11:31:32.937 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:32.919 7faddce02700 1 -- 172.21.2.102:0/3481401498 learned_addr learned my addr 172.21.2.102:0/3481401498 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:31:32.937 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:32.919 7fadcbfff700 1 -- 172.21.2.102:0/3481401498 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3500568794 0 0) 0x7fadd40010b0 con 0x7fadd81b0a10 2021-02-10T11:31:32.937 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:32.919 7fadcbfff700 1 -- 172.21.2.102:0/3481401498 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fadc4002fe0 con 0x7fadd81b0a10 2021-02-10T11:31:32.937 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:32.919 7faddd603700 1 -- 172.21.2.102:0/3481401498 >> v2:172.21.2.113:3300/0 conn(0x7fadd80f0650 msgr2=0x7fadd80cfb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:32.937 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:32.919 7faddd603700 1 --2- 172.21.2.102:0/3481401498 >> v2:172.21.2.113:3300/0 conn(0x7fadd80f0650 0x7fadd80cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:32.937 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:32.920 7faddd603700 1 -- 172.21.2.102:0/3481401498 >> v1:172.21.2.102:6789/0 conn(0x7fadd81b0a10 legacy=0x7fadd80cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:32.937 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:32.920 7faddd603700 1 -- 172.21.2.102:0/3481401498 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadd80d1e90 con 0x7fadd80d1870 2021-02-10T11:31:32.937 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:32.920 7faddd603700 1 --2- 172.21.2.102:0/3481401498 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fadd80d1870 0x7fadd81b03f0 secure :-1 s=READY pgs=2138 cs=0 l=1 rev1=1 rx=0x7fadcc006650 tx=0x7fadcc009a80).ready entity=mon.0 client_cookie=39c4e4fff419f08e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:32.937 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:32.920 7fadcbfff700 1 -- 172.21.2.102:0/3481401498 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fadcc00c0c0 con 0x7fadd80d1870 2021-02-10T11:31:32.937 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:32.920 7fadcbfff700 1 -- 172.21.2.102:0/3481401498 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadcc00c340 con 0x7fadd80d1870 2021-02-10T11:31:32.938 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:32.920 7fadcbfff700 1 -- 172.21.2.102:0/3481401498 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fadcc00c860 con 0x7fadd80d1870 2021-02-10T11:31:32.938 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:32.920 7faddf7e7700 1 -- 172.21.2.102:0/3481401498 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fadd80d1870 msgr2=0x7fadd81b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:32.938 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:32.920 7faddf7e7700 1 --2- 172.21.2.102:0/3481401498 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fadd80d1870 0x7fadd81b03f0 secure :-1 s=READY pgs=2138 cs=0 l=1 rev1=1 rx=0x7fadcc006650 tx=0x7fadcc009a80).stop 2021-02-10T11:31:32.938 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:32.921 7faddf7e7700 1 -- 172.21.2.102:0/3481401498 shutdown_connections 2021-02-10T11:31:32.938 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:32.921 7faddf7e7700 1 --2- 172.21.2.102:0/3481401498 >> v2:172.21.2.113:3300/0 conn(0x7fadd80f0650 0x7fadd80cfb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:32.938 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:32.921 7faddf7e7700 1 --2- 172.21.2.102:0/3481401498 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fadd80d1870 0x7fadd81b03f0 unknown :-1 s=CLOSED pgs=2138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:32.938 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:32.921 7faddf7e7700 1 -- 172.21.2.102:0/3481401498 >> 172.21.2.102:0/3481401498 conn(0x7fadd80ce450 msgr2=0x7fadd80d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:32.938 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:32.921 7faddf7e7700 1 -- 172.21.2.102:0/3481401498 shutdown_connections 2021-02-10T11:31:32.938 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:32.921 7faddf7e7700 1 -- 172.21.2.102:0/3481401498 wait complete. 2021-02-10T11:31:32.940 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:32.935 7faddf7e7700 1 Processor -- start 2021-02-10T11:31:32.940 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:32.935 7faddf7e7700 1 -- start start 2021-02-10T11:31:32.940 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:32.936 7faddf7e7700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fadd81942b0 0x7fadd80b3550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:32.940 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:32.936 7faddf7e7700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fadd8195070 0x7fadd8195470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:32.940 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:32.936 7faddf7e7700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fadd80bb8e0 0x7fadd80bbce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:32.940 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:32.936 7faddf7e7700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadd81946b0 con 0x7fadd80bb8e0 2021-02-10T11:31:32.940 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:32.936 7faddf7e7700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadd80b3b70 con 0x7fadd8195070 2021-02-10T11:31:32.940 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:32.936 7faddf7e7700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fadd80b3d90 con 0x7fadd81942b0 2021-02-10T11:31:32.940 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:32.936 7faddde04700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fadd80bb8e0 0x7fadd80bbce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:32.941 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:32.936 7faddde04700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fadd80bb8e0 0x7fadd80bbce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:33702/0 (socket says 172.21.2.102:33702) 2021-02-10T11:31:32.941 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:32.936 7faddde04700 1 -- 172.21.2.102:0/3214517101 learned_addr learned my addr 172.21.2.102:0/3214517101 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:31:32.941 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:32.936 7faddde04700 1 -- 172.21.2.102:0/3214517101 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fadd81942b0 msgr2=0x7fadd80b3550 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:31:32.941 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:32.936 7faddde04700 1 --2- 172.21.2.102:0/3214517101 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fadd81942b0 0x7fadd80b3550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:32.941 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:32.936 7faddde04700 1 -- 172.21.2.102:0/3214517101 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fadd8195070 msgr2=0x7fadd8195470 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:31:32.941 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:32.936 7faddde04700 1 --2- 172.21.2.102:0/3214517101 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fadd8195070 0x7fadd8195470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:32.941 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:32.936 7faddde04700 1 -- 172.21.2.102:0/3214517101 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadd81a49e0 con 0x7fadd80bb8e0 2021-02-10T11:31:32.941 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:32.936 7faddde04700 1 --2- 172.21.2.102:0/3214517101 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fadd80bb8e0 0x7fadd80bbce0 secure :-1 s=READY pgs=2139 cs=0 l=1 rev1=1 rx=0x7fadd0009980 tx=0x7fadd000bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:32.941 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:32.936 7fadc9ffb700 1 -- 172.21.2.102:0/3214517101 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fadd0003020 con 0x7fadd80bb8e0 2021-02-10T11:31:32.941 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:32.937 7fadc9ffb700 1 -- 172.21.2.102:0/3214517101 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadd0005a80 con 0x7fadd80bb8e0 2021-02-10T11:31:32.941 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:32.937 7fadc9ffb700 1 -- 172.21.2.102:0/3214517101 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fadd0006030 con 0x7fadd80bb8e0 2021-02-10T11:31:32.941 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:32.937 7faddf7e7700 1 -- 172.21.2.102:0/3214517101 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fadd81a49e0 con 0x7fadd80bb8e0 2021-02-10T11:31:32.941 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:32.937 7faddf7e7700 1 -- 172.21.2.102:0/3214517101 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fadd81a49e0 con 0x7fadd80bb8e0 2021-02-10T11:31:32.942 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:32.937 7fadc9ffb700 1 -- 172.21.2.102:0/3214517101 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fadd0005c60 con 0x7fadd80bb8e0 2021-02-10T11:31:32.942 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:32.937 7fadc9ffb700 1 --2- 172.21.2.102:0/3214517101 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fadb80186c0 0x7fadb801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:32.942 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:32.937 7fadc9ffb700 1 -- 172.21.2.102:0/3214517101 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fadd0032b50 con 0x7fadd80bb8e0 2021-02-10T11:31:32.942 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:32.937 7faddd603700 1 --2- 172.21.2.102:0/3214517101 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fadb80186c0 0x7fadb801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:32.942 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:32.938 7faddd603700 1 --2- 172.21.2.102:0/3214517101 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fadb80186c0 0x7fadb801abc0 crc :-1 s=READY pgs=947 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:32.970 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:32.969 7faddf7e7700 1 -- 172.21.2.102:0/3214517101 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fadd80bc5f0 con 0x7fadd80bb8e0 2021-02-10T11:31:32.972 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:32.971 7fadc9ffb700 1 -- 172.21.2.102:0/3214517101 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fadd0005870 con 0x7fadd80bb8e0 2021-02-10T11:31:33.102 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:33.101 7faddf7e7700 1 -- 172.21.2.102:0/3214517101 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestVolumes.test_default_uid_gid_subvolume_group"]} v 0) v1 -- 0x7fadd80bc5f0 con 0x7fadd80bb8e0 2021-02-10T11:31:33.484 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:33.483 7fadc9ffb700 1 -- 172.21.2.102:0/3214517101 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestVolumes.test_default_uid_gid_subvolume_group"]}]=0 v561) v1 ==== 156+0+0 (secure 0 0 0) 0x7fadd0005a80 con 0x7fadd80bb8e0 2021-02-10T11:31:33.522 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:33.520 7faddf7e7700 1 -- 172.21.2.102:0/3214517101 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fadb80186c0 msgr2=0x7fadb801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:33.522 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:33.520 7faddf7e7700 1 --2- 172.21.2.102:0/3214517101 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fadb80186c0 0x7fadb801abc0 crc :-1 s=READY pgs=947 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:33.522 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:33.520 7faddf7e7700 1 -- 172.21.2.102:0/3214517101 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fadd80bb8e0 msgr2=0x7fadd80bbce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:33.522 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:33.520 7faddf7e7700 1 --2- 172.21.2.102:0/3214517101 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fadd80bb8e0 0x7fadd80bbce0 secure :-1 s=READY pgs=2139 cs=0 l=1 rev1=1 rx=0x7fadd0009980 tx=0x7fadd000bcd0).stop 2021-02-10T11:31:33.522 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:33.520 7faddf7e7700 1 -- 172.21.2.102:0/3214517101 shutdown_connections 2021-02-10T11:31:33.522 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:33.520 7faddf7e7700 1 --2- 172.21.2.102:0/3214517101 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fadd80bb8e0 0x7fadd80bbce0 unknown :-1 s=CLOSED pgs=2139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:33.522 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:33.520 7faddf7e7700 1 --2- 172.21.2.102:0/3214517101 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fadb80186c0 0x7fadb801abc0 unknown :-1 s=CLOSED pgs=947 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:33.522 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:33.520 7faddf7e7700 1 --2- 172.21.2.102:0/3214517101 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fadd8195070 0x7fadd8195470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:33.522 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:33.520 7faddf7e7700 1 --2- 172.21.2.102:0/3214517101 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fadd81942b0 0x7fadd80b3550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:33.522 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:33.520 7faddf7e7700 1 -- 172.21.2.102:0/3214517101 >> 172.21.2.102:0/3214517101 conn(0x7fadd80b3040 msgr2=0x7fadd80c3250 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:33.523 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:33.521 7faddf7e7700 1 -- 172.21.2.102:0/3214517101 shutdown_connections 2021-02-10T11:31:33.523 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:33.521 7faddf7e7700 1 -- 172.21.2.102:0/3214517101 wait complete. 2021-02-10T11:31:33.530 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config set mon mon_allow_pool_delete True 2021-02-10T11:31:33.961 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:33.943 7f6da3c1d700 1 Processor -- start 2021-02-10T11:31:33.962 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:33.943 7f6da3c1d700 1 -- start start 2021-02-10T11:31:33.962 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:33.943 7f6da3c1d700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f6d9c0d1870 0x7f6d9c1b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:33.962 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:33.943 7f6da3c1d700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6d9c0cff70 0x7f6d9c109130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:33.962 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:33.943 7f6da3c1d700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d9c109a60 con 0x7f6d9c1b0ae0 2021-02-10T11:31:33.962 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:33.943 7f6da3c1d700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d9c109750 con 0x7f6d9c0d1870 2021-02-10T11:31:33.962 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:33.943 7f6da3c1d700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d9c088070 con 0x7f6d9c0cff70 2021-02-10T11:31:33.962 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:33.944 7f6da1238700 1 --1- >> v1:172.21.2.113:6789/0 conn(0x7f6d9c1b0ae0 0x7f6d9c108940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.113:6789/0 says I am v1:172.21.2.102:33254/0 (socket says 172.21.2.102:33254) 2021-02-10T11:31:33.962 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:33.944 7f6da223a700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6d9c0cff70 0x7f6d9c109130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:33.962 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:33.944 7f6da1a39700 1 --2- 172.21.2.102:0/3452075462 >> v2:172.21.2.113:3300/0 conn(0x7f6d9c0d1870 0x7f6d9c1b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:33.962 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:33.944 7f6da1238700 1 -- 172.21.2.102:0/3452075462 learned_addr learned my addr 172.21.2.102:0/3452075462 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:31:33.962 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:33.945 7f6da223a700 1 -- 172.21.2.102:0/3452075462 >> v2:172.21.2.113:3300/0 conn(0x7f6d9c0d1870 msgr2=0x7f6d9c1b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:33.963 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:33.945 7f6da223a700 1 --2- 172.21.2.102:0/3452075462 >> v2:172.21.2.113:3300/0 conn(0x7f6d9c0d1870 0x7f6d9c1b04c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:33.963 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:33.945 7f6da223a700 1 -- 172.21.2.102:0/3452075462 >> v1:172.21.2.113:6789/0 conn(0x7f6d9c1b0ae0 legacy=0x7f6d9c108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:33.963 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:33.945 7f6da223a700 1 -- 172.21.2.102:0/3452075462 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d9c0d1c70 con 0x7f6d9c0cff70 2021-02-10T11:31:33.963 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:33.945 7f6da1a39700 1 --2- 172.21.2.102:0/3452075462 >> v2:172.21.2.113:3300/0 conn(0x7f6d9c0d1870 0x7f6d9c1b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:31:33.963 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:33.945 7f6da223a700 1 --2- 172.21.2.102:0/3452075462 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6d9c0cff70 0x7f6d9c109130 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f6d94006e50 tx=0x7f6d94009e70).ready entity=mon.2 client_cookie=4e772fed2bfb7897 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:33.963 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:33.945 7f6da0a37700 1 -- 172.21.2.102:0/3452075462 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6d9400b510 con 0x7f6d9c0cff70 2021-02-10T11:31:33.963 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:33.945 7f6da0a37700 1 -- 172.21.2.102:0/3452075462 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d94006790 con 0x7f6d9c0cff70 2021-02-10T11:31:33.963 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:33.945 7f6da0a37700 1 -- 172.21.2.102:0/3452075462 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6d9400b940 con 0x7f6d9c0cff70 2021-02-10T11:31:33.963 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:33.945 7f6da3c1d700 1 -- 172.21.2.102:0/3452075462 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6d9c0cff70 msgr2=0x7f6d9c109130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:33.963 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:33.945 7f6da3c1d700 1 --2- 172.21.2.102:0/3452075462 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6d9c0cff70 0x7f6d9c109130 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f6d94006e50 tx=0x7f6d94009e70).stop 2021-02-10T11:31:33.964 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:33.946 7f6da3c1d700 1 -- 172.21.2.102:0/3452075462 shutdown_connections 2021-02-10T11:31:33.964 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:33.946 7f6da3c1d700 1 --2- 172.21.2.102:0/3452075462 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6d9c0cff70 0x7f6d9c109130 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:33.964 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:33.946 7f6da3c1d700 1 --2- 172.21.2.102:0/3452075462 >> v2:172.21.2.113:3300/0 conn(0x7f6d9c0d1870 0x7f6d9c1b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:33.964 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:33.946 7f6da3c1d700 1 -- 172.21.2.102:0/3452075462 >> 172.21.2.102:0/3452075462 conn(0x7f6d9c0ce420 msgr2=0x7f6d9c0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:33.964 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:33.946 7f6da3c1d700 1 -- 172.21.2.102:0/3452075462 shutdown_connections 2021-02-10T11:31:33.964 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:33.946 7f6da3c1d700 1 -- 172.21.2.102:0/3452075462 wait complete. 2021-02-10T11:31:33.966 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:33.961 7f6da3c1d700 1 Processor -- start 2021-02-10T11:31:33.966 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:33.961 7f6da3c1d700 1 -- start start 2021-02-10T11:31:33.966 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:33.961 7f6da3c1d700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6d9c0a7950 0x7f6d9c0b8df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:33.966 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:33.961 7f6da3c1d700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6d9c1a7490 0x7f6d9c1a7890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:33.967 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:33.961 7f6da3c1d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6d9c1a7eb0 0x7f6d9c0ac630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:33.967 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:33.961 7f6da3c1d700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d9c0a7fb0 con 0x7f6d9c1a7eb0 2021-02-10T11:31:33.967 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:33.961 7f6da3c1d700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d9c0acc50 con 0x7f6d9c0a7950 2021-02-10T11:31:33.967 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:33.961 7f6da3c1d700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d9c0ace30 con 0x7f6d9c1a7490 2021-02-10T11:31:33.967 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:33.961 7f6da223a700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6d9c1a7eb0 0x7f6d9c0ac630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:33.967 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:33.961 7f6da223a700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6d9c1a7eb0 0x7f6d9c0ac630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:33716/0 (socket says 172.21.2.102:33716) 2021-02-10T11:31:33.967 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:33.961 7f6da223a700 1 -- 172.21.2.102:0/3090792074 learned_addr learned my addr 172.21.2.102:0/3090792074 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:31:33.968 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:33.962 7f6da1a39700 1 --2- 172.21.2.102:0/3090792074 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6d9c0a7950 0x7f6d9c0b8df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:33.968 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:33.962 7f6da1238700 1 --2- 172.21.2.102:0/3090792074 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6d9c1a7490 0x7f6d9c1a7890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:33.968 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:33.962 7f6da223a700 1 -- 172.21.2.102:0/3090792074 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6d9c1a7490 msgr2=0x7f6d9c1a7890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:33.968 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:33.962 7f6da223a700 1 --2- 172.21.2.102:0/3090792074 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6d9c1a7490 0x7f6d9c1a7890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:33.968 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:33.962 7f6da223a700 1 -- 172.21.2.102:0/3090792074 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6d9c0a7950 msgr2=0x7f6d9c0b8df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:33.968 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:33.962 7f6da223a700 1 --2- 172.21.2.102:0/3090792074 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6d9c0a7950 0x7f6d9c0b8df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:33.968 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:33.962 7f6da223a700 1 -- 172.21.2.102:0/3090792074 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d9c0ad050 con 0x7f6d9c1a7eb0 2021-02-10T11:31:33.969 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:33.962 7f6da1a39700 1 --2- 172.21.2.102:0/3090792074 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6d9c0a7950 0x7f6d9c0b8df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:31:33.969 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:33.962 7f6da1238700 1 --2- 172.21.2.102:0/3090792074 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6d9c1a7490 0x7f6d9c1a7890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:31:33.969 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:33.962 7f6da223a700 1 --2- 172.21.2.102:0/3090792074 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6d9c1a7eb0 0x7f6d9c0ac630 secure :-1 s=READY pgs=2140 cs=0 l=1 rev1=1 rx=0x7f6d94006c50 tx=0x7f6d9400bd20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:33.969 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:33.962 7f6d8e7fc700 1 -- 172.21.2.102:0/3090792074 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6d940093d0 con 0x7f6d9c1a7eb0 2021-02-10T11:31:33.969 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:33.962 7f6d8e7fc700 1 -- 172.21.2.102:0/3090792074 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d9400be60 con 0x7f6d9c1a7eb0 2021-02-10T11:31:33.969 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:33.962 7f6d8e7fc700 1 -- 172.21.2.102:0/3090792074 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6d9400c570 con 0x7f6d9c1a7eb0 2021-02-10T11:31:33.969 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:33.962 7f6da3c1d700 1 -- 172.21.2.102:0/3090792074 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d9c0ad050 con 0x7f6d9c1a7eb0 2021-02-10T11:31:33.969 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:33.963 7f6da3c1d700 1 -- 172.21.2.102:0/3090792074 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d9c0ad050 con 0x7f6d9c1a7eb0 2021-02-10T11:31:33.969 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:33.963 7f6d8e7fc700 1 -- 172.21.2.102:0/3090792074 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f6d940093d0 con 0x7f6d9c1a7eb0 2021-02-10T11:31:33.969 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:33.963 7f6d8e7fc700 1 --2- 172.21.2.102:0/3090792074 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6d840186c0 0x7f6d8401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:33.969 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:33.963 7f6d8e7fc700 1 -- 172.21.2.102:0/3090792074 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f6d94032910 con 0x7f6d9c1a7eb0 2021-02-10T11:31:33.970 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:33.963 7f6da1a39700 1 --2- 172.21.2.102:0/3090792074 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6d840186c0 0x7f6d8401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:33.970 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:33.964 7f6da1a39700 1 --2- 172.21.2.102:0/3090792074 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6d840186c0 0x7f6d8401abc0 crc :-1 s=READY pgs=948 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:33.995 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:33.993 7f6da3c1d700 1 -- 172.21.2.102:0/3090792074 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d9c082a70 con 0x7f6d9c1a7eb0 2021-02-10T11:31:33.997 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:33.996 7f6d8e7fc700 1 -- 172.21.2.102:0/3090792074 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f6d94028f10 con 0x7f6d9c1a7eb0 2021-02-10T11:31:34.147 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:34.146 7f6da3c1d700 1 -- 172.21.2.102:0/3090792074 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f6d9c082a70 con 0x7f6d9c1a7eb0 2021-02-10T11:31:34.151 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:34.150 7f6d8e7fc700 1 -- 172.21.2.102:0/3090792074 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v28)=0 v28) v1 ==== 122+0+0 (secure 0 0 0) 0x7f6d94028f10 con 0x7f6d9c1a7eb0 2021-02-10T11:31:34.156 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:34.155 7f6da3c1d700 1 -- 172.21.2.102:0/3090792074 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6d840186c0 msgr2=0x7f6d8401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:34.156 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:34.155 7f6da3c1d700 1 --2- 172.21.2.102:0/3090792074 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6d840186c0 0x7f6d8401abc0 crc :-1 s=READY pgs=948 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:34.156 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:34.155 7f6da3c1d700 1 -- 172.21.2.102:0/3090792074 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6d9c1a7eb0 msgr2=0x7f6d9c0ac630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:34.156 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:34.155 7f6da3c1d700 1 --2- 172.21.2.102:0/3090792074 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6d9c1a7eb0 0x7f6d9c0ac630 secure :-1 s=READY pgs=2140 cs=0 l=1 rev1=1 rx=0x7f6d94006c50 tx=0x7f6d9400bd20).stop 2021-02-10T11:31:34.156 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:34.155 7f6da3c1d700 1 -- 172.21.2.102:0/3090792074 shutdown_connections 2021-02-10T11:31:34.156 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:34.155 7f6da3c1d700 1 --2- 172.21.2.102:0/3090792074 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6d9c1a7eb0 0x7f6d9c0ac630 unknown :-1 s=CLOSED pgs=2140 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:34.156 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:34.155 7f6da3c1d700 1 --2- 172.21.2.102:0/3090792074 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6d9c1a7490 0x7f6d9c1a7890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:34.156 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:34.155 7f6da3c1d700 1 --2- 172.21.2.102:0/3090792074 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6d840186c0 0x7f6d8401abc0 unknown :-1 s=CLOSED pgs=948 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:34.157 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:34.155 7f6da3c1d700 1 --2- 172.21.2.102:0/3090792074 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6d9c0a7950 0x7f6d9c0b8df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:34.157 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:34.155 7f6da3c1d700 1 -- 172.21.2.102:0/3090792074 >> 172.21.2.102:0/3090792074 conn(0x7f6d9c0a02d0 msgr2=0x7f6d9c09f200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:34.157 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:34.155 7f6da3c1d700 1 -- 172.21.2.102:0/3090792074 shutdown_connections 2021-02-10T11:31:34.157 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:34.155 7f6da3c1d700 1 -- 172.21.2.102:0/3090792074 wait complete. 2021-02-10T11:31:34.165 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2021-02-10T11:31:34.165 DEBUG:teuthology.orchestra.run.gibba002:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:31:34.511 DEBUG:teuthology.orchestra.run.gibba013:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:31:34.855 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2021-02-10T11:31:34.855 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:31:34.856 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:31:34.856 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:31:35.193 INFO:teuthology.orchestra.run.gibba002.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.0’: No such file or directory 2021-02-10T11:31:35.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:31:35.193 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.0 2021-02-10T11:31:35.193 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:31:35.193 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:31:35.194 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:31:35.529 INFO:teuthology.orchestra.run.gibba002.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.1’: No such file or directory 2021-02-10T11:31:35.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:31:35.530 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2021-02-10T11:31:35.531 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:31:35.531 DEBUG:teuthology.orchestra.run.gibba013:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:31:35.531 DEBUG:teuthology.orchestra.run.gibba013:> ' -- /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:31:35.866 INFO:teuthology.orchestra.run.gibba013.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.2’: No such file or directory 2021-02-10T11:31:35.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:31:35.867 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2021-02-10T11:31:35.867 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:31:35.867 DEBUG:teuthology.orchestra.run.gibba013:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:31:35.867 DEBUG:teuthology.orchestra.run.gibba013:> ' -- /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:31:36.201 INFO:teuthology.orchestra.run.gibba013.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.3’: No such file or directory 2021-02-10T11:31:36.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:31:36.201 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.3 2021-02-10T11:31:36.202 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json-pretty 2021-02-10T11:31:36.628 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:36.611 7f0d2f9df700 1 Processor -- start 2021-02-10T11:31:36.629 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:36.611 7f0d2f9df700 1 -- start start 2021-02-10T11:31:36.629 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:36.612 7f0d2f9df700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f0d280d1870 0x7f0d281b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:36.629 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:36.612 7f0d2f9df700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f0d281b0a10 0x7f0d280cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:36.629 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:36.612 7f0d2f9df700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d280f0b10 con 0x7f0d280d1c70 2021-02-10T11:31:36.629 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:36.612 7f0d2f9df700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d280e5ed0 con 0x7f0d281b0a10 2021-02-10T11:31:36.629 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:36.612 7f0d2f9df700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d280f0650 con 0x7f0d280d1870 2021-02-10T11:31:36.629 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:36.612 7f0d2cffa700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f0d281b0a10 0x7f0d280cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:36.629 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:36.612 7f0d2cffa700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f0d281b0a10 0x7f0d280cf330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:33720/0 (socket says 172.21.2.102:33720) 2021-02-10T11:31:36.630 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:36.612 7f0d2cffa700 1 -- 172.21.2.102:0/2102571898 learned_addr learned my addr 172.21.2.102:0/2102571898 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:31:36.630 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:36.612 7f0d2d7fb700 1 --2- 172.21.2.102:0/2102571898 >> v2:172.21.2.113:3300/0 conn(0x7f0d280d1870 0x7f0d281b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:36.630 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:36.612 7f0d2cffa700 1 -- 172.21.2.102:0/2102571898 >> v2:172.21.2.113:3300/0 conn(0x7f0d280d1870 msgr2=0x7f0d281b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:36.630 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:36.612 7f0d2cffa700 1 --2- 172.21.2.102:0/2102571898 >> v2:172.21.2.113:3300/0 conn(0x7f0d280d1870 0x7f0d281b03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:36.630 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:36.612 7f0d2cffa700 1 -- 172.21.2.102:0/2102571898 >> v1:172.21.2.113:6789/0 conn(0x7f0d280d1c70 legacy=0x7f0d280cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:36.630 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:36.612 7f0d2cffa700 1 -- 172.21.2.102:0/2102571898 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d280f0840 con 0x7f0d281b0a10 2021-02-10T11:31:36.630 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:36.612 7f0d2cffa700 1 --2- 172.21.2.102:0/2102571898 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0d281b0a10 0x7f0d280cf330 secure :-1 s=READY pgs=2141 cs=0 l=1 rev1=1 rx=0x7f0d1c006700 tx=0x7f0d1c009b00).ready entity=mon.0 client_cookie=261c9149a29b4623 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:36.630 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:36.613 7f0d1bfff700 1 -- 172.21.2.102:0/2102571898 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0d1c00c0c0 con 0x7f0d281b0a10 2021-02-10T11:31:36.630 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:36.613 7f0d1bfff700 1 -- 172.21.2.102:0/2102571898 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d1c00c400 con 0x7f0d281b0a10 2021-02-10T11:31:36.630 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:36.613 7f0d1bfff700 1 -- 172.21.2.102:0/2102571898 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0d1c00c8e0 con 0x7f0d281b0a10 2021-02-10T11:31:36.630 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:36.613 7f0d2f9df700 1 -- 172.21.2.102:0/2102571898 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0d281b0a10 msgr2=0x7f0d280cf330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:36.630 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:36.613 7f0d2f9df700 1 --2- 172.21.2.102:0/2102571898 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0d281b0a10 0x7f0d280cf330 secure :-1 s=READY pgs=2141 cs=0 l=1 rev1=1 rx=0x7f0d1c006700 tx=0x7f0d1c009b00).stop 2021-02-10T11:31:36.630 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:36.613 7f0d2f9df700 1 -- 172.21.2.102:0/2102571898 shutdown_connections 2021-02-10T11:31:36.631 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:36.613 7f0d2f9df700 1 --2- 172.21.2.102:0/2102571898 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0d281b0a10 0x7f0d280cf330 unknown :-1 s=CLOSED pgs=2141 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:36.631 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:36.613 7f0d2f9df700 1 --2- 172.21.2.102:0/2102571898 >> v2:172.21.2.113:3300/0 conn(0x7f0d280d1870 0x7f0d281b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:36.631 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:36.613 7f0d2f9df700 1 -- 172.21.2.102:0/2102571898 >> 172.21.2.102:0/2102571898 conn(0x7f0d280ce450 msgr2=0x7f0d280d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:36.631 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:36.613 7f0d2f9df700 1 -- 172.21.2.102:0/2102571898 shutdown_connections 2021-02-10T11:31:36.631 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:36.613 7f0d2f9df700 1 -- 172.21.2.102:0/2102571898 wait complete. 2021-02-10T11:31:36.633 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:36.628 7f0d2f9df700 1 Processor -- start 2021-02-10T11:31:36.633 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:36.628 7f0d2f9df700 1 -- start start 2021-02-10T11:31:36.633 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:36.628 7f0d2f9df700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0d28195650 0x7f0d280b5560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:36.633 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:36.628 7f0d2f9df700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0d280b5b80 0x7f0d281a2e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:36.633 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:36.628 7f0d2f9df700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0d280acfa0 0x7f0d281a3440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:36.633 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:36.628 7f0d2f9df700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d280b6010 con 0x7f0d280b5b80 2021-02-10T11:31:36.633 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:36.628 7f0d2f9df700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d281a3a60 con 0x7f0d28195650 2021-02-10T11:31:36.633 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:36.628 7f0d2f9df700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d281a3c40 con 0x7f0d280acfa0 2021-02-10T11:31:36.633 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:36.629 7f0d2cffa700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0d280b5b80 0x7f0d281a2e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:36.633 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:36.629 7f0d2cffa700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0d280b5b80 0x7f0d281a2e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:33724/0 (socket says 172.21.2.102:33724) 2021-02-10T11:31:36.633 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:36.629 7f0d2cffa700 1 -- 172.21.2.102:0/1917446514 learned_addr learned my addr 172.21.2.102:0/1917446514 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:31:36.633 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:36.629 7f0d2d7fb700 1 --2- 172.21.2.102:0/1917446514 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0d28195650 0x7f0d280b5560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:36.634 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:36.629 7f0d2dffc700 1 --2- 172.21.2.102:0/1917446514 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0d280acfa0 0x7f0d281a3440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:36.634 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:36.629 7f0d2cffa700 1 -- 172.21.2.102:0/1917446514 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0d280acfa0 msgr2=0x7f0d281a3440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:36.634 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:36.629 7f0d2cffa700 1 --2- 172.21.2.102:0/1917446514 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0d280acfa0 0x7f0d281a3440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:36.634 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:36.629 7f0d2cffa700 1 -- 172.21.2.102:0/1917446514 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0d28195650 msgr2=0x7f0d280b5560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:36.634 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:36.629 7f0d2cffa700 1 --2- 172.21.2.102:0/1917446514 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0d28195650 0x7f0d280b5560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:36.634 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:36.629 7f0d2cffa700 1 -- 172.21.2.102:0/1917446514 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d280bcae0 con 0x7f0d280b5b80 2021-02-10T11:31:36.634 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:36.629 7f0d2cffa700 1 --2- 172.21.2.102:0/1917446514 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0d280b5b80 0x7f0d281a2e20 secure :-1 s=READY pgs=2142 cs=0 l=1 rev1=1 rx=0x7f0d1c006f20 tx=0x7f0d1c00b9e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:36.634 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:36.629 7f0d19ffb700 1 -- 172.21.2.102:0/1917446514 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0d1c00bdd0 con 0x7f0d280b5b80 2021-02-10T11:31:36.634 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:36.629 7f0d19ffb700 1 -- 172.21.2.102:0/1917446514 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d1c006440 con 0x7f0d280b5b80 2021-02-10T11:31:36.635 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:36.629 7f0d2f9df700 1 -- 172.21.2.102:0/1917446514 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d280bcae0 con 0x7f0d280b5b80 2021-02-10T11:31:36.635 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:36.629 7f0d2f9df700 1 -- 172.21.2.102:0/1917446514 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d280bcae0 con 0x7f0d280b5b80 2021-02-10T11:31:36.635 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:36.629 7f0d19ffb700 1 -- 172.21.2.102:0/1917446514 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0d1c0052e0 con 0x7f0d280b5b80 2021-02-10T11:31:36.635 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:36.630 7f0d19ffb700 1 -- 172.21.2.102:0/1917446514 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f0d1c006030 con 0x7f0d280b5b80 2021-02-10T11:31:36.635 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:36.630 7f0d19ffb700 1 --2- 172.21.2.102:0/1917446514 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0d08018700 0x7f0d0801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:36.635 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:36.630 7f0d19ffb700 1 -- 172.21.2.102:0/1917446514 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f0d1c0327d0 con 0x7f0d280b5b80 2021-02-10T11:31:36.635 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:36.630 7f0d2d7fb700 1 --2- 172.21.2.102:0/1917446514 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0d08018700 0x7f0d0801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:36.635 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:36.630 7f0d2d7fb700 1 --2- 172.21.2.102:0/1917446514 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0d08018700 0x7f0d0801abc0 crc :-1 s=READY pgs=949 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:36.663 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:36.661 7f0d2f9df700 1 -- 172.21.2.102:0/1917446514 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d281a3c40 con 0x7f0d280b5b80 2021-02-10T11:31:36.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:36.664 7f0d19ffb700 1 -- 172.21.2.102:0/1917446514 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f0d1c006030 con 0x7f0d280b5b80 2021-02-10T11:31:36.790 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:36.788 7f0d2f9df700 1 -- 172.21.2.102:0/1917446514 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7f0d281a3c40 con 0x7f0d280b5b80 2021-02-10T11:31:36.790 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:36.789 7f0d19ffb700 1 -- 172.21.2.102:0/1917446514 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v103) v1 ==== 81+0+30372 (secure 0 0 0) 0x7f0d1c006440 con 0x7f0d280b5b80 2021-02-10T11:31:36.791 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:31:36.791 INFO:teuthology.orchestra.run.gibba002.stdout:{ 2021-02-10T11:31:36.791 INFO:teuthology.orchestra.run.gibba002.stdout: "epoch": 103, 2021-02-10T11:31:36.791 INFO:teuthology.orchestra.run.gibba002.stdout: "fsid": "68db30ea-dbce-4aef-a6df-f01c6fea0513", 2021-02-10T11:31:36.791 INFO:teuthology.orchestra.run.gibba002.stdout: "created": "2021-02-10 11:03:06.331557", 2021-02-10T11:31:36.791 INFO:teuthology.orchestra.run.gibba002.stdout: "modified": "2021-02-10 11:31:00.055039", 2021-02-10T11:31:36.791 INFO:teuthology.orchestra.run.gibba002.stdout: "last_up_change": "2021-02-10 11:03:32.690562", 2021-02-10T11:31:36.791 INFO:teuthology.orchestra.run.gibba002.stdout: "last_in_change": "2021-02-10 11:03:32.690562", 2021-02-10T11:31:36.791 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2021-02-10T11:31:36.792 INFO:teuthology.orchestra.run.gibba002.stdout: "flags_num": 5799936, 2021-02-10T11:31:36.792 INFO:teuthology.orchestra.run.gibba002.stdout: "flags_set": [ 2021-02-10T11:31:36.792 INFO:teuthology.orchestra.run.gibba002.stdout: "pglog_hardlimit", 2021-02-10T11:31:36.792 INFO:teuthology.orchestra.run.gibba002.stdout: "purged_snapdirs", 2021-02-10T11:31:36.792 INFO:teuthology.orchestra.run.gibba002.stdout: "recovery_deletes", 2021-02-10T11:31:36.792 INFO:teuthology.orchestra.run.gibba002.stdout: "sortbitwise" 2021-02-10T11:31:36.792 INFO:teuthology.orchestra.run.gibba002.stdout: ], 2021-02-10T11:31:36.792 INFO:teuthology.orchestra.run.gibba002.stdout: "crush_version": 4, 2021-02-10T11:31:36.792 INFO:teuthology.orchestra.run.gibba002.stdout: "full_ratio": 0.94999998807907104, 2021-02-10T11:31:36.792 INFO:teuthology.orchestra.run.gibba002.stdout: "backfillfull_ratio": 0.89999997615814209, 2021-02-10T11:31:36.792 INFO:teuthology.orchestra.run.gibba002.stdout: "nearfull_ratio": 0.85000002384185791, 2021-02-10T11:31:36.792 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_snapshot": "", 2021-02-10T11:31:36.792 INFO:teuthology.orchestra.run.gibba002.stdout: "pool_max": 21, 2021-02-10T11:31:36.793 INFO:teuthology.orchestra.run.gibba002.stdout: "max_osd": 8, 2021-02-10T11:31:36.793 INFO:teuthology.orchestra.run.gibba002.stdout: "require_min_compat_client": "jewel", 2021-02-10T11:31:36.793 INFO:teuthology.orchestra.run.gibba002.stdout: "min_compat_client": "jewel", 2021-02-10T11:31:36.793 INFO:teuthology.orchestra.run.gibba002.stdout: "require_osd_release": "nautilus", 2021-02-10T11:31:36.793 INFO:teuthology.orchestra.run.gibba002.stdout: "pools": [ 2021-02-10T11:31:36.793 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:31:36.793 INFO:teuthology.orchestra.run.gibba002.stdout: "pool": 1, 2021-02-10T11:31:36.793 INFO:teuthology.orchestra.run.gibba002.stdout: "pool_name": "rbd", 2021-02-10T11:31:36.793 INFO:teuthology.orchestra.run.gibba002.stdout: "create_time": "2021-02-10 11:03:37.322519", 2021-02-10T11:31:36.793 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 1, 2021-02-10T11:31:36.793 INFO:teuthology.orchestra.run.gibba002.stdout: "flags_names": "hashpspool", 2021-02-10T11:31:36.793 INFO:teuthology.orchestra.run.gibba002.stdout: "type": 1, 2021-02-10T11:31:36.793 INFO:teuthology.orchestra.run.gibba002.stdout: "size": 2, 2021-02-10T11:31:36.793 INFO:teuthology.orchestra.run.gibba002.stdout: "min_size": 1, 2021-02-10T11:31:36.794 INFO:teuthology.orchestra.run.gibba002.stdout: "crush_rule": 0, 2021-02-10T11:31:36.794 INFO:teuthology.orchestra.run.gibba002.stdout: "object_hash": 2, 2021-02-10T11:31:36.794 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:31:36.794 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num": 8, 2021-02-10T11:31:36.794 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_placement_num": 8, 2021-02-10T11:31:36.794 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_placement_num_target": 8, 2021-02-10T11:31:36.794 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num_target": 8, 2021-02-10T11:31:36.794 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num_pending": 8, 2021-02-10T11:31:36.794 INFO:teuthology.orchestra.run.gibba002.stdout: "last_pg_merge_meta": { 2021-02-10T11:31:36.794 INFO:teuthology.orchestra.run.gibba002.stdout: "source_pgid": "0.0", 2021-02-10T11:31:36.794 INFO:teuthology.orchestra.run.gibba002.stdout: "ready_epoch": 0, 2021-02-10T11:31:36.794 INFO:teuthology.orchestra.run.gibba002.stdout: "last_epoch_started": 0, 2021-02-10T11:31:36.794 INFO:teuthology.orchestra.run.gibba002.stdout: "last_epoch_clean": 0, 2021-02-10T11:31:36.795 INFO:teuthology.orchestra.run.gibba002.stdout: "source_version": "0'0", 2021-02-10T11:31:36.795 INFO:teuthology.orchestra.run.gibba002.stdout: "target_version": "0'0" 2021-02-10T11:31:36.795 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:31:36.795 INFO:teuthology.orchestra.run.gibba002.stdout: "last_change": "16", 2021-02-10T11:31:36.795 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend": "0", 2021-02-10T11:31:36.795 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:31:36.795 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:31:36.795 INFO:teuthology.orchestra.run.gibba002.stdout: "auid": 0, 2021-02-10T11:31:36.795 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:31:36.795 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_seq": 0, 2021-02-10T11:31:36.795 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_epoch": 0, 2021-02-10T11:31:36.795 INFO:teuthology.orchestra.run.gibba002.stdout: "pool_snaps": [], 2021-02-10T11:31:36.795 INFO:teuthology.orchestra.run.gibba002.stdout: "removed_snaps": "[]", 2021-02-10T11:31:36.795 INFO:teuthology.orchestra.run.gibba002.stdout: "quota_max_bytes": 0, 2021-02-10T11:31:36.796 INFO:teuthology.orchestra.run.gibba002.stdout: "quota_max_objects": 0, 2021-02-10T11:31:36.796 INFO:teuthology.orchestra.run.gibba002.stdout: "tiers": [], 2021-02-10T11:31:36.796 INFO:teuthology.orchestra.run.gibba002.stdout: "tier_of": -1, 2021-02-10T11:31:36.796 INFO:teuthology.orchestra.run.gibba002.stdout: "read_tier": -1, 2021-02-10T11:31:36.796 INFO:teuthology.orchestra.run.gibba002.stdout: "write_tier": -1, 2021-02-10T11:31:36.796 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_mode": "none", 2021-02-10T11:31:36.796 INFO:teuthology.orchestra.run.gibba002.stdout: "target_max_bytes": 0, 2021-02-10T11:31:36.796 INFO:teuthology.orchestra.run.gibba002.stdout: "target_max_objects": 0, 2021-02-10T11:31:36.796 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:31:36.796 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:31:36.796 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:31:36.796 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_min_flush_age": 0, 2021-02-10T11:31:36.796 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_min_evict_age": 0, 2021-02-10T11:31:36.797 INFO:teuthology.orchestra.run.gibba002.stdout: "erasure_code_profile": "", 2021-02-10T11:31:36.797 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_params": { 2021-02-10T11:31:36.797 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "none" 2021-02-10T11:31:36.797 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:31:36.797 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_period": 0, 2021-02-10T11:31:36.797 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_count": 0, 2021-02-10T11:31:36.797 INFO:teuthology.orchestra.run.gibba002.stdout: "use_gmt_hitset": true, 2021-02-10T11:31:36.797 INFO:teuthology.orchestra.run.gibba002.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:31:36.797 INFO:teuthology.orchestra.run.gibba002.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:31:36.797 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:31:36.797 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:31:36.797 INFO:teuthology.orchestra.run.gibba002.stdout: "grade_table": [], 2021-02-10T11:31:36.797 INFO:teuthology.orchestra.run.gibba002.stdout: "stripe_width": 0, 2021-02-10T11:31:36.797 INFO:teuthology.orchestra.run.gibba002.stdout: "expected_num_objects": 0, 2021-02-10T11:31:36.798 INFO:teuthology.orchestra.run.gibba002.stdout: "fast_read": false, 2021-02-10T11:31:36.798 INFO:teuthology.orchestra.run.gibba002.stdout: "options": {}, 2021-02-10T11:31:36.798 INFO:teuthology.orchestra.run.gibba002.stdout: "application_metadata": { 2021-02-10T11:31:36.798 INFO:teuthology.orchestra.run.gibba002.stdout: "rbd": {} 2021-02-10T11:31:36.798 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:31:36.798 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:31:36.798 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:31:36.798 INFO:teuthology.orchestra.run.gibba002.stdout: "pool": 20, 2021-02-10T11:31:36.798 INFO:teuthology.orchestra.run.gibba002.stdout: "pool_name": "cephfs_metadata", 2021-02-10T11:31:36.798 INFO:teuthology.orchestra.run.gibba002.stdout: "create_time": "2021-02-10 11:30:56.015905", 2021-02-10T11:31:36.798 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 1, 2021-02-10T11:31:36.798 INFO:teuthology.orchestra.run.gibba002.stdout: "flags_names": "hashpspool", 2021-02-10T11:31:36.798 INFO:teuthology.orchestra.run.gibba002.stdout: "type": 1, 2021-02-10T11:31:36.798 INFO:teuthology.orchestra.run.gibba002.stdout: "size": 2, 2021-02-10T11:31:36.799 INFO:teuthology.orchestra.run.gibba002.stdout: "min_size": 1, 2021-02-10T11:31:36.799 INFO:teuthology.orchestra.run.gibba002.stdout: "crush_rule": 0, 2021-02-10T11:31:36.799 INFO:teuthology.orchestra.run.gibba002.stdout: "object_hash": 2, 2021-02-10T11:31:36.799 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:31:36.799 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num": 8, 2021-02-10T11:31:36.799 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_placement_num": 8, 2021-02-10T11:31:36.799 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_placement_num_target": 8, 2021-02-10T11:31:36.799 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num_target": 8, 2021-02-10T11:31:36.799 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num_pending": 8, 2021-02-10T11:31:36.799 INFO:teuthology.orchestra.run.gibba002.stdout: "last_pg_merge_meta": { 2021-02-10T11:31:36.799 INFO:teuthology.orchestra.run.gibba002.stdout: "source_pgid": "0.0", 2021-02-10T11:31:36.799 INFO:teuthology.orchestra.run.gibba002.stdout: "ready_epoch": 0, 2021-02-10T11:31:36.799 INFO:teuthology.orchestra.run.gibba002.stdout: "last_epoch_started": 0, 2021-02-10T11:31:36.799 INFO:teuthology.orchestra.run.gibba002.stdout: "last_epoch_clean": 0, 2021-02-10T11:31:36.800 INFO:teuthology.orchestra.run.gibba002.stdout: "source_version": "0'0", 2021-02-10T11:31:36.800 INFO:teuthology.orchestra.run.gibba002.stdout: "target_version": "0'0" 2021-02-10T11:31:36.800 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:31:36.800 INFO:teuthology.orchestra.run.gibba002.stdout: "last_change": "101", 2021-02-10T11:31:36.800 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend": "0", 2021-02-10T11:31:36.800 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:31:36.800 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:31:36.800 INFO:teuthology.orchestra.run.gibba002.stdout: "auid": 0, 2021-02-10T11:31:36.800 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:31:36.800 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_seq": 0, 2021-02-10T11:31:36.800 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_epoch": 0, 2021-02-10T11:31:36.800 INFO:teuthology.orchestra.run.gibba002.stdout: "pool_snaps": [], 2021-02-10T11:31:36.800 INFO:teuthology.orchestra.run.gibba002.stdout: "removed_snaps": "[]", 2021-02-10T11:31:36.800 INFO:teuthology.orchestra.run.gibba002.stdout: "quota_max_bytes": 0, 2021-02-10T11:31:36.801 INFO:teuthology.orchestra.run.gibba002.stdout: "quota_max_objects": 0, 2021-02-10T11:31:36.801 INFO:teuthology.orchestra.run.gibba002.stdout: "tiers": [], 2021-02-10T11:31:36.801 INFO:teuthology.orchestra.run.gibba002.stdout: "tier_of": -1, 2021-02-10T11:31:36.801 INFO:teuthology.orchestra.run.gibba002.stdout: "read_tier": -1, 2021-02-10T11:31:36.801 INFO:teuthology.orchestra.run.gibba002.stdout: "write_tier": -1, 2021-02-10T11:31:36.801 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_mode": "none", 2021-02-10T11:31:36.801 INFO:teuthology.orchestra.run.gibba002.stdout: "target_max_bytes": 0, 2021-02-10T11:31:36.801 INFO:teuthology.orchestra.run.gibba002.stdout: "target_max_objects": 0, 2021-02-10T11:31:36.801 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:31:36.801 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:31:36.801 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:31:36.801 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_min_flush_age": 0, 2021-02-10T11:31:36.801 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_min_evict_age": 0, 2021-02-10T11:31:36.802 INFO:teuthology.orchestra.run.gibba002.stdout: "erasure_code_profile": "", 2021-02-10T11:31:36.802 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_params": { 2021-02-10T11:31:36.802 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "none" 2021-02-10T11:31:36.802 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:31:36.802 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_period": 0, 2021-02-10T11:31:36.802 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_count": 0, 2021-02-10T11:31:36.802 INFO:teuthology.orchestra.run.gibba002.stdout: "use_gmt_hitset": true, 2021-02-10T11:31:36.802 INFO:teuthology.orchestra.run.gibba002.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:31:36.802 INFO:teuthology.orchestra.run.gibba002.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:31:36.802 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:31:36.802 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:31:36.802 INFO:teuthology.orchestra.run.gibba002.stdout: "grade_table": [], 2021-02-10T11:31:36.802 INFO:teuthology.orchestra.run.gibba002.stdout: "stripe_width": 0, 2021-02-10T11:31:36.802 INFO:teuthology.orchestra.run.gibba002.stdout: "expected_num_objects": 0, 2021-02-10T11:31:36.803 INFO:teuthology.orchestra.run.gibba002.stdout: "fast_read": false, 2021-02-10T11:31:36.803 INFO:teuthology.orchestra.run.gibba002.stdout: "options": { 2021-02-10T11:31:36.803 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_autoscale_bias": 4, 2021-02-10T11:31:36.803 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num_min": 16, 2021-02-10T11:31:36.803 INFO:teuthology.orchestra.run.gibba002.stdout: "recovery_priority": 5 2021-02-10T11:31:36.803 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:31:36.803 INFO:teuthology.orchestra.run.gibba002.stdout: "application_metadata": { 2021-02-10T11:31:36.803 INFO:teuthology.orchestra.run.gibba002.stdout: "cephfs": { 2021-02-10T11:31:36.803 INFO:teuthology.orchestra.run.gibba002.stdout: "metadata": "cephfs" 2021-02-10T11:31:36.803 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:31:36.803 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:31:36.803 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:31:36.803 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:31:36.804 INFO:teuthology.orchestra.run.gibba002.stdout: "pool": 21, 2021-02-10T11:31:36.804 INFO:teuthology.orchestra.run.gibba002.stdout: "pool_name": "cephfs_data", 2021-02-10T11:31:36.804 INFO:teuthology.orchestra.run.gibba002.stdout: "create_time": "2021-02-10 11:30:57.042942", 2021-02-10T11:31:36.804 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 1, 2021-02-10T11:31:36.804 INFO:teuthology.orchestra.run.gibba002.stdout: "flags_names": "hashpspool", 2021-02-10T11:31:36.804 INFO:teuthology.orchestra.run.gibba002.stdout: "type": 1, 2021-02-10T11:31:36.804 INFO:teuthology.orchestra.run.gibba002.stdout: "size": 2, 2021-02-10T11:31:36.804 INFO:teuthology.orchestra.run.gibba002.stdout: "min_size": 1, 2021-02-10T11:31:36.804 INFO:teuthology.orchestra.run.gibba002.stdout: "crush_rule": 0, 2021-02-10T11:31:36.804 INFO:teuthology.orchestra.run.gibba002.stdout: "object_hash": 2, 2021-02-10T11:31:36.804 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:31:36.804 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num": 8, 2021-02-10T11:31:36.804 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_placement_num": 8, 2021-02-10T11:31:36.805 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_placement_num_target": 8, 2021-02-10T11:31:36.805 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num_target": 8, 2021-02-10T11:31:36.805 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num_pending": 8, 2021-02-10T11:31:36.805 INFO:teuthology.orchestra.run.gibba002.stdout: "last_pg_merge_meta": { 2021-02-10T11:31:36.805 INFO:teuthology.orchestra.run.gibba002.stdout: "source_pgid": "0.0", 2021-02-10T11:31:36.805 INFO:teuthology.orchestra.run.gibba002.stdout: "ready_epoch": 0, 2021-02-10T11:31:36.805 INFO:teuthology.orchestra.run.gibba002.stdout: "last_epoch_started": 0, 2021-02-10T11:31:36.805 INFO:teuthology.orchestra.run.gibba002.stdout: "last_epoch_clean": 0, 2021-02-10T11:31:36.805 INFO:teuthology.orchestra.run.gibba002.stdout: "source_version": "0'0", 2021-02-10T11:31:36.805 INFO:teuthology.orchestra.run.gibba002.stdout: "target_version": "0'0" 2021-02-10T11:31:36.805 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:31:36.805 INFO:teuthology.orchestra.run.gibba002.stdout: "last_change": "103", 2021-02-10T11:31:36.805 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend": "0", 2021-02-10T11:31:36.805 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:31:36.806 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:31:36.806 INFO:teuthology.orchestra.run.gibba002.stdout: "auid": 0, 2021-02-10T11:31:36.806 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:31:36.806 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_seq": 0, 2021-02-10T11:31:36.806 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_epoch": 0, 2021-02-10T11:31:36.806 INFO:teuthology.orchestra.run.gibba002.stdout: "pool_snaps": [], 2021-02-10T11:31:36.806 INFO:teuthology.orchestra.run.gibba002.stdout: "removed_snaps": "[]", 2021-02-10T11:31:36.806 INFO:teuthology.orchestra.run.gibba002.stdout: "quota_max_bytes": 0, 2021-02-10T11:31:36.806 INFO:teuthology.orchestra.run.gibba002.stdout: "quota_max_objects": 0, 2021-02-10T11:31:36.806 INFO:teuthology.orchestra.run.gibba002.stdout: "tiers": [], 2021-02-10T11:31:36.806 INFO:teuthology.orchestra.run.gibba002.stdout: "tier_of": -1, 2021-02-10T11:31:36.806 INFO:teuthology.orchestra.run.gibba002.stdout: "read_tier": -1, 2021-02-10T11:31:36.806 INFO:teuthology.orchestra.run.gibba002.stdout: "write_tier": -1, 2021-02-10T11:31:36.806 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_mode": "none", 2021-02-10T11:31:36.807 INFO:teuthology.orchestra.run.gibba002.stdout: "target_max_bytes": 0, 2021-02-10T11:31:36.807 INFO:teuthology.orchestra.run.gibba002.stdout: "target_max_objects": 0, 2021-02-10T11:31:36.807 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:31:36.807 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:31:36.807 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:31:36.807 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_min_flush_age": 0, 2021-02-10T11:31:36.807 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_min_evict_age": 0, 2021-02-10T11:31:36.807 INFO:teuthology.orchestra.run.gibba002.stdout: "erasure_code_profile": "", 2021-02-10T11:31:36.807 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_params": { 2021-02-10T11:31:36.807 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "none" 2021-02-10T11:31:36.807 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:31:36.807 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_period": 0, 2021-02-10T11:31:36.807 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_count": 0, 2021-02-10T11:31:36.808 INFO:teuthology.orchestra.run.gibba002.stdout: "use_gmt_hitset": true, 2021-02-10T11:31:36.808 INFO:teuthology.orchestra.run.gibba002.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:31:36.808 INFO:teuthology.orchestra.run.gibba002.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:31:36.808 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:31:36.808 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:31:36.808 INFO:teuthology.orchestra.run.gibba002.stdout: "grade_table": [], 2021-02-10T11:31:36.808 INFO:teuthology.orchestra.run.gibba002.stdout: "stripe_width": 0, 2021-02-10T11:31:36.808 INFO:teuthology.orchestra.run.gibba002.stdout: "expected_num_objects": 0, 2021-02-10T11:31:36.808 INFO:teuthology.orchestra.run.gibba002.stdout: "fast_read": false, 2021-02-10T11:31:36.808 INFO:teuthology.orchestra.run.gibba002.stdout: "options": {}, 2021-02-10T11:31:36.808 INFO:teuthology.orchestra.run.gibba002.stdout: "application_metadata": { 2021-02-10T11:31:36.808 INFO:teuthology.orchestra.run.gibba002.stdout: "cephfs": { 2021-02-10T11:31:36.808 INFO:teuthology.orchestra.run.gibba002.stdout: "data": "cephfs" 2021-02-10T11:31:36.808 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:31:36.809 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:31:36.809 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:31:36.809 INFO:teuthology.orchestra.run.gibba002.stdout: ], 2021-02-10T11:31:36.809 INFO:teuthology.orchestra.run.gibba002.stdout: "osds": [ 2021-02-10T11:31:36.809 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:31:36.809 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 0, 2021-02-10T11:31:36.809 INFO:teuthology.orchestra.run.gibba002.stdout: "uuid": "be691b4d-a039-408a-af80-9d53f02b6f1f", 2021-02-10T11:31:36.809 INFO:teuthology.orchestra.run.gibba002.stdout: "up": 1, 2021-02-10T11:31:36.809 INFO:teuthology.orchestra.run.gibba002.stdout: "in": 1, 2021-02-10T11:31:36.809 INFO:teuthology.orchestra.run.gibba002.stdout: "weight": 1, 2021-02-10T11:31:36.809 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_affinity": 1, 2021-02-10T11:31:36.809 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_begin": 0, 2021-02-10T11:31:36.809 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_end": 0, 2021-02-10T11:31:36.809 INFO:teuthology.orchestra.run.gibba002.stdout: "up_from": 13, 2021-02-10T11:31:36.810 INFO:teuthology.orchestra.run.gibba002.stdout: "up_thru": 98, 2021-02-10T11:31:36.810 INFO:teuthology.orchestra.run.gibba002.stdout: "down_at": 0, 2021-02-10T11:31:36.810 INFO:teuthology.orchestra.run.gibba002.stdout: "lost_at": 0, 2021-02-10T11:31:36.810 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addrs": { 2021-02-10T11:31:36.810 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:31:36.810 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:31:36.810 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:31:36.810 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6818", 2021-02-10T11:31:36.810 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36088 2021-02-10T11:31:36.810 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:31:36.810 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:31:36.810 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:31:36.810 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6820", 2021-02-10T11:31:36.811 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36088 2021-02-10T11:31:36.811 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:31:36.811 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:31:36.811 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:31:36.811 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addrs": { 2021-02-10T11:31:36.811 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:31:36.811 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:31:36.811 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:31:36.811 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6822", 2021-02-10T11:31:36.811 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36088 2021-02-10T11:31:36.811 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:31:36.811 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:31:36.811 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:31:36.811 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6824", 2021-02-10T11:31:36.812 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36088 2021-02-10T11:31:36.812 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:31:36.812 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:31:36.812 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:31:36.812 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addrs": { 2021-02-10T11:31:36.812 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:31:36.812 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:31:36.812 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:31:36.812 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6830", 2021-02-10T11:31:36.812 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36088 2021-02-10T11:31:36.812 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:31:36.812 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:31:36.812 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:31:36.812 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6832", 2021-02-10T11:31:36.813 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36088 2021-02-10T11:31:36.813 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:31:36.813 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:31:36.813 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:31:36.813 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addrs": { 2021-02-10T11:31:36.813 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:31:36.813 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:31:36.813 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:31:36.813 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6826", 2021-02-10T11:31:36.813 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36088 2021-02-10T11:31:36.813 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:31:36.813 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:31:36.813 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:31:36.813 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6828", 2021-02-10T11:31:36.814 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36088 2021-02-10T11:31:36.814 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:31:36.814 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:31:36.814 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:31:36.814 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addr": "172.21.2.102:6820/36088", 2021-02-10T11:31:36.814 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addr": "172.21.2.102:6824/36088", 2021-02-10T11:31:36.814 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addr": "172.21.2.102:6832/36088", 2021-02-10T11:31:36.814 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addr": "172.21.2.102:6828/36088", 2021-02-10T11:31:36.814 INFO:teuthology.orchestra.run.gibba002.stdout: "state": [ 2021-02-10T11:31:36.814 INFO:teuthology.orchestra.run.gibba002.stdout: "exists", 2021-02-10T11:31:36.814 INFO:teuthology.orchestra.run.gibba002.stdout: "up" 2021-02-10T11:31:36.814 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:31:36.814 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:31:36.815 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:31:36.815 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 1, 2021-02-10T11:31:36.815 INFO:teuthology.orchestra.run.gibba002.stdout: "uuid": "64e3ccc8-9c69-4d7c-865e-af8e83822635", 2021-02-10T11:31:36.815 INFO:teuthology.orchestra.run.gibba002.stdout: "up": 1, 2021-02-10T11:31:36.815 INFO:teuthology.orchestra.run.gibba002.stdout: "in": 1, 2021-02-10T11:31:36.815 INFO:teuthology.orchestra.run.gibba002.stdout: "weight": 1, 2021-02-10T11:31:36.815 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_affinity": 1, 2021-02-10T11:31:36.815 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_begin": 0, 2021-02-10T11:31:36.815 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_end": 0, 2021-02-10T11:31:36.815 INFO:teuthology.orchestra.run.gibba002.stdout: "up_from": 13, 2021-02-10T11:31:36.815 INFO:teuthology.orchestra.run.gibba002.stdout: "up_thru": 100, 2021-02-10T11:31:36.815 INFO:teuthology.orchestra.run.gibba002.stdout: "down_at": 0, 2021-02-10T11:31:36.815 INFO:teuthology.orchestra.run.gibba002.stdout: "lost_at": 0, 2021-02-10T11:31:36.815 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addrs": { 2021-02-10T11:31:36.816 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:31:36.816 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:31:36.816 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:31:36.816 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6819", 2021-02-10T11:31:36.816 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36087 2021-02-10T11:31:36.816 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:31:36.816 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:31:36.816 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:31:36.816 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6821", 2021-02-10T11:31:36.816 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36087 2021-02-10T11:31:36.816 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:31:36.816 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:31:36.816 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:31:36.816 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addrs": { 2021-02-10T11:31:36.817 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:31:36.817 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:31:36.817 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:31:36.817 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6823", 2021-02-10T11:31:36.817 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36087 2021-02-10T11:31:36.817 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:31:36.817 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:31:36.817 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:31:36.817 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6825", 2021-02-10T11:31:36.817 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36087 2021-02-10T11:31:36.817 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:31:36.817 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:31:36.818 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:31:36.818 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addrs": { 2021-02-10T11:31:36.818 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:31:36.818 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:31:36.818 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:31:36.818 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6831", 2021-02-10T11:31:36.818 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36087 2021-02-10T11:31:36.818 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:31:36.818 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:31:36.818 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:31:36.818 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6833", 2021-02-10T11:31:36.818 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36087 2021-02-10T11:31:36.819 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:31:36.819 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:31:36.819 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:31:36.819 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addrs": { 2021-02-10T11:31:36.819 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:31:36.819 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:31:36.819 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:31:36.819 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6827", 2021-02-10T11:31:36.819 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36087 2021-02-10T11:31:36.819 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:31:36.819 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:31:36.819 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:31:36.819 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6829", 2021-02-10T11:31:36.819 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36087 2021-02-10T11:31:36.820 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:31:36.820 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:31:36.820 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:31:36.820 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addr": "172.21.2.102:6821/36087", 2021-02-10T11:31:36.820 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addr": "172.21.2.102:6825/36087", 2021-02-10T11:31:36.820 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addr": "172.21.2.102:6833/36087", 2021-02-10T11:31:36.820 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addr": "172.21.2.102:6829/36087", 2021-02-10T11:31:36.820 INFO:teuthology.orchestra.run.gibba002.stdout: "state": [ 2021-02-10T11:31:36.820 INFO:teuthology.orchestra.run.gibba002.stdout: "exists", 2021-02-10T11:31:36.820 INFO:teuthology.orchestra.run.gibba002.stdout: "up" 2021-02-10T11:31:36.820 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:31:36.820 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:31:36.820 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:31:36.821 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 2, 2021-02-10T11:31:36.821 INFO:teuthology.orchestra.run.gibba002.stdout: "uuid": "153700af-747f-481f-bdf4-d6072d3a842f", 2021-02-10T11:31:36.821 INFO:teuthology.orchestra.run.gibba002.stdout: "up": 1, 2021-02-10T11:31:36.821 INFO:teuthology.orchestra.run.gibba002.stdout: "in": 1, 2021-02-10T11:31:36.821 INFO:teuthology.orchestra.run.gibba002.stdout: "weight": 1, 2021-02-10T11:31:36.821 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_affinity": 1, 2021-02-10T11:31:36.821 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_begin": 0, 2021-02-10T11:31:36.821 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_end": 0, 2021-02-10T11:31:36.821 INFO:teuthology.orchestra.run.gibba002.stdout: "up_from": 13, 2021-02-10T11:31:36.821 INFO:teuthology.orchestra.run.gibba002.stdout: "up_thru": 100, 2021-02-10T11:31:36.821 INFO:teuthology.orchestra.run.gibba002.stdout: "down_at": 0, 2021-02-10T11:31:36.821 INFO:teuthology.orchestra.run.gibba002.stdout: "lost_at": 0, 2021-02-10T11:31:36.821 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addrs": { 2021-02-10T11:31:36.821 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:31:36.822 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:31:36.822 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:31:36.822 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6810", 2021-02-10T11:31:36.822 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36089 2021-02-10T11:31:36.822 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:31:36.822 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:31:36.822 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:31:36.822 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6811", 2021-02-10T11:31:36.822 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36089 2021-02-10T11:31:36.822 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:31:36.822 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:31:36.822 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:31:36.822 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addrs": { 2021-02-10T11:31:36.822 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:31:36.823 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:31:36.823 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:31:36.823 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6812", 2021-02-10T11:31:36.823 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36089 2021-02-10T11:31:36.823 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:31:36.823 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:31:36.823 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:31:36.823 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6813", 2021-02-10T11:31:36.823 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36089 2021-02-10T11:31:36.823 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:31:36.823 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:31:36.823 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:31:36.823 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addrs": { 2021-02-10T11:31:36.823 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:31:36.824 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:31:36.824 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:31:36.824 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6816", 2021-02-10T11:31:36.824 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36089 2021-02-10T11:31:36.824 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:31:36.824 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:31:36.824 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:31:36.824 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6817", 2021-02-10T11:31:36.824 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36089 2021-02-10T11:31:36.824 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:31:36.824 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:31:36.824 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:31:36.824 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addrs": { 2021-02-10T11:31:36.824 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:31:36.825 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:31:36.825 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:31:36.825 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6814", 2021-02-10T11:31:36.825 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36089 2021-02-10T11:31:36.825 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:31:36.825 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:31:36.825 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:31:36.825 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6815", 2021-02-10T11:31:36.825 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36089 2021-02-10T11:31:36.825 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:31:36.825 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:31:36.825 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:31:36.825 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addr": "172.21.2.102:6811/36089", 2021-02-10T11:31:36.826 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addr": "172.21.2.102:6813/36089", 2021-02-10T11:31:36.826 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addr": "172.21.2.102:6817/36089", 2021-02-10T11:31:36.826 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addr": "172.21.2.102:6815/36089", 2021-02-10T11:31:36.826 INFO:teuthology.orchestra.run.gibba002.stdout: "state": [ 2021-02-10T11:31:36.826 INFO:teuthology.orchestra.run.gibba002.stdout: "exists", 2021-02-10T11:31:36.826 INFO:teuthology.orchestra.run.gibba002.stdout: "up" 2021-02-10T11:31:36.826 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:31:36.826 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:31:36.826 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:31:36.826 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 3, 2021-02-10T11:31:36.826 INFO:teuthology.orchestra.run.gibba002.stdout: "uuid": "cbc678f4-83a9-45e6-8830-2f7c232b1ea7", 2021-02-10T11:31:36.826 INFO:teuthology.orchestra.run.gibba002.stdout: "up": 1, 2021-02-10T11:31:36.826 INFO:teuthology.orchestra.run.gibba002.stdout: "in": 1, 2021-02-10T11:31:36.826 INFO:teuthology.orchestra.run.gibba002.stdout: "weight": 1, 2021-02-10T11:31:36.827 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_affinity": 1, 2021-02-10T11:31:36.827 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_begin": 0, 2021-02-10T11:31:36.827 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_end": 0, 2021-02-10T11:31:36.827 INFO:teuthology.orchestra.run.gibba002.stdout: "up_from": 13, 2021-02-10T11:31:36.827 INFO:teuthology.orchestra.run.gibba002.stdout: "up_thru": 101, 2021-02-10T11:31:36.827 INFO:teuthology.orchestra.run.gibba002.stdout: "down_at": 0, 2021-02-10T11:31:36.827 INFO:teuthology.orchestra.run.gibba002.stdout: "lost_at": 0, 2021-02-10T11:31:36.827 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addrs": { 2021-02-10T11:31:36.827 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:31:36.827 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:31:36.827 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:31:36.827 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6802", 2021-02-10T11:31:36.827 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36090 2021-02-10T11:31:36.828 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:31:36.828 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:31:36.828 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:31:36.828 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6803", 2021-02-10T11:31:36.828 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36090 2021-02-10T11:31:36.828 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:31:36.828 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:31:36.828 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:31:36.828 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addrs": { 2021-02-10T11:31:36.828 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:31:36.828 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:31:36.828 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:31:36.828 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6804", 2021-02-10T11:31:36.828 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36090 2021-02-10T11:31:36.829 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:31:36.829 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:31:36.829 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:31:36.829 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6805", 2021-02-10T11:31:36.829 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36090 2021-02-10T11:31:36.829 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:31:36.829 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:31:36.829 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:31:36.829 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addrs": { 2021-02-10T11:31:36.829 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:31:36.829 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:31:36.829 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:31:36.829 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6808", 2021-02-10T11:31:36.830 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36090 2021-02-10T11:31:36.830 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:31:36.830 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:31:36.830 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:31:36.830 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6809", 2021-02-10T11:31:36.830 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36090 2021-02-10T11:31:36.830 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:31:36.830 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:31:36.830 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:31:36.830 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addrs": { 2021-02-10T11:31:36.830 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:31:36.830 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:31:36.830 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:31:36.830 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6806", 2021-02-10T11:31:36.831 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36090 2021-02-10T11:31:36.831 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:31:36.831 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:31:36.831 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:31:36.831 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6807", 2021-02-10T11:31:36.831 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36090 2021-02-10T11:31:36.831 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:31:36.831 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:31:36.831 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:31:36.831 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addr": "172.21.2.102:6803/36090", 2021-02-10T11:31:36.831 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addr": "172.21.2.102:6805/36090", 2021-02-10T11:31:36.831 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addr": "172.21.2.102:6809/36090", 2021-02-10T11:31:36.831 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addr": "172.21.2.102:6807/36090", 2021-02-10T11:31:36.832 INFO:teuthology.orchestra.run.gibba002.stdout: "state": [ 2021-02-10T11:31:36.832 INFO:teuthology.orchestra.run.gibba002.stdout: "exists", 2021-02-10T11:31:36.832 INFO:teuthology.orchestra.run.gibba002.stdout: "up" 2021-02-10T11:31:36.832 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:31:36.832 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:31:36.832 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:31:36.832 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 4, 2021-02-10T11:31:36.832 INFO:teuthology.orchestra.run.gibba002.stdout: "uuid": "c01e4008-b258-4f10-afd2-9e7f37214351", 2021-02-10T11:31:36.832 INFO:teuthology.orchestra.run.gibba002.stdout: "up": 1, 2021-02-10T11:31:36.832 INFO:teuthology.orchestra.run.gibba002.stdout: "in": 1, 2021-02-10T11:31:36.832 INFO:teuthology.orchestra.run.gibba002.stdout: "weight": 1, 2021-02-10T11:31:36.832 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_affinity": 1, 2021-02-10T11:31:36.832 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_begin": 0, 2021-02-10T11:31:36.832 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_end": 0, 2021-02-10T11:31:36.833 INFO:teuthology.orchestra.run.gibba002.stdout: "up_from": 13, 2021-02-10T11:31:36.833 INFO:teuthology.orchestra.run.gibba002.stdout: "up_thru": 91, 2021-02-10T11:31:36.833 INFO:teuthology.orchestra.run.gibba002.stdout: "down_at": 0, 2021-02-10T11:31:36.833 INFO:teuthology.orchestra.run.gibba002.stdout: "lost_at": 0, 2021-02-10T11:31:36.833 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addrs": { 2021-02-10T11:31:36.833 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:31:36.833 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:31:36.833 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:31:36.833 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6808", 2021-02-10T11:31:36.833 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36368 2021-02-10T11:31:36.833 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:31:36.833 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:31:36.833 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:31:36.833 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6811", 2021-02-10T11:31:36.834 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36368 2021-02-10T11:31:36.834 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:31:36.834 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:31:36.834 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:31:36.834 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addrs": { 2021-02-10T11:31:36.834 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:31:36.834 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:31:36.834 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:31:36.834 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6814", 2021-02-10T11:31:36.834 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36368 2021-02-10T11:31:36.834 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:31:36.834 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:31:36.834 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:31:36.835 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6818", 2021-02-10T11:31:36.835 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36368 2021-02-10T11:31:36.835 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:31:36.835 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:31:36.835 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:31:36.835 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addrs": { 2021-02-10T11:31:36.835 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:31:36.835 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:31:36.835 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:31:36.835 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6826", 2021-02-10T11:31:36.835 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36368 2021-02-10T11:31:36.835 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:31:36.835 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:31:36.835 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:31:36.836 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6828", 2021-02-10T11:31:36.836 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36368 2021-02-10T11:31:36.836 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:31:36.836 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:31:36.836 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:31:36.836 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addrs": { 2021-02-10T11:31:36.836 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:31:36.836 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:31:36.836 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:31:36.836 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6820", 2021-02-10T11:31:36.836 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36368 2021-02-10T11:31:36.836 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:31:36.836 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:31:36.837 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:31:36.837 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6824", 2021-02-10T11:31:36.837 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36368 2021-02-10T11:31:36.837 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:31:36.837 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:31:36.837 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:31:36.837 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addr": "172.21.2.113:6811/36368", 2021-02-10T11:31:36.837 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addr": "172.21.2.113:6818/36368", 2021-02-10T11:31:36.837 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addr": "172.21.2.113:6828/36368", 2021-02-10T11:31:36.837 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addr": "172.21.2.113:6824/36368", 2021-02-10T11:31:36.837 INFO:teuthology.orchestra.run.gibba002.stdout: "state": [ 2021-02-10T11:31:36.837 INFO:teuthology.orchestra.run.gibba002.stdout: "exists", 2021-02-10T11:31:36.837 INFO:teuthology.orchestra.run.gibba002.stdout: "up" 2021-02-10T11:31:36.837 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:31:36.838 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:31:36.838 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:31:36.838 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 5, 2021-02-10T11:31:36.838 INFO:teuthology.orchestra.run.gibba002.stdout: "uuid": "551f6007-daf8-4a98-8ee0-c2b16bcccb5e", 2021-02-10T11:31:36.838 INFO:teuthology.orchestra.run.gibba002.stdout: "up": 1, 2021-02-10T11:31:36.838 INFO:teuthology.orchestra.run.gibba002.stdout: "in": 1, 2021-02-10T11:31:36.838 INFO:teuthology.orchestra.run.gibba002.stdout: "weight": 1, 2021-02-10T11:31:36.838 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_affinity": 1, 2021-02-10T11:31:36.838 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_begin": 0, 2021-02-10T11:31:36.838 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_end": 0, 2021-02-10T11:31:36.838 INFO:teuthology.orchestra.run.gibba002.stdout: "up_from": 13, 2021-02-10T11:31:36.838 INFO:teuthology.orchestra.run.gibba002.stdout: "up_thru": 101, 2021-02-10T11:31:36.838 INFO:teuthology.orchestra.run.gibba002.stdout: "down_at": 0, 2021-02-10T11:31:36.839 INFO:teuthology.orchestra.run.gibba002.stdout: "lost_at": 0, 2021-02-10T11:31:36.839 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addrs": { 2021-02-10T11:31:36.839 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:31:36.839 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:31:36.839 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:31:36.839 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6802", 2021-02-10T11:31:36.839 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36366 2021-02-10T11:31:36.839 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:31:36.839 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:31:36.839 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:31:36.839 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6805", 2021-02-10T11:31:36.839 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36366 2021-02-10T11:31:36.839 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:31:36.839 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:31:36.840 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:31:36.840 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addrs": { 2021-02-10T11:31:36.840 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:31:36.840 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:31:36.840 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:31:36.840 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6807", 2021-02-10T11:31:36.840 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36366 2021-02-10T11:31:36.840 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:31:36.840 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:31:36.840 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:31:36.840 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6810", 2021-02-10T11:31:36.840 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36366 2021-02-10T11:31:36.840 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:31:36.840 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:31:36.841 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:31:36.841 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addrs": { 2021-02-10T11:31:36.841 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:31:36.841 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:31:36.841 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:31:36.841 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6819", 2021-02-10T11:31:36.841 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36366 2021-02-10T11:31:36.841 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:31:36.841 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:31:36.841 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:31:36.841 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6822", 2021-02-10T11:31:36.841 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36366 2021-02-10T11:31:36.841 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:31:36.841 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:31:36.842 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:31:36.842 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addrs": { 2021-02-10T11:31:36.842 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:31:36.842 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:31:36.842 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:31:36.842 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6812", 2021-02-10T11:31:36.842 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36366 2021-02-10T11:31:36.842 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:31:36.842 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:31:36.842 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:31:36.842 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6815", 2021-02-10T11:31:36.842 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36366 2021-02-10T11:31:36.843 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:31:36.843 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:31:36.843 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:31:36.843 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addr": "172.21.2.113:6805/36366", 2021-02-10T11:31:36.843 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addr": "172.21.2.113:6810/36366", 2021-02-10T11:31:36.843 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addr": "172.21.2.113:6822/36366", 2021-02-10T11:31:36.843 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addr": "172.21.2.113:6815/36366", 2021-02-10T11:31:36.843 INFO:teuthology.orchestra.run.gibba002.stdout: "state": [ 2021-02-10T11:31:36.843 INFO:teuthology.orchestra.run.gibba002.stdout: "exists", 2021-02-10T11:31:36.843 INFO:teuthology.orchestra.run.gibba002.stdout: "up" 2021-02-10T11:31:36.843 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:31:36.843 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:31:36.843 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:31:36.843 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 6, 2021-02-10T11:31:36.844 INFO:teuthology.orchestra.run.gibba002.stdout: "uuid": "bf67c040-0e50-48be-b102-c3943bc9ebef", 2021-02-10T11:31:36.844 INFO:teuthology.orchestra.run.gibba002.stdout: "up": 1, 2021-02-10T11:31:36.844 INFO:teuthology.orchestra.run.gibba002.stdout: "in": 1, 2021-02-10T11:31:36.844 INFO:teuthology.orchestra.run.gibba002.stdout: "weight": 1, 2021-02-10T11:31:36.844 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_affinity": 1, 2021-02-10T11:31:36.844 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_begin": 0, 2021-02-10T11:31:36.844 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_end": 0, 2021-02-10T11:31:36.844 INFO:teuthology.orchestra.run.gibba002.stdout: "up_from": 13, 2021-02-10T11:31:36.844 INFO:teuthology.orchestra.run.gibba002.stdout: "up_thru": 100, 2021-02-10T11:31:36.844 INFO:teuthology.orchestra.run.gibba002.stdout: "down_at": 0, 2021-02-10T11:31:36.844 INFO:teuthology.orchestra.run.gibba002.stdout: "lost_at": 0, 2021-02-10T11:31:36.844 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addrs": { 2021-02-10T11:31:36.844 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:31:36.845 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:31:36.845 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:31:36.845 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6800", 2021-02-10T11:31:36.845 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36367 2021-02-10T11:31:36.845 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:31:36.845 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:31:36.845 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:31:36.845 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6801", 2021-02-10T11:31:36.845 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36367 2021-02-10T11:31:36.845 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:31:36.845 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:31:36.845 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:31:36.845 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addrs": { 2021-02-10T11:31:36.845 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:31:36.846 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:31:36.846 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:31:36.846 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6803", 2021-02-10T11:31:36.846 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36367 2021-02-10T11:31:36.846 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:31:36.846 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:31:36.846 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:31:36.846 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6804", 2021-02-10T11:31:36.846 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36367 2021-02-10T11:31:36.846 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:31:36.846 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:31:36.846 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:31:36.846 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addrs": { 2021-02-10T11:31:36.846 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:31:36.847 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:31:36.847 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:31:36.847 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6813", 2021-02-10T11:31:36.847 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36367 2021-02-10T11:31:36.847 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:31:36.847 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:31:36.847 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:31:36.847 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6816", 2021-02-10T11:31:36.847 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36367 2021-02-10T11:31:36.847 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:31:36.847 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:31:36.847 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:31:36.847 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addrs": { 2021-02-10T11:31:36.848 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:31:36.848 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:31:36.848 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:31:36.848 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6806", 2021-02-10T11:31:36.848 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36367 2021-02-10T11:31:36.848 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:31:36.848 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:31:36.848 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:31:36.848 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6809", 2021-02-10T11:31:36.848 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36367 2021-02-10T11:31:36.848 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:31:36.848 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:31:36.848 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:31:36.848 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addr": "172.21.2.113:6801/36367", 2021-02-10T11:31:36.849 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addr": "172.21.2.113:6804/36367", 2021-02-10T11:31:36.849 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addr": "172.21.2.113:6816/36367", 2021-02-10T11:31:36.849 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addr": "172.21.2.113:6809/36367", 2021-02-10T11:31:36.849 INFO:teuthology.orchestra.run.gibba002.stdout: "state": [ 2021-02-10T11:31:36.849 INFO:teuthology.orchestra.run.gibba002.stdout: "exists", 2021-02-10T11:31:36.849 INFO:teuthology.orchestra.run.gibba002.stdout: "up" 2021-02-10T11:31:36.849 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:31:36.849 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:31:36.849 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:31:36.849 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 7, 2021-02-10T11:31:36.849 INFO:teuthology.orchestra.run.gibba002.stdout: "uuid": "f5461f80-018d-428f-be89-2eb2bcbe7bb6", 2021-02-10T11:31:36.849 INFO:teuthology.orchestra.run.gibba002.stdout: "up": 1, 2021-02-10T11:31:36.849 INFO:teuthology.orchestra.run.gibba002.stdout: "in": 1, 2021-02-10T11:31:36.850 INFO:teuthology.orchestra.run.gibba002.stdout: "weight": 1, 2021-02-10T11:31:36.850 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_affinity": 1, 2021-02-10T11:31:36.850 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_begin": 0, 2021-02-10T11:31:36.850 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_end": 0, 2021-02-10T11:31:36.850 INFO:teuthology.orchestra.run.gibba002.stdout: "up_from": 13, 2021-02-10T11:31:36.850 INFO:teuthology.orchestra.run.gibba002.stdout: "up_thru": 100, 2021-02-10T11:31:36.850 INFO:teuthology.orchestra.run.gibba002.stdout: "down_at": 0, 2021-02-10T11:31:36.850 INFO:teuthology.orchestra.run.gibba002.stdout: "lost_at": 0, 2021-02-10T11:31:36.850 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addrs": { 2021-02-10T11:31:36.850 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:31:36.850 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:31:36.850 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:31:36.850 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6817", 2021-02-10T11:31:36.850 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36365 2021-02-10T11:31:36.851 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:31:36.851 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:31:36.851 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:31:36.851 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6821", 2021-02-10T11:31:36.851 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36365 2021-02-10T11:31:36.851 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:31:36.851 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:31:36.851 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:31:36.851 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addrs": { 2021-02-10T11:31:36.851 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:31:36.851 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:31:36.851 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:31:36.851 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6823", 2021-02-10T11:31:36.852 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36365 2021-02-10T11:31:36.852 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:31:36.852 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:31:36.852 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:31:36.852 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6825", 2021-02-10T11:31:36.852 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36365 2021-02-10T11:31:36.852 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:31:36.852 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:31:36.852 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:31:36.852 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addrs": { 2021-02-10T11:31:36.852 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:31:36.852 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:31:36.852 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:31:36.852 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6830", 2021-02-10T11:31:36.852 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36365 2021-02-10T11:31:36.853 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:31:36.853 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:31:36.853 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:31:36.853 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6831", 2021-02-10T11:31:36.853 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36365 2021-02-10T11:31:36.853 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:31:36.853 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:31:36.853 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:31:36.853 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addrs": { 2021-02-10T11:31:36.853 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:31:36.853 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:31:36.853 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:31:36.853 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6827", 2021-02-10T11:31:36.854 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36365 2021-02-10T11:31:36.854 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:31:36.854 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:31:36.854 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:31:36.854 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6829", 2021-02-10T11:31:36.854 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36365 2021-02-10T11:31:36.854 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:31:36.854 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:31:36.854 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:31:36.854 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addr": "172.21.2.113:6821/36365", 2021-02-10T11:31:36.854 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addr": "172.21.2.113:6825/36365", 2021-02-10T11:31:36.854 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addr": "172.21.2.113:6831/36365", 2021-02-10T11:31:36.854 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addr": "172.21.2.113:6829/36365", 2021-02-10T11:31:36.854 INFO:teuthology.orchestra.run.gibba002.stdout: "state": [ 2021-02-10T11:31:36.855 INFO:teuthology.orchestra.run.gibba002.stdout: "exists", 2021-02-10T11:31:36.855 INFO:teuthology.orchestra.run.gibba002.stdout: "up" 2021-02-10T11:31:36.855 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:31:36.855 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:31:36.855 INFO:teuthology.orchestra.run.gibba002.stdout: ], 2021-02-10T11:31:36.855 INFO:teuthology.orchestra.run.gibba002.stdout: "osd_xinfo": [ 2021-02-10T11:31:36.855 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:31:36.855 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 0, 2021-02-10T11:31:36.855 INFO:teuthology.orchestra.run.gibba002.stdout: "down_stamp": "0.000000", 2021-02-10T11:31:36.855 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_probability": 0, 2021-02-10T11:31:36.855 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_interval": 0, 2021-02-10T11:31:36.855 INFO:teuthology.orchestra.run.gibba002.stdout: "features": 4611087854035861503, 2021-02-10T11:31:36.855 INFO:teuthology.orchestra.run.gibba002.stdout: "old_weight": 0 2021-02-10T11:31:36.855 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:31:36.856 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:31:36.856 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 1, 2021-02-10T11:31:36.856 INFO:teuthology.orchestra.run.gibba002.stdout: "down_stamp": "0.000000", 2021-02-10T11:31:36.856 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_probability": 0, 2021-02-10T11:31:36.856 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_interval": 0, 2021-02-10T11:31:36.856 INFO:teuthology.orchestra.run.gibba002.stdout: "features": 4611087854035861503, 2021-02-10T11:31:36.856 INFO:teuthology.orchestra.run.gibba002.stdout: "old_weight": 0 2021-02-10T11:31:36.856 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:31:36.856 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:31:36.856 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 2, 2021-02-10T11:31:36.856 INFO:teuthology.orchestra.run.gibba002.stdout: "down_stamp": "0.000000", 2021-02-10T11:31:36.856 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_probability": 0, 2021-02-10T11:31:36.856 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_interval": 0, 2021-02-10T11:31:36.856 INFO:teuthology.orchestra.run.gibba002.stdout: "features": 4611087854035861503, 2021-02-10T11:31:36.857 INFO:teuthology.orchestra.run.gibba002.stdout: "old_weight": 0 2021-02-10T11:31:36.857 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:31:36.857 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:31:36.857 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 3, 2021-02-10T11:31:36.857 INFO:teuthology.orchestra.run.gibba002.stdout: "down_stamp": "0.000000", 2021-02-10T11:31:36.857 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_probability": 0, 2021-02-10T11:31:36.857 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_interval": 0, 2021-02-10T11:31:36.857 INFO:teuthology.orchestra.run.gibba002.stdout: "features": 4611087854035861503, 2021-02-10T11:31:36.857 INFO:teuthology.orchestra.run.gibba002.stdout: "old_weight": 0 2021-02-10T11:31:36.857 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:31:36.857 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:31:36.857 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 4, 2021-02-10T11:31:36.857 INFO:teuthology.orchestra.run.gibba002.stdout: "down_stamp": "0.000000", 2021-02-10T11:31:36.857 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_probability": 0, 2021-02-10T11:31:36.858 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_interval": 0, 2021-02-10T11:31:36.858 INFO:teuthology.orchestra.run.gibba002.stdout: "features": 4611087854035861503, 2021-02-10T11:31:36.858 INFO:teuthology.orchestra.run.gibba002.stdout: "old_weight": 0 2021-02-10T11:31:36.858 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:31:36.858 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:31:36.858 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 5, 2021-02-10T11:31:36.858 INFO:teuthology.orchestra.run.gibba002.stdout: "down_stamp": "0.000000", 2021-02-10T11:31:36.858 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_probability": 0, 2021-02-10T11:31:36.858 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_interval": 0, 2021-02-10T11:31:36.858 INFO:teuthology.orchestra.run.gibba002.stdout: "features": 4611087854035861503, 2021-02-10T11:31:36.858 INFO:teuthology.orchestra.run.gibba002.stdout: "old_weight": 0 2021-02-10T11:31:36.858 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:31:36.859 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:31:36.859 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 6, 2021-02-10T11:31:36.859 INFO:teuthology.orchestra.run.gibba002.stdout: "down_stamp": "0.000000", 2021-02-10T11:31:36.859 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_probability": 0, 2021-02-10T11:31:36.859 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_interval": 0, 2021-02-10T11:31:36.859 INFO:teuthology.orchestra.run.gibba002.stdout: "features": 4611087854035861503, 2021-02-10T11:31:36.859 INFO:teuthology.orchestra.run.gibba002.stdout: "old_weight": 0 2021-02-10T11:31:36.859 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:31:36.859 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:31:36.859 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 7, 2021-02-10T11:31:36.859 INFO:teuthology.orchestra.run.gibba002.stdout: "down_stamp": "0.000000", 2021-02-10T11:31:36.859 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_probability": 0, 2021-02-10T11:31:36.859 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_interval": 0, 2021-02-10T11:31:36.859 INFO:teuthology.orchestra.run.gibba002.stdout: "features": 4611087854035861503, 2021-02-10T11:31:36.860 INFO:teuthology.orchestra.run.gibba002.stdout: "old_weight": 0 2021-02-10T11:31:36.860 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:31:36.860 INFO:teuthology.orchestra.run.gibba002.stdout: ], 2021-02-10T11:31:36.860 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_upmap": [], 2021-02-10T11:31:36.860 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_upmap_items": [], 2021-02-10T11:31:36.860 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_temp": [], 2021-02-10T11:31:36.860 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_temp": [], 2021-02-10T11:31:36.860 INFO:teuthology.orchestra.run.gibba002.stdout: "blacklist": {}, 2021-02-10T11:31:36.860 INFO:teuthology.orchestra.run.gibba002.stdout: "erasure_code_profiles": { 2021-02-10T11:31:36.860 INFO:teuthology.orchestra.run.gibba002.stdout: "default": { 2021-02-10T11:31:36.860 INFO:teuthology.orchestra.run.gibba002.stdout: "crush-failure-domain": "osd", 2021-02-10T11:31:36.860 INFO:teuthology.orchestra.run.gibba002.stdout: "k": "2", 2021-02-10T11:31:36.860 INFO:teuthology.orchestra.run.gibba002.stdout: "m": "1", 2021-02-10T11:31:36.860 INFO:teuthology.orchestra.run.gibba002.stdout: "plugin": "jerasure", 2021-02-10T11:31:36.861 INFO:teuthology.orchestra.run.gibba002.stdout: "ruleset-failure-domain": "osd", 2021-02-10T11:31:36.861 INFO:teuthology.orchestra.run.gibba002.stdout: "technique": "reed_sol_van" 2021-02-10T11:31:36.861 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:31:36.861 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:31:36.861 INFO:teuthology.orchestra.run.gibba002.stdout: "removed_snaps_queue": [], 2021-02-10T11:31:36.861 INFO:teuthology.orchestra.run.gibba002.stdout: "new_removed_snaps": [], 2021-02-10T11:31:36.861 INFO:teuthology.orchestra.run.gibba002.stdout: "new_purged_snaps": [], 2021-02-10T11:31:36.861 INFO:teuthology.orchestra.run.gibba002.stdout: "crush_node_flags": {}, 2021-02-10T11:31:36.861 INFO:teuthology.orchestra.run.gibba002.stdout: "device_class_flags": {} 2021-02-10T11:31:36.861 INFO:teuthology.orchestra.run.gibba002.stdout:} 2021-02-10T11:31:36.862 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:36.791 7f0d2f9df700 1 -- 172.21.2.102:0/1917446514 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0d08018700 msgr2=0x7f0d0801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:36.863 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:36.791 7f0d2f9df700 1 --2- 172.21.2.102:0/1917446514 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0d08018700 0x7f0d0801abc0 crc :-1 s=READY pgs=949 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:36.863 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:36.791 7f0d2f9df700 1 -- 172.21.2.102:0/1917446514 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0d280b5b80 msgr2=0x7f0d281a2e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:36.863 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:36.791 7f0d2f9df700 1 --2- 172.21.2.102:0/1917446514 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0d280b5b80 0x7f0d281a2e20 secure :-1 s=READY pgs=2142 cs=0 l=1 rev1=1 rx=0x7f0d1c006f20 tx=0x7f0d1c00b9e0).stop 2021-02-10T11:31:36.863 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:36.791 7f0d2f9df700 1 -- 172.21.2.102:0/1917446514 shutdown_connections 2021-02-10T11:31:36.863 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:36.791 7f0d2f9df700 1 --2- 172.21.2.102:0/1917446514 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0d280acfa0 0x7f0d281a3440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:36.863 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:36.791 7f0d2f9df700 1 --2- 172.21.2.102:0/1917446514 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0d280b5b80 0x7f0d281a2e20 unknown :-1 s=CLOSED pgs=2142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:36.863 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:36.791 7f0d2f9df700 1 --2- 172.21.2.102:0/1917446514 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0d08018700 0x7f0d0801abc0 unknown :-1 s=CLOSED pgs=949 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:36.863 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:36.791 7f0d2f9df700 1 --2- 172.21.2.102:0/1917446514 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0d28195650 0x7f0d280b5560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:36.863 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:36.791 7f0d2f9df700 1 -- 172.21.2.102:0/1917446514 >> 172.21.2.102:0/1917446514 conn(0x7f0d280b3040 msgr2=0x7f0d280c3250 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:36.863 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:36.791 7f0d2f9df700 1 -- 172.21.2.102:0/1917446514 shutdown_connections 2021-02-10T11:31:36.863 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:36.791 7f0d2f9df700 1 -- 172.21.2.102:0/1917446514 wait complete. 2021-02-10T11:31:36.864 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:31:37.291 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:37.273 7f1d85979700 1 Processor -- start 2021-02-10T11:31:37.291 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:37.273 7f1d85979700 1 -- start start 2021-02-10T11:31:37.291 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:37.273 7f1d85979700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1d800d1870 0x7f1d801b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:37.291 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:37.273 7f1d85979700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d800f0b10 con 0x7f1d801b0a10 2021-02-10T11:31:37.292 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:37.273 7f1d85979700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d800d0310 con 0x7f1d800f0650 2021-02-10T11:31:37.292 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:37.273 7f1d85979700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d800e5ed0 con 0x7f1d800d1870 2021-02-10T11:31:37.292 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:37.273 7f1d7ed73700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f1d801b0a10 0x7f1d800cf330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:54188/0 (socket says 172.21.2.102:54188) 2021-02-10T11:31:37.292 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:37.273 7f1d7ed73700 1 -- 172.21.2.102:0/3579939116 learned_addr learned my addr 172.21.2.102:0/3579939116 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:31:37.292 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:37.274 7f1d7f574700 1 --2- 172.21.2.102:0/3579939116 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1d800d1870 0x7f1d801b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:37.292 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:37.274 7f1d7e572700 1 -- 172.21.2.102:0/3579939116 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2780517608 0 0) 0x7f1d700010b0 con 0x7f1d801b0a10 2021-02-10T11:31:37.292 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:37.274 7f1d7e572700 1 -- 172.21.2.102:0/3579939116 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d6c002fe0 con 0x7f1d801b0a10 2021-02-10T11:31:37.292 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:37.274 7f1d7e572700 1 -- 172.21.2.102:0/3579939116 <== mon.1 v1:172.21.2.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1395058352 0 0) 0x7f1d78001400 con 0x7f1d800f0650 2021-02-10T11:31:37.292 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:37.274 7f1d7e572700 1 -- 172.21.2.102:0/3579939116 --> v1:172.21.2.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d6c003210 con 0x7f1d800f0650 2021-02-10T11:31:37.292 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:37.274 7f1d7f574700 1 -- 172.21.2.102:0/3579939116 >> v1:172.21.2.113:6789/0 conn(0x7f1d800f0650 legacy=0x7f1d800cfb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:37.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:37.274 7f1d7f574700 1 -- 172.21.2.102:0/3579939116 >> v1:172.21.2.102:6789/0 conn(0x7f1d801b0a10 legacy=0x7f1d800cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:37.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:37.274 7f1d7f574700 1 -- 172.21.2.102:0/3579939116 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d800d0530 con 0x7f1d800d1870 2021-02-10T11:31:37.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:37.274 7f1d7f574700 1 --2- 172.21.2.102:0/3579939116 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1d800d1870 0x7f1d801b03f0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7f1d68006650 tx=0x7f1d68009a80).ready entity=mon.2 client_cookie=ca1eef6ec17a2141 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:37.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:37.274 7f1d7e572700 1 -- 172.21.2.102:0/3579939116 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1d680009b0 con 0x7f1d800d1870 2021-02-10T11:31:37.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:37.274 7f1d7e572700 1 -- 172.21.2.102:0/3579939116 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d6800c0c0 con 0x7f1d800d1870 2021-02-10T11:31:37.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:37.274 7f1d7e572700 1 -- 172.21.2.102:0/3579939116 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1d6800c520 con 0x7f1d800d1870 2021-02-10T11:31:37.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:37.275 7f1d85979700 1 -- 172.21.2.102:0/3579939116 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1d800d1870 msgr2=0x7f1d801b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:37.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:37.275 7f1d85979700 1 --2- 172.21.2.102:0/3579939116 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1d800d1870 0x7f1d801b03f0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7f1d68006650 tx=0x7f1d68009a80).stop 2021-02-10T11:31:37.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:37.275 7f1d85979700 1 -- 172.21.2.102:0/3579939116 shutdown_connections 2021-02-10T11:31:37.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:37.275 7f1d85979700 1 --2- 172.21.2.102:0/3579939116 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1d800d1870 0x7f1d801b03f0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:37.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:37.275 7f1d85979700 1 -- 172.21.2.102:0/3579939116 >> 172.21.2.102:0/3579939116 conn(0x7f1d800ce450 msgr2=0x7f1d800d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:37.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:37.275 7f1d85979700 1 -- 172.21.2.102:0/3579939116 shutdown_connections 2021-02-10T11:31:37.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:37.275 7f1d85979700 1 -- 172.21.2.102:0/3579939116 wait complete. 2021-02-10T11:31:37.294 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:37.290 7f1d85979700 1 Processor -- start 2021-02-10T11:31:37.294 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:37.290 7f1d85979700 1 -- start start 2021-02-10T11:31:37.295 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:37.290 7f1d85979700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1d800970b0 0x7f1d800ebe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:37.295 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:37.290 7f1d85979700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1d80098810 0x7f1d80098c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:37.295 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:37.290 7f1d85979700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1d80099230 0x7f1d800b18c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:37.295 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:37.290 7f1d85979700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d800ec6d0 con 0x7f1d80098810 2021-02-10T11:31:37.295 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:37.290 7f1d85979700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d800974b0 con 0x7f1d80099230 2021-02-10T11:31:37.295 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:37.290 7f1d85979700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d800b1ee0 con 0x7f1d800970b0 2021-02-10T11:31:37.296 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:37.290 7f1d7ed73700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1d80098810 0x7f1d80098c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:37.296 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:37.290 7f1d7ed73700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1d80098810 0x7f1d80098c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:33738/0 (socket says 172.21.2.102:33738) 2021-02-10T11:31:37.296 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:37.290 7f1d7ed73700 1 -- 172.21.2.102:0/3972206096 learned_addr learned my addr 172.21.2.102:0/3972206096 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:31:37.296 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:37.290 7f1d7f574700 1 --2- 172.21.2.102:0/3972206096 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1d800970b0 0x7f1d800ebe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:37.296 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:37.290 7f1d7fd75700 1 --2- 172.21.2.102:0/3972206096 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1d80099230 0x7f1d800b18c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:37.296 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:37.291 7f1d7ed73700 1 -- 172.21.2.102:0/3972206096 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1d800970b0 msgr2=0x7f1d800ebe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:37.296 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:37.291 7f1d7ed73700 1 --2- 172.21.2.102:0/3972206096 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1d800970b0 0x7f1d800ebe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:37.296 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:37.291 7f1d7ed73700 1 -- 172.21.2.102:0/3972206096 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1d80099230 msgr2=0x7f1d800b18c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:37.296 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:37.291 7f1d7ed73700 1 --2- 172.21.2.102:0/3972206096 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1d80099230 0x7f1d800b18c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:37.296 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:37.291 7f1d7ed73700 1 -- 172.21.2.102:0/3972206096 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d800b2100 con 0x7f1d80098810 2021-02-10T11:31:37.296 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:37.291 7f1d7ed73700 1 --2- 172.21.2.102:0/3972206096 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1d80098810 0x7f1d80098c10 secure :-1 s=READY pgs=2144 cs=0 l=1 rev1=1 rx=0x7f1d70001c60 tx=0x7f1d7000a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:37.297 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:37.291 7f1d77fff700 1 -- 172.21.2.102:0/3972206096 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1d7000acc0 con 0x7f1d80098810 2021-02-10T11:31:37.297 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:37.291 7f1d77fff700 1 -- 172.21.2.102:0/3972206096 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d70004350 con 0x7f1d80098810 2021-02-10T11:31:37.297 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:37.291 7f1d77fff700 1 -- 172.21.2.102:0/3972206096 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1d70004870 con 0x7f1d80098810 2021-02-10T11:31:37.297 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:37.291 7f1d85979700 1 -- 172.21.2.102:0/3972206096 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d800b2100 con 0x7f1d80098810 2021-02-10T11:31:37.297 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:37.291 7f1d85979700 1 -- 172.21.2.102:0/3972206096 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d800b2690 con 0x7f1d80098810 2021-02-10T11:31:37.297 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:37.292 7f1d77fff700 1 -- 172.21.2.102:0/3972206096 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1d70004530 con 0x7f1d80098810 2021-02-10T11:31:37.297 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:37.292 7f1d77fff700 1 --2- 172.21.2.102:0/3972206096 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1d640186c0 0x7f1d6401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:37.297 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:37.292 7f1d77fff700 1 -- 172.21.2.102:0/3972206096 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f1d70032400 con 0x7f1d80098810 2021-02-10T11:31:37.297 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:37.292 7f1d7f574700 1 --2- 172.21.2.102:0/3972206096 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1d640186c0 0x7f1d6401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:37.297 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:37.292 7f1d7f574700 1 --2- 172.21.2.102:0/3972206096 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1d640186c0 0x7f1d6401abc0 crc :-1 s=READY pgs=950 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:37.325 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:37.324 7f1d85979700 1 -- 172.21.2.102:0/3972206096 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d800ebc00 con 0x7f1d80098810 2021-02-10T11:31:37.327 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:37.326 7f1d77fff700 1 -- 172.21.2.102:0/3972206096 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f1d70000960 con 0x7f1d80098810 2021-02-10T11:31:37.456 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:37.455 7f1d85979700 1 -- 172.21.2.102:0/3972206096 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1d800f0b10 con 0x7f1d80098810 2021-02-10T11:31:37.456 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:37.455 7f1d77fff700 1 -- 172.21.2.102:0/3972206096 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+2101 (secure 0 0 0) 0x7f1d70004510 con 0x7f1d80098810 2021-02-10T11:31:37.457 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:31:37.457 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":83,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10056,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.102:6835/711305601","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6834","nonce":711305601},{"type":"v1","addr":"172.21.2.102:6835","nonce":711305601}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":78}],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:30:58.047057","modified":"2021-02-10 11:31:00.055485","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6353},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6353":{"gid":6353,"name":"b","rank":0,"incarnation":79,"state":"up:active","state_seq":3,"addr":"172.21.2.113:6833/1091618344","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6832","nonce":1091618344},{"type":"v1","addr":"172.21.2.113:6833","nonce":1091618344}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":10}]} 2021-02-10T11:31:37.457 INFO:teuthology.orchestra.run.gibba002.stderr:dumped fsmap epoch 83 2021-02-10T11:31:37.458 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:37.457 7f1d85979700 1 -- 172.21.2.102:0/3972206096 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1d640186c0 msgr2=0x7f1d6401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:37.458 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:37.457 7f1d85979700 1 --2- 172.21.2.102:0/3972206096 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1d640186c0 0x7f1d6401abc0 crc :-1 s=READY pgs=950 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:37.458 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:37.457 7f1d85979700 1 -- 172.21.2.102:0/3972206096 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1d80098810 msgr2=0x7f1d80098c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:37.458 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:37.457 7f1d85979700 1 --2- 172.21.2.102:0/3972206096 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1d80098810 0x7f1d80098c10 secure :-1 s=READY pgs=2144 cs=0 l=1 rev1=1 rx=0x7f1d70001c60 tx=0x7f1d7000a8d0).stop 2021-02-10T11:31:37.459 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:37.457 7f1d85979700 1 -- 172.21.2.102:0/3972206096 shutdown_connections 2021-02-10T11:31:37.459 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:37.457 7f1d85979700 1 --2- 172.21.2.102:0/3972206096 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1d640186c0 0x7f1d6401abc0 unknown :-1 s=CLOSED pgs=950 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:37.459 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:37.457 7f1d85979700 1 --2- 172.21.2.102:0/3972206096 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1d80099230 0x7f1d800b18c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:37.459 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:37.457 7f1d85979700 1 --2- 172.21.2.102:0/3972206096 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1d80098810 0x7f1d80098c10 unknown :-1 s=CLOSED pgs=2144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:37.459 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:37.457 7f1d85979700 1 --2- 172.21.2.102:0/3972206096 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1d800970b0 0x7f1d800ebe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:37.459 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:37.457 7f1d85979700 1 -- 172.21.2.102:0/3972206096 >> 172.21.2.102:0/3972206096 conn(0x7f1d8019a730 msgr2=0x7f1d8019e480 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:37.459 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:37.457 7f1d85979700 1 -- 172.21.2.102:0/3972206096 shutdown_connections 2021-02-10T11:31:37.459 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:37.458 7f1d85979700 1 -- 172.21.2.102:0/3972206096 wait complete. 2021-02-10T11:31:37.467 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs fail cephfs 2021-02-10T11:31:37.895 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:37.877 7f747c1da700 1 Processor -- start 2021-02-10T11:31:37.895 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:37.877 7f747c1da700 1 -- start start 2021-02-10T11:31:37.895 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:37.878 7f747c1da700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f74740d1860 0x7f74741b03e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:37.895 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:37.878 7f747c1da700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74740f0b00 con 0x7f74740f0640 2021-02-10T11:31:37.895 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:37.878 7f747c1da700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74740d0300 con 0x7f74741b0a00 2021-02-10T11:31:37.895 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:37.878 7f747c1da700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74740d0520 con 0x7f74740d1860 2021-02-10T11:31:37.895 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:37.878 7f7479ff6700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f74740d1860 0x7f74741b03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:37.895 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:37.878 7f747a7f7700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f74740f0640 0x7f74740cfb10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:54200/0 (socket says 172.21.2.102:54200) 2021-02-10T11:31:37.895 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:37.878 7f7479ff6700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f74740d1860 0x7f74741b03e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:33744/0 (socket says 172.21.2.102:33744) 2021-02-10T11:31:37.895 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:37.878 7f747a7f7700 1 -- 172.21.2.102:0/666863785 learned_addr learned my addr 172.21.2.102:0/666863785 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:31:37.896 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:37.878 7f7479ff6700 1 -- 172.21.2.102:0/666863785 >> v1:172.21.2.113:6789/0 conn(0x7f74741b0a00 legacy=0x7f74740cf320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:37.896 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:37.878 7f7479ff6700 1 -- 172.21.2.102:0/666863785 >> v1:172.21.2.102:6789/0 conn(0x7f74740f0640 legacy=0x7f74740cfb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:37.896 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:37.878 7f7479ff6700 1 -- 172.21.2.102:0/666863785 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74740d1c60 con 0x7f74740d1860 2021-02-10T11:31:37.896 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:37.878 7f7479ff6700 1 --2- 172.21.2.102:0/666863785 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f74740d1860 0x7f74741b03e0 secure :-1 s=READY pgs=2146 cs=0 l=1 rev1=1 rx=0x7f7468006650 tx=0x7f7468009a80).ready entity=mon.0 client_cookie=66c456882d0a4cf8 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:37.896 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:37.878 7f7478ff4700 1 -- 172.21.2.102:0/666863785 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f746800c0c0 con 0x7f74740d1860 2021-02-10T11:31:37.896 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:37.879 7f7478ff4700 1 -- 172.21.2.102:0/666863785 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f746800c340 con 0x7f74740d1860 2021-02-10T11:31:37.896 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:37.879 7f7478ff4700 1 -- 172.21.2.102:0/666863785 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f746800c860 con 0x7f74740d1860 2021-02-10T11:31:37.896 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:37.879 7f747c1da700 1 -- 172.21.2.102:0/666863785 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f74740d1860 msgr2=0x7f74741b03e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:37.896 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:37.879 7f747c1da700 1 --2- 172.21.2.102:0/666863785 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f74740d1860 0x7f74741b03e0 secure :-1 s=READY pgs=2146 cs=0 l=1 rev1=1 rx=0x7f7468006650 tx=0x7f7468009a80).stop 2021-02-10T11:31:37.896 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:37.879 7f747c1da700 1 -- 172.21.2.102:0/666863785 shutdown_connections 2021-02-10T11:31:37.896 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:37.879 7f747c1da700 1 --2- 172.21.2.102:0/666863785 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f74740d1860 0x7f74741b03e0 unknown :-1 s=CLOSED pgs=2146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:37.896 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:37.879 7f747c1da700 1 -- 172.21.2.102:0/666863785 >> 172.21.2.102:0/666863785 conn(0x7f74740ce440 msgr2=0x7f74740d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:37.896 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:37.879 7f747c1da700 1 -- 172.21.2.102:0/666863785 shutdown_connections 2021-02-10T11:31:37.897 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:37.879 7f747c1da700 1 -- 172.21.2.102:0/666863785 wait complete. 2021-02-10T11:31:37.898 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:37.894 7f747c1da700 1 Processor -- start 2021-02-10T11:31:37.898 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:37.894 7f747c1da700 1 -- start start 2021-02-10T11:31:37.898 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:37.894 7f747c1da700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f74740a7930 0x7f74740a1b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:37.898 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:37.894 7f747c1da700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f74740c16b0 0x7f74740c1ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:37.898 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:37.894 7f747c1da700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f74740c20d0 0x7f74740a8640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:37.898 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:37.894 7f747c1da700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74740c24d0 con 0x7f74740c16b0 2021-02-10T11:31:37.899 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:37.894 7f747c1da700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74740a7d30 con 0x7f74740c20d0 2021-02-10T11:31:37.899 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:37.894 7f747c1da700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74740a2150 con 0x7f74740a7930 2021-02-10T11:31:37.899 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:37.894 7f74797f5700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f74740c16b0 0x7f74740c1ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:37.899 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:37.895 7f74797f5700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f74740c16b0 0x7f74740c1ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:33750/0 (socket says 172.21.2.102:33750) 2021-02-10T11:31:37.899 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:37.895 7f74797f5700 1 -- 172.21.2.102:0/1318369684 learned_addr learned my addr 172.21.2.102:0/1318369684 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:31:37.899 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:37.895 7f74797f5700 1 -- 172.21.2.102:0/1318369684 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f74740a7930 msgr2=0x7f74740a1b30 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:31:37.899 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:37.895 7f74797f5700 1 --2- 172.21.2.102:0/1318369684 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f74740a7930 0x7f74740a1b30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:37.899 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:37.895 7f74797f5700 1 -- 172.21.2.102:0/1318369684 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f74740c20d0 msgr2=0x7f74740a8640 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:31:37.899 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:37.895 7f74797f5700 1 --2- 172.21.2.102:0/1318369684 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f74740c20d0 0x7f74740a8640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:37.899 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:37.895 7f74797f5700 1 -- 172.21.2.102:0/1318369684 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74740a8c60 con 0x7f74740c16b0 2021-02-10T11:31:37.899 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:37.895 7f74797f5700 1 --2- 172.21.2.102:0/1318369684 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f74740c16b0 0x7f74740c1ab0 secure :-1 s=READY pgs=2147 cs=0 l=1 rev1=1 rx=0x7f747000b5b0 tx=0x7f747000b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:37.899 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:37.895 7f74667fc700 1 -- 172.21.2.102:0/1318369684 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f747000bd40 con 0x7f74740c16b0 2021-02-10T11:31:37.899 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:37.895 7f74667fc700 1 -- 172.21.2.102:0/1318369684 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7470000ce0 con 0x7f74740c16b0 2021-02-10T11:31:37.900 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:37.895 7f74667fc700 1 -- 172.21.2.102:0/1318369684 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f74700062a0 con 0x7f74740c16b0 2021-02-10T11:31:37.900 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:37.895 7f747c1da700 1 -- 172.21.2.102:0/1318369684 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74740a8c60 con 0x7f74740c16b0 2021-02-10T11:31:37.900 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:37.895 7f747c1da700 1 -- 172.21.2.102:0/1318369684 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74740a8c60 con 0x7f74740c16b0 2021-02-10T11:31:37.900 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:37.896 7f74667fc700 1 -- 172.21.2.102:0/1318369684 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f7470006020 con 0x7f74740c16b0 2021-02-10T11:31:37.900 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:37.896 7f74667fc700 1 --2- 172.21.2.102:0/1318369684 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f745c0186c0 0x7f745c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:37.900 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:37.896 7f74667fc700 1 -- 172.21.2.102:0/1318369684 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f7470032800 con 0x7f74740c16b0 2021-02-10T11:31:37.900 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:37.896 7f7479ff6700 1 --2- 172.21.2.102:0/1318369684 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f745c0186c0 0x7f745c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:37.900 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:37.897 7f7479ff6700 1 --2- 172.21.2.102:0/1318369684 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f745c0186c0 0x7f745c01abc0 crc :-1 s=READY pgs=951 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:37.929 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:37.928 7f747c1da700 1 -- 172.21.2.102:0/1318369684 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74740e75b0 con 0x7f74740c16b0 2021-02-10T11:31:37.931 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:37.930 7f74667fc700 1 -- 172.21.2.102:0/1318369684 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f747001f540 con 0x7f74740c16b0 2021-02-10T11:31:38.060 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:38.059 7f747c1da700 1 -- 172.21.2.102:0/1318369684 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f74740d1ed0 con 0x7f74740c16b0 2021-02-10T11:31:38.099 INFO:tasks.ceph.mon.a.gibba002.stderr:2021-02-10 11:31:38.098 7f8dd2755700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-02-10T11:31:38.104 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:38.103 7f74667fc700 1 -- 172.21.2.102:0/1318369684 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v84) v1 ==== 161+0+0 (secure 0 0 0) 0x7f7470006020 con 0x7f74740c16b0 2021-02-10T11:31:38.124 INFO:teuthology.orchestra.run.gibba002.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2021-02-10T11:31:38.126 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:38.125 7f747c1da700 1 -- 172.21.2.102:0/1318369684 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f745c0186c0 msgr2=0x7f745c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:38.127 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:38.125 7f747c1da700 1 --2- 172.21.2.102:0/1318369684 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f745c0186c0 0x7f745c01abc0 crc :-1 s=READY pgs=951 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:38.127 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:38.125 7f747c1da700 1 -- 172.21.2.102:0/1318369684 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f74740c16b0 msgr2=0x7f74740c1ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:38.127 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:38.125 7f747c1da700 1 --2- 172.21.2.102:0/1318369684 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f74740c16b0 0x7f74740c1ab0 secure :-1 s=READY pgs=2147 cs=0 l=1 rev1=1 rx=0x7f747000b5b0 tx=0x7f747000b950).stop 2021-02-10T11:31:38.127 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:38.125 7f747c1da700 1 -- 172.21.2.102:0/1318369684 shutdown_connections 2021-02-10T11:31:38.127 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:38.125 7f747c1da700 1 --2- 172.21.2.102:0/1318369684 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f745c0186c0 0x7f745c01abc0 unknown :-1 s=CLOSED pgs=951 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:38.127 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:38.125 7f747c1da700 1 --2- 172.21.2.102:0/1318369684 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f74740c20d0 0x7f74740a8640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:38.127 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:38.125 7f747c1da700 1 --2- 172.21.2.102:0/1318369684 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f74740c16b0 0x7f74740c1ab0 unknown :-1 s=CLOSED pgs=2147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:38.127 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:38.125 7f747c1da700 1 --2- 172.21.2.102:0/1318369684 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f74740a7930 0x7f74740a1b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:38.127 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:38.125 7f747c1da700 1 -- 172.21.2.102:0/1318369684 >> 172.21.2.102:0/1318369684 conn(0x7f74740b34f0 msgr2=0x7f74740aaa60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:38.127 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:38.125 7f747c1da700 1 -- 172.21.2.102:0/1318369684 shutdown_connections 2021-02-10T11:31:38.128 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:38.125 7f747c1da700 1 -- 172.21.2.102:0/1318369684 wait complete. 2021-02-10T11:31:38.135 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:31:38.156 INFO:tasks.ceph.mds.b.gibba013.stdout:starting mds.b at 2021-02-10T11:31:38.562 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:38.544 7f3984783700 1 Processor -- start 2021-02-10T11:31:38.563 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:38.545 7f3984783700 1 -- start start 2021-02-10T11:31:38.563 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:38.545 7f3984783700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f397c0d0340 0x7f397c0ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:38.563 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:38.545 7f3984783700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f397c0d1c70 0x7f397c0efc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:38.563 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:38.545 7f3984783700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f397c0f0b10 con 0x7f397c0d1870 2021-02-10T11:31:38.563 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:38.545 7f3984783700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f397c0e5ed0 con 0x7f397c0d0340 2021-02-10T11:31:38.563 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:38.545 7f3984783700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f397c1b0be0 con 0x7f397c0d1c70 2021-02-10T11:31:38.563 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:38.545 7f3981d9e700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f397c0d0340 0x7f397c0ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:38.563 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:38.545 7f398259f700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f397c0d1870 0x7f397c1b03f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:54214/0 (socket says 172.21.2.102:54214) 2021-02-10T11:31:38.563 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:38.545 7f3982da0700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f397c0d1c70 0x7f397c0efc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:38.563 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:38.545 7f398259f700 1 -- 172.21.2.102:0/1626375724 learned_addr learned my addr 172.21.2.102:0/1626375724 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:31:38.563 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:38.546 7f3982da0700 1 -- 172.21.2.102:0/1626375724 >> v2:172.21.2.102:3300/0 conn(0x7f397c0d0340 msgr2=0x7f397c0ef640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:38.563 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:38.546 7f3982da0700 1 --2- 172.21.2.102:0/1626375724 >> v2:172.21.2.102:3300/0 conn(0x7f397c0d0340 0x7f397c0ef640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:38.564 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:38.546 7f3982da0700 1 -- 172.21.2.102:0/1626375724 >> v1:172.21.2.102:6789/0 conn(0x7f397c0d1870 legacy=0x7f397c1b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:38.564 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:38.546 7f3982da0700 1 -- 172.21.2.102:0/1626375724 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f397c0f20a0 con 0x7f397c0d1c70 2021-02-10T11:31:38.564 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:38.546 7f3981d9e700 1 --2- 172.21.2.102:0/1626375724 >> v2:172.21.2.102:3300/0 conn(0x7f397c0d0340 0x7f397c0ef640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:31:38.564 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:38.546 7f3982da0700 1 --2- 172.21.2.102:0/1626375724 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f397c0d1c70 0x7f397c0efc60 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 rx=0x7f3974006e50 tx=0x7f3974009e70).ready entity=mon.1 client_cookie=258bca0be2ccf3a2 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:38.564 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:38.546 7f398159d700 1 -- 172.21.2.102:0/1626375724 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f397400b510 con 0x7f397c0d1c70 2021-02-10T11:31:38.564 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:38.546 7f398159d700 1 -- 172.21.2.102:0/1626375724 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3974006790 con 0x7f397c0d1c70 2021-02-10T11:31:38.564 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:38.546 7f398159d700 1 -- 172.21.2.102:0/1626375724 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f397400b9d0 con 0x7f397c0d1c70 2021-02-10T11:31:38.564 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:38.546 7f3984783700 1 -- 172.21.2.102:0/1626375724 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f397c0d1c70 msgr2=0x7f397c0efc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:38.564 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:38.546 7f3984783700 1 --2- 172.21.2.102:0/1626375724 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f397c0d1c70 0x7f397c0efc60 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 rx=0x7f3974006e50 tx=0x7f3974009e70).stop 2021-02-10T11:31:38.564 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:38.547 7f3984783700 1 -- 172.21.2.102:0/1626375724 shutdown_connections 2021-02-10T11:31:38.564 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:38.547 7f3984783700 1 --2- 172.21.2.102:0/1626375724 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f397c0d1c70 0x7f397c0efc60 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:38.564 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:38.547 7f3984783700 1 --2- 172.21.2.102:0/1626375724 >> v2:172.21.2.102:3300/0 conn(0x7f397c0d0340 0x7f397c0ef640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:38.564 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:38.547 7f3984783700 1 -- 172.21.2.102:0/1626375724 >> 172.21.2.102:0/1626375724 conn(0x7f397c0ce450 msgr2=0x7f397c0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:38.565 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:38.547 7f3984783700 1 -- 172.21.2.102:0/1626375724 shutdown_connections 2021-02-10T11:31:38.565 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:38.547 7f3984783700 1 -- 172.21.2.102:0/1626375724 wait complete. 2021-02-10T11:31:38.566 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:38.562 7f3984783700 1 Processor -- start 2021-02-10T11:31:38.566 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:38.562 7f3984783700 1 -- start start 2021-02-10T11:31:38.567 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:38.562 7f3984783700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f397c1942b0 0x7f397c0b3550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:38.567 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:38.562 7f3984783700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f397c195070 0x7f397c195470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:38.567 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:38.562 7f3984783700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f397c0bb8e0 0x7f397c0bbce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:38.567 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:38.562 7f3984783700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f397c1946b0 con 0x7f397c0bb8e0 2021-02-10T11:31:38.567 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:38.562 7f3984783700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f397c0b3b70 con 0x7f397c1942b0 2021-02-10T11:31:38.567 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:38.562 7f3984783700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f397c0b3d90 con 0x7f397c195070 2021-02-10T11:31:38.567 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:38.562 7f3982da0700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f397c0bb8e0 0x7f397c0bbce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:38.567 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:38.563 7f3982da0700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f397c0bb8e0 0x7f397c0bbce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:33764/0 (socket says 172.21.2.102:33764) 2021-02-10T11:31:38.567 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:38.563 7f3982da0700 1 -- 172.21.2.102:0/1404375239 learned_addr learned my addr 172.21.2.102:0/1404375239 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:31:38.567 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:38.563 7f398259f700 1 --2- 172.21.2.102:0/1404375239 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f397c1942b0 0x7f397c0b3550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:38.567 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:38.563 7f3981d9e700 1 --2- 172.21.2.102:0/1404375239 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f397c195070 0x7f397c195470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:38.567 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:38.563 7f3982da0700 1 -- 172.21.2.102:0/1404375239 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f397c195070 msgr2=0x7f397c195470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:38.567 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:38.563 7f3982da0700 1 --2- 172.21.2.102:0/1404375239 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f397c195070 0x7f397c195470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:38.568 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:38.563 7f3982da0700 1 -- 172.21.2.102:0/1404375239 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f397c1942b0 msgr2=0x7f397c0b3550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:38.568 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:38.563 7f3982da0700 1 --2- 172.21.2.102:0/1404375239 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f397c1942b0 0x7f397c0b3550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:38.568 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:38.563 7f3982da0700 1 -- 172.21.2.102:0/1404375239 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f397c1a49e0 con 0x7f397c0bb8e0 2021-02-10T11:31:38.568 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:38.563 7f3982da0700 1 --2- 172.21.2.102:0/1404375239 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f397c0bb8e0 0x7f397c0bbce0 secure :-1 s=READY pgs=2150 cs=0 l=1 rev1=1 rx=0x7f397400b970 tx=0x7f397400bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:38.568 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:38.563 7f3972ffd700 1 -- 172.21.2.102:0/1404375239 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f39740093d0 con 0x7f397c0bb8e0 2021-02-10T11:31:38.568 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:38.563 7f3972ffd700 1 -- 172.21.2.102:0/1404375239 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3974009020 con 0x7f397c0bb8e0 2021-02-10T11:31:38.568 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:38.563 7f3984783700 1 -- 172.21.2.102:0/1404375239 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f397c1a49e0 con 0x7f397c0bb8e0 2021-02-10T11:31:38.568 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:38.563 7f3972ffd700 1 -- 172.21.2.102:0/1404375239 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f397400c560 con 0x7f397c0bb8e0 2021-02-10T11:31:38.568 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:38.563 7f3984783700 1 -- 172.21.2.102:0/1404375239 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f397c1a49e0 con 0x7f397c0bb8e0 2021-02-10T11:31:38.568 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:38.564 7f3972ffd700 1 -- 172.21.2.102:0/1404375239 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f3974001030 con 0x7f397c0bb8e0 2021-02-10T11:31:38.568 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:38.564 7f3972ffd700 1 --2- 172.21.2.102:0/1404375239 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3964018730 0x7f396401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:38.568 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:38.564 7f3972ffd700 1 -- 172.21.2.102:0/1404375239 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 5659+0+0 (secure 0 0 0) 0x7f3974032a10 con 0x7f397c0bb8e0 2021-02-10T11:31:38.568 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:38.564 7f398259f700 1 --2- 172.21.2.102:0/1404375239 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3964018730 0x7f396401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:38.569 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:38.564 7f398259f700 1 --2- 172.21.2.102:0/1404375239 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3964018730 0x7f396401abc0 crc :-1 s=READY pgs=952 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:38.595 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:38.594 7f3984783700 1 -- 172.21.2.102:0/1404375239 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f397c0bc5f0 con 0x7f397c0bb8e0 2021-02-10T11:31:38.598 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:38.597 7f3972ffd700 1 -- 172.21.2.102:0/1404375239 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f3974001030 con 0x7f397c0bb8e0 2021-02-10T11:31:38.726 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:38.725 7f3984783700 1 -- 172.21.2.102:0/1404375239 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f397c0bc5f0 con 0x7f397c0bb8e0 2021-02-10T11:31:38.727 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:38.726 7f3972ffd700 1 -- 172.21.2.102:0/1404375239 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 84 v84) v1 ==== 94+0+1757 (secure 0 0 0) 0x7f3974009020 con 0x7f397c0bb8e0 2021-02-10T11:31:38.727 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:31:38.728 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":84,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10056,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.102:6835/711305601","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6834","nonce":711305601},{"type":"v1","addr":"172.21.2.102:6835","nonce":711305601}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":78}],"filesystems":[{"mdsmap":{"epoch":84,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:30:58.047057","modified":"2021-02-10 11:31:38.099378","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":104,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":10}]} 2021-02-10T11:31:38.728 INFO:teuthology.orchestra.run.gibba002.stderr:dumped fsmap epoch 84 2021-02-10T11:31:38.729 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:38.728 7f3984783700 1 -- 172.21.2.102:0/1404375239 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3964018730 msgr2=0x7f396401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:38.729 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:38.728 7f3984783700 1 --2- 172.21.2.102:0/1404375239 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3964018730 0x7f396401abc0 crc :-1 s=READY pgs=952 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:38.729 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:38.728 7f3984783700 1 -- 172.21.2.102:0/1404375239 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f397c0bb8e0 msgr2=0x7f397c0bbce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:38.729 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:38.728 7f3984783700 1 --2- 172.21.2.102:0/1404375239 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f397c0bb8e0 0x7f397c0bbce0 secure :-1 s=READY pgs=2150 cs=0 l=1 rev1=1 rx=0x7f397400b970 tx=0x7f397400bd10).stop 2021-02-10T11:31:38.729 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:38.728 7f3984783700 1 -- 172.21.2.102:0/1404375239 shutdown_connections 2021-02-10T11:31:38.729 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:38.728 7f3984783700 1 --2- 172.21.2.102:0/1404375239 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f397c0bb8e0 0x7f397c0bbce0 unknown :-1 s=CLOSED pgs=2150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:38.730 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:38.728 7f3984783700 1 --2- 172.21.2.102:0/1404375239 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f397c195070 0x7f397c195470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:38.730 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:38.728 7f3984783700 1 --2- 172.21.2.102:0/1404375239 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3964018730 0x7f396401abc0 unknown :-1 s=CLOSED pgs=952 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:38.730 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:38.728 7f3984783700 1 --2- 172.21.2.102:0/1404375239 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f397c1942b0 0x7f397c0b3550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:38.730 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:38.728 7f3984783700 1 -- 172.21.2.102:0/1404375239 >> 172.21.2.102:0/1404375239 conn(0x7f397c0b3040 msgr2=0x7f397c0c3250 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:38.730 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:38.728 7f3984783700 1 -- 172.21.2.102:0/1404375239 shutdown_connections 2021-02-10T11:31:38.730 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:38.728 7f3984783700 1 -- 172.21.2.102:0/1404375239 wait complete. 2021-02-10T11:31:38.738 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2021-02-10T11:31:39.172 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:39.156 7f0725403700 1 Processor -- start 2021-02-10T11:31:39.173 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:39.156 7f0725403700 1 -- start start 2021-02-10T11:31:39.173 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:39.156 7f0725403700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f07200d0330 0x7f07200ef630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:39.173 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:39.156 7f0725403700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f07200d1c60 0x7f07200efc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:39.173 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:39.156 7f0725403700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07200f0b00 con 0x7f07200d1860 2021-02-10T11:31:39.173 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:39.156 7f0725403700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07200e5ec0 con 0x7f07200d0330 2021-02-10T11:31:39.173 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:39.156 7f0725403700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07201b0bd0 con 0x7f07200d1c60 2021-02-10T11:31:39.173 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:39.156 7f071687c700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f07200d0330 0x7f07200ef630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:39.173 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:39.156 7f071f07d700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f07200d1860 0x7f07201b03e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:54226/0 (socket says 172.21.2.102:54226) 2021-02-10T11:31:39.173 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:39.156 7f071f07d700 1 -- 172.21.2.102:0/3750518482 learned_addr learned my addr 172.21.2.102:0/3750518482 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:31:39.173 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:39.156 7f071f87e700 1 --2- 172.21.2.102:0/3750518482 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f07200d1c60 0x7f07200efc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:39.174 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:39.157 7f071687c700 1 -- 172.21.2.102:0/3750518482 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f07200d1c60 msgr2=0x7f07200efc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:39.174 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:39.157 7f071687c700 1 --2- 172.21.2.102:0/3750518482 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f07200d1c60 0x7f07200efc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:39.174 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:39.157 7f071687c700 1 -- 172.21.2.102:0/3750518482 >> v1:172.21.2.102:6789/0 conn(0x7f07200d1860 legacy=0x7f07201b03e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:39.174 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:39.157 7f071687c700 1 -- 172.21.2.102:0/3750518482 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07200f2090 con 0x7f07200d0330 2021-02-10T11:31:39.174 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:39.157 7f071687c700 1 --2- 172.21.2.102:0/3750518482 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f07200d0330 0x7f07200ef630 secure :-1 s=READY pgs=2152 cs=0 l=1 rev1=1 rx=0x7f0708006650 tx=0x7f0708009a00).ready entity=mon.0 client_cookie=dd22686687621472 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:39.174 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:39.157 7f071e87c700 1 -- 172.21.2.102:0/3750518482 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0708009df0 con 0x7f07200d0330 2021-02-10T11:31:39.174 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:39.157 7f071e87c700 1 -- 172.21.2.102:0/3750518482 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07080009b0 con 0x7f07200d0330 2021-02-10T11:31:39.174 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:39.157 7f071e87c700 1 -- 172.21.2.102:0/3750518482 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f070800c2b0 con 0x7f07200d0330 2021-02-10T11:31:39.174 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:39.157 7f0725403700 1 -- 172.21.2.102:0/3750518482 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f07200d0330 msgr2=0x7f07200ef630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:39.174 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:39.157 7f0725403700 1 --2- 172.21.2.102:0/3750518482 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f07200d0330 0x7f07200ef630 secure :-1 s=READY pgs=2152 cs=0 l=1 rev1=1 rx=0x7f0708006650 tx=0x7f0708009a00).stop 2021-02-10T11:31:39.174 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:39.157 7f0725403700 1 -- 172.21.2.102:0/3750518482 shutdown_connections 2021-02-10T11:31:39.174 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:39.157 7f0725403700 1 --2- 172.21.2.102:0/3750518482 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f07200d0330 0x7f07200ef630 unknown :-1 s=CLOSED pgs=2152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:39.174 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:39.157 7f0725403700 1 --2- 172.21.2.102:0/3750518482 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f07200d1c60 0x7f07200efc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:39.175 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:39.157 7f0725403700 1 -- 172.21.2.102:0/3750518482 >> 172.21.2.102:0/3750518482 conn(0x7f07200ce440 msgr2=0x7f07200d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:39.175 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:39.157 7f0725403700 1 -- 172.21.2.102:0/3750518482 shutdown_connections 2021-02-10T11:31:39.175 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:39.157 7f0725403700 1 -- 172.21.2.102:0/3750518482 wait complete. 2021-02-10T11:31:39.176 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:39.172 7f0725403700 1 Processor -- start 2021-02-10T11:31:39.176 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:39.172 7f0725403700 1 -- start start 2021-02-10T11:31:39.176 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:39.172 7f0725403700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f072009eaf0 0x7f07200ebdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:39.177 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:39.172 7f0725403700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f07200b72e0 0x7f072009f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:39.177 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:39.172 7f0725403700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f07200b7dd0 0x7f07200b81d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:39.177 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:39.172 7f0725403700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07200b7850 con 0x7f07200b72e0 2021-02-10T11:31:39.177 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:39.172 7f0725403700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f072009eef0 con 0x7f072009eaf0 2021-02-10T11:31:39.177 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:39.172 7f0725403700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07200ec3f0 con 0x7f07200b7dd0 2021-02-10T11:31:39.177 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:39.172 7f071687c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f07200b72e0 0x7f072009f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:39.177 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:39.172 7f071687c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f07200b72e0 0x7f072009f1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:33776/0 (socket says 172.21.2.102:33776) 2021-02-10T11:31:39.177 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:39.172 7f071687c700 1 -- 172.21.2.102:0/1553105201 learned_addr learned my addr 172.21.2.102:0/1553105201 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:31:39.177 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:39.173 7f071f07d700 1 --2- 172.21.2.102:0/1553105201 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f072009eaf0 0x7f07200ebdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:39.177 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:39.173 7f071f87e700 1 --2- 172.21.2.102:0/1553105201 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f07200b7dd0 0x7f07200b81d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:39.178 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:39.173 7f071687c700 1 -- 172.21.2.102:0/1553105201 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f07200b7dd0 msgr2=0x7f07200b81d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:39.178 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:39.173 7f071687c700 1 --2- 172.21.2.102:0/1553105201 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f07200b7dd0 0x7f07200b81d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:39.178 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:39.173 7f071687c700 1 -- 172.21.2.102:0/1553105201 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f072009eaf0 msgr2=0x7f07200ebdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:39.178 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:39.173 7f071687c700 1 --2- 172.21.2.102:0/1553105201 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f072009eaf0 0x7f07200ebdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:39.178 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:39.173 7f071687c700 1 -- 172.21.2.102:0/1553105201 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f072009f7c0 con 0x7f07200b72e0 2021-02-10T11:31:39.178 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:39.173 7f071687c700 1 --2- 172.21.2.102:0/1553105201 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f07200b72e0 0x7f072009f1a0 secure :-1 s=READY pgs=2153 cs=0 l=1 rev1=1 rx=0x7f07080096a0 tx=0x7f070800b8e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:39.178 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:39.173 7f071c878700 1 -- 172.21.2.102:0/1553105201 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f07080093a0 con 0x7f07200b72e0 2021-02-10T11:31:39.178 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:39.173 7f071c878700 1 -- 172.21.2.102:0/1553105201 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07080067f0 con 0x7f07200b72e0 2021-02-10T11:31:39.178 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:39.173 7f071c878700 1 -- 172.21.2.102:0/1553105201 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0708002220 con 0x7f07200b72e0 2021-02-10T11:31:39.178 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:39.173 7f0725403700 1 -- 172.21.2.102:0/1553105201 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f072009f7c0 con 0x7f07200b72e0 2021-02-10T11:31:39.178 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:39.173 7f0725403700 1 -- 172.21.2.102:0/1553105201 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f072009f7c0 con 0x7f07200b72e0 2021-02-10T11:31:39.178 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:39.174 7f071c878700 1 -- 172.21.2.102:0/1553105201 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f07080052e0 con 0x7f07200b72e0 2021-02-10T11:31:39.179 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:39.174 7f071c878700 1 --2- 172.21.2.102:0/1553105201 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f07040186c0 0x7f070401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:39.179 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:39.174 7f071c878700 1 -- 172.21.2.102:0/1553105201 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 5659+0+0 (secure 0 0 0) 0x7f0708032570 con 0x7f07200b72e0 2021-02-10T11:31:39.179 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:39.174 7f071f07d700 1 --2- 172.21.2.102:0/1553105201 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f07040186c0 0x7f070401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:39.179 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:39.174 7f071f07d700 1 --2- 172.21.2.102:0/1553105201 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f07040186c0 0x7f070401abc0 crc :-1 s=READY pgs=953 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:39.206 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:39.205 7f0725403700 1 -- 172.21.2.102:0/1553105201 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07200ebba0 con 0x7f07200b72e0 2021-02-10T11:31:39.209 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:39.208 7f071c878700 1 -- 172.21.2.102:0/1553105201 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f070802deb0 con 0x7f07200b72e0 2021-02-10T11:31:39.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:39.336 7f0725403700 1 -- 172.21.2.102:0/1553105201 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f07200f0b00 con 0x7f07200b72e0 2021-02-10T11:31:39.426 DEBUG:teuthology.orchestra.run.gibba002:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:31:39.430 DEBUG:teuthology.orchestra.run.gibba013:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:31:40.113 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:40.112 7f071c878700 1 -- 172.21.2.102:0/1553105201 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v85) v1 ==== 104+0+0 (secure 0 0 0) 0x7f070802deb0 con 0x7f07200b72e0 2021-02-10T11:31:40.159 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:40.157 7f0725403700 1 -- 172.21.2.102:0/1553105201 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f07040186c0 msgr2=0x7f070401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:40.159 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:40.157 7f0725403700 1 --2- 172.21.2.102:0/1553105201 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f07040186c0 0x7f070401abc0 crc :-1 s=READY pgs=953 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:40.159 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:40.157 7f0725403700 1 -- 172.21.2.102:0/1553105201 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f07200b72e0 msgr2=0x7f072009f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:40.159 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:40.157 7f0725403700 1 --2- 172.21.2.102:0/1553105201 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f07200b72e0 0x7f072009f1a0 secure :-1 s=READY pgs=2153 cs=0 l=1 rev1=1 rx=0x7f07080096a0 tx=0x7f070800b8e0).stop 2021-02-10T11:31:40.159 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:40.157 7f0725403700 1 -- 172.21.2.102:0/1553105201 shutdown_connections 2021-02-10T11:31:40.159 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:40.157 7f0725403700 1 --2- 172.21.2.102:0/1553105201 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f07200b7dd0 0x7f07200b81d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:40.159 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:40.157 7f0725403700 1 --2- 172.21.2.102:0/1553105201 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f07200b72e0 0x7f072009f1a0 unknown :-1 s=CLOSED pgs=2153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:40.159 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:40.157 7f0725403700 1 --2- 172.21.2.102:0/1553105201 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f07040186c0 0x7f070401abc0 unknown :-1 s=CLOSED pgs=953 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:40.159 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:40.157 7f0725403700 1 --2- 172.21.2.102:0/1553105201 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f072009eaf0 0x7f07200ebdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:40.160 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:40.157 7f0725403700 1 -- 172.21.2.102:0/1553105201 >> 172.21.2.102:0/1553105201 conn(0x7f07200b8df0 msgr2=0x7f07200add50 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:40.160 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:40.157 7f0725403700 1 -- 172.21.2.102:0/1553105201 shutdown_connections 2021-02-10T11:31:40.160 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:40.158 7f0725403700 1 -- 172.21.2.102:0/1553105201 wait complete. 2021-02-10T11:31:40.167 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool delete cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2021-02-10T11:31:40.597 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:40.580 7fcf3fe38700 1 Processor -- start 2021-02-10T11:31:40.598 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:40.580 7fcf3fe38700 1 -- start start 2021-02-10T11:31:40.598 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:40.581 7fcf3fe38700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcf380d1c80 0x7fcf380cef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:40.598 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:40.581 7fcf3fe38700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fcf380cf590 0x7fcf380cf990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:40.598 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:40.581 7fcf3fe38700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf380fe1d0 con 0x7fcf380d1880 2021-02-10T11:31:40.598 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:40.581 7fcf3fe38700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf381b0cc0 con 0x7fcf380cf590 2021-02-10T11:31:40.598 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:40.581 7fcf3fe38700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf38109a70 con 0x7fcf380d1c80 2021-02-10T11:31:40.598 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:40.581 7fcf3e455700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fcf380cf590 0x7fcf380cf990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:40.598 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:40.581 7fcf3dc54700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fcf380d1880 0x7fcf381b04d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:54240/0 (socket says 172.21.2.102:54240) 2021-02-10T11:31:40.599 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:40.581 7fcf3dc54700 1 -- 172.21.2.102:0/2129785224 learned_addr learned my addr 172.21.2.102:0/2129785224 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:31:40.599 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:40.581 7fcf3e455700 1 -- 172.21.2.102:0/2129785224 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcf380d1c80 msgr2=0x7fcf380cef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:40.599 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:40.581 7fcf3e455700 1 --2- 172.21.2.102:0/2129785224 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcf380d1c80 0x7fcf380cef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:40.599 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:40.581 7fcf3e455700 1 -- 172.21.2.102:0/2129785224 >> v1:172.21.2.102:6789/0 conn(0x7fcf380d1880 legacy=0x7fcf381b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:40.599 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:40.581 7fcf3e455700 1 -- 172.21.2.102:0/2129785224 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf38088070 con 0x7fcf380cf590 2021-02-10T11:31:40.599 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:40.581 7fcf3e455700 1 --2- 172.21.2.102:0/2129785224 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcf380cf590 0x7fcf380cf990 secure :-1 s=READY pgs=2155 cs=0 l=1 rev1=1 rx=0x7fcf30006e30 tx=0x7fcf30009e60).ready entity=mon.0 client_cookie=dd750c85694162bd server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:40.599 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:40.581 7fcf3cc52700 1 -- 172.21.2.102:0/2129785224 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcf30006a50 con 0x7fcf380cf590 2021-02-10T11:31:40.599 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:40.581 7fcf3cc52700 1 -- 172.21.2.102:0/2129785224 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf3000b300 con 0x7fcf380cf590 2021-02-10T11:31:40.599 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:40.581 7fcf3cc52700 1 -- 172.21.2.102:0/2129785224 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcf3000b820 con 0x7fcf380cf590 2021-02-10T11:31:40.599 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:40.582 7fcf3fe38700 1 -- 172.21.2.102:0/2129785224 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcf380cf590 msgr2=0x7fcf380cf990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:40.599 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:40.582 7fcf3fe38700 1 --2- 172.21.2.102:0/2129785224 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcf380cf590 0x7fcf380cf990 secure :-1 s=READY pgs=2155 cs=0 l=1 rev1=1 rx=0x7fcf30006e30 tx=0x7fcf30009e60).stop 2021-02-10T11:31:40.599 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:40.582 7fcf3fe38700 1 -- 172.21.2.102:0/2129785224 shutdown_connections 2021-02-10T11:31:40.600 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:40.582 7fcf3fe38700 1 --2- 172.21.2.102:0/2129785224 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcf380cf590 0x7fcf380cf990 unknown :-1 s=CLOSED pgs=2155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:40.600 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:40.582 7fcf3fe38700 1 --2- 172.21.2.102:0/2129785224 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcf380d1c80 0x7fcf380cef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:40.600 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:40.582 7fcf3fe38700 1 -- 172.21.2.102:0/2129785224 >> 172.21.2.102:0/2129785224 conn(0x7fcf380ce430 msgr2=0x7fcf380d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:40.600 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:40.582 7fcf3fe38700 1 -- 172.21.2.102:0/2129785224 shutdown_connections 2021-02-10T11:31:40.600 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:40.582 7fcf3fe38700 1 -- 172.21.2.102:0/2129785224 wait complete. 2021-02-10T11:31:40.602 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:40.597 7fcf3fe38700 1 Processor -- start 2021-02-10T11:31:40.602 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:40.597 7fcf3fe38700 1 -- start start 2021-02-10T11:31:40.602 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:40.597 7fcf3fe38700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcf3819e5b0 0x7fcf381afc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:40.602 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:40.597 7fcf3fe38700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcf3819e9b0 0x7fcf381b0dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:40.602 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:40.597 7fcf3fe38700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fcf381b0250 0x7fcf381b13e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:40.603 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:40.597 7fcf3fe38700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf38195bd0 con 0x7fcf3819e5b0 2021-02-10T11:31:40.603 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:40.597 7fcf3fe38700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf381b0840 con 0x7fcf381b0250 2021-02-10T11:31:40.603 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:40.597 7fcf3fe38700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf3819eed0 con 0x7fcf3819e9b0 2021-02-10T11:31:40.603 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:40.597 7fcf3dc54700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcf3819e5b0 0x7fcf381afc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:40.603 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:40.597 7fcf3dc54700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcf3819e5b0 0x7fcf381afc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:33788/0 (socket says 172.21.2.102:33788) 2021-02-10T11:31:40.603 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:40.597 7fcf3dc54700 1 -- 172.21.2.102:0/3931372106 learned_addr learned my addr 172.21.2.102:0/3931372106 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:31:40.603 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:40.598 7fcf3d453700 1 --2- 172.21.2.102:0/3931372106 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcf3819e9b0 0x7fcf381b0dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:40.603 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:40.598 7fcf3e455700 1 --2- 172.21.2.102:0/3931372106 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fcf381b0250 0x7fcf381b13e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:40.603 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:40.598 7fcf3dc54700 1 -- 172.21.2.102:0/3931372106 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcf3819e9b0 msgr2=0x7fcf381b0dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:40.603 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:40.598 7fcf3dc54700 1 --2- 172.21.2.102:0/3931372106 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcf3819e9b0 0x7fcf381b0dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:40.603 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:40.598 7fcf3dc54700 1 -- 172.21.2.102:0/3931372106 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fcf381b0250 msgr2=0x7fcf381b13e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:40.603 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:40.598 7fcf3dc54700 1 --2- 172.21.2.102:0/3931372106 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fcf381b0250 0x7fcf381b13e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:40.603 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:40.598 7fcf3dc54700 1 -- 172.21.2.102:0/3931372106 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf3819f0b0 con 0x7fcf3819e5b0 2021-02-10T11:31:40.604 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:40.598 7fcf3e455700 1 --2- 172.21.2.102:0/3931372106 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fcf381b0250 0x7fcf381b13e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:31:40.604 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:40.598 7fcf3dc54700 1 --2- 172.21.2.102:0/3931372106 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcf3819e5b0 0x7fcf381afc30 secure :-1 s=READY pgs=2156 cs=0 l=1 rev1=1 rx=0x7fcf34001c60 tx=0x7fcf3400a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:40.604 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:40.598 7fcf2a7fc700 1 -- 172.21.2.102:0/3931372106 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcf3400acc0 con 0x7fcf3819e5b0 2021-02-10T11:31:40.604 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:40.598 7fcf2a7fc700 1 -- 172.21.2.102:0/3931372106 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf34004140 con 0x7fcf3819e5b0 2021-02-10T11:31:40.604 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:40.598 7fcf3fe38700 1 -- 172.21.2.102:0/3931372106 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf3819f0b0 con 0x7fcf3819e5b0 2021-02-10T11:31:40.604 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:40.598 7fcf2a7fc700 1 -- 172.21.2.102:0/3931372106 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcf340045a0 con 0x7fcf3819e5b0 2021-02-10T11:31:40.604 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:40.599 7fcf3fe38700 1 -- 172.21.2.102:0/3931372106 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf381b1cf0 con 0x7fcf3819e5b0 2021-02-10T11:31:40.604 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:40.599 7fcf2a7fc700 1 -- 172.21.2.102:0/3931372106 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fcf34000db0 con 0x7fcf3819e5b0 2021-02-10T11:31:40.604 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:40.599 7fcf2a7fc700 1 --2- 172.21.2.102:0/3931372106 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcf20018730 0x7fcf2001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:40.604 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:40.599 7fcf2a7fc700 1 -- 172.21.2.102:0/3931372106 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 5659+0+0 (secure 0 0 0) 0x7fcf34032640 con 0x7fcf3819e5b0 2021-02-10T11:31:40.604 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:40.599 7fcf3d453700 1 --2- 172.21.2.102:0/3931372106 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcf20018730 0x7fcf2001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:40.604 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:40.599 7fcf3d453700 1 --2- 172.21.2.102:0/3931372106 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcf20018730 0x7fcf2001ac10 crc :-1 s=READY pgs=954 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:40.632 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:40.631 7fcf3fe38700 1 -- 172.21.2.102:0/3931372106 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf381b2210 con 0x7fcf3819e5b0 2021-02-10T11:31:40.634 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:40.633 7fcf2a7fc700 1 -- 172.21.2.102:0/3931372106 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fcf340009b0 con 0x7fcf3819e5b0 2021-02-10T11:31:40.763 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:40.762 7fcf3fe38700 1 -- 172.21.2.102:0/3931372106 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"pool2": "cephfs_metadata", "prefix": "osd pool delete", "pool": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fcf381b2210 con 0x7fcf3819e5b0 2021-02-10T11:31:41.115 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:41.114 7fcf2a7fc700 1 -- 172.21.2.102:0/3931372106 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"pool2": "cephfs_metadata", "prefix": "osd pool delete", "pool": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v105) v1 ==== 185+0+0 (secure 0 0 0) 0x7fcf3401f4b0 con 0x7fcf3819e5b0 2021-02-10T11:31:41.127 INFO:teuthology.orchestra.run.gibba002.stderr:pool 'cephfs_metadata' removed 2021-02-10T11:31:41.129 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:41.127 7fcf3fe38700 1 -- 172.21.2.102:0/3931372106 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcf20018730 msgr2=0x7fcf2001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:41.129 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:41.127 7fcf3fe38700 1 --2- 172.21.2.102:0/3931372106 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcf20018730 0x7fcf2001ac10 crc :-1 s=READY pgs=954 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:41.129 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:41.127 7fcf3fe38700 1 -- 172.21.2.102:0/3931372106 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcf3819e5b0 msgr2=0x7fcf381afc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:41.129 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:41.127 7fcf3fe38700 1 --2- 172.21.2.102:0/3931372106 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcf3819e5b0 0x7fcf381afc30 secure :-1 s=READY pgs=2156 cs=0 l=1 rev1=1 rx=0x7fcf34001c60 tx=0x7fcf3400a8d0).stop 2021-02-10T11:31:41.129 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:41.127 7fcf3fe38700 1 -- 172.21.2.102:0/3931372106 shutdown_connections 2021-02-10T11:31:41.129 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:41.127 7fcf3fe38700 1 --2- 172.21.2.102:0/3931372106 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcf20018730 0x7fcf2001ac10 unknown :-1 s=CLOSED pgs=954 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:41.129 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:41.127 7fcf3fe38700 1 --2- 172.21.2.102:0/3931372106 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fcf381b0250 0x7fcf381b13e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:41.129 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:41.127 7fcf3fe38700 1 --2- 172.21.2.102:0/3931372106 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcf3819e9b0 0x7fcf381b0dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:41.129 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:41.127 7fcf3fe38700 1 --2- 172.21.2.102:0/3931372106 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcf3819e5b0 0x7fcf381afc30 unknown :-1 s=CLOSED pgs=2156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:41.129 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:41.127 7fcf3fe38700 1 -- 172.21.2.102:0/3931372106 >> 172.21.2.102:0/3931372106 conn(0x7fcf3819f940 msgr2=0x7fcf3818c290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:41.129 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:41.127 7fcf3fe38700 1 -- 172.21.2.102:0/3931372106 shutdown_connections 2021-02-10T11:31:41.130 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:41.127 7fcf3fe38700 1 -- 172.21.2.102:0/3931372106 wait complete. 2021-02-10T11:31:41.137 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool delete cephfs_data cephfs_data --yes-i-really-really-mean-it 2021-02-10T11:31:41.571 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:41.552 7f6cbeace700 1 Processor -- start 2021-02-10T11:31:41.571 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:41.552 7f6cbeace700 1 -- start start 2021-02-10T11:31:41.571 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:41.553 7f6cbeace700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6cb810b000 0x7f6cb80cf760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:41.571 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:41.553 7f6cbeace700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cb80fe1d0 con 0x7f6cb80d1880 2021-02-10T11:31:41.571 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:41.553 7f6cbeace700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cb81b0cc0 con 0x7f6cb80d1c80 2021-02-10T11:31:41.572 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:41.553 7f6cbeace700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6cb8109a70 con 0x7f6cb810b000 2021-02-10T11:31:41.572 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:41.553 7f6cbc8ea700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f6cb80d1880 0x7f6cb81b04d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:54252/0 (socket says 172.21.2.102:54252) 2021-02-10T11:31:41.572 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:41.553 7f6cbc8ea700 1 -- 172.21.2.102:0/3248447465 learned_addr learned my addr 172.21.2.102:0/3248447465 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:31:41.572 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:41.553 7f6cb77fe700 1 -- 172.21.2.102:0/3248447465 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2049671947 0 0) 0x7f6ca00009b0 con 0x7f6cb80d1880 2021-02-10T11:31:41.572 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:41.553 7f6cb77fe700 1 -- 172.21.2.102:0/3248447465 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ca4002fe0 con 0x7f6cb80d1880 2021-02-10T11:31:41.572 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:41.553 7f6cb77fe700 1 -- 172.21.2.102:0/3248447465 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2065485738 0 0) 0x7f6ca0001010 con 0x7f6cb80d1880 2021-02-10T11:31:41.572 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:41.553 7f6cb77fe700 1 -- 172.21.2.102:0/3248447465 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6cb810b000 msgr2=0x7f6cb80cf760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:41.572 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:41.553 7f6cbd0eb700 1 --2- 172.21.2.102:0/3248447465 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6cb810b000 0x7f6cb80cf760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:41.572 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:41.554 7f6cb77fe700 1 --2- 172.21.2.102:0/3248447465 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6cb810b000 0x7f6cb80cf760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:41.572 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:41.554 7f6cb77fe700 1 -- 172.21.2.102:0/3248447465 >> v1:172.21.2.113:6789/0 conn(0x7f6cb80d1c80 legacy=0x7f6cb80cef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:41.572 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:41.554 7f6cb77fe700 1 -- 172.21.2.102:0/3248447465 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cb810b400 con 0x7f6cb80d1880 2021-02-10T11:31:41.573 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:41.554 7f6cb77fe700 1 -- 172.21.2.102:0/3248447465 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7f6ca0001590 con 0x7f6cb80d1880 2021-02-10T11:31:41.573 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:41.554 7f6cb77fe700 1 -- 172.21.2.102:0/3248447465 >> v1:172.21.2.102:6789/0 conn(0x7f6cb80d1880 legacy=0x7f6cb81b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:41.573 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:41.554 7f6cb77fe700 1 --2- 172.21.2.102:0/3248447465 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6ca4003af0 0x7f6ca4005fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:41.573 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:41.554 7f6cb77fe700 1 --2- 172.21.2.102:0/3248447465 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6ca4006800 0x7f6ca4008d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:41.573 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:41.554 7f6cb77fe700 1 --2- 172.21.2.102:0/3248447465 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6ca40095c0 0x7f6ca400bae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:41.573 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:41.554 7f6cb77fe700 1 -- 172.21.2.102:0/3248447465 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ca400c370 con 0x7f6ca40095c0 2021-02-10T11:31:41.573 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:41.554 7f6cb77fe700 1 -- 172.21.2.102:0/3248447465 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ca400c590 con 0x7f6ca4006800 2021-02-10T11:31:41.573 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:41.554 7f6cb77fe700 1 -- 172.21.2.102:0/3248447465 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ca400c7b0 con 0x7f6ca4003af0 2021-02-10T11:31:41.573 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:41.554 7f6cbd0eb700 1 --2- 172.21.2.102:0/3248447465 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6cb810b000 0x7f6cb80cf760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:31:41.573 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:41.554 7f6cbd0eb700 1 --2- 172.21.2.102:0/3248447465 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6ca40095c0 0x7f6ca400bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:41.573 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:41.554 7f6cbc8ea700 1 --2- 172.21.2.102:0/3248447465 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6ca4003af0 0x7f6ca4005fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:41.573 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:41.554 7f6cb7fff700 1 --2- 172.21.2.102:0/3248447465 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6ca4006800 0x7f6ca4008d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:41.573 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:41.554 7f6cbd0eb700 1 -- 172.21.2.102:0/3248447465 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6ca4003af0 msgr2=0x7f6ca4005fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:41.574 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:41.554 7f6cbd0eb700 1 --2- 172.21.2.102:0/3248447465 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6ca4003af0 0x7f6ca4005fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:41.574 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:41.554 7f6cbd0eb700 1 -- 172.21.2.102:0/3248447465 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6ca4006800 msgr2=0x7f6ca4008d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:41.574 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:41.554 7f6cbd0eb700 1 --2- 172.21.2.102:0/3248447465 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6ca4006800 0x7f6ca4008d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:41.574 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:41.554 7f6cbd0eb700 1 -- 172.21.2.102:0/3248447465 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ca400ca50 con 0x7f6ca40095c0 2021-02-10T11:31:41.574 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:41.554 7f6cbd0eb700 1 --2- 172.21.2.102:0/3248447465 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6ca40095c0 0x7f6ca400bae0 secure :-1 s=READY pgs=2158 cs=0 l=1 rev1=1 rx=0x7f6cb0001140 tx=0x7f6cb004dde0).ready entity=mon.0 client_cookie=8a8d1e404347d829 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:41.574 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:41.554 7f6cb77fe700 1 -- 172.21.2.102:0/3248447465 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6cb0006240 con 0x7f6ca40095c0 2021-02-10T11:31:41.574 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:41.554 7f6cb77fe700 1 -- 172.21.2.102:0/3248447465 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cb0000f60 con 0x7f6ca40095c0 2021-02-10T11:31:41.574 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:41.554 7f6cb77fe700 1 -- 172.21.2.102:0/3248447465 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6cb00056b0 con 0x7f6ca40095c0 2021-02-10T11:31:41.574 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:41.554 7f6cbeace700 1 -- 172.21.2.102:0/3248447465 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6ca40095c0 msgr2=0x7f6ca400bae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:41.574 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:41.554 7f6cbeace700 1 --2- 172.21.2.102:0/3248447465 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6ca40095c0 0x7f6ca400bae0 secure :-1 s=READY pgs=2158 cs=0 l=1 rev1=1 rx=0x7f6cb0001140 tx=0x7f6cb004dde0).stop 2021-02-10T11:31:41.574 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:41.554 7f6cbd0eb700 1 -- 172.21.2.102:0/3248447465 reap_dead start 2021-02-10T11:31:41.574 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:41.555 7f6cbeace700 1 -- 172.21.2.102:0/3248447465 shutdown_connections 2021-02-10T11:31:41.575 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:41.555 7f6cbeace700 1 -- 172.21.2.102:0/3248447465 >> 172.21.2.102:0/3248447465 conn(0x7f6cb80ce430 msgr2=0x7f6cb80d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:41.575 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:41.555 7f6cbeace700 1 -- 172.21.2.102:0/3248447465 shutdown_connections 2021-02-10T11:31:41.575 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:41.555 7f6cbeace700 1 -- 172.21.2.102:0/3248447465 wait complete. 2021-02-10T11:31:41.575 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:41.570 7f6cbeace700 1 Processor -- start 2021-02-10T11:31:41.575 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:41.570 7f6cbeace700 1 -- start start 2021-02-10T11:31:41.576 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:41.570 7f6cbeace700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6cb8188400 0x7f6cb8188800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:41.576 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:41.570 7f6cbeace700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6cb8188e20 0x7f6cb8189220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:41.576 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:41.570 7f6cbeace700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6cb809cbc0 0x7f6cb809cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:41.576 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:41.570 7f6cbeace700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cb8187e80 con 0x7f6cb8188400 2021-02-10T11:31:41.577 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:41.570 7f6cbeace700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cb80b27c0 con 0x7f6cb8188e20 2021-02-10T11:31:41.577 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:41.570 7f6cbeace700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6cb80b29a0 con 0x7f6cb809cbc0 2021-02-10T11:31:41.577 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:41.570 7f6cbc8ea700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6cb8188400 0x7f6cb8188800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:41.577 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:41.570 7f6cbc8ea700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6cb8188400 0x7f6cb8188800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:33810/0 (socket says 172.21.2.102:33810) 2021-02-10T11:31:41.577 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:41.570 7f6cbc8ea700 1 -- 172.21.2.102:0/195172105 learned_addr learned my addr 172.21.2.102:0/195172105 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:31:41.577 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:41.570 7f6cb7fff700 1 --2- 172.21.2.102:0/195172105 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6cb8188e20 0x7f6cb8189220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:41.577 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:41.571 7f6cbd0eb700 1 --2- 172.21.2.102:0/195172105 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6cb809cbc0 0x7f6cb809cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:41.577 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:41.571 7f6cbc8ea700 1 -- 172.21.2.102:0/195172105 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6cb809cbc0 msgr2=0x7f6cb809cfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:41.577 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:41.571 7f6cbc8ea700 1 --2- 172.21.2.102:0/195172105 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6cb809cbc0 0x7f6cb809cfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:41.577 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:41.571 7f6cbc8ea700 1 -- 172.21.2.102:0/195172105 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6cb8188e20 msgr2=0x7f6cb8189220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:41.577 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:41.571 7f6cbc8ea700 1 --2- 172.21.2.102:0/195172105 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6cb8188e20 0x7f6cb8189220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:41.578 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:41.571 7f6cbc8ea700 1 -- 172.21.2.102:0/195172105 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cb809d5e0 con 0x7f6cb8188400 2021-02-10T11:31:41.578 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:41.571 7f6cbc8ea700 1 --2- 172.21.2.102:0/195172105 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6cb8188400 0x7f6cb8188800 secure :-1 s=READY pgs=2159 cs=0 l=1 rev1=1 rx=0x7f6ca0001660 tx=0x7f6ca000a930).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:41.578 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:41.571 7f6cb57fa700 1 -- 172.21.2.102:0/195172105 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6ca000ad20 con 0x7f6cb8188400 2021-02-10T11:31:41.578 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:41.571 7f6cb57fa700 1 -- 172.21.2.102:0/195172105 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ca00016a0 con 0x7f6cb8188400 2021-02-10T11:31:41.578 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:41.571 7f6cb57fa700 1 -- 172.21.2.102:0/195172105 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6ca0004330 con 0x7f6cb8188400 2021-02-10T11:31:41.578 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:41.571 7f6cbeace700 1 -- 172.21.2.102:0/195172105 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6cb809d5e0 con 0x7f6cb8188400 2021-02-10T11:31:41.578 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:41.571 7f6cbeace700 1 -- 172.21.2.102:0/195172105 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6cb809d5e0 con 0x7f6cb8188400 2021-02-10T11:31:41.578 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:41.571 7f6cb57fa700 1 -- 172.21.2.102:0/195172105 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f6ca0004140 con 0x7f6cb8188400 2021-02-10T11:31:41.578 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:41.571 7f6cb57fa700 1 --2- 172.21.2.102:0/195172105 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6c9c0186c0 0x7f6c9c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:41.578 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:41.572 7f6cb57fa700 1 -- 172.21.2.102:0/195172105 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 5221+0+0 (secure 0 0 0) 0x7f6ca00323c0 con 0x7f6cb8188400 2021-02-10T11:31:41.578 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:41.572 7f6cb7fff700 1 --2- 172.21.2.102:0/195172105 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6c9c0186c0 0x7f6c9c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:41.578 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:41.572 7f6cb7fff700 1 --2- 172.21.2.102:0/195172105 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6c9c0186c0 0x7f6c9c01abc0 crc :-1 s=READY pgs=955 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:41.603 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:41.602 7f6cbeace700 1 -- 172.21.2.102:0/195172105 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6cb809db70 con 0x7f6cb8188400 2021-02-10T11:31:41.606 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:41.605 7f6cb57fa700 1 -- 172.21.2.102:0/195172105 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f6ca0001180 con 0x7f6cb8188400 2021-02-10T11:31:41.733 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:41.732 7f6cbeace700 1 -- 172.21.2.102:0/195172105 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"pool2": "cephfs_data", "prefix": "osd pool delete", "pool": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f6cb8082a70 con 0x7f6cb8188400 2021-02-10T11:31:42.122 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:42.120 7f6cb57fa700 1 -- 172.21.2.102:0/195172105 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"pool2": "cephfs_data", "prefix": "osd pool delete", "pool": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v106) v1 ==== 173+0+0 (secure 0 0 0) 0x7f6ca0004140 con 0x7f6cb8188400 2021-02-10T11:31:42.148 INFO:teuthology.orchestra.run.gibba002.stderr:pool 'cephfs_data' removed 2021-02-10T11:31:42.150 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:42.149 7f6cbeace700 1 -- 172.21.2.102:0/195172105 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6c9c0186c0 msgr2=0x7f6c9c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:42.150 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:42.149 7f6cbeace700 1 --2- 172.21.2.102:0/195172105 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6c9c0186c0 0x7f6c9c01abc0 crc :-1 s=READY pgs=955 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:42.150 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:42.149 7f6cbeace700 1 -- 172.21.2.102:0/195172105 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6cb8188400 msgr2=0x7f6cb8188800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:42.150 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:42.149 7f6cbeace700 1 --2- 172.21.2.102:0/195172105 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6cb8188400 0x7f6cb8188800 secure :-1 s=READY pgs=2159 cs=0 l=1 rev1=1 rx=0x7f6ca0001660 tx=0x7f6ca000a930).stop 2021-02-10T11:31:42.151 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:42.149 7f6cbeace700 1 -- 172.21.2.102:0/195172105 shutdown_connections 2021-02-10T11:31:42.151 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:42.149 7f6cbeace700 1 --2- 172.21.2.102:0/195172105 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6cb809cbc0 0x7f6cb809cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:42.151 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:42.149 7f6cbeace700 1 --2- 172.21.2.102:0/195172105 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6c9c0186c0 0x7f6c9c01abc0 unknown :-1 s=CLOSED pgs=955 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:42.151 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:42.149 7f6cbeace700 1 --2- 172.21.2.102:0/195172105 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6cb8188e20 0x7f6cb8189220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:42.151 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:42.149 7f6cbeace700 1 --2- 172.21.2.102:0/195172105 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6cb8188400 0x7f6cb8188800 unknown :-1 s=CLOSED pgs=2159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:42.151 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:42.149 7f6cbeace700 1 -- 172.21.2.102:0/195172105 >> 172.21.2.102:0/195172105 conn(0x7f6cb80c2cd0 msgr2=0x7f6cb80b5950 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:42.151 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:42.149 7f6cbeace700 1 -- 172.21.2.102:0/195172105 shutdown_connections 2021-02-10T11:31:42.151 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:42.149 7f6cbeace700 1 -- 172.21.2.102:0/195172105 wait complete. 2021-02-10T11:31:42.158 INFO:tasks.ceph.mds.a:Restarting daemon 2021-02-10T11:31:42.159 INFO:tasks.ceph.mds.a:Stopping old one... 2021-02-10T11:31:42.159 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2021-02-10T11:31:42.159 INFO:teuthology.orchestra.run:waiting for 300 2021-02-10T11:31:42.159 INFO:tasks.ceph.mds.b:Restarting daemon 2021-02-10T11:31:42.159 INFO:tasks.ceph.mds.b:Stopping old one... 2021-02-10T11:31:42.159 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2021-02-10T11:31:42.159 INFO:teuthology.orchestra.run:waiting for 300 2021-02-10T11:31:42.168 INFO:tasks.ceph.mds.b:Stopped 2021-02-10T11:31:42.169 DEBUG:teuthology.orchestra.run.gibba013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2021-02-10T11:31:42.169 INFO:tasks.ceph.mds.a:Stopped 2021-02-10T11:31:42.169 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2021-02-10T11:31:42.210 INFO:tasks.ceph.mds.b:Started 2021-02-10T11:31:42.210 DEBUG:teuthology.parallel:result is None 2021-02-10T11:31:42.211 INFO:tasks.ceph.mds.a:Started 2021-02-10T11:31:42.211 DEBUG:teuthology.parallel:result is None 2021-02-10T11:31:42.212 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd blacklist clear 2021-02-10T11:31:42.614 INFO:tasks.ceph.mds.b.gibba013.stdout:starting mds.b at 2021-02-10T11:31:42.618 INFO:tasks.ceph.mds.a.gibba002.stdout:starting mds.a at 2021-02-10T11:31:42.638 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:42.620 7f397a962700 1 Processor -- start 2021-02-10T11:31:42.638 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:42.620 7f397a962700 1 -- start start 2021-02-10T11:31:42.638 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:42.620 7f397a962700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f39740d0340 0x7f39740ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:42.638 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:42.620 7f397a962700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f39740d1c70 0x7f39740efc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:42.638 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:42.620 7f397a962700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39740f0b10 con 0x7f39740d1870 2021-02-10T11:31:42.638 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:42.620 7f397a962700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39740e5ed0 con 0x7f39740d0340 2021-02-10T11:31:42.638 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:42.620 7f397a962700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39741b0be0 con 0x7f39740d1c70 2021-02-10T11:31:42.638 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:42.621 7f396bfff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f39740d0340 0x7f39740ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:42.639 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:42.621 7f396bfff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f39740d0340 0x7f39740ef640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:33826/0 (socket says 172.21.2.102:33826) 2021-02-10T11:31:42.639 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:42.621 7f396bfff700 1 -- 172.21.2.102:0/2870045962 learned_addr learned my addr 172.21.2.102:0/2870045962 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:31:42.639 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:42.621 7f3978f7f700 1 --2- 172.21.2.102:0/2870045962 >> v2:172.21.2.113:3300/0 conn(0x7f39740d1c70 0x7f39740efc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:42.639 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:42.621 7f396bfff700 1 -- 172.21.2.102:0/2870045962 >> v2:172.21.2.113:3300/0 conn(0x7f39740d1c70 msgr2=0x7f39740efc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:42.639 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:42.621 7f396bfff700 1 --2- 172.21.2.102:0/2870045962 >> v2:172.21.2.113:3300/0 conn(0x7f39740d1c70 0x7f39740efc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:42.639 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:42.621 7f396bfff700 1 -- 172.21.2.102:0/2870045962 >> v1:172.21.2.113:6789/0 conn(0x7f39740d1870 legacy=0x7f39741b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:42.639 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:42.621 7f396bfff700 1 -- 172.21.2.102:0/2870045962 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39740f20a0 con 0x7f39740d0340 2021-02-10T11:31:42.639 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:42.621 7f396bfff700 1 --2- 172.21.2.102:0/2870045962 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f39740d0340 0x7f39740ef640 secure :-1 s=READY pgs=2165 cs=0 l=1 rev1=1 rx=0x7f3964006a80 tx=0x7f3964009f00).ready entity=mon.0 client_cookie=32d9ffdd40f305fd server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:42.639 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:42.621 7f39737fe700 1 -- 172.21.2.102:0/2870045962 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3964000d30 con 0x7f39740d0340 2021-02-10T11:31:42.639 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:42.621 7f39737fe700 1 -- 172.21.2.102:0/2870045962 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f396400b5c0 con 0x7f39740d0340 2021-02-10T11:31:42.639 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:42.621 7f39737fe700 1 -- 172.21.2.102:0/2870045962 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f396400ba50 con 0x7f39740d0340 2021-02-10T11:31:42.640 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:42.621 7f397a962700 1 -- 172.21.2.102:0/2870045962 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f39740d0340 msgr2=0x7f39740ef640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:42.640 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:42.621 7f397a962700 1 --2- 172.21.2.102:0/2870045962 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f39740d0340 0x7f39740ef640 secure :-1 s=READY pgs=2165 cs=0 l=1 rev1=1 rx=0x7f3964006a80 tx=0x7f3964009f00).stop 2021-02-10T11:31:42.640 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:42.622 7f397a962700 1 -- 172.21.2.102:0/2870045962 shutdown_connections 2021-02-10T11:31:42.640 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:42.622 7f397a962700 1 --2- 172.21.2.102:0/2870045962 >> v2:172.21.2.113:3300/0 conn(0x7f39740d1c70 0x7f39740efc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:42.640 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:42.622 7f397a962700 1 --2- 172.21.2.102:0/2870045962 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f39740d0340 0x7f39740ef640 unknown :-1 s=CLOSED pgs=2165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:42.640 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:42.622 7f397a962700 1 -- 172.21.2.102:0/2870045962 >> 172.21.2.102:0/2870045962 conn(0x7f39740ce450 msgr2=0x7f39740d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:42.640 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:42.622 7f397a962700 1 -- 172.21.2.102:0/2870045962 shutdown_connections 2021-02-10T11:31:42.640 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:42.622 7f397a962700 1 -- 172.21.2.102:0/2870045962 wait complete. 2021-02-10T11:31:42.640 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:42.637 7f397a962700 1 Processor -- start 2021-02-10T11:31:42.640 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:42.637 7f397a962700 1 -- start start 2021-02-10T11:31:42.642 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:42.637 7f397a962700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f397419ffa0 0x7f39740eda10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:42.642 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:42.637 7f397a962700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f39741a03a0 0x7f39740ee030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:42.642 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:42.637 7f397a962700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f39740ee650 0x7f39740ebe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:42.642 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:42.637 7f397a962700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39740ed490 con 0x7f39741a03a0 2021-02-10T11:31:42.642 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:42.637 7f397a962700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39740eea50 con 0x7f397419ffa0 2021-02-10T11:31:42.642 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:42.637 7f397a962700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39740eec30 con 0x7f39740ee650 2021-02-10T11:31:42.642 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:42.637 7f396bfff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f39741a03a0 0x7f39740ee030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:42.642 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:42.638 7f396bfff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f39741a03a0 0x7f39740ee030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:33832/0 (socket says 172.21.2.102:33832) 2021-02-10T11:31:42.642 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:42.638 7f396bfff700 1 -- 172.21.2.102:0/1641322238 learned_addr learned my addr 172.21.2.102:0/1641322238 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:31:42.642 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:42.638 7f396bfff700 1 -- 172.21.2.102:0/1641322238 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f39740ee650 msgr2=0x7f39740ebe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:42.643 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:42.638 7f396bfff700 1 --2- 172.21.2.102:0/1641322238 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f39740ee650 0x7f39740ebe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:42.643 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:42.638 7f396bfff700 1 -- 172.21.2.102:0/1641322238 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f397419ffa0 msgr2=0x7f39740eda10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:42.643 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:42.638 7f396bfff700 1 --2- 172.21.2.102:0/1641322238 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f397419ffa0 0x7f39740eda10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:42.643 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:42.638 7f396bfff700 1 -- 172.21.2.102:0/1641322238 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39740ec4a0 con 0x7f39741a03a0 2021-02-10T11:31:42.643 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:42.638 7f396bfff700 1 --2- 172.21.2.102:0/1641322238 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f39741a03a0 0x7f39740ee030 secure :-1 s=READY pgs=2166 cs=0 l=1 rev1=1 rx=0x7f3964006a90 tx=0x7f396400bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:42.643 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:42.638 7f39717fa700 1 -- 172.21.2.102:0/1641322238 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3964000d30 con 0x7f39741a03a0 2021-02-10T11:31:42.643 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:42.638 7f39717fa700 1 -- 172.21.2.102:0/1641322238 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3964006710 con 0x7f39741a03a0 2021-02-10T11:31:42.643 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:42.638 7f39717fa700 1 -- 172.21.2.102:0/1641322238 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f396400c570 con 0x7f39741a03a0 2021-02-10T11:31:42.643 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:42.638 7f397a962700 1 -- 172.21.2.102:0/1641322238 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39740ec4a0 con 0x7f39741a03a0 2021-02-10T11:31:42.643 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:42.638 7f397a962700 1 -- 172.21.2.102:0/1641322238 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39740ec4a0 con 0x7f39741a03a0 2021-02-10T11:31:42.643 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:42.639 7f39717fa700 1 -- 172.21.2.102:0/1641322238 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f3964007000 con 0x7f39741a03a0 2021-02-10T11:31:42.643 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:42.639 7f39717fa700 1 --2- 172.21.2.102:0/1641322238 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f39580186c0 0x7f395801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:42.643 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:42.639 7f39717fa700 1 -- 172.21.2.102:0/1641322238 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 4839+0+0 (secure 0 0 0) 0x7f3964032810 con 0x7f39741a03a0 2021-02-10T11:31:42.644 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:42.639 7f3973fff700 1 --2- 172.21.2.102:0/1641322238 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f39580186c0 0x7f395801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:42.644 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:42.639 7f3973fff700 1 --2- 172.21.2.102:0/1641322238 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f39580186c0 0x7f395801abc0 crc :-1 s=READY pgs=956 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:42.670 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:42.669 7f397a962700 1 -- 172.21.2.102:0/1641322238 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39740ebc50 con 0x7f39741a03a0 2021-02-10T11:31:42.674 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:42.672 7f39717fa700 1 -- 172.21.2.102:0/1641322238 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f396401f830 con 0x7f39741a03a0 2021-02-10T11:31:42.812 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:42.811 7f397a962700 1 -- 172.21.2.102:0/1641322238 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd blacklist clear"} v 0) v1 -- 0x7f39740ebc50 con 0x7f39741a03a0 2021-02-10T11:31:43.164 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:43.163 7f39717fa700 1 -- 172.21.2.102:0/1641322238 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd blacklist clear"}]=0 removed all blacklist entries v107) v1 ==== 97+0+0 (secure 0 0 0) 0x7f3964007000 con 0x7f39741a03a0 2021-02-10T11:31:43.178 INFO:teuthology.orchestra.run.gibba002.stderr: removed all blacklist entries 2021-02-10T11:31:43.179 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:43.178 7f397a962700 1 -- 172.21.2.102:0/1641322238 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f39580186c0 msgr2=0x7f395801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:43.180 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:43.178 7f397a962700 1 --2- 172.21.2.102:0/1641322238 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f39580186c0 0x7f395801abc0 crc :-1 s=READY pgs=956 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:43.180 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:43.178 7f397a962700 1 -- 172.21.2.102:0/1641322238 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f39741a03a0 msgr2=0x7f39740ee030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:43.180 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:43.178 7f397a962700 1 --2- 172.21.2.102:0/1641322238 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f39741a03a0 0x7f39740ee030 secure :-1 s=READY pgs=2166 cs=0 l=1 rev1=1 rx=0x7f3964006a90 tx=0x7f396400bd10).stop 2021-02-10T11:31:43.180 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:43.178 7f397a962700 1 -- 172.21.2.102:0/1641322238 shutdown_connections 2021-02-10T11:31:43.180 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:43.178 7f397a962700 1 --2- 172.21.2.102:0/1641322238 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f39740ee650 0x7f39740ebe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:43.180 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:43.178 7f397a962700 1 --2- 172.21.2.102:0/1641322238 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f39741a03a0 0x7f39740ee030 unknown :-1 s=CLOSED pgs=2166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:43.180 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:43.178 7f397a962700 1 --2- 172.21.2.102:0/1641322238 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f39580186c0 0x7f395801abc0 unknown :-1 s=CLOSED pgs=956 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:43.180 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:43.178 7f397a962700 1 --2- 172.21.2.102:0/1641322238 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f397419ffa0 0x7f39740eda10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:43.180 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:43.178 7f397a962700 1 -- 172.21.2.102:0/1641322238 >> 172.21.2.102:0/1641322238 conn(0x7f3974094d90 msgr2=0x7f397408a420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:43.180 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:43.178 7f397a962700 1 -- 172.21.2.102:0/1641322238 shutdown_connections 2021-02-10T11:31:43.181 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:43.178 7f397a962700 1 -- 172.21.2.102:0/1641322238 wait complete. 2021-02-10T11:31:43.188 INFO:tasks.cephfs.cephfs_test_case:['0', '1', '2', '3'] 2021-02-10T11:31:43.188 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth ls --format=json-pretty 2021-02-10T11:31:43.618 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:43.599 7fd3a01ed700 1 Processor -- start 2021-02-10T11:31:43.619 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:43.599 7fd3a01ed700 1 -- start start 2021-02-10T11:31:43.619 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:43.599 7fd3a01ed700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd3980d0340 0x7fd3980ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:43.619 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:43.599 7fd3a01ed700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fd3980d1c70 0x7fd3980efc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:43.619 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:43.599 7fd3a01ed700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3980f0b10 con 0x7fd3980d1870 2021-02-10T11:31:43.619 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:43.599 7fd3a01ed700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3980e5ed0 con 0x7fd3980d1c70 2021-02-10T11:31:43.619 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:43.599 7fd3a01ed700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3981b0be0 con 0x7fd3980d0340 2021-02-10T11:31:43.619 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:43.600 7fd39e009700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fd3980d1870 0x7fd3981b03f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:54294/0 (socket says 172.21.2.102:54294) 2021-02-10T11:31:43.619 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:43.600 7fd39e009700 1 -- 172.21.2.102:0/1299566823 learned_addr learned my addr 172.21.2.102:0/1299566823 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:31:43.619 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:43.600 7fd39d007700 1 -- 172.21.2.102:0/1299566823 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2186223830 0 0) 0x7fd38c0010b0 con 0x7fd3980d1870 2021-02-10T11:31:43.619 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:43.600 7fd39d007700 1 -- 172.21.2.102:0/1299566823 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd384002fe0 con 0x7fd3980d1870 2021-02-10T11:31:43.620 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:43.600 7fd39e80a700 1 --2- 172.21.2.102:0/1299566823 >> v2:172.21.2.113:3300/0 conn(0x7fd3980d1c70 0x7fd3980efc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:43.620 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:43.600 7fd39d808700 1 --2- 172.21.2.102:0/1299566823 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd3980d0340 0x7fd3980ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:43.620 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:43.600 7fd39d007700 1 -- 172.21.2.102:0/1299566823 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2225841226 0 0) 0x7fd38c0010b0 con 0x7fd3980d1870 2021-02-10T11:31:43.620 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:43.600 7fd39d007700 1 -- 172.21.2.102:0/1299566823 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd3980d0340 msgr2=0x7fd3980ef640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:43.620 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:43.600 7fd39d007700 1 --2- 172.21.2.102:0/1299566823 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd3980d0340 0x7fd3980ef640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:43.620 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:43.600 7fd39d007700 1 -- 172.21.2.102:0/1299566823 >> v2:172.21.2.113:3300/0 conn(0x7fd3980d1c70 msgr2=0x7fd3980efc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:43.620 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:43.600 7fd39d007700 1 --2- 172.21.2.102:0/1299566823 >> v2:172.21.2.113:3300/0 conn(0x7fd3980d1c70 0x7fd3980efc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:43.620 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:43.600 7fd39d007700 1 -- 172.21.2.102:0/1299566823 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3980f20a0 con 0x7fd3980d1870 2021-02-10T11:31:43.620 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:43.600 7fd39d007700 1 -- 172.21.2.102:0/1299566823 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7fd38c0015b0 con 0x7fd3980d1870 2021-02-10T11:31:43.620 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:43.600 7fd39d007700 1 -- 172.21.2.102:0/1299566823 >> v1:172.21.2.102:6789/0 conn(0x7fd3980d1870 legacy=0x7fd3981b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:43.620 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:43.600 7fd39d007700 1 --2- 172.21.2.102:0/1299566823 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd384003b10 0x7fd384006010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:43.620 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:43.600 7fd39d007700 1 --2- 172.21.2.102:0/1299566823 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd384006820 0x7fd384008c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:43.620 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:43.601 7fd39d007700 1 --2- 172.21.2.102:0/1299566823 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd38404d540 0x7fd38404fa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:43.621 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:43.601 7fd39d007700 1 -- 172.21.2.102:0/1299566823 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3840502d0 con 0x7fd384003b10 2021-02-10T11:31:43.621 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:43.601 7fd39d007700 1 -- 172.21.2.102:0/1299566823 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3840504f0 con 0x7fd38404d540 2021-02-10T11:31:43.621 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:43.601 7fd39d007700 1 -- 172.21.2.102:0/1299566823 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd384050710 con 0x7fd384006820 2021-02-10T11:31:43.621 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:43.601 7fd39e009700 1 --2- 172.21.2.102:0/1299566823 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd384003b10 0x7fd384006010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:43.621 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:43.601 7fd39d808700 1 --2- 172.21.2.102:0/1299566823 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd384006820 0x7fd384008c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:43.621 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:43.601 7fd39e80a700 1 --2- 172.21.2.102:0/1299566823 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd38404d540 0x7fd38404fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:43.621 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:43.601 7fd39e009700 1 -- 172.21.2.102:0/1299566823 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd384006820 msgr2=0x7fd384008c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:43.621 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:43.601 7fd39e009700 1 --2- 172.21.2.102:0/1299566823 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd384006820 0x7fd384008c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:43.621 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:43.601 7fd39e009700 1 -- 172.21.2.102:0/1299566823 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd38404d540 msgr2=0x7fd38404fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:43.621 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:43.601 7fd39e009700 1 --2- 172.21.2.102:0/1299566823 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd38404d540 0x7fd38404fa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:43.621 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:43.601 7fd39e009700 1 -- 172.21.2.102:0/1299566823 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3840509b0 con 0x7fd384003b10 2021-02-10T11:31:43.621 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:43.601 7fd39e009700 1 --2- 172.21.2.102:0/1299566823 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd384003b10 0x7fd384006010 secure :-1 s=READY pgs=2168 cs=0 l=1 rev1=1 rx=0x7fd38c000b50 tx=0x7fd38c007cf0).ready entity=mon.0 client_cookie=919653d5025c9b73 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:43.621 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:43.601 7fd39d007700 1 -- 172.21.2.102:0/1299566823 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd38c00a310 con 0x7fd384003b10 2021-02-10T11:31:43.622 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:43.601 7fd39d007700 1 -- 172.21.2.102:0/1299566823 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd38c00a590 con 0x7fd384003b10 2021-02-10T11:31:43.622 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:43.601 7fd39d007700 1 -- 172.21.2.102:0/1299566823 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd38c00aab0 con 0x7fd384003b10 2021-02-10T11:31:43.622 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:43.601 7fd3a01ed700 1 -- 172.21.2.102:0/1299566823 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd384003b10 msgr2=0x7fd384006010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:43.622 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:43.601 7fd3a01ed700 1 --2- 172.21.2.102:0/1299566823 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd384003b10 0x7fd384006010 secure :-1 s=READY pgs=2168 cs=0 l=1 rev1=1 rx=0x7fd38c000b50 tx=0x7fd38c007cf0).stop 2021-02-10T11:31:43.622 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:43.601 7fd39e80a700 1 -- 172.21.2.102:0/1299566823 reap_dead start 2021-02-10T11:31:43.622 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:43.602 7fd3a01ed700 1 -- 172.21.2.102:0/1299566823 shutdown_connections 2021-02-10T11:31:43.622 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:43.602 7fd3a01ed700 1 -- 172.21.2.102:0/1299566823 >> 172.21.2.102:0/1299566823 conn(0x7fd3980ce450 msgr2=0x7fd3980d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:43.622 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:43.602 7fd3a01ed700 1 -- 172.21.2.102:0/1299566823 shutdown_connections 2021-02-10T11:31:43.622 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:43.602 7fd3a01ed700 1 -- 172.21.2.102:0/1299566823 wait complete. 2021-02-10T11:31:43.622 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:43.617 7fd3a01ed700 1 Processor -- start 2021-02-10T11:31:43.622 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:43.617 7fd3a01ed700 1 -- start start 2021-02-10T11:31:43.622 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:43.617 7fd3a01ed700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd398195650 0x7fd3980b5560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:43.623 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:43.617 7fd3a01ed700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd3980b5b80 0x7fd3981a2e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:43.623 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:43.617 7fd3a01ed700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd3980acfa0 0x7fd3981a3440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:43.623 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:43.617 7fd3a01ed700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3980b6010 con 0x7fd3980b5b80 2021-02-10T11:31:43.623 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:43.617 7fd3a01ed700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3981a3a60 con 0x7fd3980acfa0 2021-02-10T11:31:43.623 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:43.617 7fd3a01ed700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3981a3c40 con 0x7fd398195650 2021-02-10T11:31:43.623 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:43.617 7fd39d808700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd3980b5b80 0x7fd3981a2e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:43.623 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:43.617 7fd39d808700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd3980b5b80 0x7fd3981a2e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:33850/0 (socket says 172.21.2.102:33850) 2021-02-10T11:31:43.623 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:43.617 7fd39d808700 1 -- 172.21.2.102:0/3211948513 learned_addr learned my addr 172.21.2.102:0/3211948513 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:31:43.624 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:43.618 7fd39d808700 1 -- 172.21.2.102:0/3211948513 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd398195650 msgr2=0x7fd3980b5560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:43.624 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:43.618 7fd39d808700 1 --2- 172.21.2.102:0/3211948513 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd398195650 0x7fd3980b5560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:43.624 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:43.618 7fd39d808700 1 -- 172.21.2.102:0/3211948513 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd3980acfa0 msgr2=0x7fd3981a3440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:43.624 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:43.618 7fd39d808700 1 --2- 172.21.2.102:0/3211948513 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd3980acfa0 0x7fd3981a3440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:43.625 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:43.618 7fd39d808700 1 -- 172.21.2.102:0/3211948513 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3980bcae0 con 0x7fd3980b5b80 2021-02-10T11:31:43.625 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:43.618 7fd39d808700 1 --2- 172.21.2.102:0/3211948513 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd3980b5b80 0x7fd3981a2e20 secure :-1 s=READY pgs=2169 cs=0 l=1 rev1=1 rx=0x7fd394006f10 tx=0x7fd39400b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:43.625 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:43.618 7fd38affd700 1 -- 172.21.2.102:0/3211948513 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd39400bd80 con 0x7fd3980b5b80 2021-02-10T11:31:43.625 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:43.618 7fd38affd700 1 -- 172.21.2.102:0/3211948513 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd394000c90 con 0x7fd3980b5b80 2021-02-10T11:31:43.625 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:43.618 7fd3a01ed700 1 -- 172.21.2.102:0/3211948513 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3980bcae0 con 0x7fd3980b5b80 2021-02-10T11:31:43.625 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:43.618 7fd3a01ed700 1 -- 172.21.2.102:0/3211948513 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3980bd070 con 0x7fd3980b5b80 2021-02-10T11:31:43.625 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:43.618 7fd38affd700 1 -- 172.21.2.102:0/3211948513 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd3940022b0 con 0x7fd3980b5b80 2021-02-10T11:31:43.625 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:43.619 7fd38affd700 1 -- 172.21.2.102:0/3211948513 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd394009fe0 con 0x7fd3980b5b80 2021-02-10T11:31:43.625 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:43.619 7fd38affd700 1 --2- 172.21.2.102:0/3211948513 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd380018730 0x7fd38001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:43.625 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:43.619 7fd38affd700 1 -- 172.21.2.102:0/3211948513 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 4753+0+0 (secure 0 0 0) 0x7fd394032490 con 0x7fd3980b5b80 2021-02-10T11:31:43.625 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:43.619 7fd39e009700 1 --2- 172.21.2.102:0/3211948513 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd380018730 0x7fd38001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:43.625 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:43.619 7fd39e009700 1 --2- 172.21.2.102:0/3211948513 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd380018730 0x7fd38001ac10 crc :-1 s=READY pgs=957 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:43.650 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:43.649 7fd3a01ed700 1 -- 172.21.2.102:0/3211948513 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3980f0ab0 con 0x7fd3980b5b80 2021-02-10T11:31:43.652 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:43.652 7fd38affd700 1 -- 172.21.2.102:0/3211948513 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd394000960 con 0x7fd3980b5b80 2021-02-10T11:31:43.785 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:43.784 7fd3a01ed700 1 -- 172.21.2.102:0/3211948513 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fd3980f0ab0 con 0x7fd3980b5b80 2021-02-10T11:31:43.785 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:43.784 7fd38affd700 1 -- 172.21.2.102:0/3211948513 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v62) v1 ==== 80+0+4513 (secure 0 0 0) 0x7fd39400bd80 con 0x7fd3980b5b80 2021-02-10T11:31:43.786 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:31:43.786 INFO:teuthology.orchestra.run.gibba002.stdout:{ 2021-02-10T11:31:43.786 INFO:teuthology.orchestra.run.gibba002.stdout: "auth_dump": [ 2021-02-10T11:31:43.786 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:31:43.786 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "mds.a", 2021-02-10T11:31:43.786 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAevSNgsgyEARAAvr0cUjjCoEHLxaZSZ2HtVA==", 2021-02-10T11:31:43.786 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:31:43.786 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow", 2021-02-10T11:31:43.787 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:31:43.787 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:31:43.787 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:31:43.787 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:31:43.787 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:31:43.787 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:31:43.787 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "mds.b", 2021-02-10T11:31:43.787 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAevSNgwOFMMRAALfb1fbX56ao1TU8ClnTW1A==", 2021-02-10T11:31:43.787 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:31:43.787 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow", 2021-02-10T11:31:43.787 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:31:43.787 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:31:43.787 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:31:43.788 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:31:43.788 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:31:43.788 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:31:43.788 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.0", 2021-02-10T11:31:43.788 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAmvSNgj+bMMBAAD3HUD6fLiMAg+++iNzLJKQ==", 2021-02-10T11:31:43.788 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:31:43.788 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:31:43.788 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:31:43.788 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:31:43.788 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:31:43.788 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:31:43.788 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:31:43.788 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.1", 2021-02-10T11:31:43.788 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQApvSNgnvc+MRAAAOxqlNwiyLHqNITOrYeFKQ==", 2021-02-10T11:31:43.789 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:31:43.789 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:31:43.789 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:31:43.789 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:31:43.789 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:31:43.789 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:31:43.789 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:31:43.789 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.2", 2021-02-10T11:31:43.789 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAsvSNgZ4SbMRAA76IOYWfbiAuBoM/4k81GxA==", 2021-02-10T11:31:43.789 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:31:43.789 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:31:43.789 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:31:43.789 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:31:43.790 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:31:43.790 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:31:43.790 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:31:43.790 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.3", 2021-02-10T11:31:43.790 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAvvSNgFk3rMRAAWCmKKyw0LDQ+UlAHZAITOw==", 2021-02-10T11:31:43.790 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:31:43.790 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:31:43.790 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:31:43.790 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:31:43.790 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:31:43.790 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:31:43.790 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:31:43.790 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.4", 2021-02-10T11:31:43.791 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQA4vSNgpKcNJBAApJMmvj5YxVHPDOmvmQONOg==", 2021-02-10T11:31:43.791 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:31:43.791 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:31:43.791 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:31:43.791 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:31:43.791 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:31:43.791 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:31:43.791 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:31:43.791 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.5", 2021-02-10T11:31:43.791 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQA7vSNgf2ujJBAAfkuYpyb3WTzc8V5kw+Wktg==", 2021-02-10T11:31:43.791 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:31:43.791 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:31:43.791 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:31:43.791 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:31:43.792 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:31:43.792 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:31:43.792 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:31:43.792 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.6", 2021-02-10T11:31:43.792 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQA+vSNgqw9uJRAAFMU0R+UxU9C3o/9byb3e0g==", 2021-02-10T11:31:43.792 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:31:43.792 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:31:43.792 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:31:43.792 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:31:43.792 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:31:43.792 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:31:43.792 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:31:43.792 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.7", 2021-02-10T11:31:43.793 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQBBvSNgvWSrJRAA2KaCHmLpzovRgYzLvyofTQ==", 2021-02-10T11:31:43.793 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:31:43.793 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:31:43.793 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:31:43.793 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:31:43.793 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:31:43.793 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:31:43.793 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:31:43.793 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "client.0", 2021-02-10T11:31:43.793 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAfvSNgZd9XIBAAvyrvl8fdKuHnJZfczfRoFw==", 2021-02-10T11:31:43.793 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:31:43.793 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow", 2021-02-10T11:31:43.793 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow r", 2021-02-10T11:31:43.793 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:31:43.794 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:31:43.794 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:31:43.794 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:31:43.794 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "client.1", 2021-02-10T11:31:43.794 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAfvSNgSyfXNhAAIduPnF7Vd/6HSkJ0MlcMSA==", 2021-02-10T11:31:43.794 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:31:43.794 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow", 2021-02-10T11:31:43.794 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow r", 2021-02-10T11:31:43.794 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:31:43.794 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:31:43.794 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:31:43.794 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:31:43.794 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "client.2", 2021-02-10T11:31:43.795 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAgvSNgiIG/ERAA9ZTamaTI7ZsPSnxBqjplCg==", 2021-02-10T11:31:43.795 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:31:43.795 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow", 2021-02-10T11:31:43.795 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow r", 2021-02-10T11:31:43.795 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:31:43.795 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:31:43.795 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:31:43.795 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:31:43.795 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "client.3", 2021-02-10T11:31:43.795 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAgvSNgOWpOKBAAlXsEB04Q/GvtL06PqRMLeA==", 2021-02-10T11:31:43.795 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:31:43.795 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow", 2021-02-10T11:31:43.795 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow r", 2021-02-10T11:31:43.795 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:31:43.796 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:31:43.796 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:31:43.796 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:31:43.796 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "client.admin", 2021-02-10T11:31:43.796 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAavSNgO3GQBhAAL+NVnYskQRKVWLgyDPY/xw==", 2021-02-10T11:31:43.796 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:31:43.796 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow *", 2021-02-10T11:31:43.796 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:31:43.796 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:31:43.796 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:31:43.796 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:31:43.796 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:31:43.796 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:31:43.796 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "mgr.x", 2021-02-10T11:31:43.797 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAdvSNgXZZgJhAA7dtBSGzH+DsBwnddZpOBJA==", 2021-02-10T11:31:43.797 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:31:43.797 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow *", 2021-02-10T11:31:43.797 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow profile mgr", 2021-02-10T11:31:43.797 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:31:43.797 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:31:43.797 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:31:43.797 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:31:43.797 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "mgr.y", 2021-02-10T11:31:43.797 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAdvSNgVqQFDRAAwo6YEdWZCAr5GdOWVL/Qvg==", 2021-02-10T11:31:43.797 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:31:43.797 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow *", 2021-02-10T11:31:43.797 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow profile mgr", 2021-02-10T11:31:43.797 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:31:43.798 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:31:43.798 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:31:43.798 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:31:43.798 INFO:teuthology.orchestra.run.gibba002.stdout:} 2021-02-10T11:31:43.799 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:43.786 7fd3a01ed700 1 -- 172.21.2.102:0/3211948513 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd380018730 msgr2=0x7fd38001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:43.799 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:43.786 7fd3a01ed700 1 --2- 172.21.2.102:0/3211948513 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd380018730 0x7fd38001ac10 crc :-1 s=READY pgs=957 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:43.799 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:43.786 7fd3a01ed700 1 -- 172.21.2.102:0/3211948513 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd3980b5b80 msgr2=0x7fd3981a2e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:43.799 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:43.786 7fd3a01ed700 1 --2- 172.21.2.102:0/3211948513 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd3980b5b80 0x7fd3981a2e20 secure :-1 s=READY pgs=2169 cs=0 l=1 rev1=1 rx=0x7fd394006f10 tx=0x7fd39400b990).stop 2021-02-10T11:31:43.799 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:43.786 7fd3a01ed700 1 -- 172.21.2.102:0/3211948513 shutdown_connections 2021-02-10T11:31:43.799 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:43.786 7fd3a01ed700 1 --2- 172.21.2.102:0/3211948513 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd380018730 0x7fd38001ac10 unknown :-1 s=CLOSED pgs=957 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:43.799 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:43.786 7fd3a01ed700 1 --2- 172.21.2.102:0/3211948513 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd3980acfa0 0x7fd3981a3440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:43.799 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:43.786 7fd3a01ed700 1 --2- 172.21.2.102:0/3211948513 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd3980b5b80 0x7fd3981a2e20 unknown :-1 s=CLOSED pgs=2169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:43.800 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:43.786 7fd3a01ed700 1 --2- 172.21.2.102:0/3211948513 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd398195650 0x7fd3980b5560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:43.800 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:43.786 7fd3a01ed700 1 -- 172.21.2.102:0/3211948513 >> 172.21.2.102:0/3211948513 conn(0x7fd3980b3040 msgr2=0x7fd3980c3250 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:43.800 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:43.787 7fd3a01ed700 1 -- 172.21.2.102:0/3211948513 shutdown_connections 2021-02-10T11:31:43.800 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:43.787 7fd3a01ed700 1 -- 172.21.2.102:0/3211948513 wait complete. 2021-02-10T11:31:43.800 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:31:44.233 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:44.213 7fac52d02700 1 Processor -- start 2021-02-10T11:31:44.233 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:44.213 7fac52d02700 1 -- start start 2021-02-10T11:31:44.233 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:44.214 7fac52d02700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fac4c1b0470 0x7fac4c1b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:44.233 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:44.214 7fac52d02700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fac4c0f20e0 0x7fac4c0f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:44.233 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:44.214 7fac52d02700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac4c0f0b50 con 0x7fac4c0ef610 2021-02-10T11:31:44.233 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:44.214 7fac52d02700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac4c0f0820 con 0x7fac4c0f20e0 2021-02-10T11:31:44.233 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:44.214 7fac52d02700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac4c0f24e0 con 0x7fac4c1b0470 2021-02-10T11:31:44.233 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:44.214 7fac4bfff700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fac4c0ef610 0x7fac4c0efa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:54312/0 (socket says 172.21.2.102:54312) 2021-02-10T11:31:44.233 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:44.214 7fac4bfff700 1 -- 172.21.2.102:0/4236125777 learned_addr learned my addr 172.21.2.102:0/4236125777 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:31:44.234 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:44.214 7fac5131f700 1 --2- 172.21.2.102:0/4236125777 >> v2:172.21.2.102:3300/0 conn(0x7fac4c0f20e0 0x7fac4c0f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:44.234 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:44.215 7fac5131f700 1 -- 172.21.2.102:0/4236125777 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fac4c1b0470 msgr2=0x7fac4c1b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:44.234 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:44.215 7fac5131f700 1 --2- 172.21.2.102:0/4236125777 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fac4c1b0470 0x7fac4c1b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:44.234 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:44.215 7fac5131f700 1 -- 172.21.2.102:0/4236125777 >> v1:172.21.2.102:6789/0 conn(0x7fac4c0ef610 legacy=0x7fac4c0efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:44.234 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:44.215 7fac5131f700 1 -- 172.21.2.102:0/4236125777 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac4c0e52e0 con 0x7fac4c0f20e0 2021-02-10T11:31:44.234 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:44.215 7fac5131f700 1 --2- 172.21.2.102:0/4236125777 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fac4c0f20e0 0x7fac4c0f0200 secure :-1 s=READY pgs=2171 cs=0 l=1 rev1=1 rx=0x7fac44006e30 tx=0x7fac44009e60).ready entity=mon.0 client_cookie=8c50740f3aa794ce server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:44.234 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:44.215 7fac4b7fe700 1 -- 172.21.2.102:0/4236125777 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fac44006a50 con 0x7fac4c0f20e0 2021-02-10T11:31:44.234 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:44.215 7fac4b7fe700 1 -- 172.21.2.102:0/4236125777 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac4400b300 con 0x7fac4c0f20e0 2021-02-10T11:31:44.234 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:44.215 7fac4b7fe700 1 -- 172.21.2.102:0/4236125777 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fac4400b7e0 con 0x7fac4c0f20e0 2021-02-10T11:31:44.234 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:44.215 7fac52d02700 1 -- 172.21.2.102:0/4236125777 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fac4c0f20e0 msgr2=0x7fac4c0f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:44.234 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:44.215 7fac52d02700 1 --2- 172.21.2.102:0/4236125777 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fac4c0f20e0 0x7fac4c0f0200 secure :-1 s=READY pgs=2171 cs=0 l=1 rev1=1 rx=0x7fac44006e30 tx=0x7fac44009e60).stop 2021-02-10T11:31:44.234 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:44.216 7fac52d02700 1 -- 172.21.2.102:0/4236125777 shutdown_connections 2021-02-10T11:31:44.234 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:44.216 7fac52d02700 1 --2- 172.21.2.102:0/4236125777 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fac4c0f20e0 0x7fac4c0f0200 unknown :-1 s=CLOSED pgs=2171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:44.235 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:44.216 7fac52d02700 1 --2- 172.21.2.102:0/4236125777 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fac4c1b0470 0x7fac4c1b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:44.235 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:44.216 7fac52d02700 1 -- 172.21.2.102:0/4236125777 >> 172.21.2.102:0/4236125777 conn(0x7fac4c0ce3e0 msgr2=0x7fac4c0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:44.235 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:44.216 7fac52d02700 1 -- 172.21.2.102:0/4236125777 shutdown_connections 2021-02-10T11:31:44.235 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:44.216 7fac52d02700 1 -- 172.21.2.102:0/4236125777 wait complete. 2021-02-10T11:31:44.236 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:44.232 7fac52d02700 1 Processor -- start 2021-02-10T11:31:44.236 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:44.232 7fac52d02700 1 -- start start 2021-02-10T11:31:44.236 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:44.233 7fac52d02700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fac4c1acc20 0x7fac4c1ad020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:44.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:44.233 7fac52d02700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fac4c1ad640 0x7fac4c1ada40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:44.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:44.233 7fac52d02700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fac4c188110 0x7fac4c188510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:44.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:44.233 7fac52d02700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac4c1ac720 con 0x7fac4c1ad640 2021-02-10T11:31:44.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:44.233 7fac52d02700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac4c0b1b10 con 0x7fac4c1acc20 2021-02-10T11:31:44.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:44.233 7fac52d02700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac4c188b30 con 0x7fac4c188110 2021-02-10T11:31:44.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:44.233 7fac4bfff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fac4c1ad640 0x7fac4c1ada40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:44.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:44.233 7fac4bfff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fac4c1ad640 0x7fac4c1ada40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:33862/0 (socket says 172.21.2.102:33862) 2021-02-10T11:31:44.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:44.233 7fac4bfff700 1 -- 172.21.2.102:0/2790726340 learned_addr learned my addr 172.21.2.102:0/2790726340 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:31:44.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:44.233 7fac5131f700 1 --2- 172.21.2.102:0/2790726340 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fac4c188110 0x7fac4c188510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:44.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:44.233 7fac50b1e700 1 --2- 172.21.2.102:0/2790726340 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fac4c1acc20 0x7fac4c1ad020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:44.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:44.233 7fac5131f700 1 -- 172.21.2.102:0/2790726340 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fac4c1acc20 msgr2=0x7fac4c1ad020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:44.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:44.233 7fac5131f700 1 --2- 172.21.2.102:0/2790726340 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fac4c1acc20 0x7fac4c1ad020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:44.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:44.233 7fac5131f700 1 -- 172.21.2.102:0/2790726340 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fac4c1ad640 msgr2=0x7fac4c1ada40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:44.238 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:44.233 7fac5131f700 1 --2- 172.21.2.102:0/2790726340 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fac4c1ad640 0x7fac4c1ada40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:44.238 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:44.233 7fac5131f700 1 -- 172.21.2.102:0/2790726340 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac4c188d50 con 0x7fac4c188110 2021-02-10T11:31:44.238 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:44.234 7fac5131f700 1 --2- 172.21.2.102:0/2790726340 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fac4c188110 0x7fac4c188510 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7fac4400b8f0 tx=0x7fac4400bc90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:44.238 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:44.234 7fac497fa700 1 -- 172.21.2.102:0/2790726340 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fac4400bdd0 con 0x7fac4c188110 2021-02-10T11:31:44.238 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:44.234 7fac497fa700 1 -- 172.21.2.102:0/2790726340 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac44005820 con 0x7fac4c188110 2021-02-10T11:31:44.238 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:44.234 7fac497fa700 1 -- 172.21.2.102:0/2790726340 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fac44005d40 con 0x7fac4c188110 2021-02-10T11:31:44.238 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:44.234 7fac52d02700 1 -- 172.21.2.102:0/2790726340 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac4c188d50 con 0x7fac4c188110 2021-02-10T11:31:44.238 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:44.234 7fac52d02700 1 -- 172.21.2.102:0/2790726340 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac4c1892e0 con 0x7fac4c188110 2021-02-10T11:31:44.238 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:44.234 7fac497fa700 1 -- 172.21.2.102:0/2790726340 <== mon.2 v2:172.21.2.113:3301/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fac44005a00 con 0x7fac4c188110 2021-02-10T11:31:44.238 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:44.235 7fac497fa700 1 --2- 172.21.2.102:0/2790726340 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fac300186c0 0x7fac3001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:44.238 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:44.235 7fac497fa700 1 -- 172.21.2.102:0/2790726340 <== mon.2 v2:172.21.2.113:3301/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 4753+0+0 (secure 0 0 0) 0x7fac44032540 con 0x7fac4c188110 2021-02-10T11:31:44.238 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:44.235 7fac50b1e700 1 --2- 172.21.2.102:0/2790726340 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fac300186c0 0x7fac3001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:44.238 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:44.235 7fac50b1e700 1 --2- 172.21.2.102:0/2790726340 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fac300186c0 0x7fac3001abc0 crc :-1 s=READY pgs=958 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:44.265 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:44.264 7fac52d02700 1 -- 172.21.2.102:0/2790726340 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac4c0eddf0 con 0x7fac4c188110 2021-02-10T11:31:44.269 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:44.268 7fac497fa700 1 -- 172.21.2.102:0/2790726340 <== mon.2 v2:172.21.2.113:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fac440093d0 con 0x7fac4c188110 2021-02-10T11:31:44.398 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:44.397 7fac52d02700 1 -- 172.21.2.102:0/2790726340 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fac4c0e52e0 con 0x7fac4c188110 2021-02-10T11:31:44.399 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:44.399 7fac497fa700 1 -- 172.21.2.102:0/2790726340 <== mon.2 v2:172.21.2.113:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v107) v1 ==== 74+0+11152 (secure 0 0 0) 0x7fac44005a00 con 0x7fac4c188110 2021-02-10T11:31:44.401 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:31:44.402 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":107,"fsid":"68db30ea-dbce-4aef-a6df-f01c6fea0513","created":"2021-02-10 11:03:06.331557","modified":"2021-02-10 11:31:43.161063","last_up_change":"2021-02-10 11:03:32.690562","last_in_change":"2021-02-10 11:03:32.690562","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:03:37.322519","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"be691b4d-a039-408a-af80-9d53f02b6f1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6818","nonce":36088},{"type":"v1","addr":"172.21.2.102:6820","nonce":36088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6822","nonce":36088},{"type":"v1","addr":"172.21.2.102:6824","nonce":36088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6830","nonce":36088},{"type":"v1","addr":"172.21.2.102:6832","nonce":36088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6826","nonce":36088},{"type":"v1","addr":"172.21.2.102:6828","nonce":36088}]},"public_addr":"172.21.2.102:6820/36088","cluster_addr":"172.21.2.102:6824/36088","heartbeat_back_addr":"172.21.2.102:6832/36088","heartbeat_front_addr":"172.21.2.102:6828/36088","state":["exists","up"]},{"osd":1,"uuid":"64e3ccc8-9c69-4d7c-865e-af8e83822635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6819","nonce":36087},{"type":"v1","addr":"172.21.2.102:6821","nonce":36087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6823","nonce":36087},{"type":"v1","addr":"172.21.2.102:6825","nonce":36087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6831","nonce":36087},{"type":"v1","addr":"172.21.2.102:6833","nonce":36087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6827","nonce":36087},{"type":"v1","addr":"172.21.2.102:6829","nonce":36087}]},"public_addr":"172.21.2.102:6821/36087","cluster_addr":"172.21.2.102:6825/36087","heartbeat_back_addr":"172.21.2.102:6833/36087","heartbeat_front_addr":"172.21.2.102:6829/36087","state":["exists","up"]},{"osd":2,"uuid":"153700af-747f-481f-bdf4-d6072d3a842f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6810","nonce":36089},{"type":"v1","addr":"172.21.2.102:6811","nonce":36089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6812","nonce":36089},{"type":"v1","addr":"172.21.2.102:6813","nonce":36089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6816","nonce":36089},{"type":"v1","addr":"172.21.2.102:6817","nonce":36089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6814","nonce":36089},{"type":"v1","addr":"172.21.2.102:6815","nonce":36089}]},"public_addr":"172.21.2.102:6811/36089","cluster_addr":"172.21.2.102:6813/36089","heartbeat_back_addr":"172.21.2.102:6817/36089","heartbeat_front_addr":"172.21.2.102:6815/36089","state":["exists","up"]},{"osd":3,"uuid":"cbc678f4-83a9-45e6-8830-2f7c232b1ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6802","nonce":36090},{"type":"v1","addr":"172.21.2.102:6803","nonce":36090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6804","nonce":36090},{"type":"v1","addr":"172.21.2.102:6805","nonce":36090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6808","nonce":36090},{"type":"v1","addr":"172.21.2.102:6809","nonce":36090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6806","nonce":36090},{"type":"v1","addr":"172.21.2.102:6807","nonce":36090}]},"public_addr":"172.21.2.102:6803/36090","cluster_addr":"172.21.2.102:6805/36090","heartbeat_back_addr":"172.21.2.102:6809/36090","heartbeat_front_addr":"172.21.2.102:6807/36090","state":["exists","up"]},{"osd":4,"uuid":"c01e4008-b258-4f10-afd2-9e7f37214351","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6808","nonce":36368},{"type":"v1","addr":"172.21.2.113:6811","nonce":36368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6814","nonce":36368},{"type":"v1","addr":"172.21.2.113:6818","nonce":36368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6826","nonce":36368},{"type":"v1","addr":"172.21.2.113:6828","nonce":36368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6820","nonce":36368},{"type":"v1","addr":"172.21.2.113:6824","nonce":36368}]},"public_addr":"172.21.2.113:6811/36368","cluster_addr":"172.21.2.113:6818/36368","heartbeat_back_addr":"172.21.2.113:6828/36368","heartbeat_front_addr":"172.21.2.113:6824/36368","state":["exists","up"]},{"osd":5,"uuid":"551f6007-daf8-4a98-8ee0-c2b16bcccb5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6802","nonce":36366},{"type":"v1","addr":"172.21.2.113:6805","nonce":36366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6807","nonce":36366},{"type":"v1","addr":"172.21.2.113:6810","nonce":36366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6819","nonce":36366},{"type":"v1","addr":"172.21.2.113:6822","nonce":36366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6812","nonce":36366},{"type":"v1","addr":"172.21.2.113:6815","nonce":36366}]},"public_addr":"172.21.2.113:6805/36366","cluster_addr":"172.21.2.113:6810/36366","heartbeat_back_addr":"172.21.2.113:6822/36366","heartbeat_front_addr":"172.21.2.113:6815/36366","state":["exists","up"]},{"osd":6,"uuid":"bf67c040-0e50-48be-b102-c3943bc9ebef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6800","nonce":36367},{"type":"v1","addr":"172.21.2.113:6801","nonce":36367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6803","nonce":36367},{"type":"v1","addr":"172.21.2.113:6804","nonce":36367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6813","nonce":36367},{"type":"v1","addr":"172.21.2.113:6816","nonce":36367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6806","nonce":36367},{"type":"v1","addr":"172.21.2.113:6809","nonce":36367}]},"public_addr":"172.21.2.113:6801/36367","cluster_addr":"172.21.2.113:6804/36367","heartbeat_back_addr":"172.21.2.113:6816/36367","heartbeat_front_addr":"172.21.2.113:6809/36367","state":["exists","up"]},{"osd":7,"uuid":"f5461f80-018d-428f-be89-2eb2bcbe7bb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6817","nonce":36365},{"type":"v1","addr":"172.21.2.113:6821","nonce":36365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6823","nonce":36365},{"type":"v1","addr":"172.21.2.113:6825","nonce":36365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6830","nonce":36365},{"type":"v1","addr":"172.21.2.113:6831","nonce":36365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6827","nonce":36365},{"type":"v1","addr":"172.21.2.113:6829","nonce":36365}]},"public_addr":"172.21.2.113:6821/36365","cluster_addr":"172.21.2.113:6825/36365","heartbeat_back_addr":"172.21.2.113:6831/36365","heartbeat_front_addr":"172.21.2.113:6829/36365","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:31:44.403 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:44.402 7fac52d02700 1 -- 172.21.2.102:0/2790726340 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fac300186c0 msgr2=0x7fac3001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:44.403 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:44.402 7fac52d02700 1 --2- 172.21.2.102:0/2790726340 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fac300186c0 0x7fac3001abc0 crc :-1 s=READY pgs=958 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:44.403 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:44.402 7fac52d02700 1 -- 172.21.2.102:0/2790726340 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fac4c188110 msgr2=0x7fac4c188510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:44.403 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:44.402 7fac52d02700 1 --2- 172.21.2.102:0/2790726340 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fac4c188110 0x7fac4c188510 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7fac4400b8f0 tx=0x7fac4400bc90).stop 2021-02-10T11:31:44.403 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:44.402 7fac52d02700 1 -- 172.21.2.102:0/2790726340 shutdown_connections 2021-02-10T11:31:44.404 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:44.402 7fac52d02700 1 --2- 172.21.2.102:0/2790726340 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fac4c188110 0x7fac4c188510 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:44.404 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:44.402 7fac52d02700 1 --2- 172.21.2.102:0/2790726340 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fac4c1ad640 0x7fac4c1ada40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:44.404 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:44.402 7fac52d02700 1 --2- 172.21.2.102:0/2790726340 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fac300186c0 0x7fac3001abc0 unknown :-1 s=CLOSED pgs=958 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:44.404 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:44.402 7fac52d02700 1 --2- 172.21.2.102:0/2790726340 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fac4c1acc20 0x7fac4c1ad020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:44.404 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:44.402 7fac52d02700 1 -- 172.21.2.102:0/2790726340 >> 172.21.2.102:0/2790726340 conn(0x7fac4c19f7a0 msgr2=0x7fac4c1922a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:44.404 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:44.402 7fac52d02700 1 -- 172.21.2.102:0/2790726340 shutdown_connections 2021-02-10T11:31:44.404 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:44.402 7fac52d02700 1 -- 172.21.2.102:0/2790726340 wait complete. 2021-02-10T11:31:44.412 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:03:37.322519', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2021-02-10T11:31:44.412 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:31:44.842 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:44.824 7fb1fe8ff700 1 Processor -- start 2021-02-10T11:31:44.842 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:44.824 7fb1fe8ff700 1 -- start start 2021-02-10T11:31:44.842 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:44.825 7fb1fe8ff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fb1f80d1c70 0x7fb1f80cef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:44.842 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:44.825 7fb1fe8ff700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fb1f80cf580 0x7fb1f80cf980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:44.842 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:44.825 7fb1fe8ff700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1f80fe1c0 con 0x7fb1f80d1870 2021-02-10T11:31:44.842 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:44.825 7fb1fe8ff700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1f81b0cb0 con 0x7fb1f80d1c70 2021-02-10T11:31:44.842 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:44.825 7fb1fe8ff700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1f8109a60 con 0x7fb1f80cf580 2021-02-10T11:31:44.843 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:44.825 7fb1f77fe700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fb1f80d1c70 0x7fb1f80cef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:44.843 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:44.825 7fb1f77fe700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fb1f80d1c70 0x7fb1f80cef60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:33868/0 (socket says 172.21.2.102:33868) 2021-02-10T11:31:44.843 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:44.825 7fb1f77fe700 1 -- 172.21.2.102:0/2022561900 learned_addr learned my addr 172.21.2.102:0/2022561900 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:31:44.843 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:44.825 7fb1f77fe700 1 -- 172.21.2.102:0/2022561900 >> v2:172.21.2.113:3300/0 conn(0x7fb1f80cf580 msgr2=0x7fb1f80cf980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:44.843 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:44.825 7fb1f77fe700 1 --2- 172.21.2.102:0/2022561900 >> v2:172.21.2.113:3300/0 conn(0x7fb1f80cf580 0x7fb1f80cf980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:44.843 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:44.825 7fb1f77fe700 1 -- 172.21.2.102:0/2022561900 >> v1:172.21.2.113:6789/0 conn(0x7fb1f80d1870 legacy=0x7fb1f81b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:44.843 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:44.825 7fb1f77fe700 1 -- 172.21.2.102:0/2022561900 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1f8088070 con 0x7fb1f80d1c70 2021-02-10T11:31:44.843 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:44.826 7fb1f77fe700 1 --2- 172.21.2.102:0/2022561900 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb1f80d1c70 0x7fb1f80cef60 secure :-1 s=READY pgs=2172 cs=0 l=1 rev1=1 rx=0x7fb1ec006700 tx=0x7fb1ec009b80).ready entity=mon.0 client_cookie=fae44664d605dbaf server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:44.843 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:44.826 7fb1f6ffd700 1 -- 172.21.2.102:0/2022561900 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb1ec00c0c0 con 0x7fb1f80d1c70 2021-02-10T11:31:44.843 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:44.826 7fb1f6ffd700 1 -- 172.21.2.102:0/2022561900 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1ec00c400 con 0x7fb1f80d1c70 2021-02-10T11:31:44.843 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:44.826 7fb1f6ffd700 1 -- 172.21.2.102:0/2022561900 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb1ec00c920 con 0x7fb1f80d1c70 2021-02-10T11:31:44.843 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:44.826 7fb1fe8ff700 1 -- 172.21.2.102:0/2022561900 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb1f80d1c70 msgr2=0x7fb1f80cef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:44.843 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:44.826 7fb1fe8ff700 1 --2- 172.21.2.102:0/2022561900 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb1f80d1c70 0x7fb1f80cef60 secure :-1 s=READY pgs=2172 cs=0 l=1 rev1=1 rx=0x7fb1ec006700 tx=0x7fb1ec009b80).stop 2021-02-10T11:31:44.844 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:44.826 7fb1fe8ff700 1 -- 172.21.2.102:0/2022561900 shutdown_connections 2021-02-10T11:31:44.844 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:44.826 7fb1fe8ff700 1 --2- 172.21.2.102:0/2022561900 >> v2:172.21.2.113:3300/0 conn(0x7fb1f80cf580 0x7fb1f80cf980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:44.844 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:44.826 7fb1fe8ff700 1 --2- 172.21.2.102:0/2022561900 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb1f80d1c70 0x7fb1f80cef60 unknown :-1 s=CLOSED pgs=2172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:44.844 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:44.826 7fb1fe8ff700 1 -- 172.21.2.102:0/2022561900 >> 172.21.2.102:0/2022561900 conn(0x7fb1f80ce420 msgr2=0x7fb1f80d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:44.844 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:44.827 7fb1fe8ff700 1 -- 172.21.2.102:0/2022561900 shutdown_connections 2021-02-10T11:31:44.844 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:44.827 7fb1fe8ff700 1 -- 172.21.2.102:0/2022561900 wait complete. 2021-02-10T11:31:44.845 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:44.841 7fb1fe8ff700 1 Processor -- start 2021-02-10T11:31:44.845 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:44.841 7fb1fe8ff700 1 -- start start 2021-02-10T11:31:44.845 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:44.842 7fb1fe8ff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb1f80c3820 0x7fb1f80a7950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:44.846 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:44.842 7fb1fe8ff700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb1f81a44c0 0x7fb1f81a48c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:44.846 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:44.842 7fb1fe8ff700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb1f80863d0 0x7fb1f80867d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:44.846 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:44.842 7fb1fe8ff700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1f8085ed0 con 0x7fb1f80c3820 2021-02-10T11:31:44.846 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:44.842 7fb1fe8ff700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1f80c0740 con 0x7fb1f80863d0 2021-02-10T11:31:44.846 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:44.842 7fb1fe8ff700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1f8086df0 con 0x7fb1f81a44c0 2021-02-10T11:31:44.846 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:44.842 7fb1f7fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb1f80c3820 0x7fb1f80a7950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:44.846 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:44.842 7fb1f7fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb1f80c3820 0x7fb1f80a7950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:33874/0 (socket says 172.21.2.102:33874) 2021-02-10T11:31:44.846 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:44.842 7fb1f7fff700 1 -- 172.21.2.102:0/1808358640 learned_addr learned my addr 172.21.2.102:0/1808358640 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:31:44.846 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:44.842 7fb1f7fff700 1 -- 172.21.2.102:0/1808358640 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb1f81a44c0 msgr2=0x7fb1f81a48c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:44.846 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:44.842 7fb1f7fff700 1 --2- 172.21.2.102:0/1808358640 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb1f81a44c0 0x7fb1f81a48c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:44.846 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:44.842 7fb1f7fff700 1 -- 172.21.2.102:0/1808358640 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb1f80863d0 msgr2=0x7fb1f80867d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:44.846 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:44.842 7fb1f7fff700 1 --2- 172.21.2.102:0/1808358640 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb1f80863d0 0x7fb1f80867d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:44.847 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:44.842 7fb1f7fff700 1 -- 172.21.2.102:0/1808358640 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1f8087010 con 0x7fb1f80c3820 2021-02-10T11:31:44.847 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:44.842 7fb1f7fff700 1 --2- 172.21.2.102:0/1808358640 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb1f80c3820 0x7fb1f80a7950 secure :-1 s=READY pgs=2173 cs=0 l=1 rev1=1 rx=0x7fb1e800b530 tx=0x7fb1e800b8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:44.847 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:44.843 7fb1f4ff9700 1 -- 172.21.2.102:0/1808358640 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb1e80009b0 con 0x7fb1f80c3820 2021-02-10T11:31:44.847 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:44.843 7fb1f4ff9700 1 -- 172.21.2.102:0/1808358640 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1e800bcc0 con 0x7fb1f80c3820 2021-02-10T11:31:44.847 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:44.843 7fb1f4ff9700 1 -- 172.21.2.102:0/1808358640 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb1e80062a0 con 0x7fb1f80c3820 2021-02-10T11:31:44.847 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:44.843 7fb1fe8ff700 1 -- 172.21.2.102:0/1808358640 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1f8087010 con 0x7fb1f80c3820 2021-02-10T11:31:44.847 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:44.843 7fb1fe8ff700 1 -- 172.21.2.102:0/1808358640 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1f8083000 con 0x7fb1f80c3820 2021-02-10T11:31:44.847 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:44.843 7fb1f4ff9700 1 -- 172.21.2.102:0/1808358640 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fb1e8006020 con 0x7fb1f80c3820 2021-02-10T11:31:44.847 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:44.844 7fb1f4ff9700 1 --2- 172.21.2.102:0/1808358640 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb1d80186c0 0x7fb1d801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:44.847 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:44.844 7fb1f4ff9700 1 -- 172.21.2.102:0/1808358640 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 4753+0+0 (secure 0 0 0) 0x7fb1e8032270 con 0x7fb1f80c3820 2021-02-10T11:31:44.847 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:44.844 7fb1f77fe700 1 --2- 172.21.2.102:0/1808358640 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb1d80186c0 0x7fb1d801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:44.847 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:44.844 7fb1f77fe700 1 --2- 172.21.2.102:0/1808358640 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb1d80186c0 0x7fb1d801abc0 crc :-1 s=READY pgs=959 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:44.876 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:44.875 7fb1fe8ff700 1 -- 172.21.2.102:0/1808358640 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1f8085e50 con 0x7fb1f80c3820 2021-02-10T11:31:44.878 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:44.877 7fb1f4ff9700 1 -- 172.21.2.102:0/1808358640 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fb1e8020e90 con 0x7fb1f80c3820 2021-02-10T11:31:45.005 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:45.004 7fb1fe8ff700 1 -- 172.21.2.102:0/1808358640 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7fb1f8085e50 con 0x7fb1f80c3820 2021-02-10T11:31:45.006 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:45.005 7fb1f4ff9700 1 -- 172.21.2.102:0/1808358640 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v107) v1 ==== 92+0+10 (secure 0 0 0) 0x7fb1e8020e90 con 0x7fb1f80c3820 2021-02-10T11:31:45.006 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:31:45.008 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:45.006 7fb1fe8ff700 1 -- 172.21.2.102:0/1808358640 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb1d80186c0 msgr2=0x7fb1d801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:45.008 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:45.006 7fb1fe8ff700 1 --2- 172.21.2.102:0/1808358640 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb1d80186c0 0x7fb1d801abc0 crc :-1 s=READY pgs=959 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:45.008 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:45.006 7fb1fe8ff700 1 -- 172.21.2.102:0/1808358640 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb1f80c3820 msgr2=0x7fb1f80a7950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:45.008 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:45.006 7fb1fe8ff700 1 --2- 172.21.2.102:0/1808358640 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb1f80c3820 0x7fb1f80a7950 secure :-1 s=READY pgs=2173 cs=0 l=1 rev1=1 rx=0x7fb1e800b530 tx=0x7fb1e800b8d0).stop 2021-02-10T11:31:45.008 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:45.007 7fb1fe8ff700 1 -- 172.21.2.102:0/1808358640 shutdown_connections 2021-02-10T11:31:45.008 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:45.007 7fb1fe8ff700 1 --2- 172.21.2.102:0/1808358640 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb1d80186c0 0x7fb1d801abc0 unknown :-1 s=CLOSED pgs=959 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:45.008 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:45.007 7fb1fe8ff700 1 --2- 172.21.2.102:0/1808358640 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb1f80863d0 0x7fb1f80867d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:45.008 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:45.007 7fb1fe8ff700 1 --2- 172.21.2.102:0/1808358640 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb1f81a44c0 0x7fb1f81a48c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:45.008 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:45.007 7fb1fe8ff700 1 --2- 172.21.2.102:0/1808358640 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb1f80c3820 0x7fb1f80a7950 unknown :-1 s=CLOSED pgs=2173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:45.008 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:45.007 7fb1fe8ff700 1 -- 172.21.2.102:0/1808358640 >> 172.21.2.102:0/1808358640 conn(0x7fb1f80872c0 msgr2=0x7fb1f8095700 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:45.009 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:45.007 7fb1fe8ff700 1 -- 172.21.2.102:0/1808358640 shutdown_connections 2021-02-10T11:31:45.009 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:45.007 7fb1fe8ff700 1 -- 172.21.2.102:0/1808358640 wait complete. 2021-02-10T11:31:45.017 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --format=json-pretty osd lspools 2021-02-10T11:31:45.443 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:45.426 7fbca6de1700 1 Processor -- start 2021-02-10T11:31:45.444 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:45.426 7fbca6de1700 1 -- start start 2021-02-10T11:31:45.444 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:45.427 7fbca6de1700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fbca00d1870 0x7fbca01b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:45.444 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:45.427 7fbca6de1700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fbca00f0650 0x7fbca00cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:45.444 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:45.427 7fbca6de1700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbca00f0b10 con 0x7fbca01b0a10 2021-02-10T11:31:45.444 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:45.427 7fbca6de1700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbca00e5ed0 con 0x7fbca00d1870 2021-02-10T11:31:45.444 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:45.427 7fbca6de1700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbca00d1c70 con 0x7fbca00f0650 2021-02-10T11:31:45.444 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:45.427 7fbca4bfd700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fbca00d1870 0x7fbca01b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:45.444 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:45.427 7fbca4bfd700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fbca00d1870 0x7fbca01b03f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:33882/0 (socket says 172.21.2.102:33882) 2021-02-10T11:31:45.444 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:45.427 7fbca4bfd700 1 -- 172.21.2.102:0/2011989600 learned_addr learned my addr 172.21.2.102:0/2011989600 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:31:45.444 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:45.427 7fbca53fe700 1 --2- 172.21.2.102:0/2011989600 >> v2:172.21.2.113:3300/0 conn(0x7fbca00f0650 0x7fbca00cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:45.444 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:45.427 7fbca4bfd700 1 -- 172.21.2.102:0/2011989600 >> v2:172.21.2.113:3300/0 conn(0x7fbca00f0650 msgr2=0x7fbca00cfb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:45.445 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:45.427 7fbca4bfd700 1 --2- 172.21.2.102:0/2011989600 >> v2:172.21.2.113:3300/0 conn(0x7fbca00f0650 0x7fbca00cfb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:45.445 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:45.427 7fbca4bfd700 1 -- 172.21.2.102:0/2011989600 >> v1:172.21.2.113:6789/0 conn(0x7fbca01b0a10 legacy=0x7fbca00cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:45.445 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:45.427 7fbca4bfd700 1 -- 172.21.2.102:0/2011989600 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbca00d1e90 con 0x7fbca00d1870 2021-02-10T11:31:45.445 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:45.427 7fbca4bfd700 1 --2- 172.21.2.102:0/2011989600 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbca00d1870 0x7fbca01b03f0 secure :-1 s=READY pgs=2174 cs=0 l=1 rev1=1 rx=0x7fbc88006700 tx=0x7fbc88009b80).ready entity=mon.0 client_cookie=bde64ec53ee4a592 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:45.445 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:45.428 7fbc9f7fe700 1 -- 172.21.2.102:0/2011989600 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbc8800c0c0 con 0x7fbca00d1870 2021-02-10T11:31:45.445 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:45.428 7fbc9f7fe700 1 -- 172.21.2.102:0/2011989600 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc8800c340 con 0x7fbca00d1870 2021-02-10T11:31:45.445 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:45.428 7fbc9f7fe700 1 -- 172.21.2.102:0/2011989600 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbc8800c860 con 0x7fbca00d1870 2021-02-10T11:31:45.445 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:45.428 7fbca6de1700 1 -- 172.21.2.102:0/2011989600 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbca00d1870 msgr2=0x7fbca01b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:45.445 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:45.428 7fbca6de1700 1 --2- 172.21.2.102:0/2011989600 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbca00d1870 0x7fbca01b03f0 secure :-1 s=READY pgs=2174 cs=0 l=1 rev1=1 rx=0x7fbc88006700 tx=0x7fbc88009b80).stop 2021-02-10T11:31:45.445 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:45.428 7fbca6de1700 1 -- 172.21.2.102:0/2011989600 shutdown_connections 2021-02-10T11:31:45.445 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:45.428 7fbca6de1700 1 --2- 172.21.2.102:0/2011989600 >> v2:172.21.2.113:3300/0 conn(0x7fbca00f0650 0x7fbca00cfb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:45.445 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:45.428 7fbca6de1700 1 --2- 172.21.2.102:0/2011989600 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbca00d1870 0x7fbca01b03f0 unknown :-1 s=CLOSED pgs=2174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:45.445 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:45.428 7fbca6de1700 1 -- 172.21.2.102:0/2011989600 >> 172.21.2.102:0/2011989600 conn(0x7fbca00ce450 msgr2=0x7fbca00d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:45.445 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:45.428 7fbca6de1700 1 -- 172.21.2.102:0/2011989600 shutdown_connections 2021-02-10T11:31:45.446 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:45.428 7fbca6de1700 1 -- 172.21.2.102:0/2011989600 wait complete. 2021-02-10T11:31:45.447 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:45.443 7fbca6de1700 1 Processor -- start 2021-02-10T11:31:45.447 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:45.443 7fbca6de1700 1 -- start start 2021-02-10T11:31:45.447 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:45.443 7fbca6de1700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbca00ada90 0x7fbca00ade90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:45.447 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:45.443 7fbca6de1700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbca0099d20 0x7fbca009a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:45.447 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:45.443 7fbca6de1700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fbca009a740 0x7fbca0097bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:45.447 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:45.443 7fbca6de1700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbca00997a0 con 0x7fbca00ada90 2021-02-10T11:31:45.448 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:45.443 7fbca6de1700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbca018f0c0 con 0x7fbca009a740 2021-02-10T11:31:45.448 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:45.443 7fbca6de1700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbca009ab40 con 0x7fbca0099d20 2021-02-10T11:31:45.448 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:45.443 7fbca53fe700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fbca009a740 0x7fbca0097bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:45.448 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:45.444 7fbca53fe700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fbca009a740 0x7fbca0097bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.113:3300/0 says I am v2:172.21.2.102:37320/0 (socket says 172.21.2.102:37320) 2021-02-10T11:31:45.448 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:45.444 7fbca53fe700 1 -- 172.21.2.102:0/2588685682 learned_addr learned my addr 172.21.2.102:0/2588685682 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:31:45.448 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:45.444 7fbc9ffff700 1 --2- 172.21.2.102:0/2588685682 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbca0099d20 0x7fbca009a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:45.448 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:45.444 7fbca4bfd700 1 --2- 172.21.2.102:0/2588685682 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbca00ada90 0x7fbca00ade90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:45.448 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:45.444 7fbca4bfd700 1 -- 172.21.2.102:0/2588685682 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbca0099d20 msgr2=0x7fbca009a120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:45.448 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:45.444 7fbca4bfd700 1 --2- 172.21.2.102:0/2588685682 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbca0099d20 0x7fbca009a120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:45.448 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:45.444 7fbca4bfd700 1 -- 172.21.2.102:0/2588685682 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fbca009a740 msgr2=0x7fbca0097bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:45.448 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:45.444 7fbca4bfd700 1 --2- 172.21.2.102:0/2588685682 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fbca009a740 0x7fbca0097bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:45.448 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:45.444 7fbca4bfd700 1 -- 172.21.2.102:0/2588685682 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbca009ad60 con 0x7fbca00ada90 2021-02-10T11:31:45.448 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:45.444 7fbca4bfd700 1 --2- 172.21.2.102:0/2588685682 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbca00ada90 0x7fbca00ade90 secure :-1 s=READY pgs=2175 cs=0 l=1 rev1=1 rx=0x7fbc88006f20 tx=0x7fbc8800b8b0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:45.449 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:45.444 7fbc9d7fa700 1 -- 172.21.2.102:0/2588685682 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbc88000c90 con 0x7fbca00ada90 2021-02-10T11:31:45.449 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:45.444 7fbc9d7fa700 1 -- 172.21.2.102:0/2588685682 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc880090b0 con 0x7fbca00ada90 2021-02-10T11:31:45.449 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:45.444 7fbc9d7fa700 1 -- 172.21.2.102:0/2588685682 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbc880022b0 con 0x7fbca00ada90 2021-02-10T11:31:45.449 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:45.444 7fbca6de1700 1 -- 172.21.2.102:0/2588685682 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbca009ad60 con 0x7fbca00ada90 2021-02-10T11:31:45.449 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:45.444 7fbca6de1700 1 -- 172.21.2.102:0/2588685682 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbca00984d0 con 0x7fbca00ada90 2021-02-10T11:31:45.449 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:45.445 7fbc9d7fa700 1 -- 172.21.2.102:0/2588685682 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fbc880052e0 con 0x7fbca00ada90 2021-02-10T11:31:45.449 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:45.445 7fbc9d7fa700 1 --2- 172.21.2.102:0/2588685682 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fbc840186c0 0x7fbc8401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:45.449 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:45.445 7fbc9d7fa700 1 -- 172.21.2.102:0/2588685682 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 4753+0+0 (secure 0 0 0) 0x7fbc88032180 con 0x7fbca00ada90 2021-02-10T11:31:45.449 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:45.445 7fbc9ffff700 1 --2- 172.21.2.102:0/2588685682 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fbc840186c0 0x7fbc8401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:45.449 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:45.445 7fbc9ffff700 1 --2- 172.21.2.102:0/2588685682 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fbc840186c0 0x7fbc8401abc0 crc :-1 s=READY pgs=960 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:45.478 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:45.477 7fbca6de1700 1 -- 172.21.2.102:0/2588685682 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbca00ebc00 con 0x7fbca00ada90 2021-02-10T11:31:45.480 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:45.479 7fbc9d7fa700 1 -- 172.21.2.102:0/2588685682 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fbc88030c60 con 0x7fbca00ada90 2021-02-10T11:31:45.602 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:45.601 7fbca6de1700 1 -- 172.21.2.102:0/2588685682 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fbca00f0b10 con 0x7fbca00ada90 2021-02-10T11:31:45.603 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:45.602 7fbc9d7fa700 1 -- 172.21.2.102:0/2588685682 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v107) v1 ==== 84+0+64 (secure 0 0 0) 0x7fbc8800bca0 con 0x7fbca00ada90 2021-02-10T11:31:45.603 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:31:45.603 INFO:teuthology.orchestra.run.gibba002.stdout:[ 2021-02-10T11:31:45.604 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:31:45.604 INFO:teuthology.orchestra.run.gibba002.stdout: "poolnum": 1, 2021-02-10T11:31:45.604 INFO:teuthology.orchestra.run.gibba002.stdout: "poolname": "rbd" 2021-02-10T11:31:45.604 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:31:45.604 INFO:teuthology.orchestra.run.gibba002.stdout:] 2021-02-10T11:31:45.604 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:45.603 7fbca6de1700 1 -- 172.21.2.102:0/2588685682 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fbc840186c0 msgr2=0x7fbc8401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:45.604 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:45.603 7fbca6de1700 1 --2- 172.21.2.102:0/2588685682 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fbc840186c0 0x7fbc8401abc0 crc :-1 s=READY pgs=960 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:45.605 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:45.603 7fbca6de1700 1 -- 172.21.2.102:0/2588685682 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbca00ada90 msgr2=0x7fbca00ade90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:45.605 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:45.603 7fbca6de1700 1 --2- 172.21.2.102:0/2588685682 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbca00ada90 0x7fbca00ade90 secure :-1 s=READY pgs=2175 cs=0 l=1 rev1=1 rx=0x7fbc88006f20 tx=0x7fbc8800b8b0).stop 2021-02-10T11:31:45.605 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:45.604 7fbca6de1700 1 -- 172.21.2.102:0/2588685682 shutdown_connections 2021-02-10T11:31:45.605 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:45.604 7fbca6de1700 1 --2- 172.21.2.102:0/2588685682 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fbc840186c0 0x7fbc8401abc0 unknown :-1 s=CLOSED pgs=960 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:45.605 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:45.604 7fbca6de1700 1 --2- 172.21.2.102:0/2588685682 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fbca009a740 0x7fbca0097bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:45.605 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:45.604 7fbca6de1700 1 --2- 172.21.2.102:0/2588685682 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbca0099d20 0x7fbca009a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:45.605 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:45.604 7fbca6de1700 1 --2- 172.21.2.102:0/2588685682 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbca00ada90 0x7fbca00ade90 unknown :-1 s=CLOSED pgs=2175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:45.605 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:45.604 7fbca6de1700 1 -- 172.21.2.102:0/2588685682 >> 172.21.2.102:0/2588685682 conn(0x7fbca00f0550 msgr2=0x7fbca0191730 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:45.606 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:45.604 7fbca6de1700 1 -- 172.21.2.102:0/2588685682 shutdown_connections 2021-02-10T11:31:45.606 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:45.604 7fbca6de1700 1 -- 172.21.2.102:0/2588685682 wait complete. 2021-02-10T11:31:45.614 INFO:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2021-02-10T11:31:45.614 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool create cephfs_metadata 8 2021-02-10T11:31:46.047 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:46.028 7fd3e3c9a700 1 Processor -- start 2021-02-10T11:31:46.048 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:46.028 7fd3e3c9a700 1 -- start start 2021-02-10T11:31:46.048 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:46.028 7fd3e3c9a700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fd3dc0d1870 0x7fd3dc1b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:46.048 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:46.029 7fd3e3c9a700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3dc0fe1c0 con 0x7fd3dc0cff70 2021-02-10T11:31:46.048 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:46.029 7fd3e3c9a700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3dc109a60 con 0x7fd3dc1b0ae0 2021-02-10T11:31:46.048 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:46.029 7fd3e3c9a700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3dc088070 con 0x7fd3dc0d1870 2021-02-10T11:31:46.048 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:46.029 7fd3e22b7700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fd3dc0cff70 0x7fd3dc109130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:54352/0 (socket says 172.21.2.102:54352) 2021-02-10T11:31:46.048 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:46.029 7fd3e22b7700 1 -- 172.21.2.102:0/389291140 learned_addr learned my addr 172.21.2.102:0/389291140 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:31:46.048 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:46.029 7fd3e0ab4700 1 -- 172.21.2.102:0/389291140 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3702193883 0 0) 0x7fd3d4001500 con 0x7fd3dc0cff70 2021-02-10T11:31:46.048 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:46.029 7fd3e0ab4700 1 -- 172.21.2.102:0/389291140 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd3c8002fe0 con 0x7fd3dc0cff70 2021-02-10T11:31:46.049 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:46.029 7fd3e0ab4700 1 -- 172.21.2.102:0/389291140 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1203012483 0 0) 0x7fd3d4001460 con 0x7fd3dc0cff70 2021-02-10T11:31:46.049 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:46.029 7fd3e0ab4700 1 -- 172.21.2.102:0/389291140 >> v2:172.21.2.113:3300/0 conn(0x7fd3dc0d1870 msgr2=0x7fd3dc1b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:46.049 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:46.029 7fd3e0ab4700 1 --2- 172.21.2.102:0/389291140 >> v2:172.21.2.113:3300/0 conn(0x7fd3dc0d1870 0x7fd3dc1b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:46.049 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:46.029 7fd3e0ab4700 1 -- 172.21.2.102:0/389291140 >> v1:172.21.2.113:6789/0 conn(0x7fd3dc1b0ae0 legacy=0x7fd3dc108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:46.049 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:46.029 7fd3e0ab4700 1 -- 172.21.2.102:0/389291140 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3dc088260 con 0x7fd3dc0cff70 2021-02-10T11:31:46.049 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:46.029 7fd3e0ab4700 1 -- 172.21.2.102:0/389291140 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7fd3d4001980 con 0x7fd3dc0cff70 2021-02-10T11:31:46.049 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:46.029 7fd3e0ab4700 1 -- 172.21.2.102:0/389291140 >> v1:172.21.2.102:6789/0 conn(0x7fd3dc0cff70 legacy=0x7fd3dc109130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:46.049 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:46.030 7fd3e0ab4700 1 --2- 172.21.2.102:0/389291140 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd3c8003a40 0x7fd3c8005ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:46.049 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:46.030 7fd3e0ab4700 1 --2- 172.21.2.102:0/389291140 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd3c8006770 0x7fd3c8008c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:46.049 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:46.030 7fd3e0ab4700 1 --2- 172.21.2.102:0/389291140 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd3c8009590 0x7fd3c800bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:46.049 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:46.030 7fd3e0ab4700 1 -- 172.21.2.102:0/389291140 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3c800c300 con 0x7fd3c8006770 2021-02-10T11:31:46.049 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:46.030 7fd3e0ab4700 1 -- 172.21.2.102:0/389291140 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3c800c520 con 0x7fd3c8009590 2021-02-10T11:31:46.049 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:46.030 7fd3e0ab4700 1 -- 172.21.2.102:0/389291140 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3c800c740 con 0x7fd3c8003a40 2021-02-10T11:31:46.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:46.030 7fd3e12b5700 1 --2- 172.21.2.102:0/389291140 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd3c8006770 0x7fd3c8008c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:46.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:46.030 7fd3e22b7700 1 --2- 172.21.2.102:0/389291140 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd3c8009590 0x7fd3c800bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:46.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:46.030 7fd3e1ab6700 1 --2- 172.21.2.102:0/389291140 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd3c8003a40 0x7fd3c8005ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:46.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:46.030 7fd3e12b5700 1 -- 172.21.2.102:0/389291140 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd3c8003a40 msgr2=0x7fd3c8005ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:46.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:46.030 7fd3e12b5700 1 --2- 172.21.2.102:0/389291140 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd3c8003a40 0x7fd3c8005ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:46.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:46.030 7fd3e12b5700 1 -- 172.21.2.102:0/389291140 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd3c8009590 msgr2=0x7fd3c800bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:46.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:46.030 7fd3e12b5700 1 --2- 172.21.2.102:0/389291140 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd3c8009590 0x7fd3c800bab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:46.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:46.030 7fd3e22b7700 1 -- 172.21.2.102:0/389291140 reap_dead start 2021-02-10T11:31:46.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:46.030 7fd3e12b5700 1 -- 172.21.2.102:0/389291140 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3c800c9e0 con 0x7fd3c8006770 2021-02-10T11:31:46.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:46.030 7fd3e12b5700 1 --2- 172.21.2.102:0/389291140 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd3c8006770 0x7fd3c8008c90 secure :-1 s=READY pgs=2177 cs=0 l=1 rev1=1 rx=0x7fd3d8006ab0 tx=0x7fd3d8009a90).ready entity=mon.0 client_cookie=5f757473328c2c67 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:46.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:46.031 7fd3e0ab4700 1 -- 172.21.2.102:0/389291140 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd3d80082e0 con 0x7fd3c8006770 2021-02-10T11:31:46.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:46.031 7fd3e0ab4700 1 -- 172.21.2.102:0/389291140 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3d8000e20 con 0x7fd3c8006770 2021-02-10T11:31:46.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:46.031 7fd3e0ab4700 1 -- 172.21.2.102:0/389291140 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd3d80085e0 con 0x7fd3c8006770 2021-02-10T11:31:46.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:46.031 7fd3e3c9a700 1 -- 172.21.2.102:0/389291140 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd3c8006770 msgr2=0x7fd3c8008c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:46.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:46.031 7fd3e3c9a700 1 --2- 172.21.2.102:0/389291140 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd3c8006770 0x7fd3c8008c90 secure :-1 s=READY pgs=2177 cs=0 l=1 rev1=1 rx=0x7fd3d8006ab0 tx=0x7fd3d8009a90).stop 2021-02-10T11:31:46.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:46.031 7fd3e3c9a700 1 -- 172.21.2.102:0/389291140 shutdown_connections 2021-02-10T11:31:46.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:46.031 7fd3e3c9a700 1 --2- 172.21.2.102:0/389291140 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd3c8006770 0x7fd3c8008c90 unknown :-1 s=CLOSED pgs=2177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:46.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:46.031 7fd3e3c9a700 1 -- 172.21.2.102:0/389291140 >> 172.21.2.102:0/389291140 conn(0x7fd3dc0ce420 msgr2=0x7fd3dc0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:46.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:46.031 7fd3e3c9a700 1 -- 172.21.2.102:0/389291140 shutdown_connections 2021-02-10T11:31:46.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:46.031 7fd3e3c9a700 1 -- 172.21.2.102:0/389291140 wait complete. 2021-02-10T11:31:46.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:46.046 7fd3e3c9a700 1 Processor -- start 2021-02-10T11:31:46.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:46.046 7fd3e3c9a700 1 -- start start 2021-02-10T11:31:46.053 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:46.046 7fd3e3c9a700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd3dc085bc0 0x7fd3dc0863e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:46.053 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:46.046 7fd3e3c9a700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd3dc0954c0 0x7fd3dc096300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:46.053 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:46.047 7fd3e3c9a700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd3dc0958c0 0x7fd3dc096920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:46.053 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:46.047 7fd3e3c9a700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3dc095d80 con 0x7fd3dc0954c0 2021-02-10T11:31:46.053 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:46.047 7fd3e3c9a700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3dc0bfa60 con 0x7fd3dc0958c0 2021-02-10T11:31:46.053 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:46.047 7fd3e3c9a700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3dc085fc0 con 0x7fd3dc085bc0 2021-02-10T11:31:46.053 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:46.047 7fd3e12b5700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd3dc0954c0 0x7fd3dc096300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:46.053 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:46.047 7fd3e12b5700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd3dc0954c0 0x7fd3dc096300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:33912/0 (socket says 172.21.2.102:33912) 2021-02-10T11:31:46.053 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:46.047 7fd3e12b5700 1 -- 172.21.2.102:0/1479395639 learned_addr learned my addr 172.21.2.102:0/1479395639 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:31:46.053 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:46.047 7fd3e1ab6700 1 --2- 172.21.2.102:0/1479395639 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd3dc085bc0 0x7fd3dc0863e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:46.053 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:46.047 7fd3e22b7700 1 --2- 172.21.2.102:0/1479395639 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd3dc0958c0 0x7fd3dc096920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:46.054 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:46.047 7fd3e12b5700 1 -- 172.21.2.102:0/1479395639 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd3dc085bc0 msgr2=0x7fd3dc0863e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:46.054 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:46.047 7fd3e12b5700 1 --2- 172.21.2.102:0/1479395639 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd3dc085bc0 0x7fd3dc0863e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:46.054 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:46.047 7fd3e12b5700 1 -- 172.21.2.102:0/1479395639 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd3dc0958c0 msgr2=0x7fd3dc096920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:46.054 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:46.047 7fd3e12b5700 1 --2- 172.21.2.102:0/1479395639 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd3dc0958c0 0x7fd3dc096920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:46.054 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:46.047 7fd3e12b5700 1 -- 172.21.2.102:0/1479395639 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3dc086c10 con 0x7fd3dc0954c0 2021-02-10T11:31:46.054 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:46.047 7fd3e22b7700 1 --2- 172.21.2.102:0/1479395639 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd3dc0958c0 0x7fd3dc096920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:31:46.054 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:46.047 7fd3e12b5700 1 --2- 172.21.2.102:0/1479395639 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd3dc0954c0 0x7fd3dc096300 secure :-1 s=READY pgs=2178 cs=0 l=1 rev1=1 rx=0x7fd3d80095b0 tx=0x7fd3d800b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:46.054 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:46.047 7fd3ce7fc700 1 -- 172.21.2.102:0/1479395639 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd3d800bd30 con 0x7fd3dc0954c0 2021-02-10T11:31:46.054 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:46.047 7fd3ce7fc700 1 -- 172.21.2.102:0/1479395639 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3d8002030 con 0x7fd3dc0954c0 2021-02-10T11:31:46.054 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:46.047 7fd3ce7fc700 1 -- 172.21.2.102:0/1479395639 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd3d8002550 con 0x7fd3dc0954c0 2021-02-10T11:31:46.054 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:46.047 7fd3e3c9a700 1 -- 172.21.2.102:0/1479395639 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3dc086c10 con 0x7fd3dc0954c0 2021-02-10T11:31:46.054 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:46.048 7fd3e3c9a700 1 -- 172.21.2.102:0/1479395639 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3dc086c10 con 0x7fd3dc0954c0 2021-02-10T11:31:46.054 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:46.048 7fd3ce7fc700 1 -- 172.21.2.102:0/1479395639 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd3d8006b20 con 0x7fd3dc0954c0 2021-02-10T11:31:46.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:46.048 7fd3ce7fc700 1 --2- 172.21.2.102:0/1479395639 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd3c40186c0 0x7fd3c401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:46.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:46.048 7fd3ce7fc700 1 -- 172.21.2.102:0/1479395639 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 4753+0+0 (secure 0 0 0) 0x7fd3d80322d0 con 0x7fd3dc0954c0 2021-02-10T11:31:46.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:46.048 7fd3e1ab6700 1 --2- 172.21.2.102:0/1479395639 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd3c40186c0 0x7fd3c401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:46.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:46.048 7fd3e1ab6700 1 --2- 172.21.2.102:0/1479395639 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd3c40186c0 0x7fd3c401abc0 crc :-1 s=READY pgs=961 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:46.080 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:46.078 7fd3e3c9a700 1 -- 172.21.2.102:0/1479395639 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3dc0fe1c0 con 0x7fd3dc0954c0 2021-02-10T11:31:46.082 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:46.081 7fd3ce7fc700 1 -- 172.21.2.102:0/1479395639 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd3d8000940 con 0x7fd3dc0954c0 2021-02-10T11:31:46.211 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:46.210 7fd3e3c9a700 1 -- 172.21.2.102:0/1479395639 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"} v 0) v1 -- 0x7fd3dc0fe1c0 con 0x7fd3dc0954c0 2021-02-10T11:31:46.621 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:46.620 7fd3ce7fc700 1 -- 172.21.2.102:0/1479395639 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"}]=0 pool 'cephfs_metadata' created v108) v1 ==== 133+0+0 (secure 0 0 0) 0x7fd3d8006b20 con 0x7fd3dc0954c0 2021-02-10T11:31:46.626 INFO:teuthology.orchestra.run.gibba002.stderr:pool 'cephfs_metadata' created 2021-02-10T11:31:46.628 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:46.627 7fd3e3c9a700 1 -- 172.21.2.102:0/1479395639 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd3c40186c0 msgr2=0x7fd3c401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:46.628 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:46.627 7fd3e3c9a700 1 --2- 172.21.2.102:0/1479395639 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd3c40186c0 0x7fd3c401abc0 crc :-1 s=READY pgs=961 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:46.629 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:46.627 7fd3e3c9a700 1 -- 172.21.2.102:0/1479395639 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd3dc0954c0 msgr2=0x7fd3dc096300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:46.629 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:46.627 7fd3e3c9a700 1 --2- 172.21.2.102:0/1479395639 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd3dc0954c0 0x7fd3dc096300 secure :-1 s=READY pgs=2178 cs=0 l=1 rev1=1 rx=0x7fd3d80095b0 tx=0x7fd3d800b940).stop 2021-02-10T11:31:46.629 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:46.627 7fd3e3c9a700 1 -- 172.21.2.102:0/1479395639 shutdown_connections 2021-02-10T11:31:46.629 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:46.627 7fd3e3c9a700 1 --2- 172.21.2.102:0/1479395639 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd3c40186c0 0x7fd3c401abc0 unknown :-1 s=CLOSED pgs=961 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:46.629 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:46.627 7fd3e3c9a700 1 --2- 172.21.2.102:0/1479395639 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd3dc0958c0 0x7fd3dc096920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:46.629 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:46.627 7fd3e3c9a700 1 --2- 172.21.2.102:0/1479395639 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd3dc0954c0 0x7fd3dc096300 unknown :-1 s=CLOSED pgs=2178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:46.629 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:46.627 7fd3e3c9a700 1 --2- 172.21.2.102:0/1479395639 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd3dc085bc0 0x7fd3dc0863e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:46.629 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:46.627 7fd3e3c9a700 1 -- 172.21.2.102:0/1479395639 >> 172.21.2.102:0/1479395639 conn(0x7fd3dc0bef00 msgr2=0x7fd3dc09d1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:46.629 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:46.627 7fd3e3c9a700 1 -- 172.21.2.102:0/1479395639 shutdown_connections 2021-02-10T11:31:46.629 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:46.627 7fd3e3c9a700 1 -- 172.21.2.102:0/1479395639 wait complete. 2021-02-10T11:31:46.637 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool create cephfs_data 8 2021-02-10T11:31:47.071 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:47.050 7f03d0ace700 1 Processor -- start 2021-02-10T11:31:47.071 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:47.050 7f03d0ace700 1 -- start start 2021-02-10T11:31:47.071 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:47.051 7f03d0ace700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f03cc0d1860 0x7f03cc1b04b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:47.071 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:47.051 7f03d0ace700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f03cc1b0ad0 0x7f03cc108930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:47.072 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:47.051 7f03d0ace700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03cc0fe1b0 con 0x7f03cc0d1c60 2021-02-10T11:31:47.072 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:47.051 7f03d0ace700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03cc109a50 con 0x7f03cc1b0ad0 2021-02-10T11:31:47.072 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:47.051 7f03d0ace700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03cc109740 con 0x7f03cc0d1860 2021-02-10T11:31:47.072 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:47.051 7f03ca05d700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f03cc1b0ad0 0x7f03cc108930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:47.072 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:47.051 7f03ca05d700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f03cc1b0ad0 0x7f03cc108930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:33916/0 (socket says 172.21.2.102:33916) 2021-02-10T11:31:47.073 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:47.051 7f03ca05d700 1 -- 172.21.2.102:0/2891579947 learned_addr learned my addr 172.21.2.102:0/2891579947 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:31:47.073 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:47.051 7f03ca85e700 1 --2- 172.21.2.102:0/2891579947 >> v2:172.21.2.113:3300/0 conn(0x7f03cc0d1860 0x7f03cc1b04b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:47.073 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:47.052 7f03ca85e700 1 -- 172.21.2.102:0/2891579947 >> v2:172.21.2.102:3300/0 conn(0x7f03cc1b0ad0 msgr2=0x7f03cc108930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:47.073 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:47.052 7f03ca85e700 1 --2- 172.21.2.102:0/2891579947 >> v2:172.21.2.102:3300/0 conn(0x7f03cc1b0ad0 0x7f03cc108930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:47.073 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:47.052 7f03ca85e700 1 -- 172.21.2.102:0/2891579947 >> v1:172.21.2.113:6789/0 conn(0x7f03cc0d1c60 legacy=0x7f03cc108f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:47.073 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:47.052 7f03ca85e700 1 -- 172.21.2.102:0/2891579947 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03cc088070 con 0x7f03cc0d1860 2021-02-10T11:31:47.074 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:47.052 7f03ca05d700 1 --2- 172.21.2.102:0/2891579947 >> v2:172.21.2.102:3300/0 conn(0x7f03cc1b0ad0 0x7f03cc108930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:31:47.074 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:47.052 7f03ca85e700 1 --2- 172.21.2.102:0/2891579947 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f03cc0d1860 0x7f03cc1b04b0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 rx=0x7f03b4006650 tx=0x7f03b4009a80).ready entity=mon.1 client_cookie=49bb90574b159060 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:47.074 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:47.052 7f03c985c700 1 -- 172.21.2.102:0/2891579947 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f03b40009b0 con 0x7f03cc0d1860 2021-02-10T11:31:47.074 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:47.052 7f03c985c700 1 -- 172.21.2.102:0/2891579947 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03b4006740 con 0x7f03cc0d1860 2021-02-10T11:31:47.074 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:47.052 7f03c985c700 1 -- 172.21.2.102:0/2891579947 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f03b400c0c0 con 0x7f03cc0d1860 2021-02-10T11:31:47.075 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:47.052 7f03d0ace700 1 -- 172.21.2.102:0/2891579947 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f03cc0d1860 msgr2=0x7f03cc1b04b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:47.075 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:47.052 7f03d0ace700 1 --2- 172.21.2.102:0/2891579947 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f03cc0d1860 0x7f03cc1b04b0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 rx=0x7f03b4006650 tx=0x7f03b4009a80).stop 2021-02-10T11:31:47.075 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:47.053 7f03d0ace700 1 -- 172.21.2.102:0/2891579947 shutdown_connections 2021-02-10T11:31:47.075 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:47.053 7f03d0ace700 1 --2- 172.21.2.102:0/2891579947 >> v2:172.21.2.102:3300/0 conn(0x7f03cc1b0ad0 0x7f03cc108930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:47.075 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:47.053 7f03d0ace700 1 --2- 172.21.2.102:0/2891579947 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f03cc0d1860 0x7f03cc1b04b0 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:47.075 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:47.053 7f03d0ace700 1 -- 172.21.2.102:0/2891579947 >> 172.21.2.102:0/2891579947 conn(0x7f03cc0ce410 msgr2=0x7f03cc0d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:47.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:47.053 7f03d0ace700 1 -- 172.21.2.102:0/2891579947 shutdown_connections 2021-02-10T11:31:47.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:47.053 7f03d0ace700 1 -- 172.21.2.102:0/2891579947 wait complete. 2021-02-10T11:31:47.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:47.068 7f03d0ace700 1 Processor -- start 2021-02-10T11:31:47.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:47.068 7f03d0ace700 1 -- start start 2021-02-10T11:31:47.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:47.068 7f03d0ace700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f03cc0badd0 0x7f03cc09eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:47.077 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:47.068 7f03d0ace700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f03cc0adf50 0x7f03cc09f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:47.077 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:47.068 7f03d0ace700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f03cc1a7470 0x7f03cc1a7870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:47.077 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:47.068 7f03d0ace700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03cc09e5a0 con 0x7f03cc1a7470 2021-02-10T11:31:47.077 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:47.068 7f03d0ace700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03cc09f6e0 con 0x7f03cc0adf50 2021-02-10T11:31:47.077 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:47.068 7f03d0ace700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03cc0bb220 con 0x7f03cc0badd0 2021-02-10T11:31:47.077 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:47.068 7f03cb05f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f03cc1a7470 0x7f03cc1a7870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:47.078 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:47.068 7f03cb05f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f03cc1a7470 0x7f03cc1a7870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:33920/0 (socket says 172.21.2.102:33920) 2021-02-10T11:31:47.078 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:47.068 7f03cb05f700 1 -- 172.21.2.102:0/368405699 learned_addr learned my addr 172.21.2.102:0/368405699 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:31:47.078 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:47.068 7f03cb05f700 1 -- 172.21.2.102:0/368405699 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f03cc0badd0 msgr2=0x7f03cc09eaa0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:31:47.078 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:47.069 7f03cb05f700 1 --2- 172.21.2.102:0/368405699 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f03cc0badd0 0x7f03cc09eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:47.078 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:47.069 7f03cb05f700 1 -- 172.21.2.102:0/368405699 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f03cc0adf50 msgr2=0x7f03cc09f0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:47.079 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:47.069 7f03cb05f700 1 --2- 172.21.2.102:0/368405699 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f03cc0adf50 0x7f03cc09f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:47.079 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:47.069 7f03cb05f700 1 -- 172.21.2.102:0/368405699 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03cc1a7e90 con 0x7f03cc1a7470 2021-02-10T11:31:47.079 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:47.069 7f03cb05f700 1 --2- 172.21.2.102:0/368405699 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f03cc1a7470 0x7f03cc1a7870 secure :-1 s=READY pgs=2179 cs=0 l=1 rev1=1 rx=0x7f03c4009980 tx=0x7f03c400bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:47.079 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:47.069 7f03c37fe700 1 -- 172.21.2.102:0/368405699 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f03c4003030 con 0x7f03cc1a7470 2021-02-10T11:31:47.079 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:47.069 7f03c37fe700 1 -- 172.21.2.102:0/368405699 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03c4001400 con 0x7f03cc1a7470 2021-02-10T11:31:47.080 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:47.069 7f03c37fe700 1 -- 172.21.2.102:0/368405699 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f03c4001920 con 0x7f03cc1a7470 2021-02-10T11:31:47.080 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:47.069 7f03d0ace700 1 -- 172.21.2.102:0/368405699 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03cc1a7e90 con 0x7f03cc1a7470 2021-02-10T11:31:47.080 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:47.069 7f03d0ace700 1 -- 172.21.2.102:0/368405699 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03cc1a7e90 con 0x7f03cc1a7470 2021-02-10T11:31:47.081 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:47.070 7f03c37fe700 1 -- 172.21.2.102:0/368405699 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f03c40015e0 con 0x7f03cc1a7470 2021-02-10T11:31:47.081 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:47.070 7f03c37fe700 1 --2- 172.21.2.102:0/368405699 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f03b00186c0 0x7f03b001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:47.082 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:47.070 7f03c37fe700 1 -- 172.21.2.102:0/368405699 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5107+0+0 (secure 0 0 0) 0x7f03c4032730 con 0x7f03cc1a7470 2021-02-10T11:31:47.082 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:47.070 7f03ca85e700 1 --2- 172.21.2.102:0/368405699 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f03b00186c0 0x7f03b001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:47.082 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:47.071 7f03ca85e700 1 --2- 172.21.2.102:0/368405699 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f03b00186c0 0x7f03b001abc0 crc :-1 s=READY pgs=962 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:47.102 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:47.101 7f03d0ace700 1 -- 172.21.2.102:0/368405699 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03cc09f6e0 con 0x7f03cc1a7470 2021-02-10T11:31:47.105 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:47.104 7f03c37fe700 1 -- 172.21.2.102:0/368405699 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f03c400be10 con 0x7f03cc1a7470 2021-02-10T11:31:47.236 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:47.234 7f03d0ace700 1 -- 172.21.2.102:0/368405699 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_data"} v 0) v1 -- 0x7f03cc09f6e0 con 0x7f03cc1a7470 2021-02-10T11:31:47.629 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:47.628 7f03c37fe700 1 -- 172.21.2.102:0/368405699 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_data"}]=0 pool 'cephfs_data' created v110) v1 ==== 125+0+0 (secure 0 0 0) 0x7f03c40015a0 con 0x7f03cc1a7470 2021-02-10T11:31:47.652 INFO:teuthology.orchestra.run.gibba002.stderr:pool 'cephfs_data' created 2021-02-10T11:31:47.653 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:47.652 7f03d0ace700 1 -- 172.21.2.102:0/368405699 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f03b00186c0 msgr2=0x7f03b001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:47.653 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:47.652 7f03d0ace700 1 --2- 172.21.2.102:0/368405699 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f03b00186c0 0x7f03b001abc0 crc :-1 s=READY pgs=962 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:47.653 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:47.652 7f03d0ace700 1 -- 172.21.2.102:0/368405699 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f03cc1a7470 msgr2=0x7f03cc1a7870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:47.653 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:47.652 7f03d0ace700 1 --2- 172.21.2.102:0/368405699 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f03cc1a7470 0x7f03cc1a7870 secure :-1 s=READY pgs=2179 cs=0 l=1 rev1=1 rx=0x7f03c4009980 tx=0x7f03c400bcd0).stop 2021-02-10T11:31:47.654 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:47.652 7f03d0ace700 1 -- 172.21.2.102:0/368405699 shutdown_connections 2021-02-10T11:31:47.654 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:47.652 7f03d0ace700 1 --2- 172.21.2.102:0/368405699 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f03cc1a7470 0x7f03cc1a7870 unknown :-1 s=CLOSED pgs=2179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:47.654 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:47.652 7f03d0ace700 1 --2- 172.21.2.102:0/368405699 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f03b00186c0 0x7f03b001abc0 unknown :-1 s=CLOSED pgs=962 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:47.654 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:47.652 7f03d0ace700 1 --2- 172.21.2.102:0/368405699 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f03cc0adf50 0x7f03cc09f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:47.654 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:47.652 7f03d0ace700 1 --2- 172.21.2.102:0/368405699 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f03cc0badd0 0x7f03cc09eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:47.654 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:47.652 7f03d0ace700 1 -- 172.21.2.102:0/368405699 >> 172.21.2.102:0/368405699 conn(0x7f03cc0bee90 msgr2=0x7f03cc0bb880 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:47.654 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:47.652 7f03d0ace700 1 -- 172.21.2.102:0/368405699 shutdown_connections 2021-02-10T11:31:47.654 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:47.652 7f03d0ace700 1 -- 172.21.2.102:0/368405699 wait complete. 2021-02-10T11:31:47.661 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2021-02-10T11:31:48.095 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.076 7fb56e582700 1 Processor -- start 2021-02-10T11:31:48.095 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.076 7fb56e582700 1 -- start start 2021-02-10T11:31:48.095 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.076 7fb56e582700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fb5680d1880 0x7fb5681b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:48.095 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.076 7fb56e582700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5680fe1d0 con 0x7fb5680cff80 2021-02-10T11:31:48.095 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.076 7fb56e582700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb568109a70 con 0x7fb5681b0af0 2021-02-10T11:31:48.095 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.076 7fb56e582700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb568088070 con 0x7fb5680d1880 2021-02-10T11:31:48.096 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.076 7fb56cb9f700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fb5680cff80 0x7fb568109140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:54390/0 (socket says 172.21.2.102:54390) 2021-02-10T11:31:48.096 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.076 7fb56cb9f700 1 -- 172.21.2.102:0/4101321918 learned_addr learned my addr 172.21.2.102:0/4101321918 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:31:48.096 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.077 7fb567fff700 1 --2- 172.21.2.102:0/4101321918 >> v2:172.21.2.113:3300/0 conn(0x7fb5680d1880 0x7fb5681b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:48.096 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.077 7fb5677fe700 1 -- 172.21.2.102:0/4101321918 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3547845107 0 0) 0x7fb560000f90 con 0x7fb5680cff80 2021-02-10T11:31:48.096 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.077 7fb5677fe700 1 -- 172.21.2.102:0/4101321918 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb554002fe0 con 0x7fb5680cff80 2021-02-10T11:31:48.096 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.077 7fb5677fe700 1 -- 172.21.2.102:0/4101321918 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4158139102 0 0) 0x7fb5600014c0 con 0x7fb5680cff80 2021-02-10T11:31:48.096 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.077 7fb5677fe700 1 -- 172.21.2.102:0/4101321918 >> v2:172.21.2.113:3300/0 conn(0x7fb5680d1880 msgr2=0x7fb5681b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:48.096 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.077 7fb5677fe700 1 --2- 172.21.2.102:0/4101321918 >> v2:172.21.2.113:3300/0 conn(0x7fb5680d1880 0x7fb5681b04d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:48.096 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.077 7fb5677fe700 1 -- 172.21.2.102:0/4101321918 >> v1:172.21.2.113:6789/0 conn(0x7fb5681b0af0 legacy=0x7fb568108950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:48.096 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.077 7fb5677fe700 1 -- 172.21.2.102:0/4101321918 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb568088260 con 0x7fb5680cff80 2021-02-10T11:31:48.096 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.077 7fb5677fe700 1 -- 172.21.2.102:0/4101321918 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7fb5600019c0 con 0x7fb5680cff80 2021-02-10T11:31:48.096 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.077 7fb5677fe700 1 -- 172.21.2.102:0/4101321918 >> v1:172.21.2.102:6789/0 conn(0x7fb5680cff80 legacy=0x7fb568109140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:48.096 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.077 7fb5677fe700 1 --2- 172.21.2.102:0/4101321918 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb554003b90 0x7fb5540060b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:48.097 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.077 7fb5677fe700 1 --2- 172.21.2.102:0/4101321918 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb554006840 0x7fb554008d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:48.097 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.077 7fb5677fe700 1 --2- 172.21.2.102:0/4101321918 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb554009660 0x7fb55400bb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:48.097 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.077 7fb5677fe700 1 -- 172.21.2.102:0/4101321918 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb55400c390 con 0x7fb554003b90 2021-02-10T11:31:48.097 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.077 7fb5677fe700 1 -- 172.21.2.102:0/4101321918 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb55400c5b0 con 0x7fb554006840 2021-02-10T11:31:48.097 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.077 7fb5677fe700 1 -- 172.21.2.102:0/4101321918 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb55400c7d0 con 0x7fb554009660 2021-02-10T11:31:48.097 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.078 7fb567fff700 1 --2- 172.21.2.102:0/4101321918 >> v2:172.21.2.113:3300/0 conn(0x7fb5680d1880 0x7fb5681b04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:31:48.097 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.078 7fb55ffff700 1 --2- 172.21.2.102:0/4101321918 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb554006840 0x7fb554008d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:48.097 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.078 7fb567fff700 1 --2- 172.21.2.102:0/4101321918 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb554003b90 0x7fb5540060b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:48.097 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.078 7fb56cb9f700 1 --2- 172.21.2.102:0/4101321918 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb554009660 0x7fb55400bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:48.097 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.078 7fb567fff700 1 -- 172.21.2.102:0/4101321918 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb554009660 msgr2=0x7fb55400bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:48.097 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.078 7fb567fff700 1 --2- 172.21.2.102:0/4101321918 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb554009660 0x7fb55400bb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:48.097 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.078 7fb567fff700 1 -- 172.21.2.102:0/4101321918 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb554006840 msgr2=0x7fb554008d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:48.097 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.078 7fb567fff700 1 --2- 172.21.2.102:0/4101321918 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb554006840 0x7fb554008d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:48.098 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.078 7fb567fff700 1 -- 172.21.2.102:0/4101321918 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb554050a40 con 0x7fb554003b90 2021-02-10T11:31:48.098 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.078 7fb56cb9f700 1 --2- 172.21.2.102:0/4101321918 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb554009660 0x7fb55400bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:31:48.098 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.078 7fb56cb9f700 1 -- 172.21.2.102:0/4101321918 reap_dead start 2021-02-10T11:31:48.098 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.078 7fb567fff700 1 --2- 172.21.2.102:0/4101321918 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb554003b90 0x7fb5540060b0 secure :-1 s=READY pgs=2181 cs=0 l=1 rev1=1 rx=0x7fb550000b80 tx=0x7fb550009e80).ready entity=mon.0 client_cookie=c19f2f45ed3791af server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:48.098 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.078 7fb5677fe700 1 -- 172.21.2.102:0/4101321918 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb550006dc0 con 0x7fb554003b90 2021-02-10T11:31:48.098 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.078 7fb5677fe700 1 -- 172.21.2.102:0/4101321918 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb55000c370 con 0x7fb554003b90 2021-02-10T11:31:48.098 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.078 7fb5677fe700 1 -- 172.21.2.102:0/4101321918 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb55000c7c0 con 0x7fb554003b90 2021-02-10T11:31:48.098 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.078 7fb56e582700 1 -- 172.21.2.102:0/4101321918 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb554003b90 msgr2=0x7fb5540060b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:48.098 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.078 7fb56e582700 1 --2- 172.21.2.102:0/4101321918 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb554003b90 0x7fb5540060b0 secure :-1 s=READY pgs=2181 cs=0 l=1 rev1=1 rx=0x7fb550000b80 tx=0x7fb550009e80).stop 2021-02-10T11:31:48.098 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.079 7fb56e582700 1 -- 172.21.2.102:0/4101321918 shutdown_connections 2021-02-10T11:31:48.098 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.079 7fb56e582700 1 --2- 172.21.2.102:0/4101321918 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb554003b90 0x7fb5540060b0 unknown :-1 s=CLOSED pgs=2181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:48.098 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.079 7fb56e582700 1 -- 172.21.2.102:0/4101321918 >> 172.21.2.102:0/4101321918 conn(0x7fb5680ce430 msgr2=0x7fb5680d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:48.099 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.079 7fb56e582700 1 -- 172.21.2.102:0/4101321918 shutdown_connections 2021-02-10T11:31:48.099 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.079 7fb56e582700 1 -- 172.21.2.102:0/4101321918 wait complete. 2021-02-10T11:31:48.099 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.094 7fb56e582700 1 Processor -- start 2021-02-10T11:31:48.099 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.094 7fb56e582700 1 -- start start 2021-02-10T11:31:48.100 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.094 7fb56e582700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb5681a3570 0x7fb5681a3970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:48.100 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.094 7fb56e582700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb56809c9f0 0x7fb56809cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:48.100 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.094 7fb56e582700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb56809d410 0x7fb568098170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:48.100 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.094 7fb56e582700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb56809c470 con 0x7fb56809c9f0 2021-02-10T11:31:48.100 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.094 7fb56e582700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5680b2840 con 0x7fb56809d410 2021-02-10T11:31:48.101 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.094 7fb56e582700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5680b2a20 con 0x7fb5681a3570 2021-02-10T11:31:48.101 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.094 7fb55ffff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb56809c9f0 0x7fb56809cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:48.101 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.094 7fb55ffff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb56809c9f0 0x7fb56809cdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:33944/0 (socket says 172.21.2.102:33944) 2021-02-10T11:31:48.101 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.094 7fb55ffff700 1 -- 172.21.2.102:0/2595615777 learned_addr learned my addr 172.21.2.102:0/2595615777 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:31:48.101 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.095 7fb55ffff700 1 -- 172.21.2.102:0/2595615777 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb5681a3570 msgr2=0x7fb5681a3970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:48.101 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.095 7fb55ffff700 1 --2- 172.21.2.102:0/2595615777 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb5681a3570 0x7fb5681a3970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:48.101 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.095 7fb55ffff700 1 -- 172.21.2.102:0/2595615777 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb56809d410 msgr2=0x7fb568098170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:48.101 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.095 7fb55ffff700 1 --2- 172.21.2.102:0/2595615777 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb56809d410 0x7fb568098170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:48.101 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.095 7fb55ffff700 1 -- 172.21.2.102:0/2595615777 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb56809d880 con 0x7fb56809c9f0 2021-02-10T11:31:48.101 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.095 7fb55ffff700 1 --2- 172.21.2.102:0/2595615777 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb56809c9f0 0x7fb56809cdf0 secure :-1 s=READY pgs=2182 cs=0 l=1 rev1=1 rx=0x7fb55800b610 tx=0x7fb55800b9b0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:48.101 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.095 7fb5657fa700 1 -- 172.21.2.102:0/2595615777 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb558000960 con 0x7fb56809c9f0 2021-02-10T11:31:48.101 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.095 7fb5657fa700 1 -- 172.21.2.102:0/2595615777 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb55800bda0 con 0x7fb56809c9f0 2021-02-10T11:31:48.102 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.095 7fb56e582700 1 -- 172.21.2.102:0/2595615777 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb56809d880 con 0x7fb56809c9f0 2021-02-10T11:31:48.102 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.095 7fb56e582700 1 -- 172.21.2.102:0/2595615777 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb568098a80 con 0x7fb56809c9f0 2021-02-10T11:31:48.102 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.095 7fb5657fa700 1 -- 172.21.2.102:0/2595615777 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb558002030 con 0x7fb56809c9f0 2021-02-10T11:31:48.102 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.096 7fb5657fa700 1 -- 172.21.2.102:0/2595615777 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fb558000ca0 con 0x7fb56809c9f0 2021-02-10T11:31:48.102 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.096 7fb5657fa700 1 --2- 172.21.2.102:0/2595615777 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb54c0186c0 0x7fb54c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:48.102 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.096 7fb5657fa700 1 -- 172.21.2.102:0/2595615777 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5457+0+0 (secure 0 0 0) 0x7fb558032560 con 0x7fb56809c9f0 2021-02-10T11:31:48.102 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.096 7fb567fff700 1 --2- 172.21.2.102:0/2595615777 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb54c0186c0 0x7fb54c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:48.102 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.096 7fb567fff700 1 --2- 172.21.2.102:0/2595615777 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb54c0186c0 0x7fb54c01abc0 crc :-1 s=READY pgs=965 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:48.127 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.126 7fb56e582700 1 -- 172.21.2.102:0/2595615777 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb568082a70 con 0x7fb56809c9f0 2021-02-10T11:31:48.129 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.129 7fb5657fa700 1 -- 172.21.2.102:0/2595615777 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fb558000940 con 0x7fb56809c9f0 2021-02-10T11:31:48.259 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.257 7fb56e582700 1 -- 172.21.2.102:0/2595615777 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"data": "cephfs_data", "prefix": "fs new", "force": true, "fs_name": "cephfs", "metadata": "cephfs_metadata"} v 0) v1 -- 0x7fb568082a70 con 0x7fb56809c9f0 2021-02-10T11:31:48.263 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.261 7fb5657fa700 1 -- 172.21.2.102:0/2595615777 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"data": "cephfs_data", "prefix": "fs new", "force": true, "fs_name": "cephfs", "metadata": "cephfs_metadata"}]=0 new fs with metadata pool 22 and data pool 23 v88) v1 ==== 189+0+0 (secure 0 0 0) 0x7fb558000ca0 con 0x7fb56809c9f0 2021-02-10T11:31:48.266 INFO:teuthology.orchestra.run.gibba002.stderr:new fs with metadata pool 22 and data pool 23 2021-02-10T11:31:48.267 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.267 7fb56e582700 1 -- 172.21.2.102:0/2595615777 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb54c0186c0 msgr2=0x7fb54c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:48.268 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.267 7fb56e582700 1 --2- 172.21.2.102:0/2595615777 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb54c0186c0 0x7fb54c01abc0 crc :-1 s=READY pgs=965 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:48.268 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.267 7fb56e582700 1 -- 172.21.2.102:0/2595615777 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb56809c9f0 msgr2=0x7fb56809cdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:48.268 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.267 7fb56e582700 1 --2- 172.21.2.102:0/2595615777 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb56809c9f0 0x7fb56809cdf0 secure :-1 s=READY pgs=2182 cs=0 l=1 rev1=1 rx=0x7fb55800b610 tx=0x7fb55800b9b0).stop 2021-02-10T11:31:48.269 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.267 7fb56e582700 1 -- 172.21.2.102:0/2595615777 shutdown_connections 2021-02-10T11:31:48.269 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.267 7fb56e582700 1 --2- 172.21.2.102:0/2595615777 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb54c0186c0 0x7fb54c01abc0 unknown :-1 s=CLOSED pgs=965 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:48.269 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.267 7fb56e582700 1 --2- 172.21.2.102:0/2595615777 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb56809d410 0x7fb568098170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:48.269 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.267 7fb56e582700 1 --2- 172.21.2.102:0/2595615777 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb56809c9f0 0x7fb56809cdf0 unknown :-1 s=CLOSED pgs=2182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:48.269 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.267 7fb56e582700 1 --2- 172.21.2.102:0/2595615777 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb5681a3570 0x7fb5681a3970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:48.269 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.267 7fb56e582700 1 -- 172.21.2.102:0/2595615777 >> 172.21.2.102:0/2595615777 conn(0x7fb5680c2cd0 msgr2=0x7fb5680b5950 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:48.269 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.267 7fb56e582700 1 -- 172.21.2.102:0/2595615777 shutdown_connections 2021-02-10T11:31:48.269 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.267 7fb56e582700 1 -- 172.21.2.102:0/2595615777 wait complete. 2021-02-10T11:31:48.276 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:31:48.707 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.688 7f6209aa4700 1 Processor -- start 2021-02-10T11:31:48.708 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.688 7f6209aa4700 1 -- start start 2021-02-10T11:31:48.708 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.689 7f6209aa4700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f62040d1870 0x7f62041b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:48.708 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.689 7f6209aa4700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f62041b0a10 0x7f62040cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:48.708 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.689 7f6209aa4700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62040f0b10 con 0x7f62040d1c70 2021-02-10T11:31:48.708 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.689 7f6209aa4700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62040fe230 con 0x7f62041b0a10 2021-02-10T11:31:48.708 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.689 7f6209aa4700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62040e5ed0 con 0x7f62040d1870 2021-02-10T11:31:48.708 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.689 7f6203fff700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f62040d1c70 0x7f62040cf950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:54406/0 (socket says 172.21.2.102:54406) 2021-02-10T11:31:48.708 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.689 7f6203fff700 1 -- 172.21.2.102:0/3894880831 learned_addr learned my addr 172.21.2.102:0/3894880831 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:31:48.708 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.689 7f62027fc700 1 -- 172.21.2.102:0/3894880831 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1095875092 0 0) 0x7f61fc001500 con 0x7f62040d1c70 2021-02-10T11:31:48.708 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.689 7f62027fc700 1 -- 172.21.2.102:0/3894880831 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61f0002fe0 con 0x7f62040d1c70 2021-02-10T11:31:48.708 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.690 7f62027fc700 1 -- 172.21.2.102:0/3894880831 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1600727082 0 0) 0x7f61fc001500 con 0x7f62040d1c70 2021-02-10T11:31:48.708 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.690 7f62027fc700 1 -- 172.21.2.102:0/3894880831 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f62040d1870 msgr2=0x7f62041b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:48.708 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.690 7f62027fc700 1 --2- 172.21.2.102:0/3894880831 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f62040d1870 0x7f62041b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:48.709 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.690 7f62027fc700 1 -- 172.21.2.102:0/3894880831 >> v2:172.21.2.113:3300/0 conn(0x7f62041b0a10 msgr2=0x7f62040cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:48.709 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.690 7f62027fc700 1 --2- 172.21.2.102:0/3894880831 >> v2:172.21.2.113:3300/0 conn(0x7f62041b0a10 0x7f62040cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:48.709 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.690 7f62027fc700 1 -- 172.21.2.102:0/3894880831 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62040f0650 con 0x7f62040d1c70 2021-02-10T11:31:48.709 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.690 7f62027fc700 1 -- 172.21.2.102:0/3894880831 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7f61fc001a00 con 0x7f62040d1c70 2021-02-10T11:31:48.709 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.690 7f62027fc700 1 -- 172.21.2.102:0/3894880831 >> v1:172.21.2.102:6789/0 conn(0x7f62040d1c70 legacy=0x7f62040cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:48.709 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.690 7f62027fc700 1 --2- 172.21.2.102:0/3894880831 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f61f0003a60 0x7f61f0005ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:48.709 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.690 7f62027fc700 1 --2- 172.21.2.102:0/3894880831 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f61f0006770 0x7f61f0008c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:48.709 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.690 7f62027fc700 1 --2- 172.21.2.102:0/3894880831 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f61f0009590 0x7f61f000ba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:48.709 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.690 7f62027fc700 1 -- 172.21.2.102:0/3894880831 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61f000c2d0 con 0x7f61f0006770 2021-02-10T11:31:48.709 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.690 7f62027fc700 1 -- 172.21.2.102:0/3894880831 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61f000c4f0 con 0x7f61f0003a60 2021-02-10T11:31:48.709 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.690 7f62027fc700 1 -- 172.21.2.102:0/3894880831 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61f000c710 con 0x7f61f0009590 2021-02-10T11:31:48.709 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.690 7f6203fff700 1 --2- 172.21.2.102:0/3894880831 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f61f0009590 0x7f61f000ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:48.709 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.690 7f6202ffd700 1 --2- 172.21.2.102:0/3894880831 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f61f0006770 0x7f61f0008c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:48.710 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.690 7f62037fe700 1 --2- 172.21.2.102:0/3894880831 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f61f0003a60 0x7f61f0005ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:48.710 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.690 7f6203fff700 1 -- 172.21.2.102:0/3894880831 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f61f0003a60 msgr2=0x7f61f0005ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:48.710 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.690 7f6203fff700 1 --2- 172.21.2.102:0/3894880831 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f61f0003a60 0x7f61f0005ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:48.710 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.690 7f6203fff700 1 -- 172.21.2.102:0/3894880831 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f61f0006770 msgr2=0x7f61f0008c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:48.710 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.690 7f6203fff700 1 --2- 172.21.2.102:0/3894880831 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f61f0006770 0x7f61f0008c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:48.710 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.690 7f6203fff700 1 -- 172.21.2.102:0/3894880831 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61f000c9b0 con 0x7f61f0009590 2021-02-10T11:31:48.710 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.690 7f6202ffd700 1 --2- 172.21.2.102:0/3894880831 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f61f0006770 0x7f61f0008c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:31:48.710 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.691 7f6203fff700 1 --2- 172.21.2.102:0/3894880831 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f61f0009590 0x7f61f000ba80 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f61fc007fc0 tx=0x7f61fc04e320).ready entity=mon.2 client_cookie=59f952939754bc06 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:48.710 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.691 7f62027fc700 1 -- 172.21.2.102:0/3894880831 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f61fc007870 con 0x7f61f0009590 2021-02-10T11:31:48.710 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.691 7f62027fc700 1 -- 172.21.2.102:0/3894880831 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61fc000ee0 con 0x7f61f0009590 2021-02-10T11:31:48.710 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.691 7f62027fc700 1 -- 172.21.2.102:0/3894880831 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f61fc007300 con 0x7f61f0009590 2021-02-10T11:31:48.710 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.691 7f6209aa4700 1 -- 172.21.2.102:0/3894880831 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f61f0009590 msgr2=0x7f61f000ba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:48.710 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.691 7f6209aa4700 1 --2- 172.21.2.102:0/3894880831 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f61f0009590 0x7f61f000ba80 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f61fc007fc0 tx=0x7f61fc04e320).stop 2021-02-10T11:31:48.711 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.691 7f6203fff700 1 -- 172.21.2.102:0/3894880831 reap_dead start 2021-02-10T11:31:48.711 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.691 7f6209aa4700 1 -- 172.21.2.102:0/3894880831 shutdown_connections 2021-02-10T11:31:48.711 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.691 7f6209aa4700 1 -- 172.21.2.102:0/3894880831 >> 172.21.2.102:0/3894880831 conn(0x7f62040ce450 msgr2=0x7f62040d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:48.711 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.691 7f6209aa4700 1 -- 172.21.2.102:0/3894880831 shutdown_connections 2021-02-10T11:31:48.711 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.691 7f6209aa4700 1 -- 172.21.2.102:0/3894880831 wait complete. 2021-02-10T11:31:48.711 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.706 7f6209aa4700 1 Processor -- start 2021-02-10T11:31:48.711 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.706 7f6209aa4700 1 -- start start 2021-02-10T11:31:48.713 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.707 7f6209aa4700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f62041a05d0 0x7f62040a87c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:48.713 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.707 7f6209aa4700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f62040a8de0 0x7f62040a2630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:48.713 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.707 7f6209aa4700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f62040a2c50 0x7f62040a3050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:48.713 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.707 7f6209aa4700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62040b32d0 con 0x7f62040a8de0 2021-02-10T11:31:48.713 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.707 7f6209aa4700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62040a91e0 con 0x7f62040a2c50 2021-02-10T11:31:48.713 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.707 7f6209aa4700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62040b37d0 con 0x7f62041a05d0 2021-02-10T11:31:48.713 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.707 7f6202ffd700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f62040a8de0 0x7f62040a2630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:48.713 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.707 7f6202ffd700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f62040a8de0 0x7f62040a2630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:33964/0 (socket says 172.21.2.102:33964) 2021-02-10T11:31:48.713 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.707 7f6202ffd700 1 -- 172.21.2.102:0/1900381140 learned_addr learned my addr 172.21.2.102:0/1900381140 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:31:48.713 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.707 7f6203fff700 1 --2- 172.21.2.102:0/1900381140 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f62040a2c50 0x7f62040a3050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:48.713 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.707 7f6202ffd700 1 -- 172.21.2.102:0/1900381140 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f62041a05d0 msgr2=0x7f62040a87c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:48.713 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.707 7f62037fe700 1 --2- 172.21.2.102:0/1900381140 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f62041a05d0 0x7f62040a87c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:48.713 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.707 7f6202ffd700 1 --2- 172.21.2.102:0/1900381140 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f62041a05d0 0x7f62040a87c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:48.714 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.707 7f6202ffd700 1 -- 172.21.2.102:0/1900381140 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f62040a2c50 msgr2=0x7f62040a3050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:48.714 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.707 7f6202ffd700 1 --2- 172.21.2.102:0/1900381140 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f62040a2c50 0x7f62040a3050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:48.714 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.707 7f6202ffd700 1 -- 172.21.2.102:0/1900381140 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62040b39b0 con 0x7f62040a8de0 2021-02-10T11:31:48.714 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.707 7f6203fff700 1 --2- 172.21.2.102:0/1900381140 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f62040a2c50 0x7f62040a3050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:31:48.714 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.707 7f62037fe700 1 --2- 172.21.2.102:0/1900381140 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f62041a05d0 0x7f62040a87c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:31:48.714 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.707 7f6202ffd700 1 --2- 172.21.2.102:0/1900381140 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f62040a8de0 0x7f62040a2630 secure :-1 s=READY pgs=2184 cs=0 l=1 rev1=1 rx=0x7f61f4006f20 tx=0x7f61f400b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:48.714 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.708 7f61fbfff700 1 -- 172.21.2.102:0/1900381140 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f61f400bd50 con 0x7f62040a8de0 2021-02-10T11:31:48.714 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.708 7f61fbfff700 1 -- 172.21.2.102:0/1900381140 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61f4002080 con 0x7f62040a8de0 2021-02-10T11:31:48.714 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.708 7f61fbfff700 1 -- 172.21.2.102:0/1900381140 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f61f40024e0 con 0x7f62040a8de0 2021-02-10T11:31:48.714 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.708 7f6209aa4700 1 -- 172.21.2.102:0/1900381140 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62040b39b0 con 0x7f62040a8de0 2021-02-10T11:31:48.714 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.708 7f6209aa4700 1 -- 172.21.2.102:0/1900381140 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62040b39b0 con 0x7f62040a8de0 2021-02-10T11:31:48.714 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.708 7f61fbfff700 1 -- 172.21.2.102:0/1900381140 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f61f4000c90 con 0x7f62040a8de0 2021-02-10T11:31:48.715 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.708 7f61fbfff700 1 --2- 172.21.2.102:0/1900381140 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f61e80186c0 0x7f61e801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:48.715 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.708 7f61fbfff700 1 -- 172.21.2.102:0/1900381140 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f61f4032590 con 0x7f62040a8de0 2021-02-10T11:31:48.715 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.709 7f62037fe700 1 --2- 172.21.2.102:0/1900381140 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f61e80186c0 0x7f61e801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:48.715 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.709 7f62037fe700 1 --2- 172.21.2.102:0/1900381140 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f61e80186c0 0x7f61e801abc0 crc :-1 s=READY pgs=966 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:48.740 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.739 7f6209aa4700 1 -- 172.21.2.102:0/1900381140 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62040ebc80 con 0x7f62040a8de0 2021-02-10T11:31:48.742 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.741 7f61fbfff700 1 -- 172.21.2.102:0/1900381140 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f61f4006bb0 con 0x7f62040a8de0 2021-02-10T11:31:48.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.870 7f6209aa4700 1 -- 172.21.2.102:0/1900381140 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f62040f0b10 con 0x7f62040a8de0 2021-02-10T11:31:48.872 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.871 7f61fbfff700 1 -- 172.21.2.102:0/1900381140 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v111) v1 ==== 74+0+14020 (secure 0 0 0) 0x7f61f402bcc0 con 0x7f62040a8de0 2021-02-10T11:31:48.873 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:31:48.873 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":111,"fsid":"68db30ea-dbce-4aef-a6df-f01c6fea0513","created":"2021-02-10 11:03:06.331557","modified":"2021-02-10 11:31:48.259497","last_up_change":"2021-02-10 11:03:32.690562","last_in_change":"2021-02-10 11:03:32.690562","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":23,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:03:37.322519","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":22,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:31:46.228075","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"111","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":23,"pool_name":"cephfs_data","create_time":"2021-02-10 11:31:47.252676","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"111","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"be691b4d-a039-408a-af80-9d53f02b6f1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6818","nonce":36088},{"type":"v1","addr":"172.21.2.102:6820","nonce":36088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6822","nonce":36088},{"type":"v1","addr":"172.21.2.102:6824","nonce":36088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6830","nonce":36088},{"type":"v1","addr":"172.21.2.102:6832","nonce":36088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6826","nonce":36088},{"type":"v1","addr":"172.21.2.102:6828","nonce":36088}]},"public_addr":"172.21.2.102:6820/36088","cluster_addr":"172.21.2.102:6824/36088","heartbeat_back_addr":"172.21.2.102:6832/36088","heartbeat_front_addr":"172.21.2.102:6828/36088","state":["exists","up"]},{"osd":1,"uuid":"64e3ccc8-9c69-4d7c-865e-af8e83822635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6819","nonce":36087},{"type":"v1","addr":"172.21.2.102:6821","nonce":36087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6823","nonce":36087},{"type":"v1","addr":"172.21.2.102:6825","nonce":36087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6831","nonce":36087},{"type":"v1","addr":"172.21.2.102:6833","nonce":36087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6827","nonce":36087},{"type":"v1","addr":"172.21.2.102:6829","nonce":36087}]},"public_addr":"172.21.2.102:6821/36087","cluster_addr":"172.21.2.102:6825/36087","heartbeat_back_addr":"172.21.2.102:6833/36087","heartbeat_front_addr":"172.21.2.102:6829/36087","state":["exists","up"]},{"osd":2,"uuid":"153700af-747f-481f-bdf4-d6072d3a842f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6810","nonce":36089},{"type":"v1","addr":"172.21.2.102:6811","nonce":36089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6812","nonce":36089},{"type":"v1","addr":"172.21.2.102:6813","nonce":36089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6816","nonce":36089},{"type":"v1","addr":"172.21.2.102:6817","nonce":36089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6814","nonce":36089},{"type":"v1","addr":"172.21.2.102:6815","nonce":36089}]},"public_addr":"172.21.2.102:6811/36089","cluster_addr":"172.21.2.102:6813/36089","heartbeat_back_addr":"172.21.2.102:6817/36089","heartbeat_front_addr":"172.21.2.102:6815/36089","state":["exists","up"]},{"osd":3,"uuid":"cbc678f4-83a9-45e6-8830-2f7c232b1ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6802","nonce":36090},{"type":"v1","addr":"172.21.2.102:6803","nonce":36090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6804","nonce":36090},{"type":"v1","addr":"172.21.2.102:6805","nonce":36090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6808","nonce":36090},{"type":"v1","addr":"172.21.2.102:6809","nonce":36090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6806","nonce":36090},{"type":"v1","addr":"172.21.2.102:6807","nonce":36090}]},"public_addr":"172.21.2.102:6803/36090","cluster_addr":"172.21.2.102:6805/36090","heartbeat_back_addr":"172.21.2.102:6809/36090","heartbeat_front_addr":"172.21.2.102:6807/36090","state":["exists","up"]},{"osd":4,"uuid":"c01e4008-b258-4f10-afd2-9e7f37214351","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6808","nonce":36368},{"type":"v1","addr":"172.21.2.113:6811","nonce":36368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6814","nonce":36368},{"type":"v1","addr":"172.21.2.113:6818","nonce":36368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6826","nonce":36368},{"type":"v1","addr":"172.21.2.113:6828","nonce":36368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6820","nonce":36368},{"type":"v1","addr":"172.21.2.113:6824","nonce":36368}]},"public_addr":"172.21.2.113:6811/36368","cluster_addr":"172.21.2.113:6818/36368","heartbeat_back_addr":"172.21.2.113:6828/36368","heartbeat_front_addr":"172.21.2.113:6824/36368","state":["exists","up"]},{"osd":5,"uuid":"551f6007-daf8-4a98-8ee0-c2b16bcccb5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6802","nonce":36366},{"type":"v1","addr":"172.21.2.113:6805","nonce":36366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6807","nonce":36366},{"type":"v1","addr":"172.21.2.113:6810","nonce":36366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6819","nonce":36366},{"type":"v1","addr":"172.21.2.113:6822","nonce":36366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6812","nonce":36366},{"type":"v1","addr":"172.21.2.113:6815","nonce":36366}]},"public_addr":"172.21.2.113:6805/36366","cluster_addr":"172.21.2.113:6810/36366","heartbeat_back_addr":"172.21.2.113:6822/36366","heartbeat_front_addr":"172.21.2.113:6815/36366","state":["exists","up"]},{"osd":6,"uuid":"bf67c040-0e50-48be-b102-c3943bc9ebef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6800","nonce":36367},{"type":"v1","addr":"172.21.2.113:6801","nonce":36367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6803","nonce":36367},{"type":"v1","addr":"172.21.2.113:6804","nonce":36367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6813","nonce":36367},{"type":"v1","addr":"172.21.2.113:6816","nonce":36367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6806","nonce":36367},{"type":"v1","addr":"172.21.2.113:6809","nonce":36367}]},"public_addr":"172.21.2.113:6801/36367","cluster_addr":"172.21.2.113:6804/36367","heartbeat_back_addr":"172.21.2.113:6816/36367","heartbeat_front_addr":"172.21.2.113:6809/36367","state":["exists","up"]},{"osd":7,"uuid":"f5461f80-018d-428f-be89-2eb2bcbe7bb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6817","nonce":36365},{"type":"v1","addr":"172.21.2.113:6821","nonce":36365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6823","nonce":36365},{"type":"v1","addr":"172.21.2.113:6825","nonce":36365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6830","nonce":36365},{"type":"v1","addr":"172.21.2.113:6831","nonce":36365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6827","nonce":36365},{"type":"v1","addr":"172.21.2.113:6829","nonce":36365}]},"public_addr":"172.21.2.113:6821/36365","cluster_addr":"172.21.2.113:6825/36365","heartbeat_back_addr":"172.21.2.113:6831/36365","heartbeat_front_addr":"172.21.2.113:6829/36365","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:31:48.874 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.872 7f6209aa4700 1 -- 172.21.2.102:0/1900381140 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f61e80186c0 msgr2=0x7f61e801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:48.874 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.872 7f6209aa4700 1 --2- 172.21.2.102:0/1900381140 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f61e80186c0 0x7f61e801abc0 crc :-1 s=READY pgs=966 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:48.874 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.872 7f6209aa4700 1 -- 172.21.2.102:0/1900381140 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f62040a8de0 msgr2=0x7f62040a2630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:48.874 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.873 7f6209aa4700 1 --2- 172.21.2.102:0/1900381140 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f62040a8de0 0x7f62040a2630 secure :-1 s=READY pgs=2184 cs=0 l=1 rev1=1 rx=0x7f61f4006f20 tx=0x7f61f400b960).stop 2021-02-10T11:31:48.874 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.873 7f6209aa4700 1 -- 172.21.2.102:0/1900381140 shutdown_connections 2021-02-10T11:31:48.874 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.873 7f6209aa4700 1 --2- 172.21.2.102:0/1900381140 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f61e80186c0 0x7f61e801abc0 unknown :-1 s=CLOSED pgs=966 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:48.874 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.873 7f6209aa4700 1 --2- 172.21.2.102:0/1900381140 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f62040a2c50 0x7f62040a3050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:48.875 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.873 7f6209aa4700 1 --2- 172.21.2.102:0/1900381140 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f62040a8de0 0x7f62040a2630 unknown :-1 s=CLOSED pgs=2184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:48.875 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.873 7f6209aa4700 1 --2- 172.21.2.102:0/1900381140 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f62041a05d0 0x7f62040a87c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:48.875 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.873 7f6209aa4700 1 -- 172.21.2.102:0/1900381140 >> 172.21.2.102:0/1900381140 conn(0x7f62040a9bf0 msgr2=0x7f62040ebeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:48.875 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.873 7f6209aa4700 1 -- 172.21.2.102:0/1900381140 shutdown_connections 2021-02-10T11:31:48.875 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:48.873 7f6209aa4700 1 -- 172.21.2.102:0/1900381140 wait complete. 2021-02-10T11:31:48.883 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:31:49.314 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:49.296 7f63c2a2e700 1 Processor -- start 2021-02-10T11:31:49.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:49.296 7f63c2a2e700 1 -- start start 2021-02-10T11:31:49.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:49.297 7f63c2a2e700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f63bc0d1870 0x7f63bc1b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:49.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:49.297 7f63c2a2e700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f63bc0f0650 0x7f63bc0cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:49.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:49.297 7f63c2a2e700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63bc0f0b10 con 0x7f63bc1b0a10 2021-02-10T11:31:49.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:49.297 7f63c2a2e700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63bc0fe230 con 0x7f63bc0f0650 2021-02-10T11:31:49.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:49.297 7f63c2a2e700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63bc0e5ed0 con 0x7f63bc0d1870 2021-02-10T11:31:49.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:49.297 7f63c104b700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f63bc0f0650 0x7f63bc0cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:49.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:49.297 7f63bbfff700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f63bc1b0a10 0x7f63bc0cf330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:54424/0 (socket says 172.21.2.102:54424) 2021-02-10T11:31:49.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:49.297 7f63c104b700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f63bc0f0650 0x7f63bc0cfb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:33972/0 (socket says 172.21.2.102:33972) 2021-02-10T11:31:49.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:49.297 7f63bbfff700 1 -- 172.21.2.102:0/3795700498 learned_addr learned my addr 172.21.2.102:0/3795700498 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:31:49.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:49.297 7f63c084a700 1 --2- 172.21.2.102:0/3795700498 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f63bc0d1870 0x7f63bc1b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:49.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:49.298 7f63c104b700 1 -- 172.21.2.102:0/3795700498 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f63bc0d1870 msgr2=0x7f63bc1b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:49.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:49.298 7f63c104b700 1 --2- 172.21.2.102:0/3795700498 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f63bc0d1870 0x7f63bc1b03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:49.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:49.298 7f63c104b700 1 -- 172.21.2.102:0/3795700498 >> v1:172.21.2.102:6789/0 conn(0x7f63bc1b0a10 legacy=0x7f63bc0cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:49.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:49.298 7f63c104b700 1 -- 172.21.2.102:0/3795700498 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63bc0f20a0 con 0x7f63bc0f0650 2021-02-10T11:31:49.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:49.298 7f63c104b700 1 --2- 172.21.2.102:0/3795700498 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f63bc0f0650 0x7f63bc0cfb20 secure :-1 s=READY pgs=2186 cs=0 l=1 rev1=1 rx=0x7f63b4006e30 tx=0x7f63b4009de0).ready entity=mon.0 client_cookie=bacd9dd037341c93 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:49.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:49.298 7f63bb7fe700 1 -- 172.21.2.102:0/3795700498 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f63b4000f90 con 0x7f63bc0f0650 2021-02-10T11:31:49.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:49.298 7f63bb7fe700 1 -- 172.21.2.102:0/3795700498 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63b400b300 con 0x7f63bc0f0650 2021-02-10T11:31:49.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:49.298 7f63bb7fe700 1 -- 172.21.2.102:0/3795700498 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f63b400b820 con 0x7f63bc0f0650 2021-02-10T11:31:49.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:49.298 7f63c2a2e700 1 -- 172.21.2.102:0/3795700498 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f63bc0f0650 msgr2=0x7f63bc0cfb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:49.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:49.298 7f63c2a2e700 1 --2- 172.21.2.102:0/3795700498 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f63bc0f0650 0x7f63bc0cfb20 secure :-1 s=READY pgs=2186 cs=0 l=1 rev1=1 rx=0x7f63b4006e30 tx=0x7f63b4009de0).stop 2021-02-10T11:31:49.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:49.299 7f63c2a2e700 1 -- 172.21.2.102:0/3795700498 shutdown_connections 2021-02-10T11:31:49.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:49.299 7f63c2a2e700 1 --2- 172.21.2.102:0/3795700498 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f63bc0f0650 0x7f63bc0cfb20 unknown :-1 s=CLOSED pgs=2186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:49.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:49.299 7f63c2a2e700 1 --2- 172.21.2.102:0/3795700498 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f63bc0d1870 0x7f63bc1b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:49.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:49.299 7f63c2a2e700 1 -- 172.21.2.102:0/3795700498 >> 172.21.2.102:0/3795700498 conn(0x7f63bc0ce450 msgr2=0x7f63bc0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:49.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:49.299 7f63c2a2e700 1 -- 172.21.2.102:0/3795700498 shutdown_connections 2021-02-10T11:31:49.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:49.299 7f63c2a2e700 1 -- 172.21.2.102:0/3795700498 wait complete. 2021-02-10T11:31:49.318 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:49.314 7f63c2a2e700 1 Processor -- start 2021-02-10T11:31:49.319 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:49.314 7f63c2a2e700 1 -- start start 2021-02-10T11:31:49.319 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:49.314 7f63c2a2e700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f63bc1942b0 0x7f63bc0b3550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:49.319 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:49.314 7f63c2a2e700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f63bc195070 0x7f63bc195470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:49.319 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:49.314 7f63c2a2e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f63bc0bb8e0 0x7f63bc0bbce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:49.319 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:49.314 7f63c2a2e700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63bc1946b0 con 0x7f63bc0bb8e0 2021-02-10T11:31:49.319 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:49.314 7f63c2a2e700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63bc0b3b70 con 0x7f63bc1942b0 2021-02-10T11:31:49.319 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:49.314 7f63c2a2e700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63bc0b3d90 con 0x7f63bc195070 2021-02-10T11:31:49.319 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:49.314 7f63c104b700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f63bc0bb8e0 0x7f63bc0bbce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:49.319 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:49.314 7f63c104b700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f63bc0bb8e0 0x7f63bc0bbce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:33976/0 (socket says 172.21.2.102:33976) 2021-02-10T11:31:49.319 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:49.314 7f63c104b700 1 -- 172.21.2.102:0/1460018757 learned_addr learned my addr 172.21.2.102:0/1460018757 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:31:49.319 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:49.314 7f63bbfff700 1 --2- 172.21.2.102:0/1460018757 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f63bc195070 0x7f63bc195470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:49.319 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:49.314 7f63c084a700 1 --2- 172.21.2.102:0/1460018757 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f63bc1942b0 0x7f63bc0b3550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:49.320 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:49.315 7f63c104b700 1 -- 172.21.2.102:0/1460018757 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f63bc195070 msgr2=0x7f63bc195470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:49.320 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:49.315 7f63c104b700 1 --2- 172.21.2.102:0/1460018757 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f63bc195070 0x7f63bc195470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:49.320 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:49.315 7f63c104b700 1 -- 172.21.2.102:0/1460018757 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f63bc1942b0 msgr2=0x7f63bc0b3550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:49.320 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:49.315 7f63c104b700 1 --2- 172.21.2.102:0/1460018757 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f63bc1942b0 0x7f63bc0b3550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:49.320 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:49.315 7f63c104b700 1 -- 172.21.2.102:0/1460018757 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63bc1a49e0 con 0x7f63bc0bb8e0 2021-02-10T11:31:49.320 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:49.315 7f63c104b700 1 --2- 172.21.2.102:0/1460018757 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f63bc0bb8e0 0x7f63bc0bbce0 secure :-1 s=READY pgs=2187 cs=0 l=1 rev1=1 rx=0x7f63b400b930 tx=0x7f63b400bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:49.320 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:49.315 7f63b97fa700 1 -- 172.21.2.102:0/1460018757 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f63b400be10 con 0x7f63bc0bb8e0 2021-02-10T11:31:49.320 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:49.315 7f63b97fa700 1 -- 172.21.2.102:0/1460018757 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63b4005820 con 0x7f63bc0bb8e0 2021-02-10T11:31:49.320 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:49.315 7f63c2a2e700 1 -- 172.21.2.102:0/1460018757 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63bc1a49e0 con 0x7f63bc0bb8e0 2021-02-10T11:31:49.320 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:49.315 7f63b97fa700 1 -- 172.21.2.102:0/1460018757 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f63b4005cc0 con 0x7f63bc0bb8e0 2021-02-10T11:31:49.320 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:49.315 7f63c2a2e700 1 -- 172.21.2.102:0/1460018757 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63bc0bc5f0 con 0x7f63bc0bb8e0 2021-02-10T11:31:49.320 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:49.316 7f63b97fa700 1 -- 172.21.2.102:0/1460018757 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f63b4005a00 con 0x7f63bc0bb8e0 2021-02-10T11:31:49.321 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:49.316 7f63b97fa700 1 --2- 172.21.2.102:0/1460018757 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f63a00186c0 0x7f63a001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:49.321 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:49.316 7f63b97fa700 1 -- 172.21.2.102:0/1460018757 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f63b40327a0 con 0x7f63bc0bb8e0 2021-02-10T11:31:49.321 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:49.316 7f63c084a700 1 --2- 172.21.2.102:0/1460018757 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f63a00186c0 0x7f63a001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:49.321 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:49.316 7f63c084a700 1 --2- 172.21.2.102:0/1460018757 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f63a00186c0 0x7f63a001abc0 crc :-1 s=READY pgs=967 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:49.348 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:49.346 7f63c2a2e700 1 -- 172.21.2.102:0/1460018757 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63bc0bc5f0 con 0x7f63bc0bb8e0 2021-02-10T11:31:49.350 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:49.349 7f63b97fa700 1 -- 172.21.2.102:0/1460018757 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f63b402dfc0 con 0x7f63bc0bb8e0 2021-02-10T11:31:49.474 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:49.473 7f63c2a2e700 1 -- 172.21.2.102:0/1460018757 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f63bc0bc5f0 con 0x7f63bc0bb8e0 2021-02-10T11:31:49.474 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:49.474 7f63b97fa700 1 -- 172.21.2.102:0/1460018757 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v112) v1 ==== 74+0+14007 (secure 0 0 0) 0x7f63b402dfc0 con 0x7f63bc0bb8e0 2021-02-10T11:31:49.475 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:31:49.475 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":112,"fsid":"68db30ea-dbce-4aef-a6df-f01c6fea0513","created":"2021-02-10 11:03:06.331557","modified":"2021-02-10 11:31:49.261779","last_up_change":"2021-02-10 11:03:32.690562","last_in_change":"2021-02-10 11:03:32.690562","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":23,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:03:37.322519","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":22,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:31:46.228075","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"111","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":23,"pool_name":"cephfs_data","create_time":"2021-02-10 11:31:47.252676","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"112","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"be691b4d-a039-408a-af80-9d53f02b6f1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6818","nonce":36088},{"type":"v1","addr":"172.21.2.102:6820","nonce":36088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6822","nonce":36088},{"type":"v1","addr":"172.21.2.102:6824","nonce":36088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6830","nonce":36088},{"type":"v1","addr":"172.21.2.102:6832","nonce":36088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6826","nonce":36088},{"type":"v1","addr":"172.21.2.102:6828","nonce":36088}]},"public_addr":"172.21.2.102:6820/36088","cluster_addr":"172.21.2.102:6824/36088","heartbeat_back_addr":"172.21.2.102:6832/36088","heartbeat_front_addr":"172.21.2.102:6828/36088","state":["exists","up"]},{"osd":1,"uuid":"64e3ccc8-9c69-4d7c-865e-af8e83822635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6819","nonce":36087},{"type":"v1","addr":"172.21.2.102:6821","nonce":36087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6823","nonce":36087},{"type":"v1","addr":"172.21.2.102:6825","nonce":36087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6831","nonce":36087},{"type":"v1","addr":"172.21.2.102:6833","nonce":36087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6827","nonce":36087},{"type":"v1","addr":"172.21.2.102:6829","nonce":36087}]},"public_addr":"172.21.2.102:6821/36087","cluster_addr":"172.21.2.102:6825/36087","heartbeat_back_addr":"172.21.2.102:6833/36087","heartbeat_front_addr":"172.21.2.102:6829/36087","state":["exists","up"]},{"osd":2,"uuid":"153700af-747f-481f-bdf4-d6072d3a842f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6810","nonce":36089},{"type":"v1","addr":"172.21.2.102:6811","nonce":36089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6812","nonce":36089},{"type":"v1","addr":"172.21.2.102:6813","nonce":36089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6816","nonce":36089},{"type":"v1","addr":"172.21.2.102:6817","nonce":36089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6814","nonce":36089},{"type":"v1","addr":"172.21.2.102:6815","nonce":36089}]},"public_addr":"172.21.2.102:6811/36089","cluster_addr":"172.21.2.102:6813/36089","heartbeat_back_addr":"172.21.2.102:6817/36089","heartbeat_front_addr":"172.21.2.102:6815/36089","state":["exists","up"]},{"osd":3,"uuid":"cbc678f4-83a9-45e6-8830-2f7c232b1ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6802","nonce":36090},{"type":"v1","addr":"172.21.2.102:6803","nonce":36090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6804","nonce":36090},{"type":"v1","addr":"172.21.2.102:6805","nonce":36090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6808","nonce":36090},{"type":"v1","addr":"172.21.2.102:6809","nonce":36090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6806","nonce":36090},{"type":"v1","addr":"172.21.2.102:6807","nonce":36090}]},"public_addr":"172.21.2.102:6803/36090","cluster_addr":"172.21.2.102:6805/36090","heartbeat_back_addr":"172.21.2.102:6809/36090","heartbeat_front_addr":"172.21.2.102:6807/36090","state":["exists","up"]},{"osd":4,"uuid":"c01e4008-b258-4f10-afd2-9e7f37214351","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6808","nonce":36368},{"type":"v1","addr":"172.21.2.113:6811","nonce":36368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6814","nonce":36368},{"type":"v1","addr":"172.21.2.113:6818","nonce":36368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6826","nonce":36368},{"type":"v1","addr":"172.21.2.113:6828","nonce":36368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6820","nonce":36368},{"type":"v1","addr":"172.21.2.113:6824","nonce":36368}]},"public_addr":"172.21.2.113:6811/36368","cluster_addr":"172.21.2.113:6818/36368","heartbeat_back_addr":"172.21.2.113:6828/36368","heartbeat_front_addr":"172.21.2.113:6824/36368","state":["exists","up"]},{"osd":5,"uuid":"551f6007-daf8-4a98-8ee0-c2b16bcccb5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6802","nonce":36366},{"type":"v1","addr":"172.21.2.113:6805","nonce":36366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6807","nonce":36366},{"type":"v1","addr":"172.21.2.113:6810","nonce":36366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6819","nonce":36366},{"type":"v1","addr":"172.21.2.113:6822","nonce":36366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6812","nonce":36366},{"type":"v1","addr":"172.21.2.113:6815","nonce":36366}]},"public_addr":"172.21.2.113:6805/36366","cluster_addr":"172.21.2.113:6810/36366","heartbeat_back_addr":"172.21.2.113:6822/36366","heartbeat_front_addr":"172.21.2.113:6815/36366","state":["exists","up"]},{"osd":6,"uuid":"bf67c040-0e50-48be-b102-c3943bc9ebef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6800","nonce":36367},{"type":"v1","addr":"172.21.2.113:6801","nonce":36367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6803","nonce":36367},{"type":"v1","addr":"172.21.2.113:6804","nonce":36367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6813","nonce":36367},{"type":"v1","addr":"172.21.2.113:6816","nonce":36367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6806","nonce":36367},{"type":"v1","addr":"172.21.2.113:6809","nonce":36367}]},"public_addr":"172.21.2.113:6801/36367","cluster_addr":"172.21.2.113:6804/36367","heartbeat_back_addr":"172.21.2.113:6816/36367","heartbeat_front_addr":"172.21.2.113:6809/36367","state":["exists","up"]},{"osd":7,"uuid":"f5461f80-018d-428f-be89-2eb2bcbe7bb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6817","nonce":36365},{"type":"v1","addr":"172.21.2.113:6821","nonce":36365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6823","nonce":36365},{"type":"v1","addr":"172.21.2.113:6825","nonce":36365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6830","nonce":36365},{"type":"v1","addr":"172.21.2.113:6831","nonce":36365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6827","nonce":36365},{"type":"v1","addr":"172.21.2.113:6829","nonce":36365}]},"public_addr":"172.21.2.113:6821/36365","cluster_addr":"172.21.2.113:6825/36365","heartbeat_back_addr":"172.21.2.113:6831/36365","heartbeat_front_addr":"172.21.2.113:6829/36365","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:31:49.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:49.475 7f63c2a2e700 1 -- 172.21.2.102:0/1460018757 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f63a00186c0 msgr2=0x7f63a001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:49.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:49.475 7f63c2a2e700 1 --2- 172.21.2.102:0/1460018757 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f63a00186c0 0x7f63a001abc0 crc :-1 s=READY pgs=967 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:49.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:49.475 7f63c2a2e700 1 -- 172.21.2.102:0/1460018757 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f63bc0bb8e0 msgr2=0x7f63bc0bbce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:49.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:49.475 7f63c2a2e700 1 --2- 172.21.2.102:0/1460018757 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f63bc0bb8e0 0x7f63bc0bbce0 secure :-1 s=READY pgs=2187 cs=0 l=1 rev1=1 rx=0x7f63b400b930 tx=0x7f63b400bcd0).stop 2021-02-10T11:31:49.477 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:49.475 7f63c2a2e700 1 -- 172.21.2.102:0/1460018757 shutdown_connections 2021-02-10T11:31:49.477 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:49.475 7f63c2a2e700 1 --2- 172.21.2.102:0/1460018757 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f63bc0bb8e0 0x7f63bc0bbce0 unknown :-1 s=CLOSED pgs=2187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:49.477 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:49.475 7f63c2a2e700 1 --2- 172.21.2.102:0/1460018757 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f63bc195070 0x7f63bc195470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:49.477 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:49.475 7f63c2a2e700 1 --2- 172.21.2.102:0/1460018757 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f63a00186c0 0x7f63a001abc0 unknown :-1 s=CLOSED pgs=967 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:49.477 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:49.475 7f63c2a2e700 1 --2- 172.21.2.102:0/1460018757 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f63bc1942b0 0x7f63bc0b3550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:49.477 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:49.475 7f63c2a2e700 1 -- 172.21.2.102:0/1460018757 >> 172.21.2.102:0/1460018757 conn(0x7f63bc0b3040 msgr2=0x7f63bc0c3250 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:49.477 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:49.475 7f63c2a2e700 1 -- 172.21.2.102:0/1460018757 shutdown_connections 2021-02-10T11:31:49.477 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:49.476 7f63c2a2e700 1 -- 172.21.2.102:0/1460018757 wait complete. 2021-02-10T11:31:49.486 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2021-02-10T11:31:49.918 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:49.899 7f843977f700 1 Processor -- start 2021-02-10T11:31:49.918 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:49.899 7f843977f700 1 -- start start 2021-02-10T11:31:49.918 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:49.899 7f843977f700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f84340d1c70 0x7f84340cef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:49.919 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:49.899 7f843977f700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f84340cf580 0x7f84340cf980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:49.919 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:49.899 7f843977f700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84340fe1c0 con 0x7f84340d1870 2021-02-10T11:31:49.919 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:49.899 7f843977f700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84341b0cb0 con 0x7f84340cf580 2021-02-10T11:31:49.919 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:49.899 7f843977f700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8434109a60 con 0x7f84340d1c70 2021-02-10T11:31:49.919 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:49.900 7f8433574700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f84340d1870 0x7f84341b04c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:54438/0 (socket says 172.21.2.102:54438) 2021-02-10T11:31:49.919 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:49.900 7f8433574700 1 -- 172.21.2.102:0/3720799142 learned_addr learned my addr 172.21.2.102:0/3720799142 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:31:49.919 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:49.900 7f8432572700 1 -- 172.21.2.102:0/3720799142 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4056039183 0 0) 0x7f841c0010b0 con 0x7f84340d1870 2021-02-10T11:31:49.919 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:49.900 7f8432572700 1 -- 172.21.2.102:0/3720799142 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8420002fe0 con 0x7f84340d1870 2021-02-10T11:31:49.919 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:49.900 7f8432572700 1 -- 172.21.2.102:0/3720799142 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1981453608 0 0) 0x7f841c0010b0 con 0x7f84340d1870 2021-02-10T11:31:49.919 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:49.900 7f8433d75700 1 --2- 172.21.2.102:0/3720799142 >> v2:172.21.2.113:3300/0 conn(0x7f84340cf580 0x7f84340cf980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:49.919 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:49.900 7f8432572700 1 -- 172.21.2.102:0/3720799142 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f84340d1c70 msgr2=0x7f84340cef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:49.919 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:49.900 7f8432572700 1 --2- 172.21.2.102:0/3720799142 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f84340d1c70 0x7f84340cef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:49.919 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:49.900 7f8432572700 1 -- 172.21.2.102:0/3720799142 >> v2:172.21.2.113:3300/0 conn(0x7f84340cf580 msgr2=0x7f84340cf980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:49.920 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:49.900 7f8432572700 1 --2- 172.21.2.102:0/3720799142 >> v2:172.21.2.113:3300/0 conn(0x7f84340cf580 0x7f84340cf980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:49.920 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:49.900 7f8432572700 1 -- 172.21.2.102:0/3720799142 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8434088070 con 0x7f84340d1870 2021-02-10T11:31:49.920 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:49.900 7f8432572700 1 -- 172.21.2.102:0/3720799142 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7f841c0015b0 con 0x7f84340d1870 2021-02-10T11:31:49.920 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:49.900 7f8433d75700 1 --2- 172.21.2.102:0/3720799142 >> v2:172.21.2.113:3300/0 conn(0x7f84340cf580 0x7f84340cf980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:31:49.920 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:49.900 7f8432572700 1 -- 172.21.2.102:0/3720799142 >> v1:172.21.2.102:6789/0 conn(0x7f84340d1870 legacy=0x7f84341b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:49.920 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:49.900 7f8432572700 1 --2- 172.21.2.102:0/3720799142 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8420003a60 0x7f8420005ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:49.920 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:49.900 7f8432572700 1 --2- 172.21.2.102:0/3720799142 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8420006790 0x7f8420008c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:49.920 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:49.900 7f8432572700 1 --2- 172.21.2.102:0/3720799142 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8420009590 0x7f842000ba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:49.920 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:49.900 7f8432572700 1 -- 172.21.2.102:0/3720799142 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f842000c2d0 con 0x7f8420006790 2021-02-10T11:31:49.920 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:49.900 7f8432572700 1 -- 172.21.2.102:0/3720799142 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f842000c4f0 con 0x7f8420003a60 2021-02-10T11:31:49.920 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:49.900 7f8432572700 1 -- 172.21.2.102:0/3720799142 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f842000c710 con 0x7f8420009590 2021-02-10T11:31:49.920 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:49.901 7f8433574700 1 --2- 172.21.2.102:0/3720799142 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8420003a60 0x7f8420005ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:49.920 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:49.901 7f8432d73700 1 --2- 172.21.2.102:0/3720799142 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8420006790 0x7f8420008c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:49.920 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:49.901 7f8433d75700 1 --2- 172.21.2.102:0/3720799142 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8420009590 0x7f842000ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:49.921 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:49.901 7f8432d73700 1 -- 172.21.2.102:0/3720799142 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8420009590 msgr2=0x7f842000ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:49.921 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:49.901 7f8432d73700 1 --2- 172.21.2.102:0/3720799142 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8420009590 0x7f842000ba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:49.921 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:49.901 7f8432d73700 1 -- 172.21.2.102:0/3720799142 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8420003a60 msgr2=0x7f8420005ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:49.921 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:49.901 7f8432d73700 1 --2- 172.21.2.102:0/3720799142 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8420003a60 0x7f8420005ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:49.921 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:49.901 7f8432d73700 1 -- 172.21.2.102:0/3720799142 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84200509c0 con 0x7f8420006790 2021-02-10T11:31:49.921 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:49.901 7f8432d73700 1 --2- 172.21.2.102:0/3720799142 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8420006790 0x7f8420008c90 secure :-1 s=READY pgs=2189 cs=0 l=1 rev1=1 rx=0x7f8424006ab0 tx=0x7f8424009b40).ready entity=mon.0 client_cookie=52ad5fb11e169791 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:49.921 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:49.901 7f8432572700 1 -- 172.21.2.102:0/3720799142 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8424006680 con 0x7f8420006790 2021-02-10T11:31:49.921 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:49.901 7f8432572700 1 -- 172.21.2.102:0/3720799142 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84240063f0 con 0x7f8420006790 2021-02-10T11:31:49.921 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:49.901 7f8432572700 1 -- 172.21.2.102:0/3720799142 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f842400c2b0 con 0x7f8420006790 2021-02-10T11:31:49.921 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:49.901 7f843977f700 1 -- 172.21.2.102:0/3720799142 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8420006790 msgr2=0x7f8420008c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:49.921 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:49.901 7f843977f700 1 --2- 172.21.2.102:0/3720799142 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8420006790 0x7f8420008c90 secure :-1 s=READY pgs=2189 cs=0 l=1 rev1=1 rx=0x7f8424006ab0 tx=0x7f8424009b40).stop 2021-02-10T11:31:49.921 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:49.902 7f8433d75700 1 -- 172.21.2.102:0/3720799142 reap_dead start 2021-02-10T11:31:49.921 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:49.902 7f843977f700 1 -- 172.21.2.102:0/3720799142 shutdown_connections 2021-02-10T11:31:49.922 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:49.902 7f843977f700 1 -- 172.21.2.102:0/3720799142 >> 172.21.2.102:0/3720799142 conn(0x7f84340ce420 msgr2=0x7f84340d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:49.922 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:49.902 7f843977f700 1 -- 172.21.2.102:0/3720799142 shutdown_connections 2021-02-10T11:31:49.922 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:49.902 7f843977f700 1 -- 172.21.2.102:0/3720799142 wait complete. 2021-02-10T11:31:49.922 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:49.917 7f843977f700 1 Processor -- start 2021-02-10T11:31:49.922 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:49.917 7f843977f700 1 -- start start 2021-02-10T11:31:49.923 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:49.918 7f843977f700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8434085bc0 0x7f84340863e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:49.923 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:49.918 7f843977f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f84340954c0 0x7f8434096300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:49.923 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:49.918 7f843977f700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f84340958c0 0x7f8434096920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:49.923 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:49.918 7f843977f700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8434095d80 con 0x7f84340954c0 2021-02-10T11:31:49.923 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:49.918 7f843977f700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84340bfa60 con 0x7f8434085bc0 2021-02-10T11:31:49.924 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:49.918 7f843977f700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8434085fc0 con 0x7f84340958c0 2021-02-10T11:31:49.924 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:49.918 7f8432d73700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f84340954c0 0x7f8434096300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:49.924 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:49.918 7f8432d73700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f84340954c0 0x7f8434096300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:33994/0 (socket says 172.21.2.102:33994) 2021-02-10T11:31:49.924 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:49.918 7f8432d73700 1 -- 172.21.2.102:0/4252621620 learned_addr learned my addr 172.21.2.102:0/4252621620 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:31:49.924 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:49.918 7f8433d75700 1 --2- 172.21.2.102:0/4252621620 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f84340958c0 0x7f8434096920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:49.924 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:49.918 7f8433574700 1 --2- 172.21.2.102:0/4252621620 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8434085bc0 0x7f84340863e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:49.924 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:49.918 7f8432d73700 1 -- 172.21.2.102:0/4252621620 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f84340958c0 msgr2=0x7f8434096920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:49.924 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:49.918 7f8432d73700 1 --2- 172.21.2.102:0/4252621620 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f84340958c0 0x7f8434096920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:49.924 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:49.918 7f8432d73700 1 -- 172.21.2.102:0/4252621620 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8434085bc0 msgr2=0x7f84340863e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:49.924 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:49.918 7f8432d73700 1 --2- 172.21.2.102:0/4252621620 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8434085bc0 0x7f84340863e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:49.924 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:49.918 7f8432d73700 1 -- 172.21.2.102:0/4252621620 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8434086c10 con 0x7f84340954c0 2021-02-10T11:31:49.924 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:49.918 7f8432d73700 1 --2- 172.21.2.102:0/4252621620 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f84340954c0 0x7f8434096300 secure :-1 s=READY pgs=2190 cs=0 l=1 rev1=1 rx=0x7f842400b610 tx=0x7f842400b9b0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:49.924 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:49.919 7f842bfff700 1 -- 172.21.2.102:0/4252621620 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8424000c90 con 0x7f84340954c0 2021-02-10T11:31:49.925 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:49.919 7f842bfff700 1 -- 172.21.2.102:0/4252621620 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8424002030 con 0x7f84340954c0 2021-02-10T11:31:49.925 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:49.919 7f843977f700 1 -- 172.21.2.102:0/4252621620 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8434086c10 con 0x7f84340954c0 2021-02-10T11:31:49.925 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:49.919 7f843977f700 1 -- 172.21.2.102:0/4252621620 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8434097230 con 0x7f84340954c0 2021-02-10T11:31:49.925 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:49.919 7f842bfff700 1 -- 172.21.2.102:0/4252621620 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8424002360 con 0x7f84340954c0 2021-02-10T11:31:49.925 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:49.919 7f842bfff700 1 -- 172.21.2.102:0/4252621620 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f8424002210 con 0x7f84340954c0 2021-02-10T11:31:49.925 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:49.919 7f842bfff700 1 --2- 172.21.2.102:0/4252621620 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f84180186c0 0x7f841801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:49.925 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:49.919 7f842bfff700 1 -- 172.21.2.102:0/4252621620 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f84240325a0 con 0x7f84340954c0 2021-02-10T11:31:49.925 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:49.919 7f8433574700 1 --2- 172.21.2.102:0/4252621620 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f84180186c0 0x7f841801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:49.925 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:49.920 7f8433574700 1 --2- 172.21.2.102:0/4252621620 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f84180186c0 0x7f841801abc0 crc :-1 s=READY pgs=968 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:49.951 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:49.949 7f843977f700 1 -- 172.21.2.102:0/4252621620 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84340fe1c0 con 0x7f84340954c0 2021-02-10T11:31:49.953 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:49.952 7f842bfff700 1 -- 172.21.2.102:0/4252621620 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f842402aca0 con 0x7f84340954c0 2021-02-10T11:31:50.090 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:50.089 7f843977f700 1 -- 172.21.2.102:0/4252621620 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"} v 0) v1 -- 0x7f84340fe1c0 con 0x7f84340954c0 2021-02-10T11:31:50.271 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:50.270 7f842bfff700 1 -- 172.21.2.102:0/4252621620 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"}]=0 v91) v1 ==== 118+0+0 (secure 0 0 0) 0x7f842402aca0 con 0x7f84340954c0 2021-02-10T11:31:50.308 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:50.306 7f843977f700 1 -- 172.21.2.102:0/4252621620 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f84180186c0 msgr2=0x7f841801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:50.308 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:50.306 7f843977f700 1 --2- 172.21.2.102:0/4252621620 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f84180186c0 0x7f841801abc0 crc :-1 s=READY pgs=968 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:50.308 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:50.306 7f843977f700 1 -- 172.21.2.102:0/4252621620 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f84340954c0 msgr2=0x7f8434096300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:50.308 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:50.306 7f843977f700 1 --2- 172.21.2.102:0/4252621620 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f84340954c0 0x7f8434096300 secure :-1 s=READY pgs=2190 cs=0 l=1 rev1=1 rx=0x7f842400b610 tx=0x7f842400b9b0).stop 2021-02-10T11:31:50.308 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:50.306 7f843977f700 1 -- 172.21.2.102:0/4252621620 shutdown_connections 2021-02-10T11:31:50.308 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:50.306 7f843977f700 1 --2- 172.21.2.102:0/4252621620 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f84340958c0 0x7f8434096920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:50.308 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:50.306 7f843977f700 1 --2- 172.21.2.102:0/4252621620 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f84340954c0 0x7f8434096300 unknown :-1 s=CLOSED pgs=2190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:50.309 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:50.306 7f843977f700 1 --2- 172.21.2.102:0/4252621620 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f84180186c0 0x7f841801abc0 unknown :-1 s=CLOSED pgs=968 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:50.309 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:50.306 7f843977f700 1 --2- 172.21.2.102:0/4252621620 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8434085bc0 0x7f84340863e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:50.309 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:50.306 7f843977f700 1 -- 172.21.2.102:0/4252621620 >> 172.21.2.102:0/4252621620 conn(0x7f84340bef00 msgr2=0x7f843409d1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:50.309 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:50.306 7f843977f700 1 -- 172.21.2.102:0/4252621620 shutdown_connections 2021-02-10T11:31:50.309 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:50.306 7f843977f700 1 -- 172.21.2.102:0/4252621620 wait complete. 2021-02-10T11:31:50.315 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:31:50.747 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:50.727 7fb7250f4700 1 Processor -- start 2021-02-10T11:31:50.747 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:50.727 7fb7250f4700 1 -- start start 2021-02-10T11:31:50.747 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:50.728 7fb7250f4700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fb7200d0340 0x7fb7200ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:50.747 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:50.728 7fb7250f4700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb7200d1c70 0x7fb7200efc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:50.747 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:50.728 7fb7250f4700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7200f0b10 con 0x7fb7200d1870 2021-02-10T11:31:50.747 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:50.728 7fb7250f4700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7200e5ed0 con 0x7fb7200d0340 2021-02-10T11:31:50.747 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:50.728 7fb7250f4700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7201b0be0 con 0x7fb7200d1c70 2021-02-10T11:31:50.748 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:50.729 7fb71f59e700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb7200d1c70 0x7fb7200efc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:50.748 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:50.729 7fb71e59c700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fb7200d0340 0x7fb7200ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:50.748 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:50.729 7fb71ed9d700 1 --1- >> v1:172.21.2.113:6789/0 conn(0x7fb7200d1870 0x7fb7201b03f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.113:6789/0 says I am v1:172.21.2.102:33548/0 (socket says 172.21.2.102:33548) 2021-02-10T11:31:50.748 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:50.729 7fb71ed9d700 1 -- 172.21.2.102:0/3817839862 learned_addr learned my addr 172.21.2.102:0/3817839862 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:31:50.748 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:50.730 7fb71e59c700 1 -- 172.21.2.102:0/3817839862 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb7200d1c70 msgr2=0x7fb7200efc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:50.748 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:50.730 7fb71e59c700 1 --2- 172.21.2.102:0/3817839862 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb7200d1c70 0x7fb7200efc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:50.748 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:50.730 7fb71e59c700 1 -- 172.21.2.102:0/3817839862 >> v1:172.21.2.113:6789/0 conn(0x7fb7200d1870 legacy=0x7fb7201b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:50.748 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:50.730 7fb71e59c700 1 -- 172.21.2.102:0/3817839862 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7200f20a0 con 0x7fb7200d0340 2021-02-10T11:31:50.748 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:50.730 7fb71f59e700 1 --2- 172.21.2.102:0/3817839862 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb7200d1c70 0x7fb7200efc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:31:50.748 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:50.730 7fb71e59c700 1 --2- 172.21.2.102:0/3817839862 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb7200d0340 0x7fb7200ef640 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 rx=0x7fb710006650 tx=0x7fb710009a80).ready entity=mon.1 client_cookie=f38743241c78a057 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:50.748 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:50.731 7fb71dd9b700 1 -- 172.21.2.102:0/3817839862 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb71000c0c0 con 0x7fb7200d0340 2021-02-10T11:31:50.748 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:50.731 7fb71dd9b700 1 -- 172.21.2.102:0/3817839862 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb71000c340 con 0x7fb7200d0340 2021-02-10T11:31:50.748 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:50.731 7fb71dd9b700 1 -- 172.21.2.102:0/3817839862 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb71000c860 con 0x7fb7200d0340 2021-02-10T11:31:50.749 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:50.731 7fb7250f4700 1 -- 172.21.2.102:0/3817839862 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb7200d0340 msgr2=0x7fb7200ef640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:50.749 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:50.731 7fb7250f4700 1 --2- 172.21.2.102:0/3817839862 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb7200d0340 0x7fb7200ef640 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 rx=0x7fb710006650 tx=0x7fb710009a80).stop 2021-02-10T11:31:50.749 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:50.731 7fb7250f4700 1 -- 172.21.2.102:0/3817839862 shutdown_connections 2021-02-10T11:31:50.749 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:50.731 7fb7250f4700 1 --2- 172.21.2.102:0/3817839862 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb7200d1c70 0x7fb7200efc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:50.749 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:50.731 7fb7250f4700 1 --2- 172.21.2.102:0/3817839862 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb7200d0340 0x7fb7200ef640 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:50.749 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:50.731 7fb7250f4700 1 -- 172.21.2.102:0/3817839862 >> 172.21.2.102:0/3817839862 conn(0x7fb7200ce450 msgr2=0x7fb7200d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:50.749 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:50.732 7fb7250f4700 1 -- 172.21.2.102:0/3817839862 shutdown_connections 2021-02-10T11:31:50.749 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:50.732 7fb7250f4700 1 -- 172.21.2.102:0/3817839862 wait complete. 2021-02-10T11:31:50.751 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:50.746 7fb7250f4700 1 Processor -- start 2021-02-10T11:31:50.751 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:50.746 7fb7250f4700 1 -- start start 2021-02-10T11:31:50.751 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:50.747 7fb7250f4700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb72018d870 0x7fb7200b19b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:50.751 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:50.747 7fb7250f4700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb7200b1fd0 0x7fb720099ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:50.751 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:50.747 7fb7250f4700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb7200b23d0 0x7fb72009a510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:50.751 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:50.747 7fb7250f4700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb72018dc70 con 0x7fb72018d870 2021-02-10T11:31:50.751 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:50.747 7fb7250f4700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb720199e60 con 0x7fb7200b23d0 2021-02-10T11:31:50.751 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:50.747 7fb7250f4700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb72009ab30 con 0x7fb7200b1fd0 2021-02-10T11:31:50.751 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:50.747 7fb71ed9d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb72018d870 0x7fb7200b19b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:50.751 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:50.747 7fb71ed9d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb72018d870 0x7fb7200b19b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:34006/0 (socket says 172.21.2.102:34006) 2021-02-10T11:31:50.751 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:50.747 7fb71ed9d700 1 -- 172.21.2.102:0/1988282190 learned_addr learned my addr 172.21.2.102:0/1988282190 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:31:50.752 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:50.747 7fb71ed9d700 1 -- 172.21.2.102:0/1988282190 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb7200b1fd0 msgr2=0x7fb720099ef0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:31:50.752 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:50.747 7fb71ed9d700 1 --2- 172.21.2.102:0/1988282190 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb7200b1fd0 0x7fb720099ef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:50.752 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:50.747 7fb71ed9d700 1 -- 172.21.2.102:0/1988282190 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb7200b23d0 msgr2=0x7fb72009a510 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:31:50.752 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:50.747 7fb71ed9d700 1 --2- 172.21.2.102:0/1988282190 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb7200b23d0 0x7fb72009a510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:50.752 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:50.747 7fb71ed9d700 1 -- 172.21.2.102:0/1988282190 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb72009ad50 con 0x7fb72018d870 2021-02-10T11:31:50.752 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:50.747 7fb71ed9d700 1 --2- 172.21.2.102:0/1988282190 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb72018d870 0x7fb7200b19b0 secure :-1 s=READY pgs=2191 cs=0 l=1 rev1=1 rx=0x7fb70800a580 tx=0x7fb70800a920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:50.752 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:50.747 7fb7177fe700 1 -- 172.21.2.102:0/1988282190 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb70800ad10 con 0x7fb72018d870 2021-02-10T11:31:50.752 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:50.748 7fb7177fe700 1 -- 172.21.2.102:0/1988282190 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb708000f00 con 0x7fb72018d870 2021-02-10T11:31:50.752 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:50.748 7fb7250f4700 1 -- 172.21.2.102:0/1988282190 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb72009ad50 con 0x7fb72018d870 2021-02-10T11:31:50.752 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:50.748 7fb7250f4700 1 -- 172.21.2.102:0/1988282190 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb72018e3f0 con 0x7fb72018d870 2021-02-10T11:31:50.752 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:50.748 7fb7177fe700 1 -- 172.21.2.102:0/1988282190 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb7080045a0 con 0x7fb72018d870 2021-02-10T11:31:50.752 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:50.748 7fb7177fe700 1 -- 172.21.2.102:0/1988282190 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fb708008000 con 0x7fb72018d870 2021-02-10T11:31:50.752 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:50.748 7fb7177fe700 1 --2- 172.21.2.102:0/1988282190 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb704018730 0x7fb70401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:50.753 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:50.748 7fb7177fe700 1 -- 172.21.2.102:0/1988282190 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fb7080327b0 con 0x7fb72018d870 2021-02-10T11:31:50.753 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:50.748 7fb71e59c700 1 --2- 172.21.2.102:0/1988282190 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb704018730 0x7fb70401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:50.753 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:50.749 7fb71e59c700 1 --2- 172.21.2.102:0/1988282190 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb704018730 0x7fb70401ac10 crc :-1 s=READY pgs=969 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:50.779 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:50.778 7fb7250f4700 1 -- 172.21.2.102:0/1988282190 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7200999c0 con 0x7fb72018d870 2021-02-10T11:31:50.781 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:50.780 7fb7177fe700 1 -- 172.21.2.102:0/1988282190 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fb708000c60 con 0x7fb72018d870 2021-02-10T11:31:50.908 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:50.906 7fb7250f4700 1 -- 172.21.2.102:0/1988282190 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb7200999c0 con 0x7fb72018d870 2021-02-10T11:31:50.908 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:50.907 7fb7177fe700 1 -- 172.21.2.102:0/1988282190 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 91 v91) v1 ==== 94+0+2104 (secure 0 0 0) 0x7fb708000e50 con 0x7fb72018d870 2021-02-10T11:31:50.909 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:31:50.909 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":91,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10404,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.102:6835/1661316191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6834","nonce":1661316191},{"type":"v1","addr":"172.21.2.102:6835","nonce":1661316191}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:31:48.259584","modified":"2021-02-10 11:31:50.267046","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10401},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10401":{"gid":10401,"name":"b","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.2.113:6833/313019011","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6832","nonce":313019011},{"type":"v1","addr":"172.21.2.113:6833","nonce":313019011}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[23],"metadata_pool":22,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":11}]} 2021-02-10T11:31:50.909 INFO:teuthology.orchestra.run.gibba002.stderr:dumped fsmap epoch 91 2021-02-10T11:31:50.910 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:50.909 7fb7250f4700 1 -- 172.21.2.102:0/1988282190 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb704018730 msgr2=0x7fb70401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:50.910 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:50.909 7fb7250f4700 1 --2- 172.21.2.102:0/1988282190 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb704018730 0x7fb70401ac10 crc :-1 s=READY pgs=969 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:50.911 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:50.909 7fb7250f4700 1 -- 172.21.2.102:0/1988282190 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb72018d870 msgr2=0x7fb7200b19b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:50.911 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:50.909 7fb7250f4700 1 --2- 172.21.2.102:0/1988282190 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb72018d870 0x7fb7200b19b0 secure :-1 s=READY pgs=2191 cs=0 l=1 rev1=1 rx=0x7fb70800a580 tx=0x7fb70800a920).stop 2021-02-10T11:31:50.911 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:50.909 7fb7250f4700 1 -- 172.21.2.102:0/1988282190 shutdown_connections 2021-02-10T11:31:50.911 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:50.909 7fb7250f4700 1 --2- 172.21.2.102:0/1988282190 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb704018730 0x7fb70401ac10 unknown :-1 s=CLOSED pgs=969 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:50.911 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:50.909 7fb7250f4700 1 --2- 172.21.2.102:0/1988282190 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb7200b23d0 0x7fb72009a510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:50.911 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:50.909 7fb7250f4700 1 --2- 172.21.2.102:0/1988282190 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb7200b1fd0 0x7fb720099ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:50.911 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:50.909 7fb7250f4700 1 --2- 172.21.2.102:0/1988282190 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb72018d870 0x7fb7200b19b0 unknown :-1 s=CLOSED pgs=2191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:50.911 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:50.909 7fb7250f4700 1 -- 172.21.2.102:0/1988282190 >> 172.21.2.102:0/1988282190 conn(0x7fb7200f0550 msgr2=0x7fb720191730 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:50.911 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:50.909 7fb7250f4700 1 -- 172.21.2.102:0/1988282190 shutdown_connections 2021-02-10T11:31:50.911 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:50.909 7fb7250f4700 1 -- 172.21.2.102:0/1988282190 wait complete. 2021-02-10T11:31:50.919 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:31:51.349 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:51.331 7f1c0b8ea700 1 Processor -- start 2021-02-10T11:31:51.350 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:51.331 7f1c0b8ea700 1 -- start start 2021-02-10T11:31:51.350 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:51.332 7f1c0b8ea700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f1c040d1870 0x7f1c041b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:51.350 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:51.332 7f1c0b8ea700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1c040f0650 0x7f1c040cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:51.350 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:51.332 7f1c0b8ea700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c040f0b10 con 0x7f1c041b0a10 2021-02-10T11:31:51.350 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:51.332 7f1c0b8ea700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c040e5ed0 con 0x7f1c040d1870 2021-02-10T11:31:51.350 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:51.332 7f1c0b8ea700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c040d1c70 con 0x7f1c040f0650 2021-02-10T11:31:51.350 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:51.332 7f1c09706700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f1c040d1870 0x7f1c041b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:51.350 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:51.332 7f1c09706700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f1c040d1870 0x7f1c041b03f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:34014/0 (socket says 172.21.2.102:34014) 2021-02-10T11:31:51.350 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:51.332 7f1c09706700 1 -- 172.21.2.102:0/3930539963 learned_addr learned my addr 172.21.2.102:0/3930539963 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:31:51.350 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:51.332 7f1c09f07700 1 --2- 172.21.2.102:0/3930539963 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1c040f0650 0x7f1c040cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:51.351 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:51.332 7f1c09706700 1 -- 172.21.2.102:0/3930539963 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1c040f0650 msgr2=0x7f1c040cfb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:51.351 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:51.332 7f1c09706700 1 --2- 172.21.2.102:0/3930539963 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1c040f0650 0x7f1c040cfb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:51.351 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:51.332 7f1c09706700 1 -- 172.21.2.102:0/3930539963 >> v1:172.21.2.113:6789/0 conn(0x7f1c041b0a10 legacy=0x7f1c040cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:51.351 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:51.333 7f1c09706700 1 -- 172.21.2.102:0/3930539963 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c040d1e90 con 0x7f1c040d1870 2021-02-10T11:31:51.351 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:51.333 7f1c09706700 1 --2- 172.21.2.102:0/3930539963 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1c040d1870 0x7f1c041b03f0 secure :-1 s=READY pgs=2192 cs=0 l=1 rev1=1 rx=0x7f1c00006700 tx=0x7f1c00009b80).ready entity=mon.0 client_cookie=6539fba56e8a1e6b server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:51.351 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:51.333 7f1bfbfff700 1 -- 172.21.2.102:0/3930539963 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1c0000c0c0 con 0x7f1c040d1870 2021-02-10T11:31:51.351 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:51.333 7f1bfbfff700 1 -- 172.21.2.102:0/3930539963 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c0000c340 con 0x7f1c040d1870 2021-02-10T11:31:51.351 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:51.333 7f1bfbfff700 1 -- 172.21.2.102:0/3930539963 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1c0000c860 con 0x7f1c040d1870 2021-02-10T11:31:51.351 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:51.333 7f1c0b8ea700 1 -- 172.21.2.102:0/3930539963 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1c040d1870 msgr2=0x7f1c041b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:51.351 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:51.333 7f1c0b8ea700 1 --2- 172.21.2.102:0/3930539963 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1c040d1870 0x7f1c041b03f0 secure :-1 s=READY pgs=2192 cs=0 l=1 rev1=1 rx=0x7f1c00006700 tx=0x7f1c00009b80).stop 2021-02-10T11:31:51.351 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:51.334 7f1c0b8ea700 1 -- 172.21.2.102:0/3930539963 shutdown_connections 2021-02-10T11:31:51.351 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:51.334 7f1c0b8ea700 1 --2- 172.21.2.102:0/3930539963 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1c040f0650 0x7f1c040cfb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:51.351 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:51.334 7f1c0b8ea700 1 --2- 172.21.2.102:0/3930539963 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1c040d1870 0x7f1c041b03f0 unknown :-1 s=CLOSED pgs=2192 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:51.352 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:51.334 7f1c0b8ea700 1 -- 172.21.2.102:0/3930539963 >> 172.21.2.102:0/3930539963 conn(0x7f1c040ce450 msgr2=0x7f1c040d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:51.352 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:51.334 7f1c0b8ea700 1 -- 172.21.2.102:0/3930539963 shutdown_connections 2021-02-10T11:31:51.352 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:51.334 7f1c0b8ea700 1 -- 172.21.2.102:0/3930539963 wait complete. 2021-02-10T11:31:51.353 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:51.349 7f1c0b8ea700 1 Processor -- start 2021-02-10T11:31:51.353 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:51.349 7f1c0b8ea700 1 -- start start 2021-02-10T11:31:51.353 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:51.349 7f1c0b8ea700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1c041a4a60 0x7f1c040c1c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:51.353 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:51.349 7f1c0b8ea700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1c040c2290 0x7f1c040c2690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:51.353 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:51.349 7f1c0b8ea700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1c040c2cb0 0x7f1c040bfd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:51.353 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:51.349 7f1c0b8ea700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c040f05b0 con 0x7f1c040c2cb0 2021-02-10T11:31:51.353 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:51.349 7f1c0b8ea700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c040f07d0 con 0x7f1c040c2290 2021-02-10T11:31:51.354 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:51.349 7f1c0b8ea700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c040c0340 con 0x7f1c041a4a60 2021-02-10T11:31:51.354 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:51.349 7f1c09f07700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1c040c2cb0 0x7f1c040bfd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:51.354 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:51.349 7f1c09f07700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1c040c2cb0 0x7f1c040bfd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:34024/0 (socket says 172.21.2.102:34024) 2021-02-10T11:31:51.354 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:51.349 7f1c09f07700 1 -- 172.21.2.102:0/4029222670 learned_addr learned my addr 172.21.2.102:0/4029222670 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:31:51.354 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:51.350 7f1c09f07700 1 -- 172.21.2.102:0/4029222670 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1c041a4a60 msgr2=0x7f1c040c1c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:51.354 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:51.350 7f1c09f07700 1 --2- 172.21.2.102:0/4029222670 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1c041a4a60 0x7f1c040c1c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:51.354 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:51.350 7f1c09f07700 1 -- 172.21.2.102:0/4029222670 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1c040c2290 msgr2=0x7f1c040c2690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:51.354 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:51.350 7f1c09f07700 1 --2- 172.21.2.102:0/4029222670 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1c040c2290 0x7f1c040c2690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:51.354 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:51.350 7f1c09f07700 1 -- 172.21.2.102:0/4029222670 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c040c0560 con 0x7f1c040c2cb0 2021-02-10T11:31:51.354 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:51.350 7f1c09f07700 1 --2- 172.21.2.102:0/4029222670 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1c040c2cb0 0x7f1c040bfd20 secure :-1 s=READY pgs=2193 cs=0 l=1 rev1=1 rx=0x7f1bfc00b970 tx=0x7f1bfc00bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:51.355 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:51.350 7f1bf9ffb700 1 -- 172.21.2.102:0/4029222670 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1bfc000f90 con 0x7f1c040c2cb0 2021-02-10T11:31:51.355 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:51.350 7f1bf9ffb700 1 -- 172.21.2.102:0/4029222670 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bfc006aa0 con 0x7f1c040c2cb0 2021-02-10T11:31:51.355 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:51.350 7f1bf9ffb700 1 -- 172.21.2.102:0/4029222670 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1bfc001680 con 0x7f1c040c2cb0 2021-02-10T11:31:51.355 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:51.350 7f1c0b8ea700 1 -- 172.21.2.102:0/4029222670 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c040c0560 con 0x7f1c040c2cb0 2021-02-10T11:31:51.355 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:51.350 7f1c0b8ea700 1 -- 172.21.2.102:0/4029222670 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c040c0af0 con 0x7f1c040c2cb0 2021-02-10T11:31:51.355 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:51.351 7f1bf9ffb700 1 -- 172.21.2.102:0/4029222670 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1bfc0011f0 con 0x7f1c040c2cb0 2021-02-10T11:31:51.355 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:51.351 7f1bf9ffb700 1 --2- 172.21.2.102:0/4029222670 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1be40186c0 0x7f1be401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:51.355 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:51.351 7f1bf9ffb700 1 -- 172.21.2.102:0/4029222670 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f1bfc0328d0 con 0x7f1c040c2cb0 2021-02-10T11:31:51.355 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:51.351 7f1c09706700 1 --2- 172.21.2.102:0/4029222670 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1be40186c0 0x7f1be401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:51.356 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:51.351 7f1c09706700 1 --2- 172.21.2.102:0/4029222670 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1be40186c0 0x7f1be401abc0 crc :-1 s=READY pgs=970 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:51.383 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:51.382 7f1c0b8ea700 1 -- 172.21.2.102:0/4029222670 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c040f0b10 con 0x7f1c040c2cb0 2021-02-10T11:31:51.385 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:51.384 7f1bf9ffb700 1 -- 172.21.2.102:0/4029222670 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f1bfc025d60 con 0x7f1c040c2cb0 2021-02-10T11:31:51.511 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:51.510 7f1c0b8ea700 1 -- 172.21.2.102:0/4029222670 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1c040f0b10 con 0x7f1c040c2cb0 2021-02-10T11:31:51.511 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:51.510 7f1bf9ffb700 1 -- 172.21.2.102:0/4029222670 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v112) v1 ==== 74+0+14007 (secure 0 0 0) 0x7f1bfc025d60 con 0x7f1c040c2cb0 2021-02-10T11:31:51.512 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:31:51.512 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":112,"fsid":"68db30ea-dbce-4aef-a6df-f01c6fea0513","created":"2021-02-10 11:03:06.331557","modified":"2021-02-10 11:31:49.261779","last_up_change":"2021-02-10 11:03:32.690562","last_in_change":"2021-02-10 11:03:32.690562","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":23,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:03:37.322519","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":22,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:31:46.228075","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"111","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":23,"pool_name":"cephfs_data","create_time":"2021-02-10 11:31:47.252676","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"112","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"be691b4d-a039-408a-af80-9d53f02b6f1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6818","nonce":36088},{"type":"v1","addr":"172.21.2.102:6820","nonce":36088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6822","nonce":36088},{"type":"v1","addr":"172.21.2.102:6824","nonce":36088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6830","nonce":36088},{"type":"v1","addr":"172.21.2.102:6832","nonce":36088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6826","nonce":36088},{"type":"v1","addr":"172.21.2.102:6828","nonce":36088}]},"public_addr":"172.21.2.102:6820/36088","cluster_addr":"172.21.2.102:6824/36088","heartbeat_back_addr":"172.21.2.102:6832/36088","heartbeat_front_addr":"172.21.2.102:6828/36088","state":["exists","up"]},{"osd":1,"uuid":"64e3ccc8-9c69-4d7c-865e-af8e83822635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6819","nonce":36087},{"type":"v1","addr":"172.21.2.102:6821","nonce":36087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6823","nonce":36087},{"type":"v1","addr":"172.21.2.102:6825","nonce":36087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6831","nonce":36087},{"type":"v1","addr":"172.21.2.102:6833","nonce":36087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6827","nonce":36087},{"type":"v1","addr":"172.21.2.102:6829","nonce":36087}]},"public_addr":"172.21.2.102:6821/36087","cluster_addr":"172.21.2.102:6825/36087","heartbeat_back_addr":"172.21.2.102:6833/36087","heartbeat_front_addr":"172.21.2.102:6829/36087","state":["exists","up"]},{"osd":2,"uuid":"153700af-747f-481f-bdf4-d6072d3a842f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6810","nonce":36089},{"type":"v1","addr":"172.21.2.102:6811","nonce":36089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6812","nonce":36089},{"type":"v1","addr":"172.21.2.102:6813","nonce":36089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6816","nonce":36089},{"type":"v1","addr":"172.21.2.102:6817","nonce":36089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6814","nonce":36089},{"type":"v1","addr":"172.21.2.102:6815","nonce":36089}]},"public_addr":"172.21.2.102:6811/36089","cluster_addr":"172.21.2.102:6813/36089","heartbeat_back_addr":"172.21.2.102:6817/36089","heartbeat_front_addr":"172.21.2.102:6815/36089","state":["exists","up"]},{"osd":3,"uuid":"cbc678f4-83a9-45e6-8830-2f7c232b1ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6802","nonce":36090},{"type":"v1","addr":"172.21.2.102:6803","nonce":36090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6804","nonce":36090},{"type":"v1","addr":"172.21.2.102:6805","nonce":36090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6808","nonce":36090},{"type":"v1","addr":"172.21.2.102:6809","nonce":36090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6806","nonce":36090},{"type":"v1","addr":"172.21.2.102:6807","nonce":36090}]},"public_addr":"172.21.2.102:6803/36090","cluster_addr":"172.21.2.102:6805/36090","heartbeat_back_addr":"172.21.2.102:6809/36090","heartbeat_front_addr":"172.21.2.102:6807/36090","state":["exists","up"]},{"osd":4,"uuid":"c01e4008-b258-4f10-afd2-9e7f37214351","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6808","nonce":36368},{"type":"v1","addr":"172.21.2.113:6811","nonce":36368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6814","nonce":36368},{"type":"v1","addr":"172.21.2.113:6818","nonce":36368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6826","nonce":36368},{"type":"v1","addr":"172.21.2.113:6828","nonce":36368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6820","nonce":36368},{"type":"v1","addr":"172.21.2.113:6824","nonce":36368}]},"public_addr":"172.21.2.113:6811/36368","cluster_addr":"172.21.2.113:6818/36368","heartbeat_back_addr":"172.21.2.113:6828/36368","heartbeat_front_addr":"172.21.2.113:6824/36368","state":["exists","up"]},{"osd":5,"uuid":"551f6007-daf8-4a98-8ee0-c2b16bcccb5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6802","nonce":36366},{"type":"v1","addr":"172.21.2.113:6805","nonce":36366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6807","nonce":36366},{"type":"v1","addr":"172.21.2.113:6810","nonce":36366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6819","nonce":36366},{"type":"v1","addr":"172.21.2.113:6822","nonce":36366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6812","nonce":36366},{"type":"v1","addr":"172.21.2.113:6815","nonce":36366}]},"public_addr":"172.21.2.113:6805/36366","cluster_addr":"172.21.2.113:6810/36366","heartbeat_back_addr":"172.21.2.113:6822/36366","heartbeat_front_addr":"172.21.2.113:6815/36366","state":["exists","up"]},{"osd":6,"uuid":"bf67c040-0e50-48be-b102-c3943bc9ebef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6800","nonce":36367},{"type":"v1","addr":"172.21.2.113:6801","nonce":36367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6803","nonce":36367},{"type":"v1","addr":"172.21.2.113:6804","nonce":36367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6813","nonce":36367},{"type":"v1","addr":"172.21.2.113:6816","nonce":36367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6806","nonce":36367},{"type":"v1","addr":"172.21.2.113:6809","nonce":36367}]},"public_addr":"172.21.2.113:6801/36367","cluster_addr":"172.21.2.113:6804/36367","heartbeat_back_addr":"172.21.2.113:6816/36367","heartbeat_front_addr":"172.21.2.113:6809/36367","state":["exists","up"]},{"osd":7,"uuid":"f5461f80-018d-428f-be89-2eb2bcbe7bb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6817","nonce":36365},{"type":"v1","addr":"172.21.2.113:6821","nonce":36365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6823","nonce":36365},{"type":"v1","addr":"172.21.2.113:6825","nonce":36365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6830","nonce":36365},{"type":"v1","addr":"172.21.2.113:6831","nonce":36365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6827","nonce":36365},{"type":"v1","addr":"172.21.2.113:6829","nonce":36365}]},"public_addr":"172.21.2.113:6821/36365","cluster_addr":"172.21.2.113:6825/36365","heartbeat_back_addr":"172.21.2.113:6831/36365","heartbeat_front_addr":"172.21.2.113:6829/36365","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:31:51.513 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:51.512 7f1c0b8ea700 1 -- 172.21.2.102:0/4029222670 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1be40186c0 msgr2=0x7f1be401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:51.513 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:51.512 7f1c0b8ea700 1 --2- 172.21.2.102:0/4029222670 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1be40186c0 0x7f1be401abc0 crc :-1 s=READY pgs=970 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:51.513 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:51.512 7f1c0b8ea700 1 -- 172.21.2.102:0/4029222670 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1c040c2cb0 msgr2=0x7f1c040bfd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:51.513 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:51.512 7f1c0b8ea700 1 --2- 172.21.2.102:0/4029222670 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1c040c2cb0 0x7f1c040bfd20 secure :-1 s=READY pgs=2193 cs=0 l=1 rev1=1 rx=0x7f1bfc00b970 tx=0x7f1bfc00bd10).stop 2021-02-10T11:31:51.514 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:51.512 7f1c0b8ea700 1 -- 172.21.2.102:0/4029222670 shutdown_connections 2021-02-10T11:31:51.514 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:51.512 7f1c0b8ea700 1 --2- 172.21.2.102:0/4029222670 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1c040c2cb0 0x7f1c040bfd20 unknown :-1 s=CLOSED pgs=2193 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:51.514 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:51.512 7f1c0b8ea700 1 --2- 172.21.2.102:0/4029222670 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1be40186c0 0x7f1be401abc0 unknown :-1 s=CLOSED pgs=970 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:51.514 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:51.512 7f1c0b8ea700 1 --2- 172.21.2.102:0/4029222670 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1c040c2290 0x7f1c040c2690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:51.514 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:51.512 7f1c0b8ea700 1 --2- 172.21.2.102:0/4029222670 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1c041a4a60 0x7f1c040c1c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:51.514 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:51.512 7f1c0b8ea700 1 -- 172.21.2.102:0/4029222670 >> 172.21.2.102:0/4029222670 conn(0x7f1c040ebe30 msgr2=0x7f1c040bf200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:51.514 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:51.513 7f1c0b8ea700 1 -- 172.21.2.102:0/4029222670 shutdown_connections 2021-02-10T11:31:51.514 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:51.513 7f1c0b8ea700 1 -- 172.21.2.102:0/4029222670 wait complete. 2021-02-10T11:31:51.522 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:31:51.964 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:51.946 7fba61cc9700 1 Processor -- start 2021-02-10T11:31:51.964 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:51.946 7fba61cc9700 1 -- start start 2021-02-10T11:31:51.964 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:51.946 7fba61cc9700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fba5c0d1c80 0x7fba5c0cef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:51.964 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:51.946 7fba61cc9700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fba5c0cf590 0x7fba5c0cf990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:51.964 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:51.946 7fba61cc9700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba5c0fe1d0 con 0x7fba5c0d1880 2021-02-10T11:31:51.964 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:51.946 7fba61cc9700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba5c1b0cc0 con 0x7fba5c0cf590 2021-02-10T11:31:51.964 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:51.946 7fba61cc9700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba5c109a70 con 0x7fba5c0d1c80 2021-02-10T11:31:51.964 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:51.946 7fba5bfff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fba5c0cf590 0x7fba5c0cf990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:51.964 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:51.946 7fba5b7fe700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fba5c0d1880 0x7fba5c1b04d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:54484/0 (socket says 172.21.2.102:54484) 2021-02-10T11:31:51.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:51.946 7fba5b7fe700 1 -- 172.21.2.102:0/3749514068 learned_addr learned my addr 172.21.2.102:0/3749514068 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:31:51.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:51.946 7fba5bfff700 1 --2- 172.21.2.102:0/3749514068 >> v2:172.21.2.102:3300/0 conn(0x7fba5c0cf590 0x7fba5c0cf990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:34030/0 (socket says 172.21.2.102:34030) 2021-02-10T11:31:51.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:51.947 7fba5bfff700 1 -- 172.21.2.102:0/3749514068 >> v2:172.21.2.113:3300/0 conn(0x7fba5c0d1c80 msgr2=0x7fba5c0cef70 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:31:51.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:51.947 7fba5bfff700 1 --2- 172.21.2.102:0/3749514068 >> v2:172.21.2.113:3300/0 conn(0x7fba5c0d1c80 0x7fba5c0cef70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:51.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:51.947 7fba5bfff700 1 -- 172.21.2.102:0/3749514068 >> v1:172.21.2.102:6789/0 conn(0x7fba5c0d1880 legacy=0x7fba5c1b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:51.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:51.947 7fba5a7fc700 1 -- 172.21.2.102:0/3749514068 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4206597764 0 0) 0x7fba44001180 con 0x7fba5c0d1880 2021-02-10T11:31:51.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:51.947 7fba5bfff700 1 -- 172.21.2.102:0/3749514068 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba5c088070 con 0x7fba5c0cf590 2021-02-10T11:31:51.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:51.947 7fba5bfff700 1 --2- 172.21.2.102:0/3749514068 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fba5c0cf590 0x7fba5c0cf990 secure :-1 s=READY pgs=2195 cs=0 l=1 rev1=1 rx=0x7fba54006e30 tx=0x7fba54009de0).ready entity=mon.0 client_cookie=56fbe0dd08e176b4 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:51.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:51.947 7fba5a7fc700 1 -- 172.21.2.102:0/3749514068 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fba5400b300 con 0x7fba5c0cf590 2021-02-10T11:31:51.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:51.947 7fba5a7fc700 1 -- 172.21.2.102:0/3749514068 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba5400b580 con 0x7fba5c0cf590 2021-02-10T11:31:51.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:51.947 7fba5a7fc700 1 -- 172.21.2.102:0/3749514068 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fba5400baa0 con 0x7fba5c0cf590 2021-02-10T11:31:51.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:51.947 7fba61cc9700 1 -- 172.21.2.102:0/3749514068 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fba5c0cf590 msgr2=0x7fba5c0cf990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:51.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:51.947 7fba61cc9700 1 --2- 172.21.2.102:0/3749514068 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fba5c0cf590 0x7fba5c0cf990 secure :-1 s=READY pgs=2195 cs=0 l=1 rev1=1 rx=0x7fba54006e30 tx=0x7fba54009de0).stop 2021-02-10T11:31:51.966 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:51.948 7fba61cc9700 1 -- 172.21.2.102:0/3749514068 shutdown_connections 2021-02-10T11:31:51.966 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:51.948 7fba61cc9700 1 --2- 172.21.2.102:0/3749514068 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fba5c0cf590 0x7fba5c0cf990 unknown :-1 s=CLOSED pgs=2195 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:51.966 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:51.948 7fba61cc9700 1 --2- 172.21.2.102:0/3749514068 >> v2:172.21.2.113:3300/0 conn(0x7fba5c0d1c80 0x7fba5c0cef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:51.966 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:51.948 7fba61cc9700 1 -- 172.21.2.102:0/3749514068 >> 172.21.2.102:0/3749514068 conn(0x7fba5c0ce430 msgr2=0x7fba5c0d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:51.966 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:51.948 7fba61cc9700 1 -- 172.21.2.102:0/3749514068 shutdown_connections 2021-02-10T11:31:51.966 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:51.948 7fba61cc9700 1 -- 172.21.2.102:0/3749514068 wait complete. 2021-02-10T11:31:51.966 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:51.962 7fba61cc9700 1 Processor -- start 2021-02-10T11:31:51.966 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:51.963 7fba61cc9700 1 -- start start 2021-02-10T11:31:51.967 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:51.963 7fba61cc9700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fba5c1b9470 0x7fba5c1ae6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:51.967 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:51.963 7fba61cc9700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fba5c1aecf0 0x7fba5c0957f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:51.968 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:51.963 7fba61cc9700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fba5c1af0f0 0x7fba5c095e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:51.968 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:51.963 7fba61cc9700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba5c096d70 con 0x7fba5c1af0f0 2021-02-10T11:31:51.968 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:51.963 7fba61cc9700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba5c0a7780 con 0x7fba5c1b9470 2021-02-10T11:31:51.968 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:51.963 7fba61cc9700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba5c1b9870 con 0x7fba5c1aecf0 2021-02-10T11:31:51.968 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:51.963 7fba5bfff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fba5c1af0f0 0x7fba5c095e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:51.968 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:51.963 7fba5bfff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fba5c1af0f0 0x7fba5c095e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:34036/0 (socket says 172.21.2.102:34036) 2021-02-10T11:31:51.968 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:51.963 7fba5bfff700 1 -- 172.21.2.102:0/818421373 learned_addr learned my addr 172.21.2.102:0/818421373 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:31:51.968 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:51.963 7fba5bfff700 1 -- 172.21.2.102:0/818421373 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fba5c1aecf0 msgr2=0x7fba5c0957f0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:31:51.968 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:51.963 7fba5bfff700 1 --2- 172.21.2.102:0/818421373 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fba5c1aecf0 0x7fba5c0957f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:51.968 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:51.963 7fba5bfff700 1 -- 172.21.2.102:0/818421373 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fba5c1b9470 msgr2=0x7fba5c1ae6d0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:31:51.968 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:51.963 7fba5bfff700 1 --2- 172.21.2.102:0/818421373 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fba5c1b9470 0x7fba5c1ae6d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:51.968 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:51.964 7fba5bfff700 1 -- 172.21.2.102:0/818421373 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba5c19e6b0 con 0x7fba5c1af0f0 2021-02-10T11:31:51.969 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:51.964 7fba5bfff700 1 --2- 172.21.2.102:0/818421373 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fba5c1af0f0 0x7fba5c095e10 secure :-1 s=READY pgs=2196 cs=0 l=1 rev1=1 rx=0x7fba5400b970 tx=0x7fba5400bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:51.969 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:51.964 7fba53fff700 1 -- 172.21.2.102:0/818421373 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fba54009380 con 0x7fba5c1af0f0 2021-02-10T11:31:51.969 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:51.964 7fba53fff700 1 -- 172.21.2.102:0/818421373 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba54001270 con 0x7fba5c1af0f0 2021-02-10T11:31:51.969 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:51.964 7fba61cc9700 1 -- 172.21.2.102:0/818421373 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba5c19e6b0 con 0x7fba5c1af0f0 2021-02-10T11:31:51.969 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:51.964 7fba53fff700 1 -- 172.21.2.102:0/818421373 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fba540016d0 con 0x7fba5c1af0f0 2021-02-10T11:31:51.969 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:51.964 7fba61cc9700 1 -- 172.21.2.102:0/818421373 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba5c19e6b0 con 0x7fba5c1af0f0 2021-02-10T11:31:51.969 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:51.964 7fba53fff700 1 -- 172.21.2.102:0/818421373 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fba54001450 con 0x7fba5c1af0f0 2021-02-10T11:31:51.969 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:51.964 7fba53fff700 1 --2- 172.21.2.102:0/818421373 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fba400186d0 0x7fba4001abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:51.969 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:51.964 7fba53fff700 1 -- 172.21.2.102:0/818421373 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fba540329e0 con 0x7fba5c1af0f0 2021-02-10T11:31:51.969 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:51.965 7fba5b7fe700 1 --2- 172.21.2.102:0/818421373 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fba400186d0 0x7fba4001abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:51.969 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:51.965 7fba5b7fe700 1 --2- 172.21.2.102:0/818421373 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fba400186d0 0x7fba4001abd0 crc :-1 s=READY pgs=971 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:51.997 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:51.996 7fba61cc9700 1 -- 172.21.2.102:0/818421373 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba5c0829a0 con 0x7fba5c1af0f0 2021-02-10T11:31:51.999 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:51.998 7fba53fff700 1 -- 172.21.2.102:0/818421373 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fba54001480 con 0x7fba5c1af0f0 2021-02-10T11:31:52.129 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:52.127 7fba61cc9700 1 -- 172.21.2.102:0/818421373 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7fba5c0a7780 con 0x7fba5c1af0f0 2021-02-10T11:31:52.133 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:52.131 7fba53fff700 1 -- 172.21.2.102:0/818421373 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.0 v63) v1 ==== 184+0+0 (secure 0 0 0) 0x7fba54005820 con 0x7fba5c1af0f0 2021-02-10T11:31:52.136 INFO:teuthology.orchestra.run.gibba002.stderr:updated caps for client.0 2021-02-10T11:31:52.138 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:52.136 7fba61cc9700 1 -- 172.21.2.102:0/818421373 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fba400186d0 msgr2=0x7fba4001abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:52.138 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:52.136 7fba61cc9700 1 --2- 172.21.2.102:0/818421373 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fba400186d0 0x7fba4001abd0 crc :-1 s=READY pgs=971 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:52.138 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:52.136 7fba61cc9700 1 -- 172.21.2.102:0/818421373 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fba5c1af0f0 msgr2=0x7fba5c095e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:52.138 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:52.136 7fba61cc9700 1 --2- 172.21.2.102:0/818421373 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fba5c1af0f0 0x7fba5c095e10 secure :-1 s=READY pgs=2196 cs=0 l=1 rev1=1 rx=0x7fba5400b970 tx=0x7fba5400bd10).stop 2021-02-10T11:31:52.138 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:52.137 7fba61cc9700 1 -- 172.21.2.102:0/818421373 shutdown_connections 2021-02-10T11:31:52.138 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:52.137 7fba61cc9700 1 --2- 172.21.2.102:0/818421373 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fba5c1af0f0 0x7fba5c095e10 unknown :-1 s=CLOSED pgs=2196 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:52.138 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:52.137 7fba61cc9700 1 --2- 172.21.2.102:0/818421373 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fba5c1aecf0 0x7fba5c0957f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:52.138 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:52.137 7fba61cc9700 1 --2- 172.21.2.102:0/818421373 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fba400186d0 0x7fba4001abd0 unknown :-1 s=CLOSED pgs=971 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:52.138 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:52.137 7fba61cc9700 1 --2- 172.21.2.102:0/818421373 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fba5c1b9470 0x7fba5c1ae6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:52.138 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:52.137 7fba61cc9700 1 -- 172.21.2.102:0/818421373 >> 172.21.2.102:0/818421373 conn(0x7fba5c0845b0 msgr2=0x7fba5c18c290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:52.139 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:52.137 7fba61cc9700 1 -- 172.21.2.102:0/818421373 shutdown_connections 2021-02-10T11:31:52.139 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:52.137 7fba61cc9700 1 -- 172.21.2.102:0/818421373 wait complete. 2021-02-10T11:31:52.146 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.1 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:31:52.589 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:52.569 7eff09944700 1 Processor -- start 2021-02-10T11:31:52.589 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:52.569 7eff09944700 1 -- start start 2021-02-10T11:31:52.589 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:52.569 7eff09944700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7eff0410afe0 0x7eff040cf740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:52.589 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:52.569 7eff09944700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff040fe1b0 con 0x7eff040d1c60 2021-02-10T11:31:52.590 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:52.569 7eff09944700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff041b0ca0 con 0x7eff040d1860 2021-02-10T11:31:52.590 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:52.569 7eff09944700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff04109a50 con 0x7eff0410afe0 2021-02-10T11:31:52.590 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:52.569 7eff02d73700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7eff040d1c60 0x7eff040cef50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:54498/0 (socket says 172.21.2.102:54498) 2021-02-10T11:31:52.590 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:52.569 7eff02d73700 1 -- 172.21.2.102:0/2570045826 learned_addr learned my addr 172.21.2.102:0/2570045826 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:31:52.590 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:52.570 7eff02572700 1 -- 172.21.2.102:0/2570045826 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 495413020 0 0) 0x7efef80010b0 con 0x7eff040d1c60 2021-02-10T11:31:52.590 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:52.570 7eff02572700 1 -- 172.21.2.102:0/2570045826 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efeec002fe0 con 0x7eff040d1c60 2021-02-10T11:31:52.590 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:52.570 7eff03d75700 1 --2- 172.21.2.102:0/2570045826 >> v2:172.21.2.113:3300/0 conn(0x7eff0410afe0 0x7eff040cf740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:52.590 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:52.570 7eff02572700 1 -- 172.21.2.102:0/2570045826 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1610658077 0 0) 0x7efef80010b0 con 0x7eff040d1c60 2021-02-10T11:31:52.590 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:52.570 7eff02572700 1 -- 172.21.2.102:0/2570045826 >> v2:172.21.2.113:3300/0 conn(0x7eff0410afe0 msgr2=0x7eff040cf740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:52.590 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:52.570 7eff02572700 1 --2- 172.21.2.102:0/2570045826 >> v2:172.21.2.113:3300/0 conn(0x7eff0410afe0 0x7eff040cf740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:52.590 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:52.570 7eff02572700 1 -- 172.21.2.102:0/2570045826 >> v1:172.21.2.113:6789/0 conn(0x7eff040d1860 legacy=0x7eff041b04b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:52.590 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:52.570 7eff02572700 1 -- 172.21.2.102:0/2570045826 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff0410b3e0 con 0x7eff040d1c60 2021-02-10T11:31:52.591 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:52.570 7eff02572700 1 -- 172.21.2.102:0/2570045826 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7efef80015b0 con 0x7eff040d1c60 2021-02-10T11:31:52.591 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:52.570 7eff02572700 1 -- 172.21.2.102:0/2570045826 >> v1:172.21.2.102:6789/0 conn(0x7eff040d1c60 legacy=0x7eff040cef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:52.591 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:52.570 7eff02572700 1 --2- 172.21.2.102:0/2570045826 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efeec003ac0 0x7efeec005f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:52.591 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:52.570 7eff02572700 1 --2- 172.21.2.102:0/2570045826 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7efeec0067b0 0x7efeec008c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:52.591 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:52.570 7eff02572700 1 --2- 172.21.2.102:0/2570045826 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7efeec009590 0x7efeec00bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:52.591 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:52.570 7eff02572700 1 -- 172.21.2.102:0/2570045826 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efeec00c300 con 0x7efeec003ac0 2021-02-10T11:31:52.591 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:52.570 7eff02572700 1 -- 172.21.2.102:0/2570045826 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efeec00c520 con 0x7efeec009590 2021-02-10T11:31:52.591 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:52.570 7eff02572700 1 -- 172.21.2.102:0/2570045826 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efeec00c740 con 0x7efeec0067b0 2021-02-10T11:31:52.591 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:52.571 7eff03574700 1 --2- 172.21.2.102:0/2570045826 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efeec003ac0 0x7efeec005f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:52.591 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:52.571 7eff03d75700 1 --2- 172.21.2.102:0/2570045826 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7efeec009590 0x7efeec00bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:52.591 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:52.571 7eff02d73700 1 --2- 172.21.2.102:0/2570045826 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7efeec0067b0 0x7efeec008c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:52.591 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:52.571 7eff03574700 1 -- 172.21.2.102:0/2570045826 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7efeec0067b0 msgr2=0x7efeec008c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:52.592 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:52.571 7eff03574700 1 --2- 172.21.2.102:0/2570045826 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7efeec0067b0 0x7efeec008c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:52.592 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:52.571 7eff03574700 1 -- 172.21.2.102:0/2570045826 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7efeec009590 msgr2=0x7efeec00bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:52.592 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:52.571 7eff03574700 1 --2- 172.21.2.102:0/2570045826 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7efeec009590 0x7efeec00bab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:52.592 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:52.571 7eff03574700 1 -- 172.21.2.102:0/2570045826 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efeec0509b0 con 0x7efeec003ac0 2021-02-10T11:31:52.592 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:52.571 7eff03d75700 1 -- 172.21.2.102:0/2570045826 reap_dead start 2021-02-10T11:31:52.592 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:52.571 7eff03574700 1 --2- 172.21.2.102:0/2570045826 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efeec003ac0 0x7efeec005f10 secure :-1 s=READY pgs=2198 cs=0 l=1 rev1=1 rx=0x7efef4006ab0 tx=0x7efef4009b40).ready entity=mon.0 client_cookie=841ed891f9bfa345 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:52.592 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:52.571 7eff02572700 1 -- 172.21.2.102:0/2570045826 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efef400b2e0 con 0x7efeec003ac0 2021-02-10T11:31:52.592 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:52.571 7eff02572700 1 -- 172.21.2.102:0/2570045826 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efef400b660 con 0x7efeec003ac0 2021-02-10T11:31:52.592 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:52.571 7eff02572700 1 -- 172.21.2.102:0/2570045826 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efef400bb40 con 0x7efeec003ac0 2021-02-10T11:31:52.592 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:52.571 7eff09944700 1 -- 172.21.2.102:0/2570045826 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efeec003ac0 msgr2=0x7efeec005f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:52.592 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:52.571 7eff09944700 1 --2- 172.21.2.102:0/2570045826 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efeec003ac0 0x7efeec005f10 secure :-1 s=READY pgs=2198 cs=0 l=1 rev1=1 rx=0x7efef4006ab0 tx=0x7efef4009b40).stop 2021-02-10T11:31:52.592 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:52.572 7eff09944700 1 -- 172.21.2.102:0/2570045826 shutdown_connections 2021-02-10T11:31:52.592 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:52.572 7eff09944700 1 --2- 172.21.2.102:0/2570045826 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efeec003ac0 0x7efeec005f10 unknown :-1 s=CLOSED pgs=2198 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:52.593 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:52.572 7eff09944700 1 -- 172.21.2.102:0/2570045826 >> 172.21.2.102:0/2570045826 conn(0x7eff040ce410 msgr2=0x7eff040d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:52.593 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:52.572 7eff09944700 1 -- 172.21.2.102:0/2570045826 shutdown_connections 2021-02-10T11:31:52.593 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:52.572 7eff09944700 1 -- 172.21.2.102:0/2570045826 wait complete. 2021-02-10T11:31:52.593 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:52.587 7eff09944700 1 Processor -- start 2021-02-10T11:31:52.593 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:52.587 7eff09944700 1 -- start start 2021-02-10T11:31:52.593 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:52.587 7eff09944700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7eff040b5d30 0x7eff040c2910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:52.593 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:52.587 7eff09944700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7eff040ae630 0x7eff040aea30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:52.593 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:52.587 7eff09944700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7eff040bee80 0x7eff040bf280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:52.593 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:52.587 7eff09944700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff040c2f30 con 0x7eff040b5d30 2021-02-10T11:31:52.593 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:52.587 7eff09944700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff040c2390 con 0x7eff040bee80 2021-02-10T11:31:52.593 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:52.587 7eff09944700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff040b6130 con 0x7eff040ae630 2021-02-10T11:31:52.593 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:52.588 7eff03574700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7eff040b5d30 0x7eff040c2910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:52.593 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:52.588 7eff03574700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7eff040b5d30 0x7eff040c2910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:34054/0 (socket says 172.21.2.102:34054) 2021-02-10T11:31:52.594 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:52.588 7eff03574700 1 -- 172.21.2.102:0/1835405071 learned_addr learned my addr 172.21.2.102:0/1835405071 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:31:52.594 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:52.588 7eff03574700 1 -- 172.21.2.102:0/1835405071 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7eff040ae630 msgr2=0x7eff040aea30 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:31:52.594 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:52.588 7eff03574700 1 --2- 172.21.2.102:0/1835405071 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7eff040ae630 0x7eff040aea30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:52.594 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:52.588 7eff03574700 1 -- 172.21.2.102:0/1835405071 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7eff040bee80 msgr2=0x7eff040bf280 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:31:52.594 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:52.588 7eff03574700 1 --2- 172.21.2.102:0/1835405071 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7eff040bee80 0x7eff040bf280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:52.594 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:52.588 7eff03574700 1 -- 172.21.2.102:0/1835405071 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff040bf8a0 con 0x7eff040b5d30 2021-02-10T11:31:52.594 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:52.588 7eff03574700 1 --2- 172.21.2.102:0/1835405071 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7eff040b5d30 0x7eff040c2910 secure :-1 s=READY pgs=2199 cs=0 l=1 rev1=1 rx=0x7efef40093a0 tx=0x7efef400b9c0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:52.595 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:52.588 7efeebfff700 1 -- 172.21.2.102:0/1835405071 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efef400bdb0 con 0x7eff040b5d30 2021-02-10T11:31:52.595 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:52.588 7efeebfff700 1 -- 172.21.2.102:0/1835405071 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efef4000a10 con 0x7eff040b5d30 2021-02-10T11:31:52.595 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:52.588 7efeebfff700 1 -- 172.21.2.102:0/1835405071 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efef40054a0 con 0x7eff040b5d30 2021-02-10T11:31:52.595 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:52.588 7eff09944700 1 -- 172.21.2.102:0/1835405071 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff040bf8a0 con 0x7eff040b5d30 2021-02-10T11:31:52.595 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:52.588 7eff09944700 1 -- 172.21.2.102:0/1835405071 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff040bfe30 con 0x7eff040b5d30 2021-02-10T11:31:52.596 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:52.589 7efeebfff700 1 -- 172.21.2.102:0/1835405071 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7efef400a000 con 0x7eff040b5d30 2021-02-10T11:31:52.596 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:52.589 7efeebfff700 1 --2- 172.21.2.102:0/1835405071 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7efee40186c0 0x7efee401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:52.596 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:52.589 7efeebfff700 1 -- 172.21.2.102:0/1835405071 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5573+0+0 (secure 0 0 0) 0x7efef4032580 con 0x7eff040b5d30 2021-02-10T11:31:52.596 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:52.589 7eff02d73700 1 --2- 172.21.2.102:0/1835405071 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7efee40186c0 0x7efee401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:52.596 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:52.589 7eff02d73700 1 --2- 172.21.2.102:0/1835405071 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7efee40186c0 0x7efee401abc0 crc :-1 s=READY pgs=972 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:52.620 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:52.619 7eff09944700 1 -- 172.21.2.102:0/1835405071 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff040fe1b0 con 0x7eff040b5d30 2021-02-10T11:31:52.623 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:52.622 7efeebfff700 1 -- 172.21.2.102:0/1835405071 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7efef40219a0 con 0x7eff040b5d30 2021-02-10T11:31:52.753 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:52.752 7eff09944700 1 -- 172.21.2.102:0/1835405071 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7eff040fe1b0 con 0x7eff040b5d30 2021-02-10T11:31:52.756 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:52.755 7efeebfff700 1 -- 172.21.2.102:0/1835405071 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.1 v64) v1 ==== 184+0+0 (secure 0 0 0) 0x7efef400a000 con 0x7eff040b5d30 2021-02-10T11:31:52.760 INFO:teuthology.orchestra.run.gibba002.stderr:updated caps for client.1 2021-02-10T11:31:52.762 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:52.760 7eff09944700 1 -- 172.21.2.102:0/1835405071 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7efee40186c0 msgr2=0x7efee401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:52.762 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:52.760 7eff09944700 1 --2- 172.21.2.102:0/1835405071 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7efee40186c0 0x7efee401abc0 crc :-1 s=READY pgs=972 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:52.762 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:52.760 7eff09944700 1 -- 172.21.2.102:0/1835405071 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7eff040b5d30 msgr2=0x7eff040c2910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:52.762 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:52.760 7eff09944700 1 --2- 172.21.2.102:0/1835405071 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7eff040b5d30 0x7eff040c2910 secure :-1 s=READY pgs=2199 cs=0 l=1 rev1=1 rx=0x7efef40093a0 tx=0x7efef400b9c0).stop 2021-02-10T11:31:52.762 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:52.761 7eff09944700 1 -- 172.21.2.102:0/1835405071 shutdown_connections 2021-02-10T11:31:52.762 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:52.761 7eff09944700 1 --2- 172.21.2.102:0/1835405071 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7efee40186c0 0x7efee401abc0 unknown :-1 s=CLOSED pgs=972 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:52.762 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:52.761 7eff09944700 1 --2- 172.21.2.102:0/1835405071 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7eff040bee80 0x7eff040bf280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:52.762 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:52.761 7eff09944700 1 --2- 172.21.2.102:0/1835405071 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7eff040ae630 0x7eff040aea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:52.763 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:52.761 7eff09944700 1 --2- 172.21.2.102:0/1835405071 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7eff040b5d30 0x7eff040c2910 unknown :-1 s=CLOSED pgs=2199 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:52.763 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:52.761 7eff09944700 1 -- 172.21.2.102:0/1835405071 >> 172.21.2.102:0/1835405071 conn(0x7eff040874f0 msgr2=0x7eff0409e240 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:52.763 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:52.761 7eff09944700 1 -- 172.21.2.102:0/1835405071 shutdown_connections 2021-02-10T11:31:52.763 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:52.761 7eff09944700 1 -- 172.21.2.102:0/1835405071 wait complete. 2021-02-10T11:31:52.771 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.2 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:31:53.201 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:53.183 7f30473f3700 1 Processor -- start 2021-02-10T11:31:53.202 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:53.183 7f30473f3700 1 -- start start 2021-02-10T11:31:53.202 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:53.184 7f30473f3700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f30400d1880 0x7f30401b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:53.202 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:53.184 7f30473f3700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30400fe1d0 con 0x7f30401b0af0 2021-02-10T11:31:53.202 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:53.184 7f30473f3700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3040109a70 con 0x7f30400cff80 2021-02-10T11:31:53.202 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:53.184 7f30473f3700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3040088070 con 0x7f30400d1880 2021-02-10T11:31:53.202 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:53.184 7f3044a0e700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f30401b0af0 0x7f3040108950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:54516/0 (socket says 172.21.2.102:54516) 2021-02-10T11:31:53.202 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:53.184 7f3044a0e700 1 -- 172.21.2.102:0/3736397066 learned_addr learned my addr 172.21.2.102:0/3736397066 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:31:53.202 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:53.184 7f304520f700 1 --2- 172.21.2.102:0/3736397066 >> v2:172.21.2.113:3300/0 conn(0x7f30400d1880 0x7f30401b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:53.202 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:53.185 7f3033fff700 1 -- 172.21.2.102:0/3736397066 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2299834918 0 0) 0x7f303c0009b0 con 0x7f30401b0af0 2021-02-10T11:31:53.202 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:53.185 7f3033fff700 1 -- 172.21.2.102:0/3736397066 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f302c002fe0 con 0x7f30401b0af0 2021-02-10T11:31:53.202 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:53.185 7f3033fff700 1 -- 172.21.2.102:0/3736397066 <== mon.1 v1:172.21.2.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1006182677 0 0) 0x7f3038000f90 con 0x7f30400cff80 2021-02-10T11:31:53.203 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:53.185 7f3033fff700 1 -- 172.21.2.102:0/3736397066 --> v1:172.21.2.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f302c003210 con 0x7f30400cff80 2021-02-10T11:31:53.203 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:53.185 7f304520f700 1 -- 172.21.2.102:0/3736397066 >> v1:172.21.2.113:6789/0 conn(0x7f30400cff80 legacy=0x7f3040109140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:53.203 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:53.185 7f304520f700 1 -- 172.21.2.102:0/3736397066 >> v1:172.21.2.102:6789/0 conn(0x7f30401b0af0 legacy=0x7f3040108950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:53.203 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:53.185 7f3033fff700 1 -- 172.21.2.102:0/3736397066 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2697939311 0 0) 0x7f303c001010 con 0x7f30401b0af0 2021-02-10T11:31:53.203 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:53.185 7f304520f700 1 -- 172.21.2.102:0/3736397066 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3040088260 con 0x7f30400d1880 2021-02-10T11:31:53.203 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:53.185 7f304520f700 1 --2- 172.21.2.102:0/3736397066 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f30400d1880 0x7f30401b04d0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 rx=0x7f3034006650 tx=0x7f3034009a80).ready entity=mon.1 client_cookie=eeebaf04aab3ac02 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:53.203 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:53.185 7f3033fff700 1 -- 172.21.2.102:0/3736397066 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f30340009b0 con 0x7f30400d1880 2021-02-10T11:31:53.203 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:53.185 7f3033fff700 1 -- 172.21.2.102:0/3736397066 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3034006740 con 0x7f30400d1880 2021-02-10T11:31:53.203 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:53.185 7f3033fff700 1 -- 172.21.2.102:0/3736397066 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f303400c0c0 con 0x7f30400d1880 2021-02-10T11:31:53.203 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:53.185 7f30473f3700 1 -- 172.21.2.102:0/3736397066 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f30400d1880 msgr2=0x7f30401b04d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:53.203 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:53.185 7f30473f3700 1 --2- 172.21.2.102:0/3736397066 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f30400d1880 0x7f30401b04d0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 rx=0x7f3034006650 tx=0x7f3034009a80).stop 2021-02-10T11:31:53.203 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:53.186 7f30473f3700 1 -- 172.21.2.102:0/3736397066 shutdown_connections 2021-02-10T11:31:53.203 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:53.186 7f30473f3700 1 --2- 172.21.2.102:0/3736397066 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f30400d1880 0x7f30401b04d0 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:53.204 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:53.186 7f30473f3700 1 -- 172.21.2.102:0/3736397066 >> 172.21.2.102:0/3736397066 conn(0x7f30400ce430 msgr2=0x7f30400d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:53.204 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:53.186 7f30473f3700 1 -- 172.21.2.102:0/3736397066 shutdown_connections 2021-02-10T11:31:53.204 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:53.186 7f30473f3700 1 -- 172.21.2.102:0/3736397066 wait complete. 2021-02-10T11:31:53.205 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:53.201 7f30473f3700 1 Processor -- start 2021-02-10T11:31:53.205 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:53.201 7f30473f3700 1 -- start start 2021-02-10T11:31:53.205 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:53.201 7f30473f3700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f30401a0580 0x7f3040086fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:53.205 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:53.201 7f30473f3700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f30400b66b0 0x7f30400b6ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:53.205 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:53.201 7f30473f3700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3040082bb0 0x7f3040082fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:53.205 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:53.201 7f30473f3700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30400875d0 con 0x7f30401a0580 2021-02-10T11:31:53.206 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:53.201 7f30473f3700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30400835d0 con 0x7f30400b66b0 2021-02-10T11:31:53.206 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:53.201 7f30473f3700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30400837f0 con 0x7f3040082bb0 2021-02-10T11:31:53.206 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:53.202 7f3045a10700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3040082bb0 0x7f3040082fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:53.206 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:53.202 7f304520f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f30401a0580 0x7f3040086fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:53.206 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:53.202 7f3045a10700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3040082bb0 0x7f3040082fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.113:3301/0 says I am v2:172.21.2.102:39600/0 (socket says 172.21.2.102:39600) 2021-02-10T11:31:53.206 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:53.202 7f3045a10700 1 -- 172.21.2.102:0/3514766872 learned_addr learned my addr 172.21.2.102:0/3514766872 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:31:53.206 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:53.202 7f3044a0e700 1 --2- 172.21.2.102:0/3514766872 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f30400b66b0 0x7f30400b6ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:53.206 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:53.202 7f304520f700 1 -- 172.21.2.102:0/3514766872 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3040082bb0 msgr2=0x7f3040082fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:53.206 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:53.202 7f304520f700 1 --2- 172.21.2.102:0/3514766872 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3040082bb0 0x7f3040082fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:53.206 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:53.202 7f304520f700 1 -- 172.21.2.102:0/3514766872 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f30400b66b0 msgr2=0x7f30400b6ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:53.206 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:53.202 7f304520f700 1 --2- 172.21.2.102:0/3514766872 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f30400b66b0 0x7f30400b6ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:53.206 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:53.202 7f304520f700 1 -- 172.21.2.102:0/3514766872 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3040083a10 con 0x7f30401a0580 2021-02-10T11:31:53.206 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:53.202 7f3044a0e700 1 --2- 172.21.2.102:0/3514766872 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f30400b66b0 0x7f30400b6ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:31:53.207 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:53.202 7f304520f700 1 --2- 172.21.2.102:0/3514766872 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f30401a0580 0x7f3040086fb0 secure :-1 s=READY pgs=2201 cs=0 l=1 rev1=1 rx=0x7f3034009650 tx=0x7f303400b8c0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:53.207 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:53.202 7f3031ffb700 1 -- 172.21.2.102:0/3514766872 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3034006740 con 0x7f30401a0580 2021-02-10T11:31:53.207 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:53.202 7f3031ffb700 1 -- 172.21.2.102:0/3514766872 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3034006390 con 0x7f30401a0580 2021-02-10T11:31:53.207 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:53.202 7f3031ffb700 1 -- 172.21.2.102:0/3514766872 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f30340052e0 con 0x7f30401a0580 2021-02-10T11:31:53.207 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:53.202 7f30473f3700 1 -- 172.21.2.102:0/3514766872 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3040083a10 con 0x7f30401a0580 2021-02-10T11:31:53.207 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:53.202 7f30473f3700 1 -- 172.21.2.102:0/3514766872 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3040083a10 con 0x7f30401a0580 2021-02-10T11:31:53.207 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:53.203 7f3031ffb700 1 -- 172.21.2.102:0/3514766872 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f30340052e0 con 0x7f30401a0580 2021-02-10T11:31:53.207 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:53.203 7f3031ffb700 1 --2- 172.21.2.102:0/3514766872 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f30200186c0 0x7f302001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:53.207 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:53.203 7f3031ffb700 1 -- 172.21.2.102:0/3514766872 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f30340324e0 con 0x7f30401a0580 2021-02-10T11:31:53.207 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:53.203 7f3044a0e700 1 --2- 172.21.2.102:0/3514766872 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f30200186c0 0x7f302001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:53.207 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:53.204 7f3044a0e700 1 --2- 172.21.2.102:0/3514766872 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f30200186c0 0x7f302001abc0 crc :-1 s=READY pgs=973 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:53.235 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:53.234 7f30473f3700 1 -- 172.21.2.102:0/3514766872 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30400875d0 con 0x7f30401a0580 2021-02-10T11:31:53.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:53.236 7f3031ffb700 1 -- 172.21.2.102:0/3514766872 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f303401c960 con 0x7f30401a0580 2021-02-10T11:31:53.368 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:53.367 7f30473f3700 1 -- 172.21.2.102:0/3514766872 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f30400875d0 con 0x7f30401a0580 2021-02-10T11:31:53.371 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:53.370 7f3031ffb700 1 -- 172.21.2.102:0/3514766872 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.2 v65) v1 ==== 184+0+0 (secure 0 0 0) 0x7f303401c960 con 0x7f30401a0580 2021-02-10T11:31:53.375 INFO:teuthology.orchestra.run.gibba002.stderr:updated caps for client.2 2021-02-10T11:31:53.377 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:53.375 7f30473f3700 1 -- 172.21.2.102:0/3514766872 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f30200186c0 msgr2=0x7f302001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:53.377 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:53.375 7f30473f3700 1 --2- 172.21.2.102:0/3514766872 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f30200186c0 0x7f302001abc0 crc :-1 s=READY pgs=973 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:53.377 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:53.375 7f30473f3700 1 -- 172.21.2.102:0/3514766872 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f30401a0580 msgr2=0x7f3040086fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:53.377 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:53.375 7f30473f3700 1 --2- 172.21.2.102:0/3514766872 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f30401a0580 0x7f3040086fb0 secure :-1 s=READY pgs=2201 cs=0 l=1 rev1=1 rx=0x7f3034009650 tx=0x7f303400b8c0).stop 2021-02-10T11:31:53.377 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:53.376 7f30473f3700 1 -- 172.21.2.102:0/3514766872 shutdown_connections 2021-02-10T11:31:53.377 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:53.376 7f30473f3700 1 --2- 172.21.2.102:0/3514766872 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3040082bb0 0x7f3040082fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:53.377 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:53.376 7f30473f3700 1 --2- 172.21.2.102:0/3514766872 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f30200186c0 0x7f302001abc0 unknown :-1 s=CLOSED pgs=973 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:53.377 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:53.376 7f30473f3700 1 --2- 172.21.2.102:0/3514766872 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f30400b66b0 0x7f30400b6ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:53.377 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:53.376 7f30473f3700 1 --2- 172.21.2.102:0/3514766872 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f30401a0580 0x7f3040086fb0 unknown :-1 s=CLOSED pgs=2201 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:53.378 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:53.376 7f30473f3700 1 -- 172.21.2.102:0/3514766872 >> 172.21.2.102:0/3514766872 conn(0x7f30400afc60 msgr2=0x7f30400b50d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:53.378 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:53.376 7f30473f3700 1 -- 172.21.2.102:0/3514766872 shutdown_connections 2021-02-10T11:31:53.378 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:53.376 7f30473f3700 1 -- 172.21.2.102:0/3514766872 wait complete. 2021-02-10T11:31:53.385 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.3 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:31:53.827 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:53.809 7ff5ac655700 1 Processor -- start 2021-02-10T11:31:53.934 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:53.809 7ff5ac655700 1 -- start start 2021-02-10T11:31:53.934 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:53.809 7ff5ac655700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7ff5a40d1c60 0x7ff5a40cef50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:53.934 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:53.809 7ff5ac655700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff5a40cf570 0x7ff5a40cf970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:53.934 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:53.809 7ff5ac655700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5a40fe1b0 con 0x7ff5a40d1860 2021-02-10T11:31:53.934 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:53.809 7ff5ac655700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5a41b0ca0 con 0x7ff5a40d1c60 2021-02-10T11:31:53.934 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:53.809 7ff5ac655700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5a4109a50 con 0x7ff5a40cf570 2021-02-10T11:31:53.934 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:53.810 7ff5aa471700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7ff5a40d1860 0x7ff5a41b04b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:54528/0 (socket says 172.21.2.102:54528) 2021-02-10T11:31:53.934 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:53.810 7ff5aa471700 1 -- 172.21.2.102:0/2429311551 learned_addr learned my addr 172.21.2.102:0/2429311551 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:31:53.934 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:53.810 7ff5a9c70700 1 --2- 172.21.2.102:0/2429311551 >> v2:172.21.2.102:3300/0 conn(0x7ff5a40d1c60 0x7ff5a40cef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:53.935 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:53.810 7ff5a9c70700 1 -- 172.21.2.102:0/2429311551 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff5a40cf570 msgr2=0x7ff5a40cf970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:53.935 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:53.810 7ff5a9c70700 1 --2- 172.21.2.102:0/2429311551 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff5a40cf570 0x7ff5a40cf970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:53.935 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:53.810 7ff5a9c70700 1 -- 172.21.2.102:0/2429311551 >> v1:172.21.2.102:6789/0 conn(0x7ff5a40d1860 legacy=0x7ff5a41b04b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:53.935 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:53.810 7ff5a9c70700 1 -- 172.21.2.102:0/2429311551 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5a4088070 con 0x7ff5a40d1c60 2021-02-10T11:31:53.935 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:53.810 7ff5a9c70700 1 --2- 172.21.2.102:0/2429311551 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff5a40d1c60 0x7ff5a40cef50 secure :-1 s=READY pgs=2203 cs=0 l=1 rev1=1 rx=0x7ff5a0006650 tx=0x7ff5a0009a80).ready entity=mon.0 client_cookie=ff46dd7b0dd58cf server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:53.935 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:53.810 7ff5a946f700 1 -- 172.21.2.102:0/2429311551 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff5a000c0c0 con 0x7ff5a40d1c60 2021-02-10T11:31:53.935 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:53.810 7ff5a946f700 1 -- 172.21.2.102:0/2429311551 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5a000c340 con 0x7ff5a40d1c60 2021-02-10T11:31:53.935 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:53.810 7ff5a946f700 1 -- 172.21.2.102:0/2429311551 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff5a000c860 con 0x7ff5a40d1c60 2021-02-10T11:31:53.935 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:53.810 7ff5ac655700 1 -- 172.21.2.102:0/2429311551 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff5a40d1c60 msgr2=0x7ff5a40cef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:53.935 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:53.810 7ff5ac655700 1 --2- 172.21.2.102:0/2429311551 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff5a40d1c60 0x7ff5a40cef50 secure :-1 s=READY pgs=2203 cs=0 l=1 rev1=1 rx=0x7ff5a0006650 tx=0x7ff5a0009a80).stop 2021-02-10T11:31:53.935 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:53.811 7ff5ac655700 1 -- 172.21.2.102:0/2429311551 shutdown_connections 2021-02-10T11:31:53.935 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:53.811 7ff5ac655700 1 --2- 172.21.2.102:0/2429311551 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff5a40d1c60 0x7ff5a40cef50 unknown :-1 s=CLOSED pgs=2203 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:53.935 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:53.811 7ff5ac655700 1 --2- 172.21.2.102:0/2429311551 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff5a40cf570 0x7ff5a40cf970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:53.936 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:53.811 7ff5ac655700 1 -- 172.21.2.102:0/2429311551 >> 172.21.2.102:0/2429311551 conn(0x7ff5a40ce410 msgr2=0x7ff5a40d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:53.936 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:53.811 7ff5ac655700 1 -- 172.21.2.102:0/2429311551 shutdown_connections 2021-02-10T11:31:53.936 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:53.811 7ff5ac655700 1 -- 172.21.2.102:0/2429311551 wait complete. 2021-02-10T11:31:53.938 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:53.826 7ff5ac655700 1 Processor -- start 2021-02-10T11:31:53.938 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:53.826 7ff5ac655700 1 -- start start 2021-02-10T11:31:53.938 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:53.826 7ff5ac655700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff5a4099860 0x7ff5a4194f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:53.938 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:53.826 7ff5ac655700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff5a4099c60 0x7ff5a4195540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:53.938 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:53.826 7ff5ac655700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff5a418a540 0x7ff5a418a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:53.938 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:53.826 7ff5ac655700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5a4189fc0 con 0x7ff5a418a540 2021-02-10T11:31:53.938 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:53.826 7ff5ac655700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5a4090820 con 0x7ff5a4099c60 2021-02-10T11:31:53.938 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:53.826 7ff5ac655700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5a4090a00 con 0x7ff5a4099860 2021-02-10T11:31:53.938 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:53.826 7ff5aac72700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff5a418a540 0x7ff5a418a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:53.939 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:53.827 7ff5aac72700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff5a418a540 0x7ff5a418a940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:34076/0 (socket says 172.21.2.102:34076) 2021-02-10T11:31:53.939 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:53.827 7ff5aac72700 1 -- 172.21.2.102:0/3542097489 learned_addr learned my addr 172.21.2.102:0/3542097489 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:31:53.939 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:53.827 7ff5aac72700 1 -- 172.21.2.102:0/3542097489 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff5a4099860 msgr2=0x7ff5a4194f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:53.939 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:53.827 7ff5aac72700 1 --2- 172.21.2.102:0/3542097489 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff5a4099860 0x7ff5a4194f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:53.939 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:53.827 7ff5aac72700 1 -- 172.21.2.102:0/3542097489 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff5a4099c60 msgr2=0x7ff5a4195540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:53.939 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:53.827 7ff5aac72700 1 --2- 172.21.2.102:0/3542097489 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff5a4099c60 0x7ff5a4195540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:53.939 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:53.827 7ff5aac72700 1 -- 172.21.2.102:0/3542097489 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5a418af60 con 0x7ff5a418a540 2021-02-10T11:31:53.940 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:53.827 7ff5aac72700 1 --2- 172.21.2.102:0/3542097489 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff5a418a540 0x7ff5a418a940 secure :-1 s=READY pgs=2204 cs=0 l=1 rev1=1 rx=0x7ff59c009980 tx=0x7ff59c00bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:53.940 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:53.827 7ff596ffd700 1 -- 172.21.2.102:0/3542097489 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff59c005870 con 0x7ff5a418a540 2021-02-10T11:31:53.940 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:53.827 7ff596ffd700 1 -- 172.21.2.102:0/3542097489 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff59c005b30 con 0x7ff5a418a540 2021-02-10T11:31:53.940 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:53.827 7ff596ffd700 1 -- 172.21.2.102:0/3542097489 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff59c006030 con 0x7ff5a418a540 2021-02-10T11:31:53.940 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:53.827 7ff5ac655700 1 -- 172.21.2.102:0/3542097489 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5a418af60 con 0x7ff5a418a540 2021-02-10T11:31:53.940 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:53.827 7ff5ac655700 1 -- 172.21.2.102:0/3542097489 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5a418b4f0 con 0x7ff5a418a540 2021-02-10T11:31:53.941 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:53.828 7ff596ffd700 1 -- 172.21.2.102:0/3542097489 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ff59c005d10 con 0x7ff5a418a540 2021-02-10T11:31:53.941 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:53.828 7ff596ffd700 1 --2- 172.21.2.102:0/3542097489 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff58c0186c0 0x7ff58c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:53.941 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:53.828 7ff596ffd700 1 -- 172.21.2.102:0/3542097489 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5573+0+0 (secure 0 0 0) 0x7ff59c032800 con 0x7ff5a418a540 2021-02-10T11:31:53.941 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:53.828 7ff5aa471700 1 --2- 172.21.2.102:0/3542097489 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff58c0186c0 0x7ff58c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:53.941 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:53.828 7ff5aa471700 1 --2- 172.21.2.102:0/3542097489 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff58c0186c0 0x7ff58c01abc0 crc :-1 s=READY pgs=974 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:53.941 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:53.859 7ff5ac655700 1 -- 172.21.2.102:0/3542097489 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5a4082980 con 0x7ff5a418a540 2021-02-10T11:31:53.942 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:53.862 7ff596ffd700 1 -- 172.21.2.102:0/3542097489 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ff59c006290 con 0x7ff5a418a540 2021-02-10T11:31:53.991 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:53.990 7ff5ac655700 1 -- 172.21.2.102:0/3542097489 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.3", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7ff5a4082980 con 0x7ff5a418a540 2021-02-10T11:31:53.994 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:53.993 7ff596ffd700 1 -- 172.21.2.102:0/3542097489 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.3", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.3 v66) v1 ==== 184+0+0 (secure 0 0 0) 0x7ff59c006f20 con 0x7ff5a418a540 2021-02-10T11:31:53.999 INFO:teuthology.orchestra.run.gibba002.stderr:updated caps for client.3 2021-02-10T11:31:54.000 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:53.999 7ff5ac655700 1 -- 172.21.2.102:0/3542097489 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff58c0186c0 msgr2=0x7ff58c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:54.001 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:53.999 7ff5ac655700 1 --2- 172.21.2.102:0/3542097489 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff58c0186c0 0x7ff58c01abc0 crc :-1 s=READY pgs=974 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:54.001 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:53.999 7ff5ac655700 1 -- 172.21.2.102:0/3542097489 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff5a418a540 msgr2=0x7ff5a418a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:54.001 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:53.999 7ff5ac655700 1 --2- 172.21.2.102:0/3542097489 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff5a418a540 0x7ff5a418a940 secure :-1 s=READY pgs=2204 cs=0 l=1 rev1=1 rx=0x7ff59c009980 tx=0x7ff59c00bcd0).stop 2021-02-10T11:31:54.001 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:53.999 7ff5ac655700 1 -- 172.21.2.102:0/3542097489 shutdown_connections 2021-02-10T11:31:54.001 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:53.999 7ff5ac655700 1 --2- 172.21.2.102:0/3542097489 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff5a418a540 0x7ff5a418a940 unknown :-1 s=CLOSED pgs=2204 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:54.002 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:53.999 7ff5ac655700 1 --2- 172.21.2.102:0/3542097489 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff58c0186c0 0x7ff58c01abc0 unknown :-1 s=CLOSED pgs=974 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:54.002 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:53.999 7ff5ac655700 1 --2- 172.21.2.102:0/3542097489 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff5a4099c60 0x7ff5a4195540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:54.002 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:53.999 7ff5ac655700 1 --2- 172.21.2.102:0/3542097489 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff5a4099860 0x7ff5a4194f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:54.002 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:53.999 7ff5ac655700 1 -- 172.21.2.102:0/3542097489 >> 172.21.2.102:0/3542097489 conn(0x7ff5a419f820 msgr2=0x7ff5a41a5cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:54.002 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:53.999 7ff5ac655700 1 -- 172.21.2.102:0/3542097489 shutdown_connections 2021-02-10T11:31:54.002 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:53.999 7ff5ac655700 1 -- 172.21.2.102:0/3542097489 wait complete. 2021-02-10T11:31:54.011 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:31:54.444 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:54.425 7f87f0188700 1 Processor -- start 2021-02-10T11:31:54.444 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:54.425 7f87f0188700 1 -- start start 2021-02-10T11:31:54.445 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:54.425 7f87f0188700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f87e80d1c70 0x7f87e80f20a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:54.445 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:54.425 7f87f0188700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87e80f0b10 con 0x7f87e80d1870 2021-02-10T11:31:54.445 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:54.425 7f87f0188700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87e81b0be0 con 0x7f87e80d0340 2021-02-10T11:31:54.445 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:54.425 7f87f0188700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87e80e5ed0 con 0x7f87e80d1c70 2021-02-10T11:31:54.445 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:54.425 7f87edfa4700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f87e80d1870 0x7f87e81b03f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:54538/0 (socket says 172.21.2.102:54538) 2021-02-10T11:31:54.445 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:54.425 7f87edfa4700 1 -- 172.21.2.102:0/4228285498 learned_addr learned my addr 172.21.2.102:0/4228285498 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:31:54.446 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:54.426 7f87ecfa2700 1 -- 172.21.2.102:0/4228285498 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4263143088 0 0) 0x7f87e40010b0 con 0x7f87e80d1870 2021-02-10T11:31:54.446 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:54.426 7f87ecfa2700 1 -- 172.21.2.102:0/4228285498 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87d4002fe0 con 0x7f87e80d1870 2021-02-10T11:31:54.446 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:54.426 7f87ee7a5700 1 --2- 172.21.2.102:0/4228285498 >> v2:172.21.2.113:3300/0 conn(0x7f87e80d1c70 0x7f87e80f20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:54.446 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:54.426 7f87ecfa2700 1 -- 172.21.2.102:0/4228285498 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3975570954 0 0) 0x7f87e40010b0 con 0x7f87e80d1870 2021-02-10T11:31:54.446 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:54.426 7f87ecfa2700 1 -- 172.21.2.102:0/4228285498 >> v2:172.21.2.113:3300/0 conn(0x7f87e80d1c70 msgr2=0x7f87e80f20a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:54.446 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:54.426 7f87ecfa2700 1 --2- 172.21.2.102:0/4228285498 >> v2:172.21.2.113:3300/0 conn(0x7f87e80d1c70 0x7f87e80f20a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:54.447 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:54.426 7f87ecfa2700 1 -- 172.21.2.102:0/4228285498 >> v1:172.21.2.113:6789/0 conn(0x7f87e80d0340 legacy=0x7f87e80ef640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:54.447 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:54.426 7f87ecfa2700 1 -- 172.21.2.102:0/4228285498 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87e80efe30 con 0x7f87e80d1870 2021-02-10T11:31:54.447 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:54.426 7f87ecfa2700 1 -- 172.21.2.102:0/4228285498 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7f87e40015b0 con 0x7f87e80d1870 2021-02-10T11:31:54.447 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:54.426 7f87ecfa2700 1 -- 172.21.2.102:0/4228285498 >> v1:172.21.2.102:6789/0 conn(0x7f87e80d1870 legacy=0x7f87e81b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:54.447 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:54.426 7f87ecfa2700 1 --2- 172.21.2.102:0/4228285498 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f87d4003a70 0x7f87d4005f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:54.447 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:54.426 7f87ecfa2700 1 --2- 172.21.2.102:0/4228285498 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f87d40067a0 0x7f87d4008ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:54.447 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:54.426 7f87ecfa2700 1 --2- 172.21.2.102:0/4228285498 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f87d40095a0 0x7f87d400bac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:54.447 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:54.426 7f87ecfa2700 1 -- 172.21.2.102:0/4228285498 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87d400c310 con 0x7f87d40067a0 2021-02-10T11:31:54.447 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:54.426 7f87ecfa2700 1 -- 172.21.2.102:0/4228285498 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87d400c530 con 0x7f87d4003a70 2021-02-10T11:31:54.447 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:54.426 7f87ecfa2700 1 -- 172.21.2.102:0/4228285498 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87d400c750 con 0x7f87d40095a0 2021-02-10T11:31:54.447 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:54.426 7f87ed7a3700 1 --2- 172.21.2.102:0/4228285498 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f87d40067a0 0x7f87d4008ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:54.448 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:54.427 7f87ee7a5700 1 --2- 172.21.2.102:0/4228285498 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f87d40095a0 0x7f87d400bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:54.448 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:54.427 7f87edfa4700 1 --2- 172.21.2.102:0/4228285498 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f87d4003a70 0x7f87d4005f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:54.448 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:54.427 7f87ed7a3700 1 -- 172.21.2.102:0/4228285498 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f87d40095a0 msgr2=0x7f87d400bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:54.448 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:54.427 7f87ed7a3700 1 --2- 172.21.2.102:0/4228285498 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f87d40095a0 0x7f87d400bac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:54.448 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:54.427 7f87ed7a3700 1 -- 172.21.2.102:0/4228285498 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f87d4003a70 msgr2=0x7f87d4005f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:54.448 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:54.427 7f87ed7a3700 1 --2- 172.21.2.102:0/4228285498 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f87d4003a70 0x7f87d4005f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:54.448 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:54.427 7f87ed7a3700 1 -- 172.21.2.102:0/4228285498 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87d400c9f0 con 0x7f87d40067a0 2021-02-10T11:31:54.448 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:54.427 7f87ee7a5700 1 -- 172.21.2.102:0/4228285498 reap_dead start 2021-02-10T11:31:54.450 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:54.427 7f87ed7a3700 1 --2- 172.21.2.102:0/4228285498 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f87d40067a0 0x7f87d4008ca0 secure :-1 s=READY pgs=2206 cs=0 l=1 rev1=1 rx=0x7f87dc006ae0 tx=0x7f87dc04db00).ready entity=mon.0 client_cookie=a2dd09e82ff9494f server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:54.451 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:54.427 7f87ecfa2700 1 -- 172.21.2.102:0/4228285498 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f87dc0500c0 con 0x7f87d40067a0 2021-02-10T11:31:54.451 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:54.427 7f87ecfa2700 1 -- 172.21.2.102:0/4228285498 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87dc0502c0 con 0x7f87d40067a0 2021-02-10T11:31:54.451 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:54.427 7f87ecfa2700 1 -- 172.21.2.102:0/4228285498 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f87dc0507e0 con 0x7f87d40067a0 2021-02-10T11:31:54.451 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:54.427 7f87f0188700 1 -- 172.21.2.102:0/4228285498 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f87d40067a0 msgr2=0x7f87d4008ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:54.451 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:54.427 7f87f0188700 1 --2- 172.21.2.102:0/4228285498 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f87d40067a0 0x7f87d4008ca0 secure :-1 s=READY pgs=2206 cs=0 l=1 rev1=1 rx=0x7f87dc006ae0 tx=0x7f87dc04db00).stop 2021-02-10T11:31:54.451 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:54.428 7f87f0188700 1 -- 172.21.2.102:0/4228285498 shutdown_connections 2021-02-10T11:31:54.451 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:54.428 7f87f0188700 1 --2- 172.21.2.102:0/4228285498 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f87d40067a0 0x7f87d4008ca0 unknown :-1 s=CLOSED pgs=2206 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:54.451 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:54.428 7f87f0188700 1 -- 172.21.2.102:0/4228285498 >> 172.21.2.102:0/4228285498 conn(0x7f87e80ce450 msgr2=0x7f87e80d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:54.451 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:54.428 7f87f0188700 1 -- 172.21.2.102:0/4228285498 shutdown_connections 2021-02-10T11:31:54.452 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:54.428 7f87f0188700 1 -- 172.21.2.102:0/4228285498 wait complete. 2021-02-10T11:31:54.452 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:54.443 7f87f0188700 1 Processor -- start 2021-02-10T11:31:54.452 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:54.443 7f87f0188700 1 -- start start 2021-02-10T11:31:54.452 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:54.443 7f87f0188700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f87e8195650 0x7f87e80b5560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:54.452 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:54.443 7f87f0188700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f87e80b5b80 0x7f87e81a2e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:54.452 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:54.443 7f87f0188700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f87e80acfa0 0x7f87e81a3440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:54.452 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:54.443 7f87f0188700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87e80b6010 con 0x7f87e8195650 2021-02-10T11:31:54.452 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:54.443 7f87f0188700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87e81a3a60 con 0x7f87e80b5b80 2021-02-10T11:31:54.452 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:54.443 7f87f0188700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87e81a3c40 con 0x7f87e80acfa0 2021-02-10T11:31:54.452 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:54.443 7f87edfa4700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f87e8195650 0x7f87e80b5560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:54.452 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:54.443 7f87edfa4700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f87e8195650 0x7f87e80b5560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:34096/0 (socket says 172.21.2.102:34096) 2021-02-10T11:31:54.453 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:54.443 7f87edfa4700 1 -- 172.21.2.102:0/1679462269 learned_addr learned my addr 172.21.2.102:0/1679462269 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:31:54.453 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:54.444 7f87edfa4700 1 -- 172.21.2.102:0/1679462269 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f87e80acfa0 msgr2=0x7f87e81a3440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:54.453 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:54.444 7f87edfa4700 1 --2- 172.21.2.102:0/1679462269 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f87e80acfa0 0x7f87e81a3440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:54.453 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:54.444 7f87edfa4700 1 -- 172.21.2.102:0/1679462269 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f87e80b5b80 msgr2=0x7f87e81a2e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:54.453 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:54.444 7f87edfa4700 1 --2- 172.21.2.102:0/1679462269 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f87e80b5b80 0x7f87e81a2e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:54.453 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:54.444 7f87edfa4700 1 -- 172.21.2.102:0/1679462269 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87e80bcae0 con 0x7f87e8195650 2021-02-10T11:31:54.453 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:54.444 7f87edfa4700 1 --2- 172.21.2.102:0/1679462269 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f87e8195650 0x7f87e80b5560 secure :-1 s=READY pgs=2207 cs=0 l=1 rev1=1 rx=0x7f87e4001c00 tx=0x7f87e400a8e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:54.453 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:54.444 7f87da7fc700 1 -- 172.21.2.102:0/1679462269 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f87e40009b0 con 0x7f87e8195650 2021-02-10T11:31:54.453 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:54.444 7f87da7fc700 1 -- 172.21.2.102:0/1679462269 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87e400acd0 con 0x7f87e8195650 2021-02-10T11:31:54.453 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:54.444 7f87da7fc700 1 -- 172.21.2.102:0/1679462269 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f87e40045d0 con 0x7f87e8195650 2021-02-10T11:31:54.454 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:54.444 7f87f0188700 1 -- 172.21.2.102:0/1679462269 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87e80bcae0 con 0x7f87e8195650 2021-02-10T11:31:54.454 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:54.444 7f87f0188700 1 -- 172.21.2.102:0/1679462269 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87e80bcae0 con 0x7f87e8195650 2021-02-10T11:31:54.454 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:54.445 7f87da7fc700 1 -- 172.21.2.102:0/1679462269 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f87e4004350 con 0x7f87e8195650 2021-02-10T11:31:54.454 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:54.445 7f87da7fc700 1 --2- 172.21.2.102:0/1679462269 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f87d00186c0 0x7f87d001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:54.454 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:54.445 7f87da7fc700 1 -- 172.21.2.102:0/1679462269 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f87e40325e0 con 0x7f87e8195650 2021-02-10T11:31:54.454 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:54.445 7f87ed7a3700 1 --2- 172.21.2.102:0/1679462269 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f87d00186c0 0x7f87d001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:54.454 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:54.445 7f87ed7a3700 1 --2- 172.21.2.102:0/1679462269 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f87d00186c0 0x7f87d001abc0 crc :-1 s=READY pgs=975 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:54.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:54.475 7f87f0188700 1 -- 172.21.2.102:0/1679462269 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87e80b6010 con 0x7f87e8195650 2021-02-10T11:31:54.478 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:54.477 7f87da7fc700 1 -- 172.21.2.102:0/1679462269 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f87e4004140 con 0x7f87e8195650 2021-02-10T11:31:54.607 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:54.606 7f87f0188700 1 -- 172.21.2.102:0/1679462269 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f87e80b6010 con 0x7f87e8195650 2021-02-10T11:31:54.607 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:54.606 7f87da7fc700 1 -- 172.21.2.102:0/1679462269 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 91 v91) v1 ==== 94+0+2104 (secure 0 0 0) 0x7f87e400acd0 con 0x7f87e8195650 2021-02-10T11:31:54.608 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:31:54.608 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":91,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10404,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.102:6835/1661316191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6834","nonce":1661316191},{"type":"v1","addr":"172.21.2.102:6835","nonce":1661316191}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:31:48.259584","modified":"2021-02-10 11:31:50.267046","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10401},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10401":{"gid":10401,"name":"b","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.2.113:6833/313019011","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6832","nonce":313019011},{"type":"v1","addr":"172.21.2.113:6833","nonce":313019011}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[23],"metadata_pool":22,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":11}]} 2021-02-10T11:31:54.608 INFO:teuthology.orchestra.run.gibba002.stderr:dumped fsmap epoch 91 2021-02-10T11:31:54.609 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:54.608 7f87f0188700 1 -- 172.21.2.102:0/1679462269 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f87d00186c0 msgr2=0x7f87d001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:54.609 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:54.608 7f87f0188700 1 --2- 172.21.2.102:0/1679462269 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f87d00186c0 0x7f87d001abc0 crc :-1 s=READY pgs=975 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:54.610 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:54.608 7f87f0188700 1 -- 172.21.2.102:0/1679462269 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f87e8195650 msgr2=0x7f87e80b5560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:54.610 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:54.608 7f87f0188700 1 --2- 172.21.2.102:0/1679462269 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f87e8195650 0x7f87e80b5560 secure :-1 s=READY pgs=2207 cs=0 l=1 rev1=1 rx=0x7f87e4001c00 tx=0x7f87e400a8e0).stop 2021-02-10T11:31:54.610 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:54.608 7f87f0188700 1 -- 172.21.2.102:0/1679462269 shutdown_connections 2021-02-10T11:31:54.610 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:54.608 7f87f0188700 1 --2- 172.21.2.102:0/1679462269 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f87e80acfa0 0x7f87e81a3440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:54.610 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:54.608 7f87f0188700 1 --2- 172.21.2.102:0/1679462269 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f87d00186c0 0x7f87d001abc0 unknown :-1 s=CLOSED pgs=975 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:54.610 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:54.608 7f87f0188700 1 --2- 172.21.2.102:0/1679462269 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f87e80b5b80 0x7f87e81a2e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:54.610 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:54.608 7f87f0188700 1 --2- 172.21.2.102:0/1679462269 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f87e8195650 0x7f87e80b5560 unknown :-1 s=CLOSED pgs=2207 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:54.610 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:54.608 7f87f0188700 1 -- 172.21.2.102:0/1679462269 >> 172.21.2.102:0/1679462269 conn(0x7f87e80b3040 msgr2=0x7f87e80c3250 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:54.610 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:54.608 7f87f0188700 1 -- 172.21.2.102:0/1679462269 shutdown_connections 2021-02-10T11:31:54.610 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:54.608 7f87f0188700 1 -- 172.21.2.102:0/1679462269 wait complete. 2021-02-10T11:31:54.823 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 91, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:31:48.259584', 'modified': '2021-02-10 11:31:50.267046', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 10401}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10401': {'gid': 10401, 'name': 'b', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.113:6833/313019011', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.113:6832', 'nonce': 313019011}, {'type': 'v1', 'addr': '172.21.2.113:6833', 'nonce': 313019011}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [23], 'metadata_pool': 22, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:31:54.823 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:31:54.824 DEBUG:teuthology.orchestra.run.gibba013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:31:55.225 INFO:teuthology.orchestra.run.gibba013.stdout:{ 2021-02-10T11:31:55.225 INFO:teuthology.orchestra.run.gibba013.stdout: "cluster_fsid": "68db30ea-dbce-4aef-a6df-f01c6fea0513", 2021-02-10T11:31:55.225 INFO:teuthology.orchestra.run.gibba013.stdout: "whoami": 0, 2021-02-10T11:31:55.225 INFO:teuthology.orchestra.run.gibba013.stdout: "id": 10401, 2021-02-10T11:31:55.225 INFO:teuthology.orchestra.run.gibba013.stdout: "want_state": "up:active", 2021-02-10T11:31:55.225 INFO:teuthology.orchestra.run.gibba013.stdout: "state": "up:active", 2021-02-10T11:31:55.225 INFO:teuthology.orchestra.run.gibba013.stdout: "rank_uptime": 6.9631957010000001, 2021-02-10T11:31:55.226 INFO:teuthology.orchestra.run.gibba013.stdout: "mdsmap_epoch": 91, 2021-02-10T11:31:55.226 INFO:teuthology.orchestra.run.gibba013.stdout: "osdmap_epoch": 112, 2021-02-10T11:31:55.226 INFO:teuthology.orchestra.run.gibba013.stdout: "osdmap_epoch_barrier": 112, 2021-02-10T11:31:55.226 INFO:teuthology.orchestra.run.gibba013.stdout: "uptime": 12.609584864 2021-02-10T11:31:55.226 INFO:teuthology.orchestra.run.gibba013.stdout:} 2021-02-10T11:31:55.229 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "68db30ea-dbce-4aef-a6df-f01c6fea0513", "whoami": 0, "id": 10401, "want_state": "up:active", "state": "up:active", "rank_uptime": 6.9631957010000001, "mdsmap_epoch": 91, "osdmap_epoch": 112, "osdmap_epoch_barrier": 112, "uptime": 12.609584864 } 2021-02-10T11:31:55.229 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:31:55.656 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:55.639 7f7b463bb700 1 Processor -- start 2021-02-10T11:31:55.657 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:55.639 7f7b463bb700 1 -- start start 2021-02-10T11:31:55.657 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:55.639 7f7b463bb700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f7b400d1870 0x7f7b401b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:55.657 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:55.639 7f7b463bb700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f7b401b0a10 0x7f7b400cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:55.657 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:55.639 7f7b463bb700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7b400d1c70 0x7f7b400cf950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:55.657 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:55.639 7f7b463bb700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b400e5ed0 con 0x7f7b400d1870 2021-02-10T11:31:55.657 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:55.639 7f7b463bb700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b400f0b10 con 0x7f7b401b0a10 2021-02-10T11:31:55.657 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:55.639 7f7b463bb700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b400cff70 con 0x7f7b400d1c70 2021-02-10T11:31:55.657 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:55.639 7f7b3ffff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f7b400d1870 0x7f7b401b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:55.657 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:55.639 7f7b3ffff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f7b400d1870 0x7f7b401b03f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:34102/0 (socket says 172.21.2.102:34102) 2021-02-10T11:31:55.657 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:55.639 7f7b3ffff700 1 -- 172.21.2.102:0/2331711134 learned_addr learned my addr 172.21.2.102:0/2331711134 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:31:55.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:55.640 7f7b3ffff700 1 -- 172.21.2.102:0/2331711134 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7b400d1c70 msgr2=0x7f7b400cf950 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:31:55.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:55.640 7f7b3ffff700 1 --2- 172.21.2.102:0/2331711134 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7b400d1c70 0x7f7b400cf950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:55.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:55.640 7f7b3ffff700 1 -- 172.21.2.102:0/2331711134 >> v2:172.21.2.113:3300/0 conn(0x7f7b401b0a10 msgr2=0x7f7b400cf330 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:31:55.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:55.640 7f7b3ffff700 1 --2- 172.21.2.102:0/2331711134 >> v2:172.21.2.113:3300/0 conn(0x7f7b401b0a10 0x7f7b400cf330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:55.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:55.640 7f7b3ffff700 1 -- 172.21.2.102:0/2331711134 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b400d0150 con 0x7f7b400d1870 2021-02-10T11:31:55.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:55.640 7f7b3ffff700 1 --2- 172.21.2.102:0/2331711134 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7b400d1870 0x7f7b401b03f0 secure :-1 s=READY pgs=2208 cs=0 l=1 rev1=1 rx=0x7f7b30006700 tx=0x7f7b30009b00).ready entity=mon.0 client_cookie=ad602980d30bc152 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:55.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:55.640 7f7b3effd700 1 -- 172.21.2.102:0/2331711134 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7b3000c0c0 con 0x7f7b400d1870 2021-02-10T11:31:55.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:55.640 7f7b3effd700 1 -- 172.21.2.102:0/2331711134 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b3000c400 con 0x7f7b400d1870 2021-02-10T11:31:55.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:55.640 7f7b3effd700 1 -- 172.21.2.102:0/2331711134 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7b3000c920 con 0x7f7b400d1870 2021-02-10T11:31:55.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:55.640 7f7b463bb700 1 -- 172.21.2.102:0/2331711134 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7b400d1870 msgr2=0x7f7b401b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:55.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:55.640 7f7b463bb700 1 --2- 172.21.2.102:0/2331711134 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7b400d1870 0x7f7b401b03f0 secure :-1 s=READY pgs=2208 cs=0 l=1 rev1=1 rx=0x7f7b30006700 tx=0x7f7b30009b00).stop 2021-02-10T11:31:55.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:55.641 7f7b463bb700 1 -- 172.21.2.102:0/2331711134 shutdown_connections 2021-02-10T11:31:55.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:55.641 7f7b463bb700 1 --2- 172.21.2.102:0/2331711134 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7b400d1c70 0x7f7b400cf950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:55.659 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:55.641 7f7b463bb700 1 --2- 172.21.2.102:0/2331711134 >> v2:172.21.2.113:3300/0 conn(0x7f7b401b0a10 0x7f7b400cf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:55.659 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:55.641 7f7b463bb700 1 --2- 172.21.2.102:0/2331711134 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7b400d1870 0x7f7b401b03f0 unknown :-1 s=CLOSED pgs=2208 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:55.659 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:55.641 7f7b463bb700 1 -- 172.21.2.102:0/2331711134 >> 172.21.2.102:0/2331711134 conn(0x7f7b400ce450 msgr2=0x7f7b400d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:55.659 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:55.641 7f7b463bb700 1 -- 172.21.2.102:0/2331711134 shutdown_connections 2021-02-10T11:31:55.659 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:55.641 7f7b463bb700 1 -- 172.21.2.102:0/2331711134 wait complete. 2021-02-10T11:31:55.661 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:55.656 7f7b463bb700 1 Processor -- start 2021-02-10T11:31:55.661 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:55.656 7f7b463bb700 1 -- start start 2021-02-10T11:31:55.661 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:55.656 7f7b463bb700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7b401b8ad0 0x7f7b40187e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:55.661 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:55.656 7f7b463bb700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7b401b8ed0 0x7f7b400b1af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:55.661 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:55.656 7f7b463bb700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7b400b2110 0x7f7b401b7a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:55.661 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:55.656 7f7b463bb700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b40188470 con 0x7f7b400b2110 2021-02-10T11:31:55.661 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:55.656 7f7b463bb700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b4018be10 con 0x7f7b401b8ad0 2021-02-10T11:31:55.661 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:55.656 7f7b463bb700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b4018c030 con 0x7f7b401b8ed0 2021-02-10T11:31:55.661 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:55.656 7f7b449d8700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7b400b2110 0x7f7b401b7a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:55.661 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:55.656 7f7b449d8700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7b400b2110 0x7f7b401b7a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:34108/0 (socket says 172.21.2.102:34108) 2021-02-10T11:31:55.661 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:55.657 7f7b449d8700 1 -- 172.21.2.102:0/2920916539 learned_addr learned my addr 172.21.2.102:0/2920916539 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:31:55.661 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:55.657 7f7b3f7fe700 1 --2- 172.21.2.102:0/2920916539 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7b401b8ed0 0x7f7b400b1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:55.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:55.657 7f7b3ffff700 1 --2- 172.21.2.102:0/2920916539 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7b401b8ad0 0x7f7b40187e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:55.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:55.657 7f7b449d8700 1 -- 172.21.2.102:0/2920916539 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7b401b8ed0 msgr2=0x7f7b400b1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:55.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:55.657 7f7b449d8700 1 --2- 172.21.2.102:0/2920916539 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7b401b8ed0 0x7f7b400b1af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:55.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:55.657 7f7b449d8700 1 -- 172.21.2.102:0/2920916539 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7b401b8ad0 msgr2=0x7f7b40187e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:55.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:55.657 7f7b449d8700 1 --2- 172.21.2.102:0/2920916539 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7b401b8ad0 0x7f7b40187e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:55.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:55.657 7f7b449d8700 1 -- 172.21.2.102:0/2920916539 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b400b2510 con 0x7f7b400b2110 2021-02-10T11:31:55.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:55.657 7f7b3ffff700 1 --2- 172.21.2.102:0/2920916539 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7b401b8ad0 0x7f7b40187e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:31:55.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:55.657 7f7b449d8700 1 --2- 172.21.2.102:0/2920916539 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7b400b2110 0x7f7b401b7a00 secure :-1 s=READY pgs=2209 cs=0 l=1 rev1=1 rx=0x7f7b38009960 tx=0x7f7b3800bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:55.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:55.657 7f7b3cff9700 1 -- 172.21.2.102:0/2920916539 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7b38000f70 con 0x7f7b400b2110 2021-02-10T11:31:55.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:55.657 7f7b3cff9700 1 -- 172.21.2.102:0/2920916539 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b380030c0 con 0x7f7b400b2110 2021-02-10T11:31:55.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:55.657 7f7b463bb700 1 -- 172.21.2.102:0/2920916539 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b400b2510 con 0x7f7b400b2110 2021-02-10T11:31:55.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:55.657 7f7b463bb700 1 -- 172.21.2.102:0/2920916539 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b401b8020 con 0x7f7b400b2110 2021-02-10T11:31:55.663 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:55.657 7f7b3cff9700 1 -- 172.21.2.102:0/2920916539 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7b38001500 con 0x7f7b400b2110 2021-02-10T11:31:55.663 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:55.658 7f7b3cff9700 1 -- 172.21.2.102:0/2920916539 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f7b38001310 con 0x7f7b400b2110 2021-02-10T11:31:55.663 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:55.658 7f7b3cff9700 1 --2- 172.21.2.102:0/2920916539 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7b200186c0 0x7f7b2001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:55.663 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:55.658 7f7b3cff9700 1 -- 172.21.2.102:0/2920916539 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f7b38032950 con 0x7f7b400b2110 2021-02-10T11:31:55.663 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:55.658 7f7b3ffff700 1 --2- 172.21.2.102:0/2920916539 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7b200186c0 0x7f7b2001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:55.663 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:55.658 7f7b3ffff700 1 --2- 172.21.2.102:0/2920916539 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7b200186c0 0x7f7b2001abc0 crc :-1 s=READY pgs=976 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:55.691 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:55.689 7f7b463bb700 1 -- 172.21.2.102:0/2920916539 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b401b88b0 con 0x7f7b400b2110 2021-02-10T11:31:55.693 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:55.692 7f7b3cff9700 1 -- 172.21.2.102:0/2920916539 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f7b38000cc0 con 0x7f7b400b2110 2021-02-10T11:31:55.817 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:55.816 7f7b463bb700 1 -- 172.21.2.102:0/2920916539 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7b401b88b0 con 0x7f7b400b2110 2021-02-10T11:31:55.818 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:55.817 7f7b3cff9700 1 -- 172.21.2.102:0/2920916539 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v112) v1 ==== 74+0+14007 (secure 0 0 0) 0x7f7b38001310 con 0x7f7b400b2110 2021-02-10T11:31:55.818 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:31:55.819 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":112,"fsid":"68db30ea-dbce-4aef-a6df-f01c6fea0513","created":"2021-02-10 11:03:06.331557","modified":"2021-02-10 11:31:49.261779","last_up_change":"2021-02-10 11:03:32.690562","last_in_change":"2021-02-10 11:03:32.690562","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":23,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:03:37.322519","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":22,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:31:46.228075","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"111","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":23,"pool_name":"cephfs_data","create_time":"2021-02-10 11:31:47.252676","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"112","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"be691b4d-a039-408a-af80-9d53f02b6f1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6818","nonce":36088},{"type":"v1","addr":"172.21.2.102:6820","nonce":36088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6822","nonce":36088},{"type":"v1","addr":"172.21.2.102:6824","nonce":36088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6830","nonce":36088},{"type":"v1","addr":"172.21.2.102:6832","nonce":36088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6826","nonce":36088},{"type":"v1","addr":"172.21.2.102:6828","nonce":36088}]},"public_addr":"172.21.2.102:6820/36088","cluster_addr":"172.21.2.102:6824/36088","heartbeat_back_addr":"172.21.2.102:6832/36088","heartbeat_front_addr":"172.21.2.102:6828/36088","state":["exists","up"]},{"osd":1,"uuid":"64e3ccc8-9c69-4d7c-865e-af8e83822635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6819","nonce":36087},{"type":"v1","addr":"172.21.2.102:6821","nonce":36087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6823","nonce":36087},{"type":"v1","addr":"172.21.2.102:6825","nonce":36087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6831","nonce":36087},{"type":"v1","addr":"172.21.2.102:6833","nonce":36087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6827","nonce":36087},{"type":"v1","addr":"172.21.2.102:6829","nonce":36087}]},"public_addr":"172.21.2.102:6821/36087","cluster_addr":"172.21.2.102:6825/36087","heartbeat_back_addr":"172.21.2.102:6833/36087","heartbeat_front_addr":"172.21.2.102:6829/36087","state":["exists","up"]},{"osd":2,"uuid":"153700af-747f-481f-bdf4-d6072d3a842f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6810","nonce":36089},{"type":"v1","addr":"172.21.2.102:6811","nonce":36089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6812","nonce":36089},{"type":"v1","addr":"172.21.2.102:6813","nonce":36089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6816","nonce":36089},{"type":"v1","addr":"172.21.2.102:6817","nonce":36089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6814","nonce":36089},{"type":"v1","addr":"172.21.2.102:6815","nonce":36089}]},"public_addr":"172.21.2.102:6811/36089","cluster_addr":"172.21.2.102:6813/36089","heartbeat_back_addr":"172.21.2.102:6817/36089","heartbeat_front_addr":"172.21.2.102:6815/36089","state":["exists","up"]},{"osd":3,"uuid":"cbc678f4-83a9-45e6-8830-2f7c232b1ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6802","nonce":36090},{"type":"v1","addr":"172.21.2.102:6803","nonce":36090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6804","nonce":36090},{"type":"v1","addr":"172.21.2.102:6805","nonce":36090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6808","nonce":36090},{"type":"v1","addr":"172.21.2.102:6809","nonce":36090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6806","nonce":36090},{"type":"v1","addr":"172.21.2.102:6807","nonce":36090}]},"public_addr":"172.21.2.102:6803/36090","cluster_addr":"172.21.2.102:6805/36090","heartbeat_back_addr":"172.21.2.102:6809/36090","heartbeat_front_addr":"172.21.2.102:6807/36090","state":["exists","up"]},{"osd":4,"uuid":"c01e4008-b258-4f10-afd2-9e7f37214351","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6808","nonce":36368},{"type":"v1","addr":"172.21.2.113:6811","nonce":36368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6814","nonce":36368},{"type":"v1","addr":"172.21.2.113:6818","nonce":36368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6826","nonce":36368},{"type":"v1","addr":"172.21.2.113:6828","nonce":36368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6820","nonce":36368},{"type":"v1","addr":"172.21.2.113:6824","nonce":36368}]},"public_addr":"172.21.2.113:6811/36368","cluster_addr":"172.21.2.113:6818/36368","heartbeat_back_addr":"172.21.2.113:6828/36368","heartbeat_front_addr":"172.21.2.113:6824/36368","state":["exists","up"]},{"osd":5,"uuid":"551f6007-daf8-4a98-8ee0-c2b16bcccb5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6802","nonce":36366},{"type":"v1","addr":"172.21.2.113:6805","nonce":36366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6807","nonce":36366},{"type":"v1","addr":"172.21.2.113:6810","nonce":36366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6819","nonce":36366},{"type":"v1","addr":"172.21.2.113:6822","nonce":36366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6812","nonce":36366},{"type":"v1","addr":"172.21.2.113:6815","nonce":36366}]},"public_addr":"172.21.2.113:6805/36366","cluster_addr":"172.21.2.113:6810/36366","heartbeat_back_addr":"172.21.2.113:6822/36366","heartbeat_front_addr":"172.21.2.113:6815/36366","state":["exists","up"]},{"osd":6,"uuid":"bf67c040-0e50-48be-b102-c3943bc9ebef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6800","nonce":36367},{"type":"v1","addr":"172.21.2.113:6801","nonce":36367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6803","nonce":36367},{"type":"v1","addr":"172.21.2.113:6804","nonce":36367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6813","nonce":36367},{"type":"v1","addr":"172.21.2.113:6816","nonce":36367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6806","nonce":36367},{"type":"v1","addr":"172.21.2.113:6809","nonce":36367}]},"public_addr":"172.21.2.113:6801/36367","cluster_addr":"172.21.2.113:6804/36367","heartbeat_back_addr":"172.21.2.113:6816/36367","heartbeat_front_addr":"172.21.2.113:6809/36367","state":["exists","up"]},{"osd":7,"uuid":"f5461f80-018d-428f-be89-2eb2bcbe7bb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6817","nonce":36365},{"type":"v1","addr":"172.21.2.113:6821","nonce":36365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6823","nonce":36365},{"type":"v1","addr":"172.21.2.113:6825","nonce":36365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6830","nonce":36365},{"type":"v1","addr":"172.21.2.113:6831","nonce":36365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6827","nonce":36365},{"type":"v1","addr":"172.21.2.113:6829","nonce":36365}]},"public_addr":"172.21.2.113:6821/36365","cluster_addr":"172.21.2.113:6825/36365","heartbeat_back_addr":"172.21.2.113:6831/36365","heartbeat_front_addr":"172.21.2.113:6829/36365","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:31:55.820 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:55.818 7f7b463bb700 1 -- 172.21.2.102:0/2920916539 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7b200186c0 msgr2=0x7f7b2001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:55.820 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:55.818 7f7b463bb700 1 --2- 172.21.2.102:0/2920916539 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7b200186c0 0x7f7b2001abc0 crc :-1 s=READY pgs=976 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:55.820 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:55.818 7f7b463bb700 1 -- 172.21.2.102:0/2920916539 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7b400b2110 msgr2=0x7f7b401b7a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:55.820 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:55.818 7f7b463bb700 1 --2- 172.21.2.102:0/2920916539 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7b400b2110 0x7f7b401b7a00 secure :-1 s=READY pgs=2209 cs=0 l=1 rev1=1 rx=0x7f7b38009960 tx=0x7f7b3800bcf0).stop 2021-02-10T11:31:55.821 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:55.818 7f7b463bb700 1 -- 172.21.2.102:0/2920916539 shutdown_connections 2021-02-10T11:31:55.821 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:55.818 7f7b463bb700 1 --2- 172.21.2.102:0/2920916539 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7b400b2110 0x7f7b401b7a00 unknown :-1 s=CLOSED pgs=2209 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:55.821 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:55.818 7f7b463bb700 1 --2- 172.21.2.102:0/2920916539 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7b401b8ed0 0x7f7b400b1af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:55.821 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:55.818 7f7b463bb700 1 --2- 172.21.2.102:0/2920916539 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7b200186c0 0x7f7b2001abc0 unknown :-1 s=CLOSED pgs=976 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:55.821 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:55.818 7f7b463bb700 1 --2- 172.21.2.102:0/2920916539 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7b401b8ad0 0x7f7b40187e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:55.821 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:55.818 7f7b463bb700 1 -- 172.21.2.102:0/2920916539 >> 172.21.2.102:0/2920916539 conn(0x7f7b400ed810 msgr2=0x7f7b40094980 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:55.821 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:55.819 7f7b463bb700 1 -- 172.21.2.102:0/2920916539 shutdown_connections 2021-02-10T11:31:55.821 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:55.819 7f7b463bb700 1 -- 172.21.2.102:0/2920916539 wait complete. 2021-02-10T11:31:55.828 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:03:37.322519', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 22, 'pool_name': 'cephfs_metadata', 'create_time': '2021-02-10 11:31:46.228075', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '111', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 23, 'pool_name': 'cephfs_data', 'create_time': '2021-02-10 11:31:47.252676', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '112', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-02-10T11:31:55.829 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:31:56.259 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:56.242 7f048d4e5700 1 Processor -- start 2021-02-10T11:31:56.260 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:56.242 7f048d4e5700 1 -- start start 2021-02-10T11:31:56.260 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:56.242 7f048d4e5700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f04880d1870 0x7f04881b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:56.260 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:56.242 7f048d4e5700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f04880cff70 0x7f0488109130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:56.260 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:56.242 7f048d4e5700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04880fe1c0 con 0x7f04881b0ae0 2021-02-10T11:31:56.260 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:56.242 7f048d4e5700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0488109ad0 con 0x7f04880cff70 2021-02-10T11:31:56.260 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:56.242 7f048d4e5700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0488109750 con 0x7f04880d1870 2021-02-10T11:31:56.260 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:56.243 7f0487ac0700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f04880cff70 0x7f0488109130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:56.260 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:56.243 7f0486abe700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f04881b0ae0 0x7f0488108940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:54570/0 (socket says 172.21.2.102:54570) 2021-02-10T11:31:56.260 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:56.243 7f0486abe700 1 -- 172.21.2.102:0/1426103332 learned_addr learned my addr 172.21.2.102:0/1426103332 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:31:56.260 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:56.243 7f0487ac0700 1 -- 172.21.2.102:0/1426103332 >> v2:172.21.2.113:3300/0 conn(0x7f04880d1870 msgr2=0x7f04881b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:56.260 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:56.243 7f0487ac0700 1 --2- 172.21.2.102:0/1426103332 >> v2:172.21.2.113:3300/0 conn(0x7f04880d1870 0x7f04881b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:56.260 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:56.243 7f0487ac0700 1 -- 172.21.2.102:0/1426103332 >> v1:172.21.2.102:6789/0 conn(0x7f04881b0ae0 legacy=0x7f0488108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:56.260 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:56.243 7f0487ac0700 1 -- 172.21.2.102:0/1426103332 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0488088070 con 0x7f04880cff70 2021-02-10T11:31:56.261 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:56.243 7f0487ac0700 1 --2- 172.21.2.102:0/1426103332 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f04880cff70 0x7f0488109130 secure :-1 s=READY pgs=2211 cs=0 l=1 rev1=1 rx=0x7f0480006e30 tx=0x7f0480009e60).ready entity=mon.0 client_cookie=e428e6f96207eb25 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:56.261 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:56.243 7f04862bd700 1 -- 172.21.2.102:0/1426103332 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0480006a50 con 0x7f04880cff70 2021-02-10T11:31:56.261 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:56.244 7f04862bd700 1 -- 172.21.2.102:0/1426103332 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f048000b300 con 0x7f04880cff70 2021-02-10T11:31:56.261 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:56.244 7f04862bd700 1 -- 172.21.2.102:0/1426103332 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f048000b790 con 0x7f04880cff70 2021-02-10T11:31:56.261 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:56.244 7f048d4e5700 1 -- 172.21.2.102:0/1426103332 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f04880cff70 msgr2=0x7f0488109130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:56.261 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:56.244 7f048d4e5700 1 --2- 172.21.2.102:0/1426103332 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f04880cff70 0x7f0488109130 secure :-1 s=READY pgs=2211 cs=0 l=1 rev1=1 rx=0x7f0480006e30 tx=0x7f0480009e60).stop 2021-02-10T11:31:56.261 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:56.244 7f048d4e5700 1 -- 172.21.2.102:0/1426103332 shutdown_connections 2021-02-10T11:31:56.261 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:56.244 7f048d4e5700 1 --2- 172.21.2.102:0/1426103332 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f04880cff70 0x7f0488109130 unknown :-1 s=CLOSED pgs=2211 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:56.261 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:56.244 7f048d4e5700 1 --2- 172.21.2.102:0/1426103332 >> v2:172.21.2.113:3300/0 conn(0x7f04880d1870 0x7f04881b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:56.261 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:56.244 7f048d4e5700 1 -- 172.21.2.102:0/1426103332 >> 172.21.2.102:0/1426103332 conn(0x7f04880ce420 msgr2=0x7f04880d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:56.261 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:56.244 7f048d4e5700 1 -- 172.21.2.102:0/1426103332 shutdown_connections 2021-02-10T11:31:56.261 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:56.244 7f048d4e5700 1 -- 172.21.2.102:0/1426103332 wait complete. 2021-02-10T11:31:56.263 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:56.259 7f048d4e5700 1 Processor -- start 2021-02-10T11:31:56.263 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:56.259 7f048d4e5700 1 -- start start 2021-02-10T11:31:56.263 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:56.259 7f048d4e5700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f04880c3820 0x7f0488084cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:56.263 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:56.259 7f048d4e5700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f04880a0290 0x7f04880852d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:56.263 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:56.259 7f048d4e5700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0488086270 0x7f0488086670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:56.263 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:56.259 7f048d4e5700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04880c0700 con 0x7f04880c3820 2021-02-10T11:31:56.263 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:56.259 7f048d4e5700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04880858f0 con 0x7f0488086270 2021-02-10T11:31:56.263 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:56.259 7f048d4e5700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0488085b10 con 0x7f04880a0290 2021-02-10T11:31:56.263 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:56.259 7f04872bf700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f04880c3820 0x7f0488084cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:56.264 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:56.259 7f04872bf700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f04880c3820 0x7f0488084cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:34122/0 (socket says 172.21.2.102:34122) 2021-02-10T11:31:56.264 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:56.259 7f04872bf700 1 -- 172.21.2.102:0/2369140944 learned_addr learned my addr 172.21.2.102:0/2369140944 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:31:56.264 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:56.260 7f04872bf700 1 -- 172.21.2.102:0/2369140944 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f04880a0290 msgr2=0x7f04880852d0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:31:56.264 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:56.260 7f04872bf700 1 --2- 172.21.2.102:0/2369140944 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f04880a0290 0x7f04880852d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:56.264 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:56.260 7f04872bf700 1 -- 172.21.2.102:0/2369140944 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0488086270 msgr2=0x7f0488086670 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:31:56.264 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:56.260 7f04872bf700 1 --2- 172.21.2.102:0/2369140944 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0488086270 0x7f0488086670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:56.264 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:56.260 7f04872bf700 1 -- 172.21.2.102:0/2369140944 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0488086c90 con 0x7f04880c3820 2021-02-10T11:31:56.264 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:56.260 7f04872bf700 1 --2- 172.21.2.102:0/2369140944 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f04880c3820 0x7f0488084cb0 secure :-1 s=READY pgs=2212 cs=0 l=1 rev1=1 rx=0x7f0478006f00 tx=0x7f047800b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:56.265 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:56.260 7f047ffff700 1 -- 172.21.2.102:0/2369140944 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0478006350 con 0x7f04880c3820 2021-02-10T11:31:56.265 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:56.260 7f047ffff700 1 -- 172.21.2.102:0/2369140944 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f047800be00 con 0x7f04880c3820 2021-02-10T11:31:56.265 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:56.260 7f047ffff700 1 -- 172.21.2.102:0/2369140944 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0478002020 con 0x7f04880c3820 2021-02-10T11:31:56.265 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:56.260 7f048d4e5700 1 -- 172.21.2.102:0/2369140944 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0488086c90 con 0x7f04880c3820 2021-02-10T11:31:56.265 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:56.260 7f048d4e5700 1 -- 172.21.2.102:0/2369140944 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0488086c90 con 0x7f04880c3820 2021-02-10T11:31:56.265 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:56.261 7f047ffff700 1 -- 172.21.2.102:0/2369140944 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f0478005310 con 0x7f04880c3820 2021-02-10T11:31:56.265 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:56.261 7f047ffff700 1 --2- 172.21.2.102:0/2369140944 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f046c0186c0 0x7f046c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:56.265 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:56.261 7f047ffff700 1 -- 172.21.2.102:0/2369140944 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f0478032790 con 0x7f04880c3820 2021-02-10T11:31:56.265 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:56.261 7f0486abe700 1 --2- 172.21.2.102:0/2369140944 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f046c0186c0 0x7f046c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:56.265 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:56.261 7f0486abe700 1 --2- 172.21.2.102:0/2369140944 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f046c0186c0 0x7f046c01abc0 crc :-1 s=READY pgs=977 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:56.294 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:56.292 7f048d4e5700 1 -- 172.21.2.102:0/2369140944 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0488082a70 con 0x7f04880c3820 2021-02-10T11:31:56.296 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:56.295 7f047ffff700 1 -- 172.21.2.102:0/2369140944 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f0478006b90 con 0x7f04880c3820 2021-02-10T11:31:56.422 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:56.421 7f048d4e5700 1 -- 172.21.2.102:0/2369140944 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f0488082a70 con 0x7f04880c3820 2021-02-10T11:31:56.423 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:56.422 7f047ffff700 1 -- 172.21.2.102:0/2369140944 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v112) v1 ==== 92+0+10 (secure 0 0 0) 0x7f047800be00 con 0x7f04880c3820 2021-02-10T11:31:56.423 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:31:56.425 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:56.424 7f048d4e5700 1 -- 172.21.2.102:0/2369140944 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f046c0186c0 msgr2=0x7f046c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:56.425 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:56.424 7f048d4e5700 1 --2- 172.21.2.102:0/2369140944 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f046c0186c0 0x7f046c01abc0 crc :-1 s=READY pgs=977 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:56.425 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:56.424 7f048d4e5700 1 -- 172.21.2.102:0/2369140944 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f04880c3820 msgr2=0x7f0488084cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:56.425 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:56.424 7f048d4e5700 1 --2- 172.21.2.102:0/2369140944 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f04880c3820 0x7f0488084cb0 secure :-1 s=READY pgs=2212 cs=0 l=1 rev1=1 rx=0x7f0478006f00 tx=0x7f047800b940).stop 2021-02-10T11:31:56.425 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:56.424 7f048d4e5700 1 -- 172.21.2.102:0/2369140944 shutdown_connections 2021-02-10T11:31:56.425 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:56.424 7f048d4e5700 1 --2- 172.21.2.102:0/2369140944 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f046c0186c0 0x7f046c01abc0 unknown :-1 s=CLOSED pgs=977 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:56.425 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:56.424 7f048d4e5700 1 --2- 172.21.2.102:0/2369140944 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0488086270 0x7f0488086670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:56.425 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:56.424 7f048d4e5700 1 --2- 172.21.2.102:0/2369140944 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f04880a0290 0x7f04880852d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:56.425 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:56.424 7f048d4e5700 1 --2- 172.21.2.102:0/2369140944 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f04880c3820 0x7f0488084cb0 unknown :-1 s=CLOSED pgs=2212 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:56.425 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:56.424 7f048d4e5700 1 -- 172.21.2.102:0/2369140944 >> 172.21.2.102:0/2369140944 conn(0x7f04880872c0 msgr2=0x7f0488095700 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:56.426 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:56.424 7f048d4e5700 1 -- 172.21.2.102:0/2369140944 shutdown_connections 2021-02-10T11:31:56.426 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:56.424 7f048d4e5700 1 -- 172.21.2.102:0/2369140944 wait complete. 2021-02-10T11:31:56.434 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-02-10T11:31:56.866 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:56.847 7f1224b46700 1 Processor -- start 2021-02-10T11:31:56.867 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:56.847 7f1224b46700 1 -- start start 2021-02-10T11:31:56.867 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:56.848 7f1224b46700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f12200d1c70 0x7f12200cef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:56.867 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:56.848 7f1224b46700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f12200cf580 0x7f12200cf980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:56.867 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:56.848 7f1224b46700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12200fe1c0 con 0x7f12200d1870 2021-02-10T11:31:56.867 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:56.848 7f1224b46700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12201b0cb0 con 0x7f12200cf580 2021-02-10T11:31:56.867 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:56.848 7f1224b46700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1220109a60 con 0x7f12200d1c70 2021-02-10T11:31:56.867 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:56.849 7f121e85e700 1 --1- >> v1:172.21.2.113:6789/0 conn(0x7f12200d1870 0x7f12201b04c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.113:6789/0 says I am v1:172.21.2.102:33674/0 (socket says 172.21.2.102:33674) 2021-02-10T11:31:56.867 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:56.849 7f121e85e700 1 -- 172.21.2.102:0/968527798 learned_addr learned my addr 172.21.2.102:0/968527798 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:31:56.867 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:56.849 7f121f05f700 1 --2- 172.21.2.102:0/968527798 >> v2:172.21.2.113:3300/0 conn(0x7f12200cf580 0x7f12200cf980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:56.867 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:56.849 7f121e05d700 1 --2- 172.21.2.102:0/968527798 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f12200d1c70 0x7f12200cef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:56.867 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:56.849 7f121e05d700 1 -- 172.21.2.102:0/968527798 >> v2:172.21.2.113:3300/0 conn(0x7f12200cf580 msgr2=0x7f12200cf980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:56.867 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:56.849 7f121e05d700 1 --2- 172.21.2.102:0/968527798 >> v2:172.21.2.113:3300/0 conn(0x7f12200cf580 0x7f12200cf980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:56.868 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:56.849 7f121e05d700 1 -- 172.21.2.102:0/968527798 >> v1:172.21.2.113:6789/0 conn(0x7f12200d1870 legacy=0x7f12201b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:56.868 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:56.849 7f121e05d700 1 -- 172.21.2.102:0/968527798 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1220088070 con 0x7f12200d1c70 2021-02-10T11:31:56.868 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:56.849 7f121e05d700 1 --2- 172.21.2.102:0/968527798 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f12200d1c70 0x7f12200cef60 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7f1214006650 tx=0x7f1214009a80).ready entity=mon.2 client_cookie=c0bdf2a8ea18fcb server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:56.868 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:56.850 7f121d85c700 1 -- 172.21.2.102:0/968527798 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f12140009b0 con 0x7f12200d1c70 2021-02-10T11:31:56.868 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:56.850 7f121d85c700 1 -- 172.21.2.102:0/968527798 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f121400c0c0 con 0x7f12200d1c70 2021-02-10T11:31:56.868 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:56.850 7f121d85c700 1 -- 172.21.2.102:0/968527798 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f121400c5e0 con 0x7f12200d1c70 2021-02-10T11:31:56.868 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:56.850 7f1224b46700 1 -- 172.21.2.102:0/968527798 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f12200d1c70 msgr2=0x7f12200cef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:56.868 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:56.850 7f1224b46700 1 --2- 172.21.2.102:0/968527798 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f12200d1c70 0x7f12200cef60 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7f1214006650 tx=0x7f1214009a80).stop 2021-02-10T11:31:56.868 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:56.851 7f1224b46700 1 -- 172.21.2.102:0/968527798 shutdown_connections 2021-02-10T11:31:56.868 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:56.851 7f1224b46700 1 --2- 172.21.2.102:0/968527798 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f12200d1c70 0x7f12200cef60 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:56.868 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:56.851 7f1224b46700 1 --2- 172.21.2.102:0/968527798 >> v2:172.21.2.113:3300/0 conn(0x7f12200cf580 0x7f12200cf980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:56.868 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:56.851 7f1224b46700 1 -- 172.21.2.102:0/968527798 >> 172.21.2.102:0/968527798 conn(0x7f12200ce420 msgr2=0x7f12200d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:56.868 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:56.851 7f1224b46700 1 -- 172.21.2.102:0/968527798 shutdown_connections 2021-02-10T11:31:56.869 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:56.851 7f1224b46700 1 -- 172.21.2.102:0/968527798 wait complete. 2021-02-10T11:31:56.870 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:56.866 7f1224b46700 1 Processor -- start 2021-02-10T11:31:56.870 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:56.866 7f1224b46700 1 -- start start 2021-02-10T11:31:56.870 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:56.866 7f1224b46700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1220082ca0 0x7f12200aa200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:56.870 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:56.866 7f1224b46700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f12200830a0 0x7f122008a710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:56.870 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:56.866 7f1224b46700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f12200aa820 0x7f122008ad30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:56.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:56.866 7f1224b46700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12200b5e70 con 0x7f12200aa820 2021-02-10T11:31:56.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:56.866 7f1224b46700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1220083650 con 0x7f1220082ca0 2021-02-10T11:31:56.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:56.866 7f1224b46700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1220198fa0 con 0x7f12200830a0 2021-02-10T11:31:56.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:56.867 7f121f05f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f12200aa820 0x7f122008ad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:56.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:56.867 7f121f05f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f12200aa820 0x7f122008ad30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:34134/0 (socket says 172.21.2.102:34134) 2021-02-10T11:31:56.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:56.867 7f121f05f700 1 -- 172.21.2.102:0/2757715938 learned_addr learned my addr 172.21.2.102:0/2757715938 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:31:56.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:56.867 7f121f05f700 1 -- 172.21.2.102:0/2757715938 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f12200830a0 msgr2=0x7f122008a710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:56.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:56.867 7f121e05d700 1 --2- 172.21.2.102:0/2757715938 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f12200830a0 0x7f122008a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:56.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:56.867 7f121e85e700 1 --2- 172.21.2.102:0/2757715938 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1220082ca0 0x7f12200aa200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:56.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:56.867 7f121f05f700 1 --2- 172.21.2.102:0/2757715938 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f12200830a0 0x7f122008a710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:56.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:56.867 7f121f05f700 1 -- 172.21.2.102:0/2757715938 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1220082ca0 msgr2=0x7f12200aa200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:56.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:56.867 7f121f05f700 1 --2- 172.21.2.102:0/2757715938 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1220082ca0 0x7f12200aa200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:56.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:56.867 7f121f05f700 1 -- 172.21.2.102:0/2757715938 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12200a9cc0 con 0x7f12200aa820 2021-02-10T11:31:56.872 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:56.867 7f121e05d700 1 --2- 172.21.2.102:0/2757715938 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f12200830a0 0x7f122008a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:31:56.872 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:56.867 7f121e85e700 1 --2- 172.21.2.102:0/2757715938 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1220082ca0 0x7f12200aa200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:31:56.872 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:56.867 7f121f05f700 1 --2- 172.21.2.102:0/2757715938 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f12200aa820 0x7f122008ad30 secure :-1 s=READY pgs=2213 cs=0 l=1 rev1=1 rx=0x7f121800b970 tx=0x7f121800bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:56.872 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:56.868 7f120f7fe700 1 -- 172.21.2.102:0/2757715938 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1218000f90 con 0x7f12200aa820 2021-02-10T11:31:56.872 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:56.868 7f120f7fe700 1 -- 172.21.2.102:0/2757715938 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1218003030 con 0x7f12200aa820 2021-02-10T11:31:56.872 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:56.868 7f120f7fe700 1 -- 172.21.2.102:0/2757715938 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1218001400 con 0x7f12200aa820 2021-02-10T11:31:56.872 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:56.868 7f1224b46700 1 -- 172.21.2.102:0/2757715938 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12200a9cc0 con 0x7f12200aa820 2021-02-10T11:31:56.872 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:56.868 7f1224b46700 1 -- 172.21.2.102:0/2757715938 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12200a9cc0 con 0x7f12200aa820 2021-02-10T11:31:56.872 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:56.868 7f120f7fe700 1 -- 172.21.2.102:0/2757715938 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f12180011f0 con 0x7f12200aa820 2021-02-10T11:31:56.872 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:56.868 7f120f7fe700 1 --2- 172.21.2.102:0/2757715938 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f12040186c0 0x7f120401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:56.872 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:56.868 7f120f7fe700 1 -- 172.21.2.102:0/2757715938 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f1218032980 con 0x7f12200aa820 2021-02-10T11:31:56.872 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:56.868 7f121e85e700 1 --2- 172.21.2.102:0/2757715938 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f12040186c0 0x7f120401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:56.872 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:56.869 7f121e85e700 1 --2- 172.21.2.102:0/2757715938 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f12040186c0 0x7f120401abc0 crc :-1 s=READY pgs=978 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:56.899 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:56.897 7f1224b46700 1 -- 172.21.2.102:0/2757715938 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1220198f70 con 0x7f12200aa820 2021-02-10T11:31:56.901 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:56.900 7f120f7fe700 1 -- 172.21.2.102:0/2757715938 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f12180066f0 con 0x7f12200aa820 2021-02-10T11:31:57.029 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:57.027 7f1224b46700 1 -- 172.21.2.102:0/2757715938 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7f1220198f70 con 0x7f12200aa820 2021-02-10T11:31:57.029 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:57.028 7f120f7fe700 1 -- 172.21.2.102:0/2757715938 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v112) v1 ==== 104+0+10 (secure 0 0 0) 0x7f12180011f0 con 0x7f12200aa820 2021-02-10T11:31:57.030 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:31:57.031 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:57.030 7f1224b46700 1 -- 172.21.2.102:0/2757715938 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f12040186c0 msgr2=0x7f120401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:57.031 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:57.030 7f1224b46700 1 --2- 172.21.2.102:0/2757715938 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f12040186c0 0x7f120401abc0 crc :-1 s=READY pgs=978 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:57.031 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:57.030 7f1224b46700 1 -- 172.21.2.102:0/2757715938 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f12200aa820 msgr2=0x7f122008ad30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:57.032 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:57.030 7f1224b46700 1 --2- 172.21.2.102:0/2757715938 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f12200aa820 0x7f122008ad30 secure :-1 s=READY pgs=2213 cs=0 l=1 rev1=1 rx=0x7f121800b970 tx=0x7f121800bd10).stop 2021-02-10T11:31:57.032 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:57.030 7f1224b46700 1 -- 172.21.2.102:0/2757715938 shutdown_connections 2021-02-10T11:31:57.032 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:57.030 7f1224b46700 1 --2- 172.21.2.102:0/2757715938 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f12200aa820 0x7f122008ad30 unknown :-1 s=CLOSED pgs=2213 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:57.032 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:57.030 7f1224b46700 1 --2- 172.21.2.102:0/2757715938 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f12200830a0 0x7f122008a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:57.032 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:57.030 7f1224b46700 1 --2- 172.21.2.102:0/2757715938 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f12040186c0 0x7f120401abc0 unknown :-1 s=CLOSED pgs=978 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:57.032 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:57.030 7f1224b46700 1 --2- 172.21.2.102:0/2757715938 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1220082ca0 0x7f12200aa200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:57.032 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:57.030 7f1224b46700 1 -- 172.21.2.102:0/2757715938 >> 172.21.2.102:0/2757715938 conn(0x7f12200a02d0 msgr2=0x7f122009f200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:57.032 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:57.030 7f1224b46700 1 -- 172.21.2.102:0/2757715938 shutdown_connections 2021-02-10T11:31:57.032 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:57.030 7f1224b46700 1 -- 172.21.2.102:0/2757715938 wait complete. 2021-02-10T11:31:57.040 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-02-10T11:31:57.474 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:57.454 7faa51b81700 1 Processor -- start 2021-02-10T11:31:57.474 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:57.454 7faa51b81700 1 -- start start 2021-02-10T11:31:57.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:57.455 7faa51b81700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7faa4c0d1c70 0x7faa4c0cef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:57.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:57.455 7faa51b81700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa4c0d0170 con 0x7faa4c0d1870 2021-02-10T11:31:57.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:57.455 7faa51b81700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa4c0fe1c0 con 0x7faa4c0cf580 2021-02-10T11:31:57.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:57.455 7faa51b81700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa4c1b0cb0 con 0x7faa4c0d1c70 2021-02-10T11:31:57.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:57.455 7faa4b7fe700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7faa4c0d1870 0x7faa4c1b04c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:54594/0 (socket says 172.21.2.102:54594) 2021-02-10T11:31:57.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:57.455 7faa4b7fe700 1 -- 172.21.2.102:0/2673068647 learned_addr learned my addr 172.21.2.102:0/2673068647 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:31:57.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:57.456 7faa4a7fc700 1 -- 172.21.2.102:0/2673068647 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1696044655 0 0) 0x7faa3c000ce0 con 0x7faa4c0d1870 2021-02-10T11:31:57.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:57.456 7faa4a7fc700 1 -- 172.21.2.102:0/2673068647 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa38002fe0 con 0x7faa4c0d1870 2021-02-10T11:31:57.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:57.456 7faa4affd700 1 --2- 172.21.2.102:0/2673068647 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7faa4c0d1c70 0x7faa4c0cef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:57.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:57.456 7faa4a7fc700 1 -- 172.21.2.102:0/2673068647 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3938035768 0 0) 0x7faa3c001180 con 0x7faa4c0d1870 2021-02-10T11:31:57.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:57.456 7faa4a7fc700 1 -- 172.21.2.102:0/2673068647 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7faa4c0d1c70 msgr2=0x7faa4c0cef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:57.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:57.456 7faa4a7fc700 1 --2- 172.21.2.102:0/2673068647 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7faa4c0d1c70 0x7faa4c0cef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:57.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:57.456 7faa4a7fc700 1 -- 172.21.2.102:0/2673068647 >> v1:172.21.2.113:6789/0 conn(0x7faa4c0cf580 legacy=0x7faa4c0cf980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:57.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:57.456 7faa4a7fc700 1 -- 172.21.2.102:0/2673068647 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa4c109a60 con 0x7faa4c0d1870 2021-02-10T11:31:57.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:57.456 7faa4a7fc700 1 -- 172.21.2.102:0/2673068647 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7faa3c001680 con 0x7faa4c0d1870 2021-02-10T11:31:57.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:57.456 7faa4a7fc700 1 -- 172.21.2.102:0/2673068647 >> v1:172.21.2.102:6789/0 conn(0x7faa4c0d1870 legacy=0x7faa4c1b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:57.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:57.456 7faa4a7fc700 1 --2- 172.21.2.102:0/2673068647 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7faa38003ac0 0x7faa38005f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:57.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:57.456 7faa4a7fc700 1 --2- 172.21.2.102:0/2673068647 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7faa380067b0 0x7faa38008cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:57.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:57.456 7faa4a7fc700 1 --2- 172.21.2.102:0/2673068647 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7faa38009590 0x7faa3800ba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:57.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:57.456 7faa4a7fc700 1 -- 172.21.2.102:0/2673068647 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa3800c2d0 con 0x7faa38003ac0 2021-02-10T11:31:57.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:57.456 7faa4a7fc700 1 -- 172.21.2.102:0/2673068647 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa3800c4f0 con 0x7faa380067b0 2021-02-10T11:31:57.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:57.456 7faa4a7fc700 1 -- 172.21.2.102:0/2673068647 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa3800c710 con 0x7faa38009590 2021-02-10T11:31:57.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:57.456 7faa4b7fe700 1 --2- 172.21.2.102:0/2673068647 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7faa38003ac0 0x7faa38005f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:57.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:57.456 7faa4affd700 1 --2- 172.21.2.102:0/2673068647 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7faa380067b0 0x7faa38008cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:57.477 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:57.457 7faa4bfff700 1 --2- 172.21.2.102:0/2673068647 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7faa38009590 0x7faa3800ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:57.477 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:57.457 7faa4b7fe700 1 -- 172.21.2.102:0/2673068647 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7faa38009590 msgr2=0x7faa3800ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:57.477 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:57.457 7faa4b7fe700 1 --2- 172.21.2.102:0/2673068647 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7faa38009590 0x7faa3800ba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:57.477 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:57.457 7faa4b7fe700 1 -- 172.21.2.102:0/2673068647 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7faa380067b0 msgr2=0x7faa38008cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:57.477 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:57.457 7faa4b7fe700 1 --2- 172.21.2.102:0/2673068647 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7faa380067b0 0x7faa38008cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:57.477 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:57.457 7faa4b7fe700 1 -- 172.21.2.102:0/2673068647 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa38050980 con 0x7faa38003ac0 2021-02-10T11:31:57.477 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:57.457 7faa4bfff700 1 --2- 172.21.2.102:0/2673068647 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7faa38009590 0x7faa3800ba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:31:57.477 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:57.457 7faa4b7fe700 1 --2- 172.21.2.102:0/2673068647 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7faa38003ac0 0x7faa38005f10 secure :-1 s=READY pgs=2215 cs=0 l=1 rev1=1 rx=0x7faa3c000b60 tx=0x7faa3c007da0).ready entity=mon.0 client_cookie=2a49f3f411370f35 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:57.477 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:57.457 7faa4a7fc700 1 -- 172.21.2.102:0/2673068647 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faa3c0070c0 con 0x7faa38003ac0 2021-02-10T11:31:57.477 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:57.457 7faa4a7fc700 1 -- 172.21.2.102:0/2673068647 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa3c0043a0 con 0x7faa38003ac0 2021-02-10T11:31:57.477 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:57.457 7faa4a7fc700 1 -- 172.21.2.102:0/2673068647 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faa3c0048c0 con 0x7faa38003ac0 2021-02-10T11:31:57.477 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:57.457 7faa51b81700 1 -- 172.21.2.102:0/2673068647 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7faa38003ac0 msgr2=0x7faa38005f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:57.477 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:57.457 7faa51b81700 1 --2- 172.21.2.102:0/2673068647 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7faa38003ac0 0x7faa38005f10 secure :-1 s=READY pgs=2215 cs=0 l=1 rev1=1 rx=0x7faa3c000b60 tx=0x7faa3c007da0).stop 2021-02-10T11:31:57.477 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:57.457 7faa4bfff700 1 -- 172.21.2.102:0/2673068647 reap_dead start 2021-02-10T11:31:57.478 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:57.458 7faa51b81700 1 -- 172.21.2.102:0/2673068647 shutdown_connections 2021-02-10T11:31:57.478 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:57.458 7faa51b81700 1 -- 172.21.2.102:0/2673068647 >> 172.21.2.102:0/2673068647 conn(0x7faa4c0ce420 msgr2=0x7faa4c0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:57.478 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:57.458 7faa51b81700 1 -- 172.21.2.102:0/2673068647 shutdown_connections 2021-02-10T11:31:57.478 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:57.458 7faa51b81700 1 -- 172.21.2.102:0/2673068647 wait complete. 2021-02-10T11:31:57.478 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:57.473 7faa51b81700 1 Processor -- start 2021-02-10T11:31:57.478 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:57.473 7faa51b81700 1 -- start start 2021-02-10T11:31:57.479 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:57.473 7faa51b81700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7faa4c085bc0 0x7faa4c0863e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:57.480 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:57.473 7faa51b81700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7faa4c0954c0 0x7faa4c096300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:57.480 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:57.473 7faa51b81700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7faa4c0958c0 0x7faa4c096920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:57.480 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:57.473 7faa51b81700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa4c095d80 con 0x7faa4c0954c0 2021-02-10T11:31:57.480 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:57.473 7faa51b81700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa4c0bfa60 con 0x7faa4c0958c0 2021-02-10T11:31:57.480 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:57.473 7faa51b81700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa4c085fc0 con 0x7faa4c085bc0 2021-02-10T11:31:57.480 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:57.474 7faa4affd700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7faa4c0954c0 0x7faa4c096300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:57.480 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:57.474 7faa4affd700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7faa4c0954c0 0x7faa4c096300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:34150/0 (socket says 172.21.2.102:34150) 2021-02-10T11:31:57.480 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:57.474 7faa4affd700 1 -- 172.21.2.102:0/3768116329 learned_addr learned my addr 172.21.2.102:0/3768116329 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:31:57.480 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:57.474 7faa4b7fe700 1 --2- 172.21.2.102:0/3768116329 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7faa4c085bc0 0x7faa4c0863e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:57.480 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:57.474 7faa4bfff700 1 --2- 172.21.2.102:0/3768116329 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7faa4c0958c0 0x7faa4c096920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:57.480 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:57.474 7faa4affd700 1 -- 172.21.2.102:0/3768116329 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7faa4c085bc0 msgr2=0x7faa4c0863e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:57.481 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:57.474 7faa4affd700 1 --2- 172.21.2.102:0/3768116329 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7faa4c085bc0 0x7faa4c0863e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:57.481 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:57.474 7faa4affd700 1 -- 172.21.2.102:0/3768116329 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7faa4c0958c0 msgr2=0x7faa4c096920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:57.481 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:57.474 7faa4affd700 1 --2- 172.21.2.102:0/3768116329 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7faa4c0958c0 0x7faa4c096920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:57.481 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:57.474 7faa4affd700 1 -- 172.21.2.102:0/3768116329 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa4c086c10 con 0x7faa4c0954c0 2021-02-10T11:31:57.481 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:57.474 7faa4affd700 1 --2- 172.21.2.102:0/3768116329 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7faa4c0954c0 0x7faa4c096300 secure :-1 s=READY pgs=2216 cs=0 l=1 rev1=1 rx=0x7faa3400b610 tx=0x7faa3400b9b0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:57.481 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:57.474 7faa43fff700 1 -- 172.21.2.102:0/3768116329 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faa340009b0 con 0x7faa4c0954c0 2021-02-10T11:31:57.481 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:57.474 7faa43fff700 1 -- 172.21.2.102:0/3768116329 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa3400bda0 con 0x7faa4c0954c0 2021-02-10T11:31:57.481 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:57.474 7faa51b81700 1 -- 172.21.2.102:0/3768116329 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa4c086c10 con 0x7faa4c0954c0 2021-02-10T11:31:57.482 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:57.475 7faa43fff700 1 -- 172.21.2.102:0/3768116329 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faa340022b0 con 0x7faa4c0954c0 2021-02-10T11:31:57.482 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:57.475 7faa51b81700 1 -- 172.21.2.102:0/3768116329 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa4c086c10 con 0x7faa4c0954c0 2021-02-10T11:31:57.482 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:57.475 7faa43fff700 1 -- 172.21.2.102:0/3768116329 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7faa34009fe0 con 0x7faa4c0954c0 2021-02-10T11:31:57.482 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:57.475 7faa43fff700 1 --2- 172.21.2.102:0/3768116329 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7faa300186d0 0x7faa3001abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:57.482 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:57.475 7faa43fff700 1 -- 172.21.2.102:0/3768116329 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5573+0+0 (secure 0 0 0) 0x7faa340327d0 con 0x7faa4c0954c0 2021-02-10T11:31:57.482 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:57.475 7faa4b7fe700 1 --2- 172.21.2.102:0/3768116329 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7faa300186d0 0x7faa3001abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:57.482 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:57.476 7faa4b7fe700 1 --2- 172.21.2.102:0/3768116329 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7faa300186d0 0x7faa3001abd0 crc :-1 s=READY pgs=979 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:57.506 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:57.505 7faa51b81700 1 -- 172.21.2.102:0/3768116329 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa4c0fe1c0 con 0x7faa4c0954c0 2021-02-10T11:31:57.509 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:57.508 7faa43fff700 1 -- 172.21.2.102:0/3768116329 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7faa34000c90 con 0x7faa4c0954c0 2021-02-10T11:31:57.637 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:57.636 7faa51b81700 1 -- 172.21.2.102:0/3768116329 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7faa4c0fe1c0 con 0x7faa4c0954c0 2021-02-10T11:31:57.637 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:57.636 7faa43fff700 1 -- 172.21.2.102:0/3768116329 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v112) v1 ==== 100+0+10 (secure 0 0 0) 0x7faa3400bda0 con 0x7faa4c0954c0 2021-02-10T11:31:57.638 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:31:57.640 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:57.638 7faa51b81700 1 -- 172.21.2.102:0/3768116329 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7faa300186d0 msgr2=0x7faa3001abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:57.640 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:57.638 7faa51b81700 1 --2- 172.21.2.102:0/3768116329 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7faa300186d0 0x7faa3001abd0 crc :-1 s=READY pgs=979 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:57.640 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:57.638 7faa51b81700 1 -- 172.21.2.102:0/3768116329 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7faa4c0954c0 msgr2=0x7faa4c096300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:57.640 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:57.638 7faa51b81700 1 --2- 172.21.2.102:0/3768116329 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7faa4c0954c0 0x7faa4c096300 secure :-1 s=READY pgs=2216 cs=0 l=1 rev1=1 rx=0x7faa3400b610 tx=0x7faa3400b9b0).stop 2021-02-10T11:31:57.640 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:57.638 7faa51b81700 1 -- 172.21.2.102:0/3768116329 shutdown_connections 2021-02-10T11:31:57.640 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:57.638 7faa51b81700 1 --2- 172.21.2.102:0/3768116329 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7faa300186d0 0x7faa3001abd0 unknown :-1 s=CLOSED pgs=979 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:57.640 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:57.638 7faa51b81700 1 --2- 172.21.2.102:0/3768116329 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7faa4c0958c0 0x7faa4c096920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:57.640 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:57.638 7faa51b81700 1 --2- 172.21.2.102:0/3768116329 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7faa4c0954c0 0x7faa4c096300 unknown :-1 s=CLOSED pgs=2216 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:57.640 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:57.638 7faa51b81700 1 --2- 172.21.2.102:0/3768116329 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7faa4c085bc0 0x7faa4c0863e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:57.640 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:57.638 7faa51b81700 1 -- 172.21.2.102:0/3768116329 >> 172.21.2.102:0/3768116329 conn(0x7faa4c0bef00 msgr2=0x7faa4c09d1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:57.640 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:57.638 7faa51b81700 1 -- 172.21.2.102:0/3768116329 shutdown_connections 2021-02-10T11:31:57.641 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:57.638 7faa51b81700 1 -- 172.21.2.102:0/3768116329 wait complete. 2021-02-10T11:31:57.648 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-02-10T11:31:57.648 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:31:58.074 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:58.057 7f30a6137700 1 Processor -- start 2021-02-10T11:31:58.075 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:58.057 7f30a6137700 1 -- start start 2021-02-10T11:31:58.075 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:58.058 7f30a6137700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f30a00d1870 0x7f30a01b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:58.075 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:58.058 7f30a6137700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30a00f0b10 con 0x7f30a01b0a10 2021-02-10T11:31:58.075 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:58.058 7f30a6137700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30a00d0310 con 0x7f30a00f0650 2021-02-10T11:31:58.075 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:58.058 7f30a6137700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30a00d0530 con 0x7f30a00d1870 2021-02-10T11:31:58.075 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:58.058 7f309f7fe700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f30a00d1870 0x7f30a01b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:58.075 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:58.058 7f309f7fe700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f30a00d1870 0x7f30a01b03f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:34158/0 (socket says 172.21.2.102:34158) 2021-02-10T11:31:58.075 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:58.058 7f309f7fe700 1 -- 172.21.2.102:0/651869605 learned_addr learned my addr 172.21.2.102:0/651869605 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:31:58.075 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:58.058 7f309f7fe700 1 -- 172.21.2.102:0/651869605 >> v1:172.21.2.113:6789/0 conn(0x7f30a00f0650 legacy=0x7f30a00cfb20 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:31:58.075 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:58.058 7f309f7fe700 1 -- 172.21.2.102:0/651869605 >> v1:172.21.2.102:6789/0 conn(0x7f30a01b0a10 legacy=0x7f30a00cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:58.075 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:58.058 7f309f7fe700 1 -- 172.21.2.102:0/651869605 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30a00d1c70 con 0x7f30a00d1870 2021-02-10T11:31:58.075 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:58.058 7f309f7fe700 1 --2- 172.21.2.102:0/651869605 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f30a00d1870 0x7f30a01b03f0 secure :-1 s=READY pgs=2218 cs=0 l=1 rev1=1 rx=0x7f3090006700 tx=0x7f3090009b80).ready entity=mon.0 client_cookie=f78f5ec86cf5cf87 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:58.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:58.058 7f309e7fc700 1 -- 172.21.2.102:0/651869605 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f309000c0c0 con 0x7f30a00d1870 2021-02-10T11:31:58.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:58.058 7f309e7fc700 1 -- 172.21.2.102:0/651869605 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f309000c340 con 0x7f30a00d1870 2021-02-10T11:31:58.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:58.058 7f309e7fc700 1 -- 172.21.2.102:0/651869605 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f309000c820 con 0x7f30a00d1870 2021-02-10T11:31:58.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:58.059 7f30a6137700 1 -- 172.21.2.102:0/651869605 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f30a00d1870 msgr2=0x7f30a01b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:58.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:58.059 7f30a6137700 1 --2- 172.21.2.102:0/651869605 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f30a00d1870 0x7f30a01b03f0 secure :-1 s=READY pgs=2218 cs=0 l=1 rev1=1 rx=0x7f3090006700 tx=0x7f3090009b80).stop 2021-02-10T11:31:58.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:58.059 7f30a6137700 1 -- 172.21.2.102:0/651869605 shutdown_connections 2021-02-10T11:31:58.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:58.059 7f30a6137700 1 --2- 172.21.2.102:0/651869605 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f30a00d1870 0x7f30a01b03f0 unknown :-1 s=CLOSED pgs=2218 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:58.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:58.059 7f30a6137700 1 -- 172.21.2.102:0/651869605 >> 172.21.2.102:0/651869605 conn(0x7f30a00ce450 msgr2=0x7f30a00d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:58.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:58.059 7f30a6137700 1 -- 172.21.2.102:0/651869605 shutdown_connections 2021-02-10T11:31:58.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:58.059 7f30a6137700 1 -- 172.21.2.102:0/651869605 wait complete. 2021-02-10T11:31:58.077 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:58.073 7f30a6137700 1 Processor -- start 2021-02-10T11:31:58.078 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:58.074 7f30a6137700 1 -- start start 2021-02-10T11:31:58.078 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:58.074 7f30a6137700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f30a00ae6b0 0x7f30a00a8bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:58.078 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:58.074 7f30a6137700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f30a00adbd0 0x7f30a00a9a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:58.078 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:58.074 7f30a6137700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f30a00adfd0 0x7f30a00aa090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:58.078 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:58.074 7f30a6137700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30a01bf350 con 0x7f30a00ae6b0 2021-02-10T11:31:58.078 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:58.074 7f30a6137700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30a00a9560 con 0x7f30a00adbd0 2021-02-10T11:31:58.078 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:58.074 7f30a6137700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30a00a9210 con 0x7f30a00adfd0 2021-02-10T11:31:58.078 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:58.074 7f309f7fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f30a00ae6b0 0x7f30a00a8bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:58.078 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:58.074 7f309f7fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f30a00ae6b0 0x7f30a00a8bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:34168/0 (socket says 172.21.2.102:34168) 2021-02-10T11:31:58.078 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:58.074 7f309f7fe700 1 -- 172.21.2.102:0/2870540393 learned_addr learned my addr 172.21.2.102:0/2870540393 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:31:58.078 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:58.074 7f309f7fe700 1 -- 172.21.2.102:0/2870540393 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f30a00adfd0 msgr2=0x7f30a00aa090 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:31:58.078 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:58.074 7f309f7fe700 1 --2- 172.21.2.102:0/2870540393 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f30a00adfd0 0x7f30a00aa090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:58.079 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:58.074 7f309f7fe700 1 -- 172.21.2.102:0/2870540393 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f30a00adbd0 msgr2=0x7f30a00a9a70 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:31:58.079 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:58.074 7f309f7fe700 1 --2- 172.21.2.102:0/2870540393 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f30a00adbd0 0x7f30a00a9a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:58.079 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:58.074 7f309f7fe700 1 -- 172.21.2.102:0/2870540393 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30a00aeab0 con 0x7f30a00ae6b0 2021-02-10T11:31:58.079 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:58.075 7f309f7fe700 1 --2- 172.21.2.102:0/2870540393 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f30a00ae6b0 0x7f30a00a8bc0 secure :-1 s=READY pgs=2219 cs=0 l=1 rev1=1 rx=0x7f309000b5e0 tx=0x7f309000b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:58.079 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:58.075 7f3097fff700 1 -- 172.21.2.102:0/2870540393 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f309000bd70 con 0x7f30a00ae6b0 2021-02-10T11:31:58.079 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:58.075 7f3097fff700 1 -- 172.21.2.102:0/2870540393 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3090002030 con 0x7f30a00ae6b0 2021-02-10T11:31:58.079 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:58.075 7f30a6137700 1 -- 172.21.2.102:0/2870540393 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30a00aeab0 con 0x7f30a00ae6b0 2021-02-10T11:31:58.079 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:58.075 7f30a6137700 1 -- 172.21.2.102:0/2870540393 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30a00aa9a0 con 0x7f30a00ae6b0 2021-02-10T11:31:58.079 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:58.075 7f3097fff700 1 -- 172.21.2.102:0/2870540393 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3090002550 con 0x7f30a00ae6b0 2021-02-10T11:31:58.079 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:58.076 7f3097fff700 1 -- 172.21.2.102:0/2870540393 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f3090000c90 con 0x7f30a00ae6b0 2021-02-10T11:31:58.079 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:58.076 7f3097fff700 1 --2- 172.21.2.102:0/2870540393 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f307c0186c0 0x7f307c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:58.079 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:58.076 7f3097fff700 1 -- 172.21.2.102:0/2870540393 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f3090032590 con 0x7f30a00ae6b0 2021-02-10T11:31:58.079 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:58.076 7f309effd700 1 --2- 172.21.2.102:0/2870540393 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f307c0186c0 0x7f307c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:58.080 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:58.076 7f309effd700 1 --2- 172.21.2.102:0/2870540393 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f307c0186c0 0x7f307c01abc0 crc :-1 s=READY pgs=980 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:58.109 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:58.108 7f30a6137700 1 -- 172.21.2.102:0/2870540393 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30a00e7de0 con 0x7f30a00ae6b0 2021-02-10T11:31:58.111 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:58.110 7f3097fff700 1 -- 172.21.2.102:0/2870540393 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f30900009b0 con 0x7f30a00ae6b0 2021-02-10T11:31:58.240 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:58.239 7f30a6137700 1 -- 172.21.2.102:0/2870540393 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f30a00d1ee0 con 0x7f30a00ae6b0 2021-02-10T11:31:58.240 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:58.239 7f3097fff700 1 -- 172.21.2.102:0/2870540393 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 91 v91) v1 ==== 94+0+2104 (secure 0 0 0) 0x7f3090000c90 con 0x7f30a00ae6b0 2021-02-10T11:31:58.241 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:31:58.241 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":91,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10404,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.102:6835/1661316191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6834","nonce":1661316191},{"type":"v1","addr":"172.21.2.102:6835","nonce":1661316191}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:31:48.259584","modified":"2021-02-10 11:31:50.267046","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10401},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10401":{"gid":10401,"name":"b","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.2.113:6833/313019011","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6832","nonce":313019011},{"type":"v1","addr":"172.21.2.113:6833","nonce":313019011}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[23],"metadata_pool":22,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":11}]} 2021-02-10T11:31:58.241 INFO:teuthology.orchestra.run.gibba002.stderr:dumped fsmap epoch 91 2021-02-10T11:31:58.242 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:58.241 7f30a6137700 1 -- 172.21.2.102:0/2870540393 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f307c0186c0 msgr2=0x7f307c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:58.242 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:58.241 7f30a6137700 1 --2- 172.21.2.102:0/2870540393 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f307c0186c0 0x7f307c01abc0 crc :-1 s=READY pgs=980 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:58.242 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:58.241 7f30a6137700 1 -- 172.21.2.102:0/2870540393 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f30a00ae6b0 msgr2=0x7f30a00a8bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:58.242 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:58.241 7f30a6137700 1 --2- 172.21.2.102:0/2870540393 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f30a00ae6b0 0x7f30a00a8bc0 secure :-1 s=READY pgs=2219 cs=0 l=1 rev1=1 rx=0x7f309000b5e0 tx=0x7f309000b980).stop 2021-02-10T11:31:58.243 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:58.241 7f30a6137700 1 -- 172.21.2.102:0/2870540393 shutdown_connections 2021-02-10T11:31:58.243 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:58.241 7f30a6137700 1 --2- 172.21.2.102:0/2870540393 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f30a00adfd0 0x7f30a00aa090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:58.243 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:58.241 7f30a6137700 1 --2- 172.21.2.102:0/2870540393 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f307c0186c0 0x7f307c01abc0 unknown :-1 s=CLOSED pgs=980 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:58.243 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:58.241 7f30a6137700 1 --2- 172.21.2.102:0/2870540393 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f30a00adbd0 0x7f30a00a9a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:58.243 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:58.241 7f30a6137700 1 --2- 172.21.2.102:0/2870540393 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f30a00ae6b0 0x7f30a00a8bc0 unknown :-1 s=CLOSED pgs=2219 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:58.243 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:58.241 7f30a6137700 1 -- 172.21.2.102:0/2870540393 >> 172.21.2.102:0/2870540393 conn(0x7f30a00a0030 msgr2=0x7f30a00eb1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:58.243 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:58.241 7f30a6137700 1 -- 172.21.2.102:0/2870540393 shutdown_connections 2021-02-10T11:31:58.243 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:31:58.241 7f30a6137700 1 -- 172.21.2.102:0/2870540393 wait complete. 2021-02-10T11:31:58.452 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 91, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:31:48.259584', 'modified': '2021-02-10 11:31:50.267046', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 10401}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10401': {'gid': 10401, 'name': 'b', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.113:6833/313019011', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.113:6832', 'nonce': 313019011}, {'type': 'v1', 'addr': '172.21.2.113:6833', 'nonce': 313019011}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [23], 'metadata_pool': 22, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:31:58.452 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:31:58.452 DEBUG:teuthology.orchestra.run.gibba013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:31:58.851 INFO:teuthology.orchestra.run.gibba013.stdout:{ 2021-02-10T11:31:58.851 INFO:teuthology.orchestra.run.gibba013.stdout: "cluster_fsid": "68db30ea-dbce-4aef-a6df-f01c6fea0513", 2021-02-10T11:31:58.851 INFO:teuthology.orchestra.run.gibba013.stdout: "whoami": 0, 2021-02-10T11:31:58.851 INFO:teuthology.orchestra.run.gibba013.stdout: "id": 10401, 2021-02-10T11:31:58.851 INFO:teuthology.orchestra.run.gibba013.stdout: "want_state": "up:active", 2021-02-10T11:31:58.851 INFO:teuthology.orchestra.run.gibba013.stdout: "state": "up:active", 2021-02-10T11:31:58.851 INFO:teuthology.orchestra.run.gibba013.stdout: "rank_uptime": 10.589417219, 2021-02-10T11:31:58.852 INFO:teuthology.orchestra.run.gibba013.stdout: "mdsmap_epoch": 91, 2021-02-10T11:31:58.852 INFO:teuthology.orchestra.run.gibba013.stdout: "osdmap_epoch": 112, 2021-02-10T11:31:58.852 INFO:teuthology.orchestra.run.gibba013.stdout: "osdmap_epoch_barrier": 112, 2021-02-10T11:31:58.852 INFO:teuthology.orchestra.run.gibba013.stdout: "uptime": 16.235807411 2021-02-10T11:31:58.852 INFO:teuthology.orchestra.run.gibba013.stdout:} 2021-02-10T11:31:58.855 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "68db30ea-dbce-4aef-a6df-f01c6fea0513", "whoami": 0, "id": 10401, "want_state": "up:active", "state": "up:active", "rank_uptime": 10.589417219, "mdsmap_epoch": 91, "osdmap_epoch": 112, "osdmap_epoch_barrier": 112, "uptime": 16.235807411 } 2021-02-10T11:31:58.855 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2021-02-10T11:31:58.855 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2021-02-10T11:31:58.855 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@gibba002.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2021-02-10T11:31:58.855 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:31:58.856 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec mkdir -p /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:31:59.191 DEBUG:teuthology.orchestra.run.gibba002:> sudo modprobe fuse 2021-02-10T11:31:59.535 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:31:59.535 DEBUG:teuthology.orchestra.run.gibba002:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:31:59.878 INFO:teuthology.orchestra.run.gibba002.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:31:59.879 INFO:teuthology.orchestra.run.gibba002.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:31:59.879 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:31:59.880 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:31:59.880 DEBUG:teuthology.orchestra.run.gibba002:> ls /sys/fs/fuse/connections 2021-02-10T11:32:00.214 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2021-02-10T11:32:00.215 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:32:00.217 DEBUG:teuthology.orchestra.run.gibba002:> sudo modprobe fuse 2021-02-10T11:32:00.561 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:32:00.561 DEBUG:teuthology.orchestra.run.gibba002:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:32:00.627 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba002.stderr:2021-02-10 11:32:00.625 7f1cea26bf80 -1 init, newargv = 0x55622f93d1a0 newargc=9 2021-02-10T11:32:00.627 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba002.stderr:ceph-fuse[182073]: starting ceph client 2021-02-10T11:32:00.633 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba002.stderr:ceph-fuse[182073]: starting fuse 2021-02-10T11:32:00.903 INFO:teuthology.orchestra.run.gibba002.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:32:00.904 INFO:teuthology.orchestra.run.gibba002.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:32:00.904 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:32:00.904 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:32:00.905 DEBUG:teuthology.orchestra.run.gibba002:> ls /sys/fs/fuse/connections 2021-02-10T11:32:01.238 INFO:teuthology.orchestra.run.gibba002.stdout:43 2021-02-10T11:32:01.239 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2021-02-10T11:32:01.239 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:32:01.239 DEBUG:teuthology.orchestra.run.gibba002:> sudo python3 -c ' 2021-02-10T11:32:01.239 DEBUG:teuthology.orchestra.run.gibba002:> import glob 2021-02-10T11:32:01.239 DEBUG:teuthology.orchestra.run.gibba002:> import re 2021-02-10T11:32:01.239 DEBUG:teuthology.orchestra.run.gibba002:> import os 2021-02-10T11:32:01.239 DEBUG:teuthology.orchestra.run.gibba002:> import subprocess 2021-02-10T11:32:01.239 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:32:01.239 DEBUG:teuthology.orchestra.run.gibba002:> def find_socket(client_name): 2021-02-10T11:32:01.240 DEBUG:teuthology.orchestra.run.gibba002:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2021-02-10T11:32:01.240 DEBUG:teuthology.orchestra.run.gibba002:> files = glob.glob(asok_path) 2021-02-10T11:32:01.240 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:32:01.240 DEBUG:teuthology.orchestra.run.gibba002:> # Given a non-glob path, it better be there 2021-02-10T11:32:01.240 DEBUG:teuthology.orchestra.run.gibba002:> if "*" not in asok_path: 2021-02-10T11:32:01.240 DEBUG:teuthology.orchestra.run.gibba002:> assert(len(files) == 1) 2021-02-10T11:32:01.240 DEBUG:teuthology.orchestra.run.gibba002:> return files[0] 2021-02-10T11:32:01.240 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:32:01.240 DEBUG:teuthology.orchestra.run.gibba002:> for f in files: 2021-02-10T11:32:01.240 DEBUG:teuthology.orchestra.run.gibba002:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2021-02-10T11:32:01.240 DEBUG:teuthology.orchestra.run.gibba002:> if os.path.exists("/proc/{0}".format(pid)): 2021-02-10T11:32:01.240 DEBUG:teuthology.orchestra.run.gibba002:> return f 2021-02-10T11:32:01.240 DEBUG:teuthology.orchestra.run.gibba002:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2021-02-10T11:32:01.240 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:32:01.241 DEBUG:teuthology.orchestra.run.gibba002:> print(find_socket("client.0")) 2021-02-10T11:32:01.241 DEBUG:teuthology.orchestra.run.gibba002:> ' 2021-02-10T11:32:01.603 INFO:teuthology.orchestra.run.gibba002.stdout:/var/run/ceph/ceph-client.0.182073.asok 2021-02-10T11:32:01.605 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.182073.asok 2021-02-10T11:32:01.605 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:32:01.606 DEBUG:teuthology.orchestra.run.gibba002:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.182073.asok status 2021-02-10T11:32:01.997 INFO:teuthology.orchestra.run.gibba002.stdout:{ 2021-02-10T11:32:01.997 INFO:teuthology.orchestra.run.gibba002.stdout: "metadata": { 2021-02-10T11:32:01.998 INFO:teuthology.orchestra.run.gibba002.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:32:01.998 INFO:teuthology.orchestra.run.gibba002.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:32:01.998 INFO:teuthology.orchestra.run.gibba002.stdout: "entity_id": "0", 2021-02-10T11:32:01.998 INFO:teuthology.orchestra.run.gibba002.stdout: "hostname": "gibba002", 2021-02-10T11:32:01.998 INFO:teuthology.orchestra.run.gibba002.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2021-02-10T11:32:01.998 INFO:teuthology.orchestra.run.gibba002.stdout: "pid": "182073", 2021-02-10T11:32:01.998 INFO:teuthology.orchestra.run.gibba002.stdout: "root": "/" 2021-02-10T11:32:01.998 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:32:01.998 INFO:teuthology.orchestra.run.gibba002.stdout: "dentry_count": 0, 2021-02-10T11:32:01.998 INFO:teuthology.orchestra.run.gibba002.stdout: "dentry_pinned_count": 0, 2021-02-10T11:32:01.998 INFO:teuthology.orchestra.run.gibba002.stdout: "id": 10563, 2021-02-10T11:32:01.998 INFO:teuthology.orchestra.run.gibba002.stdout: "inst": { 2021-02-10T11:32:01.998 INFO:teuthology.orchestra.run.gibba002.stdout: "name": { 2021-02-10T11:32:01.998 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "client", 2021-02-10T11:32:01.999 INFO:teuthology.orchestra.run.gibba002.stdout: "num": 10563 2021-02-10T11:32:01.999 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:32:01.999 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": { 2021-02-10T11:32:01.999 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:32:01.999 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:0", 2021-02-10T11:32:01.999 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 2187742070 2021-02-10T11:32:01.999 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:32:01.999 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:32:01.999 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": { 2021-02-10T11:32:01.999 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:32:01.999 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:0", 2021-02-10T11:32:01.999 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 2187742070 2021-02-10T11:32:01.999 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:32:02.000 INFO:teuthology.orchestra.run.gibba002.stdout: "inst_str": "client.10563 172.21.2.102:0/2187742070", 2021-02-10T11:32:02.000 INFO:teuthology.orchestra.run.gibba002.stdout: "addr_str": "172.21.2.102:0/2187742070", 2021-02-10T11:32:02.000 INFO:teuthology.orchestra.run.gibba002.stdout: "inode_count": 1, 2021-02-10T11:32:02.000 INFO:teuthology.orchestra.run.gibba002.stdout: "mds_epoch": 91, 2021-02-10T11:32:02.000 INFO:teuthology.orchestra.run.gibba002.stdout: "osd_epoch": 112, 2021-02-10T11:32:02.000 INFO:teuthology.orchestra.run.gibba002.stdout: "osd_epoch_barrier": 0, 2021-02-10T11:32:02.000 INFO:teuthology.orchestra.run.gibba002.stdout: "blacklisted": false 2021-02-10T11:32:02.000 INFO:teuthology.orchestra.run.gibba002.stdout:} 2021-02-10T11:32:02.001 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:32:02.002 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:32:02.002 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:32:02.337 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:32:02.337 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:32:02.338 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:32:02.338 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:32:02.684 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:32:03.111 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:03.094 7ffad51d1700 1 Processor -- start 2021-02-10T11:32:03.111 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:03.094 7ffad51d1700 1 -- start start 2021-02-10T11:32:03.112 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:03.095 7ffad51d1700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7ffad00d1870 0x7ffad01b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:03.112 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:03.095 7ffad51d1700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7ffad01b0a10 0x7ffad00cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:03.112 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:03.095 7ffad51d1700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffad00f0b10 con 0x7ffad00d1c70 2021-02-10T11:32:03.112 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:03.095 7ffad51d1700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffad00e5ed0 con 0x7ffad01b0a10 2021-02-10T11:32:03.112 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:03.095 7ffad51d1700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffad00f0650 con 0x7ffad00d1870 2021-02-10T11:32:03.112 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:03.095 7fface59c700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7ffad01b0a10 0x7ffad00cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:03.112 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:03.095 7fface59c700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7ffad01b0a10 0x7ffad00cf330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:34188/0 (socket says 172.21.2.102:34188) 2021-02-10T11:32:03.112 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:03.095 7fface59c700 1 -- 172.21.2.102:0/2277342824 learned_addr learned my addr 172.21.2.102:0/2277342824 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:32:03.112 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:03.095 7fface59c700 1 -- 172.21.2.102:0/2277342824 >> v2:172.21.2.113:3300/0 conn(0x7ffad00d1870 msgr2=0x7ffad01b03f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:32:03.112 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:03.095 7fface59c700 1 --2- 172.21.2.102:0/2277342824 >> v2:172.21.2.113:3300/0 conn(0x7ffad00d1870 0x7ffad01b03f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:03.112 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:03.095 7fface59c700 1 -- 172.21.2.102:0/2277342824 >> v1:172.21.2.113:6789/0 conn(0x7ffad00d1c70 legacy=0x7ffad00cf950 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:32:03.112 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:03.095 7fface59c700 1 -- 172.21.2.102:0/2277342824 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffad00f0840 con 0x7ffad01b0a10 2021-02-10T11:32:03.113 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:03.095 7fface59c700 1 --2- 172.21.2.102:0/2277342824 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ffad01b0a10 0x7ffad00cf330 secure :-1 s=READY pgs=2223 cs=0 l=1 rev1=1 rx=0x7ffab8006700 tx=0x7ffab8009b80).ready entity=mon.0 client_cookie=79393ab7d398e5c7 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:03.113 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:03.095 7ffacdd9b700 1 -- 172.21.2.102:0/2277342824 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ffab8000c90 con 0x7ffad01b0a10 2021-02-10T11:32:03.113 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:03.095 7ffacdd9b700 1 -- 172.21.2.102:0/2277342824 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffab800c0c0 con 0x7ffad01b0a10 2021-02-10T11:32:03.113 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:03.095 7ffacdd9b700 1 -- 172.21.2.102:0/2277342824 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ffab800c5e0 con 0x7ffad01b0a10 2021-02-10T11:32:03.113 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:03.095 7ffad51d1700 1 -- 172.21.2.102:0/2277342824 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ffad01b0a10 msgr2=0x7ffad00cf330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:03.113 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:03.095 7ffad51d1700 1 --2- 172.21.2.102:0/2277342824 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ffad01b0a10 0x7ffad00cf330 secure :-1 s=READY pgs=2223 cs=0 l=1 rev1=1 rx=0x7ffab8006700 tx=0x7ffab8009b80).stop 2021-02-10T11:32:03.113 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:03.096 7ffad51d1700 1 -- 172.21.2.102:0/2277342824 shutdown_connections 2021-02-10T11:32:03.113 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:03.096 7ffad51d1700 1 --2- 172.21.2.102:0/2277342824 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ffad01b0a10 0x7ffad00cf330 unknown :-1 s=CLOSED pgs=2223 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:03.113 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:03.096 7ffad51d1700 1 --2- 172.21.2.102:0/2277342824 >> v2:172.21.2.113:3300/0 conn(0x7ffad00d1870 0x7ffad01b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:03.113 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:03.096 7ffad51d1700 1 -- 172.21.2.102:0/2277342824 >> 172.21.2.102:0/2277342824 conn(0x7ffad00ce450 msgr2=0x7ffad00d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:03.113 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:03.096 7ffad51d1700 1 -- 172.21.2.102:0/2277342824 shutdown_connections 2021-02-10T11:32:03.113 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:03.096 7ffad51d1700 1 -- 172.21.2.102:0/2277342824 wait complete. 2021-02-10T11:32:03.115 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:03.110 7ffad51d1700 1 Processor -- start 2021-02-10T11:32:03.115 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:03.111 7ffad51d1700 1 -- start start 2021-02-10T11:32:03.115 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:03.111 7ffad51d1700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ffad00b0ad0 0x7ffad0091f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:03.115 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:03.111 7ffad51d1700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ffad01a8850 0x7ffad01a8c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:03.115 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:03.111 7ffad51d1700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ffad01a6fa0 0x7ffad01a73a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:03.115 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:03.111 7ffad51d1700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffad01abfe0 con 0x7ffad00b0ad0 2021-02-10T11:32:03.115 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:03.111 7ffad51d1700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffad01a9c40 con 0x7ffad01a6fa0 2021-02-10T11:32:03.115 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:03.111 7ffad51d1700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffad01a9730 con 0x7ffad01a8850 2021-02-10T11:32:03.115 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:03.111 7ffaced9d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ffad00b0ad0 0x7ffad0091f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:03.116 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:03.111 7ffaced9d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ffad00b0ad0 0x7ffad0091f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:34194/0 (socket says 172.21.2.102:34194) 2021-02-10T11:32:03.116 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:03.111 7ffaced9d700 1 -- 172.21.2.102:0/2169497259 learned_addr learned my addr 172.21.2.102:0/2169497259 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:32:03.116 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:03.111 7ffaced9d700 1 -- 172.21.2.102:0/2169497259 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ffad01a8850 msgr2=0x7ffad01a8c50 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:32:03.116 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:03.111 7ffaced9d700 1 --2- 172.21.2.102:0/2169497259 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ffad01a8850 0x7ffad01a8c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:03.116 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:03.111 7ffaced9d700 1 -- 172.21.2.102:0/2169497259 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ffad01a6fa0 msgr2=0x7ffad01a73a0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:32:03.116 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:03.111 7ffaced9d700 1 --2- 172.21.2.102:0/2169497259 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ffad01a6fa0 0x7ffad01a73a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:03.116 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:03.111 7ffaced9d700 1 -- 172.21.2.102:0/2169497259 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffad01a9270 con 0x7ffad00b0ad0 2021-02-10T11:32:03.116 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:03.112 7ffaced9d700 1 --2- 172.21.2.102:0/2169497259 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ffad00b0ad0 0x7ffad0091f30 secure :-1 s=READY pgs=2224 cs=0 l=1 rev1=1 rx=0x7ffac000b5c0 tx=0x7ffac000b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:03.116 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:03.112 7ffac77fe700 1 -- 172.21.2.102:0/2169497259 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ffac000bd50 con 0x7ffad00b0ad0 2021-02-10T11:32:03.116 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:03.112 7ffac77fe700 1 -- 172.21.2.102:0/2169497259 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffac0006030 con 0x7ffad00b0ad0 2021-02-10T11:32:03.116 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:03.112 7ffac77fe700 1 -- 172.21.2.102:0/2169497259 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ffac00054d0 con 0x7ffad00b0ad0 2021-02-10T11:32:03.116 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:03.112 7ffad51d1700 1 -- 172.21.2.102:0/2169497259 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffad01a9270 con 0x7ffad00b0ad0 2021-02-10T11:32:03.116 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:03.112 7ffad51d1700 1 -- 172.21.2.102:0/2169497259 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffad01a7cb0 con 0x7ffad00b0ad0 2021-02-10T11:32:03.116 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:03.112 7ffac77fe700 1 -- 172.21.2.102:0/2169497259 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ffac0006210 con 0x7ffad00b0ad0 2021-02-10T11:32:03.117 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:03.113 7ffac77fe700 1 --2- 172.21.2.102:0/2169497259 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ffab40186c0 0x7ffab401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:03.117 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:03.113 7ffac77fe700 1 -- 172.21.2.102:0/2169497259 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5573+0+0 (secure 0 0 0) 0x7ffac00325d0 con 0x7ffad00b0ad0 2021-02-10T11:32:03.117 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:03.113 7fface59c700 1 --2- 172.21.2.102:0/2169497259 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ffab40186c0 0x7ffab401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:03.117 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:03.113 7fface59c700 1 --2- 172.21.2.102:0/2169497259 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ffab40186c0 0x7ffab401abc0 crc :-1 s=READY pgs=981 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:03.146 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:03.145 7ffad51d1700 1 -- 172.21.2.102:0/2169497259 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffad01a9710 con 0x7ffad00b0ad0 2021-02-10T11:32:03.148 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:03.147 7ffac77fe700 1 -- 172.21.2.102:0/2169497259 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ffac0000960 con 0x7ffad00b0ad0 2021-02-10T11:32:03.274 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:03.272 7ffad51d1700 1 -- 172.21.2.102:0/2169497259 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ffad01a9710 con 0x7ffad00b0ad0 2021-02-10T11:32:03.275 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:03.273 7ffac77fe700 1 -- 172.21.2.102:0/2169497259 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v112) v1 ==== 74+0+14007 (secure 0 0 0) 0x7ffac00061f0 con 0x7ffad00b0ad0 2021-02-10T11:32:03.275 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:32:03.275 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":112,"fsid":"68db30ea-dbce-4aef-a6df-f01c6fea0513","created":"2021-02-10 11:03:06.331557","modified":"2021-02-10 11:31:49.261779","last_up_change":"2021-02-10 11:03:32.690562","last_in_change":"2021-02-10 11:03:32.690562","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":23,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:03:37.322519","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":22,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:31:46.228075","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"111","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":23,"pool_name":"cephfs_data","create_time":"2021-02-10 11:31:47.252676","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"112","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"be691b4d-a039-408a-af80-9d53f02b6f1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6818","nonce":36088},{"type":"v1","addr":"172.21.2.102:6820","nonce":36088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6822","nonce":36088},{"type":"v1","addr":"172.21.2.102:6824","nonce":36088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6830","nonce":36088},{"type":"v1","addr":"172.21.2.102:6832","nonce":36088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6826","nonce":36088},{"type":"v1","addr":"172.21.2.102:6828","nonce":36088}]},"public_addr":"172.21.2.102:6820/36088","cluster_addr":"172.21.2.102:6824/36088","heartbeat_back_addr":"172.21.2.102:6832/36088","heartbeat_front_addr":"172.21.2.102:6828/36088","state":["exists","up"]},{"osd":1,"uuid":"64e3ccc8-9c69-4d7c-865e-af8e83822635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6819","nonce":36087},{"type":"v1","addr":"172.21.2.102:6821","nonce":36087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6823","nonce":36087},{"type":"v1","addr":"172.21.2.102:6825","nonce":36087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6831","nonce":36087},{"type":"v1","addr":"172.21.2.102:6833","nonce":36087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6827","nonce":36087},{"type":"v1","addr":"172.21.2.102:6829","nonce":36087}]},"public_addr":"172.21.2.102:6821/36087","cluster_addr":"172.21.2.102:6825/36087","heartbeat_back_addr":"172.21.2.102:6833/36087","heartbeat_front_addr":"172.21.2.102:6829/36087","state":["exists","up"]},{"osd":2,"uuid":"153700af-747f-481f-bdf4-d6072d3a842f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6810","nonce":36089},{"type":"v1","addr":"172.21.2.102:6811","nonce":36089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6812","nonce":36089},{"type":"v1","addr":"172.21.2.102:6813","nonce":36089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6816","nonce":36089},{"type":"v1","addr":"172.21.2.102:6817","nonce":36089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6814","nonce":36089},{"type":"v1","addr":"172.21.2.102:6815","nonce":36089}]},"public_addr":"172.21.2.102:6811/36089","cluster_addr":"172.21.2.102:6813/36089","heartbeat_back_addr":"172.21.2.102:6817/36089","heartbeat_front_addr":"172.21.2.102:6815/36089","state":["exists","up"]},{"osd":3,"uuid":"cbc678f4-83a9-45e6-8830-2f7c232b1ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6802","nonce":36090},{"type":"v1","addr":"172.21.2.102:6803","nonce":36090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6804","nonce":36090},{"type":"v1","addr":"172.21.2.102:6805","nonce":36090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6808","nonce":36090},{"type":"v1","addr":"172.21.2.102:6809","nonce":36090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6806","nonce":36090},{"type":"v1","addr":"172.21.2.102:6807","nonce":36090}]},"public_addr":"172.21.2.102:6803/36090","cluster_addr":"172.21.2.102:6805/36090","heartbeat_back_addr":"172.21.2.102:6809/36090","heartbeat_front_addr":"172.21.2.102:6807/36090","state":["exists","up"]},{"osd":4,"uuid":"c01e4008-b258-4f10-afd2-9e7f37214351","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6808","nonce":36368},{"type":"v1","addr":"172.21.2.113:6811","nonce":36368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6814","nonce":36368},{"type":"v1","addr":"172.21.2.113:6818","nonce":36368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6826","nonce":36368},{"type":"v1","addr":"172.21.2.113:6828","nonce":36368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6820","nonce":36368},{"type":"v1","addr":"172.21.2.113:6824","nonce":36368}]},"public_addr":"172.21.2.113:6811/36368","cluster_addr":"172.21.2.113:6818/36368","heartbeat_back_addr":"172.21.2.113:6828/36368","heartbeat_front_addr":"172.21.2.113:6824/36368","state":["exists","up"]},{"osd":5,"uuid":"551f6007-daf8-4a98-8ee0-c2b16bcccb5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6802","nonce":36366},{"type":"v1","addr":"172.21.2.113:6805","nonce":36366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6807","nonce":36366},{"type":"v1","addr":"172.21.2.113:6810","nonce":36366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6819","nonce":36366},{"type":"v1","addr":"172.21.2.113:6822","nonce":36366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6812","nonce":36366},{"type":"v1","addr":"172.21.2.113:6815","nonce":36366}]},"public_addr":"172.21.2.113:6805/36366","cluster_addr":"172.21.2.113:6810/36366","heartbeat_back_addr":"172.21.2.113:6822/36366","heartbeat_front_addr":"172.21.2.113:6815/36366","state":["exists","up"]},{"osd":6,"uuid":"bf67c040-0e50-48be-b102-c3943bc9ebef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6800","nonce":36367},{"type":"v1","addr":"172.21.2.113:6801","nonce":36367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6803","nonce":36367},{"type":"v1","addr":"172.21.2.113:6804","nonce":36367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6813","nonce":36367},{"type":"v1","addr":"172.21.2.113:6816","nonce":36367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6806","nonce":36367},{"type":"v1","addr":"172.21.2.113:6809","nonce":36367}]},"public_addr":"172.21.2.113:6801/36367","cluster_addr":"172.21.2.113:6804/36367","heartbeat_back_addr":"172.21.2.113:6816/36367","heartbeat_front_addr":"172.21.2.113:6809/36367","state":["exists","up"]},{"osd":7,"uuid":"f5461f80-018d-428f-be89-2eb2bcbe7bb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6817","nonce":36365},{"type":"v1","addr":"172.21.2.113:6821","nonce":36365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6823","nonce":36365},{"type":"v1","addr":"172.21.2.113:6825","nonce":36365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6830","nonce":36365},{"type":"v1","addr":"172.21.2.113:6831","nonce":36365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6827","nonce":36365},{"type":"v1","addr":"172.21.2.113:6829","nonce":36365}]},"public_addr":"172.21.2.113:6821/36365","cluster_addr":"172.21.2.113:6825/36365","heartbeat_back_addr":"172.21.2.113:6831/36365","heartbeat_front_addr":"172.21.2.113:6829/36365","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:32:03.277 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:03.274 7ffad51d1700 1 -- 172.21.2.102:0/2169497259 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ffab40186c0 msgr2=0x7ffab401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:03.277 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:03.274 7ffad51d1700 1 --2- 172.21.2.102:0/2169497259 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ffab40186c0 0x7ffab401abc0 crc :-1 s=READY pgs=981 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:03.278 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:03.274 7ffad51d1700 1 -- 172.21.2.102:0/2169497259 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ffad00b0ad0 msgr2=0x7ffad0091f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:03.278 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:03.274 7ffad51d1700 1 --2- 172.21.2.102:0/2169497259 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ffad00b0ad0 0x7ffad0091f30 secure :-1 s=READY pgs=2224 cs=0 l=1 rev1=1 rx=0x7ffac000b5c0 tx=0x7ffac000b960).stop 2021-02-10T11:32:03.278 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:03.275 7ffad51d1700 1 -- 172.21.2.102:0/2169497259 shutdown_connections 2021-02-10T11:32:03.278 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:03.275 7ffad51d1700 1 --2- 172.21.2.102:0/2169497259 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ffab40186c0 0x7ffab401abc0 unknown :-1 s=CLOSED pgs=981 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:03.279 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:03.275 7ffad51d1700 1 --2- 172.21.2.102:0/2169497259 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ffad01a6fa0 0x7ffad01a73a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:03.279 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:03.275 7ffad51d1700 1 --2- 172.21.2.102:0/2169497259 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ffad01a8850 0x7ffad01a8c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:03.279 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:03.275 7ffad51d1700 1 --2- 172.21.2.102:0/2169497259 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ffad00b0ad0 0x7ffad0091f30 unknown :-1 s=CLOSED pgs=2224 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:03.279 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:03.275 7ffad51d1700 1 -- 172.21.2.102:0/2169497259 >> 172.21.2.102:0/2169497259 conn(0x7ffad019f8f0 msgr2=0x7ffad01a5f80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:03.279 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:03.275 7ffad51d1700 1 -- 172.21.2.102:0/2169497259 shutdown_connections 2021-02-10T11:32:03.280 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:03.275 7ffad51d1700 1 -- 172.21.2.102:0/2169497259 wait complete. 2021-02-10T11:32:03.286 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:03:37.322519', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 22, 'pool_name': 'cephfs_metadata', 'create_time': '2021-02-10 11:31:46.228075', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '111', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 23, 'pool_name': 'cephfs_data', 'create_time': '2021-02-10 11:31:47.252676', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '112', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-02-10T11:32:03.286 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:32:03.718 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:03.700 7fd507ac2700 1 Processor -- start 2021-02-10T11:32:03.718 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:03.700 7fd507ac2700 1 -- start start 2021-02-10T11:32:03.718 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:03.701 7fd507ac2700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fd5000d1c70 0x7fd5000cef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:03.718 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:03.701 7fd507ac2700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd5000cf580 0x7fd5000cf980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:03.718 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:03.701 7fd507ac2700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5000fe1c0 con 0x7fd5000d1870 2021-02-10T11:32:03.718 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:03.701 7fd507ac2700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5001b0cb0 con 0x7fd5000d1c70 2021-02-10T11:32:03.718 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:03.701 7fd507ac2700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd500109a60 con 0x7fd5000cf580 2021-02-10T11:32:03.719 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:03.701 7fd5050dd700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fd5000d1c70 0x7fd5000cef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:03.719 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:03.701 7fd5058de700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fd5000d1870 0x7fd5001b04c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:54656/0 (socket says 172.21.2.102:54656) 2021-02-10T11:32:03.719 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:03.701 7fd5058de700 1 -- 172.21.2.102:0/348777299 learned_addr learned my addr 172.21.2.102:0/348777299 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:32:03.719 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:03.701 7fd5050dd700 1 -- 172.21.2.102:0/348777299 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd5000cf580 msgr2=0x7fd5000cf980 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:32:03.719 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:03.701 7fd5050dd700 1 --2- 172.21.2.102:0/348777299 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd5000cf580 0x7fd5000cf980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:03.719 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:03.701 7fd5050dd700 1 -- 172.21.2.102:0/348777299 >> v1:172.21.2.102:6789/0 conn(0x7fd5000d1870 legacy=0x7fd5001b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:03.719 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:03.701 7fd5050dd700 1 -- 172.21.2.102:0/348777299 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd500088070 con 0x7fd5000d1c70 2021-02-10T11:32:03.719 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:03.701 7fd5050dd700 1 --2- 172.21.2.102:0/348777299 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd5000d1c70 0x7fd5000cef60 secure :-1 s=READY pgs=2226 cs=0 l=1 rev1=1 rx=0x7fd4fc006650 tx=0x7fd4fc009a80).ready entity=mon.0 client_cookie=8a636786deb85b85 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:03.719 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:03.702 7fd5048dc700 1 -- 172.21.2.102:0/348777299 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd4fc00c0c0 con 0x7fd5000d1c70 2021-02-10T11:32:03.719 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:03.702 7fd5048dc700 1 -- 172.21.2.102:0/348777299 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4fc00c0c0 con 0x7fd5000d1c70 2021-02-10T11:32:03.719 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:03.702 7fd5048dc700 1 -- 172.21.2.102:0/348777299 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd4fc00c340 con 0x7fd5000d1c70 2021-02-10T11:32:03.719 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:03.702 7fd507ac2700 1 -- 172.21.2.102:0/348777299 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd5000d1c70 msgr2=0x7fd5000cef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:03.720 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:03.702 7fd507ac2700 1 --2- 172.21.2.102:0/348777299 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd5000d1c70 0x7fd5000cef60 secure :-1 s=READY pgs=2226 cs=0 l=1 rev1=1 rx=0x7fd4fc006650 tx=0x7fd4fc009a80).stop 2021-02-10T11:32:03.720 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:03.702 7fd507ac2700 1 -- 172.21.2.102:0/348777299 shutdown_connections 2021-02-10T11:32:03.720 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:03.702 7fd507ac2700 1 --2- 172.21.2.102:0/348777299 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd5000d1c70 0x7fd5000cef60 unknown :-1 s=CLOSED pgs=2226 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:03.720 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:03.702 7fd507ac2700 1 --2- 172.21.2.102:0/348777299 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd5000cf580 0x7fd5000cf980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:03.720 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:03.702 7fd507ac2700 1 -- 172.21.2.102:0/348777299 >> 172.21.2.102:0/348777299 conn(0x7fd5000ce420 msgr2=0x7fd5000d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:03.720 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:03.702 7fd507ac2700 1 -- 172.21.2.102:0/348777299 shutdown_connections 2021-02-10T11:32:03.720 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:03.702 7fd507ac2700 1 -- 172.21.2.102:0/348777299 wait complete. 2021-02-10T11:32:03.722 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:03.717 7fd507ac2700 1 Processor -- start 2021-02-10T11:32:03.722 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:03.717 7fd507ac2700 1 -- start start 2021-02-10T11:32:03.722 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:03.717 7fd507ac2700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd5000c3820 0x7fd500084cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:03.722 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:03.717 7fd507ac2700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd5000a0290 0x7fd5000852d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:03.722 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:03.717 7fd507ac2700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd500086270 0x7fd500086670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:03.722 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:03.717 7fd507ac2700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5000c0700 con 0x7fd5000c3820 2021-02-10T11:32:03.722 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:03.717 7fd507ac2700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5000858f0 con 0x7fd500086270 2021-02-10T11:32:03.722 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:03.717 7fd507ac2700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd500085b10 con 0x7fd5000a0290 2021-02-10T11:32:03.722 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:03.718 7fd5058de700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd5000c3820 0x7fd500084cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:03.723 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:03.718 7fd5058de700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd5000c3820 0x7fd500084cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:34208/0 (socket says 172.21.2.102:34208) 2021-02-10T11:32:03.723 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:03.718 7fd5058de700 1 -- 172.21.2.102:0/3439338542 learned_addr learned my addr 172.21.2.102:0/3439338542 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:32:03.723 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:03.718 7fd5058de700 1 -- 172.21.2.102:0/3439338542 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd5000a0290 msgr2=0x7fd5000852d0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:32:03.723 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:03.718 7fd5058de700 1 --2- 172.21.2.102:0/3439338542 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd5000a0290 0x7fd5000852d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:03.723 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:03.718 7fd5058de700 1 -- 172.21.2.102:0/3439338542 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd500086270 msgr2=0x7fd500086670 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:32:03.723 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:03.718 7fd5058de700 1 --2- 172.21.2.102:0/3439338542 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd500086270 0x7fd500086670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:03.723 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:03.718 7fd5058de700 1 -- 172.21.2.102:0/3439338542 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd500086c90 con 0x7fd5000c3820 2021-02-10T11:32:03.723 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:03.718 7fd5058de700 1 --2- 172.21.2.102:0/3439338542 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd5000c3820 0x7fd500084cb0 secure :-1 s=READY pgs=2227 cs=0 l=1 rev1=1 rx=0x7fd4f4001c60 tx=0x7fd4f400a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:03.723 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:03.718 7fd4f27fc700 1 -- 172.21.2.102:0/3439338542 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd4f400acc0 con 0x7fd5000c3820 2021-02-10T11:32:03.723 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:03.718 7fd4f27fc700 1 -- 172.21.2.102:0/3439338542 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4f4004140 con 0x7fd5000c3820 2021-02-10T11:32:03.723 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:03.718 7fd4f27fc700 1 -- 172.21.2.102:0/3439338542 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd4f4004660 con 0x7fd5000c3820 2021-02-10T11:32:03.723 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:03.718 7fd507ac2700 1 -- 172.21.2.102:0/3439338542 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd500086c90 con 0x7fd5000c3820 2021-02-10T11:32:03.724 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:03.718 7fd507ac2700 1 -- 172.21.2.102:0/3439338542 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd500086c90 con 0x7fd5000c3820 2021-02-10T11:32:03.724 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:03.719 7fd4f27fc700 1 -- 172.21.2.102:0/3439338542 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd4f4000db0 con 0x7fd5000c3820 2021-02-10T11:32:03.724 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:03.719 7fd4f27fc700 1 --2- 172.21.2.102:0/3439338542 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd4e8018730 0x7fd4e801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:03.724 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:03.719 7fd4f27fc700 1 -- 172.21.2.102:0/3439338542 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fd4f4032620 con 0x7fd5000c3820 2021-02-10T11:32:03.724 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:03.719 7fd5050dd700 1 --2- 172.21.2.102:0/3439338542 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd4e8018730 0x7fd4e801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:03.724 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:03.719 7fd5050dd700 1 --2- 172.21.2.102:0/3439338542 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd4e8018730 0x7fd4e801ac10 crc :-1 s=READY pgs=982 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:03.752 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:03.751 7fd507ac2700 1 -- 172.21.2.102:0/3439338542 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd500082a70 con 0x7fd5000c3820 2021-02-10T11:32:03.754 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:03.753 7fd4f27fc700 1 -- 172.21.2.102:0/3439338542 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd4f40009b0 con 0x7fd5000c3820 2021-02-10T11:32:03.883 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:03.882 7fd507ac2700 1 -- 172.21.2.102:0/3439338542 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7fd500082a70 con 0x7fd5000c3820 2021-02-10T11:32:03.883 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:03.882 7fd4f27fc700 1 -- 172.21.2.102:0/3439338542 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v112) v1 ==== 92+0+10 (secure 0 0 0) 0x7fd4f4000d90 con 0x7fd5000c3820 2021-02-10T11:32:03.884 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:32:03.885 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:03.884 7fd507ac2700 1 -- 172.21.2.102:0/3439338542 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd4e8018730 msgr2=0x7fd4e801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:03.885 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:03.884 7fd507ac2700 1 --2- 172.21.2.102:0/3439338542 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd4e8018730 0x7fd4e801ac10 crc :-1 s=READY pgs=982 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:03.885 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:03.884 7fd507ac2700 1 -- 172.21.2.102:0/3439338542 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd5000c3820 msgr2=0x7fd500084cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:03.885 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:03.884 7fd507ac2700 1 --2- 172.21.2.102:0/3439338542 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd5000c3820 0x7fd500084cb0 secure :-1 s=READY pgs=2227 cs=0 l=1 rev1=1 rx=0x7fd4f4001c60 tx=0x7fd4f400a8d0).stop 2021-02-10T11:32:03.885 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:03.884 7fd507ac2700 1 -- 172.21.2.102:0/3439338542 shutdown_connections 2021-02-10T11:32:03.886 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:03.884 7fd507ac2700 1 --2- 172.21.2.102:0/3439338542 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd4e8018730 0x7fd4e801ac10 unknown :-1 s=CLOSED pgs=982 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:03.886 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:03.884 7fd507ac2700 1 --2- 172.21.2.102:0/3439338542 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd500086270 0x7fd500086670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:03.886 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:03.884 7fd507ac2700 1 --2- 172.21.2.102:0/3439338542 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd5000a0290 0x7fd5000852d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:03.886 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:03.884 7fd507ac2700 1 --2- 172.21.2.102:0/3439338542 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd5000c3820 0x7fd500084cb0 unknown :-1 s=CLOSED pgs=2227 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:03.886 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:03.884 7fd507ac2700 1 -- 172.21.2.102:0/3439338542 >> 172.21.2.102:0/3439338542 conn(0x7fd5000872c0 msgr2=0x7fd500095700 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:03.886 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:03.884 7fd507ac2700 1 -- 172.21.2.102:0/3439338542 shutdown_connections 2021-02-10T11:32:03.886 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:03.884 7fd507ac2700 1 -- 172.21.2.102:0/3439338542 wait complete. 2021-02-10T11:32:03.894 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-02-10T11:32:04.330 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:04.308 7ff8c5e21700 1 Processor -- start 2021-02-10T11:32:04.330 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:04.308 7ff8c5e21700 1 -- start start 2021-02-10T11:32:04.331 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:04.308 7ff8c5e21700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7ff8c00d1870 0x7ff8c01b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:04.331 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:04.308 7ff8c5e21700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff8c00cff70 0x7ff8c0109130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:04.331 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:04.308 7ff8c5e21700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8c0109a60 con 0x7ff8c01b0ae0 2021-02-10T11:32:04.331 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:04.308 7ff8c5e21700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8c0109750 con 0x7ff8c00d1870 2021-02-10T11:32:04.331 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:04.308 7ff8c5e21700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8c0088070 con 0x7ff8c00cff70 2021-02-10T11:32:04.331 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:04.309 7ff8beffd700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7ff8c01b0ae0 0x7ff8c0108940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:54670/0 (socket says 172.21.2.102:54670) 2021-02-10T11:32:04.332 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:04.309 7ff8beffd700 1 -- 172.21.2.102:0/3645692514 learned_addr learned my addr 172.21.2.102:0/3645692514 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:32:04.332 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:04.309 7ff8be7fc700 1 -- 172.21.2.102:0/3645692514 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1258443866 0 0) 0x7ff8b00010b0 con 0x7ff8c01b0ae0 2021-02-10T11:32:04.332 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:04.309 7ff8be7fc700 1 -- 172.21.2.102:0/3645692514 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8ac002fe0 con 0x7ff8c01b0ae0 2021-02-10T11:32:04.332 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:04.309 7ff8bf7fe700 1 --2- 172.21.2.102:0/3645692514 >> v2:172.21.2.113:3300/0 conn(0x7ff8c00d1870 0x7ff8c01b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:04.332 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:04.309 7ff8bffff700 1 --2- 172.21.2.102:0/3645692514 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff8c00cff70 0x7ff8c0109130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:04.332 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:04.309 7ff8be7fc700 1 -- 172.21.2.102:0/3645692514 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2133996079 0 0) 0x7ff8b00010b0 con 0x7ff8c01b0ae0 2021-02-10T11:32:04.333 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:04.310 7ff8be7fc700 1 -- 172.21.2.102:0/3645692514 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff8c00cff70 msgr2=0x7ff8c0109130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:32:04.333 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:04.310 7ff8be7fc700 1 --2- 172.21.2.102:0/3645692514 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff8c00cff70 0x7ff8c0109130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:04.333 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:04.310 7ff8be7fc700 1 -- 172.21.2.102:0/3645692514 >> v2:172.21.2.113:3300/0 conn(0x7ff8c00d1870 msgr2=0x7ff8c01b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:32:04.333 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:04.310 7ff8be7fc700 1 --2- 172.21.2.102:0/3645692514 >> v2:172.21.2.113:3300/0 conn(0x7ff8c00d1870 0x7ff8c01b04c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:04.333 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:04.310 7ff8be7fc700 1 -- 172.21.2.102:0/3645692514 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8c00d1c70 con 0x7ff8c01b0ae0 2021-02-10T11:32:04.333 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:04.310 7ff8be7fc700 1 -- 172.21.2.102:0/3645692514 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7ff8b00015b0 con 0x7ff8c01b0ae0 2021-02-10T11:32:04.334 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:04.310 7ff8be7fc700 1 -- 172.21.2.102:0/3645692514 >> v1:172.21.2.102:6789/0 conn(0x7ff8c01b0ae0 legacy=0x7ff8c0108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:04.334 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:04.310 7ff8be7fc700 1 --2- 172.21.2.102:0/3645692514 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff8ac003a60 0x7ff8ac005f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:04.334 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:04.310 7ff8be7fc700 1 --2- 172.21.2.102:0/3645692514 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff8ac04a7d0 0x7ff8ac04cc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:04.334 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:04.310 7ff8be7fc700 1 --2- 172.21.2.102:0/3645692514 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff8ac04d500 0x7ff8ac04fa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:04.334 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:04.310 7ff8be7fc700 1 -- 172.21.2.102:0/3645692514 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8ac0502d0 con 0x7ff8ac04d500 2021-02-10T11:32:04.334 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:04.310 7ff8be7fc700 1 -- 172.21.2.102:0/3645692514 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8ac0504f0 con 0x7ff8ac003a60 2021-02-10T11:32:04.335 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:04.310 7ff8be7fc700 1 -- 172.21.2.102:0/3645692514 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8ac050710 con 0x7ff8ac04a7d0 2021-02-10T11:32:04.335 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:04.310 7ff8bffff700 1 --2- 172.21.2.102:0/3645692514 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff8ac04d500 0x7ff8ac04fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:04.335 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:04.310 7ff8beffd700 1 --2- 172.21.2.102:0/3645692514 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff8ac04a7d0 0x7ff8ac04cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:04.335 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:04.310 7ff8bf7fe700 1 --2- 172.21.2.102:0/3645692514 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff8ac003a60 0x7ff8ac005f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:04.335 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:04.310 7ff8bffff700 1 -- 172.21.2.102:0/3645692514 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff8ac04a7d0 msgr2=0x7ff8ac04cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:32:04.335 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:04.310 7ff8bffff700 1 --2- 172.21.2.102:0/3645692514 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff8ac04a7d0 0x7ff8ac04cc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:04.336 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:04.310 7ff8bffff700 1 -- 172.21.2.102:0/3645692514 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff8ac003a60 msgr2=0x7ff8ac005f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:32:04.336 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:04.310 7ff8bffff700 1 --2- 172.21.2.102:0/3645692514 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff8ac003a60 0x7ff8ac005f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:04.336 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:04.310 7ff8bffff700 1 -- 172.21.2.102:0/3645692514 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8ac0509b0 con 0x7ff8ac04d500 2021-02-10T11:32:04.336 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:04.310 7ff8bf7fe700 1 --2- 172.21.2.102:0/3645692514 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff8ac003a60 0x7ff8ac005f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:32:04.336 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:04.310 7ff8bffff700 1 --2- 172.21.2.102:0/3645692514 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff8ac04d500 0x7ff8ac04fa20 secure :-1 s=READY pgs=2229 cs=0 l=1 rev1=1 rx=0x7ff8b8009a40 tx=0x7ff8b8009de0).ready entity=mon.0 client_cookie=2cf12fd8e9349120 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:04.336 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:04.310 7ff8be7fc700 1 -- 172.21.2.102:0/3645692514 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff8b800b510 con 0x7ff8ac04d500 2021-02-10T11:32:04.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:04.310 7ff8be7fc700 1 -- 172.21.2.102:0/3645692514 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8b8003030 con 0x7ff8ac04d500 2021-02-10T11:32:04.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:04.310 7ff8be7fc700 1 -- 172.21.2.102:0/3645692514 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff8b800b750 con 0x7ff8ac04d500 2021-02-10T11:32:04.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:04.310 7ff8c5e21700 1 -- 172.21.2.102:0/3645692514 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff8ac04d500 msgr2=0x7ff8ac04fa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:04.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:04.310 7ff8c5e21700 1 --2- 172.21.2.102:0/3645692514 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff8ac04d500 0x7ff8ac04fa20 secure :-1 s=READY pgs=2229 cs=0 l=1 rev1=1 rx=0x7ff8b8009a40 tx=0x7ff8b8009de0).stop 2021-02-10T11:32:04.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:04.310 7ff8bffff700 1 -- 172.21.2.102:0/3645692514 reap_dead start 2021-02-10T11:32:04.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:04.311 7ff8c5e21700 1 -- 172.21.2.102:0/3645692514 shutdown_connections 2021-02-10T11:32:04.338 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:04.311 7ff8c5e21700 1 -- 172.21.2.102:0/3645692514 >> 172.21.2.102:0/3645692514 conn(0x7ff8c00ce420 msgr2=0x7ff8c00d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:04.338 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:04.311 7ff8c5e21700 1 -- 172.21.2.102:0/3645692514 shutdown_connections 2021-02-10T11:32:04.338 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:04.311 7ff8c5e21700 1 -- 172.21.2.102:0/3645692514 wait complete. 2021-02-10T11:32:04.338 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:04.326 7ff8c5e21700 1 Processor -- start 2021-02-10T11:32:04.338 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:04.326 7ff8c5e21700 1 -- start start 2021-02-10T11:32:04.339 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:04.326 7ff8c5e21700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff8c01a79a0 0x7ff8c01a7da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:04.339 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:04.326 7ff8c5e21700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff8c00bdb40 0x7ff8c00bdf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:04.339 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:04.326 7ff8c5e21700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff8c0082ca0 0x7ff8c00830a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:04.339 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:04.326 7ff8c5e21700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8c019d990 con 0x7ff8c01a79a0 2021-02-10T11:32:04.339 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:04.326 7ff8c5e21700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8c019dbb0 con 0x7ff8c0082ca0 2021-02-10T11:32:04.339 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:04.326 7ff8c5e21700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8c00836c0 con 0x7ff8c00bdb40 2021-02-10T11:32:04.339 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:04.326 7ff8bf7fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff8c01a79a0 0x7ff8c01a7da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:04.339 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:04.327 7ff8bf7fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff8c01a79a0 0x7ff8c01a7da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:34230/0 (socket says 172.21.2.102:34230) 2021-02-10T11:32:04.339 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:04.327 7ff8bf7fe700 1 -- 172.21.2.102:0/3527390923 learned_addr learned my addr 172.21.2.102:0/3527390923 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:32:04.339 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:04.327 7ff8bf7fe700 1 -- 172.21.2.102:0/3527390923 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff8c00bdb40 msgr2=0x7ff8c00bdf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:04.339 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:04.327 7ff8bf7fe700 1 --2- 172.21.2.102:0/3527390923 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff8c00bdb40 0x7ff8c00bdf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:04.340 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:04.327 7ff8bf7fe700 1 -- 172.21.2.102:0/3527390923 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff8c0082ca0 msgr2=0x7ff8c00830a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:04.340 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:04.327 7ff8bf7fe700 1 --2- 172.21.2.102:0/3527390923 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff8c0082ca0 0x7ff8c00830a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:04.340 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:04.327 7ff8bf7fe700 1 -- 172.21.2.102:0/3527390923 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8c00838e0 con 0x7ff8c01a79a0 2021-02-10T11:32:04.340 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:04.327 7ff8bf7fe700 1 --2- 172.21.2.102:0/3527390923 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff8c01a79a0 0x7ff8c01a7da0 secure :-1 s=READY pgs=2230 cs=0 l=1 rev1=1 rx=0x7ff8a800b580 tx=0x7ff8a800b920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:04.340 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:04.327 7ff8b7fff700 1 -- 172.21.2.102:0/3527390923 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff8a800bd10 con 0x7ff8c01a79a0 2021-02-10T11:32:04.340 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:04.327 7ff8b7fff700 1 -- 172.21.2.102:0/3527390923 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8a8000c90 con 0x7ff8c01a79a0 2021-02-10T11:32:04.341 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:04.327 7ff8b7fff700 1 -- 172.21.2.102:0/3527390923 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff8a80022b0 con 0x7ff8c01a79a0 2021-02-10T11:32:04.341 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:04.327 7ff8c5e21700 1 -- 172.21.2.102:0/3527390923 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8c00838e0 con 0x7ff8c01a79a0 2021-02-10T11:32:04.341 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:04.327 7ff8c5e21700 1 -- 172.21.2.102:0/3527390923 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8c00838e0 con 0x7ff8c01a79a0 2021-02-10T11:32:04.341 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:04.328 7ff8b7fff700 1 -- 172.21.2.102:0/3527390923 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ff8a8006740 con 0x7ff8c01a79a0 2021-02-10T11:32:04.341 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:04.328 7ff8b7fff700 1 --2- 172.21.2.102:0/3527390923 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff8a40186d0 0x7ff8a401abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:04.341 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:04.328 7ff8b7fff700 1 -- 172.21.2.102:0/3527390923 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5573+0+0 (secure 0 0 0) 0x7ff8a80327b0 con 0x7ff8c01a79a0 2021-02-10T11:32:04.341 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:04.328 7ff8beffd700 1 --2- 172.21.2.102:0/3527390923 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff8a40186d0 0x7ff8a401abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:04.341 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:04.329 7ff8beffd700 1 --2- 172.21.2.102:0/3527390923 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff8a40186d0 0x7ff8a401abd0 crc :-1 s=READY pgs=983 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:04.360 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:04.358 7ff8c5e21700 1 -- 172.21.2.102:0/3527390923 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8c0082a70 con 0x7ff8c01a79a0 2021-02-10T11:32:04.363 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:04.361 7ff8b7fff700 1 -- 172.21.2.102:0/3527390923 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ff8a800bd10 con 0x7ff8c01a79a0 2021-02-10T11:32:04.491 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:04.490 7ff8c5e21700 1 -- 172.21.2.102:0/3527390923 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7ff8c0082a70 con 0x7ff8c01a79a0 2021-02-10T11:32:04.492 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:04.490 7ff8b7fff700 1 -- 172.21.2.102:0/3527390923 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v112) v1 ==== 104+0+10 (secure 0 0 0) 0x7ff8a8006740 con 0x7ff8c01a79a0 2021-02-10T11:32:04.492 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:32:04.494 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:04.492 7ff8c5e21700 1 -- 172.21.2.102:0/3527390923 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff8a40186d0 msgr2=0x7ff8a401abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:04.494 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:04.492 7ff8c5e21700 1 --2- 172.21.2.102:0/3527390923 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff8a40186d0 0x7ff8a401abd0 crc :-1 s=READY pgs=983 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:04.494 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:04.492 7ff8c5e21700 1 -- 172.21.2.102:0/3527390923 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff8c01a79a0 msgr2=0x7ff8c01a7da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:04.494 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:04.492 7ff8c5e21700 1 --2- 172.21.2.102:0/3527390923 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff8c01a79a0 0x7ff8c01a7da0 secure :-1 s=READY pgs=2230 cs=0 l=1 rev1=1 rx=0x7ff8a800b580 tx=0x7ff8a800b920).stop 2021-02-10T11:32:04.494 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:04.493 7ff8c5e21700 1 -- 172.21.2.102:0/3527390923 shutdown_connections 2021-02-10T11:32:04.494 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:04.493 7ff8c5e21700 1 --2- 172.21.2.102:0/3527390923 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff8a40186d0 0x7ff8a401abd0 unknown :-1 s=CLOSED pgs=983 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:04.494 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:04.493 7ff8c5e21700 1 --2- 172.21.2.102:0/3527390923 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff8c0082ca0 0x7ff8c00830a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:04.494 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:04.493 7ff8c5e21700 1 --2- 172.21.2.102:0/3527390923 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff8c00bdb40 0x7ff8c00bdf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:04.494 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:04.493 7ff8c5e21700 1 --2- 172.21.2.102:0/3527390923 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff8c01a79a0 0x7ff8c01a7da0 unknown :-1 s=CLOSED pgs=2230 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:04.494 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:04.493 7ff8c5e21700 1 -- 172.21.2.102:0/3527390923 >> 172.21.2.102:0/3527390923 conn(0x7ff8c00bef00 msgr2=0x7ff8c009f200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:04.495 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:04.493 7ff8c5e21700 1 -- 172.21.2.102:0/3527390923 shutdown_connections 2021-02-10T11:32:04.495 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:04.493 7ff8c5e21700 1 -- 172.21.2.102:0/3527390923 wait complete. 2021-02-10T11:32:04.502 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-02-10T11:32:04.935 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:04.917 7fd671a57700 1 Processor -- start 2021-02-10T11:32:04.935 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:04.917 7fd671a57700 1 -- start start 2021-02-10T11:32:04.935 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:04.918 7fd671a57700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fd66c0d1c70 0x7fd66c0cef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:04.936 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:04.918 7fd671a57700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd66c0d0170 con 0x7fd66c0d1870 2021-02-10T11:32:04.936 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:04.918 7fd671a57700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd66c0fe1c0 con 0x7fd66c0cf580 2021-02-10T11:32:04.936 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:04.918 7fd671a57700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd66c1b0cb0 con 0x7fd66c0d1c70 2021-02-10T11:32:04.936 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:04.918 7fd66affd700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fd66c0d1c70 0x7fd66c0cef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:04.936 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:04.918 7fd66b7fe700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fd66c0d1870 0x7fd66c1b04c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:54690/0 (socket says 172.21.2.102:54690) 2021-02-10T11:32:04.937 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:04.918 7fd66affd700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fd66c0d1c70 0x7fd66c0cef60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:34234/0 (socket says 172.21.2.102:34234) 2021-02-10T11:32:04.937 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:04.918 7fd66b7fe700 1 -- 172.21.2.102:0/2016252223 learned_addr learned my addr 172.21.2.102:0/2016252223 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:32:04.937 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:04.918 7fd66affd700 1 -- 172.21.2.102:0/2016252223 >> v1:172.21.2.113:6789/0 conn(0x7fd66c0cf580 legacy=0x7fd66c0cf980 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:32:04.937 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:04.918 7fd66affd700 1 -- 172.21.2.102:0/2016252223 >> v1:172.21.2.102:6789/0 conn(0x7fd66c0d1870 legacy=0x7fd66c1b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:04.937 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:04.918 7fd66affd700 1 -- 172.21.2.102:0/2016252223 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd66c109a60 con 0x7fd66c0d1c70 2021-02-10T11:32:04.937 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:04.918 7fd66affd700 1 --2- 172.21.2.102:0/2016252223 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd66c0d1c70 0x7fd66c0cef60 secure :-1 s=READY pgs=2232 cs=0 l=1 rev1=1 rx=0x7fd654006650 tx=0x7fd654009a00).ready entity=mon.0 client_cookie=a1807a4aedba2be2 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:04.938 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:04.918 7fd66a7fc700 1 -- 172.21.2.102:0/2016252223 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd6540009b0 con 0x7fd66c0d1c70 2021-02-10T11:32:04.938 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:04.919 7fd66a7fc700 1 -- 172.21.2.102:0/2016252223 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd65400c0c0 con 0x7fd66c0d1c70 2021-02-10T11:32:04.938 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:04.919 7fd66a7fc700 1 -- 172.21.2.102:0/2016252223 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd65400c5a0 con 0x7fd66c0d1c70 2021-02-10T11:32:04.938 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:04.919 7fd671a57700 1 -- 172.21.2.102:0/2016252223 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd66c0d1c70 msgr2=0x7fd66c0cef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:04.938 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:04.919 7fd671a57700 1 --2- 172.21.2.102:0/2016252223 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd66c0d1c70 0x7fd66c0cef60 secure :-1 s=READY pgs=2232 cs=0 l=1 rev1=1 rx=0x7fd654006650 tx=0x7fd654009a00).stop 2021-02-10T11:32:04.939 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:04.919 7fd671a57700 1 -- 172.21.2.102:0/2016252223 shutdown_connections 2021-02-10T11:32:04.939 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:04.919 7fd671a57700 1 --2- 172.21.2.102:0/2016252223 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd66c0d1c70 0x7fd66c0cef60 unknown :-1 s=CLOSED pgs=2232 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:04.939 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:04.919 7fd671a57700 1 -- 172.21.2.102:0/2016252223 >> 172.21.2.102:0/2016252223 conn(0x7fd66c0ce420 msgr2=0x7fd66c0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:04.939 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:04.919 7fd671a57700 1 -- 172.21.2.102:0/2016252223 shutdown_connections 2021-02-10T11:32:04.939 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:04.919 7fd671a57700 1 -- 172.21.2.102:0/2016252223 wait complete. 2021-02-10T11:32:04.942 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:04.934 7fd671a57700 1 Processor -- start 2021-02-10T11:32:04.942 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:04.934 7fd671a57700 1 -- start start 2021-02-10T11:32:04.942 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:04.934 7fd671a57700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd66c1c9af0 0x7fd66c1c9ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:04.942 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:04.934 7fd671a57700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd66c0b0910 0x7fd66c0b0d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:04.943 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:04.934 7fd671a57700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd66c0b1330 0x7fd66c0b1730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:04.943 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:04.934 7fd671a57700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd66c0b1d50 con 0x7fd66c0b0910 2021-02-10T11:32:04.943 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:04.934 7fd671a57700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd66c0a3bb0 con 0x7fd66c1c9af0 2021-02-10T11:32:04.943 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:04.934 7fd671a57700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd66c0a3dd0 con 0x7fd66c0b1330 2021-02-10T11:32:04.943 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:04.934 7fd66affd700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd66c0b0910 0x7fd66c0b0d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:04.943 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:04.934 7fd66affd700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd66c0b0910 0x7fd66c0b0d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:34242/0 (socket says 172.21.2.102:34242) 2021-02-10T11:32:04.944 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:04.934 7fd66affd700 1 -- 172.21.2.102:0/686230594 learned_addr learned my addr 172.21.2.102:0/686230594 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:32:04.944 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:04.935 7fd66affd700 1 -- 172.21.2.102:0/686230594 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd66c0b1330 msgr2=0x7fd66c0b1730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:04.944 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:04.935 7fd66affd700 1 --2- 172.21.2.102:0/686230594 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd66c0b1330 0x7fd66c0b1730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:04.944 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:04.935 7fd66affd700 1 -- 172.21.2.102:0/686230594 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd66c1c9af0 msgr2=0x7fd66c1c9ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:04.944 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:04.935 7fd66affd700 1 --2- 172.21.2.102:0/686230594 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd66c1c9af0 0x7fd66c1c9ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:04.945 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:04.935 7fd66affd700 1 -- 172.21.2.102:0/686230594 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd66c0a3ff0 con 0x7fd66c0b0910 2021-02-10T11:32:04.945 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:04.935 7fd66affd700 1 --2- 172.21.2.102:0/686230594 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd66c0b0910 0x7fd66c0b0d10 secure :-1 s=READY pgs=2233 cs=0 l=1 rev1=1 rx=0x7fd65400b590 tx=0x7fd65400b930).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:04.945 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:04.935 7fd663fff700 1 -- 172.21.2.102:0/686230594 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd65400bd20 con 0x7fd66c0b0910 2021-02-10T11:32:04.945 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:04.935 7fd663fff700 1 -- 172.21.2.102:0/686230594 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd654000c90 con 0x7fd66c0b0910 2021-02-10T11:32:04.945 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:04.935 7fd663fff700 1 -- 172.21.2.102:0/686230594 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd6540022b0 con 0x7fd66c0b0910 2021-02-10T11:32:04.945 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:04.935 7fd671a57700 1 -- 172.21.2.102:0/686230594 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd66c0a3ff0 con 0x7fd66c0b0910 2021-02-10T11:32:04.946 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:04.935 7fd671a57700 1 -- 172.21.2.102:0/686230594 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd66c0a3ff0 con 0x7fd66c0b0910 2021-02-10T11:32:04.946 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:04.936 7fd663fff700 1 -- 172.21.2.102:0/686230594 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd654006a10 con 0x7fd66c0b0910 2021-02-10T11:32:04.946 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:04.936 7fd663fff700 1 --2- 172.21.2.102:0/686230594 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd6500186d0 0x7fd65001abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:04.946 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:04.936 7fd663fff700 1 -- 172.21.2.102:0/686230594 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fd6540327a0 con 0x7fd66c0b0910 2021-02-10T11:32:04.947 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:04.936 7fd66b7fe700 1 --2- 172.21.2.102:0/686230594 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd6500186d0 0x7fd65001abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:04.947 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:04.936 7fd66b7fe700 1 --2- 172.21.2.102:0/686230594 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd6500186d0 0x7fd65001abd0 crc :-1 s=READY pgs=984 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:04.968 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:04.967 7fd671a57700 1 -- 172.21.2.102:0/686230594 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd66c0807d0 con 0x7fd66c0b0910 2021-02-10T11:32:04.971 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:04.970 7fd663fff700 1 -- 172.21.2.102:0/686230594 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd654000c90 con 0x7fd66c0b0910 2021-02-10T11:32:05.099 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:05.098 7fd671a57700 1 -- 172.21.2.102:0/686230594 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7fd66c088070 con 0x7fd66c0b0910 2021-02-10T11:32:05.100 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:05.098 7fd663fff700 1 -- 172.21.2.102:0/686230594 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v112) v1 ==== 100+0+10 (secure 0 0 0) 0x7fd65400bd20 con 0x7fd66c0b0910 2021-02-10T11:32:05.100 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:32:05.102 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:05.100 7fd671a57700 1 -- 172.21.2.102:0/686230594 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd6500186d0 msgr2=0x7fd65001abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:05.102 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:05.100 7fd671a57700 1 --2- 172.21.2.102:0/686230594 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd6500186d0 0x7fd65001abd0 crc :-1 s=READY pgs=984 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:05.102 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:05.100 7fd671a57700 1 -- 172.21.2.102:0/686230594 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd66c0b0910 msgr2=0x7fd66c0b0d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:05.102 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:05.100 7fd671a57700 1 --2- 172.21.2.102:0/686230594 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd66c0b0910 0x7fd66c0b0d10 secure :-1 s=READY pgs=2233 cs=0 l=1 rev1=1 rx=0x7fd65400b590 tx=0x7fd65400b930).stop 2021-02-10T11:32:05.102 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:05.101 7fd671a57700 1 -- 172.21.2.102:0/686230594 shutdown_connections 2021-02-10T11:32:05.103 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:05.101 7fd671a57700 1 --2- 172.21.2.102:0/686230594 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd66c0b1330 0x7fd66c0b1730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:05.103 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:05.101 7fd671a57700 1 --2- 172.21.2.102:0/686230594 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd66c0b0910 0x7fd66c0b0d10 unknown :-1 s=CLOSED pgs=2233 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:05.103 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:05.101 7fd671a57700 1 --2- 172.21.2.102:0/686230594 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd6500186d0 0x7fd65001abd0 unknown :-1 s=CLOSED pgs=984 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:05.103 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:05.101 7fd671a57700 1 --2- 172.21.2.102:0/686230594 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd66c1c9af0 0x7fd66c1c9ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:05.103 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:05.101 7fd671a57700 1 -- 172.21.2.102:0/686230594 >> 172.21.2.102:0/686230594 conn(0x7fd66c080a50 msgr2=0x7fd66c0ffa50 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:05.103 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:05.101 7fd671a57700 1 -- 172.21.2.102:0/686230594 shutdown_connections 2021-02-10T11:32:05.103 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:05.101 7fd671a57700 1 -- 172.21.2.102:0/686230594 wait complete. 2021-02-10T11:32:05.111 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-02-10T11:32:05.111 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:32:05.540 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:05.521 7f2537333700 1 Processor -- start 2021-02-10T11:32:05.540 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:05.522 7f2537333700 1 -- start start 2021-02-10T11:32:05.540 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:05.522 7f2537333700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f25300d1870 0x7f25301b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:05.540 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:05.522 7f2537333700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f25301b0a10 0x7f25300cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:05.540 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:05.522 7f2537333700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f25300d1c70 0x7f25300cf950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:05.540 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:05.522 7f2537333700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25300fe230 con 0x7f25301b0a10 2021-02-10T11:32:05.541 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:05.522 7f2537333700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25300e5ed0 con 0x7f25300d1c70 2021-02-10T11:32:05.541 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:05.522 7f2537333700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25300f0b10 con 0x7f25300d1870 2021-02-10T11:32:05.541 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:05.522 7f253494e700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f25301b0a10 0x7f25300cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:05.541 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:05.522 7f253494e700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f25301b0a10 0x7f25300cf330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:34248/0 (socket says 172.21.2.102:34248) 2021-02-10T11:32:05.541 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:05.522 7f253494e700 1 -- 172.21.2.102:0/2880841788 learned_addr learned my addr 172.21.2.102:0/2880841788 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:32:05.541 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:05.522 7f253494e700 1 -- 172.21.2.102:0/2880841788 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f25300d1870 msgr2=0x7f25301b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:32:05.541 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:05.522 7f253494e700 1 --2- 172.21.2.102:0/2880841788 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f25300d1870 0x7f25301b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:05.541 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:05.522 7f253494e700 1 -- 172.21.2.102:0/2880841788 >> v2:172.21.2.113:3300/0 conn(0x7f25300d1c70 msgr2=0x7f25300cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:32:05.541 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:05.522 7f253494e700 1 --2- 172.21.2.102:0/2880841788 >> v2:172.21.2.113:3300/0 conn(0x7f25300d1c70 0x7f25300cf950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:05.541 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:05.522 7f253494e700 1 -- 172.21.2.102:0/2880841788 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25300cff70 con 0x7f25301b0a10 2021-02-10T11:32:05.541 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:05.523 7f253494e700 1 --2- 172.21.2.102:0/2880841788 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f25301b0a10 0x7f25300cf330 secure :-1 s=READY pgs=2234 cs=0 l=1 rev1=1 rx=0x7f2520006700 tx=0x7f2520009b80).ready entity=mon.0 client_cookie=cddbdfc58514233d server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:05.541 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:05.523 7f2527fff700 1 -- 172.21.2.102:0/2880841788 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f252000c0c0 con 0x7f25301b0a10 2021-02-10T11:32:05.541 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:05.523 7f2527fff700 1 -- 172.21.2.102:0/2880841788 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f252000c440 con 0x7f25301b0a10 2021-02-10T11:32:05.542 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:05.523 7f2527fff700 1 -- 172.21.2.102:0/2880841788 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f252000c920 con 0x7f25301b0a10 2021-02-10T11:32:05.542 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:05.523 7f2537333700 1 -- 172.21.2.102:0/2880841788 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f25301b0a10 msgr2=0x7f25300cf330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:05.542 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:05.523 7f2537333700 1 --2- 172.21.2.102:0/2880841788 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f25301b0a10 0x7f25300cf330 secure :-1 s=READY pgs=2234 cs=0 l=1 rev1=1 rx=0x7f2520006700 tx=0x7f2520009b80).stop 2021-02-10T11:32:05.542 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:05.524 7f2537333700 1 -- 172.21.2.102:0/2880841788 shutdown_connections 2021-02-10T11:32:05.542 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:05.524 7f2537333700 1 --2- 172.21.2.102:0/2880841788 >> v2:172.21.2.113:3300/0 conn(0x7f25300d1c70 0x7f25300cf950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:05.542 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:05.524 7f2537333700 1 --2- 172.21.2.102:0/2880841788 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f25301b0a10 0x7f25300cf330 unknown :-1 s=CLOSED pgs=2234 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:05.542 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:05.524 7f2537333700 1 --2- 172.21.2.102:0/2880841788 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f25300d1870 0x7f25301b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:05.542 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:05.524 7f2537333700 1 -- 172.21.2.102:0/2880841788 >> 172.21.2.102:0/2880841788 conn(0x7f25300ce450 msgr2=0x7f25300d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:05.542 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:05.524 7f2537333700 1 -- 172.21.2.102:0/2880841788 shutdown_connections 2021-02-10T11:32:05.542 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:05.524 7f2537333700 1 -- 172.21.2.102:0/2880841788 wait complete. 2021-02-10T11:32:05.542 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:05.539 7f2537333700 1 Processor -- start 2021-02-10T11:32:05.542 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:05.539 7f2537333700 1 -- start start 2021-02-10T11:32:05.544 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:05.539 7f2537333700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f25301b7cf0 0x7f253019cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:05.544 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:05.539 7f2537333700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f25301b80f0 0x7f25301b84f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:05.544 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:05.539 7f2537333700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f253019da20 0x7f253019de20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:05.544 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:05.539 7f2537333700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25300a77c0 con 0x7f253019da20 2021-02-10T11:32:05.544 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:05.539 7f2537333700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f253019d4a0 con 0x7f25301b7cf0 2021-02-10T11:32:05.544 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:05.539 7f2537333700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25300954f0 con 0x7f25301b80f0 2021-02-10T11:32:05.544 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:05.539 7f2535950700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f253019da20 0x7f253019de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:05.544 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:05.539 7f2535950700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f253019da20 0x7f253019de20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:34256/0 (socket says 172.21.2.102:34256) 2021-02-10T11:32:05.544 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:05.539 7f2535950700 1 -- 172.21.2.102:0/4087616934 learned_addr learned my addr 172.21.2.102:0/4087616934 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:32:05.545 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:05.540 7f2535950700 1 -- 172.21.2.102:0/4087616934 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f25301b80f0 msgr2=0x7f25301b84f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:05.545 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:05.540 7f2535950700 1 --2- 172.21.2.102:0/4087616934 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f25301b80f0 0x7f25301b84f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:05.545 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:05.540 7f2535950700 1 -- 172.21.2.102:0/4087616934 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f25301b7cf0 msgr2=0x7f253019cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:05.545 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:05.540 7f2535950700 1 --2- 172.21.2.102:0/4087616934 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f25301b7cf0 0x7f253019cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:05.545 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:05.540 7f2535950700 1 -- 172.21.2.102:0/4087616934 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25301b8d60 con 0x7f253019da20 2021-02-10T11:32:05.545 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:05.540 7f2535950700 1 --2- 172.21.2.102:0/4087616934 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f253019da20 0x7f253019de20 secure :-1 s=READY pgs=2235 cs=0 l=1 rev1=1 rx=0x7f252c00b580 tx=0x7f252c00b920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:05.545 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:05.540 7f2525ffb700 1 -- 172.21.2.102:0/4087616934 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f252c006240 con 0x7f253019da20 2021-02-10T11:32:05.545 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:05.540 7f2525ffb700 1 -- 172.21.2.102:0/4087616934 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f252c000c10 con 0x7f253019da20 2021-02-10T11:32:05.545 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:05.540 7f2537333700 1 -- 172.21.2.102:0/4087616934 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25301b8d60 con 0x7f253019da20 2021-02-10T11:32:05.545 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:05.540 7f2525ffb700 1 -- 172.21.2.102:0/4087616934 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f252c0056e0 con 0x7f253019da20 2021-02-10T11:32:05.545 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:05.540 7f2537333700 1 -- 172.21.2.102:0/4087616934 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f253019e730 con 0x7f253019da20 2021-02-10T11:32:05.545 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:05.541 7f2525ffb700 1 -- 172.21.2.102:0/4087616934 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f252c006030 con 0x7f253019da20 2021-02-10T11:32:05.546 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:05.541 7f2525ffb700 1 --2- 172.21.2.102:0/4087616934 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f25100186d0 0x7f251001abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:05.546 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:05.541 7f2525ffb700 1 -- 172.21.2.102:0/4087616934 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f252c032650 con 0x7f253019da20 2021-02-10T11:32:05.546 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:05.541 7f253514f700 1 --2- 172.21.2.102:0/4087616934 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f25100186d0 0x7f251001abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:05.546 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:05.541 7f253514f700 1 --2- 172.21.2.102:0/4087616934 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f25100186d0 0x7f251001abd0 crc :-1 s=READY pgs=985 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:05.573 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:05.572 7f2537333700 1 -- 172.21.2.102:0/4087616934 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25300ebc00 con 0x7f253019da20 2021-02-10T11:32:05.575 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:05.574 7f2525ffb700 1 -- 172.21.2.102:0/4087616934 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f252c000bc0 con 0x7f253019da20 2021-02-10T11:32:05.704 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:05.702 7f2537333700 1 -- 172.21.2.102:0/4087616934 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f25300f0b10 con 0x7f253019da20 2021-02-10T11:32:05.704 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:05.703 7f2525ffb700 1 -- 172.21.2.102:0/4087616934 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 91 v91) v1 ==== 94+0+2104 (secure 0 0 0) 0x7f252c006030 con 0x7f253019da20 2021-02-10T11:32:05.705 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:32:05.705 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":91,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10404,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.102:6835/1661316191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6834","nonce":1661316191},{"type":"v1","addr":"172.21.2.102:6835","nonce":1661316191}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:31:48.259584","modified":"2021-02-10 11:31:50.267046","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10401},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10401":{"gid":10401,"name":"b","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.2.113:6833/313019011","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6832","nonce":313019011},{"type":"v1","addr":"172.21.2.113:6833","nonce":313019011}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[23],"metadata_pool":22,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":11}]} 2021-02-10T11:32:05.705 INFO:teuthology.orchestra.run.gibba002.stderr:dumped fsmap epoch 91 2021-02-10T11:32:05.705 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:05.704 7f2537333700 1 -- 172.21.2.102:0/4087616934 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f25100186d0 msgr2=0x7f251001abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:05.706 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:05.704 7f2537333700 1 --2- 172.21.2.102:0/4087616934 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f25100186d0 0x7f251001abd0 crc :-1 s=READY pgs=985 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:05.706 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:05.704 7f2537333700 1 -- 172.21.2.102:0/4087616934 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f253019da20 msgr2=0x7f253019de20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:05.706 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:05.704 7f2537333700 1 --2- 172.21.2.102:0/4087616934 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f253019da20 0x7f253019de20 secure :-1 s=READY pgs=2235 cs=0 l=1 rev1=1 rx=0x7f252c00b580 tx=0x7f252c00b920).stop 2021-02-10T11:32:05.706 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:05.705 7f2537333700 1 -- 172.21.2.102:0/4087616934 shutdown_connections 2021-02-10T11:32:05.706 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:05.705 7f2537333700 1 --2- 172.21.2.102:0/4087616934 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f253019da20 0x7f253019de20 unknown :-1 s=CLOSED pgs=2235 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:05.706 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:05.705 7f2537333700 1 --2- 172.21.2.102:0/4087616934 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f25301b80f0 0x7f25301b84f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:05.707 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:05.705 7f2537333700 1 --2- 172.21.2.102:0/4087616934 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f25100186d0 0x7f251001abd0 unknown :-1 s=CLOSED pgs=985 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:05.707 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:05.705 7f2537333700 1 --2- 172.21.2.102:0/4087616934 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f25301b7cf0 0x7f253019cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:05.707 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:05.705 7f2537333700 1 -- 172.21.2.102:0/4087616934 >> 172.21.2.102:0/4087616934 conn(0x7f25300ed810 msgr2=0x7f2530094980 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:05.707 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:05.705 7f2537333700 1 -- 172.21.2.102:0/4087616934 shutdown_connections 2021-02-10T11:32:05.707 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:05.705 7f2537333700 1 -- 172.21.2.102:0/4087616934 wait complete. 2021-02-10T11:32:05.916 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 91, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:31:48.259584', 'modified': '2021-02-10 11:31:50.267046', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 10401}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10401': {'gid': 10401, 'name': 'b', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.113:6833/313019011', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.113:6832', 'nonce': 313019011}, {'type': 'v1', 'addr': '172.21.2.113:6833', 'nonce': 313019011}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [23], 'metadata_pool': 22, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:32:05.916 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:32:05.917 DEBUG:teuthology.orchestra.run.gibba013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:32:06.315 INFO:teuthology.orchestra.run.gibba013.stdout:{ 2021-02-10T11:32:06.316 INFO:teuthology.orchestra.run.gibba013.stdout: "cluster_fsid": "68db30ea-dbce-4aef-a6df-f01c6fea0513", 2021-02-10T11:32:06.316 INFO:teuthology.orchestra.run.gibba013.stdout: "whoami": 0, 2021-02-10T11:32:06.316 INFO:teuthology.orchestra.run.gibba013.stdout: "id": 10401, 2021-02-10T11:32:06.316 INFO:teuthology.orchestra.run.gibba013.stdout: "want_state": "up:active", 2021-02-10T11:32:06.316 INFO:teuthology.orchestra.run.gibba013.stdout: "state": "up:active", 2021-02-10T11:32:06.316 INFO:teuthology.orchestra.run.gibba013.stdout: "rank_uptime": 18.05405545, 2021-02-10T11:32:06.316 INFO:teuthology.orchestra.run.gibba013.stdout: "mdsmap_epoch": 91, 2021-02-10T11:32:06.316 INFO:teuthology.orchestra.run.gibba013.stdout: "osdmap_epoch": 112, 2021-02-10T11:32:06.316 INFO:teuthology.orchestra.run.gibba013.stdout: "osdmap_epoch_barrier": 112, 2021-02-10T11:32:06.316 INFO:teuthology.orchestra.run.gibba013.stdout: "uptime": 23.700445329000001 2021-02-10T11:32:06.317 INFO:teuthology.orchestra.run.gibba013.stdout:} 2021-02-10T11:32:06.320 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "68db30ea-dbce-4aef-a6df-f01c6fea0513", "whoami": 0, "id": 10401, "want_state": "up:active", "state": "up:active", "rank_uptime": 18.05405545, "mdsmap_epoch": 91, "osdmap_epoch": 112, "osdmap_epoch_barrier": 112, "uptime": 23.700445329000001 } 2021-02-10T11:32:06.320 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2021-02-10T11:32:06.320 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2021-02-10T11:32:06.320 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.1 at ubuntu@gibba002.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.1... 2021-02-10T11:32:06.320 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:32:06.321 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec mkdir -p /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:32:06.658 DEBUG:teuthology.orchestra.run.gibba002:> sudo modprobe fuse 2021-02-10T11:32:07.002 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:32:07.002 DEBUG:teuthology.orchestra.run.gibba002:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:32:07.347 INFO:teuthology.orchestra.run.gibba002.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:32:07.347 INFO:teuthology.orchestra.run.gibba002.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:32:07.348 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:32:07.348 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:32:07.348 DEBUG:teuthology.orchestra.run.gibba002:> ls /sys/fs/fuse/connections 2021-02-10T11:32:07.683 INFO:teuthology.orchestra.run.gibba002.stdout:43 2021-02-10T11:32:07.683 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [43] 2021-02-10T11:32:07.683 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.1 /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:32:07.686 DEBUG:teuthology.orchestra.run.gibba002:> sudo modprobe fuse 2021-02-10T11:32:08.031 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:32:08.031 DEBUG:teuthology.orchestra.run.gibba002:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:32:08.098 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba002.stderr:2021-02-10 11:32:08.097 7f856eefdf80 -1 init, newargv = 0x5612c984f1a0 newargc=9 2021-02-10T11:32:08.098 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba002.stderr:ceph-fuse[182713]: starting ceph client 2021-02-10T11:32:08.103 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba002.stderr:ceph-fuse[182713]: starting fuse 2021-02-10T11:32:08.374 INFO:teuthology.orchestra.run.gibba002.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:32:08.374 INFO:teuthology.orchestra.run.gibba002.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:32:08.375 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:32:08.375 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:32:08.375 DEBUG:teuthology.orchestra.run.gibba002:> ls /sys/fs/fuse/connections 2021-02-10T11:32:08.711 INFO:teuthology.orchestra.run.gibba002.stdout:43 2021-02-10T11:32:08.711 INFO:teuthology.orchestra.run.gibba002.stdout:44 2021-02-10T11:32:08.711 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43, 44] 2021-02-10T11:32:08.712 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:32:08.712 DEBUG:teuthology.orchestra.run.gibba002:> sudo python3 -c ' 2021-02-10T11:32:08.712 DEBUG:teuthology.orchestra.run.gibba002:> import glob 2021-02-10T11:32:08.712 DEBUG:teuthology.orchestra.run.gibba002:> import re 2021-02-10T11:32:08.712 DEBUG:teuthology.orchestra.run.gibba002:> import os 2021-02-10T11:32:08.712 DEBUG:teuthology.orchestra.run.gibba002:> import subprocess 2021-02-10T11:32:08.712 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:32:08.712 DEBUG:teuthology.orchestra.run.gibba002:> def find_socket(client_name): 2021-02-10T11:32:08.712 DEBUG:teuthology.orchestra.run.gibba002:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2021-02-10T11:32:08.712 DEBUG:teuthology.orchestra.run.gibba002:> files = glob.glob(asok_path) 2021-02-10T11:32:08.712 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:32:08.712 DEBUG:teuthology.orchestra.run.gibba002:> # Given a non-glob path, it better be there 2021-02-10T11:32:08.712 DEBUG:teuthology.orchestra.run.gibba002:> if "*" not in asok_path: 2021-02-10T11:32:08.713 DEBUG:teuthology.orchestra.run.gibba002:> assert(len(files) == 1) 2021-02-10T11:32:08.713 DEBUG:teuthology.orchestra.run.gibba002:> return files[0] 2021-02-10T11:32:08.713 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:32:08.713 DEBUG:teuthology.orchestra.run.gibba002:> for f in files: 2021-02-10T11:32:08.713 DEBUG:teuthology.orchestra.run.gibba002:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2021-02-10T11:32:08.713 DEBUG:teuthology.orchestra.run.gibba002:> if os.path.exists("/proc/{0}".format(pid)): 2021-02-10T11:32:08.713 DEBUG:teuthology.orchestra.run.gibba002:> return f 2021-02-10T11:32:08.713 DEBUG:teuthology.orchestra.run.gibba002:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2021-02-10T11:32:08.713 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:32:08.713 DEBUG:teuthology.orchestra.run.gibba002:> print(find_socket("client.1")) 2021-02-10T11:32:08.713 DEBUG:teuthology.orchestra.run.gibba002:> ' 2021-02-10T11:32:09.076 INFO:teuthology.orchestra.run.gibba002.stdout:/var/run/ceph/ceph-client.1.182713.asok 2021-02-10T11:32:09.078 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.182713.asok 2021-02-10T11:32:09.079 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:32:09.079 DEBUG:teuthology.orchestra.run.gibba002:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.182713.asok status 2021-02-10T11:32:09.467 INFO:teuthology.orchestra.run.gibba002.stdout:{ 2021-02-10T11:32:09.468 INFO:teuthology.orchestra.run.gibba002.stdout: "metadata": { 2021-02-10T11:32:09.468 INFO:teuthology.orchestra.run.gibba002.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:32:09.468 INFO:teuthology.orchestra.run.gibba002.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:32:09.468 INFO:teuthology.orchestra.run.gibba002.stdout: "entity_id": "1", 2021-02-10T11:32:09.468 INFO:teuthology.orchestra.run.gibba002.stdout: "hostname": "gibba002", 2021-02-10T11:32:09.468 INFO:teuthology.orchestra.run.gibba002.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2021-02-10T11:32:09.468 INFO:teuthology.orchestra.run.gibba002.stdout: "pid": "182713", 2021-02-10T11:32:09.468 INFO:teuthology.orchestra.run.gibba002.stdout: "root": "/" 2021-02-10T11:32:09.468 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:32:09.468 INFO:teuthology.orchestra.run.gibba002.stdout: "dentry_count": 0, 2021-02-10T11:32:09.468 INFO:teuthology.orchestra.run.gibba002.stdout: "dentry_pinned_count": 0, 2021-02-10T11:32:09.469 INFO:teuthology.orchestra.run.gibba002.stdout: "id": 10602, 2021-02-10T11:32:09.469 INFO:teuthology.orchestra.run.gibba002.stdout: "inst": { 2021-02-10T11:32:09.469 INFO:teuthology.orchestra.run.gibba002.stdout: "name": { 2021-02-10T11:32:09.469 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "client", 2021-02-10T11:32:09.469 INFO:teuthology.orchestra.run.gibba002.stdout: "num": 10602 2021-02-10T11:32:09.469 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:32:09.469 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": { 2021-02-10T11:32:09.469 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:32:09.469 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:0", 2021-02-10T11:32:09.469 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 3086934482 2021-02-10T11:32:09.469 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:32:09.469 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:32:09.469 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": { 2021-02-10T11:32:09.469 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:32:09.470 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:0", 2021-02-10T11:32:09.470 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 3086934482 2021-02-10T11:32:09.470 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:32:09.470 INFO:teuthology.orchestra.run.gibba002.stdout: "inst_str": "client.10602 172.21.2.102:0/3086934482", 2021-02-10T11:32:09.470 INFO:teuthology.orchestra.run.gibba002.stdout: "addr_str": "172.21.2.102:0/3086934482", 2021-02-10T11:32:09.470 INFO:teuthology.orchestra.run.gibba002.stdout: "inode_count": 1, 2021-02-10T11:32:09.470 INFO:teuthology.orchestra.run.gibba002.stdout: "mds_epoch": 91, 2021-02-10T11:32:09.470 INFO:teuthology.orchestra.run.gibba002.stdout: "osd_epoch": 112, 2021-02-10T11:32:09.470 INFO:teuthology.orchestra.run.gibba002.stdout: "osd_epoch_barrier": 0, 2021-02-10T11:32:09.470 INFO:teuthology.orchestra.run.gibba002.stdout: "blacklisted": false 2021-02-10T11:32:09.470 INFO:teuthology.orchestra.run.gibba002.stdout:} 2021-02-10T11:32:09.473 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:32:09.473 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:32:09.473 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:32:09.797 DEBUG:teuthology.orchestra.run.gibba002:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:32:09.801 DEBUG:teuthology.orchestra.run.gibba013:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:32:09.809 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:32:09.809 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2021-02-10T11:32:09.809 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:32:09.809 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:32:10.146 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2021-02-10T11:32:10.578 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:10.560 7f3dcdaed700 1 Processor -- start 2021-02-10T11:32:10.578 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:10.560 7f3dcdaed700 1 -- start start 2021-02-10T11:32:10.578 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:10.560 7f3dcdaed700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3dc80d1880 0x7f3dc81b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:10.578 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:10.560 7f3dcdaed700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f3dc81b0af0 0x7f3dc8108950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:10.578 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:10.560 7f3dcdaed700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3dc8109a70 con 0x7f3dc80d1c80 2021-02-10T11:32:10.578 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:10.560 7f3dcdaed700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3dc80e5f00 con 0x7f3dc81b0af0 2021-02-10T11:32:10.578 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:10.560 7f3dcdaed700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3dc8109760 con 0x7f3dc80d1880 2021-02-10T11:32:10.578 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:10.561 7f3dc6ffd700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f3dc81b0af0 0x7f3dc8108950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:10.578 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:10.561 7f3dc6ffd700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f3dc81b0af0 0x7f3dc8108950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:34272/0 (socket says 172.21.2.102:34272) 2021-02-10T11:32:10.579 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:10.561 7f3dc6ffd700 1 -- 172.21.2.102:0/3638938370 learned_addr learned my addr 172.21.2.102:0/3638938370 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:32:10.579 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:10.561 7f3dc6ffd700 1 -- 172.21.2.102:0/3638938370 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3dc80d1880 msgr2=0x7f3dc81b04d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:32:10.579 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:10.561 7f3dc6ffd700 1 --2- 172.21.2.102:0/3638938370 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3dc80d1880 0x7f3dc81b04d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:10.579 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:10.561 7f3dc6ffd700 1 -- 172.21.2.102:0/3638938370 >> v1:172.21.2.113:6789/0 conn(0x7f3dc80d1c80 legacy=0x7f3dc8108f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:32:10.579 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:10.561 7f3dc6ffd700 1 -- 172.21.2.102:0/3638938370 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3dc8088070 con 0x7f3dc81b0af0 2021-02-10T11:32:10.579 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:10.561 7f3dc6ffd700 1 --2- 172.21.2.102:0/3638938370 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3dc81b0af0 0x7f3dc8108950 secure :-1 s=READY pgs=2237 cs=0 l=1 rev1=1 rx=0x7f3dbc006700 tx=0x7f3dbc009b00).ready entity=mon.0 client_cookie=3497ab4a507f5c48 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:10.579 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:10.562 7f3dc67fc700 1 -- 172.21.2.102:0/3638938370 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3dbc00c0c0 con 0x7f3dc81b0af0 2021-02-10T11:32:10.579 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:10.562 7f3dc67fc700 1 -- 172.21.2.102:0/3638938370 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3dbc00c400 con 0x7f3dc81b0af0 2021-02-10T11:32:10.579 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:10.562 7f3dc67fc700 1 -- 172.21.2.102:0/3638938370 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3dbc00c920 con 0x7f3dc81b0af0 2021-02-10T11:32:10.579 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:10.562 7f3dcdaed700 1 -- 172.21.2.102:0/3638938370 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3dc81b0af0 msgr2=0x7f3dc8108950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:10.579 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:10.562 7f3dcdaed700 1 --2- 172.21.2.102:0/3638938370 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3dc81b0af0 0x7f3dc8108950 secure :-1 s=READY pgs=2237 cs=0 l=1 rev1=1 rx=0x7f3dbc006700 tx=0x7f3dbc009b00).stop 2021-02-10T11:32:10.579 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:10.562 7f3dcdaed700 1 -- 172.21.2.102:0/3638938370 shutdown_connections 2021-02-10T11:32:10.580 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:10.562 7f3dcdaed700 1 --2- 172.21.2.102:0/3638938370 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3dc81b0af0 0x7f3dc8108950 unknown :-1 s=CLOSED pgs=2237 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:10.580 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:10.562 7f3dcdaed700 1 --2- 172.21.2.102:0/3638938370 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3dc80d1880 0x7f3dc81b04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:10.580 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:10.562 7f3dcdaed700 1 -- 172.21.2.102:0/3638938370 >> 172.21.2.102:0/3638938370 conn(0x7f3dc80ce430 msgr2=0x7f3dc80d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:10.580 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:10.562 7f3dcdaed700 1 -- 172.21.2.102:0/3638938370 shutdown_connections 2021-02-10T11:32:10.580 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:10.562 7f3dcdaed700 1 -- 172.21.2.102:0/3638938370 wait complete. 2021-02-10T11:32:10.581 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:10.577 7f3dcdaed700 1 Processor -- start 2021-02-10T11:32:10.582 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:10.577 7f3dcdaed700 1 -- start start 2021-02-10T11:32:10.582 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:10.577 7f3dcdaed700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3dc818adc0 0x7f3dc8192200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:10.582 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:10.577 7f3dcdaed700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3dc8192820 0x7f3dc80b7ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:10.582 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:10.577 7f3dcdaed700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3dc80b80f0 0x7f3dc81a8670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:10.582 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:10.577 7f3dcdaed700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dc81a9bf0 con 0x7f3dc80b80f0 2021-02-10T11:32:10.582 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:10.577 7f3dcdaed700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dc81aead0 con 0x7f3dc8192820 2021-02-10T11:32:10.582 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:10.577 7f3dcdaed700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3dc81a80f0 con 0x7f3dc818adc0 2021-02-10T11:32:10.582 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:10.578 7f3dc7fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3dc80b80f0 0x7f3dc81a8670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:10.582 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:10.578 7f3dc7fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3dc80b80f0 0x7f3dc81a8670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:34282/0 (socket says 172.21.2.102:34282) 2021-02-10T11:32:10.582 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:10.578 7f3dc7fff700 1 -- 172.21.2.102:0/1435508613 learned_addr learned my addr 172.21.2.102:0/1435508613 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:32:10.582 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:10.578 7f3dc7fff700 1 -- 172.21.2.102:0/1435508613 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3dc818adc0 msgr2=0x7f3dc8192200 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:32:10.582 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:10.578 7f3dc7fff700 1 --2- 172.21.2.102:0/1435508613 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3dc818adc0 0x7f3dc8192200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:10.583 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:10.578 7f3dc7fff700 1 -- 172.21.2.102:0/1435508613 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3dc8192820 msgr2=0x7f3dc80b7ad0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:32:10.583 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:10.578 7f3dc7fff700 1 --2- 172.21.2.102:0/1435508613 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3dc8192820 0x7f3dc80b7ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:10.583 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:10.578 7f3dc7fff700 1 -- 172.21.2.102:0/1435508613 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3dc8191da0 con 0x7f3dc80b80f0 2021-02-10T11:32:10.583 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:10.578 7f3dc7fff700 1 --2- 172.21.2.102:0/1435508613 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3dc80b80f0 0x7f3dc81a8670 secure :-1 s=READY pgs=2238 cs=0 l=1 rev1=1 rx=0x7f3dc000b8e0 tx=0x7f3dc000bc80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:10.583 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:10.578 7f3daffff700 1 -- 172.21.2.102:0/1435508613 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3dc0000f90 con 0x7f3dc80b80f0 2021-02-10T11:32:10.583 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:10.578 7f3daffff700 1 -- 172.21.2.102:0/1435508613 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3dc000c2f0 con 0x7f3dc80b80f0 2021-02-10T11:32:10.583 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:10.578 7f3daffff700 1 -- 172.21.2.102:0/1435508613 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3dc000c750 con 0x7f3dc80b80f0 2021-02-10T11:32:10.583 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:10.578 7f3dcdaed700 1 -- 172.21.2.102:0/1435508613 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3dc8191da0 con 0x7f3dc80b80f0 2021-02-10T11:32:10.583 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:10.579 7f3dcdaed700 1 -- 172.21.2.102:0/1435508613 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3dc80b87e0 con 0x7f3dc80b80f0 2021-02-10T11:32:10.583 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:10.579 7f3daffff700 1 -- 172.21.2.102:0/1435508613 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f3dc0001220 con 0x7f3dc80b80f0 2021-02-10T11:32:10.583 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:10.579 7f3daffff700 1 --2- 172.21.2.102:0/1435508613 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3da8018730 0x7f3da801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:10.583 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:10.579 7f3daffff700 1 -- 172.21.2.102:0/1435508613 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f3dc0032a00 con 0x7f3dc80b80f0 2021-02-10T11:32:10.584 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:10.579 7f3dc77fe700 1 --2- 172.21.2.102:0/1435508613 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3da8018730 0x7f3da801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:10.584 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:10.580 7f3dc77fe700 1 --2- 172.21.2.102:0/1435508613 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3da8018730 0x7f3da801abc0 crc :-1 s=READY pgs=986 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:10.611 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:10.610 7f3dcdaed700 1 -- 172.21.2.102:0/1435508613 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3dc81aead0 con 0x7f3dc80b80f0 2021-02-10T11:32:10.614 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:10.613 7f3daffff700 1 -- 172.21.2.102:0/1435508613 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f3dc000bdc0 con 0x7f3dc80b80f0 2021-02-10T11:32:10.751 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:10.750 7f3dcdaed700 1 -- 172.21.2.102:0/1435508613 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f3dc81aead0 con 0x7f3dc80b80f0 2021-02-10T11:32:11.481 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:11.479 7f3daffff700 1 -- 172.21.2.102:0/1435508613 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v92) v1 ==== 136+0+0 (secure 0 0 0) 0x7f3dc0001220 con 0x7f3dc80b80f0 2021-02-10T11:32:11.523 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:11.522 7f3dcdaed700 1 -- 172.21.2.102:0/1435508613 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3da8018730 msgr2=0x7f3da801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:11.523 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:11.522 7f3dcdaed700 1 --2- 172.21.2.102:0/1435508613 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3da8018730 0x7f3da801abc0 crc :-1 s=READY pgs=986 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:11.523 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:11.522 7f3dcdaed700 1 -- 172.21.2.102:0/1435508613 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3dc80b80f0 msgr2=0x7f3dc81a8670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:11.524 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:11.522 7f3dcdaed700 1 --2- 172.21.2.102:0/1435508613 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3dc80b80f0 0x7f3dc81a8670 secure :-1 s=READY pgs=2238 cs=0 l=1 rev1=1 rx=0x7f3dc000b8e0 tx=0x7f3dc000bc80).stop 2021-02-10T11:32:11.524 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:11.522 7f3dcdaed700 1 -- 172.21.2.102:0/1435508613 shutdown_connections 2021-02-10T11:32:11.524 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:11.522 7f3dcdaed700 1 --2- 172.21.2.102:0/1435508613 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3dc80b80f0 0x7f3dc81a8670 unknown :-1 s=CLOSED pgs=2238 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:11.524 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:11.522 7f3dcdaed700 1 --2- 172.21.2.102:0/1435508613 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3da8018730 0x7f3da801abc0 unknown :-1 s=CLOSED pgs=986 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:11.524 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:11.522 7f3dcdaed700 1 --2- 172.21.2.102:0/1435508613 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3dc8192820 0x7f3dc80b7ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:11.524 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:11.522 7f3dcdaed700 1 --2- 172.21.2.102:0/1435508613 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3dc818adc0 0x7f3dc8192200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:11.524 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:11.522 7f3dcdaed700 1 -- 172.21.2.102:0/1435508613 >> 172.21.2.102:0/1435508613 conn(0x7f3dc809afc0 msgr2=0x7f3dc809a670 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:11.524 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:11.522 7f3dcdaed700 1 -- 172.21.2.102:0/1435508613 shutdown_connections 2021-02-10T11:32:11.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:11.522 7f3dcdaed700 1 -- 172.21.2.102:0/1435508613 wait complete. 2021-02-10T11:32:11.531 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs volume ls 2021-02-10T11:32:11.959 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:11.940 7fa7d3394700 1 Processor -- start 2021-02-10T11:32:11.959 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:11.942 7fa7d3394700 1 -- start start 2021-02-10T11:32:11.959 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:11.942 7fa7d3394700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa7cc0d0340 0x7fa7cc0ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:11.959 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:11.942 7fa7d3394700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fa7cc0d1c70 0x7fa7cc0efc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:11.959 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:11.942 7fa7d3394700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7cc0f0b10 con 0x7fa7cc0d1870 2021-02-10T11:32:11.959 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:11.942 7fa7d3394700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7cc0e5ed0 con 0x7fa7cc0d1c70 2021-02-10T11:32:11.959 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:11.942 7fa7d3394700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7cc1b0be0 con 0x7fa7cc0d0340 2021-02-10T11:32:11.959 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:11.942 7fa7d11b0700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fa7cc0d1870 0x7fa7cc1b03f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:54742/0 (socket says 172.21.2.102:54742) 2021-02-10T11:32:11.960 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:11.943 7fa7d11b0700 1 -- 172.21.2.102:0/1704566785 learned_addr learned my addr 172.21.2.102:0/1704566785 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:32:11.960 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:11.943 7fa7d09af700 1 --2- 172.21.2.102:0/1704566785 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa7cc0d0340 0x7fa7cc0ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:11.960 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:11.943 7fa7d19b1700 1 --2- 172.21.2.102:0/1704566785 >> v2:172.21.2.113:3300/0 conn(0x7fa7cc0d1c70 0x7fa7cc0efc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:11.960 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:11.943 7fa7c3fff700 1 -- 172.21.2.102:0/1704566785 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3104123043 0 0) 0x7fa7c80010b0 con 0x7fa7cc0d1870 2021-02-10T11:32:11.960 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:11.943 7fa7c3fff700 1 -- 172.21.2.102:0/1704566785 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7b8002fe0 con 0x7fa7cc0d1870 2021-02-10T11:32:11.960 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:11.943 7fa7d09af700 1 -- 172.21.2.102:0/1704566785 >> v2:172.21.2.113:3300/0 conn(0x7fa7cc0d1c70 msgr2=0x7fa7cc0efc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:32:11.960 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:11.943 7fa7d09af700 1 --2- 172.21.2.102:0/1704566785 >> v2:172.21.2.113:3300/0 conn(0x7fa7cc0d1c70 0x7fa7cc0efc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:11.960 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:11.943 7fa7d09af700 1 -- 172.21.2.102:0/1704566785 >> v1:172.21.2.102:6789/0 conn(0x7fa7cc0d1870 legacy=0x7fa7cc1b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:11.960 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:11.943 7fa7d09af700 1 -- 172.21.2.102:0/1704566785 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7cc0f20a0 con 0x7fa7cc0d0340 2021-02-10T11:32:11.960 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:11.943 7fa7d09af700 1 --2- 172.21.2.102:0/1704566785 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa7cc0d0340 0x7fa7cc0ef640 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7fa7bc006650 tx=0x7fa7bc009a80).ready entity=mon.2 client_cookie=1bbb2918ca255fd server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:11.960 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:11.943 7fa7c3fff700 1 -- 172.21.2.102:0/1704566785 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa7bc00c0c0 con 0x7fa7cc0d0340 2021-02-10T11:32:11.960 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:11.943 7fa7c3fff700 1 -- 172.21.2.102:0/1704566785 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7bc00c340 con 0x7fa7cc0d0340 2021-02-10T11:32:11.961 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:11.943 7fa7c3fff700 1 -- 172.21.2.102:0/1704566785 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa7bc00c7a0 con 0x7fa7cc0d0340 2021-02-10T11:32:11.961 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:11.943 7fa7d3394700 1 -- 172.21.2.102:0/1704566785 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa7cc0d0340 msgr2=0x7fa7cc0ef640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:11.961 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:11.943 7fa7d3394700 1 --2- 172.21.2.102:0/1704566785 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa7cc0d0340 0x7fa7cc0ef640 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7fa7bc006650 tx=0x7fa7bc009a80).stop 2021-02-10T11:32:11.961 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:11.943 7fa7d3394700 1 -- 172.21.2.102:0/1704566785 shutdown_connections 2021-02-10T11:32:11.961 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:11.943 7fa7d3394700 1 --2- 172.21.2.102:0/1704566785 >> v2:172.21.2.113:3300/0 conn(0x7fa7cc0d1c70 0x7fa7cc0efc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:11.961 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:11.943 7fa7d3394700 1 --2- 172.21.2.102:0/1704566785 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa7cc0d0340 0x7fa7cc0ef640 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:11.961 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:11.943 7fa7d3394700 1 -- 172.21.2.102:0/1704566785 >> 172.21.2.102:0/1704566785 conn(0x7fa7cc0ce450 msgr2=0x7fa7cc0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:11.961 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:11.944 7fa7d3394700 1 -- 172.21.2.102:0/1704566785 shutdown_connections 2021-02-10T11:32:11.961 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:11.944 7fa7d3394700 1 -- 172.21.2.102:0/1704566785 wait complete. 2021-02-10T11:32:11.963 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:11.958 7fa7d3394700 1 Processor -- start 2021-02-10T11:32:11.963 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:11.958 7fa7d3394700 1 -- start start 2021-02-10T11:32:11.963 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:11.959 7fa7d3394700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa7cc1903f0 0x7fa7cc0a0180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:11.963 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:11.959 7fa7d3394700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa7cc0ecb60 0x7fa7cc0ecf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:11.963 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:11.959 7fa7d3394700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa7cc0ed580 0x7fa7cc0a65e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:11.963 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:11.959 7fa7d3394700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7cc0b6a50 con 0x7fa7cc0ed580 2021-02-10T11:32:11.963 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:11.959 7fa7d3394700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7cc0ed980 con 0x7fa7cc0ecb60 2021-02-10T11:32:11.964 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:11.959 7fa7d3394700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7cc0a3290 con 0x7fa7cc1903f0 2021-02-10T11:32:11.964 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:11.959 7fa7d19b1700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa7cc0ed580 0x7fa7cc0a65e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:11.964 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:11.959 7fa7d19b1700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa7cc0ed580 0x7fa7cc0a65e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:34294/0 (socket says 172.21.2.102:34294) 2021-02-10T11:32:11.964 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:11.959 7fa7d19b1700 1 -- 172.21.2.102:0/3159417434 learned_addr learned my addr 172.21.2.102:0/3159417434 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:32:11.964 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:11.959 7fa7d11b0700 1 --2- 172.21.2.102:0/3159417434 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa7cc1903f0 0x7fa7cc0a0180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:11.964 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:11.959 7fa7d19b1700 1 -- 172.21.2.102:0/3159417434 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa7cc1903f0 msgr2=0x7fa7cc0a0180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:11.964 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:11.959 7fa7d19b1700 1 --2- 172.21.2.102:0/3159417434 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa7cc1903f0 0x7fa7cc0a0180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:11.964 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:11.959 7fa7d09af700 1 --2- 172.21.2.102:0/3159417434 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa7cc0ecb60 0x7fa7cc0ecf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:11.964 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:11.959 7fa7d19b1700 1 -- 172.21.2.102:0/3159417434 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa7cc0ecb60 msgr2=0x7fa7cc0ecf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:11.964 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:11.959 7fa7d19b1700 1 --2- 172.21.2.102:0/3159417434 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa7cc0ecb60 0x7fa7cc0ecf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:11.964 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:11.959 7fa7d19b1700 1 -- 172.21.2.102:0/3159417434 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7cc190c20 con 0x7fa7cc0ed580 2021-02-10T11:32:11.964 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:11.959 7fa7d11b0700 1 --2- 172.21.2.102:0/3159417434 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa7cc1903f0 0x7fa7cc0a0180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:32:11.964 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:11.959 7fa7d09af700 1 --2- 172.21.2.102:0/3159417434 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa7cc0ecb60 0x7fa7cc0ecf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:32:11.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:11.960 7fa7d19b1700 1 --2- 172.21.2.102:0/3159417434 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa7cc0ed580 0x7fa7cc0a65e0 secure :-1 s=READY pgs=2240 cs=0 l=1 rev1=1 rx=0x7fa7c400b970 tx=0x7fa7c400bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:11.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:11.960 7fa7c1ffb700 1 -- 172.21.2.102:0/3159417434 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa7c4000f90 con 0x7fa7cc0ed580 2021-02-10T11:32:11.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:11.960 7fa7c1ffb700 1 -- 172.21.2.102:0/3159417434 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7c40066f0 con 0x7fa7cc0ed580 2021-02-10T11:32:11.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:11.960 7fa7d3394700 1 -- 172.21.2.102:0/3159417434 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7cc190c20 con 0x7fa7cc0ed580 2021-02-10T11:32:11.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:11.960 7fa7d3394700 1 -- 172.21.2.102:0/3159417434 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7cc0a6c00 con 0x7fa7cc0ed580 2021-02-10T11:32:11.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:11.960 7fa7c1ffb700 1 -- 172.21.2.102:0/3159417434 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa7c40015f0 con 0x7fa7cc0ed580 2021-02-10T11:32:11.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:11.961 7fa7c1ffb700 1 -- 172.21.2.102:0/3159417434 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa7c40011f0 con 0x7fa7cc0ed580 2021-02-10T11:32:11.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:11.961 7fa7c1ffb700 1 --2- 172.21.2.102:0/3159417434 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa7ac0186d0 0x7fa7ac01abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:11.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:11.961 7fa7c1ffb700 1 -- 172.21.2.102:0/3159417434 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fa7c4032b70 con 0x7fa7cc0ed580 2021-02-10T11:32:11.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:11.961 7fa7d11b0700 1 --2- 172.21.2.102:0/3159417434 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa7ac0186d0 0x7fa7ac01abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:11.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:11.961 7fa7d11b0700 1 --2- 172.21.2.102:0/3159417434 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa7ac0186d0 0x7fa7ac01abd0 crc :-1 s=READY pgs=987 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:11.992 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:11.991 7fa7d3394700 1 -- 172.21.2.102:0/3159417434 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7cc0ebc50 con 0x7fa7cc0ed580 2021-02-10T11:32:11.995 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:11.994 7fa7c1ffb700 1 -- 172.21.2.102:0/3159417434 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fa7c4000f40 con 0x7fa7cc0ed580 2021-02-10T11:32:12.130 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:12.130 7fa7d3394700 1 -- 172.21.2.102:0/3159417434 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs volume ls", "target": ["mgr", ""]}) v1 -- 0x7fa7cc0a6c00 con 0x7fa7ac0186d0 2021-02-10T11:32:12.130 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:12.130 7fa7c1ffb700 1 -- 172.21.2.102:0/3159417434 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+41 (crc 0 0 0) 0x7fa7c8000940 con 0x7fa7ac0186d0 2021-02-10T11:32:12.131 INFO:teuthology.orchestra.run.gibba002.stdout:[ 2021-02-10T11:32:12.131 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:32:12.131 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "cephfs" 2021-02-10T11:32:12.131 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:32:12.131 INFO:teuthology.orchestra.run.gibba002.stdout:] 2021-02-10T11:32:12.132 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:12.132 7fa7d3394700 1 -- 172.21.2.102:0/3159417434 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa7ac0186d0 msgr2=0x7fa7ac01abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:12.132 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:12.132 7fa7d3394700 1 --2- 172.21.2.102:0/3159417434 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa7ac0186d0 0x7fa7ac01abd0 crc :-1 s=READY pgs=987 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:12.132 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:12.132 7fa7d3394700 1 -- 172.21.2.102:0/3159417434 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa7cc0ed580 msgr2=0x7fa7cc0a65e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:12.132 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:12.132 7fa7d3394700 1 --2- 172.21.2.102:0/3159417434 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa7cc0ed580 0x7fa7cc0a65e0 secure :-1 s=READY pgs=2240 cs=0 l=1 rev1=1 rx=0x7fa7c400b970 tx=0x7fa7c400bd10).stop 2021-02-10T11:32:12.133 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:12.132 7fa7d3394700 1 -- 172.21.2.102:0/3159417434 shutdown_connections 2021-02-10T11:32:12.133 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:12.132 7fa7d3394700 1 --2- 172.21.2.102:0/3159417434 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa7cc0ed580 0x7fa7cc0a65e0 unknown :-1 s=CLOSED pgs=2240 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:12.133 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:12.132 7fa7d3394700 1 --2- 172.21.2.102:0/3159417434 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa7ac0186d0 0x7fa7ac01abd0 unknown :-1 s=CLOSED pgs=987 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:12.133 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:12.132 7fa7d3394700 1 --2- 172.21.2.102:0/3159417434 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa7cc0ecb60 0x7fa7cc0ecf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:12.133 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:12.132 7fa7d3394700 1 --2- 172.21.2.102:0/3159417434 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa7cc1903f0 0x7fa7cc0a0180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:12.133 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:12.132 7fa7d3394700 1 -- 172.21.2.102:0/3159417434 >> 172.21.2.102:0/3159417434 conn(0x7fa7cc0bc090 msgr2=0x7fa7cc0ebe80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:12.133 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:12.132 7fa7d3394700 1 -- 172.21.2.102:0/3159417434 shutdown_connections 2021-02-10T11:32:12.134 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:12.132 7fa7d3394700 1 -- 172.21.2.102:0/3159417434 wait complete. 2021-02-10T11:32:12.141 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config set mon mon_allow_pool_delete True 2021-02-10T11:32:12.573 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:12.556 7f48c6460700 1 Processor -- start 2021-02-10T11:32:12.573 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:12.556 7f48c6460700 1 -- start start 2021-02-10T11:32:12.573 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:12.556 7f48c6460700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f48c00d1c70 0x7f48c00cef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:12.573 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:12.556 7f48c6460700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f48c00cf580 0x7f48c00cf980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:12.573 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:12.556 7f48c6460700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48c00fe1c0 con 0x7f48c00d1870 2021-02-10T11:32:12.573 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:12.556 7f48c6460700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48c01b0cb0 con 0x7f48c00d1c70 2021-02-10T11:32:12.573 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:12.556 7f48c6460700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48c0109a60 con 0x7f48c00cf580 2021-02-10T11:32:12.573 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:12.556 7f48b77fe700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f48c00d1c70 0x7f48c00cef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:12.573 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:12.556 7f48bffff700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f48c00d1870 0x7f48c01b04c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:54756/0 (socket says 172.21.2.102:54756) 2021-02-10T11:32:12.573 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:12.556 7f48bffff700 1 -- 172.21.2.102:0/2727622690 learned_addr learned my addr 172.21.2.102:0/2727622690 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:32:12.574 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:12.557 7f48c4a7d700 1 --2- 172.21.2.102:0/2727622690 >> v2:172.21.2.113:3300/0 conn(0x7f48c00cf580 0x7f48c00cf980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:12.574 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:12.557 7f48bf7fe700 1 -- 172.21.2.102:0/2727622690 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3986949457 0 0) 0x7f48b0001180 con 0x7f48c00d1870 2021-02-10T11:32:12.574 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:12.557 7f48bf7fe700 1 -- 172.21.2.102:0/2727622690 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48ac002fe0 con 0x7f48c00d1870 2021-02-10T11:32:12.574 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:12.557 7f48b77fe700 1 -- 172.21.2.102:0/2727622690 >> v2:172.21.2.113:3300/0 conn(0x7f48c00cf580 msgr2=0x7f48c00cf980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:32:12.574 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:12.557 7f48b77fe700 1 --2- 172.21.2.102:0/2727622690 >> v2:172.21.2.113:3300/0 conn(0x7f48c00cf580 0x7f48c00cf980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:12.574 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:12.557 7f48b77fe700 1 -- 172.21.2.102:0/2727622690 >> v1:172.21.2.102:6789/0 conn(0x7f48c00d1870 legacy=0x7f48c01b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:12.574 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:12.557 7f48b77fe700 1 -- 172.21.2.102:0/2727622690 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48c0088070 con 0x7f48c00d1c70 2021-02-10T11:32:12.574 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:12.557 7f48b77fe700 1 --2- 172.21.2.102:0/2727622690 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f48c00d1c70 0x7f48c00cef60 secure :-1 s=READY pgs=2242 cs=0 l=1 rev1=1 rx=0x7f48a8006650 tx=0x7f48a8009a80).ready entity=mon.0 client_cookie=5324d2d3e089cfd server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:12.574 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:12.557 7f48bf7fe700 1 -- 172.21.2.102:0/2727622690 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f48a80009b0 con 0x7f48c00d1c70 2021-02-10T11:32:12.574 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:12.557 7f48bf7fe700 1 -- 172.21.2.102:0/2727622690 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48a800c0c0 con 0x7f48c00d1c70 2021-02-10T11:32:12.574 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:12.557 7f48bf7fe700 1 -- 172.21.2.102:0/2727622690 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f48a800c520 con 0x7f48c00d1c70 2021-02-10T11:32:12.574 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:12.557 7f48c6460700 1 -- 172.21.2.102:0/2727622690 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f48c00d1c70 msgr2=0x7f48c00cef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:12.574 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:12.557 7f48c6460700 1 --2- 172.21.2.102:0/2727622690 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f48c00d1c70 0x7f48c00cef60 secure :-1 s=READY pgs=2242 cs=0 l=1 rev1=1 rx=0x7f48a8006650 tx=0x7f48a8009a80).stop 2021-02-10T11:32:12.575 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:12.557 7f48c6460700 1 -- 172.21.2.102:0/2727622690 shutdown_connections 2021-02-10T11:32:12.575 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:12.557 7f48c6460700 1 --2- 172.21.2.102:0/2727622690 >> v2:172.21.2.113:3300/0 conn(0x7f48c00cf580 0x7f48c00cf980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:12.575 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:12.557 7f48c6460700 1 --2- 172.21.2.102:0/2727622690 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f48c00d1c70 0x7f48c00cef60 unknown :-1 s=CLOSED pgs=2242 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:12.575 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:12.557 7f48c6460700 1 -- 172.21.2.102:0/2727622690 >> 172.21.2.102:0/2727622690 conn(0x7f48c00ce420 msgr2=0x7f48c00d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:12.575 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:12.557 7f48c6460700 1 -- 172.21.2.102:0/2727622690 shutdown_connections 2021-02-10T11:32:12.575 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:12.557 7f48c6460700 1 -- 172.21.2.102:0/2727622690 wait complete. 2021-02-10T11:32:12.576 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:12.573 7f48c6460700 1 Processor -- start 2021-02-10T11:32:12.576 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:12.573 7f48c6460700 1 -- start start 2021-02-10T11:32:12.576 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:12.573 7f48c6460700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f48c00b22c0 0x7f48c00b26c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:12.576 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:12.573 7f48c6460700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f48c00a6df0 0x7f48c00a71f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:12.577 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:12.573 7f48c6460700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f48c00a5cd0 0x7f48c00a60d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:12.577 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:12.573 7f48c6460700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48c00c23f0 con 0x7f48c00b22c0 2021-02-10T11:32:12.577 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:12.573 7f48c6460700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48c00a66f0 con 0x7f48c00a5cd0 2021-02-10T11:32:12.577 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:12.573 7f48c6460700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48c00a6910 con 0x7f48c00a6df0 2021-02-10T11:32:12.577 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:12.573 7f48bffff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f48c00b22c0 0x7f48c00b26c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:12.577 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:12.573 7f48bffff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f48c00b22c0 0x7f48c00b26c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:34304/0 (socket says 172.21.2.102:34304) 2021-02-10T11:32:12.577 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:12.573 7f48bffff700 1 -- 172.21.2.102:0/3555691619 learned_addr learned my addr 172.21.2.102:0/3555691619 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:32:12.577 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:12.574 7f48bffff700 1 -- 172.21.2.102:0/3555691619 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f48c00a6df0 msgr2=0x7f48c00a71f0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:32:12.577 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:12.574 7f48bffff700 1 --2- 172.21.2.102:0/3555691619 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f48c00a6df0 0x7f48c00a71f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:12.577 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:12.574 7f48bffff700 1 -- 172.21.2.102:0/3555691619 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f48c00a5cd0 msgr2=0x7f48c00a60d0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:32:12.577 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:12.574 7f48bffff700 1 --2- 172.21.2.102:0/3555691619 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f48c00a5cd0 0x7f48c00a60d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:12.577 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:12.574 7f48bffff700 1 -- 172.21.2.102:0/3555691619 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48c0083980 con 0x7f48c00b22c0 2021-02-10T11:32:12.578 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:12.574 7f48bffff700 1 --2- 172.21.2.102:0/3555691619 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f48c00b22c0 0x7f48c00b26c0 secure :-1 s=READY pgs=2243 cs=0 l=1 rev1=1 rx=0x7f48b000a5d0 tx=0x7f48b000a970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:12.578 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:12.574 7f48bd7fa700 1 -- 172.21.2.102:0/3555691619 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f48b0000ce0 con 0x7f48c00b22c0 2021-02-10T11:32:12.578 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:12.574 7f48bd7fa700 1 -- 172.21.2.102:0/3555691619 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48b00043a0 con 0x7f48c00b22c0 2021-02-10T11:32:12.578 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:12.574 7f48bd7fa700 1 -- 172.21.2.102:0/3555691619 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f48b00047f0 con 0x7f48c00b22c0 2021-02-10T11:32:12.578 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:12.574 7f48c6460700 1 -- 172.21.2.102:0/3555691619 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48c0083980 con 0x7f48c00b22c0 2021-02-10T11:32:12.578 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:12.574 7f48c6460700 1 -- 172.21.2.102:0/3555691619 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48c0083f10 con 0x7f48c00b22c0 2021-02-10T11:32:12.578 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:12.575 7f48bd7fa700 1 -- 172.21.2.102:0/3555691619 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f48b0007070 con 0x7f48c00b22c0 2021-02-10T11:32:12.578 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:12.575 7f48bd7fa700 1 --2- 172.21.2.102:0/3555691619 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f48a40186c0 0x7f48a401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:12.578 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:12.575 7f48bd7fa700 1 -- 172.21.2.102:0/3555691619 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f48b0032580 con 0x7f48c00b22c0 2021-02-10T11:32:12.578 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:12.575 7f48b77fe700 1 --2- 172.21.2.102:0/3555691619 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f48a40186c0 0x7f48a401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:12.578 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:12.575 7f48b77fe700 1 --2- 172.21.2.102:0/3555691619 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f48a40186c0 0x7f48a401abc0 crc :-1 s=READY pgs=988 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:12.606 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:12.606 7f48c6460700 1 -- 172.21.2.102:0/3555691619 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48c0082a70 con 0x7f48c00b22c0 2021-02-10T11:32:12.609 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:12.609 7f48bd7fa700 1 -- 172.21.2.102:0/3555691619 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f48b0005540 con 0x7f48c00b22c0 2021-02-10T11:32:12.759 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:12.759 7f48c6460700 1 -- 172.21.2.102:0/3555691619 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f48c0082a70 con 0x7f48c00b22c0 2021-02-10T11:32:12.760 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:12.759 7f48bd7fa700 1 -- 172.21.2.102:0/3555691619 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v28)=0 v28) v1 ==== 122+0+0 (secure 0 0 0) 0x7f48b0005540 con 0x7f48c00b22c0 2021-02-10T11:32:12.761 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:12.761 7f48c6460700 1 -- 172.21.2.102:0/3555691619 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f48a40186c0 msgr2=0x7f48a401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:12.761 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:12.761 7f48c6460700 1 --2- 172.21.2.102:0/3555691619 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f48a40186c0 0x7f48a401abc0 crc :-1 s=READY pgs=988 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:12.762 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:12.761 7f48c6460700 1 -- 172.21.2.102:0/3555691619 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f48c00b22c0 msgr2=0x7f48c00b26c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:12.762 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:12.761 7f48c6460700 1 --2- 172.21.2.102:0/3555691619 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f48c00b22c0 0x7f48c00b26c0 secure :-1 s=READY pgs=2243 cs=0 l=1 rev1=1 rx=0x7f48b000a5d0 tx=0x7f48b000a970).stop 2021-02-10T11:32:12.762 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:12.762 7f48c6460700 1 -- 172.21.2.102:0/3555691619 shutdown_connections 2021-02-10T11:32:12.762 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:12.762 7f48c6460700 1 --2- 172.21.2.102:0/3555691619 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f48a40186c0 0x7f48a401abc0 unknown :-1 s=CLOSED pgs=988 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:12.762 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:12.762 7f48c6460700 1 --2- 172.21.2.102:0/3555691619 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f48c00a5cd0 0x7f48c00a60d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:12.762 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:12.762 7f48c6460700 1 --2- 172.21.2.102:0/3555691619 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f48c00a6df0 0x7f48c00a71f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:12.763 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:12.762 7f48c6460700 1 --2- 172.21.2.102:0/3555691619 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f48c00b22c0 0x7f48c00b26c0 unknown :-1 s=CLOSED pgs=2243 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:12.763 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:12.762 7f48c6460700 1 -- 172.21.2.102:0/3555691619 >> 172.21.2.102:0/3555691619 conn(0x7f48c009ffd0 msgr2=0x7f48c0082ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:12.763 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:12.762 7f48c6460700 1 -- 172.21.2.102:0/3555691619 shutdown_connections 2021-02-10T11:32:12.763 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:12.762 7f48c6460700 1 -- 172.21.2.102:0/3555691619 wait complete. 2021-02-10T11:32:12.770 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolumegroup create cephfs group_0000000000403389 2021-02-10T11:32:13.202 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:13.185 7f89081d2700 1 Processor -- start 2021-02-10T11:32:13.202 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:13.185 7f89081d2700 1 -- start start 2021-02-10T11:32:13.202 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:13.186 7f89081d2700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f89000d1870 0x7f89001b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:13.202 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:13.186 7f89081d2700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f89000cff70 0x7f8900109130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:13.202 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:13.186 7f89081d2700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8900109a60 con 0x7f89001b0ae0 2021-02-10T11:32:13.202 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:13.186 7f89081d2700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8900109750 con 0x7f89000cff70 2021-02-10T11:32:13.202 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:13.186 7f89081d2700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8900088070 con 0x7f89000d1870 2021-02-10T11:32:13.202 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:13.186 7f89067ef700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f89000cff70 0x7f8900109130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:13.202 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:13.186 7f89067ef700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f89000cff70 0x7f8900109130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:34312/0 (socket says 172.21.2.102:34312) 2021-02-10T11:32:13.202 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:13.186 7f89067ef700 1 -- 172.21.2.102:0/633447424 learned_addr learned my addr 172.21.2.102:0/633447424 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:32:13.203 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:13.186 7f89067ef700 1 -- 172.21.2.102:0/633447424 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f89000d1870 msgr2=0x7f89001b04c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:32:13.203 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:13.186 7f89067ef700 1 --2- 172.21.2.102:0/633447424 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f89000d1870 0x7f89001b04c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:13.203 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:13.186 7f89067ef700 1 -- 172.21.2.102:0/633447424 >> v1:172.21.2.113:6789/0 conn(0x7f89001b0ae0 legacy=0x7f8900108940 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:32:13.203 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:13.186 7f89067ef700 1 -- 172.21.2.102:0/633447424 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8900088260 con 0x7f89000cff70 2021-02-10T11:32:13.203 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:13.186 7f89067ef700 1 --2- 172.21.2.102:0/633447424 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f89000cff70 0x7f8900109130 secure :-1 s=READY pgs=2244 cs=0 l=1 rev1=1 rx=0x7f88f8006f00 tx=0x7f88f8009f10).ready entity=mon.0 client_cookie=f67fe57ab36a158a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:13.203 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:13.187 7f8904fec700 1 -- 172.21.2.102:0/633447424 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f88f800b3b0 con 0x7f89000cff70 2021-02-10T11:32:13.203 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:13.187 7f8904fec700 1 -- 172.21.2.102:0/633447424 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88f800b630 con 0x7f89000cff70 2021-02-10T11:32:13.203 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:13.187 7f8904fec700 1 -- 172.21.2.102:0/633447424 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f88f800bb50 con 0x7f89000cff70 2021-02-10T11:32:13.203 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:13.187 7f89081d2700 1 -- 172.21.2.102:0/633447424 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f89000cff70 msgr2=0x7f8900109130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:13.203 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:13.187 7f89081d2700 1 --2- 172.21.2.102:0/633447424 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f89000cff70 0x7f8900109130 secure :-1 s=READY pgs=2244 cs=0 l=1 rev1=1 rx=0x7f88f8006f00 tx=0x7f88f8009f10).stop 2021-02-10T11:32:13.203 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:13.187 7f89081d2700 1 -- 172.21.2.102:0/633447424 shutdown_connections 2021-02-10T11:32:13.203 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:13.187 7f89081d2700 1 --2- 172.21.2.102:0/633447424 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f89000cff70 0x7f8900109130 unknown :-1 s=CLOSED pgs=2244 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:13.204 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:13.187 7f89081d2700 1 --2- 172.21.2.102:0/633447424 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f89000d1870 0x7f89001b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:13.204 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:13.187 7f89081d2700 1 -- 172.21.2.102:0/633447424 >> 172.21.2.102:0/633447424 conn(0x7f89000ce420 msgr2=0x7f89000d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:13.204 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:13.187 7f89081d2700 1 -- 172.21.2.102:0/633447424 shutdown_connections 2021-02-10T11:32:13.204 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:13.187 7f89081d2700 1 -- 172.21.2.102:0/633447424 wait complete. 2021-02-10T11:32:13.205 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:13.202 7f89081d2700 1 Processor -- start 2021-02-10T11:32:13.205 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:13.202 7f89081d2700 1 -- start start 2021-02-10T11:32:13.205 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:13.202 7f89081d2700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f89000c3820 0x7f89000a7950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:13.205 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:13.202 7f89081d2700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f89001a44c0 0x7f89001a48c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:13.206 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:13.202 7f89081d2700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f89000863d0 0x7f89000867d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:13.206 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:13.202 7f89081d2700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8900085ed0 con 0x7f89000c3820 2021-02-10T11:32:13.206 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:13.202 7f89081d2700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89000c0740 con 0x7f89000863d0 2021-02-10T11:32:13.206 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:13.202 7f89081d2700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8900086df0 con 0x7f89001a44c0 2021-02-10T11:32:13.206 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:13.202 7f8905fee700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f89000c3820 0x7f89000a7950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:13.206 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:13.202 7f8905fee700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f89000c3820 0x7f89000a7950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:34316/0 (socket says 172.21.2.102:34316) 2021-02-10T11:32:13.206 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:13.202 7f8905fee700 1 -- 172.21.2.102:0/2945277896 learned_addr learned my addr 172.21.2.102:0/2945277896 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:32:13.206 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:13.203 7f8905fee700 1 -- 172.21.2.102:0/2945277896 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f89001a44c0 msgr2=0x7f89001a48c0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:32:13.206 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:13.203 7f8905fee700 1 --2- 172.21.2.102:0/2945277896 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f89001a44c0 0x7f89001a48c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:13.206 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:13.203 7f8905fee700 1 -- 172.21.2.102:0/2945277896 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f89000863d0 msgr2=0x7f89000867d0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:32:13.206 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:13.203 7f8905fee700 1 --2- 172.21.2.102:0/2945277896 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f89000863d0 0x7f89000867d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:13.206 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:13.203 7f8905fee700 1 -- 172.21.2.102:0/2945277896 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8900087010 con 0x7f89000c3820 2021-02-10T11:32:13.207 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:13.203 7f8905fee700 1 --2- 172.21.2.102:0/2945277896 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f89000c3820 0x7f89000a7950 secure :-1 s=READY pgs=2245 cs=0 l=1 rev1=1 rx=0x7f88fc00b5c0 tx=0x7f88fc00b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:13.207 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:13.203 7f88f27fc700 1 -- 172.21.2.102:0/2945277896 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f88fc00bd50 con 0x7f89000c3820 2021-02-10T11:32:13.207 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:13.203 7f88f27fc700 1 -- 172.21.2.102:0/2945277896 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88fc006030 con 0x7f89000c3820 2021-02-10T11:32:13.207 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:13.203 7f88f27fc700 1 -- 172.21.2.102:0/2945277896 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f88fc0054d0 con 0x7f89000c3820 2021-02-10T11:32:13.207 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:13.203 7f89081d2700 1 -- 172.21.2.102:0/2945277896 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8900087010 con 0x7f89000c3820 2021-02-10T11:32:13.207 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:13.203 7f89081d2700 1 -- 172.21.2.102:0/2945277896 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8900087010 con 0x7f89000c3820 2021-02-10T11:32:13.207 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:13.204 7f88f27fc700 1 -- 172.21.2.102:0/2945277896 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f88fc006210 con 0x7f89000c3820 2021-02-10T11:32:13.207 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:13.204 7f88f27fc700 1 --2- 172.21.2.102:0/2945277896 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f88e80186c0 0x7f88e801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:13.207 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:13.204 7f88f27fc700 1 -- 172.21.2.102:0/2945277896 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f88fc0323f0 con 0x7f89000c3820 2021-02-10T11:32:13.207 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:13.204 7f89057ed700 1 --2- 172.21.2.102:0/2945277896 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f88e80186c0 0x7f88e801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:13.207 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:13.205 7f89057ed700 1 --2- 172.21.2.102:0/2945277896 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f88e80186c0 0x7f88e801abc0 crc :-1 s=READY pgs=989 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:13.236 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:13.236 7f89081d2700 1 -- 172.21.2.102:0/2945277896 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89000fe1c0 con 0x7f89000c3820 2021-02-10T11:32:13.238 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:13.238 7f88f27fc700 1 -- 172.21.2.102:0/2945277896 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f88fc006b80 con 0x7f89000c3820 2021-02-10T11:32:13.369 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:13.369 7f89081d2700 1 -- 172.21.2.102:0/2945277896 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "target": ["mgr", ""], "group_name": "group_0000000000403389"}) v1 -- 0x7f89000fe1c0 con 0x7f88e80186c0 2021-02-10T11:32:13.373 INFO:tasks.ceph.mgr.y.gibba002.stderr:2021-02-10 11:32:13.373 7fd94bc53700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:32:13.373 INFO:tasks.ceph.mgr.y.gibba002.stderr:2021-02-10 11:32:13.373 7fd94bc53700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:32:13.373 INFO:tasks.ceph.mgr.y.gibba002.stderr:2021-02-10 11:32:13.373 7fd94bc53700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:32:13.373 INFO:tasks.ceph.mgr.y.gibba002.stderr:2021-02-10 11:32:13.373 7fd94bc53700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:32:13.373 INFO:tasks.ceph.mgr.y.gibba002.stderr:2021-02-10 11:32:13.373 7fd94bc53700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:32:13.387 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:13.387 7f88f27fc700 1 -- 172.21.2.102:0/2945277896 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f88f40009b0 con 0x7f88e80186c0 2021-02-10T11:32:13.403 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:13.402 7f89081d2700 1 -- 172.21.2.102:0/2945277896 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f88e80186c0 msgr2=0x7f88e801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:13.403 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:13.402 7f89081d2700 1 --2- 172.21.2.102:0/2945277896 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f88e80186c0 0x7f88e801abc0 crc :-1 s=READY pgs=989 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:13.403 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:13.402 7f89081d2700 1 -- 172.21.2.102:0/2945277896 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f89000c3820 msgr2=0x7f89000a7950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:13.403 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:13.402 7f89081d2700 1 --2- 172.21.2.102:0/2945277896 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f89000c3820 0x7f89000a7950 secure :-1 s=READY pgs=2245 cs=0 l=1 rev1=1 rx=0x7f88fc00b5c0 tx=0x7f88fc00b960).stop 2021-02-10T11:32:13.403 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:13.402 7f89081d2700 1 -- 172.21.2.102:0/2945277896 shutdown_connections 2021-02-10T11:32:13.403 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:13.402 7f89081d2700 1 --2- 172.21.2.102:0/2945277896 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f88e80186c0 0x7f88e801abc0 unknown :-1 s=CLOSED pgs=989 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:13.403 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:13.402 7f89081d2700 1 --2- 172.21.2.102:0/2945277896 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f89000863d0 0x7f89000867d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:13.403 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:13.402 7f89081d2700 1 --2- 172.21.2.102:0/2945277896 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f89001a44c0 0x7f89001a48c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:13.404 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:13.402 7f89081d2700 1 --2- 172.21.2.102:0/2945277896 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f89000c3820 0x7f89000a7950 unknown :-1 s=CLOSED pgs=2245 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:13.404 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:13.402 7f89081d2700 1 -- 172.21.2.102:0/2945277896 >> 172.21.2.102:0/2945277896 conn(0x7f89000872c0 msgr2=0x7f8900095700 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:13.404 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:13.402 7f89081d2700 1 -- 172.21.2.102:0/2945277896 shutdown_connections 2021-02-10T11:32:13.404 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:13.402 7f89081d2700 1 -- 172.21.2.102:0/2945277896 wait complete. 2021-02-10T11:32:13.410 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolumegroup getpath cephfs group_0000000000403389 2021-02-10T11:32:13.841 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:13.824 7fc2278d7700 1 Processor -- start 2021-02-10T11:32:13.842 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:13.825 7fc2278d7700 1 -- start start 2021-02-10T11:32:13.842 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:13.825 7fc2278d7700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fc2200d1c60 0x7fc2200cef50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:13.842 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:13.825 7fc2278d7700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc2200cf570 0x7fc2200cf970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:13.842 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:13.825 7fc2278d7700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2200fe1b0 con 0x7fc2200d1860 2021-02-10T11:32:13.842 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:13.825 7fc2278d7700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2201b0ca0 con 0x7fc2200d1c60 2021-02-10T11:32:13.842 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:13.825 7fc2278d7700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc220109a50 con 0x7fc2200cf570 2021-02-10T11:32:13.842 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:13.825 7fc225ef4700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc2200cf570 0x7fc2200cf970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:13.842 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:13.825 7fc225ef4700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc2200cf570 0x7fc2200cf970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.113:3301/0 says I am v2:172.21.2.102:39878/0 (socket says 172.21.2.102:39878) 2021-02-10T11:32:13.842 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:13.826 7fc225ef4700 1 -- 172.21.2.102:0/353607893 learned_addr learned my addr 172.21.2.102:0/353607893 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:32:13.842 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:13.826 7fc224ef2700 1 --2- 172.21.2.102:0/353607893 >> v2:172.21.2.113:3300/0 conn(0x7fc2200d1c60 0x7fc2200cef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:13.843 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:13.826 7fc225ef4700 1 -- 172.21.2.102:0/353607893 >> v2:172.21.2.113:3300/0 conn(0x7fc2200d1c60 msgr2=0x7fc2200cef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:32:13.843 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:13.826 7fc225ef4700 1 --2- 172.21.2.102:0/353607893 >> v2:172.21.2.113:3300/0 conn(0x7fc2200d1c60 0x7fc2200cef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:13.843 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:13.826 7fc225ef4700 1 -- 172.21.2.102:0/353607893 >> v1:172.21.2.113:6789/0 conn(0x7fc2200d1860 legacy=0x7fc2201b04b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:13.843 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:13.826 7fc225ef4700 1 -- 172.21.2.102:0/353607893 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc220088070 con 0x7fc2200cf570 2021-02-10T11:32:13.843 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:13.826 7fc225ef4700 1 --2- 172.21.2.102:0/353607893 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc2200cf570 0x7fc2200cf970 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7fc218006aa0 tx=0x7fc218009f20).ready entity=mon.2 client_cookie=84a564917a47da2a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:13.843 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:13.826 7fc213fff700 1 -- 172.21.2.102:0/353607893 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc21800b5c0 con 0x7fc2200cf570 2021-02-10T11:32:13.843 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:13.826 7fc213fff700 1 -- 172.21.2.102:0/353607893 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc218003030 con 0x7fc2200cf570 2021-02-10T11:32:13.843 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:13.826 7fc213fff700 1 -- 172.21.2.102:0/353607893 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc21800bac0 con 0x7fc2200cf570 2021-02-10T11:32:13.843 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:13.826 7fc2278d7700 1 -- 172.21.2.102:0/353607893 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc2200cf570 msgr2=0x7fc2200cf970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:13.843 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:13.826 7fc2278d7700 1 --2- 172.21.2.102:0/353607893 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc2200cf570 0x7fc2200cf970 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7fc218006aa0 tx=0x7fc218009f20).stop 2021-02-10T11:32:13.843 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:13.827 7fc2278d7700 1 -- 172.21.2.102:0/353607893 shutdown_connections 2021-02-10T11:32:13.843 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:13.827 7fc2278d7700 1 --2- 172.21.2.102:0/353607893 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc2200cf570 0x7fc2200cf970 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:13.843 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:13.827 7fc2278d7700 1 --2- 172.21.2.102:0/353607893 >> v2:172.21.2.113:3300/0 conn(0x7fc2200d1c60 0x7fc2200cef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:13.844 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:13.827 7fc2278d7700 1 -- 172.21.2.102:0/353607893 >> 172.21.2.102:0/353607893 conn(0x7fc2200ce410 msgr2=0x7fc2200d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:13.844 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:13.827 7fc2278d7700 1 -- 172.21.2.102:0/353607893 shutdown_connections 2021-02-10T11:32:13.844 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:13.827 7fc2278d7700 1 -- 172.21.2.102:0/353607893 wait complete. 2021-02-10T11:32:13.846 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:13.842 7fc2278d7700 1 Processor -- start 2021-02-10T11:32:13.846 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:13.842 7fc2278d7700 1 -- start start 2021-02-10T11:32:13.846 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:13.842 7fc2278d7700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc22019e3c0 0x7fc2200926d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:13.846 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:13.842 7fc2278d7700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc220193690 0x7fc220193a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:13.846 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:13.842 7fc2278d7700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc2200c3410 0x7fc2200c3810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:13.846 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:13.842 7fc2278d7700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2201940b0 con 0x7fc2200c3410 2021-02-10T11:32:13.846 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:13.842 7fc2278d7700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2200c3e30 con 0x7fc220193690 2021-02-10T11:32:13.846 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:13.842 7fc2278d7700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2200aab40 con 0x7fc22019e3c0 2021-02-10T11:32:13.846 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:13.842 7fc2256f3700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc22019e3c0 0x7fc2200926d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:13.846 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:13.842 7fc225ef4700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc2200c3410 0x7fc2200c3810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:13.846 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:13.842 7fc225ef4700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc2200c3410 0x7fc2200c3810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:34348/0 (socket says 172.21.2.102:34348) 2021-02-10T11:32:13.847 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:13.842 7fc225ef4700 1 -- 172.21.2.102:0/3552272691 learned_addr learned my addr 172.21.2.102:0/3552272691 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:32:13.847 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:13.843 7fc224ef2700 1 --2- 172.21.2.102:0/3552272691 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc220193690 0x7fc220193a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:13.847 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:13.843 7fc225ef4700 1 -- 172.21.2.102:0/3552272691 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc22019e3c0 msgr2=0x7fc2200926d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:13.847 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:13.843 7fc225ef4700 1 --2- 172.21.2.102:0/3552272691 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc22019e3c0 0x7fc2200926d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:13.847 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:13.843 7fc225ef4700 1 -- 172.21.2.102:0/3552272691 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc220193690 msgr2=0x7fc220193a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:13.847 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:13.843 7fc225ef4700 1 --2- 172.21.2.102:0/3552272691 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc220193690 0x7fc220193a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:13.847 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:13.843 7fc225ef4700 1 -- 172.21.2.102:0/3552272691 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2200aad60 con 0x7fc2200c3410 2021-02-10T11:32:13.847 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:13.843 7fc2256f3700 1 --2- 172.21.2.102:0/3552272691 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc22019e3c0 0x7fc2200926d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:32:13.847 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:13.843 7fc225ef4700 1 --2- 172.21.2.102:0/3552272691 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc2200c3410 0x7fc2200c3810 secure :-1 s=READY pgs=2248 cs=0 l=1 rev1=1 rx=0x7fc2180099b0 tx=0x7fc21800bce0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:13.847 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:13.843 7fc211ffb700 1 -- 172.21.2.102:0/3552272691 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc218009140 con 0x7fc2200c3410 2021-02-10T11:32:13.847 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:13.843 7fc211ffb700 1 -- 172.21.2.102:0/3552272691 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc218006e00 con 0x7fc2200c3410 2021-02-10T11:32:13.847 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:13.843 7fc211ffb700 1 -- 172.21.2.102:0/3552272691 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc21800c4e0 con 0x7fc2200c3410 2021-02-10T11:32:13.848 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:13.843 7fc2278d7700 1 -- 172.21.2.102:0/3552272691 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2200aad60 con 0x7fc2200c3410 2021-02-10T11:32:13.848 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:13.843 7fc2278d7700 1 -- 172.21.2.102:0/3552272691 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2200aad60 con 0x7fc2200c3410 2021-02-10T11:32:13.848 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:13.844 7fc211ffb700 1 -- 172.21.2.102:0/3552272691 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fc218009450 con 0x7fc2200c3410 2021-02-10T11:32:13.848 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:13.844 7fc211ffb700 1 --2- 172.21.2.102:0/3552272691 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc2000186c0 0x7fc20001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:13.848 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:13.844 7fc211ffb700 1 -- 172.21.2.102:0/3552272691 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fc218032b40 con 0x7fc2200c3410 2021-02-10T11:32:13.848 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:13.844 7fc2256f3700 1 --2- 172.21.2.102:0/3552272691 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc2000186c0 0x7fc20001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:13.848 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:13.844 7fc2256f3700 1 --2- 172.21.2.102:0/3552272691 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc2000186c0 0x7fc20001abc0 crc :-1 s=READY pgs=990 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:13.875 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:13.875 7fc2278d7700 1 -- 172.21.2.102:0/3552272691 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2200fe1b0 con 0x7fc2200c3410 2021-02-10T11:32:13.877 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:13.877 7fc211ffb700 1 -- 172.21.2.102:0/3552272691 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fc21800be20 con 0x7fc2200c3410 2021-02-10T11:32:14.004 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:14.004 7fc2278d7700 1 -- 172.21.2.102:0/3552272691 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolumegroup getpath", "vol_name": "cephfs", "target": ["mgr", ""], "group_name": "group_0000000000403389"}) v1 -- 0x7fc2200fe1b0 con 0x7fc2000186c0 2021-02-10T11:32:14.005 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:14.005 7fc211ffb700 1 -- 172.21.2.102:0/3552272691 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+32 (crc 0 0 0) 0x7fc21c000960 con 0x7fc2000186c0 2021-02-10T11:32:14.005 INFO:teuthology.orchestra.run.gibba002.stdout:/volumes/group_0000000000403389 2021-02-10T11:32:14.006 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:14.006 7fc2278d7700 1 -- 172.21.2.102:0/3552272691 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc2000186c0 msgr2=0x7fc20001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:14.006 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:14.006 7fc2278d7700 1 --2- 172.21.2.102:0/3552272691 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc2000186c0 0x7fc20001abc0 crc :-1 s=READY pgs=990 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:14.006 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:14.006 7fc2278d7700 1 -- 172.21.2.102:0/3552272691 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc2200c3410 msgr2=0x7fc2200c3810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:14.006 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:14.006 7fc2278d7700 1 --2- 172.21.2.102:0/3552272691 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc2200c3410 0x7fc2200c3810 secure :-1 s=READY pgs=2248 cs=0 l=1 rev1=1 rx=0x7fc2180099b0 tx=0x7fc21800bce0).stop 2021-02-10T11:32:14.007 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:14.006 7fc2278d7700 1 -- 172.21.2.102:0/3552272691 shutdown_connections 2021-02-10T11:32:14.007 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:14.006 7fc2278d7700 1 --2- 172.21.2.102:0/3552272691 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc2200c3410 0x7fc2200c3810 unknown :-1 s=CLOSED pgs=2248 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:14.007 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:14.006 7fc2278d7700 1 --2- 172.21.2.102:0/3552272691 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc2000186c0 0x7fc20001abc0 unknown :-1 s=CLOSED pgs=990 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:14.007 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:14.006 7fc2278d7700 1 --2- 172.21.2.102:0/3552272691 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc220193690 0x7fc220193a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:14.007 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:14.006 7fc2278d7700 1 --2- 172.21.2.102:0/3552272691 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc22019e3c0 0x7fc2200926d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:14.007 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:14.006 7fc2278d7700 1 -- 172.21.2.102:0/3552272691 >> 172.21.2.102:0/3552272691 conn(0x7fc2200b79f0 msgr2=0x7fc220087070 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:14.007 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:14.006 7fc2278d7700 1 -- 172.21.2.102:0/3552272691 shutdown_connections 2021-02-10T11:32:14.007 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:14.006 7fc2278d7700 1 -- 172.21.2.102:0/3552272691 wait complete. 2021-02-10T11:32:14.015 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2021-02-10T11:32:14.015 DEBUG:teuthology.orchestra.run.gibba002:> import os 2021-02-10T11:32:14.015 DEBUG:teuthology.orchestra.run.gibba002:> import stat 2021-02-10T11:32:14.015 DEBUG:teuthology.orchestra.run.gibba002:> import json 2021-02-10T11:32:14.015 DEBUG:teuthology.orchestra.run.gibba002:> import sys 2021-02-10T11:32:14.015 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:32:14.015 DEBUG:teuthology.orchestra.run.gibba002:> try: 2021-02-10T11:32:14.015 DEBUG:teuthology.orchestra.run.gibba002:> s = os.stat("/home/ubuntu/cephtest/mnt.0/volumes/group_0000000000403389") 2021-02-10T11:32:14.015 DEBUG:teuthology.orchestra.run.gibba002:> except OSError as e: 2021-02-10T11:32:14.015 DEBUG:teuthology.orchestra.run.gibba002:> sys.exit(e.errno) 2021-02-10T11:32:14.016 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:32:14.016 DEBUG:teuthology.orchestra.run.gibba002:> attrs = ["st_mode", "st_ino", "st_dev", "st_nlink", "st_uid", "st_gid", "st_size", "st_atime", "st_mtime", "st_ctime"] 2021-02-10T11:32:14.016 DEBUG:teuthology.orchestra.run.gibba002:> print(json.dumps( 2021-02-10T11:32:14.016 DEBUG:teuthology.orchestra.run.gibba002:> dict([(a, getattr(s, a)) for a in attrs]), 2021-02-10T11:32:14.016 DEBUG:teuthology.orchestra.run.gibba002:> indent=2)) 2021-02-10T11:32:14.016 DEBUG:teuthology.orchestra.run.gibba002:> ' 2021-02-10T11:32:14.408 INFO:teuthology.orchestra.run.gibba002.stdout:{ 2021-02-10T11:32:14.408 INFO:teuthology.orchestra.run.gibba002.stdout: "st_mode": 16877, 2021-02-10T11:32:14.408 INFO:teuthology.orchestra.run.gibba002.stdout: "st_ino": 1099511627777, 2021-02-10T11:32:14.408 INFO:teuthology.orchestra.run.gibba002.stdout: "st_dev": 43, 2021-02-10T11:32:14.408 INFO:teuthology.orchestra.run.gibba002.stdout: "st_nlink": 2, 2021-02-10T11:32:14.408 INFO:teuthology.orchestra.run.gibba002.stdout: "st_uid": 0, 2021-02-10T11:32:14.408 INFO:teuthology.orchestra.run.gibba002.stdout: "st_gid": 0, 2021-02-10T11:32:14.409 INFO:teuthology.orchestra.run.gibba002.stdout: "st_size": 0, 2021-02-10T11:32:14.409 INFO:teuthology.orchestra.run.gibba002.stdout: "st_atime": 1612956733.3819354, 2021-02-10T11:32:14.409 INFO:teuthology.orchestra.run.gibba002.stdout: "st_mtime": 1612956733.3819354, 2021-02-10T11:32:14.409 INFO:teuthology.orchestra.run.gibba002.stdout: "st_ctime": 1612956733.3877473 2021-02-10T11:32:14.409 INFO:teuthology.orchestra.run.gibba002.stdout:} 2021-02-10T11:32:14.591 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolumegroup rm cephfs group_0000000000403389 2021-02-10T11:32:15.062 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:15.044 7f5457556700 1 Processor -- start 2021-02-10T11:32:15.063 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:15.044 7f5457556700 1 -- start start 2021-02-10T11:32:15.063 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:15.045 7f5457556700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f54500d1870 0x7f54501b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:15.063 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:15.045 7f5457556700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f54501b0ae0 0x7f5450108940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:15.063 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:15.045 7f5457556700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5450109a60 con 0x7f54500d1c70 2021-02-10T11:32:15.063 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:15.045 7f5457556700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54500e5ef0 con 0x7f54501b0ae0 2021-02-10T11:32:15.063 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:15.045 7f5457556700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5450109750 con 0x7f54500d1870 2021-02-10T11:32:15.063 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:15.046 7f5454b71700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f54501b0ae0 0x7f5450108940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:15.063 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:15.046 7f5455b73700 1 --1- >> v1:172.21.2.113:6789/0 conn(0x7f54500d1c70 0x7f5450108f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.113:6789/0 says I am v1:172.21.2.102:33900/0 (socket says 172.21.2.102:33900) 2021-02-10T11:32:15.063 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:15.046 7f5455b73700 1 -- 172.21.2.102:0/275530264 learned_addr learned my addr 172.21.2.102:0/275530264 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:32:15.063 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:15.046 7f5455372700 1 --2- 172.21.2.102:0/275530264 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f54500d1870 0x7f54501b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:15.063 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:15.046 7f5455372700 1 -- 172.21.2.102:0/275530264 >> v2:172.21.2.113:3300/0 conn(0x7f54501b0ae0 msgr2=0x7f5450108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:32:15.064 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:15.046 7f5455372700 1 --2- 172.21.2.102:0/275530264 >> v2:172.21.2.113:3300/0 conn(0x7f54501b0ae0 0x7f5450108940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:15.064 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:15.046 7f5455372700 1 -- 172.21.2.102:0/275530264 >> v1:172.21.2.113:6789/0 conn(0x7f54500d1c70 legacy=0x7f5450108f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:15.064 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:15.046 7f5455372700 1 -- 172.21.2.102:0/275530264 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5450088070 con 0x7f54500d1870 2021-02-10T11:32:15.064 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:15.047 7f5454b71700 1 --2- 172.21.2.102:0/275530264 >> v2:172.21.2.113:3300/0 conn(0x7f54501b0ae0 0x7f5450108940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:32:15.064 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:15.047 7f5455372700 1 --2- 172.21.2.102:0/275530264 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f54500d1870 0x7f54501b04c0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f5444006650 tx=0x7f5444009a80).ready entity=mon.2 client_cookie=4f7fbdee591f4a96 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:15.064 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:15.047 7f5443fff700 1 -- 172.21.2.102:0/275530264 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f544400c0c0 con 0x7f54500d1870 2021-02-10T11:32:15.064 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:15.047 7f5443fff700 1 -- 172.21.2.102:0/275530264 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f544400c0c0 con 0x7f54500d1870 2021-02-10T11:32:15.064 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:15.047 7f5443fff700 1 -- 172.21.2.102:0/275530264 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f544400c340 con 0x7f54500d1870 2021-02-10T11:32:15.064 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:15.047 7f5457556700 1 -- 172.21.2.102:0/275530264 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f54500d1870 msgr2=0x7f54501b04c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:15.064 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:15.047 7f5457556700 1 --2- 172.21.2.102:0/275530264 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f54500d1870 0x7f54501b04c0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f5444006650 tx=0x7f5444009a80).stop 2021-02-10T11:32:15.064 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:15.048 7f5457556700 1 -- 172.21.2.102:0/275530264 shutdown_connections 2021-02-10T11:32:15.064 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:15.048 7f5457556700 1 --2- 172.21.2.102:0/275530264 >> v2:172.21.2.113:3300/0 conn(0x7f54501b0ae0 0x7f5450108940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:15.064 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:15.048 7f5457556700 1 --2- 172.21.2.102:0/275530264 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f54500d1870 0x7f54501b04c0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:15.065 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:15.048 7f5457556700 1 -- 172.21.2.102:0/275530264 >> 172.21.2.102:0/275530264 conn(0x7f54500ce420 msgr2=0x7f54500d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:15.065 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:15.048 7f5457556700 1 -- 172.21.2.102:0/275530264 shutdown_connections 2021-02-10T11:32:15.065 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:15.048 7f5457556700 1 -- 172.21.2.102:0/275530264 wait complete. 2021-02-10T11:32:15.066 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:15.063 7f5457556700 1 Processor -- start 2021-02-10T11:32:15.066 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:15.063 7f5457556700 1 -- start start 2021-02-10T11:32:15.066 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:15.063 7f5457556700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f54500abd60 0x7f54500ac160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:15.066 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:15.063 7f5457556700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f54500ac780 0x7f54500acb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:15.066 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:15.063 7f5457556700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5450083910 0x7f5450083d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:15.066 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:15.063 7f5457556700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5450087180 con 0x7f54500abd60 2021-02-10T11:32:15.067 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:15.063 7f5457556700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f545018fcd0 con 0x7f54500ac780 2021-02-10T11:32:15.067 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:15.063 7f5457556700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5450084330 con 0x7f5450083910 2021-02-10T11:32:15.067 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:15.063 7f5455372700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f54500abd60 0x7f54500ac160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:15.067 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:15.063 7f5455372700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f54500abd60 0x7f54500ac160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:34358/0 (socket says 172.21.2.102:34358) 2021-02-10T11:32:15.067 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:15.063 7f5455372700 1 -- 172.21.2.102:0/249074480 learned_addr learned my addr 172.21.2.102:0/249074480 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:32:15.067 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:15.064 7f5455372700 1 -- 172.21.2.102:0/249074480 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5450083910 msgr2=0x7f5450083d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:15.067 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:15.064 7f5455372700 1 --2- 172.21.2.102:0/249074480 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5450083910 0x7f5450083d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:15.067 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:15.064 7f5455372700 1 -- 172.21.2.102:0/249074480 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f54500ac780 msgr2=0x7f54500acb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:15.067 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:15.064 7f5455372700 1 --2- 172.21.2.102:0/249074480 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f54500ac780 0x7f54500acb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:15.067 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:15.064 7f5455372700 1 -- 172.21.2.102:0/249074480 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5450084550 con 0x7f54500abd60 2021-02-10T11:32:15.067 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:15.064 7f5455372700 1 --2- 172.21.2.102:0/249074480 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f54500abd60 0x7f54500ac160 secure :-1 s=READY pgs=2249 cs=0 l=1 rev1=1 rx=0x7f5444009630 tx=0x7f544400b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:15.067 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:15.064 7f5441ffb700 1 -- 172.21.2.102:0/249074480 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f54440093a0 con 0x7f54500abd60 2021-02-10T11:32:15.067 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:15.064 7f5441ffb700 1 -- 172.21.2.102:0/249074480 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f544400bd70 con 0x7f54500abd60 2021-02-10T11:32:15.068 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:15.064 7f5457556700 1 -- 172.21.2.102:0/249074480 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5450084550 con 0x7f54500abd60 2021-02-10T11:32:15.068 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:15.064 7f5441ffb700 1 -- 172.21.2.102:0/249074480 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5444005560 con 0x7f54500abd60 2021-02-10T11:32:15.068 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:15.065 7f5457556700 1 -- 172.21.2.102:0/249074480 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5450084ae0 con 0x7f54500abd60 2021-02-10T11:32:15.068 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:15.065 7f5441ffb700 1 -- 172.21.2.102:0/249074480 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f544400a000 con 0x7f54500abd60 2021-02-10T11:32:15.068 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:15.065 7f5441ffb700 1 --2- 172.21.2.102:0/249074480 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f54300186c0 0x7f543001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:15.068 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:15.065 7f5441ffb700 1 -- 172.21.2.102:0/249074480 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f54440323c0 con 0x7f54500abd60 2021-02-10T11:32:15.068 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:15.065 7f5454b71700 1 --2- 172.21.2.102:0/249074480 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f54300186c0 0x7f543001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:15.068 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:15.066 7f5454b71700 1 --2- 172.21.2.102:0/249074480 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f54300186c0 0x7f543001abc0 crc :-1 s=READY pgs=991 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:15.095 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:15.095 7f5457556700 1 -- 172.21.2.102:0/249074480 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5450084d90 con 0x7f54500abd60 2021-02-10T11:32:15.097 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:15.097 7f5441ffb700 1 -- 172.21.2.102:0/249074480 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f544401ecc0 con 0x7f54500abd60 2021-02-10T11:32:15.224 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:15.224 7f5457556700 1 -- 172.21.2.102:0/249074480 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolumegroup rm", "vol_name": "cephfs", "target": ["mgr", ""], "group_name": "group_0000000000403389"}) v1 -- 0x7f5450084ae0 con 0x7f54300186c0 2021-02-10T11:32:15.228 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:15.228 7f5441ffb700 1 -- 172.21.2.102:0/249074480 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f544c0009b0 con 0x7f54300186c0 2021-02-10T11:32:15.234 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:15.233 7f5457556700 1 -- 172.21.2.102:0/249074480 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f54300186c0 msgr2=0x7f543001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:15.234 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:15.233 7f5457556700 1 --2- 172.21.2.102:0/249074480 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f54300186c0 0x7f543001abc0 crc :-1 s=READY pgs=991 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:15.234 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:15.233 7f5457556700 1 -- 172.21.2.102:0/249074480 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f54500abd60 msgr2=0x7f54500ac160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:15.234 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:15.233 7f5457556700 1 --2- 172.21.2.102:0/249074480 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f54500abd60 0x7f54500ac160 secure :-1 s=READY pgs=2249 cs=0 l=1 rev1=1 rx=0x7f5444009630 tx=0x7f544400b980).stop 2021-02-10T11:32:15.235 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:15.234 7f5457556700 1 -- 172.21.2.102:0/249074480 shutdown_connections 2021-02-10T11:32:15.235 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:15.234 7f5457556700 1 --2- 172.21.2.102:0/249074480 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5450083910 0x7f5450083d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:15.235 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:15.234 7f5457556700 1 --2- 172.21.2.102:0/249074480 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f54300186c0 0x7f543001abc0 unknown :-1 s=CLOSED pgs=991 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:15.235 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:15.234 7f5457556700 1 --2- 172.21.2.102:0/249074480 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f54500ac780 0x7f54500acb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:15.235 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:15.234 7f5457556700 1 --2- 172.21.2.102:0/249074480 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f54500abd60 0x7f54500ac160 unknown :-1 s=CLOSED pgs=2249 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:15.235 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:15.234 7f5457556700 1 -- 172.21.2.102:0/249074480 >> 172.21.2.102:0/249074480 conn(0x7f54500a2b30 msgr2=0x7f545009ce60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:15.235 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:15.234 7f5457556700 1 -- 172.21.2.102:0/249074480 shutdown_connections 2021-02-10T11:32:15.235 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:15.234 7f5457556700 1 -- 172.21.2.102:0/249074480 wait complete. 2021-02-10T11:32:15.242 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2021-02-10T11:32:15.242 DEBUG:teuthology.orchestra.run.gibba002:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:32:15.589 DEBUG:teuthology.orchestra.run.gibba013:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:32:15.935 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2021-02-10T11:32:15.936 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:32:15.936 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:32:15.936 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:32:16.270 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:32:16.271 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:32:16.271 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@gibba002.front.sepia.ceph.com... 2021-02-10T11:32:16.271 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-02-10T11:32:16.271 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:32:16.622 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba002.stderr:ceph-fuse[182073]: fuse finished with error 0 and tester_r 0 2021-02-10T11:32:16.623 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:32:16.624 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:32:16.624 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:32:16.958 INFO:teuthology.orchestra.run.gibba002.stdout:ext2/ext3 2021-02-10T11:32:16.958 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2021-02-10T11:32:16.959 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:32:16.959 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:32:17.295 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:32:17.295 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:32:17.295 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:32:17.630 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:32:17.630 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2021-02-10T11:32:17.631 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@gibba002.front.sepia.ceph.com... 2021-02-10T11:32:17.631 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-02-10T11:32:17.631 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:32:17.979 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba002.stderr:ceph-fuse[182713]: fuse finished with error 0 and tester_r 0 2021-02-10T11:32:17.981 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:32:17.981 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:32:17.981 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:32:18.315 INFO:teuthology.orchestra.run.gibba002.stdout:ext2/ext3 2021-02-10T11:32:18.315 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2021-02-10T11:32:18.316 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:32:18.316 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:32:18.651 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:32:18.651 DEBUG:teuthology.orchestra.run.gibba013:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:32:18.651 DEBUG:teuthology.orchestra.run.gibba013:> ' -- /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:32:18.986 INFO:teuthology.orchestra.run.gibba013.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.2’: No such file or directory 2021-02-10T11:32:18.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:32:18.986 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2021-02-10T11:32:18.987 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:32:18.987 DEBUG:teuthology.orchestra.run.gibba013:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:32:19.322 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:32:19.322 DEBUG:teuthology.orchestra.run.gibba013:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:32:19.322 DEBUG:teuthology.orchestra.run.gibba013:> ' -- /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:32:19.657 INFO:teuthology.orchestra.run.gibba013.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.3’: No such file or directory 2021-02-10T11:32:19.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:32:19.658 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.3 2021-02-10T11:32:19.658 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:32:19.658 DEBUG:teuthology.orchestra.run.gibba013:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:32:19.994 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config rm mon mon_allow_pool_delete 2021-02-10T11:32:20.426 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:20.409 7eff7fe32700 1 Processor -- start 2021-02-10T11:32:20.426 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:20.409 7eff7fe32700 1 -- start start 2021-02-10T11:32:20.426 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:20.409 7eff7fe32700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7eff780d0340 0x7eff780ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:20.426 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:20.409 7eff7fe32700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7eff780d1c70 0x7eff780efc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:20.426 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:20.409 7eff7fe32700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff780f0b10 con 0x7eff780d1870 2021-02-10T11:32:20.426 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:20.409 7eff7fe32700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff780e5ed0 con 0x7eff780d1c70 2021-02-10T11:32:20.426 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:20.409 7eff7fe32700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff781b0be0 con 0x7eff780d0340 2021-02-10T11:32:20.426 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:20.409 7eff7e44f700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7eff780d1c70 0x7eff780efc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:20.427 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:20.409 7eff7e44f700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7eff780d1c70 0x7eff780efc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:34368/0 (socket says 172.21.2.102:34368) 2021-02-10T11:32:20.427 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:20.410 7eff7e44f700 1 -- 172.21.2.102:0/2440761775 learned_addr learned my addr 172.21.2.102:0/2440761775 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:32:20.427 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:20.410 7eff7e44f700 1 -- 172.21.2.102:0/2440761775 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7eff780d0340 msgr2=0x7eff780ef640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:32:20.427 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:20.410 7eff7e44f700 1 --2- 172.21.2.102:0/2440761775 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7eff780d0340 0x7eff780ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:20.427 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:20.410 7eff7e44f700 1 -- 172.21.2.102:0/2440761775 >> v1:172.21.2.113:6789/0 conn(0x7eff780d1870 legacy=0x7eff781b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:32:20.427 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:20.410 7eff7e44f700 1 -- 172.21.2.102:0/2440761775 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff780f20a0 con 0x7eff780d1c70 2021-02-10T11:32:20.427 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:20.410 7eff7e44f700 1 --2- 172.21.2.102:0/2440761775 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7eff780d1c70 0x7eff780efc60 secure :-1 s=READY pgs=2250 cs=0 l=1 rev1=1 rx=0x7eff70006f00 tx=0x7eff70009f10).ready entity=mon.0 client_cookie=4b442904a2b27c9a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:20.427 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:20.410 7eff7cc4c700 1 -- 172.21.2.102:0/2440761775 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7eff7000b3b0 con 0x7eff780d1c70 2021-02-10T11:32:20.427 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:20.410 7eff7cc4c700 1 -- 172.21.2.102:0/2440761775 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff7000b630 con 0x7eff780d1c70 2021-02-10T11:32:20.427 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:20.410 7eff7cc4c700 1 -- 172.21.2.102:0/2440761775 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7eff7000bb50 con 0x7eff780d1c70 2021-02-10T11:32:20.427 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:20.410 7eff7fe32700 1 -- 172.21.2.102:0/2440761775 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7eff780d1c70 msgr2=0x7eff780efc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:20.427 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:20.410 7eff7fe32700 1 --2- 172.21.2.102:0/2440761775 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7eff780d1c70 0x7eff780efc60 secure :-1 s=READY pgs=2250 cs=0 l=1 rev1=1 rx=0x7eff70006f00 tx=0x7eff70009f10).stop 2021-02-10T11:32:20.427 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:20.411 7eff7fe32700 1 -- 172.21.2.102:0/2440761775 shutdown_connections 2021-02-10T11:32:20.428 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:20.411 7eff7fe32700 1 --2- 172.21.2.102:0/2440761775 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7eff780d0340 0x7eff780ef640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:20.428 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:20.411 7eff7fe32700 1 --2- 172.21.2.102:0/2440761775 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7eff780d1c70 0x7eff780efc60 unknown :-1 s=CLOSED pgs=2250 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:20.428 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:20.411 7eff7fe32700 1 -- 172.21.2.102:0/2440761775 >> 172.21.2.102:0/2440761775 conn(0x7eff780ce450 msgr2=0x7eff780d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:20.428 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:20.411 7eff7fe32700 1 -- 172.21.2.102:0/2440761775 shutdown_connections 2021-02-10T11:32:20.428 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:20.411 7eff7fe32700 1 -- 172.21.2.102:0/2440761775 wait complete. 2021-02-10T11:32:20.429 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:20.426 7eff7fe32700 1 Processor -- start 2021-02-10T11:32:20.429 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:20.426 7eff7fe32700 1 -- start start 2021-02-10T11:32:20.430 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:20.426 7eff7fe32700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7eff780ade80 0x7eff781a3620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:20.430 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:20.426 7eff7fe32700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7eff781a3c40 0x7eff780f0070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:20.430 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:20.426 7eff7fe32700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7eff780f0690 0x7eff78091db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:20.430 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:20.426 7eff7fe32700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff780ae400 con 0x7eff781a3c40 2021-02-10T11:32:20.430 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:20.426 7eff7fe32700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff780923d0 con 0x7eff780ade80 2021-02-10T11:32:20.430 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:20.426 7eff7fe32700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff780925f0 con 0x7eff780f0690 2021-02-10T11:32:20.430 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:20.426 7eff7d44d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7eff781a3c40 0x7eff780f0070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:20.430 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:20.426 7eff7d44d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7eff781a3c40 0x7eff780f0070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:34372/0 (socket says 172.21.2.102:34372) 2021-02-10T11:32:20.430 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:20.426 7eff7d44d700 1 -- 172.21.2.102:0/2385766172 learned_addr learned my addr 172.21.2.102:0/2385766172 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:32:20.430 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:20.427 7eff7d44d700 1 -- 172.21.2.102:0/2385766172 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7eff780f0690 msgr2=0x7eff78091db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:20.430 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:20.427 7eff7d44d700 1 --2- 172.21.2.102:0/2385766172 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7eff780f0690 0x7eff78091db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:20.430 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:20.427 7eff7d44d700 1 -- 172.21.2.102:0/2385766172 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7eff780ade80 msgr2=0x7eff781a3620 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:32:20.430 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:20.427 7eff7d44d700 1 --2- 172.21.2.102:0/2385766172 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7eff780ade80 0x7eff781a3620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:20.431 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:20.427 7eff7d44d700 1 -- 172.21.2.102:0/2385766172 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff78092f50 con 0x7eff781a3c40 2021-02-10T11:32:20.431 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:20.427 7eff7d44d700 1 --2- 172.21.2.102:0/2385766172 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7eff781a3c40 0x7eff780f0070 secure :-1 s=READY pgs=2251 cs=0 l=1 rev1=1 rx=0x7eff7400b530 tx=0x7eff7400b8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:20.431 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:20.427 7eff6a7fc700 1 -- 172.21.2.102:0/2385766172 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7eff7400bcc0 con 0x7eff781a3c40 2021-02-10T11:32:20.431 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:20.427 7eff6a7fc700 1 -- 172.21.2.102:0/2385766172 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff74000c90 con 0x7eff781a3c40 2021-02-10T11:32:20.431 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:20.427 7eff6a7fc700 1 -- 172.21.2.102:0/2385766172 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7eff740022a0 con 0x7eff781a3c40 2021-02-10T11:32:20.431 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:20.427 7eff7fe32700 1 -- 172.21.2.102:0/2385766172 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff78092f50 con 0x7eff781a3c40 2021-02-10T11:32:20.431 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:20.427 7eff7fe32700 1 -- 172.21.2.102:0/2385766172 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff78092f50 con 0x7eff781a3c40 2021-02-10T11:32:20.431 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:20.428 7eff6a7fc700 1 -- 172.21.2.102:0/2385766172 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7eff74005310 con 0x7eff781a3c40 2021-02-10T11:32:20.431 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:20.428 7eff6a7fc700 1 --2- 172.21.2.102:0/2385766172 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7eff600186c0 0x7eff6001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:20.431 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:20.428 7eff6a7fc700 1 -- 172.21.2.102:0/2385766172 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5573+0+0 (secure 0 0 0) 0x7eff74032530 con 0x7eff781a3c40 2021-02-10T11:32:20.431 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:20.428 7eff7dc4e700 1 --2- 172.21.2.102:0/2385766172 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7eff600186c0 0x7eff6001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:20.431 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:20.428 7eff7dc4e700 1 --2- 172.21.2.102:0/2385766172 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7eff600186c0 0x7eff6001abc0 crc :-1 s=READY pgs=992 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:20.460 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:20.459 7eff7fe32700 1 -- 172.21.2.102:0/2385766172 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff780ebc40 con 0x7eff781a3c40 2021-02-10T11:32:20.462 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:20.462 7eff6a7fc700 1 -- 172.21.2.102:0/2385766172 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7eff74025f60 con 0x7eff781a3c40 2021-02-10T11:32:20.611 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:20.610 7eff7fe32700 1 -- 172.21.2.102:0/2385766172 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7eff780ebc40 con 0x7eff781a3c40 2021-02-10T11:32:20.614 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:20.614 7eff6a7fc700 1 -- 172.21.2.102:0/2385766172 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v29) v1 ==== 104+0+0 (secure 0 0 0) 0x7eff74025f60 con 0x7eff781a3c40 2021-02-10T11:32:20.620 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:20.619 7eff7fe32700 1 -- 172.21.2.102:0/2385766172 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7eff600186c0 msgr2=0x7eff6001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:20.620 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:20.619 7eff7fe32700 1 --2- 172.21.2.102:0/2385766172 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7eff600186c0 0x7eff6001abc0 crc :-1 s=READY pgs=992 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:20.620 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:20.619 7eff7fe32700 1 -- 172.21.2.102:0/2385766172 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7eff781a3c40 msgr2=0x7eff780f0070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:20.620 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:20.619 7eff7fe32700 1 --2- 172.21.2.102:0/2385766172 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7eff781a3c40 0x7eff780f0070 secure :-1 s=READY pgs=2251 cs=0 l=1 rev1=1 rx=0x7eff7400b530 tx=0x7eff7400b8d0).stop 2021-02-10T11:32:20.620 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:20.620 7eff7fe32700 1 -- 172.21.2.102:0/2385766172 shutdown_connections 2021-02-10T11:32:20.621 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:20.620 7eff7fe32700 1 --2- 172.21.2.102:0/2385766172 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7eff780f0690 0x7eff78091db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:20.621 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:20.620 7eff7fe32700 1 --2- 172.21.2.102:0/2385766172 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7eff781a3c40 0x7eff780f0070 unknown :-1 s=CLOSED pgs=2251 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:20.621 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:20.620 7eff7fe32700 1 --2- 172.21.2.102:0/2385766172 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7eff600186c0 0x7eff6001abc0 unknown :-1 s=CLOSED pgs=992 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:20.621 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:20.620 7eff7fe32700 1 --2- 172.21.2.102:0/2385766172 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7eff780ade80 0x7eff781a3620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:20.621 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:20.620 7eff7fe32700 1 -- 172.21.2.102:0/2385766172 >> 172.21.2.102:0/2385766172 conn(0x7eff780955e0 msgr2=0x7eff780bfa80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:20.621 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:20.620 7eff7fe32700 1 -- 172.21.2.102:0/2385766172 shutdown_connections 2021-02-10T11:32:20.621 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:20.620 7eff7fe32700 1 -- 172.21.2.102:0/2385766172 wait complete. 2021-02-10T11:32:20.628 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestVolumes.test_default_uid_gid_subvolume_group' 2021-02-10T11:32:21.056 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:21.039 7f1b9e80b700 1 Processor -- start 2021-02-10T11:32:21.056 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:21.039 7f1b9e80b700 1 -- start start 2021-02-10T11:32:21.056 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:21.040 7f1b9e80b700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1b980d1870 0x7f1b981b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:21.056 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:21.040 7f1b9e80b700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f1b981b0a10 0x7f1b980cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:21.057 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:21.040 7f1b9e80b700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b980f0b10 con 0x7f1b980d1c70 2021-02-10T11:32:21.057 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:21.040 7f1b9e80b700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b980fe230 con 0x7f1b981b0a10 2021-02-10T11:32:21.057 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:21.040 7f1b9e80b700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b980e5ed0 con 0x7f1b980d1870 2021-02-10T11:32:21.057 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:21.040 7f1b977fe700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f1b981b0a10 0x7f1b980cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:21.057 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:21.040 7f1b977fe700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f1b981b0a10 0x7f1b980cf330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:34380/0 (socket says 172.21.2.102:34380) 2021-02-10T11:32:21.057 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:21.040 7f1b977fe700 1 -- 172.21.2.102:0/1069271717 learned_addr learned my addr 172.21.2.102:0/1069271717 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:32:21.057 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:21.040 7f1b977fe700 1 -- 172.21.2.102:0/1069271717 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1b980d1870 msgr2=0x7f1b981b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:32:21.057 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:21.040 7f1b977fe700 1 --2- 172.21.2.102:0/1069271717 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1b980d1870 0x7f1b981b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:21.057 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:21.040 7f1b977fe700 1 -- 172.21.2.102:0/1069271717 >> v1:172.21.2.113:6789/0 conn(0x7f1b980d1c70 legacy=0x7f1b980cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:32:21.057 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:21.040 7f1b977fe700 1 -- 172.21.2.102:0/1069271717 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b980f0650 con 0x7f1b981b0a10 2021-02-10T11:32:21.057 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:21.041 7f1b977fe700 1 --2- 172.21.2.102:0/1069271717 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1b981b0a10 0x7f1b980cf330 secure :-1 s=READY pgs=2252 cs=0 l=1 rev1=1 rx=0x7f1b8c006700 tx=0x7f1b8c009b80).ready entity=mon.0 client_cookie=1ebfa70975e08670 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:21.058 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:21.041 7f1b96ffd700 1 -- 172.21.2.102:0/1069271717 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1b8c00c0c0 con 0x7f1b981b0a10 2021-02-10T11:32:21.058 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:21.041 7f1b96ffd700 1 -- 172.21.2.102:0/1069271717 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b8c00c440 con 0x7f1b981b0a10 2021-02-10T11:32:21.058 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:21.041 7f1b96ffd700 1 -- 172.21.2.102:0/1069271717 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1b8c00c960 con 0x7f1b981b0a10 2021-02-10T11:32:21.058 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:21.041 7f1b9e80b700 1 -- 172.21.2.102:0/1069271717 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1b981b0a10 msgr2=0x7f1b980cf330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:21.058 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:21.041 7f1b9e80b700 1 --2- 172.21.2.102:0/1069271717 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1b981b0a10 0x7f1b980cf330 secure :-1 s=READY pgs=2252 cs=0 l=1 rev1=1 rx=0x7f1b8c006700 tx=0x7f1b8c009b80).stop 2021-02-10T11:32:21.058 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:21.041 7f1b9e80b700 1 -- 172.21.2.102:0/1069271717 shutdown_connections 2021-02-10T11:32:21.058 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:21.041 7f1b9e80b700 1 --2- 172.21.2.102:0/1069271717 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1b981b0a10 0x7f1b980cf330 unknown :-1 s=CLOSED pgs=2252 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:21.058 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:21.041 7f1b9e80b700 1 --2- 172.21.2.102:0/1069271717 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1b980d1870 0x7f1b981b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:21.058 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:21.041 7f1b9e80b700 1 -- 172.21.2.102:0/1069271717 >> 172.21.2.102:0/1069271717 conn(0x7f1b980ce450 msgr2=0x7f1b980d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:21.058 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:21.042 7f1b9e80b700 1 -- 172.21.2.102:0/1069271717 shutdown_connections 2021-02-10T11:32:21.058 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:21.042 7f1b9e80b700 1 -- 172.21.2.102:0/1069271717 wait complete. 2021-02-10T11:32:21.060 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:21.056 7f1b9e80b700 1 Processor -- start 2021-02-10T11:32:21.060 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:21.056 7f1b9e80b700 1 -- start start 2021-02-10T11:32:21.060 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:21.057 7f1b9e80b700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1b980b0ad0 0x7f1b98091f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:21.060 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:21.057 7f1b9e80b700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1b981a8850 0x7f1b981a8c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:21.060 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:21.057 7f1b9e80b700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1b981a6fa0 0x7f1b981a73a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:21.060 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:21.057 7f1b9e80b700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b981abfe0 con 0x7f1b980b0ad0 2021-02-10T11:32:21.060 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:21.057 7f1b9e80b700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b981a9c40 con 0x7f1b981a8850 2021-02-10T11:32:21.060 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:21.057 7f1b9e80b700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b981a9730 con 0x7f1b981a6fa0 2021-02-10T11:32:21.060 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:21.057 7f1b97fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1b980b0ad0 0x7f1b98091f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:21.060 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:21.057 7f1b97fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1b980b0ad0 0x7f1b98091f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:34384/0 (socket says 172.21.2.102:34384) 2021-02-10T11:32:21.060 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:21.057 7f1b97fff700 1 -- 172.21.2.102:0/537548443 learned_addr learned my addr 172.21.2.102:0/537548443 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:32:21.061 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:21.057 7f1b97fff700 1 -- 172.21.2.102:0/537548443 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1b981a6fa0 msgr2=0x7f1b981a73a0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:32:21.061 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:21.057 7f1b97fff700 1 --2- 172.21.2.102:0/537548443 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1b981a6fa0 0x7f1b981a73a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:21.061 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:21.057 7f1b97fff700 1 -- 172.21.2.102:0/537548443 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1b981a8850 msgr2=0x7f1b981a8c50 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:32:21.061 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:21.057 7f1b97fff700 1 --2- 172.21.2.102:0/537548443 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1b981a8850 0x7f1b981a8c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:21.061 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:21.057 7f1b97fff700 1 -- 172.21.2.102:0/537548443 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b981a9270 con 0x7f1b980b0ad0 2021-02-10T11:32:21.061 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:21.057 7f1b97fff700 1 --2- 172.21.2.102:0/537548443 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1b980b0ad0 0x7f1b98091f30 secure :-1 s=READY pgs=2253 cs=0 l=1 rev1=1 rx=0x7f1b88006f00 tx=0x7f1b8800b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:21.061 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:21.058 7f1b94ff9700 1 -- 172.21.2.102:0/537548443 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1b8800bd30 con 0x7f1b980b0ad0 2021-02-10T11:32:21.061 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:21.058 7f1b94ff9700 1 -- 172.21.2.102:0/537548443 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b88000c90 con 0x7f1b980b0ad0 2021-02-10T11:32:21.061 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:21.058 7f1b9e80b700 1 -- 172.21.2.102:0/537548443 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b981a9270 con 0x7f1b980b0ad0 2021-02-10T11:32:21.061 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:21.058 7f1b9e80b700 1 -- 172.21.2.102:0/537548443 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b981a7cb0 con 0x7f1b980b0ad0 2021-02-10T11:32:21.062 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:21.058 7f1b94ff9700 1 -- 172.21.2.102:0/537548443 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1b880022a0 con 0x7f1b980b0ad0 2021-02-10T11:32:21.062 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:21.058 7f1b94ff9700 1 -- 172.21.2.102:0/537548443 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1b88005310 con 0x7f1b980b0ad0 2021-02-10T11:32:21.062 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:21.058 7f1b94ff9700 1 --2- 172.21.2.102:0/537548443 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1b780186d0 0x7f1b7801abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:21.062 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:21.058 7f1b94ff9700 1 -- 172.21.2.102:0/537548443 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f1b88032660 con 0x7f1b980b0ad0 2021-02-10T11:32:21.062 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:21.058 7f1b977fe700 1 --2- 172.21.2.102:0/537548443 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1b780186d0 0x7f1b7801abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:21.062 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:21.059 7f1b977fe700 1 --2- 172.21.2.102:0/537548443 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1b780186d0 0x7f1b7801abd0 crc :-1 s=READY pgs=993 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:21.090 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:21.090 7f1b9e80b700 1 -- 172.21.2.102:0/537548443 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b980ce450 con 0x7f1b980b0ad0 2021-02-10T11:32:21.093 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:21.092 7f1b94ff9700 1 -- 172.21.2.102:0/537548443 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f1b88000940 con 0x7f1b980b0ad0 2021-02-10T11:32:21.224 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:21.224 7f1b9e80b700 1 -- 172.21.2.102:0/537548443 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestVolumes.test_default_uid_gid_subvolume_group"]} v 0) v1 -- 0x7f1b980ce450 con 0x7f1b980b0ad0 2021-02-10T11:32:21.477 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:21.476 7f1b94ff9700 1 -- 172.21.2.102:0/537548443 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestVolumes.test_default_uid_gid_subvolume_group"]}]=0 v594) v1 ==== 153+0+0 (secure 0 0 0) 0x7f1b88000c90 con 0x7f1b980b0ad0 2021-02-10T11:32:21.492 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:21.491 7f1b9e80b700 1 -- 172.21.2.102:0/537548443 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1b780186d0 msgr2=0x7f1b7801abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:21.492 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:21.491 7f1b9e80b700 1 --2- 172.21.2.102:0/537548443 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1b780186d0 0x7f1b7801abd0 crc :-1 s=READY pgs=993 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:21.493 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:21.491 7f1b9e80b700 1 -- 172.21.2.102:0/537548443 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1b980b0ad0 msgr2=0x7f1b98091f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:21.493 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:21.491 7f1b9e80b700 1 --2- 172.21.2.102:0/537548443 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1b980b0ad0 0x7f1b98091f30 secure :-1 s=READY pgs=2253 cs=0 l=1 rev1=1 rx=0x7f1b88006f00 tx=0x7f1b8800b940).stop 2021-02-10T11:32:21.493 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:21.491 7f1b9e80b700 1 -- 172.21.2.102:0/537548443 shutdown_connections 2021-02-10T11:32:21.493 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:21.491 7f1b9e80b700 1 --2- 172.21.2.102:0/537548443 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1b981a6fa0 0x7f1b981a73a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:21.493 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:21.491 7f1b9e80b700 1 --2- 172.21.2.102:0/537548443 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1b780186d0 0x7f1b7801abd0 unknown :-1 s=CLOSED pgs=993 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:21.493 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:21.491 7f1b9e80b700 1 --2- 172.21.2.102:0/537548443 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1b981a8850 0x7f1b981a8c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:21.494 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:21.491 7f1b9e80b700 1 --2- 172.21.2.102:0/537548443 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1b980b0ad0 0x7f1b98091f30 unknown :-1 s=CLOSED pgs=2253 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:21.494 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:21.491 7f1b9e80b700 1 -- 172.21.2.102:0/537548443 >> 172.21.2.102:0/537548443 conn(0x7f1b9819f8f0 msgr2=0x7f1b981a5f80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:21.494 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:21.492 7f1b9e80b700 1 -- 172.21.2.102:0/537548443 shutdown_connections 2021-02-10T11:32:21.494 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:21.492 7f1b9e80b700 1 -- 172.21.2.102:0/537548443 wait complete. 2021-02-10T11:32:21.501 INFO:tasks.cephfs_test_runner:test_default_uid_gid_subvolume_group (tasks.cephfs.test_volumes.TestVolumes) ... ok 2021-02-10T11:32:21.501 INFO:tasks.cephfs_test_runner:Starting test: test_multitenant_subvolumes (tasks.cephfs.test_volumes.TestVolumes) 2021-02-10T11:32:21.502 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestVolumes.test_multitenant_subvolumes' 2021-02-10T11:32:21.930 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:21.913 7fa9fdac8700 1 Processor -- start 2021-02-10T11:32:21.931 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:21.913 7fa9fdac8700 1 -- start start 2021-02-10T11:32:21.931 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:21.913 7fa9fdac8700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fa9f80d1c70 0x7fa9f80f20a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:21.931 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:21.913 7fa9fdac8700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9f80f0b10 con 0x7fa9f80d1870 2021-02-10T11:32:21.931 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:21.913 7fa9fdac8700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9f81b0be0 con 0x7fa9f80d0340 2021-02-10T11:32:21.931 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:21.913 7fa9fdac8700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9f80e5ed0 con 0x7fa9f80d1c70 2021-02-10T11:32:21.931 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:21.914 7fa9f7fff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fa9f80d1c70 0x7fa9f80f20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:21.932 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:21.914 7fa9f7fff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fa9f80d1c70 0x7fa9f80f20a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:34396/0 (socket says 172.21.2.102:34396) 2021-02-10T11:32:21.932 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:21.914 7fa9f7fff700 1 -- 172.21.2.102:0/2970593741 learned_addr learned my addr 172.21.2.102:0/2970593741 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:32:21.932 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:21.914 7fa9f77fe700 1 --1- 172.21.2.102:0/2970593741 >> v1:172.21.2.102:6789/0 conn(0x7fa9f80d1870 0x7fa9f81b03f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:54848/0 (socket says 172.21.2.102:54848) 2021-02-10T11:32:21.932 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:21.914 7fa9f7fff700 1 -- 172.21.2.102:0/2970593741 >> v1:172.21.2.113:6789/0 conn(0x7fa9f80d0340 legacy=0x7fa9f80ef640 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:32:21.932 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:21.914 7fa9f7fff700 1 -- 172.21.2.102:0/2970593741 >> v1:172.21.2.102:6789/0 conn(0x7fa9f80d1870 legacy=0x7fa9f81b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:21.932 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:21.914 7fa9f7fff700 1 -- 172.21.2.102:0/2970593741 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9f80efe30 con 0x7fa9f80d1c70 2021-02-10T11:32:21.933 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:21.914 7fa9f7fff700 1 --2- 172.21.2.102:0/2970593741 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa9f80d1c70 0x7fa9f80f20a0 secure :-1 s=READY pgs=2255 cs=0 l=1 rev1=1 rx=0x7fa9f0006f00 tx=0x7fa9f0009f10).ready entity=mon.0 client_cookie=7a9aa0bb90f42eb4 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:21.933 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:21.914 7fa9f67fc700 1 -- 172.21.2.102:0/2970593741 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa9f000b3b0 con 0x7fa9f80d1c70 2021-02-10T11:32:21.933 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:21.914 7fa9f67fc700 1 -- 172.21.2.102:0/2970593741 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9f000b730 con 0x7fa9f80d1c70 2021-02-10T11:32:21.933 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:21.914 7fa9f67fc700 1 -- 172.21.2.102:0/2970593741 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa9f000bc10 con 0x7fa9f80d1c70 2021-02-10T11:32:21.933 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:21.914 7fa9fdac8700 1 -- 172.21.2.102:0/2970593741 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa9f80d1c70 msgr2=0x7fa9f80f20a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:21.933 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:21.914 7fa9fdac8700 1 --2- 172.21.2.102:0/2970593741 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa9f80d1c70 0x7fa9f80f20a0 secure :-1 s=READY pgs=2255 cs=0 l=1 rev1=1 rx=0x7fa9f0006f00 tx=0x7fa9f0009f10).stop 2021-02-10T11:32:21.934 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:21.915 7fa9fdac8700 1 -- 172.21.2.102:0/2970593741 shutdown_connections 2021-02-10T11:32:21.934 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:21.915 7fa9fdac8700 1 --2- 172.21.2.102:0/2970593741 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa9f80d1c70 0x7fa9f80f20a0 unknown :-1 s=CLOSED pgs=2255 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:21.934 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:21.915 7fa9fdac8700 1 -- 172.21.2.102:0/2970593741 >> 172.21.2.102:0/2970593741 conn(0x7fa9f80ce450 msgr2=0x7fa9f80d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:21.934 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:21.915 7fa9fdac8700 1 -- 172.21.2.102:0/2970593741 shutdown_connections 2021-02-10T11:32:21.934 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:21.915 7fa9fdac8700 1 -- 172.21.2.102:0/2970593741 wait complete. 2021-02-10T11:32:21.934 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:21.930 7fa9fdac8700 1 Processor -- start 2021-02-10T11:32:21.935 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:21.930 7fa9fdac8700 1 -- start start 2021-02-10T11:32:21.937 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:21.930 7fa9fdac8700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa9f81c6350 0x7fa9f809a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:21.937 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:21.930 7fa9fdac8700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa9f81c6750 0x7fa9f80b17a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:21.937 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:21.930 7fa9fdac8700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa9f80b1dc0 0x7fa9f809bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:21.937 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:21.930 7fa9fdac8700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9f81c6b50 con 0x7fa9f81c6350 2021-02-10T11:32:21.938 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:21.930 7fa9fdac8700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9f80b21c0 con 0x7fa9f80b1dc0 2021-02-10T11:32:21.938 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:21.930 7fa9fdac8700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9f809af60 con 0x7fa9f81c6750 2021-02-10T11:32:21.938 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:21.930 7fa9f77fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa9f81c6350 0x7fa9f809a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:21.938 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:21.931 7fa9f77fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa9f81c6350 0x7fa9f809a830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:34398/0 (socket says 172.21.2.102:34398) 2021-02-10T11:32:21.938 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:21.931 7fa9f77fe700 1 -- 172.21.2.102:0/955145441 learned_addr learned my addr 172.21.2.102:0/955145441 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:32:21.938 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:21.931 7fa9f77fe700 1 -- 172.21.2.102:0/955145441 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa9f81c6750 msgr2=0x7fa9f80b17a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:21.938 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:21.931 7fa9f77fe700 1 --2- 172.21.2.102:0/955145441 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa9f81c6750 0x7fa9f80b17a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:21.938 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:21.931 7fa9f77fe700 1 -- 172.21.2.102:0/955145441 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa9f80b1dc0 msgr2=0x7fa9f809bfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:21.938 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:21.931 7fa9f77fe700 1 --2- 172.21.2.102:0/955145441 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa9f80b1dc0 0x7fa9f809bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:21.939 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:21.931 7fa9f77fe700 1 -- 172.21.2.102:0/955145441 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9f80f2c00 con 0x7fa9f81c6350 2021-02-10T11:32:21.939 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:21.931 7fa9f77fe700 1 --2- 172.21.2.102:0/955145441 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa9f81c6350 0x7fa9f809a830 secure :-1 s=READY pgs=2256 cs=0 l=1 rev1=1 rx=0x7fa9e0001c60 tx=0x7fa9e000a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:21.939 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:21.931 7fa9effff700 1 -- 172.21.2.102:0/955145441 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa9e000acc0 con 0x7fa9f81c6350 2021-02-10T11:32:21.939 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:21.931 7fa9effff700 1 -- 172.21.2.102:0/955145441 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9e0004140 con 0x7fa9f81c6350 2021-02-10T11:32:21.939 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:21.931 7fa9effff700 1 -- 172.21.2.102:0/955145441 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa9e0004660 con 0x7fa9f81c6350 2021-02-10T11:32:21.939 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:21.931 7fa9fdac8700 1 -- 172.21.2.102:0/955145441 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9f80f2c00 con 0x7fa9f81c6350 2021-02-10T11:32:21.940 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:21.931 7fa9fdac8700 1 -- 172.21.2.102:0/955145441 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9f80f2c00 con 0x7fa9f81c6350 2021-02-10T11:32:21.940 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:21.932 7fa9effff700 1 -- 172.21.2.102:0/955145441 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa9e0000da0 con 0x7fa9f81c6350 2021-02-10T11:32:21.940 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:21.932 7fa9effff700 1 --2- 172.21.2.102:0/955145441 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa9dc0186c0 0x7fa9dc01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:21.940 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:21.932 7fa9effff700 1 -- 172.21.2.102:0/955145441 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fa9e0032560 con 0x7fa9f81c6350 2021-02-10T11:32:21.940 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:21.932 7fa9f6ffd700 1 --2- 172.21.2.102:0/955145441 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa9dc0186c0 0x7fa9dc01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:21.940 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:21.932 7fa9f6ffd700 1 --2- 172.21.2.102:0/955145441 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa9dc0186c0 0x7fa9dc01abc0 crc :-1 s=READY pgs=994 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:21.964 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:21.963 7fa9fdac8700 1 -- 172.21.2.102:0/955145441 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9f80e7df0 con 0x7fa9f81c6350 2021-02-10T11:32:21.966 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:21.966 7fa9effff700 1 -- 172.21.2.102:0/955145441 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fa9e00009b0 con 0x7fa9f81c6350 2021-02-10T11:32:22.097 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:22.097 7fa9fdac8700 1 -- 172.21.2.102:0/955145441 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestVolumes.test_multitenant_subvolumes"]} v 0) v1 -- 0x7fa9f80ce450 con 0x7fa9f81c6350 2021-02-10T11:32:22.480 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:22.480 7fa9effff700 1 -- 172.21.2.102:0/955145441 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestVolumes.test_multitenant_subvolumes"]}]=0 v595) v1 ==== 147+0+0 (secure 0 0 0) 0x7fa9e0000da0 con 0x7fa9f81c6350 2021-02-10T11:32:22.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:22.516 7fa9fdac8700 1 -- 172.21.2.102:0/955145441 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa9dc0186c0 msgr2=0x7fa9dc01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:22.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:22.516 7fa9fdac8700 1 --2- 172.21.2.102:0/955145441 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa9dc0186c0 0x7fa9dc01abc0 crc :-1 s=READY pgs=994 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:22.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:22.516 7fa9fdac8700 1 -- 172.21.2.102:0/955145441 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa9f81c6350 msgr2=0x7fa9f809a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:22.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:22.516 7fa9fdac8700 1 --2- 172.21.2.102:0/955145441 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa9f81c6350 0x7fa9f809a830 secure :-1 s=READY pgs=2256 cs=0 l=1 rev1=1 rx=0x7fa9e0001c60 tx=0x7fa9e000a8d0).stop 2021-02-10T11:32:22.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:22.516 7fa9fdac8700 1 -- 172.21.2.102:0/955145441 shutdown_connections 2021-02-10T11:32:22.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:22.517 7fa9fdac8700 1 --2- 172.21.2.102:0/955145441 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa9dc0186c0 0x7fa9dc01abc0 unknown :-1 s=CLOSED pgs=994 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:22.518 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:22.517 7fa9fdac8700 1 --2- 172.21.2.102:0/955145441 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa9f80b1dc0 0x7fa9f809bfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:22.518 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:22.517 7fa9fdac8700 1 --2- 172.21.2.102:0/955145441 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa9f81c6750 0x7fa9f80b17a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:22.518 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:22.517 7fa9fdac8700 1 --2- 172.21.2.102:0/955145441 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa9f81c6350 0x7fa9f809a830 unknown :-1 s=CLOSED pgs=2256 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:22.518 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:22.517 7fa9fdac8700 1 -- 172.21.2.102:0/955145441 >> 172.21.2.102:0/955145441 conn(0x7fa9f80a0450 msgr2=0x7fa9f80ec020 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:22.518 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:22.517 7fa9fdac8700 1 -- 172.21.2.102:0/955145441 shutdown_connections 2021-02-10T11:32:22.518 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:22.517 7fa9fdac8700 1 -- 172.21.2.102:0/955145441 wait complete. 2021-02-10T11:32:22.525 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config set mon mon_allow_pool_delete True 2021-02-10T11:32:22.958 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:22.940 7f25c611f700 1 Processor -- start 2021-02-10T11:32:22.958 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:22.940 7f25c611f700 1 -- start start 2021-02-10T11:32:22.958 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:22.941 7f25c611f700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f25c00d1870 0x7f25c01b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:22.959 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:22.941 7f25c611f700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f25c00cff70 0x7f25c0109130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:22.959 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:22.941 7f25c611f700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25c0109a60 con 0x7f25c01b0ae0 2021-02-10T11:32:22.959 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:22.941 7f25c611f700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25c0109750 con 0x7f25c00d1870 2021-02-10T11:32:22.959 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:22.941 7f25c611f700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25c0088070 con 0x7f25c00cff70 2021-02-10T11:32:22.959 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:22.941 7f25bf7fe700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f25c00d1870 0x7f25c01b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:22.959 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:22.941 7f25bf7fe700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f25c00d1870 0x7f25c01b04c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:34406/0 (socket says 172.21.2.102:34406) 2021-02-10T11:32:22.959 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:22.941 7f25bf7fe700 1 -- 172.21.2.102:0/1431907894 learned_addr learned my addr 172.21.2.102:0/1431907894 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:32:22.959 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:22.941 7f25bf7fe700 1 -- 172.21.2.102:0/1431907894 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f25c00cff70 msgr2=0x7f25c0109130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:32:22.959 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:22.941 7f25bf7fe700 1 --2- 172.21.2.102:0/1431907894 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f25c00cff70 0x7f25c0109130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:22.959 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:22.941 7f25bf7fe700 1 -- 172.21.2.102:0/1431907894 >> v1:172.21.2.113:6789/0 conn(0x7f25c01b0ae0 legacy=0x7f25c0108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:32:22.959 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:22.941 7f25bf7fe700 1 -- 172.21.2.102:0/1431907894 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25c00d1c70 con 0x7f25c00d1870 2021-02-10T11:32:22.959 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:22.941 7f25bf7fe700 1 --2- 172.21.2.102:0/1431907894 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f25c00d1870 0x7f25c01b04c0 secure :-1 s=READY pgs=2257 cs=0 l=1 rev1=1 rx=0x7f25b0006700 tx=0x7f25b0009b80).ready entity=mon.0 client_cookie=c6c1d2b522dce3af server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:22.960 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:22.942 7f25be7fc700 1 -- 172.21.2.102:0/1431907894 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f25b000c0c0 con 0x7f25c00d1870 2021-02-10T11:32:22.960 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:22.942 7f25be7fc700 1 -- 172.21.2.102:0/1431907894 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25b000c440 con 0x7f25c00d1870 2021-02-10T11:32:22.960 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:22.942 7f25be7fc700 1 -- 172.21.2.102:0/1431907894 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f25b000c960 con 0x7f25c00d1870 2021-02-10T11:32:22.960 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:22.942 7f25c611f700 1 -- 172.21.2.102:0/1431907894 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f25c00d1870 msgr2=0x7f25c01b04c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:22.960 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:22.942 7f25c611f700 1 --2- 172.21.2.102:0/1431907894 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f25c00d1870 0x7f25c01b04c0 secure :-1 s=READY pgs=2257 cs=0 l=1 rev1=1 rx=0x7f25b0006700 tx=0x7f25b0009b80).stop 2021-02-10T11:32:22.960 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:22.942 7f25c611f700 1 -- 172.21.2.102:0/1431907894 shutdown_connections 2021-02-10T11:32:22.960 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:22.942 7f25c611f700 1 --2- 172.21.2.102:0/1431907894 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f25c00cff70 0x7f25c0109130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:22.960 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:22.942 7f25c611f700 1 --2- 172.21.2.102:0/1431907894 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f25c00d1870 0x7f25c01b04c0 unknown :-1 s=CLOSED pgs=2257 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:22.960 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:22.942 7f25c611f700 1 -- 172.21.2.102:0/1431907894 >> 172.21.2.102:0/1431907894 conn(0x7f25c00ce420 msgr2=0x7f25c00d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:22.960 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:22.943 7f25c611f700 1 -- 172.21.2.102:0/1431907894 shutdown_connections 2021-02-10T11:32:22.960 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:22.943 7f25c611f700 1 -- 172.21.2.102:0/1431907894 wait complete. 2021-02-10T11:32:22.960 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:22.957 7f25c611f700 1 Processor -- start 2021-02-10T11:32:22.960 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:22.957 7f25c611f700 1 -- start start 2021-02-10T11:32:22.960 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:22.958 7f25c611f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f25c00c2370 0x7f25c018f870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:22.961 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:22.958 7f25c611f700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f25c0084cb0 0x7f25c01a7470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:22.961 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:22.958 7f25c611f700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f25c01a7a90 0x7f25c0085700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:22.961 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:22.958 7f25c611f700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25c00be840 con 0x7f25c00c2370 2021-02-10T11:32:22.961 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:22.958 7f25c611f700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25c00c29a0 con 0x7f25c0084cb0 2021-02-10T11:32:22.961 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:22.958 7f25c611f700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25c01a7e90 con 0x7f25c01a7a90 2021-02-10T11:32:22.961 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:22.958 7f25bf7fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f25c00c2370 0x7f25c018f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:22.961 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:22.958 7f25bf7fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f25c00c2370 0x7f25c018f870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:34410/0 (socket says 172.21.2.102:34410) 2021-02-10T11:32:22.961 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:22.958 7f25bf7fe700 1 -- 172.21.2.102:0/1983305688 learned_addr learned my addr 172.21.2.102:0/1983305688 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:32:22.962 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:22.958 7f25bf7fe700 1 -- 172.21.2.102:0/1983305688 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f25c01a7a90 msgr2=0x7f25c0085700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:22.962 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:22.958 7f25bf7fe700 1 --2- 172.21.2.102:0/1983305688 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f25c01a7a90 0x7f25c0085700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:22.962 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:22.959 7f25bf7fe700 1 -- 172.21.2.102:0/1983305688 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f25c0084cb0 msgr2=0x7f25c01a7470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:22.962 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:22.959 7f25bf7fe700 1 --2- 172.21.2.102:0/1983305688 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f25c0084cb0 0x7f25c01a7470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:22.962 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:22.959 7f25bf7fe700 1 -- 172.21.2.102:0/1983305688 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25c01a80b0 con 0x7f25c00c2370 2021-02-10T11:32:22.962 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:22.959 7f25bf7fe700 1 --2- 172.21.2.102:0/1983305688 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f25c00c2370 0x7f25c018f870 secure :-1 s=READY pgs=2258 cs=0 l=1 rev1=1 rx=0x7f25b000b5e0 tx=0x7f25b000b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:22.963 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:22.959 7f25b7fff700 1 -- 172.21.2.102:0/1983305688 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f25b000bd70 con 0x7f25c00c2370 2021-02-10T11:32:22.963 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:22.959 7f25b7fff700 1 -- 172.21.2.102:0/1983305688 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25b0002030 con 0x7f25c00c2370 2021-02-10T11:32:22.963 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:22.959 7f25b7fff700 1 -- 172.21.2.102:0/1983305688 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f25b00024d0 con 0x7f25c00c2370 2021-02-10T11:32:22.963 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:22.959 7f25c611f700 1 -- 172.21.2.102:0/1983305688 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25c01a80b0 con 0x7f25c00c2370 2021-02-10T11:32:22.963 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:22.959 7f25c611f700 1 -- 172.21.2.102:0/1983305688 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25c01a80b0 con 0x7f25c00c2370 2021-02-10T11:32:22.963 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:22.960 7f25b7fff700 1 -- 172.21.2.102:0/1983305688 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f25b0000c90 con 0x7f25c00c2370 2021-02-10T11:32:22.963 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:22.960 7f25b7fff700 1 --2- 172.21.2.102:0/1983305688 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f25a40186c0 0x7f25a401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:22.963 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:22.960 7f25b7fff700 1 -- 172.21.2.102:0/1983305688 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f25b0032470 con 0x7f25c00c2370 2021-02-10T11:32:22.963 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:22.960 7f25beffd700 1 --2- 172.21.2.102:0/1983305688 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f25a40186c0 0x7f25a401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:22.963 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:22.960 7f25beffd700 1 --2- 172.21.2.102:0/1983305688 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f25a40186c0 0x7f25a401abc0 crc :-1 s=READY pgs=995 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:22.991 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:22.991 7f25c611f700 1 -- 172.21.2.102:0/1983305688 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25c00be840 con 0x7f25c00c2370 2021-02-10T11:32:22.994 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:22.993 7f25b7fff700 1 -- 172.21.2.102:0/1983305688 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f25b0030da0 con 0x7f25c00c2370 2021-02-10T11:32:23.146 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:23.146 7f25c611f700 1 -- 172.21.2.102:0/1983305688 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f25c00be840 con 0x7f25c00c2370 2021-02-10T11:32:23.151 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:23.150 7f25b7fff700 1 -- 172.21.2.102:0/1983305688 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v30)=0 v30) v1 ==== 122+0+0 (secure 0 0 0) 0x7f25b0000c90 con 0x7f25c00c2370 2021-02-10T11:32:23.156 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:23.155 7f25c611f700 1 -- 172.21.2.102:0/1983305688 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f25a40186c0 msgr2=0x7f25a401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:23.156 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:23.155 7f25c611f700 1 --2- 172.21.2.102:0/1983305688 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f25a40186c0 0x7f25a401abc0 crc :-1 s=READY pgs=995 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:23.156 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:23.155 7f25c611f700 1 -- 172.21.2.102:0/1983305688 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f25c00c2370 msgr2=0x7f25c018f870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:23.156 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:23.155 7f25c611f700 1 --2- 172.21.2.102:0/1983305688 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f25c00c2370 0x7f25c018f870 secure :-1 s=READY pgs=2258 cs=0 l=1 rev1=1 rx=0x7f25b000b5e0 tx=0x7f25b000b980).stop 2021-02-10T11:32:23.156 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:23.155 7f25c611f700 1 -- 172.21.2.102:0/1983305688 shutdown_connections 2021-02-10T11:32:23.156 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:23.155 7f25c611f700 1 --2- 172.21.2.102:0/1983305688 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f25c01a7a90 0x7f25c0085700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:23.156 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:23.155 7f25c611f700 1 --2- 172.21.2.102:0/1983305688 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f25a40186c0 0x7f25a401abc0 unknown :-1 s=CLOSED pgs=995 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:23.156 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:23.155 7f25c611f700 1 --2- 172.21.2.102:0/1983305688 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f25c0084cb0 0x7f25c01a7470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:23.156 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:23.155 7f25c611f700 1 --2- 172.21.2.102:0/1983305688 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f25c00c2370 0x7f25c018f870 unknown :-1 s=CLOSED pgs=2258 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:23.157 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:23.155 7f25c611f700 1 -- 172.21.2.102:0/1983305688 >> 172.21.2.102:0/1983305688 conn(0x7f25c00b5dc0 msgr2=0x7f25c00c2d90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:23.157 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:23.156 7f25c611f700 1 -- 172.21.2.102:0/1983305688 shutdown_connections 2021-02-10T11:32:23.157 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:23.156 7f25c611f700 1 -- 172.21.2.102:0/1983305688 wait complete. 2021-02-10T11:32:23.164 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2021-02-10T11:32:23.165 DEBUG:teuthology.orchestra.run.gibba002:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:32:23.510 DEBUG:teuthology.orchestra.run.gibba013:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:32:23.855 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2021-02-10T11:32:23.855 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:32:23.855 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:32:23.855 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:32:24.191 INFO:teuthology.orchestra.run.gibba002.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.0’: No such file or directory 2021-02-10T11:32:24.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:32:24.191 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.0 2021-02-10T11:32:24.191 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:32:24.191 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:32:24.191 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:32:24.525 INFO:teuthology.orchestra.run.gibba002.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.1’: No such file or directory 2021-02-10T11:32:24.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:32:24.526 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2021-02-10T11:32:24.526 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:32:24.526 DEBUG:teuthology.orchestra.run.gibba013:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:32:24.526 DEBUG:teuthology.orchestra.run.gibba013:> ' -- /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:32:24.864 INFO:teuthology.orchestra.run.gibba013.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.2’: No such file or directory 2021-02-10T11:32:24.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:32:24.865 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2021-02-10T11:32:24.865 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:32:24.865 DEBUG:teuthology.orchestra.run.gibba013:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:32:24.866 DEBUG:teuthology.orchestra.run.gibba013:> ' -- /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:32:25.212 INFO:teuthology.orchestra.run.gibba013.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.3’: No such file or directory 2021-02-10T11:32:25.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:32:25.214 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.3 2021-02-10T11:32:25.214 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json-pretty 2021-02-10T11:32:25.643 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:25.626 7f9f60b07700 1 Processor -- start 2021-02-10T11:32:25.643 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:25.626 7f9f60b07700 1 -- start start 2021-02-10T11:32:25.643 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:25.626 7f9f60b07700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f9f5c0d1870 0x7f9f5c1b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:25.643 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:25.626 7f9f60b07700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f9f5c1b0a10 0x7f9f5c0cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:25.643 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:25.626 7f9f60b07700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f5c0f0b10 con 0x7f9f5c0d1c70 2021-02-10T11:32:25.643 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:25.626 7f9f60b07700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f5c0e5ed0 con 0x7f9f5c1b0a10 2021-02-10T11:32:25.643 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:25.626 7f9f60b07700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f5c0f0650 con 0x7f9f5c0d1870 2021-02-10T11:32:25.644 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:25.627 7f9f5a05d700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f9f5c1b0a10 0x7f9f5c0cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:25.644 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:25.627 7f9f5a05d700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f9f5c1b0a10 0x7f9f5c0cf330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:34422/0 (socket says 172.21.2.102:34422) 2021-02-10T11:32:25.644 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:25.627 7f9f5a05d700 1 -- 172.21.2.102:0/578557578 learned_addr learned my addr 172.21.2.102:0/578557578 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:32:25.644 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:25.627 7f9f5a05d700 1 -- 172.21.2.102:0/578557578 >> v2:172.21.2.113:3300/0 conn(0x7f9f5c0d1870 msgr2=0x7f9f5c1b03f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:32:25.644 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:25.627 7f9f5a05d700 1 --2- 172.21.2.102:0/578557578 >> v2:172.21.2.113:3300/0 conn(0x7f9f5c0d1870 0x7f9f5c1b03f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:25.644 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:25.627 7f9f5a05d700 1 -- 172.21.2.102:0/578557578 >> v1:172.21.2.113:6789/0 conn(0x7f9f5c0d1c70 legacy=0x7f9f5c0cf950 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:32:25.644 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:25.627 7f9f5a05d700 1 -- 172.21.2.102:0/578557578 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f5c0f0840 con 0x7f9f5c1b0a10 2021-02-10T11:32:25.644 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:25.627 7f9f5a05d700 1 --2- 172.21.2.102:0/578557578 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9f5c1b0a10 0x7f9f5c0cf330 secure :-1 s=READY pgs=2259 cs=0 l=1 rev1=1 rx=0x7f9f50006700 tx=0x7f9f50009b00).ready entity=mon.0 client_cookie=d67ea220cd9633b server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:25.644 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:25.628 7f9f5985c700 1 -- 172.21.2.102:0/578557578 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9f5000c0c0 con 0x7f9f5c1b0a10 2021-02-10T11:32:25.644 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:25.628 7f9f5985c700 1 -- 172.21.2.102:0/578557578 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f5000c400 con 0x7f9f5c1b0a10 2021-02-10T11:32:25.644 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:25.628 7f9f5985c700 1 -- 172.21.2.102:0/578557578 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9f5000c920 con 0x7f9f5c1b0a10 2021-02-10T11:32:25.644 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:25.628 7f9f60b07700 1 -- 172.21.2.102:0/578557578 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9f5c1b0a10 msgr2=0x7f9f5c0cf330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:25.644 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:25.628 7f9f60b07700 1 --2- 172.21.2.102:0/578557578 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9f5c1b0a10 0x7f9f5c0cf330 secure :-1 s=READY pgs=2259 cs=0 l=1 rev1=1 rx=0x7f9f50006700 tx=0x7f9f50009b00).stop 2021-02-10T11:32:25.644 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:25.628 7f9f60b07700 1 -- 172.21.2.102:0/578557578 shutdown_connections 2021-02-10T11:32:25.645 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:25.628 7f9f60b07700 1 --2- 172.21.2.102:0/578557578 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9f5c1b0a10 0x7f9f5c0cf330 unknown :-1 s=CLOSED pgs=2259 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:25.645 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:25.628 7f9f60b07700 1 --2- 172.21.2.102:0/578557578 >> v2:172.21.2.113:3300/0 conn(0x7f9f5c0d1870 0x7f9f5c1b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:25.645 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:25.628 7f9f60b07700 1 -- 172.21.2.102:0/578557578 >> 172.21.2.102:0/578557578 conn(0x7f9f5c0ce450 msgr2=0x7f9f5c0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:25.645 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:25.628 7f9f60b07700 1 -- 172.21.2.102:0/578557578 shutdown_connections 2021-02-10T11:32:25.645 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:25.628 7f9f60b07700 1 -- 172.21.2.102:0/578557578 wait complete. 2021-02-10T11:32:25.646 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:25.643 7f9f60b07700 1 Processor -- start 2021-02-10T11:32:25.646 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:25.643 7f9f60b07700 1 -- start start 2021-02-10T11:32:25.646 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:25.643 7f9f60b07700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f9f5c19af00 0x7f9f5c19b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:25.646 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:25.643 7f9f60b07700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9f5c19ba80 0x7f9f5c19be80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:25.646 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:25.643 7f9f60b07700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9f5c19c4a0 0x7f9f5c08a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:25.646 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:25.643 7f9f60b07700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f5c08fb50 con 0x7f9f5c19c4a0 2021-02-10T11:32:25.646 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:25.643 7f9f60b07700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f5c08afb0 con 0x7f9f5c19af00 2021-02-10T11:32:25.646 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:25.643 7f9f60b07700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f5c08b1a0 con 0x7f9f5c19ba80 2021-02-10T11:32:25.647 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:25.644 7f9f5b05f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9f5c19c4a0 0x7f9f5c08a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:25.647 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:25.644 7f9f5b05f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9f5c19c4a0 0x7f9f5c08a990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:34428/0 (socket says 172.21.2.102:34428) 2021-02-10T11:32:25.647 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:25.644 7f9f5b05f700 1 -- 172.21.2.102:0/2813630673 learned_addr learned my addr 172.21.2.102:0/2813630673 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:32:25.647 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:25.644 7f9f5b05f700 1 -- 172.21.2.102:0/2813630673 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9f5c19ba80 msgr2=0x7f9f5c19be80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:25.647 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:25.644 7f9f5b05f700 1 --2- 172.21.2.102:0/2813630673 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9f5c19ba80 0x7f9f5c19be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:25.647 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:25.644 7f9f5b05f700 1 -- 172.21.2.102:0/2813630673 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f9f5c19af00 msgr2=0x7f9f5c19b300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:25.647 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:25.644 7f9f5b05f700 1 --2- 172.21.2.102:0/2813630673 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f9f5c19af00 0x7f9f5c19b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:25.647 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:25.644 7f9f5b05f700 1 -- 172.21.2.102:0/2813630673 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f5c08b3c0 con 0x7f9f5c19c4a0 2021-02-10T11:32:25.647 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:25.644 7f9f5b05f700 1 --2- 172.21.2.102:0/2813630673 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9f5c19c4a0 0x7f9f5c08a990 secure :-1 s=READY pgs=2260 cs=0 l=1 rev1=1 rx=0x7f9f4c006ed0 tx=0x7f9f4c00b8e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:25.647 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:25.644 7f9f4b7fe700 1 -- 172.21.2.102:0/2813630673 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9f4c00bcd0 con 0x7f9f5c19c4a0 2021-02-10T11:32:25.647 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:25.644 7f9f4b7fe700 1 -- 172.21.2.102:0/2813630673 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f4c006020 con 0x7f9f5c19c4a0 2021-02-10T11:32:25.647 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:25.644 7f9f60b07700 1 -- 172.21.2.102:0/2813630673 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f5c08b3c0 con 0x7f9f5c19c4a0 2021-02-10T11:32:25.648 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:25.644 7f9f60b07700 1 -- 172.21.2.102:0/2813630673 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f5c08b950 con 0x7f9f5c19c4a0 2021-02-10T11:32:25.648 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:25.644 7f9f4b7fe700 1 -- 172.21.2.102:0/2813630673 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9f4c0054b0 con 0x7f9f5c19c4a0 2021-02-10T11:32:25.648 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:25.645 7f9f4b7fe700 1 -- 172.21.2.102:0/2813630673 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f9f4c006200 con 0x7f9f5c19c4a0 2021-02-10T11:32:25.648 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:25.645 7f9f4b7fe700 1 --2- 172.21.2.102:0/2813630673 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f9f40018730 0x7f9f4001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:25.648 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:25.645 7f9f4b7fe700 1 -- 172.21.2.102:0/2813630673 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f9f4c0327b0 con 0x7f9f5c19c4a0 2021-02-10T11:32:25.649 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:25.645 7f9f5a85e700 1 --2- 172.21.2.102:0/2813630673 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f9f40018730 0x7f9f4001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:25.649 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:25.646 7f9f5a85e700 1 --2- 172.21.2.102:0/2813630673 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f9f40018730 0x7f9f4001ac10 crc :-1 s=READY pgs=996 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:25.677 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:25.676 7f9f60b07700 1 -- 172.21.2.102:0/2813630673 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f5c0ee030 con 0x7f9f5c19c4a0 2021-02-10T11:32:25.679 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:25.679 7f9f4b7fe700 1 -- 172.21.2.102:0/2813630673 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f9f4c000940 con 0x7f9f5c19c4a0 2021-02-10T11:32:25.804 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:25.804 7f9f60b07700 1 -- 172.21.2.102:0/2813630673 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7f9f5c0ce450 con 0x7f9f5c19c4a0 2021-02-10T11:32:25.805 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:25.804 7f9f4b7fe700 1 -- 172.21.2.102:0/2813630673 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v112) v1 ==== 81+0+30374 (secure 0 0 0) 0x7f9f4c006020 con 0x7f9f5c19c4a0 2021-02-10T11:32:25.806 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:32:25.807 INFO:teuthology.orchestra.run.gibba002.stdout:{ 2021-02-10T11:32:25.807 INFO:teuthology.orchestra.run.gibba002.stdout: "epoch": 112, 2021-02-10T11:32:25.807 INFO:teuthology.orchestra.run.gibba002.stdout: "fsid": "68db30ea-dbce-4aef-a6df-f01c6fea0513", 2021-02-10T11:32:25.807 INFO:teuthology.orchestra.run.gibba002.stdout: "created": "2021-02-10 11:03:06.331557", 2021-02-10T11:32:25.807 INFO:teuthology.orchestra.run.gibba002.stdout: "modified": "2021-02-10 11:31:49.261779", 2021-02-10T11:32:25.808 INFO:teuthology.orchestra.run.gibba002.stdout: "last_up_change": "2021-02-10 11:03:32.690562", 2021-02-10T11:32:25.808 INFO:teuthology.orchestra.run.gibba002.stdout: "last_in_change": "2021-02-10 11:03:32.690562", 2021-02-10T11:32:25.808 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2021-02-10T11:32:25.808 INFO:teuthology.orchestra.run.gibba002.stdout: "flags_num": 5799936, 2021-02-10T11:32:25.808 INFO:teuthology.orchestra.run.gibba002.stdout: "flags_set": [ 2021-02-10T11:32:25.808 INFO:teuthology.orchestra.run.gibba002.stdout: "pglog_hardlimit", 2021-02-10T11:32:25.809 INFO:teuthology.orchestra.run.gibba002.stdout: "purged_snapdirs", 2021-02-10T11:32:25.809 INFO:teuthology.orchestra.run.gibba002.stdout: "recovery_deletes", 2021-02-10T11:32:25.809 INFO:teuthology.orchestra.run.gibba002.stdout: "sortbitwise" 2021-02-10T11:32:25.809 INFO:teuthology.orchestra.run.gibba002.stdout: ], 2021-02-10T11:32:25.809 INFO:teuthology.orchestra.run.gibba002.stdout: "crush_version": 4, 2021-02-10T11:32:25.809 INFO:teuthology.orchestra.run.gibba002.stdout: "full_ratio": 0.94999998807907104, 2021-02-10T11:32:25.810 INFO:teuthology.orchestra.run.gibba002.stdout: "backfillfull_ratio": 0.89999997615814209, 2021-02-10T11:32:25.810 INFO:teuthology.orchestra.run.gibba002.stdout: "nearfull_ratio": 0.85000002384185791, 2021-02-10T11:32:25.810 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_snapshot": "", 2021-02-10T11:32:25.810 INFO:teuthology.orchestra.run.gibba002.stdout: "pool_max": 23, 2021-02-10T11:32:25.810 INFO:teuthology.orchestra.run.gibba002.stdout: "max_osd": 8, 2021-02-10T11:32:25.811 INFO:teuthology.orchestra.run.gibba002.stdout: "require_min_compat_client": "jewel", 2021-02-10T11:32:25.811 INFO:teuthology.orchestra.run.gibba002.stdout: "min_compat_client": "jewel", 2021-02-10T11:32:25.811 INFO:teuthology.orchestra.run.gibba002.stdout: "require_osd_release": "nautilus", 2021-02-10T11:32:25.811 INFO:teuthology.orchestra.run.gibba002.stdout: "pools": [ 2021-02-10T11:32:25.811 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:32:25.811 INFO:teuthology.orchestra.run.gibba002.stdout: "pool": 1, 2021-02-10T11:32:25.812 INFO:teuthology.orchestra.run.gibba002.stdout: "pool_name": "rbd", 2021-02-10T11:32:25.812 INFO:teuthology.orchestra.run.gibba002.stdout: "create_time": "2021-02-10 11:03:37.322519", 2021-02-10T11:32:25.812 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 1, 2021-02-10T11:32:25.812 INFO:teuthology.orchestra.run.gibba002.stdout: "flags_names": "hashpspool", 2021-02-10T11:32:25.812 INFO:teuthology.orchestra.run.gibba002.stdout: "type": 1, 2021-02-10T11:32:25.813 INFO:teuthology.orchestra.run.gibba002.stdout: "size": 2, 2021-02-10T11:32:25.813 INFO:teuthology.orchestra.run.gibba002.stdout: "min_size": 1, 2021-02-10T11:32:25.813 INFO:teuthology.orchestra.run.gibba002.stdout: "crush_rule": 0, 2021-02-10T11:32:25.813 INFO:teuthology.orchestra.run.gibba002.stdout: "object_hash": 2, 2021-02-10T11:32:25.813 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:32:25.813 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num": 8, 2021-02-10T11:32:25.813 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_placement_num": 8, 2021-02-10T11:32:25.813 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_placement_num_target": 8, 2021-02-10T11:32:25.814 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num_target": 8, 2021-02-10T11:32:25.814 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num_pending": 8, 2021-02-10T11:32:25.814 INFO:teuthology.orchestra.run.gibba002.stdout: "last_pg_merge_meta": { 2021-02-10T11:32:25.814 INFO:teuthology.orchestra.run.gibba002.stdout: "source_pgid": "0.0", 2021-02-10T11:32:25.814 INFO:teuthology.orchestra.run.gibba002.stdout: "ready_epoch": 0, 2021-02-10T11:32:25.814 INFO:teuthology.orchestra.run.gibba002.stdout: "last_epoch_started": 0, 2021-02-10T11:32:25.814 INFO:teuthology.orchestra.run.gibba002.stdout: "last_epoch_clean": 0, 2021-02-10T11:32:25.814 INFO:teuthology.orchestra.run.gibba002.stdout: "source_version": "0'0", 2021-02-10T11:32:25.814 INFO:teuthology.orchestra.run.gibba002.stdout: "target_version": "0'0" 2021-02-10T11:32:25.814 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:32:25.814 INFO:teuthology.orchestra.run.gibba002.stdout: "last_change": "16", 2021-02-10T11:32:25.815 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend": "0", 2021-02-10T11:32:25.815 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:32:25.815 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:32:25.815 INFO:teuthology.orchestra.run.gibba002.stdout: "auid": 0, 2021-02-10T11:32:25.815 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:32:25.815 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_seq": 0, 2021-02-10T11:32:25.815 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_epoch": 0, 2021-02-10T11:32:25.816 INFO:teuthology.orchestra.run.gibba002.stdout: "pool_snaps": [], 2021-02-10T11:32:25.816 INFO:teuthology.orchestra.run.gibba002.stdout: "removed_snaps": "[]", 2021-02-10T11:32:25.816 INFO:teuthology.orchestra.run.gibba002.stdout: "quota_max_bytes": 0, 2021-02-10T11:32:25.816 INFO:teuthology.orchestra.run.gibba002.stdout: "quota_max_objects": 0, 2021-02-10T11:32:25.816 INFO:teuthology.orchestra.run.gibba002.stdout: "tiers": [], 2021-02-10T11:32:25.816 INFO:teuthology.orchestra.run.gibba002.stdout: "tier_of": -1, 2021-02-10T11:32:25.816 INFO:teuthology.orchestra.run.gibba002.stdout: "read_tier": -1, 2021-02-10T11:32:25.816 INFO:teuthology.orchestra.run.gibba002.stdout: "write_tier": -1, 2021-02-10T11:32:25.817 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_mode": "none", 2021-02-10T11:32:25.817 INFO:teuthology.orchestra.run.gibba002.stdout: "target_max_bytes": 0, 2021-02-10T11:32:25.817 INFO:teuthology.orchestra.run.gibba002.stdout: "target_max_objects": 0, 2021-02-10T11:32:25.817 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:32:25.817 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:32:25.817 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:32:25.817 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_min_flush_age": 0, 2021-02-10T11:32:25.817 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_min_evict_age": 0, 2021-02-10T11:32:25.818 INFO:teuthology.orchestra.run.gibba002.stdout: "erasure_code_profile": "", 2021-02-10T11:32:25.818 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_params": { 2021-02-10T11:32:25.818 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "none" 2021-02-10T11:32:25.818 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:32:25.818 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_period": 0, 2021-02-10T11:32:25.818 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_count": 0, 2021-02-10T11:32:25.818 INFO:teuthology.orchestra.run.gibba002.stdout: "use_gmt_hitset": true, 2021-02-10T11:32:25.818 INFO:teuthology.orchestra.run.gibba002.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:32:25.819 INFO:teuthology.orchestra.run.gibba002.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:32:25.819 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:32:25.819 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:32:25.819 INFO:teuthology.orchestra.run.gibba002.stdout: "grade_table": [], 2021-02-10T11:32:25.819 INFO:teuthology.orchestra.run.gibba002.stdout: "stripe_width": 0, 2021-02-10T11:32:25.819 INFO:teuthology.orchestra.run.gibba002.stdout: "expected_num_objects": 0, 2021-02-10T11:32:25.819 INFO:teuthology.orchestra.run.gibba002.stdout: "fast_read": false, 2021-02-10T11:32:25.819 INFO:teuthology.orchestra.run.gibba002.stdout: "options": {}, 2021-02-10T11:32:25.820 INFO:teuthology.orchestra.run.gibba002.stdout: "application_metadata": { 2021-02-10T11:32:25.820 INFO:teuthology.orchestra.run.gibba002.stdout: "rbd": {} 2021-02-10T11:32:25.820 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:32:25.820 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:32:25.820 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:32:25.820 INFO:teuthology.orchestra.run.gibba002.stdout: "pool": 22, 2021-02-10T11:32:25.820 INFO:teuthology.orchestra.run.gibba002.stdout: "pool_name": "cephfs_metadata", 2021-02-10T11:32:25.820 INFO:teuthology.orchestra.run.gibba002.stdout: "create_time": "2021-02-10 11:31:46.228075", 2021-02-10T11:32:25.821 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 1, 2021-02-10T11:32:25.821 INFO:teuthology.orchestra.run.gibba002.stdout: "flags_names": "hashpspool", 2021-02-10T11:32:25.821 INFO:teuthology.orchestra.run.gibba002.stdout: "type": 1, 2021-02-10T11:32:25.821 INFO:teuthology.orchestra.run.gibba002.stdout: "size": 2, 2021-02-10T11:32:25.821 INFO:teuthology.orchestra.run.gibba002.stdout: "min_size": 1, 2021-02-10T11:32:25.821 INFO:teuthology.orchestra.run.gibba002.stdout: "crush_rule": 0, 2021-02-10T11:32:25.822 INFO:teuthology.orchestra.run.gibba002.stdout: "object_hash": 2, 2021-02-10T11:32:25.822 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:32:25.822 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num": 8, 2021-02-10T11:32:25.822 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_placement_num": 8, 2021-02-10T11:32:25.822 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_placement_num_target": 8, 2021-02-10T11:32:25.822 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num_target": 8, 2021-02-10T11:32:25.823 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num_pending": 8, 2021-02-10T11:32:25.823 INFO:teuthology.orchestra.run.gibba002.stdout: "last_pg_merge_meta": { 2021-02-10T11:32:25.823 INFO:teuthology.orchestra.run.gibba002.stdout: "source_pgid": "0.0", 2021-02-10T11:32:25.823 INFO:teuthology.orchestra.run.gibba002.stdout: "ready_epoch": 0, 2021-02-10T11:32:25.823 INFO:teuthology.orchestra.run.gibba002.stdout: "last_epoch_started": 0, 2021-02-10T11:32:25.823 INFO:teuthology.orchestra.run.gibba002.stdout: "last_epoch_clean": 0, 2021-02-10T11:32:25.823 INFO:teuthology.orchestra.run.gibba002.stdout: "source_version": "0'0", 2021-02-10T11:32:25.824 INFO:teuthology.orchestra.run.gibba002.stdout: "target_version": "0'0" 2021-02-10T11:32:25.824 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:32:25.824 INFO:teuthology.orchestra.run.gibba002.stdout: "last_change": "111", 2021-02-10T11:32:25.824 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend": "0", 2021-02-10T11:32:25.824 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:32:25.824 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:32:25.824 INFO:teuthology.orchestra.run.gibba002.stdout: "auid": 0, 2021-02-10T11:32:25.824 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:32:25.824 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_seq": 0, 2021-02-10T11:32:25.824 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_epoch": 0, 2021-02-10T11:32:25.824 INFO:teuthology.orchestra.run.gibba002.stdout: "pool_snaps": [], 2021-02-10T11:32:25.824 INFO:teuthology.orchestra.run.gibba002.stdout: "removed_snaps": "[]", 2021-02-10T11:32:25.824 INFO:teuthology.orchestra.run.gibba002.stdout: "quota_max_bytes": 0, 2021-02-10T11:32:25.824 INFO:teuthology.orchestra.run.gibba002.stdout: "quota_max_objects": 0, 2021-02-10T11:32:25.825 INFO:teuthology.orchestra.run.gibba002.stdout: "tiers": [], 2021-02-10T11:32:25.825 INFO:teuthology.orchestra.run.gibba002.stdout: "tier_of": -1, 2021-02-10T11:32:25.825 INFO:teuthology.orchestra.run.gibba002.stdout: "read_tier": -1, 2021-02-10T11:32:25.825 INFO:teuthology.orchestra.run.gibba002.stdout: "write_tier": -1, 2021-02-10T11:32:25.825 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_mode": "none", 2021-02-10T11:32:25.825 INFO:teuthology.orchestra.run.gibba002.stdout: "target_max_bytes": 0, 2021-02-10T11:32:25.825 INFO:teuthology.orchestra.run.gibba002.stdout: "target_max_objects": 0, 2021-02-10T11:32:25.825 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:32:25.825 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:32:25.825 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:32:25.826 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_min_flush_age": 0, 2021-02-10T11:32:25.826 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_min_evict_age": 0, 2021-02-10T11:32:25.826 INFO:teuthology.orchestra.run.gibba002.stdout: "erasure_code_profile": "", 2021-02-10T11:32:25.826 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_params": { 2021-02-10T11:32:25.826 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "none" 2021-02-10T11:32:25.826 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:32:25.826 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_period": 0, 2021-02-10T11:32:25.826 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_count": 0, 2021-02-10T11:32:25.826 INFO:teuthology.orchestra.run.gibba002.stdout: "use_gmt_hitset": true, 2021-02-10T11:32:25.826 INFO:teuthology.orchestra.run.gibba002.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:32:25.826 INFO:teuthology.orchestra.run.gibba002.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:32:25.826 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:32:25.827 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:32:25.827 INFO:teuthology.orchestra.run.gibba002.stdout: "grade_table": [], 2021-02-10T11:32:25.827 INFO:teuthology.orchestra.run.gibba002.stdout: "stripe_width": 0, 2021-02-10T11:32:25.827 INFO:teuthology.orchestra.run.gibba002.stdout: "expected_num_objects": 0, 2021-02-10T11:32:25.827 INFO:teuthology.orchestra.run.gibba002.stdout: "fast_read": false, 2021-02-10T11:32:25.827 INFO:teuthology.orchestra.run.gibba002.stdout: "options": { 2021-02-10T11:32:25.827 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_autoscale_bias": 4, 2021-02-10T11:32:25.827 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num_min": 16, 2021-02-10T11:32:25.828 INFO:teuthology.orchestra.run.gibba002.stdout: "recovery_priority": 5 2021-02-10T11:32:25.828 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:32:25.828 INFO:teuthology.orchestra.run.gibba002.stdout: "application_metadata": { 2021-02-10T11:32:25.828 INFO:teuthology.orchestra.run.gibba002.stdout: "cephfs": { 2021-02-10T11:32:25.828 INFO:teuthology.orchestra.run.gibba002.stdout: "metadata": "cephfs" 2021-02-10T11:32:25.828 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:32:25.828 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:32:25.828 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:32:25.828 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:32:25.828 INFO:teuthology.orchestra.run.gibba002.stdout: "pool": 23, 2021-02-10T11:32:25.829 INFO:teuthology.orchestra.run.gibba002.stdout: "pool_name": "cephfs_data", 2021-02-10T11:32:25.829 INFO:teuthology.orchestra.run.gibba002.stdout: "create_time": "2021-02-10 11:31:47.252676", 2021-02-10T11:32:25.829 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 1, 2021-02-10T11:32:25.829 INFO:teuthology.orchestra.run.gibba002.stdout: "flags_names": "hashpspool", 2021-02-10T11:32:25.829 INFO:teuthology.orchestra.run.gibba002.stdout: "type": 1, 2021-02-10T11:32:25.829 INFO:teuthology.orchestra.run.gibba002.stdout: "size": 2, 2021-02-10T11:32:25.829 INFO:teuthology.orchestra.run.gibba002.stdout: "min_size": 1, 2021-02-10T11:32:25.829 INFO:teuthology.orchestra.run.gibba002.stdout: "crush_rule": 0, 2021-02-10T11:32:25.829 INFO:teuthology.orchestra.run.gibba002.stdout: "object_hash": 2, 2021-02-10T11:32:25.829 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:32:25.830 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num": 8, 2021-02-10T11:32:25.830 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_placement_num": 8, 2021-02-10T11:32:25.830 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_placement_num_target": 8, 2021-02-10T11:32:25.830 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num_target": 8, 2021-02-10T11:32:25.830 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num_pending": 8, 2021-02-10T11:32:25.830 INFO:teuthology.orchestra.run.gibba002.stdout: "last_pg_merge_meta": { 2021-02-10T11:32:25.830 INFO:teuthology.orchestra.run.gibba002.stdout: "source_pgid": "0.0", 2021-02-10T11:32:25.830 INFO:teuthology.orchestra.run.gibba002.stdout: "ready_epoch": 0, 2021-02-10T11:32:25.831 INFO:teuthology.orchestra.run.gibba002.stdout: "last_epoch_started": 0, 2021-02-10T11:32:25.831 INFO:teuthology.orchestra.run.gibba002.stdout: "last_epoch_clean": 0, 2021-02-10T11:32:25.831 INFO:teuthology.orchestra.run.gibba002.stdout: "source_version": "0'0", 2021-02-10T11:32:25.831 INFO:teuthology.orchestra.run.gibba002.stdout: "target_version": "0'0" 2021-02-10T11:32:25.831 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:32:25.831 INFO:teuthology.orchestra.run.gibba002.stdout: "last_change": "112", 2021-02-10T11:32:25.831 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend": "0", 2021-02-10T11:32:25.831 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:32:25.831 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:32:25.831 INFO:teuthology.orchestra.run.gibba002.stdout: "auid": 0, 2021-02-10T11:32:25.831 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:32:25.831 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_seq": 0, 2021-02-10T11:32:25.831 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_epoch": 0, 2021-02-10T11:32:25.831 INFO:teuthology.orchestra.run.gibba002.stdout: "pool_snaps": [], 2021-02-10T11:32:25.832 INFO:teuthology.orchestra.run.gibba002.stdout: "removed_snaps": "[]", 2021-02-10T11:32:25.832 INFO:teuthology.orchestra.run.gibba002.stdout: "quota_max_bytes": 0, 2021-02-10T11:32:25.832 INFO:teuthology.orchestra.run.gibba002.stdout: "quota_max_objects": 0, 2021-02-10T11:32:25.832 INFO:teuthology.orchestra.run.gibba002.stdout: "tiers": [], 2021-02-10T11:32:25.832 INFO:teuthology.orchestra.run.gibba002.stdout: "tier_of": -1, 2021-02-10T11:32:25.832 INFO:teuthology.orchestra.run.gibba002.stdout: "read_tier": -1, 2021-02-10T11:32:25.832 INFO:teuthology.orchestra.run.gibba002.stdout: "write_tier": -1, 2021-02-10T11:32:25.832 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_mode": "none", 2021-02-10T11:32:25.832 INFO:teuthology.orchestra.run.gibba002.stdout: "target_max_bytes": 0, 2021-02-10T11:32:25.832 INFO:teuthology.orchestra.run.gibba002.stdout: "target_max_objects": 0, 2021-02-10T11:32:25.832 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:32:25.832 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:32:25.832 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:32:25.832 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_min_flush_age": 0, 2021-02-10T11:32:25.833 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_min_evict_age": 0, 2021-02-10T11:32:25.833 INFO:teuthology.orchestra.run.gibba002.stdout: "erasure_code_profile": "", 2021-02-10T11:32:25.833 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_params": { 2021-02-10T11:32:25.833 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "none" 2021-02-10T11:32:25.833 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:32:25.833 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_period": 0, 2021-02-10T11:32:25.833 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_count": 0, 2021-02-10T11:32:25.833 INFO:teuthology.orchestra.run.gibba002.stdout: "use_gmt_hitset": true, 2021-02-10T11:32:25.833 INFO:teuthology.orchestra.run.gibba002.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:32:25.833 INFO:teuthology.orchestra.run.gibba002.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:32:25.833 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:32:25.833 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:32:25.833 INFO:teuthology.orchestra.run.gibba002.stdout: "grade_table": [], 2021-02-10T11:32:25.833 INFO:teuthology.orchestra.run.gibba002.stdout: "stripe_width": 0, 2021-02-10T11:32:25.834 INFO:teuthology.orchestra.run.gibba002.stdout: "expected_num_objects": 0, 2021-02-10T11:32:25.834 INFO:teuthology.orchestra.run.gibba002.stdout: "fast_read": false, 2021-02-10T11:32:25.834 INFO:teuthology.orchestra.run.gibba002.stdout: "options": {}, 2021-02-10T11:32:25.834 INFO:teuthology.orchestra.run.gibba002.stdout: "application_metadata": { 2021-02-10T11:32:25.834 INFO:teuthology.orchestra.run.gibba002.stdout: "cephfs": { 2021-02-10T11:32:25.834 INFO:teuthology.orchestra.run.gibba002.stdout: "data": "cephfs" 2021-02-10T11:32:25.834 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:32:25.834 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:32:25.834 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:32:25.834 INFO:teuthology.orchestra.run.gibba002.stdout: ], 2021-02-10T11:32:25.834 INFO:teuthology.orchestra.run.gibba002.stdout: "osds": [ 2021-02-10T11:32:25.834 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:32:25.835 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 0, 2021-02-10T11:32:25.835 INFO:teuthology.orchestra.run.gibba002.stdout: "uuid": "be691b4d-a039-408a-af80-9d53f02b6f1f", 2021-02-10T11:32:25.835 INFO:teuthology.orchestra.run.gibba002.stdout: "up": 1, 2021-02-10T11:32:25.835 INFO:teuthology.orchestra.run.gibba002.stdout: "in": 1, 2021-02-10T11:32:25.835 INFO:teuthology.orchestra.run.gibba002.stdout: "weight": 1, 2021-02-10T11:32:25.835 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_affinity": 1, 2021-02-10T11:32:25.835 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_begin": 0, 2021-02-10T11:32:25.835 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_end": 0, 2021-02-10T11:32:25.836 INFO:teuthology.orchestra.run.gibba002.stdout: "up_from": 13, 2021-02-10T11:32:25.836 INFO:teuthology.orchestra.run.gibba002.stdout: "up_thru": 110, 2021-02-10T11:32:25.836 INFO:teuthology.orchestra.run.gibba002.stdout: "down_at": 0, 2021-02-10T11:32:25.836 INFO:teuthology.orchestra.run.gibba002.stdout: "lost_at": 0, 2021-02-10T11:32:25.836 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addrs": { 2021-02-10T11:32:25.836 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:32:25.836 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:32:25.836 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:32:25.836 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6818", 2021-02-10T11:32:25.836 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36088 2021-02-10T11:32:25.836 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:32:25.837 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:32:25.837 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:32:25.837 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6820", 2021-02-10T11:32:25.837 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36088 2021-02-10T11:32:25.837 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:32:25.837 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:32:25.837 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:32:25.837 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addrs": { 2021-02-10T11:32:25.837 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:32:25.837 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:32:25.837 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:32:25.837 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6822", 2021-02-10T11:32:25.837 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36088 2021-02-10T11:32:25.837 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:32:25.838 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:32:25.838 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:32:25.838 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6824", 2021-02-10T11:32:25.838 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36088 2021-02-10T11:32:25.838 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:32:25.838 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:32:25.838 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:32:25.838 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addrs": { 2021-02-10T11:32:25.838 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:32:25.838 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:32:25.838 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:32:25.838 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6830", 2021-02-10T11:32:25.838 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36088 2021-02-10T11:32:25.839 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:32:25.839 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:32:25.839 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:32:25.839 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6832", 2021-02-10T11:32:25.839 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36088 2021-02-10T11:32:25.839 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:32:25.839 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:32:25.839 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:32:25.839 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addrs": { 2021-02-10T11:32:25.839 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:32:25.839 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:32:25.839 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:32:25.839 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6826", 2021-02-10T11:32:25.839 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36088 2021-02-10T11:32:25.840 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:32:25.840 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:32:25.840 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:32:25.840 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6828", 2021-02-10T11:32:25.840 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36088 2021-02-10T11:32:25.840 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:32:25.840 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:32:25.840 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:32:25.840 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addr": "172.21.2.102:6820/36088", 2021-02-10T11:32:25.840 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addr": "172.21.2.102:6824/36088", 2021-02-10T11:32:25.840 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addr": "172.21.2.102:6832/36088", 2021-02-10T11:32:25.840 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addr": "172.21.2.102:6828/36088", 2021-02-10T11:32:25.840 INFO:teuthology.orchestra.run.gibba002.stdout: "state": [ 2021-02-10T11:32:25.840 INFO:teuthology.orchestra.run.gibba002.stdout: "exists", 2021-02-10T11:32:25.841 INFO:teuthology.orchestra.run.gibba002.stdout: "up" 2021-02-10T11:32:25.841 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:32:25.841 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:32:25.841 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:32:25.841 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 1, 2021-02-10T11:32:25.841 INFO:teuthology.orchestra.run.gibba002.stdout: "uuid": "64e3ccc8-9c69-4d7c-865e-af8e83822635", 2021-02-10T11:32:25.841 INFO:teuthology.orchestra.run.gibba002.stdout: "up": 1, 2021-02-10T11:32:25.841 INFO:teuthology.orchestra.run.gibba002.stdout: "in": 1, 2021-02-10T11:32:25.841 INFO:teuthology.orchestra.run.gibba002.stdout: "weight": 1, 2021-02-10T11:32:25.841 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_affinity": 1, 2021-02-10T11:32:25.841 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_begin": 0, 2021-02-10T11:32:25.841 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_end": 0, 2021-02-10T11:32:25.841 INFO:teuthology.orchestra.run.gibba002.stdout: "up_from": 13, 2021-02-10T11:32:25.841 INFO:teuthology.orchestra.run.gibba002.stdout: "up_thru": 109, 2021-02-10T11:32:25.842 INFO:teuthology.orchestra.run.gibba002.stdout: "down_at": 0, 2021-02-10T11:32:25.842 INFO:teuthology.orchestra.run.gibba002.stdout: "lost_at": 0, 2021-02-10T11:32:25.842 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addrs": { 2021-02-10T11:32:25.842 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:32:25.842 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:32:25.842 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:32:25.842 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6819", 2021-02-10T11:32:25.842 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36087 2021-02-10T11:32:25.842 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:32:25.842 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:32:25.842 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:32:25.842 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6821", 2021-02-10T11:32:25.842 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36087 2021-02-10T11:32:25.843 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:32:25.843 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:32:25.843 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:32:25.843 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addrs": { 2021-02-10T11:32:25.843 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:32:25.843 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:32:25.843 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:32:25.843 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6823", 2021-02-10T11:32:25.843 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36087 2021-02-10T11:32:25.843 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:32:25.843 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:32:25.843 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:32:25.843 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6825", 2021-02-10T11:32:25.843 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36087 2021-02-10T11:32:25.844 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:32:25.844 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:32:25.844 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:32:25.844 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addrs": { 2021-02-10T11:32:25.844 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:32:25.844 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:32:25.844 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:32:25.844 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6831", 2021-02-10T11:32:25.844 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36087 2021-02-10T11:32:25.844 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:32:25.844 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:32:25.844 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:32:25.844 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6833", 2021-02-10T11:32:25.844 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36087 2021-02-10T11:32:25.845 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:32:25.845 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:32:25.845 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:32:25.845 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addrs": { 2021-02-10T11:32:25.845 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:32:25.845 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:32:25.845 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:32:25.845 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6827", 2021-02-10T11:32:25.845 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36087 2021-02-10T11:32:25.845 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:32:25.845 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:32:25.845 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:32:25.845 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6829", 2021-02-10T11:32:25.845 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36087 2021-02-10T11:32:25.846 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:32:25.846 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:32:25.846 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:32:25.846 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addr": "172.21.2.102:6821/36087", 2021-02-10T11:32:25.846 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addr": "172.21.2.102:6825/36087", 2021-02-10T11:32:25.846 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addr": "172.21.2.102:6833/36087", 2021-02-10T11:32:25.846 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addr": "172.21.2.102:6829/36087", 2021-02-10T11:32:25.846 INFO:teuthology.orchestra.run.gibba002.stdout: "state": [ 2021-02-10T11:32:25.846 INFO:teuthology.orchestra.run.gibba002.stdout: "exists", 2021-02-10T11:32:25.847 INFO:teuthology.orchestra.run.gibba002.stdout: "up" 2021-02-10T11:32:25.847 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:32:25.847 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:32:25.847 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:32:25.847 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 2, 2021-02-10T11:32:25.847 INFO:teuthology.orchestra.run.gibba002.stdout: "uuid": "153700af-747f-481f-bdf4-d6072d3a842f", 2021-02-10T11:32:25.847 INFO:teuthology.orchestra.run.gibba002.stdout: "up": 1, 2021-02-10T11:32:25.847 INFO:teuthology.orchestra.run.gibba002.stdout: "in": 1, 2021-02-10T11:32:25.847 INFO:teuthology.orchestra.run.gibba002.stdout: "weight": 1, 2021-02-10T11:32:25.847 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_affinity": 1, 2021-02-10T11:32:25.847 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_begin": 0, 2021-02-10T11:32:25.847 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_end": 0, 2021-02-10T11:32:25.847 INFO:teuthology.orchestra.run.gibba002.stdout: "up_from": 13, 2021-02-10T11:32:25.847 INFO:teuthology.orchestra.run.gibba002.stdout: "up_thru": 110, 2021-02-10T11:32:25.848 INFO:teuthology.orchestra.run.gibba002.stdout: "down_at": 0, 2021-02-10T11:32:25.848 INFO:teuthology.orchestra.run.gibba002.stdout: "lost_at": 0, 2021-02-10T11:32:25.848 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addrs": { 2021-02-10T11:32:25.848 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:32:25.848 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:32:25.848 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:32:25.848 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6810", 2021-02-10T11:32:25.848 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36089 2021-02-10T11:32:25.848 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:32:25.848 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:32:25.848 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:32:25.848 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6811", 2021-02-10T11:32:25.849 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36089 2021-02-10T11:32:25.849 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:32:25.849 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:32:25.849 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:32:25.849 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addrs": { 2021-02-10T11:32:25.849 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:32:25.849 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:32:25.849 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:32:25.849 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6812", 2021-02-10T11:32:25.849 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36089 2021-02-10T11:32:25.849 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:32:25.849 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:32:25.850 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:32:25.850 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6813", 2021-02-10T11:32:25.850 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36089 2021-02-10T11:32:25.850 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:32:25.850 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:32:25.850 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:32:25.850 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addrs": { 2021-02-10T11:32:25.850 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:32:25.850 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:32:25.850 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:32:25.850 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6816", 2021-02-10T11:32:25.850 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36089 2021-02-10T11:32:25.851 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:32:25.851 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:32:25.851 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:32:25.851 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6817", 2021-02-10T11:32:25.851 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36089 2021-02-10T11:32:25.851 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:32:25.851 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:32:25.851 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:32:25.851 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addrs": { 2021-02-10T11:32:25.851 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:32:25.851 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:32:25.851 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:32:25.851 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6814", 2021-02-10T11:32:25.852 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36089 2021-02-10T11:32:25.852 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:32:25.852 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:32:25.852 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:32:25.852 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6815", 2021-02-10T11:32:25.852 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36089 2021-02-10T11:32:25.852 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:32:25.852 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:32:25.852 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:32:25.853 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addr": "172.21.2.102:6811/36089", 2021-02-10T11:32:25.853 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addr": "172.21.2.102:6813/36089", 2021-02-10T11:32:25.853 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addr": "172.21.2.102:6817/36089", 2021-02-10T11:32:25.853 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addr": "172.21.2.102:6815/36089", 2021-02-10T11:32:25.853 INFO:teuthology.orchestra.run.gibba002.stdout: "state": [ 2021-02-10T11:32:25.853 INFO:teuthology.orchestra.run.gibba002.stdout: "exists", 2021-02-10T11:32:25.853 INFO:teuthology.orchestra.run.gibba002.stdout: "up" 2021-02-10T11:32:25.853 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:32:25.853 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:32:25.853 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:32:25.853 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 3, 2021-02-10T11:32:25.853 INFO:teuthology.orchestra.run.gibba002.stdout: "uuid": "cbc678f4-83a9-45e6-8830-2f7c232b1ea7", 2021-02-10T11:32:25.854 INFO:teuthology.orchestra.run.gibba002.stdout: "up": 1, 2021-02-10T11:32:25.854 INFO:teuthology.orchestra.run.gibba002.stdout: "in": 1, 2021-02-10T11:32:25.854 INFO:teuthology.orchestra.run.gibba002.stdout: "weight": 1, 2021-02-10T11:32:25.854 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_affinity": 1, 2021-02-10T11:32:25.854 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_begin": 0, 2021-02-10T11:32:25.854 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_end": 0, 2021-02-10T11:32:25.855 INFO:teuthology.orchestra.run.gibba002.stdout: "up_from": 13, 2021-02-10T11:32:25.855 INFO:teuthology.orchestra.run.gibba002.stdout: "up_thru": 110, 2021-02-10T11:32:25.855 INFO:teuthology.orchestra.run.gibba002.stdout: "down_at": 0, 2021-02-10T11:32:25.855 INFO:teuthology.orchestra.run.gibba002.stdout: "lost_at": 0, 2021-02-10T11:32:25.855 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addrs": { 2021-02-10T11:32:25.856 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:32:25.856 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:32:25.856 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:32:25.856 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6802", 2021-02-10T11:32:25.856 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36090 2021-02-10T11:32:25.856 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:32:25.857 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:32:25.857 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:32:25.857 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6803", 2021-02-10T11:32:25.857 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36090 2021-02-10T11:32:25.859 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:32:25.860 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:32:25.860 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:32:25.860 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addrs": { 2021-02-10T11:32:25.860 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:32:25.861 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:32:25.861 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:32:25.861 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6804", 2021-02-10T11:32:25.861 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36090 2021-02-10T11:32:25.861 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:32:25.861 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:32:25.862 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:32:25.862 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6805", 2021-02-10T11:32:25.862 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36090 2021-02-10T11:32:25.862 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:32:25.862 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:32:25.862 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:32:25.863 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addrs": { 2021-02-10T11:32:25.863 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:32:25.863 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:32:25.863 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:32:25.863 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6808", 2021-02-10T11:32:25.863 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36090 2021-02-10T11:32:25.864 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:32:25.864 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:32:25.864 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:32:25.864 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6809", 2021-02-10T11:32:25.864 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36090 2021-02-10T11:32:25.864 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:32:25.864 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:32:25.865 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:32:25.865 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addrs": { 2021-02-10T11:32:25.865 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:32:25.865 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:32:25.865 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:32:25.865 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6806", 2021-02-10T11:32:25.866 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36090 2021-02-10T11:32:25.866 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:32:25.866 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:32:25.866 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:32:25.866 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6807", 2021-02-10T11:32:25.866 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36090 2021-02-10T11:32:25.867 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:32:25.867 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:32:25.867 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:32:25.867 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addr": "172.21.2.102:6803/36090", 2021-02-10T11:32:25.867 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addr": "172.21.2.102:6805/36090", 2021-02-10T11:32:25.867 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addr": "172.21.2.102:6809/36090", 2021-02-10T11:32:25.868 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addr": "172.21.2.102:6807/36090", 2021-02-10T11:32:25.868 INFO:teuthology.orchestra.run.gibba002.stdout: "state": [ 2021-02-10T11:32:25.868 INFO:teuthology.orchestra.run.gibba002.stdout: "exists", 2021-02-10T11:32:25.868 INFO:teuthology.orchestra.run.gibba002.stdout: "up" 2021-02-10T11:32:25.868 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:32:25.868 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:32:25.869 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:32:25.869 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 4, 2021-02-10T11:32:25.869 INFO:teuthology.orchestra.run.gibba002.stdout: "uuid": "c01e4008-b258-4f10-afd2-9e7f37214351", 2021-02-10T11:32:25.869 INFO:teuthology.orchestra.run.gibba002.stdout: "up": 1, 2021-02-10T11:32:25.869 INFO:teuthology.orchestra.run.gibba002.stdout: "in": 1, 2021-02-10T11:32:25.869 INFO:teuthology.orchestra.run.gibba002.stdout: "weight": 1, 2021-02-10T11:32:25.870 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_affinity": 1, 2021-02-10T11:32:25.870 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_begin": 0, 2021-02-10T11:32:25.870 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_end": 0, 2021-02-10T11:32:25.870 INFO:teuthology.orchestra.run.gibba002.stdout: "up_from": 13, 2021-02-10T11:32:25.870 INFO:teuthology.orchestra.run.gibba002.stdout: "up_thru": 110, 2021-02-10T11:32:25.870 INFO:teuthology.orchestra.run.gibba002.stdout: "down_at": 0, 2021-02-10T11:32:25.871 INFO:teuthology.orchestra.run.gibba002.stdout: "lost_at": 0, 2021-02-10T11:32:25.871 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addrs": { 2021-02-10T11:32:25.871 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:32:25.871 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:32:25.871 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:32:25.871 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6808", 2021-02-10T11:32:25.872 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36368 2021-02-10T11:32:25.872 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:32:25.872 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:32:25.872 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:32:25.872 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6811", 2021-02-10T11:32:25.872 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36368 2021-02-10T11:32:25.872 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:32:25.873 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:32:25.873 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:32:25.873 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addrs": { 2021-02-10T11:32:25.873 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:32:25.873 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:32:25.873 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:32:25.874 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6814", 2021-02-10T11:32:25.874 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36368 2021-02-10T11:32:25.874 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:32:25.874 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:32:25.874 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:32:25.874 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6818", 2021-02-10T11:32:25.875 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36368 2021-02-10T11:32:25.875 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:32:25.875 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:32:25.875 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:32:25.875 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addrs": { 2021-02-10T11:32:25.875 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:32:25.876 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:32:25.876 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:32:25.876 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6826", 2021-02-10T11:32:25.876 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36368 2021-02-10T11:32:25.876 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:32:25.876 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:32:25.877 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:32:25.877 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6828", 2021-02-10T11:32:25.877 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36368 2021-02-10T11:32:25.877 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:32:25.877 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:32:25.877 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:32:25.877 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addrs": { 2021-02-10T11:32:25.878 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:32:25.878 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:32:25.878 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:32:25.878 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6820", 2021-02-10T11:32:25.878 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36368 2021-02-10T11:32:25.878 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:32:25.879 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:32:25.879 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:32:25.879 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6824", 2021-02-10T11:32:25.879 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36368 2021-02-10T11:32:25.879 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:32:25.879 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:32:25.880 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:32:25.880 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addr": "172.21.2.113:6811/36368", 2021-02-10T11:32:25.880 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addr": "172.21.2.113:6818/36368", 2021-02-10T11:32:25.880 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addr": "172.21.2.113:6828/36368", 2021-02-10T11:32:25.880 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addr": "172.21.2.113:6824/36368", 2021-02-10T11:32:25.880 INFO:teuthology.orchestra.run.gibba002.stdout: "state": [ 2021-02-10T11:32:25.880 INFO:teuthology.orchestra.run.gibba002.stdout: "exists", 2021-02-10T11:32:25.880 INFO:teuthology.orchestra.run.gibba002.stdout: "up" 2021-02-10T11:32:25.880 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:32:25.880 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:32:25.880 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:32:25.880 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 5, 2021-02-10T11:32:25.881 INFO:teuthology.orchestra.run.gibba002.stdout: "uuid": "551f6007-daf8-4a98-8ee0-c2b16bcccb5e", 2021-02-10T11:32:25.881 INFO:teuthology.orchestra.run.gibba002.stdout: "up": 1, 2021-02-10T11:32:25.881 INFO:teuthology.orchestra.run.gibba002.stdout: "in": 1, 2021-02-10T11:32:25.881 INFO:teuthology.orchestra.run.gibba002.stdout: "weight": 1, 2021-02-10T11:32:25.881 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_affinity": 1, 2021-02-10T11:32:25.881 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_begin": 0, 2021-02-10T11:32:25.881 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_end": 0, 2021-02-10T11:32:25.881 INFO:teuthology.orchestra.run.gibba002.stdout: "up_from": 13, 2021-02-10T11:32:25.881 INFO:teuthology.orchestra.run.gibba002.stdout: "up_thru": 110, 2021-02-10T11:32:25.881 INFO:teuthology.orchestra.run.gibba002.stdout: "down_at": 0, 2021-02-10T11:32:25.881 INFO:teuthology.orchestra.run.gibba002.stdout: "lost_at": 0, 2021-02-10T11:32:25.881 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addrs": { 2021-02-10T11:32:25.881 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:32:25.881 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:32:25.882 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:32:25.882 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6802", 2021-02-10T11:32:25.882 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36366 2021-02-10T11:32:25.882 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:32:25.882 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:32:25.882 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:32:25.882 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6805", 2021-02-10T11:32:25.882 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36366 2021-02-10T11:32:25.882 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:32:25.882 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:32:25.882 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:32:25.882 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addrs": { 2021-02-10T11:32:25.882 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:32:25.883 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:32:25.883 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:32:25.883 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6807", 2021-02-10T11:32:25.883 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36366 2021-02-10T11:32:25.883 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:32:25.883 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:32:25.883 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:32:25.883 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6810", 2021-02-10T11:32:25.883 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36366 2021-02-10T11:32:25.883 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:32:25.883 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:32:25.883 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:32:25.883 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addrs": { 2021-02-10T11:32:25.883 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:32:25.884 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:32:25.884 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:32:25.884 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6819", 2021-02-10T11:32:25.884 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36366 2021-02-10T11:32:25.884 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:32:25.884 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:32:25.884 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:32:25.884 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6822", 2021-02-10T11:32:25.884 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36366 2021-02-10T11:32:25.884 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:32:25.884 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:32:25.884 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:32:25.884 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addrs": { 2021-02-10T11:32:25.884 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:32:25.885 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:32:25.885 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:32:25.885 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6812", 2021-02-10T11:32:25.885 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36366 2021-02-10T11:32:25.885 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:32:25.885 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:32:25.885 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:32:25.885 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6815", 2021-02-10T11:32:25.885 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36366 2021-02-10T11:32:25.885 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:32:25.885 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:32:25.885 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:32:25.885 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addr": "172.21.2.113:6805/36366", 2021-02-10T11:32:25.885 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addr": "172.21.2.113:6810/36366", 2021-02-10T11:32:25.886 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addr": "172.21.2.113:6822/36366", 2021-02-10T11:32:25.886 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addr": "172.21.2.113:6815/36366", 2021-02-10T11:32:25.886 INFO:teuthology.orchestra.run.gibba002.stdout: "state": [ 2021-02-10T11:32:25.886 INFO:teuthology.orchestra.run.gibba002.stdout: "exists", 2021-02-10T11:32:25.886 INFO:teuthology.orchestra.run.gibba002.stdout: "up" 2021-02-10T11:32:25.886 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:32:25.886 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:32:25.886 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:32:25.886 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 6, 2021-02-10T11:32:25.886 INFO:teuthology.orchestra.run.gibba002.stdout: "uuid": "bf67c040-0e50-48be-b102-c3943bc9ebef", 2021-02-10T11:32:25.886 INFO:teuthology.orchestra.run.gibba002.stdout: "up": 1, 2021-02-10T11:32:25.886 INFO:teuthology.orchestra.run.gibba002.stdout: "in": 1, 2021-02-10T11:32:25.886 INFO:teuthology.orchestra.run.gibba002.stdout: "weight": 1, 2021-02-10T11:32:25.887 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_affinity": 1, 2021-02-10T11:32:25.887 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_begin": 0, 2021-02-10T11:32:25.887 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_end": 0, 2021-02-10T11:32:25.887 INFO:teuthology.orchestra.run.gibba002.stdout: "up_from": 13, 2021-02-10T11:32:25.887 INFO:teuthology.orchestra.run.gibba002.stdout: "up_thru": 108, 2021-02-10T11:32:25.887 INFO:teuthology.orchestra.run.gibba002.stdout: "down_at": 0, 2021-02-10T11:32:25.887 INFO:teuthology.orchestra.run.gibba002.stdout: "lost_at": 0, 2021-02-10T11:32:25.887 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addrs": { 2021-02-10T11:32:25.887 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:32:25.887 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:32:25.887 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:32:25.887 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6800", 2021-02-10T11:32:25.887 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36367 2021-02-10T11:32:25.887 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:32:25.888 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:32:25.888 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:32:25.888 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6801", 2021-02-10T11:32:25.888 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36367 2021-02-10T11:32:25.888 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:32:25.888 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:32:25.888 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:32:25.888 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addrs": { 2021-02-10T11:32:25.888 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:32:25.888 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:32:25.888 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:32:25.888 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6803", 2021-02-10T11:32:25.888 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36367 2021-02-10T11:32:25.888 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:32:25.889 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:32:25.889 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:32:25.889 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6804", 2021-02-10T11:32:25.889 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36367 2021-02-10T11:32:25.889 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:32:25.889 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:32:25.889 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:32:25.889 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addrs": { 2021-02-10T11:32:25.889 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:32:25.889 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:32:25.889 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:32:25.889 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6813", 2021-02-10T11:32:25.889 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36367 2021-02-10T11:32:25.889 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:32:25.890 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:32:25.890 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:32:25.890 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6816", 2021-02-10T11:32:25.890 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36367 2021-02-10T11:32:25.890 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:32:25.890 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:32:25.890 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:32:25.890 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addrs": { 2021-02-10T11:32:25.890 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:32:25.890 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:32:25.890 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:32:25.890 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6806", 2021-02-10T11:32:25.890 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36367 2021-02-10T11:32:25.891 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:32:25.891 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:32:25.891 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:32:25.891 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6809", 2021-02-10T11:32:25.891 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36367 2021-02-10T11:32:25.891 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:32:25.891 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:32:25.891 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:32:25.891 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addr": "172.21.2.113:6801/36367", 2021-02-10T11:32:25.891 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addr": "172.21.2.113:6804/36367", 2021-02-10T11:32:25.891 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addr": "172.21.2.113:6816/36367", 2021-02-10T11:32:25.891 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addr": "172.21.2.113:6809/36367", 2021-02-10T11:32:25.891 INFO:teuthology.orchestra.run.gibba002.stdout: "state": [ 2021-02-10T11:32:25.892 INFO:teuthology.orchestra.run.gibba002.stdout: "exists", 2021-02-10T11:32:25.892 INFO:teuthology.orchestra.run.gibba002.stdout: "up" 2021-02-10T11:32:25.892 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:32:25.892 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:32:25.892 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:32:25.892 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 7, 2021-02-10T11:32:25.892 INFO:teuthology.orchestra.run.gibba002.stdout: "uuid": "f5461f80-018d-428f-be89-2eb2bcbe7bb6", 2021-02-10T11:32:25.892 INFO:teuthology.orchestra.run.gibba002.stdout: "up": 1, 2021-02-10T11:32:25.892 INFO:teuthology.orchestra.run.gibba002.stdout: "in": 1, 2021-02-10T11:32:25.892 INFO:teuthology.orchestra.run.gibba002.stdout: "weight": 1, 2021-02-10T11:32:25.892 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_affinity": 1, 2021-02-10T11:32:25.892 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_begin": 0, 2021-02-10T11:32:25.892 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_end": 0, 2021-02-10T11:32:25.892 INFO:teuthology.orchestra.run.gibba002.stdout: "up_from": 13, 2021-02-10T11:32:25.893 INFO:teuthology.orchestra.run.gibba002.stdout: "up_thru": 100, 2021-02-10T11:32:25.893 INFO:teuthology.orchestra.run.gibba002.stdout: "down_at": 0, 2021-02-10T11:32:25.893 INFO:teuthology.orchestra.run.gibba002.stdout: "lost_at": 0, 2021-02-10T11:32:25.893 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addrs": { 2021-02-10T11:32:25.893 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:32:25.893 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:32:25.893 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:32:25.893 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6817", 2021-02-10T11:32:25.893 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36365 2021-02-10T11:32:25.893 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:32:25.893 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:32:25.893 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:32:25.893 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6821", 2021-02-10T11:32:25.894 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36365 2021-02-10T11:32:25.894 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:32:25.894 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:32:25.894 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:32:25.894 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addrs": { 2021-02-10T11:32:25.894 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:32:25.894 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:32:25.894 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:32:25.894 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6823", 2021-02-10T11:32:25.894 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36365 2021-02-10T11:32:25.894 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:32:25.894 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:32:25.894 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:32:25.894 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6825", 2021-02-10T11:32:25.895 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36365 2021-02-10T11:32:25.895 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:32:25.895 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:32:25.895 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:32:25.895 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addrs": { 2021-02-10T11:32:25.895 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:32:25.895 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:32:25.895 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:32:25.895 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6830", 2021-02-10T11:32:25.895 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36365 2021-02-10T11:32:25.895 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:32:25.895 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:32:25.895 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:32:25.895 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6831", 2021-02-10T11:32:25.896 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36365 2021-02-10T11:32:25.896 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:32:25.896 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:32:25.896 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:32:25.896 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addrs": { 2021-02-10T11:32:25.896 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:32:25.896 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:32:25.896 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:32:25.896 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6827", 2021-02-10T11:32:25.896 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36365 2021-02-10T11:32:25.896 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:32:25.896 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:32:25.896 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:32:25.896 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6829", 2021-02-10T11:32:25.897 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36365 2021-02-10T11:32:25.897 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:32:25.897 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:32:25.897 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:32:25.897 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addr": "172.21.2.113:6821/36365", 2021-02-10T11:32:25.897 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addr": "172.21.2.113:6825/36365", 2021-02-10T11:32:25.897 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addr": "172.21.2.113:6831/36365", 2021-02-10T11:32:25.897 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addr": "172.21.2.113:6829/36365", 2021-02-10T11:32:25.897 INFO:teuthology.orchestra.run.gibba002.stdout: "state": [ 2021-02-10T11:32:25.897 INFO:teuthology.orchestra.run.gibba002.stdout: "exists", 2021-02-10T11:32:25.897 INFO:teuthology.orchestra.run.gibba002.stdout: "up" 2021-02-10T11:32:25.897 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:32:25.897 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:32:25.897 INFO:teuthology.orchestra.run.gibba002.stdout: ], 2021-02-10T11:32:25.898 INFO:teuthology.orchestra.run.gibba002.stdout: "osd_xinfo": [ 2021-02-10T11:32:25.898 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:32:25.898 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 0, 2021-02-10T11:32:25.898 INFO:teuthology.orchestra.run.gibba002.stdout: "down_stamp": "0.000000", 2021-02-10T11:32:25.898 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_probability": 0, 2021-02-10T11:32:25.898 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_interval": 0, 2021-02-10T11:32:25.898 INFO:teuthology.orchestra.run.gibba002.stdout: "features": 4611087854035861503, 2021-02-10T11:32:25.898 INFO:teuthology.orchestra.run.gibba002.stdout: "old_weight": 0 2021-02-10T11:32:25.898 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:32:25.898 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:32:25.898 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 1, 2021-02-10T11:32:25.898 INFO:teuthology.orchestra.run.gibba002.stdout: "down_stamp": "0.000000", 2021-02-10T11:32:25.898 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_probability": 0, 2021-02-10T11:32:25.898 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_interval": 0, 2021-02-10T11:32:25.899 INFO:teuthology.orchestra.run.gibba002.stdout: "features": 4611087854035861503, 2021-02-10T11:32:25.899 INFO:teuthology.orchestra.run.gibba002.stdout: "old_weight": 0 2021-02-10T11:32:25.899 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:32:25.899 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:32:25.899 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 2, 2021-02-10T11:32:25.899 INFO:teuthology.orchestra.run.gibba002.stdout: "down_stamp": "0.000000", 2021-02-10T11:32:25.899 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_probability": 0, 2021-02-10T11:32:25.899 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_interval": 0, 2021-02-10T11:32:25.899 INFO:teuthology.orchestra.run.gibba002.stdout: "features": 4611087854035861503, 2021-02-10T11:32:25.899 INFO:teuthology.orchestra.run.gibba002.stdout: "old_weight": 0 2021-02-10T11:32:25.899 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:32:25.899 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:32:25.899 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 3, 2021-02-10T11:32:25.899 INFO:teuthology.orchestra.run.gibba002.stdout: "down_stamp": "0.000000", 2021-02-10T11:32:25.900 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_probability": 0, 2021-02-10T11:32:25.900 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_interval": 0, 2021-02-10T11:32:25.900 INFO:teuthology.orchestra.run.gibba002.stdout: "features": 4611087854035861503, 2021-02-10T11:32:25.900 INFO:teuthology.orchestra.run.gibba002.stdout: "old_weight": 0 2021-02-10T11:32:25.900 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:32:25.900 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:32:25.900 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 4, 2021-02-10T11:32:25.900 INFO:teuthology.orchestra.run.gibba002.stdout: "down_stamp": "0.000000", 2021-02-10T11:32:25.900 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_probability": 0, 2021-02-10T11:32:25.900 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_interval": 0, 2021-02-10T11:32:25.900 INFO:teuthology.orchestra.run.gibba002.stdout: "features": 4611087854035861503, 2021-02-10T11:32:25.900 INFO:teuthology.orchestra.run.gibba002.stdout: "old_weight": 0 2021-02-10T11:32:25.900 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:32:25.901 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:32:25.901 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 5, 2021-02-10T11:32:25.901 INFO:teuthology.orchestra.run.gibba002.stdout: "down_stamp": "0.000000", 2021-02-10T11:32:25.901 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_probability": 0, 2021-02-10T11:32:25.901 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_interval": 0, 2021-02-10T11:32:25.901 INFO:teuthology.orchestra.run.gibba002.stdout: "features": 4611087854035861503, 2021-02-10T11:32:25.901 INFO:teuthology.orchestra.run.gibba002.stdout: "old_weight": 0 2021-02-10T11:32:25.901 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:32:25.901 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:32:25.901 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 6, 2021-02-10T11:32:25.901 INFO:teuthology.orchestra.run.gibba002.stdout: "down_stamp": "0.000000", 2021-02-10T11:32:25.901 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_probability": 0, 2021-02-10T11:32:25.901 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_interval": 0, 2021-02-10T11:32:25.901 INFO:teuthology.orchestra.run.gibba002.stdout: "features": 4611087854035861503, 2021-02-10T11:32:25.902 INFO:teuthology.orchestra.run.gibba002.stdout: "old_weight": 0 2021-02-10T11:32:25.902 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:32:25.902 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:32:25.902 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 7, 2021-02-10T11:32:25.902 INFO:teuthology.orchestra.run.gibba002.stdout: "down_stamp": "0.000000", 2021-02-10T11:32:25.902 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_probability": 0, 2021-02-10T11:32:25.902 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_interval": 0, 2021-02-10T11:32:25.902 INFO:teuthology.orchestra.run.gibba002.stdout: "features": 4611087854035861503, 2021-02-10T11:32:25.902 INFO:teuthology.orchestra.run.gibba002.stdout: "old_weight": 0 2021-02-10T11:32:25.902 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:32:25.902 INFO:teuthology.orchestra.run.gibba002.stdout: ], 2021-02-10T11:32:25.902 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_upmap": [], 2021-02-10T11:32:25.902 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_upmap_items": [], 2021-02-10T11:32:25.902 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_temp": [], 2021-02-10T11:32:25.903 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_temp": [], 2021-02-10T11:32:25.903 INFO:teuthology.orchestra.run.gibba002.stdout: "blacklist": {}, 2021-02-10T11:32:25.903 INFO:teuthology.orchestra.run.gibba002.stdout: "erasure_code_profiles": { 2021-02-10T11:32:25.903 INFO:teuthology.orchestra.run.gibba002.stdout: "default": { 2021-02-10T11:32:25.903 INFO:teuthology.orchestra.run.gibba002.stdout: "crush-failure-domain": "osd", 2021-02-10T11:32:25.903 INFO:teuthology.orchestra.run.gibba002.stdout: "k": "2", 2021-02-10T11:32:25.903 INFO:teuthology.orchestra.run.gibba002.stdout: "m": "1", 2021-02-10T11:32:25.903 INFO:teuthology.orchestra.run.gibba002.stdout: "plugin": "jerasure", 2021-02-10T11:32:25.903 INFO:teuthology.orchestra.run.gibba002.stdout: "ruleset-failure-domain": "osd", 2021-02-10T11:32:25.903 INFO:teuthology.orchestra.run.gibba002.stdout: "technique": "reed_sol_van" 2021-02-10T11:32:25.903 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:32:25.903 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:32:25.903 INFO:teuthology.orchestra.run.gibba002.stdout: "removed_snaps_queue": [], 2021-02-10T11:32:25.903 INFO:teuthology.orchestra.run.gibba002.stdout: "new_removed_snaps": [], 2021-02-10T11:32:25.904 INFO:teuthology.orchestra.run.gibba002.stdout: "new_purged_snaps": [], 2021-02-10T11:32:25.904 INFO:teuthology.orchestra.run.gibba002.stdout: "crush_node_flags": {}, 2021-02-10T11:32:25.904 INFO:teuthology.orchestra.run.gibba002.stdout: "device_class_flags": {} 2021-02-10T11:32:25.904 INFO:teuthology.orchestra.run.gibba002.stdout:} 2021-02-10T11:32:25.904 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:25.806 7f9f60b07700 1 -- 172.21.2.102:0/2813630673 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f9f40018730 msgr2=0x7f9f4001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:25.904 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:25.806 7f9f60b07700 1 --2- 172.21.2.102:0/2813630673 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f9f40018730 0x7f9f4001ac10 crc :-1 s=READY pgs=996 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:25.904 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:25.806 7f9f60b07700 1 -- 172.21.2.102:0/2813630673 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9f5c19c4a0 msgr2=0x7f9f5c08a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:25.904 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:25.806 7f9f60b07700 1 --2- 172.21.2.102:0/2813630673 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9f5c19c4a0 0x7f9f5c08a990 secure :-1 s=READY pgs=2260 cs=0 l=1 rev1=1 rx=0x7f9f4c006ed0 tx=0x7f9f4c00b8e0).stop 2021-02-10T11:32:25.904 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:25.806 7f9f60b07700 1 -- 172.21.2.102:0/2813630673 shutdown_connections 2021-02-10T11:32:25.905 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:25.806 7f9f60b07700 1 --2- 172.21.2.102:0/2813630673 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9f5c19c4a0 0x7f9f5c08a990 unknown :-1 s=CLOSED pgs=2260 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:25.905 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:25.806 7f9f60b07700 1 --2- 172.21.2.102:0/2813630673 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9f5c19ba80 0x7f9f5c19be80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:25.905 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:25.806 7f9f60b07700 1 --2- 172.21.2.102:0/2813630673 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f9f40018730 0x7f9f4001ac10 unknown :-1 s=CLOSED pgs=996 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:25.905 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:25.806 7f9f60b07700 1 --2- 172.21.2.102:0/2813630673 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f9f5c19af00 0x7f9f5c19b300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:25.905 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:25.806 7f9f60b07700 1 -- 172.21.2.102:0/2813630673 >> 172.21.2.102:0/2813630673 conn(0x7f9f5c19f8c0 msgr2=0x7f9f5c1a5fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:25.905 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:25.806 7f9f60b07700 1 -- 172.21.2.102:0/2813630673 shutdown_connections 2021-02-10T11:32:25.905 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:25.806 7f9f60b07700 1 -- 172.21.2.102:0/2813630673 wait complete. 2021-02-10T11:32:25.906 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:32:26.335 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:26.318 7f813c87c700 1 Processor -- start 2021-02-10T11:32:26.335 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:26.318 7f813c87c700 1 -- start start 2021-02-10T11:32:26.335 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:26.318 7f813c87c700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f81380d1870 0x7f81381b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:26.335 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:26.318 7f813c87c700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f81380f0650 0x7f81380cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:26.335 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:26.318 7f813c87c700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81380f0b10 con 0x7f81381b0a10 2021-02-10T11:32:26.335 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:26.318 7f813c87c700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81380fe230 con 0x7f81380f0650 2021-02-10T11:32:26.335 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:26.318 7f813c87c700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81380e5ed0 con 0x7f81380d1870 2021-02-10T11:32:26.335 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:26.319 7f8136e1d700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f81380f0650 0x7f81380cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:26.336 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:26.319 7f8136e1d700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f81380f0650 0x7f81380cfb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:34432/0 (socket says 172.21.2.102:34432) 2021-02-10T11:32:26.336 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:26.319 7f8136e1d700 1 -- 172.21.2.102:0/3917480736 learned_addr learned my addr 172.21.2.102:0/3917480736 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:32:26.336 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:26.319 7f8136e1d700 1 -- 172.21.2.102:0/3917480736 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f81380d1870 msgr2=0x7f81381b03f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:32:26.336 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:26.319 7f8136e1d700 1 --2- 172.21.2.102:0/3917480736 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f81380d1870 0x7f81381b03f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:26.336 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:26.319 7f8136e1d700 1 -- 172.21.2.102:0/3917480736 >> v1:172.21.2.102:6789/0 conn(0x7f81381b0a10 legacy=0x7f81380cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:26.336 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:26.319 7f8136e1d700 1 -- 172.21.2.102:0/3917480736 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81380f20a0 con 0x7f81380f0650 2021-02-10T11:32:26.336 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:26.319 7f8136e1d700 1 --2- 172.21.2.102:0/3917480736 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f81380f0650 0x7f81380cfb20 secure :-1 s=READY pgs=2262 cs=0 l=1 rev1=1 rx=0x7f8130006f00 tx=0x7f8130009f10).ready entity=mon.0 client_cookie=3fadb002c9472e7c server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:26.336 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:26.319 7f813561a700 1 -- 172.21.2.102:0/3917480736 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f813000b3b0 con 0x7f81380f0650 2021-02-10T11:32:26.336 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:26.319 7f813561a700 1 -- 172.21.2.102:0/3917480736 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f813000b630 con 0x7f81380f0650 2021-02-10T11:32:26.336 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:26.319 7f813561a700 1 -- 172.21.2.102:0/3917480736 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f813000bb50 con 0x7f81380f0650 2021-02-10T11:32:26.336 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:26.319 7f813c87c700 1 -- 172.21.2.102:0/3917480736 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f81380f0650 msgr2=0x7f81380cfb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:26.336 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:26.319 7f813c87c700 1 --2- 172.21.2.102:0/3917480736 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f81380f0650 0x7f81380cfb20 secure :-1 s=READY pgs=2262 cs=0 l=1 rev1=1 rx=0x7f8130006f00 tx=0x7f8130009f10).stop 2021-02-10T11:32:26.336 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:26.320 7f813c87c700 1 -- 172.21.2.102:0/3917480736 shutdown_connections 2021-02-10T11:32:26.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:26.320 7f813c87c700 1 --2- 172.21.2.102:0/3917480736 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f81380f0650 0x7f81380cfb20 unknown :-1 s=CLOSED pgs=2262 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:26.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:26.320 7f813c87c700 1 --2- 172.21.2.102:0/3917480736 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f81380d1870 0x7f81381b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:26.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:26.320 7f813c87c700 1 -- 172.21.2.102:0/3917480736 >> 172.21.2.102:0/3917480736 conn(0x7f81380ce450 msgr2=0x7f81380d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:26.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:26.320 7f813c87c700 1 -- 172.21.2.102:0/3917480736 shutdown_connections 2021-02-10T11:32:26.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:26.320 7f813c87c700 1 -- 172.21.2.102:0/3917480736 wait complete. 2021-02-10T11:32:26.338 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:26.335 7f813c87c700 1 Processor -- start 2021-02-10T11:32:26.338 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:26.335 7f813c87c700 1 -- start start 2021-02-10T11:32:26.338 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:26.335 7f813c87c700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f813819af00 0x7f813819b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:26.338 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:26.335 7f813c87c700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f813819ba80 0x7f813819be80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:26.339 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:26.335 7f813c87c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f813819c4a0 0x7f813808a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:26.339 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:26.335 7f813c87c700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f813808fb50 con 0x7f813819c4a0 2021-02-10T11:32:26.339 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:26.335 7f813c87c700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f813808afb0 con 0x7f813819af00 2021-02-10T11:32:26.339 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:26.335 7f813c87c700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f813808b1a0 con 0x7f813819ba80 2021-02-10T11:32:26.339 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:26.335 7f8136e1d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f813819c4a0 0x7f813808a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:26.339 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:26.335 7f8136e1d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f813819c4a0 0x7f813808a990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:34440/0 (socket says 172.21.2.102:34440) 2021-02-10T11:32:26.339 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:26.335 7f8136e1d700 1 -- 172.21.2.102:0/1493992808 learned_addr learned my addr 172.21.2.102:0/1493992808 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:32:26.339 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:26.336 7f8136e1d700 1 -- 172.21.2.102:0/1493992808 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f813819ba80 msgr2=0x7f813819be80 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:32:26.339 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:26.336 7f8136e1d700 1 --2- 172.21.2.102:0/1493992808 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f813819ba80 0x7f813819be80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:26.339 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:26.336 7f8136e1d700 1 -- 172.21.2.102:0/1493992808 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f813819af00 msgr2=0x7f813819b300 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:32:26.339 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:26.336 7f8136e1d700 1 --2- 172.21.2.102:0/1493992808 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f813819af00 0x7f813819b300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:26.339 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:26.336 7f8136e1d700 1 -- 172.21.2.102:0/1493992808 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f813808b3c0 con 0x7f813819c4a0 2021-02-10T11:32:26.339 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:26.336 7f8136e1d700 1 --2- 172.21.2.102:0/1493992808 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f813819c4a0 0x7f813808a990 secure :-1 s=READY pgs=2263 cs=0 l=1 rev1=1 rx=0x7f8130009390 tx=0x7f813000bd40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:26.339 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:26.336 7f812effd700 1 -- 172.21.2.102:0/1493992808 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8130003030 con 0x7f813819c4a0 2021-02-10T11:32:26.340 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:26.336 7f812effd700 1 -- 172.21.2.102:0/1493992808 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8130006c80 con 0x7f813819c4a0 2021-02-10T11:32:26.340 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:26.336 7f812effd700 1 -- 172.21.2.102:0/1493992808 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8130001480 con 0x7f813819c4a0 2021-02-10T11:32:26.340 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:26.336 7f813c87c700 1 -- 172.21.2.102:0/1493992808 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f813808b3c0 con 0x7f813819c4a0 2021-02-10T11:32:26.340 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:26.336 7f813c87c700 1 -- 172.21.2.102:0/1493992808 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f813808b950 con 0x7f813819c4a0 2021-02-10T11:32:26.340 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:26.337 7f812effd700 1 -- 172.21.2.102:0/1493992808 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f8130001270 con 0x7f813819c4a0 2021-02-10T11:32:26.340 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:26.337 7f812effd700 1 --2- 172.21.2.102:0/1493992808 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f811c0186c0 0x7f811c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:26.340 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:26.337 7f812effd700 1 -- 172.21.2.102:0/1493992808 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f81300328e0 con 0x7f813819c4a0 2021-02-10T11:32:26.340 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:26.337 7f813661c700 1 --2- 172.21.2.102:0/1493992808 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f811c0186c0 0x7f811c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:26.340 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:26.337 7f813661c700 1 --2- 172.21.2.102:0/1493992808 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f811c0186c0 0x7f811c01abc0 crc :-1 s=READY pgs=997 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:26.369 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:26.368 7f813c87c700 1 -- 172.21.2.102:0/1493992808 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81380ee030 con 0x7f813819c4a0 2021-02-10T11:32:26.371 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:26.371 7f812effd700 1 -- 172.21.2.102:0/1493992808 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f8130009450 con 0x7f813819c4a0 2021-02-10T11:32:26.499 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:26.499 7f813c87c700 1 -- 172.21.2.102:0/1493992808 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f81380ce450 con 0x7f813819c4a0 2021-02-10T11:32:26.500 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:26.499 7f812effd700 1 -- 172.21.2.102:0/1493992808 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 92 v92) v1 ==== 94+0+2104 (secure 0 0 0) 0x7f8130001270 con 0x7f813819c4a0 2021-02-10T11:32:26.500 INFO:teuthology.orchestra.run.gibba002.stderr:dumped fsmap epoch 92 2021-02-10T11:32:26.501 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:32:26.501 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":92,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10404,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.102:6835/1661316191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6834","nonce":1661316191},{"type":"v1","addr":"172.21.2.102:6835","nonce":1661316191}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:31:48.259584","modified":"2021-02-10 11:31:50.267046","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10401},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10401":{"gid":10401,"name":"b","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.2.113:6833/313019011","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6832","nonce":313019011},{"type":"v1","addr":"172.21.2.113:6833","nonce":313019011}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[23],"metadata_pool":22,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":11}]} 2021-02-10T11:32:26.502 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:26.501 7f813c87c700 1 -- 172.21.2.102:0/1493992808 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f811c0186c0 msgr2=0x7f811c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:26.503 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:26.501 7f813c87c700 1 --2- 172.21.2.102:0/1493992808 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f811c0186c0 0x7f811c01abc0 crc :-1 s=READY pgs=997 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:26.503 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:26.501 7f813c87c700 1 -- 172.21.2.102:0/1493992808 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f813819c4a0 msgr2=0x7f813808a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:26.503 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:26.501 7f813c87c700 1 --2- 172.21.2.102:0/1493992808 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f813819c4a0 0x7f813808a990 secure :-1 s=READY pgs=2263 cs=0 l=1 rev1=1 rx=0x7f8130009390 tx=0x7f813000bd40).stop 2021-02-10T11:32:26.503 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:26.502 7f813c87c700 1 -- 172.21.2.102:0/1493992808 shutdown_connections 2021-02-10T11:32:26.503 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:26.502 7f813c87c700 1 --2- 172.21.2.102:0/1493992808 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f813819c4a0 0x7f813808a990 unknown :-1 s=CLOSED pgs=2263 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:26.503 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:26.502 7f813c87c700 1 --2- 172.21.2.102:0/1493992808 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f813819ba80 0x7f813819be80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:26.503 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:26.502 7f813c87c700 1 --2- 172.21.2.102:0/1493992808 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f811c0186c0 0x7f811c01abc0 unknown :-1 s=CLOSED pgs=997 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:26.503 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:26.502 7f813c87c700 1 --2- 172.21.2.102:0/1493992808 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f813819af00 0x7f813819b300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:26.503 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:26.502 7f813c87c700 1 -- 172.21.2.102:0/1493992808 >> 172.21.2.102:0/1493992808 conn(0x7f813819f8c0 msgr2=0x7f81381a5fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:26.504 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:26.502 7f813c87c700 1 -- 172.21.2.102:0/1493992808 shutdown_connections 2021-02-10T11:32:26.504 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:26.502 7f813c87c700 1 -- 172.21.2.102:0/1493992808 wait complete. 2021-02-10T11:32:26.510 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs fail cephfs 2021-02-10T11:32:26.940 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:26.924 7f7cde981700 1 Processor -- start 2021-02-10T11:32:26.940 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:26.924 7f7cde981700 1 -- start start 2021-02-10T11:32:26.940 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:26.924 7f7cde981700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f7cd80d0340 0x7f7cd80ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:26.940 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:26.924 7f7cde981700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f7cd80d1c70 0x7f7cd80efc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:26.941 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:26.924 7f7cde981700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7cd80f0b10 con 0x7f7cd80d1870 2021-02-10T11:32:26.941 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:26.924 7f7cde981700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7cd80e5ed0 con 0x7f7cd80d0340 2021-02-10T11:32:26.941 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:26.924 7f7cde981700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7cd81b0be0 con 0x7f7cd80d1c70 2021-02-10T11:32:26.941 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:26.924 7f7cd77fe700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f7cd80d0340 0x7f7cd80ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:26.941 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:26.924 7f7cd7fff700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f7cd80d1870 0x7f7cd81b03f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:54900/0 (socket says 172.21.2.102:54900) 2021-02-10T11:32:26.941 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:26.924 7f7cd77fe700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f7cd80d0340 0x7f7cd80ef640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:34446/0 (socket says 172.21.2.102:34446) 2021-02-10T11:32:26.941 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:26.924 7f7cd7fff700 1 -- 172.21.2.102:0/1464767728 learned_addr learned my addr 172.21.2.102:0/1464767728 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:32:26.941 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:26.925 7f7cd77fe700 1 -- 172.21.2.102:0/1464767728 >> v2:172.21.2.113:3300/0 conn(0x7f7cd80d1c70 msgr2=0x7f7cd80efc60 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:32:26.941 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:26.925 7f7cd77fe700 1 --2- 172.21.2.102:0/1464767728 >> v2:172.21.2.113:3300/0 conn(0x7f7cd80d1c70 0x7f7cd80efc60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:26.942 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:26.925 7f7cd77fe700 1 -- 172.21.2.102:0/1464767728 >> v1:172.21.2.102:6789/0 conn(0x7f7cd80d1870 legacy=0x7f7cd81b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:26.942 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:26.925 7f7cd77fe700 1 -- 172.21.2.102:0/1464767728 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cd80f20a0 con 0x7f7cd80d0340 2021-02-10T11:32:26.942 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:26.925 7f7cd77fe700 1 --2- 172.21.2.102:0/1464767728 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7cd80d0340 0x7f7cd80ef640 secure :-1 s=READY pgs=2265 cs=0 l=1 rev1=1 rx=0x7f7ccc006650 tx=0x7f7ccc009a80).ready entity=mon.0 client_cookie=f9126479e8bf4ae3 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:26.942 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:26.925 7f7cd6ffd700 1 -- 172.21.2.102:0/1464767728 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7ccc00c0c0 con 0x7f7cd80d0340 2021-02-10T11:32:26.942 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:26.925 7f7cd6ffd700 1 -- 172.21.2.102:0/1464767728 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ccc00c340 con 0x7f7cd80d0340 2021-02-10T11:32:26.942 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:26.925 7f7cd6ffd700 1 -- 172.21.2.102:0/1464767728 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7ccc00c860 con 0x7f7cd80d0340 2021-02-10T11:32:26.942 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:26.925 7f7cde981700 1 -- 172.21.2.102:0/1464767728 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7cd80d0340 msgr2=0x7f7cd80ef640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:26.942 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:26.925 7f7cde981700 1 --2- 172.21.2.102:0/1464767728 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7cd80d0340 0x7f7cd80ef640 secure :-1 s=READY pgs=2265 cs=0 l=1 rev1=1 rx=0x7f7ccc006650 tx=0x7f7ccc009a80).stop 2021-02-10T11:32:26.942 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:26.925 7f7cde981700 1 -- 172.21.2.102:0/1464767728 shutdown_connections 2021-02-10T11:32:26.942 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:26.925 7f7cde981700 1 --2- 172.21.2.102:0/1464767728 >> v2:172.21.2.113:3300/0 conn(0x7f7cd80d1c70 0x7f7cd80efc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:26.942 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:26.925 7f7cde981700 1 --2- 172.21.2.102:0/1464767728 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7cd80d0340 0x7f7cd80ef640 unknown :-1 s=CLOSED pgs=2265 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:26.942 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:26.925 7f7cde981700 1 -- 172.21.2.102:0/1464767728 >> 172.21.2.102:0/1464767728 conn(0x7f7cd80ce450 msgr2=0x7f7cd80d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:26.943 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:26.926 7f7cde981700 1 -- 172.21.2.102:0/1464767728 shutdown_connections 2021-02-10T11:32:26.943 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:26.926 7f7cde981700 1 -- 172.21.2.102:0/1464767728 wait complete. 2021-02-10T11:32:26.944 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:26.940 7f7cde981700 1 Processor -- start 2021-02-10T11:32:26.944 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:26.940 7f7cde981700 1 -- start start 2021-02-10T11:32:26.944 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:26.941 7f7cde981700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7cd80948c0 0x7f7cd8094d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:26.944 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:26.941 7f7cde981700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7cd8093d40 0x7f7cd8094140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:26.945 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:26.941 7f7cde981700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7cd80bfc70 0x7f7cd80c0070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:26.945 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:26.941 7f7cde981700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cd808ded0 con 0x7f7cd8093d40 2021-02-10T11:32:26.945 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:26.941 7f7cde981700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cd80953b0 con 0x7f7cd80948c0 2021-02-10T11:32:26.945 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:26.941 7f7cde981700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7cd80c0690 con 0x7f7cd80bfc70 2021-02-10T11:32:26.945 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:26.941 7f7cd77fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7cd8093d40 0x7f7cd8094140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:26.945 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:26.941 7f7cd77fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7cd8093d40 0x7f7cd8094140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:34450/0 (socket says 172.21.2.102:34450) 2021-02-10T11:32:26.945 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:26.941 7f7cd77fe700 1 -- 172.21.2.102:0/1599162902 learned_addr learned my addr 172.21.2.102:0/1599162902 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:32:26.945 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:26.941 7f7cd7fff700 1 --2- 172.21.2.102:0/1599162902 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7cd80948c0 0x7f7cd8094d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:26.945 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:26.941 7f7cdcf9e700 1 --2- 172.21.2.102:0/1599162902 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7cd80bfc70 0x7f7cd80c0070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:26.945 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:26.941 7f7cd77fe700 1 -- 172.21.2.102:0/1599162902 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7cd80bfc70 msgr2=0x7f7cd80c0070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:26.945 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:26.941 7f7cd77fe700 1 --2- 172.21.2.102:0/1599162902 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7cd80bfc70 0x7f7cd80c0070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:26.946 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:26.941 7f7cd77fe700 1 -- 172.21.2.102:0/1599162902 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7cd80948c0 msgr2=0x7f7cd8094d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:26.946 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:26.941 7f7cd77fe700 1 --2- 172.21.2.102:0/1599162902 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7cd80948c0 0x7f7cd8094d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:26.946 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:26.941 7f7cd77fe700 1 -- 172.21.2.102:0/1599162902 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cd80c08b0 con 0x7f7cd8093d40 2021-02-10T11:32:26.946 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:26.941 7f7cd77fe700 1 --2- 172.21.2.102:0/1599162902 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7cd8093d40 0x7f7cd8094140 secure :-1 s=READY pgs=2266 cs=0 l=1 rev1=1 rx=0x7f7ccc009650 tx=0x7f7ccc00b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:26.946 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:26.942 7f7cd4ff9700 1 -- 172.21.2.102:0/1599162902 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7ccc0093a0 con 0x7f7cd8093d40 2021-02-10T11:32:26.946 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:26.942 7f7cd4ff9700 1 -- 172.21.2.102:0/1599162902 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ccc00bde0 con 0x7f7cd8093d40 2021-02-10T11:32:26.946 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:26.942 7f7cd4ff9700 1 -- 172.21.2.102:0/1599162902 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7ccc0052e0 con 0x7f7cd8093d40 2021-02-10T11:32:26.946 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:26.942 7f7cde981700 1 -- 172.21.2.102:0/1599162902 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7cd80c08b0 con 0x7f7cd8093d40 2021-02-10T11:32:26.946 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:26.942 7f7cde981700 1 -- 172.21.2.102:0/1599162902 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7cd80c08b0 con 0x7f7cd8093d40 2021-02-10T11:32:26.946 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:26.942 7f7cd4ff9700 1 -- 172.21.2.102:0/1599162902 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f7ccc006030 con 0x7f7cd8093d40 2021-02-10T11:32:26.946 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:26.942 7f7cd4ff9700 1 --2- 172.21.2.102:0/1599162902 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7cb80186c0 0x7f7cb801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:26.946 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:26.942 7f7cd4ff9700 1 -- 172.21.2.102:0/1599162902 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f7ccc032400 con 0x7f7cd8093d40 2021-02-10T11:32:26.947 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:26.942 7f7cd7fff700 1 --2- 172.21.2.102:0/1599162902 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7cb80186c0 0x7f7cb801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:26.947 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:26.943 7f7cd7fff700 1 --2- 172.21.2.102:0/1599162902 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7cb80186c0 0x7f7cb801abc0 crc :-1 s=READY pgs=998 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:26.975 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:26.974 7f7cde981700 1 -- 172.21.2.102:0/1599162902 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7cd80ebc90 con 0x7f7cd8093d40 2021-02-10T11:32:26.977 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:26.977 7f7cd4ff9700 1 -- 172.21.2.102:0/1599162902 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f7ccc000940 con 0x7f7cd8093d40 2021-02-10T11:32:27.106 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:27.106 7f7cde981700 1 -- 172.21.2.102:0/1599162902 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f7cd80f0b10 con 0x7f7cd8093d40 2021-02-10T11:32:27.864 INFO:tasks.ceph.mon.a.gibba002.stderr:2021-02-10 11:32:27.863 7f8dd2755700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-02-10T11:32:27.870 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:27.869 7f7cd4ff9700 1 -- 172.21.2.102:0/1599162902 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v93) v1 ==== 161+0+0 (secure 0 0 0) 0x7f7ccc006030 con 0x7f7cd8093d40 2021-02-10T11:32:27.876 INFO:teuthology.orchestra.run.gibba002.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2021-02-10T11:32:27.878 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:27.876 7f7cde981700 1 -- 172.21.2.102:0/1599162902 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7cb80186c0 msgr2=0x7f7cb801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:27.878 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:27.876 7f7cde981700 1 --2- 172.21.2.102:0/1599162902 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7cb80186c0 0x7f7cb801abc0 crc :-1 s=READY pgs=998 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:27.878 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:27.876 7f7cde981700 1 -- 172.21.2.102:0/1599162902 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7cd8093d40 msgr2=0x7f7cd8094140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:27.878 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:27.876 7f7cde981700 1 --2- 172.21.2.102:0/1599162902 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7cd8093d40 0x7f7cd8094140 secure :-1 s=READY pgs=2266 cs=0 l=1 rev1=1 rx=0x7f7ccc009650 tx=0x7f7ccc00b960).stop 2021-02-10T11:32:27.878 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:27.876 7f7cde981700 1 -- 172.21.2.102:0/1599162902 shutdown_connections 2021-02-10T11:32:27.878 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:27.876 7f7cde981700 1 --2- 172.21.2.102:0/1599162902 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7cd80bfc70 0x7f7cd80c0070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:27.878 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:27.876 7f7cde981700 1 --2- 172.21.2.102:0/1599162902 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7cd8093d40 0x7f7cd8094140 unknown :-1 s=CLOSED pgs=2266 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:27.879 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:27.876 7f7cde981700 1 --2- 172.21.2.102:0/1599162902 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7cb80186c0 0x7f7cb801abc0 unknown :-1 s=CLOSED pgs=998 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:27.879 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:27.876 7f7cde981700 1 --2- 172.21.2.102:0/1599162902 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7cd80948c0 0x7f7cd8094d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:27.879 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:27.876 7f7cde981700 1 -- 172.21.2.102:0/1599162902 >> 172.21.2.102:0/1599162902 conn(0x7f7cd80bf640 msgr2=0x7f7cd809f200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:27.879 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:27.877 7f7cde981700 1 -- 172.21.2.102:0/1599162902 shutdown_connections 2021-02-10T11:32:27.879 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:27.877 7f7cde981700 1 -- 172.21.2.102:0/1599162902 wait complete. 2021-02-10T11:32:27.886 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:32:27.916 INFO:tasks.ceph.mds.b.gibba013.stdout:starting mds.b at 2021-02-10T11:32:28.314 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:28.296 7fa92b2f0700 1 Processor -- start 2021-02-10T11:32:28.314 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:28.296 7fa92b2f0700 1 -- start start 2021-02-10T11:32:28.314 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:28.296 7fa92b2f0700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa9240d1870 0x7fa9241b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:28.314 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:28.297 7fa92b2f0700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fa9240f0650 0x7fa9240cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:28.314 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:28.297 7fa92b2f0700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9240f0b10 con 0x7fa9241b0a10 2021-02-10T11:32:28.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:28.297 7fa92b2f0700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9240fe230 con 0x7fa9240f0650 2021-02-10T11:32:28.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:28.297 7fa92b2f0700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9240e5ed0 con 0x7fa9240d1870 2021-02-10T11:32:28.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:28.297 7fa92990d700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fa9240f0650 0x7fa9240cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:28.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:28.297 7fa92990d700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fa9240f0650 0x7fa9240cfb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:34456/0 (socket says 172.21.2.102:34456) 2021-02-10T11:32:28.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:28.297 7fa92990d700 1 -- 172.21.2.102:0/2521714539 learned_addr learned my addr 172.21.2.102:0/2521714539 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:32:28.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:28.297 7fa92890b700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fa9241b0a10 0x7fa9240cf330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:54912/0 (socket says 172.21.2.102:54912) 2021-02-10T11:32:28.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:28.297 7fa92990d700 1 -- 172.21.2.102:0/2521714539 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa9240d1870 msgr2=0x7fa9241b03f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:32:28.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:28.297 7fa92990d700 1 --2- 172.21.2.102:0/2521714539 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa9240d1870 0x7fa9241b03f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:28.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:28.297 7fa92990d700 1 -- 172.21.2.102:0/2521714539 >> v1:172.21.2.102:6789/0 conn(0x7fa9241b0a10 legacy=0x7fa9240cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:28.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:28.297 7fa92990d700 1 -- 172.21.2.102:0/2521714539 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9240f20a0 con 0x7fa9240f0650 2021-02-10T11:32:28.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:28.297 7fa917fff700 1 -- 172.21.2.102:0/2521714539 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2543698740 0 0) 0x7fa9200010b0 con 0x7fa9241b0a10 2021-02-10T11:32:28.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:28.297 7fa92990d700 1 --2- 172.21.2.102:0/2521714539 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa9240f0650 0x7fa9240cfb20 secure :-1 s=READY pgs=2268 cs=0 l=1 rev1=1 rx=0x7fa91c006f00 tx=0x7fa91c009e90).ready entity=mon.0 client_cookie=41003711e63f28d4 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:28.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:28.297 7fa917fff700 1 -- 172.21.2.102:0/2521714539 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa91c00b300 con 0x7fa9240f0650 2021-02-10T11:32:28.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:28.297 7fa917fff700 1 -- 172.21.2.102:0/2521714539 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa91c00b640 con 0x7fa9240f0650 2021-02-10T11:32:28.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:28.297 7fa917fff700 1 -- 172.21.2.102:0/2521714539 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa91c00baa0 con 0x7fa9240f0650 2021-02-10T11:32:28.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:28.297 7fa92b2f0700 1 -- 172.21.2.102:0/2521714539 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa9240f0650 msgr2=0x7fa9240cfb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:28.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:28.297 7fa92b2f0700 1 --2- 172.21.2.102:0/2521714539 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa9240f0650 0x7fa9240cfb20 secure :-1 s=READY pgs=2268 cs=0 l=1 rev1=1 rx=0x7fa91c006f00 tx=0x7fa91c009e90).stop 2021-02-10T11:32:28.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:28.298 7fa92b2f0700 1 -- 172.21.2.102:0/2521714539 shutdown_connections 2021-02-10T11:32:28.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:28.298 7fa92b2f0700 1 --2- 172.21.2.102:0/2521714539 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa9240f0650 0x7fa9240cfb20 unknown :-1 s=CLOSED pgs=2268 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:28.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:28.298 7fa92b2f0700 1 --2- 172.21.2.102:0/2521714539 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa9240d1870 0x7fa9241b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:28.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:28.298 7fa92b2f0700 1 -- 172.21.2.102:0/2521714539 >> 172.21.2.102:0/2521714539 conn(0x7fa9240ce450 msgr2=0x7fa9240d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:28.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:28.298 7fa92b2f0700 1 -- 172.21.2.102:0/2521714539 shutdown_connections 2021-02-10T11:32:28.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:28.298 7fa92b2f0700 1 -- 172.21.2.102:0/2521714539 wait complete. 2021-02-10T11:32:28.318 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:28.313 7fa92b2f0700 1 Processor -- start 2021-02-10T11:32:28.318 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:28.313 7fa92b2f0700 1 -- start start 2021-02-10T11:32:28.318 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:28.313 7fa92b2f0700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa9241971e0 0x7fa92418eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:28.318 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:28.313 7fa92b2f0700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa9241b8180 0x7fa9241b8580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:28.318 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:28.313 7fa92b2f0700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa9241b8ba0 0x7fa9240b17b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:28.318 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:28.313 7fa92b2f0700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9241b9750 con 0x7fa9241971e0 2021-02-10T11:32:28.318 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:28.313 7fa92b2f0700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa92418e930 con 0x7fa9241b8180 2021-02-10T11:32:28.319 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:28.313 7fa92b2f0700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9240b1dd0 con 0x7fa9241b8ba0 2021-02-10T11:32:28.319 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:28.314 7fa92910c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa9241971e0 0x7fa92418eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:28.319 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:28.314 7fa92910c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa9241971e0 0x7fa92418eda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:34460/0 (socket says 172.21.2.102:34460) 2021-02-10T11:32:28.319 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:28.314 7fa92910c700 1 -- 172.21.2.102:0/3083265598 learned_addr learned my addr 172.21.2.102:0/3083265598 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:32:28.319 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:28.314 7fa92910c700 1 -- 172.21.2.102:0/3083265598 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa9241b8ba0 msgr2=0x7fa9240b17b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:28.319 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:28.314 7fa92910c700 1 --2- 172.21.2.102:0/3083265598 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa9241b8ba0 0x7fa9240b17b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:28.319 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:28.314 7fa92910c700 1 -- 172.21.2.102:0/3083265598 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa9241b8180 msgr2=0x7fa9241b8580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:28.319 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:28.314 7fa92910c700 1 --2- 172.21.2.102:0/3083265598 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa9241b8180 0x7fa9241b8580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:28.319 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:28.314 7fa92910c700 1 -- 172.21.2.102:0/3083265598 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9240b1ff0 con 0x7fa9241971e0 2021-02-10T11:32:28.319 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:28.314 7fa92910c700 1 --2- 172.21.2.102:0/3083265598 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa9241971e0 0x7fa92418eda0 secure :-1 s=READY pgs=2269 cs=0 l=1 rev1=1 rx=0x7fa91800b5c0 tx=0x7fa91800b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:28.319 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:28.314 7fa915ffb700 1 -- 172.21.2.102:0/3083265598 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa91800bd50 con 0x7fa9241971e0 2021-02-10T11:32:28.319 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:28.314 7fa915ffb700 1 -- 172.21.2.102:0/3083265598 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa918006030 con 0x7fa9241971e0 2021-02-10T11:32:28.320 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:28.314 7fa92b2f0700 1 -- 172.21.2.102:0/3083265598 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9240b1ff0 con 0x7fa9241971e0 2021-02-10T11:32:28.320 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:28.315 7fa92b2f0700 1 -- 172.21.2.102:0/3083265598 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9240b1ff0 con 0x7fa9241971e0 2021-02-10T11:32:28.320 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:28.315 7fa915ffb700 1 -- 172.21.2.102:0/3083265598 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa9180054d0 con 0x7fa9241971e0 2021-02-10T11:32:28.320 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:28.315 7fa915ffb700 1 -- 172.21.2.102:0/3083265598 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa918006210 con 0x7fa9241971e0 2021-02-10T11:32:28.320 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:28.315 7fa915ffb700 1 --2- 172.21.2.102:0/3083265598 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa904018730 0x7fa90401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:28.320 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:28.315 7fa915ffb700 1 -- 172.21.2.102:0/3083265598 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5659+0+0 (secure 0 0 0) 0x7fa918032810 con 0x7fa9241971e0 2021-02-10T11:32:28.320 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:28.315 7fa92890b700 1 --2- 172.21.2.102:0/3083265598 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa904018730 0x7fa90401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:28.320 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:28.316 7fa92890b700 1 --2- 172.21.2.102:0/3083265598 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa904018730 0x7fa90401abc0 crc :-1 s=READY pgs=999 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:28.348 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:28.346 7fa92b2f0700 1 -- 172.21.2.102:0/3083265598 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9240ee060 con 0x7fa9241971e0 2021-02-10T11:32:28.350 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:28.349 7fa915ffb700 1 -- 172.21.2.102:0/3083265598 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fa918006b80 con 0x7fa9241971e0 2021-02-10T11:32:28.479 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:28.477 7fa92b2f0700 1 -- 172.21.2.102:0/3083265598 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa9240ee060 con 0x7fa9241971e0 2021-02-10T11:32:28.480 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:28.478 7fa915ffb700 1 -- 172.21.2.102:0/3083265598 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 93 v93) v1 ==== 94+0+1760 (secure 0 0 0) 0x7fa91800bd50 con 0x7fa9241971e0 2021-02-10T11:32:28.480 INFO:teuthology.orchestra.run.gibba002.stderr:dumped fsmap epoch 93 2021-02-10T11:32:28.480 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:32:28.481 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":93,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10404,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.102:6835/1661316191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6834","nonce":1661316191},{"type":"v1","addr":"172.21.2.102:6835","nonce":1661316191}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":87}],"filesystems":[{"mdsmap":{"epoch":93,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:31:48.259584","modified":"2021-02-10 11:32:27.864142","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":113,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[23],"metadata_pool":22,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":11}]} 2021-02-10T11:32:28.481 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:28.480 7fa92b2f0700 1 -- 172.21.2.102:0/3083265598 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa904018730 msgr2=0x7fa90401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:28.481 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:28.480 7fa92b2f0700 1 --2- 172.21.2.102:0/3083265598 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa904018730 0x7fa90401abc0 crc :-1 s=READY pgs=999 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:28.481 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:28.480 7fa92b2f0700 1 -- 172.21.2.102:0/3083265598 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa9241971e0 msgr2=0x7fa92418eda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:28.481 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:28.480 7fa92b2f0700 1 --2- 172.21.2.102:0/3083265598 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa9241971e0 0x7fa92418eda0 secure :-1 s=READY pgs=2269 cs=0 l=1 rev1=1 rx=0x7fa91800b5c0 tx=0x7fa91800b960).stop 2021-02-10T11:32:28.482 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:28.480 7fa92b2f0700 1 -- 172.21.2.102:0/3083265598 shutdown_connections 2021-02-10T11:32:28.482 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:28.480 7fa92b2f0700 1 --2- 172.21.2.102:0/3083265598 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa9241b8ba0 0x7fa9240b17b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:28.482 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:28.480 7fa92b2f0700 1 --2- 172.21.2.102:0/3083265598 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa904018730 0x7fa90401abc0 unknown :-1 s=CLOSED pgs=999 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:28.482 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:28.480 7fa92b2f0700 1 --2- 172.21.2.102:0/3083265598 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa9241b8180 0x7fa9241b8580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:28.482 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:28.480 7fa92b2f0700 1 --2- 172.21.2.102:0/3083265598 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa9241971e0 0x7fa92418eda0 unknown :-1 s=CLOSED pgs=2269 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:28.483 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:28.480 7fa92b2f0700 1 -- 172.21.2.102:0/3083265598 >> 172.21.2.102:0/3083265598 conn(0x7fa92419f8f0 msgr2=0x7fa92418c260 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:28.483 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:28.480 7fa92b2f0700 1 -- 172.21.2.102:0/3083265598 shutdown_connections 2021-02-10T11:32:28.483 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:28.480 7fa92b2f0700 1 -- 172.21.2.102:0/3083265598 wait complete. 2021-02-10T11:32:28.490 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2021-02-10T11:32:28.925 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:28.906 7fb5efaca700 1 Processor -- start 2021-02-10T11:32:28.925 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:28.906 7fb5efaca700 1 -- start start 2021-02-10T11:32:28.925 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:28.906 7fb5efaca700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fb5e80d1870 0x7fb5e81b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:28.925 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:28.906 7fb5efaca700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5e80f0b10 con 0x7fb5e80f0650 2021-02-10T11:32:28.926 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:28.906 7fb5efaca700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5e80d0310 con 0x7fb5e81b0a10 2021-02-10T11:32:28.926 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:28.906 7fb5efaca700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5e80d0530 con 0x7fb5e80d1870 2021-02-10T11:32:28.926 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:28.906 7fb5ed8e6700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fb5e80d1870 0x7fb5e81b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:28.926 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:28.906 7fb5ed8e6700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fb5e80d1870 0x7fb5e81b03f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:34468/0 (socket says 172.21.2.102:34468) 2021-02-10T11:32:28.926 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:28.906 7fb5ed8e6700 1 -- 172.21.2.102:0/911707827 learned_addr learned my addr 172.21.2.102:0/911707827 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:32:28.926 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:28.907 7fb5ed8e6700 1 -- 172.21.2.102:0/911707827 >> v1:172.21.2.113:6789/0 conn(0x7fb5e81b0a10 legacy=0x7fb5e80cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:28.926 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:28.907 7fb5ed8e6700 1 -- 172.21.2.102:0/911707827 >> v1:172.21.2.102:6789/0 conn(0x7fb5e80f0650 legacy=0x7fb5e80cfb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:28.926 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:28.907 7fb5ed8e6700 1 -- 172.21.2.102:0/911707827 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5e80d1c70 con 0x7fb5e80d1870 2021-02-10T11:32:28.926 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:28.907 7fb5ed8e6700 1 --2- 172.21.2.102:0/911707827 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb5e80d1870 0x7fb5e81b03f0 secure :-1 s=READY pgs=2271 cs=0 l=1 rev1=1 rx=0x7fb5e0006a80 tx=0x7fb5e0009e80).ready entity=mon.0 client_cookie=68676f879bfa4701 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:28.926 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:28.907 7fb5ec8e4700 1 -- 172.21.2.102:0/911707827 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb5e000b510 con 0x7fb5e80d1870 2021-02-10T11:32:28.926 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:28.907 7fb5ec8e4700 1 -- 172.21.2.102:0/911707827 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5e000b7d0 con 0x7fb5e80d1870 2021-02-10T11:32:28.926 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:28.907 7fb5ec8e4700 1 -- 172.21.2.102:0/911707827 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb5e000bc20 con 0x7fb5e80d1870 2021-02-10T11:32:28.926 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:28.907 7fb5efaca700 1 -- 172.21.2.102:0/911707827 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb5e80d1870 msgr2=0x7fb5e81b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:28.927 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:28.907 7fb5efaca700 1 --2- 172.21.2.102:0/911707827 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb5e80d1870 0x7fb5e81b03f0 secure :-1 s=READY pgs=2271 cs=0 l=1 rev1=1 rx=0x7fb5e0006a80 tx=0x7fb5e0009e80).stop 2021-02-10T11:32:28.927 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:28.908 7fb5efaca700 1 -- 172.21.2.102:0/911707827 shutdown_connections 2021-02-10T11:32:28.927 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:28.908 7fb5efaca700 1 --2- 172.21.2.102:0/911707827 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb5e80d1870 0x7fb5e81b03f0 unknown :-1 s=CLOSED pgs=2271 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:28.927 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:28.908 7fb5efaca700 1 -- 172.21.2.102:0/911707827 >> 172.21.2.102:0/911707827 conn(0x7fb5e80ce450 msgr2=0x7fb5e80d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:28.927 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:28.908 7fb5efaca700 1 -- 172.21.2.102:0/911707827 shutdown_connections 2021-02-10T11:32:28.927 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:28.908 7fb5efaca700 1 -- 172.21.2.102:0/911707827 wait complete. 2021-02-10T11:32:28.927 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:28.924 7fb5efaca700 1 Processor -- start 2021-02-10T11:32:28.927 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:28.924 7fb5efaca700 1 -- start start 2021-02-10T11:32:28.929 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:28.924 7fb5efaca700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb5e80e92a0 0x7fb5e81c63b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:28.929 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:28.924 7fb5efaca700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb5e81c69d0 0x7fb5e80c2e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:28.929 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:28.924 7fb5efaca700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb5e80c3450 0x7fb5e8096f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:28.929 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:28.924 7fb5efaca700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5e80e96a0 con 0x7fb5e80e92a0 2021-02-10T11:32:28.929 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:28.924 7fb5efaca700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5e80c3850 con 0x7fb5e80c3450 2021-02-10T11:32:28.929 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:28.924 7fb5efaca700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5e8097540 con 0x7fb5e81c69d0 2021-02-10T11:32:28.929 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:28.924 7fb5ed8e6700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb5e80e92a0 0x7fb5e81c63b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:28.929 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:28.924 7fb5ed8e6700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb5e80e92a0 0x7fb5e81c63b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:34474/0 (socket says 172.21.2.102:34474) 2021-02-10T11:32:28.929 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:28.924 7fb5ed8e6700 1 -- 172.21.2.102:0/3843124079 learned_addr learned my addr 172.21.2.102:0/3843124079 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:32:28.929 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:28.925 7fb5ee0e7700 1 --2- 172.21.2.102:0/3843124079 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb5e80c3450 0x7fb5e8096f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:28.929 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:28.925 7fb5ed0e5700 1 --2- 172.21.2.102:0/3843124079 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb5e81c69d0 0x7fb5e80c2e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:28.929 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:28.925 7fb5ed8e6700 1 -- 172.21.2.102:0/3843124079 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb5e81c69d0 msgr2=0x7fb5e80c2e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:28.929 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:28.925 7fb5ed8e6700 1 --2- 172.21.2.102:0/3843124079 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb5e81c69d0 0x7fb5e80c2e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:28.930 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:28.925 7fb5ed8e6700 1 -- 172.21.2.102:0/3843124079 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb5e80c3450 msgr2=0x7fb5e8096f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:28.930 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:28.925 7fb5ed8e6700 1 --2- 172.21.2.102:0/3843124079 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb5e80c3450 0x7fb5e8096f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:28.930 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:28.925 7fb5ed8e6700 1 -- 172.21.2.102:0/3843124079 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5e8097760 con 0x7fb5e80e92a0 2021-02-10T11:32:28.930 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:28.925 7fb5ee0e7700 1 --2- 172.21.2.102:0/3843124079 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb5e80c3450 0x7fb5e8096f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:32:28.930 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:28.925 7fb5ed8e6700 1 --2- 172.21.2.102:0/3843124079 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb5e80e92a0 0x7fb5e81c63b0 secure :-1 s=READY pgs=2272 cs=0 l=1 rev1=1 rx=0x7fb5e000b970 tx=0x7fb5e000bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:28.930 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:28.925 7fb5de7fc700 1 -- 172.21.2.102:0/3843124079 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb5e0000d30 con 0x7fb5e80e92a0 2021-02-10T11:32:28.930 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:28.925 7fb5de7fc700 1 -- 172.21.2.102:0/3843124079 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5e0006240 con 0x7fb5e80e92a0 2021-02-10T11:32:28.930 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:28.925 7fb5de7fc700 1 -- 172.21.2.102:0/3843124079 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb5e0005660 con 0x7fb5e80e92a0 2021-02-10T11:32:28.930 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:28.925 7fb5efaca700 1 -- 172.21.2.102:0/3843124079 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5e8097760 con 0x7fb5e80e92a0 2021-02-10T11:32:28.930 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:28.925 7fb5efaca700 1 -- 172.21.2.102:0/3843124079 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5e8097760 con 0x7fb5e80e92a0 2021-02-10T11:32:28.930 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:28.926 7fb5de7fc700 1 -- 172.21.2.102:0/3843124079 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fb5e0006420 con 0x7fb5e80e92a0 2021-02-10T11:32:28.931 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:28.926 7fb5de7fc700 1 --2- 172.21.2.102:0/3843124079 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb5d00186c0 0x7fb5d001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:28.931 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:28.926 7fb5de7fc700 1 -- 172.21.2.102:0/3843124079 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5659+0+0 (secure 0 0 0) 0x7fb5e0032b60 con 0x7fb5e80e92a0 2021-02-10T11:32:28.931 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:28.926 7fb5ed0e5700 1 --2- 172.21.2.102:0/3843124079 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb5d00186c0 0x7fb5d001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:28.931 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:28.926 7fb5ed0e5700 1 --2- 172.21.2.102:0/3843124079 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb5d00186c0 0x7fb5d001abc0 crc :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:28.957 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:28.956 7fb5efaca700 1 -- 172.21.2.102:0/3843124079 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5e80e7db0 con 0x7fb5e80e92a0 2021-02-10T11:32:28.959 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:28.958 7fb5de7fc700 1 -- 172.21.2.102:0/3843124079 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fb5e0006030 con 0x7fb5e80e92a0 2021-02-10T11:32:29.089 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:29.088 7fb5efaca700 1 -- 172.21.2.102:0/3843124079 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fb5e80d1ee0 con 0x7fb5e80e92a0 2021-02-10T11:32:29.881 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:29.879 7fb5de7fc700 1 -- 172.21.2.102:0/3843124079 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v94) v1 ==== 104+0+0 (secure 0 0 0) 0x7fb5e0006e80 con 0x7fb5e80e92a0 2021-02-10T11:32:29.911 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:29.910 7fb5efaca700 1 -- 172.21.2.102:0/3843124079 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb5d00186c0 msgr2=0x7fb5d001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:29.912 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:29.910 7fb5efaca700 1 --2- 172.21.2.102:0/3843124079 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb5d00186c0 0x7fb5d001abc0 crc :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:29.912 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:29.910 7fb5efaca700 1 -- 172.21.2.102:0/3843124079 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb5e80e92a0 msgr2=0x7fb5e81c63b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:29.912 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:29.910 7fb5efaca700 1 --2- 172.21.2.102:0/3843124079 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb5e80e92a0 0x7fb5e81c63b0 secure :-1 s=READY pgs=2272 cs=0 l=1 rev1=1 rx=0x7fb5e000b970 tx=0x7fb5e000bd10).stop 2021-02-10T11:32:29.912 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:29.910 7fb5efaca700 1 -- 172.21.2.102:0/3843124079 shutdown_connections 2021-02-10T11:32:29.912 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:29.910 7fb5efaca700 1 --2- 172.21.2.102:0/3843124079 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb5d00186c0 0x7fb5d001abc0 unknown :-1 s=CLOSED pgs=1000 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:29.912 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:29.910 7fb5efaca700 1 --2- 172.21.2.102:0/3843124079 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb5e80c3450 0x7fb5e8096f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:29.912 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:29.910 7fb5efaca700 1 --2- 172.21.2.102:0/3843124079 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb5e81c69d0 0x7fb5e80c2e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:29.912 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:29.910 7fb5efaca700 1 --2- 172.21.2.102:0/3843124079 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb5e80e92a0 0x7fb5e81c63b0 unknown :-1 s=CLOSED pgs=2272 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:29.913 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:29.910 7fb5efaca700 1 -- 172.21.2.102:0/3843124079 >> 172.21.2.102:0/3843124079 conn(0x7fb5e80bf100 msgr2=0x7fb5e80bd7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:29.913 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:29.910 7fb5efaca700 1 -- 172.21.2.102:0/3843124079 shutdown_connections 2021-02-10T11:32:29.913 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:29.910 7fb5efaca700 1 -- 172.21.2.102:0/3843124079 wait complete. 2021-02-10T11:32:29.920 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool delete cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2021-02-10T11:32:30.352 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:30.333 7fbecd38a700 1 Processor -- start 2021-02-10T11:32:30.352 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:30.333 7fbecd38a700 1 -- start start 2021-02-10T11:32:30.352 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:30.334 7fbecd38a700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fbec810b000 0x7fbec80cf760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:30.352 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:30.334 7fbecd38a700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbec80fe1d0 con 0x7fbec80d1880 2021-02-10T11:32:30.352 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:30.334 7fbecd38a700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbec81b0cc0 con 0x7fbec80d1c80 2021-02-10T11:32:30.352 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:30.334 7fbecd38a700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbec8109a70 con 0x7fbec810b000 2021-02-10T11:32:30.352 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:30.334 7fbec787e700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fbec810b000 0x7fbec80cf760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:30.352 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:30.334 7fbec787e700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fbec810b000 0x7fbec80cf760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:34486/0 (socket says 172.21.2.102:34486) 2021-02-10T11:32:30.352 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:30.334 7fbec787e700 1 -- 172.21.2.102:0/2966280854 learned_addr learned my addr 172.21.2.102:0/2966280854 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:32:30.353 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:30.334 7fbec707d700 1 --1- 172.21.2.102:0/2966280854 >> v1:172.21.2.102:6789/0 conn(0x7fbec80d1880 0x7fbec81b04d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:54938/0 (socket says 172.21.2.102:54938) 2021-02-10T11:32:30.353 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:30.334 7fbec787e700 1 -- 172.21.2.102:0/2966280854 >> v1:172.21.2.113:6789/0 conn(0x7fbec80d1c80 legacy=0x7fbec80cef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:32:30.353 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:30.335 7fbec787e700 1 -- 172.21.2.102:0/2966280854 >> v1:172.21.2.102:6789/0 conn(0x7fbec80d1880 legacy=0x7fbec81b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:30.353 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:30.335 7fbec787e700 1 -- 172.21.2.102:0/2966280854 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbec810b400 con 0x7fbec810b000 2021-02-10T11:32:30.353 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:30.335 7fbec787e700 1 --2- 172.21.2.102:0/2966280854 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbec810b000 0x7fbec80cf760 secure :-1 s=READY pgs=2274 cs=0 l=1 rev1=1 rx=0x7fbec0006f00 tx=0x7fbec0009f10).ready entity=mon.0 client_cookie=17b761a131f8372a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:30.353 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:30.335 7fbec607b700 1 -- 172.21.2.102:0/2966280854 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbec000b3b0 con 0x7fbec810b000 2021-02-10T11:32:30.353 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:30.335 7fbec607b700 1 -- 172.21.2.102:0/2966280854 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbec000b630 con 0x7fbec810b000 2021-02-10T11:32:30.353 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:30.335 7fbec607b700 1 -- 172.21.2.102:0/2966280854 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbec000bb50 con 0x7fbec810b000 2021-02-10T11:32:30.353 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:30.335 7fbecd38a700 1 -- 172.21.2.102:0/2966280854 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbec810b000 msgr2=0x7fbec80cf760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:30.353 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:30.335 7fbecd38a700 1 --2- 172.21.2.102:0/2966280854 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbec810b000 0x7fbec80cf760 secure :-1 s=READY pgs=2274 cs=0 l=1 rev1=1 rx=0x7fbec0006f00 tx=0x7fbec0009f10).stop 2021-02-10T11:32:30.353 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:30.336 7fbecd38a700 1 -- 172.21.2.102:0/2966280854 shutdown_connections 2021-02-10T11:32:30.353 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:30.336 7fbecd38a700 1 --2- 172.21.2.102:0/2966280854 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbec810b000 0x7fbec80cf760 unknown :-1 s=CLOSED pgs=2274 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:30.353 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:30.336 7fbecd38a700 1 -- 172.21.2.102:0/2966280854 >> 172.21.2.102:0/2966280854 conn(0x7fbec80ce430 msgr2=0x7fbec80d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:30.354 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:30.336 7fbecd38a700 1 -- 172.21.2.102:0/2966280854 shutdown_connections 2021-02-10T11:32:30.354 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:30.336 7fbecd38a700 1 -- 172.21.2.102:0/2966280854 wait complete. 2021-02-10T11:32:30.354 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:30.350 7fbecd38a700 1 Processor -- start 2021-02-10T11:32:30.354 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:30.351 7fbecd38a700 1 -- start start 2021-02-10T11:32:30.355 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:30.351 7fbecd38a700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbec8097820 0x7fbec8097c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:30.355 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:30.351 7fbecd38a700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fbec808ac80 0x7fbec808b080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:30.355 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:30.351 7fbecd38a700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbec808bf00 0x7fbec808c300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:30.355 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:30.351 7fbecd38a700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbec808b6a0 con 0x7fbec8097820 2021-02-10T11:32:30.355 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:30.351 7fbecd38a700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbec809a360 con 0x7fbec808ac80 2021-02-10T11:32:30.355 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:30.351 7fbecd38a700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbec809a580 con 0x7fbec808bf00 2021-02-10T11:32:30.356 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:30.351 7fbec707d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbec8097820 0x7fbec8097c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:30.356 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:30.351 7fbec707d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbec8097820 0x7fbec8097c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:34490/0 (socket says 172.21.2.102:34490) 2021-02-10T11:32:30.356 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:30.351 7fbec707d700 1 -- 172.21.2.102:0/1876892945 learned_addr learned my addr 172.21.2.102:0/1876892945 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:32:30.356 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:30.351 7fbec707d700 1 -- 172.21.2.102:0/1876892945 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbec808bf00 msgr2=0x7fbec808c300 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:32:30.356 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:30.351 7fbec707d700 1 --2- 172.21.2.102:0/1876892945 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbec808bf00 0x7fbec808c300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:30.356 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:30.351 7fbec707d700 1 -- 172.21.2.102:0/1876892945 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fbec808ac80 msgr2=0x7fbec808b080 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:32:30.356 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:30.351 7fbec707d700 1 --2- 172.21.2.102:0/1876892945 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fbec808ac80 0x7fbec808b080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:30.356 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:30.351 7fbec707d700 1 -- 172.21.2.102:0/1876892945 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbec808c920 con 0x7fbec8097820 2021-02-10T11:32:30.357 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:30.352 7fbec707d700 1 --2- 172.21.2.102:0/1876892945 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbec8097820 0x7fbec8097c20 secure :-1 s=READY pgs=2275 cs=0 l=1 rev1=1 rx=0x7fbeb000a590 tx=0x7fbeb000a930).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:30.357 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:30.352 7fbebffff700 1 -- 172.21.2.102:0/1876892945 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbeb0000ef0 con 0x7fbec8097820 2021-02-10T11:32:30.357 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:30.352 7fbebffff700 1 -- 172.21.2.102:0/1876892945 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbeb000ad20 con 0x7fbec8097820 2021-02-10T11:32:30.357 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:30.352 7fbebffff700 1 -- 172.21.2.102:0/1876892945 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbeb0004620 con 0x7fbec8097820 2021-02-10T11:32:30.357 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:30.352 7fbecd38a700 1 -- 172.21.2.102:0/1876892945 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbec808c920 con 0x7fbec8097820 2021-02-10T11:32:30.357 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:30.352 7fbecd38a700 1 -- 172.21.2.102:0/1876892945 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbec808ceb0 con 0x7fbec8097820 2021-02-10T11:32:30.358 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:30.353 7fbebffff700 1 -- 172.21.2.102:0/1876892945 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fbeb00043a0 con 0x7fbec8097820 2021-02-10T11:32:30.358 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:30.353 7fbebffff700 1 --2- 172.21.2.102:0/1876892945 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fbeac0186c0 0x7fbeac01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:30.358 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:30.353 7fbebffff700 1 -- 172.21.2.102:0/1876892945 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5659+0+0 (secure 0 0 0) 0x7fbeb0032600 con 0x7fbec8097820 2021-02-10T11:32:30.358 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:30.353 7fbec687c700 1 --2- 172.21.2.102:0/1876892945 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fbeac0186c0 0x7fbeac01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:30.358 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:30.353 7fbec687c700 1 --2- 172.21.2.102:0/1876892945 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fbeac0186c0 0x7fbeac01abc0 crc :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:30.386 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:30.384 7fbecd38a700 1 -- 172.21.2.102:0/1876892945 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbec8080340 con 0x7fbec8097820 2021-02-10T11:32:30.388 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:30.387 7fbebffff700 1 -- 172.21.2.102:0/1876892945 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fbeb0000ef0 con 0x7fbec8097820 2021-02-10T11:32:30.515 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:30.514 7fbecd38a700 1 -- 172.21.2.102:0/1876892945 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"pool2": "cephfs_metadata", "prefix": "osd pool delete", "pool": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fbec8088070 con 0x7fbec8097820 2021-02-10T11:32:30.885 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:30.883 7fbebffff700 1 -- 172.21.2.102:0/1876892945 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"pool2": "cephfs_metadata", "prefix": "osd pool delete", "pool": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v114) v1 ==== 185+0+0 (secure 0 0 0) 0x7fbeb00043a0 con 0x7fbec8097820 2021-02-10T11:32:30.931 INFO:teuthology.orchestra.run.gibba002.stderr:pool 'cephfs_metadata' removed 2021-02-10T11:32:30.933 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:30.931 7fbecd38a700 1 -- 172.21.2.102:0/1876892945 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fbeac0186c0 msgr2=0x7fbeac01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:30.933 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:30.931 7fbecd38a700 1 --2- 172.21.2.102:0/1876892945 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fbeac0186c0 0x7fbeac01abc0 crc :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:30.933 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:30.931 7fbecd38a700 1 -- 172.21.2.102:0/1876892945 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbec8097820 msgr2=0x7fbec8097c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:30.933 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:30.931 7fbecd38a700 1 --2- 172.21.2.102:0/1876892945 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbec8097820 0x7fbec8097c20 secure :-1 s=READY pgs=2275 cs=0 l=1 rev1=1 rx=0x7fbeb000a590 tx=0x7fbeb000a930).stop 2021-02-10T11:32:30.934 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:30.931 7fbecd38a700 1 -- 172.21.2.102:0/1876892945 shutdown_connections 2021-02-10T11:32:30.934 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:30.931 7fbecd38a700 1 --2- 172.21.2.102:0/1876892945 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbec808bf00 0x7fbec808c300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:30.934 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:30.931 7fbecd38a700 1 --2- 172.21.2.102:0/1876892945 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fbeac0186c0 0x7fbeac01abc0 unknown :-1 s=CLOSED pgs=1001 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:30.934 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:30.931 7fbecd38a700 1 --2- 172.21.2.102:0/1876892945 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fbec808ac80 0x7fbec808b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:30.934 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:30.931 7fbecd38a700 1 --2- 172.21.2.102:0/1876892945 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbec8097820 0x7fbec8097c20 unknown :-1 s=CLOSED pgs=2275 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:30.934 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:30.931 7fbecd38a700 1 -- 172.21.2.102:0/1876892945 >> 172.21.2.102:0/1876892945 conn(0x7fbec809fb50 msgr2=0x7fbec80fed60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:30.934 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:30.932 7fbecd38a700 1 -- 172.21.2.102:0/1876892945 shutdown_connections 2021-02-10T11:32:30.935 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:30.932 7fbecd38a700 1 -- 172.21.2.102:0/1876892945 wait complete. 2021-02-10T11:32:30.941 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool delete cephfs_data cephfs_data --yes-i-really-really-mean-it 2021-02-10T11:32:31.373 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:31.355 7fe16f222700 1 Processor -- start 2021-02-10T11:32:31.373 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:31.355 7fe16f222700 1 -- start start 2021-02-10T11:32:31.373 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:31.355 7fe16f222700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe1680d1880 0x7fe1681b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:31.373 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:31.355 7fe16f222700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fe1681b0af0 0x7fe168108950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:31.374 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:31.355 7fe16f222700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe168109a70 con 0x7fe1680d1c80 2021-02-10T11:32:31.374 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:31.355 7fe16f222700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1680e5f00 con 0x7fe1681b0af0 2021-02-10T11:32:31.374 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:31.356 7fe16f222700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe168109760 con 0x7fe1680d1880 2021-02-10T11:32:31.374 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:31.356 7fe16c83d700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fe1681b0af0 0x7fe168108950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:31.374 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:31.356 7fe16c83d700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fe1681b0af0 0x7fe168108950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:34496/0 (socket says 172.21.2.102:34496) 2021-02-10T11:32:31.374 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:31.356 7fe16c83d700 1 -- 172.21.2.102:0/828438839 learned_addr learned my addr 172.21.2.102:0/828438839 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:32:31.374 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:31.356 7fe16c83d700 1 -- 172.21.2.102:0/828438839 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe1680d1880 msgr2=0x7fe1681b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:32:31.374 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:31.356 7fe16d03e700 1 --2- 172.21.2.102:0/828438839 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe1680d1880 0x7fe1681b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:31.374 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:31.356 7fe16c83d700 1 --2- 172.21.2.102:0/828438839 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe1680d1880 0x7fe1681b04d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:31.374 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:31.356 7fe16c83d700 1 -- 172.21.2.102:0/828438839 >> v1:172.21.2.113:6789/0 conn(0x7fe1680d1c80 legacy=0x7fe168108f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:32:31.374 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:31.356 7fe16c83d700 1 -- 172.21.2.102:0/828438839 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe168088070 con 0x7fe1681b0af0 2021-02-10T11:32:31.374 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:31.356 7fe16d03e700 1 --2- 172.21.2.102:0/828438839 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe1680d1880 0x7fe1681b04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:32:31.375 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:31.357 7fe16c83d700 1 --2- 172.21.2.102:0/828438839 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe1681b0af0 0x7fe168108950 secure :-1 s=READY pgs=2276 cs=0 l=1 rev1=1 rx=0x7fe158006700 tx=0x7fe158009b80).ready entity=mon.0 client_cookie=e4ee47431b5b9f54 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:31.375 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:31.357 7fe15ffff700 1 -- 172.21.2.102:0/828438839 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe15800c0c0 con 0x7fe1681b0af0 2021-02-10T11:32:31.375 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:31.357 7fe15ffff700 1 -- 172.21.2.102:0/828438839 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe15800c440 con 0x7fe1681b0af0 2021-02-10T11:32:31.375 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:31.357 7fe15ffff700 1 -- 172.21.2.102:0/828438839 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe15800c960 con 0x7fe1681b0af0 2021-02-10T11:32:31.375 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:31.357 7fe16f222700 1 -- 172.21.2.102:0/828438839 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe1681b0af0 msgr2=0x7fe168108950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:31.375 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:31.357 7fe16f222700 1 --2- 172.21.2.102:0/828438839 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe1681b0af0 0x7fe168108950 secure :-1 s=READY pgs=2276 cs=0 l=1 rev1=1 rx=0x7fe158006700 tx=0x7fe158009b80).stop 2021-02-10T11:32:31.375 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:31.357 7fe16f222700 1 -- 172.21.2.102:0/828438839 shutdown_connections 2021-02-10T11:32:31.375 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:31.357 7fe16f222700 1 --2- 172.21.2.102:0/828438839 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe1681b0af0 0x7fe168108950 unknown :-1 s=CLOSED pgs=2276 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:31.375 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:31.357 7fe16f222700 1 --2- 172.21.2.102:0/828438839 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe1680d1880 0x7fe1681b04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:31.375 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:31.357 7fe16f222700 1 -- 172.21.2.102:0/828438839 >> 172.21.2.102:0/828438839 conn(0x7fe1680ce430 msgr2=0x7fe1680d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:31.375 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:31.358 7fe16f222700 1 -- 172.21.2.102:0/828438839 shutdown_connections 2021-02-10T11:32:31.375 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:31.358 7fe16f222700 1 -- 172.21.2.102:0/828438839 wait complete. 2021-02-10T11:32:31.377 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:31.372 7fe16f222700 1 Processor -- start 2021-02-10T11:32:31.377 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:31.372 7fe16f222700 1 -- start start 2021-02-10T11:32:31.377 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:31.373 7fe16f222700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe1680aee20 0x7fe1681b5370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:31.377 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:31.373 7fe16f222700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe1681b5990 0x7fe1681a85b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:31.377 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:31.373 7fe16f222700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe1681a8bd0 0x7fe1681b4160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:31.377 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:31.373 7fe16f222700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1681b73f0 con 0x7fe1681a8bd0 2021-02-10T11:32:31.377 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:31.373 7fe16f222700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1681b4780 con 0x7fe1681b5990 2021-02-10T11:32:31.377 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:31.373 7fe16f222700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1681b4960 con 0x7fe1680aee20 2021-02-10T11:32:31.377 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:31.373 7fe16d83f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe1681a8bd0 0x7fe1681b4160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:31.378 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:31.373 7fe16d83f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe1681a8bd0 0x7fe1681b4160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:34502/0 (socket says 172.21.2.102:34502) 2021-02-10T11:32:31.378 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:31.373 7fe16d83f700 1 -- 172.21.2.102:0/2899669040 learned_addr learned my addr 172.21.2.102:0/2899669040 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:32:31.378 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:31.373 7fe16d83f700 1 -- 172.21.2.102:0/2899669040 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe1680aee20 msgr2=0x7fe1681b5370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:31.378 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:31.373 7fe16d83f700 1 --2- 172.21.2.102:0/2899669040 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe1680aee20 0x7fe1681b5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:31.378 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:31.373 7fe16d83f700 1 -- 172.21.2.102:0/2899669040 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe1681b5990 msgr2=0x7fe1681a85b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:31.378 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:31.373 7fe16d83f700 1 --2- 172.21.2.102:0/2899669040 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe1681b5990 0x7fe1681a85b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:31.378 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:31.373 7fe16d83f700 1 -- 172.21.2.102:0/2899669040 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1681b4b80 con 0x7fe1681a8bd0 2021-02-10T11:32:31.378 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:31.373 7fe16d83f700 1 --2- 172.21.2.102:0/2899669040 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe1681a8bd0 0x7fe1681b4160 secure :-1 s=READY pgs=2277 cs=0 l=1 rev1=1 rx=0x7fe160009940 tx=0x7fe16000bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:31.378 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:31.374 7fe15dffb700 1 -- 172.21.2.102:0/2899669040 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe160000f90 con 0x7fe1681a8bd0 2021-02-10T11:32:31.378 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:31.374 7fe15dffb700 1 -- 172.21.2.102:0/2899669040 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe16000c2f0 con 0x7fe1681a8bd0 2021-02-10T11:32:31.378 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:31.374 7fe15dffb700 1 -- 172.21.2.102:0/2899669040 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe16000c810 con 0x7fe1681a8bd0 2021-02-10T11:32:31.378 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:31.374 7fe16f222700 1 -- 172.21.2.102:0/2899669040 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1681b4b80 con 0x7fe1681a8bd0 2021-02-10T11:32:31.378 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:31.374 7fe16f222700 1 -- 172.21.2.102:0/2899669040 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1681b5f50 con 0x7fe1681a8bd0 2021-02-10T11:32:31.379 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:31.375 7fe15dffb700 1 -- 172.21.2.102:0/2899669040 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fe160001220 con 0x7fe1681a8bd0 2021-02-10T11:32:31.379 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:31.375 7fe15dffb700 1 --2- 172.21.2.102:0/2899669040 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe1480186d0 0x7fe14801abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:31.379 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:31.375 7fe15dffb700 1 -- 172.21.2.102:0/2899669040 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5221+0+0 (secure 0 0 0) 0x7fe160032a20 con 0x7fe1681a8bd0 2021-02-10T11:32:31.379 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:31.375 7fe16d03e700 1 --2- 172.21.2.102:0/2899669040 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe1480186d0 0x7fe14801abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:31.379 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:31.375 7fe16d03e700 1 --2- 172.21.2.102:0/2899669040 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe1480186d0 0x7fe14801abd0 crc :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:31.407 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:31.405 7fe16f222700 1 -- 172.21.2.102:0/2899669040 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe168082a30 con 0x7fe1681a8bd0 2021-02-10T11:32:31.409 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:31.408 7fe15dffb700 1 -- 172.21.2.102:0/2899669040 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fe160002ef0 con 0x7fe1681a8bd0 2021-02-10T11:32:31.538 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:31.536 7fe16f222700 1 -- 172.21.2.102:0/2899669040 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"pool2": "cephfs_data", "prefix": "osd pool delete", "pool": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fe168109a70 con 0x7fe1681a8bd0 2021-02-10T11:32:31.888 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:31.887 7fe15dffb700 1 -- 172.21.2.102:0/2899669040 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"pool2": "cephfs_data", "prefix": "osd pool delete", "pool": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v115) v1 ==== 173+0+0 (secure 0 0 0) 0x7fe160000f40 con 0x7fe1681a8bd0 2021-02-10T11:32:31.904 INFO:teuthology.orchestra.run.gibba002.stderr:pool 'cephfs_data' removed 2021-02-10T11:32:31.906 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:31.905 7fe16f222700 1 -- 172.21.2.102:0/2899669040 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe1480186d0 msgr2=0x7fe14801abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:31.906 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:31.905 7fe16f222700 1 --2- 172.21.2.102:0/2899669040 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe1480186d0 0x7fe14801abd0 crc :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:31.906 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:31.905 7fe16f222700 1 -- 172.21.2.102:0/2899669040 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe1681a8bd0 msgr2=0x7fe1681b4160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:31.906 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:31.905 7fe16f222700 1 --2- 172.21.2.102:0/2899669040 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe1681a8bd0 0x7fe1681b4160 secure :-1 s=READY pgs=2277 cs=0 l=1 rev1=1 rx=0x7fe160009940 tx=0x7fe16000bcf0).stop 2021-02-10T11:32:31.907 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:31.905 7fe16f222700 1 -- 172.21.2.102:0/2899669040 shutdown_connections 2021-02-10T11:32:31.907 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:31.905 7fe16f222700 1 --2- 172.21.2.102:0/2899669040 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe1681a8bd0 0x7fe1681b4160 unknown :-1 s=CLOSED pgs=2277 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:31.907 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:31.905 7fe16f222700 1 --2- 172.21.2.102:0/2899669040 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe1480186d0 0x7fe14801abd0 unknown :-1 s=CLOSED pgs=1002 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:31.907 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:31.905 7fe16f222700 1 --2- 172.21.2.102:0/2899669040 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe1681b5990 0x7fe1681a85b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:31.907 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:31.905 7fe16f222700 1 --2- 172.21.2.102:0/2899669040 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe1680aee20 0x7fe1681b5370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:31.907 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:31.905 7fe16f222700 1 -- 172.21.2.102:0/2899669040 >> 172.21.2.102:0/2899669040 conn(0x7fe16819e990 msgr2=0x7fe1681918e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:31.907 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:31.905 7fe16f222700 1 -- 172.21.2.102:0/2899669040 shutdown_connections 2021-02-10T11:32:31.907 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:31.905 7fe16f222700 1 -- 172.21.2.102:0/2899669040 wait complete. 2021-02-10T11:32:31.915 INFO:tasks.ceph.mds.a:Restarting daemon 2021-02-10T11:32:31.915 INFO:tasks.ceph.mds.a:Stopping old one... 2021-02-10T11:32:31.915 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2021-02-10T11:32:31.915 INFO:teuthology.orchestra.run:waiting for 300 2021-02-10T11:32:31.916 INFO:tasks.ceph.mds.b:Restarting daemon 2021-02-10T11:32:31.916 INFO:tasks.ceph.mds.b:Stopping old one... 2021-02-10T11:32:31.916 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2021-02-10T11:32:31.916 INFO:teuthology.orchestra.run:waiting for 300 2021-02-10T11:32:31.922 INFO:tasks.ceph.mds.a:Stopped 2021-02-10T11:32:31.922 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2021-02-10T11:32:31.923 INFO:tasks.ceph.mds.b:Stopped 2021-02-10T11:32:31.923 DEBUG:teuthology.orchestra.run.gibba013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2021-02-10T11:32:31.965 INFO:tasks.ceph.mds.a:Started 2021-02-10T11:32:31.965 DEBUG:teuthology.parallel:result is None 2021-02-10T11:32:31.966 INFO:tasks.ceph.mds.b:Started 2021-02-10T11:32:31.967 DEBUG:teuthology.parallel:result is None 2021-02-10T11:32:31.967 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd blacklist clear 2021-02-10T11:32:32.371 INFO:tasks.ceph.mds.b.gibba013.stdout:starting mds.b at 2021-02-10T11:32:32.373 INFO:tasks.ceph.mds.a.gibba002.stdout:starting mds.a at 2021-02-10T11:32:32.397 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:32.378 7f7d0a48f700 1 Processor -- start 2021-02-10T11:32:32.397 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:32.378 7f7d0a48f700 1 -- start start 2021-02-10T11:32:32.397 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:32.379 7f7d0a48f700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f7d040d0330 0x7f7d040ef630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:32.397 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:32.379 7f7d0a48f700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7d040d1c60 0x7f7d040efc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:32.397 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:32.379 7f7d0a48f700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d040f0b00 con 0x7f7d040d1860 2021-02-10T11:32:32.397 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:32.379 7f7d0a48f700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d040e5ec0 con 0x7f7d040d0330 2021-02-10T11:32:32.397 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:32.379 7f7d0a48f700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d041b0bd0 con 0x7f7d040d1c60 2021-02-10T11:32:32.398 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:32.379 7f7d03fff700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f7d040d1860 0x7f7d041b03e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:54976/0 (socket says 172.21.2.102:54976) 2021-02-10T11:32:32.398 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:32.379 7f7d03fff700 1 -- 172.21.2.102:0/2359409564 learned_addr learned my addr 172.21.2.102:0/2359409564 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:32:32.398 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:32.379 7f7d037fe700 1 --2- 172.21.2.102:0/2359409564 >> v2:172.21.2.102:3300/0 conn(0x7f7d040d0330 0x7f7d040ef630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:32.398 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:32.379 7f7d08aac700 1 --2- 172.21.2.102:0/2359409564 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7d040d1c60 0x7f7d040efc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:32.398 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:32.379 7f7d037fe700 1 -- 172.21.2.102:0/2359409564 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7d040d1c60 msgr2=0x7f7d040efc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:32:32.398 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:32.379 7f7d037fe700 1 --2- 172.21.2.102:0/2359409564 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7d040d1c60 0x7f7d040efc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:32.398 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:32.379 7f7d037fe700 1 -- 172.21.2.102:0/2359409564 >> v1:172.21.2.102:6789/0 conn(0x7f7d040d1860 legacy=0x7f7d041b03e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:32.398 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:32.379 7f7d037fe700 1 -- 172.21.2.102:0/2359409564 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d040f2090 con 0x7f7d040d0330 2021-02-10T11:32:32.398 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:32.380 7f7d037fe700 1 --2- 172.21.2.102:0/2359409564 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7d040d0330 0x7f7d040ef630 secure :-1 s=READY pgs=2280 cs=0 l=1 rev1=1 rx=0x7f7cf8006650 tx=0x7f7cf8009a00).ready entity=mon.0 client_cookie=4aa6d570f376696 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:32.398 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:32.380 7f7d02ffd700 1 -- 172.21.2.102:0/2359409564 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7cf8009df0 con 0x7f7d040d0330 2021-02-10T11:32:32.398 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:32.380 7f7d02ffd700 1 -- 172.21.2.102:0/2359409564 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cf80009b0 con 0x7f7d040d0330 2021-02-10T11:32:32.398 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:32.380 7f7d02ffd700 1 -- 172.21.2.102:0/2359409564 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7cf800c2b0 con 0x7f7d040d0330 2021-02-10T11:32:32.398 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:32.380 7f7d0a48f700 1 -- 172.21.2.102:0/2359409564 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7d040d0330 msgr2=0x7f7d040ef630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:32.399 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:32.380 7f7d0a48f700 1 --2- 172.21.2.102:0/2359409564 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7d040d0330 0x7f7d040ef630 secure :-1 s=READY pgs=2280 cs=0 l=1 rev1=1 rx=0x7f7cf8006650 tx=0x7f7cf8009a00).stop 2021-02-10T11:32:32.399 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:32.381 7f7d0a48f700 1 -- 172.21.2.102:0/2359409564 shutdown_connections 2021-02-10T11:32:32.399 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:32.381 7f7d0a48f700 1 --2- 172.21.2.102:0/2359409564 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7d040d0330 0x7f7d040ef630 unknown :-1 s=CLOSED pgs=2280 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:32.399 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:32.381 7f7d0a48f700 1 --2- 172.21.2.102:0/2359409564 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7d040d1c60 0x7f7d040efc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:32.399 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:32.381 7f7d0a48f700 1 -- 172.21.2.102:0/2359409564 >> 172.21.2.102:0/2359409564 conn(0x7f7d040ce440 msgr2=0x7f7d040d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:32.399 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:32.381 7f7d0a48f700 1 -- 172.21.2.102:0/2359409564 shutdown_connections 2021-02-10T11:32:32.399 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:32.381 7f7d0a48f700 1 -- 172.21.2.102:0/2359409564 wait complete. 2021-02-10T11:32:32.399 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:32.395 7f7d0a48f700 1 Processor -- start 2021-02-10T11:32:32.399 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:32.396 7f7d0a48f700 1 -- start start 2021-02-10T11:32:32.400 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:32.396 7f7d0a48f700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7d040c2260 0x7f7d040bc050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:32.400 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:32.396 7f7d0a48f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7d040c2660 0x7f7d040bc670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:32.401 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:32.396 7f7d0a48f700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7d040bd350 0x7f7d040bd750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:32.401 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:32.396 7f7d0a48f700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d040ec090 con 0x7f7d040c2660 2021-02-10T11:32:32.401 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:32.396 7f7d0a48f700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d040c2de0 con 0x7f7d040bd350 2021-02-10T11:32:32.401 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:32.396 7f7d0a48f700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d040ebe40 con 0x7f7d040c2260 2021-02-10T11:32:32.401 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:32.396 7f7d037fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7d040c2660 0x7f7d040bc670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:32.401 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:32.396 7f7d037fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7d040c2660 0x7f7d040bc670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:34526/0 (socket says 172.21.2.102:34526) 2021-02-10T11:32:32.401 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:32.396 7f7d037fe700 1 -- 172.21.2.102:0/2663329952 learned_addr learned my addr 172.21.2.102:0/2663329952 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:32:32.401 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:32.396 7f7d037fe700 1 -- 172.21.2.102:0/2663329952 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7d040c2260 msgr2=0x7f7d040bc050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:32.401 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:32.396 7f7d037fe700 1 --2- 172.21.2.102:0/2663329952 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7d040c2260 0x7f7d040bc050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:32.401 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:32.396 7f7d037fe700 1 -- 172.21.2.102:0/2663329952 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7d040bd350 msgr2=0x7f7d040bd750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:32.401 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:32.396 7f7d037fe700 1 --2- 172.21.2.102:0/2663329952 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7d040bd350 0x7f7d040bd750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:32.401 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:32.396 7f7d037fe700 1 -- 172.21.2.102:0/2663329952 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d040bce00 con 0x7f7d040c2660 2021-02-10T11:32:32.401 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:32.397 7f7d037fe700 1 --2- 172.21.2.102:0/2663329952 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7d040c2660 0x7f7d040bc670 secure :-1 s=READY pgs=2281 cs=0 l=1 rev1=1 rx=0x7f7cf80096a0 tx=0x7f7cf800b9b0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:32.402 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:32.397 7f7d00ff9700 1 -- 172.21.2.102:0/2663329952 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7cf80093a0 con 0x7f7d040c2660 2021-02-10T11:32:32.402 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:32.397 7f7d00ff9700 1 -- 172.21.2.102:0/2663329952 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cf80067f0 con 0x7f7d040c2660 2021-02-10T11:32:32.402 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:32.397 7f7d0a48f700 1 -- 172.21.2.102:0/2663329952 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d040bce00 con 0x7f7d040c2660 2021-02-10T11:32:32.402 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:32.397 7f7d0a48f700 1 -- 172.21.2.102:0/2663329952 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d040bce00 con 0x7f7d040c2660 2021-02-10T11:32:32.402 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:32.397 7f7d00ff9700 1 -- 172.21.2.102:0/2663329952 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7cf80022b0 con 0x7f7d040c2660 2021-02-10T11:32:32.402 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:32.397 7f7d00ff9700 1 -- 172.21.2.102:0/2663329952 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f7cf8000c90 con 0x7f7d040c2660 2021-02-10T11:32:32.402 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:32.397 7f7d00ff9700 1 --2- 172.21.2.102:0/2663329952 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7ce40186c0 0x7f7ce401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:32.402 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:32.397 7f7d00ff9700 1 -- 172.21.2.102:0/2663329952 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 4839+0+0 (secure 0 0 0) 0x7f7cf8032290 con 0x7f7d040c2660 2021-02-10T11:32:32.402 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:32.398 7f7d03fff700 1 --2- 172.21.2.102:0/2663329952 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7ce40186c0 0x7f7ce401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:32.402 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:32.398 7f7d03fff700 1 --2- 172.21.2.102:0/2663329952 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7ce40186c0 0x7f7ce401abc0 crc :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:32.430 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:32.428 7f7d0a48f700 1 -- 172.21.2.102:0/2663329952 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d040ebbe0 con 0x7f7d040c2660 2021-02-10T11:32:32.432 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:32.431 7f7d00ff9700 1 -- 172.21.2.102:0/2663329952 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f7cf8030d20 con 0x7f7d040c2660 2021-02-10T11:32:32.574 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:32.573 7f7d0a48f700 1 -- 172.21.2.102:0/2663329952 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd blacklist clear"} v 0) v1 -- 0x7f7d040ebbe0 con 0x7f7d040c2660 2021-02-10T11:32:32.893 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:32.892 7f7d00ff9700 1 -- 172.21.2.102:0/2663329952 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd blacklist clear"}]=0 removed all blacklist entries v116) v1 ==== 97+0+0 (secure 0 0 0) 0x7f7cf8000c90 con 0x7f7d040c2660 2021-02-10T11:32:32.941 INFO:teuthology.orchestra.run.gibba002.stderr: removed all blacklist entries 2021-02-10T11:32:32.943 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:32.941 7f7d0a48f700 1 -- 172.21.2.102:0/2663329952 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7ce40186c0 msgr2=0x7f7ce401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:32.943 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:32.941 7f7d0a48f700 1 --2- 172.21.2.102:0/2663329952 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7ce40186c0 0x7f7ce401abc0 crc :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:32.943 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:32.941 7f7d0a48f700 1 -- 172.21.2.102:0/2663329952 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7d040c2660 msgr2=0x7f7d040bc670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:32.943 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:32.941 7f7d0a48f700 1 --2- 172.21.2.102:0/2663329952 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7d040c2660 0x7f7d040bc670 secure :-1 s=READY pgs=2281 cs=0 l=1 rev1=1 rx=0x7f7cf80096a0 tx=0x7f7cf800b9b0).stop 2021-02-10T11:32:32.944 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:32.941 7f7d0a48f700 1 -- 172.21.2.102:0/2663329952 shutdown_connections 2021-02-10T11:32:32.944 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:32.941 7f7d0a48f700 1 --2- 172.21.2.102:0/2663329952 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7ce40186c0 0x7f7ce401abc0 unknown :-1 s=CLOSED pgs=1003 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:32.944 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:32.941 7f7d0a48f700 1 --2- 172.21.2.102:0/2663329952 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7d040bd350 0x7f7d040bd750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:32.944 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:32.941 7f7d0a48f700 1 --2- 172.21.2.102:0/2663329952 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7d040c2660 0x7f7d040bc670 unknown :-1 s=CLOSED pgs=2281 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:32.944 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:32.941 7f7d0a48f700 1 --2- 172.21.2.102:0/2663329952 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7d040c2260 0x7f7d040bc050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:32.944 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:32.941 7f7d0a48f700 1 -- 172.21.2.102:0/2663329952 >> 172.21.2.102:0/2663329952 conn(0x7f7d040a0290 msgr2=0x7f7d04094c40 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:32.944 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:32.942 7f7d0a48f700 1 -- 172.21.2.102:0/2663329952 shutdown_connections 2021-02-10T11:32:32.944 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:32.942 7f7d0a48f700 1 -- 172.21.2.102:0/2663329952 wait complete. 2021-02-10T11:32:32.952 INFO:tasks.cephfs.cephfs_test_case:['0', '1', '2', '3'] 2021-02-10T11:32:32.952 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth ls --format=json-pretty 2021-02-10T11:32:33.379 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:33.361 7f043e7ef700 1 Processor -- start 2021-02-10T11:32:33.380 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:33.361 7f043e7ef700 1 -- start start 2021-02-10T11:32:33.380 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:33.362 7f043e7ef700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f04380d1870 0x7f04381b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:33.380 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:33.362 7f043e7ef700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f04380f0650 0x7f04380cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:33.380 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:33.362 7f043e7ef700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04380f0b10 con 0x7f04381b0a10 2021-02-10T11:32:33.380 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:33.362 7f043e7ef700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04380fe230 con 0x7f04380f0650 2021-02-10T11:32:33.380 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:33.362 7f043e7ef700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04380e5ed0 con 0x7f04380d1870 2021-02-10T11:32:33.380 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:33.362 7f043ce0c700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f04380f0650 0x7f04380cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:33.380 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:33.362 7f043ce0c700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f04380f0650 0x7f04380cfb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:34534/0 (socket says 172.21.2.102:34534) 2021-02-10T11:32:33.380 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:33.362 7f04377fe700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f04381b0a10 0x7f04380cf330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:54990/0 (socket says 172.21.2.102:54990) 2021-02-10T11:32:33.380 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:33.362 7f04377fe700 1 -- 172.21.2.102:0/3859904272 learned_addr learned my addr 172.21.2.102:0/3859904272 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:32:33.380 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:33.363 7f043ce0c700 1 -- 172.21.2.102:0/3859904272 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f04380d1870 msgr2=0x7f04381b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:32:33.381 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:33.363 7f043ce0c700 1 --2- 172.21.2.102:0/3859904272 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f04380d1870 0x7f04381b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:33.381 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:33.363 7f043ce0c700 1 -- 172.21.2.102:0/3859904272 >> v1:172.21.2.102:6789/0 conn(0x7f04381b0a10 legacy=0x7f04380cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:33.381 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:33.363 7f0436ffd700 1 -- 172.21.2.102:0/3859904272 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 432141710 0 0) 0x7f042c001180 con 0x7f04381b0a10 2021-02-10T11:32:33.381 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:33.363 7f043ce0c700 1 -- 172.21.2.102:0/3859904272 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04380f20a0 con 0x7f04380f0650 2021-02-10T11:32:33.381 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:33.363 7f043ce0c700 1 --2- 172.21.2.102:0/3859904272 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f04380f0650 0x7f04380cfb20 secure :-1 s=READY pgs=2283 cs=0 l=1 rev1=1 rx=0x7f0430006e30 tx=0x7f0430009de0).ready entity=mon.0 client_cookie=5ccc9bda86e74272 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:33.381 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:33.363 7f0436ffd700 1 -- 172.21.2.102:0/3859904272 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f043000b300 con 0x7f04380f0650 2021-02-10T11:32:33.381 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:33.363 7f0436ffd700 1 -- 172.21.2.102:0/3859904272 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f043000b580 con 0x7f04380f0650 2021-02-10T11:32:33.381 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:33.363 7f0436ffd700 1 -- 172.21.2.102:0/3859904272 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f043000baa0 con 0x7f04380f0650 2021-02-10T11:32:33.381 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:33.363 7f043e7ef700 1 -- 172.21.2.102:0/3859904272 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f04380f0650 msgr2=0x7f04380cfb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:33.381 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:33.363 7f043e7ef700 1 --2- 172.21.2.102:0/3859904272 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f04380f0650 0x7f04380cfb20 secure :-1 s=READY pgs=2283 cs=0 l=1 rev1=1 rx=0x7f0430006e30 tx=0x7f0430009de0).stop 2021-02-10T11:32:33.381 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:33.364 7f043e7ef700 1 -- 172.21.2.102:0/3859904272 shutdown_connections 2021-02-10T11:32:33.381 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:33.364 7f043e7ef700 1 --2- 172.21.2.102:0/3859904272 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f04380f0650 0x7f04380cfb20 unknown :-1 s=CLOSED pgs=2283 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:33.381 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:33.364 7f043e7ef700 1 --2- 172.21.2.102:0/3859904272 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f04380d1870 0x7f04381b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:33.381 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:33.364 7f043e7ef700 1 -- 172.21.2.102:0/3859904272 >> 172.21.2.102:0/3859904272 conn(0x7f04380ce450 msgr2=0x7f04380d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:33.382 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:33.364 7f043e7ef700 1 -- 172.21.2.102:0/3859904272 shutdown_connections 2021-02-10T11:32:33.382 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:33.364 7f043e7ef700 1 -- 172.21.2.102:0/3859904272 wait complete. 2021-02-10T11:32:33.383 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:33.379 7f043e7ef700 1 Processor -- start 2021-02-10T11:32:33.383 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:33.379 7f043e7ef700 1 -- start start 2021-02-10T11:32:33.383 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:33.379 7f043e7ef700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f04381b8140 0x7f04381b8540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:33.383 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:33.379 7f043e7ef700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f04381b8b60 0x7f04381969b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:33.383 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:33.379 7f043e7ef700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0438196fd0 0x7f0438197f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:33.383 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:33.379 7f043e7ef700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0438197990 con 0x7f04381b8b60 2021-02-10T11:32:33.383 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:33.379 7f043e7ef700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04381973d0 con 0x7f04381b8140 2021-02-10T11:32:33.383 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:33.379 7f043e7ef700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04381975f0 con 0x7f0438196fd0 2021-02-10T11:32:33.384 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:33.379 7f04377fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f04381b8b60 0x7f04381969b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:33.384 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:33.379 7f04377fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f04381b8b60 0x7f04381969b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:34540/0 (socket says 172.21.2.102:34540) 2021-02-10T11:32:33.384 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:33.379 7f04377fe700 1 -- 172.21.2.102:0/1712205725 learned_addr learned my addr 172.21.2.102:0/1712205725 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:32:33.384 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:33.380 7f04377fe700 1 -- 172.21.2.102:0/1712205725 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0438196fd0 msgr2=0x7f0438197f10 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:32:33.384 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:33.380 7f04377fe700 1 --2- 172.21.2.102:0/1712205725 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0438196fd0 0x7f0438197f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:33.384 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:33.380 7f04377fe700 1 -- 172.21.2.102:0/1712205725 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f04381b8140 msgr2=0x7f04381b8540 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:32:33.384 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:33.380 7f04377fe700 1 --2- 172.21.2.102:0/1712205725 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f04381b8140 0x7f04381b8540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:33.384 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:33.380 7f04377fe700 1 -- 172.21.2.102:0/1712205725 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0438198530 con 0x7f04381b8b60 2021-02-10T11:32:33.384 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:33.380 7f04377fe700 1 --2- 172.21.2.102:0/1712205725 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f04381b8b60 0x7f04381969b0 secure :-1 s=READY pgs=2284 cs=0 l=1 rev1=1 rx=0x7f042c00a570 tx=0x7f042c00a910).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:33.384 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:33.380 7f0434ff9700 1 -- 172.21.2.102:0/1712205725 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f042c00ad00 con 0x7f04381b8b60 2021-02-10T11:32:33.384 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:33.380 7f0434ff9700 1 -- 172.21.2.102:0/1712205725 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f042c001180 con 0x7f04381b8b60 2021-02-10T11:32:33.384 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:33.380 7f0434ff9700 1 -- 172.21.2.102:0/1712205725 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f042c004620 con 0x7f04381b8b60 2021-02-10T11:32:33.384 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:33.380 7f043e7ef700 1 -- 172.21.2.102:0/1712205725 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0438198530 con 0x7f04381b8b60 2021-02-10T11:32:33.384 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:33.380 7f043e7ef700 1 -- 172.21.2.102:0/1712205725 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0438198ac0 con 0x7f04381b8b60 2021-02-10T11:32:33.385 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:33.381 7f0434ff9700 1 -- 172.21.2.102:0/1712205725 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f042c0043a0 con 0x7f04381b8b60 2021-02-10T11:32:33.385 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:33.381 7f0434ff9700 1 --2- 172.21.2.102:0/1712205725 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f04180186c0 0x7f041801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:33.385 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:33.381 7f0434ff9700 1 -- 172.21.2.102:0/1712205725 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 4753+0+0 (secure 0 0 0) 0x7f042c032200 con 0x7f04381b8b60 2021-02-10T11:32:33.385 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:33.381 7f0437fff700 1 --2- 172.21.2.102:0/1712205725 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f04180186c0 0x7f041801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:33.385 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:33.381 7f0437fff700 1 --2- 172.21.2.102:0/1712205725 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f04180186c0 0x7f041801abc0 crc :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:33.413 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:33.412 7f043e7ef700 1 -- 172.21.2.102:0/1712205725 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04380ce450 con 0x7f04381b8b60 2021-02-10T11:32:33.415 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:33.414 7f0434ff9700 1 -- 172.21.2.102:0/1712205725 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f042c006fd0 con 0x7f04381b8b60 2021-02-10T11:32:33.547 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:33.546 7f043e7ef700 1 -- 172.21.2.102:0/1712205725 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f04380ce450 con 0x7f04381b8b60 2021-02-10T11:32:33.548 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:33.546 7f0434ff9700 1 -- 172.21.2.102:0/1712205725 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v66) v1 ==== 80+0+4513 (secure 0 0 0) 0x7f042c0043a0 con 0x7f04381b8b60 2021-02-10T11:32:33.548 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:32:33.549 INFO:teuthology.orchestra.run.gibba002.stdout:{ 2021-02-10T11:32:33.549 INFO:teuthology.orchestra.run.gibba002.stdout: "auth_dump": [ 2021-02-10T11:32:33.549 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:32:33.549 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "mds.a", 2021-02-10T11:32:33.549 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAevSNgsgyEARAAvr0cUjjCoEHLxaZSZ2HtVA==", 2021-02-10T11:32:33.549 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:32:33.549 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow", 2021-02-10T11:32:33.549 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:32:33.549 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:32:33.549 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:32:33.549 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:32:33.549 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:32:33.550 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:32:33.550 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "mds.b", 2021-02-10T11:32:33.550 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAevSNgwOFMMRAALfb1fbX56ao1TU8ClnTW1A==", 2021-02-10T11:32:33.550 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:32:33.550 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow", 2021-02-10T11:32:33.550 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:32:33.550 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:32:33.550 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:32:33.550 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:32:33.550 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:32:33.550 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:32:33.550 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.0", 2021-02-10T11:32:33.550 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAmvSNgj+bMMBAAD3HUD6fLiMAg+++iNzLJKQ==", 2021-02-10T11:32:33.551 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:32:33.551 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:32:33.551 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:32:33.551 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:32:33.551 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:32:33.551 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:32:33.551 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:32:33.551 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.1", 2021-02-10T11:32:33.551 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQApvSNgnvc+MRAAAOxqlNwiyLHqNITOrYeFKQ==", 2021-02-10T11:32:33.551 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:32:33.551 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:32:33.551 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:32:33.551 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:32:33.552 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:32:33.552 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:32:33.552 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:32:33.552 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.2", 2021-02-10T11:32:33.552 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAsvSNgZ4SbMRAA76IOYWfbiAuBoM/4k81GxA==", 2021-02-10T11:32:33.552 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:32:33.552 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:32:33.552 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:32:33.552 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:32:33.552 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:32:33.552 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:32:33.552 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:32:33.552 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.3", 2021-02-10T11:32:33.552 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAvvSNgFk3rMRAAWCmKKyw0LDQ+UlAHZAITOw==", 2021-02-10T11:32:33.553 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:32:33.553 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:32:33.553 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:32:33.553 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:32:33.553 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:32:33.553 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:32:33.553 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:32:33.553 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.4", 2021-02-10T11:32:33.553 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQA4vSNgpKcNJBAApJMmvj5YxVHPDOmvmQONOg==", 2021-02-10T11:32:33.553 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:32:33.553 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:32:33.553 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:32:33.553 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:32:33.553 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:32:33.554 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:32:33.554 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:32:33.554 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.5", 2021-02-10T11:32:33.554 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQA7vSNgf2ujJBAAfkuYpyb3WTzc8V5kw+Wktg==", 2021-02-10T11:32:33.554 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:32:33.554 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:32:33.554 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:32:33.554 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:32:33.554 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:32:33.554 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:32:33.554 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:32:33.554 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.6", 2021-02-10T11:32:33.554 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQA+vSNgqw9uJRAAFMU0R+UxU9C3o/9byb3e0g==", 2021-02-10T11:32:33.555 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:32:33.555 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:32:33.555 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:32:33.555 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:32:33.555 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:32:33.555 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:32:33.555 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:32:33.555 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.7", 2021-02-10T11:32:33.555 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQBBvSNgvWSrJRAA2KaCHmLpzovRgYzLvyofTQ==", 2021-02-10T11:32:33.555 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:32:33.555 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:32:33.555 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:32:33.555 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:32:33.556 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:32:33.556 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:32:33.556 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:32:33.556 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "client.0", 2021-02-10T11:32:33.556 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAfvSNgZd9XIBAAvyrvl8fdKuHnJZfczfRoFw==", 2021-02-10T11:32:33.556 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:32:33.556 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow", 2021-02-10T11:32:33.556 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow r", 2021-02-10T11:32:33.556 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:32:33.556 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:32:33.556 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:32:33.556 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:32:33.556 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "client.1", 2021-02-10T11:32:33.556 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAfvSNgSyfXNhAAIduPnF7Vd/6HSkJ0MlcMSA==", 2021-02-10T11:32:33.557 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:32:33.557 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow", 2021-02-10T11:32:33.557 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow r", 2021-02-10T11:32:33.557 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:32:33.557 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:32:33.557 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:32:33.557 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:32:33.557 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "client.2", 2021-02-10T11:32:33.557 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAgvSNgiIG/ERAA9ZTamaTI7ZsPSnxBqjplCg==", 2021-02-10T11:32:33.557 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:32:33.557 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow", 2021-02-10T11:32:33.557 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow r", 2021-02-10T11:32:33.557 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:32:33.558 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:32:33.558 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:32:33.558 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:32:33.558 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "client.3", 2021-02-10T11:32:33.558 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAgvSNgOWpOKBAAlXsEB04Q/GvtL06PqRMLeA==", 2021-02-10T11:32:33.558 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:32:33.558 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow", 2021-02-10T11:32:33.558 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow r", 2021-02-10T11:32:33.558 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:32:33.558 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:32:33.558 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:32:33.558 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:32:33.558 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "client.admin", 2021-02-10T11:32:33.559 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAavSNgO3GQBhAAL+NVnYskQRKVWLgyDPY/xw==", 2021-02-10T11:32:33.559 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:32:33.559 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow *", 2021-02-10T11:32:33.559 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:32:33.559 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:32:33.559 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:32:33.559 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:32:33.559 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:32:33.559 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:32:33.559 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "mgr.x", 2021-02-10T11:32:33.559 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAdvSNgXZZgJhAA7dtBSGzH+DsBwnddZpOBJA==", 2021-02-10T11:32:33.559 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:32:33.559 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow *", 2021-02-10T11:32:33.559 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow profile mgr", 2021-02-10T11:32:33.560 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:32:33.560 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:32:33.560 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:32:33.560 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:32:33.560 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "mgr.y", 2021-02-10T11:32:33.560 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAdvSNgVqQFDRAAwo6YEdWZCAr5GdOWVL/Qvg==", 2021-02-10T11:32:33.560 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:32:33.560 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow *", 2021-02-10T11:32:33.560 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow profile mgr", 2021-02-10T11:32:33.560 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:32:33.560 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:32:33.560 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:32:33.560 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:32:33.561 INFO:teuthology.orchestra.run.gibba002.stdout:} 2021-02-10T11:32:33.562 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:33.548 7f043e7ef700 1 -- 172.21.2.102:0/1712205725 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f04180186c0 msgr2=0x7f041801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:33.562 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:33.548 7f043e7ef700 1 --2- 172.21.2.102:0/1712205725 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f04180186c0 0x7f041801abc0 crc :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:33.562 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:33.548 7f043e7ef700 1 -- 172.21.2.102:0/1712205725 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f04381b8b60 msgr2=0x7f04381969b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:33.562 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:33.548 7f043e7ef700 1 --2- 172.21.2.102:0/1712205725 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f04381b8b60 0x7f04381969b0 secure :-1 s=READY pgs=2284 cs=0 l=1 rev1=1 rx=0x7f042c00a570 tx=0x7f042c00a910).stop 2021-02-10T11:32:33.562 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:33.549 7f043e7ef700 1 -- 172.21.2.102:0/1712205725 shutdown_connections 2021-02-10T11:32:33.562 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:33.549 7f043e7ef700 1 --2- 172.21.2.102:0/1712205725 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0438196fd0 0x7f0438197f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:33.562 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:33.549 7f043e7ef700 1 --2- 172.21.2.102:0/1712205725 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f04381b8b60 0x7f04381969b0 unknown :-1 s=CLOSED pgs=2284 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:33.562 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:33.549 7f043e7ef700 1 --2- 172.21.2.102:0/1712205725 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f04180186c0 0x7f041801abc0 unknown :-1 s=CLOSED pgs=1004 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:33.562 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:33.549 7f043e7ef700 1 --2- 172.21.2.102:0/1712205725 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f04381b8140 0x7f04381b8540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:33.562 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:33.549 7f043e7ef700 1 -- 172.21.2.102:0/1712205725 >> 172.21.2.102:0/1712205725 conn(0x7f04380efc40 msgr2=0x7f04381b9080 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:33.563 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:33.549 7f043e7ef700 1 -- 172.21.2.102:0/1712205725 shutdown_connections 2021-02-10T11:32:33.563 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:33.549 7f043e7ef700 1 -- 172.21.2.102:0/1712205725 wait complete. 2021-02-10T11:32:33.563 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:32:33.991 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:33.973 7fe1d6617700 1 Processor -- start 2021-02-10T11:32:33.991 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:33.973 7fe1d6617700 1 -- start start 2021-02-10T11:32:33.991 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:33.973 7fe1d6617700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fe1d00f05f0 0x7fe1d00f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:33.991 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:33.973 7fe1d6617700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1d01b0c70 con 0x7fe1d01b0470 2021-02-10T11:32:33.991 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:33.973 7fe1d6617700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1d00f0b50 con 0x7fe1d01b0870 2021-02-10T11:32:33.991 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:33.973 7fe1d6617700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1d00e5ea0 con 0x7fe1d00f05f0 2021-02-10T11:32:33.991 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:33.974 7fe1cffff700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fe1d01b0470 0x7fe1d00ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:55002/0 (socket says 172.21.2.102:55002) 2021-02-10T11:32:33.991 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:33.974 7fe1cffff700 1 -- 172.21.2.102:0/312299333 learned_addr learned my addr 172.21.2.102:0/312299333 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:32:33.991 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:33.974 7fe1d4c34700 1 --2- 172.21.2.102:0/312299333 >> v2:172.21.2.102:3300/0 conn(0x7fe1d00f05f0 0x7fe1d00f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:33.991 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:33.974 7fe1ceffd700 1 -- 172.21.2.102:0/312299333 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2178050222 0 0) 0x7fe1c0001180 con 0x7fe1d01b0470 2021-02-10T11:32:33.991 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:33.974 7fe1ceffd700 1 -- 172.21.2.102:0/312299333 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1b8002fe0 con 0x7fe1d01b0470 2021-02-10T11:32:33.992 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:33.974 7fe1d4c34700 1 -- 172.21.2.102:0/312299333 >> v1:172.21.2.113:6789/0 conn(0x7fe1d01b0870 legacy=0x7fe1d00efe00 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:32:33.992 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:33.974 7fe1d4c34700 1 -- 172.21.2.102:0/312299333 >> v1:172.21.2.102:6789/0 conn(0x7fe1d01b0470 legacy=0x7fe1d00ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:33.992 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:33.974 7fe1d4c34700 1 -- 172.21.2.102:0/312299333 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1d00e52e0 con 0x7fe1d00f05f0 2021-02-10T11:32:33.992 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:33.974 7fe1d4c34700 1 --2- 172.21.2.102:0/312299333 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe1d00f05f0 0x7fe1d00f20e0 secure :-1 s=READY pgs=2286 cs=0 l=1 rev1=1 rx=0x7fe1c8006e30 tx=0x7fe1c8009e60).ready entity=mon.0 client_cookie=38ae697e2266f782 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:33.992 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:33.974 7fe1ceffd700 1 -- 172.21.2.102:0/312299333 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe1c8006a50 con 0x7fe1d00f05f0 2021-02-10T11:32:33.992 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:33.974 7fe1ceffd700 1 -- 172.21.2.102:0/312299333 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1c800b300 con 0x7fe1d00f05f0 2021-02-10T11:32:33.992 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:33.974 7fe1ceffd700 1 -- 172.21.2.102:0/312299333 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe1c800b820 con 0x7fe1d00f05f0 2021-02-10T11:32:33.992 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:33.974 7fe1d6617700 1 -- 172.21.2.102:0/312299333 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe1d00f05f0 msgr2=0x7fe1d00f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:33.992 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:33.974 7fe1d6617700 1 --2- 172.21.2.102:0/312299333 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe1d00f05f0 0x7fe1d00f20e0 secure :-1 s=READY pgs=2286 cs=0 l=1 rev1=1 rx=0x7fe1c8006e30 tx=0x7fe1c8009e60).stop 2021-02-10T11:32:33.992 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:33.975 7fe1d6617700 1 -- 172.21.2.102:0/312299333 shutdown_connections 2021-02-10T11:32:33.992 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:33.975 7fe1d6617700 1 --2- 172.21.2.102:0/312299333 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe1d00f05f0 0x7fe1d00f20e0 unknown :-1 s=CLOSED pgs=2286 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:33.992 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:33.975 7fe1d6617700 1 -- 172.21.2.102:0/312299333 >> 172.21.2.102:0/312299333 conn(0x7fe1d00ce3e0 msgr2=0x7fe1d00e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:33.992 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:33.975 7fe1d6617700 1 -- 172.21.2.102:0/312299333 shutdown_connections 2021-02-10T11:32:33.993 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:33.975 7fe1d6617700 1 -- 172.21.2.102:0/312299333 wait complete. 2021-02-10T11:32:33.994 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:33.990 7fe1d6617700 1 Processor -- start 2021-02-10T11:32:33.994 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:33.990 7fe1d6617700 1 -- start start 2021-02-10T11:32:33.994 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:33.990 7fe1d6617700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe1d00b0970 0x7fe1d0092ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:33.994 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:33.990 7fe1d6617700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe1d00934f0 0x7fe1d0094dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:33.994 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:33.990 7fe1d6617700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe1d00938f0 0x7fe1d00953e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:33.994 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:33.990 7fe1d6617700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1d0195ad0 con 0x7fe1d00b0970 2021-02-10T11:32:33.994 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:33.990 7fe1d6617700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1d0095f40 con 0x7fe1d00934f0 2021-02-10T11:32:33.994 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:33.990 7fe1d6617700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1d0095a30 con 0x7fe1d00938f0 2021-02-10T11:32:33.995 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:33.990 7fe1cffff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe1d00b0970 0x7fe1d0092ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:33.995 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:33.990 7fe1cffff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe1d00b0970 0x7fe1d0092ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:34552/0 (socket says 172.21.2.102:34552) 2021-02-10T11:32:33.995 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:33.990 7fe1cffff700 1 -- 172.21.2.102:0/2066091900 learned_addr learned my addr 172.21.2.102:0/2066091900 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:32:33.995 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:33.991 7fe1cffff700 1 -- 172.21.2.102:0/2066091900 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe1d00938f0 msgr2=0x7fe1d00953e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:33.995 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:33.991 7fe1cffff700 1 --2- 172.21.2.102:0/2066091900 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe1d00938f0 0x7fe1d00953e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:33.995 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:33.991 7fe1cffff700 1 -- 172.21.2.102:0/2066091900 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe1d00934f0 msgr2=0x7fe1d0094dc0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:32:33.995 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:33.991 7fe1cffff700 1 --2- 172.21.2.102:0/2066091900 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe1d00934f0 0x7fe1d0094dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:33.995 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:33.991 7fe1cffff700 1 -- 172.21.2.102:0/2066091900 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1d0095c50 con 0x7fe1d00b0970 2021-02-10T11:32:33.996 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:33.991 7fe1cffff700 1 --2- 172.21.2.102:0/2066091900 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe1d00b0970 0x7fe1d0092ed0 secure :-1 s=READY pgs=2287 cs=0 l=1 rev1=1 rx=0x7fe1c0006f00 tx=0x7fe1c000b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:33.996 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:33.991 7fe1ccff9700 1 -- 172.21.2.102:0/2066091900 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe1c000bd30 con 0x7fe1d00b0970 2021-02-10T11:32:33.996 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:33.991 7fe1ccff9700 1 -- 172.21.2.102:0/2066091900 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1c00060c0 con 0x7fe1d00b0970 2021-02-10T11:32:33.996 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:33.991 7fe1d6617700 1 -- 172.21.2.102:0/2066091900 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1d0095c50 con 0x7fe1d00b0970 2021-02-10T11:32:33.996 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:33.991 7fe1d6617700 1 -- 172.21.2.102:0/2066091900 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1d00967b0 con 0x7fe1d00b0970 2021-02-10T11:32:33.996 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:33.991 7fe1ccff9700 1 -- 172.21.2.102:0/2066091900 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe1c0004590 con 0x7fe1d00b0970 2021-02-10T11:32:33.996 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:33.992 7fe1ccff9700 1 -- 172.21.2.102:0/2066091900 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fe1c00043a0 con 0x7fe1d00b0970 2021-02-10T11:32:33.996 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:33.992 7fe1ccff9700 1 --2- 172.21.2.102:0/2066091900 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe1b00186c0 0x7fe1b001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:33.996 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:33.992 7fe1ccff9700 1 -- 172.21.2.102:0/2066091900 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 4753+0+0 (secure 0 0 0) 0x7fe1c0032250 con 0x7fe1d00b0970 2021-02-10T11:32:33.996 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:33.992 7fe1cf7fe700 1 --2- 172.21.2.102:0/2066091900 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe1b00186c0 0x7fe1b001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:33.996 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:33.992 7fe1cf7fe700 1 --2- 172.21.2.102:0/2066091900 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe1b00186c0 0x7fe1b001abc0 crc :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:34.025 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:34.023 7fe1d6617700 1 -- 172.21.2.102:0/2066091900 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1d00e52e0 con 0x7fe1d00b0970 2021-02-10T11:32:34.027 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:34.026 7fe1ccff9700 1 -- 172.21.2.102:0/2066091900 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fe1c0030d30 con 0x7fe1d00b0970 2021-02-10T11:32:34.153 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:34.151 7fe1d6617700 1 -- 172.21.2.102:0/2066091900 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe1d00e52e0 con 0x7fe1d00b0970 2021-02-10T11:32:34.153 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:34.152 7fe1ccff9700 1 -- 172.21.2.102:0/2066091900 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v116) v1 ==== 74+0+11154 (secure 0 0 0) 0x7fe1c00043a0 con 0x7fe1d00b0970 2021-02-10T11:32:34.154 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:32:34.154 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":116,"fsid":"68db30ea-dbce-4aef-a6df-f01c6fea0513","created":"2021-02-10 11:03:06.331557","modified":"2021-02-10 11:32:32.888111","last_up_change":"2021-02-10 11:03:32.690562","last_in_change":"2021-02-10 11:03:32.690562","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":23,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:03:37.322519","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"be691b4d-a039-408a-af80-9d53f02b6f1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6818","nonce":36088},{"type":"v1","addr":"172.21.2.102:6820","nonce":36088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6822","nonce":36088},{"type":"v1","addr":"172.21.2.102:6824","nonce":36088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6830","nonce":36088},{"type":"v1","addr":"172.21.2.102:6832","nonce":36088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6826","nonce":36088},{"type":"v1","addr":"172.21.2.102:6828","nonce":36088}]},"public_addr":"172.21.2.102:6820/36088","cluster_addr":"172.21.2.102:6824/36088","heartbeat_back_addr":"172.21.2.102:6832/36088","heartbeat_front_addr":"172.21.2.102:6828/36088","state":["exists","up"]},{"osd":1,"uuid":"64e3ccc8-9c69-4d7c-865e-af8e83822635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6819","nonce":36087},{"type":"v1","addr":"172.21.2.102:6821","nonce":36087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6823","nonce":36087},{"type":"v1","addr":"172.21.2.102:6825","nonce":36087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6831","nonce":36087},{"type":"v1","addr":"172.21.2.102:6833","nonce":36087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6827","nonce":36087},{"type":"v1","addr":"172.21.2.102:6829","nonce":36087}]},"public_addr":"172.21.2.102:6821/36087","cluster_addr":"172.21.2.102:6825/36087","heartbeat_back_addr":"172.21.2.102:6833/36087","heartbeat_front_addr":"172.21.2.102:6829/36087","state":["exists","up"]},{"osd":2,"uuid":"153700af-747f-481f-bdf4-d6072d3a842f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6810","nonce":36089},{"type":"v1","addr":"172.21.2.102:6811","nonce":36089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6812","nonce":36089},{"type":"v1","addr":"172.21.2.102:6813","nonce":36089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6816","nonce":36089},{"type":"v1","addr":"172.21.2.102:6817","nonce":36089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6814","nonce":36089},{"type":"v1","addr":"172.21.2.102:6815","nonce":36089}]},"public_addr":"172.21.2.102:6811/36089","cluster_addr":"172.21.2.102:6813/36089","heartbeat_back_addr":"172.21.2.102:6817/36089","heartbeat_front_addr":"172.21.2.102:6815/36089","state":["exists","up"]},{"osd":3,"uuid":"cbc678f4-83a9-45e6-8830-2f7c232b1ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6802","nonce":36090},{"type":"v1","addr":"172.21.2.102:6803","nonce":36090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6804","nonce":36090},{"type":"v1","addr":"172.21.2.102:6805","nonce":36090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6808","nonce":36090},{"type":"v1","addr":"172.21.2.102:6809","nonce":36090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6806","nonce":36090},{"type":"v1","addr":"172.21.2.102:6807","nonce":36090}]},"public_addr":"172.21.2.102:6803/36090","cluster_addr":"172.21.2.102:6805/36090","heartbeat_back_addr":"172.21.2.102:6809/36090","heartbeat_front_addr":"172.21.2.102:6807/36090","state":["exists","up"]},{"osd":4,"uuid":"c01e4008-b258-4f10-afd2-9e7f37214351","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6808","nonce":36368},{"type":"v1","addr":"172.21.2.113:6811","nonce":36368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6814","nonce":36368},{"type":"v1","addr":"172.21.2.113:6818","nonce":36368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6826","nonce":36368},{"type":"v1","addr":"172.21.2.113:6828","nonce":36368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6820","nonce":36368},{"type":"v1","addr":"172.21.2.113:6824","nonce":36368}]},"public_addr":"172.21.2.113:6811/36368","cluster_addr":"172.21.2.113:6818/36368","heartbeat_back_addr":"172.21.2.113:6828/36368","heartbeat_front_addr":"172.21.2.113:6824/36368","state":["exists","up"]},{"osd":5,"uuid":"551f6007-daf8-4a98-8ee0-c2b16bcccb5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6802","nonce":36366},{"type":"v1","addr":"172.21.2.113:6805","nonce":36366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6807","nonce":36366},{"type":"v1","addr":"172.21.2.113:6810","nonce":36366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6819","nonce":36366},{"type":"v1","addr":"172.21.2.113:6822","nonce":36366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6812","nonce":36366},{"type":"v1","addr":"172.21.2.113:6815","nonce":36366}]},"public_addr":"172.21.2.113:6805/36366","cluster_addr":"172.21.2.113:6810/36366","heartbeat_back_addr":"172.21.2.113:6822/36366","heartbeat_front_addr":"172.21.2.113:6815/36366","state":["exists","up"]},{"osd":6,"uuid":"bf67c040-0e50-48be-b102-c3943bc9ebef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6800","nonce":36367},{"type":"v1","addr":"172.21.2.113:6801","nonce":36367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6803","nonce":36367},{"type":"v1","addr":"172.21.2.113:6804","nonce":36367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6813","nonce":36367},{"type":"v1","addr":"172.21.2.113:6816","nonce":36367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6806","nonce":36367},{"type":"v1","addr":"172.21.2.113:6809","nonce":36367}]},"public_addr":"172.21.2.113:6801/36367","cluster_addr":"172.21.2.113:6804/36367","heartbeat_back_addr":"172.21.2.113:6816/36367","heartbeat_front_addr":"172.21.2.113:6809/36367","state":["exists","up"]},{"osd":7,"uuid":"f5461f80-018d-428f-be89-2eb2bcbe7bb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6817","nonce":36365},{"type":"v1","addr":"172.21.2.113:6821","nonce":36365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6823","nonce":36365},{"type":"v1","addr":"172.21.2.113:6825","nonce":36365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6830","nonce":36365},{"type":"v1","addr":"172.21.2.113:6831","nonce":36365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6827","nonce":36365},{"type":"v1","addr":"172.21.2.113:6829","nonce":36365}]},"public_addr":"172.21.2.113:6821/36365","cluster_addr":"172.21.2.113:6825/36365","heartbeat_back_addr":"172.21.2.113:6831/36365","heartbeat_front_addr":"172.21.2.113:6829/36365","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:32:34.155 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:34.153 7fe1d6617700 1 -- 172.21.2.102:0/2066091900 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe1b00186c0 msgr2=0x7fe1b001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:34.155 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:34.153 7fe1d6617700 1 --2- 172.21.2.102:0/2066091900 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe1b00186c0 0x7fe1b001abc0 crc :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:34.155 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:34.153 7fe1d6617700 1 -- 172.21.2.102:0/2066091900 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe1d00b0970 msgr2=0x7fe1d0092ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:34.155 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:34.153 7fe1d6617700 1 --2- 172.21.2.102:0/2066091900 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe1d00b0970 0x7fe1d0092ed0 secure :-1 s=READY pgs=2287 cs=0 l=1 rev1=1 rx=0x7fe1c0006f00 tx=0x7fe1c000b940).stop 2021-02-10T11:32:34.155 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:34.154 7fe1d6617700 1 -- 172.21.2.102:0/2066091900 shutdown_connections 2021-02-10T11:32:34.155 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:34.154 7fe1d6617700 1 --2- 172.21.2.102:0/2066091900 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe1d00938f0 0x7fe1d00953e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:34.155 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:34.154 7fe1d6617700 1 --2- 172.21.2.102:0/2066091900 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe1b00186c0 0x7fe1b001abc0 unknown :-1 s=CLOSED pgs=1005 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:34.155 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:34.154 7fe1d6617700 1 --2- 172.21.2.102:0/2066091900 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe1d00934f0 0x7fe1d0094dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:34.155 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:34.154 7fe1d6617700 1 --2- 172.21.2.102:0/2066091900 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe1d00b0970 0x7fe1d0092ed0 unknown :-1 s=CLOSED pgs=2287 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:34.156 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:34.154 7fe1d6617700 1 -- 172.21.2.102:0/2066091900 >> 172.21.2.102:0/2066091900 conn(0x7fe1d019f7a0 msgr2=0x7fe1d00940e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:34.156 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:34.154 7fe1d6617700 1 -- 172.21.2.102:0/2066091900 shutdown_connections 2021-02-10T11:32:34.156 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:34.154 7fe1d6617700 1 -- 172.21.2.102:0/2066091900 wait complete. 2021-02-10T11:32:34.164 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:03:37.322519', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2021-02-10T11:32:34.164 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:32:34.596 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:34.577 7f85d1303700 1 Processor -- start 2021-02-10T11:32:34.596 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:34.578 7f85d1303700 1 -- start start 2021-02-10T11:32:34.597 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:34.578 7f85d1303700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f85cc0d1c70 0x7f85cc0cef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:34.597 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:34.578 7f85d1303700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85cc0d0170 con 0x7f85cc0d1870 2021-02-10T11:32:34.597 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:34.578 7f85d1303700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85cc0fe1c0 con 0x7f85cc0cf580 2021-02-10T11:32:34.597 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:34.578 7f85d1303700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85cc1b0cb0 con 0x7f85cc0d1c70 2021-02-10T11:32:34.597 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:34.578 7f85cb07d700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f85cc0d1870 0x7f85cc1b04c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:55016/0 (socket says 172.21.2.102:55016) 2021-02-10T11:32:34.597 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:34.578 7f85cb07d700 1 -- 172.21.2.102:0/4268493127 learned_addr learned my addr 172.21.2.102:0/4268493127 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:32:34.597 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:34.579 7f85ca07b700 1 -- 172.21.2.102:0/4268493127 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2038276212 0 0) 0x7f85b40010b0 con 0x7f85cc0d1870 2021-02-10T11:32:34.597 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:34.579 7f85ca07b700 1 -- 172.21.2.102:0/4268493127 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85b8002fe0 con 0x7f85cc0d1870 2021-02-10T11:32:34.597 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:34.579 7f85ca87c700 1 --2- 172.21.2.102:0/4268493127 >> v2:172.21.2.113:3300/0 conn(0x7f85cc0d1c70 0x7f85cc0cef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:34.597 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:34.579 7f85ca07b700 1 -- 172.21.2.102:0/4268493127 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2673881894 0 0) 0x7f85b40010b0 con 0x7f85cc0d1870 2021-02-10T11:32:34.597 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:34.579 7f85ca07b700 1 -- 172.21.2.102:0/4268493127 >> v2:172.21.2.113:3300/0 conn(0x7f85cc0d1c70 msgr2=0x7f85cc0cef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:32:34.597 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:34.579 7f85ca07b700 1 --2- 172.21.2.102:0/4268493127 >> v2:172.21.2.113:3300/0 conn(0x7f85cc0d1c70 0x7f85cc0cef60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:34.597 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:34.579 7f85ca07b700 1 -- 172.21.2.102:0/4268493127 >> v1:172.21.2.113:6789/0 conn(0x7f85cc0cf580 legacy=0x7f85cc0cf980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:32:34.598 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:34.579 7f85ca07b700 1 -- 172.21.2.102:0/4268493127 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85cc109a60 con 0x7f85cc0d1870 2021-02-10T11:32:34.598 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:34.579 7f85ca07b700 1 -- 172.21.2.102:0/4268493127 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7f85b40015b0 con 0x7f85cc0d1870 2021-02-10T11:32:34.598 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:34.579 7f85ca07b700 1 -- 172.21.2.102:0/4268493127 >> v1:172.21.2.102:6789/0 conn(0x7f85cc0d1870 legacy=0x7f85cc1b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:34.598 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:34.579 7f85ca07b700 1 --2- 172.21.2.102:0/4268493127 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f85b8003ae0 0x7f85b8005f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:34.598 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:34.579 7f85ca07b700 1 --2- 172.21.2.102:0/4268493127 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f85b80067b0 0x7f85b8008cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:34.598 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:34.579 7f85ca07b700 1 --2- 172.21.2.102:0/4268493127 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f85b80095b0 0x7f85b800bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:34.598 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:34.579 7f85ca07b700 1 -- 172.21.2.102:0/4268493127 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85b800c300 con 0x7f85b80095b0 2021-02-10T11:32:34.598 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:34.579 7f85ca07b700 1 -- 172.21.2.102:0/4268493127 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85b800c520 con 0x7f85b80067b0 2021-02-10T11:32:34.598 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:34.579 7f85ca07b700 1 -- 172.21.2.102:0/4268493127 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85b800c740 con 0x7f85b8003ae0 2021-02-10T11:32:34.598 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:34.579 7f85cb87e700 1 --2- 172.21.2.102:0/4268493127 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f85b80095b0 0x7f85b800bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:34.598 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:34.579 7f85cb07d700 1 --2- 172.21.2.102:0/4268493127 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f85b8003ae0 0x7f85b8005f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:34.598 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:34.579 7f85cb87e700 1 -- 172.21.2.102:0/4268493127 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f85b8003ae0 msgr2=0x7f85b8005f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:32:34.599 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:34.579 7f85cb87e700 1 --2- 172.21.2.102:0/4268493127 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f85b8003ae0 0x7f85b8005f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:34.599 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:34.579 7f85cb87e700 1 -- 172.21.2.102:0/4268493127 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f85b80067b0 msgr2=0x7f85b8008cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:32:34.599 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:34.579 7f85cb87e700 1 --2- 172.21.2.102:0/4268493127 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f85b80067b0 0x7f85b8008cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:34.599 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:34.579 7f85cb87e700 1 -- 172.21.2.102:0/4268493127 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85b800c9e0 con 0x7f85b80095b0 2021-02-10T11:32:34.599 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:34.579 7f85cb07d700 1 --2- 172.21.2.102:0/4268493127 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f85b8003ae0 0x7f85b8005f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:32:34.599 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:34.579 7f85cb87e700 1 -- 172.21.2.102:0/4268493127 reap_dead start 2021-02-10T11:32:34.599 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:34.579 7f85cb87e700 1 --2- 172.21.2.102:0/4268493127 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f85b80095b0 0x7f85b800bab0 secure :-1 s=READY pgs=2289 cs=0 l=1 rev1=1 rx=0x7f85c4006e30 tx=0x7f85c4009e50).ready entity=mon.0 client_cookie=eba642d4b245d2c7 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:34.599 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:34.580 7f85ca07b700 1 -- 172.21.2.102:0/4268493127 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f85c4003020 con 0x7f85b80095b0 2021-02-10T11:32:34.599 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:34.580 7f85ca07b700 1 -- 172.21.2.102:0/4268493127 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85c4001400 con 0x7f85b80095b0 2021-02-10T11:32:34.599 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:34.580 7f85ca07b700 1 -- 172.21.2.102:0/4268493127 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f85c4001880 con 0x7f85b80095b0 2021-02-10T11:32:34.599 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:34.580 7f85d1303700 1 -- 172.21.2.102:0/4268493127 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f85b80095b0 msgr2=0x7f85b800bab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:34.599 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:34.580 7f85d1303700 1 --2- 172.21.2.102:0/4268493127 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f85b80095b0 0x7f85b800bab0 secure :-1 s=READY pgs=2289 cs=0 l=1 rev1=1 rx=0x7f85c4006e30 tx=0x7f85c4009e50).stop 2021-02-10T11:32:34.599 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:34.580 7f85d1303700 1 -- 172.21.2.102:0/4268493127 shutdown_connections 2021-02-10T11:32:34.600 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:34.580 7f85d1303700 1 --2- 172.21.2.102:0/4268493127 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f85b80095b0 0x7f85b800bab0 unknown :-1 s=CLOSED pgs=2289 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:34.600 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:34.580 7f85d1303700 1 -- 172.21.2.102:0/4268493127 >> 172.21.2.102:0/4268493127 conn(0x7f85cc0ce420 msgr2=0x7f85cc0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:34.600 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:34.580 7f85d1303700 1 -- 172.21.2.102:0/4268493127 shutdown_connections 2021-02-10T11:32:34.600 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:34.580 7f85d1303700 1 -- 172.21.2.102:0/4268493127 wait complete. 2021-02-10T11:32:34.600 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:34.595 7f85d1303700 1 Processor -- start 2021-02-10T11:32:34.600 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:34.595 7f85d1303700 1 -- start start 2021-02-10T11:32:34.602 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:34.595 7f85d1303700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f85cc09bcc0 0x7f85cc09c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:34.602 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:34.595 7f85d1303700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f85cc0a0820 0x7f85cc0a0c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:34.602 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:34.595 7f85d1303700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f85cc0a1240 0x7f85cc0858d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:34.602 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:34.595 7f85d1303700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85cc0a66b0 con 0x7f85cc0a0820 2021-02-10T11:32:34.602 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:34.595 7f85d1303700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85cc1a49b0 con 0x7f85cc09bcc0 2021-02-10T11:32:34.602 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:34.595 7f85d1303700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85cc1a4b90 con 0x7f85cc0a1240 2021-02-10T11:32:34.602 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:34.596 7f85ca87c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f85cc0a0820 0x7f85cc0a0c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:34.602 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:34.596 7f85ca87c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f85cc0a0820 0x7f85cc0a0c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:34574/0 (socket says 172.21.2.102:34574) 2021-02-10T11:32:34.602 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:34.596 7f85ca87c700 1 -- 172.21.2.102:0/3155248999 learned_addr learned my addr 172.21.2.102:0/3155248999 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:32:34.602 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:34.596 7f85cb07d700 1 --2- 172.21.2.102:0/3155248999 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f85cc09bcc0 0x7f85cc09c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:34.602 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:34.596 7f85cb87e700 1 --2- 172.21.2.102:0/3155248999 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f85cc0a1240 0x7f85cc0858d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:34.603 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:34.596 7f85ca87c700 1 -- 172.21.2.102:0/3155248999 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f85cc0a1240 msgr2=0x7f85cc0858d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:34.603 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:34.596 7f85ca87c700 1 --2- 172.21.2.102:0/3155248999 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f85cc0a1240 0x7f85cc0858d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:34.603 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:34.596 7f85ca87c700 1 -- 172.21.2.102:0/3155248999 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f85cc09bcc0 msgr2=0x7f85cc09c0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:34.603 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:34.596 7f85ca87c700 1 --2- 172.21.2.102:0/3155248999 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f85cc09bcc0 0x7f85cc09c0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:34.603 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:34.596 7f85ca87c700 1 -- 172.21.2.102:0/3155248999 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85cc0a1640 con 0x7f85cc0a0820 2021-02-10T11:32:34.603 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:34.596 7f85ca87c700 1 --2- 172.21.2.102:0/3155248999 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f85cc0a0820 0x7f85cc0a0c20 secure :-1 s=READY pgs=2290 cs=0 l=1 rev1=1 rx=0x7f85bc006f20 tx=0x7f85bc00b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:34.603 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:34.596 7f85c3fff700 1 -- 172.21.2.102:0/3155248999 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f85bc006bb0 con 0x7f85cc0a0820 2021-02-10T11:32:34.603 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:34.597 7f85c3fff700 1 -- 172.21.2.102:0/3155248999 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85bc00bd80 con 0x7f85cc0a0820 2021-02-10T11:32:34.603 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:34.597 7f85c3fff700 1 -- 172.21.2.102:0/3155248999 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f85bc002220 con 0x7f85cc0a0820 2021-02-10T11:32:34.603 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:34.597 7f85d1303700 1 -- 172.21.2.102:0/3155248999 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85cc0a1640 con 0x7f85cc0a0820 2021-02-10T11:32:34.603 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:34.597 7f85d1303700 1 -- 172.21.2.102:0/3155248999 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85cc0a1640 con 0x7f85cc0a0820 2021-02-10T11:32:34.603 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:34.597 7f85c3fff700 1 -- 172.21.2.102:0/3155248999 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f85bc00a000 con 0x7f85cc0a0820 2021-02-10T11:32:34.603 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:34.597 7f85c3fff700 1 --2- 172.21.2.102:0/3155248999 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f85b00186c0 0x7f85b001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:34.604 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:34.597 7f85c3fff700 1 -- 172.21.2.102:0/3155248999 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 4753+0+0 (secure 0 0 0) 0x7f85bc032260 con 0x7f85cc0a0820 2021-02-10T11:32:34.604 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:34.597 7f85cb07d700 1 --2- 172.21.2.102:0/3155248999 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f85b00186c0 0x7f85b001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:34.604 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:34.598 7f85cb07d700 1 --2- 172.21.2.102:0/3155248999 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f85b00186c0 0x7f85b001abc0 crc :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:34.629 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:34.628 7f85d1303700 1 -- 172.21.2.102:0/3155248999 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85cc0fe1c0 con 0x7f85cc0a0820 2021-02-10T11:32:34.632 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:34.630 7f85c3fff700 1 -- 172.21.2.102:0/3155248999 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f85bc000c90 con 0x7f85cc0a0820 2021-02-10T11:32:34.760 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:34.759 7f85d1303700 1 -- 172.21.2.102:0/3155248999 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f85cc086d30 con 0x7f85cc0a0820 2021-02-10T11:32:34.761 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:34.759 7f85c3fff700 1 -- 172.21.2.102:0/3155248999 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v116) v1 ==== 92+0+10 (secure 0 0 0) 0x7f85bc00a000 con 0x7f85cc0a0820 2021-02-10T11:32:34.761 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:32:34.762 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:34.761 7f85d1303700 1 -- 172.21.2.102:0/3155248999 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f85b00186c0 msgr2=0x7f85b001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:34.762 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:34.761 7f85d1303700 1 --2- 172.21.2.102:0/3155248999 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f85b00186c0 0x7f85b001abc0 crc :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:34.762 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:34.761 7f85d1303700 1 -- 172.21.2.102:0/3155248999 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f85cc0a0820 msgr2=0x7f85cc0a0c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:34.762 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:34.761 7f85d1303700 1 --2- 172.21.2.102:0/3155248999 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f85cc0a0820 0x7f85cc0a0c20 secure :-1 s=READY pgs=2290 cs=0 l=1 rev1=1 rx=0x7f85bc006f20 tx=0x7f85bc00b990).stop 2021-02-10T11:32:34.763 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:34.761 7f85d1303700 1 -- 172.21.2.102:0/3155248999 shutdown_connections 2021-02-10T11:32:34.763 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:34.761 7f85d1303700 1 --2- 172.21.2.102:0/3155248999 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f85cc0a1240 0x7f85cc0858d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:34.763 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:34.761 7f85d1303700 1 --2- 172.21.2.102:0/3155248999 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f85cc0a0820 0x7f85cc0a0c20 unknown :-1 s=CLOSED pgs=2290 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:34.763 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:34.761 7f85d1303700 1 --2- 172.21.2.102:0/3155248999 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f85b00186c0 0x7f85b001abc0 unknown :-1 s=CLOSED pgs=1006 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:34.763 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:34.761 7f85d1303700 1 --2- 172.21.2.102:0/3155248999 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f85cc09bcc0 0x7f85cc09c0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:34.763 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:34.761 7f85d1303700 1 -- 172.21.2.102:0/3155248999 >> 172.21.2.102:0/3155248999 conn(0x7f85cc0bef00 msgr2=0x7f85cc09d1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:34.763 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:34.762 7f85d1303700 1 -- 172.21.2.102:0/3155248999 shutdown_connections 2021-02-10T11:32:34.764 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:34.762 7f85d1303700 1 -- 172.21.2.102:0/3155248999 wait complete. 2021-02-10T11:32:34.773 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --format=json-pretty osd lspools 2021-02-10T11:32:35.200 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:35.183 7f9568bc3700 1 Processor -- start 2021-02-10T11:32:35.200 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:35.183 7f9568bc3700 1 -- start start 2021-02-10T11:32:35.200 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:35.183 7f9568bc3700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f95640d0340 0x7f95640ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:35.201 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:35.183 7f9568bc3700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f95640d1c70 0x7f95640efc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:35.201 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:35.183 7f9568bc3700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95640f0b10 con 0x7f95640d1870 2021-02-10T11:32:35.201 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:35.183 7f9568bc3700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95640e5ed0 con 0x7f95640d1c70 2021-02-10T11:32:35.201 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:35.183 7f9568bc3700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95641b0be0 con 0x7f95640d0340 2021-02-10T11:32:35.201 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:35.183 7f956305f700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f95640d1c70 0x7f95640efc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:35.201 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:35.183 7f956305f700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f95640d1c70 0x7f95640efc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:34584/0 (socket says 172.21.2.102:34584) 2021-02-10T11:32:35.201 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:35.183 7f956305f700 1 -- 172.21.2.102:0/246662489 learned_addr learned my addr 172.21.2.102:0/246662489 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:32:35.201 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:35.184 7f956305f700 1 -- 172.21.2.102:0/246662489 >> v2:172.21.2.113:3300/0 conn(0x7f95640d0340 msgr2=0x7f95640ef640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:32:35.201 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:35.184 7f956305f700 1 --2- 172.21.2.102:0/246662489 >> v2:172.21.2.113:3300/0 conn(0x7f95640d0340 0x7f95640ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:35.202 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:35.184 7f956305f700 1 -- 172.21.2.102:0/246662489 >> v1:172.21.2.113:6789/0 conn(0x7f95640d1870 legacy=0x7f95641b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:32:35.202 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:35.184 7f956305f700 1 -- 172.21.2.102:0/246662489 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95640f20a0 con 0x7f95640d1c70 2021-02-10T11:32:35.202 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:35.184 7f956305f700 1 --2- 172.21.2.102:0/246662489 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f95640d1c70 0x7f95640efc60 secure :-1 s=READY pgs=2291 cs=0 l=1 rev1=1 rx=0x7f955c006f00 tx=0x7f955c009f10).ready entity=mon.0 client_cookie=841afe49ffefb271 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:35.202 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:35.184 7f956185c700 1 -- 172.21.2.102:0/246662489 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f955c00b3b0 con 0x7f95640d1c70 2021-02-10T11:32:35.202 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:35.184 7f956185c700 1 -- 172.21.2.102:0/246662489 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f955c00b630 con 0x7f95640d1c70 2021-02-10T11:32:35.202 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:35.184 7f956185c700 1 -- 172.21.2.102:0/246662489 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f955c00bb50 con 0x7f95640d1c70 2021-02-10T11:32:35.202 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:35.184 7f9568bc3700 1 -- 172.21.2.102:0/246662489 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f95640d1c70 msgr2=0x7f95640efc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:35.202 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:35.184 7f9568bc3700 1 --2- 172.21.2.102:0/246662489 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f95640d1c70 0x7f95640efc60 secure :-1 s=READY pgs=2291 cs=0 l=1 rev1=1 rx=0x7f955c006f00 tx=0x7f955c009f10).stop 2021-02-10T11:32:35.202 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:35.185 7f9568bc3700 1 -- 172.21.2.102:0/246662489 shutdown_connections 2021-02-10T11:32:35.203 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:35.185 7f9568bc3700 1 --2- 172.21.2.102:0/246662489 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f95640d1c70 0x7f95640efc60 unknown :-1 s=CLOSED pgs=2291 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:35.203 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:35.185 7f9568bc3700 1 --2- 172.21.2.102:0/246662489 >> v2:172.21.2.113:3300/0 conn(0x7f95640d0340 0x7f95640ef640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:35.203 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:35.185 7f9568bc3700 1 -- 172.21.2.102:0/246662489 >> 172.21.2.102:0/246662489 conn(0x7f95640ce450 msgr2=0x7f95640d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:35.203 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:35.185 7f9568bc3700 1 -- 172.21.2.102:0/246662489 shutdown_connections 2021-02-10T11:32:35.203 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:35.185 7f9568bc3700 1 -- 172.21.2.102:0/246662489 wait complete. 2021-02-10T11:32:35.205 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:35.199 7f9568bc3700 1 Processor -- start 2021-02-10T11:32:35.205 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:35.199 7f9568bc3700 1 -- start start 2021-02-10T11:32:35.205 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:35.200 7f9568bc3700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f956418f0a0 0x7f9564196ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:35.205 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:35.200 7f9568bc3700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f95641970d0 0x7f95641974d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:35.205 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:35.200 7f9568bc3700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9564094650 0x7f9564094a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:35.205 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:35.200 7f9568bc3700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9564095bd0 con 0x7f9564094650 2021-02-10T11:32:35.205 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:35.200 7f9568bc3700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95640ad7c0 con 0x7f95641970d0 2021-02-10T11:32:35.205 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:35.200 7f9568bc3700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f956418ec70 con 0x7f956418f0a0 2021-02-10T11:32:35.206 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:35.200 7f956305f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9564094650 0x7f9564094a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:35.206 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:35.200 7f956305f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9564094650 0x7f9564094a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:34586/0 (socket says 172.21.2.102:34586) 2021-02-10T11:32:35.206 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:35.200 7f956305f700 1 -- 172.21.2.102:0/1376829422 learned_addr learned my addr 172.21.2.102:0/1376829422 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:32:35.206 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:35.200 7f956305f700 1 -- 172.21.2.102:0/1376829422 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f956418f0a0 msgr2=0x7f9564196ab0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:32:35.206 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:35.200 7f956305f700 1 --2- 172.21.2.102:0/1376829422 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f956418f0a0 0x7f9564196ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:35.206 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:35.200 7f956305f700 1 -- 172.21.2.102:0/1376829422 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f95641970d0 msgr2=0x7f95641974d0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:32:35.206 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:35.200 7f956305f700 1 --2- 172.21.2.102:0/1376829422 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f95641970d0 0x7f95641974d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:35.206 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:35.200 7f956305f700 1 -- 172.21.2.102:0/1376829422 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9564095070 con 0x7f9564094650 2021-02-10T11:32:35.206 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:35.200 7f956305f700 1 --2- 172.21.2.102:0/1376829422 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9564094650 0x7f9564094a50 secure :-1 s=READY pgs=2292 cs=0 l=1 rev1=1 rx=0x7f955c009410 tx=0x7f955c00bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:35.206 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:35.201 7f955b7fe700 1 -- 172.21.2.102:0/1376829422 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f955c009450 con 0x7f9564094650 2021-02-10T11:32:35.206 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:35.201 7f955b7fe700 1 -- 172.21.2.102:0/1376829422 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f955c006900 con 0x7f9564094650 2021-02-10T11:32:35.207 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:35.201 7f955b7fe700 1 -- 172.21.2.102:0/1376829422 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f955c00c4e0 con 0x7f9564094650 2021-02-10T11:32:35.207 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:35.201 7f9568bc3700 1 -- 172.21.2.102:0/1376829422 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9564095070 con 0x7f9564094650 2021-02-10T11:32:35.207 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:35.201 7f9568bc3700 1 -- 172.21.2.102:0/1376829422 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9564095600 con 0x7f9564094650 2021-02-10T11:32:35.207 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:35.201 7f955b7fe700 1 -- 172.21.2.102:0/1376829422 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f955c001270 con 0x7f9564094650 2021-02-10T11:32:35.207 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:35.201 7f955b7fe700 1 --2- 172.21.2.102:0/1376829422 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f95480186c0 0x7f954801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:35.207 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:35.201 7f955b7fe700 1 -- 172.21.2.102:0/1376829422 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 4753+0+0 (secure 0 0 0) 0x7f955c0325c0 con 0x7f9564094650 2021-02-10T11:32:35.207 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:35.202 7f956285e700 1 --2- 172.21.2.102:0/1376829422 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f95480186c0 0x7f954801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:35.207 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:35.202 7f956285e700 1 --2- 172.21.2.102:0/1376829422 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f95480186c0 0x7f954801abc0 crc :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:35.235 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:35.233 7f9568bc3700 1 -- 172.21.2.102:0/1376829422 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95640ad790 con 0x7f9564094650 2021-02-10T11:32:35.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:35.236 7f955b7fe700 1 -- 172.21.2.102:0/1376829422 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f955c009450 con 0x7f9564094650 2021-02-10T11:32:35.360 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:35.359 7f9568bc3700 1 -- 172.21.2.102:0/1376829422 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f95640ad790 con 0x7f9564094650 2021-02-10T11:32:35.361 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:35.359 7f955b7fe700 1 -- 172.21.2.102:0/1376829422 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v116) v1 ==== 84+0+64 (secure 0 0 0) 0x7f955c001270 con 0x7f9564094650 2021-02-10T11:32:35.361 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:32:35.361 INFO:teuthology.orchestra.run.gibba002.stdout:[ 2021-02-10T11:32:35.361 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:32:35.361 INFO:teuthology.orchestra.run.gibba002.stdout: "poolnum": 1, 2021-02-10T11:32:35.362 INFO:teuthology.orchestra.run.gibba002.stdout: "poolname": "rbd" 2021-02-10T11:32:35.362 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:32:35.362 INFO:teuthology.orchestra.run.gibba002.stdout:] 2021-02-10T11:32:35.363 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:35.361 7f9568bc3700 1 -- 172.21.2.102:0/1376829422 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f95480186c0 msgr2=0x7f954801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:35.363 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:35.361 7f9568bc3700 1 --2- 172.21.2.102:0/1376829422 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f95480186c0 0x7f954801abc0 crc :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:35.363 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:35.361 7f9568bc3700 1 -- 172.21.2.102:0/1376829422 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9564094650 msgr2=0x7f9564094a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:35.363 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:35.361 7f9568bc3700 1 --2- 172.21.2.102:0/1376829422 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9564094650 0x7f9564094a50 secure :-1 s=READY pgs=2292 cs=0 l=1 rev1=1 rx=0x7f955c009410 tx=0x7f955c00bd10).stop 2021-02-10T11:32:35.364 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:35.361 7f9568bc3700 1 -- 172.21.2.102:0/1376829422 shutdown_connections 2021-02-10T11:32:35.364 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:35.361 7f9568bc3700 1 --2- 172.21.2.102:0/1376829422 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9564094650 0x7f9564094a50 unknown :-1 s=CLOSED pgs=2292 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:35.364 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:35.361 7f9568bc3700 1 --2- 172.21.2.102:0/1376829422 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f95480186c0 0x7f954801abc0 unknown :-1 s=CLOSED pgs=1007 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:35.364 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:35.361 7f9568bc3700 1 --2- 172.21.2.102:0/1376829422 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f95641970d0 0x7f95641974d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:35.364 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:35.362 7f9568bc3700 1 --2- 172.21.2.102:0/1376829422 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f956418f0a0 0x7f9564196ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:35.364 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:35.362 7f9568bc3700 1 -- 172.21.2.102:0/1376829422 >> 172.21.2.102:0/1376829422 conn(0x7f95641a38c0 msgr2=0x7f95641a2f00 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:35.364 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:35.362 7f9568bc3700 1 -- 172.21.2.102:0/1376829422 shutdown_connections 2021-02-10T11:32:35.364 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:35.362 7f9568bc3700 1 -- 172.21.2.102:0/1376829422 wait complete. 2021-02-10T11:32:35.371 INFO:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2021-02-10T11:32:35.372 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool create cephfs_metadata 8 2021-02-10T11:32:35.802 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:35.785 7f5e142ee700 1 Processor -- start 2021-02-10T11:32:35.803 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:35.785 7f5e142ee700 1 -- start start 2021-02-10T11:32:35.803 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:35.785 7f5e142ee700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f5e0c0d1860 0x7f5e0c1b04b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:35.803 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:35.785 7f5e142ee700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f5e0c0cff60 0x7f5e0c109120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:35.803 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:35.785 7f5e142ee700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e0c0fe1b0 con 0x7f5e0c1b0ad0 2021-02-10T11:32:35.803 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:35.785 7f5e142ee700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e0c109ac0 con 0x7f5e0c0d1860 2021-02-10T11:32:35.803 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:35.785 7f5e142ee700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e0c109740 con 0x7f5e0c0cff60 2021-02-10T11:32:35.803 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:35.786 7f5e11909700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f5e0c1b0ad0 0x7f5e0c108930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:55050/0 (socket says 172.21.2.102:55050) 2021-02-10T11:32:35.803 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:35.786 7f5e11909700 1 -- 172.21.2.102:0/316726546 learned_addr learned my addr 172.21.2.102:0/316726546 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:32:35.803 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:35.786 7f5e1210a700 1 --2- 172.21.2.102:0/316726546 >> v2:172.21.2.102:3300/0 conn(0x7f5e0c0d1860 0x7f5e0c1b04b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:35.804 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:35.786 7f5e11108700 1 -- 172.21.2.102:0/316726546 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2932126646 0 0) 0x7f5e000009b0 con 0x7f5e0c1b0ad0 2021-02-10T11:32:35.804 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:35.786 7f5e11108700 1 -- 172.21.2.102:0/316726546 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5df8002fe0 con 0x7f5e0c1b0ad0 2021-02-10T11:32:35.804 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:35.786 7f5e1210a700 1 -- 172.21.2.102:0/316726546 >> v2:172.21.2.113:3300/0 conn(0x7f5e0c0cff60 msgr2=0x7f5e0c109120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:32:35.804 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:35.786 7f5e1210a700 1 --2- 172.21.2.102:0/316726546 >> v2:172.21.2.113:3300/0 conn(0x7f5e0c0cff60 0x7f5e0c109120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:35.804 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:35.786 7f5e1210a700 1 -- 172.21.2.102:0/316726546 >> v1:172.21.2.102:6789/0 conn(0x7f5e0c1b0ad0 legacy=0x7f5e0c108930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:35.804 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:35.786 7f5e1210a700 1 -- 172.21.2.102:0/316726546 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e0c088070 con 0x7f5e0c0d1860 2021-02-10T11:32:35.804 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:35.786 7f5e1210a700 1 --2- 172.21.2.102:0/316726546 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5e0c0d1860 0x7f5e0c1b04b0 secure :-1 s=READY pgs=2294 cs=0 l=1 rev1=1 rx=0x7f5e08006650 tx=0x7f5e08009a80).ready entity=mon.0 client_cookie=fa27fddc467b09f9 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:35.804 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:35.787 7f5e11108700 1 -- 172.21.2.102:0/316726546 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5e0800c0c0 con 0x7f5e0c0d1860 2021-02-10T11:32:35.804 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:35.787 7f5e11108700 1 -- 172.21.2.102:0/316726546 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e0800c340 con 0x7f5e0c0d1860 2021-02-10T11:32:35.804 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:35.787 7f5e11108700 1 -- 172.21.2.102:0/316726546 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5e0800c860 con 0x7f5e0c0d1860 2021-02-10T11:32:35.804 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:35.787 7f5e142ee700 1 -- 172.21.2.102:0/316726546 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5e0c0d1860 msgr2=0x7f5e0c1b04b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:35.804 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:35.787 7f5e142ee700 1 --2- 172.21.2.102:0/316726546 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5e0c0d1860 0x7f5e0c1b04b0 secure :-1 s=READY pgs=2294 cs=0 l=1 rev1=1 rx=0x7f5e08006650 tx=0x7f5e08009a80).stop 2021-02-10T11:32:35.804 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:35.787 7f5e142ee700 1 -- 172.21.2.102:0/316726546 shutdown_connections 2021-02-10T11:32:35.805 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:35.787 7f5e142ee700 1 --2- 172.21.2.102:0/316726546 >> v2:172.21.2.113:3300/0 conn(0x7f5e0c0cff60 0x7f5e0c109120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:35.805 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:35.787 7f5e142ee700 1 --2- 172.21.2.102:0/316726546 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5e0c0d1860 0x7f5e0c1b04b0 unknown :-1 s=CLOSED pgs=2294 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:35.805 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:35.787 7f5e142ee700 1 -- 172.21.2.102:0/316726546 >> 172.21.2.102:0/316726546 conn(0x7f5e0c0ce410 msgr2=0x7f5e0c0d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:35.805 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:35.787 7f5e142ee700 1 -- 172.21.2.102:0/316726546 shutdown_connections 2021-02-10T11:32:35.805 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:35.787 7f5e142ee700 1 -- 172.21.2.102:0/316726546 wait complete. 2021-02-10T11:32:35.806 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:35.802 7f5e142ee700 1 Processor -- start 2021-02-10T11:32:35.806 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:35.802 7f5e142ee700 1 -- start start 2021-02-10T11:32:35.806 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:35.802 7f5e142ee700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5e0c0c2a30 0x7f5e0c0c2e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:35.806 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:35.802 7f5e142ee700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5e0c0a7950 0x7f5e0c0a7d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:35.807 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:35.802 7f5e142ee700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5e0c1a71f0 0x7f5e0c1a75f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:35.807 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:35.802 7f5e142ee700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e0c0a8440 con 0x7f5e0c1a71f0 2021-02-10T11:32:35.807 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:35.802 7f5e142ee700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e0c0c3790 con 0x7f5e0c0a7950 2021-02-10T11:32:35.807 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:35.802 7f5e142ee700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e0c1a7c10 con 0x7f5e0c0c2a30 2021-02-10T11:32:35.807 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:35.802 7f5e1290b700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5e0c1a71f0 0x7f5e0c1a75f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:35.807 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:35.802 7f5e1290b700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5e0c1a71f0 0x7f5e0c1a75f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:34602/0 (socket says 172.21.2.102:34602) 2021-02-10T11:32:35.807 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:35.802 7f5e1290b700 1 -- 172.21.2.102:0/701882212 learned_addr learned my addr 172.21.2.102:0/701882212 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:32:35.807 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:35.803 7f5e1290b700 1 -- 172.21.2.102:0/701882212 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5e0c0c2a30 msgr2=0x7f5e0c0c2e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:35.807 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:35.803 7f5e1290b700 1 --2- 172.21.2.102:0/701882212 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5e0c0c2a30 0x7f5e0c0c2e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:35.807 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:35.803 7f5e1290b700 1 -- 172.21.2.102:0/701882212 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5e0c0a7950 msgr2=0x7f5e0c0a7d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:35.807 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:35.803 7f5e1290b700 1 --2- 172.21.2.102:0/701882212 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5e0c0a7950 0x7f5e0c0a7d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:35.807 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:35.803 7f5e1290b700 1 -- 172.21.2.102:0/701882212 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e0c1a7e30 con 0x7f5e0c1a71f0 2021-02-10T11:32:35.807 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:35.803 7f5e1290b700 1 --2- 172.21.2.102:0/701882212 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5e0c1a71f0 0x7f5e0c1a75f0 secure :-1 s=READY pgs=2295 cs=0 l=1 rev1=1 rx=0x7f5e04009980 tx=0x7f5e0400bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:35.808 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:35.803 7f5dfeffd700 1 -- 172.21.2.102:0/701882212 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5e04003020 con 0x7f5e0c1a71f0 2021-02-10T11:32:35.808 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:35.803 7f5dfeffd700 1 -- 172.21.2.102:0/701882212 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e04005a80 con 0x7f5e0c1a71f0 2021-02-10T11:32:35.808 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:35.803 7f5dfeffd700 1 -- 172.21.2.102:0/701882212 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5e04006030 con 0x7f5e0c1a71f0 2021-02-10T11:32:35.808 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:35.803 7f5e142ee700 1 -- 172.21.2.102:0/701882212 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e0c1a7e30 con 0x7f5e0c1a71f0 2021-02-10T11:32:35.808 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:35.803 7f5e142ee700 1 -- 172.21.2.102:0/701882212 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e0c1a7e30 con 0x7f5e0c1a71f0 2021-02-10T11:32:35.808 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:35.804 7f5dfeffd700 1 -- 172.21.2.102:0/701882212 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f5e04006030 con 0x7f5e0c1a71f0 2021-02-10T11:32:35.808 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:35.804 7f5dfeffd700 1 --2- 172.21.2.102:0/701882212 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5df40186c0 0x7f5df401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:35.808 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:35.804 7f5dfeffd700 1 -- 172.21.2.102:0/701882212 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 4753+0+0 (secure 0 0 0) 0x7f5e040326b0 con 0x7f5e0c1a71f0 2021-02-10T11:32:35.808 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:35.804 7f5e1210a700 1 --2- 172.21.2.102:0/701882212 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5df40186c0 0x7f5df401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:35.808 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:35.804 7f5e1210a700 1 --2- 172.21.2.102:0/701882212 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5df40186c0 0x7f5df401abc0 crc :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:35.837 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:35.835 7f5e142ee700 1 -- 172.21.2.102:0/701882212 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e0c082a20 con 0x7f5e0c1a71f0 2021-02-10T11:32:35.839 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:35.838 7f5dfeffd700 1 -- 172.21.2.102:0/701882212 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f5e04005870 con 0x7f5e0c1a71f0 2021-02-10T11:32:35.969 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:35.968 7f5e142ee700 1 -- 172.21.2.102:0/701882212 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"} v 0) v1 -- 0x7f5e0c082a20 con 0x7f5e0c1a71f0 2021-02-10T11:32:36.382 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:36.380 7f5dfeffd700 1 -- 172.21.2.102:0/701882212 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"}]=0 pool 'cephfs_metadata' created v117) v1 ==== 133+0+0 (secure 0 0 0) 0x7f5e04006030 con 0x7f5e0c1a71f0 2021-02-10T11:32:36.385 INFO:teuthology.orchestra.run.gibba002.stderr:pool 'cephfs_metadata' created 2021-02-10T11:32:36.388 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:36.387 7f5e142ee700 1 -- 172.21.2.102:0/701882212 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5df40186c0 msgr2=0x7f5df401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:36.388 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:36.387 7f5e142ee700 1 --2- 172.21.2.102:0/701882212 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5df40186c0 0x7f5df401abc0 crc :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:36.388 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:36.387 7f5e142ee700 1 -- 172.21.2.102:0/701882212 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5e0c1a71f0 msgr2=0x7f5e0c1a75f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:36.388 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:36.387 7f5e142ee700 1 --2- 172.21.2.102:0/701882212 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5e0c1a71f0 0x7f5e0c1a75f0 secure :-1 s=READY pgs=2295 cs=0 l=1 rev1=1 rx=0x7f5e04009980 tx=0x7f5e0400bcd0).stop 2021-02-10T11:32:36.389 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:36.387 7f5e142ee700 1 -- 172.21.2.102:0/701882212 shutdown_connections 2021-02-10T11:32:36.389 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:36.387 7f5e142ee700 1 --2- 172.21.2.102:0/701882212 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5e0c1a71f0 0x7f5e0c1a75f0 unknown :-1 s=CLOSED pgs=2295 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:36.389 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:36.387 7f5e142ee700 1 --2- 172.21.2.102:0/701882212 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5df40186c0 0x7f5df401abc0 unknown :-1 s=CLOSED pgs=1008 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:36.389 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:36.387 7f5e142ee700 1 --2- 172.21.2.102:0/701882212 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5e0c0a7950 0x7f5e0c0a7d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:36.389 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:36.387 7f5e142ee700 1 --2- 172.21.2.102:0/701882212 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5e0c0c2a30 0x7f5e0c0c2e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:36.389 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:36.387 7f5e142ee700 1 -- 172.21.2.102:0/701882212 >> 172.21.2.102:0/701882212 conn(0x7f5e0c0b5f10 msgr2=0x7f5e0c09f200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:36.389 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:36.387 7f5e142ee700 1 -- 172.21.2.102:0/701882212 shutdown_connections 2021-02-10T11:32:36.389 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:36.387 7f5e142ee700 1 -- 172.21.2.102:0/701882212 wait complete. 2021-02-10T11:32:36.397 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool create cephfs_data 8 2021-02-10T11:32:36.828 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:36.810 7fe5c5d28700 1 Processor -- start 2021-02-10T11:32:36.828 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:36.810 7fe5c5d28700 1 -- start start 2021-02-10T11:32:36.828 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:36.810 7fe5c5d28700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fe5c010afe0 0x7fe5c00cf740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:36.828 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:36.810 7fe5c5d28700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5c00fe1b0 con 0x7fe5c00d1c60 2021-02-10T11:32:36.829 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:36.810 7fe5c5d28700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5c01b0ca0 con 0x7fe5c00d1860 2021-02-10T11:32:36.829 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:36.810 7fe5c5d28700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5c0109a50 con 0x7fe5c010afe0 2021-02-10T11:32:36.829 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:36.811 7fe5bffff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fe5c010afe0 0x7fe5c00cf740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:36.829 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:36.811 7fe5beffd700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fe5c00d1c60 0x7fe5c00cef50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:55062/0 (socket says 172.21.2.102:55062) 2021-02-10T11:32:36.829 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:36.811 7fe5beffd700 1 -- 172.21.2.102:0/733042075 learned_addr learned my addr 172.21.2.102:0/733042075 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:32:36.829 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:36.811 7fe5bffff700 1 -- 172.21.2.102:0/733042075 >> v1:172.21.2.113:6789/0 conn(0x7fe5c00d1860 legacy=0x7fe5c01b04b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:32:36.830 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:36.811 7fe5bffff700 1 -- 172.21.2.102:0/733042075 >> v1:172.21.2.102:6789/0 conn(0x7fe5c00d1c60 legacy=0x7fe5c00cef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:36.830 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:36.811 7fe5be7fc700 1 -- 172.21.2.102:0/733042075 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 341587053 0 0) 0x7fe5a8001180 con 0x7fe5c00d1c60 2021-02-10T11:32:36.830 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:36.811 7fe5bffff700 1 -- 172.21.2.102:0/733042075 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5c010b3e0 con 0x7fe5c010afe0 2021-02-10T11:32:36.830 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:36.811 7fe5bffff700 1 --2- 172.21.2.102:0/733042075 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe5c010afe0 0x7fe5c00cf740 secure :-1 s=READY pgs=2297 cs=0 l=1 rev1=1 rx=0x7fe5b8006e30 tx=0x7fe5b8009de0).ready entity=mon.0 client_cookie=59b954bbc5f014fa server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:36.830 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:36.811 7fe5be7fc700 1 -- 172.21.2.102:0/733042075 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe5b800b300 con 0x7fe5c010afe0 2021-02-10T11:32:36.830 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:36.811 7fe5be7fc700 1 -- 172.21.2.102:0/733042075 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5b800b680 con 0x7fe5c010afe0 2021-02-10T11:32:36.830 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:36.811 7fe5be7fc700 1 -- 172.21.2.102:0/733042075 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe5b800bae0 con 0x7fe5c010afe0 2021-02-10T11:32:36.830 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:36.811 7fe5c5d28700 1 -- 172.21.2.102:0/733042075 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe5c010afe0 msgr2=0x7fe5c00cf740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:36.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:36.811 7fe5c5d28700 1 --2- 172.21.2.102:0/733042075 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe5c010afe0 0x7fe5c00cf740 secure :-1 s=READY pgs=2297 cs=0 l=1 rev1=1 rx=0x7fe5b8006e30 tx=0x7fe5b8009de0).stop 2021-02-10T11:32:36.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:36.812 7fe5c5d28700 1 -- 172.21.2.102:0/733042075 shutdown_connections 2021-02-10T11:32:36.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:36.812 7fe5c5d28700 1 --2- 172.21.2.102:0/733042075 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe5c010afe0 0x7fe5c00cf740 unknown :-1 s=CLOSED pgs=2297 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:36.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:36.812 7fe5c5d28700 1 -- 172.21.2.102:0/733042075 >> 172.21.2.102:0/733042075 conn(0x7fe5c00ce410 msgr2=0x7fe5c00d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:36.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:36.812 7fe5c5d28700 1 -- 172.21.2.102:0/733042075 shutdown_connections 2021-02-10T11:32:36.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:36.812 7fe5c5d28700 1 -- 172.21.2.102:0/733042075 wait complete. 2021-02-10T11:32:36.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:36.827 7fe5c5d28700 1 Processor -- start 2021-02-10T11:32:36.833 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:36.827 7fe5c5d28700 1 -- start start 2021-02-10T11:32:36.833 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:36.827 7fe5c5d28700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe5c00bfa10 0x7fe5c00b82a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:36.833 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:36.827 7fe5c5d28700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe5c01a4a70 0x7fe5c0094c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:36.833 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:36.827 7fe5c5d28700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe5c00c0410 0x7fe5c00c0810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:36.834 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:36.827 7fe5c5d28700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5c00c3a50 con 0x7fe5c00bfa10 2021-02-10T11:32:36.834 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:36.827 7fe5c5d28700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5c00b88c0 con 0x7fe5c00c0410 2021-02-10T11:32:36.834 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:36.827 7fe5c5d28700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5c0095260 con 0x7fe5c01a4a70 2021-02-10T11:32:36.834 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:36.827 7fe5bf7fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe5c00bfa10 0x7fe5c00b82a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:36.834 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:36.827 7fe5bf7fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe5c00bfa10 0x7fe5c00b82a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:34612/0 (socket says 172.21.2.102:34612) 2021-02-10T11:32:36.834 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:36.827 7fe5bf7fe700 1 -- 172.21.2.102:0/3236870732 learned_addr learned my addr 172.21.2.102:0/3236870732 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:32:36.835 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:36.828 7fe5bf7fe700 1 -- 172.21.2.102:0/3236870732 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe5c01a4a70 msgr2=0x7fe5c0094c40 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:32:36.835 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:36.828 7fe5bf7fe700 1 --2- 172.21.2.102:0/3236870732 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe5c01a4a70 0x7fe5c0094c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:36.835 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:36.828 7fe5bf7fe700 1 -- 172.21.2.102:0/3236870732 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe5c00c0410 msgr2=0x7fe5c00c0810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:36.835 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:36.828 7fe5bf7fe700 1 --2- 172.21.2.102:0/3236870732 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe5c00c0410 0x7fe5c00c0810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:36.835 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:36.828 7fe5bf7fe700 1 -- 172.21.2.102:0/3236870732 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5c00c3540 con 0x7fe5c00bfa10 2021-02-10T11:32:36.835 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:36.828 7fe5bf7fe700 1 --2- 172.21.2.102:0/3236870732 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe5c00bfa10 0x7fe5c00b82a0 secure :-1 s=READY pgs=2298 cs=0 l=1 rev1=1 rx=0x7fe5b000b530 tx=0x7fe5b000b8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:36.835 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:36.828 7fe5b7fff700 1 -- 172.21.2.102:0/3236870732 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe5b000bcc0 con 0x7fe5c00bfa10 2021-02-10T11:32:36.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:36.828 7fe5b7fff700 1 -- 172.21.2.102:0/3236870732 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5b0006020 con 0x7fe5c00bfa10 2021-02-10T11:32:36.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:36.828 7fe5b7fff700 1 -- 172.21.2.102:0/3236870732 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe5b00054a0 con 0x7fe5c00bfa10 2021-02-10T11:32:36.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:36.828 7fe5c5d28700 1 -- 172.21.2.102:0/3236870732 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5c00c3540 con 0x7fe5c00bfa10 2021-02-10T11:32:36.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:36.828 7fe5c5d28700 1 -- 172.21.2.102:0/3236870732 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5c00c1120 con 0x7fe5c00bfa10 2021-02-10T11:32:36.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:36.829 7fe5b7fff700 1 -- 172.21.2.102:0/3236870732 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fe5b0006200 con 0x7fe5c00bfa10 2021-02-10T11:32:36.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:36.829 7fe5b7fff700 1 --2- 172.21.2.102:0/3236870732 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe5a40186c0 0x7fe5a401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:36.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:36.829 7fe5b7fff700 1 -- 172.21.2.102:0/3236870732 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 5107+0+0 (secure 0 0 0) 0x7fe5b00323b0 con 0x7fe5c00bfa10 2021-02-10T11:32:36.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:36.829 7fe5beffd700 1 --2- 172.21.2.102:0/3236870732 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe5a40186c0 0x7fe5a401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:36.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:36.829 7fe5beffd700 1 --2- 172.21.2.102:0/3236870732 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe5a40186c0 0x7fe5a401abc0 crc :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:36.862 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:36.860 7fe5c5d28700 1 -- 172.21.2.102:0/3236870732 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5c00bfee0 con 0x7fe5c00bfa10 2021-02-10T11:32:36.864 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:36.863 7fe5b7fff700 1 -- 172.21.2.102:0/3236870732 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fe5b0000c10 con 0x7fe5c00bfa10 2021-02-10T11:32:36.994 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:36.992 7fe5c5d28700 1 -- 172.21.2.102:0/3236870732 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_data"} v 0) v1 -- 0x7fe5c00bfee0 con 0x7fe5c00bfa10 2021-02-10T11:32:37.385 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:37.383 7fe5b7fff700 1 -- 172.21.2.102:0/3236870732 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_data"}]=0 pool 'cephfs_data' created v118) v1 ==== 125+0+0 (secure 0 0 0) 0x7fe5b0006200 con 0x7fe5c00bfa10 2021-02-10T11:32:37.410 INFO:teuthology.orchestra.run.gibba002.stderr:pool 'cephfs_data' created 2021-02-10T11:32:37.412 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:37.410 7fe5c5d28700 1 -- 172.21.2.102:0/3236870732 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe5a40186c0 msgr2=0x7fe5a401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:37.412 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:37.410 7fe5c5d28700 1 --2- 172.21.2.102:0/3236870732 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe5a40186c0 0x7fe5a401abc0 crc :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:37.412 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:37.410 7fe5c5d28700 1 -- 172.21.2.102:0/3236870732 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe5c00bfa10 msgr2=0x7fe5c00b82a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:37.412 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:37.410 7fe5c5d28700 1 --2- 172.21.2.102:0/3236870732 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe5c00bfa10 0x7fe5c00b82a0 secure :-1 s=READY pgs=2298 cs=0 l=1 rev1=1 rx=0x7fe5b000b530 tx=0x7fe5b000b8d0).stop 2021-02-10T11:32:37.412 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:37.410 7fe5c5d28700 1 -- 172.21.2.102:0/3236870732 shutdown_connections 2021-02-10T11:32:37.412 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:37.410 7fe5c5d28700 1 --2- 172.21.2.102:0/3236870732 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe5a40186c0 0x7fe5a401abc0 unknown :-1 s=CLOSED pgs=1009 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:37.412 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:37.410 7fe5c5d28700 1 --2- 172.21.2.102:0/3236870732 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe5c00c0410 0x7fe5c00c0810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:37.412 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:37.410 7fe5c5d28700 1 --2- 172.21.2.102:0/3236870732 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe5c01a4a70 0x7fe5c0094c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:37.412 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:37.410 7fe5c5d28700 1 --2- 172.21.2.102:0/3236870732 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe5c00bfa10 0x7fe5c00b82a0 unknown :-1 s=CLOSED pgs=2298 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:37.412 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:37.410 7fe5c5d28700 1 -- 172.21.2.102:0/3236870732 >> 172.21.2.102:0/3236870732 conn(0x7fe5c00a6a80 msgr2=0x7fe5c00ae110 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:37.413 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:37.410 7fe5c5d28700 1 -- 172.21.2.102:0/3236870732 shutdown_connections 2021-02-10T11:32:37.413 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:37.410 7fe5c5d28700 1 -- 172.21.2.102:0/3236870732 wait complete. 2021-02-10T11:32:37.420 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2021-02-10T11:32:37.852 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:37.834 7fbda9f93700 1 Processor -- start 2021-02-10T11:32:37.852 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:37.834 7fbda9f93700 1 -- start start 2021-02-10T11:32:37.852 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:37.835 7fbda9f93700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fbda40d1c80 0x7fbda40cef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:37.852 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:37.835 7fbda9f93700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fbda40cf590 0x7fbda40cf990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:37.852 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:37.835 7fbda9f93700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbda40fe1d0 con 0x7fbda40d1880 2021-02-10T11:32:37.852 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:37.835 7fbda9f93700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbda41b0cc0 con 0x7fbda40cf590 2021-02-10T11:32:37.852 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:37.835 7fbda9f93700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbda4109a70 con 0x7fbda40d1c80 2021-02-10T11:32:37.852 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:37.835 7fbda3fff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fbda40cf590 0x7fbda40cf990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:37.853 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:37.835 7fbda3fff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fbda40cf590 0x7fbda40cf990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:34626/0 (socket says 172.21.2.102:34626) 2021-02-10T11:32:37.853 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:37.835 7fbda3fff700 1 -- 172.21.2.102:0/4193590206 learned_addr learned my addr 172.21.2.102:0/4193590206 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:32:37.853 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:37.835 7fbda3fff700 1 -- 172.21.2.102:0/4193590206 >> v2:172.21.2.113:3300/0 conn(0x7fbda40d1c80 msgr2=0x7fbda40cef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:32:37.853 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:37.835 7fbda3fff700 1 --2- 172.21.2.102:0/4193590206 >> v2:172.21.2.113:3300/0 conn(0x7fbda40d1c80 0x7fbda40cef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:37.853 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:37.835 7fbda3fff700 1 -- 172.21.2.102:0/4193590206 >> v1:172.21.2.113:6789/0 conn(0x7fbda40d1880 legacy=0x7fbda41b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:32:37.853 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:37.835 7fbda3fff700 1 -- 172.21.2.102:0/4193590206 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbda4088070 con 0x7fbda40cf590 2021-02-10T11:32:37.853 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:37.836 7fbda3fff700 1 --2- 172.21.2.102:0/4193590206 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbda40cf590 0x7fbda40cf990 secure :-1 s=READY pgs=2299 cs=0 l=1 rev1=1 rx=0x7fbd9c006f00 tx=0x7fbd9c009e90).ready entity=mon.0 client_cookie=49561a71426cc97c server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:37.853 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:37.836 7fbda27fc700 1 -- 172.21.2.102:0/4193590206 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbd9c00b300 con 0x7fbda40cf590 2021-02-10T11:32:37.853 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:37.836 7fbda27fc700 1 -- 172.21.2.102:0/4193590206 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd9c00b640 con 0x7fbda40cf590 2021-02-10T11:32:37.853 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:37.836 7fbda27fc700 1 -- 172.21.2.102:0/4193590206 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbd9c00ba90 con 0x7fbda40cf590 2021-02-10T11:32:37.853 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:37.836 7fbda9f93700 1 -- 172.21.2.102:0/4193590206 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbda40cf590 msgr2=0x7fbda40cf990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:37.853 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:37.836 7fbda9f93700 1 --2- 172.21.2.102:0/4193590206 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbda40cf590 0x7fbda40cf990 secure :-1 s=READY pgs=2299 cs=0 l=1 rev1=1 rx=0x7fbd9c006f00 tx=0x7fbd9c009e90).stop 2021-02-10T11:32:37.853 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:37.836 7fbda9f93700 1 -- 172.21.2.102:0/4193590206 shutdown_connections 2021-02-10T11:32:37.854 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:37.836 7fbda9f93700 1 --2- 172.21.2.102:0/4193590206 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbda40cf590 0x7fbda40cf990 unknown :-1 s=CLOSED pgs=2299 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:37.854 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:37.836 7fbda9f93700 1 --2- 172.21.2.102:0/4193590206 >> v2:172.21.2.113:3300/0 conn(0x7fbda40d1c80 0x7fbda40cef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:37.854 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:37.836 7fbda9f93700 1 -- 172.21.2.102:0/4193590206 >> 172.21.2.102:0/4193590206 conn(0x7fbda40ce430 msgr2=0x7fbda40d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:37.854 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:37.837 7fbda9f93700 1 -- 172.21.2.102:0/4193590206 shutdown_connections 2021-02-10T11:32:37.854 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:37.837 7fbda9f93700 1 -- 172.21.2.102:0/4193590206 wait complete. 2021-02-10T11:32:37.854 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:37.851 7fbda9f93700 1 Processor -- start 2021-02-10T11:32:37.855 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:37.851 7fbda9f93700 1 -- start start 2021-02-10T11:32:37.856 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:37.851 7fbda9f93700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbda419abc0 0x7fbda409f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:37.856 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:37.851 7fbda9f93700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbda409ebd0 0x7fbda4082cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:37.856 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:37.851 7fbda9f93700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fbda40832f0 0x7fbda419b570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:37.856 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:37.851 7fbda9f93700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbda419b070 con 0x7fbda409ebd0 2021-02-10T11:32:37.856 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:37.851 7fbda9f93700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbda409f010 con 0x7fbda40832f0 2021-02-10T11:32:37.856 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:37.851 7fbda9f93700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbda41aeb30 con 0x7fbda419abc0 2021-02-10T11:32:37.856 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:37.851 7fbda2ffd700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbda409ebd0 0x7fbda4082cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:37.856 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:37.851 7fbda2ffd700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbda409ebd0 0x7fbda4082cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:34630/0 (socket says 172.21.2.102:34630) 2021-02-10T11:32:37.856 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:37.851 7fbda2ffd700 1 -- 172.21.2.102:0/3212942155 learned_addr learned my addr 172.21.2.102:0/3212942155 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:32:37.856 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:37.852 7fbda2ffd700 1 -- 172.21.2.102:0/3212942155 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbda419abc0 msgr2=0x7fbda409f370 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:32:37.856 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:37.852 7fbda2ffd700 1 --2- 172.21.2.102:0/3212942155 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbda419abc0 0x7fbda409f370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:37.856 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:37.852 7fbda2ffd700 1 -- 172.21.2.102:0/3212942155 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fbda40832f0 msgr2=0x7fbda419b570 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:32:37.856 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:37.852 7fbda2ffd700 1 --2- 172.21.2.102:0/3212942155 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fbda40832f0 0x7fbda419b570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:37.857 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:37.852 7fbda2ffd700 1 -- 172.21.2.102:0/3212942155 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbda40836f0 con 0x7fbda409ebd0 2021-02-10T11:32:37.857 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:37.852 7fbda2ffd700 1 --2- 172.21.2.102:0/3212942155 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbda409ebd0 0x7fbda4082cd0 secure :-1 s=READY pgs=2300 cs=0 l=1 rev1=1 rx=0x7fbd94006f00 tx=0x7fbd9400b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:37.857 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:37.852 7fbd9bfff700 1 -- 172.21.2.102:0/3212942155 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbd9400bd30 con 0x7fbda409ebd0 2021-02-10T11:32:37.857 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:37.852 7fbd9bfff700 1 -- 172.21.2.102:0/3212942155 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd94000c90 con 0x7fbda409ebd0 2021-02-10T11:32:37.857 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:37.852 7fbda9f93700 1 -- 172.21.2.102:0/3212942155 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbda40836f0 con 0x7fbda409ebd0 2021-02-10T11:32:37.857 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:37.852 7fbd9bfff700 1 -- 172.21.2.102:0/3212942155 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbd940022a0 con 0x7fbda409ebd0 2021-02-10T11:32:37.857 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:37.852 7fbda9f93700 1 -- 172.21.2.102:0/3212942155 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbda40836f0 con 0x7fbda409ebd0 2021-02-10T11:32:37.857 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:37.853 7fbd9bfff700 1 -- 172.21.2.102:0/3212942155 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fbd94005310 con 0x7fbda409ebd0 2021-02-10T11:32:37.857 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:37.853 7fbd9bfff700 1 --2- 172.21.2.102:0/3212942155 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fbd880186d0 0x7fbd8801abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:37.857 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:37.853 7fbd9bfff700 1 -- 172.21.2.102:0/3212942155 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5457+0+0 (secure 0 0 0) 0x7fbd940325c0 con 0x7fbda409ebd0 2021-02-10T11:32:37.857 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:37.853 7fbda37fe700 1 --2- 172.21.2.102:0/3212942155 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fbd880186d0 0x7fbd8801abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:37.857 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:37.853 7fbda37fe700 1 --2- 172.21.2.102:0/3212942155 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fbd880186d0 0x7fbd8801abd0 crc :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:37.886 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:37.884 7fbda9f93700 1 -- 172.21.2.102:0/3212942155 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbda40827b0 con 0x7fbda409ebd0 2021-02-10T11:32:37.888 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:37.887 7fbd9bfff700 1 -- 172.21.2.102:0/3212942155 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fbd94006b90 con 0x7fbda409ebd0 2021-02-10T11:32:38.017 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:38.015 7fbda9f93700 1 -- 172.21.2.102:0/3212942155 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"data": "cephfs_data", "prefix": "fs new", "force": true, "fs_name": "cephfs", "metadata": "cephfs_metadata"} v 0) v1 -- 0x7fbda40827b0 con 0x7fbda409ebd0 2021-02-10T11:32:38.020 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:38.019 7fbd9bfff700 1 -- 172.21.2.102:0/3212942155 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"data": "cephfs_data", "prefix": "fs new", "force": true, "fs_name": "cephfs", "metadata": "cephfs_metadata"}]=0 new fs with metadata pool 24 and data pool 25 v96) v1 ==== 189+0+0 (secure 0 0 0) 0x7fbd94000c90 con 0x7fbda409ebd0 2021-02-10T11:32:38.024 INFO:teuthology.orchestra.run.gibba002.stderr:new fs with metadata pool 24 and data pool 25 2021-02-10T11:32:38.026 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:38.025 7fbda9f93700 1 -- 172.21.2.102:0/3212942155 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fbd880186d0 msgr2=0x7fbd8801abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:38.026 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:38.025 7fbda9f93700 1 --2- 172.21.2.102:0/3212942155 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fbd880186d0 0x7fbd8801abd0 crc :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:38.026 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:38.025 7fbda9f93700 1 -- 172.21.2.102:0/3212942155 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbda409ebd0 msgr2=0x7fbda4082cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:38.026 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:38.025 7fbda9f93700 1 --2- 172.21.2.102:0/3212942155 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbda409ebd0 0x7fbda4082cd0 secure :-1 s=READY pgs=2300 cs=0 l=1 rev1=1 rx=0x7fbd94006f00 tx=0x7fbd9400b940).stop 2021-02-10T11:32:38.027 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:38.025 7fbda9f93700 1 -- 172.21.2.102:0/3212942155 shutdown_connections 2021-02-10T11:32:38.027 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:38.025 7fbda9f93700 1 --2- 172.21.2.102:0/3212942155 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fbd880186d0 0x7fbd8801abd0 unknown :-1 s=CLOSED pgs=1012 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:38.027 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:38.025 7fbda9f93700 1 --2- 172.21.2.102:0/3212942155 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fbda40832f0 0x7fbda419b570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:38.027 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:38.025 7fbda9f93700 1 --2- 172.21.2.102:0/3212942155 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbda409ebd0 0x7fbda4082cd0 unknown :-1 s=CLOSED pgs=2300 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:38.027 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:38.025 7fbda9f93700 1 --2- 172.21.2.102:0/3212942155 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbda419abc0 0x7fbda409f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:38.027 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:38.025 7fbda9f93700 1 -- 172.21.2.102:0/3212942155 >> 172.21.2.102:0/3212942155 conn(0x7fbda419d640 msgr2=0x7fbda4190160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:38.027 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:38.026 7fbda9f93700 1 -- 172.21.2.102:0/3212942155 shutdown_connections 2021-02-10T11:32:38.028 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:38.026 7fbda9f93700 1 -- 172.21.2.102:0/3212942155 wait complete. 2021-02-10T11:32:38.035 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:32:38.463 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:38.444 7f0f5af97700 1 Processor -- start 2021-02-10T11:32:38.464 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:38.444 7f0f5af97700 1 -- start start 2021-02-10T11:32:38.464 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:38.444 7f0f5af97700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0f540d0340 0x7f0f540ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:38.464 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:38.444 7f0f5af97700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f0f540d1c70 0x7f0f540efc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:38.464 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:38.444 7f0f5af97700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f540f0b10 con 0x7f0f540d1870 2021-02-10T11:32:38.464 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:38.444 7f0f5af97700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f540e5ed0 con 0x7f0f540d1c70 2021-02-10T11:32:38.464 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:38.444 7f0f5af97700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f541b0be0 con 0x7f0f540d0340 2021-02-10T11:32:38.464 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:38.445 7f0f595b4700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f0f540d1c70 0x7f0f540efc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:38.464 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:38.445 7f0f53fff700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0f540d0340 0x7f0f540ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:38.465 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:38.445 7f0f58db3700 1 --1- >> v1:172.21.2.113:6789/0 conn(0x7f0f540d1870 0x7f0f541b03f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.113:6789/0 says I am v1:172.21.2.102:34184/0 (socket says 172.21.2.102:34184) 2021-02-10T11:32:38.465 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:38.445 7f0f58db3700 1 -- 172.21.2.102:0/1383875025 learned_addr learned my addr 172.21.2.102:0/1383875025 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:32:38.465 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:38.446 7f0f53fff700 1 -- 172.21.2.102:0/1383875025 >> v2:172.21.2.113:3300/0 conn(0x7f0f540d1c70 msgr2=0x7f0f540efc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:32:38.465 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:38.446 7f0f53fff700 1 --2- 172.21.2.102:0/1383875025 >> v2:172.21.2.113:3300/0 conn(0x7f0f540d1c70 0x7f0f540efc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:38.465 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:38.446 7f0f53fff700 1 -- 172.21.2.102:0/1383875025 >> v1:172.21.2.113:6789/0 conn(0x7f0f540d1870 legacy=0x7f0f541b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:38.465 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:38.446 7f0f537fe700 1 -- 172.21.2.102:0/1383875025 <== mon.1 v1:172.21.2.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3765014202 0 0) 0x7f0f3c0010c0 con 0x7f0f540d1870 2021-02-10T11:32:38.465 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:38.446 7f0f53fff700 1 -- 172.21.2.102:0/1383875025 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f540f20a0 con 0x7f0f540d0340 2021-02-10T11:32:38.465 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:38.446 7f0f595b4700 1 --2- 172.21.2.102:0/1383875025 >> v2:172.21.2.113:3300/0 conn(0x7f0f540d1c70 0x7f0f540efc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:32:38.465 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:38.446 7f0f53fff700 1 --2- 172.21.2.102:0/1383875025 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0f540d0340 0x7f0f540ef640 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7f0f440069d0 tx=0x7f0f44009e40).ready entity=mon.2 client_cookie=9237c23acc84b795 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:38.465 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:38.447 7f0f537fe700 1 -- 172.21.2.102:0/1383875025 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0f44000d30 con 0x7f0f540d0340 2021-02-10T11:32:38.465 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:38.447 7f0f537fe700 1 -- 172.21.2.102:0/1383875025 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f440066c0 con 0x7f0f540d0340 2021-02-10T11:32:38.466 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:38.447 7f0f537fe700 1 -- 172.21.2.102:0/1383875025 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0f4400b7d0 con 0x7f0f540d0340 2021-02-10T11:32:38.466 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:38.447 7f0f5af97700 1 -- 172.21.2.102:0/1383875025 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0f540d0340 msgr2=0x7f0f540ef640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:38.466 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:38.447 7f0f5af97700 1 --2- 172.21.2.102:0/1383875025 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0f540d0340 0x7f0f540ef640 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7f0f440069d0 tx=0x7f0f44009e40).stop 2021-02-10T11:32:38.466 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:38.447 7f0f5af97700 1 -- 172.21.2.102:0/1383875025 shutdown_connections 2021-02-10T11:32:38.466 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:38.447 7f0f5af97700 1 --2- 172.21.2.102:0/1383875025 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0f540d0340 0x7f0f540ef640 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:38.466 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:38.448 7f0f5af97700 1 --2- 172.21.2.102:0/1383875025 >> v2:172.21.2.113:3300/0 conn(0x7f0f540d1c70 0x7f0f540efc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:38.466 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:38.448 7f0f5af97700 1 -- 172.21.2.102:0/1383875025 >> 172.21.2.102:0/1383875025 conn(0x7f0f540ce450 msgr2=0x7f0f540d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:38.466 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:38.448 7f0f5af97700 1 -- 172.21.2.102:0/1383875025 shutdown_connections 2021-02-10T11:32:38.466 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:38.448 7f0f5af97700 1 -- 172.21.2.102:0/1383875025 wait complete. 2021-02-10T11:32:38.469 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:38.462 7f0f5af97700 1 Processor -- start 2021-02-10T11:32:38.469 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:38.462 7f0f5af97700 1 -- start start 2021-02-10T11:32:38.469 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:38.463 7f0f5af97700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0f54195650 0x7f0f540b5560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:38.469 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:38.463 7f0f5af97700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0f540b5b80 0x7f0f541a2e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:38.469 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:38.463 7f0f5af97700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0f540acfa0 0x7f0f541a3440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:38.469 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:38.463 7f0f5af97700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f540b6010 con 0x7f0f54195650 2021-02-10T11:32:38.469 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:38.463 7f0f5af97700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f541a3a60 con 0x7f0f540acfa0 2021-02-10T11:32:38.469 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:38.463 7f0f5af97700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f541a3c40 con 0x7f0f540b5b80 2021-02-10T11:32:38.469 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:38.463 7f0f58db3700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0f54195650 0x7f0f540b5560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:38.469 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:38.463 7f0f58db3700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0f54195650 0x7f0f540b5560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:34642/0 (socket says 172.21.2.102:34642) 2021-02-10T11:32:38.470 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:38.463 7f0f58db3700 1 -- 172.21.2.102:0/2930795604 learned_addr learned my addr 172.21.2.102:0/2930795604 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:32:38.470 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:38.463 7f0f595b4700 1 --2- 172.21.2.102:0/2930795604 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0f540acfa0 0x7f0f541a3440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:38.470 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:38.463 7f0f53fff700 1 --2- 172.21.2.102:0/2930795604 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0f540b5b80 0x7f0f541a2e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:38.470 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:38.464 7f0f58db3700 1 -- 172.21.2.102:0/2930795604 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0f540b5b80 msgr2=0x7f0f541a2e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:38.470 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:38.464 7f0f58db3700 1 --2- 172.21.2.102:0/2930795604 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0f540b5b80 0x7f0f541a2e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:38.470 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:38.464 7f0f58db3700 1 -- 172.21.2.102:0/2930795604 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0f540acfa0 msgr2=0x7f0f541a3440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:38.470 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:38.464 7f0f58db3700 1 --2- 172.21.2.102:0/2930795604 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0f540acfa0 0x7f0f541a3440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:38.470 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:38.464 7f0f58db3700 1 -- 172.21.2.102:0/2930795604 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f540bcae0 con 0x7f0f54195650 2021-02-10T11:32:38.470 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:38.464 7f0f58db3700 1 --2- 172.21.2.102:0/2930795604 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0f54195650 0x7f0f540b5560 secure :-1 s=READY pgs=2301 cs=0 l=1 rev1=1 rx=0x7f0f3c00a5d0 tx=0x7f0f3c00a970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:38.470 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:38.464 7f0f517fa700 1 -- 172.21.2.102:0/2930795604 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0f3c000c10 con 0x7f0f54195650 2021-02-10T11:32:38.470 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:38.464 7f0f517fa700 1 -- 172.21.2.102:0/2930795604 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f3c0070d0 con 0x7f0f54195650 2021-02-10T11:32:38.471 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:38.464 7f0f5af97700 1 -- 172.21.2.102:0/2930795604 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f540bcae0 con 0x7f0f54195650 2021-02-10T11:32:38.471 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:38.464 7f0f517fa700 1 -- 172.21.2.102:0/2930795604 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0f3c0047b0 con 0x7f0f54195650 2021-02-10T11:32:38.471 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:38.464 7f0f5af97700 1 -- 172.21.2.102:0/2930795604 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f540bd070 con 0x7f0f54195650 2021-02-10T11:32:38.471 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:38.465 7f0f517fa700 1 -- 172.21.2.102:0/2930795604 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f0f3c004740 con 0x7f0f54195650 2021-02-10T11:32:38.471 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:38.465 7f0f517fa700 1 --2- 172.21.2.102:0/2930795604 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0f380186c0 0x7f0f3801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:38.471 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:38.465 7f0f517fa700 1 -- 172.21.2.102:0/2930795604 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f0f3c032390 con 0x7f0f54195650 2021-02-10T11:32:38.471 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:38.465 7f0f53fff700 1 --2- 172.21.2.102:0/2930795604 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0f380186c0 0x7f0f3801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:38.471 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:38.466 7f0f53fff700 1 --2- 172.21.2.102:0/2930795604 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0f380186c0 0x7f0f3801abc0 crc :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:38.496 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:38.494 7f0f5af97700 1 -- 172.21.2.102:0/2930795604 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f540bcaa0 con 0x7f0f54195650 2021-02-10T11:32:38.498 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:38.497 7f0f517fa700 1 -- 172.21.2.102:0/2930795604 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f0f3c004530 con 0x7f0f54195650 2021-02-10T11:32:38.621 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:38.619 7f0f5af97700 1 -- 172.21.2.102:0/2930795604 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0f540bcaa0 con 0x7f0f54195650 2021-02-10T11:32:38.621 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:38.620 7f0f517fa700 1 -- 172.21.2.102:0/2930795604 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v119) v1 ==== 74+0+14020 (secure 0 0 0) 0x7f0f3c004740 con 0x7f0f54195650 2021-02-10T11:32:38.622 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:32:38.622 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":119,"fsid":"68db30ea-dbce-4aef-a6df-f01c6fea0513","created":"2021-02-10 11:03:06.331557","modified":"2021-02-10 11:32:38.017323","last_up_change":"2021-02-10 11:03:32.690562","last_in_change":"2021-02-10 11:03:32.690562","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":25,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:03:37.322519","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":24,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:32:35.986688","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"119","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":25,"pool_name":"cephfs_data","create_time":"2021-02-10 11:32:37.011649","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"119","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"be691b4d-a039-408a-af80-9d53f02b6f1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6818","nonce":36088},{"type":"v1","addr":"172.21.2.102:6820","nonce":36088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6822","nonce":36088},{"type":"v1","addr":"172.21.2.102:6824","nonce":36088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6830","nonce":36088},{"type":"v1","addr":"172.21.2.102:6832","nonce":36088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6826","nonce":36088},{"type":"v1","addr":"172.21.2.102:6828","nonce":36088}]},"public_addr":"172.21.2.102:6820/36088","cluster_addr":"172.21.2.102:6824/36088","heartbeat_back_addr":"172.21.2.102:6832/36088","heartbeat_front_addr":"172.21.2.102:6828/36088","state":["exists","up"]},{"osd":1,"uuid":"64e3ccc8-9c69-4d7c-865e-af8e83822635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6819","nonce":36087},{"type":"v1","addr":"172.21.2.102:6821","nonce":36087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6823","nonce":36087},{"type":"v1","addr":"172.21.2.102:6825","nonce":36087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6831","nonce":36087},{"type":"v1","addr":"172.21.2.102:6833","nonce":36087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6827","nonce":36087},{"type":"v1","addr":"172.21.2.102:6829","nonce":36087}]},"public_addr":"172.21.2.102:6821/36087","cluster_addr":"172.21.2.102:6825/36087","heartbeat_back_addr":"172.21.2.102:6833/36087","heartbeat_front_addr":"172.21.2.102:6829/36087","state":["exists","up"]},{"osd":2,"uuid":"153700af-747f-481f-bdf4-d6072d3a842f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6810","nonce":36089},{"type":"v1","addr":"172.21.2.102:6811","nonce":36089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6812","nonce":36089},{"type":"v1","addr":"172.21.2.102:6813","nonce":36089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6816","nonce":36089},{"type":"v1","addr":"172.21.2.102:6817","nonce":36089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6814","nonce":36089},{"type":"v1","addr":"172.21.2.102:6815","nonce":36089}]},"public_addr":"172.21.2.102:6811/36089","cluster_addr":"172.21.2.102:6813/36089","heartbeat_back_addr":"172.21.2.102:6817/36089","heartbeat_front_addr":"172.21.2.102:6815/36089","state":["exists","up"]},{"osd":3,"uuid":"cbc678f4-83a9-45e6-8830-2f7c232b1ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6802","nonce":36090},{"type":"v1","addr":"172.21.2.102:6803","nonce":36090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6804","nonce":36090},{"type":"v1","addr":"172.21.2.102:6805","nonce":36090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6808","nonce":36090},{"type":"v1","addr":"172.21.2.102:6809","nonce":36090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6806","nonce":36090},{"type":"v1","addr":"172.21.2.102:6807","nonce":36090}]},"public_addr":"172.21.2.102:6803/36090","cluster_addr":"172.21.2.102:6805/36090","heartbeat_back_addr":"172.21.2.102:6809/36090","heartbeat_front_addr":"172.21.2.102:6807/36090","state":["exists","up"]},{"osd":4,"uuid":"c01e4008-b258-4f10-afd2-9e7f37214351","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6808","nonce":36368},{"type":"v1","addr":"172.21.2.113:6811","nonce":36368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6814","nonce":36368},{"type":"v1","addr":"172.21.2.113:6818","nonce":36368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6826","nonce":36368},{"type":"v1","addr":"172.21.2.113:6828","nonce":36368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6820","nonce":36368},{"type":"v1","addr":"172.21.2.113:6824","nonce":36368}]},"public_addr":"172.21.2.113:6811/36368","cluster_addr":"172.21.2.113:6818/36368","heartbeat_back_addr":"172.21.2.113:6828/36368","heartbeat_front_addr":"172.21.2.113:6824/36368","state":["exists","up"]},{"osd":5,"uuid":"551f6007-daf8-4a98-8ee0-c2b16bcccb5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6802","nonce":36366},{"type":"v1","addr":"172.21.2.113:6805","nonce":36366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6807","nonce":36366},{"type":"v1","addr":"172.21.2.113:6810","nonce":36366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6819","nonce":36366},{"type":"v1","addr":"172.21.2.113:6822","nonce":36366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6812","nonce":36366},{"type":"v1","addr":"172.21.2.113:6815","nonce":36366}]},"public_addr":"172.21.2.113:6805/36366","cluster_addr":"172.21.2.113:6810/36366","heartbeat_back_addr":"172.21.2.113:6822/36366","heartbeat_front_addr":"172.21.2.113:6815/36366","state":["exists","up"]},{"osd":6,"uuid":"bf67c040-0e50-48be-b102-c3943bc9ebef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6800","nonce":36367},{"type":"v1","addr":"172.21.2.113:6801","nonce":36367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6803","nonce":36367},{"type":"v1","addr":"172.21.2.113:6804","nonce":36367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6813","nonce":36367},{"type":"v1","addr":"172.21.2.113:6816","nonce":36367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6806","nonce":36367},{"type":"v1","addr":"172.21.2.113:6809","nonce":36367}]},"public_addr":"172.21.2.113:6801/36367","cluster_addr":"172.21.2.113:6804/36367","heartbeat_back_addr":"172.21.2.113:6816/36367","heartbeat_front_addr":"172.21.2.113:6809/36367","state":["exists","up"]},{"osd":7,"uuid":"f5461f80-018d-428f-be89-2eb2bcbe7bb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6817","nonce":36365},{"type":"v1","addr":"172.21.2.113:6821","nonce":36365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6823","nonce":36365},{"type":"v1","addr":"172.21.2.113:6825","nonce":36365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6830","nonce":36365},{"type":"v1","addr":"172.21.2.113:6831","nonce":36365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6827","nonce":36365},{"type":"v1","addr":"172.21.2.113:6829","nonce":36365}]},"public_addr":"172.21.2.113:6821/36365","cluster_addr":"172.21.2.113:6825/36365","heartbeat_back_addr":"172.21.2.113:6831/36365","heartbeat_front_addr":"172.21.2.113:6829/36365","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:32:38.623 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:38.622 7f0f5af97700 1 -- 172.21.2.102:0/2930795604 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0f380186c0 msgr2=0x7f0f3801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:38.623 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:38.622 7f0f5af97700 1 --2- 172.21.2.102:0/2930795604 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0f380186c0 0x7f0f3801abc0 crc :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:38.623 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:38.622 7f0f5af97700 1 -- 172.21.2.102:0/2930795604 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0f54195650 msgr2=0x7f0f540b5560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:38.623 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:38.622 7f0f5af97700 1 --2- 172.21.2.102:0/2930795604 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0f54195650 0x7f0f540b5560 secure :-1 s=READY pgs=2301 cs=0 l=1 rev1=1 rx=0x7f0f3c00a5d0 tx=0x7f0f3c00a970).stop 2021-02-10T11:32:38.624 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:38.622 7f0f5af97700 1 -- 172.21.2.102:0/2930795604 shutdown_connections 2021-02-10T11:32:38.624 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:38.622 7f0f5af97700 1 --2- 172.21.2.102:0/2930795604 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0f380186c0 0x7f0f3801abc0 unknown :-1 s=CLOSED pgs=1013 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:38.624 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:38.622 7f0f5af97700 1 --2- 172.21.2.102:0/2930795604 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0f540acfa0 0x7f0f541a3440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:38.624 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:38.622 7f0f5af97700 1 --2- 172.21.2.102:0/2930795604 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0f540b5b80 0x7f0f541a2e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:38.624 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:38.622 7f0f5af97700 1 --2- 172.21.2.102:0/2930795604 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0f54195650 0x7f0f540b5560 unknown :-1 s=CLOSED pgs=2301 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:38.624 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:38.622 7f0f5af97700 1 -- 172.21.2.102:0/2930795604 >> 172.21.2.102:0/2930795604 conn(0x7f0f540b3040 msgr2=0x7f0f540c3250 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:38.624 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:38.622 7f0f5af97700 1 -- 172.21.2.102:0/2930795604 shutdown_connections 2021-02-10T11:32:38.624 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:38.622 7f0f5af97700 1 -- 172.21.2.102:0/2930795604 wait complete. 2021-02-10T11:32:38.632 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:32:39.059 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:39.041 7fa24ebb1700 1 Processor -- start 2021-02-10T11:32:39.059 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:39.041 7fa24ebb1700 1 -- start start 2021-02-10T11:32:39.059 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:39.042 7fa24ebb1700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fa2480d1870 0x7fa2481b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:39.059 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:39.042 7fa24ebb1700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fa2481b0a10 0x7fa2480cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:39.060 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:39.042 7fa24ebb1700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa2480d1c70 0x7fa2480cf950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:39.060 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:39.042 7fa24ebb1700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2480e5ed0 con 0x7fa2481b0a10 2021-02-10T11:32:39.060 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:39.042 7fa24ebb1700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2480f0b10 con 0x7fa2480d1870 2021-02-10T11:32:39.060 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:39.042 7fa24ebb1700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2480cff70 con 0x7fa2480d1c70 2021-02-10T11:32:39.060 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:39.042 7fa247fff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fa2481b0a10 0x7fa2480cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:39.060 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:39.042 7fa247fff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fa2481b0a10 0x7fa2480cf330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:34652/0 (socket says 172.21.2.102:34652) 2021-02-10T11:32:39.060 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:39.042 7fa247fff700 1 -- 172.21.2.102:0/2516748214 learned_addr learned my addr 172.21.2.102:0/2516748214 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:32:39.060 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:39.042 7fa24c9cd700 1 --2- 172.21.2.102:0/2516748214 >> v2:172.21.2.113:3300/0 conn(0x7fa2480d1870 0x7fa2481b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:39.060 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:39.042 7fa24d1ce700 1 --2- 172.21.2.102:0/2516748214 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa2480d1c70 0x7fa2480cf950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:39.061 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:39.042 7fa247fff700 1 -- 172.21.2.102:0/2516748214 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa2480d1c70 msgr2=0x7fa2480cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:32:39.061 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:39.042 7fa247fff700 1 --2- 172.21.2.102:0/2516748214 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa2480d1c70 0x7fa2480cf950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:39.061 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:39.042 7fa247fff700 1 -- 172.21.2.102:0/2516748214 >> v2:172.21.2.113:3300/0 conn(0x7fa2480d1870 msgr2=0x7fa2481b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:32:39.061 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:39.042 7fa247fff700 1 --2- 172.21.2.102:0/2516748214 >> v2:172.21.2.113:3300/0 conn(0x7fa2480d1870 0x7fa2481b03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:39.061 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:39.042 7fa247fff700 1 -- 172.21.2.102:0/2516748214 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2480d0150 con 0x7fa2481b0a10 2021-02-10T11:32:39.061 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:39.042 7fa24c9cd700 1 --2- 172.21.2.102:0/2516748214 >> v2:172.21.2.113:3300/0 conn(0x7fa2480d1870 0x7fa2481b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:32:39.061 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:39.042 7fa247fff700 1 --2- 172.21.2.102:0/2516748214 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa2481b0a10 0x7fa2480cf330 secure :-1 s=READY pgs=2302 cs=0 l=1 rev1=1 rx=0x7fa238006700 tx=0x7fa238009b80).ready entity=mon.0 client_cookie=a2056a759b0d99d9 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:39.061 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:39.043 7fa2477fe700 1 -- 172.21.2.102:0/2516748214 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa23800c0c0 con 0x7fa2481b0a10 2021-02-10T11:32:39.061 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:39.043 7fa2477fe700 1 -- 172.21.2.102:0/2516748214 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa23800c400 con 0x7fa2481b0a10 2021-02-10T11:32:39.061 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:39.043 7fa2477fe700 1 -- 172.21.2.102:0/2516748214 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa23800c860 con 0x7fa2481b0a10 2021-02-10T11:32:39.061 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:39.043 7fa24ebb1700 1 -- 172.21.2.102:0/2516748214 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa2481b0a10 msgr2=0x7fa2480cf330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:39.061 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:39.043 7fa24ebb1700 1 --2- 172.21.2.102:0/2516748214 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa2481b0a10 0x7fa2480cf330 secure :-1 s=READY pgs=2302 cs=0 l=1 rev1=1 rx=0x7fa238006700 tx=0x7fa238009b80).stop 2021-02-10T11:32:39.062 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:39.043 7fa24ebb1700 1 -- 172.21.2.102:0/2516748214 shutdown_connections 2021-02-10T11:32:39.062 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:39.043 7fa24ebb1700 1 --2- 172.21.2.102:0/2516748214 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa2480d1c70 0x7fa2480cf950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:39.062 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:39.043 7fa24ebb1700 1 --2- 172.21.2.102:0/2516748214 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa2481b0a10 0x7fa2480cf330 unknown :-1 s=CLOSED pgs=2302 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:39.062 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:39.043 7fa24ebb1700 1 --2- 172.21.2.102:0/2516748214 >> v2:172.21.2.113:3300/0 conn(0x7fa2480d1870 0x7fa2481b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:39.062 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:39.043 7fa24ebb1700 1 -- 172.21.2.102:0/2516748214 >> 172.21.2.102:0/2516748214 conn(0x7fa2480ce450 msgr2=0x7fa2480d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:39.062 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:39.043 7fa24ebb1700 1 -- 172.21.2.102:0/2516748214 shutdown_connections 2021-02-10T11:32:39.062 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:39.043 7fa24ebb1700 1 -- 172.21.2.102:0/2516748214 wait complete. 2021-02-10T11:32:39.062 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:39.058 7fa24ebb1700 1 Processor -- start 2021-02-10T11:32:39.062 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:39.058 7fa24ebb1700 1 -- start start 2021-02-10T11:32:39.064 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:39.058 7fa24ebb1700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa24818f360 0x7fa2480a9940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:39.064 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:39.058 7fa24ebb1700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa2480a9f60 0x7fa2480ecb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:39.064 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:39.058 7fa24ebb1700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa2480ed190 0x7fa2480ed590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:39.064 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:39.058 7fa24ebb1700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa24818f760 con 0x7fa24818f360 2021-02-10T11:32:39.064 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:39.058 7fa24ebb1700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa24818f9e0 con 0x7fa2480a9f60 2021-02-10T11:32:39.064 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:39.058 7fa24ebb1700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa24808d230 con 0x7fa2480ed190 2021-02-10T11:32:39.064 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:39.059 7fa247fff700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa2480a9f60 0x7fa2480ecb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:39.064 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:39.059 7fa24c9cd700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa24818f360 0x7fa2480a9940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:39.065 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:39.059 7fa247fff700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa2480a9f60 0x7fa2480ecb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.113:3300/0 says I am v2:172.21.2.102:38086/0 (socket says 172.21.2.102:38086) 2021-02-10T11:32:39.065 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:39.059 7fa24c9cd700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa24818f360 0x7fa2480a9940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:34660/0 (socket says 172.21.2.102:34660) 2021-02-10T11:32:39.065 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:39.059 7fa247fff700 1 -- 172.21.2.102:0/3842201727 learned_addr learned my addr 172.21.2.102:0/3842201727 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:32:39.065 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:39.059 7fa24d1ce700 1 --2- 172.21.2.102:0/3842201727 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa2480ed190 0x7fa2480ed590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:39.065 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:39.059 7fa24c9cd700 1 -- 172.21.2.102:0/3842201727 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa2480ed190 msgr2=0x7fa2480ed590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:39.065 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:39.059 7fa24c9cd700 1 --2- 172.21.2.102:0/3842201727 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa2480ed190 0x7fa2480ed590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:39.065 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:39.059 7fa24c9cd700 1 -- 172.21.2.102:0/3842201727 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa2480a9f60 msgr2=0x7fa2480ecb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:39.065 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:39.059 7fa24c9cd700 1 --2- 172.21.2.102:0/3842201727 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa2480a9f60 0x7fa2480ecb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:39.065 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:39.059 7fa24c9cd700 1 -- 172.21.2.102:0/3842201727 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa24808d450 con 0x7fa24818f360 2021-02-10T11:32:39.065 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:39.059 7fa24c9cd700 1 --2- 172.21.2.102:0/3842201727 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa24818f360 0x7fa2480a9940 secure :-1 s=READY pgs=2303 cs=0 l=1 rev1=1 rx=0x7fa230009540 tx=0x7fa23000b8e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:39.065 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:39.059 7fa2457fa700 1 -- 172.21.2.102:0/3842201727 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa23000bcd0 con 0x7fa24818f360 2021-02-10T11:32:39.066 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:39.059 7fa2457fa700 1 -- 172.21.2.102:0/3842201727 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2300063e0 con 0x7fa24818f360 2021-02-10T11:32:39.066 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:39.059 7fa24ebb1700 1 -- 172.21.2.102:0/3842201727 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa24808d450 con 0x7fa24818f360 2021-02-10T11:32:39.066 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:39.059 7fa24ebb1700 1 -- 172.21.2.102:0/3842201727 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa24808d450 con 0x7fa24818f360 2021-02-10T11:32:39.066 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:39.059 7fa2457fa700 1 -- 172.21.2.102:0/3842201727 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa230002220 con 0x7fa24818f360 2021-02-10T11:32:39.066 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:39.060 7fa2457fa700 1 -- 172.21.2.102:0/3842201727 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa230009fe0 con 0x7fa24818f360 2021-02-10T11:32:39.066 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:39.060 7fa2457fa700 1 --2- 172.21.2.102:0/3842201727 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa22c018730 0x7fa22c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:39.066 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:39.060 7fa2457fa700 1 -- 172.21.2.102:0/3842201727 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fa2300326e0 con 0x7fa24818f360 2021-02-10T11:32:39.066 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:39.060 7fa247fff700 1 --2- 172.21.2.102:0/3842201727 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa22c018730 0x7fa22c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:39.066 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:39.060 7fa247fff700 1 --2- 172.21.2.102:0/3842201727 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa22c018730 0x7fa22c01abc0 crc :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:39.093 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:39.091 7fa24ebb1700 1 -- 172.21.2.102:0/3842201727 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2480ebc60 con 0x7fa24818f360 2021-02-10T11:32:39.095 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:39.094 7fa2457fa700 1 -- 172.21.2.102:0/3842201727 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fa230000c90 con 0x7fa24818f360 2021-02-10T11:32:39.220 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:39.219 7fa24ebb1700 1 -- 172.21.2.102:0/3842201727 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa2480ebc60 con 0x7fa24818f360 2021-02-10T11:32:39.221 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:39.219 7fa2457fa700 1 -- 172.21.2.102:0/3842201727 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v120) v1 ==== 74+0+14007 (secure 0 0 0) 0x7fa230009fe0 con 0x7fa24818f360 2021-02-10T11:32:39.221 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:32:39.221 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":120,"fsid":"68db30ea-dbce-4aef-a6df-f01c6fea0513","created":"2021-02-10 11:03:06.331557","modified":"2021-02-10 11:32:39.019652","last_up_change":"2021-02-10 11:03:32.690562","last_in_change":"2021-02-10 11:03:32.690562","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":25,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:03:37.322519","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":24,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:32:35.986688","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"119","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":25,"pool_name":"cephfs_data","create_time":"2021-02-10 11:32:37.011649","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"120","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"be691b4d-a039-408a-af80-9d53f02b6f1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6818","nonce":36088},{"type":"v1","addr":"172.21.2.102:6820","nonce":36088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6822","nonce":36088},{"type":"v1","addr":"172.21.2.102:6824","nonce":36088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6830","nonce":36088},{"type":"v1","addr":"172.21.2.102:6832","nonce":36088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6826","nonce":36088},{"type":"v1","addr":"172.21.2.102:6828","nonce":36088}]},"public_addr":"172.21.2.102:6820/36088","cluster_addr":"172.21.2.102:6824/36088","heartbeat_back_addr":"172.21.2.102:6832/36088","heartbeat_front_addr":"172.21.2.102:6828/36088","state":["exists","up"]},{"osd":1,"uuid":"64e3ccc8-9c69-4d7c-865e-af8e83822635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6819","nonce":36087},{"type":"v1","addr":"172.21.2.102:6821","nonce":36087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6823","nonce":36087},{"type":"v1","addr":"172.21.2.102:6825","nonce":36087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6831","nonce":36087},{"type":"v1","addr":"172.21.2.102:6833","nonce":36087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6827","nonce":36087},{"type":"v1","addr":"172.21.2.102:6829","nonce":36087}]},"public_addr":"172.21.2.102:6821/36087","cluster_addr":"172.21.2.102:6825/36087","heartbeat_back_addr":"172.21.2.102:6833/36087","heartbeat_front_addr":"172.21.2.102:6829/36087","state":["exists","up"]},{"osd":2,"uuid":"153700af-747f-481f-bdf4-d6072d3a842f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6810","nonce":36089},{"type":"v1","addr":"172.21.2.102:6811","nonce":36089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6812","nonce":36089},{"type":"v1","addr":"172.21.2.102:6813","nonce":36089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6816","nonce":36089},{"type":"v1","addr":"172.21.2.102:6817","nonce":36089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6814","nonce":36089},{"type":"v1","addr":"172.21.2.102:6815","nonce":36089}]},"public_addr":"172.21.2.102:6811/36089","cluster_addr":"172.21.2.102:6813/36089","heartbeat_back_addr":"172.21.2.102:6817/36089","heartbeat_front_addr":"172.21.2.102:6815/36089","state":["exists","up"]},{"osd":3,"uuid":"cbc678f4-83a9-45e6-8830-2f7c232b1ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6802","nonce":36090},{"type":"v1","addr":"172.21.2.102:6803","nonce":36090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6804","nonce":36090},{"type":"v1","addr":"172.21.2.102:6805","nonce":36090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6808","nonce":36090},{"type":"v1","addr":"172.21.2.102:6809","nonce":36090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6806","nonce":36090},{"type":"v1","addr":"172.21.2.102:6807","nonce":36090}]},"public_addr":"172.21.2.102:6803/36090","cluster_addr":"172.21.2.102:6805/36090","heartbeat_back_addr":"172.21.2.102:6809/36090","heartbeat_front_addr":"172.21.2.102:6807/36090","state":["exists","up"]},{"osd":4,"uuid":"c01e4008-b258-4f10-afd2-9e7f37214351","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6808","nonce":36368},{"type":"v1","addr":"172.21.2.113:6811","nonce":36368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6814","nonce":36368},{"type":"v1","addr":"172.21.2.113:6818","nonce":36368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6826","nonce":36368},{"type":"v1","addr":"172.21.2.113:6828","nonce":36368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6820","nonce":36368},{"type":"v1","addr":"172.21.2.113:6824","nonce":36368}]},"public_addr":"172.21.2.113:6811/36368","cluster_addr":"172.21.2.113:6818/36368","heartbeat_back_addr":"172.21.2.113:6828/36368","heartbeat_front_addr":"172.21.2.113:6824/36368","state":["exists","up"]},{"osd":5,"uuid":"551f6007-daf8-4a98-8ee0-c2b16bcccb5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6802","nonce":36366},{"type":"v1","addr":"172.21.2.113:6805","nonce":36366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6807","nonce":36366},{"type":"v1","addr":"172.21.2.113:6810","nonce":36366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6819","nonce":36366},{"type":"v1","addr":"172.21.2.113:6822","nonce":36366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6812","nonce":36366},{"type":"v1","addr":"172.21.2.113:6815","nonce":36366}]},"public_addr":"172.21.2.113:6805/36366","cluster_addr":"172.21.2.113:6810/36366","heartbeat_back_addr":"172.21.2.113:6822/36366","heartbeat_front_addr":"172.21.2.113:6815/36366","state":["exists","up"]},{"osd":6,"uuid":"bf67c040-0e50-48be-b102-c3943bc9ebef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6800","nonce":36367},{"type":"v1","addr":"172.21.2.113:6801","nonce":36367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6803","nonce":36367},{"type":"v1","addr":"172.21.2.113:6804","nonce":36367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6813","nonce":36367},{"type":"v1","addr":"172.21.2.113:6816","nonce":36367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6806","nonce":36367},{"type":"v1","addr":"172.21.2.113:6809","nonce":36367}]},"public_addr":"172.21.2.113:6801/36367","cluster_addr":"172.21.2.113:6804/36367","heartbeat_back_addr":"172.21.2.113:6816/36367","heartbeat_front_addr":"172.21.2.113:6809/36367","state":["exists","up"]},{"osd":7,"uuid":"f5461f80-018d-428f-be89-2eb2bcbe7bb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6817","nonce":36365},{"type":"v1","addr":"172.21.2.113:6821","nonce":36365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6823","nonce":36365},{"type":"v1","addr":"172.21.2.113:6825","nonce":36365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6830","nonce":36365},{"type":"v1","addr":"172.21.2.113:6831","nonce":36365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6827","nonce":36365},{"type":"v1","addr":"172.21.2.113:6829","nonce":36365}]},"public_addr":"172.21.2.113:6821/36365","cluster_addr":"172.21.2.113:6825/36365","heartbeat_back_addr":"172.21.2.113:6831/36365","heartbeat_front_addr":"172.21.2.113:6829/36365","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:32:39.223 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:39.221 7fa24ebb1700 1 -- 172.21.2.102:0/3842201727 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa22c018730 msgr2=0x7fa22c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:39.223 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:39.221 7fa24ebb1700 1 --2- 172.21.2.102:0/3842201727 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa22c018730 0x7fa22c01abc0 crc :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:39.223 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:39.221 7fa24ebb1700 1 -- 172.21.2.102:0/3842201727 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa24818f360 msgr2=0x7fa2480a9940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:39.223 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:39.221 7fa24ebb1700 1 --2- 172.21.2.102:0/3842201727 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa24818f360 0x7fa2480a9940 secure :-1 s=READY pgs=2303 cs=0 l=1 rev1=1 rx=0x7fa230009540 tx=0x7fa23000b8e0).stop 2021-02-10T11:32:39.223 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:39.221 7fa24ebb1700 1 -- 172.21.2.102:0/3842201727 shutdown_connections 2021-02-10T11:32:39.223 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:39.221 7fa24ebb1700 1 --2- 172.21.2.102:0/3842201727 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa2480ed190 0x7fa2480ed590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:39.223 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:39.221 7fa24ebb1700 1 --2- 172.21.2.102:0/3842201727 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa22c018730 0x7fa22c01abc0 unknown :-1 s=CLOSED pgs=1014 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:39.223 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:39.221 7fa24ebb1700 1 --2- 172.21.2.102:0/3842201727 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa2480a9f60 0x7fa2480ecb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:39.223 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:39.221 7fa24ebb1700 1 --2- 172.21.2.102:0/3842201727 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa24818f360 0x7fa2480a9940 unknown :-1 s=CLOSED pgs=2303 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:39.224 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:39.221 7fa24ebb1700 1 -- 172.21.2.102:0/3842201727 >> 172.21.2.102:0/3842201727 conn(0x7fa24809eba0 msgr2=0x7fa2480ebe90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:39.224 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:39.221 7fa24ebb1700 1 -- 172.21.2.102:0/3842201727 shutdown_connections 2021-02-10T11:32:39.224 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:39.221 7fa24ebb1700 1 -- 172.21.2.102:0/3842201727 wait complete. 2021-02-10T11:32:39.232 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2021-02-10T11:32:39.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:39.646 7f22d9325700 1 Processor -- start 2021-02-10T11:32:39.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:39.646 7f22d9325700 1 -- start start 2021-02-10T11:32:39.666 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:39.646 7f22d9325700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f22d40d1c70 0x7f22d40cef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:39.666 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:39.646 7f22d9325700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f22d40cf580 0x7f22d40cf980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:39.666 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:39.646 7f22d9325700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22d40fe1c0 con 0x7f22d40d1870 2021-02-10T11:32:39.666 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:39.646 7f22d9325700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22d41b0cb0 con 0x7f22d40d1c70 2021-02-10T11:32:39.666 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:39.646 7f22d9325700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22d4109a60 con 0x7f22d40cf580 2021-02-10T11:32:39.666 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:39.647 7f22d307d700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f22d40d1870 0x7f22d41b04c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:55120/0 (socket says 172.21.2.102:55120) 2021-02-10T11:32:39.666 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:39.647 7f22d307d700 1 -- 172.21.2.102:0/1860637691 learned_addr learned my addr 172.21.2.102:0/1860637691 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:32:39.666 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:39.647 7f22d207b700 1 -- 172.21.2.102:0/1860637691 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2288190228 0 0) 0x7f22bc000ce0 con 0x7f22d40d1870 2021-02-10T11:32:39.666 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:39.647 7f22d207b700 1 -- 172.21.2.102:0/1860637691 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22c0002fe0 con 0x7f22d40d1870 2021-02-10T11:32:39.666 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:39.647 7f22d287c700 1 --2- 172.21.2.102:0/1860637691 >> v2:172.21.2.113:3300/0 conn(0x7f22d40d1c70 0x7f22d40cef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:39.666 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:39.647 7f22d387e700 1 --2- 172.21.2.102:0/1860637691 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f22d40cf580 0x7f22d40cf980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:39.666 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:39.647 7f22d207b700 1 -- 172.21.2.102:0/1860637691 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2126519620 0 0) 0x7f22bc0010e0 con 0x7f22d40d1870 2021-02-10T11:32:39.666 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:39.648 7f22d207b700 1 -- 172.21.2.102:0/1860637691 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f22d40cf580 msgr2=0x7f22d40cf980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:32:39.667 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:39.648 7f22d207b700 1 --2- 172.21.2.102:0/1860637691 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f22d40cf580 0x7f22d40cf980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:39.667 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:39.648 7f22d207b700 1 -- 172.21.2.102:0/1860637691 >> v2:172.21.2.113:3300/0 conn(0x7f22d40d1c70 msgr2=0x7f22d40cef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:32:39.667 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:39.648 7f22d207b700 1 --2- 172.21.2.102:0/1860637691 >> v2:172.21.2.113:3300/0 conn(0x7f22d40d1c70 0x7f22d40cef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:39.667 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:39.648 7f22d207b700 1 -- 172.21.2.102:0/1860637691 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22d4088070 con 0x7f22d40d1870 2021-02-10T11:32:39.667 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:39.648 7f22d207b700 1 -- 172.21.2.102:0/1860637691 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7f22bc001660 con 0x7f22d40d1870 2021-02-10T11:32:39.667 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:39.648 7f22d207b700 1 -- 172.21.2.102:0/1860637691 >> v1:172.21.2.102:6789/0 conn(0x7f22d40d1870 legacy=0x7f22d41b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:39.667 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:39.648 7f22d207b700 1 --2- 172.21.2.102:0/1860637691 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f22c0003ae0 0x7f22c0005f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:39.667 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:39.648 7f22d207b700 1 --2- 172.21.2.102:0/1860637691 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f22c004a840 0x7f22c004cd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:39.667 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:39.648 7f22d207b700 1 --2- 172.21.2.102:0/1860637691 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f22c004d5b0 0x7f22c004fad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:39.667 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:39.648 7f22d207b700 1 -- 172.21.2.102:0/1860637691 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22c0050380 con 0x7f22c004d5b0 2021-02-10T11:32:39.667 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:39.648 7f22d207b700 1 -- 172.21.2.102:0/1860637691 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22c00505a0 con 0x7f22c0003ae0 2021-02-10T11:32:39.667 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:39.648 7f22d207b700 1 -- 172.21.2.102:0/1860637691 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22c00507c0 con 0x7f22c004a840 2021-02-10T11:32:39.667 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:39.648 7f22d287c700 1 --2- 172.21.2.102:0/1860637691 >> v2:172.21.2.113:3300/0 conn(0x7f22d40d1c70 0x7f22d40cef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:32:39.668 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:39.648 7f22d307d700 1 --2- 172.21.2.102:0/1860637691 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f22c0003ae0 0x7f22c0005f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:39.668 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:39.648 7f22d387e700 1 --2- 172.21.2.102:0/1860637691 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f22c004d5b0 0x7f22c004fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:39.668 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:39.648 7f22d287c700 1 --2- 172.21.2.102:0/1860637691 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f22c004a840 0x7f22c004cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:39.668 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:39.648 7f22d387e700 1 -- 172.21.2.102:0/1860637691 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f22c004a840 msgr2=0x7f22c004cd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:32:39.668 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:39.648 7f22d387e700 1 --2- 172.21.2.102:0/1860637691 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f22c004a840 0x7f22c004cd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:39.668 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:39.648 7f22d387e700 1 -- 172.21.2.102:0/1860637691 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f22c0003ae0 msgr2=0x7f22c0005f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:32:39.668 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:39.648 7f22d387e700 1 --2- 172.21.2.102:0/1860637691 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f22c0003ae0 0x7f22c0005f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:39.668 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:39.648 7f22d387e700 1 -- 172.21.2.102:0/1860637691 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22c0050a60 con 0x7f22c004d5b0 2021-02-10T11:32:39.668 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:39.648 7f22d387e700 1 --2- 172.21.2.102:0/1860637691 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f22c004d5b0 0x7f22c004fad0 secure :-1 s=READY pgs=2305 cs=0 l=1 rev1=1 rx=0x7f22cc009a40 tx=0x7f22cc009de0).ready entity=mon.0 client_cookie=9789ecb061ff2381 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:39.668 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:39.649 7f22d207b700 1 -- 172.21.2.102:0/1860637691 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f22cc006a90 con 0x7f22c004d5b0 2021-02-10T11:32:39.668 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:39.649 7f22d207b700 1 -- 172.21.2.102:0/1860637691 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22cc003030 con 0x7f22c004d5b0 2021-02-10T11:32:39.668 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:39.649 7f22d207b700 1 -- 172.21.2.102:0/1860637691 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f22cc00b790 con 0x7f22c004d5b0 2021-02-10T11:32:39.668 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:39.649 7f22d9325700 1 -- 172.21.2.102:0/1860637691 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f22c004d5b0 msgr2=0x7f22c004fad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:39.668 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:39.649 7f22d9325700 1 --2- 172.21.2.102:0/1860637691 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f22c004d5b0 0x7f22c004fad0 secure :-1 s=READY pgs=2305 cs=0 l=1 rev1=1 rx=0x7f22cc009a40 tx=0x7f22cc009de0).stop 2021-02-10T11:32:39.669 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:39.649 7f22d387e700 1 -- 172.21.2.102:0/1860637691 reap_dead start 2021-02-10T11:32:39.669 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:39.649 7f22d9325700 1 -- 172.21.2.102:0/1860637691 shutdown_connections 2021-02-10T11:32:39.669 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:39.649 7f22d9325700 1 -- 172.21.2.102:0/1860637691 >> 172.21.2.102:0/1860637691 conn(0x7f22d40ce420 msgr2=0x7f22d40d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:39.669 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:39.649 7f22d9325700 1 -- 172.21.2.102:0/1860637691 shutdown_connections 2021-02-10T11:32:39.669 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:39.649 7f22d9325700 1 -- 172.21.2.102:0/1860637691 wait complete. 2021-02-10T11:32:39.669 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:39.664 7f22d9325700 1 Processor -- start 2021-02-10T11:32:39.669 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:39.664 7f22d9325700 1 -- start start 2021-02-10T11:32:39.671 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:39.664 7f22d9325700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f22d409bcc0 0x7f22d409c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:39.671 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:39.664 7f22d9325700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f22d40a0820 0x7f22d40a0c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:39.671 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:39.664 7f22d9325700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f22d40a1240 0x7f22d40858d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:39.671 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:39.664 7f22d9325700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22d40a66b0 con 0x7f22d40a1240 2021-02-10T11:32:39.671 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:39.664 7f22d9325700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22d41a49b0 con 0x7f22d409bcc0 2021-02-10T11:32:39.671 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:39.664 7f22d9325700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22d41a4b90 con 0x7f22d40a0820 2021-02-10T11:32:39.671 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:39.665 7f22d387e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f22d40a1240 0x7f22d40858d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:39.671 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:39.665 7f22d387e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f22d40a1240 0x7f22d40858d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:34674/0 (socket says 172.21.2.102:34674) 2021-02-10T11:32:39.671 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:39.665 7f22d387e700 1 -- 172.21.2.102:0/3189157171 learned_addr learned my addr 172.21.2.102:0/3189157171 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:32:39.671 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:39.665 7f22d307d700 1 --2- 172.21.2.102:0/3189157171 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f22d409bcc0 0x7f22d409c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:39.671 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:39.665 7f22d287c700 1 --2- 172.21.2.102:0/3189157171 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f22d40a0820 0x7f22d40a0c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:39.672 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:39.665 7f22d287c700 1 -- 172.21.2.102:0/3189157171 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f22d409bcc0 msgr2=0x7f22d409c0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:39.672 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:39.665 7f22d287c700 1 --2- 172.21.2.102:0/3189157171 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f22d409bcc0 0x7f22d409c0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:39.672 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:39.665 7f22d287c700 1 -- 172.21.2.102:0/3189157171 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f22d40a1240 msgr2=0x7f22d40858d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:39.672 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:39.665 7f22d287c700 1 --2- 172.21.2.102:0/3189157171 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f22d40a1240 0x7f22d40858d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:39.672 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:39.665 7f22d287c700 1 -- 172.21.2.102:0/3189157171 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22d40a1640 con 0x7f22d40a0820 2021-02-10T11:32:39.672 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:39.665 7f22d387e700 1 --2- 172.21.2.102:0/3189157171 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f22d40a1240 0x7f22d40858d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:32:39.672 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:39.665 7f22d287c700 1 --2- 172.21.2.102:0/3189157171 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f22d40a0820 0x7f22d40a0c20 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 rx=0x7f22c4006aa0 tx=0x7f22c400b8e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:39.672 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:39.665 7f22cbfff700 1 -- 172.21.2.102:0/3189157171 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f22c400bcd0 con 0x7f22d40a0820 2021-02-10T11:32:39.672 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:39.666 7f22cbfff700 1 -- 172.21.2.102:0/3189157171 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22c4006c20 con 0x7f22d40a0820 2021-02-10T11:32:39.672 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:39.666 7f22cbfff700 1 -- 172.21.2.102:0/3189157171 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f22c400bcd0 con 0x7f22d40a0820 2021-02-10T11:32:39.672 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:39.666 7f22d9325700 1 -- 172.21.2.102:0/3189157171 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22d40a1640 con 0x7f22d40a0820 2021-02-10T11:32:39.672 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:39.666 7f22d9325700 1 -- 172.21.2.102:0/3189157171 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22d40a1640 con 0x7f22d40a0820 2021-02-10T11:32:39.672 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:39.666 7f22cbfff700 1 -- 172.21.2.102:0/3189157171 <== mon.2 v2:172.21.2.113:3301/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f22c4006030 con 0x7f22d40a0820 2021-02-10T11:32:39.672 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:39.666 7f22cbfff700 1 --2- 172.21.2.102:0/3189157171 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f22b8018730 0x7f22b801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:39.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:39.666 7f22cbfff700 1 -- 172.21.2.102:0/3189157171 <== mon.2 v2:172.21.2.113:3301/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f22c4032590 con 0x7f22d40a0820 2021-02-10T11:32:39.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:39.666 7f22d307d700 1 --2- 172.21.2.102:0/3189157171 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f22b8018730 0x7f22b801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:39.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:39.667 7f22d307d700 1 --2- 172.21.2.102:0/3189157171 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f22b8018730 0x7f22b801ac10 crc :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:39.698 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:39.696 7f22d9325700 1 -- 172.21.2.102:0/3189157171 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22d40fe1c0 con 0x7f22d40a0820 2021-02-10T11:32:39.702 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:39.700 7f22cbfff700 1 -- 172.21.2.102:0/3189157171 <== mon.2 v2:172.21.2.113:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f22c400bcd0 con 0x7f22d40a0820 2021-02-10T11:32:39.842 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:39.840 7f22d9325700 1 -- 172.21.2.102:0/3189157171 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_command({"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"} v 0) v1 -- 0x7f22d40fe1c0 con 0x7f22d40a0820 2021-02-10T11:32:40.030 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:40.028 7f22cbfff700 1 -- 172.21.2.102:0/3189157171 <== mon.2 v2:172.21.2.113:3301/0 7 ==== mon_command_ack([{"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"}]=0 v99) v1 ==== 118+0+0 (secure 0 0 0) 0x7f22c4006030 con 0x7f22d40a0820 2021-02-10T11:32:40.060 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:40.058 7f22d9325700 1 -- 172.21.2.102:0/3189157171 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f22b8018730 msgr2=0x7f22b801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:40.060 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:40.058 7f22d9325700 1 --2- 172.21.2.102:0/3189157171 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f22b8018730 0x7f22b801ac10 crc :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:40.060 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:40.058 7f22d9325700 1 -- 172.21.2.102:0/3189157171 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f22d40a0820 msgr2=0x7f22d40a0c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:40.060 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:40.058 7f22d9325700 1 --2- 172.21.2.102:0/3189157171 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f22d40a0820 0x7f22d40a0c20 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 rx=0x7f22c4006aa0 tx=0x7f22c400b8e0).stop 2021-02-10T11:32:40.060 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:40.059 7f22d9325700 1 -- 172.21.2.102:0/3189157171 shutdown_connections 2021-02-10T11:32:40.061 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:40.059 7f22d9325700 1 --2- 172.21.2.102:0/3189157171 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f22d40a1240 0x7f22d40858d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:40.061 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:40.059 7f22d9325700 1 --2- 172.21.2.102:0/3189157171 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f22d40a0820 0x7f22d40a0c20 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:40.061 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:40.059 7f22d9325700 1 --2- 172.21.2.102:0/3189157171 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f22b8018730 0x7f22b801ac10 unknown :-1 s=CLOSED pgs=1015 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:40.061 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:40.059 7f22d9325700 1 --2- 172.21.2.102:0/3189157171 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f22d409bcc0 0x7f22d409c0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:40.061 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:40.059 7f22d9325700 1 -- 172.21.2.102:0/3189157171 >> 172.21.2.102:0/3189157171 conn(0x7f22d40bef00 msgr2=0x7f22d409d1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:40.061 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:40.059 7f22d9325700 1 -- 172.21.2.102:0/3189157171 shutdown_connections 2021-02-10T11:32:40.061 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:40.059 7f22d9325700 1 -- 172.21.2.102:0/3189157171 wait complete. 2021-02-10T11:32:40.069 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:32:40.177 DEBUG:teuthology.orchestra.run.gibba002:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:32:40.183 DEBUG:teuthology.orchestra.run.gibba013:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:32:40.496 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:40.478 7f021f286700 1 Processor -- start 2021-02-10T11:32:40.496 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:40.479 7f021f286700 1 -- start start 2021-02-10T11:32:40.497 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:40.479 7f021f286700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f02180d1870 0x7f02181b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:40.497 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:40.479 7f021f286700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f02181b0a10 0x7f02180cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:40.497 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:40.479 7f021f286700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f02180d1c70 0x7f02180cf950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:40.497 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:40.479 7f021f286700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02180fe200 con 0x7f02180d1870 2021-02-10T11:32:40.497 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:40.479 7f021f286700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02180e5f10 con 0x7f02180d1c70 2021-02-10T11:32:40.497 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:40.479 7f021f286700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02180f0b50 con 0x7f02181b0a10 2021-02-10T11:32:40.497 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:40.479 7f021d0a2700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f02180d1870 0x7f02181b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:40.497 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:40.479 7f021d0a2700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f02180d1870 0x7f02181b03f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:34680/0 (socket says 172.21.2.102:34680) 2021-02-10T11:32:40.497 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:40.479 7f021d0a2700 1 -- 172.21.2.102:0/3735819108 learned_addr learned my addr 172.21.2.102:0/3735819108 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:32:40.497 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:40.480 7f021d0a2700 1 -- 172.21.2.102:0/3735819108 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f02181b0a10 msgr2=0x7f02180cf330 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:32:40.497 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:40.480 7f021d0a2700 1 --2- 172.21.2.102:0/3735819108 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f02181b0a10 0x7f02180cf330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:40.497 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:40.480 7f021d0a2700 1 -- 172.21.2.102:0/3735819108 >> v2:172.21.2.113:3300/0 conn(0x7f02180d1c70 msgr2=0x7f02180cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:32:40.498 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:40.480 7f021d0a2700 1 --2- 172.21.2.102:0/3735819108 >> v2:172.21.2.113:3300/0 conn(0x7f02180d1c70 0x7f02180cf950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:40.498 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:40.480 7f021d0a2700 1 -- 172.21.2.102:0/3735819108 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02180cff70 con 0x7f02180d1870 2021-02-10T11:32:40.498 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:40.480 7f021d0a2700 1 --2- 172.21.2.102:0/3735819108 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f02180d1870 0x7f02181b03f0 secure :-1 s=READY pgs=2306 cs=0 l=1 rev1=1 rx=0x7f020c006700 tx=0x7f020c009b80).ready entity=mon.0 client_cookie=f47f2424bc432ea0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:40.498 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:40.480 7f020bfff700 1 -- 172.21.2.102:0/3735819108 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f020c00c0c0 con 0x7f02180d1870 2021-02-10T11:32:40.498 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:40.480 7f020bfff700 1 -- 172.21.2.102:0/3735819108 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f020c00c3c0 con 0x7f02180d1870 2021-02-10T11:32:40.498 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:40.480 7f020bfff700 1 -- 172.21.2.102:0/3735819108 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f020c00c820 con 0x7f02180d1870 2021-02-10T11:32:40.498 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:40.480 7f021f286700 1 -- 172.21.2.102:0/3735819108 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f02180d1870 msgr2=0x7f02181b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:40.498 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:40.480 7f021f286700 1 --2- 172.21.2.102:0/3735819108 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f02180d1870 0x7f02181b03f0 secure :-1 s=READY pgs=2306 cs=0 l=1 rev1=1 rx=0x7f020c006700 tx=0x7f020c009b80).stop 2021-02-10T11:32:40.498 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:40.481 7f021f286700 1 -- 172.21.2.102:0/3735819108 shutdown_connections 2021-02-10T11:32:40.498 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:40.481 7f021f286700 1 --2- 172.21.2.102:0/3735819108 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f02181b0a10 0x7f02180cf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:40.498 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:40.481 7f021f286700 1 --2- 172.21.2.102:0/3735819108 >> v2:172.21.2.113:3300/0 conn(0x7f02180d1c70 0x7f02180cf950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:40.498 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:40.481 7f021f286700 1 --2- 172.21.2.102:0/3735819108 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f02180d1870 0x7f02181b03f0 unknown :-1 s=CLOSED pgs=2306 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:40.498 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:40.481 7f021f286700 1 -- 172.21.2.102:0/3735819108 >> 172.21.2.102:0/3735819108 conn(0x7f02180ce450 msgr2=0x7f02180d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:40.499 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:40.481 7f021f286700 1 -- 172.21.2.102:0/3735819108 shutdown_connections 2021-02-10T11:32:40.499 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:40.481 7f021f286700 1 -- 172.21.2.102:0/3735819108 wait complete. 2021-02-10T11:32:40.500 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:40.495 7f021f286700 1 Processor -- start 2021-02-10T11:32:40.500 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:40.495 7f021f286700 1 -- start start 2021-02-10T11:32:40.500 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:40.496 7f021f286700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f02181971e0 0x7f021818eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:40.500 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:40.496 7f021f286700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f02181b8180 0x7f02181b8580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:40.500 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:40.496 7f021f286700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f02181b8ba0 0x7f02180b17b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:40.500 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:40.496 7f021f286700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02181b9750 con 0x7f02181b8ba0 2021-02-10T11:32:40.500 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:40.496 7f021f286700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f021818e930 con 0x7f02181b8180 2021-02-10T11:32:40.500 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:40.496 7f021f286700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02180b1dd0 con 0x7f02181971e0 2021-02-10T11:32:40.501 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:40.496 7f021d8a3700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f02181b8ba0 0x7f02180b17b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:40.501 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:40.496 7f021d8a3700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f02181b8ba0 0x7f02180b17b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:34684/0 (socket says 172.21.2.102:34684) 2021-02-10T11:32:40.501 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:40.496 7f021d8a3700 1 -- 172.21.2.102:0/2591257740 learned_addr learned my addr 172.21.2.102:0/2591257740 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:32:40.501 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:40.496 7f021d8a3700 1 -- 172.21.2.102:0/2591257740 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f02181971e0 msgr2=0x7f021818eda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:40.501 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:40.496 7f021d8a3700 1 --2- 172.21.2.102:0/2591257740 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f02181971e0 0x7f021818eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:40.501 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:40.496 7f021d8a3700 1 -- 172.21.2.102:0/2591257740 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f02181b8180 msgr2=0x7f02181b8580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:40.501 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:40.496 7f021d8a3700 1 --2- 172.21.2.102:0/2591257740 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f02181b8180 0x7f02181b8580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:40.501 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:40.496 7f021d8a3700 1 -- 172.21.2.102:0/2591257740 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02180b1ff0 con 0x7f02181b8ba0 2021-02-10T11:32:40.501 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:40.497 7f021d8a3700 1 --2- 172.21.2.102:0/2591257740 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f02181b8ba0 0x7f02180b17b0 secure :-1 s=READY pgs=2307 cs=0 l=1 rev1=1 rx=0x7f0210009980 tx=0x7f021000bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:40.501 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:40.497 7f0209ffb700 1 -- 172.21.2.102:0/2591257740 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0210005870 con 0x7f02181b8ba0 2021-02-10T11:32:40.501 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:40.497 7f0209ffb700 1 -- 172.21.2.102:0/2591257740 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0210005af0 con 0x7f02181b8ba0 2021-02-10T11:32:40.501 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:40.497 7f0209ffb700 1 -- 172.21.2.102:0/2591257740 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0210006030 con 0x7f02181b8ba0 2021-02-10T11:32:40.501 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:40.497 7f021f286700 1 -- 172.21.2.102:0/2591257740 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02180b1ff0 con 0x7f02181b8ba0 2021-02-10T11:32:40.502 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:40.497 7f021f286700 1 -- 172.21.2.102:0/2591257740 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02180b2580 con 0x7f02181b8ba0 2021-02-10T11:32:40.502 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:40.497 7f0209ffb700 1 -- 172.21.2.102:0/2591257740 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f0210005cd0 con 0x7f02181b8ba0 2021-02-10T11:32:40.502 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:40.498 7f0209ffb700 1 --2- 172.21.2.102:0/2591257740 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f01f80186c0 0x7f01f801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:40.502 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:40.498 7f0209ffb700 1 -- 172.21.2.102:0/2591257740 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f02100328e0 con 0x7f02181b8ba0 2021-02-10T11:32:40.502 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:40.498 7f021d0a2700 1 --2- 172.21.2.102:0/2591257740 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f01f80186c0 0x7f01f801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:40.502 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:40.498 7f021d0a2700 1 --2- 172.21.2.102:0/2591257740 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f01f80186c0 0x7f01f801abc0 crc :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:40.530 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:40.529 7f021f286700 1 -- 172.21.2.102:0/2591257740 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02180ee060 con 0x7f02181b8ba0 2021-02-10T11:32:40.533 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:40.532 7f0209ffb700 1 -- 172.21.2.102:0/2591257740 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f02100062f0 con 0x7f02181b8ba0 2021-02-10T11:32:40.661 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:40.660 7f021f286700 1 -- 172.21.2.102:0/2591257740 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f02180ce450 con 0x7f02181b8ba0 2021-02-10T11:32:40.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:40.660 7f0209ffb700 1 -- 172.21.2.102:0/2591257740 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+2101 (secure 0 0 0) 0x7f0210005cd0 con 0x7f02181b8ba0 2021-02-10T11:32:40.662 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:32:40.662 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":99,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10737,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.102:6835/4014802408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6834","nonce":4014802408},{"type":"v1","addr":"172.21.2.102:6835","nonce":4014802408}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":95}],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:32:38.017410","modified":"2021-02-10 11:32:40.027261","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7693},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7693":{"gid":7693,"name":"b","rank":0,"incarnation":96,"state":"up:active","state_seq":3,"addr":"172.21.2.113:6833/238020319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6832","nonce":238020319},{"type":"v1","addr":"172.21.2.113:6833","nonce":238020319}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[25],"metadata_pool":24,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":12}]} 2021-02-10T11:32:40.662 INFO:teuthology.orchestra.run.gibba002.stderr:dumped fsmap epoch 99 2021-02-10T11:32:40.663 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:40.662 7f021f286700 1 -- 172.21.2.102:0/2591257740 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f01f80186c0 msgr2=0x7f01f801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:40.663 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:40.662 7f021f286700 1 --2- 172.21.2.102:0/2591257740 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f01f80186c0 0x7f01f801abc0 crc :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:40.663 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:40.662 7f021f286700 1 -- 172.21.2.102:0/2591257740 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f02181b8ba0 msgr2=0x7f02180b17b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:40.663 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:40.662 7f021f286700 1 --2- 172.21.2.102:0/2591257740 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f02181b8ba0 0x7f02180b17b0 secure :-1 s=READY pgs=2307 cs=0 l=1 rev1=1 rx=0x7f0210009980 tx=0x7f021000bcd0).stop 2021-02-10T11:32:40.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:40.662 7f021f286700 1 -- 172.21.2.102:0/2591257740 shutdown_connections 2021-02-10T11:32:40.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:40.662 7f021f286700 1 --2- 172.21.2.102:0/2591257740 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f02181b8ba0 0x7f02180b17b0 unknown :-1 s=CLOSED pgs=2307 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:40.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:40.662 7f021f286700 1 --2- 172.21.2.102:0/2591257740 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f01f80186c0 0x7f01f801abc0 unknown :-1 s=CLOSED pgs=1016 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:40.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:40.662 7f021f286700 1 --2- 172.21.2.102:0/2591257740 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f02181b8180 0x7f02181b8580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:40.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:40.662 7f021f286700 1 --2- 172.21.2.102:0/2591257740 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f02181971e0 0x7f021818eda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:40.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:40.662 7f021f286700 1 -- 172.21.2.102:0/2591257740 >> 172.21.2.102:0/2591257740 conn(0x7f021819f8f0 msgr2=0x7f021818c260 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:40.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:40.662 7f021f286700 1 -- 172.21.2.102:0/2591257740 shutdown_connections 2021-02-10T11:32:40.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:40.662 7f021f286700 1 -- 172.21.2.102:0/2591257740 wait complete. 2021-02-10T11:32:40.672 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:32:41.102 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:41.083 7f5b1727b700 1 Processor -- start 2021-02-10T11:32:41.102 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:41.083 7f5b1727b700 1 -- start start 2021-02-10T11:32:41.102 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:41.083 7f5b1727b700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5b100d0340 0x7f5b100ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:41.102 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:41.083 7f5b1727b700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f5b100d1c70 0x7f5b100efc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:41.102 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:41.083 7f5b1727b700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b100f0b10 con 0x7f5b100d1870 2021-02-10T11:32:41.102 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:41.083 7f5b1727b700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b100e5ed0 con 0x7f5b100d1c70 2021-02-10T11:32:41.102 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:41.083 7f5b1727b700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b101b0be0 con 0x7f5b100d0340 2021-02-10T11:32:41.102 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:41.084 7f5b15898700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f5b100d1c70 0x7f5b100efc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:41.102 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:41.084 7f5b15898700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f5b100d1c70 0x7f5b100efc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.113:3300/0 says I am v2:172.21.2.102:38122/0 (socket says 172.21.2.102:38122) 2021-02-10T11:32:41.103 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:41.084 7f5b15898700 1 -- 172.21.2.102:0/3619785362 learned_addr learned my addr 172.21.2.102:0/3619785362 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:32:41.103 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:41.084 7f5b15097700 1 --1- >> v1:172.21.2.113:6789/0 conn(0x7f5b100d1870 0x7f5b101b03f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.113:6789/0 says I am v1:172.21.2.102:34240/0 (socket says 172.21.2.102:34240) 2021-02-10T11:32:41.103 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:41.084 7f5b14896700 1 --2- 172.21.2.102:0/3619785362 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5b100d0340 0x7f5b100ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:41.103 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:41.084 7f5b14896700 1 -- 172.21.2.102:0/3619785362 >> v2:172.21.2.113:3300/0 conn(0x7f5b100d1c70 msgr2=0x7f5b100efc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:32:41.103 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:41.084 7f5b14896700 1 --2- 172.21.2.102:0/3619785362 >> v2:172.21.2.113:3300/0 conn(0x7f5b100d1c70 0x7f5b100efc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:41.103 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:41.084 7f5b14896700 1 -- 172.21.2.102:0/3619785362 >> v1:172.21.2.113:6789/0 conn(0x7f5b100d1870 legacy=0x7f5b101b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:41.103 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:41.084 7f5b03fff700 1 -- 172.21.2.102:0/3619785362 <== mon.1 v1:172.21.2.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1465456505 0 0) 0x7f5b0c0009b0 con 0x7f5b100d1870 2021-02-10T11:32:41.103 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:41.085 7f5b14896700 1 -- 172.21.2.102:0/3619785362 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b100f20a0 con 0x7f5b100d0340 2021-02-10T11:32:41.103 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:41.085 7f5b15898700 1 --2- 172.21.2.102:0/3619785362 >> v2:172.21.2.113:3300/0 conn(0x7f5b100d1c70 0x7f5b100efc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:32:41.103 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:41.085 7f5b14896700 1 --2- 172.21.2.102:0/3619785362 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5b100d0340 0x7f5b100ef640 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7f5b04006650 tx=0x7f5b04009a80).ready entity=mon.2 client_cookie=866f4ecad180bb79 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:41.103 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:41.085 7f5b03fff700 1 -- 172.21.2.102:0/3619785362 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5b0400c0c0 con 0x7f5b100d0340 2021-02-10T11:32:41.103 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:41.085 7f5b03fff700 1 -- 172.21.2.102:0/3619785362 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b0400c0c0 con 0x7f5b100d0340 2021-02-10T11:32:41.103 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:41.085 7f5b03fff700 1 -- 172.21.2.102:0/3619785362 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5b0400c340 con 0x7f5b100d0340 2021-02-10T11:32:41.104 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:41.085 7f5b1727b700 1 -- 172.21.2.102:0/3619785362 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5b100d0340 msgr2=0x7f5b100ef640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:41.104 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:41.085 7f5b1727b700 1 --2- 172.21.2.102:0/3619785362 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5b100d0340 0x7f5b100ef640 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7f5b04006650 tx=0x7f5b04009a80).stop 2021-02-10T11:32:41.104 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:41.086 7f5b1727b700 1 -- 172.21.2.102:0/3619785362 shutdown_connections 2021-02-10T11:32:41.104 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:41.086 7f5b1727b700 1 --2- 172.21.2.102:0/3619785362 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5b100d0340 0x7f5b100ef640 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:41.104 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:41.086 7f5b1727b700 1 --2- 172.21.2.102:0/3619785362 >> v2:172.21.2.113:3300/0 conn(0x7f5b100d1c70 0x7f5b100efc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:41.104 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:41.086 7f5b1727b700 1 -- 172.21.2.102:0/3619785362 >> 172.21.2.102:0/3619785362 conn(0x7f5b100ce450 msgr2=0x7f5b100d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:41.104 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:41.086 7f5b1727b700 1 -- 172.21.2.102:0/3619785362 shutdown_connections 2021-02-10T11:32:41.104 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:41.086 7f5b1727b700 1 -- 172.21.2.102:0/3619785362 wait complete. 2021-02-10T11:32:41.105 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:41.101 7f5b1727b700 1 Processor -- start 2021-02-10T11:32:41.106 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:41.101 7f5b1727b700 1 -- start start 2021-02-10T11:32:41.106 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:41.101 7f5b1727b700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5b10195650 0x7f5b100b5560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:41.106 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:41.101 7f5b1727b700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5b100b5b80 0x7f5b101a2e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:41.106 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:41.101 7f5b1727b700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5b100acfa0 0x7f5b101a3440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:41.106 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:41.101 7f5b1727b700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b100b6010 con 0x7f5b10195650 2021-02-10T11:32:41.106 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:41.101 7f5b1727b700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b101a3a60 con 0x7f5b100acfa0 2021-02-10T11:32:41.106 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:41.101 7f5b1727b700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b101a3c40 con 0x7f5b100b5b80 2021-02-10T11:32:41.106 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:41.101 7f5b15097700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5b10195650 0x7f5b100b5560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:41.106 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:41.101 7f5b15097700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5b10195650 0x7f5b100b5560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:34696/0 (socket says 172.21.2.102:34696) 2021-02-10T11:32:41.106 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:41.101 7f5b15097700 1 -- 172.21.2.102:0/3572577680 learned_addr learned my addr 172.21.2.102:0/3572577680 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:32:41.106 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:41.102 7f5b15898700 1 --2- 172.21.2.102:0/3572577680 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5b100acfa0 0x7f5b101a3440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:41.106 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:41.102 7f5b14896700 1 --2- 172.21.2.102:0/3572577680 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5b100b5b80 0x7f5b101a2e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:41.107 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:41.102 7f5b15097700 1 -- 172.21.2.102:0/3572577680 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5b100b5b80 msgr2=0x7f5b101a2e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:41.107 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:41.102 7f5b15097700 1 --2- 172.21.2.102:0/3572577680 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5b100b5b80 0x7f5b101a2e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:41.107 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:41.102 7f5b15097700 1 -- 172.21.2.102:0/3572577680 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5b100acfa0 msgr2=0x7f5b101a3440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:41.107 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:41.102 7f5b15097700 1 --2- 172.21.2.102:0/3572577680 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5b100acfa0 0x7f5b101a3440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:41.107 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:41.102 7f5b15097700 1 -- 172.21.2.102:0/3572577680 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b100bcae0 con 0x7f5b10195650 2021-02-10T11:32:41.107 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:41.102 7f5b15898700 1 --2- 172.21.2.102:0/3572577680 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5b100acfa0 0x7f5b101a3440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:32:41.107 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:41.102 7f5b15097700 1 --2- 172.21.2.102:0/3572577680 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5b10195650 0x7f5b100b5560 secure :-1 s=READY pgs=2308 cs=0 l=1 rev1=1 rx=0x7f5b0c00a4c0 tx=0x7f5b0c00a860).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:41.107 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:41.102 7f5b01ffb700 1 -- 172.21.2.102:0/3572577680 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5b0c00ac50 con 0x7f5b10195650 2021-02-10T11:32:41.107 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:41.102 7f5b01ffb700 1 -- 172.21.2.102:0/3572577680 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b0c004140 con 0x7f5b10195650 2021-02-10T11:32:41.107 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:41.102 7f5b01ffb700 1 -- 172.21.2.102:0/3572577680 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5b0c004660 con 0x7f5b10195650 2021-02-10T11:32:41.107 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:41.102 7f5b1727b700 1 -- 172.21.2.102:0/3572577680 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b100bcae0 con 0x7f5b10195650 2021-02-10T11:32:41.107 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:41.102 7f5b1727b700 1 -- 172.21.2.102:0/3572577680 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b100bd070 con 0x7f5b10195650 2021-02-10T11:32:41.107 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:41.103 7f5b01ffb700 1 -- 172.21.2.102:0/3572577680 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f5b0c009030 con 0x7f5b10195650 2021-02-10T11:32:41.107 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:41.103 7f5b01ffb700 1 --2- 172.21.2.102:0/3572577680 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5af00186c0 0x7f5af001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:41.108 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:41.103 7f5b01ffb700 1 -- 172.21.2.102:0/3572577680 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f5b0c032570 con 0x7f5b10195650 2021-02-10T11:32:41.108 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:41.103 7f5b14896700 1 --2- 172.21.2.102:0/3572577680 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5af00186c0 0x7f5af001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:41.108 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:41.103 7f5b14896700 1 --2- 172.21.2.102:0/3572577680 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5af00186c0 0x7f5af001abc0 crc :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:41.134 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:41.133 7f5b1727b700 1 -- 172.21.2.102:0/3572577680 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b100f0ab0 con 0x7f5b10195650 2021-02-10T11:32:41.137 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:41.136 7f5b01ffb700 1 -- 172.21.2.102:0/3572577680 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f5b0c030d20 con 0x7f5b10195650 2021-02-10T11:32:41.262 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:41.261 7f5b1727b700 1 -- 172.21.2.102:0/3572577680 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5b100f0ab0 con 0x7f5b10195650 2021-02-10T11:32:41.263 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:41.261 7f5b01ffb700 1 -- 172.21.2.102:0/3572577680 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v120) v1 ==== 74+0+14007 (secure 0 0 0) 0x7f5b0c009030 con 0x7f5b10195650 2021-02-10T11:32:41.263 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:32:41.263 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":120,"fsid":"68db30ea-dbce-4aef-a6df-f01c6fea0513","created":"2021-02-10 11:03:06.331557","modified":"2021-02-10 11:32:39.019652","last_up_change":"2021-02-10 11:03:32.690562","last_in_change":"2021-02-10 11:03:32.690562","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":25,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:03:37.322519","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":24,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:32:35.986688","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"119","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":25,"pool_name":"cephfs_data","create_time":"2021-02-10 11:32:37.011649","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"120","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"be691b4d-a039-408a-af80-9d53f02b6f1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6818","nonce":36088},{"type":"v1","addr":"172.21.2.102:6820","nonce":36088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6822","nonce":36088},{"type":"v1","addr":"172.21.2.102:6824","nonce":36088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6830","nonce":36088},{"type":"v1","addr":"172.21.2.102:6832","nonce":36088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6826","nonce":36088},{"type":"v1","addr":"172.21.2.102:6828","nonce":36088}]},"public_addr":"172.21.2.102:6820/36088","cluster_addr":"172.21.2.102:6824/36088","heartbeat_back_addr":"172.21.2.102:6832/36088","heartbeat_front_addr":"172.21.2.102:6828/36088","state":["exists","up"]},{"osd":1,"uuid":"64e3ccc8-9c69-4d7c-865e-af8e83822635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6819","nonce":36087},{"type":"v1","addr":"172.21.2.102:6821","nonce":36087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6823","nonce":36087},{"type":"v1","addr":"172.21.2.102:6825","nonce":36087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6831","nonce":36087},{"type":"v1","addr":"172.21.2.102:6833","nonce":36087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6827","nonce":36087},{"type":"v1","addr":"172.21.2.102:6829","nonce":36087}]},"public_addr":"172.21.2.102:6821/36087","cluster_addr":"172.21.2.102:6825/36087","heartbeat_back_addr":"172.21.2.102:6833/36087","heartbeat_front_addr":"172.21.2.102:6829/36087","state":["exists","up"]},{"osd":2,"uuid":"153700af-747f-481f-bdf4-d6072d3a842f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6810","nonce":36089},{"type":"v1","addr":"172.21.2.102:6811","nonce":36089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6812","nonce":36089},{"type":"v1","addr":"172.21.2.102:6813","nonce":36089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6816","nonce":36089},{"type":"v1","addr":"172.21.2.102:6817","nonce":36089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6814","nonce":36089},{"type":"v1","addr":"172.21.2.102:6815","nonce":36089}]},"public_addr":"172.21.2.102:6811/36089","cluster_addr":"172.21.2.102:6813/36089","heartbeat_back_addr":"172.21.2.102:6817/36089","heartbeat_front_addr":"172.21.2.102:6815/36089","state":["exists","up"]},{"osd":3,"uuid":"cbc678f4-83a9-45e6-8830-2f7c232b1ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6802","nonce":36090},{"type":"v1","addr":"172.21.2.102:6803","nonce":36090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6804","nonce":36090},{"type":"v1","addr":"172.21.2.102:6805","nonce":36090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6808","nonce":36090},{"type":"v1","addr":"172.21.2.102:6809","nonce":36090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6806","nonce":36090},{"type":"v1","addr":"172.21.2.102:6807","nonce":36090}]},"public_addr":"172.21.2.102:6803/36090","cluster_addr":"172.21.2.102:6805/36090","heartbeat_back_addr":"172.21.2.102:6809/36090","heartbeat_front_addr":"172.21.2.102:6807/36090","state":["exists","up"]},{"osd":4,"uuid":"c01e4008-b258-4f10-afd2-9e7f37214351","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6808","nonce":36368},{"type":"v1","addr":"172.21.2.113:6811","nonce":36368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6814","nonce":36368},{"type":"v1","addr":"172.21.2.113:6818","nonce":36368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6826","nonce":36368},{"type":"v1","addr":"172.21.2.113:6828","nonce":36368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6820","nonce":36368},{"type":"v1","addr":"172.21.2.113:6824","nonce":36368}]},"public_addr":"172.21.2.113:6811/36368","cluster_addr":"172.21.2.113:6818/36368","heartbeat_back_addr":"172.21.2.113:6828/36368","heartbeat_front_addr":"172.21.2.113:6824/36368","state":["exists","up"]},{"osd":5,"uuid":"551f6007-daf8-4a98-8ee0-c2b16bcccb5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6802","nonce":36366},{"type":"v1","addr":"172.21.2.113:6805","nonce":36366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6807","nonce":36366},{"type":"v1","addr":"172.21.2.113:6810","nonce":36366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6819","nonce":36366},{"type":"v1","addr":"172.21.2.113:6822","nonce":36366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6812","nonce":36366},{"type":"v1","addr":"172.21.2.113:6815","nonce":36366}]},"public_addr":"172.21.2.113:6805/36366","cluster_addr":"172.21.2.113:6810/36366","heartbeat_back_addr":"172.21.2.113:6822/36366","heartbeat_front_addr":"172.21.2.113:6815/36366","state":["exists","up"]},{"osd":6,"uuid":"bf67c040-0e50-48be-b102-c3943bc9ebef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6800","nonce":36367},{"type":"v1","addr":"172.21.2.113:6801","nonce":36367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6803","nonce":36367},{"type":"v1","addr":"172.21.2.113:6804","nonce":36367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6813","nonce":36367},{"type":"v1","addr":"172.21.2.113:6816","nonce":36367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6806","nonce":36367},{"type":"v1","addr":"172.21.2.113:6809","nonce":36367}]},"public_addr":"172.21.2.113:6801/36367","cluster_addr":"172.21.2.113:6804/36367","heartbeat_back_addr":"172.21.2.113:6816/36367","heartbeat_front_addr":"172.21.2.113:6809/36367","state":["exists","up"]},{"osd":7,"uuid":"f5461f80-018d-428f-be89-2eb2bcbe7bb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6817","nonce":36365},{"type":"v1","addr":"172.21.2.113:6821","nonce":36365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6823","nonce":36365},{"type":"v1","addr":"172.21.2.113:6825","nonce":36365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6830","nonce":36365},{"type":"v1","addr":"172.21.2.113:6831","nonce":36365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6827","nonce":36365},{"type":"v1","addr":"172.21.2.113:6829","nonce":36365}]},"public_addr":"172.21.2.113:6821/36365","cluster_addr":"172.21.2.113:6825/36365","heartbeat_back_addr":"172.21.2.113:6831/36365","heartbeat_front_addr":"172.21.2.113:6829/36365","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:32:41.264 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:41.263 7f5b1727b700 1 -- 172.21.2.102:0/3572577680 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5af00186c0 msgr2=0x7f5af001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:41.264 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:41.263 7f5b1727b700 1 --2- 172.21.2.102:0/3572577680 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5af00186c0 0x7f5af001abc0 crc :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:41.264 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:41.263 7f5b1727b700 1 -- 172.21.2.102:0/3572577680 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5b10195650 msgr2=0x7f5b100b5560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:41.264 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:41.263 7f5b1727b700 1 --2- 172.21.2.102:0/3572577680 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5b10195650 0x7f5b100b5560 secure :-1 s=READY pgs=2308 cs=0 l=1 rev1=1 rx=0x7f5b0c00a4c0 tx=0x7f5b0c00a860).stop 2021-02-10T11:32:41.265 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:41.263 7f5b1727b700 1 -- 172.21.2.102:0/3572577680 shutdown_connections 2021-02-10T11:32:41.265 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:41.263 7f5b1727b700 1 --2- 172.21.2.102:0/3572577680 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5af00186c0 0x7f5af001abc0 unknown :-1 s=CLOSED pgs=1017 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:41.265 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:41.263 7f5b1727b700 1 --2- 172.21.2.102:0/3572577680 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5b100acfa0 0x7f5b101a3440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:41.265 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:41.263 7f5b1727b700 1 --2- 172.21.2.102:0/3572577680 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5b100b5b80 0x7f5b101a2e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:41.265 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:41.263 7f5b1727b700 1 --2- 172.21.2.102:0/3572577680 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5b10195650 0x7f5b100b5560 unknown :-1 s=CLOSED pgs=2308 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:41.265 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:41.263 7f5b1727b700 1 -- 172.21.2.102:0/3572577680 >> 172.21.2.102:0/3572577680 conn(0x7f5b100b3040 msgr2=0x7f5b100c3250 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:41.265 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:41.263 7f5b1727b700 1 -- 172.21.2.102:0/3572577680 shutdown_connections 2021-02-10T11:32:41.266 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:41.263 7f5b1727b700 1 -- 172.21.2.102:0/3572577680 wait complete. 2021-02-10T11:32:41.273 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:32:41.714 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:41.694 7f191176f700 1 Processor -- start 2021-02-10T11:32:41.714 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:41.694 7f191176f700 1 -- start start 2021-02-10T11:32:41.714 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:41.694 7f191176f700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f190c0d1880 0x7f190c1b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:41.715 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:41.694 7f191176f700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f190c1b0af0 0x7f190c108950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:41.715 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:41.694 7f191176f700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f190c0d1c80 0x7f190c108f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:41.715 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:41.694 7f191176f700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f190c0e5f00 con 0x7f190c1b0af0 2021-02-10T11:32:41.715 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:41.694 7f191176f700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f190c109a70 con 0x7f190c0d1c80 2021-02-10T11:32:41.715 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:41.694 7f191176f700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f190c088070 con 0x7f190c0d1880 2021-02-10T11:32:41.715 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:41.694 7f190ad73700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f190c1b0af0 0x7f190c108950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:41.715 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:41.694 7f190ad73700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f190c1b0af0 0x7f190c108950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:34704/0 (socket says 172.21.2.102:34704) 2021-02-10T11:32:41.715 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:41.694 7f190ad73700 1 -- 172.21.2.102:0/2749257209 learned_addr learned my addr 172.21.2.102:0/2749257209 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:32:41.715 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:41.694 7f190bd75700 1 --2- 172.21.2.102:0/2749257209 >> v2:172.21.2.113:3300/0 conn(0x7f190c0d1c80 0x7f190c108f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:41.715 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:41.694 7f190b574700 1 --2- 172.21.2.102:0/2749257209 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f190c0d1880 0x7f190c1b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:41.715 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:41.695 7f190ad73700 1 -- 172.21.2.102:0/2749257209 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f190c0d1880 msgr2=0x7f190c1b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:32:41.715 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:41.695 7f190ad73700 1 --2- 172.21.2.102:0/2749257209 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f190c0d1880 0x7f190c1b04d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:41.716 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:41.695 7f190ad73700 1 -- 172.21.2.102:0/2749257209 >> v2:172.21.2.113:3300/0 conn(0x7f190c0d1c80 msgr2=0x7f190c108f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:32:41.716 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:41.695 7f190ad73700 1 --2- 172.21.2.102:0/2749257209 >> v2:172.21.2.113:3300/0 conn(0x7f190c0d1c80 0x7f190c108f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:41.716 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:41.695 7f190ad73700 1 -- 172.21.2.102:0/2749257209 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f190c109590 con 0x7f190c1b0af0 2021-02-10T11:32:41.716 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:41.695 7f190bd75700 1 --2- 172.21.2.102:0/2749257209 >> v2:172.21.2.113:3300/0 conn(0x7f190c0d1c80 0x7f190c108f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:32:41.716 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:41.695 7f190ad73700 1 --2- 172.21.2.102:0/2749257209 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f190c1b0af0 0x7f190c108950 secure :-1 s=READY pgs=2309 cs=0 l=1 rev1=1 rx=0x7f1900006700 tx=0x7f1900009b80).ready entity=mon.0 client_cookie=2de006d0e63f9dc5 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:41.716 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:41.695 7f190a572700 1 -- 172.21.2.102:0/2749257209 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f190000c0c0 con 0x7f190c1b0af0 2021-02-10T11:32:41.716 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:41.695 7f190a572700 1 -- 172.21.2.102:0/2749257209 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f190000c380 con 0x7f190c1b0af0 2021-02-10T11:32:41.716 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:41.695 7f190a572700 1 -- 172.21.2.102:0/2749257209 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f190000c860 con 0x7f190c1b0af0 2021-02-10T11:32:41.716 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:41.695 7f191176f700 1 -- 172.21.2.102:0/2749257209 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f190c1b0af0 msgr2=0x7f190c108950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:41.716 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:41.695 7f191176f700 1 --2- 172.21.2.102:0/2749257209 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f190c1b0af0 0x7f190c108950 secure :-1 s=READY pgs=2309 cs=0 l=1 rev1=1 rx=0x7f1900006700 tx=0x7f1900009b80).stop 2021-02-10T11:32:41.716 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:41.696 7f191176f700 1 -- 172.21.2.102:0/2749257209 shutdown_connections 2021-02-10T11:32:41.717 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:41.696 7f191176f700 1 --2- 172.21.2.102:0/2749257209 >> v2:172.21.2.113:3300/0 conn(0x7f190c0d1c80 0x7f190c108f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:41.717 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:41.696 7f191176f700 1 --2- 172.21.2.102:0/2749257209 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f190c1b0af0 0x7f190c108950 unknown :-1 s=CLOSED pgs=2309 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:41.717 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:41.696 7f191176f700 1 --2- 172.21.2.102:0/2749257209 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f190c0d1880 0x7f190c1b04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:41.717 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:41.696 7f191176f700 1 -- 172.21.2.102:0/2749257209 >> 172.21.2.102:0/2749257209 conn(0x7f190c0ce430 msgr2=0x7f190c0d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:41.717 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:41.696 7f191176f700 1 -- 172.21.2.102:0/2749257209 shutdown_connections 2021-02-10T11:32:41.717 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:41.696 7f191176f700 1 -- 172.21.2.102:0/2749257209 wait complete. 2021-02-10T11:32:41.717 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:41.710 7f191176f700 1 Processor -- start 2021-02-10T11:32:41.717 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:41.710 7f191176f700 1 -- start start 2021-02-10T11:32:41.717 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:41.711 7f191176f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f190c1b9470 0x7f190c1ae6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:41.717 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:41.711 7f191176f700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f190c1aecf0 0x7f190c0957f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:41.717 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:41.711 7f191176f700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f190c1af0f0 0x7f190c095e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:41.717 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:41.711 7f191176f700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f190c096d70 con 0x7f190c1b9470 2021-02-10T11:32:41.718 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:41.711 7f191176f700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f190c0a7780 con 0x7f190c1af0f0 2021-02-10T11:32:41.718 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:41.711 7f191176f700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f190c1b9870 con 0x7f190c1aecf0 2021-02-10T11:32:41.718 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:41.711 7f190b574700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f190c1b9470 0x7f190c1ae6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:41.718 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:41.711 7f190b574700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f190c1b9470 0x7f190c1ae6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:34710/0 (socket says 172.21.2.102:34710) 2021-02-10T11:32:41.718 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:41.711 7f190b574700 1 -- 172.21.2.102:0/3537785152 learned_addr learned my addr 172.21.2.102:0/3537785152 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:32:41.718 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:41.711 7f190bd75700 1 --2- 172.21.2.102:0/3537785152 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f190c1af0f0 0x7f190c095e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:41.719 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:41.711 7f190ad73700 1 --2- 172.21.2.102:0/3537785152 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f190c1aecf0 0x7f190c0957f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:41.719 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:41.712 7f190b574700 1 -- 172.21.2.102:0/3537785152 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f190c1aecf0 msgr2=0x7f190c0957f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:41.719 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:41.712 7f190b574700 1 --2- 172.21.2.102:0/3537785152 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f190c1aecf0 0x7f190c0957f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:41.719 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:41.712 7f190b574700 1 -- 172.21.2.102:0/3537785152 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f190c1af0f0 msgr2=0x7f190c095e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:41.719 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:41.712 7f190b574700 1 --2- 172.21.2.102:0/3537785152 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f190c1af0f0 0x7f190c095e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:41.719 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:41.712 7f190b574700 1 -- 172.21.2.102:0/3537785152 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f190c19e6b0 con 0x7f190c1b9470 2021-02-10T11:32:41.719 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:41.712 7f190b574700 1 --2- 172.21.2.102:0/3537785152 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f190c1b9470 0x7f190c1ae6d0 secure :-1 s=READY pgs=2310 cs=0 l=1 rev1=1 rx=0x7f1904009980 tx=0x7f190400bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:41.719 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:41.712 7f18f3fff700 1 -- 172.21.2.102:0/3537785152 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1904006030 con 0x7f190c1b9470 2021-02-10T11:32:41.719 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:41.712 7f18f3fff700 1 -- 172.21.2.102:0/3537785152 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19040062b0 con 0x7f190c1b9470 2021-02-10T11:32:41.719 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:41.712 7f18f3fff700 1 -- 172.21.2.102:0/3537785152 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1904005660 con 0x7f190c1b9470 2021-02-10T11:32:41.719 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:41.712 7f191176f700 1 -- 172.21.2.102:0/3537785152 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f190c19e6b0 con 0x7f190c1b9470 2021-02-10T11:32:41.719 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:41.712 7f191176f700 1 -- 172.21.2.102:0/3537785152 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f190c19e6b0 con 0x7f190c1b9470 2021-02-10T11:32:41.720 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:41.712 7f18f3fff700 1 -- 172.21.2.102:0/3537785152 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1904006490 con 0x7f190c1b9470 2021-02-10T11:32:41.720 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:41.713 7f18f3fff700 1 --2- 172.21.2.102:0/3537785152 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f18ec0186c0 0x7f18ec01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:41.720 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:41.713 7f18f3fff700 1 -- 172.21.2.102:0/3537785152 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f1904032b70 con 0x7f190c1b9470 2021-02-10T11:32:41.720 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:41.713 7f190ad73700 1 --2- 172.21.2.102:0/3537785152 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f18ec0186c0 0x7f18ec01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:41.721 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:41.713 7f190ad73700 1 --2- 172.21.2.102:0/3537785152 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f18ec0186c0 0x7f18ec01abc0 crc :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:41.745 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:41.744 7f191176f700 1 -- 172.21.2.102:0/3537785152 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f190c0829a0 con 0x7f190c1b9470 2021-02-10T11:32:41.748 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:41.747 7f18f3fff700 1 -- 172.21.2.102:0/3537785152 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f190400be10 con 0x7f190c1b9470 2021-02-10T11:32:41.878 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:41.877 7f191176f700 1 -- 172.21.2.102:0/3537785152 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f190c109a70 con 0x7f190c1b9470 2021-02-10T11:32:41.882 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:41.881 7f18f3fff700 1 -- 172.21.2.102:0/3537785152 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.0 v67) v1 ==== 184+0+0 (secure 0 0 0) 0x7f190400be10 con 0x7f190c1b9470 2021-02-10T11:32:41.886 INFO:teuthology.orchestra.run.gibba002.stderr:updated caps for client.0 2021-02-10T11:32:41.887 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:41.886 7f191176f700 1 -- 172.21.2.102:0/3537785152 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f18ec0186c0 msgr2=0x7f18ec01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:41.887 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:41.886 7f191176f700 1 --2- 172.21.2.102:0/3537785152 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f18ec0186c0 0x7f18ec01abc0 crc :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:41.887 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:41.886 7f191176f700 1 -- 172.21.2.102:0/3537785152 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f190c1b9470 msgr2=0x7f190c1ae6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:41.887 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:41.886 7f191176f700 1 --2- 172.21.2.102:0/3537785152 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f190c1b9470 0x7f190c1ae6d0 secure :-1 s=READY pgs=2310 cs=0 l=1 rev1=1 rx=0x7f1904009980 tx=0x7f190400bcd0).stop 2021-02-10T11:32:41.888 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:41.886 7f191176f700 1 -- 172.21.2.102:0/3537785152 shutdown_connections 2021-02-10T11:32:41.888 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:41.886 7f191176f700 1 --2- 172.21.2.102:0/3537785152 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f18ec0186c0 0x7f18ec01abc0 unknown :-1 s=CLOSED pgs=1018 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:41.888 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:41.886 7f191176f700 1 --2- 172.21.2.102:0/3537785152 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f190c1af0f0 0x7f190c095e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:41.888 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:41.886 7f191176f700 1 --2- 172.21.2.102:0/3537785152 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f190c1aecf0 0x7f190c0957f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:41.888 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:41.886 7f191176f700 1 --2- 172.21.2.102:0/3537785152 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f190c1b9470 0x7f190c1ae6d0 unknown :-1 s=CLOSED pgs=2310 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:41.888 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:41.886 7f191176f700 1 -- 172.21.2.102:0/3537785152 >> 172.21.2.102:0/3537785152 conn(0x7f190c0845b0 msgr2=0x7f190c18c290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:41.888 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:41.886 7f191176f700 1 -- 172.21.2.102:0/3537785152 shutdown_connections 2021-02-10T11:32:41.888 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:41.886 7f191176f700 1 -- 172.21.2.102:0/3537785152 wait complete. 2021-02-10T11:32:41.896 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.1 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:32:42.336 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:42.317 7fa82b69e700 1 Processor -- start 2021-02-10T11:32:42.336 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:42.317 7fa82b69e700 1 -- start start 2021-02-10T11:32:42.336 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:42.318 7fa82b69e700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa8240d1c80 0x7fa8240cef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:42.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:42.318 7fa82b69e700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8240d0180 con 0x7fa8240d1880 2021-02-10T11:32:42.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:42.318 7fa82b69e700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8240fe1d0 con 0x7fa8240cf590 2021-02-10T11:32:42.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:42.318 7fa82b69e700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8241b0cc0 con 0x7fa8240d1c80 2021-02-10T11:32:42.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:42.318 7fa8294ba700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fa8240d1880 0x7fa8241b04d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:55172/0 (socket says 172.21.2.102:55172) 2021-02-10T11:32:42.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:42.318 7fa8294ba700 1 -- 172.21.2.102:0/109971128 learned_addr learned my addr 172.21.2.102:0/109971128 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:32:42.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:42.318 7fa817fff700 1 -- 172.21.2.102:0/109971128 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4126632713 0 0) 0x7fa8200010b0 con 0x7fa8240d1880 2021-02-10T11:32:42.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:42.318 7fa817fff700 1 -- 172.21.2.102:0/109971128 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa810002fe0 con 0x7fa8240d1880 2021-02-10T11:32:42.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:42.319 7fa817fff700 1 -- 172.21.2.102:0/109971128 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3000796113 0 0) 0x7fa8200010b0 con 0x7fa8240d1880 2021-02-10T11:32:42.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:42.319 7fa817fff700 1 -- 172.21.2.102:0/109971128 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa8240d1c80 msgr2=0x7fa8240cef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:32:42.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:42.319 7fa817fff700 1 --2- 172.21.2.102:0/109971128 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa8240d1c80 0x7fa8240cef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:42.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:42.319 7fa817fff700 1 -- 172.21.2.102:0/109971128 >> v1:172.21.2.113:6789/0 conn(0x7fa8240cf590 legacy=0x7fa8240cf990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:32:42.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:42.319 7fa817fff700 1 -- 172.21.2.102:0/109971128 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa824109a70 con 0x7fa8240d1880 2021-02-10T11:32:42.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:42.319 7fa817fff700 1 -- 172.21.2.102:0/109971128 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7fa8200015b0 con 0x7fa8240d1880 2021-02-10T11:32:42.338 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:42.319 7fa817fff700 1 -- 172.21.2.102:0/109971128 >> v1:172.21.2.102:6789/0 conn(0x7fa8240d1880 legacy=0x7fa8241b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:42.338 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:42.319 7fa817fff700 1 --2- 172.21.2.102:0/109971128 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa810003aa0 0x7fa810005f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:42.338 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:42.319 7fa817fff700 1 --2- 172.21.2.102:0/109971128 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa810006780 0x7fa810008c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:42.338 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:42.319 7fa817fff700 1 --2- 172.21.2.102:0/109971128 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa810009510 0x7fa81000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:42.338 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:42.319 7fa817fff700 1 -- 172.21.2.102:0/109971128 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa81000c2c0 con 0x7fa810003aa0 2021-02-10T11:32:42.338 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:42.319 7fa817fff700 1 -- 172.21.2.102:0/109971128 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa81000c4e0 con 0x7fa810009510 2021-02-10T11:32:42.338 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:42.319 7fa817fff700 1 -- 172.21.2.102:0/109971128 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa81000c700 con 0x7fa810006780 2021-02-10T11:32:42.338 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:42.319 7fa8294ba700 1 --2- 172.21.2.102:0/109971128 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa810003aa0 0x7fa810005f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:42.338 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:42.319 7fa829cbb700 1 --2- 172.21.2.102:0/109971128 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa810009510 0x7fa81000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:42.338 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:42.319 7fa8294ba700 1 -- 172.21.2.102:0/109971128 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa810006780 msgr2=0x7fa810008c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:32:42.338 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:42.319 7fa828cb9700 1 --2- 172.21.2.102:0/109971128 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa810006780 0x7fa810008c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:42.338 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:42.319 7fa8294ba700 1 --2- 172.21.2.102:0/109971128 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa810006780 0x7fa810008c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:42.338 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:42.319 7fa8294ba700 1 -- 172.21.2.102:0/109971128 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa810009510 msgr2=0x7fa81000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:32:42.339 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:42.319 7fa8294ba700 1 --2- 172.21.2.102:0/109971128 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa810009510 0x7fa81000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:42.339 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:42.319 7fa8294ba700 1 -- 172.21.2.102:0/109971128 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa81000c9a0 con 0x7fa810003aa0 2021-02-10T11:32:42.339 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:42.319 7fa828cb9700 1 --2- 172.21.2.102:0/109971128 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa810006780 0x7fa810008c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:32:42.339 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:42.319 7fa8294ba700 1 --2- 172.21.2.102:0/109971128 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa810003aa0 0x7fa810005f60 secure :-1 s=READY pgs=2312 cs=0 l=1 rev1=1 rx=0x7fa8200017d0 tx=0x7fa820007db0).ready entity=mon.0 client_cookie=e8a5ca7d168e3fe8 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:42.339 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:42.319 7fa817fff700 1 -- 172.21.2.102:0/109971128 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa820007030 con 0x7fa810003aa0 2021-02-10T11:32:42.339 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:42.319 7fa817fff700 1 -- 172.21.2.102:0/109971128 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8200009b0 con 0x7fa810003aa0 2021-02-10T11:32:42.339 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:42.319 7fa817fff700 1 -- 172.21.2.102:0/109971128 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa82000a340 con 0x7fa810003aa0 2021-02-10T11:32:42.339 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:42.319 7fa82b69e700 1 -- 172.21.2.102:0/109971128 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa810003aa0 msgr2=0x7fa810005f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:42.339 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:42.319 7fa82b69e700 1 --2- 172.21.2.102:0/109971128 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa810003aa0 0x7fa810005f60 secure :-1 s=READY pgs=2312 cs=0 l=1 rev1=1 rx=0x7fa8200017d0 tx=0x7fa820007db0).stop 2021-02-10T11:32:42.339 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:42.319 7fa829cbb700 1 -- 172.21.2.102:0/109971128 reap_dead start 2021-02-10T11:32:42.339 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:42.320 7fa82b69e700 1 -- 172.21.2.102:0/109971128 shutdown_connections 2021-02-10T11:32:42.339 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:42.320 7fa82b69e700 1 -- 172.21.2.102:0/109971128 >> 172.21.2.102:0/109971128 conn(0x7fa8240ce430 msgr2=0x7fa8240d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:42.339 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:42.320 7fa82b69e700 1 -- 172.21.2.102:0/109971128 shutdown_connections 2021-02-10T11:32:42.340 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:42.320 7fa82b69e700 1 -- 172.21.2.102:0/109971128 wait complete. 2021-02-10T11:32:42.340 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:42.335 7fa82b69e700 1 Processor -- start 2021-02-10T11:32:42.340 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:42.335 7fa82b69e700 1 -- start start 2021-02-10T11:32:42.341 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:42.335 7fa82b69e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa8241a4a90 0x7fa8240bfd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:42.341 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:42.335 7fa82b69e700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa8240c0360 0x7fa8240c0b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:42.341 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:42.335 7fa82b69e700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa8240c1180 0x7fa8240c1580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:42.341 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:42.335 7fa82b69e700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8240c1ba0 con 0x7fa8241a4a90 2021-02-10T11:32:42.341 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:42.335 7fa82b69e700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8240c1dc0 con 0x7fa8240c1180 2021-02-10T11:32:42.341 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:42.335 7fa82b69e700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8240b6570 con 0x7fa8240c0360 2021-02-10T11:32:42.341 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:42.335 7fa8294ba700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa8241a4a90 0x7fa8240bfd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:42.341 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:42.335 7fa8294ba700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa8241a4a90 0x7fa8240bfd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:34730/0 (socket says 172.21.2.102:34730) 2021-02-10T11:32:42.341 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:42.335 7fa8294ba700 1 -- 172.21.2.102:0/3970667030 learned_addr learned my addr 172.21.2.102:0/3970667030 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:32:42.342 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:42.336 7fa8294ba700 1 -- 172.21.2.102:0/3970667030 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa8240c0360 msgr2=0x7fa8240c0b60 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:32:42.342 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:42.336 7fa8294ba700 1 --2- 172.21.2.102:0/3970667030 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa8240c0360 0x7fa8240c0b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:42.342 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:42.336 7fa8294ba700 1 -- 172.21.2.102:0/3970667030 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa8240c1180 msgr2=0x7fa8240c1580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:42.342 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:42.336 7fa8294ba700 1 --2- 172.21.2.102:0/3970667030 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa8240c1180 0x7fa8240c1580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:42.342 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:42.336 7fa8294ba700 1 -- 172.21.2.102:0/3970667030 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8240b6790 con 0x7fa8241a4a90 2021-02-10T11:32:42.342 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:42.336 7fa8294ba700 1 --2- 172.21.2.102:0/3970667030 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa8241a4a90 0x7fa8240bfd40 secure :-1 s=READY pgs=2313 cs=0 l=1 rev1=1 rx=0x7fa820004450 tx=0x7fa8200047f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:42.342 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:42.336 7fa815ffb700 1 -- 172.21.2.102:0/3970667030 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa820000990 con 0x7fa8241a4a90 2021-02-10T11:32:42.342 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:42.336 7fa815ffb700 1 -- 172.21.2.102:0/3970667030 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa820006390 con 0x7fa8241a4a90 2021-02-10T11:32:42.343 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:42.336 7fa82b69e700 1 -- 172.21.2.102:0/3970667030 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8240b6790 con 0x7fa8241a4a90 2021-02-10T11:32:42.343 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:42.336 7fa815ffb700 1 -- 172.21.2.102:0/3970667030 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa820006820 con 0x7fa8241a4a90 2021-02-10T11:32:42.343 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:42.336 7fa82b69e700 1 -- 172.21.2.102:0/3970667030 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8240b6d20 con 0x7fa8241a4a90 2021-02-10T11:32:42.343 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:42.337 7fa815ffb700 1 -- 172.21.2.102:0/3970667030 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa820006570 con 0x7fa8241a4a90 2021-02-10T11:32:42.343 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:42.337 7fa815ffb700 1 --2- 172.21.2.102:0/3970667030 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa8040186d0 0x7fa80401abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:42.343 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:42.337 7fa815ffb700 1 -- 172.21.2.102:0/3970667030 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fa820076610 con 0x7fa8241a4a90 2021-02-10T11:32:42.343 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:42.337 7fa828cb9700 1 --2- 172.21.2.102:0/3970667030 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa8040186d0 0x7fa80401abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:42.343 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:42.337 7fa828cb9700 1 --2- 172.21.2.102:0/3970667030 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa8040186d0 0x7fa80401abd0 crc :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:42.369 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:42.367 7fa82b69e700 1 -- 172.21.2.102:0/3970667030 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8240fe1d0 con 0x7fa8241a4a90 2021-02-10T11:32:42.371 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:42.370 7fa815ffb700 1 -- 172.21.2.102:0/3970667030 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fa820001070 con 0x7fa8241a4a90 2021-02-10T11:32:42.501 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:42.500 7fa82b69e700 1 -- 172.21.2.102:0/3970667030 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7fa8240fe1d0 con 0x7fa8241a4a90 2021-02-10T11:32:42.505 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:42.504 7fa815ffb700 1 -- 172.21.2.102:0/3970667030 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.1 v68) v1 ==== 184+0+0 (secure 0 0 0) 0x7fa8200634d0 con 0x7fa8241a4a90 2021-02-10T11:32:42.509 INFO:teuthology.orchestra.run.gibba002.stderr:updated caps for client.1 2021-02-10T11:32:42.510 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:42.509 7fa82b69e700 1 -- 172.21.2.102:0/3970667030 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa8040186d0 msgr2=0x7fa80401abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:42.510 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:42.509 7fa82b69e700 1 --2- 172.21.2.102:0/3970667030 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa8040186d0 0x7fa80401abd0 crc :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:42.510 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:42.509 7fa82b69e700 1 -- 172.21.2.102:0/3970667030 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa8241a4a90 msgr2=0x7fa8240bfd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:42.510 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:42.509 7fa82b69e700 1 --2- 172.21.2.102:0/3970667030 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa8241a4a90 0x7fa8240bfd40 secure :-1 s=READY pgs=2313 cs=0 l=1 rev1=1 rx=0x7fa820004450 tx=0x7fa8200047f0).stop 2021-02-10T11:32:42.511 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:42.509 7fa82b69e700 1 -- 172.21.2.102:0/3970667030 shutdown_connections 2021-02-10T11:32:42.511 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:42.509 7fa82b69e700 1 --2- 172.21.2.102:0/3970667030 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa8040186d0 0x7fa80401abd0 unknown :-1 s=CLOSED pgs=1019 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:42.511 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:42.509 7fa82b69e700 1 --2- 172.21.2.102:0/3970667030 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa8240c1180 0x7fa8240c1580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:42.511 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:42.509 7fa82b69e700 1 --2- 172.21.2.102:0/3970667030 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa8240c0360 0x7fa8240c0b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:42.511 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:42.509 7fa82b69e700 1 --2- 172.21.2.102:0/3970667030 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa8241a4a90 0x7fa8240bfd40 unknown :-1 s=CLOSED pgs=2313 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:42.511 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:42.509 7fa82b69e700 1 -- 172.21.2.102:0/3970667030 >> 172.21.2.102:0/3970667030 conn(0x7fa8240ac2b0 msgr2=0x7fa824082bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:42.511 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:42.509 7fa82b69e700 1 -- 172.21.2.102:0/3970667030 shutdown_connections 2021-02-10T11:32:42.511 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:42.509 7fa82b69e700 1 -- 172.21.2.102:0/3970667030 wait complete. 2021-02-10T11:32:42.519 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.2 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:32:42.960 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:42.942 7fb09afd1700 1 Processor -- start 2021-02-10T11:32:42.960 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:42.942 7fb09afd1700 1 -- start start 2021-02-10T11:32:42.960 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:42.943 7fb09afd1700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fb0940d1860 0x7fb0941b04b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:42.960 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:42.943 7fb09afd1700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb0940cff60 0x7fb094109120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:42.961 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:42.943 7fb09afd1700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb094109a50 con 0x7fb0941b0ad0 2021-02-10T11:32:42.961 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:42.943 7fb09afd1700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb094109740 con 0x7fb0940d1860 2021-02-10T11:32:42.961 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:42.943 7fb09afd1700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb094088070 con 0x7fb0940cff60 2021-02-10T11:32:42.961 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:42.943 7fb098ded700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fb0940d1860 0x7fb0941b04b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:42.961 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:42.943 7fb098ded700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fb0940d1860 0x7fb0941b04b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:34738/0 (socket says 172.21.2.102:34738) 2021-02-10T11:32:42.961 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:42.943 7fb098ded700 1 -- 172.21.2.102:0/563508693 learned_addr learned my addr 172.21.2.102:0/563508693 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:32:42.961 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:42.943 7fb098ded700 1 -- 172.21.2.102:0/563508693 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb0940cff60 msgr2=0x7fb094109120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:32:42.961 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:42.943 7fb098ded700 1 --2- 172.21.2.102:0/563508693 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb0940cff60 0x7fb094109120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:42.961 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:42.943 7fb098ded700 1 -- 172.21.2.102:0/563508693 >> v1:172.21.2.113:6789/0 conn(0x7fb0941b0ad0 legacy=0x7fb094108930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:32:42.961 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:42.943 7fb098ded700 1 -- 172.21.2.102:0/563508693 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0940d1c60 con 0x7fb0940d1860 2021-02-10T11:32:42.961 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:42.944 7fb098ded700 1 --2- 172.21.2.102:0/563508693 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb0940d1860 0x7fb0941b04b0 secure :-1 s=READY pgs=2314 cs=0 l=1 rev1=1 rx=0x7fb07c006700 tx=0x7fb07c009b00).ready entity=mon.0 client_cookie=731a860ad8441e7e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:42.961 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:42.944 7fb0937fe700 1 -- 172.21.2.102:0/563508693 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb07c00c0c0 con 0x7fb0940d1860 2021-02-10T11:32:42.962 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:42.944 7fb0937fe700 1 -- 172.21.2.102:0/563508693 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb07c00c400 con 0x7fb0940d1860 2021-02-10T11:32:42.962 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:42.944 7fb0937fe700 1 -- 172.21.2.102:0/563508693 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb07c00c8e0 con 0x7fb0940d1860 2021-02-10T11:32:42.962 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:42.944 7fb09afd1700 1 -- 172.21.2.102:0/563508693 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb0940d1860 msgr2=0x7fb0941b04b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:42.962 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:42.944 7fb09afd1700 1 --2- 172.21.2.102:0/563508693 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb0940d1860 0x7fb0941b04b0 secure :-1 s=READY pgs=2314 cs=0 l=1 rev1=1 rx=0x7fb07c006700 tx=0x7fb07c009b00).stop 2021-02-10T11:32:42.962 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:42.944 7fb09afd1700 1 -- 172.21.2.102:0/563508693 shutdown_connections 2021-02-10T11:32:42.962 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:42.944 7fb09afd1700 1 --2- 172.21.2.102:0/563508693 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb0940cff60 0x7fb094109120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:42.962 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:42.944 7fb09afd1700 1 --2- 172.21.2.102:0/563508693 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb0940d1860 0x7fb0941b04b0 unknown :-1 s=CLOSED pgs=2314 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:42.962 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:42.944 7fb09afd1700 1 -- 172.21.2.102:0/563508693 >> 172.21.2.102:0/563508693 conn(0x7fb0940ce410 msgr2=0x7fb0940d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:42.962 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:42.944 7fb09afd1700 1 -- 172.21.2.102:0/563508693 shutdown_connections 2021-02-10T11:32:42.962 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:42.944 7fb09afd1700 1 -- 172.21.2.102:0/563508693 wait complete. 2021-02-10T11:32:42.964 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:42.959 7fb09afd1700 1 Processor -- start 2021-02-10T11:32:42.964 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:42.959 7fb09afd1700 1 -- start start 2021-02-10T11:32:42.964 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:42.959 7fb09afd1700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb094195500 0x7fb0940b0410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:42.964 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:42.959 7fb09afd1700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb0940b0a30 0x7fb09418f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:42.964 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:42.959 7fb09afd1700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb09418f990 0x7fb094190680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:42.964 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:42.959 7fb09afd1700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb094195a60 con 0x7fb0940b0a30 2021-02-10T11:32:42.964 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:42.959 7fb09afd1700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0941a5970 con 0x7fb094195500 2021-02-10T11:32:42.964 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:42.959 7fb09afd1700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb09418fd90 con 0x7fb09418f990 2021-02-10T11:32:42.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:42.960 7fb093fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb0940b0a30 0x7fb09418f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:42.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:42.960 7fb093fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb0940b0a30 0x7fb09418f370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:34744/0 (socket says 172.21.2.102:34744) 2021-02-10T11:32:42.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:42.960 7fb093fff700 1 -- 172.21.2.102:0/2148203347 learned_addr learned my addr 172.21.2.102:0/2148203347 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:32:42.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:42.960 7fb093fff700 1 -- 172.21.2.102:0/2148203347 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb09418f990 msgr2=0x7fb094190680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:42.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:42.960 7fb093fff700 1 --2- 172.21.2.102:0/2148203347 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb09418f990 0x7fb094190680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:42.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:42.960 7fb093fff700 1 -- 172.21.2.102:0/2148203347 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb094195500 msgr2=0x7fb0940b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:42.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:42.960 7fb093fff700 1 --2- 172.21.2.102:0/2148203347 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb094195500 0x7fb0940b0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:42.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:42.960 7fb093fff700 1 -- 172.21.2.102:0/2148203347 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb094190ca0 con 0x7fb0940b0a30 2021-02-10T11:32:42.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:42.960 7fb093fff700 1 --2- 172.21.2.102:0/2148203347 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb0940b0a30 0x7fb09418f370 secure :-1 s=READY pgs=2315 cs=0 l=1 rev1=1 rx=0x7fb08400b5b0 tx=0x7fb08400b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:42.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:42.961 7fb0917fa700 1 -- 172.21.2.102:0/2148203347 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb08400bd40 con 0x7fb0940b0a30 2021-02-10T11:32:42.966 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:42.961 7fb0917fa700 1 -- 172.21.2.102:0/2148203347 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb084006020 con 0x7fb0940b0a30 2021-02-10T11:32:42.966 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:42.961 7fb0917fa700 1 -- 172.21.2.102:0/2148203347 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb0840054a0 con 0x7fb0940b0a30 2021-02-10T11:32:42.966 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:42.961 7fb09afd1700 1 -- 172.21.2.102:0/2148203347 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb094190ca0 con 0x7fb0940b0a30 2021-02-10T11:32:42.966 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:42.961 7fb09afd1700 1 -- 172.21.2.102:0/2148203347 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb094191230 con 0x7fb0940b0a30 2021-02-10T11:32:42.966 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:42.961 7fb0917fa700 1 -- 172.21.2.102:0/2148203347 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fb084006200 con 0x7fb0940b0a30 2021-02-10T11:32:42.966 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:42.961 7fb0917fa700 1 --2- 172.21.2.102:0/2148203347 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb0780186c0 0x7fb07801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:42.966 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:42.961 7fb0917fa700 1 -- 172.21.2.102:0/2148203347 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fb0840325a0 con 0x7fb0940b0a30 2021-02-10T11:32:42.966 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:42.962 7fb098ded700 1 --2- 172.21.2.102:0/2148203347 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb0780186c0 0x7fb07801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:42.966 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:42.962 7fb098ded700 1 --2- 172.21.2.102:0/2148203347 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb0780186c0 0x7fb07801abc0 crc :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:42.994 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:42.992 7fb09afd1700 1 -- 172.21.2.102:0/2148203347 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb094082980 con 0x7fb0940b0a30 2021-02-10T11:32:42.996 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:42.995 7fb0917fa700 1 -- 172.21.2.102:0/2148203347 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fb084016870 con 0x7fb0940b0a30 2021-02-10T11:32:43.126 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:43.125 7fb09afd1700 1 -- 172.21.2.102:0/2148203347 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7fb094082980 con 0x7fb0940b0a30 2021-02-10T11:32:43.129 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:43.128 7fb0917fa700 1 -- 172.21.2.102:0/2148203347 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.2 v69) v1 ==== 184+0+0 (secure 0 0 0) 0x7fb084006200 con 0x7fb0940b0a30 2021-02-10T11:32:43.133 INFO:teuthology.orchestra.run.gibba002.stderr:updated caps for client.2 2021-02-10T11:32:43.135 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:43.133 7fb09afd1700 1 -- 172.21.2.102:0/2148203347 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb0780186c0 msgr2=0x7fb07801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:43.135 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:43.133 7fb09afd1700 1 --2- 172.21.2.102:0/2148203347 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb0780186c0 0x7fb07801abc0 crc :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:43.135 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:43.133 7fb09afd1700 1 -- 172.21.2.102:0/2148203347 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb0940b0a30 msgr2=0x7fb09418f370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:43.135 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:43.133 7fb09afd1700 1 --2- 172.21.2.102:0/2148203347 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb0940b0a30 0x7fb09418f370 secure :-1 s=READY pgs=2315 cs=0 l=1 rev1=1 rx=0x7fb08400b5b0 tx=0x7fb08400b950).stop 2021-02-10T11:32:43.135 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:43.134 7fb09afd1700 1 -- 172.21.2.102:0/2148203347 shutdown_connections 2021-02-10T11:32:43.136 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:43.134 7fb09afd1700 1 --2- 172.21.2.102:0/2148203347 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb09418f990 0x7fb094190680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:43.136 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:43.134 7fb09afd1700 1 --2- 172.21.2.102:0/2148203347 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb0940b0a30 0x7fb09418f370 unknown :-1 s=CLOSED pgs=2315 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:43.136 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:43.134 7fb09afd1700 1 --2- 172.21.2.102:0/2148203347 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb0780186c0 0x7fb07801abc0 unknown :-1 s=CLOSED pgs=1020 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:43.136 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:43.134 7fb09afd1700 1 --2- 172.21.2.102:0/2148203347 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb094195500 0x7fb0940b0410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:43.136 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:43.134 7fb09afd1700 1 -- 172.21.2.102:0/2148203347 >> 172.21.2.102:0/2148203347 conn(0x7fb09419f820 msgr2=0x7fb0941a5cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:43.136 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:43.134 7fb09afd1700 1 -- 172.21.2.102:0/2148203347 shutdown_connections 2021-02-10T11:32:43.136 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:43.134 7fb09afd1700 1 -- 172.21.2.102:0/2148203347 wait complete. 2021-02-10T11:32:43.143 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.3 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:32:43.575 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:43.557 7fa60c5b8700 1 Processor -- start 2021-02-10T11:32:43.576 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:43.557 7fa60c5b8700 1 -- start start 2021-02-10T11:32:43.576 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:43.557 7fa60c5b8700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa6040d1c60 0x7fa6040cef50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:43.576 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:43.557 7fa60c5b8700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fa6040cf570 0x7fa6040cf970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:43.576 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:43.557 7fa60c5b8700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6040fe1b0 con 0x7fa6040d1860 2021-02-10T11:32:43.577 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:43.557 7fa60c5b8700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6041b0ca0 con 0x7fa6040cf570 2021-02-10T11:32:43.577 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:43.557 7fa60c5b8700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa604109a50 con 0x7fa6040d1c60 2021-02-10T11:32:43.577 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:43.558 7fa60abd5700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fa6040cf570 0x7fa6040cf970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:43.577 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:43.558 7fa60abd5700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fa6040cf570 0x7fa6040cf970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:34750/0 (socket says 172.21.2.102:34750) 2021-02-10T11:32:43.577 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:43.558 7fa60abd5700 1 -- 172.21.2.102:0/102882199 learned_addr learned my addr 172.21.2.102:0/102882199 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:32:43.577 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:43.558 7fa60abd5700 1 -- 172.21.2.102:0/102882199 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa6040d1c60 msgr2=0x7fa6040cef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:32:43.578 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:43.558 7fa60abd5700 1 --2- 172.21.2.102:0/102882199 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa6040d1c60 0x7fa6040cef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:43.578 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:43.558 7fa60abd5700 1 -- 172.21.2.102:0/102882199 >> v1:172.21.2.113:6789/0 conn(0x7fa6040d1860 legacy=0x7fa6041b04b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:32:43.578 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:43.558 7fa60abd5700 1 -- 172.21.2.102:0/102882199 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa604088070 con 0x7fa6040cf570 2021-02-10T11:32:43.578 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:43.558 7fa60abd5700 1 --2- 172.21.2.102:0/102882199 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa6040cf570 0x7fa6040cf970 secure :-1 s=READY pgs=2316 cs=0 l=1 rev1=1 rx=0x7fa5fc006f00 tx=0x7fa5fc009e90).ready entity=mon.0 client_cookie=511bed0c3c6144e9 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:43.578 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:43.558 7fa6093d2700 1 -- 172.21.2.102:0/102882199 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa5fc00b300 con 0x7fa6040cf570 2021-02-10T11:32:43.579 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:43.558 7fa6093d2700 1 -- 172.21.2.102:0/102882199 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5fc00b580 con 0x7fa6040cf570 2021-02-10T11:32:43.579 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:43.558 7fa6093d2700 1 -- 172.21.2.102:0/102882199 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa5fc00b9d0 con 0x7fa6040cf570 2021-02-10T11:32:43.579 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:43.558 7fa60c5b8700 1 -- 172.21.2.102:0/102882199 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa6040cf570 msgr2=0x7fa6040cf970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:43.579 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:43.558 7fa60c5b8700 1 --2- 172.21.2.102:0/102882199 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa6040cf570 0x7fa6040cf970 secure :-1 s=READY pgs=2316 cs=0 l=1 rev1=1 rx=0x7fa5fc006f00 tx=0x7fa5fc009e90).stop 2021-02-10T11:32:43.579 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:43.559 7fa60c5b8700 1 -- 172.21.2.102:0/102882199 shutdown_connections 2021-02-10T11:32:43.579 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:43.559 7fa60c5b8700 1 --2- 172.21.2.102:0/102882199 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa6040d1c60 0x7fa6040cef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:43.580 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:43.559 7fa60c5b8700 1 --2- 172.21.2.102:0/102882199 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa6040cf570 0x7fa6040cf970 unknown :-1 s=CLOSED pgs=2316 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:43.580 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:43.559 7fa60c5b8700 1 -- 172.21.2.102:0/102882199 >> 172.21.2.102:0/102882199 conn(0x7fa6040ce410 msgr2=0x7fa6040d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:43.580 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:43.559 7fa60c5b8700 1 -- 172.21.2.102:0/102882199 shutdown_connections 2021-02-10T11:32:43.580 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:43.559 7fa60c5b8700 1 -- 172.21.2.102:0/102882199 wait complete. 2021-02-10T11:32:43.580 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:43.574 7fa60c5b8700 1 Processor -- start 2021-02-10T11:32:43.580 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:43.574 7fa60c5b8700 1 -- start start 2021-02-10T11:32:43.581 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:43.574 7fa60c5b8700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa60418f2c0 0x7fa60418f6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:43.581 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:43.574 7fa60c5b8700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa60418fce0 0x7fa6041900e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:43.581 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:43.574 7fa60c5b8700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa60419ce80 0x7fa60419d280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:43.581 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:43.574 7fa60c5b8700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa60418ed40 con 0x7fa60418fce0 2021-02-10T11:32:43.581 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:43.574 7fa60c5b8700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6040b1b00 con 0x7fa60419ce80 2021-02-10T11:32:43.581 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:43.574 7fa60c5b8700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa60419d8a0 con 0x7fa60418f2c0 2021-02-10T11:32:43.582 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:43.574 7fa609bd3700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa60418fce0 0x7fa6041900e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:43.582 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:43.574 7fa609bd3700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa60418fce0 0x7fa6041900e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:34756/0 (socket says 172.21.2.102:34756) 2021-02-10T11:32:43.582 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:43.574 7fa609bd3700 1 -- 172.21.2.102:0/2937592713 learned_addr learned my addr 172.21.2.102:0/2937592713 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:32:43.584 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:43.574 7fa609bd3700 1 -- 172.21.2.102:0/2937592713 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa60418f2c0 msgr2=0x7fa60418f6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:43.584 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:43.574 7fa609bd3700 1 --2- 172.21.2.102:0/2937592713 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa60418f2c0 0x7fa60418f6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:43.584 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:43.574 7fa609bd3700 1 -- 172.21.2.102:0/2937592713 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa60419ce80 msgr2=0x7fa60419d280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:43.585 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:43.574 7fa609bd3700 1 --2- 172.21.2.102:0/2937592713 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa60419ce80 0x7fa60419d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:43.585 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:43.574 7fa609bd3700 1 -- 172.21.2.102:0/2937592713 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa60419dac0 con 0x7fa60418fce0 2021-02-10T11:32:43.585 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:43.575 7fa609bd3700 1 --2- 172.21.2.102:0/2937592713 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa60418fce0 0x7fa6041900e0 secure :-1 s=READY pgs=2317 cs=0 l=1 rev1=1 rx=0x7fa5f4006f00 tx=0x7fa5f400b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:43.585 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:43.575 7fa5faffd700 1 -- 172.21.2.102:0/2937592713 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa5f4006350 con 0x7fa60418fce0 2021-02-10T11:32:43.585 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:43.575 7fa5faffd700 1 -- 172.21.2.102:0/2937592713 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5f400bd30 con 0x7fa60418fce0 2021-02-10T11:32:43.585 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:43.575 7fa5faffd700 1 -- 172.21.2.102:0/2937592713 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa5f40020d0 con 0x7fa60418fce0 2021-02-10T11:32:43.586 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:43.575 7fa60c5b8700 1 -- 172.21.2.102:0/2937592713 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa60419dac0 con 0x7fa60418fce0 2021-02-10T11:32:43.586 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:43.575 7fa60c5b8700 1 -- 172.21.2.102:0/2937592713 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa60419dac0 con 0x7fa60418fce0 2021-02-10T11:32:43.586 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:43.575 7fa5faffd700 1 -- 172.21.2.102:0/2937592713 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa5f4007000 con 0x7fa60418fce0 2021-02-10T11:32:43.586 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:43.576 7fa5faffd700 1 --2- 172.21.2.102:0/2937592713 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa5ec0186c0 0x7fa5ec01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:43.586 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:43.576 7fa5faffd700 1 -- 172.21.2.102:0/2937592713 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fa5f4032560 con 0x7fa60418fce0 2021-02-10T11:32:43.586 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:43.576 7fa60a3d4700 1 --2- 172.21.2.102:0/2937592713 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa5ec0186c0 0x7fa5ec01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:43.587 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:43.576 7fa60a3d4700 1 --2- 172.21.2.102:0/2937592713 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa5ec0186c0 0x7fa5ec01abc0 crc :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:43.609 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:43.607 7fa60c5b8700 1 -- 172.21.2.102:0/2937592713 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6040b1b00 con 0x7fa60418fce0 2021-02-10T11:32:43.612 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:43.610 7fa5faffd700 1 -- 172.21.2.102:0/2937592713 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fa5f402df00 con 0x7fa60418fce0 2021-02-10T11:32:43.743 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:43.742 7fa60c5b8700 1 -- 172.21.2.102:0/2937592713 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.3", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7fa6040b1b00 con 0x7fa60418fce0 2021-02-10T11:32:43.747 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:43.746 7fa5faffd700 1 -- 172.21.2.102:0/2937592713 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.3", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.3 v70) v1 ==== 184+0+0 (secure 0 0 0) 0x7fa5f4007000 con 0x7fa60418fce0 2021-02-10T11:32:43.751 INFO:teuthology.orchestra.run.gibba002.stderr:updated caps for client.3 2021-02-10T11:32:43.753 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:43.751 7fa60c5b8700 1 -- 172.21.2.102:0/2937592713 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa5ec0186c0 msgr2=0x7fa5ec01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:43.753 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:43.751 7fa60c5b8700 1 --2- 172.21.2.102:0/2937592713 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa5ec0186c0 0x7fa5ec01abc0 crc :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:43.753 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:43.751 7fa60c5b8700 1 -- 172.21.2.102:0/2937592713 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa60418fce0 msgr2=0x7fa6041900e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:43.753 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:43.751 7fa60c5b8700 1 --2- 172.21.2.102:0/2937592713 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa60418fce0 0x7fa6041900e0 secure :-1 s=READY pgs=2317 cs=0 l=1 rev1=1 rx=0x7fa5f4006f00 tx=0x7fa5f400b940).stop 2021-02-10T11:32:43.753 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:43.751 7fa60c5b8700 1 -- 172.21.2.102:0/2937592713 shutdown_connections 2021-02-10T11:32:43.753 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:43.751 7fa60c5b8700 1 --2- 172.21.2.102:0/2937592713 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa5ec0186c0 0x7fa5ec01abc0 unknown :-1 s=CLOSED pgs=1021 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:43.753 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:43.751 7fa60c5b8700 1 --2- 172.21.2.102:0/2937592713 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa60419ce80 0x7fa60419d280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:43.753 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:43.751 7fa60c5b8700 1 --2- 172.21.2.102:0/2937592713 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa60418fce0 0x7fa6041900e0 unknown :-1 s=CLOSED pgs=2317 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:43.753 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:43.751 7fa60c5b8700 1 --2- 172.21.2.102:0/2937592713 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa60418f2c0 0x7fa60418f6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:43.754 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:43.751 7fa60c5b8700 1 -- 172.21.2.102:0/2937592713 >> 172.21.2.102:0/2937592713 conn(0x7fa60419f820 msgr2=0x7fa604085d20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:43.754 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:43.751 7fa60c5b8700 1 -- 172.21.2.102:0/2937592713 shutdown_connections 2021-02-10T11:32:43.754 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:43.752 7fa60c5b8700 1 -- 172.21.2.102:0/2937592713 wait complete. 2021-02-10T11:32:43.761 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:32:44.189 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:44.171 7fcc86bc1700 1 Processor -- start 2021-02-10T11:32:44.190 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:44.171 7fcc86bc1700 1 -- start start 2021-02-10T11:32:44.190 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:44.172 7fcc86bc1700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fcc800d1870 0x7fcc801b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:44.190 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:44.172 7fcc86bc1700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fcc801b0a10 0x7fcc800cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:44.190 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:44.172 7fcc86bc1700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc800f0b10 con 0x7fcc800d1c70 2021-02-10T11:32:44.190 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:44.172 7fcc86bc1700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc800e5ed0 con 0x7fcc800d1870 2021-02-10T11:32:44.190 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:44.172 7fcc86bc1700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc800f0650 con 0x7fcc801b0a10 2021-02-10T11:32:44.190 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:44.172 7fcc849dd700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fcc800d1870 0x7fcc801b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:44.190 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:44.172 7fcc849dd700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fcc800d1870 0x7fcc801b03f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:34764/0 (socket says 172.21.2.102:34764) 2021-02-10T11:32:44.190 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:44.172 7fcc849dd700 1 -- 172.21.2.102:0/2697215912 learned_addr learned my addr 172.21.2.102:0/2697215912 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:32:44.191 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:44.173 7fcc7ffff700 1 --2- 172.21.2.102:0/2697215912 >> v2:172.21.2.113:3300/0 conn(0x7fcc801b0a10 0x7fcc800cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:44.191 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:44.173 7fcc849dd700 1 -- 172.21.2.102:0/2697215912 >> v2:172.21.2.113:3300/0 conn(0x7fcc801b0a10 msgr2=0x7fcc800cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:32:44.191 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:44.173 7fcc849dd700 1 --2- 172.21.2.102:0/2697215912 >> v2:172.21.2.113:3300/0 conn(0x7fcc801b0a10 0x7fcc800cf330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:44.191 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:44.173 7fcc849dd700 1 -- 172.21.2.102:0/2697215912 >> v1:172.21.2.113:6789/0 conn(0x7fcc800d1c70 legacy=0x7fcc800cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:32:44.191 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:44.173 7fcc849dd700 1 -- 172.21.2.102:0/2697215912 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc800f0840 con 0x7fcc800d1870 2021-02-10T11:32:44.191 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:44.173 7fcc7ffff700 1 --2- 172.21.2.102:0/2697215912 >> v2:172.21.2.113:3300/0 conn(0x7fcc801b0a10 0x7fcc800cf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:32:44.191 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:44.173 7fcc849dd700 1 --2- 172.21.2.102:0/2697215912 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcc800d1870 0x7fcc801b03f0 secure :-1 s=READY pgs=2318 cs=0 l=1 rev1=1 rx=0x7fcc68006700 tx=0x7fcc68009b80).ready entity=mon.0 client_cookie=7c7fcbc062362f18 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:44.191 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:44.173 7fcc7f7fe700 1 -- 172.21.2.102:0/2697215912 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcc6800c0c0 con 0x7fcc800d1870 2021-02-10T11:32:44.191 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:44.173 7fcc7f7fe700 1 -- 172.21.2.102:0/2697215912 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc6800c3c0 con 0x7fcc800d1870 2021-02-10T11:32:44.191 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:44.173 7fcc7f7fe700 1 -- 172.21.2.102:0/2697215912 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcc6800c8a0 con 0x7fcc800d1870 2021-02-10T11:32:44.191 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:44.173 7fcc86bc1700 1 -- 172.21.2.102:0/2697215912 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcc800d1870 msgr2=0x7fcc801b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:44.191 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:44.173 7fcc86bc1700 1 --2- 172.21.2.102:0/2697215912 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcc800d1870 0x7fcc801b03f0 secure :-1 s=READY pgs=2318 cs=0 l=1 rev1=1 rx=0x7fcc68006700 tx=0x7fcc68009b80).stop 2021-02-10T11:32:44.192 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:44.174 7fcc86bc1700 1 -- 172.21.2.102:0/2697215912 shutdown_connections 2021-02-10T11:32:44.192 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:44.174 7fcc86bc1700 1 --2- 172.21.2.102:0/2697215912 >> v2:172.21.2.113:3300/0 conn(0x7fcc801b0a10 0x7fcc800cf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:44.192 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:44.174 7fcc86bc1700 1 --2- 172.21.2.102:0/2697215912 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcc800d1870 0x7fcc801b03f0 unknown :-1 s=CLOSED pgs=2318 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:44.192 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:44.174 7fcc86bc1700 1 -- 172.21.2.102:0/2697215912 >> 172.21.2.102:0/2697215912 conn(0x7fcc800ce450 msgr2=0x7fcc800d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:44.192 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:44.174 7fcc86bc1700 1 -- 172.21.2.102:0/2697215912 shutdown_connections 2021-02-10T11:32:44.192 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:44.174 7fcc86bc1700 1 -- 172.21.2.102:0/2697215912 wait complete. 2021-02-10T11:32:44.193 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:44.189 7fcc86bc1700 1 Processor -- start 2021-02-10T11:32:44.194 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:44.189 7fcc86bc1700 1 -- start start 2021-02-10T11:32:44.194 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:44.189 7fcc86bc1700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcc801b8ad0 0x7fcc80187e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:44.194 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:44.189 7fcc86bc1700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fcc801b8ed0 0x7fcc800b1af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:44.194 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:44.189 7fcc86bc1700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcc800b2110 0x7fcc801b7a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:44.194 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:44.189 7fcc86bc1700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc80188470 con 0x7fcc800b2110 2021-02-10T11:32:44.195 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:44.189 7fcc86bc1700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc8018be10 con 0x7fcc801b8ed0 2021-02-10T11:32:44.196 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:44.189 7fcc86bc1700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc8018c030 con 0x7fcc801b8ad0 2021-02-10T11:32:44.196 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:44.189 7fcc851de700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcc800b2110 0x7fcc801b7a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:44.196 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:44.189 7fcc851de700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcc800b2110 0x7fcc801b7a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:34770/0 (socket says 172.21.2.102:34770) 2021-02-10T11:32:44.196 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:44.189 7fcc851de700 1 -- 172.21.2.102:0/2748660783 learned_addr learned my addr 172.21.2.102:0/2748660783 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:32:44.196 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:44.190 7fcc7ffff700 1 --2- 172.21.2.102:0/2748660783 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fcc801b8ed0 0x7fcc800b1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:44.196 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:44.190 7fcc849dd700 1 --2- 172.21.2.102:0/2748660783 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcc801b8ad0 0x7fcc80187e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:44.196 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:44.190 7fcc851de700 1 -- 172.21.2.102:0/2748660783 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcc801b8ad0 msgr2=0x7fcc80187e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:44.196 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:44.190 7fcc851de700 1 --2- 172.21.2.102:0/2748660783 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcc801b8ad0 0x7fcc80187e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:44.196 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:44.190 7fcc851de700 1 -- 172.21.2.102:0/2748660783 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fcc801b8ed0 msgr2=0x7fcc800b1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:44.196 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:44.190 7fcc851de700 1 --2- 172.21.2.102:0/2748660783 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fcc801b8ed0 0x7fcc800b1af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:44.196 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:44.190 7fcc851de700 1 -- 172.21.2.102:0/2748660783 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc800b2510 con 0x7fcc800b2110 2021-02-10T11:32:44.196 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:44.190 7fcc849dd700 1 --2- 172.21.2.102:0/2748660783 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcc801b8ad0 0x7fcc80187e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:32:44.197 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:44.190 7fcc851de700 1 --2- 172.21.2.102:0/2748660783 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcc800b2110 0x7fcc801b7a00 secure :-1 s=READY pgs=2319 cs=0 l=1 rev1=1 rx=0x7fcc78006ed0 tx=0x7fcc7800b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:44.197 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:44.190 7fcc7d7fa700 1 -- 172.21.2.102:0/2748660783 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcc7800bd50 con 0x7fcc800b2110 2021-02-10T11:32:44.197 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:44.190 7fcc7d7fa700 1 -- 172.21.2.102:0/2748660783 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc78006020 con 0x7fcc800b2110 2021-02-10T11:32:44.197 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:44.190 7fcc7d7fa700 1 -- 172.21.2.102:0/2748660783 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcc780054b0 con 0x7fcc800b2110 2021-02-10T11:32:44.197 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:44.190 7fcc86bc1700 1 -- 172.21.2.102:0/2748660783 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc800b2510 con 0x7fcc800b2110 2021-02-10T11:32:44.197 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:44.190 7fcc86bc1700 1 -- 172.21.2.102:0/2748660783 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc800b2510 con 0x7fcc800b2110 2021-02-10T11:32:44.197 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:44.191 7fcc7d7fa700 1 -- 172.21.2.102:0/2748660783 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fcc78006200 con 0x7fcc800b2110 2021-02-10T11:32:44.197 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:44.191 7fcc7d7fa700 1 --2- 172.21.2.102:0/2748660783 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcc640186c0 0x7fcc6401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:44.197 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:44.191 7fcc7d7fa700 1 -- 172.21.2.102:0/2748660783 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fcc780327e0 con 0x7fcc800b2110 2021-02-10T11:32:44.197 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:44.191 7fcc849dd700 1 --2- 172.21.2.102:0/2748660783 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcc640186c0 0x7fcc6401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:44.197 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:44.191 7fcc849dd700 1 --2- 172.21.2.102:0/2748660783 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcc640186c0 0x7fcc6401abc0 crc :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:44.223 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:44.221 7fcc86bc1700 1 -- 172.21.2.102:0/2748660783 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc800ebc00 con 0x7fcc800b2110 2021-02-10T11:32:44.225 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:44.224 7fcc7d7fa700 1 -- 172.21.2.102:0/2748660783 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fcc78006230 con 0x7fcc800b2110 2021-02-10T11:32:44.355 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:44.354 7fcc86bc1700 1 -- 172.21.2.102:0/2748660783 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcc800ebc00 con 0x7fcc800b2110 2021-02-10T11:32:44.355 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:44.354 7fcc7d7fa700 1 -- 172.21.2.102:0/2748660783 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+2101 (secure 0 0 0) 0x7fcc78006000 con 0x7fcc800b2110 2021-02-10T11:32:44.356 INFO:teuthology.orchestra.run.gibba002.stderr:dumped fsmap epoch 99 2021-02-10T11:32:44.356 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:32:44.356 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":99,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10737,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.102:6835/4014802408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6834","nonce":4014802408},{"type":"v1","addr":"172.21.2.102:6835","nonce":4014802408}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":95}],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:32:38.017410","modified":"2021-02-10 11:32:40.027261","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7693},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7693":{"gid":7693,"name":"b","rank":0,"incarnation":96,"state":"up:active","state_seq":3,"addr":"172.21.2.113:6833/238020319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6832","nonce":238020319},{"type":"v1","addr":"172.21.2.113:6833","nonce":238020319}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[25],"metadata_pool":24,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":12}]} 2021-02-10T11:32:44.357 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:44.356 7fcc86bc1700 1 -- 172.21.2.102:0/2748660783 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcc640186c0 msgr2=0x7fcc6401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:44.357 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:44.356 7fcc86bc1700 1 --2- 172.21.2.102:0/2748660783 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcc640186c0 0x7fcc6401abc0 crc :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:44.358 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:44.356 7fcc86bc1700 1 -- 172.21.2.102:0/2748660783 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcc800b2110 msgr2=0x7fcc801b7a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:44.358 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:44.356 7fcc86bc1700 1 --2- 172.21.2.102:0/2748660783 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcc800b2110 0x7fcc801b7a00 secure :-1 s=READY pgs=2319 cs=0 l=1 rev1=1 rx=0x7fcc78006ed0 tx=0x7fcc7800b960).stop 2021-02-10T11:32:44.358 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:44.356 7fcc86bc1700 1 -- 172.21.2.102:0/2748660783 shutdown_connections 2021-02-10T11:32:44.358 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:44.356 7fcc86bc1700 1 --2- 172.21.2.102:0/2748660783 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcc800b2110 0x7fcc801b7a00 unknown :-1 s=CLOSED pgs=2319 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:44.358 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:44.356 7fcc86bc1700 1 --2- 172.21.2.102:0/2748660783 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcc640186c0 0x7fcc6401abc0 unknown :-1 s=CLOSED pgs=1022 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:44.359 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:44.356 7fcc86bc1700 1 --2- 172.21.2.102:0/2748660783 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fcc801b8ed0 0x7fcc800b1af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:44.359 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:44.356 7fcc86bc1700 1 --2- 172.21.2.102:0/2748660783 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcc801b8ad0 0x7fcc80187e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:44.359 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:44.356 7fcc86bc1700 1 -- 172.21.2.102:0/2748660783 >> 172.21.2.102:0/2748660783 conn(0x7fcc800ed810 msgr2=0x7fcc80094980 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:44.359 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:44.356 7fcc86bc1700 1 -- 172.21.2.102:0/2748660783 shutdown_connections 2021-02-10T11:32:44.359 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:44.357 7fcc86bc1700 1 -- 172.21.2.102:0/2748660783 wait complete. 2021-02-10T11:32:44.568 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 99, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:32:38.017410', 'modified': '2021-02-10 11:32:40.027261', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7693}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7693': {'gid': 7693, 'name': 'b', 'rank': 0, 'incarnation': 96, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.113:6833/238020319', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.113:6832', 'nonce': 238020319}, {'type': 'v1', 'addr': '172.21.2.113:6833', 'nonce': 238020319}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [25], 'metadata_pool': 24, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:32:44.568 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:32:44.568 DEBUG:teuthology.orchestra.run.gibba013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:32:44.968 INFO:teuthology.orchestra.run.gibba013.stdout:{ 2021-02-10T11:32:44.968 INFO:teuthology.orchestra.run.gibba013.stdout: "cluster_fsid": "68db30ea-dbce-4aef-a6df-f01c6fea0513", 2021-02-10T11:32:44.968 INFO:teuthology.orchestra.run.gibba013.stdout: "whoami": 0, 2021-02-10T11:32:44.968 INFO:teuthology.orchestra.run.gibba013.stdout: "id": 7693, 2021-02-10T11:32:44.968 INFO:teuthology.orchestra.run.gibba013.stdout: "want_state": "up:active", 2021-02-10T11:32:44.968 INFO:teuthology.orchestra.run.gibba013.stdout: "state": "up:active", 2021-02-10T11:32:44.968 INFO:teuthology.orchestra.run.gibba013.stdout: "rank_uptime": 6.9474236380000001, 2021-02-10T11:32:44.968 INFO:teuthology.orchestra.run.gibba013.stdout: "mdsmap_epoch": 99, 2021-02-10T11:32:44.969 INFO:teuthology.orchestra.run.gibba013.stdout: "osdmap_epoch": 120, 2021-02-10T11:32:44.969 INFO:teuthology.orchestra.run.gibba013.stdout: "osdmap_epoch_barrier": 120, 2021-02-10T11:32:44.969 INFO:teuthology.orchestra.run.gibba013.stdout: "uptime": 12.596087601000001 2021-02-10T11:32:44.969 INFO:teuthology.orchestra.run.gibba013.stdout:} 2021-02-10T11:32:44.972 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "68db30ea-dbce-4aef-a6df-f01c6fea0513", "whoami": 0, "id": 7693, "want_state": "up:active", "state": "up:active", "rank_uptime": 6.9474236380000001, "mdsmap_epoch": 99, "osdmap_epoch": 120, "osdmap_epoch_barrier": 120, "uptime": 12.596087601000001 } 2021-02-10T11:32:44.973 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:32:45.403 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:45.384 7f4a98166700 1 Processor -- start 2021-02-10T11:32:45.403 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:45.384 7f4a98166700 1 -- start start 2021-02-10T11:32:45.403 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:45.384 7f4a98166700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f4a900d1870 0x7f4a901b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:45.403 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:45.384 7f4a98166700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4a901b0a10 0x7f4a900cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:45.403 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:45.384 7f4a98166700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a900f0b10 con 0x7f4a900d1c70 2021-02-10T11:32:45.404 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:45.384 7f4a98166700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a900e5ed0 con 0x7f4a900d1870 2021-02-10T11:32:45.404 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:45.384 7f4a98166700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a900f0650 con 0x7f4a901b0a10 2021-02-10T11:32:45.404 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:45.385 7f4a96783700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f4a900d1c70 0x7f4a900cf950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:55230/0 (socket says 172.21.2.102:55230) 2021-02-10T11:32:45.404 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:45.385 7f4a96783700 1 -- 172.21.2.102:0/38007677 learned_addr learned my addr 172.21.2.102:0/38007677 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:32:45.404 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:45.385 7f4a94f80700 1 -- 172.21.2.102:0/38007677 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3583542973 0 0) 0x7f4a88001500 con 0x7f4a900d1c70 2021-02-10T11:32:45.404 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:45.385 7f4a94f80700 1 -- 172.21.2.102:0/38007677 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a7c002fe0 con 0x7f4a900d1c70 2021-02-10T11:32:45.404 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:45.385 7f4a95f82700 1 --2- 172.21.2.102:0/38007677 >> v2:172.21.2.113:3300/0 conn(0x7f4a900d1870 0x7f4a901b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:45.404 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:45.385 7f4a95781700 1 --2- 172.21.2.102:0/38007677 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4a901b0a10 0x7f4a900cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:45.404 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:45.385 7f4a94f80700 1 -- 172.21.2.102:0/38007677 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2173429282 0 0) 0x7f4a88001500 con 0x7f4a900d1c70 2021-02-10T11:32:45.404 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:45.385 7f4a94f80700 1 -- 172.21.2.102:0/38007677 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4a901b0a10 msgr2=0x7f4a900cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:32:45.404 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:45.385 7f4a94f80700 1 --2- 172.21.2.102:0/38007677 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4a901b0a10 0x7f4a900cf330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:45.405 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:45.385 7f4a94f80700 1 -- 172.21.2.102:0/38007677 >> v2:172.21.2.113:3300/0 conn(0x7f4a900d1870 msgr2=0x7f4a901b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:32:45.405 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:45.385 7f4a94f80700 1 --2- 172.21.2.102:0/38007677 >> v2:172.21.2.113:3300/0 conn(0x7f4a900d1870 0x7f4a901b03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:45.405 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:45.385 7f4a94f80700 1 -- 172.21.2.102:0/38007677 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a900f0870 con 0x7f4a900d1c70 2021-02-10T11:32:45.405 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:45.385 7f4a94f80700 1 -- 172.21.2.102:0/38007677 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7f4a88001a00 con 0x7f4a900d1c70 2021-02-10T11:32:45.405 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:45.385 7f4a94f80700 1 -- 172.21.2.102:0/38007677 >> v1:172.21.2.102:6789/0 conn(0x7f4a900d1c70 legacy=0x7f4a900cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:45.405 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:45.385 7f4a94f80700 1 --2- 172.21.2.102:0/38007677 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4a7c003b90 0x7f4a7c005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:45.405 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:45.385 7f4a94f80700 1 --2- 172.21.2.102:0/38007677 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4a7c006820 0x7f4a7c008d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:45.405 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:45.385 7f4a94f80700 1 --2- 172.21.2.102:0/38007677 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4a7c0095a0 0x7f4a7c00ba50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:45.405 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:45.385 7f4a94f80700 1 -- 172.21.2.102:0/38007677 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a7c00c300 con 0x7f4a7c006820 2021-02-10T11:32:45.405 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:45.385 7f4a94f80700 1 -- 172.21.2.102:0/38007677 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a7c0504f0 con 0x7f4a7c0095a0 2021-02-10T11:32:45.405 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:45.385 7f4a94f80700 1 -- 172.21.2.102:0/38007677 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a7c050710 con 0x7f4a7c003b90 2021-02-10T11:32:45.405 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:45.385 7f4a95781700 1 --2- 172.21.2.102:0/38007677 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4a7c006820 0x7f4a7c008d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:45.405 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:45.386 7f4a96783700 1 --2- 172.21.2.102:0/38007677 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4a7c0095a0 0x7f4a7c00ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:45.406 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:45.386 7f4a95f82700 1 --2- 172.21.2.102:0/38007677 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4a7c003b90 0x7f4a7c005fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:45.406 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:45.386 7f4a95781700 1 -- 172.21.2.102:0/38007677 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4a7c003b90 msgr2=0x7f4a7c005fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:32:45.406 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:45.386 7f4a95781700 1 --2- 172.21.2.102:0/38007677 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4a7c003b90 0x7f4a7c005fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:45.406 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:45.386 7f4a95781700 1 -- 172.21.2.102:0/38007677 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4a7c0095a0 msgr2=0x7f4a7c00ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:32:45.406 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:45.386 7f4a95781700 1 --2- 172.21.2.102:0/38007677 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4a7c0095a0 0x7f4a7c00ba50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:45.406 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:45.386 7f4a95781700 1 -- 172.21.2.102:0/38007677 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a7c0509b0 con 0x7f4a7c006820 2021-02-10T11:32:45.406 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:45.386 7f4a95f82700 1 --2- 172.21.2.102:0/38007677 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4a7c003b90 0x7f4a7c005fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:32:45.406 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:45.386 7f4a96783700 1 --2- 172.21.2.102:0/38007677 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4a7c0095a0 0x7f4a7c00ba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:32:45.406 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:45.386 7f4a95781700 1 --2- 172.21.2.102:0/38007677 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4a7c006820 0x7f4a7c008d00 secure :-1 s=READY pgs=2321 cs=0 l=1 rev1=1 rx=0x7f4a8c006a90 tx=0x7f4a8c009a90).ready entity=mon.0 client_cookie=3169abc65b72f0dd server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:45.406 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:45.386 7f4a94f80700 1 -- 172.21.2.102:0/38007677 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4a8c00c0c0 con 0x7f4a7c006820 2021-02-10T11:32:45.406 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:45.386 7f4a94f80700 1 -- 172.21.2.102:0/38007677 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a8c00c380 con 0x7f4a7c006820 2021-02-10T11:32:45.407 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:45.386 7f4a94f80700 1 -- 172.21.2.102:0/38007677 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4a8c00c8a0 con 0x7f4a7c006820 2021-02-10T11:32:45.407 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:45.386 7f4a98166700 1 -- 172.21.2.102:0/38007677 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4a7c006820 msgr2=0x7f4a7c008d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:45.407 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:45.386 7f4a98166700 1 --2- 172.21.2.102:0/38007677 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4a7c006820 0x7f4a7c008d00 secure :-1 s=READY pgs=2321 cs=0 l=1 rev1=1 rx=0x7f4a8c006a90 tx=0x7f4a8c009a90).stop 2021-02-10T11:32:45.407 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:45.386 7f4a96783700 1 -- 172.21.2.102:0/38007677 reap_dead start 2021-02-10T11:32:45.407 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:45.386 7f4a98166700 1 -- 172.21.2.102:0/38007677 shutdown_connections 2021-02-10T11:32:45.407 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:45.386 7f4a98166700 1 -- 172.21.2.102:0/38007677 >> 172.21.2.102:0/38007677 conn(0x7f4a900ce450 msgr2=0x7f4a900d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:45.407 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:45.387 7f4a98166700 1 -- 172.21.2.102:0/38007677 shutdown_connections 2021-02-10T11:32:45.407 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:45.387 7f4a98166700 1 -- 172.21.2.102:0/38007677 wait complete. 2021-02-10T11:32:45.407 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:45.401 7f4a98166700 1 Processor -- start 2021-02-10T11:32:45.407 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:45.401 7f4a98166700 1 -- start start 2021-02-10T11:32:45.407 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:45.402 7f4a98166700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4a901942b0 0x7f4a900b3550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:45.407 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:45.402 7f4a98166700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4a90195070 0x7f4a90195470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:45.408 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:45.402 7f4a98166700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4a900bb8e0 0x7f4a900bbce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:45.408 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:45.402 7f4a98166700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a901946b0 con 0x7f4a900bb8e0 2021-02-10T11:32:45.408 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:45.402 7f4a98166700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a900b3b70 con 0x7f4a90195070 2021-02-10T11:32:45.408 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:45.402 7f4a98166700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a900b3d90 con 0x7f4a901942b0 2021-02-10T11:32:45.409 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:45.402 7f4a96783700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4a900bb8e0 0x7f4a900bbce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:45.409 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:45.402 7f4a96783700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4a900bb8e0 0x7f4a900bbce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:34784/0 (socket says 172.21.2.102:34784) 2021-02-10T11:32:45.409 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:45.402 7f4a96783700 1 -- 172.21.2.102:0/2630714407 learned_addr learned my addr 172.21.2.102:0/2630714407 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:32:45.409 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:45.402 7f4a96783700 1 -- 172.21.2.102:0/2630714407 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4a901942b0 msgr2=0x7f4a900b3550 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:32:45.410 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:45.402 7f4a96783700 1 --2- 172.21.2.102:0/2630714407 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4a901942b0 0x7f4a900b3550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:45.410 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:45.402 7f4a96783700 1 -- 172.21.2.102:0/2630714407 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4a90195070 msgr2=0x7f4a90195470 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:32:45.410 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:45.402 7f4a96783700 1 --2- 172.21.2.102:0/2630714407 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4a90195070 0x7f4a90195470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:45.410 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:45.402 7f4a96783700 1 -- 172.21.2.102:0/2630714407 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a901a49e0 con 0x7f4a900bb8e0 2021-02-10T11:32:45.410 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:45.402 7f4a96783700 1 --2- 172.21.2.102:0/2630714407 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4a900bb8e0 0x7f4a900bbce0 secure :-1 s=READY pgs=2322 cs=0 l=1 rev1=1 rx=0x7f4a8800a8f0 tx=0x7f4a8800ac90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:45.410 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:45.402 7f4a827fc700 1 -- 172.21.2.102:0/2630714407 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4a880070d0 con 0x7f4a900bb8e0 2021-02-10T11:32:45.410 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:45.403 7f4a827fc700 1 -- 172.21.2.102:0/2630714407 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a88007410 con 0x7f4a900bb8e0 2021-02-10T11:32:45.410 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:45.403 7f4a98166700 1 -- 172.21.2.102:0/2630714407 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a901a49e0 con 0x7f4a900bb8e0 2021-02-10T11:32:45.410 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:45.403 7f4a827fc700 1 -- 172.21.2.102:0/2630714407 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4a880049a0 con 0x7f4a900bb8e0 2021-02-10T11:32:45.410 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:45.403 7f4a98166700 1 -- 172.21.2.102:0/2630714407 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a901a49e0 con 0x7f4a900bb8e0 2021-02-10T11:32:45.411 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:45.403 7f4a827fc700 1 -- 172.21.2.102:0/2630714407 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4a88000d10 con 0x7f4a900bb8e0 2021-02-10T11:32:45.411 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:45.403 7f4a827fc700 1 --2- 172.21.2.102:0/2630714407 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4a78018730 0x7f4a7801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:45.411 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:45.403 7f4a827fc700 1 -- 172.21.2.102:0/2630714407 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f4a880329d0 con 0x7f4a900bb8e0 2021-02-10T11:32:45.411 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:45.403 7f4a95f82700 1 --2- 172.21.2.102:0/2630714407 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4a78018730 0x7f4a7801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:45.411 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:45.404 7f4a95f82700 1 --2- 172.21.2.102:0/2630714407 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4a78018730 0x7f4a7801ac10 crc :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:45.436 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:45.434 7f4a98166700 1 -- 172.21.2.102:0/2630714407 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a901a49e0 con 0x7f4a900bb8e0 2021-02-10T11:32:45.438 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:45.437 7f4a827fc700 1 -- 172.21.2.102:0/2630714407 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f4a88007030 con 0x7f4a900bb8e0 2021-02-10T11:32:45.564 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:45.562 7f4a98166700 1 -- 172.21.2.102:0/2630714407 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4a901a49e0 con 0x7f4a900bb8e0 2021-02-10T11:32:45.564 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:45.563 7f4a827fc700 1 -- 172.21.2.102:0/2630714407 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v120) v1 ==== 74+0+14007 (secure 0 0 0) 0x7f4a88000d10 con 0x7f4a900bb8e0 2021-02-10T11:32:45.565 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:32:45.565 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":120,"fsid":"68db30ea-dbce-4aef-a6df-f01c6fea0513","created":"2021-02-10 11:03:06.331557","modified":"2021-02-10 11:32:39.019652","last_up_change":"2021-02-10 11:03:32.690562","last_in_change":"2021-02-10 11:03:32.690562","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":25,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:03:37.322519","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":24,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:32:35.986688","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"119","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":25,"pool_name":"cephfs_data","create_time":"2021-02-10 11:32:37.011649","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"120","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"be691b4d-a039-408a-af80-9d53f02b6f1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6818","nonce":36088},{"type":"v1","addr":"172.21.2.102:6820","nonce":36088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6822","nonce":36088},{"type":"v1","addr":"172.21.2.102:6824","nonce":36088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6830","nonce":36088},{"type":"v1","addr":"172.21.2.102:6832","nonce":36088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6826","nonce":36088},{"type":"v1","addr":"172.21.2.102:6828","nonce":36088}]},"public_addr":"172.21.2.102:6820/36088","cluster_addr":"172.21.2.102:6824/36088","heartbeat_back_addr":"172.21.2.102:6832/36088","heartbeat_front_addr":"172.21.2.102:6828/36088","state":["exists","up"]},{"osd":1,"uuid":"64e3ccc8-9c69-4d7c-865e-af8e83822635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6819","nonce":36087},{"type":"v1","addr":"172.21.2.102:6821","nonce":36087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6823","nonce":36087},{"type":"v1","addr":"172.21.2.102:6825","nonce":36087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6831","nonce":36087},{"type":"v1","addr":"172.21.2.102:6833","nonce":36087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6827","nonce":36087},{"type":"v1","addr":"172.21.2.102:6829","nonce":36087}]},"public_addr":"172.21.2.102:6821/36087","cluster_addr":"172.21.2.102:6825/36087","heartbeat_back_addr":"172.21.2.102:6833/36087","heartbeat_front_addr":"172.21.2.102:6829/36087","state":["exists","up"]},{"osd":2,"uuid":"153700af-747f-481f-bdf4-d6072d3a842f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6810","nonce":36089},{"type":"v1","addr":"172.21.2.102:6811","nonce":36089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6812","nonce":36089},{"type":"v1","addr":"172.21.2.102:6813","nonce":36089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6816","nonce":36089},{"type":"v1","addr":"172.21.2.102:6817","nonce":36089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6814","nonce":36089},{"type":"v1","addr":"172.21.2.102:6815","nonce":36089}]},"public_addr":"172.21.2.102:6811/36089","cluster_addr":"172.21.2.102:6813/36089","heartbeat_back_addr":"172.21.2.102:6817/36089","heartbeat_front_addr":"172.21.2.102:6815/36089","state":["exists","up"]},{"osd":3,"uuid":"cbc678f4-83a9-45e6-8830-2f7c232b1ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6802","nonce":36090},{"type":"v1","addr":"172.21.2.102:6803","nonce":36090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6804","nonce":36090},{"type":"v1","addr":"172.21.2.102:6805","nonce":36090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6808","nonce":36090},{"type":"v1","addr":"172.21.2.102:6809","nonce":36090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6806","nonce":36090},{"type":"v1","addr":"172.21.2.102:6807","nonce":36090}]},"public_addr":"172.21.2.102:6803/36090","cluster_addr":"172.21.2.102:6805/36090","heartbeat_back_addr":"172.21.2.102:6809/36090","heartbeat_front_addr":"172.21.2.102:6807/36090","state":["exists","up"]},{"osd":4,"uuid":"c01e4008-b258-4f10-afd2-9e7f37214351","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6808","nonce":36368},{"type":"v1","addr":"172.21.2.113:6811","nonce":36368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6814","nonce":36368},{"type":"v1","addr":"172.21.2.113:6818","nonce":36368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6826","nonce":36368},{"type":"v1","addr":"172.21.2.113:6828","nonce":36368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6820","nonce":36368},{"type":"v1","addr":"172.21.2.113:6824","nonce":36368}]},"public_addr":"172.21.2.113:6811/36368","cluster_addr":"172.21.2.113:6818/36368","heartbeat_back_addr":"172.21.2.113:6828/36368","heartbeat_front_addr":"172.21.2.113:6824/36368","state":["exists","up"]},{"osd":5,"uuid":"551f6007-daf8-4a98-8ee0-c2b16bcccb5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6802","nonce":36366},{"type":"v1","addr":"172.21.2.113:6805","nonce":36366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6807","nonce":36366},{"type":"v1","addr":"172.21.2.113:6810","nonce":36366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6819","nonce":36366},{"type":"v1","addr":"172.21.2.113:6822","nonce":36366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6812","nonce":36366},{"type":"v1","addr":"172.21.2.113:6815","nonce":36366}]},"public_addr":"172.21.2.113:6805/36366","cluster_addr":"172.21.2.113:6810/36366","heartbeat_back_addr":"172.21.2.113:6822/36366","heartbeat_front_addr":"172.21.2.113:6815/36366","state":["exists","up"]},{"osd":6,"uuid":"bf67c040-0e50-48be-b102-c3943bc9ebef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6800","nonce":36367},{"type":"v1","addr":"172.21.2.113:6801","nonce":36367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6803","nonce":36367},{"type":"v1","addr":"172.21.2.113:6804","nonce":36367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6813","nonce":36367},{"type":"v1","addr":"172.21.2.113:6816","nonce":36367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6806","nonce":36367},{"type":"v1","addr":"172.21.2.113:6809","nonce":36367}]},"public_addr":"172.21.2.113:6801/36367","cluster_addr":"172.21.2.113:6804/36367","heartbeat_back_addr":"172.21.2.113:6816/36367","heartbeat_front_addr":"172.21.2.113:6809/36367","state":["exists","up"]},{"osd":7,"uuid":"f5461f80-018d-428f-be89-2eb2bcbe7bb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6817","nonce":36365},{"type":"v1","addr":"172.21.2.113:6821","nonce":36365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6823","nonce":36365},{"type":"v1","addr":"172.21.2.113:6825","nonce":36365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6830","nonce":36365},{"type":"v1","addr":"172.21.2.113:6831","nonce":36365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6827","nonce":36365},{"type":"v1","addr":"172.21.2.113:6829","nonce":36365}]},"public_addr":"172.21.2.113:6821/36365","cluster_addr":"172.21.2.113:6825/36365","heartbeat_back_addr":"172.21.2.113:6831/36365","heartbeat_front_addr":"172.21.2.113:6829/36365","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:32:45.566 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:45.565 7f4a98166700 1 -- 172.21.2.102:0/2630714407 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4a78018730 msgr2=0x7f4a7801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:45.567 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:45.565 7f4a98166700 1 --2- 172.21.2.102:0/2630714407 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4a78018730 0x7f4a7801ac10 crc :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:45.567 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:45.565 7f4a98166700 1 -- 172.21.2.102:0/2630714407 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4a900bb8e0 msgr2=0x7f4a900bbce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:45.567 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:45.565 7f4a98166700 1 --2- 172.21.2.102:0/2630714407 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4a900bb8e0 0x7f4a900bbce0 secure :-1 s=READY pgs=2322 cs=0 l=1 rev1=1 rx=0x7f4a8800a8f0 tx=0x7f4a8800ac90).stop 2021-02-10T11:32:45.567 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:45.565 7f4a98166700 1 -- 172.21.2.102:0/2630714407 shutdown_connections 2021-02-10T11:32:45.567 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:45.565 7f4a98166700 1 --2- 172.21.2.102:0/2630714407 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4a900bb8e0 0x7f4a900bbce0 unknown :-1 s=CLOSED pgs=2322 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:45.567 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:45.565 7f4a98166700 1 --2- 172.21.2.102:0/2630714407 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4a78018730 0x7f4a7801ac10 unknown :-1 s=CLOSED pgs=1023 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:45.567 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:45.565 7f4a98166700 1 --2- 172.21.2.102:0/2630714407 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4a90195070 0x7f4a90195470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:45.567 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:45.565 7f4a98166700 1 --2- 172.21.2.102:0/2630714407 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4a901942b0 0x7f4a900b3550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:45.567 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:45.565 7f4a98166700 1 -- 172.21.2.102:0/2630714407 >> 172.21.2.102:0/2630714407 conn(0x7f4a900b3040 msgr2=0x7f4a900c3250 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:45.567 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:45.565 7f4a98166700 1 -- 172.21.2.102:0/2630714407 shutdown_connections 2021-02-10T11:32:45.567 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:45.565 7f4a98166700 1 -- 172.21.2.102:0/2630714407 wait complete. 2021-02-10T11:32:45.575 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:03:37.322519', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 24, 'pool_name': 'cephfs_metadata', 'create_time': '2021-02-10 11:32:35.986688', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '119', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 25, 'pool_name': 'cephfs_data', 'create_time': '2021-02-10 11:32:37.011649', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '120', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-02-10T11:32:45.576 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:32:46.006 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:45.988 7f5b4b9d8700 1 Processor -- start 2021-02-10T11:32:46.007 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:45.988 7f5b4b9d8700 1 -- start start 2021-02-10T11:32:46.007 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:45.988 7f5b4b9d8700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f5b4410afe0 0x7f5b440cf740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:46.007 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:45.988 7f5b4b9d8700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b440fe1b0 con 0x7f5b440d1860 2021-02-10T11:32:46.007 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:45.988 7f5b4b9d8700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b441b0ca0 con 0x7f5b440d1c60 2021-02-10T11:32:46.007 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:45.988 7f5b4b9d8700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b44109a50 con 0x7f5b4410afe0 2021-02-10T11:32:46.007 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:45.989 7f5b497f4700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f5b440d1860 0x7f5b441b04b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:55246/0 (socket says 172.21.2.102:55246) 2021-02-10T11:32:46.007 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:45.989 7f5b497f4700 1 -- 172.21.2.102:0/2231730583 learned_addr learned my addr 172.21.2.102:0/2231730583 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:32:46.007 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:45.989 7f5b49ff5700 1 --2- 172.21.2.102:0/2231730583 >> v2:172.21.2.113:3300/0 conn(0x7f5b4410afe0 0x7f5b440cf740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:46.007 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:45.989 7f5b37fff700 1 -- 172.21.2.102:0/2231730583 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 379459675 0 0) 0x7f5b400010b0 con 0x7f5b440d1860 2021-02-10T11:32:46.007 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:45.989 7f5b37fff700 1 -- 172.21.2.102:0/2231730583 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b30002fe0 con 0x7f5b440d1860 2021-02-10T11:32:46.008 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:45.989 7f5b49ff5700 1 -- 172.21.2.102:0/2231730583 >> v1:172.21.2.113:6789/0 conn(0x7f5b440d1c60 legacy=0x7f5b440cef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:46.008 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:45.989 7f5b49ff5700 1 -- 172.21.2.102:0/2231730583 >> v1:172.21.2.102:6789/0 conn(0x7f5b440d1860 legacy=0x7f5b441b04b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:46.008 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:45.989 7f5b49ff5700 1 -- 172.21.2.102:0/2231730583 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b4410b3e0 con 0x7f5b4410afe0 2021-02-10T11:32:46.008 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:45.990 7f5b49ff5700 1 --2- 172.21.2.102:0/2231730583 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5b4410afe0 0x7f5b440cf740 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 rx=0x7f5b3c006e50 tx=0x7f5b3c009e70).ready entity=mon.1 client_cookie=5e57cdc233af34b server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:46.008 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:45.990 7f5b37fff700 1 -- 172.21.2.102:0/2231730583 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5b3c006a20 con 0x7f5b4410afe0 2021-02-10T11:32:46.008 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:45.990 7f5b37fff700 1 -- 172.21.2.102:0/2231730583 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b3c006790 con 0x7f5b4410afe0 2021-02-10T11:32:46.008 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:45.990 7f5b37fff700 1 -- 172.21.2.102:0/2231730583 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5b3c00b700 con 0x7f5b4410afe0 2021-02-10T11:32:46.008 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:45.990 7f5b4b9d8700 1 -- 172.21.2.102:0/2231730583 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5b4410afe0 msgr2=0x7f5b440cf740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:46.008 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:45.990 7f5b4b9d8700 1 --2- 172.21.2.102:0/2231730583 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5b4410afe0 0x7f5b440cf740 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 rx=0x7f5b3c006e50 tx=0x7f5b3c009e70).stop 2021-02-10T11:32:46.008 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:45.990 7f5b4b9d8700 1 -- 172.21.2.102:0/2231730583 shutdown_connections 2021-02-10T11:32:46.008 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:45.990 7f5b4b9d8700 1 --2- 172.21.2.102:0/2231730583 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5b4410afe0 0x7f5b440cf740 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:46.008 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:45.990 7f5b4b9d8700 1 -- 172.21.2.102:0/2231730583 >> 172.21.2.102:0/2231730583 conn(0x7f5b440ce410 msgr2=0x7f5b440d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:46.008 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:45.991 7f5b4b9d8700 1 -- 172.21.2.102:0/2231730583 shutdown_connections 2021-02-10T11:32:46.008 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:45.991 7f5b4b9d8700 1 -- 172.21.2.102:0/2231730583 wait complete. 2021-02-10T11:32:46.009 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:46.005 7f5b4b9d8700 1 Processor -- start 2021-02-10T11:32:46.009 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:46.005 7f5b4b9d8700 1 -- start start 2021-02-10T11:32:46.010 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:46.006 7f5b4b9d8700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5b441a05e0 0x7f5b4418b970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:46.010 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:46.006 7f5b4b9d8700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5b440b8780 0x7f5b440bfa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:46.010 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:46.006 7f5b4b9d8700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5b440c0030 0x7f5b44189e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:46.010 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:46.006 7f5b4b9d8700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b440b82a0 con 0x7f5b441a05e0 2021-02-10T11:32:46.010 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:46.006 7f5b4b9d8700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b440c0430 con 0x7f5b440b8780 2021-02-10T11:32:46.010 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:46.006 7f5b4b9d8700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b440c0650 con 0x7f5b440c0030 2021-02-10T11:32:46.010 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:46.006 7f5b497f4700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5b441a05e0 0x7f5b4418b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:46.010 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:46.006 7f5b497f4700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5b441a05e0 0x7f5b4418b970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:34796/0 (socket says 172.21.2.102:34796) 2021-02-10T11:32:46.011 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:46.006 7f5b497f4700 1 -- 172.21.2.102:0/1902980847 learned_addr learned my addr 172.21.2.102:0/1902980847 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:32:46.011 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:46.006 7f5b497f4700 1 -- 172.21.2.102:0/1902980847 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5b440c0030 msgr2=0x7f5b44189e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:46.011 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:46.006 7f5b497f4700 1 --2- 172.21.2.102:0/1902980847 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5b440c0030 0x7f5b44189e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:46.011 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:46.006 7f5b497f4700 1 -- 172.21.2.102:0/1902980847 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5b440b8780 msgr2=0x7f5b440bfa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:46.011 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:46.006 7f5b497f4700 1 --2- 172.21.2.102:0/1902980847 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5b440b8780 0x7f5b440bfa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:46.011 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:46.006 7f5b497f4700 1 -- 172.21.2.102:0/1902980847 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b4418a490 con 0x7f5b441a05e0 2021-02-10T11:32:46.011 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:46.006 7f5b497f4700 1 --2- 172.21.2.102:0/1902980847 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5b441a05e0 0x7f5b4418b970 secure :-1 s=READY pgs=2324 cs=0 l=1 rev1=1 rx=0x7f5b40001c60 tx=0x7f5b4000a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:46.012 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:46.007 7f5b35ffb700 1 -- 172.21.2.102:0/1902980847 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5b4000acc0 con 0x7f5b441a05e0 2021-02-10T11:32:46.012 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:46.007 7f5b35ffb700 1 -- 172.21.2.102:0/1902980847 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b40004140 con 0x7f5b441a05e0 2021-02-10T11:32:46.012 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:46.007 7f5b35ffb700 1 -- 172.21.2.102:0/1902980847 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5b40004620 con 0x7f5b441a05e0 2021-02-10T11:32:46.012 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:46.007 7f5b4b9d8700 1 -- 172.21.2.102:0/1902980847 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b4418a490 con 0x7f5b441a05e0 2021-02-10T11:32:46.012 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:46.007 7f5b4b9d8700 1 -- 172.21.2.102:0/1902980847 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b4418aa20 con 0x7f5b441a05e0 2021-02-10T11:32:46.012 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:46.007 7f5b35ffb700 1 -- 172.21.2.102:0/1902980847 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f5b40004320 con 0x7f5b441a05e0 2021-02-10T11:32:46.012 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:46.008 7f5b35ffb700 1 --2- 172.21.2.102:0/1902980847 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5b240186c0 0x7f5b2401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:46.012 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:46.008 7f5b35ffb700 1 -- 172.21.2.102:0/1902980847 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f5b40032360 con 0x7f5b441a05e0 2021-02-10T11:32:46.012 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:46.008 7f5b48ff3700 1 --2- 172.21.2.102:0/1902980847 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5b240186c0 0x7f5b2401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:46.012 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:46.008 7f5b48ff3700 1 --2- 172.21.2.102:0/1902980847 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5b240186c0 0x7f5b2401abc0 crc :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:46.040 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:46.038 7f5b4b9d8700 1 -- 172.21.2.102:0/1902980847 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b44082a20 con 0x7f5b441a05e0 2021-02-10T11:32:46.042 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:46.041 7f5b35ffb700 1 -- 172.21.2.102:0/1902980847 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f5b400009b0 con 0x7f5b441a05e0 2021-02-10T11:32:46.170 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:46.169 7f5b4b9d8700 1 -- 172.21.2.102:0/1902980847 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f5b44082a20 con 0x7f5b441a05e0 2021-02-10T11:32:46.170 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:46.169 7f5b35ffb700 1 -- 172.21.2.102:0/1902980847 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v120) v1 ==== 92+0+10 (secure 0 0 0) 0x7f5b40004320 con 0x7f5b441a05e0 2021-02-10T11:32:46.171 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:32:46.172 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:46.171 7f5b4b9d8700 1 -- 172.21.2.102:0/1902980847 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5b240186c0 msgr2=0x7f5b2401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:46.172 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:46.171 7f5b4b9d8700 1 --2- 172.21.2.102:0/1902980847 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5b240186c0 0x7f5b2401abc0 crc :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:46.172 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:46.171 7f5b4b9d8700 1 -- 172.21.2.102:0/1902980847 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5b441a05e0 msgr2=0x7f5b4418b970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:46.172 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:46.171 7f5b4b9d8700 1 --2- 172.21.2.102:0/1902980847 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5b441a05e0 0x7f5b4418b970 secure :-1 s=READY pgs=2324 cs=0 l=1 rev1=1 rx=0x7f5b40001c60 tx=0x7f5b4000a8d0).stop 2021-02-10T11:32:46.173 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:46.171 7f5b4b9d8700 1 -- 172.21.2.102:0/1902980847 shutdown_connections 2021-02-10T11:32:46.173 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:46.171 7f5b4b9d8700 1 --2- 172.21.2.102:0/1902980847 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5b440c0030 0x7f5b44189e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:46.173 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:46.171 7f5b4b9d8700 1 --2- 172.21.2.102:0/1902980847 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5b240186c0 0x7f5b2401abc0 unknown :-1 s=CLOSED pgs=1024 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:46.173 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:46.171 7f5b4b9d8700 1 --2- 172.21.2.102:0/1902980847 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5b440b8780 0x7f5b440bfa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:46.173 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:46.171 7f5b4b9d8700 1 --2- 172.21.2.102:0/1902980847 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5b441a05e0 0x7f5b4418b970 unknown :-1 s=CLOSED pgs=2324 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:46.173 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:46.171 7f5b4b9d8700 1 -- 172.21.2.102:0/1902980847 >> 172.21.2.102:0/1902980847 conn(0x7f5b440a6a80 msgr2=0x7f5b440ae110 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:46.173 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:46.171 7f5b4b9d8700 1 -- 172.21.2.102:0/1902980847 shutdown_connections 2021-02-10T11:32:46.174 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:46.171 7f5b4b9d8700 1 -- 172.21.2.102:0/1902980847 wait complete. 2021-02-10T11:32:46.181 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-02-10T11:32:46.611 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:46.594 7fb726bd6700 1 Processor -- start 2021-02-10T11:32:46.612 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:46.595 7fb726bd6700 1 -- start start 2021-02-10T11:32:46.612 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:46.595 7fb726bd6700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fb72010aff0 0x7fb7200cf750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:46.612 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:46.595 7fb726bd6700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7200fe1c0 con 0x7fb7200d1870 2021-02-10T11:32:46.612 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:46.595 7fb726bd6700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7201b0cb0 con 0x7fb7200d1c70 2021-02-10T11:32:46.612 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:46.595 7fb726bd6700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb720109a60 con 0x7fb72010aff0 2021-02-10T11:32:46.612 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:46.595 7fb7249f2700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fb7200d1870 0x7fb7201b04c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:55258/0 (socket says 172.21.2.102:55258) 2021-02-10T11:32:46.612 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:46.595 7fb7249f2700 1 -- 172.21.2.102:0/2901664098 learned_addr learned my addr 172.21.2.102:0/2901664098 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:32:46.612 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:46.595 7fb7251f3700 1 --2- 172.21.2.102:0/2901664098 >> v2:172.21.2.102:3300/0 conn(0x7fb72010aff0 0x7fb7200cf750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:46.612 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:46.595 7fb71f7fe700 1 -- 172.21.2.102:0/2901664098 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3365670711 0 0) 0x7fb708001180 con 0x7fb7200d1870 2021-02-10T11:32:46.612 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:46.595 7fb7251f3700 1 -- 172.21.2.102:0/2901664098 >> v1:172.21.2.113:6789/0 conn(0x7fb7200d1c70 legacy=0x7fb7200cef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:46.612 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:46.595 7fb7251f3700 1 -- 172.21.2.102:0/2901664098 >> v1:172.21.2.102:6789/0 conn(0x7fb7200d1870 legacy=0x7fb7201b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:46.613 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:46.595 7fb7251f3700 1 -- 172.21.2.102:0/2901664098 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb72010b3f0 con 0x7fb72010aff0 2021-02-10T11:32:46.613 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:46.596 7fb7251f3700 1 --2- 172.21.2.102:0/2901664098 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb72010aff0 0x7fb7200cf750 secure :-1 s=READY pgs=2326 cs=0 l=1 rev1=1 rx=0x7fb710006e30 tx=0x7fb710009e60).ready entity=mon.0 client_cookie=d4032ed9e9e882fa server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:46.613 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:46.596 7fb71f7fe700 1 -- 172.21.2.102:0/2901664098 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb710006a50 con 0x7fb72010aff0 2021-02-10T11:32:46.613 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:46.596 7fb71f7fe700 1 -- 172.21.2.102:0/2901664098 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb71000b300 con 0x7fb72010aff0 2021-02-10T11:32:46.613 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:46.596 7fb71f7fe700 1 -- 172.21.2.102:0/2901664098 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb71000b790 con 0x7fb72010aff0 2021-02-10T11:32:46.613 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:46.596 7fb726bd6700 1 -- 172.21.2.102:0/2901664098 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb72010aff0 msgr2=0x7fb7200cf750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:46.613 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:46.596 7fb726bd6700 1 --2- 172.21.2.102:0/2901664098 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb72010aff0 0x7fb7200cf750 secure :-1 s=READY pgs=2326 cs=0 l=1 rev1=1 rx=0x7fb710006e30 tx=0x7fb710009e60).stop 2021-02-10T11:32:46.613 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:46.596 7fb726bd6700 1 -- 172.21.2.102:0/2901664098 shutdown_connections 2021-02-10T11:32:46.613 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:46.596 7fb726bd6700 1 --2- 172.21.2.102:0/2901664098 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb72010aff0 0x7fb7200cf750 unknown :-1 s=CLOSED pgs=2326 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:46.613 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:46.596 7fb726bd6700 1 -- 172.21.2.102:0/2901664098 >> 172.21.2.102:0/2901664098 conn(0x7fb7200ce420 msgr2=0x7fb7200d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:46.613 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:46.596 7fb726bd6700 1 -- 172.21.2.102:0/2901664098 shutdown_connections 2021-02-10T11:32:46.613 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:46.596 7fb726bd6700 1 -- 172.21.2.102:0/2901664098 wait complete. 2021-02-10T11:32:46.615 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:46.611 7fb726bd6700 1 Processor -- start 2021-02-10T11:32:46.615 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:46.611 7fb726bd6700 1 -- start start 2021-02-10T11:32:46.615 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:46.611 7fb726bd6700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb7200c3820 0x7fb7200a7950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:46.615 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:46.611 7fb726bd6700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb7201a44c0 0x7fb7201a48c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:46.615 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:46.611 7fb726bd6700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb7200863d0 0x7fb7200867d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:46.615 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:46.611 7fb726bd6700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb720085ed0 con 0x7fb7200863d0 2021-02-10T11:32:46.615 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:46.611 7fb726bd6700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7200c0740 con 0x7fb7201a44c0 2021-02-10T11:32:46.615 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:46.611 7fb726bd6700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb720086df0 con 0x7fb7200c3820 2021-02-10T11:32:46.615 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:46.611 7fb7251f3700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb7200863d0 0x7fb7200867d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:46.616 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:46.611 7fb7251f3700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb7200863d0 0x7fb7200867d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:34808/0 (socket says 172.21.2.102:34808) 2021-02-10T11:32:46.616 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:46.611 7fb7251f3700 1 -- 172.21.2.102:0/295104064 learned_addr learned my addr 172.21.2.102:0/295104064 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:32:46.616 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:46.611 7fb7251f3700 1 -- 172.21.2.102:0/295104064 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb7200c3820 msgr2=0x7fb7200a7950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:46.616 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:46.611 7fb7251f3700 1 --2- 172.21.2.102:0/295104064 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb7200c3820 0x7fb7200a7950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:46.616 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:46.611 7fb7251f3700 1 -- 172.21.2.102:0/295104064 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb7201a44c0 msgr2=0x7fb7201a48c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:46.616 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:46.611 7fb7251f3700 1 --2- 172.21.2.102:0/295104064 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb7201a44c0 0x7fb7201a48c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:46.616 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:46.612 7fb7251f3700 1 -- 172.21.2.102:0/295104064 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb720087010 con 0x7fb7200863d0 2021-02-10T11:32:46.616 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:46.612 7fb7251f3700 1 --2- 172.21.2.102:0/295104064 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb7200863d0 0x7fb7200867d0 secure :-1 s=READY pgs=2327 cs=0 l=1 rev1=1 rx=0x7fb710001070 tx=0x7fb71000bd40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:46.616 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:46.612 7fb71d7fa700 1 -- 172.21.2.102:0/295104064 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb710001270 con 0x7fb7200863d0 2021-02-10T11:32:46.616 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:46.612 7fb71d7fa700 1 -- 172.21.2.102:0/295104064 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb710006df0 con 0x7fb7200863d0 2021-02-10T11:32:46.616 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:46.612 7fb726bd6700 1 -- 172.21.2.102:0/295104064 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb720087010 con 0x7fb7200863d0 2021-02-10T11:32:46.616 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:46.612 7fb71d7fa700 1 -- 172.21.2.102:0/295104064 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb7100017b0 con 0x7fb7200863d0 2021-02-10T11:32:46.617 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:46.612 7fb726bd6700 1 -- 172.21.2.102:0/295104064 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb720083000 con 0x7fb7200863d0 2021-02-10T11:32:46.617 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:46.613 7fb71d7fa700 1 -- 172.21.2.102:0/295104064 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fb710001740 con 0x7fb7200863d0 2021-02-10T11:32:46.617 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:46.613 7fb71d7fa700 1 --2- 172.21.2.102:0/295104064 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb7040186c0 0x7fb70401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:46.617 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:46.613 7fb71d7fa700 1 -- 172.21.2.102:0/295104064 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fb710032800 con 0x7fb7200863d0 2021-02-10T11:32:46.617 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:46.613 7fb7249f2700 1 --2- 172.21.2.102:0/295104064 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb7040186c0 0x7fb70401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:46.617 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:46.613 7fb7249f2700 1 --2- 172.21.2.102:0/295104064 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb7040186c0 0x7fb70401abc0 crc :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:46.646 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:46.645 7fb726bd6700 1 -- 172.21.2.102:0/295104064 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb720085e50 con 0x7fb7200863d0 2021-02-10T11:32:46.649 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:46.648 7fb71d7fa700 1 -- 172.21.2.102:0/295104064 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fb710001530 con 0x7fb7200863d0 2021-02-10T11:32:46.777 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:46.775 7fb726bd6700 1 -- 172.21.2.102:0/295104064 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7fb720085e50 con 0x7fb7200863d0 2021-02-10T11:32:46.777 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:46.775 7fb71d7fa700 1 -- 172.21.2.102:0/295104064 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v120) v1 ==== 104+0+10 (secure 0 0 0) 0x7fb710001620 con 0x7fb7200863d0 2021-02-10T11:32:46.777 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:32:46.778 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:46.777 7fb726bd6700 1 -- 172.21.2.102:0/295104064 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb7040186c0 msgr2=0x7fb70401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:46.779 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:46.777 7fb726bd6700 1 --2- 172.21.2.102:0/295104064 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb7040186c0 0x7fb70401abc0 crc :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:46.779 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:46.777 7fb726bd6700 1 -- 172.21.2.102:0/295104064 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb7200863d0 msgr2=0x7fb7200867d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:46.779 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:46.777 7fb726bd6700 1 --2- 172.21.2.102:0/295104064 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb7200863d0 0x7fb7200867d0 secure :-1 s=READY pgs=2327 cs=0 l=1 rev1=1 rx=0x7fb710001070 tx=0x7fb71000bd40).stop 2021-02-10T11:32:46.779 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:46.777 7fb726bd6700 1 -- 172.21.2.102:0/295104064 shutdown_connections 2021-02-10T11:32:46.780 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:46.777 7fb726bd6700 1 --2- 172.21.2.102:0/295104064 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb7200863d0 0x7fb7200867d0 unknown :-1 s=CLOSED pgs=2327 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:46.780 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:46.777 7fb726bd6700 1 --2- 172.21.2.102:0/295104064 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb7040186c0 0x7fb70401abc0 unknown :-1 s=CLOSED pgs=1025 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:46.780 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:46.777 7fb726bd6700 1 --2- 172.21.2.102:0/295104064 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb7201a44c0 0x7fb7201a48c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:46.780 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:46.777 7fb726bd6700 1 --2- 172.21.2.102:0/295104064 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb7200c3820 0x7fb7200a7950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:46.780 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:46.777 7fb726bd6700 1 -- 172.21.2.102:0/295104064 >> 172.21.2.102:0/295104064 conn(0x7fb7200872c0 msgr2=0x7fb720095700 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:46.780 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:46.777 7fb726bd6700 1 -- 172.21.2.102:0/295104064 shutdown_connections 2021-02-10T11:32:46.780 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:46.778 7fb726bd6700 1 -- 172.21.2.102:0/295104064 wait complete. 2021-02-10T11:32:46.788 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-02-10T11:32:47.223 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:47.203 7ffb1f2b8700 1 Processor -- start 2021-02-10T11:32:47.223 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:47.203 7ffb1f2b8700 1 -- start start 2021-02-10T11:32:47.223 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:47.203 7ffb1f2b8700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7ffb180d1870 0x7ffb181b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:47.224 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:47.203 7ffb1f2b8700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ffb181b0ae0 0x7ffb18108940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:47.224 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:47.203 7ffb1f2b8700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb180fe1c0 con 0x7ffb180d1c70 2021-02-10T11:32:47.224 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:47.203 7ffb1f2b8700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb18109a60 con 0x7ffb180d1870 2021-02-10T11:32:47.224 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:47.203 7ffb1f2b8700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb18109750 con 0x7ffb181b0ae0 2021-02-10T11:32:47.224 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:47.203 7ffb1d8d5700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7ffb180d1c70 0x7ffb18108f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:55272/0 (socket says 172.21.2.102:55272) 2021-02-10T11:32:47.224 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:47.203 7ffb1d8d5700 1 -- 172.21.2.102:0/127116536 learned_addr learned my addr 172.21.2.102:0/127116536 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:32:47.224 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:47.204 7ffb0ffff700 1 -- 172.21.2.102:0/127116536 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 470231115 0 0) 0x7ffb10001500 con 0x7ffb180d1c70 2021-02-10T11:32:47.224 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:47.204 7ffb0ffff700 1 -- 172.21.2.102:0/127116536 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb04002fe0 con 0x7ffb180d1c70 2021-02-10T11:32:47.224 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:47.204 7ffb1d0d4700 1 --2- 172.21.2.102:0/127116536 >> v2:172.21.2.113:3300/0 conn(0x7ffb180d1870 0x7ffb181b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:47.225 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:47.204 7ffb0ffff700 1 -- 172.21.2.102:0/127116536 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 197459706 0 0) 0x7ffb10001460 con 0x7ffb180d1c70 2021-02-10T11:32:47.225 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:47.204 7ffb1c8d3700 1 --2- 172.21.2.102:0/127116536 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ffb181b0ae0 0x7ffb18108940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:47.225 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:47.204 7ffb0ffff700 1 -- 172.21.2.102:0/127116536 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ffb181b0ae0 msgr2=0x7ffb18108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:32:47.225 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:47.204 7ffb0ffff700 1 --2- 172.21.2.102:0/127116536 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ffb181b0ae0 0x7ffb18108940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:47.225 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:47.204 7ffb0ffff700 1 -- 172.21.2.102:0/127116536 >> v2:172.21.2.113:3300/0 conn(0x7ffb180d1870 msgr2=0x7ffb181b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:32:47.225 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:47.204 7ffb0ffff700 1 --2- 172.21.2.102:0/127116536 >> v2:172.21.2.113:3300/0 conn(0x7ffb180d1870 0x7ffb181b04c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:47.225 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:47.204 7ffb0ffff700 1 -- 172.21.2.102:0/127116536 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb18088070 con 0x7ffb180d1c70 2021-02-10T11:32:47.225 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:47.204 7ffb0ffff700 1 -- 172.21.2.102:0/127116536 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7ffb10001980 con 0x7ffb180d1c70 2021-02-10T11:32:47.225 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:47.204 7ffb0ffff700 1 -- 172.21.2.102:0/127116536 >> v1:172.21.2.102:6789/0 conn(0x7ffb180d1c70 legacy=0x7ffb18108f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:47.226 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:47.204 7ffb0ffff700 1 --2- 172.21.2.102:0/127116536 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ffb04003a70 0x7ffb04005ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:47.226 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:47.204 7ffb0ffff700 1 --2- 172.21.2.102:0/127116536 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ffb04006760 0x7ffb04008c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:47.226 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:47.204 7ffb0ffff700 1 --2- 172.21.2.102:0/127116536 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ffb04009500 0x7ffb0400ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:47.226 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:47.204 7ffb0ffff700 1 -- 172.21.2.102:0/127116536 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb0400c2d0 con 0x7ffb04006760 2021-02-10T11:32:47.226 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:47.204 7ffb0ffff700 1 -- 172.21.2.102:0/127116536 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb0400c4f0 con 0x7ffb04009500 2021-02-10T11:32:47.226 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:47.204 7ffb0ffff700 1 -- 172.21.2.102:0/127116536 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb04050720 con 0x7ffb04003a70 2021-02-10T11:32:47.226 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:47.204 7ffb1c8d3700 1 --2- 172.21.2.102:0/127116536 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ffb04006760 0x7ffb04008c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:47.226 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:47.205 7ffb1d8d5700 1 --2- 172.21.2.102:0/127116536 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ffb04009500 0x7ffb0400ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:47.226 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:47.205 7ffb1c8d3700 1 -- 172.21.2.102:0/127116536 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ffb04003a70 msgr2=0x7ffb04005ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:32:47.226 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:47.205 7ffb1d0d4700 1 --2- 172.21.2.102:0/127116536 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ffb04003a70 0x7ffb04005ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:47.226 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:47.205 7ffb1c8d3700 1 --2- 172.21.2.102:0/127116536 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ffb04003a70 0x7ffb04005ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:47.226 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:47.205 7ffb1c8d3700 1 -- 172.21.2.102:0/127116536 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ffb04009500 msgr2=0x7ffb0400ba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:32:47.226 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:47.205 7ffb1c8d3700 1 --2- 172.21.2.102:0/127116536 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ffb04009500 0x7ffb0400ba20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:47.227 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:47.205 7ffb1c8d3700 1 -- 172.21.2.102:0/127116536 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb04050980 con 0x7ffb04006760 2021-02-10T11:32:47.227 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:47.205 7ffb1d8d5700 1 --2- 172.21.2.102:0/127116536 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ffb04009500 0x7ffb0400ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:32:47.227 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:47.205 7ffb1c8d3700 1 --2- 172.21.2.102:0/127116536 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ffb04006760 0x7ffb04008c60 secure :-1 s=READY pgs=2329 cs=0 l=1 rev1=1 rx=0x7ffb08006ac0 tx=0x7ffb08009af0).ready entity=mon.0 client_cookie=c115ac45676e717d server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:47.227 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:47.205 7ffb0ffff700 1 -- 172.21.2.102:0/127116536 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ffb0800c0c0 con 0x7ffb04006760 2021-02-10T11:32:47.227 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:47.205 7ffb0ffff700 1 -- 172.21.2.102:0/127116536 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb0800c340 con 0x7ffb04006760 2021-02-10T11:32:47.227 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:47.205 7ffb0ffff700 1 -- 172.21.2.102:0/127116536 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ffb0800c860 con 0x7ffb04006760 2021-02-10T11:32:47.227 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:47.205 7ffb1f2b8700 1 -- 172.21.2.102:0/127116536 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ffb04006760 msgr2=0x7ffb04008c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:47.227 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:47.205 7ffb1f2b8700 1 --2- 172.21.2.102:0/127116536 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ffb04006760 0x7ffb04008c60 secure :-1 s=READY pgs=2329 cs=0 l=1 rev1=1 rx=0x7ffb08006ac0 tx=0x7ffb08009af0).stop 2021-02-10T11:32:47.227 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:47.205 7ffb1d8d5700 1 -- 172.21.2.102:0/127116536 reap_dead start 2021-02-10T11:32:47.227 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:47.206 7ffb1f2b8700 1 -- 172.21.2.102:0/127116536 shutdown_connections 2021-02-10T11:32:47.227 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:47.206 7ffb1f2b8700 1 -- 172.21.2.102:0/127116536 >> 172.21.2.102:0/127116536 conn(0x7ffb180ce420 msgr2=0x7ffb180d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:47.227 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:47.206 7ffb1f2b8700 1 -- 172.21.2.102:0/127116536 shutdown_connections 2021-02-10T11:32:47.228 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:47.206 7ffb1f2b8700 1 -- 172.21.2.102:0/127116536 wait complete. 2021-02-10T11:32:47.228 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:47.221 7ffb1f2b8700 1 Processor -- start 2021-02-10T11:32:47.228 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:47.221 7ffb1f2b8700 1 -- start start 2021-02-10T11:32:47.228 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:47.221 7ffb1f2b8700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ffb181a79a0 0x7ffb181a7da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:47.228 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:47.221 7ffb1f2b8700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ffb180bdb40 0x7ffb180bdf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:47.228 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:47.221 7ffb1f2b8700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ffb18082ca0 0x7ffb180830a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:47.228 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:47.221 7ffb1f2b8700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb1819d990 con 0x7ffb18082ca0 2021-02-10T11:32:47.228 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:47.221 7ffb1f2b8700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb1819dbb0 con 0x7ffb181a79a0 2021-02-10T11:32:47.228 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:47.221 7ffb1f2b8700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb180836c0 con 0x7ffb180bdb40 2021-02-10T11:32:47.228 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:47.221 7ffb1d8d5700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ffb18082ca0 0x7ffb180830a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:47.228 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:47.221 7ffb1d8d5700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ffb18082ca0 0x7ffb180830a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:34828/0 (socket says 172.21.2.102:34828) 2021-02-10T11:32:47.228 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:47.221 7ffb1d8d5700 1 -- 172.21.2.102:0/2695914977 learned_addr learned my addr 172.21.2.102:0/2695914977 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:32:47.229 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:47.222 7ffb1d8d5700 1 -- 172.21.2.102:0/2695914977 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ffb180bdb40 msgr2=0x7ffb180bdf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:47.230 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:47.222 7ffb1d8d5700 1 --2- 172.21.2.102:0/2695914977 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ffb180bdb40 0x7ffb180bdf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:47.230 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:47.222 7ffb1d8d5700 1 -- 172.21.2.102:0/2695914977 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ffb181a79a0 msgr2=0x7ffb181a7da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:47.230 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:47.222 7ffb1d8d5700 1 --2- 172.21.2.102:0/2695914977 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ffb181a79a0 0x7ffb181a7da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:47.230 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:47.222 7ffb1d8d5700 1 -- 172.21.2.102:0/2695914977 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb180838e0 con 0x7ffb18082ca0 2021-02-10T11:32:47.230 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:47.222 7ffb1d8d5700 1 --2- 172.21.2.102:0/2695914977 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ffb18082ca0 0x7ffb180830a0 secure :-1 s=READY pgs=2330 cs=0 l=1 rev1=1 rx=0x7ffb10007f60 tx=0x7ffb1000ac70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:47.230 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:47.222 7ffb0dffb700 1 -- 172.21.2.102:0/2695914977 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ffb10000da0 con 0x7ffb18082ca0 2021-02-10T11:32:47.230 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:47.222 7ffb0dffb700 1 -- 172.21.2.102:0/2695914977 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb10004930 con 0x7ffb18082ca0 2021-02-10T11:32:47.230 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:47.222 7ffb0dffb700 1 -- 172.21.2.102:0/2695914977 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ffb100070d0 con 0x7ffb18082ca0 2021-02-10T11:32:47.230 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:47.222 7ffb1f2b8700 1 -- 172.21.2.102:0/2695914977 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb180838e0 con 0x7ffb18082ca0 2021-02-10T11:32:47.231 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:47.222 7ffb1f2b8700 1 -- 172.21.2.102:0/2695914977 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb18084420 con 0x7ffb18082ca0 2021-02-10T11:32:47.231 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:47.223 7ffb0dffb700 1 -- 172.21.2.102:0/2695914977 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ffb10000da0 con 0x7ffb18082ca0 2021-02-10T11:32:47.231 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:47.223 7ffb0dffb700 1 --2- 172.21.2.102:0/2695914977 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ffaf80186c0 0x7ffaf801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:47.231 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:47.223 7ffb0dffb700 1 -- 172.21.2.102:0/2695914977 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 5573+0+0 (secure 0 0 0) 0x7ffb100328b0 con 0x7ffb18082ca0 2021-02-10T11:32:47.231 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:47.223 7ffb1d0d4700 1 --2- 172.21.2.102:0/2695914977 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ffaf80186c0 0x7ffaf801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:47.231 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:47.224 7ffb1d0d4700 1 --2- 172.21.2.102:0/2695914977 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ffaf80186c0 0x7ffaf801abc0 crc :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:47.255 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:47.253 7ffb1f2b8700 1 -- 172.21.2.102:0/2695914977 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb18082a70 con 0x7ffb18082ca0 2021-02-10T11:32:47.257 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:47.256 7ffb0dffb700 1 -- 172.21.2.102:0/2695914977 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ffb10004dc0 con 0x7ffb18082ca0 2021-02-10T11:32:47.386 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:47.384 7ffb1f2b8700 1 -- 172.21.2.102:0/2695914977 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7ffb180838e0 con 0x7ffb18082ca0 2021-02-10T11:32:47.386 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:47.385 7ffb0dffb700 1 -- 172.21.2.102:0/2695914977 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v120) v1 ==== 100+0+10 (secure 0 0 0) 0x7ffb10000da0 con 0x7ffb18082ca0 2021-02-10T11:32:47.387 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:32:47.388 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:47.387 7ffb1f2b8700 1 -- 172.21.2.102:0/2695914977 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ffaf80186c0 msgr2=0x7ffaf801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:47.388 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:47.387 7ffb1f2b8700 1 --2- 172.21.2.102:0/2695914977 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ffaf80186c0 0x7ffaf801abc0 crc :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:47.388 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:47.387 7ffb1f2b8700 1 -- 172.21.2.102:0/2695914977 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ffb18082ca0 msgr2=0x7ffb180830a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:47.388 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:47.387 7ffb1f2b8700 1 --2- 172.21.2.102:0/2695914977 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ffb18082ca0 0x7ffb180830a0 secure :-1 s=READY pgs=2330 cs=0 l=1 rev1=1 rx=0x7ffb10007f60 tx=0x7ffb1000ac70).stop 2021-02-10T11:32:47.389 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:47.387 7ffb1f2b8700 1 -- 172.21.2.102:0/2695914977 shutdown_connections 2021-02-10T11:32:47.389 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:47.387 7ffb1f2b8700 1 --2- 172.21.2.102:0/2695914977 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ffb18082ca0 0x7ffb180830a0 unknown :-1 s=CLOSED pgs=2330 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:47.389 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:47.387 7ffb1f2b8700 1 --2- 172.21.2.102:0/2695914977 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ffb180bdb40 0x7ffb180bdf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:47.389 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:47.387 7ffb1f2b8700 1 --2- 172.21.2.102:0/2695914977 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ffaf80186c0 0x7ffaf801abc0 unknown :-1 s=CLOSED pgs=1026 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:47.389 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:47.387 7ffb1f2b8700 1 --2- 172.21.2.102:0/2695914977 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ffb181a79a0 0x7ffb181a7da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:47.389 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:47.387 7ffb1f2b8700 1 -- 172.21.2.102:0/2695914977 >> 172.21.2.102:0/2695914977 conn(0x7ffb180bef00 msgr2=0x7ffb1809f200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:47.389 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:47.387 7ffb1f2b8700 1 -- 172.21.2.102:0/2695914977 shutdown_connections 2021-02-10T11:32:47.389 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:47.387 7ffb1f2b8700 1 -- 172.21.2.102:0/2695914977 wait complete. 2021-02-10T11:32:47.397 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-02-10T11:32:47.397 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:32:47.827 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:47.808 7fd4b3bac700 1 Processor -- start 2021-02-10T11:32:47.827 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:47.808 7fd4b3bac700 1 -- start start 2021-02-10T11:32:47.827 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:47.809 7fd4b3bac700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd4ac0d1870 0x7fd4ac1b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:47.827 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:47.809 7fd4b3bac700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fd4ac0f0650 0x7fd4ac0cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:47.827 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:47.809 7fd4b3bac700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4ac0f0b10 con 0x7fd4ac1b0a10 2021-02-10T11:32:47.827 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:47.809 7fd4b3bac700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4ac0fe230 con 0x7fd4ac0f0650 2021-02-10T11:32:47.827 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:47.809 7fd4b3bac700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4ac0e5ed0 con 0x7fd4ac0d1870 2021-02-10T11:32:47.827 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:47.810 7fd4b21c9700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fd4ac0f0650 0x7fd4ac0cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:47.827 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:47.810 7fd4b19c8700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd4ac0d1870 0x7fd4ac1b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:47.828 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:47.810 7fd4b11c7700 1 --1- >> v1:172.21.2.113:6789/0 conn(0x7fd4ac1b0a10 0x7fd4ac0cf330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.113:6789/0 says I am v1:172.21.2.102:34382/0 (socket says 172.21.2.102:34382) 2021-02-10T11:32:47.828 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:47.810 7fd4b11c7700 1 -- 172.21.2.102:0/2111097751 learned_addr learned my addr 172.21.2.102:0/2111097751 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:32:47.828 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:47.810 7fd4b21c9700 1 -- 172.21.2.102:0/2111097751 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd4ac0d1870 msgr2=0x7fd4ac1b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:32:47.828 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:47.810 7fd4b21c9700 1 --2- 172.21.2.102:0/2111097751 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd4ac0d1870 0x7fd4ac1b03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:47.828 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:47.810 7fd4b21c9700 1 -- 172.21.2.102:0/2111097751 >> v1:172.21.2.113:6789/0 conn(0x7fd4ac1b0a10 legacy=0x7fd4ac0cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:47.828 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:47.810 7fd4b21c9700 1 -- 172.21.2.102:0/2111097751 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4ac0f20a0 con 0x7fd4ac0f0650 2021-02-10T11:32:47.828 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:47.810 7fd4b21c9700 1 --2- 172.21.2.102:0/2111097751 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd4ac0f0650 0x7fd4ac0cfb20 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 rx=0x7fd4a4006e50 tx=0x7fd4a4009e70).ready entity=mon.1 client_cookie=4037459858982e67 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:47.828 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:47.810 7fd4b09c6700 1 -- 172.21.2.102:0/2111097751 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd4a4006a20 con 0x7fd4ac0f0650 2021-02-10T11:32:47.828 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:47.810 7fd4b09c6700 1 -- 172.21.2.102:0/2111097751 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4a4006790 con 0x7fd4ac0f0650 2021-02-10T11:32:47.828 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:47.810 7fd4b09c6700 1 -- 172.21.2.102:0/2111097751 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd4a400b700 con 0x7fd4ac0f0650 2021-02-10T11:32:47.828 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:47.811 7fd4b3bac700 1 -- 172.21.2.102:0/2111097751 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd4ac0f0650 msgr2=0x7fd4ac0cfb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:47.828 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:47.811 7fd4b3bac700 1 --2- 172.21.2.102:0/2111097751 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd4ac0f0650 0x7fd4ac0cfb20 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 rx=0x7fd4a4006e50 tx=0x7fd4a4009e70).stop 2021-02-10T11:32:47.828 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:47.811 7fd4b3bac700 1 -- 172.21.2.102:0/2111097751 shutdown_connections 2021-02-10T11:32:47.829 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:47.811 7fd4b3bac700 1 --2- 172.21.2.102:0/2111097751 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd4ac0f0650 0x7fd4ac0cfb20 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:47.829 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:47.811 7fd4b3bac700 1 --2- 172.21.2.102:0/2111097751 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd4ac0d1870 0x7fd4ac1b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:47.829 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:47.811 7fd4b3bac700 1 -- 172.21.2.102:0/2111097751 >> 172.21.2.102:0/2111097751 conn(0x7fd4ac0ce450 msgr2=0x7fd4ac0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:47.829 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:47.811 7fd4b3bac700 1 -- 172.21.2.102:0/2111097751 shutdown_connections 2021-02-10T11:32:47.829 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:47.811 7fd4b3bac700 1 -- 172.21.2.102:0/2111097751 wait complete. 2021-02-10T11:32:47.830 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:47.826 7fd4b3bac700 1 Processor -- start 2021-02-10T11:32:47.830 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:47.826 7fd4b3bac700 1 -- start start 2021-02-10T11:32:47.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:47.826 7fd4b3bac700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd4ac0b2860 0x7fd4ac1a4480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:47.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:47.826 7fd4b3bac700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd4ac1a4aa0 0x7fd4ac0ec620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:47.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:47.826 7fd4b3bac700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd4ac0ecc40 0x7fd4ac097790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:47.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:47.826 7fd4b3bac700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4ac0b2ce0 con 0x7fd4ac0b2860 2021-02-10T11:32:47.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:47.826 7fd4b3bac700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4ac0ed040 con 0x7fd4ac1a4aa0 2021-02-10T11:32:47.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:47.826 7fd4b3bac700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4ac0ed260 con 0x7fd4ac0ecc40 2021-02-10T11:32:47.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:47.826 7fd4b19c8700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd4ac0b2860 0x7fd4ac1a4480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:47.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:47.826 7fd4b19c8700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd4ac0b2860 0x7fd4ac1a4480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:34840/0 (socket says 172.21.2.102:34840) 2021-02-10T11:32:47.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:47.826 7fd4b19c8700 1 -- 172.21.2.102:0/2137788194 learned_addr learned my addr 172.21.2.102:0/2137788194 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:32:47.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:47.827 7fd4b19c8700 1 -- 172.21.2.102:0/2137788194 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd4ac0ecc40 msgr2=0x7fd4ac097790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:47.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:47.827 7fd4b19c8700 1 --2- 172.21.2.102:0/2137788194 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd4ac0ecc40 0x7fd4ac097790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:47.832 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:47.827 7fd4b11c7700 1 --2- 172.21.2.102:0/2137788194 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd4ac1a4aa0 0x7fd4ac0ec620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:47.832 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:47.827 7fd4b19c8700 1 -- 172.21.2.102:0/2137788194 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd4ac1a4aa0 msgr2=0x7fd4ac0ec620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:47.832 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:47.827 7fd4b19c8700 1 --2- 172.21.2.102:0/2137788194 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd4ac1a4aa0 0x7fd4ac0ec620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:47.832 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:47.827 7fd4b19c8700 1 -- 172.21.2.102:0/2137788194 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4ac097db0 con 0x7fd4ac0b2860 2021-02-10T11:32:47.832 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:47.827 7fd4b11c7700 1 --2- 172.21.2.102:0/2137788194 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd4ac1a4aa0 0x7fd4ac0ec620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:32:47.832 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:47.827 7fd4b19c8700 1 --2- 172.21.2.102:0/2137788194 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd4ac0b2860 0x7fd4ac1a4480 secure :-1 s=READY pgs=2331 cs=0 l=1 rev1=1 rx=0x7fd4a800b560 tx=0x7fd4a800b900).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:47.832 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:47.827 7fd4a27fc700 1 -- 172.21.2.102:0/2137788194 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd4a80009b0 con 0x7fd4ac0b2860 2021-02-10T11:32:47.832 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:47.827 7fd4a27fc700 1 -- 172.21.2.102:0/2137788194 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4a800bcf0 con 0x7fd4ac0b2860 2021-02-10T11:32:47.832 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:47.827 7fd4a27fc700 1 -- 172.21.2.102:0/2137788194 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd4a80022b0 con 0x7fd4ac0b2860 2021-02-10T11:32:47.833 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:47.827 7fd4b3bac700 1 -- 172.21.2.102:0/2137788194 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4ac097db0 con 0x7fd4ac0b2860 2021-02-10T11:32:47.833 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:47.827 7fd4b3bac700 1 -- 172.21.2.102:0/2137788194 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4ac097db0 con 0x7fd4ac0b2860 2021-02-10T11:32:47.833 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:47.828 7fd4a27fc700 1 -- 172.21.2.102:0/2137788194 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd4a8006b50 con 0x7fd4ac0b2860 2021-02-10T11:32:47.833 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:47.828 7fd4a27fc700 1 --2- 172.21.2.102:0/2137788194 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd4940186c0 0x7fd49401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:47.833 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:47.828 7fd4a27fc700 1 -- 172.21.2.102:0/2137788194 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fd4a80327b0 con 0x7fd4ac0b2860 2021-02-10T11:32:47.833 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:47.828 7fd4b11c7700 1 --2- 172.21.2.102:0/2137788194 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd4940186c0 0x7fd49401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:47.833 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:47.828 7fd4b11c7700 1 --2- 172.21.2.102:0/2137788194 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd4940186c0 0x7fd49401abc0 crc :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:47.860 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:47.858 7fd4b3bac700 1 -- 172.21.2.102:0/2137788194 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4ac0f0b10 con 0x7fd4ac0b2860 2021-02-10T11:32:47.862 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:47.861 7fd4a27fc700 1 -- 172.21.2.102:0/2137788194 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd4a8006340 con 0x7fd4ac0b2860 2021-02-10T11:32:47.991 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:47.990 7fd4b3bac700 1 -- 172.21.2.102:0/2137788194 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd4ac0f0b10 con 0x7fd4ac0b2860 2021-02-10T11:32:47.992 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:47.990 7fd4a27fc700 1 -- 172.21.2.102:0/2137788194 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+2101 (secure 0 0 0) 0x7fd4a8006b50 con 0x7fd4ac0b2860 2021-02-10T11:32:47.992 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:32:47.993 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":99,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10737,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.102:6835/4014802408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6834","nonce":4014802408},{"type":"v1","addr":"172.21.2.102:6835","nonce":4014802408}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":95}],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:32:38.017410","modified":"2021-02-10 11:32:40.027261","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7693},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7693":{"gid":7693,"name":"b","rank":0,"incarnation":96,"state":"up:active","state_seq":3,"addr":"172.21.2.113:6833/238020319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6832","nonce":238020319},{"type":"v1","addr":"172.21.2.113:6833","nonce":238020319}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[25],"metadata_pool":24,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":12}]} 2021-02-10T11:32:47.993 INFO:teuthology.orchestra.run.gibba002.stderr:dumped fsmap epoch 99 2021-02-10T11:32:47.994 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:47.992 7fd4b3bac700 1 -- 172.21.2.102:0/2137788194 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd4940186c0 msgr2=0x7fd49401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:47.994 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:47.992 7fd4b3bac700 1 --2- 172.21.2.102:0/2137788194 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd4940186c0 0x7fd49401abc0 crc :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:47.994 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:47.992 7fd4b3bac700 1 -- 172.21.2.102:0/2137788194 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd4ac0b2860 msgr2=0x7fd4ac1a4480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:47.994 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:47.992 7fd4b3bac700 1 --2- 172.21.2.102:0/2137788194 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd4ac0b2860 0x7fd4ac1a4480 secure :-1 s=READY pgs=2331 cs=0 l=1 rev1=1 rx=0x7fd4a800b560 tx=0x7fd4a800b900).stop 2021-02-10T11:32:47.994 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:47.993 7fd4b3bac700 1 -- 172.21.2.102:0/2137788194 shutdown_connections 2021-02-10T11:32:47.994 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:47.993 7fd4b3bac700 1 --2- 172.21.2.102:0/2137788194 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd4ac0ecc40 0x7fd4ac097790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:47.994 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:47.993 7fd4b3bac700 1 --2- 172.21.2.102:0/2137788194 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd4940186c0 0x7fd49401abc0 unknown :-1 s=CLOSED pgs=1027 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:47.995 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:47.993 7fd4b3bac700 1 --2- 172.21.2.102:0/2137788194 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd4ac1a4aa0 0x7fd4ac0ec620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:47.995 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:47.993 7fd4b3bac700 1 --2- 172.21.2.102:0/2137788194 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd4ac0b2860 0x7fd4ac1a4480 unknown :-1 s=CLOSED pgs=2331 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:47.995 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:47.993 7fd4b3bac700 1 -- 172.21.2.102:0/2137788194 >> 172.21.2.102:0/2137788194 conn(0x7fd4ac0bf500 msgr2=0x7fd4ac0b5050 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:47.995 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:47.993 7fd4b3bac700 1 -- 172.21.2.102:0/2137788194 shutdown_connections 2021-02-10T11:32:47.995 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:47.993 7fd4b3bac700 1 -- 172.21.2.102:0/2137788194 wait complete. 2021-02-10T11:32:48.204 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 99, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:32:38.017410', 'modified': '2021-02-10 11:32:40.027261', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7693}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7693': {'gid': 7693, 'name': 'b', 'rank': 0, 'incarnation': 96, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.113:6833/238020319', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.113:6832', 'nonce': 238020319}, {'type': 'v1', 'addr': '172.21.2.113:6833', 'nonce': 238020319}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [25], 'metadata_pool': 24, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:32:48.204 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:32:48.204 DEBUG:teuthology.orchestra.run.gibba013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:32:48.603 INFO:teuthology.orchestra.run.gibba013.stdout:{ 2021-02-10T11:32:48.603 INFO:teuthology.orchestra.run.gibba013.stdout: "cluster_fsid": "68db30ea-dbce-4aef-a6df-f01c6fea0513", 2021-02-10T11:32:48.604 INFO:teuthology.orchestra.run.gibba013.stdout: "whoami": 0, 2021-02-10T11:32:48.604 INFO:teuthology.orchestra.run.gibba013.stdout: "id": 7693, 2021-02-10T11:32:48.604 INFO:teuthology.orchestra.run.gibba013.stdout: "want_state": "up:active", 2021-02-10T11:32:48.604 INFO:teuthology.orchestra.run.gibba013.stdout: "state": "up:active", 2021-02-10T11:32:48.604 INFO:teuthology.orchestra.run.gibba013.stdout: "rank_uptime": 10.582845817000001, 2021-02-10T11:32:48.604 INFO:teuthology.orchestra.run.gibba013.stdout: "mdsmap_epoch": 99, 2021-02-10T11:32:48.604 INFO:teuthology.orchestra.run.gibba013.stdout: "osdmap_epoch": 120, 2021-02-10T11:32:48.604 INFO:teuthology.orchestra.run.gibba013.stdout: "osdmap_epoch_barrier": 120, 2021-02-10T11:32:48.604 INFO:teuthology.orchestra.run.gibba013.stdout: "uptime": 16.231512175999999 2021-02-10T11:32:48.604 INFO:teuthology.orchestra.run.gibba013.stdout:} 2021-02-10T11:32:48.608 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "68db30ea-dbce-4aef-a6df-f01c6fea0513", "whoami": 0, "id": 7693, "want_state": "up:active", "state": "up:active", "rank_uptime": 10.582845817000001, "mdsmap_epoch": 99, "osdmap_epoch": 120, "osdmap_epoch_barrier": 120, "uptime": 16.231512175999999 } 2021-02-10T11:32:48.608 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2021-02-10T11:32:48.608 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2021-02-10T11:32:48.608 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@gibba002.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2021-02-10T11:32:48.608 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:32:48.608 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec mkdir -p /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:32:48.943 DEBUG:teuthology.orchestra.run.gibba002:> sudo modprobe fuse 2021-02-10T11:32:49.286 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:32:49.286 DEBUG:teuthology.orchestra.run.gibba002:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:32:49.629 INFO:teuthology.orchestra.run.gibba002.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:32:49.629 INFO:teuthology.orchestra.run.gibba002.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:32:49.630 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:32:49.630 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:32:49.630 DEBUG:teuthology.orchestra.run.gibba002:> ls /sys/fs/fuse/connections 2021-02-10T11:32:49.963 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2021-02-10T11:32:49.964 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:32:49.966 DEBUG:teuthology.orchestra.run.gibba002:> sudo modprobe fuse 2021-02-10T11:32:50.311 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:32:50.311 DEBUG:teuthology.orchestra.run.gibba002:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:32:50.378 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba002.stderr:ceph-fuse[185822]: starting ceph client 2021-02-10T11:32:50.378 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba002.stderr:2021-02-10 11:32:50.377 7f13b6654f80 -1 init, newargv = 0x562e3f19ab80 newargc=9 2021-02-10T11:32:50.384 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba002.stderr:ceph-fuse[185822]: starting fuse 2021-02-10T11:32:50.654 INFO:teuthology.orchestra.run.gibba002.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:32:50.654 INFO:teuthology.orchestra.run.gibba002.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:32:50.655 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:32:50.656 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:32:50.656 DEBUG:teuthology.orchestra.run.gibba002:> ls /sys/fs/fuse/connections 2021-02-10T11:32:50.991 INFO:teuthology.orchestra.run.gibba002.stdout:43 2021-02-10T11:32:50.991 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2021-02-10T11:32:50.992 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:32:50.992 DEBUG:teuthology.orchestra.run.gibba002:> sudo python3 -c ' 2021-02-10T11:32:50.992 DEBUG:teuthology.orchestra.run.gibba002:> import glob 2021-02-10T11:32:50.992 DEBUG:teuthology.orchestra.run.gibba002:> import re 2021-02-10T11:32:50.992 DEBUG:teuthology.orchestra.run.gibba002:> import os 2021-02-10T11:32:50.992 DEBUG:teuthology.orchestra.run.gibba002:> import subprocess 2021-02-10T11:32:50.992 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:32:50.992 DEBUG:teuthology.orchestra.run.gibba002:> def find_socket(client_name): 2021-02-10T11:32:50.992 DEBUG:teuthology.orchestra.run.gibba002:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2021-02-10T11:32:50.992 DEBUG:teuthology.orchestra.run.gibba002:> files = glob.glob(asok_path) 2021-02-10T11:32:50.992 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:32:50.992 DEBUG:teuthology.orchestra.run.gibba002:> # Given a non-glob path, it better be there 2021-02-10T11:32:50.992 DEBUG:teuthology.orchestra.run.gibba002:> if "*" not in asok_path: 2021-02-10T11:32:50.993 DEBUG:teuthology.orchestra.run.gibba002:> assert(len(files) == 1) 2021-02-10T11:32:50.993 DEBUG:teuthology.orchestra.run.gibba002:> return files[0] 2021-02-10T11:32:50.993 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:32:50.993 DEBUG:teuthology.orchestra.run.gibba002:> for f in files: 2021-02-10T11:32:50.993 DEBUG:teuthology.orchestra.run.gibba002:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2021-02-10T11:32:50.993 DEBUG:teuthology.orchestra.run.gibba002:> if os.path.exists("/proc/{0}".format(pid)): 2021-02-10T11:32:50.993 DEBUG:teuthology.orchestra.run.gibba002:> return f 2021-02-10T11:32:50.993 DEBUG:teuthology.orchestra.run.gibba002:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2021-02-10T11:32:50.993 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:32:50.993 DEBUG:teuthology.orchestra.run.gibba002:> print(find_socket("client.0")) 2021-02-10T11:32:50.993 DEBUG:teuthology.orchestra.run.gibba002:> ' 2021-02-10T11:32:51.355 INFO:teuthology.orchestra.run.gibba002.stdout:/var/run/ceph/ceph-client.0.185822.asok 2021-02-10T11:32:51.357 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.185822.asok 2021-02-10T11:32:51.358 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:32:51.358 DEBUG:teuthology.orchestra.run.gibba002:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.185822.asok status 2021-02-10T11:32:51.747 INFO:teuthology.orchestra.run.gibba002.stdout:{ 2021-02-10T11:32:51.747 INFO:teuthology.orchestra.run.gibba002.stdout: "metadata": { 2021-02-10T11:32:51.747 INFO:teuthology.orchestra.run.gibba002.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:32:51.748 INFO:teuthology.orchestra.run.gibba002.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:32:51.748 INFO:teuthology.orchestra.run.gibba002.stdout: "entity_id": "0", 2021-02-10T11:32:51.748 INFO:teuthology.orchestra.run.gibba002.stdout: "hostname": "gibba002", 2021-02-10T11:32:51.748 INFO:teuthology.orchestra.run.gibba002.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2021-02-10T11:32:51.748 INFO:teuthology.orchestra.run.gibba002.stdout: "pid": "185822", 2021-02-10T11:32:51.748 INFO:teuthology.orchestra.run.gibba002.stdout: "root": "/" 2021-02-10T11:32:51.748 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:32:51.748 INFO:teuthology.orchestra.run.gibba002.stdout: "dentry_count": 0, 2021-02-10T11:32:51.748 INFO:teuthology.orchestra.run.gibba002.stdout: "dentry_pinned_count": 0, 2021-02-10T11:32:51.748 INFO:teuthology.orchestra.run.gibba002.stdout: "id": 10893, 2021-02-10T11:32:51.748 INFO:teuthology.orchestra.run.gibba002.stdout: "inst": { 2021-02-10T11:32:51.748 INFO:teuthology.orchestra.run.gibba002.stdout: "name": { 2021-02-10T11:32:51.748 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "client", 2021-02-10T11:32:51.748 INFO:teuthology.orchestra.run.gibba002.stdout: "num": 10893 2021-02-10T11:32:51.749 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:32:51.749 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": { 2021-02-10T11:32:51.749 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:32:51.749 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:0", 2021-02-10T11:32:51.749 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 589667214 2021-02-10T11:32:51.749 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:32:51.749 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:32:51.749 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": { 2021-02-10T11:32:51.749 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:32:51.749 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:0", 2021-02-10T11:32:51.749 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 589667214 2021-02-10T11:32:51.749 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:32:51.749 INFO:teuthology.orchestra.run.gibba002.stdout: "inst_str": "client.10893 172.21.2.102:0/589667214", 2021-02-10T11:32:51.750 INFO:teuthology.orchestra.run.gibba002.stdout: "addr_str": "172.21.2.102:0/589667214", 2021-02-10T11:32:51.750 INFO:teuthology.orchestra.run.gibba002.stdout: "inode_count": 1, 2021-02-10T11:32:51.750 INFO:teuthology.orchestra.run.gibba002.stdout: "mds_epoch": 99, 2021-02-10T11:32:51.750 INFO:teuthology.orchestra.run.gibba002.stdout: "osd_epoch": 120, 2021-02-10T11:32:51.750 INFO:teuthology.orchestra.run.gibba002.stdout: "osd_epoch_barrier": 0, 2021-02-10T11:32:51.750 INFO:teuthology.orchestra.run.gibba002.stdout: "blacklisted": false 2021-02-10T11:32:51.750 INFO:teuthology.orchestra.run.gibba002.stdout:} 2021-02-10T11:32:51.751 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:32:51.752 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:32:51.752 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:32:52.088 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:32:52.089 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:32:52.089 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:32:52.089 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:32:52.434 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:32:52.863 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:52.845 7f331d603700 1 Processor -- start 2021-02-10T11:32:52.863 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:52.845 7f331d603700 1 -- start start 2021-02-10T11:32:52.863 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:52.845 7f331d603700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f33180d1870 0x7f33181b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:52.863 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:52.845 7f331d603700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f33181b0a10 0x7f33180cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:52.863 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:52.845 7f331d603700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33180f0b10 con 0x7f33180d1c70 2021-02-10T11:32:52.863 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:52.845 7f331d603700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33180fe230 con 0x7f33181b0a10 2021-02-10T11:32:52.863 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:52.845 7f331d603700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33180e5ed0 con 0x7f33180d1870 2021-02-10T11:32:52.863 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:52.845 7f3316abe700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f33181b0a10 0x7f33180cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:52.863 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:52.846 7f3316abe700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f33181b0a10 0x7f33180cf330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:34856/0 (socket says 172.21.2.102:34856) 2021-02-10T11:32:52.863 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:52.846 7f3316abe700 1 -- 172.21.2.102:0/2700286102 learned_addr learned my addr 172.21.2.102:0/2700286102 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:32:52.864 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:52.846 7f3316abe700 1 -- 172.21.2.102:0/2700286102 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f33180d1870 msgr2=0x7f33181b03f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:32:52.864 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:52.846 7f3316abe700 1 --2- 172.21.2.102:0/2700286102 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f33180d1870 0x7f33181b03f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:52.864 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:52.846 7f3316abe700 1 -- 172.21.2.102:0/2700286102 >> v1:172.21.2.113:6789/0 conn(0x7f33180d1c70 legacy=0x7f33180cf950 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:32:52.864 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:52.846 7f3316abe700 1 -- 172.21.2.102:0/2700286102 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33180f0650 con 0x7f33181b0a10 2021-02-10T11:32:52.864 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:52.846 7f3316abe700 1 --2- 172.21.2.102:0/2700286102 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f33181b0a10 0x7f33180cf330 secure :-1 s=READY pgs=2335 cs=0 l=1 rev1=1 rx=0x7f3300006700 tx=0x7f3300009b00).ready entity=mon.0 client_cookie=4df359350eff3ee0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:52.864 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:52.846 7f33162bd700 1 -- 172.21.2.102:0/2700286102 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f330000c0c0 con 0x7f33181b0a10 2021-02-10T11:32:52.864 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:52.846 7f33162bd700 1 -- 172.21.2.102:0/2700286102 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f330000c340 con 0x7f33181b0a10 2021-02-10T11:32:52.864 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:52.846 7f33162bd700 1 -- 172.21.2.102:0/2700286102 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f330000c820 con 0x7f33181b0a10 2021-02-10T11:32:52.864 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:52.846 7f331d603700 1 -- 172.21.2.102:0/2700286102 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f33181b0a10 msgr2=0x7f33180cf330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:52.864 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:52.846 7f331d603700 1 --2- 172.21.2.102:0/2700286102 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f33181b0a10 0x7f33180cf330 secure :-1 s=READY pgs=2335 cs=0 l=1 rev1=1 rx=0x7f3300006700 tx=0x7f3300009b00).stop 2021-02-10T11:32:52.864 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:52.847 7f331d603700 1 -- 172.21.2.102:0/2700286102 shutdown_connections 2021-02-10T11:32:52.864 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:52.847 7f331d603700 1 --2- 172.21.2.102:0/2700286102 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f33181b0a10 0x7f33180cf330 unknown :-1 s=CLOSED pgs=2335 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:52.864 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:52.847 7f331d603700 1 --2- 172.21.2.102:0/2700286102 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f33180d1870 0x7f33181b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:52.865 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:52.847 7f331d603700 1 -- 172.21.2.102:0/2700286102 >> 172.21.2.102:0/2700286102 conn(0x7f33180ce450 msgr2=0x7f33180d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:52.865 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:52.847 7f331d603700 1 -- 172.21.2.102:0/2700286102 shutdown_connections 2021-02-10T11:32:52.865 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:52.847 7f331d603700 1 -- 172.21.2.102:0/2700286102 wait complete. 2021-02-10T11:32:52.866 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:52.862 7f331d603700 1 Processor -- start 2021-02-10T11:32:52.866 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:52.862 7f331d603700 1 -- start start 2021-02-10T11:32:52.867 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:52.862 7f331d603700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f33181ae550 0x7f33181924f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:52.867 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:52.862 7f331d603700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f33181ae950 0x7f33181a9aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:52.867 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:52.862 7f331d603700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f33181aa0c0 0x7f33181aa4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:52.867 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:52.862 7f331d603700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33181a9520 con 0x7f33181ae550 2021-02-10T11:32:52.867 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:52.862 7f331d603700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f331818fee0 con 0x7f33181aa0c0 2021-02-10T11:32:52.867 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:52.862 7f331d603700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f331818f9d0 con 0x7f33181ae950 2021-02-10T11:32:52.867 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:52.863 7f33172bf700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f33181ae550 0x7f33181924f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:52.867 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:52.863 7f3316abe700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f33181ae950 0x7f33181a9aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:52.867 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:52.863 7f3317ac0700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f33181aa0c0 0x7f33181aa4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:52.867 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:52.863 7f33172bf700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f33181ae550 0x7f33181924f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:34860/0 (socket says 172.21.2.102:34860) 2021-02-10T11:32:52.867 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:52.863 7f33172bf700 1 -- 172.21.2.102:0/1260247398 learned_addr learned my addr 172.21.2.102:0/1260247398 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:32:52.867 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:52.863 7f33172bf700 1 -- 172.21.2.102:0/1260247398 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f33181ae950 msgr2=0x7f33181a9aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:52.868 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:52.863 7f33172bf700 1 --2- 172.21.2.102:0/1260247398 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f33181ae950 0x7f33181a9aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:52.868 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:52.863 7f33172bf700 1 -- 172.21.2.102:0/1260247398 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f33181aa0c0 msgr2=0x7f33181aa4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:52.868 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:52.863 7f33172bf700 1 --2- 172.21.2.102:0/1260247398 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f33181aa0c0 0x7f33181aa4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:52.868 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:52.863 7f33172bf700 1 -- 172.21.2.102:0/1260247398 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33181a9170 con 0x7f33181ae550 2021-02-10T11:32:52.868 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:52.863 7f33172bf700 1 --2- 172.21.2.102:0/1260247398 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f33181ae550 0x7f33181924f0 secure :-1 s=READY pgs=2336 cs=0 l=1 rev1=1 rx=0x7f330800b5c0 tx=0x7f330800b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:52.868 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:52.863 7f330ffff700 1 -- 172.21.2.102:0/1260247398 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3308000c40 con 0x7f33181ae550 2021-02-10T11:32:52.868 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:52.863 7f330ffff700 1 -- 172.21.2.102:0/1260247398 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f330800be20 con 0x7f33181ae550 2021-02-10T11:32:52.868 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:52.863 7f330ffff700 1 -- 172.21.2.102:0/1260247398 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f33080062e0 con 0x7f33181ae550 2021-02-10T11:32:52.868 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:52.864 7f331d603700 1 -- 172.21.2.102:0/1260247398 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33181a9170 con 0x7f33181ae550 2021-02-10T11:32:52.868 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:52.864 7f331d603700 1 -- 172.21.2.102:0/1260247398 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33181aaae0 con 0x7f33181ae550 2021-02-10T11:32:52.868 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:52.864 7f330ffff700 1 -- 172.21.2.102:0/1260247398 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f3308006030 con 0x7f33181ae550 2021-02-10T11:32:52.868 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:52.864 7f330ffff700 1 --2- 172.21.2.102:0/1260247398 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f32fc0186c0 0x7f32fc01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:52.868 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:52.864 7f330ffff700 1 -- 172.21.2.102:0/1260247398 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f33080325c0 con 0x7f33181ae550 2021-02-10T11:32:52.869 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:52.864 7f3316abe700 1 --2- 172.21.2.102:0/1260247398 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f32fc0186c0 0x7f32fc01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:52.869 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:52.865 7f3316abe700 1 --2- 172.21.2.102:0/1260247398 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f32fc0186c0 0x7f32fc01abc0 crc :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:52.897 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:52.895 7f331d603700 1 -- 172.21.2.102:0/1260247398 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33181aafc0 con 0x7f33181ae550 2021-02-10T11:32:52.899 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:52.898 7f330ffff700 1 -- 172.21.2.102:0/1260247398 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f3308006b80 con 0x7f33181ae550 2021-02-10T11:32:53.023 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:53.021 7f331d603700 1 -- 172.21.2.102:0/1260247398 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f33181aafc0 con 0x7f33181ae550 2021-02-10T11:32:53.023 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:53.022 7f330ffff700 1 -- 172.21.2.102:0/1260247398 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v120) v1 ==== 74+0+14007 (secure 0 0 0) 0x7f3308025d90 con 0x7f33181ae550 2021-02-10T11:32:53.024 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:32:53.024 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":120,"fsid":"68db30ea-dbce-4aef-a6df-f01c6fea0513","created":"2021-02-10 11:03:06.331557","modified":"2021-02-10 11:32:39.019652","last_up_change":"2021-02-10 11:03:32.690562","last_in_change":"2021-02-10 11:03:32.690562","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":25,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:03:37.322519","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":24,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:32:35.986688","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"119","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":25,"pool_name":"cephfs_data","create_time":"2021-02-10 11:32:37.011649","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"120","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"be691b4d-a039-408a-af80-9d53f02b6f1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6818","nonce":36088},{"type":"v1","addr":"172.21.2.102:6820","nonce":36088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6822","nonce":36088},{"type":"v1","addr":"172.21.2.102:6824","nonce":36088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6830","nonce":36088},{"type":"v1","addr":"172.21.2.102:6832","nonce":36088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6826","nonce":36088},{"type":"v1","addr":"172.21.2.102:6828","nonce":36088}]},"public_addr":"172.21.2.102:6820/36088","cluster_addr":"172.21.2.102:6824/36088","heartbeat_back_addr":"172.21.2.102:6832/36088","heartbeat_front_addr":"172.21.2.102:6828/36088","state":["exists","up"]},{"osd":1,"uuid":"64e3ccc8-9c69-4d7c-865e-af8e83822635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6819","nonce":36087},{"type":"v1","addr":"172.21.2.102:6821","nonce":36087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6823","nonce":36087},{"type":"v1","addr":"172.21.2.102:6825","nonce":36087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6831","nonce":36087},{"type":"v1","addr":"172.21.2.102:6833","nonce":36087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6827","nonce":36087},{"type":"v1","addr":"172.21.2.102:6829","nonce":36087}]},"public_addr":"172.21.2.102:6821/36087","cluster_addr":"172.21.2.102:6825/36087","heartbeat_back_addr":"172.21.2.102:6833/36087","heartbeat_front_addr":"172.21.2.102:6829/36087","state":["exists","up"]},{"osd":2,"uuid":"153700af-747f-481f-bdf4-d6072d3a842f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6810","nonce":36089},{"type":"v1","addr":"172.21.2.102:6811","nonce":36089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6812","nonce":36089},{"type":"v1","addr":"172.21.2.102:6813","nonce":36089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6816","nonce":36089},{"type":"v1","addr":"172.21.2.102:6817","nonce":36089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6814","nonce":36089},{"type":"v1","addr":"172.21.2.102:6815","nonce":36089}]},"public_addr":"172.21.2.102:6811/36089","cluster_addr":"172.21.2.102:6813/36089","heartbeat_back_addr":"172.21.2.102:6817/36089","heartbeat_front_addr":"172.21.2.102:6815/36089","state":["exists","up"]},{"osd":3,"uuid":"cbc678f4-83a9-45e6-8830-2f7c232b1ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6802","nonce":36090},{"type":"v1","addr":"172.21.2.102:6803","nonce":36090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6804","nonce":36090},{"type":"v1","addr":"172.21.2.102:6805","nonce":36090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6808","nonce":36090},{"type":"v1","addr":"172.21.2.102:6809","nonce":36090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6806","nonce":36090},{"type":"v1","addr":"172.21.2.102:6807","nonce":36090}]},"public_addr":"172.21.2.102:6803/36090","cluster_addr":"172.21.2.102:6805/36090","heartbeat_back_addr":"172.21.2.102:6809/36090","heartbeat_front_addr":"172.21.2.102:6807/36090","state":["exists","up"]},{"osd":4,"uuid":"c01e4008-b258-4f10-afd2-9e7f37214351","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6808","nonce":36368},{"type":"v1","addr":"172.21.2.113:6811","nonce":36368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6814","nonce":36368},{"type":"v1","addr":"172.21.2.113:6818","nonce":36368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6826","nonce":36368},{"type":"v1","addr":"172.21.2.113:6828","nonce":36368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6820","nonce":36368},{"type":"v1","addr":"172.21.2.113:6824","nonce":36368}]},"public_addr":"172.21.2.113:6811/36368","cluster_addr":"172.21.2.113:6818/36368","heartbeat_back_addr":"172.21.2.113:6828/36368","heartbeat_front_addr":"172.21.2.113:6824/36368","state":["exists","up"]},{"osd":5,"uuid":"551f6007-daf8-4a98-8ee0-c2b16bcccb5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6802","nonce":36366},{"type":"v1","addr":"172.21.2.113:6805","nonce":36366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6807","nonce":36366},{"type":"v1","addr":"172.21.2.113:6810","nonce":36366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6819","nonce":36366},{"type":"v1","addr":"172.21.2.113:6822","nonce":36366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6812","nonce":36366},{"type":"v1","addr":"172.21.2.113:6815","nonce":36366}]},"public_addr":"172.21.2.113:6805/36366","cluster_addr":"172.21.2.113:6810/36366","heartbeat_back_addr":"172.21.2.113:6822/36366","heartbeat_front_addr":"172.21.2.113:6815/36366","state":["exists","up"]},{"osd":6,"uuid":"bf67c040-0e50-48be-b102-c3943bc9ebef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6800","nonce":36367},{"type":"v1","addr":"172.21.2.113:6801","nonce":36367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6803","nonce":36367},{"type":"v1","addr":"172.21.2.113:6804","nonce":36367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6813","nonce":36367},{"type":"v1","addr":"172.21.2.113:6816","nonce":36367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6806","nonce":36367},{"type":"v1","addr":"172.21.2.113:6809","nonce":36367}]},"public_addr":"172.21.2.113:6801/36367","cluster_addr":"172.21.2.113:6804/36367","heartbeat_back_addr":"172.21.2.113:6816/36367","heartbeat_front_addr":"172.21.2.113:6809/36367","state":["exists","up"]},{"osd":7,"uuid":"f5461f80-018d-428f-be89-2eb2bcbe7bb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6817","nonce":36365},{"type":"v1","addr":"172.21.2.113:6821","nonce":36365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6823","nonce":36365},{"type":"v1","addr":"172.21.2.113:6825","nonce":36365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6830","nonce":36365},{"type":"v1","addr":"172.21.2.113:6831","nonce":36365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6827","nonce":36365},{"type":"v1","addr":"172.21.2.113:6829","nonce":36365}]},"public_addr":"172.21.2.113:6821/36365","cluster_addr":"172.21.2.113:6825/36365","heartbeat_back_addr":"172.21.2.113:6831/36365","heartbeat_front_addr":"172.21.2.113:6829/36365","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:32:53.025 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:53.023 7f331d603700 1 -- 172.21.2.102:0/1260247398 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f32fc0186c0 msgr2=0x7f32fc01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:53.025 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:53.023 7f331d603700 1 --2- 172.21.2.102:0/1260247398 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f32fc0186c0 0x7f32fc01abc0 crc :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:53.025 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:53.023 7f331d603700 1 -- 172.21.2.102:0/1260247398 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f33181ae550 msgr2=0x7f33181924f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:53.025 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:53.023 7f331d603700 1 --2- 172.21.2.102:0/1260247398 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f33181ae550 0x7f33181924f0 secure :-1 s=READY pgs=2336 cs=0 l=1 rev1=1 rx=0x7f330800b5c0 tx=0x7f330800b960).stop 2021-02-10T11:32:53.025 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:53.023 7f331d603700 1 -- 172.21.2.102:0/1260247398 shutdown_connections 2021-02-10T11:32:53.025 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:53.023 7f331d603700 1 --2- 172.21.2.102:0/1260247398 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f32fc0186c0 0x7f32fc01abc0 unknown :-1 s=CLOSED pgs=1028 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:53.025 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:53.023 7f331d603700 1 --2- 172.21.2.102:0/1260247398 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f33181aa0c0 0x7f33181aa4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:53.025 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:53.023 7f331d603700 1 --2- 172.21.2.102:0/1260247398 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f33181ae950 0x7f33181a9aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:53.025 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:53.023 7f331d603700 1 --2- 172.21.2.102:0/1260247398 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f33181ae550 0x7f33181924f0 unknown :-1 s=CLOSED pgs=2336 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:53.025 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:53.023 7f331d603700 1 -- 172.21.2.102:0/1260247398 >> 172.21.2.102:0/1260247398 conn(0x7f33181a8c90 msgr2=0x7f33181b38a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:53.026 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:53.024 7f331d603700 1 -- 172.21.2.102:0/1260247398 shutdown_connections 2021-02-10T11:32:53.026 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:53.024 7f331d603700 1 -- 172.21.2.102:0/1260247398 wait complete. 2021-02-10T11:32:53.033 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:03:37.322519', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 24, 'pool_name': 'cephfs_metadata', 'create_time': '2021-02-10 11:32:35.986688', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '119', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 25, 'pool_name': 'cephfs_data', 'create_time': '2021-02-10 11:32:37.011649', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '120', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-02-10T11:32:53.034 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:32:53.467 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:53.449 7f562dec1700 1 Processor -- start 2021-02-10T11:32:53.468 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:53.449 7f562dec1700 1 -- start start 2021-02-10T11:32:53.468 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:53.450 7f562dec1700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f56280d1c70 0x7f56280cef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:53.468 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:53.450 7f562dec1700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56280d0170 con 0x7f56280cf580 2021-02-10T11:32:53.468 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:53.450 7f562dec1700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56280fe1c0 con 0x7f56280d1870 2021-02-10T11:32:53.468 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:53.450 7f562dec1700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56281b0cb0 con 0x7f56280d1c70 2021-02-10T11:32:53.468 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:53.450 7f5626ffd700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f56280d1c70 0x7f56280cef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:53.468 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:53.450 7f5626ffd700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f56280d1c70 0x7f56280cef60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:34868/0 (socket says 172.21.2.102:34868) 2021-02-10T11:32:53.468 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:53.450 7f5627fff700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f56280cf580 0x7f56280cf980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:55328/0 (socket says 172.21.2.102:55328) 2021-02-10T11:32:53.468 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:53.450 7f5626ffd700 1 -- 172.21.2.102:0/2874043474 learned_addr learned my addr 172.21.2.102:0/2874043474 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:32:53.468 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:53.451 7f5626ffd700 1 -- 172.21.2.102:0/2874043474 >> v1:172.21.2.113:6789/0 conn(0x7f56280d1870 legacy=0x7f56281b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:32:53.468 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:53.451 7f5626ffd700 1 -- 172.21.2.102:0/2874043474 >> v1:172.21.2.102:6789/0 conn(0x7f56280cf580 legacy=0x7f56280cf980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:53.468 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:53.451 7f5626ffd700 1 -- 172.21.2.102:0/2874043474 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5628109a60 con 0x7f56280d1c70 2021-02-10T11:32:53.469 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:53.451 7f5626ffd700 1 --2- 172.21.2.102:0/2874043474 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f56280d1c70 0x7f56280cef60 secure :-1 s=READY pgs=2338 cs=0 l=1 rev1=1 rx=0x7f561c006700 tx=0x7f561c009b00).ready entity=mon.0 client_cookie=819fdff847d4f631 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:53.469 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:53.451 7f56267fc700 1 -- 172.21.2.102:0/2874043474 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f561c00c0c0 con 0x7f56280d1c70 2021-02-10T11:32:53.469 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:53.451 7f56267fc700 1 -- 172.21.2.102:0/2874043474 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f561c00c340 con 0x7f56280d1c70 2021-02-10T11:32:53.469 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:53.451 7f56267fc700 1 -- 172.21.2.102:0/2874043474 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f561c00c820 con 0x7f56280d1c70 2021-02-10T11:32:53.469 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:53.451 7f562dec1700 1 -- 172.21.2.102:0/2874043474 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f56280d1c70 msgr2=0x7f56280cef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:53.469 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:53.451 7f562dec1700 1 --2- 172.21.2.102:0/2874043474 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f56280d1c70 0x7f56280cef60 secure :-1 s=READY pgs=2338 cs=0 l=1 rev1=1 rx=0x7f561c006700 tx=0x7f561c009b00).stop 2021-02-10T11:32:53.469 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:53.452 7f562dec1700 1 -- 172.21.2.102:0/2874043474 shutdown_connections 2021-02-10T11:32:53.469 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:53.452 7f562dec1700 1 --2- 172.21.2.102:0/2874043474 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f56280d1c70 0x7f56280cef60 unknown :-1 s=CLOSED pgs=2338 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:53.469 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:53.452 7f562dec1700 1 -- 172.21.2.102:0/2874043474 >> 172.21.2.102:0/2874043474 conn(0x7f56280ce420 msgr2=0x7f56280d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:53.469 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:53.452 7f562dec1700 1 -- 172.21.2.102:0/2874043474 shutdown_connections 2021-02-10T11:32:53.469 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:53.452 7f562dec1700 1 -- 172.21.2.102:0/2874043474 wait complete. 2021-02-10T11:32:53.470 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:53.467 7f562dec1700 1 Processor -- start 2021-02-10T11:32:53.471 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:53.467 7f562dec1700 1 -- start start 2021-02-10T11:32:53.471 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:53.467 7f562dec1700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f56280c0740 0x7f56281c5580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:53.471 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:53.467 7f562dec1700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f56281c5ba0 0x7f56280a5c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:53.471 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:53.467 7f562dec1700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f56280b6e60 0x7f56280a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:53.471 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:53.467 7f562dec1700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56280a6880 con 0x7f56281c5ba0 2021-02-10T11:32:53.471 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:53.467 7f562dec1700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56280a6aa0 con 0x7f56280b6e60 2021-02-10T11:32:53.471 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:53.467 7f562dec1700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56280a6cc0 con 0x7f56280c0740 2021-02-10T11:32:53.471 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:53.467 7f5626ffd700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f56281c5ba0 0x7f56280a5c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:53.471 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:53.467 7f5626ffd700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f56281c5ba0 0x7f56280a5c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:34878/0 (socket says 172.21.2.102:34878) 2021-02-10T11:32:53.471 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:53.467 7f5626ffd700 1 -- 172.21.2.102:0/3475164088 learned_addr learned my addr 172.21.2.102:0/3475164088 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:32:53.471 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:53.468 7f5626ffd700 1 -- 172.21.2.102:0/3475164088 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f56280c0740 msgr2=0x7f56281c5580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:32:53.471 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:53.468 7f5626ffd700 1 --2- 172.21.2.102:0/3475164088 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f56280c0740 0x7f56281c5580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:53.472 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:53.468 7f5626ffd700 1 -- 172.21.2.102:0/3475164088 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f56280b6e60 msgr2=0x7f56280a6260 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:32:53.472 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:53.468 7f5626ffd700 1 --2- 172.21.2.102:0/3475164088 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f56280b6e60 0x7f56280a6260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:53.472 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:53.468 7f5626ffd700 1 -- 172.21.2.102:0/3475164088 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56280a6ee0 con 0x7f56281c5ba0 2021-02-10T11:32:53.472 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:53.468 7f5626ffd700 1 --2- 172.21.2.102:0/3475164088 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f56281c5ba0 0x7f56280a5c40 secure :-1 s=READY pgs=2339 cs=0 l=1 rev1=1 rx=0x7f561c006f20 tx=0x7f561c00b9b0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:53.472 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:53.468 7f560ffff700 1 -- 172.21.2.102:0/3475164088 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f561c00bda0 con 0x7f56281c5ba0 2021-02-10T11:32:53.472 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:53.468 7f560ffff700 1 -- 172.21.2.102:0/3475164088 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f561c009420 con 0x7f56281c5ba0 2021-02-10T11:32:53.472 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:53.468 7f562dec1700 1 -- 172.21.2.102:0/3475164088 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56280a6ee0 con 0x7f56281c5ba0 2021-02-10T11:32:53.472 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:53.468 7f562dec1700 1 -- 172.21.2.102:0/3475164088 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56280b5650 con 0x7f56281c5ba0 2021-02-10T11:32:53.472 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:53.468 7f560ffff700 1 -- 172.21.2.102:0/3475164088 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f561c0052e0 con 0x7f56281c5ba0 2021-02-10T11:32:53.472 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:53.469 7f560ffff700 1 -- 172.21.2.102:0/3475164088 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f561c006030 con 0x7f56281c5ba0 2021-02-10T11:32:53.472 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:53.469 7f560ffff700 1 --2- 172.21.2.102:0/3475164088 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5608018730 0x7f560801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:53.472 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:53.469 7f560ffff700 1 -- 172.21.2.102:0/3475164088 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f561c032780 con 0x7f56281c5ba0 2021-02-10T11:32:53.472 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:53.469 7f56277fe700 1 --2- 172.21.2.102:0/3475164088 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5608018730 0x7f560801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:53.473 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:53.469 7f56277fe700 1 --2- 172.21.2.102:0/3475164088 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5608018730 0x7f560801ac10 crc :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:53.501 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:53.500 7f562dec1700 1 -- 172.21.2.102:0/3475164088 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56280807d0 con 0x7f56281c5ba0 2021-02-10T11:32:53.504 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:53.502 7f560ffff700 1 -- 172.21.2.102:0/3475164088 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f561c009100 con 0x7f56281c5ba0 2021-02-10T11:32:53.631 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:53.630 7f562dec1700 1 -- 172.21.2.102:0/3475164088 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f5628088070 con 0x7f56281c5ba0 2021-02-10T11:32:53.631 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:53.630 7f560ffff700 1 -- 172.21.2.102:0/3475164088 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v120) v1 ==== 92+0+10 (secure 0 0 0) 0x7f561c00bda0 con 0x7f56281c5ba0 2021-02-10T11:32:53.632 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:32:53.633 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:53.632 7f562dec1700 1 -- 172.21.2.102:0/3475164088 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5608018730 msgr2=0x7f560801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:53.633 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:53.632 7f562dec1700 1 --2- 172.21.2.102:0/3475164088 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5608018730 0x7f560801ac10 crc :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:53.633 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:53.632 7f562dec1700 1 -- 172.21.2.102:0/3475164088 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f56281c5ba0 msgr2=0x7f56280a5c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:53.633 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:53.632 7f562dec1700 1 --2- 172.21.2.102:0/3475164088 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f56281c5ba0 0x7f56280a5c40 secure :-1 s=READY pgs=2339 cs=0 l=1 rev1=1 rx=0x7f561c006f20 tx=0x7f561c00b9b0).stop 2021-02-10T11:32:53.634 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:53.632 7f562dec1700 1 -- 172.21.2.102:0/3475164088 shutdown_connections 2021-02-10T11:32:53.634 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:53.632 7f562dec1700 1 --2- 172.21.2.102:0/3475164088 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5608018730 0x7f560801ac10 unknown :-1 s=CLOSED pgs=1029 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:53.634 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:53.632 7f562dec1700 1 --2- 172.21.2.102:0/3475164088 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f56280b6e60 0x7f56280a6260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:53.634 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:53.632 7f562dec1700 1 --2- 172.21.2.102:0/3475164088 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f56281c5ba0 0x7f56280a5c40 unknown :-1 s=CLOSED pgs=2339 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:53.634 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:53.632 7f562dec1700 1 --2- 172.21.2.102:0/3475164088 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f56280c0740 0x7f56281c5580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:53.634 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:53.632 7f562dec1700 1 -- 172.21.2.102:0/3475164088 >> 172.21.2.102:0/3475164088 conn(0x7f56280c4120 msgr2=0x7f56280ffd60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:53.634 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:53.632 7f562dec1700 1 -- 172.21.2.102:0/3475164088 shutdown_connections 2021-02-10T11:32:53.634 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:53.632 7f562dec1700 1 -- 172.21.2.102:0/3475164088 wait complete. 2021-02-10T11:32:53.642 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-02-10T11:32:54.073 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:54.055 7f33b5696700 1 Processor -- start 2021-02-10T11:32:54.073 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:54.055 7f33b5696700 1 -- start start 2021-02-10T11:32:54.073 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:54.056 7f33b5696700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f33b00d1870 0x7f33b01b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:54.073 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:54.056 7f33b5696700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33b00fe1c0 con 0x7f33b00cff70 2021-02-10T11:32:54.073 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:54.056 7f33b5696700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33b0109a60 con 0x7f33b01b0ae0 2021-02-10T11:32:54.073 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:54.056 7f33b5696700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33b0088070 con 0x7f33b00d1870 2021-02-10T11:32:54.073 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:54.056 7f33af2bf700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f33b00d1870 0x7f33b01b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:54.073 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:54.056 7f33afac0700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f33b00cff70 0x7f33b0109130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:55338/0 (socket says 172.21.2.102:55338) 2021-02-10T11:32:54.074 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:54.056 7f33afac0700 1 -- 172.21.2.102:0/2218477907 learned_addr learned my addr 172.21.2.102:0/2218477907 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:32:54.074 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:54.056 7f33af2bf700 1 -- 172.21.2.102:0/2218477907 >> v1:172.21.2.113:6789/0 conn(0x7f33b01b0ae0 legacy=0x7f33b0108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:32:54.074 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:54.056 7f33af2bf700 1 -- 172.21.2.102:0/2218477907 >> v1:172.21.2.102:6789/0 conn(0x7f33b00cff70 legacy=0x7f33b0109130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:54.074 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:54.056 7f33af2bf700 1 -- 172.21.2.102:0/2218477907 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33b0088260 con 0x7f33b00d1870 2021-02-10T11:32:54.074 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:54.056 7f33af2bf700 1 --2- 172.21.2.102:0/2218477907 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f33b00d1870 0x7f33b01b04c0 secure :-1 s=READY pgs=2341 cs=0 l=1 rev1=1 rx=0x7f33a0006650 tx=0x7f33a0009a00).ready entity=mon.0 client_cookie=f9ae06e039d6b0a4 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:54.074 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:54.057 7f33ae2bd700 1 -- 172.21.2.102:0/2218477907 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f33a0006390 con 0x7f33b00d1870 2021-02-10T11:32:54.074 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:54.057 7f33ae2bd700 1 -- 172.21.2.102:0/2218477907 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33a00009b0 con 0x7f33b00d1870 2021-02-10T11:32:54.074 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:54.057 7f33ae2bd700 1 -- 172.21.2.102:0/2218477907 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f33a000c2b0 con 0x7f33b00d1870 2021-02-10T11:32:54.074 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:54.057 7f33b5696700 1 -- 172.21.2.102:0/2218477907 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f33b00d1870 msgr2=0x7f33b01b04c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:54.074 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:54.057 7f33b5696700 1 --2- 172.21.2.102:0/2218477907 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f33b00d1870 0x7f33b01b04c0 secure :-1 s=READY pgs=2341 cs=0 l=1 rev1=1 rx=0x7f33a0006650 tx=0x7f33a0009a00).stop 2021-02-10T11:32:54.074 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:54.057 7f33b5696700 1 -- 172.21.2.102:0/2218477907 shutdown_connections 2021-02-10T11:32:54.074 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:54.057 7f33b5696700 1 --2- 172.21.2.102:0/2218477907 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f33b00d1870 0x7f33b01b04c0 unknown :-1 s=CLOSED pgs=2341 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:54.075 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:54.057 7f33b5696700 1 -- 172.21.2.102:0/2218477907 >> 172.21.2.102:0/2218477907 conn(0x7f33b00ce420 msgr2=0x7f33b00d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:54.075 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:54.057 7f33b5696700 1 -- 172.21.2.102:0/2218477907 shutdown_connections 2021-02-10T11:32:54.075 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:54.057 7f33b5696700 1 -- 172.21.2.102:0/2218477907 wait complete. 2021-02-10T11:32:54.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:54.072 7f33b5696700 1 Processor -- start 2021-02-10T11:32:54.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:54.072 7f33b5696700 1 -- start start 2021-02-10T11:32:54.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:54.072 7f33b5696700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f33b008a650 0x7f33b01c1f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:54.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:54.072 7f33b5696700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f33b0091e10 0x7f33b0092210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:54.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:54.072 7f33b5696700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f33b01076a0 0x7f33b0107aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:54.077 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:54.072 7f33b5696700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33b0092830 con 0x7f33b008a650 2021-02-10T11:32:54.077 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:54.072 7f33b5696700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33b01003e0 con 0x7f33b01076a0 2021-02-10T11:32:54.077 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:54.072 7f33b5696700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33b01c2540 con 0x7f33b0091e10 2021-02-10T11:32:54.077 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:54.073 7f33af2bf700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f33b008a650 0x7f33b01c1f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:54.077 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:54.073 7f33af2bf700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f33b008a650 0x7f33b01c1f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:34884/0 (socket says 172.21.2.102:34884) 2021-02-10T11:32:54.077 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:54.073 7f33af2bf700 1 -- 172.21.2.102:0/1958250950 learned_addr learned my addr 172.21.2.102:0/1958250950 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:32:54.077 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:54.073 7f33afac0700 1 --2- 172.21.2.102:0/1958250950 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f33b01076a0 0x7f33b0107aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:54.077 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:54.073 7f33aeabe700 1 --2- 172.21.2.102:0/1958250950 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f33b0091e10 0x7f33b0092210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:54.077 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:54.073 7f33af2bf700 1 -- 172.21.2.102:0/1958250950 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f33b0091e10 msgr2=0x7f33b0092210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:54.077 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:54.073 7f33af2bf700 1 --2- 172.21.2.102:0/1958250950 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f33b0091e10 0x7f33b0092210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:54.077 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:54.073 7f33af2bf700 1 -- 172.21.2.102:0/1958250950 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f33b01076a0 msgr2=0x7f33b0107aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:54.077 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:54.073 7f33af2bf700 1 --2- 172.21.2.102:0/1958250950 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f33b01076a0 0x7f33b0107aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:54.078 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:54.073 7f33af2bf700 1 -- 172.21.2.102:0/1958250950 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33b008aa50 con 0x7f33b008a650 2021-02-10T11:32:54.078 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:54.073 7f33aeabe700 1 --2- 172.21.2.102:0/1958250950 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f33b0091e10 0x7f33b0092210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:32:54.078 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:54.073 7f33af2bf700 1 --2- 172.21.2.102:0/1958250950 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f33b008a650 0x7f33b01c1f20 secure :-1 s=READY pgs=2342 cs=0 l=1 rev1=1 rx=0x7f33a0006f70 tx=0x7f33a000b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:54.078 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:54.073 7f3397fff700 1 -- 172.21.2.102:0/1958250950 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f33a00064a0 con 0x7f33b008a650 2021-02-10T11:32:54.078 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:54.073 7f3397fff700 1 -- 172.21.2.102:0/1958250950 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33a000bde0 con 0x7f33b008a650 2021-02-10T11:32:54.078 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:54.073 7f33b5696700 1 -- 172.21.2.102:0/1958250950 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33b008aa50 con 0x7f33b008a650 2021-02-10T11:32:54.078 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:54.073 7f3397fff700 1 -- 172.21.2.102:0/1958250950 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f33a0002030 con 0x7f33b008a650 2021-02-10T11:32:54.078 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:54.073 7f33b5696700 1 -- 172.21.2.102:0/1958250950 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33b01083b0 con 0x7f33b008a650 2021-02-10T11:32:54.078 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:54.074 7f3397fff700 1 -- 172.21.2.102:0/1958250950 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f33a0000c90 con 0x7f33b008a650 2021-02-10T11:32:54.078 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:54.074 7f3397fff700 1 --2- 172.21.2.102:0/1958250950 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3390018730 0x7f339001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:54.078 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:54.074 7f3397fff700 1 -- 172.21.2.102:0/1958250950 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f33a0032620 con 0x7f33b008a650 2021-02-10T11:32:54.079 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:54.074 7f33aeabe700 1 --2- 172.21.2.102:0/1958250950 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3390018730 0x7f339001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:54.079 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:54.075 7f33aeabe700 1 --2- 172.21.2.102:0/1958250950 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3390018730 0x7f339001ac10 crc :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:54.107 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:54.106 7f33b5696700 1 -- 172.21.2.102:0/1958250950 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33b00918e0 con 0x7f33b008a650 2021-02-10T11:32:54.109 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:54.108 7f3397fff700 1 -- 172.21.2.102:0/1958250950 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f33a00009b0 con 0x7f33b008a650 2021-02-10T11:32:54.238 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:54.236 7f33b5696700 1 -- 172.21.2.102:0/1958250950 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7f33b00918e0 con 0x7f33b008a650 2021-02-10T11:32:54.238 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:54.237 7f3397fff700 1 -- 172.21.2.102:0/1958250950 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v120) v1 ==== 104+0+10 (secure 0 0 0) 0x7f33a000bd50 con 0x7f33b008a650 2021-02-10T11:32:54.239 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:32:54.240 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:54.238 7f33b5696700 1 -- 172.21.2.102:0/1958250950 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3390018730 msgr2=0x7f339001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:54.240 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:54.238 7f33b5696700 1 --2- 172.21.2.102:0/1958250950 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3390018730 0x7f339001ac10 crc :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:54.240 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:54.238 7f33b5696700 1 -- 172.21.2.102:0/1958250950 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f33b008a650 msgr2=0x7f33b01c1f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:54.240 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:54.238 7f33b5696700 1 --2- 172.21.2.102:0/1958250950 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f33b008a650 0x7f33b01c1f20 secure :-1 s=READY pgs=2342 cs=0 l=1 rev1=1 rx=0x7f33a0006f70 tx=0x7f33a000b960).stop 2021-02-10T11:32:54.241 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:54.239 7f33b5696700 1 -- 172.21.2.102:0/1958250950 shutdown_connections 2021-02-10T11:32:54.241 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:54.239 7f33b5696700 1 --2- 172.21.2.102:0/1958250950 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3390018730 0x7f339001ac10 unknown :-1 s=CLOSED pgs=1030 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:54.241 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:54.239 7f33b5696700 1 --2- 172.21.2.102:0/1958250950 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f33b01076a0 0x7f33b0107aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:54.241 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:54.239 7f33b5696700 1 --2- 172.21.2.102:0/1958250950 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f33b0091e10 0x7f33b0092210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:54.241 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:54.239 7f33b5696700 1 --2- 172.21.2.102:0/1958250950 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f33b008a650 0x7f33b01c1f20 unknown :-1 s=CLOSED pgs=2342 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:54.241 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:54.239 7f33b5696700 1 -- 172.21.2.102:0/1958250950 >> 172.21.2.102:0/1958250950 conn(0x7f33b00bcff0 msgr2=0x7f33b008e9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:54.241 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:54.239 7f33b5696700 1 -- 172.21.2.102:0/1958250950 shutdown_connections 2021-02-10T11:32:54.241 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:54.239 7f33b5696700 1 -- 172.21.2.102:0/1958250950 wait complete. 2021-02-10T11:32:54.248 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-02-10T11:32:54.681 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:54.662 7f1af0c4a700 1 Processor -- start 2021-02-10T11:32:54.681 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:54.662 7f1af0c4a700 1 -- start start 2021-02-10T11:32:54.681 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:54.663 7f1af0c4a700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1aec0d1870 0x7f1aec1b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:54.681 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:54.663 7f1af0c4a700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1aec109a60 con 0x7f1aec1b0ae0 2021-02-10T11:32:54.682 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:54.663 7f1af0c4a700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1aec088070 con 0x7f1aec0cff70 2021-02-10T11:32:54.682 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:54.663 7f1af0c4a700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1aec0e5ef0 con 0x7f1aec0d1870 2021-02-10T11:32:54.682 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:54.663 7f1aea05d700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f1aec1b0ae0 0x7f1aec108940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:55348/0 (socket says 172.21.2.102:55348) 2021-02-10T11:32:54.682 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:54.663 7f1aea05d700 1 -- 172.21.2.102:0/1940832124 learned_addr learned my addr 172.21.2.102:0/1940832124 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:32:54.682 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:54.663 7f1aea85e700 1 --2- 172.21.2.102:0/1940832124 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1aec0d1870 0x7f1aec1b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:54.682 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:54.663 7f1ae985c700 1 -- 172.21.2.102:0/1940832124 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2503550621 0 0) 0x7f1ae0000ce0 con 0x7f1aec1b0ae0 2021-02-10T11:32:54.682 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:54.663 7f1ae985c700 1 -- 172.21.2.102:0/1940832124 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ad4002fe0 con 0x7f1aec1b0ae0 2021-02-10T11:32:54.682 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:54.663 7f1ae985c700 1 -- 172.21.2.102:0/1940832124 <== mon.1 v1:172.21.2.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1290168415 0 0) 0x7f1ae4001400 con 0x7f1aec0cff70 2021-02-10T11:32:54.682 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:54.663 7f1ae985c700 1 -- 172.21.2.102:0/1940832124 --> v1:172.21.2.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ad4003210 con 0x7f1aec0cff70 2021-02-10T11:32:54.682 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:54.664 7f1aea85e700 1 -- 172.21.2.102:0/1940832124 >> v1:172.21.2.113:6789/0 conn(0x7f1aec0cff70 legacy=0x7f1aec109130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:54.682 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:54.664 7f1aea85e700 1 -- 172.21.2.102:0/1940832124 >> v1:172.21.2.102:6789/0 conn(0x7f1aec1b0ae0 legacy=0x7f1aec108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:54.682 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:54.664 7f1aea85e700 1 -- 172.21.2.102:0/1940832124 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1aec0d1c70 con 0x7f1aec0d1870 2021-02-10T11:32:54.683 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:54.664 7f1aea85e700 1 --2- 172.21.2.102:0/1940832124 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1aec0d1870 0x7f1aec1b04c0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 rx=0x7f1adc006650 tx=0x7f1adc009a80).ready entity=mon.2 client_cookie=66f52b3fb801b0b5 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:54.683 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:54.664 7f1ae985c700 1 -- 172.21.2.102:0/1940832124 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1adc00c0c0 con 0x7f1aec0d1870 2021-02-10T11:32:54.683 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:54.664 7f1ae985c700 1 -- 172.21.2.102:0/1940832124 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1adc00c340 con 0x7f1aec0d1870 2021-02-10T11:32:54.683 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:54.665 7f1ae985c700 1 -- 172.21.2.102:0/1940832124 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1adc00c860 con 0x7f1aec0d1870 2021-02-10T11:32:54.683 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:54.665 7f1af0c4a700 1 -- 172.21.2.102:0/1940832124 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1aec0d1870 msgr2=0x7f1aec1b04c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:54.683 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:54.665 7f1af0c4a700 1 --2- 172.21.2.102:0/1940832124 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1aec0d1870 0x7f1aec1b04c0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 rx=0x7f1adc006650 tx=0x7f1adc009a80).stop 2021-02-10T11:32:54.683 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:54.665 7f1af0c4a700 1 -- 172.21.2.102:0/1940832124 shutdown_connections 2021-02-10T11:32:54.683 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:54.665 7f1af0c4a700 1 --2- 172.21.2.102:0/1940832124 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1aec0d1870 0x7f1aec1b04c0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:54.683 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:54.665 7f1af0c4a700 1 -- 172.21.2.102:0/1940832124 >> 172.21.2.102:0/1940832124 conn(0x7f1aec0ce420 msgr2=0x7f1aec0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:54.683 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:54.665 7f1af0c4a700 1 -- 172.21.2.102:0/1940832124 shutdown_connections 2021-02-10T11:32:54.683 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:54.665 7f1af0c4a700 1 -- 172.21.2.102:0/1940832124 wait complete. 2021-02-10T11:32:54.685 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:54.680 7f1af0c4a700 1 Processor -- start 2021-02-10T11:32:54.685 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:54.680 7f1af0c4a700 1 -- start start 2021-02-10T11:32:54.685 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:54.680 7f1af0c4a700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1aec1a7bf0 0x7f1aec1a7ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:54.685 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:54.680 7f1af0c4a700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1aec195560 0x7f1aec195960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:54.685 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:54.680 7f1af0c4a700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1aec195f80 0x7f1aec196380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:54.685 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:54.680 7f1af0c4a700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1aec194fe0 con 0x7f1aec195560 2021-02-10T11:32:54.686 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:54.680 7f1af0c4a700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1aec193020 con 0x7f1aec195f80 2021-02-10T11:32:54.686 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:54.680 7f1af0c4a700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1aec193440 con 0x7f1aec1a7bf0 2021-02-10T11:32:54.686 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:54.680 7f1aea05d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1aec195560 0x7f1aec195960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:54.686 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:54.680 7f1aea05d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1aec195560 0x7f1aec195960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:34898/0 (socket says 172.21.2.102:34898) 2021-02-10T11:32:54.686 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:54.680 7f1aea05d700 1 -- 172.21.2.102:0/2976349939 learned_addr learned my addr 172.21.2.102:0/2976349939 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:32:54.686 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:54.681 7f1aea05d700 1 -- 172.21.2.102:0/2976349939 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1aec1a7bf0 msgr2=0x7f1aec1a7ff0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:32:54.686 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:54.681 7f1aea05d700 1 --2- 172.21.2.102:0/2976349939 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1aec1a7bf0 0x7f1aec1a7ff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:54.686 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:54.681 7f1aea05d700 1 -- 172.21.2.102:0/2976349939 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1aec195f80 msgr2=0x7f1aec196380 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:32:54.686 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:54.681 7f1aea05d700 1 --2- 172.21.2.102:0/2976349939 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1aec195f80 0x7f1aec196380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:54.687 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:54.681 7f1aea05d700 1 -- 172.21.2.102:0/2976349939 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1aec193660 con 0x7f1aec195560 2021-02-10T11:32:54.687 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:54.681 7f1aea05d700 1 --2- 172.21.2.102:0/2976349939 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1aec195560 0x7f1aec195960 secure :-1 s=READY pgs=2344 cs=0 l=1 rev1=1 rx=0x7f1ae0001490 tx=0x7f1ae000a950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:54.687 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:54.681 7f1adb7fe700 1 -- 172.21.2.102:0/2976349939 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1ae0000ce0 con 0x7f1aec195560 2021-02-10T11:32:54.687 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:54.681 7f1adb7fe700 1 -- 172.21.2.102:0/2976349939 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ae00043a0 con 0x7f1aec195560 2021-02-10T11:32:54.687 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:54.681 7f1adb7fe700 1 -- 172.21.2.102:0/2976349939 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1ae00047f0 con 0x7f1aec195560 2021-02-10T11:32:54.687 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:54.681 7f1af0c4a700 1 -- 172.21.2.102:0/2976349939 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1aec193660 con 0x7f1aec195560 2021-02-10T11:32:54.687 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:54.681 7f1af0c4a700 1 -- 172.21.2.102:0/2976349939 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1aec193660 con 0x7f1aec195560 2021-02-10T11:32:54.687 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:54.681 7f1adb7fe700 1 -- 172.21.2.102:0/2976349939 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1ae0004580 con 0x7f1aec195560 2021-02-10T11:32:54.687 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:54.681 7f1adb7fe700 1 --2- 172.21.2.102:0/2976349939 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1ad0018730 0x7f1ad001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:54.687 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:54.681 7f1adb7fe700 1 -- 172.21.2.102:0/2976349939 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f1ae0032670 con 0x7f1aec195560 2021-02-10T11:32:54.687 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:54.682 7f1aea85e700 1 --2- 172.21.2.102:0/2976349939 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1ad0018730 0x7f1ad001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:54.687 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:54.682 7f1aea85e700 1 --2- 172.21.2.102:0/2976349939 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1ad0018730 0x7f1ad001ac10 crc :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:54.714 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:54.712 7f1af0c4a700 1 -- 172.21.2.102:0/2976349939 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1aec082980 con 0x7f1aec195560 2021-02-10T11:32:54.716 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:54.715 7f1adb7fe700 1 -- 172.21.2.102:0/2976349939 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f1ae0000940 con 0x7f1aec195560 2021-02-10T11:32:54.845 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:54.843 7f1af0c4a700 1 -- 172.21.2.102:0/2976349939 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7f1aec109a60 con 0x7f1aec195560 2021-02-10T11:32:54.845 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:54.844 7f1adb7fe700 1 -- 172.21.2.102:0/2976349939 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v120) v1 ==== 100+0+10 (secure 0 0 0) 0x7f1ae000ad40 con 0x7f1aec195560 2021-02-10T11:32:54.846 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:32:54.847 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:54.846 7f1af0c4a700 1 -- 172.21.2.102:0/2976349939 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1ad0018730 msgr2=0x7f1ad001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:54.847 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:54.846 7f1af0c4a700 1 --2- 172.21.2.102:0/2976349939 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1ad0018730 0x7f1ad001ac10 crc :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:54.847 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:54.846 7f1af0c4a700 1 -- 172.21.2.102:0/2976349939 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1aec195560 msgr2=0x7f1aec195960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:54.847 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:54.846 7f1af0c4a700 1 --2- 172.21.2.102:0/2976349939 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1aec195560 0x7f1aec195960 secure :-1 s=READY pgs=2344 cs=0 l=1 rev1=1 rx=0x7f1ae0001490 tx=0x7f1ae000a950).stop 2021-02-10T11:32:54.848 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:54.846 7f1af0c4a700 1 -- 172.21.2.102:0/2976349939 shutdown_connections 2021-02-10T11:32:54.848 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:54.846 7f1af0c4a700 1 --2- 172.21.2.102:0/2976349939 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1ad0018730 0x7f1ad001ac10 unknown :-1 s=CLOSED pgs=1031 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:54.848 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:54.846 7f1af0c4a700 1 --2- 172.21.2.102:0/2976349939 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1aec195f80 0x7f1aec196380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:54.848 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:54.846 7f1af0c4a700 1 --2- 172.21.2.102:0/2976349939 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1aec195560 0x7f1aec195960 unknown :-1 s=CLOSED pgs=2344 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:54.848 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:54.846 7f1af0c4a700 1 --2- 172.21.2.102:0/2976349939 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1aec1a7bf0 0x7f1aec1a7ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:54.848 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:54.846 7f1af0c4a700 1 -- 172.21.2.102:0/2976349939 >> 172.21.2.102:0/2976349939 conn(0x7f1aec0a3270 msgr2=0x7f1aec095970 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:54.848 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:54.846 7f1af0c4a700 1 -- 172.21.2.102:0/2976349939 shutdown_connections 2021-02-10T11:32:54.848 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:54.846 7f1af0c4a700 1 -- 172.21.2.102:0/2976349939 wait complete. 2021-02-10T11:32:54.856 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-02-10T11:32:54.856 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:32:55.284 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:55.266 7f2fa772a700 1 Processor -- start 2021-02-10T11:32:55.284 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:55.266 7f2fa772a700 1 -- start start 2021-02-10T11:32:55.284 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:55.266 7f2fa772a700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f2fa00d1870 0x7f2fa01b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:55.285 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:55.266 7f2fa772a700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fa00f0b10 con 0x7f2fa00f0650 2021-02-10T11:32:55.285 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:55.266 7f2fa772a700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fa00d0310 con 0x7f2fa01b0a10 2021-02-10T11:32:55.285 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:55.266 7f2fa772a700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fa00d0530 con 0x7f2fa00d1870 2021-02-10T11:32:55.285 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:55.266 7f2fa5546700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f2fa00d1870 0x7f2fa01b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:55.285 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:55.267 7f2fa5d47700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f2fa00f0650 0x7f2fa00cfb20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:55360/0 (socket says 172.21.2.102:55360) 2021-02-10T11:32:55.285 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:55.267 7f2fa5d47700 1 -- 172.21.2.102:0/3391745294 learned_addr learned my addr 172.21.2.102:0/3391745294 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:32:55.285 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:55.267 7f2fa5546700 1 -- 172.21.2.102:0/3391745294 >> v1:172.21.2.113:6789/0 conn(0x7f2fa01b0a10 legacy=0x7f2fa00cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:32:55.285 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:55.267 7f2fa5546700 1 -- 172.21.2.102:0/3391745294 >> v1:172.21.2.102:6789/0 conn(0x7f2fa00f0650 legacy=0x7f2fa00cfb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:55.285 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:55.267 7f2fa5546700 1 -- 172.21.2.102:0/3391745294 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fa00d1c70 con 0x7f2fa00d1870 2021-02-10T11:32:55.285 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:55.267 7f2fa5546700 1 --2- 172.21.2.102:0/3391745294 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2fa00d1870 0x7f2fa01b03f0 secure :-1 s=READY pgs=2346 cs=0 l=1 rev1=1 rx=0x7f2f9c006650 tx=0x7f2f9c009a00).ready entity=mon.0 client_cookie=89df1a4104ef32c1 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:55.285 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:55.267 7f2f93fff700 1 -- 172.21.2.102:0/3391745294 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2f9c009df0 con 0x7f2fa00d1870 2021-02-10T11:32:55.285 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:55.268 7f2f93fff700 1 -- 172.21.2.102:0/3391745294 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f9c0009b0 con 0x7f2fa00d1870 2021-02-10T11:32:55.285 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:55.268 7f2f93fff700 1 -- 172.21.2.102:0/3391745294 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2f9c00c2b0 con 0x7f2fa00d1870 2021-02-10T11:32:55.286 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:55.268 7f2fa772a700 1 -- 172.21.2.102:0/3391745294 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2fa00d1870 msgr2=0x7f2fa01b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:55.286 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:55.268 7f2fa772a700 1 --2- 172.21.2.102:0/3391745294 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2fa00d1870 0x7f2fa01b03f0 secure :-1 s=READY pgs=2346 cs=0 l=1 rev1=1 rx=0x7f2f9c006650 tx=0x7f2f9c009a00).stop 2021-02-10T11:32:55.286 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:55.268 7f2fa772a700 1 -- 172.21.2.102:0/3391745294 shutdown_connections 2021-02-10T11:32:55.286 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:55.268 7f2fa772a700 1 --2- 172.21.2.102:0/3391745294 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2fa00d1870 0x7f2fa01b03f0 unknown :-1 s=CLOSED pgs=2346 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:55.286 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:55.268 7f2fa772a700 1 -- 172.21.2.102:0/3391745294 >> 172.21.2.102:0/3391745294 conn(0x7f2fa00ce450 msgr2=0x7f2fa00d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:55.286 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:55.269 7f2fa772a700 1 -- 172.21.2.102:0/3391745294 shutdown_connections 2021-02-10T11:32:55.286 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:55.269 7f2fa772a700 1 -- 172.21.2.102:0/3391745294 wait complete. 2021-02-10T11:32:55.287 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:55.283 7f2fa772a700 1 Processor -- start 2021-02-10T11:32:55.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:55.283 7f2fa772a700 1 -- start start 2021-02-10T11:32:55.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:55.284 7f2fa772a700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2fa00c33c0 0x7f2fa00c37c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:55.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:55.284 7f2fa772a700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2fa008beb0 0x7f2fa008c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:55.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:55.284 7f2fa772a700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2fa008c8d0 0x7f2fa00ec000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:55.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:55.284 7f2fa772a700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fa00ec620 con 0x7f2fa00c33c0 2021-02-10T11:32:55.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:55.284 7f2fa772a700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fa00ec840 con 0x7f2fa008c8d0 2021-02-10T11:32:55.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:55.284 7f2fa772a700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fa00eca60 con 0x7f2fa008beb0 2021-02-10T11:32:55.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:55.284 7f2fa5546700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2fa00c33c0 0x7f2fa00c37c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:55.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:55.284 7f2fa5546700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2fa00c33c0 0x7f2fa00c37c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:34908/0 (socket says 172.21.2.102:34908) 2021-02-10T11:32:55.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:55.284 7f2fa5546700 1 -- 172.21.2.102:0/3026967833 learned_addr learned my addr 172.21.2.102:0/3026967833 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:32:55.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:55.284 7f2fa5546700 1 -- 172.21.2.102:0/3026967833 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2fa008beb0 msgr2=0x7f2fa008c2b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:32:55.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:55.284 7f2fa5546700 1 --2- 172.21.2.102:0/3026967833 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2fa008beb0 0x7f2fa008c2b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:55.289 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:55.284 7f2fa5546700 1 -- 172.21.2.102:0/3026967833 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2fa008c8d0 msgr2=0x7f2fa00ec000 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:32:55.289 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:55.284 7f2fa5546700 1 --2- 172.21.2.102:0/3026967833 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2fa008c8d0 0x7f2fa00ec000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:55.289 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:55.284 7f2fa5546700 1 -- 172.21.2.102:0/3026967833 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fa00ecc80 con 0x7f2fa00c33c0 2021-02-10T11:32:55.289 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:55.284 7f2fa5546700 1 --2- 172.21.2.102:0/3026967833 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2fa00c33c0 0x7f2fa00c37c0 secure :-1 s=READY pgs=2347 cs=0 l=1 rev1=1 rx=0x7f2f9c0096a0 tx=0x7f2f9c00b9b0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:55.289 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:55.284 7f2f91ffb700 1 -- 172.21.2.102:0/3026967833 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2f9c0093a0 con 0x7f2fa00c33c0 2021-02-10T11:32:55.289 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:55.285 7f2f91ffb700 1 -- 172.21.2.102:0/3026967833 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f9c0067f0 con 0x7f2fa00c33c0 2021-02-10T11:32:55.289 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:55.285 7f2f91ffb700 1 -- 172.21.2.102:0/3026967833 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2f9c002220 con 0x7f2fa00c33c0 2021-02-10T11:32:55.289 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:55.285 7f2fa772a700 1 -- 172.21.2.102:0/3026967833 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fa00ecc80 con 0x7f2fa00c33c0 2021-02-10T11:32:55.289 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:55.285 7f2fa772a700 1 -- 172.21.2.102:0/3026967833 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fa00ed210 con 0x7f2fa00c33c0 2021-02-10T11:32:55.289 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:55.285 7f2f91ffb700 1 -- 172.21.2.102:0/3026967833 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f2f9c0052e0 con 0x7f2fa00c33c0 2021-02-10T11:32:55.289 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:55.285 7f2f91ffb700 1 --2- 172.21.2.102:0/3026967833 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2f800186c0 0x7f2f8001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:55.289 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:55.285 7f2f91ffb700 1 -- 172.21.2.102:0/3026967833 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f2f9c032780 con 0x7f2fa00c33c0 2021-02-10T11:32:55.289 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:55.285 7f2fa4d45700 1 --2- 172.21.2.102:0/3026967833 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2f800186c0 0x7f2f8001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:55.290 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:55.286 7f2fa4d45700 1 --2- 172.21.2.102:0/3026967833 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2f800186c0 0x7f2f8001abc0 crc :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:55.318 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:55.316 7f2fa772a700 1 -- 172.21.2.102:0/3026967833 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fa00ed4c0 con 0x7f2fa00c33c0 2021-02-10T11:32:55.320 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:55.319 7f2f91ffb700 1 -- 172.21.2.102:0/3026967833 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f2f9c0093a0 con 0x7f2fa00c33c0 2021-02-10T11:32:55.449 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:55.447 7f2fa772a700 1 -- 172.21.2.102:0/3026967833 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2fa00ed4c0 con 0x7f2fa00c33c0 2021-02-10T11:32:55.449 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:55.448 7f2f91ffb700 1 -- 172.21.2.102:0/3026967833 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+2101 (secure 0 0 0) 0x7f2f9c00bda0 con 0x7f2fa00c33c0 2021-02-10T11:32:55.450 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:32:55.450 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":99,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10737,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.102:6835/4014802408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6834","nonce":4014802408},{"type":"v1","addr":"172.21.2.102:6835","nonce":4014802408}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":95}],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:32:38.017410","modified":"2021-02-10 11:32:40.027261","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7693},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7693":{"gid":7693,"name":"b","rank":0,"incarnation":96,"state":"up:active","state_seq":3,"addr":"172.21.2.113:6833/238020319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6832","nonce":238020319},{"type":"v1","addr":"172.21.2.113:6833","nonce":238020319}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[25],"metadata_pool":24,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":12}]} 2021-02-10T11:32:55.450 INFO:teuthology.orchestra.run.gibba002.stderr:dumped fsmap epoch 99 2021-02-10T11:32:55.451 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:55.450 7f2fa772a700 1 -- 172.21.2.102:0/3026967833 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2f800186c0 msgr2=0x7f2f8001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:55.451 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:55.450 7f2fa772a700 1 --2- 172.21.2.102:0/3026967833 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2f800186c0 0x7f2f8001abc0 crc :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:55.451 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:55.450 7f2fa772a700 1 -- 172.21.2.102:0/3026967833 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2fa00c33c0 msgr2=0x7f2fa00c37c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:55.451 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:55.450 7f2fa772a700 1 --2- 172.21.2.102:0/3026967833 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2fa00c33c0 0x7f2fa00c37c0 secure :-1 s=READY pgs=2347 cs=0 l=1 rev1=1 rx=0x7f2f9c0096a0 tx=0x7f2f9c00b9b0).stop 2021-02-10T11:32:55.452 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:55.450 7f2fa772a700 1 -- 172.21.2.102:0/3026967833 shutdown_connections 2021-02-10T11:32:55.452 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:55.450 7f2fa772a700 1 --2- 172.21.2.102:0/3026967833 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2f800186c0 0x7f2f8001abc0 unknown :-1 s=CLOSED pgs=1032 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:55.452 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:55.450 7f2fa772a700 1 --2- 172.21.2.102:0/3026967833 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2fa008c8d0 0x7f2fa00ec000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:55.452 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:55.450 7f2fa772a700 1 --2- 172.21.2.102:0/3026967833 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2fa008beb0 0x7f2fa008c2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:55.452 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:55.450 7f2fa772a700 1 --2- 172.21.2.102:0/3026967833 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2fa00c33c0 0x7f2fa00c37c0 unknown :-1 s=CLOSED pgs=2347 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:55.452 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:55.450 7f2fa772a700 1 -- 172.21.2.102:0/3026967833 >> 172.21.2.102:0/3026967833 conn(0x7f2fa00a0450 msgr2=0x7f2fa00f0460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:55.452 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:55.450 7f2fa772a700 1 -- 172.21.2.102:0/3026967833 shutdown_connections 2021-02-10T11:32:55.452 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:32:55.450 7f2fa772a700 1 -- 172.21.2.102:0/3026967833 wait complete. 2021-02-10T11:32:55.661 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 99, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:32:38.017410', 'modified': '2021-02-10 11:32:40.027261', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7693}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7693': {'gid': 7693, 'name': 'b', 'rank': 0, 'incarnation': 96, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.113:6833/238020319', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.113:6832', 'nonce': 238020319}, {'type': 'v1', 'addr': '172.21.2.113:6833', 'nonce': 238020319}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [25], 'metadata_pool': 24, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:32:55.721 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:32:55.723 DEBUG:teuthology.orchestra.run.gibba013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:32:56.144 INFO:teuthology.orchestra.run.gibba013.stdout:{ 2021-02-10T11:32:56.144 INFO:teuthology.orchestra.run.gibba013.stdout: "cluster_fsid": "68db30ea-dbce-4aef-a6df-f01c6fea0513", 2021-02-10T11:32:56.144 INFO:teuthology.orchestra.run.gibba013.stdout: "whoami": 0, 2021-02-10T11:32:56.144 INFO:teuthology.orchestra.run.gibba013.stdout: "id": 7693, 2021-02-10T11:32:56.144 INFO:teuthology.orchestra.run.gibba013.stdout: "want_state": "up:active", 2021-02-10T11:32:56.144 INFO:teuthology.orchestra.run.gibba013.stdout: "state": "up:active", 2021-02-10T11:32:56.144 INFO:teuthology.orchestra.run.gibba013.stdout: "rank_uptime": 18.123036697, 2021-02-10T11:32:56.145 INFO:teuthology.orchestra.run.gibba013.stdout: "mdsmap_epoch": 99, 2021-02-10T11:32:56.145 INFO:teuthology.orchestra.run.gibba013.stdout: "osdmap_epoch": 120, 2021-02-10T11:32:56.145 INFO:teuthology.orchestra.run.gibba013.stdout: "osdmap_epoch_barrier": 120, 2021-02-10T11:32:56.145 INFO:teuthology.orchestra.run.gibba013.stdout: "uptime": 23.771702915999999 2021-02-10T11:32:56.145 INFO:teuthology.orchestra.run.gibba013.stdout:} 2021-02-10T11:32:56.148 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "68db30ea-dbce-4aef-a6df-f01c6fea0513", "whoami": 0, "id": 7693, "want_state": "up:active", "state": "up:active", "rank_uptime": 18.123036697, "mdsmap_epoch": 99, "osdmap_epoch": 120, "osdmap_epoch_barrier": 120, "uptime": 23.771702915999999 } 2021-02-10T11:32:56.148 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2021-02-10T11:32:56.149 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2021-02-10T11:32:56.149 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.1 at ubuntu@gibba002.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.1... 2021-02-10T11:32:56.149 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:32:56.149 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec mkdir -p /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:32:56.487 DEBUG:teuthology.orchestra.run.gibba002:> sudo modprobe fuse 2021-02-10T11:32:56.838 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:32:56.838 DEBUG:teuthology.orchestra.run.gibba002:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:32:57.189 INFO:teuthology.orchestra.run.gibba002.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:32:57.189 INFO:teuthology.orchestra.run.gibba002.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:32:57.190 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:32:57.190 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:32:57.190 DEBUG:teuthology.orchestra.run.gibba002:> ls /sys/fs/fuse/connections 2021-02-10T11:32:57.524 INFO:teuthology.orchestra.run.gibba002.stdout:43 2021-02-10T11:32:57.525 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [43] 2021-02-10T11:32:57.525 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.1 /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:32:57.528 DEBUG:teuthology.orchestra.run.gibba002:> sudo modprobe fuse 2021-02-10T11:32:57.872 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:32:57.873 DEBUG:teuthology.orchestra.run.gibba002:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:32:57.938 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba002.stderr:ceph-fuse[186462]: starting ceph client 2021-02-10T11:32:57.938 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba002.stderr:2021-02-10 11:32:57.936 7f49d69bcf80 -1 init, newargv = 0x559eb4f571a0 newargc=9 2021-02-10T11:32:57.943 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba002.stderr:ceph-fuse[186462]: starting fuse 2021-02-10T11:32:58.215 INFO:teuthology.orchestra.run.gibba002.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:32:58.215 INFO:teuthology.orchestra.run.gibba002.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:32:58.216 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:32:58.216 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:32:58.216 DEBUG:teuthology.orchestra.run.gibba002:> ls /sys/fs/fuse/connections 2021-02-10T11:32:58.551 INFO:teuthology.orchestra.run.gibba002.stdout:43 2021-02-10T11:32:58.551 INFO:teuthology.orchestra.run.gibba002.stdout:44 2021-02-10T11:32:58.552 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43, 44] 2021-02-10T11:32:58.552 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:32:58.552 DEBUG:teuthology.orchestra.run.gibba002:> sudo python3 -c ' 2021-02-10T11:32:58.552 DEBUG:teuthology.orchestra.run.gibba002:> import glob 2021-02-10T11:32:58.552 DEBUG:teuthology.orchestra.run.gibba002:> import re 2021-02-10T11:32:58.552 DEBUG:teuthology.orchestra.run.gibba002:> import os 2021-02-10T11:32:58.552 DEBUG:teuthology.orchestra.run.gibba002:> import subprocess 2021-02-10T11:32:58.552 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:32:58.552 DEBUG:teuthology.orchestra.run.gibba002:> def find_socket(client_name): 2021-02-10T11:32:58.552 DEBUG:teuthology.orchestra.run.gibba002:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2021-02-10T11:32:58.553 DEBUG:teuthology.orchestra.run.gibba002:> files = glob.glob(asok_path) 2021-02-10T11:32:58.553 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:32:58.553 DEBUG:teuthology.orchestra.run.gibba002:> # Given a non-glob path, it better be there 2021-02-10T11:32:58.553 DEBUG:teuthology.orchestra.run.gibba002:> if "*" not in asok_path: 2021-02-10T11:32:58.553 DEBUG:teuthology.orchestra.run.gibba002:> assert(len(files) == 1) 2021-02-10T11:32:58.553 DEBUG:teuthology.orchestra.run.gibba002:> return files[0] 2021-02-10T11:32:58.553 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:32:58.553 DEBUG:teuthology.orchestra.run.gibba002:> for f in files: 2021-02-10T11:32:58.553 DEBUG:teuthology.orchestra.run.gibba002:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2021-02-10T11:32:58.553 DEBUG:teuthology.orchestra.run.gibba002:> if os.path.exists("/proc/{0}".format(pid)): 2021-02-10T11:32:58.553 DEBUG:teuthology.orchestra.run.gibba002:> return f 2021-02-10T11:32:58.553 DEBUG:teuthology.orchestra.run.gibba002:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2021-02-10T11:32:58.553 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:32:58.553 DEBUG:teuthology.orchestra.run.gibba002:> print(find_socket("client.1")) 2021-02-10T11:32:58.554 DEBUG:teuthology.orchestra.run.gibba002:> ' 2021-02-10T11:32:58.916 INFO:teuthology.orchestra.run.gibba002.stdout:/var/run/ceph/ceph-client.1.186462.asok 2021-02-10T11:32:58.919 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.186462.asok 2021-02-10T11:32:58.919 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:32:58.920 DEBUG:teuthology.orchestra.run.gibba002:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.186462.asok status 2021-02-10T11:32:59.308 INFO:teuthology.orchestra.run.gibba002.stdout:{ 2021-02-10T11:32:59.308 INFO:teuthology.orchestra.run.gibba002.stdout: "metadata": { 2021-02-10T11:32:59.309 INFO:teuthology.orchestra.run.gibba002.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:32:59.309 INFO:teuthology.orchestra.run.gibba002.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:32:59.309 INFO:teuthology.orchestra.run.gibba002.stdout: "entity_id": "1", 2021-02-10T11:32:59.309 INFO:teuthology.orchestra.run.gibba002.stdout: "hostname": "gibba002", 2021-02-10T11:32:59.309 INFO:teuthology.orchestra.run.gibba002.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2021-02-10T11:32:59.309 INFO:teuthology.orchestra.run.gibba002.stdout: "pid": "186462", 2021-02-10T11:32:59.309 INFO:teuthology.orchestra.run.gibba002.stdout: "root": "/" 2021-02-10T11:32:59.309 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:32:59.309 INFO:teuthology.orchestra.run.gibba002.stdout: "dentry_count": 0, 2021-02-10T11:32:59.309 INFO:teuthology.orchestra.run.gibba002.stdout: "dentry_pinned_count": 0, 2021-02-10T11:32:59.309 INFO:teuthology.orchestra.run.gibba002.stdout: "id": 10938, 2021-02-10T11:32:59.309 INFO:teuthology.orchestra.run.gibba002.stdout: "inst": { 2021-02-10T11:32:59.310 INFO:teuthology.orchestra.run.gibba002.stdout: "name": { 2021-02-10T11:32:59.310 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "client", 2021-02-10T11:32:59.310 INFO:teuthology.orchestra.run.gibba002.stdout: "num": 10938 2021-02-10T11:32:59.310 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:32:59.310 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": { 2021-02-10T11:32:59.310 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:32:59.310 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:0", 2021-02-10T11:32:59.310 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 2703441897 2021-02-10T11:32:59.310 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:32:59.310 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:32:59.310 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": { 2021-02-10T11:32:59.310 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:32:59.310 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:0", 2021-02-10T11:32:59.311 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 2703441897 2021-02-10T11:32:59.311 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:32:59.311 INFO:teuthology.orchestra.run.gibba002.stdout: "inst_str": "client.10938 172.21.2.102:0/2703441897", 2021-02-10T11:32:59.311 INFO:teuthology.orchestra.run.gibba002.stdout: "addr_str": "172.21.2.102:0/2703441897", 2021-02-10T11:32:59.311 INFO:teuthology.orchestra.run.gibba002.stdout: "inode_count": 1, 2021-02-10T11:32:59.311 INFO:teuthology.orchestra.run.gibba002.stdout: "mds_epoch": 99, 2021-02-10T11:32:59.311 INFO:teuthology.orchestra.run.gibba002.stdout: "osd_epoch": 120, 2021-02-10T11:32:59.311 INFO:teuthology.orchestra.run.gibba002.stdout: "osd_epoch_barrier": 0, 2021-02-10T11:32:59.311 INFO:teuthology.orchestra.run.gibba002.stdout: "blacklisted": false 2021-02-10T11:32:59.311 INFO:teuthology.orchestra.run.gibba002.stdout:} 2021-02-10T11:32:59.313 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:32:59.313 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:32:59.313 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:32:59.650 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:32:59.650 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2021-02-10T11:32:59.651 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:32:59.651 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:32:59.997 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2021-02-10T11:33:00.429 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:00.411 7fb98f22b700 1 Processor -- start 2021-02-10T11:33:00.429 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:00.411 7fb98f22b700 1 -- start start 2021-02-10T11:33:00.430 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:00.411 7fb98f22b700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb9880d1880 0x7fb9881b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:00.430 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:00.411 7fb98f22b700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fb9881b0af0 0x7fb988108950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:00.430 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:00.411 7fb98f22b700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fb9880d1c80 0x7fb988108f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:00.430 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:00.411 7fb98f22b700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9880e5f00 con 0x7fb9881b0af0 2021-02-10T11:33:00.430 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:00.411 7fb98f22b700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb988109a70 con 0x7fb9880d1c80 2021-02-10T11:33:00.430 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:00.411 7fb98f22b700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb988088070 con 0x7fb9880d1880 2021-02-10T11:33:00.430 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:00.411 7fb98c846700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fb9881b0af0 0x7fb988108950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:00.430 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:00.411 7fb98c846700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fb9881b0af0 0x7fb988108950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:34930/0 (socket says 172.21.2.102:34930) 2021-02-10T11:33:00.430 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:00.411 7fb98c846700 1 -- 172.21.2.102:0/1361145181 learned_addr learned my addr 172.21.2.102:0/1361145181 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:33:00.430 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:00.411 7fb98d848700 1 --2- 172.21.2.102:0/1361145181 >> v2:172.21.2.113:3300/0 conn(0x7fb9880d1c80 0x7fb988108f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:00.430 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:00.412 7fb98d047700 1 --2- 172.21.2.102:0/1361145181 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb9880d1880 0x7fb9881b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:00.431 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:00.412 7fb98c846700 1 -- 172.21.2.102:0/1361145181 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb9880d1880 msgr2=0x7fb9881b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:33:00.431 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:00.412 7fb98c846700 1 --2- 172.21.2.102:0/1361145181 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb9880d1880 0x7fb9881b04d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:00.431 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:00.412 7fb98c846700 1 -- 172.21.2.102:0/1361145181 >> v2:172.21.2.113:3300/0 conn(0x7fb9880d1c80 msgr2=0x7fb988108f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:33:00.431 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:00.412 7fb98c846700 1 --2- 172.21.2.102:0/1361145181 >> v2:172.21.2.113:3300/0 conn(0x7fb9880d1c80 0x7fb988108f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:00.431 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:00.412 7fb98c846700 1 -- 172.21.2.102:0/1361145181 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb988109590 con 0x7fb9881b0af0 2021-02-10T11:33:00.431 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:00.412 7fb98c846700 1 --2- 172.21.2.102:0/1361145181 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb9881b0af0 0x7fb988108950 secure :-1 s=READY pgs=2350 cs=0 l=1 rev1=1 rx=0x7fb978006700 tx=0x7fb978009b80).ready entity=mon.0 client_cookie=60e53ed7516ddac8 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:00.431 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:00.412 7fb97ffff700 1 -- 172.21.2.102:0/1361145181 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb97800c0c0 con 0x7fb9881b0af0 2021-02-10T11:33:00.431 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:00.412 7fb97ffff700 1 -- 172.21.2.102:0/1361145181 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb97800c340 con 0x7fb9881b0af0 2021-02-10T11:33:00.431 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:00.412 7fb97ffff700 1 -- 172.21.2.102:0/1361145181 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb97800c860 con 0x7fb9881b0af0 2021-02-10T11:33:00.431 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:00.412 7fb98f22b700 1 -- 172.21.2.102:0/1361145181 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb9881b0af0 msgr2=0x7fb988108950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:00.431 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:00.412 7fb98f22b700 1 --2- 172.21.2.102:0/1361145181 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb9881b0af0 0x7fb988108950 secure :-1 s=READY pgs=2350 cs=0 l=1 rev1=1 rx=0x7fb978006700 tx=0x7fb978009b80).stop 2021-02-10T11:33:00.431 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:00.413 7fb98f22b700 1 -- 172.21.2.102:0/1361145181 shutdown_connections 2021-02-10T11:33:00.431 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:00.413 7fb98f22b700 1 --2- 172.21.2.102:0/1361145181 >> v2:172.21.2.113:3300/0 conn(0x7fb9880d1c80 0x7fb988108f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:00.431 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:00.413 7fb98f22b700 1 --2- 172.21.2.102:0/1361145181 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb9881b0af0 0x7fb988108950 unknown :-1 s=CLOSED pgs=2350 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:00.432 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:00.413 7fb98f22b700 1 --2- 172.21.2.102:0/1361145181 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb9880d1880 0x7fb9881b04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:00.432 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:00.413 7fb98f22b700 1 -- 172.21.2.102:0/1361145181 >> 172.21.2.102:0/1361145181 conn(0x7fb9880ce430 msgr2=0x7fb9880d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:00.432 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:00.413 7fb98f22b700 1 -- 172.21.2.102:0/1361145181 shutdown_connections 2021-02-10T11:33:00.432 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:00.413 7fb98f22b700 1 -- 172.21.2.102:0/1361145181 wait complete. 2021-02-10T11:33:00.432 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:00.428 7fb98f22b700 1 Processor -- start 2021-02-10T11:33:00.432 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:00.428 7fb98f22b700 1 -- start start 2021-02-10T11:33:00.434 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:00.428 7fb98f22b700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb9881a0580 0x7fb988082c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:00.434 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:00.428 7fb98f22b700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb9880832a0 0x7fb9880836a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:00.434 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:00.428 7fb98f22b700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb9880b66d0 0x7fb9880b6ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:00.434 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:00.428 7fb98f22b700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9880ba540 con 0x7fb9880b66d0 2021-02-10T11:33:00.434 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:00.428 7fb98f22b700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9880baac0 con 0x7fb9880832a0 2021-02-10T11:33:00.434 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:00.428 7fb98f22b700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9880b70f0 con 0x7fb9881a0580 2021-02-10T11:33:00.434 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:00.429 7fb98d848700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb9880b66d0 0x7fb9880b6ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:00.434 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:00.429 7fb98d848700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb9880b66d0 0x7fb9880b6ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:34936/0 (socket says 172.21.2.102:34936) 2021-02-10T11:33:00.434 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:00.429 7fb98d848700 1 -- 172.21.2.102:0/3242893347 learned_addr learned my addr 172.21.2.102:0/3242893347 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:33:00.434 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:00.429 7fb98d047700 1 --2- 172.21.2.102:0/3242893347 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb9881a0580 0x7fb988082c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:00.435 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:00.429 7fb98c846700 1 --2- 172.21.2.102:0/3242893347 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb9880832a0 0x7fb9880836a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:00.435 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:00.429 7fb98d848700 1 -- 172.21.2.102:0/3242893347 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb9881a0580 msgr2=0x7fb988082c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:00.435 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:00.429 7fb98d848700 1 --2- 172.21.2.102:0/3242893347 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb9881a0580 0x7fb988082c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:00.435 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:00.429 7fb98d848700 1 -- 172.21.2.102:0/3242893347 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb9880832a0 msgr2=0x7fb9880836a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:00.435 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:00.429 7fb98d848700 1 --2- 172.21.2.102:0/3242893347 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb9880832a0 0x7fb9880836a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:00.435 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:00.429 7fb98d848700 1 -- 172.21.2.102:0/3242893347 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9880b7310 con 0x7fb9880b66d0 2021-02-10T11:33:00.435 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:00.429 7fb98d047700 1 --2- 172.21.2.102:0/3242893347 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb9881a0580 0x7fb988082c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:33:00.435 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:00.429 7fb98d848700 1 --2- 172.21.2.102:0/3242893347 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb9880b66d0 0x7fb9880b6ad0 secure :-1 s=READY pgs=2351 cs=0 l=1 rev1=1 rx=0x7fb98000b970 tx=0x7fb98000bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:00.435 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:00.429 7fb97dffb700 1 -- 172.21.2.102:0/3242893347 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb980000f90 con 0x7fb9880b66d0 2021-02-10T11:33:00.435 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:00.430 7fb97dffb700 1 -- 172.21.2.102:0/3242893347 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb980006bb0 con 0x7fb9880b66d0 2021-02-10T11:33:00.435 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:00.430 7fb97dffb700 1 -- 172.21.2.102:0/3242893347 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb980001400 con 0x7fb9880b66d0 2021-02-10T11:33:00.435 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:00.430 7fb98f22b700 1 -- 172.21.2.102:0/3242893347 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9880b7310 con 0x7fb9880b66d0 2021-02-10T11:33:00.435 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:00.430 7fb98f22b700 1 -- 172.21.2.102:0/3242893347 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9880b7310 con 0x7fb9880b66d0 2021-02-10T11:33:00.435 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:00.430 7fb97dffb700 1 -- 172.21.2.102:0/3242893347 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fb9800011f0 con 0x7fb9880b66d0 2021-02-10T11:33:00.436 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:00.430 7fb97dffb700 1 --2- 172.21.2.102:0/3242893347 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb9680186c0 0x7fb96801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:00.436 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:00.430 7fb97dffb700 1 -- 172.21.2.102:0/3242893347 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fb9800328a0 con 0x7fb9880b66d0 2021-02-10T11:33:00.436 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:00.430 7fb98d047700 1 --2- 172.21.2.102:0/3242893347 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb9680186c0 0x7fb96801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:00.436 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:00.431 7fb98d047700 1 --2- 172.21.2.102:0/3242893347 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb9680186c0 0x7fb96801abc0 crc :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:00.463 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:00.461 7fb98f22b700 1 -- 172.21.2.102:0/3242893347 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb988082a50 con 0x7fb9880b66d0 2021-02-10T11:33:00.465 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:00.464 7fb97dffb700 1 -- 172.21.2.102:0/3242893347 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fb9800066f0 con 0x7fb9880b66d0 2021-02-10T11:33:00.601 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:00.600 7fb98f22b700 1 -- 172.21.2.102:0/3242893347 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fb988082a50 con 0x7fb9880b66d0 2021-02-10T11:33:01.481 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:01.479 7fb97dffb700 1 -- 172.21.2.102:0/3242893347 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v100) v1 ==== 136+0+0 (secure 0 0 0) 0x7fb9800011f0 con 0x7fb9880b66d0 2021-02-10T11:33:01.524 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:01.522 7fb98f22b700 1 -- 172.21.2.102:0/3242893347 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb9680186c0 msgr2=0x7fb96801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:01.524 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:01.522 7fb98f22b700 1 --2- 172.21.2.102:0/3242893347 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb9680186c0 0x7fb96801abc0 crc :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:01.524 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:01.522 7fb98f22b700 1 -- 172.21.2.102:0/3242893347 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb9880b66d0 msgr2=0x7fb9880b6ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:01.524 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:01.522 7fb98f22b700 1 --2- 172.21.2.102:0/3242893347 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb9880b66d0 0x7fb9880b6ad0 secure :-1 s=READY pgs=2351 cs=0 l=1 rev1=1 rx=0x7fb98000b970 tx=0x7fb98000bd10).stop 2021-02-10T11:33:01.524 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:01.523 7fb98f22b700 1 -- 172.21.2.102:0/3242893347 shutdown_connections 2021-02-10T11:33:01.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:01.523 7fb98f22b700 1 --2- 172.21.2.102:0/3242893347 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb9880b66d0 0x7fb9880b6ad0 unknown :-1 s=CLOSED pgs=2351 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:01.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:01.523 7fb98f22b700 1 --2- 172.21.2.102:0/3242893347 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb9680186c0 0x7fb96801abc0 unknown :-1 s=CLOSED pgs=1033 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:01.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:01.523 7fb98f22b700 1 --2- 172.21.2.102:0/3242893347 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb9880832a0 0x7fb9880836a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:01.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:01.523 7fb98f22b700 1 --2- 172.21.2.102:0/3242893347 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb9881a0580 0x7fb988082c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:01.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:01.523 7fb98f22b700 1 -- 172.21.2.102:0/3242893347 >> 172.21.2.102:0/3242893347 conn(0x7fb9880c20f0 msgr2=0x7fb98809e410 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:01.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:01.523 7fb98f22b700 1 -- 172.21.2.102:0/3242893347 shutdown_connections 2021-02-10T11:33:01.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:01.523 7fb98f22b700 1 -- 172.21.2.102:0/3242893347 wait complete. 2021-02-10T11:33:01.532 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs volume ls 2021-02-10T11:33:01.961 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:01.942 7f5dbf2c1700 1 Processor -- start 2021-02-10T11:33:01.961 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:01.942 7f5dbf2c1700 1 -- start start 2021-02-10T11:33:01.962 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:01.942 7f5dbf2c1700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f5db80d0340 0x7f5db80ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:01.962 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:01.942 7f5dbf2c1700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5db80d1c70 0x7f5db80efc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:01.962 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:01.942 7f5dbf2c1700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5db80f0b10 con 0x7f5db80d1870 2021-02-10T11:33:01.962 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:01.942 7f5dbf2c1700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5db80e5ed0 con 0x7f5db80d0340 2021-02-10T11:33:01.962 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:01.942 7f5dbf2c1700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5db81b0be0 con 0x7f5db80d1c70 2021-02-10T11:33:01.962 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:01.943 7f5dbd8de700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5db80d1c70 0x7f5db80efc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:01.962 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:01.943 7f5dbc8dc700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f5db80d0340 0x7f5db80ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:01.962 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:01.943 7f5dbd0dd700 1 --1- >> v1:172.21.2.113:6789/0 conn(0x7f5db80d1870 0x7f5db81b03f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.113:6789/0 says I am v1:172.21.2.102:34490/0 (socket says 172.21.2.102:34490) 2021-02-10T11:33:01.962 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:01.943 7f5dbd0dd700 1 -- 172.21.2.102:0/735332145 learned_addr learned my addr 172.21.2.102:0/735332145 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:33:01.962 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:01.944 7f5dbd8de700 1 -- 172.21.2.102:0/735332145 >> v2:172.21.2.113:3300/0 conn(0x7f5db80d0340 msgr2=0x7f5db80ef640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:33:01.962 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:01.944 7f5dbd8de700 1 --2- 172.21.2.102:0/735332145 >> v2:172.21.2.113:3300/0 conn(0x7f5db80d0340 0x7f5db80ef640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:01.963 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:01.944 7f5dbd8de700 1 -- 172.21.2.102:0/735332145 >> v1:172.21.2.113:6789/0 conn(0x7f5db80d1870 legacy=0x7f5db81b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:01.963 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:01.944 7f5dbd8de700 1 -- 172.21.2.102:0/735332145 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5db80f20a0 con 0x7f5db80d1c70 2021-02-10T11:33:01.963 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:01.944 7f5dbd8de700 1 --2- 172.21.2.102:0/735332145 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5db80d1c70 0x7f5db80efc60 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7f5db0006e50 tx=0x7f5db0009e70).ready entity=mon.2 client_cookie=3e8b97be0300da60 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:01.963 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:01.944 7f5daffff700 1 -- 172.21.2.102:0/735332145 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5db000b510 con 0x7f5db80d1c70 2021-02-10T11:33:01.963 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:01.945 7f5daffff700 1 -- 172.21.2.102:0/735332145 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5db0006790 con 0x7f5db80d1c70 2021-02-10T11:33:01.963 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:01.945 7f5daffff700 1 -- 172.21.2.102:0/735332145 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5db000b940 con 0x7f5db80d1c70 2021-02-10T11:33:01.963 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:01.945 7f5dbf2c1700 1 -- 172.21.2.102:0/735332145 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5db80d1c70 msgr2=0x7f5db80efc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:01.963 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:01.945 7f5dbf2c1700 1 --2- 172.21.2.102:0/735332145 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5db80d1c70 0x7f5db80efc60 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7f5db0006e50 tx=0x7f5db0009e70).stop 2021-02-10T11:33:01.963 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:01.945 7f5dbf2c1700 1 -- 172.21.2.102:0/735332145 shutdown_connections 2021-02-10T11:33:01.963 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:01.945 7f5dbf2c1700 1 --2- 172.21.2.102:0/735332145 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5db80d1c70 0x7f5db80efc60 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:01.963 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:01.945 7f5dbf2c1700 1 --2- 172.21.2.102:0/735332145 >> v2:172.21.2.113:3300/0 conn(0x7f5db80d0340 0x7f5db80ef640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:01.963 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:01.945 7f5dbf2c1700 1 -- 172.21.2.102:0/735332145 >> 172.21.2.102:0/735332145 conn(0x7f5db80ce450 msgr2=0x7f5db80d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:01.963 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:01.945 7f5dbf2c1700 1 -- 172.21.2.102:0/735332145 shutdown_connections 2021-02-10T11:33:01.964 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:01.946 7f5dbf2c1700 1 -- 172.21.2.102:0/735332145 wait complete. 2021-02-10T11:33:01.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:01.960 7f5dbf2c1700 1 Processor -- start 2021-02-10T11:33:01.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:01.960 7f5dbf2c1700 1 -- start start 2021-02-10T11:33:01.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:01.961 7f5dbf2c1700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5db81a06a0 0x7f5db809f380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:01.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:01.961 7f5dbf2c1700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5db809f9a0 0x7f5db808b560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:01.966 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:01.961 7f5dbf2c1700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5db808bb80 0x7f5db808bf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:01.966 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:01.961 7f5dbf2c1700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5db808c780 con 0x7f5db81a06a0 2021-02-10T11:33:01.966 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:01.961 7f5dbf2c1700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5db809fda0 con 0x7f5db808bb80 2021-02-10T11:33:01.966 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:01.961 7f5dbf2c1700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5db808cd00 con 0x7f5db809f9a0 2021-02-10T11:33:01.966 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:01.961 7f5dbd0dd700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5db81a06a0 0x7f5db809f380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:01.966 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:01.961 7f5dbd0dd700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5db81a06a0 0x7f5db809f380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:34948/0 (socket says 172.21.2.102:34948) 2021-02-10T11:33:01.966 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:01.961 7f5dbd0dd700 1 -- 172.21.2.102:0/3264649110 learned_addr learned my addr 172.21.2.102:0/3264649110 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:33:01.966 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:01.961 7f5dbd8de700 1 --2- 172.21.2.102:0/3264649110 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5db808bb80 0x7f5db808bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:01.966 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:01.961 7f5dbc8dc700 1 --2- 172.21.2.102:0/3264649110 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5db809f9a0 0x7f5db808b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:01.966 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:01.962 7f5dbd8de700 1 -- 172.21.2.102:0/3264649110 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5db809f9a0 msgr2=0x7f5db808b560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:01.966 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:01.962 7f5dbd8de700 1 --2- 172.21.2.102:0/3264649110 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5db809f9a0 0x7f5db808b560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:01.966 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:01.962 7f5dbd8de700 1 -- 172.21.2.102:0/3264649110 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5db81a06a0 msgr2=0x7f5db809f380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:01.967 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:01.962 7f5dbd8de700 1 --2- 172.21.2.102:0/3264649110 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5db81a06a0 0x7f5db809f380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:01.967 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:01.962 7f5dbd8de700 1 -- 172.21.2.102:0/3264649110 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5db808cf20 con 0x7f5db808bb80 2021-02-10T11:33:01.967 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:01.962 7f5dbd0dd700 1 --2- 172.21.2.102:0/3264649110 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5db81a06a0 0x7f5db809f380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:33:01.967 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:01.962 7f5dbd8de700 1 --2- 172.21.2.102:0/3264649110 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5db808bb80 0x7f5db808bf80 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 rx=0x7f5db0009890 tx=0x7f5db000bc20).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:01.967 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:01.962 7f5dadffb700 1 -- 172.21.2.102:0/3264649110 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5db0006ce0 con 0x7f5db808bb80 2021-02-10T11:33:01.967 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:01.962 7f5dadffb700 1 -- 172.21.2.102:0/3264649110 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5db000bd60 con 0x7f5db808bb80 2021-02-10T11:33:01.967 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:01.962 7f5dadffb700 1 -- 172.21.2.102:0/3264649110 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5db00022b0 con 0x7f5db808bb80 2021-02-10T11:33:01.967 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:01.962 7f5dbf2c1700 1 -- 172.21.2.102:0/3264649110 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5db808cf20 con 0x7f5db808bb80 2021-02-10T11:33:01.967 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:01.962 7f5dbf2c1700 1 -- 172.21.2.102:0/3264649110 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5db808cf20 con 0x7f5db808bb80 2021-02-10T11:33:01.967 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:01.963 7f5dadffb700 1 -- 172.21.2.102:0/3264649110 <== mon.1 v2:172.21.2.113:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f5db00056b0 con 0x7f5db808bb80 2021-02-10T11:33:01.967 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:01.963 7f5dadffb700 1 --2- 172.21.2.102:0/3264649110 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5d980186c0 0x7f5d9801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:01.967 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:01.963 7f5dadffb700 1 -- 172.21.2.102:0/3264649110 <== mon.1 v2:172.21.2.113:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f5db0032860 con 0x7f5db808bb80 2021-02-10T11:33:01.967 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:01.963 7f5dbd0dd700 1 --2- 172.21.2.102:0/3264649110 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5d980186c0 0x7f5d9801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:01.967 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:01.963 7f5dbd0dd700 1 --2- 172.21.2.102:0/3264649110 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5d980186c0 0x7f5d9801abc0 crc :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:01.994 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:01.992 7f5dbf2c1700 1 -- 172.21.2.102:0/3264649110 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5db80ebc50 con 0x7f5db808bb80 2021-02-10T11:33:01.996 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:01.995 7f5dadffb700 1 -- 172.21.2.102:0/3264649110 <== mon.1 v2:172.21.2.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f5db002cc10 con 0x7f5db808bb80 2021-02-10T11:33:02.132 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:02.131 7f5dbf2c1700 1 -- 172.21.2.102:0/3264649110 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs volume ls", "target": ["mgr", ""]}) v1 -- 0x7f5db80ebc50 con 0x7f5d980186c0 2021-02-10T11:33:02.133 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:02.131 7f5dadffb700 1 -- 172.21.2.102:0/3264649110 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+41 (crc 0 0 0) 0x7f5db4000c10 con 0x7f5d980186c0 2021-02-10T11:33:02.133 INFO:teuthology.orchestra.run.gibba002.stdout:[ 2021-02-10T11:33:02.133 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:33:02.133 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "cephfs" 2021-02-10T11:33:02.133 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:33:02.133 INFO:teuthology.orchestra.run.gibba002.stdout:] 2021-02-10T11:33:02.134 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:02.133 7f5dbf2c1700 1 -- 172.21.2.102:0/3264649110 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5d980186c0 msgr2=0x7f5d9801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:02.134 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:02.133 7f5dbf2c1700 1 --2- 172.21.2.102:0/3264649110 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5d980186c0 0x7f5d9801abc0 crc :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:02.135 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:02.133 7f5dbf2c1700 1 -- 172.21.2.102:0/3264649110 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5db808bb80 msgr2=0x7f5db808bf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:02.135 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:02.133 7f5dbf2c1700 1 --2- 172.21.2.102:0/3264649110 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5db808bb80 0x7f5db808bf80 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 rx=0x7f5db0009890 tx=0x7f5db000bc20).stop 2021-02-10T11:33:02.135 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:02.133 7f5dbf2c1700 1 -- 172.21.2.102:0/3264649110 shutdown_connections 2021-02-10T11:33:02.135 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:02.133 7f5dbf2c1700 1 --2- 172.21.2.102:0/3264649110 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5d980186c0 0x7f5d9801abc0 unknown :-1 s=CLOSED pgs=1034 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:02.135 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:02.133 7f5dbf2c1700 1 --2- 172.21.2.102:0/3264649110 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5db808bb80 0x7f5db808bf80 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:02.136 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:02.133 7f5dbf2c1700 1 --2- 172.21.2.102:0/3264649110 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5db809f9a0 0x7f5db808b560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:02.136 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:02.133 7f5dbf2c1700 1 --2- 172.21.2.102:0/3264649110 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5db81a06a0 0x7f5db809f380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:02.136 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:02.133 7f5dbf2c1700 1 -- 172.21.2.102:0/3264649110 >> 172.21.2.102:0/3264649110 conn(0x7f5db80b83b0 msgr2=0x7f5db80a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:02.136 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:02.133 7f5dbf2c1700 1 -- 172.21.2.102:0/3264649110 shutdown_connections 2021-02-10T11:33:02.136 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:02.133 7f5dbf2c1700 1 -- 172.21.2.102:0/3264649110 wait complete. 2021-02-10T11:33:02.143 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config set mon mon_allow_pool_delete True 2021-02-10T11:33:02.577 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:02.557 7fb8c3e5b700 1 Processor -- start 2021-02-10T11:33:02.577 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:02.557 7fb8c3e5b700 1 -- start start 2021-02-10T11:33:02.577 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:02.558 7fb8c3e5b700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb8bc10aff0 0x7fb8bc0cf750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:02.577 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:02.558 7fb8c3e5b700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8bc0fe1c0 con 0x7fb8bc0d1870 2021-02-10T11:33:02.577 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:02.558 7fb8c3e5b700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8bc1b0cb0 con 0x7fb8bc0d1c70 2021-02-10T11:33:02.578 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:02.558 7fb8c3e5b700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8bc109a60 con 0x7fb8bc10aff0 2021-02-10T11:33:02.578 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:02.558 7fb8c1476700 1 --1- >> v1:172.21.2.113:6789/0 conn(0x7fb8bc0d1c70 0x7fb8bc0cef60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.113:6789/0 says I am v1:172.21.2.102:34504/0 (socket says 172.21.2.102:34504) 2021-02-10T11:33:02.578 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:02.558 7fb8c1c77700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fb8bc0d1870 0x7fb8bc1b04c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:55412/0 (socket says 172.21.2.102:55412) 2021-02-10T11:33:02.578 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:02.558 7fb8c1476700 1 -- 172.21.2.102:0/3864501640 learned_addr learned my addr 172.21.2.102:0/3864501640 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:33:02.578 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:02.558 7fb8c2478700 1 --2- 172.21.2.102:0/3864501640 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb8bc10aff0 0x7fb8bc0cf750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:02.578 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:02.559 7fb8c0c75700 1 -- 172.21.2.102:0/3864501640 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 645393810 0 0) 0x7fb8b8000ce0 con 0x7fb8bc0d1870 2021-02-10T11:33:02.578 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:02.559 7fb8c0c75700 1 -- 172.21.2.102:0/3864501640 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8a8002fe0 con 0x7fb8bc0d1870 2021-02-10T11:33:02.579 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:02.559 7fb8c0c75700 1 -- 172.21.2.102:0/3864501640 <== mon.1 v1:172.21.2.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2352263437 0 0) 0x7fb8b0001190 con 0x7fb8bc0d1c70 2021-02-10T11:33:02.579 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:02.559 7fb8c2478700 1 -- 172.21.2.102:0/3864501640 >> v1:172.21.2.113:6789/0 conn(0x7fb8bc0d1c70 legacy=0x7fb8bc0cef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:02.579 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:02.559 7fb8c2478700 1 -- 172.21.2.102:0/3864501640 >> v1:172.21.2.102:6789/0 conn(0x7fb8bc0d1870 legacy=0x7fb8bc1b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:02.579 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:02.559 7fb8c2478700 1 -- 172.21.2.102:0/3864501640 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8bc10b3f0 con 0x7fb8bc10aff0 2021-02-10T11:33:02.579 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:02.559 7fb8c2478700 1 --2- 172.21.2.102:0/3864501640 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb8bc10aff0 0x7fb8bc0cf750 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 rx=0x7fb8b4006e50 tx=0x7fb8b4009e70).ready entity=mon.2 client_cookie=c5c09b54392c3d9d server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:02.579 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:02.559 7fb8c0c75700 1 -- 172.21.2.102:0/3864501640 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb8b400b510 con 0x7fb8bc10aff0 2021-02-10T11:33:02.579 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:02.559 7fb8c0c75700 1 -- 172.21.2.102:0/3864501640 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8b4006790 con 0x7fb8bc10aff0 2021-02-10T11:33:02.579 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:02.559 7fb8c0c75700 1 -- 172.21.2.102:0/3864501640 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb8b400b940 con 0x7fb8bc10aff0 2021-02-10T11:33:02.579 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:02.559 7fb8c3e5b700 1 -- 172.21.2.102:0/3864501640 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb8bc10aff0 msgr2=0x7fb8bc0cf750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:02.579 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:02.559 7fb8c3e5b700 1 --2- 172.21.2.102:0/3864501640 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb8bc10aff0 0x7fb8bc0cf750 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 rx=0x7fb8b4006e50 tx=0x7fb8b4009e70).stop 2021-02-10T11:33:02.579 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:02.560 7fb8c3e5b700 1 -- 172.21.2.102:0/3864501640 shutdown_connections 2021-02-10T11:33:02.579 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:02.560 7fb8c3e5b700 1 --2- 172.21.2.102:0/3864501640 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb8bc10aff0 0x7fb8bc0cf750 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:02.579 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:02.560 7fb8c3e5b700 1 -- 172.21.2.102:0/3864501640 >> 172.21.2.102:0/3864501640 conn(0x7fb8bc0ce420 msgr2=0x7fb8bc0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:02.580 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:02.560 7fb8c3e5b700 1 -- 172.21.2.102:0/3864501640 shutdown_connections 2021-02-10T11:33:02.580 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:02.560 7fb8c3e5b700 1 -- 172.21.2.102:0/3864501640 wait complete. 2021-02-10T11:33:02.580 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:02.575 7fb8c3e5b700 1 Processor -- start 2021-02-10T11:33:02.580 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:02.575 7fb8c3e5b700 1 -- start start 2021-02-10T11:33:02.580 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:02.575 7fb8c3e5b700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb8bc09bcc0 0x7fb8bc09c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:02.580 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:02.575 7fb8c3e5b700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb8bc0a0820 0x7fb8bc0a0c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:02.580 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:02.575 7fb8c3e5b700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb8bc0a1240 0x7fb8bc0858d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:02.580 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:02.575 7fb8c3e5b700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8bc0a66b0 con 0x7fb8bc0a1240 2021-02-10T11:33:02.580 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:02.575 7fb8c3e5b700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8bc1a49b0 con 0x7fb8bc09bcc0 2021-02-10T11:33:02.580 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:02.575 7fb8c3e5b700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8bc1a4b90 con 0x7fb8bc0a0820 2021-02-10T11:33:02.580 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:02.576 7fb8c1c77700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb8bc09bcc0 0x7fb8bc09c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:02.580 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:02.576 7fb8c1c77700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb8bc09bcc0 0x7fb8bc09c0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.113:3300/0 says I am v2:172.21.2.102:38394/0 (socket says 172.21.2.102:38394) 2021-02-10T11:33:02.580 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:02.576 7fb8c1c77700 1 -- 172.21.2.102:0/605166002 learned_addr learned my addr 172.21.2.102:0/605166002 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:33:02.580 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:02.576 7fb8c2478700 1 --2- 172.21.2.102:0/605166002 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb8bc0a1240 0x7fb8bc0858d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:02.581 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:02.576 7fb8c1476700 1 --2- 172.21.2.102:0/605166002 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb8bc0a0820 0x7fb8bc0a0c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:02.582 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:02.576 7fb8c1c77700 1 -- 172.21.2.102:0/605166002 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb8bc0a0820 msgr2=0x7fb8bc0a0c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:02.582 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:02.576 7fb8c1c77700 1 --2- 172.21.2.102:0/605166002 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb8bc0a0820 0x7fb8bc0a0c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:02.582 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:02.576 7fb8c1c77700 1 -- 172.21.2.102:0/605166002 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb8bc0a1240 msgr2=0x7fb8bc0858d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:02.582 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:02.576 7fb8c1c77700 1 --2- 172.21.2.102:0/605166002 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb8bc0a1240 0x7fb8bc0858d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:02.582 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:02.576 7fb8c1c77700 1 -- 172.21.2.102:0/605166002 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8bc0a1640 con 0x7fb8bc09bcc0 2021-02-10T11:33:02.582 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:02.576 7fb8c2478700 1 --2- 172.21.2.102:0/605166002 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb8bc0a1240 0x7fb8bc0858d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:33:02.582 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:02.576 7fb8c1476700 1 --2- 172.21.2.102:0/605166002 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb8bc0a0820 0x7fb8bc0a0c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:33:02.582 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:02.576 7fb8c1c77700 1 --2- 172.21.2.102:0/605166002 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb8bc09bcc0 0x7fb8bc09c0c0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 rx=0x7fb8b800b5c0 tx=0x7fb8b800b960).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:02.582 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:02.576 7fb8ae7fc700 1 -- 172.21.2.102:0/605166002 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb8b800bd50 con 0x7fb8bc09bcc0 2021-02-10T11:33:02.582 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:02.576 7fb8ae7fc700 1 -- 172.21.2.102:0/605166002 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8b8006280 con 0x7fb8bc09bcc0 2021-02-10T11:33:02.582 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:02.576 7fb8c3e5b700 1 -- 172.21.2.102:0/605166002 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8bc0a1640 con 0x7fb8bc09bcc0 2021-02-10T11:33:02.583 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:02.576 7fb8ae7fc700 1 -- 172.21.2.102:0/605166002 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb8b8004620 con 0x7fb8bc09bcc0 2021-02-10T11:33:02.583 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:02.576 7fb8c3e5b700 1 -- 172.21.2.102:0/605166002 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8bc0a1640 con 0x7fb8bc09bcc0 2021-02-10T11:33:02.583 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:02.577 7fb8ae7fc700 1 -- 172.21.2.102:0/605166002 <== mon.1 v2:172.21.2.113:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fb8b80043a0 con 0x7fb8bc09bcc0 2021-02-10T11:33:02.583 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:02.577 7fb8ae7fc700 1 --2- 172.21.2.102:0/605166002 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb8a4018730 0x7fb8a401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:02.583 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:02.577 7fb8ae7fc700 1 -- 172.21.2.102:0/605166002 <== mon.1 v2:172.21.2.113:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fb8b80327a0 con 0x7fb8bc09bcc0 2021-02-10T11:33:02.583 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:02.577 7fb8c1476700 1 --2- 172.21.2.102:0/605166002 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb8a4018730 0x7fb8a401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:02.583 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:02.577 7fb8c1476700 1 --2- 172.21.2.102:0/605166002 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb8a4018730 0x7fb8a401ac10 crc :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:02.610 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:02.608 7fb8c3e5b700 1 -- 172.21.2.102:0/605166002 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8bc0fe1c0 con 0x7fb8bc09bcc0 2021-02-10T11:33:02.612 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:02.611 7fb8ae7fc700 1 -- 172.21.2.102:0/605166002 <== mon.1 v2:172.21.2.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fb8b8006070 con 0x7fb8bc09bcc0 2021-02-10T11:33:02.765 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:02.763 7fb8c3e5b700 1 -- 172.21.2.102:0/605166002 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fb8bc0fe1c0 con 0x7fb8bc09bcc0 2021-02-10T11:33:02.765 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:02.764 7fb8ae7fc700 1 -- 172.21.2.102:0/605166002 <== mon.1 v2:172.21.2.113:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v30)=0 v30) v1 ==== 122+0+0 (secure 0 0 0) 0x7fb8b80043a0 con 0x7fb8bc09bcc0 2021-02-10T11:33:02.767 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:02.766 7fb8c3e5b700 1 -- 172.21.2.102:0/605166002 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb8a4018730 msgr2=0x7fb8a401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:02.767 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:02.766 7fb8c3e5b700 1 --2- 172.21.2.102:0/605166002 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb8a4018730 0x7fb8a401ac10 crc :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:02.767 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:02.766 7fb8c3e5b700 1 -- 172.21.2.102:0/605166002 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb8bc09bcc0 msgr2=0x7fb8bc09c0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:02.767 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:02.766 7fb8c3e5b700 1 --2- 172.21.2.102:0/605166002 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb8bc09bcc0 0x7fb8bc09c0c0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 rx=0x7fb8b800b5c0 tx=0x7fb8b800b960).stop 2021-02-10T11:33:02.768 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:02.766 7fb8c3e5b700 1 -- 172.21.2.102:0/605166002 shutdown_connections 2021-02-10T11:33:02.768 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:02.766 7fb8c3e5b700 1 --2- 172.21.2.102:0/605166002 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb8bc0a1240 0x7fb8bc0858d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:02.768 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:02.766 7fb8c3e5b700 1 --2- 172.21.2.102:0/605166002 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb8bc0a0820 0x7fb8bc0a0c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:02.768 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:02.766 7fb8c3e5b700 1 --2- 172.21.2.102:0/605166002 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb8a4018730 0x7fb8a401ac10 unknown :-1 s=CLOSED pgs=1035 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:02.768 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:02.766 7fb8c3e5b700 1 --2- 172.21.2.102:0/605166002 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb8bc09bcc0 0x7fb8bc09c0c0 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:02.768 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:02.766 7fb8c3e5b700 1 -- 172.21.2.102:0/605166002 >> 172.21.2.102:0/605166002 conn(0x7fb8bc0bef00 msgr2=0x7fb8bc09d1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:02.768 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:02.766 7fb8c3e5b700 1 -- 172.21.2.102:0/605166002 shutdown_connections 2021-02-10T11:33:02.768 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:02.766 7fb8c3e5b700 1 -- 172.21.2.102:0/605166002 wait complete. 2021-02-10T11:33:02.776 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolumegroup create cephfs group_0000000000668580 2021-02-10T11:33:03.208 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:03.190 7fd9aa25c700 1 Processor -- start 2021-02-10T11:33:03.208 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:03.190 7fd9aa25c700 1 -- start start 2021-02-10T11:33:03.208 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:03.190 7fd9aa25c700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd9a40d1870 0x7fd9a41b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:03.208 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:03.190 7fd9aa25c700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fd9a41b0ae0 0x7fd9a4108940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:03.208 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:03.190 7fd9aa25c700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fd9a40d1c70 0x7fd9a4108f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:03.208 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:03.190 7fd9aa25c700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9a40e5ef0 con 0x7fd9a40d1c70 2021-02-10T11:33:03.209 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:03.190 7fd9aa25c700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9a4109a60 con 0x7fd9a41b0ae0 2021-02-10T11:33:03.209 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:03.190 7fd9aa25c700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9a4088070 con 0x7fd9a40d1870 2021-02-10T11:33:03.209 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:03.191 7fd9a8879700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fd9a40d1c70 0x7fd9a4108f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:03.209 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:03.191 7fd9a8879700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fd9a40d1c70 0x7fd9a4108f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:34970/0 (socket says 172.21.2.102:34970) 2021-02-10T11:33:03.209 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:03.191 7fd9a8879700 1 -- 172.21.2.102:0/1237894405 learned_addr learned my addr 172.21.2.102:0/1237894405 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:33:03.209 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:03.191 7fd9a3fff700 1 --2- 172.21.2.102:0/1237894405 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd9a40d1870 0x7fd9a41b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:03.209 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:03.191 7fd9a37fe700 1 --2- 172.21.2.102:0/1237894405 >> v2:172.21.2.113:3300/0 conn(0x7fd9a41b0ae0 0x7fd9a4108940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:03.209 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:03.191 7fd9a8879700 1 -- 172.21.2.102:0/1237894405 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd9a40d1870 msgr2=0x7fd9a41b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:33:03.209 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:03.191 7fd9a8879700 1 --2- 172.21.2.102:0/1237894405 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd9a40d1870 0x7fd9a41b04c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:03.209 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:03.191 7fd9a8879700 1 -- 172.21.2.102:0/1237894405 >> v2:172.21.2.113:3300/0 conn(0x7fd9a41b0ae0 msgr2=0x7fd9a4108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:33:03.209 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:03.191 7fd9a8879700 1 --2- 172.21.2.102:0/1237894405 >> v2:172.21.2.113:3300/0 conn(0x7fd9a41b0ae0 0x7fd9a4108940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:03.209 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:03.191 7fd9a8879700 1 -- 172.21.2.102:0/1237894405 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9a4109580 con 0x7fd9a40d1c70 2021-02-10T11:33:03.209 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:03.191 7fd9a37fe700 1 --2- 172.21.2.102:0/1237894405 >> v2:172.21.2.113:3300/0 conn(0x7fd9a41b0ae0 0x7fd9a4108940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:33:03.210 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:03.191 7fd9a8879700 1 --2- 172.21.2.102:0/1237894405 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd9a40d1c70 0x7fd9a4108f60 secure :-1 s=READY pgs=2353 cs=0 l=1 rev1=1 rx=0x7fd99c006f00 tx=0x7fd99c009f10).ready entity=mon.0 client_cookie=ee01725b09f36593 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:03.210 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:03.191 7fd9a2ffd700 1 -- 172.21.2.102:0/1237894405 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd99c00b3b0 con 0x7fd9a40d1c70 2021-02-10T11:33:03.210 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:03.192 7fd9a2ffd700 1 -- 172.21.2.102:0/1237894405 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd99c00b730 con 0x7fd9a40d1c70 2021-02-10T11:33:03.210 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:03.192 7fd9a2ffd700 1 -- 172.21.2.102:0/1237894405 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd99c00bc50 con 0x7fd9a40d1c70 2021-02-10T11:33:03.210 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:03.192 7fd9aa25c700 1 -- 172.21.2.102:0/1237894405 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd9a40d1c70 msgr2=0x7fd9a4108f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:03.210 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:03.192 7fd9aa25c700 1 --2- 172.21.2.102:0/1237894405 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd9a40d1c70 0x7fd9a4108f60 secure :-1 s=READY pgs=2353 cs=0 l=1 rev1=1 rx=0x7fd99c006f00 tx=0x7fd99c009f10).stop 2021-02-10T11:33:03.210 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:03.192 7fd9aa25c700 1 -- 172.21.2.102:0/1237894405 shutdown_connections 2021-02-10T11:33:03.210 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:03.192 7fd9aa25c700 1 --2- 172.21.2.102:0/1237894405 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd9a40d1c70 0x7fd9a4108f60 unknown :-1 s=CLOSED pgs=2353 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:03.210 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:03.192 7fd9aa25c700 1 --2- 172.21.2.102:0/1237894405 >> v2:172.21.2.113:3300/0 conn(0x7fd9a41b0ae0 0x7fd9a4108940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:03.210 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:03.192 7fd9aa25c700 1 --2- 172.21.2.102:0/1237894405 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd9a40d1870 0x7fd9a41b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:03.210 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:03.192 7fd9aa25c700 1 -- 172.21.2.102:0/1237894405 >> 172.21.2.102:0/1237894405 conn(0x7fd9a40ce420 msgr2=0x7fd9a40d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:03.210 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:03.192 7fd9aa25c700 1 -- 172.21.2.102:0/1237894405 shutdown_connections 2021-02-10T11:33:03.211 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:03.192 7fd9aa25c700 1 -- 172.21.2.102:0/1237894405 wait complete. 2021-02-10T11:33:03.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:03.207 7fd9aa25c700 1 Processor -- start 2021-02-10T11:33:03.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:03.207 7fd9aa25c700 1 -- start start 2021-02-10T11:33:03.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:03.207 7fd9aa25c700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd9a40abd60 0x7fd9a40ac160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:03.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:03.207 7fd9aa25c700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd9a40ac780 0x7fd9a40acb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:03.213 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:03.207 7fd9aa25c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd9a4083910 0x7fd9a4083d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:03.213 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:03.207 7fd9aa25c700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9a4087180 con 0x7fd9a4083910 2021-02-10T11:33:03.213 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:03.207 7fd9aa25c700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9a418fcd0 con 0x7fd9a40ac780 2021-02-10T11:33:03.213 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:03.207 7fd9aa25c700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9a4084330 con 0x7fd9a40abd60 2021-02-10T11:33:03.213 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:03.208 7fd9a8879700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd9a4083910 0x7fd9a4083d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:03.213 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:03.208 7fd9a8879700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd9a4083910 0x7fd9a4083d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:34980/0 (socket says 172.21.2.102:34980) 2021-02-10T11:33:03.213 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:03.208 7fd9a37fe700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd9a40ac780 0x7fd9a40acb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:03.213 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:03.208 7fd9a8879700 1 -- 172.21.2.102:0/477263868 learned_addr learned my addr 172.21.2.102:0/477263868 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:33:03.213 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:03.208 7fd9a3fff700 1 --2- 172.21.2.102:0/477263868 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd9a40abd60 0x7fd9a40ac160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:03.213 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:03.208 7fd9a8879700 1 -- 172.21.2.102:0/477263868 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd9a40abd60 msgr2=0x7fd9a40ac160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:03.213 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:03.208 7fd9a8879700 1 --2- 172.21.2.102:0/477263868 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd9a40abd60 0x7fd9a40ac160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:03.213 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:03.208 7fd9a8879700 1 -- 172.21.2.102:0/477263868 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd9a40ac780 msgr2=0x7fd9a40acb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:03.213 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:03.208 7fd9a8879700 1 --2- 172.21.2.102:0/477263868 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd9a40ac780 0x7fd9a40acb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:03.214 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:03.208 7fd9a8879700 1 -- 172.21.2.102:0/477263868 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9a4084550 con 0x7fd9a4083910 2021-02-10T11:33:03.214 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:03.208 7fd9a37fe700 1 --2- 172.21.2.102:0/477263868 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd9a40ac780 0x7fd9a40acb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:33:03.214 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:03.208 7fd9a8879700 1 --2- 172.21.2.102:0/477263868 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd9a4083910 0x7fd9a4083d10 secure :-1 s=READY pgs=2354 cs=0 l=1 rev1=1 rx=0x7fd99c009390 tx=0x7fd99c00bd40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:03.214 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:03.208 7fd9a0ff9700 1 -- 172.21.2.102:0/477263868 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd99c003030 con 0x7fd9a4083910 2021-02-10T11:33:03.214 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:03.208 7fd9a0ff9700 1 -- 172.21.2.102:0/477263868 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd99c0068e0 con 0x7fd9a4083910 2021-02-10T11:33:03.214 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:03.208 7fd9a0ff9700 1 -- 172.21.2.102:0/477263868 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd99c00c570 con 0x7fd9a4083910 2021-02-10T11:33:03.214 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:03.208 7fd9aa25c700 1 -- 172.21.2.102:0/477263868 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9a4084550 con 0x7fd9a4083910 2021-02-10T11:33:03.214 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:03.208 7fd9aa25c700 1 -- 172.21.2.102:0/477263868 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9a4084ae0 con 0x7fd9a4083910 2021-02-10T11:33:03.214 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:03.209 7fd9a0ff9700 1 -- 172.21.2.102:0/477263868 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd99c00a000 con 0x7fd9a4083910 2021-02-10T11:33:03.214 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:03.209 7fd9a0ff9700 1 --2- 172.21.2.102:0/477263868 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd9840186c0 0x7fd98401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:03.214 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:03.209 7fd9a0ff9700 1 -- 172.21.2.102:0/477263868 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fd99c0328a0 con 0x7fd9a4083910 2021-02-10T11:33:03.214 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:03.209 7fd9a3fff700 1 --2- 172.21.2.102:0/477263868 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd9840186c0 0x7fd98401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:03.214 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:03.210 7fd9a3fff700 1 --2- 172.21.2.102:0/477263868 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd9840186c0 0x7fd98401abc0 crc :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:03.242 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:03.240 7fd9aa25c700 1 -- 172.21.2.102:0/477263868 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9a4084d90 con 0x7fd9a4083910 2021-02-10T11:33:03.244 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:03.243 7fd9a0ff9700 1 -- 172.21.2.102:0/477263868 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd99c02ff50 con 0x7fd9a4083910 2021-02-10T11:33:03.373 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:03.371 7fd9aa25c700 1 -- 172.21.2.102:0/477263868 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "target": ["mgr", ""], "group_name": "group_0000000000668580"}) v1 -- 0x7fd9a4084ae0 con 0x7fd9840186c0 2021-02-10T11:33:03.377 INFO:tasks.ceph.mgr.y.gibba002.stderr:2021-02-10 11:33:03.376 7fd94bc53700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:33:03.377 INFO:tasks.ceph.mgr.y.gibba002.stderr:2021-02-10 11:33:03.376 7fd94bc53700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:33:03.377 INFO:tasks.ceph.mgr.y.gibba002.stderr:2021-02-10 11:33:03.376 7fd94bc53700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:33:03.377 INFO:tasks.ceph.mgr.y.gibba002.stderr:2021-02-10 11:33:03.376 7fd94bc53700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:33:03.378 INFO:tasks.ceph.mgr.y.gibba002.stderr:2021-02-10 11:33:03.376 7fd94bc53700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:33:03.391 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:03.390 7fd9a0ff9700 1 -- 172.21.2.102:0/477263868 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fd994006a70 con 0x7fd9840186c0 2021-02-10T11:33:03.406 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:03.404 7fd9aa25c700 1 -- 172.21.2.102:0/477263868 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd9840186c0 msgr2=0x7fd98401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:03.406 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:03.404 7fd9aa25c700 1 --2- 172.21.2.102:0/477263868 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd9840186c0 0x7fd98401abc0 crc :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:03.406 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:03.404 7fd9aa25c700 1 -- 172.21.2.102:0/477263868 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd9a4083910 msgr2=0x7fd9a4083d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:03.406 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:03.404 7fd9aa25c700 1 --2- 172.21.2.102:0/477263868 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd9a4083910 0x7fd9a4083d10 secure :-1 s=READY pgs=2354 cs=0 l=1 rev1=1 rx=0x7fd99c009390 tx=0x7fd99c00bd40).stop 2021-02-10T11:33:03.406 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:03.404 7fd9aa25c700 1 -- 172.21.2.102:0/477263868 shutdown_connections 2021-02-10T11:33:03.406 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:03.404 7fd9aa25c700 1 --2- 172.21.2.102:0/477263868 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd9a4083910 0x7fd9a4083d10 unknown :-1 s=CLOSED pgs=2354 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:03.406 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:03.404 7fd9aa25c700 1 --2- 172.21.2.102:0/477263868 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd9840186c0 0x7fd98401abc0 unknown :-1 s=CLOSED pgs=1036 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:03.406 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:03.404 7fd9aa25c700 1 --2- 172.21.2.102:0/477263868 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd9a40ac780 0x7fd9a40acb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:03.406 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:03.404 7fd9aa25c700 1 --2- 172.21.2.102:0/477263868 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd9a40abd60 0x7fd9a40ac160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:03.407 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:03.404 7fd9aa25c700 1 -- 172.21.2.102:0/477263868 >> 172.21.2.102:0/477263868 conn(0x7fd9a40a2b30 msgr2=0x7fd9a409ce60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:03.407 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:03.404 7fd9aa25c700 1 -- 172.21.2.102:0/477263868 shutdown_connections 2021-02-10T11:33:03.407 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:03.404 7fd9aa25c700 1 -- 172.21.2.102:0/477263868 wait complete. 2021-02-10T11:33:03.414 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000594354 --group_name group_0000000000668580 2021-02-10T11:33:03.848 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:03.830 7fdefe7ae700 1 Processor -- start 2021-02-10T11:33:03.849 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:03.830 7fdefe7ae700 1 -- start start 2021-02-10T11:33:03.849 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:03.830 7fdefe7ae700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fdef80d1c80 0x7fdef80cef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:03.849 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:03.830 7fdefe7ae700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fdef80cf590 0x7fdef80cf990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:03.849 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:03.830 7fdefe7ae700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdef80fe1d0 con 0x7fdef80d1880 2021-02-10T11:33:03.849 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:03.830 7fdefe7ae700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdef81b0cc0 con 0x7fdef80cf590 2021-02-10T11:33:03.849 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:03.830 7fdefe7ae700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdef8109a70 con 0x7fdef80d1c80 2021-02-10T11:33:03.849 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:03.830 7fdef77fe700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fdef80d1c80 0x7fdef80cef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:03.849 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:03.831 7fdefcdcb700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fdef80cf590 0x7fdef80cf990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:03.849 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:03.830 7fdef7fff700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fdef80d1880 0x7fdef81b04d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:55454/0 (socket says 172.21.2.102:55454) 2021-02-10T11:33:03.849 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:03.831 7fdef77fe700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fdef80d1c80 0x7fdef80cef70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.113:3300/0 says I am v2:172.21.2.102:38428/0 (socket says 172.21.2.102:38428) 2021-02-10T11:33:03.850 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:03.831 7fdef7fff700 1 -- 172.21.2.102:0/1745617281 learned_addr learned my addr 172.21.2.102:0/1745617281 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:33:03.850 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:03.831 7fdef77fe700 1 -- 172.21.2.102:0/1745617281 >> v2:172.21.2.102:3300/0 conn(0x7fdef80cf590 msgr2=0x7fdef80cf990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:33:03.850 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:03.831 7fdef77fe700 1 --2- 172.21.2.102:0/1745617281 >> v2:172.21.2.102:3300/0 conn(0x7fdef80cf590 0x7fdef80cf990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:03.850 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:03.831 7fdef77fe700 1 -- 172.21.2.102:0/1745617281 >> v1:172.21.2.102:6789/0 conn(0x7fdef80d1880 legacy=0x7fdef81b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:03.850 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:03.831 7fdef77fe700 1 -- 172.21.2.102:0/1745617281 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdef8088070 con 0x7fdef80d1c80 2021-02-10T11:33:03.850 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:03.831 7fdef6ffd700 1 -- 172.21.2.102:0/1745617281 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3752441387 0 0) 0x7fdee80009b0 con 0x7fdef80d1880 2021-02-10T11:33:03.850 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:03.831 7fdef77fe700 1 --2- 172.21.2.102:0/1745617281 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fdef80d1c80 0x7fdef80cef70 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 rx=0x7fdeec006650 tx=0x7fdeec009a00).ready entity=mon.1 client_cookie=f6be7cabccdfdcf4 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:03.850 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:03.831 7fdef6ffd700 1 -- 172.21.2.102:0/1745617281 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdeec009df0 con 0x7fdef80d1c80 2021-02-10T11:33:03.850 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:03.831 7fdef6ffd700 1 -- 172.21.2.102:0/1745617281 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdeec006740 con 0x7fdef80d1c80 2021-02-10T11:33:03.850 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:03.831 7fdefe7ae700 1 -- 172.21.2.102:0/1745617281 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fdef80d1c80 msgr2=0x7fdef80cef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:03.850 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:03.831 7fdefe7ae700 1 --2- 172.21.2.102:0/1745617281 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fdef80d1c80 0x7fdef80cef70 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 rx=0x7fdeec006650 tx=0x7fdeec009a00).stop 2021-02-10T11:33:03.850 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:03.831 7fdef6ffd700 1 -- 172.21.2.102:0/1745617281 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdeec009df0 con 0x7fdef80d1c80 2021-02-10T11:33:03.850 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:03.832 7fdefe7ae700 1 -- 172.21.2.102:0/1745617281 shutdown_connections 2021-02-10T11:33:03.851 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:03.832 7fdefe7ae700 1 --2- 172.21.2.102:0/1745617281 >> v2:172.21.2.102:3300/0 conn(0x7fdef80cf590 0x7fdef80cf990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:03.851 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:03.832 7fdefe7ae700 1 --2- 172.21.2.102:0/1745617281 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fdef80d1c80 0x7fdef80cef70 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:03.851 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:03.832 7fdefe7ae700 1 -- 172.21.2.102:0/1745617281 >> 172.21.2.102:0/1745617281 conn(0x7fdef80ce430 msgr2=0x7fdef80d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:03.851 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:03.832 7fdefe7ae700 1 -- 172.21.2.102:0/1745617281 shutdown_connections 2021-02-10T11:33:03.851 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:03.832 7fdefe7ae700 1 -- 172.21.2.102:0/1745617281 wait complete. 2021-02-10T11:33:03.851 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:03.846 7fdefe7ae700 1 Processor -- start 2021-02-10T11:33:03.851 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:03.846 7fdefe7ae700 1 -- start start 2021-02-10T11:33:03.851 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:03.847 7fdefe7ae700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fdef80aef00 0x7fdef8099c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:03.851 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:03.847 7fdefe7ae700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdef809a260 0x7fdef809a660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:03.851 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:03.847 7fdefe7ae700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdef809ac80 0x7fdef819a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:03.851 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:03.847 7fdefe7ae700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdef819bdd0 con 0x7fdef809a260 2021-02-10T11:33:03.851 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:03.847 7fdefe7ae700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdef819de70 con 0x7fdef80aef00 2021-02-10T11:33:03.851 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:03.847 7fdefe7ae700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdef819a2d0 con 0x7fdef809ac80 2021-02-10T11:33:03.851 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:03.847 7fdef77fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdef809a260 0x7fdef809a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:03.852 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:03.847 7fdef77fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdef809a260 0x7fdef809a660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:35004/0 (socket says 172.21.2.102:35004) 2021-02-10T11:33:03.852 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:03.847 7fdef77fe700 1 -- 172.21.2.102:0/2612099620 learned_addr learned my addr 172.21.2.102:0/2612099620 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:33:03.852 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:03.847 7fdef7fff700 1 --2- 172.21.2.102:0/2612099620 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fdef80aef00 0x7fdef8099c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:03.853 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:03.847 7fdefcdcb700 1 --2- 172.21.2.102:0/2612099620 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdef809ac80 0x7fdef819a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:03.853 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:03.847 7fdef7fff700 1 -- 172.21.2.102:0/2612099620 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdef809ac80 msgr2=0x7fdef819a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:03.853 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:03.847 7fdef7fff700 1 --2- 172.21.2.102:0/2612099620 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdef809ac80 0x7fdef819a850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:03.854 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:03.847 7fdef7fff700 1 -- 172.21.2.102:0/2612099620 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdef809a260 msgr2=0x7fdef809a660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:03.854 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:03.847 7fdef7fff700 1 --2- 172.21.2.102:0/2612099620 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdef809a260 0x7fdef809a660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:03.854 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:03.848 7fdef7fff700 1 -- 172.21.2.102:0/2612099620 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdef8199f50 con 0x7fdef80aef00 2021-02-10T11:33:03.854 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:03.848 7fdef77fe700 1 --2- 172.21.2.102:0/2612099620 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdef809a260 0x7fdef809a660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:33:03.854 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:03.848 7fdef7fff700 1 --2- 172.21.2.102:0/2612099620 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fdef80aef00 0x7fdef8099c40 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 rx=0x7fdee800a4c0 tx=0x7fdee800a860).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:03.854 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:03.848 7fdef4ff9700 1 -- 172.21.2.102:0/2612099620 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdee8000e20 con 0x7fdef80aef00 2021-02-10T11:33:03.854 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:03.848 7fdef4ff9700 1 -- 172.21.2.102:0/2612099620 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdee800ac50 con 0x7fdef80aef00 2021-02-10T11:33:03.854 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:03.848 7fdef4ff9700 1 -- 172.21.2.102:0/2612099620 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdee80045d0 con 0x7fdef80aef00 2021-02-10T11:33:03.854 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:03.848 7fdefe7ae700 1 -- 172.21.2.102:0/2612099620 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdef8199f50 con 0x7fdef80aef00 2021-02-10T11:33:03.854 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:03.848 7fdefe7ae700 1 -- 172.21.2.102:0/2612099620 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdef8199f50 con 0x7fdef80aef00 2021-02-10T11:33:03.854 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:03.848 7fdef4ff9700 1 -- 172.21.2.102:0/2612099620 <== mon.1 v2:172.21.2.113:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fdee8000d30 con 0x7fdef80aef00 2021-02-10T11:33:03.854 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:03.848 7fdef4ff9700 1 --2- 172.21.2.102:0/2612099620 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fded80186c0 0x7fded801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:03.855 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:03.848 7fdef4ff9700 1 -- 172.21.2.102:0/2612099620 <== mon.1 v2:172.21.2.113:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fdee8032470 con 0x7fdef80aef00 2021-02-10T11:33:03.855 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:03.849 7fdef77fe700 1 --2- 172.21.2.102:0/2612099620 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fded80186c0 0x7fded801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:03.855 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:03.849 7fdef77fe700 1 --2- 172.21.2.102:0/2612099620 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fded80186c0 0x7fded801abc0 crc :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:03.881 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:03.880 7fdefe7ae700 1 -- 172.21.2.102:0/2612099620 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdef8082a70 con 0x7fdef80aef00 2021-02-10T11:33:03.884 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:03.883 7fdef4ff9700 1 -- 172.21.2.102:0/2612099620 <== mon.1 v2:172.21.2.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fdee8004140 con 0x7fdef80aef00 2021-02-10T11:33:04.013 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:04.011 7fdefe7ae700 1 -- 172.21.2.102:0/2612099620 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000594354", "target": ["mgr", ""], "group_name": "group_0000000000668580"}) v1 -- 0x7fdef8199f50 con 0x7fded80186c0 2021-02-10T11:33:04.037 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:04.035 7fdef4ff9700 1 -- 172.21.2.102:0/2612099620 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fdeec006c60 con 0x7fded80186c0 2021-02-10T11:33:04.046 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:04.044 7fdefe7ae700 1 -- 172.21.2.102:0/2612099620 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fded80186c0 msgr2=0x7fded801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:04.046 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:04.044 7fdefe7ae700 1 --2- 172.21.2.102:0/2612099620 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fded80186c0 0x7fded801abc0 crc :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:04.046 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:04.044 7fdefe7ae700 1 -- 172.21.2.102:0/2612099620 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fdef80aef00 msgr2=0x7fdef8099c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:04.046 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:04.044 7fdefe7ae700 1 --2- 172.21.2.102:0/2612099620 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fdef80aef00 0x7fdef8099c40 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 rx=0x7fdee800a4c0 tx=0x7fdee800a860).stop 2021-02-10T11:33:04.047 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:04.044 7fdefe7ae700 1 -- 172.21.2.102:0/2612099620 shutdown_connections 2021-02-10T11:33:04.047 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:04.044 7fdefe7ae700 1 --2- 172.21.2.102:0/2612099620 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdef809ac80 0x7fdef819a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:04.047 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:04.044 7fdefe7ae700 1 --2- 172.21.2.102:0/2612099620 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdef809a260 0x7fdef809a660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:04.047 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:04.044 7fdefe7ae700 1 --2- 172.21.2.102:0/2612099620 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fded80186c0 0x7fded801abc0 unknown :-1 s=CLOSED pgs=1037 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:04.047 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:04.044 7fdefe7ae700 1 --2- 172.21.2.102:0/2612099620 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fdef80aef00 0x7fdef8099c40 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:04.047 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:04.044 7fdefe7ae700 1 -- 172.21.2.102:0/2612099620 >> 172.21.2.102:0/2612099620 conn(0x7fdef818e810 msgr2=0x7fdef818c290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:04.047 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:04.045 7fdefe7ae700 1 -- 172.21.2.102:0/2612099620 shutdown_connections 2021-02-10T11:33:04.047 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:04.045 7fdefe7ae700 1 -- 172.21.2.102:0/2612099620 wait complete. 2021-02-10T11:33:04.055 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.1 && sudo ls volumes 2021-02-10T11:33:04.407 INFO:teuthology.orchestra.run.gibba002.stdout:group_0000000000668580 2021-02-10T11:33:04.407 INFO:teuthology.orchestra.run.gibba002.stdout:_group_0000000000668580:subvolume_0000000000594354.meta 2021-02-10T11:33:04.408 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume authorize cephfs subvolume_0000000000594354 alice --group_name group_0000000000668580 --tenant_id tenant1 2021-02-10T11:33:04.849 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:04.830 7f513abc6700 1 Processor -- start 2021-02-10T11:33:04.849 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:04.831 7f513abc6700 1 -- start start 2021-02-10T11:33:04.849 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:04.831 7f513abc6700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f51340e53a0 0x7f51341b0550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:04.849 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:04.831 7f513abc6700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f513410b050 0x7f51341091c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:04.849 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:04.831 7f513abc6700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5134109ac0 con 0x7f51341b0b70 2021-02-10T11:33:04.849 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:04.831 7f513abc6700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51340e57a0 con 0x7f51340e53a0 2021-02-10T11:33:04.849 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:04.831 7f513abc6700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51341097e0 con 0x7f513410b050 2021-02-10T11:33:04.849 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:04.831 7f5133fff700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f51341b0b70 0x7f51341089d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:55470/0 (socket says 172.21.2.102:55470) 2021-02-10T11:33:04.849 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:04.831 7f5133fff700 1 -- 172.21.2.102:0/3187381307 learned_addr learned my addr 172.21.2.102:0/3187381307 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:33:04.849 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:04.831 7f51389e2700 1 --2- 172.21.2.102:0/3187381307 >> v2:172.21.2.102:3300/0 conn(0x7f51340e53a0 0x7f51341b0550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:04.849 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:04.831 7f51391e3700 1 --2- 172.21.2.102:0/3187381307 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f513410b050 0x7f51341091c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:04.850 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:04.831 7f51337fe700 1 -- 172.21.2.102:0/3187381307 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1748534849 0 0) 0x7f5124001500 con 0x7f51341b0b70 2021-02-10T11:33:04.850 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:04.832 7f51337fe700 1 -- 172.21.2.102:0/3187381307 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5120002fe0 con 0x7f51341b0b70 2021-02-10T11:33:04.850 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:04.832 7f51389e2700 1 -- 172.21.2.102:0/3187381307 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f513410b050 msgr2=0x7f51341091c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:33:04.850 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:04.832 7f51389e2700 1 --2- 172.21.2.102:0/3187381307 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f513410b050 0x7f51341091c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:04.850 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:04.832 7f51389e2700 1 -- 172.21.2.102:0/3187381307 >> v1:172.21.2.102:6789/0 conn(0x7f51341b0b70 legacy=0x7f51341089d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:04.850 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:04.832 7f51389e2700 1 -- 172.21.2.102:0/3187381307 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f513410b450 con 0x7f51340e53a0 2021-02-10T11:33:04.850 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:04.832 7f51389e2700 1 --2- 172.21.2.102:0/3187381307 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f51340e53a0 0x7f51341b0550 secure :-1 s=READY pgs=2359 cs=0 l=1 rev1=1 rx=0x7f511c006650 tx=0x7f511c009a80).ready entity=mon.0 client_cookie=8664026a06274609 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:04.850 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:04.832 7f51337fe700 1 -- 172.21.2.102:0/3187381307 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f511c00c0c0 con 0x7f51340e53a0 2021-02-10T11:33:04.850 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:04.832 7f51337fe700 1 -- 172.21.2.102:0/3187381307 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f511c00c400 con 0x7f51340e53a0 2021-02-10T11:33:04.850 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:04.832 7f51337fe700 1 -- 172.21.2.102:0/3187381307 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f511c00c860 con 0x7f51340e53a0 2021-02-10T11:33:04.850 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:04.832 7f513abc6700 1 -- 172.21.2.102:0/3187381307 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f51340e53a0 msgr2=0x7f51341b0550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:04.850 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:04.832 7f513abc6700 1 --2- 172.21.2.102:0/3187381307 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f51340e53a0 0x7f51341b0550 secure :-1 s=READY pgs=2359 cs=0 l=1 rev1=1 rx=0x7f511c006650 tx=0x7f511c009a80).stop 2021-02-10T11:33:04.851 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:04.833 7f513abc6700 1 -- 172.21.2.102:0/3187381307 shutdown_connections 2021-02-10T11:33:04.851 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:04.833 7f513abc6700 1 --2- 172.21.2.102:0/3187381307 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f513410b050 0x7f51341091c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:04.851 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:04.833 7f513abc6700 1 --2- 172.21.2.102:0/3187381307 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f51340e53a0 0x7f51341b0550 unknown :-1 s=CLOSED pgs=2359 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:04.851 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:04.833 7f513abc6700 1 -- 172.21.2.102:0/3187381307 >> 172.21.2.102:0/3187381307 conn(0x7f51340ce4b0 msgr2=0x7f51340e45d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:04.851 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:04.833 7f513abc6700 1 -- 172.21.2.102:0/3187381307 shutdown_connections 2021-02-10T11:33:04.851 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:04.833 7f513abc6700 1 -- 172.21.2.102:0/3187381307 wait complete. 2021-02-10T11:33:04.852 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:04.847 7f513abc6700 1 Processor -- start 2021-02-10T11:33:04.852 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:04.847 7f513abc6700 1 -- start start 2021-02-10T11:33:04.853 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:04.848 7f513abc6700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f513409d960 0x7f51340a8c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:04.853 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:04.848 7f513abc6700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f513409c6a0 0x7f513409caa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:04.853 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:04.848 7f513abc6700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f51340a3440 0x7f51340a3840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:04.853 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:04.848 7f513abc6700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f513409d0c0 con 0x7f513409c6a0 2021-02-10T11:33:04.853 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:04.848 7f513abc6700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5134106f50 con 0x7f513409d960 2021-02-10T11:33:04.853 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:04.848 7f513abc6700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f513409dd60 con 0x7f51340a3440 2021-02-10T11:33:04.853 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:04.848 7f5133fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f513409c6a0 0x7f513409caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:04.853 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:04.848 7f5133fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f513409c6a0 0x7f513409caa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:35020/0 (socket says 172.21.2.102:35020) 2021-02-10T11:33:04.853 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:04.848 7f5133fff700 1 -- 172.21.2.102:0/214922934 learned_addr learned my addr 172.21.2.102:0/214922934 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:33:04.853 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:04.848 7f51391e3700 1 --2- 172.21.2.102:0/214922934 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f51340a3440 0x7f51340a3840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:04.853 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:04.848 7f51389e2700 1 --2- 172.21.2.102:0/214922934 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f513409d960 0x7f51340a8c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:04.853 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:04.849 7f51391e3700 1 -- 172.21.2.102:0/214922934 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f513409d960 msgr2=0x7f51340a8c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:04.853 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:04.849 7f51391e3700 1 --2- 172.21.2.102:0/214922934 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f513409d960 0x7f51340a8c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:04.854 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:04.849 7f51391e3700 1 -- 172.21.2.102:0/214922934 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f513409c6a0 msgr2=0x7f513409caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:04.854 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:04.849 7f51391e3700 1 --2- 172.21.2.102:0/214922934 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f513409c6a0 0x7f513409caa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:04.854 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:04.849 7f51391e3700 1 -- 172.21.2.102:0/214922934 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51340a3e60 con 0x7f51340a3440 2021-02-10T11:33:04.854 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:04.849 7f5133fff700 1 --2- 172.21.2.102:0/214922934 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f513409c6a0 0x7f513409caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:33:04.854 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:04.849 7f51391e3700 1 --2- 172.21.2.102:0/214922934 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f51340a3440 0x7f51340a3840 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0x7f512c006ed0 tx=0x7f512c00b910).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:04.854 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:04.849 7f51317fa700 1 -- 172.21.2.102:0/214922934 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f512c006b30 con 0x7f51340a3440 2021-02-10T11:33:04.854 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:04.849 7f51317fa700 1 -- 172.21.2.102:0/214922934 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f512c00bd00 con 0x7f51340a3440 2021-02-10T11:33:04.854 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:04.849 7f51317fa700 1 -- 172.21.2.102:0/214922934 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f512c0022b0 con 0x7f51340a3440 2021-02-10T11:33:04.854 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:04.849 7f513abc6700 1 -- 172.21.2.102:0/214922934 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51340a3e60 con 0x7f51340a3440 2021-02-10T11:33:04.854 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:04.849 7f513abc6700 1 -- 172.21.2.102:0/214922934 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51340a3e60 con 0x7f51340a3440 2021-02-10T11:33:04.854 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:04.850 7f51317fa700 1 -- 172.21.2.102:0/214922934 <== mon.2 v2:172.21.2.113:3301/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f512c005330 con 0x7f51340a3440 2021-02-10T11:33:04.854 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:04.850 7f51317fa700 1 --2- 172.21.2.102:0/214922934 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f51180186c0 0x7f511801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:04.854 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:04.850 7f51317fa700 1 -- 172.21.2.102:0/214922934 <== mon.2 v2:172.21.2.113:3301/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f512c032680 con 0x7f51340a3440 2021-02-10T11:33:04.855 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:04.850 7f51389e2700 1 --2- 172.21.2.102:0/214922934 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f51180186c0 0x7f511801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:04.855 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:04.850 7f51389e2700 1 --2- 172.21.2.102:0/214922934 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f51180186c0 0x7f511801abc0 crc :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:04.882 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:04.881 7f513abc6700 1 -- 172.21.2.102:0/214922934 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51340a4100 con 0x7f51340a3440 2021-02-10T11:33:04.885 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:04.884 7f51317fa700 1 -- 172.21.2.102:0/214922934 <== mon.2 v2:172.21.2.113:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f512c006370 con 0x7f51340a3440 2021-02-10T11:33:05.010 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:05.009 7f513abc6700 1 -- 172.21.2.102:0/214922934 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume authorize", "vol_name": "cephfs", "sub_name": "subvolume_0000000000594354", "target": ["mgr", ""], "tenant_id": "tenant1", "auth_id": "alice", "group_name": "group_0000000000668580"}) v1 -- 0x7f51340a4100 con 0x7f51180186c0 2021-02-10T11:33:05.016 INFO:tasks.ceph.mgr.y.gibba002.stderr:Creating meta for ID alice with tenant tenant1 2021-02-10T11:33:05.046 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:05.045 7f51317fa700 1 -- 172.21.2.102:0/214922934 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+41 (crc 0 0 0) 0x7f511c000990 con 0x7f51180186c0 2021-02-10T11:33:05.074 INFO:teuthology.orchestra.run.gibba002.stdout:AQBxxCNgj3ytARAAP4sE76cNoucyYWD2ySOHXA== 2021-02-10T11:33:05.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:05.074 7f513abc6700 1 -- 172.21.2.102:0/214922934 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f51180186c0 msgr2=0x7f511801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:05.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:05.074 7f513abc6700 1 --2- 172.21.2.102:0/214922934 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f51180186c0 0x7f511801abc0 crc :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:05.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:05.074 7f513abc6700 1 -- 172.21.2.102:0/214922934 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f51340a3440 msgr2=0x7f51340a3840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:05.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:05.074 7f513abc6700 1 --2- 172.21.2.102:0/214922934 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f51340a3440 0x7f51340a3840 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0x7f512c006ed0 tx=0x7f512c00b910).stop 2021-02-10T11:33:05.077 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:05.075 7f513abc6700 1 -- 172.21.2.102:0/214922934 shutdown_connections 2021-02-10T11:33:05.077 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:05.075 7f513abc6700 1 --2- 172.21.2.102:0/214922934 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f51340a3440 0x7f51340a3840 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:05.077 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:05.075 7f513abc6700 1 --2- 172.21.2.102:0/214922934 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f513409c6a0 0x7f513409caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:05.077 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:05.075 7f513abc6700 1 --2- 172.21.2.102:0/214922934 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f51180186c0 0x7f511801abc0 unknown :-1 s=CLOSED pgs=1038 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:05.077 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:05.075 7f513abc6700 1 --2- 172.21.2.102:0/214922934 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f513409d960 0x7f51340a8c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:05.077 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:05.075 7f513abc6700 1 -- 172.21.2.102:0/214922934 >> 172.21.2.102:0/214922934 conn(0x7f513409e470 msgr2=0x7f51340b5960 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:05.077 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:05.075 7f513abc6700 1 -- 172.21.2.102:0/214922934 shutdown_connections 2021-02-10T11:33:05.077 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:05.075 7f513abc6700 1 -- 172.21.2.102:0/214922934 wait complete. 2021-02-10T11:33:05.084 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.1 && sudo ls volumes 2021-02-10T11:33:05.431 INFO:teuthology.orchestra.run.gibba002.stdout:$alice.meta 2021-02-10T11:33:05.432 INFO:teuthology.orchestra.run.gibba002.stdout:group_0000000000668580 2021-02-10T11:33:05.432 INFO:teuthology.orchestra.run.gibba002.stdout:_group_0000000000668580:subvolume_0000000000594354.meta 2021-02-10T11:33:05.433 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.1 && sudo sudo cat '/home/ubuntu/cephtest/mnt.1/volumes/$alice.meta' 2021-02-10T11:33:05.795 INFO:teuthology.orchestra.run.gibba002.stdout:{"compat_version": 1, "tenant_id": "tenant1", "version": 5, "dirty": false, "subvolumes": {"group_0000000000668580/subvolume_0000000000594354": {"access_level": "rw", "dirty": false}}} 2021-02-10T11:33:05.795 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.1 && sudo sudo cat /home/ubuntu/cephtest/mnt.1/volumes/_group_0000000000668580:subvolume_0000000000594354.meta 2021-02-10T11:33:06.157 INFO:teuthology.orchestra.run.gibba002.stdout:{"auths": {"alice": {"access_level": "rw", "dirty": false}}, "version": 5, "compat_version": 1} 2021-02-10T11:33:06.158 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume authorize cephfs subvolume_0000000000594354 alice --group_name group_0000000000668580 --tenant_id tenant2 2021-02-10T11:33:06.597 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:06.579 7f7bc3bac700 1 Processor -- start 2021-02-10T11:33:06.598 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:06.579 7f7bc3bac700 1 -- start start 2021-02-10T11:33:06.598 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:06.580 7f7bc3bac700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7bbc0e53a0 0x7f7bbc1b0550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:06.598 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:06.580 7f7bc3bac700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f7bbc10b050 0x7f7bbc1091c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:06.598 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:06.580 7f7bc3bac700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bbc109ac0 con 0x7f7bbc1b0b70 2021-02-10T11:33:06.598 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:06.580 7f7bc3bac700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7bbc0fe2b0 con 0x7f7bbc10b050 2021-02-10T11:33:06.598 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:06.580 7f7bc3bac700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7bbc0e5f80 con 0x7f7bbc0e53a0 2021-02-10T11:33:06.598 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:06.580 7f7bc21c9700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f7bbc10b050 0x7f7bbc1091c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:06.598 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:06.580 7f7bc21c9700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f7bbc10b050 0x7f7bbc1091c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:35034/0 (socket says 172.21.2.102:35034) 2021-02-10T11:33:06.599 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:06.580 7f7bc21c9700 1 -- 172.21.2.102:0/269280290 learned_addr learned my addr 172.21.2.102:0/269280290 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:33:06.599 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:06.580 7f7bc19c8700 1 --2- 172.21.2.102:0/269280290 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7bbc0e53a0 0x7f7bbc1b0550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:06.599 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:06.580 7f7bc21c9700 1 -- 172.21.2.102:0/269280290 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7bbc0e53a0 msgr2=0x7f7bbc1b0550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:33:06.599 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:06.580 7f7bc21c9700 1 --2- 172.21.2.102:0/269280290 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7bbc0e53a0 0x7f7bbc1b0550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:06.599 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:06.580 7f7bc21c9700 1 -- 172.21.2.102:0/269280290 >> v1:172.21.2.113:6789/0 conn(0x7f7bbc1b0b70 legacy=0x7f7bbc1089d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:06.599 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:06.580 7f7bc21c9700 1 -- 172.21.2.102:0/269280290 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bbc0e57a0 con 0x7f7bbc10b050 2021-02-10T11:33:06.599 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:06.580 7f7bc21c9700 1 --2- 172.21.2.102:0/269280290 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7bbc10b050 0x7f7bbc1091c0 secure :-1 s=READY pgs=2360 cs=0 l=1 rev1=1 rx=0x7f7bb4006f00 tx=0x7f7bb4009f10).ready entity=mon.0 client_cookie=c8ae148f96dabe51 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:06.599 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:06.580 7f7bc09c6700 1 -- 172.21.2.102:0/269280290 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7bb400b3b0 con 0x7f7bbc10b050 2021-02-10T11:33:06.599 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:06.581 7f7bc09c6700 1 -- 172.21.2.102:0/269280290 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7bb400b670 con 0x7f7bbc10b050 2021-02-10T11:33:06.599 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:06.581 7f7bc09c6700 1 -- 172.21.2.102:0/269280290 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7bb400bb50 con 0x7f7bbc10b050 2021-02-10T11:33:06.600 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:06.581 7f7bc3bac700 1 -- 172.21.2.102:0/269280290 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7bbc10b050 msgr2=0x7f7bbc1091c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:06.600 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:06.581 7f7bc3bac700 1 --2- 172.21.2.102:0/269280290 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7bbc10b050 0x7f7bbc1091c0 secure :-1 s=READY pgs=2360 cs=0 l=1 rev1=1 rx=0x7f7bb4006f00 tx=0x7f7bb4009f10).stop 2021-02-10T11:33:06.600 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:06.581 7f7bc3bac700 1 -- 172.21.2.102:0/269280290 shutdown_connections 2021-02-10T11:33:06.600 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:06.581 7f7bc3bac700 1 --2- 172.21.2.102:0/269280290 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7bbc10b050 0x7f7bbc1091c0 unknown :-1 s=CLOSED pgs=2360 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:06.600 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:06.581 7f7bc3bac700 1 --2- 172.21.2.102:0/269280290 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7bbc0e53a0 0x7f7bbc1b0550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:06.600 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:06.581 7f7bc3bac700 1 -- 172.21.2.102:0/269280290 >> 172.21.2.102:0/269280290 conn(0x7f7bbc0ce4b0 msgr2=0x7f7bbc0e45d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:06.600 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:06.581 7f7bc3bac700 1 -- 172.21.2.102:0/269280290 shutdown_connections 2021-02-10T11:33:06.600 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:06.581 7f7bc3bac700 1 -- 172.21.2.102:0/269280290 wait complete. 2021-02-10T11:33:06.600 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:06.596 7f7bc3bac700 1 Processor -- start 2021-02-10T11:33:06.600 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:06.596 7f7bc3bac700 1 -- start start 2021-02-10T11:33:06.602 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:06.596 7f7bc3bac700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7bbc19a540 0x7f7bbc0b1fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:06.602 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:06.596 7f7bc3bac700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7bbc0b2600 0x7f7bbc187f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:06.602 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:06.596 7f7bc3bac700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7bbc191390 0x7f7bbc188520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:06.602 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:06.596 7f7bc3bac700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bbc190e10 con 0x7f7bbc0b2600 2021-02-10T11:33:06.602 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:06.596 7f7bc3bac700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bbc19a0a0 con 0x7f7bbc19a540 2021-02-10T11:33:06.602 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:06.596 7f7bc3bac700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7bbc19a280 con 0x7f7bbc191390 2021-02-10T11:33:06.602 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:06.596 7f7bc11c7700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7bbc0b2600 0x7f7bbc187f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:06.602 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:06.596 7f7bc11c7700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7bbc0b2600 0x7f7bbc187f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:35040/0 (socket says 172.21.2.102:35040) 2021-02-10T11:33:06.603 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:06.596 7f7bc11c7700 1 -- 172.21.2.102:0/3715509134 learned_addr learned my addr 172.21.2.102:0/3715509134 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:33:06.603 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:06.596 7f7bc21c9700 1 --2- 172.21.2.102:0/3715509134 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7bbc191390 0x7f7bbc188520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:06.603 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:06.596 7f7bc19c8700 1 --2- 172.21.2.102:0/3715509134 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7bbc19a540 0x7f7bbc0b1fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:06.603 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:06.597 7f7bc11c7700 1 -- 172.21.2.102:0/3715509134 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7bbc191390 msgr2=0x7f7bbc188520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:06.603 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:06.597 7f7bc11c7700 1 --2- 172.21.2.102:0/3715509134 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7bbc191390 0x7f7bbc188520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:06.603 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:06.597 7f7bc11c7700 1 -- 172.21.2.102:0/3715509134 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7bbc19a540 msgr2=0x7f7bbc0b1fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:06.603 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:06.597 7f7bc11c7700 1 --2- 172.21.2.102:0/3715509134 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7bbc19a540 0x7f7bbc0b1fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:06.603 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:06.597 7f7bc11c7700 1 -- 172.21.2.102:0/3715509134 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bbc188b40 con 0x7f7bbc0b2600 2021-02-10T11:33:06.603 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:06.597 7f7bc21c9700 1 --2- 172.21.2.102:0/3715509134 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7bbc191390 0x7f7bbc188520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:33:06.603 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:06.597 7f7bc11c7700 1 --2- 172.21.2.102:0/3715509134 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7bbc0b2600 0x7f7bbc187f00 secure :-1 s=READY pgs=2361 cs=0 l=1 rev1=1 rx=0x7f7bb8006ed0 tx=0x7f7bb800b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:06.603 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:06.597 7f7bae7fc700 1 -- 172.21.2.102:0/3715509134 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7bb8006020 con 0x7f7bbc0b2600 2021-02-10T11:33:06.603 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:06.597 7f7bae7fc700 1 -- 172.21.2.102:0/3715509134 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7bb8006360 con 0x7f7bbc0b2600 2021-02-10T11:33:06.603 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:06.597 7f7bae7fc700 1 -- 172.21.2.102:0/3715509134 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7bb8005730 con 0x7f7bbc0b2600 2021-02-10T11:33:06.603 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:06.597 7f7bc3bac700 1 -- 172.21.2.102:0/3715509134 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7bbc188b40 con 0x7f7bbc0b2600 2021-02-10T11:33:06.604 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:06.597 7f7bc3bac700 1 -- 172.21.2.102:0/3715509134 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7bbc188b40 con 0x7f7bbc0b2600 2021-02-10T11:33:06.604 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:06.598 7f7bae7fc700 1 -- 172.21.2.102:0/3715509134 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f7bb80054b0 con 0x7f7bbc0b2600 2021-02-10T11:33:06.604 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:06.598 7f7bae7fc700 1 --2- 172.21.2.102:0/3715509134 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7ba40186c0 0x7f7ba401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:06.604 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:06.598 7f7bae7fc700 1 -- 172.21.2.102:0/3715509134 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f7bb80325b0 con 0x7f7bbc0b2600 2021-02-10T11:33:06.604 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:06.598 7f7bc19c8700 1 --2- 172.21.2.102:0/3715509134 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7ba40186c0 0x7f7ba401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:06.604 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:06.598 7f7bc19c8700 1 --2- 172.21.2.102:0/3715509134 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7ba40186c0 0x7f7ba401abc0 crc :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:06.631 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:06.630 7f7bc3bac700 1 -- 172.21.2.102:0/3715509134 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7bbc109ac0 con 0x7f7bbc0b2600 2021-02-10T11:33:06.634 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:06.632 7f7bae7fc700 1 -- 172.21.2.102:0/3715509134 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f7bb802ef30 con 0x7f7bbc0b2600 2021-02-10T11:33:06.759 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:06.758 7f7bc3bac700 1 -- 172.21.2.102:0/3715509134 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume authorize", "vol_name": "cephfs", "sub_name": "subvolume_0000000000594354", "target": ["mgr", ""], "tenant_id": "tenant2", "auth_id": "alice", "group_name": "group_0000000000668580"}) v1 -- 0x7f7bbc109ac0 con 0x7f7ba40186c0 2021-02-10T11:33:06.765 INFO:tasks.ceph.mgr.y.gibba002.stderr:2021-02-10 11:33:06.763 7fd94bc53700 -1 mgr.server reply reply (1) Operation not permitted auth ID: alice is already in use 2021-02-10T11:33:06.765 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:06.763 7f7bae7fc700 1 -- 172.21.2.102:0/3715509134 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: -1 auth ID: alice is already in use) v1 ==== 40+0+0 (crc 0 0 0) 0x7f7bb00009b0 con 0x7f7ba40186c0 2021-02-10T11:33:06.766 INFO:teuthology.orchestra.run.gibba002.stderr:Error EPERM: auth ID: alice is already in use 2021-02-10T11:33:06.769 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:06.766 7f7bc3bac700 1 -- 172.21.2.102:0/3715509134 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7ba40186c0 msgr2=0x7f7ba401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:06.769 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:06.766 7f7bc3bac700 1 --2- 172.21.2.102:0/3715509134 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7ba40186c0 0x7f7ba401abc0 crc :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:06.769 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:06.766 7f7bc3bac700 1 -- 172.21.2.102:0/3715509134 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7bbc0b2600 msgr2=0x7f7bbc187f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:06.769 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:06.766 7f7bc3bac700 1 --2- 172.21.2.102:0/3715509134 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7bbc0b2600 0x7f7bbc187f00 secure :-1 s=READY pgs=2361 cs=0 l=1 rev1=1 rx=0x7f7bb8006ed0 tx=0x7f7bb800b960).stop 2021-02-10T11:33:06.769 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:06.766 7f7bc3bac700 1 -- 172.21.2.102:0/3715509134 shutdown_connections 2021-02-10T11:33:06.769 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:06.766 7f7bc3bac700 1 --2- 172.21.2.102:0/3715509134 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7bbc191390 0x7f7bbc188520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:06.769 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:06.766 7f7bc3bac700 1 --2- 172.21.2.102:0/3715509134 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7bbc0b2600 0x7f7bbc187f00 unknown :-1 s=CLOSED pgs=2361 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:06.769 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:06.766 7f7bc3bac700 1 --2- 172.21.2.102:0/3715509134 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7ba40186c0 0x7f7ba401abc0 unknown :-1 s=CLOSED pgs=1039 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:06.769 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:06.766 7f7bc3bac700 1 --2- 172.21.2.102:0/3715509134 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7bbc19a540 0x7f7bbc0b1fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:06.769 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:06.766 7f7bc3bac700 1 -- 172.21.2.102:0/3715509134 >> 172.21.2.102:0/3715509134 conn(0x7f7bbc19f8e0 msgr2=0x7f7bbc19aa50 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:06.769 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:06.766 7f7bc3bac700 1 -- 172.21.2.102:0/3715509134 shutdown_connections 2021-02-10T11:33:06.770 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:06.767 7f7bc3bac700 1 -- 172.21.2.102:0/3715509134 wait complete. 2021-02-10T11:33:06.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:33:06.777 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume deauthorize cephfs subvolume_0000000000594354 alice --group_name group_0000000000668580 2021-02-10T11:33:07.211 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:07.193 7fb6e48ad700 1 Processor -- start 2021-02-10T11:33:07.211 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:07.193 7fb6e48ad700 1 -- start start 2021-02-10T11:33:07.211 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:07.193 7fb6e48ad700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fb6e00dce80 0x7fb6e00d14f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:07.211 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:07.194 7fb6e48ad700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb6e010b2c0 0x7fb6e00eb620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:07.211 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:07.194 7fb6e48ad700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6e010a550 con 0x7fb6e00eb220 2021-02-10T11:33:07.211 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:07.194 7fb6e48ad700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6e0088240 con 0x7fb6e00dce80 2021-02-10T11:33:07.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:07.194 7fb6e48ad700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6e00d1f30 con 0x7fb6e010b2c0 2021-02-10T11:33:07.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:07.194 7fb6de61c700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fb6e00dce80 0x7fb6e00d14f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:07.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:07.194 7fb6de61c700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fb6e00dce80 0x7fb6e00d14f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:35046/0 (socket says 172.21.2.102:35046) 2021-02-10T11:33:07.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:07.194 7fb6de61c700 1 -- 172.21.2.102:0/3683600417 learned_addr learned my addr 172.21.2.102:0/3683600417 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:33:07.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:07.194 7fb6dee1d700 1 --2- 172.21.2.102:0/3683600417 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb6e010b2c0 0x7fb6e00eb620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:07.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:07.194 7fb6de61c700 1 -- 172.21.2.102:0/3683600417 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb6e010b2c0 msgr2=0x7fb6e00eb620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:33:07.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:07.194 7fb6de61c700 1 --2- 172.21.2.102:0/3683600417 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb6e010b2c0 0x7fb6e00eb620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:07.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:07.194 7fb6de61c700 1 -- 172.21.2.102:0/3683600417 >> v1:172.21.2.113:6789/0 conn(0x7fb6e00eb220 legacy=0x7fb6e010aad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:07.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:07.194 7fb6de61c700 1 -- 172.21.2.102:0/3683600417 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6e00dd280 con 0x7fb6e00dce80 2021-02-10T11:33:07.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:07.194 7fb6de61c700 1 --2- 172.21.2.102:0/3683600417 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb6e00dce80 0x7fb6e00d14f0 secure :-1 s=READY pgs=2362 cs=0 l=1 rev1=1 rx=0x7fb6c8006700 tx=0x7fb6c8009b80).ready entity=mon.0 client_cookie=970a7d6d8739b0a0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:07.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:07.194 7fb6dd61a700 1 -- 172.21.2.102:0/3683600417 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb6c800c0c0 con 0x7fb6e00dce80 2021-02-10T11:33:07.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:07.194 7fb6dd61a700 1 -- 172.21.2.102:0/3683600417 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6c800c440 con 0x7fb6e00dce80 2021-02-10T11:33:07.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:07.194 7fb6dd61a700 1 -- 172.21.2.102:0/3683600417 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb6c800c960 con 0x7fb6e00dce80 2021-02-10T11:33:07.213 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:07.194 7fb6e48ad700 1 -- 172.21.2.102:0/3683600417 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb6e00dce80 msgr2=0x7fb6e00d14f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:07.213 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:07.194 7fb6e48ad700 1 --2- 172.21.2.102:0/3683600417 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb6e00dce80 0x7fb6e00d14f0 secure :-1 s=READY pgs=2362 cs=0 l=1 rev1=1 rx=0x7fb6c8006700 tx=0x7fb6c8009b80).stop 2021-02-10T11:33:07.213 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:07.195 7fb6e48ad700 1 -- 172.21.2.102:0/3683600417 shutdown_connections 2021-02-10T11:33:07.213 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:07.195 7fb6e48ad700 1 --2- 172.21.2.102:0/3683600417 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb6e010b2c0 0x7fb6e00eb620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:07.213 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:07.195 7fb6e48ad700 1 --2- 172.21.2.102:0/3683600417 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb6e00dce80 0x7fb6e00d14f0 unknown :-1 s=CLOSED pgs=2362 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:07.213 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:07.195 7fb6e48ad700 1 -- 172.21.2.102:0/3683600417 >> 172.21.2.102:0/3683600417 conn(0x7fb6e00ea630 msgr2=0x7fb6e00d0800 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:07.213 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:07.195 7fb6e48ad700 1 -- 172.21.2.102:0/3683600417 shutdown_connections 2021-02-10T11:33:07.213 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:07.195 7fb6e48ad700 1 -- 172.21.2.102:0/3683600417 wait complete. 2021-02-10T11:33:07.214 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:07.210 7fb6e48ad700 1 Processor -- start 2021-02-10T11:33:07.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:07.210 7fb6e48ad700 1 -- start start 2021-02-10T11:33:07.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:07.210 7fb6e48ad700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb6e01a3580 0x7fb6e01a3980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:07.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:07.210 7fb6e48ad700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb6e0099280 0x7fb6e00924d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:07.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:07.210 7fb6e48ad700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb6e0099680 0x7fb6e0092af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:07.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:07.210 7fb6e48ad700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6e0109b70 con 0x7fb6e0099680 2021-02-10T11:33:07.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:07.210 7fb6e48ad700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6e0096b80 con 0x7fb6e0099280 2021-02-10T11:33:07.216 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:07.210 7fb6e48ad700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6e0093750 con 0x7fb6e01a3580 2021-02-10T11:33:07.216 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:07.211 7fb6de61c700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb6e01a3580 0x7fb6e01a3980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:07.216 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:07.211 7fb6dee1d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb6e0099680 0x7fb6e0092af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:07.216 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:07.211 7fb6de61c700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb6e01a3580 0x7fb6e01a3980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.113:3301/0 says I am v2:172.21.2.102:40588/0 (socket says 172.21.2.102:40588) 2021-02-10T11:33:07.216 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:07.211 7fb6de61c700 1 -- 172.21.2.102:0/937727121 learned_addr learned my addr 172.21.2.102:0/937727121 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:33:07.216 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:07.211 7fb6dde1b700 1 --2- 172.21.2.102:0/937727121 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb6e0099280 0x7fb6e00924d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:07.216 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:07.211 7fb6dee1d700 1 -- 172.21.2.102:0/937727121 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb6e01a3580 msgr2=0x7fb6e01a3980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:07.216 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:07.211 7fb6dee1d700 1 --2- 172.21.2.102:0/937727121 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb6e01a3580 0x7fb6e01a3980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:07.216 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:07.211 7fb6dee1d700 1 -- 172.21.2.102:0/937727121 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb6e0099280 msgr2=0x7fb6e00924d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:07.216 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:07.211 7fb6dee1d700 1 --2- 172.21.2.102:0/937727121 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb6e0099280 0x7fb6e00924d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:07.216 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:07.211 7fb6dee1d700 1 -- 172.21.2.102:0/937727121 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6e0093970 con 0x7fb6e0099680 2021-02-10T11:33:07.217 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:07.211 7fb6dee1d700 1 --2- 172.21.2.102:0/937727121 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb6e0099680 0x7fb6e0092af0 secure :-1 s=READY pgs=2363 cs=0 l=1 rev1=1 rx=0x7fb6d800b8f0 tx=0x7fb6d800bc90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:07.217 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:07.211 7fb6d6ffd700 1 -- 172.21.2.102:0/937727121 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb6d800c2f0 con 0x7fb6e0099680 2021-02-10T11:33:07.217 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:07.211 7fb6d6ffd700 1 -- 172.21.2.102:0/937727121 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6d800c570 con 0x7fb6e0099680 2021-02-10T11:33:07.217 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:07.211 7fb6d6ffd700 1 -- 172.21.2.102:0/937727121 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb6d800ca90 con 0x7fb6e0099680 2021-02-10T11:33:07.217 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:07.211 7fb6e48ad700 1 -- 172.21.2.102:0/937727121 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6e0093970 con 0x7fb6e0099680 2021-02-10T11:33:07.217 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:07.211 7fb6e48ad700 1 -- 172.21.2.102:0/937727121 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6e0093f00 con 0x7fb6e0099680 2021-02-10T11:33:07.217 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:07.212 7fb6d6ffd700 1 -- 172.21.2.102:0/937727121 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fb6d800c750 con 0x7fb6e0099680 2021-02-10T11:33:07.217 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:07.212 7fb6d6ffd700 1 --2- 172.21.2.102:0/937727121 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb6c40186c0 0x7fb6c401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:07.217 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:07.212 7fb6d6ffd700 1 -- 172.21.2.102:0/937727121 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fb6d8032820 con 0x7fb6e0099680 2021-02-10T11:33:07.217 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:07.212 7fb6de61c700 1 --2- 172.21.2.102:0/937727121 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb6c40186c0 0x7fb6c401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:07.217 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:07.212 7fb6de61c700 1 --2- 172.21.2.102:0/937727121 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb6c40186c0 0x7fb6c401abc0 crc :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:07.245 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:07.243 7fb6e48ad700 1 -- 172.21.2.102:0/937727121 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6e00f9390 con 0x7fb6e0099680 2021-02-10T11:33:07.247 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:07.246 7fb6d6ffd700 1 -- 172.21.2.102:0/937727121 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fb6d801f7e0 con 0x7fb6e0099680 2021-02-10T11:33:07.374 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:07.372 7fb6e48ad700 1 -- 172.21.2.102:0/937727121 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume deauthorize", "vol_name": "cephfs", "sub_name": "subvolume_0000000000594354", "target": ["mgr", ""], "auth_id": "alice", "group_name": "group_0000000000668580"}) v1 -- 0x7fb6e0093f00 con 0x7fb6c40186c0 2021-02-10T11:33:07.408 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:07.407 7fb6d6ffd700 1 -- 172.21.2.102:0/937727121 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fb6c80009f0 con 0x7fb6c40186c0 2021-02-10T11:33:07.439 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:07.437 7fb6e48ad700 1 -- 172.21.2.102:0/937727121 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb6c40186c0 msgr2=0x7fb6c401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:07.439 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:07.437 7fb6e48ad700 1 --2- 172.21.2.102:0/937727121 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb6c40186c0 0x7fb6c401abc0 crc :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:07.439 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:07.437 7fb6e48ad700 1 -- 172.21.2.102:0/937727121 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb6e0099680 msgr2=0x7fb6e0092af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:07.440 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:07.437 7fb6e48ad700 1 --2- 172.21.2.102:0/937727121 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb6e0099680 0x7fb6e0092af0 secure :-1 s=READY pgs=2363 cs=0 l=1 rev1=1 rx=0x7fb6d800b8f0 tx=0x7fb6d800bc90).stop 2021-02-10T11:33:07.440 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:07.437 7fb6e48ad700 1 -- 172.21.2.102:0/937727121 shutdown_connections 2021-02-10T11:33:07.440 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:07.437 7fb6e48ad700 1 --2- 172.21.2.102:0/937727121 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb6e0099680 0x7fb6e0092af0 unknown :-1 s=CLOSED pgs=2363 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:07.440 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:07.437 7fb6e48ad700 1 --2- 172.21.2.102:0/937727121 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb6c40186c0 0x7fb6c401abc0 unknown :-1 s=CLOSED pgs=1040 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:07.440 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:07.437 7fb6e48ad700 1 --2- 172.21.2.102:0/937727121 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb6e0099280 0x7fb6e00924d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:07.440 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:07.437 7fb6e48ad700 1 --2- 172.21.2.102:0/937727121 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb6e01a3580 0x7fb6e01a3980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:07.440 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:07.437 7fb6e48ad700 1 -- 172.21.2.102:0/937727121 >> 172.21.2.102:0/937727121 conn(0x7fb6e0109df0 msgr2=0x7fb6e00fc8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:07.440 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:07.438 7fb6e48ad700 1 -- 172.21.2.102:0/937727121 shutdown_connections 2021-02-10T11:33:07.440 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:07.438 7fb6e48ad700 1 -- 172.21.2.102:0/937727121 wait complete. 2021-02-10T11:33:07.447 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.1 && sudo ls volumes 2021-02-10T11:33:07.797 INFO:teuthology.orchestra.run.gibba002.stdout:group_0000000000668580 2021-02-10T11:33:07.798 INFO:teuthology.orchestra.run.gibba002.stdout:_group_0000000000668580:subvolume_0000000000594354.meta 2021-02-10T11:33:07.799 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000594354 --group_name group_0000000000668580 2021-02-10T11:33:08.233 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:08.214 7f3b70552700 1 Processor -- start 2021-02-10T11:33:08.233 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:08.214 7f3b70552700 1 -- start start 2021-02-10T11:33:08.234 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:08.215 7f3b70552700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3b680d1880 0x7f3b681b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:08.234 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:08.215 7f3b70552700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f3b681b0af0 0x7f3b68108950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:08.234 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:08.215 7f3b70552700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f3b680d1c80 0x7f3b68108f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:08.234 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:08.215 7f3b70552700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b680e5f00 con 0x7f3b681b0af0 2021-02-10T11:33:08.234 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:08.215 7f3b70552700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b68109a70 con 0x7f3b680d1c80 2021-02-10T11:33:08.234 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:08.215 7f3b70552700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b68088070 con 0x7f3b680d1880 2021-02-10T11:33:08.234 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:08.215 7f3b6db6d700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f3b681b0af0 0x7f3b68108950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:08.235 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:08.215 7f3b6eb6f700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f3b680d1c80 0x7f3b68108f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:08.235 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:08.215 7f3b6eb6f700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f3b680d1c80 0x7f3b68108f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.113:3300/0 says I am v2:172.21.2.102:38490/0 (socket says 172.21.2.102:38490) 2021-02-10T11:33:08.235 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:08.215 7f3b6db6d700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f3b681b0af0 0x7f3b68108950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:35060/0 (socket says 172.21.2.102:35060) 2021-02-10T11:33:08.235 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:08.215 7f3b6eb6f700 1 -- 172.21.2.102:0/3508062699 learned_addr learned my addr 172.21.2.102:0/3508062699 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:33:08.235 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:08.215 7f3b6e36e700 1 --2- 172.21.2.102:0/3508062699 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3b680d1880 0x7f3b681b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:08.235 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:08.215 7f3b6eb6f700 1 -- 172.21.2.102:0/3508062699 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3b680d1880 msgr2=0x7f3b681b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:33:08.235 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:08.215 7f3b6eb6f700 1 --2- 172.21.2.102:0/3508062699 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3b680d1880 0x7f3b681b04d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:08.235 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:08.215 7f3b6eb6f700 1 -- 172.21.2.102:0/3508062699 >> v2:172.21.2.102:3300/0 conn(0x7f3b681b0af0 msgr2=0x7f3b68108950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:33:08.236 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:08.215 7f3b6eb6f700 1 --2- 172.21.2.102:0/3508062699 >> v2:172.21.2.102:3300/0 conn(0x7f3b681b0af0 0x7f3b68108950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:08.236 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:08.216 7f3b6eb6f700 1 -- 172.21.2.102:0/3508062699 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b68109590 con 0x7f3b680d1c80 2021-02-10T11:33:08.236 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:08.216 7f3b6eb6f700 1 --2- 172.21.2.102:0/3508062699 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3b680d1c80 0x7f3b68108f70 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 rx=0x7f3b60006aa0 tx=0x7f3b60009f20).ready entity=mon.1 client_cookie=c1bc3ff355272c74 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:08.236 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:08.216 7f3b6d36c700 1 -- 172.21.2.102:0/3508062699 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3b6000b5c0 con 0x7f3b680d1c80 2021-02-10T11:33:08.236 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:08.216 7f3b6d36c700 1 -- 172.21.2.102:0/3508062699 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b60003030 con 0x7f3b680d1c80 2021-02-10T11:33:08.236 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:08.216 7f3b6d36c700 1 -- 172.21.2.102:0/3508062699 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3b6000bac0 con 0x7f3b680d1c80 2021-02-10T11:33:08.236 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:08.216 7f3b70552700 1 -- 172.21.2.102:0/3508062699 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3b680d1c80 msgr2=0x7f3b68108f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:08.236 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:08.216 7f3b70552700 1 --2- 172.21.2.102:0/3508062699 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3b680d1c80 0x7f3b68108f70 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 rx=0x7f3b60006aa0 tx=0x7f3b60009f20).stop 2021-02-10T11:33:08.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:08.217 7f3b70552700 1 -- 172.21.2.102:0/3508062699 shutdown_connections 2021-02-10T11:33:08.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:08.217 7f3b70552700 1 --2- 172.21.2.102:0/3508062699 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3b680d1c80 0x7f3b68108f70 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:08.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:08.217 7f3b70552700 1 --2- 172.21.2.102:0/3508062699 >> v2:172.21.2.102:3300/0 conn(0x7f3b681b0af0 0x7f3b68108950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:08.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:08.217 7f3b70552700 1 --2- 172.21.2.102:0/3508062699 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3b680d1880 0x7f3b681b04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:08.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:08.217 7f3b70552700 1 -- 172.21.2.102:0/3508062699 >> 172.21.2.102:0/3508062699 conn(0x7f3b680ce430 msgr2=0x7f3b680d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:08.238 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:08.217 7f3b70552700 1 -- 172.21.2.102:0/3508062699 shutdown_connections 2021-02-10T11:33:08.238 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:08.217 7f3b70552700 1 -- 172.21.2.102:0/3508062699 wait complete. 2021-02-10T11:33:08.238 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:08.231 7f3b70552700 1 Processor -- start 2021-02-10T11:33:08.238 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:08.232 7f3b70552700 1 -- start start 2021-02-10T11:33:08.240 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:08.232 7f3b70552700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3b681a0580 0x7f3b68082c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:08.240 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:08.232 7f3b70552700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3b680832a0 0x7f3b680836a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:08.241 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:08.232 7f3b70552700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3b680b66d0 0x7f3b680b6ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:08.241 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:08.232 7f3b70552700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b680ba540 con 0x7f3b680832a0 2021-02-10T11:33:08.241 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:08.232 7f3b70552700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b680baac0 con 0x7f3b680b66d0 2021-02-10T11:33:08.241 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:08.232 7f3b70552700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b680b70f0 con 0x7f3b681a0580 2021-02-10T11:33:08.241 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:08.232 7f3b6db6d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3b680832a0 0x7f3b680836a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:08.242 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:08.232 7f3b6db6d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3b680832a0 0x7f3b680836a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:35064/0 (socket says 172.21.2.102:35064) 2021-02-10T11:33:08.242 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:08.232 7f3b6db6d700 1 -- 172.21.2.102:0/3915720762 learned_addr learned my addr 172.21.2.102:0/3915720762 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:33:08.242 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:08.232 7f3b6e36e700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3b681a0580 0x7f3b68082c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:08.242 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:08.232 7f3b6eb6f700 1 --2- 172.21.2.102:0/3915720762 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3b680b66d0 0x7f3b680b6ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:08.242 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:08.233 7f3b6e36e700 1 -- 172.21.2.102:0/3915720762 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3b680b66d0 msgr2=0x7f3b680b6ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:08.243 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:08.233 7f3b6e36e700 1 --2- 172.21.2.102:0/3915720762 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3b680b66d0 0x7f3b680b6ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:08.243 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:08.233 7f3b6e36e700 1 -- 172.21.2.102:0/3915720762 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3b680832a0 msgr2=0x7f3b680836a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:08.243 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:08.233 7f3b6e36e700 1 --2- 172.21.2.102:0/3915720762 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3b680832a0 0x7f3b680836a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:08.243 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:08.233 7f3b6e36e700 1 -- 172.21.2.102:0/3915720762 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b680b7310 con 0x7f3b681a0580 2021-02-10T11:33:08.243 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:08.233 7f3b6e36e700 1 --2- 172.21.2.102:0/3915720762 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3b681a0580 0x7f3b68082c80 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 rx=0x7f3b5c009540 tx=0x7f3b5c00b8e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:08.243 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:08.233 7f3b5affd700 1 -- 172.21.2.102:0/3915720762 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3b5c00bcd0 con 0x7f3b681a0580 2021-02-10T11:33:08.244 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:08.233 7f3b5affd700 1 -- 172.21.2.102:0/3915720762 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b5c006740 con 0x7f3b681a0580 2021-02-10T11:33:08.244 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:08.233 7f3b70552700 1 -- 172.21.2.102:0/3915720762 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b680b7310 con 0x7f3b681a0580 2021-02-10T11:33:08.244 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:08.233 7f3b5affd700 1 -- 172.21.2.102:0/3915720762 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3b5c002030 con 0x7f3b681a0580 2021-02-10T11:33:08.244 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:08.233 7f3b70552700 1 -- 172.21.2.102:0/3915720762 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b680b7310 con 0x7f3b681a0580 2021-02-10T11:33:08.244 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:08.234 7f3b5affd700 1 -- 172.21.2.102:0/3915720762 <== mon.2 v2:172.21.2.113:3301/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f3b5c000c90 con 0x7f3b681a0580 2021-02-10T11:33:08.245 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:08.234 7f3b5affd700 1 --2- 172.21.2.102:0/3915720762 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3b50018730 0x7f3b5001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:08.245 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:08.234 7f3b5affd700 1 -- 172.21.2.102:0/3915720762 <== mon.2 v2:172.21.2.113:3301/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f3b5c0326f0 con 0x7f3b681a0580 2021-02-10T11:33:08.245 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:08.234 7f3b6db6d700 1 --2- 172.21.2.102:0/3915720762 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3b50018730 0x7f3b5001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:08.245 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:08.234 7f3b6db6d700 1 --2- 172.21.2.102:0/3915720762 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3b50018730 0x7f3b5001ac10 crc :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:08.266 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:08.265 7f3b70552700 1 -- 172.21.2.102:0/3915720762 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b68082a50 con 0x7f3b681a0580 2021-02-10T11:33:08.269 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:08.268 7f3b5affd700 1 -- 172.21.2.102:0/3915720762 <== mon.2 v2:172.21.2.113:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f3b5c006ae0 con 0x7f3b681a0580 2021-02-10T11:33:08.401 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:08.398 7f3b70552700 1 -- 172.21.2.102:0/3915720762 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000594354", "target": ["mgr", ""], "group_name": "group_0000000000668580"}) v1 -- 0x7f3b68082a50 con 0x7f3b50018730 2021-02-10T11:33:08.412 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:08.411 7f3b5affd700 1 -- 172.21.2.102:0/3915720762 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f3b640009b0 con 0x7f3b50018730 2021-02-10T11:33:08.416 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:08.415 7f3b70552700 1 -- 172.21.2.102:0/3915720762 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3b50018730 msgr2=0x7f3b5001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:08.416 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:08.415 7f3b70552700 1 --2- 172.21.2.102:0/3915720762 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3b50018730 0x7f3b5001ac10 crc :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:08.416 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:08.415 7f3b70552700 1 -- 172.21.2.102:0/3915720762 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3b681a0580 msgr2=0x7f3b68082c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:08.416 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:08.415 7f3b70552700 1 --2- 172.21.2.102:0/3915720762 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3b681a0580 0x7f3b68082c80 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 rx=0x7f3b5c009540 tx=0x7f3b5c00b8e0).stop 2021-02-10T11:33:08.417 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:08.415 7f3b70552700 1 -- 172.21.2.102:0/3915720762 shutdown_connections 2021-02-10T11:33:08.417 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:08.415 7f3b70552700 1 --2- 172.21.2.102:0/3915720762 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3b50018730 0x7f3b5001ac10 unknown :-1 s=CLOSED pgs=1041 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:08.417 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:08.415 7f3b70552700 1 --2- 172.21.2.102:0/3915720762 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3b680b66d0 0x7f3b680b6ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:08.417 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:08.415 7f3b70552700 1 --2- 172.21.2.102:0/3915720762 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3b680832a0 0x7f3b680836a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:08.417 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:08.415 7f3b70552700 1 --2- 172.21.2.102:0/3915720762 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3b681a0580 0x7f3b68082c80 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:08.417 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:08.415 7f3b70552700 1 -- 172.21.2.102:0/3915720762 >> 172.21.2.102:0/3915720762 conn(0x7f3b680c20f0 msgr2=0x7f3b6809e410 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:08.417 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:08.415 7f3b70552700 1 -- 172.21.2.102:0/3915720762 shutdown_connections 2021-02-10T11:33:08.417 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:08.415 7f3b70552700 1 -- 172.21.2.102:0/3915720762 wait complete. 2021-02-10T11:33:08.426 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.1 && sudo ls volumes 2021-02-10T11:33:08.774 INFO:teuthology.orchestra.run.gibba002.stdout:_deleting 2021-02-10T11:33:08.774 INFO:teuthology.orchestra.run.gibba002.stdout:group_0000000000668580 2021-02-10T11:33:08.775 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:33:08.775 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:33:08.775 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:33:09.110 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:33:09.110 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2021-02-10T11:33:09.110 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@gibba002.front.sepia.ceph.com... 2021-02-10T11:33:09.111 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-02-10T11:33:09.111 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:33:09.464 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba002.stderr:ceph-fuse[186462]: fuse finished with error 0 and tester_r 0 2021-02-10T11:33:09.465 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:33:09.465 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:33:09.465 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:33:09.801 INFO:teuthology.orchestra.run.gibba002.stdout:ext2/ext3 2021-02-10T11:33:09.801 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2021-02-10T11:33:09.801 INFO:teuthology.orchestra.run:waiting for 900 2021-02-10T11:33:09.802 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:33:09.802 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:33:10.138 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolumegroup rm cephfs group_0000000000668580 2021-02-10T11:33:10.571 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:10.552 7f0dc024c700 1 Processor -- start 2021-02-10T11:33:10.571 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:10.552 7f0dc024c700 1 -- start start 2021-02-10T11:33:10.571 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:10.553 7f0dc024c700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0db80d1870 0x7f0db81b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:10.571 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:10.553 7f0dc024c700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f0db81b0ae0 0x7f0db8108940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:10.571 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:10.553 7f0dc024c700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0db8109a60 con 0x7f0db80d1c70 2021-02-10T11:33:10.571 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:10.553 7f0dc024c700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0db80e5ef0 con 0x7f0db81b0ae0 2021-02-10T11:33:10.571 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:10.553 7f0dc024c700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0db8109750 con 0x7f0db80d1870 2021-02-10T11:33:10.571 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:10.553 7f0dbe869700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f0db80d1c70 0x7f0db8108f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:55530/0 (socket says 172.21.2.102:55530) 2021-02-10T11:33:10.572 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:10.553 7f0dbe869700 1 -- 172.21.2.102:0/1117535264 learned_addr learned my addr 172.21.2.102:0/1117535264 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:33:10.572 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:10.553 7f0dbd867700 1 --2- 172.21.2.102:0/1117535264 >> v2:172.21.2.113:3300/0 conn(0x7f0db81b0ae0 0x7f0db8108940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:10.572 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:10.553 7f0dbe068700 1 --2- 172.21.2.102:0/1117535264 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0db80d1870 0x7f0db81b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:10.572 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:10.554 7f0dbd867700 1 -- 172.21.2.102:0/1117535264 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0db80d1870 msgr2=0x7f0db81b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:33:10.572 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:10.554 7f0dbd867700 1 --2- 172.21.2.102:0/1117535264 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0db80d1870 0x7f0db81b04c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:10.572 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:10.554 7f0dbd867700 1 -- 172.21.2.102:0/1117535264 >> v1:172.21.2.102:6789/0 conn(0x7f0db80d1c70 legacy=0x7f0db8108f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:10.572 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:10.554 7f0dbd867700 1 -- 172.21.2.102:0/1117535264 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0db8088070 con 0x7f0db81b0ae0 2021-02-10T11:33:10.572 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:10.554 7f0dbd066700 1 -- 172.21.2.102:0/1117535264 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 623835032 0 0) 0x7f0db0001180 con 0x7f0db80d1c70 2021-02-10T11:33:10.572 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:10.554 7f0dbd867700 1 --2- 172.21.2.102:0/1117535264 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0db81b0ae0 0x7f0db8108940 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 rx=0x7f0db4006650 tx=0x7f0db4009a80).ready entity=mon.1 client_cookie=b2419513164117e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:10.572 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:10.554 7f0dbd066700 1 -- 172.21.2.102:0/1117535264 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0db400c0c0 con 0x7f0db81b0ae0 2021-02-10T11:33:10.572 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:10.554 7f0dbd066700 1 -- 172.21.2.102:0/1117535264 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0db400c340 con 0x7f0db81b0ae0 2021-02-10T11:33:10.572 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:10.554 7f0dbd066700 1 -- 172.21.2.102:0/1117535264 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0db400c860 con 0x7f0db81b0ae0 2021-02-10T11:33:10.572 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:10.554 7f0dc024c700 1 -- 172.21.2.102:0/1117535264 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0db81b0ae0 msgr2=0x7f0db8108940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:10.572 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:10.554 7f0dc024c700 1 --2- 172.21.2.102:0/1117535264 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0db81b0ae0 0x7f0db8108940 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 rx=0x7f0db4006650 tx=0x7f0db4009a80).stop 2021-02-10T11:33:10.573 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:10.555 7f0dc024c700 1 -- 172.21.2.102:0/1117535264 shutdown_connections 2021-02-10T11:33:10.573 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:10.555 7f0dc024c700 1 --2- 172.21.2.102:0/1117535264 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0db81b0ae0 0x7f0db8108940 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:10.573 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:10.555 7f0dc024c700 1 --2- 172.21.2.102:0/1117535264 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0db80d1870 0x7f0db81b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:10.573 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:10.555 7f0dc024c700 1 -- 172.21.2.102:0/1117535264 >> 172.21.2.102:0/1117535264 conn(0x7f0db80ce420 msgr2=0x7f0db80d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:10.573 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:10.555 7f0dc024c700 1 -- 172.21.2.102:0/1117535264 shutdown_connections 2021-02-10T11:33:10.573 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:10.555 7f0dc024c700 1 -- 172.21.2.102:0/1117535264 wait complete. 2021-02-10T11:33:10.574 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:10.570 7f0dc024c700 1 Processor -- start 2021-02-10T11:33:10.574 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:10.570 7f0dc024c700 1 -- start start 2021-02-10T11:33:10.575 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:10.570 7f0dc024c700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0db80a58c0 0x7f0db80a4d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:10.575 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:10.570 7f0dc024c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0db80b1ba0 0x7f0db80b1fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:10.575 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:10.570 7f0dc024c700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0db80b25c0 0x7f0db80b3270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:10.575 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:10.570 7f0dc024c700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0db80b2a40 con 0x7f0db80b1ba0 2021-02-10T11:33:10.575 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:10.570 7f0dc024c700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0db80a5340 con 0x7f0db80a58c0 2021-02-10T11:33:10.575 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:10.570 7f0dc024c700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0db80a5560 con 0x7f0db80b25c0 2021-02-10T11:33:10.575 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:10.571 7f0dbd867700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0db80b1ba0 0x7f0db80b1fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:10.575 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:10.571 7f0dbd867700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0db80b1ba0 0x7f0db80b1fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:35078/0 (socket says 172.21.2.102:35078) 2021-02-10T11:33:10.575 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:10.571 7f0dbd867700 1 -- 172.21.2.102:0/850752124 learned_addr learned my addr 172.21.2.102:0/850752124 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:33:10.575 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:10.571 7f0dbe869700 1 --2- 172.21.2.102:0/850752124 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0db80b25c0 0x7f0db80b3270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:10.575 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:10.571 7f0dbe068700 1 --2- 172.21.2.102:0/850752124 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0db80a58c0 0x7f0db80a4d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:10.575 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:10.571 7f0dbd867700 1 -- 172.21.2.102:0/850752124 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0db80b25c0 msgr2=0x7f0db80b3270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:10.575 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:10.571 7f0dbd867700 1 --2- 172.21.2.102:0/850752124 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0db80b25c0 0x7f0db80b3270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:10.576 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:10.571 7f0dbd867700 1 -- 172.21.2.102:0/850752124 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0db80a58c0 msgr2=0x7f0db80a4d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:10.576 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:10.571 7f0dbd867700 1 --2- 172.21.2.102:0/850752124 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0db80a58c0 0x7f0db80a4d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:10.576 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:10.571 7f0dbd867700 1 -- 172.21.2.102:0/850752124 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0db80b3890 con 0x7f0db80b1ba0 2021-02-10T11:33:10.576 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:10.571 7f0dbd867700 1 --2- 172.21.2.102:0/850752124 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0db80b1ba0 0x7f0db80b1fa0 secure :-1 s=READY pgs=2365 cs=0 l=1 rev1=1 rx=0x7f0db4009630 tx=0x7f0db400b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:10.576 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:10.571 7f0daaffd700 1 -- 172.21.2.102:0/850752124 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0db40093a0 con 0x7f0db80b1ba0 2021-02-10T11:33:10.576 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:10.571 7f0daaffd700 1 -- 172.21.2.102:0/850752124 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0db400bd70 con 0x7f0db80b1ba0 2021-02-10T11:33:10.576 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:10.571 7f0dc024c700 1 -- 172.21.2.102:0/850752124 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0db80b3890 con 0x7f0db80b1ba0 2021-02-10T11:33:10.576 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:10.571 7f0daaffd700 1 -- 172.21.2.102:0/850752124 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0db40054d0 con 0x7f0db80b1ba0 2021-02-10T11:33:10.576 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:10.571 7f0dc024c700 1 -- 172.21.2.102:0/850752124 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0db80b3de0 con 0x7f0db80b1ba0 2021-02-10T11:33:10.576 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:10.572 7f0daaffd700 1 -- 172.21.2.102:0/850752124 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f0db40052e0 con 0x7f0db80b1ba0 2021-02-10T11:33:10.576 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:10.572 7f0daaffd700 1 --2- 172.21.2.102:0/850752124 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0da00186c0 0x7f0da001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:10.576 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:10.572 7f0daaffd700 1 -- 172.21.2.102:0/850752124 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f0db4032580 con 0x7f0db80b1ba0 2021-02-10T11:33:10.576 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:10.572 7f0dbe068700 1 --2- 172.21.2.102:0/850752124 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0da00186c0 0x7f0da001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:10.576 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:10.573 7f0dbe068700 1 --2- 172.21.2.102:0/850752124 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0da00186c0 0x7f0da001abc0 crc :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:10.584 DEBUG:teuthology.orchestra.run.gibba002:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:33:10.587 DEBUG:teuthology.orchestra.run.gibba013:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:33:10.604 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:10.603 7f0dc024c700 1 -- 172.21.2.102:0/850752124 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0db8082980 con 0x7f0db80b1ba0 2021-02-10T11:33:10.607 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:10.606 7f0daaffd700 1 -- 172.21.2.102:0/850752124 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f0db4009340 con 0x7f0db80b1ba0 2021-02-10T11:33:10.736 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:10.735 7f0dc024c700 1 -- 172.21.2.102:0/850752124 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolumegroup rm", "vol_name": "cephfs", "target": ["mgr", ""], "group_name": "group_0000000000668580"}) v1 -- 0x7f0db80b3de0 con 0x7f0da00186c0 2021-02-10T11:33:10.738 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:10.738 7f0daaffd700 1 -- 172.21.2.102:0/850752124 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f0dac006bf0 con 0x7f0da00186c0 2021-02-10T11:33:10.741 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:10.740 7f0dc024c700 1 -- 172.21.2.102:0/850752124 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0da00186c0 msgr2=0x7f0da001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:10.741 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:10.740 7f0dc024c700 1 --2- 172.21.2.102:0/850752124 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0da00186c0 0x7f0da001abc0 crc :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:10.741 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:10.740 7f0dc024c700 1 -- 172.21.2.102:0/850752124 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0db80b1ba0 msgr2=0x7f0db80b1fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:10.741 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:10.740 7f0dc024c700 1 --2- 172.21.2.102:0/850752124 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0db80b1ba0 0x7f0db80b1fa0 secure :-1 s=READY pgs=2365 cs=0 l=1 rev1=1 rx=0x7f0db4009630 tx=0x7f0db400b980).stop 2021-02-10T11:33:10.742 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:10.740 7f0dc024c700 1 -- 172.21.2.102:0/850752124 shutdown_connections 2021-02-10T11:33:10.742 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:10.740 7f0dc024c700 1 --2- 172.21.2.102:0/850752124 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0db80b25c0 0x7f0db80b3270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:10.742 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:10.740 7f0dc024c700 1 --2- 172.21.2.102:0/850752124 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0db80b1ba0 0x7f0db80b1fa0 unknown :-1 s=CLOSED pgs=2365 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:10.742 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:10.740 7f0dc024c700 1 --2- 172.21.2.102:0/850752124 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0da00186c0 0x7f0da001abc0 unknown :-1 s=CLOSED pgs=1047 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:10.742 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:10.740 7f0dc024c700 1 --2- 172.21.2.102:0/850752124 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0db80a58c0 0x7f0db80a4d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:10.742 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:10.740 7f0dc024c700 1 -- 172.21.2.102:0/850752124 >> 172.21.2.102:0/850752124 conn(0x7f0db80b5dc0 msgr2=0x7f0db8095960 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:10.742 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:10.740 7f0dc024c700 1 -- 172.21.2.102:0/850752124 shutdown_connections 2021-02-10T11:33:10.742 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:10.740 7f0dc024c700 1 -- 172.21.2.102:0/850752124 wait complete. 2021-02-10T11:33:10.750 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2021-02-10T11:33:10.750 DEBUG:teuthology.orchestra.run.gibba002:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:33:11.136 DEBUG:teuthology.orchestra.run.gibba013:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:33:11.481 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2021-02-10T11:33:11.481 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:33:11.481 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:33:11.481 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:33:11.817 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:33:11.817 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:33:11.818 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@gibba002.front.sepia.ceph.com... 2021-02-10T11:33:11.818 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-02-10T11:33:11.818 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:33:12.174 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba002.stderr:ceph-fuse[185822]: fuse finished with error 0 and tester_r 0 2021-02-10T11:33:12.175 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:33:12.175 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:33:12.175 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:33:12.509 INFO:teuthology.orchestra.run.gibba002.stdout:ext2/ext3 2021-02-10T11:33:12.510 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2021-02-10T11:33:12.510 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:33:12.510 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:33:12.845 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:33:12.845 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:33:12.846 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:33:13.182 INFO:teuthology.orchestra.run.gibba002.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.1’: No such file or directory 2021-02-10T11:33:13.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:33:13.182 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2021-02-10T11:33:13.183 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:33:13.183 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:33:13.518 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:33:13.518 DEBUG:teuthology.orchestra.run.gibba013:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:33:13.518 DEBUG:teuthology.orchestra.run.gibba013:> ' -- /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:33:13.851 INFO:teuthology.orchestra.run.gibba013.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.2’: No such file or directory 2021-02-10T11:33:13.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:33:13.852 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2021-02-10T11:33:13.852 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:33:13.852 DEBUG:teuthology.orchestra.run.gibba013:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:33:14.187 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:33:14.187 DEBUG:teuthology.orchestra.run.gibba013:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:33:14.187 DEBUG:teuthology.orchestra.run.gibba013:> ' -- /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:33:14.520 INFO:teuthology.orchestra.run.gibba013.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.3’: No such file or directory 2021-02-10T11:33:14.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:33:14.520 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.3 2021-02-10T11:33:14.521 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:33:14.521 DEBUG:teuthology.orchestra.run.gibba013:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:33:14.857 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config rm mon mon_allow_pool_delete 2021-02-10T11:33:15.289 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:15.270 7fbc493a9700 1 Processor -- start 2021-02-10T11:33:15.289 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:15.270 7fbc493a9700 1 -- start start 2021-02-10T11:33:15.289 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:15.271 7fbc493a9700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbc440d0340 0x7fbc440ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:15.289 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:15.271 7fbc493a9700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc440f0b10 con 0x7fbc440d1c70 2021-02-10T11:33:15.289 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:15.271 7fbc493a9700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc441b0be0 con 0x7fbc440d1870 2021-02-10T11:33:15.289 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:15.271 7fbc493a9700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc440e5ed0 con 0x7fbc440d0340 2021-02-10T11:33:15.289 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:15.271 7fbc4387e700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fbc440d1c70 0x7fbc440efc60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:55546/0 (socket says 172.21.2.102:55546) 2021-02-10T11:33:15.289 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:15.271 7fbc4387e700 1 -- 172.21.2.102:0/798564890 learned_addr learned my addr 172.21.2.102:0/798564890 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:33:15.289 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:15.271 7fbc4287c700 1 --2- 172.21.2.102:0/798564890 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbc440d0340 0x7fbc440ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:15.290 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:15.271 7fbc4207b700 1 -- 172.21.2.102:0/798564890 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2565521611 0 0) 0x7fbc3c001500 con 0x7fbc440d1c70 2021-02-10T11:33:15.290 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:15.271 7fbc4207b700 1 -- 172.21.2.102:0/798564890 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc2c002fe0 con 0x7fbc440d1c70 2021-02-10T11:33:15.290 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:15.272 7fbc4207b700 1 -- 172.21.2.102:0/798564890 <== mon.1 v1:172.21.2.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1804160825 0 0) 0x7fbc34000c60 con 0x7fbc440d1870 2021-02-10T11:33:15.290 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:15.272 7fbc4287c700 1 -- 172.21.2.102:0/798564890 >> v1:172.21.2.113:6789/0 conn(0x7fbc440d1870 legacy=0x7fbc441b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:15.290 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:15.272 7fbc4287c700 1 -- 172.21.2.102:0/798564890 >> v1:172.21.2.102:6789/0 conn(0x7fbc440d1c70 legacy=0x7fbc440efc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:15.290 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:15.272 7fbc4287c700 1 -- 172.21.2.102:0/798564890 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc440f0450 con 0x7fbc440d0340 2021-02-10T11:33:15.290 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:15.272 7fbc4287c700 1 --2- 172.21.2.102:0/798564890 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbc440d0340 0x7fbc440ef640 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0x7fbc38006650 tx=0x7fbc38009a00).ready entity=mon.2 client_cookie=9428b6c0a94bb8a9 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:15.290 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:15.272 7fbc4207b700 1 -- 172.21.2.102:0/798564890 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbc380009b0 con 0x7fbc440d0340 2021-02-10T11:33:15.290 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:15.272 7fbc4207b700 1 -- 172.21.2.102:0/798564890 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc3800c0c0 con 0x7fbc440d0340 2021-02-10T11:33:15.290 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:15.272 7fbc4207b700 1 -- 172.21.2.102:0/798564890 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbc3800c5e0 con 0x7fbc440d0340 2021-02-10T11:33:15.290 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:15.272 7fbc493a9700 1 -- 172.21.2.102:0/798564890 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbc440d0340 msgr2=0x7fbc440ef640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:15.290 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:15.272 7fbc493a9700 1 --2- 172.21.2.102:0/798564890 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbc440d0340 0x7fbc440ef640 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0x7fbc38006650 tx=0x7fbc38009a00).stop 2021-02-10T11:33:15.291 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:15.273 7fbc493a9700 1 -- 172.21.2.102:0/798564890 shutdown_connections 2021-02-10T11:33:15.291 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:15.273 7fbc493a9700 1 --2- 172.21.2.102:0/798564890 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbc440d0340 0x7fbc440ef640 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:15.291 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:15.273 7fbc493a9700 1 -- 172.21.2.102:0/798564890 >> 172.21.2.102:0/798564890 conn(0x7fbc440ce450 msgr2=0x7fbc440d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:15.291 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:15.273 7fbc493a9700 1 -- 172.21.2.102:0/798564890 shutdown_connections 2021-02-10T11:33:15.291 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:15.273 7fbc493a9700 1 -- 172.21.2.102:0/798564890 wait complete. 2021-02-10T11:33:15.292 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:15.287 7fbc493a9700 1 Processor -- start 2021-02-10T11:33:15.292 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:15.288 7fbc493a9700 1 -- start start 2021-02-10T11:33:15.292 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:15.288 7fbc493a9700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbc4409e770 0x7fbc440aeae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:15.292 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:15.288 7fbc493a9700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fbc4419cdb0 0x7fbc4409ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:15.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:15.288 7fbc493a9700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbc4419d1b0 0x7fbc4409f220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:15.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:15.288 7fbc493a9700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc4409e260 con 0x7fbc4419d1b0 2021-02-10T11:33:15.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:15.288 7fbc493a9700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc4409de80 con 0x7fbc4419cdb0 2021-02-10T11:33:15.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:15.288 7fbc493a9700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc4409e070 con 0x7fbc4409e770 2021-02-10T11:33:15.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:15.288 7fbc4387e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbc4419d1b0 0x7fbc4409f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:15.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:15.288 7fbc4387e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbc4419d1b0 0x7fbc4409f220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:35092/0 (socket says 172.21.2.102:35092) 2021-02-10T11:33:15.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:15.288 7fbc4387e700 1 -- 172.21.2.102:0/957859411 learned_addr learned my addr 172.21.2.102:0/957859411 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:33:15.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:15.288 7fbc4287c700 1 --2- 172.21.2.102:0/957859411 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fbc4419cdb0 0x7fbc4409ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:15.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:15.288 7fbc4387e700 1 -- 172.21.2.102:0/957859411 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbc4409e770 msgr2=0x7fbc440aeae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:15.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:15.288 7fbc4387e700 1 --2- 172.21.2.102:0/957859411 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbc4409e770 0x7fbc440aeae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:15.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:15.288 7fbc4387e700 1 -- 172.21.2.102:0/957859411 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fbc4419cdb0 msgr2=0x7fbc4409ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:15.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:15.288 7fbc4387e700 1 --2- 172.21.2.102:0/957859411 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fbc4419cdb0 0x7fbc4409ec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:15.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:15.288 7fbc4387e700 1 -- 172.21.2.102:0/957859411 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc440a7930 con 0x7fbc4419d1b0 2021-02-10T11:33:15.294 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:15.288 7fbc4287c700 1 --2- 172.21.2.102:0/957859411 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fbc4419cdb0 0x7fbc4409ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:33:15.294 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:15.288 7fbc4387e700 1 --2- 172.21.2.102:0/957859411 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbc4419d1b0 0x7fbc4409f220 secure :-1 s=READY pgs=2367 cs=0 l=1 rev1=1 rx=0x7fbc3c00a900 tx=0x7fbc3c00aca0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:15.294 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:15.289 7fbc2bfff700 1 -- 172.21.2.102:0/957859411 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbc3c001270 con 0x7fbc4419d1b0 2021-02-10T11:33:15.294 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:15.289 7fbc2bfff700 1 -- 172.21.2.102:0/957859411 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc3c0070d0 con 0x7fbc4419d1b0 2021-02-10T11:33:15.294 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:15.289 7fbc493a9700 1 -- 172.21.2.102:0/957859411 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc440a7930 con 0x7fbc4419d1b0 2021-02-10T11:33:15.294 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:15.289 7fbc493a9700 1 -- 172.21.2.102:0/957859411 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc440a7930 con 0x7fbc4419d1b0 2021-02-10T11:33:15.294 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:15.289 7fbc2bfff700 1 -- 172.21.2.102:0/957859411 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbc3c004720 con 0x7fbc4419d1b0 2021-02-10T11:33:15.294 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:15.289 7fbc2bfff700 1 -- 172.21.2.102:0/957859411 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fbc3c0072b0 con 0x7fbc4419d1b0 2021-02-10T11:33:15.294 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:15.289 7fbc2bfff700 1 --2- 172.21.2.102:0/957859411 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fbc24018730 0x7fbc2401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:15.294 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:15.289 7fbc2bfff700 1 -- 172.21.2.102:0/957859411 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fbc3c032b50 con 0x7fbc4419d1b0 2021-02-10T11:33:15.294 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:15.289 7fbc4307d700 1 --2- 172.21.2.102:0/957859411 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fbc24018730 0x7fbc2401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:15.294 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:15.290 7fbc4307d700 1 --2- 172.21.2.102:0/957859411 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fbc24018730 0x7fbc2401ac10 crc :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:15.322 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:15.321 7fbc493a9700 1 -- 172.21.2.102:0/957859411 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc440ebc40 con 0x7fbc4419d1b0 2021-02-10T11:33:15.324 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:15.323 7fbc2bfff700 1 -- 172.21.2.102:0/957859411 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fbc3c007c90 con 0x7fbc4419d1b0 2021-02-10T11:33:15.474 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:15.472 7fbc493a9700 1 -- 172.21.2.102:0/957859411 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fbc440ebc40 con 0x7fbc4419d1b0 2021-02-10T11:33:15.477 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:15.476 7fbc2bfff700 1 -- 172.21.2.102:0/957859411 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v31) v1 ==== 104+0+0 (secure 0 0 0) 0x7fbc3c0070d0 con 0x7fbc4419d1b0 2021-02-10T11:33:15.482 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:15.481 7fbc493a9700 1 -- 172.21.2.102:0/957859411 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fbc24018730 msgr2=0x7fbc2401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:15.483 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:15.481 7fbc493a9700 1 --2- 172.21.2.102:0/957859411 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fbc24018730 0x7fbc2401ac10 crc :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:15.483 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:15.481 7fbc493a9700 1 -- 172.21.2.102:0/957859411 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbc4419d1b0 msgr2=0x7fbc4409f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:15.483 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:15.481 7fbc493a9700 1 --2- 172.21.2.102:0/957859411 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbc4419d1b0 0x7fbc4409f220 secure :-1 s=READY pgs=2367 cs=0 l=1 rev1=1 rx=0x7fbc3c00a900 tx=0x7fbc3c00aca0).stop 2021-02-10T11:33:15.483 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:15.481 7fbc493a9700 1 -- 172.21.2.102:0/957859411 shutdown_connections 2021-02-10T11:33:15.484 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:15.481 7fbc493a9700 1 --2- 172.21.2.102:0/957859411 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbc4419d1b0 0x7fbc4409f220 unknown :-1 s=CLOSED pgs=2367 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:15.484 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:15.481 7fbc493a9700 1 --2- 172.21.2.102:0/957859411 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fbc24018730 0x7fbc2401ac10 unknown :-1 s=CLOSED pgs=1048 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:15.484 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:15.481 7fbc493a9700 1 --2- 172.21.2.102:0/957859411 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fbc4419cdb0 0x7fbc4409ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:15.484 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:15.481 7fbc493a9700 1 --2- 172.21.2.102:0/957859411 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbc4409e770 0x7fbc440aeae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:15.484 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:15.481 7fbc493a9700 1 -- 172.21.2.102:0/957859411 >> 172.21.2.102:0/957859411 conn(0x7fbc440a7460 msgr2=0x7fbc44091a50 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:15.484 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:15.482 7fbc493a9700 1 -- 172.21.2.102:0/957859411 shutdown_connections 2021-02-10T11:33:15.484 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:15.482 7fbc493a9700 1 -- 172.21.2.102:0/957859411 wait complete. 2021-02-10T11:33:15.491 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestVolumes.test_multitenant_subvolumes' 2021-02-10T11:33:15.919 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:15.901 7f085057e700 1 Processor -- start 2021-02-10T11:33:15.920 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:15.901 7f085057e700 1 -- start start 2021-02-10T11:33:15.920 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:15.902 7f085057e700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f08480d0340 0x7f08480ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:15.920 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:15.902 7f085057e700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08480f0b10 con 0x7f08480d1c70 2021-02-10T11:33:15.920 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:15.902 7f085057e700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08481b0be0 con 0x7f08480d1870 2021-02-10T11:33:15.920 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:15.902 7f085057e700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08480e5ed0 con 0x7f08480d0340 2021-02-10T11:33:15.920 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:15.902 7f084eb9b700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f08480d1c70 0x7f08480efc60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:55558/0 (socket says 172.21.2.102:55558) 2021-02-10T11:33:15.920 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:15.902 7f084eb9b700 1 -- 172.21.2.102:0/3856479318 learned_addr learned my addr 172.21.2.102:0/3856479318 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:33:15.920 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:15.902 7f084db99700 1 --2- 172.21.2.102:0/3856479318 >> v2:172.21.2.113:3300/0 conn(0x7f08480d0340 0x7f08480ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:15.920 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:15.902 7f084d398700 1 -- 172.21.2.102:0/3856479318 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3875257440 0 0) 0x7f0840001500 con 0x7f08480d1c70 2021-02-10T11:33:15.920 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:15.902 7f084d398700 1 -- 172.21.2.102:0/3856479318 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0834002fe0 con 0x7f08480d1c70 2021-02-10T11:33:15.920 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:15.903 7f084db99700 1 -- 172.21.2.102:0/3856479318 >> v1:172.21.2.113:6789/0 conn(0x7f08480d1870 legacy=0x7f08481b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:15.920 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:15.903 7f084db99700 1 -- 172.21.2.102:0/3856479318 >> v1:172.21.2.102:6789/0 conn(0x7f08480d1c70 legacy=0x7f08480efc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:15.921 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:15.903 7f084db99700 1 -- 172.21.2.102:0/3856479318 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08480f0450 con 0x7f08480d0340 2021-02-10T11:33:15.921 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:15.903 7f084db99700 1 --2- 172.21.2.102:0/3856479318 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f08480d0340 0x7f08480ef640 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 rx=0x7f083c006650 tx=0x7f083c009a80).ready entity=mon.1 client_cookie=164f402300bacba7 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:15.921 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:15.903 7f084d398700 1 -- 172.21.2.102:0/3856479318 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f083c00c0c0 con 0x7f08480d0340 2021-02-10T11:33:15.921 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:15.903 7f084d398700 1 -- 172.21.2.102:0/3856479318 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f083c00c340 con 0x7f08480d0340 2021-02-10T11:33:15.921 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:15.903 7f084d398700 1 -- 172.21.2.102:0/3856479318 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f083c00c860 con 0x7f08480d0340 2021-02-10T11:33:15.921 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:15.903 7f085057e700 1 -- 172.21.2.102:0/3856479318 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f08480d0340 msgr2=0x7f08480ef640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:15.921 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:15.903 7f085057e700 1 --2- 172.21.2.102:0/3856479318 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f08480d0340 0x7f08480ef640 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 rx=0x7f083c006650 tx=0x7f083c009a80).stop 2021-02-10T11:33:15.921 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:15.904 7f085057e700 1 -- 172.21.2.102:0/3856479318 shutdown_connections 2021-02-10T11:33:15.921 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:15.904 7f085057e700 1 --2- 172.21.2.102:0/3856479318 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f08480d0340 0x7f08480ef640 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:15.921 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:15.904 7f085057e700 1 -- 172.21.2.102:0/3856479318 >> 172.21.2.102:0/3856479318 conn(0x7f08480ce450 msgr2=0x7f08480d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:15.921 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:15.904 7f085057e700 1 -- 172.21.2.102:0/3856479318 shutdown_connections 2021-02-10T11:33:15.921 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:15.904 7f085057e700 1 -- 172.21.2.102:0/3856479318 wait complete. 2021-02-10T11:33:15.923 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:15.918 7f085057e700 1 Processor -- start 2021-02-10T11:33:15.923 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:15.918 7f085057e700 1 -- start start 2021-02-10T11:33:15.923 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:15.919 7f085057e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f08480f2520 0x7f08481b0e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:15.923 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:15.919 7f085057e700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f08481b14b0 0x7f08481afda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:15.923 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:15.919 7f085057e700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f08480f2920 0x7f08481b03c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:15.924 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:15.919 7f085057e700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08481b33e0 con 0x7f08480f2520 2021-02-10T11:33:15.924 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:15.919 7f085057e700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08481b2e60 con 0x7f08481b14b0 2021-02-10T11:33:15.924 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:15.919 7f085057e700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08481b09e0 con 0x7f08480f2920 2021-02-10T11:33:15.924 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:15.919 7f084e39a700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f08480f2520 0x7f08481b0e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:15.924 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:15.919 7f084e39a700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f08480f2520 0x7f08481b0e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:35106/0 (socket says 172.21.2.102:35106) 2021-02-10T11:33:15.924 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:15.919 7f084e39a700 1 -- 172.21.2.102:0/605144082 learned_addr learned my addr 172.21.2.102:0/605144082 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:33:15.924 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:15.919 7f084db99700 1 --2- 172.21.2.102:0/605144082 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f08481b14b0 0x7f08481afda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:15.924 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:15.919 7f084eb9b700 1 --2- 172.21.2.102:0/605144082 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f08480f2920 0x7f08481b03c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:15.924 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:15.919 7f084e39a700 1 -- 172.21.2.102:0/605144082 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f08480f2920 msgr2=0x7f08481b03c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:15.924 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:15.919 7f084e39a700 1 --2- 172.21.2.102:0/605144082 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f08480f2920 0x7f08481b03c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:15.924 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:15.919 7f084e39a700 1 -- 172.21.2.102:0/605144082 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f08481b14b0 msgr2=0x7f08481afda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:15.924 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:15.919 7f084e39a700 1 --2- 172.21.2.102:0/605144082 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f08481b14b0 0x7f08481afda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:15.924 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:15.919 7f084e39a700 1 -- 172.21.2.102:0/605144082 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08481b0bc0 con 0x7f08480f2520 2021-02-10T11:33:15.925 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:15.919 7f084e39a700 1 --2- 172.21.2.102:0/605144082 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f08480f2520 0x7f08481b0e90 secure :-1 s=READY pgs=2369 cs=0 l=1 rev1=1 rx=0x7f084400b5b0 tx=0x7f084400b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:15.925 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:15.920 7f083affd700 1 -- 172.21.2.102:0/605144082 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f084400bd40 con 0x7f08480f2520 2021-02-10T11:33:15.925 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:15.920 7f083affd700 1 -- 172.21.2.102:0/605144082 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0844006240 con 0x7f08480f2520 2021-02-10T11:33:15.925 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:15.920 7f085057e700 1 -- 172.21.2.102:0/605144082 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08481b0bc0 con 0x7f08480f2520 2021-02-10T11:33:15.925 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:15.920 7f083affd700 1 -- 172.21.2.102:0/605144082 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0844005690 con 0x7f08480f2520 2021-02-10T11:33:15.925 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:15.920 7f085057e700 1 -- 172.21.2.102:0/605144082 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08481b0bc0 con 0x7f08480f2520 2021-02-10T11:33:15.925 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:15.920 7f083affd700 1 -- 172.21.2.102:0/605144082 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f08440054a0 con 0x7f08480f2520 2021-02-10T11:33:15.925 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:15.920 7f083affd700 1 --2- 172.21.2.102:0/605144082 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f08300186d0 0x7f083001abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:15.925 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:15.920 7f083affd700 1 -- 172.21.2.102:0/605144082 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f08440327a0 con 0x7f08480f2520 2021-02-10T11:33:15.925 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:15.921 7f084db99700 1 --2- 172.21.2.102:0/605144082 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f08300186d0 0x7f083001abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:15.926 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:15.921 7f084db99700 1 --2- 172.21.2.102:0/605144082 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f08300186d0 0x7f083001abd0 crc :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:15.953 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:15.952 7f085057e700 1 -- 172.21.2.102:0/605144082 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08480ee0e0 con 0x7f08480f2520 2021-02-10T11:33:15.956 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:15.954 7f083affd700 1 -- 172.21.2.102:0/605144082 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f0844000c60 con 0x7f08480f2520 2021-02-10T11:33:16.088 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:16.087 7f085057e700 1 -- 172.21.2.102:0/605144082 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestVolumes.test_multitenant_subvolumes"]} v 0) v1 -- 0x7f08480ee0e0 con 0x7f08480f2520 2021-02-10T11:33:16.534 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:16.532 7f083affd700 1 -- 172.21.2.102:0/605144082 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestVolumes.test_multitenant_subvolumes"]}]=0 v633) v1 ==== 144+0+0 (secure 0 0 0) 0x7f08440054a0 con 0x7f08480f2520 2021-02-10T11:33:16.559 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:16.558 7f085057e700 1 -- 172.21.2.102:0/605144082 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f08300186d0 msgr2=0x7f083001abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:16.559 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:16.558 7f085057e700 1 --2- 172.21.2.102:0/605144082 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f08300186d0 0x7f083001abd0 crc :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:16.559 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:16.558 7f085057e700 1 -- 172.21.2.102:0/605144082 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f08480f2520 msgr2=0x7f08481b0e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:16.560 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:16.558 7f085057e700 1 --2- 172.21.2.102:0/605144082 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f08480f2520 0x7f08481b0e90 secure :-1 s=READY pgs=2369 cs=0 l=1 rev1=1 rx=0x7f084400b5b0 tx=0x7f084400b950).stop 2021-02-10T11:33:16.560 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:16.558 7f085057e700 1 -- 172.21.2.102:0/605144082 shutdown_connections 2021-02-10T11:33:16.560 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:16.558 7f085057e700 1 --2- 172.21.2.102:0/605144082 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f08480f2920 0x7f08481b03c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:16.560 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:16.558 7f085057e700 1 --2- 172.21.2.102:0/605144082 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f08300186d0 0x7f083001abd0 unknown :-1 s=CLOSED pgs=1049 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:16.560 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:16.558 7f085057e700 1 --2- 172.21.2.102:0/605144082 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f08481b14b0 0x7f08481afda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:16.560 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:16.558 7f085057e700 1 --2- 172.21.2.102:0/605144082 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f08480f2520 0x7f08481b0e90 unknown :-1 s=CLOSED pgs=2369 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:16.560 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:16.558 7f085057e700 1 -- 172.21.2.102:0/605144082 >> 172.21.2.102:0/605144082 conn(0x7f08481b3860 msgr2=0x7f08481af430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:16.560 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:16.558 7f085057e700 1 -- 172.21.2.102:0/605144082 shutdown_connections 2021-02-10T11:33:16.560 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:16.558 7f085057e700 1 -- 172.21.2.102:0/605144082 wait complete. 2021-02-10T11:33:16.567 INFO:tasks.cephfs_test_runner:test_multitenant_subvolumes (tasks.cephfs.test_volumes.TestVolumes) ... ok 2021-02-10T11:33:16.567 INFO:tasks.cephfs_test_runner:Starting test: test_non_clone_status (tasks.cephfs.test_volumes.TestVolumes) 2021-02-10T11:33:16.567 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestVolumes.test_non_clone_status' 2021-02-10T11:33:16.994 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:16.976 7fcf47920700 1 Processor -- start 2021-02-10T11:33:16.994 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:16.976 7fcf47920700 1 -- start start 2021-02-10T11:33:16.994 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:16.977 7fcf47920700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fcf400d1870 0x7fcf401b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:16.994 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:16.977 7fcf47920700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fcf401b0a10 0x7fcf400cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:16.995 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:16.977 7fcf47920700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf400f0b10 con 0x7fcf400d1c70 2021-02-10T11:33:16.995 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:16.977 7fcf47920700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf400e5ed0 con 0x7fcf400d1870 2021-02-10T11:33:16.995 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:16.977 7fcf47920700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf400f0650 con 0x7fcf401b0a10 2021-02-10T11:33:16.995 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:16.977 7fcf45f3d700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fcf400d1c70 0x7fcf400cf950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:55568/0 (socket says 172.21.2.102:55568) 2021-02-10T11:33:16.995 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:16.977 7fcf45f3d700 1 -- 172.21.2.102:0/4268479229 learned_addr learned my addr 172.21.2.102:0/4268479229 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:33:16.995 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:16.977 7fcf4573c700 1 --2- 172.21.2.102:0/4268479229 >> v2:172.21.2.102:3300/0 conn(0x7fcf400d1870 0x7fcf401b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:16.995 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:16.977 7fcf4573c700 1 -- 172.21.2.102:0/4268479229 >> v2:172.21.2.113:3300/0 conn(0x7fcf401b0a10 msgr2=0x7fcf400cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:33:16.995 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:16.977 7fcf4573c700 1 --2- 172.21.2.102:0/4268479229 >> v2:172.21.2.113:3300/0 conn(0x7fcf401b0a10 0x7fcf400cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:16.995 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:16.977 7fcf4573c700 1 -- 172.21.2.102:0/4268479229 >> v1:172.21.2.102:6789/0 conn(0x7fcf400d1c70 legacy=0x7fcf400cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:16.995 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:16.977 7fcf4573c700 1 -- 172.21.2.102:0/4268479229 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf400f0840 con 0x7fcf400d1870 2021-02-10T11:33:16.995 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:16.978 7fcf4573c700 1 --2- 172.21.2.102:0/4268479229 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcf400d1870 0x7fcf401b03f0 secure :-1 s=READY pgs=2371 cs=0 l=1 rev1=1 rx=0x7fcf34006650 tx=0x7fcf34009a80).ready entity=mon.0 client_cookie=8d1fd0c8130b0e1d server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:16.995 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:16.978 7fcf33fff700 1 -- 172.21.2.102:0/4268479229 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcf3400c0c0 con 0x7fcf400d1870 2021-02-10T11:33:16.996 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:16.978 7fcf33fff700 1 -- 172.21.2.102:0/4268479229 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf3400c340 con 0x7fcf400d1870 2021-02-10T11:33:16.996 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:16.978 7fcf33fff700 1 -- 172.21.2.102:0/4268479229 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcf3400c860 con 0x7fcf400d1870 2021-02-10T11:33:16.996 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:16.978 7fcf47920700 1 -- 172.21.2.102:0/4268479229 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcf400d1870 msgr2=0x7fcf401b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:16.996 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:16.978 7fcf47920700 1 --2- 172.21.2.102:0/4268479229 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcf400d1870 0x7fcf401b03f0 secure :-1 s=READY pgs=2371 cs=0 l=1 rev1=1 rx=0x7fcf34006650 tx=0x7fcf34009a80).stop 2021-02-10T11:33:16.996 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:16.978 7fcf47920700 1 -- 172.21.2.102:0/4268479229 shutdown_connections 2021-02-10T11:33:16.996 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:16.978 7fcf47920700 1 --2- 172.21.2.102:0/4268479229 >> v2:172.21.2.113:3300/0 conn(0x7fcf401b0a10 0x7fcf400cf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:16.996 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:16.978 7fcf47920700 1 --2- 172.21.2.102:0/4268479229 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcf400d1870 0x7fcf401b03f0 unknown :-1 s=CLOSED pgs=2371 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:16.996 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:16.978 7fcf47920700 1 -- 172.21.2.102:0/4268479229 >> 172.21.2.102:0/4268479229 conn(0x7fcf400ce450 msgr2=0x7fcf400d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:16.996 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:16.979 7fcf47920700 1 -- 172.21.2.102:0/4268479229 shutdown_connections 2021-02-10T11:33:16.996 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:16.979 7fcf47920700 1 -- 172.21.2.102:0/4268479229 wait complete. 2021-02-10T11:33:16.997 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:16.993 7fcf47920700 1 Processor -- start 2021-02-10T11:33:16.998 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:16.993 7fcf47920700 1 -- start start 2021-02-10T11:33:16.998 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:16.993 7fcf47920700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fcf400ae8a0 0x7fcf401ae550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:16.998 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:16.993 7fcf47920700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcf400eef70 0x7fcf400ef370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:16.998 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:16.993 7fcf47920700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcf401ab380 0x7fcf401ab780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:16.998 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:16.993 7fcf47920700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf400ef990 con 0x7fcf401ab380 2021-02-10T11:33:16.998 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:16.993 7fcf47920700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf401aeb70 con 0x7fcf400ae8a0 2021-02-10T11:33:16.998 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:16.993 7fcf47920700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf401abda0 con 0x7fcf400eef70 2021-02-10T11:33:16.998 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:16.994 7fcf45f3d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcf401ab380 0x7fcf401ab780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:16.998 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:16.994 7fcf45f3d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcf401ab380 0x7fcf401ab780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:35116/0 (socket says 172.21.2.102:35116) 2021-02-10T11:33:16.998 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:16.994 7fcf45f3d700 1 -- 172.21.2.102:0/3961111016 learned_addr learned my addr 172.21.2.102:0/3961111016 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:33:16.998 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:16.994 7fcf45f3d700 1 -- 172.21.2.102:0/3961111016 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcf400eef70 msgr2=0x7fcf400ef370 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:33:16.999 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:16.994 7fcf45f3d700 1 --2- 172.21.2.102:0/3961111016 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcf400eef70 0x7fcf400ef370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:16.999 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:16.994 7fcf45f3d700 1 -- 172.21.2.102:0/3961111016 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fcf400ae8a0 msgr2=0x7fcf401ae550 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:33:16.999 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:16.994 7fcf45f3d700 1 --2- 172.21.2.102:0/3961111016 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fcf400ae8a0 0x7fcf401ae550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:16.999 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:16.994 7fcf45f3d700 1 -- 172.21.2.102:0/3961111016 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf401abfc0 con 0x7fcf401ab380 2021-02-10T11:33:16.999 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:16.994 7fcf45f3d700 1 --2- 172.21.2.102:0/3961111016 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcf401ab380 0x7fcf401ab780 secure :-1 s=READY pgs=2372 cs=0 l=1 rev1=1 rx=0x7fcf38001bf0 tx=0x7fcf3800ac50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:16.999 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:16.994 7fcf31ffb700 1 -- 172.21.2.102:0/3961111016 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcf380070d0 con 0x7fcf401ab380 2021-02-10T11:33:16.999 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:16.994 7fcf31ffb700 1 -- 172.21.2.102:0/3961111016 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf38007390 con 0x7fcf401ab380 2021-02-10T11:33:16.999 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:16.994 7fcf47920700 1 -- 172.21.2.102:0/3961111016 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf401abfc0 con 0x7fcf401ab380 2021-02-10T11:33:16.999 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:16.994 7fcf47920700 1 -- 172.21.2.102:0/3961111016 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf401af9f0 con 0x7fcf401ab380 2021-02-10T11:33:16.999 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:16.994 7fcf31ffb700 1 -- 172.21.2.102:0/3961111016 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcf38004720 con 0x7fcf401ab380 2021-02-10T11:33:16.999 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:16.995 7fcf31ffb700 1 -- 172.21.2.102:0/3961111016 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fcf38007570 con 0x7fcf401ab380 2021-02-10T11:33:16.999 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:16.995 7fcf31ffb700 1 --2- 172.21.2.102:0/3961111016 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcf20018700 0x7fcf2001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:17.000 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:16.995 7fcf31ffb700 1 -- 172.21.2.102:0/3961111016 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fcf38032970 con 0x7fcf401ab380 2021-02-10T11:33:17.000 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:16.995 7fcf4573c700 1 --2- 172.21.2.102:0/3961111016 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcf20018700 0x7fcf2001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:17.000 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:16.995 7fcf4573c700 1 --2- 172.21.2.102:0/3961111016 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcf20018700 0x7fcf2001abc0 crc :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:17.028 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:17.027 7fcf47920700 1 -- 172.21.2.102:0/3961111016 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf401b0d30 con 0x7fcf401ab380 2021-02-10T11:33:17.030 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:17.029 7fcf31ffb700 1 -- 172.21.2.102:0/3961111016 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fcf380014e0 con 0x7fcf401ab380 2021-02-10T11:33:17.163 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:17.161 7fcf47920700 1 -- 172.21.2.102:0/3961111016 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestVolumes.test_non_clone_status"]} v 0) v1 -- 0x7fcf401b0d30 con 0x7fcf401ab380 2021-02-10T11:33:17.536 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:17.535 7fcf31ffb700 1 -- 172.21.2.102:0/3961111016 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestVolumes.test_non_clone_status"]}]=0 v634) v1 ==== 141+0+0 (secure 0 0 0) 0x7fcf38007390 con 0x7fcf401ab380 2021-02-10T11:33:17.581 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:17.579 7fcf47920700 1 -- 172.21.2.102:0/3961111016 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcf20018700 msgr2=0x7fcf2001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:17.582 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:17.579 7fcf47920700 1 --2- 172.21.2.102:0/3961111016 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcf20018700 0x7fcf2001abc0 crc :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:17.582 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:17.579 7fcf47920700 1 -- 172.21.2.102:0/3961111016 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcf401ab380 msgr2=0x7fcf401ab780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:17.582 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:17.579 7fcf47920700 1 --2- 172.21.2.102:0/3961111016 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcf401ab380 0x7fcf401ab780 secure :-1 s=READY pgs=2372 cs=0 l=1 rev1=1 rx=0x7fcf38001bf0 tx=0x7fcf3800ac50).stop 2021-02-10T11:33:17.582 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:17.581 7fcf47920700 1 -- 172.21.2.102:0/3961111016 shutdown_connections 2021-02-10T11:33:17.582 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:17.581 7fcf47920700 1 --2- 172.21.2.102:0/3961111016 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcf401ab380 0x7fcf401ab780 unknown :-1 s=CLOSED pgs=2372 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:17.582 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:17.581 7fcf47920700 1 --2- 172.21.2.102:0/3961111016 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcf400eef70 0x7fcf400ef370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:17.582 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:17.581 7fcf47920700 1 --2- 172.21.2.102:0/3961111016 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcf20018700 0x7fcf2001abc0 unknown :-1 s=CLOSED pgs=1050 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:17.582 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:17.581 7fcf47920700 1 --2- 172.21.2.102:0/3961111016 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fcf400ae8a0 0x7fcf401ae550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:17.582 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:17.581 7fcf47920700 1 -- 172.21.2.102:0/3961111016 >> 172.21.2.102:0/3961111016 conn(0x7fcf401a8c90 msgr2=0x7fcf401b38a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:17.582 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:17.581 7fcf47920700 1 -- 172.21.2.102:0/3961111016 shutdown_connections 2021-02-10T11:33:17.582 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:17.581 7fcf47920700 1 -- 172.21.2.102:0/3961111016 wait complete. 2021-02-10T11:33:17.590 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config set mon mon_allow_pool_delete True 2021-02-10T11:33:18.024 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:18.004 7fa122460700 1 Processor -- start 2021-02-10T11:33:18.024 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:18.004 7fa122460700 1 -- start start 2021-02-10T11:33:18.024 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:18.004 7fa122460700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fa11c0d1860 0x7fa11c1b04b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:18.024 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:18.004 7fa122460700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa11c0fe1b0 con 0x7fa11c1b0ad0 2021-02-10T11:33:18.024 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:18.004 7fa122460700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa11c109a50 con 0x7fa11c0cff60 2021-02-10T11:33:18.025 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:18.004 7fa122460700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa11c088070 con 0x7fa11c0d1860 2021-02-10T11:33:18.025 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:18.004 7fa11b7fe700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fa11c1b0ad0 0x7fa11c108930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:55580/0 (socket says 172.21.2.102:55580) 2021-02-10T11:33:18.025 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:18.004 7fa11b7fe700 1 -- 172.21.2.102:0/2876095587 learned_addr learned my addr 172.21.2.102:0/2876095587 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:33:18.025 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:18.005 7fa11affd700 1 -- 172.21.2.102:0/2876095587 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3393428016 0 0) 0x7fa1100010b0 con 0x7fa11c1b0ad0 2021-02-10T11:33:18.025 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:18.005 7fa11affd700 1 -- 172.21.2.102:0/2876095587 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa104002fe0 con 0x7fa11c1b0ad0 2021-02-10T11:33:18.025 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:18.005 7fa11bfff700 1 --2- 172.21.2.102:0/2876095587 >> v2:172.21.2.113:3300/0 conn(0x7fa11c0d1860 0x7fa11c1b04b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:18.026 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:18.005 7fa11affd700 1 -- 172.21.2.102:0/2876095587 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 712419207 0 0) 0x7fa1100010b0 con 0x7fa11c1b0ad0 2021-02-10T11:33:18.026 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:18.005 7fa11affd700 1 -- 172.21.2.102:0/2876095587 >> v2:172.21.2.113:3300/0 conn(0x7fa11c0d1860 msgr2=0x7fa11c1b04b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:33:18.026 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:18.005 7fa11affd700 1 --2- 172.21.2.102:0/2876095587 >> v2:172.21.2.113:3300/0 conn(0x7fa11c0d1860 0x7fa11c1b04b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:18.026 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:18.005 7fa11affd700 1 -- 172.21.2.102:0/2876095587 >> v1:172.21.2.113:6789/0 conn(0x7fa11c0cff60 legacy=0x7fa11c109120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:18.026 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:18.005 7fa11affd700 1 -- 172.21.2.102:0/2876095587 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa11c088260 con 0x7fa11c1b0ad0 2021-02-10T11:33:18.026 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:18.005 7fa11affd700 1 -- 172.21.2.102:0/2876095587 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7fa1100015b0 con 0x7fa11c1b0ad0 2021-02-10T11:33:18.027 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:18.005 7fa11affd700 1 -- 172.21.2.102:0/2876095587 >> v1:172.21.2.102:6789/0 conn(0x7fa11c1b0ad0 legacy=0x7fa11c108930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:18.027 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:18.005 7fa11affd700 1 --2- 172.21.2.102:0/2876095587 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa104003af0 0x7fa104005f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:18.027 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:18.005 7fa11affd700 1 --2- 172.21.2.102:0/2876095587 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa1040067a0 0x7fa104008cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:18.027 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:18.005 7fa11affd700 1 --2- 172.21.2.102:0/2876095587 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa1040095a0 0x7fa10400baa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:18.027 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:18.005 7fa11affd700 1 -- 172.21.2.102:0/2876095587 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa10400c2f0 con 0x7fa1040067a0 2021-02-10T11:33:18.027 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:18.005 7fa11affd700 1 -- 172.21.2.102:0/2876095587 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa10400c510 con 0x7fa104003af0 2021-02-10T11:33:18.028 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:18.005 7fa11affd700 1 -- 172.21.2.102:0/2876095587 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa104050740 con 0x7fa1040095a0 2021-02-10T11:33:18.028 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:18.005 7fa11b7fe700 1 --2- 172.21.2.102:0/2876095587 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa1040067a0 0x7fa104008cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:18.028 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:18.005 7fa11bfff700 1 --2- 172.21.2.102:0/2876095587 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa104003af0 0x7fa104005f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:18.028 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:18.006 7fa120a7d700 1 --2- 172.21.2.102:0/2876095587 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa1040095a0 0x7fa10400baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:18.028 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:18.006 7fa11b7fe700 1 -- 172.21.2.102:0/2876095587 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa1040095a0 msgr2=0x7fa10400baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:33:18.028 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:18.006 7fa11b7fe700 1 --2- 172.21.2.102:0/2876095587 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa1040095a0 0x7fa10400baa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:18.029 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:18.006 7fa11b7fe700 1 -- 172.21.2.102:0/2876095587 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa104003af0 msgr2=0x7fa104005f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:33:18.029 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:18.006 7fa11b7fe700 1 --2- 172.21.2.102:0/2876095587 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa104003af0 0x7fa104005f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:18.029 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:18.006 7fa11b7fe700 1 -- 172.21.2.102:0/2876095587 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1040509a0 con 0x7fa1040067a0 2021-02-10T11:33:18.029 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:18.006 7fa120a7d700 1 --2- 172.21.2.102:0/2876095587 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa1040095a0 0x7fa10400baa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:33:18.029 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:18.006 7fa120a7d700 1 -- 172.21.2.102:0/2876095587 reap_dead start 2021-02-10T11:33:18.029 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:18.006 7fa11b7fe700 1 --2- 172.21.2.102:0/2876095587 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa1040067a0 0x7fa104008cc0 secure :-1 s=READY pgs=2374 cs=0 l=1 rev1=1 rx=0x7fa110000b60 tx=0x7fa110007d40).ready entity=mon.0 client_cookie=e156d2ce10fc12d3 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:18.030 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:18.006 7fa11affd700 1 -- 172.21.2.102:0/2876095587 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa110001010 con 0x7fa1040067a0 2021-02-10T11:33:18.030 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:18.006 7fa11affd700 1 -- 172.21.2.102:0/2876095587 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa110000980 con 0x7fa1040067a0 2021-02-10T11:33:18.030 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:18.006 7fa11affd700 1 -- 172.21.2.102:0/2876095587 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa11000a520 con 0x7fa1040067a0 2021-02-10T11:33:18.030 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:18.006 7fa122460700 1 -- 172.21.2.102:0/2876095587 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa1040067a0 msgr2=0x7fa104008cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:18.030 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:18.006 7fa122460700 1 --2- 172.21.2.102:0/2876095587 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa1040067a0 0x7fa104008cc0 secure :-1 s=READY pgs=2374 cs=0 l=1 rev1=1 rx=0x7fa110000b60 tx=0x7fa110007d40).stop 2021-02-10T11:33:18.031 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:18.006 7fa122460700 1 -- 172.21.2.102:0/2876095587 shutdown_connections 2021-02-10T11:33:18.031 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:18.006 7fa122460700 1 --2- 172.21.2.102:0/2876095587 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa1040067a0 0x7fa104008cc0 unknown :-1 s=CLOSED pgs=2374 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:18.031 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:18.006 7fa122460700 1 -- 172.21.2.102:0/2876095587 >> 172.21.2.102:0/2876095587 conn(0x7fa11c0ce410 msgr2=0x7fa11c0d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:18.031 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:18.006 7fa122460700 1 -- 172.21.2.102:0/2876095587 shutdown_connections 2021-02-10T11:33:18.031 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:18.007 7fa122460700 1 -- 172.21.2.102:0/2876095587 wait complete. 2021-02-10T11:33:18.031 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:18.021 7fa122460700 1 Processor -- start 2021-02-10T11:33:18.031 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:18.021 7fa122460700 1 -- start start 2021-02-10T11:33:18.031 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:18.021 7fa122460700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa11c0badd0 0x7fa11c09eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:18.032 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:18.021 7fa122460700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa11c0adf50 0x7fa11c09f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:18.032 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:18.022 7fa122460700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa11c1a7470 0x7fa11c1a7870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:18.032 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:18.022 7fa122460700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa11c09e5a0 con 0x7fa11c1a7470 2021-02-10T11:33:18.032 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:18.022 7fa122460700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa11c09f6e0 con 0x7fa11c0adf50 2021-02-10T11:33:18.032 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:18.022 7fa122460700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa11c0bb220 con 0x7fa11c0badd0 2021-02-10T11:33:18.032 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:18.022 7fa120a7d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa11c1a7470 0x7fa11c1a7870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:18.032 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:18.022 7fa120a7d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa11c1a7470 0x7fa11c1a7870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:35138/0 (socket says 172.21.2.102:35138) 2021-02-10T11:33:18.032 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:18.022 7fa120a7d700 1 -- 172.21.2.102:0/2459333252 learned_addr learned my addr 172.21.2.102:0/2459333252 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:33:18.032 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:18.022 7fa11b7fe700 1 --2- 172.21.2.102:0/2459333252 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa11c0adf50 0x7fa11c09f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:18.032 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:18.022 7fa11bfff700 1 --2- 172.21.2.102:0/2459333252 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa11c0badd0 0x7fa11c09eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:18.032 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:18.022 7fa120a7d700 1 -- 172.21.2.102:0/2459333252 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa11c0badd0 msgr2=0x7fa11c09eaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:18.032 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:18.022 7fa120a7d700 1 --2- 172.21.2.102:0/2459333252 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa11c0badd0 0x7fa11c09eaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:18.032 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:18.022 7fa120a7d700 1 -- 172.21.2.102:0/2459333252 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa11c0adf50 msgr2=0x7fa11c09f0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:18.033 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:18.022 7fa120a7d700 1 --2- 172.21.2.102:0/2459333252 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa11c0adf50 0x7fa11c09f0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:18.033 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:18.022 7fa120a7d700 1 -- 172.21.2.102:0/2459333252 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa11c1a7e90 con 0x7fa11c1a7470 2021-02-10T11:33:18.033 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:18.022 7fa120a7d700 1 --2- 172.21.2.102:0/2459333252 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa11c1a7470 0x7fa11c1a7870 secure :-1 s=READY pgs=2375 cs=0 l=1 rev1=1 rx=0x7fa10c006f30 tx=0x7fa10c00b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:18.034 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:18.023 7fa118ff9700 1 -- 172.21.2.102:0/2459333252 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa10c000960 con 0x7fa11c1a7470 2021-02-10T11:33:18.034 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:18.023 7fa118ff9700 1 -- 172.21.2.102:0/2459333252 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa10c00bd50 con 0x7fa11c1a7470 2021-02-10T11:33:18.034 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:18.023 7fa118ff9700 1 -- 172.21.2.102:0/2459333252 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa10c005320 con 0x7fa11c1a7470 2021-02-10T11:33:18.034 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:18.023 7fa122460700 1 -- 172.21.2.102:0/2459333252 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa11c1a7e90 con 0x7fa11c1a7470 2021-02-10T11:33:18.034 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:18.023 7fa122460700 1 -- 172.21.2.102:0/2459333252 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa11c09d1f0 con 0x7fa11c1a7470 2021-02-10T11:33:18.034 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:18.023 7fa118ff9700 1 -- 172.21.2.102:0/2459333252 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa10c006030 con 0x7fa11c1a7470 2021-02-10T11:33:18.034 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:18.023 7fa118ff9700 1 --2- 172.21.2.102:0/2459333252 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa0fc0186d0 0x7fa0fc01abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:18.034 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:18.023 7fa118ff9700 1 -- 172.21.2.102:0/2459333252 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fa10c032620 con 0x7fa11c1a7470 2021-02-10T11:33:18.034 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:18.024 7fa11bfff700 1 --2- 172.21.2.102:0/2459333252 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa0fc0186d0 0x7fa0fc01abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:18.034 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:18.024 7fa11bfff700 1 --2- 172.21.2.102:0/2459333252 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa0fc0186d0 0x7fa0fc01abd0 crc :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:18.056 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:18.054 7fa122460700 1 -- 172.21.2.102:0/2459333252 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa11c082a20 con 0x7fa11c1a7470 2021-02-10T11:33:18.058 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:18.058 7fa118ff9700 1 -- 172.21.2.102:0/2459333252 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fa10c006c20 con 0x7fa11c1a7470 2021-02-10T11:33:18.206 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:18.205 7fa122460700 1 -- 172.21.2.102:0/2459333252 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fa11c082a20 con 0x7fa11c1a7470 2021-02-10T11:33:18.210 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:18.209 7fa118ff9700 1 -- 172.21.2.102:0/2459333252 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v32)=0 v32) v1 ==== 122+0+0 (secure 0 0 0) 0x7fa10c006030 con 0x7fa11c1a7470 2021-02-10T11:33:18.214 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:18.213 7fa122460700 1 -- 172.21.2.102:0/2459333252 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa0fc0186d0 msgr2=0x7fa0fc01abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:18.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:18.213 7fa122460700 1 --2- 172.21.2.102:0/2459333252 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa0fc0186d0 0x7fa0fc01abd0 crc :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:18.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:18.213 7fa122460700 1 -- 172.21.2.102:0/2459333252 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa11c1a7470 msgr2=0x7fa11c1a7870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:18.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:18.213 7fa122460700 1 --2- 172.21.2.102:0/2459333252 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa11c1a7470 0x7fa11c1a7870 secure :-1 s=READY pgs=2375 cs=0 l=1 rev1=1 rx=0x7fa10c006f30 tx=0x7fa10c00b960).stop 2021-02-10T11:33:18.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:18.213 7fa122460700 1 -- 172.21.2.102:0/2459333252 shutdown_connections 2021-02-10T11:33:18.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:18.213 7fa122460700 1 --2- 172.21.2.102:0/2459333252 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa11c1a7470 0x7fa11c1a7870 unknown :-1 s=CLOSED pgs=2375 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:18.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:18.213 7fa122460700 1 --2- 172.21.2.102:0/2459333252 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa0fc0186d0 0x7fa0fc01abd0 unknown :-1 s=CLOSED pgs=1051 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:18.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:18.213 7fa122460700 1 --2- 172.21.2.102:0/2459333252 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa11c0adf50 0x7fa11c09f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:18.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:18.213 7fa122460700 1 --2- 172.21.2.102:0/2459333252 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa11c0badd0 0x7fa11c09eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:18.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:18.213 7fa122460700 1 -- 172.21.2.102:0/2459333252 >> 172.21.2.102:0/2459333252 conn(0x7fa11c0bee90 msgr2=0x7fa11c0bb880 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:18.216 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:18.213 7fa122460700 1 -- 172.21.2.102:0/2459333252 shutdown_connections 2021-02-10T11:33:18.216 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:18.213 7fa122460700 1 -- 172.21.2.102:0/2459333252 wait complete. 2021-02-10T11:33:18.224 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2021-02-10T11:33:18.225 DEBUG:teuthology.orchestra.run.gibba002:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:33:18.571 DEBUG:teuthology.orchestra.run.gibba013:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:33:18.924 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2021-02-10T11:33:18.925 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:33:18.925 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:33:18.925 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:33:19.260 INFO:teuthology.orchestra.run.gibba002.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.0’: No such file or directory 2021-02-10T11:33:19.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:33:19.260 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.0 2021-02-10T11:33:19.260 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:33:19.260 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:33:19.261 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:33:19.596 INFO:teuthology.orchestra.run.gibba002.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.1’: No such file or directory 2021-02-10T11:33:19.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:33:19.597 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2021-02-10T11:33:19.597 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:33:19.597 DEBUG:teuthology.orchestra.run.gibba013:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:33:19.597 DEBUG:teuthology.orchestra.run.gibba013:> ' -- /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:33:19.933 INFO:teuthology.orchestra.run.gibba013.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.2’: No such file or directory 2021-02-10T11:33:19.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:33:19.933 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2021-02-10T11:33:19.934 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:33:19.934 DEBUG:teuthology.orchestra.run.gibba013:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:33:19.934 DEBUG:teuthology.orchestra.run.gibba013:> ' -- /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:33:20.268 INFO:teuthology.orchestra.run.gibba013.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.3’: No such file or directory 2021-02-10T11:33:20.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:33:20.268 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.3 2021-02-10T11:33:20.269 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json-pretty 2021-02-10T11:33:20.697 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:20.680 7f0908e76700 1 Processor -- start 2021-02-10T11:33:20.697 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:20.680 7f0908e76700 1 -- start start 2021-02-10T11:33:20.697 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:20.680 7f0908e76700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f09041b0870 0x7f09040f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:20.697 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:20.680 7f0908e76700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f09040efe00 0x7f09040f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:20.697 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:20.680 7f0908e76700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09041b0c70 con 0x7f09041b0470 2021-02-10T11:33:20.697 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:20.680 7f0908e76700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09040e5ea0 con 0x7f09041b0870 2021-02-10T11:33:20.697 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:20.680 7f0908e76700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09040f0b50 con 0x7f09040efe00 2021-02-10T11:33:20.697 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:20.680 7f0902b13700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f09041b0470 0x7f09040ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:55598/0 (socket says 172.21.2.102:55598) 2021-02-10T11:33:20.698 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:20.680 7f0902b13700 1 -- 172.21.2.102:0/578485312 learned_addr learned my addr 172.21.2.102:0/578485312 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:33:20.698 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:20.680 7f0902312700 1 --2- 172.21.2.102:0/578485312 >> v2:172.21.2.102:3300/0 conn(0x7f09041b0870 0x7f09040f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:20.698 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:20.681 7f0901b11700 1 -- 172.21.2.102:0/578485312 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1938921646 0 0) 0x7f08ec000ce0 con 0x7f09041b0470 2021-02-10T11:33:20.698 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:20.681 7f0901b11700 1 -- 172.21.2.102:0/578485312 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08f0002fe0 con 0x7f09041b0470 2021-02-10T11:33:20.698 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:20.681 7f0902312700 1 -- 172.21.2.102:0/578485312 >> v2:172.21.2.113:3300/0 conn(0x7f09040efe00 msgr2=0x7f09040f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:33:20.698 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:20.681 7f0902312700 1 --2- 172.21.2.102:0/578485312 >> v2:172.21.2.113:3300/0 conn(0x7f09040efe00 0x7f09040f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:20.698 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:20.681 7f0902312700 1 -- 172.21.2.102:0/578485312 >> v1:172.21.2.102:6789/0 conn(0x7f09041b0470 legacy=0x7f09040ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:20.698 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:20.681 7f0902312700 1 -- 172.21.2.102:0/578485312 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09040f0820 con 0x7f09041b0870 2021-02-10T11:33:20.698 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:20.681 7f0902312700 1 --2- 172.21.2.102:0/578485312 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f09041b0870 0x7f09040f20e0 secure :-1 s=READY pgs=2377 cs=0 l=1 rev1=1 rx=0x7f08f4006650 tx=0x7f08f4009a00).ready entity=mon.0 client_cookie=9566d783d0053cf3 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:20.698 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:20.681 7f0901b11700 1 -- 172.21.2.102:0/578485312 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f08f40009b0 con 0x7f09041b0870 2021-02-10T11:33:20.698 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:20.681 7f0901b11700 1 -- 172.21.2.102:0/578485312 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08f400c0c0 con 0x7f09041b0870 2021-02-10T11:33:20.698 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:20.681 7f0901b11700 1 -- 172.21.2.102:0/578485312 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f08f400c5a0 con 0x7f09041b0870 2021-02-10T11:33:20.699 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:20.681 7f0908e76700 1 -- 172.21.2.102:0/578485312 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f09041b0870 msgr2=0x7f09040f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:20.699 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:20.681 7f0908e76700 1 --2- 172.21.2.102:0/578485312 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f09041b0870 0x7f09040f20e0 secure :-1 s=READY pgs=2377 cs=0 l=1 rev1=1 rx=0x7f08f4006650 tx=0x7f08f4009a00).stop 2021-02-10T11:33:20.699 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:20.682 7f0908e76700 1 -- 172.21.2.102:0/578485312 shutdown_connections 2021-02-10T11:33:20.699 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:20.682 7f0908e76700 1 --2- 172.21.2.102:0/578485312 >> v2:172.21.2.113:3300/0 conn(0x7f09040efe00 0x7f09040f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:20.699 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:20.682 7f0908e76700 1 --2- 172.21.2.102:0/578485312 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f09041b0870 0x7f09040f20e0 unknown :-1 s=CLOSED pgs=2377 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:20.699 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:20.682 7f0908e76700 1 -- 172.21.2.102:0/578485312 >> 172.21.2.102:0/578485312 conn(0x7f09040ce3e0 msgr2=0x7f09040e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:20.699 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:20.682 7f0908e76700 1 -- 172.21.2.102:0/578485312 shutdown_connections 2021-02-10T11:33:20.699 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:20.682 7f0908e76700 1 -- 172.21.2.102:0/578485312 wait complete. 2021-02-10T11:33:20.700 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:20.697 7f0908e76700 1 Processor -- start 2021-02-10T11:33:20.701 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:20.697 7f0908e76700 1 -- start start 2021-02-10T11:33:20.701 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:20.697 7f0908e76700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f09041072b0 0x7f09041097d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:20.701 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:20.697 7f0908e76700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f09040bebd0 0x7f09041a7350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:20.701 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:20.697 7f0908e76700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f09040befd0 0x7f09041a7970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:20.701 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:20.697 7f0908e76700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0904106ef0 con 0x7f09040befd0 2021-02-10T11:33:20.701 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:20.697 7f0908e76700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09041a7f90 con 0x7f09041072b0 2021-02-10T11:33:20.701 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:20.697 7f0908e76700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09041a81b0 con 0x7f09040bebd0 2021-02-10T11:33:20.701 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:20.697 7f0902b13700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f09041072b0 0x7f09041097d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:20.701 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:20.697 7f0903314700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f09040befd0 0x7f09041a7970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:20.701 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:20.697 7f0902312700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f09040bebd0 0x7f09041a7350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:20.701 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:20.697 7f0902b13700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f09041072b0 0x7f09041097d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.113:3300/0 says I am v2:172.21.2.102:38578/0 (socket says 172.21.2.102:38578) 2021-02-10T11:33:20.701 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:20.697 7f0902b13700 1 -- 172.21.2.102:0/2201648316 learned_addr learned my addr 172.21.2.102:0/2201648316 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:33:20.702 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:20.698 7f0903314700 1 -- 172.21.2.102:0/2201648316 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f09040bebd0 msgr2=0x7f09041a7350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:20.702 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:20.698 7f0903314700 1 --2- 172.21.2.102:0/2201648316 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f09040bebd0 0x7f09041a7350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:20.702 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:20.698 7f0903314700 1 -- 172.21.2.102:0/2201648316 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f09041072b0 msgr2=0x7f09041097d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:20.702 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:20.698 7f0903314700 1 --2- 172.21.2.102:0/2201648316 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f09041072b0 0x7f09041097d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:20.702 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:20.698 7f0903314700 1 -- 172.21.2.102:0/2201648316 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09040bfa60 con 0x7f09040befd0 2021-02-10T11:33:20.702 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:20.698 7f0903314700 1 --2- 172.21.2.102:0/2201648316 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f09040befd0 0x7f09041a7970 secure :-1 s=READY pgs=2378 cs=0 l=1 rev1=1 rx=0x7f08fc00b890 tx=0x7f08fc00bc30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:20.702 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:20.698 7f08fb7fe700 1 -- 172.21.2.102:0/2201648316 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f08fc005870 con 0x7f09040befd0 2021-02-10T11:33:20.702 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:20.698 7f08fb7fe700 1 -- 172.21.2.102:0/2201648316 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08fc005ab0 con 0x7f09040befd0 2021-02-10T11:33:20.702 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:20.698 7f08fb7fe700 1 -- 172.21.2.102:0/2201648316 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f08fc006030 con 0x7f09040befd0 2021-02-10T11:33:20.702 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:20.698 7f0908e76700 1 -- 172.21.2.102:0/2201648316 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09040bfa60 con 0x7f09040befd0 2021-02-10T11:33:20.702 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:20.698 7f0908e76700 1 -- 172.21.2.102:0/2201648316 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09040bfff0 con 0x7f09040befd0 2021-02-10T11:33:20.702 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:20.699 7f08fb7fe700 1 -- 172.21.2.102:0/2201648316 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f08fc005c90 con 0x7f09040befd0 2021-02-10T11:33:20.702 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:20.699 7f08fb7fe700 1 --2- 172.21.2.102:0/2201648316 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f08e80186c0 0x7f08e801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:20.703 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:20.699 7f08fb7fe700 1 -- 172.21.2.102:0/2201648316 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f08fc032870 con 0x7f09040befd0 2021-02-10T11:33:20.703 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:20.699 7f0902b13700 1 --2- 172.21.2.102:0/2201648316 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f08e80186c0 0x7f08e801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:20.703 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:20.699 7f0902b13700 1 --2- 172.21.2.102:0/2201648316 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f08e80186c0 0x7f08e801abc0 crc :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:20.731 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:20.730 7f0908e76700 1 -- 172.21.2.102:0/2201648316 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09040bf530 con 0x7f09040befd0 2021-02-10T11:33:20.733 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:20.733 7f08fb7fe700 1 -- 172.21.2.102:0/2201648316 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f08fc0030d0 con 0x7f09040befd0 2021-02-10T11:33:20.858 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:20.857 7f0908e76700 1 -- 172.21.2.102:0/2201648316 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7f09040bf530 con 0x7f09040befd0 2021-02-10T11:33:20.859 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:20.858 7f08fb7fe700 1 -- 172.21.2.102:0/2201648316 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v120) v1 ==== 81+0+30374 (secure 0 0 0) 0x7f08fc005c90 con 0x7f09040befd0 2021-02-10T11:33:20.859 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:33:20.860 INFO:teuthology.orchestra.run.gibba002.stdout:{ 2021-02-10T11:33:20.860 INFO:teuthology.orchestra.run.gibba002.stdout: "epoch": 120, 2021-02-10T11:33:20.860 INFO:teuthology.orchestra.run.gibba002.stdout: "fsid": "68db30ea-dbce-4aef-a6df-f01c6fea0513", 2021-02-10T11:33:20.860 INFO:teuthology.orchestra.run.gibba002.stdout: "created": "2021-02-10 11:03:06.331557", 2021-02-10T11:33:20.860 INFO:teuthology.orchestra.run.gibba002.stdout: "modified": "2021-02-10 11:32:39.019652", 2021-02-10T11:33:20.860 INFO:teuthology.orchestra.run.gibba002.stdout: "last_up_change": "2021-02-10 11:03:32.690562", 2021-02-10T11:33:20.860 INFO:teuthology.orchestra.run.gibba002.stdout: "last_in_change": "2021-02-10 11:03:32.690562", 2021-02-10T11:33:20.860 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2021-02-10T11:33:20.860 INFO:teuthology.orchestra.run.gibba002.stdout: "flags_num": 5799936, 2021-02-10T11:33:20.860 INFO:teuthology.orchestra.run.gibba002.stdout: "flags_set": [ 2021-02-10T11:33:20.860 INFO:teuthology.orchestra.run.gibba002.stdout: "pglog_hardlimit", 2021-02-10T11:33:20.860 INFO:teuthology.orchestra.run.gibba002.stdout: "purged_snapdirs", 2021-02-10T11:33:20.860 INFO:teuthology.orchestra.run.gibba002.stdout: "recovery_deletes", 2021-02-10T11:33:20.861 INFO:teuthology.orchestra.run.gibba002.stdout: "sortbitwise" 2021-02-10T11:33:20.861 INFO:teuthology.orchestra.run.gibba002.stdout: ], 2021-02-10T11:33:20.861 INFO:teuthology.orchestra.run.gibba002.stdout: "crush_version": 4, 2021-02-10T11:33:20.861 INFO:teuthology.orchestra.run.gibba002.stdout: "full_ratio": 0.94999998807907104, 2021-02-10T11:33:20.861 INFO:teuthology.orchestra.run.gibba002.stdout: "backfillfull_ratio": 0.89999997615814209, 2021-02-10T11:33:20.861 INFO:teuthology.orchestra.run.gibba002.stdout: "nearfull_ratio": 0.85000002384185791, 2021-02-10T11:33:20.861 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_snapshot": "", 2021-02-10T11:33:20.861 INFO:teuthology.orchestra.run.gibba002.stdout: "pool_max": 25, 2021-02-10T11:33:20.861 INFO:teuthology.orchestra.run.gibba002.stdout: "max_osd": 8, 2021-02-10T11:33:20.861 INFO:teuthology.orchestra.run.gibba002.stdout: "require_min_compat_client": "jewel", 2021-02-10T11:33:20.861 INFO:teuthology.orchestra.run.gibba002.stdout: "min_compat_client": "jewel", 2021-02-10T11:33:20.861 INFO:teuthology.orchestra.run.gibba002.stdout: "require_osd_release": "nautilus", 2021-02-10T11:33:20.861 INFO:teuthology.orchestra.run.gibba002.stdout: "pools": [ 2021-02-10T11:33:20.862 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:33:20.862 INFO:teuthology.orchestra.run.gibba002.stdout: "pool": 1, 2021-02-10T11:33:20.862 INFO:teuthology.orchestra.run.gibba002.stdout: "pool_name": "rbd", 2021-02-10T11:33:20.862 INFO:teuthology.orchestra.run.gibba002.stdout: "create_time": "2021-02-10 11:03:37.322519", 2021-02-10T11:33:20.862 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 1, 2021-02-10T11:33:20.862 INFO:teuthology.orchestra.run.gibba002.stdout: "flags_names": "hashpspool", 2021-02-10T11:33:20.862 INFO:teuthology.orchestra.run.gibba002.stdout: "type": 1, 2021-02-10T11:33:20.862 INFO:teuthology.orchestra.run.gibba002.stdout: "size": 2, 2021-02-10T11:33:20.862 INFO:teuthology.orchestra.run.gibba002.stdout: "min_size": 1, 2021-02-10T11:33:20.862 INFO:teuthology.orchestra.run.gibba002.stdout: "crush_rule": 0, 2021-02-10T11:33:20.862 INFO:teuthology.orchestra.run.gibba002.stdout: "object_hash": 2, 2021-02-10T11:33:20.862 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:33:20.863 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num": 8, 2021-02-10T11:33:20.863 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_placement_num": 8, 2021-02-10T11:33:20.863 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_placement_num_target": 8, 2021-02-10T11:33:20.863 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num_target": 8, 2021-02-10T11:33:20.863 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num_pending": 8, 2021-02-10T11:33:20.863 INFO:teuthology.orchestra.run.gibba002.stdout: "last_pg_merge_meta": { 2021-02-10T11:33:20.863 INFO:teuthology.orchestra.run.gibba002.stdout: "source_pgid": "0.0", 2021-02-10T11:33:20.863 INFO:teuthology.orchestra.run.gibba002.stdout: "ready_epoch": 0, 2021-02-10T11:33:20.863 INFO:teuthology.orchestra.run.gibba002.stdout: "last_epoch_started": 0, 2021-02-10T11:33:20.863 INFO:teuthology.orchestra.run.gibba002.stdout: "last_epoch_clean": 0, 2021-02-10T11:33:20.863 INFO:teuthology.orchestra.run.gibba002.stdout: "source_version": "0'0", 2021-02-10T11:33:20.863 INFO:teuthology.orchestra.run.gibba002.stdout: "target_version": "0'0" 2021-02-10T11:33:20.863 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:33:20.863 INFO:teuthology.orchestra.run.gibba002.stdout: "last_change": "16", 2021-02-10T11:33:20.864 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend": "0", 2021-02-10T11:33:20.864 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:33:20.864 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:33:20.864 INFO:teuthology.orchestra.run.gibba002.stdout: "auid": 0, 2021-02-10T11:33:20.864 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:33:20.864 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_seq": 0, 2021-02-10T11:33:20.864 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_epoch": 0, 2021-02-10T11:33:20.864 INFO:teuthology.orchestra.run.gibba002.stdout: "pool_snaps": [], 2021-02-10T11:33:20.864 INFO:teuthology.orchestra.run.gibba002.stdout: "removed_snaps": "[]", 2021-02-10T11:33:20.864 INFO:teuthology.orchestra.run.gibba002.stdout: "quota_max_bytes": 0, 2021-02-10T11:33:20.864 INFO:teuthology.orchestra.run.gibba002.stdout: "quota_max_objects": 0, 2021-02-10T11:33:20.864 INFO:teuthology.orchestra.run.gibba002.stdout: "tiers": [], 2021-02-10T11:33:20.864 INFO:teuthology.orchestra.run.gibba002.stdout: "tier_of": -1, 2021-02-10T11:33:20.864 INFO:teuthology.orchestra.run.gibba002.stdout: "read_tier": -1, 2021-02-10T11:33:20.865 INFO:teuthology.orchestra.run.gibba002.stdout: "write_tier": -1, 2021-02-10T11:33:20.865 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_mode": "none", 2021-02-10T11:33:20.865 INFO:teuthology.orchestra.run.gibba002.stdout: "target_max_bytes": 0, 2021-02-10T11:33:20.865 INFO:teuthology.orchestra.run.gibba002.stdout: "target_max_objects": 0, 2021-02-10T11:33:20.865 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:33:20.865 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:33:20.865 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:33:20.865 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_min_flush_age": 0, 2021-02-10T11:33:20.865 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_min_evict_age": 0, 2021-02-10T11:33:20.865 INFO:teuthology.orchestra.run.gibba002.stdout: "erasure_code_profile": "", 2021-02-10T11:33:20.865 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_params": { 2021-02-10T11:33:20.865 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "none" 2021-02-10T11:33:20.865 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:33:20.866 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_period": 0, 2021-02-10T11:33:20.866 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_count": 0, 2021-02-10T11:33:20.866 INFO:teuthology.orchestra.run.gibba002.stdout: "use_gmt_hitset": true, 2021-02-10T11:33:20.866 INFO:teuthology.orchestra.run.gibba002.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:33:20.866 INFO:teuthology.orchestra.run.gibba002.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:33:20.866 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:33:20.866 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:33:20.866 INFO:teuthology.orchestra.run.gibba002.stdout: "grade_table": [], 2021-02-10T11:33:20.866 INFO:teuthology.orchestra.run.gibba002.stdout: "stripe_width": 0, 2021-02-10T11:33:20.866 INFO:teuthology.orchestra.run.gibba002.stdout: "expected_num_objects": 0, 2021-02-10T11:33:20.866 INFO:teuthology.orchestra.run.gibba002.stdout: "fast_read": false, 2021-02-10T11:33:20.866 INFO:teuthology.orchestra.run.gibba002.stdout: "options": {}, 2021-02-10T11:33:20.867 INFO:teuthology.orchestra.run.gibba002.stdout: "application_metadata": { 2021-02-10T11:33:20.867 INFO:teuthology.orchestra.run.gibba002.stdout: "rbd": {} 2021-02-10T11:33:20.867 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:33:20.867 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:33:20.867 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:33:20.867 INFO:teuthology.orchestra.run.gibba002.stdout: "pool": 24, 2021-02-10T11:33:20.867 INFO:teuthology.orchestra.run.gibba002.stdout: "pool_name": "cephfs_metadata", 2021-02-10T11:33:20.867 INFO:teuthology.orchestra.run.gibba002.stdout: "create_time": "2021-02-10 11:32:35.986688", 2021-02-10T11:33:20.867 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 1, 2021-02-10T11:33:20.867 INFO:teuthology.orchestra.run.gibba002.stdout: "flags_names": "hashpspool", 2021-02-10T11:33:20.867 INFO:teuthology.orchestra.run.gibba002.stdout: "type": 1, 2021-02-10T11:33:20.868 INFO:teuthology.orchestra.run.gibba002.stdout: "size": 2, 2021-02-10T11:33:20.868 INFO:teuthology.orchestra.run.gibba002.stdout: "min_size": 1, 2021-02-10T11:33:20.868 INFO:teuthology.orchestra.run.gibba002.stdout: "crush_rule": 0, 2021-02-10T11:33:20.868 INFO:teuthology.orchestra.run.gibba002.stdout: "object_hash": 2, 2021-02-10T11:33:20.868 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:33:20.868 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num": 8, 2021-02-10T11:33:20.868 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_placement_num": 8, 2021-02-10T11:33:20.868 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_placement_num_target": 8, 2021-02-10T11:33:20.868 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num_target": 8, 2021-02-10T11:33:20.868 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num_pending": 8, 2021-02-10T11:33:20.868 INFO:teuthology.orchestra.run.gibba002.stdout: "last_pg_merge_meta": { 2021-02-10T11:33:20.868 INFO:teuthology.orchestra.run.gibba002.stdout: "source_pgid": "0.0", 2021-02-10T11:33:20.869 INFO:teuthology.orchestra.run.gibba002.stdout: "ready_epoch": 0, 2021-02-10T11:33:20.869 INFO:teuthology.orchestra.run.gibba002.stdout: "last_epoch_started": 0, 2021-02-10T11:33:20.869 INFO:teuthology.orchestra.run.gibba002.stdout: "last_epoch_clean": 0, 2021-02-10T11:33:20.869 INFO:teuthology.orchestra.run.gibba002.stdout: "source_version": "0'0", 2021-02-10T11:33:20.869 INFO:teuthology.orchestra.run.gibba002.stdout: "target_version": "0'0" 2021-02-10T11:33:20.869 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:33:20.869 INFO:teuthology.orchestra.run.gibba002.stdout: "last_change": "119", 2021-02-10T11:33:20.869 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend": "0", 2021-02-10T11:33:20.869 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:33:20.869 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:33:20.869 INFO:teuthology.orchestra.run.gibba002.stdout: "auid": 0, 2021-02-10T11:33:20.869 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:33:20.869 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_seq": 0, 2021-02-10T11:33:20.870 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_epoch": 0, 2021-02-10T11:33:20.870 INFO:teuthology.orchestra.run.gibba002.stdout: "pool_snaps": [], 2021-02-10T11:33:20.870 INFO:teuthology.orchestra.run.gibba002.stdout: "removed_snaps": "[]", 2021-02-10T11:33:20.870 INFO:teuthology.orchestra.run.gibba002.stdout: "quota_max_bytes": 0, 2021-02-10T11:33:20.870 INFO:teuthology.orchestra.run.gibba002.stdout: "quota_max_objects": 0, 2021-02-10T11:33:20.870 INFO:teuthology.orchestra.run.gibba002.stdout: "tiers": [], 2021-02-10T11:33:20.870 INFO:teuthology.orchestra.run.gibba002.stdout: "tier_of": -1, 2021-02-10T11:33:20.870 INFO:teuthology.orchestra.run.gibba002.stdout: "read_tier": -1, 2021-02-10T11:33:20.870 INFO:teuthology.orchestra.run.gibba002.stdout: "write_tier": -1, 2021-02-10T11:33:20.870 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_mode": "none", 2021-02-10T11:33:20.870 INFO:teuthology.orchestra.run.gibba002.stdout: "target_max_bytes": 0, 2021-02-10T11:33:20.870 INFO:teuthology.orchestra.run.gibba002.stdout: "target_max_objects": 0, 2021-02-10T11:33:20.870 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:33:20.871 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:33:20.871 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:33:20.871 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_min_flush_age": 0, 2021-02-10T11:33:20.871 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_min_evict_age": 0, 2021-02-10T11:33:20.871 INFO:teuthology.orchestra.run.gibba002.stdout: "erasure_code_profile": "", 2021-02-10T11:33:20.871 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_params": { 2021-02-10T11:33:20.871 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "none" 2021-02-10T11:33:20.871 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:33:20.871 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_period": 0, 2021-02-10T11:33:20.871 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_count": 0, 2021-02-10T11:33:20.871 INFO:teuthology.orchestra.run.gibba002.stdout: "use_gmt_hitset": true, 2021-02-10T11:33:20.871 INFO:teuthology.orchestra.run.gibba002.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:33:20.871 INFO:teuthology.orchestra.run.gibba002.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:33:20.872 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:33:20.872 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:33:20.872 INFO:teuthology.orchestra.run.gibba002.stdout: "grade_table": [], 2021-02-10T11:33:20.872 INFO:teuthology.orchestra.run.gibba002.stdout: "stripe_width": 0, 2021-02-10T11:33:20.872 INFO:teuthology.orchestra.run.gibba002.stdout: "expected_num_objects": 0, 2021-02-10T11:33:20.872 INFO:teuthology.orchestra.run.gibba002.stdout: "fast_read": false, 2021-02-10T11:33:20.872 INFO:teuthology.orchestra.run.gibba002.stdout: "options": { 2021-02-10T11:33:20.872 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_autoscale_bias": 4, 2021-02-10T11:33:20.872 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num_min": 16, 2021-02-10T11:33:20.872 INFO:teuthology.orchestra.run.gibba002.stdout: "recovery_priority": 5 2021-02-10T11:33:20.872 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:33:20.872 INFO:teuthology.orchestra.run.gibba002.stdout: "application_metadata": { 2021-02-10T11:33:20.872 INFO:teuthology.orchestra.run.gibba002.stdout: "cephfs": { 2021-02-10T11:33:20.873 INFO:teuthology.orchestra.run.gibba002.stdout: "metadata": "cephfs" 2021-02-10T11:33:20.873 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:33:20.873 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:33:20.873 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:33:20.873 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:33:20.873 INFO:teuthology.orchestra.run.gibba002.stdout: "pool": 25, 2021-02-10T11:33:20.873 INFO:teuthology.orchestra.run.gibba002.stdout: "pool_name": "cephfs_data", 2021-02-10T11:33:20.873 INFO:teuthology.orchestra.run.gibba002.stdout: "create_time": "2021-02-10 11:32:37.011649", 2021-02-10T11:33:20.873 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 1, 2021-02-10T11:33:20.873 INFO:teuthology.orchestra.run.gibba002.stdout: "flags_names": "hashpspool", 2021-02-10T11:33:20.873 INFO:teuthology.orchestra.run.gibba002.stdout: "type": 1, 2021-02-10T11:33:20.873 INFO:teuthology.orchestra.run.gibba002.stdout: "size": 2, 2021-02-10T11:33:20.874 INFO:teuthology.orchestra.run.gibba002.stdout: "min_size": 1, 2021-02-10T11:33:20.874 INFO:teuthology.orchestra.run.gibba002.stdout: "crush_rule": 0, 2021-02-10T11:33:20.874 INFO:teuthology.orchestra.run.gibba002.stdout: "object_hash": 2, 2021-02-10T11:33:20.874 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:33:20.874 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num": 8, 2021-02-10T11:33:20.874 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_placement_num": 8, 2021-02-10T11:33:20.874 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_placement_num_target": 8, 2021-02-10T11:33:20.874 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num_target": 8, 2021-02-10T11:33:20.874 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num_pending": 8, 2021-02-10T11:33:20.874 INFO:teuthology.orchestra.run.gibba002.stdout: "last_pg_merge_meta": { 2021-02-10T11:33:20.874 INFO:teuthology.orchestra.run.gibba002.stdout: "source_pgid": "0.0", 2021-02-10T11:33:20.874 INFO:teuthology.orchestra.run.gibba002.stdout: "ready_epoch": 0, 2021-02-10T11:33:20.874 INFO:teuthology.orchestra.run.gibba002.stdout: "last_epoch_started": 0, 2021-02-10T11:33:20.874 INFO:teuthology.orchestra.run.gibba002.stdout: "last_epoch_clean": 0, 2021-02-10T11:33:20.875 INFO:teuthology.orchestra.run.gibba002.stdout: "source_version": "0'0", 2021-02-10T11:33:20.875 INFO:teuthology.orchestra.run.gibba002.stdout: "target_version": "0'0" 2021-02-10T11:33:20.875 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:33:20.875 INFO:teuthology.orchestra.run.gibba002.stdout: "last_change": "120", 2021-02-10T11:33:20.875 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend": "0", 2021-02-10T11:33:20.875 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:33:20.875 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:33:20.875 INFO:teuthology.orchestra.run.gibba002.stdout: "auid": 0, 2021-02-10T11:33:20.875 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:33:20.875 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_seq": 0, 2021-02-10T11:33:20.875 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_epoch": 0, 2021-02-10T11:33:20.875 INFO:teuthology.orchestra.run.gibba002.stdout: "pool_snaps": [], 2021-02-10T11:33:20.875 INFO:teuthology.orchestra.run.gibba002.stdout: "removed_snaps": "[]", 2021-02-10T11:33:20.876 INFO:teuthology.orchestra.run.gibba002.stdout: "quota_max_bytes": 0, 2021-02-10T11:33:20.876 INFO:teuthology.orchestra.run.gibba002.stdout: "quota_max_objects": 0, 2021-02-10T11:33:20.876 INFO:teuthology.orchestra.run.gibba002.stdout: "tiers": [], 2021-02-10T11:33:20.876 INFO:teuthology.orchestra.run.gibba002.stdout: "tier_of": -1, 2021-02-10T11:33:20.876 INFO:teuthology.orchestra.run.gibba002.stdout: "read_tier": -1, 2021-02-10T11:33:20.876 INFO:teuthology.orchestra.run.gibba002.stdout: "write_tier": -1, 2021-02-10T11:33:20.876 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_mode": "none", 2021-02-10T11:33:20.876 INFO:teuthology.orchestra.run.gibba002.stdout: "target_max_bytes": 0, 2021-02-10T11:33:20.876 INFO:teuthology.orchestra.run.gibba002.stdout: "target_max_objects": 0, 2021-02-10T11:33:20.876 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:33:20.876 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:33:20.876 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:33:20.876 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_min_flush_age": 0, 2021-02-10T11:33:20.877 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_min_evict_age": 0, 2021-02-10T11:33:20.877 INFO:teuthology.orchestra.run.gibba002.stdout: "erasure_code_profile": "", 2021-02-10T11:33:20.877 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_params": { 2021-02-10T11:33:20.877 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "none" 2021-02-10T11:33:20.877 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:33:20.877 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_period": 0, 2021-02-10T11:33:20.877 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_count": 0, 2021-02-10T11:33:20.877 INFO:teuthology.orchestra.run.gibba002.stdout: "use_gmt_hitset": true, 2021-02-10T11:33:20.877 INFO:teuthology.orchestra.run.gibba002.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:33:20.877 INFO:teuthology.orchestra.run.gibba002.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:33:20.877 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:33:20.877 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:33:20.877 INFO:teuthology.orchestra.run.gibba002.stdout: "grade_table": [], 2021-02-10T11:33:20.878 INFO:teuthology.orchestra.run.gibba002.stdout: "stripe_width": 0, 2021-02-10T11:33:20.878 INFO:teuthology.orchestra.run.gibba002.stdout: "expected_num_objects": 0, 2021-02-10T11:33:20.878 INFO:teuthology.orchestra.run.gibba002.stdout: "fast_read": false, 2021-02-10T11:33:20.878 INFO:teuthology.orchestra.run.gibba002.stdout: "options": {}, 2021-02-10T11:33:20.878 INFO:teuthology.orchestra.run.gibba002.stdout: "application_metadata": { 2021-02-10T11:33:20.878 INFO:teuthology.orchestra.run.gibba002.stdout: "cephfs": { 2021-02-10T11:33:20.878 INFO:teuthology.orchestra.run.gibba002.stdout: "data": "cephfs" 2021-02-10T11:33:20.878 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:33:20.878 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:33:20.878 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:33:20.878 INFO:teuthology.orchestra.run.gibba002.stdout: ], 2021-02-10T11:33:20.878 INFO:teuthology.orchestra.run.gibba002.stdout: "osds": [ 2021-02-10T11:33:20.878 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:33:20.879 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 0, 2021-02-10T11:33:20.879 INFO:teuthology.orchestra.run.gibba002.stdout: "uuid": "be691b4d-a039-408a-af80-9d53f02b6f1f", 2021-02-10T11:33:20.879 INFO:teuthology.orchestra.run.gibba002.stdout: "up": 1, 2021-02-10T11:33:20.879 INFO:teuthology.orchestra.run.gibba002.stdout: "in": 1, 2021-02-10T11:33:20.879 INFO:teuthology.orchestra.run.gibba002.stdout: "weight": 1, 2021-02-10T11:33:20.879 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_affinity": 1, 2021-02-10T11:33:20.879 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_begin": 0, 2021-02-10T11:33:20.879 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_end": 0, 2021-02-10T11:33:20.879 INFO:teuthology.orchestra.run.gibba002.stdout: "up_from": 13, 2021-02-10T11:33:20.879 INFO:teuthology.orchestra.run.gibba002.stdout: "up_thru": 118, 2021-02-10T11:33:20.879 INFO:teuthology.orchestra.run.gibba002.stdout: "down_at": 0, 2021-02-10T11:33:20.879 INFO:teuthology.orchestra.run.gibba002.stdout: "lost_at": 0, 2021-02-10T11:33:20.880 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addrs": { 2021-02-10T11:33:20.880 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:33:20.880 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:33:20.880 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:33:20.880 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6818", 2021-02-10T11:33:20.880 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36088 2021-02-10T11:33:20.880 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:33:20.880 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:33:20.880 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:33:20.880 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6820", 2021-02-10T11:33:20.880 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36088 2021-02-10T11:33:20.880 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:33:20.880 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:33:20.880 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:33:20.881 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addrs": { 2021-02-10T11:33:20.881 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:33:20.881 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:33:20.881 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:33:20.881 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6822", 2021-02-10T11:33:20.881 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36088 2021-02-10T11:33:20.881 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:33:20.881 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:33:20.881 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:33:20.881 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6824", 2021-02-10T11:33:20.881 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36088 2021-02-10T11:33:20.881 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:33:20.881 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:33:20.881 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:33:20.882 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addrs": { 2021-02-10T11:33:20.882 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:33:20.882 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:33:20.882 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:33:20.882 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6830", 2021-02-10T11:33:20.882 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36088 2021-02-10T11:33:20.882 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:33:20.882 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:33:20.882 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:33:20.882 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6832", 2021-02-10T11:33:20.882 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36088 2021-02-10T11:33:20.882 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:33:20.882 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:33:20.883 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:33:20.883 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addrs": { 2021-02-10T11:33:20.883 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:33:20.883 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:33:20.883 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:33:20.883 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6826", 2021-02-10T11:33:20.883 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36088 2021-02-10T11:33:20.883 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:33:20.883 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:33:20.883 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:33:20.883 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6828", 2021-02-10T11:33:20.883 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36088 2021-02-10T11:33:20.883 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:33:20.883 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:33:20.884 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:33:20.884 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addr": "172.21.2.102:6820/36088", 2021-02-10T11:33:20.884 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addr": "172.21.2.102:6824/36088", 2021-02-10T11:33:20.884 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addr": "172.21.2.102:6832/36088", 2021-02-10T11:33:20.884 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addr": "172.21.2.102:6828/36088", 2021-02-10T11:33:20.884 INFO:teuthology.orchestra.run.gibba002.stdout: "state": [ 2021-02-10T11:33:20.884 INFO:teuthology.orchestra.run.gibba002.stdout: "exists", 2021-02-10T11:33:20.884 INFO:teuthology.orchestra.run.gibba002.stdout: "up" 2021-02-10T11:33:20.884 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:33:20.884 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:33:20.884 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:33:20.884 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 1, 2021-02-10T11:33:20.884 INFO:teuthology.orchestra.run.gibba002.stdout: "uuid": "64e3ccc8-9c69-4d7c-865e-af8e83822635", 2021-02-10T11:33:20.884 INFO:teuthology.orchestra.run.gibba002.stdout: "up": 1, 2021-02-10T11:33:20.885 INFO:teuthology.orchestra.run.gibba002.stdout: "in": 1, 2021-02-10T11:33:20.885 INFO:teuthology.orchestra.run.gibba002.stdout: "weight": 1, 2021-02-10T11:33:20.885 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_affinity": 1, 2021-02-10T11:33:20.885 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_begin": 0, 2021-02-10T11:33:20.885 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_end": 0, 2021-02-10T11:33:20.885 INFO:teuthology.orchestra.run.gibba002.stdout: "up_from": 13, 2021-02-10T11:33:20.885 INFO:teuthology.orchestra.run.gibba002.stdout: "up_thru": 117, 2021-02-10T11:33:20.885 INFO:teuthology.orchestra.run.gibba002.stdout: "down_at": 0, 2021-02-10T11:33:20.885 INFO:teuthology.orchestra.run.gibba002.stdout: "lost_at": 0, 2021-02-10T11:33:20.885 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addrs": { 2021-02-10T11:33:20.885 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:33:20.885 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:33:20.885 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:33:20.886 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6819", 2021-02-10T11:33:20.886 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36087 2021-02-10T11:33:20.886 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:33:20.886 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:33:20.886 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:33:20.886 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6821", 2021-02-10T11:33:20.886 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36087 2021-02-10T11:33:20.886 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:33:20.886 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:33:20.886 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:33:20.886 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addrs": { 2021-02-10T11:33:20.886 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:33:20.886 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:33:20.887 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:33:20.887 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6823", 2021-02-10T11:33:20.887 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36087 2021-02-10T11:33:20.887 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:33:20.887 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:33:20.887 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:33:20.887 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6825", 2021-02-10T11:33:20.887 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36087 2021-02-10T11:33:20.887 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:33:20.887 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:33:20.887 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:33:20.887 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addrs": { 2021-02-10T11:33:20.887 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:33:20.887 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:33:20.888 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:33:20.888 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6831", 2021-02-10T11:33:20.888 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36087 2021-02-10T11:33:20.888 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:33:20.888 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:33:20.888 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:33:20.888 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6833", 2021-02-10T11:33:20.888 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36087 2021-02-10T11:33:20.888 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:33:20.888 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:33:20.888 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:33:20.888 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addrs": { 2021-02-10T11:33:20.888 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:33:20.888 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:33:20.889 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:33:20.889 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6827", 2021-02-10T11:33:20.889 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36087 2021-02-10T11:33:20.889 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:33:20.889 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:33:20.889 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:33:20.889 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6829", 2021-02-10T11:33:20.889 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36087 2021-02-10T11:33:20.889 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:33:20.889 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:33:20.889 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:33:20.889 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addr": "172.21.2.102:6821/36087", 2021-02-10T11:33:20.889 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addr": "172.21.2.102:6825/36087", 2021-02-10T11:33:20.890 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addr": "172.21.2.102:6833/36087", 2021-02-10T11:33:20.890 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addr": "172.21.2.102:6829/36087", 2021-02-10T11:33:20.890 INFO:teuthology.orchestra.run.gibba002.stdout: "state": [ 2021-02-10T11:33:20.890 INFO:teuthology.orchestra.run.gibba002.stdout: "exists", 2021-02-10T11:33:20.890 INFO:teuthology.orchestra.run.gibba002.stdout: "up" 2021-02-10T11:33:20.890 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:33:20.890 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:33:20.890 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:33:20.890 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 2, 2021-02-10T11:33:20.890 INFO:teuthology.orchestra.run.gibba002.stdout: "uuid": "153700af-747f-481f-bdf4-d6072d3a842f", 2021-02-10T11:33:20.890 INFO:teuthology.orchestra.run.gibba002.stdout: "up": 1, 2021-02-10T11:33:20.890 INFO:teuthology.orchestra.run.gibba002.stdout: "in": 1, 2021-02-10T11:33:20.890 INFO:teuthology.orchestra.run.gibba002.stdout: "weight": 1, 2021-02-10T11:33:20.891 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_affinity": 1, 2021-02-10T11:33:20.891 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_begin": 0, 2021-02-10T11:33:20.891 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_end": 0, 2021-02-10T11:33:20.891 INFO:teuthology.orchestra.run.gibba002.stdout: "up_from": 13, 2021-02-10T11:33:20.891 INFO:teuthology.orchestra.run.gibba002.stdout: "up_thru": 118, 2021-02-10T11:33:20.891 INFO:teuthology.orchestra.run.gibba002.stdout: "down_at": 0, 2021-02-10T11:33:20.891 INFO:teuthology.orchestra.run.gibba002.stdout: "lost_at": 0, 2021-02-10T11:33:20.891 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addrs": { 2021-02-10T11:33:20.891 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:33:20.891 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:33:20.891 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:33:20.891 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6810", 2021-02-10T11:33:20.891 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36089 2021-02-10T11:33:20.892 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:33:20.892 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:33:20.892 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:33:20.892 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6811", 2021-02-10T11:33:20.892 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36089 2021-02-10T11:33:20.892 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:33:20.892 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:33:20.892 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:33:20.892 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addrs": { 2021-02-10T11:33:20.892 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:33:20.892 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:33:20.892 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:33:20.892 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6812", 2021-02-10T11:33:20.893 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36089 2021-02-10T11:33:20.893 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:33:20.893 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:33:20.893 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:33:20.893 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6813", 2021-02-10T11:33:20.893 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36089 2021-02-10T11:33:20.893 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:33:20.893 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:33:20.893 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:33:20.893 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addrs": { 2021-02-10T11:33:20.893 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:33:20.893 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:33:20.893 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:33:20.893 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6816", 2021-02-10T11:33:20.894 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36089 2021-02-10T11:33:20.894 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:33:20.894 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:33:20.894 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:33:20.894 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6817", 2021-02-10T11:33:20.894 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36089 2021-02-10T11:33:20.894 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:33:20.894 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:33:20.894 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:33:20.894 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addrs": { 2021-02-10T11:33:20.894 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:33:20.894 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:33:20.894 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:33:20.895 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6814", 2021-02-10T11:33:20.895 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36089 2021-02-10T11:33:20.895 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:33:20.895 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:33:20.895 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:33:20.895 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6815", 2021-02-10T11:33:20.895 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36089 2021-02-10T11:33:20.895 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:33:20.895 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:33:20.895 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:33:20.895 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addr": "172.21.2.102:6811/36089", 2021-02-10T11:33:20.895 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addr": "172.21.2.102:6813/36089", 2021-02-10T11:33:20.895 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addr": "172.21.2.102:6817/36089", 2021-02-10T11:33:20.895 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addr": "172.21.2.102:6815/36089", 2021-02-10T11:33:20.896 INFO:teuthology.orchestra.run.gibba002.stdout: "state": [ 2021-02-10T11:33:20.896 INFO:teuthology.orchestra.run.gibba002.stdout: "exists", 2021-02-10T11:33:20.896 INFO:teuthology.orchestra.run.gibba002.stdout: "up" 2021-02-10T11:33:20.896 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:33:20.896 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:33:20.896 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:33:20.896 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 3, 2021-02-10T11:33:20.896 INFO:teuthology.orchestra.run.gibba002.stdout: "uuid": "cbc678f4-83a9-45e6-8830-2f7c232b1ea7", 2021-02-10T11:33:20.896 INFO:teuthology.orchestra.run.gibba002.stdout: "up": 1, 2021-02-10T11:33:20.896 INFO:teuthology.orchestra.run.gibba002.stdout: "in": 1, 2021-02-10T11:33:20.896 INFO:teuthology.orchestra.run.gibba002.stdout: "weight": 1, 2021-02-10T11:33:20.896 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_affinity": 1, 2021-02-10T11:33:20.896 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_begin": 0, 2021-02-10T11:33:20.896 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_end": 0, 2021-02-10T11:33:20.897 INFO:teuthology.orchestra.run.gibba002.stdout: "up_from": 13, 2021-02-10T11:33:20.897 INFO:teuthology.orchestra.run.gibba002.stdout: "up_thru": 118, 2021-02-10T11:33:20.897 INFO:teuthology.orchestra.run.gibba002.stdout: "down_at": 0, 2021-02-10T11:33:20.897 INFO:teuthology.orchestra.run.gibba002.stdout: "lost_at": 0, 2021-02-10T11:33:20.897 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addrs": { 2021-02-10T11:33:20.897 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:33:20.897 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:33:20.897 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:33:20.897 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6802", 2021-02-10T11:33:20.897 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36090 2021-02-10T11:33:20.897 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:33:20.897 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:33:20.897 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:33:20.897 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6803", 2021-02-10T11:33:20.898 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36090 2021-02-10T11:33:20.898 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:33:20.898 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:33:20.898 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:33:20.898 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addrs": { 2021-02-10T11:33:20.898 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:33:20.898 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:33:20.898 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:33:20.898 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6804", 2021-02-10T11:33:20.898 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36090 2021-02-10T11:33:20.898 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:33:20.898 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:33:20.899 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:33:20.899 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6805", 2021-02-10T11:33:20.899 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36090 2021-02-10T11:33:20.899 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:33:20.899 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:33:20.899 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:33:20.899 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addrs": { 2021-02-10T11:33:20.899 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:33:20.899 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:33:20.899 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:33:20.899 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6808", 2021-02-10T11:33:20.899 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36090 2021-02-10T11:33:20.899 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:33:20.900 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:33:20.900 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:33:20.900 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6809", 2021-02-10T11:33:20.900 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36090 2021-02-10T11:33:20.900 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:33:20.900 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:33:20.900 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:33:20.900 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addrs": { 2021-02-10T11:33:20.900 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:33:20.900 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:33:20.900 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:33:20.900 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6806", 2021-02-10T11:33:20.900 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36090 2021-02-10T11:33:20.901 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:33:20.901 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:33:20.901 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:33:20.901 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6807", 2021-02-10T11:33:20.901 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36090 2021-02-10T11:33:20.901 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:33:20.901 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:33:20.901 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:33:20.901 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addr": "172.21.2.102:6803/36090", 2021-02-10T11:33:20.901 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addr": "172.21.2.102:6805/36090", 2021-02-10T11:33:20.901 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addr": "172.21.2.102:6809/36090", 2021-02-10T11:33:20.901 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addr": "172.21.2.102:6807/36090", 2021-02-10T11:33:20.901 INFO:teuthology.orchestra.run.gibba002.stdout: "state": [ 2021-02-10T11:33:20.902 INFO:teuthology.orchestra.run.gibba002.stdout: "exists", 2021-02-10T11:33:20.902 INFO:teuthology.orchestra.run.gibba002.stdout: "up" 2021-02-10T11:33:20.902 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:33:20.902 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:33:20.902 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:33:20.902 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 4, 2021-02-10T11:33:20.902 INFO:teuthology.orchestra.run.gibba002.stdout: "uuid": "c01e4008-b258-4f10-afd2-9e7f37214351", 2021-02-10T11:33:20.902 INFO:teuthology.orchestra.run.gibba002.stdout: "up": 1, 2021-02-10T11:33:20.902 INFO:teuthology.orchestra.run.gibba002.stdout: "in": 1, 2021-02-10T11:33:20.902 INFO:teuthology.orchestra.run.gibba002.stdout: "weight": 1, 2021-02-10T11:33:20.902 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_affinity": 1, 2021-02-10T11:33:20.902 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_begin": 0, 2021-02-10T11:33:20.902 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_end": 0, 2021-02-10T11:33:20.903 INFO:teuthology.orchestra.run.gibba002.stdout: "up_from": 13, 2021-02-10T11:33:20.903 INFO:teuthology.orchestra.run.gibba002.stdout: "up_thru": 110, 2021-02-10T11:33:20.903 INFO:teuthology.orchestra.run.gibba002.stdout: "down_at": 0, 2021-02-10T11:33:20.903 INFO:teuthology.orchestra.run.gibba002.stdout: "lost_at": 0, 2021-02-10T11:33:20.903 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addrs": { 2021-02-10T11:33:20.903 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:33:20.903 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:33:20.903 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:33:20.903 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6808", 2021-02-10T11:33:20.903 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36368 2021-02-10T11:33:20.903 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:33:20.903 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:33:20.904 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:33:20.904 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6811", 2021-02-10T11:33:20.904 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36368 2021-02-10T11:33:20.904 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:33:20.904 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:33:20.904 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:33:20.904 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addrs": { 2021-02-10T11:33:20.904 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:33:20.904 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:33:20.904 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:33:20.904 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6814", 2021-02-10T11:33:20.904 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36368 2021-02-10T11:33:20.904 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:33:20.905 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:33:20.905 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:33:20.905 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6818", 2021-02-10T11:33:20.905 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36368 2021-02-10T11:33:20.905 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:33:20.905 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:33:20.905 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:33:20.905 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addrs": { 2021-02-10T11:33:20.905 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:33:20.905 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:33:20.905 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:33:20.905 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6826", 2021-02-10T11:33:20.905 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36368 2021-02-10T11:33:20.906 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:33:20.906 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:33:20.906 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:33:20.906 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6828", 2021-02-10T11:33:20.906 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36368 2021-02-10T11:33:20.906 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:33:20.906 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:33:20.906 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:33:20.906 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addrs": { 2021-02-10T11:33:20.906 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:33:20.906 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:33:20.906 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:33:20.906 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6820", 2021-02-10T11:33:20.906 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36368 2021-02-10T11:33:20.907 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:33:20.907 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:33:20.907 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:33:20.907 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6824", 2021-02-10T11:33:20.907 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36368 2021-02-10T11:33:20.907 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:33:20.907 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:33:20.907 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:33:20.907 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addr": "172.21.2.113:6811/36368", 2021-02-10T11:33:20.907 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addr": "172.21.2.113:6818/36368", 2021-02-10T11:33:20.907 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addr": "172.21.2.113:6828/36368", 2021-02-10T11:33:20.907 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addr": "172.21.2.113:6824/36368", 2021-02-10T11:33:20.907 INFO:teuthology.orchestra.run.gibba002.stdout: "state": [ 2021-02-10T11:33:20.907 INFO:teuthology.orchestra.run.gibba002.stdout: "exists", 2021-02-10T11:33:20.908 INFO:teuthology.orchestra.run.gibba002.stdout: "up" 2021-02-10T11:33:20.908 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:33:20.908 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:33:20.908 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:33:20.908 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 5, 2021-02-10T11:33:20.908 INFO:teuthology.orchestra.run.gibba002.stdout: "uuid": "551f6007-daf8-4a98-8ee0-c2b16bcccb5e", 2021-02-10T11:33:20.908 INFO:teuthology.orchestra.run.gibba002.stdout: "up": 1, 2021-02-10T11:33:20.908 INFO:teuthology.orchestra.run.gibba002.stdout: "in": 1, 2021-02-10T11:33:20.908 INFO:teuthology.orchestra.run.gibba002.stdout: "weight": 1, 2021-02-10T11:33:20.908 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_affinity": 1, 2021-02-10T11:33:20.908 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_begin": 0, 2021-02-10T11:33:20.908 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_end": 0, 2021-02-10T11:33:20.908 INFO:teuthology.orchestra.run.gibba002.stdout: "up_from": 13, 2021-02-10T11:33:20.909 INFO:teuthology.orchestra.run.gibba002.stdout: "up_thru": 117, 2021-02-10T11:33:20.909 INFO:teuthology.orchestra.run.gibba002.stdout: "down_at": 0, 2021-02-10T11:33:20.909 INFO:teuthology.orchestra.run.gibba002.stdout: "lost_at": 0, 2021-02-10T11:33:20.909 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addrs": { 2021-02-10T11:33:20.909 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:33:20.909 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:33:20.909 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:33:20.909 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6802", 2021-02-10T11:33:20.909 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36366 2021-02-10T11:33:20.909 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:33:20.909 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:33:20.909 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:33:20.909 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6805", 2021-02-10T11:33:20.909 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36366 2021-02-10T11:33:20.910 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:33:20.910 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:33:20.910 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:33:20.910 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addrs": { 2021-02-10T11:33:20.910 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:33:20.910 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:33:20.910 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:33:20.910 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6807", 2021-02-10T11:33:20.910 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36366 2021-02-10T11:33:20.910 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:33:20.910 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:33:20.910 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:33:20.910 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6810", 2021-02-10T11:33:20.911 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36366 2021-02-10T11:33:20.911 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:33:20.911 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:33:20.911 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:33:20.911 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addrs": { 2021-02-10T11:33:20.911 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:33:20.911 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:33:20.911 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:33:20.911 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6819", 2021-02-10T11:33:20.911 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36366 2021-02-10T11:33:20.911 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:33:20.911 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:33:20.911 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:33:20.911 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6822", 2021-02-10T11:33:20.912 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36366 2021-02-10T11:33:20.912 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:33:20.912 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:33:20.912 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:33:20.912 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addrs": { 2021-02-10T11:33:20.912 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:33:20.912 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:33:20.912 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:33:20.912 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6812", 2021-02-10T11:33:20.912 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36366 2021-02-10T11:33:20.912 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:33:20.912 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:33:20.912 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:33:20.912 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6815", 2021-02-10T11:33:20.913 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36366 2021-02-10T11:33:20.913 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:33:20.913 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:33:20.913 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:33:20.913 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addr": "172.21.2.113:6805/36366", 2021-02-10T11:33:20.913 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addr": "172.21.2.113:6810/36366", 2021-02-10T11:33:20.913 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addr": "172.21.2.113:6822/36366", 2021-02-10T11:33:20.913 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addr": "172.21.2.113:6815/36366", 2021-02-10T11:33:20.913 INFO:teuthology.orchestra.run.gibba002.stdout: "state": [ 2021-02-10T11:33:20.913 INFO:teuthology.orchestra.run.gibba002.stdout: "exists", 2021-02-10T11:33:20.913 INFO:teuthology.orchestra.run.gibba002.stdout: "up" 2021-02-10T11:33:20.913 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:33:20.913 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:33:20.913 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:33:20.914 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 6, 2021-02-10T11:33:20.914 INFO:teuthology.orchestra.run.gibba002.stdout: "uuid": "bf67c040-0e50-48be-b102-c3943bc9ebef", 2021-02-10T11:33:20.914 INFO:teuthology.orchestra.run.gibba002.stdout: "up": 1, 2021-02-10T11:33:20.914 INFO:teuthology.orchestra.run.gibba002.stdout: "in": 1, 2021-02-10T11:33:20.914 INFO:teuthology.orchestra.run.gibba002.stdout: "weight": 1, 2021-02-10T11:33:20.914 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_affinity": 1, 2021-02-10T11:33:20.914 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_begin": 0, 2021-02-10T11:33:20.914 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_end": 0, 2021-02-10T11:33:20.914 INFO:teuthology.orchestra.run.gibba002.stdout: "up_from": 13, 2021-02-10T11:33:20.914 INFO:teuthology.orchestra.run.gibba002.stdout: "up_thru": 118, 2021-02-10T11:33:20.914 INFO:teuthology.orchestra.run.gibba002.stdout: "down_at": 0, 2021-02-10T11:33:20.914 INFO:teuthology.orchestra.run.gibba002.stdout: "lost_at": 0, 2021-02-10T11:33:20.914 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addrs": { 2021-02-10T11:33:20.915 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:33:20.915 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:33:20.915 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:33:20.915 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6800", 2021-02-10T11:33:20.915 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36367 2021-02-10T11:33:20.915 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:33:20.915 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:33:20.915 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:33:20.915 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6801", 2021-02-10T11:33:20.915 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36367 2021-02-10T11:33:20.915 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:33:20.915 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:33:20.915 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:33:20.915 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addrs": { 2021-02-10T11:33:20.916 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:33:20.916 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:33:20.916 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:33:20.916 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6803", 2021-02-10T11:33:20.916 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36367 2021-02-10T11:33:20.916 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:33:20.916 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:33:20.916 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:33:20.916 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6804", 2021-02-10T11:33:20.916 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36367 2021-02-10T11:33:20.916 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:33:20.916 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:33:20.916 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:33:20.917 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addrs": { 2021-02-10T11:33:20.917 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:33:20.917 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:33:20.917 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:33:20.917 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6813", 2021-02-10T11:33:20.917 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36367 2021-02-10T11:33:20.917 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:33:20.917 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:33:20.917 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:33:20.917 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6816", 2021-02-10T11:33:20.917 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36367 2021-02-10T11:33:20.917 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:33:20.917 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:33:20.917 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:33:20.918 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addrs": { 2021-02-10T11:33:20.918 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:33:20.918 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:33:20.918 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:33:20.918 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6806", 2021-02-10T11:33:20.918 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36367 2021-02-10T11:33:20.918 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:33:20.918 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:33:20.918 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:33:20.918 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6809", 2021-02-10T11:33:20.918 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36367 2021-02-10T11:33:20.918 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:33:20.918 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:33:20.919 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:33:20.919 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addr": "172.21.2.113:6801/36367", 2021-02-10T11:33:20.919 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addr": "172.21.2.113:6804/36367", 2021-02-10T11:33:20.919 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addr": "172.21.2.113:6816/36367", 2021-02-10T11:33:20.919 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addr": "172.21.2.113:6809/36367", 2021-02-10T11:33:20.919 INFO:teuthology.orchestra.run.gibba002.stdout: "state": [ 2021-02-10T11:33:20.919 INFO:teuthology.orchestra.run.gibba002.stdout: "exists", 2021-02-10T11:33:20.919 INFO:teuthology.orchestra.run.gibba002.stdout: "up" 2021-02-10T11:33:20.919 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:33:20.919 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:33:20.919 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:33:20.919 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 7, 2021-02-10T11:33:20.919 INFO:teuthology.orchestra.run.gibba002.stdout: "uuid": "f5461f80-018d-428f-be89-2eb2bcbe7bb6", 2021-02-10T11:33:20.919 INFO:teuthology.orchestra.run.gibba002.stdout: "up": 1, 2021-02-10T11:33:20.920 INFO:teuthology.orchestra.run.gibba002.stdout: "in": 1, 2021-02-10T11:33:20.920 INFO:teuthology.orchestra.run.gibba002.stdout: "weight": 1, 2021-02-10T11:33:20.920 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_affinity": 1, 2021-02-10T11:33:20.920 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_begin": 0, 2021-02-10T11:33:20.920 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_end": 0, 2021-02-10T11:33:20.920 INFO:teuthology.orchestra.run.gibba002.stdout: "up_from": 13, 2021-02-10T11:33:20.920 INFO:teuthology.orchestra.run.gibba002.stdout: "up_thru": 118, 2021-02-10T11:33:20.920 INFO:teuthology.orchestra.run.gibba002.stdout: "down_at": 0, 2021-02-10T11:33:20.920 INFO:teuthology.orchestra.run.gibba002.stdout: "lost_at": 0, 2021-02-10T11:33:20.920 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addrs": { 2021-02-10T11:33:20.920 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:33:20.920 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:33:20.920 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:33:20.920 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6817", 2021-02-10T11:33:20.921 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36365 2021-02-10T11:33:20.921 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:33:20.921 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:33:20.921 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:33:20.921 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6821", 2021-02-10T11:33:20.921 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36365 2021-02-10T11:33:20.921 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:33:20.921 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:33:20.921 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:33:20.921 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addrs": { 2021-02-10T11:33:20.921 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:33:20.921 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:33:20.921 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:33:20.921 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6823", 2021-02-10T11:33:20.922 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36365 2021-02-10T11:33:20.922 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:33:20.922 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:33:20.922 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:33:20.922 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6825", 2021-02-10T11:33:20.922 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36365 2021-02-10T11:33:20.922 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:33:20.922 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:33:20.922 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:33:20.922 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addrs": { 2021-02-10T11:33:20.922 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:33:20.922 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:33:20.922 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:33:20.922 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6830", 2021-02-10T11:33:20.923 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36365 2021-02-10T11:33:20.923 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:33:20.923 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:33:20.923 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:33:20.923 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6831", 2021-02-10T11:33:20.923 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36365 2021-02-10T11:33:20.923 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:33:20.923 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:33:20.923 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:33:20.923 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addrs": { 2021-02-10T11:33:20.923 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:33:20.923 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:33:20.923 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:33:20.923 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6827", 2021-02-10T11:33:20.924 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36365 2021-02-10T11:33:20.924 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:33:20.924 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:33:20.924 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:33:20.924 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6829", 2021-02-10T11:33:20.924 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36365 2021-02-10T11:33:20.924 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:33:20.924 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:33:20.924 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:33:20.924 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addr": "172.21.2.113:6821/36365", 2021-02-10T11:33:20.924 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addr": "172.21.2.113:6825/36365", 2021-02-10T11:33:20.924 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addr": "172.21.2.113:6831/36365", 2021-02-10T11:33:20.924 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addr": "172.21.2.113:6829/36365", 2021-02-10T11:33:20.925 INFO:teuthology.orchestra.run.gibba002.stdout: "state": [ 2021-02-10T11:33:20.925 INFO:teuthology.orchestra.run.gibba002.stdout: "exists", 2021-02-10T11:33:20.925 INFO:teuthology.orchestra.run.gibba002.stdout: "up" 2021-02-10T11:33:20.925 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:33:20.925 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:33:20.925 INFO:teuthology.orchestra.run.gibba002.stdout: ], 2021-02-10T11:33:20.925 INFO:teuthology.orchestra.run.gibba002.stdout: "osd_xinfo": [ 2021-02-10T11:33:20.925 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:33:20.925 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 0, 2021-02-10T11:33:20.925 INFO:teuthology.orchestra.run.gibba002.stdout: "down_stamp": "0.000000", 2021-02-10T11:33:20.925 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_probability": 0, 2021-02-10T11:33:20.925 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_interval": 0, 2021-02-10T11:33:20.925 INFO:teuthology.orchestra.run.gibba002.stdout: "features": 4611087854035861503, 2021-02-10T11:33:20.925 INFO:teuthology.orchestra.run.gibba002.stdout: "old_weight": 0 2021-02-10T11:33:20.926 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:33:20.926 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:33:20.926 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 1, 2021-02-10T11:33:20.926 INFO:teuthology.orchestra.run.gibba002.stdout: "down_stamp": "0.000000", 2021-02-10T11:33:20.926 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_probability": 0, 2021-02-10T11:33:20.926 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_interval": 0, 2021-02-10T11:33:20.926 INFO:teuthology.orchestra.run.gibba002.stdout: "features": 4611087854035861503, 2021-02-10T11:33:20.926 INFO:teuthology.orchestra.run.gibba002.stdout: "old_weight": 0 2021-02-10T11:33:20.926 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:33:20.926 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:33:20.926 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 2, 2021-02-10T11:33:20.926 INFO:teuthology.orchestra.run.gibba002.stdout: "down_stamp": "0.000000", 2021-02-10T11:33:20.926 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_probability": 0, 2021-02-10T11:33:20.927 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_interval": 0, 2021-02-10T11:33:20.927 INFO:teuthology.orchestra.run.gibba002.stdout: "features": 4611087854035861503, 2021-02-10T11:33:20.927 INFO:teuthology.orchestra.run.gibba002.stdout: "old_weight": 0 2021-02-10T11:33:20.927 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:33:20.927 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:33:20.927 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 3, 2021-02-10T11:33:20.927 INFO:teuthology.orchestra.run.gibba002.stdout: "down_stamp": "0.000000", 2021-02-10T11:33:20.927 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_probability": 0, 2021-02-10T11:33:20.927 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_interval": 0, 2021-02-10T11:33:20.927 INFO:teuthology.orchestra.run.gibba002.stdout: "features": 4611087854035861503, 2021-02-10T11:33:20.927 INFO:teuthology.orchestra.run.gibba002.stdout: "old_weight": 0 2021-02-10T11:33:20.927 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:33:20.927 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:33:20.927 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 4, 2021-02-10T11:33:20.928 INFO:teuthology.orchestra.run.gibba002.stdout: "down_stamp": "0.000000", 2021-02-10T11:33:20.928 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_probability": 0, 2021-02-10T11:33:20.928 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_interval": 0, 2021-02-10T11:33:20.928 INFO:teuthology.orchestra.run.gibba002.stdout: "features": 4611087854035861503, 2021-02-10T11:33:20.928 INFO:teuthology.orchestra.run.gibba002.stdout: "old_weight": 0 2021-02-10T11:33:20.928 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:33:20.928 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:33:20.928 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 5, 2021-02-10T11:33:20.928 INFO:teuthology.orchestra.run.gibba002.stdout: "down_stamp": "0.000000", 2021-02-10T11:33:20.928 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_probability": 0, 2021-02-10T11:33:20.928 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_interval": 0, 2021-02-10T11:33:20.928 INFO:teuthology.orchestra.run.gibba002.stdout: "features": 4611087854035861503, 2021-02-10T11:33:20.928 INFO:teuthology.orchestra.run.gibba002.stdout: "old_weight": 0 2021-02-10T11:33:20.929 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:33:20.929 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:33:20.929 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 6, 2021-02-10T11:33:20.929 INFO:teuthology.orchestra.run.gibba002.stdout: "down_stamp": "0.000000", 2021-02-10T11:33:20.929 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_probability": 0, 2021-02-10T11:33:20.929 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_interval": 0, 2021-02-10T11:33:20.929 INFO:teuthology.orchestra.run.gibba002.stdout: "features": 4611087854035861503, 2021-02-10T11:33:20.929 INFO:teuthology.orchestra.run.gibba002.stdout: "old_weight": 0 2021-02-10T11:33:20.929 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:33:20.929 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:33:20.929 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 7, 2021-02-10T11:33:20.929 INFO:teuthology.orchestra.run.gibba002.stdout: "down_stamp": "0.000000", 2021-02-10T11:33:20.929 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_probability": 0, 2021-02-10T11:33:20.929 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_interval": 0, 2021-02-10T11:33:20.930 INFO:teuthology.orchestra.run.gibba002.stdout: "features": 4611087854035861503, 2021-02-10T11:33:20.930 INFO:teuthology.orchestra.run.gibba002.stdout: "old_weight": 0 2021-02-10T11:33:20.930 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:33:20.930 INFO:teuthology.orchestra.run.gibba002.stdout: ], 2021-02-10T11:33:20.930 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_upmap": [], 2021-02-10T11:33:20.930 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_upmap_items": [], 2021-02-10T11:33:20.930 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_temp": [], 2021-02-10T11:33:20.930 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_temp": [], 2021-02-10T11:33:20.930 INFO:teuthology.orchestra.run.gibba002.stdout: "blacklist": {}, 2021-02-10T11:33:20.930 INFO:teuthology.orchestra.run.gibba002.stdout: "erasure_code_profiles": { 2021-02-10T11:33:20.930 INFO:teuthology.orchestra.run.gibba002.stdout: "default": { 2021-02-10T11:33:20.930 INFO:teuthology.orchestra.run.gibba002.stdout: "crush-failure-domain": "osd", 2021-02-10T11:33:20.930 INFO:teuthology.orchestra.run.gibba002.stdout: "k": "2", 2021-02-10T11:33:20.930 INFO:teuthology.orchestra.run.gibba002.stdout: "m": "1", 2021-02-10T11:33:20.931 INFO:teuthology.orchestra.run.gibba002.stdout: "plugin": "jerasure", 2021-02-10T11:33:20.931 INFO:teuthology.orchestra.run.gibba002.stdout: "ruleset-failure-domain": "osd", 2021-02-10T11:33:20.931 INFO:teuthology.orchestra.run.gibba002.stdout: "technique": "reed_sol_van" 2021-02-10T11:33:20.931 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:33:20.931 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:33:20.931 INFO:teuthology.orchestra.run.gibba002.stdout: "removed_snaps_queue": [], 2021-02-10T11:33:20.931 INFO:teuthology.orchestra.run.gibba002.stdout: "new_removed_snaps": [], 2021-02-10T11:33:20.931 INFO:teuthology.orchestra.run.gibba002.stdout: "new_purged_snaps": [], 2021-02-10T11:33:20.931 INFO:teuthology.orchestra.run.gibba002.stdout: "crush_node_flags": {}, 2021-02-10T11:33:20.931 INFO:teuthology.orchestra.run.gibba002.stdout: "device_class_flags": {} 2021-02-10T11:33:20.931 INFO:teuthology.orchestra.run.gibba002.stdout:} 2021-02-10T11:33:20.932 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:20.860 7f0908e76700 1 -- 172.21.2.102:0/2201648316 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f08e80186c0 msgr2=0x7f08e801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:20.933 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:20.860 7f0908e76700 1 --2- 172.21.2.102:0/2201648316 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f08e80186c0 0x7f08e801abc0 crc :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:20.933 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:20.860 7f0908e76700 1 -- 172.21.2.102:0/2201648316 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f09040befd0 msgr2=0x7f09041a7970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:20.933 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:20.860 7f0908e76700 1 --2- 172.21.2.102:0/2201648316 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f09040befd0 0x7f09041a7970 secure :-1 s=READY pgs=2378 cs=0 l=1 rev1=1 rx=0x7f08fc00b890 tx=0x7f08fc00bc30).stop 2021-02-10T11:33:20.933 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:20.860 7f0908e76700 1 -- 172.21.2.102:0/2201648316 shutdown_connections 2021-02-10T11:33:20.933 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:20.860 7f0908e76700 1 --2- 172.21.2.102:0/2201648316 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f09040befd0 0x7f09041a7970 unknown :-1 s=CLOSED pgs=2378 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:20.933 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:20.860 7f0908e76700 1 --2- 172.21.2.102:0/2201648316 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f09040bebd0 0x7f09041a7350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:20.933 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:20.860 7f0908e76700 1 --2- 172.21.2.102:0/2201648316 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f08e80186c0 0x7f08e801abc0 unknown :-1 s=CLOSED pgs=1052 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:20.933 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:20.860 7f0908e76700 1 --2- 172.21.2.102:0/2201648316 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f09041072b0 0x7f09041097d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:20.933 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:20.860 7f0908e76700 1 -- 172.21.2.102:0/2201648316 >> 172.21.2.102:0/2201648316 conn(0x7f09040b5df0 msgr2=0x7f09040c3c80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:20.933 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:20.860 7f0908e76700 1 -- 172.21.2.102:0/2201648316 shutdown_connections 2021-02-10T11:33:20.933 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:20.860 7f0908e76700 1 -- 172.21.2.102:0/2201648316 wait complete. 2021-02-10T11:33:20.934 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:33:21.362 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:21.344 7f56ea890700 1 Processor -- start 2021-02-10T11:33:21.363 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:21.345 7f56ea890700 1 -- start start 2021-02-10T11:33:21.363 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:21.345 7f56ea890700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f56e40f05f0 0x7f56e40f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:21.363 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:21.345 7f56ea890700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56e41b0c70 con 0x7f56e41b0470 2021-02-10T11:33:21.363 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:21.345 7f56ea890700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56e40f0b50 con 0x7f56e41b0870 2021-02-10T11:33:21.363 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:21.345 7f56ea890700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56e40e5ea0 con 0x7f56e40f05f0 2021-02-10T11:33:21.363 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:21.345 7f56e3fff700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f56e41b0470 0x7f56e40ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:55612/0 (socket says 172.21.2.102:55612) 2021-02-10T11:33:21.363 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:21.345 7f56e3fff700 1 -- 172.21.2.102:0/3497616238 learned_addr learned my addr 172.21.2.102:0/3497616238 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:33:21.363 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:21.346 7f56e2ffd700 1 -- 172.21.2.102:0/3497616238 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2222130575 0 0) 0x7f56d40010b0 con 0x7f56e41b0470 2021-02-10T11:33:21.363 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:21.346 7f56e2ffd700 1 -- 172.21.2.102:0/3497616238 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f56cc002fe0 con 0x7f56e41b0470 2021-02-10T11:33:21.363 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:21.346 7f56e8ead700 1 --2- 172.21.2.102:0/3497616238 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f56e40f05f0 0x7f56e40f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:21.363 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:21.346 7f56e2ffd700 1 -- 172.21.2.102:0/3497616238 <== mon.1 v1:172.21.2.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2796657219 0 0) 0x7f56d8000ff0 con 0x7f56e41b0870 2021-02-10T11:33:21.364 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:21.346 7f56e8ead700 1 -- 172.21.2.102:0/3497616238 >> v1:172.21.2.113:6789/0 conn(0x7f56e41b0870 legacy=0x7f56e40efe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:21.364 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:21.346 7f56e8ead700 1 -- 172.21.2.102:0/3497616238 >> v1:172.21.2.102:6789/0 conn(0x7f56e41b0470 legacy=0x7f56e40ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:21.364 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:21.346 7f56e8ead700 1 -- 172.21.2.102:0/3497616238 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56e40e52e0 con 0x7f56e40f05f0 2021-02-10T11:33:21.364 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:21.347 7f56e8ead700 1 --2- 172.21.2.102:0/3497616238 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f56e40f05f0 0x7f56e40f20e0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 rx=0x7f56dc006e50 tx=0x7f56dc009e70).ready entity=mon.2 client_cookie=f93685d070403189 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:21.364 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:21.347 7f56e2ffd700 1 -- 172.21.2.102:0/3497616238 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f56dc006a20 con 0x7f56e40f05f0 2021-02-10T11:33:21.364 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:21.347 7f56e2ffd700 1 -- 172.21.2.102:0/3497616238 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56dc006790 con 0x7f56e40f05f0 2021-02-10T11:33:21.364 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:21.347 7f56e2ffd700 1 -- 172.21.2.102:0/3497616238 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f56dc00b700 con 0x7f56e40f05f0 2021-02-10T11:33:21.364 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:21.347 7f56ea890700 1 -- 172.21.2.102:0/3497616238 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f56e40f05f0 msgr2=0x7f56e40f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:21.364 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:21.347 7f56ea890700 1 --2- 172.21.2.102:0/3497616238 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f56e40f05f0 0x7f56e40f20e0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 rx=0x7f56dc006e50 tx=0x7f56dc009e70).stop 2021-02-10T11:33:21.364 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:21.348 7f56ea890700 1 -- 172.21.2.102:0/3497616238 shutdown_connections 2021-02-10T11:33:21.364 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:21.348 7f56ea890700 1 --2- 172.21.2.102:0/3497616238 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f56e40f05f0 0x7f56e40f20e0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:21.364 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:21.348 7f56ea890700 1 -- 172.21.2.102:0/3497616238 >> 172.21.2.102:0/3497616238 conn(0x7f56e40ce3e0 msgr2=0x7f56e40e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:21.364 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:21.348 7f56ea890700 1 -- 172.21.2.102:0/3497616238 shutdown_connections 2021-02-10T11:33:21.365 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:21.348 7f56ea890700 1 -- 172.21.2.102:0/3497616238 wait complete. 2021-02-10T11:33:21.366 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:21.363 7f56ea890700 1 Processor -- start 2021-02-10T11:33:21.366 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:21.363 7f56ea890700 1 -- start start 2021-02-10T11:33:21.366 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:21.363 7f56ea890700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f56e40a6550 0x7f56e418c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:21.366 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:21.363 7f56ea890700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f56e41b2200 0x7f56e41b2600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:21.367 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:21.363 7f56ea890700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f56e41b2c20 0x7f56e41b3020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:21.367 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:21.363 7f56ea890700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56e41b1c80 con 0x7f56e41b2200 2021-02-10T11:33:21.367 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:21.363 7f56ea890700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56e41b0320 con 0x7f56e40a6550 2021-02-10T11:33:21.367 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:21.363 7f56ea890700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56e41b0500 con 0x7f56e41b2c20 2021-02-10T11:33:21.367 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:21.363 7f56e37fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f56e41b2200 0x7f56e41b2600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:21.367 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:21.363 7f56e37fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f56e41b2200 0x7f56e41b2600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:35166/0 (socket says 172.21.2.102:35166) 2021-02-10T11:33:21.367 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:21.363 7f56e37fe700 1 -- 172.21.2.102:0/1970014257 learned_addr learned my addr 172.21.2.102:0/1970014257 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:33:21.367 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:21.364 7f56e37fe700 1 -- 172.21.2.102:0/1970014257 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f56e41b2c20 msgr2=0x7f56e41b3020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:21.367 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:21.364 7f56e37fe700 1 --2- 172.21.2.102:0/1970014257 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f56e41b2c20 0x7f56e41b3020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:21.367 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:21.364 7f56e37fe700 1 -- 172.21.2.102:0/1970014257 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f56e40a6550 msgr2=0x7f56e418c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:21.367 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:21.364 7f56e37fe700 1 --2- 172.21.2.102:0/1970014257 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f56e40a6550 0x7f56e418c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:21.367 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:21.364 7f56e37fe700 1 -- 172.21.2.102:0/1970014257 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56e41b0720 con 0x7f56e41b2200 2021-02-10T11:33:21.367 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:21.364 7f56e37fe700 1 --2- 172.21.2.102:0/1970014257 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f56e41b2200 0x7f56e41b2600 secure :-1 s=READY pgs=2380 cs=0 l=1 rev1=1 rx=0x7f56d800a570 tx=0x7f56d800a910).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:21.368 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:21.364 7f56e0ff9700 1 -- 172.21.2.102:0/1970014257 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f56d800c0d0 con 0x7f56e41b2200 2021-02-10T11:33:21.368 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:21.364 7f56e0ff9700 1 -- 172.21.2.102:0/1970014257 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56d800c3d0 con 0x7f56e41b2200 2021-02-10T11:33:21.368 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:21.364 7f56e0ff9700 1 -- 172.21.2.102:0/1970014257 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f56d800c820 con 0x7f56e41b2200 2021-02-10T11:33:21.368 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:21.364 7f56ea890700 1 -- 172.21.2.102:0/1970014257 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56e41b0720 con 0x7f56e41b2200 2021-02-10T11:33:21.368 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:21.364 7f56ea890700 1 -- 172.21.2.102:0/1970014257 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56e41b0cb0 con 0x7f56e41b2200 2021-02-10T11:33:21.368 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:21.365 7f56e0ff9700 1 -- 172.21.2.102:0/1970014257 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f56d8006540 con 0x7f56e41b2200 2021-02-10T11:33:21.368 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:21.365 7f56e0ff9700 1 --2- 172.21.2.102:0/1970014257 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f56c40186c0 0x7f56c401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:21.368 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:21.365 7f56e0ff9700 1 -- 172.21.2.102:0/1970014257 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f56d8032570 con 0x7f56e41b2200 2021-02-10T11:33:21.368 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:21.365 7f56e3fff700 1 --2- 172.21.2.102:0/1970014257 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f56c40186c0 0x7f56c401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:21.368 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:21.366 7f56e3fff700 1 --2- 172.21.2.102:0/1970014257 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f56c40186c0 0x7f56c401abc0 crc :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:21.395 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:21.395 7f56ea890700 1 -- 172.21.2.102:0/1970014257 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56e41b02d0 con 0x7f56e41b2200 2021-02-10T11:33:21.398 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:21.397 7f56e0ff9700 1 -- 172.21.2.102:0/1970014257 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f56d8030d20 con 0x7f56e41b2200 2021-02-10T11:33:21.527 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:21.527 7f56ea890700 1 -- 172.21.2.102:0/1970014257 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f56e41b02d0 con 0x7f56e41b2200 2021-02-10T11:33:21.528 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:21.527 7f56e0ff9700 1 -- 172.21.2.102:0/1970014257 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+2102 (secure 0 0 0) 0x7f56d8006540 con 0x7f56e41b2200 2021-02-10T11:33:21.528 INFO:teuthology.orchestra.run.gibba002.stderr:dumped fsmap epoch 100 2021-02-10T11:33:21.529 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:33:21.529 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":100,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10737,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.102:6835/4014802408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6834","nonce":4014802408},{"type":"v1","addr":"172.21.2.102:6835","nonce":4014802408}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":95}],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:32:38.017410","modified":"2021-02-10 11:32:40.027261","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7693},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7693":{"gid":7693,"name":"b","rank":0,"incarnation":96,"state":"up:active","state_seq":3,"addr":"172.21.2.113:6833/238020319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6832","nonce":238020319},{"type":"v1","addr":"172.21.2.113:6833","nonce":238020319}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[25],"metadata_pool":24,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":12}]} 2021-02-10T11:33:21.531 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:21.529 7f56ea890700 1 -- 172.21.2.102:0/1970014257 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f56c40186c0 msgr2=0x7f56c401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:21.531 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:21.529 7f56ea890700 1 --2- 172.21.2.102:0/1970014257 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f56c40186c0 0x7f56c401abc0 crc :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:21.531 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:21.529 7f56ea890700 1 -- 172.21.2.102:0/1970014257 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f56e41b2200 msgr2=0x7f56e41b2600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:21.531 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:21.529 7f56ea890700 1 --2- 172.21.2.102:0/1970014257 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f56e41b2200 0x7f56e41b2600 secure :-1 s=READY pgs=2380 cs=0 l=1 rev1=1 rx=0x7f56d800a570 tx=0x7f56d800a910).stop 2021-02-10T11:33:21.531 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:21.529 7f56ea890700 1 -- 172.21.2.102:0/1970014257 shutdown_connections 2021-02-10T11:33:21.531 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:21.529 7f56ea890700 1 --2- 172.21.2.102:0/1970014257 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f56e41b2c20 0x7f56e41b3020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:21.531 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:21.529 7f56ea890700 1 --2- 172.21.2.102:0/1970014257 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f56e41b2200 0x7f56e41b2600 unknown :-1 s=CLOSED pgs=2380 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:21.531 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:21.529 7f56ea890700 1 --2- 172.21.2.102:0/1970014257 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f56c40186c0 0x7f56c401abc0 unknown :-1 s=CLOSED pgs=1053 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:21.531 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:21.529 7f56ea890700 1 --2- 172.21.2.102:0/1970014257 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f56e40a6550 0x7f56e418c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:21.531 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:21.529 7f56ea890700 1 -- 172.21.2.102:0/1970014257 >> 172.21.2.102:0/1970014257 conn(0x7f56e41076a0 msgr2=0x7f56e4107b40 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:21.531 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:21.530 7f56ea890700 1 -- 172.21.2.102:0/1970014257 shutdown_connections 2021-02-10T11:33:21.532 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:21.530 7f56ea890700 1 -- 172.21.2.102:0/1970014257 wait complete. 2021-02-10T11:33:21.539 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs fail cephfs 2021-02-10T11:33:21.973 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:21.955 7f126781f700 1 Processor -- start 2021-02-10T11:33:21.973 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:21.955 7f126781f700 1 -- start start 2021-02-10T11:33:21.973 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:21.956 7f126781f700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f12600d1870 0x7f12601b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:21.973 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:21.956 7f126781f700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f12600f0650 0x7f12600cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:21.973 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:21.956 7f126781f700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12600f0b10 con 0x7f12601b0a10 2021-02-10T11:33:21.973 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:21.956 7f126781f700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12600e5ed0 con 0x7f12600d1870 2021-02-10T11:33:21.973 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:21.956 7f126781f700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12600d1c70 con 0x7f12600f0650 2021-02-10T11:33:21.973 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:21.956 7f1264e3a700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f12601b0a10 0x7f12600cf330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:55626/0 (socket says 172.21.2.102:55626) 2021-02-10T11:33:21.973 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:21.956 7f1264e3a700 1 -- 172.21.2.102:0/1321617714 learned_addr learned my addr 172.21.2.102:0/1321617714 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:33:21.973 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:21.956 7f126563b700 1 --2- 172.21.2.102:0/1321617714 >> v2:172.21.2.102:3300/0 conn(0x7f12600d1870 0x7f12601b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:21.973 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:21.956 7f1265e3c700 1 --2- 172.21.2.102:0/1321617714 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f12600f0650 0x7f12600cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:21.974 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:21.957 7f1257fff700 1 -- 172.21.2.102:0/1321617714 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 993618812 0 0) 0x7f12500009b0 con 0x7f12601b0a10 2021-02-10T11:33:21.974 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:21.957 7f126563b700 1 -- 172.21.2.102:0/1321617714 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f12600f0650 msgr2=0x7f12600cfb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:33:21.974 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:21.957 7f126563b700 1 --2- 172.21.2.102:0/1321617714 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f12600f0650 0x7f12600cfb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:21.974 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:21.957 7f126563b700 1 -- 172.21.2.102:0/1321617714 >> v1:172.21.2.102:6789/0 conn(0x7f12601b0a10 legacy=0x7f12600cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:21.974 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:21.957 7f126563b700 1 -- 172.21.2.102:0/1321617714 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12600d1e90 con 0x7f12600d1870 2021-02-10T11:33:21.974 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:21.957 7f1265e3c700 1 --2- 172.21.2.102:0/1321617714 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f12600f0650 0x7f12600cfb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:33:21.974 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:21.957 7f126563b700 1 --2- 172.21.2.102:0/1321617714 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f12600d1870 0x7f12601b03f0 secure :-1 s=READY pgs=2382 cs=0 l=1 rev1=1 rx=0x7f125c006650 tx=0x7f125c009a80).ready entity=mon.0 client_cookie=79e4df135a52421 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:21.974 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:21.957 7f1257fff700 1 -- 172.21.2.102:0/1321617714 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f125c006390 con 0x7f12600d1870 2021-02-10T11:33:21.974 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:21.957 7f1257fff700 1 -- 172.21.2.102:0/1321617714 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f125c0009b0 con 0x7f12600d1870 2021-02-10T11:33:21.974 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:21.957 7f1257fff700 1 -- 172.21.2.102:0/1321617714 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f125c00c2b0 con 0x7f12600d1870 2021-02-10T11:33:21.974 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:21.957 7f126781f700 1 -- 172.21.2.102:0/1321617714 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f12600d1870 msgr2=0x7f12601b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:21.974 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:21.957 7f126781f700 1 --2- 172.21.2.102:0/1321617714 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f12600d1870 0x7f12601b03f0 secure :-1 s=READY pgs=2382 cs=0 l=1 rev1=1 rx=0x7f125c006650 tx=0x7f125c009a80).stop 2021-02-10T11:33:21.975 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:21.958 7f126781f700 1 -- 172.21.2.102:0/1321617714 shutdown_connections 2021-02-10T11:33:21.975 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:21.958 7f126781f700 1 --2- 172.21.2.102:0/1321617714 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f12600f0650 0x7f12600cfb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:21.975 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:21.958 7f126781f700 1 --2- 172.21.2.102:0/1321617714 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f12600d1870 0x7f12601b03f0 unknown :-1 s=CLOSED pgs=2382 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:21.975 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:21.958 7f126781f700 1 -- 172.21.2.102:0/1321617714 >> 172.21.2.102:0/1321617714 conn(0x7f12600ce450 msgr2=0x7f12600d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:21.975 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:21.958 7f126781f700 1 -- 172.21.2.102:0/1321617714 shutdown_connections 2021-02-10T11:33:21.975 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:21.958 7f126781f700 1 -- 172.21.2.102:0/1321617714 wait complete. 2021-02-10T11:33:21.976 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:21.972 7f126781f700 1 Processor -- start 2021-02-10T11:33:21.976 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:21.973 7f126781f700 1 -- start start 2021-02-10T11:33:21.976 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:21.973 7f126781f700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f12601a06a0 0x7f126009f380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:21.977 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:21.973 7f126781f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f126009f9a0 0x7f126008b560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:21.977 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:21.973 7f126781f700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f126008bb80 0x7f126008bf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:21.977 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:21.973 7f126781f700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f126008c780 con 0x7f126009f9a0 2021-02-10T11:33:21.977 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:21.973 7f126781f700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f126009fda0 con 0x7f126008bb80 2021-02-10T11:33:21.977 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:21.973 7f126781f700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f126008cd00 con 0x7f12601a06a0 2021-02-10T11:33:21.977 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:21.973 7f1264e3a700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f126009f9a0 0x7f126008b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:21.977 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:21.973 7f1264e3a700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f126009f9a0 0x7f126008b560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:35176/0 (socket says 172.21.2.102:35176) 2021-02-10T11:33:21.977 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:21.973 7f1264e3a700 1 -- 172.21.2.102:0/3972904789 learned_addr learned my addr 172.21.2.102:0/3972904789 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:33:21.977 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:21.974 7f1264e3a700 1 -- 172.21.2.102:0/3972904789 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f12601a06a0 msgr2=0x7f126009f380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:21.977 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:21.974 7f1264e3a700 1 --2- 172.21.2.102:0/3972904789 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f12601a06a0 0x7f126009f380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:21.977 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:21.974 7f1264e3a700 1 -- 172.21.2.102:0/3972904789 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f126008bb80 msgr2=0x7f126008bf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:21.977 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:21.974 7f1264e3a700 1 --2- 172.21.2.102:0/3972904789 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f126008bb80 0x7f126008bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:21.977 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:21.974 7f1264e3a700 1 -- 172.21.2.102:0/3972904789 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f126008cf20 con 0x7f126009f9a0 2021-02-10T11:33:21.978 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:21.974 7f1264e3a700 1 --2- 172.21.2.102:0/3972904789 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f126009f9a0 0x7f126008b560 secure :-1 s=READY pgs=2383 cs=0 l=1 rev1=1 rx=0x7f1250007650 tx=0x7f125000a880).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:21.978 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:21.974 7f1255ffb700 1 -- 172.21.2.102:0/3972904789 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f125000ac70 con 0x7f126009f9a0 2021-02-10T11:33:21.978 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:21.974 7f1255ffb700 1 -- 172.21.2.102:0/3972904789 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1250004140 con 0x7f126009f9a0 2021-02-10T11:33:21.978 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:21.974 7f1255ffb700 1 -- 172.21.2.102:0/3972904789 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f12500045a0 con 0x7f126009f9a0 2021-02-10T11:33:21.978 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:21.974 7f126781f700 1 -- 172.21.2.102:0/3972904789 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f126008cf20 con 0x7f126009f9a0 2021-02-10T11:33:21.978 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:21.974 7f126781f700 1 -- 172.21.2.102:0/3972904789 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f126008cf20 con 0x7f126009f9a0 2021-02-10T11:33:21.978 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:21.975 7f1255ffb700 1 -- 172.21.2.102:0/3972904789 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1250000dd0 con 0x7f126009f9a0 2021-02-10T11:33:21.978 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:21.975 7f1255ffb700 1 --2- 172.21.2.102:0/3972904789 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1240018730 0x7f124001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:21.978 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:21.975 7f1255ffb700 1 -- 172.21.2.102:0/3972904789 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f1250032630 con 0x7f126009f9a0 2021-02-10T11:33:21.978 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:21.975 7f126563b700 1 --2- 172.21.2.102:0/3972904789 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1240018730 0x7f124001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:21.978 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:21.975 7f126563b700 1 --2- 172.21.2.102:0/3972904789 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1240018730 0x7f124001ac10 crc :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:22.006 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:22.006 7f126781f700 1 -- 172.21.2.102:0/3972904789 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12600ebc50 con 0x7f126009f9a0 2021-02-10T11:33:22.009 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:22.008 7f1255ffb700 1 -- 172.21.2.102:0/3972904789 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f1250000940 con 0x7f126009f9a0 2021-02-10T11:33:22.137 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:22.136 7f126781f700 1 -- 172.21.2.102:0/3972904789 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f12600f0b10 con 0x7f126009f9a0 2021-02-10T11:33:22.549 INFO:tasks.ceph.mon.a.gibba002.stderr:2021-02-10 11:33:22.548 7f8dd2755700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-02-10T11:33:22.555 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:22.554 7f1255ffb700 1 -- 172.21.2.102:0/3972904789 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v101) v1 ==== 161+0+0 (secure 0 0 0) 0x7f125001f4f0 con 0x7f126009f9a0 2021-02-10T11:33:22.603 INFO:teuthology.orchestra.run.gibba002.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2021-02-10T11:33:22.604 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:22.604 7f126781f700 1 -- 172.21.2.102:0/3972904789 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1240018730 msgr2=0x7f124001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:22.605 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:22.604 7f126781f700 1 --2- 172.21.2.102:0/3972904789 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1240018730 0x7f124001ac10 crc :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:22.605 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:22.604 7f126781f700 1 -- 172.21.2.102:0/3972904789 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f126009f9a0 msgr2=0x7f126008b560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:22.605 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:22.604 7f126781f700 1 --2- 172.21.2.102:0/3972904789 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f126009f9a0 0x7f126008b560 secure :-1 s=READY pgs=2383 cs=0 l=1 rev1=1 rx=0x7f1250007650 tx=0x7f125000a880).stop 2021-02-10T11:33:22.605 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:22.604 7f126781f700 1 -- 172.21.2.102:0/3972904789 shutdown_connections 2021-02-10T11:33:22.605 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:22.604 7f126781f700 1 --2- 172.21.2.102:0/3972904789 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1240018730 0x7f124001ac10 unknown :-1 s=CLOSED pgs=1054 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:22.606 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:22.604 7f126781f700 1 --2- 172.21.2.102:0/3972904789 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f126008bb80 0x7f126008bf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:22.606 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:22.604 7f126781f700 1 --2- 172.21.2.102:0/3972904789 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f126009f9a0 0x7f126008b560 unknown :-1 s=CLOSED pgs=2383 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:22.606 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:22.604 7f126781f700 1 --2- 172.21.2.102:0/3972904789 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f12601a06a0 0x7f126009f380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:22.606 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:22.604 7f126781f700 1 -- 172.21.2.102:0/3972904789 >> 172.21.2.102:0/3972904789 conn(0x7f12600b83b0 msgr2=0x7f12600a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:22.606 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:22.605 7f126781f700 1 -- 172.21.2.102:0/3972904789 shutdown_connections 2021-02-10T11:33:22.606 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:22.605 7f126781f700 1 -- 172.21.2.102:0/3972904789 wait complete. 2021-02-10T11:33:22.607 INFO:tasks.ceph.mds.b.gibba013.stdout:starting mds.b at 2021-02-10T11:33:22.614 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:33:23.042 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:23.025 7febd7d41700 1 Processor -- start 2021-02-10T11:33:23.042 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:23.025 7febd7d41700 1 -- start start 2021-02-10T11:33:23.042 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:23.026 7febd7d41700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7febd00d0340 0x7febd00ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:23.042 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:23.026 7febd7d41700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7febd00d1c70 0x7febd00efc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:23.042 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:23.026 7febd7d41700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febd00f0b10 con 0x7febd00d1870 2021-02-10T11:33:23.042 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:23.026 7febd7d41700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7febd00e5ed0 con 0x7febd00d0340 2021-02-10T11:33:23.042 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:23.026 7febd7d41700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febd01b0be0 con 0x7febd00d1c70 2021-02-10T11:33:23.042 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:23.026 7febd535c700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7febd00d0340 0x7febd00ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:23.043 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:23.026 7febd535c700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7febd00d0340 0x7febd00ef640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:35182/0 (socket says 172.21.2.102:35182) 2021-02-10T11:33:23.043 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:23.026 7febd535c700 1 -- 172.21.2.102:0/1230708113 learned_addr learned my addr 172.21.2.102:0/1230708113 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:33:23.043 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:23.026 7febd535c700 1 -- 172.21.2.102:0/1230708113 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7febd00d1c70 msgr2=0x7febd00efc60 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:33:23.043 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:23.026 7febd535c700 1 --2- 172.21.2.102:0/1230708113 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7febd00d1c70 0x7febd00efc60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:23.043 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:23.026 7febd535c700 1 -- 172.21.2.102:0/1230708113 >> v1:172.21.2.113:6789/0 conn(0x7febd00d1870 legacy=0x7febd01b03f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:33:23.043 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:23.026 7febd535c700 1 -- 172.21.2.102:0/1230708113 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febd00f20a0 con 0x7febd00d0340 2021-02-10T11:33:23.043 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:23.026 7febd535c700 1 --2- 172.21.2.102:0/1230708113 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7febd00d0340 0x7febd00ef640 secure :-1 s=READY pgs=2384 cs=0 l=1 rev1=1 rx=0x7febcc006700 tx=0x7febcc009b80).ready entity=mon.0 client_cookie=4e251c531439ffbf server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:23.043 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:23.027 7febd4b5b700 1 -- 172.21.2.102:0/1230708113 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7febcc00c0c0 con 0x7febd00d0340 2021-02-10T11:33:23.043 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:23.027 7febd4b5b700 1 -- 172.21.2.102:0/1230708113 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febcc00c440 con 0x7febd00d0340 2021-02-10T11:33:23.043 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:23.027 7febd4b5b700 1 -- 172.21.2.102:0/1230708113 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7febcc00c8a0 con 0x7febd00d0340 2021-02-10T11:33:23.043 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:23.027 7febd7d41700 1 -- 172.21.2.102:0/1230708113 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7febd00d0340 msgr2=0x7febd00ef640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:23.044 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:23.027 7febd7d41700 1 --2- 172.21.2.102:0/1230708113 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7febd00d0340 0x7febd00ef640 secure :-1 s=READY pgs=2384 cs=0 l=1 rev1=1 rx=0x7febcc006700 tx=0x7febcc009b80).stop 2021-02-10T11:33:23.044 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:23.027 7febd7d41700 1 -- 172.21.2.102:0/1230708113 shutdown_connections 2021-02-10T11:33:23.044 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:23.027 7febd7d41700 1 --2- 172.21.2.102:0/1230708113 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7febd00d1c70 0x7febd00efc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:23.044 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:23.027 7febd7d41700 1 --2- 172.21.2.102:0/1230708113 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7febd00d0340 0x7febd00ef640 unknown :-1 s=CLOSED pgs=2384 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:23.044 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:23.027 7febd7d41700 1 -- 172.21.2.102:0/1230708113 >> 172.21.2.102:0/1230708113 conn(0x7febd00ce450 msgr2=0x7febd00d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:23.044 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:23.027 7febd7d41700 1 -- 172.21.2.102:0/1230708113 shutdown_connections 2021-02-10T11:33:23.044 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:23.027 7febd7d41700 1 -- 172.21.2.102:0/1230708113 wait complete. 2021-02-10T11:33:23.046 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:23.042 7febd7d41700 1 Processor -- start 2021-02-10T11:33:23.046 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:23.042 7febd7d41700 1 -- start start 2021-02-10T11:33:23.046 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:23.042 7febd7d41700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7febd00b4f30 0x7febd018b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:23.046 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:23.042 7febd7d41700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7febd0093f40 0x7febd0094340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:23.046 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:23.042 7febd7d41700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7febd0094960 0x7febd0187e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:23.046 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:23.042 7febd7d41700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febd0094e90 con 0x7febd00b4f30 2021-02-10T11:33:23.046 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:23.042 7febd7d41700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febd018bff0 con 0x7febd0094960 2021-02-10T11:33:23.046 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:23.042 7febd7d41700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febd008f780 con 0x7febd0093f40 2021-02-10T11:33:23.046 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:23.042 7febd5b5d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7febd00b4f30 0x7febd018b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:23.046 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:23.042 7febd5b5d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7febd00b4f30 0x7febd018b9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:35192/0 (socket says 172.21.2.102:35192) 2021-02-10T11:33:23.047 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:23.042 7febd5b5d700 1 -- 172.21.2.102:0/1445110153 learned_addr learned my addr 172.21.2.102:0/1445110153 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:33:23.047 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:23.043 7febd5b5d700 1 -- 172.21.2.102:0/1445110153 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7febd0093f40 msgr2=0x7febd0094340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:23.047 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:23.043 7febd5b5d700 1 --2- 172.21.2.102:0/1445110153 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7febd0093f40 0x7febd0094340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:23.047 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:23.043 7febd5b5d700 1 -- 172.21.2.102:0/1445110153 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7febd0094960 msgr2=0x7febd0187e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:23.047 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:23.043 7febd5b5d700 1 --2- 172.21.2.102:0/1445110153 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7febd0094960 0x7febd0187e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:23.047 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:23.043 7febd5b5d700 1 -- 172.21.2.102:0/1445110153 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febd0188470 con 0x7febd00b4f30 2021-02-10T11:33:23.047 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:23.043 7febd5b5d700 1 --2- 172.21.2.102:0/1445110153 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7febd00b4f30 0x7febd018b9d0 secure :-1 s=READY pgs=2385 cs=0 l=1 rev1=1 rx=0x7febc400b5b0 tx=0x7febc400b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:23.047 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:23.043 7febc27fc700 1 -- 172.21.2.102:0/1445110153 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7febc400bd40 con 0x7febd00b4f30 2021-02-10T11:33:23.047 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:23.043 7febc27fc700 1 -- 172.21.2.102:0/1445110153 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febc4006020 con 0x7febd00b4f30 2021-02-10T11:33:23.047 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:23.043 7febd7d41700 1 -- 172.21.2.102:0/1445110153 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febd0188470 con 0x7febd00b4f30 2021-02-10T11:33:23.047 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:23.043 7febc27fc700 1 -- 172.21.2.102:0/1445110153 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7febc40054a0 con 0x7febd00b4f30 2021-02-10T11:33:23.047 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:23.043 7febd7d41700 1 -- 172.21.2.102:0/1445110153 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7febd0188470 con 0x7febd00b4f30 2021-02-10T11:33:23.047 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:23.044 7febc27fc700 1 -- 172.21.2.102:0/1445110153 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7febc4006200 con 0x7febd00b4f30 2021-02-10T11:33:23.048 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:23.044 7febc27fc700 1 --2- 172.21.2.102:0/1445110153 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7febb80186c0 0x7febb801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:23.048 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:23.044 7febc27fc700 1 -- 172.21.2.102:0/1445110153 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 5659+0+0 (secure 0 0 0) 0x7febc40327f0 con 0x7febd00b4f30 2021-02-10T11:33:23.048 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:23.044 7febd535c700 1 --2- 172.21.2.102:0/1445110153 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7febb80186c0 0x7febb801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:23.048 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:23.044 7febd535c700 1 --2- 172.21.2.102:0/1445110153 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7febb80186c0 0x7febb801abc0 crc :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:23.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:23.076 7febd7d41700 1 -- 172.21.2.102:0/1445110153 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febd00ce450 con 0x7febd00b4f30 2021-02-10T11:33:23.079 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:23.078 7febc27fc700 1 -- 172.21.2.102:0/1445110153 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7febc4006230 con 0x7febd00b4f30 2021-02-10T11:33:23.209 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:23.208 7febd7d41700 1 -- 172.21.2.102:0/1445110153 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7febd00ce450 con 0x7febd00b4f30 2021-02-10T11:33:23.209 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:23.208 7febc27fc700 1 -- 172.21.2.102:0/1445110153 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+1762 (secure 0 0 0) 0x7febc4006000 con 0x7febd00b4f30 2021-02-10T11:33:23.209 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:33:23.210 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":101,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10737,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.102:6835/4014802408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6834","nonce":4014802408},{"type":"v1","addr":"172.21.2.102:6835","nonce":4014802408}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":95}],"filesystems":[{"mdsmap":{"epoch":101,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:32:38.017410","modified":"2021-02-10 11:33:22.548658","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":121,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[25],"metadata_pool":24,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":12}]} 2021-02-10T11:33:23.210 INFO:teuthology.orchestra.run.gibba002.stderr:dumped fsmap epoch 101 2021-02-10T11:33:23.211 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:23.210 7febd7d41700 1 -- 172.21.2.102:0/1445110153 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7febb80186c0 msgr2=0x7febb801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:23.211 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:23.210 7febd7d41700 1 --2- 172.21.2.102:0/1445110153 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7febb80186c0 0x7febb801abc0 crc :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:23.211 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:23.210 7febd7d41700 1 -- 172.21.2.102:0/1445110153 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7febd00b4f30 msgr2=0x7febd018b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:23.211 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:23.210 7febd7d41700 1 --2- 172.21.2.102:0/1445110153 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7febd00b4f30 0x7febd018b9d0 secure :-1 s=READY pgs=2385 cs=0 l=1 rev1=1 rx=0x7febc400b5b0 tx=0x7febc400b950).stop 2021-02-10T11:33:23.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:23.211 7febd7d41700 1 -- 172.21.2.102:0/1445110153 shutdown_connections 2021-02-10T11:33:23.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:23.211 7febd7d41700 1 --2- 172.21.2.102:0/1445110153 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7febb80186c0 0x7febb801abc0 unknown :-1 s=CLOSED pgs=1055 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:23.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:23.211 7febd7d41700 1 --2- 172.21.2.102:0/1445110153 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7febd0094960 0x7febd0187e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:23.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:23.211 7febd7d41700 1 --2- 172.21.2.102:0/1445110153 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7febd0093f40 0x7febd0094340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:23.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:23.211 7febd7d41700 1 --2- 172.21.2.102:0/1445110153 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7febd00b4f30 0x7febd018b9d0 unknown :-1 s=CLOSED pgs=2385 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:23.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:23.211 7febd7d41700 1 -- 172.21.2.102:0/1445110153 >> 172.21.2.102:0/1445110153 conn(0x7febd019f8f0 msgr2=0x7febd01a5f80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:23.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:23.211 7febd7d41700 1 -- 172.21.2.102:0/1445110153 shutdown_connections 2021-02-10T11:33:23.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:23.211 7febd7d41700 1 -- 172.21.2.102:0/1445110153 wait complete. 2021-02-10T11:33:23.220 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2021-02-10T11:33:23.656 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:23.638 7fe57c148700 1 Processor -- start 2021-02-10T11:33:23.656 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:23.638 7fe57c148700 1 -- start start 2021-02-10T11:33:23.656 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:23.639 7fe57c148700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe5740d1870 0x7fe5741b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:23.657 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:23.639 7fe57c148700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fe5741b0a10 0x7fe5740cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:23.657 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:23.639 7fe57c148700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5740f0b10 con 0x7fe5740d1c70 2021-02-10T11:33:23.657 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:23.639 7fe57c148700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5740fe230 con 0x7fe5741b0a10 2021-02-10T11:33:23.657 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:23.639 7fe57c148700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5740e5ed0 con 0x7fe5740d1870 2021-02-10T11:33:23.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:23.639 7fe57a765700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fe5740d1c70 0x7fe5740cf950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:55652/0 (socket says 172.21.2.102:55652) 2021-02-10T11:33:23.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:23.639 7fe57a765700 1 -- 172.21.2.102:0/302884661 learned_addr learned my addr 172.21.2.102:0/302884661 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:33:23.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:23.639 7fe579763700 1 --2- 172.21.2.102:0/302884661 >> v2:172.21.2.102:3300/0 conn(0x7fe5741b0a10 0x7fe5740cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:23.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:23.639 7fe578f62700 1 -- 172.21.2.102:0/302884661 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1629997314 0 0) 0x7fe56c001500 con 0x7fe5740d1c70 2021-02-10T11:33:23.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:23.640 7fe578f62700 1 -- 172.21.2.102:0/302884661 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe560002fe0 con 0x7fe5740d1c70 2021-02-10T11:33:23.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:23.640 7fe579763700 1 -- 172.21.2.102:0/302884661 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe5740d1870 msgr2=0x7fe5741b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:33:23.659 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:23.640 7fe579763700 1 --2- 172.21.2.102:0/302884661 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe5740d1870 0x7fe5741b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:23.659 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:23.640 7fe579763700 1 -- 172.21.2.102:0/302884661 >> v1:172.21.2.102:6789/0 conn(0x7fe5740d1c70 legacy=0x7fe5740cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:23.659 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:23.640 7fe579763700 1 -- 172.21.2.102:0/302884661 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5740f0650 con 0x7fe5741b0a10 2021-02-10T11:33:23.659 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:23.640 7fe579763700 1 --2- 172.21.2.102:0/302884661 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe5741b0a10 0x7fe5740cf330 secure :-1 s=READY pgs=2387 cs=0 l=1 rev1=1 rx=0x7fe570006650 tx=0x7fe570009a80).ready entity=mon.0 client_cookie=3af3b7d341851d68 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:23.659 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:23.640 7fe578f62700 1 -- 172.21.2.102:0/302884661 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe57000c0c0 con 0x7fe5741b0a10 2021-02-10T11:33:23.659 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:23.640 7fe578f62700 1 -- 172.21.2.102:0/302884661 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe57000c340 con 0x7fe5741b0a10 2021-02-10T11:33:23.659 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:23.640 7fe578f62700 1 -- 172.21.2.102:0/302884661 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe57000c860 con 0x7fe5741b0a10 2021-02-10T11:33:23.660 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:23.640 7fe57c148700 1 -- 172.21.2.102:0/302884661 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe5741b0a10 msgr2=0x7fe5740cf330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:23.660 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:23.640 7fe57c148700 1 --2- 172.21.2.102:0/302884661 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe5741b0a10 0x7fe5740cf330 secure :-1 s=READY pgs=2387 cs=0 l=1 rev1=1 rx=0x7fe570006650 tx=0x7fe570009a80).stop 2021-02-10T11:33:23.660 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:23.640 7fe57c148700 1 -- 172.21.2.102:0/302884661 shutdown_connections 2021-02-10T11:33:23.660 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:23.640 7fe57c148700 1 --2- 172.21.2.102:0/302884661 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe5741b0a10 0x7fe5740cf330 unknown :-1 s=CLOSED pgs=2387 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:23.660 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:23.640 7fe57c148700 1 --2- 172.21.2.102:0/302884661 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe5740d1870 0x7fe5741b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:23.660 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:23.640 7fe57c148700 1 -- 172.21.2.102:0/302884661 >> 172.21.2.102:0/302884661 conn(0x7fe5740ce450 msgr2=0x7fe5740d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:23.660 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:23.641 7fe57c148700 1 -- 172.21.2.102:0/302884661 shutdown_connections 2021-02-10T11:33:23.660 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:23.641 7fe57c148700 1 -- 172.21.2.102:0/302884661 wait complete. 2021-02-10T11:33:23.660 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:23.655 7fe57c148700 1 Processor -- start 2021-02-10T11:33:23.660 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:23.655 7fe57c148700 1 -- start start 2021-02-10T11:33:23.663 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:23.656 7fe57c148700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe5741a3500 0x7fe5741a3900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:23.663 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:23.656 7fe57c148700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe57409d5d0 0x7fe57409d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:23.663 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:23.656 7fe57c148700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe5741932d0 0x7fe5741936d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:23.663 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:23.656 7fe57c148700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe57409dff0 con 0x7fe5741932d0 2021-02-10T11:33:23.663 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:23.656 7fe57c148700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe574193cf0 con 0x7fe5741a3500 2021-02-10T11:33:23.663 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:23.656 7fe57c148700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe574193f10 con 0x7fe57409d5d0 2021-02-10T11:33:23.663 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:23.656 7fe57a765700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe5741932d0 0x7fe5741936d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:23.663 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:23.656 7fe57a765700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe5741932d0 0x7fe5741936d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:35206/0 (socket says 172.21.2.102:35206) 2021-02-10T11:33:23.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:23.656 7fe57a765700 1 -- 172.21.2.102:0/85833262 learned_addr learned my addr 172.21.2.102:0/85833262 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:33:23.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:23.656 7fe57a765700 1 -- 172.21.2.102:0/85833262 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe57409d5d0 msgr2=0x7fe57409d9d0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:33:23.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:23.656 7fe57a765700 1 --2- 172.21.2.102:0/85833262 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe57409d5d0 0x7fe57409d9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:23.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:23.656 7fe57a765700 1 -- 172.21.2.102:0/85833262 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe5741a3500 msgr2=0x7fe5741a3900 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:33:23.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:23.656 7fe57a765700 1 --2- 172.21.2.102:0/85833262 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe5741a3500 0x7fe5741a3900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:23.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:23.656 7fe57a765700 1 -- 172.21.2.102:0/85833262 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe574194130 con 0x7fe5741932d0 2021-02-10T11:33:23.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:23.656 7fe57a765700 1 --2- 172.21.2.102:0/85833262 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe5741932d0 0x7fe5741936d0 secure :-1 s=READY pgs=2388 cs=0 l=1 rev1=1 rx=0x7fe56c00a8f0 tx=0x7fe56c00ac90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:23.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:23.656 7fe5667fc700 1 -- 172.21.2.102:0/85833262 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe56c001270 con 0x7fe5741932d0 2021-02-10T11:33:23.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:23.656 7fe5667fc700 1 -- 172.21.2.102:0/85833262 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe56c007160 con 0x7fe5741932d0 2021-02-10T11:33:23.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:23.656 7fe5667fc700 1 -- 172.21.2.102:0/85833262 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe56c007530 con 0x7fe5741932d0 2021-02-10T11:33:23.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:23.656 7fe57c148700 1 -- 172.21.2.102:0/85833262 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe574194130 con 0x7fe5741932d0 2021-02-10T11:33:23.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:23.657 7fe57c148700 1 -- 172.21.2.102:0/85833262 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe574194130 con 0x7fe5741932d0 2021-02-10T11:33:23.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:23.657 7fe5667fc700 1 -- 172.21.2.102:0/85833262 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fe56c000d30 con 0x7fe5741932d0 2021-02-10T11:33:23.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:23.657 7fe5667fc700 1 --2- 172.21.2.102:0/85833262 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe55c018730 0x7fe55c01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:23.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:23.657 7fe5667fc700 1 -- 172.21.2.102:0/85833262 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 5659+0+0 (secure 0 0 0) 0x7fe56c032bc0 con 0x7fe5741932d0 2021-02-10T11:33:23.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:23.657 7fe579f64700 1 --2- 172.21.2.102:0/85833262 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe55c018730 0x7fe55c01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:23.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:23.658 7fe579f64700 1 --2- 172.21.2.102:0/85833262 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe55c018730 0x7fe55c01ac10 crc :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:23.689 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:23.688 7fe57c148700 1 -- 172.21.2.102:0/85833262 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5740f0b10 con 0x7fe5741932d0 2021-02-10T11:33:23.692 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:23.691 7fe5667fc700 1 -- 172.21.2.102:0/85833262 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fe56c007d10 con 0x7fe5741932d0 2021-02-10T11:33:23.818 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:23.817 7fe57c148700 1 -- 172.21.2.102:0/85833262 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fe5740f0b10 con 0x7fe5741932d0 2021-02-10T11:33:24.562 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:24.561 7fe5667fc700 1 -- 172.21.2.102:0/85833262 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v102) v1 ==== 104+0+0 (secure 0 0 0) 0x7fe56c000d30 con 0x7fe5741932d0 2021-02-10T11:33:24.590 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:24.589 7fe57c148700 1 -- 172.21.2.102:0/85833262 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe55c018730 msgr2=0x7fe55c01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:24.591 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:24.589 7fe57c148700 1 --2- 172.21.2.102:0/85833262 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe55c018730 0x7fe55c01ac10 crc :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:24.591 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:24.589 7fe57c148700 1 -- 172.21.2.102:0/85833262 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe5741932d0 msgr2=0x7fe5741936d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:24.591 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:24.589 7fe57c148700 1 --2- 172.21.2.102:0/85833262 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe5741932d0 0x7fe5741936d0 secure :-1 s=READY pgs=2388 cs=0 l=1 rev1=1 rx=0x7fe56c00a8f0 tx=0x7fe56c00ac90).stop 2021-02-10T11:33:24.591 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:24.589 7fe57c148700 1 -- 172.21.2.102:0/85833262 shutdown_connections 2021-02-10T11:33:24.592 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:24.589 7fe57c148700 1 --2- 172.21.2.102:0/85833262 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe5741932d0 0x7fe5741936d0 unknown :-1 s=CLOSED pgs=2388 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:24.592 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:24.589 7fe57c148700 1 --2- 172.21.2.102:0/85833262 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe57409d5d0 0x7fe57409d9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:24.592 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:24.589 7fe57c148700 1 --2- 172.21.2.102:0/85833262 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe55c018730 0x7fe55c01ac10 unknown :-1 s=CLOSED pgs=1056 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:24.592 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:24.589 7fe57c148700 1 --2- 172.21.2.102:0/85833262 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe5741a3500 0x7fe5741a3900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:24.592 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:24.589 7fe57c148700 1 -- 172.21.2.102:0/85833262 >> 172.21.2.102:0/85833262 conn(0x7fe5740ed860 msgr2=0x7fe5740bf0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:24.593 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:24.589 7fe57c148700 1 -- 172.21.2.102:0/85833262 shutdown_connections 2021-02-10T11:33:24.593 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:24.589 7fe57c148700 1 -- 172.21.2.102:0/85833262 wait complete. 2021-02-10T11:33:24.598 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool delete cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2021-02-10T11:33:25.030 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:25.013 7f14c132c700 1 Processor -- start 2021-02-10T11:33:25.030 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:25.013 7f14c132c700 1 -- start start 2021-02-10T11:33:25.030 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:25.013 7f14c132c700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f14bc0d1880 0x7f14bc1b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:25.030 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:25.013 7f14c132c700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f14bc1b0af0 0x7f14bc108950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:25.030 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:25.013 7f14c132c700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14bc0fe1d0 con 0x7f14bc0d1c80 2021-02-10T11:33:25.030 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:25.013 7f14c132c700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14bc109a70 con 0x7f14bc0d1880 2021-02-10T11:33:25.030 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:25.013 7f14c132c700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14bc109760 con 0x7f14bc1b0af0 2021-02-10T11:33:25.030 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:25.013 7f14bb87e700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f14bc0d1c80 0x7f14bc108f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:55666/0 (socket says 172.21.2.102:55666) 2021-02-10T11:33:25.030 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:25.013 7f14bb87e700 1 -- 172.21.2.102:0/1731111040 learned_addr learned my addr 172.21.2.102:0/1731111040 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:33:25.031 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:25.013 7f14bb07d700 1 --2- 172.21.2.102:0/1731111040 >> v2:172.21.2.102:3300/0 conn(0x7f14bc0d1880 0x7f14bc1b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:25.031 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:25.014 7f14ba07b700 1 -- 172.21.2.102:0/1731111040 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1499908026 0 0) 0x7f14b4001500 con 0x7f14bc0d1c80 2021-02-10T11:33:25.031 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:25.014 7f14ba07b700 1 -- 172.21.2.102:0/1731111040 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14a8002fe0 con 0x7f14bc0d1c80 2021-02-10T11:33:25.031 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:25.014 7f14bb07d700 1 -- 172.21.2.102:0/1731111040 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f14bc1b0af0 msgr2=0x7f14bc108950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:33:25.031 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:25.014 7f14bb07d700 1 --2- 172.21.2.102:0/1731111040 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f14bc1b0af0 0x7f14bc108950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:25.031 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:25.014 7f14bb07d700 1 -- 172.21.2.102:0/1731111040 >> v1:172.21.2.102:6789/0 conn(0x7f14bc0d1c80 legacy=0x7f14bc108f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:25.031 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:25.014 7f14bb07d700 1 -- 172.21.2.102:0/1731111040 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14bc088070 con 0x7f14bc0d1880 2021-02-10T11:33:25.031 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:25.014 7f14bb07d700 1 --2- 172.21.2.102:0/1731111040 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f14bc0d1880 0x7f14bc1b04d0 secure :-1 s=READY pgs=2390 cs=0 l=1 rev1=1 rx=0x7f14ac006650 tx=0x7f14ac009a80).ready entity=mon.0 client_cookie=e5b2e86a72a96c09 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:25.031 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:25.014 7f14ba07b700 1 -- 172.21.2.102:0/1731111040 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f14ac00c0c0 con 0x7f14bc0d1880 2021-02-10T11:33:25.031 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:25.014 7f14ba07b700 1 -- 172.21.2.102:0/1731111040 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14ac00c340 con 0x7f14bc0d1880 2021-02-10T11:33:25.031 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:25.014 7f14ba07b700 1 -- 172.21.2.102:0/1731111040 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f14ac00c860 con 0x7f14bc0d1880 2021-02-10T11:33:25.031 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:25.014 7f14c132c700 1 -- 172.21.2.102:0/1731111040 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f14bc0d1880 msgr2=0x7f14bc1b04d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:25.031 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:25.014 7f14c132c700 1 --2- 172.21.2.102:0/1731111040 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f14bc0d1880 0x7f14bc1b04d0 secure :-1 s=READY pgs=2390 cs=0 l=1 rev1=1 rx=0x7f14ac006650 tx=0x7f14ac009a80).stop 2021-02-10T11:33:25.032 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:25.015 7f14c132c700 1 -- 172.21.2.102:0/1731111040 shutdown_connections 2021-02-10T11:33:25.032 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:25.015 7f14c132c700 1 --2- 172.21.2.102:0/1731111040 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f14bc1b0af0 0x7f14bc108950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:25.032 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:25.015 7f14c132c700 1 --2- 172.21.2.102:0/1731111040 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f14bc0d1880 0x7f14bc1b04d0 unknown :-1 s=CLOSED pgs=2390 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:25.032 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:25.015 7f14c132c700 1 -- 172.21.2.102:0/1731111040 >> 172.21.2.102:0/1731111040 conn(0x7f14bc0ce430 msgr2=0x7f14bc0d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:25.032 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:25.015 7f14c132c700 1 -- 172.21.2.102:0/1731111040 shutdown_connections 2021-02-10T11:33:25.032 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:25.015 7f14c132c700 1 -- 172.21.2.102:0/1731111040 wait complete. 2021-02-10T11:33:25.032 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:25.029 7f14c132c700 1 Processor -- start 2021-02-10T11:33:25.034 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:25.030 7f14c132c700 1 -- start start 2021-02-10T11:33:25.034 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:25.030 7f14c132c700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f14bc0ad520 0x7f14bc0ad920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:25.034 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:25.030 7f14c132c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f14bc0adf40 0x7f14bc097c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:25.034 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:25.030 7f14c132c700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f14bc098250 0x7f14bc098650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:25.034 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:25.030 7f14c132c700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14bc098c70 con 0x7f14bc0adf40 2021-02-10T11:33:25.034 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:25.030 7f14c132c700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14bc0910f0 con 0x7f14bc0ad520 2021-02-10T11:33:25.034 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:25.030 7f14c132c700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14bc091310 con 0x7f14bc098250 2021-02-10T11:33:25.034 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:25.030 7f14ba87c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f14bc0adf40 0x7f14bc097c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:25.034 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:25.030 7f14ba87c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f14bc0adf40 0x7f14bc097c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:35218/0 (socket says 172.21.2.102:35218) 2021-02-10T11:33:25.034 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:25.030 7f14ba87c700 1 -- 172.21.2.102:0/2939846484 learned_addr learned my addr 172.21.2.102:0/2939846484 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:33:25.034 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:25.030 7f14ba87c700 1 -- 172.21.2.102:0/2939846484 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f14bc098250 msgr2=0x7f14bc098650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:25.034 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:25.031 7f14ba87c700 1 --2- 172.21.2.102:0/2939846484 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f14bc098250 0x7f14bc098650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:25.035 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:25.031 7f14ba87c700 1 -- 172.21.2.102:0/2939846484 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f14bc0ad520 msgr2=0x7f14bc0ad920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:25.035 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:25.031 7f14bb07d700 1 --2- 172.21.2.102:0/2939846484 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f14bc0ad520 0x7f14bc0ad920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:25.035 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:25.031 7f14ba87c700 1 --2- 172.21.2.102:0/2939846484 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f14bc0ad520 0x7f14bc0ad920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:25.035 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:25.031 7f14ba87c700 1 -- 172.21.2.102:0/2939846484 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14bc19a3b0 con 0x7f14bc0adf40 2021-02-10T11:33:25.035 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:25.031 7f14bb07d700 1 --2- 172.21.2.102:0/2939846484 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f14bc0ad520 0x7f14bc0ad920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:33:25.035 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:25.031 7f14ba87c700 1 --2- 172.21.2.102:0/2939846484 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f14bc0adf40 0x7f14bc097c30 secure :-1 s=READY pgs=2391 cs=0 l=1 rev1=1 rx=0x7f14a4006f00 tx=0x7f14a400b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:25.035 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:25.031 7f14b3fff700 1 -- 172.21.2.102:0/2939846484 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f14a400bd30 con 0x7f14bc0adf40 2021-02-10T11:33:25.035 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:25.031 7f14b3fff700 1 -- 172.21.2.102:0/2939846484 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14a4000c90 con 0x7f14bc0adf40 2021-02-10T11:33:25.035 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:25.031 7f14b3fff700 1 -- 172.21.2.102:0/2939846484 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f14a40022a0 con 0x7f14bc0adf40 2021-02-10T11:33:25.035 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:25.031 7f14c132c700 1 -- 172.21.2.102:0/2939846484 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14bc19a3b0 con 0x7f14bc0adf40 2021-02-10T11:33:25.035 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:25.031 7f14c132c700 1 -- 172.21.2.102:0/2939846484 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14bc19a3b0 con 0x7f14bc0adf40 2021-02-10T11:33:25.035 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:25.032 7f14b3fff700 1 -- 172.21.2.102:0/2939846484 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f14a4005310 con 0x7f14bc0adf40 2021-02-10T11:33:25.035 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:25.032 7f14b3fff700 1 --2- 172.21.2.102:0/2939846484 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f14a00186c0 0x7f14a001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:25.035 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:25.032 7f14b3fff700 1 -- 172.21.2.102:0/2939846484 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 5659+0+0 (secure 0 0 0) 0x7f14a4032400 con 0x7f14bc0adf40 2021-02-10T11:33:25.036 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:25.032 7f14bb07d700 1 --2- 172.21.2.102:0/2939846484 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f14a00186c0 0x7f14a001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:25.036 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:25.032 7f14bb07d700 1 --2- 172.21.2.102:0/2939846484 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f14a00186c0 0x7f14a001abc0 crc :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:25.064 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:25.063 7f14c132c700 1 -- 172.21.2.102:0/2939846484 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14bc098c70 con 0x7f14bc0adf40 2021-02-10T11:33:25.066 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:25.066 7f14b3fff700 1 -- 172.21.2.102:0/2939846484 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f14a4006b90 con 0x7f14bc0adf40 2021-02-10T11:33:25.193 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:25.193 7f14c132c700 1 -- 172.21.2.102:0/2939846484 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"pool2": "cephfs_metadata", "prefix": "osd pool delete", "pool": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f14bc098c70 con 0x7f14bc0adf40 2021-02-10T11:33:25.565 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:25.564 7f14b3fff700 1 -- 172.21.2.102:0/2939846484 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"pool2": "cephfs_metadata", "prefix": "osd pool delete", "pool": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v122) v1 ==== 185+0+0 (secure 0 0 0) 0x7f14a4027ea0 con 0x7f14bc0adf40 2021-02-10T11:33:25.610 INFO:teuthology.orchestra.run.gibba002.stderr:pool 'cephfs_metadata' removed 2021-02-10T11:33:25.612 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:25.611 7f14c132c700 1 -- 172.21.2.102:0/2939846484 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f14a00186c0 msgr2=0x7f14a001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:25.612 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:25.611 7f14c132c700 1 --2- 172.21.2.102:0/2939846484 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f14a00186c0 0x7f14a001abc0 crc :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:25.613 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:25.612 7f14c132c700 1 -- 172.21.2.102:0/2939846484 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f14bc0adf40 msgr2=0x7f14bc097c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:25.613 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:25.612 7f14c132c700 1 --2- 172.21.2.102:0/2939846484 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f14bc0adf40 0x7f14bc097c30 secure :-1 s=READY pgs=2391 cs=0 l=1 rev1=1 rx=0x7f14a4006f00 tx=0x7f14a400b940).stop 2021-02-10T11:33:25.613 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:25.612 7f14c132c700 1 -- 172.21.2.102:0/2939846484 shutdown_connections 2021-02-10T11:33:25.613 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:25.612 7f14c132c700 1 --2- 172.21.2.102:0/2939846484 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f14bc098250 0x7f14bc098650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:25.613 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:25.612 7f14c132c700 1 --2- 172.21.2.102:0/2939846484 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f14bc0adf40 0x7f14bc097c30 unknown :-1 s=CLOSED pgs=2391 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:25.613 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:25.612 7f14c132c700 1 --2- 172.21.2.102:0/2939846484 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f14a00186c0 0x7f14a001abc0 unknown :-1 s=CLOSED pgs=1057 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:25.613 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:25.612 7f14c132c700 1 --2- 172.21.2.102:0/2939846484 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f14bc0ad520 0x7f14bc0ad920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:25.613 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:25.612 7f14c132c700 1 -- 172.21.2.102:0/2939846484 >> 172.21.2.102:0/2939846484 conn(0x7f14bc18e810 msgr2=0x7f14bc18c290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:25.613 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:25.612 7f14c132c700 1 -- 172.21.2.102:0/2939846484 shutdown_connections 2021-02-10T11:33:25.613 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:25.612 7f14c132c700 1 -- 172.21.2.102:0/2939846484 wait complete. 2021-02-10T11:33:25.620 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool delete cephfs_data cephfs_data --yes-i-really-really-mean-it 2021-02-10T11:33:26.053 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:26.036 7fb98fe8f700 1 Processor -- start 2021-02-10T11:33:26.053 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:26.036 7fb98fe8f700 1 -- start start 2021-02-10T11:33:26.053 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:26.037 7fb98fe8f700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fb9880d1880 0x7fb9881b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:26.054 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:26.037 7fb98fe8f700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fb9880cff80 0x7fb988109140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:26.054 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:26.037 7fb98fe8f700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9880fe1d0 con 0x7fb9881b0af0 2021-02-10T11:33:26.054 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:26.037 7fb98fe8f700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb988109ae0 con 0x7fb9880d1880 2021-02-10T11:33:26.054 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:26.037 7fb98fe8f700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb988109760 con 0x7fb9880cff80 2021-02-10T11:33:26.054 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:26.037 7fb98d4aa700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fb9881b0af0 0x7fb988108950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:55680/0 (socket says 172.21.2.102:55680) 2021-02-10T11:33:26.054 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:26.037 7fb98d4aa700 1 -- 172.21.2.102:0/955873388 learned_addr learned my addr 172.21.2.102:0/955873388 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:33:26.054 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:26.037 7fb98dcab700 1 --2- 172.21.2.102:0/955873388 >> v2:172.21.2.102:3300/0 conn(0x7fb9880d1880 0x7fb9881b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:26.054 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:26.037 7fb98dcab700 1 -- 172.21.2.102:0/955873388 >> v2:172.21.2.113:3300/0 conn(0x7fb9880cff80 msgr2=0x7fb988109140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:33:26.054 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:26.037 7fb98dcab700 1 --2- 172.21.2.102:0/955873388 >> v2:172.21.2.113:3300/0 conn(0x7fb9880cff80 0x7fb988109140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:26.054 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:26.037 7fb98dcab700 1 -- 172.21.2.102:0/955873388 >> v1:172.21.2.102:6789/0 conn(0x7fb9881b0af0 legacy=0x7fb988108950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:26.054 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:26.037 7fb98dcab700 1 -- 172.21.2.102:0/955873388 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb988088070 con 0x7fb9880d1880 2021-02-10T11:33:26.054 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:26.037 7fb98dcab700 1 --2- 172.21.2.102:0/955873388 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb9880d1880 0x7fb9881b04d0 secure :-1 s=READY pgs=2393 cs=0 l=1 rev1=1 rx=0x7fb97c006650 tx=0x7fb97c009a80).ready entity=mon.0 client_cookie=c616fa030497298c server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:26.054 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:26.038 7fb98cca9700 1 -- 172.21.2.102:0/955873388 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb97c00c0c0 con 0x7fb9880d1880 2021-02-10T11:33:26.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:26.038 7fb98cca9700 1 -- 172.21.2.102:0/955873388 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb97c00c340 con 0x7fb9880d1880 2021-02-10T11:33:26.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:26.038 7fb98cca9700 1 -- 172.21.2.102:0/955873388 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb97c00c860 con 0x7fb9880d1880 2021-02-10T11:33:26.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:26.038 7fb98fe8f700 1 -- 172.21.2.102:0/955873388 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb9880d1880 msgr2=0x7fb9881b04d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:26.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:26.038 7fb98fe8f700 1 --2- 172.21.2.102:0/955873388 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb9880d1880 0x7fb9881b04d0 secure :-1 s=READY pgs=2393 cs=0 l=1 rev1=1 rx=0x7fb97c006650 tx=0x7fb97c009a80).stop 2021-02-10T11:33:26.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:26.038 7fb98fe8f700 1 -- 172.21.2.102:0/955873388 shutdown_connections 2021-02-10T11:33:26.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:26.038 7fb98fe8f700 1 --2- 172.21.2.102:0/955873388 >> v2:172.21.2.113:3300/0 conn(0x7fb9880cff80 0x7fb988109140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:26.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:26.038 7fb98fe8f700 1 --2- 172.21.2.102:0/955873388 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb9880d1880 0x7fb9881b04d0 unknown :-1 s=CLOSED pgs=2393 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:26.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:26.038 7fb98fe8f700 1 -- 172.21.2.102:0/955873388 >> 172.21.2.102:0/955873388 conn(0x7fb9880ce430 msgr2=0x7fb9880d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:26.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:26.038 7fb98fe8f700 1 -- 172.21.2.102:0/955873388 shutdown_connections 2021-02-10T11:33:26.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:26.038 7fb98fe8f700 1 -- 172.21.2.102:0/955873388 wait complete. 2021-02-10T11:33:26.056 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:26.053 7fb98fe8f700 1 Processor -- start 2021-02-10T11:33:26.057 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:26.053 7fb98fe8f700 1 -- start start 2021-02-10T11:33:26.057 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:26.053 7fb98fe8f700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb98819e5b0 0x7fb9881afc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:26.057 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:26.053 7fb98fe8f700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb98819e9b0 0x7fb9881b0dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:26.057 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:26.053 7fb98fe8f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb9881b0250 0x7fb9881b13e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:26.057 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:26.053 7fb98fe8f700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb988195bd0 con 0x7fb9881b0250 2021-02-10T11:33:26.057 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:26.053 7fb98fe8f700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9881b0840 con 0x7fb98819e5b0 2021-02-10T11:33:26.057 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:26.053 7fb98fe8f700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb98819eed0 con 0x7fb98819e9b0 2021-02-10T11:33:26.057 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:26.054 7fb98e4ac700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb9881b0250 0x7fb9881b13e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:26.057 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:26.054 7fb98e4ac700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb9881b0250 0x7fb9881b13e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:35232/0 (socket says 172.21.2.102:35232) 2021-02-10T11:33:26.057 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:26.054 7fb98e4ac700 1 -- 172.21.2.102:0/1110141904 learned_addr learned my addr 172.21.2.102:0/1110141904 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:33:26.057 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:26.054 7fb98e4ac700 1 -- 172.21.2.102:0/1110141904 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb98819e9b0 msgr2=0x7fb9881b0dc0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:33:26.057 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:26.054 7fb98e4ac700 1 --2- 172.21.2.102:0/1110141904 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb98819e9b0 0x7fb9881b0dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:26.057 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:26.054 7fb98e4ac700 1 -- 172.21.2.102:0/1110141904 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb98819e5b0 msgr2=0x7fb9881afc30 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:33:26.057 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:26.054 7fb98e4ac700 1 --2- 172.21.2.102:0/1110141904 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb98819e5b0 0x7fb9881afc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:26.058 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:26.054 7fb98e4ac700 1 -- 172.21.2.102:0/1110141904 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb98819f0b0 con 0x7fb9881b0250 2021-02-10T11:33:26.058 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:26.054 7fb98e4ac700 1 --2- 172.21.2.102:0/1110141904 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb9881b0250 0x7fb9881b13e0 secure :-1 s=READY pgs=2394 cs=0 l=1 rev1=1 rx=0x7fb980009980 tx=0x7fb98000bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:26.058 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:26.054 7fb97a7fc700 1 -- 172.21.2.102:0/1110141904 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb980003020 con 0x7fb9881b0250 2021-02-10T11:33:26.058 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:26.054 7fb97a7fc700 1 -- 172.21.2.102:0/1110141904 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb980005a80 con 0x7fb9881b0250 2021-02-10T11:33:26.058 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:26.054 7fb97a7fc700 1 -- 172.21.2.102:0/1110141904 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb980006030 con 0x7fb9881b0250 2021-02-10T11:33:26.058 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:26.054 7fb98fe8f700 1 -- 172.21.2.102:0/1110141904 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb98819f0b0 con 0x7fb9881b0250 2021-02-10T11:33:26.058 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:26.054 7fb98fe8f700 1 -- 172.21.2.102:0/1110141904 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9881b1cf0 con 0x7fb9881b0250 2021-02-10T11:33:26.058 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:26.055 7fb97a7fc700 1 -- 172.21.2.102:0/1110141904 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fb980006030 con 0x7fb9881b0250 2021-02-10T11:33:26.058 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:26.055 7fb97a7fc700 1 --2- 172.21.2.102:0/1110141904 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb9700186c0 0x7fb97001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:26.058 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:26.055 7fb97a7fc700 1 -- 172.21.2.102:0/1110141904 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 5221+0+0 (secure 0 0 0) 0x7fb9800327c0 con 0x7fb9881b0250 2021-02-10T11:33:26.058 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:26.055 7fb98dcab700 1 --2- 172.21.2.102:0/1110141904 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb9700186c0 0x7fb97001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:26.058 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:26.055 7fb98dcab700 1 --2- 172.21.2.102:0/1110141904 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb9700186c0 0x7fb97001abc0 crc :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:26.087 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:26.086 7fb98fe8f700 1 -- 172.21.2.102:0/1110141904 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9881b2210 con 0x7fb9881b0250 2021-02-10T11:33:26.089 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:26.089 7fb97a7fc700 1 -- 172.21.2.102:0/1110141904 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fb98002df90 con 0x7fb9881b0250 2021-02-10T11:33:26.214 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:26.214 7fb98fe8f700 1 -- 172.21.2.102:0/1110141904 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"pool2": "cephfs_data", "prefix": "osd pool delete", "pool": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fb9881b2210 con 0x7fb9881b0250 2021-02-10T11:33:26.570 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:26.570 7fb97a7fc700 1 -- 172.21.2.102:0/1110141904 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"pool2": "cephfs_data", "prefix": "osd pool delete", "pool": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v123) v1 ==== 173+0+0 (secure 0 0 0) 0x7fb98002df90 con 0x7fb9881b0250 2021-02-10T11:33:26.579 INFO:teuthology.orchestra.run.gibba002.stderr:pool 'cephfs_data' removed 2021-02-10T11:33:26.581 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:26.580 7fb98fe8f700 1 -- 172.21.2.102:0/1110141904 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb9700186c0 msgr2=0x7fb97001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:26.582 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:26.580 7fb98fe8f700 1 --2- 172.21.2.102:0/1110141904 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb9700186c0 0x7fb97001abc0 crc :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:26.582 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:26.581 7fb98fe8f700 1 -- 172.21.2.102:0/1110141904 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb9881b0250 msgr2=0x7fb9881b13e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:26.582 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:26.581 7fb98fe8f700 1 --2- 172.21.2.102:0/1110141904 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb9881b0250 0x7fb9881b13e0 secure :-1 s=READY pgs=2394 cs=0 l=1 rev1=1 rx=0x7fb980009980 tx=0x7fb98000bcd0).stop 2021-02-10T11:33:26.582 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:26.581 7fb98fe8f700 1 -- 172.21.2.102:0/1110141904 shutdown_connections 2021-02-10T11:33:26.582 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:26.581 7fb98fe8f700 1 --2- 172.21.2.102:0/1110141904 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb9881b0250 0x7fb9881b13e0 unknown :-1 s=CLOSED pgs=2394 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:26.582 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:26.581 7fb98fe8f700 1 --2- 172.21.2.102:0/1110141904 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb98819e9b0 0x7fb9881b0dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:26.582 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:26.581 7fb98fe8f700 1 --2- 172.21.2.102:0/1110141904 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb9700186c0 0x7fb97001abc0 unknown :-1 s=CLOSED pgs=1058 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:26.582 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:26.581 7fb98fe8f700 1 --2- 172.21.2.102:0/1110141904 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb98819e5b0 0x7fb9881afc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:26.582 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:26.581 7fb98fe8f700 1 -- 172.21.2.102:0/1110141904 >> 172.21.2.102:0/1110141904 conn(0x7fb98819f940 msgr2=0x7fb98818c290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:26.582 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:26.581 7fb98fe8f700 1 -- 172.21.2.102:0/1110141904 shutdown_connections 2021-02-10T11:33:26.582 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:26.581 7fb98fe8f700 1 -- 172.21.2.102:0/1110141904 wait complete. 2021-02-10T11:33:26.590 INFO:tasks.ceph.mds.a:Restarting daemon 2021-02-10T11:33:26.590 INFO:tasks.ceph.mds.a:Stopping old one... 2021-02-10T11:33:26.590 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2021-02-10T11:33:26.590 INFO:teuthology.orchestra.run:waiting for 300 2021-02-10T11:33:26.590 INFO:tasks.ceph.mds.b:Restarting daemon 2021-02-10T11:33:26.590 INFO:tasks.ceph.mds.b:Stopping old one... 2021-02-10T11:33:26.591 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2021-02-10T11:33:26.591 INFO:teuthology.orchestra.run:waiting for 300 2021-02-10T11:33:26.596 INFO:tasks.ceph.mds.a:Stopped 2021-02-10T11:33:26.767 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2021-02-10T11:33:26.768 INFO:tasks.ceph.mds.b:Stopped 2021-02-10T11:33:26.768 DEBUG:teuthology.orchestra.run.gibba013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2021-02-10T11:33:26.788 INFO:tasks.ceph.mds.a:Started 2021-02-10T11:33:26.789 DEBUG:teuthology.parallel:result is None 2021-02-10T11:33:26.829 INFO:tasks.ceph.mds.b:Started 2021-02-10T11:33:26.830 DEBUG:teuthology.parallel:result is None 2021-02-10T11:33:26.830 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd blacklist clear 2021-02-10T11:33:27.194 INFO:tasks.ceph.mds.a.gibba002.stdout:starting mds.a at 2021-02-10T11:33:27.214 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:27.197 7fc187916700 1 Processor -- start 2021-02-10T11:33:27.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:27.197 7fc187916700 1 -- start start 2021-02-10T11:33:27.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:27.197 7fc187916700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fc1800d0340 0x7fc1800ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:27.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:27.197 7fc187916700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fc1800d1c70 0x7fc1800efc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:27.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:27.197 7fc187916700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1800f0b10 con 0x7fc1800d1870 2021-02-10T11:33:27.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:27.197 7fc187916700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1800e5ed0 con 0x7fc1800d0340 2021-02-10T11:33:27.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:27.197 7fc187916700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1800d2fe0 con 0x7fc1800d1c70 2021-02-10T11:33:27.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:27.197 7fc184f31700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fc1800d0340 0x7fc1800ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:27.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:27.198 7fc184f31700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fc1800d0340 0x7fc1800ef640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:35252/0 (socket says 172.21.2.102:35252) 2021-02-10T11:33:27.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:27.198 7fc184f31700 1 -- 172.21.2.102:0/2966259552 learned_addr learned my addr 172.21.2.102:0/2966259552 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:33:27.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:27.198 7fc185f33700 1 --2- 172.21.2.102:0/2966259552 >> v2:172.21.2.113:3300/0 conn(0x7fc1800d1c70 0x7fc1800efc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:27.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:27.198 7fc184f31700 1 -- 172.21.2.102:0/2966259552 >> v2:172.21.2.113:3300/0 conn(0x7fc1800d1c70 msgr2=0x7fc1800efc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:33:27.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:27.198 7fc184f31700 1 --2- 172.21.2.102:0/2966259552 >> v2:172.21.2.113:3300/0 conn(0x7fc1800d1c70 0x7fc1800efc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:27.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:27.198 7fc184f31700 1 -- 172.21.2.102:0/2966259552 >> v1:172.21.2.113:6789/0 conn(0x7fc1800d1870 legacy=0x7fc1801b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:33:27.216 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:27.198 7fc184f31700 1 -- 172.21.2.102:0/2966259552 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1801b0be0 con 0x7fc1800d0340 2021-02-10T11:33:27.216 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:27.198 7fc184f31700 1 --2- 172.21.2.102:0/2966259552 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc1800d0340 0x7fc1800ef640 secure :-1 s=READY pgs=2397 cs=0 l=1 rev1=1 rx=0x7fc170006700 tx=0x7fc170009b00).ready entity=mon.0 client_cookie=3b0371e9d24a6b79 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:27.216 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:27.198 7fc177fff700 1 -- 172.21.2.102:0/2966259552 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc17000c0c0 con 0x7fc1800d0340 2021-02-10T11:33:27.216 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:27.198 7fc177fff700 1 -- 172.21.2.102:0/2966259552 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc17000c340 con 0x7fc1800d0340 2021-02-10T11:33:27.216 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:27.198 7fc177fff700 1 -- 172.21.2.102:0/2966259552 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc17000c860 con 0x7fc1800d0340 2021-02-10T11:33:27.216 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:27.198 7fc187916700 1 -- 172.21.2.102:0/2966259552 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc1800d0340 msgr2=0x7fc1800ef640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:27.216 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:27.198 7fc187916700 1 --2- 172.21.2.102:0/2966259552 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc1800d0340 0x7fc1800ef640 secure :-1 s=READY pgs=2397 cs=0 l=1 rev1=1 rx=0x7fc170006700 tx=0x7fc170009b00).stop 2021-02-10T11:33:27.216 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:27.199 7fc187916700 1 -- 172.21.2.102:0/2966259552 shutdown_connections 2021-02-10T11:33:27.216 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:27.199 7fc187916700 1 --2- 172.21.2.102:0/2966259552 >> v2:172.21.2.113:3300/0 conn(0x7fc1800d1c70 0x7fc1800efc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:27.216 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:27.199 7fc187916700 1 --2- 172.21.2.102:0/2966259552 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc1800d0340 0x7fc1800ef640 unknown :-1 s=CLOSED pgs=2397 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:27.216 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:27.199 7fc187916700 1 -- 172.21.2.102:0/2966259552 >> 172.21.2.102:0/2966259552 conn(0x7fc1800ce450 msgr2=0x7fc1800d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:27.216 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:27.199 7fc187916700 1 -- 172.21.2.102:0/2966259552 shutdown_connections 2021-02-10T11:33:27.216 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:27.199 7fc187916700 1 -- 172.21.2.102:0/2966259552 wait complete. 2021-02-10T11:33:27.216 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:27.214 7fc187916700 1 Processor -- start 2021-02-10T11:33:27.217 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:27.214 7fc187916700 1 -- start start 2021-02-10T11:33:27.218 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:27.214 7fc187916700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc1801a06a0 0x7fc18009f380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:27.218 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:27.214 7fc187916700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc18009f9a0 0x7fc18008b560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:27.218 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:27.214 7fc187916700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc18008bb80 0x7fc18008bf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:27.218 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:27.214 7fc187916700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc18008c780 con 0x7fc1801a06a0 2021-02-10T11:33:27.218 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:27.214 7fc187916700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc18009fda0 con 0x7fc18009f9a0 2021-02-10T11:33:27.218 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:27.214 7fc187916700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc18008cd00 con 0x7fc18008bb80 2021-02-10T11:33:27.218 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:27.215 7fc185732700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc1801a06a0 0x7fc18009f380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:27.218 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:27.215 7fc185732700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc1801a06a0 0x7fc18009f380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:35256/0 (socket says 172.21.2.102:35256) 2021-02-10T11:33:27.219 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:27.215 7fc185732700 1 -- 172.21.2.102:0/887704806 learned_addr learned my addr 172.21.2.102:0/887704806 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:33:27.219 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:27.215 7fc185f33700 1 --2- 172.21.2.102:0/887704806 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc18008bb80 0x7fc18008bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:27.219 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:27.215 7fc184f31700 1 --2- 172.21.2.102:0/887704806 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc18009f9a0 0x7fc18008b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:27.219 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:27.215 7fc185732700 1 -- 172.21.2.102:0/887704806 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc18008bb80 msgr2=0x7fc18008bf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:27.219 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:27.215 7fc185732700 1 --2- 172.21.2.102:0/887704806 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc18008bb80 0x7fc18008bf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:27.219 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:27.215 7fc185732700 1 -- 172.21.2.102:0/887704806 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc18009f9a0 msgr2=0x7fc18008b560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:27.219 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:27.215 7fc185732700 1 --2- 172.21.2.102:0/887704806 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc18009f9a0 0x7fc18008b560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:27.219 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:27.215 7fc185732700 1 -- 172.21.2.102:0/887704806 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc18008cf20 con 0x7fc1801a06a0 2021-02-10T11:33:27.219 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:27.215 7fc185732700 1 --2- 172.21.2.102:0/887704806 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc1801a06a0 0x7fc18009f380 secure :-1 s=READY pgs=2398 cs=0 l=1 rev1=1 rx=0x7fc17800b920 tx=0x7fc17800bcc0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:27.219 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:27.215 7fc175ffb700 1 -- 172.21.2.102:0/887704806 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc178005ac0 con 0x7fc1801a06a0 2021-02-10T11:33:27.219 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:27.215 7fc175ffb700 1 -- 172.21.2.102:0/887704806 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc178005dc0 con 0x7fc1801a06a0 2021-02-10T11:33:27.219 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:27.215 7fc175ffb700 1 -- 172.21.2.102:0/887704806 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc1780062b0 con 0x7fc1801a06a0 2021-02-10T11:33:27.219 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:27.215 7fc187916700 1 -- 172.21.2.102:0/887704806 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc18008cf20 con 0x7fc1801a06a0 2021-02-10T11:33:27.220 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:27.215 7fc187916700 1 -- 172.21.2.102:0/887704806 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc18008d4b0 con 0x7fc1801a06a0 2021-02-10T11:33:27.220 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:27.216 7fc175ffb700 1 -- 172.21.2.102:0/887704806 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fc178005ac0 con 0x7fc1801a06a0 2021-02-10T11:33:27.220 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:27.216 7fc175ffb700 1 --2- 172.21.2.102:0/887704806 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc1600186c0 0x7fc16001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:27.220 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:27.216 7fc175ffb700 1 -- 172.21.2.102:0/887704806 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(123..123 src has 1..123) v4 ==== 4839+0+0 (secure 0 0 0) 0x7fc178032450 con 0x7fc1801a06a0 2021-02-10T11:33:27.220 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:27.216 7fc184f31700 1 --2- 172.21.2.102:0/887704806 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc1600186c0 0x7fc16001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:27.220 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:27.217 7fc184f31700 1 --2- 172.21.2.102:0/887704806 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc1600186c0 0x7fc16001abc0 crc :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:27.236 INFO:tasks.ceph.mds.b.gibba013.stdout:starting mds.b at 2021-02-10T11:33:27.247 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:27.246 7fc187916700 1 -- 172.21.2.102:0/887704806 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1800ebc50 con 0x7fc1801a06a0 2021-02-10T11:33:27.249 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:27.249 7fc175ffb700 1 -- 172.21.2.102:0/887704806 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fc178006570 con 0x7fc1801a06a0 2021-02-10T11:33:27.388 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:27.387 7fc187916700 1 -- 172.21.2.102:0/887704806 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd blacklist clear"} v 0) v1 -- 0x7fc1800f0b10 con 0x7fc1801a06a0 2021-02-10T11:33:27.576 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:27.575 7fc175ffb700 1 -- 172.21.2.102:0/887704806 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd blacklist clear"}]=0 removed all blacklist entries v124) v1 ==== 97+0+0 (secure 0 0 0) 0x7fc178006f20 con 0x7fc1801a06a0 2021-02-10T11:33:27.603 INFO:teuthology.orchestra.run.gibba002.stderr: removed all blacklist entries 2021-02-10T11:33:27.605 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:27.604 7fc187916700 1 -- 172.21.2.102:0/887704806 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc1600186c0 msgr2=0x7fc16001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:27.605 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:27.604 7fc187916700 1 --2- 172.21.2.102:0/887704806 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc1600186c0 0x7fc16001abc0 crc :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:27.605 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:27.604 7fc187916700 1 -- 172.21.2.102:0/887704806 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc1801a06a0 msgr2=0x7fc18009f380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:27.605 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:27.604 7fc187916700 1 --2- 172.21.2.102:0/887704806 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc1801a06a0 0x7fc18009f380 secure :-1 s=READY pgs=2398 cs=0 l=1 rev1=1 rx=0x7fc17800b920 tx=0x7fc17800bcc0).stop 2021-02-10T11:33:27.606 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:27.605 7fc187916700 1 -- 172.21.2.102:0/887704806 shutdown_connections 2021-02-10T11:33:27.606 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:27.605 7fc187916700 1 --2- 172.21.2.102:0/887704806 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc18008bb80 0x7fc18008bf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:27.606 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:27.605 7fc187916700 1 --2- 172.21.2.102:0/887704806 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc1600186c0 0x7fc16001abc0 unknown :-1 s=CLOSED pgs=1059 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:27.606 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:27.605 7fc187916700 1 --2- 172.21.2.102:0/887704806 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc18009f9a0 0x7fc18008b560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:27.606 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:27.605 7fc187916700 1 --2- 172.21.2.102:0/887704806 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc1801a06a0 0x7fc18009f380 unknown :-1 s=CLOSED pgs=2398 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:27.606 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:27.605 7fc187916700 1 -- 172.21.2.102:0/887704806 >> 172.21.2.102:0/887704806 conn(0x7fc1800b83b0 msgr2=0x7fc1800a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:27.606 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:27.605 7fc187916700 1 -- 172.21.2.102:0/887704806 shutdown_connections 2021-02-10T11:33:27.606 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:27.605 7fc187916700 1 -- 172.21.2.102:0/887704806 wait complete. 2021-02-10T11:33:27.614 INFO:tasks.cephfs.cephfs_test_case:['0', '1', '2', '3'] 2021-02-10T11:33:27.615 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth ls --format=json-pretty 2021-02-10T11:33:28.044 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:28.027 7fcfb8abe700 1 Processor -- start 2021-02-10T11:33:28.044 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:28.027 7fcfb8abe700 1 -- start start 2021-02-10T11:33:28.044 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:28.027 7fcfb8abe700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fcfb41b0470 0x7fcfb41b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:28.044 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:28.027 7fcfb8abe700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fcfb40f20e0 0x7fcfb40f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:28.044 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:28.027 7fcfb8abe700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcfb40f0b50 con 0x7fcfb40ef610 2021-02-10T11:33:28.045 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:28.027 7fcfb8abe700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcfb40f0820 con 0x7fcfb41b0470 2021-02-10T11:33:28.045 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:28.027 7fcfb8abe700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcfb40f24e0 con 0x7fcfb40f20e0 2021-02-10T11:33:28.045 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:28.028 7fcfb285e700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fcfb41b0470 0x7fcfb41b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:28.045 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:28.028 7fcfb285e700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fcfb41b0470 0x7fcfb41b0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:35262/0 (socket says 172.21.2.102:35262) 2021-02-10T11:33:28.045 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:28.028 7fcfb285e700 1 -- 172.21.2.102:0/23542161 learned_addr learned my addr 172.21.2.102:0/23542161 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:33:28.045 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:28.028 7fcfb285e700 1 -- 172.21.2.102:0/23542161 >> v2:172.21.2.113:3300/0 conn(0x7fcfb40f20e0 msgr2=0x7fcfb40f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:33:28.045 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:28.028 7fcfb285e700 1 --2- 172.21.2.102:0/23542161 >> v2:172.21.2.113:3300/0 conn(0x7fcfb40f20e0 0x7fcfb40f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:28.045 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:28.028 7fcfb285e700 1 -- 172.21.2.102:0/23542161 >> v1:172.21.2.113:6789/0 conn(0x7fcfb40ef610 legacy=0x7fcfb40efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:33:28.045 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:28.028 7fcfb285e700 1 -- 172.21.2.102:0/23542161 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfb40e52e0 con 0x7fcfb41b0470 2021-02-10T11:33:28.045 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:28.028 7fcfb285e700 1 --2- 172.21.2.102:0/23542161 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcfb41b0470 0x7fcfb41b0870 secure :-1 s=READY pgs=2401 cs=0 l=1 rev1=1 rx=0x7fcfa4006700 tx=0x7fcfa4009b80).ready entity=mon.0 client_cookie=997454bae2ff5d0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:28.045 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:28.028 7fcfb185c700 1 -- 172.21.2.102:0/23542161 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcfa400c0c0 con 0x7fcfb41b0470 2021-02-10T11:33:28.045 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:28.028 7fcfb185c700 1 -- 172.21.2.102:0/23542161 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfa400c340 con 0x7fcfb41b0470 2021-02-10T11:33:28.045 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:28.028 7fcfb185c700 1 -- 172.21.2.102:0/23542161 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcfa400c860 con 0x7fcfb41b0470 2021-02-10T11:33:28.046 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:28.028 7fcfb8abe700 1 -- 172.21.2.102:0/23542161 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcfb41b0470 msgr2=0x7fcfb41b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:28.046 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:28.028 7fcfb8abe700 1 --2- 172.21.2.102:0/23542161 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcfb41b0470 0x7fcfb41b0870 secure :-1 s=READY pgs=2401 cs=0 l=1 rev1=1 rx=0x7fcfa4006700 tx=0x7fcfa4009b80).stop 2021-02-10T11:33:28.046 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:28.029 7fcfb8abe700 1 -- 172.21.2.102:0/23542161 shutdown_connections 2021-02-10T11:33:28.046 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:28.029 7fcfb8abe700 1 --2- 172.21.2.102:0/23542161 >> v2:172.21.2.113:3300/0 conn(0x7fcfb40f20e0 0x7fcfb40f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:28.046 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:28.029 7fcfb8abe700 1 --2- 172.21.2.102:0/23542161 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcfb41b0470 0x7fcfb41b0870 unknown :-1 s=CLOSED pgs=2401 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:28.046 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:28.029 7fcfb8abe700 1 -- 172.21.2.102:0/23542161 >> 172.21.2.102:0/23542161 conn(0x7fcfb40ce3e0 msgr2=0x7fcfb40e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:28.046 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:28.029 7fcfb8abe700 1 -- 172.21.2.102:0/23542161 shutdown_connections 2021-02-10T11:33:28.046 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:28.029 7fcfb8abe700 1 -- 172.21.2.102:0/23542161 wait complete. 2021-02-10T11:33:28.047 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:28.044 7fcfb8abe700 1 Processor -- start 2021-02-10T11:33:28.048 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:28.044 7fcfb8abe700 1 -- start start 2021-02-10T11:33:28.048 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:28.044 7fcfb8abe700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcfb41acc20 0x7fcfb41ad020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:28.048 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:28.044 7fcfb8abe700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fcfb41ad640 0x7fcfb41ada40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:28.048 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:28.044 7fcfb8abe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcfb4188110 0x7fcfb4188510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:28.048 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:28.044 7fcfb8abe700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfb41ac720 con 0x7fcfb4188110 2021-02-10T11:33:28.048 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:28.044 7fcfb8abe700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfb40b1b10 con 0x7fcfb41ad640 2021-02-10T11:33:28.048 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:28.044 7fcfb8abe700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfb4188b30 con 0x7fcfb41acc20 2021-02-10T11:33:28.048 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:28.045 7fcfb305f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcfb4188110 0x7fcfb4188510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:28.048 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:28.045 7fcfb305f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcfb4188110 0x7fcfb4188510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:35268/0 (socket says 172.21.2.102:35268) 2021-02-10T11:33:28.048 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:28.045 7fcfb305f700 1 -- 172.21.2.102:0/697294726 learned_addr learned my addr 172.21.2.102:0/697294726 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:33:28.048 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:28.045 7fcfb305f700 1 -- 172.21.2.102:0/697294726 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcfb41acc20 msgr2=0x7fcfb41ad020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:28.048 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:28.045 7fcfb305f700 1 --2- 172.21.2.102:0/697294726 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcfb41acc20 0x7fcfb41ad020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:28.049 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:28.045 7fcfb305f700 1 -- 172.21.2.102:0/697294726 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fcfb41ad640 msgr2=0x7fcfb41ada40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:28.049 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:28.045 7fcfb305f700 1 --2- 172.21.2.102:0/697294726 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fcfb41ad640 0x7fcfb41ada40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:28.049 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:28.045 7fcfb305f700 1 -- 172.21.2.102:0/697294726 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfb4188d50 con 0x7fcfb4188110 2021-02-10T11:33:28.049 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:28.045 7fcfb305f700 1 --2- 172.21.2.102:0/697294726 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcfb4188110 0x7fcfb4188510 secure :-1 s=READY pgs=2402 cs=0 l=1 rev1=1 rx=0x7fcfac009980 tx=0x7fcfac00bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:28.049 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:28.045 7fcfab7fe700 1 -- 172.21.2.102:0/697294726 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcfac005af0 con 0x7fcfb4188110 2021-02-10T11:33:28.049 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:28.045 7fcfab7fe700 1 -- 172.21.2.102:0/697294726 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfac005d70 con 0x7fcfb4188110 2021-02-10T11:33:28.049 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:28.045 7fcfb8abe700 1 -- 172.21.2.102:0/697294726 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfb4188d50 con 0x7fcfb4188110 2021-02-10T11:33:28.049 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:28.045 7fcfb8abe700 1 -- 172.21.2.102:0/697294726 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfb41892e0 con 0x7fcfb4188110 2021-02-10T11:33:28.049 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:28.045 7fcfab7fe700 1 -- 172.21.2.102:0/697294726 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcfac0062b0 con 0x7fcfb4188110 2021-02-10T11:33:28.049 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:28.046 7fcfab7fe700 1 -- 172.21.2.102:0/697294726 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fcfac006030 con 0x7fcfb4188110 2021-02-10T11:33:28.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:28.046 7fcfab7fe700 1 --2- 172.21.2.102:0/697294726 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcf98018730 0x7fcf9801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:28.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:28.046 7fcfab7fe700 1 -- 172.21.2.102:0/697294726 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 4753+0+0 (secure 0 0 0) 0x7fcfac032820 con 0x7fcfb4188110 2021-02-10T11:33:28.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:28.046 7fcfb285e700 1 --2- 172.21.2.102:0/697294726 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcf98018730 0x7fcf9801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:28.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:28.047 7fcfb285e700 1 --2- 172.21.2.102:0/697294726 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcf98018730 0x7fcf9801ac10 crc :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:28.078 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:28.077 7fcfb8abe700 1 -- 172.21.2.102:0/697294726 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfb40eddf0 con 0x7fcfb4188110 2021-02-10T11:33:28.080 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:28.080 7fcfab7fe700 1 -- 172.21.2.102:0/697294726 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fcfac006570 con 0x7fcfb4188110 2021-02-10T11:33:28.210 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:28.209 7fcfb8abe700 1 -- 172.21.2.102:0/697294726 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fcfb40e52e0 con 0x7fcfb4188110 2021-02-10T11:33:28.210 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:28.210 7fcfab7fe700 1 -- 172.21.2.102:0/697294726 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v72) v1 ==== 80+0+4513 (secure 0 0 0) 0x7fcfac006030 con 0x7fcfb4188110 2021-02-10T11:33:28.211 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:33:28.211 INFO:teuthology.orchestra.run.gibba002.stdout:{ 2021-02-10T11:33:28.211 INFO:teuthology.orchestra.run.gibba002.stdout: "auth_dump": [ 2021-02-10T11:33:28.211 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:33:28.211 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "mds.a", 2021-02-10T11:33:28.211 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAevSNgsgyEARAAvr0cUjjCoEHLxaZSZ2HtVA==", 2021-02-10T11:33:28.211 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:33:28.211 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow", 2021-02-10T11:33:28.212 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:33:28.212 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:33:28.212 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:33:28.212 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:33:28.212 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:33:28.212 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:33:28.212 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "mds.b", 2021-02-10T11:33:28.212 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAevSNgwOFMMRAALfb1fbX56ao1TU8ClnTW1A==", 2021-02-10T11:33:28.212 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:33:28.212 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow", 2021-02-10T11:33:28.212 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:33:28.212 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:33:28.213 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:33:28.213 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:33:28.213 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:33:28.213 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:33:28.213 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.0", 2021-02-10T11:33:28.213 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAmvSNgj+bMMBAAD3HUD6fLiMAg+++iNzLJKQ==", 2021-02-10T11:33:28.213 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:33:28.213 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:33:28.213 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:33:28.213 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:33:28.213 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:33:28.213 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:33:28.213 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:33:28.213 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.1", 2021-02-10T11:33:28.214 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQApvSNgnvc+MRAAAOxqlNwiyLHqNITOrYeFKQ==", 2021-02-10T11:33:28.214 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:33:28.214 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:33:28.214 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:33:28.214 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:33:28.214 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:33:28.214 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:33:28.214 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:33:28.214 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.2", 2021-02-10T11:33:28.214 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAsvSNgZ4SbMRAA76IOYWfbiAuBoM/4k81GxA==", 2021-02-10T11:33:28.214 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:33:28.214 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:33:28.214 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:33:28.214 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:33:28.215 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:33:28.215 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:33:28.215 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:33:28.215 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.3", 2021-02-10T11:33:28.215 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAvvSNgFk3rMRAAWCmKKyw0LDQ+UlAHZAITOw==", 2021-02-10T11:33:28.215 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:33:28.215 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:33:28.215 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:33:28.215 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:33:28.215 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:33:28.215 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:33:28.215 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:33:28.215 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.4", 2021-02-10T11:33:28.216 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQA4vSNgpKcNJBAApJMmvj5YxVHPDOmvmQONOg==", 2021-02-10T11:33:28.216 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:33:28.216 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:33:28.216 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:33:28.216 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:33:28.216 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:33:28.216 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:33:28.216 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:33:28.216 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.5", 2021-02-10T11:33:28.216 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQA7vSNgf2ujJBAAfkuYpyb3WTzc8V5kw+Wktg==", 2021-02-10T11:33:28.216 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:33:28.216 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:33:28.216 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:33:28.216 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:33:28.217 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:33:28.217 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:33:28.217 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:33:28.217 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.6", 2021-02-10T11:33:28.217 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQA+vSNgqw9uJRAAFMU0R+UxU9C3o/9byb3e0g==", 2021-02-10T11:33:28.217 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:33:28.217 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:33:28.217 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:33:28.217 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:33:28.217 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:33:28.217 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:33:28.217 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:33:28.217 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.7", 2021-02-10T11:33:28.217 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQBBvSNgvWSrJRAA2KaCHmLpzovRgYzLvyofTQ==", 2021-02-10T11:33:28.218 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:33:28.218 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:33:28.218 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:33:28.218 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:33:28.218 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:33:28.218 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:33:28.218 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:33:28.218 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "client.0", 2021-02-10T11:33:28.218 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAfvSNgZd9XIBAAvyrvl8fdKuHnJZfczfRoFw==", 2021-02-10T11:33:28.218 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:33:28.218 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow", 2021-02-10T11:33:28.218 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow r", 2021-02-10T11:33:28.218 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:33:28.219 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:33:28.219 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:33:28.219 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:33:28.219 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "client.1", 2021-02-10T11:33:28.219 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAfvSNgSyfXNhAAIduPnF7Vd/6HSkJ0MlcMSA==", 2021-02-10T11:33:28.219 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:33:28.219 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow", 2021-02-10T11:33:28.219 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow r", 2021-02-10T11:33:28.219 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:33:28.219 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:33:28.219 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:33:28.219 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:33:28.219 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "client.2", 2021-02-10T11:33:28.220 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAgvSNgiIG/ERAA9ZTamaTI7ZsPSnxBqjplCg==", 2021-02-10T11:33:28.220 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:33:28.220 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow", 2021-02-10T11:33:28.220 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow r", 2021-02-10T11:33:28.220 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:33:28.220 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:33:28.220 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:33:28.220 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:33:28.220 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "client.3", 2021-02-10T11:33:28.220 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAgvSNgOWpOKBAAlXsEB04Q/GvtL06PqRMLeA==", 2021-02-10T11:33:28.220 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:33:28.220 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow", 2021-02-10T11:33:28.220 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow r", 2021-02-10T11:33:28.220 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:33:28.220 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:33:28.221 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:33:28.221 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:33:28.221 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "client.admin", 2021-02-10T11:33:28.221 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAavSNgO3GQBhAAL+NVnYskQRKVWLgyDPY/xw==", 2021-02-10T11:33:28.221 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:33:28.221 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow *", 2021-02-10T11:33:28.221 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:33:28.221 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:33:28.221 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:33:28.221 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:33:28.221 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:33:28.221 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:33:28.221 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "mgr.x", 2021-02-10T11:33:28.222 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAdvSNgXZZgJhAA7dtBSGzH+DsBwnddZpOBJA==", 2021-02-10T11:33:28.222 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:33:28.222 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow *", 2021-02-10T11:33:28.222 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow profile mgr", 2021-02-10T11:33:28.222 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:33:28.222 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:33:28.222 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:33:28.222 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:33:28.222 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "mgr.y", 2021-02-10T11:33:28.222 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAdvSNgVqQFDRAAwo6YEdWZCAr5GdOWVL/Qvg==", 2021-02-10T11:33:28.222 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:33:28.222 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow *", 2021-02-10T11:33:28.222 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow profile mgr", 2021-02-10T11:33:28.222 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:33:28.223 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:33:28.223 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:33:28.223 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:33:28.223 INFO:teuthology.orchestra.run.gibba002.stdout:} 2021-02-10T11:33:28.224 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:28.212 7fcfb8abe700 1 -- 172.21.2.102:0/697294726 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcf98018730 msgr2=0x7fcf9801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:28.224 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:28.212 7fcfb8abe700 1 --2- 172.21.2.102:0/697294726 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcf98018730 0x7fcf9801ac10 crc :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:28.224 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:28.212 7fcfb8abe700 1 -- 172.21.2.102:0/697294726 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcfb4188110 msgr2=0x7fcfb4188510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:28.224 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:28.212 7fcfb8abe700 1 --2- 172.21.2.102:0/697294726 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcfb4188110 0x7fcfb4188510 secure :-1 s=READY pgs=2402 cs=0 l=1 rev1=1 rx=0x7fcfac009980 tx=0x7fcfac00bcd0).stop 2021-02-10T11:33:28.224 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:28.212 7fcfb8abe700 1 -- 172.21.2.102:0/697294726 shutdown_connections 2021-02-10T11:33:28.224 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:28.212 7fcfb8abe700 1 --2- 172.21.2.102:0/697294726 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcfb4188110 0x7fcfb4188510 unknown :-1 s=CLOSED pgs=2402 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:28.224 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:28.212 7fcfb8abe700 1 --2- 172.21.2.102:0/697294726 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcf98018730 0x7fcf9801ac10 unknown :-1 s=CLOSED pgs=1060 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:28.224 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:28.212 7fcfb8abe700 1 --2- 172.21.2.102:0/697294726 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fcfb41ad640 0x7fcfb41ada40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:28.224 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:28.212 7fcfb8abe700 1 --2- 172.21.2.102:0/697294726 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcfb41acc20 0x7fcfb41ad020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:28.225 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:28.212 7fcfb8abe700 1 -- 172.21.2.102:0/697294726 >> 172.21.2.102:0/697294726 conn(0x7fcfb419f7a0 msgr2=0x7fcfb41922a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:28.225 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:28.212 7fcfb8abe700 1 -- 172.21.2.102:0/697294726 shutdown_connections 2021-02-10T11:33:28.225 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:28.212 7fcfb8abe700 1 -- 172.21.2.102:0/697294726 wait complete. 2021-02-10T11:33:28.225 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:33:28.657 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:28.638 7f8a43328700 1 Processor -- start 2021-02-10T11:33:28.657 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:28.638 7f8a43328700 1 -- start start 2021-02-10T11:33:28.657 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:28.639 7f8a43328700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8a3c0d1870 0x7f8a3c1b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:28.657 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:28.639 7f8a43328700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f8a3c1b0a10 0x7f8a3c0cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:28.657 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:28.639 7f8a43328700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a3c0f0b10 con 0x7f8a3c0d1c70 2021-02-10T11:33:28.657 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:28.639 7f8a43328700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a3c0fe230 con 0x7f8a3c1b0a10 2021-02-10T11:33:28.657 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:28.639 7f8a43328700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a3c0e5ed0 con 0x7f8a3c0d1870 2021-02-10T11:33:28.657 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:28.639 7f8a41945700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f8a3c0d1c70 0x7f8a3c0cf950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:55730/0 (socket says 172.21.2.102:55730) 2021-02-10T11:33:28.657 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:28.639 7f8a41945700 1 -- 172.21.2.102:0/3246256019 learned_addr learned my addr 172.21.2.102:0/3246256019 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:33:28.657 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:28.639 7f8a2ffff700 1 -- 172.21.2.102:0/3246256019 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1334764187 0 0) 0x7f8a34001180 con 0x7f8a3c0d1c70 2021-02-10T11:33:28.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:28.639 7f8a40943700 1 --2- 172.21.2.102:0/3246256019 >> v2:172.21.2.113:3300/0 conn(0x7f8a3c1b0a10 0x7f8a3c0cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:28.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:28.639 7f8a2ffff700 1 -- 172.21.2.102:0/3246256019 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a28002fe0 con 0x7f8a3c0d1c70 2021-02-10T11:33:28.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:28.639 7f8a41144700 1 --2- 172.21.2.102:0/3246256019 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8a3c0d1870 0x7f8a3c1b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:28.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:28.640 7f8a2ffff700 1 -- 172.21.2.102:0/3246256019 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2723816725 0 0) 0x7f8a34001180 con 0x7f8a3c0d1c70 2021-02-10T11:33:28.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:28.640 7f8a2ffff700 1 -- 172.21.2.102:0/3246256019 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8a3c0d1870 msgr2=0x7f8a3c1b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:33:28.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:28.640 7f8a2ffff700 1 --2- 172.21.2.102:0/3246256019 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8a3c0d1870 0x7f8a3c1b03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:28.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:28.640 7f8a2ffff700 1 -- 172.21.2.102:0/3246256019 >> v2:172.21.2.113:3300/0 conn(0x7f8a3c1b0a10 msgr2=0x7f8a3c0cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:33:28.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:28.640 7f8a2ffff700 1 --2- 172.21.2.102:0/3246256019 >> v2:172.21.2.113:3300/0 conn(0x7f8a3c1b0a10 0x7f8a3c0cf330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:28.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:28.640 7f8a2ffff700 1 -- 172.21.2.102:0/3246256019 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a3c0f0650 con 0x7f8a3c0d1c70 2021-02-10T11:33:28.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:28.640 7f8a2ffff700 1 -- 172.21.2.102:0/3246256019 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7f8a34001680 con 0x7f8a3c0d1c70 2021-02-10T11:33:28.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:28.640 7f8a2ffff700 1 -- 172.21.2.102:0/3246256019 >> v1:172.21.2.102:6789/0 conn(0x7f8a3c0d1c70 legacy=0x7f8a3c0cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:28.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:28.640 7f8a2ffff700 1 --2- 172.21.2.102:0/3246256019 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8a28003a60 0x7f8a28005f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:28.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:28.640 7f8a2ffff700 1 --2- 172.21.2.102:0/3246256019 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8a2804a790 0x7f8a2804cc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:28.659 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:28.640 7f8a2ffff700 1 --2- 172.21.2.102:0/3246256019 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8a2804d500 0x7f8a2804fa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:28.659 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:28.640 7f8a2ffff700 1 -- 172.21.2.102:0/3246256019 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a280502d0 con 0x7f8a28003a60 2021-02-10T11:33:28.659 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:28.640 7f8a2ffff700 1 -- 172.21.2.102:0/3246256019 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a280504f0 con 0x7f8a2804d500 2021-02-10T11:33:28.659 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:28.640 7f8a2ffff700 1 -- 172.21.2.102:0/3246256019 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a28050710 con 0x7f8a2804a790 2021-02-10T11:33:28.659 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:28.640 7f8a41144700 1 --2- 172.21.2.102:0/3246256019 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8a28003a60 0x7f8a28005f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:28.659 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:28.640 7f8a40943700 1 --2- 172.21.2.102:0/3246256019 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8a2804a790 0x7f8a2804cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:28.659 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:28.640 7f8a41945700 1 --2- 172.21.2.102:0/3246256019 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8a2804d500 0x7f8a2804fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:28.659 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:28.640 7f8a41144700 1 -- 172.21.2.102:0/3246256019 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8a2804a790 msgr2=0x7f8a2804cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:33:28.659 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:28.640 7f8a41144700 1 --2- 172.21.2.102:0/3246256019 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8a2804a790 0x7f8a2804cc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:28.659 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:28.640 7f8a41144700 1 -- 172.21.2.102:0/3246256019 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8a2804d500 msgr2=0x7f8a2804fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:33:28.659 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:28.640 7f8a41144700 1 --2- 172.21.2.102:0/3246256019 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8a2804d500 0x7f8a2804fa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:28.659 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:28.640 7f8a41144700 1 -- 172.21.2.102:0/3246256019 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a280509b0 con 0x7f8a28003a60 2021-02-10T11:33:28.660 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:28.640 7f8a41144700 1 --2- 172.21.2.102:0/3246256019 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8a28003a60 0x7f8a28005f60 secure :-1 s=READY pgs=2404 cs=0 l=1 rev1=1 rx=0x7f8a3800c370 tx=0x7f8a3800cb00).ready entity=mon.0 client_cookie=66dd5520fe8c304c server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:28.660 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:28.641 7f8a2ffff700 1 -- 172.21.2.102:0/3246256019 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8a380054f0 con 0x7f8a28003a60 2021-02-10T11:33:28.660 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:28.641 7f8a2ffff700 1 -- 172.21.2.102:0/3246256019 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a38005870 con 0x7f8a28003a60 2021-02-10T11:33:28.660 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:28.641 7f8a2ffff700 1 -- 172.21.2.102:0/3246256019 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8a38005d90 con 0x7f8a28003a60 2021-02-10T11:33:28.660 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:28.641 7f8a43328700 1 -- 172.21.2.102:0/3246256019 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8a28003a60 msgr2=0x7f8a28005f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:28.660 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:28.641 7f8a43328700 1 --2- 172.21.2.102:0/3246256019 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8a28003a60 0x7f8a28005f60 secure :-1 s=READY pgs=2404 cs=0 l=1 rev1=1 rx=0x7f8a3800c370 tx=0x7f8a3800cb00).stop 2021-02-10T11:33:28.660 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:28.641 7f8a41945700 1 -- 172.21.2.102:0/3246256019 reap_dead start 2021-02-10T11:33:28.660 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:28.641 7f8a43328700 1 -- 172.21.2.102:0/3246256019 shutdown_connections 2021-02-10T11:33:28.660 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:28.641 7f8a43328700 1 -- 172.21.2.102:0/3246256019 >> 172.21.2.102:0/3246256019 conn(0x7f8a3c0ce450 msgr2=0x7f8a3c0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:28.660 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:28.642 7f8a43328700 1 -- 172.21.2.102:0/3246256019 shutdown_connections 2021-02-10T11:33:28.660 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:28.642 7f8a43328700 1 -- 172.21.2.102:0/3246256019 wait complete. 2021-02-10T11:33:28.660 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:28.656 7f8a43328700 1 Processor -- start 2021-02-10T11:33:28.660 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:28.656 7f8a43328700 1 -- start start 2021-02-10T11:33:28.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:28.657 7f8a43328700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8a3c195650 0x7f8a3c0b5560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:28.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:28.657 7f8a43328700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8a3c0b5b80 0x7f8a3c1a2e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:28.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:28.657 7f8a43328700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8a3c0acfa0 0x7f8a3c1a3440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:28.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:28.657 7f8a43328700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a3c0b6010 con 0x7f8a3c0b5b80 2021-02-10T11:33:28.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:28.657 7f8a43328700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a3c1a3a60 con 0x7f8a3c0acfa0 2021-02-10T11:33:28.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:28.657 7f8a43328700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a3c1a3c40 con 0x7f8a3c195650 2021-02-10T11:33:28.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:28.657 7f8a40943700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8a3c0b5b80 0x7f8a3c1a2e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:28.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:28.657 7f8a40943700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8a3c0b5b80 0x7f8a3c1a2e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:35288/0 (socket says 172.21.2.102:35288) 2021-02-10T11:33:28.663 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:28.657 7f8a40943700 1 -- 172.21.2.102:0/1679438444 learned_addr learned my addr 172.21.2.102:0/1679438444 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:33:28.663 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:28.657 7f8a41945700 1 --2- 172.21.2.102:0/1679438444 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8a3c0acfa0 0x7f8a3c1a3440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:28.663 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:28.657 7f8a41144700 1 --2- 172.21.2.102:0/1679438444 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8a3c195650 0x7f8a3c0b5560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:28.663 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:28.657 7f8a40943700 1 -- 172.21.2.102:0/1679438444 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8a3c195650 msgr2=0x7f8a3c0b5560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:28.663 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:28.657 7f8a40943700 1 --2- 172.21.2.102:0/1679438444 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8a3c195650 0x7f8a3c0b5560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:28.663 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:28.657 7f8a40943700 1 -- 172.21.2.102:0/1679438444 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8a3c0acfa0 msgr2=0x7f8a3c1a3440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:28.663 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:28.657 7f8a40943700 1 --2- 172.21.2.102:0/1679438444 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8a3c0acfa0 0x7f8a3c1a3440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:28.663 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:28.657 7f8a40943700 1 -- 172.21.2.102:0/1679438444 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a3c0bcae0 con 0x7f8a3c0b5b80 2021-02-10T11:33:28.663 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:28.657 7f8a41144700 1 --2- 172.21.2.102:0/1679438444 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8a3c195650 0x7f8a3c0b5560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:33:28.663 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:28.657 7f8a40943700 1 --2- 172.21.2.102:0/1679438444 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8a3c0b5b80 0x7f8a3c1a2e20 secure :-1 s=READY pgs=2405 cs=0 l=1 rev1=1 rx=0x7f8a300099e0 tx=0x7f8a3000bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:28.663 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:28.658 7f8a2dffb700 1 -- 172.21.2.102:0/1679438444 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8a3000c2f0 con 0x7f8a3c0b5b80 2021-02-10T11:33:28.663 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:28.658 7f8a2dffb700 1 -- 172.21.2.102:0/1679438444 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a3000c630 con 0x7f8a3c0b5b80 2021-02-10T11:33:28.663 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:28.658 7f8a2dffb700 1 -- 172.21.2.102:0/1679438444 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8a3000ca80 con 0x7f8a3c0b5b80 2021-02-10T11:33:28.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:28.658 7f8a43328700 1 -- 172.21.2.102:0/1679438444 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a3c0bcae0 con 0x7f8a3c0b5b80 2021-02-10T11:33:28.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:28.658 7f8a43328700 1 -- 172.21.2.102:0/1679438444 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a3c0bd070 con 0x7f8a3c0b5b80 2021-02-10T11:33:28.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:28.658 7f8a2dffb700 1 -- 172.21.2.102:0/1679438444 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f8a3000c2f0 con 0x7f8a3c0b5b80 2021-02-10T11:33:28.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:28.658 7f8a2dffb700 1 --2- 172.21.2.102:0/1679438444 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8a1c0186c0 0x7f8a1c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:28.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:28.659 7f8a2dffb700 1 -- 172.21.2.102:0/1679438444 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 4753+0+0 (secure 0 0 0) 0x7f8a300325a0 con 0x7f8a3c0b5b80 2021-02-10T11:33:28.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:28.659 7f8a41144700 1 --2- 172.21.2.102:0/1679438444 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8a1c0186c0 0x7f8a1c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:28.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:28.659 7f8a41144700 1 --2- 172.21.2.102:0/1679438444 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8a1c0186c0 0x7f8a1c01abc0 crc :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:28.690 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:28.689 7f8a43328700 1 -- 172.21.2.102:0/1679438444 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a3c0b6010 con 0x7f8a3c0b5b80 2021-02-10T11:33:28.692 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:28.691 7f8a2dffb700 1 -- 172.21.2.102:0/1679438444 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f8a30000cc0 con 0x7f8a3c0b5b80 2021-02-10T11:33:28.815 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:28.815 7f8a43328700 1 -- 172.21.2.102:0/1679438444 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8a3c0b6010 con 0x7f8a3c0b5b80 2021-02-10T11:33:28.816 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:28.815 7f8a2dffb700 1 -- 172.21.2.102:0/1679438444 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v124) v1 ==== 74+0+11154 (secure 0 0 0) 0x7f8a3000c2f0 con 0x7f8a3c0b5b80 2021-02-10T11:33:28.816 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:33:28.817 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":124,"fsid":"68db30ea-dbce-4aef-a6df-f01c6fea0513","created":"2021-02-10 11:03:06.331557","modified":"2021-02-10 11:33:27.569425","last_up_change":"2021-02-10 11:03:32.690562","last_in_change":"2021-02-10 11:03:32.690562","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":25,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:03:37.322519","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"be691b4d-a039-408a-af80-9d53f02b6f1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6818","nonce":36088},{"type":"v1","addr":"172.21.2.102:6820","nonce":36088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6822","nonce":36088},{"type":"v1","addr":"172.21.2.102:6824","nonce":36088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6830","nonce":36088},{"type":"v1","addr":"172.21.2.102:6832","nonce":36088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6826","nonce":36088},{"type":"v1","addr":"172.21.2.102:6828","nonce":36088}]},"public_addr":"172.21.2.102:6820/36088","cluster_addr":"172.21.2.102:6824/36088","heartbeat_back_addr":"172.21.2.102:6832/36088","heartbeat_front_addr":"172.21.2.102:6828/36088","state":["exists","up"]},{"osd":1,"uuid":"64e3ccc8-9c69-4d7c-865e-af8e83822635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6819","nonce":36087},{"type":"v1","addr":"172.21.2.102:6821","nonce":36087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6823","nonce":36087},{"type":"v1","addr":"172.21.2.102:6825","nonce":36087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6831","nonce":36087},{"type":"v1","addr":"172.21.2.102:6833","nonce":36087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6827","nonce":36087},{"type":"v1","addr":"172.21.2.102:6829","nonce":36087}]},"public_addr":"172.21.2.102:6821/36087","cluster_addr":"172.21.2.102:6825/36087","heartbeat_back_addr":"172.21.2.102:6833/36087","heartbeat_front_addr":"172.21.2.102:6829/36087","state":["exists","up"]},{"osd":2,"uuid":"153700af-747f-481f-bdf4-d6072d3a842f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6810","nonce":36089},{"type":"v1","addr":"172.21.2.102:6811","nonce":36089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6812","nonce":36089},{"type":"v1","addr":"172.21.2.102:6813","nonce":36089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6816","nonce":36089},{"type":"v1","addr":"172.21.2.102:6817","nonce":36089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6814","nonce":36089},{"type":"v1","addr":"172.21.2.102:6815","nonce":36089}]},"public_addr":"172.21.2.102:6811/36089","cluster_addr":"172.21.2.102:6813/36089","heartbeat_back_addr":"172.21.2.102:6817/36089","heartbeat_front_addr":"172.21.2.102:6815/36089","state":["exists","up"]},{"osd":3,"uuid":"cbc678f4-83a9-45e6-8830-2f7c232b1ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6802","nonce":36090},{"type":"v1","addr":"172.21.2.102:6803","nonce":36090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6804","nonce":36090},{"type":"v1","addr":"172.21.2.102:6805","nonce":36090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6808","nonce":36090},{"type":"v1","addr":"172.21.2.102:6809","nonce":36090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6806","nonce":36090},{"type":"v1","addr":"172.21.2.102:6807","nonce":36090}]},"public_addr":"172.21.2.102:6803/36090","cluster_addr":"172.21.2.102:6805/36090","heartbeat_back_addr":"172.21.2.102:6809/36090","heartbeat_front_addr":"172.21.2.102:6807/36090","state":["exists","up"]},{"osd":4,"uuid":"c01e4008-b258-4f10-afd2-9e7f37214351","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6808","nonce":36368},{"type":"v1","addr":"172.21.2.113:6811","nonce":36368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6814","nonce":36368},{"type":"v1","addr":"172.21.2.113:6818","nonce":36368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6826","nonce":36368},{"type":"v1","addr":"172.21.2.113:6828","nonce":36368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6820","nonce":36368},{"type":"v1","addr":"172.21.2.113:6824","nonce":36368}]},"public_addr":"172.21.2.113:6811/36368","cluster_addr":"172.21.2.113:6818/36368","heartbeat_back_addr":"172.21.2.113:6828/36368","heartbeat_front_addr":"172.21.2.113:6824/36368","state":["exists","up"]},{"osd":5,"uuid":"551f6007-daf8-4a98-8ee0-c2b16bcccb5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6802","nonce":36366},{"type":"v1","addr":"172.21.2.113:6805","nonce":36366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6807","nonce":36366},{"type":"v1","addr":"172.21.2.113:6810","nonce":36366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6819","nonce":36366},{"type":"v1","addr":"172.21.2.113:6822","nonce":36366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6812","nonce":36366},{"type":"v1","addr":"172.21.2.113:6815","nonce":36366}]},"public_addr":"172.21.2.113:6805/36366","cluster_addr":"172.21.2.113:6810/36366","heartbeat_back_addr":"172.21.2.113:6822/36366","heartbeat_front_addr":"172.21.2.113:6815/36366","state":["exists","up"]},{"osd":6,"uuid":"bf67c040-0e50-48be-b102-c3943bc9ebef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6800","nonce":36367},{"type":"v1","addr":"172.21.2.113:6801","nonce":36367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6803","nonce":36367},{"type":"v1","addr":"172.21.2.113:6804","nonce":36367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6813","nonce":36367},{"type":"v1","addr":"172.21.2.113:6816","nonce":36367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6806","nonce":36367},{"type":"v1","addr":"172.21.2.113:6809","nonce":36367}]},"public_addr":"172.21.2.113:6801/36367","cluster_addr":"172.21.2.113:6804/36367","heartbeat_back_addr":"172.21.2.113:6816/36367","heartbeat_front_addr":"172.21.2.113:6809/36367","state":["exists","up"]},{"osd":7,"uuid":"f5461f80-018d-428f-be89-2eb2bcbe7bb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6817","nonce":36365},{"type":"v1","addr":"172.21.2.113:6821","nonce":36365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6823","nonce":36365},{"type":"v1","addr":"172.21.2.113:6825","nonce":36365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6830","nonce":36365},{"type":"v1","addr":"172.21.2.113:6831","nonce":36365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6827","nonce":36365},{"type":"v1","addr":"172.21.2.113:6829","nonce":36365}]},"public_addr":"172.21.2.113:6821/36365","cluster_addr":"172.21.2.113:6825/36365","heartbeat_back_addr":"172.21.2.113:6831/36365","heartbeat_front_addr":"172.21.2.113:6829/36365","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:33:28.818 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:28.817 7f8a43328700 1 -- 172.21.2.102:0/1679438444 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8a1c0186c0 msgr2=0x7f8a1c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:28.818 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:28.817 7f8a43328700 1 --2- 172.21.2.102:0/1679438444 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8a1c0186c0 0x7f8a1c01abc0 crc :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:28.818 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:28.817 7f8a43328700 1 -- 172.21.2.102:0/1679438444 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8a3c0b5b80 msgr2=0x7f8a3c1a2e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:28.818 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:28.817 7f8a43328700 1 --2- 172.21.2.102:0/1679438444 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8a3c0b5b80 0x7f8a3c1a2e20 secure :-1 s=READY pgs=2405 cs=0 l=1 rev1=1 rx=0x7f8a300099e0 tx=0x7f8a3000bcf0).stop 2021-02-10T11:33:28.818 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:28.817 7f8a43328700 1 -- 172.21.2.102:0/1679438444 shutdown_connections 2021-02-10T11:33:28.818 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:28.817 7f8a43328700 1 --2- 172.21.2.102:0/1679438444 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8a1c0186c0 0x7f8a1c01abc0 unknown :-1 s=CLOSED pgs=1061 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:28.819 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:28.817 7f8a43328700 1 --2- 172.21.2.102:0/1679438444 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8a3c0acfa0 0x7f8a3c1a3440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:28.819 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:28.817 7f8a43328700 1 --2- 172.21.2.102:0/1679438444 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8a3c0b5b80 0x7f8a3c1a2e20 unknown :-1 s=CLOSED pgs=2405 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:28.819 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:28.817 7f8a43328700 1 --2- 172.21.2.102:0/1679438444 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8a3c195650 0x7f8a3c0b5560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:28.819 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:28.818 7f8a43328700 1 -- 172.21.2.102:0/1679438444 >> 172.21.2.102:0/1679438444 conn(0x7f8a3c0b3040 msgr2=0x7f8a3c0c3250 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:28.819 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:28.818 7f8a43328700 1 -- 172.21.2.102:0/1679438444 shutdown_connections 2021-02-10T11:33:28.819 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:28.818 7f8a43328700 1 -- 172.21.2.102:0/1679438444 wait complete. 2021-02-10T11:33:28.827 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:03:37.322519', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2021-02-10T11:33:28.827 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:33:29.260 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:29.243 7fe2c695b700 1 Processor -- start 2021-02-10T11:33:29.260 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:29.243 7fe2c695b700 1 -- start start 2021-02-10T11:33:29.260 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:29.243 7fe2c695b700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fe2c00d1870 0x7fe2c01b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:29.260 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:29.243 7fe2c695b700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fe2c00cff70 0x7fe2c0109130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:29.260 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:29.243 7fe2c695b700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2c00fe1c0 con 0x7fe2c01b0ae0 2021-02-10T11:33:29.260 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:29.243 7fe2c695b700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2c0109ad0 con 0x7fe2c00cff70 2021-02-10T11:33:29.260 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:29.243 7fe2c695b700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2c0109750 con 0x7fe2c00d1870 2021-02-10T11:33:29.260 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:29.244 7fe2c4f78700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fe2c00cff70 0x7fe2c0109130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:29.260 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:29.244 7fe2c4f78700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fe2c00cff70 0x7fe2c0109130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:35292/0 (socket says 172.21.2.102:35292) 2021-02-10T11:33:29.261 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:29.244 7fe2c4f78700 1 -- 172.21.2.102:0/2908026663 learned_addr learned my addr 172.21.2.102:0/2908026663 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:33:29.261 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:29.244 7fe2c4f78700 1 -- 172.21.2.102:0/2908026663 >> v2:172.21.2.113:3300/0 conn(0x7fe2c00d1870 msgr2=0x7fe2c01b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:33:29.261 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:29.244 7fe2c4f78700 1 --2- 172.21.2.102:0/2908026663 >> v2:172.21.2.113:3300/0 conn(0x7fe2c00d1870 0x7fe2c01b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:29.261 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:29.244 7fe2c4f78700 1 -- 172.21.2.102:0/2908026663 >> v1:172.21.2.113:6789/0 conn(0x7fe2c01b0ae0 legacy=0x7fe2c0108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:33:29.261 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:29.244 7fe2c4f78700 1 -- 172.21.2.102:0/2908026663 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2c0088070 con 0x7fe2c00cff70 2021-02-10T11:33:29.261 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:29.244 7fe2c4f78700 1 --2- 172.21.2.102:0/2908026663 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe2c00cff70 0x7fe2c0109130 secure :-1 s=READY pgs=2406 cs=0 l=1 rev1=1 rx=0x7fe2b8006f00 tx=0x7fe2b8009f10).ready entity=mon.0 client_cookie=bb6c4bc398160019 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:29.261 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:29.244 7fe2beffd700 1 -- 172.21.2.102:0/2908026663 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe2b800b3b0 con 0x7fe2c00cff70 2021-02-10T11:33:29.261 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:29.244 7fe2beffd700 1 -- 172.21.2.102:0/2908026663 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2b800b630 con 0x7fe2c00cff70 2021-02-10T11:33:29.261 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:29.244 7fe2beffd700 1 -- 172.21.2.102:0/2908026663 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe2b800bb50 con 0x7fe2c00cff70 2021-02-10T11:33:29.261 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:29.245 7fe2c695b700 1 -- 172.21.2.102:0/2908026663 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe2c00cff70 msgr2=0x7fe2c0109130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:29.261 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:29.245 7fe2c695b700 1 --2- 172.21.2.102:0/2908026663 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe2c00cff70 0x7fe2c0109130 secure :-1 s=READY pgs=2406 cs=0 l=1 rev1=1 rx=0x7fe2b8006f00 tx=0x7fe2b8009f10).stop 2021-02-10T11:33:29.261 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:29.245 7fe2c695b700 1 -- 172.21.2.102:0/2908026663 shutdown_connections 2021-02-10T11:33:29.261 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:29.245 7fe2c695b700 1 --2- 172.21.2.102:0/2908026663 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe2c00cff70 0x7fe2c0109130 unknown :-1 s=CLOSED pgs=2406 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:29.262 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:29.245 7fe2c695b700 1 --2- 172.21.2.102:0/2908026663 >> v2:172.21.2.113:3300/0 conn(0x7fe2c00d1870 0x7fe2c01b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:29.262 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:29.245 7fe2c695b700 1 -- 172.21.2.102:0/2908026663 >> 172.21.2.102:0/2908026663 conn(0x7fe2c00ce420 msgr2=0x7fe2c00d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:29.262 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:29.245 7fe2c695b700 1 -- 172.21.2.102:0/2908026663 shutdown_connections 2021-02-10T11:33:29.262 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:29.245 7fe2c695b700 1 -- 172.21.2.102:0/2908026663 wait complete. 2021-02-10T11:33:29.263 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:29.260 7fe2c695b700 1 Processor -- start 2021-02-10T11:33:29.263 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:29.260 7fe2c695b700 1 -- start start 2021-02-10T11:33:29.263 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:29.260 7fe2c695b700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe2c018f9d0 0x7fe2c0084cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:29.263 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:29.260 7fe2c695b700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe2c00c23e0 0x7fe2c01a44b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:29.263 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:29.260 7fe2c695b700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe2c00c27e0 0x7fe2c00859f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:29.264 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:29.260 7fe2c695b700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2c00854f0 con 0x7fe2c00c23e0 2021-02-10T11:33:29.264 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:29.260 7fe2c695b700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2c018fdd0 con 0x7fe2c00c27e0 2021-02-10T11:33:29.264 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:29.260 7fe2c695b700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2c01a4ad0 con 0x7fe2c018f9d0 2021-02-10T11:33:29.264 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:29.260 7fe2bf7fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe2c00c23e0 0x7fe2c01a44b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:29.264 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:29.260 7fe2bf7fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe2c00c23e0 0x7fe2c01a44b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:35298/0 (socket says 172.21.2.102:35298) 2021-02-10T11:33:29.264 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:29.260 7fe2bf7fe700 1 -- 172.21.2.102:0/245307882 learned_addr learned my addr 172.21.2.102:0/245307882 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:33:29.264 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:29.261 7fe2bf7fe700 1 -- 172.21.2.102:0/245307882 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe2c018f9d0 msgr2=0x7fe2c0084cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:29.264 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:29.261 7fe2bf7fe700 1 --2- 172.21.2.102:0/245307882 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe2c018f9d0 0x7fe2c0084cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:29.264 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:29.261 7fe2bf7fe700 1 -- 172.21.2.102:0/245307882 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe2c00c27e0 msgr2=0x7fe2c00859f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:29.264 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:29.261 7fe2bf7fe700 1 --2- 172.21.2.102:0/245307882 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe2c00c27e0 0x7fe2c00859f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:29.264 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:29.261 7fe2bf7fe700 1 -- 172.21.2.102:0/245307882 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2c01a4cf0 con 0x7fe2c00c23e0 2021-02-10T11:33:29.264 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:29.261 7fe2bf7fe700 1 --2- 172.21.2.102:0/245307882 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe2c00c23e0 0x7fe2c01a44b0 secure :-1 s=READY pgs=2407 cs=0 l=1 rev1=1 rx=0x7fe2b400b5b0 tx=0x7fe2b400b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:29.264 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:29.261 7fe2bcff9700 1 -- 172.21.2.102:0/245307882 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe2b40009b0 con 0x7fe2c00c23e0 2021-02-10T11:33:29.265 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:29.261 7fe2bcff9700 1 -- 172.21.2.102:0/245307882 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2b400bd40 con 0x7fe2c00c23e0 2021-02-10T11:33:29.265 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:29.261 7fe2c695b700 1 -- 172.21.2.102:0/245307882 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2c01a4cf0 con 0x7fe2c00c23e0 2021-02-10T11:33:29.265 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:29.261 7fe2c695b700 1 -- 172.21.2.102:0/245307882 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2c0086300 con 0x7fe2c00c23e0 2021-02-10T11:33:29.265 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:29.262 7fe2bcff9700 1 -- 172.21.2.102:0/245307882 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe2b4006210 con 0x7fe2c00c23e0 2021-02-10T11:33:29.265 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:29.262 7fe2bcff9700 1 -- 172.21.2.102:0/245307882 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fe2b4006020 con 0x7fe2c00c23e0 2021-02-10T11:33:29.265 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:29.262 7fe2bcff9700 1 --2- 172.21.2.102:0/245307882 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe2a00186c0 0x7fe2a001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:29.265 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:29.262 7fe2bcff9700 1 -- 172.21.2.102:0/245307882 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 4753+0+0 (secure 0 0 0) 0x7fe2b40324a0 con 0x7fe2c00c23e0 2021-02-10T11:33:29.265 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:29.262 7fe2bffff700 1 --2- 172.21.2.102:0/245307882 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe2a00186c0 0x7fe2a001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:29.265 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:29.263 7fe2bffff700 1 --2- 172.21.2.102:0/245307882 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe2a00186c0 0x7fe2a001abc0 crc :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:29.294 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:29.293 7fe2c695b700 1 -- 172.21.2.102:0/245307882 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2c0082a70 con 0x7fe2c00c23e0 2021-02-10T11:33:29.296 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:29.296 7fe2bcff9700 1 -- 172.21.2.102:0/245307882 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fe2b4000940 con 0x7fe2c00c23e0 2021-02-10T11:33:29.423 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:29.422 7fe2c695b700 1 -- 172.21.2.102:0/245307882 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7fe2c0082a70 con 0x7fe2c00c23e0 2021-02-10T11:33:29.423 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:29.422 7fe2bcff9700 1 -- 172.21.2.102:0/245307882 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v124) v1 ==== 92+0+10 (secure 0 0 0) 0x7fe2b400bd40 con 0x7fe2c00c23e0 2021-02-10T11:33:29.423 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:33:29.425 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:29.424 7fe2c695b700 1 -- 172.21.2.102:0/245307882 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe2a00186c0 msgr2=0x7fe2a001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:29.425 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:29.424 7fe2c695b700 1 --2- 172.21.2.102:0/245307882 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe2a00186c0 0x7fe2a001abc0 crc :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:29.425 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:29.424 7fe2c695b700 1 -- 172.21.2.102:0/245307882 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe2c00c23e0 msgr2=0x7fe2c01a44b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:29.425 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:29.424 7fe2c695b700 1 --2- 172.21.2.102:0/245307882 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe2c00c23e0 0x7fe2c01a44b0 secure :-1 s=READY pgs=2407 cs=0 l=1 rev1=1 rx=0x7fe2b400b5b0 tx=0x7fe2b400b950).stop 2021-02-10T11:33:29.425 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:29.425 7fe2c695b700 1 -- 172.21.2.102:0/245307882 shutdown_connections 2021-02-10T11:33:29.425 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:29.425 7fe2c695b700 1 --2- 172.21.2.102:0/245307882 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe2a00186c0 0x7fe2a001abc0 unknown :-1 s=CLOSED pgs=1062 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:29.425 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:29.425 7fe2c695b700 1 --2- 172.21.2.102:0/245307882 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe2c00c27e0 0x7fe2c00859f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:29.425 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:29.425 7fe2c695b700 1 --2- 172.21.2.102:0/245307882 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe2c00c23e0 0x7fe2c01a44b0 unknown :-1 s=CLOSED pgs=2407 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:29.426 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:29.425 7fe2c695b700 1 --2- 172.21.2.102:0/245307882 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe2c018f9d0 0x7fe2c0084cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:29.426 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:29.425 7fe2c695b700 1 -- 172.21.2.102:0/245307882 >> 172.21.2.102:0/245307882 conn(0x7fe2c00b5dc0 msgr2=0x7fe2c00c2d90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:29.426 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:29.425 7fe2c695b700 1 -- 172.21.2.102:0/245307882 shutdown_connections 2021-02-10T11:33:29.426 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:29.425 7fe2c695b700 1 -- 172.21.2.102:0/245307882 wait complete. 2021-02-10T11:33:29.435 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --format=json-pretty osd lspools 2021-02-10T11:33:29.863 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:29.845 7f2f9667b700 1 Processor -- start 2021-02-10T11:33:29.863 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:29.845 7f2f9667b700 1 -- start start 2021-02-10T11:33:29.863 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:29.846 7f2f9667b700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2f900d0340 0x7f2f900ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:29.864 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:29.846 7f2f9667b700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f900f0b10 con 0x7f2f900d1870 2021-02-10T11:33:29.864 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:29.846 7f2f9667b700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f901b0be0 con 0x7f2f900d1c70 2021-02-10T11:33:29.864 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:29.846 7f2f9667b700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f900e5ed0 con 0x7f2f900d0340 2021-02-10T11:33:29.864 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:29.846 7f2f8ffff700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f2f900d1870 0x7f2f901b03f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:55760/0 (socket says 172.21.2.102:55760) 2021-02-10T11:33:29.864 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:29.846 7f2f8ffff700 1 -- 172.21.2.102:0/2268941598 learned_addr learned my addr 172.21.2.102:0/2268941598 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:33:29.864 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:29.846 7f2f8f7fe700 1 --2- 172.21.2.102:0/2268941598 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2f900d0340 0x7f2f900ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:29.864 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:29.847 7f2f8effd700 1 -- 172.21.2.102:0/2268941598 <== mon.1 v1:172.21.2.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4224865638 0 0) 0x7f2f88000f90 con 0x7f2f900d1c70 2021-02-10T11:33:29.864 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:29.847 7f2f8effd700 1 -- 172.21.2.102:0/2268941598 --> v1:172.21.2.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f78002fe0 con 0x7f2f900d1c70 2021-02-10T11:33:29.864 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:29.847 7f2f8effd700 1 -- 172.21.2.102:0/2268941598 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2482346667 0 0) 0x7f2f800009b0 con 0x7f2f900d1870 2021-02-10T11:33:29.864 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:29.847 7f2f8effd700 1 -- 172.21.2.102:0/2268941598 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f78004370 con 0x7f2f900d1870 2021-02-10T11:33:29.864 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:29.847 7f2f8f7fe700 1 -- 172.21.2.102:0/2268941598 >> v1:172.21.2.113:6789/0 conn(0x7f2f900d1c70 legacy=0x7f2f900efc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:29.864 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:29.847 7f2f8f7fe700 1 -- 172.21.2.102:0/2268941598 >> v1:172.21.2.102:6789/0 conn(0x7f2f900d1870 legacy=0x7f2f901b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:29.864 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:29.847 7f2f8f7fe700 1 -- 172.21.2.102:0/2268941598 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f900f0450 con 0x7f2f900d0340 2021-02-10T11:33:29.865 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:29.847 7f2f8f7fe700 1 --2- 172.21.2.102:0/2268941598 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2f900d0340 0x7f2f900ef640 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7f2f84006650 tx=0x7f2f84009a80).ready entity=mon.2 client_cookie=61ac9ff711c34ed5 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:29.865 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:29.847 7f2f8effd700 1 -- 172.21.2.102:0/2268941598 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2f840009b0 con 0x7f2f900d0340 2021-02-10T11:33:29.865 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:29.847 7f2f8effd700 1 -- 172.21.2.102:0/2268941598 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f8400c0c0 con 0x7f2f900d0340 2021-02-10T11:33:29.865 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:29.847 7f2f8effd700 1 -- 172.21.2.102:0/2268941598 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2f8400c5e0 con 0x7f2f900d0340 2021-02-10T11:33:29.865 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:29.847 7f2f9667b700 1 -- 172.21.2.102:0/2268941598 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2f900d0340 msgr2=0x7f2f900ef640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:29.865 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:29.847 7f2f9667b700 1 --2- 172.21.2.102:0/2268941598 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2f900d0340 0x7f2f900ef640 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7f2f84006650 tx=0x7f2f84009a80).stop 2021-02-10T11:33:29.865 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:29.848 7f2f9667b700 1 -- 172.21.2.102:0/2268941598 shutdown_connections 2021-02-10T11:33:29.865 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:29.848 7f2f9667b700 1 --2- 172.21.2.102:0/2268941598 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2f900d0340 0x7f2f900ef640 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:29.865 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:29.848 7f2f9667b700 1 -- 172.21.2.102:0/2268941598 >> 172.21.2.102:0/2268941598 conn(0x7f2f900ce450 msgr2=0x7f2f900d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:29.865 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:29.848 7f2f9667b700 1 -- 172.21.2.102:0/2268941598 shutdown_connections 2021-02-10T11:33:29.865 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:29.848 7f2f9667b700 1 -- 172.21.2.102:0/2268941598 wait complete. 2021-02-10T11:33:29.867 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:29.863 7f2f9667b700 1 Processor -- start 2021-02-10T11:33:29.867 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:29.863 7f2f9667b700 1 -- start start 2021-02-10T11:33:29.867 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:29.864 7f2f9667b700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2f9018d870 0x7f2f900b19b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:29.867 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:29.864 7f2f9667b700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2f900b1fd0 0x7f2f90099ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:29.867 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:29.864 7f2f9667b700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2f900b23d0 0x7f2f9009a510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:29.867 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:29.864 7f2f9667b700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f9018dc70 con 0x7f2f900b23d0 2021-02-10T11:33:29.868 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:29.864 7f2f9667b700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f90199e60 con 0x7f2f900b1fd0 2021-02-10T11:33:29.868 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:29.864 7f2f9667b700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f9009ab30 con 0x7f2f9018d870 2021-02-10T11:33:29.868 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:29.864 7f2f8ffff700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2f9018d870 0x7f2f900b19b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:29.868 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:29.864 7f2f8f7fe700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2f900b1fd0 0x7f2f90099ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:29.868 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:29.864 7f2f8ffff700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2f9018d870 0x7f2f900b19b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.113:3301/0 says I am v2:172.21.2.102:40844/0 (socket says 172.21.2.102:40844) 2021-02-10T11:33:29.868 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:29.864 7f2f8ffff700 1 -- 172.21.2.102:0/1904485176 learned_addr learned my addr 172.21.2.102:0/1904485176 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:33:29.868 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:29.864 7f2f94c98700 1 --2- 172.21.2.102:0/1904485176 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2f900b23d0 0x7f2f9009a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:29.868 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:29.865 7f2f8ffff700 1 -- 172.21.2.102:0/1904485176 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2f900b1fd0 msgr2=0x7f2f90099ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:29.868 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:29.865 7f2f8ffff700 1 --2- 172.21.2.102:0/1904485176 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2f900b1fd0 0x7f2f90099ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:29.868 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:29.865 7f2f8ffff700 1 -- 172.21.2.102:0/1904485176 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2f900b23d0 msgr2=0x7f2f9009a510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:29.868 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:29.865 7f2f8ffff700 1 --2- 172.21.2.102:0/1904485176 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2f900b23d0 0x7f2f9009a510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:29.868 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:29.865 7f2f8ffff700 1 -- 172.21.2.102:0/1904485176 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f9009ad50 con 0x7f2f9018d870 2021-02-10T11:33:29.868 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:29.865 7f2f8f7fe700 1 --2- 172.21.2.102:0/1904485176 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2f900b1fd0 0x7f2f90099ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:33:29.869 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:29.865 7f2f8ffff700 1 --2- 172.21.2.102:0/1904485176 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2f9018d870 0x7f2f900b19b0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 rx=0x7f2f8000a540 tx=0x7f2f8000a8e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:29.869 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:29.865 7f2f8cff9700 1 -- 172.21.2.102:0/1904485176 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2f8000acd0 con 0x7f2f9018d870 2021-02-10T11:33:29.869 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:29.865 7f2f8cff9700 1 -- 172.21.2.102:0/1904485176 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f80000d50 con 0x7f2f9018d870 2021-02-10T11:33:29.869 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:29.865 7f2f8cff9700 1 -- 172.21.2.102:0/1904485176 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2f800045d0 con 0x7f2f9018d870 2021-02-10T11:33:29.869 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:29.865 7f2f9667b700 1 -- 172.21.2.102:0/1904485176 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f9009ad50 con 0x7f2f9018d870 2021-02-10T11:33:29.869 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:29.865 7f2f9667b700 1 -- 172.21.2.102:0/1904485176 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f9018e3f0 con 0x7f2f9018d870 2021-02-10T11:33:29.869 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:29.866 7f2f8cff9700 1 -- 172.21.2.102:0/1904485176 <== mon.2 v2:172.21.2.113:3301/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f2f80004140 con 0x7f2f9018d870 2021-02-10T11:33:29.869 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:29.866 7f2f8cff9700 1 --2- 172.21.2.102:0/1904485176 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2f700186c0 0x7f2f7001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:29.869 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:29.866 7f2f8cff9700 1 -- 172.21.2.102:0/1904485176 <== mon.2 v2:172.21.2.113:3301/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 4753+0+0 (secure 0 0 0) 0x7f2f800320e0 con 0x7f2f9018d870 2021-02-10T11:33:29.869 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:29.866 7f2f8f7fe700 1 --2- 172.21.2.102:0/1904485176 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2f700186c0 0x7f2f7001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:29.869 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:29.867 7f2f8f7fe700 1 --2- 172.21.2.102:0/1904485176 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2f700186c0 0x7f2f7001abc0 crc :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:29.896 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:29.896 7f2f9667b700 1 -- 172.21.2.102:0/1904485176 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f900999c0 con 0x7f2f9018d870 2021-02-10T11:33:29.900 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:29.900 7f2f8cff9700 1 -- 172.21.2.102:0/1904485176 <== mon.2 v2:172.21.2.113:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f2f80000960 con 0x7f2f9018d870 2021-02-10T11:33:30.027 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:30.026 7f2f9667b700 1 -- 172.21.2.102:0/1904485176 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f2f900999c0 con 0x7f2f9018d870 2021-02-10T11:33:30.027 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:30.027 7f2f8cff9700 1 -- 172.21.2.102:0/1904485176 <== mon.2 v2:172.21.2.113:3301/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v124) v1 ==== 84+0+64 (secure 0 0 0) 0x7f2f80004140 con 0x7f2f9018d870 2021-02-10T11:33:30.027 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:33:30.028 INFO:teuthology.orchestra.run.gibba002.stdout:[ 2021-02-10T11:33:30.028 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:33:30.028 INFO:teuthology.orchestra.run.gibba002.stdout: "poolnum": 1, 2021-02-10T11:33:30.028 INFO:teuthology.orchestra.run.gibba002.stdout: "poolname": "rbd" 2021-02-10T11:33:30.028 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:33:30.028 INFO:teuthology.orchestra.run.gibba002.stdout:] 2021-02-10T11:33:30.029 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:30.028 7f2f9667b700 1 -- 172.21.2.102:0/1904485176 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2f700186c0 msgr2=0x7f2f7001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:30.029 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:30.028 7f2f9667b700 1 --2- 172.21.2.102:0/1904485176 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2f700186c0 0x7f2f7001abc0 crc :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:30.029 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:30.028 7f2f9667b700 1 -- 172.21.2.102:0/1904485176 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2f9018d870 msgr2=0x7f2f900b19b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:30.029 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:30.028 7f2f9667b700 1 --2- 172.21.2.102:0/1904485176 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2f9018d870 0x7f2f900b19b0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 rx=0x7f2f8000a540 tx=0x7f2f8000a8e0).stop 2021-02-10T11:33:30.029 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:30.028 7f2f9667b700 1 -- 172.21.2.102:0/1904485176 shutdown_connections 2021-02-10T11:33:30.029 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:30.028 7f2f9667b700 1 --2- 172.21.2.102:0/1904485176 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2f900b23d0 0x7f2f9009a510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:30.029 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:30.028 7f2f9667b700 1 --2- 172.21.2.102:0/1904485176 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2f700186c0 0x7f2f7001abc0 unknown :-1 s=CLOSED pgs=1063 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:30.029 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:30.028 7f2f9667b700 1 --2- 172.21.2.102:0/1904485176 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2f900b1fd0 0x7f2f90099ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:30.029 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:30.028 7f2f9667b700 1 --2- 172.21.2.102:0/1904485176 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2f9018d870 0x7f2f900b19b0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:30.029 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:30.028 7f2f9667b700 1 -- 172.21.2.102:0/1904485176 >> 172.21.2.102:0/1904485176 conn(0x7f2f900f0550 msgr2=0x7f2f90191730 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:30.030 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:30.029 7f2f9667b700 1 -- 172.21.2.102:0/1904485176 shutdown_connections 2021-02-10T11:33:30.030 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:30.029 7f2f9667b700 1 -- 172.21.2.102:0/1904485176 wait complete. 2021-02-10T11:33:30.038 INFO:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2021-02-10T11:33:30.038 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool create cephfs_metadata 8 2021-02-10T11:33:30.471 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:30.454 7ff9f2739700 1 Processor -- start 2021-02-10T11:33:30.471 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:30.454 7ff9f2739700 1 -- start start 2021-02-10T11:33:30.471 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:30.454 7ff9f2739700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7ff9ec0d1870 0x7ff9ec1b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:30.471 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:30.454 7ff9f2739700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7ff9ec1b0ae0 0x7ff9ec108940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:30.471 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:30.454 7ff9f2739700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9ec0fe1c0 con 0x7ff9ec0d1c70 2021-02-10T11:33:30.471 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:30.454 7ff9f2739700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9ec109a60 con 0x7ff9ec0d1870 2021-02-10T11:33:30.471 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:30.455 7ff9f2739700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9ec109750 con 0x7ff9ec1b0ae0 2021-02-10T11:33:30.471 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:30.455 7ff9ebfff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7ff9ec0d1870 0x7ff9ec1b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:30.472 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:30.455 7ff9ebfff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7ff9ec0d1870 0x7ff9ec1b04c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:35316/0 (socket says 172.21.2.102:35316) 2021-02-10T11:33:30.472 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:30.455 7ff9ebfff700 1 -- 172.21.2.102:0/1524930106 learned_addr learned my addr 172.21.2.102:0/1524930106 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:33:30.472 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:30.455 7ff9ebfff700 1 -- 172.21.2.102:0/1524930106 >> v2:172.21.2.113:3300/0 conn(0x7ff9ec1b0ae0 msgr2=0x7ff9ec108940 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:33:30.472 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:30.455 7ff9ebfff700 1 --2- 172.21.2.102:0/1524930106 >> v2:172.21.2.113:3300/0 conn(0x7ff9ec1b0ae0 0x7ff9ec108940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:30.472 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:30.455 7ff9ebfff700 1 -- 172.21.2.102:0/1524930106 >> v1:172.21.2.113:6789/0 conn(0x7ff9ec0d1c70 legacy=0x7ff9ec108f60 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:33:30.472 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:30.455 7ff9ebfff700 1 -- 172.21.2.102:0/1524930106 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9ec088070 con 0x7ff9ec0d1870 2021-02-10T11:33:30.472 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:30.455 7ff9ebfff700 1 --2- 172.21.2.102:0/1524930106 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff9ec0d1870 0x7ff9ec1b04c0 secure :-1 s=READY pgs=2409 cs=0 l=1 rev1=1 rx=0x7ff9dc006700 tx=0x7ff9dc009b80).ready entity=mon.0 client_cookie=faf008595dac8d4 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:30.472 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:30.455 7ff9eaffd700 1 -- 172.21.2.102:0/1524930106 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff9dc00c0c0 con 0x7ff9ec0d1870 2021-02-10T11:33:30.472 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:30.455 7ff9eaffd700 1 -- 172.21.2.102:0/1524930106 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9dc00c400 con 0x7ff9ec0d1870 2021-02-10T11:33:30.472 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:30.455 7ff9eaffd700 1 -- 172.21.2.102:0/1524930106 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff9dc00c920 con 0x7ff9ec0d1870 2021-02-10T11:33:30.472 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:30.456 7ff9f2739700 1 -- 172.21.2.102:0/1524930106 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff9ec0d1870 msgr2=0x7ff9ec1b04c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:30.472 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:30.456 7ff9f2739700 1 --2- 172.21.2.102:0/1524930106 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff9ec0d1870 0x7ff9ec1b04c0 secure :-1 s=READY pgs=2409 cs=0 l=1 rev1=1 rx=0x7ff9dc006700 tx=0x7ff9dc009b80).stop 2021-02-10T11:33:30.472 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:30.456 7ff9f2739700 1 -- 172.21.2.102:0/1524930106 shutdown_connections 2021-02-10T11:33:30.472 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:30.456 7ff9f2739700 1 --2- 172.21.2.102:0/1524930106 >> v2:172.21.2.113:3300/0 conn(0x7ff9ec1b0ae0 0x7ff9ec108940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:30.473 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:30.456 7ff9f2739700 1 --2- 172.21.2.102:0/1524930106 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff9ec0d1870 0x7ff9ec1b04c0 unknown :-1 s=CLOSED pgs=2409 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:30.473 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:30.456 7ff9f2739700 1 -- 172.21.2.102:0/1524930106 >> 172.21.2.102:0/1524930106 conn(0x7ff9ec0ce420 msgr2=0x7ff9ec0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:30.473 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:30.456 7ff9f2739700 1 -- 172.21.2.102:0/1524930106 shutdown_connections 2021-02-10T11:33:30.473 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:30.456 7ff9f2739700 1 -- 172.21.2.102:0/1524930106 wait complete. 2021-02-10T11:33:30.474 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:30.471 7ff9f2739700 1 Processor -- start 2021-02-10T11:33:30.474 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:30.471 7ff9f2739700 1 -- start start 2021-02-10T11:33:30.474 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:30.471 7ff9f2739700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff9ec0c3820 0x7ff9ec084cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:30.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:30.471 7ff9f2739700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff9ec0a0290 0x7ff9ec0852d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:30.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:30.471 7ff9f2739700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff9ec086270 0x7ff9ec086670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:30.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:30.471 7ff9f2739700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9ec0c0700 con 0x7ff9ec086270 2021-02-10T11:33:30.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:30.471 7ff9f2739700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9ec0858f0 con 0x7ff9ec0a0290 2021-02-10T11:33:30.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:30.471 7ff9f2739700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9ec085b10 con 0x7ff9ec0c3820 2021-02-10T11:33:30.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:30.472 7ff9f0d56700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff9ec086270 0x7ff9ec086670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:30.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:30.472 7ff9f0d56700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff9ec086270 0x7ff9ec086670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:35322/0 (socket says 172.21.2.102:35322) 2021-02-10T11:33:30.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:30.472 7ff9f0d56700 1 -- 172.21.2.102:0/3200956798 learned_addr learned my addr 172.21.2.102:0/3200956798 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:33:30.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:30.472 7ff9f0d56700 1 -- 172.21.2.102:0/3200956798 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff9ec0c3820 msgr2=0x7ff9ec084cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:30.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:30.472 7ff9f0d56700 1 --2- 172.21.2.102:0/3200956798 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff9ec0c3820 0x7ff9ec084cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:30.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:30.472 7ff9f0d56700 1 -- 172.21.2.102:0/3200956798 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff9ec0a0290 msgr2=0x7ff9ec0852d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:30.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:30.472 7ff9f0d56700 1 --2- 172.21.2.102:0/3200956798 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff9ec0a0290 0x7ff9ec0852d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:30.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:30.472 7ff9f0d56700 1 -- 172.21.2.102:0/3200956798 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9ec086c90 con 0x7ff9ec086270 2021-02-10T11:33:30.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:30.472 7ff9f0d56700 1 --2- 172.21.2.102:0/3200956798 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff9ec086270 0x7ff9ec086670 secure :-1 s=READY pgs=2410 cs=0 l=1 rev1=1 rx=0x7ff9e4009940 tx=0x7ff9e400bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:30.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:30.472 7ff9e8ff9700 1 -- 172.21.2.102:0/3200956798 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff9e4003030 con 0x7ff9ec086270 2021-02-10T11:33:30.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:30.472 7ff9e8ff9700 1 -- 172.21.2.102:0/3200956798 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9e4000d30 con 0x7ff9ec086270 2021-02-10T11:33:30.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:30.472 7ff9e8ff9700 1 -- 172.21.2.102:0/3200956798 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff9e40014f0 con 0x7ff9ec086270 2021-02-10T11:33:30.477 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:30.472 7ff9f2739700 1 -- 172.21.2.102:0/3200956798 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9ec086c90 con 0x7ff9ec086270 2021-02-10T11:33:30.477 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:30.473 7ff9f2739700 1 -- 172.21.2.102:0/3200956798 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9ec1a2f00 con 0x7ff9ec086270 2021-02-10T11:33:30.477 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:30.473 7ff9e8ff9700 1 -- 172.21.2.102:0/3200956798 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ff9e400a000 con 0x7ff9ec086270 2021-02-10T11:33:30.477 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:30.473 7ff9e8ff9700 1 --2- 172.21.2.102:0/3200956798 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff9cc0186c0 0x7ff9cc01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:30.477 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:30.473 7ff9e8ff9700 1 -- 172.21.2.102:0/3200956798 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 4753+0+0 (secure 0 0 0) 0x7ff9e40325e0 con 0x7ff9ec086270 2021-02-10T11:33:30.477 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:30.473 7ff9ebfff700 1 --2- 172.21.2.102:0/3200956798 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff9cc0186c0 0x7ff9cc01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:30.477 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:30.474 7ff9ebfff700 1 --2- 172.21.2.102:0/3200956798 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff9cc0186c0 0x7ff9cc01abc0 crc :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:30.505 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:30.505 7ff9f2739700 1 -- 172.21.2.102:0/3200956798 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9ec082a70 con 0x7ff9ec086270 2021-02-10T11:33:30.508 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:30.507 7ff9e8ff9700 1 -- 172.21.2.102:0/3200956798 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ff9e4000f90 con 0x7ff9ec086270 2021-02-10T11:33:30.636 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:30.635 7ff9f2739700 1 -- 172.21.2.102:0/3200956798 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"} v 0) v1 -- 0x7ff9ec082a70 con 0x7ff9ec086270 2021-02-10T11:33:31.202 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:31.201 7ff9e8ff9700 1 -- 172.21.2.102:0/3200956798 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"}]=0 pool 'cephfs_metadata' created v125) v1 ==== 133+0+0 (secure 0 0 0) 0x7ff9e400a000 con 0x7ff9ec086270 2021-02-10T11:33:31.202 INFO:teuthology.orchestra.run.gibba002.stderr:pool 'cephfs_metadata' created 2021-02-10T11:33:31.205 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:31.204 7ff9f2739700 1 -- 172.21.2.102:0/3200956798 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff9cc0186c0 msgr2=0x7ff9cc01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:31.205 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:31.204 7ff9f2739700 1 --2- 172.21.2.102:0/3200956798 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff9cc0186c0 0x7ff9cc01abc0 crc :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:31.205 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:31.204 7ff9f2739700 1 -- 172.21.2.102:0/3200956798 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff9ec086270 msgr2=0x7ff9ec086670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:31.205 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:31.204 7ff9f2739700 1 --2- 172.21.2.102:0/3200956798 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff9ec086270 0x7ff9ec086670 secure :-1 s=READY pgs=2410 cs=0 l=1 rev1=1 rx=0x7ff9e4009940 tx=0x7ff9e400bcf0).stop 2021-02-10T11:33:31.205 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:31.204 7ff9f2739700 1 -- 172.21.2.102:0/3200956798 shutdown_connections 2021-02-10T11:33:31.205 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:31.204 7ff9f2739700 1 --2- 172.21.2.102:0/3200956798 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff9ec086270 0x7ff9ec086670 unknown :-1 s=CLOSED pgs=2410 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:31.205 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:31.204 7ff9f2739700 1 --2- 172.21.2.102:0/3200956798 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff9cc0186c0 0x7ff9cc01abc0 unknown :-1 s=CLOSED pgs=1064 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:31.205 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:31.204 7ff9f2739700 1 --2- 172.21.2.102:0/3200956798 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff9ec0a0290 0x7ff9ec0852d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:31.205 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:31.204 7ff9f2739700 1 --2- 172.21.2.102:0/3200956798 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff9ec0c3820 0x7ff9ec084cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:31.205 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:31.204 7ff9f2739700 1 -- 172.21.2.102:0/3200956798 >> 172.21.2.102:0/3200956798 conn(0x7ff9ec0872c0 msgr2=0x7ff9ec095700 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:31.206 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:31.204 7ff9f2739700 1 -- 172.21.2.102:0/3200956798 shutdown_connections 2021-02-10T11:33:31.206 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:31.204 7ff9f2739700 1 -- 172.21.2.102:0/3200956798 wait complete. 2021-02-10T11:33:31.213 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool create cephfs_data 8 2021-02-10T11:33:31.649 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:31.631 7f613e4aa700 1 Processor -- start 2021-02-10T11:33:31.649 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:31.632 7f613e4aa700 1 -- start start 2021-02-10T11:33:31.650 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:31.632 7f613e4aa700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f613810aff0 0x7f61380cf750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:31.650 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:31.632 7f613e4aa700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61380fe1c0 con 0x7f61380d1c70 2021-02-10T11:33:31.650 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:31.632 7f613e4aa700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61381b0cb0 con 0x7f61380d1870 2021-02-10T11:33:31.650 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:31.632 7f613e4aa700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61380d2fe0 con 0x7f613810aff0 2021-02-10T11:33:31.650 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:31.632 7f61377fe700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f61380d1c70 0x7f61380cef60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:55790/0 (socket says 172.21.2.102:55790) 2021-02-10T11:33:31.650 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:31.632 7f61377fe700 1 -- 172.21.2.102:0/4896489 learned_addr learned my addr 172.21.2.102:0/4896489 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:33:31.650 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:31.632 7f613cac7700 1 --2- 172.21.2.102:0/4896489 >> v2:172.21.2.113:3300/0 conn(0x7f613810aff0 0x7f61380cf750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:31.650 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:31.633 7f6136ffd700 1 -- 172.21.2.102:0/4896489 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1599756069 0 0) 0x7f612c0009b0 con 0x7f61380d1c70 2021-02-10T11:33:31.650 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:31.633 7f6136ffd700 1 -- 172.21.2.102:0/4896489 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6120002fe0 con 0x7f61380d1c70 2021-02-10T11:33:31.650 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:31.633 7f6136ffd700 1 -- 172.21.2.102:0/4896489 <== mon.1 v1:172.21.2.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3403359018 0 0) 0x7f6130000fe0 con 0x7f61380d1870 2021-02-10T11:33:31.650 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:31.633 7f6136ffd700 1 -- 172.21.2.102:0/4896489 --> v1:172.21.2.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6120004370 con 0x7f61380d1870 2021-02-10T11:33:31.650 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:31.633 7f613cac7700 1 -- 172.21.2.102:0/4896489 >> v1:172.21.2.113:6789/0 conn(0x7f61380d1870 legacy=0x7f61381b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:31.650 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:31.633 7f613cac7700 1 -- 172.21.2.102:0/4896489 >> v1:172.21.2.102:6789/0 conn(0x7f61380d1c70 legacy=0x7f61380cef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:31.651 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:31.633 7f613cac7700 1 -- 172.21.2.102:0/4896489 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6138109a60 con 0x7f613810aff0 2021-02-10T11:33:31.651 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:31.633 7f613cac7700 1 --2- 172.21.2.102:0/4896489 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f613810aff0 0x7f61380cf750 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 rx=0x7f6128006ad0 tx=0x7f6128009ad0).ready entity=mon.1 client_cookie=4c62c56d1f952d75 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:31.651 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:31.633 7f6136ffd700 1 -- 172.21.2.102:0/4896489 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f61280022e0 con 0x7f613810aff0 2021-02-10T11:33:31.651 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:31.633 7f6136ffd700 1 -- 172.21.2.102:0/4896489 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6128006410 con 0x7f613810aff0 2021-02-10T11:33:31.651 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:31.633 7f6136ffd700 1 -- 172.21.2.102:0/4896489 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6128002620 con 0x7f613810aff0 2021-02-10T11:33:31.651 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:31.633 7f613e4aa700 1 -- 172.21.2.102:0/4896489 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f613810aff0 msgr2=0x7f61380cf750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:31.651 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:31.633 7f613e4aa700 1 --2- 172.21.2.102:0/4896489 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f613810aff0 0x7f61380cf750 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 rx=0x7f6128006ad0 tx=0x7f6128009ad0).stop 2021-02-10T11:33:31.651 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:31.634 7f613e4aa700 1 -- 172.21.2.102:0/4896489 shutdown_connections 2021-02-10T11:33:31.651 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:31.634 7f613e4aa700 1 --2- 172.21.2.102:0/4896489 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f613810aff0 0x7f61380cf750 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:31.651 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:31.634 7f613e4aa700 1 -- 172.21.2.102:0/4896489 >> 172.21.2.102:0/4896489 conn(0x7f61380ce420 msgr2=0x7f61380d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:31.651 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:31.634 7f613e4aa700 1 -- 172.21.2.102:0/4896489 shutdown_connections 2021-02-10T11:33:31.651 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:31.634 7f613e4aa700 1 -- 172.21.2.102:0/4896489 wait complete. 2021-02-10T11:33:31.651 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:31.649 7f613e4aa700 1 Processor -- start 2021-02-10T11:33:31.652 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:31.649 7f613e4aa700 1 -- start start 2021-02-10T11:33:31.653 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:31.649 7f613e4aa700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6138191b10 0x7f61380ae640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:31.653 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:31.649 7f613e4aa700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6138191f10 0x7f613819ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:31.653 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:31.649 7f613e4aa700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f61381a05f0 0x7f6138082ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:31.654 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:31.649 7f613e4aa700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6138192440 con 0x7f6138191b10 2021-02-10T11:33:31.654 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:31.649 7f613e4aa700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61381916e0 con 0x7f61381a05f0 2021-02-10T11:33:31.654 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:31.649 7f613e4aa700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6138191900 con 0x7f6138191f10 2021-02-10T11:33:31.654 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:31.649 7f6137fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6138191b10 0x7f61380ae640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:31.654 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:31.649 7f6137fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6138191b10 0x7f61380ae640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:35338/0 (socket says 172.21.2.102:35338) 2021-02-10T11:33:31.654 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:31.649 7f6137fff700 1 -- 172.21.2.102:0/2991152914 learned_addr learned my addr 172.21.2.102:0/2991152914 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:33:31.654 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:31.650 7f613cac7700 1 --2- 172.21.2.102:0/2991152914 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f61381a05f0 0x7f6138082ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:31.654 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:31.650 7f61377fe700 1 --2- 172.21.2.102:0/2991152914 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6138191f10 0x7f613819ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:31.654 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:31.650 7f6137fff700 1 -- 172.21.2.102:0/2991152914 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6138191f10 msgr2=0x7f613819ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:31.654 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:31.650 7f6137fff700 1 --2- 172.21.2.102:0/2991152914 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6138191f10 0x7f613819ffd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:31.654 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:31.650 7f6137fff700 1 -- 172.21.2.102:0/2991152914 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f61381a05f0 msgr2=0x7f6138082ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:31.654 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:31.650 7f6137fff700 1 --2- 172.21.2.102:0/2991152914 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f61381a05f0 0x7f6138082ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:31.655 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:31.650 7f6137fff700 1 -- 172.21.2.102:0/2991152914 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61380834f0 con 0x7f6138191b10 2021-02-10T11:33:31.655 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:31.650 7f613cac7700 1 --2- 172.21.2.102:0/2991152914 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f61381a05f0 0x7f6138082ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:33:31.655 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:31.650 7f6137fff700 1 --2- 172.21.2.102:0/2991152914 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6138191b10 0x7f61380ae640 secure :-1 s=READY pgs=2412 cs=0 l=1 rev1=1 rx=0x7f6130009980 tx=0x7f613000bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:31.655 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:31.650 7f6134ff9700 1 -- 172.21.2.102:0/2991152914 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f613000be10 con 0x7f6138191b10 2021-02-10T11:33:31.655 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:31.650 7f6134ff9700 1 -- 172.21.2.102:0/2991152914 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f613000c2f0 con 0x7f6138191b10 2021-02-10T11:33:31.655 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:31.650 7f6134ff9700 1 -- 172.21.2.102:0/2991152914 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f613000c790 con 0x7f6138191b10 2021-02-10T11:33:31.655 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:31.650 7f613e4aa700 1 -- 172.21.2.102:0/2991152914 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61380834f0 con 0x7f6138191b10 2021-02-10T11:33:31.655 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:31.650 7f613e4aa700 1 -- 172.21.2.102:0/2991152914 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6138083a80 con 0x7f6138191b10 2021-02-10T11:33:31.655 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:31.651 7f6134ff9700 1 -- 172.21.2.102:0/2991152914 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f6130003070 con 0x7f6138191b10 2021-02-10T11:33:31.655 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:31.651 7f6134ff9700 1 --2- 172.21.2.102:0/2991152914 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f61180186c0 0x7f611801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:31.655 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:31.651 7f6134ff9700 1 -- 172.21.2.102:0/2991152914 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5107+0+0 (secure 0 0 0) 0x7f6130032780 con 0x7f6138191b10 2021-02-10T11:33:31.655 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:31.651 7f61377fe700 1 --2- 172.21.2.102:0/2991152914 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f61180186c0 0x7f611801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:31.655 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:31.651 7f61377fe700 1 --2- 172.21.2.102:0/2991152914 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f61180186c0 0x7f611801abc0 crc :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:31.683 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:31.682 7f613e4aa700 1 -- 172.21.2.102:0/2991152914 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6138191900 con 0x7f6138191b10 2021-02-10T11:33:31.685 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:31.685 7f6134ff9700 1 -- 172.21.2.102:0/2991152914 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f6130021da0 con 0x7f6138191b10 2021-02-10T11:33:31.815 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:31.814 7f613e4aa700 1 -- 172.21.2.102:0/2991152914 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_data"} v 0) v1 -- 0x7f6138191900 con 0x7f6138191b10 2021-02-10T11:33:32.208 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:32.208 7f6134ff9700 1 -- 172.21.2.102:0/2991152914 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_data"}]=0 pool 'cephfs_data' created v126) v1 ==== 125+0+0 (secure 0 0 0) 0x7f6130021da0 con 0x7f6138191b10 2021-02-10T11:33:32.231 INFO:teuthology.orchestra.run.gibba002.stderr:pool 'cephfs_data' created 2021-02-10T11:33:32.232 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:32.232 7f613e4aa700 1 -- 172.21.2.102:0/2991152914 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f61180186c0 msgr2=0x7f611801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:32.232 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:32.232 7f613e4aa700 1 --2- 172.21.2.102:0/2991152914 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f61180186c0 0x7f611801abc0 crc :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:32.232 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:32.232 7f613e4aa700 1 -- 172.21.2.102:0/2991152914 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6138191b10 msgr2=0x7f61380ae640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:32.232 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:32.232 7f613e4aa700 1 --2- 172.21.2.102:0/2991152914 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6138191b10 0x7f61380ae640 secure :-1 s=READY pgs=2412 cs=0 l=1 rev1=1 rx=0x7f6130009980 tx=0x7f613000bcd0).stop 2021-02-10T11:33:32.233 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:32.232 7f613e4aa700 1 -- 172.21.2.102:0/2991152914 shutdown_connections 2021-02-10T11:33:32.233 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:32.232 7f613e4aa700 1 --2- 172.21.2.102:0/2991152914 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f61180186c0 0x7f611801abc0 unknown :-1 s=CLOSED pgs=1067 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:32.233 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:32.232 7f613e4aa700 1 --2- 172.21.2.102:0/2991152914 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f61381a05f0 0x7f6138082ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:32.233 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:32.232 7f613e4aa700 1 --2- 172.21.2.102:0/2991152914 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6138191f10 0x7f613819ffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:32.233 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:32.232 7f613e4aa700 1 --2- 172.21.2.102:0/2991152914 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6138191b10 0x7f61380ae640 unknown :-1 s=CLOSED pgs=2412 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:32.233 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:32.232 7f613e4aa700 1 -- 172.21.2.102:0/2991152914 >> 172.21.2.102:0/2991152914 conn(0x7f61380a02d0 msgr2=0x7f613809f200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:32.233 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:32.232 7f613e4aa700 1 -- 172.21.2.102:0/2991152914 shutdown_connections 2021-02-10T11:33:32.233 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:32.232 7f613e4aa700 1 -- 172.21.2.102:0/2991152914 wait complete. 2021-02-10T11:33:32.241 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2021-02-10T11:33:32.681 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:32.659 7f3bf6a34700 1 Processor -- start 2021-02-10T11:33:32.681 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:32.659 7f3bf6a34700 1 -- start start 2021-02-10T11:33:32.682 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:32.659 7f3bf6a34700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f3bf00d1880 0x7f3bf01b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:32.682 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:32.659 7f3bf6a34700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f3bf01b0af0 0x7f3bf0108950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:32.682 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:32.659 7f3bf6a34700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3bf00fe1d0 con 0x7f3bf00d1c80 2021-02-10T11:33:32.682 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:32.659 7f3bf6a34700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3bf0109a70 con 0x7f3bf00d1880 2021-02-10T11:33:32.682 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:32.659 7f3bf6a34700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3bf0109760 con 0x7f3bf01b0af0 2021-02-10T11:33:32.682 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:32.659 7f3bf4850700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f3bf00d1880 0x7f3bf01b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:32.682 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:32.659 7f3bf5051700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f3bf00d1c80 0x7f3bf0108f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:55802/0 (socket says 172.21.2.102:55802) 2021-02-10T11:33:32.682 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:32.659 7f3bf5051700 1 -- 172.21.2.102:0/3046985100 learned_addr learned my addr 172.21.2.102:0/3046985100 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:33:32.682 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:32.660 7f3beffff700 1 --2- 172.21.2.102:0/3046985100 >> v2:172.21.2.113:3300/0 conn(0x7f3bf01b0af0 0x7f3bf0108950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:32.682 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:32.660 7f3bf4850700 1 -- 172.21.2.102:0/3046985100 >> v2:172.21.2.113:3300/0 conn(0x7f3bf01b0af0 msgr2=0x7f3bf0108950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:33:32.682 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:32.660 7f3bf4850700 1 --2- 172.21.2.102:0/3046985100 >> v2:172.21.2.113:3300/0 conn(0x7f3bf01b0af0 0x7f3bf0108950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:32.683 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:32.660 7f3bf4850700 1 -- 172.21.2.102:0/3046985100 >> v1:172.21.2.102:6789/0 conn(0x7f3bf00d1c80 legacy=0x7f3bf0108f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:32.683 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:32.660 7f3bef7fe700 1 -- 172.21.2.102:0/3046985100 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4158124793 0 0) 0x7f3be8000f90 con 0x7f3bf00d1c80 2021-02-10T11:33:32.683 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:32.660 7f3bf4850700 1 -- 172.21.2.102:0/3046985100 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bf0088070 con 0x7f3bf00d1880 2021-02-10T11:33:32.683 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:32.660 7f3bf4850700 1 --2- 172.21.2.102:0/3046985100 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3bf00d1880 0x7f3bf01b04d0 secure :-1 s=READY pgs=2414 cs=0 l=1 rev1=1 rx=0x7f3bd8006650 tx=0x7f3bd8009a00).ready entity=mon.0 client_cookie=830fdee301a6e899 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:32.683 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:32.661 7f3bef7fe700 1 -- 172.21.2.102:0/3046985100 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3bd8009df0 con 0x7f3bf00d1880 2021-02-10T11:33:32.683 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:32.661 7f3bef7fe700 1 -- 172.21.2.102:0/3046985100 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bd80009b0 con 0x7f3bf00d1880 2021-02-10T11:33:32.683 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:32.661 7f3bef7fe700 1 -- 172.21.2.102:0/3046985100 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3bd800c2b0 con 0x7f3bf00d1880 2021-02-10T11:33:32.683 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:32.661 7f3bf6a34700 1 -- 172.21.2.102:0/3046985100 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3bf00d1880 msgr2=0x7f3bf01b04d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:32.683 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:32.661 7f3bf6a34700 1 --2- 172.21.2.102:0/3046985100 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3bf00d1880 0x7f3bf01b04d0 secure :-1 s=READY pgs=2414 cs=0 l=1 rev1=1 rx=0x7f3bd8006650 tx=0x7f3bd8009a00).stop 2021-02-10T11:33:32.683 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:32.661 7f3bf6a34700 1 -- 172.21.2.102:0/3046985100 shutdown_connections 2021-02-10T11:33:32.683 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:32.661 7f3bf6a34700 1 --2- 172.21.2.102:0/3046985100 >> v2:172.21.2.113:3300/0 conn(0x7f3bf01b0af0 0x7f3bf0108950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:32.683 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:32.661 7f3bf6a34700 1 --2- 172.21.2.102:0/3046985100 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3bf00d1880 0x7f3bf01b04d0 unknown :-1 s=CLOSED pgs=2414 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:32.683 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:32.661 7f3bf6a34700 1 -- 172.21.2.102:0/3046985100 >> 172.21.2.102:0/3046985100 conn(0x7f3bf00ce430 msgr2=0x7f3bf00d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:32.683 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:32.661 7f3bf6a34700 1 -- 172.21.2.102:0/3046985100 shutdown_connections 2021-02-10T11:33:32.684 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:32.661 7f3bf6a34700 1 -- 172.21.2.102:0/3046985100 wait complete. 2021-02-10T11:33:32.684 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:32.676 7f3bf6a34700 1 Processor -- start 2021-02-10T11:33:32.684 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:32.676 7f3bf6a34700 1 -- start start 2021-02-10T11:33:32.684 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:32.677 7f3bf6a34700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3bf00aef00 0x7f3bf0099c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:32.684 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:32.677 7f3bf6a34700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3bf009a260 0x7f3bf009a660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:32.684 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:32.677 7f3bf6a34700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3bf009ac80 0x7f3bf019a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:32.684 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:32.677 7f3bf6a34700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bf019bdd0 con 0x7f3bf00aef00 2021-02-10T11:33:32.684 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:32.677 7f3bf6a34700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bf019de70 con 0x7f3bf009a260 2021-02-10T11:33:32.684 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:32.677 7f3bf6a34700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3bf019a2d0 con 0x7f3bf009ac80 2021-02-10T11:33:32.684 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:32.677 7f3bf4850700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3bf00aef00 0x7f3bf0099c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:32.684 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:32.677 7f3bf4850700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3bf00aef00 0x7f3bf0099c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:35352/0 (socket says 172.21.2.102:35352) 2021-02-10T11:33:32.684 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:32.677 7f3bf4850700 1 -- 172.21.2.102:0/3501452823 learned_addr learned my addr 172.21.2.102:0/3501452823 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:33:32.684 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:32.677 7f3bf4850700 1 -- 172.21.2.102:0/3501452823 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3bf009ac80 msgr2=0x7f3bf019a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:32.685 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:32.677 7f3bf4850700 1 --2- 172.21.2.102:0/3501452823 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3bf009ac80 0x7f3bf019a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:32.685 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:32.677 7f3bf4850700 1 -- 172.21.2.102:0/3501452823 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3bf009a260 msgr2=0x7f3bf009a660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:32.685 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:32.677 7f3bf4850700 1 --2- 172.21.2.102:0/3501452823 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3bf009a260 0x7f3bf009a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:32.685 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:32.677 7f3bf4850700 1 -- 172.21.2.102:0/3501452823 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bf0199f50 con 0x7f3bf00aef00 2021-02-10T11:33:32.685 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:32.677 7f3bf4850700 1 --2- 172.21.2.102:0/3501452823 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3bf00aef00 0x7f3bf0099c40 secure :-1 s=READY pgs=2415 cs=0 l=1 rev1=1 rx=0x7f3bd80096a0 tx=0x7f3bd800b9b0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:32.685 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:32.677 7f3bed7fa700 1 -- 172.21.2.102:0/3501452823 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3bd80093a0 con 0x7f3bf00aef00 2021-02-10T11:33:32.685 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:32.677 7f3bed7fa700 1 -- 172.21.2.102:0/3501452823 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bd80067f0 con 0x7f3bf00aef00 2021-02-10T11:33:32.685 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:32.677 7f3bf6a34700 1 -- 172.21.2.102:0/3501452823 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3bf0199f50 con 0x7f3bf00aef00 2021-02-10T11:33:32.685 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:32.677 7f3bed7fa700 1 -- 172.21.2.102:0/3501452823 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3bd80022b0 con 0x7f3bf00aef00 2021-02-10T11:33:32.685 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:32.677 7f3bf6a34700 1 -- 172.21.2.102:0/3501452823 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3bf019b160 con 0x7f3bf00aef00 2021-02-10T11:33:32.685 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:32.678 7f3bed7fa700 1 -- 172.21.2.102:0/3501452823 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f3bd8000c90 con 0x7f3bf00aef00 2021-02-10T11:33:32.685 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:32.678 7f3bed7fa700 1 --2- 172.21.2.102:0/3501452823 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3bd40186c0 0x7f3bd401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:32.686 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:32.678 7f3bed7fa700 1 -- 172.21.2.102:0/3501452823 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 5457+0+0 (secure 0 0 0) 0x7f3bd80324e0 con 0x7f3bf00aef00 2021-02-10T11:33:32.686 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:32.678 7f3beffff700 1 --2- 172.21.2.102:0/3501452823 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3bd40186c0 0x7f3bd401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:32.686 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:32.678 7f3beffff700 1 --2- 172.21.2.102:0/3501452823 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3bd40186c0 0x7f3bd401abc0 crc :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:32.710 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:32.710 7f3bf6a34700 1 -- 172.21.2.102:0/3501452823 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3bf0082a70 con 0x7f3bf00aef00 2021-02-10T11:33:32.713 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:32.713 7f3bed7fa700 1 -- 172.21.2.102:0/3501452823 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f3bd8020610 con 0x7f3bf00aef00 2021-02-10T11:33:32.844 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:32.843 7f3bf6a34700 1 -- 172.21.2.102:0/3501452823 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"data": "cephfs_data", "prefix": "fs new", "force": true, "fs_name": "cephfs", "metadata": "cephfs_metadata"} v 0) v1 -- 0x7f3bf0199f50 con 0x7f3bf00aef00 2021-02-10T11:33:32.848 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:32.848 7f3bed7fa700 1 -- 172.21.2.102:0/3501452823 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"data": "cephfs_data", "prefix": "fs new", "force": true, "fs_name": "cephfs", "metadata": "cephfs_metadata"}]=0 new fs with metadata pool 26 and data pool 27 v104) v1 ==== 189+0+0 (secure 0 0 0) 0x7f3bd8020610 con 0x7f3bf00aef00 2021-02-10T11:33:32.851 INFO:teuthology.orchestra.run.gibba002.stderr:new fs with metadata pool 26 and data pool 27 2021-02-10T11:33:32.853 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:32.852 7f3bf6a34700 1 -- 172.21.2.102:0/3501452823 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3bd40186c0 msgr2=0x7f3bd401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:32.853 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:32.852 7f3bf6a34700 1 --2- 172.21.2.102:0/3501452823 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3bd40186c0 0x7f3bd401abc0 crc :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:32.853 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:32.852 7f3bf6a34700 1 -- 172.21.2.102:0/3501452823 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3bf00aef00 msgr2=0x7f3bf0099c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:32.853 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:32.852 7f3bf6a34700 1 --2- 172.21.2.102:0/3501452823 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3bf00aef00 0x7f3bf0099c40 secure :-1 s=READY pgs=2415 cs=0 l=1 rev1=1 rx=0x7f3bd80096a0 tx=0x7f3bd800b9b0).stop 2021-02-10T11:33:32.853 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:32.853 7f3bf6a34700 1 -- 172.21.2.102:0/3501452823 shutdown_connections 2021-02-10T11:33:32.853 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:32.853 7f3bf6a34700 1 --2- 172.21.2.102:0/3501452823 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3bf009ac80 0x7f3bf019a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:32.853 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:32.853 7f3bf6a34700 1 --2- 172.21.2.102:0/3501452823 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3bd40186c0 0x7f3bd401abc0 unknown :-1 s=CLOSED pgs=1068 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:32.854 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:32.853 7f3bf6a34700 1 --2- 172.21.2.102:0/3501452823 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3bf009a260 0x7f3bf009a660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:32.854 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:32.853 7f3bf6a34700 1 --2- 172.21.2.102:0/3501452823 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3bf00aef00 0x7f3bf0099c40 unknown :-1 s=CLOSED pgs=2415 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:32.854 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:32.853 7f3bf6a34700 1 -- 172.21.2.102:0/3501452823 >> 172.21.2.102:0/3501452823 conn(0x7f3bf018e810 msgr2=0x7f3bf018c290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:32.854 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:32.853 7f3bf6a34700 1 -- 172.21.2.102:0/3501452823 shutdown_connections 2021-02-10T11:33:32.854 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:32.853 7f3bf6a34700 1 -- 172.21.2.102:0/3501452823 wait complete. 2021-02-10T11:33:32.861 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:33:33.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:33.271 7f237f182700 1 Processor -- start 2021-02-10T11:33:33.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:33.271 7f237f182700 1 -- start start 2021-02-10T11:33:33.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:33.272 7f237f182700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f23780d1870 0x7f23781b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:33.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:33.272 7f237f182700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f23780f0650 0x7f23780cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:33.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:33.272 7f237f182700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23780f0b10 con 0x7f23781b0a10 2021-02-10T11:33:33.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:33.272 7f237f182700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23780e5ed0 con 0x7f23780d1870 2021-02-10T11:33:33.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:33.272 7f237f182700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23780d1c70 con 0x7f23780f0650 2021-02-10T11:33:33.289 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:33.272 7f237cf9e700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f23780d1870 0x7f23781b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:33.289 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:33.272 7f237cf9e700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f23780d1870 0x7f23781b03f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:35358/0 (socket says 172.21.2.102:35358) 2021-02-10T11:33:33.289 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:33.272 7f237cf9e700 1 -- 172.21.2.102:0/1632060771 learned_addr learned my addr 172.21.2.102:0/1632060771 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:33:33.289 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:33.272 7f237cf9e700 1 -- 172.21.2.102:0/1632060771 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f23780f0650 msgr2=0x7f23780cfb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:33:33.289 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:33.272 7f237cf9e700 1 --2- 172.21.2.102:0/1632060771 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f23780f0650 0x7f23780cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:33.289 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:33.272 7f237cf9e700 1 -- 172.21.2.102:0/1632060771 >> v1:172.21.2.113:6789/0 conn(0x7f23781b0a10 legacy=0x7f23780cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:33:33.289 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:33.272 7f237cf9e700 1 -- 172.21.2.102:0/1632060771 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23780d1e90 con 0x7f23780d1870 2021-02-10T11:33:33.289 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:33.273 7f237cf9e700 1 --2- 172.21.2.102:0/1632060771 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f23780d1870 0x7f23781b03f0 secure :-1 s=READY pgs=2416 cs=0 l=1 rev1=1 rx=0x7f2360006700 tx=0x7f2360009b80).ready entity=mon.0 client_cookie=5d5179412a9af7c1 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:33.289 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:33.273 7f23777fe700 1 -- 172.21.2.102:0/1632060771 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f236000c0c0 con 0x7f23780d1870 2021-02-10T11:33:33.289 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:33.273 7f23777fe700 1 -- 172.21.2.102:0/1632060771 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f236000c400 con 0x7f23780d1870 2021-02-10T11:33:33.289 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:33.273 7f23777fe700 1 -- 172.21.2.102:0/1632060771 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f236000c8e0 con 0x7f23780d1870 2021-02-10T11:33:33.289 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:33.273 7f237f182700 1 -- 172.21.2.102:0/1632060771 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f23780d1870 msgr2=0x7f23781b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:33.289 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:33.273 7f237f182700 1 --2- 172.21.2.102:0/1632060771 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f23780d1870 0x7f23781b03f0 secure :-1 s=READY pgs=2416 cs=0 l=1 rev1=1 rx=0x7f2360006700 tx=0x7f2360009b80).stop 2021-02-10T11:33:33.290 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:33.273 7f237f182700 1 -- 172.21.2.102:0/1632060771 shutdown_connections 2021-02-10T11:33:33.290 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:33.273 7f237f182700 1 --2- 172.21.2.102:0/1632060771 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f23780f0650 0x7f23780cfb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:33.290 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:33.273 7f237f182700 1 --2- 172.21.2.102:0/1632060771 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f23780d1870 0x7f23781b03f0 unknown :-1 s=CLOSED pgs=2416 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:33.290 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:33.273 7f237f182700 1 -- 172.21.2.102:0/1632060771 >> 172.21.2.102:0/1632060771 conn(0x7f23780ce450 msgr2=0x7f23780d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:33.290 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:33.273 7f237f182700 1 -- 172.21.2.102:0/1632060771 shutdown_connections 2021-02-10T11:33:33.290 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:33.273 7f237f182700 1 -- 172.21.2.102:0/1632060771 wait complete. 2021-02-10T11:33:33.291 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:33.288 7f237f182700 1 Processor -- start 2021-02-10T11:33:33.292 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:33.288 7f237f182700 1 -- start start 2021-02-10T11:33:33.292 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:33.288 7f237f182700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f23780b4f30 0x7f237818b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:33.292 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:33.288 7f237f182700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2378093f40 0x7f2378094340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:33.292 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:33.288 7f237f182700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2378094960 0x7f2378187e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:33.292 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:33.288 7f237f182700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2378094e90 con 0x7f2378093f40 2021-02-10T11:33:33.292 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:33.288 7f237f182700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f237818bff0 con 0x7f23780b4f30 2021-02-10T11:33:33.292 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:33.288 7f237f182700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f237808f780 con 0x7f2378094960 2021-02-10T11:33:33.292 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:33.288 7f2377fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2378093f40 0x7f2378094340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:33.292 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:33.289 7f2377fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2378093f40 0x7f2378094340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:35364/0 (socket says 172.21.2.102:35364) 2021-02-10T11:33:33.292 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:33.289 7f2377fff700 1 -- 172.21.2.102:0/1912248618 learned_addr learned my addr 172.21.2.102:0/1912248618 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:33:33.292 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:33.289 7f2377fff700 1 -- 172.21.2.102:0/1912248618 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2378094960 msgr2=0x7f2378187e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:33.292 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:33.289 7f2377fff700 1 --2- 172.21.2.102:0/1912248618 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2378094960 0x7f2378187e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:33.292 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:33.289 7f2377fff700 1 -- 172.21.2.102:0/1912248618 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f23780b4f30 msgr2=0x7f237818b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:33.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:33.289 7f2377fff700 1 --2- 172.21.2.102:0/1912248618 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f23780b4f30 0x7f237818b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:33.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:33.289 7f2377fff700 1 -- 172.21.2.102:0/1912248618 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2378188470 con 0x7f2378093f40 2021-02-10T11:33:33.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:33.289 7f2377fff700 1 --2- 172.21.2.102:0/1912248618 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2378093f40 0x7f2378094340 secure :-1 s=READY pgs=2417 cs=0 l=1 rev1=1 rx=0x7f236800b8e0 tx=0x7f236800bc80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:33.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:33.289 7f23757fa700 1 -- 172.21.2.102:0/1912248618 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2368000fe0 con 0x7f2378093f40 2021-02-10T11:33:33.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:33.289 7f23757fa700 1 -- 172.21.2.102:0/1912248618 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f236800c2f0 con 0x7f2378093f40 2021-02-10T11:33:33.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:33.289 7f237f182700 1 -- 172.21.2.102:0/1912248618 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2378188470 con 0x7f2378093f40 2021-02-10T11:33:33.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:33.289 7f237f182700 1 -- 172.21.2.102:0/1912248618 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2378188a00 con 0x7f2378093f40 2021-02-10T11:33:33.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:33.289 7f23757fa700 1 -- 172.21.2.102:0/1912248618 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f236800c810 con 0x7f2378093f40 2021-02-10T11:33:33.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:33.290 7f23757fa700 1 -- 172.21.2.102:0/1912248618 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f236800c4d0 con 0x7f2378093f40 2021-02-10T11:33:33.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:33.290 7f23757fa700 1 --2- 172.21.2.102:0/1912248618 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f235c0186c0 0x7f235c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:33.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:33.290 7f23757fa700 1 -- 172.21.2.102:0/1912248618 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f2368032970 con 0x7f2378093f40 2021-02-10T11:33:33.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:33.290 7f237cf9e700 1 --2- 172.21.2.102:0/1912248618 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f235c0186c0 0x7f235c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:33.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:33.290 7f237cf9e700 1 --2- 172.21.2.102:0/1912248618 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f235c0186c0 0x7f235c01abc0 crc :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:33.322 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:33.322 7f237f182700 1 -- 172.21.2.102:0/1912248618 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f237808f760 con 0x7f2378093f40 2021-02-10T11:33:33.325 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:33.324 7f23757fa700 1 -- 172.21.2.102:0/1912248618 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f2368021280 con 0x7f2378093f40 2021-02-10T11:33:33.448 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:33.447 7f237f182700 1 -- 172.21.2.102:0/1912248618 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f237808f760 con 0x7f2378093f40 2021-02-10T11:33:33.448 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:33.448 7f23757fa700 1 -- 172.21.2.102:0/1912248618 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v127) v1 ==== 74+0+14020 (secure 0 0 0) 0x7f2368021280 con 0x7f2378093f40 2021-02-10T11:33:33.449 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:33:33.449 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":127,"fsid":"68db30ea-dbce-4aef-a6df-f01c6fea0513","created":"2021-02-10 11:03:06.331557","modified":"2021-02-10 11:33:32.844346","last_up_change":"2021-02-10 11:03:32.690562","last_in_change":"2021-02-10 11:03:32.690562","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":27,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:03:37.322519","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":26,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:33:30.653221","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"127","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":27,"pool_name":"cephfs_data","create_time":"2021-02-10 11:33:31.831765","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"127","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"be691b4d-a039-408a-af80-9d53f02b6f1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6818","nonce":36088},{"type":"v1","addr":"172.21.2.102:6820","nonce":36088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6822","nonce":36088},{"type":"v1","addr":"172.21.2.102:6824","nonce":36088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6830","nonce":36088},{"type":"v1","addr":"172.21.2.102:6832","nonce":36088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6826","nonce":36088},{"type":"v1","addr":"172.21.2.102:6828","nonce":36088}]},"public_addr":"172.21.2.102:6820/36088","cluster_addr":"172.21.2.102:6824/36088","heartbeat_back_addr":"172.21.2.102:6832/36088","heartbeat_front_addr":"172.21.2.102:6828/36088","state":["exists","up"]},{"osd":1,"uuid":"64e3ccc8-9c69-4d7c-865e-af8e83822635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6819","nonce":36087},{"type":"v1","addr":"172.21.2.102:6821","nonce":36087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6823","nonce":36087},{"type":"v1","addr":"172.21.2.102:6825","nonce":36087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6831","nonce":36087},{"type":"v1","addr":"172.21.2.102:6833","nonce":36087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6827","nonce":36087},{"type":"v1","addr":"172.21.2.102:6829","nonce":36087}]},"public_addr":"172.21.2.102:6821/36087","cluster_addr":"172.21.2.102:6825/36087","heartbeat_back_addr":"172.21.2.102:6833/36087","heartbeat_front_addr":"172.21.2.102:6829/36087","state":["exists","up"]},{"osd":2,"uuid":"153700af-747f-481f-bdf4-d6072d3a842f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6810","nonce":36089},{"type":"v1","addr":"172.21.2.102:6811","nonce":36089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6812","nonce":36089},{"type":"v1","addr":"172.21.2.102:6813","nonce":36089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6816","nonce":36089},{"type":"v1","addr":"172.21.2.102:6817","nonce":36089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6814","nonce":36089},{"type":"v1","addr":"172.21.2.102:6815","nonce":36089}]},"public_addr":"172.21.2.102:6811/36089","cluster_addr":"172.21.2.102:6813/36089","heartbeat_back_addr":"172.21.2.102:6817/36089","heartbeat_front_addr":"172.21.2.102:6815/36089","state":["exists","up"]},{"osd":3,"uuid":"cbc678f4-83a9-45e6-8830-2f7c232b1ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6802","nonce":36090},{"type":"v1","addr":"172.21.2.102:6803","nonce":36090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6804","nonce":36090},{"type":"v1","addr":"172.21.2.102:6805","nonce":36090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6808","nonce":36090},{"type":"v1","addr":"172.21.2.102:6809","nonce":36090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6806","nonce":36090},{"type":"v1","addr":"172.21.2.102:6807","nonce":36090}]},"public_addr":"172.21.2.102:6803/36090","cluster_addr":"172.21.2.102:6805/36090","heartbeat_back_addr":"172.21.2.102:6809/36090","heartbeat_front_addr":"172.21.2.102:6807/36090","state":["exists","up"]},{"osd":4,"uuid":"c01e4008-b258-4f10-afd2-9e7f37214351","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6808","nonce":36368},{"type":"v1","addr":"172.21.2.113:6811","nonce":36368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6814","nonce":36368},{"type":"v1","addr":"172.21.2.113:6818","nonce":36368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6826","nonce":36368},{"type":"v1","addr":"172.21.2.113:6828","nonce":36368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6820","nonce":36368},{"type":"v1","addr":"172.21.2.113:6824","nonce":36368}]},"public_addr":"172.21.2.113:6811/36368","cluster_addr":"172.21.2.113:6818/36368","heartbeat_back_addr":"172.21.2.113:6828/36368","heartbeat_front_addr":"172.21.2.113:6824/36368","state":["exists","up"]},{"osd":5,"uuid":"551f6007-daf8-4a98-8ee0-c2b16bcccb5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6802","nonce":36366},{"type":"v1","addr":"172.21.2.113:6805","nonce":36366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6807","nonce":36366},{"type":"v1","addr":"172.21.2.113:6810","nonce":36366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6819","nonce":36366},{"type":"v1","addr":"172.21.2.113:6822","nonce":36366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6812","nonce":36366},{"type":"v1","addr":"172.21.2.113:6815","nonce":36366}]},"public_addr":"172.21.2.113:6805/36366","cluster_addr":"172.21.2.113:6810/36366","heartbeat_back_addr":"172.21.2.113:6822/36366","heartbeat_front_addr":"172.21.2.113:6815/36366","state":["exists","up"]},{"osd":6,"uuid":"bf67c040-0e50-48be-b102-c3943bc9ebef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6800","nonce":36367},{"type":"v1","addr":"172.21.2.113:6801","nonce":36367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6803","nonce":36367},{"type":"v1","addr":"172.21.2.113:6804","nonce":36367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6813","nonce":36367},{"type":"v1","addr":"172.21.2.113:6816","nonce":36367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6806","nonce":36367},{"type":"v1","addr":"172.21.2.113:6809","nonce":36367}]},"public_addr":"172.21.2.113:6801/36367","cluster_addr":"172.21.2.113:6804/36367","heartbeat_back_addr":"172.21.2.113:6816/36367","heartbeat_front_addr":"172.21.2.113:6809/36367","state":["exists","up"]},{"osd":7,"uuid":"f5461f80-018d-428f-be89-2eb2bcbe7bb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6817","nonce":36365},{"type":"v1","addr":"172.21.2.113:6821","nonce":36365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6823","nonce":36365},{"type":"v1","addr":"172.21.2.113:6825","nonce":36365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6830","nonce":36365},{"type":"v1","addr":"172.21.2.113:6831","nonce":36365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6827","nonce":36365},{"type":"v1","addr":"172.21.2.113:6829","nonce":36365}]},"public_addr":"172.21.2.113:6821/36365","cluster_addr":"172.21.2.113:6825/36365","heartbeat_back_addr":"172.21.2.113:6831/36365","heartbeat_front_addr":"172.21.2.113:6829/36365","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:33:33.450 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:33.449 7f237f182700 1 -- 172.21.2.102:0/1912248618 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f235c0186c0 msgr2=0x7f235c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:33.450 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:33.449 7f237f182700 1 --2- 172.21.2.102:0/1912248618 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f235c0186c0 0x7f235c01abc0 crc :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:33.450 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:33.449 7f237f182700 1 -- 172.21.2.102:0/1912248618 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2378093f40 msgr2=0x7f2378094340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:33.450 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:33.450 7f237f182700 1 --2- 172.21.2.102:0/1912248618 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2378093f40 0x7f2378094340 secure :-1 s=READY pgs=2417 cs=0 l=1 rev1=1 rx=0x7f236800b8e0 tx=0x7f236800bc80).stop 2021-02-10T11:33:33.451 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:33.450 7f237f182700 1 -- 172.21.2.102:0/1912248618 shutdown_connections 2021-02-10T11:33:33.451 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:33.450 7f237f182700 1 --2- 172.21.2.102:0/1912248618 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2378094960 0x7f2378187e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:33.451 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:33.450 7f237f182700 1 --2- 172.21.2.102:0/1912248618 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2378093f40 0x7f2378094340 unknown :-1 s=CLOSED pgs=2417 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:33.451 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:33.450 7f237f182700 1 --2- 172.21.2.102:0/1912248618 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f235c0186c0 0x7f235c01abc0 unknown :-1 s=CLOSED pgs=1069 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:33.451 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:33.450 7f237f182700 1 --2- 172.21.2.102:0/1912248618 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f23780b4f30 0x7f237818b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:33.451 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:33.450 7f237f182700 1 -- 172.21.2.102:0/1912248618 >> 172.21.2.102:0/1912248618 conn(0x7f237819f8f0 msgr2=0x7f23781a5f80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:33.451 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:33.450 7f237f182700 1 -- 172.21.2.102:0/1912248618 shutdown_connections 2021-02-10T11:33:33.452 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:33.450 7f237f182700 1 -- 172.21.2.102:0/1912248618 wait complete. 2021-02-10T11:33:33.459 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:33:33.885 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:33.867 7fac997b9700 1 Processor -- start 2021-02-10T11:33:33.885 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:33.867 7fac997b9700 1 -- start start 2021-02-10T11:33:33.885 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:33.868 7fac997b9700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fac940d1c70 0x7fac940f20a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:33.885 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:33.868 7fac997b9700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac940f0b10 con 0x7fac940d1870 2021-02-10T11:33:33.886 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:33.868 7fac997b9700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac941b0be0 con 0x7fac940d0340 2021-02-10T11:33:33.886 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:33.868 7fac997b9700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac940e5ed0 con 0x7fac940d1c70 2021-02-10T11:33:33.886 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:33.868 7fac93574700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fac940d1870 0x7fac941b03f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:55836/0 (socket says 172.21.2.102:55836) 2021-02-10T11:33:33.886 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:33.868 7fac93574700 1 -- 172.21.2.102:0/76545373 learned_addr learned my addr 172.21.2.102:0/76545373 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:33:33.886 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:33.868 7fac93d75700 1 --2- 172.21.2.102:0/76545373 >> v2:172.21.2.113:3300/0 conn(0x7fac940d1c70 0x7fac940f20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:33.886 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:33.869 7fac92572700 1 -- 172.21.2.102:0/76545373 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2679864406 0 0) 0x7fac840010b0 con 0x7fac940d1870 2021-02-10T11:33:33.886 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:33.869 7fac92572700 1 -- 172.21.2.102:0/76545373 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac80002fe0 con 0x7fac940d1870 2021-02-10T11:33:33.886 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:33.869 7fac92572700 1 -- 172.21.2.102:0/76545373 <== mon.1 v1:172.21.2.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2931946841 0 0) 0x7fac7c000ff0 con 0x7fac940d0340 2021-02-10T11:33:33.886 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:33.869 7fac92572700 1 -- 172.21.2.102:0/76545373 --> v1:172.21.2.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac80003210 con 0x7fac940d0340 2021-02-10T11:33:33.887 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:33.869 7fac93d75700 1 -- 172.21.2.102:0/76545373 >> v1:172.21.2.113:6789/0 conn(0x7fac940d0340 legacy=0x7fac940ef640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:33.887 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:33.869 7fac93d75700 1 -- 172.21.2.102:0/76545373 >> v1:172.21.2.102:6789/0 conn(0x7fac940d1870 legacy=0x7fac941b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:33.887 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:33.869 7fac93d75700 1 -- 172.21.2.102:0/76545373 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac940efe30 con 0x7fac940d1c70 2021-02-10T11:33:33.887 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:33.869 7fac93d75700 1 --2- 172.21.2.102:0/76545373 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fac940d1c70 0x7fac940f20a0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 rx=0x7fac8c006e50 tx=0x7fac8c009e70).ready entity=mon.1 client_cookie=14e45f0b66a6b2e8 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:33.887 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:33.869 7fac92572700 1 -- 172.21.2.102:0/76545373 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fac8c00b510 con 0x7fac940d1c70 2021-02-10T11:33:33.887 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:33.869 7fac92572700 1 -- 172.21.2.102:0/76545373 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac8c006790 con 0x7fac940d1c70 2021-02-10T11:33:33.887 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:33.869 7fac92572700 1 -- 172.21.2.102:0/76545373 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fac8c00b940 con 0x7fac940d1c70 2021-02-10T11:33:33.887 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:33.869 7fac997b9700 1 -- 172.21.2.102:0/76545373 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fac940d1c70 msgr2=0x7fac940f20a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:33.887 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:33.869 7fac997b9700 1 --2- 172.21.2.102:0/76545373 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fac940d1c70 0x7fac940f20a0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 rx=0x7fac8c006e50 tx=0x7fac8c009e70).stop 2021-02-10T11:33:33.887 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:33.870 7fac997b9700 1 -- 172.21.2.102:0/76545373 shutdown_connections 2021-02-10T11:33:33.887 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:33.870 7fac997b9700 1 --2- 172.21.2.102:0/76545373 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fac940d1c70 0x7fac940f20a0 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:33.887 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:33.870 7fac997b9700 1 -- 172.21.2.102:0/76545373 >> 172.21.2.102:0/76545373 conn(0x7fac940ce450 msgr2=0x7fac940d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:33.888 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:33.870 7fac997b9700 1 -- 172.21.2.102:0/76545373 shutdown_connections 2021-02-10T11:33:33.888 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:33.870 7fac997b9700 1 -- 172.21.2.102:0/76545373 wait complete. 2021-02-10T11:33:33.889 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:33.885 7fac997b9700 1 Processor -- start 2021-02-10T11:33:33.889 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:33.885 7fac997b9700 1 -- start start 2021-02-10T11:33:33.889 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:33.885 7fac997b9700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fac9418d870 0x7fac940b19b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:33.889 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:33.885 7fac997b9700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fac940b1fd0 0x7fac94099ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:33.889 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:33.885 7fac997b9700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fac940b23d0 0x7fac9409a510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:33.890 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:33.885 7fac997b9700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac9418dc70 con 0x7fac9418d870 2021-02-10T11:33:33.890 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:33.885 7fac997b9700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac94199e60 con 0x7fac940b1fd0 2021-02-10T11:33:33.890 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:33.885 7fac997b9700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac9409ab30 con 0x7fac940b23d0 2021-02-10T11:33:33.890 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:33.886 7fac93574700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fac9418d870 0x7fac940b19b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:33.890 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:33.886 7fac93574700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fac9418d870 0x7fac940b19b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:35386/0 (socket says 172.21.2.102:35386) 2021-02-10T11:33:33.890 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:33.886 7fac93574700 1 -- 172.21.2.102:0/3397795181 learned_addr learned my addr 172.21.2.102:0/3397795181 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:33:33.890 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:33.886 7fac93574700 1 -- 172.21.2.102:0/3397795181 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fac940b23d0 msgr2=0x7fac9409a510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:33.890 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:33.886 7fac93574700 1 --2- 172.21.2.102:0/3397795181 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fac940b23d0 0x7fac9409a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:33.890 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:33.886 7fac92d73700 1 --2- 172.21.2.102:0/3397795181 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fac940b1fd0 0x7fac94099ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:33.890 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:33.886 7fac93574700 1 -- 172.21.2.102:0/3397795181 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fac940b1fd0 msgr2=0x7fac94099ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:33.890 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:33.886 7fac93574700 1 --2- 172.21.2.102:0/3397795181 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fac940b1fd0 0x7fac94099ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:33.891 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:33.886 7fac93574700 1 -- 172.21.2.102:0/3397795181 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac9409ad50 con 0x7fac9418d870 2021-02-10T11:33:33.891 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:33.886 7fac93574700 1 --2- 172.21.2.102:0/3397795181 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fac9418d870 0x7fac940b19b0 secure :-1 s=READY pgs=2419 cs=0 l=1 rev1=1 rx=0x7fac84001c60 tx=0x7fac8400a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:33.891 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:33.886 7fac8bfff700 1 -- 172.21.2.102:0/3397795181 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fac8400acc0 con 0x7fac9418d870 2021-02-10T11:33:33.891 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:33.886 7fac8bfff700 1 -- 172.21.2.102:0/3397795181 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac84004350 con 0x7fac9418d870 2021-02-10T11:33:33.891 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:33.886 7fac997b9700 1 -- 172.21.2.102:0/3397795181 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac9409ad50 con 0x7fac9418d870 2021-02-10T11:33:33.891 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:33.886 7fac997b9700 1 -- 172.21.2.102:0/3397795181 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac9409ad50 con 0x7fac9418d870 2021-02-10T11:33:33.891 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:33.886 7fac8bfff700 1 -- 172.21.2.102:0/3397795181 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fac84004830 con 0x7fac9418d870 2021-02-10T11:33:33.891 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:33.887 7fac8bfff700 1 -- 172.21.2.102:0/3397795181 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fac84004800 con 0x7fac9418d870 2021-02-10T11:33:33.891 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:33.887 7fac8bfff700 1 --2- 172.21.2.102:0/3397795181 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fac780186d0 0x7fac7801abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:33.891 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:33.887 7fac8bfff700 1 -- 172.21.2.102:0/3397795181 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fac84032770 con 0x7fac9418d870 2021-02-10T11:33:33.891 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:33.887 7fac92d73700 1 --2- 172.21.2.102:0/3397795181 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fac780186d0 0x7fac7801abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:33.891 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:33.887 7fac92d73700 1 --2- 172.21.2.102:0/3397795181 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fac780186d0 0x7fac7801abd0 crc :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:33.918 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:33.918 7fac997b9700 1 -- 172.21.2.102:0/3397795181 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac940999c0 con 0x7fac9418d870 2021-02-10T11:33:33.921 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:33.921 7fac8bfff700 1 -- 172.21.2.102:0/3397795181 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fac84004140 con 0x7fac9418d870 2021-02-10T11:33:34.047 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:34.046 7fac997b9700 1 -- 172.21.2.102:0/3397795181 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fac940999c0 con 0x7fac9418d870 2021-02-10T11:33:34.047 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:34.047 7fac8bfff700 1 -- 172.21.2.102:0/3397795181 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v128) v1 ==== 74+0+14007 (secure 0 0 0) 0x7fac84004350 con 0x7fac9418d870 2021-02-10T11:33:34.048 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:33:34.048 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":128,"fsid":"68db30ea-dbce-4aef-a6df-f01c6fea0513","created":"2021-02-10 11:03:06.331557","modified":"2021-02-10 11:33:33.846924","last_up_change":"2021-02-10 11:03:32.690562","last_in_change":"2021-02-10 11:03:32.690562","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":27,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:03:37.322519","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":26,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:33:30.653221","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"127","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":27,"pool_name":"cephfs_data","create_time":"2021-02-10 11:33:31.831765","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"128","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"be691b4d-a039-408a-af80-9d53f02b6f1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6818","nonce":36088},{"type":"v1","addr":"172.21.2.102:6820","nonce":36088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6822","nonce":36088},{"type":"v1","addr":"172.21.2.102:6824","nonce":36088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6830","nonce":36088},{"type":"v1","addr":"172.21.2.102:6832","nonce":36088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6826","nonce":36088},{"type":"v1","addr":"172.21.2.102:6828","nonce":36088}]},"public_addr":"172.21.2.102:6820/36088","cluster_addr":"172.21.2.102:6824/36088","heartbeat_back_addr":"172.21.2.102:6832/36088","heartbeat_front_addr":"172.21.2.102:6828/36088","state":["exists","up"]},{"osd":1,"uuid":"64e3ccc8-9c69-4d7c-865e-af8e83822635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6819","nonce":36087},{"type":"v1","addr":"172.21.2.102:6821","nonce":36087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6823","nonce":36087},{"type":"v1","addr":"172.21.2.102:6825","nonce":36087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6831","nonce":36087},{"type":"v1","addr":"172.21.2.102:6833","nonce":36087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6827","nonce":36087},{"type":"v1","addr":"172.21.2.102:6829","nonce":36087}]},"public_addr":"172.21.2.102:6821/36087","cluster_addr":"172.21.2.102:6825/36087","heartbeat_back_addr":"172.21.2.102:6833/36087","heartbeat_front_addr":"172.21.2.102:6829/36087","state":["exists","up"]},{"osd":2,"uuid":"153700af-747f-481f-bdf4-d6072d3a842f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6810","nonce":36089},{"type":"v1","addr":"172.21.2.102:6811","nonce":36089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6812","nonce":36089},{"type":"v1","addr":"172.21.2.102:6813","nonce":36089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6816","nonce":36089},{"type":"v1","addr":"172.21.2.102:6817","nonce":36089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6814","nonce":36089},{"type":"v1","addr":"172.21.2.102:6815","nonce":36089}]},"public_addr":"172.21.2.102:6811/36089","cluster_addr":"172.21.2.102:6813/36089","heartbeat_back_addr":"172.21.2.102:6817/36089","heartbeat_front_addr":"172.21.2.102:6815/36089","state":["exists","up"]},{"osd":3,"uuid":"cbc678f4-83a9-45e6-8830-2f7c232b1ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6802","nonce":36090},{"type":"v1","addr":"172.21.2.102:6803","nonce":36090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6804","nonce":36090},{"type":"v1","addr":"172.21.2.102:6805","nonce":36090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6808","nonce":36090},{"type":"v1","addr":"172.21.2.102:6809","nonce":36090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6806","nonce":36090},{"type":"v1","addr":"172.21.2.102:6807","nonce":36090}]},"public_addr":"172.21.2.102:6803/36090","cluster_addr":"172.21.2.102:6805/36090","heartbeat_back_addr":"172.21.2.102:6809/36090","heartbeat_front_addr":"172.21.2.102:6807/36090","state":["exists","up"]},{"osd":4,"uuid":"c01e4008-b258-4f10-afd2-9e7f37214351","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6808","nonce":36368},{"type":"v1","addr":"172.21.2.113:6811","nonce":36368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6814","nonce":36368},{"type":"v1","addr":"172.21.2.113:6818","nonce":36368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6826","nonce":36368},{"type":"v1","addr":"172.21.2.113:6828","nonce":36368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6820","nonce":36368},{"type":"v1","addr":"172.21.2.113:6824","nonce":36368}]},"public_addr":"172.21.2.113:6811/36368","cluster_addr":"172.21.2.113:6818/36368","heartbeat_back_addr":"172.21.2.113:6828/36368","heartbeat_front_addr":"172.21.2.113:6824/36368","state":["exists","up"]},{"osd":5,"uuid":"551f6007-daf8-4a98-8ee0-c2b16bcccb5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6802","nonce":36366},{"type":"v1","addr":"172.21.2.113:6805","nonce":36366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6807","nonce":36366},{"type":"v1","addr":"172.21.2.113:6810","nonce":36366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6819","nonce":36366},{"type":"v1","addr":"172.21.2.113:6822","nonce":36366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6812","nonce":36366},{"type":"v1","addr":"172.21.2.113:6815","nonce":36366}]},"public_addr":"172.21.2.113:6805/36366","cluster_addr":"172.21.2.113:6810/36366","heartbeat_back_addr":"172.21.2.113:6822/36366","heartbeat_front_addr":"172.21.2.113:6815/36366","state":["exists","up"]},{"osd":6,"uuid":"bf67c040-0e50-48be-b102-c3943bc9ebef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6800","nonce":36367},{"type":"v1","addr":"172.21.2.113:6801","nonce":36367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6803","nonce":36367},{"type":"v1","addr":"172.21.2.113:6804","nonce":36367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6813","nonce":36367},{"type":"v1","addr":"172.21.2.113:6816","nonce":36367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6806","nonce":36367},{"type":"v1","addr":"172.21.2.113:6809","nonce":36367}]},"public_addr":"172.21.2.113:6801/36367","cluster_addr":"172.21.2.113:6804/36367","heartbeat_back_addr":"172.21.2.113:6816/36367","heartbeat_front_addr":"172.21.2.113:6809/36367","state":["exists","up"]},{"osd":7,"uuid":"f5461f80-018d-428f-be89-2eb2bcbe7bb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6817","nonce":36365},{"type":"v1","addr":"172.21.2.113:6821","nonce":36365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6823","nonce":36365},{"type":"v1","addr":"172.21.2.113:6825","nonce":36365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6830","nonce":36365},{"type":"v1","addr":"172.21.2.113:6831","nonce":36365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6827","nonce":36365},{"type":"v1","addr":"172.21.2.113:6829","nonce":36365}]},"public_addr":"172.21.2.113:6821/36365","cluster_addr":"172.21.2.113:6825/36365","heartbeat_back_addr":"172.21.2.113:6831/36365","heartbeat_front_addr":"172.21.2.113:6829/36365","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:33:34.049 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:34.048 7fac997b9700 1 -- 172.21.2.102:0/3397795181 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fac780186d0 msgr2=0x7fac7801abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:34.049 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:34.048 7fac997b9700 1 --2- 172.21.2.102:0/3397795181 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fac780186d0 0x7fac7801abd0 crc :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:34.049 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:34.048 7fac997b9700 1 -- 172.21.2.102:0/3397795181 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fac9418d870 msgr2=0x7fac940b19b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:34.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:34.048 7fac997b9700 1 --2- 172.21.2.102:0/3397795181 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fac9418d870 0x7fac940b19b0 secure :-1 s=READY pgs=2419 cs=0 l=1 rev1=1 rx=0x7fac84001c60 tx=0x7fac8400a8d0).stop 2021-02-10T11:33:34.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:34.049 7fac997b9700 1 -- 172.21.2.102:0/3397795181 shutdown_connections 2021-02-10T11:33:34.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:34.049 7fac997b9700 1 --2- 172.21.2.102:0/3397795181 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fac940b23d0 0x7fac9409a510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:34.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:34.049 7fac997b9700 1 --2- 172.21.2.102:0/3397795181 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fac780186d0 0x7fac7801abd0 unknown :-1 s=CLOSED pgs=1077 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:34.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:34.049 7fac997b9700 1 --2- 172.21.2.102:0/3397795181 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fac940b1fd0 0x7fac94099ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:34.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:34.049 7fac997b9700 1 --2- 172.21.2.102:0/3397795181 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fac9418d870 0x7fac940b19b0 unknown :-1 s=CLOSED pgs=2419 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:34.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:34.049 7fac997b9700 1 -- 172.21.2.102:0/3397795181 >> 172.21.2.102:0/3397795181 conn(0x7fac940f0550 msgr2=0x7fac94191730 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:34.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:34.049 7fac997b9700 1 -- 172.21.2.102:0/3397795181 shutdown_connections 2021-02-10T11:33:34.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:34.049 7fac997b9700 1 -- 172.21.2.102:0/3397795181 wait complete. 2021-02-10T11:33:34.058 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2021-02-10T11:33:34.490 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:34.472 7f03d0051700 1 Processor -- start 2021-02-10T11:33:34.490 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:34.472 7f03d0051700 1 -- start start 2021-02-10T11:33:34.491 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:34.473 7f03d0051700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f03c80d1870 0x7f03c81b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:34.491 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:34.473 7f03d0051700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f03c80cff70 0x7f03c8109130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:34.491 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:34.473 7f03d0051700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03c80fe1c0 con 0x7f03c81b0ae0 2021-02-10T11:33:34.491 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:34.473 7f03d0051700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03c8109ad0 con 0x7f03c80cff70 2021-02-10T11:33:34.491 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:34.473 7f03d0051700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03c8109750 con 0x7f03c80d1870 2021-02-10T11:33:34.491 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:34.473 7f03ce66e700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f03c80cff70 0x7f03c8109130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:34.491 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:34.473 7f03ce66e700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f03c80cff70 0x7f03c8109130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:35394/0 (socket says 172.21.2.102:35394) 2021-02-10T11:33:34.491 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:34.473 7f03ce66e700 1 -- 172.21.2.102:0/867589968 learned_addr learned my addr 172.21.2.102:0/867589968 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:33:34.491 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:34.473 7f03ce66e700 1 -- 172.21.2.102:0/867589968 >> v2:172.21.2.113:3300/0 conn(0x7f03c80d1870 msgr2=0x7f03c81b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:33:34.491 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:34.473 7f03ce66e700 1 --2- 172.21.2.102:0/867589968 >> v2:172.21.2.113:3300/0 conn(0x7f03c80d1870 0x7f03c81b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:34.491 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:34.473 7f03ce66e700 1 -- 172.21.2.102:0/867589968 >> v1:172.21.2.113:6789/0 conn(0x7f03c81b0ae0 legacy=0x7f03c8108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:33:34.491 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:34.473 7f03ce66e700 1 -- 172.21.2.102:0/867589968 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03c8088070 con 0x7f03c80cff70 2021-02-10T11:33:34.491 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:34.474 7f03ce66e700 1 --2- 172.21.2.102:0/867589968 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f03c80cff70 0x7f03c8109130 secure :-1 s=READY pgs=2420 cs=0 l=1 rev1=1 rx=0x7f03c0006f00 tx=0x7f03c0009f10).ready entity=mon.0 client_cookie=74b5cff145f87045 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:34.492 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:34.474 7f03cce6b700 1 -- 172.21.2.102:0/867589968 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f03c000b3b0 con 0x7f03c80cff70 2021-02-10T11:33:34.492 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:34.474 7f03cce6b700 1 -- 172.21.2.102:0/867589968 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03c000b730 con 0x7f03c80cff70 2021-02-10T11:33:34.492 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:34.474 7f03cce6b700 1 -- 172.21.2.102:0/867589968 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f03c000bb90 con 0x7f03c80cff70 2021-02-10T11:33:34.492 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:34.474 7f03d0051700 1 -- 172.21.2.102:0/867589968 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f03c80cff70 msgr2=0x7f03c8109130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:34.492 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:34.474 7f03d0051700 1 --2- 172.21.2.102:0/867589968 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f03c80cff70 0x7f03c8109130 secure :-1 s=READY pgs=2420 cs=0 l=1 rev1=1 rx=0x7f03c0006f00 tx=0x7f03c0009f10).stop 2021-02-10T11:33:34.492 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:34.474 7f03d0051700 1 -- 172.21.2.102:0/867589968 shutdown_connections 2021-02-10T11:33:34.492 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:34.474 7f03d0051700 1 --2- 172.21.2.102:0/867589968 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f03c80cff70 0x7f03c8109130 unknown :-1 s=CLOSED pgs=2420 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:34.492 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:34.474 7f03d0051700 1 --2- 172.21.2.102:0/867589968 >> v2:172.21.2.113:3300/0 conn(0x7f03c80d1870 0x7f03c81b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:34.492 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:34.474 7f03d0051700 1 -- 172.21.2.102:0/867589968 >> 172.21.2.102:0/867589968 conn(0x7f03c80ce420 msgr2=0x7f03c80d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:34.492 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:34.475 7f03d0051700 1 -- 172.21.2.102:0/867589968 shutdown_connections 2021-02-10T11:33:34.492 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:34.475 7f03d0051700 1 -- 172.21.2.102:0/867589968 wait complete. 2021-02-10T11:33:34.494 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:34.489 7f03d0051700 1 Processor -- start 2021-02-10T11:33:34.494 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:34.489 7f03d0051700 1 -- start start 2021-02-10T11:33:34.494 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:34.490 7f03d0051700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f03c80c3820 0x7f03c8084cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:34.494 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:34.490 7f03d0051700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f03c80a0290 0x7f03c80852d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:34.494 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:34.490 7f03d0051700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f03c8086270 0x7f03c8086670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:34.494 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:34.490 7f03d0051700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03c80c0700 con 0x7f03c8086270 2021-02-10T11:33:34.494 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:34.490 7f03d0051700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03c80858f0 con 0x7f03c80c3820 2021-02-10T11:33:34.494 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:34.490 7f03d0051700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03c8085b10 con 0x7f03c80a0290 2021-02-10T11:33:34.494 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:34.490 7f03ce66e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f03c8086270 0x7f03c8086670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:34.494 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:34.490 7f03ce66e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f03c8086270 0x7f03c8086670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:35402/0 (socket says 172.21.2.102:35402) 2021-02-10T11:33:34.494 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:34.490 7f03ce66e700 1 -- 172.21.2.102:0/2178796782 learned_addr learned my addr 172.21.2.102:0/2178796782 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:33:34.495 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:34.490 7f03ce66e700 1 -- 172.21.2.102:0/2178796782 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f03c80a0290 msgr2=0x7f03c80852d0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:33:34.495 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:34.490 7f03ce66e700 1 --2- 172.21.2.102:0/2178796782 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f03c80a0290 0x7f03c80852d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:34.495 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:34.490 7f03ce66e700 1 -- 172.21.2.102:0/2178796782 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f03c80c3820 msgr2=0x7f03c8084cb0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:33:34.495 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:34.490 7f03ce66e700 1 --2- 172.21.2.102:0/2178796782 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f03c80c3820 0x7f03c8084cb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:34.495 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:34.490 7f03ce66e700 1 -- 172.21.2.102:0/2178796782 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03c8086c90 con 0x7f03c8086270 2021-02-10T11:33:34.495 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:34.490 7f03ce66e700 1 --2- 172.21.2.102:0/2178796782 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f03c8086270 0x7f03c8086670 secure :-1 s=READY pgs=2421 cs=0 l=1 rev1=1 rx=0x7f03c0009390 tx=0x7f03c000bd40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:34.495 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:34.490 7f03ba7fc700 1 -- 172.21.2.102:0/2178796782 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f03c0003030 con 0x7f03c8086270 2021-02-10T11:33:34.495 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:34.490 7f03ba7fc700 1 -- 172.21.2.102:0/2178796782 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03c00068e0 con 0x7f03c8086270 2021-02-10T11:33:34.495 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:34.490 7f03ba7fc700 1 -- 172.21.2.102:0/2178796782 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f03c000c570 con 0x7f03c8086270 2021-02-10T11:33:34.495 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:34.490 7f03d0051700 1 -- 172.21.2.102:0/2178796782 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03c8086c90 con 0x7f03c8086270 2021-02-10T11:33:34.495 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:34.491 7f03d0051700 1 -- 172.21.2.102:0/2178796782 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03c8086c90 con 0x7f03c8086270 2021-02-10T11:33:34.495 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:34.491 7f03ba7fc700 1 -- 172.21.2.102:0/2178796782 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f03c0001270 con 0x7f03c8086270 2021-02-10T11:33:34.495 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:34.491 7f03ba7fc700 1 --2- 172.21.2.102:0/2178796782 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f03b00186c0 0x7f03b001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:34.496 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:34.491 7f03ba7fc700 1 -- 172.21.2.102:0/2178796782 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f03c0032750 con 0x7f03c8086270 2021-02-10T11:33:34.496 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:34.491 7f03cde6d700 1 --2- 172.21.2.102:0/2178796782 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f03b00186c0 0x7f03b001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:34.496 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:34.492 7f03cde6d700 1 --2- 172.21.2.102:0/2178796782 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f03b00186c0 0x7f03b001abc0 crc :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:34.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:34.523 7f03d0051700 1 -- 172.21.2.102:0/2178796782 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03c8082a70 con 0x7f03c8086270 2021-02-10T11:33:34.527 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:34.526 7f03ba7fc700 1 -- 172.21.2.102:0/2178796782 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f03c0006c80 con 0x7f03c8086270 2021-02-10T11:33:34.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:34.661 7f03d0051700 1 -- 172.21.2.102:0/2178796782 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"} v 0) v1 -- 0x7f03c8082a70 con 0x7f03c8086270 2021-02-10T11:33:34.855 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:34.853 7f03ba7fc700 1 -- 172.21.2.102:0/2178796782 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"}]=0 v107) v1 ==== 118+0+0 (secure 0 0 0) 0x7f03c001f880 con 0x7f03c8086270 2021-02-10T11:33:34.880 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:34.878 7f03d0051700 1 -- 172.21.2.102:0/2178796782 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f03b00186c0 msgr2=0x7f03b001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:34.881 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:34.878 7f03d0051700 1 --2- 172.21.2.102:0/2178796782 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f03b00186c0 0x7f03b001abc0 crc :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:34.881 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:34.878 7f03d0051700 1 -- 172.21.2.102:0/2178796782 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f03c8086270 msgr2=0x7f03c8086670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:34.881 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:34.878 7f03d0051700 1 --2- 172.21.2.102:0/2178796782 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f03c8086270 0x7f03c8086670 secure :-1 s=READY pgs=2421 cs=0 l=1 rev1=1 rx=0x7f03c0009390 tx=0x7f03c000bd40).stop 2021-02-10T11:33:34.881 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:34.879 7f03d0051700 1 -- 172.21.2.102:0/2178796782 shutdown_connections 2021-02-10T11:33:34.881 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:34.879 7f03d0051700 1 --2- 172.21.2.102:0/2178796782 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f03c8086270 0x7f03c8086670 unknown :-1 s=CLOSED pgs=2421 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:34.881 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:34.879 7f03d0051700 1 --2- 172.21.2.102:0/2178796782 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f03c80a0290 0x7f03c80852d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:34.881 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:34.879 7f03d0051700 1 --2- 172.21.2.102:0/2178796782 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f03b00186c0 0x7f03b001abc0 unknown :-1 s=CLOSED pgs=1078 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:34.881 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:34.879 7f03d0051700 1 --2- 172.21.2.102:0/2178796782 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f03c80c3820 0x7f03c8084cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:34.881 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:34.879 7f03d0051700 1 -- 172.21.2.102:0/2178796782 >> 172.21.2.102:0/2178796782 conn(0x7f03c80872c0 msgr2=0x7f03c8095700 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:34.881 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:34.879 7f03d0051700 1 -- 172.21.2.102:0/2178796782 shutdown_connections 2021-02-10T11:33:34.881 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:34.879 7f03d0051700 1 -- 172.21.2.102:0/2178796782 wait complete. 2021-02-10T11:33:34.888 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:33:35.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:35.299 7f7d9bc0e700 1 Processor -- start 2021-02-10T11:33:35.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:35.299 7f7d9bc0e700 1 -- start start 2021-02-10T11:33:35.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:35.299 7f7d9bc0e700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f7d940d1870 0x7f7d941b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:35.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:35.299 7f7d9bc0e700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7d941b0a10 0x7f7d940cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:35.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:35.299 7f7d9bc0e700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f7d940d1c70 0x7f7d940cf950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:35.318 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:35.299 7f7d9bc0e700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d940fe200 con 0x7f7d940d1c70 2021-02-10T11:33:35.318 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:35.299 7f7d9bc0e700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d940e5f10 con 0x7f7d940d1870 2021-02-10T11:33:35.318 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:35.299 7f7d9bc0e700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d940f0b50 con 0x7f7d941b0a10 2021-02-10T11:33:35.318 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:35.300 7f7d9a22b700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f7d940d1c70 0x7f7d940cf950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:35.318 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:35.300 7f7d9a22b700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f7d940d1c70 0x7f7d940cf950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:35406/0 (socket says 172.21.2.102:35406) 2021-02-10T11:33:35.318 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:35.300 7f7d9a22b700 1 -- 172.21.2.102:0/780851157 learned_addr learned my addr 172.21.2.102:0/780851157 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:33:35.318 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:35.300 7f7d99a2a700 1 --2- 172.21.2.102:0/780851157 >> v2:172.21.2.113:3300/0 conn(0x7f7d940d1870 0x7f7d941b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:35.318 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:35.300 7f7d99229700 1 --2- 172.21.2.102:0/780851157 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7d941b0a10 0x7f7d940cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:35.318 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:35.300 7f7d9a22b700 1 -- 172.21.2.102:0/780851157 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7d941b0a10 msgr2=0x7f7d940cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:33:35.318 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:35.300 7f7d9a22b700 1 --2- 172.21.2.102:0/780851157 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7d941b0a10 0x7f7d940cf330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:35.319 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:35.300 7f7d9a22b700 1 -- 172.21.2.102:0/780851157 >> v2:172.21.2.113:3300/0 conn(0x7f7d940d1870 msgr2=0x7f7d941b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:33:35.319 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:35.300 7f7d9a22b700 1 --2- 172.21.2.102:0/780851157 >> v2:172.21.2.113:3300/0 conn(0x7f7d940d1870 0x7f7d941b03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:35.319 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:35.300 7f7d9a22b700 1 -- 172.21.2.102:0/780851157 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d940cff70 con 0x7f7d940d1c70 2021-02-10T11:33:35.319 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:35.300 7f7d99229700 1 --2- 172.21.2.102:0/780851157 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7d941b0a10 0x7f7d940cf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:33:35.319 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:35.300 7f7d9a22b700 1 --2- 172.21.2.102:0/780851157 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7d940d1c70 0x7f7d940cf950 secure :-1 s=READY pgs=2422 cs=0 l=1 rev1=1 rx=0x7f7d8c006f00 tx=0x7f7d8c009f10).ready entity=mon.0 client_cookie=7793ae06a2bfa5ed server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:35.319 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:35.300 7f7d98a28700 1 -- 172.21.2.102:0/780851157 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7d8c00b3b0 con 0x7f7d940d1c70 2021-02-10T11:33:35.319 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:35.300 7f7d98a28700 1 -- 172.21.2.102:0/780851157 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d8c00b630 con 0x7f7d940d1c70 2021-02-10T11:33:35.319 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:35.300 7f7d98a28700 1 -- 172.21.2.102:0/780851157 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7d8c00bb10 con 0x7f7d940d1c70 2021-02-10T11:33:35.319 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:35.300 7f7d9bc0e700 1 -- 172.21.2.102:0/780851157 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7d940d1c70 msgr2=0x7f7d940cf950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:35.319 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:35.300 7f7d9bc0e700 1 --2- 172.21.2.102:0/780851157 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7d940d1c70 0x7f7d940cf950 secure :-1 s=READY pgs=2422 cs=0 l=1 rev1=1 rx=0x7f7d8c006f00 tx=0x7f7d8c009f10).stop 2021-02-10T11:33:35.319 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:35.301 7f7d9bc0e700 1 -- 172.21.2.102:0/780851157 shutdown_connections 2021-02-10T11:33:35.319 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:35.301 7f7d9bc0e700 1 --2- 172.21.2.102:0/780851157 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7d941b0a10 0x7f7d940cf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:35.319 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:35.301 7f7d9bc0e700 1 --2- 172.21.2.102:0/780851157 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7d940d1c70 0x7f7d940cf950 unknown :-1 s=CLOSED pgs=2422 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:35.320 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:35.301 7f7d9bc0e700 1 --2- 172.21.2.102:0/780851157 >> v2:172.21.2.113:3300/0 conn(0x7f7d940d1870 0x7f7d941b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:35.320 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:35.301 7f7d9bc0e700 1 -- 172.21.2.102:0/780851157 >> 172.21.2.102:0/780851157 conn(0x7f7d940ce450 msgr2=0x7f7d940d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:35.320 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:35.301 7f7d9bc0e700 1 -- 172.21.2.102:0/780851157 shutdown_connections 2021-02-10T11:33:35.320 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:35.301 7f7d9bc0e700 1 -- 172.21.2.102:0/780851157 wait complete. 2021-02-10T11:33:35.321 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:35.316 7f7d9bc0e700 1 Processor -- start 2021-02-10T11:33:35.321 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:35.316 7f7d9bc0e700 1 -- start start 2021-02-10T11:33:35.321 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:35.316 7f7d9bc0e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7d941942b0 0x7f7d940b3550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:35.321 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:35.316 7f7d9bc0e700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7d94195070 0x7f7d94195470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:35.322 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:35.316 7f7d9bc0e700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7d940bb8e0 0x7f7d940bbce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:35.322 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:35.316 7f7d9bc0e700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d941946b0 con 0x7f7d941942b0 2021-02-10T11:33:35.322 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:35.316 7f7d9bc0e700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d940b3b70 con 0x7f7d940bb8e0 2021-02-10T11:33:35.322 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:35.316 7f7d9bc0e700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d940b3d90 con 0x7f7d94195070 2021-02-10T11:33:35.322 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:35.316 7f7d99a2a700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7d941942b0 0x7f7d940b3550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:35.322 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:35.316 7f7d99a2a700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7d941942b0 0x7f7d940b3550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:35412/0 (socket says 172.21.2.102:35412) 2021-02-10T11:33:35.322 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:35.316 7f7d99a2a700 1 -- 172.21.2.102:0/1727722559 learned_addr learned my addr 172.21.2.102:0/1727722559 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:33:35.322 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:35.317 7f7d99a2a700 1 -- 172.21.2.102:0/1727722559 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7d94195070 msgr2=0x7f7d94195470 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:33:35.322 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:35.317 7f7d99a2a700 1 --2- 172.21.2.102:0/1727722559 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7d94195070 0x7f7d94195470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:35.322 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:35.317 7f7d99a2a700 1 -- 172.21.2.102:0/1727722559 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7d940bb8e0 msgr2=0x7f7d940bbce0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:33:35.322 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:35.317 7f7d99a2a700 1 --2- 172.21.2.102:0/1727722559 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7d940bb8e0 0x7f7d940bbce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:35.322 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:35.317 7f7d99a2a700 1 -- 172.21.2.102:0/1727722559 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d941a49e0 con 0x7f7d941942b0 2021-02-10T11:33:35.322 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:35.317 7f7d99a2a700 1 --2- 172.21.2.102:0/1727722559 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7d941942b0 0x7f7d940b3550 secure :-1 s=READY pgs=2423 cs=0 l=1 rev1=1 rx=0x7f7d88006f60 tx=0x7f7d8800b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:35.323 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:35.317 7f7d867fc700 1 -- 172.21.2.102:0/1727722559 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7d880009b0 con 0x7f7d941942b0 2021-02-10T11:33:35.323 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:35.317 7f7d867fc700 1 -- 172.21.2.102:0/1727722559 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d8800bd60 con 0x7f7d941942b0 2021-02-10T11:33:35.323 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:35.317 7f7d867fc700 1 -- 172.21.2.102:0/1727722559 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7d88002030 con 0x7f7d941942b0 2021-02-10T11:33:35.323 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:35.317 7f7d9bc0e700 1 -- 172.21.2.102:0/1727722559 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d941a49e0 con 0x7f7d941942b0 2021-02-10T11:33:35.323 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:35.317 7f7d9bc0e700 1 -- 172.21.2.102:0/1727722559 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d941a49e0 con 0x7f7d941942b0 2021-02-10T11:33:35.323 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:35.318 7f7d867fc700 1 -- 172.21.2.102:0/1727722559 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f7d88000c90 con 0x7f7d941942b0 2021-02-10T11:33:35.323 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:35.318 7f7d867fc700 1 --2- 172.21.2.102:0/1727722559 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7d7c0186c0 0x7f7d7c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:35.323 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:35.318 7f7d867fc700 1 -- 172.21.2.102:0/1727722559 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f7d88032560 con 0x7f7d941942b0 2021-02-10T11:33:35.323 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:35.318 7f7d99229700 1 --2- 172.21.2.102:0/1727722559 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7d7c0186c0 0x7f7d7c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:35.323 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:35.318 7f7d99229700 1 --2- 172.21.2.102:0/1727722559 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7d7c0186c0 0x7f7d7c01abc0 crc :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:35.350 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:35.349 7f7d9bc0e700 1 -- 172.21.2.102:0/1727722559 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d941a49e0 con 0x7f7d941942b0 2021-02-10T11:33:35.353 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:35.351 7f7d867fc700 1 -- 172.21.2.102:0/1727722559 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f7d88006b50 con 0x7f7d941942b0 2021-02-10T11:33:35.483 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:35.481 7f7d9bc0e700 1 -- 172.21.2.102:0/1727722559 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7d941a49e0 con 0x7f7d941942b0 2021-02-10T11:33:35.483 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:35.481 7f7d867fc700 1 -- 172.21.2.102:0/1727722559 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+2102 (secure 0 0 0) 0x7f7d88000c90 con 0x7f7d941942b0 2021-02-10T11:33:35.484 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:33:35.484 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":107,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":11103,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.102:6835/160551241","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6834","nonce":160551241},{"type":"v1","addr":"172.21.2.102:6835","nonce":160551241}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":103}],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:33:32.844442","modified":"2021-02-10 11:33:34.850935","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7882},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7882":{"gid":7882,"name":"b","rank":0,"incarnation":104,"state":"up:active","state_seq":3,"addr":"172.21.2.113:6833/121050001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6832","nonce":121050001},{"type":"v1","addr":"172.21.2.113:6833","nonce":121050001}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[27],"metadata_pool":26,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":13}]} 2021-02-10T11:33:35.484 INFO:teuthology.orchestra.run.gibba002.stderr:dumped fsmap epoch 107 2021-02-10T11:33:35.485 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:35.483 7f7d9bc0e700 1 -- 172.21.2.102:0/1727722559 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7d7c0186c0 msgr2=0x7f7d7c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:35.485 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:35.483 7f7d9bc0e700 1 --2- 172.21.2.102:0/1727722559 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7d7c0186c0 0x7f7d7c01abc0 crc :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:35.486 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:35.483 7f7d9bc0e700 1 -- 172.21.2.102:0/1727722559 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7d941942b0 msgr2=0x7f7d940b3550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:35.486 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:35.483 7f7d9bc0e700 1 --2- 172.21.2.102:0/1727722559 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7d941942b0 0x7f7d940b3550 secure :-1 s=READY pgs=2423 cs=0 l=1 rev1=1 rx=0x7f7d88006f60 tx=0x7f7d8800b970).stop 2021-02-10T11:33:35.486 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:35.484 7f7d9bc0e700 1 -- 172.21.2.102:0/1727722559 shutdown_connections 2021-02-10T11:33:35.486 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:35.484 7f7d9bc0e700 1 --2- 172.21.2.102:0/1727722559 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7d7c0186c0 0x7f7d7c01abc0 unknown :-1 s=CLOSED pgs=1079 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:35.486 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:35.484 7f7d9bc0e700 1 --2- 172.21.2.102:0/1727722559 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7d940bb8e0 0x7f7d940bbce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:35.486 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:35.484 7f7d9bc0e700 1 --2- 172.21.2.102:0/1727722559 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7d94195070 0x7f7d94195470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:35.486 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:35.484 7f7d9bc0e700 1 --2- 172.21.2.102:0/1727722559 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7d941942b0 0x7f7d940b3550 unknown :-1 s=CLOSED pgs=2423 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:35.486 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:35.484 7f7d9bc0e700 1 -- 172.21.2.102:0/1727722559 >> 172.21.2.102:0/1727722559 conn(0x7f7d940b3040 msgr2=0x7f7d940c3250 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:35.486 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:35.484 7f7d9bc0e700 1 -- 172.21.2.102:0/1727722559 shutdown_connections 2021-02-10T11:33:35.486 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:35.484 7f7d9bc0e700 1 -- 172.21.2.102:0/1727722559 wait complete. 2021-02-10T11:33:35.494 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:33:35.924 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:35.905 7fdf2f5e5700 1 Processor -- start 2021-02-10T11:33:35.924 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:35.906 7fdf2f5e5700 1 -- start start 2021-02-10T11:33:35.924 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:35.906 7fdf2f5e5700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdf280d0340 0x7fdf280ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:35.924 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:35.906 7fdf2f5e5700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fdf280d1c70 0x7fdf280efc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:35.924 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:35.906 7fdf2f5e5700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf280f0b10 con 0x7fdf280d1870 2021-02-10T11:33:35.924 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:35.906 7fdf2f5e5700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf280e5ed0 con 0x7fdf280d1c70 2021-02-10T11:33:35.924 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:35.906 7fdf2f5e5700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf281b0be0 con 0x7fdf280d0340 2021-02-10T11:33:35.924 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:35.906 7fdf2d401700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fdf280d1870 0x7fdf281b03f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:55876/0 (socket says 172.21.2.102:55876) 2021-02-10T11:33:35.925 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:35.906 7fdf2d401700 1 -- 172.21.2.102:0/2518758438 learned_addr learned my addr 172.21.2.102:0/2518758438 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:33:35.925 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:35.906 7fdf2dc02700 1 --2- 172.21.2.102:0/2518758438 >> v2:172.21.2.102:3300/0 conn(0x7fdf280d1c70 0x7fdf280efc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:35.925 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:35.906 7fdf2cc00700 1 --2- 172.21.2.102:0/2518758438 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdf280d0340 0x7fdf280ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:35.925 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:35.907 7fdf2dc02700 1 -- 172.21.2.102:0/2518758438 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdf280d0340 msgr2=0x7fdf280ef640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:33:35.925 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:35.907 7fdf2dc02700 1 --2- 172.21.2.102:0/2518758438 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdf280d0340 0x7fdf280ef640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:35.925 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:35.907 7fdf2dc02700 1 -- 172.21.2.102:0/2518758438 >> v1:172.21.2.102:6789/0 conn(0x7fdf280d1870 legacy=0x7fdf281b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:35.925 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:35.907 7fdf2dc02700 1 -- 172.21.2.102:0/2518758438 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf280f20a0 con 0x7fdf280d1c70 2021-02-10T11:33:35.925 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:35.907 7fdf2dc02700 1 --2- 172.21.2.102:0/2518758438 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdf280d1c70 0x7fdf280efc60 secure :-1 s=READY pgs=2425 cs=0 l=1 rev1=1 rx=0x7fdf200068b0 tx=0x7fdf20009ac0).ready entity=mon.0 client_cookie=487eb2a54a0b63a7 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:35.925 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:35.907 7fdf1ffff700 1 -- 172.21.2.102:0/2518758438 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdf2000c0c0 con 0x7fdf280d1c70 2021-02-10T11:33:35.925 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:35.907 7fdf1ffff700 1 -- 172.21.2.102:0/2518758438 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf2000c340 con 0x7fdf280d1c70 2021-02-10T11:33:35.925 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:35.907 7fdf1ffff700 1 -- 172.21.2.102:0/2518758438 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdf2000c860 con 0x7fdf280d1c70 2021-02-10T11:33:35.925 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:35.907 7fdf2f5e5700 1 -- 172.21.2.102:0/2518758438 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdf280d1c70 msgr2=0x7fdf280efc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:35.926 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:35.907 7fdf2f5e5700 1 --2- 172.21.2.102:0/2518758438 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdf280d1c70 0x7fdf280efc60 secure :-1 s=READY pgs=2425 cs=0 l=1 rev1=1 rx=0x7fdf200068b0 tx=0x7fdf20009ac0).stop 2021-02-10T11:33:35.926 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:35.907 7fdf2f5e5700 1 -- 172.21.2.102:0/2518758438 shutdown_connections 2021-02-10T11:33:35.926 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:35.907 7fdf2f5e5700 1 --2- 172.21.2.102:0/2518758438 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdf280d1c70 0x7fdf280efc60 unknown :-1 s=CLOSED pgs=2425 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:35.926 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:35.907 7fdf2f5e5700 1 --2- 172.21.2.102:0/2518758438 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdf280d0340 0x7fdf280ef640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:35.926 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:35.907 7fdf2f5e5700 1 -- 172.21.2.102:0/2518758438 >> 172.21.2.102:0/2518758438 conn(0x7fdf280ce450 msgr2=0x7fdf280d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:35.926 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:35.908 7fdf2f5e5700 1 -- 172.21.2.102:0/2518758438 shutdown_connections 2021-02-10T11:33:35.926 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:35.908 7fdf2f5e5700 1 -- 172.21.2.102:0/2518758438 wait complete. 2021-02-10T11:33:35.927 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:35.923 7fdf2f5e5700 1 Processor -- start 2021-02-10T11:33:35.927 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:35.923 7fdf2f5e5700 1 -- start start 2021-02-10T11:33:35.928 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:35.923 7fdf2f5e5700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdf280ada90 0x7fdf280ade90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:35.928 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:35.923 7fdf2f5e5700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fdf28099d20 0x7fdf2809a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:35.928 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:35.923 7fdf2f5e5700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdf2809a740 0x7fdf28097bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:35.928 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:35.923 7fdf2f5e5700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf280997a0 con 0x7fdf2809a740 2021-02-10T11:33:35.928 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:35.923 7fdf2f5e5700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf2818f0c0 con 0x7fdf28099d20 2021-02-10T11:33:35.928 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:35.923 7fdf2f5e5700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf2809ab40 con 0x7fdf280ada90 2021-02-10T11:33:35.928 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:35.923 7fdf2d401700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdf280ada90 0x7fdf280ade90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:35.928 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:35.923 7fdf2dc02700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdf2809a740 0x7fdf28097bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:35.928 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:35.923 7fdf2d401700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdf280ada90 0x7fdf280ade90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.113:3301/0 says I am v2:172.21.2.102:40962/0 (socket says 172.21.2.102:40962) 2021-02-10T11:33:35.928 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:35.923 7fdf2d401700 1 -- 172.21.2.102:0/3313173618 learned_addr learned my addr 172.21.2.102:0/3313173618 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:33:35.928 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:35.923 7fdf2cc00700 1 --2- 172.21.2.102:0/3313173618 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fdf28099d20 0x7fdf2809a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:35.928 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:35.924 7fdf2dc02700 1 -- 172.21.2.102:0/3313173618 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdf280ada90 msgr2=0x7fdf280ade90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:35.929 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:35.924 7fdf2dc02700 1 --2- 172.21.2.102:0/3313173618 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdf280ada90 0x7fdf280ade90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:35.929 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:35.924 7fdf2dc02700 1 -- 172.21.2.102:0/3313173618 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fdf28099d20 msgr2=0x7fdf2809a120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:35.929 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:35.924 7fdf2dc02700 1 --2- 172.21.2.102:0/3313173618 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fdf28099d20 0x7fdf2809a120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:35.929 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:35.924 7fdf2dc02700 1 -- 172.21.2.102:0/3313173618 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf2809ad60 con 0x7fdf2809a740 2021-02-10T11:33:35.929 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:35.924 7fdf2dc02700 1 --2- 172.21.2.102:0/3313173618 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdf2809a740 0x7fdf28097bc0 secure :-1 s=READY pgs=2426 cs=0 l=1 rev1=1 rx=0x7fdf20006ec0 tx=0x7fdf2000b8f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:35.929 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:35.924 7fdf1dffb700 1 -- 172.21.2.102:0/3313173618 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdf20006240 con 0x7fdf2809a740 2021-02-10T11:33:35.929 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:35.924 7fdf1dffb700 1 -- 172.21.2.102:0/3313173618 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf20009250 con 0x7fdf2809a740 2021-02-10T11:33:35.929 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:35.924 7fdf2f5e5700 1 -- 172.21.2.102:0/3313173618 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf2809ad60 con 0x7fdf2809a740 2021-02-10T11:33:35.929 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:35.924 7fdf2f5e5700 1 -- 172.21.2.102:0/3313173618 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf280984d0 con 0x7fdf2809a740 2021-02-10T11:33:35.929 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:35.924 7fdf1dffb700 1 -- 172.21.2.102:0/3313173618 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdf200057c0 con 0x7fdf2809a740 2021-02-10T11:33:35.929 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:35.925 7fdf1dffb700 1 -- 172.21.2.102:0/3313173618 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fdf20006030 con 0x7fdf2809a740 2021-02-10T11:33:35.929 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:35.925 7fdf1dffb700 1 --2- 172.21.2.102:0/3313173618 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fdf080186c0 0x7fdf0801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:35.929 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:35.925 7fdf1dffb700 1 -- 172.21.2.102:0/3313173618 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fdf200324a0 con 0x7fdf2809a740 2021-02-10T11:33:35.930 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:35.925 7fdf2d401700 1 --2- 172.21.2.102:0/3313173618 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fdf080186c0 0x7fdf0801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:35.930 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:35.925 7fdf2d401700 1 --2- 172.21.2.102:0/3313173618 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fdf080186c0 0x7fdf0801abc0 crc :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:35.957 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:35.956 7fdf2f5e5700 1 -- 172.21.2.102:0/3313173618 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf280ebc00 con 0x7fdf2809a740 2021-02-10T11:33:35.960 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:35.958 7fdf1dffb700 1 -- 172.21.2.102:0/3313173618 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fdf20030c50 con 0x7fdf2809a740 2021-02-10T11:33:36.084 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:36.082 7fdf2f5e5700 1 -- 172.21.2.102:0/3313173618 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdf280f0b10 con 0x7fdf2809a740 2021-02-10T11:33:36.084 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:36.083 7fdf1dffb700 1 -- 172.21.2.102:0/3313173618 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v128) v1 ==== 74+0+14007 (secure 0 0 0) 0x7fdf20006030 con 0x7fdf2809a740 2021-02-10T11:33:36.085 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:33:36.085 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":128,"fsid":"68db30ea-dbce-4aef-a6df-f01c6fea0513","created":"2021-02-10 11:03:06.331557","modified":"2021-02-10 11:33:33.846924","last_up_change":"2021-02-10 11:03:32.690562","last_in_change":"2021-02-10 11:03:32.690562","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":27,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:03:37.322519","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":26,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:33:30.653221","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"127","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":27,"pool_name":"cephfs_data","create_time":"2021-02-10 11:33:31.831765","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"128","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"be691b4d-a039-408a-af80-9d53f02b6f1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6818","nonce":36088},{"type":"v1","addr":"172.21.2.102:6820","nonce":36088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6822","nonce":36088},{"type":"v1","addr":"172.21.2.102:6824","nonce":36088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6830","nonce":36088},{"type":"v1","addr":"172.21.2.102:6832","nonce":36088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6826","nonce":36088},{"type":"v1","addr":"172.21.2.102:6828","nonce":36088}]},"public_addr":"172.21.2.102:6820/36088","cluster_addr":"172.21.2.102:6824/36088","heartbeat_back_addr":"172.21.2.102:6832/36088","heartbeat_front_addr":"172.21.2.102:6828/36088","state":["exists","up"]},{"osd":1,"uuid":"64e3ccc8-9c69-4d7c-865e-af8e83822635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6819","nonce":36087},{"type":"v1","addr":"172.21.2.102:6821","nonce":36087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6823","nonce":36087},{"type":"v1","addr":"172.21.2.102:6825","nonce":36087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6831","nonce":36087},{"type":"v1","addr":"172.21.2.102:6833","nonce":36087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6827","nonce":36087},{"type":"v1","addr":"172.21.2.102:6829","nonce":36087}]},"public_addr":"172.21.2.102:6821/36087","cluster_addr":"172.21.2.102:6825/36087","heartbeat_back_addr":"172.21.2.102:6833/36087","heartbeat_front_addr":"172.21.2.102:6829/36087","state":["exists","up"]},{"osd":2,"uuid":"153700af-747f-481f-bdf4-d6072d3a842f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6810","nonce":36089},{"type":"v1","addr":"172.21.2.102:6811","nonce":36089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6812","nonce":36089},{"type":"v1","addr":"172.21.2.102:6813","nonce":36089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6816","nonce":36089},{"type":"v1","addr":"172.21.2.102:6817","nonce":36089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6814","nonce":36089},{"type":"v1","addr":"172.21.2.102:6815","nonce":36089}]},"public_addr":"172.21.2.102:6811/36089","cluster_addr":"172.21.2.102:6813/36089","heartbeat_back_addr":"172.21.2.102:6817/36089","heartbeat_front_addr":"172.21.2.102:6815/36089","state":["exists","up"]},{"osd":3,"uuid":"cbc678f4-83a9-45e6-8830-2f7c232b1ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6802","nonce":36090},{"type":"v1","addr":"172.21.2.102:6803","nonce":36090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6804","nonce":36090},{"type":"v1","addr":"172.21.2.102:6805","nonce":36090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6808","nonce":36090},{"type":"v1","addr":"172.21.2.102:6809","nonce":36090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6806","nonce":36090},{"type":"v1","addr":"172.21.2.102:6807","nonce":36090}]},"public_addr":"172.21.2.102:6803/36090","cluster_addr":"172.21.2.102:6805/36090","heartbeat_back_addr":"172.21.2.102:6809/36090","heartbeat_front_addr":"172.21.2.102:6807/36090","state":["exists","up"]},{"osd":4,"uuid":"c01e4008-b258-4f10-afd2-9e7f37214351","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6808","nonce":36368},{"type":"v1","addr":"172.21.2.113:6811","nonce":36368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6814","nonce":36368},{"type":"v1","addr":"172.21.2.113:6818","nonce":36368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6826","nonce":36368},{"type":"v1","addr":"172.21.2.113:6828","nonce":36368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6820","nonce":36368},{"type":"v1","addr":"172.21.2.113:6824","nonce":36368}]},"public_addr":"172.21.2.113:6811/36368","cluster_addr":"172.21.2.113:6818/36368","heartbeat_back_addr":"172.21.2.113:6828/36368","heartbeat_front_addr":"172.21.2.113:6824/36368","state":["exists","up"]},{"osd":5,"uuid":"551f6007-daf8-4a98-8ee0-c2b16bcccb5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6802","nonce":36366},{"type":"v1","addr":"172.21.2.113:6805","nonce":36366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6807","nonce":36366},{"type":"v1","addr":"172.21.2.113:6810","nonce":36366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6819","nonce":36366},{"type":"v1","addr":"172.21.2.113:6822","nonce":36366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6812","nonce":36366},{"type":"v1","addr":"172.21.2.113:6815","nonce":36366}]},"public_addr":"172.21.2.113:6805/36366","cluster_addr":"172.21.2.113:6810/36366","heartbeat_back_addr":"172.21.2.113:6822/36366","heartbeat_front_addr":"172.21.2.113:6815/36366","state":["exists","up"]},{"osd":6,"uuid":"bf67c040-0e50-48be-b102-c3943bc9ebef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6800","nonce":36367},{"type":"v1","addr":"172.21.2.113:6801","nonce":36367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6803","nonce":36367},{"type":"v1","addr":"172.21.2.113:6804","nonce":36367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6813","nonce":36367},{"type":"v1","addr":"172.21.2.113:6816","nonce":36367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6806","nonce":36367},{"type":"v1","addr":"172.21.2.113:6809","nonce":36367}]},"public_addr":"172.21.2.113:6801/36367","cluster_addr":"172.21.2.113:6804/36367","heartbeat_back_addr":"172.21.2.113:6816/36367","heartbeat_front_addr":"172.21.2.113:6809/36367","state":["exists","up"]},{"osd":7,"uuid":"f5461f80-018d-428f-be89-2eb2bcbe7bb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6817","nonce":36365},{"type":"v1","addr":"172.21.2.113:6821","nonce":36365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6823","nonce":36365},{"type":"v1","addr":"172.21.2.113:6825","nonce":36365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6830","nonce":36365},{"type":"v1","addr":"172.21.2.113:6831","nonce":36365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6827","nonce":36365},{"type":"v1","addr":"172.21.2.113:6829","nonce":36365}]},"public_addr":"172.21.2.113:6821/36365","cluster_addr":"172.21.2.113:6825/36365","heartbeat_back_addr":"172.21.2.113:6831/36365","heartbeat_front_addr":"172.21.2.113:6829/36365","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:33:36.087 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:36.084 7fdf2f5e5700 1 -- 172.21.2.102:0/3313173618 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fdf080186c0 msgr2=0x7fdf0801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:36.087 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:36.084 7fdf2f5e5700 1 --2- 172.21.2.102:0/3313173618 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fdf080186c0 0x7fdf0801abc0 crc :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:36.087 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:36.084 7fdf2f5e5700 1 -- 172.21.2.102:0/3313173618 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdf2809a740 msgr2=0x7fdf28097bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:36.087 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:36.084 7fdf2f5e5700 1 --2- 172.21.2.102:0/3313173618 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdf2809a740 0x7fdf28097bc0 secure :-1 s=READY pgs=2426 cs=0 l=1 rev1=1 rx=0x7fdf20006ec0 tx=0x7fdf2000b8f0).stop 2021-02-10T11:33:36.087 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:36.085 7fdf2f5e5700 1 -- 172.21.2.102:0/3313173618 shutdown_connections 2021-02-10T11:33:36.087 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:36.085 7fdf2f5e5700 1 --2- 172.21.2.102:0/3313173618 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdf2809a740 0x7fdf28097bc0 unknown :-1 s=CLOSED pgs=2426 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:36.087 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:36.085 7fdf2f5e5700 1 --2- 172.21.2.102:0/3313173618 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fdf080186c0 0x7fdf0801abc0 unknown :-1 s=CLOSED pgs=1080 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:36.087 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:36.085 7fdf2f5e5700 1 --2- 172.21.2.102:0/3313173618 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fdf28099d20 0x7fdf2809a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:36.087 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:36.085 7fdf2f5e5700 1 --2- 172.21.2.102:0/3313173618 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdf280ada90 0x7fdf280ade90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:36.087 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:36.085 7fdf2f5e5700 1 -- 172.21.2.102:0/3313173618 >> 172.21.2.102:0/3313173618 conn(0x7fdf280f0550 msgr2=0x7fdf28191730 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:36.087 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:36.085 7fdf2f5e5700 1 -- 172.21.2.102:0/3313173618 shutdown_connections 2021-02-10T11:33:36.088 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:36.085 7fdf2f5e5700 1 -- 172.21.2.102:0/3313173618 wait complete. 2021-02-10T11:33:36.095 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:33:36.535 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:36.516 7f496f5df700 1 Processor -- start 2021-02-10T11:33:36.535 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:36.516 7f496f5df700 1 -- start start 2021-02-10T11:33:36.535 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:36.516 7f496f5df700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f49680d1880 0x7f49681b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:36.535 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:36.516 7f496f5df700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f49680cff80 0x7f4968109140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:36.535 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:36.516 7f496f5df700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4968109a70 con 0x7f49681b0af0 2021-02-10T11:33:36.535 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:36.516 7f496f5df700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4968109760 con 0x7f49680cff80 2021-02-10T11:33:36.535 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:36.516 7f496f5df700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4968088070 con 0x7f49680d1880 2021-02-10T11:33:36.536 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:36.516 7f496cbfa700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f49681b0af0 0x7f4968108950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:55888/0 (socket says 172.21.2.102:55888) 2021-02-10T11:33:36.536 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:36.517 7f496cbfa700 1 -- 172.21.2.102:0/379832907 learned_addr learned my addr 172.21.2.102:0/379832907 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:33:36.536 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:36.517 7f496dbfc700 1 --2- 172.21.2.102:0/379832907 >> v2:172.21.2.102:3300/0 conn(0x7f49680cff80 0x7f4968109140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:36.536 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:36.517 7f495ffff700 1 -- 172.21.2.102:0/379832907 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1343491921 0 0) 0x7f49580009b0 con 0x7f49681b0af0 2021-02-10T11:33:36.536 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:36.517 7f495ffff700 1 -- 172.21.2.102:0/379832907 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4954002fe0 con 0x7f49681b0af0 2021-02-10T11:33:36.536 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:36.517 7f496dbfc700 1 -- 172.21.2.102:0/379832907 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f49680d1880 msgr2=0x7f49681b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:33:36.536 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:36.517 7f496dbfc700 1 --2- 172.21.2.102:0/379832907 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f49680d1880 0x7f49681b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:36.536 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:36.517 7f496dbfc700 1 -- 172.21.2.102:0/379832907 >> v1:172.21.2.102:6789/0 conn(0x7f49681b0af0 legacy=0x7f4968108950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:36.536 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:36.517 7f496dbfc700 1 -- 172.21.2.102:0/379832907 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4968088260 con 0x7f49680cff80 2021-02-10T11:33:36.536 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:36.517 7f496dbfc700 1 --2- 172.21.2.102:0/379832907 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f49680cff80 0x7f4968109140 secure :-1 s=READY pgs=2428 cs=0 l=1 rev1=1 rx=0x7f4960006e30 tx=0x7f4960009e60).ready entity=mon.0 client_cookie=c0afeed43a14db6b server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:36.536 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:36.517 7f495ffff700 1 -- 172.21.2.102:0/379832907 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4960006a50 con 0x7f49680cff80 2021-02-10T11:33:36.536 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:36.518 7f495ffff700 1 -- 172.21.2.102:0/379832907 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f496000b300 con 0x7f49680cff80 2021-02-10T11:33:36.536 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:36.518 7f495ffff700 1 -- 172.21.2.102:0/379832907 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f496000b820 con 0x7f49680cff80 2021-02-10T11:33:36.536 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:36.518 7f496f5df700 1 -- 172.21.2.102:0/379832907 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f49680cff80 msgr2=0x7f4968109140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:36.537 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:36.518 7f496f5df700 1 --2- 172.21.2.102:0/379832907 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f49680cff80 0x7f4968109140 secure :-1 s=READY pgs=2428 cs=0 l=1 rev1=1 rx=0x7f4960006e30 tx=0x7f4960009e60).stop 2021-02-10T11:33:36.537 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:36.518 7f496f5df700 1 -- 172.21.2.102:0/379832907 shutdown_connections 2021-02-10T11:33:36.537 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:36.518 7f496f5df700 1 --2- 172.21.2.102:0/379832907 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f49680cff80 0x7f4968109140 unknown :-1 s=CLOSED pgs=2428 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:36.537 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:36.518 7f496f5df700 1 --2- 172.21.2.102:0/379832907 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f49680d1880 0x7f49681b04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:36.537 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:36.518 7f496f5df700 1 -- 172.21.2.102:0/379832907 >> 172.21.2.102:0/379832907 conn(0x7f49680ce430 msgr2=0x7f49680d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:36.537 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:36.519 7f496f5df700 1 -- 172.21.2.102:0/379832907 shutdown_connections 2021-02-10T11:33:36.537 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:36.519 7f496f5df700 1 -- 172.21.2.102:0/379832907 wait complete. 2021-02-10T11:33:36.538 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:36.534 7f496f5df700 1 Processor -- start 2021-02-10T11:33:36.538 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:36.534 7f496f5df700 1 -- start start 2021-02-10T11:33:36.539 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:36.534 7f496f5df700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4968085300 0x7f49680bf1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:36.539 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:36.534 7f496f5df700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f49680bf7d0 0x7f4968085fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:36.539 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:36.534 7f496f5df700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f49680bfbd0 0x7f49680865d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:36.539 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:36.534 7f496f5df700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4968084d80 con 0x7f49680bfbd0 2021-02-10T11:33:36.539 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:36.534 7f496f5df700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4968085700 con 0x7f4968085300 2021-02-10T11:33:36.539 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:36.534 7f496f5df700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4968086bf0 con 0x7f49680bf7d0 2021-02-10T11:33:36.539 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:36.534 7f496dbfc700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f49680bfbd0 0x7f49680865d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:36.539 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:36.534 7f496dbfc700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f49680bfbd0 0x7f49680865d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:35442/0 (socket says 172.21.2.102:35442) 2021-02-10T11:33:36.539 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:36.534 7f496dbfc700 1 -- 172.21.2.102:0/1112306057 learned_addr learned my addr 172.21.2.102:0/1112306057 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:33:36.539 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:36.535 7f496d3fb700 1 --2- 172.21.2.102:0/1112306057 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4968085300 0x7f49680bf1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:36.539 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:36.535 7f496cbfa700 1 --2- 172.21.2.102:0/1112306057 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f49680bf7d0 0x7f4968085fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:36.539 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:36.535 7f496dbfc700 1 -- 172.21.2.102:0/1112306057 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f49680bf7d0 msgr2=0x7f4968085fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:36.539 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:36.535 7f496dbfc700 1 --2- 172.21.2.102:0/1112306057 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f49680bf7d0 0x7f4968085fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:36.540 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:36.535 7f496dbfc700 1 -- 172.21.2.102:0/1112306057 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4968085300 msgr2=0x7f49680bf1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:36.540 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:36.535 7f496dbfc700 1 --2- 172.21.2.102:0/1112306057 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4968085300 0x7f49680bf1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:36.540 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:36.535 7f496dbfc700 1 -- 172.21.2.102:0/1112306057 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4968086e10 con 0x7f49680bfbd0 2021-02-10T11:33:36.540 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:36.535 7f496dbfc700 1 --2- 172.21.2.102:0/1112306057 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f49680bfbd0 0x7f49680865d0 secure :-1 s=READY pgs=2429 cs=0 l=1 rev1=1 rx=0x7f496000b970 tx=0x7f496000bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:36.540 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:36.535 7f495dffb700 1 -- 172.21.2.102:0/1112306057 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4960009380 con 0x7f49680bfbd0 2021-02-10T11:33:36.540 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:36.535 7f495dffb700 1 -- 172.21.2.102:0/1112306057 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4960000ff0 con 0x7f49680bfbd0 2021-02-10T11:33:36.540 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:36.535 7f495dffb700 1 -- 172.21.2.102:0/1112306057 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4960005a50 con 0x7f49680bfbd0 2021-02-10T11:33:36.540 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:36.535 7f496f5df700 1 -- 172.21.2.102:0/1112306057 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4968086e10 con 0x7f49680bfbd0 2021-02-10T11:33:36.540 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:36.535 7f496f5df700 1 -- 172.21.2.102:0/1112306057 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4968086e10 con 0x7f49680bfbd0 2021-02-10T11:33:36.540 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:36.536 7f495dffb700 1 -- 172.21.2.102:0/1112306057 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4960005820 con 0x7f49680bfbd0 2021-02-10T11:33:36.540 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:36.536 7f495dffb700 1 --2- 172.21.2.102:0/1112306057 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f49480186c0 0x7f494801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:36.540 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:36.536 7f495dffb700 1 -- 172.21.2.102:0/1112306057 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f4960032a20 con 0x7f49680bfbd0 2021-02-10T11:33:36.540 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:36.536 7f496d3fb700 1 --2- 172.21.2.102:0/1112306057 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f49480186c0 0x7f494801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:36.541 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:36.536 7f496d3fb700 1 --2- 172.21.2.102:0/1112306057 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f49480186c0 0x7f494801abc0 crc :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:36.568 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:36.566 7f496f5df700 1 -- 172.21.2.102:0/1112306057 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49680829a0 con 0x7f49680bfbd0 2021-02-10T11:33:36.570 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:36.569 7f495dffb700 1 -- 172.21.2.102:0/1112306057 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f4960009380 con 0x7f49680bfbd0 2021-02-10T11:33:36.700 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:36.699 7f496f5df700 1 -- 172.21.2.102:0/1112306057 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f4968109a70 con 0x7f49680bfbd0 2021-02-10T11:33:36.704 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:36.702 7f495dffb700 1 -- 172.21.2.102:0/1112306057 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.0 v73) v1 ==== 184+0+0 (secure 0 0 0) 0x7f4960000f60 con 0x7f49680bfbd0 2021-02-10T11:33:36.710 INFO:teuthology.orchestra.run.gibba002.stderr:updated caps for client.0 2021-02-10T11:33:36.711 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:36.707 7f496f5df700 1 -- 172.21.2.102:0/1112306057 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f49480186c0 msgr2=0x7f494801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:36.711 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:36.707 7f496f5df700 1 --2- 172.21.2.102:0/1112306057 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f49480186c0 0x7f494801abc0 crc :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:36.711 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:36.707 7f496f5df700 1 -- 172.21.2.102:0/1112306057 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f49680bfbd0 msgr2=0x7f49680865d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:36.711 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:36.707 7f496f5df700 1 --2- 172.21.2.102:0/1112306057 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f49680bfbd0 0x7f49680865d0 secure :-1 s=READY pgs=2429 cs=0 l=1 rev1=1 rx=0x7f496000b970 tx=0x7f496000bd10).stop 2021-02-10T11:33:36.712 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:36.708 7f496f5df700 1 -- 172.21.2.102:0/1112306057 shutdown_connections 2021-02-10T11:33:36.712 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:36.708 7f496f5df700 1 --2- 172.21.2.102:0/1112306057 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f49680bfbd0 0x7f49680865d0 unknown :-1 s=CLOSED pgs=2429 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:36.712 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:36.708 7f496f5df700 1 --2- 172.21.2.102:0/1112306057 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f49680bf7d0 0x7f4968085fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:36.712 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:36.708 7f496f5df700 1 --2- 172.21.2.102:0/1112306057 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f49480186c0 0x7f494801abc0 unknown :-1 s=CLOSED pgs=1081 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:36.712 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:36.708 7f496f5df700 1 --2- 172.21.2.102:0/1112306057 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4968085300 0x7f49680bf1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:36.712 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:36.708 7f496f5df700 1 -- 172.21.2.102:0/1112306057 >> 172.21.2.102:0/1112306057 conn(0x7f49680bd7a0 msgr2=0x7f49680b5900 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:36.712 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:36.708 7f496f5df700 1 -- 172.21.2.102:0/1112306057 shutdown_connections 2021-02-10T11:33:36.712 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:36.708 7f496f5df700 1 -- 172.21.2.102:0/1112306057 wait complete. 2021-02-10T11:33:36.718 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.1 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:33:37.157 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:37.140 7f6dbda97700 1 Processor -- start 2021-02-10T11:33:37.157 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:37.140 7f6dbda97700 1 -- start start 2021-02-10T11:33:37.157 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:37.140 7f6dbda97700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6db80d1880 0x7f6db81b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:37.157 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:37.140 7f6dbda97700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f6db81b0af0 0x7f6db8108950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:37.157 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:37.140 7f6dbda97700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6db8109a70 con 0x7f6db80d1c80 2021-02-10T11:33:37.158 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:37.140 7f6dbda97700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6db80e5f00 con 0x7f6db81b0af0 2021-02-10T11:33:37.158 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:37.140 7f6dbda97700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6db8109760 con 0x7f6db80d1880 2021-02-10T11:33:37.158 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:37.140 7f6db6ffd700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f6db81b0af0 0x7f6db8108950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:37.158 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:37.140 7f6db7fff700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f6db80d1c80 0x7f6db8108f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:55902/0 (socket says 172.21.2.102:55902) 2021-02-10T11:33:37.158 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:37.140 7f6db6ffd700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f6db81b0af0 0x7f6db8108950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:35446/0 (socket says 172.21.2.102:35446) 2021-02-10T11:33:37.158 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:37.140 7f6db7fff700 1 -- 172.21.2.102:0/117115341 learned_addr learned my addr 172.21.2.102:0/117115341 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:33:37.158 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:37.141 7f6db6ffd700 1 -- 172.21.2.102:0/117115341 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6db80d1880 msgr2=0x7f6db81b04d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:33:37.158 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:37.141 7f6db6ffd700 1 --2- 172.21.2.102:0/117115341 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6db80d1880 0x7f6db81b04d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:37.158 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:37.141 7f6db6ffd700 1 -- 172.21.2.102:0/117115341 >> v1:172.21.2.102:6789/0 conn(0x7f6db80d1c80 legacy=0x7f6db8108f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:37.158 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:37.141 7f6db6ffd700 1 -- 172.21.2.102:0/117115341 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6db8088070 con 0x7f6db81b0af0 2021-02-10T11:33:37.158 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:37.141 7f6db6ffd700 1 --2- 172.21.2.102:0/117115341 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6db81b0af0 0x7f6db8108950 secure :-1 s=READY pgs=2431 cs=0 l=1 rev1=1 rx=0x7f6da8006650 tx=0x7f6da8009a80).ready entity=mon.0 client_cookie=2c93ba35fafe26a2 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:37.158 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:37.141 7f6db67fc700 1 -- 172.21.2.102:0/117115341 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6da800c0c0 con 0x7f6db81b0af0 2021-02-10T11:33:37.159 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:37.141 7f6db67fc700 1 -- 172.21.2.102:0/117115341 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6da800c340 con 0x7f6db81b0af0 2021-02-10T11:33:37.159 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:37.141 7f6db67fc700 1 -- 172.21.2.102:0/117115341 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6da800c860 con 0x7f6db81b0af0 2021-02-10T11:33:37.159 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:37.141 7f6dbda97700 1 -- 172.21.2.102:0/117115341 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6db81b0af0 msgr2=0x7f6db8108950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:37.159 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:37.141 7f6dbda97700 1 --2- 172.21.2.102:0/117115341 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6db81b0af0 0x7f6db8108950 secure :-1 s=READY pgs=2431 cs=0 l=1 rev1=1 rx=0x7f6da8006650 tx=0x7f6da8009a80).stop 2021-02-10T11:33:37.159 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:37.141 7f6dbda97700 1 -- 172.21.2.102:0/117115341 shutdown_connections 2021-02-10T11:33:37.159 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:37.141 7f6dbda97700 1 --2- 172.21.2.102:0/117115341 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6db81b0af0 0x7f6db8108950 unknown :-1 s=CLOSED pgs=2431 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:37.159 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:37.141 7f6dbda97700 1 --2- 172.21.2.102:0/117115341 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6db80d1880 0x7f6db81b04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:37.159 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:37.142 7f6dbda97700 1 -- 172.21.2.102:0/117115341 >> 172.21.2.102:0/117115341 conn(0x7f6db80ce430 msgr2=0x7f6db80d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:37.159 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:37.142 7f6dbda97700 1 -- 172.21.2.102:0/117115341 shutdown_connections 2021-02-10T11:33:37.159 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:37.142 7f6dbda97700 1 -- 172.21.2.102:0/117115341 wait complete. 2021-02-10T11:33:37.160 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:37.156 7f6dbda97700 1 Processor -- start 2021-02-10T11:33:37.161 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:37.156 7f6dbda97700 1 -- start start 2021-02-10T11:33:37.161 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:37.156 7f6dbda97700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6db81b9470 0x7f6db81ae6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:37.161 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:37.156 7f6dbda97700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6db81aecf0 0x7f6db80957f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:37.161 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:37.156 7f6dbda97700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6db81af0f0 0x7f6db8095e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:37.161 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:37.156 7f6dbda97700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6db8096d70 con 0x7f6db81af0f0 2021-02-10T11:33:37.161 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:37.156 7f6dbda97700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6db80a7780 con 0x7f6db81aecf0 2021-02-10T11:33:37.161 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:37.156 7f6dbda97700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6db81b9870 con 0x7f6db81b9470 2021-02-10T11:33:37.161 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:37.157 7f6db7fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6db81af0f0 0x7f6db8095e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:37.161 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:37.157 7f6db7fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6db81af0f0 0x7f6db8095e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:35452/0 (socket says 172.21.2.102:35452) 2021-02-10T11:33:37.161 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:37.157 7f6db7fff700 1 -- 172.21.2.102:0/3133340869 learned_addr learned my addr 172.21.2.102:0/3133340869 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:33:37.161 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:37.157 7f6db7fff700 1 -- 172.21.2.102:0/3133340869 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6db81b9470 msgr2=0x7f6db81ae6d0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:33:37.161 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:37.157 7f6db7fff700 1 --2- 172.21.2.102:0/3133340869 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6db81b9470 0x7f6db81ae6d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:37.161 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:37.157 7f6db7fff700 1 -- 172.21.2.102:0/3133340869 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6db81aecf0 msgr2=0x7f6db80957f0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:33:37.162 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:37.157 7f6db7fff700 1 --2- 172.21.2.102:0/3133340869 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6db81aecf0 0x7f6db80957f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:37.162 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:37.157 7f6db7fff700 1 -- 172.21.2.102:0/3133340869 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6db819e6b0 con 0x7f6db81af0f0 2021-02-10T11:33:37.162 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:37.157 7f6db7fff700 1 --2- 172.21.2.102:0/3133340869 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6db81af0f0 0x7f6db8095e10 secure :-1 s=READY pgs=2432 cs=0 l=1 rev1=1 rx=0x7f6db000a940 tx=0x7f6db000ace0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:37.162 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:37.157 7f6daffff700 1 -- 172.21.2.102:0/3133340869 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6db0000d30 con 0x7f6db81af0f0 2021-02-10T11:33:37.162 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:37.157 7f6daffff700 1 -- 172.21.2.102:0/3133340869 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6db0007020 con 0x7f6db81af0f0 2021-02-10T11:33:37.162 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:37.157 7f6dbda97700 1 -- 172.21.2.102:0/3133340869 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6db819e6b0 con 0x7f6db81af0f0 2021-02-10T11:33:37.162 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:37.157 7f6dbda97700 1 -- 172.21.2.102:0/3133340869 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6db8096720 con 0x7f6db81af0f0 2021-02-10T11:33:37.162 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:37.157 7f6daffff700 1 -- 172.21.2.102:0/3133340869 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6db0007540 con 0x7f6db81af0f0 2021-02-10T11:33:37.162 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:37.158 7f6daffff700 1 -- 172.21.2.102:0/3133340869 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f6db0007200 con 0x7f6db81af0f0 2021-02-10T11:33:37.162 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:37.158 7f6daffff700 1 --2- 172.21.2.102:0/3133340869 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6d9c0186d0 0x7f6d9c01abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:37.162 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:37.158 7f6daffff700 1 -- 172.21.2.102:0/3133340869 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f6db0032b70 con 0x7f6db81af0f0 2021-02-10T11:33:37.162 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:37.158 7f6db77fe700 1 --2- 172.21.2.102:0/3133340869 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6d9c0186d0 0x7f6d9c01abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:37.162 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:37.158 7f6db77fe700 1 --2- 172.21.2.102:0/3133340869 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6d9c0186d0 0x7f6d9c01abd0 crc :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:37.192 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:37.190 7f6dbda97700 1 -- 172.21.2.102:0/3133340869 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6db80829a0 con 0x7f6db81af0f0 2021-02-10T11:33:37.194 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:37.192 7f6daffff700 1 -- 172.21.2.102:0/3133340869 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f6db0001270 con 0x7f6db81af0f0 2021-02-10T11:33:37.323 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:37.322 7f6dbda97700 1 -- 172.21.2.102:0/3133340869 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f6db80a7780 con 0x7f6db81af0f0 2021-02-10T11:33:37.327 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:37.325 7f6daffff700 1 -- 172.21.2.102:0/3133340869 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.1 v74) v1 ==== 184+0+0 (secure 0 0 0) 0x7f6db00071c0 con 0x7f6db81af0f0 2021-02-10T11:33:37.330 INFO:teuthology.orchestra.run.gibba002.stderr:updated caps for client.1 2021-02-10T11:33:37.332 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:37.330 7f6dbda97700 1 -- 172.21.2.102:0/3133340869 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6d9c0186d0 msgr2=0x7f6d9c01abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:37.332 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:37.330 7f6dbda97700 1 --2- 172.21.2.102:0/3133340869 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6d9c0186d0 0x7f6d9c01abd0 crc :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:37.332 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:37.331 7f6dbda97700 1 -- 172.21.2.102:0/3133340869 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6db81af0f0 msgr2=0x7f6db8095e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:37.333 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:37.331 7f6dbda97700 1 --2- 172.21.2.102:0/3133340869 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6db81af0f0 0x7f6db8095e10 secure :-1 s=READY pgs=2432 cs=0 l=1 rev1=1 rx=0x7f6db000a940 tx=0x7f6db000ace0).stop 2021-02-10T11:33:37.333 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:37.331 7f6dbda97700 1 -- 172.21.2.102:0/3133340869 shutdown_connections 2021-02-10T11:33:37.333 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:37.331 7f6dbda97700 1 --2- 172.21.2.102:0/3133340869 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6db81af0f0 0x7f6db8095e10 unknown :-1 s=CLOSED pgs=2432 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:37.333 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:37.331 7f6dbda97700 1 --2- 172.21.2.102:0/3133340869 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6d9c0186d0 0x7f6d9c01abd0 unknown :-1 s=CLOSED pgs=1082 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:37.333 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:37.331 7f6dbda97700 1 --2- 172.21.2.102:0/3133340869 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6db81aecf0 0x7f6db80957f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:37.333 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:37.331 7f6dbda97700 1 --2- 172.21.2.102:0/3133340869 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6db81b9470 0x7f6db81ae6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:37.333 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:37.331 7f6dbda97700 1 -- 172.21.2.102:0/3133340869 >> 172.21.2.102:0/3133340869 conn(0x7f6db80845b0 msgr2=0x7f6db818c290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:37.333 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:37.331 7f6dbda97700 1 -- 172.21.2.102:0/3133340869 shutdown_connections 2021-02-10T11:33:37.334 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:37.331 7f6dbda97700 1 -- 172.21.2.102:0/3133340869 wait complete. 2021-02-10T11:33:37.341 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.2 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:33:37.783 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:37.762 7ff3efae1700 1 Processor -- start 2021-02-10T11:33:37.783 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:37.762 7ff3efae1700 1 -- start start 2021-02-10T11:33:37.783 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:37.763 7ff3efae1700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7ff3e80d1880 0x7ff3e81b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:37.783 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:37.763 7ff3efae1700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff3e81b0af0 0x7ff3e8108950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:37.783 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:37.763 7ff3efae1700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3e80fe1d0 con 0x7ff3e80d1c80 2021-02-10T11:33:37.783 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:37.763 7ff3efae1700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3e8109a70 con 0x7ff3e80d1880 2021-02-10T11:33:37.783 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:37.763 7ff3efae1700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3e8109760 con 0x7ff3e81b0af0 2021-02-10T11:33:37.783 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:37.763 7ff3ee0fe700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7ff3e80d1c80 0x7ff3e8108f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:55914/0 (socket says 172.21.2.102:55914) 2021-02-10T11:33:37.783 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:37.763 7ff3ee0fe700 1 -- 172.21.2.102:0/1651726218 learned_addr learned my addr 172.21.2.102:0/1651726218 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:33:37.783 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:37.764 7ff3ec8fb700 1 -- 172.21.2.102:0/1651726218 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3058443890 0 0) 0x7ff3e0001500 con 0x7ff3e80d1c80 2021-02-10T11:33:37.784 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:37.764 7ff3ec8fb700 1 -- 172.21.2.102:0/1651726218 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3d4002fe0 con 0x7ff3e80d1c80 2021-02-10T11:33:37.784 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:37.764 7ff3ed8fd700 1 --2- 172.21.2.102:0/1651726218 >> v2:172.21.2.113:3300/0 conn(0x7ff3e80d1880 0x7ff3e81b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:37.784 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:37.764 7ff3ed0fc700 1 --2- 172.21.2.102:0/1651726218 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff3e81b0af0 0x7ff3e8108950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:37.784 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:37.764 7ff3ec8fb700 1 -- 172.21.2.102:0/1651726218 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1965118105 0 0) 0x7ff3e0001460 con 0x7ff3e80d1c80 2021-02-10T11:33:37.784 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:37.764 7ff3ec8fb700 1 -- 172.21.2.102:0/1651726218 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff3e81b0af0 msgr2=0x7ff3e8108950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:33:37.784 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:37.764 7ff3ec8fb700 1 --2- 172.21.2.102:0/1651726218 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff3e81b0af0 0x7ff3e8108950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:37.784 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:37.764 7ff3ec8fb700 1 -- 172.21.2.102:0/1651726218 >> v2:172.21.2.113:3300/0 conn(0x7ff3e80d1880 msgr2=0x7ff3e81b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:33:37.784 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:37.764 7ff3ec8fb700 1 --2- 172.21.2.102:0/1651726218 >> v2:172.21.2.113:3300/0 conn(0x7ff3e80d1880 0x7ff3e81b04d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:37.784 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:37.764 7ff3ec8fb700 1 -- 172.21.2.102:0/1651726218 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3e8088070 con 0x7ff3e80d1c80 2021-02-10T11:33:37.784 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:37.764 7ff3ec8fb700 1 -- 172.21.2.102:0/1651726218 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7ff3e0001980 con 0x7ff3e80d1c80 2021-02-10T11:33:37.784 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:37.764 7ff3ec8fb700 1 -- 172.21.2.102:0/1651726218 >> v1:172.21.2.102:6789/0 conn(0x7ff3e80d1c80 legacy=0x7ff3e8108f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:37.784 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:37.764 7ff3ec8fb700 1 --2- 172.21.2.102:0/1651726218 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff3d4003af0 0x7ff3d4005f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:37.784 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:37.764 7ff3ec8fb700 1 --2- 172.21.2.102:0/1651726218 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff3d404a790 0x7ff3d404cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:37.784 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:37.764 7ff3ec8fb700 1 --2- 172.21.2.102:0/1651726218 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff3d404d590 0x7ff3d404fa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:37.785 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:37.764 7ff3ec8fb700 1 -- 172.21.2.102:0/1651726218 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3d40502d0 con 0x7ff3d404a790 2021-02-10T11:33:37.785 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:37.764 7ff3ec8fb700 1 -- 172.21.2.102:0/1651726218 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3d40504f0 con 0x7ff3d4003af0 2021-02-10T11:33:37.785 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:37.764 7ff3ec8fb700 1 -- 172.21.2.102:0/1651726218 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3d4050710 con 0x7ff3d404d590 2021-02-10T11:33:37.785 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:37.764 7ff3ed0fc700 1 --2- 172.21.2.102:0/1651726218 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff3d404a790 0x7ff3d404cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:37.785 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:37.764 7ff3ed8fd700 1 --2- 172.21.2.102:0/1651726218 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff3d4003af0 0x7ff3d4005f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:37.785 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:37.764 7ff3ee0fe700 1 --2- 172.21.2.102:0/1651726218 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff3d404d590 0x7ff3d404fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:37.785 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:37.764 7ff3ed0fc700 1 -- 172.21.2.102:0/1651726218 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff3d404d590 msgr2=0x7ff3d404fa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:33:37.785 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:37.764 7ff3ed0fc700 1 --2- 172.21.2.102:0/1651726218 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff3d404d590 0x7ff3d404fa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:37.785 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:37.764 7ff3ed0fc700 1 -- 172.21.2.102:0/1651726218 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff3d4003af0 msgr2=0x7ff3d4005f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:33:37.785 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:37.764 7ff3ed0fc700 1 --2- 172.21.2.102:0/1651726218 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff3d4003af0 0x7ff3d4005f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:37.785 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:37.764 7ff3ed0fc700 1 -- 172.21.2.102:0/1651726218 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3d40509b0 con 0x7ff3d404a790 2021-02-10T11:33:37.785 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:37.765 7ff3ee0fe700 1 --2- 172.21.2.102:0/1651726218 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff3d404d590 0x7ff3d404fa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:33:37.785 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:37.765 7ff3ed8fd700 1 --2- 172.21.2.102:0/1651726218 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff3d4003af0 0x7ff3d4005f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:33:37.785 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:37.765 7ff3ed0fc700 1 --2- 172.21.2.102:0/1651726218 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff3d404a790 0x7ff3d404cc90 secure :-1 s=READY pgs=2434 cs=0 l=1 rev1=1 rx=0x7ff3d8006ac0 tx=0x7ff3d8009ab0).ready entity=mon.0 client_cookie=613aaccda164c7e1 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:37.786 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:37.765 7ff3ec8fb700 1 -- 172.21.2.102:0/1651726218 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff3d800c0c0 con 0x7ff3d404a790 2021-02-10T11:33:37.786 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:37.765 7ff3ec8fb700 1 -- 172.21.2.102:0/1651726218 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3d800c3c0 con 0x7ff3d404a790 2021-02-10T11:33:37.786 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:37.765 7ff3ec8fb700 1 -- 172.21.2.102:0/1651726218 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff3d800c8e0 con 0x7ff3d404a790 2021-02-10T11:33:37.786 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:37.765 7ff3efae1700 1 -- 172.21.2.102:0/1651726218 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff3d404a790 msgr2=0x7ff3d404cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:37.786 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:37.765 7ff3efae1700 1 --2- 172.21.2.102:0/1651726218 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff3d404a790 0x7ff3d404cc90 secure :-1 s=READY pgs=2434 cs=0 l=1 rev1=1 rx=0x7ff3d8006ac0 tx=0x7ff3d8009ab0).stop 2021-02-10T11:33:37.786 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:37.765 7ff3ee0fe700 1 -- 172.21.2.102:0/1651726218 reap_dead start 2021-02-10T11:33:37.786 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:37.766 7ff3efae1700 1 -- 172.21.2.102:0/1651726218 shutdown_connections 2021-02-10T11:33:37.786 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:37.766 7ff3efae1700 1 -- 172.21.2.102:0/1651726218 >> 172.21.2.102:0/1651726218 conn(0x7ff3e80ce430 msgr2=0x7ff3e80d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:37.786 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:37.766 7ff3efae1700 1 -- 172.21.2.102:0/1651726218 shutdown_connections 2021-02-10T11:33:37.786 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:37.766 7ff3efae1700 1 -- 172.21.2.102:0/1651726218 wait complete. 2021-02-10T11:33:37.786 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:37.781 7ff3efae1700 1 Processor -- start 2021-02-10T11:33:37.786 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:37.781 7ff3efae1700 1 -- start start 2021-02-10T11:33:37.786 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:37.781 7ff3efae1700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff3e81a0580 0x7ff3e8086fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:37.787 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:37.781 7ff3efae1700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff3e80b66b0 0x7ff3e80b6ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:37.787 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:37.781 7ff3efae1700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff3e8082bb0 0x7ff3e8082fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:37.787 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:37.781 7ff3efae1700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3e80875d0 con 0x7ff3e80b66b0 2021-02-10T11:33:37.787 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:37.781 7ff3efae1700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3e80835d0 con 0x7ff3e81a0580 2021-02-10T11:33:37.787 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:37.781 7ff3efae1700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3e80837f0 con 0x7ff3e8082bb0 2021-02-10T11:33:37.788 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:37.781 7ff3ed0fc700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff3e80b66b0 0x7ff3e80b6ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:37.788 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:37.782 7ff3ed0fc700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff3e80b66b0 0x7ff3e80b6ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:35470/0 (socket says 172.21.2.102:35470) 2021-02-10T11:33:37.788 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:37.782 7ff3ed0fc700 1 -- 172.21.2.102:0/3859968714 learned_addr learned my addr 172.21.2.102:0/3859968714 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:33:37.788 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:37.782 7ff3ed0fc700 1 -- 172.21.2.102:0/3859968714 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff3e8082bb0 msgr2=0x7ff3e8082fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:37.788 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:37.782 7ff3ed0fc700 1 --2- 172.21.2.102:0/3859968714 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff3e8082bb0 0x7ff3e8082fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:37.788 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:37.782 7ff3ed0fc700 1 -- 172.21.2.102:0/3859968714 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff3e81a0580 msgr2=0x7ff3e8086fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:37.788 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:37.782 7ff3ed0fc700 1 --2- 172.21.2.102:0/3859968714 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff3e81a0580 0x7ff3e8086fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:37.788 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:37.782 7ff3ed0fc700 1 -- 172.21.2.102:0/3859968714 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3e8083a10 con 0x7ff3e80b66b0 2021-02-10T11:33:37.788 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:37.782 7ff3ed0fc700 1 --2- 172.21.2.102:0/3859968714 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff3e80b66b0 0x7ff3e80b6ab0 secure :-1 s=READY pgs=2435 cs=0 l=1 rev1=1 rx=0x7ff3d80097a0 tx=0x7ff3d800b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:37.789 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:37.782 7ff3de7fc700 1 -- 172.21.2.102:0/3859968714 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff3d800bd60 con 0x7ff3e80b66b0 2021-02-10T11:33:37.789 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:37.782 7ff3de7fc700 1 -- 172.21.2.102:0/3859968714 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3d8000c90 con 0x7ff3e80b66b0 2021-02-10T11:33:37.789 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:37.782 7ff3de7fc700 1 -- 172.21.2.102:0/3859968714 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff3d80022b0 con 0x7ff3e80b66b0 2021-02-10T11:33:37.789 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:37.782 7ff3efae1700 1 -- 172.21.2.102:0/3859968714 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3e8083a10 con 0x7ff3e80b66b0 2021-02-10T11:33:37.789 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:37.782 7ff3efae1700 1 -- 172.21.2.102:0/3859968714 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3e8083a10 con 0x7ff3e80b66b0 2021-02-10T11:33:37.789 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:37.783 7ff3de7fc700 1 -- 172.21.2.102:0/3859968714 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ff3d8002020 con 0x7ff3e80b66b0 2021-02-10T11:33:37.789 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:37.783 7ff3de7fc700 1 --2- 172.21.2.102:0/3859968714 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff3d00186c0 0x7ff3d001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:37.789 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:37.783 7ff3de7fc700 1 -- 172.21.2.102:0/3859968714 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 5573+0+0 (secure 0 0 0) 0x7ff3d8032540 con 0x7ff3e80b66b0 2021-02-10T11:33:37.789 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:37.783 7ff3ed8fd700 1 --2- 172.21.2.102:0/3859968714 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff3d00186c0 0x7ff3d001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:37.789 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:37.784 7ff3ed8fd700 1 --2- 172.21.2.102:0/3859968714 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff3d00186c0 0x7ff3d001abc0 crc :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:37.814 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:37.812 7ff3efae1700 1 -- 172.21.2.102:0/3859968714 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3e8082980 con 0x7ff3e80b66b0 2021-02-10T11:33:37.816 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:37.815 7ff3de7fc700 1 -- 172.21.2.102:0/3859968714 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ff3d8030cf0 con 0x7ff3e80b66b0 2021-02-10T11:33:37.948 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:37.946 7ff3efae1700 1 -- 172.21.2.102:0/3859968714 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7ff3e8082980 con 0x7ff3e80b66b0 2021-02-10T11:33:37.951 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:37.950 7ff3de7fc700 1 -- 172.21.2.102:0/3859968714 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.2 v75) v1 ==== 184+0+0 (secure 0 0 0) 0x7ff3d8002320 con 0x7ff3e80b66b0 2021-02-10T11:33:37.955 INFO:teuthology.orchestra.run.gibba002.stderr:updated caps for client.2 2021-02-10T11:33:37.957 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:37.955 7ff3efae1700 1 -- 172.21.2.102:0/3859968714 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff3d00186c0 msgr2=0x7ff3d001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:37.957 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:37.955 7ff3efae1700 1 --2- 172.21.2.102:0/3859968714 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff3d00186c0 0x7ff3d001abc0 crc :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:37.957 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:37.955 7ff3efae1700 1 -- 172.21.2.102:0/3859968714 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff3e80b66b0 msgr2=0x7ff3e80b6ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:37.957 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:37.955 7ff3efae1700 1 --2- 172.21.2.102:0/3859968714 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff3e80b66b0 0x7ff3e80b6ab0 secure :-1 s=READY pgs=2435 cs=0 l=1 rev1=1 rx=0x7ff3d80097a0 tx=0x7ff3d800b970).stop 2021-02-10T11:33:37.957 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:37.955 7ff3efae1700 1 -- 172.21.2.102:0/3859968714 shutdown_connections 2021-02-10T11:33:37.957 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:37.955 7ff3efae1700 1 --2- 172.21.2.102:0/3859968714 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff3e8082bb0 0x7ff3e8082fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:37.957 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:37.955 7ff3efae1700 1 --2- 172.21.2.102:0/3859968714 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff3e80b66b0 0x7ff3e80b6ab0 unknown :-1 s=CLOSED pgs=2435 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:37.957 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:37.955 7ff3efae1700 1 --2- 172.21.2.102:0/3859968714 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff3d00186c0 0x7ff3d001abc0 unknown :-1 s=CLOSED pgs=1083 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:37.958 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:37.955 7ff3efae1700 1 --2- 172.21.2.102:0/3859968714 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff3e81a0580 0x7ff3e8086fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:37.958 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:37.955 7ff3efae1700 1 -- 172.21.2.102:0/3859968714 >> 172.21.2.102:0/3859968714 conn(0x7ff3e80afc60 msgr2=0x7ff3e80b50d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:37.958 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:37.955 7ff3efae1700 1 -- 172.21.2.102:0/3859968714 shutdown_connections 2021-02-10T11:33:37.958 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:37.956 7ff3efae1700 1 -- 172.21.2.102:0/3859968714 wait complete. 2021-02-10T11:33:37.965 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.3 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:33:38.395 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:38.377 7ff669ff8700 1 Processor -- start 2021-02-10T11:33:38.395 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:38.377 7ff669ff8700 1 -- start start 2021-02-10T11:33:38.395 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:38.378 7ff669ff8700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7ff6640d1880 0x7ff6641b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:38.395 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:38.378 7ff669ff8700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7ff6641b0af0 0x7ff664108950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:38.395 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:38.378 7ff669ff8700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6640fe1d0 con 0x7ff6640d1c80 2021-02-10T11:33:38.395 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:38.378 7ff669ff8700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff664109a70 con 0x7ff6641b0af0 2021-02-10T11:33:38.396 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:38.378 7ff669ff8700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff664109760 con 0x7ff6640d1880 2021-02-10T11:33:38.396 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:38.378 7ff663fff700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7ff6640d1c80 0x7ff664108f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:55932/0 (socket says 172.21.2.102:55932) 2021-02-10T11:33:38.396 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:38.378 7ff663fff700 1 -- 172.21.2.102:0/1492684257 learned_addr learned my addr 172.21.2.102:0/1492684257 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:33:38.396 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:38.378 7ff662ffd700 1 --2- 172.21.2.102:0/1492684257 >> v2:172.21.2.102:3300/0 conn(0x7ff6641b0af0 0x7ff664108950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:38.396 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:38.378 7ff6637fe700 1 --2- 172.21.2.102:0/1492684257 >> v2:172.21.2.113:3300/0 conn(0x7ff6640d1880 0x7ff6641b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:38.396 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:38.378 7ff662ffd700 1 -- 172.21.2.102:0/1492684257 >> v2:172.21.2.113:3300/0 conn(0x7ff6640d1880 msgr2=0x7ff6641b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:33:38.396 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:38.378 7ff662ffd700 1 --2- 172.21.2.102:0/1492684257 >> v2:172.21.2.113:3300/0 conn(0x7ff6640d1880 0x7ff6641b04d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:38.396 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:38.378 7ff662ffd700 1 -- 172.21.2.102:0/1492684257 >> v1:172.21.2.102:6789/0 conn(0x7ff6640d1c80 legacy=0x7ff664108f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:38.396 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:38.378 7ff662ffd700 1 -- 172.21.2.102:0/1492684257 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff664088070 con 0x7ff6641b0af0 2021-02-10T11:33:38.396 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:38.378 7ff662ffd700 1 --2- 172.21.2.102:0/1492684257 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff6641b0af0 0x7ff664108950 secure :-1 s=READY pgs=2437 cs=0 l=1 rev1=1 rx=0x7ff654006650 tx=0x7ff654009a80).ready entity=mon.0 client_cookie=c0781f819adbb5d9 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:38.396 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:38.378 7ff6627fc700 1 -- 172.21.2.102:0/1492684257 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff6540009b0 con 0x7ff6641b0af0 2021-02-10T11:33:38.396 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:38.379 7ff6627fc700 1 -- 172.21.2.102:0/1492684257 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff65400c0c0 con 0x7ff6641b0af0 2021-02-10T11:33:38.397 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:38.379 7ff6627fc700 1 -- 172.21.2.102:0/1492684257 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff65400c510 con 0x7ff6641b0af0 2021-02-10T11:33:38.397 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:38.379 7ff669ff8700 1 -- 172.21.2.102:0/1492684257 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff6641b0af0 msgr2=0x7ff664108950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:38.397 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:38.379 7ff669ff8700 1 --2- 172.21.2.102:0/1492684257 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff6641b0af0 0x7ff664108950 secure :-1 s=READY pgs=2437 cs=0 l=1 rev1=1 rx=0x7ff654006650 tx=0x7ff654009a80).stop 2021-02-10T11:33:38.397 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:38.379 7ff669ff8700 1 -- 172.21.2.102:0/1492684257 shutdown_connections 2021-02-10T11:33:38.397 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:38.379 7ff669ff8700 1 --2- 172.21.2.102:0/1492684257 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff6641b0af0 0x7ff664108950 unknown :-1 s=CLOSED pgs=2437 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:38.397 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:38.379 7ff669ff8700 1 --2- 172.21.2.102:0/1492684257 >> v2:172.21.2.113:3300/0 conn(0x7ff6640d1880 0x7ff6641b04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:38.397 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:38.379 7ff669ff8700 1 -- 172.21.2.102:0/1492684257 >> 172.21.2.102:0/1492684257 conn(0x7ff6640ce430 msgr2=0x7ff6640d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:38.397 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:38.379 7ff669ff8700 1 -- 172.21.2.102:0/1492684257 shutdown_connections 2021-02-10T11:33:38.397 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:38.379 7ff669ff8700 1 -- 172.21.2.102:0/1492684257 wait complete. 2021-02-10T11:33:38.399 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:38.394 7ff669ff8700 1 Processor -- start 2021-02-10T11:33:38.399 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:38.394 7ff669ff8700 1 -- start start 2021-02-10T11:33:38.399 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:38.394 7ff669ff8700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff664095600 0x7ff664095a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:38.399 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:38.394 7ff669ff8700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff664096020 0x7ff664096420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:38.399 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:38.394 7ff669ff8700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff664092a10 0x7ff664092e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:38.399 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:38.394 7ff669ff8700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff664094100 con 0x7ff664096020 2021-02-10T11:33:38.399 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:38.394 7ff669ff8700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff66419e9e0 con 0x7ff664095600 2021-02-10T11:33:38.399 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:38.394 7ff669ff8700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff664093430 con 0x7ff664092a10 2021-02-10T11:33:38.399 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:38.394 7ff662ffd700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff664096020 0x7ff664096420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:38.399 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:38.394 7ff662ffd700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff664096020 0x7ff664096420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:35482/0 (socket says 172.21.2.102:35482) 2021-02-10T11:33:38.400 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:38.394 7ff662ffd700 1 -- 172.21.2.102:0/3557739440 learned_addr learned my addr 172.21.2.102:0/3557739440 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:33:38.400 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:38.395 7ff663fff700 1 --2- 172.21.2.102:0/3557739440 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff664092a10 0x7ff664092e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:38.400 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:38.395 7ff6637fe700 1 --2- 172.21.2.102:0/3557739440 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff664095600 0x7ff664095a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:38.400 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:38.395 7ff662ffd700 1 -- 172.21.2.102:0/3557739440 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff664092a10 msgr2=0x7ff664092e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:38.400 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:38.395 7ff662ffd700 1 --2- 172.21.2.102:0/3557739440 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff664092a10 0x7ff664092e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:38.400 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:38.395 7ff662ffd700 1 -- 172.21.2.102:0/3557739440 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff664095600 msgr2=0x7ff664095a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:38.400 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:38.395 7ff662ffd700 1 --2- 172.21.2.102:0/3557739440 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff664095600 0x7ff664095a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:38.400 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:38.395 7ff662ffd700 1 -- 172.21.2.102:0/3557739440 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff664093650 con 0x7ff664096020 2021-02-10T11:33:38.400 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:38.395 7ff662ffd700 1 --2- 172.21.2.102:0/3557739440 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff664096020 0x7ff664096420 secure :-1 s=READY pgs=2438 cs=0 l=1 rev1=1 rx=0x7ff654009650 tx=0x7ff65400b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:38.400 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:38.395 7ff65bfff700 1 -- 172.21.2.102:0/3557739440 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff65400bd50 con 0x7ff664096020 2021-02-10T11:33:38.400 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:38.395 7ff65bfff700 1 -- 172.21.2.102:0/3557739440 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff654006030 con 0x7ff664096020 2021-02-10T11:33:38.400 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:38.395 7ff65bfff700 1 -- 172.21.2.102:0/3557739440 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff654005560 con 0x7ff664096020 2021-02-10T11:33:38.400 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:38.395 7ff669ff8700 1 -- 172.21.2.102:0/3557739440 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff664093650 con 0x7ff664096020 2021-02-10T11:33:38.401 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:38.395 7ff669ff8700 1 -- 172.21.2.102:0/3557739440 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff664093be0 con 0x7ff664096020 2021-02-10T11:33:38.401 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:38.396 7ff65bfff700 1 -- 172.21.2.102:0/3557739440 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ff6540054f0 con 0x7ff664096020 2021-02-10T11:33:38.401 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:38.396 7ff65bfff700 1 --2- 172.21.2.102:0/3557739440 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff6480186c0 0x7ff64801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:38.401 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:38.396 7ff65bfff700 1 -- 172.21.2.102:0/3557739440 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 5573+0+0 (secure 0 0 0) 0x7ff654032500 con 0x7ff664096020 2021-02-10T11:33:38.401 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:38.396 7ff6637fe700 1 --2- 172.21.2.102:0/3557739440 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff6480186c0 0x7ff64801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:38.401 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:38.397 7ff6637fe700 1 --2- 172.21.2.102:0/3557739440 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff6480186c0 0x7ff64801abc0 crc :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:38.429 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:38.428 7ff669ff8700 1 -- 172.21.2.102:0/3557739440 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff664082980 con 0x7ff664096020 2021-02-10T11:33:38.432 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:38.430 7ff65bfff700 1 -- 172.21.2.102:0/3557739440 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ff6540052e0 con 0x7ff664096020 2021-02-10T11:33:38.562 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:38.560 7ff669ff8700 1 -- 172.21.2.102:0/3557739440 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.3", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7ff664094080 con 0x7ff664096020 2021-02-10T11:33:38.565 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:38.564 7ff65bfff700 1 -- 172.21.2.102:0/3557739440 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.3", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.3 v76) v1 ==== 184+0+0 (secure 0 0 0) 0x7ff6540054f0 con 0x7ff664096020 2021-02-10T11:33:38.569 INFO:teuthology.orchestra.run.gibba002.stderr:updated caps for client.3 2021-02-10T11:33:38.571 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:38.569 7ff669ff8700 1 -- 172.21.2.102:0/3557739440 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff6480186c0 msgr2=0x7ff64801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:38.571 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:38.569 7ff669ff8700 1 --2- 172.21.2.102:0/3557739440 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff6480186c0 0x7ff64801abc0 crc :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:38.571 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:38.569 7ff669ff8700 1 -- 172.21.2.102:0/3557739440 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff664096020 msgr2=0x7ff664096420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:38.571 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:38.569 7ff669ff8700 1 --2- 172.21.2.102:0/3557739440 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff664096020 0x7ff664096420 secure :-1 s=READY pgs=2438 cs=0 l=1 rev1=1 rx=0x7ff654009650 tx=0x7ff65400b960).stop 2021-02-10T11:33:38.571 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:38.569 7ff669ff8700 1 -- 172.21.2.102:0/3557739440 shutdown_connections 2021-02-10T11:33:38.571 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:38.569 7ff669ff8700 1 --2- 172.21.2.102:0/3557739440 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff664092a10 0x7ff664092e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:38.571 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:38.569 7ff669ff8700 1 --2- 172.21.2.102:0/3557739440 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff664096020 0x7ff664096420 unknown :-1 s=CLOSED pgs=2438 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:38.571 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:38.569 7ff669ff8700 1 --2- 172.21.2.102:0/3557739440 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff6480186c0 0x7ff64801abc0 unknown :-1 s=CLOSED pgs=1084 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:38.571 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:38.569 7ff669ff8700 1 --2- 172.21.2.102:0/3557739440 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff664095600 0x7ff664095a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:38.572 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:38.569 7ff669ff8700 1 -- 172.21.2.102:0/3557739440 >> 172.21.2.102:0/3557739440 conn(0x7ff6641a6190 msgr2=0x7ff6640949b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:38.572 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:38.569 7ff669ff8700 1 -- 172.21.2.102:0/3557739440 shutdown_connections 2021-02-10T11:33:38.572 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:38.569 7ff669ff8700 1 -- 172.21.2.102:0/3557739440 wait complete. 2021-02-10T11:33:38.579 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:33:39.006 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:38.988 7f3bd4b65700 1 Processor -- start 2021-02-10T11:33:39.006 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:38.988 7f3bd4b65700 1 -- start start 2021-02-10T11:33:39.006 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:38.989 7f3bd4b65700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f3bd00d0340 0x7f3bd00ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:39.006 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:38.989 7f3bd4b65700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3bd00d1c70 0x7f3bd00efc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:39.006 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:38.989 7f3bd4b65700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3bd00f0b10 con 0x7f3bd00d1870 2021-02-10T11:33:39.006 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:38.989 7f3bd4b65700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3bd00e5ed0 con 0x7f3bd00d0340 2021-02-10T11:33:39.006 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:38.989 7f3bd4b65700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3bd01b0be0 con 0x7f3bd00d1c70 2021-02-10T11:33:39.007 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:38.989 7f3bce05d700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f3bd00d0340 0x7f3bd00ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:39.007 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:38.989 7f3bce05d700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f3bd00d0340 0x7f3bd00ef640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:35490/0 (socket says 172.21.2.102:35490) 2021-02-10T11:33:39.007 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:38.989 7f3bce05d700 1 -- 172.21.2.102:0/3440676969 learned_addr learned my addr 172.21.2.102:0/3440676969 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:33:39.007 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:38.989 7f3bce05d700 1 -- 172.21.2.102:0/3440676969 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3bd00d1c70 msgr2=0x7f3bd00efc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:33:39.007 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:38.989 7f3bce05d700 1 --2- 172.21.2.102:0/3440676969 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3bd00d1c70 0x7f3bd00efc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:39.007 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:38.989 7f3bce05d700 1 -- 172.21.2.102:0/3440676969 >> v1:172.21.2.113:6789/0 conn(0x7f3bd00d1870 legacy=0x7f3bd01b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:33:39.007 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:38.989 7f3bce05d700 1 -- 172.21.2.102:0/3440676969 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bd00f20a0 con 0x7f3bd00d0340 2021-02-10T11:33:39.007 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:38.989 7f3bce05d700 1 --2- 172.21.2.102:0/3440676969 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3bd00d0340 0x7f3bd00ef640 secure :-1 s=READY pgs=2439 cs=0 l=1 rev1=1 rx=0x7f3bc0006700 tx=0x7f3bc0009b80).ready entity=mon.0 client_cookie=90b201856a3e210e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:39.007 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:38.989 7f3bcd85c700 1 -- 172.21.2.102:0/3440676969 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3bc0006340 con 0x7f3bd00d0340 2021-02-10T11:33:39.007 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:38.989 7f3bcd85c700 1 -- 172.21.2.102:0/3440676969 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bc000c0c0 con 0x7f3bd00d0340 2021-02-10T11:33:39.007 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:38.989 7f3bcd85c700 1 -- 172.21.2.102:0/3440676969 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3bc000c520 con 0x7f3bd00d0340 2021-02-10T11:33:39.007 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:38.989 7f3bd4b65700 1 -- 172.21.2.102:0/3440676969 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3bd00d0340 msgr2=0x7f3bd00ef640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:39.007 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:38.989 7f3bd4b65700 1 --2- 172.21.2.102:0/3440676969 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3bd00d0340 0x7f3bd00ef640 secure :-1 s=READY pgs=2439 cs=0 l=1 rev1=1 rx=0x7f3bc0006700 tx=0x7f3bc0009b80).stop 2021-02-10T11:33:39.008 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:38.990 7f3bd4b65700 1 -- 172.21.2.102:0/3440676969 shutdown_connections 2021-02-10T11:33:39.008 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:38.990 7f3bd4b65700 1 --2- 172.21.2.102:0/3440676969 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3bd00d1c70 0x7f3bd00efc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:39.008 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:38.990 7f3bd4b65700 1 --2- 172.21.2.102:0/3440676969 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3bd00d0340 0x7f3bd00ef640 unknown :-1 s=CLOSED pgs=2439 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:39.008 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:38.990 7f3bd4b65700 1 -- 172.21.2.102:0/3440676969 >> 172.21.2.102:0/3440676969 conn(0x7f3bd00ce450 msgr2=0x7f3bd00d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:39.008 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:38.990 7f3bd4b65700 1 -- 172.21.2.102:0/3440676969 shutdown_connections 2021-02-10T11:33:39.008 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:38.990 7f3bd4b65700 1 -- 172.21.2.102:0/3440676969 wait complete. 2021-02-10T11:33:39.010 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:39.005 7f3bd4b65700 1 Processor -- start 2021-02-10T11:33:39.010 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:39.005 7f3bd4b65700 1 -- start start 2021-02-10T11:33:39.010 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:39.005 7f3bd4b65700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3bd0199ab0 0x7f3bd01aab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:39.010 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:39.005 7f3bd4b65700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3bd01ab1b0 0x7f3bd01ab5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:39.010 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:39.005 7f3bd4b65700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3bd019d250 0x7f3bd019d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:39.010 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:39.005 7f3bd4b65700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bd01ac040 con 0x7f3bd019d250 2021-02-10T11:33:39.010 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:39.005 7f3bd4b65700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bd0199eb0 con 0x7f3bd01ab1b0 2021-02-10T11:33:39.010 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:39.005 7f3bd4b65700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3bd01abbd0 con 0x7f3bd0199ab0 2021-02-10T11:33:39.010 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:39.005 7f3bcf05f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3bd019d250 0x7f3bd019d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:39.010 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:39.005 7f3bcf05f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3bd019d250 0x7f3bd019d650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:35496/0 (socket says 172.21.2.102:35496) 2021-02-10T11:33:39.010 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:39.005 7f3bcf05f700 1 -- 172.21.2.102:0/1010899689 learned_addr learned my addr 172.21.2.102:0/1010899689 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:33:39.010 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:39.006 7f3bce85e700 1 --2- 172.21.2.102:0/1010899689 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3bd0199ab0 0x7f3bd01aab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:39.011 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:39.006 7f3bcf05f700 1 -- 172.21.2.102:0/1010899689 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3bd0199ab0 msgr2=0x7f3bd01aab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:39.011 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:39.006 7f3bcf05f700 1 --2- 172.21.2.102:0/1010899689 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3bd0199ab0 0x7f3bd01aab90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:39.011 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:39.006 7f3bcf05f700 1 -- 172.21.2.102:0/1010899689 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3bd01ab1b0 msgr2=0x7f3bd01ab5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:39.011 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:39.006 7f3bce05d700 1 --2- 172.21.2.102:0/1010899689 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3bd01ab1b0 0x7f3bd01ab5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:39.011 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:39.006 7f3bcf05f700 1 --2- 172.21.2.102:0/1010899689 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3bd01ab1b0 0x7f3bd01ab5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:39.011 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:39.006 7f3bcf05f700 1 -- 172.21.2.102:0/1010899689 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bd019dc70 con 0x7f3bd019d250 2021-02-10T11:33:39.011 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:39.006 7f3bce05d700 1 --2- 172.21.2.102:0/1010899689 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3bd01ab1b0 0x7f3bd01ab5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:33:39.011 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:39.006 7f3bcf05f700 1 --2- 172.21.2.102:0/1010899689 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3bd019d250 0x7f3bd019d650 secure :-1 s=READY pgs=2440 cs=0 l=1 rev1=1 rx=0x7f3bc800b8c0 tx=0x7f3bc800bc60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:39.011 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:39.006 7f3bc77fe700 1 -- 172.21.2.102:0/1010899689 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3bc8003020 con 0x7f3bd019d250 2021-02-10T11:33:39.011 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:39.006 7f3bc77fe700 1 -- 172.21.2.102:0/1010899689 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bc8005a80 con 0x7f3bd019d250 2021-02-10T11:33:39.011 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:39.006 7f3bc77fe700 1 -- 172.21.2.102:0/1010899689 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3bc8006030 con 0x7f3bd019d250 2021-02-10T11:33:39.011 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:39.006 7f3bd4b65700 1 -- 172.21.2.102:0/1010899689 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3bd019dc70 con 0x7f3bd019d250 2021-02-10T11:33:39.011 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:39.006 7f3bd4b65700 1 -- 172.21.2.102:0/1010899689 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3bd019e200 con 0x7f3bd019d250 2021-02-10T11:33:39.012 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:39.007 7f3bc77fe700 1 -- 172.21.2.102:0/1010899689 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f3bc8005c60 con 0x7f3bd019d250 2021-02-10T11:33:39.012 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:39.007 7f3bc77fe700 1 --2- 172.21.2.102:0/1010899689 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3bb40186d0 0x7f3bb401abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:39.012 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:39.007 7f3bc77fe700 1 -- 172.21.2.102:0/1010899689 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f3bc8032b30 con 0x7f3bd019d250 2021-02-10T11:33:39.012 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:39.007 7f3bce85e700 1 --2- 172.21.2.102:0/1010899689 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3bb40186d0 0x7f3bb401abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:39.012 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:39.007 7f3bce85e700 1 --2- 172.21.2.102:0/1010899689 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3bb40186d0 0x7f3bb401abd0 crc :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:39.040 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:39.038 7f3bd4b65700 1 -- 172.21.2.102:0/1010899689 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3bd019cd20 con 0x7f3bd019d250 2021-02-10T11:33:39.042 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:39.041 7f3bc77fe700 1 -- 172.21.2.102:0/1010899689 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f3bc8005c90 con 0x7f3bd019d250 2021-02-10T11:33:39.170 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:39.169 7f3bd4b65700 1 -- 172.21.2.102:0/1010899689 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3bd019cd20 con 0x7f3bd019d250 2021-02-10T11:33:39.171 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:39.169 7f3bc77fe700 1 -- 172.21.2.102:0/1010899689 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+2102 (secure 0 0 0) 0x7f3bc8005a80 con 0x7f3bd019d250 2021-02-10T11:33:39.171 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:33:39.172 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":107,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":11103,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.102:6835/160551241","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6834","nonce":160551241},{"type":"v1","addr":"172.21.2.102:6835","nonce":160551241}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":103}],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:33:32.844442","modified":"2021-02-10 11:33:34.850935","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7882},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7882":{"gid":7882,"name":"b","rank":0,"incarnation":104,"state":"up:active","state_seq":3,"addr":"172.21.2.113:6833/121050001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6832","nonce":121050001},{"type":"v1","addr":"172.21.2.113:6833","nonce":121050001}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[27],"metadata_pool":26,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":13}]} 2021-02-10T11:33:39.172 INFO:teuthology.orchestra.run.gibba002.stderr:dumped fsmap epoch 107 2021-02-10T11:33:39.173 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:39.171 7f3bd4b65700 1 -- 172.21.2.102:0/1010899689 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3bb40186d0 msgr2=0x7f3bb401abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:39.173 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:39.171 7f3bd4b65700 1 --2- 172.21.2.102:0/1010899689 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3bb40186d0 0x7f3bb401abd0 crc :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:39.173 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:39.171 7f3bd4b65700 1 -- 172.21.2.102:0/1010899689 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3bd019d250 msgr2=0x7f3bd019d650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:39.173 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:39.171 7f3bd4b65700 1 --2- 172.21.2.102:0/1010899689 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3bd019d250 0x7f3bd019d650 secure :-1 s=READY pgs=2440 cs=0 l=1 rev1=1 rx=0x7f3bc800b8c0 tx=0x7f3bc800bc60).stop 2021-02-10T11:33:39.173 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:39.171 7f3bd4b65700 1 -- 172.21.2.102:0/1010899689 shutdown_connections 2021-02-10T11:33:39.173 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:39.171 7f3bd4b65700 1 --2- 172.21.2.102:0/1010899689 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3bd019d250 0x7f3bd019d650 unknown :-1 s=CLOSED pgs=2440 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:39.173 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:39.171 7f3bd4b65700 1 --2- 172.21.2.102:0/1010899689 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3bb40186d0 0x7f3bb401abd0 unknown :-1 s=CLOSED pgs=1085 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:39.174 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:39.171 7f3bd4b65700 1 --2- 172.21.2.102:0/1010899689 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3bd01ab1b0 0x7f3bd01ab5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:39.174 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:39.171 7f3bd4b65700 1 --2- 172.21.2.102:0/1010899689 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3bd0199ab0 0x7f3bd01aab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:39.174 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:39.171 7f3bd4b65700 1 -- 172.21.2.102:0/1010899689 >> 172.21.2.102:0/1010899689 conn(0x7f3bd019f8c0 msgr2=0x7f3bd01a7410 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:39.174 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:39.171 7f3bd4b65700 1 -- 172.21.2.102:0/1010899689 shutdown_connections 2021-02-10T11:33:39.174 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:39.171 7f3bd4b65700 1 -- 172.21.2.102:0/1010899689 wait complete. 2021-02-10T11:33:39.383 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 107, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:33:32.844442', 'modified': '2021-02-10 11:33:34.850935', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7882}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7882': {'gid': 7882, 'name': 'b', 'rank': 0, 'incarnation': 104, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.113:6833/121050001', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.113:6832', 'nonce': 121050001}, {'type': 'v1', 'addr': '172.21.2.113:6833', 'nonce': 121050001}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [27], 'metadata_pool': 26, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:33:39.383 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:33:39.384 DEBUG:teuthology.orchestra.run.gibba013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:33:39.782 INFO:teuthology.orchestra.run.gibba013.stdout:{ 2021-02-10T11:33:39.782 INFO:teuthology.orchestra.run.gibba013.stdout: "cluster_fsid": "68db30ea-dbce-4aef-a6df-f01c6fea0513", 2021-02-10T11:33:39.782 INFO:teuthology.orchestra.run.gibba013.stdout: "whoami": 0, 2021-02-10T11:33:39.782 INFO:teuthology.orchestra.run.gibba013.stdout: "id": 7882, 2021-02-10T11:33:39.782 INFO:teuthology.orchestra.run.gibba013.stdout: "want_state": "up:active", 2021-02-10T11:33:39.782 INFO:teuthology.orchestra.run.gibba013.stdout: "state": "up:active", 2021-02-10T11:33:39.782 INFO:teuthology.orchestra.run.gibba013.stdout: "rank_uptime": 6.9340460730000002, 2021-02-10T11:33:39.783 INFO:teuthology.orchestra.run.gibba013.stdout: "mdsmap_epoch": 107, 2021-02-10T11:33:39.783 INFO:teuthology.orchestra.run.gibba013.stdout: "osdmap_epoch": 128, 2021-02-10T11:33:39.783 INFO:teuthology.orchestra.run.gibba013.stdout: "osdmap_epoch_barrier": 128, 2021-02-10T11:33:39.783 INFO:teuthology.orchestra.run.gibba013.stdout: "uptime": 12.544286206000001 2021-02-10T11:33:39.783 INFO:teuthology.orchestra.run.gibba013.stdout:} 2021-02-10T11:33:39.786 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "68db30ea-dbce-4aef-a6df-f01c6fea0513", "whoami": 0, "id": 7882, "want_state": "up:active", "state": "up:active", "rank_uptime": 6.9340460730000002, "mdsmap_epoch": 107, "osdmap_epoch": 128, "osdmap_epoch_barrier": 128, "uptime": 12.544286206000001 } 2021-02-10T11:33:39.787 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:33:40.219 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:40.197 7f83b90cf700 1 Processor -- start 2021-02-10T11:33:40.220 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:40.197 7f83b90cf700 1 -- start start 2021-02-10T11:33:40.220 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:40.198 7f83b90cf700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f83b41b0470 0x7f83b41b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:40.220 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:40.198 7f83b90cf700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f83b40ef610 0x7f83b40efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:40.220 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:40.198 7f83b90cf700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83b40f0b50 con 0x7f83b40f0030 2021-02-10T11:33:40.220 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:40.198 7f83b90cf700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83b40e52e0 con 0x7f83b41b0470 2021-02-10T11:33:40.220 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:40.198 7f83b90cf700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83b40e5490 con 0x7f83b40ef610 2021-02-10T11:33:40.220 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:40.198 7f83b359e700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f83b40f0030 0x7f83b40f10d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:55962/0 (socket says 172.21.2.102:55962) 2021-02-10T11:33:40.221 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:40.198 7f83b359e700 1 -- 172.21.2.102:0/2088924772 learned_addr learned my addr 172.21.2.102:0/2088924772 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:33:40.221 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:40.198 7f83b2d9d700 1 --2- 172.21.2.102:0/2088924772 >> v2:172.21.2.102:3300/0 conn(0x7f83b41b0470 0x7f83b41b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:40.221 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:40.199 7f83b1d9b700 1 -- 172.21.2.102:0/2088924772 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2792373104 0 0) 0x7f83ac000f90 con 0x7f83b40f0030 2021-02-10T11:33:40.221 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:40.199 7f83b1d9b700 1 -- 172.21.2.102:0/2088924772 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f83a0002fe0 con 0x7f83b40f0030 2021-02-10T11:33:40.221 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:40.199 7f83b2d9d700 1 -- 172.21.2.102:0/2088924772 >> v2:172.21.2.113:3300/0 conn(0x7f83b40ef610 msgr2=0x7f83b40efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:33:40.221 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:40.199 7f83b2d9d700 1 --2- 172.21.2.102:0/2088924772 >> v2:172.21.2.113:3300/0 conn(0x7f83b40ef610 0x7f83b40efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:40.222 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:40.199 7f83b2d9d700 1 -- 172.21.2.102:0/2088924772 >> v1:172.21.2.102:6789/0 conn(0x7f83b40f0030 legacy=0x7f83b40f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:40.222 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:40.199 7f83b2d9d700 1 -- 172.21.2.102:0/2088924772 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83b40f0430 con 0x7f83b41b0470 2021-02-10T11:33:40.222 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:40.199 7f83b2d9d700 1 --2- 172.21.2.102:0/2088924772 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f83b41b0470 0x7f83b41b0870 secure :-1 s=READY pgs=2442 cs=0 l=1 rev1=1 rx=0x7f839c006650 tx=0x7f839c009a80).ready entity=mon.0 client_cookie=2762d4e9f585f557 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:40.222 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:40.199 7f83b1d9b700 1 -- 172.21.2.102:0/2088924772 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f839c00c0c0 con 0x7f83b41b0470 2021-02-10T11:33:40.222 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:40.199 7f83b1d9b700 1 -- 172.21.2.102:0/2088924772 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f839c00c3c0 con 0x7f83b41b0470 2021-02-10T11:33:40.223 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:40.199 7f83b1d9b700 1 -- 172.21.2.102:0/2088924772 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f839c00c8e0 con 0x7f83b41b0470 2021-02-10T11:33:40.223 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:40.199 7f83b90cf700 1 -- 172.21.2.102:0/2088924772 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f83b41b0470 msgr2=0x7f83b41b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:40.223 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:40.199 7f83b90cf700 1 --2- 172.21.2.102:0/2088924772 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f83b41b0470 0x7f83b41b0870 secure :-1 s=READY pgs=2442 cs=0 l=1 rev1=1 rx=0x7f839c006650 tx=0x7f839c009a80).stop 2021-02-10T11:33:40.223 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:40.200 7f83b90cf700 1 -- 172.21.2.102:0/2088924772 shutdown_connections 2021-02-10T11:33:40.223 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:40.200 7f83b90cf700 1 --2- 172.21.2.102:0/2088924772 >> v2:172.21.2.113:3300/0 conn(0x7f83b40ef610 0x7f83b40efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:40.223 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:40.200 7f83b90cf700 1 --2- 172.21.2.102:0/2088924772 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f83b41b0470 0x7f83b41b0870 unknown :-1 s=CLOSED pgs=2442 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:40.223 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:40.200 7f83b90cf700 1 -- 172.21.2.102:0/2088924772 >> 172.21.2.102:0/2088924772 conn(0x7f83b40ce3e0 msgr2=0x7f83b40e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:40.224 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:40.200 7f83b90cf700 1 -- 172.21.2.102:0/2088924772 shutdown_connections 2021-02-10T11:33:40.224 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:40.200 7f83b90cf700 1 -- 172.21.2.102:0/2088924772 wait complete. 2021-02-10T11:33:40.224 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:40.215 7f83b90cf700 1 Processor -- start 2021-02-10T11:33:40.224 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:40.215 7f83b90cf700 1 -- start start 2021-02-10T11:33:40.224 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:40.215 7f83b90cf700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f83b40af470 0x7f83b41085e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:40.224 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:40.215 7f83b90cf700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f83b4108c00 0x7f83b4109000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:40.224 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:40.215 7f83b90cf700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f83b4109620 0x7f83b4106880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:40.225 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:40.215 7f83b90cf700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83b40af870 con 0x7f83b4108c00 2021-02-10T11:33:40.225 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:40.215 7f83b90cf700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83b4106ea0 con 0x7f83b4109620 2021-02-10T11:33:40.225 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:40.215 7f83b90cf700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83b41070c0 con 0x7f83b40af470 2021-02-10T11:33:40.226 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:40.215 7f83b259c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f83b4108c00 0x7f83b4109000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:40.226 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:40.215 7f83b259c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f83b4108c00 0x7f83b4109000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:35508/0 (socket says 172.21.2.102:35508) 2021-02-10T11:33:40.226 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:40.215 7f83b259c700 1 -- 172.21.2.102:0/260769707 learned_addr learned my addr 172.21.2.102:0/260769707 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:33:40.226 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:40.216 7f83b259c700 1 -- 172.21.2.102:0/260769707 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f83b40af470 msgr2=0x7f83b41085e0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:33:40.226 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:40.216 7f83b259c700 1 --2- 172.21.2.102:0/260769707 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f83b40af470 0x7f83b41085e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:40.226 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:40.216 7f83b259c700 1 -- 172.21.2.102:0/260769707 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f83b4109620 msgr2=0x7f83b4106880 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:33:40.227 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:40.216 7f83b259c700 1 --2- 172.21.2.102:0/260769707 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f83b4109620 0x7f83b4106880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:40.227 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:40.216 7f83b259c700 1 -- 172.21.2.102:0/260769707 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83b41072e0 con 0x7f83b4108c00 2021-02-10T11:33:40.227 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:40.216 7f83b259c700 1 --2- 172.21.2.102:0/260769707 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f83b4108c00 0x7f83b4109000 secure :-1 s=READY pgs=2443 cs=0 l=1 rev1=1 rx=0x7f83a4006f00 tx=0x7f83a400b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:40.227 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:40.216 7f83ab7fe700 1 -- 172.21.2.102:0/260769707 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f83a400bd30 con 0x7f83b4108c00 2021-02-10T11:33:40.227 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:40.216 7f83ab7fe700 1 -- 172.21.2.102:0/260769707 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83a4000c90 con 0x7f83b4108c00 2021-02-10T11:33:40.227 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:40.216 7f83b90cf700 1 -- 172.21.2.102:0/260769707 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83b41072e0 con 0x7f83b4108c00 2021-02-10T11:33:40.227 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:40.216 7f83b90cf700 1 -- 172.21.2.102:0/260769707 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83b4107870 con 0x7f83b4108c00 2021-02-10T11:33:40.228 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:40.216 7f83ab7fe700 1 -- 172.21.2.102:0/260769707 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f83a40022a0 con 0x7f83b4108c00 2021-02-10T11:33:40.228 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:40.217 7f83ab7fe700 1 -- 172.21.2.102:0/260769707 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f83a4005310 con 0x7f83b4108c00 2021-02-10T11:33:40.228 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:40.217 7f83ab7fe700 1 --2- 172.21.2.102:0/260769707 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8398018730 0x7f839801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:40.228 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:40.217 7f83ab7fe700 1 -- 172.21.2.102:0/260769707 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f83a4032770 con 0x7f83b4108c00 2021-02-10T11:33:40.228 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:40.217 7f83b2d9d700 1 --2- 172.21.2.102:0/260769707 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8398018730 0x7f839801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:40.228 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:40.217 7f83b2d9d700 1 --2- 172.21.2.102:0/260769707 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8398018730 0x7f839801ac10 crc :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:40.250 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:40.248 7f83b90cf700 1 -- 172.21.2.102:0/260769707 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83b41080b0 con 0x7f83b4108c00 2021-02-10T11:33:40.252 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:40.251 7f83ab7fe700 1 -- 172.21.2.102:0/260769707 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f83a4000960 con 0x7f83b4108c00 2021-02-10T11:33:40.378 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:40.376 7f83b90cf700 1 -- 172.21.2.102:0/260769707 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f83b41080b0 con 0x7f83b4108c00 2021-02-10T11:33:40.379 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:40.377 7f83ab7fe700 1 -- 172.21.2.102:0/260769707 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v128) v1 ==== 74+0+14007 (secure 0 0 0) 0x7f83a400bd30 con 0x7f83b4108c00 2021-02-10T11:33:40.379 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:33:40.379 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":128,"fsid":"68db30ea-dbce-4aef-a6df-f01c6fea0513","created":"2021-02-10 11:03:06.331557","modified":"2021-02-10 11:33:33.846924","last_up_change":"2021-02-10 11:03:32.690562","last_in_change":"2021-02-10 11:03:32.690562","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":27,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:03:37.322519","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":26,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:33:30.653221","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"127","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":27,"pool_name":"cephfs_data","create_time":"2021-02-10 11:33:31.831765","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"128","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"be691b4d-a039-408a-af80-9d53f02b6f1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6818","nonce":36088},{"type":"v1","addr":"172.21.2.102:6820","nonce":36088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6822","nonce":36088},{"type":"v1","addr":"172.21.2.102:6824","nonce":36088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6830","nonce":36088},{"type":"v1","addr":"172.21.2.102:6832","nonce":36088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6826","nonce":36088},{"type":"v1","addr":"172.21.2.102:6828","nonce":36088}]},"public_addr":"172.21.2.102:6820/36088","cluster_addr":"172.21.2.102:6824/36088","heartbeat_back_addr":"172.21.2.102:6832/36088","heartbeat_front_addr":"172.21.2.102:6828/36088","state":["exists","up"]},{"osd":1,"uuid":"64e3ccc8-9c69-4d7c-865e-af8e83822635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6819","nonce":36087},{"type":"v1","addr":"172.21.2.102:6821","nonce":36087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6823","nonce":36087},{"type":"v1","addr":"172.21.2.102:6825","nonce":36087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6831","nonce":36087},{"type":"v1","addr":"172.21.2.102:6833","nonce":36087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6827","nonce":36087},{"type":"v1","addr":"172.21.2.102:6829","nonce":36087}]},"public_addr":"172.21.2.102:6821/36087","cluster_addr":"172.21.2.102:6825/36087","heartbeat_back_addr":"172.21.2.102:6833/36087","heartbeat_front_addr":"172.21.2.102:6829/36087","state":["exists","up"]},{"osd":2,"uuid":"153700af-747f-481f-bdf4-d6072d3a842f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6810","nonce":36089},{"type":"v1","addr":"172.21.2.102:6811","nonce":36089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6812","nonce":36089},{"type":"v1","addr":"172.21.2.102:6813","nonce":36089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6816","nonce":36089},{"type":"v1","addr":"172.21.2.102:6817","nonce":36089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6814","nonce":36089},{"type":"v1","addr":"172.21.2.102:6815","nonce":36089}]},"public_addr":"172.21.2.102:6811/36089","cluster_addr":"172.21.2.102:6813/36089","heartbeat_back_addr":"172.21.2.102:6817/36089","heartbeat_front_addr":"172.21.2.102:6815/36089","state":["exists","up"]},{"osd":3,"uuid":"cbc678f4-83a9-45e6-8830-2f7c232b1ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6802","nonce":36090},{"type":"v1","addr":"172.21.2.102:6803","nonce":36090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6804","nonce":36090},{"type":"v1","addr":"172.21.2.102:6805","nonce":36090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6808","nonce":36090},{"type":"v1","addr":"172.21.2.102:6809","nonce":36090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6806","nonce":36090},{"type":"v1","addr":"172.21.2.102:6807","nonce":36090}]},"public_addr":"172.21.2.102:6803/36090","cluster_addr":"172.21.2.102:6805/36090","heartbeat_back_addr":"172.21.2.102:6809/36090","heartbeat_front_addr":"172.21.2.102:6807/36090","state":["exists","up"]},{"osd":4,"uuid":"c01e4008-b258-4f10-afd2-9e7f37214351","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6808","nonce":36368},{"type":"v1","addr":"172.21.2.113:6811","nonce":36368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6814","nonce":36368},{"type":"v1","addr":"172.21.2.113:6818","nonce":36368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6826","nonce":36368},{"type":"v1","addr":"172.21.2.113:6828","nonce":36368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6820","nonce":36368},{"type":"v1","addr":"172.21.2.113:6824","nonce":36368}]},"public_addr":"172.21.2.113:6811/36368","cluster_addr":"172.21.2.113:6818/36368","heartbeat_back_addr":"172.21.2.113:6828/36368","heartbeat_front_addr":"172.21.2.113:6824/36368","state":["exists","up"]},{"osd":5,"uuid":"551f6007-daf8-4a98-8ee0-c2b16bcccb5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6802","nonce":36366},{"type":"v1","addr":"172.21.2.113:6805","nonce":36366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6807","nonce":36366},{"type":"v1","addr":"172.21.2.113:6810","nonce":36366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6819","nonce":36366},{"type":"v1","addr":"172.21.2.113:6822","nonce":36366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6812","nonce":36366},{"type":"v1","addr":"172.21.2.113:6815","nonce":36366}]},"public_addr":"172.21.2.113:6805/36366","cluster_addr":"172.21.2.113:6810/36366","heartbeat_back_addr":"172.21.2.113:6822/36366","heartbeat_front_addr":"172.21.2.113:6815/36366","state":["exists","up"]},{"osd":6,"uuid":"bf67c040-0e50-48be-b102-c3943bc9ebef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6800","nonce":36367},{"type":"v1","addr":"172.21.2.113:6801","nonce":36367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6803","nonce":36367},{"type":"v1","addr":"172.21.2.113:6804","nonce":36367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6813","nonce":36367},{"type":"v1","addr":"172.21.2.113:6816","nonce":36367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6806","nonce":36367},{"type":"v1","addr":"172.21.2.113:6809","nonce":36367}]},"public_addr":"172.21.2.113:6801/36367","cluster_addr":"172.21.2.113:6804/36367","heartbeat_back_addr":"172.21.2.113:6816/36367","heartbeat_front_addr":"172.21.2.113:6809/36367","state":["exists","up"]},{"osd":7,"uuid":"f5461f80-018d-428f-be89-2eb2bcbe7bb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6817","nonce":36365},{"type":"v1","addr":"172.21.2.113:6821","nonce":36365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6823","nonce":36365},{"type":"v1","addr":"172.21.2.113:6825","nonce":36365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6830","nonce":36365},{"type":"v1","addr":"172.21.2.113:6831","nonce":36365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6827","nonce":36365},{"type":"v1","addr":"172.21.2.113:6829","nonce":36365}]},"public_addr":"172.21.2.113:6821/36365","cluster_addr":"172.21.2.113:6825/36365","heartbeat_back_addr":"172.21.2.113:6831/36365","heartbeat_front_addr":"172.21.2.113:6829/36365","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:33:40.380 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:40.379 7f83b90cf700 1 -- 172.21.2.102:0/260769707 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8398018730 msgr2=0x7f839801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:40.381 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:40.379 7f83b90cf700 1 --2- 172.21.2.102:0/260769707 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8398018730 0x7f839801ac10 crc :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:40.381 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:40.379 7f83b90cf700 1 -- 172.21.2.102:0/260769707 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f83b4108c00 msgr2=0x7f83b4109000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:40.381 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:40.379 7f83b90cf700 1 --2- 172.21.2.102:0/260769707 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f83b4108c00 0x7f83b4109000 secure :-1 s=READY pgs=2443 cs=0 l=1 rev1=1 rx=0x7f83a4006f00 tx=0x7f83a400b940).stop 2021-02-10T11:33:40.381 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:40.379 7f83b90cf700 1 -- 172.21.2.102:0/260769707 shutdown_connections 2021-02-10T11:33:40.381 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:40.379 7f83b90cf700 1 --2- 172.21.2.102:0/260769707 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8398018730 0x7f839801ac10 unknown :-1 s=CLOSED pgs=1086 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:40.381 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:40.379 7f83b90cf700 1 --2- 172.21.2.102:0/260769707 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f83b4109620 0x7f83b4106880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:40.381 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:40.379 7f83b90cf700 1 --2- 172.21.2.102:0/260769707 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f83b4108c00 0x7f83b4109000 unknown :-1 s=CLOSED pgs=2443 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:40.381 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:40.379 7f83b90cf700 1 --2- 172.21.2.102:0/260769707 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f83b40af470 0x7f83b41085e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:40.381 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:40.379 7f83b90cf700 1 -- 172.21.2.102:0/260769707 >> 172.21.2.102:0/260769707 conn(0x7f83b40b9290 msgr2=0x7f83b40aa950 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:40.381 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:40.379 7f83b90cf700 1 -- 172.21.2.102:0/260769707 shutdown_connections 2021-02-10T11:33:40.381 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:40.379 7f83b90cf700 1 -- 172.21.2.102:0/260769707 wait complete. 2021-02-10T11:33:40.389 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:03:37.322519', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 26, 'pool_name': 'cephfs_metadata', 'create_time': '2021-02-10 11:33:30.653221', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '127', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 27, 'pool_name': 'cephfs_data', 'create_time': '2021-02-10 11:33:31.831765', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '128', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-02-10T11:33:40.390 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:33:40.821 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:40.803 7f7ec48fc700 1 Processor -- start 2021-02-10T11:33:40.821 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:40.803 7f7ec48fc700 1 -- start start 2021-02-10T11:33:40.821 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:40.804 7f7ec48fc700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f7ec00d1860 0x7f7ec01b04b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:40.822 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:40.804 7f7ec48fc700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f7ec00cff60 0x7f7ec0109120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:40.822 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:40.804 7f7ec48fc700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ec00fe1b0 con 0x7f7ec01b0ad0 2021-02-10T11:33:40.822 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:40.804 7f7ec48fc700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ec0109ac0 con 0x7f7ec00d1860 2021-02-10T11:33:40.822 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:40.804 7f7ec48fc700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ec0109740 con 0x7f7ec00cff60 2021-02-10T11:33:40.822 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:40.804 7f7ebe61c700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f7ec00d1860 0x7f7ec01b04b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:40.822 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:40.804 7f7ebde1b700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f7ec01b0ad0 0x7f7ec0108930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:55972/0 (socket says 172.21.2.102:55972) 2021-02-10T11:33:40.822 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:40.804 7f7ebde1b700 1 -- 172.21.2.102:0/4113450756 learned_addr learned my addr 172.21.2.102:0/4113450756 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:33:40.822 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:40.804 7f7ebe61c700 1 -- 172.21.2.102:0/4113450756 >> v2:172.21.2.113:3300/0 conn(0x7f7ec00cff60 msgr2=0x7f7ec0109120 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:33:40.822 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:40.804 7f7ebe61c700 1 --2- 172.21.2.102:0/4113450756 >> v2:172.21.2.113:3300/0 conn(0x7f7ec00cff60 0x7f7ec0109120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:40.822 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:40.804 7f7ebe61c700 1 -- 172.21.2.102:0/4113450756 >> v1:172.21.2.102:6789/0 conn(0x7f7ec01b0ad0 legacy=0x7f7ec0108930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:40.822 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:40.804 7f7ebe61c700 1 -- 172.21.2.102:0/4113450756 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ec0088070 con 0x7f7ec00d1860 2021-02-10T11:33:40.823 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:40.804 7f7ebe61c700 1 --2- 172.21.2.102:0/4113450756 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7ec00d1860 0x7f7ec01b04b0 secure :-1 s=READY pgs=2445 cs=0 l=1 rev1=1 rx=0x7f7ea8006650 tx=0x7f7ea8009a80).ready entity=mon.0 client_cookie=be850cf9a81c7ee2 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:40.823 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:40.805 7f7ebd61a700 1 -- 172.21.2.102:0/4113450756 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7ea80009b0 con 0x7f7ec00d1860 2021-02-10T11:33:40.823 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:40.805 7f7ebd61a700 1 -- 172.21.2.102:0/4113450756 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ea800c0c0 con 0x7f7ec00d1860 2021-02-10T11:33:40.823 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:40.805 7f7ebd61a700 1 -- 172.21.2.102:0/4113450756 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7ea800c5e0 con 0x7f7ec00d1860 2021-02-10T11:33:40.823 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:40.805 7f7ec48fc700 1 -- 172.21.2.102:0/4113450756 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7ec00d1860 msgr2=0x7f7ec01b04b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:40.823 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:40.805 7f7ec48fc700 1 --2- 172.21.2.102:0/4113450756 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7ec00d1860 0x7f7ec01b04b0 secure :-1 s=READY pgs=2445 cs=0 l=1 rev1=1 rx=0x7f7ea8006650 tx=0x7f7ea8009a80).stop 2021-02-10T11:33:40.823 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:40.805 7f7ec48fc700 1 -- 172.21.2.102:0/4113450756 shutdown_connections 2021-02-10T11:33:40.823 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:40.805 7f7ec48fc700 1 --2- 172.21.2.102:0/4113450756 >> v2:172.21.2.113:3300/0 conn(0x7f7ec00cff60 0x7f7ec0109120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:40.823 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:40.805 7f7ec48fc700 1 --2- 172.21.2.102:0/4113450756 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7ec00d1860 0x7f7ec01b04b0 unknown :-1 s=CLOSED pgs=2445 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:40.823 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:40.805 7f7ec48fc700 1 -- 172.21.2.102:0/4113450756 >> 172.21.2.102:0/4113450756 conn(0x7f7ec00ce410 msgr2=0x7f7ec00d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:40.824 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:40.805 7f7ec48fc700 1 -- 172.21.2.102:0/4113450756 shutdown_connections 2021-02-10T11:33:40.824 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:40.805 7f7ec48fc700 1 -- 172.21.2.102:0/4113450756 wait complete. 2021-02-10T11:33:40.825 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:40.820 7f7ec48fc700 1 Processor -- start 2021-02-10T11:33:40.825 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:40.820 7f7ec48fc700 1 -- start start 2021-02-10T11:33:40.825 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:40.820 7f7ec48fc700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7ec01a05e0 0x7f7ec018b970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:40.825 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:40.820 7f7ec48fc700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7ec00b8780 0x7f7ec00bfa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:40.825 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:40.820 7f7ec48fc700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7ec00c0030 0x7f7ec0189e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:40.825 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:40.820 7f7ec48fc700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ec00b82a0 con 0x7f7ec00b8780 2021-02-10T11:33:40.826 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:40.820 7f7ec48fc700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ec00c0430 con 0x7f7ec01a05e0 2021-02-10T11:33:40.826 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:40.820 7f7ec48fc700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ec00c0650 con 0x7f7ec00c0030 2021-02-10T11:33:40.826 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:40.821 7f7ebde1b700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7ec00b8780 0x7f7ec00bfa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:40.826 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:40.821 7f7ebde1b700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7ec00b8780 0x7f7ec00bfa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:35520/0 (socket says 172.21.2.102:35520) 2021-02-10T11:33:40.826 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:40.821 7f7ebde1b700 1 -- 172.21.2.102:0/3337118820 learned_addr learned my addr 172.21.2.102:0/3337118820 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:33:40.826 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:40.821 7f7ebde1b700 1 -- 172.21.2.102:0/3337118820 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7ec00c0030 msgr2=0x7f7ec0189e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:40.826 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:40.821 7f7ebde1b700 1 --2- 172.21.2.102:0/3337118820 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7ec00c0030 0x7f7ec0189e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:40.826 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:40.821 7f7ebde1b700 1 -- 172.21.2.102:0/3337118820 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7ec01a05e0 msgr2=0x7f7ec018b970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:40.826 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:40.821 7f7ebde1b700 1 --2- 172.21.2.102:0/3337118820 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7ec01a05e0 0x7f7ec018b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:40.826 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:40.821 7f7ebde1b700 1 -- 172.21.2.102:0/3337118820 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ec018a490 con 0x7f7ec00b8780 2021-02-10T11:33:40.826 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:40.821 7f7ebde1b700 1 --2- 172.21.2.102:0/3337118820 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7ec00b8780 0x7f7ec00bfa10 secure :-1 s=READY pgs=2446 cs=0 l=1 rev1=1 rx=0x7f7eb0001c60 tx=0x7f7eb000a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:40.827 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:40.821 7f7eb6ffd700 1 -- 172.21.2.102:0/3337118820 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7eb000acc0 con 0x7f7ec00b8780 2021-02-10T11:33:40.827 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:40.821 7f7eb6ffd700 1 -- 172.21.2.102:0/3337118820 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7eb0004140 con 0x7f7ec00b8780 2021-02-10T11:33:40.827 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:40.821 7f7eb6ffd700 1 -- 172.21.2.102:0/3337118820 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7eb00045a0 con 0x7f7ec00b8780 2021-02-10T11:33:40.827 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:40.821 7f7ec48fc700 1 -- 172.21.2.102:0/3337118820 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ec018a490 con 0x7f7ec00b8780 2021-02-10T11:33:40.827 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:40.821 7f7ec48fc700 1 -- 172.21.2.102:0/3337118820 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ec018aa20 con 0x7f7ec00b8780 2021-02-10T11:33:40.827 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:40.822 7f7eb6ffd700 1 -- 172.21.2.102:0/3337118820 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f7eb0000d90 con 0x7f7ec00b8780 2021-02-10T11:33:40.827 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:40.822 7f7eb6ffd700 1 --2- 172.21.2.102:0/3337118820 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7ea40186c0 0x7f7ea401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:40.827 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:40.822 7f7eb6ffd700 1 -- 172.21.2.102:0/3337118820 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f7eb00323d0 con 0x7f7ec00b8780 2021-02-10T11:33:40.827 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:40.822 7f7ebe61c700 1 --2- 172.21.2.102:0/3337118820 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7ea40186c0 0x7f7ea401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:40.827 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:40.823 7f7ebe61c700 1 --2- 172.21.2.102:0/3337118820 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7ea40186c0 0x7f7ea401abc0 crc :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:40.855 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:40.854 7f7ec48fc700 1 -- 172.21.2.102:0/3337118820 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ec0082a20 con 0x7f7ec00b8780 2021-02-10T11:33:40.858 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:40.856 7f7eb6ffd700 1 -- 172.21.2.102:0/3337118820 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f7eb00009b0 con 0x7f7ec00b8780 2021-02-10T11:33:40.966 DEBUG:teuthology.orchestra.run.gibba002:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:33:40.969 DEBUG:teuthology.orchestra.run.gibba013:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:33:40.987 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:40.985 7f7ec48fc700 1 -- 172.21.2.102:0/3337118820 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f7ec0082a20 con 0x7f7ec00b8780 2021-02-10T11:33:40.987 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:40.985 7f7eb6ffd700 1 -- 172.21.2.102:0/3337118820 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v128) v1 ==== 92+0+10 (secure 0 0 0) 0x7f7eb0000d90 con 0x7f7ec00b8780 2021-02-10T11:33:40.988 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:33:40.990 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:40.988 7f7ec48fc700 1 -- 172.21.2.102:0/3337118820 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7ea40186c0 msgr2=0x7f7ea401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:40.990 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:40.988 7f7ec48fc700 1 --2- 172.21.2.102:0/3337118820 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7ea40186c0 0x7f7ea401abc0 crc :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:40.990 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:40.988 7f7ec48fc700 1 -- 172.21.2.102:0/3337118820 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7ec00b8780 msgr2=0x7f7ec00bfa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:40.990 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:40.988 7f7ec48fc700 1 --2- 172.21.2.102:0/3337118820 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7ec00b8780 0x7f7ec00bfa10 secure :-1 s=READY pgs=2446 cs=0 l=1 rev1=1 rx=0x7f7eb0001c60 tx=0x7f7eb000a8d0).stop 2021-02-10T11:33:40.990 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:40.988 7f7ec48fc700 1 -- 172.21.2.102:0/3337118820 shutdown_connections 2021-02-10T11:33:40.991 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:40.988 7f7ec48fc700 1 --2- 172.21.2.102:0/3337118820 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7ec00c0030 0x7f7ec0189e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:40.991 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:40.988 7f7ec48fc700 1 --2- 172.21.2.102:0/3337118820 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7ec00b8780 0x7f7ec00bfa10 unknown :-1 s=CLOSED pgs=2446 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:40.991 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:40.988 7f7ec48fc700 1 --2- 172.21.2.102:0/3337118820 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7ea40186c0 0x7f7ea401abc0 unknown :-1 s=CLOSED pgs=1087 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:40.991 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:40.988 7f7ec48fc700 1 --2- 172.21.2.102:0/3337118820 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7ec01a05e0 0x7f7ec018b970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:40.991 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:40.988 7f7ec48fc700 1 -- 172.21.2.102:0/3337118820 >> 172.21.2.102:0/3337118820 conn(0x7f7ec00a6a80 msgr2=0x7f7ec00ae110 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:40.991 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:40.988 7f7ec48fc700 1 -- 172.21.2.102:0/3337118820 shutdown_connections 2021-02-10T11:33:40.991 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:40.988 7f7ec48fc700 1 -- 172.21.2.102:0/3337118820 wait complete. 2021-02-10T11:33:40.998 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-02-10T11:33:41.404 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:41.385 7f5a6265b700 1 Processor -- start 2021-02-10T11:33:41.404 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:41.385 7f5a6265b700 1 -- start start 2021-02-10T11:33:41.405 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:41.385 7f5a6265b700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f5a5c0d1870 0x7f5a5c1b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:41.405 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:41.385 7f5a6265b700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5a5c0cff70 0x7f5a5c109130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:41.405 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:41.385 7f5a6265b700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a5c109a60 con 0x7f5a5c1b0ae0 2021-02-10T11:33:41.405 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:41.385 7f5a6265b700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a5c109750 con 0x7f5a5c0d1870 2021-02-10T11:33:41.405 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:41.385 7f5a6265b700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a5c088070 con 0x7f5a5c0cff70 2021-02-10T11:33:41.405 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:41.386 7f5a5bfff700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f5a5c0d1870 0x7f5a5c1b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:41.405 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:41.386 7f5a60c78700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5a5c0cff70 0x7f5a5c109130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:41.405 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:41.386 7f5a5b7fe700 1 --1- >> v1:172.21.2.113:6789/0 conn(0x7f5a5c1b0ae0 0x7f5a5c108940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.113:6789/0 says I am v1:172.21.2.102:35074/0 (socket says 172.21.2.102:35074) 2021-02-10T11:33:41.405 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:41.386 7f5a5b7fe700 1 -- 172.21.2.102:0/1266045509 learned_addr learned my addr 172.21.2.102:0/1266045509 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:33:41.405 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:41.387 7f5a60c78700 1 -- 172.21.2.102:0/1266045509 >> v2:172.21.2.113:3300/0 conn(0x7f5a5c0d1870 msgr2=0x7f5a5c1b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:33:41.405 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:41.387 7f5a60c78700 1 --2- 172.21.2.102:0/1266045509 >> v2:172.21.2.113:3300/0 conn(0x7f5a5c0d1870 0x7f5a5c1b04c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:41.405 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:41.387 7f5a60c78700 1 -- 172.21.2.102:0/1266045509 >> v1:172.21.2.113:6789/0 conn(0x7f5a5c1b0ae0 legacy=0x7f5a5c108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:41.406 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:41.387 7f5a60c78700 1 -- 172.21.2.102:0/1266045509 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a5c0d1c70 con 0x7f5a5c0cff70 2021-02-10T11:33:41.406 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:41.387 7f5a5bfff700 1 --2- 172.21.2.102:0/1266045509 >> v2:172.21.2.113:3300/0 conn(0x7f5a5c0d1870 0x7f5a5c1b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:33:41.406 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:41.387 7f5a60c78700 1 --2- 172.21.2.102:0/1266045509 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5a5c0cff70 0x7f5a5c109130 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7f5a54006e50 tx=0x7f5a54009e70).ready entity=mon.2 client_cookie=e2e6067acbd249c9 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:41.406 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:41.388 7f5a5affd700 1 -- 172.21.2.102:0/1266045509 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5a54000f90 con 0x7f5a5c0cff70 2021-02-10T11:33:41.406 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:41.388 7f5a5affd700 1 -- 172.21.2.102:0/1266045509 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a540066f0 con 0x7f5a5c0cff70 2021-02-10T11:33:41.406 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:41.388 7f5a5affd700 1 -- 172.21.2.102:0/1266045509 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5a5400b700 con 0x7f5a5c0cff70 2021-02-10T11:33:41.406 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:41.388 7f5a6265b700 1 -- 172.21.2.102:0/1266045509 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5a5c0cff70 msgr2=0x7f5a5c109130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:41.406 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:41.388 7f5a6265b700 1 --2- 172.21.2.102:0/1266045509 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5a5c0cff70 0x7f5a5c109130 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7f5a54006e50 tx=0x7f5a54009e70).stop 2021-02-10T11:33:41.406 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:41.388 7f5a6265b700 1 -- 172.21.2.102:0/1266045509 shutdown_connections 2021-02-10T11:33:41.406 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:41.388 7f5a6265b700 1 --2- 172.21.2.102:0/1266045509 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5a5c0cff70 0x7f5a5c109130 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:41.406 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:41.388 7f5a6265b700 1 --2- 172.21.2.102:0/1266045509 >> v2:172.21.2.113:3300/0 conn(0x7f5a5c0d1870 0x7f5a5c1b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:41.406 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:41.388 7f5a6265b700 1 -- 172.21.2.102:0/1266045509 >> 172.21.2.102:0/1266045509 conn(0x7f5a5c0ce420 msgr2=0x7f5a5c0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:41.406 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:41.388 7f5a6265b700 1 -- 172.21.2.102:0/1266045509 shutdown_connections 2021-02-10T11:33:41.407 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:41.388 7f5a6265b700 1 -- 172.21.2.102:0/1266045509 wait complete. 2021-02-10T11:33:41.408 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:41.403 7f5a6265b700 1 Processor -- start 2021-02-10T11:33:41.408 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:41.403 7f5a6265b700 1 -- start start 2021-02-10T11:33:41.408 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:41.404 7f5a6265b700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5a5c095160 0x7f5a5c0b76d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:41.409 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:41.404 7f5a6265b700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5a5c0b5e30 0x7f5a5c0c0e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:41.409 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:41.404 7f5a6265b700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5a5c0b6230 0x7f5a5c0c1460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:41.409 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:41.404 7f5a6265b700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a5c0b58b0 con 0x7f5a5c095160 2021-02-10T11:33:41.409 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:41.404 7f5a6265b700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a5c095560 con 0x7f5a5c0b6230 2021-02-10T11:33:41.409 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:41.404 7f5a6265b700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a5c0b7cf0 con 0x7f5a5c0b5e30 2021-02-10T11:33:41.409 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:41.404 7f5a5bfff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5a5c095160 0x7f5a5c0b76d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:41.409 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:41.404 7f5a5bfff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5a5c095160 0x7f5a5c0b76d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:35532/0 (socket says 172.21.2.102:35532) 2021-02-10T11:33:41.409 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:41.404 7f5a5bfff700 1 -- 172.21.2.102:0/2119329095 learned_addr learned my addr 172.21.2.102:0/2119329095 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:33:41.409 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:41.404 7f5a60c78700 1 --2- 172.21.2.102:0/2119329095 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5a5c0b6230 0x7f5a5c0c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:41.409 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:41.404 7f5a5bfff700 1 -- 172.21.2.102:0/2119329095 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5a5c0b5e30 msgr2=0x7f5a5c0c0e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:41.409 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:41.404 7f5a5bfff700 1 --2- 172.21.2.102:0/2119329095 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5a5c0b5e30 0x7f5a5c0c0e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:41.409 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:41.404 7f5a5bfff700 1 -- 172.21.2.102:0/2119329095 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5a5c0b6230 msgr2=0x7f5a5c0c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:41.409 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:41.404 7f5a5bfff700 1 --2- 172.21.2.102:0/2119329095 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5a5c0b6230 0x7f5a5c0c1460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:41.410 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:41.404 7f5a5bfff700 1 -- 172.21.2.102:0/2119329095 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a5c083980 con 0x7f5a5c095160 2021-02-10T11:33:41.410 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:41.404 7f5a5bfff700 1 --2- 172.21.2.102:0/2119329095 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5a5c095160 0x7f5a5c0b76d0 secure :-1 s=READY pgs=2447 cs=0 l=1 rev1=1 rx=0x7f5a4c006f40 tx=0x7f5a4c00b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:41.410 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:41.404 7f5a58ff9700 1 -- 172.21.2.102:0/2119329095 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5a4c00bd80 con 0x7f5a5c095160 2021-02-10T11:33:41.410 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:41.404 7f5a58ff9700 1 -- 172.21.2.102:0/2119329095 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a4c000c90 con 0x7f5a5c095160 2021-02-10T11:33:41.410 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:41.404 7f5a6265b700 1 -- 172.21.2.102:0/2119329095 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a5c083980 con 0x7f5a5c095160 2021-02-10T11:33:41.410 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:41.404 7f5a58ff9700 1 -- 172.21.2.102:0/2119329095 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5a4c002220 con 0x7f5a5c095160 2021-02-10T11:33:41.410 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:41.405 7f5a6265b700 1 -- 172.21.2.102:0/2119329095 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a5c083f10 con 0x7f5a5c095160 2021-02-10T11:33:41.411 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:41.405 7f5a58ff9700 1 -- 172.21.2.102:0/2119329095 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f5a4c00a000 con 0x7f5a5c095160 2021-02-10T11:33:41.411 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:41.405 7f5a58ff9700 1 --2- 172.21.2.102:0/2119329095 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5a38018730 0x7f5a3801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:41.411 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:41.405 7f5a58ff9700 1 -- 172.21.2.102:0/2119329095 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f5a4c032620 con 0x7f5a5c095160 2021-02-10T11:33:41.411 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:41.405 7f5a5b7fe700 1 --2- 172.21.2.102:0/2119329095 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5a38018730 0x7f5a3801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:41.411 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:41.405 7f5a5b7fe700 1 --2- 172.21.2.102:0/2119329095 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5a38018730 0x7f5a3801ac10 crc :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:41.436 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:41.434 7f5a6265b700 1 -- 172.21.2.102:0/2119329095 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a5c0fe1c0 con 0x7f5a5c095160 2021-02-10T11:33:41.439 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:41.437 7f5a58ff9700 1 -- 172.21.2.102:0/2119329095 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f5a4c000c90 con 0x7f5a5c095160 2021-02-10T11:33:41.567 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:41.566 7f5a6265b700 1 -- 172.21.2.102:0/2119329095 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7f5a5c0fe1c0 con 0x7f5a5c095160 2021-02-10T11:33:41.568 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:41.566 7f5a58ff9700 1 -- 172.21.2.102:0/2119329095 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v128) v1 ==== 104+0+10 (secure 0 0 0) 0x7f5a4c00a000 con 0x7f5a5c095160 2021-02-10T11:33:41.569 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:33:41.570 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:41.568 7f5a6265b700 1 -- 172.21.2.102:0/2119329095 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5a38018730 msgr2=0x7f5a3801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:41.570 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:41.568 7f5a6265b700 1 --2- 172.21.2.102:0/2119329095 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5a38018730 0x7f5a3801ac10 crc :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:41.570 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:41.568 7f5a6265b700 1 -- 172.21.2.102:0/2119329095 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5a5c095160 msgr2=0x7f5a5c0b76d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:41.570 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:41.568 7f5a6265b700 1 --2- 172.21.2.102:0/2119329095 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5a5c095160 0x7f5a5c0b76d0 secure :-1 s=READY pgs=2447 cs=0 l=1 rev1=1 rx=0x7f5a4c006f40 tx=0x7f5a4c00b990).stop 2021-02-10T11:33:41.570 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:41.568 7f5a6265b700 1 -- 172.21.2.102:0/2119329095 shutdown_connections 2021-02-10T11:33:41.570 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:41.568 7f5a6265b700 1 --2- 172.21.2.102:0/2119329095 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5a38018730 0x7f5a3801ac10 unknown :-1 s=CLOSED pgs=1088 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:41.570 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:41.568 7f5a6265b700 1 --2- 172.21.2.102:0/2119329095 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5a5c0b6230 0x7f5a5c0c1460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:41.570 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:41.568 7f5a6265b700 1 --2- 172.21.2.102:0/2119329095 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5a5c0b5e30 0x7f5a5c0c0e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:41.570 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:41.568 7f5a6265b700 1 --2- 172.21.2.102:0/2119329095 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5a5c095160 0x7f5a5c0b76d0 unknown :-1 s=CLOSED pgs=2447 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:41.570 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:41.568 7f5a6265b700 1 -- 172.21.2.102:0/2119329095 >> 172.21.2.102:0/2119329095 conn(0x7f5a5c0b6870 msgr2=0x7f5a5c0abf70 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:41.571 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:41.568 7f5a6265b700 1 -- 172.21.2.102:0/2119329095 shutdown_connections 2021-02-10T11:33:41.571 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:41.569 7f5a6265b700 1 -- 172.21.2.102:0/2119329095 wait complete. 2021-02-10T11:33:41.578 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-02-10T11:33:42.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:42.031 7f2498022700 1 Processor -- start 2021-02-10T11:33:42.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:42.031 7f2498022700 1 -- start start 2021-02-10T11:33:42.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:42.032 7f2498022700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f24900d1870 0x7f24901b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:42.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:42.032 7f2498022700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f24900cff70 0x7f2490109130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:42.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:42.032 7f2498022700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2490109a60 con 0x7f24901b0ae0 2021-02-10T11:33:42.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:42.032 7f2498022700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2490109750 con 0x7f24900cff70 2021-02-10T11:33:42.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:42.032 7f2498022700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2490088070 con 0x7f24900d1870 2021-02-10T11:33:42.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:42.032 7f249563d700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f24901b0ae0 0x7f2490108940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:55994/0 (socket says 172.21.2.102:55994) 2021-02-10T11:33:42.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:42.032 7f249563d700 1 -- 172.21.2.102:0/573120559 learned_addr learned my addr 172.21.2.102:0/573120559 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:33:42.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:42.032 7f249663f700 1 --2- 172.21.2.102:0/573120559 >> v2:172.21.2.113:3300/0 conn(0x7f24900cff70 0x7f2490109130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:42.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:42.032 7f2495e3e700 1 --2- 172.21.2.102:0/573120559 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f24900d1870 0x7f24901b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:42.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:42.032 7f2494e3c700 1 -- 172.21.2.102:0/573120559 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3914641597 0 0) 0x7f248c0010b0 con 0x7f24901b0ae0 2021-02-10T11:33:42.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:42.032 7f2494e3c700 1 -- 172.21.2.102:0/573120559 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f247c002fe0 con 0x7f24901b0ae0 2021-02-10T11:33:42.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:42.033 7f2495e3e700 1 -- 172.21.2.102:0/573120559 >> v2:172.21.2.113:3300/0 conn(0x7f24900cff70 msgr2=0x7f2490109130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:33:42.052 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:42.033 7f2495e3e700 1 --2- 172.21.2.102:0/573120559 >> v2:172.21.2.113:3300/0 conn(0x7f24900cff70 0x7f2490109130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:42.052 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:42.033 7f2495e3e700 1 -- 172.21.2.102:0/573120559 >> v1:172.21.2.102:6789/0 conn(0x7f24901b0ae0 legacy=0x7f2490108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:42.052 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:42.033 7f2495e3e700 1 -- 172.21.2.102:0/573120559 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2490088260 con 0x7f24900d1870 2021-02-10T11:33:42.052 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:42.033 7f249663f700 1 --2- 172.21.2.102:0/573120559 >> v2:172.21.2.113:3300/0 conn(0x7f24900cff70 0x7f2490109130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:33:42.052 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:42.033 7f2495e3e700 1 --2- 172.21.2.102:0/573120559 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f24900d1870 0x7f24901b04c0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 rx=0x7f2484006650 tx=0x7f2484009a80).ready entity=mon.2 client_cookie=40c913df1f4c2483 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:42.052 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:42.033 7f2494e3c700 1 -- 172.21.2.102:0/573120559 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f24840009b0 con 0x7f24900d1870 2021-02-10T11:33:42.052 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:42.033 7f2494e3c700 1 -- 172.21.2.102:0/573120559 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f248400c0c0 con 0x7f24900d1870 2021-02-10T11:33:42.052 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:42.033 7f2494e3c700 1 -- 172.21.2.102:0/573120559 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f248400c5e0 con 0x7f24900d1870 2021-02-10T11:33:42.052 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:42.034 7f2498022700 1 -- 172.21.2.102:0/573120559 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f24900d1870 msgr2=0x7f24901b04c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:42.052 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:42.034 7f2498022700 1 --2- 172.21.2.102:0/573120559 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f24900d1870 0x7f24901b04c0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 rx=0x7f2484006650 tx=0x7f2484009a80).stop 2021-02-10T11:33:42.053 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:42.034 7f2498022700 1 -- 172.21.2.102:0/573120559 shutdown_connections 2021-02-10T11:33:42.053 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:42.034 7f2498022700 1 --2- 172.21.2.102:0/573120559 >> v2:172.21.2.113:3300/0 conn(0x7f24900cff70 0x7f2490109130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:42.053 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:42.034 7f2498022700 1 --2- 172.21.2.102:0/573120559 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f24900d1870 0x7f24901b04c0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:42.053 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:42.034 7f2498022700 1 -- 172.21.2.102:0/573120559 >> 172.21.2.102:0/573120559 conn(0x7f24900ce420 msgr2=0x7f24900d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:42.053 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:42.034 7f2498022700 1 -- 172.21.2.102:0/573120559 shutdown_connections 2021-02-10T11:33:42.053 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:42.034 7f2498022700 1 -- 172.21.2.102:0/573120559 wait complete. 2021-02-10T11:33:42.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:42.049 7f2498022700 1 Processor -- start 2021-02-10T11:33:42.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:42.049 7f2498022700 1 -- start start 2021-02-10T11:33:42.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:42.050 7f2498022700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f24900a2580 0x7f24900a1cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:42.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:42.050 7f2498022700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2490094c40 0x7f24901a75e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:42.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:42.050 7f2498022700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2490095040 0x7f24901a7c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:42.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:42.050 7f2498022700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2490188180 con 0x7f2490095040 2021-02-10T11:33:42.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:42.050 7f2498022700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2490187e70 con 0x7f24900a2580 2021-02-10T11:33:42.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:42.050 7f2498022700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24901a8220 con 0x7f2490094c40 2021-02-10T11:33:42.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:42.050 7f249663f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2490095040 0x7f24901a7c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:42.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:42.050 7f249663f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2490095040 0x7f24901a7c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:35542/0 (socket says 172.21.2.102:35542) 2021-02-10T11:33:42.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:42.050 7f249663f700 1 -- 172.21.2.102:0/3143501200 learned_addr learned my addr 172.21.2.102:0/3143501200 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:33:42.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:42.050 7f2495e3e700 1 --2- 172.21.2.102:0/3143501200 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f24900a2580 0x7f24900a1cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:42.056 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:42.050 7f249563d700 1 --2- 172.21.2.102:0/3143501200 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2490094c40 0x7f24901a75e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:42.056 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:42.050 7f249663f700 1 -- 172.21.2.102:0/3143501200 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2490094c40 msgr2=0x7f24901a75e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:42.056 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:42.050 7f249663f700 1 --2- 172.21.2.102:0/3143501200 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2490094c40 0x7f24901a75e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:42.056 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:42.050 7f249663f700 1 -- 172.21.2.102:0/3143501200 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f24900a2580 msgr2=0x7f24900a1cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:42.056 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:42.050 7f249663f700 1 --2- 172.21.2.102:0/3143501200 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f24900a2580 0x7f24900a1cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:42.056 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:42.050 7f249663f700 1 -- 172.21.2.102:0/3143501200 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2490188700 con 0x7f2490095040 2021-02-10T11:33:42.056 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:42.051 7f249663f700 1 --2- 172.21.2.102:0/3143501200 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2490095040 0x7f24901a7c00 secure :-1 s=READY pgs=2449 cs=0 l=1 rev1=1 rx=0x7f248800b970 tx=0x7f248800bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:42.056 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:42.051 7f24827fc700 1 -- 172.21.2.102:0/3143501200 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2488000f90 con 0x7f2490095040 2021-02-10T11:33:42.056 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:42.051 7f24827fc700 1 -- 172.21.2.102:0/3143501200 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2488006790 con 0x7f2490095040 2021-02-10T11:33:42.056 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:42.051 7f2498022700 1 -- 172.21.2.102:0/3143501200 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2490188700 con 0x7f2490095040 2021-02-10T11:33:42.056 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:42.051 7f24827fc700 1 -- 172.21.2.102:0/3143501200 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2488001680 con 0x7f2490095040 2021-02-10T11:33:42.056 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:42.051 7f2498022700 1 -- 172.21.2.102:0/3143501200 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2490188700 con 0x7f2490095040 2021-02-10T11:33:42.057 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:42.051 7f24827fc700 1 -- 172.21.2.102:0/3143501200 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f2488000e70 con 0x7f2490095040 2021-02-10T11:33:42.057 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:42.051 7f24827fc700 1 --2- 172.21.2.102:0/3143501200 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f24780186c0 0x7f247801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:42.057 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:42.051 7f24827fc700 1 -- 172.21.2.102:0/3143501200 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f2488032b20 con 0x7f2490095040 2021-02-10T11:33:42.057 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:42.052 7f2495e3e700 1 --2- 172.21.2.102:0/3143501200 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f24780186c0 0x7f247801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:42.057 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:42.052 7f2495e3e700 1 --2- 172.21.2.102:0/3143501200 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f24780186c0 0x7f247801abc0 crc :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:42.083 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:42.082 7f2498022700 1 -- 172.21.2.102:0/3143501200 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2490082980 con 0x7f2490095040 2021-02-10T11:33:42.086 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:42.084 7f24827fc700 1 -- 172.21.2.102:0/3143501200 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f24880066f0 con 0x7f2490095040 2021-02-10T11:33:42.213 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:42.211 7f2498022700 1 -- 172.21.2.102:0/3143501200 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7f2490109a60 con 0x7f2490095040 2021-02-10T11:33:42.213 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:42.211 7f24827fc700 1 -- 172.21.2.102:0/3143501200 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v128) v1 ==== 100+0+10 (secure 0 0 0) 0x7f2488000df0 con 0x7f2490095040 2021-02-10T11:33:42.214 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:33:42.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:42.213 7f2498022700 1 -- 172.21.2.102:0/3143501200 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f24780186c0 msgr2=0x7f247801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:42.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:42.213 7f2498022700 1 --2- 172.21.2.102:0/3143501200 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f24780186c0 0x7f247801abc0 crc :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:42.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:42.213 7f2498022700 1 -- 172.21.2.102:0/3143501200 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2490095040 msgr2=0x7f24901a7c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:42.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:42.213 7f2498022700 1 --2- 172.21.2.102:0/3143501200 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2490095040 0x7f24901a7c00 secure :-1 s=READY pgs=2449 cs=0 l=1 rev1=1 rx=0x7f248800b970 tx=0x7f248800bd10).stop 2021-02-10T11:33:42.216 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:42.214 7f2498022700 1 -- 172.21.2.102:0/3143501200 shutdown_connections 2021-02-10T11:33:42.216 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:42.214 7f2498022700 1 --2- 172.21.2.102:0/3143501200 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2490095040 0x7f24901a7c00 unknown :-1 s=CLOSED pgs=2449 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:42.216 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:42.214 7f2498022700 1 --2- 172.21.2.102:0/3143501200 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2490094c40 0x7f24901a75e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:42.216 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:42.214 7f2498022700 1 --2- 172.21.2.102:0/3143501200 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f24780186c0 0x7f247801abc0 unknown :-1 s=CLOSED pgs=1089 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:42.216 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:42.214 7f2498022700 1 --2- 172.21.2.102:0/3143501200 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f24900a2580 0x7f24900a1cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:42.216 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:42.214 7f2498022700 1 -- 172.21.2.102:0/3143501200 >> 172.21.2.102:0/3143501200 conn(0x7f24900a2b30 msgr2=0x7f249009ce60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:42.216 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:42.214 7f2498022700 1 -- 172.21.2.102:0/3143501200 shutdown_connections 2021-02-10T11:33:42.216 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:42.214 7f2498022700 1 -- 172.21.2.102:0/3143501200 wait complete. 2021-02-10T11:33:42.224 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-02-10T11:33:42.224 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:33:42.652 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:42.633 7fa5c6a18700 1 Processor -- start 2021-02-10T11:33:42.652 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:42.633 7fa5c6a18700 1 -- start start 2021-02-10T11:33:42.652 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:42.634 7fa5c6a18700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fa5c00d1870 0x7fa5c01b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:42.652 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:42.634 7fa5c6a18700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa5c00f0650 0x7fa5c00cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:42.652 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:42.634 7fa5c6a18700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5c00f0b10 con 0x7fa5c01b0a10 2021-02-10T11:33:42.652 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:42.634 7fa5c6a18700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5c00e5ed0 con 0x7fa5c00d1870 2021-02-10T11:33:42.652 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:42.634 7fa5c6a18700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5c00d1c70 con 0x7fa5c00f0650 2021-02-10T11:33:42.652 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:42.634 7fa5bffff700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fa5c01b0a10 0x7fa5c00cf330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:56006/0 (socket says 172.21.2.102:56006) 2021-02-10T11:33:42.652 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:42.634 7fa5bffff700 1 -- 172.21.2.102:0/2202969493 learned_addr learned my addr 172.21.2.102:0/2202969493 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:33:42.652 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:42.634 7fa5c4834700 1 --2- 172.21.2.102:0/2202969493 >> v2:172.21.2.113:3300/0 conn(0x7fa5c00d1870 0x7fa5c01b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:42.653 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:42.634 7fa5c5035700 1 --2- 172.21.2.102:0/2202969493 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa5c00f0650 0x7fa5c00cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:42.653 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:42.634 7fa5bf7fe700 1 -- 172.21.2.102:0/2202969493 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 245809398 0 0) 0x7fa5b0001060 con 0x7fa5c01b0a10 2021-02-10T11:33:42.653 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:42.635 7fa5bf7fe700 1 -- 172.21.2.102:0/2202969493 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5ac002fe0 con 0x7fa5c01b0a10 2021-02-10T11:33:42.653 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:42.635 7fa5c5035700 1 -- 172.21.2.102:0/2202969493 >> v2:172.21.2.113:3300/0 conn(0x7fa5c00d1870 msgr2=0x7fa5c01b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:33:42.653 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:42.635 7fa5c5035700 1 --2- 172.21.2.102:0/2202969493 >> v2:172.21.2.113:3300/0 conn(0x7fa5c00d1870 0x7fa5c01b03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:42.653 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:42.635 7fa5c5035700 1 -- 172.21.2.102:0/2202969493 >> v1:172.21.2.102:6789/0 conn(0x7fa5c01b0a10 legacy=0x7fa5c00cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:42.653 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:42.635 7fa5c5035700 1 -- 172.21.2.102:0/2202969493 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5c00d1e90 con 0x7fa5c00f0650 2021-02-10T11:33:42.653 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:42.635 7fa5c4834700 1 --2- 172.21.2.102:0/2202969493 >> v2:172.21.2.113:3300/0 conn(0x7fa5c00d1870 0x7fa5c01b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:33:42.653 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:42.635 7fa5c5035700 1 --2- 172.21.2.102:0/2202969493 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa5c00f0650 0x7fa5c00cfb20 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7fa5b8006ad0 tx=0x7fa5b8009ad0).ready entity=mon.2 client_cookie=99ef82ff0f46c427 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:42.653 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:42.635 7fa5bf7fe700 1 -- 172.21.2.102:0/2202969493 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa5b800c0c0 con 0x7fa5c00f0650 2021-02-10T11:33:42.653 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:42.635 7fa5bf7fe700 1 -- 172.21.2.102:0/2202969493 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5b800c0c0 con 0x7fa5c00f0650 2021-02-10T11:33:42.653 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:42.635 7fa5bf7fe700 1 -- 172.21.2.102:0/2202969493 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa5b800c440 con 0x7fa5c00f0650 2021-02-10T11:33:42.654 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:42.635 7fa5c6a18700 1 -- 172.21.2.102:0/2202969493 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa5c00f0650 msgr2=0x7fa5c00cfb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:42.654 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:42.635 7fa5c6a18700 1 --2- 172.21.2.102:0/2202969493 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa5c00f0650 0x7fa5c00cfb20 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7fa5b8006ad0 tx=0x7fa5b8009ad0).stop 2021-02-10T11:33:42.654 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:42.636 7fa5c6a18700 1 -- 172.21.2.102:0/2202969493 shutdown_connections 2021-02-10T11:33:42.654 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:42.636 7fa5c6a18700 1 --2- 172.21.2.102:0/2202969493 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa5c00f0650 0x7fa5c00cfb20 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:42.654 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:42.636 7fa5c6a18700 1 --2- 172.21.2.102:0/2202969493 >> v2:172.21.2.113:3300/0 conn(0x7fa5c00d1870 0x7fa5c01b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:42.654 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:42.636 7fa5c6a18700 1 -- 172.21.2.102:0/2202969493 >> 172.21.2.102:0/2202969493 conn(0x7fa5c00ce450 msgr2=0x7fa5c00d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:42.654 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:42.636 7fa5c6a18700 1 -- 172.21.2.102:0/2202969493 shutdown_connections 2021-02-10T11:33:42.654 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:42.636 7fa5c6a18700 1 -- 172.21.2.102:0/2202969493 wait complete. 2021-02-10T11:33:42.656 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:42.650 7fa5c6a18700 1 Processor -- start 2021-02-10T11:33:42.656 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:42.650 7fa5c6a18700 1 -- start start 2021-02-10T11:33:42.656 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:42.651 7fa5c6a18700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa5c00ada90 0x7fa5c00ade90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:42.656 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:42.651 7fa5c6a18700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa5c0099d20 0x7fa5c009a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:42.656 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:42.651 7fa5c6a18700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa5c009a740 0x7fa5c0097bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:42.656 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:42.651 7fa5c6a18700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5c00997a0 con 0x7fa5c009a740 2021-02-10T11:33:42.656 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:42.651 7fa5c6a18700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5c018f0c0 con 0x7fa5c00ada90 2021-02-10T11:33:42.656 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:42.651 7fa5c6a18700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5c009ab40 con 0x7fa5c0099d20 2021-02-10T11:33:42.656 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:42.651 7fa5c5035700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa5c009a740 0x7fa5c0097bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:42.656 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:42.651 7fa5c5035700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa5c009a740 0x7fa5c0097bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:35558/0 (socket says 172.21.2.102:35558) 2021-02-10T11:33:42.656 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:42.651 7fa5c5035700 1 -- 172.21.2.102:0/4074610785 learned_addr learned my addr 172.21.2.102:0/4074610785 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:33:42.657 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:42.652 7fa5c5035700 1 -- 172.21.2.102:0/4074610785 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa5c0099d20 msgr2=0x7fa5c009a120 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:33:42.657 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:42.652 7fa5c5035700 1 --2- 172.21.2.102:0/4074610785 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa5c0099d20 0x7fa5c009a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:42.657 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:42.652 7fa5c5035700 1 -- 172.21.2.102:0/4074610785 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa5c00ada90 msgr2=0x7fa5c00ade90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:42.657 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:42.652 7fa5c5035700 1 --2- 172.21.2.102:0/4074610785 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa5c00ada90 0x7fa5c00ade90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:42.657 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:42.652 7fa5c5035700 1 -- 172.21.2.102:0/4074610785 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5c009ad60 con 0x7fa5c009a740 2021-02-10T11:33:42.657 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:42.652 7fa5c5035700 1 --2- 172.21.2.102:0/4074610785 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa5c009a740 0x7fa5c0097bc0 secure :-1 s=READY pgs=2451 cs=0 l=1 rev1=1 rx=0x7fa5b8009600 tx=0x7fa5b800b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:42.657 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:42.652 7fa5bd7fa700 1 -- 172.21.2.102:0/4074610785 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa5b8000cf0 con 0x7fa5c009a740 2021-02-10T11:33:42.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:42.652 7fa5bd7fa700 1 -- 172.21.2.102:0/4074610785 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5b8009070 con 0x7fa5c009a740 2021-02-10T11:33:42.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:42.652 7fa5bd7fa700 1 -- 172.21.2.102:0/4074610785 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa5b8002500 con 0x7fa5c009a740 2021-02-10T11:33:42.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:42.652 7fa5c6a18700 1 -- 172.21.2.102:0/4074610785 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5c009ad60 con 0x7fa5c009a740 2021-02-10T11:33:42.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:42.652 7fa5c6a18700 1 -- 172.21.2.102:0/4074610785 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5c00984d0 con 0x7fa5c009a740 2021-02-10T11:33:42.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:42.653 7fa5bd7fa700 1 -- 172.21.2.102:0/4074610785 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa5b8000cf0 con 0x7fa5c009a740 2021-02-10T11:33:42.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:42.653 7fa5bd7fa700 1 --2- 172.21.2.102:0/4074610785 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa5a40186c0 0x7fa5a401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:42.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:42.653 7fa5bd7fa700 1 -- 172.21.2.102:0/4074610785 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fa5b80325c0 con 0x7fa5c009a740 2021-02-10T11:33:42.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:42.653 7fa5c4834700 1 --2- 172.21.2.102:0/4074610785 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa5a40186c0 0x7fa5a401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:42.659 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:42.653 7fa5c4834700 1 --2- 172.21.2.102:0/4074610785 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa5a40186c0 0x7fa5a401abc0 crc :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:42.686 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:42.684 7fa5c6a18700 1 -- 172.21.2.102:0/4074610785 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5c00ebc00 con 0x7fa5c009a740 2021-02-10T11:33:42.688 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:42.686 7fa5bd7fa700 1 -- 172.21.2.102:0/4074610785 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fa5b8030d70 con 0x7fa5c009a740 2021-02-10T11:33:42.816 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:42.814 7fa5c6a18700 1 -- 172.21.2.102:0/4074610785 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa5c00f0b10 con 0x7fa5c009a740 2021-02-10T11:33:42.817 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:42.815 7fa5bd7fa700 1 -- 172.21.2.102:0/4074610785 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+2102 (secure 0 0 0) 0x7fa5b800bd80 con 0x7fa5c009a740 2021-02-10T11:33:42.817 INFO:teuthology.orchestra.run.gibba002.stderr:dumped fsmap epoch 107 2021-02-10T11:33:42.817 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:33:42.817 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":107,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":11103,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.102:6835/160551241","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6834","nonce":160551241},{"type":"v1","addr":"172.21.2.102:6835","nonce":160551241}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":103}],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:33:32.844442","modified":"2021-02-10 11:33:34.850935","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7882},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7882":{"gid":7882,"name":"b","rank":0,"incarnation":104,"state":"up:active","state_seq":3,"addr":"172.21.2.113:6833/121050001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6832","nonce":121050001},{"type":"v1","addr":"172.21.2.113:6833","nonce":121050001}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[27],"metadata_pool":26,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":13}]} 2021-02-10T11:33:42.818 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:42.817 7fa5c6a18700 1 -- 172.21.2.102:0/4074610785 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa5a40186c0 msgr2=0x7fa5a401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:42.818 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:42.817 7fa5c6a18700 1 --2- 172.21.2.102:0/4074610785 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa5a40186c0 0x7fa5a401abc0 crc :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:42.818 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:42.817 7fa5c6a18700 1 -- 172.21.2.102:0/4074610785 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa5c009a740 msgr2=0x7fa5c0097bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:42.819 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:42.817 7fa5c6a18700 1 --2- 172.21.2.102:0/4074610785 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa5c009a740 0x7fa5c0097bc0 secure :-1 s=READY pgs=2451 cs=0 l=1 rev1=1 rx=0x7fa5b8009600 tx=0x7fa5b800b990).stop 2021-02-10T11:33:42.819 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:42.817 7fa5c6a18700 1 -- 172.21.2.102:0/4074610785 shutdown_connections 2021-02-10T11:33:42.819 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:42.817 7fa5c6a18700 1 --2- 172.21.2.102:0/4074610785 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa5c009a740 0x7fa5c0097bc0 unknown :-1 s=CLOSED pgs=2451 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:42.819 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:42.817 7fa5c6a18700 1 --2- 172.21.2.102:0/4074610785 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa5c0099d20 0x7fa5c009a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:42.819 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:42.817 7fa5c6a18700 1 --2- 172.21.2.102:0/4074610785 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa5a40186c0 0x7fa5a401abc0 unknown :-1 s=CLOSED pgs=1090 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:42.820 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:42.817 7fa5c6a18700 1 --2- 172.21.2.102:0/4074610785 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa5c00ada90 0x7fa5c00ade90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:42.820 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:42.817 7fa5c6a18700 1 -- 172.21.2.102:0/4074610785 >> 172.21.2.102:0/4074610785 conn(0x7fa5c00f0550 msgr2=0x7fa5c0191730 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:42.820 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:42.817 7fa5c6a18700 1 -- 172.21.2.102:0/4074610785 shutdown_connections 2021-02-10T11:33:42.820 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:42.817 7fa5c6a18700 1 -- 172.21.2.102:0/4074610785 wait complete. 2021-02-10T11:33:43.029 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 107, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:33:32.844442', 'modified': '2021-02-10 11:33:34.850935', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7882}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7882': {'gid': 7882, 'name': 'b', 'rank': 0, 'incarnation': 104, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.113:6833/121050001', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.113:6832', 'nonce': 121050001}, {'type': 'v1', 'addr': '172.21.2.113:6833', 'nonce': 121050001}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [27], 'metadata_pool': 26, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:33:43.029 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:33:43.029 DEBUG:teuthology.orchestra.run.gibba013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:33:43.426 INFO:teuthology.orchestra.run.gibba013.stdout:{ 2021-02-10T11:33:43.427 INFO:teuthology.orchestra.run.gibba013.stdout: "cluster_fsid": "68db30ea-dbce-4aef-a6df-f01c6fea0513", 2021-02-10T11:33:43.427 INFO:teuthology.orchestra.run.gibba013.stdout: "whoami": 0, 2021-02-10T11:33:43.427 INFO:teuthology.orchestra.run.gibba013.stdout: "id": 7882, 2021-02-10T11:33:43.427 INFO:teuthology.orchestra.run.gibba013.stdout: "want_state": "up:active", 2021-02-10T11:33:43.427 INFO:teuthology.orchestra.run.gibba013.stdout: "state": "up:active", 2021-02-10T11:33:43.427 INFO:teuthology.orchestra.run.gibba013.stdout: "rank_uptime": 10.578570643999999, 2021-02-10T11:33:43.427 INFO:teuthology.orchestra.run.gibba013.stdout: "mdsmap_epoch": 107, 2021-02-10T11:33:43.427 INFO:teuthology.orchestra.run.gibba013.stdout: "osdmap_epoch": 128, 2021-02-10T11:33:43.427 INFO:teuthology.orchestra.run.gibba013.stdout: "osdmap_epoch_barrier": 128, 2021-02-10T11:33:43.427 INFO:teuthology.orchestra.run.gibba013.stdout: "uptime": 16.188811303000001 2021-02-10T11:33:43.427 INFO:teuthology.orchestra.run.gibba013.stdout:} 2021-02-10T11:33:43.430 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "68db30ea-dbce-4aef-a6df-f01c6fea0513", "whoami": 0, "id": 7882, "want_state": "up:active", "state": "up:active", "rank_uptime": 10.578570643999999, "mdsmap_epoch": 107, "osdmap_epoch": 128, "osdmap_epoch_barrier": 128, "uptime": 16.188811303000001 } 2021-02-10T11:33:43.431 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2021-02-10T11:33:43.431 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2021-02-10T11:33:43.431 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@gibba002.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2021-02-10T11:33:43.431 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:33:43.431 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec mkdir -p /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:33:43.767 DEBUG:teuthology.orchestra.run.gibba002:> sudo modprobe fuse 2021-02-10T11:33:44.112 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:33:44.113 DEBUG:teuthology.orchestra.run.gibba002:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:33:44.457 INFO:teuthology.orchestra.run.gibba002.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:33:44.457 INFO:teuthology.orchestra.run.gibba002.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:33:44.457 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:33:44.458 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:33:44.458 DEBUG:teuthology.orchestra.run.gibba002:> ls /sys/fs/fuse/connections 2021-02-10T11:33:44.793 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2021-02-10T11:33:44.793 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:33:44.795 DEBUG:teuthology.orchestra.run.gibba002:> sudo modprobe fuse 2021-02-10T11:33:45.141 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:33:45.142 DEBUG:teuthology.orchestra.run.gibba002:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:33:45.209 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba002.stderr:ceph-fuse[189994]: starting ceph client 2021-02-10T11:33:45.209 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba002.stderr:2021-02-10 11:33:45.207 7f0231a36f80 -1 init, newargv = 0x55ab23a111a0 newargc=9 2021-02-10T11:33:45.214 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba002.stderr:ceph-fuse[189994]: starting fuse 2021-02-10T11:33:45.485 INFO:teuthology.orchestra.run.gibba002.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:33:45.485 INFO:teuthology.orchestra.run.gibba002.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:33:45.486 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:33:45.486 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:33:45.486 DEBUG:teuthology.orchestra.run.gibba002:> ls /sys/fs/fuse/connections 2021-02-10T11:33:45.821 INFO:teuthology.orchestra.run.gibba002.stdout:43 2021-02-10T11:33:45.821 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2021-02-10T11:33:45.821 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:33:45.821 DEBUG:teuthology.orchestra.run.gibba002:> sudo python3 -c ' 2021-02-10T11:33:45.821 DEBUG:teuthology.orchestra.run.gibba002:> import glob 2021-02-10T11:33:45.822 DEBUG:teuthology.orchestra.run.gibba002:> import re 2021-02-10T11:33:45.822 DEBUG:teuthology.orchestra.run.gibba002:> import os 2021-02-10T11:33:45.822 DEBUG:teuthology.orchestra.run.gibba002:> import subprocess 2021-02-10T11:33:45.822 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:33:45.822 DEBUG:teuthology.orchestra.run.gibba002:> def find_socket(client_name): 2021-02-10T11:33:45.822 DEBUG:teuthology.orchestra.run.gibba002:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2021-02-10T11:33:45.822 DEBUG:teuthology.orchestra.run.gibba002:> files = glob.glob(asok_path) 2021-02-10T11:33:45.822 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:33:45.822 DEBUG:teuthology.orchestra.run.gibba002:> # Given a non-glob path, it better be there 2021-02-10T11:33:45.822 DEBUG:teuthology.orchestra.run.gibba002:> if "*" not in asok_path: 2021-02-10T11:33:45.822 DEBUG:teuthology.orchestra.run.gibba002:> assert(len(files) == 1) 2021-02-10T11:33:45.822 DEBUG:teuthology.orchestra.run.gibba002:> return files[0] 2021-02-10T11:33:45.822 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:33:45.822 DEBUG:teuthology.orchestra.run.gibba002:> for f in files: 2021-02-10T11:33:45.822 DEBUG:teuthology.orchestra.run.gibba002:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2021-02-10T11:33:45.823 DEBUG:teuthology.orchestra.run.gibba002:> if os.path.exists("/proc/{0}".format(pid)): 2021-02-10T11:33:45.823 DEBUG:teuthology.orchestra.run.gibba002:> return f 2021-02-10T11:33:45.823 DEBUG:teuthology.orchestra.run.gibba002:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2021-02-10T11:33:45.823 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:33:45.823 DEBUG:teuthology.orchestra.run.gibba002:> print(find_socket("client.0")) 2021-02-10T11:33:45.823 DEBUG:teuthology.orchestra.run.gibba002:> ' 2021-02-10T11:33:46.184 INFO:teuthology.orchestra.run.gibba002.stdout:/var/run/ceph/ceph-client.0.189994.asok 2021-02-10T11:33:46.187 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.189994.asok 2021-02-10T11:33:46.187 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:33:46.187 DEBUG:teuthology.orchestra.run.gibba002:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.189994.asok status 2021-02-10T11:33:46.576 INFO:teuthology.orchestra.run.gibba002.stdout:{ 2021-02-10T11:33:46.577 INFO:teuthology.orchestra.run.gibba002.stdout: "metadata": { 2021-02-10T11:33:46.577 INFO:teuthology.orchestra.run.gibba002.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:33:46.577 INFO:teuthology.orchestra.run.gibba002.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:33:46.577 INFO:teuthology.orchestra.run.gibba002.stdout: "entity_id": "0", 2021-02-10T11:33:46.577 INFO:teuthology.orchestra.run.gibba002.stdout: "hostname": "gibba002", 2021-02-10T11:33:46.577 INFO:teuthology.orchestra.run.gibba002.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2021-02-10T11:33:46.577 INFO:teuthology.orchestra.run.gibba002.stdout: "pid": "189994", 2021-02-10T11:33:46.577 INFO:teuthology.orchestra.run.gibba002.stdout: "root": "/" 2021-02-10T11:33:46.577 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:33:46.577 INFO:teuthology.orchestra.run.gibba002.stdout: "dentry_count": 0, 2021-02-10T11:33:46.577 INFO:teuthology.orchestra.run.gibba002.stdout: "dentry_pinned_count": 0, 2021-02-10T11:33:46.577 INFO:teuthology.orchestra.run.gibba002.stdout: "id": 11274, 2021-02-10T11:33:46.577 INFO:teuthology.orchestra.run.gibba002.stdout: "inst": { 2021-02-10T11:33:46.578 INFO:teuthology.orchestra.run.gibba002.stdout: "name": { 2021-02-10T11:33:46.578 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "client", 2021-02-10T11:33:46.578 INFO:teuthology.orchestra.run.gibba002.stdout: "num": 11274 2021-02-10T11:33:46.578 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:33:46.578 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": { 2021-02-10T11:33:46.578 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:33:46.578 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:0", 2021-02-10T11:33:46.578 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 1236577181 2021-02-10T11:33:46.578 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:33:46.578 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:33:46.578 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": { 2021-02-10T11:33:46.578 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:33:46.578 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:0", 2021-02-10T11:33:46.579 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 1236577181 2021-02-10T11:33:46.579 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:33:46.579 INFO:teuthology.orchestra.run.gibba002.stdout: "inst_str": "client.11274 172.21.2.102:0/1236577181", 2021-02-10T11:33:46.579 INFO:teuthology.orchestra.run.gibba002.stdout: "addr_str": "172.21.2.102:0/1236577181", 2021-02-10T11:33:46.579 INFO:teuthology.orchestra.run.gibba002.stdout: "inode_count": 1, 2021-02-10T11:33:46.579 INFO:teuthology.orchestra.run.gibba002.stdout: "mds_epoch": 107, 2021-02-10T11:33:46.579 INFO:teuthology.orchestra.run.gibba002.stdout: "osd_epoch": 128, 2021-02-10T11:33:46.579 INFO:teuthology.orchestra.run.gibba002.stdout: "osd_epoch_barrier": 0, 2021-02-10T11:33:46.579 INFO:teuthology.orchestra.run.gibba002.stdout: "blacklisted": false 2021-02-10T11:33:46.579 INFO:teuthology.orchestra.run.gibba002.stdout:} 2021-02-10T11:33:46.580 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:33:46.581 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:33:46.581 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:33:46.916 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:33:46.917 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:33:46.917 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:33:46.917 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:33:47.264 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:33:47.694 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:47.674 7f8c7d27b700 1 Processor -- start 2021-02-10T11:33:47.694 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:47.674 7f8c7d27b700 1 -- start start 2021-02-10T11:33:47.694 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:47.675 7f8c7d27b700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f8c780d1870 0x7f8c781b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:47.694 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:47.675 7f8c7d27b700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f8c781b0a10 0x7f8c780cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:47.694 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:47.675 7f8c7d27b700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c780f0b10 con 0x7f8c780d1c70 2021-02-10T11:33:47.694 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:47.675 7f8c7d27b700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c780e5ed0 con 0x7f8c781b0a10 2021-02-10T11:33:47.694 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:47.675 7f8c7d27b700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c780f0650 con 0x7f8c780d1870 2021-02-10T11:33:47.694 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:47.675 7f8c7687c700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f8c781b0a10 0x7f8c780cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:47.695 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:47.675 7f8c7687c700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f8c781b0a10 0x7f8c780cf330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:35584/0 (socket says 172.21.2.102:35584) 2021-02-10T11:33:47.695 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:47.675 7f8c7687c700 1 -- 172.21.2.102:0/2508893270 learned_addr learned my addr 172.21.2.102:0/2508893270 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:33:47.695 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:47.676 7f8c7687c700 1 -- 172.21.2.102:0/2508893270 >> v2:172.21.2.113:3300/0 conn(0x7f8c780d1870 msgr2=0x7f8c781b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:33:47.695 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:47.676 7f8c7687c700 1 --2- 172.21.2.102:0/2508893270 >> v2:172.21.2.113:3300/0 conn(0x7f8c780d1870 0x7f8c781b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:47.695 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:47.676 7f8c7687c700 1 -- 172.21.2.102:0/2508893270 >> v1:172.21.2.102:6789/0 conn(0x7f8c780d1c70 legacy=0x7f8c780cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:47.695 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:47.676 7f8c7687c700 1 -- 172.21.2.102:0/2508893270 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c780f0840 con 0x7f8c781b0a10 2021-02-10T11:33:47.695 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:47.676 7f8c7687c700 1 --2- 172.21.2.102:0/2508893270 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8c781b0a10 0x7f8c780cf330 secure :-1 s=READY pgs=2456 cs=0 l=1 rev1=1 rx=0x7f8c68006700 tx=0x7f8c68009b80).ready entity=mon.0 client_cookie=3154182dcaaf3d32 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:47.695 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:47.676 7f8c7607b700 1 -- 172.21.2.102:0/2508893270 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8c6800c0c0 con 0x7f8c781b0a10 2021-02-10T11:33:47.695 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:47.676 7f8c7607b700 1 -- 172.21.2.102:0/2508893270 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c6800c340 con 0x7f8c781b0a10 2021-02-10T11:33:47.695 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:47.676 7f8c7607b700 1 -- 172.21.2.102:0/2508893270 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8c6800c860 con 0x7f8c781b0a10 2021-02-10T11:33:47.695 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:47.676 7f8c7d27b700 1 -- 172.21.2.102:0/2508893270 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8c781b0a10 msgr2=0x7f8c780cf330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:47.695 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:47.676 7f8c7d27b700 1 --2- 172.21.2.102:0/2508893270 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8c781b0a10 0x7f8c780cf330 secure :-1 s=READY pgs=2456 cs=0 l=1 rev1=1 rx=0x7f8c68006700 tx=0x7f8c68009b80).stop 2021-02-10T11:33:47.695 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:47.677 7f8c7d27b700 1 -- 172.21.2.102:0/2508893270 shutdown_connections 2021-02-10T11:33:47.696 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:47.677 7f8c7d27b700 1 --2- 172.21.2.102:0/2508893270 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8c781b0a10 0x7f8c780cf330 unknown :-1 s=CLOSED pgs=2456 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:47.696 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:47.677 7f8c7d27b700 1 --2- 172.21.2.102:0/2508893270 >> v2:172.21.2.113:3300/0 conn(0x7f8c780d1870 0x7f8c781b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:47.696 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:47.677 7f8c7d27b700 1 -- 172.21.2.102:0/2508893270 >> 172.21.2.102:0/2508893270 conn(0x7f8c780ce450 msgr2=0x7f8c780d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:47.696 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:47.677 7f8c7d27b700 1 -- 172.21.2.102:0/2508893270 shutdown_connections 2021-02-10T11:33:47.696 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:47.677 7f8c7d27b700 1 -- 172.21.2.102:0/2508893270 wait complete. 2021-02-10T11:33:47.696 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:47.692 7f8c7d27b700 1 Processor -- start 2021-02-10T11:33:47.696 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:47.692 7f8c7d27b700 1 -- start start 2021-02-10T11:33:47.696 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:47.692 7f8c7d27b700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8c780b4f30 0x7f8c7818b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:47.696 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:47.692 7f8c7d27b700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8c78093f40 0x7f8c78094340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:47.696 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:47.692 7f8c7d27b700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8c78094960 0x7f8c78187e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:47.696 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:47.692 7f8c7d27b700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c78094e90 con 0x7f8c78093f40 2021-02-10T11:33:47.696 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:47.692 7f8c7d27b700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c7818bff0 con 0x7f8c78094960 2021-02-10T11:33:47.696 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:47.692 7f8c7d27b700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c7808f780 con 0x7f8c780b4f30 2021-02-10T11:33:47.696 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:47.692 7f8c7687c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8c78093f40 0x7f8c78094340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:47.697 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:47.692 7f8c7687c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8c78093f40 0x7f8c78094340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:35594/0 (socket says 172.21.2.102:35594) 2021-02-10T11:33:47.697 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:47.692 7f8c7687c700 1 -- 172.21.2.102:0/257890996 learned_addr learned my addr 172.21.2.102:0/257890996 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:33:47.698 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:47.693 7f8c7687c700 1 -- 172.21.2.102:0/257890996 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8c780b4f30 msgr2=0x7f8c7818b9d0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:33:47.698 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:47.693 7f8c7687c700 1 --2- 172.21.2.102:0/257890996 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8c780b4f30 0x7f8c7818b9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:47.698 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:47.693 7f8c7687c700 1 -- 172.21.2.102:0/257890996 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8c78094960 msgr2=0x7f8c78187e50 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:33:47.698 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:47.693 7f8c7687c700 1 --2- 172.21.2.102:0/257890996 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8c78094960 0x7f8c78187e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:47.698 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:47.693 7f8c7687c700 1 -- 172.21.2.102:0/257890996 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c78188470 con 0x7f8c78093f40 2021-02-10T11:33:47.698 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:47.693 7f8c7687c700 1 --2- 172.21.2.102:0/257890996 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8c78093f40 0x7f8c78094340 secure :-1 s=READY pgs=2457 cs=0 l=1 rev1=1 rx=0x7f8c6800b5e0 tx=0x7f8c6800b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:47.698 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:47.693 7f8c6ffff700 1 -- 172.21.2.102:0/257890996 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8c6800bd70 con 0x7f8c78093f40 2021-02-10T11:33:47.698 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:47.693 7f8c6ffff700 1 -- 172.21.2.102:0/257890996 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c68002030 con 0x7f8c78093f40 2021-02-10T11:33:47.698 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:47.693 7f8c7d27b700 1 -- 172.21.2.102:0/257890996 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c78188470 con 0x7f8c78093f40 2021-02-10T11:33:47.698 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:47.693 7f8c6ffff700 1 -- 172.21.2.102:0/257890996 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8c680024d0 con 0x7f8c78093f40 2021-02-10T11:33:47.698 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:47.693 7f8c7d27b700 1 -- 172.21.2.102:0/257890996 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c78188a00 con 0x7f8c78093f40 2021-02-10T11:33:47.699 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:47.694 7f8c6ffff700 1 -- 172.21.2.102:0/257890996 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f8c68002210 con 0x7f8c78093f40 2021-02-10T11:33:47.699 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:47.694 7f8c6ffff700 1 --2- 172.21.2.102:0/257890996 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8c5c018730 0x7f8c5c01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:47.699 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:47.694 7f8c6ffff700 1 -- 172.21.2.102:0/257890996 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f8c68032600 con 0x7f8c78093f40 2021-02-10T11:33:47.699 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:47.694 7f8c7707d700 1 --2- 172.21.2.102:0/257890996 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8c5c018730 0x7f8c5c01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:47.700 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:47.694 7f8c7707d700 1 --2- 172.21.2.102:0/257890996 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8c5c018730 0x7f8c5c01ac10 crc :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:47.726 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:47.725 7f8c7d27b700 1 -- 172.21.2.102:0/257890996 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c780ce450 con 0x7f8c78093f40 2021-02-10T11:33:47.729 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:47.727 7f8c6ffff700 1 -- 172.21.2.102:0/257890996 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f8c680009b0 con 0x7f8c78093f40 2021-02-10T11:33:47.852 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:47.850 7f8c7d27b700 1 -- 172.21.2.102:0/257890996 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8c780ce450 con 0x7f8c78093f40 2021-02-10T11:33:47.853 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:47.851 7f8c6ffff700 1 -- 172.21.2.102:0/257890996 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v128) v1 ==== 74+0+14007 (secure 0 0 0) 0x7f8c68002030 con 0x7f8c78093f40 2021-02-10T11:33:47.853 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:33:47.853 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":128,"fsid":"68db30ea-dbce-4aef-a6df-f01c6fea0513","created":"2021-02-10 11:03:06.331557","modified":"2021-02-10 11:33:33.846924","last_up_change":"2021-02-10 11:03:32.690562","last_in_change":"2021-02-10 11:03:32.690562","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":27,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:03:37.322519","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":26,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:33:30.653221","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"127","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":27,"pool_name":"cephfs_data","create_time":"2021-02-10 11:33:31.831765","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"128","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"be691b4d-a039-408a-af80-9d53f02b6f1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6818","nonce":36088},{"type":"v1","addr":"172.21.2.102:6820","nonce":36088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6822","nonce":36088},{"type":"v1","addr":"172.21.2.102:6824","nonce":36088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6830","nonce":36088},{"type":"v1","addr":"172.21.2.102:6832","nonce":36088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6826","nonce":36088},{"type":"v1","addr":"172.21.2.102:6828","nonce":36088}]},"public_addr":"172.21.2.102:6820/36088","cluster_addr":"172.21.2.102:6824/36088","heartbeat_back_addr":"172.21.2.102:6832/36088","heartbeat_front_addr":"172.21.2.102:6828/36088","state":["exists","up"]},{"osd":1,"uuid":"64e3ccc8-9c69-4d7c-865e-af8e83822635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6819","nonce":36087},{"type":"v1","addr":"172.21.2.102:6821","nonce":36087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6823","nonce":36087},{"type":"v1","addr":"172.21.2.102:6825","nonce":36087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6831","nonce":36087},{"type":"v1","addr":"172.21.2.102:6833","nonce":36087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6827","nonce":36087},{"type":"v1","addr":"172.21.2.102:6829","nonce":36087}]},"public_addr":"172.21.2.102:6821/36087","cluster_addr":"172.21.2.102:6825/36087","heartbeat_back_addr":"172.21.2.102:6833/36087","heartbeat_front_addr":"172.21.2.102:6829/36087","state":["exists","up"]},{"osd":2,"uuid":"153700af-747f-481f-bdf4-d6072d3a842f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6810","nonce":36089},{"type":"v1","addr":"172.21.2.102:6811","nonce":36089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6812","nonce":36089},{"type":"v1","addr":"172.21.2.102:6813","nonce":36089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6816","nonce":36089},{"type":"v1","addr":"172.21.2.102:6817","nonce":36089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6814","nonce":36089},{"type":"v1","addr":"172.21.2.102:6815","nonce":36089}]},"public_addr":"172.21.2.102:6811/36089","cluster_addr":"172.21.2.102:6813/36089","heartbeat_back_addr":"172.21.2.102:6817/36089","heartbeat_front_addr":"172.21.2.102:6815/36089","state":["exists","up"]},{"osd":3,"uuid":"cbc678f4-83a9-45e6-8830-2f7c232b1ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6802","nonce":36090},{"type":"v1","addr":"172.21.2.102:6803","nonce":36090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6804","nonce":36090},{"type":"v1","addr":"172.21.2.102:6805","nonce":36090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6808","nonce":36090},{"type":"v1","addr":"172.21.2.102:6809","nonce":36090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6806","nonce":36090},{"type":"v1","addr":"172.21.2.102:6807","nonce":36090}]},"public_addr":"172.21.2.102:6803/36090","cluster_addr":"172.21.2.102:6805/36090","heartbeat_back_addr":"172.21.2.102:6809/36090","heartbeat_front_addr":"172.21.2.102:6807/36090","state":["exists","up"]},{"osd":4,"uuid":"c01e4008-b258-4f10-afd2-9e7f37214351","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6808","nonce":36368},{"type":"v1","addr":"172.21.2.113:6811","nonce":36368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6814","nonce":36368},{"type":"v1","addr":"172.21.2.113:6818","nonce":36368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6826","nonce":36368},{"type":"v1","addr":"172.21.2.113:6828","nonce":36368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6820","nonce":36368},{"type":"v1","addr":"172.21.2.113:6824","nonce":36368}]},"public_addr":"172.21.2.113:6811/36368","cluster_addr":"172.21.2.113:6818/36368","heartbeat_back_addr":"172.21.2.113:6828/36368","heartbeat_front_addr":"172.21.2.113:6824/36368","state":["exists","up"]},{"osd":5,"uuid":"551f6007-daf8-4a98-8ee0-c2b16bcccb5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6802","nonce":36366},{"type":"v1","addr":"172.21.2.113:6805","nonce":36366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6807","nonce":36366},{"type":"v1","addr":"172.21.2.113:6810","nonce":36366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6819","nonce":36366},{"type":"v1","addr":"172.21.2.113:6822","nonce":36366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6812","nonce":36366},{"type":"v1","addr":"172.21.2.113:6815","nonce":36366}]},"public_addr":"172.21.2.113:6805/36366","cluster_addr":"172.21.2.113:6810/36366","heartbeat_back_addr":"172.21.2.113:6822/36366","heartbeat_front_addr":"172.21.2.113:6815/36366","state":["exists","up"]},{"osd":6,"uuid":"bf67c040-0e50-48be-b102-c3943bc9ebef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6800","nonce":36367},{"type":"v1","addr":"172.21.2.113:6801","nonce":36367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6803","nonce":36367},{"type":"v1","addr":"172.21.2.113:6804","nonce":36367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6813","nonce":36367},{"type":"v1","addr":"172.21.2.113:6816","nonce":36367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6806","nonce":36367},{"type":"v1","addr":"172.21.2.113:6809","nonce":36367}]},"public_addr":"172.21.2.113:6801/36367","cluster_addr":"172.21.2.113:6804/36367","heartbeat_back_addr":"172.21.2.113:6816/36367","heartbeat_front_addr":"172.21.2.113:6809/36367","state":["exists","up"]},{"osd":7,"uuid":"f5461f80-018d-428f-be89-2eb2bcbe7bb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6817","nonce":36365},{"type":"v1","addr":"172.21.2.113:6821","nonce":36365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6823","nonce":36365},{"type":"v1","addr":"172.21.2.113:6825","nonce":36365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6830","nonce":36365},{"type":"v1","addr":"172.21.2.113:6831","nonce":36365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6827","nonce":36365},{"type":"v1","addr":"172.21.2.113:6829","nonce":36365}]},"public_addr":"172.21.2.113:6821/36365","cluster_addr":"172.21.2.113:6825/36365","heartbeat_back_addr":"172.21.2.113:6831/36365","heartbeat_front_addr":"172.21.2.113:6829/36365","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:33:47.854 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:47.853 7f8c7d27b700 1 -- 172.21.2.102:0/257890996 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8c5c018730 msgr2=0x7f8c5c01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:47.854 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:47.853 7f8c7d27b700 1 --2- 172.21.2.102:0/257890996 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8c5c018730 0x7f8c5c01ac10 crc :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:47.854 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:47.853 7f8c7d27b700 1 -- 172.21.2.102:0/257890996 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8c78093f40 msgr2=0x7f8c78094340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:47.855 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:47.853 7f8c7d27b700 1 --2- 172.21.2.102:0/257890996 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8c78093f40 0x7f8c78094340 secure :-1 s=READY pgs=2457 cs=0 l=1 rev1=1 rx=0x7f8c6800b5e0 tx=0x7f8c6800b980).stop 2021-02-10T11:33:47.855 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:47.853 7f8c7d27b700 1 -- 172.21.2.102:0/257890996 shutdown_connections 2021-02-10T11:33:47.855 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:47.853 7f8c7d27b700 1 --2- 172.21.2.102:0/257890996 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8c5c018730 0x7f8c5c01ac10 unknown :-1 s=CLOSED pgs=1091 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:47.855 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:47.853 7f8c7d27b700 1 --2- 172.21.2.102:0/257890996 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8c78094960 0x7f8c78187e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:47.855 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:47.853 7f8c7d27b700 1 --2- 172.21.2.102:0/257890996 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8c78093f40 0x7f8c78094340 unknown :-1 s=CLOSED pgs=2457 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:47.855 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:47.853 7f8c7d27b700 1 --2- 172.21.2.102:0/257890996 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8c780b4f30 0x7f8c7818b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:47.855 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:47.853 7f8c7d27b700 1 -- 172.21.2.102:0/257890996 >> 172.21.2.102:0/257890996 conn(0x7f8c7819f8f0 msgr2=0x7f8c781a5f80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:47.855 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:47.853 7f8c7d27b700 1 -- 172.21.2.102:0/257890996 shutdown_connections 2021-02-10T11:33:47.856 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:47.853 7f8c7d27b700 1 -- 172.21.2.102:0/257890996 wait complete. 2021-02-10T11:33:47.863 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:03:37.322519', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 26, 'pool_name': 'cephfs_metadata', 'create_time': '2021-02-10 11:33:30.653221', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '127', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 27, 'pool_name': 'cephfs_data', 'create_time': '2021-02-10 11:33:31.831765', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '128', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-02-10T11:33:47.864 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:33:48.297 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:48.278 7fd3b3c16700 1 Processor -- start 2021-02-10T11:33:48.298 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:48.278 7fd3b3c16700 1 -- start start 2021-02-10T11:33:48.298 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:48.279 7fd3b3c16700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd3ac0d1c70 0x7fd3ac0cef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:48.298 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:48.279 7fd3b3c16700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3ac0d0170 con 0x7fd3ac0cf580 2021-02-10T11:33:48.298 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:48.279 7fd3b3c16700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3ac0fe1c0 con 0x7fd3ac0d1870 2021-02-10T11:33:48.298 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:48.279 7fd3b3c16700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3ac1b0cb0 con 0x7fd3ac0d1c70 2021-02-10T11:33:48.298 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:48.279 7fd3b2233700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fd3ac0cf580 0x7fd3ac0cf980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:56058/0 (socket says 172.21.2.102:56058) 2021-02-10T11:33:48.298 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:48.279 7fd3b2233700 1 -- 172.21.2.102:0/295605857 learned_addr learned my addr 172.21.2.102:0/295605857 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:33:48.298 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:48.279 7fd3b0a30700 1 -- 172.21.2.102:0/295605857 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3145524731 0 0) 0x7fd3a4001500 con 0x7fd3ac0cf580 2021-02-10T11:33:48.298 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:48.279 7fd3b0a30700 1 -- 172.21.2.102:0/295605857 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd398002fe0 con 0x7fd3ac0cf580 2021-02-10T11:33:48.298 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:48.280 7fd3b1231700 1 --2- 172.21.2.102:0/295605857 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd3ac0d1c70 0x7fd3ac0cef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:48.299 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:48.280 7fd3b0a30700 1 -- 172.21.2.102:0/295605857 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2455771196 0 0) 0x7fd3a4001460 con 0x7fd3ac0cf580 2021-02-10T11:33:48.299 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:48.280 7fd3b0a30700 1 -- 172.21.2.102:0/295605857 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd3ac0d1c70 msgr2=0x7fd3ac0cef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:33:48.299 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:48.280 7fd3b0a30700 1 --2- 172.21.2.102:0/295605857 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd3ac0d1c70 0x7fd3ac0cef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:48.299 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:48.280 7fd3b0a30700 1 -- 172.21.2.102:0/295605857 >> v1:172.21.2.113:6789/0 conn(0x7fd3ac0d1870 legacy=0x7fd3ac1b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:48.299 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:48.280 7fd3b0a30700 1 -- 172.21.2.102:0/295605857 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3ac109a60 con 0x7fd3ac0cf580 2021-02-10T11:33:48.299 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:48.280 7fd3b0a30700 1 -- 172.21.2.102:0/295605857 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7fd3a4001980 con 0x7fd3ac0cf580 2021-02-10T11:33:48.299 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:48.280 7fd3b0a30700 1 -- 172.21.2.102:0/295605857 >> v1:172.21.2.102:6789/0 conn(0x7fd3ac0cf580 legacy=0x7fd3ac0cf980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:48.299 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:48.280 7fd3b0a30700 1 --2- 172.21.2.102:0/295605857 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd398003ae0 0x7fd398005f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:48.299 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:48.280 7fd3b0a30700 1 --2- 172.21.2.102:0/295605857 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd398006810 0x7fd398008d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:48.299 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:48.280 7fd3b0a30700 1 --2- 172.21.2.102:0/295605857 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd3980095a0 0x7fd39800bac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:48.299 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:48.280 7fd3b0a30700 1 -- 172.21.2.102:0/295605857 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd39800c370 con 0x7fd3980095a0 2021-02-10T11:33:48.299 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:48.280 7fd3b0a30700 1 -- 172.21.2.102:0/295605857 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd39800c590 con 0x7fd398003ae0 2021-02-10T11:33:48.299 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:48.280 7fd3b0a30700 1 -- 172.21.2.102:0/295605857 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd39800c7b0 con 0x7fd398006810 2021-02-10T11:33:48.300 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:48.280 7fd3b1231700 1 --2- 172.21.2.102:0/295605857 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd3ac0d1c70 0x7fd3ac0cef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:33:48.300 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:48.280 7fd3b2233700 1 --2- 172.21.2.102:0/295605857 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd3980095a0 0x7fd39800bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:48.300 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:48.280 7fd3b1a32700 1 --2- 172.21.2.102:0/295605857 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd398003ae0 0x7fd398005f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:48.300 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:48.280 7fd3b1231700 1 --2- 172.21.2.102:0/295605857 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd398006810 0x7fd398008d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:48.300 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:48.280 7fd3b2233700 1 -- 172.21.2.102:0/295605857 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd398006810 msgr2=0x7fd398008d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:33:48.300 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:48.280 7fd3b2233700 1 --2- 172.21.2.102:0/295605857 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd398006810 0x7fd398008d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:48.300 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:48.280 7fd3b2233700 1 -- 172.21.2.102:0/295605857 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd398003ae0 msgr2=0x7fd398005f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:33:48.300 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:48.280 7fd3b2233700 1 --2- 172.21.2.102:0/295605857 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd398003ae0 0x7fd398005f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:48.300 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:48.280 7fd3b2233700 1 -- 172.21.2.102:0/295605857 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd39800ca50 con 0x7fd3980095a0 2021-02-10T11:33:48.300 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:48.280 7fd3b2233700 1 --2- 172.21.2.102:0/295605857 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd3980095a0 0x7fd39800bac0 secure :-1 s=READY pgs=2459 cs=0 l=1 rev1=1 rx=0x7fd3a4001140 tx=0x7fd3a404e320).ready entity=mon.0 client_cookie=a9d5a4d0eb1b52d0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:48.300 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:48.280 7fd3b0a30700 1 -- 172.21.2.102:0/295605857 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd3a4001460 con 0x7fd3980095a0 2021-02-10T11:33:48.300 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:48.280 7fd3b0a30700 1 -- 172.21.2.102:0/295605857 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3a40070d0 con 0x7fd3980095a0 2021-02-10T11:33:48.300 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:48.280 7fd3b0a30700 1 -- 172.21.2.102:0/295605857 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd3a4007520 con 0x7fd3980095a0 2021-02-10T11:33:48.300 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:48.280 7fd3b3c16700 1 -- 172.21.2.102:0/295605857 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd3980095a0 msgr2=0x7fd39800bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:48.301 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:48.280 7fd3b3c16700 1 --2- 172.21.2.102:0/295605857 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd3980095a0 0x7fd39800bac0 secure :-1 s=READY pgs=2459 cs=0 l=1 rev1=1 rx=0x7fd3a4001140 tx=0x7fd3a404e320).stop 2021-02-10T11:33:48.301 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:48.280 7fd3b2233700 1 -- 172.21.2.102:0/295605857 reap_dead start 2021-02-10T11:33:48.301 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:48.281 7fd3b3c16700 1 -- 172.21.2.102:0/295605857 shutdown_connections 2021-02-10T11:33:48.301 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:48.281 7fd3b3c16700 1 -- 172.21.2.102:0/295605857 >> 172.21.2.102:0/295605857 conn(0x7fd3ac0ce420 msgr2=0x7fd3ac0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:48.301 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:48.281 7fd3b3c16700 1 -- 172.21.2.102:0/295605857 shutdown_connections 2021-02-10T11:33:48.301 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:48.281 7fd3b3c16700 1 -- 172.21.2.102:0/295605857 wait complete. 2021-02-10T11:33:48.301 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:48.296 7fd3b3c16700 1 Processor -- start 2021-02-10T11:33:48.301 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:48.296 7fd3b3c16700 1 -- start start 2021-02-10T11:33:48.303 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:48.296 7fd3b3c16700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd3ac09bcc0 0x7fd3ac09c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:48.303 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:48.296 7fd3b3c16700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd3ac0a0820 0x7fd3ac0a0c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:48.303 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:48.296 7fd3b3c16700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd3ac0a1240 0x7fd3ac0858d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:48.303 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:48.296 7fd3b3c16700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3ac0a66b0 con 0x7fd3ac0a1240 2021-02-10T11:33:48.303 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:48.296 7fd3b3c16700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3ac1a49b0 con 0x7fd3ac09bcc0 2021-02-10T11:33:48.303 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:48.296 7fd3b3c16700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3ac1a4b90 con 0x7fd3ac0a0820 2021-02-10T11:33:48.303 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:48.297 7fd3b2233700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd3ac0a1240 0x7fd3ac0858d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:48.303 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:48.297 7fd3b2233700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd3ac0a1240 0x7fd3ac0858d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:35610/0 (socket says 172.21.2.102:35610) 2021-02-10T11:33:48.304 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:48.297 7fd3b2233700 1 -- 172.21.2.102:0/140914864 learned_addr learned my addr 172.21.2.102:0/140914864 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:33:48.304 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:48.297 7fd3b1a32700 1 --2- 172.21.2.102:0/140914864 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd3ac09bcc0 0x7fd3ac09c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:48.304 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:48.297 7fd3b1231700 1 --2- 172.21.2.102:0/140914864 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd3ac0a0820 0x7fd3ac0a0c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:48.304 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:48.297 7fd3b2233700 1 -- 172.21.2.102:0/140914864 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd3ac0a0820 msgr2=0x7fd3ac0a0c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:48.304 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:48.297 7fd3b2233700 1 --2- 172.21.2.102:0/140914864 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd3ac0a0820 0x7fd3ac0a0c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:48.304 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:48.297 7fd3b2233700 1 -- 172.21.2.102:0/140914864 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd3ac09bcc0 msgr2=0x7fd3ac09c0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:48.304 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:48.297 7fd3b2233700 1 --2- 172.21.2.102:0/140914864 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd3ac09bcc0 0x7fd3ac09c0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:48.304 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:48.297 7fd3b2233700 1 -- 172.21.2.102:0/140914864 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3ac0a1640 con 0x7fd3ac0a1240 2021-02-10T11:33:48.304 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:48.297 7fd3b2233700 1 --2- 172.21.2.102:0/140914864 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd3ac0a1240 0x7fd3ac0858d0 secure :-1 s=READY pgs=2460 cs=0 l=1 rev1=1 rx=0x7fd3a4002190 tx=0x7fd3a404fc50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:48.304 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:48.297 7fd39e7fc700 1 -- 172.21.2.102:0/140914864 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd3a4007d20 con 0x7fd3ac0a1240 2021-02-10T11:33:48.304 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:48.297 7fd39e7fc700 1 -- 172.21.2.102:0/140914864 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3a4004930 con 0x7fd3ac0a1240 2021-02-10T11:33:48.304 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:48.297 7fd39e7fc700 1 -- 172.21.2.102:0/140914864 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd3a4004d90 con 0x7fd3ac0a1240 2021-02-10T11:33:48.304 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:48.298 7fd3b3c16700 1 -- 172.21.2.102:0/140914864 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3ac0a1640 con 0x7fd3ac0a1240 2021-02-10T11:33:48.305 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:48.298 7fd3b3c16700 1 -- 172.21.2.102:0/140914864 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3ac0a1640 con 0x7fd3ac0a1240 2021-02-10T11:33:48.305 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:48.298 7fd39e7fc700 1 -- 172.21.2.102:0/140914864 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd3a4007d20 con 0x7fd3ac0a1240 2021-02-10T11:33:48.305 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:48.298 7fd39e7fc700 1 --2- 172.21.2.102:0/140914864 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd3940186c0 0x7fd39401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:48.305 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:48.298 7fd39e7fc700 1 -- 172.21.2.102:0/140914864 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fd3a4076910 con 0x7fd3ac0a1240 2021-02-10T11:33:48.305 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:48.298 7fd3b1a32700 1 --2- 172.21.2.102:0/140914864 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd3940186c0 0x7fd39401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:48.305 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:48.299 7fd3b1a32700 1 --2- 172.21.2.102:0/140914864 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd3940186c0 0x7fd39401abc0 crc :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:48.331 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:48.329 7fd3b3c16700 1 -- 172.21.2.102:0/140914864 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3ac0fe1c0 con 0x7fd3ac0a1240 2021-02-10T11:33:48.333 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:48.331 7fd39e7fc700 1 -- 172.21.2.102:0/140914864 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd3a4004720 con 0x7fd3ac0a1240 2021-02-10T11:33:48.461 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:48.459 7fd3b3c16700 1 -- 172.21.2.102:0/140914864 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7fd3ac0fe1c0 con 0x7fd3ac0a1240 2021-02-10T11:33:48.461 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:48.459 7fd39e7fc700 1 -- 172.21.2.102:0/140914864 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v128) v1 ==== 92+0+10 (secure 0 0 0) 0x7fd3a4007d20 con 0x7fd3ac0a1240 2021-02-10T11:33:48.462 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:33:48.463 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:48.461 7fd3b3c16700 1 -- 172.21.2.102:0/140914864 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd3940186c0 msgr2=0x7fd39401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:48.463 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:48.461 7fd3b3c16700 1 --2- 172.21.2.102:0/140914864 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd3940186c0 0x7fd39401abc0 crc :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:48.464 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:48.461 7fd3b3c16700 1 -- 172.21.2.102:0/140914864 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd3ac0a1240 msgr2=0x7fd3ac0858d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:48.464 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:48.461 7fd3b3c16700 1 --2- 172.21.2.102:0/140914864 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd3ac0a1240 0x7fd3ac0858d0 secure :-1 s=READY pgs=2460 cs=0 l=1 rev1=1 rx=0x7fd3a4002190 tx=0x7fd3a404fc50).stop 2021-02-10T11:33:48.464 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:48.461 7fd3b3c16700 1 -- 172.21.2.102:0/140914864 shutdown_connections 2021-02-10T11:33:48.464 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:48.461 7fd3b3c16700 1 --2- 172.21.2.102:0/140914864 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd3ac0a1240 0x7fd3ac0858d0 unknown :-1 s=CLOSED pgs=2460 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:48.464 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:48.461 7fd3b3c16700 1 --2- 172.21.2.102:0/140914864 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd3ac0a0820 0x7fd3ac0a0c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:48.464 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:48.461 7fd3b3c16700 1 --2- 172.21.2.102:0/140914864 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd3940186c0 0x7fd39401abc0 unknown :-1 s=CLOSED pgs=1092 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:48.464 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:48.461 7fd3b3c16700 1 --2- 172.21.2.102:0/140914864 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd3ac09bcc0 0x7fd3ac09c0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:48.464 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:48.461 7fd3b3c16700 1 -- 172.21.2.102:0/140914864 >> 172.21.2.102:0/140914864 conn(0x7fd3ac0bef00 msgr2=0x7fd3ac09d1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:48.464 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:48.461 7fd3b3c16700 1 -- 172.21.2.102:0/140914864 shutdown_connections 2021-02-10T11:33:48.464 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:48.462 7fd3b3c16700 1 -- 172.21.2.102:0/140914864 wait complete. 2021-02-10T11:33:48.471 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-02-10T11:33:48.904 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:48.886 7fad3531e700 1 Processor -- start 2021-02-10T11:33:48.905 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:48.886 7fad3531e700 1 -- start start 2021-02-10T11:33:48.905 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:48.887 7fad3531e700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fad300d1870 0x7fad301b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:48.905 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:48.887 7fad3531e700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fad300cff70 0x7fad30109130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:48.905 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:48.887 7fad3531e700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad30109a60 con 0x7fad301b0ae0 2021-02-10T11:33:48.905 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:48.887 7fad3531e700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad30109750 con 0x7fad300d1870 2021-02-10T11:33:48.905 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:48.887 7fad3531e700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad30088070 con 0x7fad300cff70 2021-02-10T11:33:48.905 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:48.887 7fad2f07d700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fad300d1870 0x7fad301b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:48.905 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:48.887 7fad2f07d700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fad300d1870 0x7fad301b04c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:35616/0 (socket says 172.21.2.102:35616) 2021-02-10T11:33:48.905 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:48.887 7fad2f07d700 1 -- 172.21.2.102:0/4160943620 learned_addr learned my addr 172.21.2.102:0/4160943620 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:33:48.905 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:48.887 7fad2f87e700 1 --2- 172.21.2.102:0/4160943620 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fad300cff70 0x7fad30109130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:48.905 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:48.887 7fad2f07d700 1 -- 172.21.2.102:0/4160943620 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fad300cff70 msgr2=0x7fad30109130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:33:48.905 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:48.887 7fad2f07d700 1 --2- 172.21.2.102:0/4160943620 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fad300cff70 0x7fad30109130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:48.906 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:48.887 7fad2f07d700 1 -- 172.21.2.102:0/4160943620 >> v1:172.21.2.113:6789/0 conn(0x7fad301b0ae0 legacy=0x7fad30108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:48.906 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:48.887 7fad2f07d700 1 -- 172.21.2.102:0/4160943620 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad300d1c70 con 0x7fad300d1870 2021-02-10T11:33:48.906 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:48.887 7fad2f07d700 1 --2- 172.21.2.102:0/4160943620 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fad300d1870 0x7fad301b04c0 secure :-1 s=READY pgs=2461 cs=0 l=1 rev1=1 rx=0x7fad18006700 tx=0x7fad18009b80).ready entity=mon.0 client_cookie=627a4f8f8e131b3d server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:48.906 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:48.887 7fad2e07b700 1 -- 172.21.2.102:0/4160943620 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fad1800c0c0 con 0x7fad300d1870 2021-02-10T11:33:48.906 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:48.888 7fad2e07b700 1 -- 172.21.2.102:0/4160943620 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad1800c0c0 con 0x7fad300d1870 2021-02-10T11:33:48.906 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:48.888 7fad2e07b700 1 -- 172.21.2.102:0/4160943620 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fad1800c440 con 0x7fad300d1870 2021-02-10T11:33:48.906 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:48.888 7fad3531e700 1 -- 172.21.2.102:0/4160943620 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fad300d1870 msgr2=0x7fad301b04c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:48.906 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:48.888 7fad3531e700 1 --2- 172.21.2.102:0/4160943620 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fad300d1870 0x7fad301b04c0 secure :-1 s=READY pgs=2461 cs=0 l=1 rev1=1 rx=0x7fad18006700 tx=0x7fad18009b80).stop 2021-02-10T11:33:48.906 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:48.888 7fad3531e700 1 -- 172.21.2.102:0/4160943620 shutdown_connections 2021-02-10T11:33:48.906 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:48.888 7fad3531e700 1 --2- 172.21.2.102:0/4160943620 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fad300cff70 0x7fad30109130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:48.906 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:48.888 7fad3531e700 1 --2- 172.21.2.102:0/4160943620 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fad300d1870 0x7fad301b04c0 unknown :-1 s=CLOSED pgs=2461 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:48.906 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:48.888 7fad3531e700 1 -- 172.21.2.102:0/4160943620 >> 172.21.2.102:0/4160943620 conn(0x7fad300ce420 msgr2=0x7fad300d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:48.907 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:48.888 7fad3531e700 1 -- 172.21.2.102:0/4160943620 shutdown_connections 2021-02-10T11:33:48.907 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:48.888 7fad3531e700 1 -- 172.21.2.102:0/4160943620 wait complete. 2021-02-10T11:33:48.907 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:48.903 7fad3531e700 1 Processor -- start 2021-02-10T11:33:48.908 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:48.903 7fad3531e700 1 -- start start 2021-02-10T11:33:48.908 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:48.904 7fad3531e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fad300926d0 0x7fad30092ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:48.908 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:48.904 7fad3531e700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fad300930f0 0x7fad30093f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:48.909 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:48.904 7fad3531e700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fad300934f0 0x7fad300945b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:48.909 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:48.904 7fad3531e700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad30093a10 con 0x7fad300926d0 2021-02-10T11:33:48.909 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:48.904 7fad3531e700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad3009efa0 con 0x7fad300930f0 2021-02-10T11:33:48.909 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:48.904 7fad3531e700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad3009f180 con 0x7fad300934f0 2021-02-10T11:33:48.909 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:48.904 7fad2f07d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fad300926d0 0x7fad30092ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:48.909 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:48.904 7fad2f07d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fad300926d0 0x7fad30092ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:35624/0 (socket says 172.21.2.102:35624) 2021-02-10T11:33:48.909 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:48.904 7fad2f07d700 1 -- 172.21.2.102:0/995750101 learned_addr learned my addr 172.21.2.102:0/995750101 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:33:48.909 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:48.904 7fad2e87c700 1 --2- 172.21.2.102:0/995750101 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fad300930f0 0x7fad30093f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:48.909 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:48.904 7fad2f87e700 1 --2- 172.21.2.102:0/995750101 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fad300934f0 0x7fad300945b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:48.909 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:48.904 7fad2f07d700 1 -- 172.21.2.102:0/995750101 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fad300934f0 msgr2=0x7fad300945b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:48.909 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:48.904 7fad2f07d700 1 --2- 172.21.2.102:0/995750101 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fad300934f0 0x7fad300945b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:48.909 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:48.904 7fad2f07d700 1 -- 172.21.2.102:0/995750101 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fad300930f0 msgr2=0x7fad30093f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:48.909 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:48.904 7fad2f07d700 1 --2- 172.21.2.102:0/995750101 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fad300930f0 0x7fad30093f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:48.910 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:48.904 7fad2f07d700 1 -- 172.21.2.102:0/995750101 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad3009f440 con 0x7fad300926d0 2021-02-10T11:33:48.910 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:48.904 7fad2f07d700 1 --2- 172.21.2.102:0/995750101 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fad300926d0 0x7fad30092ad0 secure :-1 s=READY pgs=2462 cs=0 l=1 rev1=1 rx=0x7fad1800b5e0 tx=0x7fad1800b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:48.910 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:48.905 7fad27fff700 1 -- 172.21.2.102:0/995750101 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fad1800bd70 con 0x7fad300926d0 2021-02-10T11:33:48.910 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:48.905 7fad27fff700 1 -- 172.21.2.102:0/995750101 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad18002030 con 0x7fad300926d0 2021-02-10T11:33:48.910 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:48.905 7fad3531e700 1 -- 172.21.2.102:0/995750101 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad3009f440 con 0x7fad300926d0 2021-02-10T11:33:48.910 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:48.905 7fad27fff700 1 -- 172.21.2.102:0/995750101 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fad180024d0 con 0x7fad300926d0 2021-02-10T11:33:48.910 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:48.905 7fad3531e700 1 -- 172.21.2.102:0/995750101 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad3009f440 con 0x7fad300926d0 2021-02-10T11:33:48.910 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:48.905 7fad27fff700 1 -- 172.21.2.102:0/995750101 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fad18002210 con 0x7fad300926d0 2021-02-10T11:33:48.910 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:48.906 7fad27fff700 1 --2- 172.21.2.102:0/995750101 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fad140186c0 0x7fad1401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:48.910 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:48.906 7fad27fff700 1 -- 172.21.2.102:0/995750101 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fad18032560 con 0x7fad300926d0 2021-02-10T11:33:48.910 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:48.906 7fad2e87c700 1 --2- 172.21.2.102:0/995750101 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fad140186c0 0x7fad1401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:48.910 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:48.906 7fad2e87c700 1 --2- 172.21.2.102:0/995750101 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fad140186c0 0x7fad1401abc0 crc :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:48.939 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:48.937 7fad3531e700 1 -- 172.21.2.102:0/995750101 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad30082a70 con 0x7fad300926d0 2021-02-10T11:33:48.941 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:48.939 7fad27fff700 1 -- 172.21.2.102:0/995750101 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fad18030d10 con 0x7fad300926d0 2021-02-10T11:33:49.072 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:49.070 7fad3531e700 1 -- 172.21.2.102:0/995750101 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7fad30082a70 con 0x7fad300926d0 2021-02-10T11:33:49.072 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:49.070 7fad27fff700 1 -- 172.21.2.102:0/995750101 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v128) v1 ==== 104+0+10 (secure 0 0 0) 0x7fad18002210 con 0x7fad300926d0 2021-02-10T11:33:49.073 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:33:49.074 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:49.072 7fad3531e700 1 -- 172.21.2.102:0/995750101 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fad140186c0 msgr2=0x7fad1401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:49.074 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:49.072 7fad3531e700 1 --2- 172.21.2.102:0/995750101 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fad140186c0 0x7fad1401abc0 crc :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:49.074 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:49.072 7fad3531e700 1 -- 172.21.2.102:0/995750101 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fad300926d0 msgr2=0x7fad30092ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:49.074 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:49.072 7fad3531e700 1 --2- 172.21.2.102:0/995750101 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fad300926d0 0x7fad30092ad0 secure :-1 s=READY pgs=2462 cs=0 l=1 rev1=1 rx=0x7fad1800b5e0 tx=0x7fad1800b980).stop 2021-02-10T11:33:49.074 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:49.072 7fad3531e700 1 -- 172.21.2.102:0/995750101 shutdown_connections 2021-02-10T11:33:49.075 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:49.072 7fad3531e700 1 --2- 172.21.2.102:0/995750101 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fad300934f0 0x7fad300945b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:49.075 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:49.072 7fad3531e700 1 --2- 172.21.2.102:0/995750101 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fad140186c0 0x7fad1401abc0 unknown :-1 s=CLOSED pgs=1093 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:49.075 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:49.072 7fad3531e700 1 --2- 172.21.2.102:0/995750101 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fad300930f0 0x7fad30093f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:49.075 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:49.072 7fad3531e700 1 --2- 172.21.2.102:0/995750101 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fad300926d0 0x7fad30092ad0 unknown :-1 s=CLOSED pgs=2462 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:49.075 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:49.072 7fad3531e700 1 -- 172.21.2.102:0/995750101 >> 172.21.2.102:0/995750101 conn(0x7fad30092250 msgr2=0x7fad30086fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:49.075 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:49.073 7fad3531e700 1 -- 172.21.2.102:0/995750101 shutdown_connections 2021-02-10T11:33:49.075 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:49.073 7fad3531e700 1 -- 172.21.2.102:0/995750101 wait complete. 2021-02-10T11:33:49.082 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-02-10T11:33:49.514 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:49.496 7f44ab4c5700 1 Processor -- start 2021-02-10T11:33:49.514 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:49.496 7f44ab4c5700 1 -- start start 2021-02-10T11:33:49.514 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:49.497 7f44ab4c5700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f44a40d1870 0x7f44a41b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:49.515 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:49.497 7f44ab4c5700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f44a41b0ae0 0x7f44a4108940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:49.515 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:49.497 7f44ab4c5700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44a40fe1c0 con 0x7f44a40d1c70 2021-02-10T11:33:49.515 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:49.497 7f44ab4c5700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44a4109a60 con 0x7f44a40d1870 2021-02-10T11:33:49.515 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:49.497 7f44ab4c5700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44a4109750 con 0x7f44a41b0ae0 2021-02-10T11:33:49.515 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:49.497 7f44a92e1700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f44a40d1870 0x7f44a41b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:49.515 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:49.497 7f44a92e1700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f44a40d1870 0x7f44a41b04c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:35630/0 (socket says 172.21.2.102:35630) 2021-02-10T11:33:49.515 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:49.497 7f44a92e1700 1 -- 172.21.2.102:0/2253206734 learned_addr learned my addr 172.21.2.102:0/2253206734 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:33:49.515 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:49.497 7f44a9ae2700 1 --1- 172.21.2.102:0/2253206734 >> v1:172.21.2.102:6789/0 conn(0x7f44a40d1c70 0x7f44a4108f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:56086/0 (socket says 172.21.2.102:56086) 2021-02-10T11:33:49.515 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:49.497 7f44a92e1700 1 -- 172.21.2.102:0/2253206734 >> v2:172.21.2.113:3300/0 conn(0x7f44a41b0ae0 msgr2=0x7f44a4108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:33:49.515 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:49.497 7f44a92e1700 1 --2- 172.21.2.102:0/2253206734 >> v2:172.21.2.113:3300/0 conn(0x7f44a41b0ae0 0x7f44a4108940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:49.515 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:49.497 7f44a92e1700 1 -- 172.21.2.102:0/2253206734 >> v1:172.21.2.102:6789/0 conn(0x7f44a40d1c70 legacy=0x7f44a4108f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:49.515 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:49.497 7f44a92e1700 1 -- 172.21.2.102:0/2253206734 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44a4088070 con 0x7f44a40d1870 2021-02-10T11:33:49.516 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:49.497 7f44a92e1700 1 --2- 172.21.2.102:0/2253206734 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f44a40d1870 0x7f44a41b04c0 secure :-1 s=READY pgs=2464 cs=0 l=1 rev1=1 rx=0x7f4498006700 tx=0x7f4498009b80).ready entity=mon.0 client_cookie=acc5695ef5fed0df server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:49.516 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:49.498 7f4497fff700 1 -- 172.21.2.102:0/2253206734 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f449800c0c0 con 0x7f44a40d1870 2021-02-10T11:33:49.516 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:49.498 7f4497fff700 1 -- 172.21.2.102:0/2253206734 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f449800c340 con 0x7f44a40d1870 2021-02-10T11:33:49.516 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:49.498 7f4497fff700 1 -- 172.21.2.102:0/2253206734 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f449800c860 con 0x7f44a40d1870 2021-02-10T11:33:49.516 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:49.498 7f44ab4c5700 1 -- 172.21.2.102:0/2253206734 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f44a40d1870 msgr2=0x7f44a41b04c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:49.516 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:49.498 7f44ab4c5700 1 --2- 172.21.2.102:0/2253206734 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f44a40d1870 0x7f44a41b04c0 secure :-1 s=READY pgs=2464 cs=0 l=1 rev1=1 rx=0x7f4498006700 tx=0x7f4498009b80).stop 2021-02-10T11:33:49.516 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:49.498 7f44ab4c5700 1 -- 172.21.2.102:0/2253206734 shutdown_connections 2021-02-10T11:33:49.516 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:49.498 7f44ab4c5700 1 --2- 172.21.2.102:0/2253206734 >> v2:172.21.2.113:3300/0 conn(0x7f44a41b0ae0 0x7f44a4108940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:49.516 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:49.498 7f44ab4c5700 1 --2- 172.21.2.102:0/2253206734 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f44a40d1870 0x7f44a41b04c0 unknown :-1 s=CLOSED pgs=2464 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:49.516 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:49.498 7f44ab4c5700 1 -- 172.21.2.102:0/2253206734 >> 172.21.2.102:0/2253206734 conn(0x7f44a40ce420 msgr2=0x7f44a40d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:49.516 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:49.498 7f44ab4c5700 1 -- 172.21.2.102:0/2253206734 shutdown_connections 2021-02-10T11:33:49.516 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:49.498 7f44ab4c5700 1 -- 172.21.2.102:0/2253206734 wait complete. 2021-02-10T11:33:49.518 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:49.513 7f44ab4c5700 1 Processor -- start 2021-02-10T11:33:49.518 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:49.513 7f44ab4c5700 1 -- start start 2021-02-10T11:33:49.518 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:49.513 7f44ab4c5700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f44a40c2370 0x7f44a418f870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:49.518 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:49.513 7f44ab4c5700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f44a4084cb0 0x7f44a41a7470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:49.518 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:49.513 7f44ab4c5700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f44a41a7a90 0x7f44a4085700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:49.518 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:49.513 7f44ab4c5700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44a40be840 con 0x7f44a4084cb0 2021-02-10T11:33:49.518 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:49.513 7f44ab4c5700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44a40c29a0 con 0x7f44a40c2370 2021-02-10T11:33:49.518 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:49.513 7f44ab4c5700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44a41a7e90 con 0x7f44a41a7a90 2021-02-10T11:33:49.518 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:49.514 7f44a8ae0700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f44a4084cb0 0x7f44a41a7470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:49.518 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:49.514 7f44a8ae0700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f44a4084cb0 0x7f44a41a7470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:35634/0 (socket says 172.21.2.102:35634) 2021-02-10T11:33:49.519 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:49.514 7f44a8ae0700 1 -- 172.21.2.102:0/1942248612 learned_addr learned my addr 172.21.2.102:0/1942248612 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:33:49.519 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:49.514 7f44a8ae0700 1 -- 172.21.2.102:0/1942248612 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f44a41a7a90 msgr2=0x7f44a4085700 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:33:49.519 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:49.514 7f44a8ae0700 1 --2- 172.21.2.102:0/1942248612 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f44a41a7a90 0x7f44a4085700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:49.519 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:49.514 7f44a8ae0700 1 -- 172.21.2.102:0/1942248612 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f44a40c2370 msgr2=0x7f44a418f870 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:33:49.519 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:49.514 7f44a8ae0700 1 --2- 172.21.2.102:0/1942248612 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f44a40c2370 0x7f44a418f870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:49.519 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:49.514 7f44a8ae0700 1 -- 172.21.2.102:0/1942248612 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44a41a80b0 con 0x7f44a4084cb0 2021-02-10T11:33:49.519 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:49.514 7f44a8ae0700 1 --2- 172.21.2.102:0/1942248612 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f44a4084cb0 0x7f44a41a7470 secure :-1 s=READY pgs=2465 cs=0 l=1 rev1=1 rx=0x7f44a0006f00 tx=0x7f44a000b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:49.519 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:49.514 7f4495ffb700 1 -- 172.21.2.102:0/1942248612 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f44a000bd30 con 0x7f44a4084cb0 2021-02-10T11:33:49.519 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:49.514 7f4495ffb700 1 -- 172.21.2.102:0/1942248612 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44a0000c90 con 0x7f44a4084cb0 2021-02-10T11:33:49.519 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:49.514 7f4495ffb700 1 -- 172.21.2.102:0/1942248612 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f44a0002020 con 0x7f44a4084cb0 2021-02-10T11:33:49.519 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:49.514 7f44ab4c5700 1 -- 172.21.2.102:0/1942248612 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44a41a80b0 con 0x7f44a4084cb0 2021-02-10T11:33:49.519 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:49.514 7f44ab4c5700 1 -- 172.21.2.102:0/1942248612 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44a41a80b0 con 0x7f44a4084cb0 2021-02-10T11:33:49.520 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:49.515 7f4495ffb700 1 -- 172.21.2.102:0/1942248612 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f44a0005310 con 0x7f44a4084cb0 2021-02-10T11:33:49.520 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:49.515 7f4495ffb700 1 --2- 172.21.2.102:0/1942248612 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4484018730 0x7f448401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:49.520 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:49.515 7f4495ffb700 1 -- 172.21.2.102:0/1942248612 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f44a00327d0 con 0x7f44a4084cb0 2021-02-10T11:33:49.520 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:49.515 7f44a92e1700 1 --2- 172.21.2.102:0/1942248612 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4484018730 0x7f448401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:49.520 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:49.515 7f44a92e1700 1 --2- 172.21.2.102:0/1942248612 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4484018730 0x7f448401abc0 crc :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:49.548 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:49.547 7f44ab4c5700 1 -- 172.21.2.102:0/1942248612 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44a4082a70 con 0x7f44a4084cb0 2021-02-10T11:33:49.551 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:49.549 7f4495ffb700 1 -- 172.21.2.102:0/1942248612 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f44a0006350 con 0x7f44a4084cb0 2021-02-10T11:33:49.680 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:49.678 7f44ab4c5700 1 -- 172.21.2.102:0/1942248612 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7f44a4082a70 con 0x7f44a4084cb0 2021-02-10T11:33:49.680 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:49.678 7f4495ffb700 1 -- 172.21.2.102:0/1942248612 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v128) v1 ==== 100+0+10 (secure 0 0 0) 0x7f44a000bd30 con 0x7f44a4084cb0 2021-02-10T11:33:49.681 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:33:49.682 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:49.680 7f44ab4c5700 1 -- 172.21.2.102:0/1942248612 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4484018730 msgr2=0x7f448401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:49.682 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:49.680 7f44ab4c5700 1 --2- 172.21.2.102:0/1942248612 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4484018730 0x7f448401abc0 crc :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:49.682 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:49.680 7f44ab4c5700 1 -- 172.21.2.102:0/1942248612 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f44a4084cb0 msgr2=0x7f44a41a7470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:49.682 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:49.680 7f44ab4c5700 1 --2- 172.21.2.102:0/1942248612 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f44a4084cb0 0x7f44a41a7470 secure :-1 s=READY pgs=2465 cs=0 l=1 rev1=1 rx=0x7f44a0006f00 tx=0x7f44a000b940).stop 2021-02-10T11:33:49.683 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:49.681 7f44ab4c5700 1 -- 172.21.2.102:0/1942248612 shutdown_connections 2021-02-10T11:33:49.683 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:49.681 7f44ab4c5700 1 --2- 172.21.2.102:0/1942248612 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f44a41a7a90 0x7f44a4085700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:49.683 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:49.681 7f44ab4c5700 1 --2- 172.21.2.102:0/1942248612 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f44a4084cb0 0x7f44a41a7470 unknown :-1 s=CLOSED pgs=2465 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:49.683 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:49.681 7f44ab4c5700 1 --2- 172.21.2.102:0/1942248612 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4484018730 0x7f448401abc0 unknown :-1 s=CLOSED pgs=1094 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:49.683 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:49.681 7f44ab4c5700 1 --2- 172.21.2.102:0/1942248612 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f44a40c2370 0x7f44a418f870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:49.683 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:49.681 7f44ab4c5700 1 -- 172.21.2.102:0/1942248612 >> 172.21.2.102:0/1942248612 conn(0x7f44a40b5dc0 msgr2=0x7f44a40c2d90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:49.683 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:49.681 7f44ab4c5700 1 -- 172.21.2.102:0/1942248612 shutdown_connections 2021-02-10T11:33:49.683 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:49.681 7f44ab4c5700 1 -- 172.21.2.102:0/1942248612 wait complete. 2021-02-10T11:33:49.690 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-02-10T11:33:49.691 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:33:50.119 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:50.101 7f5632579700 1 Processor -- start 2021-02-10T11:33:50.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:50.101 7f5632579700 1 -- start start 2021-02-10T11:33:50.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:50.102 7f5632579700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f562c0d1870 0x7f562c1b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:50.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:50.102 7f5632579700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f562c1b0a10 0x7f562c0cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:50.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:50.102 7f5632579700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f562c0f0b10 con 0x7f562c0d1c70 2021-02-10T11:33:50.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:50.102 7f5632579700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f562c0fe230 con 0x7f562c1b0a10 2021-02-10T11:33:50.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:50.102 7f5632579700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f562c0e5ed0 con 0x7f562c0d1870 2021-02-10T11:33:50.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:50.102 7f562b7fe700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f562c1b0a10 0x7f562c0cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:50.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:50.102 7f562b7fe700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f562c1b0a10 0x7f562c0cf330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:35644/0 (socket says 172.21.2.102:35644) 2021-02-10T11:33:50.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:50.102 7f562b7fe700 1 -- 172.21.2.102:0/4111815900 learned_addr learned my addr 172.21.2.102:0/4111815900 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:33:50.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:50.102 7f562b7fe700 1 -- 172.21.2.102:0/4111815900 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f562c0d1870 msgr2=0x7f562c1b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:33:50.121 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:50.102 7f562b7fe700 1 --2- 172.21.2.102:0/4111815900 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f562c0d1870 0x7f562c1b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:50.121 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:50.102 7f562b7fe700 1 -- 172.21.2.102:0/4111815900 >> v1:172.21.2.102:6789/0 conn(0x7f562c0d1c70 legacy=0x7f562c0cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:50.121 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:50.103 7f562b7fe700 1 -- 172.21.2.102:0/4111815900 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f562c0f0650 con 0x7f562c1b0a10 2021-02-10T11:33:50.121 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:50.103 7f562affd700 1 -- 172.21.2.102:0/4111815900 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1044285584 0 0) 0x7f5624000f90 con 0x7f562c0d1c70 2021-02-10T11:33:50.121 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:50.103 7f562b7fe700 1 --2- 172.21.2.102:0/4111815900 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f562c1b0a10 0x7f562c0cf330 secure :-1 s=READY pgs=2467 cs=0 l=1 rev1=1 rx=0x7f5620006700 tx=0x7f5620009b00).ready entity=mon.0 client_cookie=d4772c4ce6219faf server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:50.121 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:50.103 7f562affd700 1 -- 172.21.2.102:0/4111815900 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f562000c0c0 con 0x7f562c1b0a10 2021-02-10T11:33:50.121 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:50.103 7f562affd700 1 -- 172.21.2.102:0/4111815900 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f562000c400 con 0x7f562c1b0a10 2021-02-10T11:33:50.121 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:50.103 7f562affd700 1 -- 172.21.2.102:0/4111815900 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f562000c860 con 0x7f562c1b0a10 2021-02-10T11:33:50.121 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:50.103 7f5632579700 1 -- 172.21.2.102:0/4111815900 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f562c1b0a10 msgr2=0x7f562c0cf330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:50.121 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:50.103 7f5632579700 1 --2- 172.21.2.102:0/4111815900 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f562c1b0a10 0x7f562c0cf330 secure :-1 s=READY pgs=2467 cs=0 l=1 rev1=1 rx=0x7f5620006700 tx=0x7f5620009b00).stop 2021-02-10T11:33:50.121 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:50.104 7f5632579700 1 -- 172.21.2.102:0/4111815900 shutdown_connections 2021-02-10T11:33:50.121 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:50.104 7f5632579700 1 --2- 172.21.2.102:0/4111815900 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f562c1b0a10 0x7f562c0cf330 unknown :-1 s=CLOSED pgs=2467 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:50.122 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:50.104 7f5632579700 1 --2- 172.21.2.102:0/4111815900 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f562c0d1870 0x7f562c1b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:50.122 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:50.104 7f5632579700 1 -- 172.21.2.102:0/4111815900 >> 172.21.2.102:0/4111815900 conn(0x7f562c0ce450 msgr2=0x7f562c0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:50.122 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:50.104 7f5632579700 1 -- 172.21.2.102:0/4111815900 shutdown_connections 2021-02-10T11:33:50.122 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:50.104 7f5632579700 1 -- 172.21.2.102:0/4111815900 wait complete. 2021-02-10T11:33:50.123 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:50.118 7f5632579700 1 Processor -- start 2021-02-10T11:33:50.123 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:50.118 7f5632579700 1 -- start start 2021-02-10T11:33:50.123 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:50.119 7f5632579700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f562c0b4f30 0x7f562c18b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:50.124 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:50.119 7f5632579700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f562c093f40 0x7f562c094340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:50.124 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:50.119 7f5632579700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f562c094960 0x7f562c187e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:50.124 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:50.119 7f5632579700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f562c094e90 con 0x7f562c0b4f30 2021-02-10T11:33:50.124 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:50.119 7f5632579700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f562c18bff0 con 0x7f562c094960 2021-02-10T11:33:50.124 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:50.119 7f5632579700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f562c08f780 con 0x7f562c093f40 2021-02-10T11:33:50.124 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:50.119 7f562bfff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f562c0b4f30 0x7f562c18b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:50.124 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:50.119 7f562bfff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f562c0b4f30 0x7f562c18b9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:35650/0 (socket says 172.21.2.102:35650) 2021-02-10T11:33:50.124 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:50.119 7f562bfff700 1 -- 172.21.2.102:0/3943800828 learned_addr learned my addr 172.21.2.102:0/3943800828 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:33:50.124 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:50.119 7f562bfff700 1 -- 172.21.2.102:0/3943800828 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f562c093f40 msgr2=0x7f562c094340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:50.124 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:50.119 7f562bfff700 1 --2- 172.21.2.102:0/3943800828 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f562c093f40 0x7f562c094340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:50.124 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:50.119 7f562bfff700 1 -- 172.21.2.102:0/3943800828 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f562c094960 msgr2=0x7f562c187e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:50.124 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:50.119 7f5630b96700 1 --2- 172.21.2.102:0/3943800828 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f562c094960 0x7f562c187e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:50.125 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:50.119 7f562bfff700 1 --2- 172.21.2.102:0/3943800828 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f562c094960 0x7f562c187e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:50.125 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:50.120 7f562bfff700 1 -- 172.21.2.102:0/3943800828 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f562c188470 con 0x7f562c0b4f30 2021-02-10T11:33:50.125 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:50.120 7f5630b96700 1 --2- 172.21.2.102:0/3943800828 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f562c094960 0x7f562c187e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:33:50.125 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:50.120 7f562bfff700 1 --2- 172.21.2.102:0/3943800828 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f562c0b4f30 0x7f562c18b9d0 secure :-1 s=READY pgs=2468 cs=0 l=1 rev1=1 rx=0x7f561c00b530 tx=0x7f561c00b8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:50.125 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:50.120 7f5628ff9700 1 -- 172.21.2.102:0/3943800828 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f561c0009b0 con 0x7f562c0b4f30 2021-02-10T11:33:50.125 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:50.120 7f5628ff9700 1 -- 172.21.2.102:0/3943800828 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f561c0063d0 con 0x7f562c0b4f30 2021-02-10T11:33:50.125 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:50.120 7f5628ff9700 1 -- 172.21.2.102:0/3943800828 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f561c000c90 con 0x7f562c0b4f30 2021-02-10T11:33:50.125 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:50.120 7f5632579700 1 -- 172.21.2.102:0/3943800828 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f562c188470 con 0x7f562c0b4f30 2021-02-10T11:33:50.125 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:50.120 7f5632579700 1 -- 172.21.2.102:0/3943800828 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f562c188a00 con 0x7f562c0b4f30 2021-02-10T11:33:50.125 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:50.121 7f5628ff9700 1 -- 172.21.2.102:0/3943800828 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f561c007000 con 0x7f562c0b4f30 2021-02-10T11:33:50.125 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:50.121 7f5628ff9700 1 --2- 172.21.2.102:0/3943800828 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f560c0186c0 0x7f560c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:50.125 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:50.121 7f5628ff9700 1 -- 172.21.2.102:0/3943800828 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f561c032400 con 0x7f562c0b4f30 2021-02-10T11:33:50.125 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:50.121 7f562b7fe700 1 --2- 172.21.2.102:0/3943800828 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f560c0186c0 0x7f560c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:50.125 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:50.121 7f562b7fe700 1 --2- 172.21.2.102:0/3943800828 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f560c0186c0 0x7f560c01abc0 crc :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:50.153 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:50.152 7f5632579700 1 -- 172.21.2.102:0/3943800828 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f562c0ce450 con 0x7f562c0b4f30 2021-02-10T11:33:50.156 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:50.154 7f5628ff9700 1 -- 172.21.2.102:0/3943800828 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f561c000960 con 0x7f562c0b4f30 2021-02-10T11:33:50.282 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:50.280 7f5632579700 1 -- 172.21.2.102:0/3943800828 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f562c0ce450 con 0x7f562c0b4f30 2021-02-10T11:33:50.283 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:50.281 7f5628ff9700 1 -- 172.21.2.102:0/3943800828 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+2102 (secure 0 0 0) 0x7f561c007000 con 0x7f562c0b4f30 2021-02-10T11:33:50.283 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:33:50.283 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":107,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":11103,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.102:6835/160551241","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6834","nonce":160551241},{"type":"v1","addr":"172.21.2.102:6835","nonce":160551241}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":103}],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:33:32.844442","modified":"2021-02-10 11:33:34.850935","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7882},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7882":{"gid":7882,"name":"b","rank":0,"incarnation":104,"state":"up:active","state_seq":3,"addr":"172.21.2.113:6833/121050001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6832","nonce":121050001},{"type":"v1","addr":"172.21.2.113:6833","nonce":121050001}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[27],"metadata_pool":26,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":13}]} 2021-02-10T11:33:50.283 INFO:teuthology.orchestra.run.gibba002.stderr:dumped fsmap epoch 107 2021-02-10T11:33:50.284 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:50.282 7f5632579700 1 -- 172.21.2.102:0/3943800828 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f560c0186c0 msgr2=0x7f560c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:50.284 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:50.283 7f5632579700 1 --2- 172.21.2.102:0/3943800828 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f560c0186c0 0x7f560c01abc0 crc :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:50.284 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:50.283 7f5632579700 1 -- 172.21.2.102:0/3943800828 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f562c0b4f30 msgr2=0x7f562c18b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:50.284 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:50.283 7f5632579700 1 --2- 172.21.2.102:0/3943800828 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f562c0b4f30 0x7f562c18b9d0 secure :-1 s=READY pgs=2468 cs=0 l=1 rev1=1 rx=0x7f561c00b530 tx=0x7f561c00b8d0).stop 2021-02-10T11:33:50.285 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:50.283 7f5632579700 1 -- 172.21.2.102:0/3943800828 shutdown_connections 2021-02-10T11:33:50.285 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:50.283 7f5632579700 1 --2- 172.21.2.102:0/3943800828 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f560c0186c0 0x7f560c01abc0 unknown :-1 s=CLOSED pgs=1095 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:50.285 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:50.283 7f5632579700 1 --2- 172.21.2.102:0/3943800828 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f562c094960 0x7f562c187e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:50.285 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:50.283 7f5632579700 1 --2- 172.21.2.102:0/3943800828 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f562c093f40 0x7f562c094340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:50.285 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:50.283 7f5632579700 1 --2- 172.21.2.102:0/3943800828 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f562c0b4f30 0x7f562c18b9d0 unknown :-1 s=CLOSED pgs=2468 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:50.285 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:50.283 7f5632579700 1 -- 172.21.2.102:0/3943800828 >> 172.21.2.102:0/3943800828 conn(0x7f562c19f8f0 msgr2=0x7f562c1a5f80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:50.285 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:50.283 7f5632579700 1 -- 172.21.2.102:0/3943800828 shutdown_connections 2021-02-10T11:33:50.285 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:50.283 7f5632579700 1 -- 172.21.2.102:0/3943800828 wait complete. 2021-02-10T11:33:50.495 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 107, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:33:32.844442', 'modified': '2021-02-10 11:33:34.850935', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7882}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7882': {'gid': 7882, 'name': 'b', 'rank': 0, 'incarnation': 104, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.113:6833/121050001', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.113:6832', 'nonce': 121050001}, {'type': 'v1', 'addr': '172.21.2.113:6833', 'nonce': 121050001}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [27], 'metadata_pool': 26, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:33:50.495 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:33:50.495 DEBUG:teuthology.orchestra.run.gibba013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:33:50.894 INFO:teuthology.orchestra.run.gibba013.stdout:{ 2021-02-10T11:33:50.894 INFO:teuthology.orchestra.run.gibba013.stdout: "cluster_fsid": "68db30ea-dbce-4aef-a6df-f01c6fea0513", 2021-02-10T11:33:50.894 INFO:teuthology.orchestra.run.gibba013.stdout: "whoami": 0, 2021-02-10T11:33:50.894 INFO:teuthology.orchestra.run.gibba013.stdout: "id": 7882, 2021-02-10T11:33:50.895 INFO:teuthology.orchestra.run.gibba013.stdout: "want_state": "up:active", 2021-02-10T11:33:50.895 INFO:teuthology.orchestra.run.gibba013.stdout: "state": "up:active", 2021-02-10T11:33:50.895 INFO:teuthology.orchestra.run.gibba013.stdout: "rank_uptime": 18.046284892999999, 2021-02-10T11:33:50.895 INFO:teuthology.orchestra.run.gibba013.stdout: "mdsmap_epoch": 107, 2021-02-10T11:33:50.895 INFO:teuthology.orchestra.run.gibba013.stdout: "osdmap_epoch": 128, 2021-02-10T11:33:50.895 INFO:teuthology.orchestra.run.gibba013.stdout: "osdmap_epoch_barrier": 128, 2021-02-10T11:33:50.895 INFO:teuthology.orchestra.run.gibba013.stdout: "uptime": 23.656523729 2021-02-10T11:33:50.895 INFO:teuthology.orchestra.run.gibba013.stdout:} 2021-02-10T11:33:50.898 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "68db30ea-dbce-4aef-a6df-f01c6fea0513", "whoami": 0, "id": 7882, "want_state": "up:active", "state": "up:active", "rank_uptime": 18.046284892999999, "mdsmap_epoch": 107, "osdmap_epoch": 128, "osdmap_epoch_barrier": 128, "uptime": 23.656523729 } 2021-02-10T11:33:50.898 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2021-02-10T11:33:50.899 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2021-02-10T11:33:50.899 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.1 at ubuntu@gibba002.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.1... 2021-02-10T11:33:50.899 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:33:50.899 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec mkdir -p /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:33:51.237 DEBUG:teuthology.orchestra.run.gibba002:> sudo modprobe fuse 2021-02-10T11:33:51.581 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:33:51.582 DEBUG:teuthology.orchestra.run.gibba002:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:33:51.924 INFO:teuthology.orchestra.run.gibba002.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:33:51.925 INFO:teuthology.orchestra.run.gibba002.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:33:51.925 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:33:51.926 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:33:51.926 DEBUG:teuthology.orchestra.run.gibba002:> ls /sys/fs/fuse/connections 2021-02-10T11:33:52.260 INFO:teuthology.orchestra.run.gibba002.stdout:43 2021-02-10T11:33:52.261 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [43] 2021-02-10T11:33:52.261 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.1 /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:33:52.264 DEBUG:teuthology.orchestra.run.gibba002:> sudo modprobe fuse 2021-02-10T11:33:52.607 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:33:52.608 DEBUG:teuthology.orchestra.run.gibba002:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:33:52.673 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba002.stderr:2021-02-10 11:33:52.672 7f85caf89f80 -1 init, newargv = 0x55b40fab2d90 newargc=9 2021-02-10T11:33:52.674 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba002.stderr:ceph-fuse[190615]: starting ceph client 2021-02-10T11:33:52.679 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba002.stderr:ceph-fuse[190615]: starting fuse 2021-02-10T11:33:52.979 INFO:teuthology.orchestra.run.gibba002.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:33:52.979 INFO:teuthology.orchestra.run.gibba002.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:33:52.980 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:33:52.980 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:33:52.981 DEBUG:teuthology.orchestra.run.gibba002:> ls /sys/fs/fuse/connections 2021-02-10T11:33:53.355 INFO:teuthology.orchestra.run.gibba002.stdout:43 2021-02-10T11:33:53.355 INFO:teuthology.orchestra.run.gibba002.stdout:44 2021-02-10T11:33:53.355 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43, 44] 2021-02-10T11:33:53.355 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:33:53.356 DEBUG:teuthology.orchestra.run.gibba002:> sudo python3 -c ' 2021-02-10T11:33:53.356 DEBUG:teuthology.orchestra.run.gibba002:> import glob 2021-02-10T11:33:53.356 DEBUG:teuthology.orchestra.run.gibba002:> import re 2021-02-10T11:33:53.356 DEBUG:teuthology.orchestra.run.gibba002:> import os 2021-02-10T11:33:53.356 DEBUG:teuthology.orchestra.run.gibba002:> import subprocess 2021-02-10T11:33:53.356 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:33:53.356 DEBUG:teuthology.orchestra.run.gibba002:> def find_socket(client_name): 2021-02-10T11:33:53.356 DEBUG:teuthology.orchestra.run.gibba002:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2021-02-10T11:33:53.356 DEBUG:teuthology.orchestra.run.gibba002:> files = glob.glob(asok_path) 2021-02-10T11:33:53.356 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:33:53.356 DEBUG:teuthology.orchestra.run.gibba002:> # Given a non-glob path, it better be there 2021-02-10T11:33:53.356 DEBUG:teuthology.orchestra.run.gibba002:> if "*" not in asok_path: 2021-02-10T11:33:53.356 DEBUG:teuthology.orchestra.run.gibba002:> assert(len(files) == 1) 2021-02-10T11:33:53.356 DEBUG:teuthology.orchestra.run.gibba002:> return files[0] 2021-02-10T11:33:53.356 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:33:53.357 DEBUG:teuthology.orchestra.run.gibba002:> for f in files: 2021-02-10T11:33:53.357 DEBUG:teuthology.orchestra.run.gibba002:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2021-02-10T11:33:53.357 DEBUG:teuthology.orchestra.run.gibba002:> if os.path.exists("/proc/{0}".format(pid)): 2021-02-10T11:33:53.357 DEBUG:teuthology.orchestra.run.gibba002:> return f 2021-02-10T11:33:53.357 DEBUG:teuthology.orchestra.run.gibba002:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2021-02-10T11:33:53.357 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:33:53.357 DEBUG:teuthology.orchestra.run.gibba002:> print(find_socket("client.1")) 2021-02-10T11:33:53.357 DEBUG:teuthology.orchestra.run.gibba002:> ' 2021-02-10T11:33:53.720 INFO:teuthology.orchestra.run.gibba002.stdout:/var/run/ceph/ceph-client.1.190615.asok 2021-02-10T11:33:53.723 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.190615.asok 2021-02-10T11:33:53.723 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:33:53.723 DEBUG:teuthology.orchestra.run.gibba002:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.190615.asok status 2021-02-10T11:33:54.114 INFO:teuthology.orchestra.run.gibba002.stdout:{ 2021-02-10T11:33:54.114 INFO:teuthology.orchestra.run.gibba002.stdout: "metadata": { 2021-02-10T11:33:54.114 INFO:teuthology.orchestra.run.gibba002.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:33:54.114 INFO:teuthology.orchestra.run.gibba002.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:33:54.114 INFO:teuthology.orchestra.run.gibba002.stdout: "entity_id": "1", 2021-02-10T11:33:54.114 INFO:teuthology.orchestra.run.gibba002.stdout: "hostname": "gibba002", 2021-02-10T11:33:54.114 INFO:teuthology.orchestra.run.gibba002.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2021-02-10T11:33:54.114 INFO:teuthology.orchestra.run.gibba002.stdout: "pid": "190615", 2021-02-10T11:33:54.115 INFO:teuthology.orchestra.run.gibba002.stdout: "root": "/" 2021-02-10T11:33:54.115 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:33:54.115 INFO:teuthology.orchestra.run.gibba002.stdout: "dentry_count": 0, 2021-02-10T11:33:54.115 INFO:teuthology.orchestra.run.gibba002.stdout: "dentry_pinned_count": 0, 2021-02-10T11:33:54.115 INFO:teuthology.orchestra.run.gibba002.stdout: "id": 11319, 2021-02-10T11:33:54.115 INFO:teuthology.orchestra.run.gibba002.stdout: "inst": { 2021-02-10T11:33:54.115 INFO:teuthology.orchestra.run.gibba002.stdout: "name": { 2021-02-10T11:33:54.115 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "client", 2021-02-10T11:33:54.115 INFO:teuthology.orchestra.run.gibba002.stdout: "num": 11319 2021-02-10T11:33:54.115 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:33:54.115 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": { 2021-02-10T11:33:54.115 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:33:54.115 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:0", 2021-02-10T11:33:54.116 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 3950549633 2021-02-10T11:33:54.116 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:33:54.116 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:33:54.116 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": { 2021-02-10T11:33:54.116 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:33:54.116 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:0", 2021-02-10T11:33:54.116 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 3950549633 2021-02-10T11:33:54.116 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:33:54.116 INFO:teuthology.orchestra.run.gibba002.stdout: "inst_str": "client.11319 172.21.2.102:0/3950549633", 2021-02-10T11:33:54.116 INFO:teuthology.orchestra.run.gibba002.stdout: "addr_str": "172.21.2.102:0/3950549633", 2021-02-10T11:33:54.116 INFO:teuthology.orchestra.run.gibba002.stdout: "inode_count": 1, 2021-02-10T11:33:54.116 INFO:teuthology.orchestra.run.gibba002.stdout: "mds_epoch": 107, 2021-02-10T11:33:54.116 INFO:teuthology.orchestra.run.gibba002.stdout: "osd_epoch": 128, 2021-02-10T11:33:54.116 INFO:teuthology.orchestra.run.gibba002.stdout: "osd_epoch_barrier": 0, 2021-02-10T11:33:54.117 INFO:teuthology.orchestra.run.gibba002.stdout: "blacklisted": false 2021-02-10T11:33:54.117 INFO:teuthology.orchestra.run.gibba002.stdout:} 2021-02-10T11:33:54.118 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:33:54.118 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:33:54.118 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:33:54.455 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:33:54.455 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2021-02-10T11:33:54.455 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:33:54.455 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:33:54.802 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2021-02-10T11:33:55.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:55.217 7f5024cac700 1 Processor -- start 2021-02-10T11:33:55.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:55.217 7f5024cac700 1 -- start start 2021-02-10T11:33:55.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:55.217 7f5024cac700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f502010aff0 0x7f50200cf750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:55.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:55.217 7f5024cac700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50200fe1c0 con 0x7f50200d1c70 2021-02-10T11:33:55.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:55.217 7f5024cac700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50201b0cb0 con 0x7f50200d1870 2021-02-10T11:33:55.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:55.217 7f5024cac700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5020109a60 con 0x7f502010aff0 2021-02-10T11:33:55.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:55.217 7f501e05d700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f50200d1c70 0x7f50200cef60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:56128/0 (socket says 172.21.2.102:56128) 2021-02-10T11:33:55.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:55.217 7f501e05d700 1 -- 172.21.2.102:0/2318311615 learned_addr learned my addr 172.21.2.102:0/2318311615 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:33:55.238 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:55.218 7f501d85c700 1 -- 172.21.2.102:0/2318311615 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1396526427 0 0) 0x7f50080010b0 con 0x7f50200d1c70 2021-02-10T11:33:55.238 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:55.218 7f501d85c700 1 -- 172.21.2.102:0/2318311615 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f500c002fe0 con 0x7f50200d1c70 2021-02-10T11:33:55.238 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:55.218 7f501d85c700 1 -- 172.21.2.102:0/2318311615 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2417904571 0 0) 0x7f50080010b0 con 0x7f50200d1c70 2021-02-10T11:33:55.238 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:55.218 7f501d85c700 1 -- 172.21.2.102:0/2318311615 >> v2:172.21.2.113:3300/0 conn(0x7f502010aff0 msgr2=0x7f50200cf750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:33:55.238 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:55.218 7f501d85c700 1 --2- 172.21.2.102:0/2318311615 >> v2:172.21.2.113:3300/0 conn(0x7f502010aff0 0x7f50200cf750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:55.238 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:55.218 7f501d85c700 1 -- 172.21.2.102:0/2318311615 >> v1:172.21.2.113:6789/0 conn(0x7f50200d1870 legacy=0x7f50201b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:33:55.238 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:55.218 7f501d85c700 1 -- 172.21.2.102:0/2318311615 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f502010b3f0 con 0x7f50200d1c70 2021-02-10T11:33:55.239 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:55.218 7f501d85c700 1 -- 172.21.2.102:0/2318311615 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7f50080015b0 con 0x7f50200d1c70 2021-02-10T11:33:55.239 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:55.218 7f501d85c700 1 -- 172.21.2.102:0/2318311615 >> v1:172.21.2.102:6789/0 conn(0x7f50200d1c70 legacy=0x7f50200cef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:55.239 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:55.218 7f501d85c700 1 --2- 172.21.2.102:0/2318311615 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f500c003b10 0x7f500c006010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:55.239 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:55.218 7f501d85c700 1 --2- 172.21.2.102:0/2318311615 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f500c006830 0x7f500c008d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:55.239 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:55.218 7f501d85c700 1 --2- 172.21.2.102:0/2318311615 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f500c0095c0 0x7f500c00bae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:55.239 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:55.218 7f501d85c700 1 -- 172.21.2.102:0/2318311615 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f500c00c310 con 0x7f500c003b10 2021-02-10T11:33:55.240 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:55.218 7f501d85c700 1 -- 172.21.2.102:0/2318311615 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f500c00c530 con 0x7f500c006830 2021-02-10T11:33:55.240 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:55.218 7f501d85c700 1 -- 172.21.2.102:0/2318311615 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f500c00c750 con 0x7f500c0095c0 2021-02-10T11:33:55.240 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:55.218 7f501e85e700 1 --2- 172.21.2.102:0/2318311615 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f500c003b10 0x7f500c006010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:55.240 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:55.218 7f501e05d700 1 --2- 172.21.2.102:0/2318311615 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f500c006830 0x7f500c008d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:55.240 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:55.218 7f501f05f700 1 --2- 172.21.2.102:0/2318311615 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f500c0095c0 0x7f500c00bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:55.240 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:55.219 7f501e85e700 1 -- 172.21.2.102:0/2318311615 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f500c0095c0 msgr2=0x7f500c00bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:33:55.241 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:55.219 7f501e85e700 1 --2- 172.21.2.102:0/2318311615 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f500c0095c0 0x7f500c00bae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:55.241 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:55.219 7f501e85e700 1 -- 172.21.2.102:0/2318311615 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f500c006830 msgr2=0x7f500c008d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:33:55.241 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:55.219 7f501e85e700 1 --2- 172.21.2.102:0/2318311615 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f500c006830 0x7f500c008d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:55.241 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:55.219 7f501f05f700 1 -- 172.21.2.102:0/2318311615 reap_dead start 2021-02-10T11:33:55.241 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:55.219 7f501e85e700 1 -- 172.21.2.102:0/2318311615 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f500c00c9f0 con 0x7f500c003b10 2021-02-10T11:33:55.241 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:55.219 7f501e85e700 1 --2- 172.21.2.102:0/2318311615 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f500c003b10 0x7f500c006010 secure :-1 s=READY pgs=2472 cs=0 l=1 rev1=1 rx=0x7f5010000e30 tx=0x7f5010009a40).ready entity=mon.0 client_cookie=dd15182b337ab257 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:55.242 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:55.219 7f501d85c700 1 -- 172.21.2.102:0/2318311615 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5010006700 con 0x7f500c003b10 2021-02-10T11:33:55.242 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:55.219 7f501d85c700 1 -- 172.21.2.102:0/2318311615 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f501000c0c0 con 0x7f500c003b10 2021-02-10T11:33:55.242 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:55.219 7f501d85c700 1 -- 172.21.2.102:0/2318311615 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f501000c560 con 0x7f500c003b10 2021-02-10T11:33:55.242 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:55.219 7f5024cac700 1 -- 172.21.2.102:0/2318311615 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f500c003b10 msgr2=0x7f500c006010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:55.242 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:55.219 7f5024cac700 1 --2- 172.21.2.102:0/2318311615 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f500c003b10 0x7f500c006010 secure :-1 s=READY pgs=2472 cs=0 l=1 rev1=1 rx=0x7f5010000e30 tx=0x7f5010009a40).stop 2021-02-10T11:33:55.243 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:55.220 7f5024cac700 1 -- 172.21.2.102:0/2318311615 shutdown_connections 2021-02-10T11:33:55.243 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:55.220 7f5024cac700 1 --2- 172.21.2.102:0/2318311615 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f500c003b10 0x7f500c006010 unknown :-1 s=CLOSED pgs=2472 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:55.243 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:55.220 7f5024cac700 1 -- 172.21.2.102:0/2318311615 >> 172.21.2.102:0/2318311615 conn(0x7f50200ce420 msgr2=0x7f50200d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:55.243 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:55.220 7f5024cac700 1 -- 172.21.2.102:0/2318311615 shutdown_connections 2021-02-10T11:33:55.243 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:55.220 7f5024cac700 1 -- 172.21.2.102:0/2318311615 wait complete. 2021-02-10T11:33:55.243 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:55.235 7f5024cac700 1 Processor -- start 2021-02-10T11:33:55.243 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:55.235 7f5024cac700 1 -- start start 2021-02-10T11:33:55.246 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:55.236 7f5024cac700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f50201a05f0 0x7f50200a8f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:55.246 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:55.236 7f5024cac700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f50200aba50 0x7f50200b3090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:55.246 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:55.236 7f5024cac700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f50200b36b0 0x7f50200b42a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:55.246 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:55.236 7f5024cac700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50200b3d20 con 0x7f50200aba50 2021-02-10T11:33:55.246 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:55.236 7f5024cac700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50200abe50 con 0x7f50200b36b0 2021-02-10T11:33:55.246 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:55.236 7f5024cac700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50200b23c0 con 0x7f50201a05f0 2021-02-10T11:33:55.246 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:55.236 7f501e05d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f50200aba50 0x7f50200b3090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:55.246 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:55.236 7f501e05d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f50200aba50 0x7f50200b3090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:35682/0 (socket says 172.21.2.102:35682) 2021-02-10T11:33:55.247 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:55.236 7f501e05d700 1 -- 172.21.2.102:0/2698145857 learned_addr learned my addr 172.21.2.102:0/2698145857 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:33:55.247 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:55.236 7f501f05f700 1 --2- 172.21.2.102:0/2698145857 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f50200b36b0 0x7f50200b42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:55.247 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:55.236 7f501e85e700 1 --2- 172.21.2.102:0/2698145857 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f50201a05f0 0x7f50200a8f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:55.247 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:55.236 7f501e05d700 1 -- 172.21.2.102:0/2698145857 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f50201a05f0 msgr2=0x7f50200a8f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:55.247 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:55.236 7f501e05d700 1 --2- 172.21.2.102:0/2698145857 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f50201a05f0 0x7f50200a8f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:55.247 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:55.236 7f501e05d700 1 -- 172.21.2.102:0/2698145857 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f50200b36b0 msgr2=0x7f50200b42a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:55.247 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:55.236 7f501e05d700 1 --2- 172.21.2.102:0/2698145857 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f50200b36b0 0x7f50200b42a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:55.247 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:55.236 7f501e05d700 1 -- 172.21.2.102:0/2698145857 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50200b48c0 con 0x7f50200aba50 2021-02-10T11:33:55.247 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:55.236 7f501e85e700 1 --2- 172.21.2.102:0/2698145857 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f50201a05f0 0x7f50200a8f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:33:55.247 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:55.237 7f501e05d700 1 --2- 172.21.2.102:0/2698145857 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f50200aba50 0x7f50200b3090 secure :-1 s=READY pgs=2473 cs=0 l=1 rev1=1 rx=0x7f5008001b30 tx=0x7f500804f950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:55.247 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:55.237 7f50177fe700 1 -- 172.21.2.102:0/2698145857 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f50080009b0 con 0x7f50200aba50 2021-02-10T11:33:55.247 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:55.237 7f50177fe700 1 -- 172.21.2.102:0/2698145857 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f500804fd40 con 0x7f50200aba50 2021-02-10T11:33:55.247 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:55.237 7f50177fe700 1 -- 172.21.2.102:0/2698145857 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f50080045d0 con 0x7f50200aba50 2021-02-10T11:33:55.247 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:55.237 7f5024cac700 1 -- 172.21.2.102:0/2698145857 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50200b48c0 con 0x7f50200aba50 2021-02-10T11:33:55.248 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:55.237 7f5024cac700 1 -- 172.21.2.102:0/2698145857 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50200b48c0 con 0x7f50200aba50 2021-02-10T11:33:55.248 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:55.237 7f50177fe700 1 -- 172.21.2.102:0/2698145857 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f5008004350 con 0x7f50200aba50 2021-02-10T11:33:55.248 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:55.238 7f50177fe700 1 --2- 172.21.2.102:0/2698145857 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f50040186c0 0x7f500401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:55.248 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:55.238 7f50177fe700 1 -- 172.21.2.102:0/2698145857 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f5008076400 con 0x7f50200aba50 2021-02-10T11:33:55.248 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:55.238 7f501e85e700 1 --2- 172.21.2.102:0/2698145857 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f50040186c0 0x7f500401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:55.248 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:55.238 7f501e85e700 1 --2- 172.21.2.102:0/2698145857 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f50040186c0 0x7f500401abc0 crc :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:55.269 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:55.267 7f5024cac700 1 -- 172.21.2.102:0/2698145857 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50200825d0 con 0x7f50200aba50 2021-02-10T11:33:55.271 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:55.270 7f50177fe700 1 -- 172.21.2.102:0/2698145857 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f5008004140 con 0x7f50200aba50 2021-02-10T11:33:55.409 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:55.408 7f5024cac700 1 -- 172.21.2.102:0/2698145857 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f50200825d0 con 0x7f50200aba50 2021-02-10T11:33:55.464 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:55.462 7f50177fe700 1 -- 172.21.2.102:0/2698145857 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v108) v1 ==== 136+0+0 (secure 0 0 0) 0x7f5008004350 con 0x7f50200aba50 2021-02-10T11:33:55.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:55.474 7f5024cac700 1 -- 172.21.2.102:0/2698145857 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f50040186c0 msgr2=0x7f500401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:55.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:55.474 7f5024cac700 1 --2- 172.21.2.102:0/2698145857 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f50040186c0 0x7f500401abc0 crc :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:55.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:55.474 7f5024cac700 1 -- 172.21.2.102:0/2698145857 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f50200aba50 msgr2=0x7f50200b3090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:55.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:55.474 7f5024cac700 1 --2- 172.21.2.102:0/2698145857 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f50200aba50 0x7f50200b3090 secure :-1 s=READY pgs=2473 cs=0 l=1 rev1=1 rx=0x7f5008001b30 tx=0x7f500804f950).stop 2021-02-10T11:33:55.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:55.474 7f5024cac700 1 -- 172.21.2.102:0/2698145857 shutdown_connections 2021-02-10T11:33:55.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:55.474 7f5024cac700 1 --2- 172.21.2.102:0/2698145857 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f50040186c0 0x7f500401abc0 unknown :-1 s=CLOSED pgs=1096 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:55.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:55.474 7f5024cac700 1 --2- 172.21.2.102:0/2698145857 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f50200b36b0 0x7f50200b42a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:55.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:55.474 7f5024cac700 1 --2- 172.21.2.102:0/2698145857 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f50200aba50 0x7f50200b3090 unknown :-1 s=CLOSED pgs=2473 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:55.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:55.474 7f5024cac700 1 --2- 172.21.2.102:0/2698145857 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f50201a05f0 0x7f50200a8f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:55.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:55.474 7f5024cac700 1 -- 172.21.2.102:0/2698145857 >> 172.21.2.102:0/2698145857 conn(0x7f50200bc070 msgr2=0x7f50200ba770 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:55.477 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:55.474 7f5024cac700 1 -- 172.21.2.102:0/2698145857 shutdown_connections 2021-02-10T11:33:55.477 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:55.474 7f5024cac700 1 -- 172.21.2.102:0/2698145857 wait complete. 2021-02-10T11:33:55.484 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs volume ls 2021-02-10T11:33:55.910 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:55.892 7f6c738ea700 1 Processor -- start 2021-02-10T11:33:55.910 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:55.893 7f6c738ea700 1 -- start start 2021-02-10T11:33:55.910 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:55.893 7f6c738ea700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f6c6c0d0340 0x7f6c6c0ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:55.911 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:55.893 7f6c738ea700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f6c6c0d1c70 0x7f6c6c0efc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:55.911 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:55.893 7f6c738ea700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c6c0f0b10 con 0x7f6c6c0d1870 2021-02-10T11:33:55.911 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:55.893 7f6c738ea700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c6c0e5ed0 con 0x7f6c6c0d1c70 2021-02-10T11:33:55.911 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:55.893 7f6c738ea700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c6c1b0be0 con 0x7f6c6c0d0340 2021-02-10T11:33:55.911 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:55.893 7f6c71706700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f6c6c0d1870 0x7f6c6c1b03f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:56146/0 (socket says 172.21.2.102:56146) 2021-02-10T11:33:55.911 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:55.893 7f6c71706700 1 -- 172.21.2.102:0/792110782 learned_addr learned my addr 172.21.2.102:0/792110782 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:33:55.911 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:55.893 7f6c71f07700 1 --2- 172.21.2.102:0/792110782 >> v2:172.21.2.102:3300/0 conn(0x7f6c6c0d1c70 0x7f6c6c0efc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:55.911 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:55.894 7f6c71f07700 1 -- 172.21.2.102:0/792110782 >> v2:172.21.2.113:3300/0 conn(0x7f6c6c0d0340 msgr2=0x7f6c6c0ef640 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:33:55.911 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:55.894 7f6c71f07700 1 --2- 172.21.2.102:0/792110782 >> v2:172.21.2.113:3300/0 conn(0x7f6c6c0d0340 0x7f6c6c0ef640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:55.911 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:55.894 7f6c71f07700 1 -- 172.21.2.102:0/792110782 >> v1:172.21.2.102:6789/0 conn(0x7f6c6c0d1870 legacy=0x7f6c6c1b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:55.911 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:55.894 7f6c71f07700 1 -- 172.21.2.102:0/792110782 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c6c0f20a0 con 0x7f6c6c0d1c70 2021-02-10T11:33:55.911 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:55.894 7f6c71f07700 1 --2- 172.21.2.102:0/792110782 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6c6c0d1c70 0x7f6c6c0efc60 secure :-1 s=READY pgs=2475 cs=0 l=1 rev1=1 rx=0x7f6c64006e30 tx=0x7f6c64009e60).ready entity=mon.0 client_cookie=b82d5aad7285c8c3 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:55.911 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:55.894 7f6c5ffff700 1 -- 172.21.2.102:0/792110782 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6c64006a50 con 0x7f6c6c0d1c70 2021-02-10T11:33:55.912 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:55.894 7f6c5ffff700 1 -- 172.21.2.102:0/792110782 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c6400b300 con 0x7f6c6c0d1c70 2021-02-10T11:33:55.912 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:55.894 7f6c5ffff700 1 -- 172.21.2.102:0/792110782 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6c6400b820 con 0x7f6c6c0d1c70 2021-02-10T11:33:55.912 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:55.894 7f6c738ea700 1 -- 172.21.2.102:0/792110782 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6c6c0d1c70 msgr2=0x7f6c6c0efc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:55.912 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:55.894 7f6c738ea700 1 --2- 172.21.2.102:0/792110782 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6c6c0d1c70 0x7f6c6c0efc60 secure :-1 s=READY pgs=2475 cs=0 l=1 rev1=1 rx=0x7f6c64006e30 tx=0x7f6c64009e60).stop 2021-02-10T11:33:55.912 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:55.894 7f6c738ea700 1 -- 172.21.2.102:0/792110782 shutdown_connections 2021-02-10T11:33:55.912 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:55.894 7f6c738ea700 1 --2- 172.21.2.102:0/792110782 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6c6c0d1c70 0x7f6c6c0efc60 unknown :-1 s=CLOSED pgs=2475 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:55.912 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:55.894 7f6c738ea700 1 --2- 172.21.2.102:0/792110782 >> v2:172.21.2.113:3300/0 conn(0x7f6c6c0d0340 0x7f6c6c0ef640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:55.912 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:55.894 7f6c738ea700 1 -- 172.21.2.102:0/792110782 >> 172.21.2.102:0/792110782 conn(0x7f6c6c0ce450 msgr2=0x7f6c6c0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:55.912 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:55.895 7f6c738ea700 1 -- 172.21.2.102:0/792110782 shutdown_connections 2021-02-10T11:33:55.912 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:55.895 7f6c738ea700 1 -- 172.21.2.102:0/792110782 wait complete. 2021-02-10T11:33:55.914 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:55.909 7f6c738ea700 1 Processor -- start 2021-02-10T11:33:55.914 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:55.909 7f6c738ea700 1 -- start start 2021-02-10T11:33:55.914 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:55.909 7f6c738ea700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6c6c0b4840 0x7f6c6c093890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:55.914 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:55.909 7f6c738ea700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6c6c093eb0 0x7f6c6c188490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:55.914 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:55.909 7f6c738ea700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6c6c0942b0 0x7f6c6c188ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:55.914 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:55.909 7f6c738ea700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c6c0b4c40 con 0x7f6c6c093eb0 2021-02-10T11:33:55.914 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:55.909 7f6c738ea700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c6c0b4e30 con 0x7f6c6c0942b0 2021-02-10T11:33:55.914 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:55.909 7f6c738ea700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c6c189100 con 0x7f6c6c0b4840 2021-02-10T11:33:55.914 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:55.910 7f6c70f05700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6c6c093eb0 0x7f6c6c188490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:55.915 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:55.910 7f6c70f05700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6c6c093eb0 0x7f6c6c188490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:35698/0 (socket says 172.21.2.102:35698) 2021-02-10T11:33:55.915 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:55.910 7f6c70f05700 1 -- 172.21.2.102:0/1484988957 learned_addr learned my addr 172.21.2.102:0/1484988957 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:33:55.915 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:55.910 7f6c70f05700 1 -- 172.21.2.102:0/1484988957 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6c6c0b4840 msgr2=0x7f6c6c093890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:33:55.915 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:55.910 7f6c70f05700 1 --2- 172.21.2.102:0/1484988957 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6c6c0b4840 0x7f6c6c093890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:55.915 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:55.910 7f6c70f05700 1 -- 172.21.2.102:0/1484988957 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6c6c0942b0 msgr2=0x7f6c6c188ab0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:33:55.915 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:55.910 7f6c70f05700 1 --2- 172.21.2.102:0/1484988957 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6c6c0942b0 0x7f6c6c188ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:55.915 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:55.910 7f6c70f05700 1 -- 172.21.2.102:0/1484988957 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c6c189320 con 0x7f6c6c093eb0 2021-02-10T11:33:55.915 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:55.910 7f6c70f05700 1 --2- 172.21.2.102:0/1484988957 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6c6c093eb0 0x7f6c6c188490 secure :-1 s=READY pgs=2476 cs=0 l=1 rev1=1 rx=0x7f6c6000b5c0 tx=0x7f6c6000b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:55.915 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:55.910 7f6c5dffb700 1 -- 172.21.2.102:0/1484988957 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6c6000bd50 con 0x7f6c6c093eb0 2021-02-10T11:33:55.915 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:55.910 7f6c5dffb700 1 -- 172.21.2.102:0/1484988957 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c60006030 con 0x7f6c6c093eb0 2021-02-10T11:33:55.915 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:55.910 7f6c738ea700 1 -- 172.21.2.102:0/1484988957 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c6c189320 con 0x7f6c6c093eb0 2021-02-10T11:33:55.915 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:55.910 7f6c5dffb700 1 -- 172.21.2.102:0/1484988957 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6c600054d0 con 0x7f6c6c093eb0 2021-02-10T11:33:55.915 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:55.910 7f6c738ea700 1 -- 172.21.2.102:0/1484988957 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c6c189320 con 0x7f6c6c093eb0 2021-02-10T11:33:55.915 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:55.911 7f6c5dffb700 1 -- 172.21.2.102:0/1484988957 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f6c60006210 con 0x7f6c6c093eb0 2021-02-10T11:33:55.916 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:55.911 7f6c5dffb700 1 --2- 172.21.2.102:0/1484988957 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6c4c0186c0 0x7f6c4c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:55.916 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:55.911 7f6c5dffb700 1 -- 172.21.2.102:0/1484988957 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f6c600327c0 con 0x7f6c6c093eb0 2021-02-10T11:33:55.916 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:55.911 7f6c71706700 1 --2- 172.21.2.102:0/1484988957 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6c4c0186c0 0x7f6c4c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:55.916 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:55.911 7f6c71706700 1 --2- 172.21.2.102:0/1484988957 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6c4c0186c0 0x7f6c4c01abc0 crc :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:55.945 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:55.943 7f6c738ea700 1 -- 172.21.2.102:0/1484988957 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c6c0ee0f0 con 0x7f6c6c093eb0 2021-02-10T11:33:55.947 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:55.946 7f6c5dffb700 1 -- 172.21.2.102:0/1484988957 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f6c60006b80 con 0x7f6c6c093eb0 2021-02-10T11:33:56.082 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:56.080 7f6c738ea700 1 -- 172.21.2.102:0/1484988957 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs volume ls", "target": ["mgr", ""]}) v1 -- 0x7f6c6c0ce450 con 0x7f6c4c0186c0 2021-02-10T11:33:56.083 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:56.081 7f6c5dffb700 1 -- 172.21.2.102:0/1484988957 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+41 (crc 0 0 0) 0x7f6c68007690 con 0x7f6c4c0186c0 2021-02-10T11:33:56.083 INFO:teuthology.orchestra.run.gibba002.stdout:[ 2021-02-10T11:33:56.083 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:33:56.083 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "cephfs" 2021-02-10T11:33:56.083 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:33:56.083 INFO:teuthology.orchestra.run.gibba002.stdout:] 2021-02-10T11:33:56.084 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:56.082 7f6c738ea700 1 -- 172.21.2.102:0/1484988957 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6c4c0186c0 msgr2=0x7f6c4c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:56.084 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:56.082 7f6c738ea700 1 --2- 172.21.2.102:0/1484988957 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6c4c0186c0 0x7f6c4c01abc0 crc :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:56.084 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:56.082 7f6c738ea700 1 -- 172.21.2.102:0/1484988957 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6c6c093eb0 msgr2=0x7f6c6c188490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:56.084 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:56.082 7f6c738ea700 1 --2- 172.21.2.102:0/1484988957 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6c6c093eb0 0x7f6c6c188490 secure :-1 s=READY pgs=2476 cs=0 l=1 rev1=1 rx=0x7f6c6000b5c0 tx=0x7f6c6000b960).stop 2021-02-10T11:33:56.085 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:56.083 7f6c738ea700 1 -- 172.21.2.102:0/1484988957 shutdown_connections 2021-02-10T11:33:56.085 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:56.083 7f6c738ea700 1 --2- 172.21.2.102:0/1484988957 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6c4c0186c0 0x7f6c4c01abc0 unknown :-1 s=CLOSED pgs=1097 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:56.085 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:56.083 7f6c738ea700 1 --2- 172.21.2.102:0/1484988957 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6c6c0942b0 0x7f6c6c188ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:56.085 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:56.083 7f6c738ea700 1 --2- 172.21.2.102:0/1484988957 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6c6c093eb0 0x7f6c6c188490 unknown :-1 s=CLOSED pgs=2476 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:56.085 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:56.083 7f6c738ea700 1 --2- 172.21.2.102:0/1484988957 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6c6c0b4840 0x7f6c6c093890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:56.085 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:56.083 7f6c738ea700 1 -- 172.21.2.102:0/1484988957 >> 172.21.2.102:0/1484988957 conn(0x7f6c6c0946f0 msgr2=0x7f6c6c095700 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:56.085 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:56.083 7f6c738ea700 1 -- 172.21.2.102:0/1484988957 shutdown_connections 2021-02-10T11:33:56.086 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:56.083 7f6c738ea700 1 -- 172.21.2.102:0/1484988957 wait complete. 2021-02-10T11:33:56.092 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config set mon mon_allow_pool_delete True 2021-02-10T11:33:56.531 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:56.509 7f691795a700 1 Processor -- start 2021-02-10T11:33:56.531 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:56.509 7f691795a700 1 -- start start 2021-02-10T11:33:56.532 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:56.509 7f691795a700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f69100d1c60 0x7f69100cef50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:56.532 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:56.509 7f691795a700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f69100cf570 0x7f69100cf970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:56.532 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:56.509 7f691795a700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69100fe1b0 con 0x7f69100d1860 2021-02-10T11:33:56.532 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:56.509 7f691795a700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69101b0ca0 con 0x7f69100cf570 2021-02-10T11:33:56.532 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:56.509 7f691795a700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6910109a50 con 0x7f69100d1c60 2021-02-10T11:33:56.532 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:56.510 7f6915f77700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f69100cf570 0x7f69100cf970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:56.532 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:56.510 7f6914f75700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f69100d1c60 0x7f69100cef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:56.533 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:56.510 7f6915776700 1 --1- >> v1:172.21.2.113:6789/0 conn(0x7f69100d1860 0x7f69101b04b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.113:6789/0 says I am v1:172.21.2.102:35252/0 (socket says 172.21.2.102:35252) 2021-02-10T11:33:56.533 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:56.510 7f6915776700 1 -- 172.21.2.102:0/29751137 learned_addr learned my addr 172.21.2.102:0/29751137 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:33:56.533 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:56.511 7f6915f77700 1 -- 172.21.2.102:0/29751137 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f69100d1c60 msgr2=0x7f69100cef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:33:56.533 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:56.511 7f6915f77700 1 --2- 172.21.2.102:0/29751137 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f69100d1c60 0x7f69100cef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:56.533 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:56.511 7f6915f77700 1 -- 172.21.2.102:0/29751137 >> v1:172.21.2.113:6789/0 conn(0x7f69100d1860 legacy=0x7f69101b04b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:56.533 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:56.511 7f6915f77700 1 -- 172.21.2.102:0/29751137 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6910088070 con 0x7f69100cf570 2021-02-10T11:33:56.533 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:56.511 7f6914f75700 1 --2- 172.21.2.102:0/29751137 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f69100d1c60 0x7f69100cef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:33:56.533 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:56.511 7f6915f77700 1 --2- 172.21.2.102:0/29751137 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f69100cf570 0x7f69100cf970 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 rx=0x7f6908006e50 tx=0x7f6908009e70).ready entity=mon.1 client_cookie=672c5544027423bb server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:56.533 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:56.511 7f6903fff700 1 -- 172.21.2.102:0/29751137 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f69080066f0 con 0x7f69100cf570 2021-02-10T11:33:56.533 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:56.512 7f6903fff700 1 -- 172.21.2.102:0/29751137 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6908003030 con 0x7f69100cf570 2021-02-10T11:33:56.533 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:56.512 7f6903fff700 1 -- 172.21.2.102:0/29751137 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f690800b700 con 0x7f69100cf570 2021-02-10T11:33:56.534 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:56.512 7f691795a700 1 -- 172.21.2.102:0/29751137 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f69100cf570 msgr2=0x7f69100cf970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:56.534 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:56.512 7f691795a700 1 --2- 172.21.2.102:0/29751137 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f69100cf570 0x7f69100cf970 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 rx=0x7f6908006e50 tx=0x7f6908009e70).stop 2021-02-10T11:33:56.534 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:56.512 7f691795a700 1 -- 172.21.2.102:0/29751137 shutdown_connections 2021-02-10T11:33:56.534 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:56.512 7f691795a700 1 --2- 172.21.2.102:0/29751137 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f69100d1c60 0x7f69100cef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:56.534 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:56.512 7f691795a700 1 --2- 172.21.2.102:0/29751137 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f69100cf570 0x7f69100cf970 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:56.534 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:56.512 7f691795a700 1 -- 172.21.2.102:0/29751137 >> 172.21.2.102:0/29751137 conn(0x7f69100ce410 msgr2=0x7f69100d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:56.534 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:56.512 7f691795a700 1 -- 172.21.2.102:0/29751137 shutdown_connections 2021-02-10T11:33:56.534 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:56.512 7f691795a700 1 -- 172.21.2.102:0/29751137 wait complete. 2021-02-10T11:33:56.534 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:56.527 7f691795a700 1 Processor -- start 2021-02-10T11:33:56.534 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:56.527 7f691795a700 1 -- start start 2021-02-10T11:33:56.534 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:56.527 7f691795a700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f69100b97d0 0x7f69100b8c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:56.535 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:56.527 7f691795a700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f69100b9c80 0x7f69100ba080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:56.535 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:56.527 7f691795a700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6910082c50 0x7f6910083050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:56.535 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:56.527 7f691795a700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69100c3880 con 0x7f6910082c50 2021-02-10T11:33:56.535 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:56.527 7f691795a700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69100c3e00 con 0x7f69100b9c80 2021-02-10T11:33:56.535 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:56.527 7f691795a700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69100b9250 con 0x7f69100b97d0 2021-02-10T11:33:56.536 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:56.527 7f6915f77700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6910082c50 0x7f6910083050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:56.536 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:56.527 7f6915f77700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6910082c50 0x7f6910083050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:35710/0 (socket says 172.21.2.102:35710) 2021-02-10T11:33:56.536 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:56.527 7f6915f77700 1 -- 172.21.2.102:0/2455404643 learned_addr learned my addr 172.21.2.102:0/2455404643 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:33:56.536 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:56.528 7f6914f75700 1 --2- 172.21.2.102:0/2455404643 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f69100b9c80 0x7f69100ba080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:56.536 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:56.528 7f6915776700 1 --2- 172.21.2.102:0/2455404643 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f69100b97d0 0x7f69100b8c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:56.536 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:56.528 7f6915f77700 1 -- 172.21.2.102:0/2455404643 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f69100b97d0 msgr2=0x7f69100b8c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:56.536 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:56.528 7f6915f77700 1 --2- 172.21.2.102:0/2455404643 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f69100b97d0 0x7f69100b8c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:56.536 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:56.528 7f6915f77700 1 -- 172.21.2.102:0/2455404643 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f69100b9c80 msgr2=0x7f69100ba080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:56.536 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:56.528 7f6915f77700 1 --2- 172.21.2.102:0/2455404643 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f69100b9c80 0x7f69100ba080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:56.536 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:56.528 7f6915f77700 1 -- 172.21.2.102:0/2455404643 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6910083670 con 0x7f6910082c50 2021-02-10T11:33:56.536 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:56.528 7f6915f77700 1 --2- 172.21.2.102:0/2455404643 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6910082c50 0x7f6910083050 secure :-1 s=READY pgs=2477 cs=0 l=1 rev1=1 rx=0x7f69080091d0 tx=0x7f690800bd60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:56.537 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:56.528 7f6901ffb700 1 -- 172.21.2.102:0/2455404643 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6908006ca0 con 0x7f6910082c50 2021-02-10T11:33:56.537 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:56.528 7f6901ffb700 1 -- 172.21.2.102:0/2455404643 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f690800c2f0 con 0x7f6910082c50 2021-02-10T11:33:56.537 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:56.528 7f6901ffb700 1 -- 172.21.2.102:0/2455404643 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f690800c780 con 0x7f6910082c50 2021-02-10T11:33:56.537 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:56.528 7f691795a700 1 -- 172.21.2.102:0/2455404643 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6910083670 con 0x7f6910082c50 2021-02-10T11:33:56.537 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:56.528 7f691795a700 1 -- 172.21.2.102:0/2455404643 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6910083670 con 0x7f6910082c50 2021-02-10T11:33:56.537 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:56.529 7f6901ffb700 1 -- 172.21.2.102:0/2455404643 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f6908006ca0 con 0x7f6910082c50 2021-02-10T11:33:56.537 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:56.529 7f6901ffb700 1 --2- 172.21.2.102:0/2455404643 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f68f00186c0 0x7f68f001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:56.537 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:56.529 7f6901ffb700 1 -- 172.21.2.102:0/2455404643 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f6908032920 con 0x7f6910082c50 2021-02-10T11:33:56.538 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:56.529 7f6915776700 1 --2- 172.21.2.102:0/2455404643 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f68f00186c0 0x7f68f001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:56.538 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:56.529 7f6915776700 1 --2- 172.21.2.102:0/2455404643 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f68f00186c0 0x7f68f001abc0 crc :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:56.561 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:56.559 7f691795a700 1 -- 172.21.2.102:0/2455404643 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69100c3da0 con 0x7f6910082c50 2021-02-10T11:33:56.563 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:56.562 7f6901ffb700 1 -- 172.21.2.102:0/2455404643 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f69080093d0 con 0x7f6910082c50 2021-02-10T11:33:56.714 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:56.712 7f691795a700 1 -- 172.21.2.102:0/2455404643 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f69100c3da0 con 0x7f6910082c50 2021-02-10T11:33:56.714 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:56.712 7f6901ffb700 1 -- 172.21.2.102:0/2455404643 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v32)=0 v32) v1 ==== 122+0+0 (secure 0 0 0) 0x7f6908006ca0 con 0x7f6910082c50 2021-02-10T11:33:56.716 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:56.714 7f691795a700 1 -- 172.21.2.102:0/2455404643 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f68f00186c0 msgr2=0x7f68f001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:56.716 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:56.714 7f691795a700 1 --2- 172.21.2.102:0/2455404643 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f68f00186c0 0x7f68f001abc0 crc :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:56.716 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:56.714 7f691795a700 1 -- 172.21.2.102:0/2455404643 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6910082c50 msgr2=0x7f6910083050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:56.717 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:56.714 7f691795a700 1 --2- 172.21.2.102:0/2455404643 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6910082c50 0x7f6910083050 secure :-1 s=READY pgs=2477 cs=0 l=1 rev1=1 rx=0x7f69080091d0 tx=0x7f690800bd60).stop 2021-02-10T11:33:56.717 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:56.714 7f691795a700 1 -- 172.21.2.102:0/2455404643 shutdown_connections 2021-02-10T11:33:56.717 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:56.714 7f691795a700 1 --2- 172.21.2.102:0/2455404643 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6910082c50 0x7f6910083050 unknown :-1 s=CLOSED pgs=2477 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:56.717 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:56.714 7f691795a700 1 --2- 172.21.2.102:0/2455404643 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f68f00186c0 0x7f68f001abc0 unknown :-1 s=CLOSED pgs=1098 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:56.717 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:56.714 7f691795a700 1 --2- 172.21.2.102:0/2455404643 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f69100b9c80 0x7f69100ba080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:56.717 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:56.714 7f691795a700 1 --2- 172.21.2.102:0/2455404643 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f69100b97d0 0x7f69100b8c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:56.717 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:56.714 7f691795a700 1 -- 172.21.2.102:0/2455404643 >> 172.21.2.102:0/2455404643 conn(0x7f69100a7410 msgr2=0x7f6910193350 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:56.717 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:56.715 7f691795a700 1 -- 172.21.2.102:0/2455404643 shutdown_connections 2021-02-10T11:33:56.717 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:56.715 7f691795a700 1 -- 172.21.2.102:0/2455404643 wait complete. 2021-02-10T11:33:56.725 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000632662 2021-02-10T11:33:57.157 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:57.139 7f9435032700 1 Processor -- start 2021-02-10T11:33:57.157 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:57.139 7f9435032700 1 -- start start 2021-02-10T11:33:57.157 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:57.139 7f9435032700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f94300d1860 0x7f94301b04b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:57.157 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:57.139 7f9435032700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f94301b0ad0 0x7f9430108930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:57.157 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:57.139 7f9435032700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94300fe1b0 con 0x7f94300d1c60 2021-02-10T11:33:57.157 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:57.139 7f9435032700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9430109a50 con 0x7f94300d1860 2021-02-10T11:33:57.157 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:57.139 7f9435032700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9430109740 con 0x7f94301b0ad0 2021-02-10T11:33:57.157 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:57.140 7f942ed9d700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f94300d1860 0x7f94301b04b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:57.157 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:57.140 7f942ed9d700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f94300d1860 0x7f94301b04b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:35718/0 (socket says 172.21.2.102:35718) 2021-02-10T11:33:57.157 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:57.140 7f942ed9d700 1 -- 172.21.2.102:0/3333998887 learned_addr learned my addr 172.21.2.102:0/3333998887 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:33:57.158 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:57.140 7f942ed9d700 1 -- 172.21.2.102:0/3333998887 >> v2:172.21.2.113:3300/0 conn(0x7f94301b0ad0 msgr2=0x7f9430108930 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:33:57.158 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:57.140 7f942ed9d700 1 --2- 172.21.2.102:0/3333998887 >> v2:172.21.2.113:3300/0 conn(0x7f94301b0ad0 0x7f9430108930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:57.158 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:57.140 7f942ed9d700 1 -- 172.21.2.102:0/3333998887 >> v1:172.21.2.113:6789/0 conn(0x7f94300d1c60 legacy=0x7f9430108f50 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:33:57.158 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:57.140 7f942ed9d700 1 -- 172.21.2.102:0/3333998887 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9430088070 con 0x7f94300d1860 2021-02-10T11:33:57.158 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:57.140 7f942ed9d700 1 --2- 172.21.2.102:0/3333998887 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f94300d1860 0x7f94301b04b0 secure :-1 s=READY pgs=2478 cs=0 l=1 rev1=1 rx=0x7f9420006700 tx=0x7f9420009b80).ready entity=mon.0 client_cookie=f02969359cd29272 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:57.158 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:57.140 7f942dd9b700 1 -- 172.21.2.102:0/3333998887 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f942000c0c0 con 0x7f94300d1860 2021-02-10T11:33:57.158 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:57.140 7f942dd9b700 1 -- 172.21.2.102:0/3333998887 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f942000c440 con 0x7f94300d1860 2021-02-10T11:33:57.158 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:57.140 7f942dd9b700 1 -- 172.21.2.102:0/3333998887 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f942000c960 con 0x7f94300d1860 2021-02-10T11:33:57.158 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:57.140 7f9435032700 1 -- 172.21.2.102:0/3333998887 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f94300d1860 msgr2=0x7f94301b04b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:57.158 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:57.140 7f9435032700 1 --2- 172.21.2.102:0/3333998887 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f94300d1860 0x7f94301b04b0 secure :-1 s=READY pgs=2478 cs=0 l=1 rev1=1 rx=0x7f9420006700 tx=0x7f9420009b80).stop 2021-02-10T11:33:57.158 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:57.141 7f9435032700 1 -- 172.21.2.102:0/3333998887 shutdown_connections 2021-02-10T11:33:57.158 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:57.141 7f9435032700 1 --2- 172.21.2.102:0/3333998887 >> v2:172.21.2.113:3300/0 conn(0x7f94301b0ad0 0x7f9430108930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:57.158 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:57.141 7f9435032700 1 --2- 172.21.2.102:0/3333998887 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f94300d1860 0x7f94301b04b0 unknown :-1 s=CLOSED pgs=2478 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:57.159 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:57.141 7f9435032700 1 -- 172.21.2.102:0/3333998887 >> 172.21.2.102:0/3333998887 conn(0x7f94300ce410 msgr2=0x7f94300d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:57.159 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:57.141 7f9435032700 1 -- 172.21.2.102:0/3333998887 shutdown_connections 2021-02-10T11:33:57.159 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:57.141 7f9435032700 1 -- 172.21.2.102:0/3333998887 wait complete. 2021-02-10T11:33:57.160 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:57.155 7f9435032700 1 Processor -- start 2021-02-10T11:33:57.160 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:57.155 7f9435032700 1 -- start start 2021-02-10T11:33:57.160 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:57.156 7f9435032700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f94301a05e0 0x7f943018b970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:57.160 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:57.156 7f9435032700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f94300b8780 0x7f94300bfa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:57.161 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:57.156 7f9435032700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f94300c0030 0x7f9430189e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:57.161 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:57.156 7f9435032700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94300b82a0 con 0x7f94301a05e0 2021-02-10T11:33:57.161 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:57.156 7f9435032700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94300c0430 con 0x7f94300c0030 2021-02-10T11:33:57.161 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:57.156 7f9435032700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94300c0650 con 0x7f94300b8780 2021-02-10T11:33:57.161 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:57.156 7f942ed9d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f94301a05e0 0x7f943018b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:57.161 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:57.156 7f942ed9d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f94301a05e0 0x7f943018b970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:35728/0 (socket says 172.21.2.102:35728) 2021-02-10T11:33:57.161 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:57.156 7f942ed9d700 1 -- 172.21.2.102:0/1928136736 learned_addr learned my addr 172.21.2.102:0/1928136736 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:33:57.161 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:57.156 7f942ed9d700 1 -- 172.21.2.102:0/1928136736 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f94300b8780 msgr2=0x7f94300bfa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:57.161 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:57.156 7f942ed9d700 1 --2- 172.21.2.102:0/1928136736 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f94300b8780 0x7f94300bfa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:57.161 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:57.156 7f942ed9d700 1 -- 172.21.2.102:0/1928136736 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f94300c0030 msgr2=0x7f9430189e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:57.161 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:57.156 7f942ed9d700 1 --2- 172.21.2.102:0/1928136736 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f94300c0030 0x7f9430189e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:57.161 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:57.156 7f942ed9d700 1 -- 172.21.2.102:0/1928136736 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f943018a490 con 0x7f94301a05e0 2021-02-10T11:33:57.161 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:57.157 7f942ed9d700 1 --2- 172.21.2.102:0/1928136736 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f94301a05e0 0x7f943018b970 secure :-1 s=READY pgs=2479 cs=0 l=1 rev1=1 rx=0x7f942000b5e0 tx=0x7f942000b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:57.162 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:57.157 7f94277fe700 1 -- 172.21.2.102:0/1928136736 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f942000bd70 con 0x7f94301a05e0 2021-02-10T11:33:57.162 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:57.157 7f94277fe700 1 -- 172.21.2.102:0/1928136736 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9420002030 con 0x7f94301a05e0 2021-02-10T11:33:57.162 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:57.157 7f94277fe700 1 -- 172.21.2.102:0/1928136736 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9420002550 con 0x7f94301a05e0 2021-02-10T11:33:57.162 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:57.157 7f9435032700 1 -- 172.21.2.102:0/1928136736 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f943018a490 con 0x7f94301a05e0 2021-02-10T11:33:57.162 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:57.157 7f9435032700 1 -- 172.21.2.102:0/1928136736 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f943018a490 con 0x7f94301a05e0 2021-02-10T11:33:57.162 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:57.158 7f94277fe700 1 -- 172.21.2.102:0/1928136736 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f9420000c90 con 0x7f94301a05e0 2021-02-10T11:33:57.162 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:57.158 7f94277fe700 1 --2- 172.21.2.102:0/1928136736 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f94140186c0 0x7f941401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:57.162 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:57.158 7f94277fe700 1 -- 172.21.2.102:0/1928136736 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f94200325b0 con 0x7f94301a05e0 2021-02-10T11:33:57.162 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:57.158 7f942e59c700 1 --2- 172.21.2.102:0/1928136736 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f94140186c0 0x7f941401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:57.162 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:57.158 7f942e59c700 1 --2- 172.21.2.102:0/1928136736 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f94140186c0 0x7f941401abc0 crc :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:57.191 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:57.189 7f9435032700 1 -- 172.21.2.102:0/1928136736 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9430082a20 con 0x7f94301a05e0 2021-02-10T11:33:57.194 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:57.192 7f94277fe700 1 -- 172.21.2.102:0/1928136736 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f9420025fe0 con 0x7f94301a05e0 2021-02-10T11:33:57.318 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:57.316 7f9435032700 1 -- 172.21.2.102:0/1928136736 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000632662", "target": ["mgr", ""]}) v1 -- 0x7f9430082a20 con 0x7f94140186c0 2021-02-10T11:33:57.322 INFO:tasks.ceph.mgr.y.gibba002.stderr:2021-02-10 11:33:57.321 7fd94bc53700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:33:57.322 INFO:tasks.ceph.mgr.y.gibba002.stderr:2021-02-10 11:33:57.321 7fd94bc53700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:33:57.322 INFO:tasks.ceph.mgr.y.gibba002.stderr:2021-02-10 11:33:57.321 7fd94bc53700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:33:57.322 INFO:tasks.ceph.mgr.y.gibba002.stderr:2021-02-10 11:33:57.321 7fd94bc53700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:33:57.322 INFO:tasks.ceph.mgr.y.gibba002.stderr:2021-02-10 11:33:57.321 7fd94bc53700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:33:57.353 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:57.351 7f94277fe700 1 -- 172.21.2.102:0/1928136736 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f9418006e40 con 0x7f94140186c0 2021-02-10T11:33:57.384 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:57.382 7f9435032700 1 -- 172.21.2.102:0/1928136736 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f94140186c0 msgr2=0x7f941401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:57.384 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:57.382 7f9435032700 1 --2- 172.21.2.102:0/1928136736 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f94140186c0 0x7f941401abc0 crc :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:57.385 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:57.382 7f9435032700 1 -- 172.21.2.102:0/1928136736 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f94301a05e0 msgr2=0x7f943018b970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:57.385 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:57.382 7f9435032700 1 --2- 172.21.2.102:0/1928136736 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f94301a05e0 0x7f943018b970 secure :-1 s=READY pgs=2479 cs=0 l=1 rev1=1 rx=0x7f942000b5e0 tx=0x7f942000b980).stop 2021-02-10T11:33:57.385 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:57.382 7f9435032700 1 -- 172.21.2.102:0/1928136736 shutdown_connections 2021-02-10T11:33:57.385 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:57.382 7f9435032700 1 --2- 172.21.2.102:0/1928136736 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f94140186c0 0x7f941401abc0 unknown :-1 s=CLOSED pgs=1099 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:57.385 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:57.382 7f9435032700 1 --2- 172.21.2.102:0/1928136736 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f94300c0030 0x7f9430189e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:57.385 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:57.382 7f9435032700 1 --2- 172.21.2.102:0/1928136736 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f94300b8780 0x7f94300bfa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:57.386 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:57.382 7f9435032700 1 --2- 172.21.2.102:0/1928136736 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f94301a05e0 0x7f943018b970 unknown :-1 s=CLOSED pgs=2479 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:57.386 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:57.382 7f9435032700 1 -- 172.21.2.102:0/1928136736 >> 172.21.2.102:0/1928136736 conn(0x7f94300a6a80 msgr2=0x7f94300ae110 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:57.386 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:57.382 7f9435032700 1 -- 172.21.2.102:0/1928136736 shutdown_connections 2021-02-10T11:33:57.386 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:57.382 7f9435032700 1 -- 172.21.2.102:0/1928136736 wait complete. 2021-02-10T11:33:57.392 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs clone status cephfs subvolume_0000000000632662 2021-02-10T11:33:57.825 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:57.804 7fb6e4393700 1 Processor -- start 2021-02-10T11:34:00.017 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:57.804 7fb6e4393700 1 -- start start 2021-02-10T11:34:00.017 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:57.805 7fb6e4393700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fb6dc0d1870 0x7fb6dc1b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:00.017 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:57.805 7fb6e4393700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6dc0fe1c0 con 0x7fb6dc0cff70 2021-02-10T11:34:00.017 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:57.805 7fb6e4393700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6dc109a60 con 0x7fb6dc1b0ae0 2021-02-10T11:34:00.017 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:57.805 7fb6e4393700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6dc088070 con 0x7fb6dc0d1870 2021-02-10T11:34:00.017 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:57.805 7fb6e29b0700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fb6dc0cff70 0x7fb6dc109130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:56202/0 (socket says 172.21.2.102:56202) 2021-02-10T11:34:00.017 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:57.805 7fb6e29b0700 1 -- 172.21.2.102:0/3799351377 learned_addr learned my addr 172.21.2.102:0/3799351377 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:34:00.017 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:57.806 7fb6e11ad700 1 -- 172.21.2.102:0/3799351377 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 257564032 0 0) 0x7fb6d4001500 con 0x7fb6dc0cff70 2021-02-10T11:34:00.018 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:57.806 7fb6e11ad700 1 -- 172.21.2.102:0/3799351377 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6c8002fe0 con 0x7fb6dc0cff70 2021-02-10T11:34:00.018 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:57.806 7fb6e21af700 1 --2- 172.21.2.102:0/3799351377 >> v2:172.21.2.113:3300/0 conn(0x7fb6dc0d1870 0x7fb6dc1b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:00.018 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:57.806 7fb6e11ad700 1 -- 172.21.2.102:0/3799351377 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1877300526 0 0) 0x7fb6d4001500 con 0x7fb6dc0cff70 2021-02-10T11:34:00.018 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:57.806 7fb6e11ad700 1 -- 172.21.2.102:0/3799351377 >> v2:172.21.2.113:3300/0 conn(0x7fb6dc0d1870 msgr2=0x7fb6dc1b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:34:00.018 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:57.806 7fb6e11ad700 1 --2- 172.21.2.102:0/3799351377 >> v2:172.21.2.113:3300/0 conn(0x7fb6dc0d1870 0x7fb6dc1b04c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:00.018 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:57.806 7fb6e11ad700 1 -- 172.21.2.102:0/3799351377 >> v1:172.21.2.113:6789/0 conn(0x7fb6dc1b0ae0 legacy=0x7fb6dc108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:00.018 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:57.806 7fb6e11ad700 1 -- 172.21.2.102:0/3799351377 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6dc088260 con 0x7fb6dc0cff70 2021-02-10T11:34:00.018 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:57.806 7fb6e11ad700 1 -- 172.21.2.102:0/3799351377 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7fb6d4001a00 con 0x7fb6dc0cff70 2021-02-10T11:34:00.018 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:57.806 7fb6e11ad700 1 -- 172.21.2.102:0/3799351377 >> v1:172.21.2.102:6789/0 conn(0x7fb6dc0cff70 legacy=0x7fb6dc109130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:00.018 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:57.806 7fb6e11ad700 1 --2- 172.21.2.102:0/3799351377 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb6c8003af0 0x7fb6c8005f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:00.018 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:57.806 7fb6e11ad700 1 --2- 172.21.2.102:0/3799351377 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb6c8006780 0x7fb6c8008c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:00.018 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:57.806 7fb6e11ad700 1 --2- 172.21.2.102:0/3799351377 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb6c8009580 0x7fb6c800baa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:00.019 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:57.806 7fb6e11ad700 1 -- 172.21.2.102:0/3799351377 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6c800c2f0 con 0x7fb6c8006780 2021-02-10T11:34:00.019 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:57.806 7fb6e11ad700 1 -- 172.21.2.102:0/3799351377 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6c800c510 con 0x7fb6c8009580 2021-02-10T11:34:00.019 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:57.806 7fb6e11ad700 1 -- 172.21.2.102:0/3799351377 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6c800c730 con 0x7fb6c8003af0 2021-02-10T11:34:00.019 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:57.806 7fb6e19ae700 1 --2- 172.21.2.102:0/3799351377 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb6c8006780 0x7fb6c8008c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:00.019 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:57.806 7fb6e21af700 1 --2- 172.21.2.102:0/3799351377 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb6c8003af0 0x7fb6c8005f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:00.019 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:57.806 7fb6e29b0700 1 --2- 172.21.2.102:0/3799351377 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb6c8009580 0x7fb6c800baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:00.019 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:57.807 7fb6e19ae700 1 -- 172.21.2.102:0/3799351377 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb6c8003af0 msgr2=0x7fb6c8005f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:34:00.019 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:57.807 7fb6e19ae700 1 --2- 172.21.2.102:0/3799351377 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb6c8003af0 0x7fb6c8005f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:00.019 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:57.807 7fb6e19ae700 1 -- 172.21.2.102:0/3799351377 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb6c8009580 msgr2=0x7fb6c800baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:34:00.019 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:57.807 7fb6e19ae700 1 --2- 172.21.2.102:0/3799351377 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb6c8009580 0x7fb6c800baa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:00.019 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:57.807 7fb6e19ae700 1 -- 172.21.2.102:0/3799351377 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6c80509a0 con 0x7fb6c8006780 2021-02-10T11:34:00.019 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:57.807 7fb6e21af700 1 --2- 172.21.2.102:0/3799351377 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb6c8003af0 0x7fb6c8005f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:34:00.019 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:57.807 7fb6e29b0700 1 --2- 172.21.2.102:0/3799351377 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb6c8009580 0x7fb6c800baa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:34:00.019 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:57.807 7fb6e29b0700 1 -- 172.21.2.102:0/3799351377 reap_dead start 2021-02-10T11:34:00.020 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:57.807 7fb6e19ae700 1 --2- 172.21.2.102:0/3799351377 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb6c8006780 0x7fb6c8008c80 secure :-1 s=READY pgs=2483 cs=0 l=1 rev1=1 rx=0x7fb6d8006ac0 tx=0x7fb6d8009af0).ready entity=mon.0 client_cookie=30fd33bb3aaa5a9a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:00.020 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:57.807 7fb6e11ad700 1 -- 172.21.2.102:0/3799351377 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb6d8007470 con 0x7fb6c8006780 2021-02-10T11:34:00.020 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:57.807 7fb6e11ad700 1 -- 172.21.2.102:0/3799351377 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6d80009b0 con 0x7fb6c8006780 2021-02-10T11:34:00.020 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:57.807 7fb6e11ad700 1 -- 172.21.2.102:0/3799351377 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb6d80079b0 con 0x7fb6c8006780 2021-02-10T11:34:00.020 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:57.807 7fb6e4393700 1 -- 172.21.2.102:0/3799351377 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb6c8006780 msgr2=0x7fb6c8008c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:00.020 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:57.807 7fb6e4393700 1 --2- 172.21.2.102:0/3799351377 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb6c8006780 0x7fb6c8008c80 secure :-1 s=READY pgs=2483 cs=0 l=1 rev1=1 rx=0x7fb6d8006ac0 tx=0x7fb6d8009af0).stop 2021-02-10T11:34:00.020 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:57.808 7fb6e4393700 1 -- 172.21.2.102:0/3799351377 shutdown_connections 2021-02-10T11:34:00.020 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:57.808 7fb6e4393700 1 --2- 172.21.2.102:0/3799351377 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb6c8006780 0x7fb6c8008c80 unknown :-1 s=CLOSED pgs=2483 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:00.020 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:57.808 7fb6e4393700 1 -- 172.21.2.102:0/3799351377 >> 172.21.2.102:0/3799351377 conn(0x7fb6dc0ce420 msgr2=0x7fb6dc0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:00.020 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:57.808 7fb6e4393700 1 -- 172.21.2.102:0/3799351377 shutdown_connections 2021-02-10T11:34:00.020 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:57.808 7fb6e4393700 1 -- 172.21.2.102:0/3799351377 wait complete. 2021-02-10T11:34:00.020 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:57.822 7fb6e4393700 1 Processor -- start 2021-02-10T11:34:00.021 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:57.822 7fb6e4393700 1 -- start start 2021-02-10T11:34:00.021 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:57.823 7fb6e4393700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb6dc1a79a0 0x7fb6dc1a7da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:00.021 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:57.823 7fb6e4393700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb6dc0bdb40 0x7fb6dc0bdf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:00.021 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:57.823 7fb6e4393700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb6dc082ca0 0x7fb6dc0830a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:00.021 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:57.823 7fb6e4393700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6dc19d990 con 0x7fb6dc1a79a0 2021-02-10T11:34:00.021 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:57.823 7fb6e4393700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6dc19dbb0 con 0x7fb6dc082ca0 2021-02-10T11:34:00.021 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:57.823 7fb6e4393700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6dc0836c0 con 0x7fb6dc0bdb40 2021-02-10T11:34:00.021 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:57.823 7fb6e21af700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb6dc1a79a0 0x7fb6dc1a7da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:00.021 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:57.823 7fb6e21af700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb6dc1a79a0 0x7fb6dc1a7da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:35758/0 (socket says 172.21.2.102:35758) 2021-02-10T11:34:00.021 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:57.823 7fb6e21af700 1 -- 172.21.2.102:0/655810534 learned_addr learned my addr 172.21.2.102:0/655810534 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:34:00.024 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:57.824 7fb6e21af700 1 -- 172.21.2.102:0/655810534 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb6dc0bdb40 msgr2=0x7fb6dc0bdf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:00.024 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:57.824 7fb6e21af700 1 --2- 172.21.2.102:0/655810534 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb6dc0bdb40 0x7fb6dc0bdf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:00.024 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:57.824 7fb6e21af700 1 -- 172.21.2.102:0/655810534 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb6dc082ca0 msgr2=0x7fb6dc0830a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:00.024 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:57.824 7fb6e21af700 1 --2- 172.21.2.102:0/655810534 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb6dc082ca0 0x7fb6dc0830a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:00.024 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:57.824 7fb6e21af700 1 -- 172.21.2.102:0/655810534 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6dc0838e0 con 0x7fb6dc1a79a0 2021-02-10T11:34:00.024 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:57.824 7fb6e21af700 1 --2- 172.21.2.102:0/655810534 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb6dc1a79a0 0x7fb6dc1a7da0 secure :-1 s=READY pgs=2484 cs=0 l=1 rev1=1 rx=0x7fb6d000b5e0 tx=0x7fb6d000b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:00.024 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:57.824 7fb6ceffd700 1 -- 172.21.2.102:0/655810534 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb6d000bd70 con 0x7fb6dc1a79a0 2021-02-10T11:34:00.024 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:57.824 7fb6ceffd700 1 -- 172.21.2.102:0/655810534 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6d0000c90 con 0x7fb6dc1a79a0 2021-02-10T11:34:00.024 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:57.824 7fb6e4393700 1 -- 172.21.2.102:0/655810534 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6dc0838e0 con 0x7fb6dc1a79a0 2021-02-10T11:34:00.024 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:57.824 7fb6ceffd700 1 -- 172.21.2.102:0/655810534 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb6d00022b0 con 0x7fb6dc1a79a0 2021-02-10T11:34:00.024 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:57.824 7fb6e4393700 1 -- 172.21.2.102:0/655810534 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6dc084420 con 0x7fb6dc1a79a0 2021-02-10T11:34:00.025 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:57.825 7fb6ceffd700 1 -- 172.21.2.102:0/655810534 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fb6d0006740 con 0x7fb6dc1a79a0 2021-02-10T11:34:00.025 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:57.825 7fb6ceffd700 1 --2- 172.21.2.102:0/655810534 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb6c40186c0 0x7fb6c401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:00.025 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:57.825 7fb6ceffd700 1 -- 172.21.2.102:0/655810534 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fb6d0032800 con 0x7fb6dc1a79a0 2021-02-10T11:34:00.025 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:57.825 7fb6e19ae700 1 --2- 172.21.2.102:0/655810534 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb6c40186c0 0x7fb6c401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:00.025 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:57.825 7fb6e19ae700 1 --2- 172.21.2.102:0/655810534 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb6c40186c0 0x7fb6c401abc0 crc :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:00.025 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:57.855 7fb6e4393700 1 -- 172.21.2.102:0/655810534 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6dc082a70 con 0x7fb6dc1a79a0 2021-02-10T11:34:00.025 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:57.858 7fb6ceffd700 1 -- 172.21.2.102:0/655810534 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fb6d0000c90 con 0x7fb6dc1a79a0 2021-02-10T11:34:00.025 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:57.989 7fb6e4393700 1 -- 172.21.2.102:0/655810534 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvolume_0000000000632662", "target": ["mgr", ""]}) v1 -- 0x7fb6dc19d990 con 0x7fb6c40186c0 2021-02-10T11:34:00.025 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:57.992 7fb6ceffd700 1 -- 172.21.2.102:0/655810534 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: -95 operation 'clone-status' is not allowed on subvolume 'subvolume_0000000000632662' of type subvolume) v1 ==== 107+0+0 (crc 0 0 0) 0x7fb6d80092a0 con 0x7fb6c40186c0 2021-02-10T11:34:00.025 INFO:teuthology.orchestra.run.gibba002.stderr:Error ENOTSUP: operation 'clone-status' is not allowed on subvolume 'subvolume_0000000000632662' of type subvolume 2021-02-10T11:34:00.025 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:57.997 7fb6e4393700 1 -- 172.21.2.102:0/655810534 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb6c40186c0 msgr2=0x7fb6c401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:00.025 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:57.997 7fb6e4393700 1 --2- 172.21.2.102:0/655810534 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb6c40186c0 0x7fb6c401abc0 crc :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:00.025 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:57.997 7fb6e4393700 1 -- 172.21.2.102:0/655810534 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb6dc1a79a0 msgr2=0x7fb6dc1a7da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:00.025 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:57.997 7fb6e4393700 1 --2- 172.21.2.102:0/655810534 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb6dc1a79a0 0x7fb6dc1a7da0 secure :-1 s=READY pgs=2484 cs=0 l=1 rev1=1 rx=0x7fb6d000b5e0 tx=0x7fb6d000b980).stop 2021-02-10T11:34:00.026 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:57.997 7fb6e4393700 1 -- 172.21.2.102:0/655810534 shutdown_connections 2021-02-10T11:34:00.026 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:57.997 7fb6e4393700 1 --2- 172.21.2.102:0/655810534 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb6c40186c0 0x7fb6c401abc0 unknown :-1 s=CLOSED pgs=1100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:00.026 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:57.997 7fb6e4393700 1 --2- 172.21.2.102:0/655810534 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb6dc082ca0 0x7fb6dc0830a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:00.026 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:57.997 7fb6e4393700 1 --2- 172.21.2.102:0/655810534 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb6dc0bdb40 0x7fb6dc0bdf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:00.026 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:57.997 7fb6e4393700 1 --2- 172.21.2.102:0/655810534 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb6dc1a79a0 0x7fb6dc1a7da0 unknown :-1 s=CLOSED pgs=2484 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:00.026 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:57.997 7fb6e4393700 1 -- 172.21.2.102:0/655810534 >> 172.21.2.102:0/655810534 conn(0x7fb6dc0bef00 msgr2=0x7fb6dc09f200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:00.026 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:57.997 7fb6e4393700 1 -- 172.21.2.102:0/655810534 shutdown_connections 2021-02-10T11:34:00.026 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:33:57.997 7fb6e4393700 1 -- 172.21.2.102:0/655810534 wait complete. 2021-02-10T11:34:00.027 INFO:tasks.ceph.mgr.y.gibba002.stderr:2021-02-10 11:33:57.992 7fd94bc53700 -1 mgr.server reply reply (95) Operation not supported operation 'clone-status' is not allowed on subvolume 'subvolume_0000000000632662' of type subvolume 2021-02-10T11:34:00.027 DEBUG:teuthology.orchestra.run:got remote process result: 95 2021-02-10T11:34:00.027 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000632662 2021-02-10T11:34:00.458 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:00.440 7fcfe519f700 1 Processor -- start 2021-02-10T11:34:00.459 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:00.440 7fcfe519f700 1 -- start start 2021-02-10T11:34:00.459 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:00.440 7fcfe519f700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fcfe00d1870 0x7fcfe01b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:00.459 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:00.440 7fcfe519f700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcfe00fe1c0 con 0x7fcfe00cff70 2021-02-10T11:34:00.459 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:00.440 7fcfe519f700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcfe0109a60 con 0x7fcfe01b0ae0 2021-02-10T11:34:00.459 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:00.440 7fcfe519f700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcfe0088070 con 0x7fcfe00d1870 2021-02-10T11:34:00.459 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:00.440 7fcfdf59e700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fcfe00cff70 0x7fcfe0109130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:56224/0 (socket says 172.21.2.102:56224) 2021-02-10T11:34:00.459 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:00.441 7fcfdf59e700 1 -- 172.21.2.102:0/4219275052 learned_addr learned my addr 172.21.2.102:0/4219275052 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:34:00.459 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:00.441 7fcfded9d700 1 --2- 172.21.2.102:0/4219275052 >> v2:172.21.2.102:3300/0 conn(0x7fcfe00d1870 0x7fcfe01b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:00.459 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:00.441 7fcfddd9b700 1 -- 172.21.2.102:0/4219275052 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3610729227 0 0) 0x7fcfd0001500 con 0x7fcfe00cff70 2021-02-10T11:34:00.459 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:00.441 7fcfddd9b700 1 -- 172.21.2.102:0/4219275052 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcfcc002fe0 con 0x7fcfe00cff70 2021-02-10T11:34:00.459 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:00.441 7fcfded9d700 1 -- 172.21.2.102:0/4219275052 >> v1:172.21.2.113:6789/0 conn(0x7fcfe01b0ae0 legacy=0x7fcfe0108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:34:00.459 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:00.441 7fcfded9d700 1 -- 172.21.2.102:0/4219275052 >> v1:172.21.2.102:6789/0 conn(0x7fcfe00cff70 legacy=0x7fcfe0109130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:00.460 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:00.441 7fcfded9d700 1 -- 172.21.2.102:0/4219275052 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfe0088260 con 0x7fcfe00d1870 2021-02-10T11:34:00.460 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:00.441 7fcfded9d700 1 --2- 172.21.2.102:0/4219275052 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcfe00d1870 0x7fcfe01b04c0 secure :-1 s=READY pgs=2486 cs=0 l=1 rev1=1 rx=0x7fcfd8006650 tx=0x7fcfd8009a80).ready entity=mon.0 client_cookie=c92fbc61153575e5 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:00.460 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:00.441 7fcfddd9b700 1 -- 172.21.2.102:0/4219275052 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcfd800c0c0 con 0x7fcfe00d1870 2021-02-10T11:34:00.460 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:00.441 7fcfddd9b700 1 -- 172.21.2.102:0/4219275052 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfd800c340 con 0x7fcfe00d1870 2021-02-10T11:34:00.460 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:00.441 7fcfddd9b700 1 -- 172.21.2.102:0/4219275052 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcfd800c860 con 0x7fcfe00d1870 2021-02-10T11:34:00.460 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:00.441 7fcfe519f700 1 -- 172.21.2.102:0/4219275052 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcfe00d1870 msgr2=0x7fcfe01b04c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:00.460 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:00.441 7fcfe519f700 1 --2- 172.21.2.102:0/4219275052 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcfe00d1870 0x7fcfe01b04c0 secure :-1 s=READY pgs=2486 cs=0 l=1 rev1=1 rx=0x7fcfd8006650 tx=0x7fcfd8009a80).stop 2021-02-10T11:34:00.460 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:00.442 7fcfe519f700 1 -- 172.21.2.102:0/4219275052 shutdown_connections 2021-02-10T11:34:00.460 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:00.442 7fcfe519f700 1 --2- 172.21.2.102:0/4219275052 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcfe00d1870 0x7fcfe01b04c0 unknown :-1 s=CLOSED pgs=2486 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:00.460 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:00.442 7fcfe519f700 1 -- 172.21.2.102:0/4219275052 >> 172.21.2.102:0/4219275052 conn(0x7fcfe00ce420 msgr2=0x7fcfe00d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:00.460 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:00.442 7fcfe519f700 1 -- 172.21.2.102:0/4219275052 shutdown_connections 2021-02-10T11:34:00.460 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:00.442 7fcfe519f700 1 -- 172.21.2.102:0/4219275052 wait complete. 2021-02-10T11:34:00.461 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:00.457 7fcfe519f700 1 Processor -- start 2021-02-10T11:34:00.461 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:00.457 7fcfe519f700 1 -- start start 2021-02-10T11:34:00.462 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:00.457 7fcfe519f700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fcfe00c3820 0x7fcfe00a7950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:00.462 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:00.457 7fcfe519f700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcfe01a44c0 0x7fcfe01a48c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:00.462 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:00.457 7fcfe519f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcfe00863d0 0x7fcfe00867d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:00.463 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:00.457 7fcfe519f700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe0085ed0 con 0x7fcfe00863d0 2021-02-10T11:34:00.463 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:00.457 7fcfe519f700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe00c0740 con 0x7fcfe00c3820 2021-02-10T11:34:00.463 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:00.457 7fcfe519f700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe0086df0 con 0x7fcfe01a44c0 2021-02-10T11:34:00.463 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:00.457 7fcfdf59e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcfe00863d0 0x7fcfe00867d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:00.463 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:00.457 7fcfdf59e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcfe00863d0 0x7fcfe00867d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:35772/0 (socket says 172.21.2.102:35772) 2021-02-10T11:34:00.463 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:00.457 7fcfdf59e700 1 -- 172.21.2.102:0/2257183556 learned_addr learned my addr 172.21.2.102:0/2257183556 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:34:00.463 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:00.458 7fcfdf59e700 1 -- 172.21.2.102:0/2257183556 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcfe01a44c0 msgr2=0x7fcfe01a48c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:00.463 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:00.458 7fcfdf59e700 1 --2- 172.21.2.102:0/2257183556 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcfe01a44c0 0x7fcfe01a48c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:00.463 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:00.458 7fcfdf59e700 1 -- 172.21.2.102:0/2257183556 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fcfe00c3820 msgr2=0x7fcfe00a7950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:00.464 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:00.458 7fcfdf59e700 1 --2- 172.21.2.102:0/2257183556 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fcfe00c3820 0x7fcfe00a7950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:00.464 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:00.458 7fcfdf59e700 1 -- 172.21.2.102:0/2257183556 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfe0087010 con 0x7fcfe00863d0 2021-02-10T11:34:00.464 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:00.458 7fcfdf59e700 1 --2- 172.21.2.102:0/2257183556 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcfe00863d0 0x7fcfe00867d0 secure :-1 s=READY pgs=2487 cs=0 l=1 rev1=1 rx=0x7fcfd000a8f0 tx=0x7fcfd000ac90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:00.464 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:00.458 7fcfd77fe700 1 -- 172.21.2.102:0/2257183556 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcfd0007d10 con 0x7fcfe00863d0 2021-02-10T11:34:00.464 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:00.458 7fcfd77fe700 1 -- 172.21.2.102:0/2257183556 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfd0007160 con 0x7fcfe00863d0 2021-02-10T11:34:00.464 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:00.458 7fcfe519f700 1 -- 172.21.2.102:0/2257183556 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfe0087010 con 0x7fcfe00863d0 2021-02-10T11:34:00.464 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:00.458 7fcfe519f700 1 -- 172.21.2.102:0/2257183556 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfe0083000 con 0x7fcfe00863d0 2021-02-10T11:34:00.464 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:00.458 7fcfd77fe700 1 -- 172.21.2.102:0/2257183556 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcfd0004720 con 0x7fcfe00863d0 2021-02-10T11:34:00.464 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:00.459 7fcfd77fe700 1 -- 172.21.2.102:0/2257183556 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fcfd0007340 con 0x7fcfe00863d0 2021-02-10T11:34:00.464 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:00.459 7fcfd77fe700 1 --2- 172.21.2.102:0/2257183556 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcfc4018730 0x7fcfc401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:00.464 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:00.459 7fcfd77fe700 1 -- 172.21.2.102:0/2257183556 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fcfd0032b70 con 0x7fcfe00863d0 2021-02-10T11:34:00.464 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:00.459 7fcfded9d700 1 --2- 172.21.2.102:0/2257183556 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcfc4018730 0x7fcfc401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:00.464 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:00.459 7fcfded9d700 1 --2- 172.21.2.102:0/2257183556 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcfc4018730 0x7fcfc401ac10 crc :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:00.493 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:00.490 7fcfe519f700 1 -- 172.21.2.102:0/2257183556 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfe0085e50 con 0x7fcfe00863d0 2021-02-10T11:34:00.494 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:00.493 7fcfd77fe700 1 -- 172.21.2.102:0/2257183556 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fcfd0001270 con 0x7fcfe00863d0 2021-02-10T11:34:00.619 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:00.617 7fcfe519f700 1 -- 172.21.2.102:0/2257183556 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000632662", "target": ["mgr", ""]}) v1 -- 0x7fcfe0083000 con 0x7fcfc4018730 2021-02-10T11:34:00.629 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:00.627 7fcfd77fe700 1 -- 172.21.2.102:0/2257183556 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fcfd8000940 con 0x7fcfc4018730 2021-02-10T11:34:00.636 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:00.634 7fcfe519f700 1 -- 172.21.2.102:0/2257183556 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcfc4018730 msgr2=0x7fcfc401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:00.636 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:00.634 7fcfe519f700 1 --2- 172.21.2.102:0/2257183556 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcfc4018730 0x7fcfc401ac10 crc :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:00.636 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:00.634 7fcfe519f700 1 -- 172.21.2.102:0/2257183556 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcfe00863d0 msgr2=0x7fcfe00867d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:00.636 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:00.634 7fcfe519f700 1 --2- 172.21.2.102:0/2257183556 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcfe00863d0 0x7fcfe00867d0 secure :-1 s=READY pgs=2487 cs=0 l=1 rev1=1 rx=0x7fcfd000a8f0 tx=0x7fcfd000ac90).stop 2021-02-10T11:34:00.636 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:00.634 7fcfe519f700 1 -- 172.21.2.102:0/2257183556 shutdown_connections 2021-02-10T11:34:00.637 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:00.634 7fcfe519f700 1 --2- 172.21.2.102:0/2257183556 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcfe00863d0 0x7fcfe00867d0 unknown :-1 s=CLOSED pgs=2487 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:00.637 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:00.634 7fcfe519f700 1 --2- 172.21.2.102:0/2257183556 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcfe01a44c0 0x7fcfe01a48c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:00.637 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:00.634 7fcfe519f700 1 --2- 172.21.2.102:0/2257183556 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcfc4018730 0x7fcfc401ac10 unknown :-1 s=CLOSED pgs=1101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:00.637 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:00.634 7fcfe519f700 1 --2- 172.21.2.102:0/2257183556 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fcfe00c3820 0x7fcfe00a7950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:00.637 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:00.634 7fcfe519f700 1 -- 172.21.2.102:0/2257183556 >> 172.21.2.102:0/2257183556 conn(0x7fcfe00872c0 msgr2=0x7fcfe0095700 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:00.637 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:00.634 7fcfe519f700 1 -- 172.21.2.102:0/2257183556 shutdown_connections 2021-02-10T11:34:00.637 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:00.634 7fcfe519f700 1 -- 172.21.2.102:0/2257183556 wait complete. 2021-02-10T11:34:00.646 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo getfattr --only-values -n ceph.dir.entries /home/ubuntu/cephtest/mnt.0/./volumes/_deleting 2021-02-10T11:34:00.996 INFO:teuthology.orchestra.run.gibba002.stderr:getfattr: Removing leading '/' from absolute path names 2021-02-10T11:34:00.997 INFO:teuthology.orchestra.run.gibba002.stdout:0 2021-02-10T11:34:00.997 DEBUG:tasks.cephfs.mount:Directory ./volumes/_deleting seen empty from 0 after 0s 2021-02-10T11:34:00.997 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2021-02-10T11:34:00.997 DEBUG:teuthology.orchestra.run.gibba002:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:34:01.343 DEBUG:teuthology.orchestra.run.gibba013:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:34:01.688 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2021-02-10T11:34:01.688 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:34:01.688 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:34:01.688 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:34:02.024 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:34:02.025 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:34:02.025 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@gibba002.front.sepia.ceph.com... 2021-02-10T11:34:02.025 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-02-10T11:34:02.025 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:34:02.378 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba002.stderr:ceph-fuse[189994]: fuse finished with error 0 and tester_r 0 2021-02-10T11:34:02.379 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:34:02.379 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:34:02.379 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:34:02.715 INFO:teuthology.orchestra.run.gibba002.stdout:ext2/ext3 2021-02-10T11:34:02.715 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2021-02-10T11:34:02.716 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:34:02.716 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:34:03.051 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:34:03.051 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:34:03.051 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:34:03.387 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:34:03.388 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2021-02-10T11:34:03.388 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@gibba002.front.sepia.ceph.com... 2021-02-10T11:34:03.388 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-02-10T11:34:03.388 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:34:03.744 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba002.stderr:ceph-fuse[190615]: fuse finished with error 0 and tester_r 0 2021-02-10T11:34:03.745 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:34:03.745 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:34:03.745 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:34:04.079 INFO:teuthology.orchestra.run.gibba002.stdout:ext2/ext3 2021-02-10T11:34:04.080 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2021-02-10T11:34:04.080 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:34:04.080 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:34:04.415 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:34:04.415 DEBUG:teuthology.orchestra.run.gibba013:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:34:04.415 DEBUG:teuthology.orchestra.run.gibba013:> ' -- /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:34:04.751 INFO:teuthology.orchestra.run.gibba013.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.2’: No such file or directory 2021-02-10T11:34:04.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:34:04.752 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2021-02-10T11:34:04.752 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:34:04.752 DEBUG:teuthology.orchestra.run.gibba013:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:34:05.088 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:34:05.088 DEBUG:teuthology.orchestra.run.gibba013:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:34:05.088 DEBUG:teuthology.orchestra.run.gibba013:> ' -- /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:34:05.422 INFO:teuthology.orchestra.run.gibba013.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.3’: No such file or directory 2021-02-10T11:34:05.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:34:05.423 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.3 2021-02-10T11:34:05.423 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:34:05.423 DEBUG:teuthology.orchestra.run.gibba013:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:34:05.757 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config rm mon mon_allow_pool_delete 2021-02-10T11:34:06.191 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:06.173 7f6449213700 1 Processor -- start 2021-02-10T11:34:06.191 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:06.173 7f6449213700 1 -- start start 2021-02-10T11:34:06.191 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:06.173 7f6449213700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f64440d0340 0x7f64440ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:06.192 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:06.173 7f6449213700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f64440d1c70 0x7f64440efc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:06.192 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:06.173 7f6449213700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64440f0b10 con 0x7f64440d1870 2021-02-10T11:34:06.192 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:06.173 7f6449213700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64440e5ed0 con 0x7f64440d1c70 2021-02-10T11:34:06.192 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:06.173 7f6449213700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64441b0be0 con 0x7f64440d0340 2021-02-10T11:34:06.192 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:06.173 7f6442d9d700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f64440d1870 0x7f64441b03f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:56234/0 (socket says 172.21.2.102:56234) 2021-02-10T11:34:06.192 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:06.173 7f6442d9d700 1 -- 172.21.2.102:0/3858163892 learned_addr learned my addr 172.21.2.102:0/3858163892 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:34:06.192 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:06.174 7f644359e700 1 --2- 172.21.2.102:0/3858163892 >> v2:172.21.2.113:3300/0 conn(0x7f64440d1c70 0x7f64440efc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:06.192 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:06.174 7f644259c700 1 --2- 172.21.2.102:0/3858163892 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f64440d0340 0x7f64440ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:06.192 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:06.174 7f6441d9b700 1 -- 172.21.2.102:0/3858163892 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 955561978 0 0) 0x7f6434001180 con 0x7f64440d1870 2021-02-10T11:34:06.192 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:06.174 7f6441d9b700 1 -- 172.21.2.102:0/3858163892 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f642c002fe0 con 0x7f64440d1870 2021-02-10T11:34:06.192 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:06.174 7f644259c700 1 -- 172.21.2.102:0/3858163892 >> v2:172.21.2.113:3300/0 conn(0x7f64440d1c70 msgr2=0x7f64440efc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:34:06.193 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:06.174 7f644259c700 1 --2- 172.21.2.102:0/3858163892 >> v2:172.21.2.113:3300/0 conn(0x7f64440d1c70 0x7f64440efc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:06.193 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:06.174 7f644259c700 1 -- 172.21.2.102:0/3858163892 >> v1:172.21.2.102:6789/0 conn(0x7f64440d1870 legacy=0x7f64441b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:06.193 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:06.174 7f644259c700 1 -- 172.21.2.102:0/3858163892 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64440f20a0 con 0x7f64440d0340 2021-02-10T11:34:06.193 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:06.174 7f644259c700 1 --2- 172.21.2.102:0/3858163892 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f64440d0340 0x7f64440ef640 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 rx=0x7f6438006650 tx=0x7f6438009a80).ready entity=mon.2 client_cookie=b57666058a71156a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:06.193 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:06.174 7f6441d9b700 1 -- 172.21.2.102:0/3858163892 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f643800c0c0 con 0x7f64440d0340 2021-02-10T11:34:06.193 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:06.174 7f6441d9b700 1 -- 172.21.2.102:0/3858163892 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f643800c340 con 0x7f64440d0340 2021-02-10T11:34:06.193 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:06.174 7f6441d9b700 1 -- 172.21.2.102:0/3858163892 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f643800c7a0 con 0x7f64440d0340 2021-02-10T11:34:06.193 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:06.174 7f6449213700 1 -- 172.21.2.102:0/3858163892 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f64440d0340 msgr2=0x7f64440ef640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:06.193 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:06.174 7f6449213700 1 --2- 172.21.2.102:0/3858163892 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f64440d0340 0x7f64440ef640 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 rx=0x7f6438006650 tx=0x7f6438009a80).stop 2021-02-10T11:34:06.193 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:06.175 7f6449213700 1 -- 172.21.2.102:0/3858163892 shutdown_connections 2021-02-10T11:34:06.193 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:06.175 7f6449213700 1 --2- 172.21.2.102:0/3858163892 >> v2:172.21.2.113:3300/0 conn(0x7f64440d1c70 0x7f64440efc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:06.194 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:06.175 7f6449213700 1 --2- 172.21.2.102:0/3858163892 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f64440d0340 0x7f64440ef640 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:06.194 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:06.175 7f6449213700 1 -- 172.21.2.102:0/3858163892 >> 172.21.2.102:0/3858163892 conn(0x7f64440ce450 msgr2=0x7f64440d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:06.194 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:06.175 7f6449213700 1 -- 172.21.2.102:0/3858163892 shutdown_connections 2021-02-10T11:34:06.194 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:06.175 7f6449213700 1 -- 172.21.2.102:0/3858163892 wait complete. 2021-02-10T11:34:06.196 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:06.190 7f6449213700 1 Processor -- start 2021-02-10T11:34:06.196 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:06.190 7f6449213700 1 -- start start 2021-02-10T11:34:06.196 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:06.190 7f6449213700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f64440b41d0 0x7f64440b45d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:06.196 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:06.190 7f6449213700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f64440b4bf0 0x7f64440b4ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:06.196 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:06.190 7f6449213700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f64440ee600 0x7f64440eea00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:06.196 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:06.190 7f6449213700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64440b5690 con 0x7f64440ee600 2021-02-10T11:34:06.197 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:06.190 7f6449213700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64440c3dc0 con 0x7f64440b4bf0 2021-02-10T11:34:06.197 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:06.190 7f6449213700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64441a49c0 con 0x7f64440b41d0 2021-02-10T11:34:06.197 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:06.191 7f644359e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f64440ee600 0x7f64440eea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:06.197 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:06.191 7f644359e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f64440ee600 0x7f64440eea00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:35788/0 (socket says 172.21.2.102:35788) 2021-02-10T11:34:06.197 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:06.191 7f644359e700 1 -- 172.21.2.102:0/1344155332 learned_addr learned my addr 172.21.2.102:0/1344155332 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:34:06.197 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:06.191 7f644259c700 1 --2- 172.21.2.102:0/1344155332 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f64440b4bf0 0x7f64440b4ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:06.197 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:06.191 7f6442d9d700 1 --2- 172.21.2.102:0/1344155332 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f64440b41d0 0x7f64440b45d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:06.197 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:06.191 7f644359e700 1 -- 172.21.2.102:0/1344155332 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f64440b41d0 msgr2=0x7f64440b45d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:06.197 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:06.191 7f644359e700 1 --2- 172.21.2.102:0/1344155332 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f64440b41d0 0x7f64440b45d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:06.198 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:06.191 7f644359e700 1 -- 172.21.2.102:0/1344155332 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f64440b4bf0 msgr2=0x7f64440b4ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:06.198 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:06.191 7f644359e700 1 --2- 172.21.2.102:0/1344155332 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f64440b4bf0 0x7f64440b4ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:06.198 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:06.191 7f644359e700 1 -- 172.21.2.102:0/1344155332 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64440ef020 con 0x7f64440ee600 2021-02-10T11:34:06.198 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:06.191 7f644359e700 1 --2- 172.21.2.102:0/1344155332 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f64440ee600 0x7f64440eea00 secure :-1 s=READY pgs=2489 cs=0 l=1 rev1=1 rx=0x7f643c00b970 tx=0x7f643c00bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:06.198 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:06.191 7f64337fe700 1 -- 172.21.2.102:0/1344155332 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f643c000f90 con 0x7f64440ee600 2021-02-10T11:34:06.198 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:06.191 7f64337fe700 1 -- 172.21.2.102:0/1344155332 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f643c001400 con 0x7f64440ee600 2021-02-10T11:34:06.198 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:06.191 7f64337fe700 1 -- 172.21.2.102:0/1344155332 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f643c001860 con 0x7f64440ee600 2021-02-10T11:34:06.198 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:06.191 7f6449213700 1 -- 172.21.2.102:0/1344155332 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64440ef020 con 0x7f64440ee600 2021-02-10T11:34:06.198 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:06.191 7f6449213700 1 -- 172.21.2.102:0/1344155332 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64440ef5b0 con 0x7f64440ee600 2021-02-10T11:34:06.198 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:06.192 7f64337fe700 1 -- 172.21.2.102:0/1344155332 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f643c0015e0 con 0x7f64440ee600 2021-02-10T11:34:06.198 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:06.192 7f64337fe700 1 --2- 172.21.2.102:0/1344155332 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f64280186c0 0x7f642801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:06.198 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:06.192 7f64337fe700 1 -- 172.21.2.102:0/1344155332 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f643c032900 con 0x7f64440ee600 2021-02-10T11:34:06.199 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:06.192 7f6442d9d700 1 --2- 172.21.2.102:0/1344155332 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f64280186c0 0x7f642801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:06.199 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:06.192 7f6442d9d700 1 --2- 172.21.2.102:0/1344155332 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f64280186c0 0x7f642801abc0 crc :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:06.225 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:06.223 7f6449213700 1 -- 172.21.2.102:0/1344155332 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64440f0b10 con 0x7f64440ee600 2021-02-10T11:34:06.227 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:06.226 7f64337fe700 1 -- 172.21.2.102:0/1344155332 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f643c000f20 con 0x7f64440ee600 2021-02-10T11:34:06.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:06.375 7f6449213700 1 -- 172.21.2.102:0/1344155332 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f64440f0b10 con 0x7f64440ee600 2021-02-10T11:34:06.380 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:06.379 7f64337fe700 1 -- 172.21.2.102:0/1344155332 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v33) v1 ==== 104+0+0 (secure 0 0 0) 0x7f643c0015e0 con 0x7f64440ee600 2021-02-10T11:34:06.385 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:06.383 7f6449213700 1 -- 172.21.2.102:0/1344155332 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f64280186c0 msgr2=0x7f642801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:06.386 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:06.383 7f6449213700 1 --2- 172.21.2.102:0/1344155332 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f64280186c0 0x7f642801abc0 crc :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:06.386 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:06.384 7f6449213700 1 -- 172.21.2.102:0/1344155332 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f64440ee600 msgr2=0x7f64440eea00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:06.386 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:06.384 7f6449213700 1 --2- 172.21.2.102:0/1344155332 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f64440ee600 0x7f64440eea00 secure :-1 s=READY pgs=2489 cs=0 l=1 rev1=1 rx=0x7f643c00b970 tx=0x7f643c00bd10).stop 2021-02-10T11:34:06.386 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:06.384 7f6449213700 1 -- 172.21.2.102:0/1344155332 shutdown_connections 2021-02-10T11:34:06.386 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:06.384 7f6449213700 1 --2- 172.21.2.102:0/1344155332 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f64440ee600 0x7f64440eea00 unknown :-1 s=CLOSED pgs=2489 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:06.386 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:06.384 7f6449213700 1 --2- 172.21.2.102:0/1344155332 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f64280186c0 0x7f642801abc0 unknown :-1 s=CLOSED pgs=1102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:06.386 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:06.384 7f6449213700 1 --2- 172.21.2.102:0/1344155332 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f64440b4bf0 0x7f64440b4ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:06.386 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:06.384 7f6449213700 1 --2- 172.21.2.102:0/1344155332 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f64440b41d0 0x7f64440b45d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:06.386 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:06.384 7f6449213700 1 -- 172.21.2.102:0/1344155332 >> 172.21.2.102:0/1344155332 conn(0x7f64440b7070 msgr2=0x7f64440bf0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:06.386 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:06.384 7f6449213700 1 -- 172.21.2.102:0/1344155332 shutdown_connections 2021-02-10T11:34:06.387 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:06.384 7f6449213700 1 -- 172.21.2.102:0/1344155332 wait complete. 2021-02-10T11:34:06.394 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestVolumes.test_non_clone_status' 2021-02-10T11:34:06.821 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:06.803 7f11e71ac700 1 Processor -- start 2021-02-10T11:34:06.821 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:06.803 7f11e71ac700 1 -- start start 2021-02-10T11:34:06.821 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:06.803 7f11e71ac700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f11e01b0470 0x7f11e01b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:06.821 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:06.803 7f11e71ac700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f11e00ef610 0x7f11e00efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:06.821 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:06.803 7f11e71ac700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11e00f0b50 con 0x7f11e00f0030 2021-02-10T11:34:06.821 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:06.803 7f11e71ac700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11e00e52e0 con 0x7f11e01b0470 2021-02-10T11:34:06.821 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:06.803 7f11e71ac700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11e00e54d0 con 0x7f11e00ef610 2021-02-10T11:34:06.822 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:06.804 7f11e4fc8700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f11e01b0470 0x7f11e01b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:06.822 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:06.804 7f11e57c9700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f11e00f0030 0x7f11e00f10d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:56248/0 (socket says 172.21.2.102:56248) 2021-02-10T11:34:06.822 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:06.804 7f11e57c9700 1 -- 172.21.2.102:0/1776971819 learned_addr learned my addr 172.21.2.102:0/1776971819 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:34:06.822 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:06.804 7f11e4fc8700 1 --2- 172.21.2.102:0/1776971819 >> v2:172.21.2.102:3300/0 conn(0x7f11e01b0470 0x7f11e01b0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:35792/0 (socket says 172.21.2.102:35792) 2021-02-10T11:34:06.822 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:06.804 7f11dffff700 1 --2- 172.21.2.102:0/1776971819 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f11e00ef610 0x7f11e00efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:06.822 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:06.804 7f11e4fc8700 1 -- 172.21.2.102:0/1776971819 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f11e00ef610 msgr2=0x7f11e00efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:34:06.822 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:06.804 7f11e4fc8700 1 --2- 172.21.2.102:0/1776971819 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f11e00ef610 0x7f11e00efa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:06.822 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:06.804 7f11e4fc8700 1 -- 172.21.2.102:0/1776971819 >> v1:172.21.2.102:6789/0 conn(0x7f11e00f0030 legacy=0x7f11e00f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:06.822 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:06.804 7f11e4fc8700 1 -- 172.21.2.102:0/1776971819 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11e00f0430 con 0x7f11e01b0470 2021-02-10T11:34:06.822 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:06.804 7f11e4fc8700 1 --2- 172.21.2.102:0/1776971819 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f11e01b0470 0x7f11e01b0870 secure :-1 s=READY pgs=2491 cs=0 l=1 rev1=1 rx=0x7f11c8006650 tx=0x7f11c8009a80).ready entity=mon.0 client_cookie=5c2c5f2d5989728 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:06.822 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:06.804 7f11df7fe700 1 -- 172.21.2.102:0/1776971819 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f11c80009b0 con 0x7f11e01b0470 2021-02-10T11:34:06.823 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:06.804 7f11df7fe700 1 -- 172.21.2.102:0/1776971819 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11c800c0c0 con 0x7f11e01b0470 2021-02-10T11:34:06.823 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:06.804 7f11df7fe700 1 -- 172.21.2.102:0/1776971819 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f11c800c5e0 con 0x7f11e01b0470 2021-02-10T11:34:06.823 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:06.804 7f11e71ac700 1 -- 172.21.2.102:0/1776971819 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f11e01b0470 msgr2=0x7f11e01b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:06.823 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:06.804 7f11e71ac700 1 --2- 172.21.2.102:0/1776971819 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f11e01b0470 0x7f11e01b0870 secure :-1 s=READY pgs=2491 cs=0 l=1 rev1=1 rx=0x7f11c8006650 tx=0x7f11c8009a80).stop 2021-02-10T11:34:06.823 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:06.805 7f11e71ac700 1 -- 172.21.2.102:0/1776971819 shutdown_connections 2021-02-10T11:34:06.823 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:06.805 7f11e71ac700 1 --2- 172.21.2.102:0/1776971819 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f11e00ef610 0x7f11e00efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:06.823 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:06.805 7f11e71ac700 1 --2- 172.21.2.102:0/1776971819 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f11e01b0470 0x7f11e01b0870 unknown :-1 s=CLOSED pgs=2491 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:06.823 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:06.805 7f11e71ac700 1 -- 172.21.2.102:0/1776971819 >> 172.21.2.102:0/1776971819 conn(0x7f11e00ce3e0 msgr2=0x7f11e00e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:06.823 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:06.805 7f11e71ac700 1 -- 172.21.2.102:0/1776971819 shutdown_connections 2021-02-10T11:34:06.823 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:06.805 7f11e71ac700 1 -- 172.21.2.102:0/1776971819 wait complete. 2021-02-10T11:34:06.825 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:06.820 7f11e71ac700 1 Processor -- start 2021-02-10T11:34:06.825 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:06.820 7f11e71ac700 1 -- start start 2021-02-10T11:34:06.825 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:06.820 7f11e71ac700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f11e0093b00 0x7f11e01b0a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:06.825 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:06.820 7f11e71ac700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f11e01b1020 0x7f11e01a11b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:06.825 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:06.820 7f11e71ac700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f11e01a17d0 0x7f11e00922d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:06.825 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:06.820 7f11e71ac700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11e0093580 con 0x7f11e01a17d0 2021-02-10T11:34:06.825 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:06.820 7f11e71ac700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11e0093f00 con 0x7f11e0093b00 2021-02-10T11:34:06.825 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:06.820 7f11e71ac700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11e01a1bd0 con 0x7f11e01b1020 2021-02-10T11:34:06.825 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:06.820 7f11e57c9700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f11e01a17d0 0x7f11e00922d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:06.825 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:06.820 7f11e57c9700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f11e01a17d0 0x7f11e00922d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:35798/0 (socket says 172.21.2.102:35798) 2021-02-10T11:34:06.825 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:06.820 7f11e57c9700 1 -- 172.21.2.102:0/4085314313 learned_addr learned my addr 172.21.2.102:0/4085314313 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:34:06.826 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:06.821 7f11e57c9700 1 -- 172.21.2.102:0/4085314313 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f11e01b1020 msgr2=0x7f11e01a11b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:34:06.826 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:06.821 7f11e57c9700 1 --2- 172.21.2.102:0/4085314313 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f11e01b1020 0x7f11e01a11b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:06.826 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:06.821 7f11e57c9700 1 -- 172.21.2.102:0/4085314313 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f11e0093b00 msgr2=0x7f11e01b0a00 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:34:06.826 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:06.821 7f11e57c9700 1 --2- 172.21.2.102:0/4085314313 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f11e0093b00 0x7f11e01b0a00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:06.826 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:06.821 7f11e57c9700 1 -- 172.21.2.102:0/4085314313 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11e00928f0 con 0x7f11e01a17d0 2021-02-10T11:34:06.826 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:06.821 7f11e57c9700 1 --2- 172.21.2.102:0/4085314313 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f11e01a17d0 0x7f11e00922d0 secure :-1 s=READY pgs=2492 cs=0 l=1 rev1=1 rx=0x7f11d8001410 tx=0x7f11d800a950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:06.826 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:06.821 7f11dd7fa700 1 -- 172.21.2.102:0/4085314313 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f11d8000ef0 con 0x7f11e01a17d0 2021-02-10T11:34:06.826 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:06.821 7f11dd7fa700 1 -- 172.21.2.102:0/4085314313 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11d800ad40 con 0x7f11e01a17d0 2021-02-10T11:34:06.826 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:06.821 7f11dd7fa700 1 -- 172.21.2.102:0/4085314313 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f11d8004620 con 0x7f11e01a17d0 2021-02-10T11:34:06.826 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:06.821 7f11e71ac700 1 -- 172.21.2.102:0/4085314313 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11e00928f0 con 0x7f11e01a17d0 2021-02-10T11:34:06.826 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:06.821 7f11e71ac700 1 -- 172.21.2.102:0/4085314313 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11e0092e80 con 0x7f11e01a17d0 2021-02-10T11:34:06.826 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:06.822 7f11dd7fa700 1 -- 172.21.2.102:0/4085314313 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f11d80043a0 con 0x7f11e01a17d0 2021-02-10T11:34:06.826 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:06.822 7f11dd7fa700 1 --2- 172.21.2.102:0/4085314313 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f11c40186c0 0x7f11c401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:06.827 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:06.822 7f11dd7fa700 1 -- 172.21.2.102:0/4085314313 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f11d8032510 con 0x7f11e01a17d0 2021-02-10T11:34:06.827 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:06.822 7f11e4fc8700 1 --2- 172.21.2.102:0/4085314313 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f11c40186c0 0x7f11c401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:06.827 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:06.822 7f11e4fc8700 1 --2- 172.21.2.102:0/4085314313 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f11c40186c0 0x7f11c401abc0 crc :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:06.855 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:06.853 7f11e71ac700 1 -- 172.21.2.102:0/4085314313 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11e0093580 con 0x7f11e01a17d0 2021-02-10T11:34:06.857 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:06.856 7f11dd7fa700 1 -- 172.21.2.102:0/4085314313 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f11d801ec30 con 0x7f11e01a17d0 2021-02-10T11:34:06.988 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:06.987 7f11e71ac700 1 -- 172.21.2.102:0/4085314313 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestVolumes.test_non_clone_status"]} v 0) v1 -- 0x7f11e0093580 con 0x7f11e01a17d0 2021-02-10T11:34:07.132 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:07.130 7f11dd7fa700 1 -- 172.21.2.102:0/4085314313 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestVolumes.test_non_clone_status"]}]=0 v666) v1 ==== 138+0+0 (secure 0 0 0) 0x7f11d801ec30 con 0x7f11e01a17d0 2021-02-10T11:34:07.155 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:07.153 7f11e71ac700 1 -- 172.21.2.102:0/4085314313 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f11c40186c0 msgr2=0x7f11c401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:07.155 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:07.153 7f11e71ac700 1 --2- 172.21.2.102:0/4085314313 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f11c40186c0 0x7f11c401abc0 crc :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:07.156 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:07.153 7f11e71ac700 1 -- 172.21.2.102:0/4085314313 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f11e01a17d0 msgr2=0x7f11e00922d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:07.156 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:07.153 7f11e71ac700 1 --2- 172.21.2.102:0/4085314313 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f11e01a17d0 0x7f11e00922d0 secure :-1 s=READY pgs=2492 cs=0 l=1 rev1=1 rx=0x7f11d8001410 tx=0x7f11d800a950).stop 2021-02-10T11:34:07.156 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:07.153 7f11e71ac700 1 -- 172.21.2.102:0/4085314313 shutdown_connections 2021-02-10T11:34:07.156 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:07.153 7f11e71ac700 1 --2- 172.21.2.102:0/4085314313 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f11e01a17d0 0x7f11e00922d0 unknown :-1 s=CLOSED pgs=2492 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:07.156 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:07.153 7f11e71ac700 1 --2- 172.21.2.102:0/4085314313 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f11e01b1020 0x7f11e01a11b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:07.156 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:07.153 7f11e71ac700 1 --2- 172.21.2.102:0/4085314313 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f11c40186c0 0x7f11c401abc0 unknown :-1 s=CLOSED pgs=1103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:07.156 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:07.153 7f11e71ac700 1 --2- 172.21.2.102:0/4085314313 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f11e0093b00 0x7f11e01b0a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:07.156 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:07.154 7f11e71ac700 1 -- 172.21.2.102:0/4085314313 >> 172.21.2.102:0/4085314313 conn(0x7f11e01a04a0 msgr2=0x7f11e00916f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:07.156 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:07.154 7f11e71ac700 1 -- 172.21.2.102:0/4085314313 shutdown_connections 2021-02-10T11:34:07.156 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:07.154 7f11e71ac700 1 -- 172.21.2.102:0/4085314313 wait complete. 2021-02-10T11:34:07.163 INFO:tasks.cephfs_test_runner:test_non_clone_status (tasks.cephfs.test_volumes.TestVolumes) ... ok 2021-02-10T11:34:07.164 INFO:tasks.cephfs_test_runner:Starting test: test_nonexistent_subvolume_group_create (tasks.cephfs.test_volumes.TestVolumes) 2021-02-10T11:34:07.164 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestVolumes.test_nonexistent_subvolume_group_create' 2021-02-10T11:34:07.591 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:07.572 7fd8f1027700 1 Processor -- start 2021-02-10T11:34:07.591 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:07.572 7fd8f1027700 1 -- start start 2021-02-10T11:34:07.591 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:07.573 7fd8f1027700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fd8ec1b0470 0x7fd8ec1b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:07.591 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:07.573 7fd8f1027700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fd8ec0ef610 0x7fd8ec0efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:07.591 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:07.573 7fd8f1027700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8ec0f0b50 con 0x7fd8ec0f0030 2021-02-10T11:34:07.592 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:07.573 7fd8f1027700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8ec0e52e0 con 0x7fd8ec1b0470 2021-02-10T11:34:07.592 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:07.573 7fd8f1027700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8ec0e5490 con 0x7fd8ec0ef610 2021-02-10T11:34:07.592 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:07.573 7fd8ead9d700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fd8ec1b0470 0x7fd8ec1b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:07.592 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:07.573 7fd8ead9d700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fd8ec1b0470 0x7fd8ec1b0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:35804/0 (socket says 172.21.2.102:35804) 2021-02-10T11:34:07.592 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:07.573 7fd8ead9d700 1 -- 172.21.2.102:0/500395478 learned_addr learned my addr 172.21.2.102:0/500395478 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:34:07.592 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:07.573 7fd8ead9d700 1 -- 172.21.2.102:0/500395478 >> v2:172.21.2.113:3300/0 conn(0x7fd8ec0ef610 msgr2=0x7fd8ec0efa10 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:34:07.592 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:07.573 7fd8ead9d700 1 --2- 172.21.2.102:0/500395478 >> v2:172.21.2.113:3300/0 conn(0x7fd8ec0ef610 0x7fd8ec0efa10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:07.592 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:07.573 7fd8ead9d700 1 -- 172.21.2.102:0/500395478 >> v1:172.21.2.113:6789/0 conn(0x7fd8ec0f0030 legacy=0x7fd8ec0f10d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:34:07.592 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:07.573 7fd8ead9d700 1 -- 172.21.2.102:0/500395478 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8ec0f0430 con 0x7fd8ec1b0470 2021-02-10T11:34:07.592 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:07.574 7fd8ead9d700 1 --2- 172.21.2.102:0/500395478 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd8ec1b0470 0x7fd8ec1b0870 secure :-1 s=READY pgs=2493 cs=0 l=1 rev1=1 rx=0x7fd8dc006700 tx=0x7fd8dc009b80).ready entity=mon.0 client_cookie=91951841e08529f4 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:07.593 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:07.574 7fd8e9d9b700 1 -- 172.21.2.102:0/500395478 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd8dc00c0c0 con 0x7fd8ec1b0470 2021-02-10T11:34:07.593 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:07.574 7fd8e9d9b700 1 -- 172.21.2.102:0/500395478 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8dc00c400 con 0x7fd8ec1b0470 2021-02-10T11:34:07.593 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:07.574 7fd8e9d9b700 1 -- 172.21.2.102:0/500395478 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd8dc00c920 con 0x7fd8ec1b0470 2021-02-10T11:34:07.593 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:07.574 7fd8f1027700 1 -- 172.21.2.102:0/500395478 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd8ec1b0470 msgr2=0x7fd8ec1b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:07.593 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:07.574 7fd8f1027700 1 --2- 172.21.2.102:0/500395478 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd8ec1b0470 0x7fd8ec1b0870 secure :-1 s=READY pgs=2493 cs=0 l=1 rev1=1 rx=0x7fd8dc006700 tx=0x7fd8dc009b80).stop 2021-02-10T11:34:07.593 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:07.575 7fd8f1027700 1 -- 172.21.2.102:0/500395478 shutdown_connections 2021-02-10T11:34:07.593 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:07.575 7fd8f1027700 1 --2- 172.21.2.102:0/500395478 >> v2:172.21.2.113:3300/0 conn(0x7fd8ec0ef610 0x7fd8ec0efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:07.593 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:07.575 7fd8f1027700 1 --2- 172.21.2.102:0/500395478 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd8ec1b0470 0x7fd8ec1b0870 unknown :-1 s=CLOSED pgs=2493 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:07.593 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:07.575 7fd8f1027700 1 -- 172.21.2.102:0/500395478 >> 172.21.2.102:0/500395478 conn(0x7fd8ec0ce3e0 msgr2=0x7fd8ec0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:07.593 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:07.575 7fd8f1027700 1 -- 172.21.2.102:0/500395478 shutdown_connections 2021-02-10T11:34:07.593 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:07.575 7fd8f1027700 1 -- 172.21.2.102:0/500395478 wait complete. 2021-02-10T11:34:07.593 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:07.589 7fd8f1027700 1 Processor -- start 2021-02-10T11:34:07.593 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:07.589 7fd8f1027700 1 -- start start 2021-02-10T11:34:07.595 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:07.590 7fd8f1027700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd8ec0b0970 0x7fd8ec092ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:07.595 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:07.590 7fd8f1027700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd8ec0934f0 0x7fd8ec094dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:07.595 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:07.590 7fd8f1027700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd8ec0938f0 0x7fd8ec0953e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:07.595 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:07.590 7fd8f1027700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8ec195ad0 con 0x7fd8ec0934f0 2021-02-10T11:34:07.595 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:07.590 7fd8f1027700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8ec095f40 con 0x7fd8ec0938f0 2021-02-10T11:34:07.595 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:07.590 7fd8f1027700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8ec095a30 con 0x7fd8ec0b0970 2021-02-10T11:34:07.595 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:07.590 7fd8ea59c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd8ec0934f0 0x7fd8ec094dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:07.595 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:07.590 7fd8ea59c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd8ec0934f0 0x7fd8ec094dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:35810/0 (socket says 172.21.2.102:35810) 2021-02-10T11:34:07.595 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:07.590 7fd8ea59c700 1 -- 172.21.2.102:0/1767538912 learned_addr learned my addr 172.21.2.102:0/1767538912 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:34:07.596 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:07.590 7fd8ea59c700 1 -- 172.21.2.102:0/1767538912 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd8ec0b0970 msgr2=0x7fd8ec092ed0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:34:07.596 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:07.590 7fd8ea59c700 1 --2- 172.21.2.102:0/1767538912 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd8ec0b0970 0x7fd8ec092ed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:07.596 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:07.590 7fd8ea59c700 1 -- 172.21.2.102:0/1767538912 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd8ec0938f0 msgr2=0x7fd8ec0953e0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:34:07.596 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:07.590 7fd8ea59c700 1 --2- 172.21.2.102:0/1767538912 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd8ec0938f0 0x7fd8ec0953e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:07.596 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:07.590 7fd8ea59c700 1 -- 172.21.2.102:0/1767538912 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8ec095c50 con 0x7fd8ec0934f0 2021-02-10T11:34:07.596 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:07.590 7fd8ea59c700 1 --2- 172.21.2.102:0/1767538912 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd8ec0934f0 0x7fd8ec094dc0 secure :-1 s=READY pgs=2494 cs=0 l=1 rev1=1 rx=0x7fd8d400b5c0 tx=0x7fd8d400b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:07.596 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:07.591 7fd8e37fe700 1 -- 172.21.2.102:0/1767538912 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd8d4000c40 con 0x7fd8ec0934f0 2021-02-10T11:34:07.596 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:07.591 7fd8e37fe700 1 -- 172.21.2.102:0/1767538912 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8d400bd50 con 0x7fd8ec0934f0 2021-02-10T11:34:07.596 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:07.591 7fd8f1027700 1 -- 172.21.2.102:0/1767538912 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8ec095c50 con 0x7fd8ec0934f0 2021-02-10T11:34:07.596 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:07.591 7fd8f1027700 1 -- 172.21.2.102:0/1767538912 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8ec095c50 con 0x7fd8ec0934f0 2021-02-10T11:34:07.596 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:07.591 7fd8e37fe700 1 -- 172.21.2.102:0/1767538912 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd8d40054d0 con 0x7fd8ec0934f0 2021-02-10T11:34:07.596 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:07.591 7fd8e37fe700 1 -- 172.21.2.102:0/1767538912 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd8d4006240 con 0x7fd8ec0934f0 2021-02-10T11:34:07.597 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:07.591 7fd8e37fe700 1 --2- 172.21.2.102:0/1767538912 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd8d0018730 0x7fd8d001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:07.597 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:07.591 7fd8e37fe700 1 -- 172.21.2.102:0/1767538912 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fd8d40327b0 con 0x7fd8ec0934f0 2021-02-10T11:34:07.597 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:07.591 7fd8ead9d700 1 --2- 172.21.2.102:0/1767538912 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd8d0018730 0x7fd8d001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:07.597 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:07.592 7fd8ead9d700 1 --2- 172.21.2.102:0/1767538912 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd8d0018730 0x7fd8d001ac10 crc :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:07.625 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:07.623 7fd8f1027700 1 -- 172.21.2.102:0/1767538912 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8ec0e52e0 con 0x7fd8ec0934f0 2021-02-10T11:34:07.627 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:07.626 7fd8e37fe700 1 -- 172.21.2.102:0/1767538912 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd8d4006030 con 0x7fd8ec0934f0 2021-02-10T11:34:07.758 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:07.756 7fd8f1027700 1 -- 172.21.2.102:0/1767538912 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestVolumes.test_nonexistent_subvolume_group_create"]} v 0) v1 -- 0x7fd8ec0e52e0 con 0x7fd8ec0934f0 2021-02-10T11:34:08.137 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:08.135 7fd8e37fe700 1 -- 172.21.2.102:0/1767538912 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestVolumes.test_nonexistent_subvolume_group_create"]}]=0 v667) v1 ==== 159+0+0 (secure 0 0 0) 0x7fd8d4006240 con 0x7fd8ec0934f0 2021-02-10T11:34:08.177 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:08.175 7fd8f1027700 1 -- 172.21.2.102:0/1767538912 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd8d0018730 msgr2=0x7fd8d001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:08.177 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:08.175 7fd8f1027700 1 --2- 172.21.2.102:0/1767538912 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd8d0018730 0x7fd8d001ac10 crc :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:08.177 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:08.175 7fd8f1027700 1 -- 172.21.2.102:0/1767538912 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd8ec0934f0 msgr2=0x7fd8ec094dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:08.177 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:08.175 7fd8f1027700 1 --2- 172.21.2.102:0/1767538912 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd8ec0934f0 0x7fd8ec094dc0 secure :-1 s=READY pgs=2494 cs=0 l=1 rev1=1 rx=0x7fd8d400b5c0 tx=0x7fd8d400b960).stop 2021-02-10T11:34:08.177 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:08.175 7fd8f1027700 1 -- 172.21.2.102:0/1767538912 shutdown_connections 2021-02-10T11:34:08.178 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:08.175 7fd8f1027700 1 --2- 172.21.2.102:0/1767538912 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd8d0018730 0x7fd8d001ac10 unknown :-1 s=CLOSED pgs=1104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:08.178 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:08.175 7fd8f1027700 1 --2- 172.21.2.102:0/1767538912 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd8ec0938f0 0x7fd8ec0953e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:08.178 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:08.175 7fd8f1027700 1 --2- 172.21.2.102:0/1767538912 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd8ec0934f0 0x7fd8ec094dc0 unknown :-1 s=CLOSED pgs=2494 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:08.178 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:08.175 7fd8f1027700 1 --2- 172.21.2.102:0/1767538912 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd8ec0b0970 0x7fd8ec092ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:08.178 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:08.175 7fd8f1027700 1 -- 172.21.2.102:0/1767538912 >> 172.21.2.102:0/1767538912 conn(0x7fd8ec19f7a0 msgr2=0x7fd8ec0940e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:08.178 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:08.175 7fd8f1027700 1 -- 172.21.2.102:0/1767538912 shutdown_connections 2021-02-10T11:34:08.178 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:08.175 7fd8f1027700 1 -- 172.21.2.102:0/1767538912 wait complete. 2021-02-10T11:34:08.185 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config set mon mon_allow_pool_delete True 2021-02-10T11:34:08.616 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:08.597 7fcf0c780700 1 Processor -- start 2021-02-10T11:34:08.616 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:08.597 7fcf0c780700 1 -- start start 2021-02-10T11:34:08.616 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:08.598 7fcf0c780700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fcf040d1c70 0x7fcf040cef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:08.616 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:08.598 7fcf0c780700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf040d0170 con 0x7fcf040cf580 2021-02-10T11:34:08.616 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:08.598 7fcf0c780700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf040fe1c0 con 0x7fcf040d1870 2021-02-10T11:34:08.616 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:08.598 7fcf0c780700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf041b0cb0 con 0x7fcf040d1c70 2021-02-10T11:34:08.616 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:08.598 7fcf09d9b700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fcf040d1c70 0x7fcf040cef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:08.616 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:08.598 7fcf09d9b700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fcf040d1c70 0x7fcf040cef60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:35818/0 (socket says 172.21.2.102:35818) 2021-02-10T11:34:08.617 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:08.598 7fcf0ad9d700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fcf040cf580 0x7fcf040cf980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:56278/0 (socket says 172.21.2.102:56278) 2021-02-10T11:34:08.617 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:08.598 7fcf09d9b700 1 -- 172.21.2.102:0/2015231263 learned_addr learned my addr 172.21.2.102:0/2015231263 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:34:08.617 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:08.598 7fcf09d9b700 1 -- 172.21.2.102:0/2015231263 >> v1:172.21.2.113:6789/0 conn(0x7fcf040d1870 legacy=0x7fcf041b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:34:08.617 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:08.598 7fcf09d9b700 1 -- 172.21.2.102:0/2015231263 >> v1:172.21.2.102:6789/0 conn(0x7fcf040cf580 legacy=0x7fcf040cf980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:08.617 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:08.598 7fcf09d9b700 1 -- 172.21.2.102:0/2015231263 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf04109a60 con 0x7fcf040d1c70 2021-02-10T11:34:08.617 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:08.598 7fcf0959a700 1 -- 172.21.2.102:0/2015231263 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 592385452 0 0) 0x7fcefc001480 con 0x7fcf040cf580 2021-02-10T11:34:08.617 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:08.599 7fcf09d9b700 1 --2- 172.21.2.102:0/2015231263 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcf040d1c70 0x7fcf040cef60 secure :-1 s=READY pgs=2496 cs=0 l=1 rev1=1 rx=0x7fcef4006700 tx=0x7fcef4009b80).ready entity=mon.0 client_cookie=fc928322fb78a654 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:08.617 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:08.599 7fcf0959a700 1 -- 172.21.2.102:0/2015231263 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcef400c0c0 con 0x7fcf040d1c70 2021-02-10T11:34:08.617 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:08.599 7fcf0959a700 1 -- 172.21.2.102:0/2015231263 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcef400c440 con 0x7fcf040d1c70 2021-02-10T11:34:08.617 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:08.599 7fcf0959a700 1 -- 172.21.2.102:0/2015231263 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcef400c960 con 0x7fcf040d1c70 2021-02-10T11:34:08.617 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:08.599 7fcf0c780700 1 -- 172.21.2.102:0/2015231263 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcf040d1c70 msgr2=0x7fcf040cef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:08.617 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:08.599 7fcf0c780700 1 --2- 172.21.2.102:0/2015231263 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcf040d1c70 0x7fcf040cef60 secure :-1 s=READY pgs=2496 cs=0 l=1 rev1=1 rx=0x7fcef4006700 tx=0x7fcef4009b80).stop 2021-02-10T11:34:08.618 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:08.600 7fcf0c780700 1 -- 172.21.2.102:0/2015231263 shutdown_connections 2021-02-10T11:34:08.618 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:08.600 7fcf0c780700 1 --2- 172.21.2.102:0/2015231263 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcf040d1c70 0x7fcf040cef60 unknown :-1 s=CLOSED pgs=2496 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:08.618 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:08.600 7fcf0c780700 1 -- 172.21.2.102:0/2015231263 >> 172.21.2.102:0/2015231263 conn(0x7fcf040ce420 msgr2=0x7fcf040d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:08.618 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:08.600 7fcf0c780700 1 -- 172.21.2.102:0/2015231263 shutdown_connections 2021-02-10T11:34:08.618 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:08.600 7fcf0c780700 1 -- 172.21.2.102:0/2015231263 wait complete. 2021-02-10T11:34:08.618 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:08.614 7fcf0c780700 1 Processor -- start 2021-02-10T11:34:08.618 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:08.614 7fcf0c780700 1 -- start start 2021-02-10T11:34:08.619 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:08.615 7fcf0c780700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fcf040c3820 0x7fcf04084cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:08.619 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:08.615 7fcf0c780700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcf040a0290 0x7fcf040852d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:08.620 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:08.615 7fcf0c780700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcf04086270 0x7fcf04086670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:08.620 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:08.615 7fcf0c780700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf040c0700 con 0x7fcf04086270 2021-02-10T11:34:08.620 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:08.615 7fcf0c780700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf040858f0 con 0x7fcf040c3820 2021-02-10T11:34:08.620 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:08.615 7fcf0c780700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf04085b10 con 0x7fcf040a0290 2021-02-10T11:34:08.620 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:08.615 7fcf0ad9d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcf04086270 0x7fcf04086670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:08.620 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:08.615 7fcf0ad9d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcf04086270 0x7fcf04086670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:35826/0 (socket says 172.21.2.102:35826) 2021-02-10T11:34:08.620 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:08.615 7fcf0ad9d700 1 -- 172.21.2.102:0/2300621659 learned_addr learned my addr 172.21.2.102:0/2300621659 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:34:08.620 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:08.615 7fcf0ad9d700 1 -- 172.21.2.102:0/2300621659 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcf040a0290 msgr2=0x7fcf040852d0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:34:08.620 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:08.615 7fcf0ad9d700 1 --2- 172.21.2.102:0/2300621659 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcf040a0290 0x7fcf040852d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:08.620 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:08.615 7fcf0ad9d700 1 -- 172.21.2.102:0/2300621659 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fcf040c3820 msgr2=0x7fcf04084cb0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:34:08.620 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:08.615 7fcf0ad9d700 1 --2- 172.21.2.102:0/2300621659 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fcf040c3820 0x7fcf04084cb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:08.620 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:08.615 7fcf0ad9d700 1 -- 172.21.2.102:0/2300621659 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf04086c90 con 0x7fcf04086270 2021-02-10T11:34:08.620 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:08.615 7fcf0ad9d700 1 --2- 172.21.2.102:0/2300621659 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcf04086270 0x7fcf04086670 secure :-1 s=READY pgs=2497 cs=0 l=1 rev1=1 rx=0x7fcefc00a8f0 tx=0x7fcefc00ac90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:08.621 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:08.615 7fcefaffd700 1 -- 172.21.2.102:0/2300621659 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcefc007d50 con 0x7fcf04086270 2021-02-10T11:34:08.621 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:08.615 7fcefaffd700 1 -- 172.21.2.102:0/2300621659 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcefc0070d0 con 0x7fcf04086270 2021-02-10T11:34:08.621 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:08.615 7fcefaffd700 1 -- 172.21.2.102:0/2300621659 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcefc007530 con 0x7fcf04086270 2021-02-10T11:34:08.621 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:08.615 7fcf0c780700 1 -- 172.21.2.102:0/2300621659 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf04086c90 con 0x7fcf04086270 2021-02-10T11:34:08.621 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:08.616 7fcf0c780700 1 -- 172.21.2.102:0/2300621659 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf04086c90 con 0x7fcf04086270 2021-02-10T11:34:08.621 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:08.616 7fcefaffd700 1 -- 172.21.2.102:0/2300621659 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fcefc000e20 con 0x7fcf04086270 2021-02-10T11:34:08.621 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:08.616 7fcefaffd700 1 --2- 172.21.2.102:0/2300621659 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fceec0186c0 0x7fceec01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:08.621 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:08.616 7fcefaffd700 1 -- 172.21.2.102:0/2300621659 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fcefc032760 con 0x7fcf04086270 2021-02-10T11:34:08.621 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:08.616 7fcf0a59c700 1 --2- 172.21.2.102:0/2300621659 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fceec0186c0 0x7fceec01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:08.621 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:08.617 7fcf0a59c700 1 --2- 172.21.2.102:0/2300621659 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fceec0186c0 0x7fceec01abc0 crc :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:08.649 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:08.647 7fcf0c780700 1 -- 172.21.2.102:0/2300621659 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf04082a70 con 0x7fcf04086270 2021-02-10T11:34:08.652 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:08.650 7fcefaffd700 1 -- 172.21.2.102:0/2300621659 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fcefc026fc0 con 0x7fcf04086270 2021-02-10T11:34:08.800 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:08.798 7fcf0c780700 1 -- 172.21.2.102:0/2300621659 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fcf04082a70 con 0x7fcf04086270 2021-02-10T11:34:08.804 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:08.802 7fcefaffd700 1 -- 172.21.2.102:0/2300621659 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v34)=0 v34) v1 ==== 122+0+0 (secure 0 0 0) 0x7fcefc026fc0 con 0x7fcf04086270 2021-02-10T11:34:08.809 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:08.807 7fcf0c780700 1 -- 172.21.2.102:0/2300621659 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fceec0186c0 msgr2=0x7fceec01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:08.809 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:08.807 7fcf0c780700 1 --2- 172.21.2.102:0/2300621659 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fceec0186c0 0x7fceec01abc0 crc :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:08.809 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:08.807 7fcf0c780700 1 -- 172.21.2.102:0/2300621659 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcf04086270 msgr2=0x7fcf04086670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:08.809 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:08.807 7fcf0c780700 1 --2- 172.21.2.102:0/2300621659 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcf04086270 0x7fcf04086670 secure :-1 s=READY pgs=2497 cs=0 l=1 rev1=1 rx=0x7fcefc00a8f0 tx=0x7fcefc00ac90).stop 2021-02-10T11:34:08.810 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:08.808 7fcf0c780700 1 -- 172.21.2.102:0/2300621659 shutdown_connections 2021-02-10T11:34:08.810 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:08.808 7fcf0c780700 1 --2- 172.21.2.102:0/2300621659 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcf04086270 0x7fcf04086670 unknown :-1 s=CLOSED pgs=2497 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:08.810 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:08.808 7fcf0c780700 1 --2- 172.21.2.102:0/2300621659 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcf040a0290 0x7fcf040852d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:08.810 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:08.808 7fcf0c780700 1 --2- 172.21.2.102:0/2300621659 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fceec0186c0 0x7fceec01abc0 unknown :-1 s=CLOSED pgs=1105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:08.810 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:08.808 7fcf0c780700 1 --2- 172.21.2.102:0/2300621659 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fcf040c3820 0x7fcf04084cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:08.810 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:08.808 7fcf0c780700 1 -- 172.21.2.102:0/2300621659 >> 172.21.2.102:0/2300621659 conn(0x7fcf040872c0 msgr2=0x7fcf04095700 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:08.810 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:08.808 7fcf0c780700 1 -- 172.21.2.102:0/2300621659 shutdown_connections 2021-02-10T11:34:08.811 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:08.808 7fcf0c780700 1 -- 172.21.2.102:0/2300621659 wait complete. 2021-02-10T11:34:08.818 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2021-02-10T11:34:08.818 DEBUG:teuthology.orchestra.run.gibba002:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:34:09.163 DEBUG:teuthology.orchestra.run.gibba013:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:34:09.508 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2021-02-10T11:34:09.508 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:34:09.509 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:34:09.509 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:34:09.843 INFO:teuthology.orchestra.run.gibba002.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.0’: No such file or directory 2021-02-10T11:34:09.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:34:09.844 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.0 2021-02-10T11:34:09.844 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:34:09.844 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:34:09.844 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:34:10.179 INFO:teuthology.orchestra.run.gibba002.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.1’: No such file or directory 2021-02-10T11:34:10.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:34:10.179 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2021-02-10T11:34:10.179 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:34:10.180 DEBUG:teuthology.orchestra.run.gibba013:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:34:10.180 DEBUG:teuthology.orchestra.run.gibba013:> ' -- /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:34:10.515 INFO:teuthology.orchestra.run.gibba013.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.2’: No such file or directory 2021-02-10T11:34:10.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:34:10.515 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2021-02-10T11:34:10.515 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:34:10.516 DEBUG:teuthology.orchestra.run.gibba013:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:34:10.516 DEBUG:teuthology.orchestra.run.gibba013:> ' -- /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:34:10.850 INFO:teuthology.orchestra.run.gibba013.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.3’: No such file or directory 2021-02-10T11:34:10.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:34:10.851 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.3 2021-02-10T11:34:10.851 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json-pretty 2021-02-10T11:34:11.282 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:11.263 7f59b813a700 1 Processor -- start 2021-02-10T11:34:11.282 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:11.263 7f59b813a700 1 -- start start 2021-02-10T11:34:11.283 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:11.264 7f59b813a700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f59b00d1870 0x7f59b01b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:11.283 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:11.264 7f59b813a700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59b00f0b10 con 0x7f59b01b0a10 2021-02-10T11:34:11.283 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:11.264 7f59b813a700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59b00d0310 con 0x7f59b00f0650 2021-02-10T11:34:11.283 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:11.264 7f59b813a700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59b00d0530 con 0x7f59b00d1870 2021-02-10T11:34:11.283 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:11.264 7f59b5f56700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f59b00d1870 0x7f59b01b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:11.283 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:11.264 7f59b5755700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f59b01b0a10 0x7f59b00cf330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:56288/0 (socket says 172.21.2.102:56288) 2021-02-10T11:34:11.283 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:11.264 7f59b5755700 1 -- 172.21.2.102:0/877681458 learned_addr learned my addr 172.21.2.102:0/877681458 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:34:11.283 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:11.264 7f59b5f56700 1 -- 172.21.2.102:0/877681458 >> v1:172.21.2.113:6789/0 conn(0x7f59b00f0650 legacy=0x7f59b00cfb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:34:11.283 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:11.264 7f59b5f56700 1 -- 172.21.2.102:0/877681458 >> v1:172.21.2.102:6789/0 conn(0x7f59b01b0a10 legacy=0x7f59b00cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:11.283 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:11.264 7f59b5f56700 1 -- 172.21.2.102:0/877681458 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59b00d1c70 con 0x7f59b00d1870 2021-02-10T11:34:11.283 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:11.265 7f59b5f56700 1 --2- 172.21.2.102:0/877681458 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f59b00d1870 0x7f59b01b03f0 secure :-1 s=READY pgs=2499 cs=0 l=1 rev1=1 rx=0x7f59ac006650 tx=0x7f59ac009a80).ready entity=mon.0 client_cookie=d2aebdf5d1bb33b5 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:11.283 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:11.265 7f59b4f54700 1 -- 172.21.2.102:0/877681458 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f59ac00c0c0 con 0x7f59b00d1870 2021-02-10T11:34:11.284 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:11.265 7f59b4f54700 1 -- 172.21.2.102:0/877681458 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59ac00c340 con 0x7f59b00d1870 2021-02-10T11:34:11.284 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:11.265 7f59b4f54700 1 -- 172.21.2.102:0/877681458 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f59ac00c860 con 0x7f59b00d1870 2021-02-10T11:34:11.284 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:11.265 7f59b813a700 1 -- 172.21.2.102:0/877681458 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f59b00d1870 msgr2=0x7f59b01b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:11.284 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:11.265 7f59b813a700 1 --2- 172.21.2.102:0/877681458 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f59b00d1870 0x7f59b01b03f0 secure :-1 s=READY pgs=2499 cs=0 l=1 rev1=1 rx=0x7f59ac006650 tx=0x7f59ac009a80).stop 2021-02-10T11:34:11.284 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:11.266 7f59b813a700 1 -- 172.21.2.102:0/877681458 shutdown_connections 2021-02-10T11:34:11.284 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:11.266 7f59b813a700 1 --2- 172.21.2.102:0/877681458 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f59b00d1870 0x7f59b01b03f0 unknown :-1 s=CLOSED pgs=2499 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:11.284 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:11.266 7f59b813a700 1 -- 172.21.2.102:0/877681458 >> 172.21.2.102:0/877681458 conn(0x7f59b00ce450 msgr2=0x7f59b00d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:11.284 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:11.266 7f59b813a700 1 -- 172.21.2.102:0/877681458 shutdown_connections 2021-02-10T11:34:11.284 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:11.266 7f59b813a700 1 -- 172.21.2.102:0/877681458 wait complete. 2021-02-10T11:34:11.286 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:11.281 7f59b813a700 1 Processor -- start 2021-02-10T11:34:11.286 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:11.281 7f59b813a700 1 -- start start 2021-02-10T11:34:11.286 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:11.281 7f59b813a700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f59b00ae6b0 0x7f59b00a8bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:11.286 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:11.281 7f59b813a700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f59b00adbd0 0x7f59b00a9a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:11.287 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:11.281 7f59b813a700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f59b00adfd0 0x7f59b00aa090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:11.287 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:11.282 7f59b813a700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59b01bf350 con 0x7f59b00adbd0 2021-02-10T11:34:11.287 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:11.282 7f59b813a700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59b00a9560 con 0x7f59b00ae6b0 2021-02-10T11:34:11.287 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:11.282 7f59b813a700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59b00a9210 con 0x7f59b00adfd0 2021-02-10T11:34:11.287 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:11.282 7f59b5755700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f59b00adbd0 0x7f59b00a9a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:11.287 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:11.282 7f59b5755700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f59b00adbd0 0x7f59b00a9a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:35838/0 (socket says 172.21.2.102:35838) 2021-02-10T11:34:11.287 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:11.282 7f59b5755700 1 -- 172.21.2.102:0/813622409 learned_addr learned my addr 172.21.2.102:0/813622409 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:34:11.287 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:11.282 7f59b5f56700 1 --2- 172.21.2.102:0/813622409 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f59b00ae6b0 0x7f59b00a8bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:11.287 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:11.282 7f59b6757700 1 --2- 172.21.2.102:0/813622409 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f59b00adfd0 0x7f59b00aa090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:11.287 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:11.282 7f59b5755700 1 -- 172.21.2.102:0/813622409 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f59b00adfd0 msgr2=0x7f59b00aa090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:11.287 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:11.282 7f59b5755700 1 --2- 172.21.2.102:0/813622409 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f59b00adfd0 0x7f59b00aa090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:11.287 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:11.282 7f59b5755700 1 -- 172.21.2.102:0/813622409 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f59b00ae6b0 msgr2=0x7f59b00a8bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:11.287 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:11.282 7f59b5755700 1 --2- 172.21.2.102:0/813622409 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f59b00ae6b0 0x7f59b00a8bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:11.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:11.282 7f59b5755700 1 -- 172.21.2.102:0/813622409 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59b00aeab0 con 0x7f59b00adbd0 2021-02-10T11:34:11.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:11.282 7f59b6757700 1 --2- 172.21.2.102:0/813622409 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f59b00adfd0 0x7f59b00aa090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:34:11.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:11.282 7f59b5f56700 1 --2- 172.21.2.102:0/813622409 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f59b00ae6b0 0x7f59b00a8bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:34:11.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:11.282 7f59b5755700 1 --2- 172.21.2.102:0/813622409 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f59b00adbd0 0x7f59b00a9a70 secure :-1 s=READY pgs=2500 cs=0 l=1 rev1=1 rx=0x7f59a0001c60 tx=0x7f59a000a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:11.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:11.282 7f59a67fc700 1 -- 172.21.2.102:0/813622409 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f59a000acc0 con 0x7f59b00adbd0 2021-02-10T11:34:11.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:11.282 7f59a67fc700 1 -- 172.21.2.102:0/813622409 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59a0004140 con 0x7f59b00adbd0 2021-02-10T11:34:11.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:11.282 7f59b813a700 1 -- 172.21.2.102:0/813622409 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59b00aeab0 con 0x7f59b00adbd0 2021-02-10T11:34:11.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:11.283 7f59a67fc700 1 -- 172.21.2.102:0/813622409 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f59a0004620 con 0x7f59b00adbd0 2021-02-10T11:34:11.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:11.283 7f59b813a700 1 -- 172.21.2.102:0/813622409 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59b00aeab0 con 0x7f59b00adbd0 2021-02-10T11:34:11.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:11.283 7f59a67fc700 1 -- 172.21.2.102:0/813622409 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f59a0004320 con 0x7f59b00adbd0 2021-02-10T11:34:11.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:11.283 7f59a67fc700 1 --2- 172.21.2.102:0/813622409 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5998018730 0x7f599801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:11.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:11.283 7f59a67fc700 1 -- 172.21.2.102:0/813622409 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f59a00327a0 con 0x7f59b00adbd0 2021-02-10T11:34:11.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:11.283 7f59b5f56700 1 --2- 172.21.2.102:0/813622409 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5998018730 0x7f599801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:11.289 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:11.284 7f59b5f56700 1 --2- 172.21.2.102:0/813622409 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5998018730 0x7f599801ac10 crc :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:11.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:11.313 7f59b813a700 1 -- 172.21.2.102:0/813622409 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59b00e7de0 con 0x7f59b00adbd0 2021-02-10T11:34:11.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:11.316 7f59a67fc700 1 -- 172.21.2.102:0/813622409 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f59a0000e20 con 0x7f59b00adbd0 2021-02-10T11:34:11.374 DEBUG:teuthology.orchestra.run.gibba002:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:34:11.377 DEBUG:teuthology.orchestra.run.gibba013:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:34:11.445 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:11.443 7f59b813a700 1 -- 172.21.2.102:0/813622409 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7f59b00e7de0 con 0x7f59b00adbd0 2021-02-10T11:34:11.445 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:11.444 7f59a67fc700 1 -- 172.21.2.102:0/813622409 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v128) v1 ==== 81+0+30374 (secure 0 0 0) 0x7f59a0004140 con 0x7f59b00adbd0 2021-02-10T11:34:11.446 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:34:11.446 INFO:teuthology.orchestra.run.gibba002.stdout:{ 2021-02-10T11:34:11.446 INFO:teuthology.orchestra.run.gibba002.stdout: "epoch": 128, 2021-02-10T11:34:11.446 INFO:teuthology.orchestra.run.gibba002.stdout: "fsid": "68db30ea-dbce-4aef-a6df-f01c6fea0513", 2021-02-10T11:34:11.446 INFO:teuthology.orchestra.run.gibba002.stdout: "created": "2021-02-10 11:03:06.331557", 2021-02-10T11:34:11.447 INFO:teuthology.orchestra.run.gibba002.stdout: "modified": "2021-02-10 11:33:33.846924", 2021-02-10T11:34:11.447 INFO:teuthology.orchestra.run.gibba002.stdout: "last_up_change": "2021-02-10 11:03:32.690562", 2021-02-10T11:34:11.447 INFO:teuthology.orchestra.run.gibba002.stdout: "last_in_change": "2021-02-10 11:03:32.690562", 2021-02-10T11:34:11.447 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2021-02-10T11:34:11.447 INFO:teuthology.orchestra.run.gibba002.stdout: "flags_num": 5799936, 2021-02-10T11:34:11.447 INFO:teuthology.orchestra.run.gibba002.stdout: "flags_set": [ 2021-02-10T11:34:11.447 INFO:teuthology.orchestra.run.gibba002.stdout: "pglog_hardlimit", 2021-02-10T11:34:11.447 INFO:teuthology.orchestra.run.gibba002.stdout: "purged_snapdirs", 2021-02-10T11:34:11.447 INFO:teuthology.orchestra.run.gibba002.stdout: "recovery_deletes", 2021-02-10T11:34:11.447 INFO:teuthology.orchestra.run.gibba002.stdout: "sortbitwise" 2021-02-10T11:34:11.448 INFO:teuthology.orchestra.run.gibba002.stdout: ], 2021-02-10T11:34:11.448 INFO:teuthology.orchestra.run.gibba002.stdout: "crush_version": 4, 2021-02-10T11:34:11.448 INFO:teuthology.orchestra.run.gibba002.stdout: "full_ratio": 0.94999998807907104, 2021-02-10T11:34:11.448 INFO:teuthology.orchestra.run.gibba002.stdout: "backfillfull_ratio": 0.89999997615814209, 2021-02-10T11:34:11.448 INFO:teuthology.orchestra.run.gibba002.stdout: "nearfull_ratio": 0.85000002384185791, 2021-02-10T11:34:11.448 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_snapshot": "", 2021-02-10T11:34:11.448 INFO:teuthology.orchestra.run.gibba002.stdout: "pool_max": 27, 2021-02-10T11:34:11.448 INFO:teuthology.orchestra.run.gibba002.stdout: "max_osd": 8, 2021-02-10T11:34:11.448 INFO:teuthology.orchestra.run.gibba002.stdout: "require_min_compat_client": "jewel", 2021-02-10T11:34:11.448 INFO:teuthology.orchestra.run.gibba002.stdout: "min_compat_client": "jewel", 2021-02-10T11:34:11.448 INFO:teuthology.orchestra.run.gibba002.stdout: "require_osd_release": "nautilus", 2021-02-10T11:34:11.448 INFO:teuthology.orchestra.run.gibba002.stdout: "pools": [ 2021-02-10T11:34:11.448 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:11.449 INFO:teuthology.orchestra.run.gibba002.stdout: "pool": 1, 2021-02-10T11:34:11.449 INFO:teuthology.orchestra.run.gibba002.stdout: "pool_name": "rbd", 2021-02-10T11:34:11.449 INFO:teuthology.orchestra.run.gibba002.stdout: "create_time": "2021-02-10 11:03:37.322519", 2021-02-10T11:34:11.449 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 1, 2021-02-10T11:34:11.449 INFO:teuthology.orchestra.run.gibba002.stdout: "flags_names": "hashpspool", 2021-02-10T11:34:11.449 INFO:teuthology.orchestra.run.gibba002.stdout: "type": 1, 2021-02-10T11:34:11.449 INFO:teuthology.orchestra.run.gibba002.stdout: "size": 2, 2021-02-10T11:34:11.449 INFO:teuthology.orchestra.run.gibba002.stdout: "min_size": 1, 2021-02-10T11:34:11.449 INFO:teuthology.orchestra.run.gibba002.stdout: "crush_rule": 0, 2021-02-10T11:34:11.449 INFO:teuthology.orchestra.run.gibba002.stdout: "object_hash": 2, 2021-02-10T11:34:11.449 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:34:11.449 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num": 8, 2021-02-10T11:34:11.449 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_placement_num": 8, 2021-02-10T11:34:11.449 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_placement_num_target": 8, 2021-02-10T11:34:11.450 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num_target": 8, 2021-02-10T11:34:11.450 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num_pending": 8, 2021-02-10T11:34:11.450 INFO:teuthology.orchestra.run.gibba002.stdout: "last_pg_merge_meta": { 2021-02-10T11:34:11.450 INFO:teuthology.orchestra.run.gibba002.stdout: "source_pgid": "0.0", 2021-02-10T11:34:11.450 INFO:teuthology.orchestra.run.gibba002.stdout: "ready_epoch": 0, 2021-02-10T11:34:11.450 INFO:teuthology.orchestra.run.gibba002.stdout: "last_epoch_started": 0, 2021-02-10T11:34:11.450 INFO:teuthology.orchestra.run.gibba002.stdout: "last_epoch_clean": 0, 2021-02-10T11:34:11.450 INFO:teuthology.orchestra.run.gibba002.stdout: "source_version": "0'0", 2021-02-10T11:34:11.450 INFO:teuthology.orchestra.run.gibba002.stdout: "target_version": "0'0" 2021-02-10T11:34:11.450 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:11.450 INFO:teuthology.orchestra.run.gibba002.stdout: "last_change": "16", 2021-02-10T11:34:11.450 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend": "0", 2021-02-10T11:34:11.451 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:34:11.451 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:34:11.451 INFO:teuthology.orchestra.run.gibba002.stdout: "auid": 0, 2021-02-10T11:34:11.451 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:34:11.451 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_seq": 0, 2021-02-10T11:34:11.451 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_epoch": 0, 2021-02-10T11:34:11.451 INFO:teuthology.orchestra.run.gibba002.stdout: "pool_snaps": [], 2021-02-10T11:34:11.451 INFO:teuthology.orchestra.run.gibba002.stdout: "removed_snaps": "[]", 2021-02-10T11:34:11.451 INFO:teuthology.orchestra.run.gibba002.stdout: "quota_max_bytes": 0, 2021-02-10T11:34:11.451 INFO:teuthology.orchestra.run.gibba002.stdout: "quota_max_objects": 0, 2021-02-10T11:34:11.451 INFO:teuthology.orchestra.run.gibba002.stdout: "tiers": [], 2021-02-10T11:34:11.451 INFO:teuthology.orchestra.run.gibba002.stdout: "tier_of": -1, 2021-02-10T11:34:11.451 INFO:teuthology.orchestra.run.gibba002.stdout: "read_tier": -1, 2021-02-10T11:34:11.452 INFO:teuthology.orchestra.run.gibba002.stdout: "write_tier": -1, 2021-02-10T11:34:11.452 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_mode": "none", 2021-02-10T11:34:11.452 INFO:teuthology.orchestra.run.gibba002.stdout: "target_max_bytes": 0, 2021-02-10T11:34:11.452 INFO:teuthology.orchestra.run.gibba002.stdout: "target_max_objects": 0, 2021-02-10T11:34:11.452 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:34:11.452 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:34:11.452 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:34:11.452 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_min_flush_age": 0, 2021-02-10T11:34:11.452 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_min_evict_age": 0, 2021-02-10T11:34:11.452 INFO:teuthology.orchestra.run.gibba002.stdout: "erasure_code_profile": "", 2021-02-10T11:34:11.452 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_params": { 2021-02-10T11:34:11.452 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "none" 2021-02-10T11:34:11.452 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:11.452 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_period": 0, 2021-02-10T11:34:11.453 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_count": 0, 2021-02-10T11:34:11.453 INFO:teuthology.orchestra.run.gibba002.stdout: "use_gmt_hitset": true, 2021-02-10T11:34:11.453 INFO:teuthology.orchestra.run.gibba002.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:34:11.453 INFO:teuthology.orchestra.run.gibba002.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:34:11.453 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:34:11.453 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:34:11.453 INFO:teuthology.orchestra.run.gibba002.stdout: "grade_table": [], 2021-02-10T11:34:11.453 INFO:teuthology.orchestra.run.gibba002.stdout: "stripe_width": 0, 2021-02-10T11:34:11.453 INFO:teuthology.orchestra.run.gibba002.stdout: "expected_num_objects": 0, 2021-02-10T11:34:11.453 INFO:teuthology.orchestra.run.gibba002.stdout: "fast_read": false, 2021-02-10T11:34:11.453 INFO:teuthology.orchestra.run.gibba002.stdout: "options": {}, 2021-02-10T11:34:11.453 INFO:teuthology.orchestra.run.gibba002.stdout: "application_metadata": { 2021-02-10T11:34:11.453 INFO:teuthology.orchestra.run.gibba002.stdout: "rbd": {} 2021-02-10T11:34:11.454 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:34:11.454 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:11.454 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:11.454 INFO:teuthology.orchestra.run.gibba002.stdout: "pool": 26, 2021-02-10T11:34:11.454 INFO:teuthology.orchestra.run.gibba002.stdout: "pool_name": "cephfs_metadata", 2021-02-10T11:34:11.454 INFO:teuthology.orchestra.run.gibba002.stdout: "create_time": "2021-02-10 11:33:30.653221", 2021-02-10T11:34:11.454 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 1, 2021-02-10T11:34:11.454 INFO:teuthology.orchestra.run.gibba002.stdout: "flags_names": "hashpspool", 2021-02-10T11:34:11.454 INFO:teuthology.orchestra.run.gibba002.stdout: "type": 1, 2021-02-10T11:34:11.454 INFO:teuthology.orchestra.run.gibba002.stdout: "size": 2, 2021-02-10T11:34:11.454 INFO:teuthology.orchestra.run.gibba002.stdout: "min_size": 1, 2021-02-10T11:34:11.454 INFO:teuthology.orchestra.run.gibba002.stdout: "crush_rule": 0, 2021-02-10T11:34:11.454 INFO:teuthology.orchestra.run.gibba002.stdout: "object_hash": 2, 2021-02-10T11:34:11.454 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:34:11.455 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num": 8, 2021-02-10T11:34:11.455 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_placement_num": 8, 2021-02-10T11:34:11.455 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_placement_num_target": 8, 2021-02-10T11:34:11.455 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num_target": 8, 2021-02-10T11:34:11.455 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num_pending": 8, 2021-02-10T11:34:11.455 INFO:teuthology.orchestra.run.gibba002.stdout: "last_pg_merge_meta": { 2021-02-10T11:34:11.455 INFO:teuthology.orchestra.run.gibba002.stdout: "source_pgid": "0.0", 2021-02-10T11:34:11.455 INFO:teuthology.orchestra.run.gibba002.stdout: "ready_epoch": 0, 2021-02-10T11:34:11.455 INFO:teuthology.orchestra.run.gibba002.stdout: "last_epoch_started": 0, 2021-02-10T11:34:11.455 INFO:teuthology.orchestra.run.gibba002.stdout: "last_epoch_clean": 0, 2021-02-10T11:34:11.455 INFO:teuthology.orchestra.run.gibba002.stdout: "source_version": "0'0", 2021-02-10T11:34:11.455 INFO:teuthology.orchestra.run.gibba002.stdout: "target_version": "0'0" 2021-02-10T11:34:11.456 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:11.456 INFO:teuthology.orchestra.run.gibba002.stdout: "last_change": "127", 2021-02-10T11:34:11.456 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend": "0", 2021-02-10T11:34:11.456 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:34:11.456 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:34:11.456 INFO:teuthology.orchestra.run.gibba002.stdout: "auid": 0, 2021-02-10T11:34:11.456 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:34:11.456 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_seq": 0, 2021-02-10T11:34:11.456 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_epoch": 0, 2021-02-10T11:34:11.456 INFO:teuthology.orchestra.run.gibba002.stdout: "pool_snaps": [], 2021-02-10T11:34:11.456 INFO:teuthology.orchestra.run.gibba002.stdout: "removed_snaps": "[]", 2021-02-10T11:34:11.456 INFO:teuthology.orchestra.run.gibba002.stdout: "quota_max_bytes": 0, 2021-02-10T11:34:11.456 INFO:teuthology.orchestra.run.gibba002.stdout: "quota_max_objects": 0, 2021-02-10T11:34:11.457 INFO:teuthology.orchestra.run.gibba002.stdout: "tiers": [], 2021-02-10T11:34:11.457 INFO:teuthology.orchestra.run.gibba002.stdout: "tier_of": -1, 2021-02-10T11:34:11.457 INFO:teuthology.orchestra.run.gibba002.stdout: "read_tier": -1, 2021-02-10T11:34:11.457 INFO:teuthology.orchestra.run.gibba002.stdout: "write_tier": -1, 2021-02-10T11:34:11.457 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_mode": "none", 2021-02-10T11:34:11.457 INFO:teuthology.orchestra.run.gibba002.stdout: "target_max_bytes": 0, 2021-02-10T11:34:11.457 INFO:teuthology.orchestra.run.gibba002.stdout: "target_max_objects": 0, 2021-02-10T11:34:11.457 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:34:11.457 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:34:11.457 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:34:11.457 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_min_flush_age": 0, 2021-02-10T11:34:11.458 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_min_evict_age": 0, 2021-02-10T11:34:11.458 INFO:teuthology.orchestra.run.gibba002.stdout: "erasure_code_profile": "", 2021-02-10T11:34:11.458 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_params": { 2021-02-10T11:34:11.458 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "none" 2021-02-10T11:34:11.458 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:11.458 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_period": 0, 2021-02-10T11:34:11.458 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_count": 0, 2021-02-10T11:34:11.458 INFO:teuthology.orchestra.run.gibba002.stdout: "use_gmt_hitset": true, 2021-02-10T11:34:11.458 INFO:teuthology.orchestra.run.gibba002.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:34:11.458 INFO:teuthology.orchestra.run.gibba002.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:34:11.458 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:34:11.459 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:34:11.459 INFO:teuthology.orchestra.run.gibba002.stdout: "grade_table": [], 2021-02-10T11:34:11.459 INFO:teuthology.orchestra.run.gibba002.stdout: "stripe_width": 0, 2021-02-10T11:34:11.459 INFO:teuthology.orchestra.run.gibba002.stdout: "expected_num_objects": 0, 2021-02-10T11:34:11.459 INFO:teuthology.orchestra.run.gibba002.stdout: "fast_read": false, 2021-02-10T11:34:11.459 INFO:teuthology.orchestra.run.gibba002.stdout: "options": { 2021-02-10T11:34:11.459 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_autoscale_bias": 4, 2021-02-10T11:34:11.459 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num_min": 16, 2021-02-10T11:34:11.459 INFO:teuthology.orchestra.run.gibba002.stdout: "recovery_priority": 5 2021-02-10T11:34:11.459 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:11.459 INFO:teuthology.orchestra.run.gibba002.stdout: "application_metadata": { 2021-02-10T11:34:11.459 INFO:teuthology.orchestra.run.gibba002.stdout: "cephfs": { 2021-02-10T11:34:11.460 INFO:teuthology.orchestra.run.gibba002.stdout: "metadata": "cephfs" 2021-02-10T11:34:11.460 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:34:11.460 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:34:11.460 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:11.460 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:11.460 INFO:teuthology.orchestra.run.gibba002.stdout: "pool": 27, 2021-02-10T11:34:11.461 INFO:teuthology.orchestra.run.gibba002.stdout: "pool_name": "cephfs_data", 2021-02-10T11:34:11.461 INFO:teuthology.orchestra.run.gibba002.stdout: "create_time": "2021-02-10 11:33:31.831765", 2021-02-10T11:34:11.461 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 1, 2021-02-10T11:34:11.461 INFO:teuthology.orchestra.run.gibba002.stdout: "flags_names": "hashpspool", 2021-02-10T11:34:11.461 INFO:teuthology.orchestra.run.gibba002.stdout: "type": 1, 2021-02-10T11:34:11.461 INFO:teuthology.orchestra.run.gibba002.stdout: "size": 2, 2021-02-10T11:34:11.461 INFO:teuthology.orchestra.run.gibba002.stdout: "min_size": 1, 2021-02-10T11:34:11.461 INFO:teuthology.orchestra.run.gibba002.stdout: "crush_rule": 0, 2021-02-10T11:34:11.461 INFO:teuthology.orchestra.run.gibba002.stdout: "object_hash": 2, 2021-02-10T11:34:11.462 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:34:11.462 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num": 8, 2021-02-10T11:34:11.462 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_placement_num": 8, 2021-02-10T11:34:11.462 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_placement_num_target": 8, 2021-02-10T11:34:11.462 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num_target": 8, 2021-02-10T11:34:11.462 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num_pending": 8, 2021-02-10T11:34:11.462 INFO:teuthology.orchestra.run.gibba002.stdout: "last_pg_merge_meta": { 2021-02-10T11:34:11.462 INFO:teuthology.orchestra.run.gibba002.stdout: "source_pgid": "0.0", 2021-02-10T11:34:11.462 INFO:teuthology.orchestra.run.gibba002.stdout: "ready_epoch": 0, 2021-02-10T11:34:11.462 INFO:teuthology.orchestra.run.gibba002.stdout: "last_epoch_started": 0, 2021-02-10T11:34:11.462 INFO:teuthology.orchestra.run.gibba002.stdout: "last_epoch_clean": 0, 2021-02-10T11:34:11.462 INFO:teuthology.orchestra.run.gibba002.stdout: "source_version": "0'0", 2021-02-10T11:34:11.463 INFO:teuthology.orchestra.run.gibba002.stdout: "target_version": "0'0" 2021-02-10T11:34:11.463 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:11.463 INFO:teuthology.orchestra.run.gibba002.stdout: "last_change": "128", 2021-02-10T11:34:11.463 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend": "0", 2021-02-10T11:34:11.463 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:34:11.463 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:34:11.463 INFO:teuthology.orchestra.run.gibba002.stdout: "auid": 0, 2021-02-10T11:34:11.463 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:34:11.463 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_seq": 0, 2021-02-10T11:34:11.463 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_epoch": 0, 2021-02-10T11:34:11.463 INFO:teuthology.orchestra.run.gibba002.stdout: "pool_snaps": [], 2021-02-10T11:34:11.463 INFO:teuthology.orchestra.run.gibba002.stdout: "removed_snaps": "[]", 2021-02-10T11:34:11.463 INFO:teuthology.orchestra.run.gibba002.stdout: "quota_max_bytes": 0, 2021-02-10T11:34:11.464 INFO:teuthology.orchestra.run.gibba002.stdout: "quota_max_objects": 0, 2021-02-10T11:34:11.464 INFO:teuthology.orchestra.run.gibba002.stdout: "tiers": [], 2021-02-10T11:34:11.464 INFO:teuthology.orchestra.run.gibba002.stdout: "tier_of": -1, 2021-02-10T11:34:11.464 INFO:teuthology.orchestra.run.gibba002.stdout: "read_tier": -1, 2021-02-10T11:34:11.464 INFO:teuthology.orchestra.run.gibba002.stdout: "write_tier": -1, 2021-02-10T11:34:11.464 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_mode": "none", 2021-02-10T11:34:11.464 INFO:teuthology.orchestra.run.gibba002.stdout: "target_max_bytes": 0, 2021-02-10T11:34:11.464 INFO:teuthology.orchestra.run.gibba002.stdout: "target_max_objects": 0, 2021-02-10T11:34:11.465 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:34:11.465 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:34:11.465 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:34:11.465 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_min_flush_age": 0, 2021-02-10T11:34:11.465 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_min_evict_age": 0, 2021-02-10T11:34:11.465 INFO:teuthology.orchestra.run.gibba002.stdout: "erasure_code_profile": "", 2021-02-10T11:34:11.465 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_params": { 2021-02-10T11:34:11.465 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "none" 2021-02-10T11:34:11.465 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:11.465 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_period": 0, 2021-02-10T11:34:11.465 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_count": 0, 2021-02-10T11:34:11.465 INFO:teuthology.orchestra.run.gibba002.stdout: "use_gmt_hitset": true, 2021-02-10T11:34:11.465 INFO:teuthology.orchestra.run.gibba002.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:34:11.466 INFO:teuthology.orchestra.run.gibba002.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:34:11.466 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:34:11.466 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:34:11.466 INFO:teuthology.orchestra.run.gibba002.stdout: "grade_table": [], 2021-02-10T11:34:11.466 INFO:teuthology.orchestra.run.gibba002.stdout: "stripe_width": 0, 2021-02-10T11:34:11.466 INFO:teuthology.orchestra.run.gibba002.stdout: "expected_num_objects": 0, 2021-02-10T11:34:11.466 INFO:teuthology.orchestra.run.gibba002.stdout: "fast_read": false, 2021-02-10T11:34:11.466 INFO:teuthology.orchestra.run.gibba002.stdout: "options": {}, 2021-02-10T11:34:11.467 INFO:teuthology.orchestra.run.gibba002.stdout: "application_metadata": { 2021-02-10T11:34:11.467 INFO:teuthology.orchestra.run.gibba002.stdout: "cephfs": { 2021-02-10T11:34:11.467 INFO:teuthology.orchestra.run.gibba002.stdout: "data": "cephfs" 2021-02-10T11:34:11.467 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:34:11.467 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:34:11.467 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:34:11.467 INFO:teuthology.orchestra.run.gibba002.stdout: ], 2021-02-10T11:34:11.467 INFO:teuthology.orchestra.run.gibba002.stdout: "osds": [ 2021-02-10T11:34:11.467 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:11.467 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 0, 2021-02-10T11:34:11.467 INFO:teuthology.orchestra.run.gibba002.stdout: "uuid": "be691b4d-a039-408a-af80-9d53f02b6f1f", 2021-02-10T11:34:11.467 INFO:teuthology.orchestra.run.gibba002.stdout: "up": 1, 2021-02-10T11:34:11.467 INFO:teuthology.orchestra.run.gibba002.stdout: "in": 1, 2021-02-10T11:34:11.467 INFO:teuthology.orchestra.run.gibba002.stdout: "weight": 1, 2021-02-10T11:34:11.468 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_affinity": 1, 2021-02-10T11:34:11.468 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_begin": 0, 2021-02-10T11:34:11.468 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_end": 0, 2021-02-10T11:34:11.468 INFO:teuthology.orchestra.run.gibba002.stdout: "up_from": 13, 2021-02-10T11:34:11.468 INFO:teuthology.orchestra.run.gibba002.stdout: "up_thru": 126, 2021-02-10T11:34:11.468 INFO:teuthology.orchestra.run.gibba002.stdout: "down_at": 0, 2021-02-10T11:34:11.468 INFO:teuthology.orchestra.run.gibba002.stdout: "lost_at": 0, 2021-02-10T11:34:11.468 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addrs": { 2021-02-10T11:34:11.468 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:34:11.468 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:11.468 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:34:11.468 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6818", 2021-02-10T11:34:11.468 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36088 2021-02-10T11:34:11.469 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:11.469 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:11.469 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:34:11.469 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6820", 2021-02-10T11:34:11.469 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36088 2021-02-10T11:34:11.469 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:34:11.469 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:34:11.469 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:11.469 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addrs": { 2021-02-10T11:34:11.469 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:34:11.469 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:11.469 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:34:11.469 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6822", 2021-02-10T11:34:11.469 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36088 2021-02-10T11:34:11.470 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:11.470 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:11.470 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:34:11.470 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6824", 2021-02-10T11:34:11.470 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36088 2021-02-10T11:34:11.470 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:34:11.470 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:34:11.470 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:11.470 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addrs": { 2021-02-10T11:34:11.470 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:34:11.470 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:11.470 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:34:11.470 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6830", 2021-02-10T11:34:11.471 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36088 2021-02-10T11:34:11.471 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:11.471 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:11.471 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:34:11.471 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6832", 2021-02-10T11:34:11.471 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36088 2021-02-10T11:34:11.471 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:34:11.471 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:34:11.471 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:11.471 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addrs": { 2021-02-10T11:34:11.471 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:34:11.471 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:11.471 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:34:11.472 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6826", 2021-02-10T11:34:11.472 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36088 2021-02-10T11:34:11.472 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:11.472 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:11.472 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:34:11.472 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6828", 2021-02-10T11:34:11.472 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36088 2021-02-10T11:34:11.472 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:34:11.472 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:34:11.472 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:11.472 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addr": "172.21.2.102:6820/36088", 2021-02-10T11:34:11.472 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addr": "172.21.2.102:6824/36088", 2021-02-10T11:34:11.472 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addr": "172.21.2.102:6832/36088", 2021-02-10T11:34:11.473 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addr": "172.21.2.102:6828/36088", 2021-02-10T11:34:11.473 INFO:teuthology.orchestra.run.gibba002.stdout: "state": [ 2021-02-10T11:34:11.473 INFO:teuthology.orchestra.run.gibba002.stdout: "exists", 2021-02-10T11:34:11.473 INFO:teuthology.orchestra.run.gibba002.stdout: "up" 2021-02-10T11:34:11.473 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:34:11.473 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:11.473 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:11.473 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 1, 2021-02-10T11:34:11.473 INFO:teuthology.orchestra.run.gibba002.stdout: "uuid": "64e3ccc8-9c69-4d7c-865e-af8e83822635", 2021-02-10T11:34:11.473 INFO:teuthology.orchestra.run.gibba002.stdout: "up": 1, 2021-02-10T11:34:11.473 INFO:teuthology.orchestra.run.gibba002.stdout: "in": 1, 2021-02-10T11:34:11.473 INFO:teuthology.orchestra.run.gibba002.stdout: "weight": 1, 2021-02-10T11:34:11.473 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_affinity": 1, 2021-02-10T11:34:11.473 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_begin": 0, 2021-02-10T11:34:11.474 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_end": 0, 2021-02-10T11:34:11.474 INFO:teuthology.orchestra.run.gibba002.stdout: "up_from": 13, 2021-02-10T11:34:11.474 INFO:teuthology.orchestra.run.gibba002.stdout: "up_thru": 125, 2021-02-10T11:34:11.474 INFO:teuthology.orchestra.run.gibba002.stdout: "down_at": 0, 2021-02-10T11:34:11.474 INFO:teuthology.orchestra.run.gibba002.stdout: "lost_at": 0, 2021-02-10T11:34:11.474 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addrs": { 2021-02-10T11:34:11.474 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:34:11.474 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:11.474 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:34:11.474 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6819", 2021-02-10T11:34:11.474 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36087 2021-02-10T11:34:11.474 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:11.475 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:11.475 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:34:11.475 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6821", 2021-02-10T11:34:11.475 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36087 2021-02-10T11:34:11.475 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:34:11.475 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:34:11.475 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:11.475 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addrs": { 2021-02-10T11:34:11.475 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:34:11.475 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:11.475 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:34:11.475 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6823", 2021-02-10T11:34:11.475 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36087 2021-02-10T11:34:11.475 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:11.476 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:11.476 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:34:11.476 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6825", 2021-02-10T11:34:11.476 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36087 2021-02-10T11:34:11.476 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:34:11.476 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:34:11.476 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:11.476 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addrs": { 2021-02-10T11:34:11.476 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:34:11.476 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:11.476 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:34:11.476 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6831", 2021-02-10T11:34:11.476 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36087 2021-02-10T11:34:11.477 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:11.477 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:11.477 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:34:11.477 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6833", 2021-02-10T11:34:11.477 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36087 2021-02-10T11:34:11.477 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:34:11.477 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:34:11.477 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:11.477 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addrs": { 2021-02-10T11:34:11.477 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:34:11.477 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:11.477 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:34:11.477 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6827", 2021-02-10T11:34:11.478 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36087 2021-02-10T11:34:11.478 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:11.478 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:11.478 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:34:11.478 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6829", 2021-02-10T11:34:11.478 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36087 2021-02-10T11:34:11.478 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:34:11.478 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:34:11.478 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:11.478 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addr": "172.21.2.102:6821/36087", 2021-02-10T11:34:11.478 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addr": "172.21.2.102:6825/36087", 2021-02-10T11:34:11.478 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addr": "172.21.2.102:6833/36087", 2021-02-10T11:34:11.478 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addr": "172.21.2.102:6829/36087", 2021-02-10T11:34:11.479 INFO:teuthology.orchestra.run.gibba002.stdout: "state": [ 2021-02-10T11:34:11.479 INFO:teuthology.orchestra.run.gibba002.stdout: "exists", 2021-02-10T11:34:11.479 INFO:teuthology.orchestra.run.gibba002.stdout: "up" 2021-02-10T11:34:11.479 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:34:11.479 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:11.479 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:11.479 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 2, 2021-02-10T11:34:11.479 INFO:teuthology.orchestra.run.gibba002.stdout: "uuid": "153700af-747f-481f-bdf4-d6072d3a842f", 2021-02-10T11:34:11.479 INFO:teuthology.orchestra.run.gibba002.stdout: "up": 1, 2021-02-10T11:34:11.479 INFO:teuthology.orchestra.run.gibba002.stdout: "in": 1, 2021-02-10T11:34:11.479 INFO:teuthology.orchestra.run.gibba002.stdout: "weight": 1, 2021-02-10T11:34:11.479 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_affinity": 1, 2021-02-10T11:34:11.479 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_begin": 0, 2021-02-10T11:34:11.480 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_end": 0, 2021-02-10T11:34:11.480 INFO:teuthology.orchestra.run.gibba002.stdout: "up_from": 13, 2021-02-10T11:34:11.480 INFO:teuthology.orchestra.run.gibba002.stdout: "up_thru": 126, 2021-02-10T11:34:11.480 INFO:teuthology.orchestra.run.gibba002.stdout: "down_at": 0, 2021-02-10T11:34:11.480 INFO:teuthology.orchestra.run.gibba002.stdout: "lost_at": 0, 2021-02-10T11:34:11.480 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addrs": { 2021-02-10T11:34:11.480 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:34:11.480 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:11.480 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:34:11.480 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6810", 2021-02-10T11:34:11.480 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36089 2021-02-10T11:34:11.480 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:11.480 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:11.480 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:34:11.481 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6811", 2021-02-10T11:34:11.481 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36089 2021-02-10T11:34:11.481 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:34:11.481 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:34:11.481 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:11.481 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addrs": { 2021-02-10T11:34:11.481 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:34:11.481 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:11.481 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:34:11.481 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6812", 2021-02-10T11:34:11.481 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36089 2021-02-10T11:34:11.481 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:11.481 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:11.482 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:34:11.482 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6813", 2021-02-10T11:34:11.482 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36089 2021-02-10T11:34:11.482 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:34:11.482 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:34:11.482 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:11.482 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addrs": { 2021-02-10T11:34:11.482 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:34:11.482 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:11.482 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:34:11.482 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6816", 2021-02-10T11:34:11.482 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36089 2021-02-10T11:34:11.482 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:11.482 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:11.483 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:34:11.483 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6817", 2021-02-10T11:34:11.483 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36089 2021-02-10T11:34:11.483 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:34:11.483 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:34:11.483 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:11.483 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addrs": { 2021-02-10T11:34:11.483 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:34:11.483 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:11.483 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:34:11.483 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6814", 2021-02-10T11:34:11.483 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36089 2021-02-10T11:34:11.483 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:11.483 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:11.484 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:34:11.484 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6815", 2021-02-10T11:34:11.484 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36089 2021-02-10T11:34:11.484 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:34:11.484 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:34:11.484 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:11.484 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addr": "172.21.2.102:6811/36089", 2021-02-10T11:34:11.484 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addr": "172.21.2.102:6813/36089", 2021-02-10T11:34:11.484 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addr": "172.21.2.102:6817/36089", 2021-02-10T11:34:11.484 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addr": "172.21.2.102:6815/36089", 2021-02-10T11:34:11.484 INFO:teuthology.orchestra.run.gibba002.stdout: "state": [ 2021-02-10T11:34:11.484 INFO:teuthology.orchestra.run.gibba002.stdout: "exists", 2021-02-10T11:34:11.484 INFO:teuthology.orchestra.run.gibba002.stdout: "up" 2021-02-10T11:34:11.484 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:34:11.485 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:11.485 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:11.485 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 3, 2021-02-10T11:34:11.485 INFO:teuthology.orchestra.run.gibba002.stdout: "uuid": "cbc678f4-83a9-45e6-8830-2f7c232b1ea7", 2021-02-10T11:34:11.485 INFO:teuthology.orchestra.run.gibba002.stdout: "up": 1, 2021-02-10T11:34:11.485 INFO:teuthology.orchestra.run.gibba002.stdout: "in": 1, 2021-02-10T11:34:11.485 INFO:teuthology.orchestra.run.gibba002.stdout: "weight": 1, 2021-02-10T11:34:11.485 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_affinity": 1, 2021-02-10T11:34:11.485 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_begin": 0, 2021-02-10T11:34:11.485 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_end": 0, 2021-02-10T11:34:11.485 INFO:teuthology.orchestra.run.gibba002.stdout: "up_from": 13, 2021-02-10T11:34:11.485 INFO:teuthology.orchestra.run.gibba002.stdout: "up_thru": 118, 2021-02-10T11:34:11.485 INFO:teuthology.orchestra.run.gibba002.stdout: "down_at": 0, 2021-02-10T11:34:11.485 INFO:teuthology.orchestra.run.gibba002.stdout: "lost_at": 0, 2021-02-10T11:34:11.486 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addrs": { 2021-02-10T11:34:11.486 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:34:11.486 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:11.486 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:34:11.486 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6802", 2021-02-10T11:34:11.486 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36090 2021-02-10T11:34:11.486 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:11.486 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:11.486 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:34:11.486 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6803", 2021-02-10T11:34:11.486 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36090 2021-02-10T11:34:11.486 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:34:11.486 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:34:11.487 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:11.487 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addrs": { 2021-02-10T11:34:11.487 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:34:11.487 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:11.487 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:34:11.487 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6804", 2021-02-10T11:34:11.487 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36090 2021-02-10T11:34:11.487 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:11.487 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:11.487 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:34:11.487 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6805", 2021-02-10T11:34:11.487 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36090 2021-02-10T11:34:11.487 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:34:11.488 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:34:11.488 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:11.488 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addrs": { 2021-02-10T11:34:11.488 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:34:11.488 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:11.488 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:34:11.488 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6808", 2021-02-10T11:34:11.488 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36090 2021-02-10T11:34:11.488 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:11.488 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:11.488 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:34:11.488 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6809", 2021-02-10T11:34:11.488 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36090 2021-02-10T11:34:11.488 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:34:11.489 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:34:11.489 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:11.489 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addrs": { 2021-02-10T11:34:11.489 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:34:11.489 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:11.489 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:34:11.489 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6806", 2021-02-10T11:34:11.489 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36090 2021-02-10T11:34:11.489 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:11.489 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:11.489 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:34:11.489 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6807", 2021-02-10T11:34:11.489 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36090 2021-02-10T11:34:11.490 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:34:11.490 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:34:11.490 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:11.490 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addr": "172.21.2.102:6803/36090", 2021-02-10T11:34:11.490 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addr": "172.21.2.102:6805/36090", 2021-02-10T11:34:11.490 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addr": "172.21.2.102:6809/36090", 2021-02-10T11:34:11.490 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addr": "172.21.2.102:6807/36090", 2021-02-10T11:34:11.490 INFO:teuthology.orchestra.run.gibba002.stdout: "state": [ 2021-02-10T11:34:11.490 INFO:teuthology.orchestra.run.gibba002.stdout: "exists", 2021-02-10T11:34:11.490 INFO:teuthology.orchestra.run.gibba002.stdout: "up" 2021-02-10T11:34:11.490 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:34:11.490 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:11.490 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:11.490 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 4, 2021-02-10T11:34:11.491 INFO:teuthology.orchestra.run.gibba002.stdout: "uuid": "c01e4008-b258-4f10-afd2-9e7f37214351", 2021-02-10T11:34:11.491 INFO:teuthology.orchestra.run.gibba002.stdout: "up": 1, 2021-02-10T11:34:11.491 INFO:teuthology.orchestra.run.gibba002.stdout: "in": 1, 2021-02-10T11:34:11.491 INFO:teuthology.orchestra.run.gibba002.stdout: "weight": 1, 2021-02-10T11:34:11.491 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_affinity": 1, 2021-02-10T11:34:11.491 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_begin": 0, 2021-02-10T11:34:11.491 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_end": 0, 2021-02-10T11:34:11.491 INFO:teuthology.orchestra.run.gibba002.stdout: "up_from": 13, 2021-02-10T11:34:11.491 INFO:teuthology.orchestra.run.gibba002.stdout: "up_thru": 126, 2021-02-10T11:34:11.491 INFO:teuthology.orchestra.run.gibba002.stdout: "down_at": 0, 2021-02-10T11:34:11.491 INFO:teuthology.orchestra.run.gibba002.stdout: "lost_at": 0, 2021-02-10T11:34:11.491 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addrs": { 2021-02-10T11:34:11.491 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:34:11.492 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:11.492 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:34:11.492 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6808", 2021-02-10T11:34:11.492 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36368 2021-02-10T11:34:11.492 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:11.492 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:11.492 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:34:11.492 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6811", 2021-02-10T11:34:11.492 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36368 2021-02-10T11:34:11.492 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:34:11.492 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:34:11.492 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:11.492 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addrs": { 2021-02-10T11:34:11.492 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:34:11.493 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:11.493 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:34:11.493 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6814", 2021-02-10T11:34:11.493 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36368 2021-02-10T11:34:11.493 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:11.493 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:11.493 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:34:11.493 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6818", 2021-02-10T11:34:11.493 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36368 2021-02-10T11:34:11.493 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:34:11.493 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:34:11.493 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:11.493 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addrs": { 2021-02-10T11:34:11.493 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:34:11.494 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:11.494 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:34:11.494 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6826", 2021-02-10T11:34:11.494 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36368 2021-02-10T11:34:11.494 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:11.494 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:11.494 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:34:11.494 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6828", 2021-02-10T11:34:11.494 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36368 2021-02-10T11:34:11.494 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:34:11.494 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:34:11.494 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:11.494 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addrs": { 2021-02-10T11:34:11.495 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:34:11.495 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:11.495 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:34:11.495 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6820", 2021-02-10T11:34:11.495 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36368 2021-02-10T11:34:11.495 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:11.495 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:11.495 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:34:11.495 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6824", 2021-02-10T11:34:11.495 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36368 2021-02-10T11:34:11.495 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:34:11.495 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:34:11.495 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:11.495 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addr": "172.21.2.113:6811/36368", 2021-02-10T11:34:11.496 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addr": "172.21.2.113:6818/36368", 2021-02-10T11:34:11.496 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addr": "172.21.2.113:6828/36368", 2021-02-10T11:34:11.496 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addr": "172.21.2.113:6824/36368", 2021-02-10T11:34:11.496 INFO:teuthology.orchestra.run.gibba002.stdout: "state": [ 2021-02-10T11:34:11.496 INFO:teuthology.orchestra.run.gibba002.stdout: "exists", 2021-02-10T11:34:11.496 INFO:teuthology.orchestra.run.gibba002.stdout: "up" 2021-02-10T11:34:11.496 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:34:11.496 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:11.496 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:11.496 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 5, 2021-02-10T11:34:11.496 INFO:teuthology.orchestra.run.gibba002.stdout: "uuid": "551f6007-daf8-4a98-8ee0-c2b16bcccb5e", 2021-02-10T11:34:11.496 INFO:teuthology.orchestra.run.gibba002.stdout: "up": 1, 2021-02-10T11:34:11.496 INFO:teuthology.orchestra.run.gibba002.stdout: "in": 1, 2021-02-10T11:34:11.496 INFO:teuthology.orchestra.run.gibba002.stdout: "weight": 1, 2021-02-10T11:34:11.497 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_affinity": 1, 2021-02-10T11:34:11.497 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_begin": 0, 2021-02-10T11:34:11.497 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_end": 0, 2021-02-10T11:34:11.497 INFO:teuthology.orchestra.run.gibba002.stdout: "up_from": 13, 2021-02-10T11:34:11.497 INFO:teuthology.orchestra.run.gibba002.stdout: "up_thru": 126, 2021-02-10T11:34:11.497 INFO:teuthology.orchestra.run.gibba002.stdout: "down_at": 0, 2021-02-10T11:34:11.497 INFO:teuthology.orchestra.run.gibba002.stdout: "lost_at": 0, 2021-02-10T11:34:11.497 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addrs": { 2021-02-10T11:34:11.497 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:34:11.497 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:11.497 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:34:11.497 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6802", 2021-02-10T11:34:11.497 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36366 2021-02-10T11:34:11.498 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:11.498 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:11.498 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:34:11.498 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6805", 2021-02-10T11:34:11.498 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36366 2021-02-10T11:34:11.498 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:34:11.498 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:34:11.498 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:11.498 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addrs": { 2021-02-10T11:34:11.498 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:34:11.498 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:11.498 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:34:11.498 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6807", 2021-02-10T11:34:11.498 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36366 2021-02-10T11:34:11.499 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:11.499 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:11.499 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:34:11.499 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6810", 2021-02-10T11:34:11.499 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36366 2021-02-10T11:34:11.499 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:34:11.499 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:34:11.499 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:11.499 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addrs": { 2021-02-10T11:34:11.499 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:34:11.499 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:11.499 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:34:11.499 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6819", 2021-02-10T11:34:11.499 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36366 2021-02-10T11:34:11.500 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:11.500 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:11.500 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:34:11.500 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6822", 2021-02-10T11:34:11.500 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36366 2021-02-10T11:34:11.500 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:34:11.500 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:34:11.500 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:11.500 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addrs": { 2021-02-10T11:34:11.500 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:34:11.500 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:11.500 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:34:11.500 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6812", 2021-02-10T11:34:11.500 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36366 2021-02-10T11:34:11.501 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:11.501 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:11.501 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:34:11.501 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6815", 2021-02-10T11:34:11.501 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36366 2021-02-10T11:34:11.501 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:34:11.501 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:34:11.501 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:11.501 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addr": "172.21.2.113:6805/36366", 2021-02-10T11:34:11.501 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addr": "172.21.2.113:6810/36366", 2021-02-10T11:34:11.501 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addr": "172.21.2.113:6822/36366", 2021-02-10T11:34:11.501 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addr": "172.21.2.113:6815/36366", 2021-02-10T11:34:11.501 INFO:teuthology.orchestra.run.gibba002.stdout: "state": [ 2021-02-10T11:34:11.501 INFO:teuthology.orchestra.run.gibba002.stdout: "exists", 2021-02-10T11:34:11.502 INFO:teuthology.orchestra.run.gibba002.stdout: "up" 2021-02-10T11:34:11.502 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:34:11.502 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:11.502 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:11.502 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 6, 2021-02-10T11:34:11.502 INFO:teuthology.orchestra.run.gibba002.stdout: "uuid": "bf67c040-0e50-48be-b102-c3943bc9ebef", 2021-02-10T11:34:11.502 INFO:teuthology.orchestra.run.gibba002.stdout: "up": 1, 2021-02-10T11:34:11.502 INFO:teuthology.orchestra.run.gibba002.stdout: "in": 1, 2021-02-10T11:34:11.502 INFO:teuthology.orchestra.run.gibba002.stdout: "weight": 1, 2021-02-10T11:34:11.502 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_affinity": 1, 2021-02-10T11:34:11.502 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_begin": 0, 2021-02-10T11:34:11.502 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_end": 0, 2021-02-10T11:34:11.502 INFO:teuthology.orchestra.run.gibba002.stdout: "up_from": 13, 2021-02-10T11:34:11.502 INFO:teuthology.orchestra.run.gibba002.stdout: "up_thru": 126, 2021-02-10T11:34:11.503 INFO:teuthology.orchestra.run.gibba002.stdout: "down_at": 0, 2021-02-10T11:34:11.503 INFO:teuthology.orchestra.run.gibba002.stdout: "lost_at": 0, 2021-02-10T11:34:11.503 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addrs": { 2021-02-10T11:34:11.503 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:34:11.503 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:11.503 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:34:11.503 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6800", 2021-02-10T11:34:11.503 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36367 2021-02-10T11:34:11.503 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:11.503 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:11.503 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:34:11.503 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6801", 2021-02-10T11:34:11.503 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36367 2021-02-10T11:34:11.503 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:34:11.504 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:34:11.504 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:11.504 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addrs": { 2021-02-10T11:34:11.504 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:34:11.504 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:11.504 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:34:11.504 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6803", 2021-02-10T11:34:11.504 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36367 2021-02-10T11:34:11.504 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:11.504 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:11.504 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:34:11.504 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6804", 2021-02-10T11:34:11.504 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36367 2021-02-10T11:34:11.504 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:34:11.505 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:34:11.505 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:11.505 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addrs": { 2021-02-10T11:34:11.505 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:34:11.505 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:11.505 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:34:11.505 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6813", 2021-02-10T11:34:11.505 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36367 2021-02-10T11:34:11.505 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:11.505 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:11.505 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:34:11.505 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6816", 2021-02-10T11:34:11.505 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36367 2021-02-10T11:34:11.506 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:34:11.506 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:34:11.506 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:11.506 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addrs": { 2021-02-10T11:34:11.506 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:34:11.506 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:11.506 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:34:11.506 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6806", 2021-02-10T11:34:11.506 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36367 2021-02-10T11:34:11.506 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:11.506 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:11.506 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:34:11.506 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6809", 2021-02-10T11:34:11.506 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36367 2021-02-10T11:34:11.507 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:34:11.507 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:34:11.507 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:11.507 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addr": "172.21.2.113:6801/36367", 2021-02-10T11:34:11.507 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addr": "172.21.2.113:6804/36367", 2021-02-10T11:34:11.507 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addr": "172.21.2.113:6816/36367", 2021-02-10T11:34:11.507 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addr": "172.21.2.113:6809/36367", 2021-02-10T11:34:11.507 INFO:teuthology.orchestra.run.gibba002.stdout: "state": [ 2021-02-10T11:34:11.507 INFO:teuthology.orchestra.run.gibba002.stdout: "exists", 2021-02-10T11:34:11.507 INFO:teuthology.orchestra.run.gibba002.stdout: "up" 2021-02-10T11:34:11.507 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:34:11.507 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:11.507 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:11.507 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 7, 2021-02-10T11:34:11.508 INFO:teuthology.orchestra.run.gibba002.stdout: "uuid": "f5461f80-018d-428f-be89-2eb2bcbe7bb6", 2021-02-10T11:34:11.508 INFO:teuthology.orchestra.run.gibba002.stdout: "up": 1, 2021-02-10T11:34:11.508 INFO:teuthology.orchestra.run.gibba002.stdout: "in": 1, 2021-02-10T11:34:11.508 INFO:teuthology.orchestra.run.gibba002.stdout: "weight": 1, 2021-02-10T11:34:11.508 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_affinity": 1, 2021-02-10T11:34:11.508 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_begin": 0, 2021-02-10T11:34:11.508 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_end": 0, 2021-02-10T11:34:11.508 INFO:teuthology.orchestra.run.gibba002.stdout: "up_from": 13, 2021-02-10T11:34:11.508 INFO:teuthology.orchestra.run.gibba002.stdout: "up_thru": 125, 2021-02-10T11:34:11.508 INFO:teuthology.orchestra.run.gibba002.stdout: "down_at": 0, 2021-02-10T11:34:11.508 INFO:teuthology.orchestra.run.gibba002.stdout: "lost_at": 0, 2021-02-10T11:34:11.508 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addrs": { 2021-02-10T11:34:11.508 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:34:11.508 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:11.509 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:34:11.509 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6817", 2021-02-10T11:34:11.509 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36365 2021-02-10T11:34:11.509 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:11.509 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:11.509 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:34:11.509 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6821", 2021-02-10T11:34:11.509 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36365 2021-02-10T11:34:11.509 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:34:11.509 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:34:11.509 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:11.509 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addrs": { 2021-02-10T11:34:11.509 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:34:11.509 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:11.510 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:34:11.510 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6823", 2021-02-10T11:34:11.510 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36365 2021-02-10T11:34:11.510 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:11.510 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:11.510 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:34:11.510 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6825", 2021-02-10T11:34:11.510 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36365 2021-02-10T11:34:11.510 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:34:11.510 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:34:11.510 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:11.510 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addrs": { 2021-02-10T11:34:11.511 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:34:11.511 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:11.511 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:34:11.511 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6830", 2021-02-10T11:34:11.511 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36365 2021-02-10T11:34:11.511 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:11.511 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:11.511 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:34:11.511 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6831", 2021-02-10T11:34:11.511 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36365 2021-02-10T11:34:11.511 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:34:11.511 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:34:11.511 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:11.511 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addrs": { 2021-02-10T11:34:11.511 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:34:11.512 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:11.512 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:34:11.512 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6827", 2021-02-10T11:34:11.512 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36365 2021-02-10T11:34:11.512 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:11.512 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:11.512 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:34:11.512 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6829", 2021-02-10T11:34:11.512 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36365 2021-02-10T11:34:11.512 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:34:11.512 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:34:11.512 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:11.512 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addr": "172.21.2.113:6821/36365", 2021-02-10T11:34:11.513 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addr": "172.21.2.113:6825/36365", 2021-02-10T11:34:11.513 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addr": "172.21.2.113:6831/36365", 2021-02-10T11:34:11.513 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addr": "172.21.2.113:6829/36365", 2021-02-10T11:34:11.513 INFO:teuthology.orchestra.run.gibba002.stdout: "state": [ 2021-02-10T11:34:11.513 INFO:teuthology.orchestra.run.gibba002.stdout: "exists", 2021-02-10T11:34:11.513 INFO:teuthology.orchestra.run.gibba002.stdout: "up" 2021-02-10T11:34:11.513 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:34:11.513 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:34:11.513 INFO:teuthology.orchestra.run.gibba002.stdout: ], 2021-02-10T11:34:11.513 INFO:teuthology.orchestra.run.gibba002.stdout: "osd_xinfo": [ 2021-02-10T11:34:11.513 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:11.513 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 0, 2021-02-10T11:34:11.513 INFO:teuthology.orchestra.run.gibba002.stdout: "down_stamp": "0.000000", 2021-02-10T11:34:11.514 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_probability": 0, 2021-02-10T11:34:11.514 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_interval": 0, 2021-02-10T11:34:11.514 INFO:teuthology.orchestra.run.gibba002.stdout: "features": 4611087854035861503, 2021-02-10T11:34:11.514 INFO:teuthology.orchestra.run.gibba002.stdout: "old_weight": 0 2021-02-10T11:34:11.514 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:11.514 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:11.514 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 1, 2021-02-10T11:34:11.514 INFO:teuthology.orchestra.run.gibba002.stdout: "down_stamp": "0.000000", 2021-02-10T11:34:11.514 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_probability": 0, 2021-02-10T11:34:11.514 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_interval": 0, 2021-02-10T11:34:11.514 INFO:teuthology.orchestra.run.gibba002.stdout: "features": 4611087854035861503, 2021-02-10T11:34:11.514 INFO:teuthology.orchestra.run.gibba002.stdout: "old_weight": 0 2021-02-10T11:34:11.514 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:11.514 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:11.515 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 2, 2021-02-10T11:34:11.515 INFO:teuthology.orchestra.run.gibba002.stdout: "down_stamp": "0.000000", 2021-02-10T11:34:11.515 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_probability": 0, 2021-02-10T11:34:11.515 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_interval": 0, 2021-02-10T11:34:11.515 INFO:teuthology.orchestra.run.gibba002.stdout: "features": 4611087854035861503, 2021-02-10T11:34:11.515 INFO:teuthology.orchestra.run.gibba002.stdout: "old_weight": 0 2021-02-10T11:34:11.515 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:11.515 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:11.515 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 3, 2021-02-10T11:34:11.515 INFO:teuthology.orchestra.run.gibba002.stdout: "down_stamp": "0.000000", 2021-02-10T11:34:11.515 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_probability": 0, 2021-02-10T11:34:11.515 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_interval": 0, 2021-02-10T11:34:11.515 INFO:teuthology.orchestra.run.gibba002.stdout: "features": 4611087854035861503, 2021-02-10T11:34:11.516 INFO:teuthology.orchestra.run.gibba002.stdout: "old_weight": 0 2021-02-10T11:34:11.516 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:11.516 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:11.516 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 4, 2021-02-10T11:34:11.516 INFO:teuthology.orchestra.run.gibba002.stdout: "down_stamp": "0.000000", 2021-02-10T11:34:11.516 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_probability": 0, 2021-02-10T11:34:11.516 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_interval": 0, 2021-02-10T11:34:11.516 INFO:teuthology.orchestra.run.gibba002.stdout: "features": 4611087854035861503, 2021-02-10T11:34:11.516 INFO:teuthology.orchestra.run.gibba002.stdout: "old_weight": 0 2021-02-10T11:34:11.516 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:11.516 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:11.516 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 5, 2021-02-10T11:34:11.516 INFO:teuthology.orchestra.run.gibba002.stdout: "down_stamp": "0.000000", 2021-02-10T11:34:11.516 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_probability": 0, 2021-02-10T11:34:11.517 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_interval": 0, 2021-02-10T11:34:11.517 INFO:teuthology.orchestra.run.gibba002.stdout: "features": 4611087854035861503, 2021-02-10T11:34:11.517 INFO:teuthology.orchestra.run.gibba002.stdout: "old_weight": 0 2021-02-10T11:34:11.517 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:11.517 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:11.517 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 6, 2021-02-10T11:34:11.517 INFO:teuthology.orchestra.run.gibba002.stdout: "down_stamp": "0.000000", 2021-02-10T11:34:11.517 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_probability": 0, 2021-02-10T11:34:11.517 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_interval": 0, 2021-02-10T11:34:11.517 INFO:teuthology.orchestra.run.gibba002.stdout: "features": 4611087854035861503, 2021-02-10T11:34:11.517 INFO:teuthology.orchestra.run.gibba002.stdout: "old_weight": 0 2021-02-10T11:34:11.517 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:11.517 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:11.518 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 7, 2021-02-10T11:34:11.518 INFO:teuthology.orchestra.run.gibba002.stdout: "down_stamp": "0.000000", 2021-02-10T11:34:11.518 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_probability": 0, 2021-02-10T11:34:11.518 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_interval": 0, 2021-02-10T11:34:11.518 INFO:teuthology.orchestra.run.gibba002.stdout: "features": 4611087854035861503, 2021-02-10T11:34:11.518 INFO:teuthology.orchestra.run.gibba002.stdout: "old_weight": 0 2021-02-10T11:34:11.518 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:34:11.518 INFO:teuthology.orchestra.run.gibba002.stdout: ], 2021-02-10T11:34:11.518 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_upmap": [], 2021-02-10T11:34:11.518 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_upmap_items": [], 2021-02-10T11:34:11.518 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_temp": [], 2021-02-10T11:34:11.518 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_temp": [], 2021-02-10T11:34:11.518 INFO:teuthology.orchestra.run.gibba002.stdout: "blacklist": {}, 2021-02-10T11:34:11.518 INFO:teuthology.orchestra.run.gibba002.stdout: "erasure_code_profiles": { 2021-02-10T11:34:11.519 INFO:teuthology.orchestra.run.gibba002.stdout: "default": { 2021-02-10T11:34:11.519 INFO:teuthology.orchestra.run.gibba002.stdout: "crush-failure-domain": "osd", 2021-02-10T11:34:11.519 INFO:teuthology.orchestra.run.gibba002.stdout: "k": "2", 2021-02-10T11:34:11.519 INFO:teuthology.orchestra.run.gibba002.stdout: "m": "1", 2021-02-10T11:34:11.519 INFO:teuthology.orchestra.run.gibba002.stdout: "plugin": "jerasure", 2021-02-10T11:34:11.519 INFO:teuthology.orchestra.run.gibba002.stdout: "ruleset-failure-domain": "osd", 2021-02-10T11:34:11.519 INFO:teuthology.orchestra.run.gibba002.stdout: "technique": "reed_sol_van" 2021-02-10T11:34:11.519 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:34:11.519 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:11.519 INFO:teuthology.orchestra.run.gibba002.stdout: "removed_snaps_queue": [], 2021-02-10T11:34:11.519 INFO:teuthology.orchestra.run.gibba002.stdout: "new_removed_snaps": [], 2021-02-10T11:34:11.519 INFO:teuthology.orchestra.run.gibba002.stdout: "new_purged_snaps": [], 2021-02-10T11:34:11.519 INFO:teuthology.orchestra.run.gibba002.stdout: "crush_node_flags": {}, 2021-02-10T11:34:11.519 INFO:teuthology.orchestra.run.gibba002.stdout: "device_class_flags": {} 2021-02-10T11:34:11.520 INFO:teuthology.orchestra.run.gibba002.stdout:} 2021-02-10T11:34:11.521 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:11.446 7f59b813a700 1 -- 172.21.2.102:0/813622409 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5998018730 msgr2=0x7f599801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:11.521 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:11.446 7f59b813a700 1 --2- 172.21.2.102:0/813622409 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5998018730 0x7f599801ac10 crc :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:11.521 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:11.446 7f59b813a700 1 -- 172.21.2.102:0/813622409 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f59b00adbd0 msgr2=0x7f59b00a9a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:11.521 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:11.446 7f59b813a700 1 --2- 172.21.2.102:0/813622409 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f59b00adbd0 0x7f59b00a9a70 secure :-1 s=READY pgs=2500 cs=0 l=1 rev1=1 rx=0x7f59a0001c60 tx=0x7f59a000a8d0).stop 2021-02-10T11:34:11.521 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:11.446 7f59b813a700 1 -- 172.21.2.102:0/813622409 shutdown_connections 2021-02-10T11:34:11.521 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:11.446 7f59b813a700 1 --2- 172.21.2.102:0/813622409 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f59b00adfd0 0x7f59b00aa090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:11.521 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:11.446 7f59b813a700 1 --2- 172.21.2.102:0/813622409 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f59b00adbd0 0x7f59b00a9a70 unknown :-1 s=CLOSED pgs=2500 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:11.521 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:11.446 7f59b813a700 1 --2- 172.21.2.102:0/813622409 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5998018730 0x7f599801ac10 unknown :-1 s=CLOSED pgs=1106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:11.521 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:11.446 7f59b813a700 1 --2- 172.21.2.102:0/813622409 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f59b00ae6b0 0x7f59b00a8bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:11.521 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:11.446 7f59b813a700 1 -- 172.21.2.102:0/813622409 >> 172.21.2.102:0/813622409 conn(0x7f59b00a0030 msgr2=0x7f59b00eb1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:11.522 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:11.446 7f59b813a700 1 -- 172.21.2.102:0/813622409 shutdown_connections 2021-02-10T11:34:11.522 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:11.446 7f59b813a700 1 -- 172.21.2.102:0/813622409 wait complete. 2021-02-10T11:34:11.522 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:34:11.949 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:11.931 7f809c268700 1 Processor -- start 2021-02-10T11:34:11.949 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:11.931 7f809c268700 1 -- start start 2021-02-10T11:34:11.949 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:11.931 7f809c268700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f80940d1870 0x7f80941b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:11.949 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:11.931 7f809c268700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f80940f0650 0x7f80940cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:11.950 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:11.931 7f809c268700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80940f0b10 con 0x7f80941b0a10 2021-02-10T11:34:11.950 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:11.931 7f809c268700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80940e5ed0 con 0x7f80940d1870 2021-02-10T11:34:11.950 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:11.931 7f809c268700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80940d1c70 con 0x7f80940f0650 2021-02-10T11:34:11.950 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:11.931 7f809a084700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f80940d1870 0x7f80941b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:11.950 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:11.932 7f809a084700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f80940d1870 0x7f80941b03f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:35844/0 (socket says 172.21.2.102:35844) 2021-02-10T11:34:11.950 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:11.932 7f809a084700 1 -- 172.21.2.102:0/2416264003 learned_addr learned my addr 172.21.2.102:0/2416264003 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:34:11.950 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:11.932 7f809a084700 1 -- 172.21.2.102:0/2416264003 >> v2:172.21.2.113:3300/0 conn(0x7f80940f0650 msgr2=0x7f80940cfb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:34:11.950 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:11.932 7f809a084700 1 --2- 172.21.2.102:0/2416264003 >> v2:172.21.2.113:3300/0 conn(0x7f80940f0650 0x7f80940cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:11.950 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:11.932 7f809a084700 1 -- 172.21.2.102:0/2416264003 >> v1:172.21.2.113:6789/0 conn(0x7f80941b0a10 legacy=0x7f80940cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:34:11.950 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:11.932 7f809a084700 1 -- 172.21.2.102:0/2416264003 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80940d1e90 con 0x7f80940d1870 2021-02-10T11:34:11.950 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:11.932 7f809a084700 1 --2- 172.21.2.102:0/2416264003 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f80940d1870 0x7f80941b03f0 secure :-1 s=READY pgs=2501 cs=0 l=1 rev1=1 rx=0x7f8088006700 tx=0x7f8088009b80).ready entity=mon.0 client_cookie=e69bbe385b5f62de server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:11.950 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:11.932 7f8099082700 1 -- 172.21.2.102:0/2416264003 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f808800c0c0 con 0x7f80940d1870 2021-02-10T11:34:11.951 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:11.932 7f8099082700 1 -- 172.21.2.102:0/2416264003 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f808800c340 con 0x7f80940d1870 2021-02-10T11:34:11.951 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:11.932 7f8099082700 1 -- 172.21.2.102:0/2416264003 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f808800c860 con 0x7f80940d1870 2021-02-10T11:34:11.951 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:11.932 7f809c268700 1 -- 172.21.2.102:0/2416264003 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f80940d1870 msgr2=0x7f80941b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:11.951 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:11.932 7f809c268700 1 --2- 172.21.2.102:0/2416264003 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f80940d1870 0x7f80941b03f0 secure :-1 s=READY pgs=2501 cs=0 l=1 rev1=1 rx=0x7f8088006700 tx=0x7f8088009b80).stop 2021-02-10T11:34:11.951 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:11.933 7f809c268700 1 -- 172.21.2.102:0/2416264003 shutdown_connections 2021-02-10T11:34:11.951 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:11.933 7f809c268700 1 --2- 172.21.2.102:0/2416264003 >> v2:172.21.2.113:3300/0 conn(0x7f80940f0650 0x7f80940cfb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:11.951 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:11.933 7f809c268700 1 --2- 172.21.2.102:0/2416264003 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f80940d1870 0x7f80941b03f0 unknown :-1 s=CLOSED pgs=2501 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:11.951 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:11.933 7f809c268700 1 -- 172.21.2.102:0/2416264003 >> 172.21.2.102:0/2416264003 conn(0x7f80940ce450 msgr2=0x7f80940d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:11.951 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:11.933 7f809c268700 1 -- 172.21.2.102:0/2416264003 shutdown_connections 2021-02-10T11:34:11.951 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:11.933 7f809c268700 1 -- 172.21.2.102:0/2416264003 wait complete. 2021-02-10T11:34:11.953 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:11.948 7f809c268700 1 Processor -- start 2021-02-10T11:34:11.953 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:11.948 7f809c268700 1 -- start start 2021-02-10T11:34:11.953 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:11.948 7f809c268700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f809419af00 0x7f809419b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:11.953 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:11.948 7f809c268700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f809419ba80 0x7f809419be80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:11.953 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:11.948 7f809c268700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f809419c4a0 0x7f809408a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:11.953 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:11.948 7f809c268700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f809408fb50 con 0x7f809419c4a0 2021-02-10T11:34:11.953 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:11.948 7f809c268700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f809408afb0 con 0x7f809419af00 2021-02-10T11:34:11.953 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:11.948 7f809c268700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f809408b1a0 con 0x7f809419ba80 2021-02-10T11:34:11.953 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:11.948 7f809a885700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f809419c4a0 0x7f809408a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:11.954 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:11.948 7f809a885700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f809419c4a0 0x7f809408a990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:35850/0 (socket says 172.21.2.102:35850) 2021-02-10T11:34:11.954 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:11.948 7f809a885700 1 -- 172.21.2.102:0/2112834197 learned_addr learned my addr 172.21.2.102:0/2112834197 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:34:11.954 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:11.949 7f809a885700 1 -- 172.21.2.102:0/2112834197 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f809419ba80 msgr2=0x7f809419be80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:11.954 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:11.949 7f809a885700 1 --2- 172.21.2.102:0/2112834197 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f809419ba80 0x7f809419be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:11.954 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:11.949 7f809a885700 1 -- 172.21.2.102:0/2112834197 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f809419af00 msgr2=0x7f809419b300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:11.954 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:11.949 7f809a885700 1 --2- 172.21.2.102:0/2112834197 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f809419af00 0x7f809419b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:11.954 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:11.949 7f809a885700 1 -- 172.21.2.102:0/2112834197 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f809408b3c0 con 0x7f809419c4a0 2021-02-10T11:34:11.954 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:11.949 7f809a885700 1 --2- 172.21.2.102:0/2112834197 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f809419c4a0 0x7f809408a990 secure :-1 s=READY pgs=2502 cs=0 l=1 rev1=1 rx=0x7f808c009980 tx=0x7f808c00bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:11.954 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:11.949 7f8086ffd700 1 -- 172.21.2.102:0/2112834197 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f808c005870 con 0x7f809419c4a0 2021-02-10T11:34:11.954 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:11.949 7f8086ffd700 1 -- 172.21.2.102:0/2112834197 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f808c005af0 con 0x7f809419c4a0 2021-02-10T11:34:11.954 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:11.949 7f809c268700 1 -- 172.21.2.102:0/2112834197 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f809408b3c0 con 0x7f809419c4a0 2021-02-10T11:34:11.955 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:11.949 7f8086ffd700 1 -- 172.21.2.102:0/2112834197 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f808c006030 con 0x7f809419c4a0 2021-02-10T11:34:11.955 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:11.949 7f809c268700 1 -- 172.21.2.102:0/2112834197 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f809408b950 con 0x7f809419c4a0 2021-02-10T11:34:11.955 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:11.950 7f8086ffd700 1 -- 172.21.2.102:0/2112834197 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f808c005cd0 con 0x7f809419c4a0 2021-02-10T11:34:11.955 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:11.950 7f8086ffd700 1 --2- 172.21.2.102:0/2112834197 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f807c0186c0 0x7f807c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:11.955 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:11.950 7f8086ffd700 1 -- 172.21.2.102:0/2112834197 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f808c032750 con 0x7f809419c4a0 2021-02-10T11:34:11.955 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:11.950 7f809a084700 1 --2- 172.21.2.102:0/2112834197 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f807c0186c0 0x7f807c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:11.955 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:11.950 7f809a084700 1 --2- 172.21.2.102:0/2112834197 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f807c0186c0 0x7f807c01abc0 crc :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:11.983 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:11.981 7f809c268700 1 -- 172.21.2.102:0/2112834197 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80940ee030 con 0x7f809419c4a0 2021-02-10T11:34:11.985 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:11.984 7f8086ffd700 1 -- 172.21.2.102:0/2112834197 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f808c0062f0 con 0x7f809419c4a0 2021-02-10T11:34:12.114 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:12.112 7f809c268700 1 -- 172.21.2.102:0/2112834197 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f80940ce450 con 0x7f809419c4a0 2021-02-10T11:34:12.114 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:12.113 7f8086ffd700 1 -- 172.21.2.102:0/2112834197 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+2102 (secure 0 0 0) 0x7f808c005c90 con 0x7f809419c4a0 2021-02-10T11:34:12.115 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:34:12.115 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":108,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":11103,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.102:6835/160551241","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6834","nonce":160551241},{"type":"v1","addr":"172.21.2.102:6835","nonce":160551241}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":103}],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:33:32.844442","modified":"2021-02-10 11:33:34.850935","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7882},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7882":{"gid":7882,"name":"b","rank":0,"incarnation":104,"state":"up:active","state_seq":3,"addr":"172.21.2.113:6833/121050001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6832","nonce":121050001},{"type":"v1","addr":"172.21.2.113:6833","nonce":121050001}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[27],"metadata_pool":26,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":13}]} 2021-02-10T11:34:12.115 INFO:teuthology.orchestra.run.gibba002.stderr:dumped fsmap epoch 108 2021-02-10T11:34:12.116 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:12.114 7f809c268700 1 -- 172.21.2.102:0/2112834197 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f807c0186c0 msgr2=0x7f807c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:12.116 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:12.114 7f809c268700 1 --2- 172.21.2.102:0/2112834197 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f807c0186c0 0x7f807c01abc0 crc :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:12.117 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:12.114 7f809c268700 1 -- 172.21.2.102:0/2112834197 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f809419c4a0 msgr2=0x7f809408a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:12.117 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:12.114 7f809c268700 1 --2- 172.21.2.102:0/2112834197 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f809419c4a0 0x7f809408a990 secure :-1 s=READY pgs=2502 cs=0 l=1 rev1=1 rx=0x7f808c009980 tx=0x7f808c00bcd0).stop 2021-02-10T11:34:12.117 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:12.115 7f809c268700 1 -- 172.21.2.102:0/2112834197 shutdown_connections 2021-02-10T11:34:12.117 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:12.115 7f809c268700 1 --2- 172.21.2.102:0/2112834197 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f809419c4a0 0x7f809408a990 unknown :-1 s=CLOSED pgs=2502 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:12.117 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:12.115 7f809c268700 1 --2- 172.21.2.102:0/2112834197 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f809419ba80 0x7f809419be80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:12.117 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:12.115 7f809c268700 1 --2- 172.21.2.102:0/2112834197 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f807c0186c0 0x7f807c01abc0 unknown :-1 s=CLOSED pgs=1107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:12.117 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:12.115 7f809c268700 1 --2- 172.21.2.102:0/2112834197 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f809419af00 0x7f809419b300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:12.117 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:12.115 7f809c268700 1 -- 172.21.2.102:0/2112834197 >> 172.21.2.102:0/2112834197 conn(0x7f809419f8c0 msgr2=0x7f80941a5fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:12.117 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:12.115 7f809c268700 1 -- 172.21.2.102:0/2112834197 shutdown_connections 2021-02-10T11:34:12.117 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:12.115 7f809c268700 1 -- 172.21.2.102:0/2112834197 wait complete. 2021-02-10T11:34:12.125 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs fail cephfs 2021-02-10T11:34:12.553 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:12.535 7f7ac66c7700 1 Processor -- start 2021-02-10T11:34:12.553 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:12.535 7f7ac66c7700 1 -- start start 2021-02-10T11:34:12.553 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:12.535 7f7ac66c7700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7ac00d1870 0x7f7ac01b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:12.553 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:12.535 7f7ac66c7700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f7ac00f0650 0x7f7ac00cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:12.553 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:12.535 7f7ac66c7700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ac00f0b10 con 0x7f7ac01b0a10 2021-02-10T11:34:12.553 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:12.535 7f7ac66c7700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ac00fe230 con 0x7f7ac00f0650 2021-02-10T11:34:12.553 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:12.535 7f7ac66c7700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ac00e5ed0 con 0x7f7ac00d1870 2021-02-10T11:34:12.553 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:12.535 7f7abf7fe700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f7ac01b0a10 0x7f7ac00cf330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:56314/0 (socket says 172.21.2.102:56314) 2021-02-10T11:34:12.554 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:12.536 7f7abf7fe700 1 -- 172.21.2.102:0/1028056501 learned_addr learned my addr 172.21.2.102:0/1028056501 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:34:12.554 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:12.536 7f7ac4ce4700 1 --2- 172.21.2.102:0/1028056501 >> v2:172.21.2.113:3300/0 conn(0x7f7ac00f0650 0x7f7ac00cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:12.554 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:12.536 7f7abffff700 1 --2- 172.21.2.102:0/1028056501 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7ac00d1870 0x7f7ac01b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:12.554 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:12.536 7f7abeffd700 1 -- 172.21.2.102:0/1028056501 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1989026823 0 0) 0x7f7ab40009b0 con 0x7f7ac01b0a10 2021-02-10T11:34:12.554 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:12.536 7f7ac4ce4700 1 -- 172.21.2.102:0/1028056501 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7ac00d1870 msgr2=0x7f7ac01b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:34:12.554 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:12.536 7f7ac4ce4700 1 --2- 172.21.2.102:0/1028056501 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7ac00d1870 0x7f7ac01b03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:12.554 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:12.536 7f7ac4ce4700 1 -- 172.21.2.102:0/1028056501 >> v1:172.21.2.102:6789/0 conn(0x7f7ac01b0a10 legacy=0x7f7ac00cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:12.554 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:12.536 7f7ac4ce4700 1 -- 172.21.2.102:0/1028056501 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ac00f20a0 con 0x7f7ac00f0650 2021-02-10T11:34:12.554 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:12.536 7f7ac4ce4700 1 --2- 172.21.2.102:0/1028056501 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7ac00f0650 0x7f7ac00cfb20 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 rx=0x7f7ab8006e50 tx=0x7f7ab8009e70).ready entity=mon.1 client_cookie=4b9c793c6d465525 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:12.554 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:12.536 7f7abeffd700 1 -- 172.21.2.102:0/1028056501 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7ab8006a20 con 0x7f7ac00f0650 2021-02-10T11:34:12.554 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:12.536 7f7abeffd700 1 -- 172.21.2.102:0/1028056501 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ab8006790 con 0x7f7ac00f0650 2021-02-10T11:34:12.554 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:12.536 7f7abeffd700 1 -- 172.21.2.102:0/1028056501 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7ab800b700 con 0x7f7ac00f0650 2021-02-10T11:34:12.555 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:12.536 7f7ac66c7700 1 -- 172.21.2.102:0/1028056501 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7ac00f0650 msgr2=0x7f7ac00cfb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:12.555 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:12.536 7f7ac66c7700 1 --2- 172.21.2.102:0/1028056501 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7ac00f0650 0x7f7ac00cfb20 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 rx=0x7f7ab8006e50 tx=0x7f7ab8009e70).stop 2021-02-10T11:34:12.555 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:12.537 7f7ac66c7700 1 -- 172.21.2.102:0/1028056501 shutdown_connections 2021-02-10T11:34:12.555 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:12.537 7f7ac66c7700 1 --2- 172.21.2.102:0/1028056501 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7ac00f0650 0x7f7ac00cfb20 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:12.555 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:12.537 7f7ac66c7700 1 --2- 172.21.2.102:0/1028056501 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7ac00d1870 0x7f7ac01b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:12.555 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:12.537 7f7ac66c7700 1 -- 172.21.2.102:0/1028056501 >> 172.21.2.102:0/1028056501 conn(0x7f7ac00ce450 msgr2=0x7f7ac00d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:12.555 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:12.537 7f7ac66c7700 1 -- 172.21.2.102:0/1028056501 shutdown_connections 2021-02-10T11:34:12.555 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:12.537 7f7ac66c7700 1 -- 172.21.2.102:0/1028056501 wait complete. 2021-02-10T11:34:12.556 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:12.551 7f7ac66c7700 1 Processor -- start 2021-02-10T11:34:12.557 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:12.552 7f7ac66c7700 1 -- start start 2021-02-10T11:34:12.557 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:12.552 7f7ac66c7700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7ac01a06a0 0x7f7ac009f380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:12.557 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:12.552 7f7ac66c7700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7ac009f9a0 0x7f7ac008b560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:12.557 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:12.552 7f7ac66c7700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7ac008bb80 0x7f7ac008bf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:12.557 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:12.552 7f7ac66c7700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ac008c780 con 0x7f7ac009f9a0 2021-02-10T11:34:12.557 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:12.552 7f7ac66c7700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ac009fda0 con 0x7f7ac01a06a0 2021-02-10T11:34:12.557 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:12.552 7f7ac66c7700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ac008cd00 con 0x7f7ac008bb80 2021-02-10T11:34:12.557 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:12.552 7f7abf7fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7ac009f9a0 0x7f7ac008b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:12.557 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:12.552 7f7abf7fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7ac009f9a0 0x7f7ac008b560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:35862/0 (socket says 172.21.2.102:35862) 2021-02-10T11:34:12.557 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:12.552 7f7abf7fe700 1 -- 172.21.2.102:0/877512827 learned_addr learned my addr 172.21.2.102:0/877512827 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:34:12.557 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:12.553 7f7abf7fe700 1 -- 172.21.2.102:0/877512827 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7ac008bb80 msgr2=0x7f7ac008bf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:12.557 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:12.553 7f7abf7fe700 1 --2- 172.21.2.102:0/877512827 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7ac008bb80 0x7f7ac008bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:12.557 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:12.553 7f7abf7fe700 1 -- 172.21.2.102:0/877512827 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7ac01a06a0 msgr2=0x7f7ac009f380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:12.558 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:12.553 7f7abf7fe700 1 --2- 172.21.2.102:0/877512827 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7ac01a06a0 0x7f7ac009f380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:12.558 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:12.553 7f7abf7fe700 1 -- 172.21.2.102:0/877512827 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ac008cf20 con 0x7f7ac009f9a0 2021-02-10T11:34:12.558 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:12.553 7f7abf7fe700 1 --2- 172.21.2.102:0/877512827 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7ac009f9a0 0x7f7ac008b560 secure :-1 s=READY pgs=2504 cs=0 l=1 rev1=1 rx=0x7f7ab4001c60 tx=0x7f7ab400a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:12.558 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:12.553 7f7abcff9700 1 -- 172.21.2.102:0/877512827 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7ab400acc0 con 0x7f7ac009f9a0 2021-02-10T11:34:12.558 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:12.553 7f7abcff9700 1 -- 172.21.2.102:0/877512827 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ab4004140 con 0x7f7ac009f9a0 2021-02-10T11:34:12.558 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:12.553 7f7abcff9700 1 -- 172.21.2.102:0/877512827 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7ab4004660 con 0x7f7ac009f9a0 2021-02-10T11:34:12.558 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:12.553 7f7ac66c7700 1 -- 172.21.2.102:0/877512827 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ac008cf20 con 0x7f7ac009f9a0 2021-02-10T11:34:12.558 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:12.553 7f7ac66c7700 1 -- 172.21.2.102:0/877512827 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ac008d4b0 con 0x7f7ac009f9a0 2021-02-10T11:34:12.558 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:12.554 7f7abcff9700 1 -- 172.21.2.102:0/877512827 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f7ab4004320 con 0x7f7ac009f9a0 2021-02-10T11:34:12.558 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:12.554 7f7abcff9700 1 --2- 172.21.2.102:0/877512827 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7aa00186c0 0x7f7aa001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:12.558 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:12.554 7f7abcff9700 1 -- 172.21.2.102:0/877512827 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f7ab4032560 con 0x7f7ac009f9a0 2021-02-10T11:34:12.558 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:12.554 7f7abffff700 1 --2- 172.21.2.102:0/877512827 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7aa00186c0 0x7f7aa001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:12.558 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:12.554 7f7abffff700 1 --2- 172.21.2.102:0/877512827 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7aa00186c0 0x7f7aa001abc0 crc :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:12.587 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:12.585 7f7ac66c7700 1 -- 172.21.2.102:0/877512827 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ac00ebc50 con 0x7f7ac009f9a0 2021-02-10T11:34:12.590 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:12.588 7f7abcff9700 1 -- 172.21.2.102:0/877512827 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f7ab40009b0 con 0x7f7ac009f9a0 2021-02-10T11:34:12.718 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:12.716 7f7ac66c7700 1 -- 172.21.2.102:0/877512827 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f7ac00f0b10 con 0x7f7ac009f9a0 2021-02-10T11:34:13.505 INFO:tasks.ceph.mon.a.gibba002.stderr:2021-02-10 11:34:13.503 7f8dd2755700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-02-10T11:34:13.510 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:13.509 7f7abcff9700 1 -- 172.21.2.102:0/877512827 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v109) v1 ==== 161+0+0 (secure 0 0 0) 0x7f7ab4004320 con 0x7f7ac009f9a0 2021-02-10T11:34:13.536 INFO:teuthology.orchestra.run.gibba002.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2021-02-10T11:34:13.538 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:13.536 7f7ac66c7700 1 -- 172.21.2.102:0/877512827 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7aa00186c0 msgr2=0x7f7aa001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:13.538 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:13.536 7f7ac66c7700 1 --2- 172.21.2.102:0/877512827 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7aa00186c0 0x7f7aa001abc0 crc :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:13.538 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:13.536 7f7ac66c7700 1 -- 172.21.2.102:0/877512827 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7ac009f9a0 msgr2=0x7f7ac008b560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:13.538 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:13.536 7f7ac66c7700 1 --2- 172.21.2.102:0/877512827 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7ac009f9a0 0x7f7ac008b560 secure :-1 s=READY pgs=2504 cs=0 l=1 rev1=1 rx=0x7f7ab4001c60 tx=0x7f7ab400a8d0).stop 2021-02-10T11:34:13.538 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:13.536 7f7ac66c7700 1 -- 172.21.2.102:0/877512827 shutdown_connections 2021-02-10T11:34:13.538 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:13.536 7f7ac66c7700 1 --2- 172.21.2.102:0/877512827 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7ac008bb80 0x7f7ac008bf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:13.538 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:13.536 7f7ac66c7700 1 --2- 172.21.2.102:0/877512827 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7ac009f9a0 0x7f7ac008b560 unknown :-1 s=CLOSED pgs=2504 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:13.538 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:13.536 7f7ac66c7700 1 --2- 172.21.2.102:0/877512827 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7aa00186c0 0x7f7aa001abc0 unknown :-1 s=CLOSED pgs=1108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:13.539 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:13.536 7f7ac66c7700 1 --2- 172.21.2.102:0/877512827 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7ac01a06a0 0x7f7ac009f380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:13.539 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:13.536 7f7ac66c7700 1 -- 172.21.2.102:0/877512827 >> 172.21.2.102:0/877512827 conn(0x7f7ac00b83b0 msgr2=0x7f7ac00a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:13.539 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:13.537 7f7ac66c7700 1 -- 172.21.2.102:0/877512827 shutdown_connections 2021-02-10T11:34:13.539 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:13.537 7f7ac66c7700 1 -- 172.21.2.102:0/877512827 wait complete. 2021-02-10T11:34:13.547 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:34:13.562 INFO:tasks.ceph.mds.b.gibba013.stdout:starting mds.b at 2021-02-10T11:34:13.978 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:13.959 7f09c4cab700 1 Processor -- start 2021-02-10T11:34:13.978 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:13.960 7f09c4cab700 1 -- start start 2021-02-10T11:34:13.978 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:13.960 7f09c4cab700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f09c01b0470 0x7f09c01b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:13.978 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:13.960 7f09c4cab700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f09c00f20e0 0x7f09c00f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:13.979 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:13.960 7f09c4cab700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09c00f0b50 con 0x7f09c00ef610 2021-02-10T11:34:13.979 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:13.960 7f09c4cab700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09c00f0820 con 0x7f09c00f20e0 2021-02-10T11:34:13.979 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:13.960 7f09c4cab700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09c00f24e0 con 0x7f09c01b0470 2021-02-10T11:34:13.979 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:13.960 7f09bf05f700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f09c00f20e0 0x7f09c00f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:13.979 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:13.960 7f09bf05f700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f09c00f20e0 0x7f09c00f0200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:35868/0 (socket says 172.21.2.102:35868) 2021-02-10T11:34:13.979 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:13.960 7f09bf05f700 1 -- 172.21.2.102:0/963403122 learned_addr learned my addr 172.21.2.102:0/963403122 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:34:13.979 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:13.961 7f09bf05f700 1 -- 172.21.2.102:0/963403122 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f09c01b0470 msgr2=0x7f09c01b0870 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:34:13.979 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:13.961 7f09bf05f700 1 --2- 172.21.2.102:0/963403122 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f09c01b0470 0x7f09c01b0870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:13.979 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:13.961 7f09bf05f700 1 -- 172.21.2.102:0/963403122 >> v1:172.21.2.113:6789/0 conn(0x7f09c00ef610 legacy=0x7f09c00efa10 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:34:13.979 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:13.961 7f09bf05f700 1 -- 172.21.2.102:0/963403122 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09c00e52e0 con 0x7f09c00f20e0 2021-02-10T11:34:13.979 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:13.961 7f09bf05f700 1 --2- 172.21.2.102:0/963403122 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f09c00f20e0 0x7f09c00f0200 secure :-1 s=READY pgs=2505 cs=0 l=1 rev1=1 rx=0x7f09b8006f00 tx=0x7f09b8009f10).ready entity=mon.0 client_cookie=e99857c87af8bb34 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:13.979 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:13.961 7f09bd85c700 1 -- 172.21.2.102:0/963403122 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f09b800b3b0 con 0x7f09c00f20e0 2021-02-10T11:34:13.980 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:13.961 7f09bd85c700 1 -- 172.21.2.102:0/963403122 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09b800b630 con 0x7f09c00f20e0 2021-02-10T11:34:13.980 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:13.961 7f09bd85c700 1 -- 172.21.2.102:0/963403122 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f09b800bb50 con 0x7f09c00f20e0 2021-02-10T11:34:13.980 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:13.961 7f09c4cab700 1 -- 172.21.2.102:0/963403122 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f09c00f20e0 msgr2=0x7f09c00f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:13.980 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:13.961 7f09c4cab700 1 --2- 172.21.2.102:0/963403122 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f09c00f20e0 0x7f09c00f0200 secure :-1 s=READY pgs=2505 cs=0 l=1 rev1=1 rx=0x7f09b8006f00 tx=0x7f09b8009f10).stop 2021-02-10T11:34:13.980 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:13.962 7f09c4cab700 1 -- 172.21.2.102:0/963403122 shutdown_connections 2021-02-10T11:34:13.980 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:13.962 7f09c4cab700 1 --2- 172.21.2.102:0/963403122 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f09c00f20e0 0x7f09c00f0200 unknown :-1 s=CLOSED pgs=2505 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:13.980 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:13.962 7f09c4cab700 1 --2- 172.21.2.102:0/963403122 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f09c01b0470 0x7f09c01b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:13.980 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:13.962 7f09c4cab700 1 -- 172.21.2.102:0/963403122 >> 172.21.2.102:0/963403122 conn(0x7f09c00ce3e0 msgr2=0x7f09c00e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:13.980 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:13.962 7f09c4cab700 1 -- 172.21.2.102:0/963403122 shutdown_connections 2021-02-10T11:34:13.980 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:13.962 7f09c4cab700 1 -- 172.21.2.102:0/963403122 wait complete. 2021-02-10T11:34:13.982 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:13.977 7f09c4cab700 1 Processor -- start 2021-02-10T11:34:13.982 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:13.977 7f09c4cab700 1 -- start start 2021-02-10T11:34:13.982 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:13.977 7f09c4cab700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f09c0092e40 0x7f09c0093240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:13.982 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:13.977 7f09c4cab700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f09c0093860 0x7f09c01a83d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:13.982 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:13.977 7f09c4cab700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f09c01a89f0 0x7f09c01a8df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:13.982 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:13.977 7f09c4cab700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09c0195ad0 con 0x7f09c0092e40 2021-02-10T11:34:13.982 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:13.977 7f09c4cab700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09c01a9450 con 0x7f09c01a89f0 2021-02-10T11:34:13.982 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:13.977 7f09c4cab700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09c00b1b00 con 0x7f09c0093860 2021-02-10T11:34:13.982 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:13.977 7f09be85e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f09c0092e40 0x7f09c0093240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:13.982 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:13.977 7f09be85e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f09c0092e40 0x7f09c0093240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:35874/0 (socket says 172.21.2.102:35874) 2021-02-10T11:34:13.982 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:13.977 7f09be85e700 1 -- 172.21.2.102:0/1865903392 learned_addr learned my addr 172.21.2.102:0/1865903392 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:34:13.982 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:13.978 7f09be85e700 1 -- 172.21.2.102:0/1865903392 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f09c0093860 msgr2=0x7f09c01a83d0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:34:13.983 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:13.978 7f09be85e700 1 --2- 172.21.2.102:0/1865903392 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f09c0093860 0x7f09c01a83d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:13.983 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:13.978 7f09be85e700 1 -- 172.21.2.102:0/1865903392 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f09c01a89f0 msgr2=0x7f09c01a8df0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:34:13.983 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:13.978 7f09be85e700 1 --2- 172.21.2.102:0/1865903392 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f09c01a89f0 0x7f09c01a8df0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:13.983 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:13.978 7f09be85e700 1 -- 172.21.2.102:0/1865903392 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09c01a99d0 con 0x7f09c0092e40 2021-02-10T11:34:13.983 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:13.978 7f09be85e700 1 --2- 172.21.2.102:0/1865903392 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f09c0092e40 0x7f09c0093240 secure :-1 s=READY pgs=2506 cs=0 l=1 rev1=1 rx=0x7f09b000b5c0 tx=0x7f09b000b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:13.983 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:13.978 7f09af7fe700 1 -- 172.21.2.102:0/1865903392 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f09b000bd50 con 0x7f09c0092e40 2021-02-10T11:34:13.983 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:13.978 7f09af7fe700 1 -- 172.21.2.102:0/1865903392 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09b0006030 con 0x7f09c0092e40 2021-02-10T11:34:13.983 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:13.978 7f09af7fe700 1 -- 172.21.2.102:0/1865903392 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f09b00054d0 con 0x7f09c0092e40 2021-02-10T11:34:13.983 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:13.978 7f09c4cab700 1 -- 172.21.2.102:0/1865903392 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09c01a99d0 con 0x7f09c0092e40 2021-02-10T11:34:13.983 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:13.978 7f09c4cab700 1 -- 172.21.2.102:0/1865903392 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09c01a99d0 con 0x7f09c0092e40 2021-02-10T11:34:13.983 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:13.978 7f09af7fe700 1 -- 172.21.2.102:0/1865903392 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f09b0006210 con 0x7f09c0092e40 2021-02-10T11:34:13.983 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:13.978 7f09af7fe700 1 --2- 172.21.2.102:0/1865903392 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f09a4018730 0x7f09a401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:13.984 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:13.978 7f09af7fe700 1 -- 172.21.2.102:0/1865903392 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5659+0+0 (secure 0 0 0) 0x7f09b0032810 con 0x7f09c0092e40 2021-02-10T11:34:13.984 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:13.979 7f09be05d700 1 --2- 172.21.2.102:0/1865903392 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f09a4018730 0x7f09a401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:13.984 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:13.979 7f09be05d700 1 --2- 172.21.2.102:0/1865903392 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f09a4018730 0x7f09a401abc0 crc :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:14.012 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:14.010 7f09c4cab700 1 -- 172.21.2.102:0/1865903392 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09c00b1b00 con 0x7f09c0092e40 2021-02-10T11:34:14.014 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:14.012 7f09af7fe700 1 -- 172.21.2.102:0/1865903392 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f09b0006b80 con 0x7f09c0092e40 2021-02-10T11:34:14.144 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:14.143 7f09c4cab700 1 -- 172.21.2.102:0/1865903392 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f09c00b1b00 con 0x7f09c0092e40 2021-02-10T11:34:14.145 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:14.143 7f09af7fe700 1 -- 172.21.2.102:0/1865903392 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 109 v109) v1 ==== 95+0+1760 (secure 0 0 0) 0x7f09b000bd50 con 0x7f09c0092e40 2021-02-10T11:34:14.145 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:34:14.146 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":109,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":11103,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.102:6835/160551241","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6834","nonce":160551241},{"type":"v1","addr":"172.21.2.102:6835","nonce":160551241}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":103}],"filesystems":[{"mdsmap":{"epoch":109,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:33:32.844442","modified":"2021-02-10 11:34:13.504347","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":129,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[27],"metadata_pool":26,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":13}]} 2021-02-10T11:34:14.146 INFO:teuthology.orchestra.run.gibba002.stderr:dumped fsmap epoch 109 2021-02-10T11:34:14.146 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:14.145 7f09c4cab700 1 -- 172.21.2.102:0/1865903392 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f09a4018730 msgr2=0x7f09a401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:14.147 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:14.145 7f09c4cab700 1 --2- 172.21.2.102:0/1865903392 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f09a4018730 0x7f09a401abc0 crc :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:14.147 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:14.145 7f09c4cab700 1 -- 172.21.2.102:0/1865903392 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f09c0092e40 msgr2=0x7f09c0093240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:14.147 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:14.145 7f09c4cab700 1 --2- 172.21.2.102:0/1865903392 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f09c0092e40 0x7f09c0093240 secure :-1 s=READY pgs=2506 cs=0 l=1 rev1=1 rx=0x7f09b000b5c0 tx=0x7f09b000b960).stop 2021-02-10T11:34:14.147 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:14.145 7f09c4cab700 1 -- 172.21.2.102:0/1865903392 shutdown_connections 2021-02-10T11:34:14.148 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:14.145 7f09c4cab700 1 --2- 172.21.2.102:0/1865903392 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f09a4018730 0x7f09a401abc0 unknown :-1 s=CLOSED pgs=1109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:14.148 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:14.145 7f09c4cab700 1 --2- 172.21.2.102:0/1865903392 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f09c01a89f0 0x7f09c01a8df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:14.148 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:14.145 7f09c4cab700 1 --2- 172.21.2.102:0/1865903392 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f09c0093860 0x7f09c01a83d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:14.148 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:14.145 7f09c4cab700 1 --2- 172.21.2.102:0/1865903392 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f09c0092e40 0x7f09c0093240 unknown :-1 s=CLOSED pgs=2506 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:14.148 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:14.145 7f09c4cab700 1 -- 172.21.2.102:0/1865903392 >> 172.21.2.102:0/1865903392 conn(0x7f09c019f7a0 msgr2=0x7f09c00940e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:14.148 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:14.145 7f09c4cab700 1 -- 172.21.2.102:0/1865903392 shutdown_connections 2021-02-10T11:34:14.148 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:14.145 7f09c4cab700 1 -- 172.21.2.102:0/1865903392 wait complete. 2021-02-10T11:34:14.155 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2021-02-10T11:34:14.590 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:14.571 7f6fc5b7f700 1 Processor -- start 2021-02-10T11:34:14.590 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:14.571 7f6fc5b7f700 1 -- start start 2021-02-10T11:34:14.590 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:14.571 7f6fc5b7f700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6fc00d0340 0x7f6fc00ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:14.590 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:14.571 7f6fc5b7f700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fc00f0b10 con 0x7f6fc00d1c70 2021-02-10T11:34:14.590 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:14.571 7f6fc5b7f700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fc01b0be0 con 0x7f6fc00d1870 2021-02-10T11:34:14.590 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:14.571 7f6fc5b7f700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fc00e5ed0 con 0x7f6fc00d0340 2021-02-10T11:34:14.590 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:14.572 7f6fbffff700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f6fc00d1c70 0x7f6fc00efc60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:56340/0 (socket says 172.21.2.102:56340) 2021-02-10T11:34:14.591 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:14.572 7f6fbffff700 1 -- 172.21.2.102:0/4230976220 learned_addr learned my addr 172.21.2.102:0/4230976220 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:34:14.591 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:14.572 7f6fbe7fc700 1 -- 172.21.2.102:0/4230976220 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 627708673 0 0) 0x7f6fb8001500 con 0x7f6fc00d1c70 2021-02-10T11:34:14.591 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:14.572 7f6fbe7fc700 1 -- 172.21.2.102:0/4230976220 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6fac002fe0 con 0x7f6fc00d1c70 2021-02-10T11:34:14.591 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:14.572 7f6fbeffd700 1 --2- 172.21.2.102:0/4230976220 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6fc00d0340 0x7f6fc00ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:14.591 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:14.572 7f6fbe7fc700 1 -- 172.21.2.102:0/4230976220 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3627186805 0 0) 0x7f6fb8001460 con 0x7f6fc00d1c70 2021-02-10T11:34:14.591 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:14.572 7f6fbe7fc700 1 -- 172.21.2.102:0/4230976220 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6fc00d0340 msgr2=0x7f6fc00ef640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:34:14.591 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:14.572 7f6fbe7fc700 1 --2- 172.21.2.102:0/4230976220 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6fc00d0340 0x7f6fc00ef640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:14.591 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:14.572 7f6fbe7fc700 1 -- 172.21.2.102:0/4230976220 >> v1:172.21.2.113:6789/0 conn(0x7f6fc00d1870 legacy=0x7f6fc01b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:14.591 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:14.572 7f6fbe7fc700 1 -- 172.21.2.102:0/4230976220 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fc00f0450 con 0x7f6fc00d1c70 2021-02-10T11:34:14.591 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:14.572 7f6fbe7fc700 1 -- 172.21.2.102:0/4230976220 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7f6fb8001980 con 0x7f6fc00d1c70 2021-02-10T11:34:14.591 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:14.572 7f6fbe7fc700 1 -- 172.21.2.102:0/4230976220 >> v1:172.21.2.102:6789/0 conn(0x7f6fc00d1c70 legacy=0x7f6fc00efc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:14.591 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:14.572 7f6fbe7fc700 1 --2- 172.21.2.102:0/4230976220 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6fac003a60 0x7f6fac005ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:14.591 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:14.572 7f6fbe7fc700 1 --2- 172.21.2.102:0/4230976220 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6fac006770 0x7f6fac008c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:14.592 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:14.572 7f6fbe7fc700 1 --2- 172.21.2.102:0/4230976220 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6fac009590 0x7f6fac00ba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:14.592 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:14.572 7f6fbe7fc700 1 -- 172.21.2.102:0/4230976220 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fac00c2d0 con 0x7f6fac003a60 2021-02-10T11:34:14.592 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:14.572 7f6fbe7fc700 1 -- 172.21.2.102:0/4230976220 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fac00c4f0 con 0x7f6fac006770 2021-02-10T11:34:14.592 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:14.572 7f6fbe7fc700 1 -- 172.21.2.102:0/4230976220 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fac00c710 con 0x7f6fac009590 2021-02-10T11:34:14.592 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:14.572 7f6fbf7fe700 1 --2- 172.21.2.102:0/4230976220 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6fac003a60 0x7f6fac005ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:14.592 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:14.573 7f6fbeffd700 1 --2- 172.21.2.102:0/4230976220 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6fac006770 0x7f6fac008c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:14.592 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:14.573 7f6fbffff700 1 --2- 172.21.2.102:0/4230976220 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6fac009590 0x7f6fac00ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:14.592 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:14.573 7f6fbf7fe700 1 -- 172.21.2.102:0/4230976220 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6fac009590 msgr2=0x7f6fac00ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:34:14.592 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:14.573 7f6fbf7fe700 1 --2- 172.21.2.102:0/4230976220 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6fac009590 0x7f6fac00ba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:14.592 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:14.573 7f6fbf7fe700 1 -- 172.21.2.102:0/4230976220 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6fac006770 msgr2=0x7f6fac008c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:34:14.592 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:14.573 7f6fbf7fe700 1 --2- 172.21.2.102:0/4230976220 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6fac006770 0x7f6fac008c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:14.592 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:14.573 7f6fbf7fe700 1 -- 172.21.2.102:0/4230976220 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fac050980 con 0x7f6fac003a60 2021-02-10T11:34:14.592 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:14.573 7f6fbeffd700 1 --2- 172.21.2.102:0/4230976220 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6fac006770 0x7f6fac008c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:34:14.593 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:14.573 7f6fbf7fe700 1 --2- 172.21.2.102:0/4230976220 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6fac003a60 0x7f6fac005ed0 secure :-1 s=READY pgs=2508 cs=0 l=1 rev1=1 rx=0x7f6fb0006ac0 tx=0x7f6fb0009af0).ready entity=mon.0 client_cookie=4d780d0fd3644cb0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:14.593 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:14.573 7f6fbe7fc700 1 -- 172.21.2.102:0/4230976220 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6fb0007280 con 0x7f6fac003a60 2021-02-10T11:34:14.593 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:14.573 7f6fbe7fc700 1 -- 172.21.2.102:0/4230976220 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fb0007500 con 0x7f6fac003a60 2021-02-10T11:34:14.593 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:14.573 7f6fbe7fc700 1 -- 172.21.2.102:0/4230976220 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6fb0007950 con 0x7f6fac003a60 2021-02-10T11:34:14.593 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:14.573 7f6fc5b7f700 1 -- 172.21.2.102:0/4230976220 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6fac003a60 msgr2=0x7f6fac005ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:14.593 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:14.573 7f6fc5b7f700 1 --2- 172.21.2.102:0/4230976220 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6fac003a60 0x7f6fac005ed0 secure :-1 s=READY pgs=2508 cs=0 l=1 rev1=1 rx=0x7f6fb0006ac0 tx=0x7f6fb0009af0).stop 2021-02-10T11:34:14.593 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:14.573 7f6fbffff700 1 -- 172.21.2.102:0/4230976220 reap_dead start 2021-02-10T11:34:14.593 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:14.574 7f6fc5b7f700 1 -- 172.21.2.102:0/4230976220 shutdown_connections 2021-02-10T11:34:14.593 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:14.574 7f6fc5b7f700 1 -- 172.21.2.102:0/4230976220 >> 172.21.2.102:0/4230976220 conn(0x7f6fc00ce450 msgr2=0x7f6fc00d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:14.593 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:14.574 7f6fc5b7f700 1 -- 172.21.2.102:0/4230976220 shutdown_connections 2021-02-10T11:34:14.593 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:14.574 7f6fc5b7f700 1 -- 172.21.2.102:0/4230976220 wait complete. 2021-02-10T11:34:14.593 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:14.588 7f6fc5b7f700 1 Processor -- start 2021-02-10T11:34:14.593 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:14.588 7f6fc5b7f700 1 -- start start 2021-02-10T11:34:14.595 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:14.589 7f6fc5b7f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6fc0094310 0x7f6fc00ebe70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:14.595 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:14.589 7f6fc5b7f700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6fc0094710 0x7f6fc00bca30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:14.595 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:14.589 7f6fc5b7f700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6fc00ec490 0x7f6fc00bd050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:14.595 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:14.589 7f6fc5b7f700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fc0093d90 con 0x7f6fc0094310 2021-02-10T11:34:14.595 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:14.589 7f6fc5b7f700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fc00bd670 con 0x7f6fc00ec490 2021-02-10T11:34:14.595 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:14.589 7f6fc5b7f700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fc00939f0 con 0x7f6fc0094710 2021-02-10T11:34:14.595 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:14.589 7f6fbf7fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6fc0094310 0x7f6fc00ebe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:14.595 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:14.589 7f6fbf7fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6fc0094310 0x7f6fc00ebe70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:35894/0 (socket says 172.21.2.102:35894) 2021-02-10T11:34:14.595 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:14.589 7f6fbf7fe700 1 -- 172.21.2.102:0/1338712387 learned_addr learned my addr 172.21.2.102:0/1338712387 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:34:14.595 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:14.589 7f6fbf7fe700 1 -- 172.21.2.102:0/1338712387 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6fc0094710 msgr2=0x7f6fc00bca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:14.595 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:14.589 7f6fbf7fe700 1 --2- 172.21.2.102:0/1338712387 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6fc0094710 0x7f6fc00bca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:14.596 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:14.589 7f6fbf7fe700 1 -- 172.21.2.102:0/1338712387 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6fc00ec490 msgr2=0x7f6fc00bd050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:14.596 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:14.589 7f6fbf7fe700 1 --2- 172.21.2.102:0/1338712387 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6fc00ec490 0x7f6fc00bd050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:14.596 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:14.589 7f6fbf7fe700 1 -- 172.21.2.102:0/1338712387 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fc00bda80 con 0x7f6fc0094310 2021-02-10T11:34:14.596 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:14.589 7f6fbf7fe700 1 --2- 172.21.2.102:0/1338712387 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6fc0094310 0x7f6fc00ebe70 secure :-1 s=READY pgs=2509 cs=0 l=1 rev1=1 rx=0x7f6fb000a680 tx=0x7f6fb000c8e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:14.596 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:14.590 7f6fb7fff700 1 -- 172.21.2.102:0/1338712387 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6fb0007030 con 0x7f6fc0094310 2021-02-10T11:34:14.596 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:14.590 7f6fb7fff700 1 -- 172.21.2.102:0/1338712387 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fb00073b0 con 0x7f6fc0094310 2021-02-10T11:34:14.596 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:14.590 7f6fc5b7f700 1 -- 172.21.2.102:0/1338712387 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fc00bda80 con 0x7f6fc0094310 2021-02-10T11:34:14.596 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:14.590 7f6fc5b7f700 1 -- 172.21.2.102:0/1338712387 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fc01a76d0 con 0x7f6fc0094310 2021-02-10T11:34:14.596 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:14.590 7f6fb7fff700 1 -- 172.21.2.102:0/1338712387 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6fb00078d0 con 0x7f6fc0094310 2021-02-10T11:34:14.596 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:14.590 7f6fb7fff700 1 -- 172.21.2.102:0/1338712387 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f6fb0006ea0 con 0x7f6fc0094310 2021-02-10T11:34:14.596 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:14.590 7f6fb7fff700 1 --2- 172.21.2.102:0/1338712387 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6fa40186c0 0x7f6fa401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:14.596 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:14.590 7f6fb7fff700 1 -- 172.21.2.102:0/1338712387 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5659+0+0 (secure 0 0 0) 0x7f6fb0032830 con 0x7f6fc0094310 2021-02-10T11:34:14.597 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:14.591 7f6fbeffd700 1 --2- 172.21.2.102:0/1338712387 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6fa40186c0 0x7f6fa401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:14.597 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:14.591 7f6fbeffd700 1 --2- 172.21.2.102:0/1338712387 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6fa40186c0 0x7f6fa401abc0 crc :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:14.623 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:14.622 7f6fc5b7f700 1 -- 172.21.2.102:0/1338712387 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fc00ebc40 con 0x7f6fc0094310 2021-02-10T11:34:14.626 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:14.624 7f6fb7fff700 1 -- 172.21.2.102:0/1338712387 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f6fb0006c90 con 0x7f6fc0094310 2021-02-10T11:34:14.753 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:14.752 7f6fc5b7f700 1 -- 172.21.2.102:0/1338712387 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f6fc00ebc40 con 0x7f6fc0094310 2021-02-10T11:34:15.518 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:15.516 7f6fb7fff700 1 -- 172.21.2.102:0/1338712387 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v110) v1 ==== 104+0+0 (secure 0 0 0) 0x7f6fb00073b0 con 0x7f6fc0094310 2021-02-10T11:34:15.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:15.524 7f6fc5b7f700 1 -- 172.21.2.102:0/1338712387 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6fa40186c0 msgr2=0x7f6fa401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:15.526 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:15.524 7f6fc5b7f700 1 --2- 172.21.2.102:0/1338712387 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6fa40186c0 0x7f6fa401abc0 crc :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:15.526 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:15.524 7f6fc5b7f700 1 -- 172.21.2.102:0/1338712387 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6fc0094310 msgr2=0x7f6fc00ebe70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:15.526 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:15.524 7f6fc5b7f700 1 --2- 172.21.2.102:0/1338712387 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6fc0094310 0x7f6fc00ebe70 secure :-1 s=READY pgs=2509 cs=0 l=1 rev1=1 rx=0x7f6fb000a680 tx=0x7f6fb000c8e0).stop 2021-02-10T11:34:15.526 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:15.524 7f6fc5b7f700 1 -- 172.21.2.102:0/1338712387 shutdown_connections 2021-02-10T11:34:15.526 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:15.524 7f6fc5b7f700 1 --2- 172.21.2.102:0/1338712387 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6fa40186c0 0x7f6fa401abc0 unknown :-1 s=CLOSED pgs=1110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:15.526 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:15.524 7f6fc5b7f700 1 --2- 172.21.2.102:0/1338712387 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6fc00ec490 0x7f6fc00bd050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:15.526 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:15.524 7f6fc5b7f700 1 --2- 172.21.2.102:0/1338712387 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6fc0094710 0x7f6fc00bca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:15.526 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:15.524 7f6fc5b7f700 1 --2- 172.21.2.102:0/1338712387 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6fc0094310 0x7f6fc00ebe70 unknown :-1 s=CLOSED pgs=2509 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:15.526 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:15.524 7f6fc5b7f700 1 -- 172.21.2.102:0/1338712387 >> 172.21.2.102:0/1338712387 conn(0x7f6fc008fa10 msgr2=0x7f6fc00bfa60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:15.527 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:15.524 7f6fc5b7f700 1 -- 172.21.2.102:0/1338712387 shutdown_connections 2021-02-10T11:34:15.527 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:15.524 7f6fc5b7f700 1 -- 172.21.2.102:0/1338712387 wait complete. 2021-02-10T11:34:15.534 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool delete cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2021-02-10T11:34:15.964 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:15.946 7f6acbaed700 1 Processor -- start 2021-02-10T11:34:15.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:15.946 7f6acbaed700 1 -- start start 2021-02-10T11:34:15.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:15.947 7f6acbaed700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f6ac40d1c70 0x7f6ac40cef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:15.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:15.947 7f6acbaed700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6ac40cf580 0x7f6ac40cf980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:15.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:15.947 7f6acbaed700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ac40fe1c0 con 0x7f6ac40d1870 2021-02-10T11:34:15.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:15.947 7f6acbaed700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ac41b0cb0 con 0x7f6ac40d1c70 2021-02-10T11:34:15.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:15.947 7f6acbaed700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ac4109a60 con 0x7f6ac40cf580 2021-02-10T11:34:15.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:15.947 7f6ac9108700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f6ac40d1c70 0x7f6ac40cef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:15.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:15.947 7f6ac9909700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f6ac40d1870 0x7f6ac41b04c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:56356/0 (socket says 172.21.2.102:56356) 2021-02-10T11:34:15.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:15.947 7f6ac9909700 1 -- 172.21.2.102:0/3596237996 learned_addr learned my addr 172.21.2.102:0/3596237996 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:34:15.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:15.947 7f6ac9108700 1 -- 172.21.2.102:0/3596237996 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6ac40cf580 msgr2=0x7f6ac40cf980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:34:15.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:15.947 7f6ac9108700 1 --2- 172.21.2.102:0/3596237996 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6ac40cf580 0x7f6ac40cf980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:15.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:15.947 7f6ac9108700 1 -- 172.21.2.102:0/3596237996 >> v1:172.21.2.102:6789/0 conn(0x7f6ac40d1870 legacy=0x7f6ac41b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:15.966 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:15.947 7f6ac9108700 1 -- 172.21.2.102:0/3596237996 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ac4088070 con 0x7f6ac40d1c70 2021-02-10T11:34:15.966 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:15.948 7f6ac9108700 1 --2- 172.21.2.102:0/3596237996 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6ac40d1c70 0x7f6ac40cef60 secure :-1 s=READY pgs=2511 cs=0 l=1 rev1=1 rx=0x7f6ab4006650 tx=0x7f6ab4009a00).ready entity=mon.0 client_cookie=12ce351af4c87fbf server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:15.966 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:15.948 7f6ac8907700 1 -- 172.21.2.102:0/3596237996 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6ab4009df0 con 0x7f6ac40d1c70 2021-02-10T11:34:15.966 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:15.948 7f6ac8907700 1 -- 172.21.2.102:0/3596237996 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ab40009b0 con 0x7f6ac40d1c70 2021-02-10T11:34:15.966 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:15.948 7f6ac8907700 1 -- 172.21.2.102:0/3596237996 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6ab400c2b0 con 0x7f6ac40d1c70 2021-02-10T11:34:15.966 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:15.948 7f6acbaed700 1 -- 172.21.2.102:0/3596237996 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6ac40d1c70 msgr2=0x7f6ac40cef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:15.966 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:15.948 7f6acbaed700 1 --2- 172.21.2.102:0/3596237996 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6ac40d1c70 0x7f6ac40cef60 secure :-1 s=READY pgs=2511 cs=0 l=1 rev1=1 rx=0x7f6ab4006650 tx=0x7f6ab4009a00).stop 2021-02-10T11:34:15.966 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:15.948 7f6acbaed700 1 -- 172.21.2.102:0/3596237996 shutdown_connections 2021-02-10T11:34:15.966 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:15.948 7f6acbaed700 1 --2- 172.21.2.102:0/3596237996 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6ac40d1c70 0x7f6ac40cef60 unknown :-1 s=CLOSED pgs=2511 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:15.966 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:15.948 7f6acbaed700 1 --2- 172.21.2.102:0/3596237996 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6ac40cf580 0x7f6ac40cf980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:15.966 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:15.948 7f6acbaed700 1 -- 172.21.2.102:0/3596237996 >> 172.21.2.102:0/3596237996 conn(0x7f6ac40ce420 msgr2=0x7f6ac40d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:15.966 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:15.949 7f6acbaed700 1 -- 172.21.2.102:0/3596237996 shutdown_connections 2021-02-10T11:34:15.967 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:15.949 7f6acbaed700 1 -- 172.21.2.102:0/3596237996 wait complete. 2021-02-10T11:34:15.968 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:15.963 7f6acbaed700 1 Processor -- start 2021-02-10T11:34:15.968 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:15.963 7f6acbaed700 1 -- start start 2021-02-10T11:34:15.968 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:15.964 7f6acbaed700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6ac41aad00 0x7f6ac40ac0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:15.969 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:15.964 7f6acbaed700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6ac40ac6c0 0x7f6ac4187e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:15.969 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:15.964 7f6acbaed700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6ac4188490 0x7f6ac41910c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:15.969 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:15.964 7f6acbaed700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ac4188890 con 0x7f6ac4188490 2021-02-10T11:34:15.969 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:15.964 7f6acbaed700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ac4188ab0 con 0x7f6ac40ac6c0 2021-02-10T11:34:15.969 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:15.964 7f6acbaed700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ac41916e0 con 0x7f6ac41aad00 2021-02-10T11:34:15.969 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:15.964 7f6aca10a700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6ac4188490 0x7f6ac41910c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:15.969 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:15.964 7f6aca10a700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6ac4188490 0x7f6ac41910c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:35908/0 (socket says 172.21.2.102:35908) 2021-02-10T11:34:15.969 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:15.964 7f6aca10a700 1 -- 172.21.2.102:0/3511520871 learned_addr learned my addr 172.21.2.102:0/3511520871 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:34:15.969 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:15.964 7f6ac9909700 1 --2- 172.21.2.102:0/3511520871 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6ac41aad00 0x7f6ac40ac0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:15.969 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:15.964 7f6ac9108700 1 --2- 172.21.2.102:0/3511520871 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6ac40ac6c0 0x7f6ac4187e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:15.969 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:15.964 7f6aca10a700 1 -- 172.21.2.102:0/3511520871 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6ac41aad00 msgr2=0x7f6ac40ac0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:15.969 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:15.964 7f6aca10a700 1 --2- 172.21.2.102:0/3511520871 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6ac41aad00 0x7f6ac40ac0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:15.969 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:15.964 7f6aca10a700 1 -- 172.21.2.102:0/3511520871 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6ac40ac6c0 msgr2=0x7f6ac4187e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:15.970 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:15.964 7f6aca10a700 1 --2- 172.21.2.102:0/3511520871 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6ac40ac6c0 0x7f6ac4187e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:15.970 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:15.964 7f6aca10a700 1 -- 172.21.2.102:0/3511520871 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ac4191900 con 0x7f6ac4188490 2021-02-10T11:34:15.970 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:15.964 7f6ac9108700 1 --2- 172.21.2.102:0/3511520871 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6ac40ac6c0 0x7f6ac4187e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:34:15.970 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:15.964 7f6aca10a700 1 --2- 172.21.2.102:0/3511520871 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6ac4188490 0x7f6ac41910c0 secure :-1 s=READY pgs=2512 cs=0 l=1 rev1=1 rx=0x7f6abc009980 tx=0x7f6abc00bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:15.970 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:15.965 7f6aba7fc700 1 -- 172.21.2.102:0/3511520871 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6abc005870 con 0x7f6ac4188490 2021-02-10T11:34:15.970 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:15.965 7f6aba7fc700 1 -- 172.21.2.102:0/3511520871 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6abc005af0 con 0x7f6ac4188490 2021-02-10T11:34:15.970 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:15.965 7f6aba7fc700 1 -- 172.21.2.102:0/3511520871 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6abc006030 con 0x7f6ac4188490 2021-02-10T11:34:15.970 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:15.965 7f6acbaed700 1 -- 172.21.2.102:0/3511520871 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ac4191900 con 0x7f6ac4188490 2021-02-10T11:34:15.970 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:15.965 7f6acbaed700 1 -- 172.21.2.102:0/3511520871 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ac4191900 con 0x7f6ac4188490 2021-02-10T11:34:15.970 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:15.965 7f6aba7fc700 1 -- 172.21.2.102:0/3511520871 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f6abc005cd0 con 0x7f6ac4188490 2021-02-10T11:34:15.970 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:15.965 7f6aba7fc700 1 --2- 172.21.2.102:0/3511520871 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6aac0186c0 0x7f6aac01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:15.970 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:15.965 7f6aba7fc700 1 -- 172.21.2.102:0/3511520871 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5659+0+0 (secure 0 0 0) 0x7f6abc0327e0 con 0x7f6ac4188490 2021-02-10T11:34:15.970 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:15.966 7f6ac9909700 1 --2- 172.21.2.102:0/3511520871 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6aac0186c0 0x7f6aac01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:15.971 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:15.966 7f6ac9909700 1 --2- 172.21.2.102:0/3511520871 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6aac0186c0 0x7f6aac01abc0 crc :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:15.999 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:15.997 7f6acbaed700 1 -- 172.21.2.102:0/3511520871 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ac4191f10 con 0x7f6ac4188490 2021-02-10T11:34:16.001 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:15.999 7f6aba7fc700 1 -- 172.21.2.102:0/3511520871 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f6abc025ee0 con 0x7f6ac4188490 2021-02-10T11:34:16.132 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:16.125 7f6acbaed700 1 -- 172.21.2.102:0/3511520871 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"pool2": "cephfs_metadata", "prefix": "osd pool delete", "pool": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f6ac4191f10 con 0x7f6ac4188490 2021-02-10T11:34:16.522 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:16.520 7f6aba7fc700 1 -- 172.21.2.102:0/3511520871 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"pool2": "cephfs_metadata", "prefix": "osd pool delete", "pool": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v130) v1 ==== 185+0+0 (secure 0 0 0) 0x7f6abc025ee0 con 0x7f6ac4188490 2021-02-10T11:34:16.545 INFO:teuthology.orchestra.run.gibba002.stderr:pool 'cephfs_metadata' removed 2021-02-10T11:34:16.546 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:16.545 7f6acbaed700 1 -- 172.21.2.102:0/3511520871 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6aac0186c0 msgr2=0x7f6aac01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:16.547 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:16.545 7f6acbaed700 1 --2- 172.21.2.102:0/3511520871 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6aac0186c0 0x7f6aac01abc0 crc :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:16.547 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:16.545 7f6acbaed700 1 -- 172.21.2.102:0/3511520871 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6ac4188490 msgr2=0x7f6ac41910c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:16.547 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:16.545 7f6acbaed700 1 --2- 172.21.2.102:0/3511520871 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6ac4188490 0x7f6ac41910c0 secure :-1 s=READY pgs=2512 cs=0 l=1 rev1=1 rx=0x7f6abc009980 tx=0x7f6abc00bcd0).stop 2021-02-10T11:34:16.547 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:16.545 7f6acbaed700 1 -- 172.21.2.102:0/3511520871 shutdown_connections 2021-02-10T11:34:16.547 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:16.545 7f6acbaed700 1 --2- 172.21.2.102:0/3511520871 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6ac4188490 0x7f6ac41910c0 unknown :-1 s=CLOSED pgs=2512 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:16.547 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:16.545 7f6acbaed700 1 --2- 172.21.2.102:0/3511520871 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6aac0186c0 0x7f6aac01abc0 unknown :-1 s=CLOSED pgs=1111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:16.547 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:16.545 7f6acbaed700 1 --2- 172.21.2.102:0/3511520871 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6ac40ac6c0 0x7f6ac4187e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:16.548 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:16.545 7f6acbaed700 1 --2- 172.21.2.102:0/3511520871 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6ac41aad00 0x7f6ac40ac0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:16.548 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:16.545 7f6acbaed700 1 -- 172.21.2.102:0/3511520871 >> 172.21.2.102:0/3511520871 conn(0x7f6ac4094020 msgr2=0x7f6ac41922b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:16.548 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:16.545 7f6acbaed700 1 -- 172.21.2.102:0/3511520871 shutdown_connections 2021-02-10T11:34:16.548 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:16.545 7f6acbaed700 1 -- 172.21.2.102:0/3511520871 wait complete. 2021-02-10T11:34:16.555 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool delete cephfs_data cephfs_data --yes-i-really-really-mean-it 2021-02-10T11:34:16.988 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:16.969 7fcb7d5fb700 1 Processor -- start 2021-02-10T11:34:16.988 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:16.969 7fcb7d5fb700 1 -- start start 2021-02-10T11:34:16.989 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:16.969 7fcb7d5fb700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcb7810b000 0x7fcb780cf760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:16.989 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:16.970 7fcb7d5fb700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb780fe1d0 con 0x7fcb780d1880 2021-02-10T11:34:16.989 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:16.970 7fcb7d5fb700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb781b0cc0 con 0x7fcb780d1c80 2021-02-10T11:34:16.989 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:16.970 7fcb7d5fb700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb78109a70 con 0x7fcb7810b000 2021-02-10T11:34:16.989 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:16.970 7fcb772bf700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fcb780d1880 0x7fcb781b04d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:56370/0 (socket says 172.21.2.102:56370) 2021-02-10T11:34:16.989 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:16.970 7fcb772bf700 1 -- 172.21.2.102:0/1125184485 learned_addr learned my addr 172.21.2.102:0/1125184485 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:34:16.989 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:16.970 7fcb762bd700 1 -- 172.21.2.102:0/1125184485 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3505032210 0 0) 0x7fcb68000ce0 con 0x7fcb780d1880 2021-02-10T11:34:16.989 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:16.970 7fcb762bd700 1 -- 172.21.2.102:0/1125184485 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb60002fe0 con 0x7fcb780d1880 2021-02-10T11:34:16.989 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:16.970 7fcb762bd700 1 -- 172.21.2.102:0/1125184485 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1670525542 0 0) 0x7fcb68000ed0 con 0x7fcb780d1880 2021-02-10T11:34:16.989 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:16.970 7fcb77ac0700 1 --2- 172.21.2.102:0/1125184485 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcb7810b000 0x7fcb780cf760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:16.989 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:16.970 7fcb762bd700 1 -- 172.21.2.102:0/1125184485 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcb7810b000 msgr2=0x7fcb780cf760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:34:16.989 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:16.970 7fcb762bd700 1 --2- 172.21.2.102:0/1125184485 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcb7810b000 0x7fcb780cf760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:16.990 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:16.970 7fcb762bd700 1 -- 172.21.2.102:0/1125184485 >> v1:172.21.2.113:6789/0 conn(0x7fcb780d1c80 legacy=0x7fcb780cef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:34:16.990 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:16.970 7fcb762bd700 1 -- 172.21.2.102:0/1125184485 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb7810b400 con 0x7fcb780d1880 2021-02-10T11:34:16.990 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:16.970 7fcb762bd700 1 -- 172.21.2.102:0/1125184485 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7fcb680016a0 con 0x7fcb780d1880 2021-02-10T11:34:16.990 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:16.971 7fcb762bd700 1 -- 172.21.2.102:0/1125184485 >> v1:172.21.2.102:6789/0 conn(0x7fcb780d1880 legacy=0x7fcb781b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:16.990 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:16.971 7fcb762bd700 1 --2- 172.21.2.102:0/1125184485 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcb60003b20 0x7fcb60005f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:16.990 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:16.971 7fcb762bd700 1 --2- 172.21.2.102:0/1125184485 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fcb60006810 0x7fcb60008d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:16.990 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:16.971 7fcb762bd700 1 --2- 172.21.2.102:0/1125184485 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcb60009630 0x7fcb6000bb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:16.990 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:16.971 7fcb762bd700 1 -- 172.21.2.102:0/1125184485 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb6000c370 con 0x7fcb60003b20 2021-02-10T11:34:16.990 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:16.971 7fcb762bd700 1 -- 172.21.2.102:0/1125184485 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb6000c590 con 0x7fcb60006810 2021-02-10T11:34:16.990 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:16.971 7fcb762bd700 1 -- 172.21.2.102:0/1125184485 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb6000c7b0 con 0x7fcb60009630 2021-02-10T11:34:16.990 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:16.971 7fcb77ac0700 1 --2- 172.21.2.102:0/1125184485 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcb7810b000 0x7fcb780cf760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:34:16.990 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:16.971 7fcb772bf700 1 --2- 172.21.2.102:0/1125184485 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcb60003b20 0x7fcb60005f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:16.990 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:16.971 7fcb772bf700 1 -- 172.21.2.102:0/1125184485 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcb60009630 msgr2=0x7fcb6000bb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:34:16.991 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:16.971 7fcb772bf700 1 --2- 172.21.2.102:0/1125184485 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcb60009630 0x7fcb6000bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:16.991 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:16.971 7fcb772bf700 1 -- 172.21.2.102:0/1125184485 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fcb60006810 msgr2=0x7fcb60008d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:34:16.991 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:16.971 7fcb772bf700 1 --2- 172.21.2.102:0/1125184485 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fcb60006810 0x7fcb60008d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:16.991 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:16.971 7fcb772bf700 1 -- 172.21.2.102:0/1125184485 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb6000ca50 con 0x7fcb60003b20 2021-02-10T11:34:16.991 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:16.971 7fcb772bf700 1 --2- 172.21.2.102:0/1125184485 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcb60003b20 0x7fcb60005f70 secure :-1 s=READY pgs=2514 cs=0 l=1 rev1=1 rx=0x7fcb680079f0 tx=0x7fcb68007d90).ready entity=mon.0 client_cookie=ab5e839766ab4d44 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:16.991 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:16.971 7fcb762bd700 1 -- 172.21.2.102:0/1125184485 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcb68001290 con 0x7fcb60003b20 2021-02-10T11:34:16.991 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:16.971 7fcb762bd700 1 -- 172.21.2.102:0/1125184485 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb680070c0 con 0x7fcb60003b20 2021-02-10T11:34:16.991 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:16.971 7fcb762bd700 1 -- 172.21.2.102:0/1125184485 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcb680047a0 con 0x7fcb60003b20 2021-02-10T11:34:16.991 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:16.971 7fcb7d5fb700 1 -- 172.21.2.102:0/1125184485 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcb60003b20 msgr2=0x7fcb60005f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:16.991 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:16.971 7fcb7d5fb700 1 --2- 172.21.2.102:0/1125184485 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcb60003b20 0x7fcb60005f70 secure :-1 s=READY pgs=2514 cs=0 l=1 rev1=1 rx=0x7fcb680079f0 tx=0x7fcb68007d90).stop 2021-02-10T11:34:16.991 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:16.971 7fcb77ac0700 1 -- 172.21.2.102:0/1125184485 reap_dead start 2021-02-10T11:34:16.991 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:16.972 7fcb7d5fb700 1 -- 172.21.2.102:0/1125184485 shutdown_connections 2021-02-10T11:34:16.991 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:16.972 7fcb7d5fb700 1 -- 172.21.2.102:0/1125184485 >> 172.21.2.102:0/1125184485 conn(0x7fcb780ce430 msgr2=0x7fcb780d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:16.992 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:16.972 7fcb7d5fb700 1 -- 172.21.2.102:0/1125184485 shutdown_connections 2021-02-10T11:34:16.992 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:16.972 7fcb7d5fb700 1 -- 172.21.2.102:0/1125184485 wait complete. 2021-02-10T11:34:16.993 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:16.987 7fcb7d5fb700 1 Processor -- start 2021-02-10T11:34:16.993 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:16.987 7fcb7d5fb700 1 -- start start 2021-02-10T11:34:16.993 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:16.987 7fcb7d5fb700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcb780847a0 0x7fcb7818e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:16.993 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:16.987 7fcb7d5fb700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcb78084f40 0x7fcb78085340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:16.993 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:16.987 7fcb7d5fb700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fcb7809a500 0x7fcb7809a900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:16.994 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:16.987 7fcb7d5fb700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb78085960 con 0x7fcb780847a0 2021-02-10T11:34:16.994 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:16.987 7fcb7d5fb700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb78084ba0 con 0x7fcb7809a500 2021-02-10T11:34:16.994 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:16.987 7fcb7d5fb700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb7809af20 con 0x7fcb78084f40 2021-02-10T11:34:16.994 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:16.988 7fcb772bf700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcb780847a0 0x7fcb7818e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:16.994 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:16.988 7fcb772bf700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcb780847a0 0x7fcb7818e2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:35926/0 (socket says 172.21.2.102:35926) 2021-02-10T11:34:16.994 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:16.988 7fcb772bf700 1 -- 172.21.2.102:0/1109883189 learned_addr learned my addr 172.21.2.102:0/1109883189 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:34:16.994 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:16.988 7fcb77ac0700 1 --2- 172.21.2.102:0/1109883189 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fcb7809a500 0x7fcb7809a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:16.994 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:16.988 7fcb76abe700 1 --2- 172.21.2.102:0/1109883189 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcb78084f40 0x7fcb78085340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:16.994 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:16.988 7fcb772bf700 1 -- 172.21.2.102:0/1109883189 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcb78084f40 msgr2=0x7fcb78085340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:16.994 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:16.988 7fcb772bf700 1 --2- 172.21.2.102:0/1109883189 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcb78084f40 0x7fcb78085340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:16.994 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:16.988 7fcb772bf700 1 -- 172.21.2.102:0/1109883189 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fcb7809a500 msgr2=0x7fcb7809a900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:16.994 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:16.988 7fcb772bf700 1 --2- 172.21.2.102:0/1109883189 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fcb7809a500 0x7fcb7809a900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:16.994 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:16.988 7fcb772bf700 1 -- 172.21.2.102:0/1109883189 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb7809b140 con 0x7fcb780847a0 2021-02-10T11:34:16.995 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:16.988 7fcb772bf700 1 --2- 172.21.2.102:0/1109883189 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcb780847a0 0x7fcb7818e2a0 secure :-1 s=READY pgs=2515 cs=0 l=1 rev1=1 rx=0x7fcb68007ea0 tx=0x7fcb6804f950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:16.995 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:16.988 7fcb5ffff700 1 -- 172.21.2.102:0/1109883189 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcb68001950 con 0x7fcb780847a0 2021-02-10T11:34:16.995 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:16.988 7fcb5ffff700 1 -- 172.21.2.102:0/1109883189 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb6804fd40 con 0x7fcb780847a0 2021-02-10T11:34:16.995 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:16.988 7fcb7d5fb700 1 -- 172.21.2.102:0/1109883189 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb7809b140 con 0x7fcb780847a0 2021-02-10T11:34:16.995 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:16.988 7fcb7d5fb700 1 -- 172.21.2.102:0/1109883189 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb7809b6d0 con 0x7fcb780847a0 2021-02-10T11:34:16.995 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:16.989 7fcb5ffff700 1 -- 172.21.2.102:0/1109883189 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcb680502f0 con 0x7fcb780847a0 2021-02-10T11:34:16.995 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:16.989 7fcb5ffff700 1 -- 172.21.2.102:0/1109883189 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fcb6804c050 con 0x7fcb780847a0 2021-02-10T11:34:16.995 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:16.989 7fcb5ffff700 1 --2- 172.21.2.102:0/1109883189 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcb580186c0 0x7fcb5801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:16.995 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:16.989 7fcb5ffff700 1 -- 172.21.2.102:0/1109883189 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5221+0+0 (secure 0 0 0) 0x7fcb68076410 con 0x7fcb780847a0 2021-02-10T11:34:16.995 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:16.989 7fcb76abe700 1 --2- 172.21.2.102:0/1109883189 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcb580186c0 0x7fcb5801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:16.995 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:16.989 7fcb76abe700 1 --2- 172.21.2.102:0/1109883189 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcb580186c0 0x7fcb5801abc0 crc :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:17.021 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:17.019 7fcb7d5fb700 1 -- 172.21.2.102:0/1109883189 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb78082a70 con 0x7fcb780847a0 2021-02-10T11:34:17.024 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:17.022 7fcb5ffff700 1 -- 172.21.2.102:0/1109883189 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fcb68007660 con 0x7fcb780847a0 2021-02-10T11:34:17.151 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:17.150 7fcb7d5fb700 1 -- 172.21.2.102:0/1109883189 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"pool2": "cephfs_data", "prefix": "osd pool delete", "pool": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fcb78082a70 con 0x7fcb780847a0 2021-02-10T11:34:17.524 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:17.522 7fcb5ffff700 1 -- 172.21.2.102:0/1109883189 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"pool2": "cephfs_data", "prefix": "osd pool delete", "pool": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v131) v1 ==== 173+0+0 (secure 0 0 0) 0x7fcb6804c050 con 0x7fcb780847a0 2021-02-10T11:34:17.567 INFO:teuthology.orchestra.run.gibba002.stderr:pool 'cephfs_data' removed 2021-02-10T11:34:17.569 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:17.567 7fcb7d5fb700 1 -- 172.21.2.102:0/1109883189 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcb580186c0 msgr2=0x7fcb5801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:17.569 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:17.567 7fcb7d5fb700 1 --2- 172.21.2.102:0/1109883189 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcb580186c0 0x7fcb5801abc0 crc :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:17.569 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:17.567 7fcb7d5fb700 1 -- 172.21.2.102:0/1109883189 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcb780847a0 msgr2=0x7fcb7818e2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:17.569 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:17.567 7fcb7d5fb700 1 --2- 172.21.2.102:0/1109883189 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcb780847a0 0x7fcb7818e2a0 secure :-1 s=READY pgs=2515 cs=0 l=1 rev1=1 rx=0x7fcb68007ea0 tx=0x7fcb6804f950).stop 2021-02-10T11:34:17.570 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:17.567 7fcb7d5fb700 1 -- 172.21.2.102:0/1109883189 shutdown_connections 2021-02-10T11:34:17.570 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:17.567 7fcb7d5fb700 1 --2- 172.21.2.102:0/1109883189 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcb580186c0 0x7fcb5801abc0 unknown :-1 s=CLOSED pgs=1112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:17.570 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:17.567 7fcb7d5fb700 1 --2- 172.21.2.102:0/1109883189 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fcb7809a500 0x7fcb7809a900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:17.570 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:17.567 7fcb7d5fb700 1 --2- 172.21.2.102:0/1109883189 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcb78084f40 0x7fcb78085340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:17.570 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:17.567 7fcb7d5fb700 1 --2- 172.21.2.102:0/1109883189 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcb780847a0 0x7fcb7818e2a0 unknown :-1 s=CLOSED pgs=2515 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:17.570 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:17.567 7fcb7d5fb700 1 -- 172.21.2.102:0/1109883189 >> 172.21.2.102:0/1109883189 conn(0x7fcb7809ebd0 msgr2=0x7fcb78082ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:17.570 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:17.568 7fcb7d5fb700 1 -- 172.21.2.102:0/1109883189 shutdown_connections 2021-02-10T11:34:17.570 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:17.568 7fcb7d5fb700 1 -- 172.21.2.102:0/1109883189 wait complete. 2021-02-10T11:34:17.578 INFO:tasks.ceph.mds.a:Restarting daemon 2021-02-10T11:34:17.578 INFO:tasks.ceph.mds.a:Stopping old one... 2021-02-10T11:34:17.578 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2021-02-10T11:34:17.578 INFO:teuthology.orchestra.run:waiting for 300 2021-02-10T11:34:17.578 INFO:tasks.ceph.mds.b:Restarting daemon 2021-02-10T11:34:17.578 INFO:tasks.ceph.mds.b:Stopping old one... 2021-02-10T11:34:17.578 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2021-02-10T11:34:17.579 INFO:teuthology.orchestra.run:waiting for 300 2021-02-10T11:34:17.584 INFO:tasks.ceph.mds.a:Stopped 2021-02-10T11:34:17.584 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2021-02-10T11:34:17.585 INFO:tasks.ceph.mds.b:Stopped 2021-02-10T11:34:17.585 DEBUG:teuthology.orchestra.run.gibba013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2021-02-10T11:34:17.627 INFO:tasks.ceph.mds.a:Started 2021-02-10T11:34:17.627 DEBUG:teuthology.parallel:result is None 2021-02-10T11:34:17.628 INFO:tasks.ceph.mds.b:Started 2021-02-10T11:34:17.628 DEBUG:teuthology.parallel:result is None 2021-02-10T11:34:17.628 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd blacklist clear 2021-02-10T11:34:18.034 INFO:tasks.ceph.mds.b.gibba013.stdout:starting mds.b at 2021-02-10T11:34:18.035 INFO:tasks.ceph.mds.a.gibba002.stdout:starting mds.a at 2021-02-10T11:34:18.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:18.036 7f09669f3700 1 Processor -- start 2021-02-10T11:34:18.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:18.037 7f09669f3700 1 -- start start 2021-02-10T11:34:18.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:18.037 7f09669f3700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f09600d1870 0x7f09601b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:18.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:18.037 7f09669f3700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f09601b0a10 0x7f09600cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:18.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:18.037 7f09669f3700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f09600d1c70 0x7f09600cf950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:18.056 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:18.037 7f09669f3700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09600fe230 con 0x7f09601b0a10 2021-02-10T11:34:18.056 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:18.037 7f09669f3700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09600e5ed0 con 0x7f09600d1c70 2021-02-10T11:34:18.056 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:18.037 7f09669f3700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09600d2fe0 con 0x7f09600d1870 2021-02-10T11:34:18.056 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:18.037 7f095ffff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f09601b0a10 0x7f09600cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:18.056 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:18.037 7f095ffff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f09601b0a10 0x7f09600cf330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:35948/0 (socket says 172.21.2.102:35948) 2021-02-10T11:34:18.056 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:18.037 7f095ffff700 1 -- 172.21.2.102:0/1105741730 learned_addr learned my addr 172.21.2.102:0/1105741730 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:34:18.056 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:18.037 7f0965010700 1 --2- 172.21.2.102:0/1105741730 >> v2:172.21.2.113:3300/0 conn(0x7f09600d1c70 0x7f09600cf950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:18.056 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:18.037 7f096480f700 1 --2- 172.21.2.102:0/1105741730 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f09600d1870 0x7f09601b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:18.056 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:18.038 7f095ffff700 1 -- 172.21.2.102:0/1105741730 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f09600d1870 msgr2=0x7f09601b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:34:18.056 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:18.038 7f095ffff700 1 --2- 172.21.2.102:0/1105741730 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f09600d1870 0x7f09601b03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:18.056 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:18.038 7f095ffff700 1 -- 172.21.2.102:0/1105741730 >> v2:172.21.2.113:3300/0 conn(0x7f09600d1c70 msgr2=0x7f09600cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:34:18.056 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:18.038 7f095ffff700 1 --2- 172.21.2.102:0/1105741730 >> v2:172.21.2.113:3300/0 conn(0x7f09600d1c70 0x7f09600cf950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:18.056 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:18.038 7f095ffff700 1 -- 172.21.2.102:0/1105741730 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09600f0b10 con 0x7f09601b0a10 2021-02-10T11:34:18.056 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:18.038 7f0965010700 1 --2- 172.21.2.102:0/1105741730 >> v2:172.21.2.113:3300/0 conn(0x7f09600d1c70 0x7f09600cf950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:34:18.057 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:18.038 7f096480f700 1 --2- 172.21.2.102:0/1105741730 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f09600d1870 0x7f09601b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:34:18.057 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:18.038 7f095ffff700 1 --2- 172.21.2.102:0/1105741730 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f09601b0a10 0x7f09600cf330 secure :-1 s=READY pgs=2520 cs=0 l=1 rev1=1 rx=0x7f0958006a80 tx=0x7f0958009f00).ready entity=mon.0 client_cookie=f07add02e2a034ad server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:18.057 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:18.038 7f095f7fe700 1 -- 172.21.2.102:0/1105741730 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0958000d30 con 0x7f09601b0a10 2021-02-10T11:34:18.057 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:18.038 7f095f7fe700 1 -- 172.21.2.102:0/1105741730 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f095800b5c0 con 0x7f09601b0a10 2021-02-10T11:34:18.057 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:18.038 7f095f7fe700 1 -- 172.21.2.102:0/1105741730 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f095800ba20 con 0x7f09601b0a10 2021-02-10T11:34:18.057 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:18.038 7f09669f3700 1 -- 172.21.2.102:0/1105741730 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f09601b0a10 msgr2=0x7f09600cf330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:18.057 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:18.038 7f09669f3700 1 --2- 172.21.2.102:0/1105741730 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f09601b0a10 0x7f09600cf330 secure :-1 s=READY pgs=2520 cs=0 l=1 rev1=1 rx=0x7f0958006a80 tx=0x7f0958009f00).stop 2021-02-10T11:34:18.057 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:18.039 7f09669f3700 1 -- 172.21.2.102:0/1105741730 shutdown_connections 2021-02-10T11:34:18.057 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:18.039 7f09669f3700 1 --2- 172.21.2.102:0/1105741730 >> v2:172.21.2.113:3300/0 conn(0x7f09600d1c70 0x7f09600cf950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:18.057 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:18.039 7f09669f3700 1 --2- 172.21.2.102:0/1105741730 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f09601b0a10 0x7f09600cf330 unknown :-1 s=CLOSED pgs=2520 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:18.057 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:18.039 7f09669f3700 1 --2- 172.21.2.102:0/1105741730 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f09600d1870 0x7f09601b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:18.057 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:18.039 7f09669f3700 1 -- 172.21.2.102:0/1105741730 >> 172.21.2.102:0/1105741730 conn(0x7f09600ce450 msgr2=0x7f09600d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:18.057 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:18.039 7f09669f3700 1 -- 172.21.2.102:0/1105741730 shutdown_connections 2021-02-10T11:34:18.057 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:18.039 7f09669f3700 1 -- 172.21.2.102:0/1105741730 wait complete. 2021-02-10T11:34:18.058 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:18.054 7f09669f3700 1 Processor -- start 2021-02-10T11:34:18.058 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:18.054 7f09669f3700 1 -- start start 2021-02-10T11:34:18.059 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:18.054 7f09669f3700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f096019ffa0 0x7f09600eda10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:18.059 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:18.054 7f09669f3700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f09601a03a0 0x7f09600ee030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:18.060 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:18.054 7f09669f3700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f09600ee650 0x7f09600ebe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:18.060 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:18.054 7f09669f3700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09600ed490 con 0x7f09600ee650 2021-02-10T11:34:18.060 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:18.054 7f09669f3700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09600eea50 con 0x7f096019ffa0 2021-02-10T11:34:18.060 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:18.054 7f09669f3700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09600eec30 con 0x7f09601a03a0 2021-02-10T11:34:18.060 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:18.054 7f0965010700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f09600ee650 0x7f09600ebe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:18.060 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:18.054 7f0965010700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f09600ee650 0x7f09600ebe80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:35954/0 (socket says 172.21.2.102:35954) 2021-02-10T11:34:18.060 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:18.054 7f0965010700 1 -- 172.21.2.102:0/3398755656 learned_addr learned my addr 172.21.2.102:0/3398755656 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:34:18.060 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:18.055 7f096480f700 1 --2- 172.21.2.102:0/3398755656 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f096019ffa0 0x7f09600eda10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:18.060 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:18.055 7f095ffff700 1 --2- 172.21.2.102:0/3398755656 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f09601a03a0 0x7f09600ee030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:18.060 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:18.055 7f0965010700 1 -- 172.21.2.102:0/3398755656 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f09601a03a0 msgr2=0x7f09600ee030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:18.060 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:18.055 7f0965010700 1 --2- 172.21.2.102:0/3398755656 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f09601a03a0 0x7f09600ee030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:18.060 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:18.055 7f0965010700 1 -- 172.21.2.102:0/3398755656 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f096019ffa0 msgr2=0x7f09600eda10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:18.060 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:18.055 7f0965010700 1 --2- 172.21.2.102:0/3398755656 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f096019ffa0 0x7f09600eda10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:18.061 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:18.055 7f0965010700 1 -- 172.21.2.102:0/3398755656 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09600ec4a0 con 0x7f09600ee650 2021-02-10T11:34:18.061 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:18.055 7f096480f700 1 --2- 172.21.2.102:0/3398755656 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f096019ffa0 0x7f09600eda10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:34:18.061 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:18.055 7f0965010700 1 --2- 172.21.2.102:0/3398755656 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f09600ee650 0x7f09600ebe80 secure :-1 s=READY pgs=2521 cs=0 l=1 rev1=1 rx=0x7f095000b5e0 tx=0x7f095000b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:18.061 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:18.055 7f095d7fa700 1 -- 172.21.2.102:0/3398755656 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f095000bd70 con 0x7f09600ee650 2021-02-10T11:34:18.061 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:18.055 7f095d7fa700 1 -- 172.21.2.102:0/3398755656 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0950006410 con 0x7f09600ee650 2021-02-10T11:34:18.061 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:18.055 7f095d7fa700 1 -- 172.21.2.102:0/3398755656 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f09500024d0 con 0x7f09600ee650 2021-02-10T11:34:18.061 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:18.055 7f09669f3700 1 -- 172.21.2.102:0/3398755656 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09600ec4a0 con 0x7f09600ee650 2021-02-10T11:34:18.061 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:18.055 7f09669f3700 1 -- 172.21.2.102:0/3398755656 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09600ec4a0 con 0x7f09600ee650 2021-02-10T11:34:18.061 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:18.056 7f095d7fa700 1 -- 172.21.2.102:0/3398755656 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f0950005330 con 0x7f09600ee650 2021-02-10T11:34:18.061 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:18.056 7f095d7fa700 1 --2- 172.21.2.102:0/3398755656 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f09440186c0 0x7f094401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:18.061 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:18.056 7f095d7fa700 1 -- 172.21.2.102:0/3398755656 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 4839+0+0 (secure 0 0 0) 0x7f09500322f0 con 0x7f09600ee650 2021-02-10T11:34:18.061 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:18.056 7f096480f700 1 --2- 172.21.2.102:0/3398755656 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f09440186c0 0x7f094401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:18.062 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:18.056 7f096480f700 1 --2- 172.21.2.102:0/3398755656 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f09440186c0 0x7f094401abc0 crc :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:18.087 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:18.085 7f09669f3700 1 -- 172.21.2.102:0/3398755656 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09600ebc50 con 0x7f09600ee650 2021-02-10T11:34:18.089 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:18.088 7f095d7fa700 1 -- 172.21.2.102:0/3398755656 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f09500067f0 con 0x7f09600ee650 2021-02-10T11:34:18.232 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:18.230 7f09669f3700 1 -- 172.21.2.102:0/3398755656 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd blacklist clear"} v 0) v1 -- 0x7f09600ebc50 con 0x7f09600ee650 2021-02-10T11:34:18.574 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:18.573 7f095d7fa700 1 -- 172.21.2.102:0/3398755656 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd blacklist clear"}]=0 removed all blacklist entries v132) v1 ==== 97+0+0 (secure 0 0 0) 0x7f0950005330 con 0x7f09600ee650 2021-02-10T11:34:18.599 INFO:teuthology.orchestra.run.gibba002.stderr: removed all blacklist entries 2021-02-10T11:34:18.601 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:18.599 7f09669f3700 1 -- 172.21.2.102:0/3398755656 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f09440186c0 msgr2=0x7f094401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:18.601 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:18.599 7f09669f3700 1 --2- 172.21.2.102:0/3398755656 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f09440186c0 0x7f094401abc0 crc :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:18.601 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:18.599 7f09669f3700 1 -- 172.21.2.102:0/3398755656 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f09600ee650 msgr2=0x7f09600ebe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:18.601 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:18.599 7f09669f3700 1 --2- 172.21.2.102:0/3398755656 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f09600ee650 0x7f09600ebe80 secure :-1 s=READY pgs=2521 cs=0 l=1 rev1=1 rx=0x7f095000b5e0 tx=0x7f095000b980).stop 2021-02-10T11:34:18.602 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:18.600 7f09669f3700 1 -- 172.21.2.102:0/3398755656 shutdown_connections 2021-02-10T11:34:18.602 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:18.600 7f09669f3700 1 --2- 172.21.2.102:0/3398755656 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f09600ee650 0x7f09600ebe80 unknown :-1 s=CLOSED pgs=2521 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:18.602 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:18.600 7f09669f3700 1 --2- 172.21.2.102:0/3398755656 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f09601a03a0 0x7f09600ee030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:18.602 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:18.600 7f09669f3700 1 --2- 172.21.2.102:0/3398755656 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f09440186c0 0x7f094401abc0 unknown :-1 s=CLOSED pgs=1113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:18.602 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:18.600 7f09669f3700 1 --2- 172.21.2.102:0/3398755656 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f096019ffa0 0x7f09600eda10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:18.602 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:18.600 7f09669f3700 1 -- 172.21.2.102:0/3398755656 >> 172.21.2.102:0/3398755656 conn(0x7f0960094d90 msgr2=0x7f096008a420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:18.602 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:18.600 7f09669f3700 1 -- 172.21.2.102:0/3398755656 shutdown_connections 2021-02-10T11:34:18.602 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:18.600 7f09669f3700 1 -- 172.21.2.102:0/3398755656 wait complete. 2021-02-10T11:34:18.610 INFO:tasks.cephfs.cephfs_test_case:['0', '1', '2', '3'] 2021-02-10T11:34:18.610 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth ls --format=json-pretty 2021-02-10T11:34:19.039 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:19.019 7ff38dfe5700 1 Processor -- start 2021-02-10T11:34:19.039 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:19.019 7ff38dfe5700 1 -- start start 2021-02-10T11:34:19.039 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:19.020 7ff38dfe5700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff3880d0340 0x7ff3880ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:19.040 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:19.020 7ff38dfe5700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7ff3880d1c70 0x7ff3880efc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:19.040 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:19.020 7ff38dfe5700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3880f0b10 con 0x7ff3880d1870 2021-02-10T11:34:19.040 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:19.020 7ff38dfe5700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3880e5ed0 con 0x7ff3880d1c70 2021-02-10T11:34:19.040 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:19.020 7ff38dfe5700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3881b0be0 con 0x7ff3880d0340 2021-02-10T11:34:19.040 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:19.020 7ff3877fe700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7ff3880d1870 0x7ff3881b03f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:56414/0 (socket says 172.21.2.102:56414) 2021-02-10T11:34:19.040 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:19.020 7ff3877fe700 1 -- 172.21.2.102:0/2205769548 learned_addr learned my addr 172.21.2.102:0/2205769548 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:34:19.040 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:19.021 7ff3867fc700 1 -- 172.21.2.102:0/2205769548 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3521601216 0 0) 0x7ff3700009b0 con 0x7ff3880d1870 2021-02-10T11:34:19.040 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:19.021 7ff3867fc700 1 -- 172.21.2.102:0/2205769548 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff374002fe0 con 0x7ff3880d1870 2021-02-10T11:34:19.040 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:19.021 7ff386ffd700 1 --2- 172.21.2.102:0/2205769548 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff3880d0340 0x7ff3880ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:19.040 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:19.021 7ff387fff700 1 --2- 172.21.2.102:0/2205769548 >> v2:172.21.2.113:3300/0 conn(0x7ff3880d1c70 0x7ff3880efc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:19.040 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:19.021 7ff3867fc700 1 -- 172.21.2.102:0/2205769548 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4284899474 0 0) 0x7ff370001010 con 0x7ff3880d1870 2021-02-10T11:34:19.040 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:19.021 7ff3867fc700 1 -- 172.21.2.102:0/2205769548 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff3880d0340 msgr2=0x7ff3880ef640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:34:19.040 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:19.021 7ff3867fc700 1 --2- 172.21.2.102:0/2205769548 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff3880d0340 0x7ff3880ef640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:19.041 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:19.021 7ff3867fc700 1 -- 172.21.2.102:0/2205769548 >> v2:172.21.2.113:3300/0 conn(0x7ff3880d1c70 msgr2=0x7ff3880efc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:34:19.041 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:19.021 7ff3867fc700 1 --2- 172.21.2.102:0/2205769548 >> v2:172.21.2.113:3300/0 conn(0x7ff3880d1c70 0x7ff3880efc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:19.041 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:19.021 7ff3867fc700 1 -- 172.21.2.102:0/2205769548 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3880f20a0 con 0x7ff3880d1870 2021-02-10T11:34:19.041 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:19.021 7ff3867fc700 1 -- 172.21.2.102:0/2205769548 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7ff370001590 con 0x7ff3880d1870 2021-02-10T11:34:19.041 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:19.021 7ff3867fc700 1 -- 172.21.2.102:0/2205769548 >> v1:172.21.2.102:6789/0 conn(0x7ff3880d1870 legacy=0x7ff3881b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:19.041 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:19.021 7ff3867fc700 1 --2- 172.21.2.102:0/2205769548 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff374003af0 0x7ff374005f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:19.041 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:19.021 7ff3867fc700 1 --2- 172.21.2.102:0/2205769548 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff37404a7b0 0x7ff37404cc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:19.041 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:19.021 7ff3867fc700 1 --2- 172.21.2.102:0/2205769548 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff37404d500 0x7ff37404fa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:19.041 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:19.021 7ff3867fc700 1 -- 172.21.2.102:0/2205769548 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3740502d0 con 0x7ff374003af0 2021-02-10T11:34:19.041 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:19.021 7ff3867fc700 1 -- 172.21.2.102:0/2205769548 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3740504f0 con 0x7ff37404d500 2021-02-10T11:34:19.041 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:19.021 7ff3867fc700 1 -- 172.21.2.102:0/2205769548 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff374050710 con 0x7ff37404a7b0 2021-02-10T11:34:19.041 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:19.021 7ff3877fe700 1 --2- 172.21.2.102:0/2205769548 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff374003af0 0x7ff374005f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:19.041 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:19.022 7ff386ffd700 1 --2- 172.21.2.102:0/2205769548 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff37404a7b0 0x7ff37404cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:19.042 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:19.022 7ff387fff700 1 --2- 172.21.2.102:0/2205769548 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff37404d500 0x7ff37404fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:19.042 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:19.022 7ff3877fe700 1 -- 172.21.2.102:0/2205769548 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff37404a7b0 msgr2=0x7ff37404cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:34:19.042 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:19.022 7ff3877fe700 1 --2- 172.21.2.102:0/2205769548 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff37404a7b0 0x7ff37404cc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:19.042 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:19.022 7ff3877fe700 1 -- 172.21.2.102:0/2205769548 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff37404d500 msgr2=0x7ff37404fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:34:19.042 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:19.022 7ff3877fe700 1 --2- 172.21.2.102:0/2205769548 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff37404d500 0x7ff37404fa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:19.042 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:19.022 7ff3877fe700 1 -- 172.21.2.102:0/2205769548 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3740509b0 con 0x7ff374003af0 2021-02-10T11:34:19.042 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:19.022 7ff3877fe700 1 --2- 172.21.2.102:0/2205769548 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff374003af0 0x7ff374005f00 secure :-1 s=READY pgs=2523 cs=0 l=1 rev1=1 rx=0x7ff3700017d0 tx=0x7ff370007d30).ready entity=mon.0 client_cookie=735946a206f50f64 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:19.042 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:19.022 7ff3867fc700 1 -- 172.21.2.102:0/2205769548 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff37000a310 con 0x7ff374003af0 2021-02-10T11:34:19.042 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:19.022 7ff3867fc700 1 -- 172.21.2.102:0/2205769548 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff37000a690 con 0x7ff374003af0 2021-02-10T11:34:19.042 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:19.022 7ff3867fc700 1 -- 172.21.2.102:0/2205769548 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff37000aaf0 con 0x7ff374003af0 2021-02-10T11:34:19.042 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:19.022 7ff38dfe5700 1 -- 172.21.2.102:0/2205769548 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff374003af0 msgr2=0x7ff374005f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:19.042 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:19.022 7ff38dfe5700 1 --2- 172.21.2.102:0/2205769548 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff374003af0 0x7ff374005f00 secure :-1 s=READY pgs=2523 cs=0 l=1 rev1=1 rx=0x7ff3700017d0 tx=0x7ff370007d30).stop 2021-02-10T11:34:19.042 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:19.022 7ff387fff700 1 -- 172.21.2.102:0/2205769548 reap_dead start 2021-02-10T11:34:19.043 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:19.023 7ff38dfe5700 1 -- 172.21.2.102:0/2205769548 shutdown_connections 2021-02-10T11:34:19.043 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:19.023 7ff38dfe5700 1 -- 172.21.2.102:0/2205769548 >> 172.21.2.102:0/2205769548 conn(0x7ff3880ce450 msgr2=0x7ff3880d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:19.043 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:19.023 7ff38dfe5700 1 -- 172.21.2.102:0/2205769548 shutdown_connections 2021-02-10T11:34:19.043 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:19.023 7ff38dfe5700 1 -- 172.21.2.102:0/2205769548 wait complete. 2021-02-10T11:34:19.043 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:19.038 7ff38dfe5700 1 Processor -- start 2021-02-10T11:34:19.043 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:19.038 7ff38dfe5700 1 -- start start 2021-02-10T11:34:19.049 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:19.038 7ff38dfe5700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff388195650 0x7ff3880b5560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:19.049 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:19.038 7ff38dfe5700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff3880b5b80 0x7ff3881a2e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:19.049 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:19.038 7ff38dfe5700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff3880acfa0 0x7ff3881a3440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:19.049 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:19.038 7ff38dfe5700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3880b6010 con 0x7ff3880b5b80 2021-02-10T11:34:19.049 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:19.038 7ff38dfe5700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3881a3a60 con 0x7ff3880acfa0 2021-02-10T11:34:19.049 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:19.038 7ff38dfe5700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3881a3c40 con 0x7ff388195650 2021-02-10T11:34:19.049 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:19.038 7ff386ffd700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff3880b5b80 0x7ff3881a2e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:19.049 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:19.038 7ff386ffd700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff3880b5b80 0x7ff3881a2e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:35974/0 (socket says 172.21.2.102:35974) 2021-02-10T11:34:19.049 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:19.038 7ff386ffd700 1 -- 172.21.2.102:0/1278632346 learned_addr learned my addr 172.21.2.102:0/1278632346 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:34:19.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:19.038 7ff387fff700 1 --2- 172.21.2.102:0/1278632346 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff3880acfa0 0x7ff3881a3440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:19.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:19.038 7ff3877fe700 1 --2- 172.21.2.102:0/1278632346 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff388195650 0x7ff3880b5560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:19.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:19.039 7ff386ffd700 1 -- 172.21.2.102:0/1278632346 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff388195650 msgr2=0x7ff3880b5560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:19.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:19.039 7ff386ffd700 1 --2- 172.21.2.102:0/1278632346 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff388195650 0x7ff3880b5560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:19.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:19.039 7ff386ffd700 1 -- 172.21.2.102:0/1278632346 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff3880acfa0 msgr2=0x7ff3881a3440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:19.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:19.039 7ff386ffd700 1 --2- 172.21.2.102:0/1278632346 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff3880acfa0 0x7ff3881a3440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:19.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:19.039 7ff386ffd700 1 -- 172.21.2.102:0/1278632346 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3880bcae0 con 0x7ff3880b5b80 2021-02-10T11:34:19.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:19.039 7ff387fff700 1 --2- 172.21.2.102:0/1278632346 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff3880acfa0 0x7ff3881a3440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:34:19.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:19.039 7ff386ffd700 1 --2- 172.21.2.102:0/1278632346 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff3880b5b80 0x7ff3881a2e20 secure :-1 s=READY pgs=2524 cs=0 l=1 rev1=1 rx=0x7ff378006f60 tx=0x7ff37800b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:19.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:19.039 7ff37ffff700 1 -- 172.21.2.102:0/1278632346 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff37800bd60 con 0x7ff3880b5b80 2021-02-10T11:34:19.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:19.039 7ff37ffff700 1 -- 172.21.2.102:0/1278632346 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff378000c90 con 0x7ff3880b5b80 2021-02-10T11:34:19.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:19.039 7ff37ffff700 1 -- 172.21.2.102:0/1278632346 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff378002220 con 0x7ff3880b5b80 2021-02-10T11:34:19.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:19.039 7ff38dfe5700 1 -- 172.21.2.102:0/1278632346 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3880bcae0 con 0x7ff3880b5b80 2021-02-10T11:34:19.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:19.039 7ff38dfe5700 1 -- 172.21.2.102:0/1278632346 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3880bd070 con 0x7ff3880b5b80 2021-02-10T11:34:19.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:19.040 7ff37ffff700 1 -- 172.21.2.102:0/1278632346 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ff37800a000 con 0x7ff3880b5b80 2021-02-10T11:34:19.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:19.040 7ff37ffff700 1 --2- 172.21.2.102:0/1278632346 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff36c0186c0 0x7ff36c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:19.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:19.040 7ff37ffff700 1 -- 172.21.2.102:0/1278632346 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 4753+0+0 (secure 0 0 0) 0x7ff378032430 con 0x7ff3880b5b80 2021-02-10T11:34:19.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:19.040 7ff3877fe700 1 --2- 172.21.2.102:0/1278632346 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff36c0186c0 0x7ff36c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:19.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:19.040 7ff3877fe700 1 --2- 172.21.2.102:0/1278632346 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff36c0186c0 0x7ff36c01abc0 crc :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:19.072 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:19.070 7ff38dfe5700 1 -- 172.21.2.102:0/1278632346 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3880f0ab0 con 0x7ff3880b5b80 2021-02-10T11:34:19.074 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:19.072 7ff37ffff700 1 -- 172.21.2.102:0/1278632346 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ff378000940 con 0x7ff3880b5b80 2021-02-10T11:34:19.207 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:19.205 7ff38dfe5700 1 -- 172.21.2.102:0/1278632346 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7ff3880f0ab0 con 0x7ff3880b5b80 2021-02-10T11:34:19.207 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:19.205 7ff37ffff700 1 -- 172.21.2.102:0/1278632346 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v76) v1 ==== 80+0+4513 (secure 0 0 0) 0x7ff378000940 con 0x7ff3880b5b80 2021-02-10T11:34:19.208 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:34:19.208 INFO:teuthology.orchestra.run.gibba002.stdout:{ 2021-02-10T11:34:19.208 INFO:teuthology.orchestra.run.gibba002.stdout: "auth_dump": [ 2021-02-10T11:34:19.208 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:19.208 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "mds.a", 2021-02-10T11:34:19.208 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAevSNgsgyEARAAvr0cUjjCoEHLxaZSZ2HtVA==", 2021-02-10T11:34:19.208 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:34:19.208 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow", 2021-02-10T11:34:19.208 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:34:19.208 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:34:19.208 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:34:19.209 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:34:19.209 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:19.209 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:19.209 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "mds.b", 2021-02-10T11:34:19.209 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAevSNgwOFMMRAALfb1fbX56ao1TU8ClnTW1A==", 2021-02-10T11:34:19.209 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:34:19.209 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow", 2021-02-10T11:34:19.209 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:34:19.209 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:34:19.209 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:34:19.209 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:34:19.209 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:19.209 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:19.210 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.0", 2021-02-10T11:34:19.210 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAmvSNgj+bMMBAAD3HUD6fLiMAg+++iNzLJKQ==", 2021-02-10T11:34:19.210 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:34:19.210 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:34:19.210 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:34:19.210 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:34:19.210 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:34:19.210 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:19.210 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:19.210 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.1", 2021-02-10T11:34:19.210 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQApvSNgnvc+MRAAAOxqlNwiyLHqNITOrYeFKQ==", 2021-02-10T11:34:19.210 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:34:19.210 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:34:19.210 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:34:19.211 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:34:19.211 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:34:19.211 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:19.211 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:19.211 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.2", 2021-02-10T11:34:19.211 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAsvSNgZ4SbMRAA76IOYWfbiAuBoM/4k81GxA==", 2021-02-10T11:34:19.211 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:34:19.211 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:34:19.211 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:34:19.211 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:34:19.211 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:34:19.211 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:19.211 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:19.211 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.3", 2021-02-10T11:34:19.212 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAvvSNgFk3rMRAAWCmKKyw0LDQ+UlAHZAITOw==", 2021-02-10T11:34:19.212 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:34:19.212 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:34:19.212 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:34:19.212 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:34:19.212 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:34:19.212 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:19.212 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:19.212 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.4", 2021-02-10T11:34:19.212 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQA4vSNgpKcNJBAApJMmvj5YxVHPDOmvmQONOg==", 2021-02-10T11:34:19.212 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:34:19.212 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:34:19.212 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:34:19.212 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:34:19.213 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:34:19.213 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:19.213 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:19.213 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.5", 2021-02-10T11:34:19.213 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQA7vSNgf2ujJBAAfkuYpyb3WTzc8V5kw+Wktg==", 2021-02-10T11:34:19.213 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:34:19.213 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:34:19.213 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:34:19.213 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:34:19.213 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:34:19.213 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:19.213 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:19.213 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.6", 2021-02-10T11:34:19.213 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQA+vSNgqw9uJRAAFMU0R+UxU9C3o/9byb3e0g==", 2021-02-10T11:34:19.214 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:34:19.214 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:34:19.214 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:34:19.214 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:34:19.214 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:34:19.214 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:19.214 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:19.214 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.7", 2021-02-10T11:34:19.214 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQBBvSNgvWSrJRAA2KaCHmLpzovRgYzLvyofTQ==", 2021-02-10T11:34:19.214 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:34:19.214 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:34:19.214 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:34:19.214 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:34:19.214 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:34:19.215 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:19.215 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:19.215 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "client.0", 2021-02-10T11:34:19.215 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAfvSNgZd9XIBAAvyrvl8fdKuHnJZfczfRoFw==", 2021-02-10T11:34:19.215 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:34:19.215 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow", 2021-02-10T11:34:19.215 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow r", 2021-02-10T11:34:19.215 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:34:19.215 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:34:19.215 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:19.215 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:19.215 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "client.1", 2021-02-10T11:34:19.215 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAfvSNgSyfXNhAAIduPnF7Vd/6HSkJ0MlcMSA==", 2021-02-10T11:34:19.215 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:34:19.216 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow", 2021-02-10T11:34:19.216 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow r", 2021-02-10T11:34:19.216 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:34:19.216 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:34:19.216 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:19.216 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:19.216 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "client.2", 2021-02-10T11:34:19.216 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAgvSNgiIG/ERAA9ZTamaTI7ZsPSnxBqjplCg==", 2021-02-10T11:34:19.216 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:34:19.216 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow", 2021-02-10T11:34:19.216 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow r", 2021-02-10T11:34:19.216 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:34:19.216 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:34:19.216 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:19.217 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:19.217 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "client.3", 2021-02-10T11:34:19.217 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAgvSNgOWpOKBAAlXsEB04Q/GvtL06PqRMLeA==", 2021-02-10T11:34:19.217 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:34:19.217 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow", 2021-02-10T11:34:19.217 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow r", 2021-02-10T11:34:19.217 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:34:19.217 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:34:19.217 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:19.217 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:19.217 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "client.admin", 2021-02-10T11:34:19.217 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAavSNgO3GQBhAAL+NVnYskQRKVWLgyDPY/xw==", 2021-02-10T11:34:19.217 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:34:19.217 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow *", 2021-02-10T11:34:19.218 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:34:19.218 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:34:19.218 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:34:19.218 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:34:19.218 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:19.218 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:19.218 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "mgr.x", 2021-02-10T11:34:19.218 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAdvSNgXZZgJhAA7dtBSGzH+DsBwnddZpOBJA==", 2021-02-10T11:34:19.218 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:34:19.218 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow *", 2021-02-10T11:34:19.218 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow profile mgr", 2021-02-10T11:34:19.218 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:34:19.218 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:34:19.218 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:19.219 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:19.219 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "mgr.y", 2021-02-10T11:34:19.219 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAdvSNgVqQFDRAAwo6YEdWZCAr5GdOWVL/Qvg==", 2021-02-10T11:34:19.219 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:34:19.219 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow *", 2021-02-10T11:34:19.219 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow profile mgr", 2021-02-10T11:34:19.219 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:34:19.219 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:34:19.219 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:34:19.219 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:34:19.219 INFO:teuthology.orchestra.run.gibba002.stdout:} 2021-02-10T11:34:19.220 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:19.207 7ff38dfe5700 1 -- 172.21.2.102:0/1278632346 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff36c0186c0 msgr2=0x7ff36c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:19.220 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:19.207 7ff38dfe5700 1 --2- 172.21.2.102:0/1278632346 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff36c0186c0 0x7ff36c01abc0 crc :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:19.220 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:19.207 7ff38dfe5700 1 -- 172.21.2.102:0/1278632346 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff3880b5b80 msgr2=0x7ff3881a2e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:19.221 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:19.207 7ff38dfe5700 1 --2- 172.21.2.102:0/1278632346 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff3880b5b80 0x7ff3881a2e20 secure :-1 s=READY pgs=2524 cs=0 l=1 rev1=1 rx=0x7ff378006f60 tx=0x7ff37800b970).stop 2021-02-10T11:34:19.221 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:19.207 7ff38dfe5700 1 -- 172.21.2.102:0/1278632346 shutdown_connections 2021-02-10T11:34:19.221 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:19.207 7ff38dfe5700 1 --2- 172.21.2.102:0/1278632346 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff36c0186c0 0x7ff36c01abc0 unknown :-1 s=CLOSED pgs=1114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:19.221 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:19.207 7ff38dfe5700 1 --2- 172.21.2.102:0/1278632346 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff3880acfa0 0x7ff3881a3440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:19.221 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:19.207 7ff38dfe5700 1 --2- 172.21.2.102:0/1278632346 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff3880b5b80 0x7ff3881a2e20 unknown :-1 s=CLOSED pgs=2524 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:19.221 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:19.207 7ff38dfe5700 1 --2- 172.21.2.102:0/1278632346 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff388195650 0x7ff3880b5560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:19.221 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:19.207 7ff38dfe5700 1 -- 172.21.2.102:0/1278632346 >> 172.21.2.102:0/1278632346 conn(0x7ff3880b3040 msgr2=0x7ff3880c3250 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:19.221 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:19.208 7ff38dfe5700 1 -- 172.21.2.102:0/1278632346 shutdown_connections 2021-02-10T11:34:19.221 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:19.208 7ff38dfe5700 1 -- 172.21.2.102:0/1278632346 wait complete. 2021-02-10T11:34:19.222 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:34:19.651 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:19.632 7f18bbea8700 1 Processor -- start 2021-02-10T11:34:19.651 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:19.632 7f18bbea8700 1 -- start start 2021-02-10T11:34:19.651 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:19.633 7f18bbea8700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f18b40d1870 0x7f18b41b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:19.651 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:19.633 7f18bbea8700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f18b40f0650 0x7f18b40cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:19.651 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:19.633 7f18bbea8700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18b40f0b10 con 0x7f18b41b0a10 2021-02-10T11:34:19.651 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:19.633 7f18bbea8700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18b40e5ed0 con 0x7f18b40d1870 2021-02-10T11:34:19.652 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:19.633 7f18bbea8700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18b40d1c70 con 0x7f18b40f0650 2021-02-10T11:34:19.652 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:19.633 7f18b9cc4700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f18b40d1870 0x7f18b41b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:19.652 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:19.633 7f18b94c3700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f18b41b0a10 0x7f18b40cf330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:56434/0 (socket says 172.21.2.102:56434) 2021-02-10T11:34:19.652 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:19.633 7f18b94c3700 1 -- 172.21.2.102:0/2198763083 learned_addr learned my addr 172.21.2.102:0/2198763083 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:34:19.652 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:19.633 7f18ba4c5700 1 --2- 172.21.2.102:0/2198763083 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f18b40f0650 0x7f18b40cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:19.652 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:19.634 7f18b9cc4700 1 -- 172.21.2.102:0/2198763083 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f18b40f0650 msgr2=0x7f18b40cfb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:34:19.652 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:19.634 7f18b9cc4700 1 --2- 172.21.2.102:0/2198763083 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f18b40f0650 0x7f18b40cfb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:19.652 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:19.634 7f18b9cc4700 1 -- 172.21.2.102:0/2198763083 >> v1:172.21.2.102:6789/0 conn(0x7f18b41b0a10 legacy=0x7f18b40cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:19.652 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:19.634 7f18b9cc4700 1 -- 172.21.2.102:0/2198763083 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18b40d1e90 con 0x7f18b40d1870 2021-02-10T11:34:19.652 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:19.634 7f18b9cc4700 1 --2- 172.21.2.102:0/2198763083 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f18b40d1870 0x7f18b41b03f0 secure :-1 s=READY pgs=2526 cs=0 l=1 rev1=1 rx=0x7f18b0006650 tx=0x7f18b0009a80).ready entity=mon.0 client_cookie=fb3ffd7a5f38c8d server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:19.652 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:19.634 7f18b8cc2700 1 -- 172.21.2.102:0/2198763083 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f18b000c0c0 con 0x7f18b40d1870 2021-02-10T11:34:19.652 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:19.634 7f18b8cc2700 1 -- 172.21.2.102:0/2198763083 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18b000c340 con 0x7f18b40d1870 2021-02-10T11:34:19.652 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:19.634 7f18b8cc2700 1 -- 172.21.2.102:0/2198763083 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f18b000c860 con 0x7f18b40d1870 2021-02-10T11:34:19.653 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:19.634 7f18bbea8700 1 -- 172.21.2.102:0/2198763083 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f18b40d1870 msgr2=0x7f18b41b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:19.653 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:19.634 7f18bbea8700 1 --2- 172.21.2.102:0/2198763083 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f18b40d1870 0x7f18b41b03f0 secure :-1 s=READY pgs=2526 cs=0 l=1 rev1=1 rx=0x7f18b0006650 tx=0x7f18b0009a80).stop 2021-02-10T11:34:19.653 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:19.635 7f18bbea8700 1 -- 172.21.2.102:0/2198763083 shutdown_connections 2021-02-10T11:34:19.653 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:19.635 7f18bbea8700 1 --2- 172.21.2.102:0/2198763083 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f18b40f0650 0x7f18b40cfb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:19.653 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:19.635 7f18bbea8700 1 --2- 172.21.2.102:0/2198763083 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f18b40d1870 0x7f18b41b03f0 unknown :-1 s=CLOSED pgs=2526 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:19.654 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:19.635 7f18bbea8700 1 -- 172.21.2.102:0/2198763083 >> 172.21.2.102:0/2198763083 conn(0x7f18b40ce450 msgr2=0x7f18b40d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:19.654 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:19.635 7f18bbea8700 1 -- 172.21.2.102:0/2198763083 shutdown_connections 2021-02-10T11:34:19.654 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:19.635 7f18bbea8700 1 -- 172.21.2.102:0/2198763083 wait complete. 2021-02-10T11:34:19.655 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:19.650 7f18bbea8700 1 Processor -- start 2021-02-10T11:34:19.656 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:19.650 7f18bbea8700 1 -- start start 2021-02-10T11:34:19.656 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:19.650 7f18bbea8700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f18b418d870 0x7f18b40b19b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:19.656 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:19.650 7f18bbea8700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f18b40b1fd0 0x7f18b4099ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:19.656 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:19.650 7f18bbea8700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f18b40b23d0 0x7f18b409a510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:19.656 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:19.650 7f18bbea8700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18b418dc70 con 0x7f18b40b23d0 2021-02-10T11:34:19.656 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:19.650 7f18bbea8700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18b4199e60 con 0x7f18b418d870 2021-02-10T11:34:19.656 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:19.650 7f18bbea8700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18b409ab30 con 0x7f18b40b1fd0 2021-02-10T11:34:19.656 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:19.650 7f18ba4c5700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f18b40b23d0 0x7f18b409a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:19.656 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:19.650 7f18ba4c5700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f18b40b23d0 0x7f18b409a510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:35986/0 (socket says 172.21.2.102:35986) 2021-02-10T11:34:19.656 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:19.650 7f18ba4c5700 1 -- 172.21.2.102:0/2683385934 learned_addr learned my addr 172.21.2.102:0/2683385934 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:34:19.656 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:19.650 7f18b9cc4700 1 --2- 172.21.2.102:0/2683385934 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f18b418d870 0x7f18b40b19b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:19.657 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:19.650 7f18b94c3700 1 --2- 172.21.2.102:0/2683385934 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f18b40b1fd0 0x7f18b4099ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:19.657 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:19.651 7f18b9cc4700 1 -- 172.21.2.102:0/2683385934 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f18b40b1fd0 msgr2=0x7f18b4099ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:19.657 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:19.651 7f18b9cc4700 1 --2- 172.21.2.102:0/2683385934 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f18b40b1fd0 0x7f18b4099ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:19.657 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:19.651 7f18b9cc4700 1 -- 172.21.2.102:0/2683385934 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f18b40b23d0 msgr2=0x7f18b409a510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:19.657 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:19.651 7f18b9cc4700 1 --2- 172.21.2.102:0/2683385934 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f18b40b23d0 0x7f18b409a510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:19.657 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:19.651 7f18b9cc4700 1 -- 172.21.2.102:0/2683385934 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18b409ad50 con 0x7f18b418d870 2021-02-10T11:34:19.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:19.651 7f18ba4c5700 1 --2- 172.21.2.102:0/2683385934 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f18b40b23d0 0x7f18b409a510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:34:19.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:19.651 7f18b9cc4700 1 --2- 172.21.2.102:0/2683385934 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f18b418d870 0x7f18b40b19b0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 rx=0x7f18b000b530 tx=0x7f18b000b8d0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:19.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:19.651 7f18a67fc700 1 -- 172.21.2.102:0/2683385934 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f18b00093a0 con 0x7f18b418d870 2021-02-10T11:34:19.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:19.651 7f18a67fc700 1 -- 172.21.2.102:0/2683385934 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18b000bcc0 con 0x7f18b418d870 2021-02-10T11:34:19.659 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:19.651 7f18bbea8700 1 -- 172.21.2.102:0/2683385934 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18b409ad50 con 0x7f18b418d870 2021-02-10T11:34:19.659 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:19.651 7f18a67fc700 1 -- 172.21.2.102:0/2683385934 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f18b0002220 con 0x7f18b418d870 2021-02-10T11:34:19.659 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:19.651 7f18bbea8700 1 -- 172.21.2.102:0/2683385934 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18b409ad50 con 0x7f18b418d870 2021-02-10T11:34:19.659 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:19.652 7f18a67fc700 1 -- 172.21.2.102:0/2683385934 <== mon.1 v2:172.21.2.113:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f18b0009280 con 0x7f18b418d870 2021-02-10T11:34:19.659 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:19.652 7f18a67fc700 1 --2- 172.21.2.102:0/2683385934 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f189c018730 0x7f189c01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:19.659 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:19.652 7f18a67fc700 1 -- 172.21.2.102:0/2683385934 <== mon.1 v2:172.21.2.113:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 4753+0+0 (secure 0 0 0) 0x7f18b0032200 con 0x7f18b418d870 2021-02-10T11:34:19.659 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:19.652 7f18b94c3700 1 --2- 172.21.2.102:0/2683385934 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f189c018730 0x7f189c01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:19.659 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:19.652 7f18b94c3700 1 --2- 172.21.2.102:0/2683385934 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f189c018730 0x7f189c01ac10 crc :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:19.685 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:19.683 7f18bbea8700 1 -- 172.21.2.102:0/2683385934 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18b40999c0 con 0x7f18b418d870 2021-02-10T11:34:19.689 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:19.687 7f18a67fc700 1 -- 172.21.2.102:0/2683385934 <== mon.1 v2:172.21.2.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f18b0000c90 con 0x7f18b418d870 2021-02-10T11:34:19.818 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:19.816 7f18bbea8700 1 -- 172.21.2.102:0/2683385934 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f18b40999c0 con 0x7f18b418d870 2021-02-10T11:34:19.819 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:19.818 7f18a67fc700 1 -- 172.21.2.102:0/2683385934 <== mon.1 v2:172.21.2.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v132) v1 ==== 74+0+11154 (secure 0 0 0) 0x7f18b0009280 con 0x7f18b418d870 2021-02-10T11:34:19.821 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:34:19.821 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":132,"fsid":"68db30ea-dbce-4aef-a6df-f01c6fea0513","created":"2021-02-10 11:03:06.331557","modified":"2021-02-10 11:34:18.567569","last_up_change":"2021-02-10 11:03:32.690562","last_in_change":"2021-02-10 11:03:32.690562","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":27,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:03:37.322519","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"be691b4d-a039-408a-af80-9d53f02b6f1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6818","nonce":36088},{"type":"v1","addr":"172.21.2.102:6820","nonce":36088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6822","nonce":36088},{"type":"v1","addr":"172.21.2.102:6824","nonce":36088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6830","nonce":36088},{"type":"v1","addr":"172.21.2.102:6832","nonce":36088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6826","nonce":36088},{"type":"v1","addr":"172.21.2.102:6828","nonce":36088}]},"public_addr":"172.21.2.102:6820/36088","cluster_addr":"172.21.2.102:6824/36088","heartbeat_back_addr":"172.21.2.102:6832/36088","heartbeat_front_addr":"172.21.2.102:6828/36088","state":["exists","up"]},{"osd":1,"uuid":"64e3ccc8-9c69-4d7c-865e-af8e83822635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6819","nonce":36087},{"type":"v1","addr":"172.21.2.102:6821","nonce":36087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6823","nonce":36087},{"type":"v1","addr":"172.21.2.102:6825","nonce":36087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6831","nonce":36087},{"type":"v1","addr":"172.21.2.102:6833","nonce":36087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6827","nonce":36087},{"type":"v1","addr":"172.21.2.102:6829","nonce":36087}]},"public_addr":"172.21.2.102:6821/36087","cluster_addr":"172.21.2.102:6825/36087","heartbeat_back_addr":"172.21.2.102:6833/36087","heartbeat_front_addr":"172.21.2.102:6829/36087","state":["exists","up"]},{"osd":2,"uuid":"153700af-747f-481f-bdf4-d6072d3a842f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6810","nonce":36089},{"type":"v1","addr":"172.21.2.102:6811","nonce":36089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6812","nonce":36089},{"type":"v1","addr":"172.21.2.102:6813","nonce":36089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6816","nonce":36089},{"type":"v1","addr":"172.21.2.102:6817","nonce":36089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6814","nonce":36089},{"type":"v1","addr":"172.21.2.102:6815","nonce":36089}]},"public_addr":"172.21.2.102:6811/36089","cluster_addr":"172.21.2.102:6813/36089","heartbeat_back_addr":"172.21.2.102:6817/36089","heartbeat_front_addr":"172.21.2.102:6815/36089","state":["exists","up"]},{"osd":3,"uuid":"cbc678f4-83a9-45e6-8830-2f7c232b1ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6802","nonce":36090},{"type":"v1","addr":"172.21.2.102:6803","nonce":36090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6804","nonce":36090},{"type":"v1","addr":"172.21.2.102:6805","nonce":36090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6808","nonce":36090},{"type":"v1","addr":"172.21.2.102:6809","nonce":36090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6806","nonce":36090},{"type":"v1","addr":"172.21.2.102:6807","nonce":36090}]},"public_addr":"172.21.2.102:6803/36090","cluster_addr":"172.21.2.102:6805/36090","heartbeat_back_addr":"172.21.2.102:6809/36090","heartbeat_front_addr":"172.21.2.102:6807/36090","state":["exists","up"]},{"osd":4,"uuid":"c01e4008-b258-4f10-afd2-9e7f37214351","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6808","nonce":36368},{"type":"v1","addr":"172.21.2.113:6811","nonce":36368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6814","nonce":36368},{"type":"v1","addr":"172.21.2.113:6818","nonce":36368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6826","nonce":36368},{"type":"v1","addr":"172.21.2.113:6828","nonce":36368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6820","nonce":36368},{"type":"v1","addr":"172.21.2.113:6824","nonce":36368}]},"public_addr":"172.21.2.113:6811/36368","cluster_addr":"172.21.2.113:6818/36368","heartbeat_back_addr":"172.21.2.113:6828/36368","heartbeat_front_addr":"172.21.2.113:6824/36368","state":["exists","up"]},{"osd":5,"uuid":"551f6007-daf8-4a98-8ee0-c2b16bcccb5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6802","nonce":36366},{"type":"v1","addr":"172.21.2.113:6805","nonce":36366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6807","nonce":36366},{"type":"v1","addr":"172.21.2.113:6810","nonce":36366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6819","nonce":36366},{"type":"v1","addr":"172.21.2.113:6822","nonce":36366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6812","nonce":36366},{"type":"v1","addr":"172.21.2.113:6815","nonce":36366}]},"public_addr":"172.21.2.113:6805/36366","cluster_addr":"172.21.2.113:6810/36366","heartbeat_back_addr":"172.21.2.113:6822/36366","heartbeat_front_addr":"172.21.2.113:6815/36366","state":["exists","up"]},{"osd":6,"uuid":"bf67c040-0e50-48be-b102-c3943bc9ebef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6800","nonce":36367},{"type":"v1","addr":"172.21.2.113:6801","nonce":36367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6803","nonce":36367},{"type":"v1","addr":"172.21.2.113:6804","nonce":36367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6813","nonce":36367},{"type":"v1","addr":"172.21.2.113:6816","nonce":36367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6806","nonce":36367},{"type":"v1","addr":"172.21.2.113:6809","nonce":36367}]},"public_addr":"172.21.2.113:6801/36367","cluster_addr":"172.21.2.113:6804/36367","heartbeat_back_addr":"172.21.2.113:6816/36367","heartbeat_front_addr":"172.21.2.113:6809/36367","state":["exists","up"]},{"osd":7,"uuid":"f5461f80-018d-428f-be89-2eb2bcbe7bb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6817","nonce":36365},{"type":"v1","addr":"172.21.2.113:6821","nonce":36365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6823","nonce":36365},{"type":"v1","addr":"172.21.2.113:6825","nonce":36365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6830","nonce":36365},{"type":"v1","addr":"172.21.2.113:6831","nonce":36365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6827","nonce":36365},{"type":"v1","addr":"172.21.2.113:6829","nonce":36365}]},"public_addr":"172.21.2.113:6821/36365","cluster_addr":"172.21.2.113:6825/36365","heartbeat_back_addr":"172.21.2.113:6831/36365","heartbeat_front_addr":"172.21.2.113:6829/36365","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:34:19.822 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:19.820 7f18bbea8700 1 -- 172.21.2.102:0/2683385934 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f189c018730 msgr2=0x7f189c01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:19.822 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:19.820 7f18bbea8700 1 --2- 172.21.2.102:0/2683385934 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f189c018730 0x7f189c01ac10 crc :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:19.822 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:19.820 7f18bbea8700 1 -- 172.21.2.102:0/2683385934 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f18b418d870 msgr2=0x7f18b40b19b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:19.822 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:19.820 7f18bbea8700 1 --2- 172.21.2.102:0/2683385934 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f18b418d870 0x7f18b40b19b0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 rx=0x7f18b000b530 tx=0x7f18b000b8d0).stop 2021-02-10T11:34:19.823 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:19.821 7f18bbea8700 1 -- 172.21.2.102:0/2683385934 shutdown_connections 2021-02-10T11:34:19.823 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:19.821 7f18bbea8700 1 --2- 172.21.2.102:0/2683385934 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f18b40b23d0 0x7f18b409a510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:19.823 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:19.821 7f18bbea8700 1 --2- 172.21.2.102:0/2683385934 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f18b40b1fd0 0x7f18b4099ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:19.823 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:19.821 7f18bbea8700 1 --2- 172.21.2.102:0/2683385934 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f189c018730 0x7f189c01ac10 unknown :-1 s=CLOSED pgs=1115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:19.823 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:19.821 7f18bbea8700 1 --2- 172.21.2.102:0/2683385934 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f18b418d870 0x7f18b40b19b0 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:19.823 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:19.821 7f18bbea8700 1 -- 172.21.2.102:0/2683385934 >> 172.21.2.102:0/2683385934 conn(0x7f18b40f0550 msgr2=0x7f18b4191730 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:19.823 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:19.821 7f18bbea8700 1 -- 172.21.2.102:0/2683385934 shutdown_connections 2021-02-10T11:34:19.823 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:19.821 7f18bbea8700 1 -- 172.21.2.102:0/2683385934 wait complete. 2021-02-10T11:34:19.831 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:03:37.322519', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2021-02-10T11:34:19.831 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:34:20.262 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:20.243 7f18e4123700 1 Processor -- start 2021-02-10T11:34:20.262 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:20.243 7f18e4123700 1 -- start start 2021-02-10T11:34:20.262 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:20.244 7f18e4123700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f18dc0d1870 0x7f18dc1b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:20.263 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:20.244 7f18e4123700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f18dc0cff70 0x7f18dc109130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:20.263 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:20.244 7f18e4123700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18dc109a60 con 0x7f18dc1b0ae0 2021-02-10T11:34:20.263 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:20.244 7f18e4123700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18dc109750 con 0x7f18dc0cff70 2021-02-10T11:34:20.263 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:20.244 7f18e4123700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18dc088070 con 0x7f18dc0d1870 2021-02-10T11:34:20.263 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:20.244 7f18e173e700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f18dc1b0ae0 0x7f18dc108940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:56446/0 (socket says 172.21.2.102:56446) 2021-02-10T11:34:20.263 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:20.244 7f18e173e700 1 -- 172.21.2.102:0/1351408118 learned_addr learned my addr 172.21.2.102:0/1351408118 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:34:20.263 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:20.244 7f18e2740700 1 --2- 172.21.2.102:0/1351408118 >> v2:172.21.2.113:3300/0 conn(0x7f18dc0cff70 0x7f18dc109130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:20.263 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:20.244 7f18e1f3f700 1 --2- 172.21.2.102:0/1351408118 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f18dc0d1870 0x7f18dc1b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:20.263 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:20.245 7f18e2740700 1 -- 172.21.2.102:0/1351408118 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f18dc0d1870 msgr2=0x7f18dc1b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:34:20.263 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:20.245 7f18e2740700 1 --2- 172.21.2.102:0/1351408118 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f18dc0d1870 0x7f18dc1b04c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:20.263 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:20.245 7f18e2740700 1 -- 172.21.2.102:0/1351408118 >> v1:172.21.2.102:6789/0 conn(0x7f18dc1b0ae0 legacy=0x7f18dc108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:20.263 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:20.245 7f18e2740700 1 -- 172.21.2.102:0/1351408118 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18dc088260 con 0x7f18dc0cff70 2021-02-10T11:34:20.264 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:20.245 7f18e0f3d700 1 -- 172.21.2.102:0/1351408118 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3827850163 0 0) 0x7f18d00009b0 con 0x7f18dc1b0ae0 2021-02-10T11:34:20.264 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:20.245 7f18e2740700 1 --2- 172.21.2.102:0/1351408118 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f18dc0cff70 0x7f18dc109130 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 rx=0x7f18d4006e50 tx=0x7f18d4009e70).ready entity=mon.1 client_cookie=682b3fd23e2a9bcc server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:20.264 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:20.245 7f18e0f3d700 1 -- 172.21.2.102:0/1351408118 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f18d400b510 con 0x7f18dc0cff70 2021-02-10T11:34:20.264 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:20.245 7f18e0f3d700 1 -- 172.21.2.102:0/1351408118 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18d4006790 con 0x7f18dc0cff70 2021-02-10T11:34:20.264 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:20.245 7f18e0f3d700 1 -- 172.21.2.102:0/1351408118 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f18d400b940 con 0x7f18dc0cff70 2021-02-10T11:34:20.264 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:20.245 7f18e4123700 1 -- 172.21.2.102:0/1351408118 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f18dc0cff70 msgr2=0x7f18dc109130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:20.264 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:20.245 7f18e4123700 1 --2- 172.21.2.102:0/1351408118 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f18dc0cff70 0x7f18dc109130 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 rx=0x7f18d4006e50 tx=0x7f18d4009e70).stop 2021-02-10T11:34:20.264 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:20.246 7f18e4123700 1 -- 172.21.2.102:0/1351408118 shutdown_connections 2021-02-10T11:34:20.264 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:20.246 7f18e4123700 1 --2- 172.21.2.102:0/1351408118 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f18dc0cff70 0x7f18dc109130 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:20.264 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:20.246 7f18e4123700 1 --2- 172.21.2.102:0/1351408118 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f18dc0d1870 0x7f18dc1b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:20.264 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:20.246 7f18e4123700 1 -- 172.21.2.102:0/1351408118 >> 172.21.2.102:0/1351408118 conn(0x7f18dc0ce420 msgr2=0x7f18dc0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:20.264 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:20.246 7f18e4123700 1 -- 172.21.2.102:0/1351408118 shutdown_connections 2021-02-10T11:34:20.265 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:20.246 7f18e4123700 1 -- 172.21.2.102:0/1351408118 wait complete. 2021-02-10T11:34:20.266 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:20.261 7f18e4123700 1 Processor -- start 2021-02-10T11:34:20.267 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:20.261 7f18e4123700 1 -- start start 2021-02-10T11:34:20.267 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:20.261 7f18e4123700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f18dc0a58c0 0x7f18dc0a4d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:20.267 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:20.261 7f18e4123700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f18dc0b1ba0 0x7f18dc0b1fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:20.267 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:20.261 7f18e4123700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f18dc0b25c0 0x7f18dc0b3270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:20.267 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:20.261 7f18e4123700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18dc0b2a40 con 0x7f18dc0b1ba0 2021-02-10T11:34:20.267 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:20.261 7f18e4123700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18dc0a5340 con 0x7f18dc0b25c0 2021-02-10T11:34:20.267 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:20.261 7f18e4123700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18dc0a5560 con 0x7f18dc0a58c0 2021-02-10T11:34:20.267 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:20.262 7f18e173e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f18dc0b1ba0 0x7f18dc0b1fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:20.267 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:20.262 7f18e173e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f18dc0b1ba0 0x7f18dc0b1fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:35994/0 (socket says 172.21.2.102:35994) 2021-02-10T11:34:20.267 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:20.262 7f18e173e700 1 -- 172.21.2.102:0/3999302308 learned_addr learned my addr 172.21.2.102:0/3999302308 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:34:20.267 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:20.262 7f18e2740700 1 --2- 172.21.2.102:0/3999302308 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f18dc0b25c0 0x7f18dc0b3270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:20.267 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:20.262 7f18e1f3f700 1 --2- 172.21.2.102:0/3999302308 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f18dc0a58c0 0x7f18dc0a4d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:20.267 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:20.262 7f18e173e700 1 -- 172.21.2.102:0/3999302308 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f18dc0a58c0 msgr2=0x7f18dc0a4d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:20.268 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:20.262 7f18e173e700 1 --2- 172.21.2.102:0/3999302308 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f18dc0a58c0 0x7f18dc0a4d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:20.268 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:20.262 7f18e173e700 1 -- 172.21.2.102:0/3999302308 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f18dc0b25c0 msgr2=0x7f18dc0b3270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:20.268 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:20.262 7f18e173e700 1 --2- 172.21.2.102:0/3999302308 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f18dc0b25c0 0x7f18dc0b3270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:20.268 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:20.262 7f18e173e700 1 -- 172.21.2.102:0/3999302308 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18dc0b3890 con 0x7f18dc0b1ba0 2021-02-10T11:34:20.268 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:20.262 7f18e173e700 1 --2- 172.21.2.102:0/3999302308 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f18dc0b1ba0 0x7f18dc0b1fa0 secure :-1 s=READY pgs=2528 cs=0 l=1 rev1=1 rx=0x7f18d0007650 tx=0x7f18d000a900).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:20.268 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:20.263 7f18ce7fc700 1 -- 172.21.2.102:0/3999302308 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f18d000acf0 con 0x7f18dc0b1ba0 2021-02-10T11:34:20.268 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:20.263 7f18ce7fc700 1 -- 172.21.2.102:0/3999302308 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18d0004140 con 0x7f18dc0b1ba0 2021-02-10T11:34:20.268 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:20.263 7f18ce7fc700 1 -- 172.21.2.102:0/3999302308 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f18d00045a0 con 0x7f18dc0b1ba0 2021-02-10T11:34:20.268 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:20.263 7f18e4123700 1 -- 172.21.2.102:0/3999302308 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18dc0b3890 con 0x7f18dc0b1ba0 2021-02-10T11:34:20.268 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:20.263 7f18e4123700 1 -- 172.21.2.102:0/3999302308 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18dc0b3890 con 0x7f18dc0b1ba0 2021-02-10T11:34:20.268 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:20.263 7f18ce7fc700 1 -- 172.21.2.102:0/3999302308 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f18d0000dd0 con 0x7f18dc0b1ba0 2021-02-10T11:34:20.268 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:20.263 7f18ce7fc700 1 --2- 172.21.2.102:0/3999302308 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f18c40186c0 0x7f18c401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:20.268 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:20.263 7f18ce7fc700 1 -- 172.21.2.102:0/3999302308 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 4753+0+0 (secure 0 0 0) 0x7f18d0032480 con 0x7f18dc0b1ba0 2021-02-10T11:34:20.269 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:20.264 7f18e1f3f700 1 --2- 172.21.2.102:0/3999302308 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f18c40186c0 0x7f18c401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:20.269 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:20.264 7f18e1f3f700 1 --2- 172.21.2.102:0/3999302308 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f18c40186c0 0x7f18c401abc0 crc :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:20.295 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:20.294 7f18e4123700 1 -- 172.21.2.102:0/3999302308 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18dc082980 con 0x7f18dc0b1ba0 2021-02-10T11:34:20.298 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:20.296 7f18ce7fc700 1 -- 172.21.2.102:0/3999302308 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f18d0000940 con 0x7f18dc0b1ba0 2021-02-10T11:34:20.426 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:20.424 7f18e4123700 1 -- 172.21.2.102:0/3999302308 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f18dc0b3890 con 0x7f18dc0b1ba0 2021-02-10T11:34:20.426 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:20.424 7f18ce7fc700 1 -- 172.21.2.102:0/3999302308 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v132) v1 ==== 92+0+10 (secure 0 0 0) 0x7f18d0000dd0 con 0x7f18dc0b1ba0 2021-02-10T11:34:20.427 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:34:20.428 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:20.426 7f18e4123700 1 -- 172.21.2.102:0/3999302308 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f18c40186c0 msgr2=0x7f18c401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:20.428 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:20.426 7f18e4123700 1 --2- 172.21.2.102:0/3999302308 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f18c40186c0 0x7f18c401abc0 crc :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:20.429 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:20.426 7f18e4123700 1 -- 172.21.2.102:0/3999302308 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f18dc0b1ba0 msgr2=0x7f18dc0b1fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:20.429 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:20.426 7f18e4123700 1 --2- 172.21.2.102:0/3999302308 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f18dc0b1ba0 0x7f18dc0b1fa0 secure :-1 s=READY pgs=2528 cs=0 l=1 rev1=1 rx=0x7f18d0007650 tx=0x7f18d000a900).stop 2021-02-10T11:34:20.429 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:20.427 7f18e4123700 1 -- 172.21.2.102:0/3999302308 shutdown_connections 2021-02-10T11:34:20.429 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:20.427 7f18e4123700 1 --2- 172.21.2.102:0/3999302308 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f18c40186c0 0x7f18c401abc0 unknown :-1 s=CLOSED pgs=1116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:20.429 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:20.427 7f18e4123700 1 --2- 172.21.2.102:0/3999302308 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f18dc0b25c0 0x7f18dc0b3270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:20.429 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:20.427 7f18e4123700 1 --2- 172.21.2.102:0/3999302308 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f18dc0b1ba0 0x7f18dc0b1fa0 unknown :-1 s=CLOSED pgs=2528 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:20.429 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:20.427 7f18e4123700 1 --2- 172.21.2.102:0/3999302308 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f18dc0a58c0 0x7f18dc0a4d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:20.429 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:20.427 7f18e4123700 1 -- 172.21.2.102:0/3999302308 >> 172.21.2.102:0/3999302308 conn(0x7f18dc0b5dc0 msgr2=0x7f18dc095960 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:20.429 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:20.427 7f18e4123700 1 -- 172.21.2.102:0/3999302308 shutdown_connections 2021-02-10T11:34:20.429 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:20.427 7f18e4123700 1 -- 172.21.2.102:0/3999302308 wait complete. 2021-02-10T11:34:20.437 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --format=json-pretty osd lspools 2021-02-10T11:34:20.864 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:20.846 7f5f9f7cd700 1 Processor -- start 2021-02-10T11:34:20.865 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:20.846 7f5f9f7cd700 1 -- start start 2021-02-10T11:34:20.865 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:20.847 7f5f9f7cd700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f5f980d0340 0x7f5f980ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:20.865 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:20.847 7f5f9f7cd700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f5f980d1c70 0x7f5f980efc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:20.865 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:20.847 7f5f9f7cd700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f980f0b10 con 0x7f5f980d1870 2021-02-10T11:34:20.865 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:20.847 7f5f9f7cd700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f980e5ed0 con 0x7f5f980d0340 2021-02-10T11:34:20.865 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:20.847 7f5f9f7cd700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f981b0be0 con 0x7f5f980d1c70 2021-02-10T11:34:20.865 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:20.847 7f5f9cde8700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f5f980d0340 0x7f5f980ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:20.865 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:20.847 7f5f9d5e9700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f5f980d1870 0x7f5f981b03f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:56458/0 (socket says 172.21.2.102:56458) 2021-02-10T11:34:20.865 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:20.847 7f5f9d5e9700 1 -- 172.21.2.102:0/723594949 learned_addr learned my addr 172.21.2.102:0/723594949 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:34:20.865 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:20.848 7f5f9cde8700 1 -- 172.21.2.102:0/723594949 >> v2:172.21.2.113:3300/0 conn(0x7f5f980d1c70 msgr2=0x7f5f980efc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:34:20.865 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:20.848 7f5f9cde8700 1 --2- 172.21.2.102:0/723594949 >> v2:172.21.2.113:3300/0 conn(0x7f5f980d1c70 0x7f5f980efc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:20.865 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:20.848 7f5f9cde8700 1 -- 172.21.2.102:0/723594949 >> v1:172.21.2.102:6789/0 conn(0x7f5f980d1870 legacy=0x7f5f981b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:20.866 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:20.848 7f5f9cde8700 1 -- 172.21.2.102:0/723594949 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f980f20a0 con 0x7f5f980d0340 2021-02-10T11:34:20.866 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:20.848 7f5f9cde8700 1 --2- 172.21.2.102:0/723594949 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5f980d0340 0x7f5f980ef640 secure :-1 s=READY pgs=2530 cs=0 l=1 rev1=1 rx=0x7f5f94006650 tx=0x7f5f94009a80).ready entity=mon.0 client_cookie=a054fca9d5de7725 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:20.866 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:20.848 7f5f8bfff700 1 -- 172.21.2.102:0/723594949 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5f940009b0 con 0x7f5f980d0340 2021-02-10T11:34:20.866 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:20.848 7f5f8bfff700 1 -- 172.21.2.102:0/723594949 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f9400c0c0 con 0x7f5f980d0340 2021-02-10T11:34:20.866 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:20.848 7f5f8bfff700 1 -- 172.21.2.102:0/723594949 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5f9400c520 con 0x7f5f980d0340 2021-02-10T11:34:20.866 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:20.848 7f5f9f7cd700 1 -- 172.21.2.102:0/723594949 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5f980d0340 msgr2=0x7f5f980ef640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:20.866 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:20.848 7f5f9f7cd700 1 --2- 172.21.2.102:0/723594949 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5f980d0340 0x7f5f980ef640 secure :-1 s=READY pgs=2530 cs=0 l=1 rev1=1 rx=0x7f5f94006650 tx=0x7f5f94009a80).stop 2021-02-10T11:34:20.866 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:20.849 7f5f9f7cd700 1 -- 172.21.2.102:0/723594949 shutdown_connections 2021-02-10T11:34:20.866 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:20.849 7f5f9f7cd700 1 --2- 172.21.2.102:0/723594949 >> v2:172.21.2.113:3300/0 conn(0x7f5f980d1c70 0x7f5f980efc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:20.866 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:20.849 7f5f9f7cd700 1 --2- 172.21.2.102:0/723594949 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5f980d0340 0x7f5f980ef640 unknown :-1 s=CLOSED pgs=2530 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:20.866 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:20.849 7f5f9f7cd700 1 -- 172.21.2.102:0/723594949 >> 172.21.2.102:0/723594949 conn(0x7f5f980ce450 msgr2=0x7f5f980d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:20.866 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:20.849 7f5f9f7cd700 1 -- 172.21.2.102:0/723594949 shutdown_connections 2021-02-10T11:34:20.867 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:20.849 7f5f9f7cd700 1 -- 172.21.2.102:0/723594949 wait complete. 2021-02-10T11:34:20.868 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:20.863 7f5f9f7cd700 1 Processor -- start 2021-02-10T11:34:20.868 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:20.863 7f5f9f7cd700 1 -- start start 2021-02-10T11:34:20.868 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:20.864 7f5f9f7cd700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5f9818f0a0 0x7f5f98196ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:20.868 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:20.864 7f5f9f7cd700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5f981970d0 0x7f5f981974d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:20.868 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:20.864 7f5f9f7cd700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5f98094650 0x7f5f98094a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:20.868 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:20.864 7f5f9f7cd700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f98095bd0 con 0x7f5f981970d0 2021-02-10T11:34:20.869 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:20.864 7f5f9f7cd700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f980ad7c0 con 0x7f5f9818f0a0 2021-02-10T11:34:20.869 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:20.864 7f5f9f7cd700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f9818ec70 con 0x7f5f98094650 2021-02-10T11:34:20.869 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:20.864 7f5f9cde8700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5f981970d0 0x7f5f981974d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:20.869 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:20.864 7f5f9cde8700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5f981970d0 0x7f5f981974d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:36012/0 (socket says 172.21.2.102:36012) 2021-02-10T11:34:20.869 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:20.864 7f5f9cde8700 1 -- 172.21.2.102:0/1717593984 learned_addr learned my addr 172.21.2.102:0/1717593984 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:34:20.869 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:20.864 7f5f9cde8700 1 -- 172.21.2.102:0/1717593984 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5f98094650 msgr2=0x7f5f98094a50 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:34:20.869 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:20.864 7f5f9cde8700 1 --2- 172.21.2.102:0/1717593984 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5f98094650 0x7f5f98094a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:20.869 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:20.864 7f5f9cde8700 1 -- 172.21.2.102:0/1717593984 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5f9818f0a0 msgr2=0x7f5f98196ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:20.869 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:20.864 7f5f9cde8700 1 --2- 172.21.2.102:0/1717593984 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5f9818f0a0 0x7f5f98196ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:20.869 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:20.864 7f5f9cde8700 1 -- 172.21.2.102:0/1717593984 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f98095070 con 0x7f5f981970d0 2021-02-10T11:34:20.869 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:20.865 7f5f9cde8700 1 --2- 172.21.2.102:0/1717593984 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5f981970d0 0x7f5f981974d0 secure :-1 s=READY pgs=2531 cs=0 l=1 rev1=1 rx=0x7f5f94009630 tx=0x7f5f9400b900).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:20.869 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:20.865 7f5f89ffb700 1 -- 172.21.2.102:0/1717593984 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5f940093a0 con 0x7f5f981970d0 2021-02-10T11:34:20.869 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:20.865 7f5f89ffb700 1 -- 172.21.2.102:0/1717593984 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f9400bcf0 con 0x7f5f981970d0 2021-02-10T11:34:20.870 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:20.865 7f5f89ffb700 1 -- 172.21.2.102:0/1717593984 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5f94005560 con 0x7f5f981970d0 2021-02-10T11:34:20.870 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:20.865 7f5f9f7cd700 1 -- 172.21.2.102:0/1717593984 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f98095070 con 0x7f5f981970d0 2021-02-10T11:34:20.870 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:20.865 7f5f9f7cd700 1 -- 172.21.2.102:0/1717593984 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f98095070 con 0x7f5f981970d0 2021-02-10T11:34:20.870 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:20.866 7f5f89ffb700 1 -- 172.21.2.102:0/1717593984 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f5f94008240 con 0x7f5f981970d0 2021-02-10T11:34:20.870 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:20.866 7f5f89ffb700 1 --2- 172.21.2.102:0/1717593984 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5f780186c0 0x7f5f7801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:20.870 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:20.866 7f5f89ffb700 1 -- 172.21.2.102:0/1717593984 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 4753+0+0 (secure 0 0 0) 0x7f5f940321e0 con 0x7f5f981970d0 2021-02-10T11:34:20.870 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:20.866 7f5f9d5e9700 1 --2- 172.21.2.102:0/1717593984 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5f780186c0 0x7f5f7801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:20.870 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:20.866 7f5f9d5e9700 1 --2- 172.21.2.102:0/1717593984 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5f780186c0 0x7f5f7801abc0 crc :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:20.899 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:20.897 7f5f9f7cd700 1 -- 172.21.2.102:0/1717593984 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f980ad790 con 0x7f5f981970d0 2021-02-10T11:34:20.901 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:20.899 7f5f89ffb700 1 -- 172.21.2.102:0/1717593984 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f5f9402cd60 con 0x7f5f981970d0 2021-02-10T11:34:21.023 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:21.021 7f5f9f7cd700 1 -- 172.21.2.102:0/1717593984 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f5f980ad790 con 0x7f5f981970d0 2021-02-10T11:34:21.023 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:21.022 7f5f89ffb700 1 -- 172.21.2.102:0/1717593984 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v132) v1 ==== 84+0+64 (secure 0 0 0) 0x7f5f9402cd60 con 0x7f5f981970d0 2021-02-10T11:34:21.024 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:34:21.024 INFO:teuthology.orchestra.run.gibba002.stdout:[ 2021-02-10T11:34:21.024 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:21.024 INFO:teuthology.orchestra.run.gibba002.stdout: "poolnum": 1, 2021-02-10T11:34:21.024 INFO:teuthology.orchestra.run.gibba002.stdout: "poolname": "rbd" 2021-02-10T11:34:21.024 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:34:21.024 INFO:teuthology.orchestra.run.gibba002.stdout:] 2021-02-10T11:34:21.025 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:21.023 7f5f9f7cd700 1 -- 172.21.2.102:0/1717593984 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5f780186c0 msgr2=0x7f5f7801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:21.025 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:21.023 7f5f9f7cd700 1 --2- 172.21.2.102:0/1717593984 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5f780186c0 0x7f5f7801abc0 crc :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:21.025 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:21.023 7f5f9f7cd700 1 -- 172.21.2.102:0/1717593984 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5f981970d0 msgr2=0x7f5f981974d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:21.025 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:21.023 7f5f9f7cd700 1 --2- 172.21.2.102:0/1717593984 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5f981970d0 0x7f5f981974d0 secure :-1 s=READY pgs=2531 cs=0 l=1 rev1=1 rx=0x7f5f94009630 tx=0x7f5f9400b900).stop 2021-02-10T11:34:21.026 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:21.024 7f5f9f7cd700 1 -- 172.21.2.102:0/1717593984 shutdown_connections 2021-02-10T11:34:21.026 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:21.024 7f5f9f7cd700 1 --2- 172.21.2.102:0/1717593984 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5f98094650 0x7f5f98094a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:21.026 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:21.024 7f5f9f7cd700 1 --2- 172.21.2.102:0/1717593984 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5f981970d0 0x7f5f981974d0 unknown :-1 s=CLOSED pgs=2531 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:21.026 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:21.024 7f5f9f7cd700 1 --2- 172.21.2.102:0/1717593984 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5f780186c0 0x7f5f7801abc0 unknown :-1 s=CLOSED pgs=1117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:21.027 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:21.024 7f5f9f7cd700 1 --2- 172.21.2.102:0/1717593984 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5f9818f0a0 0x7f5f98196ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:21.027 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:21.024 7f5f9f7cd700 1 -- 172.21.2.102:0/1717593984 >> 172.21.2.102:0/1717593984 conn(0x7f5f981a38c0 msgr2=0x7f5f981a2f00 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:21.027 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:21.024 7f5f9f7cd700 1 -- 172.21.2.102:0/1717593984 shutdown_connections 2021-02-10T11:34:21.027 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:21.024 7f5f9f7cd700 1 -- 172.21.2.102:0/1717593984 wait complete. 2021-02-10T11:34:21.034 INFO:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2021-02-10T11:34:21.034 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool create cephfs_metadata 8 2021-02-10T11:34:21.467 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:21.449 7f82c5c56700 1 Processor -- start 2021-02-10T11:34:21.468 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:21.449 7f82c5c56700 1 -- start start 2021-02-10T11:34:21.468 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:21.449 7f82c5c56700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f82c00d1870 0x7f82c01b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:21.468 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:21.449 7f82c5c56700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82c0109a60 con 0x7f82c00cff70 2021-02-10T11:34:21.468 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:21.449 7f82c5c56700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82c0088070 con 0x7f82c01b0ae0 2021-02-10T11:34:21.468 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:21.449 7f82c5c56700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82c00e5ef0 con 0x7f82c00d1870 2021-02-10T11:34:21.468 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:21.449 7f82bffff700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f82c00cff70 0x7f82c0109130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:56472/0 (socket says 172.21.2.102:56472) 2021-02-10T11:34:21.468 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:21.449 7f82bffff700 1 -- 172.21.2.102:0/2931814615 learned_addr learned my addr 172.21.2.102:0/2931814615 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:34:21.469 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:21.449 7f82bf7fe700 1 --2- 172.21.2.102:0/2931814615 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f82c00d1870 0x7f82c01b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:21.469 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:21.450 7f82beffd700 1 -- 172.21.2.102:0/2931814615 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1337377503 0 0) 0x7f82b8001180 con 0x7f82c00cff70 2021-02-10T11:34:21.469 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:21.450 7f82bf7fe700 1 -- 172.21.2.102:0/2931814615 >> v1:172.21.2.113:6789/0 conn(0x7f82c01b0ae0 legacy=0x7f82c0108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:21.469 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:21.450 7f82bf7fe700 1 -- 172.21.2.102:0/2931814615 >> v1:172.21.2.102:6789/0 conn(0x7f82c00cff70 legacy=0x7f82c0109130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:21.469 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:21.450 7f82bf7fe700 1 -- 172.21.2.102:0/2931814615 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82c00d1c70 con 0x7f82c00d1870 2021-02-10T11:34:21.469 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:21.450 7f82bf7fe700 1 --2- 172.21.2.102:0/2931814615 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f82c00d1870 0x7f82c01b04c0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 rx=0x7f82b00069d0 tx=0x7f82b0009dc0).ready entity=mon.2 client_cookie=e7f40a2c6dc3766d server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:21.470 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:21.451 7f82beffd700 1 -- 172.21.2.102:0/2931814615 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f82b000b510 con 0x7f82c00d1870 2021-02-10T11:34:21.470 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:21.451 7f82beffd700 1 -- 172.21.2.102:0/2931814615 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82b0006ac0 con 0x7f82c00d1870 2021-02-10T11:34:21.470 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:21.451 7f82beffd700 1 -- 172.21.2.102:0/2931814615 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f82b000ba80 con 0x7f82c00d1870 2021-02-10T11:34:21.470 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:21.451 7f82c5c56700 1 -- 172.21.2.102:0/2931814615 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f82c00d1870 msgr2=0x7f82c01b04c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:21.470 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:21.451 7f82c5c56700 1 --2- 172.21.2.102:0/2931814615 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f82c00d1870 0x7f82c01b04c0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 rx=0x7f82b00069d0 tx=0x7f82b0009dc0).stop 2021-02-10T11:34:21.470 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:21.451 7f82c5c56700 1 -- 172.21.2.102:0/2931814615 shutdown_connections 2021-02-10T11:34:21.470 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:21.451 7f82c5c56700 1 --2- 172.21.2.102:0/2931814615 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f82c00d1870 0x7f82c01b04c0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:21.470 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:21.451 7f82c5c56700 1 -- 172.21.2.102:0/2931814615 >> 172.21.2.102:0/2931814615 conn(0x7f82c00ce420 msgr2=0x7f82c00d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:21.470 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:21.451 7f82c5c56700 1 -- 172.21.2.102:0/2931814615 shutdown_connections 2021-02-10T11:34:21.470 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:21.451 7f82c5c56700 1 -- 172.21.2.102:0/2931814615 wait complete. 2021-02-10T11:34:21.473 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:21.466 7f82c5c56700 1 Processor -- start 2021-02-10T11:34:21.473 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:21.466 7f82c5c56700 1 -- start start 2021-02-10T11:34:21.473 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:21.467 7f82c5c56700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f82c00c3820 0x7f82c0084cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:21.473 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:21.467 7f82c5c56700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f82c00a0290 0x7f82c00852d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:21.473 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:21.467 7f82c5c56700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f82c0086270 0x7f82c0086670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:21.473 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:21.467 7f82c5c56700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82c00c0700 con 0x7f82c0086270 2021-02-10T11:34:21.473 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:21.467 7f82c5c56700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82c00858f0 con 0x7f82c00a0290 2021-02-10T11:34:21.473 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:21.467 7f82c5c56700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82c0085b10 con 0x7f82c00c3820 2021-02-10T11:34:21.473 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:21.467 7f82bffff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f82c0086270 0x7f82c0086670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:21.474 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:21.467 7f82bffff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f82c0086270 0x7f82c0086670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:36022/0 (socket says 172.21.2.102:36022) 2021-02-10T11:34:21.474 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:21.467 7f82bffff700 1 -- 172.21.2.102:0/964440201 learned_addr learned my addr 172.21.2.102:0/964440201 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:34:21.474 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:21.467 7f82b6ffd700 1 --2- 172.21.2.102:0/964440201 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f82c00a0290 0x7f82c00852d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:21.474 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:21.467 7f82bffff700 1 -- 172.21.2.102:0/964440201 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f82c00c3820 msgr2=0x7f82c0084cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:21.474 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:21.467 7f82bffff700 1 --2- 172.21.2.102:0/964440201 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f82c00c3820 0x7f82c0084cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:21.474 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:21.467 7f82bffff700 1 -- 172.21.2.102:0/964440201 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f82c00a0290 msgr2=0x7f82c00852d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:21.474 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:21.467 7f82bffff700 1 --2- 172.21.2.102:0/964440201 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f82c00a0290 0x7f82c00852d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:21.474 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:21.467 7f82bffff700 1 -- 172.21.2.102:0/964440201 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82c0086c90 con 0x7f82c0086270 2021-02-10T11:34:21.474 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:21.467 7f82b6ffd700 1 --2- 172.21.2.102:0/964440201 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f82c00a0290 0x7f82c00852d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:34:21.474 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:21.467 7f82bffff700 1 --2- 172.21.2.102:0/964440201 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f82c0086270 0x7f82c0086670 secure :-1 s=READY pgs=2533 cs=0 l=1 rev1=1 rx=0x7f82b800a570 tx=0x7f82b800a910).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:21.474 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:21.468 7f82bcff9700 1 -- 172.21.2.102:0/964440201 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f82b80009b0 con 0x7f82c0086270 2021-02-10T11:34:21.474 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:21.468 7f82bcff9700 1 -- 172.21.2.102:0/964440201 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82b800ad00 con 0x7f82c0086270 2021-02-10T11:34:21.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:21.468 7f82bcff9700 1 -- 172.21.2.102:0/964440201 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f82b8004590 con 0x7f82c0086270 2021-02-10T11:34:21.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:21.468 7f82c5c56700 1 -- 172.21.2.102:0/964440201 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82c0086c90 con 0x7f82c0086270 2021-02-10T11:34:21.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:21.468 7f82c5c56700 1 -- 172.21.2.102:0/964440201 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82c0086c90 con 0x7f82c0086270 2021-02-10T11:34:21.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:21.468 7f82bcff9700 1 -- 172.21.2.102:0/964440201 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f82b80009b0 con 0x7f82c0086270 2021-02-10T11:34:21.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:21.468 7f82bcff9700 1 --2- 172.21.2.102:0/964440201 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f82a40186c0 0x7f82a401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:21.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:21.468 7f82bcff9700 1 -- 172.21.2.102:0/964440201 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 4753+0+0 (secure 0 0 0) 0x7f82b8032460 con 0x7f82c0086270 2021-02-10T11:34:21.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:21.468 7f82bf7fe700 1 --2- 172.21.2.102:0/964440201 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f82a40186c0 0x7f82a401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:21.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:21.469 7f82bf7fe700 1 --2- 172.21.2.102:0/964440201 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f82a40186c0 0x7f82a401abc0 crc :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:21.501 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:21.499 7f82c5c56700 1 -- 172.21.2.102:0/964440201 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82c0082a70 con 0x7f82c0086270 2021-02-10T11:34:21.503 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:21.502 7f82bcff9700 1 -- 172.21.2.102:0/964440201 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f82b8000f70 con 0x7f82c0086270 2021-02-10T11:34:21.632 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:21.630 7f82c5c56700 1 -- 172.21.2.102:0/964440201 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"} v 0) v1 -- 0x7f82c0082a70 con 0x7f82c0086270 2021-02-10T11:34:22.042 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:22.040 7f82bcff9700 1 -- 172.21.2.102:0/964440201 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"}]=0 pool 'cephfs_metadata' created v133) v1 ==== 133+0+0 (secure 0 0 0) 0x7f82b80009b0 con 0x7f82c0086270 2021-02-10T11:34:22.047 INFO:teuthology.orchestra.run.gibba002.stderr:pool 'cephfs_metadata' created 2021-02-10T11:34:22.049 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:22.047 7f82c5c56700 1 -- 172.21.2.102:0/964440201 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f82a40186c0 msgr2=0x7f82a401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:22.049 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:22.047 7f82c5c56700 1 --2- 172.21.2.102:0/964440201 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f82a40186c0 0x7f82a401abc0 crc :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:22.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:22.047 7f82c5c56700 1 -- 172.21.2.102:0/964440201 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f82c0086270 msgr2=0x7f82c0086670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:22.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:22.047 7f82c5c56700 1 --2- 172.21.2.102:0/964440201 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f82c0086270 0x7f82c0086670 secure :-1 s=READY pgs=2533 cs=0 l=1 rev1=1 rx=0x7f82b800a570 tx=0x7f82b800a910).stop 2021-02-10T11:34:22.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:22.047 7f82c5c56700 1 -- 172.21.2.102:0/964440201 shutdown_connections 2021-02-10T11:34:22.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:22.047 7f82c5c56700 1 --2- 172.21.2.102:0/964440201 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f82c0086270 0x7f82c0086670 unknown :-1 s=CLOSED pgs=2533 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:22.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:22.047 7f82c5c56700 1 --2- 172.21.2.102:0/964440201 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f82a40186c0 0x7f82a401abc0 unknown :-1 s=CLOSED pgs=1118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:22.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:22.047 7f82c5c56700 1 --2- 172.21.2.102:0/964440201 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f82c00a0290 0x7f82c00852d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:22.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:22.047 7f82c5c56700 1 --2- 172.21.2.102:0/964440201 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f82c00c3820 0x7f82c0084cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:22.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:22.047 7f82c5c56700 1 -- 172.21.2.102:0/964440201 >> 172.21.2.102:0/964440201 conn(0x7f82c00872c0 msgr2=0x7f82c0095700 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:22.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:22.048 7f82c5c56700 1 -- 172.21.2.102:0/964440201 shutdown_connections 2021-02-10T11:34:22.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:22.048 7f82c5c56700 1 -- 172.21.2.102:0/964440201 wait complete. 2021-02-10T11:34:22.058 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool create cephfs_data 8 2021-02-10T11:34:22.489 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:22.470 7f865fedd700 1 Processor -- start 2021-02-10T11:34:22.489 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:22.470 7f865fedd700 1 -- start start 2021-02-10T11:34:22.489 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:22.471 7f865fedd700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f86580d1870 0x7f86581b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:22.489 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:22.471 7f865fedd700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f86581b0ae0 0x7f8658108940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:22.489 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:22.471 7f865fedd700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8658109a60 con 0x7f86580d1c70 2021-02-10T11:34:22.489 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:22.471 7f865fedd700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86580e5ef0 con 0x7f86581b0ae0 2021-02-10T11:34:22.489 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:22.471 7f865fedd700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8658109750 con 0x7f86580d1870 2021-02-10T11:34:22.489 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:22.471 7f865d4f8700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f86581b0ae0 0x7f8658108940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:22.490 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:22.471 7f865d4f8700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f86581b0ae0 0x7f8658108940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:36028/0 (socket says 172.21.2.102:36028) 2021-02-10T11:34:22.490 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:22.471 7f865d4f8700 1 -- 172.21.2.102:0/203221658 learned_addr learned my addr 172.21.2.102:0/203221658 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:34:22.490 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:22.471 7f865e4fa700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f86580d1c70 0x7f8658108f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:56484/0 (socket says 172.21.2.102:56484) 2021-02-10T11:34:22.490 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:22.471 7f865dcf9700 1 --2- 172.21.2.102:0/203221658 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f86580d1870 0x7f86581b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:22.490 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:22.472 7f865d4f8700 1 -- 172.21.2.102:0/203221658 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f86580d1870 msgr2=0x7f86581b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:34:22.490 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:22.472 7f865d4f8700 1 --2- 172.21.2.102:0/203221658 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f86580d1870 0x7f86581b04c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:22.490 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:22.472 7f865d4f8700 1 -- 172.21.2.102:0/203221658 >> v1:172.21.2.102:6789/0 conn(0x7f86580d1c70 legacy=0x7f8658108f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:22.490 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:22.472 7f865d4f8700 1 -- 172.21.2.102:0/203221658 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8658088070 con 0x7f86581b0ae0 2021-02-10T11:34:22.490 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:22.472 7f865dcf9700 1 --2- 172.21.2.102:0/203221658 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f86580d1870 0x7f86581b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:34:22.490 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:22.472 7f865d4f8700 1 --2- 172.21.2.102:0/203221658 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f86581b0ae0 0x7f8658108940 secure :-1 s=READY pgs=2535 cs=0 l=1 rev1=1 rx=0x7f8648006700 tx=0x7f8648009b80).ready entity=mon.0 client_cookie=373d078f328359f1 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:22.490 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:22.472 7f865ccf7700 1 -- 172.21.2.102:0/203221658 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8648006340 con 0x7f86581b0ae0 2021-02-10T11:34:22.490 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:22.472 7f865ccf7700 1 -- 172.21.2.102:0/203221658 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f864800c0c0 con 0x7f86581b0ae0 2021-02-10T11:34:22.491 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:22.472 7f865ccf7700 1 -- 172.21.2.102:0/203221658 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f864800c510 con 0x7f86581b0ae0 2021-02-10T11:34:22.491 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:22.472 7f865fedd700 1 -- 172.21.2.102:0/203221658 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f86581b0ae0 msgr2=0x7f8658108940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:22.491 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:22.472 7f865fedd700 1 --2- 172.21.2.102:0/203221658 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f86581b0ae0 0x7f8658108940 secure :-1 s=READY pgs=2535 cs=0 l=1 rev1=1 rx=0x7f8648006700 tx=0x7f8648009b80).stop 2021-02-10T11:34:22.491 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:22.473 7f865fedd700 1 -- 172.21.2.102:0/203221658 shutdown_connections 2021-02-10T11:34:22.491 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:22.473 7f865fedd700 1 --2- 172.21.2.102:0/203221658 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f86581b0ae0 0x7f8658108940 unknown :-1 s=CLOSED pgs=2535 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:22.491 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:22.473 7f865fedd700 1 --2- 172.21.2.102:0/203221658 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f86580d1870 0x7f86581b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:22.491 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:22.473 7f865fedd700 1 -- 172.21.2.102:0/203221658 >> 172.21.2.102:0/203221658 conn(0x7f86580ce420 msgr2=0x7f86580d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:22.491 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:22.473 7f865fedd700 1 -- 172.21.2.102:0/203221658 shutdown_connections 2021-02-10T11:34:22.491 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:22.473 7f865fedd700 1 -- 172.21.2.102:0/203221658 wait complete. 2021-02-10T11:34:22.492 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:22.487 7f865fedd700 1 Processor -- start 2021-02-10T11:34:22.493 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:22.488 7f865fedd700 1 -- start start 2021-02-10T11:34:22.493 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:22.488 7f865fedd700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f86580b1ba0 0x7f86581a78d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:22.493 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:22.488 7f865fedd700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f86581a7ef0 0x7f86580a6a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:22.493 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:22.488 7f865fedd700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f86580a7060 0x7f86580b2920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:22.493 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:22.488 7f865fedd700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8658190c20 con 0x7f86581a7ef0 2021-02-10T11:34:22.493 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:22.488 7f865fedd700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8658190710 con 0x7f86580b1ba0 2021-02-10T11:34:22.493 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:22.488 7f865fedd700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86580b1fa0 con 0x7f86580a7060 2021-02-10T11:34:22.493 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:22.488 7f865d4f8700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f86581a7ef0 0x7f86580a6a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:22.493 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:22.488 7f865d4f8700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f86581a7ef0 0x7f86580a6a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:36036/0 (socket says 172.21.2.102:36036) 2021-02-10T11:34:22.493 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:22.488 7f865d4f8700 1 -- 172.21.2.102:0/535681610 learned_addr learned my addr 172.21.2.102:0/535681610 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:34:22.493 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:22.488 7f865d4f8700 1 -- 172.21.2.102:0/535681610 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f86580a7060 msgr2=0x7f86580b2920 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:34:22.494 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:22.488 7f865d4f8700 1 --2- 172.21.2.102:0/535681610 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f86580a7060 0x7f86580b2920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:22.494 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:22.488 7f865d4f8700 1 -- 172.21.2.102:0/535681610 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f86580b1ba0 msgr2=0x7f86581a78d0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:34:22.494 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:22.488 7f865d4f8700 1 --2- 172.21.2.102:0/535681610 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f86580b1ba0 0x7f86581a78d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:22.494 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:22.488 7f865d4f8700 1 -- 172.21.2.102:0/535681610 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86580a7460 con 0x7f86581a7ef0 2021-02-10T11:34:22.494 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:22.489 7f865d4f8700 1 --2- 172.21.2.102:0/535681610 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f86581a7ef0 0x7f86580a6a40 secure :-1 s=READY pgs=2536 cs=0 l=1 rev1=1 rx=0x7f8648009270 tx=0x7f864800b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:22.494 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:22.489 7f864e7fc700 1 -- 172.21.2.102:0/535681610 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f864800bd50 con 0x7f86581a7ef0 2021-02-10T11:34:22.494 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:22.489 7f864e7fc700 1 -- 172.21.2.102:0/535681610 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86480052e0 con 0x7f86581a7ef0 2021-02-10T11:34:22.494 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:22.489 7f864e7fc700 1 -- 172.21.2.102:0/535681610 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8648005800 con 0x7f86581a7ef0 2021-02-10T11:34:22.494 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:22.489 7f865fedd700 1 -- 172.21.2.102:0/535681610 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86580a7460 con 0x7f86581a7ef0 2021-02-10T11:34:22.494 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:22.489 7f865fedd700 1 -- 172.21.2.102:0/535681610 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86580a7460 con 0x7f86581a7ef0 2021-02-10T11:34:22.494 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:22.490 7f864e7fc700 1 -- 172.21.2.102:0/535681610 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f86480054c0 con 0x7f86581a7ef0 2021-02-10T11:34:22.494 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:22.490 7f864e7fc700 1 --2- 172.21.2.102:0/535681610 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f86400186c0 0x7f864001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:22.495 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:22.490 7f864e7fc700 1 -- 172.21.2.102:0/535681610 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5107+0+0 (secure 0 0 0) 0x7f86480323a0 con 0x7f86581a7ef0 2021-02-10T11:34:22.495 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:22.490 7f865dcf9700 1 --2- 172.21.2.102:0/535681610 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f86400186c0 0x7f864001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:22.495 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:22.490 7f865dcf9700 1 --2- 172.21.2.102:0/535681610 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f86400186c0 0x7f864001abc0 crc :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:22.522 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:22.521 7f865fedd700 1 -- 172.21.2.102:0/535681610 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8658082980 con 0x7f86581a7ef0 2021-02-10T11:34:22.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:22.523 7f864e7fc700 1 -- 172.21.2.102:0/535681610 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f8648014540 con 0x7f86581a7ef0 2021-02-10T11:34:22.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:22.656 7f865fedd700 1 -- 172.21.2.102:0/535681610 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_data"} v 0) v1 -- 0x7f8658109a60 con 0x7f86581a7ef0 2021-02-10T11:34:23.048 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:23.047 7f864e7fc700 1 -- 172.21.2.102:0/535681610 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_data"}]=0 pool 'cephfs_data' created v135) v1 ==== 125+0+0 (secure 0 0 0) 0x7f8648014540 con 0x7f86581a7ef0 2021-02-10T11:34:23.074 INFO:teuthology.orchestra.run.gibba002.stderr:pool 'cephfs_data' created 2021-02-10T11:34:23.079 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:23.074 7f865fedd700 1 -- 172.21.2.102:0/535681610 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f86400186c0 msgr2=0x7f864001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:23.080 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:23.074 7f865fedd700 1 --2- 172.21.2.102:0/535681610 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f86400186c0 0x7f864001abc0 crc :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:23.080 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:23.074 7f865fedd700 1 -- 172.21.2.102:0/535681610 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f86581a7ef0 msgr2=0x7f86580a6a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:23.080 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:23.074 7f865fedd700 1 --2- 172.21.2.102:0/535681610 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f86581a7ef0 0x7f86580a6a40 secure :-1 s=READY pgs=2536 cs=0 l=1 rev1=1 rx=0x7f8648009270 tx=0x7f864800b960).stop 2021-02-10T11:34:23.080 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:23.074 7f865fedd700 1 -- 172.21.2.102:0/535681610 shutdown_connections 2021-02-10T11:34:23.080 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:23.074 7f865fedd700 1 --2- 172.21.2.102:0/535681610 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f86580a7060 0x7f86580b2920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:23.080 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:23.074 7f865fedd700 1 --2- 172.21.2.102:0/535681610 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f86581a7ef0 0x7f86580a6a40 unknown :-1 s=CLOSED pgs=2536 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:23.080 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:23.074 7f865fedd700 1 --2- 172.21.2.102:0/535681610 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f86400186c0 0x7f864001abc0 unknown :-1 s=CLOSED pgs=1119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:23.080 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:23.074 7f865fedd700 1 --2- 172.21.2.102:0/535681610 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f86580b1ba0 0x7f86581a78d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:23.080 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:23.074 7f865fedd700 1 -- 172.21.2.102:0/535681610 >> 172.21.2.102:0/535681610 conn(0x7f86580b5dc0 msgr2=0x7f8658095960 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:23.080 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:23.074 7f865fedd700 1 -- 172.21.2.102:0/535681610 shutdown_connections 2021-02-10T11:34:23.080 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:23.074 7f865fedd700 1 -- 172.21.2.102:0/535681610 wait complete. 2021-02-10T11:34:23.085 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2021-02-10T11:34:23.518 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:23.500 7f1284f2d700 1 Processor -- start 2021-02-10T11:34:23.518 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:23.500 7f1284f2d700 1 -- start start 2021-02-10T11:34:23.518 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:23.500 7f1284f2d700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f12800d1c80 0x7f12800cef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:23.518 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:23.500 7f1284f2d700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f12800cf590 0x7f12800cf990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:23.519 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:23.500 7f1284f2d700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12800fe1d0 con 0x7f12800d1880 2021-02-10T11:34:23.519 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:23.500 7f1284f2d700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12801b0cc0 con 0x7f12800cf590 2021-02-10T11:34:23.519 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:23.500 7f1284f2d700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1280109a70 con 0x7f12800d1c80 2021-02-10T11:34:23.519 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:23.501 7f127f314700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f12800cf590 0x7f12800cf990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:23.519 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:23.501 7f127f314700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f12800cf590 0x7f12800cf990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:36046/0 (socket says 172.21.2.102:36046) 2021-02-10T11:34:23.519 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:23.501 7f127f314700 1 -- 172.21.2.102:0/3106327735 learned_addr learned my addr 172.21.2.102:0/3106327735 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:34:23.519 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:23.501 7f127e312700 1 --2- 172.21.2.102:0/3106327735 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f12800d1c80 0x7f12800cef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:23.519 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:23.501 7f127f314700 1 -- 172.21.2.102:0/3106327735 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f12800d1c80 msgr2=0x7f12800cef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:34:23.519 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:23.501 7f127f314700 1 --2- 172.21.2.102:0/3106327735 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f12800d1c80 0x7f12800cef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:23.519 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:23.501 7f127f314700 1 -- 172.21.2.102:0/3106327735 >> v1:172.21.2.113:6789/0 conn(0x7f12800d1880 legacy=0x7f12801b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:34:23.519 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:23.501 7f127f314700 1 -- 172.21.2.102:0/3106327735 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1280088070 con 0x7f12800cf590 2021-02-10T11:34:23.520 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:23.501 7f127e312700 1 --2- 172.21.2.102:0/3106327735 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f12800d1c80 0x7f12800cef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:34:23.520 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:23.501 7f127f314700 1 --2- 172.21.2.102:0/3106327735 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f12800cf590 0x7f12800cf990 secure :-1 s=READY pgs=2537 cs=0 l=1 rev1=1 rx=0x7f1278006f00 tx=0x7f1278009f10).ready entity=mon.0 client_cookie=85f3ad64ea2d2324 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:23.520 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:23.501 7f127db11700 1 -- 172.21.2.102:0/3106327735 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f127800b3b0 con 0x7f12800cf590 2021-02-10T11:34:23.520 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:23.501 7f127db11700 1 -- 172.21.2.102:0/3106327735 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1278000d30 con 0x7f12800cf590 2021-02-10T11:34:23.520 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:23.501 7f127db11700 1 -- 172.21.2.102:0/3106327735 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f127800b820 con 0x7f12800cf590 2021-02-10T11:34:23.520 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:23.501 7f1284f2d700 1 -- 172.21.2.102:0/3106327735 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f12800cf590 msgr2=0x7f12800cf990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:23.520 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:23.501 7f1284f2d700 1 --2- 172.21.2.102:0/3106327735 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f12800cf590 0x7f12800cf990 secure :-1 s=READY pgs=2537 cs=0 l=1 rev1=1 rx=0x7f1278006f00 tx=0x7f1278009f10).stop 2021-02-10T11:34:23.520 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:23.502 7f1284f2d700 1 -- 172.21.2.102:0/3106327735 shutdown_connections 2021-02-10T11:34:23.520 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:23.502 7f1284f2d700 1 --2- 172.21.2.102:0/3106327735 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f12800d1c80 0x7f12800cef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:23.520 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:23.502 7f1284f2d700 1 --2- 172.21.2.102:0/3106327735 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f12800cf590 0x7f12800cf990 unknown :-1 s=CLOSED pgs=2537 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:23.520 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:23.502 7f1284f2d700 1 -- 172.21.2.102:0/3106327735 >> 172.21.2.102:0/3106327735 conn(0x7f12800ce430 msgr2=0x7f12800d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:23.520 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:23.502 7f1284f2d700 1 -- 172.21.2.102:0/3106327735 shutdown_connections 2021-02-10T11:34:23.521 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:23.502 7f1284f2d700 1 -- 172.21.2.102:0/3106327735 wait complete. 2021-02-10T11:34:23.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:23.517 7f1284f2d700 1 Processor -- start 2021-02-10T11:34:23.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:23.517 7f1284f2d700 1 -- start start 2021-02-10T11:34:23.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:23.517 7f1284f2d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1280094030 0x7f12800aa4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:23.526 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:23.517 7f1284f2d700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f12800aaad0 0x7f128019ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:23.526 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:23.517 7f1284f2d700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1280093650 0x7f128019c0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:23.526 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:23.517 7f1284f2d700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12801b91c0 con 0x7f1280094030 2021-02-10T11:34:23.526 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:23.517 7f1284f2d700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f128019c6c0 con 0x7f12800aaad0 2021-02-10T11:34:23.526 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:23.517 7f1284f2d700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12801b7f70 con 0x7f1280093650 2021-02-10T11:34:23.526 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:23.517 7f127eb13700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1280094030 0x7f12800aa4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:23.526 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:23.517 7f127eb13700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1280094030 0x7f12800aa4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:36048/0 (socket says 172.21.2.102:36048) 2021-02-10T11:34:23.526 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:23.517 7f127eb13700 1 -- 172.21.2.102:0/3204023208 learned_addr learned my addr 172.21.2.102:0/3204023208 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:34:23.526 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:23.518 7f127eb13700 1 -- 172.21.2.102:0/3204023208 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1280093650 msgr2=0x7f128019c0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:23.526 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:23.518 7f127eb13700 1 --2- 172.21.2.102:0/3204023208 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1280093650 0x7f128019c0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:23.527 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:23.518 7f127eb13700 1 -- 172.21.2.102:0/3204023208 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f12800aaad0 msgr2=0x7f128019ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:23.527 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:23.518 7f127eb13700 1 --2- 172.21.2.102:0/3204023208 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f12800aaad0 0x7f128019ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:23.527 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:23.518 7f127eb13700 1 -- 172.21.2.102:0/3204023208 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12801b8190 con 0x7f1280094030 2021-02-10T11:34:23.527 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:23.518 7f127eb13700 1 --2- 172.21.2.102:0/3204023208 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1280094030 0x7f12800aa4b0 secure :-1 s=READY pgs=2538 cs=0 l=1 rev1=1 rx=0x7f126800b5b0 tx=0x7f126800b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:23.527 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:23.518 7f12777fe700 1 -- 172.21.2.102:0/3204023208 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f126800bd40 con 0x7f1280094030 2021-02-10T11:34:23.528 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:23.518 7f12777fe700 1 -- 172.21.2.102:0/3204023208 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1268006020 con 0x7f1280094030 2021-02-10T11:34:23.528 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:23.518 7f12777fe700 1 -- 172.21.2.102:0/3204023208 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f12680054a0 con 0x7f1280094030 2021-02-10T11:34:23.528 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:23.518 7f1284f2d700 1 -- 172.21.2.102:0/3204023208 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12801b8190 con 0x7f1280094030 2021-02-10T11:34:23.528 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:23.518 7f1284f2d700 1 -- 172.21.2.102:0/3204023208 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12801b8720 con 0x7f1280094030 2021-02-10T11:34:23.528 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:23.519 7f12777fe700 1 -- 172.21.2.102:0/3204023208 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1268006200 con 0x7f1280094030 2021-02-10T11:34:23.528 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:23.519 7f12777fe700 1 --2- 172.21.2.102:0/3204023208 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f12640186c0 0x7f126401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:23.528 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:23.519 7f12777fe700 1 -- 172.21.2.102:0/3204023208 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 5457+0+0 (secure 0 0 0) 0x7f1268032520 con 0x7f1280094030 2021-02-10T11:34:23.528 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:23.519 7f127e312700 1 --2- 172.21.2.102:0/3204023208 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f12640186c0 0x7f126401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:23.528 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:23.520 7f127e312700 1 --2- 172.21.2.102:0/3204023208 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f12640186c0 0x7f126401abc0 crc :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:23.552 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:23.550 7f1284f2d700 1 -- 172.21.2.102:0/3204023208 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1280082aa0 con 0x7f1280094030 2021-02-10T11:34:23.555 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:23.553 7f12777fe700 1 -- 172.21.2.102:0/3204023208 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f1268000940 con 0x7f1280094030 2021-02-10T11:34:23.685 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:23.683 7f1284f2d700 1 -- 172.21.2.102:0/3204023208 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"data": "cephfs_data", "prefix": "fs new", "force": true, "fs_name": "cephfs", "metadata": "cephfs_metadata"} v 0) v1 -- 0x7f1280082aa0 con 0x7f1280094030 2021-02-10T11:34:23.689 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:23.687 7f12777fe700 1 -- 172.21.2.102:0/3204023208 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"data": "cephfs_data", "prefix": "fs new", "force": true, "fs_name": "cephfs", "metadata": "cephfs_metadata"}]=0 new fs with metadata pool 28 and data pool 29 v113) v1 ==== 189+0+0 (secure 0 0 0) 0x7f126801f530 con 0x7f1280094030 2021-02-10T11:34:23.691 INFO:teuthology.orchestra.run.gibba002.stderr:new fs with metadata pool 28 and data pool 29 2021-02-10T11:34:23.694 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:23.692 7f1284f2d700 1 -- 172.21.2.102:0/3204023208 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f12640186c0 msgr2=0x7f126401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:23.694 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:23.692 7f1284f2d700 1 --2- 172.21.2.102:0/3204023208 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f12640186c0 0x7f126401abc0 crc :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:23.694 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:23.692 7f1284f2d700 1 -- 172.21.2.102:0/3204023208 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1280094030 msgr2=0x7f12800aa4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:23.694 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:23.692 7f1284f2d700 1 --2- 172.21.2.102:0/3204023208 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1280094030 0x7f12800aa4b0 secure :-1 s=READY pgs=2538 cs=0 l=1 rev1=1 rx=0x7f126800b5b0 tx=0x7f126800b950).stop 2021-02-10T11:34:23.694 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:23.692 7f1284f2d700 1 -- 172.21.2.102:0/3204023208 shutdown_connections 2021-02-10T11:34:23.694 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:23.692 7f1284f2d700 1 --2- 172.21.2.102:0/3204023208 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1280093650 0x7f128019c0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:23.695 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:23.692 7f1284f2d700 1 --2- 172.21.2.102:0/3204023208 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f12640186c0 0x7f126401abc0 crc :-1 s=CLOSED pgs=1122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:23.695 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:23.692 7f1284f2d700 1 --2- 172.21.2.102:0/3204023208 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f12800aaad0 0x7f128019ba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:23.695 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:23.692 7f1284f2d700 1 --2- 172.21.2.102:0/3204023208 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1280094030 0x7f12800aa4b0 secure :-1 s=CLOSED pgs=2538 cs=0 l=1 rev1=1 rx=0x7f126800b5b0 tx=0x7f126800b950).stop 2021-02-10T11:34:23.695 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:23.692 7f1284f2d700 1 -- 172.21.2.102:0/3204023208 >> 172.21.2.102:0/3204023208 conn(0x7f12800846b0 msgr2=0x7f1280094950 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:23.695 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:23.692 7f1284f2d700 1 -- 172.21.2.102:0/3204023208 shutdown_connections 2021-02-10T11:34:23.695 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:23.692 7f1284f2d700 1 -- 172.21.2.102:0/3204023208 wait complete. 2021-02-10T11:34:23.703 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:34:24.129 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:24.111 7f0619efd700 1 Processor -- start 2021-02-10T11:34:24.130 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:24.111 7f0619efd700 1 -- start start 2021-02-10T11:34:24.130 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:24.112 7f0619efd700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f06141b0470 0x7f06141b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:24.130 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:24.112 7f0619efd700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f06140f20e0 0x7f06140f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:24.130 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:24.112 7f0619efd700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06140f0b50 con 0x7f06140ef610 2021-02-10T11:34:24.130 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:24.112 7f0619efd700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06140f0820 con 0x7f06141b0470 2021-02-10T11:34:24.130 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:24.112 7f0619efd700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06140f24e0 con 0x7f06140f20e0 2021-02-10T11:34:24.130 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:24.112 7f06137fe700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f06141b0470 0x7f06141b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:24.130 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:24.112 7f06137fe700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f06141b0470 0x7f06141b0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:36056/0 (socket says 172.21.2.102:36056) 2021-02-10T11:34:24.130 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:24.112 7f06137fe700 1 -- 172.21.2.102:0/4026354682 learned_addr learned my addr 172.21.2.102:0/4026354682 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:34:24.130 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:24.112 7f06137fe700 1 -- 172.21.2.102:0/4026354682 >> v2:172.21.2.113:3300/0 conn(0x7f06140f20e0 msgr2=0x7f06140f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:34:24.130 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:24.112 7f06137fe700 1 --2- 172.21.2.102:0/4026354682 >> v2:172.21.2.113:3300/0 conn(0x7f06140f20e0 0x7f06140f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:24.131 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:24.112 7f06137fe700 1 -- 172.21.2.102:0/4026354682 >> v1:172.21.2.113:6789/0 conn(0x7f06140ef610 legacy=0x7f06140efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:34:24.131 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:24.113 7f06137fe700 1 -- 172.21.2.102:0/4026354682 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06140e52e0 con 0x7f06141b0470 2021-02-10T11:34:24.131 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:24.113 7f06137fe700 1 --2- 172.21.2.102:0/4026354682 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f06141b0470 0x7f06141b0870 secure :-1 s=READY pgs=2539 cs=0 l=1 rev1=1 rx=0x7f05fc006700 tx=0x7f05fc009b80).ready entity=mon.0 client_cookie=c705da7d06827bc5 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:24.131 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:24.113 7f06127fc700 1 -- 172.21.2.102:0/4026354682 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f05fc00c0c0 con 0x7f06141b0470 2021-02-10T11:34:24.131 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:24.113 7f06127fc700 1 -- 172.21.2.102:0/4026354682 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05fc00c340 con 0x7f06141b0470 2021-02-10T11:34:24.131 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:24.113 7f06127fc700 1 -- 172.21.2.102:0/4026354682 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f05fc00c860 con 0x7f06141b0470 2021-02-10T11:34:24.131 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:24.113 7f0619efd700 1 -- 172.21.2.102:0/4026354682 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f06141b0470 msgr2=0x7f06141b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:24.131 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:24.113 7f0619efd700 1 --2- 172.21.2.102:0/4026354682 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f06141b0470 0x7f06141b0870 secure :-1 s=READY pgs=2539 cs=0 l=1 rev1=1 rx=0x7f05fc006700 tx=0x7f05fc009b80).stop 2021-02-10T11:34:24.131 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:24.114 7f0619efd700 1 -- 172.21.2.102:0/4026354682 shutdown_connections 2021-02-10T11:34:24.131 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:24.114 7f0619efd700 1 --2- 172.21.2.102:0/4026354682 >> v2:172.21.2.113:3300/0 conn(0x7f06140f20e0 0x7f06140f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:24.131 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:24.114 7f0619efd700 1 --2- 172.21.2.102:0/4026354682 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f06141b0470 0x7f06141b0870 unknown :-1 s=CLOSED pgs=2539 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:24.131 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:24.114 7f0619efd700 1 -- 172.21.2.102:0/4026354682 >> 172.21.2.102:0/4026354682 conn(0x7f06140ce3e0 msgr2=0x7f06140e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:24.131 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:24.114 7f0619efd700 1 -- 172.21.2.102:0/4026354682 shutdown_connections 2021-02-10T11:34:24.132 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:24.114 7f0619efd700 1 -- 172.21.2.102:0/4026354682 wait complete. 2021-02-10T11:34:24.133 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:24.128 7f0619efd700 1 Processor -- start 2021-02-10T11:34:24.134 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:24.128 7f0619efd700 1 -- start start 2021-02-10T11:34:24.134 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:24.129 7f0619efd700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f06141acc20 0x7f06141ad020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:24.134 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:24.129 7f0619efd700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f06141ad640 0x7f06141ada40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:24.134 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:24.129 7f0619efd700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0614188110 0x7f0614188510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:24.134 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:24.129 7f0619efd700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06141ac720 con 0x7f06141acc20 2021-02-10T11:34:24.134 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:24.129 7f0619efd700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06140b1b10 con 0x7f06141ad640 2021-02-10T11:34:24.134 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:24.129 7f0619efd700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0614188b30 con 0x7f0614188110 2021-02-10T11:34:24.134 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:24.129 7f06137fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f06141acc20 0x7f06141ad020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:24.134 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:24.129 7f06137fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f06141acc20 0x7f06141ad020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:36060/0 (socket says 172.21.2.102:36060) 2021-02-10T11:34:24.134 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:24.129 7f06137fe700 1 -- 172.21.2.102:0/2236222932 learned_addr learned my addr 172.21.2.102:0/2236222932 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:34:24.134 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:24.129 7f0612ffd700 1 --2- 172.21.2.102:0/2236222932 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f06141ad640 0x7f06141ada40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:24.134 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:24.129 7f0613fff700 1 --2- 172.21.2.102:0/2236222932 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0614188110 0x7f0614188510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:24.135 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:24.130 7f06137fe700 1 -- 172.21.2.102:0/2236222932 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0614188110 msgr2=0x7f0614188510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:24.135 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:24.130 7f06137fe700 1 --2- 172.21.2.102:0/2236222932 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0614188110 0x7f0614188510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:24.135 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:24.130 7f06137fe700 1 -- 172.21.2.102:0/2236222932 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f06141ad640 msgr2=0x7f06141ada40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:24.135 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:24.130 7f06137fe700 1 --2- 172.21.2.102:0/2236222932 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f06141ad640 0x7f06141ada40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:24.135 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:24.130 7f06137fe700 1 -- 172.21.2.102:0/2236222932 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0614188d50 con 0x7f06141acc20 2021-02-10T11:34:24.135 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:24.130 7f06137fe700 1 --2- 172.21.2.102:0/2236222932 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f06141acc20 0x7f06141ad020 secure :-1 s=READY pgs=2540 cs=0 l=1 rev1=1 rx=0x7f05fc00b5e0 tx=0x7f05fc00b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:24.135 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:24.130 7f060bfff700 1 -- 172.21.2.102:0/2236222932 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f05fc00bd70 con 0x7f06141acc20 2021-02-10T11:34:24.135 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:24.130 7f060bfff700 1 -- 172.21.2.102:0/2236222932 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05fc002030 con 0x7f06141acc20 2021-02-10T11:34:24.135 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:24.130 7f060bfff700 1 -- 172.21.2.102:0/2236222932 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f05fc002550 con 0x7f06141acc20 2021-02-10T11:34:24.135 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:24.130 7f0619efd700 1 -- 172.21.2.102:0/2236222932 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0614188d50 con 0x7f06141acc20 2021-02-10T11:34:24.135 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:24.130 7f0619efd700 1 -- 172.21.2.102:0/2236222932 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06141892e0 con 0x7f06141acc20 2021-02-10T11:34:24.135 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:24.131 7f060bfff700 1 -- 172.21.2.102:0/2236222932 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f05fc000c90 con 0x7f06141acc20 2021-02-10T11:34:24.136 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:24.131 7f060bfff700 1 --2- 172.21.2.102:0/2236222932 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f05f80186c0 0x7f05f801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:24.136 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:24.131 7f060bfff700 1 -- 172.21.2.102:0/2236222932 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(136..136 src has 1..136) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f05fc032460 con 0x7f06141acc20 2021-02-10T11:34:24.136 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:24.131 7f0612ffd700 1 --2- 172.21.2.102:0/2236222932 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f05f80186c0 0x7f05f801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:24.136 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:24.131 7f0612ffd700 1 --2- 172.21.2.102:0/2236222932 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f05f80186c0 0x7f05f801abc0 crc :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:24.164 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:24.162 7f0619efd700 1 -- 172.21.2.102:0/2236222932 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06140eddf0 con 0x7f06141acc20 2021-02-10T11:34:24.166 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:24.165 7f060bfff700 1 -- 172.21.2.102:0/2236222932 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f05fc026690 con 0x7f06141acc20 2021-02-10T11:34:24.289 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:24.287 7f0619efd700 1 -- 172.21.2.102:0/2236222932 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f06140e52e0 con 0x7f06141acc20 2021-02-10T11:34:24.290 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:24.288 7f060bfff700 1 -- 172.21.2.102:0/2236222932 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v136) v1 ==== 74+0+14020 (secure 0 0 0) 0x7f05fc026690 con 0x7f06141acc20 2021-02-10T11:34:24.290 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:34:24.291 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":136,"fsid":"68db30ea-dbce-4aef-a6df-f01c6fea0513","created":"2021-02-10 11:03:06.331557","modified":"2021-02-10 11:34:23.684691","last_up_change":"2021-02-10 11:03:32.690562","last_in_change":"2021-02-10 11:03:32.690562","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":29,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:03:37.322519","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":28,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:34:21.649291","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"136","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":29,"pool_name":"cephfs_data","create_time":"2021-02-10 11:34:22.675483","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"136","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"be691b4d-a039-408a-af80-9d53f02b6f1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":134,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6818","nonce":36088},{"type":"v1","addr":"172.21.2.102:6820","nonce":36088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6822","nonce":36088},{"type":"v1","addr":"172.21.2.102:6824","nonce":36088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6830","nonce":36088},{"type":"v1","addr":"172.21.2.102:6832","nonce":36088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6826","nonce":36088},{"type":"v1","addr":"172.21.2.102:6828","nonce":36088}]},"public_addr":"172.21.2.102:6820/36088","cluster_addr":"172.21.2.102:6824/36088","heartbeat_back_addr":"172.21.2.102:6832/36088","heartbeat_front_addr":"172.21.2.102:6828/36088","state":["exists","up"]},{"osd":1,"uuid":"64e3ccc8-9c69-4d7c-865e-af8e83822635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":135,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6819","nonce":36087},{"type":"v1","addr":"172.21.2.102:6821","nonce":36087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6823","nonce":36087},{"type":"v1","addr":"172.21.2.102:6825","nonce":36087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6831","nonce":36087},{"type":"v1","addr":"172.21.2.102:6833","nonce":36087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6827","nonce":36087},{"type":"v1","addr":"172.21.2.102:6829","nonce":36087}]},"public_addr":"172.21.2.102:6821/36087","cluster_addr":"172.21.2.102:6825/36087","heartbeat_back_addr":"172.21.2.102:6833/36087","heartbeat_front_addr":"172.21.2.102:6829/36087","state":["exists","up"]},{"osd":2,"uuid":"153700af-747f-481f-bdf4-d6072d3a842f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6810","nonce":36089},{"type":"v1","addr":"172.21.2.102:6811","nonce":36089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6812","nonce":36089},{"type":"v1","addr":"172.21.2.102:6813","nonce":36089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6816","nonce":36089},{"type":"v1","addr":"172.21.2.102:6817","nonce":36089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6814","nonce":36089},{"type":"v1","addr":"172.21.2.102:6815","nonce":36089}]},"public_addr":"172.21.2.102:6811/36089","cluster_addr":"172.21.2.102:6813/36089","heartbeat_back_addr":"172.21.2.102:6817/36089","heartbeat_front_addr":"172.21.2.102:6815/36089","state":["exists","up"]},{"osd":3,"uuid":"cbc678f4-83a9-45e6-8830-2f7c232b1ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6802","nonce":36090},{"type":"v1","addr":"172.21.2.102:6803","nonce":36090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6804","nonce":36090},{"type":"v1","addr":"172.21.2.102:6805","nonce":36090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6808","nonce":36090},{"type":"v1","addr":"172.21.2.102:6809","nonce":36090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6806","nonce":36090},{"type":"v1","addr":"172.21.2.102:6807","nonce":36090}]},"public_addr":"172.21.2.102:6803/36090","cluster_addr":"172.21.2.102:6805/36090","heartbeat_back_addr":"172.21.2.102:6809/36090","heartbeat_front_addr":"172.21.2.102:6807/36090","state":["exists","up"]},{"osd":4,"uuid":"c01e4008-b258-4f10-afd2-9e7f37214351","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":135,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6808","nonce":36368},{"type":"v1","addr":"172.21.2.113:6811","nonce":36368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6814","nonce":36368},{"type":"v1","addr":"172.21.2.113:6818","nonce":36368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6826","nonce":36368},{"type":"v1","addr":"172.21.2.113:6828","nonce":36368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6820","nonce":36368},{"type":"v1","addr":"172.21.2.113:6824","nonce":36368}]},"public_addr":"172.21.2.113:6811/36368","cluster_addr":"172.21.2.113:6818/36368","heartbeat_back_addr":"172.21.2.113:6828/36368","heartbeat_front_addr":"172.21.2.113:6824/36368","state":["exists","up"]},{"osd":5,"uuid":"551f6007-daf8-4a98-8ee0-c2b16bcccb5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":134,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6802","nonce":36366},{"type":"v1","addr":"172.21.2.113:6805","nonce":36366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6807","nonce":36366},{"type":"v1","addr":"172.21.2.113:6810","nonce":36366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6819","nonce":36366},{"type":"v1","addr":"172.21.2.113:6822","nonce":36366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6812","nonce":36366},{"type":"v1","addr":"172.21.2.113:6815","nonce":36366}]},"public_addr":"172.21.2.113:6805/36366","cluster_addr":"172.21.2.113:6810/36366","heartbeat_back_addr":"172.21.2.113:6822/36366","heartbeat_front_addr":"172.21.2.113:6815/36366","state":["exists","up"]},{"osd":6,"uuid":"bf67c040-0e50-48be-b102-c3943bc9ebef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":135,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6800","nonce":36367},{"type":"v1","addr":"172.21.2.113:6801","nonce":36367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6803","nonce":36367},{"type":"v1","addr":"172.21.2.113:6804","nonce":36367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6813","nonce":36367},{"type":"v1","addr":"172.21.2.113:6816","nonce":36367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6806","nonce":36367},{"type":"v1","addr":"172.21.2.113:6809","nonce":36367}]},"public_addr":"172.21.2.113:6801/36367","cluster_addr":"172.21.2.113:6804/36367","heartbeat_back_addr":"172.21.2.113:6816/36367","heartbeat_front_addr":"172.21.2.113:6809/36367","state":["exists","up"]},{"osd":7,"uuid":"f5461f80-018d-428f-be89-2eb2bcbe7bb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":135,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6817","nonce":36365},{"type":"v1","addr":"172.21.2.113:6821","nonce":36365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6823","nonce":36365},{"type":"v1","addr":"172.21.2.113:6825","nonce":36365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6830","nonce":36365},{"type":"v1","addr":"172.21.2.113:6831","nonce":36365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6827","nonce":36365},{"type":"v1","addr":"172.21.2.113:6829","nonce":36365}]},"public_addr":"172.21.2.113:6821/36365","cluster_addr":"172.21.2.113:6825/36365","heartbeat_back_addr":"172.21.2.113:6831/36365","heartbeat_front_addr":"172.21.2.113:6829/36365","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:34:24.292 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:24.290 7f0619efd700 1 -- 172.21.2.102:0/2236222932 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f05f80186c0 msgr2=0x7f05f801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:24.292 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:24.290 7f0619efd700 1 --2- 172.21.2.102:0/2236222932 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f05f80186c0 0x7f05f801abc0 crc :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:24.292 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:24.290 7f0619efd700 1 -- 172.21.2.102:0/2236222932 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f06141acc20 msgr2=0x7f06141ad020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:24.292 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:24.290 7f0619efd700 1 --2- 172.21.2.102:0/2236222932 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f06141acc20 0x7f06141ad020 secure :-1 s=READY pgs=2540 cs=0 l=1 rev1=1 rx=0x7f05fc00b5e0 tx=0x7f05fc00b980).stop 2021-02-10T11:34:24.292 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:24.290 7f0619efd700 1 -- 172.21.2.102:0/2236222932 shutdown_connections 2021-02-10T11:34:24.292 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:24.290 7f0619efd700 1 --2- 172.21.2.102:0/2236222932 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0614188110 0x7f0614188510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:24.292 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:24.290 7f0619efd700 1 --2- 172.21.2.102:0/2236222932 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f05f80186c0 0x7f05f801abc0 unknown :-1 s=CLOSED pgs=1123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:24.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:24.290 7f0619efd700 1 --2- 172.21.2.102:0/2236222932 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f06141ad640 0x7f06141ada40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:24.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:24.290 7f0619efd700 1 --2- 172.21.2.102:0/2236222932 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f06141acc20 0x7f06141ad020 unknown :-1 s=CLOSED pgs=2540 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:24.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:24.290 7f0619efd700 1 -- 172.21.2.102:0/2236222932 >> 172.21.2.102:0/2236222932 conn(0x7f061419f7a0 msgr2=0x7f06141922a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:24.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:24.290 7f0619efd700 1 -- 172.21.2.102:0/2236222932 shutdown_connections 2021-02-10T11:34:24.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:24.290 7f0619efd700 1 -- 172.21.2.102:0/2236222932 wait complete. 2021-02-10T11:34:24.300 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:34:24.728 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:24.710 7f98e140b700 1 Processor -- start 2021-02-10T11:34:24.728 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:24.710 7f98e140b700 1 -- start start 2021-02-10T11:34:24.728 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:24.710 7f98e140b700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f98dc0d0340 0x7f98dc0ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:24.728 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:24.710 7f98e140b700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f98dc0d1c70 0x7f98dc0efc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:24.729 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:24.710 7f98e140b700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98dc0f0b10 con 0x7f98dc0d1870 2021-02-10T11:34:24.729 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:24.710 7f98e140b700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98dc0e5ed0 con 0x7f98dc0d1c70 2021-02-10T11:34:24.729 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:24.710 7f98e140b700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98dc1b0be0 con 0x7f98dc0d0340 2021-02-10T11:34:24.729 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:24.710 7f98db87e700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f98dc0d1c70 0x7f98dc0efc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:24.729 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:24.710 7f98db87e700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f98dc0d1c70 0x7f98dc0efc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:36068/0 (socket says 172.21.2.102:36068) 2021-02-10T11:34:24.729 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:24.710 7f98db87e700 1 -- 172.21.2.102:0/1968904232 learned_addr learned my addr 172.21.2.102:0/1968904232 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:34:24.729 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:24.710 7f98db07d700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f98dc0d1870 0x7f98dc1b03f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:56524/0 (socket says 172.21.2.102:56524) 2021-02-10T11:34:24.729 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:24.711 7f98d287c700 1 --2- 172.21.2.102:0/1968904232 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f98dc0d0340 0x7f98dc0ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:24.729 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:24.711 7f98db87e700 1 -- 172.21.2.102:0/1968904232 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f98dc0d0340 msgr2=0x7f98dc0ef640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:34:24.729 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:24.711 7f98db87e700 1 --2- 172.21.2.102:0/1968904232 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f98dc0d0340 0x7f98dc0ef640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:24.729 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:24.711 7f98db87e700 1 -- 172.21.2.102:0/1968904232 >> v1:172.21.2.102:6789/0 conn(0x7f98dc0d1870 legacy=0x7f98dc1b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:24.729 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:24.711 7f98db87e700 1 -- 172.21.2.102:0/1968904232 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98dc0f20a0 con 0x7f98dc0d1c70 2021-02-10T11:34:24.729 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:24.711 7f98d287c700 1 --2- 172.21.2.102:0/1968904232 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f98dc0d0340 0x7f98dc0ef640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:34:24.730 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:24.711 7f98db87e700 1 --2- 172.21.2.102:0/1968904232 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f98dc0d1c70 0x7f98dc0efc60 secure :-1 s=READY pgs=2542 cs=0 l=1 rev1=1 rx=0x7f98d4006f00 tx=0x7f98d4009f10).ready entity=mon.0 client_cookie=7f2773d4f1af453e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:24.730 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:24.711 7f98da87c700 1 -- 172.21.2.102:0/1968904232 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f98d400b3b0 con 0x7f98dc0d1c70 2021-02-10T11:34:24.730 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:24.711 7f98da87c700 1 -- 172.21.2.102:0/1968904232 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98d400b730 con 0x7f98dc0d1c70 2021-02-10T11:34:24.730 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:24.711 7f98da87c700 1 -- 172.21.2.102:0/1968904232 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f98d400bc50 con 0x7f98dc0d1c70 2021-02-10T11:34:24.730 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:24.711 7f98e140b700 1 -- 172.21.2.102:0/1968904232 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f98dc0d1c70 msgr2=0x7f98dc0efc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:24.730 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:24.711 7f98e140b700 1 --2- 172.21.2.102:0/1968904232 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f98dc0d1c70 0x7f98dc0efc60 secure :-1 s=READY pgs=2542 cs=0 l=1 rev1=1 rx=0x7f98d4006f00 tx=0x7f98d4009f10).stop 2021-02-10T11:34:24.730 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:24.712 7f98e140b700 1 -- 172.21.2.102:0/1968904232 shutdown_connections 2021-02-10T11:34:24.730 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:24.712 7f98e140b700 1 --2- 172.21.2.102:0/1968904232 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f98dc0d1c70 0x7f98dc0efc60 unknown :-1 s=CLOSED pgs=2542 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:24.730 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:24.712 7f98e140b700 1 --2- 172.21.2.102:0/1968904232 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f98dc0d0340 0x7f98dc0ef640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:24.730 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:24.712 7f98e140b700 1 -- 172.21.2.102:0/1968904232 >> 172.21.2.102:0/1968904232 conn(0x7f98dc0ce450 msgr2=0x7f98dc0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:24.730 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:24.712 7f98e140b700 1 -- 172.21.2.102:0/1968904232 shutdown_connections 2021-02-10T11:34:24.730 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:24.712 7f98e140b700 1 -- 172.21.2.102:0/1968904232 wait complete. 2021-02-10T11:34:24.730 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:24.726 7f98e140b700 1 Processor -- start 2021-02-10T11:34:24.731 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:24.726 7f98e140b700 1 -- start start 2021-02-10T11:34:24.733 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:24.727 7f98e140b700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f98dc0ba6d0 0x7f98dc0bb200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:24.733 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:24.727 7f98e140b700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f98dc0b0590 0x7f98dc0b0990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:24.733 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:24.727 7f98e140b700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f98dc0b0fb0 0x7f98dc1a7490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:24.733 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:24.727 7f98e140b700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98dc0baf30 con 0x7f98dc0b0fb0 2021-02-10T11:34:24.733 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:24.727 7f98e140b700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98dc0b0010 con 0x7f98dc0b0590 2021-02-10T11:34:24.733 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:24.727 7f98e140b700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98dc0baad0 con 0x7f98dc0ba6d0 2021-02-10T11:34:24.733 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:24.727 7f98db07d700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f98dc0ba6d0 0x7f98dc0bb200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:24.733 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:24.727 7f98db07d700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f98dc0ba6d0 0x7f98dc0bb200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.113:3301/0 says I am v2:172.21.2.102:41610/0 (socket says 172.21.2.102:41610) 2021-02-10T11:34:24.733 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:24.727 7f98db07d700 1 -- 172.21.2.102:0/2319956248 learned_addr learned my addr 172.21.2.102:0/2319956248 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:34:24.733 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:24.727 7f98db87e700 1 --2- 172.21.2.102:0/2319956248 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f98dc0b0fb0 0x7f98dc1a7490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:24.733 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:24.727 7f98d287c700 1 --2- 172.21.2.102:0/2319956248 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f98dc0b0590 0x7f98dc0b0990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:24.734 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:24.728 7f98db07d700 1 -- 172.21.2.102:0/2319956248 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f98dc0b0590 msgr2=0x7f98dc0b0990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:24.734 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:24.728 7f98db07d700 1 --2- 172.21.2.102:0/2319956248 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f98dc0b0590 0x7f98dc0b0990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:24.734 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:24.728 7f98db07d700 1 -- 172.21.2.102:0/2319956248 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f98dc0b0fb0 msgr2=0x7f98dc1a7490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:24.734 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:24.728 7f98db07d700 1 --2- 172.21.2.102:0/2319956248 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f98dc0b0fb0 0x7f98dc1a7490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:24.734 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:24.728 7f98db07d700 1 -- 172.21.2.102:0/2319956248 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98dc1a7ab0 con 0x7f98dc0ba6d0 2021-02-10T11:34:24.734 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:24.728 7f98db07d700 1 --2- 172.21.2.102:0/2319956248 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f98dc0ba6d0 0x7f98dc0bb200 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7f98cc00a590 tx=0x7f98cc00a930).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:24.734 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:24.728 7f98d8878700 1 -- 172.21.2.102:0/2319956248 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f98cc000ef0 con 0x7f98dc0ba6d0 2021-02-10T11:34:24.734 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:24.728 7f98d8878700 1 -- 172.21.2.102:0/2319956248 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98cc00ad20 con 0x7f98dc0ba6d0 2021-02-10T11:34:24.734 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:24.728 7f98e140b700 1 -- 172.21.2.102:0/2319956248 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98dc1a7ab0 con 0x7f98dc0ba6d0 2021-02-10T11:34:24.734 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:24.728 7f98d8878700 1 -- 172.21.2.102:0/2319956248 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f98cc004620 con 0x7f98dc0ba6d0 2021-02-10T11:34:24.734 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:24.728 7f98e140b700 1 -- 172.21.2.102:0/2319956248 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98dc1a7ab0 con 0x7f98dc0ba6d0 2021-02-10T11:34:24.734 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:24.729 7f98d8878700 1 -- 172.21.2.102:0/2319956248 <== mon.2 v2:172.21.2.113:3301/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f98cc0043a0 con 0x7f98dc0ba6d0 2021-02-10T11:34:24.734 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:24.729 7f98d8878700 1 --2- 172.21.2.102:0/2319956248 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f98c00186c0 0x7f98c001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:24.739 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:24.729 7f98d8878700 1 -- 172.21.2.102:0/2319956248 <== mon.2 v2:172.21.2.113:3301/0 5 ==== osd_map(137..137 src has 1..137) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f98cc032420 con 0x7f98dc0ba6d0 2021-02-10T11:34:24.739 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:24.729 7f98d287c700 1 --2- 172.21.2.102:0/2319956248 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f98c00186c0 0x7f98c001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:24.739 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:24.729 7f98d287c700 1 --2- 172.21.2.102:0/2319956248 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f98c00186c0 0x7f98c001abc0 crc :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:24.762 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:24.760 7f98e140b700 1 -- 172.21.2.102:0/2319956248 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98dc0baad0 con 0x7f98dc0ba6d0 2021-02-10T11:34:24.766 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:24.764 7f98d8878700 1 -- 172.21.2.102:0/2319956248 <== mon.2 v2:172.21.2.113:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f98cc030d50 con 0x7f98dc0ba6d0 2021-02-10T11:34:24.894 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:24.892 7f98e140b700 1 -- 172.21.2.102:0/2319956248 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f98dc0baad0 con 0x7f98dc0ba6d0 2021-02-10T11:34:24.895 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:24.894 7f98d8878700 1 -- 172.21.2.102:0/2319956248 <== mon.2 v2:172.21.2.113:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v137) v1 ==== 74+0+14007 (secure 0 0 0) 0x7f98cc01f530 con 0x7f98dc0ba6d0 2021-02-10T11:34:24.898 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:34:24.898 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":137,"fsid":"68db30ea-dbce-4aef-a6df-f01c6fea0513","created":"2021-02-10 11:03:06.331557","modified":"2021-02-10 11:34:24.687007","last_up_change":"2021-02-10 11:03:32.690562","last_in_change":"2021-02-10 11:03:32.690562","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":29,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:03:37.322519","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":28,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:34:21.649291","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"136","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":29,"pool_name":"cephfs_data","create_time":"2021-02-10 11:34:22.675483","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"137","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"be691b4d-a039-408a-af80-9d53f02b6f1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":134,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6818","nonce":36088},{"type":"v1","addr":"172.21.2.102:6820","nonce":36088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6822","nonce":36088},{"type":"v1","addr":"172.21.2.102:6824","nonce":36088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6830","nonce":36088},{"type":"v1","addr":"172.21.2.102:6832","nonce":36088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6826","nonce":36088},{"type":"v1","addr":"172.21.2.102:6828","nonce":36088}]},"public_addr":"172.21.2.102:6820/36088","cluster_addr":"172.21.2.102:6824/36088","heartbeat_back_addr":"172.21.2.102:6832/36088","heartbeat_front_addr":"172.21.2.102:6828/36088","state":["exists","up"]},{"osd":1,"uuid":"64e3ccc8-9c69-4d7c-865e-af8e83822635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":135,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6819","nonce":36087},{"type":"v1","addr":"172.21.2.102:6821","nonce":36087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6823","nonce":36087},{"type":"v1","addr":"172.21.2.102:6825","nonce":36087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6831","nonce":36087},{"type":"v1","addr":"172.21.2.102:6833","nonce":36087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6827","nonce":36087},{"type":"v1","addr":"172.21.2.102:6829","nonce":36087}]},"public_addr":"172.21.2.102:6821/36087","cluster_addr":"172.21.2.102:6825/36087","heartbeat_back_addr":"172.21.2.102:6833/36087","heartbeat_front_addr":"172.21.2.102:6829/36087","state":["exists","up"]},{"osd":2,"uuid":"153700af-747f-481f-bdf4-d6072d3a842f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6810","nonce":36089},{"type":"v1","addr":"172.21.2.102:6811","nonce":36089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6812","nonce":36089},{"type":"v1","addr":"172.21.2.102:6813","nonce":36089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6816","nonce":36089},{"type":"v1","addr":"172.21.2.102:6817","nonce":36089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6814","nonce":36089},{"type":"v1","addr":"172.21.2.102:6815","nonce":36089}]},"public_addr":"172.21.2.102:6811/36089","cluster_addr":"172.21.2.102:6813/36089","heartbeat_back_addr":"172.21.2.102:6817/36089","heartbeat_front_addr":"172.21.2.102:6815/36089","state":["exists","up"]},{"osd":3,"uuid":"cbc678f4-83a9-45e6-8830-2f7c232b1ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6802","nonce":36090},{"type":"v1","addr":"172.21.2.102:6803","nonce":36090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6804","nonce":36090},{"type":"v1","addr":"172.21.2.102:6805","nonce":36090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6808","nonce":36090},{"type":"v1","addr":"172.21.2.102:6809","nonce":36090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6806","nonce":36090},{"type":"v1","addr":"172.21.2.102:6807","nonce":36090}]},"public_addr":"172.21.2.102:6803/36090","cluster_addr":"172.21.2.102:6805/36090","heartbeat_back_addr":"172.21.2.102:6809/36090","heartbeat_front_addr":"172.21.2.102:6807/36090","state":["exists","up"]},{"osd":4,"uuid":"c01e4008-b258-4f10-afd2-9e7f37214351","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":135,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6808","nonce":36368},{"type":"v1","addr":"172.21.2.113:6811","nonce":36368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6814","nonce":36368},{"type":"v1","addr":"172.21.2.113:6818","nonce":36368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6826","nonce":36368},{"type":"v1","addr":"172.21.2.113:6828","nonce":36368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6820","nonce":36368},{"type":"v1","addr":"172.21.2.113:6824","nonce":36368}]},"public_addr":"172.21.2.113:6811/36368","cluster_addr":"172.21.2.113:6818/36368","heartbeat_back_addr":"172.21.2.113:6828/36368","heartbeat_front_addr":"172.21.2.113:6824/36368","state":["exists","up"]},{"osd":5,"uuid":"551f6007-daf8-4a98-8ee0-c2b16bcccb5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":134,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6802","nonce":36366},{"type":"v1","addr":"172.21.2.113:6805","nonce":36366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6807","nonce":36366},{"type":"v1","addr":"172.21.2.113:6810","nonce":36366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6819","nonce":36366},{"type":"v1","addr":"172.21.2.113:6822","nonce":36366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6812","nonce":36366},{"type":"v1","addr":"172.21.2.113:6815","nonce":36366}]},"public_addr":"172.21.2.113:6805/36366","cluster_addr":"172.21.2.113:6810/36366","heartbeat_back_addr":"172.21.2.113:6822/36366","heartbeat_front_addr":"172.21.2.113:6815/36366","state":["exists","up"]},{"osd":6,"uuid":"bf67c040-0e50-48be-b102-c3943bc9ebef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":135,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6800","nonce":36367},{"type":"v1","addr":"172.21.2.113:6801","nonce":36367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6803","nonce":36367},{"type":"v1","addr":"172.21.2.113:6804","nonce":36367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6813","nonce":36367},{"type":"v1","addr":"172.21.2.113:6816","nonce":36367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6806","nonce":36367},{"type":"v1","addr":"172.21.2.113:6809","nonce":36367}]},"public_addr":"172.21.2.113:6801/36367","cluster_addr":"172.21.2.113:6804/36367","heartbeat_back_addr":"172.21.2.113:6816/36367","heartbeat_front_addr":"172.21.2.113:6809/36367","state":["exists","up"]},{"osd":7,"uuid":"f5461f80-018d-428f-be89-2eb2bcbe7bb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":135,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6817","nonce":36365},{"type":"v1","addr":"172.21.2.113:6821","nonce":36365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6823","nonce":36365},{"type":"v1","addr":"172.21.2.113:6825","nonce":36365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6830","nonce":36365},{"type":"v1","addr":"172.21.2.113:6831","nonce":36365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6827","nonce":36365},{"type":"v1","addr":"172.21.2.113:6829","nonce":36365}]},"public_addr":"172.21.2.113:6821/36365","cluster_addr":"172.21.2.113:6825/36365","heartbeat_back_addr":"172.21.2.113:6831/36365","heartbeat_front_addr":"172.21.2.113:6829/36365","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:34:24.900 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:24.897 7f98e140b700 1 -- 172.21.2.102:0/2319956248 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f98c00186c0 msgr2=0x7f98c001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:24.900 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:24.897 7f98e140b700 1 --2- 172.21.2.102:0/2319956248 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f98c00186c0 0x7f98c001abc0 crc :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:24.900 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:24.897 7f98e140b700 1 -- 172.21.2.102:0/2319956248 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f98dc0ba6d0 msgr2=0x7f98dc0bb200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:24.900 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:24.897 7f98e140b700 1 --2- 172.21.2.102:0/2319956248 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f98dc0ba6d0 0x7f98dc0bb200 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7f98cc00a590 tx=0x7f98cc00a930).stop 2021-02-10T11:34:24.900 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:24.897 7f98e140b700 1 -- 172.21.2.102:0/2319956248 shutdown_connections 2021-02-10T11:34:24.900 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:24.897 7f98e140b700 1 --2- 172.21.2.102:0/2319956248 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f98dc0b0fb0 0x7f98dc1a7490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:24.901 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:24.897 7f98e140b700 1 --2- 172.21.2.102:0/2319956248 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f98c00186c0 0x7f98c001abc0 unknown :-1 s=CLOSED pgs=1124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:24.901 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:24.897 7f98e140b700 1 --2- 172.21.2.102:0/2319956248 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f98dc0b0590 0x7f98dc0b0990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:24.901 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:24.897 7f98e140b700 1 --2- 172.21.2.102:0/2319956248 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f98dc0ba6d0 0x7f98dc0bb200 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:24.901 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:24.897 7f98e140b700 1 -- 172.21.2.102:0/2319956248 >> 172.21.2.102:0/2319956248 conn(0x7f98dc09eb60 msgr2=0x7f98dc0a67b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:24.901 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:24.897 7f98e140b700 1 -- 172.21.2.102:0/2319956248 shutdown_connections 2021-02-10T11:34:24.901 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:24.898 7f98e140b700 1 -- 172.21.2.102:0/2319956248 wait complete. 2021-02-10T11:34:24.908 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2021-02-10T11:34:25.341 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:25.322 7f13cd3ca700 1 Processor -- start 2021-02-10T11:34:25.341 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:25.322 7f13cd3ca700 1 -- start start 2021-02-10T11:34:25.341 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:25.322 7f13cd3ca700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f13c80d1870 0x7f13c81b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:25.342 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:25.322 7f13cd3ca700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f13c81b0ae0 0x7f13c8108940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:25.342 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:25.322 7f13cd3ca700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13c80fe1c0 con 0x7f13c80d1c70 2021-02-10T11:34:25.342 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:25.322 7f13cd3ca700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13c8109a60 con 0x7f13c80d1870 2021-02-10T11:34:25.342 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:25.322 7f13cd3ca700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13c8109750 con 0x7f13c81b0ae0 2021-02-10T11:34:25.342 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:25.322 7f13c787e700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f13c80d1c70 0x7f13c8108f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:56542/0 (socket says 172.21.2.102:56542) 2021-02-10T11:34:25.342 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:25.322 7f13c787e700 1 -- 172.21.2.102:0/4211540866 learned_addr learned my addr 172.21.2.102:0/4211540866 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:34:25.342 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:25.323 7f13c607b700 1 -- 172.21.2.102:0/4211540866 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1474545499 0 0) 0x7f13c0001500 con 0x7f13c80d1c70 2021-02-10T11:34:25.342 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:25.323 7f13c607b700 1 -- 172.21.2.102:0/4211540866 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13b4002fe0 con 0x7f13c80d1c70 2021-02-10T11:34:25.342 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:25.323 7f13c607b700 1 -- 172.21.2.102:0/4211540866 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3281672787 0 0) 0x7f13c0001270 con 0x7f13c80d1c70 2021-02-10T11:34:25.342 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:25.323 7f13c607b700 1 -- 172.21.2.102:0/4211540866 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f13c81b0ae0 msgr2=0x7f13c8108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:34:25.343 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:25.323 7f13c707d700 1 --2- 172.21.2.102:0/4211540866 >> v2:172.21.2.113:3300/0 conn(0x7f13c80d1870 0x7f13c81b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:25.343 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:25.323 7f13c687c700 1 --2- 172.21.2.102:0/4211540866 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f13c81b0ae0 0x7f13c8108940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:25.343 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:25.323 7f13c607b700 1 --2- 172.21.2.102:0/4211540866 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f13c81b0ae0 0x7f13c8108940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:25.343 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:25.323 7f13c607b700 1 -- 172.21.2.102:0/4211540866 >> v2:172.21.2.113:3300/0 conn(0x7f13c80d1870 msgr2=0x7f13c81b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:34:25.343 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:25.323 7f13c607b700 1 --2- 172.21.2.102:0/4211540866 >> v2:172.21.2.113:3300/0 conn(0x7f13c80d1870 0x7f13c81b04c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:25.343 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:25.323 7f13c607b700 1 -- 172.21.2.102:0/4211540866 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13c8088070 con 0x7f13c80d1c70 2021-02-10T11:34:25.343 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:25.323 7f13c607b700 1 -- 172.21.2.102:0/4211540866 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7f13c0001960 con 0x7f13c80d1c70 2021-02-10T11:34:25.343 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:25.323 7f13c607b700 1 -- 172.21.2.102:0/4211540866 >> v1:172.21.2.102:6789/0 conn(0x7f13c80d1c70 legacy=0x7f13c8108f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:25.343 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:25.323 7f13c607b700 1 --2- 172.21.2.102:0/4211540866 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f13b4003b90 0x7f13b4006040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:25.343 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:25.323 7f13c607b700 1 --2- 172.21.2.102:0/4211540866 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f13b40068e0 0x7f13b4008db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:25.343 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:25.323 7f13c607b700 1 --2- 172.21.2.102:0/4211540866 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f13b4009650 0x7f13b400bb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:25.343 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:25.323 7f13c607b700 1 -- 172.21.2.102:0/4211540866 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13b400c420 con 0x7f13b4003b90 2021-02-10T11:34:25.343 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:25.323 7f13c607b700 1 -- 172.21.2.102:0/4211540866 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13b400c640 con 0x7f13b4009650 2021-02-10T11:34:25.344 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:25.323 7f13c607b700 1 -- 172.21.2.102:0/4211540866 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13b400c860 con 0x7f13b40068e0 2021-02-10T11:34:25.344 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:25.323 7f13c707d700 1 --2- 172.21.2.102:0/4211540866 >> v2:172.21.2.113:3300/0 conn(0x7f13c80d1870 0x7f13c81b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:34:25.344 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:25.323 7f13c687c700 1 --2- 172.21.2.102:0/4211540866 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f13c81b0ae0 0x7f13c8108940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:34:25.344 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:25.323 7f13c707d700 1 --2- 172.21.2.102:0/4211540866 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f13b4003b90 0x7f13b4006040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:25.344 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:25.324 7f13c687c700 1 --2- 172.21.2.102:0/4211540866 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f13b40068e0 0x7f13b4008db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:25.344 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:25.324 7f13c787e700 1 --2- 172.21.2.102:0/4211540866 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f13b4009650 0x7f13b400bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:25.344 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:25.324 7f13c707d700 1 -- 172.21.2.102:0/4211540866 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f13b40068e0 msgr2=0x7f13b4008db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:34:25.344 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:25.324 7f13c707d700 1 --2- 172.21.2.102:0/4211540866 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f13b40068e0 0x7f13b4008db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:25.344 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:25.324 7f13c707d700 1 -- 172.21.2.102:0/4211540866 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f13b4009650 msgr2=0x7f13b400bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:34:25.344 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:25.324 7f13c707d700 1 --2- 172.21.2.102:0/4211540866 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f13b4009650 0x7f13b400bb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:25.344 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:25.324 7f13c707d700 1 -- 172.21.2.102:0/4211540866 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13b400cb00 con 0x7f13b4003b90 2021-02-10T11:34:25.344 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:25.324 7f13c787e700 1 --2- 172.21.2.102:0/4211540866 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f13b4009650 0x7f13b400bb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:34:25.345 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:25.324 7f13c707d700 1 --2- 172.21.2.102:0/4211540866 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f13b4003b90 0x7f13b4006040 secure :-1 s=READY pgs=2544 cs=0 l=1 rev1=1 rx=0x7f13b0006f50 tx=0x7f13b004ecb0).ready entity=mon.0 client_cookie=e895a9c5d126e44c server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:25.345 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:25.324 7f13c607b700 1 -- 172.21.2.102:0/4211540866 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f13b0000960 con 0x7f13b4003b90 2021-02-10T11:34:25.345 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:25.324 7f13c607b700 1 -- 172.21.2.102:0/4211540866 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13b0000c90 con 0x7f13b4003b90 2021-02-10T11:34:25.345 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:25.324 7f13c607b700 1 -- 172.21.2.102:0/4211540866 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f13b00022b0 con 0x7f13b4003b90 2021-02-10T11:34:25.345 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:25.324 7f13cd3ca700 1 -- 172.21.2.102:0/4211540866 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f13b4003b90 msgr2=0x7f13b4006040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:25.345 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:25.324 7f13cd3ca700 1 --2- 172.21.2.102:0/4211540866 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f13b4003b90 0x7f13b4006040 secure :-1 s=READY pgs=2544 cs=0 l=1 rev1=1 rx=0x7f13b0006f50 tx=0x7f13b004ecb0).stop 2021-02-10T11:34:25.345 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:25.324 7f13c787e700 1 -- 172.21.2.102:0/4211540866 reap_dead start 2021-02-10T11:34:25.345 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:25.325 7f13cd3ca700 1 -- 172.21.2.102:0/4211540866 shutdown_connections 2021-02-10T11:34:25.345 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:25.325 7f13cd3ca700 1 -- 172.21.2.102:0/4211540866 >> 172.21.2.102:0/4211540866 conn(0x7f13c80ce420 msgr2=0x7f13c80d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:25.345 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:25.325 7f13cd3ca700 1 -- 172.21.2.102:0/4211540866 shutdown_connections 2021-02-10T11:34:25.345 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:25.325 7f13cd3ca700 1 -- 172.21.2.102:0/4211540866 wait complete. 2021-02-10T11:34:25.345 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:25.339 7f13cd3ca700 1 Processor -- start 2021-02-10T11:34:25.345 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:25.339 7f13cd3ca700 1 -- start start 2021-02-10T11:34:25.347 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:25.340 7f13cd3ca700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f13c81a79a0 0x7f13c81a7da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:25.347 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:25.340 7f13cd3ca700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f13c80bdb40 0x7f13c80bdf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:25.347 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:25.340 7f13cd3ca700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f13c8082ca0 0x7f13c80830a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:25.347 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:25.340 7f13cd3ca700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13c819d990 con 0x7f13c80bdb40 2021-02-10T11:34:25.347 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:25.340 7f13cd3ca700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13c819dbb0 con 0x7f13c81a79a0 2021-02-10T11:34:25.347 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:25.340 7f13cd3ca700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13c80836c0 con 0x7f13c8082ca0 2021-02-10T11:34:25.347 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:25.340 7f13c687c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f13c80bdb40 0x7f13c80bdf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:25.347 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:25.340 7f13c687c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f13c80bdb40 0x7f13c80bdf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:36098/0 (socket says 172.21.2.102:36098) 2021-02-10T11:34:25.347 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:25.340 7f13c687c700 1 -- 172.21.2.102:0/3655536880 learned_addr learned my addr 172.21.2.102:0/3655536880 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:34:25.348 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:25.340 7f13c687c700 1 -- 172.21.2.102:0/3655536880 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f13c8082ca0 msgr2=0x7f13c80830a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:25.348 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:25.340 7f13c687c700 1 --2- 172.21.2.102:0/3655536880 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f13c8082ca0 0x7f13c80830a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:25.348 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:25.340 7f13c687c700 1 -- 172.21.2.102:0/3655536880 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f13c81a79a0 msgr2=0x7f13c81a7da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:25.348 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:25.340 7f13c687c700 1 --2- 172.21.2.102:0/3655536880 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f13c81a79a0 0x7f13c81a7da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:25.348 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:25.340 7f13c687c700 1 -- 172.21.2.102:0/3655536880 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13c80838e0 con 0x7f13c80bdb40 2021-02-10T11:34:25.348 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:25.341 7f13c687c700 1 --2- 172.21.2.102:0/3655536880 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f13c80bdb40 0x7f13c80bdf40 secure :-1 s=READY pgs=2545 cs=0 l=1 rev1=1 rx=0x7f13b8006f50 tx=0x7f13b800b8e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:25.348 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:25.341 7f13bffff700 1 -- 172.21.2.102:0/3655536880 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f13b800bcd0 con 0x7f13c80bdb40 2021-02-10T11:34:25.348 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:25.341 7f13bffff700 1 -- 172.21.2.102:0/3655536880 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13b8000c90 con 0x7f13c80bdb40 2021-02-10T11:34:25.348 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:25.341 7f13bffff700 1 -- 172.21.2.102:0/3655536880 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f13b8002300 con 0x7f13c80bdb40 2021-02-10T11:34:25.349 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:25.341 7f13cd3ca700 1 -- 172.21.2.102:0/3655536880 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13c80838e0 con 0x7f13c80bdb40 2021-02-10T11:34:25.349 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:25.341 7f13cd3ca700 1 -- 172.21.2.102:0/3655536880 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13c8084420 con 0x7f13c80bdb40 2021-02-10T11:34:25.349 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:25.341 7f13bffff700 1 -- 172.21.2.102:0/3655536880 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f13b800a000 con 0x7f13c80bdb40 2021-02-10T11:34:25.349 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:25.341 7f13bffff700 1 --2- 172.21.2.102:0/3655536880 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f13ac0186c0 0x7f13ac01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:25.349 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:25.341 7f13bffff700 1 -- 172.21.2.102:0/3655536880 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(137..137 src has 1..137) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f13b80323e0 con 0x7f13c80bdb40 2021-02-10T11:34:25.349 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:25.342 7f13c707d700 1 --2- 172.21.2.102:0/3655536880 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f13ac0186c0 0x7f13ac01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:25.349 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:25.342 7f13c707d700 1 --2- 172.21.2.102:0/3655536880 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f13ac0186c0 0x7f13ac01abc0 crc :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:25.374 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:25.372 7f13cd3ca700 1 -- 172.21.2.102:0/3655536880 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13c8082a70 con 0x7f13c80bdb40 2021-02-10T11:34:25.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:25.375 7f13bffff700 1 -- 172.21.2.102:0/3655536880 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f13b8006580 con 0x7f13c80bdb40 2021-02-10T11:34:25.512 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:25.511 7f13cd3ca700 1 -- 172.21.2.102:0/3655536880 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"} v 0) v1 -- 0x7f13c8082a70 con 0x7f13c80bdb40 2021-02-10T11:34:25.698 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:25.697 7f13bffff700 1 -- 172.21.2.102:0/3655536880 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"}]=0 v116) v1 ==== 118+0+0 (secure 0 0 0) 0x7f13b800a000 con 0x7f13c80bdb40 2021-02-10T11:34:25.731 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:25.729 7f13cd3ca700 1 -- 172.21.2.102:0/3655536880 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f13ac0186c0 msgr2=0x7f13ac01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:25.731 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:25.729 7f13cd3ca700 1 --2- 172.21.2.102:0/3655536880 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f13ac0186c0 0x7f13ac01abc0 crc :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:25.731 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:25.729 7f13cd3ca700 1 -- 172.21.2.102:0/3655536880 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f13c80bdb40 msgr2=0x7f13c80bdf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:25.732 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:25.729 7f13cd3ca700 1 --2- 172.21.2.102:0/3655536880 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f13c80bdb40 0x7f13c80bdf40 secure :-1 s=READY pgs=2545 cs=0 l=1 rev1=1 rx=0x7f13b8006f50 tx=0x7f13b800b8e0).stop 2021-02-10T11:34:25.732 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:25.729 7f13cd3ca700 1 -- 172.21.2.102:0/3655536880 shutdown_connections 2021-02-10T11:34:25.732 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:25.729 7f13cd3ca700 1 --2- 172.21.2.102:0/3655536880 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f13c8082ca0 0x7f13c80830a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:25.732 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:25.729 7f13cd3ca700 1 --2- 172.21.2.102:0/3655536880 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f13c80bdb40 0x7f13c80bdf40 unknown :-1 s=CLOSED pgs=2545 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:25.732 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:25.729 7f13cd3ca700 1 --2- 172.21.2.102:0/3655536880 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f13ac0186c0 0x7f13ac01abc0 unknown :-1 s=CLOSED pgs=1125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:25.732 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:25.729 7f13cd3ca700 1 --2- 172.21.2.102:0/3655536880 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f13c81a79a0 0x7f13c81a7da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:25.732 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:25.729 7f13cd3ca700 1 -- 172.21.2.102:0/3655536880 >> 172.21.2.102:0/3655536880 conn(0x7f13c80bef00 msgr2=0x7f13c809f200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:25.732 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:25.730 7f13cd3ca700 1 -- 172.21.2.102:0/3655536880 shutdown_connections 2021-02-10T11:34:25.732 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:25.730 7f13cd3ca700 1 -- 172.21.2.102:0/3655536880 wait complete. 2021-02-10T11:34:25.739 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:34:26.169 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:26.150 7fac7688c700 1 Processor -- start 2021-02-10T11:34:26.169 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:26.150 7fac7688c700 1 -- start start 2021-02-10T11:34:26.169 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:26.150 7fac7688c700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fac700d1870 0x7fac701b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:26.169 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:26.150 7fac7688c700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fac701b0a10 0x7fac700cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:26.169 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:26.151 7fac7688c700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac700f0b10 con 0x7fac700d1c70 2021-02-10T11:34:26.170 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:26.151 7fac7688c700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac700e5ed0 con 0x7fac700d1870 2021-02-10T11:34:26.170 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:26.151 7fac7688c700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac700f0650 con 0x7fac701b0a10 2021-02-10T11:34:26.170 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:26.151 7fac74ea9700 1 --1- >> v1:172.21.2.113:6789/0 conn(0x7fac700d1c70 0x7fac700cf950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.113:6789/0 says I am v1:172.21.2.102:35650/0 (socket says 172.21.2.102:35650) 2021-02-10T11:34:26.170 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:26.151 7fac74ea9700 1 -- 172.21.2.102:0/226963424 learned_addr learned my addr 172.21.2.102:0/226963424 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:34:26.170 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:26.151 7fac6ffff700 1 --2- 172.21.2.102:0/226963424 >> v2:172.21.2.113:3300/0 conn(0x7fac700d1870 0x7fac701b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:26.170 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:26.151 7fac6f7fe700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fac701b0a10 0x7fac700cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:26.170 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:26.152 7fac6effd700 1 -- 172.21.2.102:0/226963424 <== mon.1 v1:172.21.2.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3222300859 0 0) 0x7fac68001440 con 0x7fac700d1c70 2021-02-10T11:34:26.170 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:26.152 7fac6f7fe700 1 -- 172.21.2.102:0/226963424 >> v2:172.21.2.113:3300/0 conn(0x7fac700d1870 msgr2=0x7fac701b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:34:26.170 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:26.152 7fac6f7fe700 1 --2- 172.21.2.102:0/226963424 >> v2:172.21.2.113:3300/0 conn(0x7fac700d1870 0x7fac701b03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:26.170 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:26.152 7fac6f7fe700 1 -- 172.21.2.102:0/226963424 >> v1:172.21.2.113:6789/0 conn(0x7fac700d1c70 legacy=0x7fac700cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:26.171 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:26.152 7fac6f7fe700 1 -- 172.21.2.102:0/226963424 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac700f0870 con 0x7fac701b0a10 2021-02-10T11:34:26.171 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:26.152 7fac6ffff700 1 --2- 172.21.2.102:0/226963424 >> v2:172.21.2.113:3300/0 conn(0x7fac700d1870 0x7fac701b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:34:26.171 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:26.152 7fac6f7fe700 1 --2- 172.21.2.102:0/226963424 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fac701b0a10 0x7fac700cf330 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 rx=0x7fac64006650 tx=0x7fac64009a80).ready entity=mon.2 client_cookie=cad586c865b4f001 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:26.171 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:26.153 7fac6effd700 1 -- 172.21.2.102:0/226963424 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fac640009b0 con 0x7fac701b0a10 2021-02-10T11:34:26.171 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:26.153 7fac6effd700 1 -- 172.21.2.102:0/226963424 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac6400c0c0 con 0x7fac701b0a10 2021-02-10T11:34:26.171 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:26.153 7fac6effd700 1 -- 172.21.2.102:0/226963424 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fac6400c5e0 con 0x7fac701b0a10 2021-02-10T11:34:26.171 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:26.153 7fac7688c700 1 -- 172.21.2.102:0/226963424 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fac701b0a10 msgr2=0x7fac700cf330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:26.171 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:26.153 7fac7688c700 1 --2- 172.21.2.102:0/226963424 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fac701b0a10 0x7fac700cf330 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 rx=0x7fac64006650 tx=0x7fac64009a80).stop 2021-02-10T11:34:26.171 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:26.153 7fac7688c700 1 -- 172.21.2.102:0/226963424 shutdown_connections 2021-02-10T11:34:26.171 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:26.153 7fac7688c700 1 --2- 172.21.2.102:0/226963424 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fac701b0a10 0x7fac700cf330 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:26.171 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:26.153 7fac7688c700 1 --2- 172.21.2.102:0/226963424 >> v2:172.21.2.113:3300/0 conn(0x7fac700d1870 0x7fac701b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:26.171 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:26.153 7fac7688c700 1 -- 172.21.2.102:0/226963424 >> 172.21.2.102:0/226963424 conn(0x7fac700ce450 msgr2=0x7fac700d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:26.171 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:26.153 7fac7688c700 1 -- 172.21.2.102:0/226963424 shutdown_connections 2021-02-10T11:34:26.172 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:26.153 7fac7688c700 1 -- 172.21.2.102:0/226963424 wait complete. 2021-02-10T11:34:26.173 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:26.168 7fac7688c700 1 Processor -- start 2021-02-10T11:34:26.173 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:26.168 7fac7688c700 1 -- start start 2021-02-10T11:34:26.173 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:26.168 7fac7688c700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fac700c33c0 0x7fac700ec230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:26.173 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:26.168 7fac7688c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fac700b5e30 0x7fac7019ec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:26.174 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:26.168 7fac7688c700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fac7019f260 0x7fac700ed010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:26.174 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:26.168 7fac7688c700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac700c3810 con 0x7fac700b5e30 2021-02-10T11:34:26.174 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:26.169 7fac7688c700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac700eca90 con 0x7fac700c33c0 2021-02-10T11:34:26.174 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:26.169 7fac7688c700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac700b2790 con 0x7fac7019f260 2021-02-10T11:34:26.174 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:26.169 7fac6f7fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fac700b5e30 0x7fac7019ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:26.174 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:26.169 7fac6f7fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fac700b5e30 0x7fac7019ec40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:36110/0 (socket says 172.21.2.102:36110) 2021-02-10T11:34:26.174 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:26.169 7fac6f7fe700 1 -- 172.21.2.102:0/3402421574 learned_addr learned my addr 172.21.2.102:0/3402421574 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:34:26.174 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:26.169 7fac74ea9700 1 --2- 172.21.2.102:0/3402421574 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fac7019f260 0x7fac700ed010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:26.174 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:26.169 7fac6ffff700 1 --2- 172.21.2.102:0/3402421574 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fac700c33c0 0x7fac700ec230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:26.174 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:26.169 7fac6f7fe700 1 -- 172.21.2.102:0/3402421574 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fac7019f260 msgr2=0x7fac700ed010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:26.174 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:26.169 7fac6f7fe700 1 --2- 172.21.2.102:0/3402421574 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fac7019f260 0x7fac700ed010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:26.174 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:26.169 7fac6f7fe700 1 -- 172.21.2.102:0/3402421574 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fac700c33c0 msgr2=0x7fac700ec230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:26.174 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:26.169 7fac6f7fe700 1 --2- 172.21.2.102:0/3402421574 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fac700c33c0 0x7fac700ec230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:26.175 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:26.169 7fac6f7fe700 1 -- 172.21.2.102:0/3402421574 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac7019f660 con 0x7fac700b5e30 2021-02-10T11:34:26.175 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:26.169 7fac6f7fe700 1 --2- 172.21.2.102:0/3402421574 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fac700b5e30 0x7fac7019ec40 secure :-1 s=READY pgs=2546 cs=0 l=1 rev1=1 rx=0x7fac64009630 tx=0x7fac6400b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:26.175 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:26.170 7fac6cff9700 1 -- 172.21.2.102:0/3402421574 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fac640093a0 con 0x7fac700b5e30 2021-02-10T11:34:26.175 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:26.170 7fac6cff9700 1 -- 172.21.2.102:0/3402421574 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac6400bd70 con 0x7fac700b5e30 2021-02-10T11:34:26.175 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:26.170 7fac6cff9700 1 -- 172.21.2.102:0/3402421574 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fac640054d0 con 0x7fac700b5e30 2021-02-10T11:34:26.175 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:26.170 7fac7688c700 1 -- 172.21.2.102:0/3402421574 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac7019f660 con 0x7fac700b5e30 2021-02-10T11:34:26.175 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:26.170 7fac7688c700 1 -- 172.21.2.102:0/3402421574 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac700ed630 con 0x7fac700b5e30 2021-02-10T11:34:26.175 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:26.170 7fac6cff9700 1 -- 172.21.2.102:0/3402421574 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fac6400a000 con 0x7fac700b5e30 2021-02-10T11:34:26.175 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:26.171 7fac6cff9700 1 --2- 172.21.2.102:0/3402421574 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fac500186c0 0x7fac5001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:26.175 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:26.171 7fac6cff9700 1 -- 172.21.2.102:0/3402421574 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(137..137 src has 1..137) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fac64032470 con 0x7fac700b5e30 2021-02-10T11:34:26.175 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:26.171 7fac6ffff700 1 --2- 172.21.2.102:0/3402421574 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fac500186c0 0x7fac5001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:26.175 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:26.171 7fac6ffff700 1 --2- 172.21.2.102:0/3402421574 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fac500186c0 0x7fac5001abc0 crc :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:26.202 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:26.200 7fac7688c700 1 -- 172.21.2.102:0/3402421574 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac700ec850 con 0x7fac700b5e30 2021-02-10T11:34:26.204 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:26.203 7fac6cff9700 1 -- 172.21.2.102:0/3402421574 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fac64020d80 con 0x7fac700b5e30 2021-02-10T11:34:26.333 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:26.332 7fac7688c700 1 -- 172.21.2.102:0/3402421574 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fac700ec850 con 0x7fac700b5e30 2021-02-10T11:34:26.334 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:26.332 7fac6cff9700 1 -- 172.21.2.102:0/3402421574 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 116 v116) v1 ==== 95+0+2108 (secure 0 0 0) 0x7fac64020d80 con 0x7fac700b5e30 2021-02-10T11:34:26.335 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:34:26.335 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":116,"default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":11466,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.102:6835/1274216855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6834","nonce":1274216855},{"type":"v1","addr":"172.21.2.102:6835","nonce":1274216855}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":112}],"filesystems":[{"mdsmap":{"epoch":116,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:34:23.684779","modified":"2021-02-10 11:34:25.693957","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6770},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6770":{"gid":6770,"name":"b","rank":0,"incarnation":113,"state":"up:active","state_seq":3,"addr":"172.21.2.113:6833/3752249621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6832","nonce":3752249621},{"type":"v1","addr":"172.21.2.113:6833","nonce":3752249621}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[29],"metadata_pool":28,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":14}]} 2021-02-10T11:34:26.335 INFO:teuthology.orchestra.run.gibba002.stderr:dumped fsmap epoch 116 2021-02-10T11:34:26.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:26.334 7fac7688c700 1 -- 172.21.2.102:0/3402421574 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fac500186c0 msgr2=0x7fac5001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:26.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:26.334 7fac7688c700 1 --2- 172.21.2.102:0/3402421574 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fac500186c0 0x7fac5001abc0 crc :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:26.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:26.334 7fac7688c700 1 -- 172.21.2.102:0/3402421574 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fac700b5e30 msgr2=0x7fac7019ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:26.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:26.334 7fac7688c700 1 --2- 172.21.2.102:0/3402421574 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fac700b5e30 0x7fac7019ec40 secure :-1 s=READY pgs=2546 cs=0 l=1 rev1=1 rx=0x7fac64009630 tx=0x7fac6400b980).stop 2021-02-10T11:34:26.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:26.334 7fac7688c700 1 -- 172.21.2.102:0/3402421574 shutdown_connections 2021-02-10T11:34:26.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:26.334 7fac7688c700 1 --2- 172.21.2.102:0/3402421574 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fac7019f260 0x7fac700ed010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:26.338 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:26.334 7fac7688c700 1 --2- 172.21.2.102:0/3402421574 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fac700b5e30 0x7fac7019ec40 unknown :-1 s=CLOSED pgs=2546 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:26.338 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:26.334 7fac7688c700 1 --2- 172.21.2.102:0/3402421574 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fac500186c0 0x7fac5001abc0 unknown :-1 s=CLOSED pgs=1126 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:26.338 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:26.334 7fac7688c700 1 --2- 172.21.2.102:0/3402421574 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fac700c33c0 0x7fac700ec230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:26.338 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:26.334 7fac7688c700 1 -- 172.21.2.102:0/3402421574 >> 172.21.2.102:0/3402421574 conn(0x7fac700ebe30 msgr2=0x7fac7008a880 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:26.338 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:26.334 7fac7688c700 1 -- 172.21.2.102:0/3402421574 shutdown_connections 2021-02-10T11:34:26.338 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:26.334 7fac7688c700 1 -- 172.21.2.102:0/3402421574 wait complete. 2021-02-10T11:34:26.344 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:34:26.774 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:26.755 7fbade879700 1 Processor -- start 2021-02-10T11:34:26.774 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:26.755 7fbade879700 1 -- start start 2021-02-10T11:34:26.774 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:26.755 7fbade879700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbad81b0870 0x7fbad80f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:26.775 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:26.756 7fbade879700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fbad80efe00 0x7fbad80f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:26.775 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:26.756 7fbade879700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbad81b0c70 con 0x7fbad81b0470 2021-02-10T11:34:26.775 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:26.756 7fbade879700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbad80f0b50 con 0x7fbad80efe00 2021-02-10T11:34:26.775 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:26.756 7fbade879700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbad80f0820 con 0x7fbad81b0870 2021-02-10T11:34:26.775 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:26.756 7fbad77fe700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbad81b0870 0x7fbad80f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:26.775 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:26.756 7fbadce96700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fbad80efe00 0x7fbad80f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:26.775 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:26.756 7fbad77fe700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbad81b0870 0x7fbad80f20e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.113:3301/0 says I am v2:172.21.2.102:41652/0 (socket says 172.21.2.102:41652) 2021-02-10T11:34:26.775 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:26.756 7fbad7fff700 1 --1- >> v1:172.21.2.113:6789/0 conn(0x7fbad81b0470 0x7fbad80ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.113:6789/0 says I am v1:172.21.2.102:35664/0 (socket says 172.21.2.102:35664) 2021-02-10T11:34:26.775 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:26.756 7fbad77fe700 1 -- 172.21.2.102:0/3433640101 learned_addr learned my addr 172.21.2.102:0/3433640101 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:34:26.775 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:26.756 7fbadce96700 1 --2- 172.21.2.102:0/3433640101 >> v2:172.21.2.113:3300/0 conn(0x7fbad80efe00 0x7fbad80f0200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.113:3300/0 says I am v2:172.21.2.102:39546/0 (socket says 172.21.2.102:39546) 2021-02-10T11:34:26.775 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:26.756 7fbad6ffd700 1 -- 172.21.2.102:0/3433640101 <== mon.1 v1:172.21.2.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1880738622 0 0) 0x7fbac8000c60 con 0x7fbad81b0470 2021-02-10T11:34:26.776 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:26.756 7fbad6ffd700 1 -- 172.21.2.102:0/3433640101 --> v1:172.21.2.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbac0002fe0 con 0x7fbad81b0470 2021-02-10T11:34:26.776 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:26.756 7fbad77fe700 1 -- 172.21.2.102:0/3433640101 >> v2:172.21.2.113:3300/0 conn(0x7fbad80efe00 msgr2=0x7fbad80f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:34:26.776 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:26.756 7fbad77fe700 1 --2- 172.21.2.102:0/3433640101 >> v2:172.21.2.113:3300/0 conn(0x7fbad80efe00 0x7fbad80f0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:26.776 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:26.756 7fbad77fe700 1 -- 172.21.2.102:0/3433640101 >> v1:172.21.2.113:6789/0 conn(0x7fbad81b0470 legacy=0x7fbad80ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:26.776 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:26.757 7fbad77fe700 1 -- 172.21.2.102:0/3433640101 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbad80e52e0 con 0x7fbad81b0870 2021-02-10T11:34:26.776 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:26.757 7fbadce96700 1 --2- 172.21.2.102:0/3433640101 >> v2:172.21.2.113:3300/0 conn(0x7fbad80efe00 0x7fbad80f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:34:26.776 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:26.757 7fbad77fe700 1 --2- 172.21.2.102:0/3433640101 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbad81b0870 0x7fbad80f20e0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0x7fbacc006700 tx=0x7fbacc009b80).ready entity=mon.2 client_cookie=7e6009a52cb44437 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:26.776 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:26.757 7fbad6ffd700 1 -- 172.21.2.102:0/3433640101 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbacc00c0c0 con 0x7fbad81b0870 2021-02-10T11:34:26.776 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:26.757 7fbad6ffd700 1 -- 172.21.2.102:0/3433640101 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbacc00c440 con 0x7fbad81b0870 2021-02-10T11:34:26.776 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:26.757 7fbad6ffd700 1 -- 172.21.2.102:0/3433640101 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbacc00c960 con 0x7fbad81b0870 2021-02-10T11:34:26.776 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:26.757 7fbade879700 1 -- 172.21.2.102:0/3433640101 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbad81b0870 msgr2=0x7fbad80f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:26.776 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:26.757 7fbade879700 1 --2- 172.21.2.102:0/3433640101 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbad81b0870 0x7fbad80f20e0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0x7fbacc006700 tx=0x7fbacc009b80).stop 2021-02-10T11:34:26.777 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:26.758 7fbade879700 1 -- 172.21.2.102:0/3433640101 shutdown_connections 2021-02-10T11:34:26.777 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:26.758 7fbade879700 1 --2- 172.21.2.102:0/3433640101 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbad81b0870 0x7fbad80f20e0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:26.777 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:26.758 7fbade879700 1 --2- 172.21.2.102:0/3433640101 >> v2:172.21.2.113:3300/0 conn(0x7fbad80efe00 0x7fbad80f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:26.777 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:26.758 7fbade879700 1 -- 172.21.2.102:0/3433640101 >> 172.21.2.102:0/3433640101 conn(0x7fbad80ce3e0 msgr2=0x7fbad80e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:26.777 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:26.758 7fbade879700 1 -- 172.21.2.102:0/3433640101 shutdown_connections 2021-02-10T11:34:26.777 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:26.758 7fbade879700 1 -- 172.21.2.102:0/3433640101 wait complete. 2021-02-10T11:34:26.777 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:26.773 7fbade879700 1 Processor -- start 2021-02-10T11:34:26.777 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:26.773 7fbade879700 1 -- start start 2021-02-10T11:34:26.778 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:26.773 7fbade879700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbad808ebb0 0x7fbad808efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:26.779 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:26.773 7fbade879700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbad80b65c0 0x7fbad80b69c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:26.779 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:26.773 7fbade879700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fbad80b6fe0 0x7fbad80b7bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:26.779 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:26.773 7fbade879700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbad80b76d0 con 0x7fbad808ebb0 2021-02-10T11:34:26.779 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:26.773 7fbade879700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbad808f5d0 con 0x7fbad80b6fe0 2021-02-10T11:34:26.779 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:26.773 7fbade879700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbad81a04d0 con 0x7fbad80b65c0 2021-02-10T11:34:26.779 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:26.773 7fbad7fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbad808ebb0 0x7fbad808efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:26.779 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:26.773 7fbad7fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbad808ebb0 0x7fbad808efb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:36122/0 (socket says 172.21.2.102:36122) 2021-02-10T11:34:26.779 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:26.773 7fbad7fff700 1 -- 172.21.2.102:0/610442640 learned_addr learned my addr 172.21.2.102:0/610442640 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:34:26.779 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:26.774 7fbad7fff700 1 -- 172.21.2.102:0/610442640 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbad80b65c0 msgr2=0x7fbad80b69c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:26.779 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:26.774 7fbad7fff700 1 --2- 172.21.2.102:0/610442640 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbad80b65c0 0x7fbad80b69c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:26.779 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:26.774 7fbad7fff700 1 -- 172.21.2.102:0/610442640 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fbad80b6fe0 msgr2=0x7fbad80b7bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:26.779 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:26.774 7fbad7fff700 1 --2- 172.21.2.102:0/610442640 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fbad80b6fe0 0x7fbad80b7bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:26.780 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:26.774 7fbad7fff700 1 -- 172.21.2.102:0/610442640 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbad80b81f0 con 0x7fbad808ebb0 2021-02-10T11:34:26.780 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:26.774 7fbad7fff700 1 --2- 172.21.2.102:0/610442640 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbad808ebb0 0x7fbad808efb0 secure :-1 s=READY pgs=2547 cs=0 l=1 rev1=1 rx=0x7fbac8001450 tx=0x7fbac800a930).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:26.780 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:26.774 7fbad4ff9700 1 -- 172.21.2.102:0/610442640 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbac800ad20 con 0x7fbad808ebb0 2021-02-10T11:34:26.780 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:26.774 7fbad4ff9700 1 -- 172.21.2.102:0/610442640 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbac8007590 con 0x7fbad808ebb0 2021-02-10T11:34:26.780 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:26.774 7fbad4ff9700 1 -- 172.21.2.102:0/610442640 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbac8004510 con 0x7fbad808ebb0 2021-02-10T11:34:26.780 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:26.774 7fbade879700 1 -- 172.21.2.102:0/610442640 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbad80b81f0 con 0x7fbad808ebb0 2021-02-10T11:34:26.780 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:26.774 7fbade879700 1 -- 172.21.2.102:0/610442640 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbad80b81f0 con 0x7fbad808ebb0 2021-02-10T11:34:26.780 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:26.775 7fbad4ff9700 1 -- 172.21.2.102:0/610442640 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fbac8004530 con 0x7fbad808ebb0 2021-02-10T11:34:26.780 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:26.775 7fbad4ff9700 1 --2- 172.21.2.102:0/610442640 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fbab8018730 0x7fbab801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:26.780 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:26.775 7fbad4ff9700 1 -- 172.21.2.102:0/610442640 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(137..137 src has 1..137) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fbac8032670 con 0x7fbad808ebb0 2021-02-10T11:34:26.780 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:26.775 7fbad77fe700 1 --2- 172.21.2.102:0/610442640 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fbab8018730 0x7fbab801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:26.780 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:26.775 7fbad77fe700 1 --2- 172.21.2.102:0/610442640 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fbab8018730 0x7fbab801ac10 crc :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:26.807 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:26.805 7fbade879700 1 -- 172.21.2.102:0/610442640 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbad8105a20 con 0x7fbad808ebb0 2021-02-10T11:34:26.809 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:26.808 7fbad4ff9700 1 -- 172.21.2.102:0/610442640 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fbac8004320 con 0x7fbad808ebb0 2021-02-10T11:34:26.934 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:26.932 7fbade879700 1 -- 172.21.2.102:0/610442640 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbad80f0b50 con 0x7fbad808ebb0 2021-02-10T11:34:26.934 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:26.933 7fbad4ff9700 1 -- 172.21.2.102:0/610442640 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v137) v1 ==== 74+0+14007 (secure 0 0 0) 0x7fbac8004530 con 0x7fbad808ebb0 2021-02-10T11:34:26.935 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:34:26.935 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":137,"fsid":"68db30ea-dbce-4aef-a6df-f01c6fea0513","created":"2021-02-10 11:03:06.331557","modified":"2021-02-10 11:34:24.687007","last_up_change":"2021-02-10 11:03:32.690562","last_in_change":"2021-02-10 11:03:32.690562","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":29,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:03:37.322519","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":28,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:34:21.649291","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"136","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":29,"pool_name":"cephfs_data","create_time":"2021-02-10 11:34:22.675483","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"137","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"be691b4d-a039-408a-af80-9d53f02b6f1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":134,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6818","nonce":36088},{"type":"v1","addr":"172.21.2.102:6820","nonce":36088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6822","nonce":36088},{"type":"v1","addr":"172.21.2.102:6824","nonce":36088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6830","nonce":36088},{"type":"v1","addr":"172.21.2.102:6832","nonce":36088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6826","nonce":36088},{"type":"v1","addr":"172.21.2.102:6828","nonce":36088}]},"public_addr":"172.21.2.102:6820/36088","cluster_addr":"172.21.2.102:6824/36088","heartbeat_back_addr":"172.21.2.102:6832/36088","heartbeat_front_addr":"172.21.2.102:6828/36088","state":["exists","up"]},{"osd":1,"uuid":"64e3ccc8-9c69-4d7c-865e-af8e83822635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":135,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6819","nonce":36087},{"type":"v1","addr":"172.21.2.102:6821","nonce":36087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6823","nonce":36087},{"type":"v1","addr":"172.21.2.102:6825","nonce":36087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6831","nonce":36087},{"type":"v1","addr":"172.21.2.102:6833","nonce":36087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6827","nonce":36087},{"type":"v1","addr":"172.21.2.102:6829","nonce":36087}]},"public_addr":"172.21.2.102:6821/36087","cluster_addr":"172.21.2.102:6825/36087","heartbeat_back_addr":"172.21.2.102:6833/36087","heartbeat_front_addr":"172.21.2.102:6829/36087","state":["exists","up"]},{"osd":2,"uuid":"153700af-747f-481f-bdf4-d6072d3a842f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6810","nonce":36089},{"type":"v1","addr":"172.21.2.102:6811","nonce":36089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6812","nonce":36089},{"type":"v1","addr":"172.21.2.102:6813","nonce":36089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6816","nonce":36089},{"type":"v1","addr":"172.21.2.102:6817","nonce":36089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6814","nonce":36089},{"type":"v1","addr":"172.21.2.102:6815","nonce":36089}]},"public_addr":"172.21.2.102:6811/36089","cluster_addr":"172.21.2.102:6813/36089","heartbeat_back_addr":"172.21.2.102:6817/36089","heartbeat_front_addr":"172.21.2.102:6815/36089","state":["exists","up"]},{"osd":3,"uuid":"cbc678f4-83a9-45e6-8830-2f7c232b1ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6802","nonce":36090},{"type":"v1","addr":"172.21.2.102:6803","nonce":36090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6804","nonce":36090},{"type":"v1","addr":"172.21.2.102:6805","nonce":36090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6808","nonce":36090},{"type":"v1","addr":"172.21.2.102:6809","nonce":36090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6806","nonce":36090},{"type":"v1","addr":"172.21.2.102:6807","nonce":36090}]},"public_addr":"172.21.2.102:6803/36090","cluster_addr":"172.21.2.102:6805/36090","heartbeat_back_addr":"172.21.2.102:6809/36090","heartbeat_front_addr":"172.21.2.102:6807/36090","state":["exists","up"]},{"osd":4,"uuid":"c01e4008-b258-4f10-afd2-9e7f37214351","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":135,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6808","nonce":36368},{"type":"v1","addr":"172.21.2.113:6811","nonce":36368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6814","nonce":36368},{"type":"v1","addr":"172.21.2.113:6818","nonce":36368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6826","nonce":36368},{"type":"v1","addr":"172.21.2.113:6828","nonce":36368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6820","nonce":36368},{"type":"v1","addr":"172.21.2.113:6824","nonce":36368}]},"public_addr":"172.21.2.113:6811/36368","cluster_addr":"172.21.2.113:6818/36368","heartbeat_back_addr":"172.21.2.113:6828/36368","heartbeat_front_addr":"172.21.2.113:6824/36368","state":["exists","up"]},{"osd":5,"uuid":"551f6007-daf8-4a98-8ee0-c2b16bcccb5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":134,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6802","nonce":36366},{"type":"v1","addr":"172.21.2.113:6805","nonce":36366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6807","nonce":36366},{"type":"v1","addr":"172.21.2.113:6810","nonce":36366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6819","nonce":36366},{"type":"v1","addr":"172.21.2.113:6822","nonce":36366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6812","nonce":36366},{"type":"v1","addr":"172.21.2.113:6815","nonce":36366}]},"public_addr":"172.21.2.113:6805/36366","cluster_addr":"172.21.2.113:6810/36366","heartbeat_back_addr":"172.21.2.113:6822/36366","heartbeat_front_addr":"172.21.2.113:6815/36366","state":["exists","up"]},{"osd":6,"uuid":"bf67c040-0e50-48be-b102-c3943bc9ebef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":135,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6800","nonce":36367},{"type":"v1","addr":"172.21.2.113:6801","nonce":36367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6803","nonce":36367},{"type":"v1","addr":"172.21.2.113:6804","nonce":36367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6813","nonce":36367},{"type":"v1","addr":"172.21.2.113:6816","nonce":36367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6806","nonce":36367},{"type":"v1","addr":"172.21.2.113:6809","nonce":36367}]},"public_addr":"172.21.2.113:6801/36367","cluster_addr":"172.21.2.113:6804/36367","heartbeat_back_addr":"172.21.2.113:6816/36367","heartbeat_front_addr":"172.21.2.113:6809/36367","state":["exists","up"]},{"osd":7,"uuid":"f5461f80-018d-428f-be89-2eb2bcbe7bb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":135,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6817","nonce":36365},{"type":"v1","addr":"172.21.2.113:6821","nonce":36365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6823","nonce":36365},{"type":"v1","addr":"172.21.2.113:6825","nonce":36365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6830","nonce":36365},{"type":"v1","addr":"172.21.2.113:6831","nonce":36365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6827","nonce":36365},{"type":"v1","addr":"172.21.2.113:6829","nonce":36365}]},"public_addr":"172.21.2.113:6821/36365","cluster_addr":"172.21.2.113:6825/36365","heartbeat_back_addr":"172.21.2.113:6831/36365","heartbeat_front_addr":"172.21.2.113:6829/36365","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:34:26.936 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:26.934 7fbade879700 1 -- 172.21.2.102:0/610442640 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fbab8018730 msgr2=0x7fbab801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:26.936 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:26.934 7fbade879700 1 --2- 172.21.2.102:0/610442640 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fbab8018730 0x7fbab801ac10 crc :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:26.936 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:26.934 7fbade879700 1 -- 172.21.2.102:0/610442640 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbad808ebb0 msgr2=0x7fbad808efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:26.936 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:26.934 7fbade879700 1 --2- 172.21.2.102:0/610442640 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbad808ebb0 0x7fbad808efb0 secure :-1 s=READY pgs=2547 cs=0 l=1 rev1=1 rx=0x7fbac8001450 tx=0x7fbac800a930).stop 2021-02-10T11:34:26.936 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:26.934 7fbade879700 1 -- 172.21.2.102:0/610442640 shutdown_connections 2021-02-10T11:34:26.937 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:26.934 7fbade879700 1 --2- 172.21.2.102:0/610442640 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fbab8018730 0x7fbab801ac10 unknown :-1 s=CLOSED pgs=1127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:26.937 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:26.934 7fbade879700 1 --2- 172.21.2.102:0/610442640 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fbad80b6fe0 0x7fbad80b7bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:26.937 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:26.934 7fbade879700 1 --2- 172.21.2.102:0/610442640 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbad80b65c0 0x7fbad80b69c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:26.937 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:26.934 7fbade879700 1 --2- 172.21.2.102:0/610442640 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbad808ebb0 0x7fbad808efb0 unknown :-1 s=CLOSED pgs=2547 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:26.937 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:26.934 7fbade879700 1 -- 172.21.2.102:0/610442640 >> 172.21.2.102:0/610442640 conn(0x7fbad809cb20 msgr2=0x7fbad80c3c80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:26.937 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:26.934 7fbade879700 1 -- 172.21.2.102:0/610442640 shutdown_connections 2021-02-10T11:34:26.937 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:26.934 7fbade879700 1 -- 172.21.2.102:0/610442640 wait complete. 2021-02-10T11:34:26.945 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:34:27.386 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:27.366 7fa61f783700 1 Processor -- start 2021-02-10T11:34:27.386 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:27.366 7fa61f783700 1 -- start start 2021-02-10T11:34:27.386 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:27.366 7fa61f783700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fa61810b000 0x7fa6180cf760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:27.386 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:27.366 7fa61f783700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6180fe1d0 con 0x7fa6180d1c80 2021-02-10T11:34:27.386 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:27.366 7fa61f783700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6181b0cc0 con 0x7fa6180d1880 2021-02-10T11:34:27.386 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:27.366 7fa61f783700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa618109a70 con 0x7fa61810b000 2021-02-10T11:34:27.386 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:27.367 7fa61cd9e700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fa6180d1c80 0x7fa6180cef70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:56584/0 (socket says 172.21.2.102:56584) 2021-02-10T11:34:27.386 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:27.367 7fa61cd9e700 1 -- 172.21.2.102:0/2534056137 learned_addr learned my addr 172.21.2.102:0/2534056137 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:34:27.386 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:27.367 7fa61dda0700 1 --2- 172.21.2.102:0/2534056137 >> v2:172.21.2.113:3300/0 conn(0x7fa61810b000 0x7fa6180cf760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:27.386 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:27.367 7fa60ffff700 1 -- 172.21.2.102:0/2534056137 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2412028855 0 0) 0x7fa6080010b0 con 0x7fa6180d1c80 2021-02-10T11:34:27.386 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:27.367 7fa60ffff700 1 -- 172.21.2.102:0/2534056137 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa604002fe0 con 0x7fa6180d1c80 2021-02-10T11:34:27.387 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:27.367 7fa60ffff700 1 -- 172.21.2.102:0/2534056137 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1963965261 0 0) 0x7fa6080010b0 con 0x7fa6180d1c80 2021-02-10T11:34:27.387 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:27.368 7fa60ffff700 1 -- 172.21.2.102:0/2534056137 >> v2:172.21.2.113:3300/0 conn(0x7fa61810b000 msgr2=0x7fa6180cf760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:34:27.387 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:27.368 7fa60ffff700 1 --2- 172.21.2.102:0/2534056137 >> v2:172.21.2.113:3300/0 conn(0x7fa61810b000 0x7fa6180cf760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:27.387 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:27.368 7fa60ffff700 1 -- 172.21.2.102:0/2534056137 >> v1:172.21.2.113:6789/0 conn(0x7fa6180d1880 legacy=0x7fa6181b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:27.387 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:27.368 7fa60ffff700 1 -- 172.21.2.102:0/2534056137 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa61810b400 con 0x7fa6180d1c80 2021-02-10T11:34:27.387 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:27.368 7fa60ffff700 1 -- 172.21.2.102:0/2534056137 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7fa6080015b0 con 0x7fa6180d1c80 2021-02-10T11:34:27.387 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:27.368 7fa60ffff700 1 -- 172.21.2.102:0/2534056137 >> v1:172.21.2.102:6789/0 conn(0x7fa6180d1c80 legacy=0x7fa6180cef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:27.387 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:27.368 7fa60ffff700 1 --2- 172.21.2.102:0/2534056137 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa604003b90 0x7fa604006040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:27.387 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:27.368 7fa60ffff700 1 --2- 172.21.2.102:0/2534056137 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa6040067d0 0x7fa604008cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:27.387 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:27.368 7fa60ffff700 1 --2- 172.21.2.102:0/2534056137 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa604009590 0x7fa60400bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:27.387 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:27.368 7fa60ffff700 1 -- 172.21.2.102:0/2534056137 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa60400c300 con 0x7fa6040067d0 2021-02-10T11:34:27.387 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:27.368 7fa60ffff700 1 -- 172.21.2.102:0/2534056137 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa60400c520 con 0x7fa604009590 2021-02-10T11:34:27.387 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:27.368 7fa60ffff700 1 -- 172.21.2.102:0/2534056137 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa60400c740 con 0x7fa604003b90 2021-02-10T11:34:27.387 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:27.368 7fa61cd9e700 1 --2- 172.21.2.102:0/2534056137 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa6040067d0 0x7fa604008cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:27.388 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:27.368 7fa61dda0700 1 --2- 172.21.2.102:0/2534056137 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa604009590 0x7fa60400bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:27.388 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:27.368 7fa61d59f700 1 --2- 172.21.2.102:0/2534056137 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa604003b90 0x7fa604006040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:27.388 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:27.368 7fa61cd9e700 1 -- 172.21.2.102:0/2534056137 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa604003b90 msgr2=0x7fa604006040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:34:27.388 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:27.368 7fa61cd9e700 1 --2- 172.21.2.102:0/2534056137 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa604003b90 0x7fa604006040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:27.388 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:27.368 7fa61cd9e700 1 -- 172.21.2.102:0/2534056137 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa604009590 msgr2=0x7fa60400bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:34:27.388 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:27.368 7fa61cd9e700 1 --2- 172.21.2.102:0/2534056137 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa604009590 0x7fa60400bab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:27.388 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:27.368 7fa61dda0700 1 -- 172.21.2.102:0/2534056137 reap_dead start 2021-02-10T11:34:27.388 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:27.368 7fa61cd9e700 1 -- 172.21.2.102:0/2534056137 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6040509b0 con 0x7fa6040067d0 2021-02-10T11:34:27.388 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:27.368 7fa61cd9e700 1 --2- 172.21.2.102:0/2534056137 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa6040067d0 0x7fa604008cb0 secure :-1 s=READY pgs=2549 cs=0 l=1 rev1=1 rx=0x7fa608007560 tx=0x7fa608007c60).ready entity=mon.0 client_cookie=1177b1334c46e6f5 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:27.388 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:27.368 7fa60ffff700 1 -- 172.21.2.102:0/2534056137 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa608001010 con 0x7fa6040067d0 2021-02-10T11:34:27.388 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:27.368 7fa60ffff700 1 -- 172.21.2.102:0/2534056137 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa60800a310 con 0x7fa6040067d0 2021-02-10T11:34:27.388 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:27.368 7fa60ffff700 1 -- 172.21.2.102:0/2534056137 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa60800a760 con 0x7fa6040067d0 2021-02-10T11:34:27.388 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:27.368 7fa61f783700 1 -- 172.21.2.102:0/2534056137 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa6040067d0 msgr2=0x7fa604008cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:27.389 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:27.368 7fa61f783700 1 --2- 172.21.2.102:0/2534056137 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa6040067d0 0x7fa604008cb0 secure :-1 s=READY pgs=2549 cs=0 l=1 rev1=1 rx=0x7fa608007560 tx=0x7fa608007c60).stop 2021-02-10T11:34:27.389 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:27.369 7fa61f783700 1 -- 172.21.2.102:0/2534056137 shutdown_connections 2021-02-10T11:34:27.389 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:27.369 7fa61f783700 1 --2- 172.21.2.102:0/2534056137 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa6040067d0 0x7fa604008cb0 unknown :-1 s=CLOSED pgs=2549 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:27.389 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:27.369 7fa61f783700 1 -- 172.21.2.102:0/2534056137 >> 172.21.2.102:0/2534056137 conn(0x7fa6180ce430 msgr2=0x7fa6180d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:27.389 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:27.369 7fa61f783700 1 -- 172.21.2.102:0/2534056137 shutdown_connections 2021-02-10T11:34:27.389 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:27.369 7fa61f783700 1 -- 172.21.2.102:0/2534056137 wait complete. 2021-02-10T11:34:27.389 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:27.384 7fa61f783700 1 Processor -- start 2021-02-10T11:34:27.389 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:27.384 7fa61f783700 1 -- start start 2021-02-10T11:34:27.390 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:27.384 7fa61f783700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa6180bf400 0x7fa6180c2850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:27.391 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:27.384 7fa61f783700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa6180bf800 0x7fa618099be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:27.391 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:27.384 7fa61f783700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa61809a200 0x7fa618189890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:27.391 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:27.384 7fa61f783700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6180c22d0 con 0x7fa61809a200 2021-02-10T11:34:27.391 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:27.384 7fa61f783700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6180c2e70 con 0x7fa6180bf400 2021-02-10T11:34:27.391 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:27.384 7fa61f783700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6180c3050 con 0x7fa6180bf800 2021-02-10T11:34:27.391 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:27.385 7fa61dda0700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa61809a200 0x7fa618189890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:27.391 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:27.385 7fa61dda0700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa61809a200 0x7fa618189890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:36142/0 (socket says 172.21.2.102:36142) 2021-02-10T11:34:27.391 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:27.385 7fa61dda0700 1 -- 172.21.2.102:0/1173659343 learned_addr learned my addr 172.21.2.102:0/1173659343 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:34:27.391 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:27.385 7fa61cd9e700 1 --2- 172.21.2.102:0/1173659343 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa6180bf800 0x7fa618099be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:27.391 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:27.385 7fa61d59f700 1 --2- 172.21.2.102:0/1173659343 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa6180bf400 0x7fa6180c2850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:27.391 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:27.385 7fa61dda0700 1 -- 172.21.2.102:0/1173659343 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa6180bf800 msgr2=0x7fa618099be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:27.391 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:27.385 7fa61dda0700 1 --2- 172.21.2.102:0/1173659343 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa6180bf800 0x7fa618099be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:27.392 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:27.385 7fa61dda0700 1 -- 172.21.2.102:0/1173659343 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa6180bf400 msgr2=0x7fa6180c2850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:27.392 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:27.385 7fa61dda0700 1 --2- 172.21.2.102:0/1173659343 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa6180bf400 0x7fa6180c2850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:27.392 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:27.385 7fa61dda0700 1 -- 172.21.2.102:0/1173659343 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa618189eb0 con 0x7fa61809a200 2021-02-10T11:34:27.392 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:27.385 7fa61d59f700 1 --2- 172.21.2.102:0/1173659343 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa6180bf400 0x7fa6180c2850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:34:27.392 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:27.385 7fa61dda0700 1 --2- 172.21.2.102:0/1173659343 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa61809a200 0x7fa618189890 secure :-1 s=READY pgs=2550 cs=0 l=1 rev1=1 rx=0x7fa61000b970 tx=0x7fa61000bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:27.392 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:27.385 7fa60dffb700 1 -- 172.21.2.102:0/1173659343 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa610000f90 con 0x7fa61809a200 2021-02-10T11:34:27.392 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:27.386 7fa60dffb700 1 -- 172.21.2.102:0/1173659343 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa610003030 con 0x7fa61809a200 2021-02-10T11:34:27.392 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:27.386 7fa60dffb700 1 -- 172.21.2.102:0/1173659343 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa6100013e0 con 0x7fa61809a200 2021-02-10T11:34:27.392 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:27.386 7fa61f783700 1 -- 172.21.2.102:0/1173659343 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa618189eb0 con 0x7fa61809a200 2021-02-10T11:34:27.392 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:27.386 7fa61f783700 1 -- 172.21.2.102:0/1173659343 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa618189eb0 con 0x7fa61809a200 2021-02-10T11:34:27.392 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:27.386 7fa60dffb700 1 -- 172.21.2.102:0/1173659343 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa6100011f0 con 0x7fa61809a200 2021-02-10T11:34:27.392 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:27.386 7fa60dffb700 1 --2- 172.21.2.102:0/1173659343 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa5f80186d0 0x7fa5f801abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:27.392 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:27.386 7fa60dffb700 1 -- 172.21.2.102:0/1173659343 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(137..137 src has 1..137) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fa6100329f0 con 0x7fa61809a200 2021-02-10T11:34:27.393 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:27.386 7fa61d59f700 1 --2- 172.21.2.102:0/1173659343 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa5f80186d0 0x7fa5f801abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:27.393 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:27.387 7fa61d59f700 1 --2- 172.21.2.102:0/1173659343 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa5f80186d0 0x7fa5f801abd0 crc :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:27.418 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:27.416 7fa61f783700 1 -- 172.21.2.102:0/1173659343 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa618082980 con 0x7fa61809a200 2021-02-10T11:34:27.420 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:27.419 7fa60dffb700 1 -- 172.21.2.102:0/1173659343 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fa610006720 con 0x7fa61809a200 2021-02-10T11:34:27.549 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:27.548 7fa61f783700 1 -- 172.21.2.102:0/1173659343 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7fa618082980 con 0x7fa61809a200 2021-02-10T11:34:27.553 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:27.551 7fa60dffb700 1 -- 172.21.2.102:0/1173659343 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.0 v77) v1 ==== 184+0+0 (secure 0 0 0) 0x7fa61001f8b0 con 0x7fa61809a200 2021-02-10T11:34:27.557 INFO:teuthology.orchestra.run.gibba002.stderr:updated caps for client.0 2021-02-10T11:34:27.558 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:27.557 7fa61f783700 1 -- 172.21.2.102:0/1173659343 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa5f80186d0 msgr2=0x7fa5f801abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:27.558 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:27.557 7fa61f783700 1 --2- 172.21.2.102:0/1173659343 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa5f80186d0 0x7fa5f801abd0 crc :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:27.558 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:27.557 7fa61f783700 1 -- 172.21.2.102:0/1173659343 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa61809a200 msgr2=0x7fa618189890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:27.559 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:27.557 7fa61f783700 1 --2- 172.21.2.102:0/1173659343 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa61809a200 0x7fa618189890 secure :-1 s=READY pgs=2550 cs=0 l=1 rev1=1 rx=0x7fa61000b970 tx=0x7fa61000bd10).stop 2021-02-10T11:34:27.559 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:27.557 7fa61f783700 1 -- 172.21.2.102:0/1173659343 shutdown_connections 2021-02-10T11:34:27.559 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:27.557 7fa61f783700 1 --2- 172.21.2.102:0/1173659343 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa61809a200 0x7fa618189890 unknown :-1 s=CLOSED pgs=2550 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:27.559 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:27.557 7fa61f783700 1 --2- 172.21.2.102:0/1173659343 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa6180bf800 0x7fa618099be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:27.559 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:27.557 7fa61f783700 1 --2- 172.21.2.102:0/1173659343 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa5f80186d0 0x7fa5f801abd0 unknown :-1 s=CLOSED pgs=1128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:27.559 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:27.557 7fa61f783700 1 --2- 172.21.2.102:0/1173659343 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa6180bf400 0x7fa6180c2850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:27.559 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:27.557 7fa61f783700 1 -- 172.21.2.102:0/1173659343 >> 172.21.2.102:0/1173659343 conn(0x7fa61809eba0 msgr2=0x7fa6180b5ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:27.560 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:27.557 7fa61f783700 1 -- 172.21.2.102:0/1173659343 shutdown_connections 2021-02-10T11:34:27.560 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:27.557 7fa61f783700 1 -- 172.21.2.102:0/1173659343 wait complete. 2021-02-10T11:34:27.567 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.1 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:34:27.998 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:27.980 7fda36534700 1 Processor -- start 2021-02-10T11:34:27.999 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:27.980 7fda36534700 1 -- start start 2021-02-10T11:34:27.999 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:27.980 7fda36534700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fda300d1880 0x7fda301b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:27.999 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:27.980 7fda36534700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fda300cff80 0x7fda30109140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:27.999 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:27.980 7fda36534700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda30109a70 con 0x7fda301b0af0 2021-02-10T11:34:27.999 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:27.980 7fda36534700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda30109760 con 0x7fda300cff80 2021-02-10T11:34:27.999 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:27.980 7fda36534700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda30088070 con 0x7fda300d1880 2021-02-10T11:34:27.999 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:27.981 7fda34b51700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fda300cff80 0x7fda30109140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:27.999 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:27.981 7fda34b51700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fda300cff80 0x7fda30109140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:36148/0 (socket says 172.21.2.102:36148) 2021-02-10T11:34:27.999 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:27.981 7fda34b51700 1 -- 172.21.2.102:0/2336780572 learned_addr learned my addr 172.21.2.102:0/2336780572 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:34:27.999 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:27.981 7fda34b51700 1 -- 172.21.2.102:0/2336780572 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fda300d1880 msgr2=0x7fda301b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:34:27.999 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:27.981 7fda34b51700 1 --2- 172.21.2.102:0/2336780572 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fda300d1880 0x7fda301b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:27.999 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:27.981 7fda34b51700 1 -- 172.21.2.102:0/2336780572 >> v1:172.21.2.113:6789/0 conn(0x7fda301b0af0 legacy=0x7fda30108950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:34:28.000 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:27.981 7fda34b51700 1 -- 172.21.2.102:0/2336780572 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda30088260 con 0x7fda300cff80 2021-02-10T11:34:28.000 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:27.981 7fda34b51700 1 --2- 172.21.2.102:0/2336780572 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fda300cff80 0x7fda30109140 secure :-1 s=READY pgs=2551 cs=0 l=1 rev1=1 rx=0x7fda28006f00 tx=0x7fda28009f10).ready entity=mon.0 client_cookie=14df5195cf4dae6c server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:28.000 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:27.981 7fda2effd700 1 -- 172.21.2.102:0/2336780572 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fda2800b3b0 con 0x7fda300cff80 2021-02-10T11:34:28.000 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:27.981 7fda2effd700 1 -- 172.21.2.102:0/2336780572 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda2800b730 con 0x7fda300cff80 2021-02-10T11:34:28.000 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:27.981 7fda2effd700 1 -- 172.21.2.102:0/2336780572 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fda2800bc50 con 0x7fda300cff80 2021-02-10T11:34:28.000 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:27.981 7fda36534700 1 -- 172.21.2.102:0/2336780572 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fda300cff80 msgr2=0x7fda30109140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:28.000 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:27.981 7fda36534700 1 --2- 172.21.2.102:0/2336780572 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fda300cff80 0x7fda30109140 secure :-1 s=READY pgs=2551 cs=0 l=1 rev1=1 rx=0x7fda28006f00 tx=0x7fda28009f10).stop 2021-02-10T11:34:28.000 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:27.982 7fda36534700 1 -- 172.21.2.102:0/2336780572 shutdown_connections 2021-02-10T11:34:28.000 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:27.982 7fda36534700 1 --2- 172.21.2.102:0/2336780572 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fda300cff80 0x7fda30109140 unknown :-1 s=CLOSED pgs=2551 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:28.000 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:27.982 7fda36534700 1 --2- 172.21.2.102:0/2336780572 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fda300d1880 0x7fda301b04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:28.000 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:27.982 7fda36534700 1 -- 172.21.2.102:0/2336780572 >> 172.21.2.102:0/2336780572 conn(0x7fda300ce430 msgr2=0x7fda300d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:28.000 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:27.982 7fda36534700 1 -- 172.21.2.102:0/2336780572 shutdown_connections 2021-02-10T11:34:28.001 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:27.982 7fda36534700 1 -- 172.21.2.102:0/2336780572 wait complete. 2021-02-10T11:34:28.001 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:27.997 7fda36534700 1 Processor -- start 2021-02-10T11:34:28.002 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:27.997 7fda36534700 1 -- start start 2021-02-10T11:34:28.002 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:27.997 7fda36534700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fda30086520 0x7fda301a1cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:28.002 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:27.997 7fda36534700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fda30086920 0x7fda30187e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:28.002 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:27.997 7fda36534700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fda301a22d0 0x7fda301884a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:28.002 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:27.997 7fda36534700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda30099600 con 0x7fda30086520 2021-02-10T11:34:28.002 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:27.997 7fda36534700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda300999f0 con 0x7fda301a22d0 2021-02-10T11:34:28.002 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:27.997 7fda36534700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda30188ac0 con 0x7fda30086920 2021-02-10T11:34:28.002 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:27.998 7fda2ffff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fda30086520 0x7fda301a1cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:28.003 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:27.998 7fda2ffff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fda30086520 0x7fda301a1cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:36152/0 (socket says 172.21.2.102:36152) 2021-02-10T11:34:28.003 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:27.998 7fda2ffff700 1 -- 172.21.2.102:0/3636753887 learned_addr learned my addr 172.21.2.102:0/3636753887 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:34:28.003 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:27.998 7fda2ffff700 1 -- 172.21.2.102:0/3636753887 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fda30086920 msgr2=0x7fda30187e80 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:34:28.003 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:27.998 7fda2ffff700 1 --2- 172.21.2.102:0/3636753887 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fda30086920 0x7fda30187e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:28.003 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:27.998 7fda2ffff700 1 -- 172.21.2.102:0/3636753887 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fda301a22d0 msgr2=0x7fda301884a0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:34:28.003 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:27.998 7fda2ffff700 1 --2- 172.21.2.102:0/3636753887 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fda301a22d0 0x7fda301884a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:28.003 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:27.998 7fda2ffff700 1 -- 172.21.2.102:0/3636753887 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda30188ce0 con 0x7fda30086520 2021-02-10T11:34:28.003 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:27.998 7fda2ffff700 1 --2- 172.21.2.102:0/3636753887 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fda30086520 0x7fda301a1cb0 secure :-1 s=READY pgs=2552 cs=0 l=1 rev1=1 rx=0x7fda20006f00 tx=0x7fda2000b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:28.003 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:27.998 7fda2cff9700 1 -- 172.21.2.102:0/3636753887 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fda20006b90 con 0x7fda30086520 2021-02-10T11:34:28.003 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:27.998 7fda2cff9700 1 -- 172.21.2.102:0/3636753887 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda200063d0 con 0x7fda30086520 2021-02-10T11:34:28.003 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:27.998 7fda2cff9700 1 -- 172.21.2.102:0/3636753887 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fda20000c90 con 0x7fda30086520 2021-02-10T11:34:28.003 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:27.998 7fda36534700 1 -- 172.21.2.102:0/3636753887 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda30188ce0 con 0x7fda30086520 2021-02-10T11:34:28.003 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:27.998 7fda36534700 1 -- 172.21.2.102:0/3636753887 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda30189270 con 0x7fda30086520 2021-02-10T11:34:28.004 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:28.000 7fda2cff9700 1 -- 172.21.2.102:0/3636753887 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fda20007000 con 0x7fda30086520 2021-02-10T11:34:28.004 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:28.000 7fda2cff9700 1 --2- 172.21.2.102:0/3636753887 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fda100186c0 0x7fda1001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:28.004 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:28.000 7fda2cff9700 1 -- 172.21.2.102:0/3636753887 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(137..137 src has 1..137) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fda20032540 con 0x7fda30086520 2021-02-10T11:34:28.004 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:28.000 7fda2f7fe700 1 --2- 172.21.2.102:0/3636753887 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fda100186c0 0x7fda1001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:28.004 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:28.000 7fda2f7fe700 1 --2- 172.21.2.102:0/3636753887 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fda100186c0 0x7fda1001abc0 crc :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:28.032 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:28.030 7fda36534700 1 -- 172.21.2.102:0/3636753887 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda300fe1d0 con 0x7fda30086520 2021-02-10T11:34:28.034 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:28.034 7fda2cff9700 1 -- 172.21.2.102:0/3636753887 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fda20000960 con 0x7fda30086520 2021-02-10T11:34:28.164 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:28.162 7fda36534700 1 -- 172.21.2.102:0/3636753887 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7fda300999a0 con 0x7fda30086520 2021-02-10T11:34:28.167 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:28.166 7fda2cff9700 1 -- 172.21.2.102:0/3636753887 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.1 v78) v1 ==== 184+0+0 (secure 0 0 0) 0x7fda2001f4e0 con 0x7fda30086520 2021-02-10T11:34:28.171 INFO:teuthology.orchestra.run.gibba002.stderr:updated caps for client.1 2021-02-10T11:34:28.173 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:28.172 7fda36534700 1 -- 172.21.2.102:0/3636753887 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fda100186c0 msgr2=0x7fda1001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:28.173 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:28.172 7fda36534700 1 --2- 172.21.2.102:0/3636753887 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fda100186c0 0x7fda1001abc0 crc :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:28.173 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:28.172 7fda36534700 1 -- 172.21.2.102:0/3636753887 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fda30086520 msgr2=0x7fda301a1cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:28.173 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:28.172 7fda36534700 1 --2- 172.21.2.102:0/3636753887 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fda30086520 0x7fda301a1cb0 secure :-1 s=READY pgs=2552 cs=0 l=1 rev1=1 rx=0x7fda20006f00 tx=0x7fda2000b940).stop 2021-02-10T11:34:28.173 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:28.172 7fda36534700 1 -- 172.21.2.102:0/3636753887 shutdown_connections 2021-02-10T11:34:28.173 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:28.172 7fda36534700 1 --2- 172.21.2.102:0/3636753887 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fda100186c0 0x7fda1001abc0 unknown :-1 s=CLOSED pgs=1129 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:28.174 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:28.172 7fda36534700 1 --2- 172.21.2.102:0/3636753887 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fda301a22d0 0x7fda301884a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:28.174 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:28.172 7fda36534700 1 --2- 172.21.2.102:0/3636753887 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fda30086920 0x7fda30187e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:28.174 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:28.172 7fda36534700 1 --2- 172.21.2.102:0/3636753887 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fda30086520 0x7fda301a1cb0 unknown :-1 s=CLOSED pgs=2552 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:28.174 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:28.172 7fda36534700 1 -- 172.21.2.102:0/3636753887 >> 172.21.2.102:0/3636753887 conn(0x7fda3019f880 msgr2=0x7fda3008f890 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:28.174 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:28.172 7fda36534700 1 -- 172.21.2.102:0/3636753887 shutdown_connections 2021-02-10T11:34:28.174 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:28.172 7fda36534700 1 -- 172.21.2.102:0/3636753887 wait complete. 2021-02-10T11:34:28.181 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.2 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:34:28.621 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:28.603 7f389d9ca700 1 Processor -- start 2021-02-10T11:34:28.669 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:28.603 7f389d9ca700 1 -- start start 2021-02-10T11:34:28.669 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:28.604 7f389d9ca700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f38980d1880 0x7f38981b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:28.669 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:28.604 7f389d9ca700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38980fe1d0 con 0x7f38980cff80 2021-02-10T11:34:28.669 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:28.604 7f389d9ca700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3898109a70 con 0x7f38981b0af0 2021-02-10T11:34:28.670 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:28.604 7f389d9ca700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3898088070 con 0x7f38980d1880 2021-02-10T11:34:28.670 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:28.604 7f3897574700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f38980d1880 0x7f38981b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:28.670 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:28.604 7f3897574700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f38980d1880 0x7f38981b04d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:36160/0 (socket says 172.21.2.102:36160) 2021-02-10T11:34:28.670 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:28.604 7f3897574700 1 -- 172.21.2.102:0/2646769566 learned_addr learned my addr 172.21.2.102:0/2646769566 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:34:28.670 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:28.604 7f3897d75700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f38980cff80 0x7f3898109140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:56616/0 (socket says 172.21.2.102:56616) 2021-02-10T11:34:28.670 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:28.604 7f3897574700 1 -- 172.21.2.102:0/2646769566 >> v1:172.21.2.113:6789/0 conn(0x7f38981b0af0 legacy=0x7f3898108950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:34:28.670 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:28.604 7f3897574700 1 -- 172.21.2.102:0/2646769566 >> v1:172.21.2.102:6789/0 conn(0x7f38980cff80 legacy=0x7f3898109140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:28.670 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:28.604 7f3897574700 1 -- 172.21.2.102:0/2646769566 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3898088260 con 0x7f38980d1880 2021-02-10T11:34:28.670 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:28.604 7f3897574700 1 --2- 172.21.2.102:0/2646769566 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f38980d1880 0x7f38981b04d0 secure :-1 s=READY pgs=2554 cs=0 l=1 rev1=1 rx=0x7f3888006700 tx=0x7f3888009b80).ready entity=mon.0 client_cookie=f65a98dd126d15b7 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:28.670 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:28.605 7f3896572700 1 -- 172.21.2.102:0/2646769566 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f388800c0c0 con 0x7f38980d1880 2021-02-10T11:34:28.670 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:28.605 7f3896572700 1 -- 172.21.2.102:0/2646769566 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f388800c340 con 0x7f38980d1880 2021-02-10T11:34:28.670 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:28.605 7f3896572700 1 -- 172.21.2.102:0/2646769566 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f388800c860 con 0x7f38980d1880 2021-02-10T11:34:28.671 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:28.605 7f389d9ca700 1 -- 172.21.2.102:0/2646769566 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f38980d1880 msgr2=0x7f38981b04d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:28.671 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:28.605 7f389d9ca700 1 --2- 172.21.2.102:0/2646769566 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f38980d1880 0x7f38981b04d0 secure :-1 s=READY pgs=2554 cs=0 l=1 rev1=1 rx=0x7f3888006700 tx=0x7f3888009b80).stop 2021-02-10T11:34:28.671 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:28.605 7f389d9ca700 1 -- 172.21.2.102:0/2646769566 shutdown_connections 2021-02-10T11:34:28.671 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:28.605 7f389d9ca700 1 --2- 172.21.2.102:0/2646769566 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f38980d1880 0x7f38981b04d0 unknown :-1 s=CLOSED pgs=2554 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:28.671 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:28.605 7f389d9ca700 1 -- 172.21.2.102:0/2646769566 >> 172.21.2.102:0/2646769566 conn(0x7f38980ce430 msgr2=0x7f38980d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:28.671 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:28.605 7f389d9ca700 1 -- 172.21.2.102:0/2646769566 shutdown_connections 2021-02-10T11:34:28.671 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:28.605 7f389d9ca700 1 -- 172.21.2.102:0/2646769566 wait complete. 2021-02-10T11:34:28.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:28.620 7f389d9ca700 1 Processor -- start 2021-02-10T11:34:28.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:28.620 7f389d9ca700 1 -- start start 2021-02-10T11:34:28.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:28.620 7f389d9ca700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f38980a4db0 0x7f38980b10b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:28.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:28.620 7f389d9ca700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f38980a7500 0x7f38980a7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:28.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:28.620 7f389d9ca700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f38980b1cd0 0x7f38980b20d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:28.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:28.620 7f389d9ca700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38980b1750 con 0x7f38980a4db0 2021-02-10T11:34:28.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:28.620 7f389d9ca700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38980a7900 con 0x7f38980a7500 2021-02-10T11:34:28.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:28.620 7f389d9ca700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38980b26f0 con 0x7f38980b1cd0 2021-02-10T11:34:28.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:28.621 7f3897574700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f38980a4db0 0x7f38980b10b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:28.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:28.621 7f3897574700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f38980a4db0 0x7f38980b10b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:36168/0 (socket says 172.21.2.102:36168) 2021-02-10T11:34:28.674 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:28.621 7f3897574700 1 -- 172.21.2.102:0/2490665131 learned_addr learned my addr 172.21.2.102:0/2490665131 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:34:28.674 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:28.621 7f3897574700 1 -- 172.21.2.102:0/2490665131 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f38980b1cd0 msgr2=0x7f38980b20d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:28.674 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:28.621 7f3897574700 1 --2- 172.21.2.102:0/2490665131 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f38980b1cd0 0x7f38980b20d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:28.674 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:28.621 7f3897574700 1 -- 172.21.2.102:0/2490665131 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f38980a7500 msgr2=0x7f38980a7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:28.674 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:28.621 7f3897574700 1 --2- 172.21.2.102:0/2490665131 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f38980a7500 0x7f38980a7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:28.674 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:28.621 7f3897574700 1 -- 172.21.2.102:0/2490665131 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38980b2910 con 0x7f38980a4db0 2021-02-10T11:34:28.674 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:28.621 7f3897574700 1 --2- 172.21.2.102:0/2490665131 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f38980a4db0 0x7f38980b10b0 secure :-1 s=READY pgs=2555 cs=0 l=1 rev1=1 rx=0x7f388800b5e0 tx=0x7f388800b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:28.674 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:28.621 7f388ffff700 1 -- 172.21.2.102:0/2490665131 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f388800bd70 con 0x7f38980a4db0 2021-02-10T11:34:28.674 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:28.621 7f388ffff700 1 -- 172.21.2.102:0/2490665131 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3888002030 con 0x7f38980a4db0 2021-02-10T11:34:28.674 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:28.621 7f388ffff700 1 -- 172.21.2.102:0/2490665131 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3888002550 con 0x7f38980a4db0 2021-02-10T11:34:28.674 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:28.621 7f389d9ca700 1 -- 172.21.2.102:0/2490665131 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38980b2910 con 0x7f38980a4db0 2021-02-10T11:34:28.674 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:28.622 7f389d9ca700 1 -- 172.21.2.102:0/2490665131 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38980b2910 con 0x7f38980a4db0 2021-02-10T11:34:28.675 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:28.622 7f388ffff700 1 -- 172.21.2.102:0/2490665131 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f3888000c90 con 0x7f38980a4db0 2021-02-10T11:34:28.675 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:28.622 7f388ffff700 1 --2- 172.21.2.102:0/2490665131 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f387c0186c0 0x7f387c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:28.675 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:28.622 7f388ffff700 1 -- 172.21.2.102:0/2490665131 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(137..137 src has 1..137) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f38880325a0 con 0x7f38980a4db0 2021-02-10T11:34:28.675 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:28.622 7f3896d73700 1 --2- 172.21.2.102:0/2490665131 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f387c0186c0 0x7f387c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:28.675 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:28.623 7f3896d73700 1 --2- 172.21.2.102:0/2490665131 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f387c0186c0 0x7f387c01abc0 crc :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:28.675 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:28.653 7f389d9ca700 1 -- 172.21.2.102:0/2490665131 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3898080730 con 0x7f38980a4db0 2021-02-10T11:34:28.675 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:28.657 7f388ffff700 1 -- 172.21.2.102:0/2490665131 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f388802df00 con 0x7f38980a4db0 2021-02-10T11:34:28.790 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:28.789 7f389d9ca700 1 -- 172.21.2.102:0/2490665131 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f38980ce430 con 0x7f38980a4db0 2021-02-10T11:34:28.793 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:28.792 7f388ffff700 1 -- 172.21.2.102:0/2490665131 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.2 v79) v1 ==== 184+0+0 (secure 0 0 0) 0x7f3888000c90 con 0x7f38980a4db0 2021-02-10T11:34:28.797 INFO:teuthology.orchestra.run.gibba002.stderr:updated caps for client.2 2021-02-10T11:34:28.799 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:28.798 7f389d9ca700 1 -- 172.21.2.102:0/2490665131 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f387c0186c0 msgr2=0x7f387c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:28.799 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:28.798 7f389d9ca700 1 --2- 172.21.2.102:0/2490665131 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f387c0186c0 0x7f387c01abc0 crc :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:28.799 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:28.798 7f389d9ca700 1 -- 172.21.2.102:0/2490665131 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f38980a4db0 msgr2=0x7f38980b10b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:28.799 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:28.798 7f389d9ca700 1 --2- 172.21.2.102:0/2490665131 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f38980a4db0 0x7f38980b10b0 secure :-1 s=READY pgs=2555 cs=0 l=1 rev1=1 rx=0x7f388800b5e0 tx=0x7f388800b980).stop 2021-02-10T11:34:28.800 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:28.798 7f389d9ca700 1 -- 172.21.2.102:0/2490665131 shutdown_connections 2021-02-10T11:34:28.800 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:28.798 7f389d9ca700 1 --2- 172.21.2.102:0/2490665131 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f38980b1cd0 0x7f38980b20d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:28.800 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:28.798 7f389d9ca700 1 --2- 172.21.2.102:0/2490665131 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f387c0186c0 0x7f387c01abc0 unknown :-1 s=CLOSED pgs=1130 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:28.800 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:28.798 7f389d9ca700 1 --2- 172.21.2.102:0/2490665131 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f38980a7500 0x7f38980a7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:28.800 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:28.798 7f389d9ca700 1 --2- 172.21.2.102:0/2490665131 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f38980a4db0 0x7f38980b10b0 unknown :-1 s=CLOSED pgs=2555 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:28.800 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:28.798 7f389d9ca700 1 -- 172.21.2.102:0/2490665131 >> 172.21.2.102:0/2490665131 conn(0x7f38981071a0 msgr2=0x7f3898100cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:28.800 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:28.798 7f389d9ca700 1 -- 172.21.2.102:0/2490665131 shutdown_connections 2021-02-10T11:34:28.800 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:28.798 7f389d9ca700 1 -- 172.21.2.102:0/2490665131 wait complete. 2021-02-10T11:34:28.808 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.3 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:34:29.249 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:29.232 7fde6555e700 1 Processor -- start 2021-02-10T11:34:29.250 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:29.232 7fde6555e700 1 -- start start 2021-02-10T11:34:29.250 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:29.233 7fde6555e700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fde600d1880 0x7fde601b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:29.250 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:29.233 7fde6555e700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fde600cff80 0x7fde60109140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:29.250 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:29.233 7fde6555e700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde60109a70 con 0x7fde601b0af0 2021-02-10T11:34:29.250 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:29.233 7fde6555e700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde60109760 con 0x7fde600cff80 2021-02-10T11:34:29.250 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:29.233 7fde6555e700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde60088070 con 0x7fde600d1880 2021-02-10T11:34:29.250 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:29.233 7fde5fac0700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fde600cff80 0x7fde60109140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:29.250 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:29.233 7fde5eabe700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fde601b0af0 0x7fde60108950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:56630/0 (socket says 172.21.2.102:56630) 2021-02-10T11:34:29.250 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:29.233 7fde5eabe700 1 -- 172.21.2.102:0/1790070907 learned_addr learned my addr 172.21.2.102:0/1790070907 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:34:29.250 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:29.233 7fde5f2bf700 1 --2- 172.21.2.102:0/1790070907 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fde600d1880 0x7fde601b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:29.250 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:29.233 7fde5fac0700 1 -- 172.21.2.102:0/1790070907 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fde600d1880 msgr2=0x7fde601b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:34:29.251 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:29.233 7fde5fac0700 1 --2- 172.21.2.102:0/1790070907 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fde600d1880 0x7fde601b04d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:29.251 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:29.233 7fde5fac0700 1 -- 172.21.2.102:0/1790070907 >> v1:172.21.2.102:6789/0 conn(0x7fde601b0af0 legacy=0x7fde60108950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:29.251 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:29.233 7fde5fac0700 1 -- 172.21.2.102:0/1790070907 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde60088260 con 0x7fde600cff80 2021-02-10T11:34:29.251 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:29.233 7fde5f2bf700 1 --2- 172.21.2.102:0/1790070907 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fde600d1880 0x7fde601b04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:34:29.251 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:29.233 7fde5fac0700 1 --2- 172.21.2.102:0/1790070907 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fde600cff80 0x7fde60109140 secure :-1 s=READY pgs=2557 cs=0 l=1 rev1=1 rx=0x7fde58006e30 tx=0x7fde58009e60).ready entity=mon.0 client_cookie=d482488b98cbc54b server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:29.251 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:29.234 7fde5e2bd700 1 -- 172.21.2.102:0/1790070907 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fde58006a50 con 0x7fde600cff80 2021-02-10T11:34:29.251 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:29.234 7fde5e2bd700 1 -- 172.21.2.102:0/1790070907 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde5800b300 con 0x7fde600cff80 2021-02-10T11:34:29.251 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:29.234 7fde5e2bd700 1 -- 172.21.2.102:0/1790070907 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fde5800b7e0 con 0x7fde600cff80 2021-02-10T11:34:29.251 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:29.234 7fde6555e700 1 -- 172.21.2.102:0/1790070907 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fde600cff80 msgr2=0x7fde60109140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:29.251 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:29.234 7fde6555e700 1 --2- 172.21.2.102:0/1790070907 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fde600cff80 0x7fde60109140 secure :-1 s=READY pgs=2557 cs=0 l=1 rev1=1 rx=0x7fde58006e30 tx=0x7fde58009e60).stop 2021-02-10T11:34:29.251 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:29.234 7fde6555e700 1 -- 172.21.2.102:0/1790070907 shutdown_connections 2021-02-10T11:34:29.252 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:29.234 7fde6555e700 1 --2- 172.21.2.102:0/1790070907 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fde600cff80 0x7fde60109140 unknown :-1 s=CLOSED pgs=2557 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:29.252 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:29.234 7fde6555e700 1 --2- 172.21.2.102:0/1790070907 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fde600d1880 0x7fde601b04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:29.252 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:29.234 7fde6555e700 1 -- 172.21.2.102:0/1790070907 >> 172.21.2.102:0/1790070907 conn(0x7fde600ce430 msgr2=0x7fde600d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:29.252 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:29.234 7fde6555e700 1 -- 172.21.2.102:0/1790070907 shutdown_connections 2021-02-10T11:34:29.252 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:29.234 7fde6555e700 1 -- 172.21.2.102:0/1790070907 wait complete. 2021-02-10T11:34:29.253 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:29.249 7fde6555e700 1 Processor -- start 2021-02-10T11:34:29.253 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:29.249 7fde6555e700 1 -- start start 2021-02-10T11:34:29.254 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:29.249 7fde6555e700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fde601aac50 0x7fde601ab050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:29.254 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:29.249 7fde6555e700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fde601987c0 0x7fde60198bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:29.254 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:29.249 7fde6555e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fde601991e0 0x7fde60199f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:29.254 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:29.249 7fde6555e700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde60199980 con 0x7fde601991e0 2021-02-10T11:34:29.254 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:29.249 7fde6555e700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde601ab670 con 0x7fde601aac50 2021-02-10T11:34:29.254 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:29.249 7fde6555e700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde601995e0 con 0x7fde601987c0 2021-02-10T11:34:29.254 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:29.250 7fde5fac0700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fde601991e0 0x7fde60199f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:29.254 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:29.250 7fde5fac0700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fde601991e0 0x7fde60199f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:36178/0 (socket says 172.21.2.102:36178) 2021-02-10T11:34:29.254 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:29.250 7fde5fac0700 1 -- 172.21.2.102:0/266356302 learned_addr learned my addr 172.21.2.102:0/266356302 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:34:29.254 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:29.250 7fde5fac0700 1 -- 172.21.2.102:0/266356302 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fde601987c0 msgr2=0x7fde60198bc0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:34:29.254 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:29.250 7fde5fac0700 1 --2- 172.21.2.102:0/266356302 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fde601987c0 0x7fde60198bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:29.254 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:29.250 7fde5fac0700 1 -- 172.21.2.102:0/266356302 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fde601aac50 msgr2=0x7fde601ab050 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:34:29.255 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:29.250 7fde5fac0700 1 --2- 172.21.2.102:0/266356302 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fde601aac50 0x7fde601ab050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:29.255 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:29.250 7fde5fac0700 1 -- 172.21.2.102:0/266356302 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde6019a520 con 0x7fde601991e0 2021-02-10T11:34:29.255 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:29.250 7fde5fac0700 1 --2- 172.21.2.102:0/266356302 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fde601991e0 0x7fde60199f00 secure :-1 s=READY pgs=2558 cs=0 l=1 rev1=1 rx=0x7fde58001070 tx=0x7fde5800bd40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:29.255 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:29.250 7fde57fff700 1 -- 172.21.2.102:0/266356302 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fde58001270 con 0x7fde601991e0 2021-02-10T11:34:29.255 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:29.250 7fde57fff700 1 -- 172.21.2.102:0/266356302 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde58006df0 con 0x7fde601991e0 2021-02-10T11:34:29.255 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:29.250 7fde57fff700 1 -- 172.21.2.102:0/266356302 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fde58001730 con 0x7fde601991e0 2021-02-10T11:34:29.255 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:29.250 7fde6555e700 1 -- 172.21.2.102:0/266356302 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde6019a520 con 0x7fde601991e0 2021-02-10T11:34:29.255 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:29.250 7fde6555e700 1 -- 172.21.2.102:0/266356302 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde6019a520 con 0x7fde601991e0 2021-02-10T11:34:29.255 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:29.251 7fde57fff700 1 -- 172.21.2.102:0/266356302 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fde58001270 con 0x7fde601991e0 2021-02-10T11:34:29.255 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:29.251 7fde57fff700 1 --2- 172.21.2.102:0/266356302 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fde440186c0 0x7fde4401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:29.255 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:29.251 7fde57fff700 1 -- 172.21.2.102:0/266356302 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(137..137 src has 1..137) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fde58032b60 con 0x7fde601991e0 2021-02-10T11:34:29.255 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:29.251 7fde5f2bf700 1 --2- 172.21.2.102:0/266356302 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fde440186c0 0x7fde4401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:29.255 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:29.251 7fde5f2bf700 1 --2- 172.21.2.102:0/266356302 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fde440186c0 0x7fde4401abc0 crc :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:29.283 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:29.283 7fde6555e700 1 -- 172.21.2.102:0/266356302 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde60109a70 con 0x7fde601991e0 2021-02-10T11:34:29.286 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:29.285 7fde57fff700 1 -- 172.21.2.102:0/266356302 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fde580093b0 con 0x7fde601991e0 2021-02-10T11:34:29.417 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:29.417 7fde6555e700 1 -- 172.21.2.102:0/266356302 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.3", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7fde60109a70 con 0x7fde601991e0 2021-02-10T11:34:29.421 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:29.420 7fde57fff700 1 -- 172.21.2.102:0/266356302 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.3", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.3 v80) v1 ==== 184+0+0 (secure 0 0 0) 0x7fde58001270 con 0x7fde601991e0 2021-02-10T11:34:29.425 INFO:teuthology.orchestra.run.gibba002.stderr:updated caps for client.3 2021-02-10T11:34:29.427 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:29.426 7fde6555e700 1 -- 172.21.2.102:0/266356302 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fde440186c0 msgr2=0x7fde4401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:29.427 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:29.426 7fde6555e700 1 --2- 172.21.2.102:0/266356302 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fde440186c0 0x7fde4401abc0 crc :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:29.427 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:29.426 7fde6555e700 1 -- 172.21.2.102:0/266356302 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fde601991e0 msgr2=0x7fde60199f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:29.427 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:29.426 7fde6555e700 1 --2- 172.21.2.102:0/266356302 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fde601991e0 0x7fde60199f00 secure :-1 s=READY pgs=2558 cs=0 l=1 rev1=1 rx=0x7fde58001070 tx=0x7fde5800bd40).stop 2021-02-10T11:34:29.427 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:29.426 7fde6555e700 1 -- 172.21.2.102:0/266356302 shutdown_connections 2021-02-10T11:34:29.427 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:29.426 7fde6555e700 1 --2- 172.21.2.102:0/266356302 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fde601991e0 0x7fde60199f00 unknown :-1 s=CLOSED pgs=2558 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:29.427 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:29.426 7fde6555e700 1 --2- 172.21.2.102:0/266356302 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fde601987c0 0x7fde60198bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:29.427 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:29.426 7fde6555e700 1 --2- 172.21.2.102:0/266356302 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fde440186c0 0x7fde4401abc0 unknown :-1 s=CLOSED pgs=1131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:29.427 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:29.426 7fde6555e700 1 --2- 172.21.2.102:0/266356302 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fde601aac50 0x7fde601ab050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:29.427 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:29.426 7fde6555e700 1 -- 172.21.2.102:0/266356302 >> 172.21.2.102:0/266356302 conn(0x7fde6019c560 msgr2=0x7fde6019f2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:29.428 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:29.426 7fde6555e700 1 -- 172.21.2.102:0/266356302 shutdown_connections 2021-02-10T11:34:29.428 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:29.426 7fde6555e700 1 -- 172.21.2.102:0/266356302 wait complete. 2021-02-10T11:34:29.435 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:34:29.863 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:29.845 7f6c470d7700 1 Processor -- start 2021-02-10T11:34:29.863 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:29.845 7f6c470d7700 1 -- start start 2021-02-10T11:34:29.863 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:29.846 7f6c470d7700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f6c400d1870 0x7f6c401b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:29.863 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:29.846 7f6c470d7700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c400f0b10 con 0x7f6c400f0650 2021-02-10T11:34:29.863 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:29.846 7f6c470d7700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c400d0310 con 0x7f6c401b0a10 2021-02-10T11:34:29.863 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:29.846 7f6c470d7700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c400d0530 con 0x7f6c400d1870 2021-02-10T11:34:29.863 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:29.846 7f6c44ef3700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f6c400d1870 0x7f6c401b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:29.864 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:29.846 7f6c456f4700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f6c400f0650 0x7f6c400cfb20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:56646/0 (socket says 172.21.2.102:56646) 2021-02-10T11:34:29.864 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:29.846 7f6c44ef3700 1 --2- 172.21.2.102:0/325098428 >> v2:172.21.2.102:3300/0 conn(0x7f6c400d1870 0x7f6c401b03f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:36188/0 (socket says 172.21.2.102:36188) 2021-02-10T11:34:29.864 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:29.846 7f6c456f4700 1 -- 172.21.2.102:0/325098428 learned_addr learned my addr 172.21.2.102:0/325098428 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:34:29.864 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:29.846 7f6c44ef3700 1 -- 172.21.2.102:0/325098428 >> v1:172.21.2.113:6789/0 conn(0x7f6c401b0a10 legacy=0x7f6c400cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:34:29.864 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:29.846 7f6c44ef3700 1 -- 172.21.2.102:0/325098428 >> v1:172.21.2.102:6789/0 conn(0x7f6c400f0650 legacy=0x7f6c400cfb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:29.864 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:29.846 7f6c44ef3700 1 -- 172.21.2.102:0/325098428 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c400d1c70 con 0x7f6c400d1870 2021-02-10T11:34:29.864 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:29.846 7f6c44ef3700 1 --2- 172.21.2.102:0/325098428 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6c400d1870 0x7f6c401b03f0 secure :-1 s=READY pgs=2560 cs=0 l=1 rev1=1 rx=0x7f6c28006650 tx=0x7f6c28009a80).ready entity=mon.0 client_cookie=5f7561d7a90138f4 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:29.864 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:29.847 7f6c3f7fe700 1 -- 172.21.2.102:0/325098428 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6c280009b0 con 0x7f6c400d1870 2021-02-10T11:34:29.864 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:29.847 7f6c3f7fe700 1 -- 172.21.2.102:0/325098428 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c2800c0c0 con 0x7f6c400d1870 2021-02-10T11:34:29.865 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:29.847 7f6c3f7fe700 1 -- 172.21.2.102:0/325098428 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6c2800c5e0 con 0x7f6c400d1870 2021-02-10T11:34:29.865 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:29.847 7f6c470d7700 1 -- 172.21.2.102:0/325098428 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6c400d1870 msgr2=0x7f6c401b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:29.865 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:29.847 7f6c470d7700 1 --2- 172.21.2.102:0/325098428 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6c400d1870 0x7f6c401b03f0 secure :-1 s=READY pgs=2560 cs=0 l=1 rev1=1 rx=0x7f6c28006650 tx=0x7f6c28009a80).stop 2021-02-10T11:34:29.865 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:29.847 7f6c470d7700 1 -- 172.21.2.102:0/325098428 shutdown_connections 2021-02-10T11:34:29.865 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:29.847 7f6c470d7700 1 --2- 172.21.2.102:0/325098428 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6c400d1870 0x7f6c401b03f0 unknown :-1 s=CLOSED pgs=2560 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:29.865 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:29.847 7f6c470d7700 1 -- 172.21.2.102:0/325098428 >> 172.21.2.102:0/325098428 conn(0x7f6c400ce450 msgr2=0x7f6c400d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:29.865 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:29.847 7f6c470d7700 1 -- 172.21.2.102:0/325098428 shutdown_connections 2021-02-10T11:34:29.865 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:29.847 7f6c470d7700 1 -- 172.21.2.102:0/325098428 wait complete. 2021-02-10T11:34:29.865 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:29.862 7f6c470d7700 1 Processor -- start 2021-02-10T11:34:29.865 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:29.862 7f6c470d7700 1 -- start start 2021-02-10T11:34:29.867 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:29.862 7f6c470d7700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6c400c33c0 0x7f6c400ad8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:29.867 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:29.862 7f6c470d7700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6c400adf00 0x7f6c400ae300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:29.867 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:29.862 7f6c470d7700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6c400ae920 0x7f6c400ee210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:29.867 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:29.862 7f6c470d7700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c400edc90 con 0x7f6c400adf00 2021-02-10T11:34:29.867 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:29.862 7f6c470d7700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c400ed800 con 0x7f6c400c33c0 2021-02-10T11:34:29.868 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:29.863 7f6c470d7700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c400c37c0 con 0x7f6c400ae920 2021-02-10T11:34:29.868 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:29.863 7f6c3ffff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6c400adf00 0x7f6c400ae300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:29.868 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:29.863 7f6c3ffff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6c400adf00 0x7f6c400ae300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:36192/0 (socket says 172.21.2.102:36192) 2021-02-10T11:34:29.868 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:29.863 7f6c3ffff700 1 -- 172.21.2.102:0/96479204 learned_addr learned my addr 172.21.2.102:0/96479204 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:34:29.868 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:29.863 7f6c3ffff700 1 -- 172.21.2.102:0/96479204 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6c400ae920 msgr2=0x7f6c400ee210 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:34:29.868 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:29.863 7f6c3ffff700 1 --2- 172.21.2.102:0/96479204 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6c400ae920 0x7f6c400ee210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:29.868 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:29.863 7f6c3ffff700 1 -- 172.21.2.102:0/96479204 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6c400c33c0 msgr2=0x7f6c400ad8e0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:34:29.868 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:29.863 7f6c3ffff700 1 --2- 172.21.2.102:0/96479204 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6c400c33c0 0x7f6c400ad8e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:29.868 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:29.863 7f6c3ffff700 1 -- 172.21.2.102:0/96479204 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c400ee830 con 0x7f6c400adf00 2021-02-10T11:34:29.868 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:29.863 7f6c3ffff700 1 --2- 172.21.2.102:0/96479204 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6c400adf00 0x7f6c400ae300 secure :-1 s=READY pgs=2561 cs=0 l=1 rev1=1 rx=0x7f6c30009940 tx=0x7f6c3000bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:29.868 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:29.863 7f6c3d7fa700 1 -- 172.21.2.102:0/96479204 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6c30000fe0 con 0x7f6c400adf00 2021-02-10T11:34:29.868 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:29.864 7f6c3d7fa700 1 -- 172.21.2.102:0/96479204 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c3000c2f0 con 0x7f6c400adf00 2021-02-10T11:34:29.869 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:29.864 7f6c470d7700 1 -- 172.21.2.102:0/96479204 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c400ee830 con 0x7f6c400adf00 2021-02-10T11:34:29.869 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:29.864 7f6c470d7700 1 -- 172.21.2.102:0/96479204 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c400eedc0 con 0x7f6c400adf00 2021-02-10T11:34:29.869 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:29.864 7f6c3d7fa700 1 -- 172.21.2.102:0/96479204 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6c3000c810 con 0x7f6c400adf00 2021-02-10T11:34:29.869 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:29.864 7f6c3d7fa700 1 -- 172.21.2.102:0/96479204 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f6c30001230 con 0x7f6c400adf00 2021-02-10T11:34:29.869 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:29.864 7f6c3d7fa700 1 --2- 172.21.2.102:0/96479204 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6c240186c0 0x7f6c2401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:29.869 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:29.864 7f6c3d7fa700 1 -- 172.21.2.102:0/96479204 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(137..137 src has 1..137) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f6c30032b90 con 0x7f6c400adf00 2021-02-10T11:34:29.869 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:29.865 7f6c44ef3700 1 --2- 172.21.2.102:0/96479204 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6c240186c0 0x7f6c2401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:29.870 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:29.865 7f6c44ef3700 1 --2- 172.21.2.102:0/96479204 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6c240186c0 0x7f6c2401abc0 crc :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:29.896 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:29.895 7f6c470d7700 1 -- 172.21.2.102:0/96479204 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c400e7de0 con 0x7f6c400adf00 2021-02-10T11:34:29.899 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:29.898 7f6c3d7fa700 1 -- 172.21.2.102:0/96479204 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f6c30000f90 con 0x7f6c400adf00 2021-02-10T11:34:30.027 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:30.026 7f6c470d7700 1 -- 172.21.2.102:0/96479204 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6c400d1ee0 con 0x7f6c400adf00 2021-02-10T11:34:30.028 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:30.027 7f6c3d7fa700 1 -- 172.21.2.102:0/96479204 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 116 v116) v1 ==== 95+0+2108 (secure 0 0 0) 0x7f6c30001230 con 0x7f6c400adf00 2021-02-10T11:34:30.028 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:34:30.028 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":116,"default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":11466,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.102:6835/1274216855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6834","nonce":1274216855},{"type":"v1","addr":"172.21.2.102:6835","nonce":1274216855}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":112}],"filesystems":[{"mdsmap":{"epoch":116,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:34:23.684779","modified":"2021-02-10 11:34:25.693957","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6770},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6770":{"gid":6770,"name":"b","rank":0,"incarnation":113,"state":"up:active","state_seq":3,"addr":"172.21.2.113:6833/3752249621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6832","nonce":3752249621},{"type":"v1","addr":"172.21.2.113:6833","nonce":3752249621}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[29],"metadata_pool":28,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":14}]} 2021-02-10T11:34:30.029 INFO:teuthology.orchestra.run.gibba002.stderr:dumped fsmap epoch 116 2021-02-10T11:34:30.029 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:30.029 7f6c470d7700 1 -- 172.21.2.102:0/96479204 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6c240186c0 msgr2=0x7f6c2401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:30.029 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:30.029 7f6c470d7700 1 --2- 172.21.2.102:0/96479204 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6c240186c0 0x7f6c2401abc0 crc :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:30.030 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:30.029 7f6c470d7700 1 -- 172.21.2.102:0/96479204 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6c400adf00 msgr2=0x7f6c400ae300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:30.030 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:30.029 7f6c470d7700 1 --2- 172.21.2.102:0/96479204 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6c400adf00 0x7f6c400ae300 secure :-1 s=READY pgs=2561 cs=0 l=1 rev1=1 rx=0x7f6c30009940 tx=0x7f6c3000bcf0).stop 2021-02-10T11:34:30.030 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:30.029 7f6c470d7700 1 -- 172.21.2.102:0/96479204 shutdown_connections 2021-02-10T11:34:30.030 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:30.029 7f6c470d7700 1 --2- 172.21.2.102:0/96479204 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6c400ae920 0x7f6c400ee210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:30.031 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:30.029 7f6c470d7700 1 --2- 172.21.2.102:0/96479204 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6c400adf00 0x7f6c400ae300 unknown :-1 s=CLOSED pgs=2561 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:30.031 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:30.029 7f6c470d7700 1 --2- 172.21.2.102:0/96479204 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6c240186c0 0x7f6c2401abc0 unknown :-1 s=CLOSED pgs=1132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:30.031 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:30.029 7f6c470d7700 1 --2- 172.21.2.102:0/96479204 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6c400c33c0 0x7f6c400ad8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:30.031 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:30.029 7f6c470d7700 1 -- 172.21.2.102:0/96479204 >> 172.21.2.102:0/96479204 conn(0x7f6c400a0030 msgr2=0x7f6c400eb1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:30.031 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:30.029 7f6c470d7700 1 -- 172.21.2.102:0/96479204 shutdown_connections 2021-02-10T11:34:30.031 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:30.029 7f6c470d7700 1 -- 172.21.2.102:0/96479204 wait complete. 2021-02-10T11:34:30.240 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 116, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:34:23.684779', 'modified': '2021-02-10 11:34:25.693957', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6770}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6770': {'gid': 6770, 'name': 'b', 'rank': 0, 'incarnation': 113, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.113:6833/3752249621', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.113:6832', 'nonce': 3752249621}, {'type': 'v1', 'addr': '172.21.2.113:6833', 'nonce': 3752249621}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [29], 'metadata_pool': 28, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:34:30.240 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:34:30.240 DEBUG:teuthology.orchestra.run.gibba013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:34:30.639 INFO:teuthology.orchestra.run.gibba013.stdout:{ 2021-02-10T11:34:30.639 INFO:teuthology.orchestra.run.gibba013.stdout: "cluster_fsid": "68db30ea-dbce-4aef-a6df-f01c6fea0513", 2021-02-10T11:34:30.639 INFO:teuthology.orchestra.run.gibba013.stdout: "whoami": 0, 2021-02-10T11:34:30.639 INFO:teuthology.orchestra.run.gibba013.stdout: "id": 6770, 2021-02-10T11:34:30.639 INFO:teuthology.orchestra.run.gibba013.stdout: "want_state": "up:active", 2021-02-10T11:34:30.640 INFO:teuthology.orchestra.run.gibba013.stdout: "state": "up:active", 2021-02-10T11:34:30.640 INFO:teuthology.orchestra.run.gibba013.stdout: "rank_uptime": 6.9504833159999997, 2021-02-10T11:34:30.640 INFO:teuthology.orchestra.run.gibba013.stdout: "mdsmap_epoch": 116, 2021-02-10T11:34:30.640 INFO:teuthology.orchestra.run.gibba013.stdout: "osdmap_epoch": 137, 2021-02-10T11:34:30.640 INFO:teuthology.orchestra.run.gibba013.stdout: "osdmap_epoch_barrier": 137, 2021-02-10T11:34:30.641 INFO:teuthology.orchestra.run.gibba013.stdout: "uptime": 12.603052771 2021-02-10T11:34:30.641 INFO:teuthology.orchestra.run.gibba013.stdout:} 2021-02-10T11:34:30.643 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "68db30ea-dbce-4aef-a6df-f01c6fea0513", "whoami": 0, "id": 6770, "want_state": "up:active", "state": "up:active", "rank_uptime": 6.9504833159999997, "mdsmap_epoch": 116, "osdmap_epoch": 137, "osdmap_epoch_barrier": 137, "uptime": 12.603052771 } 2021-02-10T11:34:30.644 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:34:31.074 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:31.056 7f241beee700 1 Processor -- start 2021-02-10T11:34:31.074 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:31.056 7f241beee700 1 -- start start 2021-02-10T11:34:31.074 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:31.056 7f241beee700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f24140d1870 0x7f24141b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:31.074 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:31.056 7f241beee700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f24141b0a10 0x7f24140cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:31.075 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:31.056 7f241beee700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24140f0b10 con 0x7f24140d1c70 2021-02-10T11:34:31.075 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:31.056 7f241beee700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24140e5ed0 con 0x7f24140d1870 2021-02-10T11:34:31.075 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:31.056 7f241beee700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24140f0650 con 0x7f24141b0a10 2021-02-10T11:34:31.075 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:31.057 7f2419509700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f24141b0a10 0x7f24140cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:31.075 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:31.057 7f2419d0a700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f24140d1870 0x7f24141b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:31.075 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:31.057 7f241a50b700 1 --1- >> v1:172.21.2.113:6789/0 conn(0x7f24140d1c70 0x7f24140cf950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.113:6789/0 says I am v1:172.21.2.102:35748/0 (socket says 172.21.2.102:35748) 2021-02-10T11:34:31.075 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:31.057 7f241a50b700 1 -- 172.21.2.102:0/3554668291 learned_addr learned my addr 172.21.2.102:0/3554668291 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:34:31.075 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:31.058 7f2419509700 1 -- 172.21.2.102:0/3554668291 >> v2:172.21.2.113:3300/0 conn(0x7f24140d1870 msgr2=0x7f24141b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:34:31.075 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:31.058 7f2419509700 1 --2- 172.21.2.102:0/3554668291 >> v2:172.21.2.113:3300/0 conn(0x7f24140d1870 0x7f24141b03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:31.075 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:31.058 7f2419509700 1 -- 172.21.2.102:0/3554668291 >> v1:172.21.2.113:6789/0 conn(0x7f24140d1c70 legacy=0x7f24140cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:31.075 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:31.058 7f2419509700 1 -- 172.21.2.102:0/3554668291 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24140f0870 con 0x7f24141b0a10 2021-02-10T11:34:31.075 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:31.058 7f2419d0a700 1 --2- 172.21.2.102:0/3554668291 >> v2:172.21.2.113:3300/0 conn(0x7f24140d1870 0x7f24141b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:34:31.075 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:31.058 7f2419509700 1 --2- 172.21.2.102:0/3554668291 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f24141b0a10 0x7f24140cf330 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 rx=0x7f2404006650 tx=0x7f2404009a80).ready entity=mon.2 client_cookie=5df865b245d92416 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:31.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:31.058 7f2418d08700 1 -- 172.21.2.102:0/3554668291 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f240400c0c0 con 0x7f24141b0a10 2021-02-10T11:34:31.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:31.058 7f2418d08700 1 -- 172.21.2.102:0/3554668291 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f240400c3c0 con 0x7f24141b0a10 2021-02-10T11:34:31.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:31.058 7f2418d08700 1 -- 172.21.2.102:0/3554668291 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f240400c8e0 con 0x7f24141b0a10 2021-02-10T11:34:31.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:31.058 7f241beee700 1 -- 172.21.2.102:0/3554668291 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f24141b0a10 msgr2=0x7f24140cf330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:31.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:31.058 7f241beee700 1 --2- 172.21.2.102:0/3554668291 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f24141b0a10 0x7f24140cf330 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 rx=0x7f2404006650 tx=0x7f2404009a80).stop 2021-02-10T11:34:31.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:31.059 7f241beee700 1 -- 172.21.2.102:0/3554668291 shutdown_connections 2021-02-10T11:34:31.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:31.059 7f241beee700 1 --2- 172.21.2.102:0/3554668291 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f24141b0a10 0x7f24140cf330 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:31.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:31.059 7f241beee700 1 --2- 172.21.2.102:0/3554668291 >> v2:172.21.2.113:3300/0 conn(0x7f24140d1870 0x7f24141b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:31.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:31.059 7f241beee700 1 -- 172.21.2.102:0/3554668291 >> 172.21.2.102:0/3554668291 conn(0x7f24140ce450 msgr2=0x7f24140d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:31.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:31.059 7f241beee700 1 -- 172.21.2.102:0/3554668291 shutdown_connections 2021-02-10T11:34:31.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:31.059 7f241beee700 1 -- 172.21.2.102:0/3554668291 wait complete. 2021-02-10T11:34:31.078 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:31.074 7f241beee700 1 Processor -- start 2021-02-10T11:34:31.078 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:31.074 7f241beee700 1 -- start start 2021-02-10T11:34:31.078 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:31.074 7f241beee700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2414195650 0x7f24140b5560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:31.078 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:31.074 7f241beee700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f24140b5b80 0x7f24141a2e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:31.078 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:31.074 7f241beee700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f24140acfa0 0x7f24141a3440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:31.078 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:31.074 7f241beee700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24140b6010 con 0x7f24140b5b80 2021-02-10T11:34:31.078 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:31.074 7f241beee700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24141a3a60 con 0x7f2414195650 2021-02-10T11:34:31.079 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:31.074 7f241beee700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24141a3c40 con 0x7f24140acfa0 2021-02-10T11:34:31.079 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:31.074 7f2419509700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f24140b5b80 0x7f24141a2e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:31.079 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:31.074 7f2419509700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f24140b5b80 0x7f24141a2e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:36204/0 (socket says 172.21.2.102:36204) 2021-02-10T11:34:31.079 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:31.074 7f2419509700 1 -- 172.21.2.102:0/2781210033 learned_addr learned my addr 172.21.2.102:0/2781210033 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:34:31.079 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:31.074 7f2419d0a700 1 --2- 172.21.2.102:0/2781210033 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2414195650 0x7f24140b5560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:31.079 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:31.074 7f241a50b700 1 --2- 172.21.2.102:0/2781210033 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f24140acfa0 0x7f24141a3440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:31.079 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:31.075 7f2419509700 1 -- 172.21.2.102:0/2781210033 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f24140acfa0 msgr2=0x7f24141a3440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:31.079 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:31.075 7f2419509700 1 --2- 172.21.2.102:0/2781210033 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f24140acfa0 0x7f24141a3440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:31.079 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:31.075 7f2419509700 1 -- 172.21.2.102:0/2781210033 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2414195650 msgr2=0x7f24140b5560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:31.079 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:31.075 7f2419509700 1 --2- 172.21.2.102:0/2781210033 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2414195650 0x7f24140b5560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:31.079 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:31.075 7f2419509700 1 -- 172.21.2.102:0/2781210033 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24140bcae0 con 0x7f24140b5b80 2021-02-10T11:34:31.079 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:31.075 7f241a50b700 1 --2- 172.21.2.102:0/2781210033 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f24140acfa0 0x7f24141a3440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:34:31.079 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:31.075 7f2419509700 1 --2- 172.21.2.102:0/2781210033 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f24140b5b80 0x7f24141a2e20 secure :-1 s=READY pgs=2562 cs=0 l=1 rev1=1 rx=0x7f2404006650 tx=0x7f240400b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:31.079 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:31.075 7f240a7fc700 1 -- 172.21.2.102:0/2781210033 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2404002030 con 0x7f24140b5b80 2021-02-10T11:34:31.080 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:31.075 7f240a7fc700 1 -- 172.21.2.102:0/2781210033 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24040022b0 con 0x7f24140b5b80 2021-02-10T11:34:31.080 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:31.075 7f240a7fc700 1 -- 172.21.2.102:0/2781210033 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f24040027d0 con 0x7f24140b5b80 2021-02-10T11:34:31.080 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:31.075 7f241beee700 1 -- 172.21.2.102:0/2781210033 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24140bcae0 con 0x7f24140b5b80 2021-02-10T11:34:31.080 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:31.075 7f241beee700 1 -- 172.21.2.102:0/2781210033 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24140bcae0 con 0x7f24140b5b80 2021-02-10T11:34:31.080 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:31.076 7f240a7fc700 1 -- 172.21.2.102:0/2781210033 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f2404009370 con 0x7f24140b5b80 2021-02-10T11:34:31.080 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:31.076 7f240a7fc700 1 --2- 172.21.2.102:0/2781210033 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f23fc0186c0 0x7f23fc01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:31.080 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:31.076 7f240a7fc700 1 -- 172.21.2.102:0/2781210033 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(137..137 src has 1..137) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f2404032410 con 0x7f24140b5b80 2021-02-10T11:34:31.080 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:31.076 7f2419d0a700 1 --2- 172.21.2.102:0/2781210033 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f23fc0186c0 0x7f23fc01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:31.080 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:31.076 7f2419d0a700 1 --2- 172.21.2.102:0/2781210033 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f23fc0186c0 0x7f23fc01abc0 crc :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:31.107 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:31.106 7f241beee700 1 -- 172.21.2.102:0/2781210033 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24140b6010 con 0x7f24140b5b80 2021-02-10T11:34:31.110 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:31.109 7f240a7fc700 1 -- 172.21.2.102:0/2781210033 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f2404025f00 con 0x7f24140b5b80 2021-02-10T11:34:31.234 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:31.233 7f241beee700 1 -- 172.21.2.102:0/2781210033 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f24140b6010 con 0x7f24140b5b80 2021-02-10T11:34:31.235 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:31.234 7f240a7fc700 1 -- 172.21.2.102:0/2781210033 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v137) v1 ==== 74+0+14007 (secure 0 0 0) 0x7f2404025f00 con 0x7f24140b5b80 2021-02-10T11:34:31.235 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:34:31.235 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":137,"fsid":"68db30ea-dbce-4aef-a6df-f01c6fea0513","created":"2021-02-10 11:03:06.331557","modified":"2021-02-10 11:34:24.687007","last_up_change":"2021-02-10 11:03:32.690562","last_in_change":"2021-02-10 11:03:32.690562","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":29,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:03:37.322519","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":28,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:34:21.649291","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"136","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":29,"pool_name":"cephfs_data","create_time":"2021-02-10 11:34:22.675483","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"137","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"be691b4d-a039-408a-af80-9d53f02b6f1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":134,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6818","nonce":36088},{"type":"v1","addr":"172.21.2.102:6820","nonce":36088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6822","nonce":36088},{"type":"v1","addr":"172.21.2.102:6824","nonce":36088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6830","nonce":36088},{"type":"v1","addr":"172.21.2.102:6832","nonce":36088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6826","nonce":36088},{"type":"v1","addr":"172.21.2.102:6828","nonce":36088}]},"public_addr":"172.21.2.102:6820/36088","cluster_addr":"172.21.2.102:6824/36088","heartbeat_back_addr":"172.21.2.102:6832/36088","heartbeat_front_addr":"172.21.2.102:6828/36088","state":["exists","up"]},{"osd":1,"uuid":"64e3ccc8-9c69-4d7c-865e-af8e83822635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":135,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6819","nonce":36087},{"type":"v1","addr":"172.21.2.102:6821","nonce":36087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6823","nonce":36087},{"type":"v1","addr":"172.21.2.102:6825","nonce":36087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6831","nonce":36087},{"type":"v1","addr":"172.21.2.102:6833","nonce":36087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6827","nonce":36087},{"type":"v1","addr":"172.21.2.102:6829","nonce":36087}]},"public_addr":"172.21.2.102:6821/36087","cluster_addr":"172.21.2.102:6825/36087","heartbeat_back_addr":"172.21.2.102:6833/36087","heartbeat_front_addr":"172.21.2.102:6829/36087","state":["exists","up"]},{"osd":2,"uuid":"153700af-747f-481f-bdf4-d6072d3a842f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6810","nonce":36089},{"type":"v1","addr":"172.21.2.102:6811","nonce":36089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6812","nonce":36089},{"type":"v1","addr":"172.21.2.102:6813","nonce":36089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6816","nonce":36089},{"type":"v1","addr":"172.21.2.102:6817","nonce":36089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6814","nonce":36089},{"type":"v1","addr":"172.21.2.102:6815","nonce":36089}]},"public_addr":"172.21.2.102:6811/36089","cluster_addr":"172.21.2.102:6813/36089","heartbeat_back_addr":"172.21.2.102:6817/36089","heartbeat_front_addr":"172.21.2.102:6815/36089","state":["exists","up"]},{"osd":3,"uuid":"cbc678f4-83a9-45e6-8830-2f7c232b1ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6802","nonce":36090},{"type":"v1","addr":"172.21.2.102:6803","nonce":36090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6804","nonce":36090},{"type":"v1","addr":"172.21.2.102:6805","nonce":36090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6808","nonce":36090},{"type":"v1","addr":"172.21.2.102:6809","nonce":36090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6806","nonce":36090},{"type":"v1","addr":"172.21.2.102:6807","nonce":36090}]},"public_addr":"172.21.2.102:6803/36090","cluster_addr":"172.21.2.102:6805/36090","heartbeat_back_addr":"172.21.2.102:6809/36090","heartbeat_front_addr":"172.21.2.102:6807/36090","state":["exists","up"]},{"osd":4,"uuid":"c01e4008-b258-4f10-afd2-9e7f37214351","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":135,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6808","nonce":36368},{"type":"v1","addr":"172.21.2.113:6811","nonce":36368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6814","nonce":36368},{"type":"v1","addr":"172.21.2.113:6818","nonce":36368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6826","nonce":36368},{"type":"v1","addr":"172.21.2.113:6828","nonce":36368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6820","nonce":36368},{"type":"v1","addr":"172.21.2.113:6824","nonce":36368}]},"public_addr":"172.21.2.113:6811/36368","cluster_addr":"172.21.2.113:6818/36368","heartbeat_back_addr":"172.21.2.113:6828/36368","heartbeat_front_addr":"172.21.2.113:6824/36368","state":["exists","up"]},{"osd":5,"uuid":"551f6007-daf8-4a98-8ee0-c2b16bcccb5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":134,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6802","nonce":36366},{"type":"v1","addr":"172.21.2.113:6805","nonce":36366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6807","nonce":36366},{"type":"v1","addr":"172.21.2.113:6810","nonce":36366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6819","nonce":36366},{"type":"v1","addr":"172.21.2.113:6822","nonce":36366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6812","nonce":36366},{"type":"v1","addr":"172.21.2.113:6815","nonce":36366}]},"public_addr":"172.21.2.113:6805/36366","cluster_addr":"172.21.2.113:6810/36366","heartbeat_back_addr":"172.21.2.113:6822/36366","heartbeat_front_addr":"172.21.2.113:6815/36366","state":["exists","up"]},{"osd":6,"uuid":"bf67c040-0e50-48be-b102-c3943bc9ebef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":135,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6800","nonce":36367},{"type":"v1","addr":"172.21.2.113:6801","nonce":36367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6803","nonce":36367},{"type":"v1","addr":"172.21.2.113:6804","nonce":36367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6813","nonce":36367},{"type":"v1","addr":"172.21.2.113:6816","nonce":36367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6806","nonce":36367},{"type":"v1","addr":"172.21.2.113:6809","nonce":36367}]},"public_addr":"172.21.2.113:6801/36367","cluster_addr":"172.21.2.113:6804/36367","heartbeat_back_addr":"172.21.2.113:6816/36367","heartbeat_front_addr":"172.21.2.113:6809/36367","state":["exists","up"]},{"osd":7,"uuid":"f5461f80-018d-428f-be89-2eb2bcbe7bb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":135,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6817","nonce":36365},{"type":"v1","addr":"172.21.2.113:6821","nonce":36365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6823","nonce":36365},{"type":"v1","addr":"172.21.2.113:6825","nonce":36365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6830","nonce":36365},{"type":"v1","addr":"172.21.2.113:6831","nonce":36365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6827","nonce":36365},{"type":"v1","addr":"172.21.2.113:6829","nonce":36365}]},"public_addr":"172.21.2.113:6821/36365","cluster_addr":"172.21.2.113:6825/36365","heartbeat_back_addr":"172.21.2.113:6831/36365","heartbeat_front_addr":"172.21.2.113:6829/36365","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:34:31.236 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:31.236 7f241beee700 1 -- 172.21.2.102:0/2781210033 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f23fc0186c0 msgr2=0x7f23fc01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:31.236 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:31.236 7f241beee700 1 --2- 172.21.2.102:0/2781210033 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f23fc0186c0 0x7f23fc01abc0 crc :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:31.236 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:31.236 7f241beee700 1 -- 172.21.2.102:0/2781210033 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f24140b5b80 msgr2=0x7f24141a2e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:31.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:31.236 7f241beee700 1 --2- 172.21.2.102:0/2781210033 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f24140b5b80 0x7f24141a2e20 secure :-1 s=READY pgs=2562 cs=0 l=1 rev1=1 rx=0x7f2404006650 tx=0x7f240400b960).stop 2021-02-10T11:34:31.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:31.236 7f241beee700 1 -- 172.21.2.102:0/2781210033 shutdown_connections 2021-02-10T11:34:31.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:31.236 7f241beee700 1 --2- 172.21.2.102:0/2781210033 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f24140acfa0 0x7f24141a3440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:31.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:31.236 7f241beee700 1 --2- 172.21.2.102:0/2781210033 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f24140b5b80 0x7f24141a2e20 unknown :-1 s=CLOSED pgs=2562 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:31.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:31.236 7f241beee700 1 --2- 172.21.2.102:0/2781210033 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f23fc0186c0 0x7f23fc01abc0 unknown :-1 s=CLOSED pgs=1133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:31.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:31.236 7f241beee700 1 --2- 172.21.2.102:0/2781210033 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2414195650 0x7f24140b5560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:31.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:31.236 7f241beee700 1 -- 172.21.2.102:0/2781210033 >> 172.21.2.102:0/2781210033 conn(0x7f24140b3040 msgr2=0x7f24140c3250 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:31.238 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:31.236 7f241beee700 1 -- 172.21.2.102:0/2781210033 shutdown_connections 2021-02-10T11:34:31.238 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:31.236 7f241beee700 1 -- 172.21.2.102:0/2781210033 wait complete. 2021-02-10T11:34:31.245 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:03:37.322519', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 28, 'pool_name': 'cephfs_metadata', 'create_time': '2021-02-10 11:34:21.649291', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '136', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 29, 'pool_name': 'cephfs_data', 'create_time': '2021-02-10 11:34:22.675483', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '137', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-02-10T11:34:31.246 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:34:31.679 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:31.661 7faf7b56a700 1 Processor -- start 2021-02-10T11:34:31.680 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:31.661 7faf7b56a700 1 -- start start 2021-02-10T11:34:31.680 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:31.662 7faf7b56a700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7faf740d1c70 0x7faf740cef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:31.680 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:31.662 7faf7b56a700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7faf740cf580 0x7faf740cf980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:31.680 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:31.662 7faf7b56a700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf740fe1c0 con 0x7faf740d1870 2021-02-10T11:34:31.680 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:31.662 7faf7b56a700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf741b0cb0 con 0x7faf740d1c70 2021-02-10T11:34:31.680 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:31.662 7faf7b56a700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf74109a60 con 0x7faf740cf580 2021-02-10T11:34:31.680 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:31.662 7faf78b85700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7faf740d1c70 0x7faf740cef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:31.680 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:31.662 7faf79386700 1 --1- >> v1:172.21.2.113:6789/0 conn(0x7faf740d1870 0x7faf741b04c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.113:6789/0 says I am v1:172.21.2.102:35760/0 (socket says 172.21.2.102:35760) 2021-02-10T11:34:31.680 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:31.663 7faf79386700 1 -- 172.21.2.102:0/876320590 learned_addr learned my addr 172.21.2.102:0/876320590 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:34:31.680 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:31.663 7faf79b87700 1 --2- 172.21.2.102:0/876320590 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7faf740cf580 0x7faf740cf980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:31.680 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:31.663 7faf78b85700 1 -- 172.21.2.102:0/876320590 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7faf740cf580 msgr2=0x7faf740cf980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:34:31.680 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:31.663 7faf78b85700 1 --2- 172.21.2.102:0/876320590 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7faf740cf580 0x7faf740cf980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:31.681 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:31.663 7faf78b85700 1 -- 172.21.2.102:0/876320590 >> v1:172.21.2.113:6789/0 conn(0x7faf740d1870 legacy=0x7faf741b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:31.681 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:31.663 7faf6bfff700 1 -- 172.21.2.102:0/876320590 <== mon.1 v1:172.21.2.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2866453147 0 0) 0x7faf70000c60 con 0x7faf740d1870 2021-02-10T11:34:31.681 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:31.663 7faf78b85700 1 -- 172.21.2.102:0/876320590 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf74088070 con 0x7faf740d1c70 2021-02-10T11:34:31.681 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:31.663 7faf79b87700 1 --2- 172.21.2.102:0/876320590 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7faf740cf580 0x7faf740cf980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:34:31.681 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:31.663 7faf78b85700 1 --2- 172.21.2.102:0/876320590 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7faf740d1c70 0x7faf740cef60 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 rx=0x7faf64006650 tx=0x7faf64009a80).ready entity=mon.1 client_cookie=20bf8120766f5823 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:31.681 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:31.664 7faf6bfff700 1 -- 172.21.2.102:0/876320590 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faf640009b0 con 0x7faf740d1c70 2021-02-10T11:34:31.681 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:31.664 7faf6bfff700 1 -- 172.21.2.102:0/876320590 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf6400c0c0 con 0x7faf740d1c70 2021-02-10T11:34:31.681 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:31.664 7faf6bfff700 1 -- 172.21.2.102:0/876320590 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faf6400c5e0 con 0x7faf740d1c70 2021-02-10T11:34:31.681 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:31.664 7faf7b56a700 1 -- 172.21.2.102:0/876320590 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7faf740d1c70 msgr2=0x7faf740cef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:31.681 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:31.664 7faf7b56a700 1 --2- 172.21.2.102:0/876320590 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7faf740d1c70 0x7faf740cef60 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 rx=0x7faf64006650 tx=0x7faf64009a80).stop 2021-02-10T11:34:31.681 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:31.664 7faf7b56a700 1 -- 172.21.2.102:0/876320590 shutdown_connections 2021-02-10T11:34:31.681 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:31.664 7faf7b56a700 1 --2- 172.21.2.102:0/876320590 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7faf740cf580 0x7faf740cf980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:31.681 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:31.664 7faf7b56a700 1 --2- 172.21.2.102:0/876320590 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7faf740d1c70 0x7faf740cef60 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:31.681 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:31.664 7faf7b56a700 1 -- 172.21.2.102:0/876320590 >> 172.21.2.102:0/876320590 conn(0x7faf740ce420 msgr2=0x7faf740d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:31.682 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:31.665 7faf7b56a700 1 -- 172.21.2.102:0/876320590 shutdown_connections 2021-02-10T11:34:31.682 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:31.665 7faf7b56a700 1 -- 172.21.2.102:0/876320590 wait complete. 2021-02-10T11:34:31.683 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:31.679 7faf7b56a700 1 Processor -- start 2021-02-10T11:34:31.683 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:31.679 7faf7b56a700 1 -- start start 2021-02-10T11:34:31.683 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:31.680 7faf7b56a700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7faf74085bc0 0x7faf740863e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:31.683 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:31.680 7faf7b56a700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7faf740954c0 0x7faf74096300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:31.684 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:31.680 7faf7b56a700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7faf740958c0 0x7faf74096920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:31.684 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:31.680 7faf7b56a700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf74095d80 con 0x7faf740958c0 2021-02-10T11:34:31.684 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:31.680 7faf7b56a700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf740bfa60 con 0x7faf740954c0 2021-02-10T11:34:31.684 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:31.680 7faf7b56a700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf74085fc0 con 0x7faf74085bc0 2021-02-10T11:34:31.684 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:31.680 7faf79b87700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7faf740958c0 0x7faf74096920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:31.684 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:31.680 7faf79b87700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7faf740958c0 0x7faf74096920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:36220/0 (socket says 172.21.2.102:36220) 2021-02-10T11:34:31.684 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:31.680 7faf79b87700 1 -- 172.21.2.102:0/374987380 learned_addr learned my addr 172.21.2.102:0/374987380 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:34:31.684 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:31.680 7faf78b85700 1 --2- 172.21.2.102:0/374987380 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7faf740954c0 0x7faf74096300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:31.684 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:31.680 7faf79386700 1 --2- 172.21.2.102:0/374987380 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7faf74085bc0 0x7faf740863e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:31.684 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:31.680 7faf79b87700 1 -- 172.21.2.102:0/374987380 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7faf74085bc0 msgr2=0x7faf740863e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:31.684 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:31.680 7faf79b87700 1 --2- 172.21.2.102:0/374987380 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7faf74085bc0 0x7faf740863e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:31.684 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:31.680 7faf79b87700 1 -- 172.21.2.102:0/374987380 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7faf740954c0 msgr2=0x7faf74096300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:31.684 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:31.680 7faf79b87700 1 --2- 172.21.2.102:0/374987380 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7faf740954c0 0x7faf74096300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:31.685 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:31.680 7faf79b87700 1 -- 172.21.2.102:0/374987380 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf74086c10 con 0x7faf740958c0 2021-02-10T11:34:31.685 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:31.680 7faf78b85700 1 --2- 172.21.2.102:0/374987380 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7faf740954c0 0x7faf74096300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:34:31.685 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:31.680 7faf79386700 1 --2- 172.21.2.102:0/374987380 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7faf74085bc0 0x7faf740863e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:34:31.685 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:31.680 7faf79b87700 1 --2- 172.21.2.102:0/374987380 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7faf740958c0 0x7faf74096920 secure :-1 s=READY pgs=2563 cs=0 l=1 rev1=1 rx=0x7faf6c00b970 tx=0x7faf6c00bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:31.685 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:31.681 7faf69ffb700 1 -- 172.21.2.102:0/374987380 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faf6c000f90 con 0x7faf740958c0 2021-02-10T11:34:31.685 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:31.681 7faf69ffb700 1 -- 172.21.2.102:0/374987380 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf6c003030 con 0x7faf740958c0 2021-02-10T11:34:31.685 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:31.681 7faf7b56a700 1 -- 172.21.2.102:0/374987380 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf74086c10 con 0x7faf740958c0 2021-02-10T11:34:31.685 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:31.681 7faf7b56a700 1 -- 172.21.2.102:0/374987380 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf74097230 con 0x7faf740958c0 2021-02-10T11:34:31.685 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:31.681 7faf69ffb700 1 -- 172.21.2.102:0/374987380 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faf6c001400 con 0x7faf740958c0 2021-02-10T11:34:31.685 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:31.681 7faf69ffb700 1 -- 172.21.2.102:0/374987380 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7faf6c0011f0 con 0x7faf740958c0 2021-02-10T11:34:31.685 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:31.681 7faf69ffb700 1 --2- 172.21.2.102:0/374987380 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7faf540186d0 0x7faf5401abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:31.685 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:31.681 7faf69ffb700 1 -- 172.21.2.102:0/374987380 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(137..137 src has 1..137) v4 ==== 5573+0+0 (secure 0 0 0) 0x7faf6c032b10 con 0x7faf740958c0 2021-02-10T11:34:31.685 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:31.681 7faf79386700 1 --2- 172.21.2.102:0/374987380 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7faf540186d0 0x7faf5401abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:31.685 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:31.682 7faf79386700 1 --2- 172.21.2.102:0/374987380 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7faf540186d0 0x7faf5401abd0 crc :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:31.713 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:31.712 7faf7b56a700 1 -- 172.21.2.102:0/374987380 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf740fe1c0 con 0x7faf740958c0 2021-02-10T11:34:31.715 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:31.714 7faf69ffb700 1 -- 172.21.2.102:0/374987380 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7faf6c000f40 con 0x7faf740958c0 2021-02-10T11:34:31.844 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:31.843 7faf7b56a700 1 -- 172.21.2.102:0/374987380 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7faf740fe1c0 con 0x7faf740958c0 2021-02-10T11:34:31.844 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:31.843 7faf69ffb700 1 -- 172.21.2.102:0/374987380 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v137) v1 ==== 92+0+10 (secure 0 0 0) 0x7faf6c0011f0 con 0x7faf740958c0 2021-02-10T11:34:31.845 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:34:31.846 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:31.845 7faf7b56a700 1 -- 172.21.2.102:0/374987380 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7faf540186d0 msgr2=0x7faf5401abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:31.847 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:31.845 7faf7b56a700 1 --2- 172.21.2.102:0/374987380 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7faf540186d0 0x7faf5401abd0 crc :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:31.847 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:31.845 7faf7b56a700 1 -- 172.21.2.102:0/374987380 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7faf740958c0 msgr2=0x7faf74096920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:31.847 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:31.845 7faf7b56a700 1 --2- 172.21.2.102:0/374987380 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7faf740958c0 0x7faf74096920 secure :-1 s=READY pgs=2563 cs=0 l=1 rev1=1 rx=0x7faf6c00b970 tx=0x7faf6c00bd10).stop 2021-02-10T11:34:31.847 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:31.845 7faf7b56a700 1 -- 172.21.2.102:0/374987380 shutdown_connections 2021-02-10T11:34:31.847 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:31.845 7faf7b56a700 1 --2- 172.21.2.102:0/374987380 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7faf740958c0 0x7faf74096920 unknown :-1 s=CLOSED pgs=2563 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:31.847 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:31.845 7faf7b56a700 1 --2- 172.21.2.102:0/374987380 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7faf540186d0 0x7faf5401abd0 unknown :-1 s=CLOSED pgs=1134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:31.847 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:31.845 7faf7b56a700 1 --2- 172.21.2.102:0/374987380 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7faf740954c0 0x7faf74096300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:31.848 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:31.845 7faf7b56a700 1 --2- 172.21.2.102:0/374987380 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7faf74085bc0 0x7faf740863e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:31.848 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:31.845 7faf7b56a700 1 -- 172.21.2.102:0/374987380 >> 172.21.2.102:0/374987380 conn(0x7faf740bef00 msgr2=0x7faf7409d1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:31.848 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:31.845 7faf7b56a700 1 -- 172.21.2.102:0/374987380 shutdown_connections 2021-02-10T11:34:31.848 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:31.845 7faf7b56a700 1 -- 172.21.2.102:0/374987380 wait complete. 2021-02-10T11:34:31.854 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-02-10T11:34:32.290 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:32.268 7f8647d10700 1 Processor -- start 2021-02-10T11:34:32.291 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:32.268 7f8647d10700 1 -- start start 2021-02-10T11:34:32.291 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:32.269 7f8647d10700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f86400d1870 0x7f86401b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:32.291 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:32.269 7f8647d10700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8640109a60 con 0x7f86401b0ae0 2021-02-10T11:34:32.291 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:32.269 7f8647d10700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8640088070 con 0x7f86400cff70 2021-02-10T11:34:32.291 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:32.269 7f8647d10700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86400e5ef0 con 0x7f86400d1870 2021-02-10T11:34:32.292 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:32.269 7f864632d700 1 --1- >> v1:172.21.2.113:6789/0 conn(0x7f86400cff70 0x7f8640109130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.113:6789/0 says I am v1:172.21.2.102:35774/0 (socket says 172.21.2.102:35774) 2021-02-10T11:34:32.292 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:32.269 7f864632d700 1 -- 172.21.2.102:0/2797645516 learned_addr learned my addr 172.21.2.102:0/2797645516 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:34:32.292 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:32.269 7f8645b2c700 1 --2- 172.21.2.102:0/2797645516 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f86400d1870 0x7f86401b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:32.292 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:32.270 7f8644b2a700 1 -- 172.21.2.102:0/2797645516 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2911933822 0 0) 0x7f863c000ff0 con 0x7f86401b0ae0 2021-02-10T11:34:32.292 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:32.270 7f8644b2a700 1 -- 172.21.2.102:0/2797645516 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f862c002fe0 con 0x7f86401b0ae0 2021-02-10T11:34:32.292 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:32.270 7f8644b2a700 1 -- 172.21.2.102:0/2797645516 <== mon.1 v1:172.21.2.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 974385545 0 0) 0x7f8638001440 con 0x7f86400cff70 2021-02-10T11:34:32.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:32.270 7f8644b2a700 1 -- 172.21.2.102:0/2797645516 --> v1:172.21.2.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f862c004370 con 0x7f86400cff70 2021-02-10T11:34:32.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:32.270 7f8645b2c700 1 -- 172.21.2.102:0/2797645516 >> v1:172.21.2.113:6789/0 conn(0x7f86400cff70 legacy=0x7f8640109130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:32.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:32.270 7f8645b2c700 1 -- 172.21.2.102:0/2797645516 >> v1:172.21.2.102:6789/0 conn(0x7f86401b0ae0 legacy=0x7f8640108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:32.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:32.270 7f8645b2c700 1 -- 172.21.2.102:0/2797645516 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86400d1c70 con 0x7f86400d1870 2021-02-10T11:34:32.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:32.270 7f8645b2c700 1 --2- 172.21.2.102:0/2797645516 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f86400d1870 0x7f86401b04c0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7f8634006650 tx=0x7f8634009a80).ready entity=mon.2 client_cookie=33da5aec39a6065a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:32.294 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:32.270 7f8644b2a700 1 -- 172.21.2.102:0/2797645516 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f863400c0c0 con 0x7f86400d1870 2021-02-10T11:34:32.294 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:32.270 7f8644b2a700 1 -- 172.21.2.102:0/2797645516 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f863400c340 con 0x7f86400d1870 2021-02-10T11:34:32.294 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:32.270 7f8644b2a700 1 -- 172.21.2.102:0/2797645516 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f863400c860 con 0x7f86400d1870 2021-02-10T11:34:32.294 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:32.270 7f8647d10700 1 -- 172.21.2.102:0/2797645516 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f86400d1870 msgr2=0x7f86401b04c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:32.294 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:32.270 7f8647d10700 1 --2- 172.21.2.102:0/2797645516 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f86400d1870 0x7f86401b04c0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7f8634006650 tx=0x7f8634009a80).stop 2021-02-10T11:34:32.294 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:32.271 7f8647d10700 1 -- 172.21.2.102:0/2797645516 shutdown_connections 2021-02-10T11:34:32.295 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:32.271 7f8647d10700 1 --2- 172.21.2.102:0/2797645516 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f86400d1870 0x7f86401b04c0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:32.295 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:32.271 7f8647d10700 1 -- 172.21.2.102:0/2797645516 >> 172.21.2.102:0/2797645516 conn(0x7f86400ce420 msgr2=0x7f86400d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:32.295 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:32.271 7f8647d10700 1 -- 172.21.2.102:0/2797645516 shutdown_connections 2021-02-10T11:34:32.295 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:32.271 7f8647d10700 1 -- 172.21.2.102:0/2797645516 wait complete. 2021-02-10T11:34:32.295 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:32.286 7f8647d10700 1 Processor -- start 2021-02-10T11:34:32.296 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:32.286 7f8647d10700 1 -- start start 2021-02-10T11:34:32.296 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:32.286 7f8647d10700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f86400a58c0 0x7f86400a4d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:32.296 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:32.286 7f8647d10700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f86400b1ba0 0x7f86400b1fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:32.296 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:32.286 7f8647d10700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f86400b25c0 0x7f86400b3270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:32.296 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:32.286 7f8647d10700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86400b2a40 con 0x7f86400b25c0 2021-02-10T11:34:32.296 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:32.286 7f8647d10700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86400a5340 con 0x7f86400a58c0 2021-02-10T11:34:32.297 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:32.286 7f8647d10700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86400a5560 con 0x7f86400b1ba0 2021-02-10T11:34:32.297 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:32.287 7f864632d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f86400b25c0 0x7f86400b3270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:32.297 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:32.287 7f864532b700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f86400b1ba0 0x7f86400b1fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:32.297 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:32.287 7f864632d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f86400b25c0 0x7f86400b3270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:36234/0 (socket says 172.21.2.102:36234) 2021-02-10T11:34:32.297 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:32.287 7f864532b700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f86400b1ba0 0x7f86400b1fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.113:3301/0 says I am v2:172.21.2.102:41772/0 (socket says 172.21.2.102:41772) 2021-02-10T11:34:32.298 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:32.287 7f864632d700 1 -- 172.21.2.102:0/4203534007 learned_addr learned my addr 172.21.2.102:0/4203534007 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:34:32.298 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:32.287 7f8645b2c700 1 --2- 172.21.2.102:0/4203534007 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f86400a58c0 0x7f86400a4d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:32.298 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:32.287 7f864532b700 1 -- 172.21.2.102:0/4203534007 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f86400a58c0 msgr2=0x7f86400a4d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:32.298 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:32.287 7f864532b700 1 --2- 172.21.2.102:0/4203534007 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f86400a58c0 0x7f86400a4d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:32.298 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:32.287 7f864532b700 1 -- 172.21.2.102:0/4203534007 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f86400b25c0 msgr2=0x7f86400b3270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:32.299 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:32.287 7f864532b700 1 --2- 172.21.2.102:0/4203534007 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f86400b25c0 0x7f86400b3270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:32.299 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:32.287 7f864532b700 1 -- 172.21.2.102:0/4203534007 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86400b3890 con 0x7f86400b1ba0 2021-02-10T11:34:32.299 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:32.287 7f864632d700 1 --2- 172.21.2.102:0/4203534007 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f86400b25c0 0x7f86400b3270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:34:32.299 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:32.287 7f864532b700 1 --2- 172.21.2.102:0/4203534007 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f86400b1ba0 0x7f86400b1fa0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 rx=0x7f863c006e90 tx=0x7f863c00b8c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:32.299 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:32.287 7f86327fc700 1 -- 172.21.2.102:0/4203534007 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f863c00bcb0 con 0x7f86400b1ba0 2021-02-10T11:34:32.299 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:32.287 7f86327fc700 1 -- 172.21.2.102:0/4203534007 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f863c0066d0 con 0x7f86400b1ba0 2021-02-10T11:34:32.300 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:32.287 7f8647d10700 1 -- 172.21.2.102:0/4203534007 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86400b3890 con 0x7f86400b1ba0 2021-02-10T11:34:32.300 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:32.288 7f86327fc700 1 -- 172.21.2.102:0/4203534007 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f863c006220 con 0x7f86400b1ba0 2021-02-10T11:34:32.300 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:32.288 7f8647d10700 1 -- 172.21.2.102:0/4203534007 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86400b3890 con 0x7f86400b1ba0 2021-02-10T11:34:32.300 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:32.288 7f86327fc700 1 -- 172.21.2.102:0/4203534007 <== mon.2 v2:172.21.2.113:3301/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f863c006030 con 0x7f86400b1ba0 2021-02-10T11:34:32.300 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:32.288 7f86327fc700 1 --2- 172.21.2.102:0/4203534007 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f86280186d0 0x7f862801abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:32.301 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:32.288 7f86327fc700 1 -- 172.21.2.102:0/4203534007 <== mon.2 v2:172.21.2.113:3301/0 5 ==== osd_map(137..137 src has 1..137) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f863c0326f0 con 0x7f86400b1ba0 2021-02-10T11:34:32.301 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:32.288 7f8645b2c700 1 --2- 172.21.2.102:0/4203534007 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f86280186d0 0x7f862801abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:32.301 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:32.289 7f8645b2c700 1 --2- 172.21.2.102:0/4203534007 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f86280186d0 0x7f862801abd0 crc :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:32.320 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:32.319 7f8647d10700 1 -- 172.21.2.102:0/4203534007 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8640082980 con 0x7f86400b1ba0 2021-02-10T11:34:32.323 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:32.323 7f86327fc700 1 -- 172.21.2.102:0/4203534007 <== mon.2 v2:172.21.2.113:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f863c00bcb0 con 0x7f86400b1ba0 2021-02-10T11:34:32.453 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:32.453 7f8647d10700 1 -- 172.21.2.102:0/4203534007 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7f86400b3890 con 0x7f86400b1ba0 2021-02-10T11:34:32.454 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:32.454 7f86327fc700 1 -- 172.21.2.102:0/4203534007 <== mon.2 v2:172.21.2.113:3301/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v137) v1 ==== 104+0+10 (secure 0 0 0) 0x7f863c006030 con 0x7f86400b1ba0 2021-02-10T11:34:32.456 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:34:32.458 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:32.457 7f8647d10700 1 -- 172.21.2.102:0/4203534007 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f86280186d0 msgr2=0x7f862801abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:32.458 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:32.457 7f8647d10700 1 --2- 172.21.2.102:0/4203534007 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f86280186d0 0x7f862801abd0 crc :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:32.458 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:32.457 7f8647d10700 1 -- 172.21.2.102:0/4203534007 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f86400b1ba0 msgr2=0x7f86400b1fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:32.458 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:32.457 7f8647d10700 1 --2- 172.21.2.102:0/4203534007 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f86400b1ba0 0x7f86400b1fa0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 rx=0x7f863c006e90 tx=0x7f863c00b8c0).stop 2021-02-10T11:34:32.459 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:32.457 7f8647d10700 1 -- 172.21.2.102:0/4203534007 shutdown_connections 2021-02-10T11:34:32.459 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:32.457 7f8647d10700 1 --2- 172.21.2.102:0/4203534007 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f86400b25c0 0x7f86400b3270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:32.459 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:32.457 7f8647d10700 1 --2- 172.21.2.102:0/4203534007 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f86400b1ba0 0x7f86400b1fa0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:32.459 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:32.457 7f8647d10700 1 --2- 172.21.2.102:0/4203534007 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f86280186d0 0x7f862801abd0 unknown :-1 s=CLOSED pgs=1135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:32.459 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:32.457 7f8647d10700 1 --2- 172.21.2.102:0/4203534007 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f86400a58c0 0x7f86400a4d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:32.459 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:32.457 7f8647d10700 1 -- 172.21.2.102:0/4203534007 >> 172.21.2.102:0/4203534007 conn(0x7f86400b5dc0 msgr2=0x7f8640095960 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:32.459 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:32.457 7f8647d10700 1 -- 172.21.2.102:0/4203534007 shutdown_connections 2021-02-10T11:34:32.459 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:32.458 7f8647d10700 1 -- 172.21.2.102:0/4203534007 wait complete. 2021-02-10T11:34:32.466 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-02-10T11:34:32.898 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:32.881 7f3c7146e700 1 Processor -- start 2021-02-10T11:34:32.898 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:32.881 7f3c7146e700 1 -- start start 2021-02-10T11:34:32.899 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:32.881 7f3c7146e700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f3c6c0d1860 0x7f3c6c1b04b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:32.899 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:32.881 7f3c7146e700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f3c6c0cff60 0x7f3c6c109120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:32.899 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:32.881 7f3c7146e700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c6c0fe1b0 con 0x7f3c6c1b0ad0 2021-02-10T11:34:32.899 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:32.881 7f3c7146e700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c6c109ac0 con 0x7f3c6c0cff60 2021-02-10T11:34:32.899 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:32.881 7f3c7146e700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c6c109740 con 0x7f3c6c0d1860 2021-02-10T11:34:32.899 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:32.881 7f3c6b87e700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f3c6c0cff60 0x7f3c6c109120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:32.899 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:32.882 7f3c6b87e700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f3c6c0cff60 0x7f3c6c109120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:36240/0 (socket says 172.21.2.102:36240) 2021-02-10T11:34:32.899 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:32.882 7f3c6b87e700 1 -- 172.21.2.102:0/2921079632 learned_addr learned my addr 172.21.2.102:0/2921079632 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:34:32.899 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:32.882 7f3c6b87e700 1 -- 172.21.2.102:0/2921079632 >> v2:172.21.2.113:3300/0 conn(0x7f3c6c0d1860 msgr2=0x7f3c6c1b04b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:34:32.899 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:32.882 7f3c6b87e700 1 --2- 172.21.2.102:0/2921079632 >> v2:172.21.2.113:3300/0 conn(0x7f3c6c0d1860 0x7f3c6c1b04b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:32.899 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:32.882 7f3c6b87e700 1 -- 172.21.2.102:0/2921079632 >> v1:172.21.2.113:6789/0 conn(0x7f3c6c1b0ad0 legacy=0x7f3c6c108930 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:34:32.899 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:32.882 7f3c6b87e700 1 -- 172.21.2.102:0/2921079632 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c6c088070 con 0x7f3c6c0cff60 2021-02-10T11:34:32.899 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:32.882 7f3c6b87e700 1 --2- 172.21.2.102:0/2921079632 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3c6c0cff60 0x7f3c6c109120 secure :-1 s=READY pgs=2565 cs=0 l=1 rev1=1 rx=0x7f3c64006f00 tx=0x7f3c64009e90).ready entity=mon.0 client_cookie=296fc790ad6f4f12 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:32.900 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:32.882 7f3c6a07b700 1 -- 172.21.2.102:0/2921079632 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3c6400b300 con 0x7f3c6c0cff60 2021-02-10T11:34:32.900 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:32.882 7f3c6a07b700 1 -- 172.21.2.102:0/2921079632 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c6400b640 con 0x7f3c6c0cff60 2021-02-10T11:34:32.900 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:32.882 7f3c6a07b700 1 -- 172.21.2.102:0/2921079632 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3c6400bb60 con 0x7f3c6c0cff60 2021-02-10T11:34:32.900 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:32.882 7f3c7146e700 1 -- 172.21.2.102:0/2921079632 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3c6c0cff60 msgr2=0x7f3c6c109120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:32.900 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:32.882 7f3c7146e700 1 --2- 172.21.2.102:0/2921079632 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3c6c0cff60 0x7f3c6c109120 secure :-1 s=READY pgs=2565 cs=0 l=1 rev1=1 rx=0x7f3c64006f00 tx=0x7f3c64009e90).stop 2021-02-10T11:34:32.900 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:32.883 7f3c7146e700 1 -- 172.21.2.102:0/2921079632 shutdown_connections 2021-02-10T11:34:32.900 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:32.883 7f3c7146e700 1 --2- 172.21.2.102:0/2921079632 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3c6c0cff60 0x7f3c6c109120 unknown :-1 s=CLOSED pgs=2565 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:32.900 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:32.883 7f3c7146e700 1 --2- 172.21.2.102:0/2921079632 >> v2:172.21.2.113:3300/0 conn(0x7f3c6c0d1860 0x7f3c6c1b04b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:32.900 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:32.883 7f3c7146e700 1 -- 172.21.2.102:0/2921079632 >> 172.21.2.102:0/2921079632 conn(0x7f3c6c0ce410 msgr2=0x7f3c6c0d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:32.900 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:32.883 7f3c7146e700 1 -- 172.21.2.102:0/2921079632 shutdown_connections 2021-02-10T11:34:32.900 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:32.883 7f3c7146e700 1 -- 172.21.2.102:0/2921079632 wait complete. 2021-02-10T11:34:32.902 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:32.898 7f3c7146e700 1 Processor -- start 2021-02-10T11:34:32.902 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:32.898 7f3c7146e700 1 -- start start 2021-02-10T11:34:32.902 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:32.898 7f3c7146e700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3c6c0bfa10 0x7f3c6c0b82a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:32.902 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:32.898 7f3c7146e700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3c6c1a4a70 0x7f3c6c094c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:32.902 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:32.898 7f3c7146e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3c6c0c0410 0x7f3c6c0c0810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:32.902 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:32.898 7f3c7146e700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c6c0c3a50 con 0x7f3c6c0c0410 2021-02-10T11:34:32.902 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:32.898 7f3c7146e700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c6c0b88c0 con 0x7f3c6c0bfa10 2021-02-10T11:34:32.902 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:32.898 7f3c7146e700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c6c095260 con 0x7f3c6c1a4a70 2021-02-10T11:34:32.902 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:32.899 7f3c6b87e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3c6c0c0410 0x7f3c6c0c0810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:32.902 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:32.899 7f3c6b87e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3c6c0c0410 0x7f3c6c0c0810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:36250/0 (socket says 172.21.2.102:36250) 2021-02-10T11:34:32.903 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:32.899 7f3c6b87e700 1 -- 172.21.2.102:0/3161713983 learned_addr learned my addr 172.21.2.102:0/3161713983 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:34:32.903 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:32.899 7f3c6b87e700 1 -- 172.21.2.102:0/3161713983 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3c6c1a4a70 msgr2=0x7f3c6c094c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:32.903 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:32.899 7f3c6b87e700 1 --2- 172.21.2.102:0/3161713983 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3c6c1a4a70 0x7f3c6c094c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:32.903 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:32.899 7f3c6b87e700 1 -- 172.21.2.102:0/3161713983 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3c6c0bfa10 msgr2=0x7f3c6c0b82a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:32.903 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:32.899 7f3c6b87e700 1 --2- 172.21.2.102:0/3161713983 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3c6c0bfa10 0x7f3c6c0b82a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:32.903 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:32.899 7f3c6b87e700 1 -- 172.21.2.102:0/3161713983 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c6c0c3540 con 0x7f3c6c0c0410 2021-02-10T11:34:32.903 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:32.899 7f3c6b87e700 1 --2- 172.21.2.102:0/3161713983 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3c6c0c0410 0x7f3c6c0c0810 secure :-1 s=READY pgs=2566 cs=0 l=1 rev1=1 rx=0x7f3c6400b940 tx=0x7f3c6400bce0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:32.903 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:32.899 7f3c63fff700 1 -- 172.21.2.102:0/3161713983 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3c64000f90 con 0x7f3c6c0c0410 2021-02-10T11:34:32.904 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:32.899 7f3c63fff700 1 -- 172.21.2.102:0/3161713983 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c64003030 con 0x7f3c6c0c0410 2021-02-10T11:34:32.904 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:32.899 7f3c7146e700 1 -- 172.21.2.102:0/3161713983 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c6c0c3540 con 0x7f3c6c0c0410 2021-02-10T11:34:32.904 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:32.899 7f3c7146e700 1 -- 172.21.2.102:0/3161713983 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c6c0c1120 con 0x7f3c6c0c0410 2021-02-10T11:34:32.904 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:32.899 7f3c63fff700 1 -- 172.21.2.102:0/3161713983 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3c64001460 con 0x7f3c6c0c0410 2021-02-10T11:34:32.904 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:32.900 7f3c63fff700 1 -- 172.21.2.102:0/3161713983 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f3c64001270 con 0x7f3c6c0c0410 2021-02-10T11:34:32.904 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:32.900 7f3c63fff700 1 --2- 172.21.2.102:0/3161713983 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3c500186c0 0x7f3c5001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:32.904 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:32.900 7f3c63fff700 1 -- 172.21.2.102:0/3161713983 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(137..137 src has 1..137) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f3c64032990 con 0x7f3c6c0c0410 2021-02-10T11:34:32.904 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:32.900 7f3c6b07d700 1 --2- 172.21.2.102:0/3161713983 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3c500186c0 0x7f3c5001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:32.904 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:32.901 7f3c6b07d700 1 --2- 172.21.2.102:0/3161713983 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3c500186c0 0x7f3c5001abc0 crc :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:32.932 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:32.931 7f3c7146e700 1 -- 172.21.2.102:0/3161713983 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c6c0bfee0 con 0x7f3c6c0c0410 2021-02-10T11:34:32.935 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:32.934 7f3c63fff700 1 -- 172.21.2.102:0/3161713983 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f3c64000f90 con 0x7f3c6c0c0410 2021-02-10T11:34:33.063 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:33.063 7f3c7146e700 1 -- 172.21.2.102:0/3161713983 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7f3c6c0bfee0 con 0x7f3c6c0c0410 2021-02-10T11:34:33.064 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:33.063 7f3c63fff700 1 -- 172.21.2.102:0/3161713983 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v137) v1 ==== 100+0+10 (secure 0 0 0) 0x7f3c64001270 con 0x7f3c6c0c0410 2021-02-10T11:34:33.064 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:34:33.066 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:33.065 7f3c7146e700 1 -- 172.21.2.102:0/3161713983 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3c500186c0 msgr2=0x7f3c5001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:33.066 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:33.065 7f3c7146e700 1 --2- 172.21.2.102:0/3161713983 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3c500186c0 0x7f3c5001abc0 crc :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:33.066 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:33.065 7f3c7146e700 1 -- 172.21.2.102:0/3161713983 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3c6c0c0410 msgr2=0x7f3c6c0c0810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:33.066 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:33.065 7f3c7146e700 1 --2- 172.21.2.102:0/3161713983 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3c6c0c0410 0x7f3c6c0c0810 secure :-1 s=READY pgs=2566 cs=0 l=1 rev1=1 rx=0x7f3c6400b940 tx=0x7f3c6400bce0).stop 2021-02-10T11:34:33.066 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:33.065 7f3c7146e700 1 -- 172.21.2.102:0/3161713983 shutdown_connections 2021-02-10T11:34:33.066 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:33.065 7f3c7146e700 1 --2- 172.21.2.102:0/3161713983 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3c6c0c0410 0x7f3c6c0c0810 unknown :-1 s=CLOSED pgs=2566 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:33.066 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:33.065 7f3c7146e700 1 --2- 172.21.2.102:0/3161713983 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3c6c1a4a70 0x7f3c6c094c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:33.066 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:33.065 7f3c7146e700 1 --2- 172.21.2.102:0/3161713983 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3c500186c0 0x7f3c5001abc0 unknown :-1 s=CLOSED pgs=1136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:33.066 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:33.065 7f3c7146e700 1 --2- 172.21.2.102:0/3161713983 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3c6c0bfa10 0x7f3c6c0b82a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:33.066 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:33.065 7f3c7146e700 1 -- 172.21.2.102:0/3161713983 >> 172.21.2.102:0/3161713983 conn(0x7f3c6c0a6a80 msgr2=0x7f3c6c0ae110 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:33.067 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:33.065 7f3c7146e700 1 -- 172.21.2.102:0/3161713983 shutdown_connections 2021-02-10T11:34:33.067 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:33.065 7f3c7146e700 1 -- 172.21.2.102:0/3161713983 wait complete. 2021-02-10T11:34:33.074 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-02-10T11:34:33.074 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:34:33.503 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:33.484 7f05100f8700 1 Processor -- start 2021-02-10T11:34:33.503 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:33.484 7f05100f8700 1 -- start start 2021-02-10T11:34:33.503 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:33.485 7f05100f8700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f05080d1870 0x7f05081b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:33.503 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:33.485 7f05100f8700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f05081b0a10 0x7f05080cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:33.504 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:33.485 7f05100f8700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05080f0b10 con 0x7f05080d1c70 2021-02-10T11:34:33.504 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:33.485 7f05100f8700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05080e5ed0 con 0x7f05080d1870 2021-02-10T11:34:33.504 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:33.485 7f05100f8700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05080f0650 con 0x7f05081b0a10 2021-02-10T11:34:33.504 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:33.486 7f050e715700 1 --1- >> v1:172.21.2.113:6789/0 conn(0x7f05080d1c70 0x7f05080cf950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.113:6789/0 says I am v1:172.21.2.102:35802/0 (socket says 172.21.2.102:35802) 2021-02-10T11:34:33.504 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:33.486 7f050d713700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f05081b0a10 0x7f05080cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:33.504 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:33.486 7f050e715700 1 -- 172.21.2.102:0/3481050498 learned_addr learned my addr 172.21.2.102:0/3481050498 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:34:33.504 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:33.486 7f050df14700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f05080d1870 0x7f05081b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:33.504 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:33.487 7f050d713700 1 -- 172.21.2.102:0/3481050498 >> v2:172.21.2.113:3300/0 conn(0x7f05080d1870 msgr2=0x7f05081b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:34:33.504 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:33.487 7f050d713700 1 --2- 172.21.2.102:0/3481050498 >> v2:172.21.2.113:3300/0 conn(0x7f05080d1870 0x7f05081b03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:33.504 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:33.487 7f050d713700 1 -- 172.21.2.102:0/3481050498 >> v1:172.21.2.113:6789/0 conn(0x7f05080d1c70 legacy=0x7f05080cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:33.504 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:33.487 7f050d713700 1 -- 172.21.2.102:0/3481050498 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05080f0870 con 0x7f05081b0a10 2021-02-10T11:34:33.504 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:33.487 7f050df14700 1 --2- 172.21.2.102:0/3481050498 >> v2:172.21.2.113:3300/0 conn(0x7f05080d1870 0x7f05081b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:34:33.504 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:33.487 7f050d713700 1 --2- 172.21.2.102:0/3481050498 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f05081b0a10 0x7f05080cf330 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7f04fc006650 tx=0x7f04fc009a80).ready entity=mon.2 client_cookie=9eebf35dadbfe1bb server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:33.504 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:33.487 7f050cf12700 1 -- 172.21.2.102:0/3481050498 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f04fc0009b0 con 0x7f05081b0a10 2021-02-10T11:34:33.505 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:33.487 7f050cf12700 1 -- 172.21.2.102:0/3481050498 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04fc00c0c0 con 0x7f05081b0a10 2021-02-10T11:34:33.505 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:33.487 7f050cf12700 1 -- 172.21.2.102:0/3481050498 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f04fc00c5e0 con 0x7f05081b0a10 2021-02-10T11:34:33.505 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:33.487 7f05100f8700 1 -- 172.21.2.102:0/3481050498 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f05081b0a10 msgr2=0x7f05080cf330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:33.505 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:33.487 7f05100f8700 1 --2- 172.21.2.102:0/3481050498 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f05081b0a10 0x7f05080cf330 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7f04fc006650 tx=0x7f04fc009a80).stop 2021-02-10T11:34:33.505 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:33.488 7f05100f8700 1 -- 172.21.2.102:0/3481050498 shutdown_connections 2021-02-10T11:34:33.505 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:33.488 7f05100f8700 1 --2- 172.21.2.102:0/3481050498 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f05081b0a10 0x7f05080cf330 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:33.505 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:33.488 7f05100f8700 1 --2- 172.21.2.102:0/3481050498 >> v2:172.21.2.113:3300/0 conn(0x7f05080d1870 0x7f05081b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:33.505 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:33.488 7f05100f8700 1 -- 172.21.2.102:0/3481050498 >> 172.21.2.102:0/3481050498 conn(0x7f05080ce450 msgr2=0x7f05080d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:33.505 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:33.488 7f05100f8700 1 -- 172.21.2.102:0/3481050498 shutdown_connections 2021-02-10T11:34:33.505 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:33.488 7f05100f8700 1 -- 172.21.2.102:0/3481050498 wait complete. 2021-02-10T11:34:33.507 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:33.503 7f05100f8700 1 Processor -- start 2021-02-10T11:34:33.507 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:33.503 7f05100f8700 1 -- start start 2021-02-10T11:34:33.507 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:33.503 7f05100f8700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f050818d870 0x7f05080b19b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:33.507 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:33.503 7f05100f8700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f05080b1fd0 0x7f0508099ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:33.507 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:33.503 7f05100f8700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f05080b23d0 0x7f050809a510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:33.507 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:33.503 7f05100f8700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f050818dc70 con 0x7f05080b1fd0 2021-02-10T11:34:33.507 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:33.503 7f05100f8700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0508199e60 con 0x7f050818d870 2021-02-10T11:34:33.507 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:33.503 7f05100f8700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f050809ab30 con 0x7f05080b23d0 2021-02-10T11:34:33.507 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:33.503 7f050d713700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f05080b1fd0 0x7f0508099ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:33.507 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:33.503 7f050d713700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f05080b1fd0 0x7f0508099ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:36258/0 (socket says 172.21.2.102:36258) 2021-02-10T11:34:33.507 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:33.503 7f050d713700 1 -- 172.21.2.102:0/3880099928 learned_addr learned my addr 172.21.2.102:0/3880099928 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:34:33.508 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:33.504 7f050d713700 1 -- 172.21.2.102:0/3880099928 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f05080b23d0 msgr2=0x7f050809a510 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:34:33.508 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:33.504 7f050d713700 1 --2- 172.21.2.102:0/3880099928 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f05080b23d0 0x7f050809a510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:33.508 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:33.504 7f050d713700 1 -- 172.21.2.102:0/3880099928 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f050818d870 msgr2=0x7f05080b19b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:34:33.508 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:33.504 7f050d713700 1 --2- 172.21.2.102:0/3880099928 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f050818d870 0x7f05080b19b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:33.508 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:33.504 7f050d713700 1 -- 172.21.2.102:0/3880099928 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f050809ad50 con 0x7f05080b1fd0 2021-02-10T11:34:33.508 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:33.504 7f050d713700 1 --2- 172.21.2.102:0/3880099928 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f05080b1fd0 0x7f0508099ef0 secure :-1 s=READY pgs=2567 cs=0 l=1 rev1=1 rx=0x7f04fc009630 tx=0x7f04fc00b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:33.508 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:33.504 7f04fa7fc700 1 -- 172.21.2.102:0/3880099928 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f04fc0093a0 con 0x7f05080b1fd0 2021-02-10T11:34:33.508 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:33.504 7f04fa7fc700 1 -- 172.21.2.102:0/3880099928 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04fc00bd70 con 0x7f05080b1fd0 2021-02-10T11:34:33.508 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:33.504 7f05100f8700 1 -- 172.21.2.102:0/3880099928 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f050809ad50 con 0x7f05080b1fd0 2021-02-10T11:34:33.508 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:33.504 7f04fa7fc700 1 -- 172.21.2.102:0/3880099928 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f04fc0054d0 con 0x7f05080b1fd0 2021-02-10T11:34:33.508 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:33.504 7f05100f8700 1 -- 172.21.2.102:0/3880099928 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f050809ad50 con 0x7f05080b1fd0 2021-02-10T11:34:33.508 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:33.505 7f04fa7fc700 1 -- 172.21.2.102:0/3880099928 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f04fc0054f0 con 0x7f05080b1fd0 2021-02-10T11:34:33.508 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:33.505 7f04fa7fc700 1 --2- 172.21.2.102:0/3880099928 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f04f00186c0 0x7f04f001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:33.508 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:33.505 7f04fa7fc700 1 -- 172.21.2.102:0/3880099928 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(137..137 src has 1..137) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f04fc032430 con 0x7f05080b1fd0 2021-02-10T11:34:33.509 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:33.505 7f050df14700 1 --2- 172.21.2.102:0/3880099928 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f04f00186c0 0x7f04f001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:33.509 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:33.505 7f050df14700 1 --2- 172.21.2.102:0/3880099928 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f04f00186c0 0x7f04f001abc0 crc :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:33.536 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:33.535 7f05100f8700 1 -- 172.21.2.102:0/3880099928 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05080999c0 con 0x7f05080b1fd0 2021-02-10T11:34:33.538 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:33.538 7f04fa7fc700 1 -- 172.21.2.102:0/3880099928 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f04fc0052e0 con 0x7f05080b1fd0 2021-02-10T11:34:33.666 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:33.665 7f05100f8700 1 -- 172.21.2.102:0/3880099928 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f05080999c0 con 0x7f05080b1fd0 2021-02-10T11:34:33.666 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:33.665 7f04fa7fc700 1 -- 172.21.2.102:0/3880099928 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 116 v116) v1 ==== 95+0+2108 (secure 0 0 0) 0x7f04fc0054f0 con 0x7f05080b1fd0 2021-02-10T11:34:33.667 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:34:33.667 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":116,"default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":11466,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.102:6835/1274216855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6834","nonce":1274216855},{"type":"v1","addr":"172.21.2.102:6835","nonce":1274216855}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":112}],"filesystems":[{"mdsmap":{"epoch":116,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:34:23.684779","modified":"2021-02-10 11:34:25.693957","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6770},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6770":{"gid":6770,"name":"b","rank":0,"incarnation":113,"state":"up:active","state_seq":3,"addr":"172.21.2.113:6833/3752249621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6832","nonce":3752249621},{"type":"v1","addr":"172.21.2.113:6833","nonce":3752249621}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[29],"metadata_pool":28,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":14}]} 2021-02-10T11:34:33.667 INFO:teuthology.orchestra.run.gibba002.stderr:dumped fsmap epoch 116 2021-02-10T11:34:33.668 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:33.667 7f05100f8700 1 -- 172.21.2.102:0/3880099928 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f04f00186c0 msgr2=0x7f04f001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:33.669 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:33.667 7f05100f8700 1 --2- 172.21.2.102:0/3880099928 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f04f00186c0 0x7f04f001abc0 crc :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:33.669 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:33.667 7f05100f8700 1 -- 172.21.2.102:0/3880099928 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f05080b1fd0 msgr2=0x7f0508099ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:33.669 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:33.667 7f05100f8700 1 --2- 172.21.2.102:0/3880099928 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f05080b1fd0 0x7f0508099ef0 secure :-1 s=READY pgs=2567 cs=0 l=1 rev1=1 rx=0x7f04fc009630 tx=0x7f04fc00b980).stop 2021-02-10T11:34:33.669 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:33.667 7f05100f8700 1 -- 172.21.2.102:0/3880099928 shutdown_connections 2021-02-10T11:34:33.669 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:33.667 7f05100f8700 1 --2- 172.21.2.102:0/3880099928 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f05080b23d0 0x7f050809a510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:33.669 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:33.667 7f05100f8700 1 --2- 172.21.2.102:0/3880099928 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f05080b1fd0 0x7f0508099ef0 unknown :-1 s=CLOSED pgs=2567 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:33.669 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:33.667 7f05100f8700 1 --2- 172.21.2.102:0/3880099928 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f04f00186c0 0x7f04f001abc0 unknown :-1 s=CLOSED pgs=1137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:33.669 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:33.667 7f05100f8700 1 --2- 172.21.2.102:0/3880099928 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f050818d870 0x7f05080b19b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:33.669 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:33.667 7f05100f8700 1 -- 172.21.2.102:0/3880099928 >> 172.21.2.102:0/3880099928 conn(0x7f05080f0550 msgr2=0x7f0508191730 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:33.669 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:33.668 7f05100f8700 1 -- 172.21.2.102:0/3880099928 shutdown_connections 2021-02-10T11:34:33.669 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:33.668 7f05100f8700 1 -- 172.21.2.102:0/3880099928 wait complete. 2021-02-10T11:34:33.878 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 116, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:34:23.684779', 'modified': '2021-02-10 11:34:25.693957', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6770}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6770': {'gid': 6770, 'name': 'b', 'rank': 0, 'incarnation': 113, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.113:6833/3752249621', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.113:6832', 'nonce': 3752249621}, {'type': 'v1', 'addr': '172.21.2.113:6833', 'nonce': 3752249621}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [29], 'metadata_pool': 28, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:34:33.879 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:34:33.879 DEBUG:teuthology.orchestra.run.gibba013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:34:34.278 INFO:teuthology.orchestra.run.gibba013.stdout:{ 2021-02-10T11:34:34.278 INFO:teuthology.orchestra.run.gibba013.stdout: "cluster_fsid": "68db30ea-dbce-4aef-a6df-f01c6fea0513", 2021-02-10T11:34:34.278 INFO:teuthology.orchestra.run.gibba013.stdout: "whoami": 0, 2021-02-10T11:34:34.279 INFO:teuthology.orchestra.run.gibba013.stdout: "id": 6770, 2021-02-10T11:34:34.279 INFO:teuthology.orchestra.run.gibba013.stdout: "want_state": "up:active", 2021-02-10T11:34:34.279 INFO:teuthology.orchestra.run.gibba013.stdout: "state": "up:active", 2021-02-10T11:34:34.279 INFO:teuthology.orchestra.run.gibba013.stdout: "rank_uptime": 10.590251387, 2021-02-10T11:34:34.279 INFO:teuthology.orchestra.run.gibba013.stdout: "mdsmap_epoch": 116, 2021-02-10T11:34:34.279 INFO:teuthology.orchestra.run.gibba013.stdout: "osdmap_epoch": 137, 2021-02-10T11:34:34.279 INFO:teuthology.orchestra.run.gibba013.stdout: "osdmap_epoch_barrier": 137, 2021-02-10T11:34:34.279 INFO:teuthology.orchestra.run.gibba013.stdout: "uptime": 16.242821630000002 2021-02-10T11:34:34.279 INFO:teuthology.orchestra.run.gibba013.stdout:} 2021-02-10T11:34:34.282 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "68db30ea-dbce-4aef-a6df-f01c6fea0513", "whoami": 0, "id": 6770, "want_state": "up:active", "state": "up:active", "rank_uptime": 10.590251387, "mdsmap_epoch": 116, "osdmap_epoch": 137, "osdmap_epoch_barrier": 137, "uptime": 16.242821630000002 } 2021-02-10T11:34:34.283 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2021-02-10T11:34:34.283 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2021-02-10T11:34:34.283 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@gibba002.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2021-02-10T11:34:34.283 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:34:34.283 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec mkdir -p /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:34:34.617 DEBUG:teuthology.orchestra.run.gibba002:> sudo modprobe fuse 2021-02-10T11:34:34.961 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:34:34.961 DEBUG:teuthology.orchestra.run.gibba002:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:34:35.306 INFO:teuthology.orchestra.run.gibba002.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:34:35.306 INFO:teuthology.orchestra.run.gibba002.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:34:35.307 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:34:35.307 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:34:35.307 DEBUG:teuthology.orchestra.run.gibba002:> ls /sys/fs/fuse/connections 2021-02-10T11:34:35.642 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2021-02-10T11:34:35.642 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:34:35.645 DEBUG:teuthology.orchestra.run.gibba002:> sudo modprobe fuse 2021-02-10T11:34:35.989 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:34:35.989 DEBUG:teuthology.orchestra.run.gibba002:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:34:36.056 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba002.stderr:ceph-fuse[193714]: starting ceph client2021-02-10 11:34:36.055 7f54fe159f80 -1 init, newargv = 0x55cc69ee71a0 newargc=9 2021-02-10T11:34:36.056 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba002.stderr: 2021-02-10T11:34:36.061 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba002.stderr:ceph-fuse[193714]: starting fuse 2021-02-10T11:34:36.332 INFO:teuthology.orchestra.run.gibba002.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:34:36.333 INFO:teuthology.orchestra.run.gibba002.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:34:36.333 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:34:36.334 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:34:36.334 DEBUG:teuthology.orchestra.run.gibba002:> ls /sys/fs/fuse/connections 2021-02-10T11:34:36.671 INFO:teuthology.orchestra.run.gibba002.stdout:43 2021-02-10T11:34:36.671 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2021-02-10T11:34:36.671 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:34:36.671 DEBUG:teuthology.orchestra.run.gibba002:> sudo python3 -c ' 2021-02-10T11:34:36.671 DEBUG:teuthology.orchestra.run.gibba002:> import glob 2021-02-10T11:34:36.671 DEBUG:teuthology.orchestra.run.gibba002:> import re 2021-02-10T11:34:36.671 DEBUG:teuthology.orchestra.run.gibba002:> import os 2021-02-10T11:34:36.672 DEBUG:teuthology.orchestra.run.gibba002:> import subprocess 2021-02-10T11:34:36.672 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:34:36.672 DEBUG:teuthology.orchestra.run.gibba002:> def find_socket(client_name): 2021-02-10T11:34:36.672 DEBUG:teuthology.orchestra.run.gibba002:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2021-02-10T11:34:36.672 DEBUG:teuthology.orchestra.run.gibba002:> files = glob.glob(asok_path) 2021-02-10T11:34:36.672 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:34:36.672 DEBUG:teuthology.orchestra.run.gibba002:> # Given a non-glob path, it better be there 2021-02-10T11:34:36.672 DEBUG:teuthology.orchestra.run.gibba002:> if "*" not in asok_path: 2021-02-10T11:34:36.672 DEBUG:teuthology.orchestra.run.gibba002:> assert(len(files) == 1) 2021-02-10T11:34:36.672 DEBUG:teuthology.orchestra.run.gibba002:> return files[0] 2021-02-10T11:34:36.672 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:34:36.672 DEBUG:teuthology.orchestra.run.gibba002:> for f in files: 2021-02-10T11:34:36.672 DEBUG:teuthology.orchestra.run.gibba002:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2021-02-10T11:34:36.672 DEBUG:teuthology.orchestra.run.gibba002:> if os.path.exists("/proc/{0}".format(pid)): 2021-02-10T11:34:36.673 DEBUG:teuthology.orchestra.run.gibba002:> return f 2021-02-10T11:34:36.673 DEBUG:teuthology.orchestra.run.gibba002:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2021-02-10T11:34:36.673 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:34:36.673 DEBUG:teuthology.orchestra.run.gibba002:> print(find_socket("client.0")) 2021-02-10T11:34:36.673 DEBUG:teuthology.orchestra.run.gibba002:> ' 2021-02-10T11:34:37.035 INFO:teuthology.orchestra.run.gibba002.stdout:/var/run/ceph/ceph-client.0.193714.asok 2021-02-10T11:34:37.037 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.193714.asok 2021-02-10T11:34:37.037 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:34:37.038 DEBUG:teuthology.orchestra.run.gibba002:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.193714.asok status 2021-02-10T11:34:37.427 INFO:teuthology.orchestra.run.gibba002.stdout:{ 2021-02-10T11:34:37.427 INFO:teuthology.orchestra.run.gibba002.stdout: "metadata": { 2021-02-10T11:34:37.427 INFO:teuthology.orchestra.run.gibba002.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:34:37.428 INFO:teuthology.orchestra.run.gibba002.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:34:37.428 INFO:teuthology.orchestra.run.gibba002.stdout: "entity_id": "0", 2021-02-10T11:34:37.428 INFO:teuthology.orchestra.run.gibba002.stdout: "hostname": "gibba002", 2021-02-10T11:34:37.428 INFO:teuthology.orchestra.run.gibba002.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2021-02-10T11:34:37.428 INFO:teuthology.orchestra.run.gibba002.stdout: "pid": "193714", 2021-02-10T11:34:37.428 INFO:teuthology.orchestra.run.gibba002.stdout: "root": "/" 2021-02-10T11:34:37.428 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:37.428 INFO:teuthology.orchestra.run.gibba002.stdout: "dentry_count": 0, 2021-02-10T11:34:37.428 INFO:teuthology.orchestra.run.gibba002.stdout: "dentry_pinned_count": 0, 2021-02-10T11:34:37.428 INFO:teuthology.orchestra.run.gibba002.stdout: "id": 11616, 2021-02-10T11:34:37.428 INFO:teuthology.orchestra.run.gibba002.stdout: "inst": { 2021-02-10T11:34:37.428 INFO:teuthology.orchestra.run.gibba002.stdout: "name": { 2021-02-10T11:34:37.428 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "client", 2021-02-10T11:34:37.429 INFO:teuthology.orchestra.run.gibba002.stdout: "num": 11616 2021-02-10T11:34:37.429 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:37.429 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": { 2021-02-10T11:34:37.429 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:34:37.429 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:0", 2021-02-10T11:34:37.429 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 1869552306 2021-02-10T11:34:37.429 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:34:37.429 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:37.429 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": { 2021-02-10T11:34:37.429 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:34:37.429 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:0", 2021-02-10T11:34:37.429 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 1869552306 2021-02-10T11:34:37.429 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:37.430 INFO:teuthology.orchestra.run.gibba002.stdout: "inst_str": "client.11616 172.21.2.102:0/1869552306", 2021-02-10T11:34:37.430 INFO:teuthology.orchestra.run.gibba002.stdout: "addr_str": "172.21.2.102:0/1869552306", 2021-02-10T11:34:37.430 INFO:teuthology.orchestra.run.gibba002.stdout: "inode_count": 1, 2021-02-10T11:34:37.430 INFO:teuthology.orchestra.run.gibba002.stdout: "mds_epoch": 116, 2021-02-10T11:34:37.430 INFO:teuthology.orchestra.run.gibba002.stdout: "osd_epoch": 137, 2021-02-10T11:34:37.430 INFO:teuthology.orchestra.run.gibba002.stdout: "osd_epoch_barrier": 0, 2021-02-10T11:34:37.430 INFO:teuthology.orchestra.run.gibba002.stdout: "blacklisted": false 2021-02-10T11:34:37.430 INFO:teuthology.orchestra.run.gibba002.stdout:} 2021-02-10T11:34:37.431 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:34:37.432 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:34:37.432 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:34:37.769 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:34:37.769 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:34:37.769 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:34:37.769 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:34:38.116 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:34:38.544 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:38.527 7f1697972700 1 Processor -- start 2021-02-10T11:34:38.544 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:38.527 7f1697972700 1 -- start start 2021-02-10T11:34:38.545 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:38.528 7f1697972700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f16900d1870 0x7f16901b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:38.545 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:38.528 7f1697972700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f16901b0a10 0x7f16900cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:38.545 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:38.528 7f1697972700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16900f0b10 con 0x7f16900d1c70 2021-02-10T11:34:38.545 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:38.528 7f1697972700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16900fe230 con 0x7f16901b0a10 2021-02-10T11:34:38.545 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:38.528 7f1697972700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16900e5ed0 con 0x7f16900d1870 2021-02-10T11:34:38.545 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:38.528 7f1694f8d700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f16901b0a10 0x7f16900cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:38.545 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:38.528 7f1694f8d700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f16901b0a10 0x7f16900cf330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:36286/0 (socket says 172.21.2.102:36286) 2021-02-10T11:34:38.545 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:38.528 7f1694f8d700 1 -- 172.21.2.102:0/689278811 learned_addr learned my addr 172.21.2.102:0/689278811 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:34:38.545 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:38.528 7f169578e700 1 --2- 172.21.2.102:0/689278811 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f16900d1870 0x7f16901b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:38.545 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:38.528 7f1694f8d700 1 -- 172.21.2.102:0/689278811 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f16900d1870 msgr2=0x7f16901b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:34:38.545 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:38.528 7f1694f8d700 1 --2- 172.21.2.102:0/689278811 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f16900d1870 0x7f16901b03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:38.546 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:38.528 7f1694f8d700 1 -- 172.21.2.102:0/689278811 >> v1:172.21.2.113:6789/0 conn(0x7f16900d1c70 legacy=0x7f16900cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:38.546 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:38.528 7f1694f8d700 1 -- 172.21.2.102:0/689278811 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16900f0650 con 0x7f16901b0a10 2021-02-10T11:34:38.546 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:38.528 7f1694f8d700 1 --2- 172.21.2.102:0/689278811 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f16901b0a10 0x7f16900cf330 secure :-1 s=READY pgs=2571 cs=0 l=1 rev1=1 rx=0x7f1680006700 tx=0x7f1680009b80).ready entity=mon.0 client_cookie=56ec5b0212e5b456 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:38.546 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:38.528 7f1687fff700 1 -- 172.21.2.102:0/689278811 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f168000c0c0 con 0x7f16901b0a10 2021-02-10T11:34:38.546 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:38.528 7f1687fff700 1 -- 172.21.2.102:0/689278811 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f168000c440 con 0x7f16901b0a10 2021-02-10T11:34:38.546 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:38.528 7f1687fff700 1 -- 172.21.2.102:0/689278811 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f168000c960 con 0x7f16901b0a10 2021-02-10T11:34:38.546 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:38.528 7f1697972700 1 -- 172.21.2.102:0/689278811 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f16901b0a10 msgr2=0x7f16900cf330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:38.546 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:38.528 7f1697972700 1 --2- 172.21.2.102:0/689278811 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f16901b0a10 0x7f16900cf330 secure :-1 s=READY pgs=2571 cs=0 l=1 rev1=1 rx=0x7f1680006700 tx=0x7f1680009b80).stop 2021-02-10T11:34:38.546 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:38.529 7f1697972700 1 -- 172.21.2.102:0/689278811 shutdown_connections 2021-02-10T11:34:38.546 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:38.529 7f1697972700 1 --2- 172.21.2.102:0/689278811 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f16901b0a10 0x7f16900cf330 unknown :-1 s=CLOSED pgs=2571 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:38.546 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:38.529 7f1697972700 1 --2- 172.21.2.102:0/689278811 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f16900d1870 0x7f16901b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:38.547 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:38.529 7f1697972700 1 -- 172.21.2.102:0/689278811 >> 172.21.2.102:0/689278811 conn(0x7f16900ce450 msgr2=0x7f16900d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:38.547 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:38.529 7f1697972700 1 -- 172.21.2.102:0/689278811 shutdown_connections 2021-02-10T11:34:38.547 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:38.529 7f1697972700 1 -- 172.21.2.102:0/689278811 wait complete. 2021-02-10T11:34:38.548 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:38.544 7f1697972700 1 Processor -- start 2021-02-10T11:34:38.549 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:38.544 7f1697972700 1 -- start start 2021-02-10T11:34:38.549 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:38.544 7f1697972700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f16901a4a60 0x7f16900c1c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:38.549 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:38.544 7f1697972700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f16900c2290 0x7f16900c2690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:38.549 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:38.544 7f1697972700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f16900c2cb0 0x7f16900bfd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:38.549 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:38.544 7f1697972700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16900f05b0 con 0x7f16900c2cb0 2021-02-10T11:34:38.549 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:38.544 7f1697972700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16900f07d0 con 0x7f16901a4a60 2021-02-10T11:34:38.549 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:38.544 7f1697972700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16900c0340 con 0x7f16900c2290 2021-02-10T11:34:38.549 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:38.545 7f1695f8f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f16900c2cb0 0x7f16900bfd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:38.550 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:38.545 7f1695f8f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f16900c2cb0 0x7f16900bfd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:36294/0 (socket says 172.21.2.102:36294) 2021-02-10T11:34:38.550 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:38.545 7f1695f8f700 1 -- 172.21.2.102:0/1099106169 learned_addr learned my addr 172.21.2.102:0/1099106169 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:34:38.550 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:38.545 7f1694f8d700 1 --2- 172.21.2.102:0/1099106169 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f16900c2290 0x7f16900c2690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:38.550 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:38.545 7f169578e700 1 --2- 172.21.2.102:0/1099106169 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f16901a4a60 0x7f16900c1c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:38.550 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:38.545 7f1695f8f700 1 -- 172.21.2.102:0/1099106169 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f16900c2290 msgr2=0x7f16900c2690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:38.550 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:38.545 7f1695f8f700 1 --2- 172.21.2.102:0/1099106169 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f16900c2290 0x7f16900c2690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:38.550 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:38.545 7f1695f8f700 1 -- 172.21.2.102:0/1099106169 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f16901a4a60 msgr2=0x7f16900c1c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:38.550 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:38.545 7f1695f8f700 1 --2- 172.21.2.102:0/1099106169 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f16901a4a60 0x7f16900c1c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:38.550 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:38.545 7f1695f8f700 1 -- 172.21.2.102:0/1099106169 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16900c0560 con 0x7f16900c2cb0 2021-02-10T11:34:38.550 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:38.545 7f1694f8d700 1 --2- 172.21.2.102:0/1099106169 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f16900c2290 0x7f16900c2690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:34:38.550 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:38.545 7f1695f8f700 1 --2- 172.21.2.102:0/1099106169 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f16900c2cb0 0x7f16900bfd20 secure :-1 s=READY pgs=2572 cs=0 l=1 rev1=1 rx=0x7f1688009940 tx=0x7f168800bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:38.550 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:38.545 7f1685ffb700 1 -- 172.21.2.102:0/1099106169 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1688000ce0 con 0x7f16900c2cb0 2021-02-10T11:34:38.551 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:38.545 7f1685ffb700 1 -- 172.21.2.102:0/1099106169 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f168800c2f0 con 0x7f16900c2cb0 2021-02-10T11:34:38.551 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:38.545 7f1685ffb700 1 -- 172.21.2.102:0/1099106169 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f168800c810 con 0x7f16900c2cb0 2021-02-10T11:34:38.551 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:38.545 7f1697972700 1 -- 172.21.2.102:0/1099106169 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16900c0560 con 0x7f16900c2cb0 2021-02-10T11:34:38.551 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:38.545 7f1697972700 1 -- 172.21.2.102:0/1099106169 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16900c0af0 con 0x7f16900c2cb0 2021-02-10T11:34:38.551 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:38.546 7f1685ffb700 1 -- 172.21.2.102:0/1099106169 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1688001220 con 0x7f16900c2cb0 2021-02-10T11:34:38.551 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:38.546 7f1685ffb700 1 --2- 172.21.2.102:0/1099106169 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f16700186c0 0x7f167001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:38.551 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:38.546 7f1685ffb700 1 -- 172.21.2.102:0/1099106169 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(137..137 src has 1..137) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f16880327c0 con 0x7f16900c2cb0 2021-02-10T11:34:38.551 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:38.546 7f169578e700 1 --2- 172.21.2.102:0/1099106169 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f16700186c0 0x7f167001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:38.551 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:38.546 7f169578e700 1 --2- 172.21.2.102:0/1099106169 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f16700186c0 0x7f167001abc0 crc :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:38.578 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:38.577 7f1697972700 1 -- 172.21.2.102:0/1099106169 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16900f0b10 con 0x7f16900c2cb0 2021-02-10T11:34:38.580 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:38.580 7f1685ffb700 1 -- 172.21.2.102:0/1099106169 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f1688002ef0 con 0x7f16900c2cb0 2021-02-10T11:34:38.706 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:38.705 7f1697972700 1 -- 172.21.2.102:0/1099106169 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f16900f0b10 con 0x7f16900c2cb0 2021-02-10T11:34:38.707 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:38.706 7f1685ffb700 1 -- 172.21.2.102:0/1099106169 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v137) v1 ==== 74+0+14007 (secure 0 0 0) 0x7f1688000cc0 con 0x7f16900c2cb0 2021-02-10T11:34:38.707 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:34:38.707 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":137,"fsid":"68db30ea-dbce-4aef-a6df-f01c6fea0513","created":"2021-02-10 11:03:06.331557","modified":"2021-02-10 11:34:24.687007","last_up_change":"2021-02-10 11:03:32.690562","last_in_change":"2021-02-10 11:03:32.690562","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":29,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:03:37.322519","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":28,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:34:21.649291","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"136","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":29,"pool_name":"cephfs_data","create_time":"2021-02-10 11:34:22.675483","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"137","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"be691b4d-a039-408a-af80-9d53f02b6f1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":134,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6818","nonce":36088},{"type":"v1","addr":"172.21.2.102:6820","nonce":36088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6822","nonce":36088},{"type":"v1","addr":"172.21.2.102:6824","nonce":36088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6830","nonce":36088},{"type":"v1","addr":"172.21.2.102:6832","nonce":36088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6826","nonce":36088},{"type":"v1","addr":"172.21.2.102:6828","nonce":36088}]},"public_addr":"172.21.2.102:6820/36088","cluster_addr":"172.21.2.102:6824/36088","heartbeat_back_addr":"172.21.2.102:6832/36088","heartbeat_front_addr":"172.21.2.102:6828/36088","state":["exists","up"]},{"osd":1,"uuid":"64e3ccc8-9c69-4d7c-865e-af8e83822635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":135,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6819","nonce":36087},{"type":"v1","addr":"172.21.2.102:6821","nonce":36087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6823","nonce":36087},{"type":"v1","addr":"172.21.2.102:6825","nonce":36087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6831","nonce":36087},{"type":"v1","addr":"172.21.2.102:6833","nonce":36087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6827","nonce":36087},{"type":"v1","addr":"172.21.2.102:6829","nonce":36087}]},"public_addr":"172.21.2.102:6821/36087","cluster_addr":"172.21.2.102:6825/36087","heartbeat_back_addr":"172.21.2.102:6833/36087","heartbeat_front_addr":"172.21.2.102:6829/36087","state":["exists","up"]},{"osd":2,"uuid":"153700af-747f-481f-bdf4-d6072d3a842f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6810","nonce":36089},{"type":"v1","addr":"172.21.2.102:6811","nonce":36089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6812","nonce":36089},{"type":"v1","addr":"172.21.2.102:6813","nonce":36089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6816","nonce":36089},{"type":"v1","addr":"172.21.2.102:6817","nonce":36089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6814","nonce":36089},{"type":"v1","addr":"172.21.2.102:6815","nonce":36089}]},"public_addr":"172.21.2.102:6811/36089","cluster_addr":"172.21.2.102:6813/36089","heartbeat_back_addr":"172.21.2.102:6817/36089","heartbeat_front_addr":"172.21.2.102:6815/36089","state":["exists","up"]},{"osd":3,"uuid":"cbc678f4-83a9-45e6-8830-2f7c232b1ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6802","nonce":36090},{"type":"v1","addr":"172.21.2.102:6803","nonce":36090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6804","nonce":36090},{"type":"v1","addr":"172.21.2.102:6805","nonce":36090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6808","nonce":36090},{"type":"v1","addr":"172.21.2.102:6809","nonce":36090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6806","nonce":36090},{"type":"v1","addr":"172.21.2.102:6807","nonce":36090}]},"public_addr":"172.21.2.102:6803/36090","cluster_addr":"172.21.2.102:6805/36090","heartbeat_back_addr":"172.21.2.102:6809/36090","heartbeat_front_addr":"172.21.2.102:6807/36090","state":["exists","up"]},{"osd":4,"uuid":"c01e4008-b258-4f10-afd2-9e7f37214351","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":135,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6808","nonce":36368},{"type":"v1","addr":"172.21.2.113:6811","nonce":36368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6814","nonce":36368},{"type":"v1","addr":"172.21.2.113:6818","nonce":36368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6826","nonce":36368},{"type":"v1","addr":"172.21.2.113:6828","nonce":36368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6820","nonce":36368},{"type":"v1","addr":"172.21.2.113:6824","nonce":36368}]},"public_addr":"172.21.2.113:6811/36368","cluster_addr":"172.21.2.113:6818/36368","heartbeat_back_addr":"172.21.2.113:6828/36368","heartbeat_front_addr":"172.21.2.113:6824/36368","state":["exists","up"]},{"osd":5,"uuid":"551f6007-daf8-4a98-8ee0-c2b16bcccb5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":134,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6802","nonce":36366},{"type":"v1","addr":"172.21.2.113:6805","nonce":36366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6807","nonce":36366},{"type":"v1","addr":"172.21.2.113:6810","nonce":36366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6819","nonce":36366},{"type":"v1","addr":"172.21.2.113:6822","nonce":36366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6812","nonce":36366},{"type":"v1","addr":"172.21.2.113:6815","nonce":36366}]},"public_addr":"172.21.2.113:6805/36366","cluster_addr":"172.21.2.113:6810/36366","heartbeat_back_addr":"172.21.2.113:6822/36366","heartbeat_front_addr":"172.21.2.113:6815/36366","state":["exists","up"]},{"osd":6,"uuid":"bf67c040-0e50-48be-b102-c3943bc9ebef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":135,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6800","nonce":36367},{"type":"v1","addr":"172.21.2.113:6801","nonce":36367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6803","nonce":36367},{"type":"v1","addr":"172.21.2.113:6804","nonce":36367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6813","nonce":36367},{"type":"v1","addr":"172.21.2.113:6816","nonce":36367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6806","nonce":36367},{"type":"v1","addr":"172.21.2.113:6809","nonce":36367}]},"public_addr":"172.21.2.113:6801/36367","cluster_addr":"172.21.2.113:6804/36367","heartbeat_back_addr":"172.21.2.113:6816/36367","heartbeat_front_addr":"172.21.2.113:6809/36367","state":["exists","up"]},{"osd":7,"uuid":"f5461f80-018d-428f-be89-2eb2bcbe7bb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":135,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6817","nonce":36365},{"type":"v1","addr":"172.21.2.113:6821","nonce":36365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6823","nonce":36365},{"type":"v1","addr":"172.21.2.113:6825","nonce":36365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6830","nonce":36365},{"type":"v1","addr":"172.21.2.113:6831","nonce":36365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6827","nonce":36365},{"type":"v1","addr":"172.21.2.113:6829","nonce":36365}]},"public_addr":"172.21.2.113:6821/36365","cluster_addr":"172.21.2.113:6825/36365","heartbeat_back_addr":"172.21.2.113:6831/36365","heartbeat_front_addr":"172.21.2.113:6829/36365","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:34:38.708 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:38.707 7f1697972700 1 -- 172.21.2.102:0/1099106169 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f16700186c0 msgr2=0x7f167001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:38.708 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:38.707 7f1697972700 1 --2- 172.21.2.102:0/1099106169 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f16700186c0 0x7f167001abc0 crc :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:38.708 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:38.707 7f1697972700 1 -- 172.21.2.102:0/1099106169 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f16900c2cb0 msgr2=0x7f16900bfd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:38.708 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:38.707 7f1697972700 1 --2- 172.21.2.102:0/1099106169 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f16900c2cb0 0x7f16900bfd20 secure :-1 s=READY pgs=2572 cs=0 l=1 rev1=1 rx=0x7f1688009940 tx=0x7f168800bcf0).stop 2021-02-10T11:34:38.709 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:38.708 7f1697972700 1 -- 172.21.2.102:0/1099106169 shutdown_connections 2021-02-10T11:34:38.709 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:38.708 7f1697972700 1 --2- 172.21.2.102:0/1099106169 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f16900c2cb0 0x7f16900bfd20 unknown :-1 s=CLOSED pgs=2572 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:38.709 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:38.708 7f1697972700 1 --2- 172.21.2.102:0/1099106169 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f16900c2290 0x7f16900c2690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:38.709 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:38.708 7f1697972700 1 --2- 172.21.2.102:0/1099106169 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f16700186c0 0x7f167001abc0 unknown :-1 s=CLOSED pgs=1138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:38.709 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:38.708 7f1697972700 1 --2- 172.21.2.102:0/1099106169 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f16901a4a60 0x7f16900c1c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:38.709 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:38.708 7f1697972700 1 -- 172.21.2.102:0/1099106169 >> 172.21.2.102:0/1099106169 conn(0x7f16900ebe30 msgr2=0x7f16900bf200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:38.709 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:38.708 7f1697972700 1 -- 172.21.2.102:0/1099106169 shutdown_connections 2021-02-10T11:34:38.709 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:38.708 7f1697972700 1 -- 172.21.2.102:0/1099106169 wait complete. 2021-02-10T11:34:38.717 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:03:37.322519', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 28, 'pool_name': 'cephfs_metadata', 'create_time': '2021-02-10 11:34:21.649291', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '136', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 29, 'pool_name': 'cephfs_data', 'create_time': '2021-02-10 11:34:22.675483', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '137', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-02-10T11:34:38.717 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:34:39.150 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:39.132 7fd23b6b6700 1 Processor -- start 2021-02-10T11:34:39.150 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:39.132 7fd23b6b6700 1 -- start start 2021-02-10T11:34:39.150 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:39.133 7fd23b6b6700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fd2340d1870 0x7fd2341b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:39.150 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:39.133 7fd23b6b6700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fd2340cff70 0x7fd234109130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:39.150 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:39.133 7fd23b6b6700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2340fe1c0 con 0x7fd2341b0ae0 2021-02-10T11:34:39.150 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:39.133 7fd23b6b6700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd234109ad0 con 0x7fd2340d1870 2021-02-10T11:34:39.151 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:39.133 7fd23b6b6700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd234109750 con 0x7fd2340cff70 2021-02-10T11:34:39.151 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:39.133 7fd2394d2700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fd2340d1870 0x7fd2341b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:39.151 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:39.133 7fd2394d2700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fd2340d1870 0x7fd2341b04c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:36298/0 (socket says 172.21.2.102:36298) 2021-02-10T11:34:39.151 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:39.133 7fd2394d2700 1 -- 172.21.2.102:0/448850305 learned_addr learned my addr 172.21.2.102:0/448850305 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:34:39.151 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:39.133 7fd2394d2700 1 -- 172.21.2.102:0/448850305 >> v2:172.21.2.113:3300/0 conn(0x7fd2340cff70 msgr2=0x7fd234109130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:34:39.151 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:39.133 7fd2394d2700 1 --2- 172.21.2.102:0/448850305 >> v2:172.21.2.113:3300/0 conn(0x7fd2340cff70 0x7fd234109130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:39.151 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:39.133 7fd2394d2700 1 -- 172.21.2.102:0/448850305 >> v1:172.21.2.113:6789/0 conn(0x7fd2341b0ae0 legacy=0x7fd234108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:34:39.151 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:39.133 7fd2394d2700 1 -- 172.21.2.102:0/448850305 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd234088070 con 0x7fd2340d1870 2021-02-10T11:34:39.151 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:39.134 7fd2394d2700 1 --2- 172.21.2.102:0/448850305 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd2340d1870 0x7fd2341b04c0 secure :-1 s=READY pgs=2573 cs=0 l=1 rev1=1 rx=0x7fd228006700 tx=0x7fd228009b80).ready entity=mon.0 client_cookie=75b8dd43a0ab4102 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:39.151 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:39.134 7fd227fff700 1 -- 172.21.2.102:0/448850305 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd22800c0c0 con 0x7fd2340d1870 2021-02-10T11:34:39.151 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:39.134 7fd227fff700 1 -- 172.21.2.102:0/448850305 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd22800c440 con 0x7fd2340d1870 2021-02-10T11:34:39.151 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:39.134 7fd227fff700 1 -- 172.21.2.102:0/448850305 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd22800c8a0 con 0x7fd2340d1870 2021-02-10T11:34:39.152 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:39.134 7fd23b6b6700 1 -- 172.21.2.102:0/448850305 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd2340d1870 msgr2=0x7fd2341b04c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:39.152 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:39.134 7fd23b6b6700 1 --2- 172.21.2.102:0/448850305 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd2340d1870 0x7fd2341b04c0 secure :-1 s=READY pgs=2573 cs=0 l=1 rev1=1 rx=0x7fd228006700 tx=0x7fd228009b80).stop 2021-02-10T11:34:39.152 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:39.134 7fd23b6b6700 1 -- 172.21.2.102:0/448850305 shutdown_connections 2021-02-10T11:34:39.152 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:39.134 7fd23b6b6700 1 --2- 172.21.2.102:0/448850305 >> v2:172.21.2.113:3300/0 conn(0x7fd2340cff70 0x7fd234109130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:39.152 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:39.134 7fd23b6b6700 1 --2- 172.21.2.102:0/448850305 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd2340d1870 0x7fd2341b04c0 unknown :-1 s=CLOSED pgs=2573 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:39.152 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:39.134 7fd23b6b6700 1 -- 172.21.2.102:0/448850305 >> 172.21.2.102:0/448850305 conn(0x7fd2340ce420 msgr2=0x7fd2340d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:39.152 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:39.135 7fd23b6b6700 1 -- 172.21.2.102:0/448850305 shutdown_connections 2021-02-10T11:34:39.152 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:39.135 7fd23b6b6700 1 -- 172.21.2.102:0/448850305 wait complete. 2021-02-10T11:34:39.152 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:39.149 7fd23b6b6700 1 Processor -- start 2021-02-10T11:34:39.152 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:39.149 7fd23b6b6700 1 -- start start 2021-02-10T11:34:39.154 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:39.150 7fd23b6b6700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd2340c2370 0x7fd23418f870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:39.154 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:39.150 7fd23b6b6700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd234084cb0 0x7fd2341a7470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:39.154 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:39.150 7fd23b6b6700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd2341a7a90 0x7fd234085700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:39.154 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:39.150 7fd23b6b6700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2340be840 con 0x7fd2340c2370 2021-02-10T11:34:39.154 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:39.150 7fd23b6b6700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2340c29a0 con 0x7fd2341a7a90 2021-02-10T11:34:39.154 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:39.150 7fd23b6b6700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2341a7e90 con 0x7fd234084cb0 2021-02-10T11:34:39.154 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:39.150 7fd239cd3700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd2341a7a90 0x7fd234085700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:39.154 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:39.150 7fd239cd3700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd2341a7a90 0x7fd234085700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.113:3300/0 says I am v2:172.21.2.102:39732/0 (socket says 172.21.2.102:39732) 2021-02-10T11:34:39.154 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:39.150 7fd239cd3700 1 -- 172.21.2.102:0/1751602114 learned_addr learned my addr 172.21.2.102:0/1751602114 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:34:39.154 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:39.150 7fd2394d2700 1 --2- 172.21.2.102:0/1751602114 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd2340c2370 0x7fd23418f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:39.155 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:39.150 7fd238cd1700 1 --2- 172.21.2.102:0/1751602114 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd234084cb0 0x7fd2341a7470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:39.155 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:39.150 7fd2394d2700 1 -- 172.21.2.102:0/1751602114 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd234084cb0 msgr2=0x7fd2341a7470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:39.155 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:39.150 7fd2394d2700 1 --2- 172.21.2.102:0/1751602114 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd234084cb0 0x7fd2341a7470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:39.155 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:39.150 7fd2394d2700 1 -- 172.21.2.102:0/1751602114 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd2341a7a90 msgr2=0x7fd234085700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:39.155 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:39.150 7fd2394d2700 1 --2- 172.21.2.102:0/1751602114 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd2341a7a90 0x7fd234085700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:39.155 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:39.150 7fd2394d2700 1 -- 172.21.2.102:0/1751602114 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2341a80b0 con 0x7fd2340c2370 2021-02-10T11:34:39.155 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:39.151 7fd2394d2700 1 --2- 172.21.2.102:0/1751602114 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd2340c2370 0x7fd23418f870 secure :-1 s=READY pgs=2574 cs=0 l=1 rev1=1 rx=0x7fd2280096d0 tx=0x7fd22800b8c0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:39.155 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:39.151 7fd225ffb700 1 -- 172.21.2.102:0/1751602114 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd2280009b0 con 0x7fd2340c2370 2021-02-10T11:34:39.155 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:39.151 7fd225ffb700 1 -- 172.21.2.102:0/1751602114 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd22800bcb0 con 0x7fd2340c2370 2021-02-10T11:34:39.155 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:39.151 7fd225ffb700 1 -- 172.21.2.102:0/1751602114 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd2280022b0 con 0x7fd2340c2370 2021-02-10T11:34:39.155 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:39.151 7fd23b6b6700 1 -- 172.21.2.102:0/1751602114 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2341a80b0 con 0x7fd2340c2370 2021-02-10T11:34:39.155 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:39.151 7fd23b6b6700 1 -- 172.21.2.102:0/1751602114 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2341a80b0 con 0x7fd2340c2370 2021-02-10T11:34:39.155 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:39.151 7fd225ffb700 1 -- 172.21.2.102:0/1751602114 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd228000c90 con 0x7fd2340c2370 2021-02-10T11:34:39.156 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:39.152 7fd225ffb700 1 --2- 172.21.2.102:0/1751602114 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd2140186c0 0x7fd21401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:39.156 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:39.152 7fd225ffb700 1 -- 172.21.2.102:0/1751602114 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(137..137 src has 1..137) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fd228032310 con 0x7fd2340c2370 2021-02-10T11:34:39.156 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:39.152 7fd238cd1700 1 --2- 172.21.2.102:0/1751602114 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd2140186c0 0x7fd21401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:39.156 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:39.152 7fd238cd1700 1 --2- 172.21.2.102:0/1751602114 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd2140186c0 0x7fd21401abc0 crc :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:39.183 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:39.183 7fd23b6b6700 1 -- 172.21.2.102:0/1751602114 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2340be840 con 0x7fd2340c2370 2021-02-10T11:34:39.186 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:39.185 7fd225ffb700 1 -- 172.21.2.102:0/1751602114 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd22802cf70 con 0x7fd2340c2370 2021-02-10T11:34:39.313 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:39.312 7fd23b6b6700 1 -- 172.21.2.102:0/1751602114 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7fd2340be840 con 0x7fd2340c2370 2021-02-10T11:34:39.314 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:39.313 7fd225ffb700 1 -- 172.21.2.102:0/1751602114 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v137) v1 ==== 92+0+10 (secure 0 0 0) 0x7fd22802cf70 con 0x7fd2340c2370 2021-02-10T11:34:39.314 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:34:39.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:39.315 7fd23b6b6700 1 -- 172.21.2.102:0/1751602114 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd2140186c0 msgr2=0x7fd21401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:39.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:39.315 7fd23b6b6700 1 --2- 172.21.2.102:0/1751602114 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd2140186c0 0x7fd21401abc0 crc :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:39.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:39.315 7fd23b6b6700 1 -- 172.21.2.102:0/1751602114 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd2340c2370 msgr2=0x7fd23418f870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:39.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:39.315 7fd23b6b6700 1 --2- 172.21.2.102:0/1751602114 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd2340c2370 0x7fd23418f870 secure :-1 s=READY pgs=2574 cs=0 l=1 rev1=1 rx=0x7fd2280096d0 tx=0x7fd22800b8c0).stop 2021-02-10T11:34:39.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:39.315 7fd23b6b6700 1 -- 172.21.2.102:0/1751602114 shutdown_connections 2021-02-10T11:34:39.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:39.315 7fd23b6b6700 1 --2- 172.21.2.102:0/1751602114 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd2140186c0 0x7fd21401abc0 unknown :-1 s=CLOSED pgs=1139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:39.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:39.315 7fd23b6b6700 1 --2- 172.21.2.102:0/1751602114 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd2341a7a90 0x7fd234085700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:39.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:39.315 7fd23b6b6700 1 --2- 172.21.2.102:0/1751602114 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd234084cb0 0x7fd2341a7470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:39.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:39.315 7fd23b6b6700 1 --2- 172.21.2.102:0/1751602114 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd2340c2370 0x7fd23418f870 unknown :-1 s=CLOSED pgs=2574 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:39.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:39.315 7fd23b6b6700 1 -- 172.21.2.102:0/1751602114 >> 172.21.2.102:0/1751602114 conn(0x7fd2340b5dc0 msgr2=0x7fd2340c2d90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:39.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:39.315 7fd23b6b6700 1 -- 172.21.2.102:0/1751602114 shutdown_connections 2021-02-10T11:34:39.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:39.315 7fd23b6b6700 1 -- 172.21.2.102:0/1751602114 wait complete. 2021-02-10T11:34:39.324 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-02-10T11:34:39.757 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:39.739 7fc2cd2b6700 1 Processor -- start 2021-02-10T11:34:39.757 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:39.739 7fc2cd2b6700 1 -- start start 2021-02-10T11:34:39.757 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:39.740 7fc2cd2b6700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc2c80d1c70 0x7fc2c80cef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:39.758 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:39.740 7fc2cd2b6700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fc2c80cf580 0x7fc2c80cf980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:39.758 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:39.740 7fc2cd2b6700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2c80fe1c0 con 0x7fc2c80d1870 2021-02-10T11:34:39.758 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:39.740 7fc2cd2b6700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2c81b0cb0 con 0x7fc2c80cf580 2021-02-10T11:34:39.758 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:39.740 7fc2cd2b6700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2c8109a60 con 0x7fc2c80d1c70 2021-02-10T11:34:39.758 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:39.740 7fc2c707d700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fc2c80d1870 0x7fc2c81b04c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:56766/0 (socket says 172.21.2.102:56766) 2021-02-10T11:34:39.758 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:39.740 7fc2c707d700 1 -- 172.21.2.102:0/4190582843 learned_addr learned my addr 172.21.2.102:0/4190582843 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:34:39.758 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:39.740 7fc2c607b700 1 -- 172.21.2.102:0/4190582843 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2693463641 0 0) 0x7fc2b0000ce0 con 0x7fc2c80d1870 2021-02-10T11:34:39.758 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:39.740 7fc2c687c700 1 --2- 172.21.2.102:0/4190582843 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc2c80d1c70 0x7fc2c80cef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:39.758 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:39.740 7fc2c787e700 1 --2- 172.21.2.102:0/4190582843 >> v2:172.21.2.113:3300/0 conn(0x7fc2c80cf580 0x7fc2c80cf980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:39.758 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:39.740 7fc2c607b700 1 -- 172.21.2.102:0/4190582843 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2b4002fe0 con 0x7fc2c80d1870 2021-02-10T11:34:39.758 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:39.741 7fc2c687c700 1 -- 172.21.2.102:0/4190582843 >> v2:172.21.2.113:3300/0 conn(0x7fc2c80cf580 msgr2=0x7fc2c80cf980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:34:39.758 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:39.741 7fc2c687c700 1 --2- 172.21.2.102:0/4190582843 >> v2:172.21.2.113:3300/0 conn(0x7fc2c80cf580 0x7fc2c80cf980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:39.759 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:39.741 7fc2c687c700 1 -- 172.21.2.102:0/4190582843 >> v1:172.21.2.102:6789/0 conn(0x7fc2c80d1870 legacy=0x7fc2c81b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:39.759 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:39.741 7fc2c687c700 1 -- 172.21.2.102:0/4190582843 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2c8088070 con 0x7fc2c80d1c70 2021-02-10T11:34:39.759 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:39.741 7fc2c687c700 1 --2- 172.21.2.102:0/4190582843 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc2c80d1c70 0x7fc2c80cef60 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 rx=0x7fc2b8006650 tx=0x7fc2b8009a80).ready entity=mon.2 client_cookie=4a472e40653fb2ef server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:39.759 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:39.741 7fc2c607b700 1 -- 172.21.2.102:0/4190582843 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc2b800c0c0 con 0x7fc2c80d1c70 2021-02-10T11:34:39.759 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:39.741 7fc2c607b700 1 -- 172.21.2.102:0/4190582843 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2b800c340 con 0x7fc2c80d1c70 2021-02-10T11:34:39.759 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:39.741 7fc2c607b700 1 -- 172.21.2.102:0/4190582843 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc2b800c860 con 0x7fc2c80d1c70 2021-02-10T11:34:39.759 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:39.741 7fc2cd2b6700 1 -- 172.21.2.102:0/4190582843 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc2c80d1c70 msgr2=0x7fc2c80cef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:39.759 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:39.741 7fc2cd2b6700 1 --2- 172.21.2.102:0/4190582843 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc2c80d1c70 0x7fc2c80cef60 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 rx=0x7fc2b8006650 tx=0x7fc2b8009a80).stop 2021-02-10T11:34:39.759 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:39.742 7fc2cd2b6700 1 -- 172.21.2.102:0/4190582843 shutdown_connections 2021-02-10T11:34:39.759 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:39.742 7fc2cd2b6700 1 --2- 172.21.2.102:0/4190582843 >> v2:172.21.2.113:3300/0 conn(0x7fc2c80cf580 0x7fc2c80cf980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:39.759 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:39.742 7fc2cd2b6700 1 --2- 172.21.2.102:0/4190582843 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc2c80d1c70 0x7fc2c80cef60 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:39.759 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:39.742 7fc2cd2b6700 1 -- 172.21.2.102:0/4190582843 >> 172.21.2.102:0/4190582843 conn(0x7fc2c80ce420 msgr2=0x7fc2c80d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:39.759 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:39.742 7fc2cd2b6700 1 -- 172.21.2.102:0/4190582843 shutdown_connections 2021-02-10T11:34:39.760 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:39.742 7fc2cd2b6700 1 -- 172.21.2.102:0/4190582843 wait complete. 2021-02-10T11:34:39.761 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:39.757 7fc2cd2b6700 1 Processor -- start 2021-02-10T11:34:39.761 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:39.757 7fc2cd2b6700 1 -- start start 2021-02-10T11:34:39.761 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:39.757 7fc2cd2b6700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc2c809bcc0 0x7fc2c809c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:39.761 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:39.757 7fc2cd2b6700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc2c80a0820 0x7fc2c80a0c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:39.761 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:39.757 7fc2cd2b6700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc2c80a1240 0x7fc2c80858d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:39.762 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:39.757 7fc2cd2b6700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2c80a66b0 con 0x7fc2c80a0820 2021-02-10T11:34:39.762 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:39.757 7fc2cd2b6700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2c81a49b0 con 0x7fc2c809bcc0 2021-02-10T11:34:39.762 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:39.757 7fc2cd2b6700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2c81a4b90 con 0x7fc2c80a1240 2021-02-10T11:34:39.762 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:39.758 7fc2c687c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc2c80a0820 0x7fc2c80a0c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:39.762 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:39.758 7fc2c687c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc2c80a0820 0x7fc2c80a0c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:36316/0 (socket says 172.21.2.102:36316) 2021-02-10T11:34:39.763 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:39.758 7fc2c687c700 1 -- 172.21.2.102:0/2154307011 learned_addr learned my addr 172.21.2.102:0/2154307011 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:34:39.763 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:39.758 7fc2c687c700 1 -- 172.21.2.102:0/2154307011 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc2c80a1240 msgr2=0x7fc2c80858d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:39.763 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:39.758 7fc2c687c700 1 --2- 172.21.2.102:0/2154307011 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc2c80a1240 0x7fc2c80858d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:39.763 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:39.758 7fc2c687c700 1 -- 172.21.2.102:0/2154307011 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc2c809bcc0 msgr2=0x7fc2c809c0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:39.763 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:39.758 7fc2c687c700 1 --2- 172.21.2.102:0/2154307011 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc2c809bcc0 0x7fc2c809c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:39.763 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:39.758 7fc2c687c700 1 -- 172.21.2.102:0/2154307011 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2c80a1640 con 0x7fc2c80a0820 2021-02-10T11:34:39.763 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:39.758 7fc2c687c700 1 --2- 172.21.2.102:0/2154307011 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc2c80a0820 0x7fc2c80a0c20 secure :-1 s=READY pgs=2576 cs=0 l=1 rev1=1 rx=0x7fc2b8009630 tx=0x7fc2b800b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:39.763 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:39.758 7fc2bffff700 1 -- 172.21.2.102:0/2154307011 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc2b80093a0 con 0x7fc2c80a0820 2021-02-10T11:34:39.763 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:39.758 7fc2bffff700 1 -- 172.21.2.102:0/2154307011 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2b800bd70 con 0x7fc2c80a0820 2021-02-10T11:34:39.763 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:39.758 7fc2bffff700 1 -- 172.21.2.102:0/2154307011 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc2b80054d0 con 0x7fc2c80a0820 2021-02-10T11:34:39.763 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:39.758 7fc2cd2b6700 1 -- 172.21.2.102:0/2154307011 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2c80a1640 con 0x7fc2c80a0820 2021-02-10T11:34:39.764 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:39.759 7fc2cd2b6700 1 -- 172.21.2.102:0/2154307011 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2c80a1640 con 0x7fc2c80a0820 2021-02-10T11:34:39.764 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:39.759 7fc2bffff700 1 -- 172.21.2.102:0/2154307011 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fc2b800a000 con 0x7fc2c80a0820 2021-02-10T11:34:39.764 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:39.759 7fc2bffff700 1 --2- 172.21.2.102:0/2154307011 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc2ac0186c0 0x7fc2ac01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:39.764 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:39.759 7fc2bffff700 1 -- 172.21.2.102:0/2154307011 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(137..137 src has 1..137) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fc2b8032560 con 0x7fc2c80a0820 2021-02-10T11:34:39.764 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:39.759 7fc2c707d700 1 --2- 172.21.2.102:0/2154307011 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc2ac0186c0 0x7fc2ac01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:39.764 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:39.760 7fc2c707d700 1 --2- 172.21.2.102:0/2154307011 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc2ac0186c0 0x7fc2ac01abc0 crc :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:39.791 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:39.790 7fc2cd2b6700 1 -- 172.21.2.102:0/2154307011 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2c80fe1c0 con 0x7fc2c80a0820 2021-02-10T11:34:39.793 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:39.793 7fc2bffff700 1 -- 172.21.2.102:0/2154307011 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fc2b8030d90 con 0x7fc2c80a0820 2021-02-10T11:34:39.922 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:39.921 7fc2cd2b6700 1 -- 172.21.2.102:0/2154307011 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7fc2c8086d30 con 0x7fc2c80a0820 2021-02-10T11:34:39.923 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:39.922 7fc2bffff700 1 -- 172.21.2.102:0/2154307011 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v137) v1 ==== 104+0+10 (secure 0 0 0) 0x7fc2b800a000 con 0x7fc2c80a0820 2021-02-10T11:34:39.923 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:34:39.924 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:39.923 7fc2cd2b6700 1 -- 172.21.2.102:0/2154307011 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc2ac0186c0 msgr2=0x7fc2ac01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:39.924 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:39.923 7fc2cd2b6700 1 --2- 172.21.2.102:0/2154307011 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc2ac0186c0 0x7fc2ac01abc0 crc :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:39.924 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:39.923 7fc2cd2b6700 1 -- 172.21.2.102:0/2154307011 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc2c80a0820 msgr2=0x7fc2c80a0c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:39.924 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:39.923 7fc2cd2b6700 1 --2- 172.21.2.102:0/2154307011 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc2c80a0820 0x7fc2c80a0c20 secure :-1 s=READY pgs=2576 cs=0 l=1 rev1=1 rx=0x7fc2b8009630 tx=0x7fc2b800b980).stop 2021-02-10T11:34:39.925 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:39.924 7fc2cd2b6700 1 -- 172.21.2.102:0/2154307011 shutdown_connections 2021-02-10T11:34:39.925 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:39.924 7fc2cd2b6700 1 --2- 172.21.2.102:0/2154307011 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc2c80a1240 0x7fc2c80858d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:39.925 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:39.924 7fc2cd2b6700 1 --2- 172.21.2.102:0/2154307011 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc2c80a0820 0x7fc2c80a0c20 unknown :-1 s=CLOSED pgs=2576 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:39.925 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:39.924 7fc2cd2b6700 1 --2- 172.21.2.102:0/2154307011 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc2ac0186c0 0x7fc2ac01abc0 unknown :-1 s=CLOSED pgs=1140 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:39.925 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:39.924 7fc2cd2b6700 1 --2- 172.21.2.102:0/2154307011 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc2c809bcc0 0x7fc2c809c0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:39.925 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:39.924 7fc2cd2b6700 1 -- 172.21.2.102:0/2154307011 >> 172.21.2.102:0/2154307011 conn(0x7fc2c80bef00 msgr2=0x7fc2c809d1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:39.925 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:39.924 7fc2cd2b6700 1 -- 172.21.2.102:0/2154307011 shutdown_connections 2021-02-10T11:34:39.926 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:39.924 7fc2cd2b6700 1 -- 172.21.2.102:0/2154307011 wait complete. 2021-02-10T11:34:39.933 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-02-10T11:34:40.366 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:40.347 7fea7dbd8700 1 Processor -- start 2021-02-10T11:34:40.367 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:40.347 7fea7dbd8700 1 -- start start 2021-02-10T11:34:40.367 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:40.347 7fea7dbd8700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fea780d1870 0x7fea781b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:40.367 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:40.347 7fea7dbd8700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fea781b0ae0 0x7fea78108940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:40.367 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:40.347 7fea7dbd8700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea78109a60 con 0x7fea780d1c70 2021-02-10T11:34:40.367 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:40.347 7fea7dbd8700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea780e5ef0 con 0x7fea781b0ae0 2021-02-10T11:34:40.367 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:40.347 7fea7dbd8700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea78109750 con 0x7fea780d1870 2021-02-10T11:34:40.367 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:40.349 7fea77fff700 1 --1- >> v1:172.21.2.113:6789/0 conn(0x7fea780d1c70 0x7fea78108f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.113:6789/0 says I am v1:172.21.2.102:35876/0 (socket says 172.21.2.102:35876) 2021-02-10T11:34:40.367 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:40.349 7fea77fff700 1 -- 172.21.2.102:0/4119345417 learned_addr learned my addr 172.21.2.102:0/4119345417 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:34:40.367 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:40.349 7fea76ffd700 1 --2- 172.21.2.102:0/4119345417 >> v2:172.21.2.113:3300/0 conn(0x7fea781b0ae0 0x7fea78108940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:40.368 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:40.349 7fea777fe700 1 --2- 172.21.2.102:0/4119345417 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fea780d1870 0x7fea781b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:40.368 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:40.349 7fea76ffd700 1 -- 172.21.2.102:0/4119345417 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fea780d1870 msgr2=0x7fea781b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:34:40.368 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:40.349 7fea76ffd700 1 --2- 172.21.2.102:0/4119345417 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fea780d1870 0x7fea781b04c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:40.368 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:40.350 7fea76ffd700 1 -- 172.21.2.102:0/4119345417 >> v1:172.21.2.113:6789/0 conn(0x7fea780d1c70 legacy=0x7fea78108f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:40.368 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:40.350 7fea76ffd700 1 -- 172.21.2.102:0/4119345417 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea78088070 con 0x7fea781b0ae0 2021-02-10T11:34:40.368 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:40.350 7fea777fe700 1 --2- 172.21.2.102:0/4119345417 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fea780d1870 0x7fea781b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:34:40.368 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:40.350 7fea76ffd700 1 --2- 172.21.2.102:0/4119345417 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fea781b0ae0 0x7fea78108940 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 rx=0x7fea60006650 tx=0x7fea60009a80).ready entity=mon.1 client_cookie=cdee4c04f7cbec2b server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:40.368 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:40.350 7fea767fc700 1 -- 172.21.2.102:0/4119345417 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fea6000c0c0 con 0x7fea781b0ae0 2021-02-10T11:34:40.368 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:40.350 7fea767fc700 1 -- 172.21.2.102:0/4119345417 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea6000c340 con 0x7fea781b0ae0 2021-02-10T11:34:40.369 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:40.350 7fea767fc700 1 -- 172.21.2.102:0/4119345417 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fea6000c860 con 0x7fea781b0ae0 2021-02-10T11:34:40.369 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:40.350 7fea7dbd8700 1 -- 172.21.2.102:0/4119345417 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fea781b0ae0 msgr2=0x7fea78108940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:40.369 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:40.350 7fea7dbd8700 1 --2- 172.21.2.102:0/4119345417 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fea781b0ae0 0x7fea78108940 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 rx=0x7fea60006650 tx=0x7fea60009a80).stop 2021-02-10T11:34:40.369 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:40.351 7fea7dbd8700 1 -- 172.21.2.102:0/4119345417 shutdown_connections 2021-02-10T11:34:40.369 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:40.351 7fea7dbd8700 1 --2- 172.21.2.102:0/4119345417 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fea781b0ae0 0x7fea78108940 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:40.369 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:40.351 7fea7dbd8700 1 --2- 172.21.2.102:0/4119345417 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fea780d1870 0x7fea781b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:40.369 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:40.351 7fea7dbd8700 1 -- 172.21.2.102:0/4119345417 >> 172.21.2.102:0/4119345417 conn(0x7fea780ce420 msgr2=0x7fea780d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:40.369 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:40.351 7fea7dbd8700 1 -- 172.21.2.102:0/4119345417 shutdown_connections 2021-02-10T11:34:40.369 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:40.351 7fea7dbd8700 1 -- 172.21.2.102:0/4119345417 wait complete. 2021-02-10T11:34:40.369 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:40.366 7fea7dbd8700 1 Processor -- start 2021-02-10T11:34:40.371 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:40.366 7fea7dbd8700 1 -- start start 2021-02-10T11:34:40.371 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:40.366 7fea7dbd8700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fea780a58c0 0x7fea780a4d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:40.371 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:40.366 7fea7dbd8700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fea780b1ba0 0x7fea780b1fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:40.372 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:40.366 7fea7dbd8700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fea780b25c0 0x7fea780b3270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:40.372 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:40.366 7fea7dbd8700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea780b2a40 con 0x7fea780b25c0 2021-02-10T11:34:40.372 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:40.366 7fea7dbd8700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea780a5340 con 0x7fea780a58c0 2021-02-10T11:34:40.372 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:40.366 7fea7dbd8700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea780a5560 con 0x7fea780b1ba0 2021-02-10T11:34:40.372 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:40.367 7fea77fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fea780b25c0 0x7fea780b3270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:40.372 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:40.367 7fea77fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fea780b25c0 0x7fea780b3270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:36334/0 (socket says 172.21.2.102:36334) 2021-02-10T11:34:40.372 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:40.367 7fea77fff700 1 -- 172.21.2.102:0/3673998074 learned_addr learned my addr 172.21.2.102:0/3673998074 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:34:40.372 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:40.367 7fea777fe700 1 --2- 172.21.2.102:0/3673998074 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fea780a58c0 0x7fea780a4d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:40.372 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:40.367 7fea76ffd700 1 --2- 172.21.2.102:0/3673998074 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fea780b1ba0 0x7fea780b1fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:40.372 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:40.367 7fea77fff700 1 -- 172.21.2.102:0/3673998074 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fea780b1ba0 msgr2=0x7fea780b1fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:40.372 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:40.367 7fea77fff700 1 --2- 172.21.2.102:0/3673998074 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fea780b1ba0 0x7fea780b1fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:40.373 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:40.367 7fea77fff700 1 -- 172.21.2.102:0/3673998074 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fea780a58c0 msgr2=0x7fea780a4d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:40.373 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:40.367 7fea77fff700 1 --2- 172.21.2.102:0/3673998074 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fea780a58c0 0x7fea780a4d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:40.373 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:40.367 7fea77fff700 1 -- 172.21.2.102:0/3673998074 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea780b3890 con 0x7fea780b25c0 2021-02-10T11:34:40.373 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:40.367 7fea77fff700 1 --2- 172.21.2.102:0/3673998074 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fea780b25c0 0x7fea780b3270 secure :-1 s=READY pgs=2577 cs=0 l=1 rev1=1 rx=0x7fea7000a5d0 tx=0x7fea7000a970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:40.373 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:40.367 7fea6ffff700 1 -- 172.21.2.102:0/3673998074 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fea7000ad60 con 0x7fea780b25c0 2021-02-10T11:34:40.373 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:40.367 7fea6ffff700 1 -- 172.21.2.102:0/3673998074 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea70004320 con 0x7fea780b25c0 2021-02-10T11:34:40.373 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:40.368 7fea6ffff700 1 -- 172.21.2.102:0/3673998074 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fea70004770 con 0x7fea780b25c0 2021-02-10T11:34:40.373 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:40.368 7fea7dbd8700 1 -- 172.21.2.102:0/3673998074 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea780b3890 con 0x7fea780b25c0 2021-02-10T11:34:40.373 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:40.368 7fea7dbd8700 1 -- 172.21.2.102:0/3673998074 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea780b3de0 con 0x7fea780b25c0 2021-02-10T11:34:40.373 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:40.368 7fea6ffff700 1 -- 172.21.2.102:0/3673998074 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fea70007070 con 0x7fea780b25c0 2021-02-10T11:34:40.373 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:40.368 7fea6ffff700 1 --2- 172.21.2.102:0/3673998074 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fea5c0186c0 0x7fea5c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:40.374 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:40.368 7fea6ffff700 1 -- 172.21.2.102:0/3673998074 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(137..137 src has 1..137) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fea700325a0 con 0x7fea780b25c0 2021-02-10T11:34:40.374 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:40.368 7fea777fe700 1 --2- 172.21.2.102:0/3673998074 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fea5c0186c0 0x7fea5c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:40.374 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:40.369 7fea777fe700 1 --2- 172.21.2.102:0/3673998074 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fea5c0186c0 0x7fea5c01abc0 crc :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:40.399 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:40.398 7fea7dbd8700 1 -- 172.21.2.102:0/3673998074 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea78082980 con 0x7fea780b25c0 2021-02-10T11:34:40.401 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:40.400 7fea6ffff700 1 -- 172.21.2.102:0/3673998074 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fea70000eb0 con 0x7fea780b25c0 2021-02-10T11:34:40.528 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:40.527 7fea7dbd8700 1 -- 172.21.2.102:0/3673998074 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7fea78109a60 con 0x7fea780b25c0 2021-02-10T11:34:40.529 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:40.528 7fea6ffff700 1 -- 172.21.2.102:0/3673998074 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v137) v1 ==== 100+0+10 (secure 0 0 0) 0x7fea70007030 con 0x7fea780b25c0 2021-02-10T11:34:40.529 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:34:40.530 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:40.529 7fea7dbd8700 1 -- 172.21.2.102:0/3673998074 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fea5c0186c0 msgr2=0x7fea5c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:40.531 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:40.529 7fea7dbd8700 1 --2- 172.21.2.102:0/3673998074 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fea5c0186c0 0x7fea5c01abc0 crc :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:40.531 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:40.530 7fea7dbd8700 1 -- 172.21.2.102:0/3673998074 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fea780b25c0 msgr2=0x7fea780b3270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:40.531 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:40.530 7fea7dbd8700 1 --2- 172.21.2.102:0/3673998074 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fea780b25c0 0x7fea780b3270 secure :-1 s=READY pgs=2577 cs=0 l=1 rev1=1 rx=0x7fea7000a5d0 tx=0x7fea7000a970).stop 2021-02-10T11:34:40.531 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:40.530 7fea7dbd8700 1 -- 172.21.2.102:0/3673998074 shutdown_connections 2021-02-10T11:34:40.531 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:40.530 7fea7dbd8700 1 --2- 172.21.2.102:0/3673998074 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fea780b25c0 0x7fea780b3270 unknown :-1 s=CLOSED pgs=2577 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:40.531 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:40.530 7fea7dbd8700 1 --2- 172.21.2.102:0/3673998074 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fea780b1ba0 0x7fea780b1fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:40.531 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:40.530 7fea7dbd8700 1 --2- 172.21.2.102:0/3673998074 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fea5c0186c0 0x7fea5c01abc0 unknown :-1 s=CLOSED pgs=1141 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:40.531 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:40.530 7fea7dbd8700 1 --2- 172.21.2.102:0/3673998074 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fea780a58c0 0x7fea780a4d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:40.531 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:40.530 7fea7dbd8700 1 -- 172.21.2.102:0/3673998074 >> 172.21.2.102:0/3673998074 conn(0x7fea780b5dc0 msgr2=0x7fea78095960 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:40.531 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:40.530 7fea7dbd8700 1 -- 172.21.2.102:0/3673998074 shutdown_connections 2021-02-10T11:34:40.532 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:40.530 7fea7dbd8700 1 -- 172.21.2.102:0/3673998074 wait complete. 2021-02-10T11:34:40.539 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-02-10T11:34:40.539 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:34:40.967 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:40.950 7fba88975700 1 Processor -- start 2021-02-10T11:34:40.967 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:40.950 7fba88975700 1 -- start start 2021-02-10T11:34:40.967 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:40.950 7fba88975700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fba840d0340 0x7fba840ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:40.967 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:40.950 7fba88975700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fba840d1c70 0x7fba840efc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:40.967 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:40.950 7fba88975700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba840f0b10 con 0x7fba840d1870 2021-02-10T11:34:40.968 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:40.950 7fba88975700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba840e5ed0 con 0x7fba840d0340 2021-02-10T11:34:40.968 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:40.950 7fba88975700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba841b0be0 con 0x7fba840d1c70 2021-02-10T11:34:40.968 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:40.951 7fba81e1b700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fba840d0340 0x7fba840ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:40.968 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:40.951 7fba81e1b700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fba840d0340 0x7fba840ef640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:36338/0 (socket says 172.21.2.102:36338) 2021-02-10T11:34:40.968 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:40.951 7fba81e1b700 1 -- 172.21.2.102:0/2071458369 learned_addr learned my addr 172.21.2.102:0/2071458369 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:34:40.968 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:40.951 7fba82e1d700 1 --2- 172.21.2.102:0/2071458369 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fba840d1c70 0x7fba840efc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:40.968 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:40.951 7fba81e1b700 1 -- 172.21.2.102:0/2071458369 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fba840d1c70 msgr2=0x7fba840efc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:34:40.968 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:40.951 7fba81e1b700 1 --2- 172.21.2.102:0/2071458369 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fba840d1c70 0x7fba840efc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:40.968 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:40.951 7fba81e1b700 1 -- 172.21.2.102:0/2071458369 >> v1:172.21.2.113:6789/0 conn(0x7fba840d1870 legacy=0x7fba841b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:40.968 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:40.951 7fba81e1b700 1 -- 172.21.2.102:0/2071458369 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba840f20a0 con 0x7fba840d0340 2021-02-10T11:34:40.968 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:40.951 7fba82e1d700 1 --2- 172.21.2.102:0/2071458369 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fba840d1c70 0x7fba840efc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:34:40.968 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:40.951 7fba81e1b700 1 --2- 172.21.2.102:0/2071458369 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fba840d0340 0x7fba840ef640 secure :-1 s=READY pgs=2578 cs=0 l=1 rev1=1 rx=0x7fba78006700 tx=0x7fba78009b80).ready entity=mon.0 client_cookie=fcb650f7d826250b server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:40.969 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:40.952 7fba8161a700 1 -- 172.21.2.102:0/2071458369 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fba7800c0c0 con 0x7fba840d0340 2021-02-10T11:34:40.969 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:40.952 7fba8161a700 1 -- 172.21.2.102:0/2071458369 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba7800c400 con 0x7fba840d0340 2021-02-10T11:34:40.969 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:40.952 7fba8161a700 1 -- 172.21.2.102:0/2071458369 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fba7800c860 con 0x7fba840d0340 2021-02-10T11:34:40.969 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:40.952 7fba88975700 1 -- 172.21.2.102:0/2071458369 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fba840d0340 msgr2=0x7fba840ef640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:40.969 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:40.952 7fba88975700 1 --2- 172.21.2.102:0/2071458369 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fba840d0340 0x7fba840ef640 secure :-1 s=READY pgs=2578 cs=0 l=1 rev1=1 rx=0x7fba78006700 tx=0x7fba78009b80).stop 2021-02-10T11:34:40.969 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:40.952 7fba88975700 1 -- 172.21.2.102:0/2071458369 shutdown_connections 2021-02-10T11:34:40.969 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:40.952 7fba88975700 1 --2- 172.21.2.102:0/2071458369 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fba840d1c70 0x7fba840efc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:40.969 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:40.952 7fba88975700 1 --2- 172.21.2.102:0/2071458369 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fba840d0340 0x7fba840ef640 unknown :-1 s=CLOSED pgs=2578 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:40.969 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:40.952 7fba88975700 1 -- 172.21.2.102:0/2071458369 >> 172.21.2.102:0/2071458369 conn(0x7fba840ce450 msgr2=0x7fba840d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:40.969 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:40.952 7fba88975700 1 -- 172.21.2.102:0/2071458369 shutdown_connections 2021-02-10T11:34:40.969 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:40.952 7fba88975700 1 -- 172.21.2.102:0/2071458369 wait complete. 2021-02-10T11:34:40.971 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:40.967 7fba88975700 1 Processor -- start 2021-02-10T11:34:40.971 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:40.967 7fba88975700 1 -- start start 2021-02-10T11:34:40.971 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:40.967 7fba88975700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fba84199110 0x7fba84199510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:40.971 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:40.967 7fba88975700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fba841b1c50 0x7fba8419a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:40.971 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:40.967 7fba88975700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fba8419aae0 0x7fba8419aee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:40.971 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:40.967 7fba88975700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba84199f40 con 0x7fba8419aae0 2021-02-10T11:34:40.971 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:40.968 7fba88975700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba8418f160 con 0x7fba841b1c50 2021-02-10T11:34:40.972 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:40.968 7fba88975700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba8418ecc0 con 0x7fba84199110 2021-02-10T11:34:40.972 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:40.968 7fba82e1d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fba8419aae0 0x7fba8419aee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:40.972 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:40.968 7fba82e1d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fba8419aae0 0x7fba8419aee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:36348/0 (socket says 172.21.2.102:36348) 2021-02-10T11:34:40.972 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:40.968 7fba82e1d700 1 -- 172.21.2.102:0/21871023 learned_addr learned my addr 172.21.2.102:0/21871023 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:34:40.972 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:40.968 7fba82e1d700 1 -- 172.21.2.102:0/21871023 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fba84199110 msgr2=0x7fba84199510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:40.972 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:40.968 7fba82e1d700 1 --2- 172.21.2.102:0/21871023 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fba84199110 0x7fba84199510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:40.972 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:40.968 7fba82e1d700 1 -- 172.21.2.102:0/21871023 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fba841b1c50 msgr2=0x7fba8419a4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:40.972 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:40.968 7fba82e1d700 1 --2- 172.21.2.102:0/21871023 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fba841b1c50 0x7fba8419a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:40.972 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:40.968 7fba82e1d700 1 -- 172.21.2.102:0/21871023 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba84199b30 con 0x7fba8419aae0 2021-02-10T11:34:40.972 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:40.968 7fba82e1d700 1 --2- 172.21.2.102:0/21871023 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fba8419aae0 0x7fba8419aee0 secure :-1 s=READY pgs=2579 cs=0 l=1 rev1=1 rx=0x7fba7c009980 tx=0x7fba7c00bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:40.972 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:40.969 7fba72ffd700 1 -- 172.21.2.102:0/21871023 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fba7c00c2f0 con 0x7fba8419aae0 2021-02-10T11:34:40.972 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:40.969 7fba72ffd700 1 -- 172.21.2.102:0/21871023 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba7c006790 con 0x7fba8419aae0 2021-02-10T11:34:40.972 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:40.969 7fba72ffd700 1 -- 172.21.2.102:0/21871023 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fba7c00c7f0 con 0x7fba8419aae0 2021-02-10T11:34:40.972 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:40.969 7fba88975700 1 -- 172.21.2.102:0/21871023 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba84199b30 con 0x7fba8419aae0 2021-02-10T11:34:40.973 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:40.969 7fba88975700 1 -- 172.21.2.102:0/21871023 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba84199b30 con 0x7fba8419aae0 2021-02-10T11:34:40.973 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:40.969 7fba72ffd700 1 -- 172.21.2.102:0/21871023 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fba7c0011f0 con 0x7fba8419aae0 2021-02-10T11:34:40.973 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:40.970 7fba72ffd700 1 --2- 172.21.2.102:0/21871023 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fba680186c0 0x7fba6801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:40.973 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:40.970 7fba72ffd700 1 -- 172.21.2.102:0/21871023 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(137..137 src has 1..137) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fba7c032910 con 0x7fba8419aae0 2021-02-10T11:34:40.973 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:40.970 7fba8261c700 1 --2- 172.21.2.102:0/21871023 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fba680186c0 0x7fba6801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:40.973 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:40.970 7fba8261c700 1 --2- 172.21.2.102:0/21871023 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fba680186c0 0x7fba6801abc0 crc :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:41.001 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:41.000 7fba88975700 1 -- 172.21.2.102:0/21871023 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba840ebc00 con 0x7fba8419aae0 2021-02-10T11:34:41.003 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:41.003 7fba72ffd700 1 -- 172.21.2.102:0/21871023 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fba7c00be10 con 0x7fba8419aae0 2021-02-10T11:34:41.131 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:41.130 7fba88975700 1 -- 172.21.2.102:0/21871023 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fba840f0b10 con 0x7fba8419aae0 2021-02-10T11:34:41.131 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:41.130 7fba72ffd700 1 -- 172.21.2.102:0/21871023 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 116 v116) v1 ==== 95+0+2108 (secure 0 0 0) 0x7fba7c0011f0 con 0x7fba8419aae0 2021-02-10T11:34:41.132 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:34:41.132 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":116,"default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":11466,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.102:6835/1274216855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6834","nonce":1274216855},{"type":"v1","addr":"172.21.2.102:6835","nonce":1274216855}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":112}],"filesystems":[{"mdsmap":{"epoch":116,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:34:23.684779","modified":"2021-02-10 11:34:25.693957","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6770},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6770":{"gid":6770,"name":"b","rank":0,"incarnation":113,"state":"up:active","state_seq":3,"addr":"172.21.2.113:6833/3752249621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6832","nonce":3752249621},{"type":"v1","addr":"172.21.2.113:6833","nonce":3752249621}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[29],"metadata_pool":28,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":14}]} 2021-02-10T11:34:41.132 INFO:teuthology.orchestra.run.gibba002.stderr:dumped fsmap epoch 116 2021-02-10T11:34:41.133 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:41.132 7fba88975700 1 -- 172.21.2.102:0/21871023 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fba680186c0 msgr2=0x7fba6801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:41.133 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:41.132 7fba88975700 1 --2- 172.21.2.102:0/21871023 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fba680186c0 0x7fba6801abc0 crc :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:41.133 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:41.132 7fba88975700 1 -- 172.21.2.102:0/21871023 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fba8419aae0 msgr2=0x7fba8419aee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:41.133 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:41.132 7fba88975700 1 --2- 172.21.2.102:0/21871023 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fba8419aae0 0x7fba8419aee0 secure :-1 s=READY pgs=2579 cs=0 l=1 rev1=1 rx=0x7fba7c009980 tx=0x7fba7c00bcd0).stop 2021-02-10T11:34:41.133 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:41.132 7fba88975700 1 -- 172.21.2.102:0/21871023 shutdown_connections 2021-02-10T11:34:41.134 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:41.132 7fba88975700 1 --2- 172.21.2.102:0/21871023 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fba8419aae0 0x7fba8419aee0 unknown :-1 s=CLOSED pgs=2579 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:41.134 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:41.132 7fba88975700 1 --2- 172.21.2.102:0/21871023 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fba680186c0 0x7fba6801abc0 unknown :-1 s=CLOSED pgs=1142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:41.134 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:41.132 7fba88975700 1 --2- 172.21.2.102:0/21871023 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fba841b1c50 0x7fba8419a4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:41.134 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:41.132 7fba88975700 1 --2- 172.21.2.102:0/21871023 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fba84199110 0x7fba84199510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:41.134 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:41.132 7fba88975700 1 -- 172.21.2.102:0/21871023 >> 172.21.2.102:0/21871023 conn(0x7fba8419f880 msgr2=0x7fba84198620 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:41.134 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:41.133 7fba88975700 1 -- 172.21.2.102:0/21871023 shutdown_connections 2021-02-10T11:34:41.134 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:41.133 7fba88975700 1 -- 172.21.2.102:0/21871023 wait complete. 2021-02-10T11:34:41.343 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 116, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:34:23.684779', 'modified': '2021-02-10 11:34:25.693957', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6770}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6770': {'gid': 6770, 'name': 'b', 'rank': 0, 'incarnation': 113, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.113:6833/3752249621', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.113:6832', 'nonce': 3752249621}, {'type': 'v1', 'addr': '172.21.2.113:6833', 'nonce': 3752249621}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [29], 'metadata_pool': 28, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:34:41.343 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:34:41.343 DEBUG:teuthology.orchestra.run.gibba013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:34:41.741 INFO:teuthology.orchestra.run.gibba013.stdout:{ 2021-02-10T11:34:41.742 INFO:teuthology.orchestra.run.gibba013.stdout: "cluster_fsid": "68db30ea-dbce-4aef-a6df-f01c6fea0513", 2021-02-10T11:34:41.742 INFO:teuthology.orchestra.run.gibba013.stdout: "whoami": 0, 2021-02-10T11:34:41.742 INFO:teuthology.orchestra.run.gibba013.stdout: "id": 6770, 2021-02-10T11:34:41.742 INFO:teuthology.orchestra.run.gibba013.stdout: "want_state": "up:active", 2021-02-10T11:34:41.742 INFO:teuthology.orchestra.run.gibba013.stdout: "state": "up:active", 2021-02-10T11:34:41.742 INFO:teuthology.orchestra.run.gibba013.stdout: "rank_uptime": 18.053395764000001, 2021-02-10T11:34:41.742 INFO:teuthology.orchestra.run.gibba013.stdout: "mdsmap_epoch": 116, 2021-02-10T11:34:41.742 INFO:teuthology.orchestra.run.gibba013.stdout: "osdmap_epoch": 137, 2021-02-10T11:34:41.742 INFO:teuthology.orchestra.run.gibba013.stdout: "osdmap_epoch_barrier": 137, 2021-02-10T11:34:41.742 INFO:teuthology.orchestra.run.gibba013.stdout: "uptime": 23.705965824 2021-02-10T11:34:41.742 INFO:teuthology.orchestra.run.gibba013.stdout:} 2021-02-10T11:34:41.745 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "68db30ea-dbce-4aef-a6df-f01c6fea0513", "whoami": 0, "id": 6770, "want_state": "up:active", "state": "up:active", "rank_uptime": 18.053395764000001, "mdsmap_epoch": 116, "osdmap_epoch": 137, "osdmap_epoch_barrier": 137, "uptime": 23.705965824 } 2021-02-10T11:34:41.746 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2021-02-10T11:34:41.746 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2021-02-10T11:34:41.746 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.1 at ubuntu@gibba002.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.1... 2021-02-10T11:34:41.746 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:34:41.746 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec mkdir -p /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:34:41.750 DEBUG:teuthology.orchestra.run.gibba002:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:34:41.792 DEBUG:teuthology.orchestra.run.gibba013:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:34:42.083 DEBUG:teuthology.orchestra.run.gibba002:> sudo modprobe fuse 2021-02-10T11:34:42.427 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:34:42.427 DEBUG:teuthology.orchestra.run.gibba002:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:34:42.769 INFO:teuthology.orchestra.run.gibba002.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:34:42.770 INFO:teuthology.orchestra.run.gibba002.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:34:42.770 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:34:42.770 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:34:42.771 DEBUG:teuthology.orchestra.run.gibba002:> ls /sys/fs/fuse/connections 2021-02-10T11:34:43.106 INFO:teuthology.orchestra.run.gibba002.stdout:43 2021-02-10T11:34:43.106 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [43] 2021-02-10T11:34:43.106 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.1 /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:34:43.109 DEBUG:teuthology.orchestra.run.gibba002:> sudo modprobe fuse 2021-02-10T11:34:43.454 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:34:43.455 DEBUG:teuthology.orchestra.run.gibba002:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:34:43.520 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba002.stderr:ceph-fuse[194381]: starting ceph client 2021-02-10T11:34:43.521 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba002.stderr:2021-02-10 11:34:43.519 7f9149b71f80 -1 init, newargv = 0x55dc15592200 newargc=9 2021-02-10T11:34:43.526 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba002.stderr:ceph-fuse[194381]: starting fuse 2021-02-10T11:34:43.797 INFO:teuthology.orchestra.run.gibba002.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:34:43.797 INFO:teuthology.orchestra.run.gibba002.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:34:43.798 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:34:43.798 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:34:43.798 DEBUG:teuthology.orchestra.run.gibba002:> ls /sys/fs/fuse/connections 2021-02-10T11:34:44.133 INFO:teuthology.orchestra.run.gibba002.stdout:43 2021-02-10T11:34:44.133 INFO:teuthology.orchestra.run.gibba002.stdout:44 2021-02-10T11:34:44.134 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43, 44] 2021-02-10T11:34:44.134 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:34:44.134 DEBUG:teuthology.orchestra.run.gibba002:> sudo python3 -c ' 2021-02-10T11:34:44.134 DEBUG:teuthology.orchestra.run.gibba002:> import glob 2021-02-10T11:34:44.134 DEBUG:teuthology.orchestra.run.gibba002:> import re 2021-02-10T11:34:44.134 DEBUG:teuthology.orchestra.run.gibba002:> import os 2021-02-10T11:34:44.134 DEBUG:teuthology.orchestra.run.gibba002:> import subprocess 2021-02-10T11:34:44.135 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:34:44.135 DEBUG:teuthology.orchestra.run.gibba002:> def find_socket(client_name): 2021-02-10T11:34:44.135 DEBUG:teuthology.orchestra.run.gibba002:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2021-02-10T11:34:44.135 DEBUG:teuthology.orchestra.run.gibba002:> files = glob.glob(asok_path) 2021-02-10T11:34:44.135 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:34:44.135 DEBUG:teuthology.orchestra.run.gibba002:> # Given a non-glob path, it better be there 2021-02-10T11:34:44.135 DEBUG:teuthology.orchestra.run.gibba002:> if "*" not in asok_path: 2021-02-10T11:34:44.135 DEBUG:teuthology.orchestra.run.gibba002:> assert(len(files) == 1) 2021-02-10T11:34:44.135 DEBUG:teuthology.orchestra.run.gibba002:> return files[0] 2021-02-10T11:34:44.135 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:34:44.135 DEBUG:teuthology.orchestra.run.gibba002:> for f in files: 2021-02-10T11:34:44.136 DEBUG:teuthology.orchestra.run.gibba002:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2021-02-10T11:34:44.136 DEBUG:teuthology.orchestra.run.gibba002:> if os.path.exists("/proc/{0}".format(pid)): 2021-02-10T11:34:44.136 DEBUG:teuthology.orchestra.run.gibba002:> return f 2021-02-10T11:34:44.136 DEBUG:teuthology.orchestra.run.gibba002:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2021-02-10T11:34:44.136 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:34:44.136 DEBUG:teuthology.orchestra.run.gibba002:> print(find_socket("client.1")) 2021-02-10T11:34:44.136 DEBUG:teuthology.orchestra.run.gibba002:> ' 2021-02-10T11:34:44.525 INFO:teuthology.orchestra.run.gibba002.stdout:/var/run/ceph/ceph-client.1.194381.asok 2021-02-10T11:34:44.528 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.194381.asok 2021-02-10T11:34:44.528 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:34:44.528 DEBUG:teuthology.orchestra.run.gibba002:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.194381.asok status 2021-02-10T11:34:44.928 INFO:teuthology.orchestra.run.gibba002.stdout:{ 2021-02-10T11:34:44.928 INFO:teuthology.orchestra.run.gibba002.stdout: "metadata": { 2021-02-10T11:34:44.929 INFO:teuthology.orchestra.run.gibba002.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:34:44.929 INFO:teuthology.orchestra.run.gibba002.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:34:44.929 INFO:teuthology.orchestra.run.gibba002.stdout: "entity_id": "1", 2021-02-10T11:34:44.929 INFO:teuthology.orchestra.run.gibba002.stdout: "hostname": "gibba002", 2021-02-10T11:34:44.929 INFO:teuthology.orchestra.run.gibba002.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2021-02-10T11:34:44.929 INFO:teuthology.orchestra.run.gibba002.stdout: "pid": "194381", 2021-02-10T11:34:44.929 INFO:teuthology.orchestra.run.gibba002.stdout: "root": "/" 2021-02-10T11:34:44.929 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:44.929 INFO:teuthology.orchestra.run.gibba002.stdout: "dentry_count": 0, 2021-02-10T11:34:44.929 INFO:teuthology.orchestra.run.gibba002.stdout: "dentry_pinned_count": 0, 2021-02-10T11:34:44.929 INFO:teuthology.orchestra.run.gibba002.stdout: "id": 11652, 2021-02-10T11:34:44.929 INFO:teuthology.orchestra.run.gibba002.stdout: "inst": { 2021-02-10T11:34:44.929 INFO:teuthology.orchestra.run.gibba002.stdout: "name": { 2021-02-10T11:34:44.930 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "client", 2021-02-10T11:34:44.930 INFO:teuthology.orchestra.run.gibba002.stdout: "num": 11652 2021-02-10T11:34:44.930 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:44.930 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": { 2021-02-10T11:34:44.930 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:34:44.930 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:0", 2021-02-10T11:34:44.930 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 110306745 2021-02-10T11:34:44.930 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:34:44.930 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:44.930 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": { 2021-02-10T11:34:44.930 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:34:44.930 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:0", 2021-02-10T11:34:44.931 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 110306745 2021-02-10T11:34:44.931 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:44.931 INFO:teuthology.orchestra.run.gibba002.stdout: "inst_str": "client.11652 172.21.2.102:0/110306745", 2021-02-10T11:34:44.931 INFO:teuthology.orchestra.run.gibba002.stdout: "addr_str": "172.21.2.102:0/110306745", 2021-02-10T11:34:44.931 INFO:teuthology.orchestra.run.gibba002.stdout: "inode_count": 1, 2021-02-10T11:34:44.931 INFO:teuthology.orchestra.run.gibba002.stdout: "mds_epoch": 116, 2021-02-10T11:34:44.931 INFO:teuthology.orchestra.run.gibba002.stdout: "osd_epoch": 137, 2021-02-10T11:34:44.931 INFO:teuthology.orchestra.run.gibba002.stdout: "osd_epoch_barrier": 0, 2021-02-10T11:34:44.931 INFO:teuthology.orchestra.run.gibba002.stdout: "blacklisted": false 2021-02-10T11:34:44.931 INFO:teuthology.orchestra.run.gibba002.stdout:} 2021-02-10T11:34:44.933 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:34:44.933 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:34:44.933 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:34:45.269 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:34:45.269 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2021-02-10T11:34:45.270 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:34:45.270 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:34:45.617 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2021-02-10T11:34:46.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:46.034 7f048cadf700 1 Processor -- start 2021-02-10T11:34:46.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:46.034 7f048cadf700 1 -- start start 2021-02-10T11:34:46.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:46.034 7f048cadf700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f04880d1880 0x7f04881b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:46.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:46.034 7f048cadf700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04880fe1d0 con 0x7f04881b0af0 2021-02-10T11:34:46.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:46.034 7f048cadf700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0488109a70 con 0x7f04880cff80 2021-02-10T11:34:46.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:46.034 7f048cadf700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0488088070 con 0x7f04880d1880 2021-02-10T11:34:46.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:46.034 7f048605d700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f04881b0af0 0x7f0488108950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:56822/0 (socket says 172.21.2.102:56822) 2021-02-10T11:34:46.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:46.034 7f048605d700 1 -- 172.21.2.102:0/4168576972 learned_addr learned my addr 172.21.2.102:0/4168576972 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:34:46.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:46.034 7f048685e700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f04880d1880 0x7f04881b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:46.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:46.035 7f048685e700 1 -- 172.21.2.102:0/4168576972 >> v1:172.21.2.113:6789/0 conn(0x7f04880cff80 legacy=0x7f0488109140 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:34:46.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:46.035 7f048685e700 1 -- 172.21.2.102:0/4168576972 >> v1:172.21.2.102:6789/0 conn(0x7f04881b0af0 legacy=0x7f0488108950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:46.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:46.035 7f048685e700 1 -- 172.21.2.102:0/4168576972 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0488088260 con 0x7f04880d1880 2021-02-10T11:34:46.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:46.035 7f048685e700 1 --2- 172.21.2.102:0/4168576972 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f04880d1880 0x7f04881b04d0 secure :-1 s=READY pgs=2584 cs=0 l=1 rev1=1 rx=0x7f0470006650 tx=0x7f0470009a80).ready entity=mon.0 client_cookie=1147884d996d6247 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:46.052 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:46.035 7f048585c700 1 -- 172.21.2.102:0/4168576972 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f047000c0c0 con 0x7f04880d1880 2021-02-10T11:34:46.052 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:46.035 7f048585c700 1 -- 172.21.2.102:0/4168576972 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f047000c340 con 0x7f04880d1880 2021-02-10T11:34:46.052 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:46.035 7f048585c700 1 -- 172.21.2.102:0/4168576972 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f047000c860 con 0x7f04880d1880 2021-02-10T11:34:46.052 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:46.035 7f048cadf700 1 -- 172.21.2.102:0/4168576972 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f04880d1880 msgr2=0x7f04881b04d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:46.052 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:46.035 7f048cadf700 1 --2- 172.21.2.102:0/4168576972 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f04880d1880 0x7f04881b04d0 secure :-1 s=READY pgs=2584 cs=0 l=1 rev1=1 rx=0x7f0470006650 tx=0x7f0470009a80).stop 2021-02-10T11:34:46.052 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:46.036 7f048cadf700 1 -- 172.21.2.102:0/4168576972 shutdown_connections 2021-02-10T11:34:46.052 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:46.036 7f048cadf700 1 --2- 172.21.2.102:0/4168576972 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f04880d1880 0x7f04881b04d0 unknown :-1 s=CLOSED pgs=2584 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:46.052 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:46.036 7f048cadf700 1 -- 172.21.2.102:0/4168576972 >> 172.21.2.102:0/4168576972 conn(0x7f04880ce430 msgr2=0x7f04880d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:46.052 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:46.036 7f048cadf700 1 -- 172.21.2.102:0/4168576972 shutdown_connections 2021-02-10T11:34:46.052 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:46.036 7f048cadf700 1 -- 172.21.2.102:0/4168576972 wait complete. 2021-02-10T11:34:46.054 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:46.050 7f048cadf700 1 Processor -- start 2021-02-10T11:34:46.054 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:46.050 7f048cadf700 1 -- start start 2021-02-10T11:34:46.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:46.050 7f048cadf700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f04881c69b0 0x7f048809e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:46.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:46.050 7f048cadf700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f04881b5e60 0x7f048809ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:46.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:46.050 7f048cadf700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f048809f0f0 0x7f048809f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:46.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:46.051 7f048cadf700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f048809c410 con 0x7f04881c69b0 2021-02-10T11:34:46.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:46.051 7f048cadf700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f048809c630 con 0x7f04881b5e60 2021-02-10T11:34:46.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:46.051 7f048cadf700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f048809c850 con 0x7f048809f0f0 2021-02-10T11:34:46.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:46.051 7f048685e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f04881c69b0 0x7f048809e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:46.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:46.051 7f048685e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f04881c69b0 0x7f048809e4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:36370/0 (socket says 172.21.2.102:36370) 2021-02-10T11:34:46.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:46.051 7f048685e700 1 -- 172.21.2.102:0/3674768569 learned_addr learned my addr 172.21.2.102:0/3674768569 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:34:46.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:46.051 7f048685e700 1 -- 172.21.2.102:0/3674768569 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f048809f0f0 msgr2=0x7f048809f4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:34:46.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:46.051 7f048685e700 1 --2- 172.21.2.102:0/3674768569 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f048809f0f0 0x7f048809f4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:46.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:46.051 7f048685e700 1 -- 172.21.2.102:0/3674768569 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f04881b5e60 msgr2=0x7f048809ead0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:34:46.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:46.051 7f048685e700 1 --2- 172.21.2.102:0/3674768569 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f04881b5e60 0x7f048809ead0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:46.056 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:46.051 7f048685e700 1 -- 172.21.2.102:0/3674768569 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f048809ca70 con 0x7f04881c69b0 2021-02-10T11:34:46.056 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:46.051 7f048685e700 1 --2- 172.21.2.102:0/3674768569 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f04881c69b0 0x7f048809e4b0 secure :-1 s=READY pgs=2585 cs=0 l=1 rev1=1 rx=0x7f047000b5e0 tx=0x7f047000b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:46.056 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:46.051 7f047f7fe700 1 -- 172.21.2.102:0/3674768569 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0470000940 con 0x7f04881c69b0 2021-02-10T11:34:46.056 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:46.051 7f047f7fe700 1 -- 172.21.2.102:0/3674768569 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f047000bd70 con 0x7f04881c69b0 2021-02-10T11:34:46.056 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:46.051 7f048cadf700 1 -- 172.21.2.102:0/3674768569 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f048809ca70 con 0x7f04881c69b0 2021-02-10T11:34:46.056 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:46.052 7f048cadf700 1 -- 172.21.2.102:0/3674768569 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f048809d000 con 0x7f04881c69b0 2021-02-10T11:34:46.056 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:46.052 7f047f7fe700 1 -- 172.21.2.102:0/3674768569 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0470002270 con 0x7f04881c69b0 2021-02-10T11:34:46.056 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:46.052 7f047f7fe700 1 -- 172.21.2.102:0/3674768569 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f0470000c90 con 0x7f04881c69b0 2021-02-10T11:34:46.056 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:46.052 7f047f7fe700 1 --2- 172.21.2.102:0/3674768569 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f046c018730 0x7f046c01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:46.056 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:46.052 7f047f7fe700 1 -- 172.21.2.102:0/3674768569 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(137..137 src has 1..137) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f0470032830 con 0x7f04881c69b0 2021-02-10T11:34:46.056 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:46.052 7f048605d700 1 --2- 172.21.2.102:0/3674768569 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f046c018730 0x7f046c01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:46.056 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:46.053 7f048605d700 1 --2- 172.21.2.102:0/3674768569 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f046c018730 0x7f046c01ac10 crc :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:46.085 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:46.084 7f048cadf700 1 -- 172.21.2.102:0/3674768569 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04880802b0 con 0x7f04881c69b0 2021-02-10T11:34:46.087 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:46.086 7f047f7fe700 1 -- 172.21.2.102:0/3674768569 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f0470002510 con 0x7f04881c69b0 2021-02-10T11:34:46.224 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:46.223 7f048cadf700 1 -- 172.21.2.102:0/3674768569 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f04880ce430 con 0x7f04881c69b0 2021-02-10T11:34:47.143 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:47.141 7f047f7fe700 1 -- 172.21.2.102:0/3674768569 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v117) v1 ==== 136+0+0 (secure 0 0 0) 0x7f0470000c90 con 0x7f04881c69b0 2021-02-10T11:34:47.194 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:47.191 7f048cadf700 1 -- 172.21.2.102:0/3674768569 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f046c018730 msgr2=0x7f046c01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:47.194 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:47.191 7f048cadf700 1 --2- 172.21.2.102:0/3674768569 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f046c018730 0x7f046c01ac10 crc :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:47.194 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:47.191 7f048cadf700 1 -- 172.21.2.102:0/3674768569 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f04881c69b0 msgr2=0x7f048809e4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:47.194 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:47.191 7f048cadf700 1 --2- 172.21.2.102:0/3674768569 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f04881c69b0 0x7f048809e4b0 secure :-1 s=READY pgs=2585 cs=0 l=1 rev1=1 rx=0x7f047000b5e0 tx=0x7f047000b980).stop 2021-02-10T11:34:47.195 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:47.192 7f048cadf700 1 -- 172.21.2.102:0/3674768569 shutdown_connections 2021-02-10T11:34:47.195 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:47.192 7f048cadf700 1 --2- 172.21.2.102:0/3674768569 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f048809f0f0 0x7f048809f4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:47.195 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:47.192 7f048cadf700 1 --2- 172.21.2.102:0/3674768569 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f046c018730 0x7f046c01ac10 unknown :-1 s=CLOSED pgs=1143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:47.195 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:47.192 7f048cadf700 1 --2- 172.21.2.102:0/3674768569 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f04881b5e60 0x7f048809ead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:47.195 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:47.192 7f048cadf700 1 --2- 172.21.2.102:0/3674768569 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f04881c69b0 0x7f048809e4b0 unknown :-1 s=CLOSED pgs=2585 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:47.195 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:47.192 7f048cadf700 1 -- 172.21.2.102:0/3674768569 >> 172.21.2.102:0/3674768569 conn(0x7f048809bad0 msgr2=0x7f04880804e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:47.195 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:47.192 7f048cadf700 1 -- 172.21.2.102:0/3674768569 shutdown_connections 2021-02-10T11:34:47.195 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:47.192 7f048cadf700 1 -- 172.21.2.102:0/3674768569 wait complete. 2021-02-10T11:34:47.202 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs volume ls 2021-02-10T11:34:47.630 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:47.612 7f4fb09b3700 1 Processor -- start 2021-02-10T11:34:47.630 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:47.612 7f4fb09b3700 1 -- start start 2021-02-10T11:34:47.630 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:47.612 7f4fb09b3700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f4fac0d0340 0x7f4fac0ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:47.630 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:47.612 7f4fb09b3700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4fac0d1c70 0x7f4fac0efc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:47.630 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:47.612 7f4fb09b3700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4fac0f0b10 con 0x7f4fac0d1870 2021-02-10T11:34:47.630 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:47.612 7f4fb09b3700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4fac0e5ed0 con 0x7f4fac0d0340 2021-02-10T11:34:47.631 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:47.612 7f4fb09b3700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4fac1b0be0 con 0x7f4fac0d1c70 2021-02-10T11:34:47.631 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:47.612 7f4fa9e1b700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f4fac0d0340 0x7f4fac0ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:47.631 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:47.612 7f4faa61c700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f4fac0d1870 0x7f4fac1b03f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:56834/0 (socket says 172.21.2.102:56834) 2021-02-10T11:34:47.631 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:47.612 7f4faa61c700 1 -- 172.21.2.102:0/3152133106 learned_addr learned my addr 172.21.2.102:0/3152133106 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:34:47.631 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:47.613 7f4fa9e1b700 1 -- 172.21.2.102:0/3152133106 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4fac0d1c70 msgr2=0x7f4fac0efc60 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:34:47.631 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:47.613 7f4fa9e1b700 1 --2- 172.21.2.102:0/3152133106 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4fac0d1c70 0x7f4fac0efc60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:47.631 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:47.613 7f4fa9e1b700 1 -- 172.21.2.102:0/3152133106 >> v1:172.21.2.102:6789/0 conn(0x7f4fac0d1870 legacy=0x7f4fac1b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:47.631 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:47.613 7f4fa9e1b700 1 -- 172.21.2.102:0/3152133106 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fac0f20a0 con 0x7f4fac0d0340 2021-02-10T11:34:47.631 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:47.613 7f4fa9e1b700 1 --2- 172.21.2.102:0/3152133106 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4fac0d0340 0x7f4fac0ef640 secure :-1 s=READY pgs=2587 cs=0 l=1 rev1=1 rx=0x7f4f94006650 tx=0x7f4f94009a80).ready entity=mon.0 client_cookie=f4f4b4da074ba26a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:47.631 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:47.613 7f4fa961a700 1 -- 172.21.2.102:0/3152133106 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4f940009b0 con 0x7f4fac0d0340 2021-02-10T11:34:47.631 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:47.613 7f4fa961a700 1 -- 172.21.2.102:0/3152133106 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f9400c0c0 con 0x7f4fac0d0340 2021-02-10T11:34:47.631 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:47.613 7f4fa961a700 1 -- 172.21.2.102:0/3152133106 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4f9400c520 con 0x7f4fac0d0340 2021-02-10T11:34:47.632 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:47.613 7f4fb09b3700 1 -- 172.21.2.102:0/3152133106 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4fac0d0340 msgr2=0x7f4fac0ef640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:47.632 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:47.613 7f4fb09b3700 1 --2- 172.21.2.102:0/3152133106 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4fac0d0340 0x7f4fac0ef640 secure :-1 s=READY pgs=2587 cs=0 l=1 rev1=1 rx=0x7f4f94006650 tx=0x7f4f94009a80).stop 2021-02-10T11:34:47.632 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:47.614 7f4fb09b3700 1 -- 172.21.2.102:0/3152133106 shutdown_connections 2021-02-10T11:34:47.632 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:47.614 7f4fb09b3700 1 --2- 172.21.2.102:0/3152133106 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4fac0d0340 0x7f4fac0ef640 unknown :-1 s=CLOSED pgs=2587 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:47.632 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:47.614 7f4fb09b3700 1 --2- 172.21.2.102:0/3152133106 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4fac0d1c70 0x7f4fac0efc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:47.632 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:47.614 7f4fb09b3700 1 -- 172.21.2.102:0/3152133106 >> 172.21.2.102:0/3152133106 conn(0x7f4fac0ce450 msgr2=0x7f4fac0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:47.632 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:47.614 7f4fb09b3700 1 -- 172.21.2.102:0/3152133106 shutdown_connections 2021-02-10T11:34:47.632 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:47.614 7f4fb09b3700 1 -- 172.21.2.102:0/3152133106 wait complete. 2021-02-10T11:34:47.634 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:47.628 7f4fb09b3700 1 Processor -- start 2021-02-10T11:34:47.634 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:47.628 7f4fb09b3700 1 -- start start 2021-02-10T11:34:47.634 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:47.629 7f4fb09b3700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4fac0bb5d0 0x7f4fac0bfac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:47.634 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:47.629 7f4fb09b3700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4fac0a9b50 0x7f4fac0a9f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:47.634 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:47.629 7f4fb09b3700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4fac1a7550 0x7f4fac1a7950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:47.634 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:47.629 7f4fb09b3700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fac0bb9d0 con 0x7f4fac0bb5d0 2021-02-10T11:34:47.634 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:47.629 7f4fb09b3700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fac0aa570 con 0x7f4fac1a7550 2021-02-10T11:34:47.634 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:47.629 7f4fb09b3700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4fac0aa790 con 0x7f4fac0a9b50 2021-02-10T11:34:47.634 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:47.629 7f4faa61c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4fac0bb5d0 0x7f4fac0bfac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:47.635 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:47.629 7f4faa61c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4fac0bb5d0 0x7f4fac0bfac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:36384/0 (socket says 172.21.2.102:36384) 2021-02-10T11:34:47.635 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:47.629 7f4faa61c700 1 -- 172.21.2.102:0/2964212578 learned_addr learned my addr 172.21.2.102:0/2964212578 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:34:47.635 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:47.629 7f4faa61c700 1 -- 172.21.2.102:0/2964212578 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4fac0a9b50 msgr2=0x7f4fac0a9f50 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:34:47.635 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:47.629 7f4faa61c700 1 --2- 172.21.2.102:0/2964212578 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4fac0a9b50 0x7f4fac0a9f50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:47.635 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:47.629 7f4faa61c700 1 -- 172.21.2.102:0/2964212578 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4fac1a7550 msgr2=0x7f4fac1a7950 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:34:47.635 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:47.629 7f4faa61c700 1 --2- 172.21.2.102:0/2964212578 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4fac1a7550 0x7f4fac1a7950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:47.635 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:47.629 7f4faa61c700 1 -- 172.21.2.102:0/2964212578 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fac1a7f70 con 0x7f4fac0bb5d0 2021-02-10T11:34:47.635 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:47.629 7f4faa61c700 1 --2- 172.21.2.102:0/2964212578 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4fac0bb5d0 0x7f4fac0bfac0 secure :-1 s=READY pgs=2588 cs=0 l=1 rev1=1 rx=0x7f4f9c001400 tx=0x7f4f9c00a900).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:47.635 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:47.629 7f4fa2ffd700 1 -- 172.21.2.102:0/2964212578 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4f9c00acf0 con 0x7f4fac0bb5d0 2021-02-10T11:34:47.635 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:47.630 7f4fa2ffd700 1 -- 172.21.2.102:0/2964212578 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f9c000c90 con 0x7f4fac0bb5d0 2021-02-10T11:34:47.635 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:47.630 7f4fa2ffd700 1 -- 172.21.2.102:0/2964212578 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4f9c004620 con 0x7f4fac0bb5d0 2021-02-10T11:34:47.635 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:47.630 7f4fb09b3700 1 -- 172.21.2.102:0/2964212578 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4fac1a7f70 con 0x7f4fac0bb5d0 2021-02-10T11:34:47.635 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:47.630 7f4fb09b3700 1 -- 172.21.2.102:0/2964212578 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4fac1a7f70 con 0x7f4fac0bb5d0 2021-02-10T11:34:47.636 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:47.630 7f4fa2ffd700 1 -- 172.21.2.102:0/2964212578 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4f9c0043a0 con 0x7f4fac0bb5d0 2021-02-10T11:34:47.636 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:47.630 7f4fa2ffd700 1 --2- 172.21.2.102:0/2964212578 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4f900186c0 0x7f4f9001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:47.636 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:47.630 7f4fa2ffd700 1 -- 172.21.2.102:0/2964212578 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(137..137 src has 1..137) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f4f9c032540 con 0x7f4fac0bb5d0 2021-02-10T11:34:47.636 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:47.630 7f4fa9e1b700 1 --2- 172.21.2.102:0/2964212578 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4f900186c0 0x7f4f9001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:47.636 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:47.631 7f4fa9e1b700 1 --2- 172.21.2.102:0/2964212578 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4f900186c0 0x7f4f9001abc0 crc :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:47.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:47.662 7f4fb09b3700 1 -- 172.21.2.102:0/2964212578 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4fac0ee0b0 con 0x7f4fac0bb5d0 2021-02-10T11:34:47.666 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:47.665 7f4fa2ffd700 1 -- 172.21.2.102:0/2964212578 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f4f9c00acf0 con 0x7f4fac0bb5d0 2021-02-10T11:34:47.802 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:47.800 7f4fb09b3700 1 -- 172.21.2.102:0/2964212578 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs volume ls", "target": ["mgr", ""]}) v1 -- 0x7f4fac0aa570 con 0x7f4f900186c0 2021-02-10T11:34:47.803 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:47.801 7f4fa2ffd700 1 -- 172.21.2.102:0/2964212578 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+41 (crc 0 0 0) 0x7f4f940093a0 con 0x7f4f900186c0 2021-02-10T11:34:47.803 INFO:teuthology.orchestra.run.gibba002.stdout:[ 2021-02-10T11:34:47.803 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:47.803 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "cephfs" 2021-02-10T11:34:47.803 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:34:47.804 INFO:teuthology.orchestra.run.gibba002.stdout:] 2021-02-10T11:34:47.805 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:47.803 7f4fb09b3700 1 -- 172.21.2.102:0/2964212578 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4f900186c0 msgr2=0x7f4f9001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:47.805 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:47.803 7f4fb09b3700 1 --2- 172.21.2.102:0/2964212578 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4f900186c0 0x7f4f9001abc0 crc :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:47.805 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:47.803 7f4fb09b3700 1 -- 172.21.2.102:0/2964212578 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4fac0bb5d0 msgr2=0x7f4fac0bfac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:47.805 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:47.803 7f4fb09b3700 1 --2- 172.21.2.102:0/2964212578 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4fac0bb5d0 0x7f4fac0bfac0 secure :-1 s=READY pgs=2588 cs=0 l=1 rev1=1 rx=0x7f4f9c001400 tx=0x7f4f9c00a900).stop 2021-02-10T11:34:47.805 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:47.803 7f4fb09b3700 1 -- 172.21.2.102:0/2964212578 shutdown_connections 2021-02-10T11:34:47.805 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:47.803 7f4fb09b3700 1 --2- 172.21.2.102:0/2964212578 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4f900186c0 0x7f4f9001abc0 unknown :-1 s=CLOSED pgs=1144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:47.805 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:47.803 7f4fb09b3700 1 --2- 172.21.2.102:0/2964212578 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4fac1a7550 0x7f4fac1a7950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:47.806 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:47.803 7f4fb09b3700 1 --2- 172.21.2.102:0/2964212578 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4fac0a9b50 0x7f4fac0a9f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:47.806 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:47.803 7f4fb09b3700 1 --2- 172.21.2.102:0/2964212578 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4fac0bb5d0 0x7f4fac0bfac0 unknown :-1 s=CLOSED pgs=2588 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:47.806 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:47.803 7f4fb09b3700 1 -- 172.21.2.102:0/2964212578 >> 172.21.2.102:0/2964212578 conn(0x7f4fac0a6eb0 msgr2=0x7f4fac0bf2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:47.806 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:47.803 7f4fb09b3700 1 -- 172.21.2.102:0/2964212578 shutdown_connections 2021-02-10T11:34:47.806 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:47.803 7f4fb09b3700 1 -- 172.21.2.102:0/2964212578 wait complete. 2021-02-10T11:34:47.813 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config set mon mon_allow_pool_delete True 2021-02-10T11:34:48.250 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:48.230 7ff1471c3700 1 Processor -- start 2021-02-10T11:34:48.250 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:48.230 7ff1471c3700 1 -- start start 2021-02-10T11:34:48.250 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:48.230 7ff1471c3700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7ff14010aff0 0x7ff1400cf750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:48.250 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:48.230 7ff1471c3700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1400fe1c0 con 0x7ff1400d1870 2021-02-10T11:34:48.250 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:48.230 7ff1471c3700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1401b0cb0 con 0x7ff1400d1c70 2021-02-10T11:34:48.250 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:48.230 7ff1471c3700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff140109a60 con 0x7ff14010aff0 2021-02-10T11:34:48.250 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:48.231 7ff144fdf700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7ff1400d1870 0x7ff1401b04c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:56848/0 (socket says 172.21.2.102:56848) 2021-02-10T11:34:48.250 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:48.231 7ff144fdf700 1 -- 172.21.2.102:0/1849654039 learned_addr learned my addr 172.21.2.102:0/1849654039 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:34:48.250 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:48.231 7ff13f7fe700 1 -- 172.21.2.102:0/1849654039 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1635985566 0 0) 0x7ff128000ce0 con 0x7ff1400d1870 2021-02-10T11:34:48.250 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:48.231 7ff13f7fe700 1 -- 172.21.2.102:0/1849654039 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff12c002fe0 con 0x7ff1400d1870 2021-02-10T11:34:48.251 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:48.231 7ff1457e0700 1 --2- 172.21.2.102:0/1849654039 >> v2:172.21.2.113:3300/0 conn(0x7ff14010aff0 0x7ff1400cf750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:48.251 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:48.231 7ff13f7fe700 1 -- 172.21.2.102:0/1849654039 <== mon.1 v1:172.21.2.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3366649325 0 0) 0x7ff130000c60 con 0x7ff1400d1c70 2021-02-10T11:34:48.251 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:48.231 7ff13f7fe700 1 -- 172.21.2.102:0/1849654039 --> v1:172.21.2.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff12c003210 con 0x7ff1400d1c70 2021-02-10T11:34:48.251 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:48.231 7ff13f7fe700 1 -- 172.21.2.102:0/1849654039 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2159504348 0 0) 0x7ff1280010e0 con 0x7ff1400d1870 2021-02-10T11:34:48.251 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:48.231 7ff13f7fe700 1 -- 172.21.2.102:0/1849654039 >> v2:172.21.2.113:3300/0 conn(0x7ff14010aff0 msgr2=0x7ff1400cf750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:34:48.251 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:48.231 7ff13f7fe700 1 --2- 172.21.2.102:0/1849654039 >> v2:172.21.2.113:3300/0 conn(0x7ff14010aff0 0x7ff1400cf750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:48.251 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:48.231 7ff13f7fe700 1 -- 172.21.2.102:0/1849654039 >> v1:172.21.2.113:6789/0 conn(0x7ff1400d1c70 legacy=0x7ff1400cef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:48.251 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:48.231 7ff13f7fe700 1 -- 172.21.2.102:0/1849654039 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff14010b3f0 con 0x7ff1400d1870 2021-02-10T11:34:48.251 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:48.231 7ff13f7fe700 1 -- 172.21.2.102:0/1849654039 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7ff128001660 con 0x7ff1400d1870 2021-02-10T11:34:48.251 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:48.231 7ff1457e0700 1 --2- 172.21.2.102:0/1849654039 >> v2:172.21.2.113:3300/0 conn(0x7ff14010aff0 0x7ff1400cf750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:34:48.251 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:48.231 7ff13f7fe700 1 -- 172.21.2.102:0/1849654039 >> v1:172.21.2.102:6789/0 conn(0x7ff1400d1870 legacy=0x7ff1401b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:48.251 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:48.231 7ff13f7fe700 1 --2- 172.21.2.102:0/1849654039 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff12c003b90 0x7ff12c005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:48.252 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:48.231 7ff13f7fe700 1 --2- 172.21.2.102:0/1849654039 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff12c006820 0x7ff12c008d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:48.252 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:48.231 7ff13f7fe700 1 --2- 172.21.2.102:0/1849654039 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff12c009640 0x7ff12c00bb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:48.252 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:48.231 7ff13f7fe700 1 -- 172.21.2.102:0/1849654039 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff12c0503c0 con 0x7ff12c009640 2021-02-10T11:34:48.252 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:48.231 7ff13f7fe700 1 -- 172.21.2.102:0/1849654039 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff12c050570 con 0x7ff12c003b90 2021-02-10T11:34:48.252 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:48.231 7ff13f7fe700 1 -- 172.21.2.102:0/1849654039 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff12c050750 con 0x7ff12c006820 2021-02-10T11:34:48.252 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:48.232 7ff1457e0700 1 --2- 172.21.2.102:0/1849654039 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff12c009640 0x7ff12c00bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:48.252 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:48.232 7ff144fdf700 1 --2- 172.21.2.102:0/1849654039 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff12c003b90 0x7ff12c005fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:48.252 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:48.232 7ff1457e0700 1 -- 172.21.2.102:0/1849654039 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff12c006820 msgr2=0x7ff12c008d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:34:48.252 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:48.232 7ff13ffff700 1 --2- 172.21.2.102:0/1849654039 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff12c006820 0x7ff12c008d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:48.252 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:48.232 7ff1457e0700 1 --2- 172.21.2.102:0/1849654039 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff12c006820 0x7ff12c008d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:48.252 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:48.232 7ff1457e0700 1 -- 172.21.2.102:0/1849654039 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff12c003b90 msgr2=0x7ff12c005fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:34:48.252 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:48.232 7ff1457e0700 1 --2- 172.21.2.102:0/1849654039 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff12c003b90 0x7ff12c005fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:48.252 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:48.232 7ff1457e0700 1 -- 172.21.2.102:0/1849654039 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff12c0509f0 con 0x7ff12c009640 2021-02-10T11:34:48.252 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:48.232 7ff1457e0700 1 -- 172.21.2.102:0/1849654039 reap_dead start 2021-02-10T11:34:48.253 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:48.232 7ff1457e0700 1 --2- 172.21.2.102:0/1849654039 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff12c009640 0x7ff12c00bb60 secure :-1 s=READY pgs=2590 cs=0 l=1 rev1=1 rx=0x7ff138009ec0 tx=0x7ff13800c370).ready entity=mon.0 client_cookie=f401f5ae894d1c5e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:48.253 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:48.232 7ff13f7fe700 1 -- 172.21.2.102:0/1849654039 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff138007070 con 0x7ff12c009640 2021-02-10T11:34:48.253 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:48.232 7ff13f7fe700 1 -- 172.21.2.102:0/1849654039 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff138006b00 con 0x7ff12c009640 2021-02-10T11:34:48.253 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:48.232 7ff13f7fe700 1 -- 172.21.2.102:0/1849654039 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff1380075e0 con 0x7ff12c009640 2021-02-10T11:34:48.253 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:48.232 7ff1471c3700 1 -- 172.21.2.102:0/1849654039 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff12c009640 msgr2=0x7ff12c00bb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:48.253 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:48.232 7ff1471c3700 1 --2- 172.21.2.102:0/1849654039 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff12c009640 0x7ff12c00bb60 secure :-1 s=READY pgs=2590 cs=0 l=1 rev1=1 rx=0x7ff138009ec0 tx=0x7ff13800c370).stop 2021-02-10T11:34:48.253 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:48.233 7ff1471c3700 1 -- 172.21.2.102:0/1849654039 shutdown_connections 2021-02-10T11:34:48.253 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:48.233 7ff1471c3700 1 --2- 172.21.2.102:0/1849654039 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff12c009640 0x7ff12c00bb60 unknown :-1 s=CLOSED pgs=2590 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:48.253 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:48.233 7ff1471c3700 1 -- 172.21.2.102:0/1849654039 >> 172.21.2.102:0/1849654039 conn(0x7ff1400ce420 msgr2=0x7ff1400d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:48.253 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:48.233 7ff1471c3700 1 -- 172.21.2.102:0/1849654039 shutdown_connections 2021-02-10T11:34:48.253 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:48.233 7ff1471c3700 1 -- 172.21.2.102:0/1849654039 wait complete. 2021-02-10T11:34:48.253 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:48.248 7ff1471c3700 1 Processor -- start 2021-02-10T11:34:48.253 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:48.248 7ff1471c3700 1 -- start start 2021-02-10T11:34:48.255 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:48.248 7ff1471c3700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff14009bcc0 0x7ff14009c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:48.255 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:48.248 7ff1471c3700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff1400a0820 0x7ff1400a0c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:48.255 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:48.248 7ff1471c3700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff1400a1240 0x7ff1400858d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:48.255 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:48.248 7ff1471c3700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1400a66b0 con 0x7ff1400a0820 2021-02-10T11:34:48.256 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:48.248 7ff1471c3700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1401a49b0 con 0x7ff1400a1240 2021-02-10T11:34:48.256 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:48.248 7ff1471c3700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1401a4b90 con 0x7ff14009bcc0 2021-02-10T11:34:48.256 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:48.248 7ff13ffff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff1400a0820 0x7ff1400a0c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:48.256 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:48.248 7ff13ffff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff1400a0820 0x7ff1400a0c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:36404/0 (socket says 172.21.2.102:36404) 2021-02-10T11:34:48.256 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:48.248 7ff13ffff700 1 -- 172.21.2.102:0/1618901438 learned_addr learned my addr 172.21.2.102:0/1618901438 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:34:48.256 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:48.249 7ff13ffff700 1 -- 172.21.2.102:0/1618901438 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff14009bcc0 msgr2=0x7ff14009c0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:48.256 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:48.249 7ff13ffff700 1 --2- 172.21.2.102:0/1618901438 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff14009bcc0 0x7ff14009c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:48.256 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:48.249 7ff13ffff700 1 -- 172.21.2.102:0/1618901438 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff1400a1240 msgr2=0x7ff1400858d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:48.256 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:48.249 7ff13ffff700 1 --2- 172.21.2.102:0/1618901438 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff1400a1240 0x7ff1400858d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:48.256 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:48.249 7ff13ffff700 1 -- 172.21.2.102:0/1618901438 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1400a1640 con 0x7ff1400a0820 2021-02-10T11:34:48.256 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:48.249 7ff13ffff700 1 --2- 172.21.2.102:0/1618901438 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff1400a0820 0x7ff1400a0c20 secure :-1 s=READY pgs=2591 cs=0 l=1 rev1=1 rx=0x7ff130006f60 tx=0x7ff13000b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:48.256 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:48.249 7ff13d7fa700 1 -- 172.21.2.102:0/1618901438 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff130006690 con 0x7ff1400a0820 2021-02-10T11:34:48.256 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:48.249 7ff13d7fa700 1 -- 172.21.2.102:0/1618901438 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff130006030 con 0x7ff1400a0820 2021-02-10T11:34:48.257 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:48.249 7ff13d7fa700 1 -- 172.21.2.102:0/1618901438 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff1300020d0 con 0x7ff1400a0820 2021-02-10T11:34:48.257 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:48.249 7ff1471c3700 1 -- 172.21.2.102:0/1618901438 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1400a1640 con 0x7ff1400a0820 2021-02-10T11:34:48.257 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:48.249 7ff1471c3700 1 -- 172.21.2.102:0/1618901438 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1400a1640 con 0x7ff1400a0820 2021-02-10T11:34:48.257 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:48.250 7ff13d7fa700 1 -- 172.21.2.102:0/1618901438 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ff130006690 con 0x7ff1400a0820 2021-02-10T11:34:48.257 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:48.250 7ff13d7fa700 1 --2- 172.21.2.102:0/1618901438 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff1240186c0 0x7ff12401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:48.257 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:48.250 7ff13d7fa700 1 -- 172.21.2.102:0/1618901438 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(137..137 src has 1..137) v4 ==== 5573+0+0 (secure 0 0 0) 0x7ff1300327f0 con 0x7ff1400a0820 2021-02-10T11:34:48.257 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:48.250 7ff144fdf700 1 --2- 172.21.2.102:0/1618901438 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff1240186c0 0x7ff12401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:48.257 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:48.250 7ff144fdf700 1 --2- 172.21.2.102:0/1618901438 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff1240186c0 0x7ff12401abc0 crc :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:48.281 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:48.280 7ff1471c3700 1 -- 172.21.2.102:0/1618901438 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1400fe1c0 con 0x7ff1400a0820 2021-02-10T11:34:48.284 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:48.282 7ff13d7fa700 1 -- 172.21.2.102:0/1618901438 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ff130006bb0 con 0x7ff1400a0820 2021-02-10T11:34:48.434 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:48.432 7ff1471c3700 1 -- 172.21.2.102:0/1618901438 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7ff140086d30 con 0x7ff1400a0820 2021-02-10T11:34:48.434 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:48.433 7ff13d7fa700 1 -- 172.21.2.102:0/1618901438 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v34)=0 v34) v1 ==== 122+0+0 (secure 0 0 0) 0x7ff130006030 con 0x7ff1400a0820 2021-02-10T11:34:48.437 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:48.434 7ff1471c3700 1 -- 172.21.2.102:0/1618901438 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff1240186c0 msgr2=0x7ff12401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:48.437 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:48.434 7ff1471c3700 1 --2- 172.21.2.102:0/1618901438 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff1240186c0 0x7ff12401abc0 crc :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:48.437 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:48.434 7ff1471c3700 1 -- 172.21.2.102:0/1618901438 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff1400a0820 msgr2=0x7ff1400a0c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:48.438 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:48.434 7ff1471c3700 1 --2- 172.21.2.102:0/1618901438 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff1400a0820 0x7ff1400a0c20 secure :-1 s=READY pgs=2591 cs=0 l=1 rev1=1 rx=0x7ff130006f60 tx=0x7ff13000b970).stop 2021-02-10T11:34:48.438 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:48.435 7ff1471c3700 1 -- 172.21.2.102:0/1618901438 shutdown_connections 2021-02-10T11:34:48.438 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:48.435 7ff1471c3700 1 --2- 172.21.2.102:0/1618901438 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff1240186c0 0x7ff12401abc0 unknown :-1 s=CLOSED pgs=1145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:48.438 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:48.435 7ff1471c3700 1 --2- 172.21.2.102:0/1618901438 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff1400a1240 0x7ff1400858d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:48.438 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:48.435 7ff1471c3700 1 --2- 172.21.2.102:0/1618901438 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff1400a0820 0x7ff1400a0c20 unknown :-1 s=CLOSED pgs=2591 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:48.438 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:48.435 7ff1471c3700 1 --2- 172.21.2.102:0/1618901438 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff14009bcc0 0x7ff14009c0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:48.438 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:48.435 7ff1471c3700 1 -- 172.21.2.102:0/1618901438 >> 172.21.2.102:0/1618901438 conn(0x7ff1400bef00 msgr2=0x7ff14009d1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:48.439 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:48.435 7ff1471c3700 1 -- 172.21.2.102:0/1618901438 shutdown_connections 2021-02-10T11:34:48.439 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:48.435 7ff1471c3700 1 -- 172.21.2.102:0/1618901438 wait complete. 2021-02-10T11:34:48.445 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000690036 --group_name non_existent_group 2021-02-10T11:34:48.879 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:48.861 7fdc5e9aa700 1 Processor -- start 2021-02-10T11:34:48.879 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:48.861 7fdc5e9aa700 1 -- start start 2021-02-10T11:34:48.879 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:48.861 7fdc5e9aa700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fdc580d1880 0x7fdc581b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:48.879 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:48.861 7fdc5e9aa700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdc580cff80 0x7fdc58109140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:48.879 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:48.861 7fdc5e9aa700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc58109a70 con 0x7fdc581b0af0 2021-02-10T11:34:48.879 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:48.861 7fdc5e9aa700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc58109760 con 0x7fdc580d1880 2021-02-10T11:34:48.879 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:48.861 7fdc5e9aa700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc58088070 con 0x7fdc580cff80 2021-02-10T11:34:48.879 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:48.861 7fdc57fff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fdc580d1880 0x7fdc581b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:48.879 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:48.861 7fdc57fff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fdc580d1880 0x7fdc581b04d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:36412/0 (socket says 172.21.2.102:36412) 2021-02-10T11:34:48.879 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:48.861 7fdc57fff700 1 -- 172.21.2.102:0/1770364504 learned_addr learned my addr 172.21.2.102:0/1770364504 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:34:48.880 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:48.862 7fdc5cfc7700 1 --2- 172.21.2.102:0/1770364504 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdc580cff80 0x7fdc58109140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:48.880 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:48.862 7fdc57fff700 1 -- 172.21.2.102:0/1770364504 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdc580cff80 msgr2=0x7fdc58109140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:34:48.880 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:48.862 7fdc57fff700 1 --2- 172.21.2.102:0/1770364504 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdc580cff80 0x7fdc58109140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:48.880 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:48.862 7fdc57fff700 1 -- 172.21.2.102:0/1770364504 >> v1:172.21.2.102:6789/0 conn(0x7fdc581b0af0 legacy=0x7fdc58108950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:48.880 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:48.862 7fdc57fff700 1 -- 172.21.2.102:0/1770364504 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc580d1c80 con 0x7fdc580d1880 2021-02-10T11:34:48.880 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:48.862 7fdc5cfc7700 1 --2- 172.21.2.102:0/1770364504 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdc580cff80 0x7fdc58109140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:34:48.880 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:48.862 7fdc57fff700 1 --2- 172.21.2.102:0/1770364504 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdc580d1880 0x7fdc581b04d0 secure :-1 s=READY pgs=2593 cs=0 l=1 rev1=1 rx=0x7fdc40006700 tx=0x7fdc40009b80).ready entity=mon.0 client_cookie=ef9fc90288102a27 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:48.880 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:48.862 7fdc577fe700 1 -- 172.21.2.102:0/1770364504 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdc4000c0c0 con 0x7fdc580d1880 2021-02-10T11:34:48.880 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:48.862 7fdc577fe700 1 -- 172.21.2.102:0/1770364504 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc4000c380 con 0x7fdc580d1880 2021-02-10T11:34:48.880 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:48.862 7fdc577fe700 1 -- 172.21.2.102:0/1770364504 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdc4000c880 con 0x7fdc580d1880 2021-02-10T11:34:48.880 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:48.862 7fdc5e9aa700 1 -- 172.21.2.102:0/1770364504 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdc580d1880 msgr2=0x7fdc581b04d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:48.880 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:48.862 7fdc5e9aa700 1 --2- 172.21.2.102:0/1770364504 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdc580d1880 0x7fdc581b04d0 secure :-1 s=READY pgs=2593 cs=0 l=1 rev1=1 rx=0x7fdc40006700 tx=0x7fdc40009b80).stop 2021-02-10T11:34:48.881 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:48.863 7fdc5e9aa700 1 -- 172.21.2.102:0/1770364504 shutdown_connections 2021-02-10T11:34:48.881 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:48.863 7fdc5e9aa700 1 --2- 172.21.2.102:0/1770364504 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdc580cff80 0x7fdc58109140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:48.881 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:48.863 7fdc5e9aa700 1 --2- 172.21.2.102:0/1770364504 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdc580d1880 0x7fdc581b04d0 unknown :-1 s=CLOSED pgs=2593 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:48.881 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:48.863 7fdc5e9aa700 1 -- 172.21.2.102:0/1770364504 >> 172.21.2.102:0/1770364504 conn(0x7fdc580ce430 msgr2=0x7fdc580d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:48.881 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:48.863 7fdc5e9aa700 1 -- 172.21.2.102:0/1770364504 shutdown_connections 2021-02-10T11:34:48.881 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:48.863 7fdc5e9aa700 1 -- 172.21.2.102:0/1770364504 wait complete. 2021-02-10T11:34:48.882 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:48.877 7fdc5e9aa700 1 Processor -- start 2021-02-10T11:34:48.882 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:48.877 7fdc5e9aa700 1 -- start start 2021-02-10T11:34:48.883 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:48.878 7fdc5e9aa700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fdc581b8e50 0x7fdc58091bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:48.883 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:48.878 7fdc5e9aa700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdc581b9250 0x7fdc580aa460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:48.883 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:48.878 7fdc5e9aa700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdc580aaa80 0x7fdc581acaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:48.883 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:48.878 7fdc5e9aa700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc581b9770 con 0x7fdc580aaa80 2021-02-10T11:34:48.883 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:48.878 7fdc5e9aa700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc581ac570 con 0x7fdc581b8e50 2021-02-10T11:34:48.883 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:48.878 7fdc5e9aa700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc581ac160 con 0x7fdc581b9250 2021-02-10T11:34:48.883 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:48.878 7fdc5cfc7700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdc580aaa80 0x7fdc581acaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:48.883 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:48.878 7fdc5cfc7700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdc580aaa80 0x7fdc581acaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:36418/0 (socket says 172.21.2.102:36418) 2021-02-10T11:34:48.883 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:48.878 7fdc5cfc7700 1 -- 172.21.2.102:0/388166750 learned_addr learned my addr 172.21.2.102:0/388166750 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:34:48.883 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:48.878 7fdc4ffff700 1 --2- 172.21.2.102:0/388166750 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdc581b9250 0x7fdc580aa460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:48.883 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:48.878 7fdc57fff700 1 --2- 172.21.2.102:0/388166750 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fdc581b8e50 0x7fdc58091bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:48.883 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:48.878 7fdc5cfc7700 1 -- 172.21.2.102:0/388166750 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdc581b9250 msgr2=0x7fdc580aa460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:48.884 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:48.878 7fdc5cfc7700 1 --2- 172.21.2.102:0/388166750 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdc581b9250 0x7fdc580aa460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:48.884 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:48.878 7fdc5cfc7700 1 -- 172.21.2.102:0/388166750 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fdc581b8e50 msgr2=0x7fdc58091bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:48.884 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:48.878 7fdc5cfc7700 1 --2- 172.21.2.102:0/388166750 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fdc581b8e50 0x7fdc58091bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:48.884 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:48.878 7fdc5cfc7700 1 -- 172.21.2.102:0/388166750 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc581ad110 con 0x7fdc580aaa80 2021-02-10T11:34:48.884 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:48.879 7fdc5cfc7700 1 --2- 172.21.2.102:0/388166750 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdc580aaa80 0x7fdc581acaf0 secure :-1 s=READY pgs=2594 cs=0 l=1 rev1=1 rx=0x7fdc50009980 tx=0x7fdc5000bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:48.884 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:48.879 7fdc557fa700 1 -- 172.21.2.102:0/388166750 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdc5000c2f0 con 0x7fdc580aaa80 2021-02-10T11:34:48.884 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:48.879 7fdc557fa700 1 -- 172.21.2.102:0/388166750 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc50006790 con 0x7fdc580aaa80 2021-02-10T11:34:48.884 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:48.879 7fdc557fa700 1 -- 172.21.2.102:0/388166750 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdc5000c7e0 con 0x7fdc580aaa80 2021-02-10T11:34:48.884 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:48.879 7fdc5e9aa700 1 -- 172.21.2.102:0/388166750 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc581ad110 con 0x7fdc580aaa80 2021-02-10T11:34:48.884 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:48.879 7fdc5e9aa700 1 -- 172.21.2.102:0/388166750 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc581ad6a0 con 0x7fdc580aaa80 2021-02-10T11:34:48.884 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:48.880 7fdc557fa700 1 -- 172.21.2.102:0/388166750 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fdc500011f0 con 0x7fdc580aaa80 2021-02-10T11:34:48.884 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:48.880 7fdc557fa700 1 --2- 172.21.2.102:0/388166750 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fdc3c0186c0 0x7fdc3c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:48.884 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:48.880 7fdc557fa700 1 -- 172.21.2.102:0/388166750 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(137..137 src has 1..137) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fdc500327c0 con 0x7fdc580aaa80 2021-02-10T11:34:48.884 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:48.880 7fdc57fff700 1 --2- 172.21.2.102:0/388166750 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fdc3c0186c0 0x7fdc3c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:48.885 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:48.880 7fdc57fff700 1 --2- 172.21.2.102:0/388166750 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fdc3c0186c0 0x7fdc3c01abc0 crc :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:48.913 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:48.911 7fdc5e9aa700 1 -- 172.21.2.102:0/388166750 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc581adf30 con 0x7fdc580aaa80 2021-02-10T11:34:48.915 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:48.913 7fdc557fa700 1 -- 172.21.2.102:0/388166750 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fdc50000f40 con 0x7fdc580aaa80 2021-02-10T11:34:49.042 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:49.040 7fdc5e9aa700 1 -- 172.21.2.102:0/388166750 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000690036", "target": ["mgr", ""], "group_name": "non_existent_group"}) v1 -- 0x7fdc581ad6a0 con 0x7fdc3c0186c0 2021-02-10T11:34:49.045 INFO:tasks.ceph.mgr.y.gibba002.stderr:2021-02-10 11:34:49.044 7fd94bc53700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:34:49.046 INFO:tasks.ceph.mgr.y.gibba002.stderr:2021-02-10 11:34:49.044 7fd94bc53700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:34:49.046 INFO:tasks.ceph.mgr.y.gibba002.stderr:2021-02-10 11:34:49.044 7fd94bc53700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:34:49.046 INFO:tasks.ceph.mgr.y.gibba002.stderr:2021-02-10 11:34:49.044 7fd94bc53700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:34:49.046 INFO:tasks.ceph.mgr.y.gibba002.stderr:2021-02-10 11:34:49.044 7fd94bc53700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:34:49.051 INFO:tasks.ceph.mgr.y.gibba002.stderr:2021-02-10 11:34:49.049 7fd94bc53700 -1 mgr.server reply reply (2) No such file or directory subvolume group 'non_existent_group' does not exist 2021-02-10T11:34:49.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:49.049 7fdc557fa700 1 -- 172.21.2.102:0/388166750 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: -2 subvolume group 'non_existent_group' does not exist) v1 ==== 59+0+0 (crc 0 0 0) 0x7fdc400064f0 con 0x7fdc3c0186c0 2021-02-10T11:34:49.057 INFO:teuthology.orchestra.run.gibba002.stderr:Error ENOENT: subvolume group 'non_existent_group' does not exist 2021-02-10T11:34:49.058 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:49.056 7fdc5e9aa700 1 -- 172.21.2.102:0/388166750 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fdc3c0186c0 msgr2=0x7fdc3c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:49.059 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:49.056 7fdc5e9aa700 1 --2- 172.21.2.102:0/388166750 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fdc3c0186c0 0x7fdc3c01abc0 crc :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:49.059 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:49.056 7fdc5e9aa700 1 -- 172.21.2.102:0/388166750 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdc580aaa80 msgr2=0x7fdc581acaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:49.059 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:49.056 7fdc5e9aa700 1 --2- 172.21.2.102:0/388166750 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdc580aaa80 0x7fdc581acaf0 secure :-1 s=READY pgs=2594 cs=0 l=1 rev1=1 rx=0x7fdc50009980 tx=0x7fdc5000bcd0).stop 2021-02-10T11:34:49.059 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:49.057 7fdc5e9aa700 1 -- 172.21.2.102:0/388166750 shutdown_connections 2021-02-10T11:34:49.059 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:49.057 7fdc5e9aa700 1 --2- 172.21.2.102:0/388166750 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdc580aaa80 0x7fdc581acaf0 unknown :-1 s=CLOSED pgs=2594 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:49.059 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:49.057 7fdc5e9aa700 1 --2- 172.21.2.102:0/388166750 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdc581b9250 0x7fdc580aa460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:49.059 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:49.057 7fdc5e9aa700 1 --2- 172.21.2.102:0/388166750 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fdc3c0186c0 0x7fdc3c01abc0 unknown :-1 s=CLOSED pgs=1146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:49.059 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:49.057 7fdc5e9aa700 1 --2- 172.21.2.102:0/388166750 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fdc581b8e50 0x7fdc58091bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:49.059 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:49.057 7fdc5e9aa700 1 -- 172.21.2.102:0/388166750 >> 172.21.2.102:0/388166750 conn(0x7fdc580846e0 msgr2=0x7fdc581ab3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:49.059 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:49.057 7fdc5e9aa700 1 -- 172.21.2.102:0/388166750 shutdown_connections 2021-02-10T11:34:49.059 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:49.057 7fdc5e9aa700 1 -- 172.21.2.102:0/388166750 wait complete. 2021-02-10T11:34:49.067 DEBUG:teuthology.orchestra.run:got remote process result: 2 2021-02-10T11:34:49.067 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2021-02-10T11:34:49.067 DEBUG:teuthology.orchestra.run.gibba002:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:34:49.413 DEBUG:teuthology.orchestra.run.gibba013:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:34:49.760 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2021-02-10T11:34:49.761 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:34:49.761 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:34:49.761 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:34:50.098 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:34:50.098 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:34:50.099 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@gibba002.front.sepia.ceph.com... 2021-02-10T11:34:50.099 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-02-10T11:34:50.099 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:34:50.452 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba002.stderr:ceph-fuse[193714]: fuse finished with error 0 and tester_r 0 2021-02-10T11:34:50.453 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:34:50.453 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:34:50.453 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:34:50.789 INFO:teuthology.orchestra.run.gibba002.stdout:ext2/ext3 2021-02-10T11:34:50.789 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2021-02-10T11:34:50.789 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:34:50.790 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:34:51.125 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:34:51.125 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:34:51.125 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:34:51.462 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:34:51.462 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2021-02-10T11:34:51.462 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@gibba002.front.sepia.ceph.com... 2021-02-10T11:34:51.462 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-02-10T11:34:51.463 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:34:51.818 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba002.stderr:ceph-fuse[194381]: fuse finished with error 0 and tester_r 0 2021-02-10T11:34:51.819 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:34:51.820 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:34:51.820 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:34:52.155 INFO:teuthology.orchestra.run.gibba002.stdout:ext2/ext3 2021-02-10T11:34:52.155 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2021-02-10T11:34:52.155 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:34:52.155 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:34:52.491 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:34:52.491 DEBUG:teuthology.orchestra.run.gibba013:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:34:52.492 DEBUG:teuthology.orchestra.run.gibba013:> ' -- /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:34:52.827 INFO:teuthology.orchestra.run.gibba013.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.2’: No such file or directory 2021-02-10T11:34:52.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:34:52.828 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2021-02-10T11:34:52.828 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:34:52.828 DEBUG:teuthology.orchestra.run.gibba013:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:34:53.163 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:34:53.163 DEBUG:teuthology.orchestra.run.gibba013:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:34:53.163 DEBUG:teuthology.orchestra.run.gibba013:> ' -- /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:34:53.497 INFO:teuthology.orchestra.run.gibba013.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.3’: No such file or directory 2021-02-10T11:34:53.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:34:53.497 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.3 2021-02-10T11:34:53.498 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:34:53.498 DEBUG:teuthology.orchestra.run.gibba013:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:34:53.833 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config rm mon mon_allow_pool_delete 2021-02-10T11:34:54.265 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:54.246 7fa91efbb700 1 Processor -- start 2021-02-10T11:34:54.265 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:54.246 7fa91efbb700 1 -- start start 2021-02-10T11:34:54.265 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:54.247 7fa91efbb700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa9180d1870 0x7fa9181b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:54.265 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:54.247 7fa91efbb700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fa9181b0a10 0x7fa9180cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:54.265 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:54.247 7fa91efbb700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fa9180d1c70 0x7fa9180cf950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:54.265 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:54.247 7fa91efbb700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9180fe230 con 0x7fa9180d1c70 2021-02-10T11:34:54.265 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:54.247 7fa91efbb700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9180e5ed0 con 0x7fa9181b0a10 2021-02-10T11:34:54.265 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:54.247 7fa91efbb700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9180f0b10 con 0x7fa9180d1870 2021-02-10T11:34:54.266 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:54.247 7fa91d5d8700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fa9180d1c70 0x7fa9180cf950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:54.266 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:54.247 7fa91d5d8700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fa9180d1c70 0x7fa9180cf950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:36436/0 (socket says 172.21.2.102:36436) 2021-02-10T11:34:54.266 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:54.247 7fa91d5d8700 1 -- 172.21.2.102:0/609414384 learned_addr learned my addr 172.21.2.102:0/609414384 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:34:54.266 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:54.248 7fa91d5d8700 1 -- 172.21.2.102:0/609414384 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa9180d1870 msgr2=0x7fa9181b03f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:34:54.266 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:54.248 7fa91d5d8700 1 --2- 172.21.2.102:0/609414384 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa9180d1870 0x7fa9181b03f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:54.266 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:54.248 7fa91d5d8700 1 -- 172.21.2.102:0/609414384 >> v2:172.21.2.113:3300/0 conn(0x7fa9181b0a10 msgr2=0x7fa9180cf330 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:34:54.266 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:54.248 7fa91d5d8700 1 --2- 172.21.2.102:0/609414384 >> v2:172.21.2.113:3300/0 conn(0x7fa9181b0a10 0x7fa9180cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:54.266 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:54.248 7fa91d5d8700 1 -- 172.21.2.102:0/609414384 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9180cff70 con 0x7fa9180d1c70 2021-02-10T11:34:54.266 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:54.248 7fa91d5d8700 1 --2- 172.21.2.102:0/609414384 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa9180d1c70 0x7fa9180cf950 secure :-1 s=READY pgs=2597 cs=0 l=1 rev1=1 rx=0x7fa910006b80 tx=0x7fa910009b10).ready entity=mon.0 client_cookie=404adaa96388f865 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:54.266 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:54.248 7fa9177fe700 1 -- 172.21.2.102:0/609414384 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa9100022e0 con 0x7fa9180d1c70 2021-02-10T11:34:54.266 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:54.248 7fa9177fe700 1 -- 172.21.2.102:0/609414384 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa910002620 con 0x7fa9180d1c70 2021-02-10T11:34:54.267 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:54.248 7fa9177fe700 1 -- 172.21.2.102:0/609414384 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa910002b40 con 0x7fa9180d1c70 2021-02-10T11:34:54.267 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:54.248 7fa91efbb700 1 -- 172.21.2.102:0/609414384 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa9180d1c70 msgr2=0x7fa9180cf950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:54.267 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:54.248 7fa91efbb700 1 --2- 172.21.2.102:0/609414384 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa9180d1c70 0x7fa9180cf950 secure :-1 s=READY pgs=2597 cs=0 l=1 rev1=1 rx=0x7fa910006b80 tx=0x7fa910009b10).stop 2021-02-10T11:34:54.267 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:54.249 7fa91efbb700 1 -- 172.21.2.102:0/609414384 shutdown_connections 2021-02-10T11:34:54.267 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:54.249 7fa91efbb700 1 --2- 172.21.2.102:0/609414384 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa9180d1c70 0x7fa9180cf950 unknown :-1 s=CLOSED pgs=2597 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:54.267 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:54.249 7fa91efbb700 1 --2- 172.21.2.102:0/609414384 >> v2:172.21.2.113:3300/0 conn(0x7fa9181b0a10 0x7fa9180cf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:54.267 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:54.249 7fa91efbb700 1 --2- 172.21.2.102:0/609414384 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa9180d1870 0x7fa9181b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:54.267 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:54.249 7fa91efbb700 1 -- 172.21.2.102:0/609414384 >> 172.21.2.102:0/609414384 conn(0x7fa9180ce450 msgr2=0x7fa9180d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:54.267 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:54.249 7fa91efbb700 1 -- 172.21.2.102:0/609414384 shutdown_connections 2021-02-10T11:34:54.267 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:54.249 7fa91efbb700 1 -- 172.21.2.102:0/609414384 wait complete. 2021-02-10T11:34:54.270 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:54.263 7fa91efbb700 1 Processor -- start 2021-02-10T11:34:54.271 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:54.263 7fa91efbb700 1 -- start start 2021-02-10T11:34:54.271 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:54.264 7fa91efbb700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa91818f910 0x7fa91809f230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:54.271 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:54.264 7fa91efbb700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa91809dd80 0x7fa91809e180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:54.271 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:54.264 7fa91efbb700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa91809e7a0 0x7fa9180eda70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:54.271 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:54.264 7fa91efbb700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa91818fd10 con 0x7fa91818f910 2021-02-10T11:34:54.271 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:54.264 7fa91efbb700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa91809f850 con 0x7fa91809dd80 2021-02-10T11:34:54.271 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:54.264 7fa91efbb700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa91809eba0 con 0x7fa91809e7a0 2021-02-10T11:34:54.271 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:54.264 7fa91cdd7700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa91818f910 0x7fa91809f230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:54.271 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:54.264 7fa91cdd7700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa91818f910 0x7fa91809f230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:36442/0 (socket says 172.21.2.102:36442) 2021-02-10T11:34:54.271 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:54.264 7fa91cdd7700 1 -- 172.21.2.102:0/3138326298 learned_addr learned my addr 172.21.2.102:0/3138326298 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:34:54.272 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:54.265 7fa91cdd7700 1 -- 172.21.2.102:0/3138326298 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa91809e7a0 msgr2=0x7fa9180eda70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:54.272 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:54.265 7fa91cdd7700 1 --2- 172.21.2.102:0/3138326298 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa91809e7a0 0x7fa9180eda70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:54.272 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:54.265 7fa91cdd7700 1 -- 172.21.2.102:0/3138326298 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa91809dd80 msgr2=0x7fa91809e180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:54.272 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:54.265 7fa91cdd7700 1 --2- 172.21.2.102:0/3138326298 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa91809dd80 0x7fa91809e180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:54.272 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:54.265 7fa91cdd7700 1 -- 172.21.2.102:0/3138326298 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa91809edc0 con 0x7fa91818f910 2021-02-10T11:34:54.273 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:54.265 7fa91cdd7700 1 --2- 172.21.2.102:0/3138326298 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa91818f910 0x7fa91809f230 secure :-1 s=READY pgs=2598 cs=0 l=1 rev1=1 rx=0x7fa90000b560 tx=0x7fa90000b900).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:54.273 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:54.265 7fa9157fa700 1 -- 172.21.2.102:0/3138326298 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa90000bcf0 con 0x7fa91818f910 2021-02-10T11:34:54.273 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:54.265 7fa9157fa700 1 -- 172.21.2.102:0/3138326298 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa900006030 con 0x7fa91818f910 2021-02-10T11:34:54.273 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:54.265 7fa9157fa700 1 -- 172.21.2.102:0/3138326298 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa9000054d0 con 0x7fa91818f910 2021-02-10T11:34:54.273 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:54.265 7fa91efbb700 1 -- 172.21.2.102:0/3138326298 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa91809edc0 con 0x7fa91818f910 2021-02-10T11:34:54.273 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:54.265 7fa91efbb700 1 -- 172.21.2.102:0/3138326298 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa91809edc0 con 0x7fa91818f910 2021-02-10T11:34:54.273 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:54.266 7fa9157fa700 1 -- 172.21.2.102:0/3138326298 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa900006210 con 0x7fa91818f910 2021-02-10T11:34:54.273 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:54.266 7fa9157fa700 1 --2- 172.21.2.102:0/3138326298 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa8fc0186c0 0x7fa8fc01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:54.273 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:54.266 7fa9157fa700 1 -- 172.21.2.102:0/3138326298 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(137..137 src has 1..137) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fa9000327d0 con 0x7fa91818f910 2021-02-10T11:34:54.273 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:54.266 7fa917fff700 1 --2- 172.21.2.102:0/3138326298 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa8fc0186c0 0x7fa8fc01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:54.273 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:54.266 7fa917fff700 1 --2- 172.21.2.102:0/3138326298 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa8fc0186c0 0x7fa8fc01abc0 crc :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:54.300 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:54.297 7fa91efbb700 1 -- 172.21.2.102:0/3138326298 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9180f0b10 con 0x7fa91818f910 2021-02-10T11:34:54.301 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:54.300 7fa9157fa700 1 -- 172.21.2.102:0/3138326298 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fa900006b80 con 0x7fa91818f910 2021-02-10T11:34:54.451 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:54.449 7fa91efbb700 1 -- 172.21.2.102:0/3138326298 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fa9180f0b10 con 0x7fa91818f910 2021-02-10T11:34:54.454 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:54.453 7fa9157fa700 1 -- 172.21.2.102:0/3138326298 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v35) v1 ==== 104+0+0 (secure 0 0 0) 0x7fa900006b80 con 0x7fa91818f910 2021-02-10T11:34:54.459 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:54.457 7fa91efbb700 1 -- 172.21.2.102:0/3138326298 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa8fc0186c0 msgr2=0x7fa8fc01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:54.459 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:54.457 7fa91efbb700 1 --2- 172.21.2.102:0/3138326298 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa8fc0186c0 0x7fa8fc01abc0 crc :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:54.459 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:54.457 7fa91efbb700 1 -- 172.21.2.102:0/3138326298 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa91818f910 msgr2=0x7fa91809f230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:54.459 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:54.457 7fa91efbb700 1 --2- 172.21.2.102:0/3138326298 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa91818f910 0x7fa91809f230 secure :-1 s=READY pgs=2598 cs=0 l=1 rev1=1 rx=0x7fa90000b560 tx=0x7fa90000b900).stop 2021-02-10T11:34:54.460 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:54.458 7fa91efbb700 1 -- 172.21.2.102:0/3138326298 shutdown_connections 2021-02-10T11:34:54.460 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:54.458 7fa91efbb700 1 --2- 172.21.2.102:0/3138326298 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa91809e7a0 0x7fa9180eda70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:54.460 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:54.458 7fa91efbb700 1 --2- 172.21.2.102:0/3138326298 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa8fc0186c0 0x7fa8fc01abc0 unknown :-1 s=CLOSED pgs=1147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:54.460 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:54.458 7fa91efbb700 1 --2- 172.21.2.102:0/3138326298 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa91809dd80 0x7fa91809e180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:54.460 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:54.458 7fa91efbb700 1 --2- 172.21.2.102:0/3138326298 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa91818f910 0x7fa91809f230 unknown :-1 s=CLOSED pgs=2598 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:54.460 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:54.458 7fa91efbb700 1 -- 172.21.2.102:0/3138326298 >> 172.21.2.102:0/3138326298 conn(0x7fa91809fd30 msgr2=0x7fa918190100 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:54.460 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:54.458 7fa91efbb700 1 -- 172.21.2.102:0/3138326298 shutdown_connections 2021-02-10T11:34:54.460 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:54.458 7fa91efbb700 1 -- 172.21.2.102:0/3138326298 wait complete. 2021-02-10T11:34:54.468 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestVolumes.test_nonexistent_subvolume_group_create' 2021-02-10T11:34:54.898 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:54.878 7f701c591700 1 Processor -- start 2021-02-10T11:34:54.898 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:54.878 7f701c591700 1 -- start start 2021-02-10T11:34:54.898 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:54.879 7f701c591700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f70140d1870 0x7f70141b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:54.898 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:54.879 7f701c591700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70140f0b10 con 0x7f70141b0a10 2021-02-10T11:34:54.898 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:54.879 7f701c591700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70140d0310 con 0x7f70140f0650 2021-02-10T11:34:54.898 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:54.879 7f701c591700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70140d0530 con 0x7f70140d1870 2021-02-10T11:34:54.898 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:54.879 7f7019bac700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f70141b0a10 0x7f70140cf330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:56906/0 (socket says 172.21.2.102:56906) 2021-02-10T11:34:54.898 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:54.879 7f7019bac700 1 -- 172.21.2.102:0/2268787563 learned_addr learned my addr 172.21.2.102:0/2268787563 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:34:54.898 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:54.879 7f70193ab700 1 -- 172.21.2.102:0/2268787563 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3340917889 0 0) 0x7f7010000ce0 con 0x7f70141b0a10 2021-02-10T11:34:54.899 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:54.879 7f70193ab700 1 -- 172.21.2.102:0/2268787563 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7000002fe0 con 0x7f70141b0a10 2021-02-10T11:34:54.899 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:54.879 7f701a3ad700 1 --2- 172.21.2.102:0/2268787563 >> v2:172.21.2.113:3300/0 conn(0x7f70140d1870 0x7f70141b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:54.899 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:54.880 7f70193ab700 1 -- 172.21.2.102:0/2268787563 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1055919863 0 0) 0x7f70100010e0 con 0x7f70141b0a10 2021-02-10T11:34:54.899 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:54.880 7f70193ab700 1 -- 172.21.2.102:0/2268787563 >> v2:172.21.2.113:3300/0 conn(0x7f70140d1870 msgr2=0x7f70141b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:34:54.899 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:54.880 7f70193ab700 1 --2- 172.21.2.102:0/2268787563 >> v2:172.21.2.113:3300/0 conn(0x7f70140d1870 0x7f70141b03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:54.899 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:54.880 7f70193ab700 1 -- 172.21.2.102:0/2268787563 >> v1:172.21.2.113:6789/0 conn(0x7f70140f0650 legacy=0x7f70140cfb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:54.899 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:54.880 7f70193ab700 1 -- 172.21.2.102:0/2268787563 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70140d1c70 con 0x7f70141b0a10 2021-02-10T11:34:54.899 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:54.880 7f70193ab700 1 -- 172.21.2.102:0/2268787563 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7f7010001660 con 0x7f70141b0a10 2021-02-10T11:34:54.899 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:54.880 7f70193ab700 1 -- 172.21.2.102:0/2268787563 >> v1:172.21.2.102:6789/0 conn(0x7f70141b0a10 legacy=0x7f70140cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:54.899 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:54.880 7f70193ab700 1 --2- 172.21.2.102:0/2268787563 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7000003af0 0x7f7000006010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:54.899 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:54.880 7f70193ab700 1 --2- 172.21.2.102:0/2268787563 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f70000067a0 0x7f7000008cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:54.899 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:54.880 7f70193ab700 1 --2- 172.21.2.102:0/2268787563 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f70000095c0 0x7f700000baa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:54.899 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:54.880 7f70193ab700 1 -- 172.21.2.102:0/2268787563 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f700000c2f0 con 0x7f7000003af0 2021-02-10T11:34:54.900 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:54.880 7f70193ab700 1 -- 172.21.2.102:0/2268787563 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f700000c510 con 0x7f70000067a0 2021-02-10T11:34:54.900 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:54.880 7f70193ab700 1 -- 172.21.2.102:0/2268787563 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7000050740 con 0x7f70000095c0 2021-02-10T11:34:54.900 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:54.880 7f701a3ad700 1 --2- 172.21.2.102:0/2268787563 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7000003af0 0x7f7000006010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:54.900 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:54.880 7f7019bac700 1 --2- 172.21.2.102:0/2268787563 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f70000067a0 0x7f7000008cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:54.900 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:54.880 7f701abae700 1 --2- 172.21.2.102:0/2268787563 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f70000095c0 0x7f700000baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:54.900 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:54.880 7f701a3ad700 1 -- 172.21.2.102:0/2268787563 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f70000095c0 msgr2=0x7f700000baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:34:54.900 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:54.880 7f701a3ad700 1 --2- 172.21.2.102:0/2268787563 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f70000095c0 0x7f700000baa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:54.900 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:54.880 7f701a3ad700 1 -- 172.21.2.102:0/2268787563 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f70000067a0 msgr2=0x7f7000008cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:34:54.900 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:54.880 7f701a3ad700 1 --2- 172.21.2.102:0/2268787563 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f70000067a0 0x7f7000008cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:54.900 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:54.880 7f701a3ad700 1 -- 172.21.2.102:0/2268787563 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70000509a0 con 0x7f7000003af0 2021-02-10T11:34:54.900 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:54.880 7f701abae700 1 -- 172.21.2.102:0/2268787563 reap_dead start 2021-02-10T11:34:54.900 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:54.880 7f7019bac700 1 --2- 172.21.2.102:0/2268787563 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f70000067a0 0x7f7000008cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:34:54.900 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:54.880 7f701a3ad700 1 --2- 172.21.2.102:0/2268787563 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7000003af0 0x7f7000006010 secure :-1 s=READY pgs=2600 cs=0 l=1 rev1=1 rx=0x7f700800c8f0 tx=0x7f700800cc90).ready entity=mon.0 client_cookie=c01a0e275cc961cf server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:54.901 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:54.880 7f70193ab700 1 -- 172.21.2.102:0/2268787563 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7008009310 con 0x7f7000003af0 2021-02-10T11:34:54.901 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:54.880 7f70193ab700 1 -- 172.21.2.102:0/2268787563 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7008009610 con 0x7f7000003af0 2021-02-10T11:34:54.901 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:54.880 7f70193ab700 1 -- 172.21.2.102:0/2268787563 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7008009af0 con 0x7f7000003af0 2021-02-10T11:34:54.901 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:54.880 7f701c591700 1 -- 172.21.2.102:0/2268787563 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7000003af0 msgr2=0x7f7000006010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:54.901 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:54.880 7f701c591700 1 --2- 172.21.2.102:0/2268787563 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7000003af0 0x7f7000006010 secure :-1 s=READY pgs=2600 cs=0 l=1 rev1=1 rx=0x7f700800c8f0 tx=0x7f700800cc90).stop 2021-02-10T11:34:54.901 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:54.881 7f701c591700 1 -- 172.21.2.102:0/2268787563 shutdown_connections 2021-02-10T11:34:54.901 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:54.881 7f701c591700 1 --2- 172.21.2.102:0/2268787563 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7000003af0 0x7f7000006010 unknown :-1 s=CLOSED pgs=2600 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:54.901 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:54.881 7f701c591700 1 -- 172.21.2.102:0/2268787563 >> 172.21.2.102:0/2268787563 conn(0x7f70140ce450 msgr2=0x7f70140d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:54.901 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:54.881 7f701c591700 1 -- 172.21.2.102:0/2268787563 shutdown_connections 2021-02-10T11:34:54.901 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:54.881 7f701c591700 1 -- 172.21.2.102:0/2268787563 wait complete. 2021-02-10T11:34:54.901 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:54.896 7f701c591700 1 Processor -- start 2021-02-10T11:34:54.901 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:54.896 7f701c591700 1 -- start start 2021-02-10T11:34:54.903 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:54.896 7f701c591700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f70141a4a60 0x7f70140c1c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:54.903 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:54.896 7f701c591700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f70140c2290 0x7f70140c2690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:54.903 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:54.896 7f701c591700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f70140c2cb0 0x7f70140bfd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:54.903 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:54.896 7f701c591700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70140f05b0 con 0x7f70141a4a60 2021-02-10T11:34:54.903 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:54.896 7f701c591700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70140f07d0 con 0x7f70140c2cb0 2021-02-10T11:34:54.903 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:54.896 7f701c591700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70140c0340 con 0x7f70140c2290 2021-02-10T11:34:54.903 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:54.897 7f701a3ad700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f70141a4a60 0x7f70140c1c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:54.903 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:54.897 7f701a3ad700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f70141a4a60 0x7f70140c1c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:36462/0 (socket says 172.21.2.102:36462) 2021-02-10T11:34:54.903 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:54.897 7f701a3ad700 1 -- 172.21.2.102:0/3189501076 learned_addr learned my addr 172.21.2.102:0/3189501076 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:34:54.903 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:54.897 7f701a3ad700 1 -- 172.21.2.102:0/3189501076 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f70140c2290 msgr2=0x7f70140c2690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:54.903 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:54.897 7f701a3ad700 1 --2- 172.21.2.102:0/3189501076 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f70140c2290 0x7f70140c2690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:54.904 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:54.897 7f701a3ad700 1 -- 172.21.2.102:0/3189501076 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f70140c2cb0 msgr2=0x7f70140bfd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:54.904 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:54.897 7f701a3ad700 1 --2- 172.21.2.102:0/3189501076 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f70140c2cb0 0x7f70140bfd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:54.904 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:54.897 7f701a3ad700 1 -- 172.21.2.102:0/3189501076 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70140c0560 con 0x7f70141a4a60 2021-02-10T11:34:54.904 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:54.897 7f701a3ad700 1 --2- 172.21.2.102:0/3189501076 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f70141a4a60 0x7f70140c1c70 secure :-1 s=READY pgs=2601 cs=0 l=1 rev1=1 rx=0x7f7008006fc0 tx=0x7f7008009f70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:54.904 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:54.897 7f7006ffd700 1 -- 172.21.2.102:0/3189501076 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7008006030 con 0x7f70141a4a60 2021-02-10T11:34:54.904 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:54.897 7f7006ffd700 1 -- 172.21.2.102:0/3189501076 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f700800c770 con 0x7f70141a4a60 2021-02-10T11:34:54.904 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:54.897 7f7006ffd700 1 -- 172.21.2.102:0/3189501076 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f700800cc10 con 0x7f70141a4a60 2021-02-10T11:34:54.904 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:54.897 7f701c591700 1 -- 172.21.2.102:0/3189501076 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70140c0560 con 0x7f70141a4a60 2021-02-10T11:34:54.904 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:54.897 7f701c591700 1 -- 172.21.2.102:0/3189501076 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70140c0af0 con 0x7f70141a4a60 2021-02-10T11:34:54.904 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:54.898 7f7006ffd700 1 -- 172.21.2.102:0/3189501076 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f7008006030 con 0x7f70141a4a60 2021-02-10T11:34:54.904 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:54.898 7f7006ffd700 1 --2- 172.21.2.102:0/3189501076 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6ffc0186c0 0x7f6ffc01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:54.904 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:54.898 7f7006ffd700 1 -- 172.21.2.102:0/3189501076 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(137..137 src has 1..137) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f70080323b0 con 0x7f70141a4a60 2021-02-10T11:34:54.904 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:54.898 7f7019bac700 1 --2- 172.21.2.102:0/3189501076 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6ffc0186c0 0x7f6ffc01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:54.905 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:54.899 7f7019bac700 1 --2- 172.21.2.102:0/3189501076 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6ffc0186c0 0x7f6ffc01abc0 crc :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:54.931 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:54.929 7f701c591700 1 -- 172.21.2.102:0/3189501076 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70140f0af0 con 0x7f70141a4a60 2021-02-10T11:34:54.933 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:54.931 7f7006ffd700 1 -- 172.21.2.102:0/3189501076 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f7008030ce0 con 0x7f70141a4a60 2021-02-10T11:34:55.065 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:55.063 7f701c591700 1 -- 172.21.2.102:0/3189501076 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestVolumes.test_nonexistent_subvolume_group_create"]} v 0) v1 -- 0x7f70140f0af0 con 0x7f70141a4a60 2021-02-10T11:34:55.143 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:55.141 7f7006ffd700 1 -- 172.21.2.102:0/3189501076 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestVolumes.test_nonexistent_subvolume_group_create"]}]=0 v699) v1 ==== 156+0+0 (secure 0 0 0) 0x7f7008006030 con 0x7f70141a4a60 2021-02-10T11:34:55.182 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:55.180 7f701c591700 1 -- 172.21.2.102:0/3189501076 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6ffc0186c0 msgr2=0x7f6ffc01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:55.182 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:55.180 7f701c591700 1 --2- 172.21.2.102:0/3189501076 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6ffc0186c0 0x7f6ffc01abc0 crc :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:55.182 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:55.180 7f701c591700 1 -- 172.21.2.102:0/3189501076 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f70141a4a60 msgr2=0x7f70140c1c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:55.182 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:55.180 7f701c591700 1 --2- 172.21.2.102:0/3189501076 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f70141a4a60 0x7f70140c1c70 secure :-1 s=READY pgs=2601 cs=0 l=1 rev1=1 rx=0x7f7008006fc0 tx=0x7f7008009f70).stop 2021-02-10T11:34:55.183 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:55.180 7f701c591700 1 -- 172.21.2.102:0/3189501076 shutdown_connections 2021-02-10T11:34:55.183 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:55.180 7f701c591700 1 --2- 172.21.2.102:0/3189501076 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6ffc0186c0 0x7f6ffc01abc0 unknown :-1 s=CLOSED pgs=1148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:55.183 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:55.180 7f701c591700 1 --2- 172.21.2.102:0/3189501076 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f70140c2cb0 0x7f70140bfd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:55.183 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:55.180 7f701c591700 1 --2- 172.21.2.102:0/3189501076 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f70140c2290 0x7f70140c2690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:55.183 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:55.180 7f701c591700 1 --2- 172.21.2.102:0/3189501076 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f70141a4a60 0x7f70140c1c70 unknown :-1 s=CLOSED pgs=2601 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:55.183 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:55.180 7f701c591700 1 -- 172.21.2.102:0/3189501076 >> 172.21.2.102:0/3189501076 conn(0x7f70140ebe30 msgr2=0x7f70140bf200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:55.183 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:55.180 7f701c591700 1 -- 172.21.2.102:0/3189501076 shutdown_connections 2021-02-10T11:34:55.183 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:55.180 7f701c591700 1 -- 172.21.2.102:0/3189501076 wait complete. 2021-02-10T11:34:55.190 INFO:tasks.cephfs_test_runner:test_nonexistent_subvolume_group_create (tasks.cephfs.test_volumes.TestVolumes) ... ok 2021-02-10T11:34:55.191 INFO:tasks.cephfs_test_runner:Starting test: test_nonexistent_subvolume_group_rm (tasks.cephfs.test_volumes.TestVolumes) 2021-02-10T11:34:55.191 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestVolumes.test_nonexistent_subvolume_group_rm' 2021-02-10T11:34:55.620 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:55.600 7f0ba8fdb700 1 Processor -- start 2021-02-10T11:34:55.620 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:55.600 7f0ba8fdb700 1 -- start start 2021-02-10T11:34:55.620 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:55.600 7f0ba8fdb700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f0ba40d1870 0x7f0ba41b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:55.620 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:55.600 7f0ba8fdb700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0ba40f0650 0x7f0ba40cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:55.620 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:55.600 7f0ba8fdb700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ba40f0b10 con 0x7f0ba41b0a10 2021-02-10T11:34:55.620 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:55.600 7f0ba8fdb700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ba40e5ed0 con 0x7f0ba40d1870 2021-02-10T11:34:55.620 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:55.600 7f0ba8fdb700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ba40d1c70 con 0x7f0ba40f0650 2021-02-10T11:34:55.621 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:55.601 7f0ba259c700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f0ba41b0a10 0x7f0ba40cf330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:56926/0 (socket says 172.21.2.102:56926) 2021-02-10T11:34:55.621 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:55.601 7f0ba259c700 1 -- 172.21.2.102:0/1058857348 learned_addr learned my addr 172.21.2.102:0/1058857348 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:34:55.621 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:55.601 7f0ba1d9b700 1 -- 172.21.2.102:0/1058857348 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2866320806 0 0) 0x7f0b8c0010b0 con 0x7f0ba41b0a10 2021-02-10T11:34:55.621 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:55.601 7f0ba1d9b700 1 -- 172.21.2.102:0/1058857348 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b90002fe0 con 0x7f0ba41b0a10 2021-02-10T11:34:55.621 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:55.601 7f0ba359e700 1 --2- 172.21.2.102:0/1058857348 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0ba40f0650 0x7f0ba40cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:55.621 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:55.601 7f0ba2d9d700 1 --2- 172.21.2.102:0/1058857348 >> v2:172.21.2.113:3300/0 conn(0x7f0ba40d1870 0x7f0ba41b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:55.621 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:55.602 7f0ba1d9b700 1 -- 172.21.2.102:0/1058857348 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4213948684 0 0) 0x7f0b8c0010b0 con 0x7f0ba41b0a10 2021-02-10T11:34:55.621 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:55.602 7f0ba1d9b700 1 -- 172.21.2.102:0/1058857348 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0ba40f0650 msgr2=0x7f0ba40cfb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:34:55.621 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:55.602 7f0ba1d9b700 1 --2- 172.21.2.102:0/1058857348 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0ba40f0650 0x7f0ba40cfb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:55.621 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:55.602 7f0ba1d9b700 1 -- 172.21.2.102:0/1058857348 >> v2:172.21.2.113:3300/0 conn(0x7f0ba40d1870 msgr2=0x7f0ba41b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:34:55.621 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:55.602 7f0ba1d9b700 1 --2- 172.21.2.102:0/1058857348 >> v2:172.21.2.113:3300/0 conn(0x7f0ba40d1870 0x7f0ba41b03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:55.621 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:55.602 7f0ba1d9b700 1 -- 172.21.2.102:0/1058857348 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ba40d1e90 con 0x7f0ba41b0a10 2021-02-10T11:34:55.622 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:55.602 7f0ba1d9b700 1 -- 172.21.2.102:0/1058857348 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7f0b8c0015b0 con 0x7f0ba41b0a10 2021-02-10T11:34:55.622 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:55.602 7f0ba1d9b700 1 -- 172.21.2.102:0/1058857348 >> v1:172.21.2.102:6789/0 conn(0x7f0ba41b0a10 legacy=0x7f0ba40cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:55.622 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:55.602 7f0ba1d9b700 1 --2- 172.21.2.102:0/1058857348 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0b90003a40 0x7f0b90005eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:55.622 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:55.602 7f0ba1d9b700 1 --2- 172.21.2.102:0/1058857348 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0b9004a760 0x7f0b9004cc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:55.622 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:55.602 7f0ba1d9b700 1 --2- 172.21.2.102:0/1058857348 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0b9004d500 0x7f0b9004fa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:55.622 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:55.602 7f0ba1d9b700 1 -- 172.21.2.102:0/1058857348 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b900502d0 con 0x7f0b9004a760 2021-02-10T11:34:55.622 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:55.602 7f0ba1d9b700 1 -- 172.21.2.102:0/1058857348 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b900504f0 con 0x7f0b9004d500 2021-02-10T11:34:55.622 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:55.602 7f0ba1d9b700 1 -- 172.21.2.102:0/1058857348 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b90050710 con 0x7f0b90003a40 2021-02-10T11:34:55.622 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:55.602 7f0ba259c700 1 --2- 172.21.2.102:0/1058857348 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0b9004a760 0x7f0b9004cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:55.622 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:55.602 7f0ba2d9d700 1 --2- 172.21.2.102:0/1058857348 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0b90003a40 0x7f0b90005eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:55.622 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:55.602 7f0ba359e700 1 --2- 172.21.2.102:0/1058857348 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0b9004d500 0x7f0b9004fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:55.622 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:55.602 7f0ba259c700 1 -- 172.21.2.102:0/1058857348 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0b90003a40 msgr2=0x7f0b90005eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:34:55.622 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:55.602 7f0ba259c700 1 --2- 172.21.2.102:0/1058857348 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0b90003a40 0x7f0b90005eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:55.623 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:55.602 7f0ba259c700 1 -- 172.21.2.102:0/1058857348 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0b9004d500 msgr2=0x7f0b9004fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:34:55.623 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:55.602 7f0ba259c700 1 --2- 172.21.2.102:0/1058857348 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0b9004d500 0x7f0b9004fa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:55.623 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:55.602 7f0ba259c700 1 -- 172.21.2.102:0/1058857348 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b900509b0 con 0x7f0b9004a760 2021-02-10T11:34:55.623 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:55.602 7f0ba259c700 1 --2- 172.21.2.102:0/1058857348 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0b9004a760 0x7f0b9004cc60 secure :-1 s=READY pgs=2603 cs=0 l=1 rev1=1 rx=0x7f0b8c001370 tx=0x7f0b8c007d30).ready entity=mon.0 client_cookie=9aa689626c738826 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:55.623 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:55.602 7f0ba1d9b700 1 -- 172.21.2.102:0/1058857348 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0b8c0070c0 con 0x7f0b9004a760 2021-02-10T11:34:55.623 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:55.602 7f0ba1d9b700 1 -- 172.21.2.102:0/1058857348 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b8c0010b0 con 0x7f0b9004a760 2021-02-10T11:34:55.623 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:55.602 7f0ba1d9b700 1 -- 172.21.2.102:0/1058857348 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0b8c0070c0 con 0x7f0b9004a760 2021-02-10T11:34:55.623 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:55.603 7f0ba8fdb700 1 -- 172.21.2.102:0/1058857348 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0b9004a760 msgr2=0x7f0b9004cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:55.623 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:55.603 7f0ba8fdb700 1 --2- 172.21.2.102:0/1058857348 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0b9004a760 0x7f0b9004cc60 secure :-1 s=READY pgs=2603 cs=0 l=1 rev1=1 rx=0x7f0b8c001370 tx=0x7f0b8c007d30).stop 2021-02-10T11:34:55.623 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:55.603 7f0ba359e700 1 -- 172.21.2.102:0/1058857348 reap_dead start 2021-02-10T11:34:55.623 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:55.603 7f0ba8fdb700 1 -- 172.21.2.102:0/1058857348 shutdown_connections 2021-02-10T11:34:55.623 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:55.603 7f0ba8fdb700 1 -- 172.21.2.102:0/1058857348 >> 172.21.2.102:0/1058857348 conn(0x7f0ba40ce450 msgr2=0x7f0ba40d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:55.624 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:55.603 7f0ba8fdb700 1 -- 172.21.2.102:0/1058857348 shutdown_connections 2021-02-10T11:34:55.624 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:55.603 7f0ba8fdb700 1 -- 172.21.2.102:0/1058857348 wait complete. 2021-02-10T11:34:55.624 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:55.618 7f0ba8fdb700 1 Processor -- start 2021-02-10T11:34:55.624 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:55.618 7f0ba8fdb700 1 -- start start 2021-02-10T11:34:55.625 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:55.618 7f0ba8fdb700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0ba4195650 0x7f0ba40b5560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:55.625 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:55.618 7f0ba8fdb700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0ba40b5b80 0x7f0ba41a2e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:55.625 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:55.618 7f0ba8fdb700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0ba40acfa0 0x7f0ba41a3440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:55.625 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:55.619 7f0ba8fdb700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ba40b6010 con 0x7f0ba40b5b80 2021-02-10T11:34:55.625 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:55.619 7f0ba8fdb700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ba41a3a60 con 0x7f0ba40acfa0 2021-02-10T11:34:55.626 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:55.619 7f0ba8fdb700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ba41a3c40 con 0x7f0ba4195650 2021-02-10T11:34:55.626 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:55.619 7f0ba259c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0ba40b5b80 0x7f0ba41a2e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:55.626 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:55.619 7f0ba259c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0ba40b5b80 0x7f0ba41a2e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:36480/0 (socket says 172.21.2.102:36480) 2021-02-10T11:34:55.626 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:55.619 7f0ba259c700 1 -- 172.21.2.102:0/606577982 learned_addr learned my addr 172.21.2.102:0/606577982 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:34:55.626 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:55.619 7f0ba2d9d700 1 --2- 172.21.2.102:0/606577982 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0ba4195650 0x7f0ba40b5560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:55.626 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:55.619 7f0ba359e700 1 --2- 172.21.2.102:0/606577982 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0ba40acfa0 0x7f0ba41a3440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:55.626 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:55.619 7f0ba259c700 1 -- 172.21.2.102:0/606577982 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0ba4195650 msgr2=0x7f0ba40b5560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:55.626 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:55.619 7f0ba259c700 1 --2- 172.21.2.102:0/606577982 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0ba4195650 0x7f0ba40b5560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:55.626 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:55.619 7f0ba259c700 1 -- 172.21.2.102:0/606577982 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0ba40acfa0 msgr2=0x7f0ba41a3440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:55.626 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:55.619 7f0ba259c700 1 --2- 172.21.2.102:0/606577982 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0ba40acfa0 0x7f0ba41a3440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:55.626 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:55.619 7f0ba259c700 1 -- 172.21.2.102:0/606577982 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ba40bcae0 con 0x7f0ba40b5b80 2021-02-10T11:34:55.626 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:55.619 7f0ba259c700 1 --2- 172.21.2.102:0/606577982 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0ba40b5b80 0x7f0ba41a2e20 secure :-1 s=READY pgs=2604 cs=0 l=1 rev1=1 rx=0x7f0b8c001be0 tx=0x7f0b8c00a900).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:55.627 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:55.620 7f0b9b7fe700 1 -- 172.21.2.102:0/606577982 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0b8c0010b0 con 0x7f0ba40b5b80 2021-02-10T11:34:55.627 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:55.620 7f0b9b7fe700 1 -- 172.21.2.102:0/606577982 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b8c004350 con 0x7f0ba40b5b80 2021-02-10T11:34:55.627 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:55.620 7f0ba8fdb700 1 -- 172.21.2.102:0/606577982 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ba40bcae0 con 0x7f0ba40b5b80 2021-02-10T11:34:55.627 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:55.620 7f0b9b7fe700 1 -- 172.21.2.102:0/606577982 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0b8c0047a0 con 0x7f0ba40b5b80 2021-02-10T11:34:55.627 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:55.620 7f0ba8fdb700 1 -- 172.21.2.102:0/606577982 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ba40bd070 con 0x7f0ba40b5b80 2021-02-10T11:34:55.627 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:55.620 7f0b9b7fe700 1 -- 172.21.2.102:0/606577982 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f0b8c004530 con 0x7f0ba40b5b80 2021-02-10T11:34:55.627 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:55.620 7f0b9b7fe700 1 --2- 172.21.2.102:0/606577982 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0b880186c0 0x7f0b8801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:55.627 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:55.620 7f0b9b7fe700 1 -- 172.21.2.102:0/606577982 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(137..137 src has 1..137) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f0b8c0323c0 con 0x7f0ba40b5b80 2021-02-10T11:34:55.627 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:55.621 7f0ba2d9d700 1 --2- 172.21.2.102:0/606577982 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0b880186c0 0x7f0b8801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:55.627 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:55.621 7f0ba2d9d700 1 --2- 172.21.2.102:0/606577982 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0b880186c0 0x7f0b8801abc0 crc :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:55.652 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:55.650 7f0ba8fdb700 1 -- 172.21.2.102:0/606577982 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ba40f0ab0 con 0x7f0ba40b5b80 2021-02-10T11:34:55.654 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:55.653 7f0b9b7fe700 1 -- 172.21.2.102:0/606577982 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f0b8c02cc20 con 0x7f0ba40b5b80 2021-02-10T11:34:55.789 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:55.787 7f0ba8fdb700 1 -- 172.21.2.102:0/606577982 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestVolumes.test_nonexistent_subvolume_group_rm"]} v 0) v1 -- 0x7f0ba40f0ab0 con 0x7f0ba40b5b80 2021-02-10T11:34:56.149 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:56.147 7f0b9b7fe700 1 -- 172.21.2.102:0/606577982 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestVolumes.test_nonexistent_subvolume_group_rm"]}]=0 v700) v1 ==== 155+0+0 (secure 0 0 0) 0x7f0b8c02cef0 con 0x7f0ba40b5b80 2021-02-10T11:34:56.158 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:56.157 7f0ba8fdb700 1 -- 172.21.2.102:0/606577982 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0b880186c0 msgr2=0x7f0b8801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:56.158 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:56.157 7f0ba8fdb700 1 --2- 172.21.2.102:0/606577982 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0b880186c0 0x7f0b8801abc0 crc :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:56.159 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:56.157 7f0ba8fdb700 1 -- 172.21.2.102:0/606577982 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0ba40b5b80 msgr2=0x7f0ba41a2e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:56.159 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:56.157 7f0ba8fdb700 1 --2- 172.21.2.102:0/606577982 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0ba40b5b80 0x7f0ba41a2e20 secure :-1 s=READY pgs=2604 cs=0 l=1 rev1=1 rx=0x7f0b8c001be0 tx=0x7f0b8c00a900).stop 2021-02-10T11:34:56.159 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:56.157 7f0ba8fdb700 1 -- 172.21.2.102:0/606577982 shutdown_connections 2021-02-10T11:34:56.159 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:56.157 7f0ba8fdb700 1 --2- 172.21.2.102:0/606577982 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0b880186c0 0x7f0b8801abc0 unknown :-1 s=CLOSED pgs=1149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:56.159 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:56.157 7f0ba8fdb700 1 --2- 172.21.2.102:0/606577982 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0ba40acfa0 0x7f0ba41a3440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:56.159 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:56.157 7f0ba8fdb700 1 --2- 172.21.2.102:0/606577982 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0ba40b5b80 0x7f0ba41a2e20 unknown :-1 s=CLOSED pgs=2604 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:56.159 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:56.157 7f0ba8fdb700 1 --2- 172.21.2.102:0/606577982 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0ba4195650 0x7f0ba40b5560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:56.160 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:56.157 7f0ba8fdb700 1 -- 172.21.2.102:0/606577982 >> 172.21.2.102:0/606577982 conn(0x7f0ba40b3040 msgr2=0x7f0ba40c3250 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:56.160 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:56.157 7f0ba8fdb700 1 -- 172.21.2.102:0/606577982 shutdown_connections 2021-02-10T11:34:56.160 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:56.157 7f0ba8fdb700 1 -- 172.21.2.102:0/606577982 wait complete. 2021-02-10T11:34:56.167 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config set mon mon_allow_pool_delete True 2021-02-10T11:34:56.600 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:56.582 7f54477d6700 1 Processor -- start 2021-02-10T11:34:56.600 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:56.582 7f54477d6700 1 -- start start 2021-02-10T11:34:56.600 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:56.583 7f54477d6700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f54400d1870 0x7f54401b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:56.601 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:56.583 7f54477d6700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f54401b0ae0 0x7f5440108940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:56.601 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:56.583 7f54477d6700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54400fe1c0 con 0x7f54400d1c70 2021-02-10T11:34:56.601 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:56.583 7f54477d6700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5440109a60 con 0x7f54401b0ae0 2021-02-10T11:34:56.601 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:56.583 7f54477d6700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5440109750 con 0x7f54400d1870 2021-02-10T11:34:56.601 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:56.583 7f5444df1700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f54401b0ae0 0x7f5440108940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:56.601 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:56.583 7f5444df1700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f54401b0ae0 0x7f5440108940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:36488/0 (socket says 172.21.2.102:36488) 2021-02-10T11:34:56.601 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:56.583 7f5445df3700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f54400d1c70 0x7f5440108f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:56944/0 (socket says 172.21.2.102:56944) 2021-02-10T11:34:56.601 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:56.583 7f5444df1700 1 -- 172.21.2.102:0/3398130856 learned_addr learned my addr 172.21.2.102:0/3398130856 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:34:56.601 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:56.583 7f5444df1700 1 -- 172.21.2.102:0/3398130856 >> v2:172.21.2.113:3300/0 conn(0x7f54400d1870 msgr2=0x7f54401b04c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:34:56.601 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:56.583 7f5444df1700 1 --2- 172.21.2.102:0/3398130856 >> v2:172.21.2.113:3300/0 conn(0x7f54400d1870 0x7f54401b04c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:56.601 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:56.583 7f5444df1700 1 -- 172.21.2.102:0/3398130856 >> v1:172.21.2.102:6789/0 conn(0x7f54400d1c70 legacy=0x7f5440108f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:56.601 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:56.583 7f5444df1700 1 -- 172.21.2.102:0/3398130856 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5440088070 con 0x7f54401b0ae0 2021-02-10T11:34:56.601 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:56.583 7f5444df1700 1 --2- 172.21.2.102:0/3398130856 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f54401b0ae0 0x7f5440108940 secure :-1 s=READY pgs=2606 cs=0 l=1 rev1=1 rx=0x7f543c006700 tx=0x7f543c009b80).ready entity=mon.0 client_cookie=4225a094600e77e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:56.602 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:56.583 7f5433fff700 1 -- 172.21.2.102:0/3398130856 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f543c00c0c0 con 0x7f54401b0ae0 2021-02-10T11:34:56.602 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:56.584 7f5433fff700 1 -- 172.21.2.102:0/3398130856 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f543c00c340 con 0x7f54401b0ae0 2021-02-10T11:34:56.602 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:56.584 7f5433fff700 1 -- 172.21.2.102:0/3398130856 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f543c00c7a0 con 0x7f54401b0ae0 2021-02-10T11:34:56.602 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:56.584 7f54477d6700 1 -- 172.21.2.102:0/3398130856 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f54401b0ae0 msgr2=0x7f5440108940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:56.602 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:56.584 7f54477d6700 1 --2- 172.21.2.102:0/3398130856 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f54401b0ae0 0x7f5440108940 secure :-1 s=READY pgs=2606 cs=0 l=1 rev1=1 rx=0x7f543c006700 tx=0x7f543c009b80).stop 2021-02-10T11:34:56.602 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:56.584 7f54477d6700 1 -- 172.21.2.102:0/3398130856 shutdown_connections 2021-02-10T11:34:56.602 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:56.584 7f54477d6700 1 --2- 172.21.2.102:0/3398130856 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f54401b0ae0 0x7f5440108940 unknown :-1 s=CLOSED pgs=2606 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:56.602 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:56.584 7f54477d6700 1 --2- 172.21.2.102:0/3398130856 >> v2:172.21.2.113:3300/0 conn(0x7f54400d1870 0x7f54401b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:56.602 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:56.584 7f54477d6700 1 -- 172.21.2.102:0/3398130856 >> 172.21.2.102:0/3398130856 conn(0x7f54400ce420 msgr2=0x7f54400d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:56.602 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:56.584 7f54477d6700 1 -- 172.21.2.102:0/3398130856 shutdown_connections 2021-02-10T11:34:56.602 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:56.584 7f54477d6700 1 -- 172.21.2.102:0/3398130856 wait complete. 2021-02-10T11:34:56.604 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:56.599 7f54477d6700 1 Processor -- start 2021-02-10T11:34:56.604 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:56.599 7f54477d6700 1 -- start start 2021-02-10T11:34:56.604 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:56.599 7f54477d6700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f54400c3820 0x7f5440084cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:56.604 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:56.599 7f54477d6700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f54400a0290 0x7f54400852d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:56.604 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:56.599 7f54477d6700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5440086270 0x7f5440086670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:56.604 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:56.599 7f54477d6700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54400c0700 con 0x7f54400a0290 2021-02-10T11:34:56.604 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:56.599 7f54477d6700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54400858f0 con 0x7f5440086270 2021-02-10T11:34:56.604 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:56.599 7f54477d6700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5440085b10 con 0x7f54400c3820 2021-02-10T11:34:56.604 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:56.600 7f5444df1700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f54400a0290 0x7f54400852d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:56.604 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:56.600 7f5444df1700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f54400a0290 0x7f54400852d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:36492/0 (socket says 172.21.2.102:36492) 2021-02-10T11:34:56.604 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:56.600 7f5444df1700 1 -- 172.21.2.102:0/3812067224 learned_addr learned my addr 172.21.2.102:0/3812067224 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:34:56.604 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:56.600 7f5444df1700 1 -- 172.21.2.102:0/3812067224 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f54400c3820 msgr2=0x7f5440084cb0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:34:56.604 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:56.600 7f5444df1700 1 --2- 172.21.2.102:0/3812067224 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f54400c3820 0x7f5440084cb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:56.605 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:56.600 7f5444df1700 1 -- 172.21.2.102:0/3812067224 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5440086270 msgr2=0x7f5440086670 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:34:56.605 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:56.600 7f5444df1700 1 --2- 172.21.2.102:0/3812067224 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5440086270 0x7f5440086670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:56.605 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:56.600 7f5444df1700 1 -- 172.21.2.102:0/3812067224 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5440086c90 con 0x7f54400a0290 2021-02-10T11:34:56.605 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:56.600 7f5444df1700 1 --2- 172.21.2.102:0/3812067224 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f54400a0290 0x7f54400852d0 secure :-1 s=READY pgs=2607 cs=0 l=1 rev1=1 rx=0x7f543c00b650 tx=0x7f543c00b9f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:56.605 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:56.600 7f5431ffb700 1 -- 172.21.2.102:0/3812067224 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f543c00bde0 con 0x7f54400a0290 2021-02-10T11:34:56.605 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:56.600 7f5431ffb700 1 -- 172.21.2.102:0/3812067224 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f543c000c90 con 0x7f54400a0290 2021-02-10T11:34:56.605 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:56.600 7f54477d6700 1 -- 172.21.2.102:0/3812067224 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5440086c90 con 0x7f54400a0290 2021-02-10T11:34:56.605 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:56.600 7f5431ffb700 1 -- 172.21.2.102:0/3812067224 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f543c0022b0 con 0x7f54400a0290 2021-02-10T11:34:56.605 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:56.600 7f54477d6700 1 -- 172.21.2.102:0/3812067224 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54401a2f00 con 0x7f54400a0290 2021-02-10T11:34:56.605 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:56.601 7f5431ffb700 1 -- 172.21.2.102:0/3812067224 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f543c009f90 con 0x7f54400a0290 2021-02-10T11:34:56.605 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:56.601 7f5431ffb700 1 --2- 172.21.2.102:0/3812067224 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f54200186c0 0x7f542001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:56.605 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:56.601 7f5431ffb700 1 -- 172.21.2.102:0/3812067224 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(137..137 src has 1..137) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f543c032570 con 0x7f54400a0290 2021-02-10T11:34:56.606 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:56.601 7f54455f2700 1 --2- 172.21.2.102:0/3812067224 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f54200186c0 0x7f542001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:56.606 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:56.602 7f54455f2700 1 --2- 172.21.2.102:0/3812067224 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f54200186c0 0x7f542001abc0 crc :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:56.634 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:56.632 7f54477d6700 1 -- 172.21.2.102:0/3812067224 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5440082a70 con 0x7f54400a0290 2021-02-10T11:34:56.637 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:56.635 7f5431ffb700 1 -- 172.21.2.102:0/3812067224 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f543c0009b0 con 0x7f54400a0290 2021-02-10T11:34:56.786 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:56.784 7f54477d6700 1 -- 172.21.2.102:0/3812067224 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f5440082a70 con 0x7f54400a0290 2021-02-10T11:34:56.789 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:56.788 7f5431ffb700 1 -- 172.21.2.102:0/3812067224 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v36)=0 v36) v1 ==== 122+0+0 (secure 0 0 0) 0x7f543c009f90 con 0x7f54400a0290 2021-02-10T11:34:56.794 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:56.792 7f54477d6700 1 -- 172.21.2.102:0/3812067224 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f54200186c0 msgr2=0x7f542001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:56.795 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:56.792 7f54477d6700 1 --2- 172.21.2.102:0/3812067224 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f54200186c0 0x7f542001abc0 crc :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:56.795 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:56.792 7f54477d6700 1 -- 172.21.2.102:0/3812067224 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f54400a0290 msgr2=0x7f54400852d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:56.795 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:56.792 7f54477d6700 1 --2- 172.21.2.102:0/3812067224 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f54400a0290 0x7f54400852d0 secure :-1 s=READY pgs=2607 cs=0 l=1 rev1=1 rx=0x7f543c00b650 tx=0x7f543c00b9f0).stop 2021-02-10T11:34:56.795 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:56.793 7f54477d6700 1 -- 172.21.2.102:0/3812067224 shutdown_connections 2021-02-10T11:34:56.795 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:56.793 7f54477d6700 1 --2- 172.21.2.102:0/3812067224 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f54200186c0 0x7f542001abc0 unknown :-1 s=CLOSED pgs=1150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:56.795 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:56.793 7f54477d6700 1 --2- 172.21.2.102:0/3812067224 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5440086270 0x7f5440086670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:56.795 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:56.793 7f54477d6700 1 --2- 172.21.2.102:0/3812067224 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f54400a0290 0x7f54400852d0 unknown :-1 s=CLOSED pgs=2607 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:56.795 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:56.793 7f54477d6700 1 --2- 172.21.2.102:0/3812067224 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f54400c3820 0x7f5440084cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:56.795 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:56.793 7f54477d6700 1 -- 172.21.2.102:0/3812067224 >> 172.21.2.102:0/3812067224 conn(0x7f54400872c0 msgr2=0x7f5440095700 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:56.795 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:56.793 7f54477d6700 1 -- 172.21.2.102:0/3812067224 shutdown_connections 2021-02-10T11:34:56.795 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:56.793 7f54477d6700 1 -- 172.21.2.102:0/3812067224 wait complete. 2021-02-10T11:34:56.803 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2021-02-10T11:34:56.803 DEBUG:teuthology.orchestra.run.gibba002:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:34:57.150 DEBUG:teuthology.orchestra.run.gibba013:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:34:57.496 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2021-02-10T11:34:57.497 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:34:57.497 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:34:57.497 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:34:57.833 INFO:teuthology.orchestra.run.gibba002.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.0’: No such file or directory 2021-02-10T11:34:57.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:34:57.833 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.0 2021-02-10T11:34:57.834 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:34:57.834 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:34:57.834 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:34:58.169 INFO:teuthology.orchestra.run.gibba002.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.1’: No such file or directory 2021-02-10T11:34:58.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:34:58.169 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2021-02-10T11:34:58.170 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:34:58.170 DEBUG:teuthology.orchestra.run.gibba013:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:34:58.170 DEBUG:teuthology.orchestra.run.gibba013:> ' -- /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:34:58.504 INFO:teuthology.orchestra.run.gibba013.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.2’: No such file or directory 2021-02-10T11:34:58.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:34:58.505 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2021-02-10T11:34:58.505 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:34:58.505 DEBUG:teuthology.orchestra.run.gibba013:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:34:58.505 DEBUG:teuthology.orchestra.run.gibba013:> ' -- /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:34:58.839 INFO:teuthology.orchestra.run.gibba013.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.3’: No such file or directory 2021-02-10T11:34:58.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:34:58.839 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.3 2021-02-10T11:34:58.840 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json-pretty 2021-02-10T11:34:59.270 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:59.251 7f3897086700 1 Processor -- start 2021-02-10T11:34:59.270 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:59.251 7f3897086700 1 -- start start 2021-02-10T11:34:59.270 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:59.252 7f3897086700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f38900d1870 0x7f38901b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:59.270 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:59.252 7f3897086700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f38901b0a10 0x7f38900cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:59.270 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:59.252 7f3897086700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f38900d1c70 0x7f38900cf950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:59.270 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:59.252 7f3897086700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38900fe230 con 0x7f38900d1c70 2021-02-10T11:34:59.270 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:59.252 7f3897086700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38900e5ed0 con 0x7f38901b0a10 2021-02-10T11:34:59.270 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:59.252 7f3897086700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38900f0b10 con 0x7f38900d1870 2021-02-10T11:34:59.270 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:59.252 7f38956a3700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f38900d1c70 0x7f38900cf950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:59.271 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:59.252 7f38956a3700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f38900d1c70 0x7f38900cf950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:36498/0 (socket says 172.21.2.102:36498) 2021-02-10T11:34:59.271 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:59.252 7f38956a3700 1 -- 172.21.2.102:0/2026797545 learned_addr learned my addr 172.21.2.102:0/2026797545 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:34:59.271 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:59.252 7f38956a3700 1 -- 172.21.2.102:0/2026797545 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f38900d1870 msgr2=0x7f38901b03f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:34:59.271 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:59.252 7f38956a3700 1 --2- 172.21.2.102:0/2026797545 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f38900d1870 0x7f38901b03f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:59.271 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:59.252 7f38956a3700 1 -- 172.21.2.102:0/2026797545 >> v2:172.21.2.113:3300/0 conn(0x7f38901b0a10 msgr2=0x7f38900cf330 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:34:59.271 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:59.252 7f38956a3700 1 --2- 172.21.2.102:0/2026797545 >> v2:172.21.2.113:3300/0 conn(0x7f38901b0a10 0x7f38900cf330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:59.271 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:59.252 7f38956a3700 1 -- 172.21.2.102:0/2026797545 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38900cff70 con 0x7f38900d1c70 2021-02-10T11:34:59.271 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:59.252 7f38956a3700 1 --2- 172.21.2.102:0/2026797545 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f38900d1c70 0x7f38900cf950 secure :-1 s=READY pgs=2608 cs=0 l=1 rev1=1 rx=0x7f3888006f00 tx=0x7f3888009f10).ready entity=mon.0 client_cookie=4935653ffc405631 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:59.271 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:59.253 7f388f7fe700 1 -- 172.21.2.102:0/2026797545 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f388800b3b0 con 0x7f38900d1c70 2021-02-10T11:34:59.271 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:59.253 7f388f7fe700 1 -- 172.21.2.102:0/2026797545 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f388800b730 con 0x7f38900d1c70 2021-02-10T11:34:59.271 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:59.253 7f388f7fe700 1 -- 172.21.2.102:0/2026797545 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f388800bc50 con 0x7f38900d1c70 2021-02-10T11:34:59.271 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:59.253 7f3897086700 1 -- 172.21.2.102:0/2026797545 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f38900d1c70 msgr2=0x7f38900cf950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:59.271 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:59.253 7f3897086700 1 --2- 172.21.2.102:0/2026797545 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f38900d1c70 0x7f38900cf950 secure :-1 s=READY pgs=2608 cs=0 l=1 rev1=1 rx=0x7f3888006f00 tx=0x7f3888009f10).stop 2021-02-10T11:34:59.272 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:59.253 7f3897086700 1 -- 172.21.2.102:0/2026797545 shutdown_connections 2021-02-10T11:34:59.272 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:59.253 7f3897086700 1 --2- 172.21.2.102:0/2026797545 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f38900d1c70 0x7f38900cf950 unknown :-1 s=CLOSED pgs=2608 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:59.272 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:59.253 7f3897086700 1 --2- 172.21.2.102:0/2026797545 >> v2:172.21.2.113:3300/0 conn(0x7f38901b0a10 0x7f38900cf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:59.272 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:59.253 7f3897086700 1 --2- 172.21.2.102:0/2026797545 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f38900d1870 0x7f38901b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:59.272 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:59.253 7f3897086700 1 -- 172.21.2.102:0/2026797545 >> 172.21.2.102:0/2026797545 conn(0x7f38900ce450 msgr2=0x7f38900d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:59.272 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:59.253 7f3897086700 1 -- 172.21.2.102:0/2026797545 shutdown_connections 2021-02-10T11:34:59.272 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:59.253 7f3897086700 1 -- 172.21.2.102:0/2026797545 wait complete. 2021-02-10T11:34:59.273 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:59.268 7f3897086700 1 Processor -- start 2021-02-10T11:34:59.274 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:59.268 7f3897086700 1 -- start start 2021-02-10T11:34:59.274 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:59.269 7f3897086700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f38901b7cf0 0x7f389019cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:59.274 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:59.269 7f3897086700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f38901b80f0 0x7f38901b84f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:59.274 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:59.269 7f3897086700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f389019da20 0x7f389019de20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:59.274 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:59.269 7f3897086700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38900a77c0 con 0x7f389019da20 2021-02-10T11:34:59.274 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:59.269 7f3897086700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f389019d4a0 con 0x7f38901b7cf0 2021-02-10T11:34:59.274 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:59.269 7f3897086700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38900954f0 con 0x7f38901b80f0 2021-02-10T11:34:59.274 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:59.269 7f38956a3700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f389019da20 0x7f389019de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:59.274 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:59.269 7f38956a3700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f389019da20 0x7f389019de20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:36504/0 (socket says 172.21.2.102:36504) 2021-02-10T11:34:59.274 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:59.269 7f38956a3700 1 -- 172.21.2.102:0/1604194746 learned_addr learned my addr 172.21.2.102:0/1604194746 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:34:59.274 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:59.269 7f38956a3700 1 -- 172.21.2.102:0/1604194746 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f38901b80f0 msgr2=0x7f38901b84f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:59.275 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:59.269 7f38956a3700 1 --2- 172.21.2.102:0/1604194746 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f38901b80f0 0x7f38901b84f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:59.275 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:59.269 7f38956a3700 1 -- 172.21.2.102:0/1604194746 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f38901b7cf0 msgr2=0x7f389019cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:59.275 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:59.269 7f38956a3700 1 --2- 172.21.2.102:0/1604194746 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f38901b7cf0 0x7f389019cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:59.275 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:59.269 7f38956a3700 1 -- 172.21.2.102:0/1604194746 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38901b8d60 con 0x7f389019da20 2021-02-10T11:34:59.275 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:59.270 7f38956a3700 1 --2- 172.21.2.102:0/1604194746 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f389019da20 0x7f389019de20 secure :-1 s=READY pgs=2609 cs=0 l=1 rev1=1 rx=0x7f388800b940 tx=0x7f388800bce0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:59.275 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:59.270 7f388d7fa700 1 -- 172.21.2.102:0/1604194746 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3888003030 con 0x7f389019da20 2021-02-10T11:34:59.275 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:59.270 7f388d7fa700 1 -- 172.21.2.102:0/1604194746 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3888000cc0 con 0x7f389019da20 2021-02-10T11:34:59.275 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:59.270 7f388d7fa700 1 -- 172.21.2.102:0/1604194746 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3888001460 con 0x7f389019da20 2021-02-10T11:34:59.275 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:59.270 7f3897086700 1 -- 172.21.2.102:0/1604194746 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38901b8d60 con 0x7f389019da20 2021-02-10T11:34:59.275 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:59.270 7f3897086700 1 -- 172.21.2.102:0/1604194746 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38901b8d60 con 0x7f389019da20 2021-02-10T11:34:59.275 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:59.271 7f388d7fa700 1 -- 172.21.2.102:0/1604194746 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f3888001270 con 0x7f389019da20 2021-02-10T11:34:59.275 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:59.271 7f388d7fa700 1 --2- 172.21.2.102:0/1604194746 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f38740186c0 0x7f387401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:59.275 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:59.271 7f388d7fa700 1 -- 172.21.2.102:0/1604194746 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(137..137 src has 1..137) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f3888032920 con 0x7f389019da20 2021-02-10T11:34:59.276 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:59.271 7f3894ea2700 1 --2- 172.21.2.102:0/1604194746 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f38740186c0 0x7f387401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:59.276 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:59.271 7f3894ea2700 1 --2- 172.21.2.102:0/1604194746 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f38740186c0 0x7f387401abc0 crc :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:59.303 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:59.301 7f3897086700 1 -- 172.21.2.102:0/1604194746 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38900ebc00 con 0x7f389019da20 2021-02-10T11:34:59.306 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:59.304 7f388d7fa700 1 -- 172.21.2.102:0/1604194746 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f388801f360 con 0x7f389019da20 2021-02-10T11:34:59.430 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:59.428 7f3897086700 1 -- 172.21.2.102:0/1604194746 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7f38900f0b10 con 0x7f389019da20 2021-02-10T11:34:59.430 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:59.428 7f388d7fa700 1 -- 172.21.2.102:0/1604194746 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v137) v1 ==== 81+0+30374 (secure 0 0 0) 0x7f388801f360 con 0x7f389019da20 2021-02-10T11:34:59.431 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:34:59.431 INFO:teuthology.orchestra.run.gibba002.stdout:{ 2021-02-10T11:34:59.431 INFO:teuthology.orchestra.run.gibba002.stdout: "epoch": 137, 2021-02-10T11:34:59.431 INFO:teuthology.orchestra.run.gibba002.stdout: "fsid": "68db30ea-dbce-4aef-a6df-f01c6fea0513", 2021-02-10T11:34:59.431 INFO:teuthology.orchestra.run.gibba002.stdout: "created": "2021-02-10 11:03:06.331557", 2021-02-10T11:34:59.431 INFO:teuthology.orchestra.run.gibba002.stdout: "modified": "2021-02-10 11:34:24.687007", 2021-02-10T11:34:59.431 INFO:teuthology.orchestra.run.gibba002.stdout: "last_up_change": "2021-02-10 11:03:32.690562", 2021-02-10T11:34:59.431 INFO:teuthology.orchestra.run.gibba002.stdout: "last_in_change": "2021-02-10 11:03:32.690562", 2021-02-10T11:34:59.432 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2021-02-10T11:34:59.432 INFO:teuthology.orchestra.run.gibba002.stdout: "flags_num": 5799936, 2021-02-10T11:34:59.432 INFO:teuthology.orchestra.run.gibba002.stdout: "flags_set": [ 2021-02-10T11:34:59.432 INFO:teuthology.orchestra.run.gibba002.stdout: "pglog_hardlimit", 2021-02-10T11:34:59.432 INFO:teuthology.orchestra.run.gibba002.stdout: "purged_snapdirs", 2021-02-10T11:34:59.432 INFO:teuthology.orchestra.run.gibba002.stdout: "recovery_deletes", 2021-02-10T11:34:59.432 INFO:teuthology.orchestra.run.gibba002.stdout: "sortbitwise" 2021-02-10T11:34:59.432 INFO:teuthology.orchestra.run.gibba002.stdout: ], 2021-02-10T11:34:59.432 INFO:teuthology.orchestra.run.gibba002.stdout: "crush_version": 4, 2021-02-10T11:34:59.432 INFO:teuthology.orchestra.run.gibba002.stdout: "full_ratio": 0.94999998807907104, 2021-02-10T11:34:59.432 INFO:teuthology.orchestra.run.gibba002.stdout: "backfillfull_ratio": 0.89999997615814209, 2021-02-10T11:34:59.432 INFO:teuthology.orchestra.run.gibba002.stdout: "nearfull_ratio": 0.85000002384185791, 2021-02-10T11:34:59.433 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_snapshot": "", 2021-02-10T11:34:59.433 INFO:teuthology.orchestra.run.gibba002.stdout: "pool_max": 29, 2021-02-10T11:34:59.433 INFO:teuthology.orchestra.run.gibba002.stdout: "max_osd": 8, 2021-02-10T11:34:59.433 INFO:teuthology.orchestra.run.gibba002.stdout: "require_min_compat_client": "jewel", 2021-02-10T11:34:59.433 INFO:teuthology.orchestra.run.gibba002.stdout: "min_compat_client": "jewel", 2021-02-10T11:34:59.433 INFO:teuthology.orchestra.run.gibba002.stdout: "require_osd_release": "nautilus", 2021-02-10T11:34:59.433 INFO:teuthology.orchestra.run.gibba002.stdout: "pools": [ 2021-02-10T11:34:59.433 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:59.433 INFO:teuthology.orchestra.run.gibba002.stdout: "pool": 1, 2021-02-10T11:34:59.433 INFO:teuthology.orchestra.run.gibba002.stdout: "pool_name": "rbd", 2021-02-10T11:34:59.433 INFO:teuthology.orchestra.run.gibba002.stdout: "create_time": "2021-02-10 11:03:37.322519", 2021-02-10T11:34:59.433 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 1, 2021-02-10T11:34:59.433 INFO:teuthology.orchestra.run.gibba002.stdout: "flags_names": "hashpspool", 2021-02-10T11:34:59.434 INFO:teuthology.orchestra.run.gibba002.stdout: "type": 1, 2021-02-10T11:34:59.434 INFO:teuthology.orchestra.run.gibba002.stdout: "size": 2, 2021-02-10T11:34:59.434 INFO:teuthology.orchestra.run.gibba002.stdout: "min_size": 1, 2021-02-10T11:34:59.434 INFO:teuthology.orchestra.run.gibba002.stdout: "crush_rule": 0, 2021-02-10T11:34:59.434 INFO:teuthology.orchestra.run.gibba002.stdout: "object_hash": 2, 2021-02-10T11:34:59.434 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:34:59.434 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num": 8, 2021-02-10T11:34:59.434 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_placement_num": 8, 2021-02-10T11:34:59.434 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_placement_num_target": 8, 2021-02-10T11:34:59.434 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num_target": 8, 2021-02-10T11:34:59.434 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num_pending": 8, 2021-02-10T11:34:59.434 INFO:teuthology.orchestra.run.gibba002.stdout: "last_pg_merge_meta": { 2021-02-10T11:34:59.434 INFO:teuthology.orchestra.run.gibba002.stdout: "source_pgid": "0.0", 2021-02-10T11:34:59.435 INFO:teuthology.orchestra.run.gibba002.stdout: "ready_epoch": 0, 2021-02-10T11:34:59.435 INFO:teuthology.orchestra.run.gibba002.stdout: "last_epoch_started": 0, 2021-02-10T11:34:59.435 INFO:teuthology.orchestra.run.gibba002.stdout: "last_epoch_clean": 0, 2021-02-10T11:34:59.435 INFO:teuthology.orchestra.run.gibba002.stdout: "source_version": "0'0", 2021-02-10T11:34:59.435 INFO:teuthology.orchestra.run.gibba002.stdout: "target_version": "0'0" 2021-02-10T11:34:59.435 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:59.435 INFO:teuthology.orchestra.run.gibba002.stdout: "last_change": "16", 2021-02-10T11:34:59.435 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend": "0", 2021-02-10T11:34:59.435 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:34:59.435 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:34:59.435 INFO:teuthology.orchestra.run.gibba002.stdout: "auid": 0, 2021-02-10T11:34:59.435 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:34:59.435 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_seq": 0, 2021-02-10T11:34:59.436 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_epoch": 0, 2021-02-10T11:34:59.436 INFO:teuthology.orchestra.run.gibba002.stdout: "pool_snaps": [], 2021-02-10T11:34:59.436 INFO:teuthology.orchestra.run.gibba002.stdout: "removed_snaps": "[]", 2021-02-10T11:34:59.436 INFO:teuthology.orchestra.run.gibba002.stdout: "quota_max_bytes": 0, 2021-02-10T11:34:59.436 INFO:teuthology.orchestra.run.gibba002.stdout: "quota_max_objects": 0, 2021-02-10T11:34:59.436 INFO:teuthology.orchestra.run.gibba002.stdout: "tiers": [], 2021-02-10T11:34:59.436 INFO:teuthology.orchestra.run.gibba002.stdout: "tier_of": -1, 2021-02-10T11:34:59.436 INFO:teuthology.orchestra.run.gibba002.stdout: "read_tier": -1, 2021-02-10T11:34:59.436 INFO:teuthology.orchestra.run.gibba002.stdout: "write_tier": -1, 2021-02-10T11:34:59.436 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_mode": "none", 2021-02-10T11:34:59.436 INFO:teuthology.orchestra.run.gibba002.stdout: "target_max_bytes": 0, 2021-02-10T11:34:59.436 INFO:teuthology.orchestra.run.gibba002.stdout: "target_max_objects": 0, 2021-02-10T11:34:59.436 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:34:59.437 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:34:59.437 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:34:59.437 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_min_flush_age": 0, 2021-02-10T11:34:59.437 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_min_evict_age": 0, 2021-02-10T11:34:59.437 INFO:teuthology.orchestra.run.gibba002.stdout: "erasure_code_profile": "", 2021-02-10T11:34:59.437 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_params": { 2021-02-10T11:34:59.437 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "none" 2021-02-10T11:34:59.437 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:59.437 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_period": 0, 2021-02-10T11:34:59.437 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_count": 0, 2021-02-10T11:34:59.437 INFO:teuthology.orchestra.run.gibba002.stdout: "use_gmt_hitset": true, 2021-02-10T11:34:59.437 INFO:teuthology.orchestra.run.gibba002.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:34:59.437 INFO:teuthology.orchestra.run.gibba002.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:34:59.437 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:34:59.438 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:34:59.438 INFO:teuthology.orchestra.run.gibba002.stdout: "grade_table": [], 2021-02-10T11:34:59.438 INFO:teuthology.orchestra.run.gibba002.stdout: "stripe_width": 0, 2021-02-10T11:34:59.438 INFO:teuthology.orchestra.run.gibba002.stdout: "expected_num_objects": 0, 2021-02-10T11:34:59.438 INFO:teuthology.orchestra.run.gibba002.stdout: "fast_read": false, 2021-02-10T11:34:59.438 INFO:teuthology.orchestra.run.gibba002.stdout: "options": {}, 2021-02-10T11:34:59.438 INFO:teuthology.orchestra.run.gibba002.stdout: "application_metadata": { 2021-02-10T11:34:59.438 INFO:teuthology.orchestra.run.gibba002.stdout: "rbd": {} 2021-02-10T11:34:59.438 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:34:59.438 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:59.438 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:59.438 INFO:teuthology.orchestra.run.gibba002.stdout: "pool": 28, 2021-02-10T11:34:59.438 INFO:teuthology.orchestra.run.gibba002.stdout: "pool_name": "cephfs_metadata", 2021-02-10T11:34:59.439 INFO:teuthology.orchestra.run.gibba002.stdout: "create_time": "2021-02-10 11:34:21.649291", 2021-02-10T11:34:59.439 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 1, 2021-02-10T11:34:59.439 INFO:teuthology.orchestra.run.gibba002.stdout: "flags_names": "hashpspool", 2021-02-10T11:34:59.439 INFO:teuthology.orchestra.run.gibba002.stdout: "type": 1, 2021-02-10T11:34:59.439 INFO:teuthology.orchestra.run.gibba002.stdout: "size": 2, 2021-02-10T11:34:59.439 INFO:teuthology.orchestra.run.gibba002.stdout: "min_size": 1, 2021-02-10T11:34:59.439 INFO:teuthology.orchestra.run.gibba002.stdout: "crush_rule": 0, 2021-02-10T11:34:59.439 INFO:teuthology.orchestra.run.gibba002.stdout: "object_hash": 2, 2021-02-10T11:34:59.439 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:34:59.439 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num": 8, 2021-02-10T11:34:59.439 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_placement_num": 8, 2021-02-10T11:34:59.439 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_placement_num_target": 8, 2021-02-10T11:34:59.439 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num_target": 8, 2021-02-10T11:34:59.440 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num_pending": 8, 2021-02-10T11:34:59.440 INFO:teuthology.orchestra.run.gibba002.stdout: "last_pg_merge_meta": { 2021-02-10T11:34:59.440 INFO:teuthology.orchestra.run.gibba002.stdout: "source_pgid": "0.0", 2021-02-10T11:34:59.440 INFO:teuthology.orchestra.run.gibba002.stdout: "ready_epoch": 0, 2021-02-10T11:34:59.440 INFO:teuthology.orchestra.run.gibba002.stdout: "last_epoch_started": 0, 2021-02-10T11:34:59.440 INFO:teuthology.orchestra.run.gibba002.stdout: "last_epoch_clean": 0, 2021-02-10T11:34:59.440 INFO:teuthology.orchestra.run.gibba002.stdout: "source_version": "0'0", 2021-02-10T11:34:59.440 INFO:teuthology.orchestra.run.gibba002.stdout: "target_version": "0'0" 2021-02-10T11:34:59.440 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:59.440 INFO:teuthology.orchestra.run.gibba002.stdout: "last_change": "136", 2021-02-10T11:34:59.440 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend": "0", 2021-02-10T11:34:59.440 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:34:59.440 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:34:59.441 INFO:teuthology.orchestra.run.gibba002.stdout: "auid": 0, 2021-02-10T11:34:59.441 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:34:59.441 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_seq": 0, 2021-02-10T11:34:59.441 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_epoch": 0, 2021-02-10T11:34:59.441 INFO:teuthology.orchestra.run.gibba002.stdout: "pool_snaps": [], 2021-02-10T11:34:59.441 INFO:teuthology.orchestra.run.gibba002.stdout: "removed_snaps": "[]", 2021-02-10T11:34:59.441 INFO:teuthology.orchestra.run.gibba002.stdout: "quota_max_bytes": 0, 2021-02-10T11:34:59.441 INFO:teuthology.orchestra.run.gibba002.stdout: "quota_max_objects": 0, 2021-02-10T11:34:59.441 INFO:teuthology.orchestra.run.gibba002.stdout: "tiers": [], 2021-02-10T11:34:59.441 INFO:teuthology.orchestra.run.gibba002.stdout: "tier_of": -1, 2021-02-10T11:34:59.441 INFO:teuthology.orchestra.run.gibba002.stdout: "read_tier": -1, 2021-02-10T11:34:59.441 INFO:teuthology.orchestra.run.gibba002.stdout: "write_tier": -1, 2021-02-10T11:34:59.441 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_mode": "none", 2021-02-10T11:34:59.442 INFO:teuthology.orchestra.run.gibba002.stdout: "target_max_bytes": 0, 2021-02-10T11:34:59.442 INFO:teuthology.orchestra.run.gibba002.stdout: "target_max_objects": 0, 2021-02-10T11:34:59.442 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:34:59.442 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:34:59.442 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:34:59.442 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_min_flush_age": 0, 2021-02-10T11:34:59.442 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_min_evict_age": 0, 2021-02-10T11:34:59.442 INFO:teuthology.orchestra.run.gibba002.stdout: "erasure_code_profile": "", 2021-02-10T11:34:59.442 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_params": { 2021-02-10T11:34:59.442 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "none" 2021-02-10T11:34:59.442 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:59.442 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_period": 0, 2021-02-10T11:34:59.442 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_count": 0, 2021-02-10T11:34:59.443 INFO:teuthology.orchestra.run.gibba002.stdout: "use_gmt_hitset": true, 2021-02-10T11:34:59.443 INFO:teuthology.orchestra.run.gibba002.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:34:59.443 INFO:teuthology.orchestra.run.gibba002.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:34:59.443 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:34:59.443 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:34:59.443 INFO:teuthology.orchestra.run.gibba002.stdout: "grade_table": [], 2021-02-10T11:34:59.443 INFO:teuthology.orchestra.run.gibba002.stdout: "stripe_width": 0, 2021-02-10T11:34:59.443 INFO:teuthology.orchestra.run.gibba002.stdout: "expected_num_objects": 0, 2021-02-10T11:34:59.443 INFO:teuthology.orchestra.run.gibba002.stdout: "fast_read": false, 2021-02-10T11:34:59.443 INFO:teuthology.orchestra.run.gibba002.stdout: "options": { 2021-02-10T11:34:59.443 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_autoscale_bias": 4, 2021-02-10T11:34:59.443 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num_min": 16, 2021-02-10T11:34:59.443 INFO:teuthology.orchestra.run.gibba002.stdout: "recovery_priority": 5 2021-02-10T11:34:59.444 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:59.444 INFO:teuthology.orchestra.run.gibba002.stdout: "application_metadata": { 2021-02-10T11:34:59.444 INFO:teuthology.orchestra.run.gibba002.stdout: "cephfs": { 2021-02-10T11:34:59.444 INFO:teuthology.orchestra.run.gibba002.stdout: "metadata": "cephfs" 2021-02-10T11:34:59.444 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:34:59.444 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:34:59.444 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:59.444 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:59.444 INFO:teuthology.orchestra.run.gibba002.stdout: "pool": 29, 2021-02-10T11:34:59.444 INFO:teuthology.orchestra.run.gibba002.stdout: "pool_name": "cephfs_data", 2021-02-10T11:34:59.444 INFO:teuthology.orchestra.run.gibba002.stdout: "create_time": "2021-02-10 11:34:22.675483", 2021-02-10T11:34:59.444 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 1, 2021-02-10T11:34:59.444 INFO:teuthology.orchestra.run.gibba002.stdout: "flags_names": "hashpspool", 2021-02-10T11:34:59.445 INFO:teuthology.orchestra.run.gibba002.stdout: "type": 1, 2021-02-10T11:34:59.445 INFO:teuthology.orchestra.run.gibba002.stdout: "size": 2, 2021-02-10T11:34:59.445 INFO:teuthology.orchestra.run.gibba002.stdout: "min_size": 1, 2021-02-10T11:34:59.445 INFO:teuthology.orchestra.run.gibba002.stdout: "crush_rule": 0, 2021-02-10T11:34:59.445 INFO:teuthology.orchestra.run.gibba002.stdout: "object_hash": 2, 2021-02-10T11:34:59.445 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:34:59.445 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num": 8, 2021-02-10T11:34:59.445 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_placement_num": 8, 2021-02-10T11:34:59.445 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_placement_num_target": 8, 2021-02-10T11:34:59.445 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num_target": 8, 2021-02-10T11:34:59.445 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num_pending": 8, 2021-02-10T11:34:59.445 INFO:teuthology.orchestra.run.gibba002.stdout: "last_pg_merge_meta": { 2021-02-10T11:34:59.445 INFO:teuthology.orchestra.run.gibba002.stdout: "source_pgid": "0.0", 2021-02-10T11:34:59.446 INFO:teuthology.orchestra.run.gibba002.stdout: "ready_epoch": 0, 2021-02-10T11:34:59.446 INFO:teuthology.orchestra.run.gibba002.stdout: "last_epoch_started": 0, 2021-02-10T11:34:59.446 INFO:teuthology.orchestra.run.gibba002.stdout: "last_epoch_clean": 0, 2021-02-10T11:34:59.446 INFO:teuthology.orchestra.run.gibba002.stdout: "source_version": "0'0", 2021-02-10T11:34:59.446 INFO:teuthology.orchestra.run.gibba002.stdout: "target_version": "0'0" 2021-02-10T11:34:59.446 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:59.446 INFO:teuthology.orchestra.run.gibba002.stdout: "last_change": "137", 2021-02-10T11:34:59.446 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend": "0", 2021-02-10T11:34:59.446 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:34:59.446 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:34:59.446 INFO:teuthology.orchestra.run.gibba002.stdout: "auid": 0, 2021-02-10T11:34:59.446 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:34:59.446 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_seq": 0, 2021-02-10T11:34:59.447 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_epoch": 0, 2021-02-10T11:34:59.447 INFO:teuthology.orchestra.run.gibba002.stdout: "pool_snaps": [], 2021-02-10T11:34:59.447 INFO:teuthology.orchestra.run.gibba002.stdout: "removed_snaps": "[]", 2021-02-10T11:34:59.447 INFO:teuthology.orchestra.run.gibba002.stdout: "quota_max_bytes": 0, 2021-02-10T11:34:59.447 INFO:teuthology.orchestra.run.gibba002.stdout: "quota_max_objects": 0, 2021-02-10T11:34:59.447 INFO:teuthology.orchestra.run.gibba002.stdout: "tiers": [], 2021-02-10T11:34:59.447 INFO:teuthology.orchestra.run.gibba002.stdout: "tier_of": -1, 2021-02-10T11:34:59.447 INFO:teuthology.orchestra.run.gibba002.stdout: "read_tier": -1, 2021-02-10T11:34:59.447 INFO:teuthology.orchestra.run.gibba002.stdout: "write_tier": -1, 2021-02-10T11:34:59.447 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_mode": "none", 2021-02-10T11:34:59.447 INFO:teuthology.orchestra.run.gibba002.stdout: "target_max_bytes": 0, 2021-02-10T11:34:59.447 INFO:teuthology.orchestra.run.gibba002.stdout: "target_max_objects": 0, 2021-02-10T11:34:59.447 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:34:59.447 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:34:59.448 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:34:59.448 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_min_flush_age": 0, 2021-02-10T11:34:59.448 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_min_evict_age": 0, 2021-02-10T11:34:59.448 INFO:teuthology.orchestra.run.gibba002.stdout: "erasure_code_profile": "", 2021-02-10T11:34:59.448 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_params": { 2021-02-10T11:34:59.448 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "none" 2021-02-10T11:34:59.448 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:59.448 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_period": 0, 2021-02-10T11:34:59.448 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_count": 0, 2021-02-10T11:34:59.448 INFO:teuthology.orchestra.run.gibba002.stdout: "use_gmt_hitset": true, 2021-02-10T11:34:59.448 INFO:teuthology.orchestra.run.gibba002.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:34:59.448 INFO:teuthology.orchestra.run.gibba002.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:34:59.448 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:34:59.449 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:34:59.449 INFO:teuthology.orchestra.run.gibba002.stdout: "grade_table": [], 2021-02-10T11:34:59.449 INFO:teuthology.orchestra.run.gibba002.stdout: "stripe_width": 0, 2021-02-10T11:34:59.449 INFO:teuthology.orchestra.run.gibba002.stdout: "expected_num_objects": 0, 2021-02-10T11:34:59.449 INFO:teuthology.orchestra.run.gibba002.stdout: "fast_read": false, 2021-02-10T11:34:59.449 INFO:teuthology.orchestra.run.gibba002.stdout: "options": {}, 2021-02-10T11:34:59.449 INFO:teuthology.orchestra.run.gibba002.stdout: "application_metadata": { 2021-02-10T11:34:59.449 INFO:teuthology.orchestra.run.gibba002.stdout: "cephfs": { 2021-02-10T11:34:59.449 INFO:teuthology.orchestra.run.gibba002.stdout: "data": "cephfs" 2021-02-10T11:34:59.449 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:34:59.449 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:34:59.449 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:34:59.449 INFO:teuthology.orchestra.run.gibba002.stdout: ], 2021-02-10T11:34:59.450 INFO:teuthology.orchestra.run.gibba002.stdout: "osds": [ 2021-02-10T11:34:59.450 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:59.450 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 0, 2021-02-10T11:34:59.450 INFO:teuthology.orchestra.run.gibba002.stdout: "uuid": "be691b4d-a039-408a-af80-9d53f02b6f1f", 2021-02-10T11:34:59.450 INFO:teuthology.orchestra.run.gibba002.stdout: "up": 1, 2021-02-10T11:34:59.450 INFO:teuthology.orchestra.run.gibba002.stdout: "in": 1, 2021-02-10T11:34:59.450 INFO:teuthology.orchestra.run.gibba002.stdout: "weight": 1, 2021-02-10T11:34:59.450 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_affinity": 1, 2021-02-10T11:34:59.450 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_begin": 0, 2021-02-10T11:34:59.450 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_end": 0, 2021-02-10T11:34:59.450 INFO:teuthology.orchestra.run.gibba002.stdout: "up_from": 13, 2021-02-10T11:34:59.450 INFO:teuthology.orchestra.run.gibba002.stdout: "up_thru": 134, 2021-02-10T11:34:59.451 INFO:teuthology.orchestra.run.gibba002.stdout: "down_at": 0, 2021-02-10T11:34:59.451 INFO:teuthology.orchestra.run.gibba002.stdout: "lost_at": 0, 2021-02-10T11:34:59.451 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addrs": { 2021-02-10T11:34:59.451 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:34:59.451 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:59.451 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:34:59.451 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6818", 2021-02-10T11:34:59.451 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36088 2021-02-10T11:34:59.451 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:59.451 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:59.451 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:34:59.451 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6820", 2021-02-10T11:34:59.451 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36088 2021-02-10T11:34:59.452 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:34:59.452 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:34:59.452 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:59.452 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addrs": { 2021-02-10T11:34:59.452 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:34:59.452 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:59.452 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:34:59.452 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6822", 2021-02-10T11:34:59.452 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36088 2021-02-10T11:34:59.452 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:59.452 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:59.452 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:34:59.452 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6824", 2021-02-10T11:34:59.453 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36088 2021-02-10T11:34:59.453 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:34:59.453 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:34:59.453 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:59.453 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addrs": { 2021-02-10T11:34:59.453 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:34:59.453 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:59.453 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:34:59.453 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6830", 2021-02-10T11:34:59.453 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36088 2021-02-10T11:34:59.453 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:59.453 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:59.453 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:34:59.454 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6832", 2021-02-10T11:34:59.454 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36088 2021-02-10T11:34:59.454 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:34:59.454 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:34:59.454 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:59.454 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addrs": { 2021-02-10T11:34:59.454 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:34:59.454 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:59.454 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:34:59.454 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6826", 2021-02-10T11:34:59.454 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36088 2021-02-10T11:34:59.454 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:59.454 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:59.455 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:34:59.455 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6828", 2021-02-10T11:34:59.455 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36088 2021-02-10T11:34:59.455 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:34:59.455 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:34:59.455 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:59.455 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addr": "172.21.2.102:6820/36088", 2021-02-10T11:34:59.455 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addr": "172.21.2.102:6824/36088", 2021-02-10T11:34:59.455 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addr": "172.21.2.102:6832/36088", 2021-02-10T11:34:59.455 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addr": "172.21.2.102:6828/36088", 2021-02-10T11:34:59.455 INFO:teuthology.orchestra.run.gibba002.stdout: "state": [ 2021-02-10T11:34:59.455 INFO:teuthology.orchestra.run.gibba002.stdout: "exists", 2021-02-10T11:34:59.455 INFO:teuthology.orchestra.run.gibba002.stdout: "up" 2021-02-10T11:34:59.455 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:34:59.456 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:59.456 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:59.456 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 1, 2021-02-10T11:34:59.456 INFO:teuthology.orchestra.run.gibba002.stdout: "uuid": "64e3ccc8-9c69-4d7c-865e-af8e83822635", 2021-02-10T11:34:59.456 INFO:teuthology.orchestra.run.gibba002.stdout: "up": 1, 2021-02-10T11:34:59.456 INFO:teuthology.orchestra.run.gibba002.stdout: "in": 1, 2021-02-10T11:34:59.456 INFO:teuthology.orchestra.run.gibba002.stdout: "weight": 1, 2021-02-10T11:34:59.456 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_affinity": 1, 2021-02-10T11:34:59.456 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_begin": 0, 2021-02-10T11:34:59.456 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_end": 0, 2021-02-10T11:34:59.456 INFO:teuthology.orchestra.run.gibba002.stdout: "up_from": 13, 2021-02-10T11:34:59.456 INFO:teuthology.orchestra.run.gibba002.stdout: "up_thru": 135, 2021-02-10T11:34:59.456 INFO:teuthology.orchestra.run.gibba002.stdout: "down_at": 0, 2021-02-10T11:34:59.457 INFO:teuthology.orchestra.run.gibba002.stdout: "lost_at": 0, 2021-02-10T11:34:59.457 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addrs": { 2021-02-10T11:34:59.457 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:34:59.457 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:59.457 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:34:59.457 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6819", 2021-02-10T11:34:59.457 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36087 2021-02-10T11:34:59.457 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:59.457 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:59.457 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:34:59.457 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6821", 2021-02-10T11:34:59.457 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36087 2021-02-10T11:34:59.457 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:34:59.458 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:34:59.458 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:59.458 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addrs": { 2021-02-10T11:34:59.458 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:34:59.458 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:59.458 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:34:59.458 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6823", 2021-02-10T11:34:59.458 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36087 2021-02-10T11:34:59.458 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:59.458 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:59.458 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:34:59.458 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6825", 2021-02-10T11:34:59.458 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36087 2021-02-10T11:34:59.459 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:34:59.459 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:34:59.459 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:59.459 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addrs": { 2021-02-10T11:34:59.459 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:34:59.459 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:59.459 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:34:59.459 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6831", 2021-02-10T11:34:59.459 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36087 2021-02-10T11:34:59.459 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:59.459 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:59.459 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:34:59.459 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6833", 2021-02-10T11:34:59.459 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36087 2021-02-10T11:34:59.460 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:34:59.460 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:34:59.460 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:59.460 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addrs": { 2021-02-10T11:34:59.460 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:34:59.460 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:59.460 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:34:59.460 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6827", 2021-02-10T11:34:59.460 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36087 2021-02-10T11:34:59.460 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:59.460 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:59.460 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:34:59.460 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6829", 2021-02-10T11:34:59.461 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36087 2021-02-10T11:34:59.461 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:34:59.461 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:34:59.461 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:59.461 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addr": "172.21.2.102:6821/36087", 2021-02-10T11:34:59.461 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addr": "172.21.2.102:6825/36087", 2021-02-10T11:34:59.461 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addr": "172.21.2.102:6833/36087", 2021-02-10T11:34:59.461 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addr": "172.21.2.102:6829/36087", 2021-02-10T11:34:59.461 INFO:teuthology.orchestra.run.gibba002.stdout: "state": [ 2021-02-10T11:34:59.461 INFO:teuthology.orchestra.run.gibba002.stdout: "exists", 2021-02-10T11:34:59.461 INFO:teuthology.orchestra.run.gibba002.stdout: "up" 2021-02-10T11:34:59.461 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:34:59.461 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:59.462 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:59.462 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 2, 2021-02-10T11:34:59.462 INFO:teuthology.orchestra.run.gibba002.stdout: "uuid": "153700af-747f-481f-bdf4-d6072d3a842f", 2021-02-10T11:34:59.462 INFO:teuthology.orchestra.run.gibba002.stdout: "up": 1, 2021-02-10T11:34:59.462 INFO:teuthology.orchestra.run.gibba002.stdout: "in": 1, 2021-02-10T11:34:59.462 INFO:teuthology.orchestra.run.gibba002.stdout: "weight": 1, 2021-02-10T11:34:59.462 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_affinity": 1, 2021-02-10T11:34:59.462 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_begin": 0, 2021-02-10T11:34:59.462 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_end": 0, 2021-02-10T11:34:59.462 INFO:teuthology.orchestra.run.gibba002.stdout: "up_from": 13, 2021-02-10T11:34:59.462 INFO:teuthology.orchestra.run.gibba002.stdout: "up_thru": 126, 2021-02-10T11:34:59.462 INFO:teuthology.orchestra.run.gibba002.stdout: "down_at": 0, 2021-02-10T11:34:59.464 INFO:teuthology.orchestra.run.gibba002.stdout: "lost_at": 0, 2021-02-10T11:34:59.464 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addrs": { 2021-02-10T11:34:59.464 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:34:59.464 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:59.465 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:34:59.465 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6810", 2021-02-10T11:34:59.465 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36089 2021-02-10T11:34:59.465 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:59.465 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:59.465 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:34:59.465 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6811", 2021-02-10T11:34:59.465 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36089 2021-02-10T11:34:59.465 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:34:59.465 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:34:59.465 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:59.465 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addrs": { 2021-02-10T11:34:59.465 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:34:59.466 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:59.466 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:34:59.466 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6812", 2021-02-10T11:34:59.466 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36089 2021-02-10T11:34:59.466 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:59.466 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:59.466 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:34:59.466 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6813", 2021-02-10T11:34:59.466 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36089 2021-02-10T11:34:59.466 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:34:59.466 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:34:59.466 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:59.466 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addrs": { 2021-02-10T11:34:59.467 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:34:59.467 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:59.467 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:34:59.467 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6816", 2021-02-10T11:34:59.467 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36089 2021-02-10T11:34:59.467 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:59.467 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:59.467 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:34:59.467 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6817", 2021-02-10T11:34:59.467 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36089 2021-02-10T11:34:59.467 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:34:59.467 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:34:59.467 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:59.468 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addrs": { 2021-02-10T11:34:59.468 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:34:59.468 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:59.468 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:34:59.468 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6814", 2021-02-10T11:34:59.468 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36089 2021-02-10T11:34:59.468 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:59.468 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:59.468 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:34:59.468 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6815", 2021-02-10T11:34:59.468 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36089 2021-02-10T11:34:59.468 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:34:59.468 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:34:59.468 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:59.469 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addr": "172.21.2.102:6811/36089", 2021-02-10T11:34:59.469 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addr": "172.21.2.102:6813/36089", 2021-02-10T11:34:59.469 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addr": "172.21.2.102:6817/36089", 2021-02-10T11:34:59.469 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addr": "172.21.2.102:6815/36089", 2021-02-10T11:34:59.469 INFO:teuthology.orchestra.run.gibba002.stdout: "state": [ 2021-02-10T11:34:59.469 INFO:teuthology.orchestra.run.gibba002.stdout: "exists", 2021-02-10T11:34:59.469 INFO:teuthology.orchestra.run.gibba002.stdout: "up" 2021-02-10T11:34:59.469 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:34:59.469 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:59.469 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:59.469 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 3, 2021-02-10T11:34:59.469 INFO:teuthology.orchestra.run.gibba002.stdout: "uuid": "cbc678f4-83a9-45e6-8830-2f7c232b1ea7", 2021-02-10T11:34:59.469 INFO:teuthology.orchestra.run.gibba002.stdout: "up": 1, 2021-02-10T11:34:59.470 INFO:teuthology.orchestra.run.gibba002.stdout: "in": 1, 2021-02-10T11:34:59.470 INFO:teuthology.orchestra.run.gibba002.stdout: "weight": 1, 2021-02-10T11:34:59.470 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_affinity": 1, 2021-02-10T11:34:59.470 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_begin": 0, 2021-02-10T11:34:59.470 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_end": 0, 2021-02-10T11:34:59.470 INFO:teuthology.orchestra.run.gibba002.stdout: "up_from": 13, 2021-02-10T11:34:59.470 INFO:teuthology.orchestra.run.gibba002.stdout: "up_thru": 118, 2021-02-10T11:34:59.470 INFO:teuthology.orchestra.run.gibba002.stdout: "down_at": 0, 2021-02-10T11:34:59.470 INFO:teuthology.orchestra.run.gibba002.stdout: "lost_at": 0, 2021-02-10T11:34:59.470 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addrs": { 2021-02-10T11:34:59.470 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:34:59.470 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:59.470 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:34:59.471 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6802", 2021-02-10T11:34:59.471 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36090 2021-02-10T11:34:59.471 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:59.471 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:59.471 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:34:59.471 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6803", 2021-02-10T11:34:59.471 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36090 2021-02-10T11:34:59.471 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:34:59.471 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:34:59.471 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:59.471 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addrs": { 2021-02-10T11:34:59.471 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:34:59.472 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:59.472 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:34:59.472 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6804", 2021-02-10T11:34:59.472 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36090 2021-02-10T11:34:59.472 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:59.472 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:59.472 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:34:59.472 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6805", 2021-02-10T11:34:59.472 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36090 2021-02-10T11:34:59.472 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:34:59.472 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:34:59.472 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:59.472 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addrs": { 2021-02-10T11:34:59.472 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:34:59.473 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:59.473 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:34:59.473 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6808", 2021-02-10T11:34:59.473 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36090 2021-02-10T11:34:59.473 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:59.473 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:59.473 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:34:59.473 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6809", 2021-02-10T11:34:59.473 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36090 2021-02-10T11:34:59.473 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:34:59.473 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:34:59.473 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:59.473 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addrs": { 2021-02-10T11:34:59.474 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:34:59.474 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:59.474 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:34:59.474 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6806", 2021-02-10T11:34:59.474 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36090 2021-02-10T11:34:59.474 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:59.474 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:59.474 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:34:59.474 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6807", 2021-02-10T11:34:59.474 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36090 2021-02-10T11:34:59.474 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:34:59.474 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:34:59.475 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:59.475 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addr": "172.21.2.102:6803/36090", 2021-02-10T11:34:59.475 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addr": "172.21.2.102:6805/36090", 2021-02-10T11:34:59.475 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addr": "172.21.2.102:6809/36090", 2021-02-10T11:34:59.475 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addr": "172.21.2.102:6807/36090", 2021-02-10T11:34:59.475 INFO:teuthology.orchestra.run.gibba002.stdout: "state": [ 2021-02-10T11:34:59.475 INFO:teuthology.orchestra.run.gibba002.stdout: "exists", 2021-02-10T11:34:59.475 INFO:teuthology.orchestra.run.gibba002.stdout: "up" 2021-02-10T11:34:59.475 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:34:59.475 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:59.475 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:59.475 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 4, 2021-02-10T11:34:59.475 INFO:teuthology.orchestra.run.gibba002.stdout: "uuid": "c01e4008-b258-4f10-afd2-9e7f37214351", 2021-02-10T11:34:59.475 INFO:teuthology.orchestra.run.gibba002.stdout: "up": 1, 2021-02-10T11:34:59.476 INFO:teuthology.orchestra.run.gibba002.stdout: "in": 1, 2021-02-10T11:34:59.476 INFO:teuthology.orchestra.run.gibba002.stdout: "weight": 1, 2021-02-10T11:34:59.476 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_affinity": 1, 2021-02-10T11:34:59.476 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_begin": 0, 2021-02-10T11:34:59.476 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_end": 0, 2021-02-10T11:34:59.476 INFO:teuthology.orchestra.run.gibba002.stdout: "up_from": 13, 2021-02-10T11:34:59.476 INFO:teuthology.orchestra.run.gibba002.stdout: "up_thru": 135, 2021-02-10T11:34:59.476 INFO:teuthology.orchestra.run.gibba002.stdout: "down_at": 0, 2021-02-10T11:34:59.476 INFO:teuthology.orchestra.run.gibba002.stdout: "lost_at": 0, 2021-02-10T11:34:59.476 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addrs": { 2021-02-10T11:34:59.476 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:34:59.476 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:59.476 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:34:59.477 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6808", 2021-02-10T11:34:59.477 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36368 2021-02-10T11:34:59.477 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:59.477 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:59.477 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:34:59.477 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6811", 2021-02-10T11:34:59.477 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36368 2021-02-10T11:34:59.477 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:34:59.477 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:34:59.477 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:59.477 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addrs": { 2021-02-10T11:34:59.477 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:34:59.477 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:59.478 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:34:59.478 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6814", 2021-02-10T11:34:59.478 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36368 2021-02-10T11:34:59.478 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:59.478 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:59.478 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:34:59.478 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6818", 2021-02-10T11:34:59.478 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36368 2021-02-10T11:34:59.478 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:34:59.478 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:34:59.478 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:59.478 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addrs": { 2021-02-10T11:34:59.478 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:34:59.479 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:59.479 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:34:59.479 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6826", 2021-02-10T11:34:59.479 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36368 2021-02-10T11:34:59.479 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:59.479 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:59.479 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:34:59.479 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6828", 2021-02-10T11:34:59.479 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36368 2021-02-10T11:34:59.479 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:34:59.479 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:34:59.479 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:59.479 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addrs": { 2021-02-10T11:34:59.480 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:34:59.480 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:59.480 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:34:59.480 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6820", 2021-02-10T11:34:59.480 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36368 2021-02-10T11:34:59.480 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:59.480 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:59.480 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:34:59.480 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6824", 2021-02-10T11:34:59.480 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36368 2021-02-10T11:34:59.480 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:34:59.480 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:34:59.480 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:59.481 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addr": "172.21.2.113:6811/36368", 2021-02-10T11:34:59.481 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addr": "172.21.2.113:6818/36368", 2021-02-10T11:34:59.481 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addr": "172.21.2.113:6828/36368", 2021-02-10T11:34:59.481 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addr": "172.21.2.113:6824/36368", 2021-02-10T11:34:59.481 INFO:teuthology.orchestra.run.gibba002.stdout: "state": [ 2021-02-10T11:34:59.481 INFO:teuthology.orchestra.run.gibba002.stdout: "exists", 2021-02-10T11:34:59.481 INFO:teuthology.orchestra.run.gibba002.stdout: "up" 2021-02-10T11:34:59.481 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:34:59.481 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:59.481 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:59.481 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 5, 2021-02-10T11:34:59.481 INFO:teuthology.orchestra.run.gibba002.stdout: "uuid": "551f6007-daf8-4a98-8ee0-c2b16bcccb5e", 2021-02-10T11:34:59.481 INFO:teuthology.orchestra.run.gibba002.stdout: "up": 1, 2021-02-10T11:34:59.481 INFO:teuthology.orchestra.run.gibba002.stdout: "in": 1, 2021-02-10T11:34:59.482 INFO:teuthology.orchestra.run.gibba002.stdout: "weight": 1, 2021-02-10T11:34:59.482 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_affinity": 1, 2021-02-10T11:34:59.482 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_begin": 0, 2021-02-10T11:34:59.482 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_end": 0, 2021-02-10T11:34:59.482 INFO:teuthology.orchestra.run.gibba002.stdout: "up_from": 13, 2021-02-10T11:34:59.482 INFO:teuthology.orchestra.run.gibba002.stdout: "up_thru": 134, 2021-02-10T11:34:59.482 INFO:teuthology.orchestra.run.gibba002.stdout: "down_at": 0, 2021-02-10T11:34:59.482 INFO:teuthology.orchestra.run.gibba002.stdout: "lost_at": 0, 2021-02-10T11:34:59.482 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addrs": { 2021-02-10T11:34:59.482 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:34:59.482 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:59.482 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:34:59.483 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6802", 2021-02-10T11:34:59.483 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36366 2021-02-10T11:34:59.483 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:59.483 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:59.483 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:34:59.483 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6805", 2021-02-10T11:34:59.483 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36366 2021-02-10T11:34:59.483 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:34:59.483 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:34:59.483 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:59.483 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addrs": { 2021-02-10T11:34:59.483 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:34:59.483 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:59.483 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:34:59.484 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6807", 2021-02-10T11:34:59.484 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36366 2021-02-10T11:34:59.484 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:59.484 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:59.484 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:34:59.484 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6810", 2021-02-10T11:34:59.484 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36366 2021-02-10T11:34:59.484 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:34:59.484 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:34:59.484 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:59.484 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addrs": { 2021-02-10T11:34:59.484 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:34:59.485 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:59.485 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:34:59.485 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6819", 2021-02-10T11:34:59.485 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36366 2021-02-10T11:34:59.485 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:59.485 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:59.485 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:34:59.485 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6822", 2021-02-10T11:34:59.485 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36366 2021-02-10T11:34:59.485 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:34:59.485 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:34:59.485 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:59.485 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addrs": { 2021-02-10T11:34:59.486 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:34:59.486 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:59.486 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:34:59.486 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6812", 2021-02-10T11:34:59.486 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36366 2021-02-10T11:34:59.486 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:59.486 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:59.486 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:34:59.486 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6815", 2021-02-10T11:34:59.486 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36366 2021-02-10T11:34:59.486 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:34:59.486 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:34:59.486 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:59.487 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addr": "172.21.2.113:6805/36366", 2021-02-10T11:34:59.487 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addr": "172.21.2.113:6810/36366", 2021-02-10T11:34:59.487 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addr": "172.21.2.113:6822/36366", 2021-02-10T11:34:59.487 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addr": "172.21.2.113:6815/36366", 2021-02-10T11:34:59.487 INFO:teuthology.orchestra.run.gibba002.stdout: "state": [ 2021-02-10T11:34:59.487 INFO:teuthology.orchestra.run.gibba002.stdout: "exists", 2021-02-10T11:34:59.487 INFO:teuthology.orchestra.run.gibba002.stdout: "up" 2021-02-10T11:34:59.487 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:34:59.487 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:59.487 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:59.487 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 6, 2021-02-10T11:34:59.487 INFO:teuthology.orchestra.run.gibba002.stdout: "uuid": "bf67c040-0e50-48be-b102-c3943bc9ebef", 2021-02-10T11:34:59.487 INFO:teuthology.orchestra.run.gibba002.stdout: "up": 1, 2021-02-10T11:34:59.488 INFO:teuthology.orchestra.run.gibba002.stdout: "in": 1, 2021-02-10T11:34:59.488 INFO:teuthology.orchestra.run.gibba002.stdout: "weight": 1, 2021-02-10T11:34:59.488 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_affinity": 1, 2021-02-10T11:34:59.488 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_begin": 0, 2021-02-10T11:34:59.488 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_end": 0, 2021-02-10T11:34:59.488 INFO:teuthology.orchestra.run.gibba002.stdout: "up_from": 13, 2021-02-10T11:34:59.488 INFO:teuthology.orchestra.run.gibba002.stdout: "up_thru": 135, 2021-02-10T11:34:59.488 INFO:teuthology.orchestra.run.gibba002.stdout: "down_at": 0, 2021-02-10T11:34:59.488 INFO:teuthology.orchestra.run.gibba002.stdout: "lost_at": 0, 2021-02-10T11:34:59.488 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addrs": { 2021-02-10T11:34:59.488 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:34:59.488 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:59.488 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:34:59.489 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6800", 2021-02-10T11:34:59.489 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36367 2021-02-10T11:34:59.489 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:59.489 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:59.489 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:34:59.489 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6801", 2021-02-10T11:34:59.489 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36367 2021-02-10T11:34:59.489 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:34:59.489 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:34:59.489 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:59.489 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addrs": { 2021-02-10T11:34:59.489 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:34:59.489 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:59.490 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:34:59.490 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6803", 2021-02-10T11:34:59.490 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36367 2021-02-10T11:34:59.490 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:59.490 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:59.490 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:34:59.490 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6804", 2021-02-10T11:34:59.490 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36367 2021-02-10T11:34:59.490 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:34:59.490 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:34:59.490 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:59.490 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addrs": { 2021-02-10T11:34:59.490 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:34:59.491 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:59.491 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:34:59.491 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6813", 2021-02-10T11:34:59.491 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36367 2021-02-10T11:34:59.491 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:59.491 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:59.491 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:34:59.491 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6816", 2021-02-10T11:34:59.491 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36367 2021-02-10T11:34:59.491 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:34:59.491 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:34:59.491 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:59.491 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addrs": { 2021-02-10T11:34:59.492 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:34:59.492 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:59.492 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:34:59.492 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6806", 2021-02-10T11:34:59.492 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36367 2021-02-10T11:34:59.492 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:59.492 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:59.492 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:34:59.492 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6809", 2021-02-10T11:34:59.492 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36367 2021-02-10T11:34:59.492 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:34:59.492 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:34:59.492 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:59.493 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addr": "172.21.2.113:6801/36367", 2021-02-10T11:34:59.493 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addr": "172.21.2.113:6804/36367", 2021-02-10T11:34:59.493 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addr": "172.21.2.113:6816/36367", 2021-02-10T11:34:59.493 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addr": "172.21.2.113:6809/36367", 2021-02-10T11:34:59.493 INFO:teuthology.orchestra.run.gibba002.stdout: "state": [ 2021-02-10T11:34:59.493 INFO:teuthology.orchestra.run.gibba002.stdout: "exists", 2021-02-10T11:34:59.493 INFO:teuthology.orchestra.run.gibba002.stdout: "up" 2021-02-10T11:34:59.493 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:34:59.493 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:59.493 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:59.493 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 7, 2021-02-10T11:34:59.493 INFO:teuthology.orchestra.run.gibba002.stdout: "uuid": "f5461f80-018d-428f-be89-2eb2bcbe7bb6", 2021-02-10T11:34:59.493 INFO:teuthology.orchestra.run.gibba002.stdout: "up": 1, 2021-02-10T11:34:59.494 INFO:teuthology.orchestra.run.gibba002.stdout: "in": 1, 2021-02-10T11:34:59.494 INFO:teuthology.orchestra.run.gibba002.stdout: "weight": 1, 2021-02-10T11:34:59.494 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_affinity": 1, 2021-02-10T11:34:59.494 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_begin": 0, 2021-02-10T11:34:59.494 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_end": 0, 2021-02-10T11:34:59.494 INFO:teuthology.orchestra.run.gibba002.stdout: "up_from": 13, 2021-02-10T11:34:59.494 INFO:teuthology.orchestra.run.gibba002.stdout: "up_thru": 135, 2021-02-10T11:34:59.494 INFO:teuthology.orchestra.run.gibba002.stdout: "down_at": 0, 2021-02-10T11:34:59.494 INFO:teuthology.orchestra.run.gibba002.stdout: "lost_at": 0, 2021-02-10T11:34:59.494 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addrs": { 2021-02-10T11:34:59.494 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:34:59.494 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:59.494 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:34:59.495 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6817", 2021-02-10T11:34:59.495 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36365 2021-02-10T11:34:59.495 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:59.495 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:59.495 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:34:59.495 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6821", 2021-02-10T11:34:59.495 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36365 2021-02-10T11:34:59.495 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:34:59.495 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:34:59.495 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:59.495 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addrs": { 2021-02-10T11:34:59.495 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:34:59.495 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:59.495 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:34:59.496 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6823", 2021-02-10T11:34:59.496 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36365 2021-02-10T11:34:59.496 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:59.496 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:59.496 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:34:59.496 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6825", 2021-02-10T11:34:59.496 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36365 2021-02-10T11:34:59.496 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:34:59.496 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:34:59.496 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:59.496 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addrs": { 2021-02-10T11:34:59.496 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:34:59.496 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:59.497 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:34:59.497 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6830", 2021-02-10T11:34:59.497 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36365 2021-02-10T11:34:59.497 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:59.497 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:59.497 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:34:59.497 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6831", 2021-02-10T11:34:59.497 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36365 2021-02-10T11:34:59.497 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:34:59.497 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:34:59.497 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:59.497 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addrs": { 2021-02-10T11:34:59.497 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:34:59.497 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:59.498 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:34:59.498 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6827", 2021-02-10T11:34:59.498 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36365 2021-02-10T11:34:59.498 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:59.498 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:59.498 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:34:59.498 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6829", 2021-02-10T11:34:59.498 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36365 2021-02-10T11:34:59.498 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:34:59.498 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:34:59.498 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:59.498 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addr": "172.21.2.113:6821/36365", 2021-02-10T11:34:59.499 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addr": "172.21.2.113:6825/36365", 2021-02-10T11:34:59.499 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addr": "172.21.2.113:6831/36365", 2021-02-10T11:34:59.499 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addr": "172.21.2.113:6829/36365", 2021-02-10T11:34:59.499 INFO:teuthology.orchestra.run.gibba002.stdout: "state": [ 2021-02-10T11:34:59.499 INFO:teuthology.orchestra.run.gibba002.stdout: "exists", 2021-02-10T11:34:59.499 INFO:teuthology.orchestra.run.gibba002.stdout: "up" 2021-02-10T11:34:59.499 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:34:59.499 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:34:59.499 INFO:teuthology.orchestra.run.gibba002.stdout: ], 2021-02-10T11:34:59.499 INFO:teuthology.orchestra.run.gibba002.stdout: "osd_xinfo": [ 2021-02-10T11:34:59.499 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:59.499 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 0, 2021-02-10T11:34:59.499 INFO:teuthology.orchestra.run.gibba002.stdout: "down_stamp": "0.000000", 2021-02-10T11:34:59.500 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_probability": 0, 2021-02-10T11:34:59.500 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_interval": 0, 2021-02-10T11:34:59.500 INFO:teuthology.orchestra.run.gibba002.stdout: "features": 4611087854035861503, 2021-02-10T11:34:59.500 INFO:teuthology.orchestra.run.gibba002.stdout: "old_weight": 0 2021-02-10T11:34:59.500 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:59.500 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:59.500 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 1, 2021-02-10T11:34:59.500 INFO:teuthology.orchestra.run.gibba002.stdout: "down_stamp": "0.000000", 2021-02-10T11:34:59.500 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_probability": 0, 2021-02-10T11:34:59.500 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_interval": 0, 2021-02-10T11:34:59.500 INFO:teuthology.orchestra.run.gibba002.stdout: "features": 4611087854035861503, 2021-02-10T11:34:59.500 INFO:teuthology.orchestra.run.gibba002.stdout: "old_weight": 0 2021-02-10T11:34:59.500 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:59.501 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:59.501 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 2, 2021-02-10T11:34:59.501 INFO:teuthology.orchestra.run.gibba002.stdout: "down_stamp": "0.000000", 2021-02-10T11:34:59.501 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_probability": 0, 2021-02-10T11:34:59.501 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_interval": 0, 2021-02-10T11:34:59.501 INFO:teuthology.orchestra.run.gibba002.stdout: "features": 4611087854035861503, 2021-02-10T11:34:59.501 INFO:teuthology.orchestra.run.gibba002.stdout: "old_weight": 0 2021-02-10T11:34:59.501 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:59.501 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:59.501 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 3, 2021-02-10T11:34:59.501 INFO:teuthology.orchestra.run.gibba002.stdout: "down_stamp": "0.000000", 2021-02-10T11:34:59.501 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_probability": 0, 2021-02-10T11:34:59.501 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_interval": 0, 2021-02-10T11:34:59.501 INFO:teuthology.orchestra.run.gibba002.stdout: "features": 4611087854035861503, 2021-02-10T11:34:59.502 INFO:teuthology.orchestra.run.gibba002.stdout: "old_weight": 0 2021-02-10T11:34:59.502 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:59.502 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:59.502 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 4, 2021-02-10T11:34:59.502 INFO:teuthology.orchestra.run.gibba002.stdout: "down_stamp": "0.000000", 2021-02-10T11:34:59.502 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_probability": 0, 2021-02-10T11:34:59.502 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_interval": 0, 2021-02-10T11:34:59.502 INFO:teuthology.orchestra.run.gibba002.stdout: "features": 4611087854035861503, 2021-02-10T11:34:59.502 INFO:teuthology.orchestra.run.gibba002.stdout: "old_weight": 0 2021-02-10T11:34:59.502 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:59.502 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:59.502 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 5, 2021-02-10T11:34:59.502 INFO:teuthology.orchestra.run.gibba002.stdout: "down_stamp": "0.000000", 2021-02-10T11:34:59.503 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_probability": 0, 2021-02-10T11:34:59.503 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_interval": 0, 2021-02-10T11:34:59.503 INFO:teuthology.orchestra.run.gibba002.stdout: "features": 4611087854035861503, 2021-02-10T11:34:59.503 INFO:teuthology.orchestra.run.gibba002.stdout: "old_weight": 0 2021-02-10T11:34:59.503 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:59.503 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:59.503 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 6, 2021-02-10T11:34:59.503 INFO:teuthology.orchestra.run.gibba002.stdout: "down_stamp": "0.000000", 2021-02-10T11:34:59.503 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_probability": 0, 2021-02-10T11:34:59.503 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_interval": 0, 2021-02-10T11:34:59.503 INFO:teuthology.orchestra.run.gibba002.stdout: "features": 4611087854035861503, 2021-02-10T11:34:59.503 INFO:teuthology.orchestra.run.gibba002.stdout: "old_weight": 0 2021-02-10T11:34:59.503 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:59.504 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:34:59.504 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 7, 2021-02-10T11:34:59.504 INFO:teuthology.orchestra.run.gibba002.stdout: "down_stamp": "0.000000", 2021-02-10T11:34:59.504 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_probability": 0, 2021-02-10T11:34:59.504 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_interval": 0, 2021-02-10T11:34:59.504 INFO:teuthology.orchestra.run.gibba002.stdout: "features": 4611087854035861503, 2021-02-10T11:34:59.504 INFO:teuthology.orchestra.run.gibba002.stdout: "old_weight": 0 2021-02-10T11:34:59.504 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:34:59.504 INFO:teuthology.orchestra.run.gibba002.stdout: ], 2021-02-10T11:34:59.504 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_upmap": [], 2021-02-10T11:34:59.504 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_upmap_items": [], 2021-02-10T11:34:59.504 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_temp": [], 2021-02-10T11:34:59.504 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_temp": [], 2021-02-10T11:34:59.504 INFO:teuthology.orchestra.run.gibba002.stdout: "blacklist": {}, 2021-02-10T11:34:59.505 INFO:teuthology.orchestra.run.gibba002.stdout: "erasure_code_profiles": { 2021-02-10T11:34:59.505 INFO:teuthology.orchestra.run.gibba002.stdout: "default": { 2021-02-10T11:34:59.505 INFO:teuthology.orchestra.run.gibba002.stdout: "crush-failure-domain": "osd", 2021-02-10T11:34:59.505 INFO:teuthology.orchestra.run.gibba002.stdout: "k": "2", 2021-02-10T11:34:59.505 INFO:teuthology.orchestra.run.gibba002.stdout: "m": "1", 2021-02-10T11:34:59.505 INFO:teuthology.orchestra.run.gibba002.stdout: "plugin": "jerasure", 2021-02-10T11:34:59.505 INFO:teuthology.orchestra.run.gibba002.stdout: "ruleset-failure-domain": "osd", 2021-02-10T11:34:59.505 INFO:teuthology.orchestra.run.gibba002.stdout: "technique": "reed_sol_van" 2021-02-10T11:34:59.505 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:34:59.505 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:34:59.505 INFO:teuthology.orchestra.run.gibba002.stdout: "removed_snaps_queue": [], 2021-02-10T11:34:59.505 INFO:teuthology.orchestra.run.gibba002.stdout: "new_removed_snaps": [], 2021-02-10T11:34:59.505 INFO:teuthology.orchestra.run.gibba002.stdout: "new_purged_snaps": [], 2021-02-10T11:34:59.506 INFO:teuthology.orchestra.run.gibba002.stdout: "crush_node_flags": {}, 2021-02-10T11:34:59.506 INFO:teuthology.orchestra.run.gibba002.stdout: "device_class_flags": {} 2021-02-10T11:34:59.506 INFO:teuthology.orchestra.run.gibba002.stdout:} 2021-02-10T11:34:59.507 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:59.430 7f3897086700 1 -- 172.21.2.102:0/1604194746 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f38740186c0 msgr2=0x7f387401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:59.507 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:59.430 7f3897086700 1 --2- 172.21.2.102:0/1604194746 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f38740186c0 0x7f387401abc0 crc :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:59.507 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:59.430 7f3897086700 1 -- 172.21.2.102:0/1604194746 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f389019da20 msgr2=0x7f389019de20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:59.507 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:59.430 7f3897086700 1 --2- 172.21.2.102:0/1604194746 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f389019da20 0x7f389019de20 secure :-1 s=READY pgs=2609 cs=0 l=1 rev1=1 rx=0x7f388800b940 tx=0x7f388800bce0).stop 2021-02-10T11:34:59.507 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:59.430 7f3897086700 1 -- 172.21.2.102:0/1604194746 shutdown_connections 2021-02-10T11:34:59.507 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:59.430 7f3897086700 1 --2- 172.21.2.102:0/1604194746 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f389019da20 0x7f389019de20 unknown :-1 s=CLOSED pgs=2609 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:59.508 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:59.430 7f3897086700 1 --2- 172.21.2.102:0/1604194746 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f38901b80f0 0x7f38901b84f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:59.508 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:59.430 7f3897086700 1 --2- 172.21.2.102:0/1604194746 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f38740186c0 0x7f387401abc0 unknown :-1 s=CLOSED pgs=1151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:59.508 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:59.430 7f3897086700 1 --2- 172.21.2.102:0/1604194746 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f38901b7cf0 0x7f389019cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:59.508 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:59.430 7f3897086700 1 -- 172.21.2.102:0/1604194746 >> 172.21.2.102:0/1604194746 conn(0x7f38900ed810 msgr2=0x7f3890094980 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:59.508 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:59.431 7f3897086700 1 -- 172.21.2.102:0/1604194746 shutdown_connections 2021-02-10T11:34:59.508 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:59.431 7f3897086700 1 -- 172.21.2.102:0/1604194746 wait complete. 2021-02-10T11:34:59.509 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:34:59.939 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:59.920 7f0228858700 1 Processor -- start 2021-02-10T11:34:59.939 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:59.920 7f0228858700 1 -- start start 2021-02-10T11:34:59.939 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:59.921 7f0228858700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f02240d1870 0x7f02241b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:59.939 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:59.921 7f0228858700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f02241b0a10 0x7f02240cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:59.939 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:59.921 7f0228858700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02240f0b10 con 0x7f02240d1c70 2021-02-10T11:34:59.939 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:59.921 7f0228858700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02240e5ed0 con 0x7f02240d1870 2021-02-10T11:34:59.939 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:59.921 7f0228858700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02240f0650 con 0x7f02241b0a10 2021-02-10T11:34:59.939 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:59.921 7f022261c700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f02240d1870 0x7f02241b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:59.939 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:59.921 7f022261c700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f02240d1870 0x7f02241b03f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:36510/0 (socket says 172.21.2.102:36510) 2021-02-10T11:34:59.940 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:59.921 7f022261c700 1 -- 172.21.2.102:0/3133355591 learned_addr learned my addr 172.21.2.102:0/3133355591 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:34:59.940 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:59.921 7f0221e1b700 1 --2- 172.21.2.102:0/3133355591 >> v2:172.21.2.113:3300/0 conn(0x7f02241b0a10 0x7f02240cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:59.940 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:59.921 7f022261c700 1 -- 172.21.2.102:0/3133355591 >> v2:172.21.2.113:3300/0 conn(0x7f02241b0a10 msgr2=0x7f02240cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:34:59.940 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:59.921 7f022261c700 1 --2- 172.21.2.102:0/3133355591 >> v2:172.21.2.113:3300/0 conn(0x7f02241b0a10 0x7f02240cf330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:59.940 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:59.921 7f022261c700 1 -- 172.21.2.102:0/3133355591 >> v1:172.21.2.113:6789/0 conn(0x7f02240d1c70 legacy=0x7f02240cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:59.940 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:59.921 7f022261c700 1 -- 172.21.2.102:0/3133355591 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02240f0840 con 0x7f02240d1870 2021-02-10T11:34:59.940 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:59.921 7f022261c700 1 --2- 172.21.2.102:0/3133355591 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f02240d1870 0x7f02241b03f0 secure :-1 s=READY pgs=2610 cs=0 l=1 rev1=1 rx=0x7f0214006700 tx=0x7f0214009b80).ready entity=mon.0 client_cookie=2a0c3502e6a88a4c server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:59.940 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:59.922 7f022161a700 1 -- 172.21.2.102:0/3133355591 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f021400c0c0 con 0x7f02240d1870 2021-02-10T11:34:59.940 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:59.922 7f022161a700 1 -- 172.21.2.102:0/3133355591 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f021400c3c0 con 0x7f02240d1870 2021-02-10T11:34:59.940 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:59.922 7f022161a700 1 -- 172.21.2.102:0/3133355591 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f02140009b0 con 0x7f02240d1870 2021-02-10T11:34:59.940 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:59.922 7f0228858700 1 -- 172.21.2.102:0/3133355591 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f02240d1870 msgr2=0x7f02241b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:59.940 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:59.922 7f0228858700 1 --2- 172.21.2.102:0/3133355591 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f02240d1870 0x7f02241b03f0 secure :-1 s=READY pgs=2610 cs=0 l=1 rev1=1 rx=0x7f0214006700 tx=0x7f0214009b80).stop 2021-02-10T11:34:59.940 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:59.922 7f0228858700 1 -- 172.21.2.102:0/3133355591 shutdown_connections 2021-02-10T11:34:59.941 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:59.922 7f0228858700 1 --2- 172.21.2.102:0/3133355591 >> v2:172.21.2.113:3300/0 conn(0x7f02241b0a10 0x7f02240cf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:59.941 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:59.922 7f0228858700 1 --2- 172.21.2.102:0/3133355591 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f02240d1870 0x7f02241b03f0 unknown :-1 s=CLOSED pgs=2610 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:59.941 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:59.922 7f0228858700 1 -- 172.21.2.102:0/3133355591 >> 172.21.2.102:0/3133355591 conn(0x7f02240ce450 msgr2=0x7f02240d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:59.941 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:59.923 7f0228858700 1 -- 172.21.2.102:0/3133355591 shutdown_connections 2021-02-10T11:34:59.941 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:59.923 7f0228858700 1 -- 172.21.2.102:0/3133355591 wait complete. 2021-02-10T11:34:59.942 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:59.937 7f0228858700 1 Processor -- start 2021-02-10T11:34:59.942 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:59.937 7f0228858700 1 -- start start 2021-02-10T11:34:59.942 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:59.938 7f0228858700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f02240ada90 0x7f02240ade90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:59.942 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:59.938 7f0228858700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0224099d20 0x7f022409a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:59.943 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:59.938 7f0228858700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f022409a740 0x7f0224097bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:59.943 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:59.938 7f0228858700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02240997a0 con 0x7f02240ada90 2021-02-10T11:34:59.943 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:59.938 7f0228858700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f022418f0c0 con 0x7f0224099d20 2021-02-10T11:34:59.943 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:59.938 7f0228858700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f022409ab40 con 0x7f022409a740 2021-02-10T11:34:59.943 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:59.938 7f022261c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f02240ada90 0x7f02240ade90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:59.943 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:59.938 7f022261c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f02240ada90 0x7f02240ade90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:36518/0 (socket says 172.21.2.102:36518) 2021-02-10T11:34:59.943 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:59.938 7f022261c700 1 -- 172.21.2.102:0/2814683725 learned_addr learned my addr 172.21.2.102:0/2814683725 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:34:59.943 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:59.938 7f022261c700 1 -- 172.21.2.102:0/2814683725 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f022409a740 msgr2=0x7f0224097bc0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:34:59.943 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:59.938 7f022261c700 1 --2- 172.21.2.102:0/2814683725 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f022409a740 0x7f0224097bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:59.943 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:59.938 7f022261c700 1 -- 172.21.2.102:0/2814683725 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0224099d20 msgr2=0x7f022409a120 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:34:59.943 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:59.938 7f022261c700 1 --2- 172.21.2.102:0/2814683725 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0224099d20 0x7f022409a120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:59.943 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:59.938 7f022261c700 1 -- 172.21.2.102:0/2814683725 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f022409ad60 con 0x7f02240ada90 2021-02-10T11:34:59.944 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:59.938 7f022261c700 1 --2- 172.21.2.102:0/2814683725 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f02240ada90 0x7f02240ade90 secure :-1 s=READY pgs=2611 cs=0 l=1 rev1=1 rx=0x7f02140066a0 tx=0x7f021400b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:59.944 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:59.938 7f021affd700 1 -- 172.21.2.102:0/2814683725 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f021400bd80 con 0x7f02240ada90 2021-02-10T11:34:59.944 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:59.939 7f021affd700 1 -- 172.21.2.102:0/2814683725 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02140052e0 con 0x7f02240ada90 2021-02-10T11:34:59.944 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:59.939 7f021affd700 1 -- 172.21.2.102:0/2814683725 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0214005740 con 0x7f02240ada90 2021-02-10T11:34:59.944 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:59.939 7f0228858700 1 -- 172.21.2.102:0/2814683725 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f022409ad60 con 0x7f02240ada90 2021-02-10T11:34:59.944 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:59.939 7f0228858700 1 -- 172.21.2.102:0/2814683725 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02240984d0 con 0x7f02240ada90 2021-02-10T11:34:59.944 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:59.939 7f021affd700 1 -- 172.21.2.102:0/2814683725 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f0214006030 con 0x7f02240ada90 2021-02-10T11:34:59.944 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:59.939 7f021affd700 1 --2- 172.21.2.102:0/2814683725 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f02080186c0 0x7f020801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:59.944 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:59.939 7f021affd700 1 -- 172.21.2.102:0/2814683725 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(137..137 src has 1..137) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f02140323d0 con 0x7f02240ada90 2021-02-10T11:34:59.944 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:59.939 7f0221e1b700 1 --2- 172.21.2.102:0/2814683725 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f02080186c0 0x7f020801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:59.944 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:59.940 7f0221e1b700 1 --2- 172.21.2.102:0/2814683725 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f02080186c0 0x7f020801abc0 crc :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:59.973 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:59.971 7f0228858700 1 -- 172.21.2.102:0/2814683725 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f022418f070 con 0x7f02240ada90 2021-02-10T11:34:59.975 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:34:59.973 7f021affd700 1 -- 172.21.2.102:0/2814683725 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f0214030d00 con 0x7f02240ada90 2021-02-10T11:35:00.102 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:00.100 7f0228858700 1 -- 172.21.2.102:0/2814683725 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f022418f070 con 0x7f02240ada90 2021-02-10T11:35:00.102 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:00.100 7f021affd700 1 -- 172.21.2.102:0/2814683725 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 117 v117) v1 ==== 95+0+2108 (secure 0 0 0) 0x7f0214006030 con 0x7f02240ada90 2021-02-10T11:35:00.103 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:35:00.103 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":117,"default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":11466,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.102:6835/1274216855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6834","nonce":1274216855},{"type":"v1","addr":"172.21.2.102:6835","nonce":1274216855}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":112}],"filesystems":[{"mdsmap":{"epoch":116,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:34:23.684779","modified":"2021-02-10 11:34:25.693957","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6770},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6770":{"gid":6770,"name":"b","rank":0,"incarnation":113,"state":"up:active","state_seq":3,"addr":"172.21.2.113:6833/3752249621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6832","nonce":3752249621},{"type":"v1","addr":"172.21.2.113:6833","nonce":3752249621}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[29],"metadata_pool":28,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":14}]} 2021-02-10T11:35:00.103 INFO:teuthology.orchestra.run.gibba002.stderr:dumped fsmap epoch 117 2021-02-10T11:35:00.104 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:00.102 7f0228858700 1 -- 172.21.2.102:0/2814683725 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f02080186c0 msgr2=0x7f020801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:00.104 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:00.102 7f0228858700 1 --2- 172.21.2.102:0/2814683725 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f02080186c0 0x7f020801abc0 crc :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:00.104 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:00.102 7f0228858700 1 -- 172.21.2.102:0/2814683725 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f02240ada90 msgr2=0x7f02240ade90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:00.104 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:00.102 7f0228858700 1 --2- 172.21.2.102:0/2814683725 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f02240ada90 0x7f02240ade90 secure :-1 s=READY pgs=2611 cs=0 l=1 rev1=1 rx=0x7f02140066a0 tx=0x7f021400b990).stop 2021-02-10T11:35:00.104 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:00.102 7f0228858700 1 -- 172.21.2.102:0/2814683725 shutdown_connections 2021-02-10T11:35:00.104 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:00.102 7f0228858700 1 --2- 172.21.2.102:0/2814683725 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f022409a740 0x7f0224097bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:00.105 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:00.102 7f0228858700 1 --2- 172.21.2.102:0/2814683725 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f02080186c0 0x7f020801abc0 unknown :-1 s=CLOSED pgs=1152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:00.105 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:00.102 7f0228858700 1 --2- 172.21.2.102:0/2814683725 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0224099d20 0x7f022409a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:00.105 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:00.102 7f0228858700 1 --2- 172.21.2.102:0/2814683725 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f02240ada90 0x7f02240ade90 unknown :-1 s=CLOSED pgs=2611 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:00.105 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:00.102 7f0228858700 1 -- 172.21.2.102:0/2814683725 >> 172.21.2.102:0/2814683725 conn(0x7f02240f0550 msgr2=0x7f0224191730 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:00.105 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:00.102 7f0228858700 1 -- 172.21.2.102:0/2814683725 shutdown_connections 2021-02-10T11:35:00.105 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:00.102 7f0228858700 1 -- 172.21.2.102:0/2814683725 wait complete. 2021-02-10T11:35:00.112 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs fail cephfs 2021-02-10T11:35:00.542 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:00.522 7fdf984ce700 1 Processor -- start 2021-02-10T11:35:00.542 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:00.522 7fdf984ce700 1 -- start start 2021-02-10T11:35:00.542 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:00.523 7fdf984ce700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fdf900d1c70 0x7fdf900f20a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:00.542 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:00.523 7fdf984ce700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf900f0b10 con 0x7fdf900d0340 2021-02-10T11:35:00.542 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:00.523 7fdf984ce700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf901b0be0 con 0x7fdf900d1870 2021-02-10T11:35:00.542 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:00.523 7fdf984ce700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf900e5ed0 con 0x7fdf900d1c70 2021-02-10T11:35:00.542 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:00.523 7fdf95ae9700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fdf900d0340 0x7fdf900ef640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:56980/0 (socket says 172.21.2.102:56980) 2021-02-10T11:35:00.542 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:00.523 7fdf95ae9700 1 -- 172.21.2.102:0/3182240553 learned_addr learned my addr 172.21.2.102:0/3182240553 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:35:00.543 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:00.523 7fdf952e8700 1 -- 172.21.2.102:0/3182240553 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2830814856 0 0) 0x7fdf800009b0 con 0x7fdf900d0340 2021-02-10T11:35:00.543 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:00.523 7fdf952e8700 1 -- 172.21.2.102:0/3182240553 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf7c002fe0 con 0x7fdf900d0340 2021-02-10T11:35:00.543 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:00.523 7fdf96aeb700 1 --2- 172.21.2.102:0/3182240553 >> v2:172.21.2.113:3300/0 conn(0x7fdf900d1c70 0x7fdf900f20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:00.543 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:00.524 7fdf952e8700 1 -- 172.21.2.102:0/3182240553 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 912823481 0 0) 0x7fdf80001010 con 0x7fdf900d0340 2021-02-10T11:35:00.543 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:00.524 7fdf952e8700 1 -- 172.21.2.102:0/3182240553 >> v2:172.21.2.113:3300/0 conn(0x7fdf900d1c70 msgr2=0x7fdf900f20a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:00.543 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:00.524 7fdf952e8700 1 --2- 172.21.2.102:0/3182240553 >> v2:172.21.2.113:3300/0 conn(0x7fdf900d1c70 0x7fdf900f20a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:00.543 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:00.524 7fdf952e8700 1 -- 172.21.2.102:0/3182240553 >> v1:172.21.2.113:6789/0 conn(0x7fdf900d1870 legacy=0x7fdf901b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:00.543 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:00.524 7fdf952e8700 1 -- 172.21.2.102:0/3182240553 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf900efe30 con 0x7fdf900d0340 2021-02-10T11:35:00.543 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:00.524 7fdf952e8700 1 -- 172.21.2.102:0/3182240553 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7fdf80001590 con 0x7fdf900d0340 2021-02-10T11:35:00.543 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:00.524 7fdf952e8700 1 -- 172.21.2.102:0/3182240553 >> v1:172.21.2.102:6789/0 conn(0x7fdf900d0340 legacy=0x7fdf900ef640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:00.543 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:00.524 7fdf952e8700 1 --2- 172.21.2.102:0/3182240553 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdf7c003af0 0x7fdf7c005f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:00.543 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:00.524 7fdf952e8700 1 --2- 172.21.2.102:0/3182240553 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdf7c0067a0 0x7fdf7c008ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:00.544 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:00.524 7fdf952e8700 1 --2- 172.21.2.102:0/3182240553 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fdf7c0095a0 0x7fdf7c00bac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:00.544 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:00.524 7fdf952e8700 1 -- 172.21.2.102:0/3182240553 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf7c00c2f0 con 0x7fdf7c0067a0 2021-02-10T11:35:00.544 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:00.524 7fdf952e8700 1 -- 172.21.2.102:0/3182240553 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf7c00c510 con 0x7fdf7c0095a0 2021-02-10T11:35:00.544 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:00.524 7fdf952e8700 1 -- 172.21.2.102:0/3182240553 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf7c050740 con 0x7fdf7c003af0 2021-02-10T11:35:00.544 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:00.524 7fdf95ae9700 1 --2- 172.21.2.102:0/3182240553 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdf7c0067a0 0x7fdf7c008ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:00.544 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:00.524 7fdf962ea700 1 --2- 172.21.2.102:0/3182240553 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdf7c003af0 0x7fdf7c005f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:00.544 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:00.524 7fdf96aeb700 1 --2- 172.21.2.102:0/3182240553 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fdf7c0095a0 0x7fdf7c00bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:00.544 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:00.524 7fdf95ae9700 1 -- 172.21.2.102:0/3182240553 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdf7c003af0 msgr2=0x7fdf7c005f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:00.544 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:00.524 7fdf95ae9700 1 --2- 172.21.2.102:0/3182240553 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdf7c003af0 0x7fdf7c005f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:00.544 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:00.524 7fdf95ae9700 1 -- 172.21.2.102:0/3182240553 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fdf7c0095a0 msgr2=0x7fdf7c00bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:00.544 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:00.524 7fdf95ae9700 1 --2- 172.21.2.102:0/3182240553 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fdf7c0095a0 0x7fdf7c00bac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:00.544 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:00.524 7fdf95ae9700 1 -- 172.21.2.102:0/3182240553 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf7c050970 con 0x7fdf7c0067a0 2021-02-10T11:35:00.544 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:00.524 7fdf96aeb700 1 --2- 172.21.2.102:0/3182240553 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fdf7c0095a0 0x7fdf7c00bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:35:00.545 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:00.524 7fdf96aeb700 1 -- 172.21.2.102:0/3182240553 reap_dead start 2021-02-10T11:35:00.545 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:00.525 7fdf95ae9700 1 --2- 172.21.2.102:0/3182240553 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdf7c0067a0 0x7fdf7c008ca0 secure :-1 s=READY pgs=2613 cs=0 l=1 rev1=1 rx=0x7fdf80007560 tx=0x7fdf80007d10).ready entity=mon.0 client_cookie=6a04babb90eda1b1 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:00.545 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:00.525 7fdf952e8700 1 -- 172.21.2.102:0/3182240553 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdf800009b0 con 0x7fdf7c0067a0 2021-02-10T11:35:00.545 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:00.525 7fdf952e8700 1 -- 172.21.2.102:0/3182240553 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf800013d0 con 0x7fdf7c0067a0 2021-02-10T11:35:00.545 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:00.525 7fdf952e8700 1 -- 172.21.2.102:0/3182240553 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdf8000a7a0 con 0x7fdf7c0067a0 2021-02-10T11:35:00.545 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:00.525 7fdf984ce700 1 -- 172.21.2.102:0/3182240553 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdf7c0067a0 msgr2=0x7fdf7c008ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:00.545 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:00.525 7fdf984ce700 1 --2- 172.21.2.102:0/3182240553 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdf7c0067a0 0x7fdf7c008ca0 secure :-1 s=READY pgs=2613 cs=0 l=1 rev1=1 rx=0x7fdf80007560 tx=0x7fdf80007d10).stop 2021-02-10T11:35:00.545 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:00.525 7fdf984ce700 1 -- 172.21.2.102:0/3182240553 shutdown_connections 2021-02-10T11:35:00.545 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:00.525 7fdf984ce700 1 --2- 172.21.2.102:0/3182240553 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdf7c0067a0 0x7fdf7c008ca0 unknown :-1 s=CLOSED pgs=2613 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:00.545 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:00.525 7fdf984ce700 1 -- 172.21.2.102:0/3182240553 >> 172.21.2.102:0/3182240553 conn(0x7fdf900ce450 msgr2=0x7fdf900d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:00.545 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:00.525 7fdf984ce700 1 -- 172.21.2.102:0/3182240553 shutdown_connections 2021-02-10T11:35:00.545 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:00.525 7fdf984ce700 1 -- 172.21.2.102:0/3182240553 wait complete. 2021-02-10T11:35:00.546 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:00.540 7fdf984ce700 1 Processor -- start 2021-02-10T11:35:00.546 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:00.540 7fdf984ce700 1 -- start start 2021-02-10T11:35:00.547 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:00.540 7fdf984ce700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdf901a7410 0x7fdf901a7810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:00.547 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:00.540 7fdf984ce700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fdf900ae040 0x7fdf900c1c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:00.547 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:00.540 7fdf984ce700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdf901a7e30 0x7fdf900c2260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:00.547 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:00.540 7fdf984ce700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf901a8230 con 0x7fdf901a7410 2021-02-10T11:35:00.547 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:00.540 7fdf984ce700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf900ae5e0 con 0x7fdf900ae040 2021-02-10T11:35:00.547 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:00.540 7fdf984ce700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf900c2880 con 0x7fdf901a7e30 2021-02-10T11:35:00.548 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:00.541 7fdf962ea700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdf901a7410 0x7fdf901a7810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:00.548 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:00.541 7fdf962ea700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdf901a7410 0x7fdf901a7810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:36536/0 (socket says 172.21.2.102:36536) 2021-02-10T11:35:00.548 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:00.541 7fdf962ea700 1 -- 172.21.2.102:0/2601193501 learned_addr learned my addr 172.21.2.102:0/2601193501 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:35:00.548 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:00.541 7fdf95ae9700 1 --2- 172.21.2.102:0/2601193501 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fdf900ae040 0x7fdf900c1c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:00.548 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:00.541 7fdf96aeb700 1 --2- 172.21.2.102:0/2601193501 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdf901a7e30 0x7fdf900c2260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:00.548 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:00.541 7fdf962ea700 1 -- 172.21.2.102:0/2601193501 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdf901a7e30 msgr2=0x7fdf900c2260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:00.548 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:00.541 7fdf962ea700 1 --2- 172.21.2.102:0/2601193501 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdf901a7e30 0x7fdf900c2260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:00.548 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:00.541 7fdf962ea700 1 -- 172.21.2.102:0/2601193501 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fdf900ae040 msgr2=0x7fdf900c1c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:00.548 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:00.541 7fdf962ea700 1 --2- 172.21.2.102:0/2601193501 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fdf900ae040 0x7fdf900c1c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:00.548 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:00.541 7fdf962ea700 1 -- 172.21.2.102:0/2601193501 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf900c2aa0 con 0x7fdf901a7410 2021-02-10T11:35:00.548 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:00.541 7fdf962ea700 1 --2- 172.21.2.102:0/2601193501 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdf901a7410 0x7fdf901a7810 secure :-1 s=READY pgs=2614 cs=0 l=1 rev1=1 rx=0x7fdf8c00b590 tx=0x7fdf8c00b930).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:00.548 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:00.541 7fdf86ffd700 1 -- 172.21.2.102:0/2601193501 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdf8c00bd20 con 0x7fdf901a7410 2021-02-10T11:35:00.549 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:00.541 7fdf86ffd700 1 -- 172.21.2.102:0/2601193501 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf8c000ca0 con 0x7fdf901a7410 2021-02-10T11:35:00.549 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:00.541 7fdf984ce700 1 -- 172.21.2.102:0/2601193501 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf900c2aa0 con 0x7fdf901a7410 2021-02-10T11:35:00.549 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:00.541 7fdf86ffd700 1 -- 172.21.2.102:0/2601193501 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdf8c0022b0 con 0x7fdf901a7410 2021-02-10T11:35:00.549 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:00.541 7fdf984ce700 1 -- 172.21.2.102:0/2601193501 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf900c2aa0 con 0x7fdf901a7410 2021-02-10T11:35:00.549 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:00.542 7fdf86ffd700 1 -- 172.21.2.102:0/2601193501 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fdf8c0052e0 con 0x7fdf901a7410 2021-02-10T11:35:00.549 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:00.542 7fdf86ffd700 1 --2- 172.21.2.102:0/2601193501 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fdf780186d0 0x7fdf7801abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:00.549 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:00.542 7fdf86ffd700 1 -- 172.21.2.102:0/2601193501 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(137..137 src has 1..137) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fdf8c032640 con 0x7fdf901a7410 2021-02-10T11:35:00.549 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:00.542 7fdf95ae9700 1 --2- 172.21.2.102:0/2601193501 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fdf780186d0 0x7fdf7801abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:00.549 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:00.542 7fdf95ae9700 1 --2- 172.21.2.102:0/2601193501 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fdf780186d0 0x7fdf7801abd0 crc :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:00.575 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:00.573 7fdf984ce700 1 -- 172.21.2.102:0/2601193501 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf900ebc50 con 0x7fdf901a7410 2021-02-10T11:35:00.577 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:00.576 7fdf86ffd700 1 -- 172.21.2.102:0/2601193501 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fdf8c006bb0 con 0x7fdf901a7410 2021-02-10T11:35:00.705 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:00.703 7fdf984ce700 1 -- 172.21.2.102:0/2601193501 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fdf900f0b10 con 0x7fdf901a7410 2021-02-10T11:35:01.481 INFO:tasks.ceph.mon.a.gibba002.stderr:2021-02-10 11:35:01.479 7f8dd2755700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-02-10T11:35:01.487 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:01.485 7fdf86ffd700 1 -- 172.21.2.102:0/2601193501 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v118) v1 ==== 161+0+0 (secure 0 0 0) 0x7fdf8c0052e0 con 0x7fdf901a7410 2021-02-10T11:35:01.524 INFO:teuthology.orchestra.run.gibba002.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2021-02-10T11:35:01.526 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:01.524 7fdf984ce700 1 -- 172.21.2.102:0/2601193501 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fdf780186d0 msgr2=0x7fdf7801abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:01.526 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:01.524 7fdf984ce700 1 --2- 172.21.2.102:0/2601193501 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fdf780186d0 0x7fdf7801abd0 crc :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:01.526 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:01.524 7fdf984ce700 1 -- 172.21.2.102:0/2601193501 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdf901a7410 msgr2=0x7fdf901a7810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:01.526 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:01.524 7fdf984ce700 1 --2- 172.21.2.102:0/2601193501 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdf901a7410 0x7fdf901a7810 secure :-1 s=READY pgs=2614 cs=0 l=1 rev1=1 rx=0x7fdf8c00b590 tx=0x7fdf8c00b930).stop 2021-02-10T11:35:01.527 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:01.524 7fdf984ce700 1 -- 172.21.2.102:0/2601193501 shutdown_connections 2021-02-10T11:35:01.527 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:01.524 7fdf984ce700 1 --2- 172.21.2.102:0/2601193501 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdf901a7e30 0x7fdf900c2260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:01.527 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:01.524 7fdf984ce700 1 --2- 172.21.2.102:0/2601193501 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fdf780186d0 0x7fdf7801abd0 unknown :-1 s=CLOSED pgs=1153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:01.527 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:01.524 7fdf984ce700 1 --2- 172.21.2.102:0/2601193501 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fdf900ae040 0x7fdf900c1c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:01.527 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:01.524 7fdf984ce700 1 --2- 172.21.2.102:0/2601193501 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdf901a7410 0x7fdf901a7810 unknown :-1 s=CLOSED pgs=2614 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:01.527 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:01.524 7fdf984ce700 1 -- 172.21.2.102:0/2601193501 >> 172.21.2.102:0/2601193501 conn(0x7fdf900aeb90 msgr2=0x7fdf900af100 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:01.527 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:01.524 7fdf984ce700 1 -- 172.21.2.102:0/2601193501 shutdown_connections 2021-02-10T11:35:01.527 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:01.525 7fdf984ce700 1 -- 172.21.2.102:0/2601193501 wait complete. 2021-02-10T11:35:01.530 INFO:tasks.ceph.mds.b.gibba013.stdout:starting mds.b at 2021-02-10T11:35:01.534 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:35:01.964 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:01.945 7f1f1dd76700 1 Processor -- start 2021-02-10T11:35:01.964 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:01.945 7f1f1dd76700 1 -- start start 2021-02-10T11:35:01.964 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:01.946 7f1f1dd76700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1f180d1870 0x7f1f181b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:01.964 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:01.946 7f1f1dd76700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f1f181b0a10 0x7f1f180cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:01.964 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:01.946 7f1f1dd76700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f180f0b10 con 0x7f1f180d1c70 2021-02-10T11:35:01.964 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:01.946 7f1f1dd76700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f180fe230 con 0x7f1f181b0a10 2021-02-10T11:35:01.964 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:01.946 7f1f1dd76700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f180e5ed0 con 0x7f1f180d1870 2021-02-10T11:35:01.964 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:01.946 7f1f17fff700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f1f180d1c70 0x7f1f180cf950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:56996/0 (socket says 172.21.2.102:56996) 2021-02-10T11:35:01.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:01.946 7f1f17fff700 1 -- 172.21.2.102:0/247485131 learned_addr learned my addr 172.21.2.102:0/247485131 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:35:01.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:01.946 7f1f16ffd700 1 --2- 172.21.2.102:0/247485131 >> v2:172.21.2.113:3300/0 conn(0x7f1f181b0a10 0x7f1f180cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:01.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:01.946 7f1f177fe700 1 --2- 172.21.2.102:0/247485131 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1f180d1870 0x7f1f181b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:01.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:01.946 7f1f167fc700 1 -- 172.21.2.102:0/247485131 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1479523658 0 0) 0x7f1f10001500 con 0x7f1f180d1c70 2021-02-10T11:35:01.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:01.946 7f1f167fc700 1 -- 172.21.2.102:0/247485131 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f04002fe0 con 0x7f1f180d1c70 2021-02-10T11:35:01.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:01.947 7f1f177fe700 1 -- 172.21.2.102:0/247485131 >> v2:172.21.2.113:3300/0 conn(0x7f1f181b0a10 msgr2=0x7f1f180cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:01.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:01.947 7f1f177fe700 1 --2- 172.21.2.102:0/247485131 >> v2:172.21.2.113:3300/0 conn(0x7f1f181b0a10 0x7f1f180cf330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:01.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:01.947 7f1f177fe700 1 -- 172.21.2.102:0/247485131 >> v1:172.21.2.102:6789/0 conn(0x7f1f180d1c70 legacy=0x7f1f180cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:01.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:01.947 7f1f177fe700 1 -- 172.21.2.102:0/247485131 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f180f0650 con 0x7f1f180d1870 2021-02-10T11:35:01.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:01.947 7f1f177fe700 1 --2- 172.21.2.102:0/247485131 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1f180d1870 0x7f1f181b03f0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 rx=0x7f1f08006650 tx=0x7f1f08009a80).ready entity=mon.2 client_cookie=5713dae2d4c0ce89 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:01.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:01.947 7f1f167fc700 1 -- 172.21.2.102:0/247485131 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1f080009b0 con 0x7f1f180d1870 2021-02-10T11:35:01.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:01.947 7f1f167fc700 1 -- 172.21.2.102:0/247485131 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f0800c0c0 con 0x7f1f180d1870 2021-02-10T11:35:01.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:01.947 7f1f167fc700 1 -- 172.21.2.102:0/247485131 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1f0800c5e0 con 0x7f1f180d1870 2021-02-10T11:35:01.966 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:01.947 7f1f1dd76700 1 -- 172.21.2.102:0/247485131 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1f180d1870 msgr2=0x7f1f181b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:01.966 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:01.947 7f1f1dd76700 1 --2- 172.21.2.102:0/247485131 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1f180d1870 0x7f1f181b03f0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 rx=0x7f1f08006650 tx=0x7f1f08009a80).stop 2021-02-10T11:35:01.966 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:01.948 7f1f1dd76700 1 -- 172.21.2.102:0/247485131 shutdown_connections 2021-02-10T11:35:01.966 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:01.948 7f1f1dd76700 1 --2- 172.21.2.102:0/247485131 >> v2:172.21.2.113:3300/0 conn(0x7f1f181b0a10 0x7f1f180cf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:01.966 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:01.948 7f1f1dd76700 1 --2- 172.21.2.102:0/247485131 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1f180d1870 0x7f1f181b03f0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:01.966 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:01.948 7f1f1dd76700 1 -- 172.21.2.102:0/247485131 >> 172.21.2.102:0/247485131 conn(0x7f1f180ce450 msgr2=0x7f1f180d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:01.966 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:01.948 7f1f1dd76700 1 -- 172.21.2.102:0/247485131 shutdown_connections 2021-02-10T11:35:01.966 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:01.948 7f1f1dd76700 1 -- 172.21.2.102:0/247485131 wait complete. 2021-02-10T11:35:01.968 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:01.962 7f1f1dd76700 1 Processor -- start 2021-02-10T11:35:01.968 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:01.963 7f1f1dd76700 1 -- start start 2021-02-10T11:35:01.968 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:01.963 7f1f1dd76700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1f180b54e0 0x7f1f180b58e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:01.968 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:01.963 7f1f1dd76700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1f180b5f00 0x7f1f180ee760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:01.968 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:01.963 7f1f1dd76700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1f180eed80 0x7f1f180ef180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:01.968 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:01.963 7f1f1dd76700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f180ee260 con 0x7f1f180eed80 2021-02-10T11:35:01.968 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:01.963 7f1f1dd76700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f18091380 con 0x7f1f180b54e0 2021-02-10T11:35:01.968 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:01.963 7f1f1dd76700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f18090e80 con 0x7f1f180b5f00 2021-02-10T11:35:01.968 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:01.963 7f1f17fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1f180eed80 0x7f1f180ef180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:01.968 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:01.963 7f1f17fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1f180eed80 0x7f1f180ef180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:36548/0 (socket says 172.21.2.102:36548) 2021-02-10T11:35:01.968 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:01.963 7f1f17fff700 1 -- 172.21.2.102:0/1222477617 learned_addr learned my addr 172.21.2.102:0/1222477617 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:35:01.969 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:01.964 7f1f17fff700 1 -- 172.21.2.102:0/1222477617 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1f180b5f00 msgr2=0x7f1f180ee760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:01.969 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:01.964 7f1f17fff700 1 --2- 172.21.2.102:0/1222477617 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1f180b5f00 0x7f1f180ee760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:01.969 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:01.964 7f1f17fff700 1 -- 172.21.2.102:0/1222477617 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1f180b54e0 msgr2=0x7f1f180b58e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:01.969 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:01.964 7f1f17fff700 1 --2- 172.21.2.102:0/1222477617 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1f180b54e0 0x7f1f180b58e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:01.969 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:01.964 7f1f17fff700 1 -- 172.21.2.102:0/1222477617 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f180ede40 con 0x7f1f180eed80 2021-02-10T11:35:01.969 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:01.964 7f1f17fff700 1 --2- 172.21.2.102:0/1222477617 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1f180eed80 0x7f1f180ef180 secure :-1 s=READY pgs=2617 cs=0 l=1 rev1=1 rx=0x7f1f10001c10 tx=0x7f1f1000aca0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:01.969 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:01.964 7f1f0ffff700 1 -- 172.21.2.102:0/1222477617 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1f100070d0 con 0x7f1f180eed80 2021-02-10T11:35:01.969 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:01.964 7f1f0ffff700 1 -- 172.21.2.102:0/1222477617 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f10007350 con 0x7f1f180eed80 2021-02-10T11:35:01.969 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:01.964 7f1f0ffff700 1 -- 172.21.2.102:0/1222477617 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1f100049a0 con 0x7f1f180eed80 2021-02-10T11:35:01.969 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:01.964 7f1f1dd76700 1 -- 172.21.2.102:0/1222477617 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f180ede40 con 0x7f1f180eed80 2021-02-10T11:35:01.969 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:01.964 7f1f1dd76700 1 -- 172.21.2.102:0/1222477617 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f180efa90 con 0x7f1f180eed80 2021-02-10T11:35:01.969 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:01.965 7f1f0ffff700 1 -- 172.21.2.102:0/1222477617 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1f10004720 con 0x7f1f180eed80 2021-02-10T11:35:01.970 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:01.965 7f1f0ffff700 1 --2- 172.21.2.102:0/1222477617 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1efc0186c0 0x7f1efc01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:01.970 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:01.965 7f1f0ffff700 1 -- 172.21.2.102:0/1222477617 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(138..138 src has 1..138) v4 ==== 5659+0+0 (secure 0 0 0) 0x7f1f10032940 con 0x7f1f180eed80 2021-02-10T11:35:01.970 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:01.965 7f1f177fe700 1 --2- 172.21.2.102:0/1222477617 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1efc0186c0 0x7f1efc01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:01.970 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:01.965 7f1f177fe700 1 --2- 172.21.2.102:0/1222477617 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1efc0186c0 0x7f1efc01abc0 crc :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:01.998 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:01.996 7f1f1dd76700 1 -- 172.21.2.102:0/1222477617 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f180f0b10 con 0x7f1f180eed80 2021-02-10T11:35:02.000 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:01.998 7f1f0ffff700 1 -- 172.21.2.102:0/1222477617 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f1f10000f40 con 0x7f1f180eed80 2021-02-10T11:35:02.127 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:02.125 7f1f1dd76700 1 -- 172.21.2.102:0/1222477617 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1f180f0b10 con 0x7f1f180eed80 2021-02-10T11:35:02.128 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:02.126 7f1f0ffff700 1 -- 172.21.2.102:0/1222477617 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 118 v118) v1 ==== 95+0+1763 (secure 0 0 0) 0x7f1f10004720 con 0x7f1f180eed80 2021-02-10T11:35:02.128 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:35:02.128 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":118,"default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":11466,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.102:6835/1274216855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6834","nonce":1274216855},{"type":"v1","addr":"172.21.2.102:6835","nonce":1274216855}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":112}],"filesystems":[{"mdsmap":{"epoch":118,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:34:23.684779","modified":"2021-02-10 11:35:01.480394","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":138,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[29],"metadata_pool":28,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":14}]} 2021-02-10T11:35:02.128 INFO:teuthology.orchestra.run.gibba002.stderr:dumped fsmap epoch 118 2021-02-10T11:35:02.129 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:02.128 7f1f1dd76700 1 -- 172.21.2.102:0/1222477617 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1efc0186c0 msgr2=0x7f1efc01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:02.130 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:02.128 7f1f1dd76700 1 --2- 172.21.2.102:0/1222477617 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1efc0186c0 0x7f1efc01abc0 crc :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:02.130 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:02.128 7f1f1dd76700 1 -- 172.21.2.102:0/1222477617 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1f180eed80 msgr2=0x7f1f180ef180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:02.130 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:02.128 7f1f1dd76700 1 --2- 172.21.2.102:0/1222477617 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1f180eed80 0x7f1f180ef180 secure :-1 s=READY pgs=2617 cs=0 l=1 rev1=1 rx=0x7f1f10001c10 tx=0x7f1f1000aca0).stop 2021-02-10T11:35:02.130 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:02.128 7f1f1dd76700 1 -- 172.21.2.102:0/1222477617 shutdown_connections 2021-02-10T11:35:02.130 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:02.128 7f1f1dd76700 1 --2- 172.21.2.102:0/1222477617 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1f180eed80 0x7f1f180ef180 unknown :-1 s=CLOSED pgs=2617 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:02.130 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:02.128 7f1f1dd76700 1 --2- 172.21.2.102:0/1222477617 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1f180b5f00 0x7f1f180ee760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:02.130 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:02.128 7f1f1dd76700 1 --2- 172.21.2.102:0/1222477617 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1efc0186c0 0x7f1efc01abc0 unknown :-1 s=CLOSED pgs=1154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:02.130 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:02.128 7f1f1dd76700 1 --2- 172.21.2.102:0/1222477617 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1f180b54e0 0x7f1f180b58e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:02.130 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:02.128 7f1f1dd76700 1 -- 172.21.2.102:0/1222477617 >> 172.21.2.102:0/1222477617 conn(0x7f1f180c40a0 msgr2=0x7f1f1809e050 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:02.131 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:02.128 7f1f1dd76700 1 -- 172.21.2.102:0/1222477617 shutdown_connections 2021-02-10T11:35:02.131 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:02.128 7f1f1dd76700 1 -- 172.21.2.102:0/1222477617 wait complete. 2021-02-10T11:35:02.138 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2021-02-10T11:35:02.573 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:02.554 7f45e175d700 1 Processor -- start 2021-02-10T11:35:02.573 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:02.554 7f45e175d700 1 -- start start 2021-02-10T11:35:02.573 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:02.555 7f45e175d700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f45dc0d1c70 0x7f45dc0f20a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:02.573 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:02.555 7f45e175d700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45dc0f0b10 con 0x7f45dc0d0340 2021-02-10T11:35:02.573 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:02.555 7f45e175d700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45dc1b0be0 con 0x7f45dc0d1870 2021-02-10T11:35:02.573 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:02.555 7f45e175d700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45dc0e5ed0 con 0x7f45dc0d1c70 2021-02-10T11:35:02.573 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:02.555 7f45dbd75700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f45dc0d1c70 0x7f45dc0f20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:02.573 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:02.555 7f45dbd75700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f45dc0d1c70 0x7f45dc0f20a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:36558/0 (socket says 172.21.2.102:36558) 2021-02-10T11:35:02.573 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:02.555 7f45dbd75700 1 -- 172.21.2.102:0/3255791005 learned_addr learned my addr 172.21.2.102:0/3255791005 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:35:02.574 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:02.555 7f45dad73700 1 --1- 172.21.2.102:0/3255791005 >> v1:172.21.2.102:6789/0 conn(0x7f45dc0d0340 0x7f45dc0ef640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:57010/0 (socket says 172.21.2.102:57010) 2021-02-10T11:35:02.574 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:02.555 7f45dbd75700 1 -- 172.21.2.102:0/3255791005 >> v1:172.21.2.113:6789/0 conn(0x7f45dc0d1870 legacy=0x7f45dc1b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:02.574 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:02.555 7f45dbd75700 1 -- 172.21.2.102:0/3255791005 >> v1:172.21.2.102:6789/0 conn(0x7f45dc0d0340 legacy=0x7f45dc0ef640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:02.574 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:02.555 7f45dbd75700 1 -- 172.21.2.102:0/3255791005 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45dc0efe30 con 0x7f45dc0d1c70 2021-02-10T11:35:02.574 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:02.555 7f45dbd75700 1 --2- 172.21.2.102:0/3255791005 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f45dc0d1c70 0x7f45dc0f20a0 secure :-1 s=READY pgs=2619 cs=0 l=1 rev1=1 rx=0x7f45d4006f00 tx=0x7f45d4009f10).ready entity=mon.0 client_cookie=96afe17d8f3ccce3 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:02.574 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:02.556 7f45da572700 1 -- 172.21.2.102:0/3255791005 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f45d400b3b0 con 0x7f45dc0d1c70 2021-02-10T11:35:02.574 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:02.556 7f45da572700 1 -- 172.21.2.102:0/3255791005 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45d400b730 con 0x7f45dc0d1c70 2021-02-10T11:35:02.574 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:02.556 7f45da572700 1 -- 172.21.2.102:0/3255791005 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f45d400bc50 con 0x7f45dc0d1c70 2021-02-10T11:35:02.574 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:02.556 7f45e175d700 1 -- 172.21.2.102:0/3255791005 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f45dc0d1c70 msgr2=0x7f45dc0f20a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:02.574 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:02.556 7f45e175d700 1 --2- 172.21.2.102:0/3255791005 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f45dc0d1c70 0x7f45dc0f20a0 secure :-1 s=READY pgs=2619 cs=0 l=1 rev1=1 rx=0x7f45d4006f00 tx=0x7f45d4009f10).stop 2021-02-10T11:35:02.574 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:02.556 7f45e175d700 1 -- 172.21.2.102:0/3255791005 shutdown_connections 2021-02-10T11:35:02.574 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:02.556 7f45e175d700 1 --2- 172.21.2.102:0/3255791005 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f45dc0d1c70 0x7f45dc0f20a0 unknown :-1 s=CLOSED pgs=2619 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:02.575 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:02.556 7f45e175d700 1 -- 172.21.2.102:0/3255791005 >> 172.21.2.102:0/3255791005 conn(0x7f45dc0ce450 msgr2=0x7f45dc0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:02.575 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:02.556 7f45e175d700 1 -- 172.21.2.102:0/3255791005 shutdown_connections 2021-02-10T11:35:02.575 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:02.556 7f45e175d700 1 -- 172.21.2.102:0/3255791005 wait complete. 2021-02-10T11:35:02.575 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:02.571 7f45e175d700 1 Processor -- start 2021-02-10T11:35:02.576 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:02.571 7f45e175d700 1 -- start start 2021-02-10T11:35:02.576 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:02.571 7f45e175d700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f45dc0c1970 0x7f45dc0b3d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:02.576 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:02.571 7f45e175d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f45dc0b3320 0x7f45dc0c3b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:02.576 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:02.571 7f45e175d700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f45dc0b3720 0x7f45dc0a6290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:02.577 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:02.571 7f45e175d700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45dc0b44d0 con 0x7f45dc0b3320 2021-02-10T11:35:02.577 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:02.571 7f45e175d700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45dc0a5d10 con 0x7f45dc0b3720 2021-02-10T11:35:02.577 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:02.572 7f45e175d700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45dc0b2ff0 con 0x7f45dc0c1970 2021-02-10T11:35:02.577 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:02.572 7f45dad73700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f45dc0b3320 0x7f45dc0c3b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:02.577 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:02.572 7f45dad73700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f45dc0b3320 0x7f45dc0c3b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:36562/0 (socket says 172.21.2.102:36562) 2021-02-10T11:35:02.577 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:02.572 7f45dad73700 1 -- 172.21.2.102:0/2192704270 learned_addr learned my addr 172.21.2.102:0/2192704270 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:35:02.577 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:02.572 7f45dad73700 1 -- 172.21.2.102:0/2192704270 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f45dc0c1970 msgr2=0x7f45dc0b3d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:02.577 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:02.572 7f45dad73700 1 --2- 172.21.2.102:0/2192704270 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f45dc0c1970 0x7f45dc0b3d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:02.577 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:02.572 7f45dad73700 1 -- 172.21.2.102:0/2192704270 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f45dc0b3720 msgr2=0x7f45dc0a6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:02.577 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:02.572 7f45dad73700 1 --2- 172.21.2.102:0/2192704270 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f45dc0b3720 0x7f45dc0a6290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:02.577 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:02.572 7f45dad73700 1 -- 172.21.2.102:0/2192704270 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45dc0c4160 con 0x7f45dc0b3320 2021-02-10T11:35:02.577 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:02.572 7f45dad73700 1 --2- 172.21.2.102:0/2192704270 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f45dc0b3320 0x7f45dc0c3b40 secure :-1 s=READY pgs=2620 cs=0 l=1 rev1=1 rx=0x7f45d0006f00 tx=0x7f45d000b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:02.578 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:02.572 7f45c3fff700 1 -- 172.21.2.102:0/2192704270 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f45d0006020 con 0x7f45dc0b3320 2021-02-10T11:35:02.578 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:02.572 7f45c3fff700 1 -- 172.21.2.102:0/2192704270 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45d00062a0 con 0x7f45dc0b3320 2021-02-10T11:35:02.578 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:02.572 7f45e175d700 1 -- 172.21.2.102:0/2192704270 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45dc0c4160 con 0x7f45dc0b3320 2021-02-10T11:35:02.578 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:02.572 7f45c3fff700 1 -- 172.21.2.102:0/2192704270 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f45d0004620 con 0x7f45dc0b3320 2021-02-10T11:35:02.578 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:02.572 7f45e175d700 1 -- 172.21.2.102:0/2192704270 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45dc0c4160 con 0x7f45dc0b3320 2021-02-10T11:35:02.578 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:02.573 7f45c3fff700 1 -- 172.21.2.102:0/2192704270 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f45d000a000 con 0x7f45dc0b3320 2021-02-10T11:35:02.578 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:02.573 7f45c3fff700 1 --2- 172.21.2.102:0/2192704270 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f45bc0186c0 0x7f45bc01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:02.579 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:02.573 7f45c3fff700 1 -- 172.21.2.102:0/2192704270 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(138..138 src has 1..138) v4 ==== 5659+0+0 (secure 0 0 0) 0x7f45d0032590 con 0x7f45dc0b3320 2021-02-10T11:35:02.579 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:02.573 7f45db574700 1 --2- 172.21.2.102:0/2192704270 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f45bc0186c0 0x7f45bc01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:02.579 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:02.573 7f45db574700 1 --2- 172.21.2.102:0/2192704270 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f45bc0186c0 0x7f45bc01abc0 crc :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:02.607 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:02.605 7f45e175d700 1 -- 172.21.2.102:0/2192704270 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45dc0a6e90 con 0x7f45dc0b3320 2021-02-10T11:35:02.609 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:02.607 7f45c3fff700 1 -- 172.21.2.102:0/2192704270 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f45d0020f60 con 0x7f45dc0b3320 2021-02-10T11:35:02.738 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:02.736 7f45e175d700 1 -- 172.21.2.102:0/2192704270 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f45dc0a6e90 con 0x7f45dc0b3320 2021-02-10T11:35:03.497 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:03.495 7f45c3fff700 1 -- 172.21.2.102:0/2192704270 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v119) v1 ==== 104+0+0 (secure 0 0 0) 0x7f45d0006380 con 0x7f45dc0b3320 2021-02-10T11:35:03.510 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:03.508 7f45e175d700 1 -- 172.21.2.102:0/2192704270 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f45bc0186c0 msgr2=0x7f45bc01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:03.510 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:03.508 7f45e175d700 1 --2- 172.21.2.102:0/2192704270 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f45bc0186c0 0x7f45bc01abc0 crc :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:03.510 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:03.508 7f45e175d700 1 -- 172.21.2.102:0/2192704270 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f45dc0b3320 msgr2=0x7f45dc0c3b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:03.510 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:03.508 7f45e175d700 1 --2- 172.21.2.102:0/2192704270 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f45dc0b3320 0x7f45dc0c3b40 secure :-1 s=READY pgs=2620 cs=0 l=1 rev1=1 rx=0x7f45d0006f00 tx=0x7f45d000b940).stop 2021-02-10T11:35:03.510 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:03.508 7f45e175d700 1 -- 172.21.2.102:0/2192704270 shutdown_connections 2021-02-10T11:35:03.510 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:03.508 7f45e175d700 1 --2- 172.21.2.102:0/2192704270 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f45bc0186c0 0x7f45bc01abc0 unknown :-1 s=CLOSED pgs=1155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:03.511 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:03.508 7f45e175d700 1 --2- 172.21.2.102:0/2192704270 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f45dc0b3720 0x7f45dc0a6290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:03.511 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:03.508 7f45e175d700 1 --2- 172.21.2.102:0/2192704270 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f45dc0b3320 0x7f45dc0c3b40 unknown :-1 s=CLOSED pgs=2620 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:03.511 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:03.508 7f45e175d700 1 --2- 172.21.2.102:0/2192704270 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f45dc0c1970 0x7f45dc0b3d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:03.511 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:03.508 7f45e175d700 1 -- 172.21.2.102:0/2192704270 >> 172.21.2.102:0/2192704270 conn(0x7f45dc0b7970 msgr2=0x7f45dc08a420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:03.511 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:03.508 7f45e175d700 1 -- 172.21.2.102:0/2192704270 shutdown_connections 2021-02-10T11:35:03.511 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:03.509 7f45e175d700 1 -- 172.21.2.102:0/2192704270 wait complete. 2021-02-10T11:35:03.518 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool delete cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2021-02-10T11:35:03.951 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:03.932 7f9612d16700 1 Processor -- start 2021-02-10T11:35:03.951 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:03.932 7f9612d16700 1 -- start start 2021-02-10T11:35:03.951 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:03.933 7f9612d16700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f960c0d1c70 0x7f960c0cef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:03.951 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:03.933 7f9612d16700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f960c0d0170 con 0x7f960c0cf580 2021-02-10T11:35:03.951 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:03.933 7f9612d16700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f960c0fe1c0 con 0x7f960c0d1870 2021-02-10T11:35:03.951 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:03.933 7f9612d16700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f960c1b0cb0 con 0x7f960c0d1c70 2021-02-10T11:35:03.951 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:03.933 7f960bfff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f960c0d1c70 0x7f960c0cef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:03.951 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:03.933 7f9611333700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f960c0cf580 0x7f960c0cf980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:57026/0 (socket says 172.21.2.102:57026) 2021-02-10T11:35:03.951 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:03.933 7f9611333700 1 -- 172.21.2.102:0/3854544259 learned_addr learned my addr 172.21.2.102:0/3854544259 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:35:03.951 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:03.933 7f960bfff700 1 -- 172.21.2.102:0/3854544259 >> v1:172.21.2.113:6789/0 conn(0x7f960c0d1870 legacy=0x7f960c1b04c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:35:03.952 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:03.933 7f960bfff700 1 -- 172.21.2.102:0/3854544259 >> v1:172.21.2.102:6789/0 conn(0x7f960c0cf580 legacy=0x7f960c0cf980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:03.952 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:03.933 7f960bfff700 1 -- 172.21.2.102:0/3854544259 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f960c109a60 con 0x7f960c0d1c70 2021-02-10T11:35:03.952 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:03.934 7f960bfff700 1 --2- 172.21.2.102:0/3854544259 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f960c0d1c70 0x7f960c0cef60 secure :-1 s=READY pgs=2622 cs=0 l=1 rev1=1 rx=0x7f95fc006650 tx=0x7f95fc009a80).ready entity=mon.0 client_cookie=4a68fcd1bbb99bfb server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:03.952 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:03.934 7f960b7fe700 1 -- 172.21.2.102:0/3854544259 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f95fc0009b0 con 0x7f960c0d1c70 2021-02-10T11:35:03.952 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:03.934 7f960b7fe700 1 -- 172.21.2.102:0/3854544259 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95fc00c0c0 con 0x7f960c0d1c70 2021-02-10T11:35:03.952 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:03.934 7f960b7fe700 1 -- 172.21.2.102:0/3854544259 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f95fc00c5e0 con 0x7f960c0d1c70 2021-02-10T11:35:03.952 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:03.934 7f9612d16700 1 -- 172.21.2.102:0/3854544259 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f960c0d1c70 msgr2=0x7f960c0cef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:03.952 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:03.934 7f9612d16700 1 --2- 172.21.2.102:0/3854544259 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f960c0d1c70 0x7f960c0cef60 secure :-1 s=READY pgs=2622 cs=0 l=1 rev1=1 rx=0x7f95fc006650 tx=0x7f95fc009a80).stop 2021-02-10T11:35:03.952 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:03.934 7f9612d16700 1 -- 172.21.2.102:0/3854544259 shutdown_connections 2021-02-10T11:35:03.952 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:03.934 7f9612d16700 1 --2- 172.21.2.102:0/3854544259 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f960c0d1c70 0x7f960c0cef60 unknown :-1 s=CLOSED pgs=2622 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:03.952 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:03.934 7f9612d16700 1 -- 172.21.2.102:0/3854544259 >> 172.21.2.102:0/3854544259 conn(0x7f960c0ce420 msgr2=0x7f960c0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:03.952 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:03.935 7f9612d16700 1 -- 172.21.2.102:0/3854544259 shutdown_connections 2021-02-10T11:35:03.953 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:03.935 7f9612d16700 1 -- 172.21.2.102:0/3854544259 wait complete. 2021-02-10T11:35:03.954 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:03.949 7f9612d16700 1 Processor -- start 2021-02-10T11:35:03.954 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:03.949 7f9612d16700 1 -- start start 2021-02-10T11:35:03.954 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:03.950 7f9612d16700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f960c09c190 0x7f960c09cde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:03.954 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:03.950 7f9612d16700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f960c09d400 0x7f960c09d800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:03.954 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:03.950 7f9612d16700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f960c09de20 0x7f960c08b700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:03.954 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:03.950 7f9612d16700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f960c09c590 con 0x7f960c09de20 2021-02-10T11:35:03.954 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:03.950 7f9612d16700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f960c0b6d40 con 0x7f960c09d400 2021-02-10T11:35:03.954 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:03.950 7f9612d16700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f960c08bd20 con 0x7f960c09c190 2021-02-10T11:35:03.955 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:03.950 7f9611333700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f960c09de20 0x7f960c08b700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:03.955 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:03.950 7f9611333700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f960c09de20 0x7f960c08b700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:36576/0 (socket says 172.21.2.102:36576) 2021-02-10T11:35:03.955 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:03.950 7f9611333700 1 -- 172.21.2.102:0/3681408275 learned_addr learned my addr 172.21.2.102:0/3681408275 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:35:03.955 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:03.950 7f9611333700 1 -- 172.21.2.102:0/3681408275 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f960c09c190 msgr2=0x7f960c09cde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:03.955 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:03.950 7f9611333700 1 --2- 172.21.2.102:0/3681408275 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f960c09c190 0x7f960c09cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:03.955 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:03.950 7f9611333700 1 -- 172.21.2.102:0/3681408275 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f960c09d400 msgr2=0x7f960c09d800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:03.955 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:03.950 7f9611333700 1 --2- 172.21.2.102:0/3681408275 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f960c09d400 0x7f960c09d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:03.955 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:03.950 7f9611333700 1 -- 172.21.2.102:0/3681408275 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f960c08bf40 con 0x7f960c09de20 2021-02-10T11:35:03.955 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:03.951 7f9611333700 1 --2- 172.21.2.102:0/3681408275 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f960c09de20 0x7f960c08b700 secure :-1 s=READY pgs=2623 cs=0 l=1 rev1=1 rx=0x7f960400a940 tx=0x7f960400ace0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:03.955 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:03.951 7f96097fa700 1 -- 172.21.2.102:0/3681408275 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9604001270 con 0x7f960c09de20 2021-02-10T11:35:03.955 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:03.951 7f96097fa700 1 -- 172.21.2.102:0/3681408275 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9604000d30 con 0x7f960c09de20 2021-02-10T11:35:03.955 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:03.951 7f9612d16700 1 -- 172.21.2.102:0/3681408275 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f960c08bf40 con 0x7f960c09de20 2021-02-10T11:35:03.955 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:03.951 7f96097fa700 1 -- 172.21.2.102:0/3681408275 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9604004bb0 con 0x7f960c09de20 2021-02-10T11:35:03.956 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:03.951 7f9612d16700 1 -- 172.21.2.102:0/3681408275 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f960c08c4d0 con 0x7f960c09de20 2021-02-10T11:35:03.956 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:03.951 7f96097fa700 1 -- 172.21.2.102:0/3681408275 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f960400ae20 con 0x7f960c09de20 2021-02-10T11:35:03.956 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:03.952 7f96097fa700 1 --2- 172.21.2.102:0/3681408275 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f95f00186c0 0x7f95f001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:03.956 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:03.952 7f96097fa700 1 -- 172.21.2.102:0/3681408275 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(138..138 src has 1..138) v4 ==== 5659+0+0 (secure 0 0 0) 0x7f9604032900 con 0x7f960c09de20 2021-02-10T11:35:03.956 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:03.952 7f9610b32700 1 --2- 172.21.2.102:0/3681408275 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f95f00186c0 0x7f95f001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:03.956 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:03.952 7f9610b32700 1 --2- 172.21.2.102:0/3681408275 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f95f00186c0 0x7f95f001abc0 crc :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:03.984 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:03.982 7f9612d16700 1 -- 172.21.2.102:0/3681408275 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f960c0800b0 con 0x7f960c09de20 2021-02-10T11:35:03.987 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:03.985 7f96097fa700 1 -- 172.21.2.102:0/3681408275 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f9604001270 con 0x7f960c09de20 2021-02-10T11:35:04.114 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:04.112 7f9612d16700 1 -- 172.21.2.102:0/3681408275 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"pool2": "cephfs_metadata", "prefix": "osd pool delete", "pool": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f960c088070 con 0x7f960c09de20 2021-02-10T11:35:04.501 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:04.499 7f96097fa700 1 -- 172.21.2.102:0/3681408275 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"pool2": "cephfs_metadata", "prefix": "osd pool delete", "pool": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v139) v1 ==== 185+0+0 (secure 0 0 0) 0x7f960400b1b0 con 0x7f960c09de20 2021-02-10T11:35:04.531 INFO:teuthology.orchestra.run.gibba002.stderr:pool 'cephfs_metadata' removed 2021-02-10T11:35:04.534 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:04.531 7f9612d16700 1 -- 172.21.2.102:0/3681408275 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f95f00186c0 msgr2=0x7f95f001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:04.534 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:04.531 7f9612d16700 1 --2- 172.21.2.102:0/3681408275 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f95f00186c0 0x7f95f001abc0 crc :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:04.534 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:04.531 7f9612d16700 1 -- 172.21.2.102:0/3681408275 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f960c09de20 msgr2=0x7f960c08b700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:04.534 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:04.531 7f9612d16700 1 --2- 172.21.2.102:0/3681408275 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f960c09de20 0x7f960c08b700 secure :-1 s=READY pgs=2623 cs=0 l=1 rev1=1 rx=0x7f960400a940 tx=0x7f960400ace0).stop 2021-02-10T11:35:04.534 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:04.532 7f9612d16700 1 -- 172.21.2.102:0/3681408275 shutdown_connections 2021-02-10T11:35:04.534 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:04.532 7f9612d16700 1 --2- 172.21.2.102:0/3681408275 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f960c09de20 0x7f960c08b700 unknown :-1 s=CLOSED pgs=2623 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:04.534 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:04.532 7f9612d16700 1 --2- 172.21.2.102:0/3681408275 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f95f00186c0 0x7f95f001abc0 unknown :-1 s=CLOSED pgs=1156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:04.534 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:04.532 7f9612d16700 1 --2- 172.21.2.102:0/3681408275 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f960c09d400 0x7f960c09d800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:04.535 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:04.532 7f9612d16700 1 --2- 172.21.2.102:0/3681408275 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f960c09c190 0x7f960c09cde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:04.535 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:04.532 7f9612d16700 1 -- 172.21.2.102:0/3681408275 >> 172.21.2.102:0/3681408275 conn(0x7f960c080cb0 msgr2=0x7f960c0950a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:04.535 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:04.532 7f9612d16700 1 -- 172.21.2.102:0/3681408275 shutdown_connections 2021-02-10T11:35:04.535 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:04.532 7f9612d16700 1 -- 172.21.2.102:0/3681408275 wait complete. 2021-02-10T11:35:04.542 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool delete cephfs_data cephfs_data --yes-i-really-really-mean-it 2021-02-10T11:35:04.975 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:04.956 7f735d0c8700 1 Processor -- start 2021-02-10T11:35:04.975 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:04.956 7f735d0c8700 1 -- start start 2021-02-10T11:35:04.975 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:04.957 7f735d0c8700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f73580d1c80 0x7f73580cef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:04.975 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:04.957 7f735d0c8700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f73580cf590 0x7f73580cf990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:04.976 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:04.957 7f735d0c8700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73580fe1d0 con 0x7f73580d1880 2021-02-10T11:35:04.976 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:04.957 7f735d0c8700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73581b0cc0 con 0x7f73580d1c80 2021-02-10T11:35:04.976 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:04.957 7f735d0c8700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7358109a70 con 0x7f73580cf590 2021-02-10T11:35:04.976 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:04.957 7f7356d9d700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f73580d1880 0x7f73581b04d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:57036/0 (socket says 172.21.2.102:57036) 2021-02-10T11:35:04.976 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:04.957 7f7356d9d700 1 -- 172.21.2.102:0/2722515189 learned_addr learned my addr 172.21.2.102:0/2722515189 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:35:04.976 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:04.957 7f735659c700 1 --2- 172.21.2.102:0/2722515189 >> v2:172.21.2.113:3300/0 conn(0x7f73580d1c80 0x7f73580cef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:04.976 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:04.957 7f735759e700 1 --2- 172.21.2.102:0/2722515189 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f73580cf590 0x7f73580cf990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:04.976 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:04.958 7f7355d9b700 1 -- 172.21.2.102:0/2722515189 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 922708499 0 0) 0x7f7348000ce0 con 0x7f73580d1880 2021-02-10T11:35:04.976 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:04.958 7f735659c700 1 -- 172.21.2.102:0/2722515189 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f73580cf590 msgr2=0x7f73580cf990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:04.976 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:04.958 7f735659c700 1 --2- 172.21.2.102:0/2722515189 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f73580cf590 0x7f73580cf990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:04.976 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:04.958 7f735659c700 1 -- 172.21.2.102:0/2722515189 >> v1:172.21.2.102:6789/0 conn(0x7f73580d1880 legacy=0x7f73581b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:04.976 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:04.958 7f735659c700 1 -- 172.21.2.102:0/2722515189 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7358088070 con 0x7f73580d1c80 2021-02-10T11:35:04.976 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:04.958 7f735659c700 1 --2- 172.21.2.102:0/2722515189 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f73580d1c80 0x7f73580cef70 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 rx=0x7f734c006650 tx=0x7f734c009a80).ready entity=mon.1 client_cookie=9f099b6e6352e81e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:04.977 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:04.958 7f7355d9b700 1 -- 172.21.2.102:0/2722515189 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f734c00c0c0 con 0x7f73580d1c80 2021-02-10T11:35:04.977 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:04.958 7f7355d9b700 1 -- 172.21.2.102:0/2722515189 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f734c00c340 con 0x7f73580d1c80 2021-02-10T11:35:04.977 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:04.958 7f7355d9b700 1 -- 172.21.2.102:0/2722515189 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f734c00c860 con 0x7f73580d1c80 2021-02-10T11:35:04.977 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:04.958 7f735d0c8700 1 -- 172.21.2.102:0/2722515189 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f73580d1c80 msgr2=0x7f73580cef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:04.977 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:04.958 7f735d0c8700 1 --2- 172.21.2.102:0/2722515189 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f73580d1c80 0x7f73580cef70 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 rx=0x7f734c006650 tx=0x7f734c009a80).stop 2021-02-10T11:35:04.977 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:04.959 7f735d0c8700 1 -- 172.21.2.102:0/2722515189 shutdown_connections 2021-02-10T11:35:04.977 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:04.959 7f735d0c8700 1 --2- 172.21.2.102:0/2722515189 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f73580d1c80 0x7f73580cef70 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:04.977 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:04.959 7f735d0c8700 1 --2- 172.21.2.102:0/2722515189 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f73580cf590 0x7f73580cf990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:04.977 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:04.959 7f735d0c8700 1 -- 172.21.2.102:0/2722515189 >> 172.21.2.102:0/2722515189 conn(0x7f73580ce430 msgr2=0x7f73580d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:04.977 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:04.959 7f735d0c8700 1 -- 172.21.2.102:0/2722515189 shutdown_connections 2021-02-10T11:35:04.977 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:04.959 7f735d0c8700 1 -- 172.21.2.102:0/2722515189 wait complete. 2021-02-10T11:35:04.979 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:04.974 7f735d0c8700 1 Processor -- start 2021-02-10T11:35:04.979 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:04.974 7f735d0c8700 1 -- start start 2021-02-10T11:35:04.979 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:04.974 7f735d0c8700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f73581891e0 0x7f735819c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:04.979 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:04.974 7f735d0c8700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f735818e660 0x7f735818ea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:04.979 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:04.974 7f735d0c8700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f735818f080 0x7f7358196640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:04.979 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:04.974 7f735d0c8700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7358189700 con 0x7f735818e660 2021-02-10T11:35:04.979 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:04.974 7f735d0c8700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7358196c60 con 0x7f735818f080 2021-02-10T11:35:04.979 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:04.974 7f735d0c8700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7358196e80 con 0x7f73581891e0 2021-02-10T11:35:04.979 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:04.974 7f735659c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f735818e660 0x7f735818ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:04.980 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:04.974 7f735659c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f735818e660 0x7f735818ea60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:36586/0 (socket says 172.21.2.102:36586) 2021-02-10T11:35:04.980 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:04.974 7f735659c700 1 -- 172.21.2.102:0/2263460954 learned_addr learned my addr 172.21.2.102:0/2263460954 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:35:04.980 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:04.974 7f735759e700 1 --2- 172.21.2.102:0/2263460954 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f735818f080 0x7f7358196640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:04.980 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:04.974 7f7356d9d700 1 --2- 172.21.2.102:0/2263460954 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f73581891e0 0x7f735819c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:04.980 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:04.975 7f735759e700 1 -- 172.21.2.102:0/2263460954 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f73581891e0 msgr2=0x7f735819c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:04.980 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:04.975 7f735759e700 1 --2- 172.21.2.102:0/2263460954 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f73581891e0 0x7f735819c8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:04.980 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:04.975 7f735759e700 1 -- 172.21.2.102:0/2263460954 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f735818e660 msgr2=0x7f735818ea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:04.980 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:04.975 7f735759e700 1 --2- 172.21.2.102:0/2263460954 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f735818e660 0x7f735818ea60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:04.980 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:04.975 7f735759e700 1 -- 172.21.2.102:0/2263460954 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73581970a0 con 0x7f735818f080 2021-02-10T11:35:04.981 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:04.975 7f735659c700 1 --2- 172.21.2.102:0/2263460954 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f735818e660 0x7f735818ea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:35:04.981 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:04.975 7f735759e700 1 --2- 172.21.2.102:0/2263460954 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f735818f080 0x7f7358196640 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 rx=0x7f735000b8a0 tx=0x7f735000bc40).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:04.981 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:04.975 7f73477fe700 1 -- 172.21.2.102:0/2263460954 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f735000bd80 con 0x7f735818f080 2021-02-10T11:35:04.981 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:04.975 7f73477fe700 1 -- 172.21.2.102:0/2263460954 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7350003030 con 0x7f735818f080 2021-02-10T11:35:04.981 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:04.975 7f735d0c8700 1 -- 172.21.2.102:0/2263460954 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73581970a0 con 0x7f735818f080 2021-02-10T11:35:04.981 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:04.975 7f735d0c8700 1 -- 172.21.2.102:0/2263460954 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7358195890 con 0x7f735818f080 2021-02-10T11:35:04.981 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:04.975 7f73477fe700 1 -- 172.21.2.102:0/2263460954 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7350001470 con 0x7f735818f080 2021-02-10T11:35:04.981 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:04.976 7f73477fe700 1 -- 172.21.2.102:0/2263460954 <== mon.1 v2:172.21.2.113:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f73500011f0 con 0x7f735818f080 2021-02-10T11:35:04.981 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:04.976 7f73477fe700 1 --2- 172.21.2.102:0/2263460954 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f733c0186c0 0x7f733c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:04.981 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:04.976 7f73477fe700 1 -- 172.21.2.102:0/2263460954 <== mon.1 v2:172.21.2.113:3300/0 5 ==== osd_map(139..139 src has 1..139) v4 ==== 5221+0+0 (secure 0 0 0) 0x7f7350032680 con 0x7f735818f080 2021-02-10T11:35:04.981 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:04.976 7f7356d9d700 1 --2- 172.21.2.102:0/2263460954 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f733c0186c0 0x7f733c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:04.981 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:04.976 7f7356d9d700 1 --2- 172.21.2.102:0/2263460954 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f733c0186c0 0x7f733c01abc0 crc :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:05.009 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:05.007 7f735d0c8700 1 -- 172.21.2.102:0/2263460954 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73580fe1d0 con 0x7f735818f080 2021-02-10T11:35:05.012 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:05.011 7f73477fe700 1 -- 172.21.2.102:0/2263460954 <== mon.1 v2:172.21.2.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f7350002f30 con 0x7f735818f080 2021-02-10T11:35:05.143 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:05.141 7f735d0c8700 1 -- 172.21.2.102:0/2263460954 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_command({"pool2": "cephfs_data", "prefix": "osd pool delete", "pool": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f73580fe1d0 con 0x7f735818f080 2021-02-10T11:35:05.504 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:05.502 7f73477fe700 1 -- 172.21.2.102:0/2263460954 <== mon.1 v2:172.21.2.113:3300/0 7 ==== mon_command_ack([{"pool2": "cephfs_data", "prefix": "osd pool delete", "pool": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v140) v1 ==== 173+0+0 (secure 0 0 0) 0x7f7350002f30 con 0x7f735818f080 2021-02-10T11:35:05.511 INFO:teuthology.orchestra.run.gibba002.stderr:pool 'cephfs_data' removed 2021-02-10T11:35:05.512 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:05.510 7f735d0c8700 1 -- 172.21.2.102:0/2263460954 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f733c0186c0 msgr2=0x7f733c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:05.512 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:05.510 7f735d0c8700 1 --2- 172.21.2.102:0/2263460954 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f733c0186c0 0x7f733c01abc0 crc :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:05.512 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:05.510 7f735d0c8700 1 -- 172.21.2.102:0/2263460954 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f735818f080 msgr2=0x7f7358196640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:05.513 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:05.510 7f735d0c8700 1 --2- 172.21.2.102:0/2263460954 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f735818f080 0x7f7358196640 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 rx=0x7f735000b8a0 tx=0x7f735000bc40).stop 2021-02-10T11:35:05.513 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:05.510 7f735d0c8700 1 -- 172.21.2.102:0/2263460954 shutdown_connections 2021-02-10T11:35:05.513 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:05.510 7f735d0c8700 1 --2- 172.21.2.102:0/2263460954 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f733c0186c0 0x7f733c01abc0 unknown :-1 s=CLOSED pgs=1157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:05.513 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:05.510 7f735d0c8700 1 --2- 172.21.2.102:0/2263460954 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f735818f080 0x7f7358196640 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:05.513 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:05.510 7f735d0c8700 1 --2- 172.21.2.102:0/2263460954 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f735818e660 0x7f735818ea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:05.513 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:05.510 7f735d0c8700 1 --2- 172.21.2.102:0/2263460954 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f73581891e0 0x7f735819c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:05.513 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:05.510 7f735d0c8700 1 -- 172.21.2.102:0/2263460954 >> 172.21.2.102:0/2263460954 conn(0x7f735819f940 msgr2=0x7f735819ba90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:05.513 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:05.511 7f735d0c8700 1 -- 172.21.2.102:0/2263460954 shutdown_connections 2021-02-10T11:35:05.513 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:05.511 7f735d0c8700 1 -- 172.21.2.102:0/2263460954 wait complete. 2021-02-10T11:35:05.522 INFO:tasks.ceph.mds.a:Restarting daemon 2021-02-10T11:35:05.522 INFO:tasks.ceph.mds.a:Stopping old one... 2021-02-10T11:35:05.522 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2021-02-10T11:35:05.522 INFO:teuthology.orchestra.run:waiting for 300 2021-02-10T11:35:05.522 INFO:tasks.ceph.mds.b:Restarting daemon 2021-02-10T11:35:05.522 INFO:tasks.ceph.mds.b:Stopping old one... 2021-02-10T11:35:05.523 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2021-02-10T11:35:05.523 INFO:teuthology.orchestra.run:waiting for 300 2021-02-10T11:35:05.529 INFO:tasks.ceph.mds.a:Stopped 2021-02-10T11:35:05.529 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2021-02-10T11:35:05.532 INFO:tasks.ceph.mds.b:Stopped 2021-02-10T11:35:05.532 DEBUG:teuthology.orchestra.run.gibba013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2021-02-10T11:35:05.571 INFO:tasks.ceph.mds.a:Started 2021-02-10T11:35:05.571 DEBUG:teuthology.parallel:result is None 2021-02-10T11:35:05.573 INFO:tasks.ceph.mds.b:Started 2021-02-10T11:35:05.574 DEBUG:teuthology.parallel:result is None 2021-02-10T11:35:05.574 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd blacklist clear 2021-02-10T11:35:05.978 INFO:tasks.ceph.mds.b.gibba013.stdout:starting mds.b at 2021-02-10T11:35:05.981 INFO:tasks.ceph.mds.a.gibba002.stdout:starting mds.a at 2021-02-10T11:35:06.005 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:05.985 7f41adf93700 1 Processor -- start 2021-02-10T11:35:06.006 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:05.985 7f41adf93700 1 -- start start 2021-02-10T11:35:06.006 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:05.985 7f41adf93700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f41a80d0340 0x7f41a80ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:06.006 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:05.985 7f41adf93700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f41a80d1c70 0x7f41a80efc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:06.006 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:05.985 7f41adf93700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41a80f0b10 con 0x7f41a80d1870 2021-02-10T11:35:06.006 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:05.985 7f41adf93700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41a80e5ed0 con 0x7f41a80d0340 2021-02-10T11:35:06.006 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:05.985 7f41adf93700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41a81b0be0 con 0x7f41a80d1c70 2021-02-10T11:35:06.006 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:05.985 7f41a77fe700 1 --1- >> v1:172.21.2.113:6789/0 conn(0x7f41a80d1870 0x7f41a81b03f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.113:6789/0 says I am v1:172.21.2.102:36152/0 (socket says 172.21.2.102:36152) 2021-02-10T11:35:06.006 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:05.985 7f41a6ffd700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f41a80d0340 0x7f41a80ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:06.007 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:05.985 7f41a77fe700 1 -- 172.21.2.102:0/4046837527 learned_addr learned my addr 172.21.2.102:0/4046837527 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:35:06.007 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:05.985 7f41a7fff700 1 --2- 172.21.2.102:0/4046837527 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f41a80d1c70 0x7f41a80efc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:06.007 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:05.986 7f41a67fc700 1 -- 172.21.2.102:0/4046837527 <== mon.1 v1:172.21.2.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 512050966 0 0) 0x7f4198001190 con 0x7f41a80d1870 2021-02-10T11:35:06.007 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:05.986 7f41a7fff700 1 -- 172.21.2.102:0/4046837527 >> v2:172.21.2.113:3300/0 conn(0x7f41a80d0340 msgr2=0x7f41a80ef640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:06.007 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:05.986 7f41a7fff700 1 --2- 172.21.2.102:0/4046837527 >> v2:172.21.2.113:3300/0 conn(0x7f41a80d0340 0x7f41a80ef640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:06.007 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:05.986 7f41a7fff700 1 -- 172.21.2.102:0/4046837527 >> v1:172.21.2.113:6789/0 conn(0x7f41a80d1870 legacy=0x7f41a81b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:06.007 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:05.986 7f41a7fff700 1 -- 172.21.2.102:0/4046837527 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41a80f20a0 con 0x7f41a80d1c70 2021-02-10T11:35:06.007 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:05.986 7f41a6ffd700 1 --2- 172.21.2.102:0/4046837527 >> v2:172.21.2.113:3300/0 conn(0x7f41a80d0340 0x7f41a80ef640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:35:06.008 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:05.986 7f41a7fff700 1 --2- 172.21.2.102:0/4046837527 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f41a80d1c70 0x7f41a80efc60 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7f41a0006e50 tx=0x7f41a0009e70).ready entity=mon.2 client_cookie=18f1e4dd7e09dbc4 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:06.008 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:05.986 7f41a67fc700 1 -- 172.21.2.102:0/4046837527 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f41a0000f90 con 0x7f41a80d1c70 2021-02-10T11:35:06.008 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:05.986 7f41a67fc700 1 -- 172.21.2.102:0/4046837527 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41a00066f0 con 0x7f41a80d1c70 2021-02-10T11:35:06.008 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:05.986 7f41a67fc700 1 -- 172.21.2.102:0/4046837527 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f41a000b700 con 0x7f41a80d1c70 2021-02-10T11:35:06.008 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:05.986 7f41adf93700 1 -- 172.21.2.102:0/4046837527 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f41a80d1c70 msgr2=0x7f41a80efc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:06.008 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:05.986 7f41adf93700 1 --2- 172.21.2.102:0/4046837527 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f41a80d1c70 0x7f41a80efc60 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7f41a0006e50 tx=0x7f41a0009e70).stop 2021-02-10T11:35:06.008 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:05.987 7f41adf93700 1 -- 172.21.2.102:0/4046837527 shutdown_connections 2021-02-10T11:35:06.008 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:05.987 7f41adf93700 1 --2- 172.21.2.102:0/4046837527 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f41a80d1c70 0x7f41a80efc60 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:06.008 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:05.987 7f41adf93700 1 --2- 172.21.2.102:0/4046837527 >> v2:172.21.2.113:3300/0 conn(0x7f41a80d0340 0x7f41a80ef640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:06.008 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:05.987 7f41adf93700 1 -- 172.21.2.102:0/4046837527 >> 172.21.2.102:0/4046837527 conn(0x7f41a80ce450 msgr2=0x7f41a80d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:06.009 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:05.987 7f41adf93700 1 -- 172.21.2.102:0/4046837527 shutdown_connections 2021-02-10T11:35:06.009 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:05.987 7f41adf93700 1 -- 172.21.2.102:0/4046837527 wait complete. 2021-02-10T11:35:06.009 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:06.002 7f41adf93700 1 Processor -- start 2021-02-10T11:35:06.009 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:06.002 7f41adf93700 1 -- start start 2021-02-10T11:35:06.009 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:06.002 7f41adf93700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f41a819ffa0 0x7f41a80eda10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:06.009 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:06.002 7f41adf93700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f41a81a03a0 0x7f41a80ee030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:06.009 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:06.002 7f41adf93700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f41a80ee650 0x7f41a80ebe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:06.009 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:06.002 7f41adf93700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41a80ed490 con 0x7f41a819ffa0 2021-02-10T11:35:06.009 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:06.002 7f41adf93700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41a80eea50 con 0x7f41a81a03a0 2021-02-10T11:35:06.009 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:06.002 7f41adf93700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41a80eec30 con 0x7f41a80ee650 2021-02-10T11:35:06.009 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:06.003 7f41a77fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f41a819ffa0 0x7f41a80eda10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:06.010 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:06.003 7f41a77fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f41a819ffa0 0x7f41a80eda10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:36610/0 (socket says 172.21.2.102:36610) 2021-02-10T11:35:06.010 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:06.003 7f41a77fe700 1 -- 172.21.2.102:0/2942270198 learned_addr learned my addr 172.21.2.102:0/2942270198 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:35:06.010 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:06.003 7f41a7fff700 1 --2- 172.21.2.102:0/2942270198 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f41a80ee650 0x7f41a80ebe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:06.010 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:06.003 7f41a6ffd700 1 --2- 172.21.2.102:0/2942270198 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f41a81a03a0 0x7f41a80ee030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:06.010 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:06.003 7f41a77fe700 1 -- 172.21.2.102:0/2942270198 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f41a80ee650 msgr2=0x7f41a80ebe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:06.010 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:06.003 7f41a77fe700 1 --2- 172.21.2.102:0/2942270198 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f41a80ee650 0x7f41a80ebe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:06.010 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:06.003 7f41a77fe700 1 -- 172.21.2.102:0/2942270198 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f41a81a03a0 msgr2=0x7f41a80ee030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:06.010 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:06.003 7f41a77fe700 1 --2- 172.21.2.102:0/2942270198 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f41a81a03a0 0x7f41a80ee030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:06.010 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:06.003 7f41a77fe700 1 -- 172.21.2.102:0/2942270198 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41a80ec4a0 con 0x7f41a819ffa0 2021-02-10T11:35:06.010 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:06.003 7f41a6ffd700 1 --2- 172.21.2.102:0/2942270198 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f41a81a03a0 0x7f41a80ee030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:35:06.011 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:06.003 7f41a77fe700 1 --2- 172.21.2.102:0/2942270198 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f41a819ffa0 0x7f41a80eda10 secure :-1 s=READY pgs=2627 cs=0 l=1 rev1=1 rx=0x7f419800a590 tx=0x7f419800a930).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:06.011 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:06.003 7f418ffff700 1 -- 172.21.2.102:0/2942270198 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f41980070d0 con 0x7f41a819ffa0 2021-02-10T11:35:06.011 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:06.003 7f418ffff700 1 -- 172.21.2.102:0/2942270198 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4198000c10 con 0x7f41a819ffa0 2021-02-10T11:35:06.011 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:06.003 7f418ffff700 1 -- 172.21.2.102:0/2942270198 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4198004530 con 0x7f41a819ffa0 2021-02-10T11:35:06.011 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:06.003 7f41adf93700 1 -- 172.21.2.102:0/2942270198 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41a80ec4a0 con 0x7f41a819ffa0 2021-02-10T11:35:06.011 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:06.003 7f41adf93700 1 -- 172.21.2.102:0/2942270198 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41a80ec4a0 con 0x7f41a819ffa0 2021-02-10T11:35:06.012 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:06.004 7f418ffff700 1 -- 172.21.2.102:0/2942270198 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4198001110 con 0x7f41a819ffa0 2021-02-10T11:35:06.012 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:06.004 7f418ffff700 1 --2- 172.21.2.102:0/2942270198 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f41880186c0 0x7f418801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:06.012 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:06.004 7f418ffff700 1 -- 172.21.2.102:0/2942270198 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 4839+0+0 (secure 0 0 0) 0x7f4198032140 con 0x7f41a819ffa0 2021-02-10T11:35:06.012 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:06.004 7f41a6ffd700 1 --2- 172.21.2.102:0/2942270198 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f41880186c0 0x7f418801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:06.012 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:06.004 7f41a6ffd700 1 --2- 172.21.2.102:0/2942270198 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f41880186c0 0x7f418801abc0 crc :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:06.036 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:06.034 7f41adf93700 1 -- 172.21.2.102:0/2942270198 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41a80ebc50 con 0x7f41a819ffa0 2021-02-10T11:35:06.039 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:06.037 7f418ffff700 1 -- 172.21.2.102:0/2942270198 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f41980070d0 con 0x7f41a819ffa0 2021-02-10T11:35:06.179 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:06.177 7f41adf93700 1 -- 172.21.2.102:0/2942270198 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd blacklist clear"} v 0) v1 -- 0x7f41a80ebc50 con 0x7f41a819ffa0 2021-02-10T11:35:06.514 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:06.511 7f418ffff700 1 -- 172.21.2.102:0/2942270198 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd blacklist clear"}]=0 removed all blacklist entries v141) v1 ==== 97+0+0 (secure 0 0 0) 0x7f4198001110 con 0x7f41a819ffa0 2021-02-10T11:35:06.546 INFO:teuthology.orchestra.run.gibba002.stderr: removed all blacklist entries 2021-02-10T11:35:06.547 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:06.546 7f41adf93700 1 -- 172.21.2.102:0/2942270198 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f41880186c0 msgr2=0x7f418801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:06.548 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:06.546 7f41adf93700 1 --2- 172.21.2.102:0/2942270198 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f41880186c0 0x7f418801abc0 crc :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:06.548 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:06.546 7f41adf93700 1 -- 172.21.2.102:0/2942270198 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f41a819ffa0 msgr2=0x7f41a80eda10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:06.548 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:06.546 7f41adf93700 1 --2- 172.21.2.102:0/2942270198 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f41a819ffa0 0x7f41a80eda10 secure :-1 s=READY pgs=2627 cs=0 l=1 rev1=1 rx=0x7f419800a590 tx=0x7f419800a930).stop 2021-02-10T11:35:06.549 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:06.546 7f41adf93700 1 -- 172.21.2.102:0/2942270198 shutdown_connections 2021-02-10T11:35:06.549 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:06.546 7f41adf93700 1 --2- 172.21.2.102:0/2942270198 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f41a80ee650 0x7f41a80ebe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:06.549 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:06.546 7f41adf93700 1 --2- 172.21.2.102:0/2942270198 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f41880186c0 0x7f418801abc0 unknown :-1 s=CLOSED pgs=1158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:06.549 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:06.546 7f41adf93700 1 --2- 172.21.2.102:0/2942270198 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f41a81a03a0 0x7f41a80ee030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:06.549 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:06.546 7f41adf93700 1 --2- 172.21.2.102:0/2942270198 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f41a819ffa0 0x7f41a80eda10 unknown :-1 s=CLOSED pgs=2627 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:06.549 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:06.546 7f41adf93700 1 -- 172.21.2.102:0/2942270198 >> 172.21.2.102:0/2942270198 conn(0x7f41a8094d90 msgr2=0x7f41a808a420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:06.549 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:06.546 7f41adf93700 1 -- 172.21.2.102:0/2942270198 shutdown_connections 2021-02-10T11:35:06.549 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:06.547 7f41adf93700 1 -- 172.21.2.102:0/2942270198 wait complete. 2021-02-10T11:35:06.556 INFO:tasks.cephfs.cephfs_test_case:['0', '1', '2', '3'] 2021-02-10T11:35:06.556 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth ls --format=json-pretty 2021-02-10T11:35:06.985 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:06.966 7f4836d3f700 1 Processor -- start 2021-02-10T11:35:06.985 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:06.966 7f4836d3f700 1 -- start start 2021-02-10T11:35:06.985 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:06.967 7f4836d3f700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f48300f05f0 0x7f48300f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:06.985 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:06.967 7f4836d3f700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48301b0c70 con 0x7f48301b0470 2021-02-10T11:35:06.985 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:06.967 7f4836d3f700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48300f0b50 con 0x7f48301b0870 2021-02-10T11:35:06.985 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:06.967 7f4836d3f700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48300e5ea0 con 0x7f48300f05f0 2021-02-10T11:35:06.985 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:06.967 7f4834b5b700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f48301b0470 0x7f48300ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:57076/0 (socket says 172.21.2.102:57076) 2021-02-10T11:35:06.986 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:06.967 7f4834b5b700 1 -- 172.21.2.102:0/645704738 learned_addr learned my addr 172.21.2.102:0/645704738 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:35:06.986 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:06.967 7f483535c700 1 --2- 172.21.2.102:0/645704738 >> v2:172.21.2.102:3300/0 conn(0x7f48300f05f0 0x7f48300f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:06.986 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:06.967 7f482f7fe700 1 -- 172.21.2.102:0/645704738 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 179724117 0 0) 0x7f48180010b0 con 0x7f48301b0470 2021-02-10T11:35:06.986 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:06.968 7f483535c700 1 -- 172.21.2.102:0/645704738 >> v1:172.21.2.113:6789/0 conn(0x7f48301b0870 legacy=0x7f48300efe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:06.986 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:06.968 7f483535c700 1 -- 172.21.2.102:0/645704738 >> v1:172.21.2.102:6789/0 conn(0x7f48301b0470 legacy=0x7f48300ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:06.986 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:06.968 7f483535c700 1 -- 172.21.2.102:0/645704738 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48300e52e0 con 0x7f48300f05f0 2021-02-10T11:35:06.986 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:06.968 7f483535c700 1 --2- 172.21.2.102:0/645704738 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f48300f05f0 0x7f48300f20e0 secure :-1 s=READY pgs=2629 cs=0 l=1 rev1=1 rx=0x7f4828006ab0 tx=0x7f4828009b40).ready entity=mon.0 client_cookie=de81adb0ad613ce4 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:06.987 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:06.968 7f482f7fe700 1 -- 172.21.2.102:0/645704738 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f48280022e0 con 0x7f48300f05f0 2021-02-10T11:35:06.987 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:06.968 7f482f7fe700 1 -- 172.21.2.102:0/645704738 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4828002660 con 0x7f48300f05f0 2021-02-10T11:35:06.987 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:06.968 7f482f7fe700 1 -- 172.21.2.102:0/645704738 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4828002ac0 con 0x7f48300f05f0 2021-02-10T11:35:06.987 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:06.968 7f4836d3f700 1 -- 172.21.2.102:0/645704738 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f48300f05f0 msgr2=0x7f48300f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:06.987 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:06.968 7f4836d3f700 1 --2- 172.21.2.102:0/645704738 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f48300f05f0 0x7f48300f20e0 secure :-1 s=READY pgs=2629 cs=0 l=1 rev1=1 rx=0x7f4828006ab0 tx=0x7f4828009b40).stop 2021-02-10T11:35:06.987 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:06.969 7f4836d3f700 1 -- 172.21.2.102:0/645704738 shutdown_connections 2021-02-10T11:35:06.987 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:06.969 7f4836d3f700 1 --2- 172.21.2.102:0/645704738 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f48300f05f0 0x7f48300f20e0 unknown :-1 s=CLOSED pgs=2629 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:06.987 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:06.969 7f4836d3f700 1 -- 172.21.2.102:0/645704738 >> 172.21.2.102:0/645704738 conn(0x7f48300ce3e0 msgr2=0x7f48300e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:06.987 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:06.969 7f4836d3f700 1 -- 172.21.2.102:0/645704738 shutdown_connections 2021-02-10T11:35:06.987 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:06.969 7f4836d3f700 1 -- 172.21.2.102:0/645704738 wait complete. 2021-02-10T11:35:06.989 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:06.983 7f4836d3f700 1 Processor -- start 2021-02-10T11:35:06.989 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:06.983 7f4836d3f700 1 -- start start 2021-02-10T11:35:06.989 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:06.984 7f4836d3f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4830093240 0x7f48300b48c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:06.989 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:06.984 7f4836d3f700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f48300b4ee0 0x7f4830189bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:06.989 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:06.984 7f4836d3f700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f483018a210 0x7f483018a610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:06.989 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:06.984 7f4836d3f700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48300f2a20 con 0x7f4830093240 2021-02-10T11:35:06.989 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:06.984 7f4836d3f700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f483018ac30 con 0x7f48300b4ee0 2021-02-10T11:35:06.989 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:06.984 7f4836d3f700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f483018ae50 con 0x7f483018a210 2021-02-10T11:35:06.989 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:06.984 7f4834b5b700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4830093240 0x7f48300b48c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:06.989 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:06.984 7f4834b5b700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4830093240 0x7f48300b48c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:36626/0 (socket says 172.21.2.102:36626) 2021-02-10T11:35:06.990 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:06.984 7f4834b5b700 1 -- 172.21.2.102:0/1763777515 learned_addr learned my addr 172.21.2.102:0/1763777515 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:35:06.990 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:06.984 7f4834b5b700 1 -- 172.21.2.102:0/1763777515 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f483018a210 msgr2=0x7f483018a610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:06.990 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:06.984 7f4834b5b700 1 --2- 172.21.2.102:0/1763777515 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f483018a210 0x7f483018a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:06.990 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:06.984 7f4834b5b700 1 -- 172.21.2.102:0/1763777515 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f48300b4ee0 msgr2=0x7f4830189bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:06.990 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:06.984 7f4834b5b700 1 --2- 172.21.2.102:0/1763777515 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f48300b4ee0 0x7f4830189bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:06.990 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:06.984 7f4834b5b700 1 -- 172.21.2.102:0/1763777515 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4830187e20 con 0x7f4830093240 2021-02-10T11:35:06.990 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:06.985 7f4834b5b700 1 --2- 172.21.2.102:0/1763777515 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4830093240 0x7f48300b48c0 secure :-1 s=READY pgs=2630 cs=0 l=1 rev1=1 rx=0x7f4818001c60 tx=0x7f481800a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:06.990 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:06.985 7f482d7fa700 1 -- 172.21.2.102:0/1763777515 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f481800acc0 con 0x7f4830093240 2021-02-10T11:35:06.990 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:06.985 7f482d7fa700 1 -- 172.21.2.102:0/1763777515 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4818004140 con 0x7f4830093240 2021-02-10T11:35:06.990 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:06.985 7f482d7fa700 1 -- 172.21.2.102:0/1763777515 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4818004660 con 0x7f4830093240 2021-02-10T11:35:06.990 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:06.985 7f4836d3f700 1 -- 172.21.2.102:0/1763777515 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4830187e20 con 0x7f4830093240 2021-02-10T11:35:06.990 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:06.985 7f4836d3f700 1 -- 172.21.2.102:0/1763777515 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48301883b0 con 0x7f4830093240 2021-02-10T11:35:06.991 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:06.986 7f482d7fa700 1 -- 172.21.2.102:0/1763777515 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4818004140 con 0x7f4830093240 2021-02-10T11:35:06.991 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:06.986 7f482d7fa700 1 --2- 172.21.2.102:0/1763777515 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f48140186c0 0x7f481401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:06.991 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:06.986 7f482d7fa700 1 -- 172.21.2.102:0/1763777515 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 4753+0+0 (secure 0 0 0) 0x7f48180321f0 con 0x7f4830093240 2021-02-10T11:35:06.991 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:06.986 7f482ffff700 1 --2- 172.21.2.102:0/1763777515 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f48140186c0 0x7f481401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:06.991 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:06.986 7f482ffff700 1 --2- 172.21.2.102:0/1763777515 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f48140186c0 0x7f481401abc0 crc :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:07.019 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:07.017 7f4836d3f700 1 -- 172.21.2.102:0/1763777515 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48300eddb0 con 0x7f4830093240 2021-02-10T11:35:07.021 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:07.020 7f482d7fa700 1 -- 172.21.2.102:0/1763777515 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f48180009b0 con 0x7f4830093240 2021-02-10T11:35:07.153 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:07.150 7f4836d3f700 1 -- 172.21.2.102:0/1763777515 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f48300e52e0 con 0x7f4830093240 2021-02-10T11:35:07.153 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:07.151 7f482d7fa700 1 -- 172.21.2.102:0/1763777515 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v80) v1 ==== 80+0+4513 (secure 0 0 0) 0x7f4818004140 con 0x7f4830093240 2021-02-10T11:35:07.154 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:35:07.154 INFO:teuthology.orchestra.run.gibba002.stdout:{ 2021-02-10T11:35:07.154 INFO:teuthology.orchestra.run.gibba002.stdout: "auth_dump": [ 2021-02-10T11:35:07.154 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:35:07.154 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "mds.a", 2021-02-10T11:35:07.155 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAevSNgsgyEARAAvr0cUjjCoEHLxaZSZ2HtVA==", 2021-02-10T11:35:07.155 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:35:07.155 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow", 2021-02-10T11:35:07.155 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:35:07.155 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:35:07.155 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:35:07.156 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:35:07.156 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:35:07.156 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:35:07.156 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "mds.b", 2021-02-10T11:35:07.156 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAevSNgwOFMMRAALfb1fbX56ao1TU8ClnTW1A==", 2021-02-10T11:35:07.156 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:35:07.157 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow", 2021-02-10T11:35:07.157 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:35:07.157 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:35:07.157 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:35:07.157 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:35:07.157 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:35:07.157 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:35:07.157 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.0", 2021-02-10T11:35:07.157 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAmvSNgj+bMMBAAD3HUD6fLiMAg+++iNzLJKQ==", 2021-02-10T11:35:07.158 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:35:07.158 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:35:07.158 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:35:07.158 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:35:07.158 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:35:07.158 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:35:07.158 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:35:07.158 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.1", 2021-02-10T11:35:07.158 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQApvSNgnvc+MRAAAOxqlNwiyLHqNITOrYeFKQ==", 2021-02-10T11:35:07.158 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:35:07.159 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:35:07.159 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:35:07.159 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:35:07.159 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:35:07.159 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:35:07.159 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:35:07.159 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.2", 2021-02-10T11:35:07.159 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAsvSNgZ4SbMRAA76IOYWfbiAuBoM/4k81GxA==", 2021-02-10T11:35:07.159 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:35:07.159 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:35:07.160 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:35:07.160 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:35:07.160 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:35:07.160 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:35:07.160 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:35:07.160 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.3", 2021-02-10T11:35:07.160 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAvvSNgFk3rMRAAWCmKKyw0LDQ+UlAHZAITOw==", 2021-02-10T11:35:07.160 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:35:07.160 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:35:07.160 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:35:07.160 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:35:07.160 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:35:07.160 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:35:07.161 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:35:07.161 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.4", 2021-02-10T11:35:07.161 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQA4vSNgpKcNJBAApJMmvj5YxVHPDOmvmQONOg==", 2021-02-10T11:35:07.161 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:35:07.161 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:35:07.161 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:35:07.161 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:35:07.161 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:35:07.161 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:35:07.161 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:35:07.161 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.5", 2021-02-10T11:35:07.161 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQA7vSNgf2ujJBAAfkuYpyb3WTzc8V5kw+Wktg==", 2021-02-10T11:35:07.161 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:35:07.162 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:35:07.162 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:35:07.162 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:35:07.162 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:35:07.162 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:35:07.162 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:35:07.162 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.6", 2021-02-10T11:35:07.162 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQA+vSNgqw9uJRAAFMU0R+UxU9C3o/9byb3e0g==", 2021-02-10T11:35:07.162 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:35:07.162 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:35:07.162 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:35:07.162 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:35:07.163 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:35:07.163 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:35:07.163 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:35:07.163 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.7", 2021-02-10T11:35:07.163 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQBBvSNgvWSrJRAA2KaCHmLpzovRgYzLvyofTQ==", 2021-02-10T11:35:07.163 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:35:07.163 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:35:07.163 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:35:07.163 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:35:07.163 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:35:07.164 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:35:07.164 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:35:07.164 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "client.0", 2021-02-10T11:35:07.164 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAfvSNgZd9XIBAAvyrvl8fdKuHnJZfczfRoFw==", 2021-02-10T11:35:07.164 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:35:07.164 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow", 2021-02-10T11:35:07.164 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow r", 2021-02-10T11:35:07.164 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:35:07.164 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:35:07.164 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:35:07.164 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:35:07.164 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "client.1", 2021-02-10T11:35:07.165 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAfvSNgSyfXNhAAIduPnF7Vd/6HSkJ0MlcMSA==", 2021-02-10T11:35:07.165 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:35:07.165 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow", 2021-02-10T11:35:07.165 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow r", 2021-02-10T11:35:07.165 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:35:07.165 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:35:07.165 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:35:07.165 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:35:07.165 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "client.2", 2021-02-10T11:35:07.165 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAgvSNgiIG/ERAA9ZTamaTI7ZsPSnxBqjplCg==", 2021-02-10T11:35:07.165 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:35:07.165 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow", 2021-02-10T11:35:07.165 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow r", 2021-02-10T11:35:07.166 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:35:07.166 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:35:07.166 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:35:07.166 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:35:07.166 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "client.3", 2021-02-10T11:35:07.166 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAgvSNgOWpOKBAAlXsEB04Q/GvtL06PqRMLeA==", 2021-02-10T11:35:07.166 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:35:07.166 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow", 2021-02-10T11:35:07.166 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow r", 2021-02-10T11:35:07.166 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:35:07.167 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:35:07.167 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:35:07.167 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:35:07.167 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "client.admin", 2021-02-10T11:35:07.167 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAavSNgO3GQBhAAL+NVnYskQRKVWLgyDPY/xw==", 2021-02-10T11:35:07.167 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:35:07.167 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow *", 2021-02-10T11:35:07.167 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:35:07.168 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:35:07.168 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:35:07.168 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:35:07.168 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:35:07.168 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:35:07.168 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "mgr.x", 2021-02-10T11:35:07.168 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAdvSNgXZZgJhAA7dtBSGzH+DsBwnddZpOBJA==", 2021-02-10T11:35:07.168 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:35:07.168 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow *", 2021-02-10T11:35:07.168 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow profile mgr", 2021-02-10T11:35:07.168 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:35:07.169 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:35:07.169 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:35:07.169 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:35:07.169 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "mgr.y", 2021-02-10T11:35:07.169 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAdvSNgVqQFDRAAwo6YEdWZCAr5GdOWVL/Qvg==", 2021-02-10T11:35:07.169 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:35:07.169 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow *", 2021-02-10T11:35:07.169 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow profile mgr", 2021-02-10T11:35:07.169 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:35:07.169 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:35:07.169 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:35:07.169 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:35:07.169 INFO:teuthology.orchestra.run.gibba002.stdout:} 2021-02-10T11:35:07.171 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:07.152 7f4836d3f700 1 -- 172.21.2.102:0/1763777515 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f48140186c0 msgr2=0x7f481401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:07.171 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:07.152 7f4836d3f700 1 --2- 172.21.2.102:0/1763777515 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f48140186c0 0x7f481401abc0 crc :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:07.171 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:07.152 7f4836d3f700 1 -- 172.21.2.102:0/1763777515 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4830093240 msgr2=0x7f48300b48c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:07.171 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:07.152 7f4836d3f700 1 --2- 172.21.2.102:0/1763777515 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4830093240 0x7f48300b48c0 secure :-1 s=READY pgs=2630 cs=0 l=1 rev1=1 rx=0x7f4818001c60 tx=0x7f481800a8d0).stop 2021-02-10T11:35:07.171 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:07.153 7f4836d3f700 1 -- 172.21.2.102:0/1763777515 shutdown_connections 2021-02-10T11:35:07.171 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:07.153 7f4836d3f700 1 --2- 172.21.2.102:0/1763777515 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f483018a210 0x7f483018a610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:07.171 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:07.153 7f4836d3f700 1 --2- 172.21.2.102:0/1763777515 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f48140186c0 0x7f481401abc0 unknown :-1 s=CLOSED pgs=1159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:07.171 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:07.153 7f4836d3f700 1 --2- 172.21.2.102:0/1763777515 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f48300b4ee0 0x7f4830189bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:07.172 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:07.153 7f4836d3f700 1 --2- 172.21.2.102:0/1763777515 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4830093240 0x7f48300b48c0 unknown :-1 s=CLOSED pgs=2630 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:07.172 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:07.153 7f4836d3f700 1 -- 172.21.2.102:0/1763777515 >> 172.21.2.102:0/1763777515 conn(0x7f4830094250 msgr2=0x7f483018c230 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:07.172 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:07.153 7f4836d3f700 1 -- 172.21.2.102:0/1763777515 shutdown_connections 2021-02-10T11:35:07.172 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:07.153 7f4836d3f700 1 -- 172.21.2.102:0/1763777515 wait complete. 2021-02-10T11:35:07.172 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:35:07.603 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:07.585 7f0d07c72700 1 Processor -- start 2021-02-10T11:35:07.604 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:07.585 7f0d07c72700 1 -- start start 2021-02-10T11:35:07.604 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:07.586 7f0d07c72700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f0d001b0870 0x7f0d000f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:07.604 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:07.586 7f0d07c72700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f0d000efe00 0x7f0d000f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:07.604 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:07.586 7f0d07c72700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d001b0c70 con 0x7f0d001b0470 2021-02-10T11:35:07.604 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:07.586 7f0d07c72700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d000e5ea0 con 0x7f0d000efe00 2021-02-10T11:35:07.604 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:07.586 7f0d07c72700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d000f0b50 con 0x7f0d001b0870 2021-02-10T11:35:07.604 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:07.586 7f0d0628f700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f0d000efe00 0x7f0d000f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:07.604 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:07.586 7f0d0528d700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f0d001b0870 0x7f0d000f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:07.604 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:07.586 7f0d05a8e700 1 --1- >> v1:172.21.2.113:6789/0 conn(0x7f0d001b0470 0x7f0d000ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.113:6789/0 says I am v1:172.21.2.102:36180/0 (socket says 172.21.2.102:36180) 2021-02-10T11:35:07.604 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:07.586 7f0d05a8e700 1 -- 172.21.2.102:0/3340867502 learned_addr learned my addr 172.21.2.102:0/3340867502 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:35:07.605 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:07.586 7f0d0528d700 1 -- 172.21.2.102:0/3340867502 >> v2:172.21.2.102:3300/0 conn(0x7f0d000efe00 msgr2=0x7f0d000f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:07.605 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:07.586 7f0d0528d700 1 --2- 172.21.2.102:0/3340867502 >> v2:172.21.2.102:3300/0 conn(0x7f0d000efe00 0x7f0d000f0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:07.605 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:07.586 7f0d0528d700 1 -- 172.21.2.102:0/3340867502 >> v1:172.21.2.113:6789/0 conn(0x7f0d001b0470 legacy=0x7f0d000ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:07.605 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:07.586 7f0d0528d700 1 -- 172.21.2.102:0/3340867502 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d000f0820 con 0x7f0d001b0870 2021-02-10T11:35:07.605 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:07.586 7f0d0528d700 1 --2- 172.21.2.102:0/3340867502 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0d001b0870 0x7f0d000f20e0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 rx=0x7f0cf80069d0 tx=0x7f0cf8009e40).ready entity=mon.1 client_cookie=ffba4b6740739ade server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:07.605 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:07.587 7f0d04a8c700 1 -- 172.21.2.102:0/3340867502 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0cf800b510 con 0x7f0d001b0870 2021-02-10T11:35:07.605 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:07.587 7f0d04a8c700 1 -- 172.21.2.102:0/3340867502 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cf800b890 con 0x7f0d001b0870 2021-02-10T11:35:07.605 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:07.587 7f0d04a8c700 1 -- 172.21.2.102:0/3340867502 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0cf800bdb0 con 0x7f0d001b0870 2021-02-10T11:35:07.605 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:07.587 7f0d07c72700 1 -- 172.21.2.102:0/3340867502 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0d001b0870 msgr2=0x7f0d000f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:07.605 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:07.587 7f0d07c72700 1 --2- 172.21.2.102:0/3340867502 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0d001b0870 0x7f0d000f20e0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 rx=0x7f0cf80069d0 tx=0x7f0cf8009e40).stop 2021-02-10T11:35:07.605 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:07.587 7f0d07c72700 1 -- 172.21.2.102:0/3340867502 shutdown_connections 2021-02-10T11:35:07.605 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:07.587 7f0d07c72700 1 --2- 172.21.2.102:0/3340867502 >> v2:172.21.2.102:3300/0 conn(0x7f0d000efe00 0x7f0d000f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:07.605 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:07.587 7f0d07c72700 1 --2- 172.21.2.102:0/3340867502 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0d001b0870 0x7f0d000f20e0 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:07.605 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:07.587 7f0d07c72700 1 -- 172.21.2.102:0/3340867502 >> 172.21.2.102:0/3340867502 conn(0x7f0d000ce3e0 msgr2=0x7f0d000e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:07.606 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:07.588 7f0d07c72700 1 -- 172.21.2.102:0/3340867502 shutdown_connections 2021-02-10T11:35:07.606 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:07.588 7f0d07c72700 1 -- 172.21.2.102:0/3340867502 wait complete. 2021-02-10T11:35:07.608 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:07.602 7f0d07c72700 1 Processor -- start 2021-02-10T11:35:07.608 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:07.602 7f0d07c72700 1 -- start start 2021-02-10T11:35:07.608 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:07.603 7f0d07c72700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0d001af8a0 0x7f0d00198e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:07.608 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:07.603 7f0d07c72700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0d001af050 0x7f0d000aa3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:07.608 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:07.603 7f0d07c72700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0d00197d00 0x7f0d00198100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:07.608 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:07.603 7f0d07c72700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d001988e0 con 0x7f0d001af8a0 2021-02-10T11:35:07.608 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:07.603 7f0d07c72700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d001af450 con 0x7f0d00197d00 2021-02-10T11:35:07.608 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:07.603 7f0d07c72700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d000aaa10 con 0x7f0d001af050 2021-02-10T11:35:07.608 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:07.603 7f0d05a8e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0d001af8a0 0x7f0d00198e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:07.608 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:07.603 7f0d05a8e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0d001af8a0 0x7f0d00198e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:36640/0 (socket says 172.21.2.102:36640) 2021-02-10T11:35:07.608 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:07.603 7f0d05a8e700 1 -- 172.21.2.102:0/3737040562 learned_addr learned my addr 172.21.2.102:0/3737040562 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:35:07.608 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:07.603 7f0d0528d700 1 --2- 172.21.2.102:0/3737040562 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0d001af050 0x7f0d000aa3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:07.609 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:07.603 7f0d0628f700 1 --2- 172.21.2.102:0/3737040562 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0d00197d00 0x7f0d00198100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:07.609 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:07.603 7f0d05a8e700 1 -- 172.21.2.102:0/3737040562 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0d001af050 msgr2=0x7f0d000aa3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:07.609 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:07.603 7f0d05a8e700 1 --2- 172.21.2.102:0/3737040562 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0d001af050 0x7f0d000aa3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:07.609 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:07.603 7f0d05a8e700 1 -- 172.21.2.102:0/3737040562 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0d00197d00 msgr2=0x7f0d00198100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:07.609 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:07.603 7f0d05a8e700 1 --2- 172.21.2.102:0/3737040562 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0d00197d00 0x7f0d00198100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:07.609 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:07.603 7f0d05a8e700 1 -- 172.21.2.102:0/3737040562 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d0019fec0 con 0x7f0d001af8a0 2021-02-10T11:35:07.609 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:07.604 7f0d05a8e700 1 --2- 172.21.2.102:0/3737040562 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0d001af8a0 0x7f0d00198e60 secure :-1 s=READY pgs=2631 cs=0 l=1 rev1=1 rx=0x7f0cf0007fa0 tx=0x7f0cf000a950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:07.609 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:07.604 7f0cf67fc700 1 -- 172.21.2.102:0/3737040562 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0cf00070d0 con 0x7f0d001af8a0 2021-02-10T11:35:07.609 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:07.604 7f0cf67fc700 1 -- 172.21.2.102:0/3737040562 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cf000ad40 con 0x7f0d001af8a0 2021-02-10T11:35:07.609 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:07.604 7f0d07c72700 1 -- 172.21.2.102:0/3737040562 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d0019fec0 con 0x7f0d001af8a0 2021-02-10T11:35:07.609 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:07.604 7f0cf67fc700 1 -- 172.21.2.102:0/3737040562 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0cf00045a0 con 0x7f0d001af8a0 2021-02-10T11:35:07.609 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:07.604 7f0d07c72700 1 -- 172.21.2.102:0/3737040562 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d0019fec0 con 0x7f0d001af8a0 2021-02-10T11:35:07.609 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:07.605 7f0cf67fc700 1 -- 172.21.2.102:0/3737040562 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f0cf0000c10 con 0x7f0d001af8a0 2021-02-10T11:35:07.609 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:07.605 7f0cf67fc700 1 --2- 172.21.2.102:0/3737040562 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0ce8018700 0x7f0ce801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:07.610 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:07.605 7f0cf67fc700 1 -- 172.21.2.102:0/3737040562 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 4753+0+0 (secure 0 0 0) 0x7f0cf00324a0 con 0x7f0d001af8a0 2021-02-10T11:35:07.610 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:07.605 7f0d0528d700 1 --2- 172.21.2.102:0/3737040562 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0ce8018700 0x7f0ce801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:07.610 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:07.605 7f0d0528d700 1 --2- 172.21.2.102:0/3737040562 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0ce8018700 0x7f0ce801abc0 crc :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:07.637 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:07.635 7f0d07c72700 1 -- 172.21.2.102:0/3737040562 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d00105a20 con 0x7f0d001af8a0 2021-02-10T11:35:07.639 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:07.638 7f0cf67fc700 1 -- 172.21.2.102:0/3737040562 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f0cf0007030 con 0x7f0d001af8a0 2021-02-10T11:35:07.764 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:07.762 7f0d07c72700 1 -- 172.21.2.102:0/3737040562 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0d000f0b50 con 0x7f0d001af8a0 2021-02-10T11:35:07.764 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:07.763 7f0cf67fc700 1 -- 172.21.2.102:0/3737040562 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v141) v1 ==== 74+0+11154 (secure 0 0 0) 0x7f0cf0000c10 con 0x7f0d001af8a0 2021-02-10T11:35:07.765 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:35:07.765 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":141,"fsid":"68db30ea-dbce-4aef-a6df-f01c6fea0513","created":"2021-02-10 11:03:06.331557","modified":"2021-02-10 11:35:06.506299","last_up_change":"2021-02-10 11:03:32.690562","last_in_change":"2021-02-10 11:03:32.690562","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":29,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:03:37.322519","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"be691b4d-a039-408a-af80-9d53f02b6f1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":134,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6818","nonce":36088},{"type":"v1","addr":"172.21.2.102:6820","nonce":36088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6822","nonce":36088},{"type":"v1","addr":"172.21.2.102:6824","nonce":36088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6830","nonce":36088},{"type":"v1","addr":"172.21.2.102:6832","nonce":36088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6826","nonce":36088},{"type":"v1","addr":"172.21.2.102:6828","nonce":36088}]},"public_addr":"172.21.2.102:6820/36088","cluster_addr":"172.21.2.102:6824/36088","heartbeat_back_addr":"172.21.2.102:6832/36088","heartbeat_front_addr":"172.21.2.102:6828/36088","state":["exists","up"]},{"osd":1,"uuid":"64e3ccc8-9c69-4d7c-865e-af8e83822635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":135,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6819","nonce":36087},{"type":"v1","addr":"172.21.2.102:6821","nonce":36087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6823","nonce":36087},{"type":"v1","addr":"172.21.2.102:6825","nonce":36087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6831","nonce":36087},{"type":"v1","addr":"172.21.2.102:6833","nonce":36087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6827","nonce":36087},{"type":"v1","addr":"172.21.2.102:6829","nonce":36087}]},"public_addr":"172.21.2.102:6821/36087","cluster_addr":"172.21.2.102:6825/36087","heartbeat_back_addr":"172.21.2.102:6833/36087","heartbeat_front_addr":"172.21.2.102:6829/36087","state":["exists","up"]},{"osd":2,"uuid":"153700af-747f-481f-bdf4-d6072d3a842f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6810","nonce":36089},{"type":"v1","addr":"172.21.2.102:6811","nonce":36089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6812","nonce":36089},{"type":"v1","addr":"172.21.2.102:6813","nonce":36089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6816","nonce":36089},{"type":"v1","addr":"172.21.2.102:6817","nonce":36089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6814","nonce":36089},{"type":"v1","addr":"172.21.2.102:6815","nonce":36089}]},"public_addr":"172.21.2.102:6811/36089","cluster_addr":"172.21.2.102:6813/36089","heartbeat_back_addr":"172.21.2.102:6817/36089","heartbeat_front_addr":"172.21.2.102:6815/36089","state":["exists","up"]},{"osd":3,"uuid":"cbc678f4-83a9-45e6-8830-2f7c232b1ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":118,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6802","nonce":36090},{"type":"v1","addr":"172.21.2.102:6803","nonce":36090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6804","nonce":36090},{"type":"v1","addr":"172.21.2.102:6805","nonce":36090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6808","nonce":36090},{"type":"v1","addr":"172.21.2.102:6809","nonce":36090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6806","nonce":36090},{"type":"v1","addr":"172.21.2.102:6807","nonce":36090}]},"public_addr":"172.21.2.102:6803/36090","cluster_addr":"172.21.2.102:6805/36090","heartbeat_back_addr":"172.21.2.102:6809/36090","heartbeat_front_addr":"172.21.2.102:6807/36090","state":["exists","up"]},{"osd":4,"uuid":"c01e4008-b258-4f10-afd2-9e7f37214351","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":135,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6808","nonce":36368},{"type":"v1","addr":"172.21.2.113:6811","nonce":36368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6814","nonce":36368},{"type":"v1","addr":"172.21.2.113:6818","nonce":36368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6826","nonce":36368},{"type":"v1","addr":"172.21.2.113:6828","nonce":36368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6820","nonce":36368},{"type":"v1","addr":"172.21.2.113:6824","nonce":36368}]},"public_addr":"172.21.2.113:6811/36368","cluster_addr":"172.21.2.113:6818/36368","heartbeat_back_addr":"172.21.2.113:6828/36368","heartbeat_front_addr":"172.21.2.113:6824/36368","state":["exists","up"]},{"osd":5,"uuid":"551f6007-daf8-4a98-8ee0-c2b16bcccb5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":134,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6802","nonce":36366},{"type":"v1","addr":"172.21.2.113:6805","nonce":36366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6807","nonce":36366},{"type":"v1","addr":"172.21.2.113:6810","nonce":36366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6819","nonce":36366},{"type":"v1","addr":"172.21.2.113:6822","nonce":36366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6812","nonce":36366},{"type":"v1","addr":"172.21.2.113:6815","nonce":36366}]},"public_addr":"172.21.2.113:6805/36366","cluster_addr":"172.21.2.113:6810/36366","heartbeat_back_addr":"172.21.2.113:6822/36366","heartbeat_front_addr":"172.21.2.113:6815/36366","state":["exists","up"]},{"osd":6,"uuid":"bf67c040-0e50-48be-b102-c3943bc9ebef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":135,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6800","nonce":36367},{"type":"v1","addr":"172.21.2.113:6801","nonce":36367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6803","nonce":36367},{"type":"v1","addr":"172.21.2.113:6804","nonce":36367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6813","nonce":36367},{"type":"v1","addr":"172.21.2.113:6816","nonce":36367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6806","nonce":36367},{"type":"v1","addr":"172.21.2.113:6809","nonce":36367}]},"public_addr":"172.21.2.113:6801/36367","cluster_addr":"172.21.2.113:6804/36367","heartbeat_back_addr":"172.21.2.113:6816/36367","heartbeat_front_addr":"172.21.2.113:6809/36367","state":["exists","up"]},{"osd":7,"uuid":"f5461f80-018d-428f-be89-2eb2bcbe7bb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":135,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6817","nonce":36365},{"type":"v1","addr":"172.21.2.113:6821","nonce":36365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6823","nonce":36365},{"type":"v1","addr":"172.21.2.113:6825","nonce":36365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6830","nonce":36365},{"type":"v1","addr":"172.21.2.113:6831","nonce":36365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6827","nonce":36365},{"type":"v1","addr":"172.21.2.113:6829","nonce":36365}]},"public_addr":"172.21.2.113:6821/36365","cluster_addr":"172.21.2.113:6825/36365","heartbeat_back_addr":"172.21.2.113:6831/36365","heartbeat_front_addr":"172.21.2.113:6829/36365","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:35:07.766 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:07.764 7f0d07c72700 1 -- 172.21.2.102:0/3737040562 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0ce8018700 msgr2=0x7f0ce801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:07.766 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:07.764 7f0d07c72700 1 --2- 172.21.2.102:0/3737040562 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0ce8018700 0x7f0ce801abc0 crc :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:07.767 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:07.764 7f0d07c72700 1 -- 172.21.2.102:0/3737040562 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0d001af8a0 msgr2=0x7f0d00198e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:07.767 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:07.764 7f0d07c72700 1 --2- 172.21.2.102:0/3737040562 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0d001af8a0 0x7f0d00198e60 secure :-1 s=READY pgs=2631 cs=0 l=1 rev1=1 rx=0x7f0cf0007fa0 tx=0x7f0cf000a950).stop 2021-02-10T11:35:07.767 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:07.764 7f0d07c72700 1 -- 172.21.2.102:0/3737040562 shutdown_connections 2021-02-10T11:35:07.767 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:07.764 7f0d07c72700 1 --2- 172.21.2.102:0/3737040562 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0ce8018700 0x7f0ce801abc0 unknown :-1 s=CLOSED pgs=1160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:07.767 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:07.764 7f0d07c72700 1 --2- 172.21.2.102:0/3737040562 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0d00197d00 0x7f0d00198100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:07.767 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:07.764 7f0d07c72700 1 --2- 172.21.2.102:0/3737040562 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0d001af050 0x7f0d000aa3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:07.767 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:07.764 7f0d07c72700 1 --2- 172.21.2.102:0/3737040562 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0d001af8a0 0x7f0d00198e60 unknown :-1 s=CLOSED pgs=2631 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:07.767 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:07.764 7f0d07c72700 1 -- 172.21.2.102:0/3737040562 >> 172.21.2.102:0/3737040562 conn(0x7f0d001a37a0 msgr2=0x7f0d0018c230 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:07.767 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:07.765 7f0d07c72700 1 -- 172.21.2.102:0/3737040562 shutdown_connections 2021-02-10T11:35:07.767 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:07.765 7f0d07c72700 1 -- 172.21.2.102:0/3737040562 wait complete. 2021-02-10T11:35:07.775 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:03:37.322519', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2021-02-10T11:35:07.776 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:35:08.208 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:08.188 7f5db6aa7700 1 Processor -- start 2021-02-10T11:35:08.208 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:08.188 7f5db6aa7700 1 -- start start 2021-02-10T11:35:08.208 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:08.188 7f5db6aa7700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f5db00d1870 0x7f5db01b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:08.209 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:08.188 7f5db6aa7700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5db01b0ae0 0x7f5db0108940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:08.209 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:08.189 7f5db6aa7700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5db00fe1c0 con 0x7f5db00d1c70 2021-02-10T11:35:08.209 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:08.189 7f5db6aa7700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5db0109a60 con 0x7f5db00d1870 2021-02-10T11:35:08.209 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:08.189 7f5db6aa7700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5db0109750 con 0x7f5db01b0ae0 2021-02-10T11:35:08.209 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:08.189 7f5db50c4700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f5db00d1c70 0x7f5db0108f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:57106/0 (socket says 172.21.2.102:57106) 2021-02-10T11:35:08.209 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:08.189 7f5db50c4700 1 -- 172.21.2.102:0/1419134118 learned_addr learned my addr 172.21.2.102:0/1419134118 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:35:08.209 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:08.189 7f5daf7fe700 1 -- 172.21.2.102:0/1419134118 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 863333560 0 0) 0x7f5da8001500 con 0x7f5db00d1c70 2021-02-10T11:35:08.209 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:08.189 7f5daf7fe700 1 -- 172.21.2.102:0/1419134118 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d9c002fe0 con 0x7f5db00d1c70 2021-02-10T11:35:08.209 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:08.189 7f5daffff700 1 --2- 172.21.2.102:0/1419134118 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5db01b0ae0 0x7f5db0108940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:08.210 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:08.189 7f5db48c3700 1 --2- 172.21.2.102:0/1419134118 >> v2:172.21.2.113:3300/0 conn(0x7f5db00d1870 0x7f5db01b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:08.210 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:08.190 7f5daf7fe700 1 -- 172.21.2.102:0/1419134118 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3669276524 0 0) 0x7f5da8001460 con 0x7f5db00d1c70 2021-02-10T11:35:08.210 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:08.190 7f5daf7fe700 1 -- 172.21.2.102:0/1419134118 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5db01b0ae0 msgr2=0x7f5db0108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:08.210 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:08.190 7f5daf7fe700 1 --2- 172.21.2.102:0/1419134118 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5db01b0ae0 0x7f5db0108940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:08.210 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:08.190 7f5daf7fe700 1 -- 172.21.2.102:0/1419134118 >> v2:172.21.2.113:3300/0 conn(0x7f5db00d1870 msgr2=0x7f5db01b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:08.210 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:08.190 7f5daf7fe700 1 --2- 172.21.2.102:0/1419134118 >> v2:172.21.2.113:3300/0 conn(0x7f5db00d1870 0x7f5db01b04c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:08.210 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:08.190 7f5daf7fe700 1 -- 172.21.2.102:0/1419134118 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5db0088070 con 0x7f5db00d1c70 2021-02-10T11:35:08.210 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:08.190 7f5daf7fe700 1 -- 172.21.2.102:0/1419134118 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7f5da8001980 con 0x7f5db00d1c70 2021-02-10T11:35:08.210 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:08.190 7f5daf7fe700 1 -- 172.21.2.102:0/1419134118 >> v1:172.21.2.102:6789/0 conn(0x7f5db00d1c70 legacy=0x7f5db0108f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:08.211 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:08.190 7f5daf7fe700 1 --2- 172.21.2.102:0/1419134118 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5d9c003a60 0x7f5d9c005ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:08.211 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:08.190 7f5daf7fe700 1 --2- 172.21.2.102:0/1419134118 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5d9c04a7a0 0x7f5d9c04cca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:08.211 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:08.190 7f5daf7fe700 1 --2- 172.21.2.102:0/1419134118 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5d9c04d5a0 0x7f5d9c04fa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:08.211 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:08.190 7f5daf7fe700 1 -- 172.21.2.102:0/1419134118 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d9c0502e0 con 0x7f5d9c04a7a0 2021-02-10T11:35:08.211 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:08.190 7f5daf7fe700 1 -- 172.21.2.102:0/1419134118 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d9c050500 con 0x7f5d9c003a60 2021-02-10T11:35:08.211 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:08.190 7f5daf7fe700 1 -- 172.21.2.102:0/1419134118 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d9c050720 con 0x7f5d9c04d5a0 2021-02-10T11:35:08.211 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:08.190 7f5daffff700 1 --2- 172.21.2.102:0/1419134118 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5d9c04a7a0 0x7f5d9c04cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:08.211 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:08.190 7f5db50c4700 1 --2- 172.21.2.102:0/1419134118 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5d9c04d5a0 0x7f5d9c04fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:08.211 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:08.190 7f5db48c3700 1 --2- 172.21.2.102:0/1419134118 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5d9c003a60 0x7f5d9c005ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:08.211 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:08.190 7f5daffff700 1 -- 172.21.2.102:0/1419134118 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5d9c04d5a0 msgr2=0x7f5d9c04fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:08.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:08.190 7f5daffff700 1 --2- 172.21.2.102:0/1419134118 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5d9c04d5a0 0x7f5d9c04fa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:08.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:08.190 7f5daffff700 1 -- 172.21.2.102:0/1419134118 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5d9c003a60 msgr2=0x7f5d9c005ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:08.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:08.190 7f5daffff700 1 --2- 172.21.2.102:0/1419134118 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5d9c003a60 0x7f5d9c005ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:08.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:08.190 7f5daffff700 1 -- 172.21.2.102:0/1419134118 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d9c0509c0 con 0x7f5d9c04a7a0 2021-02-10T11:35:08.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:08.191 7f5daffff700 1 --2- 172.21.2.102:0/1419134118 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5d9c04a7a0 0x7f5d9c04cca0 secure :-1 s=READY pgs=2633 cs=0 l=1 rev1=1 rx=0x7f5da0006ac0 tx=0x7f5da0009ab0).ready entity=mon.0 client_cookie=3d40102dfcccf96c server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:08.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:08.191 7f5daf7fe700 1 -- 172.21.2.102:0/1419134118 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5da000c0c0 con 0x7f5d9c04a7a0 2021-02-10T11:35:08.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:08.191 7f5daf7fe700 1 -- 172.21.2.102:0/1419134118 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5da000c3c0 con 0x7f5d9c04a7a0 2021-02-10T11:35:08.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:08.191 7f5daf7fe700 1 -- 172.21.2.102:0/1419134118 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5da000c8a0 con 0x7f5d9c04a7a0 2021-02-10T11:35:08.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:08.191 7f5db6aa7700 1 -- 172.21.2.102:0/1419134118 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5d9c04a7a0 msgr2=0x7f5d9c04cca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:08.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:08.191 7f5db6aa7700 1 --2- 172.21.2.102:0/1419134118 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5d9c04a7a0 0x7f5d9c04cca0 secure :-1 s=READY pgs=2633 cs=0 l=1 rev1=1 rx=0x7f5da0006ac0 tx=0x7f5da0009ab0).stop 2021-02-10T11:35:08.213 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:08.191 7f5db50c4700 1 -- 172.21.2.102:0/1419134118 reap_dead start 2021-02-10T11:35:08.213 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:08.191 7f5db6aa7700 1 -- 172.21.2.102:0/1419134118 shutdown_connections 2021-02-10T11:35:08.213 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:08.191 7f5db6aa7700 1 -- 172.21.2.102:0/1419134118 >> 172.21.2.102:0/1419134118 conn(0x7f5db00ce420 msgr2=0x7f5db00d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:08.213 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:08.191 7f5db6aa7700 1 -- 172.21.2.102:0/1419134118 shutdown_connections 2021-02-10T11:35:08.213 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:08.191 7f5db6aa7700 1 -- 172.21.2.102:0/1419134118 wait complete. 2021-02-10T11:35:08.213 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:08.206 7f5db6aa7700 1 Processor -- start 2021-02-10T11:35:08.213 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:08.206 7f5db6aa7700 1 -- start start 2021-02-10T11:35:08.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:08.207 7f5db6aa7700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5db0085bc0 0x7f5db00863e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:08.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:08.207 7f5db6aa7700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5db00954c0 0x7f5db0096300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:08.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:08.207 7f5db6aa7700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5db00958c0 0x7f5db0096920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:08.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:08.207 7f5db6aa7700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5db0095d80 con 0x7f5db00958c0 2021-02-10T11:35:08.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:08.207 7f5db6aa7700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5db00bfa60 con 0x7f5db00954c0 2021-02-10T11:35:08.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:08.207 7f5db6aa7700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5db0085fc0 con 0x7f5db0085bc0 2021-02-10T11:35:08.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:08.207 7f5db50c4700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5db00958c0 0x7f5db0096920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:08.216 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:08.207 7f5db50c4700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5db00958c0 0x7f5db0096920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:36658/0 (socket says 172.21.2.102:36658) 2021-02-10T11:35:08.216 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:08.207 7f5db50c4700 1 -- 172.21.2.102:0/3680247923 learned_addr learned my addr 172.21.2.102:0/3680247923 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:35:08.216 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:08.207 7f5db50c4700 1 -- 172.21.2.102:0/3680247923 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5db0085bc0 msgr2=0x7f5db00863e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:08.216 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:08.207 7f5db50c4700 1 --2- 172.21.2.102:0/3680247923 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5db0085bc0 0x7f5db00863e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:08.216 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:08.207 7f5db50c4700 1 -- 172.21.2.102:0/3680247923 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5db00954c0 msgr2=0x7f5db0096300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:08.216 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:08.207 7f5db50c4700 1 --2- 172.21.2.102:0/3680247923 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5db00954c0 0x7f5db0096300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:08.216 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:08.207 7f5db50c4700 1 -- 172.21.2.102:0/3680247923 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5db0086c10 con 0x7f5db00958c0 2021-02-10T11:35:08.216 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:08.208 7f5db50c4700 1 --2- 172.21.2.102:0/3680247923 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5db00958c0 0x7f5db0096920 secure :-1 s=READY pgs=2634 cs=0 l=1 rev1=1 rx=0x7f5da800a940 tx=0x7f5da800ace0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:08.216 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:08.208 7f5dad7fa700 1 -- 172.21.2.102:0/3680247923 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5da80013f0 con 0x7f5db00958c0 2021-02-10T11:35:08.216 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:08.208 7f5dad7fa700 1 -- 172.21.2.102:0/3680247923 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5da8007020 con 0x7f5db00958c0 2021-02-10T11:35:08.216 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:08.208 7f5dad7fa700 1 -- 172.21.2.102:0/3680247923 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5da8007540 con 0x7f5db00958c0 2021-02-10T11:35:08.217 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:08.208 7f5db6aa7700 1 -- 172.21.2.102:0/3680247923 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5db0086c10 con 0x7f5db00958c0 2021-02-10T11:35:08.217 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:08.208 7f5db6aa7700 1 -- 172.21.2.102:0/3680247923 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5db0097230 con 0x7f5db00958c0 2021-02-10T11:35:08.217 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:08.209 7f5dad7fa700 1 -- 172.21.2.102:0/3680247923 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f5da800ae20 con 0x7f5db00958c0 2021-02-10T11:35:08.217 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:08.209 7f5dad7fa700 1 --2- 172.21.2.102:0/3680247923 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5d940186c0 0x7f5d9401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:08.217 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:08.209 7f5dad7fa700 1 -- 172.21.2.102:0/3680247923 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 4753+0+0 (secure 0 0 0) 0x7f5da80325d0 con 0x7f5db00958c0 2021-02-10T11:35:08.217 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:08.209 7f5db48c3700 1 --2- 172.21.2.102:0/3680247923 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5d940186c0 0x7f5d9401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:08.217 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:08.209 7f5db48c3700 1 --2- 172.21.2.102:0/3680247923 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5d940186c0 0x7f5d9401abc0 crc :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:08.240 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:08.238 7f5db6aa7700 1 -- 172.21.2.102:0/3680247923 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5db00fe1c0 con 0x7f5db00958c0 2021-02-10T11:35:08.243 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:08.241 7f5dad7fa700 1 -- 172.21.2.102:0/3680247923 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f5da80072a0 con 0x7f5db00958c0 2021-02-10T11:35:08.370 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:08.368 7f5db6aa7700 1 -- 172.21.2.102:0/3680247923 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f5db00fe1c0 con 0x7f5db00958c0 2021-02-10T11:35:08.370 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:08.368 7f5dad7fa700 1 -- 172.21.2.102:0/3680247923 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v141) v1 ==== 92+0+10 (secure 0 0 0) 0x7f5da800ae20 con 0x7f5db00958c0 2021-02-10T11:35:08.371 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:35:08.372 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:08.370 7f5db6aa7700 1 -- 172.21.2.102:0/3680247923 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5d940186c0 msgr2=0x7f5d9401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:08.372 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:08.370 7f5db6aa7700 1 --2- 172.21.2.102:0/3680247923 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5d940186c0 0x7f5d9401abc0 crc :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:08.372 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:08.370 7f5db6aa7700 1 -- 172.21.2.102:0/3680247923 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5db00958c0 msgr2=0x7f5db0096920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:08.372 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:08.370 7f5db6aa7700 1 --2- 172.21.2.102:0/3680247923 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5db00958c0 0x7f5db0096920 secure :-1 s=READY pgs=2634 cs=0 l=1 rev1=1 rx=0x7f5da800a940 tx=0x7f5da800ace0).stop 2021-02-10T11:35:08.373 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:08.370 7f5db6aa7700 1 -- 172.21.2.102:0/3680247923 shutdown_connections 2021-02-10T11:35:08.373 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:08.370 7f5db6aa7700 1 --2- 172.21.2.102:0/3680247923 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5db00958c0 0x7f5db0096920 unknown :-1 s=CLOSED pgs=2634 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:08.373 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:08.370 7f5db6aa7700 1 --2- 172.21.2.102:0/3680247923 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5d940186c0 0x7f5d9401abc0 unknown :-1 s=CLOSED pgs=1161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:08.373 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:08.370 7f5db6aa7700 1 --2- 172.21.2.102:0/3680247923 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5db00954c0 0x7f5db0096300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:08.373 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:08.370 7f5db6aa7700 1 --2- 172.21.2.102:0/3680247923 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5db0085bc0 0x7f5db00863e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:08.373 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:08.370 7f5db6aa7700 1 -- 172.21.2.102:0/3680247923 >> 172.21.2.102:0/3680247923 conn(0x7f5db00bef00 msgr2=0x7f5db009d1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:08.373 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:08.370 7f5db6aa7700 1 -- 172.21.2.102:0/3680247923 shutdown_connections 2021-02-10T11:35:08.373 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:08.371 7f5db6aa7700 1 -- 172.21.2.102:0/3680247923 wait complete. 2021-02-10T11:35:08.381 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --format=json-pretty osd lspools 2021-02-10T11:35:08.811 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:08.792 7ffa29a85700 1 Processor -- start 2021-02-10T11:35:08.811 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:08.792 7ffa29a85700 1 -- start start 2021-02-10T11:35:08.811 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:08.793 7ffa29a85700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7ffa240d0340 0x7ffa240ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:08.812 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:08.793 7ffa29a85700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7ffa240d1c70 0x7ffa240efc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:08.812 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:08.793 7ffa29a85700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa240f0b10 con 0x7ffa240d1870 2021-02-10T11:35:08.812 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:08.793 7ffa29a85700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa240e5ed0 con 0x7ffa240d1c70 2021-02-10T11:35:08.812 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:08.793 7ffa29a85700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa241b0be0 con 0x7ffa240d0340 2021-02-10T11:35:08.812 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:08.793 7ffa23fff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7ffa240d1c70 0x7ffa240efc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:08.813 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:08.793 7ffa23fff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7ffa240d1c70 0x7ffa240efc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:36670/0 (socket says 172.21.2.102:36670) 2021-02-10T11:35:08.813 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:08.793 7ffa23fff700 1 -- 172.21.2.102:0/2263943071 learned_addr learned my addr 172.21.2.102:0/2263943071 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:35:08.813 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:08.793 7ffa23fff700 1 -- 172.21.2.102:0/2263943071 >> v2:172.21.2.113:3300/0 conn(0x7ffa240d0340 msgr2=0x7ffa240ef640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:08.813 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:08.793 7ffa23fff700 1 --2- 172.21.2.102:0/2263943071 >> v2:172.21.2.113:3300/0 conn(0x7ffa240d0340 0x7ffa240ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:08.813 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:08.793 7ffa23fff700 1 -- 172.21.2.102:0/2263943071 >> v1:172.21.2.113:6789/0 conn(0x7ffa240d1870 legacy=0x7ffa241b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:08.814 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:08.793 7ffa23fff700 1 -- 172.21.2.102:0/2263943071 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa240f20a0 con 0x7ffa240d1c70 2021-02-10T11:35:08.814 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:08.794 7ffa23fff700 1 --2- 172.21.2.102:0/2263943071 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ffa240d1c70 0x7ffa240efc60 secure :-1 s=READY pgs=2635 cs=0 l=1 rev1=1 rx=0x7ffa1c006f00 tx=0x7ffa1c009f10).ready entity=mon.0 client_cookie=86db39baa9eb7548 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:08.814 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:08.794 7ffa227fc700 1 -- 172.21.2.102:0/2263943071 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ffa1c00b3b0 con 0x7ffa240d1c70 2021-02-10T11:35:08.814 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:08.794 7ffa227fc700 1 -- 172.21.2.102:0/2263943071 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa1c00b670 con 0x7ffa240d1c70 2021-02-10T11:35:08.814 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:08.794 7ffa227fc700 1 -- 172.21.2.102:0/2263943071 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ffa1c00bb90 con 0x7ffa240d1c70 2021-02-10T11:35:08.815 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:08.794 7ffa29a85700 1 -- 172.21.2.102:0/2263943071 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ffa240d1c70 msgr2=0x7ffa240efc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:08.815 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:08.794 7ffa29a85700 1 --2- 172.21.2.102:0/2263943071 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ffa240d1c70 0x7ffa240efc60 secure :-1 s=READY pgs=2635 cs=0 l=1 rev1=1 rx=0x7ffa1c006f00 tx=0x7ffa1c009f10).stop 2021-02-10T11:35:08.815 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:08.794 7ffa29a85700 1 -- 172.21.2.102:0/2263943071 shutdown_connections 2021-02-10T11:35:08.815 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:08.794 7ffa29a85700 1 --2- 172.21.2.102:0/2263943071 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ffa240d1c70 0x7ffa240efc60 unknown :-1 s=CLOSED pgs=2635 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:08.815 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:08.794 7ffa29a85700 1 --2- 172.21.2.102:0/2263943071 >> v2:172.21.2.113:3300/0 conn(0x7ffa240d0340 0x7ffa240ef640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:08.816 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:08.794 7ffa29a85700 1 -- 172.21.2.102:0/2263943071 >> 172.21.2.102:0/2263943071 conn(0x7ffa240ce450 msgr2=0x7ffa240d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:08.816 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:08.795 7ffa29a85700 1 -- 172.21.2.102:0/2263943071 shutdown_connections 2021-02-10T11:35:08.816 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:08.795 7ffa29a85700 1 -- 172.21.2.102:0/2263943071 wait complete. 2021-02-10T11:35:08.816 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:08.809 7ffa29a85700 1 Processor -- start 2021-02-10T11:35:08.816 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:08.809 7ffa29a85700 1 -- start start 2021-02-10T11:35:08.819 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:08.809 7ffa29a85700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ffa2419a3d0 0x7ffa241ab160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:08.819 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:08.809 7ffa29a85700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ffa241ab780 0x7ffa240b3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:08.819 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:08.809 7ffa29a85700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ffa241abb80 0x7ffa240b3c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:08.819 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:08.809 7ffa29a85700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa241ae200 con 0x7ffa2419a3d0 2021-02-10T11:35:08.819 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:08.809 7ffa29a85700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa24092930 con 0x7ffa241ab780 2021-02-10T11:35:08.820 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:08.809 7ffa29a85700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa240b4290 con 0x7ffa241abb80 2021-02-10T11:35:08.820 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:08.810 7ffa237fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ffa2419a3d0 0x7ffa241ab160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:08.820 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:08.810 7ffa237fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ffa2419a3d0 0x7ffa241ab160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:36674/0 (socket says 172.21.2.102:36674) 2021-02-10T11:35:08.820 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:08.810 7ffa237fe700 1 -- 172.21.2.102:0/558535067 learned_addr learned my addr 172.21.2.102:0/558535067 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:35:08.820 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:08.810 7ffa237fe700 1 -- 172.21.2.102:0/558535067 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ffa241abb80 msgr2=0x7ffa240b3c70 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:35:08.820 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:08.810 7ffa237fe700 1 --2- 172.21.2.102:0/558535067 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ffa241abb80 0x7ffa240b3c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:08.821 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:08.810 7ffa237fe700 1 -- 172.21.2.102:0/558535067 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ffa241ab780 msgr2=0x7ffa240b3650 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:35:08.821 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:08.810 7ffa237fe700 1 --2- 172.21.2.102:0/558535067 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ffa241ab780 0x7ffa240b3650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:08.821 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:08.810 7ffa237fe700 1 -- 172.21.2.102:0/558535067 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa240ee360 con 0x7ffa2419a3d0 2021-02-10T11:35:08.821 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:08.810 7ffa237fe700 1 --2- 172.21.2.102:0/558535067 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ffa2419a3d0 0x7ffa241ab160 secure :-1 s=READY pgs=2636 cs=0 l=1 rev1=1 rx=0x7ffa1400b5b0 tx=0x7ffa1400b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:08.821 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:08.810 7ffa1bfff700 1 -- 172.21.2.102:0/558535067 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ffa1400bd40 con 0x7ffa2419a3d0 2021-02-10T11:35:08.822 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:08.810 7ffa1bfff700 1 -- 172.21.2.102:0/558535067 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa14006240 con 0x7ffa2419a3d0 2021-02-10T11:35:08.822 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:08.810 7ffa1bfff700 1 -- 172.21.2.102:0/558535067 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ffa14005720 con 0x7ffa2419a3d0 2021-02-10T11:35:08.822 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:08.810 7ffa29a85700 1 -- 172.21.2.102:0/558535067 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa240ee360 con 0x7ffa2419a3d0 2021-02-10T11:35:08.822 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:08.810 7ffa29a85700 1 -- 172.21.2.102:0/558535067 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa240ee360 con 0x7ffa2419a3d0 2021-02-10T11:35:08.822 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:08.811 7ffa1bfff700 1 -- 172.21.2.102:0/558535067 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ffa140054a0 con 0x7ffa2419a3d0 2021-02-10T11:35:08.822 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:08.811 7ffa1bfff700 1 --2- 172.21.2.102:0/558535067 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ffa080186c0 0x7ffa0801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:08.823 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:08.811 7ffa1bfff700 1 -- 172.21.2.102:0/558535067 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 4753+0+0 (secure 0 0 0) 0x7ffa14032280 con 0x7ffa2419a3d0 2021-02-10T11:35:08.823 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:08.811 7ffa22ffd700 1 --2- 172.21.2.102:0/558535067 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ffa080186c0 0x7ffa0801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:08.823 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:08.811 7ffa22ffd700 1 --2- 172.21.2.102:0/558535067 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ffa080186c0 0x7ffa0801abc0 crc :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:08.845 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:08.843 7ffa29a85700 1 -- 172.21.2.102:0/558535067 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa240ee080 con 0x7ffa2419a3d0 2021-02-10T11:35:08.847 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:08.845 7ffa1bfff700 1 -- 172.21.2.102:0/558535067 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ffa140009b0 con 0x7ffa2419a3d0 2021-02-10T11:35:08.971 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:08.969 7ffa29a85700 1 -- 172.21.2.102:0/558535067 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7ffa240ce450 con 0x7ffa2419a3d0 2021-02-10T11:35:08.972 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:08.970 7ffa1bfff700 1 -- 172.21.2.102:0/558535067 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v141) v1 ==== 84+0+64 (secure 0 0 0) 0x7ffa1400bd40 con 0x7ffa2419a3d0 2021-02-10T11:35:08.972 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:35:08.972 INFO:teuthology.orchestra.run.gibba002.stdout:[ 2021-02-10T11:35:08.973 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:35:08.973 INFO:teuthology.orchestra.run.gibba002.stdout: "poolnum": 1, 2021-02-10T11:35:08.973 INFO:teuthology.orchestra.run.gibba002.stdout: "poolname": "rbd" 2021-02-10T11:35:08.973 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:35:08.973 INFO:teuthology.orchestra.run.gibba002.stdout:] 2021-02-10T11:35:08.974 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:08.972 7ffa29a85700 1 -- 172.21.2.102:0/558535067 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ffa080186c0 msgr2=0x7ffa0801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:08.974 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:08.972 7ffa29a85700 1 --2- 172.21.2.102:0/558535067 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ffa080186c0 0x7ffa0801abc0 crc :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:08.974 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:08.972 7ffa29a85700 1 -- 172.21.2.102:0/558535067 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ffa2419a3d0 msgr2=0x7ffa241ab160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:08.974 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:08.972 7ffa29a85700 1 --2- 172.21.2.102:0/558535067 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ffa2419a3d0 0x7ffa241ab160 secure :-1 s=READY pgs=2636 cs=0 l=1 rev1=1 rx=0x7ffa1400b5b0 tx=0x7ffa1400b950).stop 2021-02-10T11:35:08.974 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:08.972 7ffa29a85700 1 -- 172.21.2.102:0/558535067 shutdown_connections 2021-02-10T11:35:08.974 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:08.972 7ffa29a85700 1 --2- 172.21.2.102:0/558535067 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ffa241abb80 0x7ffa240b3c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:08.974 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:08.972 7ffa29a85700 1 --2- 172.21.2.102:0/558535067 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ffa080186c0 0x7ffa0801abc0 unknown :-1 s=CLOSED pgs=1162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:08.974 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:08.972 7ffa29a85700 1 --2- 172.21.2.102:0/558535067 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ffa241ab780 0x7ffa240b3650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:08.974 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:08.972 7ffa29a85700 1 --2- 172.21.2.102:0/558535067 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ffa2419a3d0 0x7ffa241ab160 unknown :-1 s=CLOSED pgs=2636 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:08.974 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:08.972 7ffa29a85700 1 -- 172.21.2.102:0/558535067 >> 172.21.2.102:0/558535067 conn(0x7ffa24093640 msgr2=0x7ffa2418eb00 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:08.975 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:08.972 7ffa29a85700 1 -- 172.21.2.102:0/558535067 shutdown_connections 2021-02-10T11:35:08.975 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:08.972 7ffa29a85700 1 -- 172.21.2.102:0/558535067 wait complete. 2021-02-10T11:35:08.982 INFO:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2021-02-10T11:35:08.982 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool create cephfs_metadata 8 2021-02-10T11:35:09.414 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:09.396 7f5e8fa72700 1 Processor -- start 2021-02-10T11:35:09.414 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:09.396 7f5e8fa72700 1 -- start start 2021-02-10T11:35:09.414 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:09.397 7f5e8fa72700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f5e880d1c70 0x7f5e880cef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:09.415 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:09.397 7f5e8fa72700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e880d0170 con 0x7f5e880cf580 2021-02-10T11:35:09.415 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:09.397 7f5e8fa72700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e880fe1c0 con 0x7f5e880d1870 2021-02-10T11:35:09.415 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:09.397 7f5e8fa72700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e881b0cb0 con 0x7f5e880d1c70 2021-02-10T11:35:09.415 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:09.397 7f5e8d08d700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f5e880d1c70 0x7f5e880cef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:09.415 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:09.397 7f5e8e08f700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f5e880cf580 0x7f5e880cf980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:57136/0 (socket says 172.21.2.102:57136) 2021-02-10T11:35:09.415 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:09.397 7f5e8e08f700 1 -- 172.21.2.102:0/4283117524 learned_addr learned my addr 172.21.2.102:0/4283117524 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:35:09.415 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:09.397 7f5e8d08d700 1 -- 172.21.2.102:0/4283117524 >> v1:172.21.2.113:6789/0 conn(0x7f5e880d1870 legacy=0x7f5e881b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:09.415 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:09.397 7f5e8d08d700 1 -- 172.21.2.102:0/4283117524 >> v1:172.21.2.102:6789/0 conn(0x7f5e880cf580 legacy=0x7f5e880cf980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:09.415 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:09.397 7f5e8d08d700 1 -- 172.21.2.102:0/4283117524 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e88109a60 con 0x7f5e880d1c70 2021-02-10T11:35:09.415 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:09.398 7f5e8d08d700 1 --2- 172.21.2.102:0/4283117524 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5e880d1c70 0x7f5e880cef60 secure :-1 s=READY pgs=2638 cs=0 l=1 rev1=1 rx=0x7f5e7c006650 tx=0x7f5e7c009a80).ready entity=mon.0 client_cookie=39ff8aa787d289b5 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:09.415 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:09.398 7f5e8c88c700 1 -- 172.21.2.102:0/4283117524 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5e7c0009b0 con 0x7f5e880d1c70 2021-02-10T11:35:09.415 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:09.398 7f5e8c88c700 1 -- 172.21.2.102:0/4283117524 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e7c00c0c0 con 0x7f5e880d1c70 2021-02-10T11:35:09.415 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:09.398 7f5e8c88c700 1 -- 172.21.2.102:0/4283117524 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5e7c00c520 con 0x7f5e880d1c70 2021-02-10T11:35:09.416 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:09.398 7f5e8fa72700 1 -- 172.21.2.102:0/4283117524 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5e880d1c70 msgr2=0x7f5e880cef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:09.416 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:09.398 7f5e8fa72700 1 --2- 172.21.2.102:0/4283117524 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5e880d1c70 0x7f5e880cef60 secure :-1 s=READY pgs=2638 cs=0 l=1 rev1=1 rx=0x7f5e7c006650 tx=0x7f5e7c009a80).stop 2021-02-10T11:35:09.416 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:09.398 7f5e8fa72700 1 -- 172.21.2.102:0/4283117524 shutdown_connections 2021-02-10T11:35:09.416 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:09.398 7f5e8fa72700 1 --2- 172.21.2.102:0/4283117524 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5e880d1c70 0x7f5e880cef60 unknown :-1 s=CLOSED pgs=2638 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:09.416 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:09.398 7f5e8fa72700 1 -- 172.21.2.102:0/4283117524 >> 172.21.2.102:0/4283117524 conn(0x7f5e880ce420 msgr2=0x7f5e880d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:09.416 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:09.399 7f5e8fa72700 1 -- 172.21.2.102:0/4283117524 shutdown_connections 2021-02-10T11:35:09.416 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:09.399 7f5e8fa72700 1 -- 172.21.2.102:0/4283117524 wait complete. 2021-02-10T11:35:09.417 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:09.413 7f5e8fa72700 1 Processor -- start 2021-02-10T11:35:09.418 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:09.413 7f5e8fa72700 1 -- start start 2021-02-10T11:35:09.418 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:09.413 7f5e8fa72700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5e880a25d0 0x7f5e880a29d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:09.418 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:09.413 7f5e8fa72700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5e880a2ff0 0x7f5e880a33f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:09.418 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:09.413 7f5e8fa72700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5e880b5650 0x7f5e880b5a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:09.418 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:09.413 7f5e8fa72700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e881c3780 con 0x7f5e880a2ff0 2021-02-10T11:35:09.418 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:09.413 7f5e8fa72700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e880a20c0 con 0x7f5e880b5650 2021-02-10T11:35:09.418 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:09.413 7f5e8fa72700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e880b60a0 con 0x7f5e880a25d0 2021-02-10T11:35:09.418 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:09.413 7f5e8d08d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5e880a2ff0 0x7f5e880a33f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:09.418 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:09.413 7f5e8d08d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5e880a2ff0 0x7f5e880a33f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:36686/0 (socket says 172.21.2.102:36686) 2021-02-10T11:35:09.418 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:09.414 7f5e8d08d700 1 -- 172.21.2.102:0/784703861 learned_addr learned my addr 172.21.2.102:0/784703861 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:35:09.418 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:09.414 7f5e8d08d700 1 -- 172.21.2.102:0/784703861 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5e880a25d0 msgr2=0x7f5e880a29d0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:35:09.418 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:09.414 7f5e8d08d700 1 --2- 172.21.2.102:0/784703861 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5e880a25d0 0x7f5e880a29d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:09.419 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:09.414 7f5e8d08d700 1 -- 172.21.2.102:0/784703861 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5e880b5650 msgr2=0x7f5e880b5a50 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:35:09.419 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:09.414 7f5e8d08d700 1 --2- 172.21.2.102:0/784703861 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5e880b5650 0x7f5e880b5a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:09.419 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:09.414 7f5e8d08d700 1 -- 172.21.2.102:0/784703861 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e880b62c0 con 0x7f5e880a2ff0 2021-02-10T11:35:09.419 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:09.414 7f5e8d08d700 1 --2- 172.21.2.102:0/784703861 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5e880a2ff0 0x7f5e880a33f0 secure :-1 s=READY pgs=2639 cs=0 l=1 rev1=1 rx=0x7f5e7c009630 tx=0x7f5e7c00b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:09.419 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:09.414 7f5e7a7fc700 1 -- 172.21.2.102:0/784703861 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5e7c0093a0 con 0x7f5e880a2ff0 2021-02-10T11:35:09.419 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:09.414 7f5e7a7fc700 1 -- 172.21.2.102:0/784703861 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e7c00bd70 con 0x7f5e880a2ff0 2021-02-10T11:35:09.419 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:09.414 7f5e8fa72700 1 -- 172.21.2.102:0/784703861 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e880b62c0 con 0x7f5e880a2ff0 2021-02-10T11:35:09.419 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:09.414 7f5e7a7fc700 1 -- 172.21.2.102:0/784703861 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5e7c005560 con 0x7f5e880a2ff0 2021-02-10T11:35:09.419 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:09.414 7f5e8fa72700 1 -- 172.21.2.102:0/784703861 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e880b62c0 con 0x7f5e880a2ff0 2021-02-10T11:35:09.419 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:09.415 7f5e7a7fc700 1 -- 172.21.2.102:0/784703861 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f5e7c005740 con 0x7f5e880a2ff0 2021-02-10T11:35:09.419 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:09.415 7f5e7a7fc700 1 --2- 172.21.2.102:0/784703861 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5e70018730 0x7f5e7001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:09.419 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:09.415 7f5e7a7fc700 1 -- 172.21.2.102:0/784703861 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 4753+0+0 (secure 0 0 0) 0x7f5e7c032470 con 0x7f5e880a2ff0 2021-02-10T11:35:09.419 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:09.415 7f5e8d88e700 1 --2- 172.21.2.102:0/784703861 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5e70018730 0x7f5e7001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:09.420 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:09.415 7f5e8d88e700 1 --2- 172.21.2.102:0/784703861 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5e70018730 0x7f5e7001abc0 crc :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:09.448 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:09.446 7f5e8fa72700 1 -- 172.21.2.102:0/784703861 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e880b6eb0 con 0x7f5e880a2ff0 2021-02-10T11:35:09.451 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:09.449 7f5e7a7fc700 1 -- 172.21.2.102:0/784703861 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f5e7c01f720 con 0x7f5e880a2ff0 2021-02-10T11:35:09.578 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:09.576 7f5e8fa72700 1 -- 172.21.2.102:0/784703861 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"} v 0) v1 -- 0x7f5e880b6eb0 con 0x7f5e880a2ff0 2021-02-10T11:35:09.989 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:09.987 7f5e7a7fc700 1 -- 172.21.2.102:0/784703861 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"}]=0 pool 'cephfs_metadata' created v142) v1 ==== 133+0+0 (secure 0 0 0) 0x7f5e7c01f4e0 con 0x7f5e880a2ff0 2021-02-10T11:35:09.994 INFO:teuthology.orchestra.run.gibba002.stderr:pool 'cephfs_metadata' created 2021-02-10T11:35:09.995 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:09.993 7f5e8fa72700 1 -- 172.21.2.102:0/784703861 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5e70018730 msgr2=0x7f5e7001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:09.996 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:09.993 7f5e8fa72700 1 --2- 172.21.2.102:0/784703861 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5e70018730 0x7f5e7001abc0 crc :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:09.996 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:09.993 7f5e8fa72700 1 -- 172.21.2.102:0/784703861 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5e880a2ff0 msgr2=0x7f5e880a33f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:09.996 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:09.993 7f5e8fa72700 1 --2- 172.21.2.102:0/784703861 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5e880a2ff0 0x7f5e880a33f0 secure :-1 s=READY pgs=2639 cs=0 l=1 rev1=1 rx=0x7f5e7c009630 tx=0x7f5e7c00b980).stop 2021-02-10T11:35:09.996 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:09.993 7f5e8fa72700 1 -- 172.21.2.102:0/784703861 shutdown_connections 2021-02-10T11:35:09.996 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:09.993 7f5e8fa72700 1 --2- 172.21.2.102:0/784703861 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5e70018730 0x7f5e7001abc0 unknown :-1 s=CLOSED pgs=1163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:09.996 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:09.993 7f5e8fa72700 1 --2- 172.21.2.102:0/784703861 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5e880b5650 0x7f5e880b5a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:09.996 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:09.993 7f5e8fa72700 1 --2- 172.21.2.102:0/784703861 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5e880a2ff0 0x7f5e880a33f0 unknown :-1 s=CLOSED pgs=2639 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:09.996 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:09.993 7f5e8fa72700 1 --2- 172.21.2.102:0/784703861 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5e880a25d0 0x7f5e880a29d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:09.996 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:09.993 7f5e8fa72700 1 -- 172.21.2.102:0/784703861 >> 172.21.2.102:0/784703861 conn(0x7f5e880c4120 msgr2=0x7f5e880ffd60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:09.996 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:09.994 7f5e8fa72700 1 -- 172.21.2.102:0/784703861 shutdown_connections 2021-02-10T11:35:09.996 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:09.994 7f5e8fa72700 1 -- 172.21.2.102:0/784703861 wait complete. 2021-02-10T11:35:10.004 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool create cephfs_data 8 2021-02-10T11:35:10.436 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:10.418 7f8105c02700 1 Processor -- start 2021-02-10T11:35:10.437 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:10.418 7f8105c02700 1 -- start start 2021-02-10T11:35:10.437 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:10.418 7f8105c02700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f81000d1870 0x7f81001b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:10.437 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:10.418 7f8105c02700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f81001b0ae0 0x7f8100108940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:10.437 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:10.418 7f8105c02700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8100109a60 con 0x7f81000d1c70 2021-02-10T11:35:10.437 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:10.418 7f8105c02700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81000e5ef0 con 0x7f81001b0ae0 2021-02-10T11:35:10.437 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:10.418 7f8105c02700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8100109750 con 0x7f81000d1870 2021-02-10T11:35:10.437 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:10.419 7f80fffff700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f81000d1c70 0x7f8100108f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:57152/0 (socket says 172.21.2.102:57152) 2021-02-10T11:35:10.437 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:10.419 7f80fffff700 1 -- 172.21.2.102:0/670778140 learned_addr learned my addr 172.21.2.102:0/670778140 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:35:10.437 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:10.419 7f80feffd700 1 --2- 172.21.2.102:0/670778140 >> v2:172.21.2.102:3300/0 conn(0x7f81001b0ae0 0x7f8100108940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:10.437 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:10.419 7f80feffd700 1 -- 172.21.2.102:0/670778140 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f81000d1870 msgr2=0x7f81001b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:10.437 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:10.419 7f80feffd700 1 --2- 172.21.2.102:0/670778140 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f81000d1870 0x7f81001b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:10.437 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:10.419 7f80feffd700 1 -- 172.21.2.102:0/670778140 >> v1:172.21.2.102:6789/0 conn(0x7f81000d1c70 legacy=0x7f8100108f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:10.437 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:10.419 7f80feffd700 1 -- 172.21.2.102:0/670778140 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8100088070 con 0x7f81001b0ae0 2021-02-10T11:35:10.438 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:10.419 7f80feffd700 1 --2- 172.21.2.102:0/670778140 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f81001b0ae0 0x7f8100108940 secure :-1 s=READY pgs=2641 cs=0 l=1 rev1=1 rx=0x7f80e8006650 tx=0x7f80e8009a80).ready entity=mon.0 client_cookie=72cfb429f93d5027 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:10.438 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:10.419 7f80fe7fc700 1 -- 172.21.2.102:0/670778140 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f80e800c0c0 con 0x7f81001b0ae0 2021-02-10T11:35:10.438 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:10.419 7f80fe7fc700 1 -- 172.21.2.102:0/670778140 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80e800c340 con 0x7f81001b0ae0 2021-02-10T11:35:10.438 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:10.419 7f80fe7fc700 1 -- 172.21.2.102:0/670778140 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f80e800c860 con 0x7f81001b0ae0 2021-02-10T11:35:10.438 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:10.419 7f8105c02700 1 -- 172.21.2.102:0/670778140 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f81001b0ae0 msgr2=0x7f8100108940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:10.438 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:10.419 7f8105c02700 1 --2- 172.21.2.102:0/670778140 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f81001b0ae0 0x7f8100108940 secure :-1 s=READY pgs=2641 cs=0 l=1 rev1=1 rx=0x7f80e8006650 tx=0x7f80e8009a80).stop 2021-02-10T11:35:10.438 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:10.420 7f8105c02700 1 -- 172.21.2.102:0/670778140 shutdown_connections 2021-02-10T11:35:10.438 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:10.420 7f8105c02700 1 --2- 172.21.2.102:0/670778140 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f81001b0ae0 0x7f8100108940 unknown :-1 s=CLOSED pgs=2641 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:10.438 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:10.420 7f8105c02700 1 --2- 172.21.2.102:0/670778140 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f81000d1870 0x7f81001b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:10.438 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:10.420 7f8105c02700 1 -- 172.21.2.102:0/670778140 >> 172.21.2.102:0/670778140 conn(0x7f81000ce420 msgr2=0x7f81000d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:10.438 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:10.420 7f8105c02700 1 -- 172.21.2.102:0/670778140 shutdown_connections 2021-02-10T11:35:10.439 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:10.420 7f8105c02700 1 -- 172.21.2.102:0/670778140 wait complete. 2021-02-10T11:35:10.440 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:10.435 7f8105c02700 1 Processor -- start 2021-02-10T11:35:10.440 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:10.435 7f8105c02700 1 -- start start 2021-02-10T11:35:10.440 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:10.435 7f8105c02700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f81000c3820 0x7f81000a7950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:10.441 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:10.435 7f8105c02700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f81001a44c0 0x7f81001a48c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:10.441 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:10.435 7f8105c02700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f81000863d0 0x7f81000867d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:10.441 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:10.435 7f8105c02700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8100085ed0 con 0x7f81001a44c0 2021-02-10T11:35:10.441 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:10.435 7f8105c02700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81000c0740 con 0x7f81000c3820 2021-02-10T11:35:10.441 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:10.435 7f8105c02700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8100086df0 con 0x7f81000863d0 2021-02-10T11:35:10.441 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:10.436 7f80feffd700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f81001a44c0 0x7f81001a48c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:10.441 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:10.436 7f80feffd700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f81001a44c0 0x7f81001a48c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:36700/0 (socket says 172.21.2.102:36700) 2021-02-10T11:35:10.441 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:10.436 7f80feffd700 1 -- 172.21.2.102:0/1804916593 learned_addr learned my addr 172.21.2.102:0/1804916593 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:35:10.441 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:10.436 7f80feffd700 1 -- 172.21.2.102:0/1804916593 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f81000863d0 msgr2=0x7f81000867d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:10.441 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:10.436 7f80feffd700 1 --2- 172.21.2.102:0/1804916593 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f81000863d0 0x7f81000867d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:10.441 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:10.436 7f80feffd700 1 -- 172.21.2.102:0/1804916593 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f81000c3820 msgr2=0x7f81000a7950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:10.441 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:10.436 7f80feffd700 1 --2- 172.21.2.102:0/1804916593 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f81000c3820 0x7f81000a7950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:10.441 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:10.436 7f80feffd700 1 -- 172.21.2.102:0/1804916593 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8100087010 con 0x7f81001a44c0 2021-02-10T11:35:10.442 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:10.436 7f80feffd700 1 --2- 172.21.2.102:0/1804916593 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f81001a44c0 0x7f81001a48c0 secure :-1 s=READY pgs=2642 cs=0 l=1 rev1=1 rx=0x7f80e8009630 tx=0x7f80e800b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:10.442 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:10.436 7f80f7fff700 1 -- 172.21.2.102:0/1804916593 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f80e80093a0 con 0x7f81001a44c0 2021-02-10T11:35:10.442 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:10.436 7f80f7fff700 1 -- 172.21.2.102:0/1804916593 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80e800bd70 con 0x7f81001a44c0 2021-02-10T11:35:10.442 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:10.437 7f80f7fff700 1 -- 172.21.2.102:0/1804916593 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f80e80054d0 con 0x7f81001a44c0 2021-02-10T11:35:10.442 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:10.437 7f8105c02700 1 -- 172.21.2.102:0/1804916593 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8100087010 con 0x7f81001a44c0 2021-02-10T11:35:10.442 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:10.437 7f8105c02700 1 -- 172.21.2.102:0/1804916593 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8100083000 con 0x7f81001a44c0 2021-02-10T11:35:10.442 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:10.437 7f80f7fff700 1 -- 172.21.2.102:0/1804916593 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f80e8007020 con 0x7f81001a44c0 2021-02-10T11:35:10.442 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:10.437 7f80f7fff700 1 --2- 172.21.2.102:0/1804916593 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f80e40186c0 0x7f80e401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:10.442 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:10.437 7f80f7fff700 1 -- 172.21.2.102:0/1804916593 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(142..142 src has 1..142) v4 ==== 5107+0+0 (secure 0 0 0) 0x7f80e8032210 con 0x7f81001a44c0 2021-02-10T11:35:10.442 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:10.437 7f80ff7fe700 1 --2- 172.21.2.102:0/1804916593 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f80e40186c0 0x7f80e401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:10.442 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:10.438 7f80ff7fe700 1 --2- 172.21.2.102:0/1804916593 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f80e40186c0 0x7f80e401abc0 crc :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:10.470 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:10.468 7f8105c02700 1 -- 172.21.2.102:0/1804916593 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81000fe1c0 con 0x7f81001a44c0 2021-02-10T11:35:10.473 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:10.471 7f80f7fff700 1 -- 172.21.2.102:0/1804916593 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f80e8009340 con 0x7f81001a44c0 2021-02-10T11:35:10.602 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:10.601 7f8105c02700 1 -- 172.21.2.102:0/1804916593 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_data"} v 0) v1 -- 0x7f81000fe1c0 con 0x7f81001a44c0 2021-02-10T11:35:10.992 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:10.990 7f80f7fff700 1 -- 172.21.2.102:0/1804916593 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_data"}]=0 pool 'cephfs_data' created v143) v1 ==== 125+0+0 (secure 0 0 0) 0x7f80e8007020 con 0x7f81001a44c0 2021-02-10T11:35:11.019 INFO:teuthology.orchestra.run.gibba002.stderr:pool 'cephfs_data' created 2021-02-10T11:35:11.021 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:11.018 7f8105c02700 1 -- 172.21.2.102:0/1804916593 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f80e40186c0 msgr2=0x7f80e401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:11.021 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:11.018 7f8105c02700 1 --2- 172.21.2.102:0/1804916593 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f80e40186c0 0x7f80e401abc0 crc :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:11.021 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:11.018 7f8105c02700 1 -- 172.21.2.102:0/1804916593 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f81001a44c0 msgr2=0x7f81001a48c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:11.021 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:11.018 7f8105c02700 1 --2- 172.21.2.102:0/1804916593 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f81001a44c0 0x7f81001a48c0 secure :-1 s=READY pgs=2642 cs=0 l=1 rev1=1 rx=0x7f80e8009630 tx=0x7f80e800b980).stop 2021-02-10T11:35:11.021 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:11.018 7f8105c02700 1 -- 172.21.2.102:0/1804916593 shutdown_connections 2021-02-10T11:35:11.021 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:11.018 7f8105c02700 1 --2- 172.21.2.102:0/1804916593 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f81000863d0 0x7f81000867d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:11.021 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:11.018 7f8105c02700 1 --2- 172.21.2.102:0/1804916593 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f81001a44c0 0x7f81001a48c0 unknown :-1 s=CLOSED pgs=2642 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:11.021 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:11.018 7f8105c02700 1 --2- 172.21.2.102:0/1804916593 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f80e40186c0 0x7f80e401abc0 unknown :-1 s=CLOSED pgs=1164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:11.021 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:11.018 7f8105c02700 1 --2- 172.21.2.102:0/1804916593 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f81000c3820 0x7f81000a7950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:11.021 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:11.018 7f8105c02700 1 -- 172.21.2.102:0/1804916593 >> 172.21.2.102:0/1804916593 conn(0x7f81000872c0 msgr2=0x7f8100095700 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:11.021 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:11.019 7f8105c02700 1 -- 172.21.2.102:0/1804916593 shutdown_connections 2021-02-10T11:35:11.021 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:11.019 7f8105c02700 1 -- 172.21.2.102:0/1804916593 wait complete. 2021-02-10T11:35:11.029 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2021-02-10T11:35:11.461 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:11.443 7f42061fe700 1 Processor -- start 2021-02-10T11:35:11.462 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:11.443 7f42061fe700 1 -- start start 2021-02-10T11:35:11.462 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:11.443 7f42061fe700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f42000d1c80 0x7f42000cef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:11.463 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:11.443 7f42061fe700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f42000cf590 0x7f42000cf990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:11.463 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:11.443 7f42061fe700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42000fe1d0 con 0x7f42000d1880 2021-02-10T11:35:11.463 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:11.443 7f42061fe700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42001b0cc0 con 0x7f42000d1c80 2021-02-10T11:35:11.463 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:11.443 7f42061fe700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4200109a70 con 0x7f42000cf590 2021-02-10T11:35:11.463 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:11.444 7f420481b700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f42000cf590 0x7f42000cf990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:11.463 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:11.444 7f41f7fff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f42000d1c80 0x7f42000cef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:11.463 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:11.444 7f41fffff700 1 --1- >> v1:172.21.2.113:6789/0 conn(0x7f42000d1880 0x7f42001b04d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.113:6789/0 says I am v1:172.21.2.102:36256/0 (socket says 172.21.2.102:36256) 2021-02-10T11:35:11.463 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:11.444 7f420481b700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f42000cf590 0x7f42000cf990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.113:3300/0 says I am v2:172.21.2.102:40142/0 (socket says 172.21.2.102:40142) 2021-02-10T11:35:11.463 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:11.444 7f41fffff700 1 -- 172.21.2.102:0/1192350010 learned_addr learned my addr 172.21.2.102:0/1192350010 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:35:11.463 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:11.444 7f41f7fff700 1 -- 172.21.2.102:0/1192350010 >> v2:172.21.2.113:3300/0 conn(0x7f42000cf590 msgr2=0x7f42000cf990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:11.463 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:11.444 7f41f7fff700 1 --2- 172.21.2.102:0/1192350010 >> v2:172.21.2.113:3300/0 conn(0x7f42000cf590 0x7f42000cf990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:11.464 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:11.444 7f41f7fff700 1 -- 172.21.2.102:0/1192350010 >> v1:172.21.2.113:6789/0 conn(0x7f42000d1880 legacy=0x7f42001b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:11.464 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:11.444 7f41f7fff700 1 -- 172.21.2.102:0/1192350010 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4200088070 con 0x7f42000d1c80 2021-02-10T11:35:11.464 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:11.444 7f41f7fff700 1 --2- 172.21.2.102:0/1192350010 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f42000d1c80 0x7f42000cef70 secure :-1 s=READY pgs=2643 cs=0 l=1 rev1=1 rx=0x7f41f00069d0 tx=0x7f41f0009e40).ready entity=mon.0 client_cookie=f911a4cc0671d896 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:11.464 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:11.445 7f41ff7fe700 1 -- 172.21.2.102:0/1192350010 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f41f000b510 con 0x7f42000d1c80 2021-02-10T11:35:11.464 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:11.445 7f41ff7fe700 1 -- 172.21.2.102:0/1192350010 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41f000b890 con 0x7f42000d1c80 2021-02-10T11:35:11.464 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:11.445 7f41ff7fe700 1 -- 172.21.2.102:0/1192350010 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f41f000bcf0 con 0x7f42000d1c80 2021-02-10T11:35:11.464 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:11.445 7f42061fe700 1 -- 172.21.2.102:0/1192350010 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f42000d1c80 msgr2=0x7f42000cef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:11.464 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:11.445 7f42061fe700 1 --2- 172.21.2.102:0/1192350010 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f42000d1c80 0x7f42000cef70 secure :-1 s=READY pgs=2643 cs=0 l=1 rev1=1 rx=0x7f41f00069d0 tx=0x7f41f0009e40).stop 2021-02-10T11:35:11.464 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:11.446 7f42061fe700 1 -- 172.21.2.102:0/1192350010 shutdown_connections 2021-02-10T11:35:11.464 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:11.446 7f42061fe700 1 --2- 172.21.2.102:0/1192350010 >> v2:172.21.2.113:3300/0 conn(0x7f42000cf590 0x7f42000cf990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:11.464 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:11.446 7f42061fe700 1 --2- 172.21.2.102:0/1192350010 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f42000d1c80 0x7f42000cef70 unknown :-1 s=CLOSED pgs=2643 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:11.464 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:11.446 7f42061fe700 1 -- 172.21.2.102:0/1192350010 >> 172.21.2.102:0/1192350010 conn(0x7f42000ce430 msgr2=0x7f42000d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:11.464 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:11.446 7f42061fe700 1 -- 172.21.2.102:0/1192350010 shutdown_connections 2021-02-10T11:35:11.465 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:11.446 7f42061fe700 1 -- 172.21.2.102:0/1192350010 wait complete. 2021-02-10T11:35:11.466 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:11.460 7f42061fe700 1 Processor -- start 2021-02-10T11:35:11.466 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:11.460 7f42061fe700 1 -- start start 2021-02-10T11:35:11.466 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:11.461 7f42061fe700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4200095710 0x7f420019dcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:11.467 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:11.461 7f42061fe700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f42000aeeb0 0x7f420019e2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:11.467 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:11.461 7f42061fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f420019e900 0x7f42001b7990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:11.467 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:11.461 7f42061fe700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f420019ed00 con 0x7f420019e900 2021-02-10T11:35:11.467 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:11.461 7f42061fe700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4200094050 con 0x7f42000aeeb0 2021-02-10T11:35:11.467 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:11.461 7f42061fe700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42001b7fb0 con 0x7f4200095710 2021-02-10T11:35:11.467 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:11.461 7f420481b700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f420019e900 0x7f42001b7990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:11.467 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:11.461 7f420481b700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f420019e900 0x7f42001b7990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:36718/0 (socket says 172.21.2.102:36718) 2021-02-10T11:35:11.467 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:11.461 7f420481b700 1 -- 172.21.2.102:0/2022219131 learned_addr learned my addr 172.21.2.102:0/2022219131 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:35:11.467 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:11.461 7f41f7fff700 1 --2- 172.21.2.102:0/2022219131 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f42000aeeb0 0x7f420019e2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:11.467 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:11.461 7f41fffff700 1 --2- 172.21.2.102:0/2022219131 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4200095710 0x7f420019dcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:11.467 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:11.461 7f420481b700 1 -- 172.21.2.102:0/2022219131 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4200095710 msgr2=0x7f420019dcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:11.467 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:11.461 7f420481b700 1 --2- 172.21.2.102:0/2022219131 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4200095710 0x7f420019dcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:11.467 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:11.461 7f420481b700 1 -- 172.21.2.102:0/2022219131 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f42000aeeb0 msgr2=0x7f420019e2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:11.467 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:11.461 7f420481b700 1 --2- 172.21.2.102:0/2022219131 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f42000aeeb0 0x7f420019e2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:11.468 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:11.461 7f420481b700 1 -- 172.21.2.102:0/2022219131 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42001b81d0 con 0x7f420019e900 2021-02-10T11:35:11.468 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:11.462 7f420481b700 1 --2- 172.21.2.102:0/2022219131 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f420019e900 0x7f42001b7990 secure :-1 s=READY pgs=2644 cs=0 l=1 rev1=1 rx=0x7f41f800b5e0 tx=0x7f41f800b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:11.468 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:11.462 7f41fd7fa700 1 -- 172.21.2.102:0/2022219131 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f41f8000c10 con 0x7f420019e900 2021-02-10T11:35:11.468 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:11.462 7f41fd7fa700 1 -- 172.21.2.102:0/2022219131 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41f800bd70 con 0x7f420019e900 2021-02-10T11:35:11.468 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:11.462 7f42061fe700 1 -- 172.21.2.102:0/2022219131 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42001b81d0 con 0x7f420019e900 2021-02-10T11:35:11.468 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:11.462 7f41fd7fa700 1 -- 172.21.2.102:0/2022219131 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f41f80024d0 con 0x7f420019e900 2021-02-10T11:35:11.468 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:11.462 7f42061fe700 1 -- 172.21.2.102:0/2022219131 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42001b8760 con 0x7f420019e900 2021-02-10T11:35:11.468 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:11.463 7f41fd7fa700 1 -- 172.21.2.102:0/2022219131 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f41f8005330 con 0x7f420019e900 2021-02-10T11:35:11.468 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:11.463 7f41fd7fa700 1 --2- 172.21.2.102:0/2022219131 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f41e40186c0 0x7f41e401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:11.468 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:11.463 7f41fd7fa700 1 -- 172.21.2.102:0/2022219131 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5457+0+0 (secure 0 0 0) 0x7f41f80324f0 con 0x7f420019e900 2021-02-10T11:35:11.468 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:11.463 7f41fffff700 1 --2- 172.21.2.102:0/2022219131 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f41e40186c0 0x7f41e401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:11.468 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:11.463 7f41fffff700 1 --2- 172.21.2.102:0/2022219131 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f41e40186c0 0x7f41e401abc0 crc :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:11.495 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:11.493 7f42061fe700 1 -- 172.21.2.102:0/2022219131 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4200082aa0 con 0x7f420019e900 2021-02-10T11:35:11.498 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:11.496 7f41fd7fa700 1 -- 172.21.2.102:0/2022219131 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f41f8006370 con 0x7f420019e900 2021-02-10T11:35:11.629 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:11.627 7f42061fe700 1 -- 172.21.2.102:0/2022219131 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"data": "cephfs_data", "prefix": "fs new", "force": true, "fs_name": "cephfs", "metadata": "cephfs_metadata"} v 0) v1 -- 0x7f4200082aa0 con 0x7f420019e900 2021-02-10T11:35:11.633 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:11.631 7f41fd7fa700 1 -- 172.21.2.102:0/2022219131 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"data": "cephfs_data", "prefix": "fs new", "force": true, "fs_name": "cephfs", "metadata": "cephfs_metadata"}]=0 new fs with metadata pool 30 and data pool 31 v121) v1 ==== 189+0+0 (secure 0 0 0) 0x7f41f8005330 con 0x7f420019e900 2021-02-10T11:35:11.636 INFO:teuthology.orchestra.run.gibba002.stderr:new fs with metadata pool 30 and data pool 31 2021-02-10T11:35:11.638 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:11.636 7f42061fe700 1 -- 172.21.2.102:0/2022219131 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f41e40186c0 msgr2=0x7f41e401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:11.638 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:11.636 7f42061fe700 1 --2- 172.21.2.102:0/2022219131 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f41e40186c0 0x7f41e401abc0 crc :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:11.638 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:11.636 7f42061fe700 1 -- 172.21.2.102:0/2022219131 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f420019e900 msgr2=0x7f42001b7990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:11.638 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:11.636 7f42061fe700 1 --2- 172.21.2.102:0/2022219131 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f420019e900 0x7f42001b7990 secure :-1 s=READY pgs=2644 cs=0 l=1 rev1=1 rx=0x7f41f800b5e0 tx=0x7f41f800b980).stop 2021-02-10T11:35:11.638 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:11.636 7f42061fe700 1 -- 172.21.2.102:0/2022219131 shutdown_connections 2021-02-10T11:35:11.639 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:11.636 7f42061fe700 1 --2- 172.21.2.102:0/2022219131 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f420019e900 0x7f42001b7990 unknown :-1 s=CLOSED pgs=2644 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:11.639 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:11.636 7f42061fe700 1 --2- 172.21.2.102:0/2022219131 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f41e40186c0 0x7f41e401abc0 unknown :-1 s=CLOSED pgs=1167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:11.639 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:11.636 7f42061fe700 1 --2- 172.21.2.102:0/2022219131 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f42000aeeb0 0x7f420019e2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:11.639 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:11.636 7f42061fe700 1 --2- 172.21.2.102:0/2022219131 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4200095710 0x7f420019dcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:11.639 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:11.636 7f42061fe700 1 -- 172.21.2.102:0/2022219131 >> 172.21.2.102:0/2022219131 conn(0x7f42000846b0 msgr2=0x7f4200094950 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:11.639 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:11.636 7f42061fe700 1 -- 172.21.2.102:0/2022219131 shutdown_connections 2021-02-10T11:35:11.639 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:11.636 7f42061fe700 1 -- 172.21.2.102:0/2022219131 wait complete. 2021-02-10T11:35:11.646 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:35:12.075 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:12.056 7f7d5cfd4700 1 Processor -- start 2021-02-10T11:35:12.075 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:12.056 7f7d5cfd4700 1 -- start start 2021-02-10T11:35:12.075 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:12.057 7f7d5cfd4700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7d580d0340 0x7f7d580ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:12.075 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:12.057 7f7d5cfd4700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f7d580d1c70 0x7f7d580efc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:12.075 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:12.057 7f7d5cfd4700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d580f0b10 con 0x7f7d580d1870 2021-02-10T11:35:12.075 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:12.057 7f7d5cfd4700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d580e5ed0 con 0x7f7d580d1c70 2021-02-10T11:35:12.075 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:12.057 7f7d5cfd4700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d581b0be0 con 0x7f7d580d0340 2021-02-10T11:35:12.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:12.057 7f7d56d9d700 1 --1- >> v1:172.21.2.113:6789/0 conn(0x7f7d580d1870 0x7f7d581b03f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.113:6789/0 says I am v1:172.21.2.102:36270/0 (socket says 172.21.2.102:36270) 2021-02-10T11:35:12.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:12.057 7f7d56d9d700 1 -- 172.21.2.102:0/3915770649 learned_addr learned my addr 172.21.2.102:0/3915770649 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:35:12.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:12.057 7f7d5759e700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f7d580d1c70 0x7f7d580efc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:12.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:12.057 7f7d5659c700 1 --2- 172.21.2.102:0/3915770649 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7d580d0340 0x7f7d580ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:12.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:12.057 7f7d55d9b700 1 -- 172.21.2.102:0/3915770649 <== mon.1 v1:172.21.2.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3279824974 0 0) 0x7f7d4c001190 con 0x7f7d580d1870 2021-02-10T11:35:12.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:12.057 7f7d55d9b700 1 -- 172.21.2.102:0/3915770649 --> v1:172.21.2.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d40002fe0 con 0x7f7d580d1870 2021-02-10T11:35:12.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:12.058 7f7d5759e700 1 -- 172.21.2.102:0/3915770649 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7d580d0340 msgr2=0x7f7d580ef640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:12.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:12.058 7f7d5759e700 1 --2- 172.21.2.102:0/3915770649 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7d580d0340 0x7f7d580ef640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:12.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:12.058 7f7d5759e700 1 -- 172.21.2.102:0/3915770649 >> v1:172.21.2.113:6789/0 conn(0x7f7d580d1870 legacy=0x7f7d581b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:12.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:12.058 7f7d5759e700 1 -- 172.21.2.102:0/3915770649 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d580f20a0 con 0x7f7d580d1c70 2021-02-10T11:35:12.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:12.058 7f7d5759e700 1 --2- 172.21.2.102:0/3915770649 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7d580d1c70 0x7f7d580efc60 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 rx=0x7f7d50006e50 tx=0x7f7d50009e70).ready entity=mon.1 client_cookie=17773a93cb710481 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:12.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:12.058 7f7d55d9b700 1 -- 172.21.2.102:0/3915770649 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7d50006a20 con 0x7f7d580d1c70 2021-02-10T11:35:12.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:12.058 7f7d55d9b700 1 -- 172.21.2.102:0/3915770649 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d50006790 con 0x7f7d580d1c70 2021-02-10T11:35:12.077 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:12.058 7f7d55d9b700 1 -- 172.21.2.102:0/3915770649 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7d5000b700 con 0x7f7d580d1c70 2021-02-10T11:35:12.077 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:12.058 7f7d5cfd4700 1 -- 172.21.2.102:0/3915770649 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7d580d1c70 msgr2=0x7f7d580efc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:12.077 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:12.058 7f7d5cfd4700 1 --2- 172.21.2.102:0/3915770649 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7d580d1c70 0x7f7d580efc60 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 rx=0x7f7d50006e50 tx=0x7f7d50009e70).stop 2021-02-10T11:35:12.077 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:12.059 7f7d5cfd4700 1 -- 172.21.2.102:0/3915770649 shutdown_connections 2021-02-10T11:35:12.077 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:12.059 7f7d5cfd4700 1 --2- 172.21.2.102:0/3915770649 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7d580d0340 0x7f7d580ef640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:12.077 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:12.059 7f7d5cfd4700 1 --2- 172.21.2.102:0/3915770649 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7d580d1c70 0x7f7d580efc60 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:12.077 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:12.059 7f7d5cfd4700 1 -- 172.21.2.102:0/3915770649 >> 172.21.2.102:0/3915770649 conn(0x7f7d580ce450 msgr2=0x7f7d580d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:12.077 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:12.059 7f7d5cfd4700 1 -- 172.21.2.102:0/3915770649 shutdown_connections 2021-02-10T11:35:12.077 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:12.059 7f7d5cfd4700 1 -- 172.21.2.102:0/3915770649 wait complete. 2021-02-10T11:35:12.079 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:12.074 7f7d5cfd4700 1 Processor -- start 2021-02-10T11:35:12.079 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:12.074 7f7d5cfd4700 1 -- start start 2021-02-10T11:35:12.079 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:12.074 7f7d5cfd4700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7d581a4a60 0x7f7d580c1c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:12.079 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:12.074 7f7d5cfd4700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7d580c2290 0x7f7d580c2690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:12.079 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:12.074 7f7d5cfd4700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7d580c2cb0 0x7f7d580bfd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:12.079 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:12.074 7f7d5cfd4700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d580f05b0 con 0x7f7d580c2cb0 2021-02-10T11:35:12.079 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:12.074 7f7d5cfd4700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d580f07d0 con 0x7f7d581a4a60 2021-02-10T11:35:12.079 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:12.074 7f7d5cfd4700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d580c0340 con 0x7f7d580c2290 2021-02-10T11:35:12.079 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:12.074 7f7d5759e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7d580c2cb0 0x7f7d580bfd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:12.079 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:12.074 7f7d5759e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7d580c2cb0 0x7f7d580bfd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:36726/0 (socket says 172.21.2.102:36726) 2021-02-10T11:35:12.080 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:12.074 7f7d5759e700 1 -- 172.21.2.102:0/34098592 learned_addr learned my addr 172.21.2.102:0/34098592 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:35:12.080 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:12.074 7f7d56d9d700 1 --2- 172.21.2.102:0/34098592 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7d581a4a60 0x7f7d580c1c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:12.080 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:12.074 7f7d5659c700 1 --2- 172.21.2.102:0/34098592 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7d580c2290 0x7f7d580c2690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:12.080 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:12.075 7f7d5759e700 1 -- 172.21.2.102:0/34098592 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7d580c2290 msgr2=0x7f7d580c2690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:12.080 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:12.075 7f7d5759e700 1 --2- 172.21.2.102:0/34098592 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7d580c2290 0x7f7d580c2690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:12.080 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:12.075 7f7d5759e700 1 -- 172.21.2.102:0/34098592 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7d581a4a60 msgr2=0x7f7d580c1c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:12.080 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:12.075 7f7d5759e700 1 --2- 172.21.2.102:0/34098592 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7d581a4a60 0x7f7d580c1c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:12.080 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:12.075 7f7d5759e700 1 -- 172.21.2.102:0/34098592 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d580c0560 con 0x7f7d580c2cb0 2021-02-10T11:35:12.080 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:12.075 7f7d5759e700 1 --2- 172.21.2.102:0/34098592 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7d580c2cb0 0x7f7d580bfd20 secure :-1 s=READY pgs=2645 cs=0 l=1 rev1=1 rx=0x7f7d50006e60 tx=0x7f7d5000bc70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:12.080 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:12.075 7f7d477fe700 1 -- 172.21.2.102:0/34098592 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7d50000df0 con 0x7f7d580c2cb0 2021-02-10T11:35:12.080 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:12.075 7f7d477fe700 1 -- 172.21.2.102:0/34098592 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d5000bdb0 con 0x7f7d580c2cb0 2021-02-10T11:35:12.080 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:12.075 7f7d5cfd4700 1 -- 172.21.2.102:0/34098592 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d580c0560 con 0x7f7d580c2cb0 2021-02-10T11:35:12.080 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:12.075 7f7d477fe700 1 -- 172.21.2.102:0/34098592 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7d50001460 con 0x7f7d580c2cb0 2021-02-10T11:35:12.080 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:12.075 7f7d5cfd4700 1 -- 172.21.2.102:0/34098592 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d580c0560 con 0x7f7d580c2cb0 2021-02-10T11:35:12.081 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:12.076 7f7d477fe700 1 -- 172.21.2.102:0/34098592 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f7d500011a0 con 0x7f7d580c2cb0 2021-02-10T11:35:12.081 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:12.076 7f7d477fe700 1 --2- 172.21.2.102:0/34098592 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7d3c0186c0 0x7f7d3c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:12.081 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:12.076 7f7d477fe700 1 -- 172.21.2.102:0/34098592 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f7d50032940 con 0x7f7d580c2cb0 2021-02-10T11:35:12.081 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:12.076 7f7d56d9d700 1 --2- 172.21.2.102:0/34098592 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7d3c0186c0 0x7f7d3c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:12.081 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:12.076 7f7d56d9d700 1 --2- 172.21.2.102:0/34098592 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7d3c0186c0 0x7f7d3c01abc0 crc :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:12.108 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:12.107 7f7d5cfd4700 1 -- 172.21.2.102:0/34098592 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d580f0b10 con 0x7f7d580c2cb0 2021-02-10T11:35:12.111 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:12.109 7f7d477fe700 1 -- 172.21.2.102:0/34098592 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f7d50009130 con 0x7f7d580c2cb0 2021-02-10T11:35:12.169 DEBUG:teuthology.orchestra.run.gibba002:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:35:12.171 DEBUG:teuthology.orchestra.run.gibba013:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:35:12.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:12.235 7f7d5cfd4700 1 -- 172.21.2.102:0/34098592 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7d580f0b10 con 0x7f7d580c2cb0 2021-02-10T11:35:12.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:12.235 7f7d477fe700 1 -- 172.21.2.102:0/34098592 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v144) v1 ==== 74+0+14020 (secure 0 0 0) 0x7f7d500011a0 con 0x7f7d580c2cb0 2021-02-10T11:35:12.238 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:35:12.238 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":144,"fsid":"68db30ea-dbce-4aef-a6df-f01c6fea0513","created":"2021-02-10 11:03:06.331557","modified":"2021-02-10 11:35:11.628619","last_up_change":"2021-02-10 11:03:32.690562","last_in_change":"2021-02-10 11:03:32.690562","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":31,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:03:37.322519","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":30,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:35:09.594644","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"144","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":31,"pool_name":"cephfs_data","create_time":"2021-02-10 11:35:10.619592","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"144","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"be691b4d-a039-408a-af80-9d53f02b6f1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6818","nonce":36088},{"type":"v1","addr":"172.21.2.102:6820","nonce":36088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6822","nonce":36088},{"type":"v1","addr":"172.21.2.102:6824","nonce":36088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6830","nonce":36088},{"type":"v1","addr":"172.21.2.102:6832","nonce":36088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6826","nonce":36088},{"type":"v1","addr":"172.21.2.102:6828","nonce":36088}]},"public_addr":"172.21.2.102:6820/36088","cluster_addr":"172.21.2.102:6824/36088","heartbeat_back_addr":"172.21.2.102:6832/36088","heartbeat_front_addr":"172.21.2.102:6828/36088","state":["exists","up"]},{"osd":1,"uuid":"64e3ccc8-9c69-4d7c-865e-af8e83822635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6819","nonce":36087},{"type":"v1","addr":"172.21.2.102:6821","nonce":36087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6823","nonce":36087},{"type":"v1","addr":"172.21.2.102:6825","nonce":36087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6831","nonce":36087},{"type":"v1","addr":"172.21.2.102:6833","nonce":36087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6827","nonce":36087},{"type":"v1","addr":"172.21.2.102:6829","nonce":36087}]},"public_addr":"172.21.2.102:6821/36087","cluster_addr":"172.21.2.102:6825/36087","heartbeat_back_addr":"172.21.2.102:6833/36087","heartbeat_front_addr":"172.21.2.102:6829/36087","state":["exists","up"]},{"osd":2,"uuid":"153700af-747f-481f-bdf4-d6072d3a842f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6810","nonce":36089},{"type":"v1","addr":"172.21.2.102:6811","nonce":36089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6812","nonce":36089},{"type":"v1","addr":"172.21.2.102:6813","nonce":36089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6816","nonce":36089},{"type":"v1","addr":"172.21.2.102:6817","nonce":36089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6814","nonce":36089},{"type":"v1","addr":"172.21.2.102:6815","nonce":36089}]},"public_addr":"172.21.2.102:6811/36089","cluster_addr":"172.21.2.102:6813/36089","heartbeat_back_addr":"172.21.2.102:6817/36089","heartbeat_front_addr":"172.21.2.102:6815/36089","state":["exists","up"]},{"osd":3,"uuid":"cbc678f4-83a9-45e6-8830-2f7c232b1ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6802","nonce":36090},{"type":"v1","addr":"172.21.2.102:6803","nonce":36090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6804","nonce":36090},{"type":"v1","addr":"172.21.2.102:6805","nonce":36090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6808","nonce":36090},{"type":"v1","addr":"172.21.2.102:6809","nonce":36090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6806","nonce":36090},{"type":"v1","addr":"172.21.2.102:6807","nonce":36090}]},"public_addr":"172.21.2.102:6803/36090","cluster_addr":"172.21.2.102:6805/36090","heartbeat_back_addr":"172.21.2.102:6809/36090","heartbeat_front_addr":"172.21.2.102:6807/36090","state":["exists","up"]},{"osd":4,"uuid":"c01e4008-b258-4f10-afd2-9e7f37214351","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6808","nonce":36368},{"type":"v1","addr":"172.21.2.113:6811","nonce":36368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6814","nonce":36368},{"type":"v1","addr":"172.21.2.113:6818","nonce":36368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6826","nonce":36368},{"type":"v1","addr":"172.21.2.113:6828","nonce":36368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6820","nonce":36368},{"type":"v1","addr":"172.21.2.113:6824","nonce":36368}]},"public_addr":"172.21.2.113:6811/36368","cluster_addr":"172.21.2.113:6818/36368","heartbeat_back_addr":"172.21.2.113:6828/36368","heartbeat_front_addr":"172.21.2.113:6824/36368","state":["exists","up"]},{"osd":5,"uuid":"551f6007-daf8-4a98-8ee0-c2b16bcccb5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6802","nonce":36366},{"type":"v1","addr":"172.21.2.113:6805","nonce":36366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6807","nonce":36366},{"type":"v1","addr":"172.21.2.113:6810","nonce":36366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6819","nonce":36366},{"type":"v1","addr":"172.21.2.113:6822","nonce":36366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6812","nonce":36366},{"type":"v1","addr":"172.21.2.113:6815","nonce":36366}]},"public_addr":"172.21.2.113:6805/36366","cluster_addr":"172.21.2.113:6810/36366","heartbeat_back_addr":"172.21.2.113:6822/36366","heartbeat_front_addr":"172.21.2.113:6815/36366","state":["exists","up"]},{"osd":6,"uuid":"bf67c040-0e50-48be-b102-c3943bc9ebef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6800","nonce":36367},{"type":"v1","addr":"172.21.2.113:6801","nonce":36367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6803","nonce":36367},{"type":"v1","addr":"172.21.2.113:6804","nonce":36367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6813","nonce":36367},{"type":"v1","addr":"172.21.2.113:6816","nonce":36367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6806","nonce":36367},{"type":"v1","addr":"172.21.2.113:6809","nonce":36367}]},"public_addr":"172.21.2.113:6801/36367","cluster_addr":"172.21.2.113:6804/36367","heartbeat_back_addr":"172.21.2.113:6816/36367","heartbeat_front_addr":"172.21.2.113:6809/36367","state":["exists","up"]},{"osd":7,"uuid":"f5461f80-018d-428f-be89-2eb2bcbe7bb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6817","nonce":36365},{"type":"v1","addr":"172.21.2.113:6821","nonce":36365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6823","nonce":36365},{"type":"v1","addr":"172.21.2.113:6825","nonce":36365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6830","nonce":36365},{"type":"v1","addr":"172.21.2.113:6831","nonce":36365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6827","nonce":36365},{"type":"v1","addr":"172.21.2.113:6829","nonce":36365}]},"public_addr":"172.21.2.113:6821/36365","cluster_addr":"172.21.2.113:6825/36365","heartbeat_back_addr":"172.21.2.113:6831/36365","heartbeat_front_addr":"172.21.2.113:6829/36365","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:35:12.239 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:12.237 7f7d5cfd4700 1 -- 172.21.2.102:0/34098592 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7d3c0186c0 msgr2=0x7f7d3c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:12.240 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:12.237 7f7d5cfd4700 1 --2- 172.21.2.102:0/34098592 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7d3c0186c0 0x7f7d3c01abc0 crc :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:12.240 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:12.237 7f7d5cfd4700 1 -- 172.21.2.102:0/34098592 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7d580c2cb0 msgr2=0x7f7d580bfd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:12.240 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:12.237 7f7d5cfd4700 1 --2- 172.21.2.102:0/34098592 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7d580c2cb0 0x7f7d580bfd20 secure :-1 s=READY pgs=2645 cs=0 l=1 rev1=1 rx=0x7f7d50006e60 tx=0x7f7d5000bc70).stop 2021-02-10T11:35:12.240 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:12.237 7f7d5cfd4700 1 -- 172.21.2.102:0/34098592 shutdown_connections 2021-02-10T11:35:12.240 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:12.237 7f7d5cfd4700 1 --2- 172.21.2.102:0/34098592 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7d580c2cb0 0x7f7d580bfd20 unknown :-1 s=CLOSED pgs=2645 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:12.240 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:12.237 7f7d5cfd4700 1 --2- 172.21.2.102:0/34098592 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7d580c2290 0x7f7d580c2690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:12.240 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:12.238 7f7d5cfd4700 1 --2- 172.21.2.102:0/34098592 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7d3c0186c0 0x7f7d3c01abc0 unknown :-1 s=CLOSED pgs=1168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:12.240 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:12.238 7f7d5cfd4700 1 --2- 172.21.2.102:0/34098592 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7d581a4a60 0x7f7d580c1c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:12.240 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:12.238 7f7d5cfd4700 1 -- 172.21.2.102:0/34098592 >> 172.21.2.102:0/34098592 conn(0x7f7d580ebe30 msgr2=0x7f7d580bf200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:12.240 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:12.238 7f7d5cfd4700 1 -- 172.21.2.102:0/34098592 shutdown_connections 2021-02-10T11:35:12.240 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:12.238 7f7d5cfd4700 1 -- 172.21.2.102:0/34098592 wait complete. 2021-02-10T11:35:12.249 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:35:12.603 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:12.584 7f5e26fd8700 1 Processor -- start 2021-02-10T11:35:12.603 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:12.584 7f5e26fd8700 1 -- start start 2021-02-10T11:35:12.603 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:12.584 7f5e26fd8700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5e200d1c70 0x7f5e200f20a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:12.603 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:12.584 7f5e26fd8700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e200f0b10 con 0x7f5e200d1870 2021-02-10T11:35:12.604 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:12.584 7f5e26fd8700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e201b0be0 con 0x7f5e200d0340 2021-02-10T11:35:12.604 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:12.584 7f5e26fd8700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e200e5ed0 con 0x7f5e200d1c70 2021-02-10T11:35:12.604 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:12.584 7f5e24df4700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f5e200d1870 0x7f5e201b03f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:57190/0 (socket says 172.21.2.102:57190) 2021-02-10T11:35:12.604 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:12.584 7f5e24df4700 1 -- 172.21.2.102:0/2476916034 learned_addr learned my addr 172.21.2.102:0/2476916034 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:35:12.604 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:12.585 7f5e255f5700 1 --2- 172.21.2.102:0/2476916034 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5e200d1c70 0x7f5e200f20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:12.604 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:12.585 7f5e1f7fe700 1 -- 172.21.2.102:0/2476916034 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1875163353 0 0) 0x7f5e080009b0 con 0x7f5e200d1870 2021-02-10T11:35:12.604 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:12.585 7f5e1f7fe700 1 -- 172.21.2.102:0/2476916034 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e0c002fe0 con 0x7f5e200d1870 2021-02-10T11:35:12.604 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:12.585 7f5e1f7fe700 1 -- 172.21.2.102:0/2476916034 <== mon.1 v1:172.21.2.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2426360908 0 0) 0x7f5e10000ff0 con 0x7f5e200d0340 2021-02-10T11:35:12.604 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:12.585 7f5e1f7fe700 1 -- 172.21.2.102:0/2476916034 --> v1:172.21.2.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e0c003210 con 0x7f5e200d0340 2021-02-10T11:35:12.604 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:12.585 7f5e1f7fe700 1 -- 172.21.2.102:0/2476916034 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 516444553 0 0) 0x7f5e08001010 con 0x7f5e200d1870 2021-02-10T11:35:12.604 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:12.585 7f5e1f7fe700 1 -- 172.21.2.102:0/2476916034 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5e200d1c70 msgr2=0x7f5e200f20a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:12.604 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:12.585 7f5e1f7fe700 1 --2- 172.21.2.102:0/2476916034 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5e200d1c70 0x7f5e200f20a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:12.604 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:12.585 7f5e1f7fe700 1 -- 172.21.2.102:0/2476916034 >> v1:172.21.2.113:6789/0 conn(0x7f5e200d0340 legacy=0x7f5e200ef640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:12.605 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:12.585 7f5e1f7fe700 1 -- 172.21.2.102:0/2476916034 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e200efe30 con 0x7f5e200d1870 2021-02-10T11:35:12.605 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:12.585 7f5e1f7fe700 1 -- 172.21.2.102:0/2476916034 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7f5e08001590 con 0x7f5e200d1870 2021-02-10T11:35:12.605 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:12.585 7f5e1f7fe700 1 -- 172.21.2.102:0/2476916034 >> v1:172.21.2.102:6789/0 conn(0x7f5e200d1870 legacy=0x7f5e201b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:12.605 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:12.585 7f5e1f7fe700 1 --2- 172.21.2.102:0/2476916034 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5e0c0037e0 0x7f5e0c003be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:12.605 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:12.585 7f5e1f7fe700 1 --2- 172.21.2.102:0/2476916034 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5e0c0068e0 0x7f5e0c008cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:12.605 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:12.585 7f5e1f7fe700 1 --2- 172.21.2.102:0/2476916034 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5e0c009540 0x7f5e0c00ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:12.605 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:12.585 7f5e1f7fe700 1 -- 172.21.2.102:0/2476916034 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e0c0502e0 con 0x7f5e0c009540 2021-02-10T11:35:12.605 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:12.585 7f5e1f7fe700 1 -- 172.21.2.102:0/2476916034 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e0c0504c0 con 0x7f5e0c0068e0 2021-02-10T11:35:12.605 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:12.585 7f5e1f7fe700 1 -- 172.21.2.102:0/2476916034 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e0c0506e0 con 0x7f5e0c0037e0 2021-02-10T11:35:12.605 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:12.585 7f5e255f5700 1 --2- 172.21.2.102:0/2476916034 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5e0c009540 0x7f5e0c00ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:12.605 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:12.585 7f5e1ffff700 1 --2- 172.21.2.102:0/2476916034 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5e0c0068e0 0x7f5e0c008cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:12.605 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:12.585 7f5e24df4700 1 --2- 172.21.2.102:0/2476916034 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5e0c0037e0 0x7f5e0c003be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:12.605 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:12.586 7f5e255f5700 1 -- 172.21.2.102:0/2476916034 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5e0c0037e0 msgr2=0x7f5e0c003be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:12.605 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:12.586 7f5e255f5700 1 --2- 172.21.2.102:0/2476916034 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5e0c0037e0 0x7f5e0c003be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:12.606 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:12.586 7f5e255f5700 1 -- 172.21.2.102:0/2476916034 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5e0c0068e0 msgr2=0x7f5e0c008cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:12.606 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:12.586 7f5e255f5700 1 --2- 172.21.2.102:0/2476916034 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5e0c0068e0 0x7f5e0c008cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:12.606 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:12.586 7f5e255f5700 1 -- 172.21.2.102:0/2476916034 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e0c050980 con 0x7f5e0c009540 2021-02-10T11:35:12.606 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:12.586 7f5e255f5700 1 --2- 172.21.2.102:0/2476916034 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5e0c009540 0x7f5e0c00ba20 secure :-1 s=READY pgs=2647 cs=0 l=1 rev1=1 rx=0x7f5e18009a40 tx=0x7f5e18009de0).ready entity=mon.0 client_cookie=6138d020acee76dd server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:12.606 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:12.586 7f5e1f7fe700 1 -- 172.21.2.102:0/2476916034 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5e1800b510 con 0x7f5e0c009540 2021-02-10T11:35:12.606 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:12.586 7f5e1f7fe700 1 -- 172.21.2.102:0/2476916034 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e18003030 con 0x7f5e0c009540 2021-02-10T11:35:12.606 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:12.586 7f5e1f7fe700 1 -- 172.21.2.102:0/2476916034 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5e1800b9d0 con 0x7f5e0c009540 2021-02-10T11:35:12.606 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:12.586 7f5e26fd8700 1 -- 172.21.2.102:0/2476916034 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5e0c009540 msgr2=0x7f5e0c00ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:12.606 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:12.586 7f5e26fd8700 1 --2- 172.21.2.102:0/2476916034 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5e0c009540 0x7f5e0c00ba20 secure :-1 s=READY pgs=2647 cs=0 l=1 rev1=1 rx=0x7f5e18009a40 tx=0x7f5e18009de0).stop 2021-02-10T11:35:12.606 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:12.586 7f5e255f5700 1 -- 172.21.2.102:0/2476916034 reap_dead start 2021-02-10T11:35:12.606 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:12.586 7f5e26fd8700 1 -- 172.21.2.102:0/2476916034 shutdown_connections 2021-02-10T11:35:12.606 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:12.586 7f5e26fd8700 1 -- 172.21.2.102:0/2476916034 >> 172.21.2.102:0/2476916034 conn(0x7f5e200ce450 msgr2=0x7f5e200d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:12.606 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:12.586 7f5e26fd8700 1 -- 172.21.2.102:0/2476916034 shutdown_connections 2021-02-10T11:35:12.607 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:12.586 7f5e26fd8700 1 -- 172.21.2.102:0/2476916034 wait complete. 2021-02-10T11:35:12.607 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:12.601 7f5e26fd8700 1 Processor -- start 2021-02-10T11:35:12.607 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:12.601 7f5e26fd8700 1 -- start start 2021-02-10T11:35:12.608 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:12.602 7f5e26fd8700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5e201a4a60 0x7f5e200c1c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:12.608 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:12.602 7f5e26fd8700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5e200c2290 0x7f5e200c2690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:12.608 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:12.602 7f5e26fd8700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5e200c2cb0 0x7f5e200bfd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:12.608 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:12.602 7f5e26fd8700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e200f05b0 con 0x7f5e201a4a60 2021-02-10T11:35:12.608 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:12.602 7f5e26fd8700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e200f07d0 con 0x7f5e200c2cb0 2021-02-10T11:35:12.608 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:12.602 7f5e26fd8700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e200c0340 con 0x7f5e200c2290 2021-02-10T11:35:12.608 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:12.602 7f5e24df4700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5e201a4a60 0x7f5e200c1c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:12.609 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:12.602 7f5e24df4700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5e201a4a60 0x7f5e200c1c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:36746/0 (socket says 172.21.2.102:36746) 2021-02-10T11:35:12.609 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:12.602 7f5e24df4700 1 -- 172.21.2.102:0/3026670406 learned_addr learned my addr 172.21.2.102:0/3026670406 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:35:12.609 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:12.602 7f5e1ffff700 1 --2- 172.21.2.102:0/3026670406 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5e200c2290 0x7f5e200c2690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:12.609 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:12.602 7f5e255f5700 1 --2- 172.21.2.102:0/3026670406 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5e200c2cb0 0x7f5e200bfd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:12.609 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:12.602 7f5e24df4700 1 -- 172.21.2.102:0/3026670406 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5e200c2290 msgr2=0x7f5e200c2690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:12.609 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:12.602 7f5e24df4700 1 --2- 172.21.2.102:0/3026670406 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5e200c2290 0x7f5e200c2690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:12.609 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:12.602 7f5e24df4700 1 -- 172.21.2.102:0/3026670406 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5e200c2cb0 msgr2=0x7f5e200bfd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:12.609 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:12.602 7f5e24df4700 1 --2- 172.21.2.102:0/3026670406 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5e200c2cb0 0x7f5e200bfd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:12.609 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:12.602 7f5e24df4700 1 -- 172.21.2.102:0/3026670406 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e200c0560 con 0x7f5e201a4a60 2021-02-10T11:35:12.609 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:12.603 7f5e24df4700 1 --2- 172.21.2.102:0/3026670406 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5e201a4a60 0x7f5e200c1c70 secure :-1 s=READY pgs=2648 cs=0 l=1 rev1=1 rx=0x7f5e08001c00 tx=0x7f5e0800a8e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:12.609 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:12.603 7f5e1d7fa700 1 -- 172.21.2.102:0/3026670406 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5e080009b0 con 0x7f5e201a4a60 2021-02-10T11:35:12.609 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:12.603 7f5e1d7fa700 1 -- 172.21.2.102:0/3026670406 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e0800acd0 con 0x7f5e201a4a60 2021-02-10T11:35:12.609 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:12.603 7f5e26fd8700 1 -- 172.21.2.102:0/3026670406 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e200c0560 con 0x7f5e201a4a60 2021-02-10T11:35:12.609 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:12.603 7f5e1d7fa700 1 -- 172.21.2.102:0/3026670406 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5e08004540 con 0x7f5e201a4a60 2021-02-10T11:35:12.610 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:12.603 7f5e26fd8700 1 -- 172.21.2.102:0/3026670406 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e200c0af0 con 0x7f5e201a4a60 2021-02-10T11:35:12.610 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:12.603 7f5e1d7fa700 1 -- 172.21.2.102:0/3026670406 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f5e08004140 con 0x7f5e201a4a60 2021-02-10T11:35:12.610 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:12.603 7f5e1d7fa700 1 --2- 172.21.2.102:0/3026670406 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5e040186c0 0x7f5e0401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:12.610 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:12.603 7f5e1d7fa700 1 -- 172.21.2.102:0/3026670406 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f5e08032510 con 0x7f5e201a4a60 2021-02-10T11:35:12.610 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:12.603 7f5e1ffff700 1 --2- 172.21.2.102:0/3026670406 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5e040186c0 0x7f5e0401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:12.610 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:12.604 7f5e1ffff700 1 --2- 172.21.2.102:0/3026670406 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5e040186c0 0x7f5e0401abc0 crc :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:12.636 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:12.634 7f5e26fd8700 1 -- 172.21.2.102:0/3026670406 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e200f0af0 con 0x7f5e201a4a60 2021-02-10T11:35:12.639 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:12.637 7f5e1d7fa700 1 -- 172.21.2.102:0/3026670406 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f5e08000940 con 0x7f5e201a4a60 2021-02-10T11:35:12.771 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:12.769 7f5e26fd8700 1 -- 172.21.2.102:0/3026670406 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5e200f0af0 con 0x7f5e201a4a60 2021-02-10T11:35:12.772 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:12.770 7f5e1d7fa700 1 -- 172.21.2.102:0/3026670406 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v145) v1 ==== 74+0+14007 (secure 0 0 0) 0x7f5e0801f470 con 0x7f5e201a4a60 2021-02-10T11:35:12.772 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:35:12.772 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":145,"fsid":"68db30ea-dbce-4aef-a6df-f01c6fea0513","created":"2021-02-10 11:03:06.331557","modified":"2021-02-10 11:35:12.631037","last_up_change":"2021-02-10 11:03:32.690562","last_in_change":"2021-02-10 11:03:32.690562","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":31,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:03:37.322519","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":30,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:35:09.594644","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"144","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":31,"pool_name":"cephfs_data","create_time":"2021-02-10 11:35:10.619592","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"145","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"be691b4d-a039-408a-af80-9d53f02b6f1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6818","nonce":36088},{"type":"v1","addr":"172.21.2.102:6820","nonce":36088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6822","nonce":36088},{"type":"v1","addr":"172.21.2.102:6824","nonce":36088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6830","nonce":36088},{"type":"v1","addr":"172.21.2.102:6832","nonce":36088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6826","nonce":36088},{"type":"v1","addr":"172.21.2.102:6828","nonce":36088}]},"public_addr":"172.21.2.102:6820/36088","cluster_addr":"172.21.2.102:6824/36088","heartbeat_back_addr":"172.21.2.102:6832/36088","heartbeat_front_addr":"172.21.2.102:6828/36088","state":["exists","up"]},{"osd":1,"uuid":"64e3ccc8-9c69-4d7c-865e-af8e83822635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6819","nonce":36087},{"type":"v1","addr":"172.21.2.102:6821","nonce":36087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6823","nonce":36087},{"type":"v1","addr":"172.21.2.102:6825","nonce":36087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6831","nonce":36087},{"type":"v1","addr":"172.21.2.102:6833","nonce":36087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6827","nonce":36087},{"type":"v1","addr":"172.21.2.102:6829","nonce":36087}]},"public_addr":"172.21.2.102:6821/36087","cluster_addr":"172.21.2.102:6825/36087","heartbeat_back_addr":"172.21.2.102:6833/36087","heartbeat_front_addr":"172.21.2.102:6829/36087","state":["exists","up"]},{"osd":2,"uuid":"153700af-747f-481f-bdf4-d6072d3a842f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6810","nonce":36089},{"type":"v1","addr":"172.21.2.102:6811","nonce":36089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6812","nonce":36089},{"type":"v1","addr":"172.21.2.102:6813","nonce":36089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6816","nonce":36089},{"type":"v1","addr":"172.21.2.102:6817","nonce":36089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6814","nonce":36089},{"type":"v1","addr":"172.21.2.102:6815","nonce":36089}]},"public_addr":"172.21.2.102:6811/36089","cluster_addr":"172.21.2.102:6813/36089","heartbeat_back_addr":"172.21.2.102:6817/36089","heartbeat_front_addr":"172.21.2.102:6815/36089","state":["exists","up"]},{"osd":3,"uuid":"cbc678f4-83a9-45e6-8830-2f7c232b1ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6802","nonce":36090},{"type":"v1","addr":"172.21.2.102:6803","nonce":36090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6804","nonce":36090},{"type":"v1","addr":"172.21.2.102:6805","nonce":36090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6808","nonce":36090},{"type":"v1","addr":"172.21.2.102:6809","nonce":36090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6806","nonce":36090},{"type":"v1","addr":"172.21.2.102:6807","nonce":36090}]},"public_addr":"172.21.2.102:6803/36090","cluster_addr":"172.21.2.102:6805/36090","heartbeat_back_addr":"172.21.2.102:6809/36090","heartbeat_front_addr":"172.21.2.102:6807/36090","state":["exists","up"]},{"osd":4,"uuid":"c01e4008-b258-4f10-afd2-9e7f37214351","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6808","nonce":36368},{"type":"v1","addr":"172.21.2.113:6811","nonce":36368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6814","nonce":36368},{"type":"v1","addr":"172.21.2.113:6818","nonce":36368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6826","nonce":36368},{"type":"v1","addr":"172.21.2.113:6828","nonce":36368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6820","nonce":36368},{"type":"v1","addr":"172.21.2.113:6824","nonce":36368}]},"public_addr":"172.21.2.113:6811/36368","cluster_addr":"172.21.2.113:6818/36368","heartbeat_back_addr":"172.21.2.113:6828/36368","heartbeat_front_addr":"172.21.2.113:6824/36368","state":["exists","up"]},{"osd":5,"uuid":"551f6007-daf8-4a98-8ee0-c2b16bcccb5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6802","nonce":36366},{"type":"v1","addr":"172.21.2.113:6805","nonce":36366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6807","nonce":36366},{"type":"v1","addr":"172.21.2.113:6810","nonce":36366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6819","nonce":36366},{"type":"v1","addr":"172.21.2.113:6822","nonce":36366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6812","nonce":36366},{"type":"v1","addr":"172.21.2.113:6815","nonce":36366}]},"public_addr":"172.21.2.113:6805/36366","cluster_addr":"172.21.2.113:6810/36366","heartbeat_back_addr":"172.21.2.113:6822/36366","heartbeat_front_addr":"172.21.2.113:6815/36366","state":["exists","up"]},{"osd":6,"uuid":"bf67c040-0e50-48be-b102-c3943bc9ebef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6800","nonce":36367},{"type":"v1","addr":"172.21.2.113:6801","nonce":36367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6803","nonce":36367},{"type":"v1","addr":"172.21.2.113:6804","nonce":36367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6813","nonce":36367},{"type":"v1","addr":"172.21.2.113:6816","nonce":36367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6806","nonce":36367},{"type":"v1","addr":"172.21.2.113:6809","nonce":36367}]},"public_addr":"172.21.2.113:6801/36367","cluster_addr":"172.21.2.113:6804/36367","heartbeat_back_addr":"172.21.2.113:6816/36367","heartbeat_front_addr":"172.21.2.113:6809/36367","state":["exists","up"]},{"osd":7,"uuid":"f5461f80-018d-428f-be89-2eb2bcbe7bb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6817","nonce":36365},{"type":"v1","addr":"172.21.2.113:6821","nonce":36365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6823","nonce":36365},{"type":"v1","addr":"172.21.2.113:6825","nonce":36365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6830","nonce":36365},{"type":"v1","addr":"172.21.2.113:6831","nonce":36365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6827","nonce":36365},{"type":"v1","addr":"172.21.2.113:6829","nonce":36365}]},"public_addr":"172.21.2.113:6821/36365","cluster_addr":"172.21.2.113:6825/36365","heartbeat_back_addr":"172.21.2.113:6831/36365","heartbeat_front_addr":"172.21.2.113:6829/36365","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:35:12.773 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:12.771 7f5e26fd8700 1 -- 172.21.2.102:0/3026670406 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5e040186c0 msgr2=0x7f5e0401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:12.773 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:12.771 7f5e26fd8700 1 --2- 172.21.2.102:0/3026670406 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5e040186c0 0x7f5e0401abc0 crc :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:12.773 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:12.771 7f5e26fd8700 1 -- 172.21.2.102:0/3026670406 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5e201a4a60 msgr2=0x7f5e200c1c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:12.773 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:12.771 7f5e26fd8700 1 --2- 172.21.2.102:0/3026670406 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5e201a4a60 0x7f5e200c1c70 secure :-1 s=READY pgs=2648 cs=0 l=1 rev1=1 rx=0x7f5e08001c00 tx=0x7f5e0800a8e0).stop 2021-02-10T11:35:12.774 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:12.772 7f5e26fd8700 1 -- 172.21.2.102:0/3026670406 shutdown_connections 2021-02-10T11:35:12.774 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:12.772 7f5e26fd8700 1 --2- 172.21.2.102:0/3026670406 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5e040186c0 0x7f5e0401abc0 unknown :-1 s=CLOSED pgs=1169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:12.774 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:12.772 7f5e26fd8700 1 --2- 172.21.2.102:0/3026670406 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5e200c2cb0 0x7f5e200bfd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:12.774 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:12.772 7f5e26fd8700 1 --2- 172.21.2.102:0/3026670406 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5e200c2290 0x7f5e200c2690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:12.774 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:12.772 7f5e26fd8700 1 --2- 172.21.2.102:0/3026670406 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5e201a4a60 0x7f5e200c1c70 unknown :-1 s=CLOSED pgs=2648 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:12.775 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:12.772 7f5e26fd8700 1 -- 172.21.2.102:0/3026670406 >> 172.21.2.102:0/3026670406 conn(0x7f5e200ebe30 msgr2=0x7f5e200bf200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:12.775 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:12.772 7f5e26fd8700 1 -- 172.21.2.102:0/3026670406 shutdown_connections 2021-02-10T11:35:12.775 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:12.772 7f5e26fd8700 1 -- 172.21.2.102:0/3026670406 wait complete. 2021-02-10T11:35:12.782 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2021-02-10T11:35:13.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:13.196 7fcc5a849700 1 Processor -- start 2021-02-10T11:35:13.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:13.196 7fcc5a849700 1 -- start start 2021-02-10T11:35:13.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:13.196 7fcc5a849700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcc540d1c70 0x7fcc540cef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:13.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:13.196 7fcc5a849700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fcc540cf580 0x7fcc540cf980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:13.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:13.196 7fcc5a849700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc540fe1c0 con 0x7fcc540d1870 2021-02-10T11:35:13.216 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:13.196 7fcc5a849700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc541b0cb0 con 0x7fcc540cf580 2021-02-10T11:35:13.216 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:13.196 7fcc5a849700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc54109a60 con 0x7fcc540d1c70 2021-02-10T11:35:13.216 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:13.197 7fcc53fff700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fcc540d1870 0x7fcc541b04c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:57210/0 (socket says 172.21.2.102:57210) 2021-02-10T11:35:13.216 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:13.197 7fcc53fff700 1 -- 172.21.2.102:0/4059399252 learned_addr learned my addr 172.21.2.102:0/4059399252 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:35:13.216 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:13.197 7fcc52ffd700 1 -- 172.21.2.102:0/4059399252 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2095981134 0 0) 0x7fcc44001180 con 0x7fcc540d1870 2021-02-10T11:35:13.216 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:13.197 7fcc52ffd700 1 -- 172.21.2.102:0/4059399252 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc38002fe0 con 0x7fcc540d1870 2021-02-10T11:35:13.216 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:13.197 7fcc58e66700 1 --2- 172.21.2.102:0/4059399252 >> v2:172.21.2.113:3300/0 conn(0x7fcc540cf580 0x7fcc540cf980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:13.216 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:13.197 7fcc537fe700 1 --2- 172.21.2.102:0/4059399252 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcc540d1c70 0x7fcc540cef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:13.216 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:13.197 7fcc52ffd700 1 -- 172.21.2.102:0/4059399252 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3649149093 0 0) 0x7fcc44000ed0 con 0x7fcc540d1870 2021-02-10T11:35:13.216 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:13.197 7fcc52ffd700 1 -- 172.21.2.102:0/4059399252 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcc540d1c70 msgr2=0x7fcc540cef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:13.216 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:13.197 7fcc52ffd700 1 --2- 172.21.2.102:0/4059399252 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcc540d1c70 0x7fcc540cef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:13.216 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:13.197 7fcc52ffd700 1 -- 172.21.2.102:0/4059399252 >> v2:172.21.2.113:3300/0 conn(0x7fcc540cf580 msgr2=0x7fcc540cf980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:13.216 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:13.197 7fcc52ffd700 1 --2- 172.21.2.102:0/4059399252 >> v2:172.21.2.113:3300/0 conn(0x7fcc540cf580 0x7fcc540cf980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:13.217 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:13.197 7fcc52ffd700 1 -- 172.21.2.102:0/4059399252 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc54088070 con 0x7fcc540d1870 2021-02-10T11:35:13.217 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:13.197 7fcc52ffd700 1 -- 172.21.2.102:0/4059399252 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7fcc440015e0 con 0x7fcc540d1870 2021-02-10T11:35:13.217 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:13.197 7fcc52ffd700 1 -- 172.21.2.102:0/4059399252 >> v1:172.21.2.102:6789/0 conn(0x7fcc540d1870 legacy=0x7fcc541b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:13.217 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:13.197 7fcc52ffd700 1 --2- 172.21.2.102:0/4059399252 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcc38003a60 0x7fcc38005ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:13.217 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:13.198 7fcc52ffd700 1 --2- 172.21.2.102:0/4059399252 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcc38006770 0x7fcc38008c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:13.217 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:13.198 7fcc52ffd700 1 --2- 172.21.2.102:0/4059399252 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fcc38009500 0x7fcc3800ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:13.217 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:13.198 7fcc53fff700 1 --2- 172.21.2.102:0/4059399252 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcc38003a60 0x7fcc38005ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:13.217 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:13.198 7fcc52ffd700 1 -- 172.21.2.102:0/4059399252 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc380502e0 con 0x7fcc38003a60 2021-02-10T11:35:13.217 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:13.198 7fcc52ffd700 1 -- 172.21.2.102:0/4059399252 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc38050500 con 0x7fcc38009500 2021-02-10T11:35:13.217 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:13.198 7fcc52ffd700 1 -- 172.21.2.102:0/4059399252 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc38050720 con 0x7fcc38006770 2021-02-10T11:35:13.217 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:13.198 7fcc537fe700 1 --2- 172.21.2.102:0/4059399252 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcc38006770 0x7fcc38008c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:13.217 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:13.198 7fcc53fff700 1 -- 172.21.2.102:0/4059399252 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcc38006770 msgr2=0x7fcc38008c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:13.217 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:13.198 7fcc53fff700 1 --2- 172.21.2.102:0/4059399252 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcc38006770 0x7fcc38008c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:13.218 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:13.198 7fcc53fff700 1 -- 172.21.2.102:0/4059399252 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fcc38009500 msgr2=0x7fcc3800ba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:13.218 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:13.198 7fcc53fff700 1 --2- 172.21.2.102:0/4059399252 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fcc38009500 0x7fcc3800ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:13.218 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:13.198 7fcc53fff700 1 -- 172.21.2.102:0/4059399252 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc380509c0 con 0x7fcc38003a60 2021-02-10T11:35:13.218 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:13.198 7fcc537fe700 1 --2- 172.21.2.102:0/4059399252 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcc38006770 0x7fcc38008c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:35:13.218 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:13.198 7fcc53fff700 1 --2- 172.21.2.102:0/4059399252 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcc38003a60 0x7fcc38005ed0 secure :-1 s=READY pgs=2650 cs=0 l=1 rev1=1 rx=0x7fcc44001370 tx=0x7fcc44007d00).ready entity=mon.0 client_cookie=19f98769326c7d4b server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:13.218 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:13.198 7fcc52ffd700 1 -- 172.21.2.102:0/4059399252 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcc44000ed0 con 0x7fcc38003a60 2021-02-10T11:35:13.218 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:13.198 7fcc52ffd700 1 -- 172.21.2.102:0/4059399252 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc4400a520 con 0x7fcc38003a60 2021-02-10T11:35:13.218 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:13.198 7fcc52ffd700 1 -- 172.21.2.102:0/4059399252 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcc4400a9c0 con 0x7fcc38003a60 2021-02-10T11:35:13.218 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:13.198 7fcc5a849700 1 -- 172.21.2.102:0/4059399252 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcc38003a60 msgr2=0x7fcc38005ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:13.218 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:13.198 7fcc5a849700 1 --2- 172.21.2.102:0/4059399252 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcc38003a60 0x7fcc38005ed0 secure :-1 s=READY pgs=2650 cs=0 l=1 rev1=1 rx=0x7fcc44001370 tx=0x7fcc44007d00).stop 2021-02-10T11:35:13.218 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:13.198 7fcc58e66700 1 -- 172.21.2.102:0/4059399252 reap_dead start 2021-02-10T11:35:13.218 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:13.199 7fcc5a849700 1 -- 172.21.2.102:0/4059399252 shutdown_connections 2021-02-10T11:35:13.218 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:13.199 7fcc5a849700 1 -- 172.21.2.102:0/4059399252 >> 172.21.2.102:0/4059399252 conn(0x7fcc540ce420 msgr2=0x7fcc540d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:13.219 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:13.199 7fcc5a849700 1 -- 172.21.2.102:0/4059399252 shutdown_connections 2021-02-10T11:35:13.219 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:13.199 7fcc5a849700 1 -- 172.21.2.102:0/4059399252 wait complete. 2021-02-10T11:35:13.220 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:13.214 7fcc5a849700 1 Processor -- start 2021-02-10T11:35:13.220 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:13.214 7fcc5a849700 1 -- start start 2021-02-10T11:35:13.220 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:13.214 7fcc5a849700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcc54085bc0 0x7fcc540863e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:13.220 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:13.214 7fcc5a849700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcc540954c0 0x7fcc54096300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:13.220 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:13.214 7fcc5a849700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fcc540958c0 0x7fcc54096920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:13.220 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:13.214 7fcc5a849700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc54095d80 con 0x7fcc540954c0 2021-02-10T11:35:13.220 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:13.214 7fcc5a849700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc540bfa60 con 0x7fcc540958c0 2021-02-10T11:35:13.221 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:13.214 7fcc5a849700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc54085fc0 con 0x7fcc54085bc0 2021-02-10T11:35:13.221 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:13.214 7fcc537fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcc540954c0 0x7fcc54096300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:13.221 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:13.214 7fcc537fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcc540954c0 0x7fcc54096300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:36764/0 (socket says 172.21.2.102:36764) 2021-02-10T11:35:13.221 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:13.214 7fcc537fe700 1 -- 172.21.2.102:0/3133008808 learned_addr learned my addr 172.21.2.102:0/3133008808 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:35:13.221 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:13.215 7fcc537fe700 1 -- 172.21.2.102:0/3133008808 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcc54085bc0 msgr2=0x7fcc540863e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:13.221 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:13.215 7fcc537fe700 1 --2- 172.21.2.102:0/3133008808 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcc54085bc0 0x7fcc540863e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:13.221 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:13.215 7fcc537fe700 1 -- 172.21.2.102:0/3133008808 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fcc540958c0 msgr2=0x7fcc54096920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:13.221 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:13.215 7fcc537fe700 1 --2- 172.21.2.102:0/3133008808 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fcc540958c0 0x7fcc54096920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:13.221 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:13.215 7fcc537fe700 1 -- 172.21.2.102:0/3133008808 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc54086c10 con 0x7fcc540954c0 2021-02-10T11:35:13.221 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:13.215 7fcc537fe700 1 --2- 172.21.2.102:0/3133008808 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcc540954c0 0x7fcc54096300 secure :-1 s=READY pgs=2651 cs=0 l=1 rev1=1 rx=0x7fcc40006f40 tx=0x7fcc4000b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:13.221 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:13.215 7fcc50ff9700 1 -- 172.21.2.102:0/3133008808 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcc4000bd80 con 0x7fcc540954c0 2021-02-10T11:35:13.221 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:13.215 7fcc50ff9700 1 -- 172.21.2.102:0/3133008808 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc40000c90 con 0x7fcc540954c0 2021-02-10T11:35:13.221 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:13.215 7fcc5a849700 1 -- 172.21.2.102:0/3133008808 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc54086c10 con 0x7fcc540954c0 2021-02-10T11:35:13.222 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:13.215 7fcc5a849700 1 -- 172.21.2.102:0/3133008808 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc54097230 con 0x7fcc540954c0 2021-02-10T11:35:13.222 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:13.215 7fcc50ff9700 1 -- 172.21.2.102:0/3133008808 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcc400022b0 con 0x7fcc540954c0 2021-02-10T11:35:13.222 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:13.216 7fcc50ff9700 1 -- 172.21.2.102:0/3133008808 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fcc40006b00 con 0x7fcc540954c0 2021-02-10T11:35:13.222 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:13.216 7fcc50ff9700 1 --2- 172.21.2.102:0/3133008808 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcc300186c0 0x7fcc3001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:13.222 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:13.216 7fcc50ff9700 1 -- 172.21.2.102:0/3133008808 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fcc40032550 con 0x7fcc540954c0 2021-02-10T11:35:13.222 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:13.216 7fcc53fff700 1 --2- 172.21.2.102:0/3133008808 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcc300186c0 0x7fcc3001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:13.222 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:13.216 7fcc53fff700 1 --2- 172.21.2.102:0/3133008808 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcc300186c0 0x7fcc3001abc0 crc :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:13.248 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:13.246 7fcc5a849700 1 -- 172.21.2.102:0/3133008808 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc540fe1c0 con 0x7fcc540954c0 2021-02-10T11:35:13.250 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:13.248 7fcc50ff9700 1 -- 172.21.2.102:0/3133008808 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fcc40000990 con 0x7fcc540954c0 2021-02-10T11:35:13.385 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:13.383 7fcc5a849700 1 -- 172.21.2.102:0/3133008808 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"} v 0) v1 -- 0x7fcc540fe1c0 con 0x7fcc540954c0 2021-02-10T11:35:13.642 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:13.640 7fcc50ff9700 1 -- 172.21.2.102:0/3133008808 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"}]=0 v124) v1 ==== 118+0+0 (secure 0 0 0) 0x7fcc40006b00 con 0x7fcc540954c0 2021-02-10T11:35:13.651 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:13.650 7fcc5a849700 1 -- 172.21.2.102:0/3133008808 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcc300186c0 msgr2=0x7fcc3001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:13.651 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:13.650 7fcc5a849700 1 --2- 172.21.2.102:0/3133008808 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcc300186c0 0x7fcc3001abc0 crc :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:13.651 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:13.650 7fcc5a849700 1 -- 172.21.2.102:0/3133008808 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcc540954c0 msgr2=0x7fcc54096300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:13.652 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:13.650 7fcc5a849700 1 --2- 172.21.2.102:0/3133008808 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcc540954c0 0x7fcc54096300 secure :-1 s=READY pgs=2651 cs=0 l=1 rev1=1 rx=0x7fcc40006f40 tx=0x7fcc4000b990).stop 2021-02-10T11:35:13.652 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:13.650 7fcc5a849700 1 -- 172.21.2.102:0/3133008808 shutdown_connections 2021-02-10T11:35:13.652 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:13.650 7fcc5a849700 1 --2- 172.21.2.102:0/3133008808 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcc300186c0 0x7fcc3001abc0 unknown :-1 s=CLOSED pgs=1170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:13.652 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:13.650 7fcc5a849700 1 --2- 172.21.2.102:0/3133008808 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fcc540958c0 0x7fcc54096920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:13.652 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:13.650 7fcc5a849700 1 --2- 172.21.2.102:0/3133008808 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcc540954c0 0x7fcc54096300 unknown :-1 s=CLOSED pgs=2651 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:13.652 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:13.650 7fcc5a849700 1 --2- 172.21.2.102:0/3133008808 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcc54085bc0 0x7fcc540863e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:13.653 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:13.650 7fcc5a849700 1 -- 172.21.2.102:0/3133008808 >> 172.21.2.102:0/3133008808 conn(0x7fcc540bef00 msgr2=0x7fcc5409d1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:13.653 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:13.650 7fcc5a849700 1 -- 172.21.2.102:0/3133008808 shutdown_connections 2021-02-10T11:35:13.653 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:13.650 7fcc5a849700 1 -- 172.21.2.102:0/3133008808 wait complete. 2021-02-10T11:35:13.660 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:35:14.088 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:14.069 7f5762856700 1 Processor -- start 2021-02-10T11:35:14.088 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:14.069 7f5762856700 1 -- start start 2021-02-10T11:35:14.088 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:14.070 7f5762856700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f575c0d0340 0x7f575c0ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:14.088 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:14.070 7f5762856700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f575c0d1c70 0x7f575c0efc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:14.089 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:14.070 7f5762856700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f575c0f0b10 con 0x7f575c0d1870 2021-02-10T11:35:14.089 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:14.070 7f5762856700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f575c0e5ed0 con 0x7f575c0d0340 2021-02-10T11:35:14.089 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:14.070 7f5762856700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f575c1b0be0 con 0x7f575c0d1c70 2021-02-10T11:35:14.089 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:14.070 7f575b7fe700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f575c0d0340 0x7f575c0ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:14.089 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:14.070 7f575b7fe700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f575c0d0340 0x7f575c0ef640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:36774/0 (socket says 172.21.2.102:36774) 2021-02-10T11:35:14.089 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:14.070 7f575b7fe700 1 -- 172.21.2.102:0/3492112345 learned_addr learned my addr 172.21.2.102:0/3492112345 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:35:14.089 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:14.071 7f575b7fe700 1 -- 172.21.2.102:0/3492112345 >> v2:172.21.2.113:3300/0 conn(0x7f575c0d1c70 msgr2=0x7f575c0efc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:14.089 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:14.071 7f575b7fe700 1 --2- 172.21.2.102:0/3492112345 >> v2:172.21.2.113:3300/0 conn(0x7f575c0d1c70 0x7f575c0efc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:14.089 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:14.071 7f575b7fe700 1 -- 172.21.2.102:0/3492112345 >> v1:172.21.2.113:6789/0 conn(0x7f575c0d1870 legacy=0x7f575c1b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:14.089 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:14.071 7f575b7fe700 1 -- 172.21.2.102:0/3492112345 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f575c0f20a0 con 0x7f575c0d0340 2021-02-10T11:35:14.089 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:14.071 7f575b7fe700 1 --2- 172.21.2.102:0/3492112345 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f575c0d0340 0x7f575c0ef640 secure :-1 s=READY pgs=2652 cs=0 l=1 rev1=1 rx=0x7f5750006700 tx=0x7f5750009b00).ready entity=mon.0 client_cookie=d0a36420eb7b3b0a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:14.089 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:14.071 7f575affd700 1 -- 172.21.2.102:0/3492112345 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f575000c0c0 con 0x7f575c0d0340 2021-02-10T11:35:14.089 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:14.071 7f575affd700 1 -- 172.21.2.102:0/3492112345 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f575000c340 con 0x7f575c0d0340 2021-02-10T11:35:14.090 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:14.071 7f575affd700 1 -- 172.21.2.102:0/3492112345 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f575000c820 con 0x7f575c0d0340 2021-02-10T11:35:14.090 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:14.071 7f5762856700 1 -- 172.21.2.102:0/3492112345 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f575c0d0340 msgr2=0x7f575c0ef640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:14.090 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:14.071 7f5762856700 1 --2- 172.21.2.102:0/3492112345 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f575c0d0340 0x7f575c0ef640 secure :-1 s=READY pgs=2652 cs=0 l=1 rev1=1 rx=0x7f5750006700 tx=0x7f5750009b00).stop 2021-02-10T11:35:14.090 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:14.072 7f5762856700 1 -- 172.21.2.102:0/3492112345 shutdown_connections 2021-02-10T11:35:14.090 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:14.072 7f5762856700 1 --2- 172.21.2.102:0/3492112345 >> v2:172.21.2.113:3300/0 conn(0x7f575c0d1c70 0x7f575c0efc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:14.090 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:14.072 7f5762856700 1 --2- 172.21.2.102:0/3492112345 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f575c0d0340 0x7f575c0ef640 unknown :-1 s=CLOSED pgs=2652 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:14.090 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:14.072 7f5762856700 1 -- 172.21.2.102:0/3492112345 >> 172.21.2.102:0/3492112345 conn(0x7f575c0ce450 msgr2=0x7f575c0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:14.090 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:14.072 7f5762856700 1 -- 172.21.2.102:0/3492112345 shutdown_connections 2021-02-10T11:35:14.090 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:14.072 7f5762856700 1 -- 172.21.2.102:0/3492112345 wait complete. 2021-02-10T11:35:14.092 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:14.087 7f5762856700 1 Processor -- start 2021-02-10T11:35:14.092 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:14.087 7f5762856700 1 -- start start 2021-02-10T11:35:14.092 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:14.087 7f5762856700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f575c18f0a0 0x7f575c196ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:14.092 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:14.087 7f5762856700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f575c1970d0 0x7f575c1974d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:14.092 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:14.087 7f5762856700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f575c094650 0x7f575c094a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:14.092 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:14.087 7f5762856700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f575c095bd0 con 0x7f575c094650 2021-02-10T11:35:14.092 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:14.087 7f5762856700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f575c0ad7c0 con 0x7f575c18f0a0 2021-02-10T11:35:14.092 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:14.087 7f5762856700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f575c18ec70 con 0x7f575c1970d0 2021-02-10T11:35:14.093 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:14.087 7f5760e73700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f575c094650 0x7f575c094a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:14.093 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:14.087 7f5760e73700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f575c094650 0x7f575c094a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:36780/0 (socket says 172.21.2.102:36780) 2021-02-10T11:35:14.093 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:14.087 7f5760e73700 1 -- 172.21.2.102:0/1159932058 learned_addr learned my addr 172.21.2.102:0/1159932058 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:35:14.093 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:14.088 7f5760e73700 1 -- 172.21.2.102:0/1159932058 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f575c1970d0 msgr2=0x7f575c1974d0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:35:14.093 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:14.088 7f5760e73700 1 --2- 172.21.2.102:0/1159932058 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f575c1970d0 0x7f575c1974d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:14.093 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:14.088 7f5760e73700 1 -- 172.21.2.102:0/1159932058 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f575c18f0a0 msgr2=0x7f575c196ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:14.093 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:14.088 7f5760e73700 1 --2- 172.21.2.102:0/1159932058 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f575c18f0a0 0x7f575c196ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:14.093 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:14.088 7f5760e73700 1 -- 172.21.2.102:0/1159932058 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f575c095070 con 0x7f575c094650 2021-02-10T11:35:14.093 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:14.088 7f5760e73700 1 --2- 172.21.2.102:0/1159932058 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f575c094650 0x7f575c094a50 secure :-1 s=READY pgs=2653 cs=0 l=1 rev1=1 rx=0x7f574c00b580 tx=0x7f574c00b920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:14.093 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:14.088 7f5758ff9700 1 -- 172.21.2.102:0/1159932058 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f574c006240 con 0x7f575c094650 2021-02-10T11:35:14.093 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:14.088 7f5758ff9700 1 -- 172.21.2.102:0/1159932058 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f574c000c10 con 0x7f575c094650 2021-02-10T11:35:14.094 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:14.088 7f5758ff9700 1 -- 172.21.2.102:0/1159932058 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f574c005770 con 0x7f575c094650 2021-02-10T11:35:14.094 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:14.088 7f5762856700 1 -- 172.21.2.102:0/1159932058 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f575c095070 con 0x7f575c094650 2021-02-10T11:35:14.094 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:14.088 7f5762856700 1 -- 172.21.2.102:0/1159932058 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f575c095600 con 0x7f575c094650 2021-02-10T11:35:14.094 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:14.089 7f5758ff9700 1 -- 172.21.2.102:0/1159932058 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f574c006030 con 0x7f575c094650 2021-02-10T11:35:14.094 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:14.089 7f5758ff9700 1 --2- 172.21.2.102:0/1159932058 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f573c0186c0 0x7f573c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:14.094 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:14.089 7f5758ff9700 1 -- 172.21.2.102:0/1159932058 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f574c0325a0 con 0x7f575c094650 2021-02-10T11:35:14.094 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:14.089 7f575bfff700 1 --2- 172.21.2.102:0/1159932058 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f573c0186c0 0x7f573c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:14.094 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:14.089 7f575bfff700 1 --2- 172.21.2.102:0/1159932058 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f573c0186c0 0x7f573c01abc0 crc :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:14.122 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:14.120 7f5762856700 1 -- 172.21.2.102:0/1159932058 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f575c0ee0e0 con 0x7f575c094650 2021-02-10T11:35:14.125 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:14.123 7f5758ff9700 1 -- 172.21.2.102:0/1159932058 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f574c030d50 con 0x7f575c094650 2021-02-10T11:35:14.253 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:14.251 7f5762856700 1 -- 172.21.2.102:0/1159932058 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f575c0ce450 con 0x7f575c094650 2021-02-10T11:35:14.254 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:14.252 7f5758ff9700 1 -- 172.21.2.102:0/1159932058 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 124 v124) v1 ==== 95+0+2111 (secure 0 0 0) 0x7f574c006030 con 0x7f575c094650 2021-02-10T11:35:14.254 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:35:14.254 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":124,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":11784,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.102:6835/2097086088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6834","nonce":2097086088},{"type":"v1","addr":"172.21.2.102:6835","nonce":2097086088}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":120}],"filesystems":[{"mdsmap":{"epoch":124,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:35:11.628704","modified":"2021-02-10 11:35:13.639277","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11781},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11781":{"gid":11781,"name":"b","rank":0,"incarnation":121,"state":"up:active","state_seq":3,"addr":"172.21.2.113:6833/3334075470","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6832","nonce":3334075470},{"type":"v1","addr":"172.21.2.113:6833","nonce":3334075470}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[31],"metadata_pool":30,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":15}]} 2021-02-10T11:35:14.255 INFO:teuthology.orchestra.run.gibba002.stderr:dumped fsmap epoch 124 2021-02-10T11:35:14.256 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:14.253 7f5762856700 1 -- 172.21.2.102:0/1159932058 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f573c0186c0 msgr2=0x7f573c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:14.256 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:14.253 7f5762856700 1 --2- 172.21.2.102:0/1159932058 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f573c0186c0 0x7f573c01abc0 crc :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:14.256 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:14.253 7f5762856700 1 -- 172.21.2.102:0/1159932058 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f575c094650 msgr2=0x7f575c094a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:14.256 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:14.253 7f5762856700 1 --2- 172.21.2.102:0/1159932058 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f575c094650 0x7f575c094a50 secure :-1 s=READY pgs=2653 cs=0 l=1 rev1=1 rx=0x7f574c00b580 tx=0x7f574c00b920).stop 2021-02-10T11:35:14.256 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:14.254 7f5762856700 1 -- 172.21.2.102:0/1159932058 shutdown_connections 2021-02-10T11:35:14.256 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:14.254 7f5762856700 1 --2- 172.21.2.102:0/1159932058 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f575c094650 0x7f575c094a50 unknown :-1 s=CLOSED pgs=2653 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:14.256 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:14.254 7f5762856700 1 --2- 172.21.2.102:0/1159932058 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f575c1970d0 0x7f575c1974d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:14.256 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:14.254 7f5762856700 1 --2- 172.21.2.102:0/1159932058 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f573c0186c0 0x7f573c01abc0 unknown :-1 s=CLOSED pgs=1171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:14.256 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:14.254 7f5762856700 1 --2- 172.21.2.102:0/1159932058 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f575c18f0a0 0x7f575c196ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:14.256 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:14.254 7f5762856700 1 -- 172.21.2.102:0/1159932058 >> 172.21.2.102:0/1159932058 conn(0x7f575c1a38c0 msgr2=0x7f575c1a2f00 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:14.256 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:14.254 7f5762856700 1 -- 172.21.2.102:0/1159932058 shutdown_connections 2021-02-10T11:35:14.256 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:14.254 7f5762856700 1 -- 172.21.2.102:0/1159932058 wait complete. 2021-02-10T11:35:14.264 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:35:14.695 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:14.677 7f51350a5700 1 Processor -- start 2021-02-10T11:35:14.695 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:14.677 7f51350a5700 1 -- start start 2021-02-10T11:35:14.695 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:14.677 7f51350a5700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f51300d1870 0x7f51301b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:14.695 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:14.677 7f51350a5700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f51301b0a10 0x7f51300cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:14.695 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:14.677 7f51350a5700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f51300d1c70 0x7f51300cf950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:14.695 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:14.677 7f51350a5700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51300e5ed0 con 0x7f51300d1870 2021-02-10T11:35:14.695 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:14.677 7f51350a5700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51300f0b10 con 0x7f51301b0a10 2021-02-10T11:35:14.695 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:14.677 7f51350a5700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51300cff70 con 0x7f51300d1c70 2021-02-10T11:35:14.696 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:14.677 7f512ed9d700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f51300d1870 0x7f51301b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:14.696 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:14.677 7f512ed9d700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f51300d1870 0x7f51301b03f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:36786/0 (socket says 172.21.2.102:36786) 2021-02-10T11:35:14.696 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:14.677 7f512ed9d700 1 -- 172.21.2.102:0/444373502 learned_addr learned my addr 172.21.2.102:0/444373502 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:35:14.696 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:14.677 7f512f59e700 1 --2- 172.21.2.102:0/444373502 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f51300d1c70 0x7f51300cf950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:14.696 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:14.678 7f512e59c700 1 --2- 172.21.2.102:0/444373502 >> v2:172.21.2.113:3300/0 conn(0x7f51301b0a10 0x7f51300cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:14.696 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:14.678 7f512ed9d700 1 -- 172.21.2.102:0/444373502 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f51300d1c70 msgr2=0x7f51300cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:14.696 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:14.678 7f512ed9d700 1 --2- 172.21.2.102:0/444373502 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f51300d1c70 0x7f51300cf950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:14.696 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:14.678 7f512ed9d700 1 -- 172.21.2.102:0/444373502 >> v2:172.21.2.113:3300/0 conn(0x7f51301b0a10 msgr2=0x7f51300cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:14.696 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:14.678 7f512ed9d700 1 --2- 172.21.2.102:0/444373502 >> v2:172.21.2.113:3300/0 conn(0x7f51301b0a10 0x7f51300cf330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:14.696 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:14.678 7f512ed9d700 1 -- 172.21.2.102:0/444373502 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51300d0150 con 0x7f51300d1870 2021-02-10T11:35:14.696 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:14.678 7f512ed9d700 1 --2- 172.21.2.102:0/444373502 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f51300d1870 0x7f51301b03f0 secure :-1 s=READY pgs=2654 cs=0 l=1 rev1=1 rx=0x7f5118006700 tx=0x7f5118009b00).ready entity=mon.0 client_cookie=8482affd2dd7680 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:14.696 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:14.678 7f512dd9b700 1 -- 172.21.2.102:0/444373502 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f511800c0c0 con 0x7f51300d1870 2021-02-10T11:35:14.696 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:14.678 7f512dd9b700 1 -- 172.21.2.102:0/444373502 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f511800c340 con 0x7f51300d1870 2021-02-10T11:35:14.697 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:14.678 7f512dd9b700 1 -- 172.21.2.102:0/444373502 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f511800c820 con 0x7f51300d1870 2021-02-10T11:35:14.697 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:14.678 7f51350a5700 1 -- 172.21.2.102:0/444373502 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f51300d1870 msgr2=0x7f51301b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:14.697 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:14.678 7f51350a5700 1 --2- 172.21.2.102:0/444373502 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f51300d1870 0x7f51301b03f0 secure :-1 s=READY pgs=2654 cs=0 l=1 rev1=1 rx=0x7f5118006700 tx=0x7f5118009b00).stop 2021-02-10T11:35:14.697 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:14.679 7f51350a5700 1 -- 172.21.2.102:0/444373502 shutdown_connections 2021-02-10T11:35:14.697 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:14.679 7f51350a5700 1 --2- 172.21.2.102:0/444373502 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f51300d1c70 0x7f51300cf950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:14.697 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:14.679 7f51350a5700 1 --2- 172.21.2.102:0/444373502 >> v2:172.21.2.113:3300/0 conn(0x7f51301b0a10 0x7f51300cf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:14.697 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:14.679 7f51350a5700 1 --2- 172.21.2.102:0/444373502 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f51300d1870 0x7f51301b03f0 unknown :-1 s=CLOSED pgs=2654 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:14.697 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:14.679 7f51350a5700 1 -- 172.21.2.102:0/444373502 >> 172.21.2.102:0/444373502 conn(0x7f51300ce450 msgr2=0x7f51300d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:14.698 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:14.679 7f51350a5700 1 -- 172.21.2.102:0/444373502 shutdown_connections 2021-02-10T11:35:14.698 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:14.679 7f51350a5700 1 -- 172.21.2.102:0/444373502 wait complete. 2021-02-10T11:35:14.699 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:14.693 7f51350a5700 1 Processor -- start 2021-02-10T11:35:14.699 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:14.693 7f51350a5700 1 -- start start 2021-02-10T11:35:14.699 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:14.694 7f51350a5700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5130195650 0x7f51300b5560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:14.700 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:14.694 7f51350a5700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f51300b5b80 0x7f51301a2e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:14.700 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:14.694 7f51350a5700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f51300acfa0 0x7f51301a3440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:14.700 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:14.694 7f51350a5700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51300b6010 con 0x7f5130195650 2021-02-10T11:35:14.700 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:14.694 7f51350a5700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51301a3a60 con 0x7f51300b5b80 2021-02-10T11:35:14.700 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:14.694 7f51350a5700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51301a3c40 con 0x7f51300acfa0 2021-02-10T11:35:14.700 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:14.694 7f512ed9d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5130195650 0x7f51300b5560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:14.700 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:14.694 7f512ed9d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5130195650 0x7f51300b5560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:36792/0 (socket says 172.21.2.102:36792) 2021-02-10T11:35:14.700 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:14.694 7f512ed9d700 1 -- 172.21.2.102:0/2857819827 learned_addr learned my addr 172.21.2.102:0/2857819827 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:35:14.700 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:14.694 7f512ed9d700 1 -- 172.21.2.102:0/2857819827 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f51300acfa0 msgr2=0x7f51301a3440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:14.700 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:14.694 7f512ed9d700 1 --2- 172.21.2.102:0/2857819827 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f51300acfa0 0x7f51301a3440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:14.700 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:14.694 7f512ed9d700 1 -- 172.21.2.102:0/2857819827 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f51300b5b80 msgr2=0x7f51301a2e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:14.700 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:14.694 7f512ed9d700 1 --2- 172.21.2.102:0/2857819827 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f51300b5b80 0x7f51301a2e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:14.700 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:14.694 7f512ed9d700 1 -- 172.21.2.102:0/2857819827 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51300bcae0 con 0x7f5130195650 2021-02-10T11:35:14.701 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:14.694 7f512ed9d700 1 --2- 172.21.2.102:0/2857819827 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5130195650 0x7f51300b5560 secure :-1 s=READY pgs=2655 cs=0 l=1 rev1=1 rx=0x7f5118006f20 tx=0x7f511800b910).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:14.701 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:14.695 7f51277fe700 1 -- 172.21.2.102:0/2857819827 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f511800bd00 con 0x7f5130195650 2021-02-10T11:35:14.701 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:14.695 7f51277fe700 1 -- 172.21.2.102:0/2857819827 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5118000c90 con 0x7f5130195650 2021-02-10T11:35:14.701 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:14.695 7f51277fe700 1 -- 172.21.2.102:0/2857819827 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f51180022b0 con 0x7f5130195650 2021-02-10T11:35:14.701 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:14.695 7f51350a5700 1 -- 172.21.2.102:0/2857819827 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51300bcae0 con 0x7f5130195650 2021-02-10T11:35:14.701 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:14.695 7f51350a5700 1 -- 172.21.2.102:0/2857819827 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51300bcae0 con 0x7f5130195650 2021-02-10T11:35:14.701 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:14.695 7f51277fe700 1 -- 172.21.2.102:0/2857819827 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f5118002020 con 0x7f5130195650 2021-02-10T11:35:14.701 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:14.696 7f51277fe700 1 --2- 172.21.2.102:0/2857819827 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f51140186c0 0x7f511401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:14.701 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:14.696 7f51277fe700 1 -- 172.21.2.102:0/2857819827 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f51180325b0 con 0x7f5130195650 2021-02-10T11:35:14.701 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:14.696 7f512e59c700 1 --2- 172.21.2.102:0/2857819827 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f51140186c0 0x7f511401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:14.702 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:14.696 7f512e59c700 1 --2- 172.21.2.102:0/2857819827 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f51140186c0 0x7f511401abc0 crc :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:14.729 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:14.727 7f51350a5700 1 -- 172.21.2.102:0/2857819827 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51300b6010 con 0x7f5130195650 2021-02-10T11:35:14.731 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:14.730 7f51277fe700 1 -- 172.21.2.102:0/2857819827 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f5118006070 con 0x7f5130195650 2021-02-10T11:35:14.857 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:14.855 7f51350a5700 1 -- 172.21.2.102:0/2857819827 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f51300b6010 con 0x7f5130195650 2021-02-10T11:35:14.858 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:14.856 7f51277fe700 1 -- 172.21.2.102:0/2857819827 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v145) v1 ==== 74+0+14007 (secure 0 0 0) 0x7f5118006070 con 0x7f5130195650 2021-02-10T11:35:14.858 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:35:14.858 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":145,"fsid":"68db30ea-dbce-4aef-a6df-f01c6fea0513","created":"2021-02-10 11:03:06.331557","modified":"2021-02-10 11:35:12.631037","last_up_change":"2021-02-10 11:03:32.690562","last_in_change":"2021-02-10 11:03:32.690562","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":31,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:03:37.322519","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":30,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:35:09.594644","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"144","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":31,"pool_name":"cephfs_data","create_time":"2021-02-10 11:35:10.619592","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"145","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"be691b4d-a039-408a-af80-9d53f02b6f1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6818","nonce":36088},{"type":"v1","addr":"172.21.2.102:6820","nonce":36088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6822","nonce":36088},{"type":"v1","addr":"172.21.2.102:6824","nonce":36088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6830","nonce":36088},{"type":"v1","addr":"172.21.2.102:6832","nonce":36088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6826","nonce":36088},{"type":"v1","addr":"172.21.2.102:6828","nonce":36088}]},"public_addr":"172.21.2.102:6820/36088","cluster_addr":"172.21.2.102:6824/36088","heartbeat_back_addr":"172.21.2.102:6832/36088","heartbeat_front_addr":"172.21.2.102:6828/36088","state":["exists","up"]},{"osd":1,"uuid":"64e3ccc8-9c69-4d7c-865e-af8e83822635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6819","nonce":36087},{"type":"v1","addr":"172.21.2.102:6821","nonce":36087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6823","nonce":36087},{"type":"v1","addr":"172.21.2.102:6825","nonce":36087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6831","nonce":36087},{"type":"v1","addr":"172.21.2.102:6833","nonce":36087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6827","nonce":36087},{"type":"v1","addr":"172.21.2.102:6829","nonce":36087}]},"public_addr":"172.21.2.102:6821/36087","cluster_addr":"172.21.2.102:6825/36087","heartbeat_back_addr":"172.21.2.102:6833/36087","heartbeat_front_addr":"172.21.2.102:6829/36087","state":["exists","up"]},{"osd":2,"uuid":"153700af-747f-481f-bdf4-d6072d3a842f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6810","nonce":36089},{"type":"v1","addr":"172.21.2.102:6811","nonce":36089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6812","nonce":36089},{"type":"v1","addr":"172.21.2.102:6813","nonce":36089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6816","nonce":36089},{"type":"v1","addr":"172.21.2.102:6817","nonce":36089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6814","nonce":36089},{"type":"v1","addr":"172.21.2.102:6815","nonce":36089}]},"public_addr":"172.21.2.102:6811/36089","cluster_addr":"172.21.2.102:6813/36089","heartbeat_back_addr":"172.21.2.102:6817/36089","heartbeat_front_addr":"172.21.2.102:6815/36089","state":["exists","up"]},{"osd":3,"uuid":"cbc678f4-83a9-45e6-8830-2f7c232b1ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6802","nonce":36090},{"type":"v1","addr":"172.21.2.102:6803","nonce":36090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6804","nonce":36090},{"type":"v1","addr":"172.21.2.102:6805","nonce":36090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6808","nonce":36090},{"type":"v1","addr":"172.21.2.102:6809","nonce":36090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6806","nonce":36090},{"type":"v1","addr":"172.21.2.102:6807","nonce":36090}]},"public_addr":"172.21.2.102:6803/36090","cluster_addr":"172.21.2.102:6805/36090","heartbeat_back_addr":"172.21.2.102:6809/36090","heartbeat_front_addr":"172.21.2.102:6807/36090","state":["exists","up"]},{"osd":4,"uuid":"c01e4008-b258-4f10-afd2-9e7f37214351","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6808","nonce":36368},{"type":"v1","addr":"172.21.2.113:6811","nonce":36368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6814","nonce":36368},{"type":"v1","addr":"172.21.2.113:6818","nonce":36368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6826","nonce":36368},{"type":"v1","addr":"172.21.2.113:6828","nonce":36368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6820","nonce":36368},{"type":"v1","addr":"172.21.2.113:6824","nonce":36368}]},"public_addr":"172.21.2.113:6811/36368","cluster_addr":"172.21.2.113:6818/36368","heartbeat_back_addr":"172.21.2.113:6828/36368","heartbeat_front_addr":"172.21.2.113:6824/36368","state":["exists","up"]},{"osd":5,"uuid":"551f6007-daf8-4a98-8ee0-c2b16bcccb5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6802","nonce":36366},{"type":"v1","addr":"172.21.2.113:6805","nonce":36366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6807","nonce":36366},{"type":"v1","addr":"172.21.2.113:6810","nonce":36366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6819","nonce":36366},{"type":"v1","addr":"172.21.2.113:6822","nonce":36366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6812","nonce":36366},{"type":"v1","addr":"172.21.2.113:6815","nonce":36366}]},"public_addr":"172.21.2.113:6805/36366","cluster_addr":"172.21.2.113:6810/36366","heartbeat_back_addr":"172.21.2.113:6822/36366","heartbeat_front_addr":"172.21.2.113:6815/36366","state":["exists","up"]},{"osd":6,"uuid":"bf67c040-0e50-48be-b102-c3943bc9ebef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6800","nonce":36367},{"type":"v1","addr":"172.21.2.113:6801","nonce":36367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6803","nonce":36367},{"type":"v1","addr":"172.21.2.113:6804","nonce":36367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6813","nonce":36367},{"type":"v1","addr":"172.21.2.113:6816","nonce":36367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6806","nonce":36367},{"type":"v1","addr":"172.21.2.113:6809","nonce":36367}]},"public_addr":"172.21.2.113:6801/36367","cluster_addr":"172.21.2.113:6804/36367","heartbeat_back_addr":"172.21.2.113:6816/36367","heartbeat_front_addr":"172.21.2.113:6809/36367","state":["exists","up"]},{"osd":7,"uuid":"f5461f80-018d-428f-be89-2eb2bcbe7bb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6817","nonce":36365},{"type":"v1","addr":"172.21.2.113:6821","nonce":36365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6823","nonce":36365},{"type":"v1","addr":"172.21.2.113:6825","nonce":36365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6830","nonce":36365},{"type":"v1","addr":"172.21.2.113:6831","nonce":36365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6827","nonce":36365},{"type":"v1","addr":"172.21.2.113:6829","nonce":36365}]},"public_addr":"172.21.2.113:6821/36365","cluster_addr":"172.21.2.113:6825/36365","heartbeat_back_addr":"172.21.2.113:6831/36365","heartbeat_front_addr":"172.21.2.113:6829/36365","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:35:14.859 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:14.858 7f51350a5700 1 -- 172.21.2.102:0/2857819827 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f51140186c0 msgr2=0x7f511401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:14.859 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:14.858 7f51350a5700 1 --2- 172.21.2.102:0/2857819827 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f51140186c0 0x7f511401abc0 crc :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:14.860 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:14.858 7f51350a5700 1 -- 172.21.2.102:0/2857819827 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5130195650 msgr2=0x7f51300b5560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:14.860 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:14.858 7f51350a5700 1 --2- 172.21.2.102:0/2857819827 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5130195650 0x7f51300b5560 secure :-1 s=READY pgs=2655 cs=0 l=1 rev1=1 rx=0x7f5118006f20 tx=0x7f511800b910).stop 2021-02-10T11:35:14.860 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:14.858 7f51350a5700 1 -- 172.21.2.102:0/2857819827 shutdown_connections 2021-02-10T11:35:14.860 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:14.858 7f51350a5700 1 --2- 172.21.2.102:0/2857819827 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f51300acfa0 0x7f51301a3440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:14.860 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:14.858 7f51350a5700 1 --2- 172.21.2.102:0/2857819827 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f51140186c0 0x7f511401abc0 unknown :-1 s=CLOSED pgs=1172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:14.860 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:14.858 7f51350a5700 1 --2- 172.21.2.102:0/2857819827 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f51300b5b80 0x7f51301a2e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:14.860 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:14.858 7f51350a5700 1 --2- 172.21.2.102:0/2857819827 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5130195650 0x7f51300b5560 unknown :-1 s=CLOSED pgs=2655 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:14.860 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:14.858 7f51350a5700 1 -- 172.21.2.102:0/2857819827 >> 172.21.2.102:0/2857819827 conn(0x7f51300b3040 msgr2=0x7f51300c3250 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:14.860 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:14.858 7f51350a5700 1 -- 172.21.2.102:0/2857819827 shutdown_connections 2021-02-10T11:35:14.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:14.858 7f51350a5700 1 -- 172.21.2.102:0/2857819827 wait complete. 2021-02-10T11:35:14.869 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:35:15.310 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:15.292 7f0ac0815700 1 Processor -- start 2021-02-10T11:35:15.310 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:15.292 7f0ac0815700 1 -- start start 2021-02-10T11:35:15.311 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:15.293 7f0ac0815700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f0abc0d1c80 0x7f0abc0cef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:15.311 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:15.293 7f0ac0815700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0abc0cf590 0x7f0abc0cf990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:15.311 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:15.293 7f0ac0815700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0abc0fe1d0 con 0x7f0abc0d1880 2021-02-10T11:35:15.311 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:15.293 7f0ac0815700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0abc1b0cc0 con 0x7f0abc0d1c80 2021-02-10T11:35:15.311 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:15.293 7f0ac0815700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0abc109a70 con 0x7f0abc0cf590 2021-02-10T11:35:15.311 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:15.293 7f0ab9e1b700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f0abc0d1c80 0x7f0abc0cef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:15.311 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:15.293 7f0aba61c700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f0abc0d1880 0x7f0abc1b04d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:57256/0 (socket says 172.21.2.102:57256) 2021-02-10T11:35:15.311 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:15.293 7f0aba61c700 1 -- 172.21.2.102:0/1318857684 learned_addr learned my addr 172.21.2.102:0/1318857684 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:35:15.311 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:15.293 7f0ab9e1b700 1 -- 172.21.2.102:0/1318857684 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0abc0cf590 msgr2=0x7f0abc0cf990 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:35:15.311 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:15.293 7f0ab9e1b700 1 --2- 172.21.2.102:0/1318857684 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0abc0cf590 0x7f0abc0cf990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:15.311 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:15.293 7f0ab9e1b700 1 -- 172.21.2.102:0/1318857684 >> v1:172.21.2.102:6789/0 conn(0x7f0abc0d1880 legacy=0x7f0abc1b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:15.311 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:15.293 7f0ab9e1b700 1 -- 172.21.2.102:0/1318857684 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0abc088070 con 0x7f0abc0d1c80 2021-02-10T11:35:15.312 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:15.294 7f0ab9e1b700 1 --2- 172.21.2.102:0/1318857684 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0abc0d1c80 0x7f0abc0cef70 secure :-1 s=READY pgs=2657 cs=0 l=1 rev1=1 rx=0x7f0ab0006650 tx=0x7f0ab0009a80).ready entity=mon.0 client_cookie=791ed6c4cad2a86e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:15.312 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:15.294 7f0ab961a700 1 -- 172.21.2.102:0/1318857684 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0ab00009b0 con 0x7f0abc0d1c80 2021-02-10T11:35:15.312 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:15.294 7f0ab961a700 1 -- 172.21.2.102:0/1318857684 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ab000c0c0 con 0x7f0abc0d1c80 2021-02-10T11:35:15.312 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:15.294 7f0ab961a700 1 -- 172.21.2.102:0/1318857684 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0ab000c520 con 0x7f0abc0d1c80 2021-02-10T11:35:15.312 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:15.294 7f0ac0815700 1 -- 172.21.2.102:0/1318857684 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0abc0d1c80 msgr2=0x7f0abc0cef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:15.312 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:15.294 7f0ac0815700 1 --2- 172.21.2.102:0/1318857684 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0abc0d1c80 0x7f0abc0cef70 secure :-1 s=READY pgs=2657 cs=0 l=1 rev1=1 rx=0x7f0ab0006650 tx=0x7f0ab0009a80).stop 2021-02-10T11:35:15.312 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:15.294 7f0ac0815700 1 -- 172.21.2.102:0/1318857684 shutdown_connections 2021-02-10T11:35:15.312 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:15.294 7f0ac0815700 1 --2- 172.21.2.102:0/1318857684 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0abc0d1c80 0x7f0abc0cef70 unknown :-1 s=CLOSED pgs=2657 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:15.312 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:15.294 7f0ac0815700 1 --2- 172.21.2.102:0/1318857684 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0abc0cf590 0x7f0abc0cf990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:15.312 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:15.294 7f0ac0815700 1 -- 172.21.2.102:0/1318857684 >> 172.21.2.102:0/1318857684 conn(0x7f0abc0ce430 msgr2=0x7f0abc0d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:15.312 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:15.295 7f0ac0815700 1 -- 172.21.2.102:0/1318857684 shutdown_connections 2021-02-10T11:35:15.312 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:15.295 7f0ac0815700 1 -- 172.21.2.102:0/1318857684 wait complete. 2021-02-10T11:35:15.313 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:15.309 7f0ac0815700 1 Processor -- start 2021-02-10T11:35:15.313 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:15.309 7f0ac0815700 1 -- start start 2021-02-10T11:35:15.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:15.309 7f0ac0815700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0abc086520 0x7f0abc1a1cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:15.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:15.309 7f0ac0815700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0abc086920 0x7f0abc187e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:15.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:15.309 7f0ac0815700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0abc1a22d0 0x7f0abc1884a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:15.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:15.309 7f0ac0815700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0abc099600 con 0x7f0abc086920 2021-02-10T11:35:15.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:15.309 7f0ac0815700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0abc0999f0 con 0x7f0abc086520 2021-02-10T11:35:15.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:15.309 7f0ac0815700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0abc188ac0 con 0x7f0abc1a22d0 2021-02-10T11:35:15.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:15.309 7f0ab9e1b700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0abc086920 0x7f0abc187e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:15.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:15.309 7f0ab9e1b700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0abc086920 0x7f0abc187e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:36808/0 (socket says 172.21.2.102:36808) 2021-02-10T11:35:15.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:15.309 7f0ab9e1b700 1 -- 172.21.2.102:0/3633890446 learned_addr learned my addr 172.21.2.102:0/3633890446 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:35:15.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:15.310 7f0ab9e1b700 1 -- 172.21.2.102:0/3633890446 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0abc1a22d0 msgr2=0x7f0abc1884a0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:35:15.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:15.310 7f0ab9e1b700 1 --2- 172.21.2.102:0/3633890446 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0abc1a22d0 0x7f0abc1884a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:15.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:15.310 7f0ab9e1b700 1 -- 172.21.2.102:0/3633890446 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0abc086520 msgr2=0x7f0abc1a1cb0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:35:15.318 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:15.310 7f0ab9e1b700 1 --2- 172.21.2.102:0/3633890446 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0abc086520 0x7f0abc1a1cb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:15.318 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:15.310 7f0ab9e1b700 1 -- 172.21.2.102:0/3633890446 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0abc188ce0 con 0x7f0abc086920 2021-02-10T11:35:15.318 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:15.310 7f0ab9e1b700 1 --2- 172.21.2.102:0/3633890446 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0abc086920 0x7f0abc187e80 secure :-1 s=READY pgs=2658 cs=0 l=1 rev1=1 rx=0x7f0ab0009650 tx=0x7f0ab000b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:15.318 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:15.310 7f0aaaffd700 1 -- 172.21.2.102:0/3633890446 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0ab00093a0 con 0x7f0abc086920 2021-02-10T11:35:15.318 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:15.310 7f0aaaffd700 1 -- 172.21.2.102:0/3633890446 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ab000bde0 con 0x7f0abc086920 2021-02-10T11:35:15.318 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:15.310 7f0aaaffd700 1 -- 172.21.2.102:0/3633890446 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0ab00052e0 con 0x7f0abc086920 2021-02-10T11:35:15.318 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:15.310 7f0ac0815700 1 -- 172.21.2.102:0/3633890446 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0abc188ce0 con 0x7f0abc086920 2021-02-10T11:35:15.320 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:15.310 7f0ac0815700 1 -- 172.21.2.102:0/3633890446 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0abc189270 con 0x7f0abc086920 2021-02-10T11:35:15.320 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:15.311 7f0aaaffd700 1 -- 172.21.2.102:0/3633890446 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f0ab0006030 con 0x7f0abc086920 2021-02-10T11:35:15.320 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:15.311 7f0aaaffd700 1 --2- 172.21.2.102:0/3633890446 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0aa00186c0 0x7f0aa001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:15.320 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:15.311 7f0aaaffd700 1 -- 172.21.2.102:0/3633890446 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f0ab0032690 con 0x7f0abc086920 2021-02-10T11:35:15.320 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:15.311 7f0aba61c700 1 --2- 172.21.2.102:0/3633890446 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0aa00186c0 0x7f0aa001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:15.320 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:15.311 7f0aba61c700 1 --2- 172.21.2.102:0/3633890446 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0aa00186c0 0x7f0aa001abc0 crc :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:15.345 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:15.343 7f0ac0815700 1 -- 172.21.2.102:0/3633890446 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0abc0fe1d0 con 0x7f0abc086920 2021-02-10T11:35:15.348 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:15.346 7f0aaaffd700 1 -- 172.21.2.102:0/3633890446 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f0ab00093a0 con 0x7f0abc086920 2021-02-10T11:35:15.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:15.474 7f0ac0815700 1 -- 172.21.2.102:0/3633890446 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f0abc0999a0 con 0x7f0abc086920 2021-02-10T11:35:15.480 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:15.478 7f0aaaffd700 1 -- 172.21.2.102:0/3633890446 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.0 v81) v1 ==== 184+0+0 (secure 0 0 0) 0x7f0ab00052e0 con 0x7f0abc086920 2021-02-10T11:35:15.484 INFO:teuthology.orchestra.run.gibba002.stderr:updated caps for client.0 2021-02-10T11:35:15.485 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:15.483 7f0ac0815700 1 -- 172.21.2.102:0/3633890446 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0aa00186c0 msgr2=0x7f0aa001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:15.486 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:15.483 7f0ac0815700 1 --2- 172.21.2.102:0/3633890446 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0aa00186c0 0x7f0aa001abc0 crc :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:15.486 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:15.483 7f0ac0815700 1 -- 172.21.2.102:0/3633890446 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0abc086920 msgr2=0x7f0abc187e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:15.486 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:15.483 7f0ac0815700 1 --2- 172.21.2.102:0/3633890446 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0abc086920 0x7f0abc187e80 secure :-1 s=READY pgs=2658 cs=0 l=1 rev1=1 rx=0x7f0ab0009650 tx=0x7f0ab000b960).stop 2021-02-10T11:35:15.486 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:15.483 7f0ac0815700 1 -- 172.21.2.102:0/3633890446 shutdown_connections 2021-02-10T11:35:15.486 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:15.483 7f0ac0815700 1 --2- 172.21.2.102:0/3633890446 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0abc1a22d0 0x7f0abc1884a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:15.487 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:15.483 7f0ac0815700 1 --2- 172.21.2.102:0/3633890446 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0abc086920 0x7f0abc187e80 unknown :-1 s=CLOSED pgs=2658 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:15.487 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:15.483 7f0ac0815700 1 --2- 172.21.2.102:0/3633890446 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0aa00186c0 0x7f0aa001abc0 unknown :-1 s=CLOSED pgs=1173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:15.487 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:15.483 7f0ac0815700 1 --2- 172.21.2.102:0/3633890446 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0abc086520 0x7f0abc1a1cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:15.487 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:15.483 7f0ac0815700 1 -- 172.21.2.102:0/3633890446 >> 172.21.2.102:0/3633890446 conn(0x7f0abc19f880 msgr2=0x7f0abc08f890 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:15.487 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:15.483 7f0ac0815700 1 -- 172.21.2.102:0/3633890446 shutdown_connections 2021-02-10T11:35:15.487 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:15.483 7f0ac0815700 1 -- 172.21.2.102:0/3633890446 wait complete. 2021-02-10T11:35:15.494 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.1 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:35:15.931 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:15.912 7fc2f3c1f700 1 Processor -- start 2021-02-10T11:35:15.932 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:15.913 7fc2f3c1f700 1 -- start start 2021-02-10T11:35:15.932 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:15.913 7fc2f3c1f700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fc2ec0d1880 0x7fc2ec1b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:15.932 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:15.913 7fc2f3c1f700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc2ec1b0af0 0x7fc2ec108950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:15.932 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:15.913 7fc2f3c1f700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2ec0fe1d0 con 0x7fc2ec0d1c80 2021-02-10T11:35:15.932 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:15.913 7fc2f3c1f700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2ec109a70 con 0x7fc2ec0d1880 2021-02-10T11:35:15.932 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:15.913 7fc2f3c1f700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2ec109760 con 0x7fc2ec1b0af0 2021-02-10T11:35:15.932 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:15.913 7fc2f1a3b700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fc2ec0d1880 0x7fc2ec1b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:15.932 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:15.913 7fc2f1a3b700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fc2ec0d1880 0x7fc2ec1b04d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:36814/0 (socket says 172.21.2.102:36814) 2021-02-10T11:35:15.932 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:15.913 7fc2f223c700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fc2ec0d1c80 0x7fc2ec108f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:57270/0 (socket says 172.21.2.102:57270) 2021-02-10T11:35:15.932 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:15.913 7fc2f1a3b700 1 -- 172.21.2.102:0/2372469831 learned_addr learned my addr 172.21.2.102:0/2372469831 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:35:15.932 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:15.914 7fc2f0a39700 1 -- 172.21.2.102:0/2372469831 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 476089764 0 0) 0x7fc2e4001100 con 0x7fc2ec0d1c80 2021-02-10T11:35:15.932 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:15.914 7fc2f0a39700 1 -- 172.21.2.102:0/2372469831 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2d8002fe0 con 0x7fc2ec0d1c80 2021-02-10T11:35:15.933 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:15.914 7fc2f123a700 1 --2- 172.21.2.102:0/2372469831 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc2ec1b0af0 0x7fc2ec108950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:15.933 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:15.914 7fc2f1a3b700 1 -- 172.21.2.102:0/2372469831 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc2ec1b0af0 msgr2=0x7fc2ec108950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:15.933 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:15.914 7fc2f1a3b700 1 --2- 172.21.2.102:0/2372469831 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc2ec1b0af0 0x7fc2ec108950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:15.933 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:15.914 7fc2f1a3b700 1 -- 172.21.2.102:0/2372469831 >> v1:172.21.2.102:6789/0 conn(0x7fc2ec0d1c80 legacy=0x7fc2ec108f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:15.933 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:15.914 7fc2f1a3b700 1 -- 172.21.2.102:0/2372469831 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2ec088070 con 0x7fc2ec0d1880 2021-02-10T11:35:15.933 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:15.914 7fc2f123a700 1 --2- 172.21.2.102:0/2372469831 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc2ec1b0af0 0x7fc2ec108950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:35:15.933 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:15.914 7fc2f1a3b700 1 --2- 172.21.2.102:0/2372469831 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc2ec0d1880 0x7fc2ec1b04d0 secure :-1 s=READY pgs=2660 cs=0 l=1 rev1=1 rx=0x7fc2e8006700 tx=0x7fc2e8009b80).ready entity=mon.0 client_cookie=54c908536161c88 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:15.933 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:15.914 7fc2f0a39700 1 -- 172.21.2.102:0/2372469831 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc2e800c0c0 con 0x7fc2ec0d1880 2021-02-10T11:35:15.933 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:15.915 7fc2f0a39700 1 -- 172.21.2.102:0/2372469831 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2e800c340 con 0x7fc2ec0d1880 2021-02-10T11:35:15.933 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:15.915 7fc2f0a39700 1 -- 172.21.2.102:0/2372469831 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc2e800c860 con 0x7fc2ec0d1880 2021-02-10T11:35:15.933 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:15.915 7fc2f3c1f700 1 -- 172.21.2.102:0/2372469831 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc2ec0d1880 msgr2=0x7fc2ec1b04d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:15.933 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:15.915 7fc2f3c1f700 1 --2- 172.21.2.102:0/2372469831 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc2ec0d1880 0x7fc2ec1b04d0 secure :-1 s=READY pgs=2660 cs=0 l=1 rev1=1 rx=0x7fc2e8006700 tx=0x7fc2e8009b80).stop 2021-02-10T11:35:15.933 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:15.915 7fc2f3c1f700 1 -- 172.21.2.102:0/2372469831 shutdown_connections 2021-02-10T11:35:15.934 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:15.915 7fc2f3c1f700 1 --2- 172.21.2.102:0/2372469831 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc2ec1b0af0 0x7fc2ec108950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:15.934 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:15.915 7fc2f3c1f700 1 --2- 172.21.2.102:0/2372469831 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc2ec0d1880 0x7fc2ec1b04d0 unknown :-1 s=CLOSED pgs=2660 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:15.934 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:15.915 7fc2f3c1f700 1 -- 172.21.2.102:0/2372469831 >> 172.21.2.102:0/2372469831 conn(0x7fc2ec0ce430 msgr2=0x7fc2ec0d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:15.934 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:15.915 7fc2f3c1f700 1 -- 172.21.2.102:0/2372469831 shutdown_connections 2021-02-10T11:35:15.934 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:15.915 7fc2f3c1f700 1 -- 172.21.2.102:0/2372469831 wait complete. 2021-02-10T11:35:15.935 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:15.930 7fc2f3c1f700 1 Processor -- start 2021-02-10T11:35:15.935 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:15.930 7fc2f3c1f700 1 -- start start 2021-02-10T11:35:15.935 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:15.930 7fc2f3c1f700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc2ec1a4f80 0x7fc2ec0adac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:15.936 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:15.930 7fc2f3c1f700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc2ec0ae0e0 0x7fc2ec199a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:15.936 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:15.930 7fc2f3c1f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc2ec19a080 0x7fc2ec19a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:15.936 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:15.930 7fc2f3c1f700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2ec19ab20 con 0x7fc2ec19a080 2021-02-10T11:35:15.936 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:15.930 7fc2f3c1f700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2ec19ad40 con 0x7fc2ec0ae0e0 2021-02-10T11:35:15.936 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:15.930 7fc2f3c1f700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2ec19d310 con 0x7fc2ec1a4f80 2021-02-10T11:35:15.936 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:15.931 7fc2f223c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc2ec19a080 0x7fc2ec19a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:15.936 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:15.931 7fc2f223c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc2ec19a080 0x7fc2ec19a480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:36820/0 (socket says 172.21.2.102:36820) 2021-02-10T11:35:15.936 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:15.931 7fc2f223c700 1 -- 172.21.2.102:0/952297529 learned_addr learned my addr 172.21.2.102:0/952297529 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:35:15.936 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:15.931 7fc2f223c700 1 -- 172.21.2.102:0/952297529 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc2ec1a4f80 msgr2=0x7fc2ec0adac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:15.936 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:15.931 7fc2f223c700 1 --2- 172.21.2.102:0/952297529 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc2ec1a4f80 0x7fc2ec0adac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:15.936 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:15.931 7fc2f223c700 1 -- 172.21.2.102:0/952297529 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc2ec0ae0e0 msgr2=0x7fc2ec199a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:15.936 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:15.931 7fc2f223c700 1 --2- 172.21.2.102:0/952297529 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc2ec0ae0e0 0x7fc2ec199a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:15.936 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:15.931 7fc2f223c700 1 -- 172.21.2.102:0/952297529 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2ec19d500 con 0x7fc2ec19a080 2021-02-10T11:35:15.937 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:15.931 7fc2f223c700 1 --2- 172.21.2.102:0/952297529 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc2ec19a080 0x7fc2ec19a480 secure :-1 s=READY pgs=2661 cs=0 l=1 rev1=1 rx=0x7fc2e400b5c0 tx=0x7fc2e400b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:15.937 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:15.932 7fc2de7fc700 1 -- 172.21.2.102:0/952297529 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc2e400bd50 con 0x7fc2ec19a080 2021-02-10T11:35:15.937 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:15.932 7fc2de7fc700 1 -- 172.21.2.102:0/952297529 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2e4006030 con 0x7fc2ec19a080 2021-02-10T11:35:15.937 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:15.932 7fc2de7fc700 1 -- 172.21.2.102:0/952297529 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc2e4004590 con 0x7fc2ec19a080 2021-02-10T11:35:15.937 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:15.932 7fc2f3c1f700 1 -- 172.21.2.102:0/952297529 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2ec19d500 con 0x7fc2ec19a080 2021-02-10T11:35:15.937 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:15.932 7fc2f3c1f700 1 -- 172.21.2.102:0/952297529 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2ec19d500 con 0x7fc2ec19a080 2021-02-10T11:35:15.937 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:15.933 7fc2de7fc700 1 -- 172.21.2.102:0/952297529 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fc2e4006210 con 0x7fc2ec19a080 2021-02-10T11:35:15.937 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:15.933 7fc2de7fc700 1 --2- 172.21.2.102:0/952297529 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc2d40186c0 0x7fc2d401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:15.937 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:15.933 7fc2de7fc700 1 -- 172.21.2.102:0/952297529 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fc2e40327d0 con 0x7fc2ec19a080 2021-02-10T11:35:15.937 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:15.933 7fc2f1a3b700 1 --2- 172.21.2.102:0/952297529 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc2d40186c0 0x7fc2d401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:15.937 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:15.933 7fc2f1a3b700 1 --2- 172.21.2.102:0/952297529 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc2d40186c0 0x7fc2d401abc0 crc :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:15.966 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:15.963 7fc2f3c1f700 1 -- 172.21.2.102:0/952297529 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2ec082990 con 0x7fc2ec19a080 2021-02-10T11:35:15.968 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:15.966 7fc2de7fc700 1 -- 172.21.2.102:0/952297529 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fc2e4004ef0 con 0x7fc2ec19a080 2021-02-10T11:35:16.098 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:16.096 7fc2f3c1f700 1 -- 172.21.2.102:0/952297529 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7fc2ec082990 con 0x7fc2ec19a080 2021-02-10T11:35:16.101 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:16.100 7fc2de7fc700 1 -- 172.21.2.102:0/952297529 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.1 v82) v1 ==== 184+0+0 (secure 0 0 0) 0x7fc2e4006b80 con 0x7fc2ec19a080 2021-02-10T11:35:16.105 INFO:teuthology.orchestra.run.gibba002.stderr:updated caps for client.1 2021-02-10T11:35:16.107 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:16.105 7fc2f3c1f700 1 -- 172.21.2.102:0/952297529 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc2d40186c0 msgr2=0x7fc2d401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:16.107 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:16.105 7fc2f3c1f700 1 --2- 172.21.2.102:0/952297529 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc2d40186c0 0x7fc2d401abc0 crc :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:16.107 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:16.105 7fc2f3c1f700 1 -- 172.21.2.102:0/952297529 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc2ec19a080 msgr2=0x7fc2ec19a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:16.107 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:16.105 7fc2f3c1f700 1 --2- 172.21.2.102:0/952297529 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc2ec19a080 0x7fc2ec19a480 secure :-1 s=READY pgs=2661 cs=0 l=1 rev1=1 rx=0x7fc2e400b5c0 tx=0x7fc2e400b960).stop 2021-02-10T11:35:16.108 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:16.105 7fc2f3c1f700 1 -- 172.21.2.102:0/952297529 shutdown_connections 2021-02-10T11:35:16.108 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:16.105 7fc2f3c1f700 1 --2- 172.21.2.102:0/952297529 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc2ec19a080 0x7fc2ec19a480 unknown :-1 s=CLOSED pgs=2661 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:16.108 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:16.105 7fc2f3c1f700 1 --2- 172.21.2.102:0/952297529 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc2d40186c0 0x7fc2d401abc0 unknown :-1 s=CLOSED pgs=1174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:16.108 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:16.105 7fc2f3c1f700 1 --2- 172.21.2.102:0/952297529 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc2ec0ae0e0 0x7fc2ec199a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:16.108 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:16.105 7fc2f3c1f700 1 --2- 172.21.2.102:0/952297529 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc2ec1a4f80 0x7fc2ec0adac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:16.108 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:16.105 7fc2f3c1f700 1 -- 172.21.2.102:0/952297529 >> 172.21.2.102:0/952297529 conn(0x7fc2ec09a140 msgr2=0x7fc2ec090b40 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:16.108 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:16.105 7fc2f3c1f700 1 -- 172.21.2.102:0/952297529 shutdown_connections 2021-02-10T11:35:16.108 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:16.105 7fc2f3c1f700 1 -- 172.21.2.102:0/952297529 wait complete. 2021-02-10T11:35:16.115 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.2 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:35:16.556 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:16.538 7fbc0a9e1700 1 Processor -- start 2021-02-10T11:35:16.556 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:16.538 7fbc0a9e1700 1 -- start start 2021-02-10T11:35:16.556 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:16.538 7fbc0a9e1700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fbc040d1880 0x7fbc041b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:16.556 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:16.538 7fbc0a9e1700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fbc041b0af0 0x7fbc04108950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:16.556 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:16.538 7fbc0a9e1700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc040fe1d0 con 0x7fbc040d1c80 2021-02-10T11:35:16.556 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:16.538 7fbc0a9e1700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc04109a70 con 0x7fbc040d1880 2021-02-10T11:35:16.556 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:16.538 7fbc0a9e1700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc04109760 con 0x7fbc041b0af0 2021-02-10T11:35:16.557 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:16.539 7fbc03fff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fbc040d1880 0x7fbc041b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:16.557 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:16.539 7fbc03fff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fbc040d1880 0x7fbc041b04d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:36824/0 (socket says 172.21.2.102:36824) 2021-02-10T11:35:16.557 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:16.539 7fbc03fff700 1 -- 172.21.2.102:0/1053128563 learned_addr learned my addr 172.21.2.102:0/1053128563 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:35:16.557 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:16.539 7fbc03fff700 1 -- 172.21.2.102:0/1053128563 >> v2:172.21.2.113:3300/0 conn(0x7fbc041b0af0 msgr2=0x7fbc04108950 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:35:16.557 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:16.539 7fbc03fff700 1 --2- 172.21.2.102:0/1053128563 >> v2:172.21.2.113:3300/0 conn(0x7fbc041b0af0 0x7fbc04108950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:16.557 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:16.539 7fbc03fff700 1 -- 172.21.2.102:0/1053128563 >> v1:172.21.2.113:6789/0 conn(0x7fbc040d1c80 legacy=0x7fbc04108f70 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:35:16.557 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:16.539 7fbc03fff700 1 -- 172.21.2.102:0/1053128563 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc04088070 con 0x7fbc040d1880 2021-02-10T11:35:16.557 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:16.539 7fbc03fff700 1 --2- 172.21.2.102:0/1053128563 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbc040d1880 0x7fbc041b04d0 secure :-1 s=READY pgs=2662 cs=0 l=1 rev1=1 rx=0x7fbbf4006700 tx=0x7fbbf4009b80).ready entity=mon.0 client_cookie=cd2420f7c95390f4 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:16.557 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:16.539 7fbc02ffd700 1 -- 172.21.2.102:0/1053128563 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbbf400c0c0 con 0x7fbc040d1880 2021-02-10T11:35:16.557 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:16.539 7fbc02ffd700 1 -- 172.21.2.102:0/1053128563 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbf400c440 con 0x7fbc040d1880 2021-02-10T11:35:16.557 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:16.539 7fbc02ffd700 1 -- 172.21.2.102:0/1053128563 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbbf400c920 con 0x7fbc040d1880 2021-02-10T11:35:16.557 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:16.539 7fbc0a9e1700 1 -- 172.21.2.102:0/1053128563 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbc040d1880 msgr2=0x7fbc041b04d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:16.557 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:16.539 7fbc0a9e1700 1 --2- 172.21.2.102:0/1053128563 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbc040d1880 0x7fbc041b04d0 secure :-1 s=READY pgs=2662 cs=0 l=1 rev1=1 rx=0x7fbbf4006700 tx=0x7fbbf4009b80).stop 2021-02-10T11:35:16.558 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:16.540 7fbc0a9e1700 1 -- 172.21.2.102:0/1053128563 shutdown_connections 2021-02-10T11:35:16.558 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:16.540 7fbc0a9e1700 1 --2- 172.21.2.102:0/1053128563 >> v2:172.21.2.113:3300/0 conn(0x7fbc041b0af0 0x7fbc04108950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:16.558 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:16.540 7fbc0a9e1700 1 --2- 172.21.2.102:0/1053128563 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbc040d1880 0x7fbc041b04d0 unknown :-1 s=CLOSED pgs=2662 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:16.558 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:16.540 7fbc0a9e1700 1 -- 172.21.2.102:0/1053128563 >> 172.21.2.102:0/1053128563 conn(0x7fbc040ce430 msgr2=0x7fbc040d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:16.558 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:16.540 7fbc0a9e1700 1 -- 172.21.2.102:0/1053128563 shutdown_connections 2021-02-10T11:35:16.558 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:16.540 7fbc0a9e1700 1 -- 172.21.2.102:0/1053128563 wait complete. 2021-02-10T11:35:16.560 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:16.554 7fbc0a9e1700 1 Processor -- start 2021-02-10T11:35:16.560 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:16.554 7fbc0a9e1700 1 -- start start 2021-02-10T11:35:16.560 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:16.555 7fbc0a9e1700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbc04086520 0x7fbc041a1cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:16.560 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:16.555 7fbc0a9e1700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbc04086920 0x7fbc04187e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:16.560 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:16.555 7fbc0a9e1700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fbc041a22d0 0x7fbc041884a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:16.560 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:16.555 7fbc0a9e1700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc04099600 con 0x7fbc04086520 2021-02-10T11:35:16.560 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:16.555 7fbc0a9e1700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc040999f0 con 0x7fbc041a22d0 2021-02-10T11:35:16.560 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:16.555 7fbc0a9e1700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc04188ac0 con 0x7fbc04086920 2021-02-10T11:35:16.560 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:16.555 7fbc03fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbc04086520 0x7fbc041a1cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:16.561 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:16.555 7fbc03fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbc04086520 0x7fbc041a1cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:36834/0 (socket says 172.21.2.102:36834) 2021-02-10T11:35:16.561 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:16.555 7fbc03fff700 1 -- 172.21.2.102:0/816861427 learned_addr learned my addr 172.21.2.102:0/816861427 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:35:16.561 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:16.555 7fbc037fe700 1 --2- 172.21.2.102:0/816861427 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbc04086920 0x7fbc04187e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:16.561 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:16.555 7fbc08ffe700 1 --2- 172.21.2.102:0/816861427 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fbc041a22d0 0x7fbc041884a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:16.561 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:16.555 7fbc03fff700 1 -- 172.21.2.102:0/816861427 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbc04086920 msgr2=0x7fbc04187e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:16.561 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:16.555 7fbc03fff700 1 --2- 172.21.2.102:0/816861427 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbc04086920 0x7fbc04187e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:16.561 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:16.555 7fbc03fff700 1 -- 172.21.2.102:0/816861427 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fbc041a22d0 msgr2=0x7fbc041884a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:16.561 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:16.555 7fbc03fff700 1 --2- 172.21.2.102:0/816861427 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fbc041a22d0 0x7fbc041884a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:16.561 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:16.555 7fbc03fff700 1 -- 172.21.2.102:0/816861427 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc04188ce0 con 0x7fbc04086520 2021-02-10T11:35:16.561 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:16.556 7fbc03fff700 1 --2- 172.21.2.102:0/816861427 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbc04086520 0x7fbc041a1cb0 secure :-1 s=READY pgs=2663 cs=0 l=1 rev1=1 rx=0x7fbbf400b5e0 tx=0x7fbbf400b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:16.561 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:16.556 7fbc00ff9700 1 -- 172.21.2.102:0/816861427 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbbf400bd70 con 0x7fbc04086520 2021-02-10T11:35:16.561 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:16.556 7fbc00ff9700 1 -- 172.21.2.102:0/816861427 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbf4002030 con 0x7fbc04086520 2021-02-10T11:35:16.561 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:16.556 7fbc0a9e1700 1 -- 172.21.2.102:0/816861427 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc04188ce0 con 0x7fbc04086520 2021-02-10T11:35:16.562 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:16.556 7fbc00ff9700 1 -- 172.21.2.102:0/816861427 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbbf4002550 con 0x7fbc04086520 2021-02-10T11:35:16.562 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:16.556 7fbc0a9e1700 1 -- 172.21.2.102:0/816861427 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc04188ce0 con 0x7fbc04086520 2021-02-10T11:35:16.562 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:16.556 7fbc00ff9700 1 -- 172.21.2.102:0/816861427 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fbbf4000c90 con 0x7fbc04086520 2021-02-10T11:35:16.562 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:16.556 7fbc00ff9700 1 --2- 172.21.2.102:0/816861427 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fbbe4018730 0x7fbbe401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:16.562 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:16.556 7fbc00ff9700 1 -- 172.21.2.102:0/816861427 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fbbf40326b0 con 0x7fbc04086520 2021-02-10T11:35:16.562 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:16.556 7fbc037fe700 1 --2- 172.21.2.102:0/816861427 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fbbe4018730 0x7fbbe401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:16.562 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:16.557 7fbc037fe700 1 --2- 172.21.2.102:0/816861427 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fbbe4018730 0x7fbbe401ac10 crc :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:16.590 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:16.588 7fbc0a9e1700 1 -- 172.21.2.102:0/816861427 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc040fe1d0 con 0x7fbc04086520 2021-02-10T11:35:16.592 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:16.590 7fbc00ff9700 1 -- 172.21.2.102:0/816861427 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fbbf401f570 con 0x7fbc04086520 2021-02-10T11:35:16.722 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:16.721 7fbc0a9e1700 1 -- 172.21.2.102:0/816861427 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7fbc040fe1d0 con 0x7fbc04086520 2021-02-10T11:35:16.725 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:16.723 7fbc00ff9700 1 -- 172.21.2.102:0/816861427 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.2 v83) v1 ==== 184+0+0 (secure 0 0 0) 0x7fbbf4000c90 con 0x7fbc04086520 2021-02-10T11:35:16.725 INFO:teuthology.orchestra.run.gibba002.stderr:updated caps for client.2 2021-02-10T11:35:16.727 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:16.725 7fbc0a9e1700 1 -- 172.21.2.102:0/816861427 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fbbe4018730 msgr2=0x7fbbe401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:16.727 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:16.725 7fbc0a9e1700 1 --2- 172.21.2.102:0/816861427 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fbbe4018730 0x7fbbe401ac10 crc :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:16.727 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:16.725 7fbc0a9e1700 1 -- 172.21.2.102:0/816861427 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbc04086520 msgr2=0x7fbc041a1cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:16.727 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:16.725 7fbc0a9e1700 1 --2- 172.21.2.102:0/816861427 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbc04086520 0x7fbc041a1cb0 secure :-1 s=READY pgs=2663 cs=0 l=1 rev1=1 rx=0x7fbbf400b5e0 tx=0x7fbbf400b980).stop 2021-02-10T11:35:16.727 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:16.725 7fbc0a9e1700 1 -- 172.21.2.102:0/816861427 shutdown_connections 2021-02-10T11:35:16.727 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:16.725 7fbc0a9e1700 1 --2- 172.21.2.102:0/816861427 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fbbe4018730 0x7fbbe401ac10 unknown :-1 s=CLOSED pgs=1175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:16.727 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:16.725 7fbc0a9e1700 1 --2- 172.21.2.102:0/816861427 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fbc041a22d0 0x7fbc041884a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:16.727 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:16.725 7fbc0a9e1700 1 --2- 172.21.2.102:0/816861427 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbc04086920 0x7fbc04187e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:16.728 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:16.725 7fbc0a9e1700 1 --2- 172.21.2.102:0/816861427 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbc04086520 0x7fbc041a1cb0 unknown :-1 s=CLOSED pgs=2663 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:16.728 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:16.725 7fbc0a9e1700 1 -- 172.21.2.102:0/816861427 >> 172.21.2.102:0/816861427 conn(0x7fbc0419f880 msgr2=0x7fbc0408f890 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:16.728 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:16.725 7fbc0a9e1700 1 -- 172.21.2.102:0/816861427 shutdown_connections 2021-02-10T11:35:16.728 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:16.725 7fbc0a9e1700 1 -- 172.21.2.102:0/816861427 wait complete. 2021-02-10T11:35:16.736 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.3 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:35:17.178 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:17.159 7f8f54b02700 1 Processor -- start 2021-02-10T11:35:17.178 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:17.159 7f8f54b02700 1 -- start start 2021-02-10T11:35:17.178 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:17.159 7f8f54b02700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f8f500d1880 0x7f8f501b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:17.178 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:17.159 7f8f54b02700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8f501b0af0 0x7f8f50108950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:17.178 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:17.159 7f8f54b02700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f500fe1d0 con 0x7f8f500d1c80 2021-02-10T11:35:17.178 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:17.159 7f8f54b02700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f50109a70 con 0x7f8f500d1880 2021-02-10T11:35:17.178 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:17.159 7f8f54b02700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f50109760 con 0x7f8f501b0af0 2021-02-10T11:35:17.178 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:17.160 7f8f4f05f700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f8f500d1c80 0x7f8f50108f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:57294/0 (socket says 172.21.2.102:57294) 2021-02-10T11:35:17.179 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:17.160 7f8f4f05f700 1 -- 172.21.2.102:0/3776915072 learned_addr learned my addr 172.21.2.102:0/3776915072 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:35:17.179 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:17.160 7f8f4e85e700 1 --2- 172.21.2.102:0/3776915072 >> v2:172.21.2.113:3300/0 conn(0x7f8f500d1880 0x7f8f501b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:17.179 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:17.160 7f8f4e05d700 1 --2- 172.21.2.102:0/3776915072 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8f501b0af0 0x7f8f50108950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:17.179 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:17.160 7f8f4d85c700 1 -- 172.21.2.102:0/3776915072 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1081353672 0 0) 0x7f8f48000f90 con 0x7f8f500d1c80 2021-02-10T11:35:17.179 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:17.160 7f8f4d85c700 1 -- 172.21.2.102:0/3776915072 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f3c002fe0 con 0x7f8f500d1c80 2021-02-10T11:35:17.179 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:17.160 7f8f4e85e700 1 -- 172.21.2.102:0/3776915072 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8f501b0af0 msgr2=0x7f8f50108950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:17.179 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:17.160 7f8f4e85e700 1 --2- 172.21.2.102:0/3776915072 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8f501b0af0 0x7f8f50108950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:17.179 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:17.160 7f8f4e85e700 1 -- 172.21.2.102:0/3776915072 >> v1:172.21.2.102:6789/0 conn(0x7f8f500d1c80 legacy=0x7f8f50108f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:17.179 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:17.160 7f8f4e85e700 1 -- 172.21.2.102:0/3776915072 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f50088070 con 0x7f8f500d1880 2021-02-10T11:35:17.179 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:17.160 7f8f4e05d700 1 --2- 172.21.2.102:0/3776915072 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8f501b0af0 0x7f8f50108950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:35:17.179 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:17.160 7f8f4e85e700 1 --2- 172.21.2.102:0/3776915072 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8f500d1880 0x7f8f501b04d0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 rx=0x7f8f40006650 tx=0x7f8f40009a80).ready entity=mon.1 client_cookie=b23d23a44eec3f12 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:17.179 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:17.161 7f8f4d85c700 1 -- 172.21.2.102:0/3776915072 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8f400009b0 con 0x7f8f500d1880 2021-02-10T11:35:17.180 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:17.161 7f8f4d85c700 1 -- 172.21.2.102:0/3776915072 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f4000c0c0 con 0x7f8f500d1880 2021-02-10T11:35:17.180 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:17.161 7f8f4d85c700 1 -- 172.21.2.102:0/3776915072 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8f400009b0 con 0x7f8f500d1880 2021-02-10T11:35:17.180 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:17.161 7f8f54b02700 1 -- 172.21.2.102:0/3776915072 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8f500d1880 msgr2=0x7f8f501b04d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:17.180 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:17.161 7f8f54b02700 1 --2- 172.21.2.102:0/3776915072 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8f500d1880 0x7f8f501b04d0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 rx=0x7f8f40006650 tx=0x7f8f40009a80).stop 2021-02-10T11:35:17.180 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:17.162 7f8f54b02700 1 -- 172.21.2.102:0/3776915072 shutdown_connections 2021-02-10T11:35:17.180 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:17.162 7f8f54b02700 1 --2- 172.21.2.102:0/3776915072 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8f501b0af0 0x7f8f50108950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:17.180 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:17.162 7f8f54b02700 1 --2- 172.21.2.102:0/3776915072 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8f500d1880 0x7f8f501b04d0 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:17.180 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:17.162 7f8f54b02700 1 -- 172.21.2.102:0/3776915072 >> 172.21.2.102:0/3776915072 conn(0x7f8f500ce430 msgr2=0x7f8f500d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:17.180 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:17.162 7f8f54b02700 1 -- 172.21.2.102:0/3776915072 shutdown_connections 2021-02-10T11:35:17.180 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:17.162 7f8f54b02700 1 -- 172.21.2.102:0/3776915072 wait complete. 2021-02-10T11:35:17.182 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:17.176 7f8f54b02700 1 Processor -- start 2021-02-10T11:35:17.182 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:17.176 7f8f54b02700 1 -- start start 2021-02-10T11:35:17.182 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:17.177 7f8f54b02700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8f5008f620 0x7f8f5008fa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:17.182 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:17.177 7f8f54b02700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8f50195140 0x7f8f50195540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:17.182 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:17.177 7f8f54b02700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8f50083a40 0x7f8f50083e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:17.183 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:17.177 7f8f54b02700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f500834c0 con 0x7f8f50083a40 2021-02-10T11:35:17.183 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:17.177 7f8f54b02700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f50083220 con 0x7f8f50195140 2021-02-10T11:35:17.183 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:17.177 7f8f54b02700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f50090040 con 0x7f8f5008f620 2021-02-10T11:35:17.183 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:17.177 7f8f4f05f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8f50083a40 0x7f8f50083e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:17.183 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:17.177 7f8f4f05f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8f50083a40 0x7f8f50083e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:36846/0 (socket says 172.21.2.102:36846) 2021-02-10T11:35:17.183 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:17.177 7f8f4f05f700 1 -- 172.21.2.102:0/3323783468 learned_addr learned my addr 172.21.2.102:0/3323783468 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:35:17.183 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:17.177 7f8f4e85e700 1 --2- 172.21.2.102:0/3323783468 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8f5008f620 0x7f8f5008fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:17.183 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:17.177 7f8f4e05d700 1 --2- 172.21.2.102:0/3323783468 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8f50195140 0x7f8f50195540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:17.183 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:17.177 7f8f4f05f700 1 -- 172.21.2.102:0/3323783468 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8f5008f620 msgr2=0x7f8f5008fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:17.183 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:17.177 7f8f4f05f700 1 --2- 172.21.2.102:0/3323783468 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8f5008f620 0x7f8f5008fa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:17.183 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:17.178 7f8f4f05f700 1 -- 172.21.2.102:0/3323783468 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8f50195140 msgr2=0x7f8f50195540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:17.183 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:17.178 7f8f4f05f700 1 --2- 172.21.2.102:0/3323783468 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8f50195140 0x7f8f50195540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:17.183 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:17.178 7f8f4f05f700 1 -- 172.21.2.102:0/3323783468 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f501a4a00 con 0x7f8f50083a40 2021-02-10T11:35:17.184 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:17.178 7f8f4e85e700 1 --2- 172.21.2.102:0/3323783468 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8f5008f620 0x7f8f5008fa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:35:17.184 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:17.178 7f8f4f05f700 1 --2- 172.21.2.102:0/3323783468 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8f50083a40 0x7f8f50083e40 secure :-1 s=READY pgs=2665 cs=0 l=1 rev1=1 rx=0x7f8f48001c10 tx=0x7f8f4800aca0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:17.184 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:17.178 7f8f477fe700 1 -- 172.21.2.102:0/3323783468 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8f48007ce0 con 0x7f8f50083a40 2021-02-10T11:35:17.184 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:17.178 7f8f477fe700 1 -- 172.21.2.102:0/3323783468 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f480070d0 con 0x7f8f50083a40 2021-02-10T11:35:17.184 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:17.178 7f8f54b02700 1 -- 172.21.2.102:0/3323783468 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f501a4a00 con 0x7f8f50083a40 2021-02-10T11:35:17.184 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:17.178 7f8f54b02700 1 -- 172.21.2.102:0/3323783468 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f50084750 con 0x7f8f50083a40 2021-02-10T11:35:17.184 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:17.178 7f8f477fe700 1 -- 172.21.2.102:0/3323783468 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8f48004930 con 0x7f8f50083a40 2021-02-10T11:35:17.184 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:17.179 7f8f477fe700 1 -- 172.21.2.102:0/3323783468 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f8f48004720 con 0x7f8f50083a40 2021-02-10T11:35:17.184 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:17.179 7f8f477fe700 1 --2- 172.21.2.102:0/3323783468 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8f340186c0 0x7f8f3401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:17.184 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:17.179 7f8f477fe700 1 -- 172.21.2.102:0/3323783468 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f8f48032bb0 con 0x7f8f50083a40 2021-02-10T11:35:17.184 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:17.179 7f8f4e85e700 1 --2- 172.21.2.102:0/3323783468 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8f340186c0 0x7f8f3401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:17.184 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:17.179 7f8f4e85e700 1 --2- 172.21.2.102:0/3323783468 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8f340186c0 0x7f8f3401abc0 crc :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:17.211 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:17.209 7f8f54b02700 1 -- 172.21.2.102:0/3323783468 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f500fe1d0 con 0x7f8f50083a40 2021-02-10T11:35:17.214 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:17.212 7f8f477fe700 1 -- 172.21.2.102:0/3323783468 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f8f48007540 con 0x7f8f50083a40 2021-02-10T11:35:17.344 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:17.342 7f8f54b02700 1 -- 172.21.2.102:0/3323783468 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.3", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f8f500fe1d0 con 0x7f8f50083a40 2021-02-10T11:35:17.347 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:17.346 7f8f477fe700 1 -- 172.21.2.102:0/3323783468 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.3", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.3 v84) v1 ==== 184+0+0 (secure 0 0 0) 0x7f8f48004720 con 0x7f8f50083a40 2021-02-10T11:35:17.351 INFO:teuthology.orchestra.run.gibba002.stderr:updated caps for client.3 2021-02-10T11:35:17.353 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:17.351 7f8f54b02700 1 -- 172.21.2.102:0/3323783468 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8f340186c0 msgr2=0x7f8f3401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:17.353 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:17.351 7f8f54b02700 1 --2- 172.21.2.102:0/3323783468 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8f340186c0 0x7f8f3401abc0 crc :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:17.353 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:17.351 7f8f54b02700 1 -- 172.21.2.102:0/3323783468 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8f50083a40 msgr2=0x7f8f50083e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:17.353 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:17.351 7f8f54b02700 1 --2- 172.21.2.102:0/3323783468 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8f50083a40 0x7f8f50083e40 secure :-1 s=READY pgs=2665 cs=0 l=1 rev1=1 rx=0x7f8f48001c10 tx=0x7f8f4800aca0).stop 2021-02-10T11:35:17.353 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:17.351 7f8f54b02700 1 -- 172.21.2.102:0/3323783468 shutdown_connections 2021-02-10T11:35:17.353 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:17.351 7f8f54b02700 1 --2- 172.21.2.102:0/3323783468 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8f50083a40 0x7f8f50083e40 unknown :-1 s=CLOSED pgs=2665 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:17.353 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:17.351 7f8f54b02700 1 --2- 172.21.2.102:0/3323783468 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8f340186c0 0x7f8f3401abc0 unknown :-1 s=CLOSED pgs=1176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:17.353 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:17.351 7f8f54b02700 1 --2- 172.21.2.102:0/3323783468 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8f50195140 0x7f8f50195540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:17.353 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:17.351 7f8f54b02700 1 --2- 172.21.2.102:0/3323783468 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8f5008f620 0x7f8f5008fa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:17.354 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:17.351 7f8f54b02700 1 -- 172.21.2.102:0/3323783468 >> 172.21.2.102:0/3323783468 conn(0x7f8f500a1070 msgr2=0x7f8f5008eac0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:17.354 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:17.351 7f8f54b02700 1 -- 172.21.2.102:0/3323783468 shutdown_connections 2021-02-10T11:35:17.354 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:17.351 7f8f54b02700 1 -- 172.21.2.102:0/3323783468 wait complete. 2021-02-10T11:35:17.361 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:35:17.791 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:17.771 7f0c79f54700 1 Processor -- start 2021-02-10T11:35:17.791 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:17.771 7f0c79f54700 1 -- start start 2021-02-10T11:35:17.791 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:17.771 7f0c79f54700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0c740d1c70 0x7f0c740f20a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:17.791 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:17.771 7f0c79f54700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c740f0b10 con 0x7f0c740d0340 2021-02-10T11:35:17.792 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:17.771 7f0c79f54700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c741b0be0 con 0x7f0c740d1870 2021-02-10T11:35:17.792 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:17.771 7f0c79f54700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c740e5ed0 con 0x7f0c740d1c70 2021-02-10T11:35:17.792 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:17.772 7f0c72ffd700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f0c740d0340 0x7f0c740ef640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:57306/0 (socket says 172.21.2.102:57306) 2021-02-10T11:35:17.792 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:17.772 7f0c72ffd700 1 -- 172.21.2.102:0/2834263344 learned_addr learned my addr 172.21.2.102:0/2834263344 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:35:17.792 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:17.772 7f0c727fc700 1 -- 172.21.2.102:0/2834263344 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1770807115 0 0) 0x7f0c64001180 con 0x7f0c740d0340 2021-02-10T11:35:17.792 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:17.772 7f0c727fc700 1 -- 172.21.2.102:0/2834263344 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c60002fe0 con 0x7f0c740d0340 2021-02-10T11:35:17.792 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:17.772 7f0c73fff700 1 --2- 172.21.2.102:0/2834263344 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0c740d1c70 0x7f0c740f20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:17.792 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:17.772 7f0c727fc700 1 -- 172.21.2.102:0/2834263344 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1996335695 0 0) 0x7f0c64001180 con 0x7f0c740d0340 2021-02-10T11:35:17.792 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:17.772 7f0c727fc700 1 -- 172.21.2.102:0/2834263344 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0c740d1c70 msgr2=0x7f0c740f20a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:17.792 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:17.772 7f0c727fc700 1 --2- 172.21.2.102:0/2834263344 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0c740d1c70 0x7f0c740f20a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:17.793 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:17.772 7f0c727fc700 1 -- 172.21.2.102:0/2834263344 >> v1:172.21.2.113:6789/0 conn(0x7f0c740d1870 legacy=0x7f0c741b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:17.793 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:17.772 7f0c727fc700 1 -- 172.21.2.102:0/2834263344 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c740efe30 con 0x7f0c740d0340 2021-02-10T11:35:17.793 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:17.772 7f0c727fc700 1 -- 172.21.2.102:0/2834263344 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7f0c64001680 con 0x7f0c740d0340 2021-02-10T11:35:17.793 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:17.772 7f0c727fc700 1 -- 172.21.2.102:0/2834263344 >> v1:172.21.2.102:6789/0 conn(0x7f0c740d0340 legacy=0x7f0c740ef640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:17.793 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:17.772 7f0c727fc700 1 --2- 172.21.2.102:0/2834263344 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0c60003af0 0x7f0c60005f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:17.793 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:17.772 7f0c727fc700 1 --2- 172.21.2.102:0/2834263344 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0c60006820 0x7f0c60008cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:17.793 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:17.773 7f0c727fc700 1 --2- 172.21.2.102:0/2834263344 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0c600095b0 0x7f0c6000baa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:17.793 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:17.773 7f0c727fc700 1 -- 172.21.2.102:0/2834263344 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c6000c2d0 con 0x7f0c60003af0 2021-02-10T11:35:17.793 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:17.773 7f0c727fc700 1 -- 172.21.2.102:0/2834263344 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c6000c4f0 con 0x7f0c60006820 2021-02-10T11:35:17.793 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:17.773 7f0c727fc700 1 -- 172.21.2.102:0/2834263344 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c6000c710 con 0x7f0c600095b0 2021-02-10T11:35:17.793 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:17.773 7f0c737fe700 1 --2- 172.21.2.102:0/2834263344 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0c60003af0 0x7f0c60005f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:17.793 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:17.773 7f0c72ffd700 1 --2- 172.21.2.102:0/2834263344 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0c60006820 0x7f0c60008cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:17.794 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:17.773 7f0c73fff700 1 --2- 172.21.2.102:0/2834263344 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0c600095b0 0x7f0c6000baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:17.794 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:17.773 7f0c737fe700 1 -- 172.21.2.102:0/2834263344 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0c600095b0 msgr2=0x7f0c6000baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:17.794 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:17.773 7f0c737fe700 1 --2- 172.21.2.102:0/2834263344 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0c600095b0 0x7f0c6000baa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:17.794 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:17.773 7f0c737fe700 1 -- 172.21.2.102:0/2834263344 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0c60006820 msgr2=0x7f0c60008cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:17.794 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:17.773 7f0c737fe700 1 --2- 172.21.2.102:0/2834263344 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0c60006820 0x7f0c60008cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:17.794 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:17.773 7f0c737fe700 1 -- 172.21.2.102:0/2834263344 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c60050980 con 0x7f0c60003af0 2021-02-10T11:35:17.794 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:17.773 7f0c72ffd700 1 --2- 172.21.2.102:0/2834263344 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0c60006820 0x7f0c60008cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:35:17.794 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:17.773 7f0c737fe700 1 --2- 172.21.2.102:0/2834263344 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0c60003af0 0x7f0c60005f80 secure :-1 s=READY pgs=2667 cs=0 l=1 rev1=1 rx=0x7f0c5c006ab0 tx=0x7f0c5c009b40).ready entity=mon.0 client_cookie=ea637415252a0fa1 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:17.794 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:17.773 7f0c727fc700 1 -- 172.21.2.102:0/2834263344 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0c5c00b2e0 con 0x7f0c60003af0 2021-02-10T11:35:17.795 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:17.773 7f0c727fc700 1 -- 172.21.2.102:0/2834263344 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c5c00b560 con 0x7f0c60003af0 2021-02-10T11:35:17.795 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:17.773 7f0c727fc700 1 -- 172.21.2.102:0/2834263344 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0c5c00ba80 con 0x7f0c60003af0 2021-02-10T11:35:17.795 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:17.773 7f0c79f54700 1 -- 172.21.2.102:0/2834263344 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0c60003af0 msgr2=0x7f0c60005f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:17.795 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:17.773 7f0c79f54700 1 --2- 172.21.2.102:0/2834263344 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0c60003af0 0x7f0c60005f80 secure :-1 s=READY pgs=2667 cs=0 l=1 rev1=1 rx=0x7f0c5c006ab0 tx=0x7f0c5c009b40).stop 2021-02-10T11:35:17.795 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:17.773 7f0c73fff700 1 -- 172.21.2.102:0/2834263344 reap_dead start 2021-02-10T11:35:17.795 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:17.774 7f0c79f54700 1 -- 172.21.2.102:0/2834263344 shutdown_connections 2021-02-10T11:35:17.795 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:17.774 7f0c79f54700 1 -- 172.21.2.102:0/2834263344 >> 172.21.2.102:0/2834263344 conn(0x7f0c740ce450 msgr2=0x7f0c740d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:17.795 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:17.774 7f0c79f54700 1 -- 172.21.2.102:0/2834263344 shutdown_connections 2021-02-10T11:35:17.795 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:17.774 7f0c79f54700 1 -- 172.21.2.102:0/2834263344 wait complete. 2021-02-10T11:35:17.795 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:17.789 7f0c79f54700 1 Processor -- start 2021-02-10T11:35:17.796 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:17.789 7f0c79f54700 1 -- start start 2021-02-10T11:35:17.797 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:17.789 7f0c79f54700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0c741a4a60 0x7f0c740c1c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:17.797 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:17.789 7f0c79f54700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0c740c2290 0x7f0c740c2690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:17.798 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:17.789 7f0c79f54700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0c740c2cb0 0x7f0c740bfd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:17.798 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:17.789 7f0c79f54700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c740c31c0 con 0x7f0c741a4a60 2021-02-10T11:35:17.798 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:17.789 7f0c79f54700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c740f05f0 con 0x7f0c740c2cb0 2021-02-10T11:35:17.798 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:17.789 7f0c79f54700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c740c0340 con 0x7f0c740c2290 2021-02-10T11:35:17.798 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:17.790 7f0c737fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0c741a4a60 0x7f0c740c1c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:17.798 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:17.790 7f0c737fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0c741a4a60 0x7f0c740c1c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:36864/0 (socket says 172.21.2.102:36864) 2021-02-10T11:35:17.798 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:17.790 7f0c737fe700 1 -- 172.21.2.102:0/1540780125 learned_addr learned my addr 172.21.2.102:0/1540780125 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:35:17.798 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:17.790 7f0c737fe700 1 -- 172.21.2.102:0/1540780125 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0c740c2290 msgr2=0x7f0c740c2690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:17.798 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:17.790 7f0c737fe700 1 --2- 172.21.2.102:0/1540780125 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0c740c2290 0x7f0c740c2690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:17.798 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:17.790 7f0c737fe700 1 -- 172.21.2.102:0/1540780125 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0c740c2cb0 msgr2=0x7f0c740bfd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:17.798 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:17.790 7f0c737fe700 1 --2- 172.21.2.102:0/1540780125 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0c740c2cb0 0x7f0c740bfd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:17.799 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:17.790 7f0c737fe700 1 -- 172.21.2.102:0/1540780125 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c740c0560 con 0x7f0c741a4a60 2021-02-10T11:35:17.799 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:17.790 7f0c737fe700 1 --2- 172.21.2.102:0/1540780125 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0c741a4a60 0x7f0c740c1c70 secure :-1 s=READY pgs=2668 cs=0 l=1 rev1=1 rx=0x7f0c5c009670 tx=0x7f0c5c00b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:17.799 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:17.790 7f0c6bfff700 1 -- 172.21.2.102:0/1540780125 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0c5c00bd80 con 0x7f0c741a4a60 2021-02-10T11:35:17.799 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:17.790 7f0c6bfff700 1 -- 172.21.2.102:0/1540780125 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c5c002080 con 0x7f0c741a4a60 2021-02-10T11:35:17.799 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:17.790 7f0c6bfff700 1 -- 172.21.2.102:0/1540780125 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0c5c0025a0 con 0x7f0c741a4a60 2021-02-10T11:35:17.799 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:17.790 7f0c79f54700 1 -- 172.21.2.102:0/1540780125 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c740c0560 con 0x7f0c741a4a60 2021-02-10T11:35:17.799 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:17.790 7f0c79f54700 1 -- 172.21.2.102:0/1540780125 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c740c0560 con 0x7f0c741a4a60 2021-02-10T11:35:17.799 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:17.791 7f0c6bfff700 1 -- 172.21.2.102:0/1540780125 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f0c5c002260 con 0x7f0c741a4a60 2021-02-10T11:35:17.799 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:17.791 7f0c6bfff700 1 --2- 172.21.2.102:0/1540780125 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0c580186d0 0x7f0c5801abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:17.799 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:17.791 7f0c6bfff700 1 -- 172.21.2.102:0/1540780125 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f0c5c032820 con 0x7f0c741a4a60 2021-02-10T11:35:17.800 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:17.791 7f0c72ffd700 1 --2- 172.21.2.102:0/1540780125 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0c580186d0 0x7f0c5801abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:17.800 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:17.791 7f0c72ffd700 1 --2- 172.21.2.102:0/1540780125 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0c580186d0 0x7f0c5801abd0 crc :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:17.823 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:17.822 7f0c79f54700 1 -- 172.21.2.102:0/1540780125 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c740f0b10 con 0x7f0c741a4a60 2021-02-10T11:35:17.826 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:17.824 7f0c6bfff700 1 -- 172.21.2.102:0/1540780125 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f0c5c000a10 con 0x7f0c741a4a60 2021-02-10T11:35:17.954 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:17.952 7f0c79f54700 1 -- 172.21.2.102:0/1540780125 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0c740f0b10 con 0x7f0c741a4a60 2021-02-10T11:35:17.954 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:17.952 7f0c6bfff700 1 -- 172.21.2.102:0/1540780125 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 124 v124) v1 ==== 95+0+2111 (secure 0 0 0) 0x7f0c5c002060 con 0x7f0c741a4a60 2021-02-10T11:35:17.955 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:35:17.955 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":124,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":11784,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.102:6835/2097086088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6834","nonce":2097086088},{"type":"v1","addr":"172.21.2.102:6835","nonce":2097086088}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":120}],"filesystems":[{"mdsmap":{"epoch":124,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:35:11.628704","modified":"2021-02-10 11:35:13.639277","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11781},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11781":{"gid":11781,"name":"b","rank":0,"incarnation":121,"state":"up:active","state_seq":3,"addr":"172.21.2.113:6833/3334075470","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6832","nonce":3334075470},{"type":"v1","addr":"172.21.2.113:6833","nonce":3334075470}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[31],"metadata_pool":30,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":15}]} 2021-02-10T11:35:17.955 INFO:teuthology.orchestra.run.gibba002.stderr:dumped fsmap epoch 124 2021-02-10T11:35:17.956 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:17.954 7f0c79f54700 1 -- 172.21.2.102:0/1540780125 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0c580186d0 msgr2=0x7f0c5801abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:17.956 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:17.954 7f0c79f54700 1 --2- 172.21.2.102:0/1540780125 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0c580186d0 0x7f0c5801abd0 crc :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:17.956 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:17.954 7f0c79f54700 1 -- 172.21.2.102:0/1540780125 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0c741a4a60 msgr2=0x7f0c740c1c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:17.956 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:17.954 7f0c79f54700 1 --2- 172.21.2.102:0/1540780125 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0c741a4a60 0x7f0c740c1c70 secure :-1 s=READY pgs=2668 cs=0 l=1 rev1=1 rx=0x7f0c5c009670 tx=0x7f0c5c00b990).stop 2021-02-10T11:35:17.956 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:17.954 7f0c79f54700 1 -- 172.21.2.102:0/1540780125 shutdown_connections 2021-02-10T11:35:17.956 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:17.954 7f0c79f54700 1 --2- 172.21.2.102:0/1540780125 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0c580186d0 0x7f0c5801abd0 unknown :-1 s=CLOSED pgs=1177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:17.957 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:17.954 7f0c79f54700 1 --2- 172.21.2.102:0/1540780125 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0c740c2cb0 0x7f0c740bfd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:17.957 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:17.954 7f0c79f54700 1 --2- 172.21.2.102:0/1540780125 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0c740c2290 0x7f0c740c2690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:17.957 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:17.954 7f0c79f54700 1 --2- 172.21.2.102:0/1540780125 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0c741a4a60 0x7f0c740c1c70 unknown :-1 s=CLOSED pgs=2668 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:17.957 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:17.954 7f0c79f54700 1 -- 172.21.2.102:0/1540780125 >> 172.21.2.102:0/1540780125 conn(0x7f0c740ebe30 msgr2=0x7f0c740bf200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:17.957 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:17.954 7f0c79f54700 1 -- 172.21.2.102:0/1540780125 shutdown_connections 2021-02-10T11:35:17.957 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:17.954 7f0c79f54700 1 -- 172.21.2.102:0/1540780125 wait complete. 2021-02-10T11:35:18.166 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 124, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:35:11.628704', 'modified': '2021-02-10 11:35:13.639277', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 11781}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_11781': {'gid': 11781, 'name': 'b', 'rank': 0, 'incarnation': 121, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.113:6833/3334075470', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.113:6832', 'nonce': 3334075470}, {'type': 'v1', 'addr': '172.21.2.113:6833', 'nonce': 3334075470}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [31], 'metadata_pool': 30, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:35:18.166 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:35:18.167 DEBUG:teuthology.orchestra.run.gibba013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:35:18.565 INFO:teuthology.orchestra.run.gibba013.stdout:{ 2021-02-10T11:35:18.565 INFO:teuthology.orchestra.run.gibba013.stdout: "cluster_fsid": "68db30ea-dbce-4aef-a6df-f01c6fea0513", 2021-02-10T11:35:18.565 INFO:teuthology.orchestra.run.gibba013.stdout: "whoami": 0, 2021-02-10T11:35:18.565 INFO:teuthology.orchestra.run.gibba013.stdout: "id": 11781, 2021-02-10T11:35:18.566 INFO:teuthology.orchestra.run.gibba013.stdout: "want_state": "up:active", 2021-02-10T11:35:18.566 INFO:teuthology.orchestra.run.gibba013.stdout: "state": "up:active", 2021-02-10T11:35:18.566 INFO:teuthology.orchestra.run.gibba013.stdout: "rank_uptime": 6.9335113100000001, 2021-02-10T11:35:18.566 INFO:teuthology.orchestra.run.gibba013.stdout: "mdsmap_epoch": 124, 2021-02-10T11:35:18.566 INFO:teuthology.orchestra.run.gibba013.stdout: "osdmap_epoch": 145, 2021-02-10T11:35:18.566 INFO:teuthology.orchestra.run.gibba013.stdout: "osdmap_epoch_barrier": 145, 2021-02-10T11:35:18.566 INFO:teuthology.orchestra.run.gibba013.stdout: "uptime": 12.585624430999999 2021-02-10T11:35:18.566 INFO:teuthology.orchestra.run.gibba013.stdout:} 2021-02-10T11:35:18.570 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "68db30ea-dbce-4aef-a6df-f01c6fea0513", "whoami": 0, "id": 11781, "want_state": "up:active", "state": "up:active", "rank_uptime": 6.9335113100000001, "mdsmap_epoch": 124, "osdmap_epoch": 145, "osdmap_epoch_barrier": 145, "uptime": 12.585624430999999 } 2021-02-10T11:35:18.570 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:35:19.000 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:18.982 7ff76f2d8700 1 Processor -- start 2021-02-10T11:35:19.000 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:18.982 7ff76f2d8700 1 -- start start 2021-02-10T11:35:19.000 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:18.982 7ff76f2d8700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7ff7680d1870 0x7ff7681b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:19.000 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:18.982 7ff76f2d8700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7ff7680f0650 0x7ff7680cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:19.001 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:18.982 7ff76f2d8700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7680f0b10 con 0x7ff7681b0a10 2021-02-10T11:35:19.001 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:18.982 7ff76f2d8700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7680e5ed0 con 0x7ff7680f0650 2021-02-10T11:35:19.001 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:18.982 7ff76f2d8700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7680d1c70 con 0x7ff7680d1870 2021-02-10T11:35:19.001 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:18.983 7ff76d8f5700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7ff7680f0650 0x7ff7680cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:19.001 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:18.983 7ff76d8f5700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7ff7680f0650 0x7ff7680cfb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:36870/0 (socket says 172.21.2.102:36870) 2021-02-10T11:35:19.001 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:18.983 7ff76d8f5700 1 -- 172.21.2.102:0/2735544671 learned_addr learned my addr 172.21.2.102:0/2735544671 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:35:19.001 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:18.983 7ff76d0f4700 1 --2- 172.21.2.102:0/2735544671 >> v2:172.21.2.113:3300/0 conn(0x7ff7680d1870 0x7ff7681b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:19.001 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:18.983 7ff76d8f5700 1 -- 172.21.2.102:0/2735544671 >> v2:172.21.2.113:3300/0 conn(0x7ff7680d1870 msgr2=0x7ff7681b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:19.001 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:18.983 7ff76d8f5700 1 --2- 172.21.2.102:0/2735544671 >> v2:172.21.2.113:3300/0 conn(0x7ff7680d1870 0x7ff7681b03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:19.001 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:18.983 7ff76d8f5700 1 -- 172.21.2.102:0/2735544671 >> v1:172.21.2.113:6789/0 conn(0x7ff7681b0a10 legacy=0x7ff7680cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:19.001 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:18.983 7ff76d8f5700 1 -- 172.21.2.102:0/2735544671 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7680d1e90 con 0x7ff7680f0650 2021-02-10T11:35:19.001 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:18.983 7ff76d8f5700 1 --2- 172.21.2.102:0/2735544671 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff7680f0650 0x7ff7680cfb20 secure :-1 s=READY pgs=2669 cs=0 l=1 rev1=1 rx=0x7ff760006f00 tx=0x7ff760009f10).ready entity=mon.0 client_cookie=e3bbeb5b7dc596af server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:19.001 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:18.983 7ff75bfff700 1 -- 172.21.2.102:0/2735544671 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff76000b3b0 con 0x7ff7680f0650 2021-02-10T11:35:19.002 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:18.983 7ff75bfff700 1 -- 172.21.2.102:0/2735544671 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff76000b630 con 0x7ff7680f0650 2021-02-10T11:35:19.002 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:18.983 7ff75bfff700 1 -- 172.21.2.102:0/2735544671 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff76000bb50 con 0x7ff7680f0650 2021-02-10T11:35:19.002 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:18.983 7ff76f2d8700 1 -- 172.21.2.102:0/2735544671 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff7680f0650 msgr2=0x7ff7680cfb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:19.002 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:18.984 7ff76f2d8700 1 --2- 172.21.2.102:0/2735544671 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff7680f0650 0x7ff7680cfb20 secure :-1 s=READY pgs=2669 cs=0 l=1 rev1=1 rx=0x7ff760006f00 tx=0x7ff760009f10).stop 2021-02-10T11:35:19.002 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:18.984 7ff76f2d8700 1 -- 172.21.2.102:0/2735544671 shutdown_connections 2021-02-10T11:35:19.002 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:18.984 7ff76f2d8700 1 --2- 172.21.2.102:0/2735544671 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff7680f0650 0x7ff7680cfb20 unknown :-1 s=CLOSED pgs=2669 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:19.002 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:18.984 7ff76f2d8700 1 --2- 172.21.2.102:0/2735544671 >> v2:172.21.2.113:3300/0 conn(0x7ff7680d1870 0x7ff7681b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:19.002 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:18.984 7ff76f2d8700 1 -- 172.21.2.102:0/2735544671 >> 172.21.2.102:0/2735544671 conn(0x7ff7680ce450 msgr2=0x7ff7680d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:19.002 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:18.984 7ff76f2d8700 1 -- 172.21.2.102:0/2735544671 shutdown_connections 2021-02-10T11:35:19.002 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:18.984 7ff76f2d8700 1 -- 172.21.2.102:0/2735544671 wait complete. 2021-02-10T11:35:19.003 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:18.999 7ff76f2d8700 1 Processor -- start 2021-02-10T11:35:19.004 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:18.999 7ff76f2d8700 1 -- start start 2021-02-10T11:35:19.004 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:18.999 7ff76f2d8700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff7680970b0 0x7ff7680ebe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:19.004 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:18.999 7ff76f2d8700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff768098810 0x7ff768098c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:19.004 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:18.999 7ff76f2d8700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff768099230 0x7ff7680b18c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:19.004 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:18.999 7ff76f2d8700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7680ec6d0 con 0x7ff768099230 2021-02-10T11:35:19.004 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:18.999 7ff76f2d8700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7680974b0 con 0x7ff768098810 2021-02-10T11:35:19.004 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:18.999 7ff76f2d8700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7680b1ee0 con 0x7ff7680970b0 2021-02-10T11:35:19.004 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:18.999 7ff76d8f5700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff768099230 0x7ff7680b18c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:19.004 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:18.999 7ff76d8f5700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff768099230 0x7ff7680b18c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:36876/0 (socket says 172.21.2.102:36876) 2021-02-10T11:35:19.004 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:18.999 7ff76d8f5700 1 -- 172.21.2.102:0/2621269901 learned_addr learned my addr 172.21.2.102:0/2621269901 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:35:19.004 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:19.000 7ff76d8f5700 1 -- 172.21.2.102:0/2621269901 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff7680970b0 msgr2=0x7ff7680ebe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:19.005 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:19.000 7ff76d8f5700 1 --2- 172.21.2.102:0/2621269901 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff7680970b0 0x7ff7680ebe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:19.005 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:19.000 7ff76d8f5700 1 -- 172.21.2.102:0/2621269901 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff768098810 msgr2=0x7ff768098c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:19.005 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:19.000 7ff76d8f5700 1 --2- 172.21.2.102:0/2621269901 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff768098810 0x7ff768098c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:19.005 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:19.000 7ff76d8f5700 1 -- 172.21.2.102:0/2621269901 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7680b2100 con 0x7ff768099230 2021-02-10T11:35:19.005 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:19.000 7ff76d8f5700 1 --2- 172.21.2.102:0/2621269901 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff768099230 0x7ff7680b18c0 secure :-1 s=READY pgs=2670 cs=0 l=1 rev1=1 rx=0x7ff760009390 tx=0x7ff76000bd40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:19.005 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:19.000 7ff759ffb700 1 -- 172.21.2.102:0/2621269901 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff760003030 con 0x7ff768099230 2021-02-10T11:35:19.005 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:19.000 7ff759ffb700 1 -- 172.21.2.102:0/2621269901 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff760006c80 con 0x7ff768099230 2021-02-10T11:35:19.005 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:19.000 7ff759ffb700 1 -- 172.21.2.102:0/2621269901 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff760001480 con 0x7ff768099230 2021-02-10T11:35:19.005 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:19.000 7ff76f2d8700 1 -- 172.21.2.102:0/2621269901 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7680b2100 con 0x7ff768099230 2021-02-10T11:35:19.005 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:19.000 7ff76f2d8700 1 -- 172.21.2.102:0/2621269901 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7680b2690 con 0x7ff768099230 2021-02-10T11:35:19.005 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:19.001 7ff759ffb700 1 -- 172.21.2.102:0/2621269901 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ff760001270 con 0x7ff768099230 2021-02-10T11:35:19.005 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:19.001 7ff759ffb700 1 --2- 172.21.2.102:0/2621269901 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff7480186c0 0x7ff74801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:19.005 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:19.001 7ff759ffb700 1 -- 172.21.2.102:0/2621269901 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 5573+0+0 (secure 0 0 0) 0x7ff7600328b0 con 0x7ff768099230 2021-02-10T11:35:19.006 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:19.001 7ff76d0f4700 1 --2- 172.21.2.102:0/2621269901 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff7480186c0 0x7ff74801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:19.006 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:19.001 7ff76d0f4700 1 --2- 172.21.2.102:0/2621269901 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff7480186c0 0x7ff74801abc0 crc :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:19.034 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:19.032 7ff76f2d8700 1 -- 172.21.2.102:0/2621269901 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7680ebc00 con 0x7ff768099230 2021-02-10T11:35:19.037 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:19.035 7ff759ffb700 1 -- 172.21.2.102:0/2621269901 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ff760009450 con 0x7ff768099230 2021-02-10T11:35:19.161 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:19.160 7ff76f2d8700 1 -- 172.21.2.102:0/2621269901 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff7680f0b10 con 0x7ff768099230 2021-02-10T11:35:19.162 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:19.160 7ff759ffb700 1 -- 172.21.2.102:0/2621269901 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v145) v1 ==== 74+0+14007 (secure 0 0 0) 0x7ff760001270 con 0x7ff768099230 2021-02-10T11:35:19.163 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:35:19.163 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":145,"fsid":"68db30ea-dbce-4aef-a6df-f01c6fea0513","created":"2021-02-10 11:03:06.331557","modified":"2021-02-10 11:35:12.631037","last_up_change":"2021-02-10 11:03:32.690562","last_in_change":"2021-02-10 11:03:32.690562","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":31,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:03:37.322519","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":30,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:35:09.594644","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"144","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":31,"pool_name":"cephfs_data","create_time":"2021-02-10 11:35:10.619592","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"145","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"be691b4d-a039-408a-af80-9d53f02b6f1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6818","nonce":36088},{"type":"v1","addr":"172.21.2.102:6820","nonce":36088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6822","nonce":36088},{"type":"v1","addr":"172.21.2.102:6824","nonce":36088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6830","nonce":36088},{"type":"v1","addr":"172.21.2.102:6832","nonce":36088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6826","nonce":36088},{"type":"v1","addr":"172.21.2.102:6828","nonce":36088}]},"public_addr":"172.21.2.102:6820/36088","cluster_addr":"172.21.2.102:6824/36088","heartbeat_back_addr":"172.21.2.102:6832/36088","heartbeat_front_addr":"172.21.2.102:6828/36088","state":["exists","up"]},{"osd":1,"uuid":"64e3ccc8-9c69-4d7c-865e-af8e83822635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6819","nonce":36087},{"type":"v1","addr":"172.21.2.102:6821","nonce":36087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6823","nonce":36087},{"type":"v1","addr":"172.21.2.102:6825","nonce":36087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6831","nonce":36087},{"type":"v1","addr":"172.21.2.102:6833","nonce":36087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6827","nonce":36087},{"type":"v1","addr":"172.21.2.102:6829","nonce":36087}]},"public_addr":"172.21.2.102:6821/36087","cluster_addr":"172.21.2.102:6825/36087","heartbeat_back_addr":"172.21.2.102:6833/36087","heartbeat_front_addr":"172.21.2.102:6829/36087","state":["exists","up"]},{"osd":2,"uuid":"153700af-747f-481f-bdf4-d6072d3a842f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6810","nonce":36089},{"type":"v1","addr":"172.21.2.102:6811","nonce":36089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6812","nonce":36089},{"type":"v1","addr":"172.21.2.102:6813","nonce":36089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6816","nonce":36089},{"type":"v1","addr":"172.21.2.102:6817","nonce":36089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6814","nonce":36089},{"type":"v1","addr":"172.21.2.102:6815","nonce":36089}]},"public_addr":"172.21.2.102:6811/36089","cluster_addr":"172.21.2.102:6813/36089","heartbeat_back_addr":"172.21.2.102:6817/36089","heartbeat_front_addr":"172.21.2.102:6815/36089","state":["exists","up"]},{"osd":3,"uuid":"cbc678f4-83a9-45e6-8830-2f7c232b1ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6802","nonce":36090},{"type":"v1","addr":"172.21.2.102:6803","nonce":36090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6804","nonce":36090},{"type":"v1","addr":"172.21.2.102:6805","nonce":36090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6808","nonce":36090},{"type":"v1","addr":"172.21.2.102:6809","nonce":36090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6806","nonce":36090},{"type":"v1","addr":"172.21.2.102:6807","nonce":36090}]},"public_addr":"172.21.2.102:6803/36090","cluster_addr":"172.21.2.102:6805/36090","heartbeat_back_addr":"172.21.2.102:6809/36090","heartbeat_front_addr":"172.21.2.102:6807/36090","state":["exists","up"]},{"osd":4,"uuid":"c01e4008-b258-4f10-afd2-9e7f37214351","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6808","nonce":36368},{"type":"v1","addr":"172.21.2.113:6811","nonce":36368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6814","nonce":36368},{"type":"v1","addr":"172.21.2.113:6818","nonce":36368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6826","nonce":36368},{"type":"v1","addr":"172.21.2.113:6828","nonce":36368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6820","nonce":36368},{"type":"v1","addr":"172.21.2.113:6824","nonce":36368}]},"public_addr":"172.21.2.113:6811/36368","cluster_addr":"172.21.2.113:6818/36368","heartbeat_back_addr":"172.21.2.113:6828/36368","heartbeat_front_addr":"172.21.2.113:6824/36368","state":["exists","up"]},{"osd":5,"uuid":"551f6007-daf8-4a98-8ee0-c2b16bcccb5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6802","nonce":36366},{"type":"v1","addr":"172.21.2.113:6805","nonce":36366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6807","nonce":36366},{"type":"v1","addr":"172.21.2.113:6810","nonce":36366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6819","nonce":36366},{"type":"v1","addr":"172.21.2.113:6822","nonce":36366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6812","nonce":36366},{"type":"v1","addr":"172.21.2.113:6815","nonce":36366}]},"public_addr":"172.21.2.113:6805/36366","cluster_addr":"172.21.2.113:6810/36366","heartbeat_back_addr":"172.21.2.113:6822/36366","heartbeat_front_addr":"172.21.2.113:6815/36366","state":["exists","up"]},{"osd":6,"uuid":"bf67c040-0e50-48be-b102-c3943bc9ebef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6800","nonce":36367},{"type":"v1","addr":"172.21.2.113:6801","nonce":36367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6803","nonce":36367},{"type":"v1","addr":"172.21.2.113:6804","nonce":36367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6813","nonce":36367},{"type":"v1","addr":"172.21.2.113:6816","nonce":36367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6806","nonce":36367},{"type":"v1","addr":"172.21.2.113:6809","nonce":36367}]},"public_addr":"172.21.2.113:6801/36367","cluster_addr":"172.21.2.113:6804/36367","heartbeat_back_addr":"172.21.2.113:6816/36367","heartbeat_front_addr":"172.21.2.113:6809/36367","state":["exists","up"]},{"osd":7,"uuid":"f5461f80-018d-428f-be89-2eb2bcbe7bb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6817","nonce":36365},{"type":"v1","addr":"172.21.2.113:6821","nonce":36365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6823","nonce":36365},{"type":"v1","addr":"172.21.2.113:6825","nonce":36365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6830","nonce":36365},{"type":"v1","addr":"172.21.2.113:6831","nonce":36365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6827","nonce":36365},{"type":"v1","addr":"172.21.2.113:6829","nonce":36365}]},"public_addr":"172.21.2.113:6821/36365","cluster_addr":"172.21.2.113:6825/36365","heartbeat_back_addr":"172.21.2.113:6831/36365","heartbeat_front_addr":"172.21.2.113:6829/36365","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:35:19.164 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:19.162 7ff76f2d8700 1 -- 172.21.2.102:0/2621269901 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff7480186c0 msgr2=0x7ff74801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:19.164 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:19.162 7ff76f2d8700 1 --2- 172.21.2.102:0/2621269901 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff7480186c0 0x7ff74801abc0 crc :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:19.164 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:19.162 7ff76f2d8700 1 -- 172.21.2.102:0/2621269901 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff768099230 msgr2=0x7ff7680b18c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:19.164 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:19.162 7ff76f2d8700 1 --2- 172.21.2.102:0/2621269901 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff768099230 0x7ff7680b18c0 secure :-1 s=READY pgs=2670 cs=0 l=1 rev1=1 rx=0x7ff760009390 tx=0x7ff76000bd40).stop 2021-02-10T11:35:19.164 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:19.162 7ff76f2d8700 1 -- 172.21.2.102:0/2621269901 shutdown_connections 2021-02-10T11:35:19.165 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:19.162 7ff76f2d8700 1 --2- 172.21.2.102:0/2621269901 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff768099230 0x7ff7680b18c0 unknown :-1 s=CLOSED pgs=2670 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:19.165 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:19.162 7ff76f2d8700 1 --2- 172.21.2.102:0/2621269901 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff7480186c0 0x7ff74801abc0 unknown :-1 s=CLOSED pgs=1178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:19.165 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:19.162 7ff76f2d8700 1 --2- 172.21.2.102:0/2621269901 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff768098810 0x7ff768098c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:19.165 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:19.162 7ff76f2d8700 1 --2- 172.21.2.102:0/2621269901 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff7680970b0 0x7ff7680ebe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:19.165 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:19.162 7ff76f2d8700 1 -- 172.21.2.102:0/2621269901 >> 172.21.2.102:0/2621269901 conn(0x7ff76819a730 msgr2=0x7ff76819e480 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:19.165 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:19.162 7ff76f2d8700 1 -- 172.21.2.102:0/2621269901 shutdown_connections 2021-02-10T11:35:19.165 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:19.162 7ff76f2d8700 1 -- 172.21.2.102:0/2621269901 wait complete. 2021-02-10T11:35:19.173 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:03:37.322519', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 30, 'pool_name': 'cephfs_metadata', 'create_time': '2021-02-10 11:35:09.594644', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '144', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 31, 'pool_name': 'cephfs_data', 'create_time': '2021-02-10 11:35:10.619592', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '145', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-02-10T11:35:19.173 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:35:19.604 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:19.586 7f025fa68700 1 Processor -- start 2021-02-10T11:35:19.605 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:19.586 7f025fa68700 1 -- start start 2021-02-10T11:35:19.605 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:19.587 7f025fa68700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f02580d1870 0x7f02581b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:19.605 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:19.587 7f025fa68700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f02581b0ae0 0x7f0258108940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:19.605 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:19.587 7f025fa68700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02580fe1c0 con 0x7f02580d1c70 2021-02-10T11:35:19.605 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:19.587 7f025fa68700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0258109a60 con 0x7f02581b0ae0 2021-02-10T11:35:19.605 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:19.587 7f025fa68700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0258109750 con 0x7f02580d1870 2021-02-10T11:35:19.605 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:19.587 7f025d083700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f02581b0ae0 0x7f0258108940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:19.605 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:19.587 7f025d083700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f02581b0ae0 0x7f0258108940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:36882/0 (socket says 172.21.2.102:36882) 2021-02-10T11:35:19.605 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:19.587 7f025d083700 1 -- 172.21.2.102:0/2759066736 learned_addr learned my addr 172.21.2.102:0/2759066736 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:35:19.605 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:19.587 7f025d884700 1 --2- 172.21.2.102:0/2759066736 >> v2:172.21.2.113:3300/0 conn(0x7f02580d1870 0x7f02581b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:19.605 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:19.587 7f025d083700 1 -- 172.21.2.102:0/2759066736 >> v2:172.21.2.113:3300/0 conn(0x7f02580d1870 msgr2=0x7f02581b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:19.605 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:19.587 7f025d083700 1 --2- 172.21.2.102:0/2759066736 >> v2:172.21.2.113:3300/0 conn(0x7f02580d1870 0x7f02581b04c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:19.606 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:19.587 7f025d083700 1 -- 172.21.2.102:0/2759066736 >> v1:172.21.2.113:6789/0 conn(0x7f02580d1c70 legacy=0x7f0258108f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:19.606 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:19.587 7f025d083700 1 -- 172.21.2.102:0/2759066736 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0258088070 con 0x7f02581b0ae0 2021-02-10T11:35:19.606 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:19.587 7f025d083700 1 --2- 172.21.2.102:0/2759066736 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f02581b0ae0 0x7f0258108940 secure :-1 s=READY pgs=2671 cs=0 l=1 rev1=1 rx=0x7f024c006700 tx=0x7f024c009b80).ready entity=mon.0 client_cookie=4cd23c9249c58d99 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:19.606 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:19.588 7f025c882700 1 -- 172.21.2.102:0/2759066736 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f024c00c0c0 con 0x7f02581b0ae0 2021-02-10T11:35:19.606 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:19.588 7f025c882700 1 -- 172.21.2.102:0/2759066736 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f024c00c440 con 0x7f02581b0ae0 2021-02-10T11:35:19.606 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:19.588 7f025c882700 1 -- 172.21.2.102:0/2759066736 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f024c00c960 con 0x7f02581b0ae0 2021-02-10T11:35:19.606 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:19.588 7f025fa68700 1 -- 172.21.2.102:0/2759066736 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f02581b0ae0 msgr2=0x7f0258108940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:19.606 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:19.588 7f025fa68700 1 --2- 172.21.2.102:0/2759066736 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f02581b0ae0 0x7f0258108940 secure :-1 s=READY pgs=2671 cs=0 l=1 rev1=1 rx=0x7f024c006700 tx=0x7f024c009b80).stop 2021-02-10T11:35:19.606 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:19.588 7f025fa68700 1 -- 172.21.2.102:0/2759066736 shutdown_connections 2021-02-10T11:35:19.606 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:19.588 7f025fa68700 1 --2- 172.21.2.102:0/2759066736 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f02581b0ae0 0x7f0258108940 unknown :-1 s=CLOSED pgs=2671 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:19.606 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:19.588 7f025fa68700 1 --2- 172.21.2.102:0/2759066736 >> v2:172.21.2.113:3300/0 conn(0x7f02580d1870 0x7f02581b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:19.606 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:19.588 7f025fa68700 1 -- 172.21.2.102:0/2759066736 >> 172.21.2.102:0/2759066736 conn(0x7f02580ce420 msgr2=0x7f02580d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:19.606 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:19.588 7f025fa68700 1 -- 172.21.2.102:0/2759066736 shutdown_connections 2021-02-10T11:35:19.607 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:19.588 7f025fa68700 1 -- 172.21.2.102:0/2759066736 wait complete. 2021-02-10T11:35:19.608 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:19.603 7f025fa68700 1 Processor -- start 2021-02-10T11:35:19.608 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:19.603 7f025fa68700 1 -- start start 2021-02-10T11:35:19.608 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:19.603 7f025fa68700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f025809bcc0 0x7f025809c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:19.608 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:19.603 7f025fa68700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f02580a0820 0x7f02580a0c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:19.608 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:19.603 7f025fa68700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f02580a1240 0x7f02580858d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:19.608 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:19.603 7f025fa68700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02580a66b0 con 0x7f02580a0820 2021-02-10T11:35:19.608 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:19.603 7f025fa68700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02581a49b0 con 0x7f02580a1240 2021-02-10T11:35:19.608 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:19.603 7f025fa68700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02581a4b90 con 0x7f025809bcc0 2021-02-10T11:35:19.609 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:19.604 7f025d083700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f02580a0820 0x7f02580a0c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:19.609 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:19.604 7f025d083700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f02580a0820 0x7f02580a0c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:36890/0 (socket says 172.21.2.102:36890) 2021-02-10T11:35:19.609 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:19.604 7f025d083700 1 -- 172.21.2.102:0/1077323912 learned_addr learned my addr 172.21.2.102:0/1077323912 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:35:19.609 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:19.604 7f025d083700 1 -- 172.21.2.102:0/1077323912 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f025809bcc0 msgr2=0x7f025809c0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:19.609 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:19.604 7f025d083700 1 --2- 172.21.2.102:0/1077323912 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f025809bcc0 0x7f025809c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:19.609 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:19.604 7f025d083700 1 -- 172.21.2.102:0/1077323912 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f02580a1240 msgr2=0x7f02580858d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:19.609 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:19.604 7f025d083700 1 --2- 172.21.2.102:0/1077323912 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f02580a1240 0x7f02580858d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:19.609 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:19.604 7f025d083700 1 -- 172.21.2.102:0/1077323912 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02580a1640 con 0x7f02580a0820 2021-02-10T11:35:19.609 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:19.604 7f025d083700 1 --2- 172.21.2.102:0/1077323912 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f02580a0820 0x7f02580a0c20 secure :-1 s=READY pgs=2672 cs=0 l=1 rev1=1 rx=0x7f024c00b5e0 tx=0x7f024c00b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:19.609 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:19.604 7f024a7fc700 1 -- 172.21.2.102:0/1077323912 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f024c00bd70 con 0x7f02580a0820 2021-02-10T11:35:19.609 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:19.604 7f024a7fc700 1 -- 172.21.2.102:0/1077323912 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f024c002030 con 0x7f02580a0820 2021-02-10T11:35:19.609 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:19.604 7f024a7fc700 1 -- 172.21.2.102:0/1077323912 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f024c002550 con 0x7f02580a0820 2021-02-10T11:35:19.609 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:19.604 7f025fa68700 1 -- 172.21.2.102:0/1077323912 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02580a1640 con 0x7f02580a0820 2021-02-10T11:35:19.610 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:19.604 7f025fa68700 1 -- 172.21.2.102:0/1077323912 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02580a1640 con 0x7f02580a0820 2021-02-10T11:35:19.610 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:19.605 7f024a7fc700 1 -- 172.21.2.102:0/1077323912 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f024c000c90 con 0x7f02580a0820 2021-02-10T11:35:19.610 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:19.605 7f024a7fc700 1 --2- 172.21.2.102:0/1077323912 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0240018700 0x7f024001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:19.610 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:19.605 7f024a7fc700 1 -- 172.21.2.102:0/1077323912 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f024c0326e0 con 0x7f02580a0820 2021-02-10T11:35:19.610 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:19.605 7f025d884700 1 --2- 172.21.2.102:0/1077323912 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0240018700 0x7f024001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:19.610 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:19.605 7f025d884700 1 --2- 172.21.2.102:0/1077323912 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0240018700 0x7f024001abc0 crc :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:19.638 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:19.637 7f025fa68700 1 -- 172.21.2.102:0/1077323912 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02580fe1c0 con 0x7f02580a0820 2021-02-10T11:35:19.641 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:19.639 7f024a7fc700 1 -- 172.21.2.102:0/1077323912 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f024c01f5a0 con 0x7f02580a0820 2021-02-10T11:35:19.770 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:19.768 7f025fa68700 1 -- 172.21.2.102:0/1077323912 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f0258086d30 con 0x7f02580a0820 2021-02-10T11:35:19.771 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:19.769 7f024a7fc700 1 -- 172.21.2.102:0/1077323912 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v145) v1 ==== 92+0+10 (secure 0 0 0) 0x7f024c00bd70 con 0x7f02580a0820 2021-02-10T11:35:19.771 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:35:19.773 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:19.771 7f025fa68700 1 -- 172.21.2.102:0/1077323912 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0240018700 msgr2=0x7f024001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:19.773 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:19.771 7f025fa68700 1 --2- 172.21.2.102:0/1077323912 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0240018700 0x7f024001abc0 crc :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:19.773 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:19.771 7f025fa68700 1 -- 172.21.2.102:0/1077323912 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f02580a0820 msgr2=0x7f02580a0c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:19.773 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:19.771 7f025fa68700 1 --2- 172.21.2.102:0/1077323912 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f02580a0820 0x7f02580a0c20 secure :-1 s=READY pgs=2672 cs=0 l=1 rev1=1 rx=0x7f024c00b5e0 tx=0x7f024c00b980).stop 2021-02-10T11:35:19.773 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:19.771 7f025fa68700 1 -- 172.21.2.102:0/1077323912 shutdown_connections 2021-02-10T11:35:19.773 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:19.771 7f025fa68700 1 --2- 172.21.2.102:0/1077323912 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0240018700 0x7f024001abc0 unknown :-1 s=CLOSED pgs=1179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:19.774 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:19.771 7f025fa68700 1 --2- 172.21.2.102:0/1077323912 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f02580a1240 0x7f02580858d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:19.774 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:19.771 7f025fa68700 1 --2- 172.21.2.102:0/1077323912 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f02580a0820 0x7f02580a0c20 unknown :-1 s=CLOSED pgs=2672 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:19.774 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:19.771 7f025fa68700 1 --2- 172.21.2.102:0/1077323912 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f025809bcc0 0x7f025809c0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:19.774 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:19.771 7f025fa68700 1 -- 172.21.2.102:0/1077323912 >> 172.21.2.102:0/1077323912 conn(0x7f02580bef00 msgr2=0x7f025809d1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:19.774 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:19.771 7f025fa68700 1 -- 172.21.2.102:0/1077323912 shutdown_connections 2021-02-10T11:35:19.774 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:19.771 7f025fa68700 1 -- 172.21.2.102:0/1077323912 wait complete. 2021-02-10T11:35:19.784 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-02-10T11:35:20.217 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:20.197 7ff5fcc29700 1 Processor -- start 2021-02-10T11:35:20.217 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:20.197 7ff5fcc29700 1 -- start start 2021-02-10T11:35:20.217 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:20.198 7ff5fcc29700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7ff5f80d1870 0x7ff5f81b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:20.217 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:20.198 7ff5fcc29700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5f80fe1c0 con 0x7ff5f81b0ae0 2021-02-10T11:35:20.217 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:20.198 7ff5fcc29700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5f8109a60 con 0x7ff5f80cff70 2021-02-10T11:35:20.218 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:20.198 7ff5fcc29700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5f8088070 con 0x7ff5f80d1870 2021-02-10T11:35:20.218 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:20.198 7ff5f605d700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7ff5f81b0ae0 0x7ff5f8108940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:57352/0 (socket says 172.21.2.102:57352) 2021-02-10T11:35:20.218 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:20.198 7ff5f605d700 1 -- 172.21.2.102:0/1543014750 learned_addr learned my addr 172.21.2.102:0/1543014750 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:35:20.218 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:20.199 7ff5f585c700 1 -- 172.21.2.102:0/1543014750 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3097073661 0 0) 0x7ff5e80010b0 con 0x7ff5f81b0ae0 2021-02-10T11:35:20.218 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:20.199 7ff5f585c700 1 -- 172.21.2.102:0/1543014750 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5e4002fe0 con 0x7ff5f81b0ae0 2021-02-10T11:35:20.218 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:20.199 7ff5f685e700 1 --2- 172.21.2.102:0/1543014750 >> v2:172.21.2.113:3300/0 conn(0x7ff5f80d1870 0x7ff5f81b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:20.218 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:20.199 7ff5f585c700 1 -- 172.21.2.102:0/1543014750 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3470611909 0 0) 0x7ff5e80010b0 con 0x7ff5f81b0ae0 2021-02-10T11:35:20.218 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:20.199 7ff5f585c700 1 -- 172.21.2.102:0/1543014750 >> v2:172.21.2.113:3300/0 conn(0x7ff5f80d1870 msgr2=0x7ff5f81b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:20.218 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:20.199 7ff5f585c700 1 --2- 172.21.2.102:0/1543014750 >> v2:172.21.2.113:3300/0 conn(0x7ff5f80d1870 0x7ff5f81b04c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:20.218 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:20.199 7ff5f585c700 1 -- 172.21.2.102:0/1543014750 >> v1:172.21.2.113:6789/0 conn(0x7ff5f80cff70 legacy=0x7ff5f8109130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:20.218 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:20.199 7ff5f585c700 1 -- 172.21.2.102:0/1543014750 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5f8088260 con 0x7ff5f81b0ae0 2021-02-10T11:35:20.218 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:20.199 7ff5f585c700 1 -- 172.21.2.102:0/1543014750 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7ff5e80015b0 con 0x7ff5f81b0ae0 2021-02-10T11:35:20.218 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:20.199 7ff5f585c700 1 -- 172.21.2.102:0/1543014750 >> v1:172.21.2.102:6789/0 conn(0x7ff5f81b0ae0 legacy=0x7ff5f8108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:20.219 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:20.199 7ff5f585c700 1 --2- 172.21.2.102:0/1543014750 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff5e4003af0 0x7ff5e4005f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:20.219 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:20.199 7ff5f585c700 1 --2- 172.21.2.102:0/1543014750 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff5e4006780 0x7ff5e4008c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:20.219 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:20.199 7ff5f585c700 1 --2- 172.21.2.102:0/1543014750 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff5e4009580 0x7ff5e400baa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:20.219 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:20.199 7ff5f585c700 1 -- 172.21.2.102:0/1543014750 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5e400c2f0 con 0x7ff5e4006780 2021-02-10T11:35:20.219 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:20.199 7ff5f585c700 1 -- 172.21.2.102:0/1543014750 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5e400c510 con 0x7ff5e4003af0 2021-02-10T11:35:20.219 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:20.199 7ff5f585c700 1 -- 172.21.2.102:0/1543014750 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5e400c730 con 0x7ff5e4009580 2021-02-10T11:35:20.219 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:20.199 7ff5f605d700 1 --2- 172.21.2.102:0/1543014750 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff5e4006780 0x7ff5e4008c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:20.219 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:20.199 7ff5f685e700 1 --2- 172.21.2.102:0/1543014750 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff5e4003af0 0x7ff5e4005f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:20.219 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:20.199 7ff5f705f700 1 --2- 172.21.2.102:0/1543014750 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff5e4009580 0x7ff5e400baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:20.219 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:20.200 7ff5f605d700 1 -- 172.21.2.102:0/1543014750 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff5e4009580 msgr2=0x7ff5e400baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:20.219 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:20.200 7ff5f605d700 1 --2- 172.21.2.102:0/1543014750 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff5e4009580 0x7ff5e400baa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:20.219 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:20.200 7ff5f605d700 1 -- 172.21.2.102:0/1543014750 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff5e4003af0 msgr2=0x7ff5e4005f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:20.219 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:20.200 7ff5f605d700 1 --2- 172.21.2.102:0/1543014750 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff5e4003af0 0x7ff5e4005f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:20.220 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:20.200 7ff5f605d700 1 -- 172.21.2.102:0/1543014750 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5e400c9d0 con 0x7ff5e4006780 2021-02-10T11:35:20.220 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:20.200 7ff5f705f700 1 --2- 172.21.2.102:0/1543014750 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff5e4009580 0x7ff5e400baa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:35:20.220 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:20.200 7ff5f685e700 1 --2- 172.21.2.102:0/1543014750 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff5e4003af0 0x7ff5e4005f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:35:20.220 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:20.200 7ff5f705f700 1 -- 172.21.2.102:0/1543014750 reap_dead start 2021-02-10T11:35:20.220 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:20.200 7ff5f605d700 1 --2- 172.21.2.102:0/1543014750 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff5e4006780 0x7ff5e4008c80 secure :-1 s=READY pgs=2674 cs=0 l=1 rev1=1 rx=0x7ff5e8007560 tx=0x7ff5e8007d10).ready entity=mon.0 client_cookie=330e1dc0fb4f1475 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:20.220 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:20.200 7ff5f585c700 1 -- 172.21.2.102:0/1543014750 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff5e80070c0 con 0x7ff5e4006780 2021-02-10T11:35:20.220 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:20.200 7ff5f585c700 1 -- 172.21.2.102:0/1543014750 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5e8001400 con 0x7ff5e4006780 2021-02-10T11:35:20.220 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:20.200 7ff5f585c700 1 -- 172.21.2.102:0/1543014750 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff5e80070c0 con 0x7ff5e4006780 2021-02-10T11:35:20.220 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:20.200 7ff5fcc29700 1 -- 172.21.2.102:0/1543014750 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff5e4006780 msgr2=0x7ff5e4008c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:20.220 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:20.200 7ff5fcc29700 1 --2- 172.21.2.102:0/1543014750 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff5e4006780 0x7ff5e4008c80 secure :-1 s=READY pgs=2674 cs=0 l=1 rev1=1 rx=0x7ff5e8007560 tx=0x7ff5e8007d10).stop 2021-02-10T11:35:20.220 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:20.201 7ff5fcc29700 1 -- 172.21.2.102:0/1543014750 shutdown_connections 2021-02-10T11:35:20.220 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:20.201 7ff5fcc29700 1 --2- 172.21.2.102:0/1543014750 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff5e4006780 0x7ff5e4008c80 unknown :-1 s=CLOSED pgs=2674 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:20.220 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:20.201 7ff5fcc29700 1 -- 172.21.2.102:0/1543014750 >> 172.21.2.102:0/1543014750 conn(0x7ff5f80ce420 msgr2=0x7ff5f80d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:20.221 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:20.201 7ff5fcc29700 1 -- 172.21.2.102:0/1543014750 shutdown_connections 2021-02-10T11:35:20.221 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:20.201 7ff5fcc29700 1 -- 172.21.2.102:0/1543014750 wait complete. 2021-02-10T11:35:20.221 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:20.215 7ff5fcc29700 1 Processor -- start 2021-02-10T11:35:20.221 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:20.215 7ff5fcc29700 1 -- start start 2021-02-10T11:35:20.222 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:20.216 7ff5fcc29700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff5f8085bc0 0x7ff5f80863e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:20.222 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:20.216 7ff5fcc29700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff5f80954c0 0x7ff5f8096300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:20.222 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:20.216 7ff5fcc29700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff5f80958c0 0x7ff5f8096920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:20.223 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:20.216 7ff5fcc29700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5f8095d80 con 0x7ff5f8085bc0 2021-02-10T11:35:20.223 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:20.216 7ff5fcc29700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5f80bfa60 con 0x7ff5f80954c0 2021-02-10T11:35:20.223 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:20.216 7ff5fcc29700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5f8085fc0 con 0x7ff5f80958c0 2021-02-10T11:35:20.223 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:20.216 7ff5f685e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff5f8085bc0 0x7ff5f80863e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:20.223 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:20.216 7ff5f685e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff5f8085bc0 0x7ff5f80863e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:36910/0 (socket says 172.21.2.102:36910) 2021-02-10T11:35:20.223 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:20.216 7ff5f685e700 1 -- 172.21.2.102:0/4276410498 learned_addr learned my addr 172.21.2.102:0/4276410498 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:35:20.223 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:20.216 7ff5f685e700 1 -- 172.21.2.102:0/4276410498 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff5f80958c0 msgr2=0x7ff5f8096920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:20.223 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:20.216 7ff5f685e700 1 --2- 172.21.2.102:0/4276410498 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff5f80958c0 0x7ff5f8096920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:20.223 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:20.216 7ff5f685e700 1 -- 172.21.2.102:0/4276410498 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff5f80954c0 msgr2=0x7ff5f8096300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:20.223 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:20.216 7ff5f685e700 1 --2- 172.21.2.102:0/4276410498 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff5f80954c0 0x7ff5f8096300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:20.223 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:20.216 7ff5f685e700 1 -- 172.21.2.102:0/4276410498 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5f8086c10 con 0x7ff5f8085bc0 2021-02-10T11:35:20.223 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:20.216 7ff5f685e700 1 --2- 172.21.2.102:0/4276410498 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff5f8085bc0 0x7ff5f80863e0 secure :-1 s=READY pgs=2675 cs=0 l=1 rev1=1 rx=0x7ff5e000b530 tx=0x7ff5e000b8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:20.223 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:20.217 7ff5ef7fe700 1 -- 172.21.2.102:0/4276410498 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff5e000bcc0 con 0x7ff5f8085bc0 2021-02-10T11:35:20.224 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:20.217 7ff5ef7fe700 1 -- 172.21.2.102:0/4276410498 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5e0002030 con 0x7ff5f8085bc0 2021-02-10T11:35:20.224 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:20.217 7ff5ef7fe700 1 -- 172.21.2.102:0/4276410498 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff5e0002490 con 0x7ff5f8085bc0 2021-02-10T11:35:20.224 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:20.217 7ff5fcc29700 1 -- 172.21.2.102:0/4276410498 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5f8086c10 con 0x7ff5f8085bc0 2021-02-10T11:35:20.224 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:20.217 7ff5fcc29700 1 -- 172.21.2.102:0/4276410498 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5f8097230 con 0x7ff5f8085bc0 2021-02-10T11:35:20.224 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:20.217 7ff5ef7fe700 1 -- 172.21.2.102:0/4276410498 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ff5e0000c90 con 0x7ff5f8085bc0 2021-02-10T11:35:20.224 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:20.217 7ff5ef7fe700 1 --2- 172.21.2.102:0/4276410498 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff5dc0186c0 0x7ff5dc01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:20.224 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:20.217 7ff5ef7fe700 1 -- 172.21.2.102:0/4276410498 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 5573+0+0 (secure 0 0 0) 0x7ff5e00325f0 con 0x7ff5f8085bc0 2021-02-10T11:35:20.225 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:20.218 7ff5f605d700 1 --2- 172.21.2.102:0/4276410498 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff5dc0186c0 0x7ff5dc01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:20.225 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:20.218 7ff5f605d700 1 --2- 172.21.2.102:0/4276410498 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff5dc0186c0 0x7ff5dc01abc0 crc :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:20.250 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:20.248 7ff5fcc29700 1 -- 172.21.2.102:0/4276410498 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5f80fe1c0 con 0x7ff5f8085bc0 2021-02-10T11:35:20.252 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:20.251 7ff5ef7fe700 1 -- 172.21.2.102:0/4276410498 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ff5e0000960 con 0x7ff5f8085bc0 2021-02-10T11:35:20.379 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:20.378 7ff5fcc29700 1 -- 172.21.2.102:0/4276410498 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7ff5f80fe1c0 con 0x7ff5f8085bc0 2021-02-10T11:35:20.380 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:20.378 7ff5ef7fe700 1 -- 172.21.2.102:0/4276410498 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v145) v1 ==== 104+0+10 (secure 0 0 0) 0x7ff5e0000c90 con 0x7ff5f8085bc0 2021-02-10T11:35:20.380 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:35:20.382 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:20.380 7ff5fcc29700 1 -- 172.21.2.102:0/4276410498 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff5dc0186c0 msgr2=0x7ff5dc01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:20.382 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:20.380 7ff5fcc29700 1 --2- 172.21.2.102:0/4276410498 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff5dc0186c0 0x7ff5dc01abc0 crc :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:20.382 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:20.380 7ff5fcc29700 1 -- 172.21.2.102:0/4276410498 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff5f8085bc0 msgr2=0x7ff5f80863e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:20.383 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:20.380 7ff5fcc29700 1 --2- 172.21.2.102:0/4276410498 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff5f8085bc0 0x7ff5f80863e0 secure :-1 s=READY pgs=2675 cs=0 l=1 rev1=1 rx=0x7ff5e000b530 tx=0x7ff5e000b8d0).stop 2021-02-10T11:35:20.383 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:20.380 7ff5fcc29700 1 -- 172.21.2.102:0/4276410498 shutdown_connections 2021-02-10T11:35:20.383 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:20.380 7ff5fcc29700 1 --2- 172.21.2.102:0/4276410498 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff5f80958c0 0x7ff5f8096920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:20.383 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:20.380 7ff5fcc29700 1 --2- 172.21.2.102:0/4276410498 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff5dc0186c0 0x7ff5dc01abc0 unknown :-1 s=CLOSED pgs=1180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:20.383 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:20.380 7ff5fcc29700 1 --2- 172.21.2.102:0/4276410498 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff5f80954c0 0x7ff5f8096300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:20.383 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:20.380 7ff5fcc29700 1 --2- 172.21.2.102:0/4276410498 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff5f8085bc0 0x7ff5f80863e0 unknown :-1 s=CLOSED pgs=2675 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:20.383 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:20.380 7ff5fcc29700 1 -- 172.21.2.102:0/4276410498 >> 172.21.2.102:0/4276410498 conn(0x7ff5f80bef00 msgr2=0x7ff5f809d1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:20.383 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:20.380 7ff5fcc29700 1 -- 172.21.2.102:0/4276410498 shutdown_connections 2021-02-10T11:35:20.383 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:20.380 7ff5fcc29700 1 -- 172.21.2.102:0/4276410498 wait complete. 2021-02-10T11:35:20.391 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-02-10T11:35:20.821 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:20.803 7fc1cfd62700 1 Processor -- start 2021-02-10T11:35:20.821 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:20.803 7fc1cfd62700 1 -- start start 2021-02-10T11:35:20.821 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:20.803 7fc1cfd62700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fc1c80d1c70 0x7fc1c80cef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:20.821 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:20.803 7fc1cfd62700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1c80d0170 con 0x7fc1c80cf580 2021-02-10T11:35:20.821 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:20.803 7fc1cfd62700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1c80fe1c0 con 0x7fc1c80d1870 2021-02-10T11:35:20.822 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:20.803 7fc1cfd62700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1c81b0cb0 con 0x7fc1c80d1c70 2021-02-10T11:35:20.822 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:20.803 7fc1cd37d700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fc1c80d1c70 0x7fc1c80cef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:20.822 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:20.803 7fc1cd37d700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fc1c80d1c70 0x7fc1c80cef60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:36914/0 (socket says 172.21.2.102:36914) 2021-02-10T11:35:20.822 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:20.803 7fc1cd37d700 1 -- 172.21.2.102:0/2267737471 learned_addr learned my addr 172.21.2.102:0/2267737471 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:35:20.822 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:20.803 7fc1ce37f700 1 --1- 172.21.2.102:0/2267737471 >> v1:172.21.2.102:6789/0 conn(0x7fc1c80cf580 0x7fc1c80cf980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:57374/0 (socket says 172.21.2.102:57374) 2021-02-10T11:35:20.822 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:20.804 7fc1cd37d700 1 -- 172.21.2.102:0/2267737471 >> v1:172.21.2.113:6789/0 conn(0x7fc1c80d1870 legacy=0x7fc1c81b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:20.822 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:20.804 7fc1cd37d700 1 -- 172.21.2.102:0/2267737471 >> v1:172.21.2.102:6789/0 conn(0x7fc1c80cf580 legacy=0x7fc1c80cf980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:20.822 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:20.804 7fc1cd37d700 1 -- 172.21.2.102:0/2267737471 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1c8109a60 con 0x7fc1c80d1c70 2021-02-10T11:35:20.822 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:20.804 7fc1cd37d700 1 --2- 172.21.2.102:0/2267737471 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc1c80d1c70 0x7fc1c80cef60 secure :-1 s=READY pgs=2677 cs=0 l=1 rev1=1 rx=0x7fc1c4006700 tx=0x7fc1c4009b80).ready entity=mon.0 client_cookie=ed2e3ac6377414a8 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:20.822 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:20.804 7fc1ccb7c700 1 -- 172.21.2.102:0/2267737471 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc1c400c0c0 con 0x7fc1c80d1c70 2021-02-10T11:35:20.822 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:20.804 7fc1ccb7c700 1 -- 172.21.2.102:0/2267737471 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1c400c440 con 0x7fc1c80d1c70 2021-02-10T11:35:20.822 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:20.804 7fc1ccb7c700 1 -- 172.21.2.102:0/2267737471 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc1c400c960 con 0x7fc1c80d1c70 2021-02-10T11:35:20.822 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:20.804 7fc1cfd62700 1 -- 172.21.2.102:0/2267737471 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc1c80d1c70 msgr2=0x7fc1c80cef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:20.823 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:20.804 7fc1cfd62700 1 --2- 172.21.2.102:0/2267737471 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc1c80d1c70 0x7fc1c80cef60 secure :-1 s=READY pgs=2677 cs=0 l=1 rev1=1 rx=0x7fc1c4006700 tx=0x7fc1c4009b80).stop 2021-02-10T11:35:20.823 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:20.805 7fc1cfd62700 1 -- 172.21.2.102:0/2267737471 shutdown_connections 2021-02-10T11:35:20.823 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:20.805 7fc1cfd62700 1 --2- 172.21.2.102:0/2267737471 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc1c80d1c70 0x7fc1c80cef60 unknown :-1 s=CLOSED pgs=2677 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:20.823 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:20.805 7fc1cfd62700 1 -- 172.21.2.102:0/2267737471 >> 172.21.2.102:0/2267737471 conn(0x7fc1c80ce420 msgr2=0x7fc1c80d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:20.823 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:20.805 7fc1cfd62700 1 -- 172.21.2.102:0/2267737471 shutdown_connections 2021-02-10T11:35:20.823 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:20.805 7fc1cfd62700 1 -- 172.21.2.102:0/2267737471 wait complete. 2021-02-10T11:35:20.824 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:20.820 7fc1cfd62700 1 Processor -- start 2021-02-10T11:35:20.824 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:20.820 7fc1cfd62700 1 -- start start 2021-02-10T11:35:20.824 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:20.820 7fc1cfd62700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc1c80c0740 0x7fc1c81c5580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:20.825 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:20.820 7fc1cfd62700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc1c81c5ba0 0x7fc1c80a5c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:20.825 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:20.820 7fc1cfd62700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc1c80b6e60 0x7fc1c80a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:20.825 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:20.820 7fc1cfd62700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1c80a6880 con 0x7fc1c80b6e60 2021-02-10T11:35:20.825 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:20.820 7fc1cfd62700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1c80a6aa0 con 0x7fc1c80c0740 2021-02-10T11:35:20.825 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:20.820 7fc1cfd62700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1c80a6cc0 con 0x7fc1c81c5ba0 2021-02-10T11:35:20.825 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:20.820 7fc1ce37f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc1c80b6e60 0x7fc1c80a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:20.825 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:20.820 7fc1ce37f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc1c80b6e60 0x7fc1c80a6260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:36924/0 (socket says 172.21.2.102:36924) 2021-02-10T11:35:20.825 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:20.820 7fc1ce37f700 1 -- 172.21.2.102:0/1825855699 learned_addr learned my addr 172.21.2.102:0/1825855699 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:35:20.825 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:20.821 7fc1ce37f700 1 -- 172.21.2.102:0/1825855699 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc1c81c5ba0 msgr2=0x7fc1c80a5c40 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:35:20.825 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:20.821 7fc1ce37f700 1 --2- 172.21.2.102:0/1825855699 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc1c81c5ba0 0x7fc1c80a5c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:20.825 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:20.821 7fc1ce37f700 1 -- 172.21.2.102:0/1825855699 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc1c80c0740 msgr2=0x7fc1c81c5580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:20.825 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:20.821 7fc1ce37f700 1 --2- 172.21.2.102:0/1825855699 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc1c80c0740 0x7fc1c81c5580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:20.825 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:20.821 7fc1ce37f700 1 -- 172.21.2.102:0/1825855699 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1c80a6ee0 con 0x7fc1c80b6e60 2021-02-10T11:35:20.826 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:20.821 7fc1ce37f700 1 --2- 172.21.2.102:0/1825855699 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc1c80b6e60 0x7fc1c80a6260 secure :-1 s=READY pgs=2678 cs=0 l=1 rev1=1 rx=0x7fc1c0007e80 tx=0x7fc1c000aca0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:20.826 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:20.821 7fc1ba7fc700 1 -- 172.21.2.102:0/1825855699 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc1c0007b10 con 0x7fc1c80b6e60 2021-02-10T11:35:20.826 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:20.821 7fc1ba7fc700 1 -- 172.21.2.102:0/1825855699 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1c00070b0 con 0x7fc1c80b6e60 2021-02-10T11:35:20.826 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:20.821 7fc1cfd62700 1 -- 172.21.2.102:0/1825855699 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1c80a6ee0 con 0x7fc1c80b6e60 2021-02-10T11:35:20.826 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:20.821 7fc1ba7fc700 1 -- 172.21.2.102:0/1825855699 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc1c0007540 con 0x7fc1c80b6e60 2021-02-10T11:35:20.826 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:20.821 7fc1cfd62700 1 -- 172.21.2.102:0/1825855699 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1c80a6ee0 con 0x7fc1c80b6e60 2021-02-10T11:35:20.826 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:20.822 7fc1ba7fc700 1 -- 172.21.2.102:0/1825855699 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fc1c0007290 con 0x7fc1c80b6e60 2021-02-10T11:35:20.826 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:20.822 7fc1ba7fc700 1 --2- 172.21.2.102:0/1825855699 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc1b00186d0 0x7fc1b001abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:20.826 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:20.822 7fc1ba7fc700 1 -- 172.21.2.102:0/1825855699 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fc1c00329d0 con 0x7fc1c80b6e60 2021-02-10T11:35:20.826 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:20.822 7fc1cdb7e700 1 --2- 172.21.2.102:0/1825855699 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc1b00186d0 0x7fc1b001abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:20.826 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:20.822 7fc1cdb7e700 1 --2- 172.21.2.102:0/1825855699 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc1b00186d0 0x7fc1b001abd0 crc :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:20.855 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:20.853 7fc1cfd62700 1 -- 172.21.2.102:0/1825855699 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1c80807d0 con 0x7fc1c80b6e60 2021-02-10T11:35:20.857 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:20.856 7fc1ba7fc700 1 -- 172.21.2.102:0/1825855699 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fc1c00073c0 con 0x7fc1c80b6e60 2021-02-10T11:35:20.985 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:20.983 7fc1cfd62700 1 -- 172.21.2.102:0/1825855699 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7fc1c8088070 con 0x7fc1c80b6e60 2021-02-10T11:35:20.985 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:20.983 7fc1ba7fc700 1 -- 172.21.2.102:0/1825855699 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v145) v1 ==== 100+0+10 (secure 0 0 0) 0x7fc1c00070b0 con 0x7fc1c80b6e60 2021-02-10T11:35:20.986 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:35:20.987 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:20.985 7fc1cfd62700 1 -- 172.21.2.102:0/1825855699 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc1b00186d0 msgr2=0x7fc1b001abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:20.987 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:20.985 7fc1cfd62700 1 --2- 172.21.2.102:0/1825855699 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc1b00186d0 0x7fc1b001abd0 crc :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:20.987 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:20.985 7fc1cfd62700 1 -- 172.21.2.102:0/1825855699 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc1c80b6e60 msgr2=0x7fc1c80a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:20.987 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:20.985 7fc1cfd62700 1 --2- 172.21.2.102:0/1825855699 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc1c80b6e60 0x7fc1c80a6260 secure :-1 s=READY pgs=2678 cs=0 l=1 rev1=1 rx=0x7fc1c0007e80 tx=0x7fc1c000aca0).stop 2021-02-10T11:35:20.987 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:20.985 7fc1cfd62700 1 -- 172.21.2.102:0/1825855699 shutdown_connections 2021-02-10T11:35:20.988 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:20.985 7fc1cfd62700 1 --2- 172.21.2.102:0/1825855699 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc1c80b6e60 0x7fc1c80a6260 unknown :-1 s=CLOSED pgs=2678 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:20.988 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:20.985 7fc1cfd62700 1 --2- 172.21.2.102:0/1825855699 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc1c81c5ba0 0x7fc1c80a5c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:20.988 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:20.985 7fc1cfd62700 1 --2- 172.21.2.102:0/1825855699 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc1b00186d0 0x7fc1b001abd0 unknown :-1 s=CLOSED pgs=1181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:20.988 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:20.985 7fc1cfd62700 1 --2- 172.21.2.102:0/1825855699 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc1c80c0740 0x7fc1c81c5580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:20.988 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:20.985 7fc1cfd62700 1 -- 172.21.2.102:0/1825855699 >> 172.21.2.102:0/1825855699 conn(0x7fc1c80c4120 msgr2=0x7fc1c80ffd60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:20.988 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:20.985 7fc1cfd62700 1 -- 172.21.2.102:0/1825855699 shutdown_connections 2021-02-10T11:35:20.988 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:20.985 7fc1cfd62700 1 -- 172.21.2.102:0/1825855699 wait complete. 2021-02-10T11:35:20.996 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-02-10T11:35:20.996 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:35:21.424 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:21.405 7feef57f4700 1 Processor -- start 2021-02-10T11:35:21.425 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:21.405 7feef57f4700 1 -- start start 2021-02-10T11:35:21.425 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:21.406 7feef57f4700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7feef00d1870 0x7feef01b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:21.425 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:21.406 7feef57f4700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7feef01b0a10 0x7feef00cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:21.425 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:21.406 7feef57f4700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feef00f0b10 con 0x7feef00d1c70 2021-02-10T11:35:21.425 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:21.406 7feef57f4700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feef00e5ed0 con 0x7feef00d1870 2021-02-10T11:35:21.425 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:21.406 7feef57f4700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feef00f0650 con 0x7feef01b0a10 2021-02-10T11:35:21.425 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:21.407 7feeef574700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7feef00d1870 0x7feef01b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:21.425 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:21.407 7feeeed73700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7feef01b0a10 0x7feef00cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:21.425 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:21.407 7feeefd75700 1 --1- >> v1:172.21.2.113:6789/0 conn(0x7feef00d1c70 0x7feef00cf950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.113:6789/0 says I am v1:172.21.2.102:36476/0 (socket says 172.21.2.102:36476) 2021-02-10T11:35:21.425 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:21.407 7feeefd75700 1 -- 172.21.2.102:0/3135071543 learned_addr learned my addr 172.21.2.102:0/3135071543 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:35:21.425 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:21.407 7feeeed73700 1 -- 172.21.2.102:0/3135071543 >> v2:172.21.2.113:3300/0 conn(0x7feef00d1870 msgr2=0x7feef01b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:21.425 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:21.407 7feeeed73700 1 --2- 172.21.2.102:0/3135071543 >> v2:172.21.2.113:3300/0 conn(0x7feef00d1870 0x7feef01b03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:21.426 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:21.407 7feeeed73700 1 -- 172.21.2.102:0/3135071543 >> v1:172.21.2.113:6789/0 conn(0x7feef00d1c70 legacy=0x7feef00cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:21.426 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:21.407 7feeeed73700 1 -- 172.21.2.102:0/3135071543 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feef00f0870 con 0x7feef01b0a10 2021-02-10T11:35:21.426 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:21.407 7feeeed73700 1 --2- 172.21.2.102:0/3135071543 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7feef01b0a10 0x7feef00cf330 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 rx=0x7feee0006650 tx=0x7feee0009a80).ready entity=mon.2 client_cookie=a79450ee564be7e1 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:21.426 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:21.408 7feeee572700 1 -- 172.21.2.102:0/3135071543 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7feee000c0c0 con 0x7feef01b0a10 2021-02-10T11:35:21.426 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:21.408 7feeee572700 1 -- 172.21.2.102:0/3135071543 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feee000c0c0 con 0x7feef01b0a10 2021-02-10T11:35:21.426 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:21.408 7feeee572700 1 -- 172.21.2.102:0/3135071543 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7feee000c340 con 0x7feef01b0a10 2021-02-10T11:35:21.426 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:21.408 7feef57f4700 1 -- 172.21.2.102:0/3135071543 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7feef01b0a10 msgr2=0x7feef00cf330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:21.426 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:21.408 7feef57f4700 1 --2- 172.21.2.102:0/3135071543 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7feef01b0a10 0x7feef00cf330 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 rx=0x7feee0006650 tx=0x7feee0009a80).stop 2021-02-10T11:35:21.426 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:21.408 7feef57f4700 1 -- 172.21.2.102:0/3135071543 shutdown_connections 2021-02-10T11:35:21.426 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:21.408 7feef57f4700 1 --2- 172.21.2.102:0/3135071543 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7feef01b0a10 0x7feef00cf330 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:21.426 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:21.408 7feef57f4700 1 --2- 172.21.2.102:0/3135071543 >> v2:172.21.2.113:3300/0 conn(0x7feef00d1870 0x7feef01b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:21.426 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:21.408 7feef57f4700 1 -- 172.21.2.102:0/3135071543 >> 172.21.2.102:0/3135071543 conn(0x7feef00ce450 msgr2=0x7feef00d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:21.427 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:21.409 7feef57f4700 1 -- 172.21.2.102:0/3135071543 shutdown_connections 2021-02-10T11:35:21.427 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:21.409 7feef57f4700 1 -- 172.21.2.102:0/3135071543 wait complete. 2021-02-10T11:35:21.428 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:21.423 7feef57f4700 1 Processor -- start 2021-02-10T11:35:21.428 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:21.423 7feef57f4700 1 -- start start 2021-02-10T11:35:21.428 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:21.423 7feef57f4700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7feef018d870 0x7feef00b19b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:21.428 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:21.423 7feef57f4700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7feef00b1fd0 0x7feef0099ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:21.428 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:21.423 7feef57f4700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7feef00b23d0 0x7feef009a510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:21.428 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:21.423 7feef57f4700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feef018dc70 con 0x7feef018d870 2021-02-10T11:35:21.429 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:21.423 7feef57f4700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feef0199e60 con 0x7feef00b1fd0 2021-02-10T11:35:21.429 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:21.423 7feef57f4700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feef009ab30 con 0x7feef00b23d0 2021-02-10T11:35:21.429 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:21.424 7feeef574700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7feef018d870 0x7feef00b19b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:21.429 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:21.424 7feeef574700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7feef018d870 0x7feef00b19b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:36932/0 (socket says 172.21.2.102:36932) 2021-02-10T11:35:21.429 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:21.424 7feeef574700 1 -- 172.21.2.102:0/185068006 learned_addr learned my addr 172.21.2.102:0/185068006 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:35:21.429 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:21.424 7feeef574700 1 -- 172.21.2.102:0/185068006 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7feef00b23d0 msgr2=0x7feef009a510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:21.429 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:21.424 7feeef574700 1 --2- 172.21.2.102:0/185068006 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7feef00b23d0 0x7feef009a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:21.429 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:21.424 7feeef574700 1 -- 172.21.2.102:0/185068006 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7feef00b1fd0 msgr2=0x7feef0099ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:21.429 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:21.424 7feeef574700 1 --2- 172.21.2.102:0/185068006 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7feef00b1fd0 0x7feef0099ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:21.429 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:21.424 7feeef574700 1 -- 172.21.2.102:0/185068006 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feef009ad50 con 0x7feef018d870 2021-02-10T11:35:21.429 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:21.424 7feeef574700 1 --2- 172.21.2.102:0/185068006 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7feef018d870 0x7feef00b19b0 secure :-1 s=READY pgs=2679 cs=0 l=1 rev1=1 rx=0x7feed8006f60 tx=0x7feed800b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:21.429 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:21.425 7feee7fff700 1 -- 172.21.2.102:0/185068006 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7feed80009b0 con 0x7feef018d870 2021-02-10T11:35:21.429 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:21.425 7feee7fff700 1 -- 172.21.2.102:0/185068006 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feed800bdf0 con 0x7feef018d870 2021-02-10T11:35:21.430 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:21.425 7feee7fff700 1 -- 172.21.2.102:0/185068006 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7feed8002030 con 0x7feef018d870 2021-02-10T11:35:21.430 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:21.425 7feef57f4700 1 -- 172.21.2.102:0/185068006 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feef009ad50 con 0x7feef018d870 2021-02-10T11:35:21.430 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:21.425 7feef57f4700 1 -- 172.21.2.102:0/185068006 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feef009ad50 con 0x7feef018d870 2021-02-10T11:35:21.430 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:21.425 7feee7fff700 1 -- 172.21.2.102:0/185068006 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7feed8006b00 con 0x7feef018d870 2021-02-10T11:35:21.430 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:21.425 7feee7fff700 1 --2- 172.21.2.102:0/185068006 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7feed4018730 0x7feed401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:21.430 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:21.425 7feee7fff700 1 -- 172.21.2.102:0/185068006 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 5573+0+0 (secure 0 0 0) 0x7feed80327f0 con 0x7feef018d870 2021-02-10T11:35:21.430 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:21.425 7feeeed73700 1 --2- 172.21.2.102:0/185068006 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7feed4018730 0x7feed401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:21.430 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:21.426 7feeeed73700 1 --2- 172.21.2.102:0/185068006 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7feed4018730 0x7feed401ac10 crc :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:21.458 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:21.456 7feef57f4700 1 -- 172.21.2.102:0/185068006 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feef00999c0 con 0x7feef018d870 2021-02-10T11:35:21.460 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:21.459 7feee7fff700 1 -- 172.21.2.102:0/185068006 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7feed8006340 con 0x7feef018d870 2021-02-10T11:35:21.586 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:21.585 7feef57f4700 1 -- 172.21.2.102:0/185068006 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7feef00999c0 con 0x7feef018d870 2021-02-10T11:35:21.587 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:21.585 7feee7fff700 1 -- 172.21.2.102:0/185068006 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 124 v124) v1 ==== 95+0+2111 (secure 0 0 0) 0x7feed8006b00 con 0x7feef018d870 2021-02-10T11:35:21.587 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:35:21.588 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":124,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":11784,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.102:6835/2097086088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6834","nonce":2097086088},{"type":"v1","addr":"172.21.2.102:6835","nonce":2097086088}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":120}],"filesystems":[{"mdsmap":{"epoch":124,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:35:11.628704","modified":"2021-02-10 11:35:13.639277","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11781},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11781":{"gid":11781,"name":"b","rank":0,"incarnation":121,"state":"up:active","state_seq":3,"addr":"172.21.2.113:6833/3334075470","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6832","nonce":3334075470},{"type":"v1","addr":"172.21.2.113:6833","nonce":3334075470}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[31],"metadata_pool":30,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":15}]} 2021-02-10T11:35:21.588 INFO:teuthology.orchestra.run.gibba002.stderr:dumped fsmap epoch 124 2021-02-10T11:35:21.589 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:21.587 7feef57f4700 1 -- 172.21.2.102:0/185068006 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7feed4018730 msgr2=0x7feed401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:21.589 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:21.587 7feef57f4700 1 --2- 172.21.2.102:0/185068006 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7feed4018730 0x7feed401ac10 crc :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:21.589 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:21.587 7feef57f4700 1 -- 172.21.2.102:0/185068006 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7feef018d870 msgr2=0x7feef00b19b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:21.589 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:21.587 7feef57f4700 1 --2- 172.21.2.102:0/185068006 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7feef018d870 0x7feef00b19b0 secure :-1 s=READY pgs=2679 cs=0 l=1 rev1=1 rx=0x7feed8006f60 tx=0x7feed800b970).stop 2021-02-10T11:35:21.589 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:21.587 7feef57f4700 1 -- 172.21.2.102:0/185068006 shutdown_connections 2021-02-10T11:35:21.589 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:21.587 7feef57f4700 1 --2- 172.21.2.102:0/185068006 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7feef00b23d0 0x7feef009a510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:21.590 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:21.587 7feef57f4700 1 --2- 172.21.2.102:0/185068006 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7feed4018730 0x7feed401ac10 unknown :-1 s=CLOSED pgs=1182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:21.590 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:21.587 7feef57f4700 1 --2- 172.21.2.102:0/185068006 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7feef00b1fd0 0x7feef0099ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:21.590 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:21.587 7feef57f4700 1 --2- 172.21.2.102:0/185068006 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7feef018d870 0x7feef00b19b0 unknown :-1 s=CLOSED pgs=2679 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:21.590 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:21.587 7feef57f4700 1 -- 172.21.2.102:0/185068006 >> 172.21.2.102:0/185068006 conn(0x7feef00f0550 msgr2=0x7feef0191730 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:21.590 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:21.587 7feef57f4700 1 -- 172.21.2.102:0/185068006 shutdown_connections 2021-02-10T11:35:21.590 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:21.587 7feef57f4700 1 -- 172.21.2.102:0/185068006 wait complete. 2021-02-10T11:35:21.799 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 124, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:35:11.628704', 'modified': '2021-02-10 11:35:13.639277', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 11781}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_11781': {'gid': 11781, 'name': 'b', 'rank': 0, 'incarnation': 121, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.113:6833/3334075470', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.113:6832', 'nonce': 3334075470}, {'type': 'v1', 'addr': '172.21.2.113:6833', 'nonce': 3334075470}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [31], 'metadata_pool': 30, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:35:21.799 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:35:21.800 DEBUG:teuthology.orchestra.run.gibba013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:35:22.199 INFO:teuthology.orchestra.run.gibba013.stdout:{ 2021-02-10T11:35:22.199 INFO:teuthology.orchestra.run.gibba013.stdout: "cluster_fsid": "68db30ea-dbce-4aef-a6df-f01c6fea0513", 2021-02-10T11:35:22.199 INFO:teuthology.orchestra.run.gibba013.stdout: "whoami": 0, 2021-02-10T11:35:22.199 INFO:teuthology.orchestra.run.gibba013.stdout: "id": 11781, 2021-02-10T11:35:22.199 INFO:teuthology.orchestra.run.gibba013.stdout: "want_state": "up:active", 2021-02-10T11:35:22.199 INFO:teuthology.orchestra.run.gibba013.stdout: "state": "up:active", 2021-02-10T11:35:22.199 INFO:teuthology.orchestra.run.gibba013.stdout: "rank_uptime": 10.5670381, 2021-02-10T11:35:22.199 INFO:teuthology.orchestra.run.gibba013.stdout: "mdsmap_epoch": 124, 2021-02-10T11:35:22.199 INFO:teuthology.orchestra.run.gibba013.stdout: "osdmap_epoch": 145, 2021-02-10T11:35:22.199 INFO:teuthology.orchestra.run.gibba013.stdout: "osdmap_epoch_barrier": 145, 2021-02-10T11:35:22.199 INFO:teuthology.orchestra.run.gibba013.stdout: "uptime": 16.219151724 2021-02-10T11:35:22.200 INFO:teuthology.orchestra.run.gibba013.stdout:} 2021-02-10T11:35:22.203 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "68db30ea-dbce-4aef-a6df-f01c6fea0513", "whoami": 0, "id": 11781, "want_state": "up:active", "state": "up:active", "rank_uptime": 10.5670381, "mdsmap_epoch": 124, "osdmap_epoch": 145, "osdmap_epoch_barrier": 145, "uptime": 16.219151724 } 2021-02-10T11:35:22.203 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2021-02-10T11:35:22.203 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2021-02-10T11:35:22.203 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@gibba002.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2021-02-10T11:35:22.203 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:35:22.204 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec mkdir -p /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:35:22.539 DEBUG:teuthology.orchestra.run.gibba002:> sudo modprobe fuse 2021-02-10T11:35:22.884 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:35:22.885 DEBUG:teuthology.orchestra.run.gibba002:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:35:23.228 INFO:teuthology.orchestra.run.gibba002.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:35:23.228 INFO:teuthology.orchestra.run.gibba002.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:35:23.229 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:35:23.229 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:35:23.229 DEBUG:teuthology.orchestra.run.gibba002:> ls /sys/fs/fuse/connections 2021-02-10T11:35:23.564 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2021-02-10T11:35:23.564 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:35:23.567 DEBUG:teuthology.orchestra.run.gibba002:> sudo modprobe fuse 2021-02-10T11:35:23.911 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:35:23.911 DEBUG:teuthology.orchestra.run.gibba002:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:35:23.979 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba002.stderr:ceph-fuse[197319]: starting ceph client 2021-02-10T11:35:23.979 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba002.stderr:2021-02-10 11:35:23.977 7fc19f859f80 -1 init, newargv = 0x5591b8cf91b0 newargc=9 2021-02-10T11:35:23.984 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba002.stderr:ceph-fuse[197319]: starting fuse 2021-02-10T11:35:24.254 INFO:teuthology.orchestra.run.gibba002.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:35:24.255 INFO:teuthology.orchestra.run.gibba002.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:35:24.255 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:35:24.256 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:35:24.256 DEBUG:teuthology.orchestra.run.gibba002:> ls /sys/fs/fuse/connections 2021-02-10T11:35:24.591 INFO:teuthology.orchestra.run.gibba002.stdout:43 2021-02-10T11:35:24.591 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2021-02-10T11:35:24.592 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:35:24.592 DEBUG:teuthology.orchestra.run.gibba002:> sudo python3 -c ' 2021-02-10T11:35:24.592 DEBUG:teuthology.orchestra.run.gibba002:> import glob 2021-02-10T11:35:24.592 DEBUG:teuthology.orchestra.run.gibba002:> import re 2021-02-10T11:35:24.592 DEBUG:teuthology.orchestra.run.gibba002:> import os 2021-02-10T11:35:24.592 DEBUG:teuthology.orchestra.run.gibba002:> import subprocess 2021-02-10T11:35:24.592 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:35:24.592 DEBUG:teuthology.orchestra.run.gibba002:> def find_socket(client_name): 2021-02-10T11:35:24.592 DEBUG:teuthology.orchestra.run.gibba002:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2021-02-10T11:35:24.592 DEBUG:teuthology.orchestra.run.gibba002:> files = glob.glob(asok_path) 2021-02-10T11:35:24.592 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:35:24.593 DEBUG:teuthology.orchestra.run.gibba002:> # Given a non-glob path, it better be there 2021-02-10T11:35:24.593 DEBUG:teuthology.orchestra.run.gibba002:> if "*" not in asok_path: 2021-02-10T11:35:24.593 DEBUG:teuthology.orchestra.run.gibba002:> assert(len(files) == 1) 2021-02-10T11:35:24.593 DEBUG:teuthology.orchestra.run.gibba002:> return files[0] 2021-02-10T11:35:24.593 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:35:24.593 DEBUG:teuthology.orchestra.run.gibba002:> for f in files: 2021-02-10T11:35:24.593 DEBUG:teuthology.orchestra.run.gibba002:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2021-02-10T11:35:24.593 DEBUG:teuthology.orchestra.run.gibba002:> if os.path.exists("/proc/{0}".format(pid)): 2021-02-10T11:35:24.593 DEBUG:teuthology.orchestra.run.gibba002:> return f 2021-02-10T11:35:24.593 DEBUG:teuthology.orchestra.run.gibba002:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2021-02-10T11:35:24.593 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:35:24.593 DEBUG:teuthology.orchestra.run.gibba002:> print(find_socket("client.0")) 2021-02-10T11:35:24.593 DEBUG:teuthology.orchestra.run.gibba002:> ' 2021-02-10T11:35:24.955 INFO:teuthology.orchestra.run.gibba002.stdout:/var/run/ceph/ceph-client.0.197319.asok 2021-02-10T11:35:24.957 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.197319.asok 2021-02-10T11:35:24.957 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:35:24.958 DEBUG:teuthology.orchestra.run.gibba002:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.197319.asok status 2021-02-10T11:35:25.347 INFO:teuthology.orchestra.run.gibba002.stdout:{ 2021-02-10T11:35:25.348 INFO:teuthology.orchestra.run.gibba002.stdout: "metadata": { 2021-02-10T11:35:25.348 INFO:teuthology.orchestra.run.gibba002.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:35:25.348 INFO:teuthology.orchestra.run.gibba002.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:35:25.348 INFO:teuthology.orchestra.run.gibba002.stdout: "entity_id": "0", 2021-02-10T11:35:25.348 INFO:teuthology.orchestra.run.gibba002.stdout: "hostname": "gibba002", 2021-02-10T11:35:25.348 INFO:teuthology.orchestra.run.gibba002.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2021-02-10T11:35:25.348 INFO:teuthology.orchestra.run.gibba002.stdout: "pid": "197319", 2021-02-10T11:35:25.348 INFO:teuthology.orchestra.run.gibba002.stdout: "root": "/" 2021-02-10T11:35:25.348 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:35:25.348 INFO:teuthology.orchestra.run.gibba002.stdout: "dentry_count": 0, 2021-02-10T11:35:25.348 INFO:teuthology.orchestra.run.gibba002.stdout: "dentry_pinned_count": 0, 2021-02-10T11:35:25.348 INFO:teuthology.orchestra.run.gibba002.stdout: "id": 11937, 2021-02-10T11:35:25.349 INFO:teuthology.orchestra.run.gibba002.stdout: "inst": { 2021-02-10T11:35:25.349 INFO:teuthology.orchestra.run.gibba002.stdout: "name": { 2021-02-10T11:35:25.349 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "client", 2021-02-10T11:35:25.349 INFO:teuthology.orchestra.run.gibba002.stdout: "num": 11937 2021-02-10T11:35:25.349 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:35:25.349 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": { 2021-02-10T11:35:25.349 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:35:25.349 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:0", 2021-02-10T11:35:25.349 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 2656155659 2021-02-10T11:35:25.349 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:35:25.349 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:35:25.349 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": { 2021-02-10T11:35:25.350 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:35:25.350 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:0", 2021-02-10T11:35:25.350 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 2656155659 2021-02-10T11:35:25.350 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:35:25.350 INFO:teuthology.orchestra.run.gibba002.stdout: "inst_str": "client.11937 172.21.2.102:0/2656155659", 2021-02-10T11:35:25.350 INFO:teuthology.orchestra.run.gibba002.stdout: "addr_str": "172.21.2.102:0/2656155659", 2021-02-10T11:35:25.350 INFO:teuthology.orchestra.run.gibba002.stdout: "inode_count": 1, 2021-02-10T11:35:25.350 INFO:teuthology.orchestra.run.gibba002.stdout: "mds_epoch": 124, 2021-02-10T11:35:25.350 INFO:teuthology.orchestra.run.gibba002.stdout: "osd_epoch": 145, 2021-02-10T11:35:25.350 INFO:teuthology.orchestra.run.gibba002.stdout: "osd_epoch_barrier": 0, 2021-02-10T11:35:25.350 INFO:teuthology.orchestra.run.gibba002.stdout: "blacklisted": false 2021-02-10T11:35:25.350 INFO:teuthology.orchestra.run.gibba002.stdout:} 2021-02-10T11:35:25.352 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:35:25.352 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:35:25.352 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:35:25.687 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:35:25.687 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:35:25.687 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:35:25.687 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:35:26.033 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:35:26.462 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:26.442 7f5ef356c700 1 Processor -- start 2021-02-10T11:35:26.462 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:26.443 7f5ef356c700 1 -- start start 2021-02-10T11:35:26.463 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:26.443 7f5ef356c700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f5eec1b0870 0x7f5eec0f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:26.463 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:26.443 7f5ef356c700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5eec0efe00 0x7f5eec0f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:26.463 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:26.443 7f5ef356c700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5eec1b0c70 con 0x7f5eec1b0470 2021-02-10T11:35:26.463 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:26.443 7f5ef356c700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5eec0f0b50 con 0x7f5eec1b0870 2021-02-10T11:35:26.463 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:26.443 7f5ef356c700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5eec0f0820 con 0x7f5eec0efe00 2021-02-10T11:35:26.463 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:26.444 7f5ef0b87700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f5eec1b0870 0x7f5eec0f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:26.463 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:26.444 7f5ef1388700 1 --1- >> v1:172.21.2.113:6789/0 conn(0x7f5eec1b0470 0x7f5eec0ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.113:6789/0 says I am v1:172.21.2.102:36508/0 (socket says 172.21.2.102:36508) 2021-02-10T11:35:26.463 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:26.444 7f5ef1388700 1 -- 172.21.2.102:0/3239813905 learned_addr learned my addr 172.21.2.102:0/3239813905 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:35:26.463 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:26.444 7f5ef1b89700 1 --2- 172.21.2.102:0/3239813905 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5eec0efe00 0x7f5eec0f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:26.463 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:26.445 7f5ef0b87700 1 -- 172.21.2.102:0/3239813905 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5eec0efe00 msgr2=0x7f5eec0f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:26.463 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:26.445 7f5ef0b87700 1 --2- 172.21.2.102:0/3239813905 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5eec0efe00 0x7f5eec0f0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:26.463 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:26.445 7f5ef0b87700 1 -- 172.21.2.102:0/3239813905 >> v1:172.21.2.113:6789/0 conn(0x7f5eec1b0470 legacy=0x7f5eec0ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:26.464 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:26.445 7f5ef0b87700 1 -- 172.21.2.102:0/3239813905 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5eec0e52e0 con 0x7f5eec1b0870 2021-02-10T11:35:26.464 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:26.445 7f5edffff700 1 -- 172.21.2.102:0/3239813905 <== mon.1 v1:172.21.2.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2655152008 0 0) 0x7f5ee0001190 con 0x7f5eec1b0470 2021-02-10T11:35:26.464 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:26.445 7f5ef0b87700 1 --2- 172.21.2.102:0/3239813905 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5eec1b0870 0x7f5eec0f20e0 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 rx=0x7f5ee8006650 tx=0x7f5ee8009a80).ready entity=mon.1 client_cookie=7dbe45d636bebcbb server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:26.464 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:26.445 7f5edffff700 1 -- 172.21.2.102:0/3239813905 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5ee800c0c0 con 0x7f5eec1b0870 2021-02-10T11:35:26.464 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:26.445 7f5edffff700 1 -- 172.21.2.102:0/3239813905 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ee800c340 con 0x7f5eec1b0870 2021-02-10T11:35:26.464 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:26.445 7f5edffff700 1 -- 172.21.2.102:0/3239813905 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5ee800c860 con 0x7f5eec1b0870 2021-02-10T11:35:26.464 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:26.445 7f5ef356c700 1 -- 172.21.2.102:0/3239813905 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5eec1b0870 msgr2=0x7f5eec0f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:26.464 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:26.445 7f5ef356c700 1 --2- 172.21.2.102:0/3239813905 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5eec1b0870 0x7f5eec0f20e0 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 rx=0x7f5ee8006650 tx=0x7f5ee8009a80).stop 2021-02-10T11:35:26.464 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:26.446 7f5ef356c700 1 -- 172.21.2.102:0/3239813905 shutdown_connections 2021-02-10T11:35:26.464 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:26.446 7f5ef356c700 1 --2- 172.21.2.102:0/3239813905 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5eec0efe00 0x7f5eec0f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:26.464 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:26.446 7f5ef356c700 1 --2- 172.21.2.102:0/3239813905 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5eec1b0870 0x7f5eec0f20e0 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:26.464 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:26.446 7f5ef356c700 1 -- 172.21.2.102:0/3239813905 >> 172.21.2.102:0/3239813905 conn(0x7f5eec0ce3e0 msgr2=0x7f5eec0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:26.464 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:26.446 7f5ef356c700 1 -- 172.21.2.102:0/3239813905 shutdown_connections 2021-02-10T11:35:26.465 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:26.446 7f5ef356c700 1 -- 172.21.2.102:0/3239813905 wait complete. 2021-02-10T11:35:26.466 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:26.461 7f5ef356c700 1 Processor -- start 2021-02-10T11:35:26.466 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:26.461 7f5ef356c700 1 -- start start 2021-02-10T11:35:26.466 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:26.461 7f5ef356c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5eec1af8a0 0x7f5eec198e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:26.467 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:26.461 7f5ef356c700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5eec1af050 0x7f5eec0aa3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:26.467 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:26.461 7f5ef356c700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5eec197d00 0x7f5eec198100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:26.467 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:26.461 7f5ef356c700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5eec1988e0 con 0x7f5eec1af8a0 2021-02-10T11:35:26.467 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:26.461 7f5ef356c700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5eec1af450 con 0x7f5eec197d00 2021-02-10T11:35:26.467 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:26.461 7f5ef356c700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5eec0aaa10 con 0x7f5eec1af050 2021-02-10T11:35:26.467 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:26.462 7f5ef1388700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5eec1af8a0 0x7f5eec198e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:26.467 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:26.462 7f5ef1388700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5eec1af8a0 0x7f5eec198e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:36966/0 (socket says 172.21.2.102:36966) 2021-02-10T11:35:26.467 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:26.462 7f5ef1388700 1 -- 172.21.2.102:0/489229485 learned_addr learned my addr 172.21.2.102:0/489229485 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:35:26.467 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:26.462 7f5ef1b89700 1 --2- 172.21.2.102:0/489229485 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5eec197d00 0x7f5eec198100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:26.467 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:26.462 7f5ef0b87700 1 --2- 172.21.2.102:0/489229485 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5eec1af050 0x7f5eec0aa3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:26.467 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:26.462 7f5ef1388700 1 -- 172.21.2.102:0/489229485 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5eec1af050 msgr2=0x7f5eec0aa3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:26.467 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:26.462 7f5ef1388700 1 --2- 172.21.2.102:0/489229485 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5eec1af050 0x7f5eec0aa3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:26.467 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:26.462 7f5ef1388700 1 -- 172.21.2.102:0/489229485 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5eec197d00 msgr2=0x7f5eec198100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:26.468 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:26.462 7f5ef1388700 1 --2- 172.21.2.102:0/489229485 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5eec197d00 0x7f5eec198100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:26.468 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:26.462 7f5ef1388700 1 -- 172.21.2.102:0/489229485 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5eec19fec0 con 0x7f5eec1af8a0 2021-02-10T11:35:26.468 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:26.462 7f5ef1388700 1 --2- 172.21.2.102:0/489229485 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5eec1af8a0 0x7f5eec198e60 secure :-1 s=READY pgs=2683 cs=0 l=1 rev1=1 rx=0x7f5ee000a590 tx=0x7f5ee000a930).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:26.468 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:26.463 7f5eddffb700 1 -- 172.21.2.102:0/489229485 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5ee000ad20 con 0x7f5eec1af8a0 2021-02-10T11:35:26.468 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:26.463 7f5eddffb700 1 -- 172.21.2.102:0/489229485 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ee0000c10 con 0x7f5eec1af8a0 2021-02-10T11:35:26.468 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:26.463 7f5ef356c700 1 -- 172.21.2.102:0/489229485 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5eec19fec0 con 0x7f5eec1af8a0 2021-02-10T11:35:26.468 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:26.463 7f5ef356c700 1 -- 172.21.2.102:0/489229485 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5eec1a0450 con 0x7f5eec1af8a0 2021-02-10T11:35:26.468 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:26.463 7f5eddffb700 1 -- 172.21.2.102:0/489229485 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5ee0004530 con 0x7f5eec1af8a0 2021-02-10T11:35:26.468 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:26.463 7f5eddffb700 1 -- 172.21.2.102:0/489229485 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f5ee000ad20 con 0x7f5eec1af8a0 2021-02-10T11:35:26.468 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:26.463 7f5eddffb700 1 --2- 172.21.2.102:0/489229485 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5ecc018730 0x7f5ecc01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:26.468 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:26.463 7f5eddffb700 1 -- 172.21.2.102:0/489229485 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f5ee0032810 con 0x7f5eec1af8a0 2021-02-10T11:35:26.468 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:26.464 7f5ef0b87700 1 --2- 172.21.2.102:0/489229485 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5ecc018730 0x7f5ecc01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:26.468 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:26.464 7f5ef0b87700 1 --2- 172.21.2.102:0/489229485 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5ecc018730 0x7f5ecc01ac10 crc :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:26.495 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:26.493 7f5ef356c700 1 -- 172.21.2.102:0/489229485 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5eec105a20 con 0x7f5eec1af8a0 2021-02-10T11:35:26.497 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:26.495 7f5eddffb700 1 -- 172.21.2.102:0/489229485 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f5ee00047f0 con 0x7f5eec1af8a0 2021-02-10T11:35:26.621 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:26.619 7f5ef356c700 1 -- 172.21.2.102:0/489229485 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5eec0f0b50 con 0x7f5eec1af8a0 2021-02-10T11:35:26.621 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:26.619 7f5eddffb700 1 -- 172.21.2.102:0/489229485 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v145) v1 ==== 74+0+14007 (secure 0 0 0) 0x7f5ee001f550 con 0x7f5eec1af8a0 2021-02-10T11:35:26.622 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:35:26.622 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":145,"fsid":"68db30ea-dbce-4aef-a6df-f01c6fea0513","created":"2021-02-10 11:03:06.331557","modified":"2021-02-10 11:35:12.631037","last_up_change":"2021-02-10 11:03:32.690562","last_in_change":"2021-02-10 11:03:32.690562","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":31,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:03:37.322519","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":30,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:35:09.594644","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"144","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":31,"pool_name":"cephfs_data","create_time":"2021-02-10 11:35:10.619592","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"145","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"be691b4d-a039-408a-af80-9d53f02b6f1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6818","nonce":36088},{"type":"v1","addr":"172.21.2.102:6820","nonce":36088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6822","nonce":36088},{"type":"v1","addr":"172.21.2.102:6824","nonce":36088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6830","nonce":36088},{"type":"v1","addr":"172.21.2.102:6832","nonce":36088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6826","nonce":36088},{"type":"v1","addr":"172.21.2.102:6828","nonce":36088}]},"public_addr":"172.21.2.102:6820/36088","cluster_addr":"172.21.2.102:6824/36088","heartbeat_back_addr":"172.21.2.102:6832/36088","heartbeat_front_addr":"172.21.2.102:6828/36088","state":["exists","up"]},{"osd":1,"uuid":"64e3ccc8-9c69-4d7c-865e-af8e83822635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6819","nonce":36087},{"type":"v1","addr":"172.21.2.102:6821","nonce":36087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6823","nonce":36087},{"type":"v1","addr":"172.21.2.102:6825","nonce":36087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6831","nonce":36087},{"type":"v1","addr":"172.21.2.102:6833","nonce":36087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6827","nonce":36087},{"type":"v1","addr":"172.21.2.102:6829","nonce":36087}]},"public_addr":"172.21.2.102:6821/36087","cluster_addr":"172.21.2.102:6825/36087","heartbeat_back_addr":"172.21.2.102:6833/36087","heartbeat_front_addr":"172.21.2.102:6829/36087","state":["exists","up"]},{"osd":2,"uuid":"153700af-747f-481f-bdf4-d6072d3a842f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6810","nonce":36089},{"type":"v1","addr":"172.21.2.102:6811","nonce":36089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6812","nonce":36089},{"type":"v1","addr":"172.21.2.102:6813","nonce":36089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6816","nonce":36089},{"type":"v1","addr":"172.21.2.102:6817","nonce":36089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6814","nonce":36089},{"type":"v1","addr":"172.21.2.102:6815","nonce":36089}]},"public_addr":"172.21.2.102:6811/36089","cluster_addr":"172.21.2.102:6813/36089","heartbeat_back_addr":"172.21.2.102:6817/36089","heartbeat_front_addr":"172.21.2.102:6815/36089","state":["exists","up"]},{"osd":3,"uuid":"cbc678f4-83a9-45e6-8830-2f7c232b1ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6802","nonce":36090},{"type":"v1","addr":"172.21.2.102:6803","nonce":36090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6804","nonce":36090},{"type":"v1","addr":"172.21.2.102:6805","nonce":36090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6808","nonce":36090},{"type":"v1","addr":"172.21.2.102:6809","nonce":36090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6806","nonce":36090},{"type":"v1","addr":"172.21.2.102:6807","nonce":36090}]},"public_addr":"172.21.2.102:6803/36090","cluster_addr":"172.21.2.102:6805/36090","heartbeat_back_addr":"172.21.2.102:6809/36090","heartbeat_front_addr":"172.21.2.102:6807/36090","state":["exists","up"]},{"osd":4,"uuid":"c01e4008-b258-4f10-afd2-9e7f37214351","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6808","nonce":36368},{"type":"v1","addr":"172.21.2.113:6811","nonce":36368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6814","nonce":36368},{"type":"v1","addr":"172.21.2.113:6818","nonce":36368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6826","nonce":36368},{"type":"v1","addr":"172.21.2.113:6828","nonce":36368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6820","nonce":36368},{"type":"v1","addr":"172.21.2.113:6824","nonce":36368}]},"public_addr":"172.21.2.113:6811/36368","cluster_addr":"172.21.2.113:6818/36368","heartbeat_back_addr":"172.21.2.113:6828/36368","heartbeat_front_addr":"172.21.2.113:6824/36368","state":["exists","up"]},{"osd":5,"uuid":"551f6007-daf8-4a98-8ee0-c2b16bcccb5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6802","nonce":36366},{"type":"v1","addr":"172.21.2.113:6805","nonce":36366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6807","nonce":36366},{"type":"v1","addr":"172.21.2.113:6810","nonce":36366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6819","nonce":36366},{"type":"v1","addr":"172.21.2.113:6822","nonce":36366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6812","nonce":36366},{"type":"v1","addr":"172.21.2.113:6815","nonce":36366}]},"public_addr":"172.21.2.113:6805/36366","cluster_addr":"172.21.2.113:6810/36366","heartbeat_back_addr":"172.21.2.113:6822/36366","heartbeat_front_addr":"172.21.2.113:6815/36366","state":["exists","up"]},{"osd":6,"uuid":"bf67c040-0e50-48be-b102-c3943bc9ebef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6800","nonce":36367},{"type":"v1","addr":"172.21.2.113:6801","nonce":36367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6803","nonce":36367},{"type":"v1","addr":"172.21.2.113:6804","nonce":36367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6813","nonce":36367},{"type":"v1","addr":"172.21.2.113:6816","nonce":36367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6806","nonce":36367},{"type":"v1","addr":"172.21.2.113:6809","nonce":36367}]},"public_addr":"172.21.2.113:6801/36367","cluster_addr":"172.21.2.113:6804/36367","heartbeat_back_addr":"172.21.2.113:6816/36367","heartbeat_front_addr":"172.21.2.113:6809/36367","state":["exists","up"]},{"osd":7,"uuid":"f5461f80-018d-428f-be89-2eb2bcbe7bb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6817","nonce":36365},{"type":"v1","addr":"172.21.2.113:6821","nonce":36365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6823","nonce":36365},{"type":"v1","addr":"172.21.2.113:6825","nonce":36365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6830","nonce":36365},{"type":"v1","addr":"172.21.2.113:6831","nonce":36365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6827","nonce":36365},{"type":"v1","addr":"172.21.2.113:6829","nonce":36365}]},"public_addr":"172.21.2.113:6821/36365","cluster_addr":"172.21.2.113:6825/36365","heartbeat_back_addr":"172.21.2.113:6831/36365","heartbeat_front_addr":"172.21.2.113:6829/36365","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:35:26.623 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:26.621 7f5ef356c700 1 -- 172.21.2.102:0/489229485 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5ecc018730 msgr2=0x7f5ecc01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:26.623 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:26.621 7f5ef356c700 1 --2- 172.21.2.102:0/489229485 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5ecc018730 0x7f5ecc01ac10 crc :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:26.623 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:26.621 7f5ef356c700 1 -- 172.21.2.102:0/489229485 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5eec1af8a0 msgr2=0x7f5eec198e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:26.623 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:26.621 7f5ef356c700 1 --2- 172.21.2.102:0/489229485 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5eec1af8a0 0x7f5eec198e60 secure :-1 s=READY pgs=2683 cs=0 l=1 rev1=1 rx=0x7f5ee000a590 tx=0x7f5ee000a930).stop 2021-02-10T11:35:26.624 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:26.621 7f5ef356c700 1 -- 172.21.2.102:0/489229485 shutdown_connections 2021-02-10T11:35:26.624 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:26.621 7f5ef356c700 1 --2- 172.21.2.102:0/489229485 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5ecc018730 0x7f5ecc01ac10 unknown :-1 s=CLOSED pgs=1183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:26.624 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:26.621 7f5ef356c700 1 --2- 172.21.2.102:0/489229485 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5eec197d00 0x7f5eec198100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:26.624 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:26.621 7f5ef356c700 1 --2- 172.21.2.102:0/489229485 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5eec1af050 0x7f5eec0aa3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:26.624 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:26.621 7f5ef356c700 1 --2- 172.21.2.102:0/489229485 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5eec1af8a0 0x7f5eec198e60 unknown :-1 s=CLOSED pgs=2683 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:26.624 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:26.621 7f5ef356c700 1 -- 172.21.2.102:0/489229485 >> 172.21.2.102:0/489229485 conn(0x7f5eec1a37a0 msgr2=0x7f5eec18c230 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:26.624 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:26.621 7f5ef356c700 1 -- 172.21.2.102:0/489229485 shutdown_connections 2021-02-10T11:35:26.624 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:26.621 7f5ef356c700 1 -- 172.21.2.102:0/489229485 wait complete. 2021-02-10T11:35:26.632 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:03:37.322519', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 30, 'pool_name': 'cephfs_metadata', 'create_time': '2021-02-10 11:35:09.594644', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '144', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 31, 'pool_name': 'cephfs_data', 'create_time': '2021-02-10 11:35:10.619592', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '145', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-02-10T11:35:26.632 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:35:27.065 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:27.046 7f818802e700 1 Processor -- start 2021-02-10T11:35:27.065 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:27.047 7f818802e700 1 -- start start 2021-02-10T11:35:27.065 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:27.047 7f818802e700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f81800d1860 0x7f81801b04b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:27.066 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:27.047 7f818802e700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f81801b0ad0 0x7f8180108930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:27.066 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:27.047 7f818802e700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f81800d1c60 0x7f8180108f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:27.066 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:27.047 7f818802e700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81800fe1b0 con 0x7f81800d1860 2021-02-10T11:35:27.066 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:27.047 7f818802e700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8180109a50 con 0x7f81801b0ad0 2021-02-10T11:35:27.066 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:27.047 7f818802e700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8180088070 con 0x7f81800d1c60 2021-02-10T11:35:27.066 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:27.047 7f8185e4a700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f81800d1860 0x7f81801b04b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:27.066 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:27.047 7f8185e4a700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f81800d1860 0x7f81801b04b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:36974/0 (socket says 172.21.2.102:36974) 2021-02-10T11:35:27.066 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:27.047 7f8185e4a700 1 -- 172.21.2.102:0/1712715133 learned_addr learned my addr 172.21.2.102:0/1712715133 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:35:27.066 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:27.048 7f8185649700 1 --2- 172.21.2.102:0/1712715133 >> v2:172.21.2.113:3300/0 conn(0x7f81801b0ad0 0x7f8180108930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:27.066 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:27.048 7f818664b700 1 --2- 172.21.2.102:0/1712715133 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f81800d1c60 0x7f8180108f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:27.066 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:27.048 7f8185e4a700 1 -- 172.21.2.102:0/1712715133 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f81800d1c60 msgr2=0x7f8180108f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:27.066 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:27.048 7f8185e4a700 1 --2- 172.21.2.102:0/1712715133 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f81800d1c60 0x7f8180108f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:27.067 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:27.048 7f8185e4a700 1 -- 172.21.2.102:0/1712715133 >> v2:172.21.2.113:3300/0 conn(0x7f81801b0ad0 msgr2=0x7f8180108930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:27.067 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:27.048 7f8185e4a700 1 --2- 172.21.2.102:0/1712715133 >> v2:172.21.2.113:3300/0 conn(0x7f81801b0ad0 0x7f8180108930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:27.067 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:27.048 7f8185e4a700 1 -- 172.21.2.102:0/1712715133 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8180109570 con 0x7f81800d1860 2021-02-10T11:35:27.067 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:27.048 7f8185649700 1 --2- 172.21.2.102:0/1712715133 >> v2:172.21.2.113:3300/0 conn(0x7f81801b0ad0 0x7f8180108930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:35:27.067 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:27.048 7f8185e4a700 1 --2- 172.21.2.102:0/1712715133 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f81800d1860 0x7f81801b04b0 secure :-1 s=READY pgs=2684 cs=0 l=1 rev1=1 rx=0x7f817c006700 tx=0x7f817c009b80).ready entity=mon.0 client_cookie=78a76322081ed1d5 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:27.067 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:27.048 7f8184e48700 1 -- 172.21.2.102:0/1712715133 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f817c00c0c0 con 0x7f81800d1860 2021-02-10T11:35:27.067 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:27.048 7f8184e48700 1 -- 172.21.2.102:0/1712715133 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f817c00c440 con 0x7f81800d1860 2021-02-10T11:35:27.067 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:27.048 7f8184e48700 1 -- 172.21.2.102:0/1712715133 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f817c00c920 con 0x7f81800d1860 2021-02-10T11:35:27.067 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:27.048 7f818802e700 1 -- 172.21.2.102:0/1712715133 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f81800d1860 msgr2=0x7f81801b04b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:27.067 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:27.048 7f818802e700 1 --2- 172.21.2.102:0/1712715133 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f81800d1860 0x7f81801b04b0 secure :-1 s=READY pgs=2684 cs=0 l=1 rev1=1 rx=0x7f817c006700 tx=0x7f817c009b80).stop 2021-02-10T11:35:27.067 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:27.049 7f818802e700 1 -- 172.21.2.102:0/1712715133 shutdown_connections 2021-02-10T11:35:27.067 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:27.049 7f818802e700 1 --2- 172.21.2.102:0/1712715133 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f81800d1c60 0x7f8180108f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:27.067 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:27.049 7f818802e700 1 --2- 172.21.2.102:0/1712715133 >> v2:172.21.2.113:3300/0 conn(0x7f81801b0ad0 0x7f8180108930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:27.068 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:27.049 7f818802e700 1 --2- 172.21.2.102:0/1712715133 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f81800d1860 0x7f81801b04b0 unknown :-1 s=CLOSED pgs=2684 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:27.068 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:27.049 7f818802e700 1 -- 172.21.2.102:0/1712715133 >> 172.21.2.102:0/1712715133 conn(0x7f81800ce410 msgr2=0x7f81800d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:27.068 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:27.049 7f818802e700 1 -- 172.21.2.102:0/1712715133 shutdown_connections 2021-02-10T11:35:27.068 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:27.049 7f818802e700 1 -- 172.21.2.102:0/1712715133 wait complete. 2021-02-10T11:35:27.068 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:27.063 7f818802e700 1 Processor -- start 2021-02-10T11:35:27.068 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:27.063 7f818802e700 1 -- start start 2021-02-10T11:35:27.070 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:27.064 7f818802e700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8180085270 0x7f8180085670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:27.070 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:27.064 7f818802e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8180085c90 0x7f8180086090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:27.070 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:27.064 7f818802e700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f81800c2740 0x7f81800c2b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:27.070 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:27.064 7f818802e700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8180084cf0 con 0x7f8180085c90 2021-02-10T11:35:27.070 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:27.064 7f818802e700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81800be1b0 con 0x7f8180085270 2021-02-10T11:35:27.071 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:27.064 7f818802e700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81800be390 con 0x7f81800c2740 2021-02-10T11:35:27.071 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:27.064 7f8185649700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8180085c90 0x7f8180086090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:27.071 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:27.064 7f8185649700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8180085c90 0x7f8180086090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:36978/0 (socket says 172.21.2.102:36978) 2021-02-10T11:35:27.071 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:27.064 7f8185649700 1 -- 172.21.2.102:0/1191324316 learned_addr learned my addr 172.21.2.102:0/1191324316 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:35:27.071 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:27.064 7f8185e4a700 1 --2- 172.21.2.102:0/1191324316 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8180085270 0x7f8180085670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:27.071 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:27.064 7f818664b700 1 --2- 172.21.2.102:0/1191324316 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f81800c2740 0x7f81800c2b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:27.071 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:27.064 7f8185649700 1 -- 172.21.2.102:0/1191324316 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f81800c2740 msgr2=0x7f81800c2b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:27.071 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:27.064 7f8185649700 1 --2- 172.21.2.102:0/1191324316 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f81800c2740 0x7f81800c2b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:27.071 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:27.064 7f8185649700 1 -- 172.21.2.102:0/1191324316 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8180085270 msgr2=0x7f8180085670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:27.071 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:27.064 7f8185649700 1 --2- 172.21.2.102:0/1191324316 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8180085270 0x7f8180085670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:27.072 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:27.065 7f8185649700 1 -- 172.21.2.102:0/1191324316 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81800c3160 con 0x7f8180085c90 2021-02-10T11:35:27.072 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:27.065 7f8185649700 1 --2- 172.21.2.102:0/1191324316 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8180085c90 0x7f8180086090 secure :-1 s=READY pgs=2685 cs=0 l=1 rev1=1 rx=0x7f8170006f20 tx=0x7f817000b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:27.072 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:27.065 7f81767fc700 1 -- 172.21.2.102:0/1191324316 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f817000bd80 con 0x7f8180085c90 2021-02-10T11:35:27.072 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:27.065 7f81767fc700 1 -- 172.21.2.102:0/1191324316 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8170000c90 con 0x7f8180085c90 2021-02-10T11:35:27.072 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:27.065 7f81767fc700 1 -- 172.21.2.102:0/1191324316 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f81700022b0 con 0x7f8180085c90 2021-02-10T11:35:27.072 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:27.065 7f818802e700 1 -- 172.21.2.102:0/1191324316 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81800c3160 con 0x7f8180085c90 2021-02-10T11:35:27.072 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:27.065 7f818802e700 1 -- 172.21.2.102:0/1191324316 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81800c3160 con 0x7f8180085c90 2021-02-10T11:35:27.072 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:27.066 7f81767fc700 1 -- 172.21.2.102:0/1191324316 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f817000a000 con 0x7f8180085c90 2021-02-10T11:35:27.072 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:27.066 7f81767fc700 1 --2- 172.21.2.102:0/1191324316 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8168018700 0x7f816801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:27.072 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:27.066 7f81767fc700 1 -- 172.21.2.102:0/1191324316 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f8170032610 con 0x7f8180085c90 2021-02-10T11:35:27.072 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:27.066 7f8185e4a700 1 --2- 172.21.2.102:0/1191324316 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8168018700 0x7f816801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:27.072 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:27.066 7f8185e4a700 1 --2- 172.21.2.102:0/1191324316 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8168018700 0x7f816801abc0 crc :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:27.099 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:27.097 7f818802e700 1 -- 172.21.2.102:0/1191324316 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8180082a20 con 0x7f8180085c90 2021-02-10T11:35:27.101 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:27.099 7f81767fc700 1 -- 172.21.2.102:0/1191324316 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f8170006bb0 con 0x7f8180085c90 2021-02-10T11:35:27.230 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:27.228 7f818802e700 1 -- 172.21.2.102:0/1191324316 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f8180082a20 con 0x7f8180085c90 2021-02-10T11:35:27.230 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:27.228 7f81767fc700 1 -- 172.21.2.102:0/1191324316 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v145) v1 ==== 92+0+10 (secure 0 0 0) 0x7f817000bd80 con 0x7f8180085c90 2021-02-10T11:35:27.231 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:35:27.232 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:27.230 7f818802e700 1 -- 172.21.2.102:0/1191324316 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8168018700 msgr2=0x7f816801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:27.233 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:27.230 7f818802e700 1 --2- 172.21.2.102:0/1191324316 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8168018700 0x7f816801abc0 crc :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:27.233 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:27.230 7f818802e700 1 -- 172.21.2.102:0/1191324316 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8180085c90 msgr2=0x7f8180086090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:27.233 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:27.230 7f818802e700 1 --2- 172.21.2.102:0/1191324316 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8180085c90 0x7f8180086090 secure :-1 s=READY pgs=2685 cs=0 l=1 rev1=1 rx=0x7f8170006f20 tx=0x7f817000b990).stop 2021-02-10T11:35:27.233 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:27.231 7f818802e700 1 -- 172.21.2.102:0/1191324316 shutdown_connections 2021-02-10T11:35:27.233 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:27.231 7f818802e700 1 --2- 172.21.2.102:0/1191324316 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f81800c2740 0x7f81800c2b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:27.233 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:27.231 7f818802e700 1 --2- 172.21.2.102:0/1191324316 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8180085c90 0x7f8180086090 unknown :-1 s=CLOSED pgs=2685 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:27.233 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:27.231 7f818802e700 1 --2- 172.21.2.102:0/1191324316 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8168018700 0x7f816801abc0 unknown :-1 s=CLOSED pgs=1184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:27.233 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:27.231 7f818802e700 1 --2- 172.21.2.102:0/1191324316 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8180085270 0x7f8180085670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:27.233 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:27.231 7f818802e700 1 -- 172.21.2.102:0/1191324316 >> 172.21.2.102:0/1191324316 conn(0x7f81800b5f10 msgr2=0x7f818009f200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:27.233 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:27.231 7f818802e700 1 -- 172.21.2.102:0/1191324316 shutdown_connections 2021-02-10T11:35:27.233 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:27.231 7f818802e700 1 -- 172.21.2.102:0/1191324316 wait complete. 2021-02-10T11:35:27.241 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-02-10T11:35:27.672 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:27.653 7f4e7d130700 1 Processor -- start 2021-02-10T11:35:27.672 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:27.653 7f4e7d130700 1 -- start start 2021-02-10T11:35:27.672 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:27.654 7f4e7d130700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f4e780d1860 0x7f4e781b04b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:27.672 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:27.654 7f4e7d130700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f4e781b0ad0 0x7f4e78108930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:27.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:27.654 7f4e7d130700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4e780d1c60 0x7f4e78108f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:27.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:27.654 7f4e7d130700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e780fe1b0 con 0x7f4e781b0ad0 2021-02-10T11:35:27.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:27.654 7f4e7d130700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e78109a50 con 0x7f4e780d1860 2021-02-10T11:35:27.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:27.654 7f4e7d130700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e78088070 con 0x7f4e780d1c60 2021-02-10T11:35:27.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:27.654 7f4e7659c700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f4e781b0ad0 0x7f4e78108930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:27.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:27.654 7f4e7659c700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f4e781b0ad0 0x7f4e78108930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:36986/0 (socket says 172.21.2.102:36986) 2021-02-10T11:35:27.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:27.654 7f4e7659c700 1 -- 172.21.2.102:0/3543373542 learned_addr learned my addr 172.21.2.102:0/3543373542 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:35:27.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:27.654 7f4e7659c700 1 -- 172.21.2.102:0/3543373542 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4e780d1c60 msgr2=0x7f4e78108f50 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:35:27.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:27.654 7f4e7659c700 1 --2- 172.21.2.102:0/3543373542 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4e780d1c60 0x7f4e78108f50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:27.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:27.655 7f4e7659c700 1 -- 172.21.2.102:0/3543373542 >> v2:172.21.2.113:3300/0 conn(0x7f4e780d1860 msgr2=0x7f4e781b04b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:35:27.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:27.655 7f4e7659c700 1 --2- 172.21.2.102:0/3543373542 >> v2:172.21.2.113:3300/0 conn(0x7f4e780d1860 0x7f4e781b04b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:27.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:27.655 7f4e7659c700 1 -- 172.21.2.102:0/3543373542 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e78109570 con 0x7f4e781b0ad0 2021-02-10T11:35:27.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:27.655 7f4e7659c700 1 --2- 172.21.2.102:0/3543373542 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4e781b0ad0 0x7f4e78108930 secure :-1 s=READY pgs=2686 cs=0 l=1 rev1=1 rx=0x7f4e6c006700 tx=0x7f4e6c009b80).ready entity=mon.0 client_cookie=6de3f7d2ebb854f5 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:27.674 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:27.655 7f4e75d9b700 1 -- 172.21.2.102:0/3543373542 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4e6c00c0c0 con 0x7f4e781b0ad0 2021-02-10T11:35:27.674 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:27.655 7f4e75d9b700 1 -- 172.21.2.102:0/3543373542 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e6c00c440 con 0x7f4e781b0ad0 2021-02-10T11:35:27.674 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:27.655 7f4e75d9b700 1 -- 172.21.2.102:0/3543373542 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4e6c00c960 con 0x7f4e781b0ad0 2021-02-10T11:35:27.674 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:27.655 7f4e7d130700 1 -- 172.21.2.102:0/3543373542 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4e781b0ad0 msgr2=0x7f4e78108930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:27.674 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:27.655 7f4e7d130700 1 --2- 172.21.2.102:0/3543373542 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4e781b0ad0 0x7f4e78108930 secure :-1 s=READY pgs=2686 cs=0 l=1 rev1=1 rx=0x7f4e6c006700 tx=0x7f4e6c009b80).stop 2021-02-10T11:35:27.674 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:27.656 7f4e7d130700 1 -- 172.21.2.102:0/3543373542 shutdown_connections 2021-02-10T11:35:27.674 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:27.656 7f4e7d130700 1 --2- 172.21.2.102:0/3543373542 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4e780d1c60 0x7f4e78108f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:27.674 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:27.656 7f4e7d130700 1 --2- 172.21.2.102:0/3543373542 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4e781b0ad0 0x7f4e78108930 unknown :-1 s=CLOSED pgs=2686 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:27.674 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:27.656 7f4e7d130700 1 --2- 172.21.2.102:0/3543373542 >> v2:172.21.2.113:3300/0 conn(0x7f4e780d1860 0x7f4e781b04b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:27.674 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:27.656 7f4e7d130700 1 -- 172.21.2.102:0/3543373542 >> 172.21.2.102:0/3543373542 conn(0x7f4e780ce410 msgr2=0x7f4e780d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:27.674 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:27.656 7f4e7d130700 1 -- 172.21.2.102:0/3543373542 shutdown_connections 2021-02-10T11:35:27.674 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:27.656 7f4e7d130700 1 -- 172.21.2.102:0/3543373542 wait complete. 2021-02-10T11:35:27.676 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:27.670 7f4e7d130700 1 Processor -- start 2021-02-10T11:35:27.676 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:27.671 7f4e7d130700 1 -- start start 2021-02-10T11:35:27.676 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:27.671 7f4e7d130700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4e781a05e0 0x7f4e7818b970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:27.676 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:27.671 7f4e7d130700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4e780b8780 0x7f4e780bfa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:27.676 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:27.671 7f4e7d130700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4e780c0030 0x7f4e78189e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:27.676 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:27.671 7f4e7d130700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e780b82a0 con 0x7f4e781a05e0 2021-02-10T11:35:27.676 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:27.671 7f4e7d130700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e780c0430 con 0x7f4e780c0030 2021-02-10T11:35:27.676 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:27.671 7f4e7d130700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e780c0650 con 0x7f4e780b8780 2021-02-10T11:35:27.676 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:27.671 7f4e76d9d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4e781a05e0 0x7f4e7818b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:27.677 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:27.671 7f4e76d9d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4e781a05e0 0x7f4e7818b970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:36992/0 (socket says 172.21.2.102:36992) 2021-02-10T11:35:27.677 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:27.671 7f4e76d9d700 1 -- 172.21.2.102:0/2641991911 learned_addr learned my addr 172.21.2.102:0/2641991911 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:35:27.677 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:27.671 7f4e76d9d700 1 -- 172.21.2.102:0/2641991911 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4e780b8780 msgr2=0x7f4e780bfa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:27.677 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:27.671 7f4e76d9d700 1 --2- 172.21.2.102:0/2641991911 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4e780b8780 0x7f4e780bfa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:27.677 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:27.671 7f4e76d9d700 1 -- 172.21.2.102:0/2641991911 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4e780c0030 msgr2=0x7f4e78189e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:27.677 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:27.671 7f4e76d9d700 1 --2- 172.21.2.102:0/2641991911 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4e780c0030 0x7f4e78189e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:27.677 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:27.672 7f4e76d9d700 1 -- 172.21.2.102:0/2641991911 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e7818a490 con 0x7f4e781a05e0 2021-02-10T11:35:27.677 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:27.672 7f4e76d9d700 1 --2- 172.21.2.102:0/2641991911 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4e781a05e0 0x7f4e7818b970 secure :-1 s=READY pgs=2687 cs=0 l=1 rev1=1 rx=0x7f4e6800b5c0 tx=0x7f4e6800b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:27.677 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:27.672 7f4e677fe700 1 -- 172.21.2.102:0/2641991911 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4e6800bd50 con 0x7f4e781a05e0 2021-02-10T11:35:27.677 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:27.672 7f4e677fe700 1 -- 172.21.2.102:0/2641991911 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e68006030 con 0x7f4e781a05e0 2021-02-10T11:35:27.677 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:27.672 7f4e7d130700 1 -- 172.21.2.102:0/2641991911 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e7818a490 con 0x7f4e781a05e0 2021-02-10T11:35:27.677 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:27.672 7f4e7d130700 1 -- 172.21.2.102:0/2641991911 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e7818a490 con 0x7f4e781a05e0 2021-02-10T11:35:27.677 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:27.672 7f4e677fe700 1 -- 172.21.2.102:0/2641991911 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4e680054d0 con 0x7f4e781a05e0 2021-02-10T11:35:27.677 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:27.672 7f4e677fe700 1 -- 172.21.2.102:0/2641991911 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4e68006210 con 0x7f4e781a05e0 2021-02-10T11:35:27.678 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:27.673 7f4e677fe700 1 --2- 172.21.2.102:0/2641991911 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4e5c018730 0x7f4e5c01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:27.678 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:27.673 7f4e677fe700 1 -- 172.21.2.102:0/2641991911 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f4e680327c0 con 0x7f4e781a05e0 2021-02-10T11:35:27.678 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:27.673 7f4e7659c700 1 --2- 172.21.2.102:0/2641991911 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4e5c018730 0x7f4e5c01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:27.678 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:27.673 7f4e7659c700 1 --2- 172.21.2.102:0/2641991911 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4e5c018730 0x7f4e5c01ac10 crc :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:27.706 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:27.704 7f4e7d130700 1 -- 172.21.2.102:0/2641991911 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e78082a20 con 0x7f4e781a05e0 2021-02-10T11:35:27.708 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:27.706 7f4e677fe700 1 -- 172.21.2.102:0/2641991911 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f4e68006b80 con 0x7f4e781a05e0 2021-02-10T11:35:27.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:27.834 7f4e7d130700 1 -- 172.21.2.102:0/2641991911 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7f4e78082a20 con 0x7f4e781a05e0 2021-02-10T11:35:27.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:27.835 7f4e677fe700 1 -- 172.21.2.102:0/2641991911 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v145) v1 ==== 104+0+10 (secure 0 0 0) 0x7f4e6800bd50 con 0x7f4e781a05e0 2021-02-10T11:35:27.837 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:35:27.838 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:27.836 7f4e7d130700 1 -- 172.21.2.102:0/2641991911 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4e5c018730 msgr2=0x7f4e5c01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:27.838 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:27.836 7f4e7d130700 1 --2- 172.21.2.102:0/2641991911 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4e5c018730 0x7f4e5c01ac10 crc :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:27.838 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:27.836 7f4e7d130700 1 -- 172.21.2.102:0/2641991911 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4e781a05e0 msgr2=0x7f4e7818b970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:27.838 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:27.836 7f4e7d130700 1 --2- 172.21.2.102:0/2641991911 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4e781a05e0 0x7f4e7818b970 secure :-1 s=READY pgs=2687 cs=0 l=1 rev1=1 rx=0x7f4e6800b5c0 tx=0x7f4e6800b960).stop 2021-02-10T11:35:27.839 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:27.837 7f4e7d130700 1 -- 172.21.2.102:0/2641991911 shutdown_connections 2021-02-10T11:35:27.839 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:27.837 7f4e7d130700 1 --2- 172.21.2.102:0/2641991911 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4e5c018730 0x7f4e5c01ac10 unknown :-1 s=CLOSED pgs=1185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:27.839 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:27.837 7f4e7d130700 1 --2- 172.21.2.102:0/2641991911 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4e780c0030 0x7f4e78189e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:27.839 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:27.837 7f4e7d130700 1 --2- 172.21.2.102:0/2641991911 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4e780b8780 0x7f4e780bfa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:27.839 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:27.837 7f4e7d130700 1 --2- 172.21.2.102:0/2641991911 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4e781a05e0 0x7f4e7818b970 unknown :-1 s=CLOSED pgs=2687 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:27.839 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:27.837 7f4e7d130700 1 -- 172.21.2.102:0/2641991911 >> 172.21.2.102:0/2641991911 conn(0x7f4e780a6a80 msgr2=0x7f4e780ae110 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:27.840 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:27.837 7f4e7d130700 1 -- 172.21.2.102:0/2641991911 shutdown_connections 2021-02-10T11:35:27.840 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:27.837 7f4e7d130700 1 -- 172.21.2.102:0/2641991911 wait complete. 2021-02-10T11:35:27.847 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-02-10T11:35:28.278 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:28.260 7fab8681c700 1 Processor -- start 2021-02-10T11:35:28.279 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:28.260 7fab8681c700 1 -- start start 2021-02-10T11:35:28.279 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:28.260 7fab8681c700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fab800d1c70 0x7fab800cef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:28.279 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:28.260 7fab8681c700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fab800cf580 0x7fab800cf980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:28.279 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:28.260 7fab8681c700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab800fe1c0 con 0x7fab800d1870 2021-02-10T11:35:28.279 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:28.260 7fab8681c700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab801b0cb0 con 0x7fab800d1c70 2021-02-10T11:35:28.279 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:28.260 7fab8681c700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab80109a60 con 0x7fab800cf580 2021-02-10T11:35:28.279 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:28.261 7fab7f7fe700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fab800d1c70 0x7fab800cef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:28.279 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:28.261 7fab7f7fe700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fab800d1c70 0x7fab800cef60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:36998/0 (socket says 172.21.2.102:36998) 2021-02-10T11:35:28.279 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:28.261 7fab7f7fe700 1 -- 172.21.2.102:0/4211223805 learned_addr learned my addr 172.21.2.102:0/4211223805 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:35:28.279 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:28.261 7fab7ffff700 1 --1- 172.21.2.102:0/4211223805 >> v1:172.21.2.102:6789/0 conn(0x7fab800d1870 0x7fab801b04c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:57454/0 (socket says 172.21.2.102:57454) 2021-02-10T11:35:28.280 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:28.261 7fab7f7fe700 1 -- 172.21.2.102:0/4211223805 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fab800cf580 msgr2=0x7fab800cf980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:28.280 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:28.261 7fab7f7fe700 1 --2- 172.21.2.102:0/4211223805 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fab800cf580 0x7fab800cf980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:28.280 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:28.261 7fab7f7fe700 1 -- 172.21.2.102:0/4211223805 >> v1:172.21.2.102:6789/0 conn(0x7fab800d1870 legacy=0x7fab801b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:28.280 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:28.261 7fab7f7fe700 1 -- 172.21.2.102:0/4211223805 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab80088070 con 0x7fab800d1c70 2021-02-10T11:35:28.280 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:28.261 7fab7f7fe700 1 --2- 172.21.2.102:0/4211223805 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fab800d1c70 0x7fab800cef60 secure :-1 s=READY pgs=2689 cs=0 l=1 rev1=1 rx=0x7fab74006700 tx=0x7fab74009b80).ready entity=mon.0 client_cookie=daf722f2c255e456 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:28.280 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:28.261 7fab7effd700 1 -- 172.21.2.102:0/4211223805 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fab7400c0c0 con 0x7fab800d1c70 2021-02-10T11:35:28.280 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:28.261 7fab7effd700 1 -- 172.21.2.102:0/4211223805 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab7400c400 con 0x7fab800d1c70 2021-02-10T11:35:28.280 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:28.261 7fab7effd700 1 -- 172.21.2.102:0/4211223805 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fab7400c920 con 0x7fab800d1c70 2021-02-10T11:35:28.280 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:28.261 7fab8681c700 1 -- 172.21.2.102:0/4211223805 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fab800d1c70 msgr2=0x7fab800cef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:28.280 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:28.261 7fab8681c700 1 --2- 172.21.2.102:0/4211223805 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fab800d1c70 0x7fab800cef60 secure :-1 s=READY pgs=2689 cs=0 l=1 rev1=1 rx=0x7fab74006700 tx=0x7fab74009b80).stop 2021-02-10T11:35:28.280 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:28.262 7fab8681c700 1 -- 172.21.2.102:0/4211223805 shutdown_connections 2021-02-10T11:35:28.280 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:28.262 7fab8681c700 1 --2- 172.21.2.102:0/4211223805 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fab800d1c70 0x7fab800cef60 unknown :-1 s=CLOSED pgs=2689 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:28.280 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:28.262 7fab8681c700 1 --2- 172.21.2.102:0/4211223805 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fab800cf580 0x7fab800cf980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:28.281 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:28.262 7fab8681c700 1 -- 172.21.2.102:0/4211223805 >> 172.21.2.102:0/4211223805 conn(0x7fab800ce420 msgr2=0x7fab800d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:28.281 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:28.262 7fab8681c700 1 -- 172.21.2.102:0/4211223805 shutdown_connections 2021-02-10T11:35:28.281 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:28.262 7fab8681c700 1 -- 172.21.2.102:0/4211223805 wait complete. 2021-02-10T11:35:28.281 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:28.277 7fab8681c700 1 Processor -- start 2021-02-10T11:35:28.282 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:28.277 7fab8681c700 1 -- start start 2021-02-10T11:35:28.282 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:28.277 7fab8681c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fab800c2370 0x7fab8018f870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:28.282 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:28.277 7fab8681c700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fab80084cb0 0x7fab801a7470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:28.282 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:28.277 7fab8681c700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fab801a7a90 0x7fab80085700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:28.283 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:28.277 7fab8681c700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab800be840 con 0x7fab800c2370 2021-02-10T11:35:28.283 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:28.277 7fab8681c700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab800c29a0 con 0x7fab801a7a90 2021-02-10T11:35:28.283 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:28.277 7fab8681c700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab801a7e90 con 0x7fab80084cb0 2021-02-10T11:35:28.283 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:28.277 7fab7ffff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fab800c2370 0x7fab8018f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:28.283 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:28.277 7fab7ffff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fab800c2370 0x7fab8018f870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:37004/0 (socket says 172.21.2.102:37004) 2021-02-10T11:35:28.283 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:28.277 7fab7ffff700 1 -- 172.21.2.102:0/2029704734 learned_addr learned my addr 172.21.2.102:0/2029704734 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:35:28.283 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:28.278 7fab7ffff700 1 -- 172.21.2.102:0/2029704734 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fab80084cb0 msgr2=0x7fab801a7470 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:35:28.283 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:28.278 7fab7ffff700 1 --2- 172.21.2.102:0/2029704734 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fab80084cb0 0x7fab801a7470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:28.283 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:28.278 7fab7ffff700 1 -- 172.21.2.102:0/2029704734 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fab801a7a90 msgr2=0x7fab80085700 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:35:28.283 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:28.278 7fab7ffff700 1 --2- 172.21.2.102:0/2029704734 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fab801a7a90 0x7fab80085700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:28.283 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:28.278 7fab7ffff700 1 -- 172.21.2.102:0/2029704734 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab801a80b0 con 0x7fab800c2370 2021-02-10T11:35:28.283 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:28.278 7fab7ffff700 1 --2- 172.21.2.102:0/2029704734 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fab800c2370 0x7fab8018f870 secure :-1 s=READY pgs=2690 cs=0 l=1 rev1=1 rx=0x7fab700014d0 tx=0x7fab7000a8a0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:28.283 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:28.278 7fab7cff9700 1 -- 172.21.2.102:0/2029704734 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fab70000ef0 con 0x7fab800c2370 2021-02-10T11:35:28.284 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:28.278 7fab7cff9700 1 -- 172.21.2.102:0/2029704734 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab7000ac90 con 0x7fab800c2370 2021-02-10T11:35:28.284 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:28.278 7fab7cff9700 1 -- 172.21.2.102:0/2029704734 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fab70004590 con 0x7fab800c2370 2021-02-10T11:35:28.284 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:28.278 7fab8681c700 1 -- 172.21.2.102:0/2029704734 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab801a80b0 con 0x7fab800c2370 2021-02-10T11:35:28.284 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:28.278 7fab8681c700 1 -- 172.21.2.102:0/2029704734 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab80086010 con 0x7fab800c2370 2021-02-10T11:35:28.284 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:28.279 7fab7cff9700 1 -- 172.21.2.102:0/2029704734 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fab700043a0 con 0x7fab800c2370 2021-02-10T11:35:28.284 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:28.279 7fab7cff9700 1 --2- 172.21.2.102:0/2029704734 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fab600186c0 0x7fab6001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:28.284 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:28.279 7fab7cff9700 1 -- 172.21.2.102:0/2029704734 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fab70032420 con 0x7fab800c2370 2021-02-10T11:35:28.284 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:28.279 7fab7f7fe700 1 --2- 172.21.2.102:0/2029704734 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fab600186c0 0x7fab6001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:28.284 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:28.279 7fab7f7fe700 1 --2- 172.21.2.102:0/2029704734 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fab600186c0 0x7fab6001abc0 crc :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:28.312 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:28.310 7fab8681c700 1 -- 172.21.2.102:0/2029704734 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab80082a70 con 0x7fab800c2370 2021-02-10T11:35:28.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:28.313 7fab7cff9700 1 -- 172.21.2.102:0/2029704734 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fab70000ef0 con 0x7fab800c2370 2021-02-10T11:35:28.442 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:28.440 7fab8681c700 1 -- 172.21.2.102:0/2029704734 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7fab80082a70 con 0x7fab800c2370 2021-02-10T11:35:28.442 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:28.440 7fab7cff9700 1 -- 172.21.2.102:0/2029704734 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v145) v1 ==== 100+0+10 (secure 0 0 0) 0x7fab700043a0 con 0x7fab800c2370 2021-02-10T11:35:28.442 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:35:28.444 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:28.442 7fab8681c700 1 -- 172.21.2.102:0/2029704734 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fab600186c0 msgr2=0x7fab6001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:28.444 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:28.442 7fab8681c700 1 --2- 172.21.2.102:0/2029704734 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fab600186c0 0x7fab6001abc0 crc :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:28.444 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:28.442 7fab8681c700 1 -- 172.21.2.102:0/2029704734 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fab800c2370 msgr2=0x7fab8018f870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:28.444 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:28.442 7fab8681c700 1 --2- 172.21.2.102:0/2029704734 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fab800c2370 0x7fab8018f870 secure :-1 s=READY pgs=2690 cs=0 l=1 rev1=1 rx=0x7fab700014d0 tx=0x7fab7000a8a0).stop 2021-02-10T11:35:28.444 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:28.442 7fab8681c700 1 -- 172.21.2.102:0/2029704734 shutdown_connections 2021-02-10T11:35:28.444 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:28.442 7fab8681c700 1 --2- 172.21.2.102:0/2029704734 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fab600186c0 0x7fab6001abc0 unknown :-1 s=CLOSED pgs=1186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:28.444 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:28.442 7fab8681c700 1 --2- 172.21.2.102:0/2029704734 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fab801a7a90 0x7fab80085700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:28.444 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:28.442 7fab8681c700 1 --2- 172.21.2.102:0/2029704734 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fab80084cb0 0x7fab801a7470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:28.444 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:28.442 7fab8681c700 1 --2- 172.21.2.102:0/2029704734 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fab800c2370 0x7fab8018f870 unknown :-1 s=CLOSED pgs=2690 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:28.444 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:28.442 7fab8681c700 1 -- 172.21.2.102:0/2029704734 >> 172.21.2.102:0/2029704734 conn(0x7fab800b5dc0 msgr2=0x7fab800c2d90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:28.445 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:28.442 7fab8681c700 1 -- 172.21.2.102:0/2029704734 shutdown_connections 2021-02-10T11:35:28.445 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:28.442 7fab8681c700 1 -- 172.21.2.102:0/2029704734 wait complete. 2021-02-10T11:35:28.452 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-02-10T11:35:28.452 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:35:28.883 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:28.863 7fab401f5700 1 Processor -- start 2021-02-10T11:35:28.883 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:28.863 7fab401f5700 1 -- start start 2021-02-10T11:35:28.883 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:28.863 7fab401f5700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fab380d1870 0x7fab381b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:28.883 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:28.863 7fab401f5700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fab381b0a10 0x7fab380cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:28.883 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:28.863 7fab401f5700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab380f0b10 con 0x7fab380d1c70 2021-02-10T11:35:28.883 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:28.863 7fab401f5700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab380fe230 con 0x7fab381b0a10 2021-02-10T11:35:28.883 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:28.863 7fab401f5700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab380e5ed0 con 0x7fab380d1870 2021-02-10T11:35:28.883 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:28.864 7fab3e812700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fab380d1c70 0x7fab380cf950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:57466/0 (socket says 172.21.2.102:57466) 2021-02-10T11:35:28.884 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:28.864 7fab3e812700 1 -- 172.21.2.102:0/851636011 learned_addr learned my addr 172.21.2.102:0/851636011 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:35:28.884 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:28.864 7fab3d00f700 1 -- 172.21.2.102:0/851636011 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3771230761 0 0) 0x7fab30001180 con 0x7fab380d1c70 2021-02-10T11:35:28.884 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:28.864 7fab3d00f700 1 -- 172.21.2.102:0/851636011 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab24002fe0 con 0x7fab380d1c70 2021-02-10T11:35:28.884 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:28.864 7fab3d00f700 1 -- 172.21.2.102:0/851636011 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3517255881 0 0) 0x7fab300010e0 con 0x7fab380d1c70 2021-02-10T11:35:28.884 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:28.864 7fab3d00f700 1 -- 172.21.2.102:0/851636011 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fab380d1870 msgr2=0x7fab381b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:28.884 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:28.864 7fab3d810700 1 --2- 172.21.2.102:0/851636011 >> v2:172.21.2.113:3300/0 conn(0x7fab381b0a10 0x7fab380cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:28.884 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:28.864 7fab3e011700 1 --2- 172.21.2.102:0/851636011 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fab380d1870 0x7fab381b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:28.884 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:28.864 7fab3d00f700 1 --2- 172.21.2.102:0/851636011 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fab380d1870 0x7fab381b03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:28.884 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:28.864 7fab3d00f700 1 -- 172.21.2.102:0/851636011 >> v2:172.21.2.113:3300/0 conn(0x7fab381b0a10 msgr2=0x7fab380cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:28.884 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:28.864 7fab3d00f700 1 --2- 172.21.2.102:0/851636011 >> v2:172.21.2.113:3300/0 conn(0x7fab381b0a10 0x7fab380cf330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:28.884 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:28.864 7fab3d00f700 1 -- 172.21.2.102:0/851636011 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab380f0650 con 0x7fab380d1c70 2021-02-10T11:35:28.884 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:28.864 7fab3d00f700 1 -- 172.21.2.102:0/851636011 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7fab30001600 con 0x7fab380d1c70 2021-02-10T11:35:28.884 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:28.865 7fab3e011700 1 --2- 172.21.2.102:0/851636011 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fab380d1870 0x7fab381b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:35:28.884 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:28.865 7fab3d00f700 1 -- 172.21.2.102:0/851636011 >> v1:172.21.2.102:6789/0 conn(0x7fab380d1c70 legacy=0x7fab380cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:28.885 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:28.865 7fab3d00f700 1 --2- 172.21.2.102:0/851636011 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fab24003af0 0x7fab24005f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:28.885 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:28.865 7fab3d00f700 1 --2- 172.21.2.102:0/851636011 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fab240067e0 0x7fab24008d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:28.885 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:28.865 7fab3d00f700 1 --2- 172.21.2.102:0/851636011 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fab240095a0 0x7fab2400bac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:28.885 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:28.865 7fab3d00f700 1 -- 172.21.2.102:0/851636011 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab2400c370 con 0x7fab240095a0 2021-02-10T11:35:28.885 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:28.865 7fab3d00f700 1 -- 172.21.2.102:0/851636011 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab2400c590 con 0x7fab240067e0 2021-02-10T11:35:28.885 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:28.865 7fab3d00f700 1 -- 172.21.2.102:0/851636011 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab240507c0 con 0x7fab24003af0 2021-02-10T11:35:28.885 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:28.865 7fab3d810700 1 --2- 172.21.2.102:0/851636011 >> v2:172.21.2.113:3300/0 conn(0x7fab381b0a10 0x7fab380cf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:35:28.885 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:28.865 7fab3e812700 1 --2- 172.21.2.102:0/851636011 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fab240095a0 0x7fab2400bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:28.885 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:28.865 7fab3e011700 1 --2- 172.21.2.102:0/851636011 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fab24003af0 0x7fab24005f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:28.885 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:28.865 7fab3d810700 1 --2- 172.21.2.102:0/851636011 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fab240067e0 0x7fab24008d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:28.885 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:28.865 7fab3e812700 1 -- 172.21.2.102:0/851636011 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fab24003af0 msgr2=0x7fab24005f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:28.885 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:28.865 7fab3e812700 1 --2- 172.21.2.102:0/851636011 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fab24003af0 0x7fab24005f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:28.885 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:28.865 7fab3e812700 1 -- 172.21.2.102:0/851636011 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fab240067e0 msgr2=0x7fab24008d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:28.886 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:28.865 7fab3e812700 1 --2- 172.21.2.102:0/851636011 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fab240067e0 0x7fab24008d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:28.886 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:28.865 7fab3e812700 1 -- 172.21.2.102:0/851636011 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab24050a20 con 0x7fab240095a0 2021-02-10T11:35:28.886 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:28.865 7fab3e812700 1 --2- 172.21.2.102:0/851636011 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fab240095a0 0x7fab2400bac0 secure :-1 s=READY pgs=2692 cs=0 l=1 rev1=1 rx=0x7fab30000de0 tx=0x7fab30007c80).ready entity=mon.0 client_cookie=23a04b9453033c9f server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:28.886 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:28.865 7fab3d00f700 1 -- 172.21.2.102:0/851636011 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fab300010e0 con 0x7fab240095a0 2021-02-10T11:35:28.886 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:28.865 7fab3d00f700 1 -- 172.21.2.102:0/851636011 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab3000a520 con 0x7fab240095a0 2021-02-10T11:35:28.886 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:28.865 7fab3d00f700 1 -- 172.21.2.102:0/851636011 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fab3000aa00 con 0x7fab240095a0 2021-02-10T11:35:28.886 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:28.865 7fab401f5700 1 -- 172.21.2.102:0/851636011 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fab240095a0 msgr2=0x7fab2400bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:28.886 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:28.865 7fab401f5700 1 --2- 172.21.2.102:0/851636011 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fab240095a0 0x7fab2400bac0 secure :-1 s=READY pgs=2692 cs=0 l=1 rev1=1 rx=0x7fab30000de0 tx=0x7fab30007c80).stop 2021-02-10T11:35:28.886 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:28.865 7fab3e812700 1 -- 172.21.2.102:0/851636011 reap_dead start 2021-02-10T11:35:28.886 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:28.866 7fab401f5700 1 -- 172.21.2.102:0/851636011 shutdown_connections 2021-02-10T11:35:28.886 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:28.866 7fab401f5700 1 -- 172.21.2.102:0/851636011 >> 172.21.2.102:0/851636011 conn(0x7fab380ce450 msgr2=0x7fab380d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:28.886 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:28.866 7fab401f5700 1 -- 172.21.2.102:0/851636011 shutdown_connections 2021-02-10T11:35:28.886 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:28.866 7fab401f5700 1 -- 172.21.2.102:0/851636011 wait complete. 2021-02-10T11:35:28.887 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:28.881 7fab401f5700 1 Processor -- start 2021-02-10T11:35:28.887 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:28.881 7fab401f5700 1 -- start start 2021-02-10T11:35:28.888 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:28.881 7fab401f5700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fab380a2560 0x7fab380b4fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:28.888 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:28.881 7fab401f5700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fab380b55e0 0x7fab380b59e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:28.889 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:28.881 7fab401f5700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fab380ee200 0x7fab380ee600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:28.889 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:28.881 7fab401f5700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab3818fda0 con 0x7fab380a2560 2021-02-10T11:35:28.889 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:28.881 7fab401f5700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab380a20c0 con 0x7fab380b55e0 2021-02-10T11:35:28.889 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:28.881 7fab401f5700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab380b6000 con 0x7fab380ee200 2021-02-10T11:35:28.889 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:28.881 7fab3e011700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fab380a2560 0x7fab380b4fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:28.889 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:28.881 7fab3e011700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fab380a2560 0x7fab380b4fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:37022/0 (socket says 172.21.2.102:37022) 2021-02-10T11:35:28.889 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:28.881 7fab3e011700 1 -- 172.21.2.102:0/142345142 learned_addr learned my addr 172.21.2.102:0/142345142 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:35:28.889 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:28.881 7fab3d810700 1 --2- 172.21.2.102:0/142345142 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fab380b55e0 0x7fab380b59e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:28.889 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:28.882 7fab3e812700 1 --2- 172.21.2.102:0/142345142 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fab380ee200 0x7fab380ee600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:28.889 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:28.882 7fab3e011700 1 -- 172.21.2.102:0/142345142 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fab380ee200 msgr2=0x7fab380ee600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:28.889 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:28.882 7fab3e011700 1 --2- 172.21.2.102:0/142345142 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fab380ee200 0x7fab380ee600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:28.889 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:28.882 7fab3e011700 1 -- 172.21.2.102:0/142345142 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fab380b55e0 msgr2=0x7fab380b59e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:28.889 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:28.882 7fab3e011700 1 --2- 172.21.2.102:0/142345142 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fab380b55e0 0x7fab380b59e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:28.890 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:28.882 7fab3e011700 1 -- 172.21.2.102:0/142345142 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab380eec20 con 0x7fab380a2560 2021-02-10T11:35:28.890 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:28.882 7fab3e812700 1 --2- 172.21.2.102:0/142345142 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fab380ee200 0x7fab380ee600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:35:28.890 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:28.882 7fab3e011700 1 --2- 172.21.2.102:0/142345142 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fab380a2560 0x7fab380b4fc0 secure :-1 s=READY pgs=2693 cs=0 l=1 rev1=1 rx=0x7fab34006f30 tx=0x7fab3400b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:28.890 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:28.882 7fab2affd700 1 -- 172.21.2.102:0/142345142 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fab3400bd80 con 0x7fab380a2560 2021-02-10T11:35:28.890 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:28.882 7fab2affd700 1 -- 172.21.2.102:0/142345142 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab34002030 con 0x7fab380a2560 2021-02-10T11:35:28.890 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:28.882 7fab2affd700 1 -- 172.21.2.102:0/142345142 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fab34002400 con 0x7fab380a2560 2021-02-10T11:35:28.890 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:28.882 7fab401f5700 1 -- 172.21.2.102:0/142345142 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab380eec20 con 0x7fab380a2560 2021-02-10T11:35:28.890 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:28.882 7fab401f5700 1 -- 172.21.2.102:0/142345142 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab380ef1b0 con 0x7fab380a2560 2021-02-10T11:35:28.890 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:28.883 7fab2affd700 1 -- 172.21.2.102:0/142345142 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fab34000c90 con 0x7fab380a2560 2021-02-10T11:35:28.890 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:28.883 7fab2affd700 1 --2- 172.21.2.102:0/142345142 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fab200186c0 0x7fab2001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:28.890 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:28.883 7fab2affd700 1 -- 172.21.2.102:0/142345142 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fab34032620 con 0x7fab380a2560 2021-02-10T11:35:28.890 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:28.883 7fab3d810700 1 --2- 172.21.2.102:0/142345142 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fab200186c0 0x7fab2001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:28.890 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:28.883 7fab3d810700 1 --2- 172.21.2.102:0/142345142 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fab200186c0 0x7fab2001abc0 crc :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:28.915 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:28.913 7fab401f5700 1 -- 172.21.2.102:0/142345142 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab380f0b10 con 0x7fab380a2560 2021-02-10T11:35:28.918 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:28.916 7fab2affd700 1 -- 172.21.2.102:0/142345142 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fab34030dd0 con 0x7fab380a2560 2021-02-10T11:35:29.049 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:29.047 7fab401f5700 1 -- 172.21.2.102:0/142345142 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fab380f0b10 con 0x7fab380a2560 2021-02-10T11:35:29.049 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:29.047 7fab2affd700 1 -- 172.21.2.102:0/142345142 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 124 v124) v1 ==== 95+0+2111 (secure 0 0 0) 0x7fab3400bd80 con 0x7fab380a2560 2021-02-10T11:35:29.049 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:35:29.050 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":124,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":11784,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.102:6835/2097086088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6834","nonce":2097086088},{"type":"v1","addr":"172.21.2.102:6835","nonce":2097086088}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":120}],"filesystems":[{"mdsmap":{"epoch":124,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:35:11.628704","modified":"2021-02-10 11:35:13.639277","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11781},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11781":{"gid":11781,"name":"b","rank":0,"incarnation":121,"state":"up:active","state_seq":3,"addr":"172.21.2.113:6833/3334075470","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6832","nonce":3334075470},{"type":"v1","addr":"172.21.2.113:6833","nonce":3334075470}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[31],"metadata_pool":30,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":15}]} 2021-02-10T11:35:29.050 INFO:teuthology.orchestra.run.gibba002.stderr:dumped fsmap epoch 124 2021-02-10T11:35:29.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:29.049 7fab401f5700 1 -- 172.21.2.102:0/142345142 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fab200186c0 msgr2=0x7fab2001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:29.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:29.049 7fab401f5700 1 --2- 172.21.2.102:0/142345142 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fab200186c0 0x7fab2001abc0 crc :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:29.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:29.049 7fab401f5700 1 -- 172.21.2.102:0/142345142 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fab380a2560 msgr2=0x7fab380b4fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:29.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:29.049 7fab401f5700 1 --2- 172.21.2.102:0/142345142 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fab380a2560 0x7fab380b4fc0 secure :-1 s=READY pgs=2693 cs=0 l=1 rev1=1 rx=0x7fab34006f30 tx=0x7fab3400b990).stop 2021-02-10T11:35:29.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:29.049 7fab401f5700 1 -- 172.21.2.102:0/142345142 shutdown_connections 2021-02-10T11:35:29.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:29.049 7fab401f5700 1 --2- 172.21.2.102:0/142345142 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fab380ee200 0x7fab380ee600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:29.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:29.049 7fab401f5700 1 --2- 172.21.2.102:0/142345142 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fab200186c0 0x7fab2001abc0 unknown :-1 s=CLOSED pgs=1187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:29.052 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:29.049 7fab401f5700 1 --2- 172.21.2.102:0/142345142 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fab380b55e0 0x7fab380b59e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:29.052 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:29.049 7fab401f5700 1 --2- 172.21.2.102:0/142345142 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fab380a2560 0x7fab380b4fc0 unknown :-1 s=CLOSED pgs=2693 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:29.052 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:29.049 7fab401f5700 1 -- 172.21.2.102:0/142345142 >> 172.21.2.102:0/142345142 conn(0x7fab380ac990 msgr2=0x7fab38091b30 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:29.052 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:29.049 7fab401f5700 1 -- 172.21.2.102:0/142345142 shutdown_connections 2021-02-10T11:35:29.052 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:29.049 7fab401f5700 1 -- 172.21.2.102:0/142345142 wait complete. 2021-02-10T11:35:29.261 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 124, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:35:11.628704', 'modified': '2021-02-10 11:35:13.639277', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 11781}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_11781': {'gid': 11781, 'name': 'b', 'rank': 0, 'incarnation': 121, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.113:6833/3334075470', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.113:6832', 'nonce': 3334075470}, {'type': 'v1', 'addr': '172.21.2.113:6833', 'nonce': 3334075470}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [31], 'metadata_pool': 30, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:35:29.261 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:35:29.262 DEBUG:teuthology.orchestra.run.gibba013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:35:29.661 INFO:teuthology.orchestra.run.gibba013.stdout:{ 2021-02-10T11:35:29.661 INFO:teuthology.orchestra.run.gibba013.stdout: "cluster_fsid": "68db30ea-dbce-4aef-a6df-f01c6fea0513", 2021-02-10T11:35:29.661 INFO:teuthology.orchestra.run.gibba013.stdout: "whoami": 0, 2021-02-10T11:35:29.661 INFO:teuthology.orchestra.run.gibba013.stdout: "id": 11781, 2021-02-10T11:35:29.661 INFO:teuthology.orchestra.run.gibba013.stdout: "want_state": "up:active", 2021-02-10T11:35:29.661 INFO:teuthology.orchestra.run.gibba013.stdout: "state": "up:active", 2021-02-10T11:35:29.662 INFO:teuthology.orchestra.run.gibba013.stdout: "rank_uptime": 18.029283909, 2021-02-10T11:35:29.662 INFO:teuthology.orchestra.run.gibba013.stdout: "mdsmap_epoch": 124, 2021-02-10T11:35:29.662 INFO:teuthology.orchestra.run.gibba013.stdout: "osdmap_epoch": 145, 2021-02-10T11:35:29.662 INFO:teuthology.orchestra.run.gibba013.stdout: "osdmap_epoch_barrier": 145, 2021-02-10T11:35:29.662 INFO:teuthology.orchestra.run.gibba013.stdout: "uptime": 23.681395512999998 2021-02-10T11:35:29.662 INFO:teuthology.orchestra.run.gibba013.stdout:} 2021-02-10T11:35:29.665 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "68db30ea-dbce-4aef-a6df-f01c6fea0513", "whoami": 0, "id": 11781, "want_state": "up:active", "state": "up:active", "rank_uptime": 18.029283909, "mdsmap_epoch": 124, "osdmap_epoch": 145, "osdmap_epoch_barrier": 145, "uptime": 23.681395512999998 } 2021-02-10T11:35:29.665 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2021-02-10T11:35:29.665 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2021-02-10T11:35:29.666 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.1 at ubuntu@gibba002.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.1... 2021-02-10T11:35:29.666 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:35:29.666 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec mkdir -p /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:35:30.001 DEBUG:teuthology.orchestra.run.gibba002:> sudo modprobe fuse 2021-02-10T11:35:30.346 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:35:30.346 DEBUG:teuthology.orchestra.run.gibba002:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:35:30.691 INFO:teuthology.orchestra.run.gibba002.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:35:30.691 INFO:teuthology.orchestra.run.gibba002.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:35:30.692 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:35:30.692 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:35:30.692 DEBUG:teuthology.orchestra.run.gibba002:> ls /sys/fs/fuse/connections 2021-02-10T11:35:31.027 INFO:teuthology.orchestra.run.gibba002.stdout:43 2021-02-10T11:35:31.027 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [43] 2021-02-10T11:35:31.027 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.1 /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:35:31.030 DEBUG:teuthology.orchestra.run.gibba002:> sudo modprobe fuse 2021-02-10T11:35:31.374 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:35:31.374 DEBUG:teuthology.orchestra.run.gibba002:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:35:31.439 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba002.stderr:ceph-fuse[197959]: starting ceph client 2021-02-10T11:35:31.439 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba002.stderr:2021-02-10 11:35:31.437 7f14f39aaf80 -1 init, newargv = 0x55ae8040f1a0 newargc=9 2021-02-10T11:35:31.445 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba002.stderr:ceph-fuse[197959]: starting fuse 2021-02-10T11:35:31.717 INFO:teuthology.orchestra.run.gibba002.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:35:31.717 INFO:teuthology.orchestra.run.gibba002.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:35:31.718 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:35:31.718 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:35:31.718 DEBUG:teuthology.orchestra.run.gibba002:> ls /sys/fs/fuse/connections 2021-02-10T11:35:32.057 INFO:teuthology.orchestra.run.gibba002.stdout:43 2021-02-10T11:35:32.057 INFO:teuthology.orchestra.run.gibba002.stdout:44 2021-02-10T11:35:32.057 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43, 44] 2021-02-10T11:35:32.057 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:35:32.057 DEBUG:teuthology.orchestra.run.gibba002:> sudo python3 -c ' 2021-02-10T11:35:32.057 DEBUG:teuthology.orchestra.run.gibba002:> import glob 2021-02-10T11:35:32.057 DEBUG:teuthology.orchestra.run.gibba002:> import re 2021-02-10T11:35:32.058 DEBUG:teuthology.orchestra.run.gibba002:> import os 2021-02-10T11:35:32.058 DEBUG:teuthology.orchestra.run.gibba002:> import subprocess 2021-02-10T11:35:32.058 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:35:32.058 DEBUG:teuthology.orchestra.run.gibba002:> def find_socket(client_name): 2021-02-10T11:35:32.058 DEBUG:teuthology.orchestra.run.gibba002:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2021-02-10T11:35:32.058 DEBUG:teuthology.orchestra.run.gibba002:> files = glob.glob(asok_path) 2021-02-10T11:35:32.058 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:35:32.058 DEBUG:teuthology.orchestra.run.gibba002:> # Given a non-glob path, it better be there 2021-02-10T11:35:32.058 DEBUG:teuthology.orchestra.run.gibba002:> if "*" not in asok_path: 2021-02-10T11:35:32.058 DEBUG:teuthology.orchestra.run.gibba002:> assert(len(files) == 1) 2021-02-10T11:35:32.058 DEBUG:teuthology.orchestra.run.gibba002:> return files[0] 2021-02-10T11:35:32.058 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:35:32.058 DEBUG:teuthology.orchestra.run.gibba002:> for f in files: 2021-02-10T11:35:32.058 DEBUG:teuthology.orchestra.run.gibba002:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2021-02-10T11:35:32.059 DEBUG:teuthology.orchestra.run.gibba002:> if os.path.exists("/proc/{0}".format(pid)): 2021-02-10T11:35:32.059 DEBUG:teuthology.orchestra.run.gibba002:> return f 2021-02-10T11:35:32.059 DEBUG:teuthology.orchestra.run.gibba002:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2021-02-10T11:35:32.059 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:35:32.059 DEBUG:teuthology.orchestra.run.gibba002:> print(find_socket("client.1")) 2021-02-10T11:35:32.059 DEBUG:teuthology.orchestra.run.gibba002:> ' 2021-02-10T11:35:32.420 INFO:teuthology.orchestra.run.gibba002.stdout:/var/run/ceph/ceph-client.1.197959.asok 2021-02-10T11:35:32.423 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.197959.asok 2021-02-10T11:35:32.423 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:35:32.423 DEBUG:teuthology.orchestra.run.gibba002:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.197959.asok status 2021-02-10T11:35:32.815 INFO:teuthology.orchestra.run.gibba002.stdout:{ 2021-02-10T11:35:32.815 INFO:teuthology.orchestra.run.gibba002.stdout: "metadata": { 2021-02-10T11:35:32.815 INFO:teuthology.orchestra.run.gibba002.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:35:32.815 INFO:teuthology.orchestra.run.gibba002.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:35:32.815 INFO:teuthology.orchestra.run.gibba002.stdout: "entity_id": "1", 2021-02-10T11:35:32.815 INFO:teuthology.orchestra.run.gibba002.stdout: "hostname": "gibba002", 2021-02-10T11:35:32.815 INFO:teuthology.orchestra.run.gibba002.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2021-02-10T11:35:32.815 INFO:teuthology.orchestra.run.gibba002.stdout: "pid": "197959", 2021-02-10T11:35:32.815 INFO:teuthology.orchestra.run.gibba002.stdout: "root": "/" 2021-02-10T11:35:32.815 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:35:32.815 INFO:teuthology.orchestra.run.gibba002.stdout: "dentry_count": 0, 2021-02-10T11:35:32.816 INFO:teuthology.orchestra.run.gibba002.stdout: "dentry_pinned_count": 0, 2021-02-10T11:35:32.816 INFO:teuthology.orchestra.run.gibba002.stdout: "id": 11973, 2021-02-10T11:35:32.816 INFO:teuthology.orchestra.run.gibba002.stdout: "inst": { 2021-02-10T11:35:32.816 INFO:teuthology.orchestra.run.gibba002.stdout: "name": { 2021-02-10T11:35:32.816 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "client", 2021-02-10T11:35:32.816 INFO:teuthology.orchestra.run.gibba002.stdout: "num": 11973 2021-02-10T11:35:32.816 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:35:32.816 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": { 2021-02-10T11:35:32.816 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:35:32.816 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:0", 2021-02-10T11:35:32.816 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 3147955639 2021-02-10T11:35:32.816 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:35:32.816 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:35:32.816 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": { 2021-02-10T11:35:32.817 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:35:32.817 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:0", 2021-02-10T11:35:32.817 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 3147955639 2021-02-10T11:35:32.817 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:35:32.817 INFO:teuthology.orchestra.run.gibba002.stdout: "inst_str": "client.11973 172.21.2.102:0/3147955639", 2021-02-10T11:35:32.817 INFO:teuthology.orchestra.run.gibba002.stdout: "addr_str": "172.21.2.102:0/3147955639", 2021-02-10T11:35:32.817 INFO:teuthology.orchestra.run.gibba002.stdout: "inode_count": 1, 2021-02-10T11:35:32.817 INFO:teuthology.orchestra.run.gibba002.stdout: "mds_epoch": 124, 2021-02-10T11:35:32.817 INFO:teuthology.orchestra.run.gibba002.stdout: "osd_epoch": 145, 2021-02-10T11:35:32.817 INFO:teuthology.orchestra.run.gibba002.stdout: "osd_epoch_barrier": 0, 2021-02-10T11:35:32.817 INFO:teuthology.orchestra.run.gibba002.stdout: "blacklisted": false 2021-02-10T11:35:32.817 INFO:teuthology.orchestra.run.gibba002.stdout:} 2021-02-10T11:35:32.819 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:35:32.819 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:35:32.819 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:35:33.156 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:35:33.156 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2021-02-10T11:35:33.156 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:35:33.156 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:35:33.502 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2021-02-10T11:35:33.934 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:33.916 7f5fac0d3700 1 Processor -- start 2021-02-10T11:35:33.934 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:33.916 7f5fac0d3700 1 -- start start 2021-02-10T11:35:33.934 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:33.917 7f5fac0d3700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5fa40d1880 0x7f5fa41b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:33.934 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:33.917 7f5fac0d3700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f5fa40cff80 0x7f5fa4109140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:33.934 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:33.917 7f5fac0d3700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fa4109a70 con 0x7f5fa41b0af0 2021-02-10T11:35:33.934 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:33.917 7f5fac0d3700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5fa4109760 con 0x7f5fa40cff80 2021-02-10T11:35:33.935 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:33.917 7f5fac0d3700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fa4088070 con 0x7f5fa40d1880 2021-02-10T11:35:33.935 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:33.917 7f5faa6f0700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f5fa40cff80 0x7f5fa4109140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:33.935 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:33.917 7f5faa6f0700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f5fa40cff80 0x7f5fa4109140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:37042/0 (socket says 172.21.2.102:37042) 2021-02-10T11:35:33.935 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:33.917 7f5faa6f0700 1 -- 172.21.2.102:0/957440233 learned_addr learned my addr 172.21.2.102:0/957440233 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:35:33.935 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:33.917 7f5faa6f0700 1 -- 172.21.2.102:0/957440233 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5fa40d1880 msgr2=0x7f5fa41b04d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:35:33.935 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:33.917 7f5faa6f0700 1 --2- 172.21.2.102:0/957440233 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5fa40d1880 0x7f5fa41b04d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:33.935 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:33.917 7f5faa6f0700 1 -- 172.21.2.102:0/957440233 >> v1:172.21.2.113:6789/0 conn(0x7f5fa41b0af0 legacy=0x7f5fa4108950 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:35:33.935 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:33.917 7f5faa6f0700 1 -- 172.21.2.102:0/957440233 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fa4088260 con 0x7f5fa40cff80 2021-02-10T11:35:33.935 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:33.917 7f5faa6f0700 1 --2- 172.21.2.102:0/957440233 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5fa40cff80 0x7f5fa4109140 secure :-1 s=READY pgs=2696 cs=0 l=1 rev1=1 rx=0x7f5f9c006f00 tx=0x7f5f9c009f10).ready entity=mon.0 client_cookie=82ff2d4171eda9b3 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:33.935 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:33.917 7f5fa8eed700 1 -- 172.21.2.102:0/957440233 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5f9c00b3b0 con 0x7f5fa40cff80 2021-02-10T11:35:33.935 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:33.917 7f5fa8eed700 1 -- 172.21.2.102:0/957440233 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f9c00b630 con 0x7f5fa40cff80 2021-02-10T11:35:33.935 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:33.917 7f5fa8eed700 1 -- 172.21.2.102:0/957440233 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5f9c00bb10 con 0x7f5fa40cff80 2021-02-10T11:35:33.935 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:33.917 7f5fac0d3700 1 -- 172.21.2.102:0/957440233 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5fa40cff80 msgr2=0x7f5fa4109140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:33.935 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:33.917 7f5fac0d3700 1 --2- 172.21.2.102:0/957440233 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5fa40cff80 0x7f5fa4109140 secure :-1 s=READY pgs=2696 cs=0 l=1 rev1=1 rx=0x7f5f9c006f00 tx=0x7f5f9c009f10).stop 2021-02-10T11:35:33.936 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:33.918 7f5fac0d3700 1 -- 172.21.2.102:0/957440233 shutdown_connections 2021-02-10T11:35:33.936 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:33.918 7f5fac0d3700 1 --2- 172.21.2.102:0/957440233 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5fa40cff80 0x7f5fa4109140 unknown :-1 s=CLOSED pgs=2696 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:33.936 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:33.918 7f5fac0d3700 1 --2- 172.21.2.102:0/957440233 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5fa40d1880 0x7f5fa41b04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:33.936 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:33.918 7f5fac0d3700 1 -- 172.21.2.102:0/957440233 >> 172.21.2.102:0/957440233 conn(0x7f5fa40ce430 msgr2=0x7f5fa40d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:33.936 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:33.918 7f5fac0d3700 1 -- 172.21.2.102:0/957440233 shutdown_connections 2021-02-10T11:35:33.936 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:33.918 7f5fac0d3700 1 -- 172.21.2.102:0/957440233 wait complete. 2021-02-10T11:35:33.938 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:33.932 7f5fac0d3700 1 Processor -- start 2021-02-10T11:35:33.938 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:33.932 7f5fac0d3700 1 -- start start 2021-02-10T11:35:33.938 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:33.933 7f5fac0d3700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5fa409ef20 0x7f5fa4187e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:33.938 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:33.933 7f5fac0d3700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5fa41884a0 0x7f5fa41888a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:33.938 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:33.933 7f5fac0d3700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5fa4189840 0x7f5fa4189c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:33.938 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:33.933 7f5fac0d3700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fa41b3c10 con 0x7f5fa4189840 2021-02-10T11:35:33.938 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:33.933 7f5fac0d3700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fa41964f0 con 0x7f5fa41884a0 2021-02-10T11:35:33.938 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:33.933 7f5fac0d3700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fa4188ec0 con 0x7f5fa409ef20 2021-02-10T11:35:33.938 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:33.933 7f5faa6f0700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5fa4189840 0x7f5fa4189c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:33.938 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:33.933 7f5faa6f0700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5fa4189840 0x7f5fa4189c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:37046/0 (socket says 172.21.2.102:37046) 2021-02-10T11:35:33.938 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:33.933 7f5faa6f0700 1 -- 172.21.2.102:0/487998575 learned_addr learned my addr 172.21.2.102:0/487998575 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:35:33.938 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:33.933 7f5faa6f0700 1 -- 172.21.2.102:0/487998575 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5fa409ef20 msgr2=0x7f5fa4187e80 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:35:33.938 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:33.933 7f5faa6f0700 1 --2- 172.21.2.102:0/487998575 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5fa409ef20 0x7f5fa4187e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:33.939 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:33.933 7f5faa6f0700 1 -- 172.21.2.102:0/487998575 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5fa41884a0 msgr2=0x7f5fa41888a0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:35:33.939 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:33.933 7f5faa6f0700 1 --2- 172.21.2.102:0/487998575 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5fa41884a0 0x7f5fa41888a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:33.939 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:33.933 7f5faa6f0700 1 -- 172.21.2.102:0/487998575 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fa409f320 con 0x7f5fa4189840 2021-02-10T11:35:33.939 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:33.933 7f5faa6f0700 1 --2- 172.21.2.102:0/487998575 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5fa4189840 0x7f5fa4189c40 secure :-1 s=READY pgs=2697 cs=0 l=1 rev1=1 rx=0x7f5f9c00b9b0 tx=0x7f5f9c00bd50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:33.939 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:33.933 7f5f9a7fc700 1 -- 172.21.2.102:0/487998575 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5f9c003030 con 0x7f5fa4189840 2021-02-10T11:35:33.939 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:33.933 7f5f9a7fc700 1 -- 172.21.2.102:0/487998575 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f9c00c2f0 con 0x7f5fa4189840 2021-02-10T11:35:33.939 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:33.933 7f5fac0d3700 1 -- 172.21.2.102:0/487998575 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fa409f320 con 0x7f5fa4189840 2021-02-10T11:35:33.939 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:33.933 7f5f9a7fc700 1 -- 172.21.2.102:0/487998575 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5f9c00c790 con 0x7f5fa4189840 2021-02-10T11:35:33.939 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:33.934 7f5fac0d3700 1 -- 172.21.2.102:0/487998575 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fa409f320 con 0x7f5fa4189840 2021-02-10T11:35:33.939 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:33.934 7f5f9a7fc700 1 -- 172.21.2.102:0/487998575 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f5f9c009450 con 0x7f5fa4189840 2021-02-10T11:35:33.939 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:33.934 7f5f9a7fc700 1 --2- 172.21.2.102:0/487998575 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5f8c018730 0x7f5f8c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:33.939 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:33.934 7f5f9a7fc700 1 -- 172.21.2.102:0/487998575 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f5f9c032b80 con 0x7f5fa4189840 2021-02-10T11:35:33.939 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:33.934 7f5fa9eef700 1 --2- 172.21.2.102:0/487998575 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5f8c018730 0x7f5f8c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:33.940 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:33.935 7f5fa9eef700 1 --2- 172.21.2.102:0/487998575 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5f8c018730 0x7f5f8c01abc0 crc :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:33.968 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:33.966 7f5fac0d3700 1 -- 172.21.2.102:0/487998575 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fa41b3c10 con 0x7f5fa4189840 2021-02-10T11:35:33.970 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:33.969 7f5f9a7fc700 1 -- 172.21.2.102:0/487998575 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f5f9c000f90 con 0x7f5fa4189840 2021-02-10T11:35:34.109 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:34.107 7f5fac0d3700 1 -- 172.21.2.102:0/487998575 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f5fa41b3c10 con 0x7f5fa4189840 2021-02-10T11:35:34.163 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:34.161 7f5f9a7fc700 1 -- 172.21.2.102:0/487998575 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v125) v1 ==== 136+0+0 (secure 0 0 0) 0x7f5f9c009450 con 0x7f5fa4189840 2021-02-10T11:35:34.175 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:34.173 7f5fac0d3700 1 -- 172.21.2.102:0/487998575 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5f8c018730 msgr2=0x7f5f8c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:34.175 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:34.173 7f5fac0d3700 1 --2- 172.21.2.102:0/487998575 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5f8c018730 0x7f5f8c01abc0 crc :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:34.175 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:34.173 7f5fac0d3700 1 -- 172.21.2.102:0/487998575 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5fa4189840 msgr2=0x7f5fa4189c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:34.175 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:34.173 7f5fac0d3700 1 --2- 172.21.2.102:0/487998575 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5fa4189840 0x7f5fa4189c40 secure :-1 s=READY pgs=2697 cs=0 l=1 rev1=1 rx=0x7f5f9c00b9b0 tx=0x7f5f9c00bd50).stop 2021-02-10T11:35:34.176 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:34.173 7f5fac0d3700 1 -- 172.21.2.102:0/487998575 shutdown_connections 2021-02-10T11:35:34.176 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:34.173 7f5fac0d3700 1 --2- 172.21.2.102:0/487998575 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5fa4189840 0x7f5fa4189c40 unknown :-1 s=CLOSED pgs=2697 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:34.176 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:34.173 7f5fac0d3700 1 --2- 172.21.2.102:0/487998575 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5f8c018730 0x7f5f8c01abc0 unknown :-1 s=CLOSED pgs=1188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:34.176 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:34.173 7f5fac0d3700 1 --2- 172.21.2.102:0/487998575 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5fa41884a0 0x7f5fa41888a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:34.176 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:34.173 7f5fac0d3700 1 --2- 172.21.2.102:0/487998575 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5fa409ef20 0x7f5fa4187e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:34.176 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:34.173 7f5fac0d3700 1 -- 172.21.2.102:0/487998575 >> 172.21.2.102:0/487998575 conn(0x7f5fa419f970 msgr2=0x7f5fa41a61d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:34.176 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:34.174 7f5fac0d3700 1 -- 172.21.2.102:0/487998575 shutdown_connections 2021-02-10T11:35:34.176 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:34.174 7f5fac0d3700 1 -- 172.21.2.102:0/487998575 wait complete. 2021-02-10T11:35:34.184 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs volume ls 2021-02-10T11:35:34.610 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:34.592 7fecfb0a3700 1 Processor -- start 2021-02-10T11:35:34.611 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:34.592 7fecfb0a3700 1 -- start start 2021-02-10T11:35:34.611 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:34.592 7fecfb0a3700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fecf40d0340 0x7fecf40ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:34.611 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:34.592 7fecfb0a3700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fecf40d1c70 0x7fecf40efc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:34.611 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:34.592 7fecfb0a3700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fecf40f0b10 con 0x7fecf40d1870 2021-02-10T11:35:34.611 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:34.592 7fecfb0a3700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fecf40e5ed0 con 0x7fecf40d1c70 2021-02-10T11:35:34.611 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:34.592 7fecfb0a3700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fecf41b0be0 con 0x7fecf40d0340 2021-02-10T11:35:34.611 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:34.593 7fecf8ebf700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fecf40d1870 0x7fecf41b03f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:57510/0 (socket says 172.21.2.102:57510) 2021-02-10T11:35:34.611 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:34.593 7fecf8ebf700 1 -- 172.21.2.102:0/117072771 learned_addr learned my addr 172.21.2.102:0/117072771 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:35:34.611 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:34.593 7fecf96c0700 1 --2- 172.21.2.102:0/117072771 >> v2:172.21.2.102:3300/0 conn(0x7fecf40d1c70 0x7fecf40efc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:34.611 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:34.593 7fecf96c0700 1 -- 172.21.2.102:0/117072771 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fecf40d0340 msgr2=0x7fecf40ef640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:34.612 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:34.593 7fecf96c0700 1 --2- 172.21.2.102:0/117072771 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fecf40d0340 0x7fecf40ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:34.612 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:34.593 7fecf96c0700 1 -- 172.21.2.102:0/117072771 >> v1:172.21.2.102:6789/0 conn(0x7fecf40d1870 legacy=0x7fecf41b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:34.612 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:34.593 7fecf96c0700 1 -- 172.21.2.102:0/117072771 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecf40f20a0 con 0x7fecf40d1c70 2021-02-10T11:35:34.612 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:34.593 7fecf96c0700 1 --2- 172.21.2.102:0/117072771 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fecf40d1c70 0x7fecf40efc60 secure :-1 s=READY pgs=2699 cs=0 l=1 rev1=1 rx=0x7fecec006ab0 tx=0x7fecec009b40).ready entity=mon.0 client_cookie=346621b368c5516f server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:34.612 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:34.593 7fecf37fe700 1 -- 172.21.2.102:0/117072771 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fecec0022e0 con 0x7fecf40d1c70 2021-02-10T11:35:34.612 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:34.594 7fecf37fe700 1 -- 172.21.2.102:0/117072771 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fecec002560 con 0x7fecf40d1c70 2021-02-10T11:35:34.612 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:34.594 7fecf37fe700 1 -- 172.21.2.102:0/117072771 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fecec0029f0 con 0x7fecf40d1c70 2021-02-10T11:35:34.612 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:34.594 7fecfb0a3700 1 -- 172.21.2.102:0/117072771 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fecf40d1c70 msgr2=0x7fecf40efc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:34.612 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:34.594 7fecfb0a3700 1 --2- 172.21.2.102:0/117072771 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fecf40d1c70 0x7fecf40efc60 secure :-1 s=READY pgs=2699 cs=0 l=1 rev1=1 rx=0x7fecec006ab0 tx=0x7fecec009b40).stop 2021-02-10T11:35:34.612 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:34.594 7fecfb0a3700 1 -- 172.21.2.102:0/117072771 shutdown_connections 2021-02-10T11:35:34.612 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:34.594 7fecfb0a3700 1 --2- 172.21.2.102:0/117072771 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fecf40d1c70 0x7fecf40efc60 unknown :-1 s=CLOSED pgs=2699 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:34.612 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:34.594 7fecfb0a3700 1 --2- 172.21.2.102:0/117072771 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fecf40d0340 0x7fecf40ef640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:34.612 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:34.594 7fecfb0a3700 1 -- 172.21.2.102:0/117072771 >> 172.21.2.102:0/117072771 conn(0x7fecf40ce450 msgr2=0x7fecf40d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:34.613 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:34.594 7fecfb0a3700 1 -- 172.21.2.102:0/117072771 shutdown_connections 2021-02-10T11:35:34.613 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:34.594 7fecfb0a3700 1 -- 172.21.2.102:0/117072771 wait complete. 2021-02-10T11:35:34.614 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:34.609 7fecfb0a3700 1 Processor -- start 2021-02-10T11:35:34.614 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:34.609 7fecfb0a3700 1 -- start start 2021-02-10T11:35:34.614 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:34.609 7fecfb0a3700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fecf41908b0 0x7fecf41a4440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:34.614 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:34.609 7fecfb0a3700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fecf40bd920 0x7fecf40bdd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:34.614 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:34.609 7fecfb0a3700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fecf408d510 0x7fecf408d910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:34.614 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:34.609 7fecfb0a3700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecf408cf90 con 0x7fecf41908b0 2021-02-10T11:35:34.614 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:34.609 7fecfb0a3700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecf40be340 con 0x7fecf408d510 2021-02-10T11:35:34.615 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:34.609 7fecfb0a3700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fecf40be5f0 con 0x7fecf40bd920 2021-02-10T11:35:34.615 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:34.610 7fecf8ebf700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fecf41908b0 0x7fecf41a4440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:34.615 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:34.610 7fecf8ebf700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fecf41908b0 0x7fecf41a4440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:37064/0 (socket says 172.21.2.102:37064) 2021-02-10T11:35:34.615 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:34.610 7fecf8ebf700 1 -- 172.21.2.102:0/2353889088 learned_addr learned my addr 172.21.2.102:0/2353889088 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:35:34.615 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:34.610 7fecf8ebf700 1 -- 172.21.2.102:0/2353889088 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fecf40bd920 msgr2=0x7fecf40bdd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:34.615 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:34.610 7fecf8ebf700 1 --2- 172.21.2.102:0/2353889088 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fecf40bd920 0x7fecf40bdd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:34.615 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:34.610 7fecf8ebf700 1 -- 172.21.2.102:0/2353889088 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fecf408d510 msgr2=0x7fecf408d910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:34.615 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:34.610 7fecf8ebf700 1 --2- 172.21.2.102:0/2353889088 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fecf408d510 0x7fecf408d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:34.615 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:34.610 7fecf8ebf700 1 -- 172.21.2.102:0/2353889088 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecf41a4a60 con 0x7fecf41908b0 2021-02-10T11:35:34.615 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:34.610 7fecf8ebf700 1 --2- 172.21.2.102:0/2353889088 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fecf41908b0 0x7fecf41a4440 secure :-1 s=READY pgs=2700 cs=0 l=1 rev1=1 rx=0x7fecdc001410 tx=0x7fecdc00a950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:34.615 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:34.610 7fecf17fa700 1 -- 172.21.2.102:0/2353889088 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fecdc00ad40 con 0x7fecf41908b0 2021-02-10T11:35:34.615 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:34.611 7fecf17fa700 1 -- 172.21.2.102:0/2353889088 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fecdc000c90 con 0x7fecf41908b0 2021-02-10T11:35:34.615 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:34.611 7fecf17fa700 1 -- 172.21.2.102:0/2353889088 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fecdc004590 con 0x7fecf41908b0 2021-02-10T11:35:34.616 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:34.611 7fecfb0a3700 1 -- 172.21.2.102:0/2353889088 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fecf41a4a60 con 0x7fecf41908b0 2021-02-10T11:35:34.616 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:34.611 7fecfb0a3700 1 -- 172.21.2.102:0/2353889088 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fecf408df30 con 0x7fecf41908b0 2021-02-10T11:35:34.616 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:34.611 7fecf17fa700 1 -- 172.21.2.102:0/2353889088 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fecdc0043a0 con 0x7fecf41908b0 2021-02-10T11:35:34.616 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:34.611 7fecf17fa700 1 --2- 172.21.2.102:0/2353889088 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fecd80186c0 0x7fecd801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:34.616 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:34.611 7fecf17fa700 1 -- 172.21.2.102:0/2353889088 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fecdc032590 con 0x7fecf41908b0 2021-02-10T11:35:34.616 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:34.611 7fecf3fff700 1 --2- 172.21.2.102:0/2353889088 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fecd80186c0 0x7fecd801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:34.616 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:34.612 7fecf3fff700 1 --2- 172.21.2.102:0/2353889088 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fecd80186c0 0x7fecd801abc0 crc :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:34.644 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:34.642 7fecfb0a3700 1 -- 172.21.2.102:0/2353889088 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fecf40ebc50 con 0x7fecf41908b0 2021-02-10T11:35:34.647 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:34.645 7fecf17fa700 1 -- 172.21.2.102:0/2353889088 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fecdc000ed0 con 0x7fecf41908b0 2021-02-10T11:35:34.779 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:34.777 7fecfb0a3700 1 -- 172.21.2.102:0/2353889088 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs volume ls", "target": ["mgr", ""]}) v1 -- 0x7fecf408df30 con 0x7fecd80186c0 2021-02-10T11:35:34.780 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:34.778 7fecf17fa700 1 -- 172.21.2.102:0/2353889088 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+41 (crc 0 0 0) 0x7fece4000ce0 con 0x7fecd80186c0 2021-02-10T11:35:34.780 INFO:teuthology.orchestra.run.gibba002.stdout:[ 2021-02-10T11:35:34.780 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:35:34.780 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "cephfs" 2021-02-10T11:35:34.780 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:35:34.780 INFO:teuthology.orchestra.run.gibba002.stdout:] 2021-02-10T11:35:34.781 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:34.779 7fecfb0a3700 1 -- 172.21.2.102:0/2353889088 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fecd80186c0 msgr2=0x7fecd801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:34.782 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:34.779 7fecfb0a3700 1 --2- 172.21.2.102:0/2353889088 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fecd80186c0 0x7fecd801abc0 crc :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:34.782 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:34.779 7fecfb0a3700 1 -- 172.21.2.102:0/2353889088 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fecf41908b0 msgr2=0x7fecf41a4440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:34.782 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:34.779 7fecfb0a3700 1 --2- 172.21.2.102:0/2353889088 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fecf41908b0 0x7fecf41a4440 secure :-1 s=READY pgs=2700 cs=0 l=1 rev1=1 rx=0x7fecdc001410 tx=0x7fecdc00a950).stop 2021-02-10T11:35:34.782 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:34.779 7fecfb0a3700 1 -- 172.21.2.102:0/2353889088 shutdown_connections 2021-02-10T11:35:34.782 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:34.779 7fecfb0a3700 1 --2- 172.21.2.102:0/2353889088 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fecd80186c0 0x7fecd801abc0 unknown :-1 s=CLOSED pgs=1189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:34.782 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:34.779 7fecfb0a3700 1 --2- 172.21.2.102:0/2353889088 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fecf408d510 0x7fecf408d910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:34.782 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:34.779 7fecfb0a3700 1 --2- 172.21.2.102:0/2353889088 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fecf40bd920 0x7fecf40bdd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:34.782 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:34.779 7fecfb0a3700 1 --2- 172.21.2.102:0/2353889088 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fecf41908b0 0x7fecf41a4440 unknown :-1 s=CLOSED pgs=2700 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:34.782 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:34.779 7fecfb0a3700 1 -- 172.21.2.102:0/2353889088 >> 172.21.2.102:0/2353889088 conn(0x7fecf408f030 msgr2=0x7fecf40b5e80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:34.782 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:34.780 7fecfb0a3700 1 -- 172.21.2.102:0/2353889088 shutdown_connections 2021-02-10T11:35:34.782 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:34.780 7fecfb0a3700 1 -- 172.21.2.102:0/2353889088 wait complete. 2021-02-10T11:35:34.793 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config set mon mon_allow_pool_delete True 2021-02-10T11:35:35.229 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:35.210 7f7fdcae3700 1 Processor -- start 2021-02-10T11:35:35.229 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:35.210 7f7fdcae3700 1 -- start start 2021-02-10T11:35:35.230 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:35.211 7f7fdcae3700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f7fd80d1c70 0x7f7fd80cef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:35.230 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:35.211 7f7fdcae3700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7fd80cf580 0x7f7fd80cf980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:35.230 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:35.211 7f7fdcae3700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fd80fe1c0 con 0x7f7fd80d1870 2021-02-10T11:35:35.230 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:35.211 7f7fdcae3700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7fd81b0cb0 con 0x7f7fd80d1c70 2021-02-10T11:35:35.230 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:35.211 7f7fdcae3700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fd8109a60 con 0x7f7fd80cf580 2021-02-10T11:35:35.230 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:35.211 7f7fd685e700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f7fd80d1870 0x7f7fd81b04c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:57524/0 (socket says 172.21.2.102:57524) 2021-02-10T11:35:35.230 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:35.211 7f7fd685e700 1 -- 172.21.2.102:0/1188140386 learned_addr learned my addr 172.21.2.102:0/1188140386 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:35:35.230 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:35.211 7f7fd585c700 1 -- 172.21.2.102:0/1188140386 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 12495572 0 0) 0x7f7fc8001180 con 0x7f7fd80d1870 2021-02-10T11:35:35.230 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:35.211 7f7fd585c700 1 -- 172.21.2.102:0/1188140386 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7fc4002fe0 con 0x7f7fd80d1870 2021-02-10T11:35:35.230 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:35.211 7f7fd705f700 1 --2- 172.21.2.102:0/1188140386 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7fd80cf580 0x7f7fd80cf980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:35.230 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:35.211 7f7fd605d700 1 --2- 172.21.2.102:0/1188140386 >> v2:172.21.2.113:3300/0 conn(0x7f7fd80d1c70 0x7f7fd80cef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:35.230 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:35.211 7f7fd585c700 1 -- 172.21.2.102:0/1188140386 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2231195483 0 0) 0x7f7fc8001180 con 0x7f7fd80d1870 2021-02-10T11:35:35.231 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:35.212 7f7fd585c700 1 -- 172.21.2.102:0/1188140386 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7fd80cf580 msgr2=0x7f7fd80cf980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:35.231 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:35.212 7f7fd585c700 1 --2- 172.21.2.102:0/1188140386 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7fd80cf580 0x7f7fd80cf980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:35.231 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:35.212 7f7fd585c700 1 -- 172.21.2.102:0/1188140386 >> v2:172.21.2.113:3300/0 conn(0x7f7fd80d1c70 msgr2=0x7f7fd80cef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:35.231 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:35.212 7f7fd585c700 1 --2- 172.21.2.102:0/1188140386 >> v2:172.21.2.113:3300/0 conn(0x7f7fd80d1c70 0x7f7fd80cef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:35.231 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:35.212 7f7fd585c700 1 -- 172.21.2.102:0/1188140386 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fd8088070 con 0x7f7fd80d1870 2021-02-10T11:35:35.231 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:35.212 7f7fd585c700 1 -- 172.21.2.102:0/1188140386 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7f7fc8001680 con 0x7f7fd80d1870 2021-02-10T11:35:35.231 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:35.212 7f7fd585c700 1 -- 172.21.2.102:0/1188140386 >> v1:172.21.2.102:6789/0 conn(0x7f7fd80d1870 legacy=0x7f7fd81b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:35.231 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:35.212 7f7fd585c700 1 --2- 172.21.2.102:0/1188140386 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7fc4003b90 0x7f7fc4006040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:35.231 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:35.212 7f7fd585c700 1 --2- 172.21.2.102:0/1188140386 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7fc4006860 0x7f7fc4008d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:35.231 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:35.212 7f7fd585c700 1 --2- 172.21.2.102:0/1188140386 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7fc4009640 0x7f7fc400bb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:35.231 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:35.212 7f7fd585c700 1 -- 172.21.2.102:0/1188140386 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fc400c380 con 0x7f7fc4003b90 2021-02-10T11:35:35.231 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:35.212 7f7fd585c700 1 -- 172.21.2.102:0/1188140386 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fc400c5a0 con 0x7f7fc4006860 2021-02-10T11:35:35.231 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:35.212 7f7fd585c700 1 -- 172.21.2.102:0/1188140386 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fc40507d0 con 0x7f7fc4009640 2021-02-10T11:35:35.231 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:35.212 7f7fd685e700 1 --2- 172.21.2.102:0/1188140386 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7fc4003b90 0x7f7fc4006040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:35.232 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:35.212 7f7fd605d700 1 --2- 172.21.2.102:0/1188140386 >> v2:172.21.2.113:3300/0 conn(0x7f7fd80d1c70 0x7f7fd80cef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:35:35.232 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:35.212 7f7fd705f700 1 --2- 172.21.2.102:0/1188140386 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7fc4009640 0x7f7fc400bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:35.232 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:35.212 7f7fd685e700 1 -- 172.21.2.102:0/1188140386 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7fc4009640 msgr2=0x7f7fc400bb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:35.232 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:35.212 7f7fd685e700 1 --2- 172.21.2.102:0/1188140386 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7fc4009640 0x7f7fc400bb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:35.232 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:35.212 7f7fd685e700 1 -- 172.21.2.102:0/1188140386 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7fc4006860 msgr2=0x7f7fc4008d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:35.232 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:35.212 7f7fd685e700 1 --2- 172.21.2.102:0/1188140386 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7fc4006860 0x7f7fc4008d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:35.232 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:35.212 7f7fd685e700 1 -- 172.21.2.102:0/1188140386 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fc4050a30 con 0x7f7fc4003b90 2021-02-10T11:35:35.232 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:35.212 7f7fd705f700 1 --2- 172.21.2.102:0/1188140386 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7fc4009640 0x7f7fc400bb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:35:35.232 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:35.212 7f7fd685e700 1 --2- 172.21.2.102:0/1188140386 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7fc4003b90 0x7f7fc4006040 secure :-1 s=READY pgs=2702 cs=0 l=1 rev1=1 rx=0x7f7fc8007970 tx=0x7f7fc8007d10).ready entity=mon.0 client_cookie=46849115b8913fa5 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:35.232 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:35.212 7f7fd585c700 1 -- 172.21.2.102:0/1188140386 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7fc80070c0 con 0x7f7fc4003b90 2021-02-10T11:35:35.232 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:35.212 7f7fd585c700 1 -- 172.21.2.102:0/1188140386 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fc80043a0 con 0x7f7fc4003b90 2021-02-10T11:35:35.232 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:35.212 7f7fd585c700 1 -- 172.21.2.102:0/1188140386 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7fc8004880 con 0x7f7fc4003b90 2021-02-10T11:35:35.232 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:35.212 7f7fdcae3700 1 -- 172.21.2.102:0/1188140386 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7fc4003b90 msgr2=0x7f7fc4006040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:35.233 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:35.212 7f7fdcae3700 1 --2- 172.21.2.102:0/1188140386 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7fc4003b90 0x7f7fc4006040 secure :-1 s=READY pgs=2702 cs=0 l=1 rev1=1 rx=0x7f7fc8007970 tx=0x7f7fc8007d10).stop 2021-02-10T11:35:35.233 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:35.212 7f7fd705f700 1 -- 172.21.2.102:0/1188140386 reap_dead start 2021-02-10T11:35:35.233 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:35.213 7f7fdcae3700 1 -- 172.21.2.102:0/1188140386 shutdown_connections 2021-02-10T11:35:35.233 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:35.213 7f7fdcae3700 1 -- 172.21.2.102:0/1188140386 >> 172.21.2.102:0/1188140386 conn(0x7f7fd80ce420 msgr2=0x7f7fd80d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:35.233 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:35.213 7f7fdcae3700 1 -- 172.21.2.102:0/1188140386 shutdown_connections 2021-02-10T11:35:35.233 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:35.213 7f7fdcae3700 1 -- 172.21.2.102:0/1188140386 wait complete. 2021-02-10T11:35:35.235 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:35.228 7f7fdcae3700 1 Processor -- start 2021-02-10T11:35:35.235 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:35.228 7f7fdcae3700 1 -- start start 2021-02-10T11:35:35.235 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:35.228 7f7fdcae3700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7fd80926d0 0x7f7fd80b6820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:35.235 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:35.228 7f7fdcae3700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7fd80c3840 0x7f7fd8093270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:35.235 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:35.228 7f7fdcae3700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7fd80c3c40 0x7f7fd8093890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:35.235 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:35.228 7f7fdcae3700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fd8092ad0 con 0x7f7fd80c3c40 2021-02-10T11:35:35.235 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:35.228 7f7fdcae3700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fd8092cf0 con 0x7f7fd80926d0 2021-02-10T11:35:35.235 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:35.228 7f7fdcae3700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fd80b6f80 con 0x7f7fd80c3840 2021-02-10T11:35:35.236 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:35.228 7f7fd705f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7fd80c3c40 0x7f7fd8093890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:35.236 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:35.228 7f7fd705f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7fd80c3c40 0x7f7fd8093890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:37082/0 (socket says 172.21.2.102:37082) 2021-02-10T11:35:35.236 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:35.229 7f7fd705f700 1 -- 172.21.2.102:0/2954965946 learned_addr learned my addr 172.21.2.102:0/2954965946 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:35:35.236 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:35.229 7f7fd605d700 1 --2- 172.21.2.102:0/2954965946 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7fd80c3840 0x7f7fd8093270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:35.236 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:35.229 7f7fd685e700 1 --2- 172.21.2.102:0/2954965946 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7fd80926d0 0x7f7fd80b6820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:35.236 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:35.229 7f7fd705f700 1 -- 172.21.2.102:0/2954965946 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7fd80c3840 msgr2=0x7f7fd8093270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:35.236 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:35.229 7f7fd705f700 1 --2- 172.21.2.102:0/2954965946 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7fd80c3840 0x7f7fd8093270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:35.236 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:35.229 7f7fd705f700 1 -- 172.21.2.102:0/2954965946 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7fd80926d0 msgr2=0x7f7fd80b6820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:35.236 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:35.229 7f7fd705f700 1 --2- 172.21.2.102:0/2954965946 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7fd80926d0 0x7f7fd80b6820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:35.236 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:35.229 7f7fd705f700 1 -- 172.21.2.102:0/2954965946 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fd8093eb0 con 0x7f7fd80c3c40 2021-02-10T11:35:35.236 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:35.229 7f7fd685e700 1 --2- 172.21.2.102:0/2954965946 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7fd80926d0 0x7f7fd80b6820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:35:35.236 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:35.229 7f7fd705f700 1 --2- 172.21.2.102:0/2954965946 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7fd80c3c40 0x7f7fd8093890 secure :-1 s=READY pgs=2703 cs=0 l=1 rev1=1 rx=0x7f7fd0009980 tx=0x7f7fd000bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:35.236 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:35.230 7f7fcf7fe700 1 -- 172.21.2.102:0/2954965946 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7fd0006990 con 0x7f7fd80c3c40 2021-02-10T11:35:35.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:35.230 7f7fcf7fe700 1 -- 172.21.2.102:0/2954965946 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fd000c2f0 con 0x7f7fd80c3c40 2021-02-10T11:35:35.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:35.230 7f7fcf7fe700 1 -- 172.21.2.102:0/2954965946 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7fd000c810 con 0x7f7fd80c3c40 2021-02-10T11:35:35.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:35.230 7f7fdcae3700 1 -- 172.21.2.102:0/2954965946 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fd8093eb0 con 0x7f7fd80c3c40 2021-02-10T11:35:35.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:35.230 7f7fdcae3700 1 -- 172.21.2.102:0/2954965946 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fd8094440 con 0x7f7fd80c3c40 2021-02-10T11:35:35.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:35.230 7f7fcf7fe700 1 -- 172.21.2.102:0/2954965946 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f7fd0000df0 con 0x7f7fd80c3c40 2021-02-10T11:35:35.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:35.230 7f7fcf7fe700 1 --2- 172.21.2.102:0/2954965946 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7fbc0186c0 0x7f7fbc01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:35.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:35.230 7f7fcf7fe700 1 -- 172.21.2.102:0/2954965946 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f7fd0032940 con 0x7f7fd80c3c40 2021-02-10T11:35:35.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:35.231 7f7fd685e700 1 --2- 172.21.2.102:0/2954965946 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7fbc0186c0 0x7f7fbc01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:35.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:35.231 7f7fd685e700 1 --2- 172.21.2.102:0/2954965946 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7fbc0186c0 0x7f7fbc01abc0 crc :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:35.262 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:35.260 7f7fdcae3700 1 -- 172.21.2.102:0/2954965946 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fd8082a70 con 0x7f7fd80c3c40 2021-02-10T11:35:35.265 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:35.263 7f7fcf7fe700 1 -- 172.21.2.102:0/2954965946 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f7fd001f680 con 0x7f7fd80c3c40 2021-02-10T11:35:35.413 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:35.411 7f7fdcae3700 1 -- 172.21.2.102:0/2954965946 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f7fd8082a70 con 0x7f7fd80c3c40 2021-02-10T11:35:35.413 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:35.411 7f7fcf7fe700 1 -- 172.21.2.102:0/2954965946 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v36)=0 v36) v1 ==== 122+0+0 (secure 0 0 0) 0x7f7fd0000df0 con 0x7f7fd80c3c40 2021-02-10T11:35:35.415 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:35.413 7f7fdcae3700 1 -- 172.21.2.102:0/2954965946 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7fbc0186c0 msgr2=0x7f7fbc01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:35.415 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:35.413 7f7fdcae3700 1 --2- 172.21.2.102:0/2954965946 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7fbc0186c0 0x7f7fbc01abc0 crc :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:35.415 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:35.413 7f7fdcae3700 1 -- 172.21.2.102:0/2954965946 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7fd80c3c40 msgr2=0x7f7fd8093890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:35.415 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:35.413 7f7fdcae3700 1 --2- 172.21.2.102:0/2954965946 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7fd80c3c40 0x7f7fd8093890 secure :-1 s=READY pgs=2703 cs=0 l=1 rev1=1 rx=0x7f7fd0009980 tx=0x7f7fd000bcd0).stop 2021-02-10T11:35:35.415 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:35.413 7f7fdcae3700 1 -- 172.21.2.102:0/2954965946 shutdown_connections 2021-02-10T11:35:35.416 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:35.413 7f7fdcae3700 1 --2- 172.21.2.102:0/2954965946 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7fd80c3c40 0x7f7fd8093890 unknown :-1 s=CLOSED pgs=2703 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:35.416 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:35.413 7f7fdcae3700 1 --2- 172.21.2.102:0/2954965946 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7fd80c3840 0x7f7fd8093270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:35.416 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:35.413 7f7fdcae3700 1 --2- 172.21.2.102:0/2954965946 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7fbc0186c0 0x7f7fbc01abc0 unknown :-1 s=CLOSED pgs=1190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:35.416 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:35.413 7f7fdcae3700 1 --2- 172.21.2.102:0/2954965946 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7fd80926d0 0x7f7fd80b6820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:35.416 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:35.413 7f7fdcae3700 1 -- 172.21.2.102:0/2954965946 >> 172.21.2.102:0/2954965946 conn(0x7f7fd809ffd0 msgr2=0x7f7fd8087030 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:35.416 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:35.413 7f7fdcae3700 1 -- 172.21.2.102:0/2954965946 shutdown_connections 2021-02-10T11:35:35.416 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:35.414 7f7fdcae3700 1 -- 172.21.2.102:0/2954965946 wait complete. 2021-02-10T11:35:35.423 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolumegroup rm cephfs non_existent_group 2021-02-10T11:35:35.856 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:35.837 7fbe6f72d700 1 Processor -- start 2021-02-10T11:35:35.856 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:35.837 7fbe6f72d700 1 -- start start 2021-02-10T11:35:35.856 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:35.838 7fbe6f72d700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbe680d1c70 0x7fbe680cef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:35.856 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:35.838 7fbe6f72d700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fbe680cf580 0x7fbe680cf980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:35.856 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:35.838 7fbe6f72d700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe680fe1c0 con 0x7fbe680d1870 2021-02-10T11:35:35.856 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:35.838 7fbe6f72d700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe681b0cb0 con 0x7fbe680cf580 2021-02-10T11:35:35.857 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:35.838 7fbe6f72d700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe68109a60 con 0x7fbe680d1c70 2021-02-10T11:35:35.857 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:35.838 7fbe6d549700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fbe680d1870 0x7fbe681b04c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:57544/0 (socket says 172.21.2.102:57544) 2021-02-10T11:35:35.857 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:35.838 7fbe6d549700 1 -- 172.21.2.102:0/1004512562 learned_addr learned my addr 172.21.2.102:0/1004512562 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:35:35.857 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:35.838 7fbe6dd4a700 1 --2- 172.21.2.102:0/1004512562 >> v2:172.21.2.102:3300/0 conn(0x7fbe680cf580 0x7fbe680cf980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:35.857 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:35.839 7fbe6dd4a700 1 -- 172.21.2.102:0/1004512562 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbe680d1c70 msgr2=0x7fbe680cef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:35.857 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:35.839 7fbe6dd4a700 1 --2- 172.21.2.102:0/1004512562 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbe680d1c70 0x7fbe680cef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:35.857 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:35.839 7fbe6dd4a700 1 -- 172.21.2.102:0/1004512562 >> v1:172.21.2.102:6789/0 conn(0x7fbe680d1870 legacy=0x7fbe681b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:35.857 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:35.839 7fbe6dd4a700 1 -- 172.21.2.102:0/1004512562 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe68088070 con 0x7fbe680cf580 2021-02-10T11:35:35.857 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:35.839 7fbe6dd4a700 1 --2- 172.21.2.102:0/1004512562 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbe680cf580 0x7fbe680cf980 secure :-1 s=READY pgs=2705 cs=0 l=1 rev1=1 rx=0x7fbe60006e30 tx=0x7fbe60009e60).ready entity=mon.0 client_cookie=43ecf65de4925bcc server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:35.857 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:35.839 7fbe5bfff700 1 -- 172.21.2.102:0/1004512562 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbe60006a50 con 0x7fbe680cf580 2021-02-10T11:35:35.857 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:35.839 7fbe5bfff700 1 -- 172.21.2.102:0/1004512562 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe6000b300 con 0x7fbe680cf580 2021-02-10T11:35:35.858 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:35.839 7fbe5bfff700 1 -- 172.21.2.102:0/1004512562 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbe6000b760 con 0x7fbe680cf580 2021-02-10T11:35:35.858 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:35.839 7fbe6f72d700 1 -- 172.21.2.102:0/1004512562 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbe680cf580 msgr2=0x7fbe680cf980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:35.858 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:35.839 7fbe6f72d700 1 --2- 172.21.2.102:0/1004512562 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbe680cf580 0x7fbe680cf980 secure :-1 s=READY pgs=2705 cs=0 l=1 rev1=1 rx=0x7fbe60006e30 tx=0x7fbe60009e60).stop 2021-02-10T11:35:35.858 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:35.840 7fbe6f72d700 1 -- 172.21.2.102:0/1004512562 shutdown_connections 2021-02-10T11:35:35.858 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:35.840 7fbe6f72d700 1 --2- 172.21.2.102:0/1004512562 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbe680cf580 0x7fbe680cf980 unknown :-1 s=CLOSED pgs=2705 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:35.858 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:35.840 7fbe6f72d700 1 --2- 172.21.2.102:0/1004512562 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbe680d1c70 0x7fbe680cef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:35.858 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:35.840 7fbe6f72d700 1 -- 172.21.2.102:0/1004512562 >> 172.21.2.102:0/1004512562 conn(0x7fbe680ce420 msgr2=0x7fbe680d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:35.858 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:35.840 7fbe6f72d700 1 -- 172.21.2.102:0/1004512562 shutdown_connections 2021-02-10T11:35:35.858 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:35.840 7fbe6f72d700 1 -- 172.21.2.102:0/1004512562 wait complete. 2021-02-10T11:35:35.859 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:35.854 7fbe6f72d700 1 Processor -- start 2021-02-10T11:35:35.860 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:35.854 7fbe6f72d700 1 -- start start 2021-02-10T11:35:35.860 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:35.855 7fbe6f72d700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbe680c3820 0x7fbe680a7950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:35.860 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:35.855 7fbe6f72d700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fbe681a44c0 0x7fbe681a48c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:35.860 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:35.855 7fbe6f72d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbe680863d0 0x7fbe680867d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:35.860 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:35.855 7fbe6f72d700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe68085ed0 con 0x7fbe680863d0 2021-02-10T11:35:35.860 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:35.855 7fbe6f72d700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe680c0740 con 0x7fbe681a44c0 2021-02-10T11:35:35.860 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:35.855 7fbe6f72d700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe68086df0 con 0x7fbe680c3820 2021-02-10T11:35:35.860 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:35.855 7fbe6dd4a700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbe680863d0 0x7fbe680867d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:35.860 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:35.855 7fbe6dd4a700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbe680863d0 0x7fbe680867d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:37096/0 (socket says 172.21.2.102:37096) 2021-02-10T11:35:35.860 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:35.855 7fbe6dd4a700 1 -- 172.21.2.102:0/1964680369 learned_addr learned my addr 172.21.2.102:0/1964680369 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:35:35.860 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:35.856 7fbe6dd4a700 1 -- 172.21.2.102:0/1964680369 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbe680c3820 msgr2=0x7fbe680a7950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:35.860 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:35.856 7fbe6dd4a700 1 --2- 172.21.2.102:0/1964680369 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbe680c3820 0x7fbe680a7950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:35.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:35.856 7fbe6dd4a700 1 -- 172.21.2.102:0/1964680369 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fbe681a44c0 msgr2=0x7fbe681a48c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:35.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:35.856 7fbe6dd4a700 1 --2- 172.21.2.102:0/1964680369 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fbe681a44c0 0x7fbe681a48c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:35.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:35.856 7fbe6dd4a700 1 -- 172.21.2.102:0/1964680369 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe68087010 con 0x7fbe680863d0 2021-02-10T11:35:35.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:35.856 7fbe6dd4a700 1 --2- 172.21.2.102:0/1964680369 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbe680863d0 0x7fbe680867d0 secure :-1 s=READY pgs=2706 cs=0 l=1 rev1=1 rx=0x7fbe6000b910 tx=0x7fbe6000bcb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:35.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:35.856 7fbe59ffb700 1 -- 172.21.2.102:0/1964680369 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbe6000bdf0 con 0x7fbe680863d0 2021-02-10T11:35:35.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:35.856 7fbe59ffb700 1 -- 172.21.2.102:0/1964680369 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe60005820 con 0x7fbe680863d0 2021-02-10T11:35:35.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:35.856 7fbe6f72d700 1 -- 172.21.2.102:0/1964680369 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe68087010 con 0x7fbe680863d0 2021-02-10T11:35:35.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:35.856 7fbe6f72d700 1 -- 172.21.2.102:0/1964680369 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe68083000 con 0x7fbe680863d0 2021-02-10T11:35:35.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:35.856 7fbe59ffb700 1 -- 172.21.2.102:0/1964680369 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbe60005c70 con 0x7fbe680863d0 2021-02-10T11:35:35.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:35.857 7fbe59ffb700 1 -- 172.21.2.102:0/1964680369 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fbe60005a60 con 0x7fbe680863d0 2021-02-10T11:35:35.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:35.857 7fbe59ffb700 1 --2- 172.21.2.102:0/1964680369 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fbe480186d0 0x7fbe4801abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:35.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:35.857 7fbe59ffb700 1 -- 172.21.2.102:0/1964680369 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fbe60032ba0 con 0x7fbe680863d0 2021-02-10T11:35:35.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:35.857 7fbe6d549700 1 --2- 172.21.2.102:0/1964680369 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fbe480186d0 0x7fbe4801abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:35.862 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:35.857 7fbe6d549700 1 --2- 172.21.2.102:0/1964680369 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fbe480186d0 0x7fbe4801abd0 crc :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:35.890 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:35.888 7fbe6f72d700 1 -- 172.21.2.102:0/1964680369 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe68085e50 con 0x7fbe680863d0 2021-02-10T11:35:35.893 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:35.891 7fbe59ffb700 1 -- 172.21.2.102:0/1964680369 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fbe600093d0 con 0x7fbe680863d0 2021-02-10T11:35:36.020 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:36.018 7fbe6f72d700 1 -- 172.21.2.102:0/1964680369 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolumegroup rm", "vol_name": "cephfs", "target": ["mgr", ""], "group_name": "non_existent_group"}) v1 -- 0x7fbe68083000 con 0x7fbe480186d0 2021-02-10T11:35:36.024 INFO:tasks.ceph.mgr.y.gibba002.stderr:2021-02-10 11:35:36.023 7fd94bc53700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:35:36.024 INFO:tasks.ceph.mgr.y.gibba002.stderr:2021-02-10 11:35:36.023 7fd94bc53700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:35:36.025 INFO:tasks.ceph.mgr.y.gibba002.stderr:2021-02-10 11:35:36.023 7fd94bc53700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:35:36.025 INFO:tasks.ceph.mgr.y.gibba002.stderr:2021-02-10 11:35:36.023 7fd94bc53700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:35:36.025 INFO:tasks.ceph.mgr.y.gibba002.stderr:2021-02-10 11:35:36.023 7fd94bc53700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:35:36.029 INFO:tasks.ceph.mgr.y.gibba002.stderr:2021-02-10 11:35:36.028 7fd94bc53700 -1 mgr.server reply reply (2) No such file or directory subvolume group 'non_existent_group' does not exist 2021-02-10T11:35:36.030 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:36.028 7fbe59ffb700 1 -- 172.21.2.102:0/1964680369 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: -2 subvolume group 'non_existent_group' does not exist) v1 ==== 59+0+0 (crc 0 0 0) 0x7fbe5c000e90 con 0x7fbe480186d0 2021-02-10T11:35:36.035 INFO:teuthology.orchestra.run.gibba002.stderr:Error ENOENT: subvolume group 'non_existent_group' does not exist 2021-02-10T11:35:36.037 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:36.035 7fbe6f72d700 1 -- 172.21.2.102:0/1964680369 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fbe480186d0 msgr2=0x7fbe4801abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:36.037 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:36.035 7fbe6f72d700 1 --2- 172.21.2.102:0/1964680369 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fbe480186d0 0x7fbe4801abd0 crc :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:36.037 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:36.035 7fbe6f72d700 1 -- 172.21.2.102:0/1964680369 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbe680863d0 msgr2=0x7fbe680867d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:36.037 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:36.035 7fbe6f72d700 1 --2- 172.21.2.102:0/1964680369 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbe680863d0 0x7fbe680867d0 secure :-1 s=READY pgs=2706 cs=0 l=1 rev1=1 rx=0x7fbe6000b910 tx=0x7fbe6000bcb0).stop 2021-02-10T11:35:36.037 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:36.035 7fbe6f72d700 1 -- 172.21.2.102:0/1964680369 shutdown_connections 2021-02-10T11:35:36.038 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:36.035 7fbe6f72d700 1 --2- 172.21.2.102:0/1964680369 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbe680863d0 0x7fbe680867d0 unknown :-1 s=CLOSED pgs=2706 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:36.038 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:36.035 7fbe6f72d700 1 --2- 172.21.2.102:0/1964680369 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fbe480186d0 0x7fbe4801abd0 unknown :-1 s=CLOSED pgs=1191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:36.038 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:36.035 7fbe6f72d700 1 --2- 172.21.2.102:0/1964680369 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fbe681a44c0 0x7fbe681a48c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:36.038 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:36.035 7fbe6f72d700 1 --2- 172.21.2.102:0/1964680369 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbe680c3820 0x7fbe680a7950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:36.038 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:36.035 7fbe6f72d700 1 -- 172.21.2.102:0/1964680369 >> 172.21.2.102:0/1964680369 conn(0x7fbe680872c0 msgr2=0x7fbe68095700 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:36.038 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:36.035 7fbe6f72d700 1 -- 172.21.2.102:0/1964680369 shutdown_connections 2021-02-10T11:35:36.038 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:36.035 7fbe6f72d700 1 -- 172.21.2.102:0/1964680369 wait complete. 2021-02-10T11:35:36.046 DEBUG:teuthology.orchestra.run:got remote process result: 2 2021-02-10T11:35:36.046 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2021-02-10T11:35:36.046 DEBUG:teuthology.orchestra.run.gibba002:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:35:36.393 DEBUG:teuthology.orchestra.run.gibba013:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:35:36.739 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2021-02-10T11:35:36.740 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:35:36.740 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:35:36.740 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:35:37.075 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:35:37.076 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:35:37.076 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@gibba002.front.sepia.ceph.com... 2021-02-10T11:35:37.076 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-02-10T11:35:37.076 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:35:37.432 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba002.stderr:ceph-fuse[197319]: fuse finished with error 0 and tester_r 0 2021-02-10T11:35:37.433 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:35:37.433 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:35:37.433 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:35:37.768 INFO:teuthology.orchestra.run.gibba002.stdout:ext2/ext3 2021-02-10T11:35:37.769 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2021-02-10T11:35:37.769 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:35:37.769 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:35:38.105 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:35:38.105 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:35:38.106 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:35:38.440 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:35:38.441 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2021-02-10T11:35:38.441 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@gibba002.front.sepia.ceph.com... 2021-02-10T11:35:38.441 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-02-10T11:35:38.441 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:35:38.796 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba002.stderr:ceph-fuse[197959]: fuse finished with error 0 and tester_r 0 2021-02-10T11:35:38.797 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:35:38.797 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:35:38.797 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:35:39.134 INFO:teuthology.orchestra.run.gibba002.stdout:ext2/ext3 2021-02-10T11:35:39.135 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2021-02-10T11:35:39.135 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:35:39.135 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:35:39.471 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:35:39.471 DEBUG:teuthology.orchestra.run.gibba013:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:35:39.472 DEBUG:teuthology.orchestra.run.gibba013:> ' -- /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:35:39.808 INFO:teuthology.orchestra.run.gibba013.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.2’: No such file or directory 2021-02-10T11:35:39.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:35:39.809 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2021-02-10T11:35:39.809 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:35:39.809 DEBUG:teuthology.orchestra.run.gibba013:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:35:40.144 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:35:40.144 DEBUG:teuthology.orchestra.run.gibba013:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:35:40.144 DEBUG:teuthology.orchestra.run.gibba013:> ' -- /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:35:40.480 INFO:teuthology.orchestra.run.gibba013.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.3’: No such file or directory 2021-02-10T11:35:40.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:35:40.480 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.3 2021-02-10T11:35:40.480 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:35:40.480 DEBUG:teuthology.orchestra.run.gibba013:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:35:40.815 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config rm mon mon_allow_pool_delete 2021-02-10T11:35:41.252 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:41.232 7eff8e348700 1 Processor -- start 2021-02-10T11:35:41.252 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:41.232 7eff8e348700 1 -- start start 2021-02-10T11:35:41.252 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:41.232 7eff8e348700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7eff880d1c70 0x7eff880f20a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:41.252 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:41.232 7eff8e348700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff880f0b10 con 0x7eff880d1870 2021-02-10T11:35:41.252 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:41.232 7eff8e348700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff881b0be0 con 0x7eff880d0340 2021-02-10T11:35:41.253 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:41.232 7eff8e348700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff880e5ed0 con 0x7eff880d1c70 2021-02-10T11:35:41.253 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:41.233 7eff87fff700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7eff880d1870 0x7eff881b03f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:57570/0 (socket says 172.21.2.102:57570) 2021-02-10T11:35:41.253 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:41.233 7eff87fff700 1 -- 172.21.2.102:0/3471803805 learned_addr learned my addr 172.21.2.102:0/3471803805 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:35:41.253 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:41.233 7eff86ffd700 1 -- 172.21.2.102:0/3471803805 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1624873286 0 0) 0x7eff78001180 con 0x7eff880d1870 2021-02-10T11:35:41.253 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:41.233 7eff86ffd700 1 -- 172.21.2.102:0/3471803805 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff70002fe0 con 0x7eff880d1870 2021-02-10T11:35:41.253 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:41.233 7eff8c965700 1 --2- 172.21.2.102:0/3471803805 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7eff880d1c70 0x7eff880f20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:41.253 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:41.233 7eff86ffd700 1 -- 172.21.2.102:0/3471803805 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1017262424 0 0) 0x7eff78001180 con 0x7eff880d1870 2021-02-10T11:35:41.253 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:41.233 7eff86ffd700 1 -- 172.21.2.102:0/3471803805 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7eff880d1c70 msgr2=0x7eff880f20a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:41.253 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:41.233 7eff86ffd700 1 --2- 172.21.2.102:0/3471803805 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7eff880d1c70 0x7eff880f20a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:41.254 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:41.233 7eff86ffd700 1 -- 172.21.2.102:0/3471803805 >> v1:172.21.2.113:6789/0 conn(0x7eff880d0340 legacy=0x7eff880ef640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:41.254 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:41.233 7eff86ffd700 1 -- 172.21.2.102:0/3471803805 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff880efe30 con 0x7eff880d1870 2021-02-10T11:35:41.254 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:41.233 7eff86ffd700 1 -- 172.21.2.102:0/3471803805 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7eff78001680 con 0x7eff880d1870 2021-02-10T11:35:41.254 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:41.233 7eff86ffd700 1 -- 172.21.2.102:0/3471803805 >> v1:172.21.2.102:6789/0 conn(0x7eff880d1870 legacy=0x7eff881b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:41.254 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:41.233 7eff86ffd700 1 --2- 172.21.2.102:0/3471803805 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7eff70003aa0 0x7eff70005ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:41.254 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:41.234 7eff86ffd700 1 --2- 172.21.2.102:0/3471803805 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7eff70006770 0x7eff70008c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:41.254 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:41.234 7eff86ffd700 1 --2- 172.21.2.102:0/3471803805 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7eff70009500 0x7eff7000ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:41.254 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:41.234 7eff86ffd700 1 -- 172.21.2.102:0/3471803805 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff7000c2d0 con 0x7eff70009500 2021-02-10T11:35:41.254 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:41.234 7eff86ffd700 1 -- 172.21.2.102:0/3471803805 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff7000c4f0 con 0x7eff70003aa0 2021-02-10T11:35:41.255 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:41.234 7eff86ffd700 1 -- 172.21.2.102:0/3471803805 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff70050720 con 0x7eff70006770 2021-02-10T11:35:41.255 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:41.234 7eff8c965700 1 --2- 172.21.2.102:0/3471803805 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7eff70009500 0x7eff7000ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:41.255 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:41.234 7eff877fe700 1 --2- 172.21.2.102:0/3471803805 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7eff70006770 0x7eff70008c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:41.255 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:41.234 7eff87fff700 1 --2- 172.21.2.102:0/3471803805 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7eff70003aa0 0x7eff70005ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:41.255 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:41.234 7eff8c965700 1 -- 172.21.2.102:0/3471803805 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7eff70006770 msgr2=0x7eff70008c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:41.255 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:41.234 7eff8c965700 1 --2- 172.21.2.102:0/3471803805 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7eff70006770 0x7eff70008c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:41.255 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:41.234 7eff8c965700 1 -- 172.21.2.102:0/3471803805 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7eff70003aa0 msgr2=0x7eff70005ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:41.256 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:41.234 7eff8c965700 1 --2- 172.21.2.102:0/3471803805 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7eff70003aa0 0x7eff70005ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:41.256 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:41.234 7eff8c965700 1 -- 172.21.2.102:0/3471803805 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff70050980 con 0x7eff70009500 2021-02-10T11:35:41.256 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:41.234 7eff87fff700 1 --2- 172.21.2.102:0/3471803805 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7eff70003aa0 0x7eff70005ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:35:41.256 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:41.234 7eff8c965700 1 --2- 172.21.2.102:0/3471803805 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7eff70009500 0x7eff7000ba20 secure :-1 s=READY pgs=2710 cs=0 l=1 rev1=1 rx=0x7eff80009a40 tx=0x7eff80009de0).ready entity=mon.0 client_cookie=ff6e3c3d3eb4ba6c server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:41.256 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:41.234 7eff86ffd700 1 -- 172.21.2.102:0/3471803805 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7eff8000b510 con 0x7eff70009500 2021-02-10T11:35:41.256 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:41.234 7eff86ffd700 1 -- 172.21.2.102:0/3471803805 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff80003030 con 0x7eff70009500 2021-02-10T11:35:41.256 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:41.234 7eff86ffd700 1 -- 172.21.2.102:0/3471803805 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7eff80000f90 con 0x7eff70009500 2021-02-10T11:35:41.257 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:41.234 7eff8e348700 1 -- 172.21.2.102:0/3471803805 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7eff70009500 msgr2=0x7eff7000ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:41.257 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:41.234 7eff8e348700 1 --2- 172.21.2.102:0/3471803805 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7eff70009500 0x7eff7000ba20 secure :-1 s=READY pgs=2710 cs=0 l=1 rev1=1 rx=0x7eff80009a40 tx=0x7eff80009de0).stop 2021-02-10T11:35:41.257 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:41.234 7eff8c965700 1 -- 172.21.2.102:0/3471803805 reap_dead start 2021-02-10T11:35:41.257 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:41.235 7eff8e348700 1 -- 172.21.2.102:0/3471803805 shutdown_connections 2021-02-10T11:35:41.257 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:41.235 7eff8e348700 1 -- 172.21.2.102:0/3471803805 >> 172.21.2.102:0/3471803805 conn(0x7eff880ce450 msgr2=0x7eff880d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:41.257 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:41.235 7eff8e348700 1 -- 172.21.2.102:0/3471803805 shutdown_connections 2021-02-10T11:35:41.258 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:41.235 7eff8e348700 1 -- 172.21.2.102:0/3471803805 wait complete. 2021-02-10T11:35:41.258 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:41.249 7eff8e348700 1 Processor -- start 2021-02-10T11:35:41.258 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:41.250 7eff8e348700 1 -- start start 2021-02-10T11:35:41.258 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:41.250 7eff8e348700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7eff88094310 0x7eff880ebe70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:41.258 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:41.250 7eff8e348700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7eff88094710 0x7eff880bca30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:41.258 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:41.250 7eff8e348700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7eff880ec490 0x7eff880bd050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:41.258 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:41.250 7eff8e348700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff88093d90 con 0x7eff880ec490 2021-02-10T11:35:41.259 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:41.250 7eff8e348700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff880bd670 con 0x7eff88094310 2021-02-10T11:35:41.259 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:41.250 7eff8e348700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff880939f0 con 0x7eff88094710 2021-02-10T11:35:41.259 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:41.250 7eff8c965700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7eff880ec490 0x7eff880bd050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:41.259 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:41.250 7eff8c965700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7eff880ec490 0x7eff880bd050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:37130/0 (socket says 172.21.2.102:37130) 2021-02-10T11:35:41.259 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:41.250 7eff8c965700 1 -- 172.21.2.102:0/828348990 learned_addr learned my addr 172.21.2.102:0/828348990 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:35:41.262 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:41.250 7eff87fff700 1 --2- 172.21.2.102:0/828348990 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7eff88094310 0x7eff880ebe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:41.262 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:41.250 7eff877fe700 1 --2- 172.21.2.102:0/828348990 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7eff88094710 0x7eff880bca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:41.262 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:41.251 7eff8c965700 1 -- 172.21.2.102:0/828348990 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7eff88094710 msgr2=0x7eff880bca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:41.262 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:41.251 7eff8c965700 1 --2- 172.21.2.102:0/828348990 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7eff88094710 0x7eff880bca30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:41.262 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:41.251 7eff8c965700 1 -- 172.21.2.102:0/828348990 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7eff88094310 msgr2=0x7eff880ebe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:41.262 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:41.251 7eff8c965700 1 --2- 172.21.2.102:0/828348990 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7eff88094310 0x7eff880ebe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:41.262 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:41.251 7eff8c965700 1 -- 172.21.2.102:0/828348990 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff880bda80 con 0x7eff880ec490 2021-02-10T11:35:41.262 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:41.251 7eff8c965700 1 --2- 172.21.2.102:0/828348990 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7eff880ec490 0x7eff880bd050 secure :-1 s=READY pgs=2711 cs=0 l=1 rev1=1 rx=0x7eff80006fc0 tx=0x7eff8000bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:41.263 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:41.251 7eff84ff9700 1 -- 172.21.2.102:0/828348990 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7eff80009630 con 0x7eff880ec490 2021-02-10T11:35:41.263 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:41.251 7eff84ff9700 1 -- 172.21.2.102:0/828348990 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff80002030 con 0x7eff880ec490 2021-02-10T11:35:41.263 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:41.251 7eff8e348700 1 -- 172.21.2.102:0/828348990 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff880bda80 con 0x7eff880ec490 2021-02-10T11:35:41.263 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:41.251 7eff8e348700 1 -- 172.21.2.102:0/828348990 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff881a76d0 con 0x7eff880ec490 2021-02-10T11:35:41.263 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:41.251 7eff84ff9700 1 -- 172.21.2.102:0/828348990 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7eff800024d0 con 0x7eff880ec490 2021-02-10T11:35:41.263 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:41.252 7eff84ff9700 1 -- 172.21.2.102:0/828348990 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7eff800095e0 con 0x7eff880ec490 2021-02-10T11:35:41.263 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:41.252 7eff84ff9700 1 --2- 172.21.2.102:0/828348990 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7eff68018730 0x7eff6801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:41.263 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:41.252 7eff84ff9700 1 -- 172.21.2.102:0/828348990 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 5573+0+0 (secure 0 0 0) 0x7eff80032b40 con 0x7eff880ec490 2021-02-10T11:35:41.263 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:41.252 7eff87fff700 1 --2- 172.21.2.102:0/828348990 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7eff68018730 0x7eff6801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:41.263 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:41.252 7eff87fff700 1 --2- 172.21.2.102:0/828348990 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7eff68018730 0x7eff6801abc0 crc :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:41.284 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:41.282 7eff8e348700 1 -- 172.21.2.102:0/828348990 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff880ebc40 con 0x7eff880ec490 2021-02-10T11:35:41.286 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:41.284 7eff84ff9700 1 -- 172.21.2.102:0/828348990 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7eff800093d0 con 0x7eff880ec490 2021-02-10T11:35:41.434 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:41.432 7eff8e348700 1 -- 172.21.2.102:0/828348990 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7eff880f0b10 con 0x7eff880ec490 2021-02-10T11:35:41.438 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:41.436 7eff84ff9700 1 -- 172.21.2.102:0/828348990 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v37) v1 ==== 104+0+0 (secure 0 0 0) 0x7eff800095e0 con 0x7eff880ec490 2021-02-10T11:35:41.442 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:41.441 7eff8e348700 1 -- 172.21.2.102:0/828348990 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7eff68018730 msgr2=0x7eff6801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:41.443 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:41.441 7eff8e348700 1 --2- 172.21.2.102:0/828348990 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7eff68018730 0x7eff6801abc0 crc :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:41.443 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:41.441 7eff8e348700 1 -- 172.21.2.102:0/828348990 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7eff880ec490 msgr2=0x7eff880bd050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:41.443 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:41.441 7eff8e348700 1 --2- 172.21.2.102:0/828348990 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7eff880ec490 0x7eff880bd050 secure :-1 s=READY pgs=2711 cs=0 l=1 rev1=1 rx=0x7eff80006fc0 tx=0x7eff8000bcd0).stop 2021-02-10T11:35:41.443 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:41.441 7eff8e348700 1 -- 172.21.2.102:0/828348990 shutdown_connections 2021-02-10T11:35:41.443 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:41.441 7eff8e348700 1 --2- 172.21.2.102:0/828348990 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7eff880ec490 0x7eff880bd050 unknown :-1 s=CLOSED pgs=2711 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:41.443 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:41.441 7eff8e348700 1 --2- 172.21.2.102:0/828348990 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7eff88094710 0x7eff880bca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:41.443 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:41.441 7eff8e348700 1 --2- 172.21.2.102:0/828348990 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7eff68018730 0x7eff6801abc0 unknown :-1 s=CLOSED pgs=1192 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:41.444 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:41.441 7eff8e348700 1 --2- 172.21.2.102:0/828348990 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7eff88094310 0x7eff880ebe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:41.444 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:41.441 7eff8e348700 1 -- 172.21.2.102:0/828348990 >> 172.21.2.102:0/828348990 conn(0x7eff8808fa10 msgr2=0x7eff880bfa60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:41.444 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:41.441 7eff8e348700 1 -- 172.21.2.102:0/828348990 shutdown_connections 2021-02-10T11:35:41.444 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:41.441 7eff8e348700 1 -- 172.21.2.102:0/828348990 wait complete. 2021-02-10T11:35:41.451 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestVolumes.test_nonexistent_subvolume_group_rm' 2021-02-10T11:35:41.880 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:41.862 7f6669364700 1 Processor -- start 2021-02-10T11:35:41.880 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:41.862 7f6669364700 1 -- start start 2021-02-10T11:35:41.881 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:41.862 7f6669364700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f66640d1870 0x7f66641b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:41.881 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:41.862 7f6669364700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f66640f0650 0x7f66640cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:41.881 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:41.862 7f6669364700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66640f0b10 con 0x7f66641b0a10 2021-02-10T11:35:41.881 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:41.862 7f6669364700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66640fe230 con 0x7f66640f0650 2021-02-10T11:35:41.881 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:41.862 7f6669364700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66640e5ed0 con 0x7f66640d1870 2021-02-10T11:35:41.881 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:41.862 7f666287c700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f66641b0a10 0x7f66640cf330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:57590/0 (socket says 172.21.2.102:57590) 2021-02-10T11:35:41.881 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:41.862 7f666387e700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f66640f0650 0x7f66640cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:41.881 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:41.862 7f666287c700 1 -- 172.21.2.102:0/3761867874 learned_addr learned my addr 172.21.2.102:0/3761867874 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:35:41.881 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:41.863 7f666307d700 1 --2- 172.21.2.102:0/3761867874 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f66640d1870 0x7f66641b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:41.881 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:41.863 7f666387e700 1 -- 172.21.2.102:0/3761867874 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f66640d1870 msgr2=0x7f66641b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:41.882 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:41.863 7f666387e700 1 --2- 172.21.2.102:0/3761867874 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f66640d1870 0x7f66641b03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:41.882 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:41.863 7f666387e700 1 -- 172.21.2.102:0/3761867874 >> v1:172.21.2.102:6789/0 conn(0x7f66641b0a10 legacy=0x7f66640cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:41.882 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:41.863 7f666387e700 1 -- 172.21.2.102:0/3761867874 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66640f20a0 con 0x7f66640f0650 2021-02-10T11:35:41.882 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:41.863 7f666387e700 1 --2- 172.21.2.102:0/3761867874 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f66640f0650 0x7f66640cfb20 secure :-1 s=READY pgs=2713 cs=0 l=1 rev1=1 rx=0x7f665c006e30 tx=0x7f665c009e60).ready entity=mon.0 client_cookie=1f0fdc76186742a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:41.882 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:41.863 7f666207b700 1 -- 172.21.2.102:0/3761867874 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f665c006a50 con 0x7f66640f0650 2021-02-10T11:35:41.882 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:41.863 7f666207b700 1 -- 172.21.2.102:0/3761867874 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f665c00b300 con 0x7f66640f0650 2021-02-10T11:35:41.882 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:41.863 7f666207b700 1 -- 172.21.2.102:0/3761867874 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f665c00b820 con 0x7f66640f0650 2021-02-10T11:35:41.882 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:41.864 7f6669364700 1 -- 172.21.2.102:0/3761867874 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f66640f0650 msgr2=0x7f66640cfb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:41.882 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:41.864 7f6669364700 1 --2- 172.21.2.102:0/3761867874 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f66640f0650 0x7f66640cfb20 secure :-1 s=READY pgs=2713 cs=0 l=1 rev1=1 rx=0x7f665c006e30 tx=0x7f665c009e60).stop 2021-02-10T11:35:41.882 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:41.864 7f6669364700 1 -- 172.21.2.102:0/3761867874 shutdown_connections 2021-02-10T11:35:41.882 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:41.864 7f6669364700 1 --2- 172.21.2.102:0/3761867874 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f66640f0650 0x7f66640cfb20 unknown :-1 s=CLOSED pgs=2713 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:41.883 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:41.864 7f6669364700 1 --2- 172.21.2.102:0/3761867874 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f66640d1870 0x7f66641b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:41.883 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:41.864 7f6669364700 1 -- 172.21.2.102:0/3761867874 >> 172.21.2.102:0/3761867874 conn(0x7f66640ce450 msgr2=0x7f66640d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:41.883 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:41.864 7f6669364700 1 -- 172.21.2.102:0/3761867874 shutdown_connections 2021-02-10T11:35:41.883 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:41.864 7f6669364700 1 -- 172.21.2.102:0/3761867874 wait complete. 2021-02-10T11:35:41.884 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:41.879 7f6669364700 1 Processor -- start 2021-02-10T11:35:41.884 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:41.879 7f6669364700 1 -- start start 2021-02-10T11:35:41.884 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:41.879 7f6669364700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f66640ada90 0x7f66640ade90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:41.885 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:41.879 7f6669364700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6664099d20 0x7f666409a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:41.885 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:41.879 7f6669364700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f666409a740 0x7f6664097bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:41.885 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:41.879 7f6669364700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66640997a0 con 0x7f6664099d20 2021-02-10T11:35:41.885 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:41.879 7f6669364700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f666418f0c0 con 0x7f666409a740 2021-02-10T11:35:41.885 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:41.879 7f6669364700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f666409ab40 con 0x7f66640ada90 2021-02-10T11:35:41.885 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:41.880 7f666287c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6664099d20 0x7f666409a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:41.885 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:41.880 7f666287c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6664099d20 0x7f666409a120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:37142/0 (socket says 172.21.2.102:37142) 2021-02-10T11:35:41.885 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:41.880 7f666287c700 1 -- 172.21.2.102:0/3756081467 learned_addr learned my addr 172.21.2.102:0/3756081467 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:35:41.885 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:41.880 7f666387e700 1 --2- 172.21.2.102:0/3756081467 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f666409a740 0x7f6664097bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:41.885 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:41.880 7f666307d700 1 --2- 172.21.2.102:0/3756081467 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f66640ada90 0x7f66640ade90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:41.885 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:41.880 7f666287c700 1 -- 172.21.2.102:0/3756081467 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f66640ada90 msgr2=0x7f66640ade90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:41.885 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:41.880 7f666287c700 1 --2- 172.21.2.102:0/3756081467 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f66640ada90 0x7f66640ade90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:41.885 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:41.880 7f666287c700 1 -- 172.21.2.102:0/3756081467 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f666409a740 msgr2=0x7f6664097bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:41.886 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:41.880 7f666287c700 1 --2- 172.21.2.102:0/3756081467 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f666409a740 0x7f6664097bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:41.886 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:41.880 7f666287c700 1 -- 172.21.2.102:0/3756081467 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f666409ad60 con 0x7f6664099d20 2021-02-10T11:35:41.886 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:41.880 7f666287c700 1 --2- 172.21.2.102:0/3756081467 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6664099d20 0x7f666409a120 secure :-1 s=READY pgs=2714 cs=0 l=1 rev1=1 rx=0x7f6658001c60 tx=0x7f665800a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:41.886 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:41.880 7f664bfff700 1 -- 172.21.2.102:0/3756081467 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f665800acc0 con 0x7f6664099d20 2021-02-10T11:35:41.886 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:41.880 7f664bfff700 1 -- 172.21.2.102:0/3756081467 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6658004140 con 0x7f6664099d20 2021-02-10T11:35:41.886 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:41.880 7f664bfff700 1 -- 172.21.2.102:0/3756081467 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6658004660 con 0x7f6664099d20 2021-02-10T11:35:41.886 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:41.880 7f6669364700 1 -- 172.21.2.102:0/3756081467 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f666409ad60 con 0x7f6664099d20 2021-02-10T11:35:41.886 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:41.880 7f6669364700 1 -- 172.21.2.102:0/3756081467 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f666409ad60 con 0x7f6664099d20 2021-02-10T11:35:41.886 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:41.881 7f664bfff700 1 -- 172.21.2.102:0/3756081467 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f6658004320 con 0x7f6664099d20 2021-02-10T11:35:41.887 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:41.881 7f664bfff700 1 --2- 172.21.2.102:0/3756081467 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f66440186c0 0x7f664401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:41.887 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:41.881 7f664bfff700 1 -- 172.21.2.102:0/3756081467 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f66580325a0 con 0x7f6664099d20 2021-02-10T11:35:41.887 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:41.881 7f666307d700 1 --2- 172.21.2.102:0/3756081467 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f66440186c0 0x7f664401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:41.887 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:41.881 7f666307d700 1 --2- 172.21.2.102:0/3756081467 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f66440186c0 0x7f664401abc0 crc :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:41.914 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:41.912 7f6669364700 1 -- 172.21.2.102:0/3756081467 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66640ebc00 con 0x7f6664099d20 2021-02-10T11:35:41.917 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:41.915 7f664bfff700 1 -- 172.21.2.102:0/3756081467 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f665802e530 con 0x7f6664099d20 2021-02-10T11:35:42.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:42.048 7f6669364700 1 -- 172.21.2.102:0/3756081467 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestVolumes.test_nonexistent_subvolume_group_rm"]} v 0) v1 -- 0x7f66640ebc00 con 0x7f6664099d20 2021-02-10T11:35:42.489 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:42.487 7f664bfff700 1 -- 172.21.2.102:0/3756081467 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestVolumes.test_nonexistent_subvolume_group_rm"]}]=0 v733) v1 ==== 152+0+0 (secure 0 0 0) 0x7f665802e530 con 0x7f6664099d20 2021-02-10T11:35:42.518 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:42.516 7f6669364700 1 -- 172.21.2.102:0/3756081467 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f66440186c0 msgr2=0x7f664401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:42.518 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:42.516 7f6669364700 1 --2- 172.21.2.102:0/3756081467 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f66440186c0 0x7f664401abc0 crc :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:42.518 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:42.516 7f6669364700 1 -- 172.21.2.102:0/3756081467 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6664099d20 msgr2=0x7f666409a120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:42.519 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:42.516 7f6669364700 1 --2- 172.21.2.102:0/3756081467 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6664099d20 0x7f666409a120 secure :-1 s=READY pgs=2714 cs=0 l=1 rev1=1 rx=0x7f6658001c60 tx=0x7f665800a8d0).stop 2021-02-10T11:35:42.519 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:42.516 7f6669364700 1 -- 172.21.2.102:0/3756081467 shutdown_connections 2021-02-10T11:35:42.519 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:42.516 7f6669364700 1 --2- 172.21.2.102:0/3756081467 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f66440186c0 0x7f664401abc0 unknown :-1 s=CLOSED pgs=1193 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:42.519 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:42.516 7f6669364700 1 --2- 172.21.2.102:0/3756081467 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f666409a740 0x7f6664097bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:42.519 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:42.516 7f6669364700 1 --2- 172.21.2.102:0/3756081467 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6664099d20 0x7f666409a120 unknown :-1 s=CLOSED pgs=2714 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:42.519 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:42.516 7f6669364700 1 --2- 172.21.2.102:0/3756081467 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f66640ada90 0x7f66640ade90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:42.519 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:42.516 7f6669364700 1 -- 172.21.2.102:0/3756081467 >> 172.21.2.102:0/3756081467 conn(0x7f66640f0550 msgr2=0x7f6664191730 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:42.519 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:42.516 7f6669364700 1 -- 172.21.2.102:0/3756081467 shutdown_connections 2021-02-10T11:35:42.519 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:42.516 7f6669364700 1 -- 172.21.2.102:0/3756081467 wait complete. 2021-02-10T11:35:42.527 INFO:tasks.cephfs_test_runner:test_nonexistent_subvolume_group_rm (tasks.cephfs.test_volumes.TestVolumes) ... ok 2021-02-10T11:35:42.527 INFO:tasks.cephfs_test_runner:Starting test: test_nonexistent_subvolume_group_snapshot_rm (tasks.cephfs.test_volumes.TestVolumes) 2021-02-10T11:35:42.527 INFO:tasks.cephfs_test_runner:test_nonexistent_subvolume_group_snapshot_rm (tasks.cephfs.test_volumes.TestVolumes) ... skipped 'skipping subvolumegroup snapshot tests' 2021-02-10T11:35:42.527 INFO:tasks.cephfs_test_runner:Starting test: test_nonexistent_subvolume_rm (tasks.cephfs.test_volumes.TestVolumes) 2021-02-10T11:35:42.527 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestVolumes.test_nonexistent_subvolume_rm' 2021-02-10T11:35:42.554 DEBUG:teuthology.orchestra.run.gibba002:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:35:42.573 DEBUG:teuthology.orchestra.run.gibba013:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:35:42.957 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:42.939 7f3e4ba12700 1 Processor -- start 2021-02-10T11:35:42.957 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:42.939 7f3e4ba12700 1 -- start start 2021-02-10T11:35:42.957 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:42.939 7f3e4ba12700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3e440d1870 0x7f3e441b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:42.957 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:42.939 7f3e4ba12700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f3e440f0650 0x7f3e440cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:42.957 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:42.939 7f3e4ba12700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e440f0b10 con 0x7f3e441b0a10 2021-02-10T11:35:42.957 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:42.939 7f3e4ba12700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e440fe230 con 0x7f3e440f0650 2021-02-10T11:35:42.957 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:42.939 7f3e4ba12700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e440e5ed0 con 0x7f3e440d1870 2021-02-10T11:35:42.957 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:42.939 7f3e4a02f700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f3e440f0650 0x7f3e440cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:42.958 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:42.939 7f3e4a02f700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f3e440f0650 0x7f3e440cfb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:37148/0 (socket says 172.21.2.102:37148) 2021-02-10T11:35:42.958 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:42.939 7f3e4a02f700 1 -- 172.21.2.102:0/176360063 learned_addr learned my addr 172.21.2.102:0/176360063 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:35:42.958 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:42.940 7f3e4a02f700 1 -- 172.21.2.102:0/176360063 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3e440d1870 msgr2=0x7f3e441b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:42.958 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:42.940 7f3e4a02f700 1 --2- 172.21.2.102:0/176360063 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3e440d1870 0x7f3e441b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:42.958 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:42.940 7f3e4a02f700 1 -- 172.21.2.102:0/176360063 >> v1:172.21.2.113:6789/0 conn(0x7f3e441b0a10 legacy=0x7f3e440cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:42.958 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:42.940 7f3e4a02f700 1 -- 172.21.2.102:0/176360063 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e440f20a0 con 0x7f3e440f0650 2021-02-10T11:35:42.958 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:42.940 7f3e4a02f700 1 --2- 172.21.2.102:0/176360063 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3e440f0650 0x7f3e440cfb20 secure :-1 s=READY pgs=2715 cs=0 l=1 rev1=1 rx=0x7f3e3c006f00 tx=0x7f3e3c009f10).ready entity=mon.0 client_cookie=1b246e007cd26b23 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:42.958 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:42.940 7f3e4882c700 1 -- 172.21.2.102:0/176360063 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3e3c00b3b0 con 0x7f3e440f0650 2021-02-10T11:35:42.958 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:42.940 7f3e4882c700 1 -- 172.21.2.102:0/176360063 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e3c00b630 con 0x7f3e440f0650 2021-02-10T11:35:42.958 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:42.940 7f3e4882c700 1 -- 172.21.2.102:0/176360063 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3e3c00bb50 con 0x7f3e440f0650 2021-02-10T11:35:42.958 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:42.940 7f3e4ba12700 1 -- 172.21.2.102:0/176360063 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3e440f0650 msgr2=0x7f3e440cfb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:42.958 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:42.940 7f3e4ba12700 1 --2- 172.21.2.102:0/176360063 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3e440f0650 0x7f3e440cfb20 secure :-1 s=READY pgs=2715 cs=0 l=1 rev1=1 rx=0x7f3e3c006f00 tx=0x7f3e3c009f10).stop 2021-02-10T11:35:42.959 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:42.942 7f3e4ba12700 1 -- 172.21.2.102:0/176360063 shutdown_connections 2021-02-10T11:35:42.959 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:42.942 7f3e4ba12700 1 --2- 172.21.2.102:0/176360063 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3e440f0650 0x7f3e440cfb20 unknown :-1 s=CLOSED pgs=2715 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:42.959 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:42.942 7f3e4ba12700 1 --2- 172.21.2.102:0/176360063 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3e440d1870 0x7f3e441b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:42.959 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:42.942 7f3e4ba12700 1 -- 172.21.2.102:0/176360063 >> 172.21.2.102:0/176360063 conn(0x7f3e440ce450 msgr2=0x7f3e440d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:42.959 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:42.942 7f3e4ba12700 1 -- 172.21.2.102:0/176360063 shutdown_connections 2021-02-10T11:35:42.959 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:42.942 7f3e4ba12700 1 -- 172.21.2.102:0/176360063 wait complete. 2021-02-10T11:35:42.961 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:42.955 7f3e4ba12700 1 Processor -- start 2021-02-10T11:35:42.961 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:42.955 7f3e4ba12700 1 -- start start 2021-02-10T11:35:42.961 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:42.956 7f3e4ba12700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3e4419af00 0x7f3e4419b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:42.961 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:42.956 7f3e4ba12700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3e4419ba80 0x7f3e4419be80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:42.961 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:42.956 7f3e4ba12700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3e4419c4a0 0x7f3e4408a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:42.961 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:42.956 7f3e4ba12700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e4408fb50 con 0x7f3e4419c4a0 2021-02-10T11:35:42.961 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:42.956 7f3e4ba12700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e4408afb0 con 0x7f3e4419ba80 2021-02-10T11:35:42.961 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:42.956 7f3e4ba12700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e4408b1a0 con 0x7f3e4419af00 2021-02-10T11:35:42.961 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:42.956 7f3e4a02f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3e4419c4a0 0x7f3e4408a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:42.961 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:42.956 7f3e4a02f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3e4419c4a0 0x7f3e4408a990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:37156/0 (socket says 172.21.2.102:37156) 2021-02-10T11:35:42.961 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:42.956 7f3e4a02f700 1 -- 172.21.2.102:0/3585005213 learned_addr learned my addr 172.21.2.102:0/3585005213 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:35:42.962 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:42.956 7f3e4982e700 1 --2- 172.21.2.102:0/3585005213 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3e4419af00 0x7f3e4419b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:42.962 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:42.956 7f3e4a02f700 1 -- 172.21.2.102:0/3585005213 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3e4419af00 msgr2=0x7f3e4419b300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:42.962 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:42.956 7f3e4a02f700 1 --2- 172.21.2.102:0/3585005213 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3e4419af00 0x7f3e4419b300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:42.962 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:42.956 7f3e4a02f700 1 -- 172.21.2.102:0/3585005213 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3e4419ba80 msgr2=0x7f3e4419be80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:42.962 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:42.956 7f3e4a02f700 1 --2- 172.21.2.102:0/3585005213 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3e4419ba80 0x7f3e4419be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:42.962 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:42.956 7f3e4a02f700 1 -- 172.21.2.102:0/3585005213 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e4408b3c0 con 0x7f3e4419c4a0 2021-02-10T11:35:42.962 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:42.957 7f3e4a02f700 1 --2- 172.21.2.102:0/3585005213 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3e4419c4a0 0x7f3e4408a990 secure :-1 s=READY pgs=2716 cs=0 l=1 rev1=1 rx=0x7f3e3c009390 tx=0x7f3e3c00bd40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:42.962 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:42.957 7f3e367fc700 1 -- 172.21.2.102:0/3585005213 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3e3c009450 con 0x7f3e4419c4a0 2021-02-10T11:35:42.962 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:42.957 7f3e367fc700 1 -- 172.21.2.102:0/3585005213 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e3c003030 con 0x7f3e4419c4a0 2021-02-10T11:35:42.962 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:42.957 7f3e4ba12700 1 -- 172.21.2.102:0/3585005213 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e4408b3c0 con 0x7f3e4419c4a0 2021-02-10T11:35:42.962 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:42.957 7f3e367fc700 1 -- 172.21.2.102:0/3585005213 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3e3c001480 con 0x7f3e4419c4a0 2021-02-10T11:35:42.962 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:42.957 7f3e4ba12700 1 -- 172.21.2.102:0/3585005213 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e4408b3c0 con 0x7f3e4419c4a0 2021-02-10T11:35:42.963 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:42.958 7f3e367fc700 1 -- 172.21.2.102:0/3585005213 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f3e3c005820 con 0x7f3e4419c4a0 2021-02-10T11:35:42.963 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:42.958 7f3e367fc700 1 --2- 172.21.2.102:0/3585005213 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3e2c018730 0x7f3e2c01abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:42.963 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:42.958 7f3e367fc700 1 -- 172.21.2.102:0/3585005213 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f3e3c032b40 con 0x7f3e4419c4a0 2021-02-10T11:35:42.963 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:42.958 7f3e4982e700 1 --2- 172.21.2.102:0/3585005213 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3e2c018730 0x7f3e2c01abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:42.963 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:42.958 7f3e4982e700 1 --2- 172.21.2.102:0/3585005213 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3e2c018730 0x7f3e2c01abf0 crc :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:42.991 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:42.989 7f3e4ba12700 1 -- 172.21.2.102:0/3585005213 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e440ee030 con 0x7f3e4419c4a0 2021-02-10T11:35:42.993 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:42.993 7f3e367fc700 1 -- 172.21.2.102:0/3585005213 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f3e3c003010 con 0x7f3e4419c4a0 2021-02-10T11:35:43.127 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:43.125 7f3e4ba12700 1 -- 172.21.2.102:0/3585005213 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestVolumes.test_nonexistent_subvolume_rm"]} v 0) v1 -- 0x7f3e440ee030 con 0x7f3e4419c4a0 2021-02-10T11:35:43.492 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:43.490 7f3e367fc700 1 -- 172.21.2.102:0/3585005213 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestVolumes.test_nonexistent_subvolume_rm"]}]=0 v734) v1 ==== 149+0+0 (secure 0 0 0) 0x7f3e3c003010 con 0x7f3e4419c4a0 2021-02-10T11:35:43.495 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:43.493 7f3e4ba12700 1 -- 172.21.2.102:0/3585005213 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3e2c018730 msgr2=0x7f3e2c01abf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:43.495 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:43.493 7f3e4ba12700 1 --2- 172.21.2.102:0/3585005213 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3e2c018730 0x7f3e2c01abf0 crc :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:43.495 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:43.493 7f3e4ba12700 1 -- 172.21.2.102:0/3585005213 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3e4419c4a0 msgr2=0x7f3e4408a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:43.495 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:43.493 7f3e4ba12700 1 --2- 172.21.2.102:0/3585005213 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3e4419c4a0 0x7f3e4408a990 secure :-1 s=READY pgs=2716 cs=0 l=1 rev1=1 rx=0x7f3e3c009390 tx=0x7f3e3c00bd40).stop 2021-02-10T11:35:43.495 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:43.493 7f3e4ba12700 1 -- 172.21.2.102:0/3585005213 shutdown_connections 2021-02-10T11:35:43.495 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:43.493 7f3e4ba12700 1 --2- 172.21.2.102:0/3585005213 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3e4419c4a0 0x7f3e4408a990 unknown :-1 s=CLOSED pgs=2716 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:43.496 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:43.493 7f3e4ba12700 1 --2- 172.21.2.102:0/3585005213 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3e2c018730 0x7f3e2c01abf0 unknown :-1 s=CLOSED pgs=1194 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:43.496 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:43.493 7f3e4ba12700 1 --2- 172.21.2.102:0/3585005213 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3e4419ba80 0x7f3e4419be80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:43.496 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:43.493 7f3e4ba12700 1 --2- 172.21.2.102:0/3585005213 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3e4419af00 0x7f3e4419b300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:43.496 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:43.493 7f3e4ba12700 1 -- 172.21.2.102:0/3585005213 >> 172.21.2.102:0/3585005213 conn(0x7f3e4419f8c0 msgr2=0x7f3e441a5fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:43.496 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:43.493 7f3e4ba12700 1 -- 172.21.2.102:0/3585005213 shutdown_connections 2021-02-10T11:35:43.496 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:43.493 7f3e4ba12700 1 -- 172.21.2.102:0/3585005213 wait complete. 2021-02-10T11:35:43.504 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config set mon mon_allow_pool_delete True 2021-02-10T11:35:43.936 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:43.917 7f6efe888700 1 Processor -- start 2021-02-10T11:35:43.936 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:43.917 7f6efe888700 1 -- start start 2021-02-10T11:35:43.936 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:43.918 7f6efe888700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6ef80d1870 0x7f6ef81b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:43.936 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:43.918 7f6efe888700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f6ef81b0ae0 0x7f6ef8108940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:43.936 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:43.918 7f6efe888700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ef8109a60 con 0x7f6ef80d1c70 2021-02-10T11:35:43.936 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:43.918 7f6efe888700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ef80e5ef0 con 0x7f6ef81b0ae0 2021-02-10T11:35:43.936 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:43.918 7f6efe888700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ef8109750 con 0x7f6ef80d1870 2021-02-10T11:35:43.937 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:43.918 7f6efcea5700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f6ef80d1c70 0x7f6ef8108f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:57616/0 (socket says 172.21.2.102:57616) 2021-02-10T11:35:43.937 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:43.918 7f6efcea5700 1 -- 172.21.2.102:0/2831271510 learned_addr learned my addr 172.21.2.102:0/2831271510 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:35:43.937 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:43.918 7f6ef7fff700 1 --2- 172.21.2.102:0/2831271510 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6ef80d1870 0x7f6ef81b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:43.937 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:43.918 7f6ef77fe700 1 --2- 172.21.2.102:0/2831271510 >> v2:172.21.2.113:3300/0 conn(0x7f6ef81b0ae0 0x7f6ef8108940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:43.937 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:43.918 7f6ef6ffd700 1 -- 172.21.2.102:0/2831271510 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2697521057 0 0) 0x7f6ef0001500 con 0x7f6ef80d1c70 2021-02-10T11:35:43.937 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:43.918 7f6ef6ffd700 1 -- 172.21.2.102:0/2831271510 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ee0002fe0 con 0x7f6ef80d1c70 2021-02-10T11:35:43.937 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:43.919 7f6ef7fff700 1 -- 172.21.2.102:0/2831271510 >> v2:172.21.2.113:3300/0 conn(0x7f6ef81b0ae0 msgr2=0x7f6ef8108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:43.937 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:43.919 7f6ef7fff700 1 --2- 172.21.2.102:0/2831271510 >> v2:172.21.2.113:3300/0 conn(0x7f6ef81b0ae0 0x7f6ef8108940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:43.937 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:43.919 7f6ef7fff700 1 -- 172.21.2.102:0/2831271510 >> v1:172.21.2.102:6789/0 conn(0x7f6ef80d1c70 legacy=0x7f6ef8108f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:43.937 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:43.919 7f6ef7fff700 1 -- 172.21.2.102:0/2831271510 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ef8088070 con 0x7f6ef80d1870 2021-02-10T11:35:43.937 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:43.919 7f6ef7fff700 1 --2- 172.21.2.102:0/2831271510 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6ef80d1870 0x7f6ef81b04c0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0x7f6ee8006650 tx=0x7f6ee8009a80).ready entity=mon.2 client_cookie=b681456767e4ced0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:43.937 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:43.919 7f6ef6ffd700 1 -- 172.21.2.102:0/2831271510 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6ee800c0c0 con 0x7f6ef80d1870 2021-02-10T11:35:43.937 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:43.919 7f6ef6ffd700 1 -- 172.21.2.102:0/2831271510 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ee800c340 con 0x7f6ef80d1870 2021-02-10T11:35:43.938 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:43.919 7f6efe888700 1 -- 172.21.2.102:0/2831271510 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6ef80d1870 msgr2=0x7f6ef81b04c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:43.938 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:43.919 7f6efe888700 1 --2- 172.21.2.102:0/2831271510 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6ef80d1870 0x7f6ef81b04c0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0x7f6ee8006650 tx=0x7f6ee8009a80).stop 2021-02-10T11:35:43.938 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:43.919 7f6ef6ffd700 1 -- 172.21.2.102:0/2831271510 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6ee800c790 con 0x7f6ef80d1870 2021-02-10T11:35:43.938 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:43.920 7f6efe888700 1 -- 172.21.2.102:0/2831271510 shutdown_connections 2021-02-10T11:35:43.938 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:43.920 7f6efe888700 1 --2- 172.21.2.102:0/2831271510 >> v2:172.21.2.113:3300/0 conn(0x7f6ef81b0ae0 0x7f6ef8108940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:43.938 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:43.920 7f6efe888700 1 --2- 172.21.2.102:0/2831271510 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6ef80d1870 0x7f6ef81b04c0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:43.938 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:43.920 7f6efe888700 1 -- 172.21.2.102:0/2831271510 >> 172.21.2.102:0/2831271510 conn(0x7f6ef80ce420 msgr2=0x7f6ef80d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:43.938 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:43.920 7f6efe888700 1 -- 172.21.2.102:0/2831271510 shutdown_connections 2021-02-10T11:35:43.938 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:43.920 7f6efe888700 1 -- 172.21.2.102:0/2831271510 wait complete. 2021-02-10T11:35:43.940 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:43.934 7f6efe888700 1 Processor -- start 2021-02-10T11:35:43.940 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:43.934 7f6efe888700 1 -- start start 2021-02-10T11:35:43.940 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:43.935 7f6efe888700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6ef808aa00 0x7f6ef81a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:43.940 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:43.935 7f6efe888700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6ef809f580 0x7f6ef809f980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:43.940 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:43.935 7f6efe888700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6ef80a0830 0x7f6ef80a0c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:43.940 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:43.935 7f6efe888700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ef80a0070 con 0x7f6ef808aa00 2021-02-10T11:35:43.940 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:43.935 7f6efe888700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ef808ae30 con 0x7f6ef809f580 2021-02-10T11:35:43.940 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:43.935 7f6efe888700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ef808d2a0 con 0x7f6ef80a0830 2021-02-10T11:35:43.940 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:43.935 7f6ef7fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6ef808aa00 0x7f6ef81a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:43.940 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:43.935 7f6ef7fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6ef808aa00 0x7f6ef81a43b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:37166/0 (socket says 172.21.2.102:37166) 2021-02-10T11:35:43.941 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:43.935 7f6ef7fff700 1 -- 172.21.2.102:0/956667641 learned_addr learned my addr 172.21.2.102:0/956667641 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:35:43.941 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:43.935 7f6ef7fff700 1 -- 172.21.2.102:0/956667641 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6ef80a0830 msgr2=0x7f6ef80a0c30 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:35:43.941 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:43.935 7f6ef7fff700 1 --2- 172.21.2.102:0/956667641 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6ef80a0830 0x7f6ef80a0c30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:43.941 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:43.935 7f6ef7fff700 1 -- 172.21.2.102:0/956667641 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6ef809f580 msgr2=0x7f6ef809f980 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:35:43.941 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:43.935 7f6ef7fff700 1 --2- 172.21.2.102:0/956667641 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6ef809f580 0x7f6ef809f980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:43.941 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:43.935 7f6ef7fff700 1 -- 172.21.2.102:0/956667641 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ef81a49d0 con 0x7f6ef808aa00 2021-02-10T11:35:43.941 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:43.936 7f6ef7fff700 1 --2- 172.21.2.102:0/956667641 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6ef808aa00 0x7f6ef81a43b0 secure :-1 s=READY pgs=2718 cs=0 l=1 rev1=1 rx=0x7f6ee800b5e0 tx=0x7f6ee800b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:43.941 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:43.936 7f6ef4ff9700 1 -- 172.21.2.102:0/956667641 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6ee8006800 con 0x7f6ef808aa00 2021-02-10T11:35:43.941 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:43.936 7f6ef4ff9700 1 -- 172.21.2.102:0/956667641 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ee800bd70 con 0x7f6ef808aa00 2021-02-10T11:35:43.941 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:43.936 7f6ef4ff9700 1 -- 172.21.2.102:0/956667641 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6ee8002300 con 0x7f6ef808aa00 2021-02-10T11:35:43.941 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:43.936 7f6efe888700 1 -- 172.21.2.102:0/956667641 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ef81a49d0 con 0x7f6ef808aa00 2021-02-10T11:35:43.941 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:43.936 7f6efe888700 1 -- 172.21.2.102:0/956667641 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ef80a1540 con 0x7f6ef808aa00 2021-02-10T11:35:43.941 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:43.936 7f6ef4ff9700 1 -- 172.21.2.102:0/956667641 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f6ee8000c90 con 0x7f6ef808aa00 2021-02-10T11:35:43.942 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:43.937 7f6ef4ff9700 1 --2- 172.21.2.102:0/956667641 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6ed80186c0 0x7f6ed801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:43.942 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:43.937 7f6ef4ff9700 1 -- 172.21.2.102:0/956667641 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f6ee8032570 con 0x7f6ef808aa00 2021-02-10T11:35:43.942 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:43.937 7f6ef77fe700 1 --2- 172.21.2.102:0/956667641 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6ed80186c0 0x7f6ed801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:43.942 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:43.937 7f6ef77fe700 1 --2- 172.21.2.102:0/956667641 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6ed80186c0 0x7f6ed801abc0 crc :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:43.969 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:43.968 7f6efe888700 1 -- 172.21.2.102:0/956667641 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ef8082980 con 0x7f6ef808aa00 2021-02-10T11:35:43.972 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:43.971 7f6ef4ff9700 1 -- 172.21.2.102:0/956667641 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f6ee8025ce0 con 0x7f6ef808aa00 2021-02-10T11:35:44.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:44.118 7f6efe888700 1 -- 172.21.2.102:0/956667641 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f6ef8109a60 con 0x7f6ef808aa00 2021-02-10T11:35:44.124 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:44.122 7f6ef4ff9700 1 -- 172.21.2.102:0/956667641 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v38)=0 v38) v1 ==== 122+0+0 (secure 0 0 0) 0x7f6ee8025ce0 con 0x7f6ef808aa00 2021-02-10T11:35:44.129 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:44.127 7f6efe888700 1 -- 172.21.2.102:0/956667641 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6ed80186c0 msgr2=0x7f6ed801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:44.129 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:44.127 7f6efe888700 1 --2- 172.21.2.102:0/956667641 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6ed80186c0 0x7f6ed801abc0 crc :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:44.129 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:44.127 7f6efe888700 1 -- 172.21.2.102:0/956667641 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6ef808aa00 msgr2=0x7f6ef81a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:44.129 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:44.127 7f6efe888700 1 --2- 172.21.2.102:0/956667641 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6ef808aa00 0x7f6ef81a43b0 secure :-1 s=READY pgs=2718 cs=0 l=1 rev1=1 rx=0x7f6ee800b5e0 tx=0x7f6ee800b980).stop 2021-02-10T11:35:44.130 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:44.127 7f6efe888700 1 -- 172.21.2.102:0/956667641 shutdown_connections 2021-02-10T11:35:44.130 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:44.127 7f6efe888700 1 --2- 172.21.2.102:0/956667641 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6ef80a0830 0x7f6ef80a0c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:44.130 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:44.127 7f6efe888700 1 --2- 172.21.2.102:0/956667641 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6ed80186c0 0x7f6ed801abc0 unknown :-1 s=CLOSED pgs=1195 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:44.130 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:44.127 7f6efe888700 1 --2- 172.21.2.102:0/956667641 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6ef809f580 0x7f6ef809f980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:44.130 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:44.127 7f6efe888700 1 --2- 172.21.2.102:0/956667641 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6ef808aa00 0x7f6ef81a43b0 unknown :-1 s=CLOSED pgs=2718 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:44.130 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:44.127 7f6efe888700 1 -- 172.21.2.102:0/956667641 >> 172.21.2.102:0/956667641 conn(0x7f6ef80bfb90 msgr2=0x7f6ef80a6a00 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:44.130 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:44.127 7f6efe888700 1 -- 172.21.2.102:0/956667641 shutdown_connections 2021-02-10T11:35:44.130 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:44.128 7f6efe888700 1 -- 172.21.2.102:0/956667641 wait complete. 2021-02-10T11:35:44.137 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2021-02-10T11:35:44.138 DEBUG:teuthology.orchestra.run.gibba002:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:35:44.484 DEBUG:teuthology.orchestra.run.gibba013:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:35:44.832 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2021-02-10T11:35:44.832 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:35:44.832 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:35:44.832 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:35:45.166 INFO:teuthology.orchestra.run.gibba002.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.0’: No such file or directory 2021-02-10T11:35:45.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:35:45.166 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.0 2021-02-10T11:35:45.167 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:35:45.167 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:35:45.167 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:35:45.502 INFO:teuthology.orchestra.run.gibba002.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.1’: No such file or directory 2021-02-10T11:35:45.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:35:45.503 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2021-02-10T11:35:45.503 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:35:45.503 DEBUG:teuthology.orchestra.run.gibba013:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:35:45.503 DEBUG:teuthology.orchestra.run.gibba013:> ' -- /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:35:45.839 INFO:teuthology.orchestra.run.gibba013.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.2’: No such file or directory 2021-02-10T11:35:45.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:35:45.839 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2021-02-10T11:35:45.840 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:35:45.840 DEBUG:teuthology.orchestra.run.gibba013:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:35:45.840 DEBUG:teuthology.orchestra.run.gibba013:> ' -- /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:35:46.175 INFO:teuthology.orchestra.run.gibba013.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.3’: No such file or directory 2021-02-10T11:35:46.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:35:46.176 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.3 2021-02-10T11:35:46.176 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json-pretty 2021-02-10T11:35:46.640 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:46.623 7fb022ff7700 1 Processor -- start 2021-02-10T11:35:46.641 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:46.623 7fb022ff7700 1 -- start start 2021-02-10T11:35:46.641 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:46.623 7fb022ff7700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb01c0d1870 0x7fb01c1b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:46.641 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:46.623 7fb022ff7700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fb01c1b0a10 0x7fb01c0cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:46.641 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:46.623 7fb022ff7700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb01c0f0b10 con 0x7fb01c0d1c70 2021-02-10T11:35:46.641 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:46.623 7fb022ff7700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb01c0fe230 con 0x7fb01c1b0a10 2021-02-10T11:35:46.641 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:46.623 7fb022ff7700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb01c0e5ed0 con 0x7fb01c0d1870 2021-02-10T11:35:46.641 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:46.624 7fb021614700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fb01c0d1c70 0x7fb01c0cf950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:57630/0 (socket says 172.21.2.102:57630) 2021-02-10T11:35:46.641 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:46.624 7fb021614700 1 -- 172.21.2.102:0/605631218 learned_addr learned my addr 172.21.2.102:0/605631218 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:35:46.641 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:46.624 7fb01bfff700 1 --2- 172.21.2.102:0/605631218 >> v2:172.21.2.102:3300/0 conn(0x7fb01c1b0a10 0x7fb01c0cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:46.641 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:46.624 7fb01bfff700 1 -- 172.21.2.102:0/605631218 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb01c0d1870 msgr2=0x7fb01c1b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:46.641 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:46.624 7fb01bfff700 1 --2- 172.21.2.102:0/605631218 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb01c0d1870 0x7fb01c1b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:46.641 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:46.624 7fb01bfff700 1 -- 172.21.2.102:0/605631218 >> v1:172.21.2.102:6789/0 conn(0x7fb01c0d1c70 legacy=0x7fb01c0cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:46.641 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:46.624 7fb01bfff700 1 -- 172.21.2.102:0/605631218 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb01c0f0650 con 0x7fb01c1b0a10 2021-02-10T11:35:46.642 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:46.624 7fb01b7fe700 1 -- 172.21.2.102:0/605631218 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1111697047 0 0) 0x7fb014001500 con 0x7fb01c0d1c70 2021-02-10T11:35:46.642 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:46.624 7fb01bfff700 1 --2- 172.21.2.102:0/605631218 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb01c1b0a10 0x7fb01c0cf330 secure :-1 s=READY pgs=2720 cs=0 l=1 rev1=1 rx=0x7fb010006650 tx=0x7fb010009a00).ready entity=mon.0 client_cookie=6556e9ae44b5467d server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:46.642 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:46.624 7fb01b7fe700 1 -- 172.21.2.102:0/605631218 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb010009df0 con 0x7fb01c1b0a10 2021-02-10T11:35:46.642 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:46.624 7fb01b7fe700 1 -- 172.21.2.102:0/605631218 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0100009b0 con 0x7fb01c1b0a10 2021-02-10T11:35:46.642 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:46.624 7fb01b7fe700 1 -- 172.21.2.102:0/605631218 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb01000c2b0 con 0x7fb01c1b0a10 2021-02-10T11:35:46.642 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:46.624 7fb022ff7700 1 -- 172.21.2.102:0/605631218 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb01c1b0a10 msgr2=0x7fb01c0cf330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:46.642 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:46.624 7fb022ff7700 1 --2- 172.21.2.102:0/605631218 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb01c1b0a10 0x7fb01c0cf330 secure :-1 s=READY pgs=2720 cs=0 l=1 rev1=1 rx=0x7fb010006650 tx=0x7fb010009a00).stop 2021-02-10T11:35:46.642 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:46.625 7fb022ff7700 1 -- 172.21.2.102:0/605631218 shutdown_connections 2021-02-10T11:35:46.642 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:46.625 7fb022ff7700 1 --2- 172.21.2.102:0/605631218 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb01c1b0a10 0x7fb01c0cf330 unknown :-1 s=CLOSED pgs=2720 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:46.642 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:46.625 7fb022ff7700 1 --2- 172.21.2.102:0/605631218 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb01c0d1870 0x7fb01c1b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:46.642 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:46.625 7fb022ff7700 1 -- 172.21.2.102:0/605631218 >> 172.21.2.102:0/605631218 conn(0x7fb01c0ce450 msgr2=0x7fb01c0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:46.642 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:46.625 7fb022ff7700 1 -- 172.21.2.102:0/605631218 shutdown_connections 2021-02-10T11:35:46.642 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:46.625 7fb022ff7700 1 -- 172.21.2.102:0/605631218 wait complete. 2021-02-10T11:35:46.644 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:46.640 7fb022ff7700 1 Processor -- start 2021-02-10T11:35:46.644 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:46.640 7fb022ff7700 1 -- start start 2021-02-10T11:35:46.644 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:46.640 7fb022ff7700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb01c1ac980 0x7fb01c1ace80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:46.644 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:46.640 7fb022ff7700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb01c0aa400 0x7fb01c1b72d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:46.644 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:46.640 7fb022ff7700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb01c0aa800 0x7fb01c1b78f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:46.644 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:46.640 7fb022ff7700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb01c1b6d50 con 0x7fb01c0aa800 2021-02-10T11:35:46.645 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:46.640 7fb022ff7700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb01c1ad4a0 con 0x7fb01c1ac980 2021-02-10T11:35:46.645 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:46.640 7fb022ff7700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb01c18f0e0 con 0x7fb01c0aa400 2021-02-10T11:35:46.645 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:46.641 7fb021614700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb01c0aa800 0x7fb01c1b78f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:46.645 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:46.641 7fb021614700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb01c0aa800 0x7fb01c1b78f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:37178/0 (socket says 172.21.2.102:37178) 2021-02-10T11:35:46.645 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:46.641 7fb021614700 1 -- 172.21.2.102:0/1782913514 learned_addr learned my addr 172.21.2.102:0/1782913514 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:35:46.645 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:46.641 7fb01bfff700 1 --2- 172.21.2.102:0/1782913514 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb01c0aa400 0x7fb01c1b72d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:46.645 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:46.641 7fb020e13700 1 --2- 172.21.2.102:0/1782913514 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb01c1ac980 0x7fb01c1ace80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:46.645 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:46.641 7fb021614700 1 -- 172.21.2.102:0/1782913514 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb01c0aa400 msgr2=0x7fb01c1b72d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:46.645 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:46.641 7fb021614700 1 --2- 172.21.2.102:0/1782913514 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb01c0aa400 0x7fb01c1b72d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:46.645 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:46.641 7fb021614700 1 -- 172.21.2.102:0/1782913514 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb01c1ac980 msgr2=0x7fb01c1ace80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:46.645 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:46.641 7fb021614700 1 --2- 172.21.2.102:0/1782913514 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb01c1ac980 0x7fb01c1ace80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:46.645 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:46.641 7fb021614700 1 -- 172.21.2.102:0/1782913514 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb01c1ac4c0 con 0x7fb01c0aa800 2021-02-10T11:35:46.646 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:46.641 7fb01bfff700 1 --2- 172.21.2.102:0/1782913514 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb01c0aa400 0x7fb01c1b72d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:35:46.646 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:46.641 7fb021614700 1 --2- 172.21.2.102:0/1782913514 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb01c0aa800 0x7fb01c1b78f0 secure :-1 s=READY pgs=2721 cs=0 l=1 rev1=1 rx=0x7fb014001c10 tx=0x7fb01400ac60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:46.646 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:46.641 7fb0197fa700 1 -- 172.21.2.102:0/1782913514 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb014007020 con 0x7fb01c0aa800 2021-02-10T11:35:46.646 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:46.641 7fb0197fa700 1 -- 172.21.2.102:0/1782913514 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0140072a0 con 0x7fb01c0aa800 2021-02-10T11:35:46.646 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:46.641 7fb0197fa700 1 -- 172.21.2.102:0/1782913514 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb014004720 con 0x7fb01c0aa800 2021-02-10T11:35:46.646 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:46.641 7fb022ff7700 1 -- 172.21.2.102:0/1782913514 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb01c1ac4c0 con 0x7fb01c0aa800 2021-02-10T11:35:46.646 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:46.641 7fb022ff7700 1 -- 172.21.2.102:0/1782913514 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb01c1b8200 con 0x7fb01c0aa800 2021-02-10T11:35:46.646 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:46.642 7fb0197fa700 1 -- 172.21.2.102:0/1782913514 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fb014007480 con 0x7fb01c0aa800 2021-02-10T11:35:46.646 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:46.642 7fb0197fa700 1 --2- 172.21.2.102:0/1782913514 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7faffc0186c0 0x7faffc01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:46.646 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:46.642 7fb0197fa700 1 -- 172.21.2.102:0/1782913514 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fb014032af0 con 0x7fb01c0aa800 2021-02-10T11:35:46.646 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:46.642 7fb020e13700 1 --2- 172.21.2.102:0/1782913514 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7faffc0186c0 0x7faffc01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:46.646 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:46.642 7fb020e13700 1 --2- 172.21.2.102:0/1782913514 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7faffc0186c0 0x7faffc01abc0 crc :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:46.675 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:46.673 7fb022ff7700 1 -- 172.21.2.102:0/1782913514 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb01c0ebc60 con 0x7fb01c0aa800 2021-02-10T11:35:46.677 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:46.676 7fb0197fa700 1 -- 172.21.2.102:0/1782913514 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fb014000f90 con 0x7fb01c0aa800 2021-02-10T11:35:46.800 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:46.799 7fb022ff7700 1 -- 172.21.2.102:0/1782913514 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7fb01c0f0b10 con 0x7fb01c0aa800 2021-02-10T11:35:46.801 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:46.800 7fb0197fa700 1 -- 172.21.2.102:0/1782913514 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v145) v1 ==== 81+0+30374 (secure 0 0 0) 0x7fb0140073d0 con 0x7fb01c0aa800 2021-02-10T11:35:46.801 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:35:46.801 INFO:teuthology.orchestra.run.gibba002.stdout:{ 2021-02-10T11:35:46.801 INFO:teuthology.orchestra.run.gibba002.stdout: "epoch": 145, 2021-02-10T11:35:46.801 INFO:teuthology.orchestra.run.gibba002.stdout: "fsid": "68db30ea-dbce-4aef-a6df-f01c6fea0513", 2021-02-10T11:35:46.802 INFO:teuthology.orchestra.run.gibba002.stdout: "created": "2021-02-10 11:03:06.331557", 2021-02-10T11:35:46.802 INFO:teuthology.orchestra.run.gibba002.stdout: "modified": "2021-02-10 11:35:12.631037", 2021-02-10T11:35:46.802 INFO:teuthology.orchestra.run.gibba002.stdout: "last_up_change": "2021-02-10 11:03:32.690562", 2021-02-10T11:35:46.802 INFO:teuthology.orchestra.run.gibba002.stdout: "last_in_change": "2021-02-10 11:03:32.690562", 2021-02-10T11:35:46.802 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2021-02-10T11:35:46.802 INFO:teuthology.orchestra.run.gibba002.stdout: "flags_num": 5799936, 2021-02-10T11:35:46.802 INFO:teuthology.orchestra.run.gibba002.stdout: "flags_set": [ 2021-02-10T11:35:46.802 INFO:teuthology.orchestra.run.gibba002.stdout: "pglog_hardlimit", 2021-02-10T11:35:46.802 INFO:teuthology.orchestra.run.gibba002.stdout: "purged_snapdirs", 2021-02-10T11:35:46.802 INFO:teuthology.orchestra.run.gibba002.stdout: "recovery_deletes", 2021-02-10T11:35:46.802 INFO:teuthology.orchestra.run.gibba002.stdout: "sortbitwise" 2021-02-10T11:35:46.802 INFO:teuthology.orchestra.run.gibba002.stdout: ], 2021-02-10T11:35:46.803 INFO:teuthology.orchestra.run.gibba002.stdout: "crush_version": 4, 2021-02-10T11:35:46.803 INFO:teuthology.orchestra.run.gibba002.stdout: "full_ratio": 0.94999998807907104, 2021-02-10T11:35:46.803 INFO:teuthology.orchestra.run.gibba002.stdout: "backfillfull_ratio": 0.89999997615814209, 2021-02-10T11:35:46.803 INFO:teuthology.orchestra.run.gibba002.stdout: "nearfull_ratio": 0.85000002384185791, 2021-02-10T11:35:46.803 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_snapshot": "", 2021-02-10T11:35:46.803 INFO:teuthology.orchestra.run.gibba002.stdout: "pool_max": 31, 2021-02-10T11:35:46.803 INFO:teuthology.orchestra.run.gibba002.stdout: "max_osd": 8, 2021-02-10T11:35:46.803 INFO:teuthology.orchestra.run.gibba002.stdout: "require_min_compat_client": "jewel", 2021-02-10T11:35:46.803 INFO:teuthology.orchestra.run.gibba002.stdout: "min_compat_client": "jewel", 2021-02-10T11:35:46.803 INFO:teuthology.orchestra.run.gibba002.stdout: "require_osd_release": "nautilus", 2021-02-10T11:35:46.803 INFO:teuthology.orchestra.run.gibba002.stdout: "pools": [ 2021-02-10T11:35:46.803 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:35:46.803 INFO:teuthology.orchestra.run.gibba002.stdout: "pool": 1, 2021-02-10T11:35:46.804 INFO:teuthology.orchestra.run.gibba002.stdout: "pool_name": "rbd", 2021-02-10T11:35:46.804 INFO:teuthology.orchestra.run.gibba002.stdout: "create_time": "2021-02-10 11:03:37.322519", 2021-02-10T11:35:46.804 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 1, 2021-02-10T11:35:46.804 INFO:teuthology.orchestra.run.gibba002.stdout: "flags_names": "hashpspool", 2021-02-10T11:35:46.804 INFO:teuthology.orchestra.run.gibba002.stdout: "type": 1, 2021-02-10T11:35:46.804 INFO:teuthology.orchestra.run.gibba002.stdout: "size": 2, 2021-02-10T11:35:46.804 INFO:teuthology.orchestra.run.gibba002.stdout: "min_size": 1, 2021-02-10T11:35:46.804 INFO:teuthology.orchestra.run.gibba002.stdout: "crush_rule": 0, 2021-02-10T11:35:46.804 INFO:teuthology.orchestra.run.gibba002.stdout: "object_hash": 2, 2021-02-10T11:35:46.804 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:35:46.804 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num": 8, 2021-02-10T11:35:46.804 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_placement_num": 8, 2021-02-10T11:35:46.804 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_placement_num_target": 8, 2021-02-10T11:35:46.805 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num_target": 8, 2021-02-10T11:35:46.805 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num_pending": 8, 2021-02-10T11:35:46.805 INFO:teuthology.orchestra.run.gibba002.stdout: "last_pg_merge_meta": { 2021-02-10T11:35:46.805 INFO:teuthology.orchestra.run.gibba002.stdout: "source_pgid": "0.0", 2021-02-10T11:35:46.805 INFO:teuthology.orchestra.run.gibba002.stdout: "ready_epoch": 0, 2021-02-10T11:35:46.805 INFO:teuthology.orchestra.run.gibba002.stdout: "last_epoch_started": 0, 2021-02-10T11:35:46.805 INFO:teuthology.orchestra.run.gibba002.stdout: "last_epoch_clean": 0, 2021-02-10T11:35:46.805 INFO:teuthology.orchestra.run.gibba002.stdout: "source_version": "0'0", 2021-02-10T11:35:46.805 INFO:teuthology.orchestra.run.gibba002.stdout: "target_version": "0'0" 2021-02-10T11:35:46.805 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:35:46.806 INFO:teuthology.orchestra.run.gibba002.stdout: "last_change": "16", 2021-02-10T11:35:46.806 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend": "0", 2021-02-10T11:35:46.806 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:35:46.806 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:35:46.806 INFO:teuthology.orchestra.run.gibba002.stdout: "auid": 0, 2021-02-10T11:35:46.806 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:35:46.806 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_seq": 0, 2021-02-10T11:35:46.806 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_epoch": 0, 2021-02-10T11:35:46.806 INFO:teuthology.orchestra.run.gibba002.stdout: "pool_snaps": [], 2021-02-10T11:35:46.806 INFO:teuthology.orchestra.run.gibba002.stdout: "removed_snaps": "[]", 2021-02-10T11:35:46.806 INFO:teuthology.orchestra.run.gibba002.stdout: "quota_max_bytes": 0, 2021-02-10T11:35:46.807 INFO:teuthology.orchestra.run.gibba002.stdout: "quota_max_objects": 0, 2021-02-10T11:35:46.807 INFO:teuthology.orchestra.run.gibba002.stdout: "tiers": [], 2021-02-10T11:35:46.807 INFO:teuthology.orchestra.run.gibba002.stdout: "tier_of": -1, 2021-02-10T11:35:46.807 INFO:teuthology.orchestra.run.gibba002.stdout: "read_tier": -1, 2021-02-10T11:35:46.807 INFO:teuthology.orchestra.run.gibba002.stdout: "write_tier": -1, 2021-02-10T11:35:46.807 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_mode": "none", 2021-02-10T11:35:46.807 INFO:teuthology.orchestra.run.gibba002.stdout: "target_max_bytes": 0, 2021-02-10T11:35:46.807 INFO:teuthology.orchestra.run.gibba002.stdout: "target_max_objects": 0, 2021-02-10T11:35:46.807 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:35:46.807 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:35:46.807 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:35:46.807 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_min_flush_age": 0, 2021-02-10T11:35:46.807 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_min_evict_age": 0, 2021-02-10T11:35:46.808 INFO:teuthology.orchestra.run.gibba002.stdout: "erasure_code_profile": "", 2021-02-10T11:35:46.808 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_params": { 2021-02-10T11:35:46.808 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "none" 2021-02-10T11:35:46.808 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:35:46.808 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_period": 0, 2021-02-10T11:35:46.808 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_count": 0, 2021-02-10T11:35:46.808 INFO:teuthology.orchestra.run.gibba002.stdout: "use_gmt_hitset": true, 2021-02-10T11:35:46.808 INFO:teuthology.orchestra.run.gibba002.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:35:46.808 INFO:teuthology.orchestra.run.gibba002.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:35:46.808 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:35:46.808 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:35:46.808 INFO:teuthology.orchestra.run.gibba002.stdout: "grade_table": [], 2021-02-10T11:35:46.808 INFO:teuthology.orchestra.run.gibba002.stdout: "stripe_width": 0, 2021-02-10T11:35:46.809 INFO:teuthology.orchestra.run.gibba002.stdout: "expected_num_objects": 0, 2021-02-10T11:35:46.809 INFO:teuthology.orchestra.run.gibba002.stdout: "fast_read": false, 2021-02-10T11:35:46.809 INFO:teuthology.orchestra.run.gibba002.stdout: "options": {}, 2021-02-10T11:35:46.809 INFO:teuthology.orchestra.run.gibba002.stdout: "application_metadata": { 2021-02-10T11:35:46.809 INFO:teuthology.orchestra.run.gibba002.stdout: "rbd": {} 2021-02-10T11:35:46.809 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:35:46.809 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:35:46.809 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:35:46.809 INFO:teuthology.orchestra.run.gibba002.stdout: "pool": 30, 2021-02-10T11:35:46.809 INFO:teuthology.orchestra.run.gibba002.stdout: "pool_name": "cephfs_metadata", 2021-02-10T11:35:46.809 INFO:teuthology.orchestra.run.gibba002.stdout: "create_time": "2021-02-10 11:35:09.594644", 2021-02-10T11:35:46.809 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 1, 2021-02-10T11:35:46.809 INFO:teuthology.orchestra.run.gibba002.stdout: "flags_names": "hashpspool", 2021-02-10T11:35:46.810 INFO:teuthology.orchestra.run.gibba002.stdout: "type": 1, 2021-02-10T11:35:46.810 INFO:teuthology.orchestra.run.gibba002.stdout: "size": 2, 2021-02-10T11:35:46.810 INFO:teuthology.orchestra.run.gibba002.stdout: "min_size": 1, 2021-02-10T11:35:46.810 INFO:teuthology.orchestra.run.gibba002.stdout: "crush_rule": 0, 2021-02-10T11:35:46.810 INFO:teuthology.orchestra.run.gibba002.stdout: "object_hash": 2, 2021-02-10T11:35:46.810 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:35:46.810 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num": 8, 2021-02-10T11:35:46.810 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_placement_num": 8, 2021-02-10T11:35:46.810 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_placement_num_target": 8, 2021-02-10T11:35:46.810 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num_target": 8, 2021-02-10T11:35:46.810 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num_pending": 8, 2021-02-10T11:35:46.811 INFO:teuthology.orchestra.run.gibba002.stdout: "last_pg_merge_meta": { 2021-02-10T11:35:46.811 INFO:teuthology.orchestra.run.gibba002.stdout: "source_pgid": "0.0", 2021-02-10T11:35:46.811 INFO:teuthology.orchestra.run.gibba002.stdout: "ready_epoch": 0, 2021-02-10T11:35:46.811 INFO:teuthology.orchestra.run.gibba002.stdout: "last_epoch_started": 0, 2021-02-10T11:35:46.811 INFO:teuthology.orchestra.run.gibba002.stdout: "last_epoch_clean": 0, 2021-02-10T11:35:46.811 INFO:teuthology.orchestra.run.gibba002.stdout: "source_version": "0'0", 2021-02-10T11:35:46.811 INFO:teuthology.orchestra.run.gibba002.stdout: "target_version": "0'0" 2021-02-10T11:35:46.811 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:35:46.811 INFO:teuthology.orchestra.run.gibba002.stdout: "last_change": "144", 2021-02-10T11:35:46.811 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend": "0", 2021-02-10T11:35:46.811 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:35:46.811 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:35:46.812 INFO:teuthology.orchestra.run.gibba002.stdout: "auid": 0, 2021-02-10T11:35:46.812 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:35:46.812 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_seq": 0, 2021-02-10T11:35:46.812 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_epoch": 0, 2021-02-10T11:35:46.812 INFO:teuthology.orchestra.run.gibba002.stdout: "pool_snaps": [], 2021-02-10T11:35:46.812 INFO:teuthology.orchestra.run.gibba002.stdout: "removed_snaps": "[]", 2021-02-10T11:35:46.812 INFO:teuthology.orchestra.run.gibba002.stdout: "quota_max_bytes": 0, 2021-02-10T11:35:46.812 INFO:teuthology.orchestra.run.gibba002.stdout: "quota_max_objects": 0, 2021-02-10T11:35:46.812 INFO:teuthology.orchestra.run.gibba002.stdout: "tiers": [], 2021-02-10T11:35:46.812 INFO:teuthology.orchestra.run.gibba002.stdout: "tier_of": -1, 2021-02-10T11:35:46.812 INFO:teuthology.orchestra.run.gibba002.stdout: "read_tier": -1, 2021-02-10T11:35:46.812 INFO:teuthology.orchestra.run.gibba002.stdout: "write_tier": -1, 2021-02-10T11:35:46.812 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_mode": "none", 2021-02-10T11:35:46.813 INFO:teuthology.orchestra.run.gibba002.stdout: "target_max_bytes": 0, 2021-02-10T11:35:46.813 INFO:teuthology.orchestra.run.gibba002.stdout: "target_max_objects": 0, 2021-02-10T11:35:46.813 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:35:46.813 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:35:46.813 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:35:46.813 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_min_flush_age": 0, 2021-02-10T11:35:46.813 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_min_evict_age": 0, 2021-02-10T11:35:46.813 INFO:teuthology.orchestra.run.gibba002.stdout: "erasure_code_profile": "", 2021-02-10T11:35:46.813 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_params": { 2021-02-10T11:35:46.813 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "none" 2021-02-10T11:35:46.813 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:35:46.813 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_period": 0, 2021-02-10T11:35:46.813 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_count": 0, 2021-02-10T11:35:46.814 INFO:teuthology.orchestra.run.gibba002.stdout: "use_gmt_hitset": true, 2021-02-10T11:35:46.814 INFO:teuthology.orchestra.run.gibba002.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:35:46.814 INFO:teuthology.orchestra.run.gibba002.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:35:46.814 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:35:46.814 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:35:46.814 INFO:teuthology.orchestra.run.gibba002.stdout: "grade_table": [], 2021-02-10T11:35:46.814 INFO:teuthology.orchestra.run.gibba002.stdout: "stripe_width": 0, 2021-02-10T11:35:46.814 INFO:teuthology.orchestra.run.gibba002.stdout: "expected_num_objects": 0, 2021-02-10T11:35:46.814 INFO:teuthology.orchestra.run.gibba002.stdout: "fast_read": false, 2021-02-10T11:35:46.814 INFO:teuthology.orchestra.run.gibba002.stdout: "options": { 2021-02-10T11:35:46.814 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_autoscale_bias": 4, 2021-02-10T11:35:46.814 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num_min": 16, 2021-02-10T11:35:46.814 INFO:teuthology.orchestra.run.gibba002.stdout: "recovery_priority": 5 2021-02-10T11:35:46.815 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:35:46.815 INFO:teuthology.orchestra.run.gibba002.stdout: "application_metadata": { 2021-02-10T11:35:46.815 INFO:teuthology.orchestra.run.gibba002.stdout: "cephfs": { 2021-02-10T11:35:46.815 INFO:teuthology.orchestra.run.gibba002.stdout: "metadata": "cephfs" 2021-02-10T11:35:46.815 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:35:46.815 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:35:46.815 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:35:46.815 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:35:46.815 INFO:teuthology.orchestra.run.gibba002.stdout: "pool": 31, 2021-02-10T11:35:46.815 INFO:teuthology.orchestra.run.gibba002.stdout: "pool_name": "cephfs_data", 2021-02-10T11:35:46.815 INFO:teuthology.orchestra.run.gibba002.stdout: "create_time": "2021-02-10 11:35:10.619592", 2021-02-10T11:35:46.815 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 1, 2021-02-10T11:35:46.815 INFO:teuthology.orchestra.run.gibba002.stdout: "flags_names": "hashpspool", 2021-02-10T11:35:46.816 INFO:teuthology.orchestra.run.gibba002.stdout: "type": 1, 2021-02-10T11:35:46.816 INFO:teuthology.orchestra.run.gibba002.stdout: "size": 2, 2021-02-10T11:35:46.816 INFO:teuthology.orchestra.run.gibba002.stdout: "min_size": 1, 2021-02-10T11:35:46.816 INFO:teuthology.orchestra.run.gibba002.stdout: "crush_rule": 0, 2021-02-10T11:35:46.816 INFO:teuthology.orchestra.run.gibba002.stdout: "object_hash": 2, 2021-02-10T11:35:46.816 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:35:46.816 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num": 8, 2021-02-10T11:35:46.816 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_placement_num": 8, 2021-02-10T11:35:46.816 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_placement_num_target": 8, 2021-02-10T11:35:46.816 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num_target": 8, 2021-02-10T11:35:46.816 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num_pending": 8, 2021-02-10T11:35:46.816 INFO:teuthology.orchestra.run.gibba002.stdout: "last_pg_merge_meta": { 2021-02-10T11:35:46.816 INFO:teuthology.orchestra.run.gibba002.stdout: "source_pgid": "0.0", 2021-02-10T11:35:46.817 INFO:teuthology.orchestra.run.gibba002.stdout: "ready_epoch": 0, 2021-02-10T11:35:46.817 INFO:teuthology.orchestra.run.gibba002.stdout: "last_epoch_started": 0, 2021-02-10T11:35:46.817 INFO:teuthology.orchestra.run.gibba002.stdout: "last_epoch_clean": 0, 2021-02-10T11:35:46.817 INFO:teuthology.orchestra.run.gibba002.stdout: "source_version": "0'0", 2021-02-10T11:35:46.817 INFO:teuthology.orchestra.run.gibba002.stdout: "target_version": "0'0" 2021-02-10T11:35:46.817 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:35:46.817 INFO:teuthology.orchestra.run.gibba002.stdout: "last_change": "145", 2021-02-10T11:35:46.817 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend": "0", 2021-02-10T11:35:46.817 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:35:46.817 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:35:46.817 INFO:teuthology.orchestra.run.gibba002.stdout: "auid": 0, 2021-02-10T11:35:46.817 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:35:46.817 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_seq": 0, 2021-02-10T11:35:46.817 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_epoch": 0, 2021-02-10T11:35:46.818 INFO:teuthology.orchestra.run.gibba002.stdout: "pool_snaps": [], 2021-02-10T11:35:46.818 INFO:teuthology.orchestra.run.gibba002.stdout: "removed_snaps": "[]", 2021-02-10T11:35:46.818 INFO:teuthology.orchestra.run.gibba002.stdout: "quota_max_bytes": 0, 2021-02-10T11:35:46.818 INFO:teuthology.orchestra.run.gibba002.stdout: "quota_max_objects": 0, 2021-02-10T11:35:46.818 INFO:teuthology.orchestra.run.gibba002.stdout: "tiers": [], 2021-02-10T11:35:46.818 INFO:teuthology.orchestra.run.gibba002.stdout: "tier_of": -1, 2021-02-10T11:35:46.818 INFO:teuthology.orchestra.run.gibba002.stdout: "read_tier": -1, 2021-02-10T11:35:46.818 INFO:teuthology.orchestra.run.gibba002.stdout: "write_tier": -1, 2021-02-10T11:35:46.818 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_mode": "none", 2021-02-10T11:35:46.818 INFO:teuthology.orchestra.run.gibba002.stdout: "target_max_bytes": 0, 2021-02-10T11:35:46.818 INFO:teuthology.orchestra.run.gibba002.stdout: "target_max_objects": 0, 2021-02-10T11:35:46.818 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:35:46.819 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:35:46.819 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:35:46.819 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_min_flush_age": 0, 2021-02-10T11:35:46.819 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_min_evict_age": 0, 2021-02-10T11:35:46.819 INFO:teuthology.orchestra.run.gibba002.stdout: "erasure_code_profile": "", 2021-02-10T11:35:46.819 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_params": { 2021-02-10T11:35:46.819 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "none" 2021-02-10T11:35:46.819 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:35:46.819 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_period": 0, 2021-02-10T11:35:46.819 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_count": 0, 2021-02-10T11:35:46.819 INFO:teuthology.orchestra.run.gibba002.stdout: "use_gmt_hitset": true, 2021-02-10T11:35:46.819 INFO:teuthology.orchestra.run.gibba002.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:35:46.819 INFO:teuthology.orchestra.run.gibba002.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:35:46.820 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:35:46.820 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:35:46.820 INFO:teuthology.orchestra.run.gibba002.stdout: "grade_table": [], 2021-02-10T11:35:46.820 INFO:teuthology.orchestra.run.gibba002.stdout: "stripe_width": 0, 2021-02-10T11:35:46.820 INFO:teuthology.orchestra.run.gibba002.stdout: "expected_num_objects": 0, 2021-02-10T11:35:46.820 INFO:teuthology.orchestra.run.gibba002.stdout: "fast_read": false, 2021-02-10T11:35:46.820 INFO:teuthology.orchestra.run.gibba002.stdout: "options": {}, 2021-02-10T11:35:46.820 INFO:teuthology.orchestra.run.gibba002.stdout: "application_metadata": { 2021-02-10T11:35:46.820 INFO:teuthology.orchestra.run.gibba002.stdout: "cephfs": { 2021-02-10T11:35:46.820 INFO:teuthology.orchestra.run.gibba002.stdout: "data": "cephfs" 2021-02-10T11:35:46.820 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:35:46.820 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:35:46.820 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:35:46.820 INFO:teuthology.orchestra.run.gibba002.stdout: ], 2021-02-10T11:35:46.821 INFO:teuthology.orchestra.run.gibba002.stdout: "osds": [ 2021-02-10T11:35:46.821 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:35:46.821 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 0, 2021-02-10T11:35:46.821 INFO:teuthology.orchestra.run.gibba002.stdout: "uuid": "be691b4d-a039-408a-af80-9d53f02b6f1f", 2021-02-10T11:35:46.821 INFO:teuthology.orchestra.run.gibba002.stdout: "up": 1, 2021-02-10T11:35:46.821 INFO:teuthology.orchestra.run.gibba002.stdout: "in": 1, 2021-02-10T11:35:46.821 INFO:teuthology.orchestra.run.gibba002.stdout: "weight": 1, 2021-02-10T11:35:46.821 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_affinity": 1, 2021-02-10T11:35:46.821 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_begin": 0, 2021-02-10T11:35:46.821 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_end": 0, 2021-02-10T11:35:46.821 INFO:teuthology.orchestra.run.gibba002.stdout: "up_from": 13, 2021-02-10T11:35:46.821 INFO:teuthology.orchestra.run.gibba002.stdout: "up_thru": 143, 2021-02-10T11:35:46.822 INFO:teuthology.orchestra.run.gibba002.stdout: "down_at": 0, 2021-02-10T11:35:46.822 INFO:teuthology.orchestra.run.gibba002.stdout: "lost_at": 0, 2021-02-10T11:35:46.822 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addrs": { 2021-02-10T11:35:46.822 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:35:46.822 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:35:46.822 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:35:46.822 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6818", 2021-02-10T11:35:46.822 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36088 2021-02-10T11:35:46.822 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:35:46.822 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:35:46.822 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:35:46.822 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6820", 2021-02-10T11:35:46.822 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36088 2021-02-10T11:35:46.823 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:35:46.823 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:35:46.823 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:35:46.823 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addrs": { 2021-02-10T11:35:46.823 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:35:46.823 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:35:46.823 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:35:46.823 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6822", 2021-02-10T11:35:46.823 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36088 2021-02-10T11:35:46.823 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:35:46.823 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:35:46.824 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:35:46.824 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6824", 2021-02-10T11:35:46.824 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36088 2021-02-10T11:35:46.824 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:35:46.824 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:35:46.824 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:35:46.824 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addrs": { 2021-02-10T11:35:46.824 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:35:46.824 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:35:46.824 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:35:46.824 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6830", 2021-02-10T11:35:46.824 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36088 2021-02-10T11:35:46.824 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:35:46.825 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:35:46.825 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:35:46.825 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6832", 2021-02-10T11:35:46.825 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36088 2021-02-10T11:35:46.825 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:35:46.825 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:35:46.825 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:35:46.825 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addrs": { 2021-02-10T11:35:46.825 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:35:46.825 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:35:46.825 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:35:46.825 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6826", 2021-02-10T11:35:46.825 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36088 2021-02-10T11:35:46.826 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:35:46.826 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:35:46.826 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:35:46.826 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6828", 2021-02-10T11:35:46.826 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36088 2021-02-10T11:35:46.826 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:35:46.826 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:35:46.826 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:35:46.826 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addr": "172.21.2.102:6820/36088", 2021-02-10T11:35:46.826 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addr": "172.21.2.102:6824/36088", 2021-02-10T11:35:46.826 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addr": "172.21.2.102:6832/36088", 2021-02-10T11:35:46.826 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addr": "172.21.2.102:6828/36088", 2021-02-10T11:35:46.826 INFO:teuthology.orchestra.run.gibba002.stdout: "state": [ 2021-02-10T11:35:46.827 INFO:teuthology.orchestra.run.gibba002.stdout: "exists", 2021-02-10T11:35:46.827 INFO:teuthology.orchestra.run.gibba002.stdout: "up" 2021-02-10T11:35:46.827 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:35:46.827 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:35:46.827 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:35:46.827 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 1, 2021-02-10T11:35:46.827 INFO:teuthology.orchestra.run.gibba002.stdout: "uuid": "64e3ccc8-9c69-4d7c-865e-af8e83822635", 2021-02-10T11:35:46.827 INFO:teuthology.orchestra.run.gibba002.stdout: "up": 1, 2021-02-10T11:35:46.827 INFO:teuthology.orchestra.run.gibba002.stdout: "in": 1, 2021-02-10T11:35:46.827 INFO:teuthology.orchestra.run.gibba002.stdout: "weight": 1, 2021-02-10T11:35:46.827 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_affinity": 1, 2021-02-10T11:35:46.827 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_begin": 0, 2021-02-10T11:35:46.827 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_end": 0, 2021-02-10T11:35:46.828 INFO:teuthology.orchestra.run.gibba002.stdout: "up_from": 13, 2021-02-10T11:35:46.828 INFO:teuthology.orchestra.run.gibba002.stdout: "up_thru": 143, 2021-02-10T11:35:46.828 INFO:teuthology.orchestra.run.gibba002.stdout: "down_at": 0, 2021-02-10T11:35:46.828 INFO:teuthology.orchestra.run.gibba002.stdout: "lost_at": 0, 2021-02-10T11:35:46.828 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addrs": { 2021-02-10T11:35:46.828 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:35:46.828 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:35:46.828 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:35:46.828 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6819", 2021-02-10T11:35:46.828 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36087 2021-02-10T11:35:46.828 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:35:46.828 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:35:46.828 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:35:46.829 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6821", 2021-02-10T11:35:46.829 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36087 2021-02-10T11:35:46.829 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:35:46.829 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:35:46.829 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:35:46.829 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addrs": { 2021-02-10T11:35:46.829 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:35:46.829 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:35:46.829 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:35:46.829 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6823", 2021-02-10T11:35:46.829 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36087 2021-02-10T11:35:46.829 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:35:46.829 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:35:46.830 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:35:46.830 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6825", 2021-02-10T11:35:46.830 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36087 2021-02-10T11:35:46.830 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:35:46.830 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:35:46.830 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:35:46.830 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addrs": { 2021-02-10T11:35:46.830 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:35:46.830 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:35:46.830 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:35:46.830 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6831", 2021-02-10T11:35:46.830 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36087 2021-02-10T11:35:46.830 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:35:46.830 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:35:46.831 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:35:46.831 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6833", 2021-02-10T11:35:46.831 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36087 2021-02-10T11:35:46.831 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:35:46.831 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:35:46.831 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:35:46.831 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addrs": { 2021-02-10T11:35:46.831 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:35:46.831 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:35:46.831 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:35:46.831 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6827", 2021-02-10T11:35:46.831 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36087 2021-02-10T11:35:46.831 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:35:46.832 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:35:46.832 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:35:46.832 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6829", 2021-02-10T11:35:46.832 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36087 2021-02-10T11:35:46.832 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:35:46.832 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:35:46.832 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:35:46.832 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addr": "172.21.2.102:6821/36087", 2021-02-10T11:35:46.832 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addr": "172.21.2.102:6825/36087", 2021-02-10T11:35:46.832 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addr": "172.21.2.102:6833/36087", 2021-02-10T11:35:46.832 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addr": "172.21.2.102:6829/36087", 2021-02-10T11:35:46.832 INFO:teuthology.orchestra.run.gibba002.stdout: "state": [ 2021-02-10T11:35:46.832 INFO:teuthology.orchestra.run.gibba002.stdout: "exists", 2021-02-10T11:35:46.833 INFO:teuthology.orchestra.run.gibba002.stdout: "up" 2021-02-10T11:35:46.833 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:35:46.833 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:35:46.833 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:35:46.833 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 2, 2021-02-10T11:35:46.833 INFO:teuthology.orchestra.run.gibba002.stdout: "uuid": "153700af-747f-481f-bdf4-d6072d3a842f", 2021-02-10T11:35:46.833 INFO:teuthology.orchestra.run.gibba002.stdout: "up": 1, 2021-02-10T11:35:46.833 INFO:teuthology.orchestra.run.gibba002.stdout: "in": 1, 2021-02-10T11:35:46.833 INFO:teuthology.orchestra.run.gibba002.stdout: "weight": 1, 2021-02-10T11:35:46.833 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_affinity": 1, 2021-02-10T11:35:46.833 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_begin": 0, 2021-02-10T11:35:46.833 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_end": 0, 2021-02-10T11:35:46.833 INFO:teuthology.orchestra.run.gibba002.stdout: "up_from": 13, 2021-02-10T11:35:46.834 INFO:teuthology.orchestra.run.gibba002.stdout: "up_thru": 142, 2021-02-10T11:35:46.834 INFO:teuthology.orchestra.run.gibba002.stdout: "down_at": 0, 2021-02-10T11:35:46.834 INFO:teuthology.orchestra.run.gibba002.stdout: "lost_at": 0, 2021-02-10T11:35:46.834 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addrs": { 2021-02-10T11:35:46.834 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:35:46.834 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:35:46.834 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:35:46.834 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6810", 2021-02-10T11:35:46.834 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36089 2021-02-10T11:35:46.834 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:35:46.834 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:35:46.835 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:35:46.836 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6811", 2021-02-10T11:35:46.836 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36089 2021-02-10T11:35:46.836 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:35:46.836 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:35:46.836 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:35:46.836 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addrs": { 2021-02-10T11:35:46.836 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:35:46.836 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:35:46.837 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:35:46.837 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6812", 2021-02-10T11:35:46.837 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36089 2021-02-10T11:35:46.837 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:35:46.837 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:35:46.837 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:35:46.837 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6813", 2021-02-10T11:35:46.837 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36089 2021-02-10T11:35:46.837 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:35:46.837 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:35:46.837 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:35:46.837 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addrs": { 2021-02-10T11:35:46.837 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:35:46.838 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:35:46.838 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:35:46.838 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6816", 2021-02-10T11:35:46.838 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36089 2021-02-10T11:35:46.838 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:35:46.838 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:35:46.838 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:35:46.838 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6817", 2021-02-10T11:35:46.838 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36089 2021-02-10T11:35:46.838 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:35:46.839 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:35:46.839 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:35:46.839 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addrs": { 2021-02-10T11:35:46.839 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:35:46.839 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:35:46.839 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:35:46.839 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6814", 2021-02-10T11:35:46.839 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36089 2021-02-10T11:35:46.839 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:35:46.839 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:35:46.839 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:35:46.839 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6815", 2021-02-10T11:35:46.839 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36089 2021-02-10T11:35:46.840 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:35:46.840 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:35:46.840 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:35:46.840 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addr": "172.21.2.102:6811/36089", 2021-02-10T11:35:46.840 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addr": "172.21.2.102:6813/36089", 2021-02-10T11:35:46.840 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addr": "172.21.2.102:6817/36089", 2021-02-10T11:35:46.840 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addr": "172.21.2.102:6815/36089", 2021-02-10T11:35:46.840 INFO:teuthology.orchestra.run.gibba002.stdout: "state": [ 2021-02-10T11:35:46.840 INFO:teuthology.orchestra.run.gibba002.stdout: "exists", 2021-02-10T11:35:46.840 INFO:teuthology.orchestra.run.gibba002.stdout: "up" 2021-02-10T11:35:46.840 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:35:46.840 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:35:46.840 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:35:46.841 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 3, 2021-02-10T11:35:46.841 INFO:teuthology.orchestra.run.gibba002.stdout: "uuid": "cbc678f4-83a9-45e6-8830-2f7c232b1ea7", 2021-02-10T11:35:46.841 INFO:teuthology.orchestra.run.gibba002.stdout: "up": 1, 2021-02-10T11:35:46.841 INFO:teuthology.orchestra.run.gibba002.stdout: "in": 1, 2021-02-10T11:35:46.841 INFO:teuthology.orchestra.run.gibba002.stdout: "weight": 1, 2021-02-10T11:35:46.841 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_affinity": 1, 2021-02-10T11:35:46.841 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_begin": 0, 2021-02-10T11:35:46.841 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_end": 0, 2021-02-10T11:35:46.841 INFO:teuthology.orchestra.run.gibba002.stdout: "up_from": 13, 2021-02-10T11:35:46.841 INFO:teuthology.orchestra.run.gibba002.stdout: "up_thru": 142, 2021-02-10T11:35:46.841 INFO:teuthology.orchestra.run.gibba002.stdout: "down_at": 0, 2021-02-10T11:35:46.841 INFO:teuthology.orchestra.run.gibba002.stdout: "lost_at": 0, 2021-02-10T11:35:46.841 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addrs": { 2021-02-10T11:35:46.841 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:35:46.842 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:35:46.842 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:35:46.842 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6802", 2021-02-10T11:35:46.842 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36090 2021-02-10T11:35:46.842 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:35:46.842 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:35:46.842 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:35:46.842 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6803", 2021-02-10T11:35:46.842 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36090 2021-02-10T11:35:46.842 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:35:46.842 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:35:46.843 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:35:46.843 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addrs": { 2021-02-10T11:35:46.843 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:35:46.843 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:35:46.843 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:35:46.843 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6804", 2021-02-10T11:35:46.843 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36090 2021-02-10T11:35:46.843 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:35:46.843 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:35:46.843 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:35:46.843 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6805", 2021-02-10T11:35:46.843 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36090 2021-02-10T11:35:46.844 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:35:46.844 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:35:46.844 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:35:46.844 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addrs": { 2021-02-10T11:35:46.844 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:35:46.844 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:35:46.844 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:35:46.844 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6808", 2021-02-10T11:35:46.844 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36090 2021-02-10T11:35:46.844 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:35:46.844 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:35:46.845 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:35:46.845 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6809", 2021-02-10T11:35:46.845 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36090 2021-02-10T11:35:46.845 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:35:46.845 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:35:46.845 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:35:46.845 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addrs": { 2021-02-10T11:35:46.845 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:35:46.845 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:35:46.845 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:35:46.845 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6806", 2021-02-10T11:35:46.845 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36090 2021-02-10T11:35:46.845 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:35:46.846 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:35:46.846 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:35:46.846 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6807", 2021-02-10T11:35:46.846 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36090 2021-02-10T11:35:46.846 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:35:46.846 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:35:46.846 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:35:46.846 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addr": "172.21.2.102:6803/36090", 2021-02-10T11:35:46.846 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addr": "172.21.2.102:6805/36090", 2021-02-10T11:35:46.846 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addr": "172.21.2.102:6809/36090", 2021-02-10T11:35:46.846 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addr": "172.21.2.102:6807/36090", 2021-02-10T11:35:46.846 INFO:teuthology.orchestra.run.gibba002.stdout: "state": [ 2021-02-10T11:35:46.846 INFO:teuthology.orchestra.run.gibba002.stdout: "exists", 2021-02-10T11:35:46.846 INFO:teuthology.orchestra.run.gibba002.stdout: "up" 2021-02-10T11:35:46.847 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:35:46.847 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:35:46.847 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:35:46.847 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 4, 2021-02-10T11:35:46.847 INFO:teuthology.orchestra.run.gibba002.stdout: "uuid": "c01e4008-b258-4f10-afd2-9e7f37214351", 2021-02-10T11:35:46.847 INFO:teuthology.orchestra.run.gibba002.stdout: "up": 1, 2021-02-10T11:35:46.847 INFO:teuthology.orchestra.run.gibba002.stdout: "in": 1, 2021-02-10T11:35:46.847 INFO:teuthology.orchestra.run.gibba002.stdout: "weight": 1, 2021-02-10T11:35:46.847 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_affinity": 1, 2021-02-10T11:35:46.847 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_begin": 0, 2021-02-10T11:35:46.847 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_end": 0, 2021-02-10T11:35:46.847 INFO:teuthology.orchestra.run.gibba002.stdout: "up_from": 13, 2021-02-10T11:35:46.847 INFO:teuthology.orchestra.run.gibba002.stdout: "up_thru": 143, 2021-02-10T11:35:46.848 INFO:teuthology.orchestra.run.gibba002.stdout: "down_at": 0, 2021-02-10T11:35:46.848 INFO:teuthology.orchestra.run.gibba002.stdout: "lost_at": 0, 2021-02-10T11:35:46.848 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addrs": { 2021-02-10T11:35:46.848 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:35:46.848 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:35:46.848 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:35:46.848 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6808", 2021-02-10T11:35:46.848 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36368 2021-02-10T11:35:46.848 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:35:46.848 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:35:46.848 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:35:46.848 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6811", 2021-02-10T11:35:46.848 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36368 2021-02-10T11:35:46.849 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:35:46.849 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:35:46.849 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:35:46.849 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addrs": { 2021-02-10T11:35:46.849 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:35:46.849 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:35:46.849 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:35:46.849 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6814", 2021-02-10T11:35:46.849 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36368 2021-02-10T11:35:46.849 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:35:46.850 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:35:46.850 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:35:46.850 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6818", 2021-02-10T11:35:46.850 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36368 2021-02-10T11:35:46.850 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:35:46.850 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:35:46.850 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:35:46.850 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addrs": { 2021-02-10T11:35:46.850 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:35:46.851 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:35:46.851 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:35:46.851 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6826", 2021-02-10T11:35:46.851 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36368 2021-02-10T11:35:46.851 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:35:46.851 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:35:46.851 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:35:46.852 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6828", 2021-02-10T11:35:46.852 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36368 2021-02-10T11:35:46.852 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:35:46.852 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:35:46.852 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:35:46.852 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addrs": { 2021-02-10T11:35:46.852 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:35:46.852 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:35:46.852 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:35:46.852 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6820", 2021-02-10T11:35:46.852 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36368 2021-02-10T11:35:46.852 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:35:46.852 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:35:46.853 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:35:46.853 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6824", 2021-02-10T11:35:46.853 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36368 2021-02-10T11:35:46.853 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:35:46.853 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:35:46.853 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:35:46.853 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addr": "172.21.2.113:6811/36368", 2021-02-10T11:35:46.853 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addr": "172.21.2.113:6818/36368", 2021-02-10T11:35:46.853 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addr": "172.21.2.113:6828/36368", 2021-02-10T11:35:46.853 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addr": "172.21.2.113:6824/36368", 2021-02-10T11:35:46.853 INFO:teuthology.orchestra.run.gibba002.stdout: "state": [ 2021-02-10T11:35:46.853 INFO:teuthology.orchestra.run.gibba002.stdout: "exists", 2021-02-10T11:35:46.853 INFO:teuthology.orchestra.run.gibba002.stdout: "up" 2021-02-10T11:35:46.854 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:35:46.854 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:35:46.854 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:35:46.854 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 5, 2021-02-10T11:35:46.854 INFO:teuthology.orchestra.run.gibba002.stdout: "uuid": "551f6007-daf8-4a98-8ee0-c2b16bcccb5e", 2021-02-10T11:35:46.854 INFO:teuthology.orchestra.run.gibba002.stdout: "up": 1, 2021-02-10T11:35:46.854 INFO:teuthology.orchestra.run.gibba002.stdout: "in": 1, 2021-02-10T11:35:46.854 INFO:teuthology.orchestra.run.gibba002.stdout: "weight": 1, 2021-02-10T11:35:46.854 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_affinity": 1, 2021-02-10T11:35:46.854 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_begin": 0, 2021-02-10T11:35:46.854 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_end": 0, 2021-02-10T11:35:46.854 INFO:teuthology.orchestra.run.gibba002.stdout: "up_from": 13, 2021-02-10T11:35:46.855 INFO:teuthology.orchestra.run.gibba002.stdout: "up_thru": 142, 2021-02-10T11:35:46.855 INFO:teuthology.orchestra.run.gibba002.stdout: "down_at": 0, 2021-02-10T11:35:46.855 INFO:teuthology.orchestra.run.gibba002.stdout: "lost_at": 0, 2021-02-10T11:35:46.855 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addrs": { 2021-02-10T11:35:46.855 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:35:46.855 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:35:46.855 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:35:46.855 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6802", 2021-02-10T11:35:46.855 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36366 2021-02-10T11:35:46.855 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:35:46.855 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:35:46.855 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:35:46.856 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6805", 2021-02-10T11:35:46.856 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36366 2021-02-10T11:35:46.856 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:35:46.856 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:35:46.856 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:35:46.856 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addrs": { 2021-02-10T11:35:46.856 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:35:46.857 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:35:46.857 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:35:46.857 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6807", 2021-02-10T11:35:46.857 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36366 2021-02-10T11:35:46.857 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:35:46.857 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:35:46.857 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:35:46.857 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6810", 2021-02-10T11:35:46.857 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36366 2021-02-10T11:35:46.858 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:35:46.858 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:35:46.858 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:35:46.858 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addrs": { 2021-02-10T11:35:46.858 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:35:46.858 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:35:46.858 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:35:46.858 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6819", 2021-02-10T11:35:46.858 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36366 2021-02-10T11:35:46.858 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:35:46.858 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:35:46.858 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:35:46.859 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6822", 2021-02-10T11:35:46.859 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36366 2021-02-10T11:35:46.859 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:35:46.859 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:35:46.859 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:35:46.859 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addrs": { 2021-02-10T11:35:46.859 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:35:46.859 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:35:46.859 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:35:46.859 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6812", 2021-02-10T11:35:46.859 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36366 2021-02-10T11:35:46.859 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:35:46.859 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:35:46.860 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:35:46.860 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6815", 2021-02-10T11:35:46.860 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36366 2021-02-10T11:35:46.860 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:35:46.860 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:35:46.860 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:35:46.860 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addr": "172.21.2.113:6805/36366", 2021-02-10T11:35:46.860 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addr": "172.21.2.113:6810/36366", 2021-02-10T11:35:46.860 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addr": "172.21.2.113:6822/36366", 2021-02-10T11:35:46.860 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addr": "172.21.2.113:6815/36366", 2021-02-10T11:35:46.860 INFO:teuthology.orchestra.run.gibba002.stdout: "state": [ 2021-02-10T11:35:46.860 INFO:teuthology.orchestra.run.gibba002.stdout: "exists", 2021-02-10T11:35:46.860 INFO:teuthology.orchestra.run.gibba002.stdout: "up" 2021-02-10T11:35:46.860 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:35:46.861 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:35:46.861 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:35:46.861 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 6, 2021-02-10T11:35:46.861 INFO:teuthology.orchestra.run.gibba002.stdout: "uuid": "bf67c040-0e50-48be-b102-c3943bc9ebef", 2021-02-10T11:35:46.861 INFO:teuthology.orchestra.run.gibba002.stdout: "up": 1, 2021-02-10T11:35:46.861 INFO:teuthology.orchestra.run.gibba002.stdout: "in": 1, 2021-02-10T11:35:46.861 INFO:teuthology.orchestra.run.gibba002.stdout: "weight": 1, 2021-02-10T11:35:46.861 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_affinity": 1, 2021-02-10T11:35:46.861 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_begin": 0, 2021-02-10T11:35:46.861 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_end": 0, 2021-02-10T11:35:46.861 INFO:teuthology.orchestra.run.gibba002.stdout: "up_from": 13, 2021-02-10T11:35:46.861 INFO:teuthology.orchestra.run.gibba002.stdout: "up_thru": 143, 2021-02-10T11:35:46.861 INFO:teuthology.orchestra.run.gibba002.stdout: "down_at": 0, 2021-02-10T11:35:46.862 INFO:teuthology.orchestra.run.gibba002.stdout: "lost_at": 0, 2021-02-10T11:35:46.862 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addrs": { 2021-02-10T11:35:46.862 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:35:46.862 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:35:46.862 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:35:46.862 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6800", 2021-02-10T11:35:46.862 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36367 2021-02-10T11:35:46.862 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:35:46.862 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:35:46.862 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:35:46.862 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6801", 2021-02-10T11:35:46.862 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36367 2021-02-10T11:35:46.863 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:35:46.863 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:35:46.863 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:35:46.863 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addrs": { 2021-02-10T11:35:46.863 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:35:46.863 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:35:46.863 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:35:46.863 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6803", 2021-02-10T11:35:46.863 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36367 2021-02-10T11:35:46.863 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:35:46.863 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:35:46.863 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:35:46.864 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6804", 2021-02-10T11:35:46.864 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36367 2021-02-10T11:35:46.864 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:35:46.864 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:35:46.864 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:35:46.864 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addrs": { 2021-02-10T11:35:46.864 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:35:46.864 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:35:46.864 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:35:46.864 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6813", 2021-02-10T11:35:46.864 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36367 2021-02-10T11:35:46.864 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:35:46.865 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:35:46.865 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:35:46.865 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6816", 2021-02-10T11:35:46.865 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36367 2021-02-10T11:35:46.865 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:35:46.865 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:35:46.865 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:35:46.865 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addrs": { 2021-02-10T11:35:46.865 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:35:46.865 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:35:46.865 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:35:46.865 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6806", 2021-02-10T11:35:46.866 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36367 2021-02-10T11:35:46.866 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:35:46.866 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:35:46.866 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:35:46.866 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6809", 2021-02-10T11:35:46.866 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36367 2021-02-10T11:35:46.866 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:35:46.866 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:35:46.866 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:35:46.866 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addr": "172.21.2.113:6801/36367", 2021-02-10T11:35:46.866 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addr": "172.21.2.113:6804/36367", 2021-02-10T11:35:46.866 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addr": "172.21.2.113:6816/36367", 2021-02-10T11:35:46.867 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addr": "172.21.2.113:6809/36367", 2021-02-10T11:35:46.867 INFO:teuthology.orchestra.run.gibba002.stdout: "state": [ 2021-02-10T11:35:46.867 INFO:teuthology.orchestra.run.gibba002.stdout: "exists", 2021-02-10T11:35:46.867 INFO:teuthology.orchestra.run.gibba002.stdout: "up" 2021-02-10T11:35:46.867 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:35:46.867 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:35:46.867 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:35:46.867 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 7, 2021-02-10T11:35:46.867 INFO:teuthology.orchestra.run.gibba002.stdout: "uuid": "f5461f80-018d-428f-be89-2eb2bcbe7bb6", 2021-02-10T11:35:46.867 INFO:teuthology.orchestra.run.gibba002.stdout: "up": 1, 2021-02-10T11:35:46.867 INFO:teuthology.orchestra.run.gibba002.stdout: "in": 1, 2021-02-10T11:35:46.867 INFO:teuthology.orchestra.run.gibba002.stdout: "weight": 1, 2021-02-10T11:35:46.867 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_affinity": 1, 2021-02-10T11:35:46.868 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_begin": 0, 2021-02-10T11:35:46.868 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_end": 0, 2021-02-10T11:35:46.868 INFO:teuthology.orchestra.run.gibba002.stdout: "up_from": 13, 2021-02-10T11:35:46.868 INFO:teuthology.orchestra.run.gibba002.stdout: "up_thru": 142, 2021-02-10T11:35:46.868 INFO:teuthology.orchestra.run.gibba002.stdout: "down_at": 0, 2021-02-10T11:35:46.868 INFO:teuthology.orchestra.run.gibba002.stdout: "lost_at": 0, 2021-02-10T11:35:46.868 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addrs": { 2021-02-10T11:35:46.868 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:35:46.868 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:35:46.868 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:35:46.868 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6817", 2021-02-10T11:35:46.869 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36365 2021-02-10T11:35:46.869 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:35:46.869 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:35:46.869 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:35:46.869 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6821", 2021-02-10T11:35:46.869 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36365 2021-02-10T11:35:46.869 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:35:46.869 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:35:46.869 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:35:46.869 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addrs": { 2021-02-10T11:35:46.869 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:35:46.869 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:35:46.869 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:35:46.870 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6823", 2021-02-10T11:35:46.870 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36365 2021-02-10T11:35:46.870 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:35:46.870 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:35:46.870 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:35:46.870 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6825", 2021-02-10T11:35:46.870 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36365 2021-02-10T11:35:46.870 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:35:46.870 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:35:46.870 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:35:46.870 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addrs": { 2021-02-10T11:35:46.871 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:35:46.871 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:35:46.871 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:35:46.871 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6830", 2021-02-10T11:35:46.871 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36365 2021-02-10T11:35:46.871 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:35:46.871 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:35:46.871 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:35:46.871 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6831", 2021-02-10T11:35:46.871 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36365 2021-02-10T11:35:46.871 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:35:46.871 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:35:46.872 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:35:46.872 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addrs": { 2021-02-10T11:35:46.872 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:35:46.872 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:35:46.872 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:35:46.872 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6827", 2021-02-10T11:35:46.872 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36365 2021-02-10T11:35:46.872 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:35:46.872 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:35:46.872 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:35:46.872 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6829", 2021-02-10T11:35:46.872 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36365 2021-02-10T11:35:46.873 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:35:46.873 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:35:46.873 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:35:46.873 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addr": "172.21.2.113:6821/36365", 2021-02-10T11:35:46.873 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addr": "172.21.2.113:6825/36365", 2021-02-10T11:35:46.873 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addr": "172.21.2.113:6831/36365", 2021-02-10T11:35:46.873 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addr": "172.21.2.113:6829/36365", 2021-02-10T11:35:46.873 INFO:teuthology.orchestra.run.gibba002.stdout: "state": [ 2021-02-10T11:35:46.873 INFO:teuthology.orchestra.run.gibba002.stdout: "exists", 2021-02-10T11:35:46.873 INFO:teuthology.orchestra.run.gibba002.stdout: "up" 2021-02-10T11:35:46.873 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:35:46.873 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:35:46.874 INFO:teuthology.orchestra.run.gibba002.stdout: ], 2021-02-10T11:35:46.874 INFO:teuthology.orchestra.run.gibba002.stdout: "osd_xinfo": [ 2021-02-10T11:35:46.874 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:35:46.874 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 0, 2021-02-10T11:35:46.874 INFO:teuthology.orchestra.run.gibba002.stdout: "down_stamp": "0.000000", 2021-02-10T11:35:46.874 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_probability": 0, 2021-02-10T11:35:46.874 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_interval": 0, 2021-02-10T11:35:46.874 INFO:teuthology.orchestra.run.gibba002.stdout: "features": 4611087854035861503, 2021-02-10T11:35:46.874 INFO:teuthology.orchestra.run.gibba002.stdout: "old_weight": 0 2021-02-10T11:35:46.874 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:35:46.874 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:35:46.874 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 1, 2021-02-10T11:35:46.875 INFO:teuthology.orchestra.run.gibba002.stdout: "down_stamp": "0.000000", 2021-02-10T11:35:46.875 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_probability": 0, 2021-02-10T11:35:46.875 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_interval": 0, 2021-02-10T11:35:46.875 INFO:teuthology.orchestra.run.gibba002.stdout: "features": 4611087854035861503, 2021-02-10T11:35:46.875 INFO:teuthology.orchestra.run.gibba002.stdout: "old_weight": 0 2021-02-10T11:35:46.875 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:35:46.875 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:35:46.875 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 2, 2021-02-10T11:35:46.875 INFO:teuthology.orchestra.run.gibba002.stdout: "down_stamp": "0.000000", 2021-02-10T11:35:46.875 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_probability": 0, 2021-02-10T11:35:46.875 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_interval": 0, 2021-02-10T11:35:46.875 INFO:teuthology.orchestra.run.gibba002.stdout: "features": 4611087854035861503, 2021-02-10T11:35:46.876 INFO:teuthology.orchestra.run.gibba002.stdout: "old_weight": 0 2021-02-10T11:35:46.876 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:35:46.876 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:35:46.876 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 3, 2021-02-10T11:35:46.876 INFO:teuthology.orchestra.run.gibba002.stdout: "down_stamp": "0.000000", 2021-02-10T11:35:46.876 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_probability": 0, 2021-02-10T11:35:46.876 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_interval": 0, 2021-02-10T11:35:46.876 INFO:teuthology.orchestra.run.gibba002.stdout: "features": 4611087854035861503, 2021-02-10T11:35:46.876 INFO:teuthology.orchestra.run.gibba002.stdout: "old_weight": 0 2021-02-10T11:35:46.876 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:35:46.876 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:35:46.876 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 4, 2021-02-10T11:35:46.877 INFO:teuthology.orchestra.run.gibba002.stdout: "down_stamp": "0.000000", 2021-02-10T11:35:46.877 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_probability": 0, 2021-02-10T11:35:46.877 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_interval": 0, 2021-02-10T11:35:46.877 INFO:teuthology.orchestra.run.gibba002.stdout: "features": 4611087854035861503, 2021-02-10T11:35:46.877 INFO:teuthology.orchestra.run.gibba002.stdout: "old_weight": 0 2021-02-10T11:35:46.877 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:35:46.877 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:35:46.877 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 5, 2021-02-10T11:35:46.877 INFO:teuthology.orchestra.run.gibba002.stdout: "down_stamp": "0.000000", 2021-02-10T11:35:46.877 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_probability": 0, 2021-02-10T11:35:46.877 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_interval": 0, 2021-02-10T11:35:46.877 INFO:teuthology.orchestra.run.gibba002.stdout: "features": 4611087854035861503, 2021-02-10T11:35:46.877 INFO:teuthology.orchestra.run.gibba002.stdout: "old_weight": 0 2021-02-10T11:35:46.877 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:35:46.878 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:35:46.878 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 6, 2021-02-10T11:35:46.878 INFO:teuthology.orchestra.run.gibba002.stdout: "down_stamp": "0.000000", 2021-02-10T11:35:46.878 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_probability": 0, 2021-02-10T11:35:46.878 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_interval": 0, 2021-02-10T11:35:46.878 INFO:teuthology.orchestra.run.gibba002.stdout: "features": 4611087854035861503, 2021-02-10T11:35:46.878 INFO:teuthology.orchestra.run.gibba002.stdout: "old_weight": 0 2021-02-10T11:35:46.878 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:35:46.878 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:35:46.878 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 7, 2021-02-10T11:35:46.878 INFO:teuthology.orchestra.run.gibba002.stdout: "down_stamp": "0.000000", 2021-02-10T11:35:46.878 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_probability": 0, 2021-02-10T11:35:46.878 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_interval": 0, 2021-02-10T11:35:46.879 INFO:teuthology.orchestra.run.gibba002.stdout: "features": 4611087854035861503, 2021-02-10T11:35:46.879 INFO:teuthology.orchestra.run.gibba002.stdout: "old_weight": 0 2021-02-10T11:35:46.879 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:35:46.879 INFO:teuthology.orchestra.run.gibba002.stdout: ], 2021-02-10T11:35:46.879 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_upmap": [], 2021-02-10T11:35:46.879 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_upmap_items": [], 2021-02-10T11:35:46.879 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_temp": [], 2021-02-10T11:35:46.879 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_temp": [], 2021-02-10T11:35:46.879 INFO:teuthology.orchestra.run.gibba002.stdout: "blacklist": {}, 2021-02-10T11:35:46.879 INFO:teuthology.orchestra.run.gibba002.stdout: "erasure_code_profiles": { 2021-02-10T11:35:46.879 INFO:teuthology.orchestra.run.gibba002.stdout: "default": { 2021-02-10T11:35:46.879 INFO:teuthology.orchestra.run.gibba002.stdout: "crush-failure-domain": "osd", 2021-02-10T11:35:46.879 INFO:teuthology.orchestra.run.gibba002.stdout: "k": "2", 2021-02-10T11:35:46.880 INFO:teuthology.orchestra.run.gibba002.stdout: "m": "1", 2021-02-10T11:35:46.880 INFO:teuthology.orchestra.run.gibba002.stdout: "plugin": "jerasure", 2021-02-10T11:35:46.880 INFO:teuthology.orchestra.run.gibba002.stdout: "ruleset-failure-domain": "osd", 2021-02-10T11:35:46.880 INFO:teuthology.orchestra.run.gibba002.stdout: "technique": "reed_sol_van" 2021-02-10T11:35:46.880 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:35:46.880 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:35:46.880 INFO:teuthology.orchestra.run.gibba002.stdout: "removed_snaps_queue": [], 2021-02-10T11:35:46.880 INFO:teuthology.orchestra.run.gibba002.stdout: "new_removed_snaps": [], 2021-02-10T11:35:46.880 INFO:teuthology.orchestra.run.gibba002.stdout: "new_purged_snaps": [], 2021-02-10T11:35:46.880 INFO:teuthology.orchestra.run.gibba002.stdout: "crush_node_flags": {}, 2021-02-10T11:35:46.880 INFO:teuthology.orchestra.run.gibba002.stdout: "device_class_flags": {} 2021-02-10T11:35:46.880 INFO:teuthology.orchestra.run.gibba002.stdout:} 2021-02-10T11:35:46.882 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:46.801 7fb022ff7700 1 -- 172.21.2.102:0/1782913514 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7faffc0186c0 msgr2=0x7faffc01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:46.882 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:46.801 7fb022ff7700 1 --2- 172.21.2.102:0/1782913514 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7faffc0186c0 0x7faffc01abc0 crc :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:46.882 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:46.801 7fb022ff7700 1 -- 172.21.2.102:0/1782913514 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb01c0aa800 msgr2=0x7fb01c1b78f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:46.882 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:46.801 7fb022ff7700 1 --2- 172.21.2.102:0/1782913514 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb01c0aa800 0x7fb01c1b78f0 secure :-1 s=READY pgs=2721 cs=0 l=1 rev1=1 rx=0x7fb014001c10 tx=0x7fb01400ac60).stop 2021-02-10T11:35:46.882 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:46.801 7fb022ff7700 1 -- 172.21.2.102:0/1782913514 shutdown_connections 2021-02-10T11:35:46.882 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:46.801 7fb022ff7700 1 --2- 172.21.2.102:0/1782913514 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb01c0aa800 0x7fb01c1b78f0 unknown :-1 s=CLOSED pgs=2721 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:46.882 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:46.801 7fb022ff7700 1 --2- 172.21.2.102:0/1782913514 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb01c0aa400 0x7fb01c1b72d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:46.882 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:46.802 7fb022ff7700 1 --2- 172.21.2.102:0/1782913514 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7faffc0186c0 0x7faffc01abc0 unknown :-1 s=CLOSED pgs=1196 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:46.882 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:46.802 7fb022ff7700 1 --2- 172.21.2.102:0/1782913514 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb01c1ac980 0x7fb01c1ace80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:46.882 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:46.802 7fb022ff7700 1 -- 172.21.2.102:0/1782913514 >> 172.21.2.102:0/1782913514 conn(0x7fb01c1ac040 msgr2=0x7fb01c094980 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:46.883 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:46.802 7fb022ff7700 1 -- 172.21.2.102:0/1782913514 shutdown_connections 2021-02-10T11:35:46.883 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:46.802 7fb022ff7700 1 -- 172.21.2.102:0/1782913514 wait complete. 2021-02-10T11:35:46.883 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:35:47.311 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:47.294 7f01f5ffc700 1 Processor -- start 2021-02-10T11:35:47.311 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:47.295 7f01f5ffc700 1 -- start start 2021-02-10T11:35:47.311 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:47.295 7f01f5ffc700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f01f00d1870 0x7f01f01b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:47.312 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:47.295 7f01f5ffc700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f01f01b0a10 0x7f01f00cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:47.312 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:47.295 7f01f5ffc700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f01f00d1c70 0x7f01f00cf950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:47.312 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:47.295 7f01f5ffc700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01f00e5ed0 con 0x7f01f00d1870 2021-02-10T11:35:47.312 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:47.295 7f01f5ffc700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01f00f0b10 con 0x7f01f01b0a10 2021-02-10T11:35:47.312 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:47.295 7f01f5ffc700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01f00cff70 con 0x7f01f00d1c70 2021-02-10T11:35:47.312 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:47.295 7f01ef7fe700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f01f00d1870 0x7f01f01b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:47.312 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:47.295 7f01ef7fe700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f01f00d1870 0x7f01f01b03f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:37186/0 (socket says 172.21.2.102:37186) 2021-02-10T11:35:47.312 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:47.295 7f01ef7fe700 1 -- 172.21.2.102:0/2461192951 learned_addr learned my addr 172.21.2.102:0/2461192951 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:35:47.312 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:47.295 7f01ef7fe700 1 -- 172.21.2.102:0/2461192951 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f01f00d1c70 msgr2=0x7f01f00cf950 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:35:47.312 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:47.295 7f01ef7fe700 1 --2- 172.21.2.102:0/2461192951 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f01f00d1c70 0x7f01f00cf950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:47.312 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:47.295 7f01ef7fe700 1 -- 172.21.2.102:0/2461192951 >> v2:172.21.2.113:3300/0 conn(0x7f01f01b0a10 msgr2=0x7f01f00cf330 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:35:47.312 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:47.295 7f01ef7fe700 1 --2- 172.21.2.102:0/2461192951 >> v2:172.21.2.113:3300/0 conn(0x7f01f01b0a10 0x7f01f00cf330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:47.312 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:47.295 7f01ef7fe700 1 -- 172.21.2.102:0/2461192951 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01f00d0150 con 0x7f01f00d1870 2021-02-10T11:35:47.313 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:47.296 7f01ef7fe700 1 --2- 172.21.2.102:0/2461192951 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f01f00d1870 0x7f01f01b03f0 secure :-1 s=READY pgs=2722 cs=0 l=1 rev1=1 rx=0x7f01d8006700 tx=0x7f01d8009b80).ready entity=mon.0 client_cookie=10ea90c16be12f70 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:47.313 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:47.296 7f01ee7fc700 1 -- 172.21.2.102:0/2461192951 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f01d800c0c0 con 0x7f01f00d1870 2021-02-10T11:35:47.313 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:47.296 7f01ee7fc700 1 -- 172.21.2.102:0/2461192951 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01d800c340 con 0x7f01f00d1870 2021-02-10T11:35:47.313 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:47.296 7f01ee7fc700 1 -- 172.21.2.102:0/2461192951 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f01d800c860 con 0x7f01f00d1870 2021-02-10T11:35:47.313 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:47.296 7f01f5ffc700 1 -- 172.21.2.102:0/2461192951 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f01f00d1870 msgr2=0x7f01f01b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:47.313 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:47.296 7f01f5ffc700 1 --2- 172.21.2.102:0/2461192951 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f01f00d1870 0x7f01f01b03f0 secure :-1 s=READY pgs=2722 cs=0 l=1 rev1=1 rx=0x7f01d8006700 tx=0x7f01d8009b80).stop 2021-02-10T11:35:47.313 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:47.296 7f01f5ffc700 1 -- 172.21.2.102:0/2461192951 shutdown_connections 2021-02-10T11:35:47.313 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:47.296 7f01f5ffc700 1 --2- 172.21.2.102:0/2461192951 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f01f00d1c70 0x7f01f00cf950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:47.313 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:47.296 7f01f5ffc700 1 --2- 172.21.2.102:0/2461192951 >> v2:172.21.2.113:3300/0 conn(0x7f01f01b0a10 0x7f01f00cf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:47.313 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:47.296 7f01f5ffc700 1 --2- 172.21.2.102:0/2461192951 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f01f00d1870 0x7f01f01b03f0 unknown :-1 s=CLOSED pgs=2722 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:47.313 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:47.296 7f01f5ffc700 1 -- 172.21.2.102:0/2461192951 >> 172.21.2.102:0/2461192951 conn(0x7f01f00ce450 msgr2=0x7f01f00d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:47.313 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:47.296 7f01f5ffc700 1 -- 172.21.2.102:0/2461192951 shutdown_connections 2021-02-10T11:35:47.313 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:47.296 7f01f5ffc700 1 -- 172.21.2.102:0/2461192951 wait complete. 2021-02-10T11:35:47.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:47.311 7f01f5ffc700 1 Processor -- start 2021-02-10T11:35:47.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:47.311 7f01f5ffc700 1 -- start start 2021-02-10T11:35:47.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:47.311 7f01f5ffc700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f01f01aec10 0x7f01f01a7910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:47.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:47.311 7f01f5ffc700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f01f01aa2a0 0x7f01f00b1950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:47.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:47.311 7f01f5ffc700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f01f00b1f70 0x7f01f01aac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:47.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:47.311 7f01f5ffc700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01f01aa6e0 con 0x7f01f01aec10 2021-02-10T11:35:47.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:47.311 7f01f5ffc700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01f01af010 con 0x7f01f01aa2a0 2021-02-10T11:35:47.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:47.311 7f01f5ffc700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01f01a7f30 con 0x7f01f00b1f70 2021-02-10T11:35:47.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:47.311 7f01ef7fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f01f01aec10 0x7f01f01a7910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:47.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:47.311 7f01ef7fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f01f01aec10 0x7f01f01a7910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:37190/0 (socket says 172.21.2.102:37190) 2021-02-10T11:35:47.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:47.311 7f01ef7fe700 1 -- 172.21.2.102:0/2557368906 learned_addr learned my addr 172.21.2.102:0/2557368906 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:35:47.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:47.312 7f01ef7fe700 1 -- 172.21.2.102:0/2557368906 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f01f00b1f70 msgr2=0x7f01f01aac60 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:35:47.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:47.312 7f01ef7fe700 1 --2- 172.21.2.102:0/2557368906 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f01f00b1f70 0x7f01f01aac60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:47.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:47.312 7f01ef7fe700 1 -- 172.21.2.102:0/2557368906 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f01f01aa2a0 msgr2=0x7f01f00b1950 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:35:47.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:47.312 7f01ef7fe700 1 --2- 172.21.2.102:0/2557368906 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f01f01aa2a0 0x7f01f00b1950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:47.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:47.312 7f01ef7fe700 1 -- 172.21.2.102:0/2557368906 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01f00b2370 con 0x7f01f01aec10 2021-02-10T11:35:47.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:47.312 7f01ef7fe700 1 --2- 172.21.2.102:0/2557368906 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f01f01aec10 0x7f01f01a7910 secure :-1 s=READY pgs=2723 cs=0 l=1 rev1=1 rx=0x7f01d800b5e0 tx=0x7f01d800b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:47.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:47.312 7f01e7fff700 1 -- 172.21.2.102:0/2557368906 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f01d800bd70 con 0x7f01f01aec10 2021-02-10T11:35:47.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:47.312 7f01e7fff700 1 -- 172.21.2.102:0/2557368906 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01d8002030 con 0x7f01f01aec10 2021-02-10T11:35:47.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:47.312 7f01e7fff700 1 -- 172.21.2.102:0/2557368906 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f01d8002550 con 0x7f01f01aec10 2021-02-10T11:35:47.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:47.312 7f01f5ffc700 1 -- 172.21.2.102:0/2557368906 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01f00b2370 con 0x7f01f01aec10 2021-02-10T11:35:47.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:47.312 7f01f5ffc700 1 -- 172.21.2.102:0/2557368906 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01f01ab280 con 0x7f01f01aec10 2021-02-10T11:35:47.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:47.313 7f01e7fff700 1 -- 172.21.2.102:0/2557368906 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f01d8006020 con 0x7f01f01aec10 2021-02-10T11:35:47.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:47.313 7f01e7fff700 1 --2- 172.21.2.102:0/2557368906 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f01d40186c0 0x7f01d401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:47.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:47.313 7f01e7fff700 1 -- 172.21.2.102:0/2557368906 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f01d8032530 con 0x7f01f01aec10 2021-02-10T11:35:47.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:47.313 7f01eeffd700 1 --2- 172.21.2.102:0/2557368906 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f01d40186c0 0x7f01d401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:47.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:47.313 7f01eeffd700 1 --2- 172.21.2.102:0/2557368906 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f01d40186c0 0x7f01d401abc0 crc :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:47.346 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:47.345 7f01f5ffc700 1 -- 172.21.2.102:0/2557368906 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01f00f0b10 con 0x7f01f01aec10 2021-02-10T11:35:47.348 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:47.347 7f01e7fff700 1 -- 172.21.2.102:0/2557368906 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f01d80009b0 con 0x7f01f01aec10 2021-02-10T11:35:47.478 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:47.476 7f01f5ffc700 1 -- 172.21.2.102:0/2557368906 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f01f00f0b10 con 0x7f01f01aec10 2021-02-10T11:35:47.478 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:47.477 7f01e7fff700 1 -- 172.21.2.102:0/2557368906 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 125 v125) v1 ==== 95+0+2111 (secure 0 0 0) 0x7f01d8006020 con 0x7f01f01aec10 2021-02-10T11:35:47.479 INFO:teuthology.orchestra.run.gibba002.stderr:dumped fsmap epoch 125 2021-02-10T11:35:47.479 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:35:47.479 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":125,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":11784,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.102:6835/2097086088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6834","nonce":2097086088},{"type":"v1","addr":"172.21.2.102:6835","nonce":2097086088}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":120}],"filesystems":[{"mdsmap":{"epoch":124,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:35:11.628704","modified":"2021-02-10 11:35:13.639277","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11781},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11781":{"gid":11781,"name":"b","rank":0,"incarnation":121,"state":"up:active","state_seq":3,"addr":"172.21.2.113:6833/3334075470","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6832","nonce":3334075470},{"type":"v1","addr":"172.21.2.113:6833","nonce":3334075470}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[31],"metadata_pool":30,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":15}]} 2021-02-10T11:35:47.480 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:47.479 7f01f5ffc700 1 -- 172.21.2.102:0/2557368906 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f01d40186c0 msgr2=0x7f01d401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:47.480 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:47.479 7f01f5ffc700 1 --2- 172.21.2.102:0/2557368906 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f01d40186c0 0x7f01d401abc0 crc :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:47.480 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:47.479 7f01f5ffc700 1 -- 172.21.2.102:0/2557368906 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f01f01aec10 msgr2=0x7f01f01a7910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:47.480 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:47.479 7f01f5ffc700 1 --2- 172.21.2.102:0/2557368906 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f01f01aec10 0x7f01f01a7910 secure :-1 s=READY pgs=2723 cs=0 l=1 rev1=1 rx=0x7f01d800b5e0 tx=0x7f01d800b980).stop 2021-02-10T11:35:47.480 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:47.479 7f01f5ffc700 1 -- 172.21.2.102:0/2557368906 shutdown_connections 2021-02-10T11:35:47.480 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:47.479 7f01f5ffc700 1 --2- 172.21.2.102:0/2557368906 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f01f00b1f70 0x7f01f01aac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:47.481 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:47.479 7f01f5ffc700 1 --2- 172.21.2.102:0/2557368906 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f01d40186c0 0x7f01d401abc0 unknown :-1 s=CLOSED pgs=1197 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:47.481 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:47.479 7f01f5ffc700 1 --2- 172.21.2.102:0/2557368906 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f01f01aa2a0 0x7f01f00b1950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:47.481 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:47.479 7f01f5ffc700 1 --2- 172.21.2.102:0/2557368906 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f01f01aec10 0x7f01f01a7910 unknown :-1 s=CLOSED pgs=2723 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:47.481 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:47.479 7f01f5ffc700 1 -- 172.21.2.102:0/2557368906 >> 172.21.2.102:0/2557368906 conn(0x7f01f00ed870 msgr2=0x7f01f00edd10 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:47.481 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:47.480 7f01f5ffc700 1 -- 172.21.2.102:0/2557368906 shutdown_connections 2021-02-10T11:35:47.481 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:47.480 7f01f5ffc700 1 -- 172.21.2.102:0/2557368906 wait complete. 2021-02-10T11:35:47.489 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs fail cephfs 2021-02-10T11:35:47.918 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:47.901 7f9e06de6700 1 Processor -- start 2021-02-10T11:35:47.919 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:47.901 7f9e06de6700 1 -- start start 2021-02-10T11:35:47.919 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:47.901 7f9e06de6700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9e000d1870 0x7f9e001b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:47.919 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:47.901 7f9e06de6700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f9e001b0a10 0x7f9e000cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:47.919 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:47.901 7f9e06de6700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e000f0b10 con 0x7f9e000d1c70 2021-02-10T11:35:47.919 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:47.901 7f9e06de6700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e000fe230 con 0x7f9e001b0a10 2021-02-10T11:35:47.919 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:47.901 7f9e06de6700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e000e5ed0 con 0x7f9e000d1870 2021-02-10T11:35:47.919 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:47.902 7f9e05403700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f9e000d1c70 0x7f9e000cf950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:57656/0 (socket says 172.21.2.102:57656) 2021-02-10T11:35:47.919 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:47.902 7f9e05403700 1 -- 172.21.2.102:0/227724083 learned_addr learned my addr 172.21.2.102:0/227724083 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:35:47.919 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:47.902 7f9dfffff700 1 --2- 172.21.2.102:0/227724083 >> v2:172.21.2.102:3300/0 conn(0x7f9e001b0a10 0x7f9e000cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:47.919 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:47.902 7f9dff7fe700 1 -- 172.21.2.102:0/227724083 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1352098000 0 0) 0x7f9df8001480 con 0x7f9e000d1c70 2021-02-10T11:35:47.919 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:47.902 7f9dff7fe700 1 -- 172.21.2.102:0/227724083 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9dec002fe0 con 0x7f9e000d1c70 2021-02-10T11:35:47.920 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:47.902 7f9dfffff700 1 -- 172.21.2.102:0/227724083 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9e000d1870 msgr2=0x7f9e001b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:47.920 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:47.902 7f9dfffff700 1 --2- 172.21.2.102:0/227724083 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9e000d1870 0x7f9e001b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:47.920 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:47.902 7f9dfffff700 1 -- 172.21.2.102:0/227724083 >> v1:172.21.2.102:6789/0 conn(0x7f9e000d1c70 legacy=0x7f9e000cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:47.920 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:47.902 7f9dfffff700 1 -- 172.21.2.102:0/227724083 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e000f0650 con 0x7f9e001b0a10 2021-02-10T11:35:47.920 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:47.902 7f9dfffff700 1 --2- 172.21.2.102:0/227724083 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9e001b0a10 0x7f9e000cf330 secure :-1 s=READY pgs=2725 cs=0 l=1 rev1=1 rx=0x7f9df0006650 tx=0x7f9df0009a80).ready entity=mon.0 client_cookie=d15c4cb2bff12bfc server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:47.920 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:47.902 7f9dff7fe700 1 -- 172.21.2.102:0/227724083 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9df000c0c0 con 0x7f9e001b0a10 2021-02-10T11:35:47.920 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:47.903 7f9dff7fe700 1 -- 172.21.2.102:0/227724083 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9df000c340 con 0x7f9e001b0a10 2021-02-10T11:35:47.920 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:47.903 7f9dff7fe700 1 -- 172.21.2.102:0/227724083 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9df000c860 con 0x7f9e001b0a10 2021-02-10T11:35:47.920 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:47.903 7f9e06de6700 1 -- 172.21.2.102:0/227724083 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9e001b0a10 msgr2=0x7f9e000cf330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:47.920 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:47.903 7f9e06de6700 1 --2- 172.21.2.102:0/227724083 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9e001b0a10 0x7f9e000cf330 secure :-1 s=READY pgs=2725 cs=0 l=1 rev1=1 rx=0x7f9df0006650 tx=0x7f9df0009a80).stop 2021-02-10T11:35:47.920 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:47.903 7f9e06de6700 1 -- 172.21.2.102:0/227724083 shutdown_connections 2021-02-10T11:35:47.920 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:47.903 7f9e06de6700 1 --2- 172.21.2.102:0/227724083 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9e001b0a10 0x7f9e000cf330 unknown :-1 s=CLOSED pgs=2725 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:47.920 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:47.903 7f9e06de6700 1 --2- 172.21.2.102:0/227724083 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9e000d1870 0x7f9e001b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:47.921 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:47.903 7f9e06de6700 1 -- 172.21.2.102:0/227724083 >> 172.21.2.102:0/227724083 conn(0x7f9e000ce450 msgr2=0x7f9e000d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:47.921 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:47.903 7f9e06de6700 1 -- 172.21.2.102:0/227724083 shutdown_connections 2021-02-10T11:35:47.921 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:47.903 7f9e06de6700 1 -- 172.21.2.102:0/227724083 wait complete. 2021-02-10T11:35:47.922 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:47.918 7f9e06de6700 1 Processor -- start 2021-02-10T11:35:47.922 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:47.918 7f9e06de6700 1 -- start start 2021-02-10T11:35:47.922 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:47.918 7f9e06de6700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9e000b7510 0x7f9e000b7910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:47.922 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:47.918 7f9e06de6700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f9e0009d530 0x7f9e0009d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:47.922 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:47.918 7f9e06de6700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9e0009df50 0x7f9e0009eca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:47.923 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:47.918 7f9e06de6700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e000b8140 con 0x7f9e000b7510 2021-02-10T11:35:47.923 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:47.918 7f9e06de6700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e0009e750 con 0x7f9e0009d530 2021-02-10T11:35:47.923 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:47.918 7f9e06de6700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e0009e350 con 0x7f9e0009df50 2021-02-10T11:35:47.923 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:47.919 7f9e04c02700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9e000b7510 0x7f9e000b7910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:47.923 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:47.919 7f9e04c02700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9e000b7510 0x7f9e000b7910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:37204/0 (socket says 172.21.2.102:37204) 2021-02-10T11:35:47.923 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:47.919 7f9e04c02700 1 -- 172.21.2.102:0/2853483669 learned_addr learned my addr 172.21.2.102:0/2853483669 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:35:47.923 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:47.919 7f9e04c02700 1 -- 172.21.2.102:0/2853483669 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9e0009df50 msgr2=0x7f9e0009eca0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:35:47.923 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:47.919 7f9e04c02700 1 --2- 172.21.2.102:0/2853483669 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9e0009df50 0x7f9e0009eca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:47.923 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:47.919 7f9e04c02700 1 -- 172.21.2.102:0/2853483669 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f9e0009d530 msgr2=0x7f9e0009d930 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:35:47.923 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:47.919 7f9e04c02700 1 --2- 172.21.2.102:0/2853483669 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f9e0009d530 0x7f9e0009d930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:47.923 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:47.919 7f9e04c02700 1 -- 172.21.2.102:0/2853483669 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e0009f2c0 con 0x7f9e000b7510 2021-02-10T11:35:47.923 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:47.919 7f9e04c02700 1 --2- 172.21.2.102:0/2853483669 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9e000b7510 0x7f9e000b7910 secure :-1 s=READY pgs=2726 cs=0 l=1 rev1=1 rx=0x7f9de8006f00 tx=0x7f9de800b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:47.924 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:47.919 7f9dfd7fa700 1 -- 172.21.2.102:0/2853483669 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9de80009b0 con 0x7f9e000b7510 2021-02-10T11:35:47.924 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:47.919 7f9dfd7fa700 1 -- 172.21.2.102:0/2853483669 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9de800bdc0 con 0x7f9e000b7510 2021-02-10T11:35:47.924 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:47.919 7f9dfd7fa700 1 -- 172.21.2.102:0/2853483669 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9de8002020 con 0x7f9e000b7510 2021-02-10T11:35:47.924 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:47.919 7f9e06de6700 1 -- 172.21.2.102:0/2853483669 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e0009f2c0 con 0x7f9e000b7510 2021-02-10T11:35:47.924 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:47.919 7f9e06de6700 1 -- 172.21.2.102:0/2853483669 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e0009f850 con 0x7f9e000b7510 2021-02-10T11:35:47.924 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:47.920 7f9dfd7fa700 1 -- 172.21.2.102:0/2853483669 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f9de8005310 con 0x7f9e000b7510 2021-02-10T11:35:47.924 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:47.920 7f9dfd7fa700 1 --2- 172.21.2.102:0/2853483669 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f9de40186c0 0x7f9de401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:47.924 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:47.920 7f9dfd7fa700 1 -- 172.21.2.102:0/2853483669 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f9de8032520 con 0x7f9e000b7510 2021-02-10T11:35:47.924 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:47.920 7f9dfffff700 1 --2- 172.21.2.102:0/2853483669 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f9de40186c0 0x7f9de401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:47.924 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:47.920 7f9dfffff700 1 --2- 172.21.2.102:0/2853483669 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f9de40186c0 0x7f9de401abc0 crc :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:47.952 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:47.951 7f9e06de6700 1 -- 172.21.2.102:0/2853483669 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e000ebc50 con 0x7f9e000b7510 2021-02-10T11:35:47.954 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:47.954 7f9dfd7fa700 1 -- 172.21.2.102:0/2853483669 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f9de80009b0 con 0x7f9e000b7510 2021-02-10T11:35:48.081 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:48.080 7f9e06de6700 1 -- 172.21.2.102:0/2853483669 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f9e000f0b10 con 0x7f9e000b7510 2021-02-10T11:35:48.154 INFO:tasks.ceph.mon.a.gibba002.stderr:2021-02-10 11:35:48.153 7f8dd2755700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-02-10T11:35:48.159 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:48.158 7f9dfd7fa700 1 -- 172.21.2.102:0/2853483669 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v126) v1 ==== 161+0+0 (secure 0 0 0) 0x7f9de8005310 con 0x7f9e000b7510 2021-02-10T11:35:48.195 INFO:teuthology.orchestra.run.gibba002.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2021-02-10T11:35:48.198 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:48.196 7f9e06de6700 1 -- 172.21.2.102:0/2853483669 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f9de40186c0 msgr2=0x7f9de401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:48.198 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:48.196 7f9e06de6700 1 --2- 172.21.2.102:0/2853483669 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f9de40186c0 0x7f9de401abc0 crc :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:48.198 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:48.196 7f9e06de6700 1 -- 172.21.2.102:0/2853483669 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9e000b7510 msgr2=0x7f9e000b7910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:48.198 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:48.196 7f9e06de6700 1 --2- 172.21.2.102:0/2853483669 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9e000b7510 0x7f9e000b7910 secure :-1 s=READY pgs=2726 cs=0 l=1 rev1=1 rx=0x7f9de8006f00 tx=0x7f9de800b940).stop 2021-02-10T11:35:48.198 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:48.197 7f9e06de6700 1 -- 172.21.2.102:0/2853483669 shutdown_connections 2021-02-10T11:35:48.198 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:48.197 7f9e06de6700 1 --2- 172.21.2.102:0/2853483669 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9e0009df50 0x7f9e0009eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:48.198 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:48.197 7f9e06de6700 1 --2- 172.21.2.102:0/2853483669 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f9de40186c0 0x7f9de401abc0 unknown :-1 s=CLOSED pgs=1198 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:48.198 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:48.197 7f9e06de6700 1 --2- 172.21.2.102:0/2853483669 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f9e0009d530 0x7f9e0009d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:48.198 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:48.197 7f9e06de6700 1 --2- 172.21.2.102:0/2853483669 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9e000b7510 0x7f9e000b7910 unknown :-1 s=CLOSED pgs=2726 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:48.199 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:48.197 7f9e06de6700 1 -- 172.21.2.102:0/2853483669 >> 172.21.2.102:0/2853483669 conn(0x7f9e000b83b0 msgr2=0x7f9e000a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:48.199 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:48.197 7f9e06de6700 1 -- 172.21.2.102:0/2853483669 shutdown_connections 2021-02-10T11:35:48.199 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:48.197 7f9e06de6700 1 -- 172.21.2.102:0/2853483669 wait complete. 2021-02-10T11:35:48.203 INFO:tasks.ceph.mds.b.gibba013.stdout:starting mds.b at 2021-02-10T11:35:48.206 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:35:48.637 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:48.619 7f21bac74700 1 Processor -- start 2021-02-10T11:35:48.637 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:48.619 7f21bac74700 1 -- start start 2021-02-10T11:35:48.637 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:48.619 7f21bac74700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f21b40d0340 0x7f21b40ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:48.637 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:48.619 7f21bac74700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f21b40d1c70 0x7f21b40efc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:48.637 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:48.619 7f21bac74700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21b40f0b10 con 0x7f21b40d1870 2021-02-10T11:35:48.637 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:48.619 7f21bac74700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21b40e5ed0 con 0x7f21b40d1c70 2021-02-10T11:35:48.638 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:48.619 7f21bac74700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21b41b0be0 con 0x7f21b40d0340 2021-02-10T11:35:48.638 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:48.620 7f21b9291700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f21b40d1c70 0x7f21b40efc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:48.638 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:48.620 7f21b9291700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f21b40d1c70 0x7f21b40efc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.113:3300/0 says I am v2:172.21.2.102:40640/0 (socket says 172.21.2.102:40640) 2021-02-10T11:35:48.638 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:48.620 7f21b9291700 1 -- 172.21.2.102:0/297125350 learned_addr learned my addr 172.21.2.102:0/297125350 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:35:48.638 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:48.620 7f21b3fff700 1 --2- 172.21.2.102:0/297125350 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f21b40d0340 0x7f21b40ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:48.638 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:48.620 7f21b9291700 1 -- 172.21.2.102:0/297125350 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f21b40d0340 msgr2=0x7f21b40ef640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:48.638 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:48.620 7f21b9291700 1 --2- 172.21.2.102:0/297125350 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f21b40d0340 0x7f21b40ef640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:48.638 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:48.620 7f21b9291700 1 -- 172.21.2.102:0/297125350 >> v1:172.21.2.113:6789/0 conn(0x7f21b40d1870 legacy=0x7f21b41b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:48.638 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:48.620 7f21b9291700 1 -- 172.21.2.102:0/297125350 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21b40f20a0 con 0x7f21b40d1c70 2021-02-10T11:35:48.638 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:48.620 7f21b9291700 1 --2- 172.21.2.102:0/297125350 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f21b40d1c70 0x7f21b40efc60 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 rx=0x7f21ac006aa0 tx=0x7f21ac009f20).ready entity=mon.1 client_cookie=656ac7e8f52bb4a0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:48.638 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:48.621 7f21b37fe700 1 -- 172.21.2.102:0/297125350 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f21ac0066f0 con 0x7f21b40d1c70 2021-02-10T11:35:48.638 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:48.621 7f21b37fe700 1 -- 172.21.2.102:0/297125350 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21ac003030 con 0x7f21b40d1c70 2021-02-10T11:35:48.639 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:48.621 7f21b37fe700 1 -- 172.21.2.102:0/297125350 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f21ac00b7b0 con 0x7f21b40d1c70 2021-02-10T11:35:48.639 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:48.621 7f21bac74700 1 -- 172.21.2.102:0/297125350 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f21b40d1c70 msgr2=0x7f21b40efc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:48.639 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:48.621 7f21bac74700 1 --2- 172.21.2.102:0/297125350 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f21b40d1c70 0x7f21b40efc60 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 rx=0x7f21ac006aa0 tx=0x7f21ac009f20).stop 2021-02-10T11:35:48.639 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:48.622 7f21bac74700 1 -- 172.21.2.102:0/297125350 shutdown_connections 2021-02-10T11:35:48.639 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:48.622 7f21bac74700 1 --2- 172.21.2.102:0/297125350 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f21b40d0340 0x7f21b40ef640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:48.639 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:48.622 7f21bac74700 1 --2- 172.21.2.102:0/297125350 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f21b40d1c70 0x7f21b40efc60 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:48.639 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:48.622 7f21bac74700 1 -- 172.21.2.102:0/297125350 >> 172.21.2.102:0/297125350 conn(0x7f21b40ce450 msgr2=0x7f21b40d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:48.639 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:48.622 7f21bac74700 1 -- 172.21.2.102:0/297125350 shutdown_connections 2021-02-10T11:35:48.639 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:48.622 7f21bac74700 1 -- 172.21.2.102:0/297125350 wait complete. 2021-02-10T11:35:48.641 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:48.636 7f21bac74700 1 Processor -- start 2021-02-10T11:35:48.641 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:48.636 7f21bac74700 1 -- start start 2021-02-10T11:35:48.641 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:48.637 7f21bac74700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f21b40970b0 0x7f21b40ebe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:48.641 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:48.637 7f21bac74700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f21b4098810 0x7f21b4098c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:48.641 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:48.637 7f21bac74700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f21b4099230 0x7f21b40b18c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:48.641 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:48.637 7f21bac74700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21b40ec6d0 con 0x7f21b40970b0 2021-02-10T11:35:48.641 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:48.637 7f21bac74700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21b40974b0 con 0x7f21b4098810 2021-02-10T11:35:48.641 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:48.637 7f21bac74700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21b40b1ee0 con 0x7f21b4099230 2021-02-10T11:35:48.641 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:48.637 7f21b8a90700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f21b40970b0 0x7f21b40ebe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:48.641 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:48.637 7f21b8a90700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f21b40970b0 0x7f21b40ebe30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:37218/0 (socket says 172.21.2.102:37218) 2021-02-10T11:35:48.641 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:48.637 7f21b8a90700 1 -- 172.21.2.102:0/564807086 learned_addr learned my addr 172.21.2.102:0/564807086 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:35:48.642 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:48.637 7f21b3fff700 1 --2- 172.21.2.102:0/564807086 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f21b4098810 0x7f21b4098c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:48.642 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:48.637 7f21b9291700 1 --2- 172.21.2.102:0/564807086 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f21b4099230 0x7f21b40b18c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:48.642 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:48.637 7f21b8a90700 1 -- 172.21.2.102:0/564807086 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f21b4099230 msgr2=0x7f21b40b18c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:48.642 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:48.637 7f21b8a90700 1 --2- 172.21.2.102:0/564807086 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f21b4099230 0x7f21b40b18c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:48.642 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:48.637 7f21b8a90700 1 -- 172.21.2.102:0/564807086 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f21b4098810 msgr2=0x7f21b4098c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:48.642 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:48.637 7f21b8a90700 1 --2- 172.21.2.102:0/564807086 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f21b4098810 0x7f21b4098c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:48.642 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:48.637 7f21b8a90700 1 -- 172.21.2.102:0/564807086 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21b40b2100 con 0x7f21b40970b0 2021-02-10T11:35:48.642 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:48.638 7f21b8a90700 1 --2- 172.21.2.102:0/564807086 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f21b40970b0 0x7f21b40ebe30 secure :-1 s=READY pgs=2728 cs=0 l=1 rev1=1 rx=0x7f219c006ed0 tx=0x7f219c00b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:48.642 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:48.638 7f21b17fa700 1 -- 172.21.2.102:0/564807086 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f219c006020 con 0x7f21b40970b0 2021-02-10T11:35:48.642 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:48.638 7f21b17fa700 1 -- 172.21.2.102:0/564807086 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f219c0062a0 con 0x7f21b40970b0 2021-02-10T11:35:48.642 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:48.638 7f21b17fa700 1 -- 172.21.2.102:0/564807086 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f219c005730 con 0x7f21b40970b0 2021-02-10T11:35:48.643 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:48.638 7f21bac74700 1 -- 172.21.2.102:0/564807086 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21b40b2100 con 0x7f21b40970b0 2021-02-10T11:35:48.643 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:48.638 7f21bac74700 1 -- 172.21.2.102:0/564807086 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21b40b2100 con 0x7f21b40970b0 2021-02-10T11:35:48.643 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:48.639 7f21b17fa700 1 -- 172.21.2.102:0/564807086 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f219c0054b0 con 0x7f21b40970b0 2021-02-10T11:35:48.643 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:48.639 7f21b17fa700 1 --2- 172.21.2.102:0/564807086 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f21980186c0 0x7f219801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:48.643 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:48.639 7f21b17fa700 1 -- 172.21.2.102:0/564807086 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(146..146 src has 1..146) v4 ==== 5659+0+0 (secure 0 0 0) 0x7f219c0325d0 con 0x7f21b40970b0 2021-02-10T11:35:48.643 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:48.639 7f21b3fff700 1 --2- 172.21.2.102:0/564807086 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f21980186c0 0x7f219801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:48.643 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:48.639 7f21b3fff700 1 --2- 172.21.2.102:0/564807086 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f21980186c0 0x7f219801abc0 crc :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:48.670 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:48.669 7f21bac74700 1 -- 172.21.2.102:0/564807086 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21b40ebc00 con 0x7f21b40970b0 2021-02-10T11:35:48.672 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:48.672 7f21b17fa700 1 -- 172.21.2.102:0/564807086 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f219c02dee0 con 0x7f21b40970b0 2021-02-10T11:35:48.800 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:48.799 7f21bac74700 1 -- 172.21.2.102:0/564807086 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f21b40ebc00 con 0x7f21b40970b0 2021-02-10T11:35:48.801 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:48.800 7f21b17fa700 1 -- 172.21.2.102:0/564807086 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 126 v126) v1 ==== 95+0+1763 (secure 0 0 0) 0x7f219c02dee0 con 0x7f21b40970b0 2021-02-10T11:35:48.802 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:35:48.802 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":126,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":11784,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.102:6835/2097086088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6834","nonce":2097086088},{"type":"v1","addr":"172.21.2.102:6835","nonce":2097086088}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":120}],"filesystems":[{"mdsmap":{"epoch":126,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:35:11.628704","modified":"2021-02-10 11:35:48.153175","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":146,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[31],"metadata_pool":30,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":15}]} 2021-02-10T11:35:48.802 INFO:teuthology.orchestra.run.gibba002.stderr:dumped fsmap epoch 126 2021-02-10T11:35:48.803 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:48.802 7f21bac74700 1 -- 172.21.2.102:0/564807086 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f21980186c0 msgr2=0x7f219801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:48.803 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:48.802 7f21bac74700 1 --2- 172.21.2.102:0/564807086 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f21980186c0 0x7f219801abc0 crc :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:48.803 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:48.802 7f21bac74700 1 -- 172.21.2.102:0/564807086 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f21b40970b0 msgr2=0x7f21b40ebe30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:48.803 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:48.802 7f21bac74700 1 --2- 172.21.2.102:0/564807086 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f21b40970b0 0x7f21b40ebe30 secure :-1 s=READY pgs=2728 cs=0 l=1 rev1=1 rx=0x7f219c006ed0 tx=0x7f219c00b960).stop 2021-02-10T11:35:48.803 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:48.802 7f21bac74700 1 -- 172.21.2.102:0/564807086 shutdown_connections 2021-02-10T11:35:48.803 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:48.802 7f21bac74700 1 --2- 172.21.2.102:0/564807086 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f21b4099230 0x7f21b40b18c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:48.804 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:48.802 7f21bac74700 1 --2- 172.21.2.102:0/564807086 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f21980186c0 0x7f219801abc0 unknown :-1 s=CLOSED pgs=1199 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:48.804 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:48.802 7f21bac74700 1 --2- 172.21.2.102:0/564807086 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f21b4098810 0x7f21b4098c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:48.804 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:48.802 7f21bac74700 1 --2- 172.21.2.102:0/564807086 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f21b40970b0 0x7f21b40ebe30 unknown :-1 s=CLOSED pgs=2728 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:48.804 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:48.802 7f21bac74700 1 -- 172.21.2.102:0/564807086 >> 172.21.2.102:0/564807086 conn(0x7f21b419a730 msgr2=0x7f21b419e480 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:48.804 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:48.802 7f21bac74700 1 -- 172.21.2.102:0/564807086 shutdown_connections 2021-02-10T11:35:48.804 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:48.802 7f21bac74700 1 -- 172.21.2.102:0/564807086 wait complete. 2021-02-10T11:35:48.812 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2021-02-10T11:35:49.246 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:49.229 7fcfc85e2700 1 Processor -- start 2021-02-10T11:35:49.246 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:49.229 7fcfc85e2700 1 -- start start 2021-02-10T11:35:49.246 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:49.229 7fcfc85e2700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fcfc00d1870 0x7fcfc01b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:49.247 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:49.229 7fcfc85e2700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcfc00f0650 0x7fcfc00cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:49.247 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:49.230 7fcfc85e2700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcfc00f0b10 con 0x7fcfc01b0a10 2021-02-10T11:35:49.247 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:49.230 7fcfc85e2700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcfc00e5ed0 con 0x7fcfc00d1870 2021-02-10T11:35:49.247 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:49.230 7fcfc85e2700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfc00d1c70 con 0x7fcfc00f0650 2021-02-10T11:35:49.247 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:49.230 7fcfc63fe700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fcfc00d1870 0x7fcfc01b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:49.247 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:49.230 7fcfc63fe700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fcfc00d1870 0x7fcfc01b03f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:37222/0 (socket says 172.21.2.102:37222) 2021-02-10T11:35:49.247 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:49.230 7fcfc63fe700 1 -- 172.21.2.102:0/118734373 learned_addr learned my addr 172.21.2.102:0/118734373 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:35:49.247 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:49.230 7fcfc63fe700 1 -- 172.21.2.102:0/118734373 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcfc00f0650 msgr2=0x7fcfc00cfb20 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:35:49.247 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:49.230 7fcfc63fe700 1 --2- 172.21.2.102:0/118734373 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcfc00f0650 0x7fcfc00cfb20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:49.247 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:49.230 7fcfc63fe700 1 -- 172.21.2.102:0/118734373 >> v1:172.21.2.113:6789/0 conn(0x7fcfc01b0a10 legacy=0x7fcfc00cf330 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:35:49.247 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:49.230 7fcfc63fe700 1 -- 172.21.2.102:0/118734373 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfc00d1e90 con 0x7fcfc00d1870 2021-02-10T11:35:49.247 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:49.230 7fcfc63fe700 1 --2- 172.21.2.102:0/118734373 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcfc00d1870 0x7fcfc01b03f0 secure :-1 s=READY pgs=2729 cs=0 l=1 rev1=1 rx=0x7fcfbc006700 tx=0x7fcfbc009b80).ready entity=mon.0 client_cookie=ada0797a4e8f8c33 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:49.247 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:49.230 7fcfc53fc700 1 -- 172.21.2.102:0/118734373 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcfbc00c0c0 con 0x7fcfc00d1870 2021-02-10T11:35:49.248 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:49.230 7fcfc53fc700 1 -- 172.21.2.102:0/118734373 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfbc00c340 con 0x7fcfc00d1870 2021-02-10T11:35:49.248 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:49.230 7fcfc53fc700 1 -- 172.21.2.102:0/118734373 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcfbc00c860 con 0x7fcfc00d1870 2021-02-10T11:35:49.248 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:49.230 7fcfc85e2700 1 -- 172.21.2.102:0/118734373 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcfc00d1870 msgr2=0x7fcfc01b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:49.248 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:49.230 7fcfc85e2700 1 --2- 172.21.2.102:0/118734373 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcfc00d1870 0x7fcfc01b03f0 secure :-1 s=READY pgs=2729 cs=0 l=1 rev1=1 rx=0x7fcfbc006700 tx=0x7fcfbc009b80).stop 2021-02-10T11:35:49.248 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:49.231 7fcfc85e2700 1 -- 172.21.2.102:0/118734373 shutdown_connections 2021-02-10T11:35:49.248 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:49.231 7fcfc85e2700 1 --2- 172.21.2.102:0/118734373 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcfc00f0650 0x7fcfc00cfb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:49.248 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:49.231 7fcfc85e2700 1 --2- 172.21.2.102:0/118734373 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcfc00d1870 0x7fcfc01b03f0 unknown :-1 s=CLOSED pgs=2729 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:49.248 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:49.231 7fcfc85e2700 1 -- 172.21.2.102:0/118734373 >> 172.21.2.102:0/118734373 conn(0x7fcfc00ce450 msgr2=0x7fcfc00d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:49.248 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:49.231 7fcfc85e2700 1 -- 172.21.2.102:0/118734373 shutdown_connections 2021-02-10T11:35:49.248 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:49.231 7fcfc85e2700 1 -- 172.21.2.102:0/118734373 wait complete. 2021-02-10T11:35:49.250 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:49.246 7fcfc85e2700 1 Processor -- start 2021-02-10T11:35:49.250 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:49.246 7fcfc85e2700 1 -- start start 2021-02-10T11:35:49.250 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:49.246 7fcfc85e2700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcfc01a7a80 0x7fcfc01a7e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:49.250 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:49.246 7fcfc85e2700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fcfc00c0800 0x7fcfc00c0c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:49.250 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:49.246 7fcfc85e2700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcfc00c1220 0x7fcfc00c1620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:49.250 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:49.246 7fcfc85e2700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfc00c0300 con 0x7fcfc01a7a80 2021-02-10T11:35:49.250 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:49.246 7fcfc85e2700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfc009e9a0 con 0x7fcfc00c0800 2021-02-10T11:35:49.250 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:49.246 7fcfc85e2700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfc00f14e0 con 0x7fcfc00c1220 2021-02-10T11:35:49.250 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:49.246 7fcfc63fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcfc01a7a80 0x7fcfc01a7e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:49.250 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:49.246 7fcfc63fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcfc01a7a80 0x7fcfc01a7e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:37228/0 (socket says 172.21.2.102:37228) 2021-02-10T11:35:49.250 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:49.246 7fcfc63fe700 1 -- 172.21.2.102:0/3409808358 learned_addr learned my addr 172.21.2.102:0/3409808358 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:35:49.250 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:49.247 7fcfc63fe700 1 -- 172.21.2.102:0/3409808358 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcfc00c1220 msgr2=0x7fcfc00c1620 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:35:49.251 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:49.247 7fcfc63fe700 1 --2- 172.21.2.102:0/3409808358 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcfc00c1220 0x7fcfc00c1620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:49.251 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:49.247 7fcfc63fe700 1 -- 172.21.2.102:0/3409808358 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fcfc00c0800 msgr2=0x7fcfc00c0c00 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:35:49.251 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:49.247 7fcfc63fe700 1 --2- 172.21.2.102:0/3409808358 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fcfc00c0800 0x7fcfc00c0c00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:49.251 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:49.247 7fcfc63fe700 1 -- 172.21.2.102:0/3409808358 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfc00f1700 con 0x7fcfc01a7a80 2021-02-10T11:35:49.251 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:49.247 7fcfc63fe700 1 --2- 172.21.2.102:0/3409808358 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcfc01a7a80 0x7fcfc01a7e80 secure :-1 s=READY pgs=2730 cs=0 l=1 rev1=1 rx=0x7fcfbc00b5e0 tx=0x7fcfbc00b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:49.251 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:49.247 7fcfb2ffd700 1 -- 172.21.2.102:0/3409808358 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcfbc00bd70 con 0x7fcfc01a7a80 2021-02-10T11:35:49.251 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:49.247 7fcfb2ffd700 1 -- 172.21.2.102:0/3409808358 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfbc002030 con 0x7fcfc01a7a80 2021-02-10T11:35:49.251 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:49.247 7fcfc85e2700 1 -- 172.21.2.102:0/3409808358 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfc00f1700 con 0x7fcfc01a7a80 2021-02-10T11:35:49.251 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:49.247 7fcfc85e2700 1 -- 172.21.2.102:0/3409808358 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfc00f1c90 con 0x7fcfc01a7a80 2021-02-10T11:35:49.251 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:49.247 7fcfb2ffd700 1 -- 172.21.2.102:0/3409808358 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcfbc002550 con 0x7fcfc01a7a80 2021-02-10T11:35:49.251 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:49.248 7fcfb2ffd700 1 -- 172.21.2.102:0/3409808358 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fcfbc002210 con 0x7fcfc01a7a80 2021-02-10T11:35:49.251 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:49.248 7fcfb2ffd700 1 --2- 172.21.2.102:0/3409808358 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcfa8018730 0x7fcfa801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:49.251 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:49.248 7fcfb2ffd700 1 -- 172.21.2.102:0/3409808358 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(146..146 src has 1..146) v4 ==== 5659+0+0 (secure 0 0 0) 0x7fcfbc032650 con 0x7fcfc01a7a80 2021-02-10T11:35:49.252 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:49.248 7fcfc5bfd700 1 --2- 172.21.2.102:0/3409808358 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcfa8018730 0x7fcfa801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:49.252 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:49.248 7fcfc5bfd700 1 --2- 172.21.2.102:0/3409808358 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcfa8018730 0x7fcfa801ac10 crc :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:49.280 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:49.279 7fcfc85e2700 1 -- 172.21.2.102:0/3409808358 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfc00ee060 con 0x7fcfc01a7a80 2021-02-10T11:35:49.283 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:49.282 7fcfb2ffd700 1 -- 172.21.2.102:0/3409808358 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fcfbc0009b0 con 0x7fcfc01a7a80 2021-02-10T11:35:49.409 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:49.408 7fcfc85e2700 1 -- 172.21.2.102:0/3409808358 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fcfc00ce450 con 0x7fcfc01a7a80 2021-02-10T11:35:50.165 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:50.164 7fcfb2ffd700 1 -- 172.21.2.102:0/3409808358 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v127) v1 ==== 104+0+0 (secure 0 0 0) 0x7fcfbc002030 con 0x7fcfc01a7a80 2021-02-10T11:35:50.181 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:50.179 7fcfc85e2700 1 -- 172.21.2.102:0/3409808358 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcfa8018730 msgr2=0x7fcfa801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:50.181 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:50.179 7fcfc85e2700 1 --2- 172.21.2.102:0/3409808358 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcfa8018730 0x7fcfa801ac10 crc :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:50.181 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:50.179 7fcfc85e2700 1 -- 172.21.2.102:0/3409808358 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcfc01a7a80 msgr2=0x7fcfc01a7e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:50.181 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:50.179 7fcfc85e2700 1 --2- 172.21.2.102:0/3409808358 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcfc01a7a80 0x7fcfc01a7e80 secure :-1 s=READY pgs=2730 cs=0 l=1 rev1=1 rx=0x7fcfbc00b5e0 tx=0x7fcfbc00b980).stop 2021-02-10T11:35:50.181 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:50.180 7fcfc85e2700 1 -- 172.21.2.102:0/3409808358 shutdown_connections 2021-02-10T11:35:50.181 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:50.180 7fcfc85e2700 1 --2- 172.21.2.102:0/3409808358 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcfc00c1220 0x7fcfc00c1620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:50.181 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:50.180 7fcfc85e2700 1 --2- 172.21.2.102:0/3409808358 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcfa8018730 0x7fcfa801ac10 unknown :-1 s=CLOSED pgs=1200 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:50.181 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:50.180 7fcfc85e2700 1 --2- 172.21.2.102:0/3409808358 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fcfc00c0800 0x7fcfc00c0c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:50.181 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:50.180 7fcfc85e2700 1 --2- 172.21.2.102:0/3409808358 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcfc01a7a80 0x7fcfc01a7e80 unknown :-1 s=CLOSED pgs=2730 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:50.182 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:50.180 7fcfc85e2700 1 -- 172.21.2.102:0/3409808358 >> 172.21.2.102:0/3409808358 conn(0x7fcfc00ba800 msgr2=0x7fcfc009f7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:50.182 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:50.180 7fcfc85e2700 1 -- 172.21.2.102:0/3409808358 shutdown_connections 2021-02-10T11:35:50.182 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:50.180 7fcfc85e2700 1 -- 172.21.2.102:0/3409808358 wait complete. 2021-02-10T11:35:50.189 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool delete cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2021-02-10T11:35:50.622 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:50.604 7fbd2b10b700 1 Processor -- start 2021-02-10T11:35:50.622 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:50.604 7fbd2b10b700 1 -- start start 2021-02-10T11:35:50.622 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:50.605 7fbd2b10b700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbd240d1c80 0x7fbd240cef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:50.622 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:50.605 7fbd2b10b700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd240d0180 con 0x7fbd240cf590 2021-02-10T11:35:50.623 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:50.605 7fbd2b10b700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd240fe1d0 con 0x7fbd240d1880 2021-02-10T11:35:50.623 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:50.605 7fbd2b10b700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd241b0cc0 con 0x7fbd240d1c80 2021-02-10T11:35:50.623 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:50.605 7fbd29728700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fbd240cf590 0x7fbd240cf990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:57692/0 (socket says 172.21.2.102:57692) 2021-02-10T11:35:50.623 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:50.605 7fbd29728700 1 -- 172.21.2.102:0/2692014736 learned_addr learned my addr 172.21.2.102:0/2692014736 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:35:50.623 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:50.605 7fbd237fe700 1 -- 172.21.2.102:0/2692014736 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3733612247 0 0) 0x7fbd1c001500 con 0x7fbd240cf590 2021-02-10T11:35:50.623 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:50.605 7fbd237fe700 1 -- 172.21.2.102:0/2692014736 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd10002fe0 con 0x7fbd240cf590 2021-02-10T11:35:50.623 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:50.605 7fbd23fff700 1 --2- 172.21.2.102:0/2692014736 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbd240d1c80 0x7fbd240cef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:50.623 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:50.605 7fbd237fe700 1 -- 172.21.2.102:0/2692014736 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2694329191 0 0) 0x7fbd1c001500 con 0x7fbd240cf590 2021-02-10T11:35:50.623 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:50.605 7fbd237fe700 1 -- 172.21.2.102:0/2692014736 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbd240d1c80 msgr2=0x7fbd240cef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:50.623 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:50.605 7fbd237fe700 1 --2- 172.21.2.102:0/2692014736 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbd240d1c80 0x7fbd240cef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:50.623 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:50.605 7fbd237fe700 1 -- 172.21.2.102:0/2692014736 >> v1:172.21.2.113:6789/0 conn(0x7fbd240d1880 legacy=0x7fbd241b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:50.623 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:50.605 7fbd237fe700 1 -- 172.21.2.102:0/2692014736 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd24109a70 con 0x7fbd240cf590 2021-02-10T11:35:50.623 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:50.605 7fbd237fe700 1 -- 172.21.2.102:0/2692014736 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7fbd1c001a00 con 0x7fbd240cf590 2021-02-10T11:35:50.623 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:50.605 7fbd237fe700 1 -- 172.21.2.102:0/2692014736 >> v1:172.21.2.102:6789/0 conn(0x7fbd240cf590 legacy=0x7fbd240cf990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:50.624 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:50.605 7fbd237fe700 1 --2- 172.21.2.102:0/2692014736 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbd10003a60 0x7fbd10005eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:50.624 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:50.605 7fbd237fe700 1 --2- 172.21.2.102:0/2692014736 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fbd10006750 0x7fbd10008c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:50.624 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:50.605 7fbd237fe700 1 --2- 172.21.2.102:0/2692014736 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbd100094f0 0x7fbd1000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:50.624 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:50.605 7fbd237fe700 1 -- 172.21.2.102:0/2692014736 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd1000c2c0 con 0x7fbd100094f0 2021-02-10T11:35:50.624 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:50.605 7fbd237fe700 1 -- 172.21.2.102:0/2692014736 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd1000c4e0 con 0x7fbd10006750 2021-02-10T11:35:50.624 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:50.605 7fbd237fe700 1 -- 172.21.2.102:0/2692014736 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd10050710 con 0x7fbd10003a60 2021-02-10T11:35:50.624 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:50.605 7fbd29728700 1 --2- 172.21.2.102:0/2692014736 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbd100094f0 0x7fbd1000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:50.624 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:50.606 7fbd28f27700 1 --2- 172.21.2.102:0/2692014736 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbd10003a60 0x7fbd10005eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:50.624 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:50.606 7fbd23fff700 1 --2- 172.21.2.102:0/2692014736 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fbd10006750 0x7fbd10008c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:50.624 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:50.606 7fbd29728700 1 -- 172.21.2.102:0/2692014736 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbd10003a60 msgr2=0x7fbd10005eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:50.624 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:50.606 7fbd29728700 1 --2- 172.21.2.102:0/2692014736 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbd10003a60 0x7fbd10005eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:50.624 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:50.606 7fbd29728700 1 -- 172.21.2.102:0/2692014736 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fbd10006750 msgr2=0x7fbd10008c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:50.624 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:50.606 7fbd29728700 1 --2- 172.21.2.102:0/2692014736 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fbd10006750 0x7fbd10008c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:50.625 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:50.606 7fbd29728700 1 -- 172.21.2.102:0/2692014736 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd10050940 con 0x7fbd100094f0 2021-02-10T11:35:50.625 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:50.606 7fbd23fff700 1 --2- 172.21.2.102:0/2692014736 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fbd10006750 0x7fbd10008c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:35:50.625 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:50.606 7fbd29728700 1 --2- 172.21.2.102:0/2692014736 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbd100094f0 0x7fbd1000ba10 secure :-1 s=READY pgs=2732 cs=0 l=1 rev1=1 rx=0x7fbd1c007a70 tx=0x7fbd1c00a310).ready entity=mon.0 client_cookie=63b920990f3365c3 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:50.625 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:50.606 7fbd237fe700 1 -- 172.21.2.102:0/2692014736 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbd1c0070c0 con 0x7fbd100094f0 2021-02-10T11:35:50.625 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:50.606 7fbd237fe700 1 -- 172.21.2.102:0/2692014736 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd1c007340 con 0x7fbd100094f0 2021-02-10T11:35:50.625 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:50.606 7fbd237fe700 1 -- 172.21.2.102:0/2692014736 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbd1c004720 con 0x7fbd100094f0 2021-02-10T11:35:50.625 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:50.606 7fbd2b10b700 1 -- 172.21.2.102:0/2692014736 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbd100094f0 msgr2=0x7fbd1000ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:50.625 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:50.606 7fbd2b10b700 1 --2- 172.21.2.102:0/2692014736 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbd100094f0 0x7fbd1000ba10 secure :-1 s=READY pgs=2732 cs=0 l=1 rev1=1 rx=0x7fbd1c007a70 tx=0x7fbd1c00a310).stop 2021-02-10T11:35:50.625 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:50.606 7fbd29728700 1 -- 172.21.2.102:0/2692014736 reap_dead start 2021-02-10T11:35:50.625 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:50.606 7fbd2b10b700 1 -- 172.21.2.102:0/2692014736 shutdown_connections 2021-02-10T11:35:50.625 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:50.606 7fbd2b10b700 1 -- 172.21.2.102:0/2692014736 >> 172.21.2.102:0/2692014736 conn(0x7fbd240ce430 msgr2=0x7fbd240d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:50.625 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:50.607 7fbd2b10b700 1 -- 172.21.2.102:0/2692014736 shutdown_connections 2021-02-10T11:35:50.625 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:50.607 7fbd2b10b700 1 -- 172.21.2.102:0/2692014736 wait complete. 2021-02-10T11:35:50.625 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:50.621 7fbd2b10b700 1 Processor -- start 2021-02-10T11:35:50.626 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:50.621 7fbd2b10b700 1 -- start start 2021-02-10T11:35:50.627 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:50.622 7fbd2b10b700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fbd240913d0 0x7fbd241a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:50.627 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:50.622 7fbd2b10b700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbd241a3c30 0x7fbd240b1170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:50.627 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:50.622 7fbd2b10b700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbd240b1790 0x7fbd240b1b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:50.627 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:50.622 7fbd2b10b700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd24091850 con 0x7fbd240b1790 2021-02-10T11:35:50.627 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:50.622 7fbd2b10b700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd240b21b0 con 0x7fbd240913d0 2021-02-10T11:35:50.627 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:50.622 7fbd2b10b700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd240b23d0 con 0x7fbd241a3c30 2021-02-10T11:35:50.627 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:50.622 7fbd29728700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbd240b1790 0x7fbd240b1b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:50.627 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:50.622 7fbd29728700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbd240b1790 0x7fbd240b1b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:37248/0 (socket says 172.21.2.102:37248) 2021-02-10T11:35:50.627 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:50.622 7fbd29728700 1 -- 172.21.2.102:0/3952446022 learned_addr learned my addr 172.21.2.102:0/3952446022 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:35:50.628 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:50.622 7fbd29728700 1 -- 172.21.2.102:0/3952446022 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbd241a3c30 msgr2=0x7fbd240b1170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:50.628 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:50.622 7fbd29728700 1 --2- 172.21.2.102:0/3952446022 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbd241a3c30 0x7fbd240b1170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:50.628 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:50.622 7fbd29728700 1 -- 172.21.2.102:0/3952446022 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fbd240913d0 msgr2=0x7fbd241a3610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:50.628 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:50.622 7fbd29728700 1 --2- 172.21.2.102:0/3952446022 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fbd240913d0 0x7fbd241a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:50.628 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:50.622 7fbd29728700 1 -- 172.21.2.102:0/3952446022 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd240af2a0 con 0x7fbd240b1790 2021-02-10T11:35:50.628 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:50.622 7fbd29728700 1 --2- 172.21.2.102:0/3952446022 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbd240b1790 0x7fbd240b1b90 secure :-1 s=READY pgs=2733 cs=0 l=1 rev1=1 rx=0x7fbd1c00a8f0 tx=0x7fbd1c00ac90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:50.628 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:50.623 7fbd217fa700 1 -- 172.21.2.102:0/3952446022 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbd1c001910 con 0x7fbd240b1790 2021-02-10T11:35:50.628 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:50.623 7fbd217fa700 1 -- 172.21.2.102:0/3952446022 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd1c0070d0 con 0x7fbd240b1790 2021-02-10T11:35:50.628 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:50.623 7fbd217fa700 1 -- 172.21.2.102:0/3952446022 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbd1c004930 con 0x7fbd240b1790 2021-02-10T11:35:50.628 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:50.623 7fbd2b10b700 1 -- 172.21.2.102:0/3952446022 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd240af2a0 con 0x7fbd240b1790 2021-02-10T11:35:50.628 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:50.623 7fbd2b10b700 1 -- 172.21.2.102:0/3952446022 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd240af2a0 con 0x7fbd240b1790 2021-02-10T11:35:50.628 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:50.623 7fbd217fa700 1 -- 172.21.2.102:0/3952446022 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fbd1c0072b0 con 0x7fbd240b1790 2021-02-10T11:35:50.628 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:50.623 7fbd217fa700 1 --2- 172.21.2.102:0/3952446022 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fbd080186c0 0x7fbd0801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:50.628 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:50.623 7fbd217fa700 1 -- 172.21.2.102:0/3952446022 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(146..146 src has 1..146) v4 ==== 5659+0+0 (secure 0 0 0) 0x7fbd1c032930 con 0x7fbd240b1790 2021-02-10T11:35:50.629 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:50.623 7fbd28f27700 1 --2- 172.21.2.102:0/3952446022 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fbd080186c0 0x7fbd0801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:50.629 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:50.624 7fbd28f27700 1 --2- 172.21.2.102:0/3952446022 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fbd080186c0 0x7fbd0801abc0 crc :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:50.655 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:50.654 7fbd2b10b700 1 -- 172.21.2.102:0/3952446022 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd24082aa0 con 0x7fbd240b1790 2021-02-10T11:35:50.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:50.657 7fbd217fa700 1 -- 172.21.2.102:0/3952446022 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fbd1c001910 con 0x7fbd240b1790 2021-02-10T11:35:50.786 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:50.785 7fbd2b10b700 1 -- 172.21.2.102:0/3952446022 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"pool2": "cephfs_metadata", "prefix": "osd pool delete", "pool": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fbd24082aa0 con 0x7fbd240b1790 2021-02-10T11:35:51.172 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:51.171 7fbd217fa700 1 -- 172.21.2.102:0/3952446022 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"pool2": "cephfs_metadata", "prefix": "osd pool delete", "pool": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v147) v1 ==== 185+0+0 (secure 0 0 0) 0x7fbd1c004720 con 0x7fbd240b1790 2021-02-10T11:35:51.204 INFO:teuthology.orchestra.run.gibba002.stderr:pool 'cephfs_metadata' removed 2021-02-10T11:35:51.206 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:51.205 7fbd2b10b700 1 -- 172.21.2.102:0/3952446022 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fbd080186c0 msgr2=0x7fbd0801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:51.206 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:51.205 7fbd2b10b700 1 --2- 172.21.2.102:0/3952446022 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fbd080186c0 0x7fbd0801abc0 crc :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:51.206 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:51.205 7fbd2b10b700 1 -- 172.21.2.102:0/3952446022 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbd240b1790 msgr2=0x7fbd240b1b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:51.206 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:51.205 7fbd2b10b700 1 --2- 172.21.2.102:0/3952446022 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbd240b1790 0x7fbd240b1b90 secure :-1 s=READY pgs=2733 cs=0 l=1 rev1=1 rx=0x7fbd1c00a8f0 tx=0x7fbd1c00ac90).stop 2021-02-10T11:35:51.206 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:51.205 7fbd2b10b700 1 -- 172.21.2.102:0/3952446022 shutdown_connections 2021-02-10T11:35:51.206 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:51.205 7fbd2b10b700 1 --2- 172.21.2.102:0/3952446022 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbd240b1790 0x7fbd240b1b90 unknown :-1 s=CLOSED pgs=2733 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:51.207 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:51.205 7fbd2b10b700 1 --2- 172.21.2.102:0/3952446022 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbd241a3c30 0x7fbd240b1170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:51.207 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:51.205 7fbd2b10b700 1 --2- 172.21.2.102:0/3952446022 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fbd080186c0 0x7fbd0801abc0 unknown :-1 s=CLOSED pgs=1201 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:51.207 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:51.205 7fbd2b10b700 1 --2- 172.21.2.102:0/3952446022 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fbd240913d0 0x7fbd241a3610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:51.207 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:51.205 7fbd2b10b700 1 -- 172.21.2.102:0/3952446022 >> 172.21.2.102:0/3952446022 conn(0x7fbd24087360 msgr2=0x7fbd240b5980 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:51.207 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:51.205 7fbd2b10b700 1 -- 172.21.2.102:0/3952446022 shutdown_connections 2021-02-10T11:35:51.207 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:51.205 7fbd2b10b700 1 -- 172.21.2.102:0/3952446022 wait complete. 2021-02-10T11:35:51.214 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool delete cephfs_data cephfs_data --yes-i-really-really-mean-it 2021-02-10T11:35:51.647 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:51.630 7f4be3f6c700 1 Processor -- start 2021-02-10T11:35:51.648 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:51.630 7f4be3f6c700 1 -- start start 2021-02-10T11:35:51.648 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:51.630 7f4be3f6c700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4bdc0d1880 0x7f4bdc1b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:51.648 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:51.630 7f4be3f6c700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f4bdc1b0af0 0x7f4bdc108950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:51.648 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:51.630 7f4be3f6c700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f4bdc0d1c80 0x7f4bdc108f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:51.648 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:51.630 7f4be3f6c700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4bdc0e5f00 con 0x7f4bdc0d1c80 2021-02-10T11:35:51.648 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:51.630 7f4be3f6c700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4bdc109a70 con 0x7f4bdc1b0af0 2021-02-10T11:35:51.648 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:51.630 7f4be3f6c700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bdc088070 con 0x7f4bdc0d1880 2021-02-10T11:35:51.648 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:51.630 7f4be2589700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f4bdc0d1c80 0x7f4bdc108f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:51.648 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:51.631 7f4be2589700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f4bdc0d1c80 0x7f4bdc108f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:37252/0 (socket says 172.21.2.102:37252) 2021-02-10T11:35:51.648 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:51.631 7f4be2589700 1 -- 172.21.2.102:0/3906690173 learned_addr learned my addr 172.21.2.102:0/3906690173 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:35:51.648 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:51.631 7f4be2589700 1 -- 172.21.2.102:0/3906690173 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4bdc0d1880 msgr2=0x7f4bdc1b04d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:35:51.649 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:51.631 7f4be2589700 1 --2- 172.21.2.102:0/3906690173 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4bdc0d1880 0x7f4bdc1b04d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:51.649 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:51.631 7f4be2589700 1 -- 172.21.2.102:0/3906690173 >> v2:172.21.2.113:3300/0 conn(0x7f4bdc1b0af0 msgr2=0x7f4bdc108950 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:35:51.649 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:51.631 7f4be2589700 1 --2- 172.21.2.102:0/3906690173 >> v2:172.21.2.113:3300/0 conn(0x7f4bdc1b0af0 0x7f4bdc108950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:51.649 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:51.631 7f4be2589700 1 -- 172.21.2.102:0/3906690173 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bdc109590 con 0x7f4bdc0d1c80 2021-02-10T11:35:51.649 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:51.631 7f4be2589700 1 --2- 172.21.2.102:0/3906690173 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4bdc0d1c80 0x7f4bdc108f70 secure :-1 s=READY pgs=2734 cs=0 l=1 rev1=1 rx=0x7f4bd4006f00 tx=0x7f4bd4009f10).ready entity=mon.0 client_cookie=2182a1de089809d8 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:51.649 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:51.631 7f4be0d86700 1 -- 172.21.2.102:0/3906690173 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4bd400b3b0 con 0x7f4bdc0d1c80 2021-02-10T11:35:51.649 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:51.631 7f4be0d86700 1 -- 172.21.2.102:0/3906690173 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bd400b630 con 0x7f4bdc0d1c80 2021-02-10T11:35:51.649 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:51.631 7f4be0d86700 1 -- 172.21.2.102:0/3906690173 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4bd400bb50 con 0x7f4bdc0d1c80 2021-02-10T11:35:51.649 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:51.631 7f4be3f6c700 1 -- 172.21.2.102:0/3906690173 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4bdc0d1c80 msgr2=0x7f4bdc108f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:51.649 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:51.631 7f4be3f6c700 1 --2- 172.21.2.102:0/3906690173 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4bdc0d1c80 0x7f4bdc108f70 secure :-1 s=READY pgs=2734 cs=0 l=1 rev1=1 rx=0x7f4bd4006f00 tx=0x7f4bd4009f10).stop 2021-02-10T11:35:51.649 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:51.632 7f4be3f6c700 1 -- 172.21.2.102:0/3906690173 shutdown_connections 2021-02-10T11:35:51.649 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:51.632 7f4be3f6c700 1 --2- 172.21.2.102:0/3906690173 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4bdc0d1c80 0x7f4bdc108f70 unknown :-1 s=CLOSED pgs=2734 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:51.649 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:51.632 7f4be3f6c700 1 --2- 172.21.2.102:0/3906690173 >> v2:172.21.2.113:3300/0 conn(0x7f4bdc1b0af0 0x7f4bdc108950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:51.650 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:51.632 7f4be3f6c700 1 --2- 172.21.2.102:0/3906690173 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4bdc0d1880 0x7f4bdc1b04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:51.650 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:51.632 7f4be3f6c700 1 -- 172.21.2.102:0/3906690173 >> 172.21.2.102:0/3906690173 conn(0x7f4bdc0ce430 msgr2=0x7f4bdc0d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:51.650 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:51.632 7f4be3f6c700 1 -- 172.21.2.102:0/3906690173 shutdown_connections 2021-02-10T11:35:51.650 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:51.632 7f4be3f6c700 1 -- 172.21.2.102:0/3906690173 wait complete. 2021-02-10T11:35:51.651 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:51.647 7f4be3f6c700 1 Processor -- start 2021-02-10T11:35:51.651 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:51.647 7f4be3f6c700 1 -- start start 2021-02-10T11:35:51.652 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:51.647 7f4be3f6c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4bdc096710 0x7f4bdc096f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:51.652 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:51.647 7f4be3f6c700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4bdc19c360 0x7f4bdc0ad520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:51.652 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:51.647 7f4be3f6c700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4bdc0adb40 0x7f4bdc19cef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:51.652 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:51.647 7f4be3f6c700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bdc19c970 con 0x7f4bdc096710 2021-02-10T11:35:51.652 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:51.647 7f4be3f6c700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bdc096b10 con 0x7f4bdc19c360 2021-02-10T11:35:51.652 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:51.647 7f4be3f6c700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bdc082c60 con 0x7f4bdc0adb40 2021-02-10T11:35:51.652 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:51.648 7f4be1d88700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4bdc096710 0x7f4bdc096f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:51.652 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:51.648 7f4be1d88700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4bdc096710 0x7f4bdc096f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:37256/0 (socket says 172.21.2.102:37256) 2021-02-10T11:35:51.652 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:51.648 7f4be1d88700 1 -- 172.21.2.102:0/3905351983 learned_addr learned my addr 172.21.2.102:0/3905351983 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:35:51.652 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:51.648 7f4be1d88700 1 -- 172.21.2.102:0/3905351983 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4bdc0adb40 msgr2=0x7f4bdc19cef0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:35:51.652 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:51.648 7f4be1d88700 1 --2- 172.21.2.102:0/3905351983 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4bdc0adb40 0x7f4bdc19cef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:51.652 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:51.648 7f4be1d88700 1 -- 172.21.2.102:0/3905351983 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4bdc19c360 msgr2=0x7f4bdc0ad520 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:35:51.652 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:51.648 7f4be1d88700 1 --2- 172.21.2.102:0/3905351983 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4bdc19c360 0x7f4bdc0ad520 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:51.653 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:51.648 7f4be1d88700 1 -- 172.21.2.102:0/3905351983 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bdc097590 con 0x7f4bdc096710 2021-02-10T11:35:51.653 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:51.648 7f4be1d88700 1 --2- 172.21.2.102:0/3905351983 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4bdc096710 0x7f4bdc096f70 secure :-1 s=READY pgs=2735 cs=0 l=1 rev1=1 rx=0x7f4bd800b5c0 tx=0x7f4bd800b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:51.653 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:51.648 7f4bce7fc700 1 -- 172.21.2.102:0/3905351983 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4bd8006b80 con 0x7f4bdc096710 2021-02-10T11:35:51.653 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:51.648 7f4bce7fc700 1 -- 172.21.2.102:0/3905351983 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bd800bd50 con 0x7f4bdc096710 2021-02-10T11:35:51.653 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:51.648 7f4bce7fc700 1 -- 172.21.2.102:0/3905351983 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4bd80054d0 con 0x7f4bdc096710 2021-02-10T11:35:51.653 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:51.648 7f4be3f6c700 1 -- 172.21.2.102:0/3905351983 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4bdc097590 con 0x7f4bdc096710 2021-02-10T11:35:51.653 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:51.648 7f4be3f6c700 1 -- 172.21.2.102:0/3905351983 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4bdc097590 con 0x7f4bdc096710 2021-02-10T11:35:51.653 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:51.649 7f4bce7fc700 1 -- 172.21.2.102:0/3905351983 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4bd8006030 con 0x7f4bdc096710 2021-02-10T11:35:51.653 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:51.649 7f4bce7fc700 1 --2- 172.21.2.102:0/3905351983 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4bc40186c0 0x7f4bc401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:51.653 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:51.649 7f4bce7fc700 1 -- 172.21.2.102:0/3905351983 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 5221+0+0 (secure 0 0 0) 0x7f4bd8032460 con 0x7f4bdc096710 2021-02-10T11:35:51.653 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:51.649 7f4be1587700 1 --2- 172.21.2.102:0/3905351983 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4bc40186c0 0x7f4bc401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:51.653 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:51.649 7f4be1587700 1 --2- 172.21.2.102:0/3905351983 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4bc40186c0 0x7f4bc401abc0 crc :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:51.681 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:51.680 7f4be3f6c700 1 -- 172.21.2.102:0/3905351983 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4bdc109a70 con 0x7f4bdc096710 2021-02-10T11:35:51.684 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:51.683 7f4bce7fc700 1 -- 172.21.2.102:0/3905351983 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f4bd8006b80 con 0x7f4bdc096710 2021-02-10T11:35:51.811 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:51.810 7f4be3f6c700 1 -- 172.21.2.102:0/3905351983 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"pool2": "cephfs_data", "prefix": "osd pool delete", "pool": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f4bdc109a70 con 0x7f4bdc096710 2021-02-10T11:35:52.177 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:52.176 7f4bce7fc700 1 -- 172.21.2.102:0/3905351983 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"pool2": "cephfs_data", "prefix": "osd pool delete", "pool": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v148) v1 ==== 173+0+0 (secure 0 0 0) 0x7f4bd8006030 con 0x7f4bdc096710 2021-02-10T11:35:52.177 INFO:teuthology.orchestra.run.gibba002.stderr:pool 'cephfs_data' removed 2021-02-10T11:35:52.179 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:52.178 7f4be3f6c700 1 -- 172.21.2.102:0/3905351983 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4bc40186c0 msgr2=0x7f4bc401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:52.179 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:52.178 7f4be3f6c700 1 --2- 172.21.2.102:0/3905351983 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4bc40186c0 0x7f4bc401abc0 crc :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:52.179 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:52.178 7f4be3f6c700 1 -- 172.21.2.102:0/3905351983 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4bdc096710 msgr2=0x7f4bdc096f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:52.179 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:52.178 7f4be3f6c700 1 --2- 172.21.2.102:0/3905351983 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4bdc096710 0x7f4bdc096f70 secure :-1 s=READY pgs=2735 cs=0 l=1 rev1=1 rx=0x7f4bd800b5c0 tx=0x7f4bd800b960).stop 2021-02-10T11:35:52.179 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:52.178 7f4be3f6c700 1 -- 172.21.2.102:0/3905351983 shutdown_connections 2021-02-10T11:35:52.180 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:52.178 7f4be3f6c700 1 --2- 172.21.2.102:0/3905351983 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4bdc0adb40 0x7f4bdc19cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:52.180 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:52.178 7f4be3f6c700 1 --2- 172.21.2.102:0/3905351983 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4bc40186c0 0x7f4bc401abc0 unknown :-1 s=CLOSED pgs=1202 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:52.180 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:52.178 7f4be3f6c700 1 --2- 172.21.2.102:0/3905351983 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4bdc19c360 0x7f4bdc0ad520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:52.180 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:52.178 7f4be3f6c700 1 --2- 172.21.2.102:0/3905351983 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4bdc096710 0x7f4bdc096f70 unknown :-1 s=CLOSED pgs=2735 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:52.180 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:52.178 7f4be3f6c700 1 -- 172.21.2.102:0/3905351983 >> 172.21.2.102:0/3905351983 conn(0x7f4bdc19e990 msgr2=0x7f4bdc1918e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:52.180 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:52.178 7f4be3f6c700 1 -- 172.21.2.102:0/3905351983 shutdown_connections 2021-02-10T11:35:52.180 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:52.178 7f4be3f6c700 1 -- 172.21.2.102:0/3905351983 wait complete. 2021-02-10T11:35:52.188 INFO:tasks.ceph.mds.a:Restarting daemon 2021-02-10T11:35:52.189 INFO:tasks.ceph.mds.a:Stopping old one... 2021-02-10T11:35:52.189 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2021-02-10T11:35:52.189 INFO:teuthology.orchestra.run:waiting for 300 2021-02-10T11:35:52.189 INFO:tasks.ceph.mds.b:Restarting daemon 2021-02-10T11:35:52.189 INFO:tasks.ceph.mds.b:Stopping old one... 2021-02-10T11:35:52.189 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2021-02-10T11:35:52.190 INFO:teuthology.orchestra.run:waiting for 300 2021-02-10T11:35:52.195 INFO:tasks.ceph.mds.a:Stopped 2021-02-10T11:35:52.195 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2021-02-10T11:35:52.196 INFO:tasks.ceph.mds.b:Stopped 2021-02-10T11:35:52.196 DEBUG:teuthology.orchestra.run.gibba013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2021-02-10T11:35:52.237 INFO:tasks.ceph.mds.a:Started 2021-02-10T11:35:52.237 DEBUG:teuthology.parallel:result is None 2021-02-10T11:35:52.238 INFO:tasks.ceph.mds.b:Started 2021-02-10T11:35:52.238 DEBUG:teuthology.parallel:result is None 2021-02-10T11:35:52.239 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd blacklist clear 2021-02-10T11:35:52.642 INFO:tasks.ceph.mds.b.gibba013.stdout:starting mds.b at 2021-02-10T11:35:52.647 INFO:tasks.ceph.mds.a.gibba002.stdout:starting mds.a at 2021-02-10T11:35:52.667 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:52.649 7f9730063700 1 Processor -- start 2021-02-10T11:35:52.668 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:52.649 7f9730063700 1 -- start start 2021-02-10T11:35:52.668 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:52.650 7f9730063700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f97280d1870 0x7f97281b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:52.668 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:52.650 7f9730063700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f97281b0a10 0x7f97280cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:52.668 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:52.650 7f9730063700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97280f0b10 con 0x7f97280d1c70 2021-02-10T11:35:52.668 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:52.650 7f9730063700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97280e5ed0 con 0x7f97281b0a10 2021-02-10T11:35:52.668 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:52.650 7f9730063700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97280d2fe0 con 0x7f97280d1870 2021-02-10T11:35:52.668 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:52.650 7f972d67e700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f97281b0a10 0x7f97280cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:52.668 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:52.650 7f972d67e700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f97281b0a10 0x7f97280cf330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:37278/0 (socket says 172.21.2.102:37278) 2021-02-10T11:35:52.668 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:52.650 7f972d67e700 1 -- 172.21.2.102:0/2702102985 learned_addr learned my addr 172.21.2.102:0/2702102985 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:35:52.668 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:52.650 7f972e680700 1 --1- 172.21.2.102:0/2702102985 >> v1:172.21.2.102:6789/0 conn(0x7f97280d1c70 0x7f97280cf950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:57736/0 (socket says 172.21.2.102:57736) 2021-02-10T11:35:52.668 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:52.650 7f972de7f700 1 --2- 172.21.2.102:0/2702102985 >> v2:172.21.2.113:3300/0 conn(0x7f97280d1870 0x7f97281b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:52.668 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:52.650 7f972d67e700 1 -- 172.21.2.102:0/2702102985 >> v2:172.21.2.113:3300/0 conn(0x7f97280d1870 msgr2=0x7f97281b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:52.668 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:52.650 7f972d67e700 1 --2- 172.21.2.102:0/2702102985 >> v2:172.21.2.113:3300/0 conn(0x7f97280d1870 0x7f97281b03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:52.669 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:52.650 7f972d67e700 1 -- 172.21.2.102:0/2702102985 >> v1:172.21.2.102:6789/0 conn(0x7f97280d1c70 legacy=0x7f97280cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:52.669 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:52.650 7f972d67e700 1 -- 172.21.2.102:0/2702102985 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97280f0650 con 0x7f97281b0a10 2021-02-10T11:35:52.669 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:52.651 7f972d67e700 1 --2- 172.21.2.102:0/2702102985 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f97281b0a10 0x7f97280cf330 secure :-1 s=READY pgs=2738 cs=0 l=1 rev1=1 rx=0x7f9724006a80 tx=0x7f9724009f00).ready entity=mon.0 client_cookie=e4a7b990ec7451f2 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:52.669 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:52.651 7f972ce7d700 1 -- 172.21.2.102:0/2702102985 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f972400b5c0 con 0x7f97281b0a10 2021-02-10T11:35:52.669 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:52.651 7f972ce7d700 1 -- 172.21.2.102:0/2702102985 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f972400b8c0 con 0x7f97281b0a10 2021-02-10T11:35:52.669 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:52.651 7f972ce7d700 1 -- 172.21.2.102:0/2702102985 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f972400bda0 con 0x7f97281b0a10 2021-02-10T11:35:52.669 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:52.651 7f9730063700 1 -- 172.21.2.102:0/2702102985 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f97281b0a10 msgr2=0x7f97280cf330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:52.669 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:52.651 7f9730063700 1 --2- 172.21.2.102:0/2702102985 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f97281b0a10 0x7f97280cf330 secure :-1 s=READY pgs=2738 cs=0 l=1 rev1=1 rx=0x7f9724006a80 tx=0x7f9724009f00).stop 2021-02-10T11:35:52.669 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:52.652 7f9730063700 1 -- 172.21.2.102:0/2702102985 shutdown_connections 2021-02-10T11:35:52.669 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:52.652 7f9730063700 1 --2- 172.21.2.102:0/2702102985 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f97281b0a10 0x7f97280cf330 unknown :-1 s=CLOSED pgs=2738 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:52.669 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:52.652 7f9730063700 1 --2- 172.21.2.102:0/2702102985 >> v2:172.21.2.113:3300/0 conn(0x7f97280d1870 0x7f97281b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:52.669 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:52.652 7f9730063700 1 -- 172.21.2.102:0/2702102985 >> 172.21.2.102:0/2702102985 conn(0x7f97280ce450 msgr2=0x7f97280d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:52.670 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:52.652 7f9730063700 1 -- 172.21.2.102:0/2702102985 shutdown_connections 2021-02-10T11:35:52.670 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:52.652 7f9730063700 1 -- 172.21.2.102:0/2702102985 wait complete. 2021-02-10T11:35:52.671 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:52.667 7f9730063700 1 Processor -- start 2021-02-10T11:35:52.671 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:52.667 7f9730063700 1 -- start start 2021-02-10T11:35:52.671 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:52.667 7f9730063700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f972819ffa0 0x7f97280eda10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:52.671 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:52.667 7f9730063700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f97281a03a0 0x7f97280ee030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:52.671 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:52.667 7f9730063700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f97280ee650 0x7f97280ebe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:52.671 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:52.667 7f9730063700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97280ed490 con 0x7f97281a03a0 2021-02-10T11:35:52.672 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:52.667 7f9730063700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97280eea50 con 0x7f97280ee650 2021-02-10T11:35:52.672 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:52.667 7f9730063700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97280eec30 con 0x7f972819ffa0 2021-02-10T11:35:52.672 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:52.668 7f972de7f700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f972819ffa0 0x7f97280eda10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:52.672 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:52.668 7f972d67e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f97281a03a0 0x7f97280ee030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:52.672 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:52.668 7f972de7f700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f972819ffa0 0x7f97280eda10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.113:3301/0 says I am v2:172.21.2.102:42818/0 (socket says 172.21.2.102:42818) 2021-02-10T11:35:52.672 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:52.668 7f972de7f700 1 -- 172.21.2.102:0/110496849 learned_addr learned my addr 172.21.2.102:0/110496849 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:35:52.672 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:52.668 7f972e680700 1 --2- 172.21.2.102:0/110496849 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f97280ee650 0x7f97280ebe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:52.672 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:52.668 7f972d67e700 1 -- 172.21.2.102:0/110496849 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f972819ffa0 msgr2=0x7f97280eda10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:52.672 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:52.668 7f972d67e700 1 --2- 172.21.2.102:0/110496849 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f972819ffa0 0x7f97280eda10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:52.672 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:52.668 7f972d67e700 1 -- 172.21.2.102:0/110496849 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f97280ee650 msgr2=0x7f97280ebe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:52.672 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:52.668 7f972d67e700 1 --2- 172.21.2.102:0/110496849 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f97280ee650 0x7f97280ebe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:52.672 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:52.668 7f972d67e700 1 -- 172.21.2.102:0/110496849 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97280ec4a0 con 0x7f97281a03a0 2021-02-10T11:35:52.672 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:52.668 7f972d67e700 1 --2- 172.21.2.102:0/110496849 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f97281a03a0 0x7f97280ee030 secure :-1 s=READY pgs=2739 cs=0 l=1 rev1=1 rx=0x7f9724006ed0 tx=0x7f972400bc40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:52.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:52.668 7f971e7fc700 1 -- 172.21.2.102:0/110496849 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f972400c2f0 con 0x7f97281a03a0 2021-02-10T11:35:52.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:52.669 7f971e7fc700 1 -- 172.21.2.102:0/110496849 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f972400c630 con 0x7f97281a03a0 2021-02-10T11:35:52.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:52.669 7f971e7fc700 1 -- 172.21.2.102:0/110496849 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f972400cb50 con 0x7f97281a03a0 2021-02-10T11:35:52.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:52.669 7f9730063700 1 -- 172.21.2.102:0/110496849 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97280ec4a0 con 0x7f97281a03a0 2021-02-10T11:35:52.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:52.669 7f9730063700 1 -- 172.21.2.102:0/110496849 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97280ec4a0 con 0x7f97281a03a0 2021-02-10T11:35:52.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:52.669 7f971e7fc700 1 -- 172.21.2.102:0/110496849 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f972400c2f0 con 0x7f97281a03a0 2021-02-10T11:35:52.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:52.669 7f971e7fc700 1 --2- 172.21.2.102:0/110496849 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f97080186c0 0x7f970801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:52.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:52.669 7f971e7fc700 1 -- 172.21.2.102:0/110496849 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 4839+0+0 (secure 0 0 0) 0x7f97240325c0 con 0x7f97281a03a0 2021-02-10T11:35:52.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:52.670 7f972de7f700 1 --2- 172.21.2.102:0/110496849 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f97080186c0 0x7f970801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:52.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:52.670 7f972de7f700 1 --2- 172.21.2.102:0/110496849 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f97080186c0 0x7f970801abc0 crc :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:52.699 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:52.698 7f9730063700 1 -- 172.21.2.102:0/110496849 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97280ebc50 con 0x7f97281a03a0 2021-02-10T11:35:52.702 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:52.701 7f971e7fc700 1 -- 172.21.2.102:0/110496849 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f9724026f40 con 0x7f97281a03a0 2021-02-10T11:35:52.840 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:52.839 7f9730063700 1 -- 172.21.2.102:0/110496849 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd blacklist clear"} v 0) v1 -- 0x7f97280ebc50 con 0x7f97281a03a0 2021-02-10T11:35:53.179 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:53.178 7f971e7fc700 1 -- 172.21.2.102:0/110496849 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd blacklist clear"}]=0 removed all blacklist entries v149) v1 ==== 97+0+0 (secure 0 0 0) 0x7f9724026f40 con 0x7f97281a03a0 2021-02-10T11:35:53.205 INFO:teuthology.orchestra.run.gibba002.stderr: removed all blacklist entries 2021-02-10T11:35:53.206 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:53.206 7f9730063700 1 -- 172.21.2.102:0/110496849 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f97080186c0 msgr2=0x7f970801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:53.207 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:53.206 7f9730063700 1 --2- 172.21.2.102:0/110496849 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f97080186c0 0x7f970801abc0 crc :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:53.207 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:53.206 7f9730063700 1 -- 172.21.2.102:0/110496849 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f97281a03a0 msgr2=0x7f97280ee030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:53.207 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:53.206 7f9730063700 1 --2- 172.21.2.102:0/110496849 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f97281a03a0 0x7f97280ee030 secure :-1 s=READY pgs=2739 cs=0 l=1 rev1=1 rx=0x7f9724006ed0 tx=0x7f972400bc40).stop 2021-02-10T11:35:53.207 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:53.206 7f9730063700 1 -- 172.21.2.102:0/110496849 shutdown_connections 2021-02-10T11:35:53.207 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:53.206 7f9730063700 1 --2- 172.21.2.102:0/110496849 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f97080186c0 0x7f970801abc0 unknown :-1 s=CLOSED pgs=1203 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:53.208 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:53.206 7f9730063700 1 --2- 172.21.2.102:0/110496849 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f97280ee650 0x7f97280ebe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:53.208 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:53.206 7f9730063700 1 --2- 172.21.2.102:0/110496849 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f97281a03a0 0x7f97280ee030 unknown :-1 s=CLOSED pgs=2739 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:53.208 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:53.206 7f9730063700 1 --2- 172.21.2.102:0/110496849 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f972819ffa0 0x7f97280eda10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:53.208 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:53.206 7f9730063700 1 -- 172.21.2.102:0/110496849 >> 172.21.2.102:0/110496849 conn(0x7f9728094d90 msgr2=0x7f972808a420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:53.208 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:53.206 7f9730063700 1 -- 172.21.2.102:0/110496849 shutdown_connections 2021-02-10T11:35:53.208 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:53.206 7f9730063700 1 -- 172.21.2.102:0/110496849 wait complete. 2021-02-10T11:35:53.215 INFO:tasks.cephfs.cephfs_test_case:['0', '1', '2', '3'] 2021-02-10T11:35:53.215 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth ls --format=json-pretty 2021-02-10T11:35:53.645 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:53.628 7ff730ed8700 1 Processor -- start 2021-02-10T11:35:53.645 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:53.628 7ff730ed8700 1 -- start start 2021-02-10T11:35:53.645 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:53.628 7ff730ed8700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7ff72c0d1c70 0x7ff72c0f20a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:53.645 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:53.628 7ff730ed8700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff72c0f0b10 con 0x7ff72c0d0340 2021-02-10T11:35:53.645 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:53.628 7ff730ed8700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff72c1b0be0 con 0x7ff72c0d1870 2021-02-10T11:35:53.645 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:53.628 7ff730ed8700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff72c0e5ed0 con 0x7ff72c0d1c70 2021-02-10T11:35:53.645 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:53.628 7ff722312700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7ff72c0d0340 0x7ff72c0ef640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:57750/0 (socket says 172.21.2.102:57750) 2021-02-10T11:35:53.645 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:53.628 7ff722312700 1 -- 172.21.2.102:0/3532123477 learned_addr learned my addr 172.21.2.102:0/3532123477 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:35:53.646 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:53.628 7ff72b314700 1 --2- 172.21.2.102:0/3532123477 >> v2:172.21.2.113:3300/0 conn(0x7ff72c0d1c70 0x7ff72c0f20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:53.646 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:53.629 7ff72a312700 1 -- 172.21.2.102:0/3532123477 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 290428348 0 0) 0x7ff71c0009b0 con 0x7ff72c0d0340 2021-02-10T11:35:53.646 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:53.629 7ff72a312700 1 -- 172.21.2.102:0/3532123477 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff70c002fe0 con 0x7ff72c0d0340 2021-02-10T11:35:53.646 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:53.629 7ff72a312700 1 -- 172.21.2.102:0/3532123477 <== mon.1 v1:172.21.2.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3398327962 0 0) 0x7ff714000c60 con 0x7ff72c0d1870 2021-02-10T11:35:53.646 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:53.629 7ff72b314700 1 -- 172.21.2.102:0/3532123477 >> v1:172.21.2.113:6789/0 conn(0x7ff72c0d1870 legacy=0x7ff72c1b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:53.646 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:53.629 7ff72b314700 1 -- 172.21.2.102:0/3532123477 >> v1:172.21.2.102:6789/0 conn(0x7ff72c0d0340 legacy=0x7ff72c0ef640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:53.646 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:53.629 7ff72b314700 1 -- 172.21.2.102:0/3532123477 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff72c0efe30 con 0x7ff72c0d1c70 2021-02-10T11:35:53.646 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:53.629 7ff72b314700 1 --2- 172.21.2.102:0/3532123477 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff72c0d1c70 0x7ff72c0f20a0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 rx=0x7ff724006e50 tx=0x7ff724009e70).ready entity=mon.1 client_cookie=24de20f5d313d094 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:53.646 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:53.629 7ff72a312700 1 -- 172.21.2.102:0/3532123477 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff724006a20 con 0x7ff72c0d1c70 2021-02-10T11:35:53.646 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:53.629 7ff72a312700 1 -- 172.21.2.102:0/3532123477 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff724006790 con 0x7ff72c0d1c70 2021-02-10T11:35:53.646 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:53.629 7ff72a312700 1 -- 172.21.2.102:0/3532123477 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff72400b700 con 0x7ff72c0d1c70 2021-02-10T11:35:53.646 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:53.629 7ff730ed8700 1 -- 172.21.2.102:0/3532123477 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff72c0d1c70 msgr2=0x7ff72c0f20a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:53.647 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:53.629 7ff730ed8700 1 --2- 172.21.2.102:0/3532123477 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff72c0d1c70 0x7ff72c0f20a0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 rx=0x7ff724006e50 tx=0x7ff724009e70).stop 2021-02-10T11:35:53.647 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:53.630 7ff730ed8700 1 -- 172.21.2.102:0/3532123477 shutdown_connections 2021-02-10T11:35:53.647 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:53.630 7ff730ed8700 1 --2- 172.21.2.102:0/3532123477 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff72c0d1c70 0x7ff72c0f20a0 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:53.647 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:53.630 7ff730ed8700 1 -- 172.21.2.102:0/3532123477 >> 172.21.2.102:0/3532123477 conn(0x7ff72c0ce450 msgr2=0x7ff72c0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:53.647 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:53.630 7ff730ed8700 1 -- 172.21.2.102:0/3532123477 shutdown_connections 2021-02-10T11:35:53.647 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:53.630 7ff730ed8700 1 -- 172.21.2.102:0/3532123477 wait complete. 2021-02-10T11:35:53.648 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:53.644 7ff730ed8700 1 Processor -- start 2021-02-10T11:35:53.649 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:53.644 7ff730ed8700 1 -- start start 2021-02-10T11:35:53.649 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:53.645 7ff730ed8700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff72c1b7cf0 0x7ff72c19cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:53.649 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:53.645 7ff730ed8700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff72c1b80f0 0x7ff72c1b84f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:53.649 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:53.645 7ff730ed8700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff72c19da20 0x7ff72c19de20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:53.649 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:53.645 7ff730ed8700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff72c0a77c0 con 0x7ff72c1b80f0 2021-02-10T11:35:53.649 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:53.645 7ff730ed8700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff72c19d4a0 con 0x7ff72c19da20 2021-02-10T11:35:53.649 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:53.645 7ff730ed8700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff72c0954f0 con 0x7ff72c1b7cf0 2021-02-10T11:35:53.649 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:53.645 7ff722312700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff72c1b80f0 0x7ff72c1b84f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:53.649 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:53.645 7ff722312700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff72c1b80f0 0x7ff72c1b84f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:37300/0 (socket says 172.21.2.102:37300) 2021-02-10T11:35:53.649 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:53.645 7ff722312700 1 -- 172.21.2.102:0/1153512687 learned_addr learned my addr 172.21.2.102:0/1153512687 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:35:53.649 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:53.645 7ff72ab13700 1 --2- 172.21.2.102:0/1153512687 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff72c1b7cf0 0x7ff72c19cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:53.649 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:53.645 7ff72b314700 1 --2- 172.21.2.102:0/1153512687 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff72c19da20 0x7ff72c19de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:53.650 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:53.645 7ff722312700 1 -- 172.21.2.102:0/1153512687 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff72c1b7cf0 msgr2=0x7ff72c19cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:53.650 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:53.645 7ff722312700 1 --2- 172.21.2.102:0/1153512687 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff72c1b7cf0 0x7ff72c19cdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:53.650 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:53.645 7ff722312700 1 -- 172.21.2.102:0/1153512687 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff72c19da20 msgr2=0x7ff72c19de20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:53.650 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:53.645 7ff722312700 1 --2- 172.21.2.102:0/1153512687 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff72c19da20 0x7ff72c19de20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:53.650 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:53.646 7ff722312700 1 -- 172.21.2.102:0/1153512687 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff72c1b8d60 con 0x7ff72c1b80f0 2021-02-10T11:35:53.650 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:53.646 7ff722312700 1 --2- 172.21.2.102:0/1153512687 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff72c1b80f0 0x7ff72c1b84f0 secure :-1 s=READY pgs=2741 cs=0 l=1 rev1=1 rx=0x7ff71c001c60 tx=0x7ff71c00a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:53.650 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:53.646 7ff723fff700 1 -- 172.21.2.102:0/1153512687 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff71c00acc0 con 0x7ff72c1b80f0 2021-02-10T11:35:53.650 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:53.646 7ff723fff700 1 -- 172.21.2.102:0/1153512687 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff71c004140 con 0x7ff72c1b80f0 2021-02-10T11:35:53.650 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:53.646 7ff723fff700 1 -- 172.21.2.102:0/1153512687 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff71c004620 con 0x7ff72c1b80f0 2021-02-10T11:35:53.650 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:53.646 7ff730ed8700 1 -- 172.21.2.102:0/1153512687 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff72c1b8d60 con 0x7ff72c1b80f0 2021-02-10T11:35:53.650 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:53.646 7ff730ed8700 1 -- 172.21.2.102:0/1153512687 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff72c19e730 con 0x7ff72c1b80f0 2021-02-10T11:35:53.650 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:53.646 7ff723fff700 1 -- 172.21.2.102:0/1153512687 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ff71c004320 con 0x7ff72c1b80f0 2021-02-10T11:35:53.651 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:53.647 7ff723fff700 1 --2- 172.21.2.102:0/1153512687 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff7040186c0 0x7ff70401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:53.651 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:53.647 7ff723fff700 1 -- 172.21.2.102:0/1153512687 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 4753+0+0 (secure 0 0 0) 0x7ff71c0324c0 con 0x7ff72c1b80f0 2021-02-10T11:35:53.651 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:53.647 7ff72ab13700 1 --2- 172.21.2.102:0/1153512687 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff7040186c0 0x7ff70401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:53.651 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:53.647 7ff72ab13700 1 --2- 172.21.2.102:0/1153512687 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff7040186c0 0x7ff70401abc0 crc :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:53.679 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:53.678 7ff730ed8700 1 -- 172.21.2.102:0/1153512687 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff72c0ebc00 con 0x7ff72c1b80f0 2021-02-10T11:35:53.682 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:53.681 7ff723fff700 1 -- 172.21.2.102:0/1153512687 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ff71c000940 con 0x7ff72c1b80f0 2021-02-10T11:35:53.815 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:53.814 7ff730ed8700 1 -- 172.21.2.102:0/1153512687 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7ff72c0f0b10 con 0x7ff72c1b80f0 2021-02-10T11:35:53.815 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:53.815 7ff723fff700 1 -- 172.21.2.102:0/1153512687 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v84) v1 ==== 80+0+4513 (secure 0 0 0) 0x7ff71c004140 con 0x7ff72c1b80f0 2021-02-10T11:35:53.816 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:35:53.816 INFO:teuthology.orchestra.run.gibba002.stdout:{ 2021-02-10T11:35:53.816 INFO:teuthology.orchestra.run.gibba002.stdout: "auth_dump": [ 2021-02-10T11:35:53.816 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:35:53.816 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "mds.a", 2021-02-10T11:35:53.816 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAevSNgsgyEARAAvr0cUjjCoEHLxaZSZ2HtVA==", 2021-02-10T11:35:53.817 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:35:53.817 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow", 2021-02-10T11:35:53.817 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:35:53.817 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:35:53.817 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:35:53.817 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:35:53.817 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:35:53.817 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:35:53.817 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "mds.b", 2021-02-10T11:35:53.817 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAevSNgwOFMMRAALfb1fbX56ao1TU8ClnTW1A==", 2021-02-10T11:35:53.817 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:35:53.817 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow", 2021-02-10T11:35:53.817 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:35:53.817 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:35:53.818 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:35:53.818 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:35:53.818 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:35:53.818 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:35:53.818 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.0", 2021-02-10T11:35:53.818 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAmvSNgj+bMMBAAD3HUD6fLiMAg+++iNzLJKQ==", 2021-02-10T11:35:53.818 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:35:53.818 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:35:53.818 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:35:53.818 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:35:53.818 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:35:53.818 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:35:53.819 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:35:53.819 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.1", 2021-02-10T11:35:53.819 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQApvSNgnvc+MRAAAOxqlNwiyLHqNITOrYeFKQ==", 2021-02-10T11:35:53.819 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:35:53.819 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:35:53.819 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:35:53.819 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:35:53.819 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:35:53.819 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:35:53.819 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:35:53.819 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.2", 2021-02-10T11:35:53.820 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAsvSNgZ4SbMRAA76IOYWfbiAuBoM/4k81GxA==", 2021-02-10T11:35:53.820 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:35:53.820 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:35:53.820 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:35:53.820 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:35:53.820 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:35:53.820 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:35:53.820 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:35:53.820 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.3", 2021-02-10T11:35:53.820 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAvvSNgFk3rMRAAWCmKKyw0LDQ+UlAHZAITOw==", 2021-02-10T11:35:53.820 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:35:53.820 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:35:53.821 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:35:53.821 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:35:53.821 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:35:53.821 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:35:53.821 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:35:53.821 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.4", 2021-02-10T11:35:53.821 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQA4vSNgpKcNJBAApJMmvj5YxVHPDOmvmQONOg==", 2021-02-10T11:35:53.821 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:35:53.821 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:35:53.821 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:35:53.821 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:35:53.821 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:35:53.822 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:35:53.822 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:35:53.822 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.5", 2021-02-10T11:35:53.822 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQA7vSNgf2ujJBAAfkuYpyb3WTzc8V5kw+Wktg==", 2021-02-10T11:35:53.822 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:35:53.822 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:35:53.822 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:35:53.822 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:35:53.823 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:35:53.823 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:35:53.823 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:35:53.823 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.6", 2021-02-10T11:35:53.823 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQA+vSNgqw9uJRAAFMU0R+UxU9C3o/9byb3e0g==", 2021-02-10T11:35:53.823 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:35:53.823 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:35:53.823 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:35:53.823 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:35:53.823 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:35:53.823 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:35:53.823 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:35:53.823 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.7", 2021-02-10T11:35:53.824 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQBBvSNgvWSrJRAA2KaCHmLpzovRgYzLvyofTQ==", 2021-02-10T11:35:53.824 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:35:53.824 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:35:53.824 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:35:53.824 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:35:53.824 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:35:53.824 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:35:53.824 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:35:53.824 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "client.0", 2021-02-10T11:35:53.824 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAfvSNgZd9XIBAAvyrvl8fdKuHnJZfczfRoFw==", 2021-02-10T11:35:53.824 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:35:53.824 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow", 2021-02-10T11:35:53.824 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow r", 2021-02-10T11:35:53.824 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:35:53.825 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:35:53.825 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:35:53.825 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:35:53.825 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "client.1", 2021-02-10T11:35:53.825 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAfvSNgSyfXNhAAIduPnF7Vd/6HSkJ0MlcMSA==", 2021-02-10T11:35:53.825 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:35:53.825 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow", 2021-02-10T11:35:53.825 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow r", 2021-02-10T11:35:53.825 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:35:53.825 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:35:53.825 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:35:53.825 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:35:53.825 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "client.2", 2021-02-10T11:35:53.825 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAgvSNgiIG/ERAA9ZTamaTI7ZsPSnxBqjplCg==", 2021-02-10T11:35:53.826 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:35:53.826 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow", 2021-02-10T11:35:53.826 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow r", 2021-02-10T11:35:53.826 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:35:53.826 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:35:53.826 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:35:53.826 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:35:53.826 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "client.3", 2021-02-10T11:35:53.826 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAgvSNgOWpOKBAAlXsEB04Q/GvtL06PqRMLeA==", 2021-02-10T11:35:53.826 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:35:53.826 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow", 2021-02-10T11:35:53.826 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow r", 2021-02-10T11:35:53.826 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:35:53.827 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:35:53.827 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:35:53.827 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:35:53.827 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "client.admin", 2021-02-10T11:35:53.827 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAavSNgO3GQBhAAL+NVnYskQRKVWLgyDPY/xw==", 2021-02-10T11:35:53.827 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:35:53.827 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow *", 2021-02-10T11:35:53.827 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:35:53.827 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:35:53.827 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:35:53.827 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:35:53.827 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:35:53.827 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:35:53.827 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "mgr.x", 2021-02-10T11:35:53.828 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAdvSNgXZZgJhAA7dtBSGzH+DsBwnddZpOBJA==", 2021-02-10T11:35:53.828 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:35:53.828 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow *", 2021-02-10T11:35:53.828 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow profile mgr", 2021-02-10T11:35:53.828 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:35:53.828 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:35:53.828 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:35:53.828 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:35:53.828 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "mgr.y", 2021-02-10T11:35:53.828 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAdvSNgVqQFDRAAwo6YEdWZCAr5GdOWVL/Qvg==", 2021-02-10T11:35:53.828 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:35:53.828 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow *", 2021-02-10T11:35:53.828 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow profile mgr", 2021-02-10T11:35:53.828 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:35:53.829 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:35:53.829 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:35:53.829 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:35:53.829 INFO:teuthology.orchestra.run.gibba002.stdout:} 2021-02-10T11:35:53.830 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:53.816 7ff730ed8700 1 -- 172.21.2.102:0/1153512687 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff7040186c0 msgr2=0x7ff70401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:53.830 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:53.816 7ff730ed8700 1 --2- 172.21.2.102:0/1153512687 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff7040186c0 0x7ff70401abc0 crc :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:53.830 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:53.816 7ff730ed8700 1 -- 172.21.2.102:0/1153512687 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff72c1b80f0 msgr2=0x7ff72c1b84f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:53.830 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:53.816 7ff730ed8700 1 --2- 172.21.2.102:0/1153512687 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff72c1b80f0 0x7ff72c1b84f0 secure :-1 s=READY pgs=2741 cs=0 l=1 rev1=1 rx=0x7ff71c001c60 tx=0x7ff71c00a8d0).stop 2021-02-10T11:35:53.830 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:53.817 7ff730ed8700 1 -- 172.21.2.102:0/1153512687 shutdown_connections 2021-02-10T11:35:53.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:53.817 7ff730ed8700 1 --2- 172.21.2.102:0/1153512687 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff7040186c0 0x7ff70401abc0 unknown :-1 s=CLOSED pgs=1204 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:53.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:53.817 7ff730ed8700 1 --2- 172.21.2.102:0/1153512687 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff72c19da20 0x7ff72c19de20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:53.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:53.817 7ff730ed8700 1 --2- 172.21.2.102:0/1153512687 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff72c1b80f0 0x7ff72c1b84f0 unknown :-1 s=CLOSED pgs=2741 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:53.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:53.817 7ff730ed8700 1 --2- 172.21.2.102:0/1153512687 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff72c1b7cf0 0x7ff72c19cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:53.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:53.817 7ff730ed8700 1 -- 172.21.2.102:0/1153512687 >> 172.21.2.102:0/1153512687 conn(0x7ff72c0ed810 msgr2=0x7ff72c094980 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:53.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:53.817 7ff730ed8700 1 -- 172.21.2.102:0/1153512687 shutdown_connections 2021-02-10T11:35:53.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:53.817 7ff730ed8700 1 -- 172.21.2.102:0/1153512687 wait complete. 2021-02-10T11:35:53.832 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:35:54.261 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:54.243 7f9792725700 1 Processor -- start 2021-02-10T11:35:54.261 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:54.243 7f9792725700 1 -- start start 2021-02-10T11:35:54.262 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:54.243 7f9792725700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f978c0d1870 0x7f978c1b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:54.262 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:54.243 7f9792725700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f978c0f0b10 con 0x7f978c0f0650 2021-02-10T11:35:54.262 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:54.243 7f9792725700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f978c0d0310 con 0x7f978c1b0a10 2021-02-10T11:35:54.262 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:54.243 7f9792725700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f978c0e5ed0 con 0x7f978c0d1870 2021-02-10T11:35:54.262 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:54.244 7f9790d42700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f978c0f0650 0x7f978c0cfb20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:57760/0 (socket says 172.21.2.102:57760) 2021-02-10T11:35:54.262 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:54.244 7f9790d42700 1 -- 172.21.2.102:0/1403092922 learned_addr learned my addr 172.21.2.102:0/1403092922 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:35:54.262 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:54.244 7f978bfff700 1 --2- 172.21.2.102:0/1403092922 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f978c0d1870 0x7f978c1b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:54.262 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:54.244 7f978affd700 1 -- 172.21.2.102:0/1403092922 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1160711085 0 0) 0x7f9784001500 con 0x7f978c0f0650 2021-02-10T11:35:54.262 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:54.244 7f978affd700 1 -- 172.21.2.102:0/1403092922 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9774002fe0 con 0x7f978c0f0650 2021-02-10T11:35:54.262 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:54.244 7f978affd700 1 -- 172.21.2.102:0/1403092922 <== mon.1 v1:172.21.2.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 247068634 0 0) 0x7f9780000ff0 con 0x7f978c1b0a10 2021-02-10T11:35:54.262 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:54.244 7f978affd700 1 -- 172.21.2.102:0/1403092922 --> v1:172.21.2.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f97740042c0 con 0x7f978c1b0a10 2021-02-10T11:35:54.263 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:54.244 7f978bfff700 1 -- 172.21.2.102:0/1403092922 >> v1:172.21.2.113:6789/0 conn(0x7f978c1b0a10 legacy=0x7f978c0cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:54.263 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:54.244 7f978bfff700 1 -- 172.21.2.102:0/1403092922 >> v1:172.21.2.102:6789/0 conn(0x7f978c0f0650 legacy=0x7f978c0cfb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:54.263 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:54.244 7f978bfff700 1 -- 172.21.2.102:0/1403092922 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f978c0d0530 con 0x7f978c0d1870 2021-02-10T11:35:54.263 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:54.244 7f978bfff700 1 --2- 172.21.2.102:0/1403092922 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f978c0d1870 0x7f978c1b03f0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 rx=0x7f977c006650 tx=0x7f977c009a80).ready entity=mon.2 client_cookie=c336070013365eba server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:54.263 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:54.245 7f978affd700 1 -- 172.21.2.102:0/1403092922 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f977c00c0c0 con 0x7f978c0d1870 2021-02-10T11:35:54.263 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:54.245 7f978affd700 1 -- 172.21.2.102:0/1403092922 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f977c00c440 con 0x7f978c0d1870 2021-02-10T11:35:54.263 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:54.245 7f978affd700 1 -- 172.21.2.102:0/1403092922 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f977c00c960 con 0x7f978c0d1870 2021-02-10T11:35:54.263 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:54.245 7f9792725700 1 -- 172.21.2.102:0/1403092922 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f978c0d1870 msgr2=0x7f978c1b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:54.263 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:54.245 7f9792725700 1 --2- 172.21.2.102:0/1403092922 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f978c0d1870 0x7f978c1b03f0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 rx=0x7f977c006650 tx=0x7f977c009a80).stop 2021-02-10T11:35:54.263 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:54.246 7f9792725700 1 -- 172.21.2.102:0/1403092922 shutdown_connections 2021-02-10T11:35:54.263 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:54.246 7f9792725700 1 --2- 172.21.2.102:0/1403092922 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f978c0d1870 0x7f978c1b03f0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:54.263 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:54.246 7f9792725700 1 -- 172.21.2.102:0/1403092922 >> 172.21.2.102:0/1403092922 conn(0x7f978c0ce450 msgr2=0x7f978c0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:54.263 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:54.246 7f9792725700 1 -- 172.21.2.102:0/1403092922 shutdown_connections 2021-02-10T11:35:54.263 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:54.246 7f9792725700 1 -- 172.21.2.102:0/1403092922 wait complete. 2021-02-10T11:35:54.265 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:54.261 7f9792725700 1 Processor -- start 2021-02-10T11:35:54.265 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:54.261 7f9792725700 1 -- start start 2021-02-10T11:35:54.265 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:54.262 7f9792725700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f978c093f10 0x7f978c094310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:54.265 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:54.262 7f9792725700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f978c0ae9e0 0x7f978c0aede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:54.265 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:54.262 7f9792725700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f978c092d90 0x7f978c093190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:54.265 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:54.262 7f9792725700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f978c19ebc0 con 0x7f978c093f10 2021-02-10T11:35:54.265 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:54.262 7f9792725700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f978c1b8760 con 0x7f978c092d90 2021-02-10T11:35:54.266 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:54.262 7f9792725700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f978c0937b0 con 0x7f978c0ae9e0 2021-02-10T11:35:54.266 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:54.262 7f978bfff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f978c093f10 0x7f978c094310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:54.266 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:54.262 7f978b7fe700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f978c0ae9e0 0x7f978c0aede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:54.266 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:54.262 7f978bfff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f978c093f10 0x7f978c094310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:37312/0 (socket says 172.21.2.102:37312) 2021-02-10T11:35:54.266 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:54.262 7f978bfff700 1 -- 172.21.2.102:0/2283421077 learned_addr learned my addr 172.21.2.102:0/2283421077 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:35:54.266 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:54.262 7f9790d42700 1 --2- 172.21.2.102:0/2283421077 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f978c092d90 0x7f978c093190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:54.266 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:54.262 7f978bfff700 1 -- 172.21.2.102:0/2283421077 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f978c0ae9e0 msgr2=0x7f978c0aede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:54.266 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:54.262 7f978bfff700 1 --2- 172.21.2.102:0/2283421077 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f978c0ae9e0 0x7f978c0aede0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:54.266 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:54.262 7f978bfff700 1 -- 172.21.2.102:0/2283421077 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f978c092d90 msgr2=0x7f978c093190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:54.266 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:54.262 7f978bfff700 1 --2- 172.21.2.102:0/2283421077 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f978c092d90 0x7f978c093190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:54.266 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:54.262 7f978bfff700 1 -- 172.21.2.102:0/2283421077 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f978c0939d0 con 0x7f978c093f10 2021-02-10T11:35:54.266 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:54.262 7f978bfff700 1 --2- 172.21.2.102:0/2283421077 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f978c093f10 0x7f978c094310 secure :-1 s=READY pgs=2743 cs=0 l=1 rev1=1 rx=0x7f977c009630 tx=0x7f977c00b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:54.266 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:54.263 7f9788ff9700 1 -- 172.21.2.102:0/2283421077 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f977c00bd60 con 0x7f978c093f10 2021-02-10T11:35:54.267 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:54.263 7f9788ff9700 1 -- 172.21.2.102:0/2283421077 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f977c006790 con 0x7f978c093f10 2021-02-10T11:35:54.267 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:54.263 7f9792725700 1 -- 172.21.2.102:0/2283421077 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f978c0939d0 con 0x7f978c093f10 2021-02-10T11:35:54.267 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:54.263 7f9792725700 1 -- 172.21.2.102:0/2283421077 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f978c189cd0 con 0x7f978c093f10 2021-02-10T11:35:54.267 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:54.263 7f9788ff9700 1 -- 172.21.2.102:0/2283421077 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f977c0052e0 con 0x7f978c093f10 2021-02-10T11:35:54.267 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:54.263 7f9788ff9700 1 -- 172.21.2.102:0/2283421077 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f977c006030 con 0x7f978c093f10 2021-02-10T11:35:54.267 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:54.263 7f9788ff9700 1 --2- 172.21.2.102:0/2283421077 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f976c0186c0 0x7f976c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:54.267 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:54.263 7f9788ff9700 1 -- 172.21.2.102:0/2283421077 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 4753+0+0 (secure 0 0 0) 0x7f977c032470 con 0x7f978c093f10 2021-02-10T11:35:54.267 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:54.264 7f978b7fe700 1 --2- 172.21.2.102:0/2283421077 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f976c0186c0 0x7f976c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:54.267 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:54.264 7f978b7fe700 1 --2- 172.21.2.102:0/2283421077 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f976c0186c0 0x7f976c01abc0 crc :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:54.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:54.293 7f9792725700 1 -- 172.21.2.102:0/2283421077 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9764002fd0 con 0x7f978c093f10 2021-02-10T11:35:54.296 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:54.295 7f9788ff9700 1 -- 172.21.2.102:0/2283421077 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f977c0093a0 con 0x7f978c093f10 2021-02-10T11:35:54.421 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:54.420 7f9792725700 1 -- 172.21.2.102:0/2283421077 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9764002de0 con 0x7f978c093f10 2021-02-10T11:35:54.421 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:54.421 7f9788ff9700 1 -- 172.21.2.102:0/2283421077 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v149) v1 ==== 74+0+11154 (secure 0 0 0) 0x7f977c006740 con 0x7f978c093f10 2021-02-10T11:35:54.422 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:35:54.422 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":149,"fsid":"68db30ea-dbce-4aef-a6df-f01c6fea0513","created":"2021-02-10 11:03:06.331557","modified":"2021-02-10 11:35:53.174620","last_up_change":"2021-02-10 11:03:32.690562","last_in_change":"2021-02-10 11:03:32.690562","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":31,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:03:37.322519","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"be691b4d-a039-408a-af80-9d53f02b6f1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6818","nonce":36088},{"type":"v1","addr":"172.21.2.102:6820","nonce":36088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6822","nonce":36088},{"type":"v1","addr":"172.21.2.102:6824","nonce":36088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6830","nonce":36088},{"type":"v1","addr":"172.21.2.102:6832","nonce":36088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6826","nonce":36088},{"type":"v1","addr":"172.21.2.102:6828","nonce":36088}]},"public_addr":"172.21.2.102:6820/36088","cluster_addr":"172.21.2.102:6824/36088","heartbeat_back_addr":"172.21.2.102:6832/36088","heartbeat_front_addr":"172.21.2.102:6828/36088","state":["exists","up"]},{"osd":1,"uuid":"64e3ccc8-9c69-4d7c-865e-af8e83822635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6819","nonce":36087},{"type":"v1","addr":"172.21.2.102:6821","nonce":36087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6823","nonce":36087},{"type":"v1","addr":"172.21.2.102:6825","nonce":36087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6831","nonce":36087},{"type":"v1","addr":"172.21.2.102:6833","nonce":36087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6827","nonce":36087},{"type":"v1","addr":"172.21.2.102:6829","nonce":36087}]},"public_addr":"172.21.2.102:6821/36087","cluster_addr":"172.21.2.102:6825/36087","heartbeat_back_addr":"172.21.2.102:6833/36087","heartbeat_front_addr":"172.21.2.102:6829/36087","state":["exists","up"]},{"osd":2,"uuid":"153700af-747f-481f-bdf4-d6072d3a842f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6810","nonce":36089},{"type":"v1","addr":"172.21.2.102:6811","nonce":36089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6812","nonce":36089},{"type":"v1","addr":"172.21.2.102:6813","nonce":36089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6816","nonce":36089},{"type":"v1","addr":"172.21.2.102:6817","nonce":36089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6814","nonce":36089},{"type":"v1","addr":"172.21.2.102:6815","nonce":36089}]},"public_addr":"172.21.2.102:6811/36089","cluster_addr":"172.21.2.102:6813/36089","heartbeat_back_addr":"172.21.2.102:6817/36089","heartbeat_front_addr":"172.21.2.102:6815/36089","state":["exists","up"]},{"osd":3,"uuid":"cbc678f4-83a9-45e6-8830-2f7c232b1ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6802","nonce":36090},{"type":"v1","addr":"172.21.2.102:6803","nonce":36090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6804","nonce":36090},{"type":"v1","addr":"172.21.2.102:6805","nonce":36090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6808","nonce":36090},{"type":"v1","addr":"172.21.2.102:6809","nonce":36090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6806","nonce":36090},{"type":"v1","addr":"172.21.2.102:6807","nonce":36090}]},"public_addr":"172.21.2.102:6803/36090","cluster_addr":"172.21.2.102:6805/36090","heartbeat_back_addr":"172.21.2.102:6809/36090","heartbeat_front_addr":"172.21.2.102:6807/36090","state":["exists","up"]},{"osd":4,"uuid":"c01e4008-b258-4f10-afd2-9e7f37214351","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6808","nonce":36368},{"type":"v1","addr":"172.21.2.113:6811","nonce":36368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6814","nonce":36368},{"type":"v1","addr":"172.21.2.113:6818","nonce":36368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6826","nonce":36368},{"type":"v1","addr":"172.21.2.113:6828","nonce":36368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6820","nonce":36368},{"type":"v1","addr":"172.21.2.113:6824","nonce":36368}]},"public_addr":"172.21.2.113:6811/36368","cluster_addr":"172.21.2.113:6818/36368","heartbeat_back_addr":"172.21.2.113:6828/36368","heartbeat_front_addr":"172.21.2.113:6824/36368","state":["exists","up"]},{"osd":5,"uuid":"551f6007-daf8-4a98-8ee0-c2b16bcccb5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6802","nonce":36366},{"type":"v1","addr":"172.21.2.113:6805","nonce":36366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6807","nonce":36366},{"type":"v1","addr":"172.21.2.113:6810","nonce":36366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6819","nonce":36366},{"type":"v1","addr":"172.21.2.113:6822","nonce":36366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6812","nonce":36366},{"type":"v1","addr":"172.21.2.113:6815","nonce":36366}]},"public_addr":"172.21.2.113:6805/36366","cluster_addr":"172.21.2.113:6810/36366","heartbeat_back_addr":"172.21.2.113:6822/36366","heartbeat_front_addr":"172.21.2.113:6815/36366","state":["exists","up"]},{"osd":6,"uuid":"bf67c040-0e50-48be-b102-c3943bc9ebef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6800","nonce":36367},{"type":"v1","addr":"172.21.2.113:6801","nonce":36367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6803","nonce":36367},{"type":"v1","addr":"172.21.2.113:6804","nonce":36367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6813","nonce":36367},{"type":"v1","addr":"172.21.2.113:6816","nonce":36367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6806","nonce":36367},{"type":"v1","addr":"172.21.2.113:6809","nonce":36367}]},"public_addr":"172.21.2.113:6801/36367","cluster_addr":"172.21.2.113:6804/36367","heartbeat_back_addr":"172.21.2.113:6816/36367","heartbeat_front_addr":"172.21.2.113:6809/36367","state":["exists","up"]},{"osd":7,"uuid":"f5461f80-018d-428f-be89-2eb2bcbe7bb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6817","nonce":36365},{"type":"v1","addr":"172.21.2.113:6821","nonce":36365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6823","nonce":36365},{"type":"v1","addr":"172.21.2.113:6825","nonce":36365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6830","nonce":36365},{"type":"v1","addr":"172.21.2.113:6831","nonce":36365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6827","nonce":36365},{"type":"v1","addr":"172.21.2.113:6829","nonce":36365}]},"public_addr":"172.21.2.113:6821/36365","cluster_addr":"172.21.2.113:6825/36365","heartbeat_back_addr":"172.21.2.113:6831/36365","heartbeat_front_addr":"172.21.2.113:6829/36365","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:35:54.423 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:54.422 7f9792725700 1 -- 172.21.2.102:0/2283421077 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f976c0186c0 msgr2=0x7f976c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:54.424 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:54.422 7f9792725700 1 --2- 172.21.2.102:0/2283421077 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f976c0186c0 0x7f976c01abc0 crc :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:54.424 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:54.422 7f9792725700 1 -- 172.21.2.102:0/2283421077 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f978c093f10 msgr2=0x7f978c094310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:54.424 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:54.422 7f9792725700 1 --2- 172.21.2.102:0/2283421077 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f978c093f10 0x7f978c094310 secure :-1 s=READY pgs=2743 cs=0 l=1 rev1=1 rx=0x7f977c009630 tx=0x7f977c00b970).stop 2021-02-10T11:35:54.424 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:54.422 7f9792725700 1 -- 172.21.2.102:0/2283421077 shutdown_connections 2021-02-10T11:35:54.424 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:54.422 7f9792725700 1 --2- 172.21.2.102:0/2283421077 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f976c0186c0 0x7f976c01abc0 unknown :-1 s=CLOSED pgs=1205 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:54.424 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:54.422 7f9792725700 1 --2- 172.21.2.102:0/2283421077 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f978c092d90 0x7f978c093190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:54.424 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:54.422 7f9792725700 1 --2- 172.21.2.102:0/2283421077 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f978c0ae9e0 0x7f978c0aede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:54.424 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:54.422 7f9792725700 1 --2- 172.21.2.102:0/2283421077 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f978c093f10 0x7f978c094310 unknown :-1 s=CLOSED pgs=2743 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:54.425 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:54.422 7f9792725700 1 -- 172.21.2.102:0/2283421077 >> 172.21.2.102:0/2283421077 conn(0x7f978c19f8a0 msgr2=0x7f978c094980 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:54.425 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:54.423 7f9792725700 1 -- 172.21.2.102:0/2283421077 shutdown_connections 2021-02-10T11:35:54.425 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:54.423 7f9792725700 1 -- 172.21.2.102:0/2283421077 wait complete. 2021-02-10T11:35:54.432 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:03:37.322519', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2021-02-10T11:35:54.432 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:35:54.865 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:54.846 7f90fbb02700 1 Processor -- start 2021-02-10T11:35:54.866 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:54.847 7f90fbb02700 1 -- start start 2021-02-10T11:35:54.866 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:54.847 7f90fbb02700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f90f40d1870 0x7f90f41b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:54.866 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:54.847 7f90fbb02700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f90f40cff70 0x7f90f4109130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:54.866 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:54.847 7f90fbb02700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90f4109a60 con 0x7f90f41b0ae0 2021-02-10T11:35:54.866 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:54.847 7f90fbb02700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90f4109750 con 0x7f90f40d1870 2021-02-10T11:35:54.866 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:54.847 7f90fbb02700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90f4088070 con 0x7f90f40cff70 2021-02-10T11:35:54.866 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:54.847 7f90f911d700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f90f41b0ae0 0x7f90f4108940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:57772/0 (socket says 172.21.2.102:57772) 2021-02-10T11:35:54.866 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:54.847 7f90f911d700 1 -- 172.21.2.102:0/105414462 learned_addr learned my addr 172.21.2.102:0/105414462 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:35:54.866 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:54.848 7f90f891c700 1 -- 172.21.2.102:0/105414462 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 406956397 0 0) 0x7f90f00009b0 con 0x7f90f41b0ae0 2021-02-10T11:35:54.866 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:54.848 7f90f891c700 1 -- 172.21.2.102:0/105414462 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f90e0002fe0 con 0x7f90f41b0ae0 2021-02-10T11:35:54.866 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:54.848 7f90fa11f700 1 --2- 172.21.2.102:0/105414462 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f90f40cff70 0x7f90f4109130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:54.867 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:54.848 7f90f991e700 1 --2- 172.21.2.102:0/105414462 >> v2:172.21.2.113:3300/0 conn(0x7f90f40d1870 0x7f90f41b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:54.867 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:54.848 7f90f891c700 1 -- 172.21.2.102:0/105414462 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3818124786 0 0) 0x7f90f0001010 con 0x7f90f41b0ae0 2021-02-10T11:35:54.867 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:54.848 7f90f891c700 1 -- 172.21.2.102:0/105414462 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f90f40cff70 msgr2=0x7f90f4109130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:54.867 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:54.848 7f90f891c700 1 --2- 172.21.2.102:0/105414462 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f90f40cff70 0x7f90f4109130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:54.867 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:54.848 7f90f891c700 1 -- 172.21.2.102:0/105414462 >> v2:172.21.2.113:3300/0 conn(0x7f90f40d1870 msgr2=0x7f90f41b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:54.867 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:54.848 7f90f891c700 1 --2- 172.21.2.102:0/105414462 >> v2:172.21.2.113:3300/0 conn(0x7f90f40d1870 0x7f90f41b04c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:54.867 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:54.848 7f90f891c700 1 -- 172.21.2.102:0/105414462 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90f40d1c70 con 0x7f90f41b0ae0 2021-02-10T11:35:54.867 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:54.848 7f90f891c700 1 -- 172.21.2.102:0/105414462 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7f90f0001530 con 0x7f90f41b0ae0 2021-02-10T11:35:54.867 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:54.848 7f90f991e700 1 --2- 172.21.2.102:0/105414462 >> v2:172.21.2.113:3300/0 conn(0x7f90f40d1870 0x7f90f41b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:35:54.867 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:54.848 7f90f891c700 1 -- 172.21.2.102:0/105414462 >> v1:172.21.2.102:6789/0 conn(0x7f90f41b0ae0 legacy=0x7f90f4108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:54.867 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:54.848 7f90f891c700 1 --2- 172.21.2.102:0/105414462 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f90e0003a40 0x7f90e0005f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:54.867 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:54.848 7f90f891c700 1 --2- 172.21.2.102:0/105414462 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f90e00067d0 0x7f90e0008cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:54.867 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:54.849 7f90f891c700 1 --2- 172.21.2.102:0/105414462 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f90e004d5a0 0x7f90e004fa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:54.867 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:54.849 7f90f891c700 1 -- 172.21.2.102:0/105414462 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90e00502e0 con 0x7f90e004d5a0 2021-02-10T11:35:54.868 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:54.849 7f90f891c700 1 -- 172.21.2.102:0/105414462 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90e0050500 con 0x7f90e00067d0 2021-02-10T11:35:54.868 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:54.849 7f90f891c700 1 -- 172.21.2.102:0/105414462 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90e0050720 con 0x7f90e0003a40 2021-02-10T11:35:54.868 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:54.849 7f90f991e700 1 --2- 172.21.2.102:0/105414462 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f90e0003a40 0x7f90e0005f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:54.868 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:54.849 7f90fa11f700 1 --2- 172.21.2.102:0/105414462 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f90e004d5a0 0x7f90e004fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:54.868 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:54.849 7f90f911d700 1 --2- 172.21.2.102:0/105414462 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f90e00067d0 0x7f90e0008cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:54.868 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:54.849 7f90fa11f700 1 -- 172.21.2.102:0/105414462 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f90e0003a40 msgr2=0x7f90e0005f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:54.868 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:54.849 7f90fa11f700 1 --2- 172.21.2.102:0/105414462 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f90e0003a40 0x7f90e0005f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:54.868 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:54.849 7f90fa11f700 1 -- 172.21.2.102:0/105414462 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f90e00067d0 msgr2=0x7f90e0008cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:54.868 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:54.849 7f90fa11f700 1 --2- 172.21.2.102:0/105414462 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f90e00067d0 0x7f90e0008cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:54.868 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:54.849 7f90fa11f700 1 -- 172.21.2.102:0/105414462 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90e00509c0 con 0x7f90e004d5a0 2021-02-10T11:35:54.868 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:54.849 7f90f911d700 1 --2- 172.21.2.102:0/105414462 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f90e00067d0 0x7f90e0008cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:35:54.868 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:54.849 7f90fa11f700 1 --2- 172.21.2.102:0/105414462 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f90e004d5a0 0x7f90e004fa30 secure :-1 s=READY pgs=2745 cs=0 l=1 rev1=1 rx=0x7f90ec006af0 tx=0x7f90ec009ad0).ready entity=mon.0 client_cookie=430b7d7bc289c391 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:54.869 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:54.849 7f90f891c700 1 -- 172.21.2.102:0/105414462 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f90ec0022e0 con 0x7f90e004d5a0 2021-02-10T11:35:54.869 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:54.850 7f90f891c700 1 -- 172.21.2.102:0/105414462 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90ec0066c0 con 0x7f90e004d5a0 2021-02-10T11:35:54.869 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:54.850 7f90f891c700 1 -- 172.21.2.102:0/105414462 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f90ec0028a0 con 0x7f90e004d5a0 2021-02-10T11:35:54.869 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:54.850 7f90fbb02700 1 -- 172.21.2.102:0/105414462 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f90e004d5a0 msgr2=0x7f90e004fa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:54.869 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:54.850 7f90fbb02700 1 --2- 172.21.2.102:0/105414462 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f90e004d5a0 0x7f90e004fa30 secure :-1 s=READY pgs=2745 cs=0 l=1 rev1=1 rx=0x7f90ec006af0 tx=0x7f90ec009ad0).stop 2021-02-10T11:35:54.869 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:54.850 7f90fa11f700 1 -- 172.21.2.102:0/105414462 reap_dead start 2021-02-10T11:35:54.869 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:54.850 7f90fbb02700 1 -- 172.21.2.102:0/105414462 shutdown_connections 2021-02-10T11:35:54.869 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:54.850 7f90fbb02700 1 -- 172.21.2.102:0/105414462 >> 172.21.2.102:0/105414462 conn(0x7f90f40ce420 msgr2=0x7f90f40d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:54.869 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:54.850 7f90fbb02700 1 -- 172.21.2.102:0/105414462 shutdown_connections 2021-02-10T11:35:54.869 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:54.850 7f90fbb02700 1 -- 172.21.2.102:0/105414462 wait complete. 2021-02-10T11:35:54.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:54.865 7f90fbb02700 1 Processor -- start 2021-02-10T11:35:54.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:54.865 7f90fbb02700 1 -- start start 2021-02-10T11:35:54.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:54.865 7f90fbb02700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f90f41a79a0 0x7f90f41a7da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:54.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:54.865 7f90fbb02700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f90f40bdb40 0x7f90f40bdf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:54.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:54.865 7f90fbb02700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f90f4082ca0 0x7f90f40830a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:54.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:54.865 7f90fbb02700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90f419d990 con 0x7f90f40bdb40 2021-02-10T11:35:54.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:54.865 7f90fbb02700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90f419dbb0 con 0x7f90f4082ca0 2021-02-10T11:35:54.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:54.865 7f90fbb02700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90f40836c0 con 0x7f90f41a79a0 2021-02-10T11:35:54.872 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:54.866 7f90f911d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f90f40bdb40 0x7f90f40bdf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:54.872 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:54.866 7f90f911d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f90f40bdb40 0x7f90f40bdf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:37330/0 (socket says 172.21.2.102:37330) 2021-02-10T11:35:54.872 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:54.866 7f90f911d700 1 -- 172.21.2.102:0/1552798593 learned_addr learned my addr 172.21.2.102:0/1552798593 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:35:54.872 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:54.866 7f90fa11f700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f90f4082ca0 0x7f90f40830a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:54.872 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:54.866 7f90f991e700 1 --2- 172.21.2.102:0/1552798593 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f90f41a79a0 0x7f90f41a7da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:54.872 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:54.866 7f90f911d700 1 -- 172.21.2.102:0/1552798593 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f90f41a79a0 msgr2=0x7f90f41a7da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:54.872 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:54.866 7f90f911d700 1 --2- 172.21.2.102:0/1552798593 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f90f41a79a0 0x7f90f41a7da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:54.872 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:54.866 7f90f911d700 1 -- 172.21.2.102:0/1552798593 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f90f4082ca0 msgr2=0x7f90f40830a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:54.872 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:54.866 7f90f911d700 1 --2- 172.21.2.102:0/1552798593 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f90f4082ca0 0x7f90f40830a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:54.872 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:54.866 7f90f911d700 1 -- 172.21.2.102:0/1552798593 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90f40838e0 con 0x7f90f40bdb40 2021-02-10T11:35:54.872 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:54.866 7f90f911d700 1 --2- 172.21.2.102:0/1552798593 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f90f40bdb40 0x7f90f40bdf40 secure :-1 s=READY pgs=2746 cs=0 l=1 rev1=1 rx=0x7f90f0001c10 tx=0x7f90f000a8f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:54.872 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:54.867 7f90e67fc700 1 -- 172.21.2.102:0/1552798593 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f90f000ace0 con 0x7f90f40bdb40 2021-02-10T11:35:54.872 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:54.867 7f90e67fc700 1 -- 172.21.2.102:0/1552798593 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90f0004350 con 0x7f90f40bdb40 2021-02-10T11:35:54.873 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:54.867 7f90fbb02700 1 -- 172.21.2.102:0/1552798593 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90f40838e0 con 0x7f90f40bdb40 2021-02-10T11:35:54.873 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:54.867 7f90e67fc700 1 -- 172.21.2.102:0/1552798593 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f90f00047b0 con 0x7f90f40bdb40 2021-02-10T11:35:54.873 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:54.867 7f90fbb02700 1 -- 172.21.2.102:0/1552798593 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90f40838e0 con 0x7f90f40bdb40 2021-02-10T11:35:54.873 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:54.867 7f90e67fc700 1 -- 172.21.2.102:0/1552798593 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f90f0007fd0 con 0x7f90f40bdb40 2021-02-10T11:35:54.873 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:54.867 7f90e67fc700 1 --2- 172.21.2.102:0/1552798593 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f90dc018730 0x7f90dc01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:54.873 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:54.867 7f90e67fc700 1 -- 172.21.2.102:0/1552798593 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 4753+0+0 (secure 0 0 0) 0x7f90f0032300 con 0x7f90f40bdb40 2021-02-10T11:35:54.873 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:54.868 7f90f991e700 1 --2- 172.21.2.102:0/1552798593 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f90dc018730 0x7f90dc01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:54.873 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:54.868 7f90f991e700 1 --2- 172.21.2.102:0/1552798593 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f90dc018730 0x7f90dc01ac10 crc :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:54.899 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:54.897 7f90fbb02700 1 -- 172.21.2.102:0/1552798593 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90f4082a70 con 0x7f90f40bdb40 2021-02-10T11:35:54.900 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:54.900 7f90e67fc700 1 -- 172.21.2.102:0/1552798593 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f90f0007fd0 con 0x7f90f40bdb40 2021-02-10T11:35:55.029 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:55.028 7f90fbb02700 1 -- 172.21.2.102:0/1552798593 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f90f4082a70 con 0x7f90f40bdb40 2021-02-10T11:35:55.029 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:55.028 7f90e67fc700 1 -- 172.21.2.102:0/1552798593 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v149) v1 ==== 92+0+10 (secure 0 0 0) 0x7f90f000ace0 con 0x7f90f40bdb40 2021-02-10T11:35:55.030 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:35:55.031 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:55.030 7f90fbb02700 1 -- 172.21.2.102:0/1552798593 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f90dc018730 msgr2=0x7f90dc01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:55.032 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:55.030 7f90fbb02700 1 --2- 172.21.2.102:0/1552798593 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f90dc018730 0x7f90dc01ac10 crc :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:55.032 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:55.030 7f90fbb02700 1 -- 172.21.2.102:0/1552798593 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f90f40bdb40 msgr2=0x7f90f40bdf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:55.032 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:55.030 7f90fbb02700 1 --2- 172.21.2.102:0/1552798593 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f90f40bdb40 0x7f90f40bdf40 secure :-1 s=READY pgs=2746 cs=0 l=1 rev1=1 rx=0x7f90f0001c10 tx=0x7f90f000a8f0).stop 2021-02-10T11:35:55.032 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:55.030 7f90fbb02700 1 -- 172.21.2.102:0/1552798593 shutdown_connections 2021-02-10T11:35:55.032 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:55.030 7f90fbb02700 1 --2- 172.21.2.102:0/1552798593 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f90dc018730 0x7f90dc01ac10 unknown :-1 s=CLOSED pgs=1206 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:55.032 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:55.030 7f90fbb02700 1 --2- 172.21.2.102:0/1552798593 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f90f4082ca0 0x7f90f40830a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:55.032 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:55.030 7f90fbb02700 1 --2- 172.21.2.102:0/1552798593 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f90f40bdb40 0x7f90f40bdf40 unknown :-1 s=CLOSED pgs=2746 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:55.032 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:55.030 7f90fbb02700 1 --2- 172.21.2.102:0/1552798593 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f90f41a79a0 0x7f90f41a7da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:55.032 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:55.030 7f90fbb02700 1 -- 172.21.2.102:0/1552798593 >> 172.21.2.102:0/1552798593 conn(0x7f90f40bef00 msgr2=0x7f90f409f200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:55.032 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:55.030 7f90fbb02700 1 -- 172.21.2.102:0/1552798593 shutdown_connections 2021-02-10T11:35:55.032 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:55.030 7f90fbb02700 1 -- 172.21.2.102:0/1552798593 wait complete. 2021-02-10T11:35:55.040 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --format=json-pretty osd lspools 2021-02-10T11:35:55.470 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:55.451 7f8e9bf13700 1 Processor -- start 2021-02-10T11:35:55.470 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:55.451 7f8e9bf13700 1 -- start start 2021-02-10T11:35:55.470 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:55.451 7f8e9bf13700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8e940d1870 0x7f8e941b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:55.470 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:55.451 7f8e9bf13700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e940f0b10 con 0x7f8e941b0a10 2021-02-10T11:35:55.470 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:55.451 7f8e9bf13700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e940d0310 con 0x7f8e940f0650 2021-02-10T11:35:55.470 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:55.451 7f8e9bf13700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e940e5ed0 con 0x7f8e940d1870 2021-02-10T11:35:55.470 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:55.451 7f8e9952e700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f8e941b0a10 0x7f8e940cf330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:57790/0 (socket says 172.21.2.102:57790) 2021-02-10T11:35:55.470 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:55.451 7f8e9952e700 1 -- 172.21.2.102:0/1819514881 learned_addr learned my addr 172.21.2.102:0/1819514881 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:35:55.470 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:55.452 7f8e99d2f700 1 --2- 172.21.2.102:0/1819514881 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8e940d1870 0x7f8e941b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:55.471 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:55.452 7f8e98d2d700 1 -- 172.21.2.102:0/1819514881 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 206850335 0 0) 0x7f8e88001430 con 0x7f8e941b0a10 2021-02-10T11:35:55.471 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:55.452 7f8e98d2d700 1 -- 172.21.2.102:0/1819514881 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e80002fe0 con 0x7f8e941b0a10 2021-02-10T11:35:55.471 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:55.452 7f8e98d2d700 1 -- 172.21.2.102:0/1819514881 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 166758954 0 0) 0x7f8e88001430 con 0x7f8e941b0a10 2021-02-10T11:35:55.471 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:55.452 7f8e98d2d700 1 -- 172.21.2.102:0/1819514881 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8e940d1870 msgr2=0x7f8e941b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:55.471 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:55.452 7f8e98d2d700 1 --2- 172.21.2.102:0/1819514881 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8e940d1870 0x7f8e941b03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:55.471 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:55.452 7f8e98d2d700 1 -- 172.21.2.102:0/1819514881 >> v1:172.21.2.113:6789/0 conn(0x7f8e940f0650 legacy=0x7f8e940cfb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:55.471 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:55.452 7f8e98d2d700 1 -- 172.21.2.102:0/1819514881 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e940d0530 con 0x7f8e941b0a10 2021-02-10T11:35:55.471 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:55.452 7f8e98d2d700 1 -- 172.21.2.102:0/1819514881 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7f8e88001930 con 0x7f8e941b0a10 2021-02-10T11:35:55.471 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:55.452 7f8e98d2d700 1 -- 172.21.2.102:0/1819514881 >> v1:172.21.2.102:6789/0 conn(0x7f8e941b0a10 legacy=0x7f8e940cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:55.471 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:55.452 7f8e98d2d700 1 --2- 172.21.2.102:0/1819514881 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8e80003a60 0x7f8e80005eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:55.471 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:55.452 7f8e98d2d700 1 --2- 172.21.2.102:0/1819514881 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8e80006750 0x7f8e80008c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:55.471 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:55.452 7f8e98d2d700 1 --2- 172.21.2.102:0/1819514881 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8e800094f0 0x7f8e8000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:55.472 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:55.452 7f8e98d2d700 1 -- 172.21.2.102:0/1819514881 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e8000c2c0 con 0x7f8e80006750 2021-02-10T11:35:55.472 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:55.452 7f8e98d2d700 1 -- 172.21.2.102:0/1819514881 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e8000c4e0 con 0x7f8e800094f0 2021-02-10T11:35:55.472 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:55.452 7f8e98d2d700 1 -- 172.21.2.102:0/1819514881 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e8000c700 con 0x7f8e80003a60 2021-02-10T11:35:55.472 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:55.452 7f8e9952e700 1 --2- 172.21.2.102:0/1819514881 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8e80006750 0x7f8e80008c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:55.472 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:55.453 7f8e99d2f700 1 --2- 172.21.2.102:0/1819514881 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8e80003a60 0x7f8e80005eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:55.472 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:55.453 7f8e9952e700 1 -- 172.21.2.102:0/1819514881 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8e80003a60 msgr2=0x7f8e80005eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:55.472 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:55.453 7f8e9952e700 1 --2- 172.21.2.102:0/1819514881 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8e80003a60 0x7f8e80005eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:55.472 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:55.453 7f8e9952e700 1 -- 172.21.2.102:0/1819514881 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8e800094f0 msgr2=0x7f8e8000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:55.472 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:55.453 7f8e9952e700 1 --2- 172.21.2.102:0/1819514881 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8e800094f0 0x7f8e8000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:55.472 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:55.453 7f8e9952e700 1 -- 172.21.2.102:0/1819514881 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e80050970 con 0x7f8e80006750 2021-02-10T11:35:55.472 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:55.453 7f8e99d2f700 1 --2- 172.21.2.102:0/1819514881 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8e80003a60 0x7f8e80005eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:35:55.473 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:55.453 7f8e9952e700 1 --2- 172.21.2.102:0/1819514881 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8e80006750 0x7f8e80008c50 secure :-1 s=READY pgs=2748 cs=0 l=1 rev1=1 rx=0x7f8e88007d10 tx=0x7f8e8800a310).ready entity=mon.0 client_cookie=61cbda20158d19cf server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:55.473 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:55.453 7f8e98d2d700 1 -- 172.21.2.102:0/1819514881 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8e880070c0 con 0x7f8e80006750 2021-02-10T11:35:55.473 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:55.453 7f8e98d2d700 1 -- 172.21.2.102:0/1819514881 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e880072c0 con 0x7f8e80006750 2021-02-10T11:35:55.473 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:55.453 7f8e98d2d700 1 -- 172.21.2.102:0/1819514881 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8e8800a980 con 0x7f8e80006750 2021-02-10T11:35:55.473 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:55.453 7f8e9bf13700 1 -- 172.21.2.102:0/1819514881 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8e80006750 msgr2=0x7f8e80008c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:55.473 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:55.453 7f8e9bf13700 1 --2- 172.21.2.102:0/1819514881 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8e80006750 0x7f8e80008c50 secure :-1 s=READY pgs=2748 cs=0 l=1 rev1=1 rx=0x7f8e88007d10 tx=0x7f8e8800a310).stop 2021-02-10T11:35:55.473 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:55.453 7f8e9a530700 1 -- 172.21.2.102:0/1819514881 reap_dead start 2021-02-10T11:35:55.473 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:55.454 7f8e9bf13700 1 -- 172.21.2.102:0/1819514881 shutdown_connections 2021-02-10T11:35:55.473 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:55.454 7f8e9bf13700 1 -- 172.21.2.102:0/1819514881 >> 172.21.2.102:0/1819514881 conn(0x7f8e940ce450 msgr2=0x7f8e940d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:55.473 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:55.454 7f8e9bf13700 1 -- 172.21.2.102:0/1819514881 shutdown_connections 2021-02-10T11:35:55.473 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:55.454 7f8e9bf13700 1 -- 172.21.2.102:0/1819514881 wait complete. 2021-02-10T11:35:55.473 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:55.469 7f8e9bf13700 1 Processor -- start 2021-02-10T11:35:55.473 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:55.469 7f8e9bf13700 1 -- start start 2021-02-10T11:35:55.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:55.469 7f8e9bf13700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8e941a05d0 0x7f8e940a87c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:55.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:55.469 7f8e9bf13700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8e940a8de0 0x7f8e940a2630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:55.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:55.469 7f8e9bf13700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8e940a2c50 0x7f8e940a3050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:55.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:55.469 7f8e9bf13700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e940b32d0 con 0x7f8e940a8de0 2021-02-10T11:35:55.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:55.469 7f8e9bf13700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e940a91e0 con 0x7f8e941a05d0 2021-02-10T11:35:55.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:55.469 7f8e9bf13700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e940b37d0 con 0x7f8e940a2c50 2021-02-10T11:35:55.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:55.469 7f8e9952e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8e940a8de0 0x7f8e940a2630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:55.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:55.469 7f8e9952e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8e940a8de0 0x7f8e940a2630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:37348/0 (socket says 172.21.2.102:37348) 2021-02-10T11:35:55.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:55.469 7f8e9952e700 1 -- 172.21.2.102:0/3451919523 learned_addr learned my addr 172.21.2.102:0/3451919523 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:35:55.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:55.470 7f8e9952e700 1 -- 172.21.2.102:0/3451919523 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8e940a2c50 msgr2=0x7f8e940a3050 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:35:55.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:55.470 7f8e9952e700 1 --2- 172.21.2.102:0/3451919523 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8e940a2c50 0x7f8e940a3050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:55.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:55.470 7f8e9952e700 1 -- 172.21.2.102:0/3451919523 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8e941a05d0 msgr2=0x7f8e940a87c0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:35:55.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:55.470 7f8e9952e700 1 --2- 172.21.2.102:0/3451919523 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8e941a05d0 0x7f8e940a87c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:55.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:55.470 7f8e9952e700 1 -- 172.21.2.102:0/3451919523 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e940b39b0 con 0x7f8e940a8de0 2021-02-10T11:35:55.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:55.470 7f8e9952e700 1 --2- 172.21.2.102:0/3451919523 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8e940a8de0 0x7f8e940a2630 secure :-1 s=READY pgs=2749 cs=0 l=1 rev1=1 rx=0x7f8e8800a920 tx=0x7f8e8800acc0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:55.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:55.470 7f8e867fc700 1 -- 172.21.2.102:0/3451919523 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8e88000d30 con 0x7f8e940a8de0 2021-02-10T11:35:55.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:55.470 7f8e867fc700 1 -- 172.21.2.102:0/3451919523 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e880070d0 con 0x7f8e940a8de0 2021-02-10T11:35:55.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:55.470 7f8e867fc700 1 -- 172.21.2.102:0/3451919523 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8e880046d0 con 0x7f8e940a8de0 2021-02-10T11:35:55.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:55.470 7f8e9bf13700 1 -- 172.21.2.102:0/3451919523 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e940b39b0 con 0x7f8e940a8de0 2021-02-10T11:35:55.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:55.470 7f8e9bf13700 1 -- 172.21.2.102:0/3451919523 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e940b39b0 con 0x7f8e940a8de0 2021-02-10T11:35:55.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:55.471 7f8e867fc700 1 -- 172.21.2.102:0/3451919523 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f8e880072b0 con 0x7f8e940a8de0 2021-02-10T11:35:55.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:55.471 7f8e867fc700 1 --2- 172.21.2.102:0/3451919523 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8e7c0186c0 0x7f8e7c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:55.477 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:55.471 7f8e867fc700 1 -- 172.21.2.102:0/3451919523 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 4753+0+0 (secure 0 0 0) 0x7f8e880325f0 con 0x7f8e940a8de0 2021-02-10T11:35:55.477 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:55.471 7f8e99d2f700 1 --2- 172.21.2.102:0/3451919523 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8e7c0186c0 0x7f8e7c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:55.477 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:55.471 7f8e99d2f700 1 --2- 172.21.2.102:0/3451919523 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8e7c0186c0 0x7f8e7c01abc0 crc :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:55.502 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:55.501 7f8e9bf13700 1 -- 172.21.2.102:0/3451919523 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e940ebc80 con 0x7f8e940a8de0 2021-02-10T11:35:55.504 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:55.504 7f8e867fc700 1 -- 172.21.2.102:0/3451919523 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f8e88001090 con 0x7f8e940a8de0 2021-02-10T11:35:55.629 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:55.628 7f8e9bf13700 1 -- 172.21.2.102:0/3451919523 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f8e940f0b10 con 0x7f8e940a8de0 2021-02-10T11:35:55.629 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:55.628 7f8e867fc700 1 -- 172.21.2.102:0/3451919523 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v149) v1 ==== 84+0+64 (secure 0 0 0) 0x7f8e880072b0 con 0x7f8e940a8de0 2021-02-10T11:35:55.630 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:35:55.630 INFO:teuthology.orchestra.run.gibba002.stdout:[ 2021-02-10T11:35:55.630 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:35:55.630 INFO:teuthology.orchestra.run.gibba002.stdout: "poolnum": 1, 2021-02-10T11:35:55.630 INFO:teuthology.orchestra.run.gibba002.stdout: "poolname": "rbd" 2021-02-10T11:35:55.630 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:35:55.630 INFO:teuthology.orchestra.run.gibba002.stdout:] 2021-02-10T11:35:55.631 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:55.630 7f8e9bf13700 1 -- 172.21.2.102:0/3451919523 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8e7c0186c0 msgr2=0x7f8e7c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:55.632 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:55.630 7f8e9bf13700 1 --2- 172.21.2.102:0/3451919523 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8e7c0186c0 0x7f8e7c01abc0 crc :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:55.632 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:55.630 7f8e9bf13700 1 -- 172.21.2.102:0/3451919523 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8e940a8de0 msgr2=0x7f8e940a2630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:55.632 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:55.630 7f8e9bf13700 1 --2- 172.21.2.102:0/3451919523 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8e940a8de0 0x7f8e940a2630 secure :-1 s=READY pgs=2749 cs=0 l=1 rev1=1 rx=0x7f8e8800a920 tx=0x7f8e8800acc0).stop 2021-02-10T11:35:55.632 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:55.630 7f8e9bf13700 1 -- 172.21.2.102:0/3451919523 shutdown_connections 2021-02-10T11:35:55.632 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:55.630 7f8e9bf13700 1 --2- 172.21.2.102:0/3451919523 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8e940a2c50 0x7f8e940a3050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:55.632 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:55.630 7f8e9bf13700 1 --2- 172.21.2.102:0/3451919523 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8e940a8de0 0x7f8e940a2630 unknown :-1 s=CLOSED pgs=2749 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:55.632 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:55.630 7f8e9bf13700 1 --2- 172.21.2.102:0/3451919523 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8e7c0186c0 0x7f8e7c01abc0 unknown :-1 s=CLOSED pgs=1207 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:55.632 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:55.630 7f8e9bf13700 1 --2- 172.21.2.102:0/3451919523 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8e941a05d0 0x7f8e940a87c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:55.632 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:55.630 7f8e9bf13700 1 -- 172.21.2.102:0/3451919523 >> 172.21.2.102:0/3451919523 conn(0x7f8e940a9bf0 msgr2=0x7f8e940ebeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:55.632 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:55.631 7f8e9bf13700 1 -- 172.21.2.102:0/3451919523 shutdown_connections 2021-02-10T11:35:55.632 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:55.631 7f8e9bf13700 1 -- 172.21.2.102:0/3451919523 wait complete. 2021-02-10T11:35:55.640 INFO:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2021-02-10T11:35:55.640 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool create cephfs_metadata 8 2021-02-10T11:35:56.072 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:56.053 7f19196f4700 1 Processor -- start 2021-02-10T11:35:56.072 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:56.053 7f19196f4700 1 -- start start 2021-02-10T11:35:56.073 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:56.054 7f19196f4700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f19140d1870 0x7f19141b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:56.073 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:56.054 7f19196f4700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f19141b0ae0 0x7f1914108940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:56.073 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:56.054 7f19196f4700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1914109a60 con 0x7f19140d1c70 2021-02-10T11:35:56.073 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:56.054 7f19196f4700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19140e5ef0 con 0x7f19141b0ae0 2021-02-10T11:35:56.073 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:56.054 7f19196f4700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1914109750 con 0x7f19140d1870 2021-02-10T11:35:56.073 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:56.055 7f19132bf700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f19140d1870 0x7f19141b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:56.074 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:56.055 7f1913ac0700 1 --1- >> v1:172.21.2.113:6789/0 conn(0x7f19140d1c70 0x7f1914108f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.113:6789/0 says I am v1:172.21.2.102:36902/0 (socket says 172.21.2.102:36902) 2021-02-10T11:35:56.074 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:56.055 7f1912abe700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f19141b0ae0 0x7f1914108940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:56.074 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:56.055 7f1913ac0700 1 -- 172.21.2.102:0/2796283760 learned_addr learned my addr 172.21.2.102:0/2796283760 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:35:56.074 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:56.055 7f19122bd700 1 -- 172.21.2.102:0/2796283760 <== mon.1 v1:172.21.2.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 627209513 0 0) 0x7f190c001440 con 0x7f19140d1c70 2021-02-10T11:35:56.074 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:56.056 7f19122bd700 1 -- 172.21.2.102:0/2796283760 --> v1:172.21.2.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18fc002fe0 con 0x7f19140d1c70 2021-02-10T11:35:56.075 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:56.056 7f19132bf700 1 -- 172.21.2.102:0/2796283760 >> v2:172.21.2.113:3300/0 conn(0x7f19141b0ae0 msgr2=0x7f1914108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:56.075 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:56.056 7f19132bf700 1 --2- 172.21.2.102:0/2796283760 >> v2:172.21.2.113:3300/0 conn(0x7f19141b0ae0 0x7f1914108940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:56.075 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:56.056 7f19132bf700 1 -- 172.21.2.102:0/2796283760 >> v1:172.21.2.113:6789/0 conn(0x7f19140d1c70 legacy=0x7f1914108f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:56.075 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:56.056 7f19132bf700 1 -- 172.21.2.102:0/2796283760 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1914088070 con 0x7f19140d1870 2021-02-10T11:35:56.075 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:56.056 7f1912abe700 1 --2- 172.21.2.102:0/2796283760 >> v2:172.21.2.113:3300/0 conn(0x7f19141b0ae0 0x7f1914108940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:35:56.075 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:56.056 7f19132bf700 1 --2- 172.21.2.102:0/2796283760 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f19140d1870 0x7f19141b04c0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0x7f1904006650 tx=0x7f1904009a80).ready entity=mon.2 client_cookie=c078ad1fe37c7f85 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:56.075 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:56.056 7f19122bd700 1 -- 172.21.2.102:0/2796283760 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f19040009b0 con 0x7f19140d1870 2021-02-10T11:35:56.075 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:56.056 7f19122bd700 1 -- 172.21.2.102:0/2796283760 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f190400c0c0 con 0x7f19140d1870 2021-02-10T11:35:56.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:56.056 7f19122bd700 1 -- 172.21.2.102:0/2796283760 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f190400c520 con 0x7f19140d1870 2021-02-10T11:35:56.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:56.056 7f19196f4700 1 -- 172.21.2.102:0/2796283760 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f19140d1870 msgr2=0x7f19141b04c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:56.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:56.056 7f19196f4700 1 --2- 172.21.2.102:0/2796283760 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f19140d1870 0x7f19141b04c0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0x7f1904006650 tx=0x7f1904009a80).stop 2021-02-10T11:35:56.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:56.057 7f19196f4700 1 -- 172.21.2.102:0/2796283760 shutdown_connections 2021-02-10T11:35:56.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:56.057 7f19196f4700 1 --2- 172.21.2.102:0/2796283760 >> v2:172.21.2.113:3300/0 conn(0x7f19141b0ae0 0x7f1914108940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:56.077 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:56.057 7f19196f4700 1 --2- 172.21.2.102:0/2796283760 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f19140d1870 0x7f19141b04c0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:56.077 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:56.057 7f19196f4700 1 -- 172.21.2.102:0/2796283760 >> 172.21.2.102:0/2796283760 conn(0x7f19140ce420 msgr2=0x7f19140d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:56.077 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:56.057 7f19196f4700 1 -- 172.21.2.102:0/2796283760 shutdown_connections 2021-02-10T11:35:56.077 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:56.057 7f19196f4700 1 -- 172.21.2.102:0/2796283760 wait complete. 2021-02-10T11:35:56.079 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:56.072 7f19196f4700 1 Processor -- start 2021-02-10T11:35:56.079 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:56.072 7f19196f4700 1 -- start start 2021-02-10T11:35:56.079 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:56.072 7f19196f4700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f19140abd60 0x7f19140ac160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:56.079 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:56.072 7f19196f4700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f19140ac780 0x7f19140acb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:56.079 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:56.072 7f19196f4700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1914083910 0x7f1914083d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:56.079 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:56.072 7f19196f4700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1914087180 con 0x7f1914083910 2021-02-10T11:35:56.079 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:56.072 7f19196f4700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f191418fcd0 con 0x7f19140abd60 2021-02-10T11:35:56.079 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:56.072 7f19196f4700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1914084330 con 0x7f19140ac780 2021-02-10T11:35:56.080 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:56.072 7f1913ac0700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1914083910 0x7f1914083d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:56.080 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:56.072 7f1913ac0700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1914083910 0x7f1914083d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:37364/0 (socket says 172.21.2.102:37364) 2021-02-10T11:35:56.080 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:56.072 7f1913ac0700 1 -- 172.21.2.102:0/2094384132 learned_addr learned my addr 172.21.2.102:0/2094384132 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:35:56.080 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:56.073 7f1913ac0700 1 -- 172.21.2.102:0/2094384132 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f19140ac780 msgr2=0x7f19140acb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:56.080 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:56.073 7f1913ac0700 1 --2- 172.21.2.102:0/2094384132 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f19140ac780 0x7f19140acb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:56.080 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:56.073 7f1913ac0700 1 -- 172.21.2.102:0/2094384132 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f19140abd60 msgr2=0x7f19140ac160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:56.080 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:56.073 7f1913ac0700 1 --2- 172.21.2.102:0/2094384132 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f19140abd60 0x7f19140ac160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:56.080 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:56.073 7f1913ac0700 1 -- 172.21.2.102:0/2094384132 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1914084550 con 0x7f1914083910 2021-02-10T11:35:56.080 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:56.073 7f1913ac0700 1 --2- 172.21.2.102:0/2094384132 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1914083910 0x7f1914083d10 secure :-1 s=READY pgs=2750 cs=0 l=1 rev1=1 rx=0x7f190c00a940 tx=0x7f190c00ace0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:56.080 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:56.073 7f18fbfff700 1 -- 172.21.2.102:0/2094384132 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f190c0048b0 con 0x7f1914083910 2021-02-10T11:35:56.081 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:56.073 7f18fbfff700 1 -- 172.21.2.102:0/2094384132 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f190c004b70 con 0x7f1914083910 2021-02-10T11:35:56.081 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:56.073 7f18fbfff700 1 -- 172.21.2.102:0/2094384132 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f190c0070d0 con 0x7f1914083910 2021-02-10T11:35:56.081 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:56.073 7f19196f4700 1 -- 172.21.2.102:0/2094384132 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1914084550 con 0x7f1914083910 2021-02-10T11:35:56.081 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:56.073 7f19196f4700 1 -- 172.21.2.102:0/2094384132 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1914084ae0 con 0x7f1914083910 2021-02-10T11:35:56.081 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:56.074 7f18fbfff700 1 -- 172.21.2.102:0/2094384132 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f190c0046a0 con 0x7f1914083910 2021-02-10T11:35:56.081 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:56.074 7f18fbfff700 1 --2- 172.21.2.102:0/2094384132 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f18f40186c0 0x7f18f401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:56.081 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:56.074 7f18fbfff700 1 -- 172.21.2.102:0/2094384132 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 4753+0+0 (secure 0 0 0) 0x7f190c0325c0 con 0x7f1914083910 2021-02-10T11:35:56.081 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:56.074 7f19132bf700 1 --2- 172.21.2.102:0/2094384132 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f18f40186c0 0x7f18f401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:56.081 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:56.074 7f19132bf700 1 --2- 172.21.2.102:0/2094384132 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f18f40186c0 0x7f18f401abc0 crc :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:56.105 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:56.104 7f19196f4700 1 -- 172.21.2.102:0/2094384132 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1914084d90 con 0x7f1914083910 2021-02-10T11:35:56.107 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:56.107 7f18fbfff700 1 -- 172.21.2.102:0/2094384132 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f190c01c7a0 con 0x7f1914083910 2021-02-10T11:35:56.235 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:56.234 7f19196f4700 1 -- 172.21.2.102:0/2094384132 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"} v 0) v1 -- 0x7f1914084d90 con 0x7f1914083910 2021-02-10T11:35:56.656 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:56.655 7f18fbfff700 1 -- 172.21.2.102:0/2094384132 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"}]=0 pool 'cephfs_metadata' created v150) v1 ==== 133+0+0 (secure 0 0 0) 0x7f190c01c7a0 con 0x7f1914083910 2021-02-10T11:35:56.700 INFO:teuthology.orchestra.run.gibba002.stderr:pool 'cephfs_metadata' created 2021-02-10T11:35:56.702 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:56.701 7f19196f4700 1 -- 172.21.2.102:0/2094384132 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f18f40186c0 msgr2=0x7f18f401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:56.702 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:56.701 7f19196f4700 1 --2- 172.21.2.102:0/2094384132 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f18f40186c0 0x7f18f401abc0 crc :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:56.703 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:56.701 7f19196f4700 1 -- 172.21.2.102:0/2094384132 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1914083910 msgr2=0x7f1914083d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:56.703 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:56.701 7f19196f4700 1 --2- 172.21.2.102:0/2094384132 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1914083910 0x7f1914083d10 secure :-1 s=READY pgs=2750 cs=0 l=1 rev1=1 rx=0x7f190c00a940 tx=0x7f190c00ace0).stop 2021-02-10T11:35:56.703 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:56.701 7f19196f4700 1 -- 172.21.2.102:0/2094384132 shutdown_connections 2021-02-10T11:35:56.703 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:56.701 7f19196f4700 1 --2- 172.21.2.102:0/2094384132 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1914083910 0x7f1914083d10 unknown :-1 s=CLOSED pgs=2750 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:56.703 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:56.701 7f19196f4700 1 --2- 172.21.2.102:0/2094384132 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f19140ac780 0x7f19140acb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:56.703 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:56.701 7f19196f4700 1 --2- 172.21.2.102:0/2094384132 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f18f40186c0 0x7f18f401abc0 unknown :-1 s=CLOSED pgs=1208 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:56.703 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:56.701 7f19196f4700 1 --2- 172.21.2.102:0/2094384132 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f19140abd60 0x7f19140ac160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:56.703 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:56.701 7f19196f4700 1 -- 172.21.2.102:0/2094384132 >> 172.21.2.102:0/2094384132 conn(0x7f19140a2b30 msgr2=0x7f191409ce60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:56.703 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:56.702 7f19196f4700 1 -- 172.21.2.102:0/2094384132 shutdown_connections 2021-02-10T11:35:56.703 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:56.702 7f19196f4700 1 -- 172.21.2.102:0/2094384132 wait complete. 2021-02-10T11:35:56.711 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool create cephfs_data 8 2021-02-10T11:35:57.145 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:57.127 7f4a94a5c700 1 Processor -- start 2021-02-10T11:35:57.145 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:57.127 7f4a94a5c700 1 -- start start 2021-02-10T11:35:57.145 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:57.128 7f4a94a5c700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4a900d1870 0x7f4a901b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:57.145 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:57.128 7f4a94a5c700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f4a900cff70 0x7f4a90109130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:57.146 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:57.128 7f4a94a5c700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a90109a60 con 0x7f4a901b0ae0 2021-02-10T11:35:57.146 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:57.128 7f4a94a5c700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a90109750 con 0x7f4a900cff70 2021-02-10T11:35:57.146 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:57.128 7f4a94a5c700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a90088070 con 0x7f4a900d1870 2021-02-10T11:35:57.146 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:57.128 7f4a8e85e700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4a900d1870 0x7f4a901b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:57.146 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:57.128 7f4a8e05d700 1 --1- >> v1:172.21.2.113:6789/0 conn(0x7f4a901b0ae0 0x7f4a90108940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.113:6789/0 says I am v1:172.21.2.102:36918/0 (socket says 172.21.2.102:36918) 2021-02-10T11:35:57.146 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:57.128 7f4a8e85e700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4a900d1870 0x7f4a901b04c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.113:3301/0 says I am v2:172.21.2.102:42904/0 (socket says 172.21.2.102:42904) 2021-02-10T11:35:57.146 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:57.128 7f4a8e05d700 1 -- 172.21.2.102:0/2348706229 learned_addr learned my addr 172.21.2.102:0/2348706229 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:35:57.146 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:57.128 7f4a8f05f700 1 --2- 172.21.2.102:0/2348706229 >> v2:172.21.2.113:3300/0 conn(0x7f4a900cff70 0x7f4a90109130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:57.146 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:57.128 7f4a8d85c700 1 -- 172.21.2.102:0/2348706229 <== mon.1 v1:172.21.2.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 578470243 0 0) 0x7f4a800010b0 con 0x7f4a901b0ae0 2021-02-10T11:35:57.146 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:57.128 7f4a8d85c700 1 -- 172.21.2.102:0/2348706229 --> v1:172.21.2.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a7c002fe0 con 0x7f4a901b0ae0 2021-02-10T11:35:57.146 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:57.128 7f4a8f05f700 1 -- 172.21.2.102:0/2348706229 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4a900d1870 msgr2=0x7f4a901b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:57.146 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:57.128 7f4a8f05f700 1 --2- 172.21.2.102:0/2348706229 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4a900d1870 0x7f4a901b04c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:57.147 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:57.129 7f4a8f05f700 1 -- 172.21.2.102:0/2348706229 >> v1:172.21.2.113:6789/0 conn(0x7f4a901b0ae0 legacy=0x7f4a90108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:57.147 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:57.129 7f4a8f05f700 1 -- 172.21.2.102:0/2348706229 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a90088260 con 0x7f4a900cff70 2021-02-10T11:35:57.147 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:57.129 7f4a8e85e700 1 --2- 172.21.2.102:0/2348706229 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4a900d1870 0x7f4a901b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:35:57.147 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:57.129 7f4a8f05f700 1 --2- 172.21.2.102:0/2348706229 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4a900cff70 0x7f4a90109130 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 rx=0x7f4a88006e50 tx=0x7f4a88009e70).ready entity=mon.1 client_cookie=87191d5f674cb23a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:57.147 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:57.129 7f4a8d85c700 1 -- 172.21.2.102:0/2348706229 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4a88000f90 con 0x7f4a900cff70 2021-02-10T11:35:57.147 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:57.129 7f4a8d85c700 1 -- 172.21.2.102:0/2348706229 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a880066f0 con 0x7f4a900cff70 2021-02-10T11:35:57.147 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:57.129 7f4a8d85c700 1 -- 172.21.2.102:0/2348706229 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4a8800b700 con 0x7f4a900cff70 2021-02-10T11:35:57.147 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:57.129 7f4a94a5c700 1 -- 172.21.2.102:0/2348706229 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4a900cff70 msgr2=0x7f4a90109130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:57.147 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:57.129 7f4a94a5c700 1 --2- 172.21.2.102:0/2348706229 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4a900cff70 0x7f4a90109130 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 rx=0x7f4a88006e50 tx=0x7f4a88009e70).stop 2021-02-10T11:35:57.147 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:57.130 7f4a94a5c700 1 -- 172.21.2.102:0/2348706229 shutdown_connections 2021-02-10T11:35:57.147 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:57.130 7f4a94a5c700 1 --2- 172.21.2.102:0/2348706229 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4a900cff70 0x7f4a90109130 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:57.147 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:57.130 7f4a94a5c700 1 --2- 172.21.2.102:0/2348706229 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4a900d1870 0x7f4a901b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:57.147 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:57.130 7f4a94a5c700 1 -- 172.21.2.102:0/2348706229 >> 172.21.2.102:0/2348706229 conn(0x7f4a900ce420 msgr2=0x7f4a900d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:57.148 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:57.130 7f4a94a5c700 1 -- 172.21.2.102:0/2348706229 shutdown_connections 2021-02-10T11:35:57.148 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:57.130 7f4a94a5c700 1 -- 172.21.2.102:0/2348706229 wait complete. 2021-02-10T11:35:57.149 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:57.145 7f4a94a5c700 1 Processor -- start 2021-02-10T11:35:57.149 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:57.145 7f4a94a5c700 1 -- start start 2021-02-10T11:35:57.149 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:57.145 7f4a94a5c700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4a900a2580 0x7f4a900a1cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:57.150 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:57.145 7f4a94a5c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4a90094c40 0x7f4a901a75e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:57.150 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:57.145 7f4a94a5c700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4a90095040 0x7f4a901a7c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:57.150 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:57.145 7f4a94a5c700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a90188180 con 0x7f4a90094c40 2021-02-10T11:35:57.150 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:57.145 7f4a94a5c700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a90187e70 con 0x7f4a90095040 2021-02-10T11:35:57.150 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:57.145 7f4a94a5c700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a901a8220 con 0x7f4a900a2580 2021-02-10T11:35:57.150 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:57.145 7f4a8e05d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4a90094c40 0x7f4a901a75e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:57.150 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:57.145 7f4a8e05d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4a90094c40 0x7f4a901a75e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:37374/0 (socket says 172.21.2.102:37374) 2021-02-10T11:35:57.150 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:57.145 7f4a8e05d700 1 -- 172.21.2.102:0/1226098622 learned_addr learned my addr 172.21.2.102:0/1226098622 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:35:57.150 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:57.146 7f4a8e05d700 1 -- 172.21.2.102:0/1226098622 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4a900a2580 msgr2=0x7f4a900a1cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:57.150 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:57.146 7f4a8e05d700 1 --2- 172.21.2.102:0/1226098622 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4a900a2580 0x7f4a900a1cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:57.150 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:57.146 7f4a8e05d700 1 -- 172.21.2.102:0/1226098622 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4a90095040 msgr2=0x7f4a901a7c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:57.150 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:57.146 7f4a8e05d700 1 --2- 172.21.2.102:0/1226098622 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4a90095040 0x7f4a901a7c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:57.151 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:57.146 7f4a8e05d700 1 -- 172.21.2.102:0/1226098622 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a90188700 con 0x7f4a90094c40 2021-02-10T11:35:57.151 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:57.146 7f4a8e05d700 1 --2- 172.21.2.102:0/1226098622 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4a90094c40 0x7f4a901a75e0 secure :-1 s=READY pgs=2751 cs=0 l=1 rev1=1 rx=0x7f4a80001c60 tx=0x7f4a8000a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:57.151 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:57.146 7f4a877fe700 1 -- 172.21.2.102:0/1226098622 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4a8000acc0 con 0x7f4a90094c40 2021-02-10T11:35:57.151 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:57.146 7f4a877fe700 1 -- 172.21.2.102:0/1226098622 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a80004350 con 0x7f4a90094c40 2021-02-10T11:35:57.151 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:57.146 7f4a877fe700 1 -- 172.21.2.102:0/1226098622 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4a80004870 con 0x7f4a90094c40 2021-02-10T11:35:57.151 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:57.146 7f4a94a5c700 1 -- 172.21.2.102:0/1226098622 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a90188700 con 0x7f4a90094c40 2021-02-10T11:35:57.151 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:57.146 7f4a94a5c700 1 -- 172.21.2.102:0/1226098622 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a90188c90 con 0x7f4a90094c40 2021-02-10T11:35:57.151 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:57.147 7f4a877fe700 1 -- 172.21.2.102:0/1226098622 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4a80004140 con 0x7f4a90094c40 2021-02-10T11:35:57.151 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:57.147 7f4a877fe700 1 --2- 172.21.2.102:0/1226098622 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4a740186c0 0x7f4a7401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:57.151 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:57.147 7f4a877fe700 1 -- 172.21.2.102:0/1226098622 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(150..150 src has 1..150) v4 ==== 5107+0+0 (secure 0 0 0) 0x7f4a800323a0 con 0x7f4a90094c40 2021-02-10T11:35:57.151 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:57.147 7f4a8e85e700 1 --2- 172.21.2.102:0/1226098622 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4a740186c0 0x7f4a7401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:57.152 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:57.148 7f4a8e85e700 1 --2- 172.21.2.102:0/1226098622 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4a740186c0 0x7f4a7401abc0 crc :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:57.178 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:57.177 7f4a94a5c700 1 -- 172.21.2.102:0/1226098622 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a90082980 con 0x7f4a90094c40 2021-02-10T11:35:57.181 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:57.180 7f4a877fe700 1 -- 172.21.2.102:0/1226098622 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f4a80000940 con 0x7f4a90094c40 2021-02-10T11:35:57.311 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:57.310 7f4a94a5c700 1 -- 172.21.2.102:0/1226098622 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_data"} v 0) v1 -- 0x7f4a90109a60 con 0x7f4a90094c40 2021-02-10T11:35:57.659 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:57.658 7f4a877fe700 1 -- 172.21.2.102:0/1226098622 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_data"}]=0 pool 'cephfs_data' created v151) v1 ==== 125+0+0 (secure 0 0 0) 0x7f4a80004140 con 0x7f4a90094c40 2021-02-10T11:35:57.677 INFO:teuthology.orchestra.run.gibba002.stderr:pool 'cephfs_data' created 2021-02-10T11:35:57.679 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:57.678 7f4a94a5c700 1 -- 172.21.2.102:0/1226098622 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4a740186c0 msgr2=0x7f4a7401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:57.679 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:57.678 7f4a94a5c700 1 --2- 172.21.2.102:0/1226098622 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4a740186c0 0x7f4a7401abc0 crc :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:57.679 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:57.678 7f4a94a5c700 1 -- 172.21.2.102:0/1226098622 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4a90094c40 msgr2=0x7f4a901a75e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:57.679 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:57.678 7f4a94a5c700 1 --2- 172.21.2.102:0/1226098622 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4a90094c40 0x7f4a901a75e0 secure :-1 s=READY pgs=2751 cs=0 l=1 rev1=1 rx=0x7f4a80001c60 tx=0x7f4a8000a8d0).stop 2021-02-10T11:35:57.679 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:57.678 7f4a94a5c700 1 -- 172.21.2.102:0/1226098622 shutdown_connections 2021-02-10T11:35:57.679 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:57.678 7f4a94a5c700 1 --2- 172.21.2.102:0/1226098622 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4a740186c0 0x7f4a7401abc0 unknown :-1 s=CLOSED pgs=1209 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:57.680 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:57.678 7f4a94a5c700 1 --2- 172.21.2.102:0/1226098622 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4a90095040 0x7f4a901a7c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:57.680 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:57.678 7f4a94a5c700 1 --2- 172.21.2.102:0/1226098622 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4a90094c40 0x7f4a901a75e0 unknown :-1 s=CLOSED pgs=2751 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:57.680 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:57.678 7f4a94a5c700 1 --2- 172.21.2.102:0/1226098622 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4a900a2580 0x7f4a900a1cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:57.680 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:57.678 7f4a94a5c700 1 -- 172.21.2.102:0/1226098622 >> 172.21.2.102:0/1226098622 conn(0x7f4a900a2b30 msgr2=0x7f4a9009ce60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:57.680 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:57.678 7f4a94a5c700 1 -- 172.21.2.102:0/1226098622 shutdown_connections 2021-02-10T11:35:57.680 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:57.678 7f4a94a5c700 1 -- 172.21.2.102:0/1226098622 wait complete. 2021-02-10T11:35:57.687 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2021-02-10T11:35:58.119 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:58.102 7f19e9cda700 1 Processor -- start 2021-02-10T11:35:58.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:58.102 7f19e9cda700 1 -- start start 2021-02-10T11:35:58.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:58.102 7f19e9cda700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f19e40d1870 0x7f19e41b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:58.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:58.102 7f19e9cda700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f19e40cff70 0x7f19e4109130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:58.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:58.102 7f19e9cda700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19e4109a60 con 0x7f19e41b0ae0 2021-02-10T11:35:58.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:58.102 7f19e9cda700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19e4109750 con 0x7f19e40d1870 2021-02-10T11:35:58.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:58.102 7f19e9cda700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19e4088070 con 0x7f19e40cff70 2021-02-10T11:35:58.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:58.102 7f19e37fe700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f19e40d1870 0x7f19e41b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:58.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:58.102 7f19e37fe700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f19e40d1870 0x7f19e41b04c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:37384/0 (socket says 172.21.2.102:37384) 2021-02-10T11:35:58.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:58.102 7f19e37fe700 1 -- 172.21.2.102:0/3214065825 learned_addr learned my addr 172.21.2.102:0/3214065825 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:35:58.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:58.103 7f19e37fe700 1 -- 172.21.2.102:0/3214065825 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f19e40cff70 msgr2=0x7f19e4109130 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:35:58.121 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:58.103 7f19e37fe700 1 --2- 172.21.2.102:0/3214065825 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f19e40cff70 0x7f19e4109130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:58.121 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:58.103 7f19e37fe700 1 -- 172.21.2.102:0/3214065825 >> v1:172.21.2.113:6789/0 conn(0x7f19e41b0ae0 legacy=0x7f19e4108940 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:35:58.121 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:58.103 7f19e37fe700 1 -- 172.21.2.102:0/3214065825 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19e40d1c70 con 0x7f19e40d1870 2021-02-10T11:35:58.121 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:58.103 7f19e37fe700 1 --2- 172.21.2.102:0/3214065825 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f19e40d1870 0x7f19e41b04c0 secure :-1 s=READY pgs=2752 cs=0 l=1 rev1=1 rx=0x7f19cc006700 tx=0x7f19cc009b80).ready entity=mon.0 client_cookie=92d0dc725130b2bd server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:58.121 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:58.103 7f19e27fc700 1 -- 172.21.2.102:0/3214065825 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f19cc00c0c0 con 0x7f19e40d1870 2021-02-10T11:35:58.121 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:58.103 7f19e27fc700 1 -- 172.21.2.102:0/3214065825 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19cc00c340 con 0x7f19e40d1870 2021-02-10T11:35:58.121 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:58.103 7f19e27fc700 1 -- 172.21.2.102:0/3214065825 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f19cc00c820 con 0x7f19e40d1870 2021-02-10T11:35:58.121 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:58.103 7f19e9cda700 1 -- 172.21.2.102:0/3214065825 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f19e40d1870 msgr2=0x7f19e41b04c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:58.121 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:58.103 7f19e9cda700 1 --2- 172.21.2.102:0/3214065825 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f19e40d1870 0x7f19e41b04c0 secure :-1 s=READY pgs=2752 cs=0 l=1 rev1=1 rx=0x7f19cc006700 tx=0x7f19cc009b80).stop 2021-02-10T11:35:58.121 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:58.103 7f19e9cda700 1 -- 172.21.2.102:0/3214065825 shutdown_connections 2021-02-10T11:35:58.121 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:58.103 7f19e9cda700 1 --2- 172.21.2.102:0/3214065825 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f19e40cff70 0x7f19e4109130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:58.122 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:58.103 7f19e9cda700 1 --2- 172.21.2.102:0/3214065825 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f19e40d1870 0x7f19e41b04c0 unknown :-1 s=CLOSED pgs=2752 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:58.122 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:58.103 7f19e9cda700 1 -- 172.21.2.102:0/3214065825 >> 172.21.2.102:0/3214065825 conn(0x7f19e40ce420 msgr2=0x7f19e40d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:58.122 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:58.104 7f19e9cda700 1 -- 172.21.2.102:0/3214065825 shutdown_connections 2021-02-10T11:35:58.122 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:58.104 7f19e9cda700 1 -- 172.21.2.102:0/3214065825 wait complete. 2021-02-10T11:35:58.122 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:58.118 7f19e9cda700 1 Processor -- start 2021-02-10T11:35:58.122 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:58.118 7f19e9cda700 1 -- start start 2021-02-10T11:35:58.122 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:58.118 7f19e9cda700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f19e41ac420 0x7f19e41a27f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:58.122 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:58.118 7f19e9cda700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f19e41a2e10 0x7f19e41ae9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:58.122 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:58.118 7f19e9cda700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f19e41af010 0x7f19e41b0a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:58.122 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:58.118 7f19e9cda700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19e41b1fb0 con 0x7f19e41ac420 2021-02-10T11:35:58.122 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:58.118 7f19e9cda700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19e41b1050 con 0x7f19e41af010 2021-02-10T11:35:58.123 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:58.118 7f19e9cda700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19e41b1270 con 0x7f19e41a2e10 2021-02-10T11:35:58.124 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:58.119 7f19e37fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f19e41ac420 0x7f19e41a27f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:58.124 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:58.119 7f19e37fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f19e41ac420 0x7f19e41a27f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:37390/0 (socket says 172.21.2.102:37390) 2021-02-10T11:35:58.124 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:58.119 7f19e37fe700 1 -- 172.21.2.102:0/4173659679 learned_addr learned my addr 172.21.2.102:0/4173659679 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:35:58.124 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:58.119 7f19e2ffd700 1 --2- 172.21.2.102:0/4173659679 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f19e41a2e10 0x7f19e41ae9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:58.124 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:58.119 7f19e3fff700 1 --2- 172.21.2.102:0/4173659679 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f19e41af010 0x7f19e41b0a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:58.124 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:58.119 7f19e37fe700 1 -- 172.21.2.102:0/4173659679 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f19e41a2e10 msgr2=0x7f19e41ae9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:58.125 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:58.119 7f19e37fe700 1 --2- 172.21.2.102:0/4173659679 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f19e41a2e10 0x7f19e41ae9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:58.125 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:58.119 7f19e37fe700 1 -- 172.21.2.102:0/4173659679 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f19e41af010 msgr2=0x7f19e41b0a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:58.125 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:58.119 7f19e37fe700 1 --2- 172.21.2.102:0/4173659679 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f19e41af010 0x7f19e41b0a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:58.125 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:58.119 7f19e37fe700 1 -- 172.21.2.102:0/4173659679 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19e41b1490 con 0x7f19e41ac420 2021-02-10T11:35:58.125 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:58.119 7f19e37fe700 1 --2- 172.21.2.102:0/4173659679 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f19e41ac420 0x7f19e41a27f0 secure :-1 s=READY pgs=2753 cs=0 l=1 rev1=1 rx=0x7f19cc00b5e0 tx=0x7f19cc00b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:58.125 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:58.119 7f19dbfff700 1 -- 172.21.2.102:0/4173659679 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f19cc00bd70 con 0x7f19e41ac420 2021-02-10T11:35:58.125 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:58.119 7f19dbfff700 1 -- 172.21.2.102:0/4173659679 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19cc002030 con 0x7f19e41ac420 2021-02-10T11:35:58.125 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:58.119 7f19dbfff700 1 -- 172.21.2.102:0/4173659679 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f19cc002550 con 0x7f19e41ac420 2021-02-10T11:35:58.125 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:58.119 7f19e9cda700 1 -- 172.21.2.102:0/4173659679 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19e41b1490 con 0x7f19e41ac420 2021-02-10T11:35:58.125 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:58.119 7f19e9cda700 1 -- 172.21.2.102:0/4173659679 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19e41b1490 con 0x7f19e41ac420 2021-02-10T11:35:58.125 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:58.120 7f19dbfff700 1 -- 172.21.2.102:0/4173659679 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f19cc002210 con 0x7f19e41ac420 2021-02-10T11:35:58.126 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:58.120 7f19dbfff700 1 --2- 172.21.2.102:0/4173659679 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f19c80186c0 0x7f19c801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:58.126 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:58.120 7f19dbfff700 1 -- 172.21.2.102:0/4173659679 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 5457+0+0 (secure 0 0 0) 0x7f19cc0324f0 con 0x7f19e41ac420 2021-02-10T11:35:58.126 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:58.120 7f19e2ffd700 1 --2- 172.21.2.102:0/4173659679 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f19c80186c0 0x7f19c801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:58.126 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:58.121 7f19e2ffd700 1 --2- 172.21.2.102:0/4173659679 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f19c80186c0 0x7f19c801abc0 crc :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:58.153 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:58.152 7f19e9cda700 1 -- 172.21.2.102:0/4173659679 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19e41b1050 con 0x7f19e41ac420 2021-02-10T11:35:58.156 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:58.155 7f19dbfff700 1 -- 172.21.2.102:0/4173659679 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f19cc030d10 con 0x7f19e41ac420 2021-02-10T11:35:58.286 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:58.285 7f19e9cda700 1 -- 172.21.2.102:0/4173659679 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"data": "cephfs_data", "prefix": "fs new", "force": true, "fs_name": "cephfs", "metadata": "cephfs_metadata"} v 0) v1 -- 0x7f19e41b1050 con 0x7f19e41ac420 2021-02-10T11:35:58.290 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:58.289 7f19dbfff700 1 -- 172.21.2.102:0/4173659679 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"data": "cephfs_data", "prefix": "fs new", "force": true, "fs_name": "cephfs", "metadata": "cephfs_metadata"}]=0 new fs with metadata pool 32 and data pool 33 v129) v1 ==== 189+0+0 (secure 0 0 0) 0x7f19cc01f510 con 0x7f19e41ac420 2021-02-10T11:35:58.293 INFO:teuthology.orchestra.run.gibba002.stderr:new fs with metadata pool 32 and data pool 33 2021-02-10T11:35:58.296 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:58.295 7f19e9cda700 1 -- 172.21.2.102:0/4173659679 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f19c80186c0 msgr2=0x7f19c801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:58.296 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:58.295 7f19e9cda700 1 --2- 172.21.2.102:0/4173659679 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f19c80186c0 0x7f19c801abc0 crc :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:58.296 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:58.295 7f19e9cda700 1 -- 172.21.2.102:0/4173659679 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f19e41ac420 msgr2=0x7f19e41a27f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:58.297 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:58.295 7f19e9cda700 1 --2- 172.21.2.102:0/4173659679 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f19e41ac420 0x7f19e41a27f0 secure :-1 s=READY pgs=2753 cs=0 l=1 rev1=1 rx=0x7f19cc00b5e0 tx=0x7f19cc00b980).stop 2021-02-10T11:35:58.297 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:58.295 7f19e9cda700 1 -- 172.21.2.102:0/4173659679 shutdown_connections 2021-02-10T11:35:58.297 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:58.295 7f19e9cda700 1 --2- 172.21.2.102:0/4173659679 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f19c80186c0 0x7f19c801abc0 unknown :-1 s=CLOSED pgs=1212 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:58.297 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:58.295 7f19e9cda700 1 --2- 172.21.2.102:0/4173659679 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f19e41af010 0x7f19e41b0a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:58.297 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:58.295 7f19e9cda700 1 --2- 172.21.2.102:0/4173659679 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f19e41a2e10 0x7f19e41ae9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:58.297 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:58.295 7f19e9cda700 1 --2- 172.21.2.102:0/4173659679 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f19e41ac420 0x7f19e41a27f0 unknown :-1 s=CLOSED pgs=2753 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:58.297 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:58.295 7f19e9cda700 1 -- 172.21.2.102:0/4173659679 >> 172.21.2.102:0/4173659679 conn(0x7f19e40841e0 msgr2=0x7f19e4082800 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:58.297 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:58.295 7f19e9cda700 1 -- 172.21.2.102:0/4173659679 shutdown_connections 2021-02-10T11:35:58.297 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:58.295 7f19e9cda700 1 -- 172.21.2.102:0/4173659679 wait complete. 2021-02-10T11:35:58.304 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:35:58.734 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:58.717 7fb5fab30700 1 Processor -- start 2021-02-10T11:35:58.734 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:58.717 7fb5fab30700 1 -- start start 2021-02-10T11:35:58.734 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:58.717 7fb5fab30700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fb5f41b0470 0x7fb5f41b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:58.734 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:58.717 7fb5fab30700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb5f40f20e0 0x7fb5f40f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:58.735 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:58.717 7fb5fab30700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5f40f0b50 con 0x7fb5f40ef610 2021-02-10T11:35:58.735 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:58.717 7fb5fab30700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5f40f0820 con 0x7fb5f41b0470 2021-02-10T11:35:58.735 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:58.717 7fb5fab30700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5f40f24e0 con 0x7fb5f40f20e0 2021-02-10T11:35:58.735 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:58.717 7fb5f894c700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fb5f41b0470 0x7fb5f41b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:58.735 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:58.718 7fb5f3fff700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fb5f40ef610 0x7fb5f40efa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:57854/0 (socket says 172.21.2.102:57854) 2021-02-10T11:35:58.735 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:58.718 7fb5f3fff700 1 -- 172.21.2.102:0/3678461587 learned_addr learned my addr 172.21.2.102:0/3678461587 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:35:58.735 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:58.718 7fb5f914d700 1 --2- 172.21.2.102:0/3678461587 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb5f40f20e0 0x7fb5f40f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:58.735 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:58.718 7fb5f894c700 1 -- 172.21.2.102:0/3678461587 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb5f40f20e0 msgr2=0x7fb5f40f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:58.735 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:58.718 7fb5f894c700 1 --2- 172.21.2.102:0/3678461587 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb5f40f20e0 0x7fb5f40f0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:58.735 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:58.718 7fb5f894c700 1 -- 172.21.2.102:0/3678461587 >> v1:172.21.2.102:6789/0 conn(0x7fb5f40ef610 legacy=0x7fb5f40efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:58.735 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:58.718 7fb5f894c700 1 -- 172.21.2.102:0/3678461587 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5f40e52e0 con 0x7fb5f41b0470 2021-02-10T11:35:58.735 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:58.718 7fb5f894c700 1 --2- 172.21.2.102:0/3678461587 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb5f41b0470 0x7fb5f41b0870 secure :-1 s=READY pgs=2755 cs=0 l=1 rev1=1 rx=0x7fb5dc006650 tx=0x7fb5dc009a80).ready entity=mon.0 client_cookie=9086a82daefe4571 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:58.736 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:58.718 7fb5f37fe700 1 -- 172.21.2.102:0/3678461587 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb5dc0009b0 con 0x7fb5f41b0470 2021-02-10T11:35:58.736 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:58.718 7fb5f37fe700 1 -- 172.21.2.102:0/3678461587 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5dc00c0c0 con 0x7fb5f41b0470 2021-02-10T11:35:58.736 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:58.718 7fb5f37fe700 1 -- 172.21.2.102:0/3678461587 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb5dc00c5a0 con 0x7fb5f41b0470 2021-02-10T11:35:58.736 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:58.718 7fb5fab30700 1 -- 172.21.2.102:0/3678461587 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb5f41b0470 msgr2=0x7fb5f41b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:58.736 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:58.718 7fb5fab30700 1 --2- 172.21.2.102:0/3678461587 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb5f41b0470 0x7fb5f41b0870 secure :-1 s=READY pgs=2755 cs=0 l=1 rev1=1 rx=0x7fb5dc006650 tx=0x7fb5dc009a80).stop 2021-02-10T11:35:58.736 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:58.719 7fb5fab30700 1 -- 172.21.2.102:0/3678461587 shutdown_connections 2021-02-10T11:35:58.736 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:58.719 7fb5fab30700 1 --2- 172.21.2.102:0/3678461587 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb5f40f20e0 0x7fb5f40f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:58.736 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:58.719 7fb5fab30700 1 --2- 172.21.2.102:0/3678461587 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb5f41b0470 0x7fb5f41b0870 unknown :-1 s=CLOSED pgs=2755 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:58.736 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:58.719 7fb5fab30700 1 -- 172.21.2.102:0/3678461587 >> 172.21.2.102:0/3678461587 conn(0x7fb5f40ce3e0 msgr2=0x7fb5f40e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:58.736 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:58.719 7fb5fab30700 1 -- 172.21.2.102:0/3678461587 shutdown_connections 2021-02-10T11:35:58.736 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:58.719 7fb5fab30700 1 -- 172.21.2.102:0/3678461587 wait complete. 2021-02-10T11:35:58.738 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:58.734 7fb5fab30700 1 Processor -- start 2021-02-10T11:35:58.738 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:58.734 7fb5fab30700 1 -- start start 2021-02-10T11:35:58.738 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:58.734 7fb5fab30700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb5f408e110 0x7fb5f40b1ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:58.738 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:58.734 7fb5fab30700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb5f40b24f0 0x7fb5f408f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:58.738 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:58.734 7fb5fab30700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb5f408fe00 0x7fb5f408c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:58.738 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:58.734 7fb5fab30700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5f408db90 con 0x7fb5f40b24f0 2021-02-10T11:35:58.739 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:58.734 7fb5fab30700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5f41a83f0 con 0x7fb5f408e110 2021-02-10T11:35:58.739 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:58.734 7fb5fab30700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5f41a85d0 con 0x7fb5f408fe00 2021-02-10T11:35:58.739 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:58.734 7fb5f3fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb5f40b24f0 0x7fb5f408f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:58.739 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:58.734 7fb5f3fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb5f40b24f0 0x7fb5f408f7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:37406/0 (socket says 172.21.2.102:37406) 2021-02-10T11:35:58.739 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:58.734 7fb5f3fff700 1 -- 172.21.2.102:0/4087527043 learned_addr learned my addr 172.21.2.102:0/4087527043 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:35:58.739 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:58.734 7fb5f914d700 1 --2- 172.21.2.102:0/4087527043 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb5f408fe00 0x7fb5f408c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:58.739 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:58.735 7fb5f894c700 1 --2- 172.21.2.102:0/4087527043 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb5f408e110 0x7fb5f40b1ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:58.739 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:58.735 7fb5f3fff700 1 -- 172.21.2.102:0/4087527043 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb5f408fe00 msgr2=0x7fb5f408c690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:58.739 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:58.735 7fb5f3fff700 1 --2- 172.21.2.102:0/4087527043 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb5f408fe00 0x7fb5f408c690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:58.739 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:58.735 7fb5f3fff700 1 -- 172.21.2.102:0/4087527043 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb5f408e110 msgr2=0x7fb5f40b1ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:58.739 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:58.735 7fb5f3fff700 1 --2- 172.21.2.102:0/4087527043 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb5f408e110 0x7fb5f40b1ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:58.739 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:58.735 7fb5f3fff700 1 -- 172.21.2.102:0/4087527043 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5f41a8890 con 0x7fb5f40b24f0 2021-02-10T11:35:58.739 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:58.735 7fb5f3fff700 1 --2- 172.21.2.102:0/4087527043 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb5f40b24f0 0x7fb5f408f7e0 secure :-1 s=READY pgs=2756 cs=0 l=1 rev1=1 rx=0x7fb5e400a920 tx=0x7fb5e400acc0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:58.740 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:58.735 7fb5f17fa700 1 -- 172.21.2.102:0/4087527043 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb5e40070d0 con 0x7fb5f40b24f0 2021-02-10T11:35:58.740 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:58.735 7fb5f17fa700 1 -- 172.21.2.102:0/4087527043 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5e40044c0 con 0x7fb5f40b24f0 2021-02-10T11:35:58.740 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:58.735 7fb5fab30700 1 -- 172.21.2.102:0/4087527043 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5f41a8890 con 0x7fb5f40b24f0 2021-02-10T11:35:58.740 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:58.735 7fb5f17fa700 1 -- 172.21.2.102:0/4087527043 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb5e4004920 con 0x7fb5f40b24f0 2021-02-10T11:35:58.740 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:58.735 7fb5fab30700 1 -- 172.21.2.102:0/4087527043 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5f40904f0 con 0x7fb5f40b24f0 2021-02-10T11:35:58.740 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:58.736 7fb5f17fa700 1 -- 172.21.2.102:0/4087527043 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fb5e40046a0 con 0x7fb5f40b24f0 2021-02-10T11:35:58.740 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:58.736 7fb5f17fa700 1 --2- 172.21.2.102:0/4087527043 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb5d80186c0 0x7fb5d801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:58.740 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:58.736 7fb5f17fa700 1 -- 172.21.2.102:0/4087527043 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(152..152 src has 1..152) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fb5e40327e0 con 0x7fb5f40b24f0 2021-02-10T11:35:58.740 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:58.736 7fb5f894c700 1 --2- 172.21.2.102:0/4087527043 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb5d80186c0 0x7fb5d801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:58.740 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:58.737 7fb5f894c700 1 --2- 172.21.2.102:0/4087527043 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb5d80186c0 0x7fb5d801abc0 crc :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:58.767 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:58.766 7fb5fab30700 1 -- 172.21.2.102:0/4087527043 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5f4105ab0 con 0x7fb5f40b24f0 2021-02-10T11:35:58.770 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:58.769 7fb5f17fa700 1 -- 172.21.2.102:0/4087527043 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fb5e4025f30 con 0x7fb5f40b24f0 2021-02-10T11:35:58.895 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:58.894 7fb5fab30700 1 -- 172.21.2.102:0/4087527043 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb5f40f0b50 con 0x7fb5f40b24f0 2021-02-10T11:35:58.896 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:58.895 7fb5f17fa700 1 -- 172.21.2.102:0/4087527043 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v152) v1 ==== 74+0+14020 (secure 0 0 0) 0x7fb5e4025f30 con 0x7fb5f40b24f0 2021-02-10T11:35:58.896 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:35:58.897 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":152,"fsid":"68db30ea-dbce-4aef-a6df-f01c6fea0513","created":"2021-02-10 11:03:06.331557","modified":"2021-02-10 11:35:58.285817","last_up_change":"2021-02-10 11:03:32.690562","last_in_change":"2021-02-10 11:03:32.690562","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":33,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:03:37.322519","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":32,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:35:56.251556","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"152","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":33,"pool_name":"cephfs_data","create_time":"2021-02-10 11:35:57.328218","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"152","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"be691b4d-a039-408a-af80-9d53f02b6f1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":150,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6818","nonce":36088},{"type":"v1","addr":"172.21.2.102:6820","nonce":36088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6822","nonce":36088},{"type":"v1","addr":"172.21.2.102:6824","nonce":36088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6830","nonce":36088},{"type":"v1","addr":"172.21.2.102:6832","nonce":36088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6826","nonce":36088},{"type":"v1","addr":"172.21.2.102:6828","nonce":36088}]},"public_addr":"172.21.2.102:6820/36088","cluster_addr":"172.21.2.102:6824/36088","heartbeat_back_addr":"172.21.2.102:6832/36088","heartbeat_front_addr":"172.21.2.102:6828/36088","state":["exists","up"]},{"osd":1,"uuid":"64e3ccc8-9c69-4d7c-865e-af8e83822635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6819","nonce":36087},{"type":"v1","addr":"172.21.2.102:6821","nonce":36087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6823","nonce":36087},{"type":"v1","addr":"172.21.2.102:6825","nonce":36087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6831","nonce":36087},{"type":"v1","addr":"172.21.2.102:6833","nonce":36087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6827","nonce":36087},{"type":"v1","addr":"172.21.2.102:6829","nonce":36087}]},"public_addr":"172.21.2.102:6821/36087","cluster_addr":"172.21.2.102:6825/36087","heartbeat_back_addr":"172.21.2.102:6833/36087","heartbeat_front_addr":"172.21.2.102:6829/36087","state":["exists","up"]},{"osd":2,"uuid":"153700af-747f-481f-bdf4-d6072d3a842f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6810","nonce":36089},{"type":"v1","addr":"172.21.2.102:6811","nonce":36089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6812","nonce":36089},{"type":"v1","addr":"172.21.2.102:6813","nonce":36089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6816","nonce":36089},{"type":"v1","addr":"172.21.2.102:6817","nonce":36089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6814","nonce":36089},{"type":"v1","addr":"172.21.2.102:6815","nonce":36089}]},"public_addr":"172.21.2.102:6811/36089","cluster_addr":"172.21.2.102:6813/36089","heartbeat_back_addr":"172.21.2.102:6817/36089","heartbeat_front_addr":"172.21.2.102:6815/36089","state":["exists","up"]},{"osd":3,"uuid":"cbc678f4-83a9-45e6-8830-2f7c232b1ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6802","nonce":36090},{"type":"v1","addr":"172.21.2.102:6803","nonce":36090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6804","nonce":36090},{"type":"v1","addr":"172.21.2.102:6805","nonce":36090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6808","nonce":36090},{"type":"v1","addr":"172.21.2.102:6809","nonce":36090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6806","nonce":36090},{"type":"v1","addr":"172.21.2.102:6807","nonce":36090}]},"public_addr":"172.21.2.102:6803/36090","cluster_addr":"172.21.2.102:6805/36090","heartbeat_back_addr":"172.21.2.102:6809/36090","heartbeat_front_addr":"172.21.2.102:6807/36090","state":["exists","up"]},{"osd":4,"uuid":"c01e4008-b258-4f10-afd2-9e7f37214351","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6808","nonce":36368},{"type":"v1","addr":"172.21.2.113:6811","nonce":36368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6814","nonce":36368},{"type":"v1","addr":"172.21.2.113:6818","nonce":36368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6826","nonce":36368},{"type":"v1","addr":"172.21.2.113:6828","nonce":36368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6820","nonce":36368},{"type":"v1","addr":"172.21.2.113:6824","nonce":36368}]},"public_addr":"172.21.2.113:6811/36368","cluster_addr":"172.21.2.113:6818/36368","heartbeat_back_addr":"172.21.2.113:6828/36368","heartbeat_front_addr":"172.21.2.113:6824/36368","state":["exists","up"]},{"osd":5,"uuid":"551f6007-daf8-4a98-8ee0-c2b16bcccb5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6802","nonce":36366},{"type":"v1","addr":"172.21.2.113:6805","nonce":36366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6807","nonce":36366},{"type":"v1","addr":"172.21.2.113:6810","nonce":36366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6819","nonce":36366},{"type":"v1","addr":"172.21.2.113:6822","nonce":36366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6812","nonce":36366},{"type":"v1","addr":"172.21.2.113:6815","nonce":36366}]},"public_addr":"172.21.2.113:6805/36366","cluster_addr":"172.21.2.113:6810/36366","heartbeat_back_addr":"172.21.2.113:6822/36366","heartbeat_front_addr":"172.21.2.113:6815/36366","state":["exists","up"]},{"osd":6,"uuid":"bf67c040-0e50-48be-b102-c3943bc9ebef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":150,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6800","nonce":36367},{"type":"v1","addr":"172.21.2.113:6801","nonce":36367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6803","nonce":36367},{"type":"v1","addr":"172.21.2.113:6804","nonce":36367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6813","nonce":36367},{"type":"v1","addr":"172.21.2.113:6816","nonce":36367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6806","nonce":36367},{"type":"v1","addr":"172.21.2.113:6809","nonce":36367}]},"public_addr":"172.21.2.113:6801/36367","cluster_addr":"172.21.2.113:6804/36367","heartbeat_back_addr":"172.21.2.113:6816/36367","heartbeat_front_addr":"172.21.2.113:6809/36367","state":["exists","up"]},{"osd":7,"uuid":"f5461f80-018d-428f-be89-2eb2bcbe7bb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6817","nonce":36365},{"type":"v1","addr":"172.21.2.113:6821","nonce":36365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6823","nonce":36365},{"type":"v1","addr":"172.21.2.113:6825","nonce":36365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6830","nonce":36365},{"type":"v1","addr":"172.21.2.113:6831","nonce":36365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6827","nonce":36365},{"type":"v1","addr":"172.21.2.113:6829","nonce":36365}]},"public_addr":"172.21.2.113:6821/36365","cluster_addr":"172.21.2.113:6825/36365","heartbeat_back_addr":"172.21.2.113:6831/36365","heartbeat_front_addr":"172.21.2.113:6829/36365","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:35:58.897 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:58.896 7fb5fab30700 1 -- 172.21.2.102:0/4087527043 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb5d80186c0 msgr2=0x7fb5d801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:58.897 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:58.896 7fb5fab30700 1 --2- 172.21.2.102:0/4087527043 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb5d80186c0 0x7fb5d801abc0 crc :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:58.898 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:58.897 7fb5fab30700 1 -- 172.21.2.102:0/4087527043 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb5f40b24f0 msgr2=0x7fb5f408f7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:58.898 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:58.897 7fb5fab30700 1 --2- 172.21.2.102:0/4087527043 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb5f40b24f0 0x7fb5f408f7e0 secure :-1 s=READY pgs=2756 cs=0 l=1 rev1=1 rx=0x7fb5e400a920 tx=0x7fb5e400acc0).stop 2021-02-10T11:35:58.898 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:58.897 7fb5fab30700 1 -- 172.21.2.102:0/4087527043 shutdown_connections 2021-02-10T11:35:58.898 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:58.897 7fb5fab30700 1 --2- 172.21.2.102:0/4087527043 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb5f408fe00 0x7fb5f408c690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:58.898 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:58.897 7fb5fab30700 1 --2- 172.21.2.102:0/4087527043 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb5f40b24f0 0x7fb5f408f7e0 unknown :-1 s=CLOSED pgs=2756 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:58.898 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:58.897 7fb5fab30700 1 --2- 172.21.2.102:0/4087527043 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb5d80186c0 0x7fb5d801abc0 unknown :-1 s=CLOSED pgs=1213 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:58.899 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:58.897 7fb5fab30700 1 --2- 172.21.2.102:0/4087527043 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb5f408e110 0x7fb5f40b1ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:58.899 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:58.897 7fb5fab30700 1 -- 172.21.2.102:0/4087527043 >> 172.21.2.102:0/4087527043 conn(0x7fb5f419f7c0 msgr2=0x7fb5f41b6b60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:58.899 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:58.897 7fb5fab30700 1 -- 172.21.2.102:0/4087527043 shutdown_connections 2021-02-10T11:35:58.899 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:58.897 7fb5fab30700 1 -- 172.21.2.102:0/4087527043 wait complete. 2021-02-10T11:35:58.907 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:35:59.336 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:59.318 7f4815d01700 1 Processor -- start 2021-02-10T11:35:59.336 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:59.318 7f4815d01700 1 -- start start 2021-02-10T11:35:59.336 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:59.319 7f4815d01700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f48100d0340 0x7f48100ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:59.336 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:59.319 7f4815d01700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f48100d1c70 0x7f48100efc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:59.336 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:59.319 7f4815d01700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48100f0b10 con 0x7f48100d1870 2021-02-10T11:35:59.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:59.319 7f4815d01700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48100e5ed0 con 0x7f48100d1c70 2021-02-10T11:35:59.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:59.319 7f4815d01700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48101b0be0 con 0x7f48100d0340 2021-02-10T11:35:59.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:59.319 7f4806ffd700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f48100d0340 0x7f48100ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:59.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:59.319 7f480f7fe700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f48100d1870 0x7f48101b03f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:57866/0 (socket says 172.21.2.102:57866) 2021-02-10T11:35:59.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:59.319 7f480f7fe700 1 -- 172.21.2.102:0/3846113226 learned_addr learned my addr 172.21.2.102:0/3846113226 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:35:59.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:59.319 7f480ffff700 1 --2- 172.21.2.102:0/3846113226 >> v2:172.21.2.113:3300/0 conn(0x7f48100d1c70 0x7f48100efc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:59.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:59.320 7f480ffff700 1 -- 172.21.2.102:0/3846113226 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f48100d0340 msgr2=0x7f48100ef640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:59.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:59.320 7f480ffff700 1 --2- 172.21.2.102:0/3846113226 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f48100d0340 0x7f48100ef640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:59.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:59.320 7f480ffff700 1 -- 172.21.2.102:0/3846113226 >> v1:172.21.2.102:6789/0 conn(0x7f48100d1870 legacy=0x7f48101b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:59.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:59.320 7f480ffff700 1 -- 172.21.2.102:0/3846113226 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48100f20a0 con 0x7f48100d1c70 2021-02-10T11:35:59.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:59.320 7f480ffff700 1 --2- 172.21.2.102:0/3846113226 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f48100d1c70 0x7f48100efc60 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 rx=0x7f4808006e50 tx=0x7f4808009e70).ready entity=mon.1 client_cookie=45990fab71449810 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:59.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:59.320 7f480effd700 1 -- 172.21.2.102:0/3846113226 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4808006a20 con 0x7f48100d1c70 2021-02-10T11:35:59.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:59.320 7f480effd700 1 -- 172.21.2.102:0/3846113226 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4808006790 con 0x7f48100d1c70 2021-02-10T11:35:59.338 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:59.320 7f480effd700 1 -- 172.21.2.102:0/3846113226 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f480800b700 con 0x7f48100d1c70 2021-02-10T11:35:59.338 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:59.320 7f4815d01700 1 -- 172.21.2.102:0/3846113226 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f48100d1c70 msgr2=0x7f48100efc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:59.338 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:59.320 7f4815d01700 1 --2- 172.21.2.102:0/3846113226 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f48100d1c70 0x7f48100efc60 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 rx=0x7f4808006e50 tx=0x7f4808009e70).stop 2021-02-10T11:35:59.338 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:59.321 7f4815d01700 1 -- 172.21.2.102:0/3846113226 shutdown_connections 2021-02-10T11:35:59.338 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:59.321 7f4815d01700 1 --2- 172.21.2.102:0/3846113226 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f48100d1c70 0x7f48100efc60 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:59.338 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:59.321 7f4815d01700 1 --2- 172.21.2.102:0/3846113226 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f48100d0340 0x7f48100ef640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:59.338 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:59.321 7f4815d01700 1 -- 172.21.2.102:0/3846113226 >> 172.21.2.102:0/3846113226 conn(0x7f48100ce450 msgr2=0x7f48100d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:59.338 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:59.321 7f4815d01700 1 -- 172.21.2.102:0/3846113226 shutdown_connections 2021-02-10T11:35:59.338 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:59.321 7f4815d01700 1 -- 172.21.2.102:0/3846113226 wait complete. 2021-02-10T11:35:59.340 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:59.336 7f4815d01700 1 Processor -- start 2021-02-10T11:35:59.340 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:59.336 7f4815d01700 1 -- start start 2021-02-10T11:35:59.340 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:59.336 7f4815d01700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f481018d870 0x7f48100b19b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:59.340 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:59.336 7f4815d01700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f48100b1fd0 0x7f4810099ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:59.340 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:59.336 7f4815d01700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f48100b23d0 0x7f481009a510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:59.340 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:59.336 7f4815d01700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f481018dc70 con 0x7f48100b1fd0 2021-02-10T11:35:59.340 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:59.336 7f4815d01700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4810199e60 con 0x7f481018d870 2021-02-10T11:35:59.340 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:59.336 7f4815d01700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f481009ab30 con 0x7f48100b23d0 2021-02-10T11:35:59.340 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:59.336 7f480ffff700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f48100b23d0 0x7f481009a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:59.340 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:59.336 7f480ffff700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f48100b23d0 0x7f481009a510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.113:3301/0 says I am v2:172.21.2.102:42954/0 (socket says 172.21.2.102:42954) 2021-02-10T11:35:59.340 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:59.336 7f480ffff700 1 -- 172.21.2.102:0/2760517368 learned_addr learned my addr 172.21.2.102:0/2760517368 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:35:59.341 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:59.336 7f4806ffd700 1 --2- 172.21.2.102:0/2760517368 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f48100b1fd0 0x7f4810099ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:59.341 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:59.336 7f480f7fe700 1 --2- 172.21.2.102:0/2760517368 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f481018d870 0x7f48100b19b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:59.341 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:59.337 7f480ffff700 1 -- 172.21.2.102:0/2760517368 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f481018d870 msgr2=0x7f48100b19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:59.341 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:59.337 7f480ffff700 1 --2- 172.21.2.102:0/2760517368 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f481018d870 0x7f48100b19b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:59.341 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:59.337 7f480ffff700 1 -- 172.21.2.102:0/2760517368 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f48100b1fd0 msgr2=0x7f4810099ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:59.341 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:59.337 7f480ffff700 1 --2- 172.21.2.102:0/2760517368 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f48100b1fd0 0x7f4810099ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:59.341 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:59.337 7f480ffff700 1 -- 172.21.2.102:0/2760517368 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f481009ad50 con 0x7f48100b23d0 2021-02-10T11:35:59.341 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:59.337 7f480ffff700 1 --2- 172.21.2.102:0/2760517368 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f48100b23d0 0x7f481009a510 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 rx=0x7f480800b940 tx=0x7f480800bce0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:59.341 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:59.337 7f480cff9700 1 -- 172.21.2.102:0/2760517368 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f48080093d0 con 0x7f48100b23d0 2021-02-10T11:35:59.341 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:59.337 7f480cff9700 1 -- 172.21.2.102:0/2760517368 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4808001070 con 0x7f48100b23d0 2021-02-10T11:35:59.341 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:59.337 7f4815d01700 1 -- 172.21.2.102:0/2760517368 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f481009ad50 con 0x7f48100b23d0 2021-02-10T11:35:59.341 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:59.337 7f4815d01700 1 -- 172.21.2.102:0/2760517368 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f481009ad50 con 0x7f48100b23d0 2021-02-10T11:35:59.341 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:59.337 7f480cff9700 1 -- 172.21.2.102:0/2760517368 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f48080014d0 con 0x7f48100b23d0 2021-02-10T11:35:59.342 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:59.337 7f480cff9700 1 -- 172.21.2.102:0/2760517368 <== mon.2 v2:172.21.2.113:3301/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f48080056b0 con 0x7f48100b23d0 2021-02-10T11:35:59.342 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:59.338 7f480cff9700 1 --2- 172.21.2.102:0/2760517368 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f47f4018730 0x7f47f401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:59.342 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:59.338 7f480cff9700 1 -- 172.21.2.102:0/2760517368 <== mon.2 v2:172.21.2.113:3301/0 5 ==== osd_map(153..153 src has 1..153) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f48080329f0 con 0x7f48100b23d0 2021-02-10T11:35:59.342 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:59.338 7f480f7fe700 1 --2- 172.21.2.102:0/2760517368 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f47f4018730 0x7f47f401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:59.342 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:59.338 7f480f7fe700 1 --2- 172.21.2.102:0/2760517368 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f47f4018730 0x7f47f401ac10 crc :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:59.369 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:59.368 7f4815d01700 1 -- 172.21.2.102:0/2760517368 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48100999c0 con 0x7f48100b23d0 2021-02-10T11:35:59.372 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:59.371 7f480cff9700 1 -- 172.21.2.102:0/2760517368 <== mon.2 v2:172.21.2.113:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f480800be20 con 0x7f48100b23d0 2021-02-10T11:35:59.498 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:59.497 7f4815d01700 1 -- 172.21.2.102:0/2760517368 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f48100999c0 con 0x7f48100b23d0 2021-02-10T11:35:59.499 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:59.498 7f480cff9700 1 -- 172.21.2.102:0/2760517368 <== mon.2 v2:172.21.2.113:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v153) v1 ==== 74+0+14007 (secure 0 0 0) 0x7f48080056b0 con 0x7f48100b23d0 2021-02-10T11:35:59.501 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:35:59.501 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":153,"fsid":"68db30ea-dbce-4aef-a6df-f01c6fea0513","created":"2021-02-10 11:03:06.331557","modified":"2021-02-10 11:35:59.288212","last_up_change":"2021-02-10 11:03:32.690562","last_in_change":"2021-02-10 11:03:32.690562","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":33,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:03:37.322519","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":32,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:35:56.251556","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"152","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":33,"pool_name":"cephfs_data","create_time":"2021-02-10 11:35:57.328218","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"153","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"be691b4d-a039-408a-af80-9d53f02b6f1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":150,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6818","nonce":36088},{"type":"v1","addr":"172.21.2.102:6820","nonce":36088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6822","nonce":36088},{"type":"v1","addr":"172.21.2.102:6824","nonce":36088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6830","nonce":36088},{"type":"v1","addr":"172.21.2.102:6832","nonce":36088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6826","nonce":36088},{"type":"v1","addr":"172.21.2.102:6828","nonce":36088}]},"public_addr":"172.21.2.102:6820/36088","cluster_addr":"172.21.2.102:6824/36088","heartbeat_back_addr":"172.21.2.102:6832/36088","heartbeat_front_addr":"172.21.2.102:6828/36088","state":["exists","up"]},{"osd":1,"uuid":"64e3ccc8-9c69-4d7c-865e-af8e83822635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6819","nonce":36087},{"type":"v1","addr":"172.21.2.102:6821","nonce":36087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6823","nonce":36087},{"type":"v1","addr":"172.21.2.102:6825","nonce":36087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6831","nonce":36087},{"type":"v1","addr":"172.21.2.102:6833","nonce":36087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6827","nonce":36087},{"type":"v1","addr":"172.21.2.102:6829","nonce":36087}]},"public_addr":"172.21.2.102:6821/36087","cluster_addr":"172.21.2.102:6825/36087","heartbeat_back_addr":"172.21.2.102:6833/36087","heartbeat_front_addr":"172.21.2.102:6829/36087","state":["exists","up"]},{"osd":2,"uuid":"153700af-747f-481f-bdf4-d6072d3a842f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6810","nonce":36089},{"type":"v1","addr":"172.21.2.102:6811","nonce":36089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6812","nonce":36089},{"type":"v1","addr":"172.21.2.102:6813","nonce":36089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6816","nonce":36089},{"type":"v1","addr":"172.21.2.102:6817","nonce":36089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6814","nonce":36089},{"type":"v1","addr":"172.21.2.102:6815","nonce":36089}]},"public_addr":"172.21.2.102:6811/36089","cluster_addr":"172.21.2.102:6813/36089","heartbeat_back_addr":"172.21.2.102:6817/36089","heartbeat_front_addr":"172.21.2.102:6815/36089","state":["exists","up"]},{"osd":3,"uuid":"cbc678f4-83a9-45e6-8830-2f7c232b1ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6802","nonce":36090},{"type":"v1","addr":"172.21.2.102:6803","nonce":36090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6804","nonce":36090},{"type":"v1","addr":"172.21.2.102:6805","nonce":36090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6808","nonce":36090},{"type":"v1","addr":"172.21.2.102:6809","nonce":36090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6806","nonce":36090},{"type":"v1","addr":"172.21.2.102:6807","nonce":36090}]},"public_addr":"172.21.2.102:6803/36090","cluster_addr":"172.21.2.102:6805/36090","heartbeat_back_addr":"172.21.2.102:6809/36090","heartbeat_front_addr":"172.21.2.102:6807/36090","state":["exists","up"]},{"osd":4,"uuid":"c01e4008-b258-4f10-afd2-9e7f37214351","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6808","nonce":36368},{"type":"v1","addr":"172.21.2.113:6811","nonce":36368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6814","nonce":36368},{"type":"v1","addr":"172.21.2.113:6818","nonce":36368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6826","nonce":36368},{"type":"v1","addr":"172.21.2.113:6828","nonce":36368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6820","nonce":36368},{"type":"v1","addr":"172.21.2.113:6824","nonce":36368}]},"public_addr":"172.21.2.113:6811/36368","cluster_addr":"172.21.2.113:6818/36368","heartbeat_back_addr":"172.21.2.113:6828/36368","heartbeat_front_addr":"172.21.2.113:6824/36368","state":["exists","up"]},{"osd":5,"uuid":"551f6007-daf8-4a98-8ee0-c2b16bcccb5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6802","nonce":36366},{"type":"v1","addr":"172.21.2.113:6805","nonce":36366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6807","nonce":36366},{"type":"v1","addr":"172.21.2.113:6810","nonce":36366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6819","nonce":36366},{"type":"v1","addr":"172.21.2.113:6822","nonce":36366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6812","nonce":36366},{"type":"v1","addr":"172.21.2.113:6815","nonce":36366}]},"public_addr":"172.21.2.113:6805/36366","cluster_addr":"172.21.2.113:6810/36366","heartbeat_back_addr":"172.21.2.113:6822/36366","heartbeat_front_addr":"172.21.2.113:6815/36366","state":["exists","up"]},{"osd":6,"uuid":"bf67c040-0e50-48be-b102-c3943bc9ebef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":150,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6800","nonce":36367},{"type":"v1","addr":"172.21.2.113:6801","nonce":36367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6803","nonce":36367},{"type":"v1","addr":"172.21.2.113:6804","nonce":36367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6813","nonce":36367},{"type":"v1","addr":"172.21.2.113:6816","nonce":36367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6806","nonce":36367},{"type":"v1","addr":"172.21.2.113:6809","nonce":36367}]},"public_addr":"172.21.2.113:6801/36367","cluster_addr":"172.21.2.113:6804/36367","heartbeat_back_addr":"172.21.2.113:6816/36367","heartbeat_front_addr":"172.21.2.113:6809/36367","state":["exists","up"]},{"osd":7,"uuid":"f5461f80-018d-428f-be89-2eb2bcbe7bb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6817","nonce":36365},{"type":"v1","addr":"172.21.2.113:6821","nonce":36365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6823","nonce":36365},{"type":"v1","addr":"172.21.2.113:6825","nonce":36365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6830","nonce":36365},{"type":"v1","addr":"172.21.2.113:6831","nonce":36365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6827","nonce":36365},{"type":"v1","addr":"172.21.2.113:6829","nonce":36365}]},"public_addr":"172.21.2.113:6821/36365","cluster_addr":"172.21.2.113:6825/36365","heartbeat_back_addr":"172.21.2.113:6831/36365","heartbeat_front_addr":"172.21.2.113:6829/36365","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:35:59.503 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:59.502 7f4815d01700 1 -- 172.21.2.102:0/2760517368 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f47f4018730 msgr2=0x7f47f401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:59.503 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:59.502 7f4815d01700 1 --2- 172.21.2.102:0/2760517368 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f47f4018730 0x7f47f401ac10 crc :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:59.503 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:59.502 7f4815d01700 1 -- 172.21.2.102:0/2760517368 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f48100b23d0 msgr2=0x7f481009a510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:59.503 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:59.502 7f4815d01700 1 --2- 172.21.2.102:0/2760517368 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f48100b23d0 0x7f481009a510 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 rx=0x7f480800b940 tx=0x7f480800bce0).stop 2021-02-10T11:35:59.503 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:59.502 7f4815d01700 1 -- 172.21.2.102:0/2760517368 shutdown_connections 2021-02-10T11:35:59.504 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:59.502 7f4815d01700 1 --2- 172.21.2.102:0/2760517368 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f48100b23d0 0x7f481009a510 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:59.504 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:59.502 7f4815d01700 1 --2- 172.21.2.102:0/2760517368 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f48100b1fd0 0x7f4810099ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:59.504 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:59.502 7f4815d01700 1 --2- 172.21.2.102:0/2760517368 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f47f4018730 0x7f47f401ac10 unknown :-1 s=CLOSED pgs=1214 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:59.504 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:59.502 7f4815d01700 1 --2- 172.21.2.102:0/2760517368 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f481018d870 0x7f48100b19b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:59.504 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:59.502 7f4815d01700 1 -- 172.21.2.102:0/2760517368 >> 172.21.2.102:0/2760517368 conn(0x7f48100f0550 msgr2=0x7f4810191730 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:59.504 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:59.502 7f4815d01700 1 -- 172.21.2.102:0/2760517368 shutdown_connections 2021-02-10T11:35:59.504 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:59.502 7f4815d01700 1 -- 172.21.2.102:0/2760517368 wait complete. 2021-02-10T11:35:59.512 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2021-02-10T11:35:59.946 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:59.926 7f5d76778700 1 Processor -- start 2021-02-10T11:35:59.946 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:59.926 7f5d76778700 1 -- start start 2021-02-10T11:35:59.946 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:59.926 7f5d76778700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f5d700d1870 0x7f5d701b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:59.946 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:59.927 7f5d76778700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d700fe1c0 con 0x7f5d701b0ae0 2021-02-10T11:35:59.946 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:59.927 7f5d76778700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d70109a60 con 0x7f5d700cff70 2021-02-10T11:35:59.946 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:59.927 7f5d76778700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d70088070 con 0x7f5d700d1870 2021-02-10T11:35:59.946 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:59.927 7f5d6f7fe700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f5d701b0ae0 0x7f5d70108940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:57884/0 (socket says 172.21.2.102:57884) 2021-02-10T11:35:59.946 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:59.927 7f5d6f7fe700 1 -- 172.21.2.102:0/1184084311 learned_addr learned my addr 172.21.2.102:0/1184084311 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:35:59.947 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:59.927 7f5d6ffff700 1 --2- 172.21.2.102:0/1184084311 >> v2:172.21.2.113:3300/0 conn(0x7f5d700d1870 0x7f5d701b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:59.947 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:59.927 7f5d6effd700 1 -- 172.21.2.102:0/1184084311 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1175533191 0 0) 0x7f5d640010b0 con 0x7f5d701b0ae0 2021-02-10T11:35:59.947 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:59.927 7f5d6effd700 1 -- 172.21.2.102:0/1184084311 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d58002fe0 con 0x7f5d701b0ae0 2021-02-10T11:35:59.947 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:59.927 7f5d6effd700 1 -- 172.21.2.102:0/1184084311 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4189227382 0 0) 0x7f5d64001010 con 0x7f5d701b0ae0 2021-02-10T11:35:59.947 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:59.927 7f5d6effd700 1 -- 172.21.2.102:0/1184084311 >> v2:172.21.2.113:3300/0 conn(0x7f5d700d1870 msgr2=0x7f5d701b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:59.947 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:59.927 7f5d6effd700 1 --2- 172.21.2.102:0/1184084311 >> v2:172.21.2.113:3300/0 conn(0x7f5d700d1870 0x7f5d701b04c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:59.947 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:59.927 7f5d6effd700 1 -- 172.21.2.102:0/1184084311 >> v1:172.21.2.113:6789/0 conn(0x7f5d700cff70 legacy=0x7f5d70109130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:59.947 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:59.927 7f5d6effd700 1 -- 172.21.2.102:0/1184084311 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d70088260 con 0x7f5d701b0ae0 2021-02-10T11:35:59.947 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:59.927 7f5d6effd700 1 -- 172.21.2.102:0/1184084311 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7f5d64001530 con 0x7f5d701b0ae0 2021-02-10T11:35:59.947 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:59.927 7f5d6effd700 1 -- 172.21.2.102:0/1184084311 >> v1:172.21.2.102:6789/0 conn(0x7f5d701b0ae0 legacy=0x7f5d70108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:59.947 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:59.927 7f5d6effd700 1 --2- 172.21.2.102:0/1184084311 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5d58003a60 0x7f5d58005ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:59.947 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:59.927 7f5d6effd700 1 --2- 172.21.2.102:0/1184084311 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5d58006770 0x7f5d58008c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:59.947 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:59.927 7f5d6effd700 1 --2- 172.21.2.102:0/1184084311 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5d58009590 0x7f5d5800bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:59.948 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:59.927 7f5d6effd700 1 -- 172.21.2.102:0/1184084311 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d5800c300 con 0x7f5d58009590 2021-02-10T11:35:59.948 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:59.927 7f5d6effd700 1 -- 172.21.2.102:0/1184084311 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d5800c520 con 0x7f5d58006770 2021-02-10T11:35:59.948 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:59.928 7f5d6effd700 1 -- 172.21.2.102:0/1184084311 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d58050750 con 0x7f5d58003a60 2021-02-10T11:35:59.948 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:59.928 7f5d74d95700 1 --2- 172.21.2.102:0/1184084311 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5d58009590 0x7f5d5800bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:59.948 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:59.928 7f5d6ffff700 1 --2- 172.21.2.102:0/1184084311 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5d58003a60 0x7f5d58005ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:59.948 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:59.928 7f5d6f7fe700 1 --2- 172.21.2.102:0/1184084311 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5d58006770 0x7f5d58008c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:59.948 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:59.928 7f5d74d95700 1 -- 172.21.2.102:0/1184084311 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5d58003a60 msgr2=0x7f5d58005ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:59.948 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:59.928 7f5d74d95700 1 --2- 172.21.2.102:0/1184084311 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5d58003a60 0x7f5d58005ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:59.948 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:59.928 7f5d74d95700 1 -- 172.21.2.102:0/1184084311 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5d58006770 msgr2=0x7f5d58008c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:59.948 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:59.928 7f5d74d95700 1 --2- 172.21.2.102:0/1184084311 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5d58006770 0x7f5d58008c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:59.948 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:59.928 7f5d74d95700 1 -- 172.21.2.102:0/1184084311 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d580509b0 con 0x7f5d58009590 2021-02-10T11:35:59.948 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:59.928 7f5d6ffff700 1 --2- 172.21.2.102:0/1184084311 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5d58003a60 0x7f5d58005ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:35:59.948 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:59.928 7f5d74d95700 1 -- 172.21.2.102:0/1184084311 reap_dead start 2021-02-10T11:35:59.949 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:59.928 7f5d74d95700 1 --2- 172.21.2.102:0/1184084311 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5d58009590 0x7f5d5800bab0 secure :-1 s=READY pgs=2759 cs=0 l=1 rev1=1 rx=0x7f5d68006e40 tx=0x7f5d68009e30).ready entity=mon.0 client_cookie=2eab5406a986508c server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:59.949 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:59.928 7f5d6effd700 1 -- 172.21.2.102:0/1184084311 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5d68006a60 con 0x7f5d58009590 2021-02-10T11:35:59.949 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:59.928 7f5d6effd700 1 -- 172.21.2.102:0/1184084311 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d6800b300 con 0x7f5d58009590 2021-02-10T11:35:59.949 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:59.928 7f5d6effd700 1 -- 172.21.2.102:0/1184084311 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5d6800b760 con 0x7f5d58009590 2021-02-10T11:35:59.949 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:59.928 7f5d76778700 1 -- 172.21.2.102:0/1184084311 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5d58009590 msgr2=0x7f5d5800bab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:59.949 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:59.928 7f5d76778700 1 --2- 172.21.2.102:0/1184084311 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5d58009590 0x7f5d5800bab0 secure :-1 s=READY pgs=2759 cs=0 l=1 rev1=1 rx=0x7f5d68006e40 tx=0x7f5d68009e30).stop 2021-02-10T11:35:59.949 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:59.929 7f5d76778700 1 -- 172.21.2.102:0/1184084311 shutdown_connections 2021-02-10T11:35:59.949 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:59.929 7f5d76778700 1 --2- 172.21.2.102:0/1184084311 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5d58009590 0x7f5d5800bab0 unknown :-1 s=CLOSED pgs=2759 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:59.949 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:59.929 7f5d76778700 1 -- 172.21.2.102:0/1184084311 >> 172.21.2.102:0/1184084311 conn(0x7f5d700ce420 msgr2=0x7f5d700d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:59.949 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:59.929 7f5d76778700 1 -- 172.21.2.102:0/1184084311 shutdown_connections 2021-02-10T11:35:59.949 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:59.929 7f5d76778700 1 -- 172.21.2.102:0/1184084311 wait complete. 2021-02-10T11:35:59.949 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:59.944 7f5d76778700 1 Processor -- start 2021-02-10T11:35:59.950 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:59.944 7f5d76778700 1 -- start start 2021-02-10T11:35:59.950 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:59.944 7f5d76778700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5d701a79a0 0x7f5d701a7da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:59.950 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:59.944 7f5d76778700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5d700bdb40 0x7f5d700bdf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:59.950 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:59.944 7f5d76778700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5d70082ca0 0x7f5d700830a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:59.950 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:59.944 7f5d76778700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d7019d990 con 0x7f5d70082ca0 2021-02-10T11:35:59.950 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:59.944 7f5d76778700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d7019dbb0 con 0x7f5d700bdb40 2021-02-10T11:35:59.950 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:59.944 7f5d76778700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d700836c0 con 0x7f5d701a79a0 2021-02-10T11:35:59.950 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:59.945 7f5d74d95700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5d70082ca0 0x7f5d700830a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:59.950 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:59.945 7f5d74d95700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5d70082ca0 0x7f5d700830a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:37436/0 (socket says 172.21.2.102:37436) 2021-02-10T11:35:59.950 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:59.945 7f5d74d95700 1 -- 172.21.2.102:0/4095013894 learned_addr learned my addr 172.21.2.102:0/4095013894 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:35:59.950 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:59.945 7f5d6ffff700 1 --2- 172.21.2.102:0/4095013894 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5d701a79a0 0x7f5d701a7da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:59.950 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:59.945 7f5d6f7fe700 1 --2- 172.21.2.102:0/4095013894 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5d700bdb40 0x7f5d700bdf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:59.952 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:59.945 7f5d6ffff700 1 -- 172.21.2.102:0/4095013894 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5d700bdb40 msgr2=0x7f5d700bdf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:59.952 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:59.945 7f5d6ffff700 1 --2- 172.21.2.102:0/4095013894 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5d700bdb40 0x7f5d700bdf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:59.952 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:59.945 7f5d6ffff700 1 -- 172.21.2.102:0/4095013894 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5d70082ca0 msgr2=0x7f5d700830a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:59.952 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:59.945 7f5d6ffff700 1 --2- 172.21.2.102:0/4095013894 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5d70082ca0 0x7f5d700830a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:59.952 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:59.945 7f5d6ffff700 1 -- 172.21.2.102:0/4095013894 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d700838e0 con 0x7f5d701a79a0 2021-02-10T11:35:59.952 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:59.945 7f5d74d95700 1 --2- 172.21.2.102:0/4095013894 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5d70082ca0 0x7f5d700830a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:35:59.952 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:59.945 7f5d6ffff700 1 --2- 172.21.2.102:0/4095013894 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5d701a79a0 0x7f5d701a7da0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0x7f5d60009540 tx=0x7f5d6000b8e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:59.952 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:59.946 7f5d6cff9700 1 -- 172.21.2.102:0/4095013894 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5d600063c0 con 0x7f5d701a79a0 2021-02-10T11:35:59.952 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:59.946 7f5d6cff9700 1 -- 172.21.2.102:0/4095013894 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d6000bd60 con 0x7f5d701a79a0 2021-02-10T11:35:59.952 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:59.946 7f5d6cff9700 1 -- 172.21.2.102:0/4095013894 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5d60002030 con 0x7f5d701a79a0 2021-02-10T11:35:59.952 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:59.946 7f5d76778700 1 -- 172.21.2.102:0/4095013894 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d700838e0 con 0x7f5d701a79a0 2021-02-10T11:35:59.952 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:59.946 7f5d76778700 1 -- 172.21.2.102:0/4095013894 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d700838e0 con 0x7f5d701a79a0 2021-02-10T11:35:59.953 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:59.946 7f5d6cff9700 1 -- 172.21.2.102:0/4095013894 <== mon.2 v2:172.21.2.113:3301/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f5d60000c90 con 0x7f5d701a79a0 2021-02-10T11:35:59.953 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:59.946 7f5d6cff9700 1 --2- 172.21.2.102:0/4095013894 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5d500186c0 0x7f5d5001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:59.953 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:59.946 7f5d6cff9700 1 -- 172.21.2.102:0/4095013894 <== mon.2 v2:172.21.2.113:3301/0 5 ==== osd_map(153..153 src has 1..153) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f5d60032440 con 0x7f5d701a79a0 2021-02-10T11:35:59.953 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:59.946 7f5d6f7fe700 1 --2- 172.21.2.102:0/4095013894 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5d500186c0 0x7f5d5001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:59.953 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:59.947 7f5d6f7fe700 1 --2- 172.21.2.102:0/4095013894 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5d500186c0 0x7f5d5001abc0 crc :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:59.978 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:59.976 7f5d76778700 1 -- 172.21.2.102:0/4095013894 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d70082a70 con 0x7f5d701a79a0 2021-02-10T11:35:59.980 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:35:59.979 7f5d6cff9700 1 -- 172.21.2.102:0/4095013894 <== mon.2 v2:172.21.2.113:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f5d60006ae0 con 0x7f5d701a79a0 2021-02-10T11:36:00.116 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:00.116 7f5d76778700 1 -- 172.21.2.102:0/4095013894 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_command({"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"} v 0) v1 -- 0x7f5d70082a70 con 0x7f5d701a79a0 2021-02-10T11:36:00.298 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:00.297 7f5d6cff9700 1 -- 172.21.2.102:0/4095013894 <== mon.2 v2:172.21.2.113:3301/0 7 ==== mon_command_ack([{"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"}]=0 v132) v1 ==== 118+0+0 (secure 0 0 0) 0x7f5d60000c90 con 0x7f5d701a79a0 2021-02-10T11:36:00.333 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:00.332 7f5d76778700 1 -- 172.21.2.102:0/4095013894 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5d500186c0 msgr2=0x7f5d5001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:00.333 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:00.332 7f5d76778700 1 --2- 172.21.2.102:0/4095013894 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5d500186c0 0x7f5d5001abc0 crc :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:00.333 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:00.332 7f5d76778700 1 -- 172.21.2.102:0/4095013894 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5d701a79a0 msgr2=0x7f5d701a7da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:00.334 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:00.332 7f5d76778700 1 --2- 172.21.2.102:0/4095013894 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5d701a79a0 0x7f5d701a7da0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0x7f5d60009540 tx=0x7f5d6000b8e0).stop 2021-02-10T11:36:00.334 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:00.332 7f5d76778700 1 -- 172.21.2.102:0/4095013894 shutdown_connections 2021-02-10T11:36:00.334 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:00.332 7f5d76778700 1 --2- 172.21.2.102:0/4095013894 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5d70082ca0 0x7f5d700830a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:00.334 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:00.332 7f5d76778700 1 --2- 172.21.2.102:0/4095013894 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5d500186c0 0x7f5d5001abc0 unknown :-1 s=CLOSED pgs=1215 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:00.334 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:00.332 7f5d76778700 1 --2- 172.21.2.102:0/4095013894 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5d700bdb40 0x7f5d700bdf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:00.334 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:00.332 7f5d76778700 1 --2- 172.21.2.102:0/4095013894 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5d701a79a0 0x7f5d701a7da0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:00.334 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:00.332 7f5d76778700 1 -- 172.21.2.102:0/4095013894 >> 172.21.2.102:0/4095013894 conn(0x7f5d700bef00 msgr2=0x7f5d7009f200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:00.334 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:00.333 7f5d76778700 1 -- 172.21.2.102:0/4095013894 shutdown_connections 2021-02-10T11:36:00.335 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:00.333 7f5d76778700 1 -- 172.21.2.102:0/4095013894 wait complete. 2021-02-10T11:36:00.342 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:36:00.774 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:00.754 7fd3da8cc700 1 Processor -- start 2021-02-10T11:36:00.774 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:00.754 7fd3da8cc700 1 -- start start 2021-02-10T11:36:00.774 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:00.754 7fd3da8cc700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd3d41b0470 0x7fd3d41b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:00.774 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:00.754 7fd3da8cc700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3d40f0b50 con 0x7fd3d40ef610 2021-02-10T11:36:00.775 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:00.754 7fd3da8cc700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3d40f24e0 con 0x7fd3d40f20e0 2021-02-10T11:36:00.775 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:00.754 7fd3da8cc700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3d40e5ea0 con 0x7fd3d41b0470 2021-02-10T11:36:00.775 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:00.755 7fd3d37fe700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fd3d40ef610 0x7fd3d40efa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:57900/0 (socket says 172.21.2.102:57900) 2021-02-10T11:36:00.775 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:00.755 7fd3d37fe700 1 -- 172.21.2.102:0/3337876473 learned_addr learned my addr 172.21.2.102:0/3337876473 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:36:00.775 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:00.755 7fd3d2ffd700 1 -- 172.21.2.102:0/3337876473 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3855074409 0 0) 0x7fd3c8001180 con 0x7fd3d40ef610 2021-02-10T11:36:00.775 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:00.755 7fd3d3fff700 1 --2- 172.21.2.102:0/3337876473 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd3d41b0470 0x7fd3d41b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:00.775 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:00.755 7fd3d2ffd700 1 -- 172.21.2.102:0/3337876473 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd3bc002fe0 con 0x7fd3d40ef610 2021-02-10T11:36:00.775 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:00.756 7fd3d2ffd700 1 -- 172.21.2.102:0/3337876473 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2295288953 0 0) 0x7fd3c80010e0 con 0x7fd3d40ef610 2021-02-10T11:36:00.775 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:00.756 7fd3d2ffd700 1 -- 172.21.2.102:0/3337876473 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd3d41b0470 msgr2=0x7fd3d41b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:00.775 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:00.756 7fd3d2ffd700 1 --2- 172.21.2.102:0/3337876473 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd3d41b0470 0x7fd3d41b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:00.775 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:00.756 7fd3d2ffd700 1 -- 172.21.2.102:0/3337876473 >> v1:172.21.2.113:6789/0 conn(0x7fd3d40f20e0 legacy=0x7fd3d40f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:00.775 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:00.756 7fd3d2ffd700 1 -- 172.21.2.102:0/3337876473 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3d40e52e0 con 0x7fd3d40ef610 2021-02-10T11:36:00.776 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:00.756 7fd3d2ffd700 1 -- 172.21.2.102:0/3337876473 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7fd3c8001600 con 0x7fd3d40ef610 2021-02-10T11:36:00.776 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:00.756 7fd3d2ffd700 1 -- 172.21.2.102:0/3337876473 >> v1:172.21.2.102:6789/0 conn(0x7fd3d40ef610 legacy=0x7fd3d40efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:00.776 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:00.756 7fd3d2ffd700 1 --2- 172.21.2.102:0/3337876473 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd3bc003a60 0x7fd3bc005f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:00.776 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:00.756 7fd3d2ffd700 1 --2- 172.21.2.102:0/3337876473 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd3bc006780 0x7fd3bc008c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:00.776 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:00.756 7fd3d2ffd700 1 --2- 172.21.2.102:0/3337876473 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd3bc009580 0x7fd3bc00ba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:00.776 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:00.756 7fd3d2ffd700 1 -- 172.21.2.102:0/3337876473 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3bc00c2c0 con 0x7fd3bc003a60 2021-02-10T11:36:00.776 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:00.756 7fd3d2ffd700 1 -- 172.21.2.102:0/3337876473 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3bc00c4e0 con 0x7fd3bc006780 2021-02-10T11:36:00.776 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:00.756 7fd3d2ffd700 1 -- 172.21.2.102:0/3337876473 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3bc050710 con 0x7fd3bc009580 2021-02-10T11:36:00.776 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:00.756 7fd3d3fff700 1 --2- 172.21.2.102:0/3337876473 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd3bc003a60 0x7fd3bc005f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:00.776 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:00.756 7fd3d37fe700 1 --2- 172.21.2.102:0/3337876473 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd3bc006780 0x7fd3bc008c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:00.776 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:00.756 7fd3d8ee9700 1 --2- 172.21.2.102:0/3337876473 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd3bc009580 0x7fd3bc00ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:00.776 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:00.756 7fd3d3fff700 1 -- 172.21.2.102:0/3337876473 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd3bc009580 msgr2=0x7fd3bc00ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:00.776 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:00.756 7fd3d3fff700 1 --2- 172.21.2.102:0/3337876473 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd3bc009580 0x7fd3bc00ba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:00.777 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:00.756 7fd3d3fff700 1 -- 172.21.2.102:0/3337876473 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd3bc006780 msgr2=0x7fd3bc008c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:00.777 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:00.756 7fd3d3fff700 1 --2- 172.21.2.102:0/3337876473 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd3bc006780 0x7fd3bc008c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:00.777 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:00.756 7fd3d3fff700 1 -- 172.21.2.102:0/3337876473 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3bc050970 con 0x7fd3bc003a60 2021-02-10T11:36:00.777 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:00.756 7fd3d3fff700 1 --2- 172.21.2.102:0/3337876473 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd3bc003a60 0x7fd3bc005f60 secure :-1 s=READY pgs=2761 cs=0 l=1 rev1=1 rx=0x7fd3c4006ac0 tx=0x7fd3c4009ab0).ready entity=mon.0 client_cookie=3e625d09de0bc62e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:00.777 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:00.756 7fd3d2ffd700 1 -- 172.21.2.102:0/3337876473 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd3c400c0c0 con 0x7fd3bc003a60 2021-02-10T11:36:00.777 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:00.756 7fd3d2ffd700 1 -- 172.21.2.102:0/3337876473 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3c400c380 con 0x7fd3bc003a60 2021-02-10T11:36:00.777 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:00.757 7fd3d2ffd700 1 -- 172.21.2.102:0/3337876473 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd3c400c7e0 con 0x7fd3bc003a60 2021-02-10T11:36:00.777 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:00.757 7fd3da8cc700 1 -- 172.21.2.102:0/3337876473 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd3bc003a60 msgr2=0x7fd3bc005f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:00.777 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:00.757 7fd3da8cc700 1 --2- 172.21.2.102:0/3337876473 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd3bc003a60 0x7fd3bc005f60 secure :-1 s=READY pgs=2761 cs=0 l=1 rev1=1 rx=0x7fd3c4006ac0 tx=0x7fd3c4009ab0).stop 2021-02-10T11:36:00.777 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:00.757 7fd3d8ee9700 1 -- 172.21.2.102:0/3337876473 reap_dead start 2021-02-10T11:36:00.777 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:00.757 7fd3da8cc700 1 -- 172.21.2.102:0/3337876473 shutdown_connections 2021-02-10T11:36:00.777 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:00.757 7fd3da8cc700 1 -- 172.21.2.102:0/3337876473 >> 172.21.2.102:0/3337876473 conn(0x7fd3d40ce3e0 msgr2=0x7fd3d40e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:00.777 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:00.757 7fd3da8cc700 1 -- 172.21.2.102:0/3337876473 shutdown_connections 2021-02-10T11:36:00.778 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:00.757 7fd3da8cc700 1 -- 172.21.2.102:0/3337876473 wait complete. 2021-02-10T11:36:00.778 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:00.772 7fd3da8cc700 1 Processor -- start 2021-02-10T11:36:00.778 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:00.772 7fd3da8cc700 1 -- start start 2021-02-10T11:36:00.778 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:00.772 7fd3da8cc700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd3d40938f0 0x7fd3d41a4310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:00.778 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:00.772 7fd3da8cc700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd3d4093cf0 0x7fd3d4092690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:00.778 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:00.772 7fd3da8cc700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd3d4092cb0 0x7fd3d40930b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:00.778 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:00.772 7fd3da8cc700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3d40b8300 con 0x7fd3d4093cf0 2021-02-10T11:36:00.778 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:00.772 7fd3da8cc700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3d40b7f40 con 0x7fd3d40938f0 2021-02-10T11:36:00.778 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:00.772 7fd3da8cc700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3d41a4930 con 0x7fd3d4092cb0 2021-02-10T11:36:00.778 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:00.773 7fd3d37fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd3d4093cf0 0x7fd3d4092690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:00.778 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:00.773 7fd3d37fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd3d4093cf0 0x7fd3d4092690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:37458/0 (socket says 172.21.2.102:37458) 2021-02-10T11:36:00.778 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:00.773 7fd3d37fe700 1 -- 172.21.2.102:0/3208998337 learned_addr learned my addr 172.21.2.102:0/3208998337 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:36:00.779 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:00.773 7fd3d8ee9700 1 --2- 172.21.2.102:0/3208998337 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd3d4092cb0 0x7fd3d40930b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:00.779 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:00.773 7fd3d3fff700 1 --2- 172.21.2.102:0/3208998337 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd3d40938f0 0x7fd3d41a4310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:00.779 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:00.773 7fd3d37fe700 1 -- 172.21.2.102:0/3208998337 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd3d4092cb0 msgr2=0x7fd3d40930b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:00.779 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:00.773 7fd3d37fe700 1 --2- 172.21.2.102:0/3208998337 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd3d4092cb0 0x7fd3d40930b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:00.779 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:00.773 7fd3d37fe700 1 -- 172.21.2.102:0/3208998337 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd3d40938f0 msgr2=0x7fd3d41a4310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:00.779 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:00.773 7fd3d37fe700 1 --2- 172.21.2.102:0/3208998337 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd3d40938f0 0x7fd3d41a4310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:00.779 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:00.773 7fd3d37fe700 1 -- 172.21.2.102:0/3208998337 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3d40ad530 con 0x7fd3d4093cf0 2021-02-10T11:36:00.779 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:00.773 7fd3d8ee9700 1 --2- 172.21.2.102:0/3208998337 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd3d4092cb0 0x7fd3d40930b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:36:00.779 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:00.773 7fd3d37fe700 1 --2- 172.21.2.102:0/3208998337 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd3d4093cf0 0x7fd3d4092690 secure :-1 s=READY pgs=2762 cs=0 l=1 rev1=1 rx=0x7fd3c800a570 tx=0x7fd3c800a910).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:00.780 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:00.773 7fd3d0ff9700 1 -- 172.21.2.102:0/3208998337 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd3c80070d0 con 0x7fd3d4093cf0 2021-02-10T11:36:00.780 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:00.773 7fd3d0ff9700 1 -- 172.21.2.102:0/3208998337 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3c80012a0 con 0x7fd3d4093cf0 2021-02-10T11:36:00.780 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:00.773 7fd3d0ff9700 1 -- 172.21.2.102:0/3208998337 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd3c80045b0 con 0x7fd3d4093cf0 2021-02-10T11:36:00.780 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:00.774 7fd3da8cc700 1 -- 172.21.2.102:0/3208998337 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3d40ad530 con 0x7fd3d4093cf0 2021-02-10T11:36:00.780 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:00.774 7fd3da8cc700 1 -- 172.21.2.102:0/3208998337 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3d40adac0 con 0x7fd3d4093cf0 2021-02-10T11:36:00.780 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:00.774 7fd3d0ff9700 1 -- 172.21.2.102:0/3208998337 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd3c80043a0 con 0x7fd3d4093cf0 2021-02-10T11:36:00.780 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:00.774 7fd3d0ff9700 1 --2- 172.21.2.102:0/3208998337 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd3b40186c0 0x7fd3b401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:00.781 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:00.774 7fd3d0ff9700 1 -- 172.21.2.102:0/3208998337 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(153..153 src has 1..153) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fd3c80323b0 con 0x7fd3d4093cf0 2021-02-10T11:36:00.781 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:00.774 7fd3d3fff700 1 --2- 172.21.2.102:0/3208998337 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd3b40186c0 0x7fd3b401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:00.781 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:00.775 7fd3d3fff700 1 --2- 172.21.2.102:0/3208998337 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd3b40186c0 0x7fd3b401abc0 crc :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:00.805 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:00.804 7fd3da8cc700 1 -- 172.21.2.102:0/3208998337 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3d4105a80 con 0x7fd3d4093cf0 2021-02-10T11:36:00.807 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:00.807 7fd3d0ff9700 1 -- 172.21.2.102:0/3208998337 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd3c8000ef0 con 0x7fd3d4093cf0 2021-02-10T11:36:00.937 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:00.936 7fd3da8cc700 1 -- 172.21.2.102:0/3208998337 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd3d40f0b50 con 0x7fd3d4093cf0 2021-02-10T11:36:00.938 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:00.937 7fd3d0ff9700 1 -- 172.21.2.102:0/3208998337 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 132 v132) v1 ==== 95+0+2107 (secure 0 0 0) 0x7fd3c80043a0 con 0x7fd3d4093cf0 2021-02-10T11:36:00.938 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:36:00.938 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":132,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7004,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.102:6835/3280534775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6834","nonce":3280534775},{"type":"v1","addr":"172.21.2.102:6835","nonce":3280534775}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":128}],"filesystems":[{"mdsmap":{"epoch":132,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:35:58.285904","modified":"2021-02-10 11:36:00.292711","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7001},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7001":{"gid":7001,"name":"b","rank":0,"incarnation":129,"state":"up:active","state_seq":3,"addr":"172.21.2.113:6833/1983258405","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6832","nonce":1983258405},{"type":"v1","addr":"172.21.2.113:6833","nonce":1983258405}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[33],"metadata_pool":32,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":16}]} 2021-02-10T11:36:00.939 INFO:teuthology.orchestra.run.gibba002.stderr:dumped fsmap epoch 132 2021-02-10T11:36:00.940 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:00.938 7fd3da8cc700 1 -- 172.21.2.102:0/3208998337 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd3b40186c0 msgr2=0x7fd3b401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:00.940 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:00.938 7fd3da8cc700 1 --2- 172.21.2.102:0/3208998337 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd3b40186c0 0x7fd3b401abc0 crc :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:00.940 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:00.938 7fd3da8cc700 1 -- 172.21.2.102:0/3208998337 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd3d4093cf0 msgr2=0x7fd3d4092690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:00.940 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:00.938 7fd3da8cc700 1 --2- 172.21.2.102:0/3208998337 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd3d4093cf0 0x7fd3d4092690 secure :-1 s=READY pgs=2762 cs=0 l=1 rev1=1 rx=0x7fd3c800a570 tx=0x7fd3c800a910).stop 2021-02-10T11:36:00.940 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:00.939 7fd3da8cc700 1 -- 172.21.2.102:0/3208998337 shutdown_connections 2021-02-10T11:36:00.940 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:00.939 7fd3da8cc700 1 --2- 172.21.2.102:0/3208998337 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd3d4092cb0 0x7fd3d40930b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:00.940 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:00.939 7fd3da8cc700 1 --2- 172.21.2.102:0/3208998337 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd3d4093cf0 0x7fd3d4092690 unknown :-1 s=CLOSED pgs=2762 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:00.940 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:00.939 7fd3da8cc700 1 --2- 172.21.2.102:0/3208998337 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd3b40186c0 0x7fd3b401abc0 unknown :-1 s=CLOSED pgs=1216 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:00.940 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:00.939 7fd3da8cc700 1 --2- 172.21.2.102:0/3208998337 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd3d40938f0 0x7fd3d41a4310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:00.940 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:00.939 7fd3da8cc700 1 -- 172.21.2.102:0/3208998337 >> 172.21.2.102:0/3208998337 conn(0x7fd3d40a1dc0 msgr2=0x7fd3d408e860 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:00.941 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:00.939 7fd3da8cc700 1 -- 172.21.2.102:0/3208998337 shutdown_connections 2021-02-10T11:36:00.941 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:00.939 7fd3da8cc700 1 -- 172.21.2.102:0/3208998337 wait complete. 2021-02-10T11:36:00.948 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:36:01.379 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:01.360 7f648c7ea700 1 Processor -- start 2021-02-10T11:36:01.379 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:01.360 7f648c7ea700 1 -- start start 2021-02-10T11:36:01.379 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:01.360 7f648c7ea700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f64840d0340 0x7f64840ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:01.379 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:01.360 7f648c7ea700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f64840d1c70 0x7f64840efc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:01.379 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:01.360 7f648c7ea700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64840f0b10 con 0x7f64840d1870 2021-02-10T11:36:01.379 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:01.360 7f648c7ea700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64840e5ed0 con 0x7f64840d0340 2021-02-10T11:36:01.379 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:01.360 7f648c7ea700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64841b0be0 con 0x7f64840d1c70 2021-02-10T11:36:01.379 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:01.362 7f648ae07700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f64840d1c70 0x7f64840efc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:01.379 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:01.362 7f648a606700 1 --1- >> v1:172.21.2.113:6789/0 conn(0x7f64840d1870 0x7f64841b03f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.113:6789/0 says I am v1:172.21.2.102:37012/0 (socket says 172.21.2.102:37012) 2021-02-10T11:36:01.380 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:01.362 7f6489e05700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f64840d0340 0x7f64840ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:01.380 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:01.362 7f648a606700 1 -- 172.21.2.102:0/1075265663 learned_addr learned my addr 172.21.2.102:0/1075265663 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:36:01.380 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:01.362 7f6489604700 1 -- 172.21.2.102:0/1075265663 <== mon.1 v1:172.21.2.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1791240107 0 0) 0x7f6480001190 con 0x7f64840d1870 2021-02-10T11:36:01.380 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:01.362 7f648ae07700 1 -- 172.21.2.102:0/1075265663 >> v2:172.21.2.113:3300/0 conn(0x7f64840d0340 msgr2=0x7f64840ef640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:01.380 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:01.362 7f648ae07700 1 --2- 172.21.2.102:0/1075265663 >> v2:172.21.2.113:3300/0 conn(0x7f64840d0340 0x7f64840ef640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:01.380 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:01.362 7f648ae07700 1 -- 172.21.2.102:0/1075265663 >> v1:172.21.2.113:6789/0 conn(0x7f64840d1870 legacy=0x7f64841b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:01.380 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:01.362 7f648ae07700 1 -- 172.21.2.102:0/1075265663 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64840f20a0 con 0x7f64840d1c70 2021-02-10T11:36:01.380 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:01.362 7f648ae07700 1 --2- 172.21.2.102:0/1075265663 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f64840d1c70 0x7f64840efc60 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 rx=0x7f647c006e50 tx=0x7f647c009e70).ready entity=mon.2 client_cookie=386f23732f451713 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:01.380 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:01.363 7f6489604700 1 -- 172.21.2.102:0/1075265663 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f647c006a20 con 0x7f64840d1c70 2021-02-10T11:36:01.380 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:01.363 7f6489604700 1 -- 172.21.2.102:0/1075265663 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f647c006790 con 0x7f64840d1c70 2021-02-10T11:36:01.380 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:01.363 7f6489604700 1 -- 172.21.2.102:0/1075265663 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f647c00b700 con 0x7f64840d1c70 2021-02-10T11:36:01.381 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:01.363 7f648c7ea700 1 -- 172.21.2.102:0/1075265663 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f64840d1c70 msgr2=0x7f64840efc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:01.381 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:01.363 7f648c7ea700 1 --2- 172.21.2.102:0/1075265663 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f64840d1c70 0x7f64840efc60 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 rx=0x7f647c006e50 tx=0x7f647c009e70).stop 2021-02-10T11:36:01.381 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:01.364 7f648c7ea700 1 -- 172.21.2.102:0/1075265663 shutdown_connections 2021-02-10T11:36:01.381 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:01.364 7f648c7ea700 1 --2- 172.21.2.102:0/1075265663 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f64840d1c70 0x7f64840efc60 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:01.381 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:01.364 7f648c7ea700 1 --2- 172.21.2.102:0/1075265663 >> v2:172.21.2.113:3300/0 conn(0x7f64840d0340 0x7f64840ef640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:01.381 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:01.364 7f648c7ea700 1 -- 172.21.2.102:0/1075265663 >> 172.21.2.102:0/1075265663 conn(0x7f64840ce450 msgr2=0x7f64840d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:01.381 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:01.364 7f648c7ea700 1 -- 172.21.2.102:0/1075265663 shutdown_connections 2021-02-10T11:36:01.381 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:01.364 7f648c7ea700 1 -- 172.21.2.102:0/1075265663 wait complete. 2021-02-10T11:36:01.383 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:01.378 7f648c7ea700 1 Processor -- start 2021-02-10T11:36:01.383 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:01.378 7f648c7ea700 1 -- start start 2021-02-10T11:36:01.384 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:01.379 7f648c7ea700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f648418d870 0x7f64840b19b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:01.384 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:01.379 7f648c7ea700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f64840b1fd0 0x7f6484099ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:01.384 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:01.379 7f648c7ea700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f64840b23d0 0x7f648409a510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:01.384 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:01.379 7f648c7ea700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f648418dc70 con 0x7f648418d870 2021-02-10T11:36:01.384 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:01.379 7f648c7ea700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6484199e60 con 0x7f64840b1fd0 2021-02-10T11:36:01.384 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:01.379 7f648c7ea700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f648409ab30 con 0x7f64840b23d0 2021-02-10T11:36:01.384 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:01.379 7f648a606700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f648418d870 0x7f64840b19b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:01.385 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:01.379 7f648a606700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f648418d870 0x7f64840b19b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:37472/0 (socket says 172.21.2.102:37472) 2021-02-10T11:36:01.385 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:01.379 7f648a606700 1 -- 172.21.2.102:0/4157363254 learned_addr learned my addr 172.21.2.102:0/4157363254 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:36:01.385 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:01.379 7f648a606700 1 -- 172.21.2.102:0/4157363254 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f64840b23d0 msgr2=0x7f648409a510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:01.385 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:01.379 7f648a606700 1 --2- 172.21.2.102:0/4157363254 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f64840b23d0 0x7f648409a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:01.385 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:01.379 7f648a606700 1 -- 172.21.2.102:0/4157363254 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f64840b1fd0 msgr2=0x7f6484099ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:01.385 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:01.379 7f648a606700 1 --2- 172.21.2.102:0/4157363254 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f64840b1fd0 0x7f6484099ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:01.386 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:01.379 7f648a606700 1 -- 172.21.2.102:0/4157363254 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f648409ad50 con 0x7f648418d870 2021-02-10T11:36:01.386 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:01.379 7f648a606700 1 --2- 172.21.2.102:0/4157363254 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f648418d870 0x7f64840b19b0 secure :-1 s=READY pgs=2763 cs=0 l=1 rev1=1 rx=0x7f6480007600 tx=0x7f648000a8a0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:01.386 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:01.379 7f647affd700 1 -- 172.21.2.102:0/4157363254 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f64800070d0 con 0x7f648418d870 2021-02-10T11:36:01.386 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:01.379 7f647affd700 1 -- 172.21.2.102:0/4157363254 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6480000c10 con 0x7f648418d870 2021-02-10T11:36:01.386 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:01.380 7f648c7ea700 1 -- 172.21.2.102:0/4157363254 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f648409ad50 con 0x7f648418d870 2021-02-10T11:36:01.386 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:01.380 7f647affd700 1 -- 172.21.2.102:0/4157363254 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6480004530 con 0x7f648418d870 2021-02-10T11:36:01.387 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:01.380 7f648c7ea700 1 -- 172.21.2.102:0/4157363254 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f648409ad50 con 0x7f648418d870 2021-02-10T11:36:01.387 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:01.380 7f647affd700 1 -- 172.21.2.102:0/4157363254 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f6480007fe0 con 0x7f648418d870 2021-02-10T11:36:01.387 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:01.380 7f647affd700 1 --2- 172.21.2.102:0/4157363254 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f646c0186c0 0x7f646c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:01.387 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:01.380 7f647affd700 1 -- 172.21.2.102:0/4157363254 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(153..153 src has 1..153) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f6480032790 con 0x7f648418d870 2021-02-10T11:36:01.387 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:01.380 7f6489e05700 1 --2- 172.21.2.102:0/4157363254 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f646c0186c0 0x7f646c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:01.387 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:01.381 7f6489e05700 1 --2- 172.21.2.102:0/4157363254 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f646c0186c0 0x7f646c01abc0 crc :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:01.412 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:01.411 7f648c7ea700 1 -- 172.21.2.102:0/4157363254 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64840999c0 con 0x7f648418d870 2021-02-10T11:36:01.414 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:01.413 7f647affd700 1 -- 172.21.2.102:0/4157363254 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f64800070d0 con 0x7f648418d870 2021-02-10T11:36:01.539 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:01.538 7f648c7ea700 1 -- 172.21.2.102:0/4157363254 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f64840999c0 con 0x7f648418d870 2021-02-10T11:36:01.579 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:01.539 7f647affd700 1 -- 172.21.2.102:0/4157363254 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v153) v1 ==== 74+0+14007 (secure 0 0 0) 0x7f6480007fe0 con 0x7f648418d870 2021-02-10T11:36:01.579 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:01.541 7f648c7ea700 1 -- 172.21.2.102:0/4157363254 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f646c0186c0 msgr2=0x7f646c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:01.579 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:01.541 7f648c7ea700 1 --2- 172.21.2.102:0/4157363254 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f646c0186c0 0x7f646c01abc0 crc :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:01.579 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:01.541 7f648c7ea700 1 -- 172.21.2.102:0/4157363254 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f648418d870 msgr2=0x7f64840b19b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:01.579 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:01.541 7f648c7ea700 1 --2- 172.21.2.102:0/4157363254 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f648418d870 0x7f64840b19b0 secure :-1 s=READY pgs=2763 cs=0 l=1 rev1=1 rx=0x7f6480007600 tx=0x7f648000a8a0).stop 2021-02-10T11:36:01.580 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:01.541 7f648c7ea700 1 -- 172.21.2.102:0/4157363254 shutdown_connections 2021-02-10T11:36:01.580 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:01.541 7f648c7ea700 1 --2- 172.21.2.102:0/4157363254 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f64840b23d0 0x7f648409a510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:01.580 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:01.541 7f648c7ea700 1 --2- 172.21.2.102:0/4157363254 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f646c0186c0 0x7f646c01abc0 unknown :-1 s=CLOSED pgs=1217 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:01.580 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:01.541 7f648c7ea700 1 --2- 172.21.2.102:0/4157363254 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f64840b1fd0 0x7f6484099ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:01.580 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:01.541 7f648c7ea700 1 --2- 172.21.2.102:0/4157363254 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f648418d870 0x7f64840b19b0 unknown :-1 s=CLOSED pgs=2763 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:01.580 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:01.541 7f648c7ea700 1 -- 172.21.2.102:0/4157363254 >> 172.21.2.102:0/4157363254 conn(0x7f64840f0550 msgr2=0x7f6484191730 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:01.580 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:01.541 7f648c7ea700 1 -- 172.21.2.102:0/4157363254 shutdown_connections 2021-02-10T11:36:01.580 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:01.541 7f648c7ea700 1 -- 172.21.2.102:0/4157363254 wait complete. 2021-02-10T11:36:01.580 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:36:01.580 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":153,"fsid":"68db30ea-dbce-4aef-a6df-f01c6fea0513","created":"2021-02-10 11:03:06.331557","modified":"2021-02-10 11:35:59.288212","last_up_change":"2021-02-10 11:03:32.690562","last_in_change":"2021-02-10 11:03:32.690562","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":33,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:03:37.322519","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":32,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:35:56.251556","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"152","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":33,"pool_name":"cephfs_data","create_time":"2021-02-10 11:35:57.328218","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"153","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"be691b4d-a039-408a-af80-9d53f02b6f1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":150,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6818","nonce":36088},{"type":"v1","addr":"172.21.2.102:6820","nonce":36088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6822","nonce":36088},{"type":"v1","addr":"172.21.2.102:6824","nonce":36088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6830","nonce":36088},{"type":"v1","addr":"172.21.2.102:6832","nonce":36088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6826","nonce":36088},{"type":"v1","addr":"172.21.2.102:6828","nonce":36088}]},"public_addr":"172.21.2.102:6820/36088","cluster_addr":"172.21.2.102:6824/36088","heartbeat_back_addr":"172.21.2.102:6832/36088","heartbeat_front_addr":"172.21.2.102:6828/36088","state":["exists","up"]},{"osd":1,"uuid":"64e3ccc8-9c69-4d7c-865e-af8e83822635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6819","nonce":36087},{"type":"v1","addr":"172.21.2.102:6821","nonce":36087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6823","nonce":36087},{"type":"v1","addr":"172.21.2.102:6825","nonce":36087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6831","nonce":36087},{"type":"v1","addr":"172.21.2.102:6833","nonce":36087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6827","nonce":36087},{"type":"v1","addr":"172.21.2.102:6829","nonce":36087}]},"public_addr":"172.21.2.102:6821/36087","cluster_addr":"172.21.2.102:6825/36087","heartbeat_back_addr":"172.21.2.102:6833/36087","heartbeat_front_addr":"172.21.2.102:6829/36087","state":["exists","up"]},{"osd":2,"uuid":"153700af-747f-481f-bdf4-d6072d3a842f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6810","nonce":36089},{"type":"v1","addr":"172.21.2.102:6811","nonce":36089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6812","nonce":36089},{"type":"v1","addr":"172.21.2.102:6813","nonce":36089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6816","nonce":36089},{"type":"v1","addr":"172.21.2.102:6817","nonce":36089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6814","nonce":36089},{"type":"v1","addr":"172.21.2.102:6815","nonce":36089}]},"public_addr":"172.21.2.102:6811/36089","cluster_addr":"172.21.2.102:6813/36089","heartbeat_back_addr":"172.21.2.102:6817/36089","heartbeat_front_addr":"172.21.2.102:6815/36089","state":["exists","up"]},{"osd":3,"uuid":"cbc678f4-83a9-45e6-8830-2f7c232b1ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6802","nonce":36090},{"type":"v1","addr":"172.21.2.102:6803","nonce":36090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6804","nonce":36090},{"type":"v1","addr":"172.21.2.102:6805","nonce":36090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6808","nonce":36090},{"type":"v1","addr":"172.21.2.102:6809","nonce":36090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6806","nonce":36090},{"type":"v1","addr":"172.21.2.102:6807","nonce":36090}]},"public_addr":"172.21.2.102:6803/36090","cluster_addr":"172.21.2.102:6805/36090","heartbeat_back_addr":"172.21.2.102:6809/36090","heartbeat_front_addr":"172.21.2.102:6807/36090","state":["exists","up"]},{"osd":4,"uuid":"c01e4008-b258-4f10-afd2-9e7f37214351","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6808","nonce":36368},{"type":"v1","addr":"172.21.2.113:6811","nonce":36368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6814","nonce":36368},{"type":"v1","addr":"172.21.2.113:6818","nonce":36368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6826","nonce":36368},{"type":"v1","addr":"172.21.2.113:6828","nonce":36368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6820","nonce":36368},{"type":"v1","addr":"172.21.2.113:6824","nonce":36368}]},"public_addr":"172.21.2.113:6811/36368","cluster_addr":"172.21.2.113:6818/36368","heartbeat_back_addr":"172.21.2.113:6828/36368","heartbeat_front_addr":"172.21.2.113:6824/36368","state":["exists","up"]},{"osd":5,"uuid":"551f6007-daf8-4a98-8ee0-c2b16bcccb5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6802","nonce":36366},{"type":"v1","addr":"172.21.2.113:6805","nonce":36366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6807","nonce":36366},{"type":"v1","addr":"172.21.2.113:6810","nonce":36366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6819","nonce":36366},{"type":"v1","addr":"172.21.2.113:6822","nonce":36366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6812","nonce":36366},{"type":"v1","addr":"172.21.2.113:6815","nonce":36366}]},"public_addr":"172.21.2.113:6805/36366","cluster_addr":"172.21.2.113:6810/36366","heartbeat_back_addr":"172.21.2.113:6822/36366","heartbeat_front_addr":"172.21.2.113:6815/36366","state":["exists","up"]},{"osd":6,"uuid":"bf67c040-0e50-48be-b102-c3943bc9ebef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":150,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6800","nonce":36367},{"type":"v1","addr":"172.21.2.113:6801","nonce":36367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6803","nonce":36367},{"type":"v1","addr":"172.21.2.113:6804","nonce":36367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6813","nonce":36367},{"type":"v1","addr":"172.21.2.113:6816","nonce":36367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6806","nonce":36367},{"type":"v1","addr":"172.21.2.113:6809","nonce":36367}]},"public_addr":"172.21.2.113:6801/36367","cluster_addr":"172.21.2.113:6804/36367","heartbeat_back_addr":"172.21.2.113:6816/36367","heartbeat_front_addr":"172.21.2.113:6809/36367","state":["exists","up"]},{"osd":7,"uuid":"f5461f80-018d-428f-be89-2eb2bcbe7bb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6817","nonce":36365},{"type":"v1","addr":"172.21.2.113:6821","nonce":36365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6823","nonce":36365},{"type":"v1","addr":"172.21.2.113:6825","nonce":36365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6830","nonce":36365},{"type":"v1","addr":"172.21.2.113:6831","nonce":36365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6827","nonce":36365},{"type":"v1","addr":"172.21.2.113:6829","nonce":36365}]},"public_addr":"172.21.2.113:6821/36365","cluster_addr":"172.21.2.113:6825/36365","heartbeat_back_addr":"172.21.2.113:6831/36365","heartbeat_front_addr":"172.21.2.113:6829/36365","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:36:01.581 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:36:02.023 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:02.003 7f6b57889700 1 Processor -- start 2021-02-10T11:36:02.023 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:02.004 7f6b57889700 1 -- start start 2021-02-10T11:36:02.023 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:02.004 7f6b57889700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f6b500d1c80 0x7f6b500cef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:02.023 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:02.004 7f6b57889700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b500d0180 con 0x7f6b500d1880 2021-02-10T11:36:02.023 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:02.004 7f6b57889700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b500fe1d0 con 0x7f6b500cf590 2021-02-10T11:36:02.023 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:02.004 7f6b57889700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b501b0cc0 con 0x7f6b500d1c80 2021-02-10T11:36:02.023 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:02.004 7f6b556a5700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f6b500d1880 0x7f6b501b04d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:57934/0 (socket says 172.21.2.102:57934) 2021-02-10T11:36:02.023 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:02.004 7f6b556a5700 1 -- 172.21.2.102:0/1552561607 learned_addr learned my addr 172.21.2.102:0/1552561607 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:36:02.023 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:02.005 7f6b43fff700 1 -- 172.21.2.102:0/1552561607 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1080914422 0 0) 0x7f6b4c001180 con 0x7f6b500d1880 2021-02-10T11:36:02.024 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:02.005 7f6b43fff700 1 -- 172.21.2.102:0/1552561607 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b3c002fe0 con 0x7f6b500d1880 2021-02-10T11:36:02.024 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:02.005 7f6b54ea4700 1 --2- 172.21.2.102:0/1552561607 >> v2:172.21.2.113:3300/0 conn(0x7f6b500d1c80 0x7f6b500cef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:02.024 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:02.005 7f6b43fff700 1 -- 172.21.2.102:0/1552561607 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 424710478 0 0) 0x7f6b4c001180 con 0x7f6b500d1880 2021-02-10T11:36:02.024 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:02.005 7f6b43fff700 1 -- 172.21.2.102:0/1552561607 >> v2:172.21.2.113:3300/0 conn(0x7f6b500d1c80 msgr2=0x7f6b500cef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:02.024 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:02.005 7f6b43fff700 1 --2- 172.21.2.102:0/1552561607 >> v2:172.21.2.113:3300/0 conn(0x7f6b500d1c80 0x7f6b500cef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:02.024 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:02.005 7f6b43fff700 1 -- 172.21.2.102:0/1552561607 >> v1:172.21.2.113:6789/0 conn(0x7f6b500cf590 legacy=0x7f6b500cf990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:02.024 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:02.005 7f6b43fff700 1 -- 172.21.2.102:0/1552561607 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b50109a70 con 0x7f6b500d1880 2021-02-10T11:36:02.024 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:02.005 7f6b43fff700 1 -- 172.21.2.102:0/1552561607 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7f6b4c001680 con 0x7f6b500d1880 2021-02-10T11:36:02.024 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:02.005 7f6b43fff700 1 -- 172.21.2.102:0/1552561607 >> v1:172.21.2.102:6789/0 conn(0x7f6b500d1880 legacy=0x7f6b501b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:02.024 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:02.005 7f6b43fff700 1 --2- 172.21.2.102:0/1552561607 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6b3c003a70 0x7f6b3c005f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:02.024 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:02.005 7f6b43fff700 1 --2- 172.21.2.102:0/1552561607 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6b3c0067a0 0x7f6b3c008ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:02.024 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:02.005 7f6b43fff700 1 --2- 172.21.2.102:0/1552561607 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6b3c0095a0 0x7f6b3c00bac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:02.024 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:02.005 7f6b43fff700 1 -- 172.21.2.102:0/1552561607 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b3c00c310 con 0x7f6b3c0095a0 2021-02-10T11:36:02.025 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:02.005 7f6b43fff700 1 -- 172.21.2.102:0/1552561607 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b3c00c530 con 0x7f6b3c003a70 2021-02-10T11:36:02.025 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:02.005 7f6b43fff700 1 -- 172.21.2.102:0/1552561607 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b3c00c750 con 0x7f6b3c0067a0 2021-02-10T11:36:02.025 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:02.005 7f6b556a5700 1 --2- 172.21.2.102:0/1552561607 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6b3c003a70 0x7f6b3c005f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:02.025 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:02.005 7f6b54ea4700 1 --2- 172.21.2.102:0/1552561607 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6b3c0067a0 0x7f6b3c008ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:02.025 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:02.005 7f6b55ea6700 1 --2- 172.21.2.102:0/1552561607 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6b3c0095a0 0x7f6b3c00bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:02.025 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:02.006 7f6b55ea6700 1 -- 172.21.2.102:0/1552561607 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6b3c0067a0 msgr2=0x7f6b3c008ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:02.025 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:02.006 7f6b55ea6700 1 --2- 172.21.2.102:0/1552561607 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6b3c0067a0 0x7f6b3c008ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:02.025 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:02.006 7f6b55ea6700 1 -- 172.21.2.102:0/1552561607 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6b3c003a70 msgr2=0x7f6b3c005f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:02.025 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:02.006 7f6b55ea6700 1 --2- 172.21.2.102:0/1552561607 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6b3c003a70 0x7f6b3c005f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:02.025 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:02.006 7f6b55ea6700 1 -- 172.21.2.102:0/1552561607 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b3c00c9f0 con 0x7f6b3c0095a0 2021-02-10T11:36:02.025 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:02.006 7f6b55ea6700 1 -- 172.21.2.102:0/1552561607 reap_dead start 2021-02-10T11:36:02.025 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:02.006 7f6b55ea6700 1 --2- 172.21.2.102:0/1552561607 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6b3c0095a0 0x7f6b3c00bac0 secure :-1 s=READY pgs=2765 cs=0 l=1 rev1=1 rx=0x7f6b480069e0 tx=0x7f6b48009e10).ready entity=mon.0 client_cookie=bc9e910f2a13aba6 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:02.025 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:02.006 7f6b43fff700 1 -- 172.21.2.102:0/1552561607 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6b4800b580 con 0x7f6b3c0095a0 2021-02-10T11:36:02.026 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:02.006 7f6b43fff700 1 -- 172.21.2.102:0/1552561607 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b4800b840 con 0x7f6b3c0095a0 2021-02-10T11:36:02.026 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:02.006 7f6b43fff700 1 -- 172.21.2.102:0/1552561607 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6b4800bd60 con 0x7f6b3c0095a0 2021-02-10T11:36:02.026 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:02.006 7f6b57889700 1 -- 172.21.2.102:0/1552561607 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6b3c0095a0 msgr2=0x7f6b3c00bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:02.026 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:02.006 7f6b57889700 1 --2- 172.21.2.102:0/1552561607 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6b3c0095a0 0x7f6b3c00bac0 secure :-1 s=READY pgs=2765 cs=0 l=1 rev1=1 rx=0x7f6b480069e0 tx=0x7f6b48009e10).stop 2021-02-10T11:36:02.026 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:02.007 7f6b57889700 1 -- 172.21.2.102:0/1552561607 shutdown_connections 2021-02-10T11:36:02.026 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:02.007 7f6b57889700 1 --2- 172.21.2.102:0/1552561607 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6b3c0095a0 0x7f6b3c00bac0 unknown :-1 s=CLOSED pgs=2765 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:02.026 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:02.007 7f6b57889700 1 -- 172.21.2.102:0/1552561607 >> 172.21.2.102:0/1552561607 conn(0x7f6b500ce430 msgr2=0x7f6b500d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:02.026 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:02.007 7f6b57889700 1 -- 172.21.2.102:0/1552561607 shutdown_connections 2021-02-10T11:36:02.026 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:02.007 7f6b57889700 1 -- 172.21.2.102:0/1552561607 wait complete. 2021-02-10T11:36:02.026 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:02.021 7f6b57889700 1 Processor -- start 2021-02-10T11:36:02.026 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:02.022 7f6b57889700 1 -- start start 2021-02-10T11:36:02.027 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:02.022 7f6b57889700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6b500abd30 0x7f6b50082bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:02.028 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:02.022 7f6b57889700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6b500831d0 0x7f6b5008a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:02.028 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:02.022 7f6b57889700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6b500ac130 0x7f6b5008aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:02.028 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:02.022 7f6b57889700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b50083700 con 0x7f6b500abd30 2021-02-10T11:36:02.028 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:02.022 7f6b57889700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b5008b0c0 con 0x7f6b500831d0 2021-02-10T11:36:02.028 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:02.022 7f6b57889700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b5008b2a0 con 0x7f6b500ac130 2021-02-10T11:36:02.028 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:02.022 7f6b556a5700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6b500abd30 0x7f6b50082bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:02.028 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:02.022 7f6b556a5700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6b500abd30 0x7f6b50082bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:37492/0 (socket says 172.21.2.102:37492) 2021-02-10T11:36:02.028 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:02.022 7f6b556a5700 1 -- 172.21.2.102:0/2186262032 learned_addr learned my addr 172.21.2.102:0/2186262032 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:36:02.028 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:02.023 7f6b556a5700 1 -- 172.21.2.102:0/2186262032 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6b500ac130 msgr2=0x7f6b5008aaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:02.028 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:02.023 7f6b556a5700 1 --2- 172.21.2.102:0/2186262032 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6b500ac130 0x7f6b5008aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:02.028 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:02.023 7f6b556a5700 1 -- 172.21.2.102:0/2186262032 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6b500831d0 msgr2=0x7f6b5008a480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:02.029 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:02.023 7f6b556a5700 1 --2- 172.21.2.102:0/2186262032 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6b500831d0 0x7f6b5008a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:02.029 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:02.023 7f6b556a5700 1 -- 172.21.2.102:0/2186262032 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b5008b5f0 con 0x7f6b500abd30 2021-02-10T11:36:02.029 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:02.023 7f6b556a5700 1 --2- 172.21.2.102:0/2186262032 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6b500abd30 0x7f6b50082bb0 secure :-1 s=READY pgs=2766 cs=0 l=1 rev1=1 rx=0x7f6b4c04bee0 tx=0x7f6b4c04d970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:02.029 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:02.023 7f6b41ffb700 1 -- 172.21.2.102:0/2186262032 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6b4c04ba80 con 0x7f6b500abd30 2021-02-10T11:36:02.029 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:02.023 7f6b41ffb700 1 -- 172.21.2.102:0/2186262032 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b4c0043a0 con 0x7f6b500abd30 2021-02-10T11:36:02.029 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:02.023 7f6b41ffb700 1 -- 172.21.2.102:0/2186262032 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6b4c0047f0 con 0x7f6b500abd30 2021-02-10T11:36:02.029 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:02.023 7f6b57889700 1 -- 172.21.2.102:0/2186262032 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b5008b5f0 con 0x7f6b500abd30 2021-02-10T11:36:02.029 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:02.023 7f6b57889700 1 -- 172.21.2.102:0/2186262032 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b5008bb50 con 0x7f6b500abd30 2021-02-10T11:36:02.029 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:02.024 7f6b41ffb700 1 -- 172.21.2.102:0/2186262032 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f6b4c04ba80 con 0x7f6b500abd30 2021-02-10T11:36:02.029 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:02.024 7f6b41ffb700 1 --2- 172.21.2.102:0/2186262032 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6b300186c0 0x7f6b3001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:02.029 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:02.024 7f6b41ffb700 1 -- 172.21.2.102:0/2186262032 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(153..153 src has 1..153) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f6b4c076410 con 0x7f6b500abd30 2021-02-10T11:36:02.029 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:02.024 7f6b54ea4700 1 --2- 172.21.2.102:0/2186262032 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6b300186c0 0x7f6b3001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:02.029 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:02.024 7f6b54ea4700 1 --2- 172.21.2.102:0/2186262032 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6b300186c0 0x7f6b3001abc0 crc :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:02.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:02.054 7f6b57889700 1 -- 172.21.2.102:0/2186262032 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b5008b2a0 con 0x7f6b500abd30 2021-02-10T11:36:02.057 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:02.056 7f6b41ffb700 1 -- 172.21.2.102:0/2186262032 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f6b4c0060c0 con 0x7f6b500abd30 2021-02-10T11:36:02.187 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:02.186 7f6b57889700 1 -- 172.21.2.102:0/2186262032 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f6b5008b2a0 con 0x7f6b500abd30 2021-02-10T11:36:02.190 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:02.189 7f6b41ffb700 1 -- 172.21.2.102:0/2186262032 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.0 v85) v1 ==== 184+0+0 (secure 0 0 0) 0x7f6b4c063520 con 0x7f6b500abd30 2021-02-10T11:36:02.194 INFO:teuthology.orchestra.run.gibba002.stderr:updated caps for client.0 2021-02-10T11:36:02.195 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:02.195 7f6b57889700 1 -- 172.21.2.102:0/2186262032 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6b300186c0 msgr2=0x7f6b3001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:02.195 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:02.195 7f6b57889700 1 --2- 172.21.2.102:0/2186262032 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6b300186c0 0x7f6b3001abc0 crc :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:02.196 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:02.195 7f6b57889700 1 -- 172.21.2.102:0/2186262032 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6b500abd30 msgr2=0x7f6b50082bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:02.196 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:02.195 7f6b57889700 1 --2- 172.21.2.102:0/2186262032 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6b500abd30 0x7f6b50082bb0 secure :-1 s=READY pgs=2766 cs=0 l=1 rev1=1 rx=0x7f6b4c04bee0 tx=0x7f6b4c04d970).stop 2021-02-10T11:36:02.196 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:02.195 7f6b57889700 1 -- 172.21.2.102:0/2186262032 shutdown_connections 2021-02-10T11:36:02.196 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:02.195 7f6b57889700 1 --2- 172.21.2.102:0/2186262032 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6b500ac130 0x7f6b5008aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:02.196 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:02.195 7f6b57889700 1 --2- 172.21.2.102:0/2186262032 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6b300186c0 0x7f6b3001abc0 unknown :-1 s=CLOSED pgs=1218 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:02.197 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:02.195 7f6b57889700 1 --2- 172.21.2.102:0/2186262032 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6b500831d0 0x7f6b5008a480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:02.197 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:02.195 7f6b57889700 1 --2- 172.21.2.102:0/2186262032 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6b500abd30 0x7f6b50082bb0 unknown :-1 s=CLOSED pgs=2766 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:02.197 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:02.195 7f6b57889700 1 -- 172.21.2.102:0/2186262032 >> 172.21.2.102:0/2186262032 conn(0x7f6b500afc60 msgr2=0x7f6b500b50d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:02.197 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:02.195 7f6b57889700 1 -- 172.21.2.102:0/2186262032 shutdown_connections 2021-02-10T11:36:02.197 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:02.195 7f6b57889700 1 -- 172.21.2.102:0/2186262032 wait complete. 2021-02-10T11:36:02.204 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.1 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:36:02.645 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:02.627 7fac359ce700 1 Processor -- start 2021-02-10T11:36:02.645 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:02.627 7fac359ce700 1 -- start start 2021-02-10T11:36:02.645 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:02.628 7fac359ce700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fac300d1880 0x7fac301b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:02.645 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:02.628 7fac359ce700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fac300cff80 0x7fac30109140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:02.645 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:02.628 7fac359ce700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac300fe1d0 con 0x7fac301b0af0 2021-02-10T11:36:02.645 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:02.628 7fac359ce700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac30109ae0 con 0x7fac300cff80 2021-02-10T11:36:02.645 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:02.628 7fac359ce700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac30109760 con 0x7fac300d1880 2021-02-10T11:36:02.646 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:02.628 7fac2fd75700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fac300cff80 0x7fac30109140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:02.646 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:02.628 7fac2fd75700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fac300cff80 0x7fac30109140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:37498/0 (socket says 172.21.2.102:37498) 2021-02-10T11:36:02.646 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:02.628 7fac2fd75700 1 -- 172.21.2.102:0/1904973071 learned_addr learned my addr 172.21.2.102:0/1904973071 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:36:02.646 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:02.628 7fac2fd75700 1 -- 172.21.2.102:0/1904973071 >> v2:172.21.2.113:3300/0 conn(0x7fac300d1880 msgr2=0x7fac301b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:02.646 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:02.628 7fac2fd75700 1 --2- 172.21.2.102:0/1904973071 >> v2:172.21.2.113:3300/0 conn(0x7fac300d1880 0x7fac301b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:02.646 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:02.628 7fac2fd75700 1 -- 172.21.2.102:0/1904973071 >> v1:172.21.2.102:6789/0 conn(0x7fac301b0af0 legacy=0x7fac30108950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:02.646 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:02.628 7fac2fd75700 1 -- 172.21.2.102:0/1904973071 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac30088070 con 0x7fac300cff80 2021-02-10T11:36:02.646 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:02.629 7fac2fd75700 1 --2- 172.21.2.102:0/1904973071 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fac300cff80 0x7fac30109140 secure :-1 s=READY pgs=2768 cs=0 l=1 rev1=1 rx=0x7fac28006f00 tx=0x7fac28009f10).ready entity=mon.0 client_cookie=14372f039502b7e6 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:02.646 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:02.629 7fac2e572700 1 -- 172.21.2.102:0/1904973071 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fac2800b3b0 con 0x7fac300cff80 2021-02-10T11:36:02.646 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:02.629 7fac2e572700 1 -- 172.21.2.102:0/1904973071 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac2800b630 con 0x7fac300cff80 2021-02-10T11:36:02.646 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:02.629 7fac2e572700 1 -- 172.21.2.102:0/1904973071 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fac2800bb50 con 0x7fac300cff80 2021-02-10T11:36:02.646 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:02.629 7fac359ce700 1 -- 172.21.2.102:0/1904973071 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fac300cff80 msgr2=0x7fac30109140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:02.647 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:02.629 7fac359ce700 1 --2- 172.21.2.102:0/1904973071 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fac300cff80 0x7fac30109140 secure :-1 s=READY pgs=2768 cs=0 l=1 rev1=1 rx=0x7fac28006f00 tx=0x7fac28009f10).stop 2021-02-10T11:36:02.647 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:02.629 7fac359ce700 1 -- 172.21.2.102:0/1904973071 shutdown_connections 2021-02-10T11:36:02.647 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:02.629 7fac359ce700 1 --2- 172.21.2.102:0/1904973071 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fac300cff80 0x7fac30109140 unknown :-1 s=CLOSED pgs=2768 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:02.647 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:02.629 7fac359ce700 1 --2- 172.21.2.102:0/1904973071 >> v2:172.21.2.113:3300/0 conn(0x7fac300d1880 0x7fac301b04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:02.647 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:02.629 7fac359ce700 1 -- 172.21.2.102:0/1904973071 >> 172.21.2.102:0/1904973071 conn(0x7fac300ce430 msgr2=0x7fac300d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:02.647 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:02.630 7fac359ce700 1 -- 172.21.2.102:0/1904973071 shutdown_connections 2021-02-10T11:36:02.647 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:02.630 7fac359ce700 1 -- 172.21.2.102:0/1904973071 wait complete. 2021-02-10T11:36:02.647 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:02.644 7fac359ce700 1 Processor -- start 2021-02-10T11:36:02.649 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:02.644 7fac359ce700 1 -- start start 2021-02-10T11:36:02.649 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:02.645 7fac359ce700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fac30086520 0x7fac301a1cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:02.649 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:02.645 7fac359ce700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fac30086920 0x7fac30187e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:02.649 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:02.645 7fac359ce700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fac301a22d0 0x7fac301884a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:02.649 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:02.645 7fac359ce700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac30099600 con 0x7fac30086520 2021-02-10T11:36:02.649 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:02.645 7fac359ce700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac300999f0 con 0x7fac30086920 2021-02-10T11:36:02.649 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:02.645 7fac359ce700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac30188ac0 con 0x7fac301a22d0 2021-02-10T11:36:02.649 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:02.645 7fac2f574700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fac30086520 0x7fac301a1cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:02.649 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:02.645 7fac2f574700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fac30086520 0x7fac301a1cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:37506/0 (socket says 172.21.2.102:37506) 2021-02-10T11:36:02.649 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:02.645 7fac2f574700 1 -- 172.21.2.102:0/1758791130 learned_addr learned my addr 172.21.2.102:0/1758791130 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:36:02.650 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:02.645 7fac2ed73700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fac30086920 0x7fac30187e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:02.650 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:02.645 7fac2fd75700 1 --2- 172.21.2.102:0/1758791130 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fac301a22d0 0x7fac301884a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:02.650 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:02.645 7fac2f574700 1 -- 172.21.2.102:0/1758791130 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fac301a22d0 msgr2=0x7fac301884a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:02.650 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:02.645 7fac2f574700 1 --2- 172.21.2.102:0/1758791130 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fac301a22d0 0x7fac301884a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:02.650 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:02.645 7fac2f574700 1 -- 172.21.2.102:0/1758791130 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fac30086920 msgr2=0x7fac30187e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:02.650 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:02.645 7fac2f574700 1 --2- 172.21.2.102:0/1758791130 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fac30086920 0x7fac30187e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:02.650 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:02.645 7fac2f574700 1 -- 172.21.2.102:0/1758791130 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac30188ce0 con 0x7fac30086520 2021-02-10T11:36:02.650 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:02.645 7fac2f574700 1 --2- 172.21.2.102:0/1758791130 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fac30086520 0x7fac301a1cb0 secure :-1 s=READY pgs=2769 cs=0 l=1 rev1=1 rx=0x7fac18006f00 tx=0x7fac1800b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:02.650 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:02.646 7fac27fff700 1 -- 172.21.2.102:0/1758791130 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fac18006b90 con 0x7fac30086520 2021-02-10T11:36:02.650 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:02.646 7fac27fff700 1 -- 172.21.2.102:0/1758791130 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac1800bd30 con 0x7fac30086520 2021-02-10T11:36:02.650 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:02.646 7fac359ce700 1 -- 172.21.2.102:0/1758791130 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac30188ce0 con 0x7fac30086520 2021-02-10T11:36:02.650 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:02.646 7fac27fff700 1 -- 172.21.2.102:0/1758791130 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fac180020d0 con 0x7fac30086520 2021-02-10T11:36:02.650 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:02.646 7fac359ce700 1 -- 172.21.2.102:0/1758791130 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac30188ce0 con 0x7fac30086520 2021-02-10T11:36:02.651 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:02.646 7fac27fff700 1 -- 172.21.2.102:0/1758791130 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fac180025a0 con 0x7fac30086520 2021-02-10T11:36:02.651 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:02.646 7fac27fff700 1 --2- 172.21.2.102:0/1758791130 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fac14018730 0x7fac1401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:02.651 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:02.646 7fac27fff700 1 -- 172.21.2.102:0/1758791130 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(153..153 src has 1..153) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fac18032620 con 0x7fac30086520 2021-02-10T11:36:02.651 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:02.646 7fac2ed73700 1 --2- 172.21.2.102:0/1758791130 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fac14018730 0x7fac1401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:02.651 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:02.647 7fac2ed73700 1 --2- 172.21.2.102:0/1758791130 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fac14018730 0x7fac1401ac10 crc :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:02.678 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:02.678 7fac359ce700 1 -- 172.21.2.102:0/1758791130 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac300fe1d0 con 0x7fac30086520 2021-02-10T11:36:02.682 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:02.680 7fac27fff700 1 -- 172.21.2.102:0/1758791130 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fac18002390 con 0x7fac30086520 2021-02-10T11:36:02.812 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:02.811 7fac359ce700 1 -- 172.21.2.102:0/1758791130 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7fac300fe1d0 con 0x7fac30086520 2021-02-10T11:36:02.815 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:02.815 7fac27fff700 1 -- 172.21.2.102:0/1758791130 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.1 v86) v1 ==== 184+0+0 (secure 0 0 0) 0x7fac180025a0 con 0x7fac30086520 2021-02-10T11:36:02.819 INFO:teuthology.orchestra.run.gibba002.stderr:updated caps for client.1 2021-02-10T11:36:02.821 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:02.819 7fac359ce700 1 -- 172.21.2.102:0/1758791130 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fac14018730 msgr2=0x7fac1401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:02.821 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:02.819 7fac359ce700 1 --2- 172.21.2.102:0/1758791130 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fac14018730 0x7fac1401ac10 crc :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:02.821 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:02.819 7fac359ce700 1 -- 172.21.2.102:0/1758791130 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fac30086520 msgr2=0x7fac301a1cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:02.821 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:02.819 7fac359ce700 1 --2- 172.21.2.102:0/1758791130 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fac30086520 0x7fac301a1cb0 secure :-1 s=READY pgs=2769 cs=0 l=1 rev1=1 rx=0x7fac18006f00 tx=0x7fac1800b940).stop 2021-02-10T11:36:02.821 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:02.820 7fac359ce700 1 -- 172.21.2.102:0/1758791130 shutdown_connections 2021-02-10T11:36:02.821 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:02.820 7fac359ce700 1 --2- 172.21.2.102:0/1758791130 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fac301a22d0 0x7fac301884a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:02.822 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:02.820 7fac359ce700 1 --2- 172.21.2.102:0/1758791130 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fac14018730 0x7fac1401ac10 unknown :-1 s=CLOSED pgs=1219 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:02.822 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:02.820 7fac359ce700 1 --2- 172.21.2.102:0/1758791130 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fac30086920 0x7fac30187e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:02.822 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:02.820 7fac359ce700 1 --2- 172.21.2.102:0/1758791130 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fac30086520 0x7fac301a1cb0 unknown :-1 s=CLOSED pgs=2769 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:02.822 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:02.820 7fac359ce700 1 -- 172.21.2.102:0/1758791130 >> 172.21.2.102:0/1758791130 conn(0x7fac3019f880 msgr2=0x7fac3008f890 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:02.822 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:02.820 7fac359ce700 1 -- 172.21.2.102:0/1758791130 shutdown_connections 2021-02-10T11:36:02.822 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:02.820 7fac359ce700 1 -- 172.21.2.102:0/1758791130 wait complete. 2021-02-10T11:36:02.829 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.2 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:36:03.260 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:03.242 7f99549d3700 1 Processor -- start 2021-02-10T11:36:03.261 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:03.242 7f99549d3700 1 -- start start 2021-02-10T11:36:03.261 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:03.243 7f99549d3700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f99500d1880 0x7f99501b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:03.261 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:03.243 7f99549d3700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f99500cff80 0x7f9950109140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:03.261 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:03.243 7f99549d3700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99500fe1d0 con 0x7f99501b0af0 2021-02-10T11:36:03.261 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:03.243 7f99549d3700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9950109ae0 con 0x7f99500d1880 2021-02-10T11:36:03.261 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:03.243 7f99549d3700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9950109760 con 0x7f99500cff80 2021-02-10T11:36:03.261 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:03.243 7f994e61c700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f99500d1880 0x7f99501b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:03.261 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:03.243 7f994e61c700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f99500d1880 0x7f99501b04d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:37510/0 (socket says 172.21.2.102:37510) 2021-02-10T11:36:03.261 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:03.243 7f994e61c700 1 -- 172.21.2.102:0/4223501663 learned_addr learned my addr 172.21.2.102:0/4223501663 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:36:03.261 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:03.243 7f994de1b700 1 --1- 172.21.2.102:0/4223501663 >> v1:172.21.2.102:6789/0 conn(0x7f99501b0af0 0x7f9950108950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:57966/0 (socket says 172.21.2.102:57966) 2021-02-10T11:36:03.262 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:03.243 7f994ee1d700 1 --2- 172.21.2.102:0/4223501663 >> v2:172.21.2.113:3300/0 conn(0x7f99500cff80 0x7f9950109140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:03.262 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:03.243 7f994e61c700 1 -- 172.21.2.102:0/4223501663 >> v2:172.21.2.113:3300/0 conn(0x7f99500cff80 msgr2=0x7f9950109140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:03.262 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:03.243 7f994e61c700 1 --2- 172.21.2.102:0/4223501663 >> v2:172.21.2.113:3300/0 conn(0x7f99500cff80 0x7f9950109140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:03.262 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:03.243 7f994e61c700 1 -- 172.21.2.102:0/4223501663 >> v1:172.21.2.102:6789/0 conn(0x7f99501b0af0 legacy=0x7f9950108950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:03.262 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:03.243 7f994e61c700 1 -- 172.21.2.102:0/4223501663 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9950088070 con 0x7f99500d1880 2021-02-10T11:36:03.262 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:03.244 7f994e61c700 1 --2- 172.21.2.102:0/4223501663 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f99500d1880 0x7f99501b04d0 secure :-1 s=READY pgs=2771 cs=0 l=1 rev1=1 rx=0x7f9938006700 tx=0x7f9938009b80).ready entity=mon.0 client_cookie=c2bea135579c1091 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:03.262 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:03.244 7f994d61a700 1 -- 172.21.2.102:0/4223501663 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f993800c0c0 con 0x7f99500d1880 2021-02-10T11:36:03.262 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:03.244 7f994d61a700 1 -- 172.21.2.102:0/4223501663 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f993800c340 con 0x7f99500d1880 2021-02-10T11:36:03.262 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:03.244 7f994d61a700 1 -- 172.21.2.102:0/4223501663 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f993800c860 con 0x7f99500d1880 2021-02-10T11:36:03.262 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:03.244 7f99549d3700 1 -- 172.21.2.102:0/4223501663 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f99500d1880 msgr2=0x7f99501b04d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:03.262 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:03.244 7f99549d3700 1 --2- 172.21.2.102:0/4223501663 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f99500d1880 0x7f99501b04d0 secure :-1 s=READY pgs=2771 cs=0 l=1 rev1=1 rx=0x7f9938006700 tx=0x7f9938009b80).stop 2021-02-10T11:36:03.262 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:03.244 7f99549d3700 1 -- 172.21.2.102:0/4223501663 shutdown_connections 2021-02-10T11:36:03.262 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:03.244 7f99549d3700 1 --2- 172.21.2.102:0/4223501663 >> v2:172.21.2.113:3300/0 conn(0x7f99500cff80 0x7f9950109140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:03.263 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:03.244 7f99549d3700 1 --2- 172.21.2.102:0/4223501663 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f99500d1880 0x7f99501b04d0 unknown :-1 s=CLOSED pgs=2771 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:03.263 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:03.244 7f99549d3700 1 -- 172.21.2.102:0/4223501663 >> 172.21.2.102:0/4223501663 conn(0x7f99500ce430 msgr2=0x7f99500d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:03.263 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:03.245 7f99549d3700 1 -- 172.21.2.102:0/4223501663 shutdown_connections 2021-02-10T11:36:03.263 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:03.245 7f99549d3700 1 -- 172.21.2.102:0/4223501663 wait complete. 2021-02-10T11:36:03.264 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:03.259 7f99549d3700 1 Processor -- start 2021-02-10T11:36:03.265 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:03.259 7f99549d3700 1 -- start start 2021-02-10T11:36:03.265 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:03.259 7f99549d3700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f99501a34e0 0x7f99501a38e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:03.265 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:03.259 7f99549d3700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9950083220 0x7f9950083620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:03.265 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:03.259 7f99549d3700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f99500acda0 0x7f99500ad1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:03.265 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:03.260 7f99549d3700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99500ad7c0 con 0x7f9950083220 2021-02-10T11:36:03.265 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:03.260 7f99549d3700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99500ad9e0 con 0x7f99501a34e0 2021-02-10T11:36:03.265 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:03.260 7f99549d3700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99500adc00 con 0x7f99500acda0 2021-02-10T11:36:03.265 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:03.260 7f994de1b700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9950083220 0x7f9950083620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:03.265 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:03.260 7f994de1b700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9950083220 0x7f9950083620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:37514/0 (socket says 172.21.2.102:37514) 2021-02-10T11:36:03.265 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:03.260 7f994de1b700 1 -- 172.21.2.102:0/1134502330 learned_addr learned my addr 172.21.2.102:0/1134502330 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:36:03.265 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:03.260 7f994ee1d700 1 --2- 172.21.2.102:0/1134502330 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f99500acda0 0x7f99500ad1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:03.265 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:03.260 7f994de1b700 1 -- 172.21.2.102:0/1134502330 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f99500acda0 msgr2=0x7f99500ad1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:03.266 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:03.260 7f994de1b700 1 --2- 172.21.2.102:0/1134502330 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f99500acda0 0x7f99500ad1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:03.266 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:03.260 7f994de1b700 1 -- 172.21.2.102:0/1134502330 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f99501a34e0 msgr2=0x7f99501a38e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:03.266 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:03.260 7f994e61c700 1 --2- 172.21.2.102:0/1134502330 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f99501a34e0 0x7f99501a38e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:03.266 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:03.260 7f994de1b700 1 --2- 172.21.2.102:0/1134502330 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f99501a34e0 0x7f99501a38e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:03.266 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:03.260 7f994de1b700 1 -- 172.21.2.102:0/1134502330 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99500ade20 con 0x7f9950083220 2021-02-10T11:36:03.266 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:03.260 7f994ee1d700 1 --2- 172.21.2.102:0/1134502330 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f99500acda0 0x7f99500ad1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:36:03.266 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:03.260 7f994e61c700 1 --2- 172.21.2.102:0/1134502330 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f99501a34e0 0x7f99501a38e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:36:03.266 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:03.260 7f994de1b700 1 --2- 172.21.2.102:0/1134502330 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9950083220 0x7f9950083620 secure :-1 s=READY pgs=2772 cs=0 l=1 rev1=1 rx=0x7f9940001c60 tx=0x7f994000a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:03.266 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:03.261 7f9946ffd700 1 -- 172.21.2.102:0/1134502330 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f994000acc0 con 0x7f9950083220 2021-02-10T11:36:03.266 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:03.261 7f9946ffd700 1 -- 172.21.2.102:0/1134502330 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9940004140 con 0x7f9950083220 2021-02-10T11:36:03.266 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:03.261 7f99549d3700 1 -- 172.21.2.102:0/1134502330 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99500ade20 con 0x7f9950083220 2021-02-10T11:36:03.266 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:03.261 7f99549d3700 1 -- 172.21.2.102:0/1134502330 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99500847d0 con 0x7f9950083220 2021-02-10T11:36:03.267 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:03.261 7f9946ffd700 1 -- 172.21.2.102:0/1134502330 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9940004620 con 0x7f9950083220 2021-02-10T11:36:03.267 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:03.261 7f9946ffd700 1 -- 172.21.2.102:0/1134502330 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f9940004320 con 0x7f9950083220 2021-02-10T11:36:03.267 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:03.261 7f9946ffd700 1 --2- 172.21.2.102:0/1134502330 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f9934018730 0x7f993401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:03.267 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:03.262 7f9946ffd700 1 -- 172.21.2.102:0/1134502330 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(153..153 src has 1..153) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f99400325e0 con 0x7f9950083220 2021-02-10T11:36:03.267 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:03.262 7f994e61c700 1 --2- 172.21.2.102:0/1134502330 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f9934018730 0x7f993401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:03.267 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:03.262 7f994e61c700 1 --2- 172.21.2.102:0/1134502330 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f9934018730 0x7f993401ac10 crc :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:03.294 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:03.292 7f99549d3700 1 -- 172.21.2.102:0/1134502330 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99500fe1d0 con 0x7f9950083220 2021-02-10T11:36:03.297 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:03.295 7f9946ffd700 1 -- 172.21.2.102:0/1134502330 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f9940000dd0 con 0x7f9950083220 2021-02-10T11:36:03.428 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:03.426 7f99549d3700 1 -- 172.21.2.102:0/1134502330 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f99500fe1d0 con 0x7f9950083220 2021-02-10T11:36:03.432 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:03.430 7f9946ffd700 1 -- 172.21.2.102:0/1134502330 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.2 v87) v1 ==== 184+0+0 (secure 0 0 0) 0x7f9940004300 con 0x7f9950083220 2021-02-10T11:36:03.435 INFO:teuthology.orchestra.run.gibba002.stderr:updated caps for client.2 2021-02-10T11:36:03.437 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:03.435 7f99549d3700 1 -- 172.21.2.102:0/1134502330 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f9934018730 msgr2=0x7f993401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:03.437 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:03.435 7f99549d3700 1 --2- 172.21.2.102:0/1134502330 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f9934018730 0x7f993401ac10 crc :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:03.437 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:03.435 7f99549d3700 1 -- 172.21.2.102:0/1134502330 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9950083220 msgr2=0x7f9950083620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:03.437 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:03.435 7f99549d3700 1 --2- 172.21.2.102:0/1134502330 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9950083220 0x7f9950083620 secure :-1 s=READY pgs=2772 cs=0 l=1 rev1=1 rx=0x7f9940001c60 tx=0x7f994000a8d0).stop 2021-02-10T11:36:03.437 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:03.435 7f99549d3700 1 -- 172.21.2.102:0/1134502330 shutdown_connections 2021-02-10T11:36:03.438 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:03.435 7f99549d3700 1 --2- 172.21.2.102:0/1134502330 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f99500acda0 0x7f99500ad1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:03.438 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:03.435 7f99549d3700 1 --2- 172.21.2.102:0/1134502330 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9950083220 0x7f9950083620 unknown :-1 s=CLOSED pgs=2772 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:03.438 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:03.435 7f99549d3700 1 --2- 172.21.2.102:0/1134502330 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f9934018730 0x7f993401ac10 unknown :-1 s=CLOSED pgs=1220 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:03.438 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:03.435 7f99549d3700 1 --2- 172.21.2.102:0/1134502330 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f99501a34e0 0x7f99501a38e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:03.438 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:03.435 7f99549d3700 1 -- 172.21.2.102:0/1134502330 >> 172.21.2.102:0/1134502330 conn(0x7f99500a1070 msgr2=0x7f995008eac0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:03.438 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:03.435 7f99549d3700 1 -- 172.21.2.102:0/1134502330 shutdown_connections 2021-02-10T11:36:03.438 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:03.435 7f99549d3700 1 -- 172.21.2.102:0/1134502330 wait complete. 2021-02-10T11:36:03.446 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.3 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:36:03.886 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:03.868 7f1859cdf700 1 Processor -- start 2021-02-10T11:36:03.886 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:03.868 7f1859cdf700 1 -- start start 2021-02-10T11:36:03.886 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:03.868 7f1859cdf700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f18540d1c80 0x7f18540cef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:03.887 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:03.868 7f1859cdf700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18540d0180 con 0x7f18540cf590 2021-02-10T11:36:03.887 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:03.868 7f1859cdf700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18540fe1d0 con 0x7f18540d1880 2021-02-10T11:36:03.887 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:03.868 7f1859cdf700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18541b0cc0 con 0x7f18540d1c80 2021-02-10T11:36:03.887 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:03.868 7f1853fff700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f18540cf590 0x7f18540cf990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:57980/0 (socket says 172.21.2.102:57980) 2021-02-10T11:36:03.887 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:03.868 7f1853fff700 1 -- 172.21.2.102:0/2562223174 learned_addr learned my addr 172.21.2.102:0/2562223174 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:36:03.887 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:03.869 7f1852ffd700 1 --2- 172.21.2.102:0/2562223174 >> v2:172.21.2.113:3300/0 conn(0x7f18540d1c80 0x7f18540cef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:03.887 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:03.869 7f18527fc700 1 -- 172.21.2.102:0/2562223174 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2945173829 0 0) 0x7f184c001500 con 0x7f18540cf590 2021-02-10T11:36:03.887 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:03.869 7f18527fc700 1 -- 172.21.2.102:0/2562223174 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1840002fe0 con 0x7f18540cf590 2021-02-10T11:36:03.887 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:03.869 7f1852ffd700 1 -- 172.21.2.102:0/2562223174 >> v1:172.21.2.113:6789/0 conn(0x7f18540d1880 legacy=0x7f18541b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:03.887 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:03.869 7f1852ffd700 1 -- 172.21.2.102:0/2562223174 >> v1:172.21.2.102:6789/0 conn(0x7f18540cf590 legacy=0x7f18540cf990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:03.887 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:03.869 7f1852ffd700 1 -- 172.21.2.102:0/2562223174 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1854109a70 con 0x7f18540d1c80 2021-02-10T11:36:03.887 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:03.869 7f1852ffd700 1 --2- 172.21.2.102:0/2562223174 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f18540d1c80 0x7f18540cef70 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 rx=0x7f183c006650 tx=0x7f183c009a80).ready entity=mon.1 client_cookie=d33ce7adba2c6b77 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:03.887 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:03.869 7f18527fc700 1 -- 172.21.2.102:0/2562223174 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f183c00c0c0 con 0x7f18540d1c80 2021-02-10T11:36:03.888 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:03.870 7f18527fc700 1 -- 172.21.2.102:0/2562223174 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f183c00c0c0 con 0x7f18540d1c80 2021-02-10T11:36:03.888 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:03.870 7f18527fc700 1 -- 172.21.2.102:0/2562223174 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f183c00c340 con 0x7f18540d1c80 2021-02-10T11:36:03.888 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:03.870 7f1859cdf700 1 -- 172.21.2.102:0/2562223174 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f18540d1c80 msgr2=0x7f18540cef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:03.888 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:03.870 7f1859cdf700 1 --2- 172.21.2.102:0/2562223174 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f18540d1c80 0x7f18540cef70 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 rx=0x7f183c006650 tx=0x7f183c009a80).stop 2021-02-10T11:36:03.888 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:03.870 7f1859cdf700 1 -- 172.21.2.102:0/2562223174 shutdown_connections 2021-02-10T11:36:03.888 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:03.870 7f1859cdf700 1 --2- 172.21.2.102:0/2562223174 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f18540d1c80 0x7f18540cef70 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:03.888 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:03.870 7f1859cdf700 1 -- 172.21.2.102:0/2562223174 >> 172.21.2.102:0/2562223174 conn(0x7f18540ce430 msgr2=0x7f18540d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:03.888 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:03.870 7f1859cdf700 1 -- 172.21.2.102:0/2562223174 shutdown_connections 2021-02-10T11:36:03.888 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:03.870 7f1859cdf700 1 -- 172.21.2.102:0/2562223174 wait complete. 2021-02-10T11:36:03.889 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:03.885 7f1859cdf700 1 Processor -- start 2021-02-10T11:36:03.890 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:03.885 7f1859cdf700 1 -- start start 2021-02-10T11:36:03.890 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:03.885 7f1859cdf700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1854086520 0x7f18541a1cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:03.890 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:03.885 7f1859cdf700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1854086920 0x7f1854187e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:03.890 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:03.885 7f1859cdf700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f18541a22d0 0x7f18541884a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:03.890 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:03.885 7f1859cdf700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1854099600 con 0x7f1854086520 2021-02-10T11:36:03.890 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:03.885 7f1859cdf700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18540999f0 con 0x7f1854086920 2021-02-10T11:36:03.890 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:03.885 7f1859cdf700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1854188ac0 con 0x7f18541a22d0 2021-02-10T11:36:03.890 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:03.885 7f18537fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1854086520 0x7f18541a1cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:03.890 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:03.885 7f18537fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1854086520 0x7f18541a1cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:37528/0 (socket says 172.21.2.102:37528) 2021-02-10T11:36:03.890 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:03.885 7f18537fe700 1 -- 172.21.2.102:0/1367116938 learned_addr learned my addr 172.21.2.102:0/1367116938 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:36:03.890 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:03.886 7f18537fe700 1 -- 172.21.2.102:0/1367116938 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f18541a22d0 msgr2=0x7f18541884a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:03.891 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:03.886 7f18537fe700 1 --2- 172.21.2.102:0/1367116938 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f18541a22d0 0x7f18541884a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:03.891 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:03.886 7f18537fe700 1 -- 172.21.2.102:0/1367116938 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1854086920 msgr2=0x7f1854187e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:03.891 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:03.886 7f18537fe700 1 --2- 172.21.2.102:0/1367116938 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1854086920 0x7f1854187e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:03.891 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:03.886 7f18537fe700 1 -- 172.21.2.102:0/1367116938 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1854188ce0 con 0x7f1854086520 2021-02-10T11:36:03.891 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:03.886 7f18537fe700 1 --2- 172.21.2.102:0/1367116938 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1854086520 0x7f18541a1cb0 secure :-1 s=READY pgs=2774 cs=0 l=1 rev1=1 rx=0x7f184400b530 tx=0x7f184400b8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:03.891 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:03.886 7f184bfff700 1 -- 172.21.2.102:0/1367116938 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f184400bcc0 con 0x7f1854086520 2021-02-10T11:36:03.891 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:03.886 7f184bfff700 1 -- 172.21.2.102:0/1367116938 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1844000c60 con 0x7f1854086520 2021-02-10T11:36:03.891 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:03.886 7f184bfff700 1 -- 172.21.2.102:0/1367116938 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f18440062a0 con 0x7f1854086520 2021-02-10T11:36:03.891 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:03.886 7f1859cdf700 1 -- 172.21.2.102:0/1367116938 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1854188ce0 con 0x7f1854086520 2021-02-10T11:36:03.891 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:03.886 7f1859cdf700 1 -- 172.21.2.102:0/1367116938 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1854189270 con 0x7f1854086520 2021-02-10T11:36:03.891 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:03.887 7f184bfff700 1 -- 172.21.2.102:0/1367116938 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1844006020 con 0x7f1854086520 2021-02-10T11:36:03.891 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:03.887 7f184bfff700 1 --2- 172.21.2.102:0/1367116938 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f18380186c0 0x7f183801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:03.891 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:03.887 7f184bfff700 1 -- 172.21.2.102:0/1367116938 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(153..153 src has 1..153) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f1844032590 con 0x7f1854086520 2021-02-10T11:36:03.892 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:03.887 7f1852ffd700 1 --2- 172.21.2.102:0/1367116938 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f18380186c0 0x7f183801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:03.892 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:03.887 7f1852ffd700 1 --2- 172.21.2.102:0/1367116938 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f18380186c0 0x7f183801abc0 crc :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:03.920 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:03.918 7f1859cdf700 1 -- 172.21.2.102:0/1367116938 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18540fe1d0 con 0x7f1854086520 2021-02-10T11:36:03.922 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:03.920 7f184bfff700 1 -- 172.21.2.102:0/1367116938 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f1844006230 con 0x7f1854086520 2021-02-10T11:36:04.052 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:04.050 7f1859cdf700 1 -- 172.21.2.102:0/1367116938 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.3", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f18540999a0 con 0x7f1854086520 2021-02-10T11:36:04.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:04.053 7f184bfff700 1 -- 172.21.2.102:0/1367116938 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.3", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.3 v88) v1 ==== 184+0+0 (secure 0 0 0) 0x7f1844000940 con 0x7f1854086520 2021-02-10T11:36:04.059 INFO:teuthology.orchestra.run.gibba002.stderr:updated caps for client.3 2021-02-10T11:36:04.061 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:04.059 7f1859cdf700 1 -- 172.21.2.102:0/1367116938 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f18380186c0 msgr2=0x7f183801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:04.061 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:04.059 7f1859cdf700 1 --2- 172.21.2.102:0/1367116938 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f18380186c0 0x7f183801abc0 crc :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:04.061 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:04.059 7f1859cdf700 1 -- 172.21.2.102:0/1367116938 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1854086520 msgr2=0x7f18541a1cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:04.061 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:04.059 7f1859cdf700 1 --2- 172.21.2.102:0/1367116938 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1854086520 0x7f18541a1cb0 secure :-1 s=READY pgs=2774 cs=0 l=1 rev1=1 rx=0x7f184400b530 tx=0x7f184400b8d0).stop 2021-02-10T11:36:04.061 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:04.059 7f1859cdf700 1 -- 172.21.2.102:0/1367116938 shutdown_connections 2021-02-10T11:36:04.061 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:04.059 7f1859cdf700 1 --2- 172.21.2.102:0/1367116938 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f18541a22d0 0x7f18541884a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:04.061 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:04.059 7f1859cdf700 1 --2- 172.21.2.102:0/1367116938 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f18380186c0 0x7f183801abc0 unknown :-1 s=CLOSED pgs=1221 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:04.061 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:04.059 7f1859cdf700 1 --2- 172.21.2.102:0/1367116938 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1854086920 0x7f1854187e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:04.062 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:04.059 7f1859cdf700 1 --2- 172.21.2.102:0/1367116938 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1854086520 0x7f18541a1cb0 unknown :-1 s=CLOSED pgs=2774 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:04.062 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:04.059 7f1859cdf700 1 -- 172.21.2.102:0/1367116938 >> 172.21.2.102:0/1367116938 conn(0x7f185419f880 msgr2=0x7f185408f890 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:04.062 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:04.059 7f1859cdf700 1 -- 172.21.2.102:0/1367116938 shutdown_connections 2021-02-10T11:36:04.062 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:04.059 7f1859cdf700 1 -- 172.21.2.102:0/1367116938 wait complete. 2021-02-10T11:36:04.069 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:36:04.498 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:04.480 7f23d4452700 1 Processor -- start 2021-02-10T11:36:04.498 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:04.480 7f23d4452700 1 -- start start 2021-02-10T11:36:04.498 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:04.480 7f23d4452700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f23cc1b0470 0x7f23cc1b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:04.499 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:04.480 7f23d4452700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23cc0f0b50 con 0x7f23cc0ef610 2021-02-10T11:36:04.499 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:04.480 7f23d4452700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23cc0f24e0 con 0x7f23cc0f20e0 2021-02-10T11:36:04.499 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:04.480 7f23d4452700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23cc0e5ea0 con 0x7f23cc1b0470 2021-02-10T11:36:04.499 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:04.481 7f23d226e700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f23cc1b0470 0x7f23cc1b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:04.499 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:04.481 7f23d1a6d700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f23cc0ef610 0x7f23cc0efa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:57990/0 (socket says 172.21.2.102:57990) 2021-02-10T11:36:04.499 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:04.481 7f23d1a6d700 1 -- 172.21.2.102:0/1172074771 learned_addr learned my addr 172.21.2.102:0/1172074771 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:36:04.499 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:04.481 7f23d226e700 1 -- 172.21.2.102:0/1172074771 >> v1:172.21.2.113:6789/0 conn(0x7f23cc0f20e0 legacy=0x7f23cc0f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:04.499 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:04.481 7f23d226e700 1 -- 172.21.2.102:0/1172074771 >> v1:172.21.2.102:6789/0 conn(0x7f23cc0ef610 legacy=0x7f23cc0efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:04.499 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:04.481 7f23d226e700 1 -- 172.21.2.102:0/1172074771 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23cc0e52e0 con 0x7f23cc1b0470 2021-02-10T11:36:04.499 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:04.481 7f23d226e700 1 --2- 172.21.2.102:0/1172074771 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f23cc1b0470 0x7f23cc1b0870 secure :-1 s=READY pgs=2776 cs=0 l=1 rev1=1 rx=0x7f23c0006650 tx=0x7f23c0009a80).ready entity=mon.0 client_cookie=5d7cb76b57ea6e11 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:04.499 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:04.481 7f23d126c700 1 -- 172.21.2.102:0/1172074771 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f23c00009b0 con 0x7f23cc1b0470 2021-02-10T11:36:04.499 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:04.481 7f23d126c700 1 -- 172.21.2.102:0/1172074771 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23c000c0c0 con 0x7f23cc1b0470 2021-02-10T11:36:04.500 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:04.481 7f23d126c700 1 -- 172.21.2.102:0/1172074771 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f23c000c510 con 0x7f23cc1b0470 2021-02-10T11:36:04.500 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:04.481 7f23d4452700 1 -- 172.21.2.102:0/1172074771 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f23cc1b0470 msgr2=0x7f23cc1b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:04.500 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:04.481 7f23d4452700 1 --2- 172.21.2.102:0/1172074771 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f23cc1b0470 0x7f23cc1b0870 secure :-1 s=READY pgs=2776 cs=0 l=1 rev1=1 rx=0x7f23c0006650 tx=0x7f23c0009a80).stop 2021-02-10T11:36:04.500 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:04.482 7f23d4452700 1 -- 172.21.2.102:0/1172074771 shutdown_connections 2021-02-10T11:36:04.500 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:04.482 7f23d4452700 1 --2- 172.21.2.102:0/1172074771 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f23cc1b0470 0x7f23cc1b0870 unknown :-1 s=CLOSED pgs=2776 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:04.500 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:04.482 7f23d4452700 1 -- 172.21.2.102:0/1172074771 >> 172.21.2.102:0/1172074771 conn(0x7f23cc0ce3e0 msgr2=0x7f23cc0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:04.500 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:04.482 7f23d4452700 1 -- 172.21.2.102:0/1172074771 shutdown_connections 2021-02-10T11:36:04.500 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:04.482 7f23d4452700 1 -- 172.21.2.102:0/1172074771 wait complete. 2021-02-10T11:36:04.501 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:04.497 7f23d4452700 1 Processor -- start 2021-02-10T11:36:04.501 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:04.497 7f23d4452700 1 -- start start 2021-02-10T11:36:04.502 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:04.497 7f23d4452700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f23cc097170 0x7f23cc0a71e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:04.502 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:04.497 7f23d4452700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f23cc0961b0 0x7f23cc0965b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:04.502 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:04.497 7f23d4452700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f23cc096bd0 0x7f23cc0a7e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:04.502 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:04.497 7f23d4452700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23cc0a8470 con 0x7f23cc0961b0 2021-02-10T11:36:04.502 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:04.497 7f23d4452700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23cc0a8650 con 0x7f23cc096bd0 2021-02-10T11:36:04.502 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:04.497 7f23d4452700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23cc0a8870 con 0x7f23cc097170 2021-02-10T11:36:04.502 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:04.497 7f23d1a6d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f23cc0961b0 0x7f23cc0965b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:04.502 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:04.497 7f23d1a6d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f23cc0961b0 0x7f23cc0965b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:37540/0 (socket says 172.21.2.102:37540) 2021-02-10T11:36:04.502 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:04.497 7f23d1a6d700 1 -- 172.21.2.102:0/333408873 learned_addr learned my addr 172.21.2.102:0/333408873 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:36:04.502 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:04.498 7f23d1a6d700 1 -- 172.21.2.102:0/333408873 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f23cc097170 msgr2=0x7f23cc0a71e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:04.502 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:04.498 7f23d1a6d700 1 --2- 172.21.2.102:0/333408873 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f23cc097170 0x7f23cc0a71e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:04.502 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:04.498 7f23d1a6d700 1 -- 172.21.2.102:0/333408873 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f23cc096bd0 msgr2=0x7f23cc0a7e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:04.503 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:04.498 7f23d1a6d700 1 --2- 172.21.2.102:0/333408873 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f23cc096bd0 0x7f23cc0a7e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:04.503 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:04.498 7f23d1a6d700 1 -- 172.21.2.102:0/333408873 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23cc0a8a90 con 0x7f23cc0961b0 2021-02-10T11:36:04.503 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:04.498 7f23d1a6d700 1 --2- 172.21.2.102:0/333408873 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f23cc0961b0 0x7f23cc0965b0 secure :-1 s=READY pgs=2777 cs=0 l=1 rev1=1 rx=0x7f23c8001c60 tx=0x7f23c800a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:04.503 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:04.498 7f23beffd700 1 -- 172.21.2.102:0/333408873 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f23c800acc0 con 0x7f23cc0961b0 2021-02-10T11:36:04.503 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:04.498 7f23beffd700 1 -- 172.21.2.102:0/333408873 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23c8004140 con 0x7f23cc0961b0 2021-02-10T11:36:04.503 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:04.498 7f23d4452700 1 -- 172.21.2.102:0/333408873 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23cc0a8a90 con 0x7f23cc0961b0 2021-02-10T11:36:04.503 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:04.498 7f23beffd700 1 -- 172.21.2.102:0/333408873 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f23c8004620 con 0x7f23cc0961b0 2021-02-10T11:36:04.503 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:04.498 7f23d4452700 1 -- 172.21.2.102:0/333408873 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23cc0a9020 con 0x7f23cc0961b0 2021-02-10T11:36:04.503 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:04.499 7f23beffd700 1 -- 172.21.2.102:0/333408873 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f23c8004320 con 0x7f23cc0961b0 2021-02-10T11:36:04.503 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:04.499 7f23beffd700 1 --2- 172.21.2.102:0/333408873 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f23b4018730 0x7f23b401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:04.503 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:04.499 7f23beffd700 1 -- 172.21.2.102:0/333408873 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(153..153 src has 1..153) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f23c8032630 con 0x7f23cc0961b0 2021-02-10T11:36:04.503 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:04.499 7f23d226e700 1 --2- 172.21.2.102:0/333408873 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f23b4018730 0x7f23b401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:04.503 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:04.500 7f23d226e700 1 --2- 172.21.2.102:0/333408873 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f23b4018730 0x7f23b401ac10 crc :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:04.532 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:04.530 7f23d4452700 1 -- 172.21.2.102:0/333408873 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23cc0a7920 con 0x7f23cc0961b0 2021-02-10T11:36:04.535 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:04.533 7f23beffd700 1 -- 172.21.2.102:0/333408873 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f23c8004350 con 0x7f23cc0961b0 2021-02-10T11:36:04.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:04.662 7f23d4452700 1 -- 172.21.2.102:0/333408873 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f23cc0a7920 con 0x7f23cc0961b0 2021-02-10T11:36:04.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:04.662 7f23beffd700 1 -- 172.21.2.102:0/333408873 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 132 v132) v1 ==== 95+0+2107 (secure 0 0 0) 0x7f23c8004140 con 0x7f23cc0961b0 2021-02-10T11:36:04.665 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:36:04.665 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":132,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7004,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.102:6835/3280534775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6834","nonce":3280534775},{"type":"v1","addr":"172.21.2.102:6835","nonce":3280534775}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":128}],"filesystems":[{"mdsmap":{"epoch":132,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:35:58.285904","modified":"2021-02-10 11:36:00.292711","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7001},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7001":{"gid":7001,"name":"b","rank":0,"incarnation":129,"state":"up:active","state_seq":3,"addr":"172.21.2.113:6833/1983258405","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6832","nonce":1983258405},{"type":"v1","addr":"172.21.2.113:6833","nonce":1983258405}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[33],"metadata_pool":32,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":16}]} 2021-02-10T11:36:04.665 INFO:teuthology.orchestra.run.gibba002.stderr:dumped fsmap epoch 132 2021-02-10T11:36:04.666 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:04.664 7f23d4452700 1 -- 172.21.2.102:0/333408873 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f23b4018730 msgr2=0x7f23b401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:04.666 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:04.664 7f23d4452700 1 --2- 172.21.2.102:0/333408873 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f23b4018730 0x7f23b401ac10 crc :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:04.666 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:04.664 7f23d4452700 1 -- 172.21.2.102:0/333408873 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f23cc0961b0 msgr2=0x7f23cc0965b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:04.666 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:04.664 7f23d4452700 1 --2- 172.21.2.102:0/333408873 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f23cc0961b0 0x7f23cc0965b0 secure :-1 s=READY pgs=2777 cs=0 l=1 rev1=1 rx=0x7f23c8001c60 tx=0x7f23c800a8d0).stop 2021-02-10T11:36:04.667 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:04.664 7f23d4452700 1 -- 172.21.2.102:0/333408873 shutdown_connections 2021-02-10T11:36:04.667 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:04.664 7f23d4452700 1 --2- 172.21.2.102:0/333408873 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f23b4018730 0x7f23b401ac10 unknown :-1 s=CLOSED pgs=1222 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:04.667 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:04.664 7f23d4452700 1 --2- 172.21.2.102:0/333408873 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f23cc096bd0 0x7f23cc0a7e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:04.667 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:04.664 7f23d4452700 1 --2- 172.21.2.102:0/333408873 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f23cc0961b0 0x7f23cc0965b0 unknown :-1 s=CLOSED pgs=2777 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:04.667 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:04.664 7f23d4452700 1 --2- 172.21.2.102:0/333408873 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f23cc097170 0x7f23cc0a71e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:04.667 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:04.664 7f23d4452700 1 -- 172.21.2.102:0/333408873 >> 172.21.2.102:0/333408873 conn(0x7f23cc0c3bd0 msgr2=0x7f23cc089210 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:04.667 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:04.664 7f23d4452700 1 -- 172.21.2.102:0/333408873 shutdown_connections 2021-02-10T11:36:04.667 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:04.664 7f23d4452700 1 -- 172.21.2.102:0/333408873 wait complete. 2021-02-10T11:36:04.876 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 132, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:35:58.285904', 'modified': '2021-02-10 11:36:00.292711', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7001}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7001': {'gid': 7001, 'name': 'b', 'rank': 0, 'incarnation': 129, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.113:6833/1983258405', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.113:6832', 'nonce': 1983258405}, {'type': 'v1', 'addr': '172.21.2.113:6833', 'nonce': 1983258405}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [33], 'metadata_pool': 32, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:36:04.876 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:36:04.877 DEBUG:teuthology.orchestra.run.gibba013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:36:05.276 INFO:teuthology.orchestra.run.gibba013.stdout:{ 2021-02-10T11:36:05.276 INFO:teuthology.orchestra.run.gibba013.stdout: "cluster_fsid": "68db30ea-dbce-4aef-a6df-f01c6fea0513", 2021-02-10T11:36:05.276 INFO:teuthology.orchestra.run.gibba013.stdout: "whoami": 0, 2021-02-10T11:36:05.276 INFO:teuthology.orchestra.run.gibba013.stdout: "id": 7001, 2021-02-10T11:36:05.276 INFO:teuthology.orchestra.run.gibba013.stdout: "want_state": "up:active", 2021-02-10T11:36:05.277 INFO:teuthology.orchestra.run.gibba013.stdout: "state": "up:active", 2021-02-10T11:36:05.277 INFO:teuthology.orchestra.run.gibba013.stdout: "rank_uptime": 6.9861410739999998, 2021-02-10T11:36:05.277 INFO:teuthology.orchestra.run.gibba013.stdout: "mdsmap_epoch": 132, 2021-02-10T11:36:05.277 INFO:teuthology.orchestra.run.gibba013.stdout: "osdmap_epoch": 153, 2021-02-10T11:36:05.277 INFO:teuthology.orchestra.run.gibba013.stdout: "osdmap_epoch_barrier": 153, 2021-02-10T11:36:05.277 INFO:teuthology.orchestra.run.gibba013.stdout: "uptime": 12.632862186000001 2021-02-10T11:36:05.277 INFO:teuthology.orchestra.run.gibba013.stdout:} 2021-02-10T11:36:05.280 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "68db30ea-dbce-4aef-a6df-f01c6fea0513", "whoami": 0, "id": 7001, "want_state": "up:active", "state": "up:active", "rank_uptime": 6.9861410739999998, "mdsmap_epoch": 132, "osdmap_epoch": 153, "osdmap_epoch_barrier": 153, "uptime": 12.632862186000001 } 2021-02-10T11:36:05.280 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:36:05.711 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:05.691 7f03472be700 1 Processor -- start 2021-02-10T11:36:05.711 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:05.691 7f03472be700 1 -- start start 2021-02-10T11:36:05.711 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:05.692 7f03472be700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f03401b0470 0x7f03401b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:05.711 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:05.692 7f03472be700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03400f0b50 con 0x7f03400ef610 2021-02-10T11:36:05.712 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:05.692 7f03472be700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03400f24e0 con 0x7f03400f20e0 2021-02-10T11:36:05.712 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:05.692 7f03472be700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03400e5ea0 con 0x7f03401b0470 2021-02-10T11:36:05.712 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:05.692 7f03448d9700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f03400ef610 0x7f03400efa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:58004/0 (socket says 172.21.2.102:58004) 2021-02-10T11:36:05.712 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:05.692 7f03448d9700 1 -- 172.21.2.102:0/3870314370 learned_addr learned my addr 172.21.2.102:0/3870314370 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:36:05.712 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:05.692 7f03450da700 1 --2- 172.21.2.102:0/3870314370 >> v2:172.21.2.113:3300/0 conn(0x7f03401b0470 0x7f03401b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:05.712 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:05.693 7f0333fff700 1 -- 172.21.2.102:0/3870314370 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4042367460 0 0) 0x7f033c0009b0 con 0x7f03400ef610 2021-02-10T11:36:05.712 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:05.693 7f0333fff700 1 -- 172.21.2.102:0/3870314370 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f032c002fe0 con 0x7f03400ef610 2021-02-10T11:36:05.712 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:05.693 7f0333fff700 1 -- 172.21.2.102:0/3870314370 <== mon.1 v1:172.21.2.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3219623570 0 0) 0x7f0338001400 con 0x7f03400f20e0 2021-02-10T11:36:05.712 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:05.693 7f0333fff700 1 -- 172.21.2.102:0/3870314370 --> v1:172.21.2.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f032c003210 con 0x7f03400f20e0 2021-02-10T11:36:05.712 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:05.693 7f0333fff700 1 -- 172.21.2.102:0/3870314370 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1006042291 0 0) 0x7f033c0010b0 con 0x7f03400ef610 2021-02-10T11:36:05.712 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:05.693 7f0333fff700 1 -- 172.21.2.102:0/3870314370 >> v2:172.21.2.113:3300/0 conn(0x7f03401b0470 msgr2=0x7f03401b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:05.712 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:05.693 7f0333fff700 1 --2- 172.21.2.102:0/3870314370 >> v2:172.21.2.113:3300/0 conn(0x7f03401b0470 0x7f03401b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:05.712 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:05.693 7f0333fff700 1 -- 172.21.2.102:0/3870314370 >> v1:172.21.2.113:6789/0 conn(0x7f03400f20e0 legacy=0x7f03400f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:05.713 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:05.693 7f0333fff700 1 -- 172.21.2.102:0/3870314370 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03400e52e0 con 0x7f03400ef610 2021-02-10T11:36:05.713 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:05.693 7f0333fff700 1 -- 172.21.2.102:0/3870314370 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7f033c0015b0 con 0x7f03400ef610 2021-02-10T11:36:05.713 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:05.693 7f0333fff700 1 -- 172.21.2.102:0/3870314370 >> v1:172.21.2.102:6789/0 conn(0x7f03400ef610 legacy=0x7f03400efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:05.713 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:05.693 7f0333fff700 1 --2- 172.21.2.102:0/3870314370 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f032c003c40 0x7f032c006050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:05.713 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:05.693 7f0333fff700 1 --2- 172.21.2.102:0/3870314370 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f032c006870 0x7f032c008d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:05.713 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:05.693 7f0333fff700 1 --2- 172.21.2.102:0/3870314370 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f032c009600 0x7f032c00bae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:05.713 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:05.693 7f0333fff700 1 -- 172.21.2.102:0/3870314370 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f032c00c330 con 0x7f032c009600 2021-02-10T11:36:05.713 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:05.693 7f0333fff700 1 -- 172.21.2.102:0/3870314370 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f032c050520 con 0x7f032c003c40 2021-02-10T11:36:05.713 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:05.693 7f0333fff700 1 -- 172.21.2.102:0/3870314370 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f032c050740 con 0x7f032c006870 2021-02-10T11:36:05.713 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:05.693 7f03458db700 1 --2- 172.21.2.102:0/3870314370 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f032c009600 0x7f032c00bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:05.713 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:05.693 7f03450da700 1 --2- 172.21.2.102:0/3870314370 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f032c003c40 0x7f032c006050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:05.713 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:05.693 7f03448d9700 1 --2- 172.21.2.102:0/3870314370 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f032c006870 0x7f032c008d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:05.713 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:05.693 7f03458db700 1 -- 172.21.2.102:0/3870314370 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f032c006870 msgr2=0x7f032c008d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:05.714 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:05.693 7f03458db700 1 --2- 172.21.2.102:0/3870314370 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f032c006870 0x7f032c008d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:05.714 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:05.693 7f03458db700 1 -- 172.21.2.102:0/3870314370 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f032c003c40 msgr2=0x7f032c006050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:05.714 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:05.693 7f03458db700 1 --2- 172.21.2.102:0/3870314370 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f032c003c40 0x7f032c006050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:05.714 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:05.693 7f03458db700 1 -- 172.21.2.102:0/3870314370 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f032c0509e0 con 0x7f032c009600 2021-02-10T11:36:05.714 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:05.694 7f03458db700 1 -- 172.21.2.102:0/3870314370 reap_dead start 2021-02-10T11:36:05.714 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:05.694 7f03458db700 1 --2- 172.21.2.102:0/3870314370 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f032c009600 0x7f032c00bae0 secure :-1 s=READY pgs=2779 cs=0 l=1 rev1=1 rx=0x7f03380013c0 tx=0x7f0338009e20).ready entity=mon.0 client_cookie=590fc2694f0b90f3 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:05.714 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:05.694 7f0333fff700 1 -- 172.21.2.102:0/3870314370 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0338001400 con 0x7f032c009600 2021-02-10T11:36:05.714 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:05.694 7f0333fff700 1 -- 172.21.2.102:0/3870314370 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0338003230 con 0x7f032c009600 2021-02-10T11:36:05.714 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:05.694 7f0333fff700 1 -- 172.21.2.102:0/3870314370 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0338001680 con 0x7f032c009600 2021-02-10T11:36:05.714 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:05.694 7f03472be700 1 -- 172.21.2.102:0/3870314370 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f032c009600 msgr2=0x7f032c00bae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:05.714 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:05.694 7f03472be700 1 --2- 172.21.2.102:0/3870314370 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f032c009600 0x7f032c00bae0 secure :-1 s=READY pgs=2779 cs=0 l=1 rev1=1 rx=0x7f03380013c0 tx=0x7f0338009e20).stop 2021-02-10T11:36:05.714 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:05.694 7f03472be700 1 -- 172.21.2.102:0/3870314370 shutdown_connections 2021-02-10T11:36:05.714 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:05.694 7f03472be700 1 --2- 172.21.2.102:0/3870314370 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f032c009600 0x7f032c00bae0 unknown :-1 s=CLOSED pgs=2779 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:05.715 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:05.694 7f03472be700 1 -- 172.21.2.102:0/3870314370 >> 172.21.2.102:0/3870314370 conn(0x7f03400ce3e0 msgr2=0x7f03400e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:05.715 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:05.694 7f03472be700 1 -- 172.21.2.102:0/3870314370 shutdown_connections 2021-02-10T11:36:05.715 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:05.695 7f03472be700 1 -- 172.21.2.102:0/3870314370 wait complete. 2021-02-10T11:36:05.715 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:05.709 7f03472be700 1 Processor -- start 2021-02-10T11:36:05.715 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:05.709 7f03472be700 1 -- start start 2021-02-10T11:36:05.716 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:05.710 7f03472be700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0340108ce0 0x7f034009da50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:05.716 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:05.710 7f03472be700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f03401090e0 0x7f034009e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:05.716 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:05.710 7f03472be700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f034009eee0 0x7f034009f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:05.716 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:05.710 7f03472be700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03401094e0 con 0x7f0340108ce0 2021-02-10T11:36:05.716 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:05.710 7f03472be700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f034009e960 con 0x7f034009eee0 2021-02-10T11:36:05.716 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:05.710 7f03472be700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f034009e690 con 0x7f03401090e0 2021-02-10T11:36:05.717 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:05.710 7f03450da700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0340108ce0 0x7f034009da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:05.717 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:05.710 7f03450da700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0340108ce0 0x7f034009da50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:37558/0 (socket says 172.21.2.102:37558) 2021-02-10T11:36:05.717 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:05.710 7f03450da700 1 -- 172.21.2.102:0/490181392 learned_addr learned my addr 172.21.2.102:0/490181392 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:36:05.717 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:05.710 7f03450da700 1 -- 172.21.2.102:0/490181392 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f03401090e0 msgr2=0x7f034009e070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:05.717 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:05.710 7f03450da700 1 --2- 172.21.2.102:0/490181392 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f03401090e0 0x7f034009e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:05.717 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:05.710 7f03450da700 1 -- 172.21.2.102:0/490181392 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f034009eee0 msgr2=0x7f034009f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:05.717 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:05.710 7f03450da700 1 --2- 172.21.2.102:0/490181392 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f034009eee0 0x7f034009f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:05.717 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:05.710 7f03450da700 1 -- 172.21.2.102:0/490181392 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f034009f900 con 0x7f0340108ce0 2021-02-10T11:36:05.717 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:05.710 7f03450da700 1 --2- 172.21.2.102:0/490181392 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0340108ce0 0x7f034009da50 secure :-1 s=READY pgs=2780 cs=0 l=1 rev1=1 rx=0x7f0334006f60 tx=0x7f033400b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:05.717 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:05.711 7f0331ffb700 1 -- 172.21.2.102:0/490181392 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f033400bd60 con 0x7f0340108ce0 2021-02-10T11:36:05.717 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:05.711 7f0331ffb700 1 -- 172.21.2.102:0/490181392 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0334000c90 con 0x7f0340108ce0 2021-02-10T11:36:05.717 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:05.711 7f0331ffb700 1 -- 172.21.2.102:0/490181392 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0334002220 con 0x7f0340108ce0 2021-02-10T11:36:05.717 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:05.711 7f03472be700 1 -- 172.21.2.102:0/490181392 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f034009f900 con 0x7f0340108ce0 2021-02-10T11:36:05.718 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:05.711 7f03472be700 1 -- 172.21.2.102:0/490181392 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f034009fe90 con 0x7f0340108ce0 2021-02-10T11:36:05.718 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:05.711 7f0331ffb700 1 -- 172.21.2.102:0/490181392 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f0334006b00 con 0x7f0340108ce0 2021-02-10T11:36:05.718 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:05.711 7f0331ffb700 1 --2- 172.21.2.102:0/490181392 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0320018730 0x7f032001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:05.718 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:05.711 7f0331ffb700 1 -- 172.21.2.102:0/490181392 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(153..153 src has 1..153) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f0334032650 con 0x7f0340108ce0 2021-02-10T11:36:05.718 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:05.712 7f03448d9700 1 --2- 172.21.2.102:0/490181392 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0320018730 0x7f032001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:05.718 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:05.712 7f03448d9700 1 --2- 172.21.2.102:0/490181392 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0320018730 0x7f032001abc0 crc :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:05.744 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:05.742 7f03472be700 1 -- 172.21.2.102:0/490181392 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f034009e960 con 0x7f0340108ce0 2021-02-10T11:36:05.746 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:05.744 7f0331ffb700 1 -- 172.21.2.102:0/490181392 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f0334000940 con 0x7f0340108ce0 2021-02-10T11:36:05.872 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:05.870 7f03472be700 1 -- 172.21.2.102:0/490181392 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f034009e960 con 0x7f0340108ce0 2021-02-10T11:36:05.872 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:05.870 7f0331ffb700 1 -- 172.21.2.102:0/490181392 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v153) v1 ==== 74+0+14007 (secure 0 0 0) 0x7f0334006b00 con 0x7f0340108ce0 2021-02-10T11:36:05.873 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:36:05.873 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":153,"fsid":"68db30ea-dbce-4aef-a6df-f01c6fea0513","created":"2021-02-10 11:03:06.331557","modified":"2021-02-10 11:35:59.288212","last_up_change":"2021-02-10 11:03:32.690562","last_in_change":"2021-02-10 11:03:32.690562","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":33,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:03:37.322519","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":32,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:35:56.251556","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"152","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":33,"pool_name":"cephfs_data","create_time":"2021-02-10 11:35:57.328218","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"153","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"be691b4d-a039-408a-af80-9d53f02b6f1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":150,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6818","nonce":36088},{"type":"v1","addr":"172.21.2.102:6820","nonce":36088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6822","nonce":36088},{"type":"v1","addr":"172.21.2.102:6824","nonce":36088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6830","nonce":36088},{"type":"v1","addr":"172.21.2.102:6832","nonce":36088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6826","nonce":36088},{"type":"v1","addr":"172.21.2.102:6828","nonce":36088}]},"public_addr":"172.21.2.102:6820/36088","cluster_addr":"172.21.2.102:6824/36088","heartbeat_back_addr":"172.21.2.102:6832/36088","heartbeat_front_addr":"172.21.2.102:6828/36088","state":["exists","up"]},{"osd":1,"uuid":"64e3ccc8-9c69-4d7c-865e-af8e83822635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6819","nonce":36087},{"type":"v1","addr":"172.21.2.102:6821","nonce":36087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6823","nonce":36087},{"type":"v1","addr":"172.21.2.102:6825","nonce":36087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6831","nonce":36087},{"type":"v1","addr":"172.21.2.102:6833","nonce":36087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6827","nonce":36087},{"type":"v1","addr":"172.21.2.102:6829","nonce":36087}]},"public_addr":"172.21.2.102:6821/36087","cluster_addr":"172.21.2.102:6825/36087","heartbeat_back_addr":"172.21.2.102:6833/36087","heartbeat_front_addr":"172.21.2.102:6829/36087","state":["exists","up"]},{"osd":2,"uuid":"153700af-747f-481f-bdf4-d6072d3a842f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6810","nonce":36089},{"type":"v1","addr":"172.21.2.102:6811","nonce":36089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6812","nonce":36089},{"type":"v1","addr":"172.21.2.102:6813","nonce":36089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6816","nonce":36089},{"type":"v1","addr":"172.21.2.102:6817","nonce":36089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6814","nonce":36089},{"type":"v1","addr":"172.21.2.102:6815","nonce":36089}]},"public_addr":"172.21.2.102:6811/36089","cluster_addr":"172.21.2.102:6813/36089","heartbeat_back_addr":"172.21.2.102:6817/36089","heartbeat_front_addr":"172.21.2.102:6815/36089","state":["exists","up"]},{"osd":3,"uuid":"cbc678f4-83a9-45e6-8830-2f7c232b1ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6802","nonce":36090},{"type":"v1","addr":"172.21.2.102:6803","nonce":36090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6804","nonce":36090},{"type":"v1","addr":"172.21.2.102:6805","nonce":36090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6808","nonce":36090},{"type":"v1","addr":"172.21.2.102:6809","nonce":36090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6806","nonce":36090},{"type":"v1","addr":"172.21.2.102:6807","nonce":36090}]},"public_addr":"172.21.2.102:6803/36090","cluster_addr":"172.21.2.102:6805/36090","heartbeat_back_addr":"172.21.2.102:6809/36090","heartbeat_front_addr":"172.21.2.102:6807/36090","state":["exists","up"]},{"osd":4,"uuid":"c01e4008-b258-4f10-afd2-9e7f37214351","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6808","nonce":36368},{"type":"v1","addr":"172.21.2.113:6811","nonce":36368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6814","nonce":36368},{"type":"v1","addr":"172.21.2.113:6818","nonce":36368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6826","nonce":36368},{"type":"v1","addr":"172.21.2.113:6828","nonce":36368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6820","nonce":36368},{"type":"v1","addr":"172.21.2.113:6824","nonce":36368}]},"public_addr":"172.21.2.113:6811/36368","cluster_addr":"172.21.2.113:6818/36368","heartbeat_back_addr":"172.21.2.113:6828/36368","heartbeat_front_addr":"172.21.2.113:6824/36368","state":["exists","up"]},{"osd":5,"uuid":"551f6007-daf8-4a98-8ee0-c2b16bcccb5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6802","nonce":36366},{"type":"v1","addr":"172.21.2.113:6805","nonce":36366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6807","nonce":36366},{"type":"v1","addr":"172.21.2.113:6810","nonce":36366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6819","nonce":36366},{"type":"v1","addr":"172.21.2.113:6822","nonce":36366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6812","nonce":36366},{"type":"v1","addr":"172.21.2.113:6815","nonce":36366}]},"public_addr":"172.21.2.113:6805/36366","cluster_addr":"172.21.2.113:6810/36366","heartbeat_back_addr":"172.21.2.113:6822/36366","heartbeat_front_addr":"172.21.2.113:6815/36366","state":["exists","up"]},{"osd":6,"uuid":"bf67c040-0e50-48be-b102-c3943bc9ebef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":150,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6800","nonce":36367},{"type":"v1","addr":"172.21.2.113:6801","nonce":36367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6803","nonce":36367},{"type":"v1","addr":"172.21.2.113:6804","nonce":36367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6813","nonce":36367},{"type":"v1","addr":"172.21.2.113:6816","nonce":36367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6806","nonce":36367},{"type":"v1","addr":"172.21.2.113:6809","nonce":36367}]},"public_addr":"172.21.2.113:6801/36367","cluster_addr":"172.21.2.113:6804/36367","heartbeat_back_addr":"172.21.2.113:6816/36367","heartbeat_front_addr":"172.21.2.113:6809/36367","state":["exists","up"]},{"osd":7,"uuid":"f5461f80-018d-428f-be89-2eb2bcbe7bb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6817","nonce":36365},{"type":"v1","addr":"172.21.2.113:6821","nonce":36365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6823","nonce":36365},{"type":"v1","addr":"172.21.2.113:6825","nonce":36365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6830","nonce":36365},{"type":"v1","addr":"172.21.2.113:6831","nonce":36365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6827","nonce":36365},{"type":"v1","addr":"172.21.2.113:6829","nonce":36365}]},"public_addr":"172.21.2.113:6821/36365","cluster_addr":"172.21.2.113:6825/36365","heartbeat_back_addr":"172.21.2.113:6831/36365","heartbeat_front_addr":"172.21.2.113:6829/36365","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:36:05.874 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:05.872 7f03472be700 1 -- 172.21.2.102:0/490181392 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0320018730 msgr2=0x7f032001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:05.874 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:05.872 7f03472be700 1 --2- 172.21.2.102:0/490181392 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0320018730 0x7f032001abc0 crc :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:05.874 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:05.872 7f03472be700 1 -- 172.21.2.102:0/490181392 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0340108ce0 msgr2=0x7f034009da50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:05.874 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:05.872 7f03472be700 1 --2- 172.21.2.102:0/490181392 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0340108ce0 0x7f034009da50 secure :-1 s=READY pgs=2780 cs=0 l=1 rev1=1 rx=0x7f0334006f60 tx=0x7f033400b970).stop 2021-02-10T11:36:05.875 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:05.872 7f03472be700 1 -- 172.21.2.102:0/490181392 shutdown_connections 2021-02-10T11:36:05.875 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:05.872 7f03472be700 1 --2- 172.21.2.102:0/490181392 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0320018730 0x7f032001abc0 unknown :-1 s=CLOSED pgs=1223 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:05.875 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:05.873 7f03472be700 1 --2- 172.21.2.102:0/490181392 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f034009eee0 0x7f034009f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:05.875 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:05.873 7f03472be700 1 --2- 172.21.2.102:0/490181392 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f03401090e0 0x7f034009e070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:05.875 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:05.873 7f03472be700 1 --2- 172.21.2.102:0/490181392 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0340108ce0 0x7f034009da50 unknown :-1 s=CLOSED pgs=2780 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:05.875 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:05.873 7f03472be700 1 -- 172.21.2.102:0/490181392 >> 172.21.2.102:0/490181392 conn(0x7f03400bc6c0 msgr2=0x7f03400b1f70 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:05.875 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:05.873 7f03472be700 1 -- 172.21.2.102:0/490181392 shutdown_connections 2021-02-10T11:36:05.875 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:05.873 7f03472be700 1 -- 172.21.2.102:0/490181392 wait complete. 2021-02-10T11:36:05.883 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:03:37.322519', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 32, 'pool_name': 'cephfs_metadata', 'create_time': '2021-02-10 11:35:56.251556', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '152', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 33, 'pool_name': 'cephfs_data', 'create_time': '2021-02-10 11:35:57.328218', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '153', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-02-10T11:36:05.884 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:36:06.318 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:06.298 7f1ad2e85700 1 Processor -- start 2021-02-10T11:36:06.319 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:06.298 7f1ad2e85700 1 -- start start 2021-02-10T11:36:06.319 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:06.298 7f1ad2e85700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f1acc0d1870 0x7f1acc1b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:06.319 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:06.298 7f1ad2e85700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1acc0fe1c0 con 0x7f1acc1b0ae0 2021-02-10T11:36:06.319 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:06.298 7f1ad2e85700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1acc109a60 con 0x7f1acc0cff70 2021-02-10T11:36:06.319 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:06.298 7f1ad2e85700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1acc088070 con 0x7f1acc0d1870 2021-02-10T11:36:06.319 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:06.298 7f1acbfff700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f1acc1b0ae0 0x7f1acc108940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:58022/0 (socket says 172.21.2.102:58022) 2021-02-10T11:36:06.319 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:06.298 7f1acbfff700 1 -- 172.21.2.102:0/4211124183 learned_addr learned my addr 172.21.2.102:0/4211124183 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:36:06.319 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:06.299 7f1ad0ca1700 1 --2- 172.21.2.102:0/4211124183 >> v2:172.21.2.113:3300/0 conn(0x7f1acc0d1870 0x7f1acc1b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:06.319 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:06.299 7f1acb7fe700 1 -- 172.21.2.102:0/4211124183 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 403768425 0 0) 0x7f1ab80010b0 con 0x7f1acc1b0ae0 2021-02-10T11:36:06.319 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:06.299 7f1acb7fe700 1 -- 172.21.2.102:0/4211124183 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1abc002fe0 con 0x7f1acc1b0ae0 2021-02-10T11:36:06.319 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:06.299 7f1acb7fe700 1 -- 172.21.2.102:0/4211124183 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 643739953 0 0) 0x7f1ab8001010 con 0x7f1acc1b0ae0 2021-02-10T11:36:06.320 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:06.299 7f1acb7fe700 1 -- 172.21.2.102:0/4211124183 >> v2:172.21.2.113:3300/0 conn(0x7f1acc0d1870 msgr2=0x7f1acc1b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:06.320 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:06.299 7f1acb7fe700 1 --2- 172.21.2.102:0/4211124183 >> v2:172.21.2.113:3300/0 conn(0x7f1acc0d1870 0x7f1acc1b04c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:06.320 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:06.299 7f1acb7fe700 1 -- 172.21.2.102:0/4211124183 >> v1:172.21.2.113:6789/0 conn(0x7f1acc0cff70 legacy=0x7f1acc109130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:06.320 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:06.299 7f1acb7fe700 1 -- 172.21.2.102:0/4211124183 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1acc088260 con 0x7f1acc1b0ae0 2021-02-10T11:36:06.320 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:06.299 7f1acb7fe700 1 -- 172.21.2.102:0/4211124183 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7f1ab8001530 con 0x7f1acc1b0ae0 2021-02-10T11:36:06.320 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:06.299 7f1acb7fe700 1 -- 172.21.2.102:0/4211124183 >> v1:172.21.2.102:6789/0 conn(0x7f1acc1b0ae0 legacy=0x7f1acc108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:06.320 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:06.299 7f1acb7fe700 1 --2- 172.21.2.102:0/4211124183 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1abc003b90 0x7f1abc0060b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:06.320 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:06.299 7f1acb7fe700 1 --2- 172.21.2.102:0/4211124183 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1abc0068c0 0x7f1abc008d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:06.320 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:06.299 7f1acb7fe700 1 --2- 172.21.2.102:0/4211124183 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1abc009670 0x7f1abc00bb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:06.320 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:06.299 7f1acb7fe700 1 -- 172.21.2.102:0/4211124183 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1abc00c3e0 con 0x7f1abc003b90 2021-02-10T11:36:06.320 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:06.299 7f1acb7fe700 1 -- 172.21.2.102:0/4211124183 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1abc00c600 con 0x7f1abc009670 2021-02-10T11:36:06.320 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:06.299 7f1acb7fe700 1 -- 172.21.2.102:0/4211124183 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1abc00c820 con 0x7f1abc0068c0 2021-02-10T11:36:06.320 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:06.299 7f1ad0ca1700 1 --2- 172.21.2.102:0/4211124183 >> v2:172.21.2.113:3300/0 conn(0x7f1acc0d1870 0x7f1acc1b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:36:06.321 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:06.299 7f1ad0ca1700 1 --2- 172.21.2.102:0/4211124183 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1abc003b90 0x7f1abc0060b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:06.321 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:06.299 7f1acbfff700 1 --2- 172.21.2.102:0/4211124183 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1abc0068c0 0x7f1abc008d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:06.321 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:06.299 7f1ad14a2700 1 --2- 172.21.2.102:0/4211124183 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1abc009670 0x7f1abc00bb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:06.321 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:06.300 7f1ad0ca1700 1 -- 172.21.2.102:0/4211124183 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1abc0068c0 msgr2=0x7f1abc008d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:06.321 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:06.300 7f1ad0ca1700 1 --2- 172.21.2.102:0/4211124183 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1abc0068c0 0x7f1abc008d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:06.321 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:06.300 7f1ad0ca1700 1 -- 172.21.2.102:0/4211124183 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1abc009670 msgr2=0x7f1abc00bb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:06.321 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:06.300 7f1ad0ca1700 1 --2- 172.21.2.102:0/4211124183 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1abc009670 0x7f1abc00bb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:06.321 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:06.300 7f1ad0ca1700 1 -- 172.21.2.102:0/4211124183 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1abc00cac0 con 0x7f1abc003b90 2021-02-10T11:36:06.321 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:06.300 7f1ad14a2700 1 -- 172.21.2.102:0/4211124183 reap_dead start 2021-02-10T11:36:06.321 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:06.300 7f1acbfff700 1 --2- 172.21.2.102:0/4211124183 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1abc0068c0 0x7f1abc008d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:36:06.321 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:06.300 7f1ad0ca1700 1 --2- 172.21.2.102:0/4211124183 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1abc003b90 0x7f1abc0060b0 secure :-1 s=READY pgs=2782 cs=0 l=1 rev1=1 rx=0x7f1ac004d9c0 tx=0x7f1ac004dd60).ready entity=mon.0 client_cookie=cad9bbdf8e170dab server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:06.321 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:06.300 7f1acb7fe700 1 -- 172.21.2.102:0/4211124183 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1ac004acf0 con 0x7f1abc003b90 2021-02-10T11:36:06.321 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:06.300 7f1acb7fe700 1 -- 172.21.2.102:0/4211124183 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ac00067d0 con 0x7f1abc003b90 2021-02-10T11:36:06.322 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:06.300 7f1acb7fe700 1 -- 172.21.2.102:0/4211124183 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1ac0002220 con 0x7f1abc003b90 2021-02-10T11:36:06.322 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:06.300 7f1ad2e85700 1 -- 172.21.2.102:0/4211124183 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1abc003b90 msgr2=0x7f1abc0060b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:06.322 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:06.300 7f1ad2e85700 1 --2- 172.21.2.102:0/4211124183 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1abc003b90 0x7f1abc0060b0 secure :-1 s=READY pgs=2782 cs=0 l=1 rev1=1 rx=0x7f1ac004d9c0 tx=0x7f1ac004dd60).stop 2021-02-10T11:36:06.322 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:06.301 7f1ad2e85700 1 -- 172.21.2.102:0/4211124183 shutdown_connections 2021-02-10T11:36:06.322 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:06.301 7f1ad2e85700 1 --2- 172.21.2.102:0/4211124183 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1abc003b90 0x7f1abc0060b0 unknown :-1 s=CLOSED pgs=2782 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:06.322 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:06.301 7f1ad2e85700 1 -- 172.21.2.102:0/4211124183 >> 172.21.2.102:0/4211124183 conn(0x7f1acc0ce420 msgr2=0x7f1acc0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:06.322 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:06.301 7f1ad2e85700 1 -- 172.21.2.102:0/4211124183 shutdown_connections 2021-02-10T11:36:06.322 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:06.301 7f1ad2e85700 1 -- 172.21.2.102:0/4211124183 wait complete. 2021-02-10T11:36:06.322 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:06.316 7f1ad2e85700 1 Processor -- start 2021-02-10T11:36:06.322 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:06.316 7f1ad2e85700 1 -- start start 2021-02-10T11:36:06.322 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:06.317 7f1ad2e85700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1acc09bcc0 0x7f1acc09c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:06.322 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:06.317 7f1ad2e85700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1acc0a0820 0x7f1acc0a0c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:06.322 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:06.317 7f1ad2e85700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1acc0a1240 0x7f1acc0858d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:06.323 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:06.317 7f1ad2e85700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1acc0a66b0 con 0x7f1acc09bcc0 2021-02-10T11:36:06.323 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:06.317 7f1ad2e85700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1acc1a49b0 con 0x7f1acc0a0820 2021-02-10T11:36:06.323 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:06.317 7f1ad2e85700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1acc1a4b90 con 0x7f1acc0a1240 2021-02-10T11:36:06.324 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:06.317 7f1ad0ca1700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1acc09bcc0 0x7f1acc09c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:06.324 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:06.317 7f1ad0ca1700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1acc09bcc0 0x7f1acc09c0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:37582/0 (socket says 172.21.2.102:37582) 2021-02-10T11:36:06.324 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:06.317 7f1ad0ca1700 1 -- 172.21.2.102:0/3886459677 learned_addr learned my addr 172.21.2.102:0/3886459677 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:36:06.324 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:06.317 7f1ad14a2700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1acc0a1240 0x7f1acc0858d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:06.324 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:06.317 7f1acbfff700 1 --2- 172.21.2.102:0/3886459677 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1acc0a0820 0x7f1acc0a0c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:06.324 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:06.317 7f1ad0ca1700 1 -- 172.21.2.102:0/3886459677 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1acc0a1240 msgr2=0x7f1acc0858d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:06.324 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:06.317 7f1ad0ca1700 1 --2- 172.21.2.102:0/3886459677 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1acc0a1240 0x7f1acc0858d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:06.325 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:06.317 7f1ad0ca1700 1 -- 172.21.2.102:0/3886459677 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1acc0a0820 msgr2=0x7f1acc0a0c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:06.325 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:06.317 7f1ad0ca1700 1 --2- 172.21.2.102:0/3886459677 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1acc0a0820 0x7f1acc0a0c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:06.325 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:06.317 7f1ad0ca1700 1 -- 172.21.2.102:0/3886459677 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1acc0a1640 con 0x7f1acc09bcc0 2021-02-10T11:36:06.325 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:06.317 7f1ad0ca1700 1 --2- 172.21.2.102:0/3886459677 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1acc09bcc0 0x7f1acc09c0c0 secure :-1 s=READY pgs=2783 cs=0 l=1 rev1=1 rx=0x7f1ac004f690 tx=0x7f1ac004fa30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:06.325 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:06.317 7f1ac97fa700 1 -- 172.21.2.102:0/3886459677 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1ac0000c90 con 0x7f1acc09bcc0 2021-02-10T11:36:06.325 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:06.318 7f1ac97fa700 1 -- 172.21.2.102:0/3886459677 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ac004d140 con 0x7f1acc09bcc0 2021-02-10T11:36:06.325 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:06.318 7f1ad2e85700 1 -- 172.21.2.102:0/3886459677 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1acc0a1640 con 0x7f1acc09bcc0 2021-02-10T11:36:06.325 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:06.318 7f1ac97fa700 1 -- 172.21.2.102:0/3886459677 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1ac0002220 con 0x7f1acc09bcc0 2021-02-10T11:36:06.325 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:06.318 7f1ad2e85700 1 -- 172.21.2.102:0/3886459677 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1acc0861e0 con 0x7f1acc09bcc0 2021-02-10T11:36:06.325 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:06.318 7f1ac97fa700 1 -- 172.21.2.102:0/3886459677 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1ac004e000 con 0x7f1acc09bcc0 2021-02-10T11:36:06.325 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:06.318 7f1ac97fa700 1 --2- 172.21.2.102:0/3886459677 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1aac018730 0x7f1aac01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:06.325 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:06.318 7f1ac97fa700 1 -- 172.21.2.102:0/3886459677 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(153..153 src has 1..153) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f1ac0076620 con 0x7f1acc09bcc0 2021-02-10T11:36:06.325 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:06.319 7f1acbfff700 1 --2- 172.21.2.102:0/3886459677 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1aac018730 0x7f1aac01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:06.326 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:06.319 7f1acbfff700 1 --2- 172.21.2.102:0/3886459677 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1aac018730 0x7f1aac01ac10 crc :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:06.350 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:06.348 7f1ad2e85700 1 -- 172.21.2.102:0/3886459677 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1acc086a70 con 0x7f1acc09bcc0 2021-02-10T11:36:06.352 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:06.350 7f1ac97fa700 1 -- 172.21.2.102:0/3886459677 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f1ac004d140 con 0x7f1acc09bcc0 2021-02-10T11:36:06.483 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:06.481 7f1ad2e85700 1 -- 172.21.2.102:0/3886459677 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f1acc086a70 con 0x7f1acc09bcc0 2021-02-10T11:36:06.483 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:06.481 7f1ac97fa700 1 -- 172.21.2.102:0/3886459677 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v153) v1 ==== 92+0+10 (secure 0 0 0) 0x7f1ac004e000 con 0x7f1acc09bcc0 2021-02-10T11:36:06.483 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:36:06.485 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:06.483 7f1ad2e85700 1 -- 172.21.2.102:0/3886459677 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1aac018730 msgr2=0x7f1aac01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:06.485 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:06.483 7f1ad2e85700 1 --2- 172.21.2.102:0/3886459677 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1aac018730 0x7f1aac01ac10 crc :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:06.485 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:06.483 7f1ad2e85700 1 -- 172.21.2.102:0/3886459677 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1acc09bcc0 msgr2=0x7f1acc09c0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:06.485 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:06.483 7f1ad2e85700 1 --2- 172.21.2.102:0/3886459677 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1acc09bcc0 0x7f1acc09c0c0 secure :-1 s=READY pgs=2783 cs=0 l=1 rev1=1 rx=0x7f1ac004f690 tx=0x7f1ac004fa30).stop 2021-02-10T11:36:06.485 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:06.483 7f1ad2e85700 1 -- 172.21.2.102:0/3886459677 shutdown_connections 2021-02-10T11:36:06.485 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:06.483 7f1ad2e85700 1 --2- 172.21.2.102:0/3886459677 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1acc0a1240 0x7f1acc0858d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:06.486 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:06.483 7f1ad2e85700 1 --2- 172.21.2.102:0/3886459677 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1aac018730 0x7f1aac01ac10 unknown :-1 s=CLOSED pgs=1224 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:06.486 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:06.483 7f1ad2e85700 1 --2- 172.21.2.102:0/3886459677 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1acc0a0820 0x7f1acc0a0c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:06.486 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:06.483 7f1ad2e85700 1 --2- 172.21.2.102:0/3886459677 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1acc09bcc0 0x7f1acc09c0c0 unknown :-1 s=CLOSED pgs=2783 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:06.486 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:06.483 7f1ad2e85700 1 -- 172.21.2.102:0/3886459677 >> 172.21.2.102:0/3886459677 conn(0x7f1acc0bef00 msgr2=0x7f1acc09d1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:06.486 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:06.483 7f1ad2e85700 1 -- 172.21.2.102:0/3886459677 shutdown_connections 2021-02-10T11:36:06.486 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:06.483 7f1ad2e85700 1 -- 172.21.2.102:0/3886459677 wait complete. 2021-02-10T11:36:06.494 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-02-10T11:36:06.927 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:06.909 7efc181d6700 1 Processor -- start 2021-02-10T11:36:06.927 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:06.909 7efc181d6700 1 -- start start 2021-02-10T11:36:06.928 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:06.909 7efc181d6700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7efc100d1870 0x7efc101b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:06.928 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:06.909 7efc181d6700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7efc100cff70 0x7efc10109130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:06.928 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:06.909 7efc181d6700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc100fe1c0 con 0x7efc101b0ae0 2021-02-10T11:36:06.928 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:06.909 7efc181d6700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc10109ad0 con 0x7efc100d1870 2021-02-10T11:36:06.928 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:06.909 7efc181d6700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc10109750 con 0x7efc100cff70 2021-02-10T11:36:06.928 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:06.910 7efc15ff2700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7efc100d1870 0x7efc101b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:06.928 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:06.910 7efc15ff2700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7efc100d1870 0x7efc101b04c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:37586/0 (socket says 172.21.2.102:37586) 2021-02-10T11:36:06.928 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:06.910 7efc15ff2700 1 -- 172.21.2.102:0/2964107224 learned_addr learned my addr 172.21.2.102:0/2964107224 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:36:06.928 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:06.910 7efc15ff2700 1 -- 172.21.2.102:0/2964107224 >> v2:172.21.2.113:3300/0 conn(0x7efc100cff70 msgr2=0x7efc10109130 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:36:06.928 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:06.910 7efc15ff2700 1 --2- 172.21.2.102:0/2964107224 >> v2:172.21.2.113:3300/0 conn(0x7efc100cff70 0x7efc10109130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:06.928 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:06.910 7efc15ff2700 1 -- 172.21.2.102:0/2964107224 >> v1:172.21.2.113:6789/0 conn(0x7efc101b0ae0 legacy=0x7efc10108940 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:36:06.928 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:06.910 7efc15ff2700 1 -- 172.21.2.102:0/2964107224 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc10088070 con 0x7efc100d1870 2021-02-10T11:36:06.928 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:06.910 7efc15ff2700 1 --2- 172.21.2.102:0/2964107224 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efc100d1870 0x7efc101b04c0 secure :-1 s=READY pgs=2784 cs=0 l=1 rev1=1 rx=0x7efc04006700 tx=0x7efc04009b80).ready entity=mon.0 client_cookie=9ce6bc7f6d2a27d7 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:06.929 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:06.910 7efc14ff0700 1 -- 172.21.2.102:0/2964107224 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efc0400c0c0 con 0x7efc100d1870 2021-02-10T11:36:06.929 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:06.910 7efc14ff0700 1 -- 172.21.2.102:0/2964107224 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc0400c340 con 0x7efc100d1870 2021-02-10T11:36:06.929 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:06.910 7efc14ff0700 1 -- 172.21.2.102:0/2964107224 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efc0400c860 con 0x7efc100d1870 2021-02-10T11:36:06.929 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:06.910 7efc181d6700 1 -- 172.21.2.102:0/2964107224 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efc100d1870 msgr2=0x7efc101b04c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:06.929 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:06.910 7efc181d6700 1 --2- 172.21.2.102:0/2964107224 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efc100d1870 0x7efc101b04c0 secure :-1 s=READY pgs=2784 cs=0 l=1 rev1=1 rx=0x7efc04006700 tx=0x7efc04009b80).stop 2021-02-10T11:36:06.929 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:06.911 7efc181d6700 1 -- 172.21.2.102:0/2964107224 shutdown_connections 2021-02-10T11:36:06.929 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:06.911 7efc181d6700 1 --2- 172.21.2.102:0/2964107224 >> v2:172.21.2.113:3300/0 conn(0x7efc100cff70 0x7efc10109130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:06.929 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:06.911 7efc181d6700 1 --2- 172.21.2.102:0/2964107224 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efc100d1870 0x7efc101b04c0 unknown :-1 s=CLOSED pgs=2784 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:06.929 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:06.911 7efc181d6700 1 -- 172.21.2.102:0/2964107224 >> 172.21.2.102:0/2964107224 conn(0x7efc100ce420 msgr2=0x7efc100d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:06.929 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:06.911 7efc181d6700 1 -- 172.21.2.102:0/2964107224 shutdown_connections 2021-02-10T11:36:06.929 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:06.911 7efc181d6700 1 -- 172.21.2.102:0/2964107224 wait complete. 2021-02-10T11:36:06.931 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:06.926 7efc181d6700 1 Processor -- start 2021-02-10T11:36:06.931 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:06.926 7efc181d6700 1 -- start start 2021-02-10T11:36:06.931 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:06.926 7efc181d6700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7efc100c2370 0x7efc1018f870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:06.931 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:06.926 7efc181d6700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efc10084cb0 0x7efc101a7470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:06.931 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:06.926 7efc181d6700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7efc101a7a90 0x7efc10085700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:06.931 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:06.926 7efc181d6700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc100be840 con 0x7efc10084cb0 2021-02-10T11:36:06.931 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:06.926 7efc181d6700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc100c29a0 con 0x7efc100c2370 2021-02-10T11:36:06.931 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:06.926 7efc181d6700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc101a7e90 con 0x7efc101a7a90 2021-02-10T11:36:06.931 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:06.926 7efc157f1700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efc10084cb0 0x7efc101a7470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:06.931 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:06.926 7efc157f1700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efc10084cb0 0x7efc101a7470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:37594/0 (socket says 172.21.2.102:37594) 2021-02-10T11:36:06.931 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:06.926 7efc157f1700 1 -- 172.21.2.102:0/3745919210 learned_addr learned my addr 172.21.2.102:0/3745919210 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:36:06.932 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:06.927 7efc157f1700 1 -- 172.21.2.102:0/3745919210 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7efc101a7a90 msgr2=0x7efc10085700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:06.932 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:06.927 7efc157f1700 1 --2- 172.21.2.102:0/3745919210 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7efc101a7a90 0x7efc10085700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:06.932 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:06.927 7efc157f1700 1 -- 172.21.2.102:0/3745919210 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7efc100c2370 msgr2=0x7efc1018f870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:06.932 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:06.927 7efc157f1700 1 --2- 172.21.2.102:0/3745919210 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7efc100c2370 0x7efc1018f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:06.932 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:06.927 7efc157f1700 1 -- 172.21.2.102:0/3745919210 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc101a80b0 con 0x7efc10084cb0 2021-02-10T11:36:06.932 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:06.927 7efc157f1700 1 --2- 172.21.2.102:0/3745919210 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efc10084cb0 0x7efc101a7470 secure :-1 s=READY pgs=2785 cs=0 l=1 rev1=1 rx=0x7efc0c00b530 tx=0x7efc0c00b8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:06.932 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:06.927 7efc027fc700 1 -- 172.21.2.102:0/3745919210 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efc0c00bcc0 con 0x7efc10084cb0 2021-02-10T11:36:06.932 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:06.927 7efc027fc700 1 -- 172.21.2.102:0/3745919210 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc0c006020 con 0x7efc10084cb0 2021-02-10T11:36:06.932 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:06.927 7efc027fc700 1 -- 172.21.2.102:0/3745919210 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efc0c0054a0 con 0x7efc10084cb0 2021-02-10T11:36:06.932 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:06.927 7efc181d6700 1 -- 172.21.2.102:0/3745919210 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc101a80b0 con 0x7efc10084cb0 2021-02-10T11:36:06.932 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:06.927 7efc181d6700 1 -- 172.21.2.102:0/3745919210 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc101a80b0 con 0x7efc10084cb0 2021-02-10T11:36:06.932 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:06.928 7efc027fc700 1 -- 172.21.2.102:0/3745919210 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7efc0c006200 con 0x7efc10084cb0 2021-02-10T11:36:06.932 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:06.928 7efc027fc700 1 --2- 172.21.2.102:0/3745919210 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7efbf80186c0 0x7efbf801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:06.933 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:06.928 7efc027fc700 1 -- 172.21.2.102:0/3745919210 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(153..153 src has 1..153) v4 ==== 5573+0+0 (secure 0 0 0) 0x7efc0c0325a0 con 0x7efc10084cb0 2021-02-10T11:36:06.933 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:06.928 7efc15ff2700 1 --2- 172.21.2.102:0/3745919210 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7efbf80186c0 0x7efbf801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:06.933 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:06.928 7efc15ff2700 1 --2- 172.21.2.102:0/3745919210 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7efbf80186c0 0x7efbf801abc0 crc :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:06.961 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:06.959 7efc181d6700 1 -- 172.21.2.102:0/3745919210 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc10082a70 con 0x7efc10084cb0 2021-02-10T11:36:06.964 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:06.962 7efc027fc700 1 -- 172.21.2.102:0/3745919210 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7efc0c030d50 con 0x7efc10084cb0 2021-02-10T11:36:07.091 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:07.089 7efc181d6700 1 -- 172.21.2.102:0/3745919210 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7efc10082a70 con 0x7efc10084cb0 2021-02-10T11:36:07.091 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:07.089 7efc027fc700 1 -- 172.21.2.102:0/3745919210 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v153) v1 ==== 104+0+10 (secure 0 0 0) 0x7efc0c00bcc0 con 0x7efc10084cb0 2021-02-10T11:36:07.091 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:36:07.093 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:07.091 7efc181d6700 1 -- 172.21.2.102:0/3745919210 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7efbf80186c0 msgr2=0x7efbf801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:07.093 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:07.091 7efc181d6700 1 --2- 172.21.2.102:0/3745919210 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7efbf80186c0 0x7efbf801abc0 crc :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:07.093 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:07.091 7efc181d6700 1 -- 172.21.2.102:0/3745919210 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efc10084cb0 msgr2=0x7efc101a7470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:07.093 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:07.091 7efc181d6700 1 --2- 172.21.2.102:0/3745919210 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efc10084cb0 0x7efc101a7470 secure :-1 s=READY pgs=2785 cs=0 l=1 rev1=1 rx=0x7efc0c00b530 tx=0x7efc0c00b8d0).stop 2021-02-10T11:36:07.093 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:07.091 7efc181d6700 1 -- 172.21.2.102:0/3745919210 shutdown_connections 2021-02-10T11:36:07.093 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:07.091 7efc181d6700 1 --2- 172.21.2.102:0/3745919210 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7efc101a7a90 0x7efc10085700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:07.094 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:07.091 7efc181d6700 1 --2- 172.21.2.102:0/3745919210 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efc10084cb0 0x7efc101a7470 unknown :-1 s=CLOSED pgs=2785 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:07.094 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:07.091 7efc181d6700 1 --2- 172.21.2.102:0/3745919210 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7efbf80186c0 0x7efbf801abc0 unknown :-1 s=CLOSED pgs=1225 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:07.094 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:07.091 7efc181d6700 1 --2- 172.21.2.102:0/3745919210 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7efc100c2370 0x7efc1018f870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:07.094 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:07.091 7efc181d6700 1 -- 172.21.2.102:0/3745919210 >> 172.21.2.102:0/3745919210 conn(0x7efc100b5dc0 msgr2=0x7efc100c2d90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:07.094 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:07.091 7efc181d6700 1 -- 172.21.2.102:0/3745919210 shutdown_connections 2021-02-10T11:36:07.094 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:07.091 7efc181d6700 1 -- 172.21.2.102:0/3745919210 wait complete. 2021-02-10T11:36:07.102 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-02-10T11:36:07.535 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:07.517 7fadfd734700 1 Processor -- start 2021-02-10T11:36:07.536 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:07.517 7fadfd734700 1 -- start start 2021-02-10T11:36:07.536 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:07.517 7fadfd734700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fadf80d1870 0x7fadf81b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:07.536 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:07.517 7fadfd734700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fadf80fe1c0 con 0x7fadf80cff70 2021-02-10T11:36:07.536 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:07.517 7fadfd734700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fadf8109a60 con 0x7fadf81b0ae0 2021-02-10T11:36:07.536 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:07.517 7fadfd734700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fadf8088070 con 0x7fadf80d1870 2021-02-10T11:36:07.536 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:07.518 7fadf7d4a700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fadf80cff70 0x7fadf8109130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:58056/0 (socket says 172.21.2.102:58056) 2021-02-10T11:36:07.536 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:07.518 7fadf7d4a700 1 -- 172.21.2.102:0/384202238 learned_addr learned my addr 172.21.2.102:0/384202238 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:36:07.536 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:07.518 7fadf6547700 1 -- 172.21.2.102:0/384202238 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3972681944 0 0) 0x7fadf0001500 con 0x7fadf80cff70 2021-02-10T11:36:07.536 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:07.518 7fadf6547700 1 -- 172.21.2.102:0/384202238 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fade4002fe0 con 0x7fadf80cff70 2021-02-10T11:36:07.536 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:07.518 7fadf6547700 1 -- 172.21.2.102:0/384202238 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2454024645 0 0) 0x7fadf0001460 con 0x7fadf80cff70 2021-02-10T11:36:07.536 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:07.518 7fadf6547700 1 -- 172.21.2.102:0/384202238 >> v2:172.21.2.113:3300/0 conn(0x7fadf80d1870 msgr2=0x7fadf81b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:07.537 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:07.518 7fadf6547700 1 --2- 172.21.2.102:0/384202238 >> v2:172.21.2.113:3300/0 conn(0x7fadf80d1870 0x7fadf81b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:07.537 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:07.518 7fadf6547700 1 -- 172.21.2.102:0/384202238 >> v1:172.21.2.113:6789/0 conn(0x7fadf81b0ae0 legacy=0x7fadf8108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:07.537 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:07.518 7fadf6547700 1 -- 172.21.2.102:0/384202238 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadf8088260 con 0x7fadf80cff70 2021-02-10T11:36:07.537 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:07.518 7fadf6547700 1 -- 172.21.2.102:0/384202238 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7fadf0001980 con 0x7fadf80cff70 2021-02-10T11:36:07.537 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:07.518 7fadf6547700 1 -- 172.21.2.102:0/384202238 >> v1:172.21.2.102:6789/0 conn(0x7fadf80cff70 legacy=0x7fadf8109130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:07.537 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:07.518 7fadf6547700 1 --2- 172.21.2.102:0/384202238 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fade4003aa0 0x7fade4005ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:07.537 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:07.518 7fadf6547700 1 --2- 172.21.2.102:0/384202238 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fade4006790 0x7fade4008c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:07.537 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:07.518 7fadf6547700 1 --2- 172.21.2.102:0/384202238 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fade4009590 0x7fade400bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:07.537 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:07.518 7fadf6547700 1 -- 172.21.2.102:0/384202238 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fade400c300 con 0x7fade4009590 2021-02-10T11:36:07.537 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:07.518 7fadf6547700 1 -- 172.21.2.102:0/384202238 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fade400c520 con 0x7fade4003aa0 2021-02-10T11:36:07.537 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:07.518 7fadf6547700 1 -- 172.21.2.102:0/384202238 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fade400c740 con 0x7fade4006790 2021-02-10T11:36:07.537 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:07.518 7fadf7d4a700 1 --2- 172.21.2.102:0/384202238 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fade4009590 0x7fade400bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:07.537 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:07.518 7fadf7549700 1 --2- 172.21.2.102:0/384202238 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fade4003aa0 0x7fade4005ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:07.538 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:07.518 7fadf6d48700 1 --2- 172.21.2.102:0/384202238 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fade4006790 0x7fade4008c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:07.538 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:07.518 7fadf7d4a700 1 -- 172.21.2.102:0/384202238 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fade4006790 msgr2=0x7fade4008c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:07.538 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:07.518 7fadf7d4a700 1 --2- 172.21.2.102:0/384202238 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fade4006790 0x7fade4008c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:07.538 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:07.518 7fadf7d4a700 1 -- 172.21.2.102:0/384202238 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fade4003aa0 msgr2=0x7fade4005ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:07.538 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:07.518 7fadf7d4a700 1 --2- 172.21.2.102:0/384202238 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fade4003aa0 0x7fade4005ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:07.538 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:07.519 7fadf7d4a700 1 -- 172.21.2.102:0/384202238 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fade400c9e0 con 0x7fade4009590 2021-02-10T11:36:07.538 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:07.519 7fadf7d4a700 1 -- 172.21.2.102:0/384202238 reap_dead start 2021-02-10T11:36:07.538 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:07.519 7fadf7d4a700 1 --2- 172.21.2.102:0/384202238 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fade4009590 0x7fade400bab0 secure :-1 s=READY pgs=2787 cs=0 l=1 rev1=1 rx=0x7fadf0007ce0 tx=0x7fadf0050310).ready entity=mon.0 client_cookie=9e5bfd8c59589e18 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:07.538 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:07.519 7fadf6547700 1 -- 172.21.2.102:0/384202238 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fadf0007dd0 con 0x7fade4009590 2021-02-10T11:36:07.538 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:07.519 7fadf6547700 1 -- 172.21.2.102:0/384202238 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadf0007030 con 0x7fade4009590 2021-02-10T11:36:07.538 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:07.519 7fadf6547700 1 -- 172.21.2.102:0/384202238 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fadf0007550 con 0x7fade4009590 2021-02-10T11:36:07.538 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:07.519 7fadfd734700 1 -- 172.21.2.102:0/384202238 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fade4009590 msgr2=0x7fade400bab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:07.538 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:07.519 7fadfd734700 1 --2- 172.21.2.102:0/384202238 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fade4009590 0x7fade400bab0 secure :-1 s=READY pgs=2787 cs=0 l=1 rev1=1 rx=0x7fadf0007ce0 tx=0x7fadf0050310).stop 2021-02-10T11:36:07.539 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:07.519 7fadfd734700 1 -- 172.21.2.102:0/384202238 shutdown_connections 2021-02-10T11:36:07.539 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:07.519 7fadfd734700 1 --2- 172.21.2.102:0/384202238 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fade4009590 0x7fade400bab0 unknown :-1 s=CLOSED pgs=2787 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:07.539 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:07.519 7fadfd734700 1 -- 172.21.2.102:0/384202238 >> 172.21.2.102:0/384202238 conn(0x7fadf80ce420 msgr2=0x7fadf80d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:07.539 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:07.519 7fadfd734700 1 -- 172.21.2.102:0/384202238 shutdown_connections 2021-02-10T11:36:07.539 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:07.519 7fadfd734700 1 -- 172.21.2.102:0/384202238 wait complete. 2021-02-10T11:36:07.539 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:07.534 7fadfd734700 1 Processor -- start 2021-02-10T11:36:07.539 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:07.534 7fadfd734700 1 -- start start 2021-02-10T11:36:07.540 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:07.534 7fadfd734700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fadf8085bc0 0x7fadf80863e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:07.540 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:07.534 7fadfd734700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fadf80954c0 0x7fadf8096300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:07.540 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:07.534 7fadfd734700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fadf80958c0 0x7fadf8096920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:07.540 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:07.534 7fadfd734700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadf8095d80 con 0x7fadf80954c0 2021-02-10T11:36:07.541 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:07.534 7fadfd734700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadf80bfa60 con 0x7fadf8085bc0 2021-02-10T11:36:07.541 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:07.534 7fadfd734700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fadf8085fc0 con 0x7fadf80958c0 2021-02-10T11:36:07.541 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:07.534 7fadf6d48700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fadf80954c0 0x7fadf8096300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:07.541 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:07.534 7fadf6d48700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fadf80954c0 0x7fadf8096300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:37610/0 (socket says 172.21.2.102:37610) 2021-02-10T11:36:07.541 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:07.534 7fadf6d48700 1 -- 172.21.2.102:0/3758588851 learned_addr learned my addr 172.21.2.102:0/3758588851 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:36:07.541 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:07.535 7fadf6d48700 1 -- 172.21.2.102:0/3758588851 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fadf80958c0 msgr2=0x7fadf8096920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:07.541 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:07.535 7fadf6d48700 1 --2- 172.21.2.102:0/3758588851 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fadf80958c0 0x7fadf8096920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:07.541 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:07.535 7fadf6d48700 1 -- 172.21.2.102:0/3758588851 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fadf8085bc0 msgr2=0x7fadf80863e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:07.541 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:07.535 7fadf6d48700 1 --2- 172.21.2.102:0/3758588851 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fadf8085bc0 0x7fadf80863e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:07.541 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:07.535 7fadf6d48700 1 -- 172.21.2.102:0/3758588851 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadf8086c10 con 0x7fadf80954c0 2021-02-10T11:36:07.541 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:07.535 7fadf6d48700 1 --2- 172.21.2.102:0/3758588851 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fadf80954c0 0x7fadf8096300 secure :-1 s=READY pgs=2788 cs=0 l=1 rev1=1 rx=0x7fade800b580 tx=0x7fade800b920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:07.541 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:07.535 7fadeffff700 1 -- 172.21.2.102:0/3758588851 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fade8006be0 con 0x7fadf80954c0 2021-02-10T11:36:07.541 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:07.535 7fadeffff700 1 -- 172.21.2.102:0/3758588851 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fade800bd10 con 0x7fadf80954c0 2021-02-10T11:36:07.542 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:07.535 7fadeffff700 1 -- 172.21.2.102:0/3758588851 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fade80054d0 con 0x7fadf80954c0 2021-02-10T11:36:07.542 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:07.535 7fadfd734700 1 -- 172.21.2.102:0/3758588851 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fadf8086c10 con 0x7fadf80954c0 2021-02-10T11:36:07.542 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:07.535 7fadfd734700 1 -- 172.21.2.102:0/3758588851 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fadf8097230 con 0x7fadf80954c0 2021-02-10T11:36:07.542 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:07.536 7fadeffff700 1 -- 172.21.2.102:0/3758588851 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fade8006030 con 0x7fadf80954c0 2021-02-10T11:36:07.542 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:07.536 7fadeffff700 1 --2- 172.21.2.102:0/3758588851 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7faddc0186c0 0x7faddc01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:07.542 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:07.536 7fadeffff700 1 -- 172.21.2.102:0/3758588851 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(153..153 src has 1..153) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fade8032430 con 0x7fadf80954c0 2021-02-10T11:36:07.542 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:07.536 7fadf7549700 1 --2- 172.21.2.102:0/3758588851 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7faddc0186c0 0x7faddc01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:07.542 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:07.536 7fadf7549700 1 --2- 172.21.2.102:0/3758588851 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7faddc0186c0 0x7faddc01abc0 crc :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:07.569 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:07.567 7fadfd734700 1 -- 172.21.2.102:0/3758588851 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fadf80fe1c0 con 0x7fadf80954c0 2021-02-10T11:36:07.572 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:07.570 7fadeffff700 1 -- 172.21.2.102:0/3758588851 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fade80009b0 con 0x7fadf80954c0 2021-02-10T11:36:07.701 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:07.699 7fadfd734700 1 -- 172.21.2.102:0/3758588851 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7fadf80fe1c0 con 0x7fadf80954c0 2021-02-10T11:36:07.701 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:07.699 7fadeffff700 1 -- 172.21.2.102:0/3758588851 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v153) v1 ==== 100+0+10 (secure 0 0 0) 0x7fade8006030 con 0x7fadf80954c0 2021-02-10T11:36:07.701 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:36:07.703 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:07.701 7fadfd734700 1 -- 172.21.2.102:0/3758588851 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7faddc0186c0 msgr2=0x7faddc01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:07.703 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:07.701 7fadfd734700 1 --2- 172.21.2.102:0/3758588851 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7faddc0186c0 0x7faddc01abc0 crc :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:07.703 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:07.701 7fadfd734700 1 -- 172.21.2.102:0/3758588851 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fadf80954c0 msgr2=0x7fadf8096300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:07.703 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:07.701 7fadfd734700 1 --2- 172.21.2.102:0/3758588851 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fadf80954c0 0x7fadf8096300 secure :-1 s=READY pgs=2788 cs=0 l=1 rev1=1 rx=0x7fade800b580 tx=0x7fade800b920).stop 2021-02-10T11:36:07.703 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:07.701 7fadfd734700 1 -- 172.21.2.102:0/3758588851 shutdown_connections 2021-02-10T11:36:07.704 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:07.701 7fadfd734700 1 --2- 172.21.2.102:0/3758588851 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fadf80958c0 0x7fadf8096920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:07.704 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:07.701 7fadfd734700 1 --2- 172.21.2.102:0/3758588851 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fadf80954c0 0x7fadf8096300 unknown :-1 s=CLOSED pgs=2788 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:07.704 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:07.701 7fadfd734700 1 --2- 172.21.2.102:0/3758588851 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7faddc0186c0 0x7faddc01abc0 unknown :-1 s=CLOSED pgs=1226 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:07.704 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:07.701 7fadfd734700 1 --2- 172.21.2.102:0/3758588851 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fadf8085bc0 0x7fadf80863e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:07.704 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:07.701 7fadfd734700 1 -- 172.21.2.102:0/3758588851 >> 172.21.2.102:0/3758588851 conn(0x7fadf80bef00 msgr2=0x7fadf809d1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:07.704 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:07.701 7fadfd734700 1 -- 172.21.2.102:0/3758588851 shutdown_connections 2021-02-10T11:36:07.704 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:07.701 7fadfd734700 1 -- 172.21.2.102:0/3758588851 wait complete. 2021-02-10T11:36:07.712 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-02-10T11:36:07.712 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:36:08.141 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:08.122 7f4346748700 1 Processor -- start 2021-02-10T11:36:08.141 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:08.122 7f4346748700 1 -- start start 2021-02-10T11:36:08.141 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:08.122 7f4346748700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f43400d1870 0x7f43401b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:08.141 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:08.122 7f4346748700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f43401b0a10 0x7f43400cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:08.141 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:08.122 7f4346748700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f43400d1c70 0x7f43400cf950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:08.141 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:08.122 7f4346748700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43400fe230 con 0x7f43401b0a10 2021-02-10T11:36:08.141 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:08.123 7f4346748700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43400e5ed0 con 0x7f43400d1c70 2021-02-10T11:36:08.141 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:08.123 7f4346748700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43400f0b10 con 0x7f43400d1870 2021-02-10T11:36:08.141 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:08.123 7f433f7fe700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f43401b0a10 0x7f43400cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:08.141 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:08.123 7f433f7fe700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f43401b0a10 0x7f43400cf330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:37618/0 (socket says 172.21.2.102:37618) 2021-02-10T11:36:08.142 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:08.123 7f433f7fe700 1 -- 172.21.2.102:0/4008617887 learned_addr learned my addr 172.21.2.102:0/4008617887 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:36:08.142 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:08.123 7f433f7fe700 1 -- 172.21.2.102:0/4008617887 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f43400d1870 msgr2=0x7f43401b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:08.142 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:08.123 7f4344d65700 1 --2- 172.21.2.102:0/4008617887 >> v2:172.21.2.113:3300/0 conn(0x7f43400d1c70 0x7f43400cf950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:08.142 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:08.123 7f433ffff700 1 --2- 172.21.2.102:0/4008617887 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f43400d1870 0x7f43401b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:08.142 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:08.123 7f433f7fe700 1 --2- 172.21.2.102:0/4008617887 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f43400d1870 0x7f43401b03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:08.142 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:08.123 7f433f7fe700 1 -- 172.21.2.102:0/4008617887 >> v2:172.21.2.113:3300/0 conn(0x7f43400d1c70 msgr2=0x7f43400cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:08.142 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:08.123 7f433f7fe700 1 --2- 172.21.2.102:0/4008617887 >> v2:172.21.2.113:3300/0 conn(0x7f43400d1c70 0x7f43400cf950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:08.142 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:08.123 7f433f7fe700 1 -- 172.21.2.102:0/4008617887 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43400cff70 con 0x7f43401b0a10 2021-02-10T11:36:08.142 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:08.123 7f4344d65700 1 --2- 172.21.2.102:0/4008617887 >> v2:172.21.2.113:3300/0 conn(0x7f43400d1c70 0x7f43400cf950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:36:08.142 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:08.123 7f433ffff700 1 --2- 172.21.2.102:0/4008617887 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f43400d1870 0x7f43401b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:36:08.142 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:08.123 7f433f7fe700 1 --2- 172.21.2.102:0/4008617887 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f43401b0a10 0x7f43400cf330 secure :-1 s=READY pgs=2789 cs=0 l=1 rev1=1 rx=0x7f4334006700 tx=0x7f4334009b80).ready entity=mon.0 client_cookie=45d7dab1254f1867 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:08.142 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:08.123 7f433effd700 1 -- 172.21.2.102:0/4008617887 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4334006340 con 0x7f43401b0a10 2021-02-10T11:36:08.143 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:08.124 7f433effd700 1 -- 172.21.2.102:0/4008617887 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f433400c0c0 con 0x7f43401b0a10 2021-02-10T11:36:08.143 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:08.124 7f433effd700 1 -- 172.21.2.102:0/4008617887 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f433400c560 con 0x7f43401b0a10 2021-02-10T11:36:08.143 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:08.124 7f4346748700 1 -- 172.21.2.102:0/4008617887 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f43401b0a10 msgr2=0x7f43400cf330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:08.143 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:08.124 7f4346748700 1 --2- 172.21.2.102:0/4008617887 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f43401b0a10 0x7f43400cf330 secure :-1 s=READY pgs=2789 cs=0 l=1 rev1=1 rx=0x7f4334006700 tx=0x7f4334009b80).stop 2021-02-10T11:36:08.143 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:08.124 7f4346748700 1 -- 172.21.2.102:0/4008617887 shutdown_connections 2021-02-10T11:36:08.143 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:08.124 7f4346748700 1 --2- 172.21.2.102:0/4008617887 >> v2:172.21.2.113:3300/0 conn(0x7f43400d1c70 0x7f43400cf950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:08.143 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:08.124 7f4346748700 1 --2- 172.21.2.102:0/4008617887 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f43401b0a10 0x7f43400cf330 unknown :-1 s=CLOSED pgs=2789 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:08.143 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:08.124 7f4346748700 1 --2- 172.21.2.102:0/4008617887 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f43400d1870 0x7f43401b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:08.143 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:08.124 7f4346748700 1 -- 172.21.2.102:0/4008617887 >> 172.21.2.102:0/4008617887 conn(0x7f43400ce450 msgr2=0x7f43400d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:08.143 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:08.124 7f4346748700 1 -- 172.21.2.102:0/4008617887 shutdown_connections 2021-02-10T11:36:08.143 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:08.124 7f4346748700 1 -- 172.21.2.102:0/4008617887 wait complete. 2021-02-10T11:36:08.143 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:08.139 7f4346748700 1 Processor -- start 2021-02-10T11:36:08.145 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:08.139 7f4346748700 1 -- start start 2021-02-10T11:36:08.145 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:08.139 7f4346748700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f434018d870 0x7f43400b19b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:08.145 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:08.139 7f4346748700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f43400b1fd0 0x7f4340099ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:08.145 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:08.139 7f4346748700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f43400b23d0 0x7f434009a510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:08.145 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:08.139 7f4346748700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f434018dc70 con 0x7f434018d870 2021-02-10T11:36:08.145 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:08.139 7f4346748700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4340199e60 con 0x7f43400b23d0 2021-02-10T11:36:08.145 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:08.139 7f4346748700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f434009ab30 con 0x7f43400b1fd0 2021-02-10T11:36:08.145 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:08.140 7f433ffff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f434018d870 0x7f43400b19b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:08.145 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:08.140 7f433ffff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f434018d870 0x7f43400b19b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:37624/0 (socket says 172.21.2.102:37624) 2021-02-10T11:36:08.145 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:08.140 7f433ffff700 1 -- 172.21.2.102:0/3761853955 learned_addr learned my addr 172.21.2.102:0/3761853955 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:36:08.145 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:08.140 7f433ffff700 1 -- 172.21.2.102:0/3761853955 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f43400b1fd0 msgr2=0x7f4340099ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:08.145 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:08.140 7f433ffff700 1 --2- 172.21.2.102:0/3761853955 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f43400b1fd0 0x7f4340099ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:08.146 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:08.140 7f433ffff700 1 -- 172.21.2.102:0/3761853955 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f43400b23d0 msgr2=0x7f434009a510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:08.146 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:08.140 7f433ffff700 1 --2- 172.21.2.102:0/3761853955 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f43400b23d0 0x7f434009a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:08.146 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:08.140 7f433ffff700 1 -- 172.21.2.102:0/3761853955 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f434009ad50 con 0x7f434018d870 2021-02-10T11:36:08.146 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:08.140 7f433ffff700 1 --2- 172.21.2.102:0/3761853955 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f434018d870 0x7f43400b19b0 secure :-1 s=READY pgs=2790 cs=0 l=1 rev1=1 rx=0x7f4330006f20 tx=0x7f433000b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:08.146 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:08.140 7f433cff9700 1 -- 172.21.2.102:0/3761853955 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f433000bd80 con 0x7f434018d870 2021-02-10T11:36:08.146 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:08.140 7f433cff9700 1 -- 172.21.2.102:0/3761853955 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4330000c90 con 0x7f434018d870 2021-02-10T11:36:08.146 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:08.140 7f4346748700 1 -- 172.21.2.102:0/3761853955 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f434009ad50 con 0x7f434018d870 2021-02-10T11:36:08.146 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:08.140 7f4346748700 1 -- 172.21.2.102:0/3761853955 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f434009ad50 con 0x7f434018d870 2021-02-10T11:36:08.146 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:08.141 7f433cff9700 1 -- 172.21.2.102:0/3761853955 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f43300022b0 con 0x7f434018d870 2021-02-10T11:36:08.146 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:08.141 7f433cff9700 1 -- 172.21.2.102:0/3761853955 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f433000a000 con 0x7f434018d870 2021-02-10T11:36:08.146 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:08.141 7f433cff9700 1 --2- 172.21.2.102:0/3761853955 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f43200186c0 0x7f432001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:08.146 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:08.141 7f433cff9700 1 -- 172.21.2.102:0/3761853955 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(153..153 src has 1..153) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f4330032540 con 0x7f434018d870 2021-02-10T11:36:08.146 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:08.141 7f433f7fe700 1 --2- 172.21.2.102:0/3761853955 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f43200186c0 0x7f432001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:08.147 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:08.141 7f433f7fe700 1 --2- 172.21.2.102:0/3761853955 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f43200186c0 0x7f432001abc0 crc :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:08.174 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:08.172 7f4346748700 1 -- 172.21.2.102:0/3761853955 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43400999c0 con 0x7f434018d870 2021-02-10T11:36:08.176 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:08.175 7f433cff9700 1 -- 172.21.2.102:0/3761853955 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f433001c3f0 con 0x7f434018d870 2021-02-10T11:36:08.304 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:08.302 7f4346748700 1 -- 172.21.2.102:0/3761853955 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f43400999c0 con 0x7f434018d870 2021-02-10T11:36:08.304 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:08.302 7f433cff9700 1 -- 172.21.2.102:0/3761853955 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 132 v132) v1 ==== 95+0+2107 (secure 0 0 0) 0x7f433001c3f0 con 0x7f434018d870 2021-02-10T11:36:08.305 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:36:08.305 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":132,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7004,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.102:6835/3280534775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6834","nonce":3280534775},{"type":"v1","addr":"172.21.2.102:6835","nonce":3280534775}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":128}],"filesystems":[{"mdsmap":{"epoch":132,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:35:58.285904","modified":"2021-02-10 11:36:00.292711","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7001},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7001":{"gid":7001,"name":"b","rank":0,"incarnation":129,"state":"up:active","state_seq":3,"addr":"172.21.2.113:6833/1983258405","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6832","nonce":1983258405},{"type":"v1","addr":"172.21.2.113:6833","nonce":1983258405}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[33],"metadata_pool":32,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":16}]} 2021-02-10T11:36:08.305 INFO:teuthology.orchestra.run.gibba002.stderr:dumped fsmap epoch 132 2021-02-10T11:36:08.307 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:08.304 7f4346748700 1 -- 172.21.2.102:0/3761853955 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f43200186c0 msgr2=0x7f432001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:08.307 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:08.304 7f4346748700 1 --2- 172.21.2.102:0/3761853955 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f43200186c0 0x7f432001abc0 crc :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:08.307 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:08.304 7f4346748700 1 -- 172.21.2.102:0/3761853955 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f434018d870 msgr2=0x7f43400b19b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:08.307 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:08.304 7f4346748700 1 --2- 172.21.2.102:0/3761853955 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f434018d870 0x7f43400b19b0 secure :-1 s=READY pgs=2790 cs=0 l=1 rev1=1 rx=0x7f4330006f20 tx=0x7f433000b990).stop 2021-02-10T11:36:08.307 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:08.305 7f4346748700 1 -- 172.21.2.102:0/3761853955 shutdown_connections 2021-02-10T11:36:08.307 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:08.305 7f4346748700 1 --2- 172.21.2.102:0/3761853955 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f43200186c0 0x7f432001abc0 unknown :-1 s=CLOSED pgs=1227 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:08.307 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:08.305 7f4346748700 1 --2- 172.21.2.102:0/3761853955 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f43400b23d0 0x7f434009a510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:08.307 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:08.305 7f4346748700 1 --2- 172.21.2.102:0/3761853955 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f43400b1fd0 0x7f4340099ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:08.307 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:08.305 7f4346748700 1 --2- 172.21.2.102:0/3761853955 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f434018d870 0x7f43400b19b0 unknown :-1 s=CLOSED pgs=2790 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:08.307 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:08.305 7f4346748700 1 -- 172.21.2.102:0/3761853955 >> 172.21.2.102:0/3761853955 conn(0x7f43400f0550 msgr2=0x7f4340191730 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:08.307 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:08.305 7f4346748700 1 -- 172.21.2.102:0/3761853955 shutdown_connections 2021-02-10T11:36:08.307 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:08.305 7f4346748700 1 -- 172.21.2.102:0/3761853955 wait complete. 2021-02-10T11:36:08.517 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 132, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:35:58.285904', 'modified': '2021-02-10 11:36:00.292711', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7001}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7001': {'gid': 7001, 'name': 'b', 'rank': 0, 'incarnation': 129, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.113:6833/1983258405', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.113:6832', 'nonce': 1983258405}, {'type': 'v1', 'addr': '172.21.2.113:6833', 'nonce': 1983258405}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [33], 'metadata_pool': 32, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:36:08.517 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:36:08.517 DEBUG:teuthology.orchestra.run.gibba013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:36:08.917 INFO:teuthology.orchestra.run.gibba013.stdout:{ 2021-02-10T11:36:08.917 INFO:teuthology.orchestra.run.gibba013.stdout: "cluster_fsid": "68db30ea-dbce-4aef-a6df-f01c6fea0513", 2021-02-10T11:36:08.917 INFO:teuthology.orchestra.run.gibba013.stdout: "whoami": 0, 2021-02-10T11:36:08.917 INFO:teuthology.orchestra.run.gibba013.stdout: "id": 7001, 2021-02-10T11:36:08.917 INFO:teuthology.orchestra.run.gibba013.stdout: "want_state": "up:active", 2021-02-10T11:36:08.917 INFO:teuthology.orchestra.run.gibba013.stdout: "state": "up:active", 2021-02-10T11:36:08.917 INFO:teuthology.orchestra.run.gibba013.stdout: "rank_uptime": 10.627184365, 2021-02-10T11:36:08.917 INFO:teuthology.orchestra.run.gibba013.stdout: "mdsmap_epoch": 132, 2021-02-10T11:36:08.917 INFO:teuthology.orchestra.run.gibba013.stdout: "osdmap_epoch": 153, 2021-02-10T11:36:08.918 INFO:teuthology.orchestra.run.gibba013.stdout: "osdmap_epoch_barrier": 153, 2021-02-10T11:36:08.918 INFO:teuthology.orchestra.run.gibba013.stdout: "uptime": 16.273905516999999 2021-02-10T11:36:08.918 INFO:teuthology.orchestra.run.gibba013.stdout:} 2021-02-10T11:36:08.921 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "68db30ea-dbce-4aef-a6df-f01c6fea0513", "whoami": 0, "id": 7001, "want_state": "up:active", "state": "up:active", "rank_uptime": 10.627184365, "mdsmap_epoch": 132, "osdmap_epoch": 153, "osdmap_epoch_barrier": 153, "uptime": 16.273905516999999 } 2021-02-10T11:36:08.921 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2021-02-10T11:36:08.921 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2021-02-10T11:36:08.921 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@gibba002.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2021-02-10T11:36:08.922 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:36:08.922 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec mkdir -p /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:36:09.257 DEBUG:teuthology.orchestra.run.gibba002:> sudo modprobe fuse 2021-02-10T11:36:09.601 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:36:09.601 DEBUG:teuthology.orchestra.run.gibba002:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:36:09.944 INFO:teuthology.orchestra.run.gibba002.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:36:09.945 INFO:teuthology.orchestra.run.gibba002.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:36:09.945 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:36:09.945 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:36:09.946 DEBUG:teuthology.orchestra.run.gibba002:> ls /sys/fs/fuse/connections 2021-02-10T11:36:10.280 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2021-02-10T11:36:10.280 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:36:10.283 DEBUG:teuthology.orchestra.run.gibba002:> sudo modprobe fuse 2021-02-10T11:36:10.627 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:36:10.627 DEBUG:teuthology.orchestra.run.gibba002:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:36:10.693 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba002.stderr:ceph-fuse[200903]: starting ceph client 2021-02-10T11:36:10.693 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba002.stderr:2021-02-10 11:36:10.690 7f7db1a2cf80 -1 init, newargv = 0x55f6ea6b5220 newargc=9 2021-02-10T11:36:10.698 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba002.stderr:ceph-fuse[200903]: starting fuse 2021-02-10T11:36:10.970 INFO:teuthology.orchestra.run.gibba002.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:36:10.970 INFO:teuthology.orchestra.run.gibba002.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:36:10.971 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:36:10.971 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:36:10.971 DEBUG:teuthology.orchestra.run.gibba002:> ls /sys/fs/fuse/connections 2021-02-10T11:36:11.306 INFO:teuthology.orchestra.run.gibba002.stdout:43 2021-02-10T11:36:11.306 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2021-02-10T11:36:11.307 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:36:11.307 DEBUG:teuthology.orchestra.run.gibba002:> sudo python3 -c ' 2021-02-10T11:36:11.307 DEBUG:teuthology.orchestra.run.gibba002:> import glob 2021-02-10T11:36:11.307 DEBUG:teuthology.orchestra.run.gibba002:> import re 2021-02-10T11:36:11.307 DEBUG:teuthology.orchestra.run.gibba002:> import os 2021-02-10T11:36:11.307 DEBUG:teuthology.orchestra.run.gibba002:> import subprocess 2021-02-10T11:36:11.307 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:36:11.307 DEBUG:teuthology.orchestra.run.gibba002:> def find_socket(client_name): 2021-02-10T11:36:11.307 DEBUG:teuthology.orchestra.run.gibba002:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2021-02-10T11:36:11.307 DEBUG:teuthology.orchestra.run.gibba002:> files = glob.glob(asok_path) 2021-02-10T11:36:11.307 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:36:11.307 DEBUG:teuthology.orchestra.run.gibba002:> # Given a non-glob path, it better be there 2021-02-10T11:36:11.308 DEBUG:teuthology.orchestra.run.gibba002:> if "*" not in asok_path: 2021-02-10T11:36:11.308 DEBUG:teuthology.orchestra.run.gibba002:> assert(len(files) == 1) 2021-02-10T11:36:11.308 DEBUG:teuthology.orchestra.run.gibba002:> return files[0] 2021-02-10T11:36:11.308 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:36:11.308 DEBUG:teuthology.orchestra.run.gibba002:> for f in files: 2021-02-10T11:36:11.308 DEBUG:teuthology.orchestra.run.gibba002:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2021-02-10T11:36:11.308 DEBUG:teuthology.orchestra.run.gibba002:> if os.path.exists("/proc/{0}".format(pid)): 2021-02-10T11:36:11.308 DEBUG:teuthology.orchestra.run.gibba002:> return f 2021-02-10T11:36:11.308 DEBUG:teuthology.orchestra.run.gibba002:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2021-02-10T11:36:11.308 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:36:11.308 DEBUG:teuthology.orchestra.run.gibba002:> print(find_socket("client.0")) 2021-02-10T11:36:11.308 DEBUG:teuthology.orchestra.run.gibba002:> ' 2021-02-10T11:36:11.671 INFO:teuthology.orchestra.run.gibba002.stdout:/var/run/ceph/ceph-client.0.200903.asok 2021-02-10T11:36:11.673 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.200903.asok 2021-02-10T11:36:11.674 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:36:11.674 DEBUG:teuthology.orchestra.run.gibba002:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.200903.asok status 2021-02-10T11:36:12.066 INFO:teuthology.orchestra.run.gibba002.stdout:{ 2021-02-10T11:36:12.066 INFO:teuthology.orchestra.run.gibba002.stdout: "metadata": { 2021-02-10T11:36:12.067 INFO:teuthology.orchestra.run.gibba002.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:36:12.067 INFO:teuthology.orchestra.run.gibba002.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:36:12.067 INFO:teuthology.orchestra.run.gibba002.stdout: "entity_id": "0", 2021-02-10T11:36:12.067 INFO:teuthology.orchestra.run.gibba002.stdout: "hostname": "gibba002", 2021-02-10T11:36:12.067 INFO:teuthology.orchestra.run.gibba002.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2021-02-10T11:36:12.067 INFO:teuthology.orchestra.run.gibba002.stdout: "pid": "200903", 2021-02-10T11:36:12.067 INFO:teuthology.orchestra.run.gibba002.stdout: "root": "/" 2021-02-10T11:36:12.067 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:36:12.067 INFO:teuthology.orchestra.run.gibba002.stdout: "dentry_count": 0, 2021-02-10T11:36:12.067 INFO:teuthology.orchestra.run.gibba002.stdout: "dentry_pinned_count": 0, 2021-02-10T11:36:12.067 INFO:teuthology.orchestra.run.gibba002.stdout: "id": 12246, 2021-02-10T11:36:12.067 INFO:teuthology.orchestra.run.gibba002.stdout: "inst": { 2021-02-10T11:36:12.068 INFO:teuthology.orchestra.run.gibba002.stdout: "name": { 2021-02-10T11:36:12.068 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "client", 2021-02-10T11:36:12.068 INFO:teuthology.orchestra.run.gibba002.stdout: "num": 12246 2021-02-10T11:36:12.068 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:36:12.068 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": { 2021-02-10T11:36:12.068 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:36:12.068 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:0", 2021-02-10T11:36:12.068 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 4217397809 2021-02-10T11:36:12.068 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:36:12.068 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:36:12.068 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": { 2021-02-10T11:36:12.068 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:36:12.068 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:0", 2021-02-10T11:36:12.069 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 4217397809 2021-02-10T11:36:12.069 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:36:12.069 INFO:teuthology.orchestra.run.gibba002.stdout: "inst_str": "client.12246 172.21.2.102:0/4217397809", 2021-02-10T11:36:12.069 INFO:teuthology.orchestra.run.gibba002.stdout: "addr_str": "172.21.2.102:0/4217397809", 2021-02-10T11:36:12.069 INFO:teuthology.orchestra.run.gibba002.stdout: "inode_count": 1, 2021-02-10T11:36:12.069 INFO:teuthology.orchestra.run.gibba002.stdout: "mds_epoch": 132, 2021-02-10T11:36:12.069 INFO:teuthology.orchestra.run.gibba002.stdout: "osd_epoch": 153, 2021-02-10T11:36:12.069 INFO:teuthology.orchestra.run.gibba002.stdout: "osd_epoch_barrier": 0, 2021-02-10T11:36:12.069 INFO:teuthology.orchestra.run.gibba002.stdout: "blacklisted": false 2021-02-10T11:36:12.069 INFO:teuthology.orchestra.run.gibba002.stdout:} 2021-02-10T11:36:12.071 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:36:12.071 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:36:12.071 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:36:12.408 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:36:12.408 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:36:12.409 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:36:12.409 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:36:12.756 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:36:12.948 DEBUG:teuthology.orchestra.run.gibba002:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:36:12.951 DEBUG:teuthology.orchestra.run.gibba013:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:36:13.184 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:13.165 7f784e7ac700 1 Processor -- start 2021-02-10T11:36:13.184 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:13.165 7f784e7ac700 1 -- start start 2021-02-10T11:36:13.184 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:13.166 7f784e7ac700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f78480d0340 0x7f78480ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:13.184 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:13.166 7f784e7ac700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f78480d1c70 0x7f78480efc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:13.185 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:13.166 7f784e7ac700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78480f0b10 con 0x7f78480d1870 2021-02-10T11:36:13.185 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:13.166 7f784e7ac700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78480e5ed0 con 0x7f78480d1c70 2021-02-10T11:36:13.185 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:13.166 7f784e7ac700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78481b0be0 con 0x7f78480d0340 2021-02-10T11:36:13.185 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:13.166 7f784cdc9700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f78480d1c70 0x7f78480efc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:13.185 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:13.166 7f784cdc9700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f78480d1c70 0x7f78480efc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:37642/0 (socket says 172.21.2.102:37642) 2021-02-10T11:36:13.185 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:13.166 7f784cdc9700 1 -- 172.21.2.102:0/738810588 learned_addr learned my addr 172.21.2.102:0/738810588 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:36:13.185 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:13.166 7f784cdc9700 1 -- 172.21.2.102:0/738810588 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f78480d0340 msgr2=0x7f78480ef640 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:36:13.185 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:13.166 7f784cdc9700 1 --2- 172.21.2.102:0/738810588 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f78480d0340 0x7f78480ef640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:13.185 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:13.166 7f784cdc9700 1 -- 172.21.2.102:0/738810588 >> v1:172.21.2.113:6789/0 conn(0x7f78480d1870 legacy=0x7f78481b03f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:36:13.185 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:13.166 7f784cdc9700 1 -- 172.21.2.102:0/738810588 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78480f20a0 con 0x7f78480d1c70 2021-02-10T11:36:13.185 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:13.167 7f784cdc9700 1 --2- 172.21.2.102:0/738810588 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f78480d1c70 0x7f78480efc60 secure :-1 s=READY pgs=2794 cs=0 l=1 rev1=1 rx=0x7f7840006f00 tx=0x7f7840009e90).ready entity=mon.0 client_cookie=e074f747bc5cc98a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:13.185 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:13.167 7f7846ffd700 1 -- 172.21.2.102:0/738810588 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f784000b300 con 0x7f78480d1c70 2021-02-10T11:36:13.186 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:13.167 7f7846ffd700 1 -- 172.21.2.102:0/738810588 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f784000b580 con 0x7f78480d1c70 2021-02-10T11:36:13.186 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:13.167 7f7846ffd700 1 -- 172.21.2.102:0/738810588 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f784000b9d0 con 0x7f78480d1c70 2021-02-10T11:36:13.186 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:13.167 7f784e7ac700 1 -- 172.21.2.102:0/738810588 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f78480d1c70 msgr2=0x7f78480efc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:13.186 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:13.167 7f784e7ac700 1 --2- 172.21.2.102:0/738810588 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f78480d1c70 0x7f78480efc60 secure :-1 s=READY pgs=2794 cs=0 l=1 rev1=1 rx=0x7f7840006f00 tx=0x7f7840009e90).stop 2021-02-10T11:36:13.186 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:13.168 7f784e7ac700 1 -- 172.21.2.102:0/738810588 shutdown_connections 2021-02-10T11:36:13.186 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:13.168 7f784e7ac700 1 --2- 172.21.2.102:0/738810588 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f78480d0340 0x7f78480ef640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:13.186 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:13.168 7f784e7ac700 1 --2- 172.21.2.102:0/738810588 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f78480d1c70 0x7f78480efc60 unknown :-1 s=CLOSED pgs=2794 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:13.186 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:13.168 7f784e7ac700 1 -- 172.21.2.102:0/738810588 >> 172.21.2.102:0/738810588 conn(0x7f78480ce450 msgr2=0x7f78480d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:13.186 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:13.168 7f784e7ac700 1 -- 172.21.2.102:0/738810588 shutdown_connections 2021-02-10T11:36:13.186 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:13.168 7f784e7ac700 1 -- 172.21.2.102:0/738810588 wait complete. 2021-02-10T11:36:13.188 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:13.183 7f784e7ac700 1 Processor -- start 2021-02-10T11:36:13.188 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:13.183 7f784e7ac700 1 -- start start 2021-02-10T11:36:13.188 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:13.183 7f784e7ac700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f784819ede0 0x7f784819f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:13.188 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:13.183 7f784e7ac700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f78481aca40 0x7f78481ace40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:13.188 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:13.183 7f784e7ac700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f784819d3c0 0x7f784819d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:13.188 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:13.183 7f784e7ac700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78481ad460 con 0x7f784819d3c0 2021-02-10T11:36:13.188 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:13.183 7f784e7ac700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f784819fca0 con 0x7f78481aca40 2021-02-10T11:36:13.188 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:13.183 7f784e7ac700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f784819dde0 con 0x7f784819ede0 2021-02-10T11:36:13.188 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:13.183 7f784cdc9700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f784819d3c0 0x7f784819d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:13.188 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:13.183 7f784cdc9700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f784819d3c0 0x7f784819d7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:37652/0 (socket says 172.21.2.102:37652) 2021-02-10T11:36:13.189 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:13.183 7f784cdc9700 1 -- 172.21.2.102:0/3979738219 learned_addr learned my addr 172.21.2.102:0/3979738219 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:36:13.189 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:13.184 7f784cdc9700 1 -- 172.21.2.102:0/3979738219 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f784819ede0 msgr2=0x7f784819f1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:13.189 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:13.184 7f784cdc9700 1 --2- 172.21.2.102:0/3979738219 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f784819ede0 0x7f784819f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:13.189 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:13.184 7f784cdc9700 1 -- 172.21.2.102:0/3979738219 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f78481aca40 msgr2=0x7f78481ace40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:13.189 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:13.184 7f784cdc9700 1 --2- 172.21.2.102:0/3979738219 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f78481aca40 0x7f78481ace40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:13.189 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:13.184 7f784cdc9700 1 -- 172.21.2.102:0/3979738219 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f784819e000 con 0x7f784819d3c0 2021-02-10T11:36:13.189 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:13.184 7f784cdc9700 1 --2- 172.21.2.102:0/3979738219 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f784819d3c0 0x7f784819d7c0 secure :-1 s=READY pgs=2795 cs=0 l=1 rev1=1 rx=0x7f784000b910 tx=0x7f784000bcb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:13.189 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:13.184 7f7844ff9700 1 -- 172.21.2.102:0/3979738219 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f784000bdf0 con 0x7f784819d3c0 2021-02-10T11:36:13.189 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:13.184 7f7844ff9700 1 -- 172.21.2.102:0/3979738219 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7840005820 con 0x7f784819d3c0 2021-02-10T11:36:13.189 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:13.184 7f7844ff9700 1 -- 172.21.2.102:0/3979738219 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7840005c70 con 0x7f784819d3c0 2021-02-10T11:36:13.189 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:13.184 7f784e7ac700 1 -- 172.21.2.102:0/3979738219 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f784819e000 con 0x7f784819d3c0 2021-02-10T11:36:13.189 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:13.184 7f784e7ac700 1 -- 172.21.2.102:0/3979738219 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f784819e590 con 0x7f784819d3c0 2021-02-10T11:36:13.189 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:13.185 7f7844ff9700 1 -- 172.21.2.102:0/3979738219 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f7840005a00 con 0x7f784819d3c0 2021-02-10T11:36:13.190 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:13.185 7f7844ff9700 1 --2- 172.21.2.102:0/3979738219 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f78280186c0 0x7f782801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:13.190 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:13.185 7f7844ff9700 1 -- 172.21.2.102:0/3979738219 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(153..153 src has 1..153) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f7840032930 con 0x7f784819d3c0 2021-02-10T11:36:13.190 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:13.185 7f7847fff700 1 --2- 172.21.2.102:0/3979738219 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f78280186c0 0x7f782801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:13.190 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:13.185 7f7847fff700 1 --2- 172.21.2.102:0/3979738219 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f78280186c0 0x7f782801abc0 crc :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:13.218 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:13.216 7f784e7ac700 1 -- 172.21.2.102:0/3979738219 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f784819e8b0 con 0x7f784819d3c0 2021-02-10T11:36:13.220 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:13.219 7f7844ff9700 1 -- 172.21.2.102:0/3979738219 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f784002dfa0 con 0x7f784819d3c0 2021-02-10T11:36:13.347 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:13.345 7f784e7ac700 1 -- 172.21.2.102:0/3979738219 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f784819e8b0 con 0x7f784819d3c0 2021-02-10T11:36:13.348 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:13.346 7f7844ff9700 1 -- 172.21.2.102:0/3979738219 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v153) v1 ==== 74+0+14007 (secure 0 0 0) 0x7f784002dfa0 con 0x7f784819d3c0 2021-02-10T11:36:13.348 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:36:13.348 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":153,"fsid":"68db30ea-dbce-4aef-a6df-f01c6fea0513","created":"2021-02-10 11:03:06.331557","modified":"2021-02-10 11:35:59.288212","last_up_change":"2021-02-10 11:03:32.690562","last_in_change":"2021-02-10 11:03:32.690562","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":33,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:03:37.322519","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":32,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:35:56.251556","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"152","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":33,"pool_name":"cephfs_data","create_time":"2021-02-10 11:35:57.328218","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"153","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"be691b4d-a039-408a-af80-9d53f02b6f1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":150,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6818","nonce":36088},{"type":"v1","addr":"172.21.2.102:6820","nonce":36088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6822","nonce":36088},{"type":"v1","addr":"172.21.2.102:6824","nonce":36088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6830","nonce":36088},{"type":"v1","addr":"172.21.2.102:6832","nonce":36088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6826","nonce":36088},{"type":"v1","addr":"172.21.2.102:6828","nonce":36088}]},"public_addr":"172.21.2.102:6820/36088","cluster_addr":"172.21.2.102:6824/36088","heartbeat_back_addr":"172.21.2.102:6832/36088","heartbeat_front_addr":"172.21.2.102:6828/36088","state":["exists","up"]},{"osd":1,"uuid":"64e3ccc8-9c69-4d7c-865e-af8e83822635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6819","nonce":36087},{"type":"v1","addr":"172.21.2.102:6821","nonce":36087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6823","nonce":36087},{"type":"v1","addr":"172.21.2.102:6825","nonce":36087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6831","nonce":36087},{"type":"v1","addr":"172.21.2.102:6833","nonce":36087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6827","nonce":36087},{"type":"v1","addr":"172.21.2.102:6829","nonce":36087}]},"public_addr":"172.21.2.102:6821/36087","cluster_addr":"172.21.2.102:6825/36087","heartbeat_back_addr":"172.21.2.102:6833/36087","heartbeat_front_addr":"172.21.2.102:6829/36087","state":["exists","up"]},{"osd":2,"uuid":"153700af-747f-481f-bdf4-d6072d3a842f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6810","nonce":36089},{"type":"v1","addr":"172.21.2.102:6811","nonce":36089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6812","nonce":36089},{"type":"v1","addr":"172.21.2.102:6813","nonce":36089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6816","nonce":36089},{"type":"v1","addr":"172.21.2.102:6817","nonce":36089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6814","nonce":36089},{"type":"v1","addr":"172.21.2.102:6815","nonce":36089}]},"public_addr":"172.21.2.102:6811/36089","cluster_addr":"172.21.2.102:6813/36089","heartbeat_back_addr":"172.21.2.102:6817/36089","heartbeat_front_addr":"172.21.2.102:6815/36089","state":["exists","up"]},{"osd":3,"uuid":"cbc678f4-83a9-45e6-8830-2f7c232b1ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6802","nonce":36090},{"type":"v1","addr":"172.21.2.102:6803","nonce":36090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6804","nonce":36090},{"type":"v1","addr":"172.21.2.102:6805","nonce":36090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6808","nonce":36090},{"type":"v1","addr":"172.21.2.102:6809","nonce":36090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6806","nonce":36090},{"type":"v1","addr":"172.21.2.102:6807","nonce":36090}]},"public_addr":"172.21.2.102:6803/36090","cluster_addr":"172.21.2.102:6805/36090","heartbeat_back_addr":"172.21.2.102:6809/36090","heartbeat_front_addr":"172.21.2.102:6807/36090","state":["exists","up"]},{"osd":4,"uuid":"c01e4008-b258-4f10-afd2-9e7f37214351","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6808","nonce":36368},{"type":"v1","addr":"172.21.2.113:6811","nonce":36368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6814","nonce":36368},{"type":"v1","addr":"172.21.2.113:6818","nonce":36368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6826","nonce":36368},{"type":"v1","addr":"172.21.2.113:6828","nonce":36368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6820","nonce":36368},{"type":"v1","addr":"172.21.2.113:6824","nonce":36368}]},"public_addr":"172.21.2.113:6811/36368","cluster_addr":"172.21.2.113:6818/36368","heartbeat_back_addr":"172.21.2.113:6828/36368","heartbeat_front_addr":"172.21.2.113:6824/36368","state":["exists","up"]},{"osd":5,"uuid":"551f6007-daf8-4a98-8ee0-c2b16bcccb5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6802","nonce":36366},{"type":"v1","addr":"172.21.2.113:6805","nonce":36366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6807","nonce":36366},{"type":"v1","addr":"172.21.2.113:6810","nonce":36366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6819","nonce":36366},{"type":"v1","addr":"172.21.2.113:6822","nonce":36366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6812","nonce":36366},{"type":"v1","addr":"172.21.2.113:6815","nonce":36366}]},"public_addr":"172.21.2.113:6805/36366","cluster_addr":"172.21.2.113:6810/36366","heartbeat_back_addr":"172.21.2.113:6822/36366","heartbeat_front_addr":"172.21.2.113:6815/36366","state":["exists","up"]},{"osd":6,"uuid":"bf67c040-0e50-48be-b102-c3943bc9ebef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":150,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6800","nonce":36367},{"type":"v1","addr":"172.21.2.113:6801","nonce":36367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6803","nonce":36367},{"type":"v1","addr":"172.21.2.113:6804","nonce":36367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6813","nonce":36367},{"type":"v1","addr":"172.21.2.113:6816","nonce":36367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6806","nonce":36367},{"type":"v1","addr":"172.21.2.113:6809","nonce":36367}]},"public_addr":"172.21.2.113:6801/36367","cluster_addr":"172.21.2.113:6804/36367","heartbeat_back_addr":"172.21.2.113:6816/36367","heartbeat_front_addr":"172.21.2.113:6809/36367","state":["exists","up"]},{"osd":7,"uuid":"f5461f80-018d-428f-be89-2eb2bcbe7bb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6817","nonce":36365},{"type":"v1","addr":"172.21.2.113:6821","nonce":36365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6823","nonce":36365},{"type":"v1","addr":"172.21.2.113:6825","nonce":36365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6830","nonce":36365},{"type":"v1","addr":"172.21.2.113:6831","nonce":36365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6827","nonce":36365},{"type":"v1","addr":"172.21.2.113:6829","nonce":36365}]},"public_addr":"172.21.2.113:6821/36365","cluster_addr":"172.21.2.113:6825/36365","heartbeat_back_addr":"172.21.2.113:6831/36365","heartbeat_front_addr":"172.21.2.113:6829/36365","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:36:13.350 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:13.348 7f784e7ac700 1 -- 172.21.2.102:0/3979738219 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f78280186c0 msgr2=0x7f782801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:13.350 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:13.348 7f784e7ac700 1 --2- 172.21.2.102:0/3979738219 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f78280186c0 0x7f782801abc0 crc :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:13.350 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:13.348 7f784e7ac700 1 -- 172.21.2.102:0/3979738219 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f784819d3c0 msgr2=0x7f784819d7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:13.351 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:13.348 7f784e7ac700 1 --2- 172.21.2.102:0/3979738219 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f784819d3c0 0x7f784819d7c0 secure :-1 s=READY pgs=2795 cs=0 l=1 rev1=1 rx=0x7f784000b910 tx=0x7f784000bcb0).stop 2021-02-10T11:36:13.351 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:13.348 7f784e7ac700 1 -- 172.21.2.102:0/3979738219 shutdown_connections 2021-02-10T11:36:13.351 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:13.348 7f784e7ac700 1 --2- 172.21.2.102:0/3979738219 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f784819d3c0 0x7f784819d7c0 unknown :-1 s=CLOSED pgs=2795 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:13.351 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:13.348 7f784e7ac700 1 --2- 172.21.2.102:0/3979738219 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f78280186c0 0x7f782801abc0 unknown :-1 s=CLOSED pgs=1228 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:13.351 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:13.348 7f784e7ac700 1 --2- 172.21.2.102:0/3979738219 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f78481aca40 0x7f78481ace40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:13.351 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:13.348 7f784e7ac700 1 --2- 172.21.2.102:0/3979738219 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f784819ede0 0x7f784819f1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:13.351 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:13.348 7f784e7ac700 1 -- 172.21.2.102:0/3979738219 >> 172.21.2.102:0/3979738219 conn(0x7f784819f880 msgr2=0x7f7848094ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:13.351 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:13.348 7f784e7ac700 1 -- 172.21.2.102:0/3979738219 shutdown_connections 2021-02-10T11:36:13.351 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:13.349 7f784e7ac700 1 -- 172.21.2.102:0/3979738219 wait complete. 2021-02-10T11:36:13.359 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:03:37.322519', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 32, 'pool_name': 'cephfs_metadata', 'create_time': '2021-02-10 11:35:56.251556', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '152', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 33, 'pool_name': 'cephfs_data', 'create_time': '2021-02-10 11:35:57.328218', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '153', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-02-10T11:36:13.359 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:36:13.828 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:13.810 7f5e2c0b6700 1 Processor -- start 2021-02-10T11:36:13.829 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:13.810 7f5e2c0b6700 1 -- start start 2021-02-10T11:36:13.829 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:13.811 7f5e2c0b6700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5e240d1870 0x7f5e241b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:13.829 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:13.811 7f5e2c0b6700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f5e241b0ae0 0x7f5e24108940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:13.829 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:13.811 7f5e2c0b6700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e24109a60 con 0x7f5e240d1c70 2021-02-10T11:36:13.829 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:13.811 7f5e2c0b6700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e240e5ef0 con 0x7f5e241b0ae0 2021-02-10T11:36:13.829 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:13.811 7f5e2c0b6700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e24109750 con 0x7f5e240d1870 2021-02-10T11:36:13.829 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:13.811 7f5e296d1700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f5e241b0ae0 0x7f5e24108940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:13.829 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:13.811 7f5e296d1700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f5e241b0ae0 0x7f5e24108940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:37656/0 (socket says 172.21.2.102:37656) 2021-02-10T11:36:13.829 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:13.811 7f5e296d1700 1 -- 172.21.2.102:0/3475372240 learned_addr learned my addr 172.21.2.102:0/3475372240 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:36:13.829 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:13.811 7f5e296d1700 1 -- 172.21.2.102:0/3475372240 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5e240d1870 msgr2=0x7f5e241b04c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:36:13.829 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:13.811 7f5e296d1700 1 --2- 172.21.2.102:0/3475372240 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5e240d1870 0x7f5e241b04c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:13.830 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:13.811 7f5e296d1700 1 -- 172.21.2.102:0/3475372240 >> v1:172.21.2.102:6789/0 conn(0x7f5e240d1c70 legacy=0x7f5e24108f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:13.830 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:13.811 7f5e296d1700 1 -- 172.21.2.102:0/3475372240 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e24088070 con 0x7f5e241b0ae0 2021-02-10T11:36:13.830 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:13.811 7f5e296d1700 1 --2- 172.21.2.102:0/3475372240 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5e241b0ae0 0x7f5e24108940 secure :-1 s=READY pgs=2797 cs=0 l=1 rev1=1 rx=0x7f5e20006700 tx=0x7f5e20009b80).ready entity=mon.0 client_cookie=98fb6af9e06d42bb server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:13.830 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:13.811 7f5e28ed0700 1 -- 172.21.2.102:0/3475372240 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5e2000c0c0 con 0x7f5e241b0ae0 2021-02-10T11:36:13.830 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:13.811 7f5e28ed0700 1 -- 172.21.2.102:0/3475372240 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e2000c400 con 0x7f5e241b0ae0 2021-02-10T11:36:13.830 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:13.811 7f5e28ed0700 1 -- 172.21.2.102:0/3475372240 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5e2000c920 con 0x7f5e241b0ae0 2021-02-10T11:36:13.830 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:13.811 7f5e2c0b6700 1 -- 172.21.2.102:0/3475372240 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5e241b0ae0 msgr2=0x7f5e24108940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:13.830 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:13.811 7f5e2c0b6700 1 --2- 172.21.2.102:0/3475372240 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5e241b0ae0 0x7f5e24108940 secure :-1 s=READY pgs=2797 cs=0 l=1 rev1=1 rx=0x7f5e20006700 tx=0x7f5e20009b80).stop 2021-02-10T11:36:13.830 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:13.812 7f5e2c0b6700 1 -- 172.21.2.102:0/3475372240 shutdown_connections 2021-02-10T11:36:13.830 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:13.812 7f5e2c0b6700 1 --2- 172.21.2.102:0/3475372240 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5e241b0ae0 0x7f5e24108940 unknown :-1 s=CLOSED pgs=2797 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:13.830 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:13.812 7f5e2c0b6700 1 --2- 172.21.2.102:0/3475372240 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5e240d1870 0x7f5e241b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:13.830 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:13.812 7f5e2c0b6700 1 -- 172.21.2.102:0/3475372240 >> 172.21.2.102:0/3475372240 conn(0x7f5e240ce420 msgr2=0x7f5e240d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:13.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:13.812 7f5e2c0b6700 1 -- 172.21.2.102:0/3475372240 shutdown_connections 2021-02-10T11:36:13.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:13.812 7f5e2c0b6700 1 -- 172.21.2.102:0/3475372240 wait complete. 2021-02-10T11:36:13.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:13.826 7f5e2c0b6700 1 Processor -- start 2021-02-10T11:36:13.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:13.826 7f5e2c0b6700 1 -- start start 2021-02-10T11:36:13.832 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:13.827 7f5e2c0b6700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5e240c2370 0x7f5e2418f870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:13.832 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:13.827 7f5e2c0b6700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5e24084cb0 0x7f5e241a7470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:13.832 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:13.827 7f5e2c0b6700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5e241a7a90 0x7f5e24085700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:13.832 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:13.827 7f5e2c0b6700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e240be840 con 0x7f5e240c2370 2021-02-10T11:36:13.832 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:13.827 7f5e2c0b6700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e240c29a0 con 0x7f5e24084cb0 2021-02-10T11:36:13.833 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:13.827 7f5e2c0b6700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e241a7e90 con 0x7f5e241a7a90 2021-02-10T11:36:13.833 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:13.827 7f5e29ed2700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5e240c2370 0x7f5e2418f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:13.833 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:13.827 7f5e29ed2700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5e240c2370 0x7f5e2418f870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:37662/0 (socket says 172.21.2.102:37662) 2021-02-10T11:36:13.833 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:13.827 7f5e29ed2700 1 -- 172.21.2.102:0/280145807 learned_addr learned my addr 172.21.2.102:0/280145807 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:36:13.833 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:13.827 7f5e29ed2700 1 -- 172.21.2.102:0/280145807 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5e241a7a90 msgr2=0x7f5e24085700 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:36:13.833 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:13.827 7f5e29ed2700 1 --2- 172.21.2.102:0/280145807 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5e241a7a90 0x7f5e24085700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:13.833 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:13.827 7f5e29ed2700 1 -- 172.21.2.102:0/280145807 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5e24084cb0 msgr2=0x7f5e241a7470 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:36:13.833 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:13.827 7f5e29ed2700 1 --2- 172.21.2.102:0/280145807 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5e24084cb0 0x7f5e241a7470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:13.833 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:13.827 7f5e29ed2700 1 -- 172.21.2.102:0/280145807 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e241a80b0 con 0x7f5e240c2370 2021-02-10T11:36:13.833 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:13.828 7f5e29ed2700 1 --2- 172.21.2.102:0/280145807 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5e240c2370 0x7f5e2418f870 secure :-1 s=READY pgs=2798 cs=0 l=1 rev1=1 rx=0x7f5e1800b5c0 tx=0x7f5e1800b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:13.833 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:13.828 7f5e167fc700 1 -- 172.21.2.102:0/280145807 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5e1800bd50 con 0x7f5e240c2370 2021-02-10T11:36:13.833 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:13.828 7f5e167fc700 1 -- 172.21.2.102:0/280145807 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e18006030 con 0x7f5e240c2370 2021-02-10T11:36:13.834 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:13.828 7f5e2c0b6700 1 -- 172.21.2.102:0/280145807 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e241a80b0 con 0x7f5e240c2370 2021-02-10T11:36:13.834 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:13.828 7f5e167fc700 1 -- 172.21.2.102:0/280145807 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5e180054d0 con 0x7f5e240c2370 2021-02-10T11:36:13.834 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:13.828 7f5e2c0b6700 1 -- 172.21.2.102:0/280145807 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e241a80b0 con 0x7f5e240c2370 2021-02-10T11:36:13.834 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:13.829 7f5e167fc700 1 -- 172.21.2.102:0/280145807 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f5e18006210 con 0x7f5e240c2370 2021-02-10T11:36:13.834 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:13.829 7f5e167fc700 1 --2- 172.21.2.102:0/280145807 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5e0c0186c0 0x7f5e0c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:13.834 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:13.829 7f5e167fc700 1 -- 172.21.2.102:0/280145807 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(153..153 src has 1..153) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f5e180325b0 con 0x7f5e240c2370 2021-02-10T11:36:13.834 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:13.829 7f5e296d1700 1 --2- 172.21.2.102:0/280145807 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5e0c0186c0 0x7f5e0c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:13.834 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:13.829 7f5e296d1700 1 --2- 172.21.2.102:0/280145807 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5e0c0186c0 0x7f5e0c01abc0 crc :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:13.863 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:13.861 7f5e2c0b6700 1 -- 172.21.2.102:0/280145807 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e24082a70 con 0x7f5e240c2370 2021-02-10T11:36:13.865 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:13.863 7f5e167fc700 1 -- 172.21.2.102:0/280145807 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f5e180223c0 con 0x7f5e240c2370 2021-02-10T11:36:13.995 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:13.993 7f5e2c0b6700 1 -- 172.21.2.102:0/280145807 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f5e24082a70 con 0x7f5e240c2370 2021-02-10T11:36:13.995 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:13.993 7f5e167fc700 1 -- 172.21.2.102:0/280145807 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v153) v1 ==== 92+0+10 (secure 0 0 0) 0x7f5e180223c0 con 0x7f5e240c2370 2021-02-10T11:36:13.996 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:36:13.997 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:13.995 7f5e2c0b6700 1 -- 172.21.2.102:0/280145807 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5e0c0186c0 msgr2=0x7f5e0c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:13.997 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:13.995 7f5e2c0b6700 1 --2- 172.21.2.102:0/280145807 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5e0c0186c0 0x7f5e0c01abc0 crc :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:13.997 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:13.995 7f5e2c0b6700 1 -- 172.21.2.102:0/280145807 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5e240c2370 msgr2=0x7f5e2418f870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:13.997 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:13.995 7f5e2c0b6700 1 --2- 172.21.2.102:0/280145807 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5e240c2370 0x7f5e2418f870 secure :-1 s=READY pgs=2798 cs=0 l=1 rev1=1 rx=0x7f5e1800b5c0 tx=0x7f5e1800b960).stop 2021-02-10T11:36:13.998 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:13.995 7f5e2c0b6700 1 -- 172.21.2.102:0/280145807 shutdown_connections 2021-02-10T11:36:13.998 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:13.995 7f5e2c0b6700 1 --2- 172.21.2.102:0/280145807 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5e241a7a90 0x7f5e24085700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:13.998 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:13.995 7f5e2c0b6700 1 --2- 172.21.2.102:0/280145807 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5e0c0186c0 0x7f5e0c01abc0 unknown :-1 s=CLOSED pgs=1229 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:13.998 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:13.995 7f5e2c0b6700 1 --2- 172.21.2.102:0/280145807 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5e24084cb0 0x7f5e241a7470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:13.998 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:13.995 7f5e2c0b6700 1 --2- 172.21.2.102:0/280145807 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5e240c2370 0x7f5e2418f870 unknown :-1 s=CLOSED pgs=2798 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:13.998 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:13.995 7f5e2c0b6700 1 -- 172.21.2.102:0/280145807 >> 172.21.2.102:0/280145807 conn(0x7f5e240b5dc0 msgr2=0x7f5e240c2d90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:13.998 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:13.996 7f5e2c0b6700 1 -- 172.21.2.102:0/280145807 shutdown_connections 2021-02-10T11:36:13.998 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:13.996 7f5e2c0b6700 1 -- 172.21.2.102:0/280145807 wait complete. 2021-02-10T11:36:14.006 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-02-10T11:36:14.438 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:14.420 7f867c8e8700 1 Processor -- start 2021-02-10T11:36:14.439 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:14.420 7f867c8e8700 1 -- start start 2021-02-10T11:36:14.439 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:14.420 7f867c8e8700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f86780d1870 0x7f86781b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:14.439 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:14.420 7f867c8e8700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f86780cff70 0x7f8678109130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:14.439 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:14.420 7f867c8e8700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8678109a60 con 0x7f86781b0ae0 2021-02-10T11:36:14.439 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:14.420 7f867c8e8700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8678109750 con 0x7f86780d1870 2021-02-10T11:36:14.439 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:14.420 7f867c8e8700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8678088070 con 0x7f86780cff70 2021-02-10T11:36:14.439 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:14.420 7f867661c700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f86780d1870 0x7f86781b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:14.439 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:14.420 7f867661c700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f86780d1870 0x7f86781b04c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:37668/0 (socket says 172.21.2.102:37668) 2021-02-10T11:36:14.439 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:14.421 7f867661c700 1 -- 172.21.2.102:0/537691183 learned_addr learned my addr 172.21.2.102:0/537691183 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:36:14.439 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:14.421 7f8676e1d700 1 --2- 172.21.2.102:0/537691183 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f86780cff70 0x7f8678109130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:14.439 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:14.421 7f867661c700 1 -- 172.21.2.102:0/537691183 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f86780cff70 msgr2=0x7f8678109130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:14.439 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:14.421 7f867661c700 1 --2- 172.21.2.102:0/537691183 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f86780cff70 0x7f8678109130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:14.440 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:14.421 7f867661c700 1 -- 172.21.2.102:0/537691183 >> v1:172.21.2.113:6789/0 conn(0x7f86781b0ae0 legacy=0x7f8678108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:14.440 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:14.421 7f867661c700 1 -- 172.21.2.102:0/537691183 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86780d1c70 con 0x7f86780d1870 2021-02-10T11:36:14.440 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:14.421 7f867661c700 1 --2- 172.21.2.102:0/537691183 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f86780d1870 0x7f86781b04c0 secure :-1 s=READY pgs=2799 cs=0 l=1 rev1=1 rx=0x7f8660006700 tx=0x7f8660009b80).ready entity=mon.0 client_cookie=a9aba932e336bcf9 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:14.440 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:14.421 7f867561a700 1 -- 172.21.2.102:0/537691183 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f866000c0c0 con 0x7f86780d1870 2021-02-10T11:36:14.440 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:14.421 7f867561a700 1 -- 172.21.2.102:0/537691183 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f866000c340 con 0x7f86780d1870 2021-02-10T11:36:14.440 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:14.421 7f867561a700 1 -- 172.21.2.102:0/537691183 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f866000c860 con 0x7f86780d1870 2021-02-10T11:36:14.440 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:14.421 7f867c8e8700 1 -- 172.21.2.102:0/537691183 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f86780d1870 msgr2=0x7f86781b04c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:14.440 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:14.421 7f867c8e8700 1 --2- 172.21.2.102:0/537691183 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f86780d1870 0x7f86781b04c0 secure :-1 s=READY pgs=2799 cs=0 l=1 rev1=1 rx=0x7f8660006700 tx=0x7f8660009b80).stop 2021-02-10T11:36:14.440 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:14.422 7f867c8e8700 1 -- 172.21.2.102:0/537691183 shutdown_connections 2021-02-10T11:36:14.440 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:14.422 7f867c8e8700 1 --2- 172.21.2.102:0/537691183 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f86780cff70 0x7f8678109130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:14.440 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:14.422 7f867c8e8700 1 --2- 172.21.2.102:0/537691183 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f86780d1870 0x7f86781b04c0 unknown :-1 s=CLOSED pgs=2799 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:14.440 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:14.422 7f867c8e8700 1 -- 172.21.2.102:0/537691183 >> 172.21.2.102:0/537691183 conn(0x7f86780ce420 msgr2=0x7f86780d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:14.440 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:14.422 7f867c8e8700 1 -- 172.21.2.102:0/537691183 shutdown_connections 2021-02-10T11:36:14.441 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:14.422 7f867c8e8700 1 -- 172.21.2.102:0/537691183 wait complete. 2021-02-10T11:36:14.442 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:14.437 7f867c8e8700 1 Processor -- start 2021-02-10T11:36:14.442 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:14.437 7f867c8e8700 1 -- start start 2021-02-10T11:36:14.443 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:14.437 7f867c8e8700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f86780a7950 0x7f86780b8df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:14.443 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:14.437 7f867c8e8700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f86781a7490 0x7f86781a7890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:14.443 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:14.437 7f867c8e8700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f86781a7eb0 0x7f86780ac630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:14.443 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:14.437 7f867c8e8700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86780a7fb0 con 0x7f86781a7490 2021-02-10T11:36:14.443 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:14.437 7f867c8e8700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86780acc50 con 0x7f86780a7950 2021-02-10T11:36:14.443 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:14.437 7f867c8e8700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86780ace30 con 0x7f86781a7eb0 2021-02-10T11:36:14.443 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:14.437 7f8675e1b700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f86781a7490 0x7f86781a7890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:14.443 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:14.438 7f8676e1d700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f86781a7eb0 0x7f86780ac630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:14.443 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:14.438 7f8675e1b700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f86781a7490 0x7f86781a7890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:37674/0 (socket says 172.21.2.102:37674) 2021-02-10T11:36:14.443 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:14.438 7f8675e1b700 1 -- 172.21.2.102:0/559387349 learned_addr learned my addr 172.21.2.102:0/559387349 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:36:14.443 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:14.438 7f867661c700 1 --2- 172.21.2.102:0/559387349 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f86780a7950 0x7f86780b8df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:14.443 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:14.438 7f8676e1d700 1 -- 172.21.2.102:0/559387349 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f86780a7950 msgr2=0x7f86780b8df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:14.443 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:14.438 7f8676e1d700 1 --2- 172.21.2.102:0/559387349 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f86780a7950 0x7f86780b8df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:14.444 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:14.438 7f8676e1d700 1 -- 172.21.2.102:0/559387349 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f86781a7490 msgr2=0x7f86781a7890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:14.444 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:14.438 7f8676e1d700 1 --2- 172.21.2.102:0/559387349 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f86781a7490 0x7f86781a7890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:14.444 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:14.438 7f8676e1d700 1 -- 172.21.2.102:0/559387349 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86780ad050 con 0x7f86781a7eb0 2021-02-10T11:36:14.444 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:14.438 7f8675e1b700 1 --2- 172.21.2.102:0/559387349 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f86781a7490 0x7f86781a7890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:36:14.444 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:14.438 7f8676e1d700 1 --2- 172.21.2.102:0/559387349 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f86781a7eb0 0x7f86780ac630 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 rx=0x7f867000b8a0 tx=0x7f867000bc40).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:14.444 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:14.438 7f866effd700 1 -- 172.21.2.102:0/559387349 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8670000f90 con 0x7f86781a7eb0 2021-02-10T11:36:14.444 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:14.438 7f866effd700 1 -- 172.21.2.102:0/559387349 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f867000bd80 con 0x7f86781a7eb0 2021-02-10T11:36:14.444 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:14.438 7f866effd700 1 -- 172.21.2.102:0/559387349 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8670001470 con 0x7f86781a7eb0 2021-02-10T11:36:14.444 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:14.438 7f867c8e8700 1 -- 172.21.2.102:0/559387349 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86780ad050 con 0x7f86781a7eb0 2021-02-10T11:36:14.444 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:14.438 7f867c8e8700 1 -- 172.21.2.102:0/559387349 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86780ad050 con 0x7f86781a7eb0 2021-02-10T11:36:14.444 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:14.439 7f866effd700 1 -- 172.21.2.102:0/559387349 <== mon.2 v2:172.21.2.113:3301/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f86700011f0 con 0x7f86781a7eb0 2021-02-10T11:36:14.444 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:14.439 7f866effd700 1 --2- 172.21.2.102:0/559387349 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f865c0186c0 0x7f865c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:14.444 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:14.439 7f866effd700 1 -- 172.21.2.102:0/559387349 <== mon.2 v2:172.21.2.113:3301/0 5 ==== osd_map(153..153 src has 1..153) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f86700326e0 con 0x7f86781a7eb0 2021-02-10T11:36:14.445 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:14.439 7f867661c700 1 --2- 172.21.2.102:0/559387349 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f865c0186c0 0x7f865c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:14.445 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:14.440 7f867661c700 1 --2- 172.21.2.102:0/559387349 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f865c0186c0 0x7f865c01abc0 crc :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:14.472 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:14.470 7f867c8e8700 1 -- 172.21.2.102:0/559387349 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8678082a70 con 0x7f86781a7eb0 2021-02-10T11:36:14.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:14.473 7f866effd700 1 -- 172.21.2.102:0/559387349 <== mon.2 v2:172.21.2.113:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f86700066f0 con 0x7f86781a7eb0 2021-02-10T11:36:14.604 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:14.602 7f867c8e8700 1 -- 172.21.2.102:0/559387349 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7f8678082a70 con 0x7f86781a7eb0 2021-02-10T11:36:14.605 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:14.602 7f866effd700 1 -- 172.21.2.102:0/559387349 <== mon.2 v2:172.21.2.113:3301/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v153) v1 ==== 104+0+10 (secure 0 0 0) 0x7f86700011f0 con 0x7f86781a7eb0 2021-02-10T11:36:14.605 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:36:14.606 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:14.605 7f867c8e8700 1 -- 172.21.2.102:0/559387349 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f865c0186c0 msgr2=0x7f865c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:14.606 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:14.605 7f867c8e8700 1 --2- 172.21.2.102:0/559387349 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f865c0186c0 0x7f865c01abc0 crc :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:14.607 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:14.605 7f867c8e8700 1 -- 172.21.2.102:0/559387349 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f86781a7eb0 msgr2=0x7f86780ac630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:14.607 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:14.605 7f867c8e8700 1 --2- 172.21.2.102:0/559387349 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f86781a7eb0 0x7f86780ac630 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 rx=0x7f867000b8a0 tx=0x7f867000bc40).stop 2021-02-10T11:36:14.607 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:14.605 7f867c8e8700 1 -- 172.21.2.102:0/559387349 shutdown_connections 2021-02-10T11:36:14.607 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:14.605 7f867c8e8700 1 --2- 172.21.2.102:0/559387349 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f86781a7eb0 0x7f86780ac630 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:14.607 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:14.605 7f867c8e8700 1 --2- 172.21.2.102:0/559387349 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f86781a7490 0x7f86781a7890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:14.608 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:14.605 7f867c8e8700 1 --2- 172.21.2.102:0/559387349 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f865c0186c0 0x7f865c01abc0 unknown :-1 s=CLOSED pgs=1230 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:14.608 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:14.605 7f867c8e8700 1 --2- 172.21.2.102:0/559387349 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f86780a7950 0x7f86780b8df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:14.608 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:14.605 7f867c8e8700 1 -- 172.21.2.102:0/559387349 >> 172.21.2.102:0/559387349 conn(0x7f86780a02d0 msgr2=0x7f867809f200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:14.608 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:14.605 7f867c8e8700 1 -- 172.21.2.102:0/559387349 shutdown_connections 2021-02-10T11:36:14.608 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:14.605 7f867c8e8700 1 -- 172.21.2.102:0/559387349 wait complete. 2021-02-10T11:36:14.615 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-02-10T11:36:15.048 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.029 7f42c0b19700 1 Processor -- start 2021-02-10T11:36:15.049 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.029 7f42c0b19700 1 -- start start 2021-02-10T11:36:15.049 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.029 7f42c0b19700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f42bc0d1c70 0x7f42bc0cef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:15.049 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.029 7f42c0b19700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f42bc0cf580 0x7f42bc0cf980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:15.049 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.029 7f42c0b19700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42bc0fe1c0 con 0x7f42bc0d1870 2021-02-10T11:36:15.049 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.029 7f42c0b19700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42bc1b0cb0 con 0x7f42bc0cf580 2021-02-10T11:36:15.049 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.029 7f42c0b19700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42bc109a60 con 0x7f42bc0d1c70 2021-02-10T11:36:15.049 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.030 7f42ba85e700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f42bc0d1870 0x7f42bc1b04c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:58136/0 (socket says 172.21.2.102:58136) 2021-02-10T11:36:15.049 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.030 7f42ba85e700 1 -- 172.21.2.102:0/20293983 learned_addr learned my addr 172.21.2.102:0/20293983 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:36:15.049 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.030 7f42b985c700 1 -- 172.21.2.102:0/20293983 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3213498509 0 0) 0x7f42a4001180 con 0x7f42bc0d1870 2021-02-10T11:36:15.049 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.030 7f42b985c700 1 -- 172.21.2.102:0/20293983 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42a8002fe0 con 0x7f42bc0d1870 2021-02-10T11:36:15.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.030 7f42bb05f700 1 --2- 172.21.2.102:0/20293983 >> v2:172.21.2.113:3300/0 conn(0x7f42bc0cf580 0x7f42bc0cf980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:15.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.030 7f42ba05d700 1 --2- 172.21.2.102:0/20293983 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f42bc0d1c70 0x7f42bc0cef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:15.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.030 7f42b985c700 1 -- 172.21.2.102:0/20293983 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3156507100 0 0) 0x7f42a4001180 con 0x7f42bc0d1870 2021-02-10T11:36:15.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.030 7f42b985c700 1 -- 172.21.2.102:0/20293983 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f42bc0d1c70 msgr2=0x7f42bc0cef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:15.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.030 7f42b985c700 1 --2- 172.21.2.102:0/20293983 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f42bc0d1c70 0x7f42bc0cef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:15.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.030 7f42b985c700 1 -- 172.21.2.102:0/20293983 >> v2:172.21.2.113:3300/0 conn(0x7f42bc0cf580 msgr2=0x7f42bc0cf980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:15.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.030 7f42b985c700 1 --2- 172.21.2.102:0/20293983 >> v2:172.21.2.113:3300/0 conn(0x7f42bc0cf580 0x7f42bc0cf980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:15.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.030 7f42b985c700 1 -- 172.21.2.102:0/20293983 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42bc088070 con 0x7f42bc0d1870 2021-02-10T11:36:15.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.030 7f42b985c700 1 -- 172.21.2.102:0/20293983 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7f42a4001680 con 0x7f42bc0d1870 2021-02-10T11:36:15.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.030 7f42b985c700 1 -- 172.21.2.102:0/20293983 >> v1:172.21.2.102:6789/0 conn(0x7f42bc0d1870 legacy=0x7f42bc1b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:15.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.030 7f42b985c700 1 --2- 172.21.2.102:0/20293983 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f42a8003af0 0x7f42a8005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:15.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.030 7f42b985c700 1 --2- 172.21.2.102:0/20293983 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f42a804a7a0 0x7f42a804cca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:15.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.030 7f42b985c700 1 --2- 172.21.2.102:0/20293983 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f42a804d5a0 0x7f42a804fa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:15.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.030 7f42b985c700 1 -- 172.21.2.102:0/20293983 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42a80502e0 con 0x7f42a804a7a0 2021-02-10T11:36:15.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.030 7f42b985c700 1 -- 172.21.2.102:0/20293983 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42a8050500 con 0x7f42a8003af0 2021-02-10T11:36:15.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.030 7f42b985c700 1 -- 172.21.2.102:0/20293983 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42a8050720 con 0x7f42a804d5a0 2021-02-10T11:36:15.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.030 7f42ba05d700 1 --2- 172.21.2.102:0/20293983 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f42a804a7a0 0x7f42a804cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:15.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.031 7f42ba85e700 1 --2- 172.21.2.102:0/20293983 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f42a8003af0 0x7f42a8005fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:15.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.031 7f42ba05d700 1 -- 172.21.2.102:0/20293983 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f42a804d5a0 msgr2=0x7f42a804fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:15.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.031 7f42bb05f700 1 --2- 172.21.2.102:0/20293983 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f42a804d5a0 0x7f42a804fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:15.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.031 7f42ba05d700 1 --2- 172.21.2.102:0/20293983 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f42a804d5a0 0x7f42a804fa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:15.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.031 7f42ba05d700 1 -- 172.21.2.102:0/20293983 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f42a8003af0 msgr2=0x7f42a8005fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:15.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.031 7f42ba05d700 1 --2- 172.21.2.102:0/20293983 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f42a8003af0 0x7f42a8005fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:15.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.031 7f42ba05d700 1 -- 172.21.2.102:0/20293983 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42a80509c0 con 0x7f42a804a7a0 2021-02-10T11:36:15.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.031 7f42bb05f700 1 --2- 172.21.2.102:0/20293983 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f42a804d5a0 0x7f42a804fa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:36:15.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.031 7f42ba05d700 1 --2- 172.21.2.102:0/20293983 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f42a804a7a0 0x7f42a804cca0 secure :-1 s=READY pgs=2801 cs=0 l=1 rev1=1 rx=0x7f42ac006ac0 tx=0x7f42ac009ab0).ready entity=mon.0 client_cookie=bff52d6df3213de3 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:15.052 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.031 7f42b985c700 1 -- 172.21.2.102:0/20293983 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f42ac00c0c0 con 0x7f42a804a7a0 2021-02-10T11:36:15.052 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.031 7f42b985c700 1 -- 172.21.2.102:0/20293983 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42ac00c3c0 con 0x7f42a804a7a0 2021-02-10T11:36:15.052 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.031 7f42b985c700 1 -- 172.21.2.102:0/20293983 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f42ac00c8a0 con 0x7f42a804a7a0 2021-02-10T11:36:15.052 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.031 7f42c0b19700 1 -- 172.21.2.102:0/20293983 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f42a804a7a0 msgr2=0x7f42a804cca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:15.052 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.031 7f42c0b19700 1 --2- 172.21.2.102:0/20293983 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f42a804a7a0 0x7f42a804cca0 secure :-1 s=READY pgs=2801 cs=0 l=1 rev1=1 rx=0x7f42ac006ac0 tx=0x7f42ac009ab0).stop 2021-02-10T11:36:15.052 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.031 7f42bb05f700 1 -- 172.21.2.102:0/20293983 reap_dead start 2021-02-10T11:36:15.052 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.032 7f42c0b19700 1 -- 172.21.2.102:0/20293983 shutdown_connections 2021-02-10T11:36:15.052 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.032 7f42c0b19700 1 -- 172.21.2.102:0/20293983 >> 172.21.2.102:0/20293983 conn(0x7f42bc0ce420 msgr2=0x7f42bc0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:15.052 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.032 7f42c0b19700 1 -- 172.21.2.102:0/20293983 shutdown_connections 2021-02-10T11:36:15.052 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.032 7f42c0b19700 1 -- 172.21.2.102:0/20293983 wait complete. 2021-02-10T11:36:15.052 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.046 7f42c0b19700 1 Processor -- start 2021-02-10T11:36:15.052 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.047 7f42c0b19700 1 -- start start 2021-02-10T11:36:15.054 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.047 7f42c0b19700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f42bc1a79a0 0x7f42bc1a7da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:15.054 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.047 7f42c0b19700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f42bc0bdb40 0x7f42bc0bdf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:15.054 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.047 7f42c0b19700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f42bc082ca0 0x7f42bc0830a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:15.054 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.047 7f42c0b19700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42bc19d990 con 0x7f42bc082ca0 2021-02-10T11:36:15.054 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.047 7f42c0b19700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42bc19dbb0 con 0x7f42bc1a79a0 2021-02-10T11:36:15.054 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.047 7f42c0b19700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42bc0836c0 con 0x7f42bc0bdb40 2021-02-10T11:36:15.054 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.047 7f42bb05f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f42bc082ca0 0x7f42bc0830a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:15.054 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.047 7f42bb05f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f42bc082ca0 0x7f42bc0830a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:37696/0 (socket says 172.21.2.102:37696) 2021-02-10T11:36:15.054 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.047 7f42bb05f700 1 -- 172.21.2.102:0/1686438056 learned_addr learned my addr 172.21.2.102:0/1686438056 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:36:15.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.047 7f42bb05f700 1 -- 172.21.2.102:0/1686438056 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f42bc0bdb40 msgr2=0x7f42bc0bdf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:15.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.047 7f42bb05f700 1 --2- 172.21.2.102:0/1686438056 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f42bc0bdb40 0x7f42bc0bdf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:15.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.047 7f42bb05f700 1 -- 172.21.2.102:0/1686438056 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f42bc1a79a0 msgr2=0x7f42bc1a7da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:15.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.047 7f42bb05f700 1 --2- 172.21.2.102:0/1686438056 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f42bc1a79a0 0x7f42bc1a7da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:15.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.047 7f42bb05f700 1 -- 172.21.2.102:0/1686438056 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42bc0838e0 con 0x7f42bc082ca0 2021-02-10T11:36:15.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.048 7f42bb05f700 1 --2- 172.21.2.102:0/1686438056 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f42bc082ca0 0x7f42bc0830a0 secure :-1 s=READY pgs=2802 cs=0 l=1 rev1=1 rx=0x7f42b40099a0 tx=0x7f42b400bc70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:15.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.048 7f42b37fe700 1 -- 172.21.2.102:0/1686438056 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f42b400c2f0 con 0x7f42bc082ca0 2021-02-10T11:36:15.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.048 7f42b37fe700 1 -- 172.21.2.102:0/1686438056 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42b4006d00 con 0x7f42bc082ca0 2021-02-10T11:36:15.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.048 7f42b37fe700 1 -- 172.21.2.102:0/1686438056 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f42b400c870 con 0x7f42bc082ca0 2021-02-10T11:36:15.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.048 7f42c0b19700 1 -- 172.21.2.102:0/1686438056 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42bc0838e0 con 0x7f42bc082ca0 2021-02-10T11:36:15.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.048 7f42c0b19700 1 -- 172.21.2.102:0/1686438056 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42bc084420 con 0x7f42bc082ca0 2021-02-10T11:36:15.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.049 7f42b37fe700 1 -- 172.21.2.102:0/1686438056 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f42b40011f0 con 0x7f42bc082ca0 2021-02-10T11:36:15.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.049 7f42b37fe700 1 --2- 172.21.2.102:0/1686438056 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f42a00186c0 0x7f42a001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:15.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.049 7f42b37fe700 1 -- 172.21.2.102:0/1686438056 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(153..153 src has 1..153) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f42b4032930 con 0x7f42bc082ca0 2021-02-10T11:36:15.056 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.049 7f42ba85e700 1 --2- 172.21.2.102:0/1686438056 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f42a00186c0 0x7f42a001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:15.056 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.049 7f42ba85e700 1 --2- 172.21.2.102:0/1686438056 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f42a00186c0 0x7f42a001abc0 crc :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:15.081 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.080 7f42c0b19700 1 -- 172.21.2.102:0/1686438056 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42bc082a70 con 0x7f42bc082ca0 2021-02-10T11:36:15.084 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.082 7f42b37fe700 1 -- 172.21.2.102:0/1686438056 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f42b4021fc0 con 0x7f42bc082ca0 2021-02-10T11:36:15.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.210 7f42c0b19700 1 -- 172.21.2.102:0/1686438056 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7f42bc082a70 con 0x7f42bc082ca0 2021-02-10T11:36:15.213 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.211 7f42b37fe700 1 -- 172.21.2.102:0/1686438056 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v153) v1 ==== 100+0+10 (secure 0 0 0) 0x7f42b4021fc0 con 0x7f42bc082ca0 2021-02-10T11:36:15.213 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:36:15.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.213 7f42c0b19700 1 -- 172.21.2.102:0/1686438056 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f42a00186c0 msgr2=0x7f42a001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:15.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.213 7f42c0b19700 1 --2- 172.21.2.102:0/1686438056 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f42a00186c0 0x7f42a001abc0 crc :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:15.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.213 7f42c0b19700 1 -- 172.21.2.102:0/1686438056 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f42bc082ca0 msgr2=0x7f42bc0830a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:15.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.213 7f42c0b19700 1 --2- 172.21.2.102:0/1686438056 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f42bc082ca0 0x7f42bc0830a0 secure :-1 s=READY pgs=2802 cs=0 l=1 rev1=1 rx=0x7f42b40099a0 tx=0x7f42b400bc70).stop 2021-02-10T11:36:15.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.213 7f42c0b19700 1 -- 172.21.2.102:0/1686438056 shutdown_connections 2021-02-10T11:36:15.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.213 7f42c0b19700 1 --2- 172.21.2.102:0/1686438056 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f42bc082ca0 0x7f42bc0830a0 unknown :-1 s=CLOSED pgs=2802 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:15.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.213 7f42c0b19700 1 --2- 172.21.2.102:0/1686438056 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f42bc0bdb40 0x7f42bc0bdf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:15.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.213 7f42c0b19700 1 --2- 172.21.2.102:0/1686438056 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f42a00186c0 0x7f42a001abc0 unknown :-1 s=CLOSED pgs=1231 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:15.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.213 7f42c0b19700 1 --2- 172.21.2.102:0/1686438056 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f42bc1a79a0 0x7f42bc1a7da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:15.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.213 7f42c0b19700 1 -- 172.21.2.102:0/1686438056 >> 172.21.2.102:0/1686438056 conn(0x7f42bc0bef00 msgr2=0x7f42bc09f200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:15.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.213 7f42c0b19700 1 -- 172.21.2.102:0/1686438056 shutdown_connections 2021-02-10T11:36:15.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.213 7f42c0b19700 1 -- 172.21.2.102:0/1686438056 wait complete. 2021-02-10T11:36:15.224 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-02-10T11:36:15.224 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:36:15.653 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.635 7f4aab3cc700 1 Processor -- start 2021-02-10T11:36:15.653 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.635 7f4aab3cc700 1 -- start start 2021-02-10T11:36:15.654 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.636 7f4aab3cc700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f4aa41b0470 0x7f4aa41b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:15.654 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.636 7f4aab3cc700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f4aa40ef610 0x7f4aa40efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:15.654 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.636 7f4aab3cc700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4aa40f0030 0x7f4aa40f0430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:15.654 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.636 7f4aab3cc700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4aa40e5ea0 con 0x7f4aa40ef610 2021-02-10T11:36:15.654 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.636 7f4aab3cc700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4aa40f0b50 con 0x7f4aa41b0470 2021-02-10T11:36:15.654 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.636 7f4aab3cc700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4aa40e52e0 con 0x7f4aa40f0030 2021-02-10T11:36:15.654 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.636 7f4aa89e7700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f4aa40ef610 0x7f4aa40efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:15.654 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.636 7f4aa89e7700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f4aa40ef610 0x7f4aa40efa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:37700/0 (socket says 172.21.2.102:37700) 2021-02-10T11:36:15.654 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.636 7f4aa89e7700 1 -- 172.21.2.102:0/1613602562 learned_addr learned my addr 172.21.2.102:0/1613602562 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:36:15.654 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.636 7f4aa99e9700 1 --2- 172.21.2.102:0/1613602562 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4aa40f0030 0x7f4aa40f0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:15.654 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.636 7f4aa91e8700 1 --2- 172.21.2.102:0/1613602562 >> v2:172.21.2.113:3300/0 conn(0x7f4aa41b0470 0x7f4aa41b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:15.655 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.636 7f4aa89e7700 1 -- 172.21.2.102:0/1613602562 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4aa40f0030 msgr2=0x7f4aa40f0430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:15.655 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.636 7f4aa89e7700 1 --2- 172.21.2.102:0/1613602562 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4aa40f0030 0x7f4aa40f0430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:15.655 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.636 7f4aa89e7700 1 -- 172.21.2.102:0/1613602562 >> v2:172.21.2.113:3300/0 conn(0x7f4aa41b0470 msgr2=0x7f4aa41b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:15.655 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.636 7f4aa89e7700 1 --2- 172.21.2.102:0/1613602562 >> v2:172.21.2.113:3300/0 conn(0x7f4aa41b0470 0x7f4aa41b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:15.655 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.636 7f4aa89e7700 1 -- 172.21.2.102:0/1613602562 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4aa40f10d0 con 0x7f4aa40ef610 2021-02-10T11:36:15.655 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.636 7f4aa91e8700 1 --2- 172.21.2.102:0/1613602562 >> v2:172.21.2.113:3300/0 conn(0x7f4aa41b0470 0x7f4aa41b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:36:15.655 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.636 7f4aa89e7700 1 --2- 172.21.2.102:0/1613602562 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4aa40ef610 0x7f4aa40efa10 secure :-1 s=READY pgs=2803 cs=0 l=1 rev1=1 rx=0x7f4aa0006700 tx=0x7f4aa0009b00).ready entity=mon.0 client_cookie=e30a0ab344646bc3 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:15.655 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.636 7f4a97fff700 1 -- 172.21.2.102:0/1613602562 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4aa000c0c0 con 0x7f4aa40ef610 2021-02-10T11:36:15.655 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.636 7f4a97fff700 1 -- 172.21.2.102:0/1613602562 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4aa000c400 con 0x7f4aa40ef610 2021-02-10T11:36:15.655 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.637 7f4a97fff700 1 -- 172.21.2.102:0/1613602562 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4aa000c860 con 0x7f4aa40ef610 2021-02-10T11:36:15.655 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.637 7f4aab3cc700 1 -- 172.21.2.102:0/1613602562 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4aa40ef610 msgr2=0x7f4aa40efa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:15.655 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.637 7f4aab3cc700 1 --2- 172.21.2.102:0/1613602562 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4aa40ef610 0x7f4aa40efa10 secure :-1 s=READY pgs=2803 cs=0 l=1 rev1=1 rx=0x7f4aa0006700 tx=0x7f4aa0009b00).stop 2021-02-10T11:36:15.655 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.637 7f4aab3cc700 1 -- 172.21.2.102:0/1613602562 shutdown_connections 2021-02-10T11:36:15.656 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.637 7f4aab3cc700 1 --2- 172.21.2.102:0/1613602562 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4aa40f0030 0x7f4aa40f0430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:15.656 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.637 7f4aab3cc700 1 --2- 172.21.2.102:0/1613602562 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4aa40ef610 0x7f4aa40efa10 unknown :-1 s=CLOSED pgs=2803 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:15.656 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.637 7f4aab3cc700 1 --2- 172.21.2.102:0/1613602562 >> v2:172.21.2.113:3300/0 conn(0x7f4aa41b0470 0x7f4aa41b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:15.656 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.637 7f4aab3cc700 1 -- 172.21.2.102:0/1613602562 >> 172.21.2.102:0/1613602562 conn(0x7f4aa40ce3e0 msgr2=0x7f4aa40e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:15.656 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.637 7f4aab3cc700 1 -- 172.21.2.102:0/1613602562 shutdown_connections 2021-02-10T11:36:15.656 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.637 7f4aab3cc700 1 -- 172.21.2.102:0/1613602562 wait complete. 2021-02-10T11:36:15.657 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.652 7f4aab3cc700 1 Processor -- start 2021-02-10T11:36:15.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.652 7f4aab3cc700 1 -- start start 2021-02-10T11:36:15.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.652 7f4aab3cc700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4aa41b5200 0x7f4aa418bee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:15.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.652 7f4aab3cc700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4aa418c500 0x7f4aa418c900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:15.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.652 7f4aab3cc700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4aa40b1630 0x7f4aa40b1a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:15.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.652 7f4aab3cc700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4aa41b58e0 con 0x7f4aa41b5200 2021-02-10T11:36:15.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.652 7f4aab3cc700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4aa40b2050 con 0x7f4aa40b1630 2021-02-10T11:36:15.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.652 7f4aab3cc700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4aa40b2270 con 0x7f4aa418c500 2021-02-10T11:36:15.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.652 7f4aa99e9700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4aa40b1630 0x7f4aa40b1a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:15.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.652 7f4aa91e8700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4aa41b5200 0x7f4aa418bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:15.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.652 7f4aa99e9700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4aa40b1630 0x7f4aa40b1a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.113:3300/0 says I am v2:172.21.2.102:41134/0 (socket says 172.21.2.102:41134) 2021-02-10T11:36:15.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.652 7f4aa99e9700 1 -- 172.21.2.102:0/2145627710 learned_addr learned my addr 172.21.2.102:0/2145627710 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:36:15.659 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.652 7f4aa91e8700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4aa41b5200 0x7f4aa418bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:37706/0 (socket says 172.21.2.102:37706) 2021-02-10T11:36:15.659 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.653 7f4aa89e7700 1 --2- 172.21.2.102:0/2145627710 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4aa418c500 0x7f4aa418c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:15.659 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.653 7f4aa99e9700 1 -- 172.21.2.102:0/2145627710 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4aa418c500 msgr2=0x7f4aa418c900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:15.659 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.653 7f4aa99e9700 1 --2- 172.21.2.102:0/2145627710 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4aa418c500 0x7f4aa418c900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:15.659 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.653 7f4aa99e9700 1 -- 172.21.2.102:0/2145627710 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4aa41b5200 msgr2=0x7f4aa418bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:15.659 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.653 7f4aa99e9700 1 --2- 172.21.2.102:0/2145627710 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4aa41b5200 0x7f4aa418bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:15.659 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.653 7f4aa99e9700 1 -- 172.21.2.102:0/2145627710 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4aa40b2490 con 0x7f4aa40b1630 2021-02-10T11:36:15.659 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.653 7f4aa99e9700 1 --2- 172.21.2.102:0/2145627710 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4aa40b1630 0x7f4aa40b1a30 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 rx=0x7f4a9c00b940 tx=0x7f4a9c00bce0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:15.659 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.653 7f4a95ffb700 1 -- 172.21.2.102:0/2145627710 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4a9c0066f0 con 0x7f4aa40b1630 2021-02-10T11:36:15.659 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.653 7f4a95ffb700 1 -- 172.21.2.102:0/2145627710 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a9c00c2f0 con 0x7f4aa40b1630 2021-02-10T11:36:15.659 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.653 7f4aab3cc700 1 -- 172.21.2.102:0/2145627710 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4aa40b2490 con 0x7f4aa40b1630 2021-02-10T11:36:15.659 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.653 7f4a95ffb700 1 -- 172.21.2.102:0/2145627710 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4a9c00c750 con 0x7f4aa40b1630 2021-02-10T11:36:15.659 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.653 7f4aab3cc700 1 -- 172.21.2.102:0/2145627710 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4aa40b2490 con 0x7f4aa40b1630 2021-02-10T11:36:15.660 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.654 7f4a95ffb700 1 -- 172.21.2.102:0/2145627710 <== mon.1 v2:172.21.2.113:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4a9c0011f0 con 0x7f4aa40b1630 2021-02-10T11:36:15.660 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.654 7f4a95ffb700 1 --2- 172.21.2.102:0/2145627710 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4a840186c0 0x7f4a8401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:15.660 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.654 7f4a95ffb700 1 -- 172.21.2.102:0/2145627710 <== mon.1 v2:172.21.2.113:3300/0 5 ==== osd_map(153..153 src has 1..153) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f4a9c0327b0 con 0x7f4aa40b1630 2021-02-10T11:36:15.660 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.654 7f4aa91e8700 1 --2- 172.21.2.102:0/2145627710 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4a840186c0 0x7f4a8401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:15.660 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.654 7f4aa91e8700 1 --2- 172.21.2.102:0/2145627710 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4a840186c0 0x7f4a8401abc0 crc :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:15.687 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.685 7f4aab3cc700 1 -- 172.21.2.102:0/2145627710 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4aa4105a80 con 0x7f4aa40b1630 2021-02-10T11:36:15.690 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.688 7f4a95ffb700 1 -- 172.21.2.102:0/2145627710 <== mon.1 v2:172.21.2.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f4a9c02e000 con 0x7f4aa40b1630 2021-02-10T11:36:15.822 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.820 7f4aab3cc700 1 -- 172.21.2.102:0/2145627710 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4aa40f0b50 con 0x7f4aa40b1630 2021-02-10T11:36:15.823 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.821 7f4a95ffb700 1 -- 172.21.2.102:0/2145627710 <== mon.1 v2:172.21.2.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 132 v132) v1 ==== 95+0+2107 (secure 0 0 0) 0x7f4a9c02e000 con 0x7f4aa40b1630 2021-02-10T11:36:15.825 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:36:15.825 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":132,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7004,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.102:6835/3280534775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6834","nonce":3280534775},{"type":"v1","addr":"172.21.2.102:6835","nonce":3280534775}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":128}],"filesystems":[{"mdsmap":{"epoch":132,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:35:58.285904","modified":"2021-02-10 11:36:00.292711","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7001},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7001":{"gid":7001,"name":"b","rank":0,"incarnation":129,"state":"up:active","state_seq":3,"addr":"172.21.2.113:6833/1983258405","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6832","nonce":1983258405},{"type":"v1","addr":"172.21.2.113:6833","nonce":1983258405}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[33],"metadata_pool":32,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":16}]} 2021-02-10T11:36:15.825 INFO:teuthology.orchestra.run.gibba002.stderr:dumped fsmap epoch 132 2021-02-10T11:36:15.827 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.824 7f4aab3cc700 1 -- 172.21.2.102:0/2145627710 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4a840186c0 msgr2=0x7f4a8401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:15.827 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.824 7f4aab3cc700 1 --2- 172.21.2.102:0/2145627710 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4a840186c0 0x7f4a8401abc0 crc :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:15.827 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.824 7f4aab3cc700 1 -- 172.21.2.102:0/2145627710 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4aa40b1630 msgr2=0x7f4aa40b1a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:15.827 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.824 7f4aab3cc700 1 --2- 172.21.2.102:0/2145627710 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4aa40b1630 0x7f4aa40b1a30 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 rx=0x7f4a9c00b940 tx=0x7f4a9c00bce0).stop 2021-02-10T11:36:15.827 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.824 7f4aab3cc700 1 -- 172.21.2.102:0/2145627710 shutdown_connections 2021-02-10T11:36:15.827 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.824 7f4aab3cc700 1 --2- 172.21.2.102:0/2145627710 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4a840186c0 0x7f4a8401abc0 unknown :-1 s=CLOSED pgs=1232 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:15.827 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.824 7f4aab3cc700 1 --2- 172.21.2.102:0/2145627710 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4aa40b1630 0x7f4aa40b1a30 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:15.827 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.824 7f4aab3cc700 1 --2- 172.21.2.102:0/2145627710 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4aa418c500 0x7f4aa418c900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:15.827 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.824 7f4aab3cc700 1 --2- 172.21.2.102:0/2145627710 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4aa41b5200 0x7f4aa418bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:15.827 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.824 7f4aab3cc700 1 -- 172.21.2.102:0/2145627710 >> 172.21.2.102:0/2145627710 conn(0x7f4aa41b9980 msgr2=0x7f4aa4105cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:15.827 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.825 7f4aab3cc700 1 -- 172.21.2.102:0/2145627710 shutdown_connections 2021-02-10T11:36:15.827 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:15.825 7f4aab3cc700 1 -- 172.21.2.102:0/2145627710 wait complete. 2021-02-10T11:36:16.037 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 132, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:35:58.285904', 'modified': '2021-02-10 11:36:00.292711', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7001}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7001': {'gid': 7001, 'name': 'b', 'rank': 0, 'incarnation': 129, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.113:6833/1983258405', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.113:6832', 'nonce': 1983258405}, {'type': 'v1', 'addr': '172.21.2.113:6833', 'nonce': 1983258405}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [33], 'metadata_pool': 32, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:36:16.037 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:36:16.037 DEBUG:teuthology.orchestra.run.gibba013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:36:16.438 INFO:teuthology.orchestra.run.gibba013.stdout:{ 2021-02-10T11:36:16.439 INFO:teuthology.orchestra.run.gibba013.stdout: "cluster_fsid": "68db30ea-dbce-4aef-a6df-f01c6fea0513", 2021-02-10T11:36:16.439 INFO:teuthology.orchestra.run.gibba013.stdout: "whoami": 0, 2021-02-10T11:36:16.439 INFO:teuthology.orchestra.run.gibba013.stdout: "id": 7001, 2021-02-10T11:36:16.439 INFO:teuthology.orchestra.run.gibba013.stdout: "want_state": "up:active", 2021-02-10T11:36:16.439 INFO:teuthology.orchestra.run.gibba013.stdout: "state": "up:active", 2021-02-10T11:36:16.439 INFO:teuthology.orchestra.run.gibba013.stdout: "rank_uptime": 18.148851449999999, 2021-02-10T11:36:16.439 INFO:teuthology.orchestra.run.gibba013.stdout: "mdsmap_epoch": 132, 2021-02-10T11:36:16.439 INFO:teuthology.orchestra.run.gibba013.stdout: "osdmap_epoch": 153, 2021-02-10T11:36:16.439 INFO:teuthology.orchestra.run.gibba013.stdout: "osdmap_epoch_barrier": 153, 2021-02-10T11:36:16.439 INFO:teuthology.orchestra.run.gibba013.stdout: "uptime": 23.795571416000001 2021-02-10T11:36:16.439 INFO:teuthology.orchestra.run.gibba013.stdout:} 2021-02-10T11:36:16.443 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "68db30ea-dbce-4aef-a6df-f01c6fea0513", "whoami": 0, "id": 7001, "want_state": "up:active", "state": "up:active", "rank_uptime": 18.148851449999999, "mdsmap_epoch": 132, "osdmap_epoch": 153, "osdmap_epoch_barrier": 153, "uptime": 23.795571416000001 } 2021-02-10T11:36:16.443 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2021-02-10T11:36:16.443 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2021-02-10T11:36:16.443 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.1 at ubuntu@gibba002.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.1... 2021-02-10T11:36:16.443 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:36:16.443 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec mkdir -p /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:36:16.779 DEBUG:teuthology.orchestra.run.gibba002:> sudo modprobe fuse 2021-02-10T11:36:17.123 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:36:17.123 DEBUG:teuthology.orchestra.run.gibba002:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:36:17.467 INFO:teuthology.orchestra.run.gibba002.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:36:17.467 INFO:teuthology.orchestra.run.gibba002.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:36:17.468 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:36:17.468 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:36:17.468 DEBUG:teuthology.orchestra.run.gibba002:> ls /sys/fs/fuse/connections 2021-02-10T11:36:17.803 INFO:teuthology.orchestra.run.gibba002.stdout:43 2021-02-10T11:36:17.803 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [43] 2021-02-10T11:36:17.804 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.1 /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:36:17.807 DEBUG:teuthology.orchestra.run.gibba002:> sudo modprobe fuse 2021-02-10T11:36:18.153 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:36:18.153 DEBUG:teuthology.orchestra.run.gibba002:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:36:18.219 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba002.stderr:ceph-fuse[201572]: starting ceph client 2021-02-10T11:36:18.219 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba002.stderr:2021-02-10 11:36:18.217 7f688a87cf80 -1 init, newargv = 0x55af4bc191a0 newargc=9 2021-02-10T11:36:18.226 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba002.stderr:ceph-fuse[201572]: starting fuse 2021-02-10T11:36:18.497 INFO:teuthology.orchestra.run.gibba002.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:36:18.497 INFO:teuthology.orchestra.run.gibba002.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:36:18.498 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:36:18.498 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:36:18.498 DEBUG:teuthology.orchestra.run.gibba002:> ls /sys/fs/fuse/connections 2021-02-10T11:36:18.834 INFO:teuthology.orchestra.run.gibba002.stdout:43 2021-02-10T11:36:18.834 INFO:teuthology.orchestra.run.gibba002.stdout:44 2021-02-10T11:36:18.834 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43, 44] 2021-02-10T11:36:18.835 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:36:18.835 DEBUG:teuthology.orchestra.run.gibba002:> sudo python3 -c ' 2021-02-10T11:36:18.835 DEBUG:teuthology.orchestra.run.gibba002:> import glob 2021-02-10T11:36:18.835 DEBUG:teuthology.orchestra.run.gibba002:> import re 2021-02-10T11:36:18.835 DEBUG:teuthology.orchestra.run.gibba002:> import os 2021-02-10T11:36:18.835 DEBUG:teuthology.orchestra.run.gibba002:> import subprocess 2021-02-10T11:36:18.835 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:36:18.835 DEBUG:teuthology.orchestra.run.gibba002:> def find_socket(client_name): 2021-02-10T11:36:18.835 DEBUG:teuthology.orchestra.run.gibba002:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2021-02-10T11:36:18.835 DEBUG:teuthology.orchestra.run.gibba002:> files = glob.glob(asok_path) 2021-02-10T11:36:18.835 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:36:18.836 DEBUG:teuthology.orchestra.run.gibba002:> # Given a non-glob path, it better be there 2021-02-10T11:36:18.836 DEBUG:teuthology.orchestra.run.gibba002:> if "*" not in asok_path: 2021-02-10T11:36:18.836 DEBUG:teuthology.orchestra.run.gibba002:> assert(len(files) == 1) 2021-02-10T11:36:18.836 DEBUG:teuthology.orchestra.run.gibba002:> return files[0] 2021-02-10T11:36:18.836 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:36:18.836 DEBUG:teuthology.orchestra.run.gibba002:> for f in files: 2021-02-10T11:36:18.836 DEBUG:teuthology.orchestra.run.gibba002:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2021-02-10T11:36:18.836 DEBUG:teuthology.orchestra.run.gibba002:> if os.path.exists("/proc/{0}".format(pid)): 2021-02-10T11:36:18.836 DEBUG:teuthology.orchestra.run.gibba002:> return f 2021-02-10T11:36:18.836 DEBUG:teuthology.orchestra.run.gibba002:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2021-02-10T11:36:18.836 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:36:18.836 DEBUG:teuthology.orchestra.run.gibba002:> print(find_socket("client.1")) 2021-02-10T11:36:18.837 DEBUG:teuthology.orchestra.run.gibba002:> ' 2021-02-10T11:36:19.198 INFO:teuthology.orchestra.run.gibba002.stdout:/var/run/ceph/ceph-client.1.201572.asok 2021-02-10T11:36:19.200 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.201572.asok 2021-02-10T11:36:19.200 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:36:19.200 DEBUG:teuthology.orchestra.run.gibba002:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.201572.asok status 2021-02-10T11:36:19.592 INFO:teuthology.orchestra.run.gibba002.stdout:{ 2021-02-10T11:36:19.592 INFO:teuthology.orchestra.run.gibba002.stdout: "metadata": { 2021-02-10T11:36:19.593 INFO:teuthology.orchestra.run.gibba002.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:36:19.593 INFO:teuthology.orchestra.run.gibba002.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:36:19.593 INFO:teuthology.orchestra.run.gibba002.stdout: "entity_id": "1", 2021-02-10T11:36:19.593 INFO:teuthology.orchestra.run.gibba002.stdout: "hostname": "gibba002", 2021-02-10T11:36:19.593 INFO:teuthology.orchestra.run.gibba002.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2021-02-10T11:36:19.593 INFO:teuthology.orchestra.run.gibba002.stdout: "pid": "201572", 2021-02-10T11:36:19.593 INFO:teuthology.orchestra.run.gibba002.stdout: "root": "/" 2021-02-10T11:36:19.593 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:36:19.593 INFO:teuthology.orchestra.run.gibba002.stdout: "dentry_count": 0, 2021-02-10T11:36:19.593 INFO:teuthology.orchestra.run.gibba002.stdout: "dentry_pinned_count": 0, 2021-02-10T11:36:19.593 INFO:teuthology.orchestra.run.gibba002.stdout: "id": 12288, 2021-02-10T11:36:19.593 INFO:teuthology.orchestra.run.gibba002.stdout: "inst": { 2021-02-10T11:36:19.593 INFO:teuthology.orchestra.run.gibba002.stdout: "name": { 2021-02-10T11:36:19.594 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "client", 2021-02-10T11:36:19.594 INFO:teuthology.orchestra.run.gibba002.stdout: "num": 12288 2021-02-10T11:36:19.594 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:36:19.594 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": { 2021-02-10T11:36:19.594 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:36:19.594 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:0", 2021-02-10T11:36:19.594 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 1424100616 2021-02-10T11:36:19.594 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:36:19.594 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:36:19.594 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": { 2021-02-10T11:36:19.594 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:36:19.594 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:0", 2021-02-10T11:36:19.595 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 1424100616 2021-02-10T11:36:19.595 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:36:19.595 INFO:teuthology.orchestra.run.gibba002.stdout: "inst_str": "client.12288 172.21.2.102:0/1424100616", 2021-02-10T11:36:19.595 INFO:teuthology.orchestra.run.gibba002.stdout: "addr_str": "172.21.2.102:0/1424100616", 2021-02-10T11:36:19.595 INFO:teuthology.orchestra.run.gibba002.stdout: "inode_count": 1, 2021-02-10T11:36:19.595 INFO:teuthology.orchestra.run.gibba002.stdout: "mds_epoch": 132, 2021-02-10T11:36:19.595 INFO:teuthology.orchestra.run.gibba002.stdout: "osd_epoch": 153, 2021-02-10T11:36:19.595 INFO:teuthology.orchestra.run.gibba002.stdout: "osd_epoch_barrier": 0, 2021-02-10T11:36:19.595 INFO:teuthology.orchestra.run.gibba002.stdout: "blacklisted": false 2021-02-10T11:36:19.595 INFO:teuthology.orchestra.run.gibba002.stdout:} 2021-02-10T11:36:19.596 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:36:19.597 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:36:19.597 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:36:19.932 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:36:19.932 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2021-02-10T11:36:19.932 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:36:19.933 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:36:20.279 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2021-02-10T11:36:20.713 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:20.694 7f91d7780700 1 Processor -- start 2021-02-10T11:36:20.713 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:20.694 7f91d7780700 1 -- start start 2021-02-10T11:36:20.713 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:20.694 7f91d7780700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f91d00d1870 0x7f91d01b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:20.713 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:20.694 7f91d7780700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f91d01b0ae0 0x7f91d0108940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:20.714 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:20.694 7f91d7780700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91d0109a60 con 0x7f91d00d1c70 2021-02-10T11:36:20.714 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:20.694 7f91d7780700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91d00e5ef0 con 0x7f91d01b0ae0 2021-02-10T11:36:20.714 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:20.694 7f91d7780700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91d0109750 con 0x7f91d00d1870 2021-02-10T11:36:20.714 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:20.694 7f91d5d9d700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f91d00d1c70 0x7f91d0108f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:58180/0 (socket says 172.21.2.102:58180) 2021-02-10T11:36:20.714 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:20.694 7f91d5d9d700 1 -- 172.21.2.102:0/1041659749 learned_addr learned my addr 172.21.2.102:0/1041659749 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:36:20.714 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:20.695 7f91d4d9b700 1 --2- 172.21.2.102:0/1041659749 >> v2:172.21.2.102:3300/0 conn(0x7f91d01b0ae0 0x7f91d0108940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:20.714 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:20.695 7f91c7fff700 1 -- 172.21.2.102:0/1041659749 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2018469573 0 0) 0x7f91c8001500 con 0x7f91d00d1c70 2021-02-10T11:36:20.714 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:20.695 7f91c7fff700 1 -- 172.21.2.102:0/1041659749 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91bc002fe0 con 0x7f91d00d1c70 2021-02-10T11:36:20.714 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:20.695 7f91d4d9b700 1 -- 172.21.2.102:0/1041659749 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f91d00d1870 msgr2=0x7f91d01b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:20.714 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:20.695 7f91d4d9b700 1 --2- 172.21.2.102:0/1041659749 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f91d00d1870 0x7f91d01b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:20.714 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:20.695 7f91c7fff700 1 -- 172.21.2.102:0/1041659749 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 771601761 0 0) 0x7f91c8001460 con 0x7f91d00d1c70 2021-02-10T11:36:20.714 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:20.695 7f91d4d9b700 1 -- 172.21.2.102:0/1041659749 >> v1:172.21.2.102:6789/0 conn(0x7f91d00d1c70 legacy=0x7f91d0108f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:20.715 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:20.695 7f91d4d9b700 1 -- 172.21.2.102:0/1041659749 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91d0088070 con 0x7f91d01b0ae0 2021-02-10T11:36:20.715 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:20.695 7f91d4d9b700 1 --2- 172.21.2.102:0/1041659749 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f91d01b0ae0 0x7f91d0108940 secure :-1 s=READY pgs=2808 cs=0 l=1 rev1=1 rx=0x7f91c0006650 tx=0x7f91c0009a80).ready entity=mon.0 client_cookie=8c01553326fe14e5 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:20.715 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:20.695 7f91c7fff700 1 -- 172.21.2.102:0/1041659749 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f91c00009b0 con 0x7f91d01b0ae0 2021-02-10T11:36:20.715 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:20.695 7f91c7fff700 1 -- 172.21.2.102:0/1041659749 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91c000c0c0 con 0x7f91d01b0ae0 2021-02-10T11:36:20.715 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:20.695 7f91c7fff700 1 -- 172.21.2.102:0/1041659749 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f91c000c5e0 con 0x7f91d01b0ae0 2021-02-10T11:36:20.715 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:20.695 7f91d7780700 1 -- 172.21.2.102:0/1041659749 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f91d01b0ae0 msgr2=0x7f91d0108940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:20.715 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:20.696 7f91d7780700 1 --2- 172.21.2.102:0/1041659749 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f91d01b0ae0 0x7f91d0108940 secure :-1 s=READY pgs=2808 cs=0 l=1 rev1=1 rx=0x7f91c0006650 tx=0x7f91c0009a80).stop 2021-02-10T11:36:20.715 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:20.696 7f91d7780700 1 -- 172.21.2.102:0/1041659749 shutdown_connections 2021-02-10T11:36:20.715 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:20.696 7f91d7780700 1 --2- 172.21.2.102:0/1041659749 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f91d01b0ae0 0x7f91d0108940 unknown :-1 s=CLOSED pgs=2808 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:20.715 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:20.696 7f91d7780700 1 --2- 172.21.2.102:0/1041659749 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f91d00d1870 0x7f91d01b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:20.715 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:20.696 7f91d7780700 1 -- 172.21.2.102:0/1041659749 >> 172.21.2.102:0/1041659749 conn(0x7f91d00ce420 msgr2=0x7f91d00d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:20.715 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:20.696 7f91d7780700 1 -- 172.21.2.102:0/1041659749 shutdown_connections 2021-02-10T11:36:20.715 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:20.696 7f91d7780700 1 -- 172.21.2.102:0/1041659749 wait complete. 2021-02-10T11:36:20.715 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:20.711 7f91d7780700 1 Processor -- start 2021-02-10T11:36:20.716 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:20.711 7f91d7780700 1 -- start start 2021-02-10T11:36:20.717 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:20.712 7f91d7780700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f91d00be800 0x7f91d00bec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:20.717 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:20.712 7f91d7780700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f91d00bf220 0x7f91d0099f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:20.717 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:20.712 7f91d7780700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f91d00bf620 0x7f91d009a560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:20.717 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:20.712 7f91d7780700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91d009da90 con 0x7f91d00bf220 2021-02-10T11:36:20.718 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:20.712 7f91d7780700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91d009ab80 con 0x7f91d00be800 2021-02-10T11:36:20.718 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:20.712 7f91d7780700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91d009ada0 con 0x7f91d00bf620 2021-02-10T11:36:20.718 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:20.712 7f91d4d9b700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f91d00bf220 0x7f91d0099f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:20.718 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:20.712 7f91d4d9b700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f91d00bf220 0x7f91d0099f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:37732/0 (socket says 172.21.2.102:37732) 2021-02-10T11:36:20.718 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:20.712 7f91d559c700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f91d00be800 0x7f91d00bec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:20.718 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:20.712 7f91d4d9b700 1 -- 172.21.2.102:0/1989992806 learned_addr learned my addr 172.21.2.102:0/1989992806 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:36:20.718 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:20.712 7f91d5d9d700 1 --2- 172.21.2.102:0/1989992806 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f91d00bf620 0x7f91d009a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:20.718 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:20.712 7f91d4d9b700 1 -- 172.21.2.102:0/1989992806 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f91d00bf620 msgr2=0x7f91d009a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:20.718 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:20.712 7f91d4d9b700 1 --2- 172.21.2.102:0/1989992806 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f91d00bf620 0x7f91d009a560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:20.718 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:20.712 7f91d4d9b700 1 -- 172.21.2.102:0/1989992806 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f91d00be800 msgr2=0x7f91d00bec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:20.719 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:20.713 7f91d4d9b700 1 --2- 172.21.2.102:0/1989992806 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f91d00be800 0x7f91d00bec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:20.719 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:20.713 7f91d4d9b700 1 -- 172.21.2.102:0/1989992806 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91d009afc0 con 0x7f91d00bf220 2021-02-10T11:36:20.719 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:20.713 7f91d5d9d700 1 --2- 172.21.2.102:0/1989992806 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f91d00bf620 0x7f91d009a560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:36:20.719 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:20.713 7f91d559c700 1 --2- 172.21.2.102:0/1989992806 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f91d00be800 0x7f91d00bec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:36:20.719 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:20.713 7f91d4d9b700 1 --2- 172.21.2.102:0/1989992806 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f91d00bf220 0x7f91d0099f40 secure :-1 s=READY pgs=2809 cs=0 l=1 rev1=1 rx=0x7f91c0009630 tx=0x7f91c000b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:20.719 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:20.713 7f91c5ffb700 1 -- 172.21.2.102:0/1989992806 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f91c00093a0 con 0x7f91d00bf220 2021-02-10T11:36:20.719 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:20.713 7f91c5ffb700 1 -- 172.21.2.102:0/1989992806 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91c000bd70 con 0x7f91d00bf220 2021-02-10T11:36:20.719 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:20.713 7f91d7780700 1 -- 172.21.2.102:0/1989992806 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91d009afc0 con 0x7f91d00bf220 2021-02-10T11:36:20.719 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:20.713 7f91c5ffb700 1 -- 172.21.2.102:0/1989992806 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f91c0005560 con 0x7f91d00bf220 2021-02-10T11:36:20.719 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:20.713 7f91d7780700 1 -- 172.21.2.102:0/1989992806 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91d009afc0 con 0x7f91d00bf220 2021-02-10T11:36:20.719 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:20.713 7f91c5ffb700 1 -- 172.21.2.102:0/1989992806 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f91c0005700 con 0x7f91d00bf220 2021-02-10T11:36:20.719 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:20.713 7f91c5ffb700 1 --2- 172.21.2.102:0/1989992806 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f91b00186d0 0x7f91b001abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:20.719 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:20.714 7f91c5ffb700 1 -- 172.21.2.102:0/1989992806 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(153..153 src has 1..153) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f91c0032620 con 0x7f91d00bf220 2021-02-10T11:36:20.720 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:20.714 7f91d559c700 1 --2- 172.21.2.102:0/1989992806 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f91b00186d0 0x7f91b001abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:20.720 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:20.714 7f91d559c700 1 --2- 172.21.2.102:0/1989992806 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f91b00186d0 0x7f91b001abd0 crc :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:20.746 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:20.744 7f91d7780700 1 -- 172.21.2.102:0/1989992806 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91d0082520 con 0x7f91d00bf220 2021-02-10T11:36:20.749 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:20.747 7f91c5ffb700 1 -- 172.21.2.102:0/1989992806 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f91c00054f0 con 0x7f91d00bf220 2021-02-10T11:36:20.886 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:20.884 7f91d7780700 1 -- 172.21.2.102:0/1989992806 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f91d0109a60 con 0x7f91d00bf220 2021-02-10T11:36:21.490 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:21.487 7f91c5ffb700 1 -- 172.21.2.102:0/1989992806 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v133) v1 ==== 136+0+0 (secure 0 0 0) 0x7f91c0005700 con 0x7f91d00bf220 2021-02-10T11:36:21.506 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:21.504 7f91d7780700 1 -- 172.21.2.102:0/1989992806 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f91b00186d0 msgr2=0x7f91b001abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:21.507 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:21.504 7f91d7780700 1 --2- 172.21.2.102:0/1989992806 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f91b00186d0 0x7f91b001abd0 crc :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:21.507 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:21.504 7f91d7780700 1 -- 172.21.2.102:0/1989992806 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f91d00bf220 msgr2=0x7f91d0099f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:21.507 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:21.504 7f91d7780700 1 --2- 172.21.2.102:0/1989992806 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f91d00bf220 0x7f91d0099f40 secure :-1 s=READY pgs=2809 cs=0 l=1 rev1=1 rx=0x7f91c0009630 tx=0x7f91c000b980).stop 2021-02-10T11:36:21.507 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:21.505 7f91d7780700 1 -- 172.21.2.102:0/1989992806 shutdown_connections 2021-02-10T11:36:21.507 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:21.505 7f91d7780700 1 --2- 172.21.2.102:0/1989992806 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f91d00bf620 0x7f91d009a560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:21.507 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:21.505 7f91d7780700 1 --2- 172.21.2.102:0/1989992806 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f91d00bf220 0x7f91d0099f40 unknown :-1 s=CLOSED pgs=2809 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:21.508 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:21.505 7f91d7780700 1 --2- 172.21.2.102:0/1989992806 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f91b00186d0 0x7f91b001abd0 unknown :-1 s=CLOSED pgs=1233 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:21.508 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:21.505 7f91d7780700 1 --2- 172.21.2.102:0/1989992806 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f91d00be800 0x7f91d00bec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:21.508 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:21.505 7f91d7780700 1 -- 172.21.2.102:0/1989992806 >> 172.21.2.102:0/1989992806 conn(0x7f91d00b2b50 msgr2=0x7f91d0087010 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:21.508 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:21.505 7f91d7780700 1 -- 172.21.2.102:0/1989992806 shutdown_connections 2021-02-10T11:36:21.508 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:21.505 7f91d7780700 1 -- 172.21.2.102:0/1989992806 wait complete. 2021-02-10T11:36:21.515 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs volume ls 2021-02-10T11:36:21.945 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:21.925 7f0c3cfb1700 1 Processor -- start 2021-02-10T11:36:21.945 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:21.925 7f0c3cfb1700 1 -- start start 2021-02-10T11:36:21.946 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:21.926 7f0c3cfb1700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0c380d1870 0x7f0c381b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:21.946 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:21.926 7f0c3cfb1700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f0c380f0650 0x7f0c380cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:21.946 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:21.926 7f0c3cfb1700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c380f0b10 con 0x7f0c381b0a10 2021-02-10T11:36:21.946 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:21.926 7f0c3cfb1700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c380fe230 con 0x7f0c380f0650 2021-02-10T11:36:21.946 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:21.926 7f0c3cfb1700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c380e5ed0 con 0x7f0c380d1870 2021-02-10T11:36:21.946 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:21.927 7f0c36d9d700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0c380d1870 0x7f0c381b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:21.946 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:21.927 7f0c36d9d700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0c380d1870 0x7f0c381b03f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.113:3301/0 says I am v2:172.21.2.102:43274/0 (socket says 172.21.2.102:43274) 2021-02-10T11:36:21.946 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:21.927 7f0c36d9d700 1 -- 172.21.2.102:0/2389368837 learned_addr learned my addr 172.21.2.102:0/2389368837 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:36:21.946 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:21.927 7f0c3659c700 1 --1- 172.21.2.102:0/2389368837 >> v1:172.21.2.113:6789/0 conn(0x7f0c381b0a10 0x7f0c380cf330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.113:6789/0 says I am v1:172.21.2.102:37288/0 (socket says 172.21.2.102:37288) 2021-02-10T11:36:21.947 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:21.927 7f0c3759e700 1 --2- 172.21.2.102:0/2389368837 >> v2:172.21.2.113:3300/0 conn(0x7f0c380f0650 0x7f0c380cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:21.947 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:21.927 7f0c36d9d700 1 -- 172.21.2.102:0/2389368837 >> v2:172.21.2.113:3300/0 conn(0x7f0c380f0650 msgr2=0x7f0c380cfb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:21.947 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:21.927 7f0c36d9d700 1 --2- 172.21.2.102:0/2389368837 >> v2:172.21.2.113:3300/0 conn(0x7f0c380f0650 0x7f0c380cfb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:21.947 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:21.927 7f0c36d9d700 1 -- 172.21.2.102:0/2389368837 >> v1:172.21.2.113:6789/0 conn(0x7f0c381b0a10 legacy=0x7f0c380cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:21.947 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:21.927 7f0c36d9d700 1 -- 172.21.2.102:0/2389368837 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c380f20a0 con 0x7f0c380d1870 2021-02-10T11:36:21.947 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:21.927 7f0c3759e700 1 --2- 172.21.2.102:0/2389368837 >> v2:172.21.2.113:3300/0 conn(0x7f0c380f0650 0x7f0c380cfb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:36:21.947 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:21.928 7f0c36d9d700 1 --2- 172.21.2.102:0/2389368837 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0c380d1870 0x7f0c381b03f0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 rx=0x7f0c20006700 tx=0x7f0c20009b80).ready entity=mon.2 client_cookie=e1dfdc89d893d612 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:21.947 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:21.928 7f0c35d9b700 1 -- 172.21.2.102:0/2389368837 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0c2000c0c0 con 0x7f0c380d1870 2021-02-10T11:36:21.947 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:21.928 7f0c35d9b700 1 -- 172.21.2.102:0/2389368837 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c2000c340 con 0x7f0c380d1870 2021-02-10T11:36:21.948 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:21.928 7f0c35d9b700 1 -- 172.21.2.102:0/2389368837 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0c2000c860 con 0x7f0c380d1870 2021-02-10T11:36:21.948 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:21.928 7f0c3cfb1700 1 -- 172.21.2.102:0/2389368837 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0c380d1870 msgr2=0x7f0c381b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:21.948 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:21.928 7f0c3cfb1700 1 --2- 172.21.2.102:0/2389368837 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0c380d1870 0x7f0c381b03f0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 rx=0x7f0c20006700 tx=0x7f0c20009b80).stop 2021-02-10T11:36:21.948 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:21.929 7f0c3cfb1700 1 -- 172.21.2.102:0/2389368837 shutdown_connections 2021-02-10T11:36:21.948 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:21.929 7f0c3cfb1700 1 --2- 172.21.2.102:0/2389368837 >> v2:172.21.2.113:3300/0 conn(0x7f0c380f0650 0x7f0c380cfb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:21.948 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:21.929 7f0c3cfb1700 1 --2- 172.21.2.102:0/2389368837 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0c380d1870 0x7f0c381b03f0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:21.948 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:21.929 7f0c3cfb1700 1 -- 172.21.2.102:0/2389368837 >> 172.21.2.102:0/2389368837 conn(0x7f0c380ce450 msgr2=0x7f0c380d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:21.948 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:21.929 7f0c3cfb1700 1 -- 172.21.2.102:0/2389368837 shutdown_connections 2021-02-10T11:36:21.948 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:21.929 7f0c3cfb1700 1 -- 172.21.2.102:0/2389368837 wait complete. 2021-02-10T11:36:21.950 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:21.944 7f0c3cfb1700 1 Processor -- start 2021-02-10T11:36:21.950 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:21.944 7f0c3cfb1700 1 -- start start 2021-02-10T11:36:21.950 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:21.944 7f0c3cfb1700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0c381a06a0 0x7f0c3809f380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:21.950 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:21.944 7f0c3cfb1700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0c3809f9a0 0x7f0c3808b560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:21.950 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:21.944 7f0c3cfb1700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0c3808bb80 0x7f0c3808bf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:21.950 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:21.944 7f0c3cfb1700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c3808c780 con 0x7f0c3808bb80 2021-02-10T11:36:21.950 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:21.944 7f0c3cfb1700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c3809fda0 con 0x7f0c3809f9a0 2021-02-10T11:36:21.951 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:21.944 7f0c3cfb1700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c3808cd00 con 0x7f0c381a06a0 2021-02-10T11:36:21.951 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:21.944 7f0c3759e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0c3808bb80 0x7f0c3808bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:21.951 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:21.944 7f0c3759e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0c3808bb80 0x7f0c3808bf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:37748/0 (socket says 172.21.2.102:37748) 2021-02-10T11:36:21.951 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:21.944 7f0c3759e700 1 -- 172.21.2.102:0/4035796625 learned_addr learned my addr 172.21.2.102:0/4035796625 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:36:21.951 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:21.945 7f0c3759e700 1 -- 172.21.2.102:0/4035796625 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0c381a06a0 msgr2=0x7f0c3809f380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:21.951 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:21.945 7f0c3759e700 1 --2- 172.21.2.102:0/4035796625 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0c381a06a0 0x7f0c3809f380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:21.951 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:21.945 7f0c3759e700 1 -- 172.21.2.102:0/4035796625 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0c3809f9a0 msgr2=0x7f0c3808b560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:21.951 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:21.945 7f0c3759e700 1 --2- 172.21.2.102:0/4035796625 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0c3809f9a0 0x7f0c3808b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:21.951 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:21.945 7f0c3759e700 1 -- 172.21.2.102:0/4035796625 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c3808cf20 con 0x7f0c3808bb80 2021-02-10T11:36:21.951 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:21.945 7f0c3759e700 1 --2- 172.21.2.102:0/4035796625 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0c3808bb80 0x7f0c3808bf80 secure :-1 s=READY pgs=2810 cs=0 l=1 rev1=1 rx=0x7f0c3000b970 tx=0x7f0c3000bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:21.951 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:21.945 7f0c2f7fe700 1 -- 172.21.2.102:0/4035796625 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0c30000f90 con 0x7f0c3808bb80 2021-02-10T11:36:21.951 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:21.945 7f0c2f7fe700 1 -- 172.21.2.102:0/4035796625 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c30003030 con 0x7f0c3808bb80 2021-02-10T11:36:21.952 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:21.945 7f0c2f7fe700 1 -- 172.21.2.102:0/4035796625 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0c30001400 con 0x7f0c3808bb80 2021-02-10T11:36:21.952 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:21.945 7f0c3cfb1700 1 -- 172.21.2.102:0/4035796625 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c3808cf20 con 0x7f0c3808bb80 2021-02-10T11:36:21.952 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:21.945 7f0c3cfb1700 1 -- 172.21.2.102:0/4035796625 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c3808d4b0 con 0x7f0c3808bb80 2021-02-10T11:36:21.952 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:21.946 7f0c2f7fe700 1 -- 172.21.2.102:0/4035796625 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f0c300011f0 con 0x7f0c3808bb80 2021-02-10T11:36:21.952 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:21.946 7f0c2f7fe700 1 --2- 172.21.2.102:0/4035796625 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0c1c0186c0 0x7f0c1c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:21.952 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:21.946 7f0c2f7fe700 1 -- 172.21.2.102:0/4035796625 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(153..153 src has 1..153) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f0c300328d0 con 0x7f0c3808bb80 2021-02-10T11:36:21.952 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:21.946 7f0c36d9d700 1 --2- 172.21.2.102:0/4035796625 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0c1c0186c0 0x7f0c1c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:21.952 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:21.947 7f0c36d9d700 1 --2- 172.21.2.102:0/4035796625 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0c1c0186c0 0x7f0c1c01abc0 crc :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:21.977 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:21.975 7f0c3cfb1700 1 -- 172.21.2.102:0/4035796625 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c380ebc50 con 0x7f0c3808bb80 2021-02-10T11:36:21.980 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:21.978 7f0c2f7fe700 1 -- 172.21.2.102:0/4035796625 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f0c30000f90 con 0x7f0c3808bb80 2021-02-10T11:36:22.119 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:22.113 7f0c3cfb1700 1 -- 172.21.2.102:0/4035796625 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs volume ls", "target": ["mgr", ""]}) v1 -- 0x7f0c3808d4b0 con 0x7f0c1c0186c0 2021-02-10T11:36:22.119 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:22.113 7f0c2f7fe700 1 -- 172.21.2.102:0/4035796625 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+41 (crc 0 0 0) 0x7f0c20000940 con 0x7f0c1c0186c0 2021-02-10T11:36:22.119 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:22.115 7f0c3cfb1700 1 -- 172.21.2.102:0/4035796625 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0c1c0186c0 msgr2=0x7f0c1c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:22.119 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:22.115 7f0c3cfb1700 1 --2- 172.21.2.102:0/4035796625 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0c1c0186c0 0x7f0c1c01abc0 crc :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:22.119 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:22.115 7f0c3cfb1700 1 -- 172.21.2.102:0/4035796625 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0c3808bb80 msgr2=0x7f0c3808bf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:22.119 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:22.115 7f0c3cfb1700 1 --2- 172.21.2.102:0/4035796625 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0c3808bb80 0x7f0c3808bf80 secure :-1 s=READY pgs=2810 cs=0 l=1 rev1=1 rx=0x7f0c3000b970 tx=0x7f0c3000bd10).stop 2021-02-10T11:36:22.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:22.115 7f0c3cfb1700 1 -- 172.21.2.102:0/4035796625 shutdown_connections 2021-02-10T11:36:22.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:22.115 7f0c3cfb1700 1 --2- 172.21.2.102:0/4035796625 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0c3808bb80 0x7f0c3808bf80 unknown :-1 s=CLOSED pgs=2810 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:22.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:22.115 7f0c3cfb1700 1 --2- 172.21.2.102:0/4035796625 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0c1c0186c0 0x7f0c1c01abc0 unknown :-1 s=CLOSED pgs=1234 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:22.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:22.115 7f0c3cfb1700 1 --2- 172.21.2.102:0/4035796625 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0c3809f9a0 0x7f0c3808b560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:22.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:22.115 7f0c3cfb1700 1 --2- 172.21.2.102:0/4035796625 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0c381a06a0 0x7f0c3809f380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:22.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:22.115 7f0c3cfb1700 1 -- 172.21.2.102:0/4035796625 >> 172.21.2.102:0/4035796625 conn(0x7f0c380b83b0 msgr2=0x7f0c380a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:22.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:22.115 7f0c3cfb1700 1 -- 172.21.2.102:0/4035796625 shutdown_connections 2021-02-10T11:36:22.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:22.115 7f0c3cfb1700 1 -- 172.21.2.102:0/4035796625 wait complete. 2021-02-10T11:36:22.120 INFO:teuthology.orchestra.run.gibba002.stdout:[ 2021-02-10T11:36:22.120 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:36:22.120 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "cephfs" 2021-02-10T11:36:22.120 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:36:22.121 INFO:teuthology.orchestra.run.gibba002.stdout:] 2021-02-10T11:36:22.126 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config set mon mon_allow_pool_delete True 2021-02-10T11:36:22.560 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:22.541 7f2bce96e700 1 Processor -- start 2021-02-10T11:36:22.561 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:22.541 7f2bce96e700 1 -- start start 2021-02-10T11:36:22.561 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:22.542 7f2bce96e700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f2bc80d1870 0x7f2bc81b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:22.561 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:22.542 7f2bce96e700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2bc81b0ae0 0x7f2bc8108940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:22.561 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:22.542 7f2bce96e700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f2bc80d1c70 0x7f2bc8108f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:22.561 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:22.542 7f2bce96e700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2bc80fe1c0 con 0x7f2bc80d1c70 2021-02-10T11:36:22.561 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:22.542 7f2bce96e700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2bc8109a60 con 0x7f2bc80d1870 2021-02-10T11:36:22.561 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:22.542 7f2bce96e700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2bc8088070 con 0x7f2bc81b0ae0 2021-02-10T11:36:22.561 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:22.542 7f2bccf8b700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f2bc80d1c70 0x7f2bc8108f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:22.561 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:22.542 7f2bccf8b700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f2bc80d1c70 0x7f2bc8108f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:37756/0 (socket says 172.21.2.102:37756) 2021-02-10T11:36:22.561 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:22.542 7f2bccf8b700 1 -- 172.21.2.102:0/2598872820 learned_addr learned my addr 172.21.2.102:0/2598872820 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:36:22.561 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:22.542 7f2bccf8b700 1 -- 172.21.2.102:0/2598872820 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2bc81b0ae0 msgr2=0x7f2bc8108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:22.561 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:22.542 7f2bccf8b700 1 --2- 172.21.2.102:0/2598872820 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2bc81b0ae0 0x7f2bc8108940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:22.561 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:22.542 7f2bccf8b700 1 -- 172.21.2.102:0/2598872820 >> v2:172.21.2.113:3300/0 conn(0x7f2bc80d1870 msgr2=0x7f2bc81b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:22.562 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:22.542 7f2bccf8b700 1 --2- 172.21.2.102:0/2598872820 >> v2:172.21.2.113:3300/0 conn(0x7f2bc80d1870 0x7f2bc81b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:22.562 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:22.542 7f2bccf8b700 1 -- 172.21.2.102:0/2598872820 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bc8109580 con 0x7f2bc80d1c70 2021-02-10T11:36:22.562 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:22.543 7f2bccf8b700 1 --2- 172.21.2.102:0/2598872820 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2bc80d1c70 0x7f2bc8108f60 secure :-1 s=READY pgs=2811 cs=0 l=1 rev1=1 rx=0x7f2bc0006f00 tx=0x7f2bc0009f10).ready entity=mon.0 client_cookie=3f3f06dc08bb956a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:22.562 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:22.543 7f2bc6ffd700 1 -- 172.21.2.102:0/2598872820 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2bc000b3b0 con 0x7f2bc80d1c70 2021-02-10T11:36:22.562 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:22.543 7f2bc6ffd700 1 -- 172.21.2.102:0/2598872820 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bc000b730 con 0x7f2bc80d1c70 2021-02-10T11:36:22.562 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:22.543 7f2bc6ffd700 1 -- 172.21.2.102:0/2598872820 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2bc000bb90 con 0x7f2bc80d1c70 2021-02-10T11:36:22.562 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:22.543 7f2bce96e700 1 -- 172.21.2.102:0/2598872820 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2bc80d1c70 msgr2=0x7f2bc8108f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:22.562 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:22.543 7f2bce96e700 1 --2- 172.21.2.102:0/2598872820 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2bc80d1c70 0x7f2bc8108f60 secure :-1 s=READY pgs=2811 cs=0 l=1 rev1=1 rx=0x7f2bc0006f00 tx=0x7f2bc0009f10).stop 2021-02-10T11:36:22.562 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:22.544 7f2bce96e700 1 -- 172.21.2.102:0/2598872820 shutdown_connections 2021-02-10T11:36:22.562 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:22.544 7f2bce96e700 1 --2- 172.21.2.102:0/2598872820 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2bc81b0ae0 0x7f2bc8108940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:22.562 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:22.544 7f2bce96e700 1 --2- 172.21.2.102:0/2598872820 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2bc80d1c70 0x7f2bc8108f60 unknown :-1 s=CLOSED pgs=2811 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:22.562 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:22.544 7f2bce96e700 1 --2- 172.21.2.102:0/2598872820 >> v2:172.21.2.113:3300/0 conn(0x7f2bc80d1870 0x7f2bc81b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:22.562 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:22.544 7f2bce96e700 1 -- 172.21.2.102:0/2598872820 >> 172.21.2.102:0/2598872820 conn(0x7f2bc80ce420 msgr2=0x7f2bc80d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:22.563 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:22.544 7f2bce96e700 1 -- 172.21.2.102:0/2598872820 shutdown_connections 2021-02-10T11:36:22.563 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:22.544 7f2bce96e700 1 -- 172.21.2.102:0/2598872820 wait complete. 2021-02-10T11:36:22.564 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:22.559 7f2bce96e700 1 Processor -- start 2021-02-10T11:36:22.564 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:22.559 7f2bce96e700 1 -- start start 2021-02-10T11:36:22.564 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:22.559 7f2bce96e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2bc80c3820 0x7f2bc8084cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:22.564 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:22.560 7f2bce96e700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2bc80a0290 0x7f2bc80852d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:22.565 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:22.560 7f2bce96e700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2bc8086270 0x7f2bc8086670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:22.565 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:22.560 7f2bce96e700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bc80c0700 con 0x7f2bc80c3820 2021-02-10T11:36:22.565 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:22.560 7f2bce96e700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bc80858f0 con 0x7f2bc80a0290 2021-02-10T11:36:22.565 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:22.560 7f2bce96e700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2bc8085b10 con 0x7f2bc8086270 2021-02-10T11:36:22.565 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:22.560 7f2bc7fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2bc80c3820 0x7f2bc8084cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:22.565 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:22.560 7f2bc7fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2bc80c3820 0x7f2bc8084cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:37758/0 (socket says 172.21.2.102:37758) 2021-02-10T11:36:22.565 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:22.560 7f2bc7fff700 1 -- 172.21.2.102:0/500790248 learned_addr learned my addr 172.21.2.102:0/500790248 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:36:22.565 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:22.560 7f2bccf8b700 1 --2- 172.21.2.102:0/500790248 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2bc8086270 0x7f2bc8086670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:22.565 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:22.560 7f2bc77fe700 1 --2- 172.21.2.102:0/500790248 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2bc80a0290 0x7f2bc80852d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:22.565 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:22.560 7f2bc7fff700 1 -- 172.21.2.102:0/500790248 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2bc8086270 msgr2=0x7f2bc8086670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:22.565 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:22.560 7f2bc7fff700 1 --2- 172.21.2.102:0/500790248 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2bc8086270 0x7f2bc8086670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:22.565 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:22.560 7f2bc7fff700 1 -- 172.21.2.102:0/500790248 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2bc80a0290 msgr2=0x7f2bc80852d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:22.566 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:22.560 7f2bc7fff700 1 --2- 172.21.2.102:0/500790248 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2bc80a0290 0x7f2bc80852d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:22.566 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:22.560 7f2bc7fff700 1 -- 172.21.2.102:0/500790248 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bc8086c90 con 0x7f2bc80c3820 2021-02-10T11:36:22.566 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:22.560 7f2bc7fff700 1 --2- 172.21.2.102:0/500790248 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2bc80c3820 0x7f2bc8084cb0 secure :-1 s=READY pgs=2812 cs=0 l=1 rev1=1 rx=0x7f2bb8006f00 tx=0x7f2bb800b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:22.566 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:22.561 7f2bc4ff9700 1 -- 172.21.2.102:0/500790248 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2bb800bd30 con 0x7f2bc80c3820 2021-02-10T11:36:22.566 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:22.561 7f2bc4ff9700 1 -- 172.21.2.102:0/500790248 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bb8000c90 con 0x7f2bc80c3820 2021-02-10T11:36:22.566 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:22.561 7f2bce96e700 1 -- 172.21.2.102:0/500790248 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2bc8086c90 con 0x7f2bc80c3820 2021-02-10T11:36:22.566 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:22.561 7f2bc4ff9700 1 -- 172.21.2.102:0/500790248 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2bb80022a0 con 0x7f2bc80c3820 2021-02-10T11:36:22.566 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:22.561 7f2bce96e700 1 -- 172.21.2.102:0/500790248 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2bc81a2f00 con 0x7f2bc80c3820 2021-02-10T11:36:22.566 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:22.561 7f2bc4ff9700 1 -- 172.21.2.102:0/500790248 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f2bb8005310 con 0x7f2bc80c3820 2021-02-10T11:36:22.566 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:22.561 7f2bc4ff9700 1 --2- 172.21.2.102:0/500790248 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2ba8018730 0x7f2ba801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:22.566 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:22.561 7f2bc4ff9700 1 -- 172.21.2.102:0/500790248 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(153..153 src has 1..153) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f2bb8032650 con 0x7f2bc80c3820 2021-02-10T11:36:22.566 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:22.561 7f2bc77fe700 1 --2- 172.21.2.102:0/500790248 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2ba8018730 0x7f2ba801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:22.566 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:22.562 7f2bc77fe700 1 --2- 172.21.2.102:0/500790248 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2ba8018730 0x7f2ba801abc0 crc :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:22.594 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:22.592 7f2bce96e700 1 -- 172.21.2.102:0/500790248 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2bc8082a70 con 0x7f2bc80c3820 2021-02-10T11:36:22.596 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:22.595 7f2bc4ff9700 1 -- 172.21.2.102:0/500790248 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f2bb8000940 con 0x7f2bc80c3820 2021-02-10T11:36:22.748 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:22.746 7f2bce96e700 1 -- 172.21.2.102:0/500790248 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f2bc8082a70 con 0x7f2bc80c3820 2021-02-10T11:36:22.748 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:22.746 7f2bc4ff9700 1 -- 172.21.2.102:0/500790248 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v38)=0 v38) v1 ==== 122+0+0 (secure 0 0 0) 0x7f2bb8005310 con 0x7f2bc80c3820 2021-02-10T11:36:22.750 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:22.748 7f2bce96e700 1 -- 172.21.2.102:0/500790248 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2ba8018730 msgr2=0x7f2ba801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:22.750 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:22.748 7f2bce96e700 1 --2- 172.21.2.102:0/500790248 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2ba8018730 0x7f2ba801abc0 crc :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:22.750 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:22.748 7f2bce96e700 1 -- 172.21.2.102:0/500790248 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2bc80c3820 msgr2=0x7f2bc8084cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:22.750 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:22.748 7f2bce96e700 1 --2- 172.21.2.102:0/500790248 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2bc80c3820 0x7f2bc8084cb0 secure :-1 s=READY pgs=2812 cs=0 l=1 rev1=1 rx=0x7f2bb8006f00 tx=0x7f2bb800b940).stop 2021-02-10T11:36:22.750 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:22.748 7f2bce96e700 1 -- 172.21.2.102:0/500790248 shutdown_connections 2021-02-10T11:36:22.750 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:22.748 7f2bce96e700 1 --2- 172.21.2.102:0/500790248 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2bc8086270 0x7f2bc8086670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:22.751 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:22.748 7f2bce96e700 1 --2- 172.21.2.102:0/500790248 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2ba8018730 0x7f2ba801abc0 unknown :-1 s=CLOSED pgs=1235 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:22.751 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:22.748 7f2bce96e700 1 --2- 172.21.2.102:0/500790248 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2bc80a0290 0x7f2bc80852d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:22.751 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:22.748 7f2bce96e700 1 --2- 172.21.2.102:0/500790248 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2bc80c3820 0x7f2bc8084cb0 unknown :-1 s=CLOSED pgs=2812 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:22.751 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:22.748 7f2bce96e700 1 -- 172.21.2.102:0/500790248 >> 172.21.2.102:0/500790248 conn(0x7f2bc80872c0 msgr2=0x7f2bc8095700 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:22.751 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:22.749 7f2bce96e700 1 -- 172.21.2.102:0/500790248 shutdown_connections 2021-02-10T11:36:22.751 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:22.749 7f2bce96e700 1 -- 172.21.2.102:0/500790248 wait complete. 2021-02-10T11:36:22.759 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs non_existent_subvolume 2021-02-10T11:36:23.190 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:23.172 7f49dcdcf700 1 Processor -- start 2021-02-10T11:36:23.191 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:23.172 7f49dcdcf700 1 -- start start 2021-02-10T11:36:23.191 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:23.173 7f49dcdcf700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f49d80d1870 0x7f49d81b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:23.191 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:23.173 7f49dcdcf700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f49d80cff70 0x7f49d8109130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:23.191 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:23.173 7f49dcdcf700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49d8109a60 con 0x7f49d81b0ae0 2021-02-10T11:36:23.191 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:23.173 7f49dcdcf700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49d8109750 con 0x7f49d80d1870 2021-02-10T11:36:23.191 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:23.173 7f49dcdcf700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49d8088070 con 0x7f49d80cff70 2021-02-10T11:36:23.191 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:23.173 7f49d6b13700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f49d80d1870 0x7f49d81b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:23.191 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:23.173 7f49d6312700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f49d81b0ae0 0x7f49d8108940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:58224/0 (socket says 172.21.2.102:58224) 2021-02-10T11:36:23.191 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:23.173 7f49d6312700 1 -- 172.21.2.102:0/4009523967 learned_addr learned my addr 172.21.2.102:0/4009523967 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:36:23.191 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:23.173 7f49d6b13700 1 -- 172.21.2.102:0/4009523967 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f49d80cff70 msgr2=0x7f49d8109130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:23.191 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:23.173 7f49d6b13700 1 --2- 172.21.2.102:0/4009523967 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f49d80cff70 0x7f49d8109130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:23.192 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:23.173 7f49d6b13700 1 -- 172.21.2.102:0/4009523967 >> v1:172.21.2.102:6789/0 conn(0x7f49d81b0ae0 legacy=0x7f49d8108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:23.192 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:23.173 7f49d5b11700 1 -- 172.21.2.102:0/4009523967 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3517557561 0 0) 0x7f49c00010b0 con 0x7f49d81b0ae0 2021-02-10T11:36:23.192 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:23.173 7f49d6b13700 1 -- 172.21.2.102:0/4009523967 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49d80d1c70 con 0x7f49d80d1870 2021-02-10T11:36:23.192 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:23.174 7f49d6b13700 1 --2- 172.21.2.102:0/4009523967 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f49d80d1870 0x7f49d81b04c0 secure :-1 s=READY pgs=2814 cs=0 l=1 rev1=1 rx=0x7f49c8006650 tx=0x7f49c8009a00).ready entity=mon.0 client_cookie=f8e638e141aba721 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:23.192 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:23.174 7f49d5b11700 1 -- 172.21.2.102:0/4009523967 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f49c8009df0 con 0x7f49d80d1870 2021-02-10T11:36:23.192 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:23.174 7f49d5b11700 1 -- 172.21.2.102:0/4009523967 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49c80009b0 con 0x7f49d80d1870 2021-02-10T11:36:23.192 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:23.174 7f49d5b11700 1 -- 172.21.2.102:0/4009523967 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f49c800c2b0 con 0x7f49d80d1870 2021-02-10T11:36:23.192 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:23.174 7f49dcdcf700 1 -- 172.21.2.102:0/4009523967 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f49d80d1870 msgr2=0x7f49d81b04c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:23.192 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:23.174 7f49dcdcf700 1 --2- 172.21.2.102:0/4009523967 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f49d80d1870 0x7f49d81b04c0 secure :-1 s=READY pgs=2814 cs=0 l=1 rev1=1 rx=0x7f49c8006650 tx=0x7f49c8009a00).stop 2021-02-10T11:36:23.192 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:23.174 7f49dcdcf700 1 -- 172.21.2.102:0/4009523967 shutdown_connections 2021-02-10T11:36:23.192 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:23.174 7f49dcdcf700 1 --2- 172.21.2.102:0/4009523967 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f49d80cff70 0x7f49d8109130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:23.192 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:23.174 7f49dcdcf700 1 --2- 172.21.2.102:0/4009523967 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f49d80d1870 0x7f49d81b04c0 unknown :-1 s=CLOSED pgs=2814 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:23.192 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:23.174 7f49dcdcf700 1 -- 172.21.2.102:0/4009523967 >> 172.21.2.102:0/4009523967 conn(0x7f49d80ce420 msgr2=0x7f49d80d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:23.193 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:23.175 7f49dcdcf700 1 -- 172.21.2.102:0/4009523967 shutdown_connections 2021-02-10T11:36:23.193 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:23.175 7f49dcdcf700 1 -- 172.21.2.102:0/4009523967 wait complete. 2021-02-10T11:36:23.194 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:23.189 7f49dcdcf700 1 Processor -- start 2021-02-10T11:36:23.194 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:23.189 7f49dcdcf700 1 -- start start 2021-02-10T11:36:23.194 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:23.189 7f49dcdcf700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f49d818f9d0 0x7f49d8084cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:23.194 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:23.189 7f49dcdcf700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f49d80c23e0 0x7f49d81a44b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:23.194 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:23.189 7f49dcdcf700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f49d80c27e0 0x7f49d80859f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:23.194 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:23.189 7f49dcdcf700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49d8085470 con 0x7f49d818f9d0 2021-02-10T11:36:23.195 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:23.189 7f49dcdcf700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49d818fdd0 con 0x7f49d80c27e0 2021-02-10T11:36:23.195 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:23.189 7f49dcdcf700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49d81a4ad0 con 0x7f49d80c23e0 2021-02-10T11:36:23.195 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:23.189 7f49d6b13700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f49d818f9d0 0x7f49d8084cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:23.195 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:23.189 7f49d6b13700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f49d818f9d0 0x7f49d8084cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:37774/0 (socket says 172.21.2.102:37774) 2021-02-10T11:36:23.195 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:23.189 7f49d6b13700 1 -- 172.21.2.102:0/3960083585 learned_addr learned my addr 172.21.2.102:0/3960083585 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:36:23.195 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:23.190 7f49d6b13700 1 -- 172.21.2.102:0/3960083585 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f49d80c23e0 msgr2=0x7f49d81a44b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:36:23.195 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:23.190 7f49d6b13700 1 --2- 172.21.2.102:0/3960083585 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f49d80c23e0 0x7f49d81a44b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:23.195 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:23.190 7f49d6b13700 1 -- 172.21.2.102:0/3960083585 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f49d80c27e0 msgr2=0x7f49d80859f0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:36:23.195 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:23.190 7f49d6b13700 1 --2- 172.21.2.102:0/3960083585 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f49d80c27e0 0x7f49d80859f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:23.195 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:23.190 7f49d6b13700 1 -- 172.21.2.102:0/3960083585 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49d81a4cf0 con 0x7f49d818f9d0 2021-02-10T11:36:23.195 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:23.190 7f49d6b13700 1 --2- 172.21.2.102:0/3960083585 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f49d818f9d0 0x7f49d8084cb0 secure :-1 s=READY pgs=2815 cs=0 l=1 rev1=1 rx=0x7f49c80096a0 tx=0x7f49c800b9b0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:23.195 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:23.190 7f49cf7fe700 1 -- 172.21.2.102:0/3960083585 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f49c80093a0 con 0x7f49d818f9d0 2021-02-10T11:36:23.195 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:23.190 7f49cf7fe700 1 -- 172.21.2.102:0/3960083585 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49c80067f0 con 0x7f49d818f9d0 2021-02-10T11:36:23.196 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:23.190 7f49cf7fe700 1 -- 172.21.2.102:0/3960083585 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f49c8002220 con 0x7f49d818f9d0 2021-02-10T11:36:23.196 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:23.190 7f49dcdcf700 1 -- 172.21.2.102:0/3960083585 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49d81a4cf0 con 0x7f49d818f9d0 2021-02-10T11:36:23.196 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:23.190 7f49dcdcf700 1 -- 172.21.2.102:0/3960083585 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49d8086300 con 0x7f49d818f9d0 2021-02-10T11:36:23.196 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:23.191 7f49cf7fe700 1 -- 172.21.2.102:0/3960083585 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f49c80052e0 con 0x7f49d818f9d0 2021-02-10T11:36:23.196 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:23.191 7f49cf7fe700 1 --2- 172.21.2.102:0/3960083585 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f49bc0186d0 0x7f49bc01abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:23.196 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:23.191 7f49cf7fe700 1 -- 172.21.2.102:0/3960083585 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(153..153 src has 1..153) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f49c8032600 con 0x7f49d818f9d0 2021-02-10T11:36:23.196 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:23.191 7f49d6312700 1 --2- 172.21.2.102:0/3960083585 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f49bc0186d0 0x7f49bc01abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:23.196 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:23.191 7f49d6312700 1 --2- 172.21.2.102:0/3960083585 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f49bc0186d0 0x7f49bc01abd0 crc :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:23.225 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:23.223 7f49dcdcf700 1 -- 172.21.2.102:0/3960083585 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49d8082a70 con 0x7f49d818f9d0 2021-02-10T11:36:23.227 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:23.225 7f49cf7fe700 1 -- 172.21.2.102:0/3960083585 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f49c80093a0 con 0x7f49d818f9d0 2021-02-10T11:36:23.354 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:23.352 7f49dcdcf700 1 -- 172.21.2.102:0/3960083585 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "non_existent_subvolume", "target": ["mgr", ""]}) v1 -- 0x7f49d818fdd0 con 0x7f49bc0186d0 2021-02-10T11:36:23.358 INFO:tasks.ceph.mgr.y.gibba002.stderr:2021-02-10 11:36:23.356 7fd94bc53700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:36:23.358 INFO:tasks.ceph.mgr.y.gibba002.stderr:2021-02-10 11:36:23.356 7fd94bc53700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:36:23.358 INFO:tasks.ceph.mgr.y.gibba002.stderr:2021-02-10 11:36:23.356 7fd94bc53700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:36:23.358 INFO:tasks.ceph.mgr.y.gibba002.stderr:2021-02-10 11:36:23.356 7fd94bc53700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:36:23.358 INFO:tasks.ceph.mgr.y.gibba002.stderr:2021-02-10 11:36:23.356 7fd94bc53700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:36:23.364 INFO:tasks.ceph.mgr.y.gibba002.stderr:2021-02-10 11:36:23.362 7fd94bc53700 -1 mgr.server reply reply (2) No such file or directory subvolume 'non_existent_subvolume' does not exist 2021-02-10T11:36:23.364 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:23.362 7f49cf7fe700 1 -- 172.21.2.102:0/3960083585 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: -2 subvolume 'non_existent_subvolume' does not exist) v1 ==== 57+0+0 (crc 0 0 0) 0x7f49c00014b0 con 0x7f49bc0186d0 2021-02-10T11:36:23.369 INFO:teuthology.orchestra.run.gibba002.stderr:Error ENOENT: subvolume 'non_existent_subvolume' does not exist 2021-02-10T11:36:23.370 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:23.369 7f49dcdcf700 1 -- 172.21.2.102:0/3960083585 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f49bc0186d0 msgr2=0x7f49bc01abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:23.371 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:23.369 7f49dcdcf700 1 --2- 172.21.2.102:0/3960083585 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f49bc0186d0 0x7f49bc01abd0 crc :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:23.371 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:23.369 7f49dcdcf700 1 -- 172.21.2.102:0/3960083585 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f49d818f9d0 msgr2=0x7f49d8084cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:23.371 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:23.369 7f49dcdcf700 1 --2- 172.21.2.102:0/3960083585 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f49d818f9d0 0x7f49d8084cb0 secure :-1 s=READY pgs=2815 cs=0 l=1 rev1=1 rx=0x7f49c80096a0 tx=0x7f49c800b9b0).stop 2021-02-10T11:36:23.371 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:23.369 7f49dcdcf700 1 -- 172.21.2.102:0/3960083585 shutdown_connections 2021-02-10T11:36:23.371 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:23.369 7f49dcdcf700 1 --2- 172.21.2.102:0/3960083585 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f49bc0186d0 0x7f49bc01abd0 unknown :-1 s=CLOSED pgs=1236 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:23.371 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:23.369 7f49dcdcf700 1 --2- 172.21.2.102:0/3960083585 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f49d80c27e0 0x7f49d80859f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:23.372 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:23.369 7f49dcdcf700 1 --2- 172.21.2.102:0/3960083585 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f49d80c23e0 0x7f49d81a44b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:23.372 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:23.369 7f49dcdcf700 1 --2- 172.21.2.102:0/3960083585 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f49d818f9d0 0x7f49d8084cb0 unknown :-1 s=CLOSED pgs=2815 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:23.372 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:23.369 7f49dcdcf700 1 -- 172.21.2.102:0/3960083585 >> 172.21.2.102:0/3960083585 conn(0x7f49d80b5dc0 msgr2=0x7f49d80c2d90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:23.372 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:23.369 7f49dcdcf700 1 -- 172.21.2.102:0/3960083585 shutdown_connections 2021-02-10T11:36:23.372 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:23.369 7f49dcdcf700 1 -- 172.21.2.102:0/3960083585 wait complete. 2021-02-10T11:36:23.379 DEBUG:teuthology.orchestra.run:got remote process result: 2 2021-02-10T11:36:23.379 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2021-02-10T11:36:23.379 DEBUG:teuthology.orchestra.run.gibba002:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:36:23.725 DEBUG:teuthology.orchestra.run.gibba013:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:36:24.070 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2021-02-10T11:36:24.071 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:36:24.071 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:36:24.071 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:36:24.407 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:36:24.407 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:36:24.408 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@gibba002.front.sepia.ceph.com... 2021-02-10T11:36:24.408 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-02-10T11:36:24.408 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:36:24.765 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba002.stderr:ceph-fuse[200903]: fuse finished with error 0 and tester_r 0 2021-02-10T11:36:24.767 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:36:24.767 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:36:24.767 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:36:25.101 INFO:teuthology.orchestra.run.gibba002.stdout:ext2/ext3 2021-02-10T11:36:25.101 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2021-02-10T11:36:25.102 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:36:25.102 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:36:25.438 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:36:25.438 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:36:25.438 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:36:25.775 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:36:25.775 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2021-02-10T11:36:25.775 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@gibba002.front.sepia.ceph.com... 2021-02-10T11:36:25.776 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-02-10T11:36:25.776 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:36:26.128 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba002.stderr:ceph-fuse[201572]: fuse finished with error 0 and tester_r 0 2021-02-10T11:36:26.130 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:36:26.130 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:36:26.130 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:36:26.466 INFO:teuthology.orchestra.run.gibba002.stdout:ext2/ext3 2021-02-10T11:36:26.467 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2021-02-10T11:36:26.467 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:36:26.467 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:36:26.802 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:36:26.802 DEBUG:teuthology.orchestra.run.gibba013:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:36:26.802 DEBUG:teuthology.orchestra.run.gibba013:> ' -- /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:36:27.138 INFO:teuthology.orchestra.run.gibba013.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.2’: No such file or directory 2021-02-10T11:36:27.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:36:27.138 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2021-02-10T11:36:27.138 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:36:27.139 DEBUG:teuthology.orchestra.run.gibba013:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:36:27.473 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:36:27.473 DEBUG:teuthology.orchestra.run.gibba013:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:36:27.473 DEBUG:teuthology.orchestra.run.gibba013:> ' -- /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:36:27.808 INFO:teuthology.orchestra.run.gibba013.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.3’: No such file or directory 2021-02-10T11:36:27.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:36:27.808 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.3 2021-02-10T11:36:27.809 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:36:27.809 DEBUG:teuthology.orchestra.run.gibba013:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:36:28.144 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config rm mon mon_allow_pool_delete 2021-02-10T11:36:28.580 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:28.560 7ff38032f700 1 Processor -- start 2021-02-10T11:36:28.580 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:28.560 7ff38032f700 1 -- start start 2021-02-10T11:36:28.580 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:28.560 7ff38032f700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff3780d1c60 0x7ff3780f2090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:28.580 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:28.560 7ff38032f700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3780f0b00 con 0x7ff3780d0330 2021-02-10T11:36:28.581 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:28.560 7ff38032f700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3781b0bd0 con 0x7ff3780d1860 2021-02-10T11:36:28.581 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:28.560 7ff38032f700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3780e5ec0 con 0x7ff3780d1c60 2021-02-10T11:36:28.581 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:28.560 7ff37d94a700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7ff3780d0330 0x7ff3780ef630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:58250/0 (socket says 172.21.2.102:58250) 2021-02-10T11:36:28.581 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:28.560 7ff37d94a700 1 -- 172.21.2.102:0/921233503 learned_addr learned my addr 172.21.2.102:0/921233503 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:36:28.581 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:28.561 7ff37d149700 1 -- 172.21.2.102:0/921233503 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4065248669 0 0) 0x7ff374001180 con 0x7ff3780d0330 2021-02-10T11:36:28.581 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:28.561 7ff37d149700 1 -- 172.21.2.102:0/921233503 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff364002fe0 con 0x7ff3780d0330 2021-02-10T11:36:28.581 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:28.561 7ff37e94c700 1 --2- 172.21.2.102:0/921233503 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff3780d1c60 0x7ff3780f2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:28.581 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:28.561 7ff37d149700 1 -- 172.21.2.102:0/921233503 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3601837026 0 0) 0x7ff3740010e0 con 0x7ff3780d0330 2021-02-10T11:36:28.581 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:28.561 7ff37d149700 1 -- 172.21.2.102:0/921233503 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff3780d1c60 msgr2=0x7ff3780f2090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:28.581 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:28.561 7ff37d149700 1 --2- 172.21.2.102:0/921233503 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff3780d1c60 0x7ff3780f2090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:28.581 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:28.561 7ff37d149700 1 -- 172.21.2.102:0/921233503 >> v1:172.21.2.113:6789/0 conn(0x7ff3780d1860 legacy=0x7ff3781b03e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:28.581 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:28.561 7ff37d149700 1 -- 172.21.2.102:0/921233503 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3780efe20 con 0x7ff3780d0330 2021-02-10T11:36:28.582 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:28.561 7ff37d149700 1 -- 172.21.2.102:0/921233503 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7ff374001600 con 0x7ff3780d0330 2021-02-10T11:36:28.582 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:28.561 7ff37d149700 1 -- 172.21.2.102:0/921233503 >> v1:172.21.2.102:6789/0 conn(0x7ff3780d0330 legacy=0x7ff3780ef630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:28.582 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:28.561 7ff37d149700 1 --2- 172.21.2.102:0/921233503 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff364003a60 0x7ff364005ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:28.582 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:28.561 7ff37d149700 1 --2- 172.21.2.102:0/921233503 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff364006790 0x7ff364008c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:28.582 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:28.561 7ff37d149700 1 --2- 172.21.2.102:0/921233503 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff364009500 0x7ff36400ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:28.582 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:28.561 7ff37d149700 1 -- 172.21.2.102:0/921233503 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff36400c2d0 con 0x7ff364003a60 2021-02-10T11:36:28.582 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:28.561 7ff37d149700 1 -- 172.21.2.102:0/921233503 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff36400c4f0 con 0x7ff364009500 2021-02-10T11:36:28.582 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:28.561 7ff37d149700 1 -- 172.21.2.102:0/921233503 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff36400c710 con 0x7ff364006790 2021-02-10T11:36:28.582 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:28.561 7ff37e14b700 1 --2- 172.21.2.102:0/921233503 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff364003a60 0x7ff364005ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:28.582 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:28.562 7ff37d94a700 1 --2- 172.21.2.102:0/921233503 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff364006790 0x7ff364008c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:28.582 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:28.562 7ff37e94c700 1 --2- 172.21.2.102:0/921233503 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff364009500 0x7ff36400ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:28.582 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:28.562 7ff37e14b700 1 -- 172.21.2.102:0/921233503 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff364006790 msgr2=0x7ff364008c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:28.582 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:28.562 7ff37e14b700 1 --2- 172.21.2.102:0/921233503 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff364006790 0x7ff364008c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:28.583 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:28.562 7ff37e14b700 1 -- 172.21.2.102:0/921233503 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff364009500 msgr2=0x7ff36400ba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:28.583 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:28.562 7ff37e14b700 1 --2- 172.21.2.102:0/921233503 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff364009500 0x7ff36400ba20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:28.583 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:28.562 7ff37e14b700 1 -- 172.21.2.102:0/921233503 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff36400c9b0 con 0x7ff364003a60 2021-02-10T11:36:28.583 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:28.562 7ff37e14b700 1 --2- 172.21.2.102:0/921233503 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff364003a60 0x7ff364005ef0 secure :-1 s=READY pgs=2819 cs=0 l=1 rev1=1 rx=0x7ff36c04ae40 tx=0x7ff36c04de20).ready entity=mon.0 client_cookie=85751cd0019b1890 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:28.583 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:28.562 7ff37d149700 1 -- 172.21.2.102:0/921233503 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff36c001400 con 0x7ff364003a60 2021-02-10T11:36:28.583 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:28.562 7ff37d149700 1 -- 172.21.2.102:0/921233503 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff36c001680 con 0x7ff364003a60 2021-02-10T11:36:28.583 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:28.562 7ff37d149700 1 -- 172.21.2.102:0/921233503 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff36c001ba0 con 0x7ff364003a60 2021-02-10T11:36:28.583 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:28.562 7ff38032f700 1 -- 172.21.2.102:0/921233503 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff364003a60 msgr2=0x7ff364005ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:28.583 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:28.562 7ff38032f700 1 --2- 172.21.2.102:0/921233503 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff364003a60 0x7ff364005ef0 secure :-1 s=READY pgs=2819 cs=0 l=1 rev1=1 rx=0x7ff36c04ae40 tx=0x7ff36c04de20).stop 2021-02-10T11:36:28.583 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:28.562 7ff37e94c700 1 -- 172.21.2.102:0/921233503 reap_dead start 2021-02-10T11:36:28.583 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:28.563 7ff38032f700 1 -- 172.21.2.102:0/921233503 shutdown_connections 2021-02-10T11:36:28.583 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:28.563 7ff38032f700 1 -- 172.21.2.102:0/921233503 >> 172.21.2.102:0/921233503 conn(0x7ff3780ce440 msgr2=0x7ff3780d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:28.584 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:28.563 7ff38032f700 1 -- 172.21.2.102:0/921233503 shutdown_connections 2021-02-10T11:36:28.584 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:28.563 7ff38032f700 1 -- 172.21.2.102:0/921233503 wait complete. 2021-02-10T11:36:28.584 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:28.578 7ff38032f700 1 Processor -- start 2021-02-10T11:36:28.584 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:28.578 7ff38032f700 1 -- start start 2021-02-10T11:36:28.585 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:28.579 7ff38032f700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff37809fb20 0x7ff37808dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:28.586 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:28.579 7ff38032f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff3780908e0 0x7ff378090ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:28.586 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:28.579 7ff38032f700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff378091300 0x7ff37808c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:28.586 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:28.579 7ff38032f700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff37808d6c0 con 0x7ff3780908e0 2021-02-10T11:36:28.586 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:28.579 7ff38032f700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff37809ff60 con 0x7ff378091300 2021-02-10T11:36:28.586 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:28.579 7ff38032f700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff37809c240 con 0x7ff37809fb20 2021-02-10T11:36:28.586 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:28.579 7ff37d94a700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff3780908e0 0x7ff378090ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:28.586 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:28.579 7ff37d94a700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff3780908e0 0x7ff378090ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:37808/0 (socket says 172.21.2.102:37808) 2021-02-10T11:36:28.586 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:28.579 7ff37d94a700 1 -- 172.21.2.102:0/2829400280 learned_addr learned my addr 172.21.2.102:0/2829400280 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:36:28.586 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:28.579 7ff37e14b700 1 --2- 172.21.2.102:0/2829400280 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff37809fb20 0x7ff37808dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:28.586 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:28.579 7ff37e94c700 1 --2- 172.21.2.102:0/2829400280 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff378091300 0x7ff37808c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:28.586 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:28.579 7ff37d94a700 1 -- 172.21.2.102:0/2829400280 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff37809fb20 msgr2=0x7ff37808dcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:28.587 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:28.579 7ff37d94a700 1 --2- 172.21.2.102:0/2829400280 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff37809fb20 0x7ff37808dcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:28.587 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:28.579 7ff37d94a700 1 -- 172.21.2.102:0/2829400280 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff378091300 msgr2=0x7ff37808c720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:28.587 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:28.579 7ff37d94a700 1 --2- 172.21.2.102:0/2829400280 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff378091300 0x7ff37808c720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:28.587 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:28.579 7ff37d94a700 1 -- 172.21.2.102:0/2829400280 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff37808cd40 con 0x7ff3780908e0 2021-02-10T11:36:28.587 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:28.579 7ff37d94a700 1 --2- 172.21.2.102:0/2829400280 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff3780908e0 0x7ff378090ce0 secure :-1 s=READY pgs=2820 cs=0 l=1 rev1=1 rx=0x7ff37400a5d0 tx=0x7ff37400a970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:28.587 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:28.579 7ff36affd700 1 -- 172.21.2.102:0/2829400280 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff374001c20 con 0x7ff3780908e0 2021-02-10T11:36:28.587 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:28.579 7ff36affd700 1 -- 172.21.2.102:0/2829400280 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff374001930 con 0x7ff3780908e0 2021-02-10T11:36:28.587 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:28.579 7ff36affd700 1 -- 172.21.2.102:0/2829400280 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff3740045b0 con 0x7ff3780908e0 2021-02-10T11:36:28.587 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:28.579 7ff38032f700 1 -- 172.21.2.102:0/2829400280 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff37808cd40 con 0x7ff3780908e0 2021-02-10T11:36:28.587 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:28.579 7ff38032f700 1 -- 172.21.2.102:0/2829400280 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff37808cd40 con 0x7ff3780908e0 2021-02-10T11:36:28.587 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:28.580 7ff36affd700 1 -- 172.21.2.102:0/2829400280 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ff3740043a0 con 0x7ff3780908e0 2021-02-10T11:36:28.587 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:28.580 7ff36affd700 1 --2- 172.21.2.102:0/2829400280 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff3600186c0 0x7ff36001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:28.588 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:28.580 7ff36affd700 1 -- 172.21.2.102:0/2829400280 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(153..153 src has 1..153) v4 ==== 5573+0+0 (secure 0 0 0) 0x7ff3740323a0 con 0x7ff3780908e0 2021-02-10T11:36:28.588 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:28.580 7ff37e14b700 1 --2- 172.21.2.102:0/2829400280 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff3600186c0 0x7ff36001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:28.588 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:28.580 7ff37e14b700 1 --2- 172.21.2.102:0/2829400280 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff3600186c0 0x7ff36001abc0 crc :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:28.612 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:28.611 7ff38032f700 1 -- 172.21.2.102:0/2829400280 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3780ebba0 con 0x7ff3780908e0 2021-02-10T11:36:28.615 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:28.613 7ff36affd700 1 -- 172.21.2.102:0/2829400280 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ff374030cd0 con 0x7ff3780908e0 2021-02-10T11:36:28.767 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:28.765 7ff38032f700 1 -- 172.21.2.102:0/2829400280 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7ff37809ff20 con 0x7ff3780908e0 2021-02-10T11:36:28.770 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:28.768 7ff36affd700 1 -- 172.21.2.102:0/2829400280 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v39) v1 ==== 104+0+0 (secure 0 0 0) 0x7ff3740043a0 con 0x7ff3780908e0 2021-02-10T11:36:28.776 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:28.774 7ff38032f700 1 -- 172.21.2.102:0/2829400280 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff3600186c0 msgr2=0x7ff36001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:28.776 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:28.774 7ff38032f700 1 --2- 172.21.2.102:0/2829400280 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff3600186c0 0x7ff36001abc0 crc :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:28.776 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:28.774 7ff38032f700 1 -- 172.21.2.102:0/2829400280 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff3780908e0 msgr2=0x7ff378090ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:28.776 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:28.774 7ff38032f700 1 --2- 172.21.2.102:0/2829400280 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff3780908e0 0x7ff378090ce0 secure :-1 s=READY pgs=2820 cs=0 l=1 rev1=1 rx=0x7ff37400a5d0 tx=0x7ff37400a970).stop 2021-02-10T11:36:28.776 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:28.774 7ff38032f700 1 -- 172.21.2.102:0/2829400280 shutdown_connections 2021-02-10T11:36:28.776 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:28.774 7ff38032f700 1 --2- 172.21.2.102:0/2829400280 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff3600186c0 0x7ff36001abc0 unknown :-1 s=CLOSED pgs=1237 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:28.776 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:28.774 7ff38032f700 1 --2- 172.21.2.102:0/2829400280 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff378091300 0x7ff37808c720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:28.777 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:28.774 7ff38032f700 1 --2- 172.21.2.102:0/2829400280 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff3780908e0 0x7ff378090ce0 unknown :-1 s=CLOSED pgs=2820 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:28.777 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:28.774 7ff38032f700 1 --2- 172.21.2.102:0/2829400280 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff37809fb20 0x7ff37808dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:28.777 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:28.774 7ff38032f700 1 -- 172.21.2.102:0/2829400280 >> 172.21.2.102:0/2829400280 conn(0x7ff378094e60 msgr2=0x7ff37808fb80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:28.777 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:28.774 7ff38032f700 1 -- 172.21.2.102:0/2829400280 shutdown_connections 2021-02-10T11:36:28.777 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:28.774 7ff38032f700 1 -- 172.21.2.102:0/2829400280 wait complete. 2021-02-10T11:36:28.784 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestVolumes.test_nonexistent_subvolume_rm' 2021-02-10T11:36:29.213 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:29.195 7f8cc26bf700 1 Processor -- start 2021-02-10T11:36:29.214 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:29.195 7f8cc26bf700 1 -- start start 2021-02-10T11:36:29.214 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:29.196 7f8cc26bf700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8cbc0d1870 0x7f8cbc1b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:29.214 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:29.196 7f8cc26bf700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f8cbc0f0650 0x7f8cbc0cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:29.214 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:29.196 7f8cc26bf700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cbc0f0b10 con 0x7f8cbc1b0a10 2021-02-10T11:36:29.214 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:29.196 7f8cc26bf700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8cbc0fe230 con 0x7f8cbc0f0650 2021-02-10T11:36:29.214 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:29.196 7f8cc26bf700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cbc0e5ed0 con 0x7f8cbc0d1870 2021-02-10T11:36:29.214 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:29.196 7f8cc0cdc700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f8cbc0f0650 0x7f8cbc0cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:29.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:29.196 7f8cc0cdc700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f8cbc0f0650 0x7f8cbc0cfb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:37812/0 (socket says 172.21.2.102:37812) 2021-02-10T11:36:29.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:29.196 7f8cc0cdc700 1 -- 172.21.2.102:0/3972309906 learned_addr learned my addr 172.21.2.102:0/3972309906 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:36:29.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:29.196 7f8cc0cdc700 1 -- 172.21.2.102:0/3972309906 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8cbc0d1870 msgr2=0x7f8cbc1b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:29.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:29.196 7f8cc0cdc700 1 --2- 172.21.2.102:0/3972309906 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8cbc0d1870 0x7f8cbc1b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:29.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:29.196 7f8cc0cdc700 1 -- 172.21.2.102:0/3972309906 >> v1:172.21.2.113:6789/0 conn(0x7f8cbc1b0a10 legacy=0x7f8cbc0cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:29.216 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:29.196 7f8cc0cdc700 1 -- 172.21.2.102:0/3972309906 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cbc0f20a0 con 0x7f8cbc0f0650 2021-02-10T11:36:29.216 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:29.196 7f8cc0cdc700 1 --2- 172.21.2.102:0/3972309906 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8cbc0f0650 0x7f8cbc0cfb20 secure :-1 s=READY pgs=2821 cs=0 l=1 rev1=1 rx=0x7f8cb4006f00 tx=0x7f8cb4009f10).ready entity=mon.0 client_cookie=28386b54ff3c3220 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:29.216 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:29.196 7f8cbaffd700 1 -- 172.21.2.102:0/3972309906 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8cb400b3b0 con 0x7f8cbc0f0650 2021-02-10T11:36:29.216 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:29.196 7f8cbaffd700 1 -- 172.21.2.102:0/3972309906 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cb400b730 con 0x7f8cbc0f0650 2021-02-10T11:36:29.216 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:29.196 7f8cbaffd700 1 -- 172.21.2.102:0/3972309906 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8cb400bc50 con 0x7f8cbc0f0650 2021-02-10T11:36:29.216 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:29.196 7f8cc26bf700 1 -- 172.21.2.102:0/3972309906 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8cbc0f0650 msgr2=0x7f8cbc0cfb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:29.216 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:29.196 7f8cc26bf700 1 --2- 172.21.2.102:0/3972309906 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8cbc0f0650 0x7f8cbc0cfb20 secure :-1 s=READY pgs=2821 cs=0 l=1 rev1=1 rx=0x7f8cb4006f00 tx=0x7f8cb4009f10).stop 2021-02-10T11:36:29.217 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:29.197 7f8cc26bf700 1 -- 172.21.2.102:0/3972309906 shutdown_connections 2021-02-10T11:36:29.217 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:29.197 7f8cc26bf700 1 --2- 172.21.2.102:0/3972309906 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8cbc0f0650 0x7f8cbc0cfb20 unknown :-1 s=CLOSED pgs=2821 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:29.217 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:29.197 7f8cc26bf700 1 --2- 172.21.2.102:0/3972309906 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8cbc0d1870 0x7f8cbc1b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:29.217 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:29.197 7f8cc26bf700 1 -- 172.21.2.102:0/3972309906 >> 172.21.2.102:0/3972309906 conn(0x7f8cbc0ce450 msgr2=0x7f8cbc0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:29.217 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:29.197 7f8cc26bf700 1 -- 172.21.2.102:0/3972309906 shutdown_connections 2021-02-10T11:36:29.217 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:29.197 7f8cc26bf700 1 -- 172.21.2.102:0/3972309906 wait complete. 2021-02-10T11:36:29.217 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:29.211 7f8cc26bf700 1 Processor -- start 2021-02-10T11:36:29.217 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:29.211 7f8cc26bf700 1 -- start start 2021-02-10T11:36:29.219 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:29.212 7f8cc26bf700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8cbc19af00 0x7f8cbc19b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:29.219 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:29.212 7f8cc26bf700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8cbc19ba80 0x7f8cbc19be80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:29.219 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:29.212 7f8cc26bf700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8cbc19c4a0 0x7f8cbc08a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:29.219 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:29.212 7f8cc26bf700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cbc08fb50 con 0x7f8cbc19c4a0 2021-02-10T11:36:29.219 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:29.212 7f8cc26bf700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cbc08afb0 con 0x7f8cbc19ba80 2021-02-10T11:36:29.219 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:29.212 7f8cc26bf700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cbc08b1a0 con 0x7f8cbc19af00 2021-02-10T11:36:29.219 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:29.212 7f8cc0cdc700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8cbc19c4a0 0x7f8cbc08a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:29.219 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:29.212 7f8cc0cdc700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8cbc19c4a0 0x7f8cbc08a990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:37820/0 (socket says 172.21.2.102:37820) 2021-02-10T11:36:29.220 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:29.212 7f8cc0cdc700 1 -- 172.21.2.102:0/150310134 learned_addr learned my addr 172.21.2.102:0/150310134 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:36:29.220 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:29.212 7f8cc0cdc700 1 -- 172.21.2.102:0/150310134 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8cbc19af00 msgr2=0x7f8cbc19b300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:29.220 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:29.212 7f8cc0cdc700 1 --2- 172.21.2.102:0/150310134 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8cbc19af00 0x7f8cbc19b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:29.220 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:29.212 7f8cc0cdc700 1 -- 172.21.2.102:0/150310134 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8cbc19ba80 msgr2=0x7f8cbc19be80 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:36:29.220 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:29.212 7f8cc0cdc700 1 --2- 172.21.2.102:0/150310134 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8cbc19ba80 0x7f8cbc19be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:29.220 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:29.212 7f8cc0cdc700 1 -- 172.21.2.102:0/150310134 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cbc08b3c0 con 0x7f8cbc19c4a0 2021-02-10T11:36:29.220 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:29.212 7f8cc0cdc700 1 --2- 172.21.2.102:0/150310134 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8cbc19c4a0 0x7f8cbc08a990 secure :-1 s=READY pgs=2822 cs=0 l=1 rev1=1 rx=0x7f8cb400b8e0 tx=0x7f8cb400bc80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:29.220 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:29.213 7f8cb8ff9700 1 -- 172.21.2.102:0/150310134 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8cb4003030 con 0x7f8cbc19c4a0 2021-02-10T11:36:29.220 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:29.213 7f8cb8ff9700 1 -- 172.21.2.102:0/150310134 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cb400bdc0 con 0x7f8cbc19c4a0 2021-02-10T11:36:29.220 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:29.213 7f8cc26bf700 1 -- 172.21.2.102:0/150310134 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cbc08b3c0 con 0x7f8cbc19c4a0 2021-02-10T11:36:29.220 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:29.213 7f8cc26bf700 1 -- 172.21.2.102:0/150310134 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cbc08b950 con 0x7f8cbc19c4a0 2021-02-10T11:36:29.220 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:29.213 7f8cb8ff9700 1 -- 172.21.2.102:0/150310134 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8cb400c570 con 0x7f8cbc19c4a0 2021-02-10T11:36:29.221 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:29.213 7f8cb8ff9700 1 -- 172.21.2.102:0/150310134 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f8cb4001270 con 0x7f8cbc19c4a0 2021-02-10T11:36:29.221 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:29.213 7f8cb8ff9700 1 --2- 172.21.2.102:0/150310134 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8c9c0186c0 0x7f8c9c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:29.221 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:29.213 7f8cb8ff9700 1 -- 172.21.2.102:0/150310134 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(153..153 src has 1..153) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f8cb40328b0 con 0x7f8cbc19c4a0 2021-02-10T11:36:29.221 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:29.214 7f8cbbfff700 1 --2- 172.21.2.102:0/150310134 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8c9c0186c0 0x7f8c9c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:29.221 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:29.214 7f8cbbfff700 1 --2- 172.21.2.102:0/150310134 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8c9c0186c0 0x7f8c9c01abc0 crc :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:29.247 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:29.245 7f8cc26bf700 1 -- 172.21.2.102:0/150310134 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8cbc0ee030 con 0x7f8cbc19c4a0 2021-02-10T11:36:29.250 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:29.248 7f8cb8ff9700 1 -- 172.21.2.102:0/150310134 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f8cb4024380 con 0x7f8cbc19c4a0 2021-02-10T11:36:29.382 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:29.379 7f8cc26bf700 1 -- 172.21.2.102:0/150310134 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestVolumes.test_nonexistent_subvolume_rm"]} v 0) v1 -- 0x7f8cbc0ce450 con 0x7f8cbc19c4a0 2021-02-10T11:36:30.166 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:30.164 7f8cb8ff9700 1 -- 172.21.2.102:0/150310134 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestVolumes.test_nonexistent_subvolume_rm"]}]=0 v767) v1 ==== 146+0+0 (secure 0 0 0) 0x7f8cb4024380 con 0x7f8cbc19c4a0 2021-02-10T11:36:30.203 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:30.201 7f8cc26bf700 1 -- 172.21.2.102:0/150310134 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8c9c0186c0 msgr2=0x7f8c9c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:30.203 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:30.201 7f8cc26bf700 1 --2- 172.21.2.102:0/150310134 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8c9c0186c0 0x7f8c9c01abc0 crc :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:30.204 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:30.201 7f8cc26bf700 1 -- 172.21.2.102:0/150310134 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8cbc19c4a0 msgr2=0x7f8cbc08a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:30.204 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:30.201 7f8cc26bf700 1 --2- 172.21.2.102:0/150310134 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8cbc19c4a0 0x7f8cbc08a990 secure :-1 s=READY pgs=2822 cs=0 l=1 rev1=1 rx=0x7f8cb400b8e0 tx=0x7f8cb400bc80).stop 2021-02-10T11:36:30.204 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:30.201 7f8cc26bf700 1 -- 172.21.2.102:0/150310134 shutdown_connections 2021-02-10T11:36:30.204 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:30.201 7f8cc26bf700 1 --2- 172.21.2.102:0/150310134 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8cbc19c4a0 0x7f8cbc08a990 unknown :-1 s=CLOSED pgs=2822 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:30.204 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:30.201 7f8cc26bf700 1 --2- 172.21.2.102:0/150310134 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8c9c0186c0 0x7f8c9c01abc0 unknown :-1 s=CLOSED pgs=1238 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:30.204 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:30.201 7f8cc26bf700 1 --2- 172.21.2.102:0/150310134 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8cbc19ba80 0x7f8cbc19be80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:30.204 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:30.201 7f8cc26bf700 1 --2- 172.21.2.102:0/150310134 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8cbc19af00 0x7f8cbc19b300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:30.204 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:30.201 7f8cc26bf700 1 -- 172.21.2.102:0/150310134 >> 172.21.2.102:0/150310134 conn(0x7f8cbc19f8c0 msgr2=0x7f8cbc1a5fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:30.204 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:30.201 7f8cc26bf700 1 -- 172.21.2.102:0/150310134 shutdown_connections 2021-02-10T11:36:30.204 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:30.201 7f8cc26bf700 1 -- 172.21.2.102:0/150310134 wait complete. 2021-02-10T11:36:30.212 INFO:tasks.cephfs_test_runner:test_nonexistent_subvolume_rm (tasks.cephfs.test_volumes.TestVolumes) ... ok 2021-02-10T11:36:30.212 INFO:tasks.cephfs_test_runner:Starting test: test_nonexistent_subvolume_snapshot_rm (tasks.cephfs.test_volumes.TestVolumes) 2021-02-10T11:36:30.212 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestVolumes.test_nonexistent_subvolume_snapshot_rm' 2021-02-10T11:36:30.641 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:30.622 7f4ebd363700 1 Processor -- start 2021-02-10T11:36:30.641 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:30.622 7f4ebd363700 1 -- start start 2021-02-10T11:36:30.641 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:30.623 7f4ebd363700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f4eb81b0470 0x7f4eb81b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:30.641 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:30.623 7f4ebd363700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4eb80ef610 0x7f4eb80efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:30.641 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:30.623 7f4ebd363700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f4eb80f0030 0x7f4eb80f0430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:30.641 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:30.623 7f4ebd363700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4eb80e5ea0 con 0x7f4eb81b0470 2021-02-10T11:36:30.641 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:30.623 7f4ebd363700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4eb80f0b50 con 0x7f4eb80f0030 2021-02-10T11:36:30.641 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:30.623 7f4ebd363700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4eb80e52e0 con 0x7f4eb80ef610 2021-02-10T11:36:30.641 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:30.623 7f4eb707d700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f4eb81b0470 0x7f4eb81b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:30.641 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:30.623 7f4eb707d700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f4eb81b0470 0x7f4eb81b0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:37824/0 (socket says 172.21.2.102:37824) 2021-02-10T11:36:30.642 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:30.623 7f4eb707d700 1 -- 172.21.2.102:0/2822386683 learned_addr learned my addr 172.21.2.102:0/2822386683 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:36:30.642 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:30.623 7f4eb707d700 1 -- 172.21.2.102:0/2822386683 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4eb80ef610 msgr2=0x7f4eb80efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:30.642 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:30.623 7f4eb707d700 1 --2- 172.21.2.102:0/2822386683 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4eb80ef610 0x7f4eb80efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:30.642 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:30.623 7f4eb707d700 1 -- 172.21.2.102:0/2822386683 >> v2:172.21.2.113:3300/0 conn(0x7f4eb80f0030 msgr2=0x7f4eb80f0430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:30.642 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:30.623 7f4eb707d700 1 --2- 172.21.2.102:0/2822386683 >> v2:172.21.2.113:3300/0 conn(0x7f4eb80f0030 0x7f4eb80f0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:30.642 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:30.623 7f4eb707d700 1 -- 172.21.2.102:0/2822386683 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4eb80f10d0 con 0x7f4eb81b0470 2021-02-10T11:36:30.642 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:30.623 7f4eb707d700 1 --2- 172.21.2.102:0/2822386683 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4eb81b0470 0x7f4eb81b0870 secure :-1 s=READY pgs=2823 cs=0 l=1 rev1=1 rx=0x7f4ea0006700 tx=0x7f4ea0009b00).ready entity=mon.0 client_cookie=e430ec7bff2b8c0c server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:30.642 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:30.624 7f4eb607b700 1 -- 172.21.2.102:0/2822386683 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4ea000c0c0 con 0x7f4eb81b0470 2021-02-10T11:36:30.642 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:30.624 7f4eb607b700 1 -- 172.21.2.102:0/2822386683 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ea000c380 con 0x7f4eb81b0470 2021-02-10T11:36:30.642 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:30.624 7f4eb607b700 1 -- 172.21.2.102:0/2822386683 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4ea000c860 con 0x7f4eb81b0470 2021-02-10T11:36:30.642 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:30.624 7f4ebd363700 1 -- 172.21.2.102:0/2822386683 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4eb81b0470 msgr2=0x7f4eb81b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:30.643 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:30.624 7f4ebd363700 1 --2- 172.21.2.102:0/2822386683 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4eb81b0470 0x7f4eb81b0870 secure :-1 s=READY pgs=2823 cs=0 l=1 rev1=1 rx=0x7f4ea0006700 tx=0x7f4ea0009b00).stop 2021-02-10T11:36:30.643 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:30.624 7f4ebd363700 1 -- 172.21.2.102:0/2822386683 shutdown_connections 2021-02-10T11:36:30.643 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:30.624 7f4ebd363700 1 --2- 172.21.2.102:0/2822386683 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4eb80ef610 0x7f4eb80efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:30.643 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:30.624 7f4ebd363700 1 --2- 172.21.2.102:0/2822386683 >> v2:172.21.2.113:3300/0 conn(0x7f4eb80f0030 0x7f4eb80f0430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:30.643 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:30.624 7f4ebd363700 1 --2- 172.21.2.102:0/2822386683 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4eb81b0470 0x7f4eb81b0870 unknown :-1 s=CLOSED pgs=2823 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:30.643 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:30.624 7f4ebd363700 1 -- 172.21.2.102:0/2822386683 >> 172.21.2.102:0/2822386683 conn(0x7f4eb80ce3e0 msgr2=0x7f4eb80e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:30.643 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:30.624 7f4ebd363700 1 -- 172.21.2.102:0/2822386683 shutdown_connections 2021-02-10T11:36:30.643 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:30.624 7f4ebd363700 1 -- 172.21.2.102:0/2822386683 wait complete. 2021-02-10T11:36:30.645 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:30.639 7f4ebd363700 1 Processor -- start 2021-02-10T11:36:30.645 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:30.639 7f4ebd363700 1 -- start start 2021-02-10T11:36:30.645 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:30.639 7f4ebd363700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4eb81b6e50 0x7f4eb8198a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:30.645 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:30.639 7f4ebd363700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4eb81990b0 0x7f4eb80ad260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:30.645 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:30.639 7f4ebd363700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4eb80ad880 0x7f4eb80adc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:30.646 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:30.639 7f4ebd363700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4eb8199580 con 0x7f4eb80ad880 2021-02-10T11:36:30.646 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:30.639 7f4ebd363700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4eb81b96f0 con 0x7f4eb81990b0 2021-02-10T11:36:30.646 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:30.639 7f4ebd363700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4eb81b7250 con 0x7f4eb81b6e50 2021-02-10T11:36:30.646 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:30.640 7f4eb787e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4eb80ad880 0x7f4eb80adc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:30.646 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:30.640 7f4eb787e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4eb80ad880 0x7f4eb80adc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:37830/0 (socket says 172.21.2.102:37830) 2021-02-10T11:36:30.646 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:30.640 7f4eb787e700 1 -- 172.21.2.102:0/2982235462 learned_addr learned my addr 172.21.2.102:0/2982235462 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:36:30.646 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:30.640 7f4eb787e700 1 -- 172.21.2.102:0/2982235462 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4eb81b6e50 msgr2=0x7f4eb8198a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:30.646 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:30.640 7f4eb787e700 1 --2- 172.21.2.102:0/2982235462 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4eb81b6e50 0x7f4eb8198a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:30.646 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:30.640 7f4eb787e700 1 -- 172.21.2.102:0/2982235462 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4eb81990b0 msgr2=0x7f4eb80ad260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:30.646 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:30.640 7f4eb787e700 1 --2- 172.21.2.102:0/2982235462 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4eb81990b0 0x7f4eb80ad260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:30.646 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:30.640 7f4eb787e700 1 -- 172.21.2.102:0/2982235462 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4eb8199d70 con 0x7f4eb80ad880 2021-02-10T11:36:30.647 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:30.640 7f4eb787e700 1 --2- 172.21.2.102:0/2982235462 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4eb80ad880 0x7f4eb80adc80 secure :-1 s=READY pgs=2824 cs=0 l=1 rev1=1 rx=0x7f4eb0009980 tx=0x7f4eb000bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:30.647 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:30.640 7f4eaffff700 1 -- 172.21.2.102:0/2982235462 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4eb0005af0 con 0x7f4eb80ad880 2021-02-10T11:36:30.647 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:30.641 7f4eaffff700 1 -- 172.21.2.102:0/2982235462 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4eb0005df0 con 0x7f4eb80ad880 2021-02-10T11:36:30.647 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:30.641 7f4eaffff700 1 -- 172.21.2.102:0/2982235462 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4eb00062b0 con 0x7f4eb80ad880 2021-02-10T11:36:30.647 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:30.641 7f4ebd363700 1 -- 172.21.2.102:0/2982235462 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4eb8199d70 con 0x7f4eb80ad880 2021-02-10T11:36:30.647 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:30.641 7f4ebd363700 1 -- 172.21.2.102:0/2982235462 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4eb819a300 con 0x7f4eb80ad880 2021-02-10T11:36:30.647 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:30.641 7f4eaffff700 1 -- 172.21.2.102:0/2982235462 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4eb0005af0 con 0x7f4eb80ad880 2021-02-10T11:36:30.647 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:30.641 7f4eaffff700 1 --2- 172.21.2.102:0/2982235462 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4e9c0186c0 0x7f4e9c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:30.647 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:30.641 7f4eaffff700 1 -- 172.21.2.102:0/2982235462 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(153..153 src has 1..153) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f4eb00327c0 con 0x7f4eb80ad880 2021-02-10T11:36:30.647 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:30.641 7f4eb707d700 1 --2- 172.21.2.102:0/2982235462 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4e9c0186c0 0x7f4e9c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:30.648 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:30.642 7f4eb707d700 1 --2- 172.21.2.102:0/2982235462 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4e9c0186c0 0x7f4e9c01abc0 crc :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:30.675 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:30.673 7f4ebd363700 1 -- 172.21.2.102:0/2982235462 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4eb8105ac0 con 0x7f4eb80ad880 2021-02-10T11:36:30.677 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:30.676 7f4eaffff700 1 -- 172.21.2.102:0/2982235462 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f4eb0029ff0 con 0x7f4eb80ad880 2021-02-10T11:36:30.810 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:30.808 7f4ebd363700 1 -- 172.21.2.102:0/2982235462 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestVolumes.test_nonexistent_subvolume_snapshot_rm"]} v 0) v1 -- 0x7f4eb80f0b50 con 0x7f4eb80ad880 2021-02-10T11:36:31.168 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:31.166 7f4eaffff700 1 -- 172.21.2.102:0/2982235462 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestVolumes.test_nonexistent_subvolume_snapshot_rm"]}]=0 v768) v1 ==== 158+0+0 (secure 0 0 0) 0x7f4eb0006f20 con 0x7f4eb80ad880 2021-02-10T11:36:31.178 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:31.176 7f4ebd363700 1 -- 172.21.2.102:0/2982235462 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4e9c0186c0 msgr2=0x7f4e9c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:31.178 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:31.176 7f4ebd363700 1 --2- 172.21.2.102:0/2982235462 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4e9c0186c0 0x7f4e9c01abc0 crc :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:31.178 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:31.176 7f4ebd363700 1 -- 172.21.2.102:0/2982235462 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4eb80ad880 msgr2=0x7f4eb80adc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:31.178 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:31.176 7f4ebd363700 1 --2- 172.21.2.102:0/2982235462 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4eb80ad880 0x7f4eb80adc80 secure :-1 s=READY pgs=2824 cs=0 l=1 rev1=1 rx=0x7f4eb0009980 tx=0x7f4eb000bcd0).stop 2021-02-10T11:36:31.178 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:31.176 7f4ebd363700 1 -- 172.21.2.102:0/2982235462 shutdown_connections 2021-02-10T11:36:31.178 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:31.176 7f4ebd363700 1 --2- 172.21.2.102:0/2982235462 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4eb80ad880 0x7f4eb80adc80 unknown :-1 s=CLOSED pgs=2824 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:31.178 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:31.176 7f4ebd363700 1 --2- 172.21.2.102:0/2982235462 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4e9c0186c0 0x7f4e9c01abc0 unknown :-1 s=CLOSED pgs=1239 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:31.179 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:31.176 7f4ebd363700 1 --2- 172.21.2.102:0/2982235462 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4eb81990b0 0x7f4eb80ad260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:31.179 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:31.176 7f4ebd363700 1 --2- 172.21.2.102:0/2982235462 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4eb81b6e50 0x7f4eb8198a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:31.179 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:31.176 7f4ebd363700 1 -- 172.21.2.102:0/2982235462 >> 172.21.2.102:0/2982235462 conn(0x7f4eb819f7c0 msgr2=0x7f4eb818c230 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:31.179 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:31.176 7f4ebd363700 1 -- 172.21.2.102:0/2982235462 shutdown_connections 2021-02-10T11:36:31.179 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:31.176 7f4ebd363700 1 -- 172.21.2.102:0/2982235462 wait complete. 2021-02-10T11:36:31.187 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config set mon mon_allow_pool_delete True 2021-02-10T11:36:31.620 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:31.601 7f01d9019700 1 Processor -- start 2021-02-10T11:36:31.621 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:31.601 7f01d9019700 1 -- start start 2021-02-10T11:36:31.621 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:31.602 7f01d9019700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f01d40d1870 0x7f01d41b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:31.621 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:31.602 7f01d9019700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f01d40cff70 0x7f01d4109130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:31.621 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:31.602 7f01d9019700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01d40fe1c0 con 0x7f01d41b0ae0 2021-02-10T11:36:31.621 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:31.602 7f01d9019700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01d4109ad0 con 0x7f01d40d1870 2021-02-10T11:36:31.621 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:31.602 7f01d9019700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01d4109750 con 0x7f01d40cff70 2021-02-10T11:36:31.621 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:31.602 7f01d2d9d700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f01d40d1870 0x7f01d41b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:31.621 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:31.602 7f01d2d9d700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f01d40d1870 0x7f01d41b04c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:37836/0 (socket says 172.21.2.102:37836) 2021-02-10T11:36:31.622 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:31.602 7f01d2d9d700 1 -- 172.21.2.102:0/54257878 learned_addr learned my addr 172.21.2.102:0/54257878 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:36:31.622 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:31.602 7f01d2d9d700 1 -- 172.21.2.102:0/54257878 >> v2:172.21.2.113:3300/0 conn(0x7f01d40cff70 msgr2=0x7f01d4109130 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:36:31.622 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:31.602 7f01d2d9d700 1 --2- 172.21.2.102:0/54257878 >> v2:172.21.2.113:3300/0 conn(0x7f01d40cff70 0x7f01d4109130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:31.622 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:31.602 7f01d2d9d700 1 -- 172.21.2.102:0/54257878 >> v1:172.21.2.113:6789/0 conn(0x7f01d41b0ae0 legacy=0x7f01d4108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:31.622 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:31.603 7f01d2d9d700 1 -- 172.21.2.102:0/54257878 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01d4088070 con 0x7f01d40d1870 2021-02-10T11:36:31.622 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:31.603 7f01d2d9d700 1 --2- 172.21.2.102:0/54257878 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f01d40d1870 0x7f01d41b04c0 secure :-1 s=READY pgs=2825 cs=0 l=1 rev1=1 rx=0x7f01c4006700 tx=0x7f01c4009b00).ready entity=mon.0 client_cookie=a3195dc6a054635e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:31.622 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:31.603 7f01d1d9b700 1 -- 172.21.2.102:0/54257878 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f01c400c0c0 con 0x7f01d40d1870 2021-02-10T11:36:31.622 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:31.603 7f01d1d9b700 1 -- 172.21.2.102:0/54257878 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01c400c340 con 0x7f01d40d1870 2021-02-10T11:36:31.622 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:31.603 7f01d1d9b700 1 -- 172.21.2.102:0/54257878 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f01c400c820 con 0x7f01d40d1870 2021-02-10T11:36:31.622 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:31.603 7f01d9019700 1 -- 172.21.2.102:0/54257878 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f01d40d1870 msgr2=0x7f01d41b04c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:31.623 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:31.603 7f01d9019700 1 --2- 172.21.2.102:0/54257878 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f01d40d1870 0x7f01d41b04c0 secure :-1 s=READY pgs=2825 cs=0 l=1 rev1=1 rx=0x7f01c4006700 tx=0x7f01c4009b00).stop 2021-02-10T11:36:31.623 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:31.604 7f01d9019700 1 -- 172.21.2.102:0/54257878 shutdown_connections 2021-02-10T11:36:31.623 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:31.604 7f01d9019700 1 --2- 172.21.2.102:0/54257878 >> v2:172.21.2.113:3300/0 conn(0x7f01d40cff70 0x7f01d4109130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:31.623 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:31.604 7f01d9019700 1 --2- 172.21.2.102:0/54257878 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f01d40d1870 0x7f01d41b04c0 unknown :-1 s=CLOSED pgs=2825 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:31.623 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:31.604 7f01d9019700 1 -- 172.21.2.102:0/54257878 >> 172.21.2.102:0/54257878 conn(0x7f01d40ce420 msgr2=0x7f01d40d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:31.623 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:31.604 7f01d9019700 1 -- 172.21.2.102:0/54257878 shutdown_connections 2021-02-10T11:36:31.623 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:31.604 7f01d9019700 1 -- 172.21.2.102:0/54257878 wait complete. 2021-02-10T11:36:31.623 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:31.618 7f01d9019700 1 Processor -- start 2021-02-10T11:36:31.623 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:31.618 7f01d9019700 1 -- start start 2021-02-10T11:36:31.625 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:31.619 7f01d9019700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f01d40c2370 0x7f01d418f870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:31.625 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:31.619 7f01d9019700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f01d4084cb0 0x7f01d41a7470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:31.625 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:31.619 7f01d9019700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f01d41a7a90 0x7f01d4085700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:31.625 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:31.619 7f01d9019700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01d40be840 con 0x7f01d4084cb0 2021-02-10T11:36:31.625 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:31.619 7f01d9019700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01d40c29a0 con 0x7f01d40c2370 2021-02-10T11:36:31.625 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:31.619 7f01d9019700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01d41a7e90 con 0x7f01d41a7a90 2021-02-10T11:36:31.625 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:31.619 7f01d259c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f01d4084cb0 0x7f01d41a7470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:31.625 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:31.619 7f01d259c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f01d4084cb0 0x7f01d41a7470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:37842/0 (socket says 172.21.2.102:37842) 2021-02-10T11:36:31.625 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:31.619 7f01d259c700 1 -- 172.21.2.102:0/3981814515 learned_addr learned my addr 172.21.2.102:0/3981814515 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:36:31.625 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:31.619 7f01d259c700 1 -- 172.21.2.102:0/3981814515 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f01d41a7a90 msgr2=0x7f01d4085700 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:36:31.626 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:31.619 7f01d259c700 1 --2- 172.21.2.102:0/3981814515 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f01d41a7a90 0x7f01d4085700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:31.626 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:31.619 7f01d259c700 1 -- 172.21.2.102:0/3981814515 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f01d40c2370 msgr2=0x7f01d418f870 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:36:31.626 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:31.619 7f01d259c700 1 --2- 172.21.2.102:0/3981814515 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f01d40c2370 0x7f01d418f870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:31.626 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:31.619 7f01d259c700 1 -- 172.21.2.102:0/3981814515 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01d41a80b0 con 0x7f01d4084cb0 2021-02-10T11:36:31.626 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:31.620 7f01d259c700 1 --2- 172.21.2.102:0/3981814515 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f01d4084cb0 0x7f01d41a7470 secure :-1 s=READY pgs=2826 cs=0 l=1 rev1=1 rx=0x7f01bc00b5b0 tx=0x7f01bc00b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:31.626 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:31.620 7f01cb7fe700 1 -- 172.21.2.102:0/3981814515 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f01bc00bd40 con 0x7f01d4084cb0 2021-02-10T11:36:31.626 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:31.620 7f01cb7fe700 1 -- 172.21.2.102:0/3981814515 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01bc000e20 con 0x7f01d4084cb0 2021-02-10T11:36:31.626 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:31.620 7f01cb7fe700 1 -- 172.21.2.102:0/3981814515 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f01bc006210 con 0x7f01d4084cb0 2021-02-10T11:36:31.626 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:31.620 7f01d9019700 1 -- 172.21.2.102:0/3981814515 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01d41a80b0 con 0x7f01d4084cb0 2021-02-10T11:36:31.626 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:31.620 7f01d9019700 1 -- 172.21.2.102:0/3981814515 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01d4086010 con 0x7f01d4084cb0 2021-02-10T11:36:31.627 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:31.620 7f01cb7fe700 1 -- 172.21.2.102:0/3981814515 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f01bc006020 con 0x7f01d4084cb0 2021-02-10T11:36:31.627 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:31.620 7f01cb7fe700 1 --2- 172.21.2.102:0/3981814515 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f01b80186c0 0x7f01b801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:31.627 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:31.621 7f01cb7fe700 1 -- 172.21.2.102:0/3981814515 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(153..153 src has 1..153) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f01bc032590 con 0x7f01d4084cb0 2021-02-10T11:36:31.627 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:31.621 7f01d2d9d700 1 --2- 172.21.2.102:0/3981814515 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f01b80186c0 0x7f01b801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:31.627 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:31.621 7f01d2d9d700 1 --2- 172.21.2.102:0/3981814515 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f01b80186c0 0x7f01b801abc0 crc :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:31.654 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:31.652 7f01d9019700 1 -- 172.21.2.102:0/3981814515 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01d4082a70 con 0x7f01d4084cb0 2021-02-10T11:36:31.656 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:31.654 7f01cb7fe700 1 -- 172.21.2.102:0/3981814515 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f01bc023cd0 con 0x7f01d4084cb0 2021-02-10T11:36:31.808 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:31.805 7f01d9019700 1 -- 172.21.2.102:0/3981814515 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f01d4082a70 con 0x7f01d4084cb0 2021-02-10T11:36:31.812 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:31.810 7f01cb7fe700 1 -- 172.21.2.102:0/3981814515 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v40)=0 v40) v1 ==== 122+0+0 (secure 0 0 0) 0x7f01bc023cd0 con 0x7f01d4084cb0 2021-02-10T11:36:31.817 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:31.815 7f01d9019700 1 -- 172.21.2.102:0/3981814515 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f01b80186c0 msgr2=0x7f01b801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:31.817 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:31.815 7f01d9019700 1 --2- 172.21.2.102:0/3981814515 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f01b80186c0 0x7f01b801abc0 crc :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:31.817 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:31.815 7f01d9019700 1 -- 172.21.2.102:0/3981814515 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f01d4084cb0 msgr2=0x7f01d41a7470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:31.817 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:31.815 7f01d9019700 1 --2- 172.21.2.102:0/3981814515 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f01d4084cb0 0x7f01d41a7470 secure :-1 s=READY pgs=2826 cs=0 l=1 rev1=1 rx=0x7f01bc00b5b0 tx=0x7f01bc00b950).stop 2021-02-10T11:36:31.817 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:31.815 7f01d9019700 1 -- 172.21.2.102:0/3981814515 shutdown_connections 2021-02-10T11:36:31.817 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:31.815 7f01d9019700 1 --2- 172.21.2.102:0/3981814515 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f01d41a7a90 0x7f01d4085700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:31.818 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:31.815 7f01d9019700 1 --2- 172.21.2.102:0/3981814515 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f01d4084cb0 0x7f01d41a7470 unknown :-1 s=CLOSED pgs=2826 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:31.818 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:31.815 7f01d9019700 1 --2- 172.21.2.102:0/3981814515 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f01b80186c0 0x7f01b801abc0 unknown :-1 s=CLOSED pgs=1240 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:31.818 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:31.815 7f01d9019700 1 --2- 172.21.2.102:0/3981814515 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f01d40c2370 0x7f01d418f870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:31.818 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:31.815 7f01d9019700 1 -- 172.21.2.102:0/3981814515 >> 172.21.2.102:0/3981814515 conn(0x7f01d40b5dc0 msgr2=0x7f01d40c2d90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:31.818 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:31.815 7f01d9019700 1 -- 172.21.2.102:0/3981814515 shutdown_connections 2021-02-10T11:36:31.818 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:31.815 7f01d9019700 1 -- 172.21.2.102:0/3981814515 wait complete. 2021-02-10T11:36:31.825 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2021-02-10T11:36:31.826 DEBUG:teuthology.orchestra.run.gibba002:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:36:32.173 DEBUG:teuthology.orchestra.run.gibba013:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:36:32.519 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2021-02-10T11:36:32.519 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:36:32.519 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:36:32.519 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:36:32.861 INFO:teuthology.orchestra.run.gibba002.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.0’: No such file or directory 2021-02-10T11:36:32.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:36:32.863 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.0 2021-02-10T11:36:32.864 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:36:32.864 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:36:32.864 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:36:33.200 INFO:teuthology.orchestra.run.gibba002.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.1’: No such file or directory 2021-02-10T11:36:33.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:36:33.201 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2021-02-10T11:36:33.201 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:36:33.201 DEBUG:teuthology.orchestra.run.gibba013:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:36:33.201 DEBUG:teuthology.orchestra.run.gibba013:> ' -- /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:36:33.536 INFO:teuthology.orchestra.run.gibba013.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.2’: No such file or directory 2021-02-10T11:36:33.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:36:33.537 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2021-02-10T11:36:33.537 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:36:33.537 DEBUG:teuthology.orchestra.run.gibba013:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:36:33.537 DEBUG:teuthology.orchestra.run.gibba013:> ' -- /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:36:33.872 INFO:teuthology.orchestra.run.gibba013.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.3’: No such file or directory 2021-02-10T11:36:33.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:36:33.872 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.3 2021-02-10T11:36:33.873 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json-pretty 2021-02-10T11:36:34.301 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:34.282 7f7933929700 1 Processor -- start 2021-02-10T11:36:34.301 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:34.282 7f7933929700 1 -- start start 2021-02-10T11:36:34.301 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:34.282 7f7933929700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f792c0d1870 0x7f792c1b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:34.301 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:34.282 7f7933929700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f792c1b0a10 0x7f792c0cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:34.301 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:34.282 7f7933929700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f792c0f0b10 con 0x7f792c0d1c70 2021-02-10T11:36:34.301 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:34.282 7f7933929700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f792c0e5ed0 con 0x7f792c0d1870 2021-02-10T11:36:34.301 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:34.282 7f7933929700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f792c0f0650 con 0x7f792c1b0a10 2021-02-10T11:36:34.301 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:34.283 7f7931745700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f792c0d1870 0x7f792c1b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:34.302 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:34.283 7f7931745700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f792c0d1870 0x7f792c1b03f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:37850/0 (socket says 172.21.2.102:37850) 2021-02-10T11:36:34.302 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:34.283 7f7931f46700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f792c0d1c70 0x7f792c0cf950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:58306/0 (socket says 172.21.2.102:58306) 2021-02-10T11:36:34.302 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:34.283 7f7931745700 1 -- 172.21.2.102:0/2424605105 learned_addr learned my addr 172.21.2.102:0/2424605105 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:36:34.302 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:34.283 7f7930f44700 1 --2- 172.21.2.102:0/2424605105 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f792c1b0a10 0x7f792c0cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:34.302 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:34.283 7f7931745700 1 -- 172.21.2.102:0/2424605105 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f792c1b0a10 msgr2=0x7f792c0cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:34.302 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:34.283 7f7931745700 1 --2- 172.21.2.102:0/2424605105 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f792c1b0a10 0x7f792c0cf330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:34.302 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:34.283 7f7931745700 1 -- 172.21.2.102:0/2424605105 >> v1:172.21.2.102:6789/0 conn(0x7f792c0d1c70 legacy=0x7f792c0cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:34.302 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:34.283 7f7931745700 1 -- 172.21.2.102:0/2424605105 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f792c0f0870 con 0x7f792c0d1870 2021-02-10T11:36:34.302 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:34.283 7f7930f44700 1 --2- 172.21.2.102:0/2424605105 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f792c1b0a10 0x7f792c0cf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:36:34.302 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:34.283 7f7931745700 1 --2- 172.21.2.102:0/2424605105 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f792c0d1870 0x7f792c1b03f0 secure :-1 s=READY pgs=2828 cs=0 l=1 rev1=1 rx=0x7f7928006700 tx=0x7f7928009b80).ready entity=mon.0 client_cookie=1b9f9333a4748705 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:34.302 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:34.283 7f791ffff700 1 -- 172.21.2.102:0/2424605105 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f792800c0c0 con 0x7f792c0d1870 2021-02-10T11:36:34.303 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:34.283 7f791ffff700 1 -- 172.21.2.102:0/2424605105 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f792800c340 con 0x7f792c0d1870 2021-02-10T11:36:34.303 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:34.283 7f791ffff700 1 -- 172.21.2.102:0/2424605105 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f792800c860 con 0x7f792c0d1870 2021-02-10T11:36:34.303 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:34.283 7f7933929700 1 -- 172.21.2.102:0/2424605105 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f792c0d1870 msgr2=0x7f792c1b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:34.303 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:34.283 7f7933929700 1 --2- 172.21.2.102:0/2424605105 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f792c0d1870 0x7f792c1b03f0 secure :-1 s=READY pgs=2828 cs=0 l=1 rev1=1 rx=0x7f7928006700 tx=0x7f7928009b80).stop 2021-02-10T11:36:34.303 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:34.284 7f7933929700 1 -- 172.21.2.102:0/2424605105 shutdown_connections 2021-02-10T11:36:34.303 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:34.284 7f7933929700 1 --2- 172.21.2.102:0/2424605105 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f792c1b0a10 0x7f792c0cf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:34.303 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:34.284 7f7933929700 1 --2- 172.21.2.102:0/2424605105 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f792c0d1870 0x7f792c1b03f0 unknown :-1 s=CLOSED pgs=2828 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:34.303 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:34.284 7f7933929700 1 -- 172.21.2.102:0/2424605105 >> 172.21.2.102:0/2424605105 conn(0x7f792c0ce450 msgr2=0x7f792c0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:34.303 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:34.284 7f7933929700 1 -- 172.21.2.102:0/2424605105 shutdown_connections 2021-02-10T11:36:34.303 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:34.284 7f7933929700 1 -- 172.21.2.102:0/2424605105 wait complete. 2021-02-10T11:36:34.303 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:34.299 7f7933929700 1 Processor -- start 2021-02-10T11:36:34.303 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:34.299 7f7933929700 1 -- start start 2021-02-10T11:36:34.305 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:34.299 7f7933929700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f792c195650 0x7f792c0b5560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:34.305 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:34.299 7f7933929700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f792c0b5b80 0x7f792c1a2e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:34.305 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:34.299 7f7933929700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f792c0acfa0 0x7f792c1a3440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:34.305 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:34.299 7f7933929700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f792c0b6010 con 0x7f792c195650 2021-02-10T11:36:34.305 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:34.299 7f7933929700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f792c1a3a60 con 0x7f792c0acfa0 2021-02-10T11:36:34.305 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:34.299 7f7933929700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f792c1a3c40 con 0x7f792c0b5b80 2021-02-10T11:36:34.305 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:34.299 7f7931745700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f792c195650 0x7f792c0b5560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:34.306 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:34.300 7f7931745700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f792c195650 0x7f792c0b5560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:37858/0 (socket says 172.21.2.102:37858) 2021-02-10T11:36:34.306 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:34.300 7f7931745700 1 -- 172.21.2.102:0/103158089 learned_addr learned my addr 172.21.2.102:0/103158089 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:36:34.306 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:34.300 7f7930f44700 1 --2- 172.21.2.102:0/103158089 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f792c0b5b80 0x7f792c1a2e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:34.306 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:34.300 7f7931f46700 1 --2- 172.21.2.102:0/103158089 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f792c0acfa0 0x7f792c1a3440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:34.306 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:34.300 7f7931745700 1 -- 172.21.2.102:0/103158089 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f792c0b5b80 msgr2=0x7f792c1a2e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:34.306 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:34.300 7f7931745700 1 --2- 172.21.2.102:0/103158089 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f792c0b5b80 0x7f792c1a2e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:34.306 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:34.300 7f7931745700 1 -- 172.21.2.102:0/103158089 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f792c0acfa0 msgr2=0x7f792c1a3440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:34.306 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:34.300 7f7931745700 1 --2- 172.21.2.102:0/103158089 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f792c0acfa0 0x7f792c1a3440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:34.306 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:34.300 7f7931745700 1 -- 172.21.2.102:0/103158089 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f792c0bcae0 con 0x7f792c195650 2021-02-10T11:36:34.306 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:34.300 7f7931f46700 1 --2- 172.21.2.102:0/103158089 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f792c0acfa0 0x7f792c1a3440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:36:34.307 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:34.300 7f7931745700 1 --2- 172.21.2.102:0/103158089 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f792c195650 0x7f792c0b5560 secure :-1 s=READY pgs=2829 cs=0 l=1 rev1=1 rx=0x7f792800b5e0 tx=0x7f792800b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:34.307 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:34.300 7f791dffb700 1 -- 172.21.2.102:0/103158089 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f792800bd70 con 0x7f792c195650 2021-02-10T11:36:34.307 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:34.300 7f791dffb700 1 -- 172.21.2.102:0/103158089 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7928002030 con 0x7f792c195650 2021-02-10T11:36:34.307 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:34.300 7f791dffb700 1 -- 172.21.2.102:0/103158089 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7928002550 con 0x7f792c195650 2021-02-10T11:36:34.307 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:34.300 7f7933929700 1 -- 172.21.2.102:0/103158089 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f792c0bcae0 con 0x7f792c195650 2021-02-10T11:36:34.307 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:34.300 7f7933929700 1 -- 172.21.2.102:0/103158089 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f792c0bcae0 con 0x7f792c195650 2021-02-10T11:36:34.307 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:34.301 7f791dffb700 1 -- 172.21.2.102:0/103158089 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f7928000c90 con 0x7f792c195650 2021-02-10T11:36:34.307 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:34.301 7f791dffb700 1 --2- 172.21.2.102:0/103158089 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f790c0186c0 0x7f790c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:34.307 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:34.301 7f791dffb700 1 -- 172.21.2.102:0/103158089 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(153..153 src has 1..153) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f7928032400 con 0x7f792c195650 2021-02-10T11:36:34.307 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:34.301 7f7930f44700 1 --2- 172.21.2.102:0/103158089 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f790c0186c0 0x7f790c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:34.307 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:34.302 7f7930f44700 1 --2- 172.21.2.102:0/103158089 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f790c0186c0 0x7f790c01abc0 crc :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:34.334 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:34.332 7f7933929700 1 -- 172.21.2.102:0/103158089 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f792c0b6010 con 0x7f792c195650 2021-02-10T11:36:34.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:34.335 7f791dffb700 1 -- 172.21.2.102:0/103158089 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f7928030d30 con 0x7f792c195650 2021-02-10T11:36:34.463 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:34.460 7f7933929700 1 -- 172.21.2.102:0/103158089 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7f792c0b6010 con 0x7f792c195650 2021-02-10T11:36:34.463 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:34.461 7f791dffb700 1 -- 172.21.2.102:0/103158089 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v153) v1 ==== 81+0+30374 (secure 0 0 0) 0x7f7928000c90 con 0x7f792c195650 2021-02-10T11:36:34.464 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:36:34.465 INFO:teuthology.orchestra.run.gibba002.stdout:{ 2021-02-10T11:36:34.465 INFO:teuthology.orchestra.run.gibba002.stdout: "epoch": 153, 2021-02-10T11:36:34.465 INFO:teuthology.orchestra.run.gibba002.stdout: "fsid": "68db30ea-dbce-4aef-a6df-f01c6fea0513", 2021-02-10T11:36:34.465 INFO:teuthology.orchestra.run.gibba002.stdout: "created": "2021-02-10 11:03:06.331557", 2021-02-10T11:36:34.465 INFO:teuthology.orchestra.run.gibba002.stdout: "modified": "2021-02-10 11:35:59.288212", 2021-02-10T11:36:34.465 INFO:teuthology.orchestra.run.gibba002.stdout: "last_up_change": "2021-02-10 11:03:32.690562", 2021-02-10T11:36:34.465 INFO:teuthology.orchestra.run.gibba002.stdout: "last_in_change": "2021-02-10 11:03:32.690562", 2021-02-10T11:36:34.466 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2021-02-10T11:36:34.466 INFO:teuthology.orchestra.run.gibba002.stdout: "flags_num": 5799936, 2021-02-10T11:36:34.466 INFO:teuthology.orchestra.run.gibba002.stdout: "flags_set": [ 2021-02-10T11:36:34.466 INFO:teuthology.orchestra.run.gibba002.stdout: "pglog_hardlimit", 2021-02-10T11:36:34.466 INFO:teuthology.orchestra.run.gibba002.stdout: "purged_snapdirs", 2021-02-10T11:36:34.466 INFO:teuthology.orchestra.run.gibba002.stdout: "recovery_deletes", 2021-02-10T11:36:34.466 INFO:teuthology.orchestra.run.gibba002.stdout: "sortbitwise" 2021-02-10T11:36:34.467 INFO:teuthology.orchestra.run.gibba002.stdout: ], 2021-02-10T11:36:34.467 INFO:teuthology.orchestra.run.gibba002.stdout: "crush_version": 4, 2021-02-10T11:36:34.467 INFO:teuthology.orchestra.run.gibba002.stdout: "full_ratio": 0.94999998807907104, 2021-02-10T11:36:34.467 INFO:teuthology.orchestra.run.gibba002.stdout: "backfillfull_ratio": 0.89999997615814209, 2021-02-10T11:36:34.467 INFO:teuthology.orchestra.run.gibba002.stdout: "nearfull_ratio": 0.85000002384185791, 2021-02-10T11:36:34.467 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_snapshot": "", 2021-02-10T11:36:34.467 INFO:teuthology.orchestra.run.gibba002.stdout: "pool_max": 33, 2021-02-10T11:36:34.468 INFO:teuthology.orchestra.run.gibba002.stdout: "max_osd": 8, 2021-02-10T11:36:34.468 INFO:teuthology.orchestra.run.gibba002.stdout: "require_min_compat_client": "jewel", 2021-02-10T11:36:34.468 INFO:teuthology.orchestra.run.gibba002.stdout: "min_compat_client": "jewel", 2021-02-10T11:36:34.468 INFO:teuthology.orchestra.run.gibba002.stdout: "require_osd_release": "nautilus", 2021-02-10T11:36:34.468 INFO:teuthology.orchestra.run.gibba002.stdout: "pools": [ 2021-02-10T11:36:34.468 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:36:34.468 INFO:teuthology.orchestra.run.gibba002.stdout: "pool": 1, 2021-02-10T11:36:34.469 INFO:teuthology.orchestra.run.gibba002.stdout: "pool_name": "rbd", 2021-02-10T11:36:34.469 INFO:teuthology.orchestra.run.gibba002.stdout: "create_time": "2021-02-10 11:03:37.322519", 2021-02-10T11:36:34.469 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 1, 2021-02-10T11:36:34.469 INFO:teuthology.orchestra.run.gibba002.stdout: "flags_names": "hashpspool", 2021-02-10T11:36:34.469 INFO:teuthology.orchestra.run.gibba002.stdout: "type": 1, 2021-02-10T11:36:34.469 INFO:teuthology.orchestra.run.gibba002.stdout: "size": 2, 2021-02-10T11:36:34.470 INFO:teuthology.orchestra.run.gibba002.stdout: "min_size": 1, 2021-02-10T11:36:34.470 INFO:teuthology.orchestra.run.gibba002.stdout: "crush_rule": 0, 2021-02-10T11:36:34.470 INFO:teuthology.orchestra.run.gibba002.stdout: "object_hash": 2, 2021-02-10T11:36:34.470 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:36:34.470 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num": 8, 2021-02-10T11:36:34.470 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_placement_num": 8, 2021-02-10T11:36:34.470 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_placement_num_target": 8, 2021-02-10T11:36:34.471 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num_target": 8, 2021-02-10T11:36:34.471 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num_pending": 8, 2021-02-10T11:36:34.471 INFO:teuthology.orchestra.run.gibba002.stdout: "last_pg_merge_meta": { 2021-02-10T11:36:34.471 INFO:teuthology.orchestra.run.gibba002.stdout: "source_pgid": "0.0", 2021-02-10T11:36:34.471 INFO:teuthology.orchestra.run.gibba002.stdout: "ready_epoch": 0, 2021-02-10T11:36:34.471 INFO:teuthology.orchestra.run.gibba002.stdout: "last_epoch_started": 0, 2021-02-10T11:36:34.471 INFO:teuthology.orchestra.run.gibba002.stdout: "last_epoch_clean": 0, 2021-02-10T11:36:34.472 INFO:teuthology.orchestra.run.gibba002.stdout: "source_version": "0'0", 2021-02-10T11:36:34.472 INFO:teuthology.orchestra.run.gibba002.stdout: "target_version": "0'0" 2021-02-10T11:36:34.472 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:36:34.472 INFO:teuthology.orchestra.run.gibba002.stdout: "last_change": "16", 2021-02-10T11:36:34.472 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend": "0", 2021-02-10T11:36:34.472 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:36:34.472 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:36:34.473 INFO:teuthology.orchestra.run.gibba002.stdout: "auid": 0, 2021-02-10T11:36:34.473 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:36:34.473 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_seq": 0, 2021-02-10T11:36:34.473 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_epoch": 0, 2021-02-10T11:36:34.473 INFO:teuthology.orchestra.run.gibba002.stdout: "pool_snaps": [], 2021-02-10T11:36:34.473 INFO:teuthology.orchestra.run.gibba002.stdout: "removed_snaps": "[]", 2021-02-10T11:36:34.473 INFO:teuthology.orchestra.run.gibba002.stdout: "quota_max_bytes": 0, 2021-02-10T11:36:34.474 INFO:teuthology.orchestra.run.gibba002.stdout: "quota_max_objects": 0, 2021-02-10T11:36:34.474 INFO:teuthology.orchestra.run.gibba002.stdout: "tiers": [], 2021-02-10T11:36:34.474 INFO:teuthology.orchestra.run.gibba002.stdout: "tier_of": -1, 2021-02-10T11:36:34.474 INFO:teuthology.orchestra.run.gibba002.stdout: "read_tier": -1, 2021-02-10T11:36:34.474 INFO:teuthology.orchestra.run.gibba002.stdout: "write_tier": -1, 2021-02-10T11:36:34.474 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_mode": "none", 2021-02-10T11:36:34.474 INFO:teuthology.orchestra.run.gibba002.stdout: "target_max_bytes": 0, 2021-02-10T11:36:34.475 INFO:teuthology.orchestra.run.gibba002.stdout: "target_max_objects": 0, 2021-02-10T11:36:34.475 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:36:34.475 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:36:34.475 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:36:34.475 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_min_flush_age": 0, 2021-02-10T11:36:34.475 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_min_evict_age": 0, 2021-02-10T11:36:34.475 INFO:teuthology.orchestra.run.gibba002.stdout: "erasure_code_profile": "", 2021-02-10T11:36:34.476 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_params": { 2021-02-10T11:36:34.476 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "none" 2021-02-10T11:36:34.476 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:36:34.476 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_period": 0, 2021-02-10T11:36:34.476 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_count": 0, 2021-02-10T11:36:34.476 INFO:teuthology.orchestra.run.gibba002.stdout: "use_gmt_hitset": true, 2021-02-10T11:36:34.476 INFO:teuthology.orchestra.run.gibba002.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:36:34.477 INFO:teuthology.orchestra.run.gibba002.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:36:34.477 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:36:34.477 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:36:34.477 INFO:teuthology.orchestra.run.gibba002.stdout: "grade_table": [], 2021-02-10T11:36:34.477 INFO:teuthology.orchestra.run.gibba002.stdout: "stripe_width": 0, 2021-02-10T11:36:34.477 INFO:teuthology.orchestra.run.gibba002.stdout: "expected_num_objects": 0, 2021-02-10T11:36:34.477 INFO:teuthology.orchestra.run.gibba002.stdout: "fast_read": false, 2021-02-10T11:36:34.478 INFO:teuthology.orchestra.run.gibba002.stdout: "options": {}, 2021-02-10T11:36:34.478 INFO:teuthology.orchestra.run.gibba002.stdout: "application_metadata": { 2021-02-10T11:36:34.478 INFO:teuthology.orchestra.run.gibba002.stdout: "rbd": {} 2021-02-10T11:36:34.478 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:36:34.478 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:36:34.478 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:36:34.478 INFO:teuthology.orchestra.run.gibba002.stdout: "pool": 32, 2021-02-10T11:36:34.479 INFO:teuthology.orchestra.run.gibba002.stdout: "pool_name": "cephfs_metadata", 2021-02-10T11:36:34.479 INFO:teuthology.orchestra.run.gibba002.stdout: "create_time": "2021-02-10 11:35:56.251556", 2021-02-10T11:36:34.479 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 1, 2021-02-10T11:36:34.479 INFO:teuthology.orchestra.run.gibba002.stdout: "flags_names": "hashpspool", 2021-02-10T11:36:34.479 INFO:teuthology.orchestra.run.gibba002.stdout: "type": 1, 2021-02-10T11:36:34.479 INFO:teuthology.orchestra.run.gibba002.stdout: "size": 2, 2021-02-10T11:36:34.479 INFO:teuthology.orchestra.run.gibba002.stdout: "min_size": 1, 2021-02-10T11:36:34.480 INFO:teuthology.orchestra.run.gibba002.stdout: "crush_rule": 0, 2021-02-10T11:36:34.480 INFO:teuthology.orchestra.run.gibba002.stdout: "object_hash": 2, 2021-02-10T11:36:34.480 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:36:34.480 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num": 8, 2021-02-10T11:36:34.480 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_placement_num": 8, 2021-02-10T11:36:34.480 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_placement_num_target": 8, 2021-02-10T11:36:34.481 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num_target": 8, 2021-02-10T11:36:34.481 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num_pending": 8, 2021-02-10T11:36:34.481 INFO:teuthology.orchestra.run.gibba002.stdout: "last_pg_merge_meta": { 2021-02-10T11:36:34.481 INFO:teuthology.orchestra.run.gibba002.stdout: "source_pgid": "0.0", 2021-02-10T11:36:34.481 INFO:teuthology.orchestra.run.gibba002.stdout: "ready_epoch": 0, 2021-02-10T11:36:34.481 INFO:teuthology.orchestra.run.gibba002.stdout: "last_epoch_started": 0, 2021-02-10T11:36:34.481 INFO:teuthology.orchestra.run.gibba002.stdout: "last_epoch_clean": 0, 2021-02-10T11:36:34.481 INFO:teuthology.orchestra.run.gibba002.stdout: "source_version": "0'0", 2021-02-10T11:36:34.482 INFO:teuthology.orchestra.run.gibba002.stdout: "target_version": "0'0" 2021-02-10T11:36:34.482 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:36:34.482 INFO:teuthology.orchestra.run.gibba002.stdout: "last_change": "152", 2021-02-10T11:36:34.482 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend": "0", 2021-02-10T11:36:34.482 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:36:34.482 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:36:34.482 INFO:teuthology.orchestra.run.gibba002.stdout: "auid": 0, 2021-02-10T11:36:34.483 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:36:34.483 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_seq": 0, 2021-02-10T11:36:34.483 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_epoch": 0, 2021-02-10T11:36:34.483 INFO:teuthology.orchestra.run.gibba002.stdout: "pool_snaps": [], 2021-02-10T11:36:34.483 INFO:teuthology.orchestra.run.gibba002.stdout: "removed_snaps": "[]", 2021-02-10T11:36:34.483 INFO:teuthology.orchestra.run.gibba002.stdout: "quota_max_bytes": 0, 2021-02-10T11:36:34.483 INFO:teuthology.orchestra.run.gibba002.stdout: "quota_max_objects": 0, 2021-02-10T11:36:34.483 INFO:teuthology.orchestra.run.gibba002.stdout: "tiers": [], 2021-02-10T11:36:34.484 INFO:teuthology.orchestra.run.gibba002.stdout: "tier_of": -1, 2021-02-10T11:36:34.484 INFO:teuthology.orchestra.run.gibba002.stdout: "read_tier": -1, 2021-02-10T11:36:34.484 INFO:teuthology.orchestra.run.gibba002.stdout: "write_tier": -1, 2021-02-10T11:36:34.484 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_mode": "none", 2021-02-10T11:36:34.484 INFO:teuthology.orchestra.run.gibba002.stdout: "target_max_bytes": 0, 2021-02-10T11:36:34.484 INFO:teuthology.orchestra.run.gibba002.stdout: "target_max_objects": 0, 2021-02-10T11:36:34.484 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:36:34.485 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:36:34.485 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:36:34.485 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_min_flush_age": 0, 2021-02-10T11:36:34.485 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_min_evict_age": 0, 2021-02-10T11:36:34.485 INFO:teuthology.orchestra.run.gibba002.stdout: "erasure_code_profile": "", 2021-02-10T11:36:34.485 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_params": { 2021-02-10T11:36:34.485 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "none" 2021-02-10T11:36:34.485 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:36:34.486 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_period": 0, 2021-02-10T11:36:34.486 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_count": 0, 2021-02-10T11:36:34.486 INFO:teuthology.orchestra.run.gibba002.stdout: "use_gmt_hitset": true, 2021-02-10T11:36:34.486 INFO:teuthology.orchestra.run.gibba002.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:36:34.486 INFO:teuthology.orchestra.run.gibba002.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:36:34.486 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:36:34.486 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:36:34.487 INFO:teuthology.orchestra.run.gibba002.stdout: "grade_table": [], 2021-02-10T11:36:34.487 INFO:teuthology.orchestra.run.gibba002.stdout: "stripe_width": 0, 2021-02-10T11:36:34.487 INFO:teuthology.orchestra.run.gibba002.stdout: "expected_num_objects": 0, 2021-02-10T11:36:34.487 INFO:teuthology.orchestra.run.gibba002.stdout: "fast_read": false, 2021-02-10T11:36:34.487 INFO:teuthology.orchestra.run.gibba002.stdout: "options": { 2021-02-10T11:36:34.487 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_autoscale_bias": 4, 2021-02-10T11:36:34.487 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num_min": 16, 2021-02-10T11:36:34.488 INFO:teuthology.orchestra.run.gibba002.stdout: "recovery_priority": 5 2021-02-10T11:36:34.488 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:36:34.488 INFO:teuthology.orchestra.run.gibba002.stdout: "application_metadata": { 2021-02-10T11:36:34.488 INFO:teuthology.orchestra.run.gibba002.stdout: "cephfs": { 2021-02-10T11:36:34.488 INFO:teuthology.orchestra.run.gibba002.stdout: "metadata": "cephfs" 2021-02-10T11:36:34.488 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:36:34.488 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:36:34.488 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:36:34.489 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:36:34.489 INFO:teuthology.orchestra.run.gibba002.stdout: "pool": 33, 2021-02-10T11:36:34.489 INFO:teuthology.orchestra.run.gibba002.stdout: "pool_name": "cephfs_data", 2021-02-10T11:36:34.489 INFO:teuthology.orchestra.run.gibba002.stdout: "create_time": "2021-02-10 11:35:57.328218", 2021-02-10T11:36:34.489 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 1, 2021-02-10T11:36:34.489 INFO:teuthology.orchestra.run.gibba002.stdout: "flags_names": "hashpspool", 2021-02-10T11:36:34.489 INFO:teuthology.orchestra.run.gibba002.stdout: "type": 1, 2021-02-10T11:36:34.490 INFO:teuthology.orchestra.run.gibba002.stdout: "size": 2, 2021-02-10T11:36:34.490 INFO:teuthology.orchestra.run.gibba002.stdout: "min_size": 1, 2021-02-10T11:36:34.490 INFO:teuthology.orchestra.run.gibba002.stdout: "crush_rule": 0, 2021-02-10T11:36:34.490 INFO:teuthology.orchestra.run.gibba002.stdout: "object_hash": 2, 2021-02-10T11:36:34.490 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:36:34.490 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num": 8, 2021-02-10T11:36:34.490 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_placement_num": 8, 2021-02-10T11:36:34.491 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_placement_num_target": 8, 2021-02-10T11:36:34.491 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num_target": 8, 2021-02-10T11:36:34.491 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num_pending": 8, 2021-02-10T11:36:34.491 INFO:teuthology.orchestra.run.gibba002.stdout: "last_pg_merge_meta": { 2021-02-10T11:36:34.491 INFO:teuthology.orchestra.run.gibba002.stdout: "source_pgid": "0.0", 2021-02-10T11:36:34.491 INFO:teuthology.orchestra.run.gibba002.stdout: "ready_epoch": 0, 2021-02-10T11:36:34.491 INFO:teuthology.orchestra.run.gibba002.stdout: "last_epoch_started": 0, 2021-02-10T11:36:34.492 INFO:teuthology.orchestra.run.gibba002.stdout: "last_epoch_clean": 0, 2021-02-10T11:36:34.492 INFO:teuthology.orchestra.run.gibba002.stdout: "source_version": "0'0", 2021-02-10T11:36:34.492 INFO:teuthology.orchestra.run.gibba002.stdout: "target_version": "0'0" 2021-02-10T11:36:34.492 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:36:34.492 INFO:teuthology.orchestra.run.gibba002.stdout: "last_change": "153", 2021-02-10T11:36:34.492 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend": "0", 2021-02-10T11:36:34.493 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:36:34.493 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:36:34.493 INFO:teuthology.orchestra.run.gibba002.stdout: "auid": 0, 2021-02-10T11:36:34.493 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:36:34.493 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_seq": 0, 2021-02-10T11:36:34.493 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_epoch": 0, 2021-02-10T11:36:34.494 INFO:teuthology.orchestra.run.gibba002.stdout: "pool_snaps": [], 2021-02-10T11:36:34.494 INFO:teuthology.orchestra.run.gibba002.stdout: "removed_snaps": "[]", 2021-02-10T11:36:34.494 INFO:teuthology.orchestra.run.gibba002.stdout: "quota_max_bytes": 0, 2021-02-10T11:36:34.494 INFO:teuthology.orchestra.run.gibba002.stdout: "quota_max_objects": 0, 2021-02-10T11:36:34.494 INFO:teuthology.orchestra.run.gibba002.stdout: "tiers": [], 2021-02-10T11:36:34.494 INFO:teuthology.orchestra.run.gibba002.stdout: "tier_of": -1, 2021-02-10T11:36:34.495 INFO:teuthology.orchestra.run.gibba002.stdout: "read_tier": -1, 2021-02-10T11:36:34.495 INFO:teuthology.orchestra.run.gibba002.stdout: "write_tier": -1, 2021-02-10T11:36:34.495 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_mode": "none", 2021-02-10T11:36:34.495 INFO:teuthology.orchestra.run.gibba002.stdout: "target_max_bytes": 0, 2021-02-10T11:36:34.495 INFO:teuthology.orchestra.run.gibba002.stdout: "target_max_objects": 0, 2021-02-10T11:36:34.495 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:36:34.495 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:36:34.495 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:36:34.496 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_min_flush_age": 0, 2021-02-10T11:36:34.496 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_min_evict_age": 0, 2021-02-10T11:36:34.496 INFO:teuthology.orchestra.run.gibba002.stdout: "erasure_code_profile": "", 2021-02-10T11:36:34.496 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_params": { 2021-02-10T11:36:34.496 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "none" 2021-02-10T11:36:34.496 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:36:34.496 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_period": 0, 2021-02-10T11:36:34.497 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_count": 0, 2021-02-10T11:36:34.497 INFO:teuthology.orchestra.run.gibba002.stdout: "use_gmt_hitset": true, 2021-02-10T11:36:34.497 INFO:teuthology.orchestra.run.gibba002.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:36:34.497 INFO:teuthology.orchestra.run.gibba002.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:36:34.497 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:36:34.497 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:36:34.497 INFO:teuthology.orchestra.run.gibba002.stdout: "grade_table": [], 2021-02-10T11:36:34.498 INFO:teuthology.orchestra.run.gibba002.stdout: "stripe_width": 0, 2021-02-10T11:36:34.498 INFO:teuthology.orchestra.run.gibba002.stdout: "expected_num_objects": 0, 2021-02-10T11:36:34.498 INFO:teuthology.orchestra.run.gibba002.stdout: "fast_read": false, 2021-02-10T11:36:34.498 INFO:teuthology.orchestra.run.gibba002.stdout: "options": {}, 2021-02-10T11:36:34.498 INFO:teuthology.orchestra.run.gibba002.stdout: "application_metadata": { 2021-02-10T11:36:34.498 INFO:teuthology.orchestra.run.gibba002.stdout: "cephfs": { 2021-02-10T11:36:34.499 INFO:teuthology.orchestra.run.gibba002.stdout: "data": "cephfs" 2021-02-10T11:36:34.499 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:36:34.499 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:36:34.499 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:36:34.499 INFO:teuthology.orchestra.run.gibba002.stdout: ], 2021-02-10T11:36:34.499 INFO:teuthology.orchestra.run.gibba002.stdout: "osds": [ 2021-02-10T11:36:34.500 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:36:34.500 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 0, 2021-02-10T11:36:34.500 INFO:teuthology.orchestra.run.gibba002.stdout: "uuid": "be691b4d-a039-408a-af80-9d53f02b6f1f", 2021-02-10T11:36:34.500 INFO:teuthology.orchestra.run.gibba002.stdout: "up": 1, 2021-02-10T11:36:34.500 INFO:teuthology.orchestra.run.gibba002.stdout: "in": 1, 2021-02-10T11:36:34.500 INFO:teuthology.orchestra.run.gibba002.stdout: "weight": 1, 2021-02-10T11:36:34.500 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_affinity": 1, 2021-02-10T11:36:34.500 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_begin": 0, 2021-02-10T11:36:34.501 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_end": 0, 2021-02-10T11:36:34.501 INFO:teuthology.orchestra.run.gibba002.stdout: "up_from": 13, 2021-02-10T11:36:34.501 INFO:teuthology.orchestra.run.gibba002.stdout: "up_thru": 150, 2021-02-10T11:36:34.501 INFO:teuthology.orchestra.run.gibba002.stdout: "down_at": 0, 2021-02-10T11:36:34.501 INFO:teuthology.orchestra.run.gibba002.stdout: "lost_at": 0, 2021-02-10T11:36:34.501 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addrs": { 2021-02-10T11:36:34.501 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:36:34.501 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:36:34.501 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:36:34.502 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6818", 2021-02-10T11:36:34.502 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36088 2021-02-10T11:36:34.502 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:36:34.502 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:36:34.502 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:36:34.502 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6820", 2021-02-10T11:36:34.502 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36088 2021-02-10T11:36:34.502 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:36:34.502 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:36:34.502 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:36:34.502 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addrs": { 2021-02-10T11:36:34.502 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:36:34.503 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:36:34.503 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:36:34.503 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6822", 2021-02-10T11:36:34.503 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36088 2021-02-10T11:36:34.503 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:36:34.503 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:36:34.503 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:36:34.503 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6824", 2021-02-10T11:36:34.503 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36088 2021-02-10T11:36:34.503 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:36:34.503 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:36:34.503 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:36:34.503 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addrs": { 2021-02-10T11:36:34.504 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:36:34.504 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:36:34.504 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:36:34.504 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6830", 2021-02-10T11:36:34.504 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36088 2021-02-10T11:36:34.504 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:36:34.504 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:36:34.504 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:36:34.504 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6832", 2021-02-10T11:36:34.504 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36088 2021-02-10T11:36:34.504 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:36:34.504 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:36:34.504 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:36:34.504 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addrs": { 2021-02-10T11:36:34.505 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:36:34.505 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:36:34.505 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:36:34.505 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6826", 2021-02-10T11:36:34.505 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36088 2021-02-10T11:36:34.505 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:36:34.505 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:36:34.505 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:36:34.505 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6828", 2021-02-10T11:36:34.505 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36088 2021-02-10T11:36:34.505 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:36:34.505 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:36:34.505 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:36:34.506 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addr": "172.21.2.102:6820/36088", 2021-02-10T11:36:34.506 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addr": "172.21.2.102:6824/36088", 2021-02-10T11:36:34.506 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addr": "172.21.2.102:6832/36088", 2021-02-10T11:36:34.506 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addr": "172.21.2.102:6828/36088", 2021-02-10T11:36:34.506 INFO:teuthology.orchestra.run.gibba002.stdout: "state": [ 2021-02-10T11:36:34.506 INFO:teuthology.orchestra.run.gibba002.stdout: "exists", 2021-02-10T11:36:34.506 INFO:teuthology.orchestra.run.gibba002.stdout: "up" 2021-02-10T11:36:34.506 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:36:34.506 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:36:34.506 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:36:34.507 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 1, 2021-02-10T11:36:34.507 INFO:teuthology.orchestra.run.gibba002.stdout: "uuid": "64e3ccc8-9c69-4d7c-865e-af8e83822635", 2021-02-10T11:36:34.507 INFO:teuthology.orchestra.run.gibba002.stdout: "up": 1, 2021-02-10T11:36:34.507 INFO:teuthology.orchestra.run.gibba002.stdout: "in": 1, 2021-02-10T11:36:34.507 INFO:teuthology.orchestra.run.gibba002.stdout: "weight": 1, 2021-02-10T11:36:34.507 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_affinity": 1, 2021-02-10T11:36:34.507 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_begin": 0, 2021-02-10T11:36:34.507 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_end": 0, 2021-02-10T11:36:34.507 INFO:teuthology.orchestra.run.gibba002.stdout: "up_from": 13, 2021-02-10T11:36:34.507 INFO:teuthology.orchestra.run.gibba002.stdout: "up_thru": 143, 2021-02-10T11:36:34.507 INFO:teuthology.orchestra.run.gibba002.stdout: "down_at": 0, 2021-02-10T11:36:34.507 INFO:teuthology.orchestra.run.gibba002.stdout: "lost_at": 0, 2021-02-10T11:36:34.507 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addrs": { 2021-02-10T11:36:34.507 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:36:34.508 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:36:34.508 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:36:34.508 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6819", 2021-02-10T11:36:34.508 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36087 2021-02-10T11:36:34.508 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:36:34.508 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:36:34.508 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:36:34.508 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6821", 2021-02-10T11:36:34.508 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36087 2021-02-10T11:36:34.508 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:36:34.508 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:36:34.508 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:36:34.508 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addrs": { 2021-02-10T11:36:34.509 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:36:34.509 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:36:34.509 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:36:34.509 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6823", 2021-02-10T11:36:34.509 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36087 2021-02-10T11:36:34.509 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:36:34.509 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:36:34.509 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:36:34.509 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6825", 2021-02-10T11:36:34.509 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36087 2021-02-10T11:36:34.509 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:36:34.509 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:36:34.509 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:36:34.509 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addrs": { 2021-02-10T11:36:34.510 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:36:34.510 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:36:34.510 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:36:34.510 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6831", 2021-02-10T11:36:34.510 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36087 2021-02-10T11:36:34.510 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:36:34.510 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:36:34.510 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:36:34.510 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6833", 2021-02-10T11:36:34.510 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36087 2021-02-10T11:36:34.510 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:36:34.510 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:36:34.510 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:36:34.511 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addrs": { 2021-02-10T11:36:34.511 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:36:34.511 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:36:34.511 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:36:34.511 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6827", 2021-02-10T11:36:34.511 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36087 2021-02-10T11:36:34.511 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:36:34.511 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:36:34.511 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:36:34.511 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6829", 2021-02-10T11:36:34.511 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36087 2021-02-10T11:36:34.511 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:36:34.511 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:36:34.512 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:36:34.512 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addr": "172.21.2.102:6821/36087", 2021-02-10T11:36:34.512 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addr": "172.21.2.102:6825/36087", 2021-02-10T11:36:34.512 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addr": "172.21.2.102:6833/36087", 2021-02-10T11:36:34.512 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addr": "172.21.2.102:6829/36087", 2021-02-10T11:36:34.512 INFO:teuthology.orchestra.run.gibba002.stdout: "state": [ 2021-02-10T11:36:34.512 INFO:teuthology.orchestra.run.gibba002.stdout: "exists", 2021-02-10T11:36:34.512 INFO:teuthology.orchestra.run.gibba002.stdout: "up" 2021-02-10T11:36:34.512 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:36:34.512 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:36:34.513 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:36:34.513 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 2, 2021-02-10T11:36:34.513 INFO:teuthology.orchestra.run.gibba002.stdout: "uuid": "153700af-747f-481f-bdf4-d6072d3a842f", 2021-02-10T11:36:34.513 INFO:teuthology.orchestra.run.gibba002.stdout: "up": 1, 2021-02-10T11:36:34.513 INFO:teuthology.orchestra.run.gibba002.stdout: "in": 1, 2021-02-10T11:36:34.513 INFO:teuthology.orchestra.run.gibba002.stdout: "weight": 1, 2021-02-10T11:36:34.513 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_affinity": 1, 2021-02-10T11:36:34.513 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_begin": 0, 2021-02-10T11:36:34.513 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_end": 0, 2021-02-10T11:36:34.513 INFO:teuthology.orchestra.run.gibba002.stdout: "up_from": 13, 2021-02-10T11:36:34.513 INFO:teuthology.orchestra.run.gibba002.stdout: "up_thru": 151, 2021-02-10T11:36:34.513 INFO:teuthology.orchestra.run.gibba002.stdout: "down_at": 0, 2021-02-10T11:36:34.513 INFO:teuthology.orchestra.run.gibba002.stdout: "lost_at": 0, 2021-02-10T11:36:34.513 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addrs": { 2021-02-10T11:36:34.514 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:36:34.514 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:36:34.514 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:36:34.514 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6810", 2021-02-10T11:36:34.514 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36089 2021-02-10T11:36:34.514 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:36:34.514 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:36:34.514 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:36:34.514 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6811", 2021-02-10T11:36:34.514 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36089 2021-02-10T11:36:34.514 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:36:34.514 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:36:34.514 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:36:34.515 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addrs": { 2021-02-10T11:36:34.515 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:36:34.515 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:36:34.515 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:36:34.515 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6812", 2021-02-10T11:36:34.515 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36089 2021-02-10T11:36:34.515 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:36:34.515 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:36:34.515 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:36:34.515 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6813", 2021-02-10T11:36:34.515 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36089 2021-02-10T11:36:34.515 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:36:34.515 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:36:34.515 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:36:34.516 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addrs": { 2021-02-10T11:36:34.516 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:36:34.516 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:36:34.516 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:36:34.516 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6816", 2021-02-10T11:36:34.516 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36089 2021-02-10T11:36:34.516 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:36:34.516 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:36:34.516 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:36:34.516 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6817", 2021-02-10T11:36:34.516 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36089 2021-02-10T11:36:34.516 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:36:34.516 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:36:34.517 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:36:34.517 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addrs": { 2021-02-10T11:36:34.517 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:36:34.517 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:36:34.517 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:36:34.517 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6814", 2021-02-10T11:36:34.517 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36089 2021-02-10T11:36:34.517 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:36:34.517 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:36:34.517 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:36:34.517 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6815", 2021-02-10T11:36:34.517 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36089 2021-02-10T11:36:34.517 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:36:34.517 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:36:34.518 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:36:34.518 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addr": "172.21.2.102:6811/36089", 2021-02-10T11:36:34.518 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addr": "172.21.2.102:6813/36089", 2021-02-10T11:36:34.518 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addr": "172.21.2.102:6817/36089", 2021-02-10T11:36:34.518 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addr": "172.21.2.102:6815/36089", 2021-02-10T11:36:34.518 INFO:teuthology.orchestra.run.gibba002.stdout: "state": [ 2021-02-10T11:36:34.518 INFO:teuthology.orchestra.run.gibba002.stdout: "exists", 2021-02-10T11:36:34.518 INFO:teuthology.orchestra.run.gibba002.stdout: "up" 2021-02-10T11:36:34.518 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:36:34.518 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:36:34.518 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:36:34.518 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 3, 2021-02-10T11:36:34.518 INFO:teuthology.orchestra.run.gibba002.stdout: "uuid": "cbc678f4-83a9-45e6-8830-2f7c232b1ea7", 2021-02-10T11:36:34.519 INFO:teuthology.orchestra.run.gibba002.stdout: "up": 1, 2021-02-10T11:36:34.519 INFO:teuthology.orchestra.run.gibba002.stdout: "in": 1, 2021-02-10T11:36:34.519 INFO:teuthology.orchestra.run.gibba002.stdout: "weight": 1, 2021-02-10T11:36:34.519 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_affinity": 1, 2021-02-10T11:36:34.519 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_begin": 0, 2021-02-10T11:36:34.519 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_end": 0, 2021-02-10T11:36:34.519 INFO:teuthology.orchestra.run.gibba002.stdout: "up_from": 13, 2021-02-10T11:36:34.519 INFO:teuthology.orchestra.run.gibba002.stdout: "up_thru": 151, 2021-02-10T11:36:34.519 INFO:teuthology.orchestra.run.gibba002.stdout: "down_at": 0, 2021-02-10T11:36:34.519 INFO:teuthology.orchestra.run.gibba002.stdout: "lost_at": 0, 2021-02-10T11:36:34.519 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addrs": { 2021-02-10T11:36:34.519 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:36:34.519 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:36:34.520 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:36:34.520 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6802", 2021-02-10T11:36:34.520 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36090 2021-02-10T11:36:34.520 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:36:34.520 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:36:34.520 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:36:34.520 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6803", 2021-02-10T11:36:34.520 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36090 2021-02-10T11:36:34.520 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:36:34.520 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:36:34.520 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:36:34.520 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addrs": { 2021-02-10T11:36:34.520 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:36:34.520 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:36:34.521 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:36:34.521 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6804", 2021-02-10T11:36:34.521 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36090 2021-02-10T11:36:34.521 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:36:34.521 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:36:34.521 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:36:34.521 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6805", 2021-02-10T11:36:34.521 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36090 2021-02-10T11:36:34.521 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:36:34.521 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:36:34.521 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:36:34.521 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addrs": { 2021-02-10T11:36:34.521 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:36:34.522 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:36:34.522 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:36:34.522 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6808", 2021-02-10T11:36:34.522 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36090 2021-02-10T11:36:34.522 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:36:34.522 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:36:34.522 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:36:34.522 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6809", 2021-02-10T11:36:34.522 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36090 2021-02-10T11:36:34.522 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:36:34.522 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:36:34.522 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:36:34.522 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addrs": { 2021-02-10T11:36:34.523 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:36:34.523 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:36:34.523 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:36:34.523 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6806", 2021-02-10T11:36:34.523 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36090 2021-02-10T11:36:34.523 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:36:34.523 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:36:34.523 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:36:34.523 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6807", 2021-02-10T11:36:34.523 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36090 2021-02-10T11:36:34.523 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:36:34.523 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:36:34.523 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:36:34.524 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addr": "172.21.2.102:6803/36090", 2021-02-10T11:36:34.524 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addr": "172.21.2.102:6805/36090", 2021-02-10T11:36:34.524 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addr": "172.21.2.102:6809/36090", 2021-02-10T11:36:34.524 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addr": "172.21.2.102:6807/36090", 2021-02-10T11:36:34.524 INFO:teuthology.orchestra.run.gibba002.stdout: "state": [ 2021-02-10T11:36:34.524 INFO:teuthology.orchestra.run.gibba002.stdout: "exists", 2021-02-10T11:36:34.524 INFO:teuthology.orchestra.run.gibba002.stdout: "up" 2021-02-10T11:36:34.524 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:36:34.524 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:36:34.524 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:36:34.524 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 4, 2021-02-10T11:36:34.524 INFO:teuthology.orchestra.run.gibba002.stdout: "uuid": "c01e4008-b258-4f10-afd2-9e7f37214351", 2021-02-10T11:36:34.524 INFO:teuthology.orchestra.run.gibba002.stdout: "up": 1, 2021-02-10T11:36:34.524 INFO:teuthology.orchestra.run.gibba002.stdout: "in": 1, 2021-02-10T11:36:34.525 INFO:teuthology.orchestra.run.gibba002.stdout: "weight": 1, 2021-02-10T11:36:34.525 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_affinity": 1, 2021-02-10T11:36:34.525 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_begin": 0, 2021-02-10T11:36:34.525 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_end": 0, 2021-02-10T11:36:34.525 INFO:teuthology.orchestra.run.gibba002.stdout: "up_from": 13, 2021-02-10T11:36:34.525 INFO:teuthology.orchestra.run.gibba002.stdout: "up_thru": 143, 2021-02-10T11:36:34.525 INFO:teuthology.orchestra.run.gibba002.stdout: "down_at": 0, 2021-02-10T11:36:34.525 INFO:teuthology.orchestra.run.gibba002.stdout: "lost_at": 0, 2021-02-10T11:36:34.525 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addrs": { 2021-02-10T11:36:34.525 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:36:34.525 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:36:34.525 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:36:34.525 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6808", 2021-02-10T11:36:34.525 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36368 2021-02-10T11:36:34.526 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:36:34.526 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:36:34.526 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:36:34.526 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6811", 2021-02-10T11:36:34.526 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36368 2021-02-10T11:36:34.526 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:36:34.526 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:36:34.526 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:36:34.526 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addrs": { 2021-02-10T11:36:34.526 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:36:34.526 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:36:34.526 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:36:34.526 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6814", 2021-02-10T11:36:34.527 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36368 2021-02-10T11:36:34.527 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:36:34.527 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:36:34.527 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:36:34.527 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6818", 2021-02-10T11:36:34.527 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36368 2021-02-10T11:36:34.527 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:36:34.527 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:36:34.527 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:36:34.527 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addrs": { 2021-02-10T11:36:34.527 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:36:34.527 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:36:34.527 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:36:34.527 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6826", 2021-02-10T11:36:34.528 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36368 2021-02-10T11:36:34.528 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:36:34.528 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:36:34.528 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:36:34.528 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6828", 2021-02-10T11:36:34.528 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36368 2021-02-10T11:36:34.528 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:36:34.528 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:36:34.528 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:36:34.528 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addrs": { 2021-02-10T11:36:34.528 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:36:34.528 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:36:34.528 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:36:34.529 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6820", 2021-02-10T11:36:34.529 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36368 2021-02-10T11:36:34.529 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:36:34.529 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:36:34.529 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:36:34.529 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6824", 2021-02-10T11:36:34.529 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36368 2021-02-10T11:36:34.529 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:36:34.529 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:36:34.529 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:36:34.529 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addr": "172.21.2.113:6811/36368", 2021-02-10T11:36:34.529 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addr": "172.21.2.113:6818/36368", 2021-02-10T11:36:34.529 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addr": "172.21.2.113:6828/36368", 2021-02-10T11:36:34.529 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addr": "172.21.2.113:6824/36368", 2021-02-10T11:36:34.530 INFO:teuthology.orchestra.run.gibba002.stdout: "state": [ 2021-02-10T11:36:34.530 INFO:teuthology.orchestra.run.gibba002.stdout: "exists", 2021-02-10T11:36:34.530 INFO:teuthology.orchestra.run.gibba002.stdout: "up" 2021-02-10T11:36:34.530 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:36:34.530 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:36:34.530 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:36:34.530 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 5, 2021-02-10T11:36:34.530 INFO:teuthology.orchestra.run.gibba002.stdout: "uuid": "551f6007-daf8-4a98-8ee0-c2b16bcccb5e", 2021-02-10T11:36:34.530 INFO:teuthology.orchestra.run.gibba002.stdout: "up": 1, 2021-02-10T11:36:34.530 INFO:teuthology.orchestra.run.gibba002.stdout: "in": 1, 2021-02-10T11:36:34.530 INFO:teuthology.orchestra.run.gibba002.stdout: "weight": 1, 2021-02-10T11:36:34.530 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_affinity": 1, 2021-02-10T11:36:34.530 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_begin": 0, 2021-02-10T11:36:34.530 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_end": 0, 2021-02-10T11:36:34.531 INFO:teuthology.orchestra.run.gibba002.stdout: "up_from": 13, 2021-02-10T11:36:34.531 INFO:teuthology.orchestra.run.gibba002.stdout: "up_thru": 151, 2021-02-10T11:36:34.531 INFO:teuthology.orchestra.run.gibba002.stdout: "down_at": 0, 2021-02-10T11:36:34.531 INFO:teuthology.orchestra.run.gibba002.stdout: "lost_at": 0, 2021-02-10T11:36:34.531 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addrs": { 2021-02-10T11:36:34.531 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:36:34.531 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:36:34.531 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:36:34.531 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6802", 2021-02-10T11:36:34.531 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36366 2021-02-10T11:36:34.531 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:36:34.531 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:36:34.531 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:36:34.532 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6805", 2021-02-10T11:36:34.532 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36366 2021-02-10T11:36:34.532 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:36:34.532 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:36:34.532 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:36:34.532 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addrs": { 2021-02-10T11:36:34.532 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:36:34.532 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:36:34.532 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:36:34.532 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6807", 2021-02-10T11:36:34.532 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36366 2021-02-10T11:36:34.532 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:36:34.532 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:36:34.532 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:36:34.533 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6810", 2021-02-10T11:36:34.533 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36366 2021-02-10T11:36:34.533 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:36:34.533 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:36:34.533 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:36:34.533 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addrs": { 2021-02-10T11:36:34.533 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:36:34.533 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:36:34.533 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:36:34.533 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6819", 2021-02-10T11:36:34.533 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36366 2021-02-10T11:36:34.533 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:36:34.533 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:36:34.533 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:36:34.534 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6822", 2021-02-10T11:36:34.534 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36366 2021-02-10T11:36:34.534 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:36:34.534 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:36:34.534 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:36:34.534 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addrs": { 2021-02-10T11:36:34.534 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:36:34.534 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:36:34.534 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:36:34.534 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6812", 2021-02-10T11:36:34.534 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36366 2021-02-10T11:36:34.534 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:36:34.534 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:36:34.535 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:36:34.535 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6815", 2021-02-10T11:36:34.535 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36366 2021-02-10T11:36:34.535 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:36:34.535 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:36:34.535 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:36:34.535 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addr": "172.21.2.113:6805/36366", 2021-02-10T11:36:34.535 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addr": "172.21.2.113:6810/36366", 2021-02-10T11:36:34.535 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addr": "172.21.2.113:6822/36366", 2021-02-10T11:36:34.535 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addr": "172.21.2.113:6815/36366", 2021-02-10T11:36:34.535 INFO:teuthology.orchestra.run.gibba002.stdout: "state": [ 2021-02-10T11:36:34.535 INFO:teuthology.orchestra.run.gibba002.stdout: "exists", 2021-02-10T11:36:34.535 INFO:teuthology.orchestra.run.gibba002.stdout: "up" 2021-02-10T11:36:34.535 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:36:34.536 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:36:34.536 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:36:34.536 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 6, 2021-02-10T11:36:34.536 INFO:teuthology.orchestra.run.gibba002.stdout: "uuid": "bf67c040-0e50-48be-b102-c3943bc9ebef", 2021-02-10T11:36:34.536 INFO:teuthology.orchestra.run.gibba002.stdout: "up": 1, 2021-02-10T11:36:34.536 INFO:teuthology.orchestra.run.gibba002.stdout: "in": 1, 2021-02-10T11:36:34.536 INFO:teuthology.orchestra.run.gibba002.stdout: "weight": 1, 2021-02-10T11:36:34.536 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_affinity": 1, 2021-02-10T11:36:34.536 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_begin": 0, 2021-02-10T11:36:34.536 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_end": 0, 2021-02-10T11:36:34.536 INFO:teuthology.orchestra.run.gibba002.stdout: "up_from": 13, 2021-02-10T11:36:34.536 INFO:teuthology.orchestra.run.gibba002.stdout: "up_thru": 150, 2021-02-10T11:36:34.536 INFO:teuthology.orchestra.run.gibba002.stdout: "down_at": 0, 2021-02-10T11:36:34.536 INFO:teuthology.orchestra.run.gibba002.stdout: "lost_at": 0, 2021-02-10T11:36:34.537 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addrs": { 2021-02-10T11:36:34.537 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:36:34.537 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:36:34.537 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:36:34.537 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6800", 2021-02-10T11:36:34.537 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36367 2021-02-10T11:36:34.537 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:36:34.537 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:36:34.537 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:36:34.537 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6801", 2021-02-10T11:36:34.537 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36367 2021-02-10T11:36:34.537 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:36:34.537 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:36:34.537 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:36:34.538 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addrs": { 2021-02-10T11:36:34.538 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:36:34.538 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:36:34.538 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:36:34.538 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6803", 2021-02-10T11:36:34.538 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36367 2021-02-10T11:36:34.538 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:36:34.538 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:36:34.538 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:36:34.538 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6804", 2021-02-10T11:36:34.538 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36367 2021-02-10T11:36:34.538 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:36:34.538 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:36:34.539 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:36:34.539 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addrs": { 2021-02-10T11:36:34.539 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:36:34.539 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:36:34.539 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:36:34.539 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6813", 2021-02-10T11:36:34.539 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36367 2021-02-10T11:36:34.539 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:36:34.539 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:36:34.539 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:36:34.539 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6816", 2021-02-10T11:36:34.539 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36367 2021-02-10T11:36:34.539 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:36:34.540 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:36:34.540 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:36:34.540 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addrs": { 2021-02-10T11:36:34.540 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:36:34.540 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:36:34.540 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:36:34.540 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6806", 2021-02-10T11:36:34.540 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36367 2021-02-10T11:36:34.540 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:36:34.540 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:36:34.540 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:36:34.540 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6809", 2021-02-10T11:36:34.540 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36367 2021-02-10T11:36:34.540 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:36:34.541 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:36:34.541 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:36:34.541 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addr": "172.21.2.113:6801/36367", 2021-02-10T11:36:34.541 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addr": "172.21.2.113:6804/36367", 2021-02-10T11:36:34.541 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addr": "172.21.2.113:6816/36367", 2021-02-10T11:36:34.541 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addr": "172.21.2.113:6809/36367", 2021-02-10T11:36:34.541 INFO:teuthology.orchestra.run.gibba002.stdout: "state": [ 2021-02-10T11:36:34.541 INFO:teuthology.orchestra.run.gibba002.stdout: "exists", 2021-02-10T11:36:34.541 INFO:teuthology.orchestra.run.gibba002.stdout: "up" 2021-02-10T11:36:34.541 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:36:34.541 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:36:34.541 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:36:34.541 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 7, 2021-02-10T11:36:34.541 INFO:teuthology.orchestra.run.gibba002.stdout: "uuid": "f5461f80-018d-428f-be89-2eb2bcbe7bb6", 2021-02-10T11:36:34.542 INFO:teuthology.orchestra.run.gibba002.stdout: "up": 1, 2021-02-10T11:36:34.542 INFO:teuthology.orchestra.run.gibba002.stdout: "in": 1, 2021-02-10T11:36:34.542 INFO:teuthology.orchestra.run.gibba002.stdout: "weight": 1, 2021-02-10T11:36:34.542 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_affinity": 1, 2021-02-10T11:36:34.542 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_begin": 0, 2021-02-10T11:36:34.542 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_end": 0, 2021-02-10T11:36:34.542 INFO:teuthology.orchestra.run.gibba002.stdout: "up_from": 13, 2021-02-10T11:36:34.542 INFO:teuthology.orchestra.run.gibba002.stdout: "up_thru": 142, 2021-02-10T11:36:34.542 INFO:teuthology.orchestra.run.gibba002.stdout: "down_at": 0, 2021-02-10T11:36:34.542 INFO:teuthology.orchestra.run.gibba002.stdout: "lost_at": 0, 2021-02-10T11:36:34.542 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addrs": { 2021-02-10T11:36:34.542 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:36:34.542 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:36:34.543 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:36:34.543 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6817", 2021-02-10T11:36:34.543 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36365 2021-02-10T11:36:34.543 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:36:34.543 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:36:34.543 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:36:34.543 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6821", 2021-02-10T11:36:34.543 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36365 2021-02-10T11:36:34.543 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:36:34.543 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:36:34.543 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:36:34.543 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addrs": { 2021-02-10T11:36:34.543 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:36:34.543 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:36:34.544 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:36:34.544 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6823", 2021-02-10T11:36:34.544 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36365 2021-02-10T11:36:34.544 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:36:34.544 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:36:34.544 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:36:34.544 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6825", 2021-02-10T11:36:34.544 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36365 2021-02-10T11:36:34.544 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:36:34.544 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:36:34.544 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:36:34.544 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addrs": { 2021-02-10T11:36:34.544 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:36:34.544 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:36:34.545 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:36:34.545 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6830", 2021-02-10T11:36:34.545 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36365 2021-02-10T11:36:34.545 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:36:34.545 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:36:34.545 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:36:34.545 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6831", 2021-02-10T11:36:34.545 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36365 2021-02-10T11:36:34.545 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:36:34.545 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:36:34.545 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:36:34.545 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addrs": { 2021-02-10T11:36:34.545 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:36:34.545 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:36:34.546 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:36:34.546 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6827", 2021-02-10T11:36:34.546 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36365 2021-02-10T11:36:34.546 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:36:34.546 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:36:34.546 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:36:34.546 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6829", 2021-02-10T11:36:34.546 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36365 2021-02-10T11:36:34.546 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:36:34.546 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:36:34.546 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:36:34.546 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addr": "172.21.2.113:6821/36365", 2021-02-10T11:36:34.546 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addr": "172.21.2.113:6825/36365", 2021-02-10T11:36:34.547 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addr": "172.21.2.113:6831/36365", 2021-02-10T11:36:34.547 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addr": "172.21.2.113:6829/36365", 2021-02-10T11:36:34.547 INFO:teuthology.orchestra.run.gibba002.stdout: "state": [ 2021-02-10T11:36:34.547 INFO:teuthology.orchestra.run.gibba002.stdout: "exists", 2021-02-10T11:36:34.547 INFO:teuthology.orchestra.run.gibba002.stdout: "up" 2021-02-10T11:36:34.547 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:36:34.547 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:36:34.547 INFO:teuthology.orchestra.run.gibba002.stdout: ], 2021-02-10T11:36:34.547 INFO:teuthology.orchestra.run.gibba002.stdout: "osd_xinfo": [ 2021-02-10T11:36:34.547 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:36:34.547 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 0, 2021-02-10T11:36:34.547 INFO:teuthology.orchestra.run.gibba002.stdout: "down_stamp": "0.000000", 2021-02-10T11:36:34.547 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_probability": 0, 2021-02-10T11:36:34.547 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_interval": 0, 2021-02-10T11:36:34.548 INFO:teuthology.orchestra.run.gibba002.stdout: "features": 4611087854035861503, 2021-02-10T11:36:34.548 INFO:teuthology.orchestra.run.gibba002.stdout: "old_weight": 0 2021-02-10T11:36:34.548 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:36:34.548 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:36:34.548 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 1, 2021-02-10T11:36:34.548 INFO:teuthology.orchestra.run.gibba002.stdout: "down_stamp": "0.000000", 2021-02-10T11:36:34.548 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_probability": 0, 2021-02-10T11:36:34.548 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_interval": 0, 2021-02-10T11:36:34.548 INFO:teuthology.orchestra.run.gibba002.stdout: "features": 4611087854035861503, 2021-02-10T11:36:34.548 INFO:teuthology.orchestra.run.gibba002.stdout: "old_weight": 0 2021-02-10T11:36:34.548 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:36:34.548 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:36:34.548 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 2, 2021-02-10T11:36:34.548 INFO:teuthology.orchestra.run.gibba002.stdout: "down_stamp": "0.000000", 2021-02-10T11:36:34.549 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_probability": 0, 2021-02-10T11:36:34.549 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_interval": 0, 2021-02-10T11:36:34.549 INFO:teuthology.orchestra.run.gibba002.stdout: "features": 4611087854035861503, 2021-02-10T11:36:34.549 INFO:teuthology.orchestra.run.gibba002.stdout: "old_weight": 0 2021-02-10T11:36:34.549 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:36:34.549 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:36:34.549 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 3, 2021-02-10T11:36:34.549 INFO:teuthology.orchestra.run.gibba002.stdout: "down_stamp": "0.000000", 2021-02-10T11:36:34.549 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_probability": 0, 2021-02-10T11:36:34.549 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_interval": 0, 2021-02-10T11:36:34.549 INFO:teuthology.orchestra.run.gibba002.stdout: "features": 4611087854035861503, 2021-02-10T11:36:34.549 INFO:teuthology.orchestra.run.gibba002.stdout: "old_weight": 0 2021-02-10T11:36:34.549 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:36:34.549 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:36:34.550 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 4, 2021-02-10T11:36:34.550 INFO:teuthology.orchestra.run.gibba002.stdout: "down_stamp": "0.000000", 2021-02-10T11:36:34.550 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_probability": 0, 2021-02-10T11:36:34.550 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_interval": 0, 2021-02-10T11:36:34.550 INFO:teuthology.orchestra.run.gibba002.stdout: "features": 4611087854035861503, 2021-02-10T11:36:34.550 INFO:teuthology.orchestra.run.gibba002.stdout: "old_weight": 0 2021-02-10T11:36:34.550 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:36:34.550 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:36:34.550 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 5, 2021-02-10T11:36:34.550 INFO:teuthology.orchestra.run.gibba002.stdout: "down_stamp": "0.000000", 2021-02-10T11:36:34.550 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_probability": 0, 2021-02-10T11:36:34.550 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_interval": 0, 2021-02-10T11:36:34.550 INFO:teuthology.orchestra.run.gibba002.stdout: "features": 4611087854035861503, 2021-02-10T11:36:34.551 INFO:teuthology.orchestra.run.gibba002.stdout: "old_weight": 0 2021-02-10T11:36:34.551 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:36:34.551 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:36:34.551 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 6, 2021-02-10T11:36:34.551 INFO:teuthology.orchestra.run.gibba002.stdout: "down_stamp": "0.000000", 2021-02-10T11:36:34.551 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_probability": 0, 2021-02-10T11:36:34.551 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_interval": 0, 2021-02-10T11:36:34.551 INFO:teuthology.orchestra.run.gibba002.stdout: "features": 4611087854035861503, 2021-02-10T11:36:34.551 INFO:teuthology.orchestra.run.gibba002.stdout: "old_weight": 0 2021-02-10T11:36:34.551 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:36:34.551 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:36:34.551 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 7, 2021-02-10T11:36:34.551 INFO:teuthology.orchestra.run.gibba002.stdout: "down_stamp": "0.000000", 2021-02-10T11:36:34.551 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_probability": 0, 2021-02-10T11:36:34.552 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_interval": 0, 2021-02-10T11:36:34.552 INFO:teuthology.orchestra.run.gibba002.stdout: "features": 4611087854035861503, 2021-02-10T11:36:34.552 INFO:teuthology.orchestra.run.gibba002.stdout: "old_weight": 0 2021-02-10T11:36:34.552 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:36:34.552 INFO:teuthology.orchestra.run.gibba002.stdout: ], 2021-02-10T11:36:34.552 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_upmap": [], 2021-02-10T11:36:34.552 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_upmap_items": [], 2021-02-10T11:36:34.552 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_temp": [], 2021-02-10T11:36:34.552 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_temp": [], 2021-02-10T11:36:34.552 INFO:teuthology.orchestra.run.gibba002.stdout: "blacklist": {}, 2021-02-10T11:36:34.552 INFO:teuthology.orchestra.run.gibba002.stdout: "erasure_code_profiles": { 2021-02-10T11:36:34.552 INFO:teuthology.orchestra.run.gibba002.stdout: "default": { 2021-02-10T11:36:34.552 INFO:teuthology.orchestra.run.gibba002.stdout: "crush-failure-domain": "osd", 2021-02-10T11:36:34.552 INFO:teuthology.orchestra.run.gibba002.stdout: "k": "2", 2021-02-10T11:36:34.553 INFO:teuthology.orchestra.run.gibba002.stdout: "m": "1", 2021-02-10T11:36:34.553 INFO:teuthology.orchestra.run.gibba002.stdout: "plugin": "jerasure", 2021-02-10T11:36:34.553 INFO:teuthology.orchestra.run.gibba002.stdout: "ruleset-failure-domain": "osd", 2021-02-10T11:36:34.553 INFO:teuthology.orchestra.run.gibba002.stdout: "technique": "reed_sol_van" 2021-02-10T11:36:34.553 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:36:34.553 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:36:34.553 INFO:teuthology.orchestra.run.gibba002.stdout: "removed_snaps_queue": [], 2021-02-10T11:36:34.553 INFO:teuthology.orchestra.run.gibba002.stdout: "new_removed_snaps": [], 2021-02-10T11:36:34.553 INFO:teuthology.orchestra.run.gibba002.stdout: "new_purged_snaps": [], 2021-02-10T11:36:34.553 INFO:teuthology.orchestra.run.gibba002.stdout: "crush_node_flags": {}, 2021-02-10T11:36:34.553 INFO:teuthology.orchestra.run.gibba002.stdout: "device_class_flags": {} 2021-02-10T11:36:34.553 INFO:teuthology.orchestra.run.gibba002.stdout:} 2021-02-10T11:36:34.554 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:34.463 7f7933929700 1 -- 172.21.2.102:0/103158089 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f790c0186c0 msgr2=0x7f790c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:34.555 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:34.463 7f7933929700 1 --2- 172.21.2.102:0/103158089 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f790c0186c0 0x7f790c01abc0 crc :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:34.555 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:34.463 7f7933929700 1 -- 172.21.2.102:0/103158089 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f792c195650 msgr2=0x7f792c0b5560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:34.555 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:34.463 7f7933929700 1 --2- 172.21.2.102:0/103158089 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f792c195650 0x7f792c0b5560 secure :-1 s=READY pgs=2829 cs=0 l=1 rev1=1 rx=0x7f792800b5e0 tx=0x7f792800b980).stop 2021-02-10T11:36:34.555 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:34.463 7f7933929700 1 -- 172.21.2.102:0/103158089 shutdown_connections 2021-02-10T11:36:34.555 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:34.463 7f7933929700 1 --2- 172.21.2.102:0/103158089 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f790c0186c0 0x7f790c01abc0 unknown :-1 s=CLOSED pgs=1241 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:34.555 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:34.463 7f7933929700 1 --2- 172.21.2.102:0/103158089 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f792c0acfa0 0x7f792c1a3440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:34.555 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:34.463 7f7933929700 1 --2- 172.21.2.102:0/103158089 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f792c0b5b80 0x7f792c1a2e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:34.555 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:34.463 7f7933929700 1 --2- 172.21.2.102:0/103158089 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f792c195650 0x7f792c0b5560 unknown :-1 s=CLOSED pgs=2829 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:34.555 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:34.463 7f7933929700 1 -- 172.21.2.102:0/103158089 >> 172.21.2.102:0/103158089 conn(0x7f792c0b3040 msgr2=0x7f792c0c3250 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:34.555 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:34.463 7f7933929700 1 -- 172.21.2.102:0/103158089 shutdown_connections 2021-02-10T11:36:34.555 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:34.463 7f7933929700 1 -- 172.21.2.102:0/103158089 wait complete. 2021-02-10T11:36:34.556 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:36:34.987 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:34.967 7f2b6b022700 1 Processor -- start 2021-02-10T11:36:34.987 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:34.967 7f2b6b022700 1 -- start start 2021-02-10T11:36:34.987 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:34.967 7f2b6b022700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f2b640d1870 0x7f2b641b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:34.987 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:34.967 7f2b6b022700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2b641b0a10 0x7f2b640cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:34.988 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:34.967 7f2b6b022700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b640f0b10 con 0x7f2b640d1c70 2021-02-10T11:36:34.988 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:34.967 7f2b6b022700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b640e5ed0 con 0x7f2b640d1870 2021-02-10T11:36:34.988 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:34.967 7f2b6b022700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b640f0650 con 0x7f2b641b0a10 2021-02-10T11:36:34.988 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:34.969 7f2b6963f700 1 --1- >> v1:172.21.2.113:6789/0 conn(0x7f2b640d1c70 0x7f2b640cf950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.113:6789/0 says I am v1:172.21.2.102:37412/0 (socket says 172.21.2.102:37412) 2021-02-10T11:36:34.988 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:34.969 7f2b63fff700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2b641b0a10 0x7f2b640cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:34.988 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:34.969 7f2b6963f700 1 -- 172.21.2.102:0/2918561264 learned_addr learned my addr 172.21.2.102:0/2918561264 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:36:34.988 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:34.969 7f2b68e3e700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f2b640d1870 0x7f2b641b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:34.988 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:34.969 7f2b63fff700 1 -- 172.21.2.102:0/2918561264 >> v2:172.21.2.113:3300/0 conn(0x7f2b640d1870 msgr2=0x7f2b641b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:34.988 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:34.969 7f2b63fff700 1 --2- 172.21.2.102:0/2918561264 >> v2:172.21.2.113:3300/0 conn(0x7f2b640d1870 0x7f2b641b03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:34.988 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:34.969 7f2b637fe700 1 -- 172.21.2.102:0/2918561264 <== mon.1 v1:172.21.2.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2067161032 0 0) 0x7f2b5c0010c0 con 0x7f2b640d1c70 2021-02-10T11:36:34.988 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:34.969 7f2b63fff700 1 -- 172.21.2.102:0/2918561264 >> v1:172.21.2.113:6789/0 conn(0x7f2b640d1c70 legacy=0x7f2b640cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:34.988 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:34.969 7f2b63fff700 1 -- 172.21.2.102:0/2918561264 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b640f0870 con 0x7f2b641b0a10 2021-02-10T11:36:34.989 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:34.969 7f2b68e3e700 1 --2- 172.21.2.102:0/2918561264 >> v2:172.21.2.113:3300/0 conn(0x7f2b640d1870 0x7f2b641b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:36:34.989 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:34.970 7f2b63fff700 1 --2- 172.21.2.102:0/2918561264 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2b641b0a10 0x7f2b640cf330 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 rx=0x7f2b540069d0 tx=0x7f2b54009e40).ready entity=mon.2 client_cookie=287a61a1d3568a00 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:34.989 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:34.970 7f2b637fe700 1 -- 172.21.2.102:0/2918561264 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2b5400b510 con 0x7f2b641b0a10 2021-02-10T11:36:34.989 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:34.970 7f2b637fe700 1 -- 172.21.2.102:0/2918561264 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b5400b890 con 0x7f2b641b0a10 2021-02-10T11:36:34.989 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:34.970 7f2b637fe700 1 -- 172.21.2.102:0/2918561264 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2b5400bdb0 con 0x7f2b641b0a10 2021-02-10T11:36:34.989 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:34.970 7f2b6b022700 1 -- 172.21.2.102:0/2918561264 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2b641b0a10 msgr2=0x7f2b640cf330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:34.989 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:34.970 7f2b6b022700 1 --2- 172.21.2.102:0/2918561264 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2b641b0a10 0x7f2b640cf330 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 rx=0x7f2b540069d0 tx=0x7f2b54009e40).stop 2021-02-10T11:36:34.989 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:34.971 7f2b6b022700 1 -- 172.21.2.102:0/2918561264 shutdown_connections 2021-02-10T11:36:34.989 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:34.971 7f2b6b022700 1 --2- 172.21.2.102:0/2918561264 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2b641b0a10 0x7f2b640cf330 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:34.989 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:34.971 7f2b6b022700 1 --2- 172.21.2.102:0/2918561264 >> v2:172.21.2.113:3300/0 conn(0x7f2b640d1870 0x7f2b641b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:34.989 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:34.971 7f2b6b022700 1 -- 172.21.2.102:0/2918561264 >> 172.21.2.102:0/2918561264 conn(0x7f2b640ce450 msgr2=0x7f2b640d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:34.989 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:34.971 7f2b6b022700 1 -- 172.21.2.102:0/2918561264 shutdown_connections 2021-02-10T11:36:34.989 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:34.971 7f2b6b022700 1 -- 172.21.2.102:0/2918561264 wait complete. 2021-02-10T11:36:34.991 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:34.985 7f2b6b022700 1 Processor -- start 2021-02-10T11:36:34.991 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:34.985 7f2b6b022700 1 -- start start 2021-02-10T11:36:34.991 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:34.986 7f2b6b022700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2b64195650 0x7f2b640b5560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:34.991 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:34.986 7f2b6b022700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2b640b5b80 0x7f2b641a2e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:34.991 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:34.986 7f2b6b022700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2b640acfa0 0x7f2b641a3440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:34.991 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:34.986 7f2b6b022700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b640b6010 con 0x7f2b640acfa0 2021-02-10T11:36:34.991 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:34.986 7f2b6b022700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b641a3a60 con 0x7f2b64195650 2021-02-10T11:36:34.991 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:34.986 7f2b6b022700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b641a3c40 con 0x7f2b640b5b80 2021-02-10T11:36:34.992 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:34.986 7f2b6963f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2b640acfa0 0x7f2b641a3440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:34.992 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:34.986 7f2b6963f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2b640acfa0 0x7f2b641a3440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:37866/0 (socket says 172.21.2.102:37866) 2021-02-10T11:36:34.992 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:34.986 7f2b6963f700 1 -- 172.21.2.102:0/399047248 learned_addr learned my addr 172.21.2.102:0/399047248 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:36:34.992 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:34.986 7f2b68e3e700 1 --2- 172.21.2.102:0/399047248 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2b64195650 0x7f2b640b5560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:34.992 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:34.986 7f2b63fff700 1 --2- 172.21.2.102:0/399047248 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2b640b5b80 0x7f2b641a2e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:34.992 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:34.987 7f2b6963f700 1 -- 172.21.2.102:0/399047248 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2b640b5b80 msgr2=0x7f2b641a2e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:34.992 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:34.987 7f2b6963f700 1 --2- 172.21.2.102:0/399047248 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2b640b5b80 0x7f2b641a2e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:34.992 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:34.987 7f2b6963f700 1 -- 172.21.2.102:0/399047248 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2b64195650 msgr2=0x7f2b640b5560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:34.992 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:34.987 7f2b6963f700 1 --2- 172.21.2.102:0/399047248 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2b64195650 0x7f2b640b5560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:34.992 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:34.987 7f2b6963f700 1 -- 172.21.2.102:0/399047248 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b640bcae0 con 0x7f2b640acfa0 2021-02-10T11:36:34.992 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:34.987 7f2b6963f700 1 --2- 172.21.2.102:0/399047248 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2b640acfa0 0x7f2b641a3440 secure :-1 s=READY pgs=2830 cs=0 l=1 rev1=1 rx=0x7f2b5c00a590 tx=0x7f2b5c00a930).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:34.992 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:34.987 7f2b617fa700 1 -- 172.21.2.102:0/399047248 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2b5c000eb0 con 0x7f2b640acfa0 2021-02-10T11:36:34.992 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:34.987 7f2b617fa700 1 -- 172.21.2.102:0/399047248 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b5c00ad20 con 0x7f2b640acfa0 2021-02-10T11:36:34.993 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:34.987 7f2b617fa700 1 -- 172.21.2.102:0/399047248 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2b5c0070d0 con 0x7f2b640acfa0 2021-02-10T11:36:34.993 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:34.987 7f2b6b022700 1 -- 172.21.2.102:0/399047248 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b640bcae0 con 0x7f2b640acfa0 2021-02-10T11:36:34.993 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:34.987 7f2b6b022700 1 -- 172.21.2.102:0/399047248 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b640bcae0 con 0x7f2b640acfa0 2021-02-10T11:36:34.993 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:34.988 7f2b617fa700 1 -- 172.21.2.102:0/399047248 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f2b5c004320 con 0x7f2b640acfa0 2021-02-10T11:36:34.993 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:34.988 7f2b617fa700 1 --2- 172.21.2.102:0/399047248 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2b480186c0 0x7f2b4801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:34.993 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:34.988 7f2b617fa700 1 -- 172.21.2.102:0/399047248 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(153..153 src has 1..153) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f2b5c032560 con 0x7f2b640acfa0 2021-02-10T11:36:34.993 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:34.988 7f2b68e3e700 1 --2- 172.21.2.102:0/399047248 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2b480186c0 0x7f2b4801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:34.993 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:34.988 7f2b68e3e700 1 --2- 172.21.2.102:0/399047248 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2b480186c0 0x7f2b4801abc0 crc :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:35.019 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:35.017 7f2b6b022700 1 -- 172.21.2.102:0/399047248 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b640b6010 con 0x7f2b640acfa0 2021-02-10T11:36:35.022 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:35.020 7f2b617fa700 1 -- 172.21.2.102:0/399047248 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f2b5c000eb0 con 0x7f2b640acfa0 2021-02-10T11:36:35.149 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:35.147 7f2b6b022700 1 -- 172.21.2.102:0/399047248 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2b640b6010 con 0x7f2b640acfa0 2021-02-10T11:36:35.149 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:35.147 7f2b617fa700 1 -- 172.21.2.102:0/399047248 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 133 v133) v1 ==== 95+0+2107 (secure 0 0 0) 0x7f2b5c004320 con 0x7f2b640acfa0 2021-02-10T11:36:35.150 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:36:35.150 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":133,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7004,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.102:6835/3280534775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6834","nonce":3280534775},{"type":"v1","addr":"172.21.2.102:6835","nonce":3280534775}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":128}],"filesystems":[{"mdsmap":{"epoch":132,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:35:58.285904","modified":"2021-02-10 11:36:00.292711","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7001},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7001":{"gid":7001,"name":"b","rank":0,"incarnation":129,"state":"up:active","state_seq":3,"addr":"172.21.2.113:6833/1983258405","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6832","nonce":1983258405},{"type":"v1","addr":"172.21.2.113:6833","nonce":1983258405}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[33],"metadata_pool":32,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":16}]} 2021-02-10T11:36:35.150 INFO:teuthology.orchestra.run.gibba002.stderr:dumped fsmap epoch 133 2021-02-10T11:36:35.151 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:35.149 7f2b6b022700 1 -- 172.21.2.102:0/399047248 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2b480186c0 msgr2=0x7f2b4801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:35.151 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:35.149 7f2b6b022700 1 --2- 172.21.2.102:0/399047248 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2b480186c0 0x7f2b4801abc0 crc :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:35.151 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:35.149 7f2b6b022700 1 -- 172.21.2.102:0/399047248 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2b640acfa0 msgr2=0x7f2b641a3440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:35.151 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:35.149 7f2b6b022700 1 --2- 172.21.2.102:0/399047248 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2b640acfa0 0x7f2b641a3440 secure :-1 s=READY pgs=2830 cs=0 l=1 rev1=1 rx=0x7f2b5c00a590 tx=0x7f2b5c00a930).stop 2021-02-10T11:36:35.152 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:35.149 7f2b6b022700 1 -- 172.21.2.102:0/399047248 shutdown_connections 2021-02-10T11:36:35.152 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:35.149 7f2b6b022700 1 --2- 172.21.2.102:0/399047248 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2b640acfa0 0x7f2b641a3440 unknown :-1 s=CLOSED pgs=2830 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:35.152 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:35.149 7f2b6b022700 1 --2- 172.21.2.102:0/399047248 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2b640b5b80 0x7f2b641a2e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:35.152 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:35.149 7f2b6b022700 1 --2- 172.21.2.102:0/399047248 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2b480186c0 0x7f2b4801abc0 unknown :-1 s=CLOSED pgs=1242 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:35.152 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:35.149 7f2b6b022700 1 --2- 172.21.2.102:0/399047248 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2b64195650 0x7f2b640b5560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:35.152 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:35.149 7f2b6b022700 1 -- 172.21.2.102:0/399047248 >> 172.21.2.102:0/399047248 conn(0x7f2b640b3040 msgr2=0x7f2b640c3250 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:35.152 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:35.149 7f2b6b022700 1 -- 172.21.2.102:0/399047248 shutdown_connections 2021-02-10T11:36:35.152 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:35.149 7f2b6b022700 1 -- 172.21.2.102:0/399047248 wait complete. 2021-02-10T11:36:35.160 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs fail cephfs 2021-02-10T11:36:35.588 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:35.570 7f4dcae10700 1 Processor -- start 2021-02-10T11:36:35.588 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:35.570 7f4dcae10700 1 -- start start 2021-02-10T11:36:35.589 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:35.570 7f4dcae10700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f4dc40d1870 0x7f4dc41b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:35.589 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:35.570 7f4dcae10700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4dc40f0650 0x7f4dc40cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:35.589 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:35.570 7f4dcae10700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4dc40f0b10 con 0x7f4dc41b0a10 2021-02-10T11:36:35.589 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:35.570 7f4dcae10700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4dc40e5ed0 con 0x7f4dc40d1870 2021-02-10T11:36:35.589 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:35.570 7f4dcae10700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4dc40d1c70 con 0x7f4dc40f0650 2021-02-10T11:36:35.589 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:35.570 7f4dc8c2c700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f4dc40d1870 0x7f4dc41b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:35.589 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:35.570 7f4dc8c2c700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f4dc40d1870 0x7f4dc41b03f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:37876/0 (socket says 172.21.2.102:37876) 2021-02-10T11:36:35.589 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:35.570 7f4dc8c2c700 1 -- 172.21.2.102:0/2350584337 learned_addr learned my addr 172.21.2.102:0/2350584337 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:36:35.589 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:35.571 7f4dc8c2c700 1 -- 172.21.2.102:0/2350584337 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4dc40f0650 msgr2=0x7f4dc40cfb20 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:36:35.589 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:35.571 7f4dc8c2c700 1 --2- 172.21.2.102:0/2350584337 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4dc40f0650 0x7f4dc40cfb20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:35.589 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:35.571 7f4dc8c2c700 1 -- 172.21.2.102:0/2350584337 >> v1:172.21.2.113:6789/0 conn(0x7f4dc41b0a10 legacy=0x7f4dc40cf330 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:36:35.589 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:35.571 7f4dc8c2c700 1 -- 172.21.2.102:0/2350584337 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dc40d1e90 con 0x7f4dc40d1870 2021-02-10T11:36:35.589 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:35.571 7f4dc8c2c700 1 --2- 172.21.2.102:0/2350584337 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4dc40d1870 0x7f4dc41b03f0 secure :-1 s=READY pgs=2831 cs=0 l=1 rev1=1 rx=0x7f4dac006700 tx=0x7f4dac009b00).ready entity=mon.0 client_cookie=2ba2e946fa777395 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:35.590 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:35.571 7f4dc37fe700 1 -- 172.21.2.102:0/2350584337 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4dac00c0c0 con 0x7f4dc40d1870 2021-02-10T11:36:35.590 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:35.571 7f4dc37fe700 1 -- 172.21.2.102:0/2350584337 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4dac00c340 con 0x7f4dc40d1870 2021-02-10T11:36:35.590 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:35.571 7f4dc37fe700 1 -- 172.21.2.102:0/2350584337 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4dac00c820 con 0x7f4dc40d1870 2021-02-10T11:36:35.590 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:35.571 7f4dcae10700 1 -- 172.21.2.102:0/2350584337 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4dc40d1870 msgr2=0x7f4dc41b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:35.590 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:35.571 7f4dcae10700 1 --2- 172.21.2.102:0/2350584337 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4dc40d1870 0x7f4dc41b03f0 secure :-1 s=READY pgs=2831 cs=0 l=1 rev1=1 rx=0x7f4dac006700 tx=0x7f4dac009b00).stop 2021-02-10T11:36:35.590 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:35.572 7f4dcae10700 1 -- 172.21.2.102:0/2350584337 shutdown_connections 2021-02-10T11:36:35.590 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:35.572 7f4dcae10700 1 --2- 172.21.2.102:0/2350584337 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4dc40f0650 0x7f4dc40cfb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:35.590 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:35.572 7f4dcae10700 1 --2- 172.21.2.102:0/2350584337 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4dc40d1870 0x7f4dc41b03f0 unknown :-1 s=CLOSED pgs=2831 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:35.590 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:35.572 7f4dcae10700 1 -- 172.21.2.102:0/2350584337 >> 172.21.2.102:0/2350584337 conn(0x7f4dc40ce450 msgr2=0x7f4dc40d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:35.590 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:35.572 7f4dcae10700 1 -- 172.21.2.102:0/2350584337 shutdown_connections 2021-02-10T11:36:35.590 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:35.572 7f4dcae10700 1 -- 172.21.2.102:0/2350584337 wait complete. 2021-02-10T11:36:35.592 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:35.587 7f4dcae10700 1 Processor -- start 2021-02-10T11:36:35.592 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:35.587 7f4dcae10700 1 -- start start 2021-02-10T11:36:35.592 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:35.587 7f4dcae10700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4dc40bdda0 0x7f4dc418b250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:35.592 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:35.587 7f4dcae10700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4dc418b870 0x7f4dc40c0030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:35.592 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:35.587 7f4dcae10700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4dc418bc70 0x7f4dc40c0650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:35.592 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:35.587 7f4dcae10700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dc419d390 con 0x7f4dc40bdda0 2021-02-10T11:36:35.592 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:35.587 7f4dcae10700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dc40bd740 con 0x7f4dc418bc70 2021-02-10T11:36:35.592 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:35.587 7f4dcae10700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4dc40be1a0 con 0x7f4dc418b870 2021-02-10T11:36:35.592 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:35.587 7f4dc8c2c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4dc40bdda0 0x7f4dc418b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:35.592 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:35.587 7f4dc8c2c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4dc40bdda0 0x7f4dc418b250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:37882/0 (socket says 172.21.2.102:37882) 2021-02-10T11:36:35.592 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:35.587 7f4dc8c2c700 1 -- 172.21.2.102:0/3853171754 learned_addr learned my addr 172.21.2.102:0/3853171754 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:36:35.592 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:35.588 7f4dc8c2c700 1 -- 172.21.2.102:0/3853171754 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4dc418b870 msgr2=0x7f4dc40c0030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:35.593 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:35.588 7f4dc8c2c700 1 --2- 172.21.2.102:0/3853171754 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4dc418b870 0x7f4dc40c0030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:35.593 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:35.588 7f4dc8c2c700 1 -- 172.21.2.102:0/3853171754 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4dc418bc70 msgr2=0x7f4dc40c0650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:35.593 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:35.588 7f4dc8c2c700 1 --2- 172.21.2.102:0/3853171754 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4dc418bc70 0x7f4dc40c0650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:35.593 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:35.588 7f4dc8c2c700 1 -- 172.21.2.102:0/3853171754 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dc40c0c70 con 0x7f4dc40bdda0 2021-02-10T11:36:35.593 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:35.588 7f4dc8c2c700 1 --2- 172.21.2.102:0/3853171754 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4dc40bdda0 0x7f4dc418b250 secure :-1 s=READY pgs=2832 cs=0 l=1 rev1=1 rx=0x7f4dac006f20 tx=0x7f4dac00b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:35.593 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:35.588 7f4dc17fa700 1 -- 172.21.2.102:0/3853171754 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4dac009100 con 0x7f4dc40bdda0 2021-02-10T11:36:35.593 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:35.588 7f4dc17fa700 1 -- 172.21.2.102:0/3853171754 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4dac00bd80 con 0x7f4dc40bdda0 2021-02-10T11:36:35.593 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:35.588 7f4dc17fa700 1 -- 172.21.2.102:0/3853171754 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4dac005560 con 0x7f4dc40bdda0 2021-02-10T11:36:35.593 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:35.588 7f4dcae10700 1 -- 172.21.2.102:0/3853171754 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4dc40c0c70 con 0x7f4dc40bdda0 2021-02-10T11:36:35.593 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:35.588 7f4dcae10700 1 -- 172.21.2.102:0/3853171754 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4dc40c1200 con 0x7f4dc40bdda0 2021-02-10T11:36:35.593 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:35.589 7f4dc17fa700 1 -- 172.21.2.102:0/3853171754 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4dac007000 con 0x7f4dc40bdda0 2021-02-10T11:36:35.593 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:35.589 7f4dc17fa700 1 --2- 172.21.2.102:0/3853171754 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4da80186c0 0x7f4da801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:35.593 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:35.589 7f4dc17fa700 1 -- 172.21.2.102:0/3853171754 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(153..153 src has 1..153) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f4dac032550 con 0x7f4dc40bdda0 2021-02-10T11:36:35.594 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:35.589 7f4dc3fff700 1 --2- 172.21.2.102:0/3853171754 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4da80186c0 0x7f4da801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:35.594 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:35.589 7f4dc3fff700 1 --2- 172.21.2.102:0/3853171754 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4da80186c0 0x7f4da801abc0 crc :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:35.622 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:35.620 7f4dcae10700 1 -- 172.21.2.102:0/3853171754 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4dc40ee0b0 con 0x7f4dc40bdda0 2021-02-10T11:36:35.625 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:35.623 7f4dc17fa700 1 -- 172.21.2.102:0/3853171754 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f4dac009100 con 0x7f4dc40bdda0 2021-02-10T11:36:35.754 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:35.752 7f4dcae10700 1 -- 172.21.2.102:0/3853171754 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f4dc40ee0b0 con 0x7f4dc40bdda0 2021-02-10T11:36:36.162 INFO:tasks.ceph.mon.a.gibba002.stderr:2021-02-10 11:36:36.160 7f8dd2755700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-02-10T11:36:36.169 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:36.167 7f4dc17fa700 1 -- 172.21.2.102:0/3853171754 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v134) v1 ==== 161+0+0 (secure 0 0 0) 0x7f4dac007320 con 0x7f4dc40bdda0 2021-02-10T11:36:36.169 INFO:teuthology.orchestra.run.gibba002.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2021-02-10T11:36:36.171 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:36.169 7f4dcae10700 1 -- 172.21.2.102:0/3853171754 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4da80186c0 msgr2=0x7f4da801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:36.171 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:36.169 7f4dcae10700 1 --2- 172.21.2.102:0/3853171754 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4da80186c0 0x7f4da801abc0 crc :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:36.171 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:36.169 7f4dcae10700 1 -- 172.21.2.102:0/3853171754 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4dc40bdda0 msgr2=0x7f4dc418b250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:36.171 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:36.169 7f4dcae10700 1 --2- 172.21.2.102:0/3853171754 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4dc40bdda0 0x7f4dc418b250 secure :-1 s=READY pgs=2832 cs=0 l=1 rev1=1 rx=0x7f4dac006f20 tx=0x7f4dac00b990).stop 2021-02-10T11:36:36.172 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:36.169 7f4dcae10700 1 -- 172.21.2.102:0/3853171754 shutdown_connections 2021-02-10T11:36:36.172 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:36.169 7f4dcae10700 1 --2- 172.21.2.102:0/3853171754 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4da80186c0 0x7f4da801abc0 unknown :-1 s=CLOSED pgs=1243 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:36.172 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:36.169 7f4dcae10700 1 --2- 172.21.2.102:0/3853171754 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4dc418bc70 0x7f4dc40c0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:36.172 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:36.169 7f4dcae10700 1 --2- 172.21.2.102:0/3853171754 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4dc418b870 0x7f4dc40c0030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:36.172 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:36.169 7f4dcae10700 1 --2- 172.21.2.102:0/3853171754 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4dc40bdda0 0x7f4dc418b250 unknown :-1 s=CLOSED pgs=2832 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:36.172 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:36.169 7f4dcae10700 1 -- 172.21.2.102:0/3853171754 >> 172.21.2.102:0/3853171754 conn(0x7f4dc40a6eb0 msgr2=0x7f4dc4098ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:36.172 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:36.170 7f4dcae10700 1 -- 172.21.2.102:0/3853171754 shutdown_connections 2021-02-10T11:36:36.172 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:36.170 7f4dcae10700 1 -- 172.21.2.102:0/3853171754 wait complete. 2021-02-10T11:36:36.180 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:36:36.212 INFO:tasks.ceph.mds.b.gibba013.stdout:starting mds.b at 2021-02-10T11:36:36.611 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:36.592 7fc9377b3700 1 Processor -- start 2021-02-10T11:36:36.611 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:36.592 7fc9377b3700 1 -- start start 2021-02-10T11:36:36.612 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:36.592 7fc9377b3700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fc9300d1870 0x7fc9301b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:36.612 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:36.592 7fc9377b3700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fc9301b0a10 0x7fc9300cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:36.612 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:36.592 7fc9377b3700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc9300d1c70 0x7fc9300cf950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:36.612 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:36.592 7fc9377b3700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9300e5ed0 con 0x7fc9300d1870 2021-02-10T11:36:36.612 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:36.592 7fc9377b3700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9300f0b10 con 0x7fc9301b0a10 2021-02-10T11:36:36.612 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:36.592 7fc9377b3700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9300cff70 con 0x7fc9300d1c70 2021-02-10T11:36:36.612 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:36.592 7fc9355cf700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fc9300d1870 0x7fc9301b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:36.612 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:36.593 7fc9355cf700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fc9300d1870 0x7fc9301b03f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:37888/0 (socket says 172.21.2.102:37888) 2021-02-10T11:36:36.612 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:36.593 7fc9355cf700 1 -- 172.21.2.102:0/892418703 learned_addr learned my addr 172.21.2.102:0/892418703 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:36:36.612 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:36.593 7fc935dd0700 1 --2- 172.21.2.102:0/892418703 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc9300d1c70 0x7fc9300cf950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:36.612 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:36.593 7fc9355cf700 1 -- 172.21.2.102:0/892418703 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc9300d1c70 msgr2=0x7fc9300cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:36.613 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:36.593 7fc9355cf700 1 --2- 172.21.2.102:0/892418703 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc9300d1c70 0x7fc9300cf950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:36.613 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:36.593 7fc934dce700 1 --2- 172.21.2.102:0/892418703 >> v2:172.21.2.113:3300/0 conn(0x7fc9301b0a10 0x7fc9300cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:36.613 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:36.593 7fc9355cf700 1 -- 172.21.2.102:0/892418703 >> v2:172.21.2.113:3300/0 conn(0x7fc9301b0a10 msgr2=0x7fc9300cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:36.613 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:36.593 7fc9355cf700 1 --2- 172.21.2.102:0/892418703 >> v2:172.21.2.113:3300/0 conn(0x7fc9301b0a10 0x7fc9300cf330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:36.613 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:36.593 7fc9355cf700 1 -- 172.21.2.102:0/892418703 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9300d0150 con 0x7fc9300d1870 2021-02-10T11:36:36.613 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:36.593 7fc934dce700 1 --2- 172.21.2.102:0/892418703 >> v2:172.21.2.113:3300/0 conn(0x7fc9301b0a10 0x7fc9300cf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:36:36.613 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:36.593 7fc9355cf700 1 --2- 172.21.2.102:0/892418703 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc9300d1870 0x7fc9301b03f0 secure :-1 s=READY pgs=2834 cs=0 l=1 rev1=1 rx=0x7fc924006700 tx=0x7fc924009b00).ready entity=mon.0 client_cookie=4589de6ea6ae255e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:36.613 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:36.593 7fc923fff700 1 -- 172.21.2.102:0/892418703 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc92400c0c0 con 0x7fc9300d1870 2021-02-10T11:36:36.613 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:36.593 7fc923fff700 1 -- 172.21.2.102:0/892418703 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc92400c340 con 0x7fc9300d1870 2021-02-10T11:36:36.613 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:36.593 7fc923fff700 1 -- 172.21.2.102:0/892418703 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc92400c820 con 0x7fc9300d1870 2021-02-10T11:36:36.613 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:36.593 7fc9377b3700 1 -- 172.21.2.102:0/892418703 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc9300d1870 msgr2=0x7fc9301b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:36.613 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:36.593 7fc9377b3700 1 --2- 172.21.2.102:0/892418703 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc9300d1870 0x7fc9301b03f0 secure :-1 s=READY pgs=2834 cs=0 l=1 rev1=1 rx=0x7fc924006700 tx=0x7fc924009b00).stop 2021-02-10T11:36:36.614 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:36.594 7fc9377b3700 1 -- 172.21.2.102:0/892418703 shutdown_connections 2021-02-10T11:36:36.614 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:36.594 7fc9377b3700 1 --2- 172.21.2.102:0/892418703 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc9300d1c70 0x7fc9300cf950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:36.614 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:36.594 7fc9377b3700 1 --2- 172.21.2.102:0/892418703 >> v2:172.21.2.113:3300/0 conn(0x7fc9301b0a10 0x7fc9300cf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:36.614 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:36.594 7fc9377b3700 1 --2- 172.21.2.102:0/892418703 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc9300d1870 0x7fc9301b03f0 unknown :-1 s=CLOSED pgs=2834 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:36.614 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:36.594 7fc9377b3700 1 -- 172.21.2.102:0/892418703 >> 172.21.2.102:0/892418703 conn(0x7fc9300ce450 msgr2=0x7fc9300d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:36.614 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:36.594 7fc9377b3700 1 -- 172.21.2.102:0/892418703 shutdown_connections 2021-02-10T11:36:36.614 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:36.595 7fc9377b3700 1 -- 172.21.2.102:0/892418703 wait complete. 2021-02-10T11:36:36.614 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:36.609 7fc9377b3700 1 Processor -- start 2021-02-10T11:36:36.614 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:36.609 7fc9377b3700 1 -- start start 2021-02-10T11:36:36.616 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:36.610 7fc9377b3700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc9300ada90 0x7fc9300ade90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:36.616 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:36.610 7fc9377b3700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc930099d20 0x7fc93009a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:36.616 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:36.610 7fc9377b3700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc93009a740 0x7fc930097bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:36.617 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:36.610 7fc9377b3700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9300997a0 con 0x7fc93009a740 2021-02-10T11:36:36.617 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:36.610 7fc9377b3700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc93018f0c0 con 0x7fc930099d20 2021-02-10T11:36:36.617 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:36.610 7fc9377b3700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc93009ab40 con 0x7fc9300ada90 2021-02-10T11:36:36.617 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:36.610 7fc935dd0700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc93009a740 0x7fc930097bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:36.617 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:36.610 7fc935dd0700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc93009a740 0x7fc930097bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:37896/0 (socket says 172.21.2.102:37896) 2021-02-10T11:36:36.617 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:36.610 7fc935dd0700 1 -- 172.21.2.102:0/2746865312 learned_addr learned my addr 172.21.2.102:0/2746865312 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:36:36.617 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:36.610 7fc935dd0700 1 -- 172.21.2.102:0/2746865312 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc9300ada90 msgr2=0x7fc9300ade90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:36.617 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:36.610 7fc935dd0700 1 --2- 172.21.2.102:0/2746865312 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc9300ada90 0x7fc9300ade90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:36.617 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:36.610 7fc935dd0700 1 -- 172.21.2.102:0/2746865312 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc930099d20 msgr2=0x7fc93009a120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:36.617 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:36.610 7fc935dd0700 1 --2- 172.21.2.102:0/2746865312 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc930099d20 0x7fc93009a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:36.617 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:36.610 7fc935dd0700 1 -- 172.21.2.102:0/2746865312 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc93009ad60 con 0x7fc93009a740 2021-02-10T11:36:36.617 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:36.611 7fc935dd0700 1 --2- 172.21.2.102:0/2746865312 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc93009a740 0x7fc930097bc0 secure :-1 s=READY pgs=2835 cs=0 l=1 rev1=1 rx=0x7fc928009980 tx=0x7fc92800bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:36.617 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:36.611 7fc921ffb700 1 -- 172.21.2.102:0/2746865312 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc9280011f0 con 0x7fc93009a740 2021-02-10T11:36:36.617 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:36.611 7fc921ffb700 1 -- 172.21.2.102:0/2746865312 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc928003030 con 0x7fc93009a740 2021-02-10T11:36:36.618 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:36.611 7fc921ffb700 1 -- 172.21.2.102:0/2746865312 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc9280016e0 con 0x7fc93009a740 2021-02-10T11:36:36.618 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:36.611 7fc9377b3700 1 -- 172.21.2.102:0/2746865312 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc93009ad60 con 0x7fc93009a740 2021-02-10T11:36:36.618 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:36.611 7fc9377b3700 1 -- 172.21.2.102:0/2746865312 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc93009ad60 con 0x7fc93009a740 2021-02-10T11:36:36.618 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:36.611 7fc921ffb700 1 -- 172.21.2.102:0/2746865312 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fc928000e20 con 0x7fc93009a740 2021-02-10T11:36:36.618 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:36.611 7fc921ffb700 1 --2- 172.21.2.102:0/2746865312 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc9100186c0 0x7fc91001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:36.618 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:36.612 7fc921ffb700 1 -- 172.21.2.102:0/2746865312 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(154..154 src has 1..154) v4 ==== 5659+0+0 (secure 0 0 0) 0x7fc928032950 con 0x7fc93009a740 2021-02-10T11:36:36.618 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:36.612 7fc9355cf700 1 --2- 172.21.2.102:0/2746865312 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc9100186c0 0x7fc91001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:36.618 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:36.612 7fc9355cf700 1 --2- 172.21.2.102:0/2746865312 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc9100186c0 0x7fc91001abc0 crc :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:36.644 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:36.642 7fc9377b3700 1 -- 172.21.2.102:0/2746865312 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9300ebc00 con 0x7fc93009a740 2021-02-10T11:36:36.647 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:36.644 7fc921ffb700 1 -- 172.21.2.102:0/2746865312 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fc92800be10 con 0x7fc93009a740 2021-02-10T11:36:36.776 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:36.774 7fc9377b3700 1 -- 172.21.2.102:0/2746865312 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc9300ebc00 con 0x7fc93009a740 2021-02-10T11:36:36.776 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:36.774 7fc921ffb700 1 -- 172.21.2.102:0/2746865312 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 134 v134) v1 ==== 95+0+1762 (secure 0 0 0) 0x7fc928000e20 con 0x7fc93009a740 2021-02-10T11:36:36.777 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:36:36.777 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":134,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7004,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.102:6835/3280534775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6834","nonce":3280534775},{"type":"v1","addr":"172.21.2.102:6835","nonce":3280534775}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":128}],"filesystems":[{"mdsmap":{"epoch":134,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:35:58.285904","modified":"2021-02-10 11:36:36.161559","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":154,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[33],"metadata_pool":32,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":16}]} 2021-02-10T11:36:36.777 INFO:teuthology.orchestra.run.gibba002.stderr:dumped fsmap epoch 134 2021-02-10T11:36:36.778 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:36.776 7fc9377b3700 1 -- 172.21.2.102:0/2746865312 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc9100186c0 msgr2=0x7fc91001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:36.778 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:36.776 7fc9377b3700 1 --2- 172.21.2.102:0/2746865312 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc9100186c0 0x7fc91001abc0 crc :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:36.778 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:36.776 7fc9377b3700 1 -- 172.21.2.102:0/2746865312 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc93009a740 msgr2=0x7fc930097bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:36.778 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:36.776 7fc9377b3700 1 --2- 172.21.2.102:0/2746865312 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc93009a740 0x7fc930097bc0 secure :-1 s=READY pgs=2835 cs=0 l=1 rev1=1 rx=0x7fc928009980 tx=0x7fc92800bcd0).stop 2021-02-10T11:36:36.779 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:36.776 7fc9377b3700 1 -- 172.21.2.102:0/2746865312 shutdown_connections 2021-02-10T11:36:36.779 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:36.776 7fc9377b3700 1 --2- 172.21.2.102:0/2746865312 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc93009a740 0x7fc930097bc0 unknown :-1 s=CLOSED pgs=2835 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:36.779 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:36.776 7fc9377b3700 1 --2- 172.21.2.102:0/2746865312 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc9100186c0 0x7fc91001abc0 unknown :-1 s=CLOSED pgs=1244 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:36.779 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:36.776 7fc9377b3700 1 --2- 172.21.2.102:0/2746865312 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc930099d20 0x7fc93009a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:36.779 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:36.776 7fc9377b3700 1 --2- 172.21.2.102:0/2746865312 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc9300ada90 0x7fc9300ade90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:36.779 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:36.776 7fc9377b3700 1 -- 172.21.2.102:0/2746865312 >> 172.21.2.102:0/2746865312 conn(0x7fc9300f0550 msgr2=0x7fc930191730 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:36.779 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:36.777 7fc9377b3700 1 -- 172.21.2.102:0/2746865312 shutdown_connections 2021-02-10T11:36:36.779 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:36.777 7fc9377b3700 1 -- 172.21.2.102:0/2746865312 wait complete. 2021-02-10T11:36:36.787 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2021-02-10T11:36:37.223 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:37.203 7f6be31cc700 1 Processor -- start 2021-02-10T11:36:37.223 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:37.203 7f6be31cc700 1 -- start start 2021-02-10T11:36:37.223 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:37.204 7f6be31cc700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f6bdc0d1c70 0x7f6bdc0f20a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:37.223 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:37.204 7f6be31cc700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bdc0f0b10 con 0x7f6bdc0d1870 2021-02-10T11:36:37.223 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:37.204 7f6be31cc700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bdc1b0be0 con 0x7f6bdc0d0340 2021-02-10T11:36:37.224 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:37.204 7f6be31cc700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6bdc0e5ed0 con 0x7f6bdc0d1c70 2021-02-10T11:36:37.224 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:37.204 7f6be0fe8700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f6bdc0d1870 0x7f6bdc1b03f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:58358/0 (socket says 172.21.2.102:58358) 2021-02-10T11:36:37.224 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:37.204 7f6be0fe8700 1 -- 172.21.2.102:0/1286846068 learned_addr learned my addr 172.21.2.102:0/1286846068 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:36:37.224 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:37.204 7f6bdb7fe700 1 -- 172.21.2.102:0/1286846068 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4207586401 0 0) 0x7f6bc40009b0 con 0x7f6bdc0d1870 2021-02-10T11:36:37.224 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:37.204 7f6bdb7fe700 1 -- 172.21.2.102:0/1286846068 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6bc8002fe0 con 0x7f6bdc0d1870 2021-02-10T11:36:37.224 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:37.204 7f6be17e9700 1 --2- 172.21.2.102:0/1286846068 >> v2:172.21.2.113:3300/0 conn(0x7f6bdc0d1c70 0x7f6bdc0f20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:37.224 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:37.205 7f6bdb7fe700 1 -- 172.21.2.102:0/1286846068 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2320717072 0 0) 0x7f6bc4001010 con 0x7f6bdc0d1870 2021-02-10T11:36:37.224 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:37.205 7f6bdb7fe700 1 -- 172.21.2.102:0/1286846068 >> v2:172.21.2.113:3300/0 conn(0x7f6bdc0d1c70 msgr2=0x7f6bdc0f20a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:37.224 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:37.205 7f6bdb7fe700 1 --2- 172.21.2.102:0/1286846068 >> v2:172.21.2.113:3300/0 conn(0x7f6bdc0d1c70 0x7f6bdc0f20a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:37.224 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:37.205 7f6bdb7fe700 1 -- 172.21.2.102:0/1286846068 >> v1:172.21.2.113:6789/0 conn(0x7f6bdc0d0340 legacy=0x7f6bdc0ef640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:37.224 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:37.205 7f6bdb7fe700 1 -- 172.21.2.102:0/1286846068 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bdc0efe30 con 0x7f6bdc0d1870 2021-02-10T11:36:37.224 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:37.205 7f6bdb7fe700 1 -- 172.21.2.102:0/1286846068 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7f6bc4001590 con 0x7f6bdc0d1870 2021-02-10T11:36:37.225 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:37.205 7f6bdb7fe700 1 -- 172.21.2.102:0/1286846068 >> v1:172.21.2.102:6789/0 conn(0x7f6bdc0d1870 legacy=0x7f6bdc1b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:37.225 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:37.205 7f6bdb7fe700 1 --2- 172.21.2.102:0/1286846068 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6bc8003ac0 0x7f6bc8005f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:37.225 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:37.205 7f6bdb7fe700 1 --2- 172.21.2.102:0/1286846068 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6bc80067b0 0x7f6bc8008c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:37.225 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:37.205 7f6bdb7fe700 1 --2- 172.21.2.102:0/1286846068 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6bc8009590 0x7f6bc800bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:37.225 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:37.205 7f6bdb7fe700 1 -- 172.21.2.102:0/1286846068 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bc800c300 con 0x7f6bc8009590 2021-02-10T11:36:37.225 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:37.205 7f6bdb7fe700 1 -- 172.21.2.102:0/1286846068 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bc800c520 con 0x7f6bc8003ac0 2021-02-10T11:36:37.225 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:37.205 7f6bdb7fe700 1 -- 172.21.2.102:0/1286846068 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bc800c740 con 0x7f6bc80067b0 2021-02-10T11:36:37.225 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:37.205 7f6be17e9700 1 --2- 172.21.2.102:0/1286846068 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6bc8009590 0x7f6bc800bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:37.225 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:37.205 7f6be0fe8700 1 --2- 172.21.2.102:0/1286846068 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6bc8003ac0 0x7f6bc8005f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:37.225 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:37.205 7f6bdbfff700 1 --2- 172.21.2.102:0/1286846068 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6bc80067b0 0x7f6bc8008c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:37.225 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:37.205 7f6be17e9700 1 -- 172.21.2.102:0/1286846068 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6bc80067b0 msgr2=0x7f6bc8008c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:37.225 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:37.205 7f6be17e9700 1 --2- 172.21.2.102:0/1286846068 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6bc80067b0 0x7f6bc8008c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:37.225 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:37.205 7f6be17e9700 1 -- 172.21.2.102:0/1286846068 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6bc8003ac0 msgr2=0x7f6bc8005f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:37.225 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:37.205 7f6be17e9700 1 --2- 172.21.2.102:0/1286846068 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6bc8003ac0 0x7f6bc8005f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:37.226 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:37.205 7f6be17e9700 1 -- 172.21.2.102:0/1286846068 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bc80509b0 con 0x7f6bc8009590 2021-02-10T11:36:37.226 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:37.205 7f6be17e9700 1 -- 172.21.2.102:0/1286846068 reap_dead start 2021-02-10T11:36:37.226 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:37.205 7f6be17e9700 1 --2- 172.21.2.102:0/1286846068 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6bc8009590 0x7f6bc800bab0 secure :-1 s=READY pgs=2837 cs=0 l=1 rev1=1 rx=0x7f6bd4006a40 tx=0x7f6bd400c0c0).ready entity=mon.0 client_cookie=74c8ac187214089e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:37.226 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:37.206 7f6bdb7fe700 1 -- 172.21.2.102:0/1286846068 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6bd4006a80 con 0x7f6bc8009590 2021-02-10T11:36:37.226 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:37.206 7f6bdb7fe700 1 -- 172.21.2.102:0/1286846068 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bd4007070 con 0x7f6bc8009590 2021-02-10T11:36:37.226 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:37.206 7f6bdb7fe700 1 -- 172.21.2.102:0/1286846068 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6bd40066f0 con 0x7f6bc8009590 2021-02-10T11:36:37.226 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:37.206 7f6be31cc700 1 -- 172.21.2.102:0/1286846068 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6bc8009590 msgr2=0x7f6bc800bab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:37.226 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:37.206 7f6be31cc700 1 --2- 172.21.2.102:0/1286846068 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6bc8009590 0x7f6bc800bab0 secure :-1 s=READY pgs=2837 cs=0 l=1 rev1=1 rx=0x7f6bd4006a40 tx=0x7f6bd400c0c0).stop 2021-02-10T11:36:37.226 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:37.206 7f6be31cc700 1 -- 172.21.2.102:0/1286846068 shutdown_connections 2021-02-10T11:36:37.226 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:37.206 7f6be31cc700 1 --2- 172.21.2.102:0/1286846068 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6bc8009590 0x7f6bc800bab0 unknown :-1 s=CLOSED pgs=2837 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:37.226 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:37.206 7f6be31cc700 1 -- 172.21.2.102:0/1286846068 >> 172.21.2.102:0/1286846068 conn(0x7f6bdc0ce450 msgr2=0x7f6bdc0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:37.226 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:37.206 7f6be31cc700 1 -- 172.21.2.102:0/1286846068 shutdown_connections 2021-02-10T11:36:37.227 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:37.206 7f6be31cc700 1 -- 172.21.2.102:0/1286846068 wait complete. 2021-02-10T11:36:37.227 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:37.221 7f6be31cc700 1 Processor -- start 2021-02-10T11:36:37.227 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:37.221 7f6be31cc700 1 -- start start 2021-02-10T11:36:37.229 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:37.222 7f6be31cc700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6bdc094310 0x7f6bdc0ebe70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:37.229 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:37.222 7f6be31cc700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6bdc094710 0x7f6bdc0bca30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:37.229 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:37.222 7f6be31cc700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6bdc0ec490 0x7f6bdc0bd050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:37.229 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:37.222 7f6be31cc700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bdc093d90 con 0x7f6bdc0ec490 2021-02-10T11:36:37.229 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:37.222 7f6be31cc700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bdc0bd670 con 0x7f6bdc094310 2021-02-10T11:36:37.229 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:37.222 7f6be31cc700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bdc0939f0 con 0x7f6bdc094710 2021-02-10T11:36:37.229 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:37.222 7f6bdbfff700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6bdc094710 0x7f6bdc0bca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:37.229 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:37.222 7f6be17e9700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6bdc0ec490 0x7f6bdc0bd050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:37.229 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:37.222 7f6bdbfff700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6bdc094710 0x7f6bdc0bca30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.113:3301/0 says I am v2:172.21.2.102:43448/0 (socket says 172.21.2.102:43448) 2021-02-10T11:36:37.229 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:37.222 7f6bdbfff700 1 -- 172.21.2.102:0/1105724509 learned_addr learned my addr 172.21.2.102:0/1105724509 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:36:37.229 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:37.222 7f6be0fe8700 1 --2- 172.21.2.102:0/1105724509 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6bdc094310 0x7f6bdc0ebe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:37.229 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:37.222 7f6be17e9700 1 -- 172.21.2.102:0/1105724509 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6bdc094710 msgr2=0x7f6bdc0bca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:37.230 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:37.222 7f6be17e9700 1 --2- 172.21.2.102:0/1105724509 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6bdc094710 0x7f6bdc0bca30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:37.230 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:37.222 7f6be17e9700 1 -- 172.21.2.102:0/1105724509 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6bdc094310 msgr2=0x7f6bdc0ebe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:37.230 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:37.222 7f6be17e9700 1 --2- 172.21.2.102:0/1105724509 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6bdc094310 0x7f6bdc0ebe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:37.230 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:37.222 7f6be17e9700 1 -- 172.21.2.102:0/1105724509 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bdc0bda80 con 0x7f6bdc0ec490 2021-02-10T11:36:37.230 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:37.222 7f6be0fe8700 1 --2- 172.21.2.102:0/1105724509 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6bdc094310 0x7f6bdc0ebe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:36:37.230 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:37.223 7f6be17e9700 1 --2- 172.21.2.102:0/1105724509 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6bdc0ec490 0x7f6bdc0bd050 secure :-1 s=READY pgs=2838 cs=0 l=1 rev1=1 rx=0x7f6bd40096b0 tx=0x7f6bd400bc90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:37.230 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:37.223 7f6bd97fa700 1 -- 172.21.2.102:0/1105724509 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6bd4008030 con 0x7f6bdc0ec490 2021-02-10T11:36:37.230 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:37.223 7f6bd97fa700 1 -- 172.21.2.102:0/1105724509 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bd40068e0 con 0x7f6bdc0ec490 2021-02-10T11:36:37.230 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:37.223 7f6be31cc700 1 -- 172.21.2.102:0/1105724509 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6bdc0bda80 con 0x7f6bdc0ec490 2021-02-10T11:36:37.230 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:37.223 7f6bd97fa700 1 -- 172.21.2.102:0/1105724509 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6bd4008630 con 0x7f6bdc0ec490 2021-02-10T11:36:37.230 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:37.223 7f6be31cc700 1 -- 172.21.2.102:0/1105724509 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bdc1a76d0 con 0x7f6bdc0ec490 2021-02-10T11:36:37.231 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:37.223 7f6bd97fa700 1 -- 172.21.2.102:0/1105724509 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f6bd4008330 con 0x7f6bdc0ec490 2021-02-10T11:36:37.231 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:37.223 7f6bd97fa700 1 --2- 172.21.2.102:0/1105724509 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6bc00186d0 0x7f6bc001abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:37.231 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:37.223 7f6bd97fa700 1 -- 172.21.2.102:0/1105724509 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(154..154 src has 1..154) v4 ==== 5659+0+0 (secure 0 0 0) 0x7f6bd4032950 con 0x7f6bdc0ec490 2021-02-10T11:36:37.231 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:37.224 7f6be0fe8700 1 --2- 172.21.2.102:0/1105724509 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6bc00186d0 0x7f6bc001abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:37.231 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:37.224 7f6be0fe8700 1 --2- 172.21.2.102:0/1105724509 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6bc00186d0 0x7f6bc001abd0 crc :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:37.255 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:37.253 7f6be31cc700 1 -- 172.21.2.102:0/1105724509 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6bdc0ebc40 con 0x7f6bdc0ec490 2021-02-10T11:36:37.258 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:37.256 7f6bd97fa700 1 -- 172.21.2.102:0/1105724509 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f6bd4008540 con 0x7f6bdc0ec490 2021-02-10T11:36:37.389 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:37.387 7f6be31cc700 1 -- 172.21.2.102:0/1105724509 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f6bdc0f0b10 con 0x7f6bdc0ec490 2021-02-10T11:36:38.174 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:38.172 7f6bd97fa700 1 -- 172.21.2.102:0/1105724509 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v135) v1 ==== 104+0+0 (secure 0 0 0) 0x7f6bd4008330 con 0x7f6bdc0ec490 2021-02-10T11:36:38.211 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:38.209 7f6be31cc700 1 -- 172.21.2.102:0/1105724509 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6bc00186d0 msgr2=0x7f6bc001abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:38.211 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:38.209 7f6be31cc700 1 --2- 172.21.2.102:0/1105724509 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6bc00186d0 0x7f6bc001abd0 crc :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:38.211 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:38.209 7f6be31cc700 1 -- 172.21.2.102:0/1105724509 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6bdc0ec490 msgr2=0x7f6bdc0bd050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:38.211 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:38.209 7f6be31cc700 1 --2- 172.21.2.102:0/1105724509 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6bdc0ec490 0x7f6bdc0bd050 secure :-1 s=READY pgs=2838 cs=0 l=1 rev1=1 rx=0x7f6bd40096b0 tx=0x7f6bd400bc90).stop 2021-02-10T11:36:38.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:38.209 7f6be31cc700 1 -- 172.21.2.102:0/1105724509 shutdown_connections 2021-02-10T11:36:38.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:38.209 7f6be31cc700 1 --2- 172.21.2.102:0/1105724509 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6bdc0ec490 0x7f6bdc0bd050 unknown :-1 s=CLOSED pgs=2838 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:38.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:38.209 7f6be31cc700 1 --2- 172.21.2.102:0/1105724509 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6bdc094710 0x7f6bdc0bca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:38.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:38.209 7f6be31cc700 1 --2- 172.21.2.102:0/1105724509 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6bc00186d0 0x7f6bc001abd0 unknown :-1 s=CLOSED pgs=1245 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:38.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:38.209 7f6be31cc700 1 --2- 172.21.2.102:0/1105724509 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6bdc094310 0x7f6bdc0ebe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:38.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:38.209 7f6be31cc700 1 -- 172.21.2.102:0/1105724509 >> 172.21.2.102:0/1105724509 conn(0x7f6bdc08fa10 msgr2=0x7f6bdc0bfa60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:38.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:38.209 7f6be31cc700 1 -- 172.21.2.102:0/1105724509 shutdown_connections 2021-02-10T11:36:38.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:38.209 7f6be31cc700 1 -- 172.21.2.102:0/1105724509 wait complete. 2021-02-10T11:36:38.219 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool delete cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2021-02-10T11:36:38.655 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:38.635 7fad3fbb3700 1 Processor -- start 2021-02-10T11:36:38.655 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:38.635 7fad3fbb3700 1 -- start start 2021-02-10T11:36:38.655 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:38.635 7fad3fbb3700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fad3810b000 0x7fad380cf760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:38.655 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:38.635 7fad3fbb3700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad380fe1d0 con 0x7fad380d1880 2021-02-10T11:36:38.655 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:38.635 7fad3fbb3700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad381b0cc0 con 0x7fad380d1c80 2021-02-10T11:36:38.655 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:38.635 7fad3fbb3700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad38109a70 con 0x7fad3810b000 2021-02-10T11:36:38.655 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:38.635 7fad3d9cf700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fad380d1880 0x7fad381b04d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:58374/0 (socket says 172.21.2.102:58374) 2021-02-10T11:36:38.655 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:38.635 7fad3d9cf700 1 -- 172.21.2.102:0/383366584 learned_addr learned my addr 172.21.2.102:0/383366584 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:36:38.656 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:38.636 7fad3c9cd700 1 -- 172.21.2.102:0/383366584 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3919731663 0 0) 0x7fad30000d30 con 0x7fad380d1880 2021-02-10T11:36:38.656 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:38.636 7fad3c9cd700 1 -- 172.21.2.102:0/383366584 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad24002fe0 con 0x7fad380d1880 2021-02-10T11:36:38.656 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:38.636 7fad3e1d0700 1 --2- 172.21.2.102:0/383366584 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fad3810b000 0x7fad380cf760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:38.656 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:38.636 7fad3c9cd700 1 -- 172.21.2.102:0/383366584 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3359999183 0 0) 0x7fad30001390 con 0x7fad380d1880 2021-02-10T11:36:38.656 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:38.636 7fad3c9cd700 1 -- 172.21.2.102:0/383366584 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fad3810b000 msgr2=0x7fad380cf760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:38.656 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:38.636 7fad3c9cd700 1 --2- 172.21.2.102:0/383366584 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fad3810b000 0x7fad380cf760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:38.656 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:38.636 7fad3c9cd700 1 -- 172.21.2.102:0/383366584 >> v1:172.21.2.113:6789/0 conn(0x7fad380d1c80 legacy=0x7fad380cef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:38.656 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:38.636 7fad3c9cd700 1 -- 172.21.2.102:0/383366584 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad3810b400 con 0x7fad380d1880 2021-02-10T11:36:38.656 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:38.636 7fad3c9cd700 1 -- 172.21.2.102:0/383366584 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7fad30001910 con 0x7fad380d1880 2021-02-10T11:36:38.656 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:38.636 7fad3c9cd700 1 -- 172.21.2.102:0/383366584 >> v1:172.21.2.102:6789/0 conn(0x7fad380d1880 legacy=0x7fad381b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:38.656 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:38.636 7fad3c9cd700 1 --2- 172.21.2.102:0/383366584 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fad24003b90 0x7fad24005fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:38.657 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:38.636 7fad3c9cd700 1 --2- 172.21.2.102:0/383366584 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fad24006770 0x7fad24008c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:38.657 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:38.636 7fad3c9cd700 1 --2- 172.21.2.102:0/383366584 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fad24009510 0x7fad2400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:38.657 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:38.636 7fad3c9cd700 1 -- 172.21.2.102:0/383366584 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad2400c2c0 con 0x7fad24006770 2021-02-10T11:36:38.657 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:38.636 7fad3c9cd700 1 -- 172.21.2.102:0/383366584 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad2400c4e0 con 0x7fad24009510 2021-02-10T11:36:38.657 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:38.636 7fad3c9cd700 1 -- 172.21.2.102:0/383366584 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad2400c700 con 0x7fad24003b90 2021-02-10T11:36:38.657 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:38.636 7fad3d1ce700 1 --2- 172.21.2.102:0/383366584 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fad24006770 0x7fad24008c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:38.657 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:38.637 7fad3e1d0700 1 --2- 172.21.2.102:0/383366584 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fad24009510 0x7fad2400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:38.657 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:38.637 7fad3d1ce700 1 -- 172.21.2.102:0/383366584 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fad24003b90 msgr2=0x7fad24005fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:38.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:38.637 7fad3d1ce700 1 --2- 172.21.2.102:0/383366584 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fad24003b90 0x7fad24005fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:38.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:38.637 7fad3d1ce700 1 -- 172.21.2.102:0/383366584 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fad24009510 msgr2=0x7fad2400ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:38.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:38.637 7fad3d1ce700 1 --2- 172.21.2.102:0/383366584 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fad24009510 0x7fad2400ba10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:38.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:38.637 7fad3d1ce700 1 -- 172.21.2.102:0/383366584 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad24050970 con 0x7fad24006770 2021-02-10T11:36:38.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:38.637 7fad3d1ce700 1 --2- 172.21.2.102:0/383366584 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fad24006770 0x7fad24008c90 secure :-1 s=READY pgs=2840 cs=0 l=1 rev1=1 rx=0x7fad28006ab0 tx=0x7fad28009ac0).ready entity=mon.0 client_cookie=48b7277e3e28359b server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:38.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:38.637 7fad3c9cd700 1 -- 172.21.2.102:0/383366584 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fad280066d0 con 0x7fad24006770 2021-02-10T11:36:38.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:38.637 7fad3c9cd700 1 -- 172.21.2.102:0/383366584 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad280022e0 con 0x7fad24006770 2021-02-10T11:36:38.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:38.637 7fad3c9cd700 1 -- 172.21.2.102:0/383366584 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fad28000960 con 0x7fad24006770 2021-02-10T11:36:38.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:38.637 7fad3fbb3700 1 -- 172.21.2.102:0/383366584 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fad24006770 msgr2=0x7fad24008c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:38.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:38.637 7fad3fbb3700 1 --2- 172.21.2.102:0/383366584 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fad24006770 0x7fad24008c90 secure :-1 s=READY pgs=2840 cs=0 l=1 rev1=1 rx=0x7fad28006ab0 tx=0x7fad28009ac0).stop 2021-02-10T11:36:38.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:38.637 7fad3e1d0700 1 -- 172.21.2.102:0/383366584 reap_dead start 2021-02-10T11:36:38.659 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:38.638 7fad3fbb3700 1 -- 172.21.2.102:0/383366584 shutdown_connections 2021-02-10T11:36:38.659 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:38.638 7fad3fbb3700 1 -- 172.21.2.102:0/383366584 >> 172.21.2.102:0/383366584 conn(0x7fad380ce430 msgr2=0x7fad380d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:38.659 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:38.638 7fad3fbb3700 1 -- 172.21.2.102:0/383366584 shutdown_connections 2021-02-10T11:36:38.659 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:38.638 7fad3fbb3700 1 -- 172.21.2.102:0/383366584 wait complete. 2021-02-10T11:36:38.659 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:38.653 7fad3fbb3700 1 Processor -- start 2021-02-10T11:36:38.659 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:38.653 7fad3fbb3700 1 -- start start 2021-02-10T11:36:38.661 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:38.653 7fad3fbb3700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fad381a3570 0x7fad381a3970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:38.661 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:38.653 7fad3fbb3700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fad3809c9f0 0x7fad3809cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:38.661 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:38.653 7fad3fbb3700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fad3809d410 0x7fad38098170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:38.661 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:38.653 7fad3fbb3700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad3809c470 con 0x7fad3809c9f0 2021-02-10T11:36:38.661 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:38.653 7fad3fbb3700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad380b2840 con 0x7fad3809d410 2021-02-10T11:36:38.661 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:38.653 7fad3fbb3700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad380b2a20 con 0x7fad381a3570 2021-02-10T11:36:38.661 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:38.653 7fad3d1ce700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fad3809c9f0 0x7fad3809cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:38.661 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:38.653 7fad3d1ce700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fad3809c9f0 0x7fad3809cdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:37930/0 (socket says 172.21.2.102:37930) 2021-02-10T11:36:38.661 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:38.653 7fad3d1ce700 1 -- 172.21.2.102:0/3626020421 learned_addr learned my addr 172.21.2.102:0/3626020421 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:36:38.661 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:38.654 7fad3d1ce700 1 -- 172.21.2.102:0/3626020421 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fad381a3570 msgr2=0x7fad381a3970 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:36:38.661 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:38.654 7fad3d1ce700 1 --2- 172.21.2.102:0/3626020421 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fad381a3570 0x7fad381a3970 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:38.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:38.654 7fad3d1ce700 1 -- 172.21.2.102:0/3626020421 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fad3809d410 msgr2=0x7fad38098170 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:36:38.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:38.654 7fad3d1ce700 1 --2- 172.21.2.102:0/3626020421 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fad3809d410 0x7fad38098170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:38.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:38.654 7fad3d1ce700 1 -- 172.21.2.102:0/3626020421 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad3809d880 con 0x7fad3809c9f0 2021-02-10T11:36:38.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:38.654 7fad3d1ce700 1 --2- 172.21.2.102:0/3626020421 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fad3809c9f0 0x7fad3809cdf0 secure :-1 s=READY pgs=2841 cs=0 l=1 rev1=1 rx=0x7fad2800b580 tx=0x7fad2800b920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:38.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:38.654 7fad2e7fc700 1 -- 172.21.2.102:0/3626020421 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fad28000960 con 0x7fad3809c9f0 2021-02-10T11:36:38.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:38.654 7fad2e7fc700 1 -- 172.21.2.102:0/3626020421 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad2800bd10 con 0x7fad3809c9f0 2021-02-10T11:36:38.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:38.654 7fad3fbb3700 1 -- 172.21.2.102:0/3626020421 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad3809d880 con 0x7fad3809c9f0 2021-02-10T11:36:38.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:38.654 7fad2e7fc700 1 -- 172.21.2.102:0/3626020421 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fad28006210 con 0x7fad3809c9f0 2021-02-10T11:36:38.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:38.654 7fad3fbb3700 1 -- 172.21.2.102:0/3626020421 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad3809d880 con 0x7fad3809c9f0 2021-02-10T11:36:38.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:38.654 7fad2e7fc700 1 -- 172.21.2.102:0/3626020421 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fad2800a000 con 0x7fad3809c9f0 2021-02-10T11:36:38.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:38.655 7fad2e7fc700 1 --2- 172.21.2.102:0/3626020421 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fad20018730 0x7fad2001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:38.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:38.655 7fad2e7fc700 1 -- 172.21.2.102:0/3626020421 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(154..154 src has 1..154) v4 ==== 5659+0+0 (secure 0 0 0) 0x7fad28032a30 con 0x7fad3809c9f0 2021-02-10T11:36:38.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:38.655 7fad3d9cf700 1 --2- 172.21.2.102:0/3626020421 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fad20018730 0x7fad2001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:38.663 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:38.655 7fad3d9cf700 1 --2- 172.21.2.102:0/3626020421 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fad20018730 0x7fad2001ac10 crc :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:38.687 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:38.685 7fad3fbb3700 1 -- 172.21.2.102:0/3626020421 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad38082a70 con 0x7fad3809c9f0 2021-02-10T11:36:38.689 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:38.687 7fad2e7fc700 1 -- 172.21.2.102:0/3626020421 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fad280093e0 con 0x7fad3809c9f0 2021-02-10T11:36:38.817 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:38.815 7fad3fbb3700 1 -- 172.21.2.102:0/3626020421 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"pool2": "cephfs_metadata", "prefix": "osd pool delete", "pool": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fad38082a70 con 0x7fad3809c9f0 2021-02-10T11:36:39.176 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:39.174 7fad2e7fc700 1 -- 172.21.2.102:0/3626020421 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"pool2": "cephfs_metadata", "prefix": "osd pool delete", "pool": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v155) v1 ==== 185+0+0 (secure 0 0 0) 0x7fad2801f510 con 0x7fad3809c9f0 2021-02-10T11:36:39.182 INFO:teuthology.orchestra.run.gibba002.stderr:pool 'cephfs_metadata' removed 2021-02-10T11:36:39.184 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:39.182 7fad3fbb3700 1 -- 172.21.2.102:0/3626020421 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fad20018730 msgr2=0x7fad2001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:39.184 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:39.182 7fad3fbb3700 1 --2- 172.21.2.102:0/3626020421 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fad20018730 0x7fad2001ac10 crc :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:39.184 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:39.182 7fad3fbb3700 1 -- 172.21.2.102:0/3626020421 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fad3809c9f0 msgr2=0x7fad3809cdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:39.185 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:39.182 7fad3fbb3700 1 --2- 172.21.2.102:0/3626020421 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fad3809c9f0 0x7fad3809cdf0 secure :-1 s=READY pgs=2841 cs=0 l=1 rev1=1 rx=0x7fad2800b580 tx=0x7fad2800b920).stop 2021-02-10T11:36:39.185 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:39.182 7fad3fbb3700 1 -- 172.21.2.102:0/3626020421 shutdown_connections 2021-02-10T11:36:39.185 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:39.182 7fad3fbb3700 1 --2- 172.21.2.102:0/3626020421 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fad20018730 0x7fad2001ac10 unknown :-1 s=CLOSED pgs=1246 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:39.185 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:39.182 7fad3fbb3700 1 --2- 172.21.2.102:0/3626020421 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fad3809d410 0x7fad38098170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:39.185 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:39.182 7fad3fbb3700 1 --2- 172.21.2.102:0/3626020421 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fad3809c9f0 0x7fad3809cdf0 unknown :-1 s=CLOSED pgs=2841 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:39.185 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:39.182 7fad3fbb3700 1 --2- 172.21.2.102:0/3626020421 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fad381a3570 0x7fad381a3970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:39.185 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:39.182 7fad3fbb3700 1 -- 172.21.2.102:0/3626020421 >> 172.21.2.102:0/3626020421 conn(0x7fad380c2cd0 msgr2=0x7fad380b5950 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:39.185 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:39.182 7fad3fbb3700 1 -- 172.21.2.102:0/3626020421 shutdown_connections 2021-02-10T11:36:39.185 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:39.182 7fad3fbb3700 1 -- 172.21.2.102:0/3626020421 wait complete. 2021-02-10T11:36:39.192 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool delete cephfs_data cephfs_data --yes-i-really-really-mean-it 2021-02-10T11:36:39.627 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:39.607 7fa10c426700 1 Processor -- start 2021-02-10T11:36:39.627 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:39.607 7fa10c426700 1 -- start start 2021-02-10T11:36:39.628 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:39.608 7fa10c426700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa1040d1c80 0x7fa1040cef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:39.628 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:39.608 7fa10c426700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fa1040cf590 0x7fa1040cf990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:39.628 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:39.608 7fa10c426700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1040fe1d0 con 0x7fa1040d1880 2021-02-10T11:36:39.628 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:39.608 7fa10c426700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1041b0cc0 con 0x7fa1040cf590 2021-02-10T11:36:39.628 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:39.608 7fa10c426700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa104109a70 con 0x7fa1040d1c80 2021-02-10T11:36:39.628 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:39.609 7fa109a41700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa1040d1c80 0x7fa1040cef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:39.628 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:39.609 7fa10a242700 1 --1- >> v1:172.21.2.113:6789/0 conn(0x7fa1040d1880 0x7fa1041b04d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.113:6789/0 says I am v1:172.21.2.102:37486/0 (socket says 172.21.2.102:37486) 2021-02-10T11:36:39.628 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:39.609 7fa109a41700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa1040d1c80 0x7fa1040cef70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.113:3301/0 says I am v2:172.21.2.102:43474/0 (socket says 172.21.2.102:43474) 2021-02-10T11:36:39.628 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:39.609 7fa10a242700 1 -- 172.21.2.102:0/3097501725 learned_addr learned my addr 172.21.2.102:0/3097501725 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:36:39.628 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:39.609 7fa10aa43700 1 --2- 172.21.2.102:0/3097501725 >> v2:172.21.2.113:3300/0 conn(0x7fa1040cf590 0x7fa1040cf990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:39.628 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:39.609 7fa109240700 1 -- 172.21.2.102:0/3097501725 <== mon.1 v1:172.21.2.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3812829048 0 0) 0x7fa1000010b0 con 0x7fa1040d1880 2021-02-10T11:36:39.628 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:39.609 7fa109a41700 1 -- 172.21.2.102:0/3097501725 >> v2:172.21.2.113:3300/0 conn(0x7fa1040cf590 msgr2=0x7fa1040cf990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:39.629 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:39.609 7fa109a41700 1 --2- 172.21.2.102:0/3097501725 >> v2:172.21.2.113:3300/0 conn(0x7fa1040cf590 0x7fa1040cf990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:39.629 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:39.609 7fa109a41700 1 -- 172.21.2.102:0/3097501725 >> v1:172.21.2.113:6789/0 conn(0x7fa1040d1880 legacy=0x7fa1041b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:39.629 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:39.609 7fa109a41700 1 -- 172.21.2.102:0/3097501725 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa104088070 con 0x7fa1040d1c80 2021-02-10T11:36:39.629 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:39.609 7fa10aa43700 1 --2- 172.21.2.102:0/3097501725 >> v2:172.21.2.113:3300/0 conn(0x7fa1040cf590 0x7fa1040cf990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:36:39.629 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:39.610 7fa109a41700 1 --2- 172.21.2.102:0/3097501725 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa1040d1c80 0x7fa1040cef70 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 rx=0x7fa0f8006650 tx=0x7fa0f8009a80).ready entity=mon.2 client_cookie=28a72448c7e24374 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:39.629 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:39.610 7fa109240700 1 -- 172.21.2.102:0/3097501725 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa0f800c0c0 con 0x7fa1040d1c80 2021-02-10T11:36:39.629 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:39.610 7fa109240700 1 -- 172.21.2.102:0/3097501725 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0f800c340 con 0x7fa1040d1c80 2021-02-10T11:36:39.629 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:39.610 7fa109240700 1 -- 172.21.2.102:0/3097501725 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa0f800c860 con 0x7fa1040d1c80 2021-02-10T11:36:39.629 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:39.610 7fa10c426700 1 -- 172.21.2.102:0/3097501725 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa1040d1c80 msgr2=0x7fa1040cef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:39.629 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:39.610 7fa10c426700 1 --2- 172.21.2.102:0/3097501725 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa1040d1c80 0x7fa1040cef70 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 rx=0x7fa0f8006650 tx=0x7fa0f8009a80).stop 2021-02-10T11:36:39.629 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:39.611 7fa10c426700 1 -- 172.21.2.102:0/3097501725 shutdown_connections 2021-02-10T11:36:39.629 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:39.611 7fa10c426700 1 --2- 172.21.2.102:0/3097501725 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa1040d1c80 0x7fa1040cef70 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:39.629 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:39.611 7fa10c426700 1 --2- 172.21.2.102:0/3097501725 >> v2:172.21.2.113:3300/0 conn(0x7fa1040cf590 0x7fa1040cf990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:39.629 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:39.611 7fa10c426700 1 -- 172.21.2.102:0/3097501725 >> 172.21.2.102:0/3097501725 conn(0x7fa1040ce430 msgr2=0x7fa1040d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:39.630 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:39.611 7fa10c426700 1 -- 172.21.2.102:0/3097501725 shutdown_connections 2021-02-10T11:36:39.630 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:39.611 7fa10c426700 1 -- 172.21.2.102:0/3097501725 wait complete. 2021-02-10T11:36:39.631 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:39.626 7fa10c426700 1 Processor -- start 2021-02-10T11:36:39.631 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:39.626 7fa10c426700 1 -- start start 2021-02-10T11:36:39.631 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:39.626 7fa10c426700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa1040b50a0 0x7fa1040ac2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:39.631 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:39.626 7fa10c426700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa1040b54a0 0x7fa1040bf5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:39.632 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:39.626 7fa10c426700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa1040ac900 0x7fa1040bfbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:39.632 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:39.626 7fa10c426700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1040ad390 con 0x7fa1040b50a0 2021-02-10T11:36:39.632 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:39.626 7fa10c426700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1040bf040 con 0x7fa1040b54a0 2021-02-10T11:36:39.632 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:39.626 7fa10c426700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1040aced0 con 0x7fa1040ac900 2021-02-10T11:36:39.632 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:39.626 7fa10a242700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa1040b50a0 0x7fa1040ac2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:39.632 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:39.626 7fa10a242700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa1040b50a0 0x7fa1040ac2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:37948/0 (socket says 172.21.2.102:37948) 2021-02-10T11:36:39.632 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:39.626 7fa10a242700 1 -- 172.21.2.102:0/1254595117 learned_addr learned my addr 172.21.2.102:0/1254595117 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:36:39.632 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:39.627 7fa10aa43700 1 --2- 172.21.2.102:0/1254595117 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa1040ac900 0x7fa1040bfbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:39.632 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:39.627 7fa109a41700 1 --2- 172.21.2.102:0/1254595117 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa1040b54a0 0x7fa1040bf5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:39.632 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:39.627 7fa10a242700 1 -- 172.21.2.102:0/1254595117 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa1040ac900 msgr2=0x7fa1040bfbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:39.632 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:39.627 7fa10a242700 1 --2- 172.21.2.102:0/1254595117 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa1040ac900 0x7fa1040bfbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:39.632 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:39.627 7fa10a242700 1 -- 172.21.2.102:0/1254595117 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa1040b54a0 msgr2=0x7fa1040bf5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:39.632 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:39.627 7fa10a242700 1 --2- 172.21.2.102:0/1254595117 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa1040b54a0 0x7fa1040bf5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:39.632 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:39.627 7fa10a242700 1 -- 172.21.2.102:0/1254595117 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1040c0200 con 0x7fa1040b50a0 2021-02-10T11:36:39.633 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:39.627 7fa10a242700 1 --2- 172.21.2.102:0/1254595117 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa1040b50a0 0x7fa1040ac2e0 secure :-1 s=READY pgs=2842 cs=0 l=1 rev1=1 rx=0x7fa100001c10 tx=0x7fa10000a8f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:39.633 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:39.627 7fa0f6ffd700 1 -- 172.21.2.102:0/1254595117 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa1000009b0 con 0x7fa1040b50a0 2021-02-10T11:36:39.633 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:39.627 7fa0f6ffd700 1 -- 172.21.2.102:0/1254595117 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1000012c0 con 0x7fa1040b50a0 2021-02-10T11:36:39.633 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:39.627 7fa10c426700 1 -- 172.21.2.102:0/1254595117 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1040c0200 con 0x7fa1040b50a0 2021-02-10T11:36:39.633 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:39.627 7fa10c426700 1 -- 172.21.2.102:0/1254595117 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1040c0200 con 0x7fa1040b50a0 2021-02-10T11:36:39.633 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:39.627 7fa0f6ffd700 1 -- 172.21.2.102:0/1254595117 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa100004350 con 0x7fa1040b50a0 2021-02-10T11:36:39.633 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:39.628 7fa0f6ffd700 1 -- 172.21.2.102:0/1254595117 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa100004350 con 0x7fa1040b50a0 2021-02-10T11:36:39.633 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:39.628 7fa0f6ffd700 1 --2- 172.21.2.102:0/1254595117 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa0ec018730 0x7fa0ec01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:39.633 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:39.628 7fa0f6ffd700 1 -- 172.21.2.102:0/1254595117 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(155..155 src has 1..155) v4 ==== 5221+0+0 (secure 0 0 0) 0x7fa100032620 con 0x7fa1040b50a0 2021-02-10T11:36:39.633 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:39.628 7fa109a41700 1 --2- 172.21.2.102:0/1254595117 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa0ec018730 0x7fa0ec01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:39.633 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:39.628 7fa109a41700 1 --2- 172.21.2.102:0/1254595117 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa0ec018730 0x7fa0ec01ac10 crc :-1 s=READY pgs=1247 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:39.659 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:39.657 7fa10c426700 1 -- 172.21.2.102:0/1254595117 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1040fe1d0 con 0x7fa1040b50a0 2021-02-10T11:36:39.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:39.660 7fa0f6ffd700 1 -- 172.21.2.102:0/1254595117 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fa100004140 con 0x7fa1040b50a0 2021-02-10T11:36:39.788 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:39.786 7fa10c426700 1 -- 172.21.2.102:0/1254595117 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"pool2": "cephfs_data", "prefix": "osd pool delete", "pool": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fa1040fe1d0 con 0x7fa1040b50a0 2021-02-10T11:36:40.181 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:40.179 7fa0f6ffd700 1 -- 172.21.2.102:0/1254595117 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"pool2": "cephfs_data", "prefix": "osd pool delete", "pool": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v156) v1 ==== 173+0+0 (secure 0 0 0) 0x7fa100004350 con 0x7fa1040b50a0 2021-02-10T11:36:40.203 INFO:teuthology.orchestra.run.gibba002.stderr:pool 'cephfs_data' removed 2021-02-10T11:36:40.205 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:40.203 7fa10c426700 1 -- 172.21.2.102:0/1254595117 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa0ec018730 msgr2=0x7fa0ec01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:40.205 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:40.203 7fa10c426700 1 --2- 172.21.2.102:0/1254595117 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa0ec018730 0x7fa0ec01ac10 crc :-1 s=READY pgs=1247 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:40.205 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:40.203 7fa10c426700 1 -- 172.21.2.102:0/1254595117 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa1040b50a0 msgr2=0x7fa1040ac2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:40.205 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:40.203 7fa10c426700 1 --2- 172.21.2.102:0/1254595117 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa1040b50a0 0x7fa1040ac2e0 secure :-1 s=READY pgs=2842 cs=0 l=1 rev1=1 rx=0x7fa100001c10 tx=0x7fa10000a8f0).stop 2021-02-10T11:36:40.206 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:40.203 7fa10c426700 1 -- 172.21.2.102:0/1254595117 shutdown_connections 2021-02-10T11:36:40.206 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:40.203 7fa10c426700 1 --2- 172.21.2.102:0/1254595117 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa1040ac900 0x7fa1040bfbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:40.206 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:40.203 7fa10c426700 1 --2- 172.21.2.102:0/1254595117 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa0ec018730 0x7fa0ec01ac10 unknown :-1 s=CLOSED pgs=1247 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:40.206 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:40.203 7fa10c426700 1 --2- 172.21.2.102:0/1254595117 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa1040b54a0 0x7fa1040bf5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:40.206 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:40.204 7fa10c426700 1 --2- 172.21.2.102:0/1254595117 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa1040b50a0 0x7fa1040ac2e0 unknown :-1 s=CLOSED pgs=2842 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:40.206 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:40.204 7fa10c426700 1 -- 172.21.2.102:0/1254595117 >> 172.21.2.102:0/1254595117 conn(0x7fa1040a1870 msgr2=0x7fa10409e300 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:40.206 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:40.204 7fa10c426700 1 -- 172.21.2.102:0/1254595117 shutdown_connections 2021-02-10T11:36:40.207 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:40.204 7fa10c426700 1 -- 172.21.2.102:0/1254595117 wait complete. 2021-02-10T11:36:40.214 INFO:tasks.ceph.mds.a:Restarting daemon 2021-02-10T11:36:40.214 INFO:tasks.ceph.mds.a:Stopping old one... 2021-02-10T11:36:40.215 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2021-02-10T11:36:40.215 INFO:teuthology.orchestra.run:waiting for 300 2021-02-10T11:36:40.215 INFO:tasks.ceph.mds.b:Restarting daemon 2021-02-10T11:36:40.215 INFO:tasks.ceph.mds.b:Stopping old one... 2021-02-10T11:36:40.215 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2021-02-10T11:36:40.215 INFO:teuthology.orchestra.run:waiting for 300 2021-02-10T11:36:40.221 INFO:tasks.ceph.mds.a:Stopped 2021-02-10T11:36:40.222 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2021-02-10T11:36:40.222 INFO:tasks.ceph.mds.b:Stopped 2021-02-10T11:36:40.223 DEBUG:teuthology.orchestra.run.gibba013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2021-02-10T11:36:40.265 INFO:tasks.ceph.mds.a:Started 2021-02-10T11:36:40.265 DEBUG:teuthology.parallel:result is None 2021-02-10T11:36:40.266 INFO:tasks.ceph.mds.b:Started 2021-02-10T11:36:40.266 DEBUG:teuthology.parallel:result is None 2021-02-10T11:36:40.266 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd blacklist clear 2021-02-10T11:36:40.669 INFO:tasks.ceph.mds.b.gibba013.stdout:starting mds.b at 2021-02-10T11:36:40.674 INFO:tasks.ceph.mds.a.gibba002.stdout:starting mds.a at 2021-02-10T11:36:40.694 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:40.675 7f228625e700 1 Processor -- start 2021-02-10T11:36:40.694 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:40.675 7f228625e700 1 -- start start 2021-02-10T11:36:40.694 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:40.675 7f228625e700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f22800d1860 0x7f22801b03e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:40.694 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:40.675 7f228625e700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22800f0b00 con 0x7f22800f0640 2021-02-10T11:36:40.694 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:40.675 7f228625e700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22800d0300 con 0x7f22801b0a00 2021-02-10T11:36:40.694 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:40.675 7f228625e700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22800d0520 con 0x7f22800d1860 2021-02-10T11:36:40.694 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:40.676 7f227ffff700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f22800d1860 0x7f22801b03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:40.694 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:40.676 7f227ffff700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f22800d1860 0x7f22801b03e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.113:3300/0 says I am v2:172.21.2.102:41392/0 (socket says 172.21.2.102:41392) 2021-02-10T11:36:40.694 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:40.676 7f227ffff700 1 -- 172.21.2.102:0/2745763901 learned_addr learned my addr 172.21.2.102:0/2745763901 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:36:40.694 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:40.676 7f227f7fe700 1 -- 172.21.2.102:0/2745763901 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2289341500 0 0) 0x7f2278000f90 con 0x7f22800f0640 2021-02-10T11:36:40.695 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:40.676 7f227f7fe700 1 -- 172.21.2.102:0/2745763901 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f226c002fe0 con 0x7f22800f0640 2021-02-10T11:36:40.695 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:40.676 7f227ffff700 1 -- 172.21.2.102:0/2745763901 >> v1:172.21.2.113:6789/0 conn(0x7f22801b0a00 legacy=0x7f22800cf320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:40.695 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:40.676 7f227ffff700 1 -- 172.21.2.102:0/2745763901 >> v1:172.21.2.102:6789/0 conn(0x7f22800f0640 legacy=0x7f22800cfb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:40.695 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:40.676 7f227ffff700 1 -- 172.21.2.102:0/2745763901 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22800d2fd0 con 0x7f22800d1860 2021-02-10T11:36:40.695 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:40.676 7f227ffff700 1 --2- 172.21.2.102:0/2745763901 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f22800d1860 0x7f22801b03e0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 rx=0x7f2270006700 tx=0x7f2270009b80).ready entity=mon.1 client_cookie=e1f4dce99dd7954 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:40.695 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:40.676 7f227f7fe700 1 -- 172.21.2.102:0/2745763901 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f227000c0c0 con 0x7f22800d1860 2021-02-10T11:36:40.695 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:40.676 7f227f7fe700 1 -- 172.21.2.102:0/2745763901 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f227000c340 con 0x7f22800d1860 2021-02-10T11:36:40.695 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:40.676 7f227f7fe700 1 -- 172.21.2.102:0/2745763901 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f227000c860 con 0x7f22800d1860 2021-02-10T11:36:40.695 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:40.676 7f228625e700 1 -- 172.21.2.102:0/2745763901 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f22800d1860 msgr2=0x7f22801b03e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:40.695 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:40.676 7f228625e700 1 --2- 172.21.2.102:0/2745763901 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f22800d1860 0x7f22801b03e0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 rx=0x7f2270006700 tx=0x7f2270009b80).stop 2021-02-10T11:36:40.695 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:40.677 7f228625e700 1 -- 172.21.2.102:0/2745763901 shutdown_connections 2021-02-10T11:36:40.695 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:40.677 7f228625e700 1 --2- 172.21.2.102:0/2745763901 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f22800d1860 0x7f22801b03e0 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:40.696 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:40.677 7f228625e700 1 -- 172.21.2.102:0/2745763901 >> 172.21.2.102:0/2745763901 conn(0x7f22800ce440 msgr2=0x7f22800d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:40.696 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:40.677 7f228625e700 1 -- 172.21.2.102:0/2745763901 shutdown_connections 2021-02-10T11:36:40.696 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:40.677 7f228625e700 1 -- 172.21.2.102:0/2745763901 wait complete. 2021-02-10T11:36:40.697 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:40.692 7f228625e700 1 Processor -- start 2021-02-10T11:36:40.697 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:40.692 7f228625e700 1 -- start start 2021-02-10T11:36:40.697 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:40.692 7f228625e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f22800c2cf0 0x7f22800960a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:40.698 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:40.692 7f228625e700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f22801a7450 0x7f22801a7850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:40.698 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:40.692 7f228625e700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f22801a7e70 0x7f2280099a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:40.698 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:40.692 7f228625e700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22801a8270 con 0x7f22800c2cf0 2021-02-10T11:36:40.698 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:40.692 7f228625e700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22800968d0 con 0x7f22801a7450 2021-02-10T11:36:40.698 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:40.692 7f228625e700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f228009a040 con 0x7f22801a7e70 2021-02-10T11:36:40.698 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:40.693 7f227ffff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f22800c2cf0 0x7f22800960a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:40.699 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:40.693 7f227ffff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f22800c2cf0 0x7f22800960a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:37968/0 (socket says 172.21.2.102:37968) 2021-02-10T11:36:40.699 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:40.693 7f227ffff700 1 -- 172.21.2.102:0/3811649845 learned_addr learned my addr 172.21.2.102:0/3811649845 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:36:40.699 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:40.693 7f227ffff700 1 -- 172.21.2.102:0/3811649845 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f22801a7e70 msgr2=0x7f2280099a20 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:36:40.699 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:40.693 7f227ffff700 1 --2- 172.21.2.102:0/3811649845 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f22801a7e70 0x7f2280099a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:40.699 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:40.693 7f227ffff700 1 -- 172.21.2.102:0/3811649845 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f22801a7450 msgr2=0x7f22801a7850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:40.699 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:40.693 7f227ffff700 1 --2- 172.21.2.102:0/3811649845 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f22801a7450 0x7f22801a7850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:40.699 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:40.693 7f227ffff700 1 -- 172.21.2.102:0/3811649845 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f228009a220 con 0x7f22800c2cf0 2021-02-10T11:36:40.699 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:40.693 7f227ffff700 1 --2- 172.21.2.102:0/3811649845 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f22800c2cf0 0x7f22800960a0 secure :-1 s=READY pgs=2846 cs=0 l=1 rev1=1 rx=0x7f227000b650 tx=0x7f227000b9f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:40.699 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:40.693 7f227d7fa700 1 -- 172.21.2.102:0/3811649845 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f22700009b0 con 0x7f22800c2cf0 2021-02-10T11:36:40.700 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:40.693 7f227d7fa700 1 -- 172.21.2.102:0/3811649845 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2270006570 con 0x7f22800c2cf0 2021-02-10T11:36:40.700 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:40.693 7f227d7fa700 1 -- 172.21.2.102:0/3811649845 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2270002030 con 0x7f22800c2cf0 2021-02-10T11:36:40.700 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:40.693 7f228625e700 1 -- 172.21.2.102:0/3811649845 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f228009a220 con 0x7f22800c2cf0 2021-02-10T11:36:40.700 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:40.693 7f228625e700 1 -- 172.21.2.102:0/3811649845 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f228009a220 con 0x7f22800c2cf0 2021-02-10T11:36:40.700 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:40.694 7f227d7fa700 1 -- 172.21.2.102:0/3811649845 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f2270009f90 con 0x7f22800c2cf0 2021-02-10T11:36:40.700 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:40.694 7f227d7fa700 1 --2- 172.21.2.102:0/3811649845 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f22640186c0 0x7f226401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:40.700 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:40.694 7f227d7fa700 1 -- 172.21.2.102:0/3811649845 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(156..156 src has 1..156) v4 ==== 4839+0+0 (secure 0 0 0) 0x7f2270032290 con 0x7f22800c2cf0 2021-02-10T11:36:40.700 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:40.694 7f2277fff700 1 --2- 172.21.2.102:0/3811649845 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f22640186c0 0x7f226401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:40.700 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:40.694 7f2277fff700 1 --2- 172.21.2.102:0/3811649845 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f22640186c0 0x7f226401abc0 crc :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:40.726 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:40.724 7f228625e700 1 -- 172.21.2.102:0/3811649845 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22800ee060 con 0x7f22800c2cf0 2021-02-10T11:36:40.729 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:40.727 7f227d7fa700 1 -- 172.21.2.102:0/3811649845 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f2270000c90 con 0x7f22800c2cf0 2021-02-10T11:36:40.870 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:40.868 7f228625e700 1 -- 172.21.2.102:0/3811649845 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd blacklist clear"} v 0) v1 -- 0x7f22800ce440 con 0x7f22800c2cf0 2021-02-10T11:36:41.221 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:41.219 7f227d7fa700 1 -- 172.21.2.102:0/3811649845 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd blacklist clear"}]=0 removed all blacklist entries v157) v1 ==== 97+0+0 (secure 0 0 0) 0x7f2270009f90 con 0x7f22800c2cf0 2021-02-10T11:36:41.236 INFO:teuthology.orchestra.run.gibba002.stderr: removed all blacklist entries 2021-02-10T11:36:41.239 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:41.236 7f228625e700 1 -- 172.21.2.102:0/3811649845 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f22640186c0 msgr2=0x7f226401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:41.239 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:41.236 7f228625e700 1 --2- 172.21.2.102:0/3811649845 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f22640186c0 0x7f226401abc0 crc :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:41.239 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:41.236 7f228625e700 1 -- 172.21.2.102:0/3811649845 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f22800c2cf0 msgr2=0x7f22800960a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:41.239 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:41.236 7f228625e700 1 --2- 172.21.2.102:0/3811649845 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f22800c2cf0 0x7f22800960a0 secure :-1 s=READY pgs=2846 cs=0 l=1 rev1=1 rx=0x7f227000b650 tx=0x7f227000b9f0).stop 2021-02-10T11:36:41.239 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:41.237 7f228625e700 1 -- 172.21.2.102:0/3811649845 shutdown_connections 2021-02-10T11:36:41.239 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:41.237 7f228625e700 1 --2- 172.21.2.102:0/3811649845 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f22801a7e70 0x7f2280099a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:41.239 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:41.237 7f228625e700 1 --2- 172.21.2.102:0/3811649845 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f22640186c0 0x7f226401abc0 unknown :-1 s=CLOSED pgs=1248 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:41.239 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:41.237 7f228625e700 1 --2- 172.21.2.102:0/3811649845 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f22801a7450 0x7f22801a7850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:41.239 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:41.237 7f228625e700 1 --2- 172.21.2.102:0/3811649845 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f22800c2cf0 0x7f22800960a0 unknown :-1 s=CLOSED pgs=2846 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:41.239 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:41.237 7f228625e700 1 -- 172.21.2.102:0/3811649845 >> 172.21.2.102:0/3811649845 conn(0x7f22800be6a0 msgr2=0x7f22800b6080 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:41.239 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:41.237 7f228625e700 1 -- 172.21.2.102:0/3811649845 shutdown_connections 2021-02-10T11:36:41.240 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:41.237 7f228625e700 1 -- 172.21.2.102:0/3811649845 wait complete. 2021-02-10T11:36:41.247 INFO:tasks.cephfs.cephfs_test_case:['0', '1', '2', '3'] 2021-02-10T11:36:41.247 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth ls --format=json-pretty 2021-02-10T11:36:41.677 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:41.658 7fbd68844700 1 Processor -- start 2021-02-10T11:36:41.677 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:41.658 7fbd68844700 1 -- start start 2021-02-10T11:36:41.677 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:41.658 7fbd68844700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbd640d0340 0x7fbd640ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:41.678 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:41.658 7fbd68844700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fbd640d1c70 0x7fbd640efc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:41.678 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:41.658 7fbd68844700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd640f0b10 con 0x7fbd640d1870 2021-02-10T11:36:41.678 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:41.658 7fbd68844700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd640e5ed0 con 0x7fbd640d1c70 2021-02-10T11:36:41.678 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:41.658 7fbd68844700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd641b0be0 con 0x7fbd640d0340 2021-02-10T11:36:41.678 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:41.659 7fbd62e1d700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fbd640d1c70 0x7fbd640efc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:41.678 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:41.659 7fbd62e1d700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fbd640d1c70 0x7fbd640efc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:37976/0 (socket says 172.21.2.102:37976) 2021-02-10T11:36:41.678 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:41.659 7fbd62e1d700 1 -- 172.21.2.102:0/102712909 learned_addr learned my addr 172.21.2.102:0/102712909 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:36:41.678 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:41.659 7fbd62e1d700 1 -- 172.21.2.102:0/102712909 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbd640d0340 msgr2=0x7fbd640ef640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:41.678 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:41.659 7fbd61e1b700 1 --2- 172.21.2.102:0/102712909 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbd640d0340 0x7fbd640ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:41.678 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:41.659 7fbd62e1d700 1 --2- 172.21.2.102:0/102712909 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbd640d0340 0x7fbd640ef640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:41.678 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:41.659 7fbd62e1d700 1 -- 172.21.2.102:0/102712909 >> v1:172.21.2.113:6789/0 conn(0x7fbd640d1870 legacy=0x7fbd641b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:41.678 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:41.659 7fbd62e1d700 1 -- 172.21.2.102:0/102712909 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd640f20a0 con 0x7fbd640d1c70 2021-02-10T11:36:41.679 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:41.659 7fbd61e1b700 1 --2- 172.21.2.102:0/102712909 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbd640d0340 0x7fbd640ef640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:36:41.679 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:41.660 7fbd62e1d700 1 --2- 172.21.2.102:0/102712909 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbd640d1c70 0x7fbd640efc60 secure :-1 s=READY pgs=2847 cs=0 l=1 rev1=1 rx=0x7fbd5c006f00 tx=0x7fbd5c009f10).ready entity=mon.0 client_cookie=d50f700c7ccf6f4 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:41.679 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:41.660 7fbd6161a700 1 -- 172.21.2.102:0/102712909 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbd5c00b3b0 con 0x7fbd640d1c70 2021-02-10T11:36:41.679 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:41.660 7fbd6161a700 1 -- 172.21.2.102:0/102712909 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd5c00b630 con 0x7fbd640d1c70 2021-02-10T11:36:41.679 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:41.660 7fbd6161a700 1 -- 172.21.2.102:0/102712909 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbd5c00bb50 con 0x7fbd640d1c70 2021-02-10T11:36:41.679 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:41.660 7fbd68844700 1 -- 172.21.2.102:0/102712909 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbd640d1c70 msgr2=0x7fbd640efc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:41.679 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:41.660 7fbd68844700 1 --2- 172.21.2.102:0/102712909 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbd640d1c70 0x7fbd640efc60 secure :-1 s=READY pgs=2847 cs=0 l=1 rev1=1 rx=0x7fbd5c006f00 tx=0x7fbd5c009f10).stop 2021-02-10T11:36:41.679 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:41.661 7fbd68844700 1 -- 172.21.2.102:0/102712909 shutdown_connections 2021-02-10T11:36:41.679 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:41.661 7fbd68844700 1 --2- 172.21.2.102:0/102712909 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbd640d0340 0x7fbd640ef640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:41.679 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:41.661 7fbd68844700 1 --2- 172.21.2.102:0/102712909 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbd640d1c70 0x7fbd640efc60 unknown :-1 s=CLOSED pgs=2847 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:41.679 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:41.661 7fbd68844700 1 -- 172.21.2.102:0/102712909 >> 172.21.2.102:0/102712909 conn(0x7fbd640ce450 msgr2=0x7fbd640d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:41.679 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:41.661 7fbd68844700 1 -- 172.21.2.102:0/102712909 shutdown_connections 2021-02-10T11:36:41.680 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:41.661 7fbd68844700 1 -- 172.21.2.102:0/102712909 wait complete. 2021-02-10T11:36:41.682 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:41.676 7fbd68844700 1 Processor -- start 2021-02-10T11:36:41.682 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:41.676 7fbd68844700 1 -- start start 2021-02-10T11:36:41.682 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:41.676 7fbd68844700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbd641b8ad0 0x7fbd64187e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:41.682 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:41.676 7fbd68844700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fbd641b8ed0 0x7fbd640b1af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:41.682 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:41.676 7fbd68844700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbd640b2110 0x7fbd641b7a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:41.682 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:41.676 7fbd68844700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd64188470 con 0x7fbd641b8ad0 2021-02-10T11:36:41.682 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:41.676 7fbd68844700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd6418be10 con 0x7fbd641b8ed0 2021-02-10T11:36:41.682 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:41.676 7fbd68844700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd6418c030 con 0x7fbd640b2110 2021-02-10T11:36:41.683 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:41.676 7fbd6261c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbd641b8ad0 0x7fbd64187e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:41.683 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:41.676 7fbd6261c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbd641b8ad0 0x7fbd64187e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:37984/0 (socket says 172.21.2.102:37984) 2021-02-10T11:36:41.683 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:41.676 7fbd6261c700 1 -- 172.21.2.102:0/354124768 learned_addr learned my addr 172.21.2.102:0/354124768 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:36:41.683 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:41.677 7fbd62e1d700 1 --2- 172.21.2.102:0/354124768 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbd640b2110 0x7fbd641b7a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:41.683 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:41.677 7fbd6261c700 1 -- 172.21.2.102:0/354124768 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbd640b2110 msgr2=0x7fbd641b7a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:41.683 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:41.677 7fbd6261c700 1 --2- 172.21.2.102:0/354124768 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbd640b2110 0x7fbd641b7a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:41.683 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:41.677 7fbd61e1b700 1 --2- 172.21.2.102:0/354124768 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fbd641b8ed0 0x7fbd640b1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:41.683 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:41.677 7fbd6261c700 1 -- 172.21.2.102:0/354124768 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fbd641b8ed0 msgr2=0x7fbd640b1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:41.683 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:41.677 7fbd6261c700 1 --2- 172.21.2.102:0/354124768 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fbd641b8ed0 0x7fbd640b1af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:41.683 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:41.677 7fbd6261c700 1 -- 172.21.2.102:0/354124768 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd640b2510 con 0x7fbd641b8ad0 2021-02-10T11:36:41.683 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:41.677 7fbd62e1d700 1 --2- 172.21.2.102:0/354124768 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbd640b2110 0x7fbd641b7a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:36:41.683 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:41.677 7fbd61e1b700 1 --2- 172.21.2.102:0/354124768 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fbd641b8ed0 0x7fbd640b1af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:36:41.683 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:41.677 7fbd6261c700 1 --2- 172.21.2.102:0/354124768 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbd641b8ad0 0x7fbd64187e50 secure :-1 s=READY pgs=2848 cs=0 l=1 rev1=1 rx=0x7fbd54006ed0 tx=0x7fbd5400b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:41.684 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:41.677 7fbd52ffd700 1 -- 172.21.2.102:0/354124768 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbd54006240 con 0x7fbd641b8ad0 2021-02-10T11:36:41.684 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:41.677 7fbd52ffd700 1 -- 172.21.2.102:0/354124768 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd540009b0 con 0x7fbd641b8ad0 2021-02-10T11:36:41.684 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:41.677 7fbd52ffd700 1 -- 172.21.2.102:0/354124768 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbd54005730 con 0x7fbd641b8ad0 2021-02-10T11:36:41.684 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:41.677 7fbd68844700 1 -- 172.21.2.102:0/354124768 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd640b2510 con 0x7fbd641b8ad0 2021-02-10T11:36:41.684 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:41.677 7fbd68844700 1 -- 172.21.2.102:0/354124768 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd640b2510 con 0x7fbd641b8ad0 2021-02-10T11:36:41.684 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:41.678 7fbd52ffd700 1 -- 172.21.2.102:0/354124768 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fbd54006030 con 0x7fbd641b8ad0 2021-02-10T11:36:41.684 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:41.678 7fbd52ffd700 1 --2- 172.21.2.102:0/354124768 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fbd480186c0 0x7fbd4801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:41.684 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:41.678 7fbd52ffd700 1 -- 172.21.2.102:0/354124768 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(157..157 src has 1..157) v4 ==== 4753+0+0 (secure 0 0 0) 0x7fbd54032210 con 0x7fbd641b8ad0 2021-02-10T11:36:41.684 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:41.678 7fbd61e1b700 1 --2- 172.21.2.102:0/354124768 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fbd480186c0 0x7fbd4801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:41.684 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:41.679 7fbd61e1b700 1 --2- 172.21.2.102:0/354124768 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fbd480186c0 0x7fbd4801abc0 crc :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:41.711 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:41.709 7fbd68844700 1 -- 172.21.2.102:0/354124768 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd640ebc00 con 0x7fbd641b8ad0 2021-02-10T11:36:41.713 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:41.711 7fbd52ffd700 1 -- 172.21.2.102:0/354124768 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fbd54000d00 con 0x7fbd641b8ad0 2021-02-10T11:36:41.846 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:41.844 7fbd68844700 1 -- 172.21.2.102:0/354124768 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fbd640ebc00 con 0x7fbd641b8ad0 2021-02-10T11:36:41.847 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:41.844 7fbd52ffd700 1 -- 172.21.2.102:0/354124768 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v88) v1 ==== 80+0+4513 (secure 0 0 0) 0x7fbd54006030 con 0x7fbd641b8ad0 2021-02-10T11:36:41.847 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:36:41.847 INFO:teuthology.orchestra.run.gibba002.stdout:{ 2021-02-10T11:36:41.847 INFO:teuthology.orchestra.run.gibba002.stdout: "auth_dump": [ 2021-02-10T11:36:41.847 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:36:41.847 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "mds.a", 2021-02-10T11:36:41.848 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAevSNgsgyEARAAvr0cUjjCoEHLxaZSZ2HtVA==", 2021-02-10T11:36:41.848 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:36:41.848 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow", 2021-02-10T11:36:41.848 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:36:41.848 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:36:41.848 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:36:41.848 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:36:41.848 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:36:41.848 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:36:41.848 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "mds.b", 2021-02-10T11:36:41.848 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAevSNgwOFMMRAALfb1fbX56ao1TU8ClnTW1A==", 2021-02-10T11:36:41.849 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:36:41.849 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow", 2021-02-10T11:36:41.849 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:36:41.849 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:36:41.849 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:36:41.849 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:36:41.849 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:36:41.849 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:36:41.849 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.0", 2021-02-10T11:36:41.849 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAmvSNgj+bMMBAAD3HUD6fLiMAg+++iNzLJKQ==", 2021-02-10T11:36:41.849 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:36:41.849 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:36:41.850 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:36:41.850 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:36:41.850 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:36:41.850 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:36:41.850 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:36:41.850 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.1", 2021-02-10T11:36:41.850 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQApvSNgnvc+MRAAAOxqlNwiyLHqNITOrYeFKQ==", 2021-02-10T11:36:41.850 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:36:41.850 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:36:41.850 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:36:41.850 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:36:41.851 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:36:41.851 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:36:41.851 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:36:41.851 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.2", 2021-02-10T11:36:41.851 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAsvSNgZ4SbMRAA76IOYWfbiAuBoM/4k81GxA==", 2021-02-10T11:36:41.851 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:36:41.851 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:36:41.851 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:36:41.851 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:36:41.851 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:36:41.851 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:36:41.851 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:36:41.852 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.3", 2021-02-10T11:36:41.852 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAvvSNgFk3rMRAAWCmKKyw0LDQ+UlAHZAITOw==", 2021-02-10T11:36:41.852 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:36:41.852 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:36:41.852 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:36:41.852 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:36:41.852 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:36:41.852 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:36:41.852 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:36:41.852 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.4", 2021-02-10T11:36:41.852 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQA4vSNgpKcNJBAApJMmvj5YxVHPDOmvmQONOg==", 2021-02-10T11:36:41.852 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:36:41.853 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:36:41.853 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:36:41.853 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:36:41.853 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:36:41.853 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:36:41.853 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:36:41.853 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.5", 2021-02-10T11:36:41.853 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQA7vSNgf2ujJBAAfkuYpyb3WTzc8V5kw+Wktg==", 2021-02-10T11:36:41.853 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:36:41.853 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:36:41.853 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:36:41.853 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:36:41.854 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:36:41.854 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:36:41.854 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:36:41.854 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.6", 2021-02-10T11:36:41.854 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQA+vSNgqw9uJRAAFMU0R+UxU9C3o/9byb3e0g==", 2021-02-10T11:36:41.854 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:36:41.854 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:36:41.854 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:36:41.854 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:36:41.854 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:36:41.854 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:36:41.855 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:36:41.855 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.7", 2021-02-10T11:36:41.855 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQBBvSNgvWSrJRAA2KaCHmLpzovRgYzLvyofTQ==", 2021-02-10T11:36:41.855 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:36:41.855 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:36:41.855 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:36:41.855 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:36:41.855 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:36:41.855 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:36:41.855 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:36:41.855 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "client.0", 2021-02-10T11:36:41.855 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAfvSNgZd9XIBAAvyrvl8fdKuHnJZfczfRoFw==", 2021-02-10T11:36:41.856 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:36:41.856 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow", 2021-02-10T11:36:41.856 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow r", 2021-02-10T11:36:41.856 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:36:41.856 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:36:41.856 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:36:41.856 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:36:41.856 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "client.1", 2021-02-10T11:36:41.856 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAfvSNgSyfXNhAAIduPnF7Vd/6HSkJ0MlcMSA==", 2021-02-10T11:36:41.856 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:36:41.856 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow", 2021-02-10T11:36:41.856 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow r", 2021-02-10T11:36:41.857 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:36:41.857 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:36:41.857 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:36:41.857 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:36:41.857 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "client.2", 2021-02-10T11:36:41.857 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAgvSNgiIG/ERAA9ZTamaTI7ZsPSnxBqjplCg==", 2021-02-10T11:36:41.857 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:36:41.857 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow", 2021-02-10T11:36:41.857 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow r", 2021-02-10T11:36:41.857 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:36:41.857 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:36:41.858 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:36:41.858 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:36:41.858 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "client.3", 2021-02-10T11:36:41.858 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAgvSNgOWpOKBAAlXsEB04Q/GvtL06PqRMLeA==", 2021-02-10T11:36:41.858 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:36:41.858 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow", 2021-02-10T11:36:41.858 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow r", 2021-02-10T11:36:41.858 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:36:41.858 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:36:41.859 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:36:41.859 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:36:41.859 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "client.admin", 2021-02-10T11:36:41.859 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAavSNgO3GQBhAAL+NVnYskQRKVWLgyDPY/xw==", 2021-02-10T11:36:41.859 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:36:41.859 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow *", 2021-02-10T11:36:41.859 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:36:41.859 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:36:41.859 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:36:41.859 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:36:41.859 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:36:41.860 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:36:41.860 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "mgr.x", 2021-02-10T11:36:41.860 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAdvSNgXZZgJhAA7dtBSGzH+DsBwnddZpOBJA==", 2021-02-10T11:36:41.860 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:36:41.860 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow *", 2021-02-10T11:36:41.860 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow profile mgr", 2021-02-10T11:36:41.860 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:36:41.860 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:36:41.860 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:36:41.860 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:36:41.860 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "mgr.y", 2021-02-10T11:36:41.861 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAdvSNgVqQFDRAAwo6YEdWZCAr5GdOWVL/Qvg==", 2021-02-10T11:36:41.861 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:36:41.861 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow *", 2021-02-10T11:36:41.861 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow profile mgr", 2021-02-10T11:36:41.861 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:36:41.861 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:36:41.861 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:36:41.861 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:36:41.861 INFO:teuthology.orchestra.run.gibba002.stdout:} 2021-02-10T11:36:41.862 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:41.846 7fbd68844700 1 -- 172.21.2.102:0/354124768 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fbd480186c0 msgr2=0x7fbd4801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:41.863 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:41.846 7fbd68844700 1 --2- 172.21.2.102:0/354124768 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fbd480186c0 0x7fbd4801abc0 crc :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:41.863 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:41.846 7fbd68844700 1 -- 172.21.2.102:0/354124768 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbd641b8ad0 msgr2=0x7fbd64187e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:41.863 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:41.846 7fbd68844700 1 --2- 172.21.2.102:0/354124768 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbd641b8ad0 0x7fbd64187e50 secure :-1 s=READY pgs=2848 cs=0 l=1 rev1=1 rx=0x7fbd54006ed0 tx=0x7fbd5400b960).stop 2021-02-10T11:36:41.863 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:41.847 7fbd68844700 1 -- 172.21.2.102:0/354124768 shutdown_connections 2021-02-10T11:36:41.863 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:41.847 7fbd68844700 1 --2- 172.21.2.102:0/354124768 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fbd640b2110 0x7fbd641b7a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:41.863 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:41.847 7fbd68844700 1 --2- 172.21.2.102:0/354124768 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fbd480186c0 0x7fbd4801abc0 unknown :-1 s=CLOSED pgs=1249 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:41.863 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:41.847 7fbd68844700 1 --2- 172.21.2.102:0/354124768 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fbd641b8ed0 0x7fbd640b1af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:41.863 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:41.847 7fbd68844700 1 --2- 172.21.2.102:0/354124768 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fbd641b8ad0 0x7fbd64187e50 unknown :-1 s=CLOSED pgs=2848 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:41.863 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:41.847 7fbd68844700 1 -- 172.21.2.102:0/354124768 >> 172.21.2.102:0/354124768 conn(0x7fbd640ed810 msgr2=0x7fbd64094980 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:41.863 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:41.847 7fbd68844700 1 -- 172.21.2.102:0/354124768 shutdown_connections 2021-02-10T11:36:41.863 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:41.847 7fbd68844700 1 -- 172.21.2.102:0/354124768 wait complete. 2021-02-10T11:36:41.864 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:36:42.297 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:42.276 7f0549466700 1 Processor -- start 2021-02-10T11:36:42.298 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:42.276 7f0549466700 1 -- start start 2021-02-10T11:36:42.298 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:42.277 7f0549466700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f05440d1870 0x7f05441b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:42.298 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:42.277 7f0549466700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f05441b0a10 0x7f05440cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:42.298 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:42.277 7f0549466700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05440f0b10 con 0x7f05440d1c70 2021-02-10T11:36:42.298 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:42.277 7f0549466700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05440fe230 con 0x7f05441b0a10 2021-02-10T11:36:42.298 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:42.277 7f0549466700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05440e5ed0 con 0x7f05440d1870 2021-02-10T11:36:42.298 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:42.277 7f054387e700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f05440d1c70 0x7f05440cf950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:58444/0 (socket says 172.21.2.102:58444) 2021-02-10T11:36:42.298 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:42.277 7f054387e700 1 -- 172.21.2.102:0/14344781 learned_addr learned my addr 172.21.2.102:0/14344781 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:36:42.298 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:42.277 7f054207b700 1 -- 172.21.2.102:0/14344781 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1746900667 0 0) 0x7f053c001500 con 0x7f05440d1c70 2021-02-10T11:36:42.298 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:42.277 7f054207b700 1 -- 172.21.2.102:0/14344781 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0530002fe0 con 0x7f05440d1c70 2021-02-10T11:36:42.299 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:42.277 7f054287c700 1 --2- 172.21.2.102:0/14344781 >> v2:172.21.2.113:3300/0 conn(0x7f05441b0a10 0x7f05440cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:42.299 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:42.278 7f054307d700 1 --2- 172.21.2.102:0/14344781 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f05440d1870 0x7f05441b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:42.299 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:42.278 7f054207b700 1 -- 172.21.2.102:0/14344781 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2380836178 0 0) 0x7f053c001500 con 0x7f05440d1c70 2021-02-10T11:36:42.299 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:42.278 7f054207b700 1 -- 172.21.2.102:0/14344781 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f05440d1870 msgr2=0x7f05441b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:42.299 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:42.278 7f054207b700 1 --2- 172.21.2.102:0/14344781 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f05440d1870 0x7f05441b03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:42.299 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:42.278 7f054207b700 1 -- 172.21.2.102:0/14344781 >> v2:172.21.2.113:3300/0 conn(0x7f05441b0a10 msgr2=0x7f05440cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:42.299 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:42.278 7f054207b700 1 --2- 172.21.2.102:0/14344781 >> v2:172.21.2.113:3300/0 conn(0x7f05441b0a10 0x7f05440cf330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:42.299 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:42.278 7f054207b700 1 -- 172.21.2.102:0/14344781 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05440f0650 con 0x7f05440d1c70 2021-02-10T11:36:42.299 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:42.278 7f054207b700 1 -- 172.21.2.102:0/14344781 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7f053c001a00 con 0x7f05440d1c70 2021-02-10T11:36:42.299 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:42.278 7f054207b700 1 -- 172.21.2.102:0/14344781 >> v1:172.21.2.102:6789/0 conn(0x7f05440d1c70 legacy=0x7f05440cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:42.299 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:42.278 7f054207b700 1 --2- 172.21.2.102:0/14344781 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0530003af0 0x7f0530005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:42.299 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:42.278 7f054207b700 1 --2- 172.21.2.102:0/14344781 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0530006840 0x7f0530008d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:42.299 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:42.278 7f054207b700 1 --2- 172.21.2.102:0/14344781 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0530009640 0x7f053000bb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:42.299 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:42.278 7f054207b700 1 -- 172.21.2.102:0/14344781 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f053000c380 con 0x7f0530006840 2021-02-10T11:36:42.300 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:42.278 7f054207b700 1 -- 172.21.2.102:0/14344781 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0530050570 con 0x7f0530003af0 2021-02-10T11:36:42.300 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:42.278 7f054207b700 1 -- 172.21.2.102:0/14344781 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0530050790 con 0x7f0530009640 2021-02-10T11:36:42.300 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:42.278 7f054287c700 1 --2- 172.21.2.102:0/14344781 >> v2:172.21.2.113:3300/0 conn(0x7f05441b0a10 0x7f05440cf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:36:42.300 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:42.278 7f054307d700 1 --2- 172.21.2.102:0/14344781 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0530003af0 0x7f0530005fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:42.300 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:42.278 7f054287c700 1 --2- 172.21.2.102:0/14344781 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0530006840 0x7f0530008d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:42.300 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:42.278 7f054387e700 1 --2- 172.21.2.102:0/14344781 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0530009640 0x7f053000bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:42.300 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:42.278 7f054287c700 1 -- 172.21.2.102:0/14344781 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0530009640 msgr2=0x7f053000bb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:42.300 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:42.278 7f054287c700 1 --2- 172.21.2.102:0/14344781 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0530009640 0x7f053000bb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:42.300 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:42.278 7f054287c700 1 -- 172.21.2.102:0/14344781 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0530003af0 msgr2=0x7f0530005fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:42.300 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:42.278 7f054287c700 1 --2- 172.21.2.102:0/14344781 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0530003af0 0x7f0530005fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:42.300 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:42.278 7f054287c700 1 -- 172.21.2.102:0/14344781 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0530050a30 con 0x7f0530006840 2021-02-10T11:36:42.300 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:42.278 7f054287c700 1 --2- 172.21.2.102:0/14344781 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0530006840 0x7f0530008d40 secure :-1 s=READY pgs=2850 cs=0 l=1 rev1=1 rx=0x7f0534009950 tx=0x7f0534009cf0).ready entity=mon.0 client_cookie=a00c82642a598e44 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:42.300 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:42.279 7f054207b700 1 -- 172.21.2.102:0/14344781 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0534000960 con 0x7f0530006840 2021-02-10T11:36:42.301 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:42.279 7f054207b700 1 -- 172.21.2.102:0/14344781 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0534000c90 con 0x7f0530006840 2021-02-10T11:36:42.301 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:42.279 7f054207b700 1 -- 172.21.2.102:0/14344781 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f05340022b0 con 0x7f0530006840 2021-02-10T11:36:42.301 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:42.279 7f0549466700 1 -- 172.21.2.102:0/14344781 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0530006840 msgr2=0x7f0530008d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:42.301 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:42.279 7f0549466700 1 --2- 172.21.2.102:0/14344781 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0530006840 0x7f0530008d40 secure :-1 s=READY pgs=2850 cs=0 l=1 rev1=1 rx=0x7f0534009950 tx=0x7f0534009cf0).stop 2021-02-10T11:36:42.301 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:42.279 7f054387e700 1 -- 172.21.2.102:0/14344781 reap_dead start 2021-02-10T11:36:42.301 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:42.279 7f0549466700 1 -- 172.21.2.102:0/14344781 shutdown_connections 2021-02-10T11:36:42.301 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:42.279 7f0549466700 1 -- 172.21.2.102:0/14344781 >> 172.21.2.102:0/14344781 conn(0x7f05440ce450 msgr2=0x7f05440d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:42.301 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:42.279 7f0549466700 1 -- 172.21.2.102:0/14344781 shutdown_connections 2021-02-10T11:36:42.301 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:42.279 7f0549466700 1 -- 172.21.2.102:0/14344781 wait complete. 2021-02-10T11:36:42.301 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:42.294 7f0549466700 1 Processor -- start 2021-02-10T11:36:42.301 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:42.294 7f0549466700 1 -- start start 2021-02-10T11:36:42.301 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:42.294 7f0549466700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f05441a05d0 0x7f05440a87c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:42.302 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:42.294 7f0549466700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f05440a8de0 0x7f05440a2630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:42.302 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:42.294 7f0549466700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f05440a2c50 0x7f05440a3050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:42.302 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:42.294 7f0549466700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05440b32d0 con 0x7f05440a2c50 2021-02-10T11:36:42.302 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:42.294 7f0549466700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05440a91e0 con 0x7f05441a05d0 2021-02-10T11:36:42.302 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:42.294 7f0549466700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05440b37d0 con 0x7f05440a8de0 2021-02-10T11:36:42.302 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:42.295 7f054387e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f05440a2c50 0x7f05440a3050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:42.302 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:42.295 7f054387e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f05440a2c50 0x7f05440a3050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:38000/0 (socket says 172.21.2.102:38000) 2021-02-10T11:36:42.302 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:42.295 7f054387e700 1 -- 172.21.2.102:0/3213745615 learned_addr learned my addr 172.21.2.102:0/3213745615 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:36:42.302 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:42.295 7f054387e700 1 -- 172.21.2.102:0/3213745615 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f05440a8de0 msgr2=0x7f05440a2630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:42.302 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:42.295 7f054387e700 1 --2- 172.21.2.102:0/3213745615 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f05440a8de0 0x7f05440a2630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:42.302 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:42.295 7f054387e700 1 -- 172.21.2.102:0/3213745615 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f05441a05d0 msgr2=0x7f05440a87c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:42.302 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:42.295 7f054387e700 1 --2- 172.21.2.102:0/3213745615 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f05441a05d0 0x7f05440a87c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:42.303 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:42.295 7f054387e700 1 -- 172.21.2.102:0/3213745615 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05440b39b0 con 0x7f05440a2c50 2021-02-10T11:36:42.303 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:42.295 7f054387e700 1 --2- 172.21.2.102:0/3213745615 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f05440a2c50 0x7f05440a3050 secure :-1 s=READY pgs=2851 cs=0 l=1 rev1=1 rx=0x7f053c00a900 tx=0x7f053c00aca0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:42.303 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:42.295 7f053bfff700 1 -- 172.21.2.102:0/3213745615 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f053c007d90 con 0x7f05440a2c50 2021-02-10T11:36:42.303 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:42.295 7f053bfff700 1 -- 172.21.2.102:0/3213745615 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f053c0070d0 con 0x7f05440a2c50 2021-02-10T11:36:42.303 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:42.295 7f0549466700 1 -- 172.21.2.102:0/3213745615 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05440b39b0 con 0x7f05440a2c50 2021-02-10T11:36:42.303 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:42.295 7f053bfff700 1 -- 172.21.2.102:0/3213745615 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f053c004930 con 0x7f05440a2c50 2021-02-10T11:36:42.303 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:42.295 7f0549466700 1 -- 172.21.2.102:0/3213745615 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05440b3f40 con 0x7f05440a2c50 2021-02-10T11:36:42.304 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:42.296 7f053bfff700 1 -- 172.21.2.102:0/3213745615 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f053c007d10 con 0x7f05440a2c50 2021-02-10T11:36:42.304 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:42.296 7f053bfff700 1 --2- 172.21.2.102:0/3213745615 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f05280186c0 0x7f052801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:42.304 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:42.296 7f053bfff700 1 -- 172.21.2.102:0/3213745615 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(157..157 src has 1..157) v4 ==== 4753+0+0 (secure 0 0 0) 0x7f053c032620 con 0x7f05440a2c50 2021-02-10T11:36:42.304 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:42.296 7f054307d700 1 --2- 172.21.2.102:0/3213745615 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f05280186c0 0x7f052801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:42.304 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:42.296 7f054307d700 1 --2- 172.21.2.102:0/3213745615 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f05280186c0 0x7f052801abc0 crc :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:42.329 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:42.326 7f0549466700 1 -- 172.21.2.102:0/3213745615 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05440ebc80 con 0x7f05440a2c50 2021-02-10T11:36:42.331 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:42.329 7f053bfff700 1 -- 172.21.2.102:0/3213745615 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f053c007370 con 0x7f05440a2c50 2021-02-10T11:36:42.459 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:42.457 7f0549466700 1 -- 172.21.2.102:0/3213745615 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f05440f0b10 con 0x7f05440a2c50 2021-02-10T11:36:42.459 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:42.457 7f053bfff700 1 -- 172.21.2.102:0/3213745615 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v157) v1 ==== 74+0+11154 (secure 0 0 0) 0x7f053c007d10 con 0x7f05440a2c50 2021-02-10T11:36:42.460 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:36:42.460 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":157,"fsid":"68db30ea-dbce-4aef-a6df-f01c6fea0513","created":"2021-02-10 11:03:06.331557","modified":"2021-02-10 11:36:41.216847","last_up_change":"2021-02-10 11:03:32.690562","last_in_change":"2021-02-10 11:03:32.690562","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":33,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:03:37.322519","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"be691b4d-a039-408a-af80-9d53f02b6f1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":150,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6818","nonce":36088},{"type":"v1","addr":"172.21.2.102:6820","nonce":36088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6822","nonce":36088},{"type":"v1","addr":"172.21.2.102:6824","nonce":36088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6830","nonce":36088},{"type":"v1","addr":"172.21.2.102:6832","nonce":36088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6826","nonce":36088},{"type":"v1","addr":"172.21.2.102:6828","nonce":36088}]},"public_addr":"172.21.2.102:6820/36088","cluster_addr":"172.21.2.102:6824/36088","heartbeat_back_addr":"172.21.2.102:6832/36088","heartbeat_front_addr":"172.21.2.102:6828/36088","state":["exists","up"]},{"osd":1,"uuid":"64e3ccc8-9c69-4d7c-865e-af8e83822635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6819","nonce":36087},{"type":"v1","addr":"172.21.2.102:6821","nonce":36087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6823","nonce":36087},{"type":"v1","addr":"172.21.2.102:6825","nonce":36087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6831","nonce":36087},{"type":"v1","addr":"172.21.2.102:6833","nonce":36087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6827","nonce":36087},{"type":"v1","addr":"172.21.2.102:6829","nonce":36087}]},"public_addr":"172.21.2.102:6821/36087","cluster_addr":"172.21.2.102:6825/36087","heartbeat_back_addr":"172.21.2.102:6833/36087","heartbeat_front_addr":"172.21.2.102:6829/36087","state":["exists","up"]},{"osd":2,"uuid":"153700af-747f-481f-bdf4-d6072d3a842f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6810","nonce":36089},{"type":"v1","addr":"172.21.2.102:6811","nonce":36089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6812","nonce":36089},{"type":"v1","addr":"172.21.2.102:6813","nonce":36089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6816","nonce":36089},{"type":"v1","addr":"172.21.2.102:6817","nonce":36089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6814","nonce":36089},{"type":"v1","addr":"172.21.2.102:6815","nonce":36089}]},"public_addr":"172.21.2.102:6811/36089","cluster_addr":"172.21.2.102:6813/36089","heartbeat_back_addr":"172.21.2.102:6817/36089","heartbeat_front_addr":"172.21.2.102:6815/36089","state":["exists","up"]},{"osd":3,"uuid":"cbc678f4-83a9-45e6-8830-2f7c232b1ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6802","nonce":36090},{"type":"v1","addr":"172.21.2.102:6803","nonce":36090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6804","nonce":36090},{"type":"v1","addr":"172.21.2.102:6805","nonce":36090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6808","nonce":36090},{"type":"v1","addr":"172.21.2.102:6809","nonce":36090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6806","nonce":36090},{"type":"v1","addr":"172.21.2.102:6807","nonce":36090}]},"public_addr":"172.21.2.102:6803/36090","cluster_addr":"172.21.2.102:6805/36090","heartbeat_back_addr":"172.21.2.102:6809/36090","heartbeat_front_addr":"172.21.2.102:6807/36090","state":["exists","up"]},{"osd":4,"uuid":"c01e4008-b258-4f10-afd2-9e7f37214351","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6808","nonce":36368},{"type":"v1","addr":"172.21.2.113:6811","nonce":36368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6814","nonce":36368},{"type":"v1","addr":"172.21.2.113:6818","nonce":36368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6826","nonce":36368},{"type":"v1","addr":"172.21.2.113:6828","nonce":36368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6820","nonce":36368},{"type":"v1","addr":"172.21.2.113:6824","nonce":36368}]},"public_addr":"172.21.2.113:6811/36368","cluster_addr":"172.21.2.113:6818/36368","heartbeat_back_addr":"172.21.2.113:6828/36368","heartbeat_front_addr":"172.21.2.113:6824/36368","state":["exists","up"]},{"osd":5,"uuid":"551f6007-daf8-4a98-8ee0-c2b16bcccb5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6802","nonce":36366},{"type":"v1","addr":"172.21.2.113:6805","nonce":36366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6807","nonce":36366},{"type":"v1","addr":"172.21.2.113:6810","nonce":36366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6819","nonce":36366},{"type":"v1","addr":"172.21.2.113:6822","nonce":36366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6812","nonce":36366},{"type":"v1","addr":"172.21.2.113:6815","nonce":36366}]},"public_addr":"172.21.2.113:6805/36366","cluster_addr":"172.21.2.113:6810/36366","heartbeat_back_addr":"172.21.2.113:6822/36366","heartbeat_front_addr":"172.21.2.113:6815/36366","state":["exists","up"]},{"osd":6,"uuid":"bf67c040-0e50-48be-b102-c3943bc9ebef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":150,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6800","nonce":36367},{"type":"v1","addr":"172.21.2.113:6801","nonce":36367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6803","nonce":36367},{"type":"v1","addr":"172.21.2.113:6804","nonce":36367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6813","nonce":36367},{"type":"v1","addr":"172.21.2.113:6816","nonce":36367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6806","nonce":36367},{"type":"v1","addr":"172.21.2.113:6809","nonce":36367}]},"public_addr":"172.21.2.113:6801/36367","cluster_addr":"172.21.2.113:6804/36367","heartbeat_back_addr":"172.21.2.113:6816/36367","heartbeat_front_addr":"172.21.2.113:6809/36367","state":["exists","up"]},{"osd":7,"uuid":"f5461f80-018d-428f-be89-2eb2bcbe7bb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6817","nonce":36365},{"type":"v1","addr":"172.21.2.113:6821","nonce":36365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6823","nonce":36365},{"type":"v1","addr":"172.21.2.113:6825","nonce":36365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6830","nonce":36365},{"type":"v1","addr":"172.21.2.113:6831","nonce":36365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6827","nonce":36365},{"type":"v1","addr":"172.21.2.113:6829","nonce":36365}]},"public_addr":"172.21.2.113:6821/36365","cluster_addr":"172.21.2.113:6825/36365","heartbeat_back_addr":"172.21.2.113:6831/36365","heartbeat_front_addr":"172.21.2.113:6829/36365","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:36:42.461 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:42.459 7f0549466700 1 -- 172.21.2.102:0/3213745615 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f05280186c0 msgr2=0x7f052801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:42.461 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:42.459 7f0549466700 1 --2- 172.21.2.102:0/3213745615 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f05280186c0 0x7f052801abc0 crc :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:42.461 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:42.459 7f0549466700 1 -- 172.21.2.102:0/3213745615 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f05440a2c50 msgr2=0x7f05440a3050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:42.461 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:42.459 7f0549466700 1 --2- 172.21.2.102:0/3213745615 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f05440a2c50 0x7f05440a3050 secure :-1 s=READY pgs=2851 cs=0 l=1 rev1=1 rx=0x7f053c00a900 tx=0x7f053c00aca0).stop 2021-02-10T11:36:42.462 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:42.459 7f0549466700 1 -- 172.21.2.102:0/3213745615 shutdown_connections 2021-02-10T11:36:42.462 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:42.459 7f0549466700 1 --2- 172.21.2.102:0/3213745615 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f05440a2c50 0x7f05440a3050 unknown :-1 s=CLOSED pgs=2851 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:42.462 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:42.459 7f0549466700 1 --2- 172.21.2.102:0/3213745615 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f05440a8de0 0x7f05440a2630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:42.462 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:42.459 7f0549466700 1 --2- 172.21.2.102:0/3213745615 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f05280186c0 0x7f052801abc0 unknown :-1 s=CLOSED pgs=1250 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:42.462 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:42.459 7f0549466700 1 --2- 172.21.2.102:0/3213745615 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f05441a05d0 0x7f05440a87c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:42.462 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:42.459 7f0549466700 1 -- 172.21.2.102:0/3213745615 >> 172.21.2.102:0/3213745615 conn(0x7f05440a9bf0 msgr2=0x7f05440ebeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:42.462 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:42.459 7f0549466700 1 -- 172.21.2.102:0/3213745615 shutdown_connections 2021-02-10T11:36:42.462 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:42.459 7f0549466700 1 -- 172.21.2.102:0/3213745615 wait complete. 2021-02-10T11:36:42.470 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:03:37.322519', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2021-02-10T11:36:42.470 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:36:42.906 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:42.886 7fee0a342700 1 Processor -- start 2021-02-10T11:36:42.906 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:42.886 7fee0a342700 1 -- start start 2021-02-10T11:36:42.907 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:42.886 7fee0a342700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fee040d1870 0x7fee041b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:42.907 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:42.886 7fee0a342700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fee040cff70 0x7fee04109130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:42.907 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:42.886 7fee0a342700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee04109a60 con 0x7fee041b0ae0 2021-02-10T11:36:42.907 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:42.886 7fee0a342700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee04109750 con 0x7fee040cff70 2021-02-10T11:36:42.907 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:42.886 7fee0a342700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee04088070 con 0x7fee040d1870 2021-02-10T11:36:42.907 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:42.887 7fee0895f700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fee040cff70 0x7fee04109130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:42.907 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:42.887 7fee03fff700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fee040d1870 0x7fee041b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:42.907 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:42.887 7fee037fe700 1 --1- >> v1:172.21.2.113:6789/0 conn(0x7fee041b0ae0 0x7fee04108940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.113:6789/0 says I am v1:172.21.2.102:37554/0 (socket says 172.21.2.102:37554) 2021-02-10T11:36:42.907 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:42.887 7fee037fe700 1 -- 172.21.2.102:0/1531938923 learned_addr learned my addr 172.21.2.102:0/1531938923 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:36:42.907 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:42.888 7fee03fff700 1 -- 172.21.2.102:0/1531938923 >> v2:172.21.2.113:3300/0 conn(0x7fee040cff70 msgr2=0x7fee04109130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:42.908 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:42.888 7fee03fff700 1 --2- 172.21.2.102:0/1531938923 >> v2:172.21.2.113:3300/0 conn(0x7fee040cff70 0x7fee04109130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:42.908 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:42.888 7fee03fff700 1 -- 172.21.2.102:0/1531938923 >> v1:172.21.2.113:6789/0 conn(0x7fee041b0ae0 legacy=0x7fee04108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:42.908 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:42.888 7fee03fff700 1 -- 172.21.2.102:0/1531938923 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee04088260 con 0x7fee040d1870 2021-02-10T11:36:42.908 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:42.888 7fee02ffd700 1 -- 172.21.2.102:0/1531938923 <== mon.1 v1:172.21.2.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3168774650 0 0) 0x7fedf80010b0 con 0x7fee041b0ae0 2021-02-10T11:36:42.908 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:42.888 7fee0895f700 1 --2- 172.21.2.102:0/1531938923 >> v2:172.21.2.113:3300/0 conn(0x7fee040cff70 0x7fee04109130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:36:42.908 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:42.888 7fee03fff700 1 --2- 172.21.2.102:0/1531938923 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fee040d1870 0x7fee041b04c0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 rx=0x7fedf4006650 tx=0x7fedf4009a80).ready entity=mon.2 client_cookie=f2c95b2737dd5b18 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:42.908 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:42.889 7fee02ffd700 1 -- 172.21.2.102:0/1531938923 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fedf400c0c0 con 0x7fee040d1870 2021-02-10T11:36:42.909 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:42.889 7fee02ffd700 1 -- 172.21.2.102:0/1531938923 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedf400c340 con 0x7fee040d1870 2021-02-10T11:36:42.909 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:42.889 7fee02ffd700 1 -- 172.21.2.102:0/1531938923 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fedf400c860 con 0x7fee040d1870 2021-02-10T11:36:42.909 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:42.889 7fee0a342700 1 -- 172.21.2.102:0/1531938923 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fee040d1870 msgr2=0x7fee041b04c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:42.909 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:42.889 7fee0a342700 1 --2- 172.21.2.102:0/1531938923 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fee040d1870 0x7fee041b04c0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 rx=0x7fedf4006650 tx=0x7fedf4009a80).stop 2021-02-10T11:36:42.909 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:42.890 7fee0a342700 1 -- 172.21.2.102:0/1531938923 shutdown_connections 2021-02-10T11:36:42.909 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:42.890 7fee0a342700 1 --2- 172.21.2.102:0/1531938923 >> v2:172.21.2.113:3300/0 conn(0x7fee040cff70 0x7fee04109130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:42.909 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:42.890 7fee0a342700 1 --2- 172.21.2.102:0/1531938923 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fee040d1870 0x7fee041b04c0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:42.909 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:42.890 7fee0a342700 1 -- 172.21.2.102:0/1531938923 >> 172.21.2.102:0/1531938923 conn(0x7fee040ce420 msgr2=0x7fee040d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:42.909 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:42.890 7fee0a342700 1 -- 172.21.2.102:0/1531938923 shutdown_connections 2021-02-10T11:36:42.910 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:42.890 7fee0a342700 1 -- 172.21.2.102:0/1531938923 wait complete. 2021-02-10T11:36:42.910 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:42.904 7fee0a342700 1 Processor -- start 2021-02-10T11:36:42.910 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:42.904 7fee0a342700 1 -- start start 2021-02-10T11:36:42.911 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:42.905 7fee0a342700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fee040a2580 0x7fee040a1cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:42.911 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:42.905 7fee0a342700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fee04094c40 0x7fee041a75e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:42.911 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:42.905 7fee0a342700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fee04095040 0x7fee041a7c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:42.911 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:42.905 7fee0a342700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee04188180 con 0x7fee040a2580 2021-02-10T11:36:42.912 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:42.905 7fee0a342700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee04187e70 con 0x7fee04095040 2021-02-10T11:36:42.912 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:42.905 7fee0a342700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee041a8220 con 0x7fee04094c40 2021-02-10T11:36:42.912 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:42.905 7fee03fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fee040a2580 0x7fee040a1cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:42.912 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:42.905 7fee03fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fee040a2580 0x7fee040a1cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:38010/0 (socket says 172.21.2.102:38010) 2021-02-10T11:36:42.912 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:42.905 7fee03fff700 1 -- 172.21.2.102:0/2019652491 learned_addr learned my addr 172.21.2.102:0/2019652491 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:36:42.912 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:42.905 7fee03fff700 1 -- 172.21.2.102:0/2019652491 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fee04094c40 msgr2=0x7fee041a75e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:42.912 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:42.905 7fee03fff700 1 --2- 172.21.2.102:0/2019652491 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fee04094c40 0x7fee041a75e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:42.912 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:42.905 7fee03fff700 1 -- 172.21.2.102:0/2019652491 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fee04095040 msgr2=0x7fee041a7c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:42.912 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:42.905 7fee03fff700 1 --2- 172.21.2.102:0/2019652491 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fee04095040 0x7fee041a7c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:42.912 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:42.905 7fee03fff700 1 -- 172.21.2.102:0/2019652491 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee04188700 con 0x7fee040a2580 2021-02-10T11:36:42.913 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:42.905 7fee03fff700 1 --2- 172.21.2.102:0/2019652491 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fee040a2580 0x7fee040a1cf0 secure :-1 s=READY pgs=2852 cs=0 l=1 rev1=1 rx=0x7fedf4009630 tx=0x7fedf400b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:42.913 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:42.906 7fee00ff9700 1 -- 172.21.2.102:0/2019652491 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fedf40093a0 con 0x7fee040a2580 2021-02-10T11:36:42.913 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:42.906 7fee00ff9700 1 -- 172.21.2.102:0/2019652491 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedf400bd70 con 0x7fee040a2580 2021-02-10T11:36:42.913 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:42.906 7fee00ff9700 1 -- 172.21.2.102:0/2019652491 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fedf4005560 con 0x7fee040a2580 2021-02-10T11:36:42.913 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:42.906 7fee0a342700 1 -- 172.21.2.102:0/2019652491 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee04188700 con 0x7fee040a2580 2021-02-10T11:36:42.913 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:42.906 7fee0a342700 1 -- 172.21.2.102:0/2019652491 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee04188c90 con 0x7fee040a2580 2021-02-10T11:36:42.913 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:42.906 7fee00ff9700 1 -- 172.21.2.102:0/2019652491 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fedf4007020 con 0x7fee040a2580 2021-02-10T11:36:42.913 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:42.906 7fee00ff9700 1 --2- 172.21.2.102:0/2019652491 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fede40186c0 0x7fede401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:42.913 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:42.906 7fee00ff9700 1 -- 172.21.2.102:0/2019652491 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(157..157 src has 1..157) v4 ==== 4753+0+0 (secure 0 0 0) 0x7fedf4032360 con 0x7fee040a2580 2021-02-10T11:36:42.913 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:42.907 7fee037fe700 1 --2- 172.21.2.102:0/2019652491 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fede40186c0 0x7fede401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:42.913 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:42.907 7fee037fe700 1 --2- 172.21.2.102:0/2019652491 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fede40186c0 0x7fede401abc0 crc :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:42.938 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:42.936 7fee0a342700 1 -- 172.21.2.102:0/2019652491 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee04082980 con 0x7fee040a2580 2021-02-10T11:36:42.940 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:42.938 7fee00ff9700 1 -- 172.21.2.102:0/2019652491 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fedf4009340 con 0x7fee040a2580 2021-02-10T11:36:43.070 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:43.068 7fee0a342700 1 -- 172.21.2.102:0/2019652491 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7fee04109a60 con 0x7fee040a2580 2021-02-10T11:36:43.070 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:43.068 7fee00ff9700 1 -- 172.21.2.102:0/2019652491 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v157) v1 ==== 92+0+10 (secure 0 0 0) 0x7fedf400bd70 con 0x7fee040a2580 2021-02-10T11:36:43.071 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:36:43.072 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:43.070 7fee0a342700 1 -- 172.21.2.102:0/2019652491 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fede40186c0 msgr2=0x7fede401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:43.072 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:43.070 7fee0a342700 1 --2- 172.21.2.102:0/2019652491 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fede40186c0 0x7fede401abc0 crc :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:43.072 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:43.070 7fee0a342700 1 -- 172.21.2.102:0/2019652491 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fee040a2580 msgr2=0x7fee040a1cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:43.072 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:43.070 7fee0a342700 1 --2- 172.21.2.102:0/2019652491 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fee040a2580 0x7fee040a1cf0 secure :-1 s=READY pgs=2852 cs=0 l=1 rev1=1 rx=0x7fedf4009630 tx=0x7fedf400b980).stop 2021-02-10T11:36:43.073 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:43.070 7fee0a342700 1 -- 172.21.2.102:0/2019652491 shutdown_connections 2021-02-10T11:36:43.073 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:43.070 7fee0a342700 1 --2- 172.21.2.102:0/2019652491 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fede40186c0 0x7fede401abc0 unknown :-1 s=CLOSED pgs=1251 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:43.073 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:43.070 7fee0a342700 1 --2- 172.21.2.102:0/2019652491 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fee04095040 0x7fee041a7c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:43.073 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:43.070 7fee0a342700 1 --2- 172.21.2.102:0/2019652491 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fee04094c40 0x7fee041a75e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:43.073 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:43.070 7fee0a342700 1 --2- 172.21.2.102:0/2019652491 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fee040a2580 0x7fee040a1cf0 unknown :-1 s=CLOSED pgs=2852 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:43.073 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:43.070 7fee0a342700 1 -- 172.21.2.102:0/2019652491 >> 172.21.2.102:0/2019652491 conn(0x7fee040a2b30 msgr2=0x7fee0409ce60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:43.073 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:43.071 7fee0a342700 1 -- 172.21.2.102:0/2019652491 shutdown_connections 2021-02-10T11:36:43.073 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:43.071 7fee0a342700 1 -- 172.21.2.102:0/2019652491 wait complete. 2021-02-10T11:36:43.081 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --format=json-pretty osd lspools 2021-02-10T11:36:43.325 DEBUG:teuthology.orchestra.run.gibba002:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:36:43.329 DEBUG:teuthology.orchestra.run.gibba013:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:36:43.508 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:43.490 7f17cfa56700 1 Processor -- start 2021-02-10T11:36:43.509 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:43.490 7f17cfa56700 1 -- start start 2021-02-10T11:36:43.509 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:43.491 7f17cfa56700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f17c80d1870 0x7f17c81b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:43.509 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:43.491 7f17cfa56700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f17c81b0a10 0x7f17c80cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:43.509 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:43.491 7f17cfa56700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17c80f0b10 con 0x7f17c80d1c70 2021-02-10T11:36:43.509 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:43.491 7f17cfa56700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17c80e5ed0 con 0x7f17c81b0a10 2021-02-10T11:36:43.509 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:43.491 7f17cfa56700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17c80f0650 con 0x7f17c80d1870 2021-02-10T11:36:43.509 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:43.491 7f17cd071700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f17c81b0a10 0x7f17c80cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:43.509 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:43.491 7f17cd071700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f17c81b0a10 0x7f17c80cf330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:38018/0 (socket says 172.21.2.102:38018) 2021-02-10T11:36:43.509 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:43.491 7f17cd071700 1 -- 172.21.2.102:0/2820210322 learned_addr learned my addr 172.21.2.102:0/2820210322 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:36:43.509 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:43.491 7f17cd071700 1 -- 172.21.2.102:0/2820210322 >> v2:172.21.2.113:3300/0 conn(0x7f17c80d1870 msgr2=0x7f17c81b03f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:36:43.510 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:43.491 7f17cd071700 1 --2- 172.21.2.102:0/2820210322 >> v2:172.21.2.113:3300/0 conn(0x7f17c80d1870 0x7f17c81b03f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:43.510 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:43.491 7f17cd071700 1 -- 172.21.2.102:0/2820210322 >> v1:172.21.2.113:6789/0 conn(0x7f17c80d1c70 legacy=0x7f17c80cf950 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:36:43.510 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:43.491 7f17cd071700 1 -- 172.21.2.102:0/2820210322 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17c80f0840 con 0x7f17c81b0a10 2021-02-10T11:36:43.510 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:43.491 7f17cd071700 1 --2- 172.21.2.102:0/2820210322 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f17c81b0a10 0x7f17c80cf330 secure :-1 s=READY pgs=2853 cs=0 l=1 rev1=1 rx=0x7f17c4006700 tx=0x7f17c4009b00).ready entity=mon.0 client_cookie=6994a4669519adac server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:43.510 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:43.492 7f17cc870700 1 -- 172.21.2.102:0/2820210322 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f17c400c0c0 con 0x7f17c81b0a10 2021-02-10T11:36:43.510 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:43.492 7f17cc870700 1 -- 172.21.2.102:0/2820210322 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17c400c400 con 0x7f17c81b0a10 2021-02-10T11:36:43.510 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:43.492 7f17cc870700 1 -- 172.21.2.102:0/2820210322 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f17c400c8e0 con 0x7f17c81b0a10 2021-02-10T11:36:43.510 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:43.492 7f17cfa56700 1 -- 172.21.2.102:0/2820210322 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f17c81b0a10 msgr2=0x7f17c80cf330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:43.510 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:43.492 7f17cfa56700 1 --2- 172.21.2.102:0/2820210322 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f17c81b0a10 0x7f17c80cf330 secure :-1 s=READY pgs=2853 cs=0 l=1 rev1=1 rx=0x7f17c4006700 tx=0x7f17c4009b00).stop 2021-02-10T11:36:43.510 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:43.492 7f17cfa56700 1 -- 172.21.2.102:0/2820210322 shutdown_connections 2021-02-10T11:36:43.510 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:43.492 7f17cfa56700 1 --2- 172.21.2.102:0/2820210322 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f17c81b0a10 0x7f17c80cf330 unknown :-1 s=CLOSED pgs=2853 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:43.510 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:43.492 7f17cfa56700 1 --2- 172.21.2.102:0/2820210322 >> v2:172.21.2.113:3300/0 conn(0x7f17c80d1870 0x7f17c81b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:43.510 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:43.492 7f17cfa56700 1 -- 172.21.2.102:0/2820210322 >> 172.21.2.102:0/2820210322 conn(0x7f17c80ce450 msgr2=0x7f17c80d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:43.511 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:43.492 7f17cfa56700 1 -- 172.21.2.102:0/2820210322 shutdown_connections 2021-02-10T11:36:43.511 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:43.492 7f17cfa56700 1 -- 172.21.2.102:0/2820210322 wait complete. 2021-02-10T11:36:43.511 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:43.507 7f17cfa56700 1 Processor -- start 2021-02-10T11:36:43.511 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:43.507 7f17cfa56700 1 -- start start 2021-02-10T11:36:43.512 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:43.507 7f17cfa56700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f17c819af00 0x7f17c819b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:43.512 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:43.507 7f17cfa56700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f17c819ba80 0x7f17c819be80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:43.512 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:43.507 7f17cfa56700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f17c819c4a0 0x7f17c808a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:43.512 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:43.507 7f17cfa56700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17c808fb50 con 0x7f17c819c4a0 2021-02-10T11:36:43.513 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:43.507 7f17cfa56700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17c808afb0 con 0x7f17c819af00 2021-02-10T11:36:43.513 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:43.507 7f17cfa56700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17c808b1a0 con 0x7f17c819ba80 2021-02-10T11:36:43.513 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:43.507 7f17ce073700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f17c819c4a0 0x7f17c808a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:43.513 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:43.507 7f17ce073700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f17c819c4a0 0x7f17c808a990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:38024/0 (socket says 172.21.2.102:38024) 2021-02-10T11:36:43.513 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:43.507 7f17ce073700 1 -- 172.21.2.102:0/4177509695 learned_addr learned my addr 172.21.2.102:0/4177509695 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:36:43.513 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:43.507 7f17ce073700 1 -- 172.21.2.102:0/4177509695 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f17c819ba80 msgr2=0x7f17c819be80 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:36:43.513 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:43.507 7f17ce073700 1 --2- 172.21.2.102:0/4177509695 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f17c819ba80 0x7f17c819be80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:43.513 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:43.507 7f17ce073700 1 -- 172.21.2.102:0/4177509695 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f17c819af00 msgr2=0x7f17c819b300 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:36:43.513 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:43.507 7f17ce073700 1 --2- 172.21.2.102:0/4177509695 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f17c819af00 0x7f17c819b300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:43.513 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:43.507 7f17ce073700 1 -- 172.21.2.102:0/4177509695 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17c808b3c0 con 0x7f17c819c4a0 2021-02-10T11:36:43.513 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:43.508 7f17ce073700 1 --2- 172.21.2.102:0/4177509695 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f17c819c4a0 0x7f17c808a990 secure :-1 s=READY pgs=2854 cs=0 l=1 rev1=1 rx=0x7f17c0009940 tx=0x7f17c000bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:43.513 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:43.508 7f17ba7fc700 1 -- 172.21.2.102:0/4177509695 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f17c0000f90 con 0x7f17c819c4a0 2021-02-10T11:36:43.513 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:43.508 7f17ba7fc700 1 -- 172.21.2.102:0/4177509695 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17c0001270 con 0x7f17c819c4a0 2021-02-10T11:36:43.514 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:43.508 7f17cfa56700 1 -- 172.21.2.102:0/4177509695 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17c808b3c0 con 0x7f17c819c4a0 2021-02-10T11:36:43.514 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:43.508 7f17ba7fc700 1 -- 172.21.2.102:0/4177509695 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f17c0001750 con 0x7f17c819c4a0 2021-02-10T11:36:43.514 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:43.508 7f17cfa56700 1 -- 172.21.2.102:0/4177509695 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17c808b3c0 con 0x7f17c819c4a0 2021-02-10T11:36:43.514 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:43.508 7f17ba7fc700 1 -- 172.21.2.102:0/4177509695 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f17c0001450 con 0x7f17c819c4a0 2021-02-10T11:36:43.514 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:43.508 7f17ba7fc700 1 --2- 172.21.2.102:0/4177509695 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f17b0018730 0x7f17b001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:43.514 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:43.509 7f17ba7fc700 1 -- 172.21.2.102:0/4177509695 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(157..157 src has 1..157) v4 ==== 4753+0+0 (secure 0 0 0) 0x7f17c0032870 con 0x7f17c819c4a0 2021-02-10T11:36:43.514 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:43.509 7f17cd872700 1 --2- 172.21.2.102:0/4177509695 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f17b0018730 0x7f17b001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:43.514 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:43.509 7f17cd872700 1 --2- 172.21.2.102:0/4177509695 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f17b0018730 0x7f17b001ac10 crc :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:43.543 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:43.541 7f17cfa56700 1 -- 172.21.2.102:0/4177509695 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17c80ee030 con 0x7f17c819c4a0 2021-02-10T11:36:43.545 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:43.543 7f17ba7fc700 1 -- 172.21.2.102:0/4177509695 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f17c0000f90 con 0x7f17c819c4a0 2021-02-10T11:36:43.671 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:43.668 7f17cfa56700 1 -- 172.21.2.102:0/4177509695 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f17c80ee030 con 0x7f17c819c4a0 2021-02-10T11:36:43.671 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:43.669 7f17ba7fc700 1 -- 172.21.2.102:0/4177509695 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v157) v1 ==== 84+0+64 (secure 0 0 0) 0x7f17c0001270 con 0x7f17c819c4a0 2021-02-10T11:36:43.671 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:36:43.672 INFO:teuthology.orchestra.run.gibba002.stdout:[ 2021-02-10T11:36:43.672 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:36:43.672 INFO:teuthology.orchestra.run.gibba002.stdout: "poolnum": 1, 2021-02-10T11:36:43.672 INFO:teuthology.orchestra.run.gibba002.stdout: "poolname": "rbd" 2021-02-10T11:36:43.672 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:36:43.672 INFO:teuthology.orchestra.run.gibba002.stdout:] 2021-02-10T11:36:43.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:43.671 7f17cfa56700 1 -- 172.21.2.102:0/4177509695 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f17b0018730 msgr2=0x7f17b001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:43.674 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:43.671 7f17cfa56700 1 --2- 172.21.2.102:0/4177509695 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f17b0018730 0x7f17b001ac10 crc :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:43.674 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:43.671 7f17cfa56700 1 -- 172.21.2.102:0/4177509695 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f17c819c4a0 msgr2=0x7f17c808a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:43.674 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:43.671 7f17cfa56700 1 --2- 172.21.2.102:0/4177509695 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f17c819c4a0 0x7f17c808a990 secure :-1 s=READY pgs=2854 cs=0 l=1 rev1=1 rx=0x7f17c0009940 tx=0x7f17c000bcf0).stop 2021-02-10T11:36:43.674 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:43.672 7f17cfa56700 1 -- 172.21.2.102:0/4177509695 shutdown_connections 2021-02-10T11:36:43.674 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:43.672 7f17cfa56700 1 --2- 172.21.2.102:0/4177509695 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f17c819c4a0 0x7f17c808a990 unknown :-1 s=CLOSED pgs=2854 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:43.674 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:43.672 7f17cfa56700 1 --2- 172.21.2.102:0/4177509695 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f17c819ba80 0x7f17c819be80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:43.674 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:43.672 7f17cfa56700 1 --2- 172.21.2.102:0/4177509695 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f17b0018730 0x7f17b001ac10 unknown :-1 s=CLOSED pgs=1252 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:43.674 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:43.672 7f17cfa56700 1 --2- 172.21.2.102:0/4177509695 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f17c819af00 0x7f17c819b300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:43.674 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:43.672 7f17cfa56700 1 -- 172.21.2.102:0/4177509695 >> 172.21.2.102:0/4177509695 conn(0x7f17c819f8c0 msgr2=0x7f17c81a5fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:43.674 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:43.672 7f17cfa56700 1 -- 172.21.2.102:0/4177509695 shutdown_connections 2021-02-10T11:36:43.675 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:43.672 7f17cfa56700 1 -- 172.21.2.102:0/4177509695 wait complete. 2021-02-10T11:36:43.682 INFO:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2021-02-10T11:36:43.682 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool create cephfs_metadata 8 2021-02-10T11:36:44.115 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:44.097 7fe4ccbcc700 1 Processor -- start 2021-02-10T11:36:44.115 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:44.097 7fe4ccbcc700 1 -- start start 2021-02-10T11:36:44.115 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:44.097 7fe4ccbcc700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe4c80d1c70 0x7fe4c80cef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:44.115 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:44.097 7fe4ccbcc700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fe4c80cf580 0x7fe4c80cf980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:44.116 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:44.097 7fe4ccbcc700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4c80fe1c0 con 0x7fe4c80d1870 2021-02-10T11:36:44.116 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:44.097 7fe4ccbcc700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4c81b0cb0 con 0x7fe4c80cf580 2021-02-10T11:36:44.116 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:44.097 7fe4ccbcc700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4c8109a60 con 0x7fe4c80d1c70 2021-02-10T11:36:44.116 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:44.097 7fe4c705f700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fe4c80cf580 0x7fe4c80cf980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:44.116 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:44.097 7fe4c705f700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fe4c80cf580 0x7fe4c80cf980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:38034/0 (socket says 172.21.2.102:38034) 2021-02-10T11:36:44.116 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:44.097 7fe4c705f700 1 -- 172.21.2.102:0/4054760270 learned_addr learned my addr 172.21.2.102:0/4054760270 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:36:44.116 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:44.097 7fe4c685e700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fe4c80d1870 0x7fe4c81b04c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:58486/0 (socket says 172.21.2.102:58486) 2021-02-10T11:36:44.116 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:44.098 7fe4c705f700 1 -- 172.21.2.102:0/4054760270 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe4c80d1c70 msgr2=0x7fe4c80cef60 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:36:44.116 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:44.098 7fe4c705f700 1 --2- 172.21.2.102:0/4054760270 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe4c80d1c70 0x7fe4c80cef60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:44.116 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:44.098 7fe4c705f700 1 -- 172.21.2.102:0/4054760270 >> v1:172.21.2.102:6789/0 conn(0x7fe4c80d1870 legacy=0x7fe4c81b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:44.116 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:44.098 7fe4c705f700 1 -- 172.21.2.102:0/4054760270 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4c8088070 con 0x7fe4c80cf580 2021-02-10T11:36:44.116 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:44.098 7fe4c705f700 1 --2- 172.21.2.102:0/4054760270 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe4c80cf580 0x7fe4c80cf980 secure :-1 s=READY pgs=2856 cs=0 l=1 rev1=1 rx=0x7fe4c0006f00 tx=0x7fe4c0009f10).ready entity=mon.0 client_cookie=6d0bbf14882e7186 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:44.116 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:44.098 7fe4c585c700 1 -- 172.21.2.102:0/4054760270 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe4c000b3b0 con 0x7fe4c80cf580 2021-02-10T11:36:44.117 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:44.098 7fe4c585c700 1 -- 172.21.2.102:0/4054760270 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4c000b630 con 0x7fe4c80cf580 2021-02-10T11:36:44.117 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:44.098 7fe4c585c700 1 -- 172.21.2.102:0/4054760270 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe4c000bb50 con 0x7fe4c80cf580 2021-02-10T11:36:44.117 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:44.098 7fe4ccbcc700 1 -- 172.21.2.102:0/4054760270 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe4c80cf580 msgr2=0x7fe4c80cf980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:44.117 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:44.098 7fe4ccbcc700 1 --2- 172.21.2.102:0/4054760270 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe4c80cf580 0x7fe4c80cf980 secure :-1 s=READY pgs=2856 cs=0 l=1 rev1=1 rx=0x7fe4c0006f00 tx=0x7fe4c0009f10).stop 2021-02-10T11:36:44.117 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:44.099 7fe4ccbcc700 1 -- 172.21.2.102:0/4054760270 shutdown_connections 2021-02-10T11:36:44.117 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:44.099 7fe4ccbcc700 1 --2- 172.21.2.102:0/4054760270 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe4c80cf580 0x7fe4c80cf980 unknown :-1 s=CLOSED pgs=2856 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:44.117 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:44.099 7fe4ccbcc700 1 --2- 172.21.2.102:0/4054760270 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe4c80d1c70 0x7fe4c80cef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:44.117 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:44.099 7fe4ccbcc700 1 -- 172.21.2.102:0/4054760270 >> 172.21.2.102:0/4054760270 conn(0x7fe4c80ce420 msgr2=0x7fe4c80d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:44.117 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:44.099 7fe4ccbcc700 1 -- 172.21.2.102:0/4054760270 shutdown_connections 2021-02-10T11:36:44.117 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:44.099 7fe4ccbcc700 1 -- 172.21.2.102:0/4054760270 wait complete. 2021-02-10T11:36:44.119 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:44.113 7fe4ccbcc700 1 Processor -- start 2021-02-10T11:36:44.119 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:44.114 7fe4ccbcc700 1 -- start start 2021-02-10T11:36:44.119 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:44.114 7fe4ccbcc700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe4c80c3820 0x7fe4c80a7950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:44.119 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:44.114 7fe4ccbcc700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe4c81a44c0 0x7fe4c81a48c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:44.119 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:44.114 7fe4ccbcc700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe4c80863d0 0x7fe4c80867d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:44.119 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:44.114 7fe4ccbcc700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4c8085ed0 con 0x7fe4c80c3820 2021-02-10T11:36:44.119 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:44.114 7fe4ccbcc700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4c80c0740 con 0x7fe4c81a44c0 2021-02-10T11:36:44.119 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:44.114 7fe4ccbcc700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4c8086df0 con 0x7fe4c80863d0 2021-02-10T11:36:44.119 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:44.114 7fe4c685e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe4c80c3820 0x7fe4c80a7950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:44.119 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:44.114 7fe4c685e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe4c80c3820 0x7fe4c80a7950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:38036/0 (socket says 172.21.2.102:38036) 2021-02-10T11:36:44.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:44.114 7fe4c685e700 1 -- 172.21.2.102:0/4134574704 learned_addr learned my addr 172.21.2.102:0/4134574704 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:36:44.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:44.114 7fe4c685e700 1 -- 172.21.2.102:0/4134574704 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe4c80863d0 msgr2=0x7fe4c80867d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:44.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:44.114 7fe4c685e700 1 --2- 172.21.2.102:0/4134574704 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe4c80863d0 0x7fe4c80867d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:44.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:44.114 7fe4c685e700 1 -- 172.21.2.102:0/4134574704 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe4c81a44c0 msgr2=0x7fe4c81a48c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:44.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:44.114 7fe4c685e700 1 --2- 172.21.2.102:0/4134574704 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe4c81a44c0 0x7fe4c81a48c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:44.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:44.114 7fe4c685e700 1 -- 172.21.2.102:0/4134574704 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4c8087010 con 0x7fe4c80c3820 2021-02-10T11:36:44.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:44.115 7fe4c685e700 1 --2- 172.21.2.102:0/4134574704 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe4c80c3820 0x7fe4c80a7950 secure :-1 s=READY pgs=2857 cs=0 l=1 rev1=1 rx=0x7fe4b000a590 tx=0x7fe4b000a930).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:44.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:44.115 7fe4bf7fe700 1 -- 172.21.2.102:0/4134574704 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe4b0000ef0 con 0x7fe4c80c3820 2021-02-10T11:36:44.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:44.115 7fe4bf7fe700 1 -- 172.21.2.102:0/4134574704 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4b000ad20 con 0x7fe4c80c3820 2021-02-10T11:36:44.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:44.115 7fe4ccbcc700 1 -- 172.21.2.102:0/4134574704 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4c8087010 con 0x7fe4c80c3820 2021-02-10T11:36:44.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:44.115 7fe4ccbcc700 1 -- 172.21.2.102:0/4134574704 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4c8087010 con 0x7fe4c80c3820 2021-02-10T11:36:44.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:44.115 7fe4bf7fe700 1 -- 172.21.2.102:0/4134574704 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe4b0004620 con 0x7fe4c80c3820 2021-02-10T11:36:44.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:44.115 7fe4bf7fe700 1 -- 172.21.2.102:0/4134574704 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fe4b00043a0 con 0x7fe4c80c3820 2021-02-10T11:36:44.121 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:44.115 7fe4bf7fe700 1 --2- 172.21.2.102:0/4134574704 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe4ac018730 0x7fe4ac01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:44.121 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:44.116 7fe4bf7fe700 1 -- 172.21.2.102:0/4134574704 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(157..157 src has 1..157) v4 ==== 4753+0+0 (secure 0 0 0) 0x7fe4b00324c0 con 0x7fe4c80c3820 2021-02-10T11:36:44.121 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:44.116 7fe4c605d700 1 --2- 172.21.2.102:0/4134574704 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe4ac018730 0x7fe4ac01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:44.121 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:44.116 7fe4c605d700 1 --2- 172.21.2.102:0/4134574704 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe4ac018730 0x7fe4ac01ac10 crc :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:44.149 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:44.147 7fe4ccbcc700 1 -- 172.21.2.102:0/4134574704 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4c80fe1c0 con 0x7fe4c80c3820 2021-02-10T11:36:44.152 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:44.150 7fe4bf7fe700 1 -- 172.21.2.102:0/4134574704 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fe4b0007070 con 0x7fe4c80c3820 2021-02-10T11:36:44.280 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:44.278 7fe4ccbcc700 1 -- 172.21.2.102:0/4134574704 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"} v 0) v1 -- 0x7fe4c80fe1c0 con 0x7fe4c80c3820 2021-02-10T11:36:44.677 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:44.674 7fe4bf7fe700 1 -- 172.21.2.102:0/4134574704 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"}]=0 pool 'cephfs_metadata' created v158) v1 ==== 133+0+0 (secure 0 0 0) 0x7fe4b00043a0 con 0x7fe4c80c3820 2021-02-10T11:36:44.695 INFO:teuthology.orchestra.run.gibba002.stderr:pool 'cephfs_metadata' created 2021-02-10T11:36:44.697 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:44.695 7fe4ccbcc700 1 -- 172.21.2.102:0/4134574704 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe4ac018730 msgr2=0x7fe4ac01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:44.697 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:44.695 7fe4ccbcc700 1 --2- 172.21.2.102:0/4134574704 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe4ac018730 0x7fe4ac01ac10 crc :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:44.698 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:44.695 7fe4ccbcc700 1 -- 172.21.2.102:0/4134574704 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe4c80c3820 msgr2=0x7fe4c80a7950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:44.698 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:44.695 7fe4ccbcc700 1 --2- 172.21.2.102:0/4134574704 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe4c80c3820 0x7fe4c80a7950 secure :-1 s=READY pgs=2857 cs=0 l=1 rev1=1 rx=0x7fe4b000a590 tx=0x7fe4b000a930).stop 2021-02-10T11:36:44.698 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:44.695 7fe4ccbcc700 1 -- 172.21.2.102:0/4134574704 shutdown_connections 2021-02-10T11:36:44.698 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:44.695 7fe4ccbcc700 1 --2- 172.21.2.102:0/4134574704 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe4c80863d0 0x7fe4c80867d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:44.698 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:44.695 7fe4ccbcc700 1 --2- 172.21.2.102:0/4134574704 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe4ac018730 0x7fe4ac01ac10 unknown :-1 s=CLOSED pgs=1253 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:44.698 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:44.695 7fe4ccbcc700 1 --2- 172.21.2.102:0/4134574704 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe4c81a44c0 0x7fe4c81a48c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:44.698 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:44.695 7fe4ccbcc700 1 --2- 172.21.2.102:0/4134574704 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe4c80c3820 0x7fe4c80a7950 unknown :-1 s=CLOSED pgs=2857 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:44.698 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:44.695 7fe4ccbcc700 1 -- 172.21.2.102:0/4134574704 >> 172.21.2.102:0/4134574704 conn(0x7fe4c80872c0 msgr2=0x7fe4c8095700 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:44.698 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:44.695 7fe4ccbcc700 1 -- 172.21.2.102:0/4134574704 shutdown_connections 2021-02-10T11:36:44.698 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:44.695 7fe4ccbcc700 1 -- 172.21.2.102:0/4134574704 wait complete. 2021-02-10T11:36:44.706 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool create cephfs_data 8 2021-02-10T11:36:45.140 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:45.120 7f486a733700 1 Processor -- start 2021-02-10T11:36:45.140 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:45.120 7f486a733700 1 -- start start 2021-02-10T11:36:45.140 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:45.121 7f486a733700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f48640d1c70 0x7f48640cef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:45.140 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:45.121 7f486a733700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48640d0170 con 0x7f48640cf580 2021-02-10T11:36:45.140 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:45.121 7f486a733700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48640fe1c0 con 0x7f48640d1870 2021-02-10T11:36:45.140 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:45.121 7f486a733700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48641b0cb0 con 0x7f48640d1c70 2021-02-10T11:36:45.141 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:45.121 7f4868d50700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f48640cf580 0x7f48640cf980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:58504/0 (socket says 172.21.2.102:58504) 2021-02-10T11:36:45.141 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:45.121 7f4868d50700 1 -- 172.21.2.102:0/229691531 learned_addr learned my addr 172.21.2.102:0/229691531 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:36:45.141 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:45.121 7f4862ffd700 1 -- 172.21.2.102:0/229691531 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2512788823 0 0) 0x7f485c001500 con 0x7f48640cf580 2021-02-10T11:36:45.141 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:45.121 7f4862ffd700 1 -- 172.21.2.102:0/229691531 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f484c002fe0 con 0x7f48640cf580 2021-02-10T11:36:45.141 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:45.122 7f48637fe700 1 --2- 172.21.2.102:0/229691531 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f48640d1c70 0x7f48640cef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:45.141 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:45.122 7f4862ffd700 1 -- 172.21.2.102:0/229691531 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1273269710 0 0) 0x7f485c001500 con 0x7f48640cf580 2021-02-10T11:36:45.141 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:45.122 7f4862ffd700 1 -- 172.21.2.102:0/229691531 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f48640d1c70 msgr2=0x7f48640cef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:45.141 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:45.122 7f4862ffd700 1 --2- 172.21.2.102:0/229691531 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f48640d1c70 0x7f48640cef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:45.141 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:45.122 7f4862ffd700 1 -- 172.21.2.102:0/229691531 >> v1:172.21.2.113:6789/0 conn(0x7f48640d1870 legacy=0x7f48641b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:45.141 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:45.122 7f4862ffd700 1 -- 172.21.2.102:0/229691531 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4864109a60 con 0x7f48640cf580 2021-02-10T11:36:45.141 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:45.122 7f4862ffd700 1 -- 172.21.2.102:0/229691531 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7f485c001a00 con 0x7f48640cf580 2021-02-10T11:36:45.141 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:45.122 7f4862ffd700 1 -- 172.21.2.102:0/229691531 >> v1:172.21.2.102:6789/0 conn(0x7f48640cf580 legacy=0x7f48640cf980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:45.142 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:45.122 7f4862ffd700 1 --2- 172.21.2.102:0/229691531 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f484c003a60 0x7f484c005ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:45.142 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:45.122 7f4862ffd700 1 --2- 172.21.2.102:0/229691531 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f484c006790 0x7f484c008c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:45.142 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:45.122 7f4862ffd700 1 --2- 172.21.2.102:0/229691531 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f484c009590 0x7f484c00ba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:45.142 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:45.122 7f4862ffd700 1 -- 172.21.2.102:0/229691531 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f484c00c2d0 con 0x7f484c006790 2021-02-10T11:36:45.142 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:45.122 7f4862ffd700 1 -- 172.21.2.102:0/229691531 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f484c00c4f0 con 0x7f484c003a60 2021-02-10T11:36:45.142 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:45.122 7f4862ffd700 1 -- 172.21.2.102:0/229691531 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f484c00c710 con 0x7f484c009590 2021-02-10T11:36:45.142 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:45.122 7f48637fe700 1 --2- 172.21.2.102:0/229691531 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f484c006790 0x7f484c008c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:45.142 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:45.122 7f4863fff700 1 --2- 172.21.2.102:0/229691531 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f484c003a60 0x7f484c005ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:45.142 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:45.122 7f4868d50700 1 --2- 172.21.2.102:0/229691531 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f484c009590 0x7f484c00ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:45.142 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:45.122 7f48637fe700 1 -- 172.21.2.102:0/229691531 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f484c009590 msgr2=0x7f484c00ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:45.142 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:45.122 7f48637fe700 1 --2- 172.21.2.102:0/229691531 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f484c009590 0x7f484c00ba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:45.142 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:45.122 7f48637fe700 1 -- 172.21.2.102:0/229691531 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f484c003a60 msgr2=0x7f484c005ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:45.143 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:45.122 7f48637fe700 1 --2- 172.21.2.102:0/229691531 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f484c003a60 0x7f484c005ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:45.143 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:45.122 7f48637fe700 1 -- 172.21.2.102:0/229691531 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f484c050980 con 0x7f484c006790 2021-02-10T11:36:45.143 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:45.122 7f48637fe700 1 --2- 172.21.2.102:0/229691531 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f484c006790 0x7f484c008c90 secure :-1 s=READY pgs=2859 cs=0 l=1 rev1=1 rx=0x7f4858006ac0 tx=0x7f4858009ab0).ready entity=mon.0 client_cookie=135516e1a14e572a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:45.143 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:45.123 7f4862ffd700 1 -- 172.21.2.102:0/229691531 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f485800c0c0 con 0x7f484c006790 2021-02-10T11:36:45.143 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:45.123 7f4862ffd700 1 -- 172.21.2.102:0/229691531 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f485800c400 con 0x7f484c006790 2021-02-10T11:36:45.143 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:45.123 7f4862ffd700 1 -- 172.21.2.102:0/229691531 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f485800c890 con 0x7f484c006790 2021-02-10T11:36:45.143 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:45.123 7f486a733700 1 -- 172.21.2.102:0/229691531 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f484c006790 msgr2=0x7f484c008c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:45.143 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:45.123 7f486a733700 1 --2- 172.21.2.102:0/229691531 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f484c006790 0x7f484c008c90 secure :-1 s=READY pgs=2859 cs=0 l=1 rev1=1 rx=0x7f4858006ac0 tx=0x7f4858009ab0).stop 2021-02-10T11:36:45.143 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:45.123 7f4868d50700 1 -- 172.21.2.102:0/229691531 reap_dead start 2021-02-10T11:36:45.143 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:45.123 7f486a733700 1 -- 172.21.2.102:0/229691531 shutdown_connections 2021-02-10T11:36:45.143 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:45.123 7f486a733700 1 -- 172.21.2.102:0/229691531 >> 172.21.2.102:0/229691531 conn(0x7f48640ce420 msgr2=0x7f48640d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:45.143 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:45.123 7f486a733700 1 -- 172.21.2.102:0/229691531 shutdown_connections 2021-02-10T11:36:45.143 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:45.123 7f486a733700 1 -- 172.21.2.102:0/229691531 wait complete. 2021-02-10T11:36:45.144 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:45.138 7f486a733700 1 Processor -- start 2021-02-10T11:36:45.144 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:45.138 7f486a733700 1 -- start start 2021-02-10T11:36:45.145 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:45.138 7f486a733700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f48640ab680 0x7f486408a420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:45.145 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:45.138 7f486a733700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f486408aa40 0x7f48640a9960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:45.145 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:45.138 7f486a733700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f486408ae40 0x7f48640a9f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:45.145 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:45.138 7f486a733700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48640aba80 con 0x7f486408aa40 2021-02-10T11:36:45.145 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:45.138 7f486a733700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48640ab100 con 0x7f486408ae40 2021-02-10T11:36:45.145 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:45.138 7f486a733700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f486409c330 con 0x7f48640ab680 2021-02-10T11:36:45.145 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:45.139 7f48637fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f486408aa40 0x7f48640a9960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:45.146 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:45.139 7f48637fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f486408aa40 0x7f48640a9960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:38058/0 (socket says 172.21.2.102:38058) 2021-02-10T11:36:45.146 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:45.139 7f48637fe700 1 -- 172.21.2.102:0/3812889044 learned_addr learned my addr 172.21.2.102:0/3812889044 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:36:45.146 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:45.139 7f48637fe700 1 -- 172.21.2.102:0/3812889044 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f48640ab680 msgr2=0x7f486408a420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:45.146 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:45.139 7f48637fe700 1 --2- 172.21.2.102:0/3812889044 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f48640ab680 0x7f486408a420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:45.146 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:45.139 7f48637fe700 1 -- 172.21.2.102:0/3812889044 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f486408ae40 msgr2=0x7f48640a9f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:45.146 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:45.139 7f48637fe700 1 --2- 172.21.2.102:0/3812889044 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f486408ae40 0x7f48640a9f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:45.146 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:45.139 7f48637fe700 1 -- 172.21.2.102:0/3812889044 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f486419e300 con 0x7f486408aa40 2021-02-10T11:36:45.146 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:45.139 7f48637fe700 1 --2- 172.21.2.102:0/3812889044 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f486408aa40 0x7f48640a9960 secure :-1 s=READY pgs=2860 cs=0 l=1 rev1=1 rx=0x7f485800b670 tx=0x7f485800ba10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:45.146 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:45.139 7f4860ff9700 1 -- 172.21.2.102:0/3812889044 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4858009250 con 0x7f486408aa40 2021-02-10T11:36:45.146 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:45.139 7f4860ff9700 1 -- 172.21.2.102:0/3812889044 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4858006920 con 0x7f486408aa40 2021-02-10T11:36:45.146 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:45.139 7f486a733700 1 -- 172.21.2.102:0/3812889044 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f486419e300 con 0x7f486408aa40 2021-02-10T11:36:45.146 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:45.139 7f4860ff9700 1 -- 172.21.2.102:0/3812889044 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f48580052e0 con 0x7f486408aa40 2021-02-10T11:36:45.146 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:45.140 7f486a733700 1 -- 172.21.2.102:0/3812889044 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f486419e300 con 0x7f486408aa40 2021-02-10T11:36:45.147 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:45.140 7f4860ff9700 1 -- 172.21.2.102:0/3812889044 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4858009250 con 0x7f486408aa40 2021-02-10T11:36:45.147 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:45.140 7f4860ff9700 1 --2- 172.21.2.102:0/3812889044 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f48440186c0 0x7f484401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:45.147 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:45.140 7f4860ff9700 1 -- 172.21.2.102:0/3812889044 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(159..159 src has 1..159) v4 ==== 5107+0+0 (secure 0 0 0) 0x7f48580325f0 con 0x7f486408aa40 2021-02-10T11:36:45.147 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:45.140 7f4863fff700 1 --2- 172.21.2.102:0/3812889044 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f48440186c0 0x7f484401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:45.147 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:45.141 7f4863fff700 1 --2- 172.21.2.102:0/3812889044 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f48440186c0 0x7f484401abc0 crc :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:45.173 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:45.170 7f486a733700 1 -- 172.21.2.102:0/3812889044 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4864082a70 con 0x7f486408aa40 2021-02-10T11:36:45.175 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:45.173 7f4860ff9700 1 -- 172.21.2.102:0/3812889044 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f4858000940 con 0x7f486408aa40 2021-02-10T11:36:45.304 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:45.302 7f486a733700 1 -- 172.21.2.102:0/3812889044 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_data"} v 0) v1 -- 0x7f4864082a70 con 0x7f486408aa40 2021-02-10T11:36:45.685 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:45.683 7f4860ff9700 1 -- 172.21.2.102:0/3812889044 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_data"}]=0 pool 'cephfs_data' created v160) v1 ==== 125+0+0 (secure 0 0 0) 0x7f4858009250 con 0x7f486408aa40 2021-02-10T11:36:45.721 INFO:teuthology.orchestra.run.gibba002.stderr:pool 'cephfs_data' created 2021-02-10T11:36:45.723 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:45.720 7f486a733700 1 -- 172.21.2.102:0/3812889044 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f48440186c0 msgr2=0x7f484401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:45.723 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:45.720 7f486a733700 1 --2- 172.21.2.102:0/3812889044 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f48440186c0 0x7f484401abc0 crc :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:45.723 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:45.720 7f486a733700 1 -- 172.21.2.102:0/3812889044 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f486408aa40 msgr2=0x7f48640a9960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:45.723 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:45.720 7f486a733700 1 --2- 172.21.2.102:0/3812889044 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f486408aa40 0x7f48640a9960 secure :-1 s=READY pgs=2860 cs=0 l=1 rev1=1 rx=0x7f485800b670 tx=0x7f485800ba10).stop 2021-02-10T11:36:45.723 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:45.720 7f486a733700 1 -- 172.21.2.102:0/3812889044 shutdown_connections 2021-02-10T11:36:45.723 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:45.720 7f486a733700 1 --2- 172.21.2.102:0/3812889044 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f48440186c0 0x7f484401abc0 unknown :-1 s=CLOSED pgs=1254 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:45.723 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:45.720 7f486a733700 1 --2- 172.21.2.102:0/3812889044 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f486408ae40 0x7f48640a9f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:45.723 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:45.720 7f486a733700 1 --2- 172.21.2.102:0/3812889044 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f486408aa40 0x7f48640a9960 unknown :-1 s=CLOSED pgs=2860 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:45.723 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:45.720 7f486a733700 1 --2- 172.21.2.102:0/3812889044 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f48640ab680 0x7f486408a420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:45.723 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:45.720 7f486a733700 1 -- 172.21.2.102:0/3812889044 >> 172.21.2.102:0/3812889044 conn(0x7f4864091d20 msgr2=0x7f48640a6cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:45.723 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:45.721 7f486a733700 1 -- 172.21.2.102:0/3812889044 shutdown_connections 2021-02-10T11:36:45.724 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:45.721 7f486a733700 1 -- 172.21.2.102:0/3812889044 wait complete. 2021-02-10T11:36:45.731 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2021-02-10T11:36:46.165 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:46.146 7fc1e7d8e700 1 Processor -- start 2021-02-10T11:36:46.165 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:46.146 7fc1e7d8e700 1 -- start start 2021-02-10T11:36:46.165 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:46.146 7fc1e7d8e700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc1e00d1c80 0x7fc1e00cef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:46.165 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:46.146 7fc1e7d8e700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fc1e00cf590 0x7fc1e00cf990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:46.165 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:46.146 7fc1e7d8e700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1e00fe1d0 con 0x7fc1e00d1880 2021-02-10T11:36:46.165 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:46.146 7fc1e7d8e700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1e01b0cc0 con 0x7fc1e00cf590 2021-02-10T11:36:46.165 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:46.146 7fc1e7d8e700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1e0109a70 con 0x7fc1e00d1c80 2021-02-10T11:36:46.165 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:46.147 7fc1e63ab700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fc1e00cf590 0x7fc1e00cf990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:46.165 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:46.147 7fc1e63ab700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fc1e00cf590 0x7fc1e00cf990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:38066/0 (socket says 172.21.2.102:38066) 2021-02-10T11:36:46.166 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:46.147 7fc1e63ab700 1 -- 172.21.2.102:0/1220977182 learned_addr learned my addr 172.21.2.102:0/1220977182 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:36:46.166 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:46.147 7fc1e5baa700 1 --1- 172.21.2.102:0/1220977182 >> v1:172.21.2.102:6789/0 conn(0x7fc1e00d1880 0x7fc1e01b04d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:58522/0 (socket says 172.21.2.102:58522) 2021-02-10T11:36:46.166 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:46.147 7fc1e63ab700 1 -- 172.21.2.102:0/1220977182 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc1e00d1c80 msgr2=0x7fc1e00cef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:46.166 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:46.147 7fc1e63ab700 1 --2- 172.21.2.102:0/1220977182 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc1e00d1c80 0x7fc1e00cef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:46.166 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:46.147 7fc1e63ab700 1 -- 172.21.2.102:0/1220977182 >> v1:172.21.2.102:6789/0 conn(0x7fc1e00d1880 legacy=0x7fc1e01b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:46.166 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:46.147 7fc1e63ab700 1 -- 172.21.2.102:0/1220977182 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1e0088070 con 0x7fc1e00cf590 2021-02-10T11:36:46.166 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:46.147 7fc1e63ab700 1 --2- 172.21.2.102:0/1220977182 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc1e00cf590 0x7fc1e00cf990 secure :-1 s=READY pgs=2862 cs=0 l=1 rev1=1 rx=0x7fc1d8006f00 tx=0x7fc1d8009f10).ready entity=mon.0 client_cookie=6ae43b89b0724822 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:46.166 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:46.148 7fc1e4ba8700 1 -- 172.21.2.102:0/1220977182 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc1d800b3b0 con 0x7fc1e00cf590 2021-02-10T11:36:46.166 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:46.148 7fc1e4ba8700 1 -- 172.21.2.102:0/1220977182 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1d800b730 con 0x7fc1e00cf590 2021-02-10T11:36:46.166 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:46.148 7fc1e4ba8700 1 -- 172.21.2.102:0/1220977182 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc1d800bc50 con 0x7fc1e00cf590 2021-02-10T11:36:46.166 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:46.148 7fc1e7d8e700 1 -- 172.21.2.102:0/1220977182 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc1e00cf590 msgr2=0x7fc1e00cf990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:46.166 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:46.148 7fc1e7d8e700 1 --2- 172.21.2.102:0/1220977182 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc1e00cf590 0x7fc1e00cf990 secure :-1 s=READY pgs=2862 cs=0 l=1 rev1=1 rx=0x7fc1d8006f00 tx=0x7fc1d8009f10).stop 2021-02-10T11:36:46.167 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:46.148 7fc1e7d8e700 1 -- 172.21.2.102:0/1220977182 shutdown_connections 2021-02-10T11:36:46.167 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:46.148 7fc1e7d8e700 1 --2- 172.21.2.102:0/1220977182 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc1e00cf590 0x7fc1e00cf990 unknown :-1 s=CLOSED pgs=2862 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:46.167 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:46.148 7fc1e7d8e700 1 --2- 172.21.2.102:0/1220977182 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc1e00d1c80 0x7fc1e00cef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:46.167 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:46.148 7fc1e7d8e700 1 -- 172.21.2.102:0/1220977182 >> 172.21.2.102:0/1220977182 conn(0x7fc1e00ce430 msgr2=0x7fc1e00d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:46.167 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:46.149 7fc1e7d8e700 1 -- 172.21.2.102:0/1220977182 shutdown_connections 2021-02-10T11:36:46.167 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:46.149 7fc1e7d8e700 1 -- 172.21.2.102:0/1220977182 wait complete. 2021-02-10T11:36:46.169 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:46.163 7fc1e7d8e700 1 Processor -- start 2021-02-10T11:36:46.169 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:46.163 7fc1e7d8e700 1 -- start start 2021-02-10T11:36:46.169 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:46.164 7fc1e7d8e700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc1e01b58d0 0x7fc1e00aa4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:46.169 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:46.164 7fc1e7d8e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc1e00aaaf0 0x7fc1e01b80a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:46.169 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:46.164 7fc1e7d8e700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc1e01b86c0 0x7fc1e019de70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:46.169 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:46.164 7fc1e7d8e700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1e01b8ac0 con 0x7fc1e00aaaf0 2021-02-10T11:36:46.169 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:46.164 7fc1e7d8e700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1e01b8e20 con 0x7fc1e01b58d0 2021-02-10T11:36:46.169 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:46.164 7fc1e7d8e700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1e01b5cd0 con 0x7fc1e01b86c0 2021-02-10T11:36:46.169 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:46.164 7fc1e53a9700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc1e00aaaf0 0x7fc1e01b80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:46.169 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:46.164 7fc1e53a9700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc1e00aaaf0 0x7fc1e01b80a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:38070/0 (socket says 172.21.2.102:38070) 2021-02-10T11:36:46.170 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:46.164 7fc1e53a9700 1 -- 172.21.2.102:0/3048179561 learned_addr learned my addr 172.21.2.102:0/3048179561 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:36:46.170 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:46.164 7fc1e63ab700 1 --2- 172.21.2.102:0/3048179561 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc1e01b86c0 0x7fc1e019de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:46.170 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:46.164 7fc1e5baa700 1 --2- 172.21.2.102:0/3048179561 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc1e01b58d0 0x7fc1e00aa4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:46.170 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:46.164 7fc1e53a9700 1 -- 172.21.2.102:0/3048179561 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc1e01b86c0 msgr2=0x7fc1e019de70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:46.170 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:46.165 7fc1e53a9700 1 --2- 172.21.2.102:0/3048179561 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc1e01b86c0 0x7fc1e019de70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:46.170 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:46.165 7fc1e53a9700 1 -- 172.21.2.102:0/3048179561 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc1e01b58d0 msgr2=0x7fc1e00aa4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:46.170 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:46.165 7fc1e53a9700 1 --2- 172.21.2.102:0/3048179561 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc1e01b58d0 0x7fc1e00aa4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:46.170 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:46.165 7fc1e53a9700 1 -- 172.21.2.102:0/3048179561 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1e0095710 con 0x7fc1e00aaaf0 2021-02-10T11:36:46.170 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:46.165 7fc1e5baa700 1 --2- 172.21.2.102:0/3048179561 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc1e01b58d0 0x7fc1e00aa4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:36:46.170 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:46.165 7fc1e53a9700 1 --2- 172.21.2.102:0/3048179561 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc1e00aaaf0 0x7fc1e01b80a0 secure :-1 s=READY pgs=2863 cs=0 l=1 rev1=1 rx=0x7fc1d000b530 tx=0x7fc1d000b8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:46.170 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:46.165 7fc1d67fc700 1 -- 172.21.2.102:0/3048179561 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc1d000bcc0 con 0x7fc1e00aaaf0 2021-02-10T11:36:46.170 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:46.165 7fc1d67fc700 1 -- 172.21.2.102:0/3048179561 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1d0000c90 con 0x7fc1e00aaaf0 2021-02-10T11:36:46.171 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:46.165 7fc1d67fc700 1 -- 172.21.2.102:0/3048179561 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc1d00022a0 con 0x7fc1e00aaaf0 2021-02-10T11:36:46.171 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:46.165 7fc1e7d8e700 1 -- 172.21.2.102:0/3048179561 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1e0095710 con 0x7fc1e00aaaf0 2021-02-10T11:36:46.171 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:46.165 7fc1e7d8e700 1 -- 172.21.2.102:0/3048179561 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1e0095710 con 0x7fc1e00aaaf0 2021-02-10T11:36:46.171 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:46.166 7fc1d67fc700 1 -- 172.21.2.102:0/3048179561 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fc1d0005310 con 0x7fc1e00aaaf0 2021-02-10T11:36:46.171 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:46.166 7fc1d67fc700 1 --2- 172.21.2.102:0/3048179561 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc1c80186c0 0x7fc1c801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:46.171 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:46.166 7fc1d67fc700 1 -- 172.21.2.102:0/3048179561 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(160..160 src has 1..160) v4 ==== 5457+0+0 (secure 0 0 0) 0x7fc1d0032500 con 0x7fc1e00aaaf0 2021-02-10T11:36:46.171 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:46.166 7fc1e5baa700 1 --2- 172.21.2.102:0/3048179561 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc1c80186c0 0x7fc1c801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:46.171 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:46.167 7fc1e5baa700 1 --2- 172.21.2.102:0/3048179561 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc1c80186c0 0x7fc1c801abc0 crc :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:46.198 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:46.196 7fc1e7d8e700 1 -- 172.21.2.102:0/3048179561 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1e019ede0 con 0x7fc1e00aaaf0 2021-02-10T11:36:46.201 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:46.199 7fc1d67fc700 1 -- 172.21.2.102:0/3048179561 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fc1d0006b90 con 0x7fc1e00aaaf0 2021-02-10T11:36:46.332 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:46.330 7fc1e7d8e700 1 -- 172.21.2.102:0/3048179561 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"data": "cephfs_data", "prefix": "fs new", "force": true, "fs_name": "cephfs", "metadata": "cephfs_metadata"} v 0) v1 -- 0x7fc1e019ede0 con 0x7fc1e00aaaf0 2021-02-10T11:36:46.335 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:46.334 7fc1d67fc700 1 -- 172.21.2.102:0/3048179561 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"data": "cephfs_data", "prefix": "fs new", "force": true, "fs_name": "cephfs", "metadata": "cephfs_metadata"}]=0 new fs with metadata pool 34 and data pool 35 v138) v1 ==== 189+0+0 (secure 0 0 0) 0x7fc1d0005310 con 0x7fc1e00aaaf0 2021-02-10T11:36:46.340 INFO:teuthology.orchestra.run.gibba002.stderr:new fs with metadata pool 34 and data pool 35 2021-02-10T11:36:46.341 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:46.339 7fc1e7d8e700 1 -- 172.21.2.102:0/3048179561 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc1c80186c0 msgr2=0x7fc1c801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:46.341 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:46.339 7fc1e7d8e700 1 --2- 172.21.2.102:0/3048179561 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc1c80186c0 0x7fc1c801abc0 crc :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:46.342 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:46.339 7fc1e7d8e700 1 -- 172.21.2.102:0/3048179561 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc1e00aaaf0 msgr2=0x7fc1e01b80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:46.342 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:46.339 7fc1e7d8e700 1 --2- 172.21.2.102:0/3048179561 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc1e00aaaf0 0x7fc1e01b80a0 secure :-1 s=READY pgs=2863 cs=0 l=1 rev1=1 rx=0x7fc1d000b530 tx=0x7fc1d000b8d0).stop 2021-02-10T11:36:46.342 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:46.339 7fc1e7d8e700 1 -- 172.21.2.102:0/3048179561 shutdown_connections 2021-02-10T11:36:46.342 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:46.339 7fc1e7d8e700 1 --2- 172.21.2.102:0/3048179561 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc1e01b86c0 0x7fc1e019de70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:46.342 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:46.339 7fc1e7d8e700 1 --2- 172.21.2.102:0/3048179561 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc1e00aaaf0 0x7fc1e01b80a0 unknown :-1 s=CLOSED pgs=2863 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:46.342 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:46.339 7fc1e7d8e700 1 --2- 172.21.2.102:0/3048179561 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc1c80186c0 0x7fc1c801abc0 unknown :-1 s=CLOSED pgs=1257 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:46.342 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:46.339 7fc1e7d8e700 1 --2- 172.21.2.102:0/3048179561 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc1e01b58d0 0x7fc1e00aa4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:46.342 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:46.339 7fc1e7d8e700 1 -- 172.21.2.102:0/3048179561 >> 172.21.2.102:0/3048179561 conn(0x7fc1e019f940 msgr2=0x7fc1e018c290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:46.342 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:46.339 7fc1e7d8e700 1 -- 172.21.2.102:0/3048179561 shutdown_connections 2021-02-10T11:36:46.342 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:46.339 7fc1e7d8e700 1 -- 172.21.2.102:0/3048179561 wait complete. 2021-02-10T11:36:46.350 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:36:46.779 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:46.760 7f288bd90700 1 Processor -- start 2021-02-10T11:36:46.780 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:46.760 7f288bd90700 1 -- start start 2021-02-10T11:36:46.780 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:46.761 7f288bd90700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f28840d0340 0x7f28840ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:46.780 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:46.761 7f288bd90700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28840f0b10 con 0x7f28840d1870 2021-02-10T11:36:46.780 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:46.761 7f288bd90700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28841b0be0 con 0x7f28840d1c70 2021-02-10T11:36:46.780 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:46.761 7f288bd90700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28840e5ed0 con 0x7f28840d0340 2021-02-10T11:36:46.780 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:46.761 7f2889bac700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f28840d1870 0x7f28841b03f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:58534/0 (socket says 172.21.2.102:58534) 2021-02-10T11:36:46.780 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:46.761 7f2889bac700 1 -- 172.21.2.102:0/4197190733 learned_addr learned my addr 172.21.2.102:0/4197190733 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:36:46.780 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:46.761 7f28893ab700 1 --2- 172.21.2.102:0/4197190733 >> v2:172.21.2.113:3300/0 conn(0x7f28840d0340 0x7f28840ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:46.780 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:46.761 7f2888baa700 1 -- 172.21.2.102:0/4197190733 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1118515136 0 0) 0x7f2880001180 con 0x7f28840d1870 2021-02-10T11:36:46.780 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:46.762 7f2888baa700 1 -- 172.21.2.102:0/4197190733 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2870002fe0 con 0x7f28840d1870 2021-02-10T11:36:46.780 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:46.762 7f2888baa700 1 -- 172.21.2.102:0/4197190733 <== mon.1 v1:172.21.2.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3416738878 0 0) 0x7f287c001080 con 0x7f28840d1c70 2021-02-10T11:36:46.780 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:46.762 7f2888baa700 1 -- 172.21.2.102:0/4197190733 --> v1:172.21.2.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2870004370 con 0x7f28840d1c70 2021-02-10T11:36:46.781 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:46.762 7f28893ab700 1 -- 172.21.2.102:0/4197190733 >> v1:172.21.2.113:6789/0 conn(0x7f28840d1c70 legacy=0x7f28840efc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:46.781 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:46.762 7f28893ab700 1 -- 172.21.2.102:0/4197190733 >> v1:172.21.2.102:6789/0 conn(0x7f28840d1870 legacy=0x7f28841b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:46.781 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:46.762 7f28893ab700 1 -- 172.21.2.102:0/4197190733 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28840f0450 con 0x7f28840d0340 2021-02-10T11:36:46.781 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:46.762 7f28893ab700 1 --2- 172.21.2.102:0/4197190733 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f28840d0340 0x7f28840ef640 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 rx=0x7f28780069d0 tx=0x7f2878009dc0).ready entity=mon.1 client_cookie=9b93a24204b71834 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:46.781 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:46.762 7f2888baa700 1 -- 172.21.2.102:0/4197190733 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f287800b510 con 0x7f28840d0340 2021-02-10T11:36:46.781 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:46.762 7f2888baa700 1 -- 172.21.2.102:0/4197190733 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2878006ac0 con 0x7f28840d0340 2021-02-10T11:36:46.781 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:46.762 7f2888baa700 1 -- 172.21.2.102:0/4197190733 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f287800ba80 con 0x7f28840d0340 2021-02-10T11:36:46.781 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:46.762 7f288bd90700 1 -- 172.21.2.102:0/4197190733 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f28840d0340 msgr2=0x7f28840ef640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:46.781 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:46.762 7f288bd90700 1 --2- 172.21.2.102:0/4197190733 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f28840d0340 0x7f28840ef640 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 rx=0x7f28780069d0 tx=0x7f2878009dc0).stop 2021-02-10T11:36:46.781 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:46.763 7f288bd90700 1 -- 172.21.2.102:0/4197190733 shutdown_connections 2021-02-10T11:36:46.781 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:46.763 7f288bd90700 1 --2- 172.21.2.102:0/4197190733 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f28840d0340 0x7f28840ef640 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:46.781 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:46.763 7f288bd90700 1 -- 172.21.2.102:0/4197190733 >> 172.21.2.102:0/4197190733 conn(0x7f28840ce450 msgr2=0x7f28840d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:46.782 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:46.763 7f288bd90700 1 -- 172.21.2.102:0/4197190733 shutdown_connections 2021-02-10T11:36:46.782 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:46.763 7f288bd90700 1 -- 172.21.2.102:0/4197190733 wait complete. 2021-02-10T11:36:46.783 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:46.778 7f288bd90700 1 Processor -- start 2021-02-10T11:36:46.784 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:46.778 7f288bd90700 1 -- start start 2021-02-10T11:36:46.784 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:46.778 7f288bd90700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f288418d870 0x7f28840b19b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:46.784 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:46.778 7f288bd90700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f28840b1fd0 0x7f2884099ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:46.784 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:46.778 7f288bd90700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f28840b23d0 0x7f288409a510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:46.784 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:46.778 7f288bd90700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f288418dc70 con 0x7f288418d870 2021-02-10T11:36:46.784 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:46.778 7f288bd90700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2884199e60 con 0x7f28840b1fd0 2021-02-10T11:36:46.784 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:46.778 7f288bd90700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f288409ab30 con 0x7f28840b23d0 2021-02-10T11:36:46.784 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:46.778 7f2889bac700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f288418d870 0x7f28840b19b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:46.784 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:46.778 7f2889bac700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f288418d870 0x7f28840b19b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:38084/0 (socket says 172.21.2.102:38084) 2021-02-10T11:36:46.784 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:46.778 7f2889bac700 1 -- 172.21.2.102:0/2314193025 learned_addr learned my addr 172.21.2.102:0/2314193025 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:36:46.784 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:46.779 7f288a3ad700 1 --2- 172.21.2.102:0/2314193025 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f28840b23d0 0x7f288409a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:46.785 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:46.779 7f28893ab700 1 --2- 172.21.2.102:0/2314193025 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f28840b1fd0 0x7f2884099ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:46.785 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:46.779 7f2889bac700 1 -- 172.21.2.102:0/2314193025 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f28840b23d0 msgr2=0x7f288409a510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:46.785 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:46.779 7f2889bac700 1 --2- 172.21.2.102:0/2314193025 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f28840b23d0 0x7f288409a510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:46.785 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:46.779 7f2889bac700 1 -- 172.21.2.102:0/2314193025 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f28840b1fd0 msgr2=0x7f2884099ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:46.785 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:46.779 7f2889bac700 1 --2- 172.21.2.102:0/2314193025 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f28840b1fd0 0x7f2884099ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:46.785 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:46.779 7f2889bac700 1 -- 172.21.2.102:0/2314193025 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f288409ad50 con 0x7f288418d870 2021-02-10T11:36:46.785 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:46.779 7f28893ab700 1 --2- 172.21.2.102:0/2314193025 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f28840b1fd0 0x7f2884099ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:36:46.785 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:46.779 7f288a3ad700 1 --2- 172.21.2.102:0/2314193025 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f28840b23d0 0x7f288409a510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:36:46.785 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:46.779 7f2889bac700 1 --2- 172.21.2.102:0/2314193025 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f288418d870 0x7f28840b19b0 secure :-1 s=READY pgs=2865 cs=0 l=1 rev1=1 rx=0x7f288000a5d0 tx=0x7f288000a970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:46.785 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:46.779 7f28767fc700 1 -- 172.21.2.102:0/2314193025 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2880000ce0 con 0x7f288418d870 2021-02-10T11:36:46.785 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:46.779 7f28767fc700 1 -- 172.21.2.102:0/2314193025 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28800043a0 con 0x7f288418d870 2021-02-10T11:36:46.785 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:46.779 7f28767fc700 1 -- 172.21.2.102:0/2314193025 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2880004800 con 0x7f288418d870 2021-02-10T11:36:46.786 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:46.779 7f288bd90700 1 -- 172.21.2.102:0/2314193025 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f288409ad50 con 0x7f288418d870 2021-02-10T11:36:46.786 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:46.779 7f288bd90700 1 -- 172.21.2.102:0/2314193025 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f288409ad50 con 0x7f288418d870 2021-02-10T11:36:46.786 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:46.780 7f28767fc700 1 -- 172.21.2.102:0/2314193025 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f2880004580 con 0x7f288418d870 2021-02-10T11:36:46.786 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:46.780 7f28767fc700 1 --2- 172.21.2.102:0/2314193025 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f286c0186c0 0x7f286c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:46.786 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:46.780 7f28767fc700 1 -- 172.21.2.102:0/2314193025 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f2880032510 con 0x7f288418d870 2021-02-10T11:36:46.786 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:46.780 7f28893ab700 1 --2- 172.21.2.102:0/2314193025 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f286c0186c0 0x7f286c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:46.786 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:46.780 7f28893ab700 1 --2- 172.21.2.102:0/2314193025 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f286c0186c0 0x7f286c01abc0 crc :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:46.812 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:46.810 7f288bd90700 1 -- 172.21.2.102:0/2314193025 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28840999c0 con 0x7f288418d870 2021-02-10T11:36:46.815 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:46.813 7f28767fc700 1 -- 172.21.2.102:0/2314193025 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f2880000cc0 con 0x7f288418d870 2021-02-10T11:36:46.939 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:46.937 7f288bd90700 1 -- 172.21.2.102:0/2314193025 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f28840999c0 con 0x7f288418d870 2021-02-10T11:36:46.939 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:46.938 7f28767fc700 1 -- 172.21.2.102:0/2314193025 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v161) v1 ==== 74+0+14020 (secure 0 0 0) 0x7f2880004550 con 0x7f288418d870 2021-02-10T11:36:46.940 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:36:46.940 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":161,"fsid":"68db30ea-dbce-4aef-a6df-f01c6fea0513","created":"2021-02-10 11:03:06.331557","modified":"2021-02-10 11:36:46.331787","last_up_change":"2021-02-10 11:03:32.690562","last_in_change":"2021-02-10 11:03:32.690562","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":35,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:03:37.322519","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":34,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:36:44.297293","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"161","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":35,"pool_name":"cephfs_data","create_time":"2021-02-10 11:36:45.321943","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"161","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"be691b4d-a039-408a-af80-9d53f02b6f1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6818","nonce":36088},{"type":"v1","addr":"172.21.2.102:6820","nonce":36088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6822","nonce":36088},{"type":"v1","addr":"172.21.2.102:6824","nonce":36088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6830","nonce":36088},{"type":"v1","addr":"172.21.2.102:6832","nonce":36088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6826","nonce":36088},{"type":"v1","addr":"172.21.2.102:6828","nonce":36088}]},"public_addr":"172.21.2.102:6820/36088","cluster_addr":"172.21.2.102:6824/36088","heartbeat_back_addr":"172.21.2.102:6832/36088","heartbeat_front_addr":"172.21.2.102:6828/36088","state":["exists","up"]},{"osd":1,"uuid":"64e3ccc8-9c69-4d7c-865e-af8e83822635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6819","nonce":36087},{"type":"v1","addr":"172.21.2.102:6821","nonce":36087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6823","nonce":36087},{"type":"v1","addr":"172.21.2.102:6825","nonce":36087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6831","nonce":36087},{"type":"v1","addr":"172.21.2.102:6833","nonce":36087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6827","nonce":36087},{"type":"v1","addr":"172.21.2.102:6829","nonce":36087}]},"public_addr":"172.21.2.102:6821/36087","cluster_addr":"172.21.2.102:6825/36087","heartbeat_back_addr":"172.21.2.102:6833/36087","heartbeat_front_addr":"172.21.2.102:6829/36087","state":["exists","up"]},{"osd":2,"uuid":"153700af-747f-481f-bdf4-d6072d3a842f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6810","nonce":36089},{"type":"v1","addr":"172.21.2.102:6811","nonce":36089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6812","nonce":36089},{"type":"v1","addr":"172.21.2.102:6813","nonce":36089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6816","nonce":36089},{"type":"v1","addr":"172.21.2.102:6817","nonce":36089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6814","nonce":36089},{"type":"v1","addr":"172.21.2.102:6815","nonce":36089}]},"public_addr":"172.21.2.102:6811/36089","cluster_addr":"172.21.2.102:6813/36089","heartbeat_back_addr":"172.21.2.102:6817/36089","heartbeat_front_addr":"172.21.2.102:6815/36089","state":["exists","up"]},{"osd":3,"uuid":"cbc678f4-83a9-45e6-8830-2f7c232b1ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6802","nonce":36090},{"type":"v1","addr":"172.21.2.102:6803","nonce":36090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6804","nonce":36090},{"type":"v1","addr":"172.21.2.102:6805","nonce":36090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6808","nonce":36090},{"type":"v1","addr":"172.21.2.102:6809","nonce":36090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6806","nonce":36090},{"type":"v1","addr":"172.21.2.102:6807","nonce":36090}]},"public_addr":"172.21.2.102:6803/36090","cluster_addr":"172.21.2.102:6805/36090","heartbeat_back_addr":"172.21.2.102:6809/36090","heartbeat_front_addr":"172.21.2.102:6807/36090","state":["exists","up"]},{"osd":4,"uuid":"c01e4008-b258-4f10-afd2-9e7f37214351","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6808","nonce":36368},{"type":"v1","addr":"172.21.2.113:6811","nonce":36368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6814","nonce":36368},{"type":"v1","addr":"172.21.2.113:6818","nonce":36368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6826","nonce":36368},{"type":"v1","addr":"172.21.2.113:6828","nonce":36368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6820","nonce":36368},{"type":"v1","addr":"172.21.2.113:6824","nonce":36368}]},"public_addr":"172.21.2.113:6811/36368","cluster_addr":"172.21.2.113:6818/36368","heartbeat_back_addr":"172.21.2.113:6828/36368","heartbeat_front_addr":"172.21.2.113:6824/36368","state":["exists","up"]},{"osd":5,"uuid":"551f6007-daf8-4a98-8ee0-c2b16bcccb5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6802","nonce":36366},{"type":"v1","addr":"172.21.2.113:6805","nonce":36366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6807","nonce":36366},{"type":"v1","addr":"172.21.2.113:6810","nonce":36366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6819","nonce":36366},{"type":"v1","addr":"172.21.2.113:6822","nonce":36366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6812","nonce":36366},{"type":"v1","addr":"172.21.2.113:6815","nonce":36366}]},"public_addr":"172.21.2.113:6805/36366","cluster_addr":"172.21.2.113:6810/36366","heartbeat_back_addr":"172.21.2.113:6822/36366","heartbeat_front_addr":"172.21.2.113:6815/36366","state":["exists","up"]},{"osd":6,"uuid":"bf67c040-0e50-48be-b102-c3943bc9ebef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6800","nonce":36367},{"type":"v1","addr":"172.21.2.113:6801","nonce":36367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6803","nonce":36367},{"type":"v1","addr":"172.21.2.113:6804","nonce":36367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6813","nonce":36367},{"type":"v1","addr":"172.21.2.113:6816","nonce":36367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6806","nonce":36367},{"type":"v1","addr":"172.21.2.113:6809","nonce":36367}]},"public_addr":"172.21.2.113:6801/36367","cluster_addr":"172.21.2.113:6804/36367","heartbeat_back_addr":"172.21.2.113:6816/36367","heartbeat_front_addr":"172.21.2.113:6809/36367","state":["exists","up"]},{"osd":7,"uuid":"f5461f80-018d-428f-be89-2eb2bcbe7bb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6817","nonce":36365},{"type":"v1","addr":"172.21.2.113:6821","nonce":36365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6823","nonce":36365},{"type":"v1","addr":"172.21.2.113:6825","nonce":36365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6830","nonce":36365},{"type":"v1","addr":"172.21.2.113:6831","nonce":36365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6827","nonce":36365},{"type":"v1","addr":"172.21.2.113:6829","nonce":36365}]},"public_addr":"172.21.2.113:6821/36365","cluster_addr":"172.21.2.113:6825/36365","heartbeat_back_addr":"172.21.2.113:6831/36365","heartbeat_front_addr":"172.21.2.113:6829/36365","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:36:46.942 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:46.939 7f288bd90700 1 -- 172.21.2.102:0/2314193025 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f286c0186c0 msgr2=0x7f286c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:46.942 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:46.939 7f288bd90700 1 --2- 172.21.2.102:0/2314193025 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f286c0186c0 0x7f286c01abc0 crc :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:46.942 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:46.939 7f288bd90700 1 -- 172.21.2.102:0/2314193025 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f288418d870 msgr2=0x7f28840b19b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:46.942 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:46.939 7f288bd90700 1 --2- 172.21.2.102:0/2314193025 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f288418d870 0x7f28840b19b0 secure :-1 s=READY pgs=2865 cs=0 l=1 rev1=1 rx=0x7f288000a5d0 tx=0x7f288000a970).stop 2021-02-10T11:36:46.942 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:46.939 7f288bd90700 1 -- 172.21.2.102:0/2314193025 shutdown_connections 2021-02-10T11:36:46.942 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:46.939 7f288bd90700 1 --2- 172.21.2.102:0/2314193025 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f28840b23d0 0x7f288409a510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:46.942 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:46.939 7f288bd90700 1 --2- 172.21.2.102:0/2314193025 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f286c0186c0 0x7f286c01abc0 unknown :-1 s=CLOSED pgs=1258 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:46.942 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:46.939 7f288bd90700 1 --2- 172.21.2.102:0/2314193025 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f28840b1fd0 0x7f2884099ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:46.942 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:46.939 7f288bd90700 1 --2- 172.21.2.102:0/2314193025 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f288418d870 0x7f28840b19b0 unknown :-1 s=CLOSED pgs=2865 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:46.942 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:46.939 7f288bd90700 1 -- 172.21.2.102:0/2314193025 >> 172.21.2.102:0/2314193025 conn(0x7f28840f0550 msgr2=0x7f2884191730 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:46.942 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:46.939 7f288bd90700 1 -- 172.21.2.102:0/2314193025 shutdown_connections 2021-02-10T11:36:46.942 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:46.940 7f288bd90700 1 -- 172.21.2.102:0/2314193025 wait complete. 2021-02-10T11:36:46.950 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:36:47.379 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:47.360 7f1dd63aa700 1 Processor -- start 2021-02-10T11:36:47.379 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:47.361 7f1dd63aa700 1 -- start start 2021-02-10T11:36:47.380 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:47.361 7f1dd63aa700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f1dd00d0340 0x7f1dd00ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:47.380 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:47.361 7f1dd63aa700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1dd00d1c70 0x7f1dd00efc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:47.380 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:47.361 7f1dd63aa700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1dd00f0b10 con 0x7f1dd00d1870 2021-02-10T11:36:47.380 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:47.361 7f1dd63aa700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1dd00e5ed0 con 0x7f1dd00d0340 2021-02-10T11:36:47.380 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:47.361 7f1dd63aa700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1dd01b0be0 con 0x7f1dd00d1c70 2021-02-10T11:36:47.380 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:47.361 7f1dcf7fe700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f1dd00d0340 0x7f1dd00ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:47.380 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:47.361 7f1dcf7fe700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f1dd00d0340 0x7f1dd00ef640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:38092/0 (socket says 172.21.2.102:38092) 2021-02-10T11:36:47.380 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:47.361 7f1dcf7fe700 1 -- 172.21.2.102:0/3484802272 learned_addr learned my addr 172.21.2.102:0/3484802272 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:36:47.380 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:47.362 7f1dd49c7700 1 --2- 172.21.2.102:0/3484802272 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1dd00d1c70 0x7f1dd00efc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:47.380 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:47.362 7f1dcf7fe700 1 -- 172.21.2.102:0/3484802272 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1dd00d1c70 msgr2=0x7f1dd00efc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:47.380 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:47.362 7f1dcf7fe700 1 --2- 172.21.2.102:0/3484802272 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1dd00d1c70 0x7f1dd00efc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:47.380 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:47.362 7f1dcf7fe700 1 -- 172.21.2.102:0/3484802272 >> v1:172.21.2.102:6789/0 conn(0x7f1dd00d1870 legacy=0x7f1dd01b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:47.381 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:47.362 7f1dcf7fe700 1 -- 172.21.2.102:0/3484802272 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dd00f20a0 con 0x7f1dd00d0340 2021-02-10T11:36:47.381 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:47.362 7f1dd49c7700 1 --2- 172.21.2.102:0/3484802272 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1dd00d1c70 0x7f1dd00efc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:36:47.381 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:47.362 7f1dcf7fe700 1 --2- 172.21.2.102:0/3484802272 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1dd00d0340 0x7f1dd00ef640 secure :-1 s=READY pgs=2867 cs=0 l=1 rev1=1 rx=0x7f1dc0006700 tx=0x7f1dc0009b80).ready entity=mon.0 client_cookie=a58917410089b2c7 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:47.381 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:47.362 7f1dceffd700 1 -- 172.21.2.102:0/3484802272 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1dc000c0c0 con 0x7f1dd00d0340 2021-02-10T11:36:47.381 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:47.362 7f1dceffd700 1 -- 172.21.2.102:0/3484802272 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1dc000c440 con 0x7f1dd00d0340 2021-02-10T11:36:47.381 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:47.362 7f1dceffd700 1 -- 172.21.2.102:0/3484802272 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1dc000c960 con 0x7f1dd00d0340 2021-02-10T11:36:47.381 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:47.362 7f1dd63aa700 1 -- 172.21.2.102:0/3484802272 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1dd00d0340 msgr2=0x7f1dd00ef640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:47.381 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:47.362 7f1dd63aa700 1 --2- 172.21.2.102:0/3484802272 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1dd00d0340 0x7f1dd00ef640 secure :-1 s=READY pgs=2867 cs=0 l=1 rev1=1 rx=0x7f1dc0006700 tx=0x7f1dc0009b80).stop 2021-02-10T11:36:47.381 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:47.363 7f1dd63aa700 1 -- 172.21.2.102:0/3484802272 shutdown_connections 2021-02-10T11:36:47.381 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:47.363 7f1dd63aa700 1 --2- 172.21.2.102:0/3484802272 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1dd00d0340 0x7f1dd00ef640 unknown :-1 s=CLOSED pgs=2867 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:47.381 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:47.363 7f1dd63aa700 1 --2- 172.21.2.102:0/3484802272 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1dd00d1c70 0x7f1dd00efc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:47.381 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:47.363 7f1dd63aa700 1 -- 172.21.2.102:0/3484802272 >> 172.21.2.102:0/3484802272 conn(0x7f1dd00ce450 msgr2=0x7f1dd00d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:47.381 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:47.363 7f1dd63aa700 1 -- 172.21.2.102:0/3484802272 shutdown_connections 2021-02-10T11:36:47.381 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:47.363 7f1dd63aa700 1 -- 172.21.2.102:0/3484802272 wait complete. 2021-02-10T11:36:47.383 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:47.378 7f1dd63aa700 1 Processor -- start 2021-02-10T11:36:47.383 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:47.378 7f1dd63aa700 1 -- start start 2021-02-10T11:36:47.383 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:47.378 7f1dd63aa700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1dd01b8ad0 0x7f1dd0187e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:47.383 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:47.378 7f1dd63aa700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1dd01b8ed0 0x7f1dd00b1af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:47.383 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:47.378 7f1dd63aa700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1dd00b2110 0x7f1dd01b7a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:47.383 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:47.378 7f1dd63aa700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dd0188470 con 0x7f1dd01b8ad0 2021-02-10T11:36:47.383 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:47.378 7f1dd63aa700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dd018be10 con 0x7f1dd01b8ed0 2021-02-10T11:36:47.383 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:47.378 7f1dd63aa700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1dd018c030 con 0x7f1dd00b2110 2021-02-10T11:36:47.384 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:47.378 7f1dcffff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1dd01b8ad0 0x7f1dd0187e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:47.384 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:47.378 7f1dcffff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1dd01b8ad0 0x7f1dd0187e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:38098/0 (socket says 172.21.2.102:38098) 2021-02-10T11:36:47.384 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:47.378 7f1dcffff700 1 -- 172.21.2.102:0/448577710 learned_addr learned my addr 172.21.2.102:0/448577710 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:36:47.384 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:47.379 7f1dcf7fe700 1 --2- 172.21.2.102:0/448577710 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1dd01b8ed0 0x7f1dd00b1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:47.384 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:47.379 7f1dd49c7700 1 --2- 172.21.2.102:0/448577710 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1dd00b2110 0x7f1dd01b7a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:47.384 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:47.379 7f1dcffff700 1 -- 172.21.2.102:0/448577710 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1dd00b2110 msgr2=0x7f1dd01b7a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:47.384 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:47.379 7f1dcffff700 1 --2- 172.21.2.102:0/448577710 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1dd00b2110 0x7f1dd01b7a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:47.384 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:47.379 7f1dcffff700 1 -- 172.21.2.102:0/448577710 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1dd01b8ed0 msgr2=0x7f1dd00b1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:47.384 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:47.379 7f1dcffff700 1 --2- 172.21.2.102:0/448577710 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1dd01b8ed0 0x7f1dd00b1af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:47.384 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:47.379 7f1dcffff700 1 -- 172.21.2.102:0/448577710 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dd00b2510 con 0x7f1dd01b8ad0 2021-02-10T11:36:47.384 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:47.379 7f1dcffff700 1 --2- 172.21.2.102:0/448577710 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1dd01b8ad0 0x7f1dd0187e50 secure :-1 s=READY pgs=2868 cs=0 l=1 rev1=1 rx=0x7f1db8006f00 tx=0x7f1db800b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:47.384 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:47.379 7f1dccff9700 1 -- 172.21.2.102:0/448577710 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1db8007070 con 0x7f1dd01b8ad0 2021-02-10T11:36:47.385 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:47.379 7f1dccff9700 1 -- 172.21.2.102:0/448577710 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1db8007270 con 0x7f1dd01b8ad0 2021-02-10T11:36:47.385 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:47.379 7f1dd63aa700 1 -- 172.21.2.102:0/448577710 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1dd00b2510 con 0x7f1dd01b8ad0 2021-02-10T11:36:47.385 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:47.379 7f1dd63aa700 1 -- 172.21.2.102:0/448577710 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1dd01b8020 con 0x7f1dd01b8ad0 2021-02-10T11:36:47.385 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:47.379 7f1dccff9700 1 -- 172.21.2.102:0/448577710 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1db8007790 con 0x7f1dd01b8ad0 2021-02-10T11:36:47.385 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:47.380 7f1dccff9700 1 -- 172.21.2.102:0/448577710 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1db8007450 con 0x7f1dd01b8ad0 2021-02-10T11:36:47.385 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:47.380 7f1dccff9700 1 --2- 172.21.2.102:0/448577710 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1db40186c0 0x7f1db401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:47.385 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:47.380 7f1dccff9700 1 -- 172.21.2.102:0/448577710 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(162..162 src has 1..162) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f1db8032850 con 0x7f1dd01b8ad0 2021-02-10T11:36:47.385 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:47.380 7f1dcf7fe700 1 --2- 172.21.2.102:0/448577710 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1db40186c0 0x7f1db401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:47.385 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:47.381 7f1dcf7fe700 1 --2- 172.21.2.102:0/448577710 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1db40186c0 0x7f1db401abc0 crc :-1 s=READY pgs=1259 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:47.413 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:47.411 7f1dd63aa700 1 -- 172.21.2.102:0/448577710 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1da8002fd0 con 0x7f1dd01b8ad0 2021-02-10T11:36:47.416 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:47.414 7f1dccff9700 1 -- 172.21.2.102:0/448577710 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f1db8007a50 con 0x7f1dd01b8ad0 2021-02-10T11:36:47.541 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:47.539 7f1dd63aa700 1 -- 172.21.2.102:0/448577710 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1da8002de0 con 0x7f1dd01b8ad0 2021-02-10T11:36:47.542 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:47.540 7f1dccff9700 1 -- 172.21.2.102:0/448577710 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v162) v1 ==== 74+0+14007 (secure 0 0 0) 0x7f1db8007250 con 0x7f1dd01b8ad0 2021-02-10T11:36:47.542 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:36:47.543 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":162,"fsid":"68db30ea-dbce-4aef-a6df-f01c6fea0513","created":"2021-02-10 11:03:06.331557","modified":"2021-02-10 11:36:47.333734","last_up_change":"2021-02-10 11:03:32.690562","last_in_change":"2021-02-10 11:03:32.690562","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":35,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:03:37.322519","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":34,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:36:44.297293","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"161","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":35,"pool_name":"cephfs_data","create_time":"2021-02-10 11:36:45.321943","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"162","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"be691b4d-a039-408a-af80-9d53f02b6f1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6818","nonce":36088},{"type":"v1","addr":"172.21.2.102:6820","nonce":36088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6822","nonce":36088},{"type":"v1","addr":"172.21.2.102:6824","nonce":36088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6830","nonce":36088},{"type":"v1","addr":"172.21.2.102:6832","nonce":36088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6826","nonce":36088},{"type":"v1","addr":"172.21.2.102:6828","nonce":36088}]},"public_addr":"172.21.2.102:6820/36088","cluster_addr":"172.21.2.102:6824/36088","heartbeat_back_addr":"172.21.2.102:6832/36088","heartbeat_front_addr":"172.21.2.102:6828/36088","state":["exists","up"]},{"osd":1,"uuid":"64e3ccc8-9c69-4d7c-865e-af8e83822635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6819","nonce":36087},{"type":"v1","addr":"172.21.2.102:6821","nonce":36087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6823","nonce":36087},{"type":"v1","addr":"172.21.2.102:6825","nonce":36087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6831","nonce":36087},{"type":"v1","addr":"172.21.2.102:6833","nonce":36087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6827","nonce":36087},{"type":"v1","addr":"172.21.2.102:6829","nonce":36087}]},"public_addr":"172.21.2.102:6821/36087","cluster_addr":"172.21.2.102:6825/36087","heartbeat_back_addr":"172.21.2.102:6833/36087","heartbeat_front_addr":"172.21.2.102:6829/36087","state":["exists","up"]},{"osd":2,"uuid":"153700af-747f-481f-bdf4-d6072d3a842f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6810","nonce":36089},{"type":"v1","addr":"172.21.2.102:6811","nonce":36089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6812","nonce":36089},{"type":"v1","addr":"172.21.2.102:6813","nonce":36089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6816","nonce":36089},{"type":"v1","addr":"172.21.2.102:6817","nonce":36089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6814","nonce":36089},{"type":"v1","addr":"172.21.2.102:6815","nonce":36089}]},"public_addr":"172.21.2.102:6811/36089","cluster_addr":"172.21.2.102:6813/36089","heartbeat_back_addr":"172.21.2.102:6817/36089","heartbeat_front_addr":"172.21.2.102:6815/36089","state":["exists","up"]},{"osd":3,"uuid":"cbc678f4-83a9-45e6-8830-2f7c232b1ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6802","nonce":36090},{"type":"v1","addr":"172.21.2.102:6803","nonce":36090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6804","nonce":36090},{"type":"v1","addr":"172.21.2.102:6805","nonce":36090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6808","nonce":36090},{"type":"v1","addr":"172.21.2.102:6809","nonce":36090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6806","nonce":36090},{"type":"v1","addr":"172.21.2.102:6807","nonce":36090}]},"public_addr":"172.21.2.102:6803/36090","cluster_addr":"172.21.2.102:6805/36090","heartbeat_back_addr":"172.21.2.102:6809/36090","heartbeat_front_addr":"172.21.2.102:6807/36090","state":["exists","up"]},{"osd":4,"uuid":"c01e4008-b258-4f10-afd2-9e7f37214351","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6808","nonce":36368},{"type":"v1","addr":"172.21.2.113:6811","nonce":36368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6814","nonce":36368},{"type":"v1","addr":"172.21.2.113:6818","nonce":36368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6826","nonce":36368},{"type":"v1","addr":"172.21.2.113:6828","nonce":36368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6820","nonce":36368},{"type":"v1","addr":"172.21.2.113:6824","nonce":36368}]},"public_addr":"172.21.2.113:6811/36368","cluster_addr":"172.21.2.113:6818/36368","heartbeat_back_addr":"172.21.2.113:6828/36368","heartbeat_front_addr":"172.21.2.113:6824/36368","state":["exists","up"]},{"osd":5,"uuid":"551f6007-daf8-4a98-8ee0-c2b16bcccb5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6802","nonce":36366},{"type":"v1","addr":"172.21.2.113:6805","nonce":36366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6807","nonce":36366},{"type":"v1","addr":"172.21.2.113:6810","nonce":36366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6819","nonce":36366},{"type":"v1","addr":"172.21.2.113:6822","nonce":36366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6812","nonce":36366},{"type":"v1","addr":"172.21.2.113:6815","nonce":36366}]},"public_addr":"172.21.2.113:6805/36366","cluster_addr":"172.21.2.113:6810/36366","heartbeat_back_addr":"172.21.2.113:6822/36366","heartbeat_front_addr":"172.21.2.113:6815/36366","state":["exists","up"]},{"osd":6,"uuid":"bf67c040-0e50-48be-b102-c3943bc9ebef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6800","nonce":36367},{"type":"v1","addr":"172.21.2.113:6801","nonce":36367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6803","nonce":36367},{"type":"v1","addr":"172.21.2.113:6804","nonce":36367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6813","nonce":36367},{"type":"v1","addr":"172.21.2.113:6816","nonce":36367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6806","nonce":36367},{"type":"v1","addr":"172.21.2.113:6809","nonce":36367}]},"public_addr":"172.21.2.113:6801/36367","cluster_addr":"172.21.2.113:6804/36367","heartbeat_back_addr":"172.21.2.113:6816/36367","heartbeat_front_addr":"172.21.2.113:6809/36367","state":["exists","up"]},{"osd":7,"uuid":"f5461f80-018d-428f-be89-2eb2bcbe7bb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6817","nonce":36365},{"type":"v1","addr":"172.21.2.113:6821","nonce":36365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6823","nonce":36365},{"type":"v1","addr":"172.21.2.113:6825","nonce":36365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6830","nonce":36365},{"type":"v1","addr":"172.21.2.113:6831","nonce":36365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6827","nonce":36365},{"type":"v1","addr":"172.21.2.113:6829","nonce":36365}]},"public_addr":"172.21.2.113:6821/36365","cluster_addr":"172.21.2.113:6825/36365","heartbeat_back_addr":"172.21.2.113:6831/36365","heartbeat_front_addr":"172.21.2.113:6829/36365","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:36:47.544 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:47.541 7f1dd63aa700 1 -- 172.21.2.102:0/448577710 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1db40186c0 msgr2=0x7f1db401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:47.544 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:47.541 7f1dd63aa700 1 --2- 172.21.2.102:0/448577710 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1db40186c0 0x7f1db401abc0 crc :-1 s=READY pgs=1259 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:47.544 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:47.541 7f1dd63aa700 1 -- 172.21.2.102:0/448577710 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1dd01b8ad0 msgr2=0x7f1dd0187e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:47.544 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:47.541 7f1dd63aa700 1 --2- 172.21.2.102:0/448577710 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1dd01b8ad0 0x7f1dd0187e50 secure :-1 s=READY pgs=2868 cs=0 l=1 rev1=1 rx=0x7f1db8006f00 tx=0x7f1db800b940).stop 2021-02-10T11:36:47.544 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:47.542 7f1dd63aa700 1 -- 172.21.2.102:0/448577710 shutdown_connections 2021-02-10T11:36:47.544 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:47.542 7f1dd63aa700 1 --2- 172.21.2.102:0/448577710 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1dd00b2110 0x7f1dd01b7a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:47.544 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:47.542 7f1dd63aa700 1 --2- 172.21.2.102:0/448577710 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1db40186c0 0x7f1db401abc0 unknown :-1 s=CLOSED pgs=1259 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:47.544 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:47.542 7f1dd63aa700 1 --2- 172.21.2.102:0/448577710 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1dd01b8ed0 0x7f1dd00b1af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:47.544 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:47.542 7f1dd63aa700 1 --2- 172.21.2.102:0/448577710 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1dd01b8ad0 0x7f1dd0187e50 unknown :-1 s=CLOSED pgs=2868 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:47.544 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:47.542 7f1dd63aa700 1 -- 172.21.2.102:0/448577710 >> 172.21.2.102:0/448577710 conn(0x7f1dd00ed810 msgr2=0x7f1dd0094980 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:47.545 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:47.542 7f1dd63aa700 1 -- 172.21.2.102:0/448577710 shutdown_connections 2021-02-10T11:36:47.545 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:47.542 7f1dd63aa700 1 -- 172.21.2.102:0/448577710 wait complete. 2021-02-10T11:36:47.553 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2021-02-10T11:36:47.988 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:47.968 7f2842718700 1 Processor -- start 2021-02-10T11:36:47.988 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:47.968 7f2842718700 1 -- start start 2021-02-10T11:36:47.988 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:47.968 7f2842718700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f283c0d1870 0x7f283c1b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:47.988 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:47.968 7f2842718700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f283c0cff70 0x7f283c109130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:47.988 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:47.968 7f2842718700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f283c109a60 con 0x7f283c1b0ae0 2021-02-10T11:36:47.988 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:47.968 7f2842718700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f283c109750 con 0x7f283c0cff70 2021-02-10T11:36:47.988 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:47.968 7f2842718700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f283c088070 con 0x7f283c0d1870 2021-02-10T11:36:47.988 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:47.968 7f283b7fe700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f283c1b0ae0 0x7f283c108940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:58564/0 (socket says 172.21.2.102:58564) 2021-02-10T11:36:47.989 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:47.969 7f283b7fe700 1 -- 172.21.2.102:0/2122894549 learned_addr learned my addr 172.21.2.102:0/2122894549 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:36:47.989 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:47.969 7f283affd700 1 -- 172.21.2.102:0/2122894549 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 320825143 0 0) 0x7f28300009b0 con 0x7f283c1b0ae0 2021-02-10T11:36:47.989 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:47.969 7f283affd700 1 -- 172.21.2.102:0/2122894549 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2824002fe0 con 0x7f283c1b0ae0 2021-02-10T11:36:47.989 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:47.969 7f283affd700 1 -- 172.21.2.102:0/2122894549 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3409716486 0 0) 0x7f28300010b0 con 0x7f283c1b0ae0 2021-02-10T11:36:47.989 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:47.969 7f283affd700 1 -- 172.21.2.102:0/2122894549 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f283c0d1870 msgr2=0x7f283c1b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:47.989 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:47.969 7f283affd700 1 --2- 172.21.2.102:0/2122894549 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f283c0d1870 0x7f283c1b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:47.989 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:47.969 7f283affd700 1 -- 172.21.2.102:0/2122894549 >> v2:172.21.2.113:3300/0 conn(0x7f283c0cff70 msgr2=0x7f283c109130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:47.989 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:47.969 7f283affd700 1 --2- 172.21.2.102:0/2122894549 >> v2:172.21.2.113:3300/0 conn(0x7f283c0cff70 0x7f283c109130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:47.989 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:47.969 7f283affd700 1 -- 172.21.2.102:0/2122894549 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f283c088260 con 0x7f283c1b0ae0 2021-02-10T11:36:47.989 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:47.969 7f283affd700 1 -- 172.21.2.102:0/2122894549 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7f28300015b0 con 0x7f283c1b0ae0 2021-02-10T11:36:47.989 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:47.969 7f283affd700 1 -- 172.21.2.102:0/2122894549 >> v1:172.21.2.102:6789/0 conn(0x7f283c1b0ae0 legacy=0x7f283c108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:47.989 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:47.969 7f283affd700 1 --2- 172.21.2.102:0/2122894549 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2824003a60 0x7f2824005ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:47.989 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:47.969 7f283affd700 1 --2- 172.21.2.102:0/2122894549 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2824006790 0x7f2824008c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:47.989 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:47.969 7f283affd700 1 --2- 172.21.2.102:0/2122894549 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2824009590 0x7f282400ba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:47.990 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:47.969 7f283affd700 1 -- 172.21.2.102:0/2122894549 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f282400c2d0 con 0x7f2824006790 2021-02-10T11:36:47.990 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:47.969 7f283affd700 1 -- 172.21.2.102:0/2122894549 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f282400c4f0 con 0x7f2824003a60 2021-02-10T11:36:47.990 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:47.969 7f283affd700 1 -- 172.21.2.102:0/2122894549 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f282400c710 con 0x7f2824009590 2021-02-10T11:36:47.990 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:47.969 7f283b7fe700 1 --2- 172.21.2.102:0/2122894549 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2824006790 0x7f2824008c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:47.990 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:47.970 7f283bfff700 1 --2- 172.21.2.102:0/2122894549 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2824003a60 0x7f2824005ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:47.990 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:47.970 7f2840d35700 1 --2- 172.21.2.102:0/2122894549 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2824009590 0x7f282400ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:47.990 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:47.970 7f283b7fe700 1 -- 172.21.2.102:0/2122894549 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2824009590 msgr2=0x7f282400ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:47.990 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:47.970 7f283b7fe700 1 --2- 172.21.2.102:0/2122894549 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2824009590 0x7f282400ba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:47.990 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:47.970 7f283b7fe700 1 -- 172.21.2.102:0/2122894549 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2824003a60 msgr2=0x7f2824005ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:47.990 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:47.970 7f283b7fe700 1 --2- 172.21.2.102:0/2122894549 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2824003a60 0x7f2824005ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:47.990 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:47.970 7f283b7fe700 1 -- 172.21.2.102:0/2122894549 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f282400c9b0 con 0x7f2824006790 2021-02-10T11:36:47.990 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:47.970 7f283bfff700 1 --2- 172.21.2.102:0/2122894549 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2824003a60 0x7f2824005ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:36:47.990 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:47.970 7f283b7fe700 1 --2- 172.21.2.102:0/2122894549 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2824006790 0x7f2824008c90 secure :-1 s=READY pgs=2870 cs=0 l=1 rev1=1 rx=0x7f2830001c60 tx=0x7f2830007d40).ready entity=mon.0 client_cookie=b42f028d23e7a9c5 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:47.991 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:47.970 7f283affd700 1 -- 172.21.2.102:0/2122894549 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f28300070d0 con 0x7f2824006790 2021-02-10T11:36:47.991 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:47.970 7f283affd700 1 -- 172.21.2.102:0/2122894549 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2830001010 con 0x7f2824006790 2021-02-10T11:36:47.991 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:47.970 7f283affd700 1 -- 172.21.2.102:0/2122894549 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f283004e5a0 con 0x7f2824006790 2021-02-10T11:36:47.991 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:47.970 7f2842718700 1 -- 172.21.2.102:0/2122894549 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2824006790 msgr2=0x7f2824008c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:47.991 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:47.970 7f2842718700 1 --2- 172.21.2.102:0/2122894549 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2824006790 0x7f2824008c90 secure :-1 s=READY pgs=2870 cs=0 l=1 rev1=1 rx=0x7f2830001c60 tx=0x7f2830007d40).stop 2021-02-10T11:36:47.991 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:47.970 7f2840d35700 1 -- 172.21.2.102:0/2122894549 reap_dead start 2021-02-10T11:36:47.991 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:47.971 7f2842718700 1 -- 172.21.2.102:0/2122894549 shutdown_connections 2021-02-10T11:36:47.991 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:47.971 7f2842718700 1 -- 172.21.2.102:0/2122894549 >> 172.21.2.102:0/2122894549 conn(0x7f283c0ce420 msgr2=0x7f283c0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:47.991 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:47.971 7f2842718700 1 -- 172.21.2.102:0/2122894549 shutdown_connections 2021-02-10T11:36:47.991 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:47.971 7f2842718700 1 -- 172.21.2.102:0/2122894549 wait complete. 2021-02-10T11:36:47.991 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:47.986 7f2842718700 1 Processor -- start 2021-02-10T11:36:47.991 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:47.986 7f2842718700 1 -- start start 2021-02-10T11:36:47.993 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:47.986 7f2842718700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f283c0c1850 0x7f283c0c1f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:47.993 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:47.986 7f2842718700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f283c0c2810 0x7f283c0c2c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:47.993 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:47.986 7f2842718700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f283c0c3230 0x7f283c0c3630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:47.993 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:47.986 7f2842718700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f283c0c1c50 con 0x7f283c0c2810 2021-02-10T11:36:47.993 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:47.986 7f2842718700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f283c084920 con 0x7f283c0c3230 2021-02-10T11:36:47.993 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:47.986 7f2842718700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f283c084b40 con 0x7f283c0c1850 2021-02-10T11:36:47.993 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:47.986 7f283b7fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f283c0c2810 0x7f283c0c2c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:47.993 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:47.986 7f283b7fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f283c0c2810 0x7f283c0c2c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:38120/0 (socket says 172.21.2.102:38120) 2021-02-10T11:36:47.993 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:47.986 7f283b7fe700 1 -- 172.21.2.102:0/387396953 learned_addr learned my addr 172.21.2.102:0/387396953 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:36:47.993 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:47.986 7f2840d35700 1 --2- 172.21.2.102:0/387396953 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f283c0c3230 0x7f283c0c3630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:47.994 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:47.987 7f283bfff700 1 --2- 172.21.2.102:0/387396953 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f283c0c1850 0x7f283c0c1f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:47.994 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:47.987 7f283b7fe700 1 -- 172.21.2.102:0/387396953 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f283c0c1850 msgr2=0x7f283c0c1f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:47.994 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:47.987 7f283b7fe700 1 --2- 172.21.2.102:0/387396953 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f283c0c1850 0x7f283c0c1f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:47.994 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:47.987 7f283b7fe700 1 -- 172.21.2.102:0/387396953 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f283c0c3230 msgr2=0x7f283c0c3630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:47.994 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:47.987 7f283b7fe700 1 --2- 172.21.2.102:0/387396953 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f283c0c3230 0x7f283c0c3630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:47.994 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:47.987 7f283b7fe700 1 -- 172.21.2.102:0/387396953 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f283c084d60 con 0x7f283c0c2810 2021-02-10T11:36:47.994 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:47.987 7f283b7fe700 1 --2- 172.21.2.102:0/387396953 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f283c0c2810 0x7f283c0c2c10 secure :-1 s=READY pgs=2871 cs=0 l=1 rev1=1 rx=0x7f2830001270 tx=0x7f283004f930).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:47.994 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:47.987 7f2838ff9700 1 -- 172.21.2.102:0/387396953 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f283004fd20 con 0x7f283c0c2810 2021-02-10T11:36:47.994 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:47.987 7f2838ff9700 1 -- 172.21.2.102:0/387396953 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28300074f0 con 0x7f283c0c2810 2021-02-10T11:36:47.994 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:47.987 7f2838ff9700 1 -- 172.21.2.102:0/387396953 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f28300502f0 con 0x7f283c0c2810 2021-02-10T11:36:47.994 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:47.987 7f2842718700 1 -- 172.21.2.102:0/387396953 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f283c084d60 con 0x7f283c0c2810 2021-02-10T11:36:47.994 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:47.987 7f2842718700 1 -- 172.21.2.102:0/387396953 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f283c0852f0 con 0x7f283c0c2810 2021-02-10T11:36:47.994 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:47.988 7f2838ff9700 1 -- 172.21.2.102:0/387396953 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f283004c050 con 0x7f283c0c2810 2021-02-10T11:36:47.995 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:47.988 7f2838ff9700 1 --2- 172.21.2.102:0/387396953 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f281c0186c0 0x7f281c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:47.995 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:47.988 7f2838ff9700 1 -- 172.21.2.102:0/387396953 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(162..162 src has 1..162) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f2830076430 con 0x7f283c0c2810 2021-02-10T11:36:47.995 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:47.988 7f283bfff700 1 --2- 172.21.2.102:0/387396953 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f281c0186c0 0x7f281c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:47.995 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:47.988 7f283bfff700 1 --2- 172.21.2.102:0/387396953 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f281c0186c0 0x7f281c01abc0 crc :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:48.020 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:48.018 7f2842718700 1 -- 172.21.2.102:0/387396953 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f283c109a60 con 0x7f283c0c2810 2021-02-10T11:36:48.023 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:48.021 7f2838ff9700 1 -- 172.21.2.102:0/387396953 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f283006cec0 con 0x7f283c0c2810 2021-02-10T11:36:48.158 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:48.156 7f2842718700 1 -- 172.21.2.102:0/387396953 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"} v 0) v1 -- 0x7f283c109a60 con 0x7f283c0c2810 2021-02-10T11:36:48.345 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:48.343 7f2838ff9700 1 -- 172.21.2.102:0/387396953 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"}]=0 v141) v1 ==== 118+0+0 (secure 0 0 0) 0x7f283006cec0 con 0x7f283c0c2810 2021-02-10T11:36:48.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:48.373 7f2842718700 1 -- 172.21.2.102:0/387396953 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f281c0186c0 msgr2=0x7f281c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:48.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:48.373 7f2842718700 1 --2- 172.21.2.102:0/387396953 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f281c0186c0 0x7f281c01abc0 crc :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:48.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:48.373 7f2842718700 1 -- 172.21.2.102:0/387396953 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f283c0c2810 msgr2=0x7f283c0c2c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:48.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:48.373 7f2842718700 1 --2- 172.21.2.102:0/387396953 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f283c0c2810 0x7f283c0c2c10 secure :-1 s=READY pgs=2871 cs=0 l=1 rev1=1 rx=0x7f2830001270 tx=0x7f283004f930).stop 2021-02-10T11:36:48.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:48.374 7f2842718700 1 -- 172.21.2.102:0/387396953 shutdown_connections 2021-02-10T11:36:48.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:48.374 7f2842718700 1 --2- 172.21.2.102:0/387396953 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f281c0186c0 0x7f281c01abc0 unknown :-1 s=CLOSED pgs=1260 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:48.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:48.374 7f2842718700 1 --2- 172.21.2.102:0/387396953 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f283c0c3230 0x7f283c0c3630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:48.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:48.374 7f2842718700 1 --2- 172.21.2.102:0/387396953 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f283c0c2810 0x7f283c0c2c10 unknown :-1 s=CLOSED pgs=2871 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:48.377 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:48.374 7f2842718700 1 --2- 172.21.2.102:0/387396953 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f283c0c1850 0x7f283c0c1f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:48.377 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:48.374 7f2842718700 1 -- 172.21.2.102:0/387396953 >> 172.21.2.102:0/387396953 conn(0x7f283c0af690 msgr2=0x7f283c094c40 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:48.377 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:48.374 7f2842718700 1 -- 172.21.2.102:0/387396953 shutdown_connections 2021-02-10T11:36:48.377 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:48.374 7f2842718700 1 -- 172.21.2.102:0/387396953 wait complete. 2021-02-10T11:36:48.384 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:36:48.814 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:48.795 7f3ef48b5700 1 Processor -- start 2021-02-10T11:36:48.814 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:48.795 7f3ef48b5700 1 -- start start 2021-02-10T11:36:48.814 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:48.796 7f3ef48b5700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f3ef00d0340 0x7f3ef00ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:48.814 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:48.796 7f3ef48b5700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3ef00d1c70 0x7f3ef00efc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:48.814 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:48.796 7f3ef48b5700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ef00f0b10 con 0x7f3ef00d1870 2021-02-10T11:36:48.814 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:48.796 7f3ef48b5700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ef00e5ed0 con 0x7f3ef00d0340 2021-02-10T11:36:48.815 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:48.796 7f3ef48b5700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ef01b0be0 con 0x7f3ef00d1c70 2021-02-10T11:36:48.815 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:48.796 7f3eede1b700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f3ef00d0340 0x7f3ef00ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:48.815 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:48.796 7f3eede1b700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f3ef00d0340 0x7f3ef00ef640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:38126/0 (socket says 172.21.2.102:38126) 2021-02-10T11:36:48.815 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:48.796 7f3eede1b700 1 -- 172.21.2.102:0/3326593531 learned_addr learned my addr 172.21.2.102:0/3326593531 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:36:48.815 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:48.796 7f3eede1b700 1 -- 172.21.2.102:0/3326593531 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3ef00d1c70 msgr2=0x7f3ef00efc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:48.815 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:48.796 7f3eede1b700 1 --2- 172.21.2.102:0/3326593531 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3ef00d1c70 0x7f3ef00efc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:48.815 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:48.796 7f3eede1b700 1 -- 172.21.2.102:0/3326593531 >> v1:172.21.2.113:6789/0 conn(0x7f3ef00d1870 legacy=0x7f3ef01b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:48.815 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:48.796 7f3eede1b700 1 -- 172.21.2.102:0/3326593531 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ef00f20a0 con 0x7f3ef00d0340 2021-02-10T11:36:48.815 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:48.796 7f3eede1b700 1 --2- 172.21.2.102:0/3326593531 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3ef00d0340 0x7f3ef00ef640 secure :-1 s=READY pgs=2872 cs=0 l=1 rev1=1 rx=0x7f3ed8006700 tx=0x7f3ed8009b80).ready entity=mon.0 client_cookie=e44782c01497c7cb server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:48.815 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:48.797 7f3eed61a700 1 -- 172.21.2.102:0/3326593531 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3ed800c0c0 con 0x7f3ef00d0340 2021-02-10T11:36:48.815 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:48.797 7f3eed61a700 1 -- 172.21.2.102:0/3326593531 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ed800c340 con 0x7f3ef00d0340 2021-02-10T11:36:48.815 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:48.797 7f3eed61a700 1 -- 172.21.2.102:0/3326593531 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3ed800c860 con 0x7f3ef00d0340 2021-02-10T11:36:48.815 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:48.797 7f3ef48b5700 1 -- 172.21.2.102:0/3326593531 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3ef00d0340 msgr2=0x7f3ef00ef640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:48.816 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:48.797 7f3ef48b5700 1 --2- 172.21.2.102:0/3326593531 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3ef00d0340 0x7f3ef00ef640 secure :-1 s=READY pgs=2872 cs=0 l=1 rev1=1 rx=0x7f3ed8006700 tx=0x7f3ed8009b80).stop 2021-02-10T11:36:48.816 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:48.798 7f3ef48b5700 1 -- 172.21.2.102:0/3326593531 shutdown_connections 2021-02-10T11:36:48.816 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:48.798 7f3ef48b5700 1 --2- 172.21.2.102:0/3326593531 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3ef00d1c70 0x7f3ef00efc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:48.816 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:48.798 7f3ef48b5700 1 --2- 172.21.2.102:0/3326593531 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3ef00d0340 0x7f3ef00ef640 unknown :-1 s=CLOSED pgs=2872 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:48.816 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:48.798 7f3ef48b5700 1 -- 172.21.2.102:0/3326593531 >> 172.21.2.102:0/3326593531 conn(0x7f3ef00ce450 msgr2=0x7f3ef00d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:48.816 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:48.798 7f3ef48b5700 1 -- 172.21.2.102:0/3326593531 shutdown_connections 2021-02-10T11:36:48.816 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:48.798 7f3ef48b5700 1 -- 172.21.2.102:0/3326593531 wait complete. 2021-02-10T11:36:48.817 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:48.812 7f3ef48b5700 1 Processor -- start 2021-02-10T11:36:48.817 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:48.812 7f3ef48b5700 1 -- start start 2021-02-10T11:36:48.817 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:48.813 7f3ef48b5700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3ef00b0ad0 0x7f3ef0091f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:48.818 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:48.813 7f3ef48b5700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3ef01a8850 0x7f3ef01a8c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:48.818 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:48.813 7f3ef48b5700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3ef01a6fa0 0x7f3ef01a73a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:48.818 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:48.813 7f3ef48b5700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ef01abfe0 con 0x7f3ef01a6fa0 2021-02-10T11:36:48.818 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:48.813 7f3ef48b5700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ef01a9c40 con 0x7f3ef00b0ad0 2021-02-10T11:36:48.818 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:48.813 7f3ef48b5700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ef01a9730 con 0x7f3ef01a8850 2021-02-10T11:36:48.818 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:48.813 7f3eeee1d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3ef01a6fa0 0x7f3ef01a73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:48.818 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:48.813 7f3eeee1d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3ef01a6fa0 0x7f3ef01a73a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:38136/0 (socket says 172.21.2.102:38136) 2021-02-10T11:36:48.818 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:48.813 7f3eeee1d700 1 -- 172.21.2.102:0/3811123444 learned_addr learned my addr 172.21.2.102:0/3811123444 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:36:48.818 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:48.813 7f3eeee1d700 1 -- 172.21.2.102:0/3811123444 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3ef01a8850 msgr2=0x7f3ef01a8c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:48.818 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:48.813 7f3eeee1d700 1 --2- 172.21.2.102:0/3811123444 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3ef01a8850 0x7f3ef01a8c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:48.818 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:48.813 7f3eeee1d700 1 -- 172.21.2.102:0/3811123444 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3ef00b0ad0 msgr2=0x7f3ef0091f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:48.818 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:48.813 7f3eeee1d700 1 --2- 172.21.2.102:0/3811123444 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3ef00b0ad0 0x7f3ef0091f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:48.818 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:48.813 7f3eeee1d700 1 -- 172.21.2.102:0/3811123444 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ef01a9270 con 0x7f3ef01a6fa0 2021-02-10T11:36:48.819 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:48.813 7f3eeee1d700 1 --2- 172.21.2.102:0/3811123444 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3ef01a6fa0 0x7f3ef01a73a0 secure :-1 s=READY pgs=2873 cs=0 l=1 rev1=1 rx=0x7f3ee8009980 tx=0x7f3ee800bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:48.819 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:48.813 7f3ee6ffd700 1 -- 172.21.2.102:0/3811123444 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3ee8005870 con 0x7f3ef01a6fa0 2021-02-10T11:36:48.819 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:48.813 7f3ee6ffd700 1 -- 172.21.2.102:0/3811123444 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ee8005b30 con 0x7f3ef01a6fa0 2021-02-10T11:36:48.819 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:48.813 7f3ef48b5700 1 -- 172.21.2.102:0/3811123444 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ef01a9270 con 0x7f3ef01a6fa0 2021-02-10T11:36:48.819 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:48.814 7f3ef48b5700 1 -- 172.21.2.102:0/3811123444 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ef01a9270 con 0x7f3ef01a6fa0 2021-02-10T11:36:48.819 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:48.814 7f3ee6ffd700 1 -- 172.21.2.102:0/3811123444 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3ee8006030 con 0x7f3ef01a6fa0 2021-02-10T11:36:48.819 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:48.814 7f3ee6ffd700 1 -- 172.21.2.102:0/3811123444 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f3ee8005d10 con 0x7f3ef01a6fa0 2021-02-10T11:36:48.819 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:48.814 7f3ee6ffd700 1 --2- 172.21.2.102:0/3811123444 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3ed4018730 0x7f3ed401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:48.819 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:48.814 7f3ee6ffd700 1 -- 172.21.2.102:0/3811123444 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(162..162 src has 1..162) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f3ee8032b70 con 0x7f3ef01a6fa0 2021-02-10T11:36:48.819 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:48.814 7f3eee61c700 1 --2- 172.21.2.102:0/3811123444 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3ed4018730 0x7f3ed401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:48.819 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:48.814 7f3eee61c700 1 --2- 172.21.2.102:0/3811123444 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3ed4018730 0x7f3ed401ac10 crc :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:48.847 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:48.845 7f3ef48b5700 1 -- 172.21.2.102:0/3811123444 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ef01abfe0 con 0x7f3ef01a6fa0 2021-02-10T11:36:48.850 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:48.848 7f3ee6ffd700 1 -- 172.21.2.102:0/3811123444 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f3ee8005d40 con 0x7f3ef01a6fa0 2021-02-10T11:36:48.979 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:48.977 7f3ef48b5700 1 -- 172.21.2.102:0/3811123444 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3ef01abfe0 con 0x7f3ef01a6fa0 2021-02-10T11:36:48.979 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:48.977 7f3ee6ffd700 1 -- 172.21.2.102:0/3811123444 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 141 v141) v1 ==== 95+0+2108 (secure 0 0 0) 0x7f3ee8006f20 con 0x7f3ef01a6fa0 2021-02-10T11:36:48.980 INFO:teuthology.orchestra.run.gibba002.stderr:dumped fsmap epoch 141 2021-02-10T11:36:48.980 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:36:48.980 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":141,"default_fscid":17,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":12405,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.102:6835/4152930669","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6834","nonce":4152930669},{"type":"v1","addr":"172.21.2.102:6835","nonce":4152930669}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":137}],"filesystems":[{"mdsmap":{"epoch":141,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:36:46.331874","modified":"2021-02-10 11:36:48.340104","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7130},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7130":{"gid":7130,"name":"b","rank":0,"incarnation":138,"state":"up:active","state_seq":3,"addr":"172.21.2.113:6833/2966977555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6832","nonce":2966977555},{"type":"v1","addr":"172.21.2.113:6833","nonce":2966977555}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[35],"metadata_pool":34,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":17}]} 2021-02-10T11:36:48.981 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:48.979 7f3ef48b5700 1 -- 172.21.2.102:0/3811123444 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3ed4018730 msgr2=0x7f3ed401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:48.982 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:48.979 7f3ef48b5700 1 --2- 172.21.2.102:0/3811123444 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3ed4018730 0x7f3ed401ac10 crc :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:48.982 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:48.979 7f3ef48b5700 1 -- 172.21.2.102:0/3811123444 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3ef01a6fa0 msgr2=0x7f3ef01a73a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:48.982 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:48.979 7f3ef48b5700 1 --2- 172.21.2.102:0/3811123444 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3ef01a6fa0 0x7f3ef01a73a0 secure :-1 s=READY pgs=2873 cs=0 l=1 rev1=1 rx=0x7f3ee8009980 tx=0x7f3ee800bcd0).stop 2021-02-10T11:36:48.982 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:48.979 7f3ef48b5700 1 -- 172.21.2.102:0/3811123444 shutdown_connections 2021-02-10T11:36:48.982 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:48.979 7f3ef48b5700 1 --2- 172.21.2.102:0/3811123444 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3ef01a6fa0 0x7f3ef01a73a0 unknown :-1 s=CLOSED pgs=2873 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:48.982 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:48.979 7f3ef48b5700 1 --2- 172.21.2.102:0/3811123444 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3ef01a8850 0x7f3ef01a8c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:48.982 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:48.979 7f3ef48b5700 1 --2- 172.21.2.102:0/3811123444 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3ed4018730 0x7f3ed401ac10 unknown :-1 s=CLOSED pgs=1261 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:48.982 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:48.979 7f3ef48b5700 1 --2- 172.21.2.102:0/3811123444 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3ef00b0ad0 0x7f3ef0091f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:48.982 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:48.979 7f3ef48b5700 1 -- 172.21.2.102:0/3811123444 >> 172.21.2.102:0/3811123444 conn(0x7f3ef019f8f0 msgr2=0x7f3ef01a5f80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:48.982 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:48.980 7f3ef48b5700 1 -- 172.21.2.102:0/3811123444 shutdown_connections 2021-02-10T11:36:48.982 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:48.980 7f3ef48b5700 1 -- 172.21.2.102:0/3811123444 wait complete. 2021-02-10T11:36:48.990 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:36:49.420 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:49.401 7fc912944700 1 Processor -- start 2021-02-10T11:36:49.420 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:49.401 7fc912944700 1 -- start start 2021-02-10T11:36:49.420 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:49.402 7fc912944700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fc90c0d1870 0x7fc90c1b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:49.420 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:49.402 7fc912944700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc90c0f0650 0x7fc90c0cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:49.420 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:49.402 7fc912944700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc90c0f0b10 con 0x7fc90c1b0a10 2021-02-10T11:36:49.420 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:49.402 7fc912944700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc90c0e5ed0 con 0x7fc90c0d1870 2021-02-10T11:36:49.420 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:49.402 7fc912944700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc90c0d1c70 con 0x7fc90c0f0650 2021-02-10T11:36:49.420 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:49.402 7fc90bfff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fc90c0d1870 0x7fc90c1b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:49.420 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:49.402 7fc90bfff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fc90c0d1870 0x7fc90c1b03f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:38140/0 (socket says 172.21.2.102:38140) 2021-02-10T11:36:49.420 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:49.402 7fc90bfff700 1 -- 172.21.2.102:0/2130290555 learned_addr learned my addr 172.21.2.102:0/2130290555 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:36:49.420 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:49.402 7fc90bfff700 1 -- 172.21.2.102:0/2130290555 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc90c0f0650 msgr2=0x7fc90c0cfb20 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:36:49.421 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:49.402 7fc90bfff700 1 --2- 172.21.2.102:0/2130290555 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc90c0f0650 0x7fc90c0cfb20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:49.421 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:49.402 7fc90bfff700 1 -- 172.21.2.102:0/2130290555 >> v1:172.21.2.113:6789/0 conn(0x7fc90c1b0a10 legacy=0x7fc90c0cf330 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:36:49.421 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:49.402 7fc90bfff700 1 -- 172.21.2.102:0/2130290555 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc90c0d1e90 con 0x7fc90c0d1870 2021-02-10T11:36:49.421 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:49.402 7fc90bfff700 1 --2- 172.21.2.102:0/2130290555 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc90c0d1870 0x7fc90c1b03f0 secure :-1 s=READY pgs=2874 cs=0 l=1 rev1=1 rx=0x7fc8fc006700 tx=0x7fc8fc009b00).ready entity=mon.0 client_cookie=99d5a00248a3f190 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:49.421 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:49.402 7fc90affd700 1 -- 172.21.2.102:0/2130290555 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc8fc00c0c0 con 0x7fc90c0d1870 2021-02-10T11:36:49.421 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:49.403 7fc90affd700 1 -- 172.21.2.102:0/2130290555 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8fc00c400 con 0x7fc90c0d1870 2021-02-10T11:36:49.421 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:49.403 7fc90affd700 1 -- 172.21.2.102:0/2130290555 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc8fc00c8e0 con 0x7fc90c0d1870 2021-02-10T11:36:49.421 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:49.403 7fc912944700 1 -- 172.21.2.102:0/2130290555 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc90c0d1870 msgr2=0x7fc90c1b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:49.421 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:49.403 7fc912944700 1 --2- 172.21.2.102:0/2130290555 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc90c0d1870 0x7fc90c1b03f0 secure :-1 s=READY pgs=2874 cs=0 l=1 rev1=1 rx=0x7fc8fc006700 tx=0x7fc8fc009b00).stop 2021-02-10T11:36:49.421 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:49.403 7fc912944700 1 -- 172.21.2.102:0/2130290555 shutdown_connections 2021-02-10T11:36:49.421 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:49.403 7fc912944700 1 --2- 172.21.2.102:0/2130290555 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc90c0f0650 0x7fc90c0cfb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:49.421 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:49.403 7fc912944700 1 --2- 172.21.2.102:0/2130290555 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc90c0d1870 0x7fc90c1b03f0 unknown :-1 s=CLOSED pgs=2874 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:49.422 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:49.403 7fc912944700 1 -- 172.21.2.102:0/2130290555 >> 172.21.2.102:0/2130290555 conn(0x7fc90c0ce450 msgr2=0x7fc90c0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:49.422 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:49.403 7fc912944700 1 -- 172.21.2.102:0/2130290555 shutdown_connections 2021-02-10T11:36:49.422 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:49.403 7fc912944700 1 -- 172.21.2.102:0/2130290555 wait complete. 2021-02-10T11:36:49.422 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:49.418 7fc912944700 1 Processor -- start 2021-02-10T11:36:49.423 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:49.418 7fc912944700 1 -- start start 2021-02-10T11:36:49.423 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:49.418 7fc912944700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc90c18a540 0x7fc90c197480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:49.423 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:49.418 7fc912944700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc90c18aa50 0x7fc90c0b77d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:49.424 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:49.418 7fc912944700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc90c0b7df0 0x7fc90c0b81f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:49.424 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:49.418 7fc912944700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc90c1a79d0 con 0x7fc90c0b7df0 2021-02-10T11:36:49.424 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:49.418 7fc912944700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc90c1a9e20 con 0x7fc90c18a540 2021-02-10T11:36:49.424 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:49.418 7fc912944700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc90c1aa780 con 0x7fc90c18aa50 2021-02-10T11:36:49.424 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:49.419 7fc910f61700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc90c0b7df0 0x7fc90c0b81f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:49.424 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:49.419 7fc90bfff700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc90c18a540 0x7fc90c197480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:49.424 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:49.419 7fc910f61700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc90c0b7df0 0x7fc90c0b81f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:38150/0 (socket says 172.21.2.102:38150) 2021-02-10T11:36:49.424 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:49.419 7fc910f61700 1 -- 172.21.2.102:0/1816194183 learned_addr learned my addr 172.21.2.102:0/1816194183 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:36:49.424 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:49.419 7fc90b7fe700 1 --2- 172.21.2.102:0/1816194183 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc90c18aa50 0x7fc90c0b77d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:49.424 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:49.419 7fc90bfff700 1 -- 172.21.2.102:0/1816194183 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc90c18aa50 msgr2=0x7fc90c0b77d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:49.424 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:49.419 7fc90bfff700 1 --2- 172.21.2.102:0/1816194183 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc90c18aa50 0x7fc90c0b77d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:49.424 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:49.419 7fc90bfff700 1 -- 172.21.2.102:0/1816194183 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc90c0b7df0 msgr2=0x7fc90c0b81f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:49.425 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:49.419 7fc90bfff700 1 --2- 172.21.2.102:0/1816194183 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc90c0b7df0 0x7fc90c0b81f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:49.425 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:49.419 7fc90bfff700 1 -- 172.21.2.102:0/1816194183 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc90c1aa9a0 con 0x7fc90c18a540 2021-02-10T11:36:49.425 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:49.419 7fc90b7fe700 1 --2- 172.21.2.102:0/1816194183 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc90c18aa50 0x7fc90c0b77d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:36:49.425 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:49.419 7fc90bfff700 1 --2- 172.21.2.102:0/1816194183 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc90c18a540 0x7fc90c197480 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 rx=0x7fc8fc00b550 tx=0x7fc8fc00b8f0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:49.425 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:49.419 7fc908ff9700 1 -- 172.21.2.102:0/1816194183 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc8fc002030 con 0x7fc90c18a540 2021-02-10T11:36:49.425 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:49.419 7fc908ff9700 1 -- 172.21.2.102:0/1816194183 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8fc0022b0 con 0x7fc90c18a540 2021-02-10T11:36:49.425 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:49.419 7fc912944700 1 -- 172.21.2.102:0/1816194183 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc90c1aa9a0 con 0x7fc90c18a540 2021-02-10T11:36:49.425 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:49.419 7fc908ff9700 1 -- 172.21.2.102:0/1816194183 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc8fc002710 con 0x7fc90c18a540 2021-02-10T11:36:49.425 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:49.419 7fc912944700 1 -- 172.21.2.102:0/1816194183 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc90c1aa9a0 con 0x7fc90c18a540 2021-02-10T11:36:49.425 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:49.420 7fc908ff9700 1 -- 172.21.2.102:0/1816194183 <== mon.1 v2:172.21.2.113:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fc8fc002490 con 0x7fc90c18a540 2021-02-10T11:36:49.425 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:49.420 7fc908ff9700 1 --2- 172.21.2.102:0/1816194183 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc8ec0186c0 0x7fc8ec01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:49.425 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:49.420 7fc908ff9700 1 -- 172.21.2.102:0/1816194183 <== mon.1 v2:172.21.2.113:3300/0 5 ==== osd_map(162..162 src has 1..162) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fc8fc032340 con 0x7fc90c18a540 2021-02-10T11:36:49.425 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:49.420 7fc90b7fe700 1 --2- 172.21.2.102:0/1816194183 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc8ec0186c0 0x7fc8ec01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:49.426 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:49.420 7fc90b7fe700 1 --2- 172.21.2.102:0/1816194183 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc8ec0186c0 0x7fc8ec01abc0 crc :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:49.453 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:49.451 7fc912944700 1 -- 172.21.2.102:0/1816194183 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc90c0ce450 con 0x7fc90c18a540 2021-02-10T11:36:49.456 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:49.454 7fc908ff9700 1 -- 172.21.2.102:0/1816194183 <== mon.1 v2:172.21.2.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fc8fc024fb0 con 0x7fc90c18a540 2021-02-10T11:36:49.581 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:49.579 7fc912944700 1 -- 172.21.2.102:0/1816194183 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc90c0ce450 con 0x7fc90c18a540 2021-02-10T11:36:49.582 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:49.581 7fc908ff9700 1 -- 172.21.2.102:0/1816194183 <== mon.1 v2:172.21.2.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v162) v1 ==== 74+0+14007 (secure 0 0 0) 0x7fc8fc024fb0 con 0x7fc90c18a540 2021-02-10T11:36:49.584 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:36:49.584 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":162,"fsid":"68db30ea-dbce-4aef-a6df-f01c6fea0513","created":"2021-02-10 11:03:06.331557","modified":"2021-02-10 11:36:47.333734","last_up_change":"2021-02-10 11:03:32.690562","last_in_change":"2021-02-10 11:03:32.690562","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":35,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:03:37.322519","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":34,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:36:44.297293","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"161","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":35,"pool_name":"cephfs_data","create_time":"2021-02-10 11:36:45.321943","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"162","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"be691b4d-a039-408a-af80-9d53f02b6f1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6818","nonce":36088},{"type":"v1","addr":"172.21.2.102:6820","nonce":36088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6822","nonce":36088},{"type":"v1","addr":"172.21.2.102:6824","nonce":36088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6830","nonce":36088},{"type":"v1","addr":"172.21.2.102:6832","nonce":36088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6826","nonce":36088},{"type":"v1","addr":"172.21.2.102:6828","nonce":36088}]},"public_addr":"172.21.2.102:6820/36088","cluster_addr":"172.21.2.102:6824/36088","heartbeat_back_addr":"172.21.2.102:6832/36088","heartbeat_front_addr":"172.21.2.102:6828/36088","state":["exists","up"]},{"osd":1,"uuid":"64e3ccc8-9c69-4d7c-865e-af8e83822635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6819","nonce":36087},{"type":"v1","addr":"172.21.2.102:6821","nonce":36087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6823","nonce":36087},{"type":"v1","addr":"172.21.2.102:6825","nonce":36087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6831","nonce":36087},{"type":"v1","addr":"172.21.2.102:6833","nonce":36087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6827","nonce":36087},{"type":"v1","addr":"172.21.2.102:6829","nonce":36087}]},"public_addr":"172.21.2.102:6821/36087","cluster_addr":"172.21.2.102:6825/36087","heartbeat_back_addr":"172.21.2.102:6833/36087","heartbeat_front_addr":"172.21.2.102:6829/36087","state":["exists","up"]},{"osd":2,"uuid":"153700af-747f-481f-bdf4-d6072d3a842f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6810","nonce":36089},{"type":"v1","addr":"172.21.2.102:6811","nonce":36089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6812","nonce":36089},{"type":"v1","addr":"172.21.2.102:6813","nonce":36089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6816","nonce":36089},{"type":"v1","addr":"172.21.2.102:6817","nonce":36089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6814","nonce":36089},{"type":"v1","addr":"172.21.2.102:6815","nonce":36089}]},"public_addr":"172.21.2.102:6811/36089","cluster_addr":"172.21.2.102:6813/36089","heartbeat_back_addr":"172.21.2.102:6817/36089","heartbeat_front_addr":"172.21.2.102:6815/36089","state":["exists","up"]},{"osd":3,"uuid":"cbc678f4-83a9-45e6-8830-2f7c232b1ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6802","nonce":36090},{"type":"v1","addr":"172.21.2.102:6803","nonce":36090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6804","nonce":36090},{"type":"v1","addr":"172.21.2.102:6805","nonce":36090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6808","nonce":36090},{"type":"v1","addr":"172.21.2.102:6809","nonce":36090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6806","nonce":36090},{"type":"v1","addr":"172.21.2.102:6807","nonce":36090}]},"public_addr":"172.21.2.102:6803/36090","cluster_addr":"172.21.2.102:6805/36090","heartbeat_back_addr":"172.21.2.102:6809/36090","heartbeat_front_addr":"172.21.2.102:6807/36090","state":["exists","up"]},{"osd":4,"uuid":"c01e4008-b258-4f10-afd2-9e7f37214351","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6808","nonce":36368},{"type":"v1","addr":"172.21.2.113:6811","nonce":36368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6814","nonce":36368},{"type":"v1","addr":"172.21.2.113:6818","nonce":36368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6826","nonce":36368},{"type":"v1","addr":"172.21.2.113:6828","nonce":36368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6820","nonce":36368},{"type":"v1","addr":"172.21.2.113:6824","nonce":36368}]},"public_addr":"172.21.2.113:6811/36368","cluster_addr":"172.21.2.113:6818/36368","heartbeat_back_addr":"172.21.2.113:6828/36368","heartbeat_front_addr":"172.21.2.113:6824/36368","state":["exists","up"]},{"osd":5,"uuid":"551f6007-daf8-4a98-8ee0-c2b16bcccb5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6802","nonce":36366},{"type":"v1","addr":"172.21.2.113:6805","nonce":36366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6807","nonce":36366},{"type":"v1","addr":"172.21.2.113:6810","nonce":36366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6819","nonce":36366},{"type":"v1","addr":"172.21.2.113:6822","nonce":36366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6812","nonce":36366},{"type":"v1","addr":"172.21.2.113:6815","nonce":36366}]},"public_addr":"172.21.2.113:6805/36366","cluster_addr":"172.21.2.113:6810/36366","heartbeat_back_addr":"172.21.2.113:6822/36366","heartbeat_front_addr":"172.21.2.113:6815/36366","state":["exists","up"]},{"osd":6,"uuid":"bf67c040-0e50-48be-b102-c3943bc9ebef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6800","nonce":36367},{"type":"v1","addr":"172.21.2.113:6801","nonce":36367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6803","nonce":36367},{"type":"v1","addr":"172.21.2.113:6804","nonce":36367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6813","nonce":36367},{"type":"v1","addr":"172.21.2.113:6816","nonce":36367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6806","nonce":36367},{"type":"v1","addr":"172.21.2.113:6809","nonce":36367}]},"public_addr":"172.21.2.113:6801/36367","cluster_addr":"172.21.2.113:6804/36367","heartbeat_back_addr":"172.21.2.113:6816/36367","heartbeat_front_addr":"172.21.2.113:6809/36367","state":["exists","up"]},{"osd":7,"uuid":"f5461f80-018d-428f-be89-2eb2bcbe7bb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6817","nonce":36365},{"type":"v1","addr":"172.21.2.113:6821","nonce":36365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6823","nonce":36365},{"type":"v1","addr":"172.21.2.113:6825","nonce":36365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6830","nonce":36365},{"type":"v1","addr":"172.21.2.113:6831","nonce":36365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6827","nonce":36365},{"type":"v1","addr":"172.21.2.113:6829","nonce":36365}]},"public_addr":"172.21.2.113:6821/36365","cluster_addr":"172.21.2.113:6825/36365","heartbeat_back_addr":"172.21.2.113:6831/36365","heartbeat_front_addr":"172.21.2.113:6829/36365","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:36:49.585 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:49.583 7fc912944700 1 -- 172.21.2.102:0/1816194183 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc8ec0186c0 msgr2=0x7fc8ec01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:49.585 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:49.583 7fc912944700 1 --2- 172.21.2.102:0/1816194183 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc8ec0186c0 0x7fc8ec01abc0 crc :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:49.586 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:49.583 7fc912944700 1 -- 172.21.2.102:0/1816194183 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc90c18a540 msgr2=0x7fc90c197480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:49.586 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:49.583 7fc912944700 1 --2- 172.21.2.102:0/1816194183 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc90c18a540 0x7fc90c197480 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 rx=0x7fc8fc00b550 tx=0x7fc8fc00b8f0).stop 2021-02-10T11:36:49.586 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:49.584 7fc912944700 1 -- 172.21.2.102:0/1816194183 shutdown_connections 2021-02-10T11:36:49.586 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:49.584 7fc912944700 1 --2- 172.21.2.102:0/1816194183 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc90c0b7df0 0x7fc90c0b81f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:49.587 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:49.584 7fc912944700 1 --2- 172.21.2.102:0/1816194183 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc90c18aa50 0x7fc90c0b77d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:49.587 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:49.584 7fc912944700 1 --2- 172.21.2.102:0/1816194183 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc8ec0186c0 0x7fc8ec01abc0 unknown :-1 s=CLOSED pgs=1262 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:49.587 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:49.584 7fc912944700 1 --2- 172.21.2.102:0/1816194183 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc90c18a540 0x7fc90c197480 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:49.587 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:49.584 7fc912944700 1 -- 172.21.2.102:0/1816194183 >> 172.21.2.102:0/1816194183 conn(0x7fc90c0efc70 msgr2=0x7fc90c0ef410 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:49.587 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:49.584 7fc912944700 1 -- 172.21.2.102:0/1816194183 shutdown_connections 2021-02-10T11:36:49.587 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:49.584 7fc912944700 1 -- 172.21.2.102:0/1816194183 wait complete. 2021-02-10T11:36:49.595 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:36:50.038 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:50.018 7f2d7e526700 1 Processor -- start 2021-02-10T11:36:50.038 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:50.018 7f2d7e526700 1 -- start start 2021-02-10T11:36:50.038 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:50.019 7f2d7e526700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f2d780d1880 0x7f2d781b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:50.038 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:50.019 7f2d7e526700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d780fe1d0 con 0x7f2d780cff80 2021-02-10T11:36:50.038 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:50.019 7f2d7e526700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d78109a70 con 0x7f2d781b0af0 2021-02-10T11:36:50.038 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:50.019 7f2d7e526700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d78088070 con 0x7f2d780d1880 2021-02-10T11:36:50.038 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:50.019 7f2d7cb43700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f2d780cff80 0x7f2d78109140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:58612/0 (socket says 172.21.2.102:58612) 2021-02-10T11:36:50.038 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:50.019 7f2d7cb43700 1 -- 172.21.2.102:0/3517719800 learned_addr learned my addr 172.21.2.102:0/3517719800 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:36:50.039 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:50.019 7f2d76ffd700 1 -- 172.21.2.102:0/3517719800 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3889039355 0 0) 0x7f2d70001500 con 0x7f2d780cff80 2021-02-10T11:36:50.039 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:50.019 7f2d76ffd700 1 -- 172.21.2.102:0/3517719800 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d60002fe0 con 0x7f2d780cff80 2021-02-10T11:36:50.039 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:50.020 7f2d77fff700 1 --2- 172.21.2.102:0/3517719800 >> v2:172.21.2.113:3300/0 conn(0x7f2d780d1880 0x7f2d781b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:50.039 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:50.020 7f2d76ffd700 1 -- 172.21.2.102:0/3517719800 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 569503364 0 0) 0x7f2d70001500 con 0x7f2d780cff80 2021-02-10T11:36:50.039 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:50.020 7f2d76ffd700 1 -- 172.21.2.102:0/3517719800 >> v2:172.21.2.113:3300/0 conn(0x7f2d780d1880 msgr2=0x7f2d781b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:50.039 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:50.020 7f2d76ffd700 1 --2- 172.21.2.102:0/3517719800 >> v2:172.21.2.113:3300/0 conn(0x7f2d780d1880 0x7f2d781b04d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:50.039 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:50.020 7f2d76ffd700 1 -- 172.21.2.102:0/3517719800 >> v1:172.21.2.113:6789/0 conn(0x7f2d781b0af0 legacy=0x7f2d78108950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:50.039 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:50.020 7f2d76ffd700 1 -- 172.21.2.102:0/3517719800 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d78088260 con 0x7f2d780cff80 2021-02-10T11:36:50.039 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:50.020 7f2d76ffd700 1 -- 172.21.2.102:0/3517719800 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7f2d70001a00 con 0x7f2d780cff80 2021-02-10T11:36:50.039 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:50.020 7f2d76ffd700 1 -- 172.21.2.102:0/3517719800 >> v1:172.21.2.102:6789/0 conn(0x7f2d780cff80 legacy=0x7f2d78109140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:50.039 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:50.020 7f2d76ffd700 1 --2- 172.21.2.102:0/3517719800 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2d60003b20 0x7f2d60006000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:50.039 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:50.020 7f2d76ffd700 1 --2- 172.21.2.102:0/3517719800 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2d60006820 0x7f2d60008d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:50.039 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:50.020 7f2d76ffd700 1 --2- 172.21.2.102:0/3517719800 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2d60009620 0x7f2d6000bb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:50.040 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:50.020 7f2d76ffd700 1 -- 172.21.2.102:0/3517719800 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d6000c390 con 0x7f2d60006820 2021-02-10T11:36:50.040 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:50.020 7f2d76ffd700 1 -- 172.21.2.102:0/3517719800 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d6000c5b0 con 0x7f2d60009620 2021-02-10T11:36:50.040 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:50.020 7f2d76ffd700 1 -- 172.21.2.102:0/3517719800 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d6000c7d0 con 0x7f2d60003b20 2021-02-10T11:36:50.040 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:50.020 7f2d77fff700 1 --2- 172.21.2.102:0/3517719800 >> v2:172.21.2.113:3300/0 conn(0x7f2d780d1880 0x7f2d781b04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:36:50.040 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:50.020 7f2d777fe700 1 --2- 172.21.2.102:0/3517719800 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2d60006820 0x7f2d60008d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:50.040 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:50.020 7f2d777fe700 1 -- 172.21.2.102:0/3517719800 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2d60003b20 msgr2=0x7f2d60006000 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:36:50.040 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:50.020 7f2d777fe700 1 --2- 172.21.2.102:0/3517719800 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2d60003b20 0x7f2d60006000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:50.040 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:50.020 7f2d777fe700 1 -- 172.21.2.102:0/3517719800 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2d60009620 msgr2=0x7f2d6000bb40 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:36:50.040 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:50.020 7f2d777fe700 1 --2- 172.21.2.102:0/3517719800 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2d60009620 0x7f2d6000bb40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:50.040 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:50.020 7f2d777fe700 1 -- 172.21.2.102:0/3517719800 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d6000ca70 con 0x7f2d60006820 2021-02-10T11:36:50.040 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:50.020 7f2d7cb43700 1 -- 172.21.2.102:0/3517719800 reap_dead start 2021-02-10T11:36:50.040 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:50.020 7f2d777fe700 1 --2- 172.21.2.102:0/3517719800 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2d60006820 0x7f2d60008d40 secure :-1 s=READY pgs=2876 cs=0 l=1 rev1=1 rx=0x7f2d6c006ac0 tx=0x7f2d6c009af0).ready entity=mon.0 client_cookie=a08a388322e210bb server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:50.040 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:50.021 7f2d76ffd700 1 -- 172.21.2.102:0/3517719800 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2d6c007280 con 0x7f2d60006820 2021-02-10T11:36:50.041 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:50.021 7f2d76ffd700 1 -- 172.21.2.102:0/3517719800 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d6c007540 con 0x7f2d60006820 2021-02-10T11:36:50.041 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:50.021 7f2d76ffd700 1 -- 172.21.2.102:0/3517719800 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2d6c0079a0 con 0x7f2d60006820 2021-02-10T11:36:50.041 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:50.021 7f2d7e526700 1 -- 172.21.2.102:0/3517719800 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2d60006820 msgr2=0x7f2d60008d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:50.041 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:50.021 7f2d7e526700 1 --2- 172.21.2.102:0/3517719800 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2d60006820 0x7f2d60008d40 secure :-1 s=READY pgs=2876 cs=0 l=1 rev1=1 rx=0x7f2d6c006ac0 tx=0x7f2d6c009af0).stop 2021-02-10T11:36:50.041 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:50.021 7f2d7e526700 1 -- 172.21.2.102:0/3517719800 shutdown_connections 2021-02-10T11:36:50.041 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:50.021 7f2d7e526700 1 --2- 172.21.2.102:0/3517719800 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2d60006820 0x7f2d60008d40 unknown :-1 s=CLOSED pgs=2876 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:50.041 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:50.021 7f2d7e526700 1 -- 172.21.2.102:0/3517719800 >> 172.21.2.102:0/3517719800 conn(0x7f2d780ce430 msgr2=0x7f2d780d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:50.041 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:50.021 7f2d7e526700 1 -- 172.21.2.102:0/3517719800 shutdown_connections 2021-02-10T11:36:50.041 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:50.021 7f2d7e526700 1 -- 172.21.2.102:0/3517719800 wait complete. 2021-02-10T11:36:50.041 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:50.036 7f2d7e526700 1 Processor -- start 2021-02-10T11:36:50.041 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:50.036 7f2d7e526700 1 -- start start 2021-02-10T11:36:50.043 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:50.036 7f2d7e526700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2d781a0580 0x7f2d78086fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:50.043 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:50.036 7f2d7e526700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2d780b66b0 0x7f2d780b6ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:50.043 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:50.036 7f2d7e526700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2d78082bb0 0x7f2d78082fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:50.043 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:50.036 7f2d7e526700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d780875d0 con 0x7f2d78082bb0 2021-02-10T11:36:50.043 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:50.036 7f2d7e526700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d780835d0 con 0x7f2d780b66b0 2021-02-10T11:36:50.043 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:50.036 7f2d7e526700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d780837f0 con 0x7f2d781a0580 2021-02-10T11:36:50.043 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:50.037 7f2d7cb43700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2d78082bb0 0x7f2d78082fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:50.043 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:50.037 7f2d7cb43700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2d78082bb0 0x7f2d78082fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:38170/0 (socket says 172.21.2.102:38170) 2021-02-10T11:36:50.043 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:50.037 7f2d7cb43700 1 -- 172.21.2.102:0/3348333532 learned_addr learned my addr 172.21.2.102:0/3348333532 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:36:50.044 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:50.037 7f2d7cb43700 1 -- 172.21.2.102:0/3348333532 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2d781a0580 msgr2=0x7f2d78086fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:50.044 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:50.037 7f2d7cb43700 1 --2- 172.21.2.102:0/3348333532 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2d781a0580 0x7f2d78086fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:50.044 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:50.037 7f2d7cb43700 1 -- 172.21.2.102:0/3348333532 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2d780b66b0 msgr2=0x7f2d780b6ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:50.044 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:50.037 7f2d7cb43700 1 --2- 172.21.2.102:0/3348333532 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2d780b66b0 0x7f2d780b6ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:50.044 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:50.037 7f2d7cb43700 1 -- 172.21.2.102:0/3348333532 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d78083a10 con 0x7f2d78082bb0 2021-02-10T11:36:50.044 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:50.037 7f2d7cb43700 1 --2- 172.21.2.102:0/3348333532 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2d78082bb0 0x7f2d78082fb0 secure :-1 s=READY pgs=2877 cs=0 l=1 rev1=1 rx=0x7f2d7000a900 tx=0x7f2d7000aca0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:50.044 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:50.037 7f2d74ff9700 1 -- 172.21.2.102:0/3348333532 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2d700070d0 con 0x7f2d78082bb0 2021-02-10T11:36:50.044 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:50.037 7f2d74ff9700 1 -- 172.21.2.102:0/3348333532 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d700073d0 con 0x7f2d78082bb0 2021-02-10T11:36:50.044 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:50.037 7f2d7e526700 1 -- 172.21.2.102:0/3348333532 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d78083a10 con 0x7f2d78082bb0 2021-02-10T11:36:50.044 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:50.037 7f2d7e526700 1 -- 172.21.2.102:0/3348333532 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d78083a10 con 0x7f2d78082bb0 2021-02-10T11:36:50.044 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:50.037 7f2d74ff9700 1 -- 172.21.2.102:0/3348333532 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2d70004bb0 con 0x7f2d78082bb0 2021-02-10T11:36:50.044 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:50.038 7f2d74ff9700 1 -- 172.21.2.102:0/3348333532 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f2d70000da0 con 0x7f2d78082bb0 2021-02-10T11:36:50.044 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:50.038 7f2d74ff9700 1 --2- 172.21.2.102:0/3348333532 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2d58018730 0x7f2d5801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:50.045 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:50.038 7f2d74ff9700 1 -- 172.21.2.102:0/3348333532 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(162..162 src has 1..162) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f2d70032b40 con 0x7f2d78082bb0 2021-02-10T11:36:50.045 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:50.038 7f2d77fff700 1 --2- 172.21.2.102:0/3348333532 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2d58018730 0x7f2d5801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:50.045 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:50.038 7f2d77fff700 1 --2- 172.21.2.102:0/3348333532 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2d58018730 0x7f2d5801ac10 crc :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:50.071 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:50.069 7f2d7e526700 1 -- 172.21.2.102:0/3348333532 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d50002fd0 con 0x7f2d78082bb0 2021-02-10T11:36:50.073 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:50.071 7f2d74ff9700 1 -- 172.21.2.102:0/3348333532 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f2d700072c0 con 0x7f2d78082bb0 2021-02-10T11:36:50.202 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:50.200 7f2d7e526700 1 -- 172.21.2.102:0/3348333532 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f2d50002ef0 con 0x7f2d78082bb0 2021-02-10T11:36:50.205 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:50.204 7f2d74ff9700 1 -- 172.21.2.102:0/3348333532 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.0 v89) v1 ==== 184+0+0 (secure 0 0 0) 0x7f2d70000da0 con 0x7f2d78082bb0 2021-02-10T11:36:50.209 INFO:teuthology.orchestra.run.gibba002.stderr:updated caps for client.0 2021-02-10T11:36:50.211 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:50.209 7f2d7e526700 1 -- 172.21.2.102:0/3348333532 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2d58018730 msgr2=0x7f2d5801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:50.211 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:50.209 7f2d7e526700 1 --2- 172.21.2.102:0/3348333532 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2d58018730 0x7f2d5801ac10 crc :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:50.211 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:50.209 7f2d7e526700 1 -- 172.21.2.102:0/3348333532 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2d78082bb0 msgr2=0x7f2d78082fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:50.211 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:50.209 7f2d7e526700 1 --2- 172.21.2.102:0/3348333532 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2d78082bb0 0x7f2d78082fb0 secure :-1 s=READY pgs=2877 cs=0 l=1 rev1=1 rx=0x7f2d7000a900 tx=0x7f2d7000aca0).stop 2021-02-10T11:36:50.211 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:50.209 7f2d7e526700 1 -- 172.21.2.102:0/3348333532 shutdown_connections 2021-02-10T11:36:50.211 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:50.209 7f2d7e526700 1 --2- 172.21.2.102:0/3348333532 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2d78082bb0 0x7f2d78082fb0 unknown :-1 s=CLOSED pgs=2877 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:50.211 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:50.209 7f2d7e526700 1 --2- 172.21.2.102:0/3348333532 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2d58018730 0x7f2d5801ac10 unknown :-1 s=CLOSED pgs=1263 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:50.211 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:50.209 7f2d7e526700 1 --2- 172.21.2.102:0/3348333532 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2d780b66b0 0x7f2d780b6ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:50.211 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:50.209 7f2d7e526700 1 --2- 172.21.2.102:0/3348333532 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2d781a0580 0x7f2d78086fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:50.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:50.209 7f2d7e526700 1 -- 172.21.2.102:0/3348333532 >> 172.21.2.102:0/3348333532 conn(0x7f2d780afc60 msgr2=0x7f2d780b50d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:50.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:50.209 7f2d7e526700 1 -- 172.21.2.102:0/3348333532 shutdown_connections 2021-02-10T11:36:50.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:50.209 7f2d7e526700 1 -- 172.21.2.102:0/3348333532 wait complete. 2021-02-10T11:36:50.220 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.1 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:36:50.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:50.644 7f694f661700 1 Processor -- start 2021-02-10T11:36:50.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:50.644 7f694f661700 1 -- start start 2021-02-10T11:36:50.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:50.644 7f694f661700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f69480d1880 0x7f69481b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:50.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:50.644 7f694f661700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f69481b0af0 0x7f6948108950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:50.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:50.644 7f694f661700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6948109a70 con 0x7f69480d1c80 2021-02-10T11:36:50.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:50.644 7f694f661700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69480e5f00 con 0x7f69481b0af0 2021-02-10T11:36:50.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:50.644 7f694f661700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6948109760 con 0x7f69480d1880 2021-02-10T11:36:50.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:50.645 7f694d47d700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f69480d1880 0x7f69481b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:50.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:50.645 7f694cc7c700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f69481b0af0 0x7f6948108950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:50.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:50.645 7f694dc7e700 1 --1- >> v1:172.21.2.113:6789/0 conn(0x7f69480d1c80 0x7f6948108f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.113:6789/0 says I am v1:172.21.2.102:37726/0 (socket says 172.21.2.102:37726) 2021-02-10T11:36:50.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:50.645 7f694dc7e700 1 -- 172.21.2.102:0/2243763921 learned_addr learned my addr 172.21.2.102:0/2243763921 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:36:50.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:50.646 7f694cc7c700 1 -- 172.21.2.102:0/2243763921 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f69480d1880 msgr2=0x7f69481b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:50.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:50.646 7f694cc7c700 1 --2- 172.21.2.102:0/2243763921 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f69480d1880 0x7f69481b04d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:50.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:50.646 7f694cc7c700 1 -- 172.21.2.102:0/2243763921 >> v1:172.21.2.113:6789/0 conn(0x7f69480d1c80 legacy=0x7f6948108f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:50.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:50.646 7f694cc7c700 1 -- 172.21.2.102:0/2243763921 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6948088070 con 0x7f69481b0af0 2021-02-10T11:36:50.666 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:50.646 7f694d47d700 1 --2- 172.21.2.102:0/2243763921 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f69480d1880 0x7f69481b04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:36:50.666 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:50.646 7f694cc7c700 1 --2- 172.21.2.102:0/2243763921 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f69481b0af0 0x7f6948108950 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 rx=0x7f693c006650 tx=0x7f693c009a80).ready entity=mon.1 client_cookie=89a42a864b393576 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:50.666 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:50.647 7f693bfff700 1 -- 172.21.2.102:0/2243763921 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f693c00c0c0 con 0x7f69481b0af0 2021-02-10T11:36:50.666 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:50.647 7f693bfff700 1 -- 172.21.2.102:0/2243763921 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f693c00c340 con 0x7f69481b0af0 2021-02-10T11:36:50.666 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:50.647 7f693bfff700 1 -- 172.21.2.102:0/2243763921 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f693c00c860 con 0x7f69481b0af0 2021-02-10T11:36:50.666 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:50.647 7f694f661700 1 -- 172.21.2.102:0/2243763921 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f69481b0af0 msgr2=0x7f6948108950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:50.666 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:50.647 7f694f661700 1 --2- 172.21.2.102:0/2243763921 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f69481b0af0 0x7f6948108950 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 rx=0x7f693c006650 tx=0x7f693c009a80).stop 2021-02-10T11:36:50.666 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:50.647 7f694f661700 1 -- 172.21.2.102:0/2243763921 shutdown_connections 2021-02-10T11:36:50.666 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:50.647 7f694f661700 1 --2- 172.21.2.102:0/2243763921 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f69481b0af0 0x7f6948108950 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:50.666 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:50.647 7f694f661700 1 --2- 172.21.2.102:0/2243763921 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f69480d1880 0x7f69481b04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:50.666 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:50.647 7f694f661700 1 -- 172.21.2.102:0/2243763921 >> 172.21.2.102:0/2243763921 conn(0x7f69480ce430 msgr2=0x7f69480d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:50.666 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:50.648 7f694f661700 1 -- 172.21.2.102:0/2243763921 shutdown_connections 2021-02-10T11:36:50.666 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:50.648 7f694f661700 1 -- 172.21.2.102:0/2243763921 wait complete. 2021-02-10T11:36:50.667 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:50.662 7f694f661700 1 Processor -- start 2021-02-10T11:36:50.667 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:50.662 7f694f661700 1 -- start start 2021-02-10T11:36:50.668 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:50.663 7f694f661700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f69481b9280 0x7f694818f750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:50.668 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:50.663 7f694f661700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f69481b9680 0x7f694818fd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:50.668 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:50.663 7f694f661700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f69481acd60 0x7f69481ad160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:50.668 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:50.663 7f694f661700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6948190390 con 0x7f69481b9280 2021-02-10T11:36:50.668 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:50.663 7f694f661700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69481907c0 con 0x7f69481acd60 2021-02-10T11:36:50.668 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:50.663 7f694f661700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69481ac3d0 con 0x7f69481b9680 2021-02-10T11:36:50.668 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:50.663 7f694d47d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f69481b9280 0x7f694818f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:50.668 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:50.663 7f694d47d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f69481b9280 0x7f694818f750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:38180/0 (socket says 172.21.2.102:38180) 2021-02-10T11:36:50.668 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:50.663 7f694d47d700 1 -- 172.21.2.102:0/2030629263 learned_addr learned my addr 172.21.2.102:0/2030629263 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:36:50.669 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:50.663 7f694dc7e700 1 --2- 172.21.2.102:0/2030629263 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f69481acd60 0x7f69481ad160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:50.669 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:50.663 7f694cc7c700 1 --2- 172.21.2.102:0/2030629263 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f69481b9680 0x7f694818fd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:50.669 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:50.663 7f694d47d700 1 -- 172.21.2.102:0/2030629263 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f69481b9680 msgr2=0x7f694818fd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:50.669 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:50.663 7f694d47d700 1 --2- 172.21.2.102:0/2030629263 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f69481b9680 0x7f694818fd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:50.669 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:50.663 7f694d47d700 1 -- 172.21.2.102:0/2030629263 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f69481acd60 msgr2=0x7f69481ad160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:50.669 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:50.663 7f694d47d700 1 --2- 172.21.2.102:0/2030629263 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f69481acd60 0x7f69481ad160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:50.669 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:50.663 7f694d47d700 1 -- 172.21.2.102:0/2030629263 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69481ad780 con 0x7f69481b9280 2021-02-10T11:36:50.669 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:50.663 7f694cc7c700 1 --2- 172.21.2.102:0/2030629263 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f69481b9680 0x7f694818fd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:36:50.669 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:50.663 7f694d47d700 1 --2- 172.21.2.102:0/2030629263 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f69481b9280 0x7f694818f750 secure :-1 s=READY pgs=2878 cs=0 l=1 rev1=1 rx=0x7f6944006f40 tx=0x7f694400b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:50.669 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:50.664 7f6939ffb700 1 -- 172.21.2.102:0/2030629263 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f694400bd80 con 0x7f69481b9280 2021-02-10T11:36:50.669 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:50.664 7f6939ffb700 1 -- 172.21.2.102:0/2030629263 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6944000c90 con 0x7f69481b9280 2021-02-10T11:36:50.669 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:50.664 7f6939ffb700 1 -- 172.21.2.102:0/2030629263 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6944002220 con 0x7f69481b9280 2021-02-10T11:36:50.669 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:50.664 7f694f661700 1 -- 172.21.2.102:0/2030629263 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69481ad780 con 0x7f69481b9280 2021-02-10T11:36:50.670 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:50.664 7f694f661700 1 -- 172.21.2.102:0/2030629263 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69481ad780 con 0x7f69481b9280 2021-02-10T11:36:50.670 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:50.664 7f6939ffb700 1 -- 172.21.2.102:0/2030629263 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f694400a000 con 0x7f69481b9280 2021-02-10T11:36:50.670 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:50.664 7f6939ffb700 1 --2- 172.21.2.102:0/2030629263 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f69280186c0 0x7f692801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:50.670 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:50.664 7f6939ffb700 1 -- 172.21.2.102:0/2030629263 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(162..162 src has 1..162) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f6944032560 con 0x7f69481b9280 2021-02-10T11:36:50.670 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:50.665 7f694cc7c700 1 --2- 172.21.2.102:0/2030629263 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f69280186c0 0x7f692801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:50.670 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:50.665 7f694cc7c700 1 --2- 172.21.2.102:0/2030629263 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f69280186c0 0x7f692801abc0 crc :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:50.696 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:50.694 7f694f661700 1 -- 172.21.2.102:0/2030629263 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6948190390 con 0x7f69481b9280 2021-02-10T11:36:50.698 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:50.697 7f6939ffb700 1 -- 172.21.2.102:0/2030629263 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f6944006b00 con 0x7f69481b9280 2021-02-10T11:36:50.830 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:50.828 7f694f661700 1 -- 172.21.2.102:0/2030629263 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f6948190390 con 0x7f69481b9280 2021-02-10T11:36:50.834 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:50.832 7f6939ffb700 1 -- 172.21.2.102:0/2030629263 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.1 v90) v1 ==== 184+0+0 (secure 0 0 0) 0x7f694401f4d0 con 0x7f69481b9280 2021-02-10T11:36:50.837 INFO:teuthology.orchestra.run.gibba002.stderr:updated caps for client.1 2021-02-10T11:36:50.839 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:50.837 7f694f661700 1 -- 172.21.2.102:0/2030629263 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f69280186c0 msgr2=0x7f692801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:50.839 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:50.837 7f694f661700 1 --2- 172.21.2.102:0/2030629263 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f69280186c0 0x7f692801abc0 crc :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:50.839 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:50.837 7f694f661700 1 -- 172.21.2.102:0/2030629263 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f69481b9280 msgr2=0x7f694818f750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:50.839 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:50.837 7f694f661700 1 --2- 172.21.2.102:0/2030629263 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f69481b9280 0x7f694818f750 secure :-1 s=READY pgs=2878 cs=0 l=1 rev1=1 rx=0x7f6944006f40 tx=0x7f694400b990).stop 2021-02-10T11:36:50.840 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:50.837 7f694f661700 1 -- 172.21.2.102:0/2030629263 shutdown_connections 2021-02-10T11:36:50.840 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:50.837 7f694f661700 1 --2- 172.21.2.102:0/2030629263 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f69280186c0 0x7f692801abc0 unknown :-1 s=CLOSED pgs=1264 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:50.840 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:50.837 7f694f661700 1 --2- 172.21.2.102:0/2030629263 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f69481acd60 0x7f69481ad160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:50.840 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:50.837 7f694f661700 1 --2- 172.21.2.102:0/2030629263 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f69481b9680 0x7f694818fd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:50.840 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:50.837 7f694f661700 1 --2- 172.21.2.102:0/2030629263 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f69481b9280 0x7f694818f750 unknown :-1 s=CLOSED pgs=2878 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:50.840 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:50.837 7f694f661700 1 -- 172.21.2.102:0/2030629263 >> 172.21.2.102:0/2030629263 conn(0x7f69480845b0 msgr2=0x7f694818c290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:50.840 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:50.837 7f694f661700 1 -- 172.21.2.102:0/2030629263 shutdown_connections 2021-02-10T11:36:50.840 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:50.837 7f694f661700 1 -- 172.21.2.102:0/2030629263 wait complete. 2021-02-10T11:36:50.847 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.2 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:36:51.282 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:51.262 7f7f28350700 1 Processor -- start 2021-02-10T11:36:51.282 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:51.262 7f7f28350700 1 -- start start 2021-02-10T11:36:51.282 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:51.262 7f7f28350700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7f2010afe0 0x7f7f200cf740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:51.282 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:51.262 7f7f28350700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f200fe1b0 con 0x7f7f200d1860 2021-02-10T11:36:51.282 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:51.262 7f7f28350700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f201b0ca0 con 0x7f7f200d1c60 2021-02-10T11:36:51.282 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:51.262 7f7f28350700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f20109a50 con 0x7f7f2010afe0 2021-02-10T11:36:51.282 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:51.262 7f7f2616c700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f7f200d1860 0x7f7f201b04b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:58644/0 (socket says 172.21.2.102:58644) 2021-02-10T11:36:51.283 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:51.262 7f7f2616c700 1 -- 172.21.2.102:0/828697088 learned_addr learned my addr 172.21.2.102:0/828697088 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:36:51.283 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:51.263 7f7f2516a700 1 -- 172.21.2.102:0/828697088 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2321592409 0 0) 0x7f7f1c0010b0 con 0x7f7f200d1860 2021-02-10T11:36:51.283 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:51.263 7f7f2516a700 1 -- 172.21.2.102:0/828697088 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f0c002fe0 con 0x7f7f200d1860 2021-02-10T11:36:51.283 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:51.263 7f7f2696d700 1 --2- 172.21.2.102:0/828697088 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7f2010afe0 0x7f7f200cf740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:51.283 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:51.263 7f7f2516a700 1 -- 172.21.2.102:0/828697088 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2735709335 0 0) 0x7f7f1c0010b0 con 0x7f7f200d1860 2021-02-10T11:36:51.283 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:51.263 7f7f2516a700 1 -- 172.21.2.102:0/828697088 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7f2010afe0 msgr2=0x7f7f200cf740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:51.283 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:51.263 7f7f2516a700 1 --2- 172.21.2.102:0/828697088 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7f2010afe0 0x7f7f200cf740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:51.283 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:51.263 7f7f2516a700 1 -- 172.21.2.102:0/828697088 >> v1:172.21.2.113:6789/0 conn(0x7f7f200d1c60 legacy=0x7f7f200cef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:51.283 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:51.263 7f7f2516a700 1 -- 172.21.2.102:0/828697088 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f2010b3e0 con 0x7f7f200d1860 2021-02-10T11:36:51.283 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:51.264 7f7f2516a700 1 -- 172.21.2.102:0/828697088 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7f7f1c0015b0 con 0x7f7f200d1860 2021-02-10T11:36:51.283 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:51.264 7f7f2516a700 1 -- 172.21.2.102:0/828697088 >> v1:172.21.2.102:6789/0 conn(0x7f7f200d1860 legacy=0x7f7f201b04b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:51.283 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:51.264 7f7f2516a700 1 --2- 172.21.2.102:0/828697088 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7f0c003a60 0x7f7f0c005eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:51.283 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:51.264 7f7f2516a700 1 --2- 172.21.2.102:0/828697088 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7f0c006750 0x7f7f0c008c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:51.284 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:51.264 7f7f2516a700 1 --2- 172.21.2.102:0/828697088 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7f0c0094f0 0x7f7f0c00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:51.284 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:51.264 7f7f2516a700 1 -- 172.21.2.102:0/828697088 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f0c00c2c0 con 0x7f7f0c0094f0 2021-02-10T11:36:51.284 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:51.264 7f7f2516a700 1 -- 172.21.2.102:0/828697088 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f0c00c4e0 con 0x7f7f0c006750 2021-02-10T11:36:51.284 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:51.264 7f7f2516a700 1 -- 172.21.2.102:0/828697088 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f0c00c700 con 0x7f7f0c003a60 2021-02-10T11:36:51.284 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:51.264 7f7f2696d700 1 --2- 172.21.2.102:0/828697088 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7f0c0094f0 0x7f7f0c00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:51.284 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:51.264 7f7f2616c700 1 --2- 172.21.2.102:0/828697088 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7f0c003a60 0x7f7f0c005eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:51.284 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:51.264 7f7f2596b700 1 --2- 172.21.2.102:0/828697088 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7f0c006750 0x7f7f0c008c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:51.284 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:51.264 7f7f2696d700 1 -- 172.21.2.102:0/828697088 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7f0c003a60 msgr2=0x7f7f0c005eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:51.284 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:51.264 7f7f2696d700 1 --2- 172.21.2.102:0/828697088 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7f0c003a60 0x7f7f0c005eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:51.284 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:51.264 7f7f2696d700 1 -- 172.21.2.102:0/828697088 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7f0c006750 msgr2=0x7f7f0c008c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:51.284 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:51.264 7f7f2696d700 1 --2- 172.21.2.102:0/828697088 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7f0c006750 0x7f7f0c008c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:51.284 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:51.264 7f7f2696d700 1 -- 172.21.2.102:0/828697088 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f0c050940 con 0x7f7f0c0094f0 2021-02-10T11:36:51.284 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:51.264 7f7f2696d700 1 --2- 172.21.2.102:0/828697088 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7f0c0094f0 0x7f7f0c00ba10 secure :-1 s=READY pgs=2880 cs=0 l=1 rev1=1 rx=0x7f7f18009b10 tx=0x7f7f18009eb0).ready entity=mon.0 client_cookie=dbe93356e6b1ce36 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:51.285 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:51.264 7f7f2516a700 1 -- 172.21.2.102:0/828697088 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7f1800b510 con 0x7f7f0c0094f0 2021-02-10T11:36:51.285 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:51.264 7f7f2516a700 1 -- 172.21.2.102:0/828697088 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f18003030 con 0x7f7f0c0094f0 2021-02-10T11:36:51.285 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:51.265 7f7f2516a700 1 -- 172.21.2.102:0/828697088 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7f1800b790 con 0x7f7f0c0094f0 2021-02-10T11:36:51.285 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:51.265 7f7f28350700 1 -- 172.21.2.102:0/828697088 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7f0c0094f0 msgr2=0x7f7f0c00ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:51.285 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:51.265 7f7f28350700 1 --2- 172.21.2.102:0/828697088 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7f0c0094f0 0x7f7f0c00ba10 secure :-1 s=READY pgs=2880 cs=0 l=1 rev1=1 rx=0x7f7f18009b10 tx=0x7f7f18009eb0).stop 2021-02-10T11:36:51.285 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:51.265 7f7f2696d700 1 -- 172.21.2.102:0/828697088 reap_dead start 2021-02-10T11:36:51.285 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:51.265 7f7f28350700 1 -- 172.21.2.102:0/828697088 shutdown_connections 2021-02-10T11:36:51.285 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:51.265 7f7f28350700 1 -- 172.21.2.102:0/828697088 >> 172.21.2.102:0/828697088 conn(0x7f7f200ce410 msgr2=0x7f7f200d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:51.285 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:51.265 7f7f28350700 1 -- 172.21.2.102:0/828697088 shutdown_connections 2021-02-10T11:36:51.285 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:51.265 7f7f28350700 1 -- 172.21.2.102:0/828697088 wait complete. 2021-02-10T11:36:51.285 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:51.280 7f7f28350700 1 Processor -- start 2021-02-10T11:36:51.285 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:51.280 7f7f28350700 1 -- start start 2021-02-10T11:36:51.287 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:51.280 7f7f28350700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7f200bcee0 0x7f7f200b5de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:51.287 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:51.280 7f7f28350700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7f2009bce0 0x7f7f201a7620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:51.287 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:51.280 7f7f28350700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7f201a7c40 0x7f7f2009c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:51.287 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:51.280 7f7f28350700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f200b6400 con 0x7f7f201a7c40 2021-02-10T11:36:51.287 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:51.280 7f7f28350700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f2009c0e0 con 0x7f7f200bcee0 2021-02-10T11:36:51.287 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:51.280 7f7f28350700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f2009c320 con 0x7f7f2009bce0 2021-02-10T11:36:51.287 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:51.281 7f7f2696d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7f201a7c40 0x7f7f2009c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:51.287 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:51.281 7f7f2696d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7f201a7c40 0x7f7f2009c8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:38202/0 (socket says 172.21.2.102:38202) 2021-02-10T11:36:51.287 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:51.281 7f7f2696d700 1 -- 172.21.2.102:0/1292465811 learned_addr learned my addr 172.21.2.102:0/1292465811 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:36:51.287 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:51.281 7f7f2696d700 1 -- 172.21.2.102:0/1292465811 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7f2009bce0 msgr2=0x7f7f201a7620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:51.287 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:51.281 7f7f2696d700 1 --2- 172.21.2.102:0/1292465811 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7f2009bce0 0x7f7f201a7620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:51.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:51.281 7f7f2696d700 1 -- 172.21.2.102:0/1292465811 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7f200bcee0 msgr2=0x7f7f200b5de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:51.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:51.281 7f7f2696d700 1 --2- 172.21.2.102:0/1292465811 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7f200bcee0 0x7f7f200b5de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:51.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:51.281 7f7f2696d700 1 -- 172.21.2.102:0/1292465811 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f200bd2e0 con 0x7f7f201a7c40 2021-02-10T11:36:51.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:51.281 7f7f2696d700 1 --2- 172.21.2.102:0/1292465811 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7f201a7c40 0x7f7f2009c8a0 secure :-1 s=READY pgs=2881 cs=0 l=1 rev1=1 rx=0x7f7f18006f90 tx=0x7f7f1800bc90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:51.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:51.281 7f7f16ffd700 1 -- 172.21.2.102:0/1292465811 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7f18000df0 con 0x7f7f201a7c40 2021-02-10T11:36:51.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:51.281 7f7f16ffd700 1 -- 172.21.2.102:0/1292465811 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f180068d0 con 0x7f7f201a7c40 2021-02-10T11:36:51.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:51.281 7f7f28350700 1 -- 172.21.2.102:0/1292465811 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f200bd2e0 con 0x7f7f201a7c40 2021-02-10T11:36:51.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:51.281 7f7f28350700 1 -- 172.21.2.102:0/1292465811 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f201a8330 con 0x7f7f201a7c40 2021-02-10T11:36:51.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:51.281 7f7f16ffd700 1 -- 172.21.2.102:0/1292465811 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7f1800c570 con 0x7f7f201a7c40 2021-02-10T11:36:51.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:51.282 7f7f16ffd700 1 -- 172.21.2.102:0/1292465811 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f7f18000df0 con 0x7f7f201a7c40 2021-02-10T11:36:51.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:51.282 7f7f16ffd700 1 --2- 172.21.2.102:0/1292465811 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7f080186d0 0x7f7f0801abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:51.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:51.282 7f7f16ffd700 1 -- 172.21.2.102:0/1292465811 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(162..162 src has 1..162) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f7f18032b40 con 0x7f7f201a7c40 2021-02-10T11:36:51.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:51.282 7f7f2616c700 1 --2- 172.21.2.102:0/1292465811 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7f080186d0 0x7f7f0801abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:51.289 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:51.282 7f7f2616c700 1 --2- 172.21.2.102:0/1292465811 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7f080186d0 0x7f7f0801abd0 crc :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:51.314 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:51.312 7f7f28350700 1 -- 172.21.2.102:0/1292465811 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f2009c0e0 con 0x7f7f201a7c40 2021-02-10T11:36:51.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:51.314 7f7f16ffd700 1 -- 172.21.2.102:0/1292465811 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f7f1801f880 con 0x7f7f201a7c40 2021-02-10T11:36:51.447 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:51.445 7f7f28350700 1 -- 172.21.2.102:0/1292465811 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f7f2009c0e0 con 0x7f7f201a7c40 2021-02-10T11:36:51.451 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:51.449 7f7f16ffd700 1 -- 172.21.2.102:0/1292465811 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.2 v91) v1 ==== 184+0+0 (secure 0 0 0) 0x7f7f18000df0 con 0x7f7f201a7c40 2021-02-10T11:36:51.455 INFO:teuthology.orchestra.run.gibba002.stderr:updated caps for client.2 2021-02-10T11:36:51.457 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:51.454 7f7f28350700 1 -- 172.21.2.102:0/1292465811 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7f080186d0 msgr2=0x7f7f0801abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:51.457 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:51.454 7f7f28350700 1 --2- 172.21.2.102:0/1292465811 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7f080186d0 0x7f7f0801abd0 crc :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:51.457 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:51.454 7f7f28350700 1 -- 172.21.2.102:0/1292465811 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7f201a7c40 msgr2=0x7f7f2009c8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:51.457 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:51.454 7f7f28350700 1 --2- 172.21.2.102:0/1292465811 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7f201a7c40 0x7f7f2009c8a0 secure :-1 s=READY pgs=2881 cs=0 l=1 rev1=1 rx=0x7f7f18006f90 tx=0x7f7f1800bc90).stop 2021-02-10T11:36:51.458 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:51.454 7f7f28350700 1 -- 172.21.2.102:0/1292465811 shutdown_connections 2021-02-10T11:36:51.458 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:51.454 7f7f28350700 1 --2- 172.21.2.102:0/1292465811 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7f201a7c40 0x7f7f2009c8a0 unknown :-1 s=CLOSED pgs=2881 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:51.458 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:51.454 7f7f28350700 1 --2- 172.21.2.102:0/1292465811 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7f2009bce0 0x7f7f201a7620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:51.458 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:51.454 7f7f28350700 1 --2- 172.21.2.102:0/1292465811 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7f080186d0 0x7f7f0801abd0 unknown :-1 s=CLOSED pgs=1265 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:51.458 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:51.454 7f7f28350700 1 --2- 172.21.2.102:0/1292465811 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7f200bcee0 0x7f7f200b5de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:51.458 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:51.454 7f7f28350700 1 -- 172.21.2.102:0/1292465811 >> 172.21.2.102:0/1292465811 conn(0x7f7f2009e660 msgr2=0x7f7f200a9550 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:51.458 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:51.455 7f7f28350700 1 -- 172.21.2.102:0/1292465811 shutdown_connections 2021-02-10T11:36:51.458 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:51.455 7f7f28350700 1 -- 172.21.2.102:0/1292465811 wait complete. 2021-02-10T11:36:51.465 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.3 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:36:51.897 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:51.879 7f71aa0ba700 1 Processor -- start 2021-02-10T11:36:51.897 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:51.879 7f71aa0ba700 1 -- start start 2021-02-10T11:36:51.898 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:51.879 7f71aa0ba700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f71a40d1880 0x7f71a41b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:51.898 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:51.879 7f71aa0ba700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f71a41b0af0 0x7f71a4108950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:51.898 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:51.879 7f71aa0ba700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71a4109a70 con 0x7f71a40d1c80 2021-02-10T11:36:51.898 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:51.879 7f71aa0ba700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71a40e5f00 con 0x7f71a41b0af0 2021-02-10T11:36:51.898 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:51.879 7f71aa0ba700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71a4109760 con 0x7f71a40d1880 2021-02-10T11:36:51.898 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:51.880 7f71a2ffd700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f71a41b0af0 0x7f71a4108950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:51.898 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:51.880 7f71a2ffd700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f71a41b0af0 0x7f71a4108950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:38206/0 (socket says 172.21.2.102:38206) 2021-02-10T11:36:51.898 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:51.880 7f71a2ffd700 1 -- 172.21.2.102:0/2805573843 learned_addr learned my addr 172.21.2.102:0/2805573843 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:36:51.898 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:51.880 7f71a37fe700 1 --2- 172.21.2.102:0/2805573843 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f71a40d1880 0x7f71a41b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:51.898 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:51.880 7f71a2ffd700 1 -- 172.21.2.102:0/2805573843 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f71a40d1880 msgr2=0x7f71a41b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:51.898 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:51.880 7f71a2ffd700 1 --2- 172.21.2.102:0/2805573843 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f71a40d1880 0x7f71a41b04d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:51.898 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:51.880 7f71a2ffd700 1 -- 172.21.2.102:0/2805573843 >> v1:172.21.2.113:6789/0 conn(0x7f71a40d1c80 legacy=0x7f71a4108f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:51.899 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:51.880 7f71a2ffd700 1 -- 172.21.2.102:0/2805573843 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71a4088070 con 0x7f71a41b0af0 2021-02-10T11:36:51.899 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:51.880 7f71a37fe700 1 --2- 172.21.2.102:0/2805573843 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f71a40d1880 0x7f71a41b04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:36:51.899 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:51.880 7f71a2ffd700 1 --2- 172.21.2.102:0/2805573843 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f71a41b0af0 0x7f71a4108950 secure :-1 s=READY pgs=2882 cs=0 l=1 rev1=1 rx=0x7f7194006700 tx=0x7f7194009b80).ready entity=mon.0 client_cookie=9aa85cad5c19fcd6 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:51.899 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:51.880 7f71a27fc700 1 -- 172.21.2.102:0/2805573843 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f719400c0c0 con 0x7f71a41b0af0 2021-02-10T11:36:51.899 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:51.880 7f71a27fc700 1 -- 172.21.2.102:0/2805573843 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f719400c340 con 0x7f71a41b0af0 2021-02-10T11:36:51.899 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:51.880 7f71a27fc700 1 -- 172.21.2.102:0/2805573843 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f719400c860 con 0x7f71a41b0af0 2021-02-10T11:36:51.899 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:51.880 7f71aa0ba700 1 -- 172.21.2.102:0/2805573843 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f71a41b0af0 msgr2=0x7f71a4108950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:51.899 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:51.880 7f71aa0ba700 1 --2- 172.21.2.102:0/2805573843 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f71a41b0af0 0x7f71a4108950 secure :-1 s=READY pgs=2882 cs=0 l=1 rev1=1 rx=0x7f7194006700 tx=0x7f7194009b80).stop 2021-02-10T11:36:51.899 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:51.881 7f71aa0ba700 1 -- 172.21.2.102:0/2805573843 shutdown_connections 2021-02-10T11:36:51.899 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:51.881 7f71aa0ba700 1 --2- 172.21.2.102:0/2805573843 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f71a41b0af0 0x7f71a4108950 unknown :-1 s=CLOSED pgs=2882 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:51.899 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:51.881 7f71aa0ba700 1 --2- 172.21.2.102:0/2805573843 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f71a40d1880 0x7f71a41b04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:51.899 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:51.881 7f71aa0ba700 1 -- 172.21.2.102:0/2805573843 >> 172.21.2.102:0/2805573843 conn(0x7f71a40ce430 msgr2=0x7f71a40d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:51.899 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:51.881 7f71aa0ba700 1 -- 172.21.2.102:0/2805573843 shutdown_connections 2021-02-10T11:36:51.900 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:51.881 7f71aa0ba700 1 -- 172.21.2.102:0/2805573843 wait complete. 2021-02-10T11:36:51.901 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:51.896 7f71aa0ba700 1 Processor -- start 2021-02-10T11:36:51.901 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:51.896 7f71aa0ba700 1 -- start start 2021-02-10T11:36:51.901 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:51.896 7f71aa0ba700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f71a41b9470 0x7f71a41ae6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:51.901 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:51.896 7f71aa0ba700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f71a41aecf0 0x7f71a40957f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:51.901 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:51.896 7f71aa0ba700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f71a41af0f0 0x7f71a4095e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:51.901 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:51.896 7f71aa0ba700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71a4096d70 con 0x7f71a41af0f0 2021-02-10T11:36:51.901 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:51.896 7f71aa0ba700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71a40a7780 con 0x7f71a41b9470 2021-02-10T11:36:51.901 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:51.896 7f71aa0ba700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71a41b9870 con 0x7f71a41aecf0 2021-02-10T11:36:51.901 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:51.896 7f71a3fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f71a41af0f0 0x7f71a4095e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:51.902 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:51.896 7f71a3fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f71a41af0f0 0x7f71a4095e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:38212/0 (socket says 172.21.2.102:38212) 2021-02-10T11:36:51.902 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:51.896 7f71a3fff700 1 -- 172.21.2.102:0/580494945 learned_addr learned my addr 172.21.2.102:0/580494945 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:36:51.902 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:51.897 7f71a3fff700 1 -- 172.21.2.102:0/580494945 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f71a41aecf0 msgr2=0x7f71a40957f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:51.902 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:51.897 7f71a3fff700 1 --2- 172.21.2.102:0/580494945 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f71a41aecf0 0x7f71a40957f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:51.902 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:51.897 7f71a3fff700 1 -- 172.21.2.102:0/580494945 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f71a41b9470 msgr2=0x7f71a41ae6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:51.902 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:51.897 7f71a3fff700 1 --2- 172.21.2.102:0/580494945 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f71a41b9470 0x7f71a41ae6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:51.902 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:51.897 7f71a3fff700 1 -- 172.21.2.102:0/580494945 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71a419e6b0 con 0x7f71a41af0f0 2021-02-10T11:36:51.902 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:51.897 7f71a3fff700 1 --2- 172.21.2.102:0/580494945 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f71a41af0f0 0x7f71a4095e10 secure :-1 s=READY pgs=2883 cs=0 l=1 rev1=1 rx=0x7f719c009940 tx=0x7f719c00bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:51.902 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:51.897 7f719bfff700 1 -- 172.21.2.102:0/580494945 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f719c000ce0 con 0x7f71a41af0f0 2021-02-10T11:36:51.902 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:51.897 7f719bfff700 1 -- 172.21.2.102:0/580494945 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f719c000f90 con 0x7f71a41af0f0 2021-02-10T11:36:51.902 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:51.897 7f719bfff700 1 -- 172.21.2.102:0/580494945 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f719c00c570 con 0x7f71a41af0f0 2021-02-10T11:36:51.902 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:51.897 7f71aa0ba700 1 -- 172.21.2.102:0/580494945 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71a419e6b0 con 0x7f71a41af0f0 2021-02-10T11:36:51.903 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:51.897 7f71aa0ba700 1 -- 172.21.2.102:0/580494945 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71a4096720 con 0x7f71a41af0f0 2021-02-10T11:36:51.903 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:51.898 7f719bfff700 1 -- 172.21.2.102:0/580494945 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f719c001220 con 0x7f71a41af0f0 2021-02-10T11:36:51.903 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:51.898 7f719bfff700 1 --2- 172.21.2.102:0/580494945 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f71880186c0 0x7f718801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:51.903 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:51.898 7f719bfff700 1 -- 172.21.2.102:0/580494945 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(162..162 src has 1..162) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f719c032920 con 0x7f71a41af0f0 2021-02-10T11:36:51.903 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:51.898 7f71a37fe700 1 --2- 172.21.2.102:0/580494945 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f71880186c0 0x7f718801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:51.903 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:51.898 7f71a37fe700 1 --2- 172.21.2.102:0/580494945 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f71880186c0 0x7f718801abc0 crc :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:51.931 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:51.929 7f71aa0ba700 1 -- 172.21.2.102:0/580494945 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71a40829a0 con 0x7f71a41af0f0 2021-02-10T11:36:51.934 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:51.932 7f719bfff700 1 -- 172.21.2.102:0/580494945 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f719c000cc0 con 0x7f71a41af0f0 2021-02-10T11:36:52.063 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:52.061 7f71aa0ba700 1 -- 172.21.2.102:0/580494945 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.3", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f71a4109a70 con 0x7f71a41af0f0 2021-02-10T11:36:52.066 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:52.064 7f719bfff700 1 -- 172.21.2.102:0/580494945 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.3", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.3 v92) v1 ==== 184+0+0 (secure 0 0 0) 0x7f719c01f880 con 0x7f71a41af0f0 2021-02-10T11:36:52.066 INFO:teuthology.orchestra.run.gibba002.stderr:updated caps for client.3 2021-02-10T11:36:52.067 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:52.065 7f71aa0ba700 1 -- 172.21.2.102:0/580494945 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f71880186c0 msgr2=0x7f718801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:52.068 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:52.065 7f71aa0ba700 1 --2- 172.21.2.102:0/580494945 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f71880186c0 0x7f718801abc0 crc :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:52.068 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:52.065 7f71aa0ba700 1 -- 172.21.2.102:0/580494945 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f71a41af0f0 msgr2=0x7f71a4095e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:52.068 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:52.065 7f71aa0ba700 1 --2- 172.21.2.102:0/580494945 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f71a41af0f0 0x7f71a4095e10 secure :-1 s=READY pgs=2883 cs=0 l=1 rev1=1 rx=0x7f719c009940 tx=0x7f719c00bcf0).stop 2021-02-10T11:36:52.068 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:52.065 7f71aa0ba700 1 -- 172.21.2.102:0/580494945 shutdown_connections 2021-02-10T11:36:52.068 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:52.065 7f71aa0ba700 1 --2- 172.21.2.102:0/580494945 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f71a41af0f0 0x7f71a4095e10 unknown :-1 s=CLOSED pgs=2883 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:52.068 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:52.065 7f71aa0ba700 1 --2- 172.21.2.102:0/580494945 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f71a41aecf0 0x7f71a40957f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:52.068 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:52.065 7f71aa0ba700 1 --2- 172.21.2.102:0/580494945 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f71880186c0 0x7f718801abc0 unknown :-1 s=CLOSED pgs=1266 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:52.068 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:52.065 7f71aa0ba700 1 --2- 172.21.2.102:0/580494945 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f71a41b9470 0x7f71a41ae6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:52.068 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:52.065 7f71aa0ba700 1 -- 172.21.2.102:0/580494945 >> 172.21.2.102:0/580494945 conn(0x7f71a40845b0 msgr2=0x7f71a418c290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:52.068 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:52.065 7f71aa0ba700 1 -- 172.21.2.102:0/580494945 shutdown_connections 2021-02-10T11:36:52.068 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:52.065 7f71aa0ba700 1 -- 172.21.2.102:0/580494945 wait complete. 2021-02-10T11:36:52.076 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:36:52.504 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:52.486 7ff194d3d700 1 Processor -- start 2021-02-10T11:36:52.505 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:52.486 7ff194d3d700 1 -- start start 2021-02-10T11:36:52.505 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:52.486 7ff194d3d700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff1900d1870 0x7ff1901b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:52.505 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:52.486 7ff194d3d700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7ff1900f0650 0x7ff1900cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:52.505 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:52.486 7ff194d3d700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1900f0b10 con 0x7ff1901b0a10 2021-02-10T11:36:52.505 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:52.486 7ff194d3d700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1900fe230 con 0x7ff1900f0650 2021-02-10T11:36:52.505 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:52.486 7ff194d3d700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1900e5ed0 con 0x7ff1900d1870 2021-02-10T11:36:52.505 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:52.487 7ff18eb13700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff1900d1870 0x7ff1901b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:52.505 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:52.487 7ff18eb13700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff1900d1870 0x7ff1901b03f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.113:3301/0 says I am v2:172.21.2.102:43756/0 (socket says 172.21.2.102:43756) 2021-02-10T11:36:52.505 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:52.487 7ff18eb13700 1 -- 172.21.2.102:0/3747911473 learned_addr learned my addr 172.21.2.102:0/3747911473 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:36:52.505 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:52.487 7ff18f314700 1 --2- 172.21.2.102:0/3747911473 >> v2:172.21.2.113:3300/0 conn(0x7ff1900f0650 0x7ff1900cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:52.505 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:52.487 7ff18eb13700 1 -- 172.21.2.102:0/3747911473 >> v2:172.21.2.113:3300/0 conn(0x7ff1900f0650 msgr2=0x7ff1900cfb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:52.505 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:52.487 7ff18eb13700 1 --2- 172.21.2.102:0/3747911473 >> v2:172.21.2.113:3300/0 conn(0x7ff1900f0650 0x7ff1900cfb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:52.506 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:52.487 7ff18eb13700 1 -- 172.21.2.102:0/3747911473 >> v1:172.21.2.113:6789/0 conn(0x7ff1901b0a10 legacy=0x7ff1900cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:52.506 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:52.487 7ff18eb13700 1 -- 172.21.2.102:0/3747911473 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1900f20a0 con 0x7ff1900d1870 2021-02-10T11:36:52.506 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:52.487 7ff18eb13700 1 --2- 172.21.2.102:0/3747911473 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff1900d1870 0x7ff1901b03f0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7ff180006700 tx=0x7ff180009b80).ready entity=mon.2 client_cookie=7a675a90c53a26b7 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:52.506 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:52.488 7ff18db11700 1 -- 172.21.2.102:0/3747911473 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff18000c0c0 con 0x7ff1900d1870 2021-02-10T11:36:52.506 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:52.488 7ff18db11700 1 -- 172.21.2.102:0/3747911473 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff18000c340 con 0x7ff1900d1870 2021-02-10T11:36:52.506 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:52.488 7ff18db11700 1 -- 172.21.2.102:0/3747911473 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff18000c860 con 0x7ff1900d1870 2021-02-10T11:36:52.506 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:52.488 7ff194d3d700 1 -- 172.21.2.102:0/3747911473 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff1900d1870 msgr2=0x7ff1901b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:52.506 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:52.488 7ff194d3d700 1 --2- 172.21.2.102:0/3747911473 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff1900d1870 0x7ff1901b03f0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7ff180006700 tx=0x7ff180009b80).stop 2021-02-10T11:36:52.506 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:52.488 7ff194d3d700 1 -- 172.21.2.102:0/3747911473 shutdown_connections 2021-02-10T11:36:52.506 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:52.488 7ff194d3d700 1 --2- 172.21.2.102:0/3747911473 >> v2:172.21.2.113:3300/0 conn(0x7ff1900f0650 0x7ff1900cfb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:52.506 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:52.488 7ff194d3d700 1 --2- 172.21.2.102:0/3747911473 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff1900d1870 0x7ff1901b03f0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:52.507 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:52.488 7ff194d3d700 1 -- 172.21.2.102:0/3747911473 >> 172.21.2.102:0/3747911473 conn(0x7ff1900ce450 msgr2=0x7ff1900d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:52.507 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:52.489 7ff194d3d700 1 -- 172.21.2.102:0/3747911473 shutdown_connections 2021-02-10T11:36:52.507 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:52.489 7ff194d3d700 1 -- 172.21.2.102:0/3747911473 wait complete. 2021-02-10T11:36:52.508 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:52.503 7ff194d3d700 1 Processor -- start 2021-02-10T11:36:52.509 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:52.503 7ff194d3d700 1 -- start start 2021-02-10T11:36:52.509 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:52.503 7ff194d3d700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff1900ada90 0x7ff1900ade90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:52.509 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:52.503 7ff194d3d700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff190099d20 0x7ff19009a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:52.509 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:52.503 7ff194d3d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff19009a740 0x7ff190097bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:52.509 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:52.503 7ff194d3d700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1900997a0 con 0x7ff19009a740 2021-02-10T11:36:52.509 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:52.503 7ff194d3d700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff19018f0c0 con 0x7ff190099d20 2021-02-10T11:36:52.509 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:52.503 7ff194d3d700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff19009ab40 con 0x7ff1900ada90 2021-02-10T11:36:52.509 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:52.503 7ff18e312700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff190099d20 0x7ff19009a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:52.509 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:52.503 7ff18f314700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff19009a740 0x7ff190097bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:52.509 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:52.503 7ff18e312700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff190099d20 0x7ff19009a120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.113:3300/0 says I am v2:172.21.2.102:41656/0 (socket says 172.21.2.102:41656) 2021-02-10T11:36:52.509 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:52.503 7ff18e312700 1 -- 172.21.2.102:0/1820659433 learned_addr learned my addr 172.21.2.102:0/1820659433 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:36:52.510 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:52.503 7ff18f314700 1 --2- 172.21.2.102:0/1820659433 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff19009a740 0x7ff190097bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:38228/0 (socket says 172.21.2.102:38228) 2021-02-10T11:36:52.510 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:52.504 7ff18eb13700 1 --2- 172.21.2.102:0/1820659433 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff1900ada90 0x7ff1900ade90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:52.510 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:52.504 7ff18f314700 1 -- 172.21.2.102:0/1820659433 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff1900ada90 msgr2=0x7ff1900ade90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:52.510 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:52.504 7ff18f314700 1 --2- 172.21.2.102:0/1820659433 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff1900ada90 0x7ff1900ade90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:52.510 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:52.504 7ff18f314700 1 -- 172.21.2.102:0/1820659433 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff190099d20 msgr2=0x7ff19009a120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:52.510 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:52.504 7ff18f314700 1 --2- 172.21.2.102:0/1820659433 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff190099d20 0x7ff19009a120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:52.510 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:52.504 7ff18f314700 1 -- 172.21.2.102:0/1820659433 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff19009ad60 con 0x7ff19009a740 2021-02-10T11:36:52.510 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:52.504 7ff18e312700 1 --2- 172.21.2.102:0/1820659433 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff190099d20 0x7ff19009a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:36:52.510 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:52.504 7ff18f314700 1 --2- 172.21.2.102:0/1820659433 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff19009a740 0x7ff190097bc0 secure :-1 s=READY pgs=2884 cs=0 l=1 rev1=1 rx=0x7ff18800b8a0 tx=0x7ff18800bc40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:52.510 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:52.504 7ff17f7fe700 1 -- 172.21.2.102:0/1820659433 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff188000f90 con 0x7ff19009a740 2021-02-10T11:36:52.510 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:52.504 7ff17f7fe700 1 -- 172.21.2.102:0/1820659433 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1880066f0 con 0x7ff19009a740 2021-02-10T11:36:52.510 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:52.504 7ff194d3d700 1 -- 172.21.2.102:0/1820659433 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff19009ad60 con 0x7ff19009a740 2021-02-10T11:36:52.511 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:52.504 7ff17f7fe700 1 -- 172.21.2.102:0/1820659433 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff188001470 con 0x7ff19009a740 2021-02-10T11:36:52.511 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:52.504 7ff194d3d700 1 -- 172.21.2.102:0/1820659433 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1900984d0 con 0x7ff19009a740 2021-02-10T11:36:52.511 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:52.505 7ff17f7fe700 1 -- 172.21.2.102:0/1820659433 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ff1880011f0 con 0x7ff19009a740 2021-02-10T11:36:52.511 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:52.505 7ff17f7fe700 1 --2- 172.21.2.102:0/1820659433 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff174018730 0x7ff17401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:52.511 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:52.505 7ff17f7fe700 1 -- 172.21.2.102:0/1820659433 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(162..162 src has 1..162) v4 ==== 5573+0+0 (secure 0 0 0) 0x7ff1880328f0 con 0x7ff19009a740 2021-02-10T11:36:52.511 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:52.505 7ff18eb13700 1 --2- 172.21.2.102:0/1820659433 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff174018730 0x7ff17401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:52.511 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:52.506 7ff18eb13700 1 --2- 172.21.2.102:0/1820659433 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff174018730 0x7ff17401ac10 crc :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:52.538 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:52.536 7ff194d3d700 1 -- 172.21.2.102:0/1820659433 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff19018f070 con 0x7ff19009a740 2021-02-10T11:36:52.541 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:52.539 7ff17f7fe700 1 -- 172.21.2.102:0/1820659433 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ff188000f90 con 0x7ff19009a740 2021-02-10T11:36:52.668 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:52.666 7ff194d3d700 1 -- 172.21.2.102:0/1820659433 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff19018f070 con 0x7ff19009a740 2021-02-10T11:36:52.669 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:52.667 7ff17f7fe700 1 -- 172.21.2.102:0/1820659433 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 141 v141) v1 ==== 95+0+2108 (secure 0 0 0) 0x7ff1880066f0 con 0x7ff19009a740 2021-02-10T11:36:52.669 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:36:52.670 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":141,"default_fscid":17,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":12405,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.102:6835/4152930669","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6834","nonce":4152930669},{"type":"v1","addr":"172.21.2.102:6835","nonce":4152930669}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":137}],"filesystems":[{"mdsmap":{"epoch":141,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:36:46.331874","modified":"2021-02-10 11:36:48.340104","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7130},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7130":{"gid":7130,"name":"b","rank":0,"incarnation":138,"state":"up:active","state_seq":3,"addr":"172.21.2.113:6833/2966977555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6832","nonce":2966977555},{"type":"v1","addr":"172.21.2.113:6833","nonce":2966977555}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[35],"metadata_pool":34,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":17}]} 2021-02-10T11:36:52.670 INFO:teuthology.orchestra.run.gibba002.stderr:dumped fsmap epoch 141 2021-02-10T11:36:52.671 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:52.669 7ff194d3d700 1 -- 172.21.2.102:0/1820659433 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff174018730 msgr2=0x7ff17401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:52.671 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:52.669 7ff194d3d700 1 --2- 172.21.2.102:0/1820659433 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff174018730 0x7ff17401ac10 crc :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:52.671 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:52.669 7ff194d3d700 1 -- 172.21.2.102:0/1820659433 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff19009a740 msgr2=0x7ff190097bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:52.671 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:52.669 7ff194d3d700 1 --2- 172.21.2.102:0/1820659433 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff19009a740 0x7ff190097bc0 secure :-1 s=READY pgs=2884 cs=0 l=1 rev1=1 rx=0x7ff18800b8a0 tx=0x7ff18800bc40).stop 2021-02-10T11:36:52.671 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:52.669 7ff194d3d700 1 -- 172.21.2.102:0/1820659433 shutdown_connections 2021-02-10T11:36:52.671 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:52.669 7ff194d3d700 1 --2- 172.21.2.102:0/1820659433 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff19009a740 0x7ff190097bc0 unknown :-1 s=CLOSED pgs=2884 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:52.672 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:52.669 7ff194d3d700 1 --2- 172.21.2.102:0/1820659433 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff174018730 0x7ff17401ac10 unknown :-1 s=CLOSED pgs=1267 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:52.672 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:52.669 7ff194d3d700 1 --2- 172.21.2.102:0/1820659433 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff190099d20 0x7ff19009a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:52.672 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:52.669 7ff194d3d700 1 --2- 172.21.2.102:0/1820659433 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff1900ada90 0x7ff1900ade90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:52.672 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:52.669 7ff194d3d700 1 -- 172.21.2.102:0/1820659433 >> 172.21.2.102:0/1820659433 conn(0x7ff1900f0550 msgr2=0x7ff190191730 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:52.672 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:52.669 7ff194d3d700 1 -- 172.21.2.102:0/1820659433 shutdown_connections 2021-02-10T11:36:52.672 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:52.669 7ff194d3d700 1 -- 172.21.2.102:0/1820659433 wait complete. 2021-02-10T11:36:52.881 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 141, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:36:46.331874', 'modified': '2021-02-10 11:36:48.340104', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7130}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7130': {'gid': 7130, 'name': 'b', 'rank': 0, 'incarnation': 138, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.113:6833/2966977555', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.113:6832', 'nonce': 2966977555}, {'type': 'v1', 'addr': '172.21.2.113:6833', 'nonce': 2966977555}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [35], 'metadata_pool': 34, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:36:52.881 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:36:52.881 DEBUG:teuthology.orchestra.run.gibba013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:36:53.281 INFO:teuthology.orchestra.run.gibba013.stdout:{ 2021-02-10T11:36:53.281 INFO:teuthology.orchestra.run.gibba013.stdout: "cluster_fsid": "68db30ea-dbce-4aef-a6df-f01c6fea0513", 2021-02-10T11:36:53.281 INFO:teuthology.orchestra.run.gibba013.stdout: "whoami": 0, 2021-02-10T11:36:53.281 INFO:teuthology.orchestra.run.gibba013.stdout: "id": 7130, 2021-02-10T11:36:53.281 INFO:teuthology.orchestra.run.gibba013.stdout: "want_state": "up:active", 2021-02-10T11:36:53.281 INFO:teuthology.orchestra.run.gibba013.stdout: "state": "up:active", 2021-02-10T11:36:53.281 INFO:teuthology.orchestra.run.gibba013.stdout: "rank_uptime": 6.9459229320000002, 2021-02-10T11:36:53.281 INFO:teuthology.orchestra.run.gibba013.stdout: "mdsmap_epoch": 141, 2021-02-10T11:36:53.281 INFO:teuthology.orchestra.run.gibba013.stdout: "osdmap_epoch": 162, 2021-02-10T11:36:53.282 INFO:teuthology.orchestra.run.gibba013.stdout: "osdmap_epoch_barrier": 162, 2021-02-10T11:36:53.282 INFO:teuthology.orchestra.run.gibba013.stdout: "uptime": 12.610393522000001 2021-02-10T11:36:53.282 INFO:teuthology.orchestra.run.gibba013.stdout:} 2021-02-10T11:36:53.285 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "68db30ea-dbce-4aef-a6df-f01c6fea0513", "whoami": 0, "id": 7130, "want_state": "up:active", "state": "up:active", "rank_uptime": 6.9459229320000002, "mdsmap_epoch": 141, "osdmap_epoch": 162, "osdmap_epoch_barrier": 162, "uptime": 12.610393522000001 } 2021-02-10T11:36:53.286 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:36:53.713 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:53.695 7f80ea678700 1 Processor -- start 2021-02-10T11:36:53.714 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:53.695 7f80ea678700 1 -- start start 2021-02-10T11:36:53.714 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:53.695 7f80ea678700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f80e40d1870 0x7f80e41b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:53.714 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:53.695 7f80ea678700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f80e41b0a10 0x7f80e40cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:53.714 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:53.695 7f80ea678700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f80e40d1c70 0x7f80e40cf950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:53.714 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:53.695 7f80ea678700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80e40fe230 con 0x7f80e41b0a10 2021-02-10T11:36:53.714 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:53.695 7f80ea678700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80e40e5ed0 con 0x7f80e40d1c70 2021-02-10T11:36:53.714 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:53.695 7f80ea678700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80e40f0b10 con 0x7f80e40d1870 2021-02-10T11:36:53.714 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:53.696 7f80e8c95700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f80e40d1c70 0x7f80e40cf950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:53.714 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:53.696 7f80e37fe700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f80e41b0a10 0x7f80e40cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:53.714 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:53.696 7f80e8c95700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f80e40d1c70 0x7f80e40cf950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.113:3300/0 says I am v2:172.21.2.102:41668/0 (socket says 172.21.2.102:41668) 2021-02-10T11:36:53.714 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:53.696 7f80e8c95700 1 -- 172.21.2.102:0/1328995379 learned_addr learned my addr 172.21.2.102:0/1328995379 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:36:53.715 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:53.696 7f80e3fff700 1 --2- 172.21.2.102:0/1328995379 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f80e40d1870 0x7f80e41b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:53.715 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:53.696 7f80e8c95700 1 -- 172.21.2.102:0/1328995379 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f80e40d1870 msgr2=0x7f80e41b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:53.715 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:53.696 7f80e8c95700 1 --2- 172.21.2.102:0/1328995379 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f80e40d1870 0x7f80e41b03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:53.715 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:53.696 7f80e8c95700 1 -- 172.21.2.102:0/1328995379 >> v2:172.21.2.102:3300/0 conn(0x7f80e41b0a10 msgr2=0x7f80e40cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:53.715 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:53.696 7f80e8c95700 1 --2- 172.21.2.102:0/1328995379 >> v2:172.21.2.102:3300/0 conn(0x7f80e41b0a10 0x7f80e40cf330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:53.715 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:53.696 7f80e8c95700 1 -- 172.21.2.102:0/1328995379 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80e40cff70 con 0x7f80e40d1c70 2021-02-10T11:36:53.715 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:53.696 7f80e37fe700 1 --2- 172.21.2.102:0/1328995379 >> v2:172.21.2.102:3300/0 conn(0x7f80e41b0a10 0x7f80e40cf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:36:53.715 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:53.696 7f80e8c95700 1 --2- 172.21.2.102:0/1328995379 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f80e40d1c70 0x7f80e40cf950 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 rx=0x7f80dc006720 tx=0x7f80dc009ba0).ready entity=mon.1 client_cookie=66439f792a28b51e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:53.715 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:53.696 7f80e2ffd700 1 -- 172.21.2.102:0/1328995379 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f80dc006370 con 0x7f80e40d1c70 2021-02-10T11:36:53.715 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:53.697 7f80e2ffd700 1 -- 172.21.2.102:0/1328995379 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80dc0022e0 con 0x7f80e40d1c70 2021-02-10T11:36:53.715 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:53.697 7f80e2ffd700 1 -- 172.21.2.102:0/1328995379 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f80dc0027c0 con 0x7f80e40d1c70 2021-02-10T11:36:53.715 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:53.697 7f80ea678700 1 -- 172.21.2.102:0/1328995379 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f80e40d1c70 msgr2=0x7f80e40cf950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:53.715 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:53.697 7f80ea678700 1 --2- 172.21.2.102:0/1328995379 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f80e40d1c70 0x7f80e40cf950 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 rx=0x7f80dc006720 tx=0x7f80dc009ba0).stop 2021-02-10T11:36:53.716 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:53.697 7f80ea678700 1 -- 172.21.2.102:0/1328995379 shutdown_connections 2021-02-10T11:36:53.716 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:53.697 7f80ea678700 1 --2- 172.21.2.102:0/1328995379 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f80e40d1c70 0x7f80e40cf950 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:53.716 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:53.697 7f80ea678700 1 --2- 172.21.2.102:0/1328995379 >> v2:172.21.2.102:3300/0 conn(0x7f80e41b0a10 0x7f80e40cf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:53.716 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:53.697 7f80ea678700 1 --2- 172.21.2.102:0/1328995379 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f80e40d1870 0x7f80e41b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:53.716 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:53.697 7f80ea678700 1 -- 172.21.2.102:0/1328995379 >> 172.21.2.102:0/1328995379 conn(0x7f80e40ce450 msgr2=0x7f80e40d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:53.716 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:53.697 7f80ea678700 1 -- 172.21.2.102:0/1328995379 shutdown_connections 2021-02-10T11:36:53.716 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:53.697 7f80ea678700 1 -- 172.21.2.102:0/1328995379 wait complete. 2021-02-10T11:36:53.718 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:53.712 7f80ea678700 1 Processor -- start 2021-02-10T11:36:53.718 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:53.712 7f80ea678700 1 -- start start 2021-02-10T11:36:53.718 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:53.712 7f80ea678700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f80e41a4a60 0x7f80e40c1c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:53.718 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:53.712 7f80ea678700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f80e40c2290 0x7f80e40c2690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:53.718 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:53.712 7f80ea678700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f80e40c2cb0 0x7f80e40bfd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:53.718 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:53.712 7f80ea678700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80e40f05b0 con 0x7f80e40c2cb0 2021-02-10T11:36:53.718 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:53.712 7f80ea678700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80e40f07d0 con 0x7f80e40c2290 2021-02-10T11:36:53.718 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:53.712 7f80ea678700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80e40c0340 con 0x7f80e41a4a60 2021-02-10T11:36:53.718 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:53.712 7f80e8c95700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f80e40c2cb0 0x7f80e40bfd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:53.719 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:53.712 7f80e8c95700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f80e40c2cb0 0x7f80e40bfd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:38242/0 (socket says 172.21.2.102:38242) 2021-02-10T11:36:53.719 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:53.712 7f80e8c95700 1 -- 172.21.2.102:0/183468922 learned_addr learned my addr 172.21.2.102:0/183468922 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:36:53.719 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:53.712 7f80e37fe700 1 --2- 172.21.2.102:0/183468922 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f80e40c2290 0x7f80e40c2690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:53.719 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:53.713 7f80e3fff700 1 --2- 172.21.2.102:0/183468922 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f80e41a4a60 0x7f80e40c1c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:53.719 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:53.713 7f80e8c95700 1 -- 172.21.2.102:0/183468922 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f80e41a4a60 msgr2=0x7f80e40c1c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:53.719 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:53.713 7f80e8c95700 1 --2- 172.21.2.102:0/183468922 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f80e41a4a60 0x7f80e40c1c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:53.719 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:53.713 7f80e8c95700 1 -- 172.21.2.102:0/183468922 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f80e40c2290 msgr2=0x7f80e40c2690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:53.719 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:53.713 7f80e8c95700 1 --2- 172.21.2.102:0/183468922 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f80e40c2290 0x7f80e40c2690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:53.719 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:53.713 7f80e8c95700 1 -- 172.21.2.102:0/183468922 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80e40c0560 con 0x7f80e40c2cb0 2021-02-10T11:36:53.719 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:53.713 7f80e8c95700 1 --2- 172.21.2.102:0/183468922 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f80e40c2cb0 0x7f80e40bfd20 secure :-1 s=READY pgs=2885 cs=0 l=1 rev1=1 rx=0x7f80dc0095c0 tx=0x7f80dc00b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:53.719 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:53.713 7f80e0ff9700 1 -- 172.21.2.102:0/183468922 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f80dc00bd80 con 0x7f80e40c2cb0 2021-02-10T11:36:53.719 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:53.713 7f80e0ff9700 1 -- 172.21.2.102:0/183468922 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80dc006240 con 0x7f80e40c2cb0 2021-02-10T11:36:53.720 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:53.713 7f80ea678700 1 -- 172.21.2.102:0/183468922 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80e40c0560 con 0x7f80e40c2cb0 2021-02-10T11:36:53.720 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:53.713 7f80e0ff9700 1 -- 172.21.2.102:0/183468922 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f80dc005520 con 0x7f80e40c2cb0 2021-02-10T11:36:53.720 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:53.713 7f80ea678700 1 -- 172.21.2.102:0/183468922 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80e40c0560 con 0x7f80e40c2cb0 2021-02-10T11:36:53.720 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:53.714 7f80e0ff9700 1 -- 172.21.2.102:0/183468922 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f80dc005540 con 0x7f80e40c2cb0 2021-02-10T11:36:53.720 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:53.714 7f80e0ff9700 1 --2- 172.21.2.102:0/183468922 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f80c4018730 0x7f80c401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:53.720 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:53.714 7f80e0ff9700 1 -- 172.21.2.102:0/183468922 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(162..162 src has 1..162) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f80dc032600 con 0x7f80e40c2cb0 2021-02-10T11:36:53.720 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:53.714 7f80e3fff700 1 --2- 172.21.2.102:0/183468922 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f80c4018730 0x7f80c401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:53.720 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:53.714 7f80e3fff700 1 --2- 172.21.2.102:0/183468922 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f80c4018730 0x7f80c401abc0 crc :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:53.747 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:53.745 7f80ea678700 1 -- 172.21.2.102:0/183468922 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80e40f0b10 con 0x7f80e40c2cb0 2021-02-10T11:36:53.750 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:53.748 7f80e0ff9700 1 -- 172.21.2.102:0/183468922 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f80dc005330 con 0x7f80e40c2cb0 2021-02-10T11:36:53.873 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:53.871 7f80ea678700 1 -- 172.21.2.102:0/183468922 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f80e40f0b10 con 0x7f80e40c2cb0 2021-02-10T11:36:53.874 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:53.872 7f80e0ff9700 1 -- 172.21.2.102:0/183468922 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v162) v1 ==== 74+0+14007 (secure 0 0 0) 0x7f80dc006240 con 0x7f80e40c2cb0 2021-02-10T11:36:53.874 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:36:53.874 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":162,"fsid":"68db30ea-dbce-4aef-a6df-f01c6fea0513","created":"2021-02-10 11:03:06.331557","modified":"2021-02-10 11:36:47.333734","last_up_change":"2021-02-10 11:03:32.690562","last_in_change":"2021-02-10 11:03:32.690562","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":35,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:03:37.322519","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":34,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:36:44.297293","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"161","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":35,"pool_name":"cephfs_data","create_time":"2021-02-10 11:36:45.321943","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"162","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"be691b4d-a039-408a-af80-9d53f02b6f1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6818","nonce":36088},{"type":"v1","addr":"172.21.2.102:6820","nonce":36088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6822","nonce":36088},{"type":"v1","addr":"172.21.2.102:6824","nonce":36088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6830","nonce":36088},{"type":"v1","addr":"172.21.2.102:6832","nonce":36088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6826","nonce":36088},{"type":"v1","addr":"172.21.2.102:6828","nonce":36088}]},"public_addr":"172.21.2.102:6820/36088","cluster_addr":"172.21.2.102:6824/36088","heartbeat_back_addr":"172.21.2.102:6832/36088","heartbeat_front_addr":"172.21.2.102:6828/36088","state":["exists","up"]},{"osd":1,"uuid":"64e3ccc8-9c69-4d7c-865e-af8e83822635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6819","nonce":36087},{"type":"v1","addr":"172.21.2.102:6821","nonce":36087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6823","nonce":36087},{"type":"v1","addr":"172.21.2.102:6825","nonce":36087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6831","nonce":36087},{"type":"v1","addr":"172.21.2.102:6833","nonce":36087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6827","nonce":36087},{"type":"v1","addr":"172.21.2.102:6829","nonce":36087}]},"public_addr":"172.21.2.102:6821/36087","cluster_addr":"172.21.2.102:6825/36087","heartbeat_back_addr":"172.21.2.102:6833/36087","heartbeat_front_addr":"172.21.2.102:6829/36087","state":["exists","up"]},{"osd":2,"uuid":"153700af-747f-481f-bdf4-d6072d3a842f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6810","nonce":36089},{"type":"v1","addr":"172.21.2.102:6811","nonce":36089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6812","nonce":36089},{"type":"v1","addr":"172.21.2.102:6813","nonce":36089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6816","nonce":36089},{"type":"v1","addr":"172.21.2.102:6817","nonce":36089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6814","nonce":36089},{"type":"v1","addr":"172.21.2.102:6815","nonce":36089}]},"public_addr":"172.21.2.102:6811/36089","cluster_addr":"172.21.2.102:6813/36089","heartbeat_back_addr":"172.21.2.102:6817/36089","heartbeat_front_addr":"172.21.2.102:6815/36089","state":["exists","up"]},{"osd":3,"uuid":"cbc678f4-83a9-45e6-8830-2f7c232b1ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6802","nonce":36090},{"type":"v1","addr":"172.21.2.102:6803","nonce":36090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6804","nonce":36090},{"type":"v1","addr":"172.21.2.102:6805","nonce":36090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6808","nonce":36090},{"type":"v1","addr":"172.21.2.102:6809","nonce":36090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6806","nonce":36090},{"type":"v1","addr":"172.21.2.102:6807","nonce":36090}]},"public_addr":"172.21.2.102:6803/36090","cluster_addr":"172.21.2.102:6805/36090","heartbeat_back_addr":"172.21.2.102:6809/36090","heartbeat_front_addr":"172.21.2.102:6807/36090","state":["exists","up"]},{"osd":4,"uuid":"c01e4008-b258-4f10-afd2-9e7f37214351","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6808","nonce":36368},{"type":"v1","addr":"172.21.2.113:6811","nonce":36368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6814","nonce":36368},{"type":"v1","addr":"172.21.2.113:6818","nonce":36368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6826","nonce":36368},{"type":"v1","addr":"172.21.2.113:6828","nonce":36368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6820","nonce":36368},{"type":"v1","addr":"172.21.2.113:6824","nonce":36368}]},"public_addr":"172.21.2.113:6811/36368","cluster_addr":"172.21.2.113:6818/36368","heartbeat_back_addr":"172.21.2.113:6828/36368","heartbeat_front_addr":"172.21.2.113:6824/36368","state":["exists","up"]},{"osd":5,"uuid":"551f6007-daf8-4a98-8ee0-c2b16bcccb5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6802","nonce":36366},{"type":"v1","addr":"172.21.2.113:6805","nonce":36366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6807","nonce":36366},{"type":"v1","addr":"172.21.2.113:6810","nonce":36366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6819","nonce":36366},{"type":"v1","addr":"172.21.2.113:6822","nonce":36366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6812","nonce":36366},{"type":"v1","addr":"172.21.2.113:6815","nonce":36366}]},"public_addr":"172.21.2.113:6805/36366","cluster_addr":"172.21.2.113:6810/36366","heartbeat_back_addr":"172.21.2.113:6822/36366","heartbeat_front_addr":"172.21.2.113:6815/36366","state":["exists","up"]},{"osd":6,"uuid":"bf67c040-0e50-48be-b102-c3943bc9ebef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6800","nonce":36367},{"type":"v1","addr":"172.21.2.113:6801","nonce":36367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6803","nonce":36367},{"type":"v1","addr":"172.21.2.113:6804","nonce":36367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6813","nonce":36367},{"type":"v1","addr":"172.21.2.113:6816","nonce":36367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6806","nonce":36367},{"type":"v1","addr":"172.21.2.113:6809","nonce":36367}]},"public_addr":"172.21.2.113:6801/36367","cluster_addr":"172.21.2.113:6804/36367","heartbeat_back_addr":"172.21.2.113:6816/36367","heartbeat_front_addr":"172.21.2.113:6809/36367","state":["exists","up"]},{"osd":7,"uuid":"f5461f80-018d-428f-be89-2eb2bcbe7bb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6817","nonce":36365},{"type":"v1","addr":"172.21.2.113:6821","nonce":36365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6823","nonce":36365},{"type":"v1","addr":"172.21.2.113:6825","nonce":36365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6830","nonce":36365},{"type":"v1","addr":"172.21.2.113:6831","nonce":36365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6827","nonce":36365},{"type":"v1","addr":"172.21.2.113:6829","nonce":36365}]},"public_addr":"172.21.2.113:6821/36365","cluster_addr":"172.21.2.113:6825/36365","heartbeat_back_addr":"172.21.2.113:6831/36365","heartbeat_front_addr":"172.21.2.113:6829/36365","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:36:53.876 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:53.873 7f80ea678700 1 -- 172.21.2.102:0/183468922 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f80c4018730 msgr2=0x7f80c401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:53.876 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:53.873 7f80ea678700 1 --2- 172.21.2.102:0/183468922 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f80c4018730 0x7f80c401abc0 crc :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:53.876 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:53.873 7f80ea678700 1 -- 172.21.2.102:0/183468922 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f80e40c2cb0 msgr2=0x7f80e40bfd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:53.876 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:53.873 7f80ea678700 1 --2- 172.21.2.102:0/183468922 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f80e40c2cb0 0x7f80e40bfd20 secure :-1 s=READY pgs=2885 cs=0 l=1 rev1=1 rx=0x7f80dc0095c0 tx=0x7f80dc00b990).stop 2021-02-10T11:36:53.876 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:53.874 7f80ea678700 1 -- 172.21.2.102:0/183468922 shutdown_connections 2021-02-10T11:36:53.876 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:53.874 7f80ea678700 1 --2- 172.21.2.102:0/183468922 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f80e40c2cb0 0x7f80e40bfd20 unknown :-1 s=CLOSED pgs=2885 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:53.876 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:53.874 7f80ea678700 1 --2- 172.21.2.102:0/183468922 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f80c4018730 0x7f80c401abc0 unknown :-1 s=CLOSED pgs=1268 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:53.876 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:53.874 7f80ea678700 1 --2- 172.21.2.102:0/183468922 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f80e40c2290 0x7f80e40c2690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:53.876 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:53.874 7f80ea678700 1 --2- 172.21.2.102:0/183468922 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f80e41a4a60 0x7f80e40c1c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:53.876 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:53.874 7f80ea678700 1 -- 172.21.2.102:0/183468922 >> 172.21.2.102:0/183468922 conn(0x7f80e40ebe30 msgr2=0x7f80e40bf200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:53.877 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:53.874 7f80ea678700 1 -- 172.21.2.102:0/183468922 shutdown_connections 2021-02-10T11:36:53.877 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:53.874 7f80ea678700 1 -- 172.21.2.102:0/183468922 wait complete. 2021-02-10T11:36:53.885 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:03:37.322519', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 34, 'pool_name': 'cephfs_metadata', 'create_time': '2021-02-10 11:36:44.297293', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '161', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 35, 'pool_name': 'cephfs_data', 'create_time': '2021-02-10 11:36:45.321943', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '162', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-02-10T11:36:53.885 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:36:54.319 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:54.299 7fd24c37d700 1 Processor -- start 2021-02-10T11:36:54.319 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:54.299 7fd24c37d700 1 -- start start 2021-02-10T11:36:54.319 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:54.299 7fd24c37d700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fd2440d1870 0x7fd2441b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:54.319 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:54.299 7fd24c37d700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2440fe1c0 con 0x7fd2441b0ae0 2021-02-10T11:36:54.320 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:54.299 7fd24c37d700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd244109a60 con 0x7fd2440cff70 2021-02-10T11:36:54.320 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:54.299 7fd24c37d700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd244088070 con 0x7fd2440d1870 2021-02-10T11:36:54.320 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:54.299 7fd249998700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fd2441b0ae0 0x7fd244108940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:58704/0 (socket says 172.21.2.102:58704) 2021-02-10T11:36:54.320 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:54.299 7fd249998700 1 -- 172.21.2.102:0/1892595168 learned_addr learned my addr 172.21.2.102:0/1892595168 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:36:54.320 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:54.300 7fd249197700 1 -- 172.21.2.102:0/1892595168 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2930055337 0 0) 0x7fd2380010b0 con 0x7fd2441b0ae0 2021-02-10T11:36:54.320 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:54.300 7fd24a199700 1 --2- 172.21.2.102:0/1892595168 >> v2:172.21.2.113:3300/0 conn(0x7fd2440d1870 0x7fd2441b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:54.320 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:54.300 7fd249197700 1 -- 172.21.2.102:0/1892595168 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd230002fe0 con 0x7fd2441b0ae0 2021-02-10T11:36:54.320 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:54.300 7fd249197700 1 -- 172.21.2.102:0/1892595168 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1701547990 0 0) 0x7fd2380010b0 con 0x7fd2441b0ae0 2021-02-10T11:36:54.320 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:54.300 7fd249197700 1 -- 172.21.2.102:0/1892595168 >> v2:172.21.2.113:3300/0 conn(0x7fd2440d1870 msgr2=0x7fd2441b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:54.320 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:54.300 7fd249197700 1 --2- 172.21.2.102:0/1892595168 >> v2:172.21.2.113:3300/0 conn(0x7fd2440d1870 0x7fd2441b04c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:54.320 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:54.300 7fd249197700 1 -- 172.21.2.102:0/1892595168 >> v1:172.21.2.113:6789/0 conn(0x7fd2440cff70 legacy=0x7fd244109130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:54.320 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:54.300 7fd249197700 1 -- 172.21.2.102:0/1892595168 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd244088260 con 0x7fd2441b0ae0 2021-02-10T11:36:54.320 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:54.300 7fd249197700 1 -- 172.21.2.102:0/1892595168 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7fd2380015b0 con 0x7fd2441b0ae0 2021-02-10T11:36:54.321 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:54.300 7fd249197700 1 -- 172.21.2.102:0/1892595168 >> v1:172.21.2.102:6789/0 conn(0x7fd2441b0ae0 legacy=0x7fd244108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:54.321 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:54.300 7fd249197700 1 --2- 172.21.2.102:0/1892595168 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd230003a60 0x7fd230005f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:54.321 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:54.300 7fd249197700 1 --2- 172.21.2.102:0/1892595168 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd230006800 0x7fd230008cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:54.321 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:54.300 7fd249197700 1 --2- 172.21.2.102:0/1892595168 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd2300095b0 0x7fd23000bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:54.321 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:54.301 7fd249197700 1 -- 172.21.2.102:0/1892595168 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd23000c300 con 0x7fd2300095b0 2021-02-10T11:36:54.321 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:54.301 7fd249197700 1 -- 172.21.2.102:0/1892595168 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd23000c520 con 0x7fd230006800 2021-02-10T11:36:54.321 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:54.301 7fd249197700 1 -- 172.21.2.102:0/1892595168 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd23000c740 con 0x7fd230003a60 2021-02-10T11:36:54.321 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:54.301 7fd24a99a700 1 --2- 172.21.2.102:0/1892595168 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd2300095b0 0x7fd23000bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:54.321 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:54.301 7fd24a199700 1 --2- 172.21.2.102:0/1892595168 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd230003a60 0x7fd230005f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:54.321 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:54.301 7fd249998700 1 --2- 172.21.2.102:0/1892595168 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd230006800 0x7fd230008cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:54.321 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:54.301 7fd24a99a700 1 -- 172.21.2.102:0/1892595168 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd230003a60 msgr2=0x7fd230005f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:54.321 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:54.301 7fd24a99a700 1 --2- 172.21.2.102:0/1892595168 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd230003a60 0x7fd230005f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:54.321 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:54.301 7fd24a99a700 1 -- 172.21.2.102:0/1892595168 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd230006800 msgr2=0x7fd230008cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:54.322 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:54.301 7fd24a99a700 1 --2- 172.21.2.102:0/1892595168 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd230006800 0x7fd230008cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:54.322 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:54.301 7fd24a99a700 1 -- 172.21.2.102:0/1892595168 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2300509b0 con 0x7fd2300095b0 2021-02-10T11:36:54.322 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:54.301 7fd249998700 1 --2- 172.21.2.102:0/1892595168 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd230006800 0x7fd230008cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:36:54.322 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:54.301 7fd24a99a700 1 -- 172.21.2.102:0/1892595168 reap_dead start 2021-02-10T11:36:54.322 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:54.301 7fd24a99a700 1 --2- 172.21.2.102:0/1892595168 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd2300095b0 0x7fd23000bab0 secure :-1 s=READY pgs=2887 cs=0 l=1 rev1=1 rx=0x7fd23c006e40 tx=0x7fd23c009db0).ready entity=mon.0 client_cookie=551b397188450d34 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:54.322 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:54.301 7fd249197700 1 -- 172.21.2.102:0/1892595168 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd23c00b300 con 0x7fd2300095b0 2021-02-10T11:36:54.322 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:54.301 7fd249197700 1 -- 172.21.2.102:0/1892595168 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd23c00b580 con 0x7fd2300095b0 2021-02-10T11:36:54.322 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:54.301 7fd249197700 1 -- 172.21.2.102:0/1892595168 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd23c00b9f0 con 0x7fd2300095b0 2021-02-10T11:36:54.322 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:54.301 7fd24c37d700 1 -- 172.21.2.102:0/1892595168 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd2300095b0 msgr2=0x7fd23000bab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:54.322 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:54.301 7fd24c37d700 1 --2- 172.21.2.102:0/1892595168 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd2300095b0 0x7fd23000bab0 secure :-1 s=READY pgs=2887 cs=0 l=1 rev1=1 rx=0x7fd23c006e40 tx=0x7fd23c009db0).stop 2021-02-10T11:36:54.322 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:54.302 7fd24c37d700 1 -- 172.21.2.102:0/1892595168 shutdown_connections 2021-02-10T11:36:54.323 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:54.302 7fd24c37d700 1 --2- 172.21.2.102:0/1892595168 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd2300095b0 0x7fd23000bab0 unknown :-1 s=CLOSED pgs=2887 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:54.323 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:54.302 7fd24c37d700 1 -- 172.21.2.102:0/1892595168 >> 172.21.2.102:0/1892595168 conn(0x7fd2440ce420 msgr2=0x7fd2440d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:54.323 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:54.302 7fd24c37d700 1 -- 172.21.2.102:0/1892595168 shutdown_connections 2021-02-10T11:36:54.323 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:54.302 7fd24c37d700 1 -- 172.21.2.102:0/1892595168 wait complete. 2021-02-10T11:36:54.323 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:54.317 7fd24c37d700 1 Processor -- start 2021-02-10T11:36:54.323 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:54.317 7fd24c37d700 1 -- start start 2021-02-10T11:36:54.324 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:54.317 7fd24c37d700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd244085bc0 0x7fd2440863e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:54.324 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:54.317 7fd24c37d700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd2440954c0 0x7fd244096300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:54.324 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:54.317 7fd24c37d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd2440958c0 0x7fd244096920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:54.324 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:54.317 7fd24c37d700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd244095d80 con 0x7fd2440958c0 2021-02-10T11:36:54.324 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:54.317 7fd24c37d700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2440bfa60 con 0x7fd2440954c0 2021-02-10T11:36:54.325 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:54.317 7fd24c37d700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd244085fc0 con 0x7fd244085bc0 2021-02-10T11:36:54.325 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:54.318 7fd24a99a700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd2440958c0 0x7fd244096920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:54.325 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:54.318 7fd24a99a700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd2440958c0 0x7fd244096920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:38262/0 (socket says 172.21.2.102:38262) 2021-02-10T11:36:54.325 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:54.318 7fd24a99a700 1 -- 172.21.2.102:0/885119290 learned_addr learned my addr 172.21.2.102:0/885119290 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:36:54.325 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:54.318 7fd24a199700 1 --2- 172.21.2.102:0/885119290 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd244085bc0 0x7fd2440863e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:54.325 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:54.318 7fd24a99a700 1 -- 172.21.2.102:0/885119290 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd244085bc0 msgr2=0x7fd2440863e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:54.325 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:54.318 7fd24a99a700 1 --2- 172.21.2.102:0/885119290 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd244085bc0 0x7fd2440863e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:54.325 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:54.318 7fd24a99a700 1 -- 172.21.2.102:0/885119290 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd2440954c0 msgr2=0x7fd244096300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:54.325 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:54.318 7fd24a99a700 1 --2- 172.21.2.102:0/885119290 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd2440954c0 0x7fd244096300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:54.325 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:54.318 7fd24a99a700 1 -- 172.21.2.102:0/885119290 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd244086c10 con 0x7fd2440958c0 2021-02-10T11:36:54.326 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:54.318 7fd24a99a700 1 --2- 172.21.2.102:0/885119290 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd2440958c0 0x7fd244096920 secure :-1 s=READY pgs=2888 cs=0 l=1 rev1=1 rx=0x7fd23c0099a0 tx=0x7fd23c00bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:54.326 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:54.318 7fd236ffd700 1 -- 172.21.2.102:0/885119290 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd23c000f90 con 0x7fd2440958c0 2021-02-10T11:36:54.326 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:54.318 7fd236ffd700 1 -- 172.21.2.102:0/885119290 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd23c000ce0 con 0x7fd2440958c0 2021-02-10T11:36:54.326 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:54.318 7fd236ffd700 1 -- 172.21.2.102:0/885119290 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd23c00c4e0 con 0x7fd2440958c0 2021-02-10T11:36:54.326 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:54.318 7fd24c37d700 1 -- 172.21.2.102:0/885119290 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd244086c10 con 0x7fd2440958c0 2021-02-10T11:36:54.326 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:54.318 7fd24c37d700 1 -- 172.21.2.102:0/885119290 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd244086c10 con 0x7fd2440958c0 2021-02-10T11:36:54.326 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:54.319 7fd236ffd700 1 -- 172.21.2.102:0/885119290 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd23c001220 con 0x7fd2440958c0 2021-02-10T11:36:54.326 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:54.319 7fd236ffd700 1 --2- 172.21.2.102:0/885119290 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd22c018730 0x7fd22c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:54.326 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:54.319 7fd236ffd700 1 -- 172.21.2.102:0/885119290 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(162..162 src has 1..162) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fd23c032b70 con 0x7fd2440958c0 2021-02-10T11:36:54.326 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:54.319 7fd24a199700 1 --2- 172.21.2.102:0/885119290 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd22c018730 0x7fd22c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:54.326 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:54.319 7fd24a199700 1 --2- 172.21.2.102:0/885119290 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd22c018730 0x7fd22c01abc0 crc :-1 s=READY pgs=1269 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:54.351 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:54.349 7fd24c37d700 1 -- 172.21.2.102:0/885119290 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2440fe1c0 con 0x7fd2440958c0 2021-02-10T11:36:54.353 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:54.351 7fd236ffd700 1 -- 172.21.2.102:0/885119290 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd23c000cc0 con 0x7fd2440958c0 2021-02-10T11:36:54.483 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:54.480 7fd24c37d700 1 -- 172.21.2.102:0/885119290 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7fd2440fe1c0 con 0x7fd2440958c0 2021-02-10T11:36:54.483 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:54.481 7fd236ffd700 1 -- 172.21.2.102:0/885119290 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v162) v1 ==== 92+0+10 (secure 0 0 0) 0x7fd23c000cc0 con 0x7fd2440958c0 2021-02-10T11:36:54.483 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:36:54.485 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:54.483 7fd24c37d700 1 -- 172.21.2.102:0/885119290 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd22c018730 msgr2=0x7fd22c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:54.485 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:54.483 7fd24c37d700 1 --2- 172.21.2.102:0/885119290 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd22c018730 0x7fd22c01abc0 crc :-1 s=READY pgs=1269 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:54.485 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:54.483 7fd24c37d700 1 -- 172.21.2.102:0/885119290 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd2440958c0 msgr2=0x7fd244096920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:54.485 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:54.483 7fd24c37d700 1 --2- 172.21.2.102:0/885119290 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd2440958c0 0x7fd244096920 secure :-1 s=READY pgs=2888 cs=0 l=1 rev1=1 rx=0x7fd23c0099a0 tx=0x7fd23c00bcf0).stop 2021-02-10T11:36:54.485 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:54.483 7fd24c37d700 1 -- 172.21.2.102:0/885119290 shutdown_connections 2021-02-10T11:36:54.485 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:54.483 7fd24c37d700 1 --2- 172.21.2.102:0/885119290 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd2440958c0 0x7fd244096920 unknown :-1 s=CLOSED pgs=2888 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:54.485 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:54.483 7fd24c37d700 1 --2- 172.21.2.102:0/885119290 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd22c018730 0x7fd22c01abc0 unknown :-1 s=CLOSED pgs=1269 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:54.485 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:54.483 7fd24c37d700 1 --2- 172.21.2.102:0/885119290 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd2440954c0 0x7fd244096300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:54.486 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:54.483 7fd24c37d700 1 --2- 172.21.2.102:0/885119290 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd244085bc0 0x7fd2440863e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:54.486 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:54.483 7fd24c37d700 1 -- 172.21.2.102:0/885119290 >> 172.21.2.102:0/885119290 conn(0x7fd2440bef00 msgr2=0x7fd24409d1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:54.486 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:54.483 7fd24c37d700 1 -- 172.21.2.102:0/885119290 shutdown_connections 2021-02-10T11:36:54.486 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:54.483 7fd24c37d700 1 -- 172.21.2.102:0/885119290 wait complete. 2021-02-10T11:36:54.494 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-02-10T11:36:54.929 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:54.909 7f4b28e47700 1 Processor -- start 2021-02-10T11:36:54.929 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:54.909 7f4b28e47700 1 -- start start 2021-02-10T11:36:54.929 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:54.910 7f4b28e47700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4b240d1c70 0x7f4b240cef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:54.929 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:54.910 7f4b28e47700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f4b240cf580 0x7f4b240cf980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:54.929 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:54.910 7f4b28e47700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b240fe1c0 con 0x7f4b240d1870 2021-02-10T11:36:54.930 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:54.910 7f4b28e47700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b241b0cb0 con 0x7f4b240cf580 2021-02-10T11:36:54.930 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:54.910 7f4b28e47700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b24109a60 con 0x7f4b240d1c70 2021-02-10T11:36:54.930 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:54.910 7f4b22b13700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f4b240d1870 0x7f4b241b04c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:58722/0 (socket says 172.21.2.102:58722) 2021-02-10T11:36:54.930 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:54.910 7f4b22b13700 1 -- 172.21.2.102:0/2624104421 learned_addr learned my addr 172.21.2.102:0/2624104421 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:36:54.930 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:54.910 7f4b21b11700 1 -- 172.21.2.102:0/2624104421 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 772331967 0 0) 0x7f4b14000ce0 con 0x7f4b240d1870 2021-02-10T11:36:54.930 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:54.910 7f4b21b11700 1 -- 172.21.2.102:0/2624104421 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b10002fe0 con 0x7f4b240d1870 2021-02-10T11:36:54.930 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:54.910 7f4b21b11700 1 -- 172.21.2.102:0/2624104421 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 882065077 0 0) 0x7f4b140010e0 con 0x7f4b240d1870 2021-02-10T11:36:54.930 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:54.910 7f4b23314700 1 --2- 172.21.2.102:0/2624104421 >> v2:172.21.2.113:3300/0 conn(0x7f4b240cf580 0x7f4b240cf980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:54.930 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:54.910 7f4b22312700 1 --2- 172.21.2.102:0/2624104421 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4b240d1c70 0x7f4b240cef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:54.930 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:54.910 7f4b21b11700 1 -- 172.21.2.102:0/2624104421 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4b240d1c70 msgr2=0x7f4b240cef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:54.930 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:54.911 7f4b21b11700 1 --2- 172.21.2.102:0/2624104421 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4b240d1c70 0x7f4b240cef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:54.930 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:54.911 7f4b21b11700 1 -- 172.21.2.102:0/2624104421 >> v2:172.21.2.113:3300/0 conn(0x7f4b240cf580 msgr2=0x7f4b240cf980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:54.931 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:54.911 7f4b21b11700 1 --2- 172.21.2.102:0/2624104421 >> v2:172.21.2.113:3300/0 conn(0x7f4b240cf580 0x7f4b240cf980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:54.931 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:54.911 7f4b21b11700 1 -- 172.21.2.102:0/2624104421 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b24088070 con 0x7f4b240d1870 2021-02-10T11:36:54.931 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:54.911 7f4b21b11700 1 -- 172.21.2.102:0/2624104421 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7f4b14001660 con 0x7f4b240d1870 2021-02-10T11:36:54.931 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:54.911 7f4b21b11700 1 -- 172.21.2.102:0/2624104421 >> v1:172.21.2.102:6789/0 conn(0x7f4b240d1870 legacy=0x7f4b241b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:54.931 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:54.911 7f4b21b11700 1 --2- 172.21.2.102:0/2624104421 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4b10003b90 0x7f4b10006080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:54.931 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:54.911 7f4b21b11700 1 --2- 172.21.2.102:0/2624104421 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4b100068a0 0x7f4b10008dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:54.931 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:54.911 7f4b21b11700 1 --2- 172.21.2.102:0/2624104421 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4b10009660 0x7f4b1000bb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:54.931 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:54.911 7f4b21b11700 1 -- 172.21.2.102:0/2624104421 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b1000c410 con 0x7f4b100068a0 2021-02-10T11:36:54.931 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:54.911 7f4b21b11700 1 -- 172.21.2.102:0/2624104421 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b1000c630 con 0x7f4b10009660 2021-02-10T11:36:54.931 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:54.911 7f4b21b11700 1 -- 172.21.2.102:0/2624104421 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b1000c850 con 0x7f4b10003b90 2021-02-10T11:36:54.931 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:54.911 7f4b22312700 1 --2- 172.21.2.102:0/2624104421 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4b240d1c70 0x7f4b240cef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:36:54.931 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:54.911 7f4b23314700 1 --2- 172.21.2.102:0/2624104421 >> v2:172.21.2.113:3300/0 conn(0x7f4b240cf580 0x7f4b240cf980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:36:54.931 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:54.911 7f4b22312700 1 --2- 172.21.2.102:0/2624104421 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4b100068a0 0x7f4b10008dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:54.932 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:54.911 7f4b22312700 1 -- 172.21.2.102:0/2624104421 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4b10003b90 msgr2=0x7f4b10006080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:54.932 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:54.911 7f4b22312700 1 --2- 172.21.2.102:0/2624104421 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4b10003b90 0x7f4b10006080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:54.932 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:54.911 7f4b23314700 1 --2- 172.21.2.102:0/2624104421 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4b10009660 0x7f4b1000bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:54.932 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:54.911 7f4b22312700 1 -- 172.21.2.102:0/2624104421 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4b10009660 msgr2=0x7f4b1000bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:54.932 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:54.911 7f4b22312700 1 --2- 172.21.2.102:0/2624104421 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4b10009660 0x7f4b1000bb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:54.932 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:54.911 7f4b22312700 1 -- 172.21.2.102:0/2624104421 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b1000caf0 con 0x7f4b100068a0 2021-02-10T11:36:54.932 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:54.911 7f4b22312700 1 --2- 172.21.2.102:0/2624104421 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4b100068a0 0x7f4b10008dc0 secure :-1 s=READY pgs=2890 cs=0 l=1 rev1=1 rx=0x7f4b0c002e00 tx=0x7f4b0c04eab0).ready entity=mon.0 client_cookie=d544659625c48a9b server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:54.932 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:54.911 7f4b21b11700 1 -- 172.21.2.102:0/2624104421 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4b0c0510c0 con 0x7f4b100068a0 2021-02-10T11:36:54.932 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:54.911 7f4b21b11700 1 -- 172.21.2.102:0/2624104421 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b0c051380 con 0x7f4b100068a0 2021-02-10T11:36:54.932 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:54.911 7f4b21b11700 1 -- 172.21.2.102:0/2624104421 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4b0c0517f0 con 0x7f4b100068a0 2021-02-10T11:36:54.932 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:54.911 7f4b28e47700 1 -- 172.21.2.102:0/2624104421 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4b100068a0 msgr2=0x7f4b10008dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:54.932 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:54.911 7f4b28e47700 1 --2- 172.21.2.102:0/2624104421 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4b100068a0 0x7f4b10008dc0 secure :-1 s=READY pgs=2890 cs=0 l=1 rev1=1 rx=0x7f4b0c002e00 tx=0x7f4b0c04eab0).stop 2021-02-10T11:36:54.932 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:54.912 7f4b23314700 1 -- 172.21.2.102:0/2624104421 reap_dead start 2021-02-10T11:36:54.932 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:54.912 7f4b28e47700 1 -- 172.21.2.102:0/2624104421 shutdown_connections 2021-02-10T11:36:54.933 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:54.912 7f4b28e47700 1 -- 172.21.2.102:0/2624104421 >> 172.21.2.102:0/2624104421 conn(0x7f4b240ce420 msgr2=0x7f4b240d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:54.933 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:54.912 7f4b28e47700 1 -- 172.21.2.102:0/2624104421 shutdown_connections 2021-02-10T11:36:54.933 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:54.912 7f4b28e47700 1 -- 172.21.2.102:0/2624104421 wait complete. 2021-02-10T11:36:54.933 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:54.927 7f4b28e47700 1 Processor -- start 2021-02-10T11:36:54.933 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:54.927 7f4b28e47700 1 -- start start 2021-02-10T11:36:54.934 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:54.927 7f4b28e47700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4b240b22c0 0x7f4b240b26c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:54.934 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:54.927 7f4b28e47700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4b240a6df0 0x7f4b240a71f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:54.934 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:54.927 7f4b28e47700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4b240a5cd0 0x7f4b240a60d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:54.934 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:54.927 7f4b28e47700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b240c23f0 con 0x7f4b240b22c0 2021-02-10T11:36:54.934 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:54.927 7f4b28e47700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b240a66f0 con 0x7f4b240a5cd0 2021-02-10T11:36:54.934 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:54.927 7f4b28e47700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b240a68e0 con 0x7f4b240a6df0 2021-02-10T11:36:54.935 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:54.927 7f4b22b13700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4b240b22c0 0x7f4b240b26c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:54.935 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:54.927 7f4b22b13700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4b240b22c0 0x7f4b240b26c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:38282/0 (socket says 172.21.2.102:38282) 2021-02-10T11:36:54.935 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:54.927 7f4b22b13700 1 -- 172.21.2.102:0/4277584434 learned_addr learned my addr 172.21.2.102:0/4277584434 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:36:54.935 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:54.928 7f4b22b13700 1 -- 172.21.2.102:0/4277584434 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4b240a6df0 msgr2=0x7f4b240a71f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:54.935 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:54.928 7f4b22b13700 1 --2- 172.21.2.102:0/4277584434 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4b240a6df0 0x7f4b240a71f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:54.935 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:54.928 7f4b22b13700 1 -- 172.21.2.102:0/4277584434 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4b240a5cd0 msgr2=0x7f4b240a60d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:54.935 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:54.928 7f4b22b13700 1 --2- 172.21.2.102:0/4277584434 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4b240a5cd0 0x7f4b240a60d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:54.935 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:54.928 7f4b22b13700 1 -- 172.21.2.102:0/4277584434 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b24083980 con 0x7f4b240b22c0 2021-02-10T11:36:54.935 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:54.928 7f4b22b13700 1 --2- 172.21.2.102:0/4277584434 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4b240b22c0 0x7f4b240b26c0 secure :-1 s=READY pgs=2891 cs=0 l=1 rev1=1 rx=0x7f4b1400b5c0 tx=0x7f4b1400b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:54.935 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:54.928 7f4b1b7fe700 1 -- 172.21.2.102:0/4277584434 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4b14006030 con 0x7f4b240b22c0 2021-02-10T11:36:54.935 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:54.928 7f4b1b7fe700 1 -- 172.21.2.102:0/4277584434 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b14000990 con 0x7f4b240b22c0 2021-02-10T11:36:54.935 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:54.928 7f4b1b7fe700 1 -- 172.21.2.102:0/4277584434 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4b14004590 con 0x7f4b240b22c0 2021-02-10T11:36:54.936 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:54.928 7f4b28e47700 1 -- 172.21.2.102:0/4277584434 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b24083980 con 0x7f4b240b22c0 2021-02-10T11:36:54.936 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:54.928 7f4b28e47700 1 -- 172.21.2.102:0/4277584434 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b24083f10 con 0x7f4b240b22c0 2021-02-10T11:36:54.936 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:54.929 7f4b1b7fe700 1 -- 172.21.2.102:0/4277584434 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4b140043a0 con 0x7f4b240b22c0 2021-02-10T11:36:54.936 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:54.929 7f4b1b7fe700 1 --2- 172.21.2.102:0/4277584434 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4b080186c0 0x7f4b0801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:54.936 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:54.929 7f4b1b7fe700 1 -- 172.21.2.102:0/4277584434 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(162..162 src has 1..162) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f4b140325a0 con 0x7f4b240b22c0 2021-02-10T11:36:54.936 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:54.929 7f4b22312700 1 --2- 172.21.2.102:0/4277584434 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4b080186c0 0x7f4b0801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:54.936 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:54.930 7f4b22312700 1 --2- 172.21.2.102:0/4277584434 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4b080186c0 0x7f4b0801abc0 crc :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:54.961 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:54.959 7f4b28e47700 1 -- 172.21.2.102:0/4277584434 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b24082a70 con 0x7f4b240b22c0 2021-02-10T11:36:54.964 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:54.962 7f4b1b7fe700 1 -- 172.21.2.102:0/4277584434 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f4b14030d50 con 0x7f4b240b22c0 2021-02-10T11:36:55.091 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:55.089 7f4b28e47700 1 -- 172.21.2.102:0/4277584434 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7f4b24082a70 con 0x7f4b240b22c0 2021-02-10T11:36:55.092 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:55.089 7f4b1b7fe700 1 -- 172.21.2.102:0/4277584434 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v162) v1 ==== 104+0+10 (secure 0 0 0) 0x7f4b140043a0 con 0x7f4b240b22c0 2021-02-10T11:36:55.092 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:36:55.093 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:55.091 7f4b28e47700 1 -- 172.21.2.102:0/4277584434 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4b080186c0 msgr2=0x7f4b0801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:55.093 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:55.091 7f4b28e47700 1 --2- 172.21.2.102:0/4277584434 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4b080186c0 0x7f4b0801abc0 crc :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:55.094 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:55.091 7f4b28e47700 1 -- 172.21.2.102:0/4277584434 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4b240b22c0 msgr2=0x7f4b240b26c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:55.094 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:55.091 7f4b28e47700 1 --2- 172.21.2.102:0/4277584434 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4b240b22c0 0x7f4b240b26c0 secure :-1 s=READY pgs=2891 cs=0 l=1 rev1=1 rx=0x7f4b1400b5c0 tx=0x7f4b1400b960).stop 2021-02-10T11:36:55.094 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:55.092 7f4b28e47700 1 -- 172.21.2.102:0/4277584434 shutdown_connections 2021-02-10T11:36:55.094 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:55.092 7f4b28e47700 1 --2- 172.21.2.102:0/4277584434 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4b080186c0 0x7f4b0801abc0 unknown :-1 s=CLOSED pgs=1270 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:55.094 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:55.092 7f4b28e47700 1 --2- 172.21.2.102:0/4277584434 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4b240a5cd0 0x7f4b240a60d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:55.094 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:55.092 7f4b28e47700 1 --2- 172.21.2.102:0/4277584434 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4b240a6df0 0x7f4b240a71f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:55.095 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:55.092 7f4b28e47700 1 --2- 172.21.2.102:0/4277584434 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4b240b22c0 0x7f4b240b26c0 unknown :-1 s=CLOSED pgs=2891 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:55.095 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:55.092 7f4b28e47700 1 -- 172.21.2.102:0/4277584434 >> 172.21.2.102:0/4277584434 conn(0x7f4b2409ffd0 msgr2=0x7f4b24082ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:55.095 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:55.092 7f4b28e47700 1 -- 172.21.2.102:0/4277584434 shutdown_connections 2021-02-10T11:36:55.095 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:55.092 7f4b28e47700 1 -- 172.21.2.102:0/4277584434 wait complete. 2021-02-10T11:36:55.103 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-02-10T11:36:55.535 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:55.517 7f0bf7c86700 1 Processor -- start 2021-02-10T11:36:55.536 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:55.517 7f0bf7c86700 1 -- start start 2021-02-10T11:36:55.536 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:55.517 7f0bf7c86700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f0bf00d1870 0x7f0bf01b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:55.536 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:55.517 7f0bf7c86700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f0bf00cff70 0x7f0bf0109130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:55.536 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:55.517 7f0bf7c86700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bf00fe1c0 con 0x7f0bf01b0ae0 2021-02-10T11:36:55.536 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:55.517 7f0bf7c86700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0bf0109ad0 con 0x7f0bf00cff70 2021-02-10T11:36:55.536 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:55.517 7f0bf7c86700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0bf0109750 con 0x7f0bf00d1870 2021-02-10T11:36:55.536 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:55.517 7f0bf62a3700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f0bf00cff70 0x7f0bf0109130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:55.536 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:55.517 7f0bf62a3700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f0bf00cff70 0x7f0bf0109130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:38286/0 (socket says 172.21.2.102:38286) 2021-02-10T11:36:55.536 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:55.517 7f0bf62a3700 1 -- 172.21.2.102:0/2894139316 learned_addr learned my addr 172.21.2.102:0/2894139316 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:36:55.536 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:55.518 7f0bf62a3700 1 -- 172.21.2.102:0/2894139316 >> v2:172.21.2.113:3300/0 conn(0x7f0bf00d1870 msgr2=0x7f0bf01b04c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:36:55.536 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:55.518 7f0bf62a3700 1 --2- 172.21.2.102:0/2894139316 >> v2:172.21.2.113:3300/0 conn(0x7f0bf00d1870 0x7f0bf01b04c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:55.537 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:55.518 7f0bf62a3700 1 -- 172.21.2.102:0/2894139316 >> v1:172.21.2.113:6789/0 conn(0x7f0bf01b0ae0 legacy=0x7f0bf0108940 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:36:55.537 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:55.518 7f0bf62a3700 1 -- 172.21.2.102:0/2894139316 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bf0088070 con 0x7f0bf00cff70 2021-02-10T11:36:55.537 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:55.518 7f0bf62a3700 1 --2- 172.21.2.102:0/2894139316 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0bf00cff70 0x7f0bf0109130 secure :-1 s=READY pgs=2892 cs=0 l=1 rev1=1 rx=0x7f0be8006f00 tx=0x7f0be8009f10).ready entity=mon.0 client_cookie=bb5e367e5f214973 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:55.537 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:55.518 7f0bf4aa0700 1 -- 172.21.2.102:0/2894139316 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0be800b3b0 con 0x7f0bf00cff70 2021-02-10T11:36:55.537 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:55.518 7f0bf4aa0700 1 -- 172.21.2.102:0/2894139316 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0be8000d30 con 0x7f0bf00cff70 2021-02-10T11:36:55.537 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:55.518 7f0bf4aa0700 1 -- 172.21.2.102:0/2894139316 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0be800b820 con 0x7f0bf00cff70 2021-02-10T11:36:55.537 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:55.518 7f0bf7c86700 1 -- 172.21.2.102:0/2894139316 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0bf00cff70 msgr2=0x7f0bf0109130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:55.537 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:55.518 7f0bf7c86700 1 --2- 172.21.2.102:0/2894139316 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0bf00cff70 0x7f0bf0109130 secure :-1 s=READY pgs=2892 cs=0 l=1 rev1=1 rx=0x7f0be8006f00 tx=0x7f0be8009f10).stop 2021-02-10T11:36:55.537 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:55.519 7f0bf7c86700 1 -- 172.21.2.102:0/2894139316 shutdown_connections 2021-02-10T11:36:55.537 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:55.519 7f0bf7c86700 1 --2- 172.21.2.102:0/2894139316 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0bf00cff70 0x7f0bf0109130 unknown :-1 s=CLOSED pgs=2892 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:55.537 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:55.519 7f0bf7c86700 1 --2- 172.21.2.102:0/2894139316 >> v2:172.21.2.113:3300/0 conn(0x7f0bf00d1870 0x7f0bf01b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:55.537 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:55.519 7f0bf7c86700 1 -- 172.21.2.102:0/2894139316 >> 172.21.2.102:0/2894139316 conn(0x7f0bf00ce420 msgr2=0x7f0bf00d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:55.537 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:55.519 7f0bf7c86700 1 -- 172.21.2.102:0/2894139316 shutdown_connections 2021-02-10T11:36:55.538 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:55.519 7f0bf7c86700 1 -- 172.21.2.102:0/2894139316 wait complete. 2021-02-10T11:36:55.539 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:55.534 7f0bf7c86700 1 Processor -- start 2021-02-10T11:36:55.539 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:55.534 7f0bf7c86700 1 -- start start 2021-02-10T11:36:55.539 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:55.534 7f0bf7c86700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0bf00c3820 0x7f0bf0084cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:55.539 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:55.534 7f0bf7c86700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0bf00a0290 0x7f0bf00852d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:55.539 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:55.534 7f0bf7c86700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0bf0086270 0x7f0bf0086670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:55.540 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:55.534 7f0bf7c86700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bf00c0700 con 0x7f0bf00a0290 2021-02-10T11:36:55.540 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:55.534 7f0bf7c86700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bf00858f0 con 0x7f0bf00c3820 2021-02-10T11:36:55.540 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:55.534 7f0bf7c86700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bf0085b10 con 0x7f0bf0086270 2021-02-10T11:36:55.540 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:55.534 7f0bf52a1700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0bf00a0290 0x7f0bf00852d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:55.540 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:55.534 7f0bf52a1700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0bf00a0290 0x7f0bf00852d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:38290/0 (socket says 172.21.2.102:38290) 2021-02-10T11:36:55.540 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:55.534 7f0bf52a1700 1 -- 172.21.2.102:0/2600227805 learned_addr learned my addr 172.21.2.102:0/2600227805 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:36:55.540 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:55.535 7f0bf52a1700 1 -- 172.21.2.102:0/2600227805 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0bf0086270 msgr2=0x7f0bf0086670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:55.540 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:55.535 7f0bf52a1700 1 --2- 172.21.2.102:0/2600227805 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0bf0086270 0x7f0bf0086670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:55.540 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:55.535 7f0bf52a1700 1 -- 172.21.2.102:0/2600227805 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0bf00c3820 msgr2=0x7f0bf0084cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:55.540 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:55.535 7f0bf52a1700 1 --2- 172.21.2.102:0/2600227805 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0bf00c3820 0x7f0bf0084cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:55.540 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:55.535 7f0bf52a1700 1 -- 172.21.2.102:0/2600227805 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bf0086c90 con 0x7f0bf00a0290 2021-02-10T11:36:55.540 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:55.535 7f0bf52a1700 1 --2- 172.21.2.102:0/2600227805 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0bf00a0290 0x7f0bf00852d0 secure :-1 s=READY pgs=2893 cs=0 l=1 rev1=1 rx=0x7f0be000b5b0 tx=0x7f0be000b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:55.540 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:55.535 7f0be67fc700 1 -- 172.21.2.102:0/2600227805 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0be000bd40 con 0x7f0bf00a0290 2021-02-10T11:36:55.541 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:55.535 7f0be67fc700 1 -- 172.21.2.102:0/2600227805 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0be0006020 con 0x7f0bf00a0290 2021-02-10T11:36:55.541 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:55.535 7f0bf7c86700 1 -- 172.21.2.102:0/2600227805 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bf0086c90 con 0x7f0bf00a0290 2021-02-10T11:36:55.541 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:55.535 7f0be67fc700 1 -- 172.21.2.102:0/2600227805 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0be00054a0 con 0x7f0bf00a0290 2021-02-10T11:36:55.541 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:55.535 7f0bf7c86700 1 -- 172.21.2.102:0/2600227805 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bf0086c90 con 0x7f0bf00a0290 2021-02-10T11:36:55.541 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:55.536 7f0be67fc700 1 -- 172.21.2.102:0/2600227805 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f0be0006200 con 0x7f0bf00a0290 2021-02-10T11:36:55.541 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:55.536 7f0be67fc700 1 --2- 172.21.2.102:0/2600227805 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0bd8018730 0x7f0bd801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:55.541 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:55.536 7f0be67fc700 1 -- 172.21.2.102:0/2600227805 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(162..162 src has 1..162) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f0be0032680 con 0x7f0bf00a0290 2021-02-10T11:36:55.541 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:55.536 7f0bf5aa2700 1 --2- 172.21.2.102:0/2600227805 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0bd8018730 0x7f0bd801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:55.541 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:55.536 7f0bf5aa2700 1 --2- 172.21.2.102:0/2600227805 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0bd8018730 0x7f0bd801ac10 crc :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:55.569 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:55.567 7f0bf7c86700 1 -- 172.21.2.102:0/2600227805 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bf0082a70 con 0x7f0bf00a0290 2021-02-10T11:36:55.572 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:55.570 7f0be67fc700 1 -- 172.21.2.102:0/2600227805 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f0be0006230 con 0x7f0bf00a0290 2021-02-10T11:36:55.698 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:55.696 7f0bf7c86700 1 -- 172.21.2.102:0/2600227805 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7f0bf0082a70 con 0x7f0bf00a0290 2021-02-10T11:36:55.699 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:55.697 7f0be67fc700 1 -- 172.21.2.102:0/2600227805 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v162) v1 ==== 100+0+10 (secure 0 0 0) 0x7f0be0006000 con 0x7f0bf00a0290 2021-02-10T11:36:55.699 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:36:55.701 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:55.699 7f0bf7c86700 1 -- 172.21.2.102:0/2600227805 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0bd8018730 msgr2=0x7f0bd801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:55.701 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:55.699 7f0bf7c86700 1 --2- 172.21.2.102:0/2600227805 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0bd8018730 0x7f0bd801ac10 crc :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:55.701 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:55.699 7f0bf7c86700 1 -- 172.21.2.102:0/2600227805 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0bf00a0290 msgr2=0x7f0bf00852d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:55.701 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:55.699 7f0bf7c86700 1 --2- 172.21.2.102:0/2600227805 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0bf00a0290 0x7f0bf00852d0 secure :-1 s=READY pgs=2893 cs=0 l=1 rev1=1 rx=0x7f0be000b5b0 tx=0x7f0be000b950).stop 2021-02-10T11:36:55.701 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:55.699 7f0bf7c86700 1 -- 172.21.2.102:0/2600227805 shutdown_connections 2021-02-10T11:36:55.701 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:55.699 7f0bf7c86700 1 --2- 172.21.2.102:0/2600227805 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0bf0086270 0x7f0bf0086670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:55.701 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:55.699 7f0bf7c86700 1 --2- 172.21.2.102:0/2600227805 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0bf00a0290 0x7f0bf00852d0 unknown :-1 s=CLOSED pgs=2893 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:55.702 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:55.699 7f0bf7c86700 1 --2- 172.21.2.102:0/2600227805 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0bd8018730 0x7f0bd801ac10 unknown :-1 s=CLOSED pgs=1271 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:55.702 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:55.699 7f0bf7c86700 1 --2- 172.21.2.102:0/2600227805 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0bf00c3820 0x7f0bf0084cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:55.702 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:55.699 7f0bf7c86700 1 -- 172.21.2.102:0/2600227805 >> 172.21.2.102:0/2600227805 conn(0x7f0bf00872c0 msgr2=0x7f0bf0095700 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:55.702 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:55.699 7f0bf7c86700 1 -- 172.21.2.102:0/2600227805 shutdown_connections 2021-02-10T11:36:55.702 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:55.699 7f0bf7c86700 1 -- 172.21.2.102:0/2600227805 wait complete. 2021-02-10T11:36:55.710 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-02-10T11:36:55.710 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:36:56.140 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:56.122 7f3492506700 1 Processor -- start 2021-02-10T11:36:56.140 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:56.122 7f3492506700 1 -- start start 2021-02-10T11:36:56.141 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:56.123 7f3492506700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f348c0d1870 0x7f348c1b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:56.141 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:56.123 7f3492506700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f348c0f0650 0x7f348c0cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:56.141 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:56.123 7f3492506700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f348c0f0b10 con 0x7f348c1b0a10 2021-02-10T11:36:56.141 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:56.123 7f3492506700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f348c0fe230 con 0x7f348c0f0650 2021-02-10T11:36:56.141 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:56.123 7f3492506700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f348c0e5ed0 con 0x7f348c0d1870 2021-02-10T11:36:56.141 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:56.123 7f3490b23700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f348c0f0650 0x7f348c0cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:56.141 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:56.123 7f3490b23700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f348c0f0650 0x7f348c0cfb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:38302/0 (socket says 172.21.2.102:38302) 2021-02-10T11:36:56.141 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:56.123 7f3490b23700 1 -- 172.21.2.102:0/1498851640 learned_addr learned my addr 172.21.2.102:0/1498851640 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:36:56.141 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:56.123 7f3490b23700 1 -- 172.21.2.102:0/1498851640 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f348c0d1870 msgr2=0x7f348c1b03f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:36:56.141 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:56.123 7f3490b23700 1 --2- 172.21.2.102:0/1498851640 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f348c0d1870 0x7f348c1b03f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:56.141 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:56.123 7f3490b23700 1 -- 172.21.2.102:0/1498851640 >> v1:172.21.2.113:6789/0 conn(0x7f348c1b0a10 legacy=0x7f348c0cf330 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:36:56.141 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:56.123 7f3490b23700 1 -- 172.21.2.102:0/1498851640 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f348c0f20a0 con 0x7f348c0f0650 2021-02-10T11:36:56.142 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:56.123 7f3490b23700 1 --2- 172.21.2.102:0/1498851640 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f348c0f0650 0x7f348c0cfb20 secure :-1 s=READY pgs=2894 cs=0 l=1 rev1=1 rx=0x7f3484006f00 tx=0x7f3484009f10).ready entity=mon.0 client_cookie=69f37a4937f66906 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:56.142 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:56.124 7f348affd700 1 -- 172.21.2.102:0/1498851640 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f348400b3b0 con 0x7f348c0f0650 2021-02-10T11:36:56.142 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:56.124 7f348affd700 1 -- 172.21.2.102:0/1498851640 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f348400b3b0 con 0x7f348c0f0650 2021-02-10T11:36:56.142 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:56.124 7f348affd700 1 -- 172.21.2.102:0/1498851640 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f348400b730 con 0x7f348c0f0650 2021-02-10T11:36:56.142 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:56.124 7f3492506700 1 -- 172.21.2.102:0/1498851640 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f348c0f0650 msgr2=0x7f348c0cfb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:56.142 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:56.124 7f3492506700 1 --2- 172.21.2.102:0/1498851640 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f348c0f0650 0x7f348c0cfb20 secure :-1 s=READY pgs=2894 cs=0 l=1 rev1=1 rx=0x7f3484006f00 tx=0x7f3484009f10).stop 2021-02-10T11:36:56.142 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:56.124 7f3492506700 1 -- 172.21.2.102:0/1498851640 shutdown_connections 2021-02-10T11:36:56.142 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:56.124 7f3492506700 1 --2- 172.21.2.102:0/1498851640 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f348c0f0650 0x7f348c0cfb20 unknown :-1 s=CLOSED pgs=2894 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:56.142 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:56.124 7f3492506700 1 --2- 172.21.2.102:0/1498851640 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f348c0d1870 0x7f348c1b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:56.142 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:56.124 7f3492506700 1 -- 172.21.2.102:0/1498851640 >> 172.21.2.102:0/1498851640 conn(0x7f348c0ce450 msgr2=0x7f348c0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:56.142 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:56.124 7f3492506700 1 -- 172.21.2.102:0/1498851640 shutdown_connections 2021-02-10T11:36:56.142 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:56.124 7f3492506700 1 -- 172.21.2.102:0/1498851640 wait complete. 2021-02-10T11:36:56.144 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:56.139 7f3492506700 1 Processor -- start 2021-02-10T11:36:56.144 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:56.139 7f3492506700 1 -- start start 2021-02-10T11:36:56.144 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:56.139 7f3492506700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f348c0b0ad0 0x7f348c091f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:56.144 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:56.139 7f3492506700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f348c1a8850 0x7f348c1a8c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:56.144 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:56.139 7f3492506700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f348c1a6fa0 0x7f348c1a73a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:56.144 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:56.139 7f3492506700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f348c1abfe0 con 0x7f348c0b0ad0 2021-02-10T11:36:56.144 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:56.139 7f3492506700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f348c1a9c40 con 0x7f348c1a8850 2021-02-10T11:36:56.144 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:56.139 7f3492506700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f348c1a9730 con 0x7f348c1a6fa0 2021-02-10T11:36:56.145 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:56.139 7f348bfff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f348c0b0ad0 0x7f348c091f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:56.145 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:56.139 7f348bfff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f348c0b0ad0 0x7f348c091f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:38308/0 (socket says 172.21.2.102:38308) 2021-02-10T11:36:56.145 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:56.139 7f348bfff700 1 -- 172.21.2.102:0/3152430241 learned_addr learned my addr 172.21.2.102:0/3152430241 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:36:56.145 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:56.140 7f348bfff700 1 -- 172.21.2.102:0/3152430241 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f348c1a6fa0 msgr2=0x7f348c1a73a0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:36:56.145 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:56.140 7f348bfff700 1 --2- 172.21.2.102:0/3152430241 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f348c1a6fa0 0x7f348c1a73a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:56.145 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:56.140 7f348bfff700 1 -- 172.21.2.102:0/3152430241 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f348c1a8850 msgr2=0x7f348c1a8c50 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:36:56.145 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:56.140 7f348bfff700 1 --2- 172.21.2.102:0/3152430241 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f348c1a8850 0x7f348c1a8c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:56.145 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:56.140 7f348bfff700 1 -- 172.21.2.102:0/3152430241 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f348c1a9270 con 0x7f348c0b0ad0 2021-02-10T11:36:56.145 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:56.140 7f348bfff700 1 --2- 172.21.2.102:0/3152430241 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f348c0b0ad0 0x7f348c091f30 secure :-1 s=READY pgs=2895 cs=0 l=1 rev1=1 rx=0x7f347c00b5c0 tx=0x7f347c00b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:56.145 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:56.140 7f3488ff9700 1 -- 172.21.2.102:0/3152430241 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f347c00bd50 con 0x7f348c0b0ad0 2021-02-10T11:36:56.145 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:56.140 7f3488ff9700 1 -- 172.21.2.102:0/3152430241 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f347c006030 con 0x7f348c0b0ad0 2021-02-10T11:36:56.145 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:56.140 7f3492506700 1 -- 172.21.2.102:0/3152430241 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f348c1a9270 con 0x7f348c0b0ad0 2021-02-10T11:36:56.146 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:56.140 7f3488ff9700 1 -- 172.21.2.102:0/3152430241 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f347c0054d0 con 0x7f348c0b0ad0 2021-02-10T11:36:56.146 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:56.140 7f3492506700 1 -- 172.21.2.102:0/3152430241 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f348c1a7cb0 con 0x7f348c0b0ad0 2021-02-10T11:36:56.146 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:56.141 7f3488ff9700 1 -- 172.21.2.102:0/3152430241 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f347c006210 con 0x7f348c0b0ad0 2021-02-10T11:36:56.146 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:56.141 7f3488ff9700 1 --2- 172.21.2.102:0/3152430241 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f34680186c0 0x7f346801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:56.146 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:56.141 7f3488ff9700 1 -- 172.21.2.102:0/3152430241 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(162..162 src has 1..162) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f347c032440 con 0x7f348c0b0ad0 2021-02-10T11:36:56.146 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:56.141 7f348b7fe700 1 --2- 172.21.2.102:0/3152430241 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f34680186c0 0x7f346801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:56.146 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:56.141 7f348b7fe700 1 --2- 172.21.2.102:0/3152430241 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f34680186c0 0x7f346801abc0 crc :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:56.174 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:56.172 7f3492506700 1 -- 172.21.2.102:0/3152430241 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f348c0ce450 con 0x7f348c0b0ad0 2021-02-10T11:36:56.177 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:56.175 7f3488ff9700 1 -- 172.21.2.102:0/3152430241 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f347c000940 con 0x7f348c0b0ad0 2021-02-10T11:36:56.306 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:56.304 7f3492506700 1 -- 172.21.2.102:0/3152430241 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f348c0ce450 con 0x7f348c0b0ad0 2021-02-10T11:36:56.307 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:56.304 7f3488ff9700 1 -- 172.21.2.102:0/3152430241 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 141 v141) v1 ==== 95+0+2108 (secure 0 0 0) 0x7f347c006210 con 0x7f348c0b0ad0 2021-02-10T11:36:56.307 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:36:56.307 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":141,"default_fscid":17,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":12405,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.102:6835/4152930669","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6834","nonce":4152930669},{"type":"v1","addr":"172.21.2.102:6835","nonce":4152930669}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":137}],"filesystems":[{"mdsmap":{"epoch":141,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:36:46.331874","modified":"2021-02-10 11:36:48.340104","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7130},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7130":{"gid":7130,"name":"b","rank":0,"incarnation":138,"state":"up:active","state_seq":3,"addr":"172.21.2.113:6833/2966977555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6832","nonce":2966977555},{"type":"v1","addr":"172.21.2.113:6833","nonce":2966977555}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[35],"metadata_pool":34,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":17}]} 2021-02-10T11:36:56.308 INFO:teuthology.orchestra.run.gibba002.stderr:dumped fsmap epoch 141 2021-02-10T11:36:56.309 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:56.306 7f3492506700 1 -- 172.21.2.102:0/3152430241 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f34680186c0 msgr2=0x7f346801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:56.309 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:56.306 7f3492506700 1 --2- 172.21.2.102:0/3152430241 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f34680186c0 0x7f346801abc0 crc :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:56.309 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:56.306 7f3492506700 1 -- 172.21.2.102:0/3152430241 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f348c0b0ad0 msgr2=0x7f348c091f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:56.309 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:56.306 7f3492506700 1 --2- 172.21.2.102:0/3152430241 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f348c0b0ad0 0x7f348c091f30 secure :-1 s=READY pgs=2895 cs=0 l=1 rev1=1 rx=0x7f347c00b5c0 tx=0x7f347c00b960).stop 2021-02-10T11:36:56.309 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:56.307 7f3492506700 1 -- 172.21.2.102:0/3152430241 shutdown_connections 2021-02-10T11:36:56.309 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:56.307 7f3492506700 1 --2- 172.21.2.102:0/3152430241 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f348c1a6fa0 0x7f348c1a73a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:56.309 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:56.307 7f3492506700 1 --2- 172.21.2.102:0/3152430241 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f34680186c0 0x7f346801abc0 unknown :-1 s=CLOSED pgs=1272 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:56.310 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:56.307 7f3492506700 1 --2- 172.21.2.102:0/3152430241 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f348c1a8850 0x7f348c1a8c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:56.310 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:56.307 7f3492506700 1 --2- 172.21.2.102:0/3152430241 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f348c0b0ad0 0x7f348c091f30 unknown :-1 s=CLOSED pgs=2895 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:56.310 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:56.307 7f3492506700 1 -- 172.21.2.102:0/3152430241 >> 172.21.2.102:0/3152430241 conn(0x7f348c19f8f0 msgr2=0x7f348c1a5f80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:56.310 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:56.307 7f3492506700 1 -- 172.21.2.102:0/3152430241 shutdown_connections 2021-02-10T11:36:56.310 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:36:56.307 7f3492506700 1 -- 172.21.2.102:0/3152430241 wait complete. 2021-02-10T11:36:56.519 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 141, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:36:46.331874', 'modified': '2021-02-10 11:36:48.340104', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7130}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7130': {'gid': 7130, 'name': 'b', 'rank': 0, 'incarnation': 138, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.113:6833/2966977555', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.113:6832', 'nonce': 2966977555}, {'type': 'v1', 'addr': '172.21.2.113:6833', 'nonce': 2966977555}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [35], 'metadata_pool': 34, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:36:56.519 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:36:56.519 DEBUG:teuthology.orchestra.run.gibba013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:36:56.919 INFO:teuthology.orchestra.run.gibba013.stdout:{ 2021-02-10T11:36:56.919 INFO:teuthology.orchestra.run.gibba013.stdout: "cluster_fsid": "68db30ea-dbce-4aef-a6df-f01c6fea0513", 2021-02-10T11:36:56.919 INFO:teuthology.orchestra.run.gibba013.stdout: "whoami": 0, 2021-02-10T11:36:56.919 INFO:teuthology.orchestra.run.gibba013.stdout: "id": 7130, 2021-02-10T11:36:56.919 INFO:teuthology.orchestra.run.gibba013.stdout: "want_state": "up:active", 2021-02-10T11:36:56.919 INFO:teuthology.orchestra.run.gibba013.stdout: "state": "up:active", 2021-02-10T11:36:56.919 INFO:teuthology.orchestra.run.gibba013.stdout: "rank_uptime": 10.583727487000001, 2021-02-10T11:36:56.919 INFO:teuthology.orchestra.run.gibba013.stdout: "mdsmap_epoch": 141, 2021-02-10T11:36:56.919 INFO:teuthology.orchestra.run.gibba013.stdout: "osdmap_epoch": 162, 2021-02-10T11:36:56.919 INFO:teuthology.orchestra.run.gibba013.stdout: "osdmap_epoch_barrier": 162, 2021-02-10T11:36:56.919 INFO:teuthology.orchestra.run.gibba013.stdout: "uptime": 16.248198598999998 2021-02-10T11:36:56.920 INFO:teuthology.orchestra.run.gibba013.stdout:} 2021-02-10T11:36:56.923 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "68db30ea-dbce-4aef-a6df-f01c6fea0513", "whoami": 0, "id": 7130, "want_state": "up:active", "state": "up:active", "rank_uptime": 10.583727487000001, "mdsmap_epoch": 141, "osdmap_epoch": 162, "osdmap_epoch_barrier": 162, "uptime": 16.248198598999998 } 2021-02-10T11:36:56.923 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2021-02-10T11:36:56.923 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2021-02-10T11:36:56.923 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@gibba002.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2021-02-10T11:36:56.923 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:36:56.923 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec mkdir -p /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:36:57.260 DEBUG:teuthology.orchestra.run.gibba002:> sudo modprobe fuse 2021-02-10T11:36:57.606 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:36:57.607 DEBUG:teuthology.orchestra.run.gibba002:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:36:57.949 INFO:teuthology.orchestra.run.gibba002.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:36:57.949 INFO:teuthology.orchestra.run.gibba002.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:36:57.950 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:36:57.950 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:36:57.951 DEBUG:teuthology.orchestra.run.gibba002:> ls /sys/fs/fuse/connections 2021-02-10T11:36:58.285 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2021-02-10T11:36:58.285 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:36:58.288 DEBUG:teuthology.orchestra.run.gibba002:> sudo modprobe fuse 2021-02-10T11:36:58.631 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:36:58.632 DEBUG:teuthology.orchestra.run.gibba002:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:36:58.699 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba002.stderr:ceph-fuse[204511]: starting ceph client 2021-02-10T11:36:58.699 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba002.stderr:2021-02-10 11:36:58.697 7f508a83cf80 -1 init, newargv = 0x56528be36970 newargc=9 2021-02-10T11:36:58.704 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba002.stderr:ceph-fuse[204511]: starting fuse 2021-02-10T11:36:58.977 INFO:teuthology.orchestra.run.gibba002.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:36:58.977 INFO:teuthology.orchestra.run.gibba002.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:36:58.978 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:36:58.978 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:36:58.978 DEBUG:teuthology.orchestra.run.gibba002:> ls /sys/fs/fuse/connections 2021-02-10T11:36:59.314 INFO:teuthology.orchestra.run.gibba002.stdout:43 2021-02-10T11:36:59.314 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2021-02-10T11:36:59.314 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:36:59.315 DEBUG:teuthology.orchestra.run.gibba002:> sudo python3 -c ' 2021-02-10T11:36:59.315 DEBUG:teuthology.orchestra.run.gibba002:> import glob 2021-02-10T11:36:59.315 DEBUG:teuthology.orchestra.run.gibba002:> import re 2021-02-10T11:36:59.315 DEBUG:teuthology.orchestra.run.gibba002:> import os 2021-02-10T11:36:59.315 DEBUG:teuthology.orchestra.run.gibba002:> import subprocess 2021-02-10T11:36:59.315 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:36:59.316 DEBUG:teuthology.orchestra.run.gibba002:> def find_socket(client_name): 2021-02-10T11:36:59.316 DEBUG:teuthology.orchestra.run.gibba002:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2021-02-10T11:36:59.316 DEBUG:teuthology.orchestra.run.gibba002:> files = glob.glob(asok_path) 2021-02-10T11:36:59.316 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:36:59.316 DEBUG:teuthology.orchestra.run.gibba002:> # Given a non-glob path, it better be there 2021-02-10T11:36:59.316 DEBUG:teuthology.orchestra.run.gibba002:> if "*" not in asok_path: 2021-02-10T11:36:59.317 DEBUG:teuthology.orchestra.run.gibba002:> assert(len(files) == 1) 2021-02-10T11:36:59.317 DEBUG:teuthology.orchestra.run.gibba002:> return files[0] 2021-02-10T11:36:59.317 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:36:59.317 DEBUG:teuthology.orchestra.run.gibba002:> for f in files: 2021-02-10T11:36:59.317 DEBUG:teuthology.orchestra.run.gibba002:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2021-02-10T11:36:59.317 DEBUG:teuthology.orchestra.run.gibba002:> if os.path.exists("/proc/{0}".format(pid)): 2021-02-10T11:36:59.317 DEBUG:teuthology.orchestra.run.gibba002:> return f 2021-02-10T11:36:59.318 DEBUG:teuthology.orchestra.run.gibba002:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2021-02-10T11:36:59.318 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:36:59.318 DEBUG:teuthology.orchestra.run.gibba002:> print(find_socket("client.0")) 2021-02-10T11:36:59.318 DEBUG:teuthology.orchestra.run.gibba002:> ' 2021-02-10T11:36:59.680 INFO:teuthology.orchestra.run.gibba002.stdout:/var/run/ceph/ceph-client.0.204511.asok 2021-02-10T11:36:59.683 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.204511.asok 2021-02-10T11:36:59.683 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:36:59.683 DEBUG:teuthology.orchestra.run.gibba002:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.204511.asok status 2021-02-10T11:37:00.075 INFO:teuthology.orchestra.run.gibba002.stdout:{ 2021-02-10T11:37:00.076 INFO:teuthology.orchestra.run.gibba002.stdout: "metadata": { 2021-02-10T11:37:00.076 INFO:teuthology.orchestra.run.gibba002.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:37:00.076 INFO:teuthology.orchestra.run.gibba002.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:37:00.076 INFO:teuthology.orchestra.run.gibba002.stdout: "entity_id": "0", 2021-02-10T11:37:00.076 INFO:teuthology.orchestra.run.gibba002.stdout: "hostname": "gibba002", 2021-02-10T11:37:00.076 INFO:teuthology.orchestra.run.gibba002.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2021-02-10T11:37:00.076 INFO:teuthology.orchestra.run.gibba002.stdout: "pid": "204511", 2021-02-10T11:37:00.076 INFO:teuthology.orchestra.run.gibba002.stdout: "root": "/" 2021-02-10T11:37:00.076 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:37:00.076 INFO:teuthology.orchestra.run.gibba002.stdout: "dentry_count": 0, 2021-02-10T11:37:00.076 INFO:teuthology.orchestra.run.gibba002.stdout: "dentry_pinned_count": 0, 2021-02-10T11:37:00.077 INFO:teuthology.orchestra.run.gibba002.stdout: "id": 12549, 2021-02-10T11:37:00.077 INFO:teuthology.orchestra.run.gibba002.stdout: "inst": { 2021-02-10T11:37:00.077 INFO:teuthology.orchestra.run.gibba002.stdout: "name": { 2021-02-10T11:37:00.077 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "client", 2021-02-10T11:37:00.077 INFO:teuthology.orchestra.run.gibba002.stdout: "num": 12549 2021-02-10T11:37:00.077 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:37:00.077 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": { 2021-02-10T11:37:00.077 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:37:00.077 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:0", 2021-02-10T11:37:00.077 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 3716918088 2021-02-10T11:37:00.077 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:37:00.077 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:37:00.077 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": { 2021-02-10T11:37:00.078 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:37:00.078 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:0", 2021-02-10T11:37:00.078 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 3716918088 2021-02-10T11:37:00.078 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:37:00.078 INFO:teuthology.orchestra.run.gibba002.stdout: "inst_str": "client.12549 172.21.2.102:0/3716918088", 2021-02-10T11:37:00.078 INFO:teuthology.orchestra.run.gibba002.stdout: "addr_str": "172.21.2.102:0/3716918088", 2021-02-10T11:37:00.078 INFO:teuthology.orchestra.run.gibba002.stdout: "inode_count": 1, 2021-02-10T11:37:00.078 INFO:teuthology.orchestra.run.gibba002.stdout: "mds_epoch": 141, 2021-02-10T11:37:00.078 INFO:teuthology.orchestra.run.gibba002.stdout: "osd_epoch": 162, 2021-02-10T11:37:00.078 INFO:teuthology.orchestra.run.gibba002.stdout: "osd_epoch_barrier": 0, 2021-02-10T11:37:00.078 INFO:teuthology.orchestra.run.gibba002.stdout: "blacklisted": false 2021-02-10T11:37:00.078 INFO:teuthology.orchestra.run.gibba002.stdout:} 2021-02-10T11:37:00.080 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:37:00.080 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:37:00.080 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:37:00.417 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:37:00.417 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:37:00.418 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:37:00.418 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:37:00.766 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:37:01.196 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:01.177 7f05dc19c700 1 Processor -- start 2021-02-10T11:37:01.196 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:01.177 7f05dc19c700 1 -- start start 2021-02-10T11:37:01.196 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:01.178 7f05dc19c700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f05d40d1870 0x7f05d41b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:01.196 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:01.178 7f05dc19c700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f05d41b0a10 0x7f05d40cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:01.196 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:01.178 7f05dc19c700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f05d40d1c70 0x7f05d40cf950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:01.196 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:01.178 7f05dc19c700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05d40fe230 con 0x7f05d40d1c70 2021-02-10T11:37:01.196 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:01.178 7f05dc19c700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05d40e5ed0 con 0x7f05d41b0a10 2021-02-10T11:37:01.196 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:01.178 7f05dc19c700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05d40f0b10 con 0x7f05d40d1870 2021-02-10T11:37:01.196 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:01.178 7f05da7b9700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f05d40d1c70 0x7f05d40cf950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:01.197 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:01.178 7f05da7b9700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f05d40d1c70 0x7f05d40cf950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:38332/0 (socket says 172.21.2.102:38332) 2021-02-10T11:37:01.197 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:01.178 7f05da7b9700 1 -- 172.21.2.102:0/3024301590 learned_addr learned my addr 172.21.2.102:0/3024301590 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:37:01.197 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:01.178 7f05da7b9700 1 -- 172.21.2.102:0/3024301590 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f05d40d1870 msgr2=0x7f05d41b03f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:37:01.197 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:01.178 7f05da7b9700 1 --2- 172.21.2.102:0/3024301590 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f05d40d1870 0x7f05d41b03f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:01.197 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:01.178 7f05da7b9700 1 -- 172.21.2.102:0/3024301590 >> v2:172.21.2.113:3300/0 conn(0x7f05d41b0a10 msgr2=0x7f05d40cf330 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:37:01.197 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:01.178 7f05da7b9700 1 --2- 172.21.2.102:0/3024301590 >> v2:172.21.2.113:3300/0 conn(0x7f05d41b0a10 0x7f05d40cf330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:01.197 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:01.178 7f05da7b9700 1 -- 172.21.2.102:0/3024301590 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05d40cff70 con 0x7f05d40d1c70 2021-02-10T11:37:01.197 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:01.179 7f05da7b9700 1 --2- 172.21.2.102:0/3024301590 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f05d40d1c70 0x7f05d40cf950 secure :-1 s=READY pgs=2899 cs=0 l=1 rev1=1 rx=0x7f05cc006f00 tx=0x7f05cc009f10).ready entity=mon.0 client_cookie=c815380031cd8db9 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:01.197 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:01.179 7f05d8fb6700 1 -- 172.21.2.102:0/3024301590 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f05cc00b3b0 con 0x7f05d40d1c70 2021-02-10T11:37:01.197 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:01.179 7f05d8fb6700 1 -- 172.21.2.102:0/3024301590 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05cc00b730 con 0x7f05d40d1c70 2021-02-10T11:37:01.197 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:01.179 7f05d8fb6700 1 -- 172.21.2.102:0/3024301590 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f05cc00bc50 con 0x7f05d40d1c70 2021-02-10T11:37:01.197 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:01.179 7f05dc19c700 1 -- 172.21.2.102:0/3024301590 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f05d40d1c70 msgr2=0x7f05d40cf950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:01.197 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:01.179 7f05dc19c700 1 --2- 172.21.2.102:0/3024301590 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f05d40d1c70 0x7f05d40cf950 secure :-1 s=READY pgs=2899 cs=0 l=1 rev1=1 rx=0x7f05cc006f00 tx=0x7f05cc009f10).stop 2021-02-10T11:37:01.198 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:01.179 7f05dc19c700 1 -- 172.21.2.102:0/3024301590 shutdown_connections 2021-02-10T11:37:01.198 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:01.179 7f05dc19c700 1 --2- 172.21.2.102:0/3024301590 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f05d40d1c70 0x7f05d40cf950 unknown :-1 s=CLOSED pgs=2899 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:01.198 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:01.179 7f05dc19c700 1 --2- 172.21.2.102:0/3024301590 >> v2:172.21.2.113:3300/0 conn(0x7f05d41b0a10 0x7f05d40cf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:01.198 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:01.179 7f05dc19c700 1 --2- 172.21.2.102:0/3024301590 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f05d40d1870 0x7f05d41b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:01.198 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:01.179 7f05dc19c700 1 -- 172.21.2.102:0/3024301590 >> 172.21.2.102:0/3024301590 conn(0x7f05d40ce450 msgr2=0x7f05d40d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:01.198 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:01.180 7f05dc19c700 1 -- 172.21.2.102:0/3024301590 shutdown_connections 2021-02-10T11:37:01.198 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:01.180 7f05dc19c700 1 -- 172.21.2.102:0/3024301590 wait complete. 2021-02-10T11:37:01.199 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:01.194 7f05dc19c700 1 Processor -- start 2021-02-10T11:37:01.200 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:01.194 7f05dc19c700 1 -- start start 2021-02-10T11:37:01.200 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:01.194 7f05dc19c700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f05d41b7cf0 0x7f05d419cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:01.200 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:01.194 7f05dc19c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f05d41b80f0 0x7f05d41b84f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:01.200 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:01.194 7f05dc19c700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f05d419da20 0x7f05d419de20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:01.200 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:01.194 7f05dc19c700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05d40a77c0 con 0x7f05d41b80f0 2021-02-10T11:37:01.200 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:01.194 7f05dc19c700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05d419d4a0 con 0x7f05d41b7cf0 2021-02-10T11:37:01.200 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:01.194 7f05dc19c700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05d40954f0 con 0x7f05d419da20 2021-02-10T11:37:01.200 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:01.195 7f05d97b7700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f05d41b80f0 0x7f05d41b84f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:01.200 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:01.195 7f05d97b7700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f05d41b80f0 0x7f05d41b84f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:38336/0 (socket says 172.21.2.102:38336) 2021-02-10T11:37:01.200 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:01.195 7f05d97b7700 1 -- 172.21.2.102:0/2228366534 learned_addr learned my addr 172.21.2.102:0/2228366534 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:37:01.200 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:01.195 7f05d97b7700 1 -- 172.21.2.102:0/2228366534 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f05d419da20 msgr2=0x7f05d419de20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:01.200 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:01.195 7f05d97b7700 1 --2- 172.21.2.102:0/2228366534 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f05d419da20 0x7f05d419de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:01.200 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:01.195 7f05d97b7700 1 -- 172.21.2.102:0/2228366534 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f05d41b7cf0 msgr2=0x7f05d419cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:01.201 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:01.195 7f05d97b7700 1 --2- 172.21.2.102:0/2228366534 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f05d41b7cf0 0x7f05d419cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:01.201 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:01.195 7f05d97b7700 1 -- 172.21.2.102:0/2228366534 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05d41b8d60 con 0x7f05d41b80f0 2021-02-10T11:37:01.201 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:01.195 7f05d97b7700 1 --2- 172.21.2.102:0/2228366534 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f05d41b80f0 0x7f05d41b84f0 secure :-1 s=READY pgs=2900 cs=0 l=1 rev1=1 rx=0x7f05c400b5c0 tx=0x7f05c400b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:01.201 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:01.195 7f05ca7fc700 1 -- 172.21.2.102:0/2228366534 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f05c400bd50 con 0x7f05d41b80f0 2021-02-10T11:37:01.201 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:01.195 7f05ca7fc700 1 -- 172.21.2.102:0/2228366534 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05c4006030 con 0x7f05d41b80f0 2021-02-10T11:37:01.201 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:01.195 7f05ca7fc700 1 -- 172.21.2.102:0/2228366534 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f05c40054d0 con 0x7f05d41b80f0 2021-02-10T11:37:01.201 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:01.195 7f05dc19c700 1 -- 172.21.2.102:0/2228366534 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05d41b8d60 con 0x7f05d41b80f0 2021-02-10T11:37:01.201 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:01.196 7f05dc19c700 1 -- 172.21.2.102:0/2228366534 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05d41b8d60 con 0x7f05d41b80f0 2021-02-10T11:37:01.201 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:01.196 7f05ca7fc700 1 -- 172.21.2.102:0/2228366534 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f05c4006210 con 0x7f05d41b80f0 2021-02-10T11:37:01.201 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:01.196 7f05ca7fc700 1 --2- 172.21.2.102:0/2228366534 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f05bc0186c0 0x7f05bc01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:01.201 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:01.196 7f05ca7fc700 1 -- 172.21.2.102:0/2228366534 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(162..162 src has 1..162) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f05c4032650 con 0x7f05d41b80f0 2021-02-10T11:37:01.201 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:01.196 7f05d9fb8700 1 --2- 172.21.2.102:0/2228366534 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f05bc0186c0 0x7f05bc01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:01.201 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:01.197 7f05d9fb8700 1 --2- 172.21.2.102:0/2228366534 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f05bc0186c0 0x7f05bc01abc0 crc :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:01.230 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:01.228 7f05dc19c700 1 -- 172.21.2.102:0/2228366534 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05d40ebc00 con 0x7f05d41b80f0 2021-02-10T11:37:01.232 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:01.230 7f05ca7fc700 1 -- 172.21.2.102:0/2228366534 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f05c4006b80 con 0x7f05d41b80f0 2021-02-10T11:37:01.360 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:01.358 7f05dc19c700 1 -- 172.21.2.102:0/2228366534 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f05d40f0b10 con 0x7f05d41b80f0 2021-02-10T11:37:01.360 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:01.358 7f05ca7fc700 1 -- 172.21.2.102:0/2228366534 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v162) v1 ==== 74+0+14007 (secure 0 0 0) 0x7f05c4006030 con 0x7f05d41b80f0 2021-02-10T11:37:01.361 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:37:01.361 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":162,"fsid":"68db30ea-dbce-4aef-a6df-f01c6fea0513","created":"2021-02-10 11:03:06.331557","modified":"2021-02-10 11:36:47.333734","last_up_change":"2021-02-10 11:03:32.690562","last_in_change":"2021-02-10 11:03:32.690562","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":35,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:03:37.322519","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":34,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:36:44.297293","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"161","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":35,"pool_name":"cephfs_data","create_time":"2021-02-10 11:36:45.321943","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"162","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"be691b4d-a039-408a-af80-9d53f02b6f1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6818","nonce":36088},{"type":"v1","addr":"172.21.2.102:6820","nonce":36088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6822","nonce":36088},{"type":"v1","addr":"172.21.2.102:6824","nonce":36088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6830","nonce":36088},{"type":"v1","addr":"172.21.2.102:6832","nonce":36088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6826","nonce":36088},{"type":"v1","addr":"172.21.2.102:6828","nonce":36088}]},"public_addr":"172.21.2.102:6820/36088","cluster_addr":"172.21.2.102:6824/36088","heartbeat_back_addr":"172.21.2.102:6832/36088","heartbeat_front_addr":"172.21.2.102:6828/36088","state":["exists","up"]},{"osd":1,"uuid":"64e3ccc8-9c69-4d7c-865e-af8e83822635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6819","nonce":36087},{"type":"v1","addr":"172.21.2.102:6821","nonce":36087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6823","nonce":36087},{"type":"v1","addr":"172.21.2.102:6825","nonce":36087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6831","nonce":36087},{"type":"v1","addr":"172.21.2.102:6833","nonce":36087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6827","nonce":36087},{"type":"v1","addr":"172.21.2.102:6829","nonce":36087}]},"public_addr":"172.21.2.102:6821/36087","cluster_addr":"172.21.2.102:6825/36087","heartbeat_back_addr":"172.21.2.102:6833/36087","heartbeat_front_addr":"172.21.2.102:6829/36087","state":["exists","up"]},{"osd":2,"uuid":"153700af-747f-481f-bdf4-d6072d3a842f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6810","nonce":36089},{"type":"v1","addr":"172.21.2.102:6811","nonce":36089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6812","nonce":36089},{"type":"v1","addr":"172.21.2.102:6813","nonce":36089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6816","nonce":36089},{"type":"v1","addr":"172.21.2.102:6817","nonce":36089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6814","nonce":36089},{"type":"v1","addr":"172.21.2.102:6815","nonce":36089}]},"public_addr":"172.21.2.102:6811/36089","cluster_addr":"172.21.2.102:6813/36089","heartbeat_back_addr":"172.21.2.102:6817/36089","heartbeat_front_addr":"172.21.2.102:6815/36089","state":["exists","up"]},{"osd":3,"uuid":"cbc678f4-83a9-45e6-8830-2f7c232b1ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6802","nonce":36090},{"type":"v1","addr":"172.21.2.102:6803","nonce":36090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6804","nonce":36090},{"type":"v1","addr":"172.21.2.102:6805","nonce":36090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6808","nonce":36090},{"type":"v1","addr":"172.21.2.102:6809","nonce":36090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6806","nonce":36090},{"type":"v1","addr":"172.21.2.102:6807","nonce":36090}]},"public_addr":"172.21.2.102:6803/36090","cluster_addr":"172.21.2.102:6805/36090","heartbeat_back_addr":"172.21.2.102:6809/36090","heartbeat_front_addr":"172.21.2.102:6807/36090","state":["exists","up"]},{"osd":4,"uuid":"c01e4008-b258-4f10-afd2-9e7f37214351","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6808","nonce":36368},{"type":"v1","addr":"172.21.2.113:6811","nonce":36368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6814","nonce":36368},{"type":"v1","addr":"172.21.2.113:6818","nonce":36368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6826","nonce":36368},{"type":"v1","addr":"172.21.2.113:6828","nonce":36368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6820","nonce":36368},{"type":"v1","addr":"172.21.2.113:6824","nonce":36368}]},"public_addr":"172.21.2.113:6811/36368","cluster_addr":"172.21.2.113:6818/36368","heartbeat_back_addr":"172.21.2.113:6828/36368","heartbeat_front_addr":"172.21.2.113:6824/36368","state":["exists","up"]},{"osd":5,"uuid":"551f6007-daf8-4a98-8ee0-c2b16bcccb5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6802","nonce":36366},{"type":"v1","addr":"172.21.2.113:6805","nonce":36366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6807","nonce":36366},{"type":"v1","addr":"172.21.2.113:6810","nonce":36366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6819","nonce":36366},{"type":"v1","addr":"172.21.2.113:6822","nonce":36366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6812","nonce":36366},{"type":"v1","addr":"172.21.2.113:6815","nonce":36366}]},"public_addr":"172.21.2.113:6805/36366","cluster_addr":"172.21.2.113:6810/36366","heartbeat_back_addr":"172.21.2.113:6822/36366","heartbeat_front_addr":"172.21.2.113:6815/36366","state":["exists","up"]},{"osd":6,"uuid":"bf67c040-0e50-48be-b102-c3943bc9ebef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6800","nonce":36367},{"type":"v1","addr":"172.21.2.113:6801","nonce":36367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6803","nonce":36367},{"type":"v1","addr":"172.21.2.113:6804","nonce":36367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6813","nonce":36367},{"type":"v1","addr":"172.21.2.113:6816","nonce":36367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6806","nonce":36367},{"type":"v1","addr":"172.21.2.113:6809","nonce":36367}]},"public_addr":"172.21.2.113:6801/36367","cluster_addr":"172.21.2.113:6804/36367","heartbeat_back_addr":"172.21.2.113:6816/36367","heartbeat_front_addr":"172.21.2.113:6809/36367","state":["exists","up"]},{"osd":7,"uuid":"f5461f80-018d-428f-be89-2eb2bcbe7bb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6817","nonce":36365},{"type":"v1","addr":"172.21.2.113:6821","nonce":36365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6823","nonce":36365},{"type":"v1","addr":"172.21.2.113:6825","nonce":36365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6830","nonce":36365},{"type":"v1","addr":"172.21.2.113:6831","nonce":36365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6827","nonce":36365},{"type":"v1","addr":"172.21.2.113:6829","nonce":36365}]},"public_addr":"172.21.2.113:6821/36365","cluster_addr":"172.21.2.113:6825/36365","heartbeat_back_addr":"172.21.2.113:6831/36365","heartbeat_front_addr":"172.21.2.113:6829/36365","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:37:01.362 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:01.360 7f05dc19c700 1 -- 172.21.2.102:0/2228366534 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f05bc0186c0 msgr2=0x7f05bc01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:01.362 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:01.360 7f05dc19c700 1 --2- 172.21.2.102:0/2228366534 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f05bc0186c0 0x7f05bc01abc0 crc :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:01.362 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:01.360 7f05dc19c700 1 -- 172.21.2.102:0/2228366534 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f05d41b80f0 msgr2=0x7f05d41b84f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:01.363 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:01.360 7f05dc19c700 1 --2- 172.21.2.102:0/2228366534 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f05d41b80f0 0x7f05d41b84f0 secure :-1 s=READY pgs=2900 cs=0 l=1 rev1=1 rx=0x7f05c400b5c0 tx=0x7f05c400b960).stop 2021-02-10T11:37:01.363 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:01.361 7f05dc19c700 1 -- 172.21.2.102:0/2228366534 shutdown_connections 2021-02-10T11:37:01.363 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:01.361 7f05dc19c700 1 --2- 172.21.2.102:0/2228366534 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f05d419da20 0x7f05d419de20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:01.363 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:01.361 7f05dc19c700 1 --2- 172.21.2.102:0/2228366534 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f05d41b80f0 0x7f05d41b84f0 unknown :-1 s=CLOSED pgs=2900 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:01.363 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:01.361 7f05dc19c700 1 --2- 172.21.2.102:0/2228366534 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f05bc0186c0 0x7f05bc01abc0 unknown :-1 s=CLOSED pgs=1273 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:01.363 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:01.361 7f05dc19c700 1 --2- 172.21.2.102:0/2228366534 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f05d41b7cf0 0x7f05d419cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:01.364 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:01.361 7f05dc19c700 1 -- 172.21.2.102:0/2228366534 >> 172.21.2.102:0/2228366534 conn(0x7f05d40ed810 msgr2=0x7f05d4094980 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:01.364 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:01.361 7f05dc19c700 1 -- 172.21.2.102:0/2228366534 shutdown_connections 2021-02-10T11:37:01.364 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:01.361 7f05dc19c700 1 -- 172.21.2.102:0/2228366534 wait complete. 2021-02-10T11:37:01.371 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:03:37.322519', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 34, 'pool_name': 'cephfs_metadata', 'create_time': '2021-02-10 11:36:44.297293', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '161', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 35, 'pool_name': 'cephfs_data', 'create_time': '2021-02-10 11:36:45.321943', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '162', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-02-10T11:37:01.372 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:37:01.805 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:01.787 7f930be85700 1 Processor -- start 2021-02-10T11:37:01.805 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:01.787 7f930be85700 1 -- start start 2021-02-10T11:37:01.805 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:01.787 7f930be85700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f93040d1870 0x7f93041b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:01.805 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:01.787 7f930be85700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f93040cff70 0x7f9304109130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:01.805 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:01.787 7f930be85700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9304109a60 con 0x7f93041b0ae0 2021-02-10T11:37:01.805 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:01.787 7f930be85700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9304109750 con 0x7f93040cff70 2021-02-10T11:37:01.805 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:01.787 7f930be85700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9304088070 con 0x7f93040d1870 2021-02-10T11:37:01.806 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:01.787 7f930a4a2700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f93040cff70 0x7f9304109130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:01.806 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:01.787 7f930a4a2700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f93040cff70 0x7f9304109130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:38344/0 (socket says 172.21.2.102:38344) 2021-02-10T11:37:01.806 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:01.788 7f930a4a2700 1 -- 172.21.2.102:0/1241258646 learned_addr learned my addr 172.21.2.102:0/1241258646 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:37:01.806 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:01.788 7f930a4a2700 1 -- 172.21.2.102:0/1241258646 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f93040d1870 msgr2=0x7f93041b04c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:37:01.806 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:01.788 7f930a4a2700 1 --2- 172.21.2.102:0/1241258646 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f93040d1870 0x7f93041b04c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:01.806 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:01.788 7f930a4a2700 1 -- 172.21.2.102:0/1241258646 >> v1:172.21.2.113:6789/0 conn(0x7f93041b0ae0 legacy=0x7f9304108940 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:37:01.806 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:01.788 7f930a4a2700 1 -- 172.21.2.102:0/1241258646 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9304088260 con 0x7f93040cff70 2021-02-10T11:37:01.806 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:01.788 7f930a4a2700 1 --2- 172.21.2.102:0/1241258646 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f93040cff70 0x7f9304109130 secure :-1 s=READY pgs=2901 cs=0 l=1 rev1=1 rx=0x7f92fc006f00 tx=0x7f92fc009f10).ready entity=mon.0 client_cookie=ac7a8ade107e7c48 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:01.806 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:01.788 7f9308c9f700 1 -- 172.21.2.102:0/1241258646 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f92fc00b3b0 con 0x7f93040cff70 2021-02-10T11:37:01.806 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:01.788 7f9308c9f700 1 -- 172.21.2.102:0/1241258646 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92fc00b630 con 0x7f93040cff70 2021-02-10T11:37:01.806 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:01.788 7f9308c9f700 1 -- 172.21.2.102:0/1241258646 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f92fc00bb50 con 0x7f93040cff70 2021-02-10T11:37:01.806 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:01.788 7f930be85700 1 -- 172.21.2.102:0/1241258646 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f93040cff70 msgr2=0x7f9304109130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:01.807 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:01.788 7f930be85700 1 --2- 172.21.2.102:0/1241258646 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f93040cff70 0x7f9304109130 secure :-1 s=READY pgs=2901 cs=0 l=1 rev1=1 rx=0x7f92fc006f00 tx=0x7f92fc009f10).stop 2021-02-10T11:37:01.807 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:01.789 7f930be85700 1 -- 172.21.2.102:0/1241258646 shutdown_connections 2021-02-10T11:37:01.807 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:01.789 7f930be85700 1 --2- 172.21.2.102:0/1241258646 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f93040cff70 0x7f9304109130 unknown :-1 s=CLOSED pgs=2901 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:01.807 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:01.789 7f930be85700 1 --2- 172.21.2.102:0/1241258646 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f93040d1870 0x7f93041b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:01.807 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:01.789 7f930be85700 1 -- 172.21.2.102:0/1241258646 >> 172.21.2.102:0/1241258646 conn(0x7f93040ce420 msgr2=0x7f93040d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:01.807 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:01.789 7f930be85700 1 -- 172.21.2.102:0/1241258646 shutdown_connections 2021-02-10T11:37:01.807 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:01.789 7f930be85700 1 -- 172.21.2.102:0/1241258646 wait complete. 2021-02-10T11:37:01.808 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:01.803 7f930be85700 1 Processor -- start 2021-02-10T11:37:01.808 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:01.803 7f930be85700 1 -- start start 2021-02-10T11:37:01.809 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:01.804 7f930be85700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f93040c3820 0x7f9304084cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:01.809 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:01.804 7f930be85700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f93040a0290 0x7f93040852d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:01.809 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:01.804 7f930be85700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9304086270 0x7f9304086670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:01.809 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:01.804 7f930be85700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93040c0700 con 0x7f9304086270 2021-02-10T11:37:01.809 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:01.804 7f930be85700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93040858f0 con 0x7f93040c3820 2021-02-10T11:37:01.809 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:01.804 7f930be85700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9304085b10 con 0x7f93040a0290 2021-02-10T11:37:01.809 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:01.804 7f930a4a2700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9304086270 0x7f9304086670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:01.809 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:01.804 7f930a4a2700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9304086270 0x7f9304086670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:38354/0 (socket says 172.21.2.102:38354) 2021-02-10T11:37:01.809 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:01.804 7f930a4a2700 1 -- 172.21.2.102:0/1346450888 learned_addr learned my addr 172.21.2.102:0/1346450888 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:37:01.809 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:01.804 7f930a4a2700 1 -- 172.21.2.102:0/1346450888 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f93040a0290 msgr2=0x7f93040852d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:01.809 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:01.804 7f930a4a2700 1 --2- 172.21.2.102:0/1346450888 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f93040a0290 0x7f93040852d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:01.809 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:01.804 7f930a4a2700 1 -- 172.21.2.102:0/1346450888 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f93040c3820 msgr2=0x7f9304084cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:01.810 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:01.804 7f930a4a2700 1 --2- 172.21.2.102:0/1346450888 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f93040c3820 0x7f9304084cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:01.810 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:01.804 7f930a4a2700 1 -- 172.21.2.102:0/1346450888 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9304086c90 con 0x7f9304086270 2021-02-10T11:37:01.810 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:01.804 7f930a4a2700 1 --2- 172.21.2.102:0/1346450888 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9304086270 0x7f9304086670 secure :-1 s=READY pgs=2902 cs=0 l=1 rev1=1 rx=0x7f92fc009390 tx=0x7f92fc00bd40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:01.810 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:01.805 7f92f67fc700 1 -- 172.21.2.102:0/1346450888 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f92fc003030 con 0x7f9304086270 2021-02-10T11:37:01.810 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:01.805 7f92f67fc700 1 -- 172.21.2.102:0/1346450888 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92fc0068e0 con 0x7f9304086270 2021-02-10T11:37:01.810 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:01.805 7f92f67fc700 1 -- 172.21.2.102:0/1346450888 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f92fc00c570 con 0x7f9304086270 2021-02-10T11:37:01.810 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:01.805 7f930be85700 1 -- 172.21.2.102:0/1346450888 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9304086c90 con 0x7f9304086270 2021-02-10T11:37:01.810 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:01.805 7f930be85700 1 -- 172.21.2.102:0/1346450888 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9304086c90 con 0x7f9304086270 2021-02-10T11:37:01.810 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:01.805 7f92f67fc700 1 -- 172.21.2.102:0/1346450888 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f92fc005820 con 0x7f9304086270 2021-02-10T11:37:01.810 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:01.805 7f92f67fc700 1 --2- 172.21.2.102:0/1346450888 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f92ec018730 0x7f92ec01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:01.810 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:01.805 7f92f67fc700 1 -- 172.21.2.102:0/1346450888 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(162..162 src has 1..162) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f92fc032b20 con 0x7f9304086270 2021-02-10T11:37:01.810 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:01.805 7f9309ca1700 1 --2- 172.21.2.102:0/1346450888 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f92ec018730 0x7f92ec01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:01.810 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:01.806 7f9309ca1700 1 --2- 172.21.2.102:0/1346450888 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f92ec018730 0x7f92ec01ac10 crc :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:01.839 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:01.837 7f930be85700 1 -- 172.21.2.102:0/1346450888 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9304082a70 con 0x7f9304086270 2021-02-10T11:37:01.842 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:01.840 7f92f67fc700 1 -- 172.21.2.102:0/1346450888 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f92fc006c80 con 0x7f9304086270 2021-02-10T11:37:01.969 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:01.967 7f930be85700 1 -- 172.21.2.102:0/1346450888 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f9304082a70 con 0x7f9304086270 2021-02-10T11:37:01.969 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:01.967 7f92f67fc700 1 -- 172.21.2.102:0/1346450888 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v162) v1 ==== 92+0+10 (secure 0 0 0) 0x7f92fc005820 con 0x7f9304086270 2021-02-10T11:37:01.970 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:37:01.972 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:01.970 7f930be85700 1 -- 172.21.2.102:0/1346450888 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f92ec018730 msgr2=0x7f92ec01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:01.972 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:01.970 7f930be85700 1 --2- 172.21.2.102:0/1346450888 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f92ec018730 0x7f92ec01ac10 crc :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:01.972 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:01.970 7f930be85700 1 -- 172.21.2.102:0/1346450888 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9304086270 msgr2=0x7f9304086670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:01.972 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:01.970 7f930be85700 1 --2- 172.21.2.102:0/1346450888 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9304086270 0x7f9304086670 secure :-1 s=READY pgs=2902 cs=0 l=1 rev1=1 rx=0x7f92fc009390 tx=0x7f92fc00bd40).stop 2021-02-10T11:37:01.972 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:01.970 7f930be85700 1 -- 172.21.2.102:0/1346450888 shutdown_connections 2021-02-10T11:37:01.972 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:01.970 7f930be85700 1 --2- 172.21.2.102:0/1346450888 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9304086270 0x7f9304086670 unknown :-1 s=CLOSED pgs=2902 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:01.972 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:01.970 7f930be85700 1 --2- 172.21.2.102:0/1346450888 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f93040a0290 0x7f93040852d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:01.972 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:01.970 7f930be85700 1 --2- 172.21.2.102:0/1346450888 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f92ec018730 0x7f92ec01ac10 unknown :-1 s=CLOSED pgs=1274 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:01.972 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:01.970 7f930be85700 1 --2- 172.21.2.102:0/1346450888 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f93040c3820 0x7f9304084cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:01.972 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:01.970 7f930be85700 1 -- 172.21.2.102:0/1346450888 >> 172.21.2.102:0/1346450888 conn(0x7f93040872c0 msgr2=0x7f9304095700 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:01.972 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:01.970 7f930be85700 1 -- 172.21.2.102:0/1346450888 shutdown_connections 2021-02-10T11:37:01.973 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:01.970 7f930be85700 1 -- 172.21.2.102:0/1346450888 wait complete. 2021-02-10T11:37:01.980 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-02-10T11:37:02.414 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:02.395 7f4e847de700 1 Processor -- start 2021-02-10T11:37:02.415 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:02.395 7f4e847de700 1 -- start start 2021-02-10T11:37:02.415 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:02.395 7f4e847de700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f4e7c0d1c60 0x7f4e7c0cef50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:02.415 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:02.395 7f4e847de700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4e7c0cf570 0x7f4e7c0cf970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:02.415 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:02.395 7f4e847de700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e7c0fe1b0 con 0x7f4e7c0d1860 2021-02-10T11:37:02.415 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:02.395 7f4e847de700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e7c1b0ca0 con 0x7f4e7c0d1c60 2021-02-10T11:37:02.415 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:02.395 7f4e847de700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e7c109a50 con 0x7f4e7c0cf570 2021-02-10T11:37:02.415 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:02.396 7f4e825fa700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f4e7c0d1860 0x7f4e7c1b04b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:58814/0 (socket says 172.21.2.102:58814) 2021-02-10T11:37:02.415 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:02.396 7f4e825fa700 1 -- 172.21.2.102:0/3225855579 learned_addr learned my addr 172.21.2.102:0/3225855579 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:37:02.415 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:02.396 7f4e815f8700 1 -- 172.21.2.102:0/3225855579 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 602296649 0 0) 0x7f4e780010b0 con 0x7f4e7c0d1860 2021-02-10T11:37:02.416 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:02.396 7f4e815f8700 1 -- 172.21.2.102:0/3225855579 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e68002fe0 con 0x7f4e7c0d1860 2021-02-10T11:37:02.416 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:02.396 7f4e82dfb700 1 --2- 172.21.2.102:0/3225855579 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4e7c0cf570 0x7f4e7c0cf970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:02.416 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:02.396 7f4e81df9700 1 --2- 172.21.2.102:0/3225855579 >> v2:172.21.2.113:3300/0 conn(0x7f4e7c0d1c60 0x7f4e7c0cef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:02.416 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:02.396 7f4e81df9700 1 -- 172.21.2.102:0/3225855579 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4e7c0cf570 msgr2=0x7f4e7c0cf970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:37:02.416 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:02.396 7f4e81df9700 1 --2- 172.21.2.102:0/3225855579 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4e7c0cf570 0x7f4e7c0cf970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:02.416 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:02.396 7f4e815f8700 1 -- 172.21.2.102:0/3225855579 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 583495700 0 0) 0x7f4e780010b0 con 0x7f4e7c0d1860 2021-02-10T11:37:02.416 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:02.396 7f4e81df9700 1 -- 172.21.2.102:0/3225855579 >> v1:172.21.2.102:6789/0 conn(0x7f4e7c0d1860 legacy=0x7f4e7c1b04b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:02.416 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:02.396 7f4e81df9700 1 -- 172.21.2.102:0/3225855579 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e7c088070 con 0x7f4e7c0d1c60 2021-02-10T11:37:02.416 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:02.397 7f4e81df9700 1 --2- 172.21.2.102:0/3225855579 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4e7c0d1c60 0x7f4e7c0cef50 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 rx=0x7f4e70006650 tx=0x7f4e70009a80).ready entity=mon.1 client_cookie=44ddc6e964e8982 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:02.416 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:02.397 7f4e815f8700 1 -- 172.21.2.102:0/3225855579 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4e7000c0c0 con 0x7f4e7c0d1c60 2021-02-10T11:37:02.416 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:02.397 7f4e815f8700 1 -- 172.21.2.102:0/3225855579 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e7000c340 con 0x7f4e7c0d1c60 2021-02-10T11:37:02.416 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:02.397 7f4e815f8700 1 -- 172.21.2.102:0/3225855579 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4e7000c860 con 0x7f4e7c0d1c60 2021-02-10T11:37:02.416 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:02.397 7f4e847de700 1 -- 172.21.2.102:0/3225855579 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4e7c0d1c60 msgr2=0x7f4e7c0cef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:02.417 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:02.397 7f4e847de700 1 --2- 172.21.2.102:0/3225855579 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4e7c0d1c60 0x7f4e7c0cef50 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 rx=0x7f4e70006650 tx=0x7f4e70009a80).stop 2021-02-10T11:37:02.417 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:02.398 7f4e847de700 1 -- 172.21.2.102:0/3225855579 shutdown_connections 2021-02-10T11:37:02.417 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:02.398 7f4e847de700 1 --2- 172.21.2.102:0/3225855579 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4e7c0d1c60 0x7f4e7c0cef50 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:02.417 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:02.398 7f4e847de700 1 --2- 172.21.2.102:0/3225855579 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4e7c0cf570 0x7f4e7c0cf970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:02.417 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:02.398 7f4e847de700 1 -- 172.21.2.102:0/3225855579 >> 172.21.2.102:0/3225855579 conn(0x7f4e7c0ce410 msgr2=0x7f4e7c0d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:02.417 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:02.398 7f4e847de700 1 -- 172.21.2.102:0/3225855579 shutdown_connections 2021-02-10T11:37:02.417 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:02.398 7f4e847de700 1 -- 172.21.2.102:0/3225855579 wait complete. 2021-02-10T11:37:02.417 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:02.412 7f4e847de700 1 Processor -- start 2021-02-10T11:37:02.417 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:02.413 7f4e847de700 1 -- start start 2021-02-10T11:37:02.419 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:02.413 7f4e847de700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4e7c0badd0 0x7f4e7c09eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:02.419 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:02.413 7f4e847de700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4e7c0adf50 0x7f4e7c09f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:02.419 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:02.413 7f4e847de700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4e7c1a7470 0x7f4e7c1a7870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:02.419 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:02.413 7f4e847de700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e7c09e5a0 con 0x7f4e7c0adf50 2021-02-10T11:37:02.419 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:02.413 7f4e847de700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e7c09f6e0 con 0x7f4e7c1a7470 2021-02-10T11:37:02.419 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:02.413 7f4e847de700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e7c0bb220 con 0x7f4e7c0badd0 2021-02-10T11:37:02.419 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:02.413 7f4e81df9700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4e7c0adf50 0x7f4e7c09f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:02.419 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:02.413 7f4e81df9700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4e7c0adf50 0x7f4e7c09f0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:38362/0 (socket says 172.21.2.102:38362) 2021-02-10T11:37:02.419 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:02.413 7f4e81df9700 1 -- 172.21.2.102:0/1794658984 learned_addr learned my addr 172.21.2.102:0/1794658984 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:37:02.419 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:02.413 7f4e82dfb700 1 --2- 172.21.2.102:0/1794658984 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4e7c1a7470 0x7f4e7c1a7870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:02.419 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:02.413 7f4e825fa700 1 --2- 172.21.2.102:0/1794658984 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4e7c0badd0 0x7f4e7c09eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:02.420 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:02.413 7f4e81df9700 1 -- 172.21.2.102:0/1794658984 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4e7c0badd0 msgr2=0x7f4e7c09eaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:02.420 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:02.414 7f4e81df9700 1 --2- 172.21.2.102:0/1794658984 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4e7c0badd0 0x7f4e7c09eaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:02.420 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:02.414 7f4e81df9700 1 -- 172.21.2.102:0/1794658984 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4e7c1a7470 msgr2=0x7f4e7c1a7870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:02.420 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:02.414 7f4e81df9700 1 --2- 172.21.2.102:0/1794658984 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4e7c1a7470 0x7f4e7c1a7870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:02.420 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:02.414 7f4e81df9700 1 -- 172.21.2.102:0/1794658984 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e7c1a7e90 con 0x7f4e7c0adf50 2021-02-10T11:37:02.420 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:02.414 7f4e825fa700 1 --2- 172.21.2.102:0/1794658984 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4e7c0badd0 0x7f4e7c09eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:37:02.420 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:02.414 7f4e82dfb700 1 --2- 172.21.2.102:0/1794658984 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4e7c1a7470 0x7f4e7c1a7870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:37:02.420 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:02.414 7f4e81df9700 1 --2- 172.21.2.102:0/1794658984 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4e7c0adf50 0x7f4e7c09f0c0 secure :-1 s=READY pgs=2904 cs=0 l=1 rev1=1 rx=0x7f4e70009650 tx=0x7f4e7000b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:02.420 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:02.414 7f4e6effd700 1 -- 172.21.2.102:0/1794658984 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4e7000bd50 con 0x7f4e7c0adf50 2021-02-10T11:37:02.420 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:02.414 7f4e6effd700 1 -- 172.21.2.102:0/1794658984 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e70006030 con 0x7f4e7c0adf50 2021-02-10T11:37:02.420 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:02.414 7f4e6effd700 1 -- 172.21.2.102:0/1794658984 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4e70005560 con 0x7f4e7c0adf50 2021-02-10T11:37:02.420 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:02.414 7f4e847de700 1 -- 172.21.2.102:0/1794658984 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e7c1a7e90 con 0x7f4e7c0adf50 2021-02-10T11:37:02.420 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:02.414 7f4e847de700 1 -- 172.21.2.102:0/1794658984 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e7c09d1f0 con 0x7f4e7c0adf50 2021-02-10T11:37:02.421 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:02.415 7f4e6effd700 1 -- 172.21.2.102:0/1794658984 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4e700054f0 con 0x7f4e7c0adf50 2021-02-10T11:37:02.421 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:02.415 7f4e6effd700 1 --2- 172.21.2.102:0/1794658984 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4e640186c0 0x7f4e6401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:02.421 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:02.415 7f4e6effd700 1 -- 172.21.2.102:0/1794658984 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(162..162 src has 1..162) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f4e700323c0 con 0x7f4e7c0adf50 2021-02-10T11:37:02.421 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:02.415 7f4e825fa700 1 --2- 172.21.2.102:0/1794658984 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4e640186c0 0x7f4e6401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:02.421 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:02.415 7f4e825fa700 1 --2- 172.21.2.102:0/1794658984 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4e640186c0 0x7f4e6401abc0 crc :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:02.447 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:02.445 7f4e847de700 1 -- 172.21.2.102:0/1794658984 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e7c082a20 con 0x7f4e7c0adf50 2021-02-10T11:37:02.450 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:02.449 7f4e6effd700 1 -- 172.21.2.102:0/1794658984 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f4e700052e0 con 0x7f4e7c0adf50 2021-02-10T11:37:02.577 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:02.575 7f4e847de700 1 -- 172.21.2.102:0/1794658984 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7f4e7c09e520 con 0x7f4e7c0adf50 2021-02-10T11:37:02.577 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:02.575 7f4e6effd700 1 -- 172.21.2.102:0/1794658984 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v162) v1 ==== 104+0+10 (secure 0 0 0) 0x7f4e7000bd50 con 0x7f4e7c0adf50 2021-02-10T11:37:02.578 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:37:02.580 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:02.577 7f4e847de700 1 -- 172.21.2.102:0/1794658984 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4e640186c0 msgr2=0x7f4e6401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:02.580 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:02.577 7f4e847de700 1 --2- 172.21.2.102:0/1794658984 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4e640186c0 0x7f4e6401abc0 crc :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:02.580 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:02.577 7f4e847de700 1 -- 172.21.2.102:0/1794658984 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4e7c0adf50 msgr2=0x7f4e7c09f0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:02.580 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:02.577 7f4e847de700 1 --2- 172.21.2.102:0/1794658984 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4e7c0adf50 0x7f4e7c09f0c0 secure :-1 s=READY pgs=2904 cs=0 l=1 rev1=1 rx=0x7f4e70009650 tx=0x7f4e7000b960).stop 2021-02-10T11:37:02.580 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:02.578 7f4e847de700 1 -- 172.21.2.102:0/1794658984 shutdown_connections 2021-02-10T11:37:02.580 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:02.578 7f4e847de700 1 --2- 172.21.2.102:0/1794658984 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4e640186c0 0x7f4e6401abc0 unknown :-1 s=CLOSED pgs=1275 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:02.580 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:02.578 7f4e847de700 1 --2- 172.21.2.102:0/1794658984 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4e7c1a7470 0x7f4e7c1a7870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:02.580 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:02.578 7f4e847de700 1 --2- 172.21.2.102:0/1794658984 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4e7c0adf50 0x7f4e7c09f0c0 unknown :-1 s=CLOSED pgs=2904 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:02.580 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:02.578 7f4e847de700 1 --2- 172.21.2.102:0/1794658984 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4e7c0badd0 0x7f4e7c09eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:02.580 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:02.578 7f4e847de700 1 -- 172.21.2.102:0/1794658984 >> 172.21.2.102:0/1794658984 conn(0x7f4e7c0bee90 msgr2=0x7f4e7c0bb880 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:02.580 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:02.578 7f4e847de700 1 -- 172.21.2.102:0/1794658984 shutdown_connections 2021-02-10T11:37:02.580 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:02.578 7f4e847de700 1 -- 172.21.2.102:0/1794658984 wait complete. 2021-02-10T11:37:02.588 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-02-10T11:37:03.021 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:03.002 7f7979854700 1 Processor -- start 2021-02-10T11:37:03.022 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:03.002 7f7979854700 1 -- start start 2021-02-10T11:37:03.022 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:03.003 7f7979854700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f79740d1870 0x7f79741b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:03.022 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:03.003 7f7979854700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7974109a60 con 0x7f79740cff70 2021-02-10T11:37:03.022 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:03.003 7f7979854700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7974088070 con 0x7f79741b0ae0 2021-02-10T11:37:03.022 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:03.003 7f7979854700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79740e5ef0 con 0x7f79740d1870 2021-02-10T11:37:03.022 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:03.003 7f7973d75700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f79740cff70 0x7f7974109130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:58830/0 (socket says 172.21.2.102:58830) 2021-02-10T11:37:03.022 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:03.003 7f7973d75700 1 -- 172.21.2.102:0/2476148463 learned_addr learned my addr 172.21.2.102:0/2476148463 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:37:03.022 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:03.003 7f7973574700 1 --2- 172.21.2.102:0/2476148463 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f79740d1870 0x7f79741b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:03.022 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:03.003 7f7972572700 1 -- 172.21.2.102:0/2476148463 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1596263193 0 0) 0x7f796c001500 con 0x7f79740cff70 2021-02-10T11:37:03.022 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:03.003 7f7972572700 1 -- 172.21.2.102:0/2476148463 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7960002fe0 con 0x7f79740cff70 2021-02-10T11:37:03.022 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:03.003 7f7972572700 1 -- 172.21.2.102:0/2476148463 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1754806941 0 0) 0x7f796c001460 con 0x7f79740cff70 2021-02-10T11:37:03.023 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:03.003 7f7972572700 1 -- 172.21.2.102:0/2476148463 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f79740d1870 msgr2=0x7f79741b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:37:03.023 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:03.003 7f7972572700 1 --2- 172.21.2.102:0/2476148463 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f79740d1870 0x7f79741b04c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:03.023 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:03.003 7f7972572700 1 -- 172.21.2.102:0/2476148463 >> v1:172.21.2.113:6789/0 conn(0x7f79741b0ae0 legacy=0x7f7974108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:03.023 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:03.003 7f7972572700 1 -- 172.21.2.102:0/2476148463 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79740d1c70 con 0x7f79740cff70 2021-02-10T11:37:03.023 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:03.003 7f7972572700 1 -- 172.21.2.102:0/2476148463 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7f796c001980 con 0x7f79740cff70 2021-02-10T11:37:03.023 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:03.003 7f7972572700 1 -- 172.21.2.102:0/2476148463 >> v1:172.21.2.102:6789/0 conn(0x7f79740cff70 legacy=0x7f7974109130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:03.023 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:03.003 7f7972572700 1 --2- 172.21.2.102:0/2476148463 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7960003aa0 0x7f7960005ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:03.023 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:03.003 7f7972572700 1 --2- 172.21.2.102:0/2476148463 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7960006770 0x7f7960008c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:03.023 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:03.004 7f7972572700 1 --2- 172.21.2.102:0/2476148463 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7960009590 0x7f796000ba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:03.023 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:03.004 7f7972572700 1 -- 172.21.2.102:0/2476148463 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f796000c2d0 con 0x7f7960003aa0 2021-02-10T11:37:03.023 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:03.004 7f7972572700 1 -- 172.21.2.102:0/2476148463 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f796000c4f0 con 0x7f7960006770 2021-02-10T11:37:03.023 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:03.004 7f7973574700 1 --2- 172.21.2.102:0/2476148463 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7960003aa0 0x7f7960005ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:03.023 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:03.004 7f7972572700 1 -- 172.21.2.102:0/2476148463 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7960050720 con 0x7f7960009590 2021-02-10T11:37:03.024 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:03.004 7f7972d73700 1 --2- 172.21.2.102:0/2476148463 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7960006770 0x7f7960008c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:03.024 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:03.004 7f7973574700 1 -- 172.21.2.102:0/2476148463 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7960009590 msgr2=0x7f796000ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:37:03.024 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:03.004 7f7973574700 1 --2- 172.21.2.102:0/2476148463 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7960009590 0x7f796000ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:03.024 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:03.004 7f7973574700 1 -- 172.21.2.102:0/2476148463 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7960006770 msgr2=0x7f7960008c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:37:03.024 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:03.004 7f7973574700 1 --2- 172.21.2.102:0/2476148463 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7960006770 0x7f7960008c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:03.024 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:03.004 7f7973574700 1 -- 172.21.2.102:0/2476148463 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7960050980 con 0x7f7960003aa0 2021-02-10T11:37:03.024 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:03.004 7f7973574700 1 --2- 172.21.2.102:0/2476148463 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7960003aa0 0x7f7960005ed0 secure :-1 s=READY pgs=2906 cs=0 l=1 rev1=1 rx=0x7f795c0069e0 tx=0x7f795c009a80).ready entity=mon.0 client_cookie=8a2579aa9c19c408 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:03.024 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:03.004 7f7972572700 1 -- 172.21.2.102:0/2476148463 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f795c000c90 con 0x7f7960003aa0 2021-02-10T11:37:03.024 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:03.004 7f7972572700 1 -- 172.21.2.102:0/2476148463 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f795c00c0c0 con 0x7f7960003aa0 2021-02-10T11:37:03.024 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:03.004 7f7972572700 1 -- 172.21.2.102:0/2476148463 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f795c00c5a0 con 0x7f7960003aa0 2021-02-10T11:37:03.024 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:03.004 7f7979854700 1 -- 172.21.2.102:0/2476148463 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7960003aa0 msgr2=0x7f7960005ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:03.024 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:03.004 7f7979854700 1 --2- 172.21.2.102:0/2476148463 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7960003aa0 0x7f7960005ed0 secure :-1 s=READY pgs=2906 cs=0 l=1 rev1=1 rx=0x7f795c0069e0 tx=0x7f795c009a80).stop 2021-02-10T11:37:03.024 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:03.004 7f7973d75700 1 -- 172.21.2.102:0/2476148463 reap_dead start 2021-02-10T11:37:03.024 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:03.004 7f7979854700 1 -- 172.21.2.102:0/2476148463 shutdown_connections 2021-02-10T11:37:03.025 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:03.004 7f7979854700 1 -- 172.21.2.102:0/2476148463 >> 172.21.2.102:0/2476148463 conn(0x7f79740ce420 msgr2=0x7f79740d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:03.025 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:03.005 7f7979854700 1 -- 172.21.2.102:0/2476148463 shutdown_connections 2021-02-10T11:37:03.025 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:03.005 7f7979854700 1 -- 172.21.2.102:0/2476148463 wait complete. 2021-02-10T11:37:03.025 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:03.019 7f7979854700 1 Processor -- start 2021-02-10T11:37:03.025 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:03.020 7f7979854700 1 -- start start 2021-02-10T11:37:03.026 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:03.020 7f7979854700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7974085bc0 0x7f79740863e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:03.026 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:03.020 7f7979854700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f79740954c0 0x7f7974096300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:03.027 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:03.020 7f7979854700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f79740958c0 0x7f7974096920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:03.027 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:03.020 7f7979854700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7974095d80 con 0x7f79740958c0 2021-02-10T11:37:03.027 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:03.020 7f7979854700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79740bfa60 con 0x7f7974085bc0 2021-02-10T11:37:03.027 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:03.020 7f7979854700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7974085fc0 con 0x7f79740954c0 2021-02-10T11:37:03.027 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:03.020 7f7973d75700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f79740958c0 0x7f7974096920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:03.027 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:03.020 7f7973d75700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f79740958c0 0x7f7974096920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:38384/0 (socket says 172.21.2.102:38384) 2021-02-10T11:37:03.027 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:03.020 7f7973d75700 1 -- 172.21.2.102:0/307108928 learned_addr learned my addr 172.21.2.102:0/307108928 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:37:03.027 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:03.020 7f7973d75700 1 -- 172.21.2.102:0/307108928 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f79740954c0 msgr2=0x7f7974096300 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:37:03.027 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:03.020 7f7973d75700 1 --2- 172.21.2.102:0/307108928 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f79740954c0 0x7f7974096300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:03.027 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:03.020 7f7973d75700 1 -- 172.21.2.102:0/307108928 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7974085bc0 msgr2=0x7f79740863e0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:37:03.027 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:03.020 7f7973d75700 1 --2- 172.21.2.102:0/307108928 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7974085bc0 0x7f79740863e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:03.027 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:03.020 7f7973d75700 1 -- 172.21.2.102:0/307108928 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7974086c10 con 0x7f79740958c0 2021-02-10T11:37:03.027 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:03.021 7f7973d75700 1 --2- 172.21.2.102:0/307108928 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f79740958c0 0x7f7974096920 secure :-1 s=READY pgs=2907 cs=0 l=1 rev1=1 rx=0x7f796c00a950 tx=0x7f796c00acf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:03.028 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:03.021 7f796bfff700 1 -- 172.21.2.102:0/307108928 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f796c007020 con 0x7f79740958c0 2021-02-10T11:37:03.028 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:03.021 7f796bfff700 1 -- 172.21.2.102:0/307108928 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f796c007320 con 0x7f79740958c0 2021-02-10T11:37:03.028 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:03.021 7f796bfff700 1 -- 172.21.2.102:0/307108928 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f796c004910 con 0x7f79740958c0 2021-02-10T11:37:03.028 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:03.021 7f7979854700 1 -- 172.21.2.102:0/307108928 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7974086c10 con 0x7f79740958c0 2021-02-10T11:37:03.028 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:03.021 7f7979854700 1 -- 172.21.2.102:0/307108928 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7974097230 con 0x7f79740958c0 2021-02-10T11:37:03.028 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:03.021 7f796bfff700 1 -- 172.21.2.102:0/307108928 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f796c00ae30 con 0x7f79740958c0 2021-02-10T11:37:03.028 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:03.021 7f796bfff700 1 --2- 172.21.2.102:0/307108928 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f79580186c0 0x7f795801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:03.028 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:03.021 7f796bfff700 1 -- 172.21.2.102:0/307108928 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(162..162 src has 1..162) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f796c0329d0 con 0x7f79740958c0 2021-02-10T11:37:03.028 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:03.021 7f7973574700 1 --2- 172.21.2.102:0/307108928 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f79580186c0 0x7f795801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:03.028 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:03.022 7f7973574700 1 --2- 172.21.2.102:0/307108928 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f79580186c0 0x7f795801abc0 crc :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:03.054 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:03.053 7f7979854700 1 -- 172.21.2.102:0/307108928 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79740fe1c0 con 0x7f79740958c0 2021-02-10T11:37:03.057 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:03.056 7f796bfff700 1 -- 172.21.2.102:0/307108928 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f796c0016f0 con 0x7f79740958c0 2021-02-10T11:37:03.185 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:03.182 7f7979854700 1 -- 172.21.2.102:0/307108928 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7f79740fe1c0 con 0x7f79740958c0 2021-02-10T11:37:03.299 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:03.183 7f796bfff700 1 -- 172.21.2.102:0/307108928 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v162) v1 ==== 100+0+10 (secure 0 0 0) 0x7f796c007320 con 0x7f79740958c0 2021-02-10T11:37:03.299 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:03.184 7f7979854700 1 -- 172.21.2.102:0/307108928 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f79580186c0 msgr2=0x7f795801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:03.299 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:03.184 7f7979854700 1 --2- 172.21.2.102:0/307108928 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f79580186c0 0x7f795801abc0 crc :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:03.299 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:03.184 7f7979854700 1 -- 172.21.2.102:0/307108928 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f79740958c0 msgr2=0x7f7974096920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:03.300 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:03.184 7f7979854700 1 --2- 172.21.2.102:0/307108928 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f79740958c0 0x7f7974096920 secure :-1 s=READY pgs=2907 cs=0 l=1 rev1=1 rx=0x7f796c00a950 tx=0x7f796c00acf0).stop 2021-02-10T11:37:03.300 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:03.185 7f7979854700 1 -- 172.21.2.102:0/307108928 shutdown_connections 2021-02-10T11:37:03.300 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:03.185 7f7979854700 1 --2- 172.21.2.102:0/307108928 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f79740958c0 0x7f7974096920 unknown :-1 s=CLOSED pgs=2907 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:03.300 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:03.185 7f7979854700 1 --2- 172.21.2.102:0/307108928 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f79740954c0 0x7f7974096300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:03.300 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:03.185 7f7979854700 1 --2- 172.21.2.102:0/307108928 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f79580186c0 0x7f795801abc0 unknown :-1 s=CLOSED pgs=1276 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:03.300 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:03.185 7f7979854700 1 --2- 172.21.2.102:0/307108928 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7974085bc0 0x7f79740863e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:03.300 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:03.185 7f7979854700 1 -- 172.21.2.102:0/307108928 >> 172.21.2.102:0/307108928 conn(0x7f79740bef00 msgr2=0x7f797409d1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:03.300 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:03.185 7f7979854700 1 -- 172.21.2.102:0/307108928 shutdown_connections 2021-02-10T11:37:03.300 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:03.185 7f7979854700 1 -- 172.21.2.102:0/307108928 wait complete. 2021-02-10T11:37:03.300 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:37:03.301 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-02-10T11:37:03.301 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:37:03.737 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:03.714 7fb091fe9700 1 Processor -- start 2021-02-10T11:37:03.737 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:03.714 7fb091fe9700 1 -- start start 2021-02-10T11:37:03.737 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:03.714 7fb091fe9700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fb08c0d1870 0x7fb08c1b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:03.737 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:03.714 7fb091fe9700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb08c0f0650 0x7fb08c0cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:03.737 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:03.714 7fb091fe9700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb08c0f0b10 con 0x7fb08c1b0a10 2021-02-10T11:37:03.737 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:03.714 7fb091fe9700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb08c0e5ed0 con 0x7fb08c0d1870 2021-02-10T11:37:03.737 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:03.714 7fb091fe9700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb08c0d1c70 con 0x7fb08c0f0650 2021-02-10T11:37:03.738 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:03.714 7fb08affd700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fb08c1b0a10 0x7fb08c0cf330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:58846/0 (socket says 172.21.2.102:58846) 2021-02-10T11:37:03.738 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:03.714 7fb08affd700 1 -- 172.21.2.102:0/4192214360 learned_addr learned my addr 172.21.2.102:0/4192214360 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:37:03.738 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:03.715 7fb08b7fe700 1 --2- 172.21.2.102:0/4192214360 >> v2:172.21.2.113:3300/0 conn(0x7fb08c0d1870 0x7fb08c1b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:03.738 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:03.715 7fb08bfff700 1 --2- 172.21.2.102:0/4192214360 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb08c0f0650 0x7fb08c0cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:03.738 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:03.715 7fb08a7fc700 1 -- 172.21.2.102:0/4192214360 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2989666463 0 0) 0x7fb0800009b0 con 0x7fb08c1b0a10 2021-02-10T11:37:03.738 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:03.715 7fb08a7fc700 1 -- 172.21.2.102:0/4192214360 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb074002fe0 con 0x7fb08c1b0a10 2021-02-10T11:37:03.738 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:03.715 7fb08b7fe700 1 -- 172.21.2.102:0/4192214360 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb08c0f0650 msgr2=0x7fb08c0cfb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:37:03.738 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:03.715 7fb08b7fe700 1 --2- 172.21.2.102:0/4192214360 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb08c0f0650 0x7fb08c0cfb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:03.738 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:03.715 7fb08b7fe700 1 -- 172.21.2.102:0/4192214360 >> v1:172.21.2.102:6789/0 conn(0x7fb08c1b0a10 legacy=0x7fb08c0cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:03.738 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:03.715 7fb08b7fe700 1 -- 172.21.2.102:0/4192214360 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb08c0d1e90 con 0x7fb08c0d1870 2021-02-10T11:37:03.738 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:03.715 7fb08b7fe700 1 --2- 172.21.2.102:0/4192214360 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb08c0d1870 0x7fb08c1b03f0 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 rx=0x7fb07c006650 tx=0x7fb07c009a80).ready entity=mon.1 client_cookie=aed541a9aa417805 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:03.738 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:03.715 7fb08a7fc700 1 -- 172.21.2.102:0/4192214360 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb07c0009b0 con 0x7fb08c0d1870 2021-02-10T11:37:03.739 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:03.715 7fb08a7fc700 1 -- 172.21.2.102:0/4192214360 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb07c00c0c0 con 0x7fb08c0d1870 2021-02-10T11:37:03.739 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:03.715 7fb08a7fc700 1 -- 172.21.2.102:0/4192214360 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb07c00c520 con 0x7fb08c0d1870 2021-02-10T11:37:03.739 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:03.716 7fb091fe9700 1 -- 172.21.2.102:0/4192214360 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb08c0d1870 msgr2=0x7fb08c1b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:03.739 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:03.716 7fb091fe9700 1 --2- 172.21.2.102:0/4192214360 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb08c0d1870 0x7fb08c1b03f0 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 rx=0x7fb07c006650 tx=0x7fb07c009a80).stop 2021-02-10T11:37:03.739 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:03.716 7fb091fe9700 1 -- 172.21.2.102:0/4192214360 shutdown_connections 2021-02-10T11:37:03.739 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:03.716 7fb091fe9700 1 --2- 172.21.2.102:0/4192214360 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb08c0f0650 0x7fb08c0cfb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:03.739 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:03.716 7fb091fe9700 1 --2- 172.21.2.102:0/4192214360 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb08c0d1870 0x7fb08c1b03f0 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:03.739 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:03.716 7fb091fe9700 1 -- 172.21.2.102:0/4192214360 >> 172.21.2.102:0/4192214360 conn(0x7fb08c0ce450 msgr2=0x7fb08c0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:03.739 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:03.716 7fb091fe9700 1 -- 172.21.2.102:0/4192214360 shutdown_connections 2021-02-10T11:37:03.739 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:03.716 7fb091fe9700 1 -- 172.21.2.102:0/4192214360 wait complete. 2021-02-10T11:37:03.740 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:03.731 7fb091fe9700 1 Processor -- start 2021-02-10T11:37:03.740 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:03.731 7fb091fe9700 1 -- start start 2021-02-10T11:37:03.740 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:03.731 7fb091fe9700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb08c1942b0 0x7fb08c0b3550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:03.740 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:03.731 7fb091fe9700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb08c195070 0x7fb08c195470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:03.740 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:03.731 7fb091fe9700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb08c0bb8e0 0x7fb08c0bbce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:03.740 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:03.731 7fb091fe9700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb08c1946b0 con 0x7fb08c195070 2021-02-10T11:37:03.740 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:03.731 7fb091fe9700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb08c0b3b70 con 0x7fb08c1942b0 2021-02-10T11:37:03.740 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:03.731 7fb091fe9700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb08c0b3d90 con 0x7fb08c0bb8e0 2021-02-10T11:37:03.740 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:03.731 7fb08affd700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb08c195070 0x7fb08c195470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:03.740 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:03.731 7fb08affd700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb08c195070 0x7fb08c195470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:38396/0 (socket says 172.21.2.102:38396) 2021-02-10T11:37:03.741 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:03.731 7fb08affd700 1 -- 172.21.2.102:0/1952508799 learned_addr learned my addr 172.21.2.102:0/1952508799 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:37:03.741 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:03.732 7fb08bfff700 1 --2- 172.21.2.102:0/1952508799 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb08c0bb8e0 0x7fb08c0bbce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:03.741 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:03.732 7fb08affd700 1 -- 172.21.2.102:0/1952508799 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb08c0bb8e0 msgr2=0x7fb08c0bbce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:03.741 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:03.732 7fb08affd700 1 --2- 172.21.2.102:0/1952508799 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb08c0bb8e0 0x7fb08c0bbce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:03.741 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:03.732 7fb08affd700 1 -- 172.21.2.102:0/1952508799 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb08c1942b0 msgr2=0x7fb08c0b3550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:03.741 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:03.732 7fb08affd700 1 --2- 172.21.2.102:0/1952508799 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb08c1942b0 0x7fb08c0b3550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:03.741 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:03.732 7fb08affd700 1 -- 172.21.2.102:0/1952508799 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb08c1a49e0 con 0x7fb08c195070 2021-02-10T11:37:03.741 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:03.732 7fb08bfff700 1 --2- 172.21.2.102:0/1952508799 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb08c0bb8e0 0x7fb08c0bbce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:37:03.742 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:03.732 7fb08affd700 1 --2- 172.21.2.102:0/1952508799 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb08c195070 0x7fb08c195470 secure :-1 s=READY pgs=2909 cs=0 l=1 rev1=1 rx=0x7fb080001c60 tx=0x7fb08000a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:03.742 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:03.732 7fb073fff700 1 -- 172.21.2.102:0/1952508799 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb08000acc0 con 0x7fb08c195070 2021-02-10T11:37:03.742 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:03.732 7fb073fff700 1 -- 172.21.2.102:0/1952508799 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb080004140 con 0x7fb08c195070 2021-02-10T11:37:03.742 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:03.732 7fb073fff700 1 -- 172.21.2.102:0/1952508799 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb080004660 con 0x7fb08c195070 2021-02-10T11:37:03.742 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:03.732 7fb091fe9700 1 -- 172.21.2.102:0/1952508799 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb08c1a49e0 con 0x7fb08c195070 2021-02-10T11:37:03.742 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:03.732 7fb091fe9700 1 -- 172.21.2.102:0/1952508799 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb08c0bc5f0 con 0x7fb08c195070 2021-02-10T11:37:03.742 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:03.733 7fb073fff700 1 -- 172.21.2.102:0/1952508799 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fb080007f30 con 0x7fb08c195070 2021-02-10T11:37:03.743 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:03.733 7fb073fff700 1 --2- 172.21.2.102:0/1952508799 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb06c0186c0 0x7fb06c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:03.743 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:03.733 7fb073fff700 1 -- 172.21.2.102:0/1952508799 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(162..162 src has 1..162) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fb080032550 con 0x7fb08c195070 2021-02-10T11:37:03.743 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:03.733 7fb08b7fe700 1 --2- 172.21.2.102:0/1952508799 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb06c0186c0 0x7fb06c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:03.743 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:03.733 7fb08b7fe700 1 --2- 172.21.2.102:0/1952508799 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb06c0186c0 0x7fb06c01abc0 crc :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:03.766 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:03.764 7fb091fe9700 1 -- 172.21.2.102:0/1952508799 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb08c0bc5f0 con 0x7fb08c195070 2021-02-10T11:37:03.769 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:03.768 7fb073fff700 1 -- 172.21.2.102:0/1952508799 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fb080030d00 con 0x7fb08c195070 2021-02-10T11:37:03.898 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:03.896 7fb091fe9700 1 -- 172.21.2.102:0/1952508799 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb08c0bc5f0 con 0x7fb08c195070 2021-02-10T11:37:03.900 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:03.896 7fb073fff700 1 -- 172.21.2.102:0/1952508799 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 141 v141) v1 ==== 95+0+2108 (secure 0 0 0) 0x7fb080007f30 con 0x7fb08c195070 2021-02-10T11:37:03.900 INFO:teuthology.orchestra.run.gibba002.stderr:dumped fsmap epoch 141 2021-02-10T11:37:03.900 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:37:03.900 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":141,"default_fscid":17,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":12405,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.102:6835/4152930669","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6834","nonce":4152930669},{"type":"v1","addr":"172.21.2.102:6835","nonce":4152930669}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":137}],"filesystems":[{"mdsmap":{"epoch":141,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:36:46.331874","modified":"2021-02-10 11:36:48.340104","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7130},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7130":{"gid":7130,"name":"b","rank":0,"incarnation":138,"state":"up:active","state_seq":3,"addr":"172.21.2.113:6833/2966977555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6832","nonce":2966977555},{"type":"v1","addr":"172.21.2.113:6833","nonce":2966977555}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[35],"metadata_pool":34,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":17}]} 2021-02-10T11:37:03.902 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:03.898 7fb091fe9700 1 -- 172.21.2.102:0/1952508799 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb06c0186c0 msgr2=0x7fb06c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:03.902 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:03.898 7fb091fe9700 1 --2- 172.21.2.102:0/1952508799 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb06c0186c0 0x7fb06c01abc0 crc :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:03.902 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:03.898 7fb091fe9700 1 -- 172.21.2.102:0/1952508799 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb08c195070 msgr2=0x7fb08c195470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:03.902 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:03.898 7fb091fe9700 1 --2- 172.21.2.102:0/1952508799 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb08c195070 0x7fb08c195470 secure :-1 s=READY pgs=2909 cs=0 l=1 rev1=1 rx=0x7fb080001c60 tx=0x7fb08000a8d0).stop 2021-02-10T11:37:03.902 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:03.898 7fb091fe9700 1 -- 172.21.2.102:0/1952508799 shutdown_connections 2021-02-10T11:37:03.903 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:03.898 7fb091fe9700 1 --2- 172.21.2.102:0/1952508799 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb08c0bb8e0 0x7fb08c0bbce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:03.903 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:03.898 7fb091fe9700 1 --2- 172.21.2.102:0/1952508799 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb08c195070 0x7fb08c195470 unknown :-1 s=CLOSED pgs=2909 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:03.903 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:03.898 7fb091fe9700 1 --2- 172.21.2.102:0/1952508799 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb06c0186c0 0x7fb06c01abc0 unknown :-1 s=CLOSED pgs=1277 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:03.903 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:03.898 7fb091fe9700 1 --2- 172.21.2.102:0/1952508799 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb08c1942b0 0x7fb08c0b3550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:03.903 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:03.898 7fb091fe9700 1 -- 172.21.2.102:0/1952508799 >> 172.21.2.102:0/1952508799 conn(0x7fb08c0b3040 msgr2=0x7fb08c0c3250 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:03.903 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:03.899 7fb091fe9700 1 -- 172.21.2.102:0/1952508799 shutdown_connections 2021-02-10T11:37:03.903 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:03.899 7fb091fe9700 1 -- 172.21.2.102:0/1952508799 wait complete. 2021-02-10T11:37:04.111 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 141, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:36:46.331874', 'modified': '2021-02-10 11:36:48.340104', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7130}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7130': {'gid': 7130, 'name': 'b', 'rank': 0, 'incarnation': 138, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.113:6833/2966977555', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.113:6832', 'nonce': 2966977555}, {'type': 'v1', 'addr': '172.21.2.113:6833', 'nonce': 2966977555}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [35], 'metadata_pool': 34, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:37:04.111 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:37:04.111 DEBUG:teuthology.orchestra.run.gibba013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:37:04.509 INFO:teuthology.orchestra.run.gibba013.stdout:{ 2021-02-10T11:37:04.510 INFO:teuthology.orchestra.run.gibba013.stdout: "cluster_fsid": "68db30ea-dbce-4aef-a6df-f01c6fea0513", 2021-02-10T11:37:04.510 INFO:teuthology.orchestra.run.gibba013.stdout: "whoami": 0, 2021-02-10T11:37:04.510 INFO:teuthology.orchestra.run.gibba013.stdout: "id": 7130, 2021-02-10T11:37:04.510 INFO:teuthology.orchestra.run.gibba013.stdout: "want_state": "up:active", 2021-02-10T11:37:04.510 INFO:teuthology.orchestra.run.gibba013.stdout: "state": "up:active", 2021-02-10T11:37:04.510 INFO:teuthology.orchestra.run.gibba013.stdout: "rank_uptime": 18.174562420000001, 2021-02-10T11:37:04.510 INFO:teuthology.orchestra.run.gibba013.stdout: "mdsmap_epoch": 141, 2021-02-10T11:37:04.510 INFO:teuthology.orchestra.run.gibba013.stdout: "osdmap_epoch": 162, 2021-02-10T11:37:04.511 INFO:teuthology.orchestra.run.gibba013.stdout: "osdmap_epoch_barrier": 162, 2021-02-10T11:37:04.511 INFO:teuthology.orchestra.run.gibba013.stdout: "uptime": 23.839034315999999 2021-02-10T11:37:04.511 INFO:teuthology.orchestra.run.gibba013.stdout:} 2021-02-10T11:37:04.513 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "68db30ea-dbce-4aef-a6df-f01c6fea0513", "whoami": 0, "id": 7130, "want_state": "up:active", "state": "up:active", "rank_uptime": 18.174562420000001, "mdsmap_epoch": 141, "osdmap_epoch": 162, "osdmap_epoch_barrier": 162, "uptime": 23.839034315999999 } 2021-02-10T11:37:04.514 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2021-02-10T11:37:04.514 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2021-02-10T11:37:04.514 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.1 at ubuntu@gibba002.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.1... 2021-02-10T11:37:04.514 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:37:04.514 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec mkdir -p /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:37:04.856 DEBUG:teuthology.orchestra.run.gibba002:> sudo modprobe fuse 2021-02-10T11:37:05.200 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:37:05.200 DEBUG:teuthology.orchestra.run.gibba002:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:37:05.545 INFO:teuthology.orchestra.run.gibba002.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:37:05.545 INFO:teuthology.orchestra.run.gibba002.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:37:05.546 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:37:05.546 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:37:05.546 DEBUG:teuthology.orchestra.run.gibba002:> ls /sys/fs/fuse/connections 2021-02-10T11:37:05.882 INFO:teuthology.orchestra.run.gibba002.stdout:43 2021-02-10T11:37:05.882 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [43] 2021-02-10T11:37:05.882 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.1 /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:37:05.884 DEBUG:teuthology.orchestra.run.gibba002:> sudo modprobe fuse 2021-02-10T11:37:06.229 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:37:06.230 DEBUG:teuthology.orchestra.run.gibba002:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:37:06.296 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba002.stderr:ceph-fuse[205151]: starting ceph client 2021-02-10T11:37:06.296 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba002.stderr:2021-02-10 11:37:06.295 7f5bd9af7f80 -1 init, newargv = 0x55f8635e79a0 newargc=9 2021-02-10T11:37:06.301 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba002.stderr:ceph-fuse[205151]: starting fuse 2021-02-10T11:37:06.573 INFO:teuthology.orchestra.run.gibba002.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:37:06.573 INFO:teuthology.orchestra.run.gibba002.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:37:06.574 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:37:06.574 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:37:06.574 DEBUG:teuthology.orchestra.run.gibba002:> ls /sys/fs/fuse/connections 2021-02-10T11:37:06.910 INFO:teuthology.orchestra.run.gibba002.stdout:43 2021-02-10T11:37:06.911 INFO:teuthology.orchestra.run.gibba002.stdout:44 2021-02-10T11:37:06.911 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43, 44] 2021-02-10T11:37:06.911 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:37:06.911 DEBUG:teuthology.orchestra.run.gibba002:> sudo python3 -c ' 2021-02-10T11:37:06.911 DEBUG:teuthology.orchestra.run.gibba002:> import glob 2021-02-10T11:37:06.911 DEBUG:teuthology.orchestra.run.gibba002:> import re 2021-02-10T11:37:06.911 DEBUG:teuthology.orchestra.run.gibba002:> import os 2021-02-10T11:37:06.911 DEBUG:teuthology.orchestra.run.gibba002:> import subprocess 2021-02-10T11:37:06.911 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:37:06.912 DEBUG:teuthology.orchestra.run.gibba002:> def find_socket(client_name): 2021-02-10T11:37:06.912 DEBUG:teuthology.orchestra.run.gibba002:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2021-02-10T11:37:06.912 DEBUG:teuthology.orchestra.run.gibba002:> files = glob.glob(asok_path) 2021-02-10T11:37:06.912 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:37:06.912 DEBUG:teuthology.orchestra.run.gibba002:> # Given a non-glob path, it better be there 2021-02-10T11:37:06.912 DEBUG:teuthology.orchestra.run.gibba002:> if "*" not in asok_path: 2021-02-10T11:37:06.912 DEBUG:teuthology.orchestra.run.gibba002:> assert(len(files) == 1) 2021-02-10T11:37:06.912 DEBUG:teuthology.orchestra.run.gibba002:> return files[0] 2021-02-10T11:37:06.912 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:37:06.912 DEBUG:teuthology.orchestra.run.gibba002:> for f in files: 2021-02-10T11:37:06.912 DEBUG:teuthology.orchestra.run.gibba002:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2021-02-10T11:37:06.912 DEBUG:teuthology.orchestra.run.gibba002:> if os.path.exists("/proc/{0}".format(pid)): 2021-02-10T11:37:06.912 DEBUG:teuthology.orchestra.run.gibba002:> return f 2021-02-10T11:37:06.913 DEBUG:teuthology.orchestra.run.gibba002:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2021-02-10T11:37:06.913 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:37:06.913 DEBUG:teuthology.orchestra.run.gibba002:> print(find_socket("client.1")) 2021-02-10T11:37:06.913 DEBUG:teuthology.orchestra.run.gibba002:> ' 2021-02-10T11:37:07.275 INFO:teuthology.orchestra.run.gibba002.stdout:/var/run/ceph/ceph-client.1.205151.asok 2021-02-10T11:37:07.278 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.205151.asok 2021-02-10T11:37:07.278 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:37:07.278 DEBUG:teuthology.orchestra.run.gibba002:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.205151.asok status 2021-02-10T11:37:07.671 INFO:teuthology.orchestra.run.gibba002.stdout:{ 2021-02-10T11:37:07.671 INFO:teuthology.orchestra.run.gibba002.stdout: "metadata": { 2021-02-10T11:37:07.671 INFO:teuthology.orchestra.run.gibba002.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:37:07.671 INFO:teuthology.orchestra.run.gibba002.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:37:07.671 INFO:teuthology.orchestra.run.gibba002.stdout: "entity_id": "1", 2021-02-10T11:37:07.671 INFO:teuthology.orchestra.run.gibba002.stdout: "hostname": "gibba002", 2021-02-10T11:37:07.671 INFO:teuthology.orchestra.run.gibba002.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2021-02-10T11:37:07.671 INFO:teuthology.orchestra.run.gibba002.stdout: "pid": "205151", 2021-02-10T11:37:07.671 INFO:teuthology.orchestra.run.gibba002.stdout: "root": "/" 2021-02-10T11:37:07.671 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:37:07.671 INFO:teuthology.orchestra.run.gibba002.stdout: "dentry_count": 0, 2021-02-10T11:37:07.672 INFO:teuthology.orchestra.run.gibba002.stdout: "dentry_pinned_count": 0, 2021-02-10T11:37:07.672 INFO:teuthology.orchestra.run.gibba002.stdout: "id": 12588, 2021-02-10T11:37:07.672 INFO:teuthology.orchestra.run.gibba002.stdout: "inst": { 2021-02-10T11:37:07.672 INFO:teuthology.orchestra.run.gibba002.stdout: "name": { 2021-02-10T11:37:07.672 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "client", 2021-02-10T11:37:07.672 INFO:teuthology.orchestra.run.gibba002.stdout: "num": 12588 2021-02-10T11:37:07.672 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:37:07.672 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": { 2021-02-10T11:37:07.672 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:37:07.672 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:0", 2021-02-10T11:37:07.672 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 1366666634 2021-02-10T11:37:07.672 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:37:07.672 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:37:07.673 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": { 2021-02-10T11:37:07.673 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:37:07.673 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:0", 2021-02-10T11:37:07.673 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 1366666634 2021-02-10T11:37:07.673 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:37:07.673 INFO:teuthology.orchestra.run.gibba002.stdout: "inst_str": "client.12588 172.21.2.102:0/1366666634", 2021-02-10T11:37:07.673 INFO:teuthology.orchestra.run.gibba002.stdout: "addr_str": "172.21.2.102:0/1366666634", 2021-02-10T11:37:07.673 INFO:teuthology.orchestra.run.gibba002.stdout: "inode_count": 1, 2021-02-10T11:37:07.673 INFO:teuthology.orchestra.run.gibba002.stdout: "mds_epoch": 141, 2021-02-10T11:37:07.673 INFO:teuthology.orchestra.run.gibba002.stdout: "osd_epoch": 162, 2021-02-10T11:37:07.673 INFO:teuthology.orchestra.run.gibba002.stdout: "osd_epoch_barrier": 0, 2021-02-10T11:37:07.673 INFO:teuthology.orchestra.run.gibba002.stdout: "blacklisted": false 2021-02-10T11:37:07.673 INFO:teuthology.orchestra.run.gibba002.stdout:} 2021-02-10T11:37:07.675 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:37:07.675 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:37:07.675 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:37:08.010 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:37:08.010 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2021-02-10T11:37:08.010 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:37:08.010 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:37:08.357 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2021-02-10T11:37:08.792 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:08.774 7f7a3af32700 1 Processor -- start 2021-02-10T11:37:08.792 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:08.774 7f7a3af32700 1 -- start start 2021-02-10T11:37:08.792 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:08.775 7f7a3af32700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f7a340d1880 0x7f7a341b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:08.792 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:08.775 7f7a3af32700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7a340cff80 0x7f7a34109140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:08.793 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:08.775 7f7a3af32700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a34109a70 con 0x7f7a341b0af0 2021-02-10T11:37:08.793 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:08.775 7f7a3af32700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a34109760 con 0x7f7a340d1880 2021-02-10T11:37:08.793 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:08.775 7f7a3af32700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a34088070 con 0x7f7a340cff80 2021-02-10T11:37:08.793 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:08.775 7f7a2bfff700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f7a341b0af0 0x7f7a34108950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:58872/0 (socket says 172.21.2.102:58872) 2021-02-10T11:37:08.793 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:08.775 7f7a2bfff700 1 -- 172.21.2.102:0/2274525525 learned_addr learned my addr 172.21.2.102:0/2274525525 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:37:08.793 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:08.775 7f7a38d4e700 1 --2- 172.21.2.102:0/2274525525 >> v2:172.21.2.113:3300/0 conn(0x7f7a340d1880 0x7f7a341b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:08.793 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:08.775 7f7a3954f700 1 --2- 172.21.2.102:0/2274525525 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7a340cff80 0x7f7a34109140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:08.793 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:08.775 7f7a2b7fe700 1 -- 172.21.2.102:0/2274525525 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 630819111 0 0) 0x7f7a30000d30 con 0x7f7a341b0af0 2021-02-10T11:37:08.793 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:08.775 7f7a2b7fe700 1 -- 172.21.2.102:0/2274525525 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a1c002fe0 con 0x7f7a341b0af0 2021-02-10T11:37:08.793 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:08.776 7f7a2b7fe700 1 -- 172.21.2.102:0/2274525525 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1321668802 0 0) 0x7f7a30001390 con 0x7f7a341b0af0 2021-02-10T11:37:08.793 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:08.776 7f7a38d4e700 1 -- 172.21.2.102:0/2274525525 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7a340cff80 msgr2=0x7f7a34109140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:37:08.793 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:08.776 7f7a38d4e700 1 --2- 172.21.2.102:0/2274525525 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7a340cff80 0x7f7a34109140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:08.794 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:08.776 7f7a38d4e700 1 -- 172.21.2.102:0/2274525525 >> v1:172.21.2.102:6789/0 conn(0x7f7a341b0af0 legacy=0x7f7a34108950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:08.794 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:08.776 7f7a38d4e700 1 -- 172.21.2.102:0/2274525525 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a340d1c80 con 0x7f7a340d1880 2021-02-10T11:37:08.794 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:08.776 7f7a3954f700 1 --2- 172.21.2.102:0/2274525525 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7a340cff80 0x7f7a34109140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:37:08.794 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:08.776 7f7a38d4e700 1 --2- 172.21.2.102:0/2274525525 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7a340d1880 0x7f7a341b04d0 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 rx=0x7f7a24006650 tx=0x7f7a24009a80).ready entity=mon.1 client_cookie=d484eeb0f494fca5 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:08.794 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:08.776 7f7a2b7fe700 1 -- 172.21.2.102:0/2274525525 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7a240009b0 con 0x7f7a340d1880 2021-02-10T11:37:08.794 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:08.776 7f7a2b7fe700 1 -- 172.21.2.102:0/2274525525 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a24006740 con 0x7f7a340d1880 2021-02-10T11:37:08.794 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:08.776 7f7a2b7fe700 1 -- 172.21.2.102:0/2274525525 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7a2400c0c0 con 0x7f7a340d1880 2021-02-10T11:37:08.794 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:08.776 7f7a3af32700 1 -- 172.21.2.102:0/2274525525 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7a340d1880 msgr2=0x7f7a341b04d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:08.794 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:08.776 7f7a3af32700 1 --2- 172.21.2.102:0/2274525525 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7a340d1880 0x7f7a341b04d0 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 rx=0x7f7a24006650 tx=0x7f7a24009a80).stop 2021-02-10T11:37:08.794 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:08.776 7f7a3af32700 1 -- 172.21.2.102:0/2274525525 shutdown_connections 2021-02-10T11:37:08.794 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:08.776 7f7a3af32700 1 --2- 172.21.2.102:0/2274525525 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7a340cff80 0x7f7a34109140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:08.794 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:08.776 7f7a3af32700 1 --2- 172.21.2.102:0/2274525525 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7a340d1880 0x7f7a341b04d0 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:08.794 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:08.776 7f7a3af32700 1 -- 172.21.2.102:0/2274525525 >> 172.21.2.102:0/2274525525 conn(0x7f7a340ce430 msgr2=0x7f7a340d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:08.795 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:08.776 7f7a3af32700 1 -- 172.21.2.102:0/2274525525 shutdown_connections 2021-02-10T11:37:08.795 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:08.776 7f7a3af32700 1 -- 172.21.2.102:0/2274525525 wait complete. 2021-02-10T11:37:08.795 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:08.791 7f7a3af32700 1 Processor -- start 2021-02-10T11:37:08.795 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:08.791 7f7a3af32700 1 -- start start 2021-02-10T11:37:08.796 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:08.792 7f7a3af32700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7a340c3350 0x7f7a340c3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:08.796 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:08.792 7f7a3af32700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7a3409f170 0x7f7a3409f570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:08.796 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:08.792 7f7a3af32700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7a3409fb90 0x7f7a340b8ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:08.796 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:08.792 7f7a3af32700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a340b85c0 con 0x7f7a3409fb90 2021-02-10T11:37:08.796 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:08.792 7f7a3af32700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a3409ffd0 con 0x7f7a3409f170 2021-02-10T11:37:08.797 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:08.792 7f7a3af32700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a340b90e0 con 0x7f7a340c3350 2021-02-10T11:37:08.797 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:08.792 7f7a3954f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7a3409fb90 0x7f7a340b8ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:08.797 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:08.792 7f7a3954f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7a3409fb90 0x7f7a340b8ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:38424/0 (socket says 172.21.2.102:38424) 2021-02-10T11:37:08.797 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:08.792 7f7a3954f700 1 -- 172.21.2.102:0/1818475328 learned_addr learned my addr 172.21.2.102:0/1818475328 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:37:08.797 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:08.792 7f7a2bfff700 1 --2- 172.21.2.102:0/1818475328 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7a3409f170 0x7f7a3409f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:08.797 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:08.792 7f7a38d4e700 1 --2- 172.21.2.102:0/1818475328 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7a340c3350 0x7f7a340c3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:08.797 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:08.792 7f7a3954f700 1 -- 172.21.2.102:0/1818475328 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7a340c3350 msgr2=0x7f7a340c3750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:08.797 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:08.792 7f7a3954f700 1 --2- 172.21.2.102:0/1818475328 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7a340c3350 0x7f7a340c3750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:08.797 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:08.792 7f7a3954f700 1 -- 172.21.2.102:0/1818475328 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7a3409f170 msgr2=0x7f7a3409f570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:08.797 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:08.792 7f7a3954f700 1 --2- 172.21.2.102:0/1818475328 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7a3409f170 0x7f7a3409f570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:08.797 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:08.792 7f7a3954f700 1 -- 172.21.2.102:0/1818475328 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a340b9300 con 0x7f7a3409fb90 2021-02-10T11:37:08.797 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:08.792 7f7a3954f700 1 --2- 172.21.2.102:0/1818475328 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7a3409fb90 0x7f7a340b8ac0 secure :-1 s=READY pgs=2914 cs=0 l=1 rev1=1 rx=0x7f7a2c00b5e0 tx=0x7f7a2c00b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:08.797 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:08.792 7f7a297fa700 1 -- 172.21.2.102:0/1818475328 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7a2c000c10 con 0x7f7a3409fb90 2021-02-10T11:37:08.798 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:08.792 7f7a297fa700 1 -- 172.21.2.102:0/1818475328 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a2c00bd70 con 0x7f7a3409fb90 2021-02-10T11:37:08.798 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:08.793 7f7a3af32700 1 -- 172.21.2.102:0/1818475328 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a340b9300 con 0x7f7a3409fb90 2021-02-10T11:37:08.798 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:08.793 7f7a3af32700 1 -- 172.21.2.102:0/1818475328 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a340b9890 con 0x7f7a3409fb90 2021-02-10T11:37:08.798 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:08.793 7f7a297fa700 1 -- 172.21.2.102:0/1818475328 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7a2c002560 con 0x7f7a3409fb90 2021-02-10T11:37:08.798 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:08.793 7f7a297fa700 1 -- 172.21.2.102:0/1818475328 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f7a2c005330 con 0x7f7a3409fb90 2021-02-10T11:37:08.798 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:08.793 7f7a297fa700 1 --2- 172.21.2.102:0/1818475328 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7a140186c0 0x7f7a1401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:08.798 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:08.793 7f7a297fa700 1 -- 172.21.2.102:0/1818475328 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(162..162 src has 1..162) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f7a2c0325c0 con 0x7f7a3409fb90 2021-02-10T11:37:08.798 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:08.793 7f7a38d4e700 1 --2- 172.21.2.102:0/1818475328 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7a140186c0 0x7f7a1401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:08.798 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:08.794 7f7a38d4e700 1 --2- 172.21.2.102:0/1818475328 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7a140186c0 0x7f7a1401abc0 crc :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:08.825 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:08.824 7f7a3af32700 1 -- 172.21.2.102:0/1818475328 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a340b9e30 con 0x7f7a3409fb90 2021-02-10T11:37:08.827 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:08.826 7f7a297fa700 1 -- 172.21.2.102:0/1818475328 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f7a2c000bc0 con 0x7f7a3409fb90 2021-02-10T11:37:08.963 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:08.962 7f7a3af32700 1 -- 172.21.2.102:0/1818475328 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f7a340b9e30 con 0x7f7a3409fb90 2021-02-10T11:37:09.172 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:09.171 7f7a297fa700 1 -- 172.21.2.102:0/1818475328 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v142) v1 ==== 136+0+0 (secure 0 0 0) 0x7f7a2c005330 con 0x7f7a3409fb90 2021-02-10T11:37:09.180 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:09.179 7f7a3af32700 1 -- 172.21.2.102:0/1818475328 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7a140186c0 msgr2=0x7f7a1401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:09.181 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:09.179 7f7a3af32700 1 --2- 172.21.2.102:0/1818475328 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7a140186c0 0x7f7a1401abc0 crc :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:09.181 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:09.179 7f7a3af32700 1 -- 172.21.2.102:0/1818475328 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7a3409fb90 msgr2=0x7f7a340b8ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:09.181 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:09.179 7f7a3af32700 1 --2- 172.21.2.102:0/1818475328 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7a3409fb90 0x7f7a340b8ac0 secure :-1 s=READY pgs=2914 cs=0 l=1 rev1=1 rx=0x7f7a2c00b5e0 tx=0x7f7a2c00b980).stop 2021-02-10T11:37:09.181 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:09.180 7f7a3af32700 1 -- 172.21.2.102:0/1818475328 shutdown_connections 2021-02-10T11:37:09.181 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:09.180 7f7a3af32700 1 --2- 172.21.2.102:0/1818475328 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7a3409fb90 0x7f7a340b8ac0 unknown :-1 s=CLOSED pgs=2914 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:09.181 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:09.180 7f7a3af32700 1 --2- 172.21.2.102:0/1818475328 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7a140186c0 0x7f7a1401abc0 unknown :-1 s=CLOSED pgs=1278 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:09.182 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:09.180 7f7a3af32700 1 --2- 172.21.2.102:0/1818475328 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7a3409f170 0x7f7a3409f570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:09.182 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:09.180 7f7a3af32700 1 --2- 172.21.2.102:0/1818475328 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7a340c3350 0x7f7a340c3750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:09.182 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:09.180 7f7a3af32700 1 -- 172.21.2.102:0/1818475328 >> 172.21.2.102:0/1818475328 conn(0x7f7a3409ebd0 msgr2=0x7f7a34082ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:09.182 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:09.180 7f7a3af32700 1 -- 172.21.2.102:0/1818475328 shutdown_connections 2021-02-10T11:37:09.182 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:09.180 7f7a3af32700 1 -- 172.21.2.102:0/1818475328 wait complete. 2021-02-10T11:37:09.189 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs volume ls 2021-02-10T11:37:09.616 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:09.599 7f30db59e700 1 Processor -- start 2021-02-10T11:37:09.616 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:09.599 7f30db59e700 1 -- start start 2021-02-10T11:37:09.616 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:09.600 7f30db59e700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f30d40d1870 0x7f30d41b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:09.616 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:09.600 7f30db59e700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f30d41b0a10 0x7f30d40cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:09.616 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:09.600 7f30db59e700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f30d40d1c70 0x7f30d40cf950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:09.617 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:09.600 7f30db59e700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30d40fe200 con 0x7f30d40d1870 2021-02-10T11:37:09.617 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:09.600 7f30db59e700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30d40e5f10 con 0x7f30d40d1c70 2021-02-10T11:37:09.617 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:09.600 7f30db59e700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30d40f0b50 con 0x7f30d41b0a10 2021-02-10T11:37:09.617 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:09.600 7f30d93ba700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f30d40d1870 0x7f30d41b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:09.617 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:09.600 7f30d93ba700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f30d40d1870 0x7f30d41b03f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:38428/0 (socket says 172.21.2.102:38428) 2021-02-10T11:37:09.617 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:09.600 7f30d93ba700 1 -- 172.21.2.102:0/3768154409 learned_addr learned my addr 172.21.2.102:0/3768154409 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:37:09.617 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:09.600 7f30d93ba700 1 -- 172.21.2.102:0/3768154409 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f30d41b0a10 msgr2=0x7f30d40cf330 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:37:09.617 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:09.600 7f30d93ba700 1 --2- 172.21.2.102:0/3768154409 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f30d41b0a10 0x7f30d40cf330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:09.617 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:09.600 7f30d93ba700 1 -- 172.21.2.102:0/3768154409 >> v2:172.21.2.113:3300/0 conn(0x7f30d40d1c70 msgr2=0x7f30d40cf950 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:37:09.617 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:09.600 7f30d93ba700 1 --2- 172.21.2.102:0/3768154409 >> v2:172.21.2.113:3300/0 conn(0x7f30d40d1c70 0x7f30d40cf950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:09.617 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:09.600 7f30d93ba700 1 -- 172.21.2.102:0/3768154409 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30d40cff70 con 0x7f30d40d1870 2021-02-10T11:37:09.617 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:09.600 7f30d93ba700 1 --2- 172.21.2.102:0/3768154409 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f30d40d1870 0x7f30d41b03f0 secure :-1 s=READY pgs=2915 cs=0 l=1 rev1=1 rx=0x7f30c8006700 tx=0x7f30c8009b00).ready entity=mon.0 client_cookie=79f0691dc54a81ff server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:09.618 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:09.600 7f30c7fff700 1 -- 172.21.2.102:0/3768154409 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f30c800c0c0 con 0x7f30d40d1870 2021-02-10T11:37:09.618 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:09.601 7f30c7fff700 1 -- 172.21.2.102:0/3768154409 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30c800c340 con 0x7f30d40d1870 2021-02-10T11:37:09.618 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:09.601 7f30c7fff700 1 -- 172.21.2.102:0/3768154409 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f30c800c820 con 0x7f30d40d1870 2021-02-10T11:37:09.618 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:09.601 7f30db59e700 1 -- 172.21.2.102:0/3768154409 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f30d40d1870 msgr2=0x7f30d41b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:09.618 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:09.601 7f30db59e700 1 --2- 172.21.2.102:0/3768154409 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f30d40d1870 0x7f30d41b03f0 secure :-1 s=READY pgs=2915 cs=0 l=1 rev1=1 rx=0x7f30c8006700 tx=0x7f30c8009b00).stop 2021-02-10T11:37:09.618 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:09.601 7f30db59e700 1 -- 172.21.2.102:0/3768154409 shutdown_connections 2021-02-10T11:37:09.618 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:09.601 7f30db59e700 1 --2- 172.21.2.102:0/3768154409 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f30d41b0a10 0x7f30d40cf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:09.618 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:09.601 7f30db59e700 1 --2- 172.21.2.102:0/3768154409 >> v2:172.21.2.113:3300/0 conn(0x7f30d40d1c70 0x7f30d40cf950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:09.618 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:09.601 7f30db59e700 1 --2- 172.21.2.102:0/3768154409 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f30d40d1870 0x7f30d41b03f0 unknown :-1 s=CLOSED pgs=2915 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:09.618 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:09.601 7f30db59e700 1 -- 172.21.2.102:0/3768154409 >> 172.21.2.102:0/3768154409 conn(0x7f30d40ce450 msgr2=0x7f30d40d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:09.618 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:09.601 7f30db59e700 1 -- 172.21.2.102:0/3768154409 shutdown_connections 2021-02-10T11:37:09.618 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:09.601 7f30db59e700 1 -- 172.21.2.102:0/3768154409 wait complete. 2021-02-10T11:37:09.620 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:09.616 7f30db59e700 1 Processor -- start 2021-02-10T11:37:09.620 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:09.616 7f30db59e700 1 -- start start 2021-02-10T11:37:09.620 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:09.616 7f30db59e700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f30d40a5ef0 0x7f30d4188f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:09.620 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:09.616 7f30db59e700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f30d40a62f0 0x7f30d4189570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:09.620 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:09.616 7f30db59e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f30d4189b90 0x7f30d408cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:09.620 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:09.616 7f30db59e700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30d408c360 con 0x7f30d4189b90 2021-02-10T11:37:09.620 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:09.616 7f30db59e700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30d4189f90 con 0x7f30d40a62f0 2021-02-10T11:37:09.621 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:09.616 7f30db59e700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30d408d1b0 con 0x7f30d40a5ef0 2021-02-10T11:37:09.621 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:09.616 7f30d9bbb700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f30d4189b90 0x7f30d408cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:09.621 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:09.616 7f30d9bbb700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f30d4189b90 0x7f30d408cb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:38438/0 (socket says 172.21.2.102:38438) 2021-02-10T11:37:09.621 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:09.616 7f30d9bbb700 1 -- 172.21.2.102:0/1630902347 learned_addr learned my addr 172.21.2.102:0/1630902347 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:37:09.621 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:09.617 7f30d9bbb700 1 -- 172.21.2.102:0/1630902347 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f30d40a5ef0 msgr2=0x7f30d4188f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:09.621 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:09.617 7f30d9bbb700 1 --2- 172.21.2.102:0/1630902347 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f30d40a5ef0 0x7f30d4188f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:09.621 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:09.617 7f30d9bbb700 1 -- 172.21.2.102:0/1630902347 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f30d40a62f0 msgr2=0x7f30d4189570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:09.621 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:09.617 7f30d9bbb700 1 --2- 172.21.2.102:0/1630902347 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f30d40a62f0 0x7f30d4189570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:09.621 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:09.617 7f30d9bbb700 1 -- 172.21.2.102:0/1630902347 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30d408d3d0 con 0x7f30d4189b90 2021-02-10T11:37:09.621 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:09.617 7f30d9bbb700 1 --2- 172.21.2.102:0/1630902347 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f30d4189b90 0x7f30d408cb90 secure :-1 s=READY pgs=2916 cs=0 l=1 rev1=1 rx=0x7f30cc009960 tx=0x7f30cc00bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:09.621 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:09.617 7f30c5ffb700 1 -- 172.21.2.102:0/1630902347 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f30cc003030 con 0x7f30d4189b90 2021-02-10T11:37:09.621 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:09.617 7f30c5ffb700 1 -- 172.21.2.102:0/1630902347 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30cc001310 con 0x7f30d4189b90 2021-02-10T11:37:09.621 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:09.617 7f30db59e700 1 -- 172.21.2.102:0/1630902347 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30d408d3d0 con 0x7f30d4189b90 2021-02-10T11:37:09.622 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:09.617 7f30db59e700 1 -- 172.21.2.102:0/1630902347 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30d408d3d0 con 0x7f30d4189b90 2021-02-10T11:37:09.622 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:09.617 7f30c5ffb700 1 -- 172.21.2.102:0/1630902347 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f30cc001830 con 0x7f30d4189b90 2021-02-10T11:37:09.622 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:09.617 7f30c5ffb700 1 -- 172.21.2.102:0/1630902347 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f30cc0014f0 con 0x7f30d4189b90 2021-02-10T11:37:09.622 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:09.618 7f30c5ffb700 1 --2- 172.21.2.102:0/1630902347 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f30b4018730 0x7f30b401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:09.622 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:09.618 7f30c5ffb700 1 -- 172.21.2.102:0/1630902347 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(162..162 src has 1..162) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f30cc0329f0 con 0x7f30d4189b90 2021-02-10T11:37:09.622 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:09.618 7f30d93ba700 1 --2- 172.21.2.102:0/1630902347 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f30b4018730 0x7f30b401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:09.622 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:09.618 7f30d93ba700 1 --2- 172.21.2.102:0/1630902347 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f30b4018730 0x7f30b401ac10 crc :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:09.650 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:09.649 7f30db59e700 1 -- 172.21.2.102:0/1630902347 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30d40ee0b0 con 0x7f30d4189b90 2021-02-10T11:37:09.653 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:09.652 7f30c5ffb700 1 -- 172.21.2.102:0/1630902347 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f30cc001520 con 0x7f30d4189b90 2021-02-10T11:37:09.789 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:09.788 7f30db59e700 1 -- 172.21.2.102:0/1630902347 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs volume ls", "target": ["mgr", ""]}) v1 -- 0x7f30d408c360 con 0x7f30b4018730 2021-02-10T11:37:09.789 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:09.789 7f30c5ffb700 1 -- 172.21.2.102:0/1630902347 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+41 (crc 0 0 0) 0x7f30c8009360 con 0x7f30b4018730 2021-02-10T11:37:09.790 INFO:teuthology.orchestra.run.gibba002.stdout:[ 2021-02-10T11:37:09.790 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:37:09.790 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "cephfs" 2021-02-10T11:37:09.790 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:37:09.790 INFO:teuthology.orchestra.run.gibba002.stdout:] 2021-02-10T11:37:09.792 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:09.790 7f30db59e700 1 -- 172.21.2.102:0/1630902347 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f30b4018730 msgr2=0x7f30b401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:09.792 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:09.790 7f30db59e700 1 --2- 172.21.2.102:0/1630902347 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f30b4018730 0x7f30b401ac10 crc :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:09.792 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:09.790 7f30db59e700 1 -- 172.21.2.102:0/1630902347 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f30d4189b90 msgr2=0x7f30d408cb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:09.792 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:09.790 7f30db59e700 1 --2- 172.21.2.102:0/1630902347 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f30d4189b90 0x7f30d408cb90 secure :-1 s=READY pgs=2916 cs=0 l=1 rev1=1 rx=0x7f30cc009960 tx=0x7f30cc00bcf0).stop 2021-02-10T11:37:09.792 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:09.790 7f30db59e700 1 -- 172.21.2.102:0/1630902347 shutdown_connections 2021-02-10T11:37:09.792 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:09.790 7f30db59e700 1 --2- 172.21.2.102:0/1630902347 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f30d4189b90 0x7f30d408cb90 unknown :-1 s=CLOSED pgs=2916 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:09.792 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:09.790 7f30db59e700 1 --2- 172.21.2.102:0/1630902347 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f30b4018730 0x7f30b401ac10 unknown :-1 s=CLOSED pgs=1279 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:09.792 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:09.790 7f30db59e700 1 --2- 172.21.2.102:0/1630902347 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f30d40a62f0 0x7f30d4189570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:09.792 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:09.790 7f30db59e700 1 --2- 172.21.2.102:0/1630902347 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f30d40a5ef0 0x7f30d4188f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:09.792 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:09.790 7f30db59e700 1 -- 172.21.2.102:0/1630902347 >> 172.21.2.102:0/1630902347 conn(0x7f30d40a6eb0 msgr2=0x7f30d4098ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:09.792 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:09.791 7f30db59e700 1 -- 172.21.2.102:0/1630902347 shutdown_connections 2021-02-10T11:37:09.792 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:09.791 7f30db59e700 1 -- 172.21.2.102:0/1630902347 wait complete. 2021-02-10T11:37:09.800 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config set mon mon_allow_pool_delete True 2021-02-10T11:37:10.232 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:10.214 7f58255e3700 1 Processor -- start 2021-02-10T11:37:10.232 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:10.215 7f58255e3700 1 -- start start 2021-02-10T11:37:10.232 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:10.215 7f58255e3700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f58200d1870 0x7f58201b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:10.232 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:10.215 7f58255e3700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f58201b0ae0 0x7f5820108940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:10.232 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:10.215 7f58255e3700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5820109a60 con 0x7f58200d1c70 2021-02-10T11:37:10.232 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:10.215 7f58255e3700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58200e5ef0 con 0x7f58201b0ae0 2021-02-10T11:37:10.233 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:10.215 7f58255e3700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5820109750 con 0x7f58200d1870 2021-02-10T11:37:10.233 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:10.215 7f581eabe700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f58201b0ae0 0x7f5820108940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:10.233 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:10.215 7f581eabe700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f58201b0ae0 0x7f5820108940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:38442/0 (socket says 172.21.2.102:38442) 2021-02-10T11:37:10.233 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:10.215 7f581eabe700 1 -- 172.21.2.102:0/3742919086 learned_addr learned my addr 172.21.2.102:0/3742919086 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:37:10.233 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:10.216 7f581eabe700 1 -- 172.21.2.102:0/3742919086 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f58200d1870 msgr2=0x7f58201b04c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:37:10.233 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:10.216 7f581eabe700 1 --2- 172.21.2.102:0/3742919086 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f58200d1870 0x7f58201b04c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:10.233 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:10.216 7f581eabe700 1 -- 172.21.2.102:0/3742919086 >> v1:172.21.2.113:6789/0 conn(0x7f58200d1c70 legacy=0x7f5820108f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:37:10.233 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:10.216 7f581eabe700 1 -- 172.21.2.102:0/3742919086 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5820088070 con 0x7f58201b0ae0 2021-02-10T11:37:10.233 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:10.216 7f581eabe700 1 --2- 172.21.2.102:0/3742919086 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f58201b0ae0 0x7f5820108940 secure :-1 s=READY pgs=2917 cs=0 l=1 rev1=1 rx=0x7f5810006700 tx=0x7f5810009b80).ready entity=mon.0 client_cookie=731fe69f1a3dc54b server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:10.233 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:10.216 7f581e2bd700 1 -- 172.21.2.102:0/3742919086 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f581000c0c0 con 0x7f58201b0ae0 2021-02-10T11:37:10.233 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:10.216 7f581e2bd700 1 -- 172.21.2.102:0/3742919086 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f581000c340 con 0x7f58201b0ae0 2021-02-10T11:37:10.233 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:10.216 7f581e2bd700 1 -- 172.21.2.102:0/3742919086 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f581000c7a0 con 0x7f58201b0ae0 2021-02-10T11:37:10.234 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:10.216 7f58255e3700 1 -- 172.21.2.102:0/3742919086 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f58201b0ae0 msgr2=0x7f5820108940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:10.234 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:10.216 7f58255e3700 1 --2- 172.21.2.102:0/3742919086 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f58201b0ae0 0x7f5820108940 secure :-1 s=READY pgs=2917 cs=0 l=1 rev1=1 rx=0x7f5810006700 tx=0x7f5810009b80).stop 2021-02-10T11:37:10.234 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:10.216 7f58255e3700 1 -- 172.21.2.102:0/3742919086 shutdown_connections 2021-02-10T11:37:10.234 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:10.216 7f58255e3700 1 --2- 172.21.2.102:0/3742919086 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f58201b0ae0 0x7f5820108940 unknown :-1 s=CLOSED pgs=2917 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:10.234 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:10.217 7f58255e3700 1 --2- 172.21.2.102:0/3742919086 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f58200d1870 0x7f58201b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:10.234 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:10.217 7f58255e3700 1 -- 172.21.2.102:0/3742919086 >> 172.21.2.102:0/3742919086 conn(0x7f58200ce420 msgr2=0x7f58200d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:10.234 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:10.217 7f58255e3700 1 -- 172.21.2.102:0/3742919086 shutdown_connections 2021-02-10T11:37:10.234 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:10.217 7f58255e3700 1 -- 172.21.2.102:0/3742919086 wait complete. 2021-02-10T11:37:10.236 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:10.231 7f58255e3700 1 Processor -- start 2021-02-10T11:37:10.236 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:10.231 7f58255e3700 1 -- start start 2021-02-10T11:37:10.236 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:10.232 7f58255e3700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f58200c3820 0x7f5820084cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:10.236 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:10.232 7f58255e3700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f58200a0290 0x7f58200852d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:10.236 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:10.232 7f58255e3700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5820086270 0x7f5820086670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:10.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:10.232 7f58255e3700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58200c0700 con 0x7f58200a0290 2021-02-10T11:37:10.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:10.232 7f58255e3700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58200858f0 con 0x7f58200c3820 2021-02-10T11:37:10.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:10.232 7f58255e3700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5820085b10 con 0x7f5820086270 2021-02-10T11:37:10.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:10.232 7f581eabe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f58200a0290 0x7f58200852d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:10.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:10.232 7f581eabe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f58200a0290 0x7f58200852d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:38450/0 (socket says 172.21.2.102:38450) 2021-02-10T11:37:10.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:10.232 7f581eabe700 1 -- 172.21.2.102:0/3468853394 learned_addr learned my addr 172.21.2.102:0/3468853394 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:37:10.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:10.232 7f581eabe700 1 -- 172.21.2.102:0/3468853394 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5820086270 msgr2=0x7f5820086670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:10.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:10.232 7f581fac0700 1 --2- 172.21.2.102:0/3468853394 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5820086270 0x7f5820086670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:10.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:10.232 7f581f2bf700 1 --2- 172.21.2.102:0/3468853394 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f58200c3820 0x7f5820084cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:10.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:10.232 7f581eabe700 1 --2- 172.21.2.102:0/3468853394 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5820086270 0x7f5820086670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:10.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:10.232 7f581eabe700 1 -- 172.21.2.102:0/3468853394 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f58200c3820 msgr2=0x7f5820084cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:10.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:10.232 7f581eabe700 1 --2- 172.21.2.102:0/3468853394 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f58200c3820 0x7f5820084cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:10.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:10.232 7f581eabe700 1 -- 172.21.2.102:0/3468853394 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5820086c90 con 0x7f58200a0290 2021-02-10T11:37:10.238 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:10.232 7f581fac0700 1 --2- 172.21.2.102:0/3468853394 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5820086270 0x7f5820086670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:37:10.238 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:10.232 7f581f2bf700 1 --2- 172.21.2.102:0/3468853394 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f58200c3820 0x7f5820084cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:37:10.238 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:10.233 7f581eabe700 1 --2- 172.21.2.102:0/3468853394 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f58200a0290 0x7f58200852d0 secure :-1 s=READY pgs=2918 cs=0 l=1 rev1=1 rx=0x7f581000b5e0 tx=0x7f581000b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:10.238 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:10.233 7f5817fff700 1 -- 172.21.2.102:0/3468853394 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f581000bd70 con 0x7f58200a0290 2021-02-10T11:37:10.238 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:10.233 7f5817fff700 1 -- 172.21.2.102:0/3468853394 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5810002030 con 0x7f58200a0290 2021-02-10T11:37:10.238 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:10.233 7f58255e3700 1 -- 172.21.2.102:0/3468853394 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5820086c90 con 0x7f58200a0290 2021-02-10T11:37:10.238 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:10.233 7f58255e3700 1 -- 172.21.2.102:0/3468853394 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58201a2f00 con 0x7f58200a0290 2021-02-10T11:37:10.238 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:10.233 7f5817fff700 1 -- 172.21.2.102:0/3468853394 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5810002550 con 0x7f58200a0290 2021-02-10T11:37:10.238 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:10.233 7f5817fff700 1 -- 172.21.2.102:0/3468853394 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f5810002210 con 0x7f58200a0290 2021-02-10T11:37:10.238 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:10.233 7f5817fff700 1 --2- 172.21.2.102:0/3468853394 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f58040186c0 0x7f580401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:10.238 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:10.234 7f5817fff700 1 -- 172.21.2.102:0/3468853394 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(162..162 src has 1..162) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f58100327f0 con 0x7f58200a0290 2021-02-10T11:37:10.238 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:10.234 7f581f2bf700 1 --2- 172.21.2.102:0/3468853394 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f58040186c0 0x7f580401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:10.238 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:10.234 7f581f2bf700 1 --2- 172.21.2.102:0/3468853394 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f58040186c0 0x7f580401abc0 crc :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:10.266 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:10.265 7f58255e3700 1 -- 172.21.2.102:0/3468853394 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5820082a70 con 0x7f58200a0290 2021-02-10T11:37:10.269 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:10.268 7f5817fff700 1 -- 172.21.2.102:0/3468853394 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f58100009b0 con 0x7f58200a0290 2021-02-10T11:37:10.416 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:10.415 7f58255e3700 1 -- 172.21.2.102:0/3468853394 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f5820082a70 con 0x7f58200a0290 2021-02-10T11:37:10.416 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:10.415 7f5817fff700 1 -- 172.21.2.102:0/3468853394 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v40)=0 v40) v1 ==== 122+0+0 (secure 0 0 0) 0x7f5810002030 con 0x7f58200a0290 2021-02-10T11:37:10.418 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:10.417 7f58255e3700 1 -- 172.21.2.102:0/3468853394 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f58040186c0 msgr2=0x7f580401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:10.418 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:10.417 7f58255e3700 1 --2- 172.21.2.102:0/3468853394 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f58040186c0 0x7f580401abc0 crc :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:10.418 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:10.417 7f58255e3700 1 -- 172.21.2.102:0/3468853394 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f58200a0290 msgr2=0x7f58200852d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:10.419 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:10.417 7f58255e3700 1 --2- 172.21.2.102:0/3468853394 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f58200a0290 0x7f58200852d0 secure :-1 s=READY pgs=2918 cs=0 l=1 rev1=1 rx=0x7f581000b5e0 tx=0x7f581000b980).stop 2021-02-10T11:37:10.419 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:10.418 7f58255e3700 1 -- 172.21.2.102:0/3468853394 shutdown_connections 2021-02-10T11:37:10.419 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:10.418 7f58255e3700 1 --2- 172.21.2.102:0/3468853394 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5820086270 0x7f5820086670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:10.419 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:10.418 7f58255e3700 1 --2- 172.21.2.102:0/3468853394 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f58200a0290 0x7f58200852d0 unknown :-1 s=CLOSED pgs=2918 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:10.420 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:10.418 7f58255e3700 1 --2- 172.21.2.102:0/3468853394 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f58040186c0 0x7f580401abc0 unknown :-1 s=CLOSED pgs=1280 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:10.420 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:10.418 7f58255e3700 1 --2- 172.21.2.102:0/3468853394 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f58200c3820 0x7f5820084cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:10.420 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:10.418 7f58255e3700 1 -- 172.21.2.102:0/3468853394 >> 172.21.2.102:0/3468853394 conn(0x7f58200872c0 msgr2=0x7f5820095700 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:10.420 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:10.418 7f58255e3700 1 -- 172.21.2.102:0/3468853394 shutdown_connections 2021-02-10T11:37:10.420 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:10.418 7f58255e3700 1 -- 172.21.2.102:0/3468853394 wait complete. 2021-02-10T11:37:10.427 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000108316 2021-02-10T11:37:10.860 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:10.842 7fc2004e6700 1 Processor -- start 2021-02-10T11:37:10.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:10.842 7fc2004e6700 1 -- start start 2021-02-10T11:37:10.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:10.843 7fc2004e6700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fc1f80d1c70 0x7fc1f80cef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:10.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:10.843 7fc2004e6700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc1f80cf580 0x7fc1f80cf980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:10.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:10.843 7fc2004e6700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1f80fe1c0 con 0x7fc1f80d1870 2021-02-10T11:37:10.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:10.843 7fc2004e6700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1f81b0cb0 con 0x7fc1f80d1c70 2021-02-10T11:37:10.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:10.843 7fc2004e6700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1f8109a60 con 0x7fc1f80cf580 2021-02-10T11:37:10.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:10.844 7fc1fdb01700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fc1f80d1c70 0x7fc1f80cef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:10.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:10.844 7fc1fdb01700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fc1f80d1c70 0x7fc1f80cef60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.113:3300/0 says I am v2:172.21.2.102:41886/0 (socket says 172.21.2.102:41886) 2021-02-10T11:37:10.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:10.844 7fc1fdb01700 1 -- 172.21.2.102:0/4240177498 learned_addr learned my addr 172.21.2.102:0/4240177498 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:37:10.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:10.844 7fc1feb03700 1 --2- 172.21.2.102:0/4240177498 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc1f80cf580 0x7fc1f80cf980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:10.862 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:10.844 7fc1fdb01700 1 -- 172.21.2.102:0/4240177498 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc1f80cf580 msgr2=0x7fc1f80cf980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:37:10.862 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:10.844 7fc1fdb01700 1 --2- 172.21.2.102:0/4240177498 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc1f80cf580 0x7fc1f80cf980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:10.862 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:10.844 7fc1fdb01700 1 -- 172.21.2.102:0/4240177498 >> v1:172.21.2.113:6789/0 conn(0x7fc1f80d1870 legacy=0x7fc1f81b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:10.862 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:10.844 7fc1fdb01700 1 -- 172.21.2.102:0/4240177498 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1f8088070 con 0x7fc1f80d1c70 2021-02-10T11:37:10.862 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:10.844 7fc1fdb01700 1 --2- 172.21.2.102:0/4240177498 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc1f80d1c70 0x7fc1f80cef60 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 rx=0x7fc1ec006700 tx=0x7fc1ec009b80).ready entity=mon.1 client_cookie=3198c17601280448 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:10.862 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:10.845 7fc1fd300700 1 -- 172.21.2.102:0/4240177498 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc1ec00c0c0 con 0x7fc1f80d1c70 2021-02-10T11:37:10.862 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:10.845 7fc1fd300700 1 -- 172.21.2.102:0/4240177498 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1ec00c340 con 0x7fc1f80d1c70 2021-02-10T11:37:10.862 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:10.845 7fc1fd300700 1 -- 172.21.2.102:0/4240177498 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc1ec00c860 con 0x7fc1f80d1c70 2021-02-10T11:37:10.862 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:10.845 7fc2004e6700 1 -- 172.21.2.102:0/4240177498 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc1f80d1c70 msgr2=0x7fc1f80cef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:10.862 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:10.845 7fc2004e6700 1 --2- 172.21.2.102:0/4240177498 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc1f80d1c70 0x7fc1f80cef60 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 rx=0x7fc1ec006700 tx=0x7fc1ec009b80).stop 2021-02-10T11:37:10.862 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:10.846 7fc2004e6700 1 -- 172.21.2.102:0/4240177498 shutdown_connections 2021-02-10T11:37:10.862 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:10.846 7fc2004e6700 1 --2- 172.21.2.102:0/4240177498 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc1f80cf580 0x7fc1f80cf980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:10.863 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:10.846 7fc2004e6700 1 --2- 172.21.2.102:0/4240177498 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc1f80d1c70 0x7fc1f80cef60 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:10.863 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:10.846 7fc2004e6700 1 -- 172.21.2.102:0/4240177498 >> 172.21.2.102:0/4240177498 conn(0x7fc1f80ce420 msgr2=0x7fc1f80d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:10.863 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:10.846 7fc2004e6700 1 -- 172.21.2.102:0/4240177498 shutdown_connections 2021-02-10T11:37:10.863 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:10.846 7fc2004e6700 1 -- 172.21.2.102:0/4240177498 wait complete. 2021-02-10T11:37:10.864 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:10.860 7fc2004e6700 1 Processor -- start 2021-02-10T11:37:10.865 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:10.860 7fc2004e6700 1 -- start start 2021-02-10T11:37:10.865 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:10.860 7fc2004e6700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc1f80b04d0 0x7fc1f81a7a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:10.865 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:10.860 7fc2004e6700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc1f81a8090 0x7fc1f80c23a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:10.865 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:10.860 7fc2004e6700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc1f808d680 0x7fc1f80c29c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:10.865 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:10.860 7fc2004e6700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1f80b0a00 con 0x7fc1f81a8090 2021-02-10T11:37:10.865 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:10.860 7fc2004e6700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1f80c2fe0 con 0x7fc1f808d680 2021-02-10T11:37:10.865 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:10.860 7fc2004e6700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1f80c3200 con 0x7fc1f80b04d0 2021-02-10T11:37:10.865 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:10.860 7fc1fdb01700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc1f81a8090 0x7fc1f80c23a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:10.865 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:10.861 7fc1fdb01700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc1f81a8090 0x7fc1f80c23a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:38462/0 (socket says 172.21.2.102:38462) 2021-02-10T11:37:10.865 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:10.861 7fc1fdb01700 1 -- 172.21.2.102:0/1322065518 learned_addr learned my addr 172.21.2.102:0/1322065518 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:37:10.865 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:10.861 7fc1fe302700 1 --2- 172.21.2.102:0/1322065518 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc1f80b04d0 0x7fc1f81a7a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:10.865 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:10.861 7fc1feb03700 1 --2- 172.21.2.102:0/1322065518 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc1f808d680 0x7fc1f80c29c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:10.865 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:10.861 7fc1fdb01700 1 -- 172.21.2.102:0/1322065518 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc1f80b04d0 msgr2=0x7fc1f81a7a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:10.866 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:10.861 7fc1fdb01700 1 --2- 172.21.2.102:0/1322065518 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc1f80b04d0 0x7fc1f81a7a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:10.866 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:10.861 7fc1fdb01700 1 -- 172.21.2.102:0/1322065518 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc1f808d680 msgr2=0x7fc1f80c29c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:10.866 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:10.861 7fc1fdb01700 1 --2- 172.21.2.102:0/1322065518 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc1f808d680 0x7fc1f80c29c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:10.866 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:10.861 7fc1fdb01700 1 -- 172.21.2.102:0/1322065518 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1f808e080 con 0x7fc1f81a8090 2021-02-10T11:37:10.866 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:10.861 7fc1fdb01700 1 --2- 172.21.2.102:0/1322065518 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc1f81a8090 0x7fc1f80c23a0 secure :-1 s=READY pgs=2919 cs=0 l=1 rev1=1 rx=0x7fc1ec00b650 tx=0x7fc1ec00b9f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:10.866 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:10.861 7fc1eaffd700 1 -- 172.21.2.102:0/1322065518 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc1ec00bde0 con 0x7fc1f81a8090 2021-02-10T11:37:10.866 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:10.861 7fc1eaffd700 1 -- 172.21.2.102:0/1322065518 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1ec006390 con 0x7fc1f81a8090 2021-02-10T11:37:10.866 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:10.861 7fc2004e6700 1 -- 172.21.2.102:0/1322065518 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1f808e080 con 0x7fc1f81a8090 2021-02-10T11:37:10.866 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:10.861 7fc1eaffd700 1 -- 172.21.2.102:0/1322065518 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc1ec0022b0 con 0x7fc1f81a8090 2021-02-10T11:37:10.866 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:10.861 7fc2004e6700 1 -- 172.21.2.102:0/1322065518 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1f808e610 con 0x7fc1f81a8090 2021-02-10T11:37:10.866 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:10.862 7fc1eaffd700 1 -- 172.21.2.102:0/1322065518 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fc1ec000c90 con 0x7fc1f81a8090 2021-02-10T11:37:10.866 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:10.862 7fc1eaffd700 1 --2- 172.21.2.102:0/1322065518 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc1e00186d0 0x7fc1e001abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:10.866 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:10.862 7fc1eaffd700 1 -- 172.21.2.102:0/1322065518 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(162..162 src has 1..162) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fc1ec032670 con 0x7fc1f81a8090 2021-02-10T11:37:10.867 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:10.862 7fc1fe302700 1 --2- 172.21.2.102:0/1322065518 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc1e00186d0 0x7fc1e001abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:10.867 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:10.863 7fc1fe302700 1 --2- 172.21.2.102:0/1322065518 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc1e00186d0 0x7fc1e001abd0 crc :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:10.894 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:10.893 7fc2004e6700 1 -- 172.21.2.102:0/1322065518 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1f8082a70 con 0x7fc1f81a8090 2021-02-10T11:37:10.897 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:10.896 7fc1eaffd700 1 -- 172.21.2.102:0/1322065518 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fc1ec006390 con 0x7fc1f81a8090 2021-02-10T11:37:11.022 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:11.021 7fc2004e6700 1 -- 172.21.2.102:0/1322065518 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000108316", "target": ["mgr", ""]}) v1 -- 0x7fc1f80c2fe0 con 0x7fc1e00186d0 2021-02-10T11:37:11.027 INFO:tasks.ceph.mgr.y.gibba002.stderr:2021-02-10 11:37:11.026 7fd94bc53700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:37:11.027 INFO:tasks.ceph.mgr.y.gibba002.stderr:2021-02-10 11:37:11.026 7fd94bc53700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:37:11.027 INFO:tasks.ceph.mgr.y.gibba002.stderr:2021-02-10 11:37:11.026 7fd94bc53700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:37:11.027 INFO:tasks.ceph.mgr.y.gibba002.stderr:2021-02-10 11:37:11.026 7fd94bc53700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:37:11.027 INFO:tasks.ceph.mgr.y.gibba002.stderr:2021-02-10 11:37:11.026 7fd94bc53700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:37:11.059 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:11.058 7fc1eaffd700 1 -- 172.21.2.102:0/1322065518 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fc1f4006550 con 0x7fc1e00186d0 2021-02-10T11:37:11.088 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:11.086 7fc2004e6700 1 -- 172.21.2.102:0/1322065518 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc1e00186d0 msgr2=0x7fc1e001abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:11.088 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:11.086 7fc2004e6700 1 --2- 172.21.2.102:0/1322065518 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc1e00186d0 0x7fc1e001abd0 crc :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:11.088 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:11.087 7fc2004e6700 1 -- 172.21.2.102:0/1322065518 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc1f81a8090 msgr2=0x7fc1f80c23a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:11.088 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:11.087 7fc2004e6700 1 --2- 172.21.2.102:0/1322065518 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc1f81a8090 0x7fc1f80c23a0 secure :-1 s=READY pgs=2919 cs=0 l=1 rev1=1 rx=0x7fc1ec00b650 tx=0x7fc1ec00b9f0).stop 2021-02-10T11:37:11.088 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:11.087 7fc2004e6700 1 -- 172.21.2.102:0/1322065518 shutdown_connections 2021-02-10T11:37:11.089 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:11.087 7fc2004e6700 1 --2- 172.21.2.102:0/1322065518 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc1e00186d0 0x7fc1e001abd0 unknown :-1 s=CLOSED pgs=1281 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:11.089 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:11.087 7fc2004e6700 1 --2- 172.21.2.102:0/1322065518 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc1f808d680 0x7fc1f80c29c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:11.089 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:11.087 7fc2004e6700 1 --2- 172.21.2.102:0/1322065518 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc1f81a8090 0x7fc1f80c23a0 unknown :-1 s=CLOSED pgs=2919 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:11.089 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:11.087 7fc2004e6700 1 --2- 172.21.2.102:0/1322065518 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc1f80b04d0 0x7fc1f81a7a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:11.089 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:11.087 7fc2004e6700 1 -- 172.21.2.102:0/1322065518 >> 172.21.2.102:0/1322065518 conn(0x7fc1f80be530 msgr2=0x7fc1f809f1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:11.089 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:11.087 7fc2004e6700 1 -- 172.21.2.102:0/1322065518 shutdown_connections 2021-02-10T11:37:11.089 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:11.087 7fc2004e6700 1 -- 172.21.2.102:0/1322065518 wait complete. 2021-02-10T11:37:11.096 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume snapshot create cephfs subvolume_0000000000108316 snapshot_0000000000138436 2021-02-10T11:37:11.528 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:11.511 7f3c67ea9700 1 Processor -- start 2021-02-10T11:37:11.528 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:11.511 7f3c67ea9700 1 -- start start 2021-02-10T11:37:11.528 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:11.511 7f3c67ea9700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f3c600d1880 0x7f3c601b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:11.528 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:11.511 7f3c67ea9700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f3c601b0af0 0x7f3c60108950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:11.528 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:11.511 7f3c67ea9700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3c600d1c80 0x7f3c60108f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:11.528 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:11.511 7f3c67ea9700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c600fe1d0 con 0x7f3c601b0af0 2021-02-10T11:37:11.528 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:11.511 7f3c67ea9700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c60109a70 con 0x7f3c600d1880 2021-02-10T11:37:11.529 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:11.511 7f3c67ea9700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c60088070 con 0x7f3c600d1c80 2021-02-10T11:37:11.529 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:11.512 7f3c654c4700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f3c601b0af0 0x7f3c60108950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:11.529 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:11.512 7f3c654c4700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f3c601b0af0 0x7f3c60108950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:38486/0 (socket says 172.21.2.102:38486) 2021-02-10T11:37:11.529 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:11.512 7f3c654c4700 1 -- 172.21.2.102:0/4270016853 learned_addr learned my addr 172.21.2.102:0/4270016853 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:37:11.529 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:11.512 7f3c654c4700 1 -- 172.21.2.102:0/4270016853 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3c600d1c80 msgr2=0x7f3c60108f70 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:37:11.529 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:11.512 7f3c654c4700 1 --2- 172.21.2.102:0/4270016853 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3c600d1c80 0x7f3c60108f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:11.529 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:11.512 7f3c654c4700 1 -- 172.21.2.102:0/4270016853 >> v2:172.21.2.113:3300/0 conn(0x7f3c600d1880 msgr2=0x7f3c601b04d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:37:11.529 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:11.512 7f3c654c4700 1 --2- 172.21.2.102:0/4270016853 >> v2:172.21.2.113:3300/0 conn(0x7f3c600d1880 0x7f3c601b04d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:11.529 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:11.512 7f3c654c4700 1 -- 172.21.2.102:0/4270016853 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c60109590 con 0x7f3c601b0af0 2021-02-10T11:37:11.529 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:11.512 7f3c654c4700 1 --2- 172.21.2.102:0/4270016853 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3c601b0af0 0x7f3c60108950 secure :-1 s=READY pgs=2922 cs=0 l=1 rev1=1 rx=0x7f3c5c006700 tx=0x7f3c5c009b80).ready entity=mon.0 client_cookie=dfee7e6c45a5870d server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:11.530 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:11.512 7f3c64cc3700 1 -- 172.21.2.102:0/4270016853 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3c5c00c0c0 con 0x7f3c601b0af0 2021-02-10T11:37:11.530 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:11.512 7f3c64cc3700 1 -- 172.21.2.102:0/4270016853 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c5c00c340 con 0x7f3c601b0af0 2021-02-10T11:37:11.530 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:11.512 7f3c64cc3700 1 -- 172.21.2.102:0/4270016853 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3c5c00c860 con 0x7f3c601b0af0 2021-02-10T11:37:11.530 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:11.512 7f3c67ea9700 1 -- 172.21.2.102:0/4270016853 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3c601b0af0 msgr2=0x7f3c60108950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:11.530 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:11.512 7f3c67ea9700 1 --2- 172.21.2.102:0/4270016853 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3c601b0af0 0x7f3c60108950 secure :-1 s=READY pgs=2922 cs=0 l=1 rev1=1 rx=0x7f3c5c006700 tx=0x7f3c5c009b80).stop 2021-02-10T11:37:11.530 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:11.513 7f3c67ea9700 1 -- 172.21.2.102:0/4270016853 shutdown_connections 2021-02-10T11:37:11.530 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:11.513 7f3c67ea9700 1 --2- 172.21.2.102:0/4270016853 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3c600d1c80 0x7f3c60108f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:11.530 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:11.513 7f3c67ea9700 1 --2- 172.21.2.102:0/4270016853 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3c601b0af0 0x7f3c60108950 unknown :-1 s=CLOSED pgs=2922 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:11.530 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:11.513 7f3c67ea9700 1 --2- 172.21.2.102:0/4270016853 >> v2:172.21.2.113:3300/0 conn(0x7f3c600d1880 0x7f3c601b04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:11.530 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:11.513 7f3c67ea9700 1 -- 172.21.2.102:0/4270016853 >> 172.21.2.102:0/4270016853 conn(0x7f3c600ce430 msgr2=0x7f3c600d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:11.530 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:11.513 7f3c67ea9700 1 -- 172.21.2.102:0/4270016853 shutdown_connections 2021-02-10T11:37:11.531 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:11.513 7f3c67ea9700 1 -- 172.21.2.102:0/4270016853 wait complete. 2021-02-10T11:37:11.532 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:11.527 7f3c67ea9700 1 Processor -- start 2021-02-10T11:37:11.532 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:11.527 7f3c67ea9700 1 -- start start 2021-02-10T11:37:11.532 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:11.528 7f3c67ea9700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3c60190df0 0x7f3c601b7ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:11.532 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:11.528 7f3c67ea9700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3c6019ba20 0x7f3c601b87c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:11.532 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:11.528 7f3c67ea9700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3c6019be20 0x7f3c6018f830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:11.533 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:11.528 7f3c67ea9700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c601b8240 con 0x7f3c6019be20 2021-02-10T11:37:11.533 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:11.528 7f3c67ea9700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c601911f0 con 0x7f3c60190df0 2021-02-10T11:37:11.533 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:11.528 7f3c67ea9700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c601908a0 con 0x7f3c6019ba20 2021-02-10T11:37:11.533 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:11.528 7f3c664c6700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3c6019be20 0x7f3c6018f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:11.533 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:11.528 7f3c664c6700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3c6019be20 0x7f3c6018f830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:38492/0 (socket says 172.21.2.102:38492) 2021-02-10T11:37:11.533 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:11.528 7f3c664c6700 1 -- 172.21.2.102:0/1551494377 learned_addr learned my addr 172.21.2.102:0/1551494377 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:37:11.533 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:11.528 7f3c664c6700 1 -- 172.21.2.102:0/1551494377 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3c6019ba20 msgr2=0x7f3c601b87c0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:37:11.533 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:11.528 7f3c664c6700 1 --2- 172.21.2.102:0/1551494377 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3c6019ba20 0x7f3c601b87c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:11.533 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:11.528 7f3c664c6700 1 -- 172.21.2.102:0/1551494377 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3c60190df0 msgr2=0x7f3c601b7ac0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:37:11.533 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:11.528 7f3c664c6700 1 --2- 172.21.2.102:0/1551494377 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3c60190df0 0x7f3c601b7ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:11.533 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:11.528 7f3c664c6700 1 -- 172.21.2.102:0/1551494377 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c601b8de0 con 0x7f3c6019be20 2021-02-10T11:37:11.533 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:11.528 7f3c664c6700 1 --2- 172.21.2.102:0/1551494377 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3c6019be20 0x7f3c6018f830 secure :-1 s=READY pgs=2923 cs=0 l=1 rev1=1 rx=0x7f3c58009960 tx=0x7f3c5800bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:11.533 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:11.528 7f3c527fc700 1 -- 172.21.2.102:0/1551494377 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3c58000d10 con 0x7f3c6019be20 2021-02-10T11:37:11.534 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:11.528 7f3c527fc700 1 -- 172.21.2.102:0/1551494377 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c58003030 con 0x7f3c6019be20 2021-02-10T11:37:11.534 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:11.528 7f3c67ea9700 1 -- 172.21.2.102:0/1551494377 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c601b8de0 con 0x7f3c6019be20 2021-02-10T11:37:11.534 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:11.529 7f3c527fc700 1 -- 172.21.2.102:0/1551494377 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3c58001590 con 0x7f3c6019be20 2021-02-10T11:37:11.534 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:11.529 7f3c67ea9700 1 -- 172.21.2.102:0/1551494377 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c601b8de0 con 0x7f3c6019be20 2021-02-10T11:37:11.534 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:11.529 7f3c527fc700 1 -- 172.21.2.102:0/1551494377 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f3c58000d10 con 0x7f3c6019be20 2021-02-10T11:37:11.534 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:11.529 7f3c527fc700 1 --2- 172.21.2.102:0/1551494377 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3c480186c0 0x7f3c4801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:11.534 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:11.529 7f3c527fc700 1 -- 172.21.2.102:0/1551494377 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(162..162 src has 1..162) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f3c58032b30 con 0x7f3c6019be20 2021-02-10T11:37:11.534 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:11.529 7f3c65cc5700 1 --2- 172.21.2.102:0/1551494377 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3c480186c0 0x7f3c4801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:11.534 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:11.530 7f3c65cc5700 1 --2- 172.21.2.102:0/1551494377 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3c480186c0 0x7f3c4801abc0 crc :-1 s=READY pgs=1282 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:11.562 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:11.561 7f3c67ea9700 1 -- 172.21.2.102:0/1551494377 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c60082aa0 con 0x7f3c6019be20 2021-02-10T11:37:11.565 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:11.564 7f3c527fc700 1 -- 172.21.2.102:0/1551494377 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f3c5800be30 con 0x7f3c6019be20 2021-02-10T11:37:11.693 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:11.692 7f3c67ea9700 1 -- 172.21.2.102:0/1551494377 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume snapshot create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000108316", "target": ["mgr", ""], "snap_name": "snapshot_0000000000138436"}) v1 -- 0x7f3c60082aa0 con 0x7f3c480186c0 2021-02-10T11:37:11.704 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:11.703 7f3c527fc700 1 -- 172.21.2.102:0/1551494377 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f3c540009b0 con 0x7f3c480186c0 2021-02-10T11:37:11.710 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:11.709 7f3c67ea9700 1 -- 172.21.2.102:0/1551494377 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3c480186c0 msgr2=0x7f3c4801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:11.710 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:11.709 7f3c67ea9700 1 --2- 172.21.2.102:0/1551494377 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3c480186c0 0x7f3c4801abc0 crc :-1 s=READY pgs=1282 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:11.710 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:11.709 7f3c67ea9700 1 -- 172.21.2.102:0/1551494377 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3c6019be20 msgr2=0x7f3c6018f830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:11.711 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:11.709 7f3c67ea9700 1 --2- 172.21.2.102:0/1551494377 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3c6019be20 0x7f3c6018f830 secure :-1 s=READY pgs=2923 cs=0 l=1 rev1=1 rx=0x7f3c58009960 tx=0x7f3c5800bcf0).stop 2021-02-10T11:37:11.711 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:11.709 7f3c67ea9700 1 -- 172.21.2.102:0/1551494377 shutdown_connections 2021-02-10T11:37:11.711 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:11.709 7f3c67ea9700 1 --2- 172.21.2.102:0/1551494377 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3c6019be20 0x7f3c6018f830 unknown :-1 s=CLOSED pgs=2923 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:11.711 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:11.709 7f3c67ea9700 1 --2- 172.21.2.102:0/1551494377 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3c6019ba20 0x7f3c601b87c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:11.711 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:11.709 7f3c67ea9700 1 --2- 172.21.2.102:0/1551494377 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3c480186c0 0x7f3c4801abc0 unknown :-1 s=CLOSED pgs=1282 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:11.711 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:11.709 7f3c67ea9700 1 --2- 172.21.2.102:0/1551494377 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3c60190df0 0x7f3c601b7ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:11.711 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:11.709 7f3c67ea9700 1 -- 172.21.2.102:0/1551494377 >> 172.21.2.102:0/1551494377 conn(0x7f3c6019f940 msgr2=0x7f3c6018c290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:11.711 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:11.710 7f3c67ea9700 1 -- 172.21.2.102:0/1551494377 shutdown_connections 2021-02-10T11:37:11.712 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:11.710 7f3c67ea9700 1 -- 172.21.2.102:0/1551494377 wait complete. 2021-02-10T11:37:11.718 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume snapshot rm cephfs subvolume_0000000000108316 snapshot_0000000000138436 2021-02-10T11:37:12.151 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:12.133 7f4ec4571700 1 Processor -- start 2021-02-10T11:37:12.152 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:12.133 7f4ec4571700 1 -- start start 2021-02-10T11:37:12.152 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:12.134 7f4ec4571700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f4ebc0d1880 0x7f4ebc1b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:12.152 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:12.134 7f4ec4571700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ebc0fe1d0 con 0x7f4ebc1b0af0 2021-02-10T11:37:12.152 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:12.134 7f4ec4571700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ebc109a70 con 0x7f4ebc0cff80 2021-02-10T11:37:12.152 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:12.134 7f4ec4571700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ebc088070 con 0x7f4ebc0d1880 2021-02-10T11:37:12.152 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:12.134 7f4ec238d700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f4ebc0d1880 0x7f4ebc1b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:12.152 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:12.134 7f4ec1b8c700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f4ebc1b0af0 0x7f4ebc108950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:58958/0 (socket says 172.21.2.102:58958) 2021-02-10T11:37:12.152 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:12.134 7f4ec1b8c700 1 -- 172.21.2.102:0/3283180478 learned_addr learned my addr 172.21.2.102:0/3283180478 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:37:12.152 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:12.135 7f4ec238d700 1 -- 172.21.2.102:0/3283180478 >> v1:172.21.2.113:6789/0 conn(0x7f4ebc0cff80 legacy=0x7f4ebc109140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:12.152 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:12.135 7f4ec238d700 1 -- 172.21.2.102:0/3283180478 >> v1:172.21.2.102:6789/0 conn(0x7f4ebc1b0af0 legacy=0x7f4ebc108950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:12.152 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:12.135 7f4ec238d700 1 -- 172.21.2.102:0/3283180478 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ebc088260 con 0x7f4ebc0d1880 2021-02-10T11:37:12.152 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:12.135 7f4ec238d700 1 --2- 172.21.2.102:0/3283180478 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4ebc0d1880 0x7f4ebc1b04d0 secure :-1 s=READY pgs=2925 cs=0 l=1 rev1=1 rx=0x7f4eb8006650 tx=0x7f4eb8009a80).ready entity=mon.0 client_cookie=3362573cad11e4fe server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:12.152 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:12.135 7f4ec138b700 1 -- 172.21.2.102:0/3283180478 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4eb80009b0 con 0x7f4ebc0d1880 2021-02-10T11:37:12.153 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:12.135 7f4ec138b700 1 -- 172.21.2.102:0/3283180478 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4eb800c0c0 con 0x7f4ebc0d1880 2021-02-10T11:37:12.153 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:12.135 7f4ec138b700 1 -- 172.21.2.102:0/3283180478 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4eb800c5e0 con 0x7f4ebc0d1880 2021-02-10T11:37:12.153 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:12.135 7f4ec4571700 1 -- 172.21.2.102:0/3283180478 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4ebc0d1880 msgr2=0x7f4ebc1b04d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:12.153 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:12.135 7f4ec4571700 1 --2- 172.21.2.102:0/3283180478 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4ebc0d1880 0x7f4ebc1b04d0 secure :-1 s=READY pgs=2925 cs=0 l=1 rev1=1 rx=0x7f4eb8006650 tx=0x7f4eb8009a80).stop 2021-02-10T11:37:12.153 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:12.136 7f4ec4571700 1 -- 172.21.2.102:0/3283180478 shutdown_connections 2021-02-10T11:37:12.153 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:12.136 7f4ec4571700 1 --2- 172.21.2.102:0/3283180478 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4ebc0d1880 0x7f4ebc1b04d0 unknown :-1 s=CLOSED pgs=2925 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:12.153 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:12.136 7f4ec4571700 1 -- 172.21.2.102:0/3283180478 >> 172.21.2.102:0/3283180478 conn(0x7f4ebc0ce430 msgr2=0x7f4ebc0d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:12.153 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:12.136 7f4ec4571700 1 -- 172.21.2.102:0/3283180478 shutdown_connections 2021-02-10T11:37:12.154 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:12.136 7f4ec4571700 1 -- 172.21.2.102:0/3283180478 wait complete. 2021-02-10T11:37:12.155 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:12.151 7f4ec4571700 1 Processor -- start 2021-02-10T11:37:12.155 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:12.151 7f4ec4571700 1 -- start start 2021-02-10T11:37:12.155 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:12.151 7f4ec4571700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4ebc1b5bc0 0x7f4ebc0aeac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:12.156 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:12.151 7f4ec4571700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4ebc0c0d90 0x7f4ebc0c1190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:12.156 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:12.151 7f4ec4571700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4ebc0b1310 0x7f4ebc0b1710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:12.156 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:12.151 7f4ec4571700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ebc0c17b0 con 0x7f4ebc0c0d90 2021-02-10T11:37:12.156 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:12.151 7f4ec4571700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ebc0af0e0 con 0x7f4ebc0b1310 2021-02-10T11:37:12.156 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:12.151 7f4ec4571700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ebc1b5fc0 con 0x7f4ebc1b5bc0 2021-02-10T11:37:12.156 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:12.151 7f4ec1b8c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4ebc0c0d90 0x7f4ebc0c1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:12.156 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:12.151 7f4ec238d700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4ebc1b5bc0 0x7f4ebc0aeac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:12.156 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:12.152 7f4ec238d700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4ebc1b5bc0 0x7f4ebc0aeac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.113:3301/0 says I am v2:172.21.2.102:44042/0 (socket says 172.21.2.102:44042) 2021-02-10T11:37:12.156 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:12.152 7f4ec238d700 1 -- 172.21.2.102:0/3370590205 learned_addr learned my addr 172.21.2.102:0/3370590205 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:37:12.156 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:12.152 7f4ec2b8e700 1 --2- 172.21.2.102:0/3370590205 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4ebc0b1310 0x7f4ebc0b1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:12.156 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:12.152 7f4ec1b8c700 1 -- 172.21.2.102:0/3370590205 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4ebc1b5bc0 msgr2=0x7f4ebc0aeac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:12.156 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:12.152 7f4ec1b8c700 1 --2- 172.21.2.102:0/3370590205 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4ebc1b5bc0 0x7f4ebc0aeac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:12.156 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:12.152 7f4ec1b8c700 1 -- 172.21.2.102:0/3370590205 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4ebc0b1310 msgr2=0x7f4ebc0b1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:12.157 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:12.152 7f4ec1b8c700 1 --2- 172.21.2.102:0/3370590205 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4ebc0b1310 0x7f4ebc0b1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:12.157 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:12.152 7f4ec1b8c700 1 -- 172.21.2.102:0/3370590205 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ebc0b1d30 con 0x7f4ebc0c0d90 2021-02-10T11:37:12.157 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:12.152 7f4ec1b8c700 1 --2- 172.21.2.102:0/3370590205 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4ebc0c0d90 0x7f4ebc0c1190 secure :-1 s=READY pgs=2926 cs=0 l=1 rev1=1 rx=0x7f4eb000a4d0 tx=0x7f4eb000a870).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:12.157 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:12.152 7f4eaeffd700 1 -- 172.21.2.102:0/3370590205 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4eb0000e20 con 0x7f4ebc0c0d90 2021-02-10T11:37:12.157 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:12.152 7f4eaeffd700 1 -- 172.21.2.102:0/3370590205 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4eb000ac60 con 0x7f4ebc0c0d90 2021-02-10T11:37:12.157 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:12.152 7f4ec4571700 1 -- 172.21.2.102:0/3370590205 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ebc0b1d30 con 0x7f4ebc0c0d90 2021-02-10T11:37:12.157 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:12.152 7f4eaeffd700 1 -- 172.21.2.102:0/3370590205 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4eb00043c0 con 0x7f4ebc0c0d90 2021-02-10T11:37:12.157 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:12.153 7f4ec4571700 1 -- 172.21.2.102:0/3370590205 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ebc0b1d30 con 0x7f4ebc0c0d90 2021-02-10T11:37:12.157 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:12.153 7f4eaeffd700 1 -- 172.21.2.102:0/3370590205 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4eb0000ce0 con 0x7f4ebc0c0d90 2021-02-10T11:37:12.157 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:12.153 7f4eaeffd700 1 --2- 172.21.2.102:0/3370590205 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4ea40186c0 0x7f4ea401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:12.157 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:12.153 7f4eaeffd700 1 -- 172.21.2.102:0/3370590205 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(162..162 src has 1..162) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f4eb0032520 con 0x7f4ebc0c0d90 2021-02-10T11:37:12.157 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:12.153 7f4ec238d700 1 --2- 172.21.2.102:0/3370590205 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4ea40186c0 0x7f4ea401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:12.157 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:12.154 7f4ec238d700 1 --2- 172.21.2.102:0/3370590205 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4ea40186c0 0x7f4ea401abc0 crc :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:12.185 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:12.184 7f4ec4571700 1 -- 172.21.2.102:0/3370590205 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ebc0c17b0 con 0x7f4ebc0c0d90 2021-02-10T11:37:12.188 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:12.187 7f4eaeffd700 1 -- 172.21.2.102:0/3370590205 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f4eb00009b0 con 0x7f4ebc0c0d90 2021-02-10T11:37:12.314 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:12.313 7f4ec4571700 1 -- 172.21.2.102:0/3370590205 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume snapshot rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000108316", "target": ["mgr", ""], "snap_name": "snapshot_0000000000138436"}) v1 -- 0x7f4ebc0af0e0 con 0x7f4ea40186c0 2021-02-10T11:37:12.326 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:12.325 7f4eaeffd700 1 -- 172.21.2.102:0/3370590205 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f4eb8006540 con 0x7f4ea40186c0 2021-02-10T11:37:12.331 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:12.330 7f4ec4571700 1 -- 172.21.2.102:0/3370590205 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4ea40186c0 msgr2=0x7f4ea401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:12.331 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:12.330 7f4ec4571700 1 --2- 172.21.2.102:0/3370590205 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4ea40186c0 0x7f4ea401abc0 crc :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:12.331 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:12.330 7f4ec4571700 1 -- 172.21.2.102:0/3370590205 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4ebc0c0d90 msgr2=0x7f4ebc0c1190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:12.331 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:12.330 7f4ec4571700 1 --2- 172.21.2.102:0/3370590205 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4ebc0c0d90 0x7f4ebc0c1190 secure :-1 s=READY pgs=2926 cs=0 l=1 rev1=1 rx=0x7f4eb000a4d0 tx=0x7f4eb000a870).stop 2021-02-10T11:37:12.332 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:12.330 7f4ec4571700 1 -- 172.21.2.102:0/3370590205 shutdown_connections 2021-02-10T11:37:12.332 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:12.330 7f4ec4571700 1 --2- 172.21.2.102:0/3370590205 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4ea40186c0 0x7f4ea401abc0 unknown :-1 s=CLOSED pgs=1283 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:12.332 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:12.330 7f4ec4571700 1 --2- 172.21.2.102:0/3370590205 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4ebc0b1310 0x7f4ebc0b1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:12.332 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:12.330 7f4ec4571700 1 --2- 172.21.2.102:0/3370590205 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4ebc0c0d90 0x7f4ebc0c1190 unknown :-1 s=CLOSED pgs=2926 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:12.332 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:12.330 7f4ec4571700 1 --2- 172.21.2.102:0/3370590205 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4ebc1b5bc0 0x7f4ebc0aeac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:12.332 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:12.330 7f4ec4571700 1 -- 172.21.2.102:0/3370590205 >> 172.21.2.102:0/3370590205 conn(0x7f4ebc0bd400 msgr2=0x7f4ebc0804e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:12.332 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:12.330 7f4ec4571700 1 -- 172.21.2.102:0/3370590205 shutdown_connections 2021-02-10T11:37:12.332 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:12.330 7f4ec4571700 1 -- 172.21.2.102:0/3370590205 wait complete. 2021-02-10T11:37:12.339 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume snapshot rm cephfs subvolume_0000000000108316 snapshot_0000000000138436 2021-02-10T11:37:12.773 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:12.755 7f24b9eee700 1 Processor -- start 2021-02-10T11:37:12.773 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:12.755 7f24b9eee700 1 -- start start 2021-02-10T11:37:12.773 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:12.756 7f24b9eee700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f24b40d1880 0x7f24b41b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:12.774 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:12.756 7f24b9eee700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f24b41b0af0 0x7f24b4108950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:12.774 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:12.756 7f24b9eee700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24b40fe1d0 con 0x7f24b40d1c80 2021-02-10T11:37:12.774 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:12.756 7f24b9eee700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24b4109a70 con 0x7f24b40d1880 2021-02-10T11:37:12.774 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:12.756 7f24b9eee700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24b4109760 con 0x7f24b41b0af0 2021-02-10T11:37:12.774 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:12.756 7f24b3fff700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f24b40d1c80 0x7f24b4108f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:58970/0 (socket says 172.21.2.102:58970) 2021-02-10T11:37:12.774 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:12.756 7f24b3fff700 1 -- 172.21.2.102:0/841647048 learned_addr learned my addr 172.21.2.102:0/841647048 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:37:12.774 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:12.756 7f24b2ffd700 1 --2- 172.21.2.102:0/841647048 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f24b41b0af0 0x7f24b4108950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:12.774 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:12.756 7f24b37fe700 1 --2- 172.21.2.102:0/841647048 >> v2:172.21.2.113:3300/0 conn(0x7f24b40d1880 0x7f24b41b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:12.774 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:12.757 7f24b27fc700 1 -- 172.21.2.102:0/841647048 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 394377170 0 0) 0x7f24ac000f90 con 0x7f24b40d1c80 2021-02-10T11:37:12.774 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:12.757 7f24b27fc700 1 -- 172.21.2.102:0/841647048 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f249c002fe0 con 0x7f24b40d1c80 2021-02-10T11:37:12.774 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:12.757 7f24b2ffd700 1 -- 172.21.2.102:0/841647048 >> v2:172.21.2.113:3300/0 conn(0x7f24b40d1880 msgr2=0x7f24b41b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:37:12.774 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:12.757 7f24b2ffd700 1 --2- 172.21.2.102:0/841647048 >> v2:172.21.2.113:3300/0 conn(0x7f24b40d1880 0x7f24b41b04d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:12.775 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:12.757 7f24b2ffd700 1 -- 172.21.2.102:0/841647048 >> v1:172.21.2.102:6789/0 conn(0x7f24b40d1c80 legacy=0x7f24b4108f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:12.775 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:12.757 7f24b2ffd700 1 -- 172.21.2.102:0/841647048 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24b4088070 con 0x7f24b41b0af0 2021-02-10T11:37:12.775 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:12.757 7f24b37fe700 1 --2- 172.21.2.102:0/841647048 >> v2:172.21.2.113:3300/0 conn(0x7f24b40d1880 0x7f24b41b04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:37:12.775 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:12.757 7f24b2ffd700 1 --2- 172.21.2.102:0/841647048 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f24b41b0af0 0x7f24b4108950 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 rx=0x7f24a8006650 tx=0x7f24a8009a80).ready entity=mon.2 client_cookie=46786c0dfe1bb4ee server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:12.775 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:12.757 7f24b27fc700 1 -- 172.21.2.102:0/841647048 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f24a80009b0 con 0x7f24b41b0af0 2021-02-10T11:37:12.775 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:12.757 7f24b27fc700 1 -- 172.21.2.102:0/841647048 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24a800c0c0 con 0x7f24b41b0af0 2021-02-10T11:37:12.775 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:12.757 7f24b27fc700 1 -- 172.21.2.102:0/841647048 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f24a800c520 con 0x7f24b41b0af0 2021-02-10T11:37:12.775 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:12.758 7f24b9eee700 1 -- 172.21.2.102:0/841647048 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f24b41b0af0 msgr2=0x7f24b4108950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:12.775 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:12.758 7f24b9eee700 1 --2- 172.21.2.102:0/841647048 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f24b41b0af0 0x7f24b4108950 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 rx=0x7f24a8006650 tx=0x7f24a8009a80).stop 2021-02-10T11:37:12.775 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:12.758 7f24b9eee700 1 -- 172.21.2.102:0/841647048 shutdown_connections 2021-02-10T11:37:12.775 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:12.758 7f24b9eee700 1 --2- 172.21.2.102:0/841647048 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f24b41b0af0 0x7f24b4108950 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:12.775 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:12.758 7f24b9eee700 1 --2- 172.21.2.102:0/841647048 >> v2:172.21.2.113:3300/0 conn(0x7f24b40d1880 0x7f24b41b04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:12.775 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:12.758 7f24b9eee700 1 -- 172.21.2.102:0/841647048 >> 172.21.2.102:0/841647048 conn(0x7f24b40ce430 msgr2=0x7f24b40d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:12.775 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:12.758 7f24b9eee700 1 -- 172.21.2.102:0/841647048 shutdown_connections 2021-02-10T11:37:12.776 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:12.758 7f24b9eee700 1 -- 172.21.2.102:0/841647048 wait complete. 2021-02-10T11:37:12.777 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:12.772 7f24b9eee700 1 Processor -- start 2021-02-10T11:37:12.777 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:12.773 7f24b9eee700 1 -- start start 2021-02-10T11:37:12.777 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:12.773 7f24b9eee700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f24b41a7a70 0x7f24b41a7e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:12.777 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:12.773 7f24b9eee700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f24b40bf730 0x7f24b40bfb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:12.777 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:12.773 7f24b9eee700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f24b40c0150 0x7f24b40909c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:12.778 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:12.773 7f24b9eee700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24b4090fe0 con 0x7f24b41a7a70 2021-02-10T11:37:12.778 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:12.773 7f24b9eee700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24b4091200 con 0x7f24b40c0150 2021-02-10T11:37:12.778 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:12.773 7f24b9eee700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24b4091420 con 0x7f24b40bf730 2021-02-10T11:37:12.778 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:12.773 7f24b37fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f24b41a7a70 0x7f24b41a7e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:12.778 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:12.773 7f24b37fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f24b41a7a70 0x7f24b41a7e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:38518/0 (socket says 172.21.2.102:38518) 2021-02-10T11:37:12.778 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:12.773 7f24b37fe700 1 -- 172.21.2.102:0/3461148006 learned_addr learned my addr 172.21.2.102:0/3461148006 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:37:12.778 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:12.773 7f24b2ffd700 1 --2- 172.21.2.102:0/3461148006 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f24b40bf730 0x7f24b40bfb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:12.778 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:12.773 7f24b3fff700 1 --2- 172.21.2.102:0/3461148006 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f24b40c0150 0x7f24b40909c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:12.779 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:12.774 7f24b2ffd700 1 -- 172.21.2.102:0/3461148006 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f24b40c0150 msgr2=0x7f24b40909c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:12.779 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:12.774 7f24b2ffd700 1 --2- 172.21.2.102:0/3461148006 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f24b40c0150 0x7f24b40909c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:12.779 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:12.774 7f24b2ffd700 1 -- 172.21.2.102:0/3461148006 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f24b41a7a70 msgr2=0x7f24b41a7e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:12.779 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:12.774 7f24b2ffd700 1 --2- 172.21.2.102:0/3461148006 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f24b41a7a70 0x7f24b41a7e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:12.779 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:12.774 7f24b2ffd700 1 -- 172.21.2.102:0/3461148006 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24b4091640 con 0x7f24b40bf730 2021-02-10T11:37:12.779 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:12.774 7f24b37fe700 1 --2- 172.21.2.102:0/3461148006 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f24b41a7a70 0x7f24b41a7e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:37:12.779 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:12.774 7f24b2ffd700 1 --2- 172.21.2.102:0/3461148006 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f24b40bf730 0x7f24b40bfb30 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0x7f24a800b530 tx=0x7f24a800b8d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:12.779 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:12.774 7f249bfff700 1 -- 172.21.2.102:0/3461148006 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f24a80093a0 con 0x7f24b40bf730 2021-02-10T11:37:12.779 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:12.774 7f249bfff700 1 -- 172.21.2.102:0/3461148006 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24a800bcc0 con 0x7f24b40bf730 2021-02-10T11:37:12.779 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:12.774 7f24b9eee700 1 -- 172.21.2.102:0/3461148006 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24b4091640 con 0x7f24b40bf730 2021-02-10T11:37:12.779 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:12.774 7f249bfff700 1 -- 172.21.2.102:0/3461148006 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f24a80022b0 con 0x7f24b40bf730 2021-02-10T11:37:12.779 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:12.774 7f24b9eee700 1 -- 172.21.2.102:0/3461148006 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24b4091640 con 0x7f24b40bf730 2021-02-10T11:37:12.779 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:12.775 7f249bfff700 1 -- 172.21.2.102:0/3461148006 <== mon.2 v2:172.21.2.113:3301/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f24a8009fe0 con 0x7f24b40bf730 2021-02-10T11:37:12.780 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:12.775 7f249bfff700 1 --2- 172.21.2.102:0/3461148006 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f24940186c0 0x7f249401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:12.780 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:12.775 7f249bfff700 1 -- 172.21.2.102:0/3461148006 <== mon.2 v2:172.21.2.113:3301/0 5 ==== osd_map(162..162 src has 1..162) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f24a8032470 con 0x7f24b40bf730 2021-02-10T11:37:12.780 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:12.775 7f24b37fe700 1 --2- 172.21.2.102:0/3461148006 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f24940186c0 0x7f249401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:12.780 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:12.776 7f24b37fe700 1 --2- 172.21.2.102:0/3461148006 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f24940186c0 0x7f249401abc0 crc :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:12.807 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:12.806 7f24b9eee700 1 -- 172.21.2.102:0/3461148006 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24b40fe1d0 con 0x7f24b40bf730 2021-02-10T11:37:12.810 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:12.809 7f249bfff700 1 -- 172.21.2.102:0/3461148006 <== mon.2 v2:172.21.2.113:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f24a8014710 con 0x7f24b40bf730 2021-02-10T11:37:12.937 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:12.936 7f24b9eee700 1 -- 172.21.2.102:0/3461148006 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume snapshot rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000108316", "target": ["mgr", ""], "snap_name": "snapshot_0000000000138436"}) v1 -- 0x7f24b4090fe0 con 0x7f24940186c0 2021-02-10T11:37:12.941 INFO:tasks.ceph.mgr.y.gibba002.stderr:2021-02-10 11:37:12.940 7fd94bc53700 -1 mgr.server reply reply (2) No such file or directory snapshot 'snapshot_0000000000138436' does not exist 2021-02-10T11:37:12.941 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:12.940 7f249bfff700 1 -- 172.21.2.102:0/3461148006 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: -2 snapshot 'snapshot_0000000000138436' does not exist) v1 ==== 59+0+0 (crc 0 0 0) 0x7f24a40009b0 con 0x7f24940186c0 2021-02-10T11:37:12.944 INFO:teuthology.orchestra.run.gibba002.stderr:Error ENOENT: snapshot 'snapshot_0000000000138436' does not exist 2021-02-10T11:37:12.946 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:12.945 7f24b9eee700 1 -- 172.21.2.102:0/3461148006 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f24940186c0 msgr2=0x7f249401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:12.946 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:12.945 7f24b9eee700 1 --2- 172.21.2.102:0/3461148006 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f24940186c0 0x7f249401abc0 crc :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:12.946 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:12.945 7f24b9eee700 1 -- 172.21.2.102:0/3461148006 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f24b40bf730 msgr2=0x7f24b40bfb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:12.946 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:12.945 7f24b9eee700 1 --2- 172.21.2.102:0/3461148006 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f24b40bf730 0x7f24b40bfb30 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0x7f24a800b530 tx=0x7f24a800b8d0).stop 2021-02-10T11:37:12.946 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:12.945 7f24b9eee700 1 -- 172.21.2.102:0/3461148006 shutdown_connections 2021-02-10T11:37:12.946 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:12.945 7f24b9eee700 1 --2- 172.21.2.102:0/3461148006 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f24940186c0 0x7f249401abc0 unknown :-1 s=CLOSED pgs=1284 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:12.946 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:12.945 7f24b9eee700 1 --2- 172.21.2.102:0/3461148006 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f24b40c0150 0x7f24b40909c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:12.946 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:12.945 7f24b9eee700 1 --2- 172.21.2.102:0/3461148006 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f24b40bf730 0x7f24b40bfb30 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:12.947 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:12.945 7f24b9eee700 1 --2- 172.21.2.102:0/3461148006 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f24b41a7a70 0x7f24b41a7e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:12.947 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:12.945 7f24b9eee700 1 -- 172.21.2.102:0/3461148006 >> 172.21.2.102:0/3461148006 conn(0x7f24b40a69d0 msgr2=0x7f24b40b5950 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:12.947 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:12.945 7f24b9eee700 1 -- 172.21.2.102:0/3461148006 shutdown_connections 2021-02-10T11:37:12.947 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:12.945 7f24b9eee700 1 -- 172.21.2.102:0/3461148006 wait complete. 2021-02-10T11:37:12.954 DEBUG:teuthology.orchestra.run:got remote process result: 2 2021-02-10T11:37:12.954 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000108316 2021-02-10T11:37:13.388 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:13.370 7f2b50923700 1 Processor -- start 2021-02-10T11:37:13.388 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:13.370 7f2b50923700 1 -- start start 2021-02-10T11:37:13.388 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:13.371 7f2b50923700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f2b4c0d1870 0x7f2b4c1b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:13.388 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:13.371 7f2b50923700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b4c0fe1c0 con 0x7f2b4c1b0ae0 2021-02-10T11:37:13.388 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:13.371 7f2b50923700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b4c109a60 con 0x7f2b4c0cff70 2021-02-10T11:37:13.389 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:13.371 7f2b50923700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b4c088070 con 0x7f2b4c0d1870 2021-02-10T11:37:13.389 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:13.371 7f2b49e1b700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f2b4c1b0ae0 0x7f2b4c108940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:58982/0 (socket says 172.21.2.102:58982) 2021-02-10T11:37:13.389 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:13.371 7f2b4a61c700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f2b4c0d1870 0x7f2b4c1b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:13.389 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:13.371 7f2b49e1b700 1 -- 172.21.2.102:0/1813380057 learned_addr learned my addr 172.21.2.102:0/1813380057 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:37:13.389 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:13.371 7f2b4961a700 1 -- 172.21.2.102:0/1813380057 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1423768578 0 0) 0x7f2b400010b0 con 0x7f2b4c1b0ae0 2021-02-10T11:37:13.389 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:13.371 7f2b4961a700 1 -- 172.21.2.102:0/1813380057 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b34002fe0 con 0x7f2b4c1b0ae0 2021-02-10T11:37:13.389 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:13.371 7f2b4a61c700 1 -- 172.21.2.102:0/1813380057 >> v1:172.21.2.113:6789/0 conn(0x7f2b4c0cff70 legacy=0x7f2b4c109130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:37:13.389 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:13.371 7f2b4a61c700 1 -- 172.21.2.102:0/1813380057 >> v1:172.21.2.102:6789/0 conn(0x7f2b4c1b0ae0 legacy=0x7f2b4c108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:13.389 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:13.371 7f2b4a61c700 1 -- 172.21.2.102:0/1813380057 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b4c088260 con 0x7f2b4c0d1870 2021-02-10T11:37:13.389 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:13.371 7f2b4a61c700 1 --2- 172.21.2.102:0/1813380057 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2b4c0d1870 0x7f2b4c1b04c0 secure :-1 s=READY pgs=2929 cs=0 l=1 rev1=1 rx=0x7f2b3c006650 tx=0x7f2b3c009a80).ready entity=mon.0 client_cookie=4f118fc5d7a738f0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:13.389 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:13.372 7f2b4961a700 1 -- 172.21.2.102:0/1813380057 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2b3c0009b0 con 0x7f2b4c0d1870 2021-02-10T11:37:13.389 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:13.372 7f2b4961a700 1 -- 172.21.2.102:0/1813380057 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b3c00c0c0 con 0x7f2b4c0d1870 2021-02-10T11:37:13.389 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:13.372 7f2b4961a700 1 -- 172.21.2.102:0/1813380057 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2b3c00c5e0 con 0x7f2b4c0d1870 2021-02-10T11:37:13.390 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:13.372 7f2b50923700 1 -- 172.21.2.102:0/1813380057 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2b4c0d1870 msgr2=0x7f2b4c1b04c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:13.390 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:13.372 7f2b50923700 1 --2- 172.21.2.102:0/1813380057 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2b4c0d1870 0x7f2b4c1b04c0 secure :-1 s=READY pgs=2929 cs=0 l=1 rev1=1 rx=0x7f2b3c006650 tx=0x7f2b3c009a80).stop 2021-02-10T11:37:13.390 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:13.373 7f2b50923700 1 -- 172.21.2.102:0/1813380057 shutdown_connections 2021-02-10T11:37:13.390 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:13.373 7f2b50923700 1 --2- 172.21.2.102:0/1813380057 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2b4c0d1870 0x7f2b4c1b04c0 unknown :-1 s=CLOSED pgs=2929 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:13.390 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:13.373 7f2b50923700 1 -- 172.21.2.102:0/1813380057 >> 172.21.2.102:0/1813380057 conn(0x7f2b4c0ce420 msgr2=0x7f2b4c0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:13.390 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:13.373 7f2b50923700 1 -- 172.21.2.102:0/1813380057 shutdown_connections 2021-02-10T11:37:13.390 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:13.373 7f2b50923700 1 -- 172.21.2.102:0/1813380057 wait complete. 2021-02-10T11:37:13.391 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:13.387 7f2b50923700 1 Processor -- start 2021-02-10T11:37:13.391 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:13.387 7f2b50923700 1 -- start start 2021-02-10T11:37:13.392 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:13.388 7f2b50923700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2b4c0c3820 0x7f2b4c084cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:13.392 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:13.388 7f2b50923700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2b4c0a0290 0x7f2b4c0852d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:13.392 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:13.388 7f2b50923700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2b4c086270 0x7f2b4c086670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:13.392 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:13.388 7f2b50923700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b4c0c0700 con 0x7f2b4c0a0290 2021-02-10T11:37:13.392 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:13.388 7f2b50923700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b4c0858f0 con 0x7f2b4c086270 2021-02-10T11:37:13.392 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:13.388 7f2b50923700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b4c085b10 con 0x7f2b4c0c3820 2021-02-10T11:37:13.392 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:13.388 7f2b49e1b700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2b4c0a0290 0x7f2b4c0852d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:13.392 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:13.388 7f2b49e1b700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2b4c0a0290 0x7f2b4c0852d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:38530/0 (socket says 172.21.2.102:38530) 2021-02-10T11:37:13.392 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:13.388 7f2b49e1b700 1 -- 172.21.2.102:0/39813454 learned_addr learned my addr 172.21.2.102:0/39813454 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:37:13.392 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:13.388 7f2b49e1b700 1 -- 172.21.2.102:0/39813454 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2b4c0c3820 msgr2=0x7f2b4c084cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:13.392 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:13.388 7f2b49e1b700 1 --2- 172.21.2.102:0/39813454 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2b4c0c3820 0x7f2b4c084cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:13.392 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:13.388 7f2b49e1b700 1 -- 172.21.2.102:0/39813454 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2b4c086270 msgr2=0x7f2b4c086670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:13.392 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:13.388 7f2b49e1b700 1 --2- 172.21.2.102:0/39813454 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2b4c086270 0x7f2b4c086670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:13.393 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:13.388 7f2b49e1b700 1 -- 172.21.2.102:0/39813454 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b4c086c90 con 0x7f2b4c0a0290 2021-02-10T11:37:13.393 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:13.388 7f2b49e1b700 1 --2- 172.21.2.102:0/39813454 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2b4c0a0290 0x7f2b4c0852d0 secure :-1 s=READY pgs=2930 cs=0 l=1 rev1=1 rx=0x7f2b40001c60 tx=0x7f2b4000a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:13.393 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:13.389 7f2b3affd700 1 -- 172.21.2.102:0/39813454 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2b4000acc0 con 0x7f2b4c0a0290 2021-02-10T11:37:13.393 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:13.389 7f2b3affd700 1 -- 172.21.2.102:0/39813454 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b40004350 con 0x7f2b4c0a0290 2021-02-10T11:37:13.393 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:13.389 7f2b50923700 1 -- 172.21.2.102:0/39813454 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b4c086c90 con 0x7f2b4c0a0290 2021-02-10T11:37:13.393 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:13.389 7f2b3affd700 1 -- 172.21.2.102:0/39813454 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2b40004870 con 0x7f2b4c0a0290 2021-02-10T11:37:13.393 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:13.389 7f2b50923700 1 -- 172.21.2.102:0/39813454 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b4c086c90 con 0x7f2b4c0a0290 2021-02-10T11:37:13.393 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:13.389 7f2b3affd700 1 -- 172.21.2.102:0/39813454 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f2b40004530 con 0x7f2b4c0a0290 2021-02-10T11:37:13.393 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:13.389 7f2b3affd700 1 --2- 172.21.2.102:0/39813454 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2b300186c0 0x7f2b3001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:13.393 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:13.390 7f2b3affd700 1 -- 172.21.2.102:0/39813454 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(162..162 src has 1..162) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f2b400327f0 con 0x7f2b4c0a0290 2021-02-10T11:37:13.393 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:13.390 7f2b4a61c700 1 --2- 172.21.2.102:0/39813454 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2b300186c0 0x7f2b3001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:13.394 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:13.390 7f2b4a61c700 1 --2- 172.21.2.102:0/39813454 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2b300186c0 0x7f2b3001abc0 crc :-1 s=READY pgs=1285 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:13.422 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:13.420 7f2b50923700 1 -- 172.21.2.102:0/39813454 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b4c082a70 con 0x7f2b4c0a0290 2021-02-10T11:37:13.424 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:13.423 7f2b3affd700 1 -- 172.21.2.102:0/39813454 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f2b40004140 con 0x7f2b4c0a0290 2021-02-10T11:37:13.551 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:13.550 7f2b50923700 1 -- 172.21.2.102:0/39813454 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000108316", "target": ["mgr", ""]}) v1 -- 0x7f2b4c082a70 con 0x7f2b300186c0 2021-02-10T11:37:13.561 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:13.560 7f2b3affd700 1 -- 172.21.2.102:0/39813454 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f2b3c000940 con 0x7f2b300186c0 2021-02-10T11:37:13.568 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:13.567 7f2b50923700 1 -- 172.21.2.102:0/39813454 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2b300186c0 msgr2=0x7f2b3001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:13.568 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:13.567 7f2b50923700 1 --2- 172.21.2.102:0/39813454 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2b300186c0 0x7f2b3001abc0 crc :-1 s=READY pgs=1285 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:13.568 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:13.567 7f2b50923700 1 -- 172.21.2.102:0/39813454 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2b4c0a0290 msgr2=0x7f2b4c0852d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:13.568 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:13.567 7f2b50923700 1 --2- 172.21.2.102:0/39813454 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2b4c0a0290 0x7f2b4c0852d0 secure :-1 s=READY pgs=2930 cs=0 l=1 rev1=1 rx=0x7f2b40001c60 tx=0x7f2b4000a8d0).stop 2021-02-10T11:37:13.569 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:13.567 7f2b50923700 1 -- 172.21.2.102:0/39813454 shutdown_connections 2021-02-10T11:37:13.569 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:13.567 7f2b50923700 1 --2- 172.21.2.102:0/39813454 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2b300186c0 0x7f2b3001abc0 unknown :-1 s=CLOSED pgs=1285 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:13.569 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:13.567 7f2b50923700 1 --2- 172.21.2.102:0/39813454 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2b4c086270 0x7f2b4c086670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:13.569 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:13.567 7f2b50923700 1 --2- 172.21.2.102:0/39813454 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2b4c0a0290 0x7f2b4c0852d0 unknown :-1 s=CLOSED pgs=2930 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:13.569 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:13.567 7f2b50923700 1 --2- 172.21.2.102:0/39813454 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2b4c0c3820 0x7f2b4c084cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:13.569 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:13.567 7f2b50923700 1 -- 172.21.2.102:0/39813454 >> 172.21.2.102:0/39813454 conn(0x7f2b4c0872c0 msgr2=0x7f2b4c095700 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:13.569 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:13.567 7f2b50923700 1 -- 172.21.2.102:0/39813454 shutdown_connections 2021-02-10T11:37:13.569 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:13.567 7f2b50923700 1 -- 172.21.2.102:0/39813454 wait complete. 2021-02-10T11:37:13.576 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo getfattr --only-values -n ceph.dir.entries /home/ubuntu/cephtest/mnt.0/./volumes/_deleting 2021-02-10T11:37:13.698 DEBUG:teuthology.orchestra.run.gibba002:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:37:13.702 DEBUG:teuthology.orchestra.run.gibba013:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:37:13.927 INFO:teuthology.orchestra.run.gibba002.stderr:getfattr: Removing leading '/' from absolute path names 2021-02-10T11:37:13.928 INFO:teuthology.orchestra.run.gibba002.stdout:0 2021-02-10T11:37:13.929 DEBUG:tasks.cephfs.mount:Directory ./volumes/_deleting seen empty from 0 after 0s 2021-02-10T11:37:13.929 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2021-02-10T11:37:13.929 DEBUG:teuthology.orchestra.run.gibba002:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:37:14.275 DEBUG:teuthology.orchestra.run.gibba013:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:37:14.619 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2021-02-10T11:37:14.619 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:37:14.620 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:37:14.620 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:37:14.957 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:37:14.957 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:37:14.957 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@gibba002.front.sepia.ceph.com... 2021-02-10T11:37:14.957 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-02-10T11:37:14.958 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:37:15.311 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba002.stderr:ceph-fuse[204511]: fuse finished with error 0 and tester_r 0 2021-02-10T11:37:15.312 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:37:15.313 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:37:15.313 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:37:15.649 INFO:teuthology.orchestra.run.gibba002.stdout:ext2/ext3 2021-02-10T11:37:15.649 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2021-02-10T11:37:15.649 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:37:15.649 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:37:15.984 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:37:15.985 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:37:15.985 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:37:16.322 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:37:16.322 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2021-02-10T11:37:16.322 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@gibba002.front.sepia.ceph.com... 2021-02-10T11:37:16.322 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-02-10T11:37:16.323 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:37:16.676 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba002.stderr:ceph-fuse[205151]: fuse finished with error 0 and tester_r 0 2021-02-10T11:37:16.677 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:37:16.678 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:37:16.678 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:37:17.012 INFO:teuthology.orchestra.run.gibba002.stdout:ext2/ext3 2021-02-10T11:37:17.012 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2021-02-10T11:37:17.013 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:37:17.013 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:37:17.347 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:37:17.347 DEBUG:teuthology.orchestra.run.gibba013:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:37:17.347 DEBUG:teuthology.orchestra.run.gibba013:> ' -- /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:37:17.682 INFO:teuthology.orchestra.run.gibba013.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.2’: No such file or directory 2021-02-10T11:37:17.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:37:17.683 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2021-02-10T11:37:17.683 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:37:17.683 DEBUG:teuthology.orchestra.run.gibba013:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:37:18.018 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:37:18.019 DEBUG:teuthology.orchestra.run.gibba013:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:37:18.019 DEBUG:teuthology.orchestra.run.gibba013:> ' -- /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:37:18.353 INFO:teuthology.orchestra.run.gibba013.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.3’: No such file or directory 2021-02-10T11:37:18.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:37:18.353 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.3 2021-02-10T11:37:18.354 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:37:18.354 DEBUG:teuthology.orchestra.run.gibba013:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:37:18.689 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config rm mon mon_allow_pool_delete 2021-02-10T11:37:19.122 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:19.104 7fd1f5e3a700 1 Processor -- start 2021-02-10T11:37:19.122 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:19.104 7fd1f5e3a700 1 -- start start 2021-02-10T11:37:19.122 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:19.105 7fd1f5e3a700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd1f00d0340 0x7fd1f00ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:19.122 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:19.105 7fd1f5e3a700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fd1f00d1c70 0x7fd1f00efc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:19.122 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:19.105 7fd1f5e3a700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1f00f0b10 con 0x7fd1f00d1870 2021-02-10T11:37:19.122 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:19.105 7fd1f5e3a700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1f00e5ed0 con 0x7fd1f00d1c70 2021-02-10T11:37:19.122 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:19.105 7fd1f5e3a700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1f01b0be0 con 0x7fd1f00d0340 2021-02-10T11:37:19.123 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:19.105 7fd1eeffd700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd1f00d0340 0x7fd1f00ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:19.123 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:19.105 7fd1ef7fe700 1 --1- >> v1:172.21.2.113:6789/0 conn(0x7fd1f00d1870 0x7fd1f01b03f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.113:6789/0 says I am v1:172.21.2.102:38086/0 (socket says 172.21.2.102:38086) 2021-02-10T11:37:19.123 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:19.105 7fd1ef7fe700 1 -- 172.21.2.102:0/3763568297 learned_addr learned my addr 172.21.2.102:0/3763568297 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:37:19.123 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:19.105 7fd1effff700 1 --2- 172.21.2.102:0/3763568297 >> v2:172.21.2.113:3300/0 conn(0x7fd1f00d1c70 0x7fd1f00efc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:19.123 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:19.106 7fd1eeffd700 1 -- 172.21.2.102:0/3763568297 >> v2:172.21.2.113:3300/0 conn(0x7fd1f00d1c70 msgr2=0x7fd1f00efc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:37:19.123 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:19.106 7fd1eeffd700 1 --2- 172.21.2.102:0/3763568297 >> v2:172.21.2.113:3300/0 conn(0x7fd1f00d1c70 0x7fd1f00efc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:19.123 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:19.106 7fd1eeffd700 1 -- 172.21.2.102:0/3763568297 >> v1:172.21.2.113:6789/0 conn(0x7fd1f00d1870 legacy=0x7fd1f01b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:19.123 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:19.106 7fd1eeffd700 1 -- 172.21.2.102:0/3763568297 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1f00f20a0 con 0x7fd1f00d0340 2021-02-10T11:37:19.123 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:19.106 7fd1eeffd700 1 --2- 172.21.2.102:0/3763568297 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd1f00d0340 0x7fd1f00ef640 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 rx=0x7fd1e0006650 tx=0x7fd1e0009a80).ready entity=mon.2 client_cookie=d9cd0a5d6196476d server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:19.123 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:19.106 7fd1ee7fc700 1 -- 172.21.2.102:0/3763568297 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd1e00009b0 con 0x7fd1f00d0340 2021-02-10T11:37:19.123 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:19.106 7fd1ee7fc700 1 -- 172.21.2.102:0/3763568297 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1e000c0c0 con 0x7fd1f00d0340 2021-02-10T11:37:19.123 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:19.106 7fd1ee7fc700 1 -- 172.21.2.102:0/3763568297 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd1e000c5e0 con 0x7fd1f00d0340 2021-02-10T11:37:19.123 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:19.106 7fd1f5e3a700 1 -- 172.21.2.102:0/3763568297 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd1f00d0340 msgr2=0x7fd1f00ef640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:19.124 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:19.106 7fd1f5e3a700 1 --2- 172.21.2.102:0/3763568297 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd1f00d0340 0x7fd1f00ef640 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 rx=0x7fd1e0006650 tx=0x7fd1e0009a80).stop 2021-02-10T11:37:19.124 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:19.107 7fd1f5e3a700 1 -- 172.21.2.102:0/3763568297 shutdown_connections 2021-02-10T11:37:19.124 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:19.107 7fd1f5e3a700 1 --2- 172.21.2.102:0/3763568297 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd1f00d0340 0x7fd1f00ef640 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:19.124 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:19.107 7fd1f5e3a700 1 --2- 172.21.2.102:0/3763568297 >> v2:172.21.2.113:3300/0 conn(0x7fd1f00d1c70 0x7fd1f00efc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:19.124 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:19.107 7fd1f5e3a700 1 -- 172.21.2.102:0/3763568297 >> 172.21.2.102:0/3763568297 conn(0x7fd1f00ce450 msgr2=0x7fd1f00d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:19.124 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:19.107 7fd1f5e3a700 1 -- 172.21.2.102:0/3763568297 shutdown_connections 2021-02-10T11:37:19.124 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:19.107 7fd1f5e3a700 1 -- 172.21.2.102:0/3763568297 wait complete. 2021-02-10T11:37:19.126 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:19.121 7fd1f5e3a700 1 Processor -- start 2021-02-10T11:37:19.126 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:19.121 7fd1f5e3a700 1 -- start start 2021-02-10T11:37:19.126 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:19.122 7fd1f5e3a700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd1f01a06c0 0x7fd1f00ebe70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:19.126 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:19.122 7fd1f5e3a700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd1f00ec490 0x7fd1f00c30f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:19.126 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:19.122 7fd1f5e3a700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd1f00c3710 0x7fd1f00bc070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:19.126 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:19.122 7fd1f5e3a700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1f00c3b10 con 0x7fd1f00ec490 2021-02-10T11:37:19.126 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:19.122 7fd1f5e3a700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1f00eca90 con 0x7fd1f01a06c0 2021-02-10T11:37:19.126 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:19.122 7fd1f5e3a700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1f00bc690 con 0x7fd1f00c3710 2021-02-10T11:37:19.126 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:19.122 7fd1eeffd700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd1f00ec490 0x7fd1f00c30f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:19.126 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:19.122 7fd1eeffd700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd1f00ec490 0x7fd1f00c30f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:38546/0 (socket says 172.21.2.102:38546) 2021-02-10T11:37:19.126 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:19.122 7fd1eeffd700 1 -- 172.21.2.102:0/3367280887 learned_addr learned my addr 172.21.2.102:0/3367280887 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:37:19.127 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:19.122 7fd1effff700 1 --2- 172.21.2.102:0/3367280887 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd1f00c3710 0x7fd1f00bc070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:19.127 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:19.122 7fd1ef7fe700 1 --2- 172.21.2.102:0/3367280887 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd1f01a06c0 0x7fd1f00ebe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:19.127 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:19.123 7fd1eeffd700 1 -- 172.21.2.102:0/3367280887 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd1f00c3710 msgr2=0x7fd1f00bc070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:19.127 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:19.123 7fd1eeffd700 1 --2- 172.21.2.102:0/3367280887 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd1f00c3710 0x7fd1f00bc070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:19.127 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:19.123 7fd1eeffd700 1 -- 172.21.2.102:0/3367280887 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd1f01a06c0 msgr2=0x7fd1f00ebe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:19.127 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:19.123 7fd1eeffd700 1 --2- 172.21.2.102:0/3367280887 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd1f01a06c0 0x7fd1f00ebe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:19.127 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:19.123 7fd1eeffd700 1 -- 172.21.2.102:0/3367280887 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1f00bc870 con 0x7fd1f00ec490 2021-02-10T11:37:19.127 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:19.123 7fd1effff700 1 --2- 172.21.2.102:0/3367280887 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd1f00c3710 0x7fd1f00bc070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:37:19.127 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:19.123 7fd1eeffd700 1 --2- 172.21.2.102:0/3367280887 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd1f00ec490 0x7fd1f00c30f0 secure :-1 s=READY pgs=2931 cs=0 l=1 rev1=1 rx=0x7fd1e0009650 tx=0x7fd1e000b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:19.127 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:19.123 7fd1e7fff700 1 -- 172.21.2.102:0/3367280887 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd1e000bd50 con 0x7fd1f00ec490 2021-02-10T11:37:19.127 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:19.123 7fd1e7fff700 1 -- 172.21.2.102:0/3367280887 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1e0006030 con 0x7fd1f00ec490 2021-02-10T11:37:19.127 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:19.123 7fd1e7fff700 1 -- 172.21.2.102:0/3367280887 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd1e0005560 con 0x7fd1f00ec490 2021-02-10T11:37:19.128 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:19.123 7fd1f5e3a700 1 -- 172.21.2.102:0/3367280887 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1f00bc870 con 0x7fd1f00ec490 2021-02-10T11:37:19.128 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:19.123 7fd1f5e3a700 1 -- 172.21.2.102:0/3367280887 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1f00bc870 con 0x7fd1f00ec490 2021-02-10T11:37:19.128 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:19.124 7fd1e7fff700 1 -- 172.21.2.102:0/3367280887 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd1e00052e0 con 0x7fd1f00ec490 2021-02-10T11:37:19.128 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:19.124 7fd1e7fff700 1 --2- 172.21.2.102:0/3367280887 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd1d40186c0 0x7fd1d401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:19.128 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:19.124 7fd1e7fff700 1 -- 172.21.2.102:0/3367280887 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(163..163 src has 1..163) v4 ==== 5645+0+0 (secure 0 0 0) 0x7fd1e0032560 con 0x7fd1f00ec490 2021-02-10T11:37:19.128 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:19.124 7fd1ef7fe700 1 --2- 172.21.2.102:0/3367280887 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd1d40186c0 0x7fd1d401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:19.128 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:19.124 7fd1ef7fe700 1 --2- 172.21.2.102:0/3367280887 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd1d40186c0 0x7fd1d401abc0 crc :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:19.155 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:19.155 7fd1f5e3a700 1 -- 172.21.2.102:0/3367280887 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1f00ebc40 con 0x7fd1f00ec490 2021-02-10T11:37:19.158 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:19.157 7fd1e7fff700 1 -- 172.21.2.102:0/3367280887 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd1e00009a0 con 0x7fd1f00ec490 2021-02-10T11:37:19.307 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:19.306 7fd1f5e3a700 1 -- 172.21.2.102:0/3367280887 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fd1f00f0b10 con 0x7fd1f00ec490 2021-02-10T11:37:19.310 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:19.309 7fd1e7fff700 1 -- 172.21.2.102:0/3367280887 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v41) v1 ==== 104+0+0 (secure 0 0 0) 0x7fd1e00052e0 con 0x7fd1f00ec490 2021-02-10T11:37:19.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:19.314 7fd1f5e3a700 1 -- 172.21.2.102:0/3367280887 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd1d40186c0 msgr2=0x7fd1d401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:19.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:19.315 7fd1f5e3a700 1 --2- 172.21.2.102:0/3367280887 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd1d40186c0 0x7fd1d401abc0 crc :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:19.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:19.315 7fd1f5e3a700 1 -- 172.21.2.102:0/3367280887 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd1f00ec490 msgr2=0x7fd1f00c30f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:19.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:19.315 7fd1f5e3a700 1 --2- 172.21.2.102:0/3367280887 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd1f00ec490 0x7fd1f00c30f0 secure :-1 s=READY pgs=2931 cs=0 l=1 rev1=1 rx=0x7fd1e0009650 tx=0x7fd1e000b960).stop 2021-02-10T11:37:19.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:19.315 7fd1f5e3a700 1 -- 172.21.2.102:0/3367280887 shutdown_connections 2021-02-10T11:37:19.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:19.315 7fd1f5e3a700 1 --2- 172.21.2.102:0/3367280887 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd1f00c3710 0x7fd1f00bc070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:19.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:19.315 7fd1f5e3a700 1 --2- 172.21.2.102:0/3367280887 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd1f00ec490 0x7fd1f00c30f0 unknown :-1 s=CLOSED pgs=2931 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:19.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:19.315 7fd1f5e3a700 1 --2- 172.21.2.102:0/3367280887 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd1d40186c0 0x7fd1d401abc0 unknown :-1 s=CLOSED pgs=1286 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:19.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:19.315 7fd1f5e3a700 1 --2- 172.21.2.102:0/3367280887 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd1f01a06c0 0x7fd1f00ebe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:19.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:19.315 7fd1f5e3a700 1 -- 172.21.2.102:0/3367280887 >> 172.21.2.102:0/3367280887 conn(0x7fd1f00a6bb0 msgr2=0x7fd1f0095700 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:19.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:19.315 7fd1f5e3a700 1 -- 172.21.2.102:0/3367280887 shutdown_connections 2021-02-10T11:37:19.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:19.315 7fd1f5e3a700 1 -- 172.21.2.102:0/3367280887 wait complete. 2021-02-10T11:37:19.325 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestVolumes.test_nonexistent_subvolume_snapshot_rm' 2021-02-10T11:37:19.755 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:19.737 7f97e9ace700 1 Processor -- start 2021-02-10T11:37:19.755 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:19.737 7f97e9ace700 1 -- start start 2021-02-10T11:37:19.756 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:19.738 7f97e9ace700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f97e40d1870 0x7f97e41b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:19.756 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:19.738 7f97e9ace700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f97e41b0a10 0x7f97e40cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:19.756 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:19.738 7f97e9ace700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97e40f0b10 con 0x7f97e40d1c70 2021-02-10T11:37:19.756 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:19.738 7f97e9ace700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97e40e5ed0 con 0x7f97e41b0a10 2021-02-10T11:37:19.756 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:19.738 7f97e9ace700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97e40f0650 con 0x7f97e40d1870 2021-02-10T11:37:19.756 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:19.738 7f97e2ffd700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f97e41b0a10 0x7f97e40cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:19.756 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:19.738 7f97e2ffd700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f97e41b0a10 0x7f97e40cf330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:38552/0 (socket says 172.21.2.102:38552) 2021-02-10T11:37:19.756 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:19.738 7f97e2ffd700 1 -- 172.21.2.102:0/556519467 learned_addr learned my addr 172.21.2.102:0/556519467 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:37:19.756 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:19.739 7f97e37fe700 1 --2- 172.21.2.102:0/556519467 >> v2:172.21.2.113:3300/0 conn(0x7f97e40d1870 0x7f97e41b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:19.756 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:19.739 7f97e2ffd700 1 -- 172.21.2.102:0/556519467 >> v2:172.21.2.113:3300/0 conn(0x7f97e40d1870 msgr2=0x7f97e41b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:37:19.756 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:19.739 7f97e2ffd700 1 --2- 172.21.2.102:0/556519467 >> v2:172.21.2.113:3300/0 conn(0x7f97e40d1870 0x7f97e41b03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:19.756 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:19.739 7f97e2ffd700 1 -- 172.21.2.102:0/556519467 >> v1:172.21.2.113:6789/0 conn(0x7f97e40d1c70 legacy=0x7f97e40cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:37:19.756 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:19.739 7f97e2ffd700 1 -- 172.21.2.102:0/556519467 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97e40f0840 con 0x7f97e41b0a10 2021-02-10T11:37:19.757 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:19.739 7f97e37fe700 1 --2- 172.21.2.102:0/556519467 >> v2:172.21.2.113:3300/0 conn(0x7f97e40d1870 0x7f97e41b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:37:19.757 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:19.739 7f97e2ffd700 1 --2- 172.21.2.102:0/556519467 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f97e41b0a10 0x7f97e40cf330 secure :-1 s=READY pgs=2932 cs=0 l=1 rev1=1 rx=0x7f97d4006700 tx=0x7f97d4009b80).ready entity=mon.0 client_cookie=ea933093206b063e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:19.757 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:19.739 7f97e27fc700 1 -- 172.21.2.102:0/556519467 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f97d400c0c0 con 0x7f97e41b0a10 2021-02-10T11:37:19.757 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:19.739 7f97e27fc700 1 -- 172.21.2.102:0/556519467 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97d400c340 con 0x7f97e41b0a10 2021-02-10T11:37:19.757 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:19.739 7f97e27fc700 1 -- 172.21.2.102:0/556519467 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f97d400c860 con 0x7f97e41b0a10 2021-02-10T11:37:19.757 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:19.739 7f97e9ace700 1 -- 172.21.2.102:0/556519467 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f97e41b0a10 msgr2=0x7f97e40cf330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:19.757 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:19.739 7f97e9ace700 1 --2- 172.21.2.102:0/556519467 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f97e41b0a10 0x7f97e40cf330 secure :-1 s=READY pgs=2932 cs=0 l=1 rev1=1 rx=0x7f97d4006700 tx=0x7f97d4009b80).stop 2021-02-10T11:37:19.757 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:19.740 7f97e9ace700 1 -- 172.21.2.102:0/556519467 shutdown_connections 2021-02-10T11:37:19.757 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:19.740 7f97e9ace700 1 --2- 172.21.2.102:0/556519467 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f97e41b0a10 0x7f97e40cf330 unknown :-1 s=CLOSED pgs=2932 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:19.757 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:19.740 7f97e9ace700 1 --2- 172.21.2.102:0/556519467 >> v2:172.21.2.113:3300/0 conn(0x7f97e40d1870 0x7f97e41b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:19.757 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:19.740 7f97e9ace700 1 -- 172.21.2.102:0/556519467 >> 172.21.2.102:0/556519467 conn(0x7f97e40ce450 msgr2=0x7f97e40d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:19.757 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:19.740 7f97e9ace700 1 -- 172.21.2.102:0/556519467 shutdown_connections 2021-02-10T11:37:19.757 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:19.740 7f97e9ace700 1 -- 172.21.2.102:0/556519467 wait complete. 2021-02-10T11:37:19.759 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:19.755 7f97e9ace700 1 Processor -- start 2021-02-10T11:37:19.759 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:19.755 7f97e9ace700 1 -- start start 2021-02-10T11:37:19.759 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:19.755 7f97e9ace700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f97e41b7cf0 0x7f97e419cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:19.759 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:19.755 7f97e9ace700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f97e41b80f0 0x7f97e41b84f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:19.759 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:19.755 7f97e9ace700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f97e419da20 0x7f97e419de20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:19.759 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:19.755 7f97e9ace700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97e40a77c0 con 0x7f97e419da20 2021-02-10T11:37:19.759 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:19.755 7f97e9ace700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97e419d4a0 con 0x7f97e41b80f0 2021-02-10T11:37:19.760 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:19.755 7f97e9ace700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97e40954f0 con 0x7f97e41b7cf0 2021-02-10T11:37:19.760 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:19.756 7f97e3fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f97e419da20 0x7f97e419de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:19.760 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:19.756 7f97e3fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f97e419da20 0x7f97e419de20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:38560/0 (socket says 172.21.2.102:38560) 2021-02-10T11:37:19.760 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:19.756 7f97e3fff700 1 -- 172.21.2.102:0/3038126381 learned_addr learned my addr 172.21.2.102:0/3038126381 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:37:19.760 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:19.756 7f97e3fff700 1 -- 172.21.2.102:0/3038126381 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f97e41b7cf0 msgr2=0x7f97e419cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:19.760 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:19.756 7f97e3fff700 1 --2- 172.21.2.102:0/3038126381 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f97e41b7cf0 0x7f97e419cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:19.760 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:19.756 7f97e3fff700 1 -- 172.21.2.102:0/3038126381 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f97e41b80f0 msgr2=0x7f97e41b84f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:19.760 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:19.756 7f97e3fff700 1 --2- 172.21.2.102:0/3038126381 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f97e41b80f0 0x7f97e41b84f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:19.760 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:19.756 7f97e3fff700 1 -- 172.21.2.102:0/3038126381 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97e41b8d60 con 0x7f97e419da20 2021-02-10T11:37:19.760 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:19.756 7f97e3fff700 1 --2- 172.21.2.102:0/3038126381 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f97e419da20 0x7f97e419de20 secure :-1 s=READY pgs=2933 cs=0 l=1 rev1=1 rx=0x7f97dc009940 tx=0x7f97dc00bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:19.760 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:19.756 7f97dbfff700 1 -- 172.21.2.102:0/3038126381 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f97dc000f90 con 0x7f97e419da20 2021-02-10T11:37:19.760 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:19.756 7f97dbfff700 1 -- 172.21.2.102:0/3038126381 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97dc00c2f0 con 0x7f97e419da20 2021-02-10T11:37:19.761 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:19.756 7f97e9ace700 1 -- 172.21.2.102:0/3038126381 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97e41b8d60 con 0x7f97e419da20 2021-02-10T11:37:19.761 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:19.756 7f97dbfff700 1 -- 172.21.2.102:0/3038126381 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f97dc00c750 con 0x7f97e419da20 2021-02-10T11:37:19.761 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:19.757 7f97e9ace700 1 -- 172.21.2.102:0/3038126381 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97e41b8d60 con 0x7f97e419da20 2021-02-10T11:37:19.761 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:19.757 7f97dbfff700 1 -- 172.21.2.102:0/3038126381 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f97dc001220 con 0x7f97e419da20 2021-02-10T11:37:19.761 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:19.757 7f97dbfff700 1 --2- 172.21.2.102:0/3038126381 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f97c80186c0 0x7f97c801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:19.761 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:19.757 7f97dbfff700 1 -- 172.21.2.102:0/3038126381 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(163..163 src has 1..163) v4 ==== 5645+0+0 (secure 0 0 0) 0x7f97dc032800 con 0x7f97e419da20 2021-02-10T11:37:19.761 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:19.757 7f97e37fe700 1 --2- 172.21.2.102:0/3038126381 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f97c80186c0 0x7f97c801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:19.761 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:19.758 7f97e37fe700 1 --2- 172.21.2.102:0/3038126381 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f97c80186c0 0x7f97c801abc0 crc :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:19.789 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:19.788 7f97e9ace700 1 -- 172.21.2.102:0/3038126381 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97e40ebc00 con 0x7f97e419da20 2021-02-10T11:37:19.791 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:19.790 7f97dbfff700 1 -- 172.21.2.102:0/3038126381 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f97dc00be30 con 0x7f97e419da20 2021-02-10T11:37:19.924 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:19.923 7f97e9ace700 1 -- 172.21.2.102:0/3038126381 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestVolumes.test_nonexistent_subvolume_snapshot_rm"]} v 0) v1 -- 0x7f97e40f0b10 con 0x7f97e419da20 2021-02-10T11:37:20.366 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:20.365 7f97dbfff700 1 -- 172.21.2.102:0/3038126381 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestVolumes.test_nonexistent_subvolume_snapshot_rm"]}]=0 v800) v1 ==== 155+0+0 (secure 0 0 0) 0x7f97dc001220 con 0x7f97e419da20 2021-02-10T11:37:20.393 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:20.392 7f97e9ace700 1 -- 172.21.2.102:0/3038126381 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f97c80186c0 msgr2=0x7f97c801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:20.393 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:20.392 7f97e9ace700 1 --2- 172.21.2.102:0/3038126381 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f97c80186c0 0x7f97c801abc0 crc :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:20.393 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:20.392 7f97e9ace700 1 -- 172.21.2.102:0/3038126381 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f97e419da20 msgr2=0x7f97e419de20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:20.393 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:20.392 7f97e9ace700 1 --2- 172.21.2.102:0/3038126381 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f97e419da20 0x7f97e419de20 secure :-1 s=READY pgs=2933 cs=0 l=1 rev1=1 rx=0x7f97dc009940 tx=0x7f97dc00bcf0).stop 2021-02-10T11:37:20.394 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:20.392 7f97e9ace700 1 -- 172.21.2.102:0/3038126381 shutdown_connections 2021-02-10T11:37:20.394 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:20.392 7f97e9ace700 1 --2- 172.21.2.102:0/3038126381 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f97e419da20 0x7f97e419de20 unknown :-1 s=CLOSED pgs=2933 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:20.394 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:20.392 7f97e9ace700 1 --2- 172.21.2.102:0/3038126381 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f97c80186c0 0x7f97c801abc0 unknown :-1 s=CLOSED pgs=1287 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:20.394 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:20.392 7f97e9ace700 1 --2- 172.21.2.102:0/3038126381 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f97e41b80f0 0x7f97e41b84f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:20.394 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:20.392 7f97e9ace700 1 --2- 172.21.2.102:0/3038126381 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f97e41b7cf0 0x7f97e419cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:20.394 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:20.392 7f97e9ace700 1 -- 172.21.2.102:0/3038126381 >> 172.21.2.102:0/3038126381 conn(0x7f97e40ed810 msgr2=0x7f97e4094980 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:20.394 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:20.392 7f97e9ace700 1 -- 172.21.2.102:0/3038126381 shutdown_connections 2021-02-10T11:37:20.394 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:20.392 7f97e9ace700 1 -- 172.21.2.102:0/3038126381 wait complete. 2021-02-10T11:37:20.401 INFO:tasks.cephfs_test_runner:test_nonexistent_subvolume_snapshot_rm (tasks.cephfs.test_volumes.TestVolumes) ... ok 2021-02-10T11:37:20.402 INFO:tasks.cephfs_test_runner:Starting test: test_recover_auth_metadata_during_authorize (tasks.cephfs.test_volumes.TestVolumes) 2021-02-10T11:37:20.402 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestVolumes.test_recover_auth_metadata_during_authorize' 2021-02-10T11:37:20.830 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:20.813 7fb9308c7700 1 Processor -- start 2021-02-10T11:37:20.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:20.813 7fb9308c7700 1 -- start start 2021-02-10T11:37:20.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:20.813 7fb9308c7700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb92c0d1870 0x7fb92c1b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:20.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:20.813 7fb9308c7700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fb92c1b0a10 0x7fb92c0cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:20.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:20.813 7fb9308c7700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fb92c0d1c70 0x7fb92c0cf950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:20.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:20.813 7fb9308c7700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb92c0fe230 con 0x7fb92c0d1c70 2021-02-10T11:37:20.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:20.813 7fb9308c7700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb92c0e5ed0 con 0x7fb92c1b0a10 2021-02-10T11:37:20.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:20.813 7fb9308c7700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb92c0f0b10 con 0x7fb92c0d1870 2021-02-10T11:37:20.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:20.813 7fb92ae1d700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fb92c0d1c70 0x7fb92c0cf950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:20.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:20.814 7fb92ae1d700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fb92c0d1c70 0x7fb92c0cf950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:38564/0 (socket says 172.21.2.102:38564) 2021-02-10T11:37:20.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:20.814 7fb92ae1d700 1 -- 172.21.2.102:0/3205311680 learned_addr learned my addr 172.21.2.102:0/3205311680 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:37:20.832 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:20.814 7fb92ae1d700 1 -- 172.21.2.102:0/3205311680 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb92c0d1870 msgr2=0x7fb92c1b03f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:37:20.832 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:20.814 7fb92ae1d700 1 --2- 172.21.2.102:0/3205311680 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb92c0d1870 0x7fb92c1b03f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:20.832 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:20.814 7fb92ae1d700 1 -- 172.21.2.102:0/3205311680 >> v2:172.21.2.113:3300/0 conn(0x7fb92c1b0a10 msgr2=0x7fb92c0cf330 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:37:20.832 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:20.814 7fb92ae1d700 1 --2- 172.21.2.102:0/3205311680 >> v2:172.21.2.113:3300/0 conn(0x7fb92c1b0a10 0x7fb92c0cf330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:20.832 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:20.814 7fb92ae1d700 1 -- 172.21.2.102:0/3205311680 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb92c0cff70 con 0x7fb92c0d1c70 2021-02-10T11:37:20.832 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:20.814 7fb92ae1d700 1 --2- 172.21.2.102:0/3205311680 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb92c0d1c70 0x7fb92c0cf950 secure :-1 s=READY pgs=2934 cs=0 l=1 rev1=1 rx=0x7fb924006f00 tx=0x7fb924009f10).ready entity=mon.0 client_cookie=5cac8cab15ddd1bc server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:20.832 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:20.814 7fb92961a700 1 -- 172.21.2.102:0/3205311680 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb92400b3b0 con 0x7fb92c0d1c70 2021-02-10T11:37:20.832 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:20.814 7fb92961a700 1 -- 172.21.2.102:0/3205311680 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb92400b730 con 0x7fb92c0d1c70 2021-02-10T11:37:20.832 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:20.814 7fb92961a700 1 -- 172.21.2.102:0/3205311680 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb92400bc50 con 0x7fb92c0d1c70 2021-02-10T11:37:20.832 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:20.814 7fb9308c7700 1 -- 172.21.2.102:0/3205311680 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb92c0d1c70 msgr2=0x7fb92c0cf950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:20.832 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:20.814 7fb9308c7700 1 --2- 172.21.2.102:0/3205311680 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb92c0d1c70 0x7fb92c0cf950 secure :-1 s=READY pgs=2934 cs=0 l=1 rev1=1 rx=0x7fb924006f00 tx=0x7fb924009f10).stop 2021-02-10T11:37:20.832 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:20.815 7fb9308c7700 1 -- 172.21.2.102:0/3205311680 shutdown_connections 2021-02-10T11:37:20.833 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:20.815 7fb9308c7700 1 --2- 172.21.2.102:0/3205311680 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb92c0d1c70 0x7fb92c0cf950 unknown :-1 s=CLOSED pgs=2934 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:20.833 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:20.815 7fb9308c7700 1 --2- 172.21.2.102:0/3205311680 >> v2:172.21.2.113:3300/0 conn(0x7fb92c1b0a10 0x7fb92c0cf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:20.833 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:20.815 7fb9308c7700 1 --2- 172.21.2.102:0/3205311680 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb92c0d1870 0x7fb92c1b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:20.833 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:20.815 7fb9308c7700 1 -- 172.21.2.102:0/3205311680 >> 172.21.2.102:0/3205311680 conn(0x7fb92c0ce450 msgr2=0x7fb92c0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:20.833 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:20.815 7fb9308c7700 1 -- 172.21.2.102:0/3205311680 shutdown_connections 2021-02-10T11:37:20.833 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:20.815 7fb9308c7700 1 -- 172.21.2.102:0/3205311680 wait complete. 2021-02-10T11:37:20.833 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:20.829 7fb9308c7700 1 Processor -- start 2021-02-10T11:37:20.833 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:20.829 7fb9308c7700 1 -- start start 2021-02-10T11:37:20.835 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:20.830 7fb9308c7700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb92c0ed0f0 0x7fb92c0ed5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:20.835 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:20.830 7fb9308c7700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb92c0ab2e0 0x7fb92c0ab6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:20.835 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:20.830 7fb9308c7700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb92c0abd00 0x7fb92c1b86d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:20.835 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:20.830 7fb9308c7700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb92c1b8cf0 con 0x7fb92c0abd00 2021-02-10T11:37:20.835 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:20.830 7fb9308c7700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb92c1b8ee0 con 0x7fb92c0ab2e0 2021-02-10T11:37:20.835 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:20.830 7fb9308c7700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb92c1b9100 con 0x7fb92c0ed0f0 2021-02-10T11:37:20.835 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:20.830 7fb92ae1d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb92c0abd00 0x7fb92c1b86d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:20.835 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:20.830 7fb92ae1d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb92c0abd00 0x7fb92c1b86d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:38568/0 (socket says 172.21.2.102:38568) 2021-02-10T11:37:20.835 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:20.830 7fb92ae1d700 1 -- 172.21.2.102:0/361359888 learned_addr learned my addr 172.21.2.102:0/361359888 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:37:20.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:20.830 7fb92ae1d700 1 -- 172.21.2.102:0/361359888 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb92c0ed0f0 msgr2=0x7fb92c0ed5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:20.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:20.830 7fb92ae1d700 1 --2- 172.21.2.102:0/361359888 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb92c0ed0f0 0x7fb92c0ed5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:20.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:20.830 7fb92ae1d700 1 -- 172.21.2.102:0/361359888 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb92c0ab2e0 msgr2=0x7fb92c0ab6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:20.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:20.830 7fb92ae1d700 1 --2- 172.21.2.102:0/361359888 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb92c0ab2e0 0x7fb92c0ab6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:20.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:20.830 7fb92ae1d700 1 -- 172.21.2.102:0/361359888 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb92c1b9320 con 0x7fb92c0abd00 2021-02-10T11:37:20.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:20.831 7fb92ae1d700 1 --2- 172.21.2.102:0/361359888 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb92c0abd00 0x7fb92c1b86d0 secure :-1 s=READY pgs=2935 cs=0 l=1 rev1=1 rx=0x7fb9240099f0 tx=0x7fb92400bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:20.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:20.831 7fb922ffd700 1 -- 172.21.2.102:0/361359888 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb924000f90 con 0x7fb92c0abd00 2021-02-10T11:37:20.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:20.831 7fb922ffd700 1 -- 172.21.2.102:0/361359888 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb924003030 con 0x7fb92c0abd00 2021-02-10T11:37:20.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:20.831 7fb9308c7700 1 -- 172.21.2.102:0/361359888 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb92c1b9320 con 0x7fb92c0abd00 2021-02-10T11:37:20.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:20.831 7fb922ffd700 1 -- 172.21.2.102:0/361359888 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb92400c4e0 con 0x7fb92c0abd00 2021-02-10T11:37:20.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:20.831 7fb9308c7700 1 -- 172.21.2.102:0/361359888 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb92c1932a0 con 0x7fb92c0abd00 2021-02-10T11:37:20.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:20.832 7fb922ffd700 1 -- 172.21.2.102:0/361359888 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fb924001270 con 0x7fb92c0abd00 2021-02-10T11:37:20.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:20.832 7fb922ffd700 1 --2- 172.21.2.102:0/361359888 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb910018730 0x7fb91001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:20.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:20.832 7fb922ffd700 1 -- 172.21.2.102:0/361359888 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(163..163 src has 1..163) v4 ==== 5645+0+0 (secure 0 0 0) 0x7fb924032a00 con 0x7fb92c0abd00 2021-02-10T11:37:20.837 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:20.832 7fb92a61c700 1 --2- 172.21.2.102:0/361359888 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb910018730 0x7fb91001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:20.837 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:20.832 7fb92a61c700 1 --2- 172.21.2.102:0/361359888 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb910018730 0x7fb91001ac10 crc :-1 s=READY pgs=1288 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:20.864 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:20.863 7fb9308c7700 1 -- 172.21.2.102:0/361359888 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb92c192d70 con 0x7fb92c0abd00 2021-02-10T11:37:20.867 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:20.866 7fb922ffd700 1 -- 172.21.2.102:0/361359888 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fb924000f90 con 0x7fb92c0abd00 2021-02-10T11:37:20.998 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:20.997 7fb9308c7700 1 -- 172.21.2.102:0/361359888 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestVolumes.test_recover_auth_metadata_during_authorize"]} v 0) v1 -- 0x7fb92c192d70 con 0x7fb92c0abd00 2021-02-10T11:37:21.369 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:21.368 7fb922ffd700 1 -- 172.21.2.102:0/361359888 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestVolumes.test_recover_auth_metadata_during_authorize"]}]=0 v801) v1 ==== 163+0+0 (secure 0 0 0) 0x7fb924001270 con 0x7fb92c0abd00 2021-02-10T11:37:21.417 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:21.415 7fb9308c7700 1 -- 172.21.2.102:0/361359888 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb910018730 msgr2=0x7fb91001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:21.417 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:21.415 7fb9308c7700 1 --2- 172.21.2.102:0/361359888 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb910018730 0x7fb91001ac10 crc :-1 s=READY pgs=1288 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:21.417 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:21.415 7fb9308c7700 1 -- 172.21.2.102:0/361359888 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb92c0abd00 msgr2=0x7fb92c1b86d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:21.417 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:21.415 7fb9308c7700 1 --2- 172.21.2.102:0/361359888 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb92c0abd00 0x7fb92c1b86d0 secure :-1 s=READY pgs=2935 cs=0 l=1 rev1=1 rx=0x7fb9240099f0 tx=0x7fb92400bcd0).stop 2021-02-10T11:37:21.417 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:21.415 7fb9308c7700 1 -- 172.21.2.102:0/361359888 shutdown_connections 2021-02-10T11:37:21.417 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:21.415 7fb9308c7700 1 --2- 172.21.2.102:0/361359888 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb92c0abd00 0x7fb92c1b86d0 unknown :-1 s=CLOSED pgs=2935 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:21.417 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:21.415 7fb9308c7700 1 --2- 172.21.2.102:0/361359888 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb910018730 0x7fb91001ac10 unknown :-1 s=CLOSED pgs=1288 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:21.417 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:21.415 7fb9308c7700 1 --2- 172.21.2.102:0/361359888 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb92c0ab2e0 0x7fb92c0ab6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:21.418 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:21.415 7fb9308c7700 1 --2- 172.21.2.102:0/361359888 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb92c0ed0f0 0x7fb92c0ed5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:21.418 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:21.415 7fb9308c7700 1 -- 172.21.2.102:0/361359888 >> 172.21.2.102:0/361359888 conn(0x7fb92c19c580 msgr2=0x7fb92c0916f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:21.418 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:21.416 7fb9308c7700 1 -- 172.21.2.102:0/361359888 shutdown_connections 2021-02-10T11:37:21.418 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:21.416 7fb9308c7700 1 -- 172.21.2.102:0/361359888 wait complete. 2021-02-10T11:37:21.425 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config set mon mon_allow_pool_delete True 2021-02-10T11:37:21.857 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:21.839 7fc6bacd8700 1 Processor -- start 2021-02-10T11:37:21.857 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:21.839 7fc6bacd8700 1 -- start start 2021-02-10T11:37:21.857 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:21.840 7fc6bacd8700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fc6b40d1870 0x7fc6b41b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:21.857 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:21.840 7fc6bacd8700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fc6b40cff70 0x7fc6b4109130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:21.857 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:21.840 7fc6bacd8700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6b40fe1c0 con 0x7fc6b41b0ae0 2021-02-10T11:37:21.857 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:21.840 7fc6bacd8700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6b4109ad0 con 0x7fc6b40d1870 2021-02-10T11:37:21.858 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:21.840 7fc6bacd8700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6b4109750 con 0x7fc6b40cff70 2021-02-10T11:37:21.858 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:21.840 7fc6b8af4700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fc6b40d1870 0x7fc6b41b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:21.858 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:21.840 7fc6b3fff700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fc6b41b0ae0 0x7fc6b4108940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:59032/0 (socket says 172.21.2.102:59032) 2021-02-10T11:37:21.858 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:21.840 7fc6b3fff700 1 -- 172.21.2.102:0/3913560988 learned_addr learned my addr 172.21.2.102:0/3913560988 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:37:21.858 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:21.840 7fc6b8af4700 1 -- 172.21.2.102:0/3913560988 >> v2:172.21.2.113:3300/0 conn(0x7fc6b40cff70 msgr2=0x7fc6b4109130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:37:21.858 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:21.840 7fc6b8af4700 1 --2- 172.21.2.102:0/3913560988 >> v2:172.21.2.113:3300/0 conn(0x7fc6b40cff70 0x7fc6b4109130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:21.858 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:21.840 7fc6b8af4700 1 -- 172.21.2.102:0/3913560988 >> v1:172.21.2.102:6789/0 conn(0x7fc6b41b0ae0 legacy=0x7fc6b4108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:21.858 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:21.840 7fc6b8af4700 1 -- 172.21.2.102:0/3913560988 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6b4088070 con 0x7fc6b40d1870 2021-02-10T11:37:21.858 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:21.841 7fc6b8af4700 1 --2- 172.21.2.102:0/3913560988 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc6b40d1870 0x7fc6b41b04c0 secure :-1 s=READY pgs=2937 cs=0 l=1 rev1=1 rx=0x7fc69c006650 tx=0x7fc69c009a80).ready entity=mon.0 client_cookie=e59448633deb533e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:21.858 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:21.841 7fc6b37fe700 1 -- 172.21.2.102:0/3913560988 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc69c0009b0 con 0x7fc6b40d1870 2021-02-10T11:37:21.858 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:21.841 7fc6b37fe700 1 -- 172.21.2.102:0/3913560988 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc69c00c0c0 con 0x7fc6b40d1870 2021-02-10T11:37:21.858 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:21.841 7fc6b37fe700 1 -- 172.21.2.102:0/3913560988 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc69c00c5e0 con 0x7fc6b40d1870 2021-02-10T11:37:21.858 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:21.841 7fc6bacd8700 1 -- 172.21.2.102:0/3913560988 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc6b40d1870 msgr2=0x7fc6b41b04c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:21.859 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:21.841 7fc6bacd8700 1 --2- 172.21.2.102:0/3913560988 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc6b40d1870 0x7fc6b41b04c0 secure :-1 s=READY pgs=2937 cs=0 l=1 rev1=1 rx=0x7fc69c006650 tx=0x7fc69c009a80).stop 2021-02-10T11:37:21.859 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:21.842 7fc6bacd8700 1 -- 172.21.2.102:0/3913560988 shutdown_connections 2021-02-10T11:37:21.859 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:21.842 7fc6bacd8700 1 --2- 172.21.2.102:0/3913560988 >> v2:172.21.2.113:3300/0 conn(0x7fc6b40cff70 0x7fc6b4109130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:21.859 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:21.842 7fc6bacd8700 1 --2- 172.21.2.102:0/3913560988 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc6b40d1870 0x7fc6b41b04c0 unknown :-1 s=CLOSED pgs=2937 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:21.859 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:21.842 7fc6bacd8700 1 -- 172.21.2.102:0/3913560988 >> 172.21.2.102:0/3913560988 conn(0x7fc6b40ce420 msgr2=0x7fc6b40d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:21.859 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:21.842 7fc6bacd8700 1 -- 172.21.2.102:0/3913560988 shutdown_connections 2021-02-10T11:37:21.859 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:21.842 7fc6bacd8700 1 -- 172.21.2.102:0/3913560988 wait complete. 2021-02-10T11:37:21.860 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:21.856 7fc6bacd8700 1 Processor -- start 2021-02-10T11:37:21.860 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:21.856 7fc6bacd8700 1 -- start start 2021-02-10T11:37:21.860 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:21.857 7fc6bacd8700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc6b40c3820 0x7fc6b4084cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:21.860 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:21.857 7fc6bacd8700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc6b40a0290 0x7fc6b40852d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:21.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:21.857 7fc6bacd8700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc6b4086270 0x7fc6b4086670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:21.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:21.857 7fc6bacd8700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6b40c0700 con 0x7fc6b4086270 2021-02-10T11:37:21.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:21.857 7fc6bacd8700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6b40858f0 con 0x7fc6b40a0290 2021-02-10T11:37:21.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:21.857 7fc6bacd8700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6b4085b10 con 0x7fc6b40c3820 2021-02-10T11:37:21.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:21.857 7fc6b92f5700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc6b4086270 0x7fc6b4086670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:21.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:21.857 7fc6b92f5700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc6b4086270 0x7fc6b4086670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:38586/0 (socket says 172.21.2.102:38586) 2021-02-10T11:37:21.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:21.857 7fc6b92f5700 1 -- 172.21.2.102:0/2625964459 learned_addr learned my addr 172.21.2.102:0/2625964459 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:37:21.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:21.858 7fc6b92f5700 1 -- 172.21.2.102:0/2625964459 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc6b40c3820 msgr2=0x7fc6b4084cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:21.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:21.858 7fc6b92f5700 1 --2- 172.21.2.102:0/2625964459 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc6b40c3820 0x7fc6b4084cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:21.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:21.858 7fc6b92f5700 1 -- 172.21.2.102:0/2625964459 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc6b40a0290 msgr2=0x7fc6b40852d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:21.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:21.858 7fc6b92f5700 1 --2- 172.21.2.102:0/2625964459 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc6b40a0290 0x7fc6b40852d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:21.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:21.858 7fc6b92f5700 1 -- 172.21.2.102:0/2625964459 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6b4086c90 con 0x7fc6b4086270 2021-02-10T11:37:21.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:21.858 7fc6b92f5700 1 --2- 172.21.2.102:0/2625964459 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc6b4086270 0x7fc6b4086670 secure :-1 s=READY pgs=2938 cs=0 l=1 rev1=1 rx=0x7fc6ac006f10 tx=0x7fc6ac00b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:21.862 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:21.858 7fc6b17fa700 1 -- 172.21.2.102:0/2625964459 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc6ac006240 con 0x7fc6b4086270 2021-02-10T11:37:21.862 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:21.858 7fc6b17fa700 1 -- 172.21.2.102:0/2625964459 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6ac000c10 con 0x7fc6b4086270 2021-02-10T11:37:21.862 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:21.858 7fc6b17fa700 1 -- 172.21.2.102:0/2625964459 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc6ac005770 con 0x7fc6b4086270 2021-02-10T11:37:21.862 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:21.858 7fc6bacd8700 1 -- 172.21.2.102:0/2625964459 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6b4086c90 con 0x7fc6b4086270 2021-02-10T11:37:21.862 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:21.858 7fc6bacd8700 1 -- 172.21.2.102:0/2625964459 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6b4086c90 con 0x7fc6b4086270 2021-02-10T11:37:21.862 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:21.859 7fc6b17fa700 1 -- 172.21.2.102:0/2625964459 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fc6ac006030 con 0x7fc6b4086270 2021-02-10T11:37:21.862 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:21.859 7fc6b17fa700 1 --2- 172.21.2.102:0/2625964459 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc6980186c0 0x7fc69801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:21.862 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:21.859 7fc6b17fa700 1 -- 172.21.2.102:0/2625964459 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(164..164 src has 1..164) v4 ==== 5617+0+0 (secure 0 0 0) 0x7fc6ac0325c0 con 0x7fc6b4086270 2021-02-10T11:37:21.862 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:21.859 7fc6b8af4700 1 --2- 172.21.2.102:0/2625964459 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc6980186c0 0x7fc69801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:21.863 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:21.860 7fc6b8af4700 1 --2- 172.21.2.102:0/2625964459 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc6980186c0 0x7fc69801abc0 crc :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:21.891 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:21.890 7fc6bacd8700 1 -- 172.21.2.102:0/2625964459 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6b4082a70 con 0x7fc6b4086270 2021-02-10T11:37:21.893 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:21.892 7fc6b17fa700 1 -- 172.21.2.102:0/2625964459 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fc6ac025d30 con 0x7fc6b4086270 2021-02-10T11:37:22.041 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:22.040 7fc6bacd8700 1 -- 172.21.2.102:0/2625964459 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fc6b4082a70 con 0x7fc6b4086270 2021-02-10T11:37:22.045 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:22.044 7fc6b17fa700 1 -- 172.21.2.102:0/2625964459 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v42)=0 v42) v1 ==== 122+0+0 (secure 0 0 0) 0x7fc6ac025d30 con 0x7fc6b4086270 2021-02-10T11:37:22.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:22.049 7fc6bacd8700 1 -- 172.21.2.102:0/2625964459 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc6980186c0 msgr2=0x7fc69801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:22.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:22.049 7fc6bacd8700 1 --2- 172.21.2.102:0/2625964459 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc6980186c0 0x7fc69801abc0 crc :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:22.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:22.049 7fc6bacd8700 1 -- 172.21.2.102:0/2625964459 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc6b4086270 msgr2=0x7fc6b4086670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:22.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:22.049 7fc6bacd8700 1 --2- 172.21.2.102:0/2625964459 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc6b4086270 0x7fc6b4086670 secure :-1 s=READY pgs=2938 cs=0 l=1 rev1=1 rx=0x7fc6ac006f10 tx=0x7fc6ac00b990).stop 2021-02-10T11:37:22.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:22.050 7fc6bacd8700 1 -- 172.21.2.102:0/2625964459 shutdown_connections 2021-02-10T11:37:22.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:22.050 7fc6bacd8700 1 --2- 172.21.2.102:0/2625964459 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc6b4086270 0x7fc6b4086670 unknown :-1 s=CLOSED pgs=2938 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:22.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:22.050 7fc6bacd8700 1 --2- 172.21.2.102:0/2625964459 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc6980186c0 0x7fc69801abc0 unknown :-1 s=CLOSED pgs=1289 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:22.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:22.050 7fc6bacd8700 1 --2- 172.21.2.102:0/2625964459 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc6b40a0290 0x7fc6b40852d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:22.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:22.050 7fc6bacd8700 1 --2- 172.21.2.102:0/2625964459 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc6b40c3820 0x7fc6b4084cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:22.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:22.050 7fc6bacd8700 1 -- 172.21.2.102:0/2625964459 >> 172.21.2.102:0/2625964459 conn(0x7fc6b40872c0 msgr2=0x7fc6b4095700 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:22.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:22.050 7fc6bacd8700 1 -- 172.21.2.102:0/2625964459 shutdown_connections 2021-02-10T11:37:22.052 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:22.050 7fc6bacd8700 1 -- 172.21.2.102:0/2625964459 wait complete. 2021-02-10T11:37:22.059 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2021-02-10T11:37:22.059 DEBUG:teuthology.orchestra.run.gibba002:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:37:22.406 DEBUG:teuthology.orchestra.run.gibba013:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:37:22.750 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2021-02-10T11:37:22.750 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:37:22.751 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:37:22.751 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:37:23.087 INFO:teuthology.orchestra.run.gibba002.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.0’: No such file or directory 2021-02-10T11:37:23.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:37:23.088 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.0 2021-02-10T11:37:23.088 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:37:23.088 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:37:23.088 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:37:23.423 INFO:teuthology.orchestra.run.gibba002.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.1’: No such file or directory 2021-02-10T11:37:23.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:37:23.424 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2021-02-10T11:37:23.424 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:37:23.424 DEBUG:teuthology.orchestra.run.gibba013:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:37:23.424 DEBUG:teuthology.orchestra.run.gibba013:> ' -- /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:37:23.760 INFO:teuthology.orchestra.run.gibba013.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.2’: No such file or directory 2021-02-10T11:37:23.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:37:23.760 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2021-02-10T11:37:23.761 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:37:23.761 DEBUG:teuthology.orchestra.run.gibba013:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:37:23.761 DEBUG:teuthology.orchestra.run.gibba013:> ' -- /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:37:24.096 INFO:teuthology.orchestra.run.gibba013.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.3’: No such file or directory 2021-02-10T11:37:24.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:37:24.096 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.3 2021-02-10T11:37:24.096 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json-pretty 2021-02-10T11:37:24.524 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:24.506 7fd2ca4cd700 1 Processor -- start 2021-02-10T11:37:24.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:24.506 7fd2ca4cd700 1 -- start start 2021-02-10T11:37:24.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:24.507 7fd2ca4cd700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fd2c41b0470 0x7fd2c41b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:24.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:24.507 7fd2ca4cd700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2c40f0b50 con 0x7fd2c40ef610 2021-02-10T11:37:24.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:24.507 7fd2ca4cd700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2c40f24e0 con 0x7fd2c40f20e0 2021-02-10T11:37:24.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:24.507 7fd2ca4cd700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2c40e5ea0 con 0x7fd2c41b0470 2021-02-10T11:37:24.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:24.507 7fd2c3fff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fd2c41b0470 0x7fd2c41b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:24.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:24.507 7fd2c37fe700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fd2c40ef610 0x7fd2c40efa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:59046/0 (socket says 172.21.2.102:59046) 2021-02-10T11:37:24.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:24.507 7fd2c37fe700 1 -- 172.21.2.102:0/768321627 learned_addr learned my addr 172.21.2.102:0/768321627 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:37:24.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:24.507 7fd2c3fff700 1 -- 172.21.2.102:0/768321627 >> v1:172.21.2.113:6789/0 conn(0x7fd2c40f20e0 legacy=0x7fd2c40f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:24.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:24.507 7fd2c3fff700 1 -- 172.21.2.102:0/768321627 >> v1:172.21.2.102:6789/0 conn(0x7fd2c40ef610 legacy=0x7fd2c40efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:24.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:24.507 7fd2c3fff700 1 -- 172.21.2.102:0/768321627 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2c40e52e0 con 0x7fd2c41b0470 2021-02-10T11:37:24.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:24.507 7fd2c3fff700 1 --2- 172.21.2.102:0/768321627 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd2c41b0470 0x7fd2c41b0870 secure :-1 s=READY pgs=2940 cs=0 l=1 rev1=1 rx=0x7fd2b4006650 tx=0x7fd2b4009a80).ready entity=mon.0 client_cookie=c9580a10c09cb949 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:24.526 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:24.508 7fd2c2ffd700 1 -- 172.21.2.102:0/768321627 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd2b400c0c0 con 0x7fd2c41b0470 2021-02-10T11:37:24.526 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:24.508 7fd2c2ffd700 1 -- 172.21.2.102:0/768321627 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2b400c340 con 0x7fd2c41b0470 2021-02-10T11:37:24.526 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:24.508 7fd2c2ffd700 1 -- 172.21.2.102:0/768321627 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd2b400c860 con 0x7fd2c41b0470 2021-02-10T11:37:24.526 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:24.508 7fd2ca4cd700 1 -- 172.21.2.102:0/768321627 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd2c41b0470 msgr2=0x7fd2c41b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:24.526 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:24.508 7fd2ca4cd700 1 --2- 172.21.2.102:0/768321627 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd2c41b0470 0x7fd2c41b0870 secure :-1 s=READY pgs=2940 cs=0 l=1 rev1=1 rx=0x7fd2b4006650 tx=0x7fd2b4009a80).stop 2021-02-10T11:37:24.526 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:24.508 7fd2ca4cd700 1 -- 172.21.2.102:0/768321627 shutdown_connections 2021-02-10T11:37:24.526 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:24.508 7fd2ca4cd700 1 --2- 172.21.2.102:0/768321627 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd2c41b0470 0x7fd2c41b0870 unknown :-1 s=CLOSED pgs=2940 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:24.526 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:24.508 7fd2ca4cd700 1 -- 172.21.2.102:0/768321627 >> 172.21.2.102:0/768321627 conn(0x7fd2c40ce3e0 msgr2=0x7fd2c40e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:24.526 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:24.508 7fd2ca4cd700 1 -- 172.21.2.102:0/768321627 shutdown_connections 2021-02-10T11:37:24.526 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:24.508 7fd2ca4cd700 1 -- 172.21.2.102:0/768321627 wait complete. 2021-02-10T11:37:24.528 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:24.523 7fd2ca4cd700 1 Processor -- start 2021-02-10T11:37:24.528 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:24.523 7fd2ca4cd700 1 -- start start 2021-02-10T11:37:24.528 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:24.523 7fd2ca4cd700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd2c4092e30 0x7fd2c41c6320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:24.528 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:24.523 7fd2ca4cd700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd2c40964a0 0x7fd2c40968a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:24.528 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:24.523 7fd2ca4cd700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd2c4096ec0 0x7fd2c4084f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:24.528 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:24.523 7fd2ca4cd700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2c40932b0 con 0x7fd2c40964a0 2021-02-10T11:37:24.529 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:24.523 7fd2ca4cd700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2c41b5f70 con 0x7fd2c4096ec0 2021-02-10T11:37:24.529 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:24.523 7fd2ca4cd700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2c41b6150 con 0x7fd2c4092e30 2021-02-10T11:37:24.529 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:24.524 7fd2c37fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd2c40964a0 0x7fd2c40968a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:24.529 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:24.524 7fd2c37fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd2c40964a0 0x7fd2c40968a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:38598/0 (socket says 172.21.2.102:38598) 2021-02-10T11:37:24.529 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:24.524 7fd2c37fe700 1 -- 172.21.2.102:0/3690019535 learned_addr learned my addr 172.21.2.102:0/3690019535 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:37:24.529 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:24.524 7fd2c3fff700 1 --2- 172.21.2.102:0/3690019535 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd2c4092e30 0x7fd2c41c6320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:24.529 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:24.524 7fd2c8aea700 1 --2- 172.21.2.102:0/3690019535 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd2c4096ec0 0x7fd2c4084f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:24.529 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:24.524 7fd2c37fe700 1 -- 172.21.2.102:0/3690019535 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd2c4092e30 msgr2=0x7fd2c41c6320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:24.529 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:24.524 7fd2c37fe700 1 --2- 172.21.2.102:0/3690019535 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd2c4092e30 0x7fd2c41c6320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:24.529 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:24.524 7fd2c37fe700 1 -- 172.21.2.102:0/3690019535 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd2c4096ec0 msgr2=0x7fd2c4084f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:24.529 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:24.524 7fd2c37fe700 1 --2- 172.21.2.102:0/3690019535 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd2c4096ec0 0x7fd2c4084f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:24.529 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:24.524 7fd2c37fe700 1 -- 172.21.2.102:0/3690019535 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2c41c6970 con 0x7fd2c40964a0 2021-02-10T11:37:24.529 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:24.524 7fd2c37fe700 1 --2- 172.21.2.102:0/3690019535 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd2c40964a0 0x7fd2c40968a0 secure :-1 s=READY pgs=2941 cs=0 l=1 rev1=1 rx=0x7fd2b000a5a0 tx=0x7fd2b000a940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:24.530 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:24.524 7fd2c0ff9700 1 -- 172.21.2.102:0/3690019535 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd2b0000c90 con 0x7fd2c40964a0 2021-02-10T11:37:24.530 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:24.524 7fd2c0ff9700 1 -- 172.21.2.102:0/3690019535 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2b0000ed0 con 0x7fd2c40964a0 2021-02-10T11:37:24.530 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:24.524 7fd2ca4cd700 1 -- 172.21.2.102:0/3690019535 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2c41c6970 con 0x7fd2c40964a0 2021-02-10T11:37:24.530 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:24.525 7fd2ca4cd700 1 -- 172.21.2.102:0/3690019535 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2c41c6970 con 0x7fd2c40964a0 2021-02-10T11:37:24.530 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:24.525 7fd2c0ff9700 1 -- 172.21.2.102:0/3690019535 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd2b0004620 con 0x7fd2c40964a0 2021-02-10T11:37:24.530 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:24.525 7fd2c0ff9700 1 -- 172.21.2.102:0/3690019535 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd2b0007070 con 0x7fd2c40964a0 2021-02-10T11:37:24.530 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:24.525 7fd2c0ff9700 1 --2- 172.21.2.102:0/3690019535 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd2a0018730 0x7fd2a001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:24.530 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:24.525 7fd2c0ff9700 1 -- 172.21.2.102:0/3690019535 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(164..164 src has 1..164) v4 ==== 5617+0+0 (secure 0 0 0) 0x7fd2b0032810 con 0x7fd2c40964a0 2021-02-10T11:37:24.530 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:24.525 7fd2c3fff700 1 --2- 172.21.2.102:0/3690019535 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd2a0018730 0x7fd2a001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:24.530 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:24.526 7fd2c3fff700 1 --2- 172.21.2.102:0/3690019535 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd2a0018730 0x7fd2a001ac10 crc :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:24.558 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:24.556 7fd2ca4cd700 1 -- 172.21.2.102:0/3690019535 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2c407ed50 con 0x7fd2c40964a0 2021-02-10T11:37:24.561 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:24.559 7fd2c0ff9700 1 -- 172.21.2.102:0/3690019535 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd2b0000c90 con 0x7fd2c40964a0 2021-02-10T11:37:24.687 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:24.685 7fd2ca4cd700 1 -- 172.21.2.102:0/3690019535 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7fd2c40ce3e0 con 0x7fd2c40964a0 2021-02-10T11:37:24.688 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:24.686 7fd2c0ff9700 1 -- 172.21.2.102:0/3690019535 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v164) v1 ==== 81+0+30586 (secure 0 0 0) 0x7fd2b0000ea0 con 0x7fd2c40964a0 2021-02-10T11:37:24.688 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:37:24.688 INFO:teuthology.orchestra.run.gibba002.stdout:{ 2021-02-10T11:37:24.689 INFO:teuthology.orchestra.run.gibba002.stdout: "epoch": 164, 2021-02-10T11:37:24.689 INFO:teuthology.orchestra.run.gibba002.stdout: "fsid": "68db30ea-dbce-4aef-a6df-f01c6fea0513", 2021-02-10T11:37:24.689 INFO:teuthology.orchestra.run.gibba002.stdout: "created": "2021-02-10 11:03:06.331557", 2021-02-10T11:37:24.689 INFO:teuthology.orchestra.run.gibba002.stdout: "modified": "2021-02-10 11:37:21.337069", 2021-02-10T11:37:24.689 INFO:teuthology.orchestra.run.gibba002.stdout: "last_up_change": "2021-02-10 11:03:32.690562", 2021-02-10T11:37:24.689 INFO:teuthology.orchestra.run.gibba002.stdout: "last_in_change": "2021-02-10 11:03:32.690562", 2021-02-10T11:37:24.689 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2021-02-10T11:37:24.689 INFO:teuthology.orchestra.run.gibba002.stdout: "flags_num": 5799936, 2021-02-10T11:37:24.689 INFO:teuthology.orchestra.run.gibba002.stdout: "flags_set": [ 2021-02-10T11:37:24.689 INFO:teuthology.orchestra.run.gibba002.stdout: "pglog_hardlimit", 2021-02-10T11:37:24.689 INFO:teuthology.orchestra.run.gibba002.stdout: "purged_snapdirs", 2021-02-10T11:37:24.689 INFO:teuthology.orchestra.run.gibba002.stdout: "recovery_deletes", 2021-02-10T11:37:24.690 INFO:teuthology.orchestra.run.gibba002.stdout: "sortbitwise" 2021-02-10T11:37:24.690 INFO:teuthology.orchestra.run.gibba002.stdout: ], 2021-02-10T11:37:24.690 INFO:teuthology.orchestra.run.gibba002.stdout: "crush_version": 4, 2021-02-10T11:37:24.690 INFO:teuthology.orchestra.run.gibba002.stdout: "full_ratio": 0.94999998807907104, 2021-02-10T11:37:24.690 INFO:teuthology.orchestra.run.gibba002.stdout: "backfillfull_ratio": 0.89999997615814209, 2021-02-10T11:37:24.690 INFO:teuthology.orchestra.run.gibba002.stdout: "nearfull_ratio": 0.85000002384185791, 2021-02-10T11:37:24.690 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_snapshot": "", 2021-02-10T11:37:24.690 INFO:teuthology.orchestra.run.gibba002.stdout: "pool_max": 35, 2021-02-10T11:37:24.690 INFO:teuthology.orchestra.run.gibba002.stdout: "max_osd": 8, 2021-02-10T11:37:24.690 INFO:teuthology.orchestra.run.gibba002.stdout: "require_min_compat_client": "jewel", 2021-02-10T11:37:24.690 INFO:teuthology.orchestra.run.gibba002.stdout: "min_compat_client": "jewel", 2021-02-10T11:37:24.690 INFO:teuthology.orchestra.run.gibba002.stdout: "require_osd_release": "nautilus", 2021-02-10T11:37:24.690 INFO:teuthology.orchestra.run.gibba002.stdout: "pools": [ 2021-02-10T11:37:24.691 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:37:24.691 INFO:teuthology.orchestra.run.gibba002.stdout: "pool": 1, 2021-02-10T11:37:24.691 INFO:teuthology.orchestra.run.gibba002.stdout: "pool_name": "rbd", 2021-02-10T11:37:24.691 INFO:teuthology.orchestra.run.gibba002.stdout: "create_time": "2021-02-10 11:03:37.322519", 2021-02-10T11:37:24.691 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 1, 2021-02-10T11:37:24.691 INFO:teuthology.orchestra.run.gibba002.stdout: "flags_names": "hashpspool", 2021-02-10T11:37:24.691 INFO:teuthology.orchestra.run.gibba002.stdout: "type": 1, 2021-02-10T11:37:24.691 INFO:teuthology.orchestra.run.gibba002.stdout: "size": 2, 2021-02-10T11:37:24.691 INFO:teuthology.orchestra.run.gibba002.stdout: "min_size": 1, 2021-02-10T11:37:24.691 INFO:teuthology.orchestra.run.gibba002.stdout: "crush_rule": 0, 2021-02-10T11:37:24.691 INFO:teuthology.orchestra.run.gibba002.stdout: "object_hash": 2, 2021-02-10T11:37:24.691 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:37:24.691 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num": 8, 2021-02-10T11:37:24.691 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_placement_num": 8, 2021-02-10T11:37:24.692 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_placement_num_target": 8, 2021-02-10T11:37:24.692 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num_target": 8, 2021-02-10T11:37:24.692 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num_pending": 8, 2021-02-10T11:37:24.692 INFO:teuthology.orchestra.run.gibba002.stdout: "last_pg_merge_meta": { 2021-02-10T11:37:24.692 INFO:teuthology.orchestra.run.gibba002.stdout: "source_pgid": "0.0", 2021-02-10T11:37:24.692 INFO:teuthology.orchestra.run.gibba002.stdout: "ready_epoch": 0, 2021-02-10T11:37:24.692 INFO:teuthology.orchestra.run.gibba002.stdout: "last_epoch_started": 0, 2021-02-10T11:37:24.692 INFO:teuthology.orchestra.run.gibba002.stdout: "last_epoch_clean": 0, 2021-02-10T11:37:24.692 INFO:teuthology.orchestra.run.gibba002.stdout: "source_version": "0'0", 2021-02-10T11:37:24.692 INFO:teuthology.orchestra.run.gibba002.stdout: "target_version": "0'0" 2021-02-10T11:37:24.692 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:37:24.692 INFO:teuthology.orchestra.run.gibba002.stdout: "last_change": "16", 2021-02-10T11:37:24.692 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend": "0", 2021-02-10T11:37:24.693 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:37:24.693 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:37:24.693 INFO:teuthology.orchestra.run.gibba002.stdout: "auid": 0, 2021-02-10T11:37:24.693 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:37:24.693 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_seq": 0, 2021-02-10T11:37:24.693 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_epoch": 0, 2021-02-10T11:37:24.693 INFO:teuthology.orchestra.run.gibba002.stdout: "pool_snaps": [], 2021-02-10T11:37:24.693 INFO:teuthology.orchestra.run.gibba002.stdout: "removed_snaps": "[]", 2021-02-10T11:37:24.693 INFO:teuthology.orchestra.run.gibba002.stdout: "quota_max_bytes": 0, 2021-02-10T11:37:24.693 INFO:teuthology.orchestra.run.gibba002.stdout: "quota_max_objects": 0, 2021-02-10T11:37:24.693 INFO:teuthology.orchestra.run.gibba002.stdout: "tiers": [], 2021-02-10T11:37:24.693 INFO:teuthology.orchestra.run.gibba002.stdout: "tier_of": -1, 2021-02-10T11:37:24.693 INFO:teuthology.orchestra.run.gibba002.stdout: "read_tier": -1, 2021-02-10T11:37:24.693 INFO:teuthology.orchestra.run.gibba002.stdout: "write_tier": -1, 2021-02-10T11:37:24.694 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_mode": "none", 2021-02-10T11:37:24.694 INFO:teuthology.orchestra.run.gibba002.stdout: "target_max_bytes": 0, 2021-02-10T11:37:24.694 INFO:teuthology.orchestra.run.gibba002.stdout: "target_max_objects": 0, 2021-02-10T11:37:24.694 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:37:24.694 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:37:24.694 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:37:24.694 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_min_flush_age": 0, 2021-02-10T11:37:24.694 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_min_evict_age": 0, 2021-02-10T11:37:24.694 INFO:teuthology.orchestra.run.gibba002.stdout: "erasure_code_profile": "", 2021-02-10T11:37:24.694 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_params": { 2021-02-10T11:37:24.694 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "none" 2021-02-10T11:37:24.694 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:37:24.695 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_period": 0, 2021-02-10T11:37:24.695 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_count": 0, 2021-02-10T11:37:24.695 INFO:teuthology.orchestra.run.gibba002.stdout: "use_gmt_hitset": true, 2021-02-10T11:37:24.695 INFO:teuthology.orchestra.run.gibba002.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:37:24.695 INFO:teuthology.orchestra.run.gibba002.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:37:24.695 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:37:24.695 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:37:24.695 INFO:teuthology.orchestra.run.gibba002.stdout: "grade_table": [], 2021-02-10T11:37:24.695 INFO:teuthology.orchestra.run.gibba002.stdout: "stripe_width": 0, 2021-02-10T11:37:24.695 INFO:teuthology.orchestra.run.gibba002.stdout: "expected_num_objects": 0, 2021-02-10T11:37:24.695 INFO:teuthology.orchestra.run.gibba002.stdout: "fast_read": false, 2021-02-10T11:37:24.695 INFO:teuthology.orchestra.run.gibba002.stdout: "options": {}, 2021-02-10T11:37:24.695 INFO:teuthology.orchestra.run.gibba002.stdout: "application_metadata": { 2021-02-10T11:37:24.695 INFO:teuthology.orchestra.run.gibba002.stdout: "rbd": {} 2021-02-10T11:37:24.696 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:37:24.696 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:37:24.696 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:37:24.696 INFO:teuthology.orchestra.run.gibba002.stdout: "pool": 34, 2021-02-10T11:37:24.696 INFO:teuthology.orchestra.run.gibba002.stdout: "pool_name": "cephfs_metadata", 2021-02-10T11:37:24.696 INFO:teuthology.orchestra.run.gibba002.stdout: "create_time": "2021-02-10 11:36:44.297293", 2021-02-10T11:37:24.696 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 1, 2021-02-10T11:37:24.696 INFO:teuthology.orchestra.run.gibba002.stdout: "flags_names": "hashpspool", 2021-02-10T11:37:24.696 INFO:teuthology.orchestra.run.gibba002.stdout: "type": 1, 2021-02-10T11:37:24.696 INFO:teuthology.orchestra.run.gibba002.stdout: "size": 2, 2021-02-10T11:37:24.696 INFO:teuthology.orchestra.run.gibba002.stdout: "min_size": 1, 2021-02-10T11:37:24.696 INFO:teuthology.orchestra.run.gibba002.stdout: "crush_rule": 0, 2021-02-10T11:37:24.696 INFO:teuthology.orchestra.run.gibba002.stdout: "object_hash": 2, 2021-02-10T11:37:24.697 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:37:24.697 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num": 8, 2021-02-10T11:37:24.697 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_placement_num": 8, 2021-02-10T11:37:24.697 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_placement_num_target": 8, 2021-02-10T11:37:24.697 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num_target": 8, 2021-02-10T11:37:24.697 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num_pending": 8, 2021-02-10T11:37:24.697 INFO:teuthology.orchestra.run.gibba002.stdout: "last_pg_merge_meta": { 2021-02-10T11:37:24.697 INFO:teuthology.orchestra.run.gibba002.stdout: "source_pgid": "0.0", 2021-02-10T11:37:24.697 INFO:teuthology.orchestra.run.gibba002.stdout: "ready_epoch": 0, 2021-02-10T11:37:24.697 INFO:teuthology.orchestra.run.gibba002.stdout: "last_epoch_started": 0, 2021-02-10T11:37:24.697 INFO:teuthology.orchestra.run.gibba002.stdout: "last_epoch_clean": 0, 2021-02-10T11:37:24.697 INFO:teuthology.orchestra.run.gibba002.stdout: "source_version": "0'0", 2021-02-10T11:37:24.697 INFO:teuthology.orchestra.run.gibba002.stdout: "target_version": "0'0" 2021-02-10T11:37:24.698 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:37:24.698 INFO:teuthology.orchestra.run.gibba002.stdout: "last_change": "161", 2021-02-10T11:37:24.698 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend": "0", 2021-02-10T11:37:24.698 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:37:24.698 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:37:24.698 INFO:teuthology.orchestra.run.gibba002.stdout: "auid": 0, 2021-02-10T11:37:24.698 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:37:24.698 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_seq": 0, 2021-02-10T11:37:24.698 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_epoch": 0, 2021-02-10T11:37:24.698 INFO:teuthology.orchestra.run.gibba002.stdout: "pool_snaps": [], 2021-02-10T11:37:24.698 INFO:teuthology.orchestra.run.gibba002.stdout: "removed_snaps": "[]", 2021-02-10T11:37:24.698 INFO:teuthology.orchestra.run.gibba002.stdout: "quota_max_bytes": 0, 2021-02-10T11:37:24.698 INFO:teuthology.orchestra.run.gibba002.stdout: "quota_max_objects": 0, 2021-02-10T11:37:24.699 INFO:teuthology.orchestra.run.gibba002.stdout: "tiers": [], 2021-02-10T11:37:24.699 INFO:teuthology.orchestra.run.gibba002.stdout: "tier_of": -1, 2021-02-10T11:37:24.699 INFO:teuthology.orchestra.run.gibba002.stdout: "read_tier": -1, 2021-02-10T11:37:24.699 INFO:teuthology.orchestra.run.gibba002.stdout: "write_tier": -1, 2021-02-10T11:37:24.699 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_mode": "none", 2021-02-10T11:37:24.699 INFO:teuthology.orchestra.run.gibba002.stdout: "target_max_bytes": 0, 2021-02-10T11:37:24.699 INFO:teuthology.orchestra.run.gibba002.stdout: "target_max_objects": 0, 2021-02-10T11:37:24.699 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:37:24.699 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:37:24.699 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:37:24.699 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_min_flush_age": 0, 2021-02-10T11:37:24.699 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_min_evict_age": 0, 2021-02-10T11:37:24.699 INFO:teuthology.orchestra.run.gibba002.stdout: "erasure_code_profile": "", 2021-02-10T11:37:24.700 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_params": { 2021-02-10T11:37:24.700 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "none" 2021-02-10T11:37:24.700 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:37:24.700 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_period": 0, 2021-02-10T11:37:24.700 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_count": 0, 2021-02-10T11:37:24.700 INFO:teuthology.orchestra.run.gibba002.stdout: "use_gmt_hitset": true, 2021-02-10T11:37:24.700 INFO:teuthology.orchestra.run.gibba002.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:37:24.700 INFO:teuthology.orchestra.run.gibba002.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:37:24.700 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:37:24.700 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:37:24.700 INFO:teuthology.orchestra.run.gibba002.stdout: "grade_table": [], 2021-02-10T11:37:24.700 INFO:teuthology.orchestra.run.gibba002.stdout: "stripe_width": 0, 2021-02-10T11:37:24.700 INFO:teuthology.orchestra.run.gibba002.stdout: "expected_num_objects": 0, 2021-02-10T11:37:24.700 INFO:teuthology.orchestra.run.gibba002.stdout: "fast_read": false, 2021-02-10T11:37:24.701 INFO:teuthology.orchestra.run.gibba002.stdout: "options": { 2021-02-10T11:37:24.701 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_autoscale_bias": 4, 2021-02-10T11:37:24.701 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num_min": 16, 2021-02-10T11:37:24.701 INFO:teuthology.orchestra.run.gibba002.stdout: "recovery_priority": 5 2021-02-10T11:37:24.701 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:37:24.701 INFO:teuthology.orchestra.run.gibba002.stdout: "application_metadata": { 2021-02-10T11:37:24.701 INFO:teuthology.orchestra.run.gibba002.stdout: "cephfs": { 2021-02-10T11:37:24.701 INFO:teuthology.orchestra.run.gibba002.stdout: "metadata": "cephfs" 2021-02-10T11:37:24.701 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:37:24.701 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:37:24.701 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:37:24.701 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:37:24.701 INFO:teuthology.orchestra.run.gibba002.stdout: "pool": 35, 2021-02-10T11:37:24.702 INFO:teuthology.orchestra.run.gibba002.stdout: "pool_name": "cephfs_data", 2021-02-10T11:37:24.702 INFO:teuthology.orchestra.run.gibba002.stdout: "create_time": "2021-02-10 11:36:45.321943", 2021-02-10T11:37:24.702 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 8193, 2021-02-10T11:37:24.702 INFO:teuthology.orchestra.run.gibba002.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2021-02-10T11:37:24.702 INFO:teuthology.orchestra.run.gibba002.stdout: "type": 1, 2021-02-10T11:37:24.702 INFO:teuthology.orchestra.run.gibba002.stdout: "size": 2, 2021-02-10T11:37:24.702 INFO:teuthology.orchestra.run.gibba002.stdout: "min_size": 1, 2021-02-10T11:37:24.702 INFO:teuthology.orchestra.run.gibba002.stdout: "crush_rule": 0, 2021-02-10T11:37:24.702 INFO:teuthology.orchestra.run.gibba002.stdout: "object_hash": 2, 2021-02-10T11:37:24.702 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:37:24.702 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num": 8, 2021-02-10T11:37:24.702 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_placement_num": 8, 2021-02-10T11:37:24.702 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_placement_num_target": 8, 2021-02-10T11:37:24.703 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num_target": 8, 2021-02-10T11:37:24.703 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num_pending": 8, 2021-02-10T11:37:24.703 INFO:teuthology.orchestra.run.gibba002.stdout: "last_pg_merge_meta": { 2021-02-10T11:37:24.703 INFO:teuthology.orchestra.run.gibba002.stdout: "source_pgid": "0.0", 2021-02-10T11:37:24.703 INFO:teuthology.orchestra.run.gibba002.stdout: "ready_epoch": 0, 2021-02-10T11:37:24.703 INFO:teuthology.orchestra.run.gibba002.stdout: "last_epoch_started": 0, 2021-02-10T11:37:24.703 INFO:teuthology.orchestra.run.gibba002.stdout: "last_epoch_clean": 0, 2021-02-10T11:37:24.703 INFO:teuthology.orchestra.run.gibba002.stdout: "source_version": "0'0", 2021-02-10T11:37:24.703 INFO:teuthology.orchestra.run.gibba002.stdout: "target_version": "0'0" 2021-02-10T11:37:24.703 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:37:24.703 INFO:teuthology.orchestra.run.gibba002.stdout: "last_change": "163", 2021-02-10T11:37:24.703 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend": "0", 2021-02-10T11:37:24.703 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:37:24.704 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:37:24.704 INFO:teuthology.orchestra.run.gibba002.stdout: "auid": 0, 2021-02-10T11:37:24.704 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:37:24.704 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_seq": 3, 2021-02-10T11:37:24.704 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_epoch": 163, 2021-02-10T11:37:24.704 INFO:teuthology.orchestra.run.gibba002.stdout: "pool_snaps": [], 2021-02-10T11:37:24.704 INFO:teuthology.orchestra.run.gibba002.stdout: "removed_snaps": "[2~2]", 2021-02-10T11:37:24.704 INFO:teuthology.orchestra.run.gibba002.stdout: "quota_max_bytes": 0, 2021-02-10T11:37:24.704 INFO:teuthology.orchestra.run.gibba002.stdout: "quota_max_objects": 0, 2021-02-10T11:37:24.704 INFO:teuthology.orchestra.run.gibba002.stdout: "tiers": [], 2021-02-10T11:37:24.704 INFO:teuthology.orchestra.run.gibba002.stdout: "tier_of": -1, 2021-02-10T11:37:24.704 INFO:teuthology.orchestra.run.gibba002.stdout: "read_tier": -1, 2021-02-10T11:37:24.704 INFO:teuthology.orchestra.run.gibba002.stdout: "write_tier": -1, 2021-02-10T11:37:24.705 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_mode": "none", 2021-02-10T11:37:24.705 INFO:teuthology.orchestra.run.gibba002.stdout: "target_max_bytes": 0, 2021-02-10T11:37:24.705 INFO:teuthology.orchestra.run.gibba002.stdout: "target_max_objects": 0, 2021-02-10T11:37:24.705 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:37:24.705 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:37:24.705 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:37:24.705 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_min_flush_age": 0, 2021-02-10T11:37:24.705 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_min_evict_age": 0, 2021-02-10T11:37:24.705 INFO:teuthology.orchestra.run.gibba002.stdout: "erasure_code_profile": "", 2021-02-10T11:37:24.705 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_params": { 2021-02-10T11:37:24.705 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "none" 2021-02-10T11:37:24.705 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:37:24.705 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_period": 0, 2021-02-10T11:37:24.705 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_count": 0, 2021-02-10T11:37:24.706 INFO:teuthology.orchestra.run.gibba002.stdout: "use_gmt_hitset": true, 2021-02-10T11:37:24.706 INFO:teuthology.orchestra.run.gibba002.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:37:24.706 INFO:teuthology.orchestra.run.gibba002.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:37:24.706 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:37:24.706 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:37:24.706 INFO:teuthology.orchestra.run.gibba002.stdout: "grade_table": [], 2021-02-10T11:37:24.706 INFO:teuthology.orchestra.run.gibba002.stdout: "stripe_width": 0, 2021-02-10T11:37:24.706 INFO:teuthology.orchestra.run.gibba002.stdout: "expected_num_objects": 0, 2021-02-10T11:37:24.706 INFO:teuthology.orchestra.run.gibba002.stdout: "fast_read": false, 2021-02-10T11:37:24.706 INFO:teuthology.orchestra.run.gibba002.stdout: "options": {}, 2021-02-10T11:37:24.706 INFO:teuthology.orchestra.run.gibba002.stdout: "application_metadata": { 2021-02-10T11:37:24.706 INFO:teuthology.orchestra.run.gibba002.stdout: "cephfs": { 2021-02-10T11:37:24.706 INFO:teuthology.orchestra.run.gibba002.stdout: "data": "cephfs" 2021-02-10T11:37:24.707 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:37:24.707 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:37:24.707 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:37:24.707 INFO:teuthology.orchestra.run.gibba002.stdout: ], 2021-02-10T11:37:24.707 INFO:teuthology.orchestra.run.gibba002.stdout: "osds": [ 2021-02-10T11:37:24.707 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:37:24.707 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 0, 2021-02-10T11:37:24.707 INFO:teuthology.orchestra.run.gibba002.stdout: "uuid": "be691b4d-a039-408a-af80-9d53f02b6f1f", 2021-02-10T11:37:24.707 INFO:teuthology.orchestra.run.gibba002.stdout: "up": 1, 2021-02-10T11:37:24.707 INFO:teuthology.orchestra.run.gibba002.stdout: "in": 1, 2021-02-10T11:37:24.707 INFO:teuthology.orchestra.run.gibba002.stdout: "weight": 1, 2021-02-10T11:37:24.707 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_affinity": 1, 2021-02-10T11:37:24.707 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_begin": 0, 2021-02-10T11:37:24.708 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_end": 0, 2021-02-10T11:37:24.708 INFO:teuthology.orchestra.run.gibba002.stdout: "up_from": 13, 2021-02-10T11:37:24.708 INFO:teuthology.orchestra.run.gibba002.stdout: "up_thru": 159, 2021-02-10T11:37:24.708 INFO:teuthology.orchestra.run.gibba002.stdout: "down_at": 0, 2021-02-10T11:37:24.708 INFO:teuthology.orchestra.run.gibba002.stdout: "lost_at": 0, 2021-02-10T11:37:24.708 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addrs": { 2021-02-10T11:37:24.708 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:37:24.708 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:37:24.708 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:37:24.708 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6818", 2021-02-10T11:37:24.708 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36088 2021-02-10T11:37:24.708 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:37:24.708 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:37:24.708 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:37:24.709 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6820", 2021-02-10T11:37:24.709 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36088 2021-02-10T11:37:24.709 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:37:24.709 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:37:24.709 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:37:24.709 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addrs": { 2021-02-10T11:37:24.709 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:37:24.709 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:37:24.709 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:37:24.709 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6822", 2021-02-10T11:37:24.709 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36088 2021-02-10T11:37:24.709 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:37:24.709 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:37:24.709 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:37:24.710 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6824", 2021-02-10T11:37:24.710 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36088 2021-02-10T11:37:24.710 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:37:24.710 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:37:24.710 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:37:24.710 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addrs": { 2021-02-10T11:37:24.710 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:37:24.710 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:37:24.710 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:37:24.710 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6830", 2021-02-10T11:37:24.710 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36088 2021-02-10T11:37:24.710 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:37:24.711 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:37:24.711 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:37:24.711 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6832", 2021-02-10T11:37:24.711 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36088 2021-02-10T11:37:24.711 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:37:24.711 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:37:24.711 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:37:24.711 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addrs": { 2021-02-10T11:37:24.711 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:37:24.711 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:37:24.711 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:37:24.711 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6826", 2021-02-10T11:37:24.711 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36088 2021-02-10T11:37:24.711 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:37:24.712 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:37:24.712 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:37:24.712 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6828", 2021-02-10T11:37:24.712 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36088 2021-02-10T11:37:24.712 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:37:24.712 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:37:24.712 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:37:24.712 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addr": "172.21.2.102:6820/36088", 2021-02-10T11:37:24.712 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addr": "172.21.2.102:6824/36088", 2021-02-10T11:37:24.712 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addr": "172.21.2.102:6832/36088", 2021-02-10T11:37:24.712 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addr": "172.21.2.102:6828/36088", 2021-02-10T11:37:24.712 INFO:teuthology.orchestra.run.gibba002.stdout: "state": [ 2021-02-10T11:37:24.713 INFO:teuthology.orchestra.run.gibba002.stdout: "exists", 2021-02-10T11:37:24.713 INFO:teuthology.orchestra.run.gibba002.stdout: "up" 2021-02-10T11:37:24.713 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:37:24.713 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:37:24.713 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:37:24.713 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 1, 2021-02-10T11:37:24.713 INFO:teuthology.orchestra.run.gibba002.stdout: "uuid": "64e3ccc8-9c69-4d7c-865e-af8e83822635", 2021-02-10T11:37:24.713 INFO:teuthology.orchestra.run.gibba002.stdout: "up": 1, 2021-02-10T11:37:24.713 INFO:teuthology.orchestra.run.gibba002.stdout: "in": 1, 2021-02-10T11:37:24.713 INFO:teuthology.orchestra.run.gibba002.stdout: "weight": 1, 2021-02-10T11:37:24.713 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_affinity": 1, 2021-02-10T11:37:24.713 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_begin": 0, 2021-02-10T11:37:24.713 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_end": 0, 2021-02-10T11:37:24.713 INFO:teuthology.orchestra.run.gibba002.stdout: "up_from": 13, 2021-02-10T11:37:24.714 INFO:teuthology.orchestra.run.gibba002.stdout: "up_thru": 143, 2021-02-10T11:37:24.714 INFO:teuthology.orchestra.run.gibba002.stdout: "down_at": 0, 2021-02-10T11:37:24.714 INFO:teuthology.orchestra.run.gibba002.stdout: "lost_at": 0, 2021-02-10T11:37:24.714 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addrs": { 2021-02-10T11:37:24.714 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:37:24.714 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:37:24.714 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:37:24.714 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6819", 2021-02-10T11:37:24.714 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36087 2021-02-10T11:37:24.714 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:37:24.714 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:37:24.714 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:37:24.715 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6821", 2021-02-10T11:37:24.715 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36087 2021-02-10T11:37:24.715 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:37:24.715 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:37:24.715 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:37:24.715 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addrs": { 2021-02-10T11:37:24.715 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:37:24.715 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:37:24.715 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:37:24.715 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6823", 2021-02-10T11:37:24.715 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36087 2021-02-10T11:37:24.715 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:37:24.715 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:37:24.715 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:37:24.716 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6825", 2021-02-10T11:37:24.716 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36087 2021-02-10T11:37:24.716 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:37:24.716 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:37:24.716 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:37:24.716 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addrs": { 2021-02-10T11:37:24.716 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:37:24.716 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:37:24.716 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:37:24.716 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6831", 2021-02-10T11:37:24.716 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36087 2021-02-10T11:37:24.716 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:37:24.716 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:37:24.717 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:37:24.717 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6833", 2021-02-10T11:37:24.717 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36087 2021-02-10T11:37:24.717 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:37:24.717 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:37:24.717 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:37:24.717 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addrs": { 2021-02-10T11:37:24.717 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:37:24.717 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:37:24.717 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:37:24.717 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6827", 2021-02-10T11:37:24.717 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36087 2021-02-10T11:37:24.717 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:37:24.717 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:37:24.718 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:37:24.718 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6829", 2021-02-10T11:37:24.718 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36087 2021-02-10T11:37:24.718 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:37:24.718 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:37:24.718 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:37:24.718 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addr": "172.21.2.102:6821/36087", 2021-02-10T11:37:24.718 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addr": "172.21.2.102:6825/36087", 2021-02-10T11:37:24.718 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addr": "172.21.2.102:6833/36087", 2021-02-10T11:37:24.718 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addr": "172.21.2.102:6829/36087", 2021-02-10T11:37:24.718 INFO:teuthology.orchestra.run.gibba002.stdout: "state": [ 2021-02-10T11:37:24.718 INFO:teuthology.orchestra.run.gibba002.stdout: "exists", 2021-02-10T11:37:24.718 INFO:teuthology.orchestra.run.gibba002.stdout: "up" 2021-02-10T11:37:24.718 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:37:24.719 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:37:24.719 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:37:24.719 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 2, 2021-02-10T11:37:24.719 INFO:teuthology.orchestra.run.gibba002.stdout: "uuid": "153700af-747f-481f-bdf4-d6072d3a842f", 2021-02-10T11:37:24.719 INFO:teuthology.orchestra.run.gibba002.stdout: "up": 1, 2021-02-10T11:37:24.719 INFO:teuthology.orchestra.run.gibba002.stdout: "in": 1, 2021-02-10T11:37:24.719 INFO:teuthology.orchestra.run.gibba002.stdout: "weight": 1, 2021-02-10T11:37:24.719 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_affinity": 1, 2021-02-10T11:37:24.719 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_begin": 0, 2021-02-10T11:37:24.719 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_end": 0, 2021-02-10T11:37:24.719 INFO:teuthology.orchestra.run.gibba002.stdout: "up_from": 13, 2021-02-10T11:37:24.719 INFO:teuthology.orchestra.run.gibba002.stdout: "up_thru": 160, 2021-02-10T11:37:24.719 INFO:teuthology.orchestra.run.gibba002.stdout: "down_at": 0, 2021-02-10T11:37:24.720 INFO:teuthology.orchestra.run.gibba002.stdout: "lost_at": 0, 2021-02-10T11:37:24.720 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addrs": { 2021-02-10T11:37:24.720 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:37:24.720 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:37:24.720 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:37:24.720 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6810", 2021-02-10T11:37:24.720 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36089 2021-02-10T11:37:24.720 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:37:24.720 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:37:24.720 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:37:24.720 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6811", 2021-02-10T11:37:24.720 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36089 2021-02-10T11:37:24.720 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:37:24.721 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:37:24.721 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:37:24.721 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addrs": { 2021-02-10T11:37:24.721 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:37:24.721 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:37:24.721 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:37:24.721 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6812", 2021-02-10T11:37:24.721 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36089 2021-02-10T11:37:24.721 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:37:24.721 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:37:24.721 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:37:24.721 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6813", 2021-02-10T11:37:24.721 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36089 2021-02-10T11:37:24.722 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:37:24.722 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:37:24.722 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:37:24.722 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addrs": { 2021-02-10T11:37:24.722 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:37:24.722 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:37:24.722 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:37:24.722 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6816", 2021-02-10T11:37:24.722 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36089 2021-02-10T11:37:24.722 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:37:24.722 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:37:24.722 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:37:24.722 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6817", 2021-02-10T11:37:24.723 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36089 2021-02-10T11:37:24.723 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:37:24.723 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:37:24.723 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:37:24.723 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addrs": { 2021-02-10T11:37:24.723 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:37:24.723 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:37:24.723 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:37:24.723 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6814", 2021-02-10T11:37:24.723 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36089 2021-02-10T11:37:24.723 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:37:24.723 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:37:24.723 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:37:24.724 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6815", 2021-02-10T11:37:24.724 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36089 2021-02-10T11:37:24.724 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:37:24.724 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:37:24.724 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:37:24.724 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addr": "172.21.2.102:6811/36089", 2021-02-10T11:37:24.724 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addr": "172.21.2.102:6813/36089", 2021-02-10T11:37:24.724 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addr": "172.21.2.102:6817/36089", 2021-02-10T11:37:24.724 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addr": "172.21.2.102:6815/36089", 2021-02-10T11:37:24.724 INFO:teuthology.orchestra.run.gibba002.stdout: "state": [ 2021-02-10T11:37:24.724 INFO:teuthology.orchestra.run.gibba002.stdout: "exists", 2021-02-10T11:37:24.724 INFO:teuthology.orchestra.run.gibba002.stdout: "up" 2021-02-10T11:37:24.724 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:37:24.725 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:37:24.725 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:37:24.725 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 3, 2021-02-10T11:37:24.725 INFO:teuthology.orchestra.run.gibba002.stdout: "uuid": "cbc678f4-83a9-45e6-8830-2f7c232b1ea7", 2021-02-10T11:37:24.725 INFO:teuthology.orchestra.run.gibba002.stdout: "up": 1, 2021-02-10T11:37:24.725 INFO:teuthology.orchestra.run.gibba002.stdout: "in": 1, 2021-02-10T11:37:24.725 INFO:teuthology.orchestra.run.gibba002.stdout: "weight": 1, 2021-02-10T11:37:24.725 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_affinity": 1, 2021-02-10T11:37:24.725 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_begin": 0, 2021-02-10T11:37:24.725 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_end": 0, 2021-02-10T11:37:24.725 INFO:teuthology.orchestra.run.gibba002.stdout: "up_from": 13, 2021-02-10T11:37:24.725 INFO:teuthology.orchestra.run.gibba002.stdout: "up_thru": 160, 2021-02-10T11:37:24.725 INFO:teuthology.orchestra.run.gibba002.stdout: "down_at": 0, 2021-02-10T11:37:24.725 INFO:teuthology.orchestra.run.gibba002.stdout: "lost_at": 0, 2021-02-10T11:37:24.726 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addrs": { 2021-02-10T11:37:24.726 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:37:24.726 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:37:24.726 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:37:24.726 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6802", 2021-02-10T11:37:24.726 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36090 2021-02-10T11:37:24.726 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:37:24.726 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:37:24.726 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:37:24.726 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6803", 2021-02-10T11:37:24.726 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36090 2021-02-10T11:37:24.726 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:37:24.726 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:37:24.727 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:37:24.727 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addrs": { 2021-02-10T11:37:24.727 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:37:24.727 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:37:24.727 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:37:24.727 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6804", 2021-02-10T11:37:24.727 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36090 2021-02-10T11:37:24.727 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:37:24.727 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:37:24.727 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:37:24.727 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6805", 2021-02-10T11:37:24.727 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36090 2021-02-10T11:37:24.727 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:37:24.728 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:37:24.728 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:37:24.728 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addrs": { 2021-02-10T11:37:24.728 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:37:24.728 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:37:24.728 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:37:24.728 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6808", 2021-02-10T11:37:24.728 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36090 2021-02-10T11:37:24.728 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:37:24.728 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:37:24.728 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:37:24.728 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6809", 2021-02-10T11:37:24.728 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36090 2021-02-10T11:37:24.728 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:37:24.729 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:37:24.729 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:37:24.729 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addrs": { 2021-02-10T11:37:24.729 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:37:24.729 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:37:24.729 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:37:24.729 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6806", 2021-02-10T11:37:24.729 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36090 2021-02-10T11:37:24.729 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:37:24.729 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:37:24.729 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:37:24.729 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6807", 2021-02-10T11:37:24.729 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36090 2021-02-10T11:37:24.730 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:37:24.730 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:37:24.730 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:37:24.730 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addr": "172.21.2.102:6803/36090", 2021-02-10T11:37:24.730 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addr": "172.21.2.102:6805/36090", 2021-02-10T11:37:24.730 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addr": "172.21.2.102:6809/36090", 2021-02-10T11:37:24.730 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addr": "172.21.2.102:6807/36090", 2021-02-10T11:37:24.730 INFO:teuthology.orchestra.run.gibba002.stdout: "state": [ 2021-02-10T11:37:24.730 INFO:teuthology.orchestra.run.gibba002.stdout: "exists", 2021-02-10T11:37:24.730 INFO:teuthology.orchestra.run.gibba002.stdout: "up" 2021-02-10T11:37:24.730 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:37:24.730 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:37:24.730 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:37:24.731 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 4, 2021-02-10T11:37:24.731 INFO:teuthology.orchestra.run.gibba002.stdout: "uuid": "c01e4008-b258-4f10-afd2-9e7f37214351", 2021-02-10T11:37:24.731 INFO:teuthology.orchestra.run.gibba002.stdout: "up": 1, 2021-02-10T11:37:24.731 INFO:teuthology.orchestra.run.gibba002.stdout: "in": 1, 2021-02-10T11:37:24.731 INFO:teuthology.orchestra.run.gibba002.stdout: "weight": 1, 2021-02-10T11:37:24.731 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_affinity": 1, 2021-02-10T11:37:24.731 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_begin": 0, 2021-02-10T11:37:24.731 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_end": 0, 2021-02-10T11:37:24.731 INFO:teuthology.orchestra.run.gibba002.stdout: "up_from": 13, 2021-02-10T11:37:24.731 INFO:teuthology.orchestra.run.gibba002.stdout: "up_thru": 160, 2021-02-10T11:37:24.731 INFO:teuthology.orchestra.run.gibba002.stdout: "down_at": 0, 2021-02-10T11:37:24.731 INFO:teuthology.orchestra.run.gibba002.stdout: "lost_at": 0, 2021-02-10T11:37:24.731 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addrs": { 2021-02-10T11:37:24.731 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:37:24.732 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:37:24.732 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:37:24.732 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6808", 2021-02-10T11:37:24.732 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36368 2021-02-10T11:37:24.732 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:37:24.732 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:37:24.732 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:37:24.732 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6811", 2021-02-10T11:37:24.732 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36368 2021-02-10T11:37:24.732 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:37:24.732 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:37:24.732 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:37:24.732 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addrs": { 2021-02-10T11:37:24.732 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:37:24.733 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:37:24.733 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:37:24.733 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6814", 2021-02-10T11:37:24.733 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36368 2021-02-10T11:37:24.733 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:37:24.733 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:37:24.733 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:37:24.733 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6818", 2021-02-10T11:37:24.733 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36368 2021-02-10T11:37:24.733 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:37:24.733 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:37:24.733 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:37:24.733 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addrs": { 2021-02-10T11:37:24.733 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:37:24.734 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:37:24.734 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:37:24.734 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6826", 2021-02-10T11:37:24.734 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36368 2021-02-10T11:37:24.734 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:37:24.734 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:37:24.734 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:37:24.734 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6828", 2021-02-10T11:37:24.734 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36368 2021-02-10T11:37:24.734 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:37:24.734 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:37:24.734 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:37:24.735 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addrs": { 2021-02-10T11:37:24.735 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:37:24.735 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:37:24.735 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:37:24.735 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6820", 2021-02-10T11:37:24.735 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36368 2021-02-10T11:37:24.735 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:37:24.735 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:37:24.735 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:37:24.735 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6824", 2021-02-10T11:37:24.735 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36368 2021-02-10T11:37:24.735 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:37:24.735 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:37:24.735 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:37:24.736 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addr": "172.21.2.113:6811/36368", 2021-02-10T11:37:24.736 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addr": "172.21.2.113:6818/36368", 2021-02-10T11:37:24.736 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addr": "172.21.2.113:6828/36368", 2021-02-10T11:37:24.736 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addr": "172.21.2.113:6824/36368", 2021-02-10T11:37:24.736 INFO:teuthology.orchestra.run.gibba002.stdout: "state": [ 2021-02-10T11:37:24.736 INFO:teuthology.orchestra.run.gibba002.stdout: "exists", 2021-02-10T11:37:24.736 INFO:teuthology.orchestra.run.gibba002.stdout: "up" 2021-02-10T11:37:24.736 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:37:24.736 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:37:24.736 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:37:24.736 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 5, 2021-02-10T11:37:24.736 INFO:teuthology.orchestra.run.gibba002.stdout: "uuid": "551f6007-daf8-4a98-8ee0-c2b16bcccb5e", 2021-02-10T11:37:24.736 INFO:teuthology.orchestra.run.gibba002.stdout: "up": 1, 2021-02-10T11:37:24.737 INFO:teuthology.orchestra.run.gibba002.stdout: "in": 1, 2021-02-10T11:37:24.737 INFO:teuthology.orchestra.run.gibba002.stdout: "weight": 1, 2021-02-10T11:37:24.737 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_affinity": 1, 2021-02-10T11:37:24.737 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_begin": 0, 2021-02-10T11:37:24.737 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_end": 0, 2021-02-10T11:37:24.737 INFO:teuthology.orchestra.run.gibba002.stdout: "up_from": 13, 2021-02-10T11:37:24.737 INFO:teuthology.orchestra.run.gibba002.stdout: "up_thru": 160, 2021-02-10T11:37:24.737 INFO:teuthology.orchestra.run.gibba002.stdout: "down_at": 0, 2021-02-10T11:37:24.737 INFO:teuthology.orchestra.run.gibba002.stdout: "lost_at": 0, 2021-02-10T11:37:24.737 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addrs": { 2021-02-10T11:37:24.737 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:37:24.737 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:37:24.737 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:37:24.737 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6802", 2021-02-10T11:37:24.738 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36366 2021-02-10T11:37:24.738 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:37:24.738 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:37:24.738 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:37:24.738 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6805", 2021-02-10T11:37:24.738 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36366 2021-02-10T11:37:24.738 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:37:24.738 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:37:24.738 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:37:24.738 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addrs": { 2021-02-10T11:37:24.738 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:37:24.738 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:37:24.738 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:37:24.739 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6807", 2021-02-10T11:37:24.739 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36366 2021-02-10T11:37:24.739 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:37:24.739 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:37:24.739 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:37:24.739 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6810", 2021-02-10T11:37:24.739 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36366 2021-02-10T11:37:24.739 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:37:24.739 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:37:24.739 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:37:24.739 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addrs": { 2021-02-10T11:37:24.739 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:37:24.739 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:37:24.739 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:37:24.740 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6819", 2021-02-10T11:37:24.740 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36366 2021-02-10T11:37:24.740 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:37:24.740 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:37:24.740 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:37:24.740 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6822", 2021-02-10T11:37:24.740 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36366 2021-02-10T11:37:24.740 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:37:24.740 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:37:24.740 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:37:24.740 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addrs": { 2021-02-10T11:37:24.740 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:37:24.740 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:37:24.740 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:37:24.741 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6812", 2021-02-10T11:37:24.741 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36366 2021-02-10T11:37:24.741 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:37:24.741 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:37:24.741 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:37:24.741 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6815", 2021-02-10T11:37:24.741 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36366 2021-02-10T11:37:24.741 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:37:24.741 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:37:24.741 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:37:24.741 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addr": "172.21.2.113:6805/36366", 2021-02-10T11:37:24.741 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addr": "172.21.2.113:6810/36366", 2021-02-10T11:37:24.741 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addr": "172.21.2.113:6822/36366", 2021-02-10T11:37:24.742 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addr": "172.21.2.113:6815/36366", 2021-02-10T11:37:24.742 INFO:teuthology.orchestra.run.gibba002.stdout: "state": [ 2021-02-10T11:37:24.742 INFO:teuthology.orchestra.run.gibba002.stdout: "exists", 2021-02-10T11:37:24.742 INFO:teuthology.orchestra.run.gibba002.stdout: "up" 2021-02-10T11:37:24.742 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:37:24.742 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:37:24.742 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:37:24.742 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 6, 2021-02-10T11:37:24.742 INFO:teuthology.orchestra.run.gibba002.stdout: "uuid": "bf67c040-0e50-48be-b102-c3943bc9ebef", 2021-02-10T11:37:24.742 INFO:teuthology.orchestra.run.gibba002.stdout: "up": 1, 2021-02-10T11:37:24.742 INFO:teuthology.orchestra.run.gibba002.stdout: "in": 1, 2021-02-10T11:37:24.742 INFO:teuthology.orchestra.run.gibba002.stdout: "weight": 1, 2021-02-10T11:37:24.742 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_affinity": 1, 2021-02-10T11:37:24.743 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_begin": 0, 2021-02-10T11:37:24.743 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_end": 0, 2021-02-10T11:37:24.743 INFO:teuthology.orchestra.run.gibba002.stdout: "up_from": 13, 2021-02-10T11:37:24.743 INFO:teuthology.orchestra.run.gibba002.stdout: "up_thru": 160, 2021-02-10T11:37:24.743 INFO:teuthology.orchestra.run.gibba002.stdout: "down_at": 0, 2021-02-10T11:37:24.743 INFO:teuthology.orchestra.run.gibba002.stdout: "lost_at": 0, 2021-02-10T11:37:24.743 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addrs": { 2021-02-10T11:37:24.743 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:37:24.743 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:37:24.743 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:37:24.743 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6800", 2021-02-10T11:37:24.743 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36367 2021-02-10T11:37:24.743 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:37:24.743 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:37:24.744 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:37:24.744 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6801", 2021-02-10T11:37:24.744 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36367 2021-02-10T11:37:24.744 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:37:24.744 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:37:24.744 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:37:24.744 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addrs": { 2021-02-10T11:37:24.744 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:37:24.744 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:37:24.744 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:37:24.744 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6803", 2021-02-10T11:37:24.744 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36367 2021-02-10T11:37:24.744 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:37:24.745 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:37:24.745 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:37:24.745 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6804", 2021-02-10T11:37:24.745 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36367 2021-02-10T11:37:24.745 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:37:24.745 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:37:24.745 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:37:24.745 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addrs": { 2021-02-10T11:37:24.745 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:37:24.745 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:37:24.745 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:37:24.745 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6813", 2021-02-10T11:37:24.745 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36367 2021-02-10T11:37:24.745 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:37:24.746 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:37:24.746 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:37:24.746 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6816", 2021-02-10T11:37:24.746 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36367 2021-02-10T11:37:24.746 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:37:24.746 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:37:24.746 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:37:24.746 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addrs": { 2021-02-10T11:37:24.746 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:37:24.746 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:37:24.746 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:37:24.746 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6806", 2021-02-10T11:37:24.746 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36367 2021-02-10T11:37:24.747 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:37:24.747 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:37:24.747 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:37:24.747 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6809", 2021-02-10T11:37:24.747 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36367 2021-02-10T11:37:24.747 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:37:24.747 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:37:24.747 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:37:24.747 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addr": "172.21.2.113:6801/36367", 2021-02-10T11:37:24.747 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addr": "172.21.2.113:6804/36367", 2021-02-10T11:37:24.747 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addr": "172.21.2.113:6816/36367", 2021-02-10T11:37:24.747 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addr": "172.21.2.113:6809/36367", 2021-02-10T11:37:24.747 INFO:teuthology.orchestra.run.gibba002.stdout: "state": [ 2021-02-10T11:37:24.748 INFO:teuthology.orchestra.run.gibba002.stdout: "exists", 2021-02-10T11:37:24.748 INFO:teuthology.orchestra.run.gibba002.stdout: "up" 2021-02-10T11:37:24.748 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:37:24.748 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:37:24.748 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:37:24.748 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 7, 2021-02-10T11:37:24.748 INFO:teuthology.orchestra.run.gibba002.stdout: "uuid": "f5461f80-018d-428f-be89-2eb2bcbe7bb6", 2021-02-10T11:37:24.748 INFO:teuthology.orchestra.run.gibba002.stdout: "up": 1, 2021-02-10T11:37:24.748 INFO:teuthology.orchestra.run.gibba002.stdout: "in": 1, 2021-02-10T11:37:24.748 INFO:teuthology.orchestra.run.gibba002.stdout: "weight": 1, 2021-02-10T11:37:24.748 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_affinity": 1, 2021-02-10T11:37:24.748 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_begin": 0, 2021-02-10T11:37:24.748 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_end": 0, 2021-02-10T11:37:24.748 INFO:teuthology.orchestra.run.gibba002.stdout: "up_from": 13, 2021-02-10T11:37:24.749 INFO:teuthology.orchestra.run.gibba002.stdout: "up_thru": 159, 2021-02-10T11:37:24.749 INFO:teuthology.orchestra.run.gibba002.stdout: "down_at": 0, 2021-02-10T11:37:24.749 INFO:teuthology.orchestra.run.gibba002.stdout: "lost_at": 0, 2021-02-10T11:37:24.749 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addrs": { 2021-02-10T11:37:24.749 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:37:24.749 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:37:24.749 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:37:24.749 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6817", 2021-02-10T11:37:24.749 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36365 2021-02-10T11:37:24.749 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:37:24.749 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:37:24.749 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:37:24.749 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6821", 2021-02-10T11:37:24.750 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36365 2021-02-10T11:37:24.750 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:37:24.750 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:37:24.750 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:37:24.750 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addrs": { 2021-02-10T11:37:24.750 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:37:24.750 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:37:24.750 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:37:24.750 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6823", 2021-02-10T11:37:24.750 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36365 2021-02-10T11:37:24.750 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:37:24.750 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:37:24.750 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:37:24.751 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6825", 2021-02-10T11:37:24.751 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36365 2021-02-10T11:37:24.751 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:37:24.751 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:37:24.751 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:37:24.751 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addrs": { 2021-02-10T11:37:24.751 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:37:24.751 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:37:24.751 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:37:24.751 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6830", 2021-02-10T11:37:24.751 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36365 2021-02-10T11:37:24.751 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:37:24.751 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:37:24.751 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:37:24.752 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6831", 2021-02-10T11:37:24.752 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36365 2021-02-10T11:37:24.752 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:37:24.752 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:37:24.752 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:37:24.752 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addrs": { 2021-02-10T11:37:24.752 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:37:24.752 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:37:24.752 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:37:24.752 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6827", 2021-02-10T11:37:24.752 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36365 2021-02-10T11:37:24.752 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:37:24.752 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:37:24.753 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:37:24.753 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6829", 2021-02-10T11:37:24.753 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36365 2021-02-10T11:37:24.753 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:37:24.753 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:37:24.753 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:37:24.753 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addr": "172.21.2.113:6821/36365", 2021-02-10T11:37:24.753 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addr": "172.21.2.113:6825/36365", 2021-02-10T11:37:24.753 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addr": "172.21.2.113:6831/36365", 2021-02-10T11:37:24.753 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addr": "172.21.2.113:6829/36365", 2021-02-10T11:37:24.753 INFO:teuthology.orchestra.run.gibba002.stdout: "state": [ 2021-02-10T11:37:24.753 INFO:teuthology.orchestra.run.gibba002.stdout: "exists", 2021-02-10T11:37:24.753 INFO:teuthology.orchestra.run.gibba002.stdout: "up" 2021-02-10T11:37:24.753 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:37:24.754 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:37:24.754 INFO:teuthology.orchestra.run.gibba002.stdout: ], 2021-02-10T11:37:24.754 INFO:teuthology.orchestra.run.gibba002.stdout: "osd_xinfo": [ 2021-02-10T11:37:24.754 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:37:24.754 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 0, 2021-02-10T11:37:24.754 INFO:teuthology.orchestra.run.gibba002.stdout: "down_stamp": "0.000000", 2021-02-10T11:37:24.754 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_probability": 0, 2021-02-10T11:37:24.754 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_interval": 0, 2021-02-10T11:37:24.754 INFO:teuthology.orchestra.run.gibba002.stdout: "features": 4611087854035861503, 2021-02-10T11:37:24.754 INFO:teuthology.orchestra.run.gibba002.stdout: "old_weight": 0 2021-02-10T11:37:24.754 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:37:24.754 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:37:24.754 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 1, 2021-02-10T11:37:24.755 INFO:teuthology.orchestra.run.gibba002.stdout: "down_stamp": "0.000000", 2021-02-10T11:37:24.755 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_probability": 0, 2021-02-10T11:37:24.755 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_interval": 0, 2021-02-10T11:37:24.755 INFO:teuthology.orchestra.run.gibba002.stdout: "features": 4611087854035861503, 2021-02-10T11:37:24.755 INFO:teuthology.orchestra.run.gibba002.stdout: "old_weight": 0 2021-02-10T11:37:24.755 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:37:24.755 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:37:24.755 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 2, 2021-02-10T11:37:24.755 INFO:teuthology.orchestra.run.gibba002.stdout: "down_stamp": "0.000000", 2021-02-10T11:37:24.755 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_probability": 0, 2021-02-10T11:37:24.755 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_interval": 0, 2021-02-10T11:37:24.755 INFO:teuthology.orchestra.run.gibba002.stdout: "features": 4611087854035861503, 2021-02-10T11:37:24.755 INFO:teuthology.orchestra.run.gibba002.stdout: "old_weight": 0 2021-02-10T11:37:24.756 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:37:24.756 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:37:24.756 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 3, 2021-02-10T11:37:24.756 INFO:teuthology.orchestra.run.gibba002.stdout: "down_stamp": "0.000000", 2021-02-10T11:37:24.756 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_probability": 0, 2021-02-10T11:37:24.756 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_interval": 0, 2021-02-10T11:37:24.756 INFO:teuthology.orchestra.run.gibba002.stdout: "features": 4611087854035861503, 2021-02-10T11:37:24.756 INFO:teuthology.orchestra.run.gibba002.stdout: "old_weight": 0 2021-02-10T11:37:24.756 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:37:24.756 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:37:24.756 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 4, 2021-02-10T11:37:24.756 INFO:teuthology.orchestra.run.gibba002.stdout: "down_stamp": "0.000000", 2021-02-10T11:37:24.756 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_probability": 0, 2021-02-10T11:37:24.756 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_interval": 0, 2021-02-10T11:37:24.757 INFO:teuthology.orchestra.run.gibba002.stdout: "features": 4611087854035861503, 2021-02-10T11:37:24.757 INFO:teuthology.orchestra.run.gibba002.stdout: "old_weight": 0 2021-02-10T11:37:24.757 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:37:24.757 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:37:24.757 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 5, 2021-02-10T11:37:24.757 INFO:teuthology.orchestra.run.gibba002.stdout: "down_stamp": "0.000000", 2021-02-10T11:37:24.757 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_probability": 0, 2021-02-10T11:37:24.757 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_interval": 0, 2021-02-10T11:37:24.757 INFO:teuthology.orchestra.run.gibba002.stdout: "features": 4611087854035861503, 2021-02-10T11:37:24.757 INFO:teuthology.orchestra.run.gibba002.stdout: "old_weight": 0 2021-02-10T11:37:24.757 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:37:24.757 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:37:24.757 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 6, 2021-02-10T11:37:24.758 INFO:teuthology.orchestra.run.gibba002.stdout: "down_stamp": "0.000000", 2021-02-10T11:37:24.758 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_probability": 0, 2021-02-10T11:37:24.758 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_interval": 0, 2021-02-10T11:37:24.758 INFO:teuthology.orchestra.run.gibba002.stdout: "features": 4611087854035861503, 2021-02-10T11:37:24.758 INFO:teuthology.orchestra.run.gibba002.stdout: "old_weight": 0 2021-02-10T11:37:24.758 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:37:24.758 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:37:24.758 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 7, 2021-02-10T11:37:24.758 INFO:teuthology.orchestra.run.gibba002.stdout: "down_stamp": "0.000000", 2021-02-10T11:37:24.758 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_probability": 0, 2021-02-10T11:37:24.758 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_interval": 0, 2021-02-10T11:37:24.758 INFO:teuthology.orchestra.run.gibba002.stdout: "features": 4611087854035861503, 2021-02-10T11:37:24.758 INFO:teuthology.orchestra.run.gibba002.stdout: "old_weight": 0 2021-02-10T11:37:24.759 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:37:24.759 INFO:teuthology.orchestra.run.gibba002.stdout: ], 2021-02-10T11:37:24.759 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_upmap": [], 2021-02-10T11:37:24.759 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_upmap_items": [], 2021-02-10T11:37:24.759 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_temp": [], 2021-02-10T11:37:24.759 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_temp": [], 2021-02-10T11:37:24.759 INFO:teuthology.orchestra.run.gibba002.stdout: "blacklist": {}, 2021-02-10T11:37:24.759 INFO:teuthology.orchestra.run.gibba002.stdout: "erasure_code_profiles": { 2021-02-10T11:37:24.759 INFO:teuthology.orchestra.run.gibba002.stdout: "default": { 2021-02-10T11:37:24.759 INFO:teuthology.orchestra.run.gibba002.stdout: "crush-failure-domain": "osd", 2021-02-10T11:37:24.759 INFO:teuthology.orchestra.run.gibba002.stdout: "k": "2", 2021-02-10T11:37:24.759 INFO:teuthology.orchestra.run.gibba002.stdout: "m": "1", 2021-02-10T11:37:24.759 INFO:teuthology.orchestra.run.gibba002.stdout: "plugin": "jerasure", 2021-02-10T11:37:24.759 INFO:teuthology.orchestra.run.gibba002.stdout: "ruleset-failure-domain": "osd", 2021-02-10T11:37:24.760 INFO:teuthology.orchestra.run.gibba002.stdout: "technique": "reed_sol_van" 2021-02-10T11:37:24.760 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:37:24.760 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:37:24.760 INFO:teuthology.orchestra.run.gibba002.stdout: "removed_snaps_queue": [], 2021-02-10T11:37:24.760 INFO:teuthology.orchestra.run.gibba002.stdout: "new_removed_snaps": [], 2021-02-10T11:37:24.760 INFO:teuthology.orchestra.run.gibba002.stdout: "new_purged_snaps": [ 2021-02-10T11:37:24.760 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:37:24.760 INFO:teuthology.orchestra.run.gibba002.stdout: "pool": 35, 2021-02-10T11:37:24.760 INFO:teuthology.orchestra.run.gibba002.stdout: "snaps": [ 2021-02-10T11:37:24.760 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:37:24.760 INFO:teuthology.orchestra.run.gibba002.stdout: "begin": 2, 2021-02-10T11:37:24.760 INFO:teuthology.orchestra.run.gibba002.stdout: "length": 2 2021-02-10T11:37:24.760 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:37:24.760 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:37:24.761 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:37:24.761 INFO:teuthology.orchestra.run.gibba002.stdout: ], 2021-02-10T11:37:24.761 INFO:teuthology.orchestra.run.gibba002.stdout: "crush_node_flags": {}, 2021-02-10T11:37:24.761 INFO:teuthology.orchestra.run.gibba002.stdout: "device_class_flags": {} 2021-02-10T11:37:24.761 INFO:teuthology.orchestra.run.gibba002.stdout:} 2021-02-10T11:37:24.762 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:24.687 7fd2ca4cd700 1 -- 172.21.2.102:0/3690019535 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd2a0018730 msgr2=0x7fd2a001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:24.762 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:24.687 7fd2ca4cd700 1 --2- 172.21.2.102:0/3690019535 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd2a0018730 0x7fd2a001ac10 crc :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:24.762 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:24.687 7fd2ca4cd700 1 -- 172.21.2.102:0/3690019535 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd2c40964a0 msgr2=0x7fd2c40968a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:24.763 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:24.687 7fd2ca4cd700 1 --2- 172.21.2.102:0/3690019535 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd2c40964a0 0x7fd2c40968a0 secure :-1 s=READY pgs=2941 cs=0 l=1 rev1=1 rx=0x7fd2b000a5a0 tx=0x7fd2b000a940).stop 2021-02-10T11:37:24.763 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:24.687 7fd2ca4cd700 1 -- 172.21.2.102:0/3690019535 shutdown_connections 2021-02-10T11:37:24.763 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:24.687 7fd2ca4cd700 1 --2- 172.21.2.102:0/3690019535 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd2a0018730 0x7fd2a001ac10 unknown :-1 s=CLOSED pgs=1290 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:24.763 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:24.687 7fd2ca4cd700 1 --2- 172.21.2.102:0/3690019535 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd2c4096ec0 0x7fd2c4084f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:24.763 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:24.687 7fd2ca4cd700 1 --2- 172.21.2.102:0/3690019535 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd2c40964a0 0x7fd2c40968a0 unknown :-1 s=CLOSED pgs=2941 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:24.763 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:24.687 7fd2ca4cd700 1 --2- 172.21.2.102:0/3690019535 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd2c4092e30 0x7fd2c41c6320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:24.763 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:24.687 7fd2ca4cd700 1 -- 172.21.2.102:0/3690019535 >> 172.21.2.102:0/3690019535 conn(0x7fd2c40c3bd0 msgr2=0x7fd2c4089210 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:24.763 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:24.688 7fd2ca4cd700 1 -- 172.21.2.102:0/3690019535 shutdown_connections 2021-02-10T11:37:24.763 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:24.688 7fd2ca4cd700 1 -- 172.21.2.102:0/3690019535 wait complete. 2021-02-10T11:37:24.764 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:37:25.194 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:25.176 7f03b7238700 1 Processor -- start 2021-02-10T11:37:25.195 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:25.176 7f03b7238700 1 -- start start 2021-02-10T11:37:25.195 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:25.176 7f03b7238700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f03b00d1870 0x7f03b01b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:25.195 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:25.176 7f03b7238700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f03b01b0a10 0x7f03b00cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:25.195 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:25.176 7f03b7238700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03b00f0b10 con 0x7f03b00d1c70 2021-02-10T11:37:25.195 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:25.176 7f03b7238700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03b00e5ed0 con 0x7f03b01b0a10 2021-02-10T11:37:25.195 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:25.176 7f03b7238700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03b00f0650 con 0x7f03b00d1870 2021-02-10T11:37:25.195 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:25.176 7f03b4853700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f03b01b0a10 0x7f03b00cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:25.195 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:25.176 7f03b4853700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f03b01b0a10 0x7f03b00cf330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:38604/0 (socket says 172.21.2.102:38604) 2021-02-10T11:37:25.195 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:25.176 7f03b4853700 1 -- 172.21.2.102:0/2074728907 learned_addr learned my addr 172.21.2.102:0/2074728907 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:37:25.195 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:25.177 7f03b4853700 1 -- 172.21.2.102:0/2074728907 >> v2:172.21.2.113:3300/0 conn(0x7f03b00d1870 msgr2=0x7f03b01b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:37:25.195 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:25.177 7f03b4853700 1 --2- 172.21.2.102:0/2074728907 >> v2:172.21.2.113:3300/0 conn(0x7f03b00d1870 0x7f03b01b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:25.196 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:25.177 7f03b4853700 1 -- 172.21.2.102:0/2074728907 >> v1:172.21.2.113:6789/0 conn(0x7f03b00d1c70 legacy=0x7f03b00cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:37:25.196 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:25.177 7f03b4853700 1 -- 172.21.2.102:0/2074728907 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03b00f0840 con 0x7f03b01b0a10 2021-02-10T11:37:25.196 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:25.177 7f03b4853700 1 --2- 172.21.2.102:0/2074728907 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f03b01b0a10 0x7f03b00cf330 secure :-1 s=READY pgs=2942 cs=0 l=1 rev1=1 rx=0x7f03a4006700 tx=0x7f03a4009b80).ready entity=mon.0 client_cookie=c6eded65c144ec61 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:25.196 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:25.177 7f03a3fff700 1 -- 172.21.2.102:0/2074728907 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f03a400c0c0 con 0x7f03b01b0a10 2021-02-10T11:37:25.196 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:25.177 7f03a3fff700 1 -- 172.21.2.102:0/2074728907 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03a400c3c0 con 0x7f03b01b0a10 2021-02-10T11:37:25.196 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:25.177 7f03a3fff700 1 -- 172.21.2.102:0/2074728907 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f03a400c8e0 con 0x7f03b01b0a10 2021-02-10T11:37:25.196 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:25.177 7f03b7238700 1 -- 172.21.2.102:0/2074728907 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f03b01b0a10 msgr2=0x7f03b00cf330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:25.196 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:25.177 7f03b7238700 1 --2- 172.21.2.102:0/2074728907 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f03b01b0a10 0x7f03b00cf330 secure :-1 s=READY pgs=2942 cs=0 l=1 rev1=1 rx=0x7f03a4006700 tx=0x7f03a4009b80).stop 2021-02-10T11:37:25.196 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:25.178 7f03b7238700 1 -- 172.21.2.102:0/2074728907 shutdown_connections 2021-02-10T11:37:25.196 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:25.178 7f03b7238700 1 --2- 172.21.2.102:0/2074728907 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f03b01b0a10 0x7f03b00cf330 unknown :-1 s=CLOSED pgs=2942 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:25.196 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:25.178 7f03b7238700 1 --2- 172.21.2.102:0/2074728907 >> v2:172.21.2.113:3300/0 conn(0x7f03b00d1870 0x7f03b01b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:25.196 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:25.178 7f03b7238700 1 -- 172.21.2.102:0/2074728907 >> 172.21.2.102:0/2074728907 conn(0x7f03b00ce450 msgr2=0x7f03b00d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:25.196 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:25.178 7f03b7238700 1 -- 172.21.2.102:0/2074728907 shutdown_connections 2021-02-10T11:37:25.196 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:25.178 7f03b7238700 1 -- 172.21.2.102:0/2074728907 wait complete. 2021-02-10T11:37:25.198 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:25.193 7f03b7238700 1 Processor -- start 2021-02-10T11:37:25.198 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:25.193 7f03b7238700 1 -- start start 2021-02-10T11:37:25.198 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:25.193 7f03b7238700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f03b00ae820 0x7f03b01971a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:25.199 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:25.193 7f03b7238700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f03b01977c0 0x7f03b0096fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:25.199 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:25.193 7f03b7238700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f03b00975c0 0x7f03b008c660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:25.199 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:25.193 7f03b7238700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03b00979c0 con 0x7f03b00ae820 2021-02-10T11:37:25.199 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:25.193 7f03b7238700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03b0097be0 con 0x7f03b00975c0 2021-02-10T11:37:25.199 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:25.193 7f03b7238700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03b008cc80 con 0x7f03b01977c0 2021-02-10T11:37:25.199 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:25.193 7f03b5054700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f03b00ae820 0x7f03b01971a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:25.199 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:25.193 7f03b5054700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f03b00ae820 0x7f03b01971a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:38614/0 (socket says 172.21.2.102:38614) 2021-02-10T11:37:25.199 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:25.193 7f03b5054700 1 -- 172.21.2.102:0/1471092679 learned_addr learned my addr 172.21.2.102:0/1471092679 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:37:25.199 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:25.194 7f03b5054700 1 -- 172.21.2.102:0/1471092679 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f03b01977c0 msgr2=0x7f03b0096fa0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:37:25.199 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:25.194 7f03b5054700 1 --2- 172.21.2.102:0/1471092679 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f03b01977c0 0x7f03b0096fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:25.199 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:25.194 7f03b5054700 1 -- 172.21.2.102:0/1471092679 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f03b00975c0 msgr2=0x7f03b008c660 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:37:25.199 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:25.194 7f03b5054700 1 --2- 172.21.2.102:0/1471092679 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f03b00975c0 0x7f03b008c660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:25.199 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:25.194 7f03b5054700 1 -- 172.21.2.102:0/1471092679 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03b008cea0 con 0x7f03b00ae820 2021-02-10T11:37:25.200 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:25.194 7f03b5054700 1 --2- 172.21.2.102:0/1471092679 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f03b00ae820 0x7f03b01971a0 secure :-1 s=READY pgs=2943 cs=0 l=1 rev1=1 rx=0x7f03ac006f00 tx=0x7f03ac00b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:25.200 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:25.194 7f03a1ffb700 1 -- 172.21.2.102:0/1471092679 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f03ac00bd30 con 0x7f03b00ae820 2021-02-10T11:37:25.200 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:25.194 7f03a1ffb700 1 -- 172.21.2.102:0/1471092679 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03ac000c90 con 0x7f03b00ae820 2021-02-10T11:37:25.200 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:25.194 7f03b7238700 1 -- 172.21.2.102:0/1471092679 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03b008cea0 con 0x7f03b00ae820 2021-02-10T11:37:25.200 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:25.194 7f03a1ffb700 1 -- 172.21.2.102:0/1471092679 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f03ac0022a0 con 0x7f03b00ae820 2021-02-10T11:37:25.200 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:25.194 7f03b7238700 1 -- 172.21.2.102:0/1471092679 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03b008cea0 con 0x7f03b00ae820 2021-02-10T11:37:25.200 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:25.194 7f03a1ffb700 1 -- 172.21.2.102:0/1471092679 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f03ac005310 con 0x7f03b00ae820 2021-02-10T11:37:25.200 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:25.195 7f03a1ffb700 1 --2- 172.21.2.102:0/1471092679 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0390018730 0x7f039001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:25.200 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:25.195 7f03a1ffb700 1 -- 172.21.2.102:0/1471092679 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(164..164 src has 1..164) v4 ==== 5617+0+0 (secure 0 0 0) 0x7f03ac032660 con 0x7f03b00ae820 2021-02-10T11:37:25.200 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:25.195 7f03b4853700 1 --2- 172.21.2.102:0/1471092679 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0390018730 0x7f039001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:25.200 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:25.195 7f03b4853700 1 --2- 172.21.2.102:0/1471092679 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0390018730 0x7f039001ac10 crc :-1 s=READY pgs=1291 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:25.227 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:25.225 7f03b7238700 1 -- 172.21.2.102:0/1471092679 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03b008d7c0 con 0x7f03b00ae820 2021-02-10T11:37:25.230 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:25.228 7f03a1ffb700 1 -- 172.21.2.102:0/1471092679 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f03ac006b90 con 0x7f03b00ae820 2021-02-10T11:37:25.359 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:25.357 7f03b7238700 1 -- 172.21.2.102:0/1471092679 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f03b008d7c0 con 0x7f03b00ae820 2021-02-10T11:37:25.360 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:25.358 7f03a1ffb700 1 -- 172.21.2.102:0/1471092679 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 142 v142) v1 ==== 95+0+2108 (secure 0 0 0) 0x7f03ac00bd30 con 0x7f03b00ae820 2021-02-10T11:37:25.361 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:37:25.361 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":142,"default_fscid":17,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":12405,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.102:6835/4152930669","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6834","nonce":4152930669},{"type":"v1","addr":"172.21.2.102:6835","nonce":4152930669}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":137}],"filesystems":[{"mdsmap":{"epoch":141,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:36:46.331874","modified":"2021-02-10 11:36:48.340104","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7130},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7130":{"gid":7130,"name":"b","rank":0,"incarnation":138,"state":"up:active","state_seq":3,"addr":"172.21.2.113:6833/2966977555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6832","nonce":2966977555},{"type":"v1","addr":"172.21.2.113:6833","nonce":2966977555}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[35],"metadata_pool":34,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":17}]} 2021-02-10T11:37:25.361 INFO:teuthology.orchestra.run.gibba002.stderr:dumped fsmap epoch 142 2021-02-10T11:37:25.361 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:25.360 7f03b7238700 1 -- 172.21.2.102:0/1471092679 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0390018730 msgr2=0x7f039001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:25.362 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:25.360 7f03b7238700 1 --2- 172.21.2.102:0/1471092679 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0390018730 0x7f039001ac10 crc :-1 s=READY pgs=1291 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:25.362 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:25.360 7f03b7238700 1 -- 172.21.2.102:0/1471092679 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f03b00ae820 msgr2=0x7f03b01971a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:25.362 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:25.360 7f03b7238700 1 --2- 172.21.2.102:0/1471092679 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f03b00ae820 0x7f03b01971a0 secure :-1 s=READY pgs=2943 cs=0 l=1 rev1=1 rx=0x7f03ac006f00 tx=0x7f03ac00b940).stop 2021-02-10T11:37:25.362 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:25.360 7f03b7238700 1 -- 172.21.2.102:0/1471092679 shutdown_connections 2021-02-10T11:37:25.362 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:25.360 7f03b7238700 1 --2- 172.21.2.102:0/1471092679 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0390018730 0x7f039001ac10 unknown :-1 s=CLOSED pgs=1291 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:25.363 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:25.360 7f03b7238700 1 --2- 172.21.2.102:0/1471092679 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f03b00975c0 0x7f03b008c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:25.363 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:25.360 7f03b7238700 1 --2- 172.21.2.102:0/1471092679 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f03b01977c0 0x7f03b0096fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:25.363 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:25.360 7f03b7238700 1 --2- 172.21.2.102:0/1471092679 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f03b00ae820 0x7f03b01971a0 unknown :-1 s=CLOSED pgs=2943 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:25.363 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:25.360 7f03b7238700 1 -- 172.21.2.102:0/1471092679 >> 172.21.2.102:0/1471092679 conn(0x7f03b0198a50 msgr2=0x7f03b0094b90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:25.363 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:25.360 7f03b7238700 1 -- 172.21.2.102:0/1471092679 shutdown_connections 2021-02-10T11:37:25.363 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:25.360 7f03b7238700 1 -- 172.21.2.102:0/1471092679 wait complete. 2021-02-10T11:37:25.371 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs fail cephfs 2021-02-10T11:37:25.798 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:25.780 7f640c3db700 1 Processor -- start 2021-02-10T11:37:25.798 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:25.780 7f640c3db700 1 -- start start 2021-02-10T11:37:25.798 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:25.780 7f640c3db700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f64040d1870 0x7f64041b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:25.798 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:25.780 7f640c3db700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f64040f0650 0x7f64040cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:25.798 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:25.780 7f640c3db700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64040f0b10 con 0x7f64041b0a10 2021-02-10T11:37:25.799 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:25.780 7f640c3db700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64040e5ed0 con 0x7f64040d1870 2021-02-10T11:37:25.799 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:25.780 7f640c3db700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64040d1c70 con 0x7f64040f0650 2021-02-10T11:37:25.799 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:25.780 7f640a1f7700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f64040d1870 0x7f64041b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:25.799 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:25.780 7f640a1f7700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f64040d1870 0x7f64041b03f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:38618/0 (socket says 172.21.2.102:38618) 2021-02-10T11:37:25.799 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:25.780 7f640a1f7700 1 -- 172.21.2.102:0/684024024 learned_addr learned my addr 172.21.2.102:0/684024024 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:37:25.799 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:25.781 7f640a1f7700 1 -- 172.21.2.102:0/684024024 >> v2:172.21.2.113:3300/0 conn(0x7f64040f0650 msgr2=0x7f64040cfb20 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:37:25.799 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:25.781 7f640a1f7700 1 --2- 172.21.2.102:0/684024024 >> v2:172.21.2.113:3300/0 conn(0x7f64040f0650 0x7f64040cfb20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:25.799 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:25.781 7f640a1f7700 1 -- 172.21.2.102:0/684024024 >> v1:172.21.2.113:6789/0 conn(0x7f64041b0a10 legacy=0x7f64040cf330 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:37:25.799 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:25.781 7f640a1f7700 1 -- 172.21.2.102:0/684024024 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64040d1e90 con 0x7f64040d1870 2021-02-10T11:37:25.799 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:25.781 7f640a1f7700 1 --2- 172.21.2.102:0/684024024 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f64040d1870 0x7f64041b03f0 secure :-1 s=READY pgs=2944 cs=0 l=1 rev1=1 rx=0x7f63f8006700 tx=0x7f63f8009b80).ready entity=mon.0 client_cookie=b434f7e6b22da3d0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:25.799 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:25.781 7f64091f5700 1 -- 172.21.2.102:0/684024024 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f63f800c0c0 con 0x7f64040d1870 2021-02-10T11:37:25.799 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:25.781 7f64091f5700 1 -- 172.21.2.102:0/684024024 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f800c340 con 0x7f64040d1870 2021-02-10T11:37:25.799 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:25.781 7f64091f5700 1 -- 172.21.2.102:0/684024024 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f63f800c860 con 0x7f64040d1870 2021-02-10T11:37:25.800 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:25.781 7f640c3db700 1 -- 172.21.2.102:0/684024024 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f64040d1870 msgr2=0x7f64041b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:25.800 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:25.781 7f640c3db700 1 --2- 172.21.2.102:0/684024024 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f64040d1870 0x7f64041b03f0 secure :-1 s=READY pgs=2944 cs=0 l=1 rev1=1 rx=0x7f63f8006700 tx=0x7f63f8009b80).stop 2021-02-10T11:37:25.800 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:25.782 7f640c3db700 1 -- 172.21.2.102:0/684024024 shutdown_connections 2021-02-10T11:37:25.800 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:25.782 7f640c3db700 1 --2- 172.21.2.102:0/684024024 >> v2:172.21.2.113:3300/0 conn(0x7f64040f0650 0x7f64040cfb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:25.800 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:25.782 7f640c3db700 1 --2- 172.21.2.102:0/684024024 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f64040d1870 0x7f64041b03f0 unknown :-1 s=CLOSED pgs=2944 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:25.800 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:25.782 7f640c3db700 1 -- 172.21.2.102:0/684024024 >> 172.21.2.102:0/684024024 conn(0x7f64040ce450 msgr2=0x7f64040d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:25.800 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:25.782 7f640c3db700 1 -- 172.21.2.102:0/684024024 shutdown_connections 2021-02-10T11:37:25.800 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:25.782 7f640c3db700 1 -- 172.21.2.102:0/684024024 wait complete. 2021-02-10T11:37:25.801 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:25.796 7f640c3db700 1 Processor -- start 2021-02-10T11:37:25.801 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:25.796 7f640c3db700 1 -- start start 2021-02-10T11:37:25.802 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:25.797 7f640c3db700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f64040bb5d0 0x7f64040bfac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:25.802 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:25.797 7f640c3db700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f64040a9b50 0x7f64040a9f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:25.802 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:25.797 7f640c3db700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f64041a7550 0x7f64041a7950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:25.802 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:25.797 7f640c3db700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64040bb9d0 con 0x7f64040bb5d0 2021-02-10T11:37:25.802 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:25.797 7f640c3db700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64040aa570 con 0x7f64041a7550 2021-02-10T11:37:25.802 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:25.797 7f640c3db700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64040aa790 con 0x7f64040a9b50 2021-02-10T11:37:25.802 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:25.797 7f640a1f7700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f64040bb5d0 0x7f64040bfac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:25.802 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:25.797 7f640a1f7700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f64040bb5d0 0x7f64040bfac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:38624/0 (socket says 172.21.2.102:38624) 2021-02-10T11:37:25.802 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:25.797 7f640a1f7700 1 -- 172.21.2.102:0/122982309 learned_addr learned my addr 172.21.2.102:0/122982309 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:37:25.802 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:25.797 7f640a1f7700 1 -- 172.21.2.102:0/122982309 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f64040a9b50 msgr2=0x7f64040a9f50 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:37:25.802 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:25.797 7f640a1f7700 1 --2- 172.21.2.102:0/122982309 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f64040a9b50 0x7f64040a9f50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:25.802 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:25.797 7f640a1f7700 1 -- 172.21.2.102:0/122982309 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f64041a7550 msgr2=0x7f64041a7950 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:37:25.803 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:25.797 7f640a1f7700 1 --2- 172.21.2.102:0/122982309 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f64041a7550 0x7f64041a7950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:25.803 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:25.797 7f640a1f7700 1 -- 172.21.2.102:0/122982309 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64041a7f70 con 0x7f64040bb5d0 2021-02-10T11:37:25.803 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:25.798 7f640a1f7700 1 --2- 172.21.2.102:0/122982309 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f64040bb5d0 0x7f64040bfac0 secure :-1 s=READY pgs=2945 cs=0 l=1 rev1=1 rx=0x7f63f800b5e0 tx=0x7f63f800b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:25.803 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:25.798 7f63f6ffd700 1 -- 172.21.2.102:0/122982309 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f63f800bd70 con 0x7f64040bb5d0 2021-02-10T11:37:25.803 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:25.798 7f63f6ffd700 1 -- 172.21.2.102:0/122982309 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f8002030 con 0x7f64040bb5d0 2021-02-10T11:37:25.803 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:25.798 7f63f6ffd700 1 -- 172.21.2.102:0/122982309 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f63f8002550 con 0x7f64040bb5d0 2021-02-10T11:37:25.803 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:25.798 7f640c3db700 1 -- 172.21.2.102:0/122982309 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64041a7f70 con 0x7f64040bb5d0 2021-02-10T11:37:25.803 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:25.798 7f640c3db700 1 -- 172.21.2.102:0/122982309 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64041a7f70 con 0x7f64040bb5d0 2021-02-10T11:37:25.803 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:25.798 7f63f6ffd700 1 -- 172.21.2.102:0/122982309 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f63f8000c90 con 0x7f64040bb5d0 2021-02-10T11:37:25.803 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:25.799 7f63f6ffd700 1 --2- 172.21.2.102:0/122982309 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f63ec0186c0 0x7f63ec01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:25.803 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:25.799 7f63f6ffd700 1 -- 172.21.2.102:0/122982309 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(164..164 src has 1..164) v4 ==== 5617+0+0 (secure 0 0 0) 0x7f63f80325d0 con 0x7f64040bb5d0 2021-02-10T11:37:25.803 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:25.799 7f64099f6700 1 --2- 172.21.2.102:0/122982309 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f63ec0186c0 0x7f63ec01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:25.803 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:25.799 7f64099f6700 1 --2- 172.21.2.102:0/122982309 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f63ec0186c0 0x7f63ec01abc0 crc :-1 s=READY pgs=1292 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:25.832 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:25.830 7f640c3db700 1 -- 172.21.2.102:0/122982309 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64040ee0b0 con 0x7f64040bb5d0 2021-02-10T11:37:25.835 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:25.833 7f63f6ffd700 1 -- 172.21.2.102:0/122982309 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f63f8028150 con 0x7f64040bb5d0 2021-02-10T11:37:25.962 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:25.960 7f640c3db700 1 -- 172.21.2.102:0/122982309 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f64040ce450 con 0x7f64040bb5d0 2021-02-10T11:37:26.171 INFO:tasks.ceph.mon.a.gibba002.stderr:2021-02-10 11:37:26.169 7f8dd2755700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-02-10T11:37:26.177 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:26.175 7f63f6ffd700 1 -- 172.21.2.102:0/122982309 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v143) v1 ==== 161+0+0 (secure 0 0 0) 0x7f63f8028ef0 con 0x7f64040bb5d0 2021-02-10T11:37:26.227 INFO:teuthology.orchestra.run.gibba002.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2021-02-10T11:37:26.228 INFO:tasks.ceph.mds.b.gibba013.stdout:starting mds.b at 2021-02-10T11:37:26.229 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:26.227 7f640c3db700 1 -- 172.21.2.102:0/122982309 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f63ec0186c0 msgr2=0x7f63ec01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:26.229 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:26.227 7f640c3db700 1 --2- 172.21.2.102:0/122982309 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f63ec0186c0 0x7f63ec01abc0 crc :-1 s=READY pgs=1292 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:26.229 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:26.227 7f640c3db700 1 -- 172.21.2.102:0/122982309 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f64040bb5d0 msgr2=0x7f64040bfac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:26.229 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:26.227 7f640c3db700 1 --2- 172.21.2.102:0/122982309 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f64040bb5d0 0x7f64040bfac0 secure :-1 s=READY pgs=2945 cs=0 l=1 rev1=1 rx=0x7f63f800b5e0 tx=0x7f63f800b980).stop 2021-02-10T11:37:26.229 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:26.227 7f640c3db700 1 -- 172.21.2.102:0/122982309 shutdown_connections 2021-02-10T11:37:26.230 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:26.227 7f640c3db700 1 --2- 172.21.2.102:0/122982309 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f63ec0186c0 0x7f63ec01abc0 unknown :-1 s=CLOSED pgs=1292 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:26.230 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:26.227 7f640c3db700 1 --2- 172.21.2.102:0/122982309 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f64041a7550 0x7f64041a7950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:26.230 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:26.227 7f640c3db700 1 --2- 172.21.2.102:0/122982309 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f64040a9b50 0x7f64040a9f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:26.230 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:26.227 7f640c3db700 1 --2- 172.21.2.102:0/122982309 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f64040bb5d0 0x7f64040bfac0 unknown :-1 s=CLOSED pgs=2945 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:26.230 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:26.227 7f640c3db700 1 -- 172.21.2.102:0/122982309 >> 172.21.2.102:0/122982309 conn(0x7f64040a6eb0 msgr2=0x7f64040bf2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:26.230 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:26.227 7f640c3db700 1 -- 172.21.2.102:0/122982309 shutdown_connections 2021-02-10T11:37:26.230 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:26.227 7f640c3db700 1 -- 172.21.2.102:0/122982309 wait complete. 2021-02-10T11:37:26.237 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:37:26.668 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:26.647 7f266fef7700 1 Processor -- start 2021-02-10T11:37:26.668 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:26.647 7f266fef7700 1 -- start start 2021-02-10T11:37:26.668 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:26.648 7f266fef7700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f26680d1c70 0x7f26680f20a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:26.668 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:26.648 7f266fef7700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26680f0b10 con 0x7f26680d1870 2021-02-10T11:37:26.668 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:26.648 7f266fef7700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26681b0be0 con 0x7f26680d0340 2021-02-10T11:37:26.668 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:26.648 7f266fef7700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26680e5ed0 con 0x7f26680d1c70 2021-02-10T11:37:26.668 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:26.648 7f266dd13700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f26680d1870 0x7f26681b03f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:59086/0 (socket says 172.21.2.102:59086) 2021-02-10T11:37:26.668 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:26.648 7f266dd13700 1 -- 172.21.2.102:0/2456230201 learned_addr learned my addr 172.21.2.102:0/2456230201 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:37:26.668 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:26.649 7f266cd11700 1 -- 172.21.2.102:0/2456230201 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1505160536 0 0) 0x7f265c001180 con 0x7f26680d1870 2021-02-10T11:37:26.669 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:26.649 7f266cd11700 1 -- 172.21.2.102:0/2456230201 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2654002fe0 con 0x7f26680d1870 2021-02-10T11:37:26.669 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:26.649 7f266e514700 1 --2- 172.21.2.102:0/2456230201 >> v2:172.21.2.113:3300/0 conn(0x7f26680d1c70 0x7f26680f20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:26.669 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:26.649 7f266cd11700 1 -- 172.21.2.102:0/2456230201 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4292098481 0 0) 0x7f265c001180 con 0x7f26680d1870 2021-02-10T11:37:26.669 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:26.649 7f266cd11700 1 -- 172.21.2.102:0/2456230201 >> v2:172.21.2.113:3300/0 conn(0x7f26680d1c70 msgr2=0x7f26680f20a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:37:26.669 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:26.649 7f266cd11700 1 --2- 172.21.2.102:0/2456230201 >> v2:172.21.2.113:3300/0 conn(0x7f26680d1c70 0x7f26680f20a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:26.669 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:26.649 7f266cd11700 1 -- 172.21.2.102:0/2456230201 >> v1:172.21.2.113:6789/0 conn(0x7f26680d0340 legacy=0x7f26680ef640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:26.669 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:26.649 7f266cd11700 1 -- 172.21.2.102:0/2456230201 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26680efe30 con 0x7f26680d1870 2021-02-10T11:37:26.669 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:26.649 7f266cd11700 1 -- 172.21.2.102:0/2456230201 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7f265c001680 con 0x7f26680d1870 2021-02-10T11:37:26.669 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:26.649 7f266cd11700 1 -- 172.21.2.102:0/2456230201 >> v1:172.21.2.102:6789/0 conn(0x7f26680d1870 legacy=0x7f26681b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:26.669 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:26.649 7f266cd11700 1 --2- 172.21.2.102:0/2456230201 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2654003b20 0x7f2654005f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:26.669 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:26.649 7f266cd11700 1 --2- 172.21.2.102:0/2456230201 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2654006780 0x7f2654008ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:26.669 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:26.649 7f266cd11700 1 --2- 172.21.2.102:0/2456230201 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f26540095a0 0x7f265400bac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:26.669 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:26.649 7f266cd11700 1 -- 172.21.2.102:0/2456230201 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f265400c310 con 0x7f26540095a0 2021-02-10T11:37:26.670 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:26.649 7f266cd11700 1 -- 172.21.2.102:0/2456230201 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f265400c530 con 0x7f2654003b20 2021-02-10T11:37:26.670 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:26.649 7f266cd11700 1 -- 172.21.2.102:0/2456230201 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f265400c750 con 0x7f2654006780 2021-02-10T11:37:26.670 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:26.649 7f266e514700 1 --2- 172.21.2.102:0/2456230201 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f26540095a0 0x7f265400bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:26.670 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:26.649 7f266dd13700 1 --2- 172.21.2.102:0/2456230201 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2654003b20 0x7f2654005f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:26.670 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:26.649 7f266d512700 1 --2- 172.21.2.102:0/2456230201 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2654006780 0x7f2654008ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:26.670 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:26.650 7f266e514700 1 -- 172.21.2.102:0/2456230201 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2654006780 msgr2=0x7f2654008ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:37:26.670 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:26.650 7f266e514700 1 --2- 172.21.2.102:0/2456230201 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2654006780 0x7f2654008ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:26.670 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:26.650 7f266e514700 1 -- 172.21.2.102:0/2456230201 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2654003b20 msgr2=0x7f2654005f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:37:26.670 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:26.650 7f266e514700 1 --2- 172.21.2.102:0/2456230201 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2654003b20 0x7f2654005f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:26.670 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:26.650 7f266e514700 1 -- 172.21.2.102:0/2456230201 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2654050990 con 0x7f26540095a0 2021-02-10T11:37:26.670 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:26.650 7f266d512700 1 --2- 172.21.2.102:0/2456230201 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2654006780 0x7f2654008ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:37:26.670 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:26.650 7f266e514700 1 -- 172.21.2.102:0/2456230201 reap_dead start 2021-02-10T11:37:26.670 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:26.650 7f266e514700 1 --2- 172.21.2.102:0/2456230201 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f26540095a0 0x7f265400bac0 secure :-1 s=READY pgs=2947 cs=0 l=1 rev1=1 rx=0x7f2660006a40 tx=0x7f266000c0c0).ready entity=mon.0 client_cookie=7dbaf673842a2119 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:26.671 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:26.650 7f266cd11700 1 -- 172.21.2.102:0/2456230201 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2660006a80 con 0x7f26540095a0 2021-02-10T11:37:26.671 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:26.650 7f266cd11700 1 -- 172.21.2.102:0/2456230201 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2660007070 con 0x7f26540095a0 2021-02-10T11:37:26.671 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:26.650 7f266cd11700 1 -- 172.21.2.102:0/2456230201 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f26600074c0 con 0x7f26540095a0 2021-02-10T11:37:26.671 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:26.650 7f266fef7700 1 -- 172.21.2.102:0/2456230201 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f26540095a0 msgr2=0x7f265400bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:26.671 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:26.650 7f266fef7700 1 --2- 172.21.2.102:0/2456230201 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f26540095a0 0x7f265400bac0 secure :-1 s=READY pgs=2947 cs=0 l=1 rev1=1 rx=0x7f2660006a40 tx=0x7f266000c0c0).stop 2021-02-10T11:37:26.671 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:26.651 7f266fef7700 1 -- 172.21.2.102:0/2456230201 shutdown_connections 2021-02-10T11:37:26.671 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:26.651 7f266fef7700 1 --2- 172.21.2.102:0/2456230201 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f26540095a0 0x7f265400bac0 unknown :-1 s=CLOSED pgs=2947 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:26.671 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:26.651 7f266fef7700 1 -- 172.21.2.102:0/2456230201 >> 172.21.2.102:0/2456230201 conn(0x7f26680ce450 msgr2=0x7f26680d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:26.671 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:26.651 7f266fef7700 1 -- 172.21.2.102:0/2456230201 shutdown_connections 2021-02-10T11:37:26.671 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:26.651 7f266fef7700 1 -- 172.21.2.102:0/2456230201 wait complete. 2021-02-10T11:37:26.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:26.666 7f266fef7700 1 Processor -- start 2021-02-10T11:37:26.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:26.666 7f266fef7700 1 -- start start 2021-02-10T11:37:26.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:26.667 7f266fef7700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2668195650 0x7f26680b5560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:26.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:26.667 7f266fef7700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f26680b5b80 0x7f26681a2e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:26.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:26.667 7f266fef7700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f26680acfa0 0x7f26681a3440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:26.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:26.667 7f266fef7700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26680b6010 con 0x7f26680b5b80 2021-02-10T11:37:26.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:26.667 7f266fef7700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26681a3a60 con 0x7f2668195650 2021-02-10T11:37:26.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:26.667 7f266fef7700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26681a3c40 con 0x7f26680acfa0 2021-02-10T11:37:26.674 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:26.667 7f266d512700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f26680b5b80 0x7f26681a2e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:26.674 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:26.667 7f266d512700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f26680b5b80 0x7f26681a2e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:38644/0 (socket says 172.21.2.102:38644) 2021-02-10T11:37:26.674 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:26.667 7f266d512700 1 -- 172.21.2.102:0/4033284161 learned_addr learned my addr 172.21.2.102:0/4033284161 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:37:26.674 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:26.667 7f266dd13700 1 --2- 172.21.2.102:0/4033284161 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2668195650 0x7f26680b5560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:26.674 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:26.667 7f266d512700 1 -- 172.21.2.102:0/4033284161 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f26680acfa0 msgr2=0x7f26681a3440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:26.674 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:26.667 7f266d512700 1 --2- 172.21.2.102:0/4033284161 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f26680acfa0 0x7f26681a3440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:26.674 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:26.667 7f266d512700 1 -- 172.21.2.102:0/4033284161 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2668195650 msgr2=0x7f26680b5560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:26.674 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:26.667 7f266d512700 1 --2- 172.21.2.102:0/4033284161 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2668195650 0x7f26680b5560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:26.674 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:26.667 7f266d512700 1 -- 172.21.2.102:0/4033284161 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26680bcae0 con 0x7f26680b5b80 2021-02-10T11:37:26.674 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:26.667 7f266d512700 1 --2- 172.21.2.102:0/4033284161 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f26680b5b80 0x7f26681a2e20 secure :-1 s=READY pgs=2948 cs=0 l=1 rev1=1 rx=0x7f2664006f10 tx=0x7f266400b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:26.674 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:26.668 7f265a7fc700 1 -- 172.21.2.102:0/4033284161 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f266400bd80 con 0x7f26680b5b80 2021-02-10T11:37:26.674 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:26.668 7f265a7fc700 1 -- 172.21.2.102:0/4033284161 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2664000ca0 con 0x7f26680b5b80 2021-02-10T11:37:26.674 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:26.668 7f265a7fc700 1 -- 172.21.2.102:0/4033284161 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f26640022b0 con 0x7f26680b5b80 2021-02-10T11:37:26.674 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:26.668 7f266fef7700 1 -- 172.21.2.102:0/4033284161 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26680bcae0 con 0x7f26680b5b80 2021-02-10T11:37:26.675 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:26.668 7f266fef7700 1 -- 172.21.2.102:0/4033284161 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26680bcae0 con 0x7f26680b5b80 2021-02-10T11:37:26.675 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:26.669 7f265a7fc700 1 -- 172.21.2.102:0/4033284161 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f26640052e0 con 0x7f26680b5b80 2021-02-10T11:37:26.675 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:26.669 7f265a7fc700 1 --2- 172.21.2.102:0/4033284161 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f26500186c0 0x7f265001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:26.675 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:26.669 7f265a7fc700 1 -- 172.21.2.102:0/4033284161 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(165..165 src has 1..165) v4 ==== 5675+0+0 (secure 0 0 0) 0x7f26640326d0 con 0x7f26680b5b80 2021-02-10T11:37:26.675 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:26.669 7f266dd13700 1 --2- 172.21.2.102:0/4033284161 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f26500186c0 0x7f265001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:26.675 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:26.669 7f266dd13700 1 --2- 172.21.2.102:0/4033284161 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f26500186c0 0x7f265001abc0 crc :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:26.700 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:26.698 7f266fef7700 1 -- 172.21.2.102:0/4033284161 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26680b6010 con 0x7f26680b5b80 2021-02-10T11:37:26.702 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:26.700 7f265a7fc700 1 -- 172.21.2.102:0/4033284161 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f2664006be0 con 0x7f26680b5b80 2021-02-10T11:37:26.830 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:26.828 7f266fef7700 1 -- 172.21.2.102:0/4033284161 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f26680b6010 con 0x7f26680b5b80 2021-02-10T11:37:26.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:26.829 7f265a7fc700 1 -- 172.21.2.102:0/4033284161 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 143 v143) v1 ==== 95+0+1763 (secure 0 0 0) 0x7f266400bd80 con 0x7f26680b5b80 2021-02-10T11:37:26.831 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:37:26.832 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":143,"default_fscid":17,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":12405,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.102:6835/4152930669","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6834","nonce":4152930669},{"type":"v1","addr":"172.21.2.102:6835","nonce":4152930669}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":137}],"filesystems":[{"mdsmap":{"epoch":143,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:36:46.331874","modified":"2021-02-10 11:37:26.170564","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":165,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[35],"metadata_pool":34,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":17}]} 2021-02-10T11:37:26.832 INFO:teuthology.orchestra.run.gibba002.stderr:dumped fsmap epoch 143 2021-02-10T11:37:26.833 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:26.831 7f266fef7700 1 -- 172.21.2.102:0/4033284161 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f26500186c0 msgr2=0x7f265001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:26.833 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:26.831 7f266fef7700 1 --2- 172.21.2.102:0/4033284161 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f26500186c0 0x7f265001abc0 crc :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:26.833 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:26.831 7f266fef7700 1 -- 172.21.2.102:0/4033284161 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f26680b5b80 msgr2=0x7f26681a2e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:26.833 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:26.831 7f266fef7700 1 --2- 172.21.2.102:0/4033284161 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f26680b5b80 0x7f26681a2e20 secure :-1 s=READY pgs=2948 cs=0 l=1 rev1=1 rx=0x7f2664006f10 tx=0x7f266400b990).stop 2021-02-10T11:37:26.833 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:26.831 7f266fef7700 1 -- 172.21.2.102:0/4033284161 shutdown_connections 2021-02-10T11:37:26.834 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:26.831 7f266fef7700 1 --2- 172.21.2.102:0/4033284161 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f26680acfa0 0x7f26681a3440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:26.834 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:26.831 7f266fef7700 1 --2- 172.21.2.102:0/4033284161 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f26680b5b80 0x7f26681a2e20 unknown :-1 s=CLOSED pgs=2948 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:26.834 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:26.831 7f266fef7700 1 --2- 172.21.2.102:0/4033284161 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f26500186c0 0x7f265001abc0 unknown :-1 s=CLOSED pgs=1293 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:26.834 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:26.831 7f266fef7700 1 --2- 172.21.2.102:0/4033284161 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2668195650 0x7f26680b5560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:26.834 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:26.831 7f266fef7700 1 -- 172.21.2.102:0/4033284161 >> 172.21.2.102:0/4033284161 conn(0x7f26680b3040 msgr2=0x7f26680c3250 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:26.834 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:26.831 7f266fef7700 1 -- 172.21.2.102:0/4033284161 shutdown_connections 2021-02-10T11:37:26.834 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:26.831 7f266fef7700 1 -- 172.21.2.102:0/4033284161 wait complete. 2021-02-10T11:37:26.841 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2021-02-10T11:37:27.274 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:27.255 7fc021a83700 1 Processor -- start 2021-02-10T11:37:27.275 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:27.256 7fc021a83700 1 -- start start 2021-02-10T11:37:27.275 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:27.256 7fc021a83700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc01c0d1870 0x7fc01c1b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:27.275 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:27.256 7fc021a83700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fc01c0f0650 0x7fc01c0cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:27.275 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:27.256 7fc021a83700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc01c0f0b10 con 0x7fc01c1b0a10 2021-02-10T11:37:27.275 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:27.256 7fc021a83700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc01c0fe230 con 0x7fc01c0f0650 2021-02-10T11:37:27.275 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:27.256 7fc021a83700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc01c0e5ed0 con 0x7fc01c0d1870 2021-02-10T11:37:27.275 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:27.256 7fc01affd700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fc01c1b0a10 0x7fc01c0cf330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:59106/0 (socket says 172.21.2.102:59106) 2021-02-10T11:37:27.275 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:27.256 7fc01affd700 1 -- 172.21.2.102:0/1043946583 learned_addr learned my addr 172.21.2.102:0/1043946583 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:37:27.275 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:27.257 7fc01b7fe700 1 --2- 172.21.2.102:0/1043946583 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc01c0d1870 0x7fc01c1b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:27.275 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:27.257 7fc01bfff700 1 --2- 172.21.2.102:0/1043946583 >> v2:172.21.2.113:3300/0 conn(0x7fc01c0f0650 0x7fc01c0cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:27.275 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:27.257 7fc01a7fc700 1 -- 172.21.2.102:0/1043946583 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1963316227 0 0) 0x7fc00c0009b0 con 0x7fc01c1b0a10 2021-02-10T11:37:27.276 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:27.257 7fc01a7fc700 1 -- 172.21.2.102:0/1043946583 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc008002fe0 con 0x7fc01c1b0a10 2021-02-10T11:37:27.276 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:27.257 7fc01b7fe700 1 -- 172.21.2.102:0/1043946583 >> v2:172.21.2.113:3300/0 conn(0x7fc01c0f0650 msgr2=0x7fc01c0cfb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:37:27.276 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:27.257 7fc01b7fe700 1 --2- 172.21.2.102:0/1043946583 >> v2:172.21.2.113:3300/0 conn(0x7fc01c0f0650 0x7fc01c0cfb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:27.276 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:27.257 7fc01b7fe700 1 -- 172.21.2.102:0/1043946583 >> v1:172.21.2.102:6789/0 conn(0x7fc01c1b0a10 legacy=0x7fc01c0cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:27.276 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:27.257 7fc01b7fe700 1 -- 172.21.2.102:0/1043946583 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc01c0f20a0 con 0x7fc01c0d1870 2021-02-10T11:37:27.276 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:27.257 7fc01b7fe700 1 --2- 172.21.2.102:0/1043946583 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc01c0d1870 0x7fc01c1b03f0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0x7fc0040069d0 tx=0x7fc004009e40).ready entity=mon.2 client_cookie=25d09abc2d7b986e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:27.276 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:27.258 7fc01a7fc700 1 -- 172.21.2.102:0/1043946583 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc00400b510 con 0x7fc01c0d1870 2021-02-10T11:37:27.276 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:27.258 7fc01a7fc700 1 -- 172.21.2.102:0/1043946583 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc00400b890 con 0x7fc01c0d1870 2021-02-10T11:37:27.276 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:27.258 7fc01a7fc700 1 -- 172.21.2.102:0/1043946583 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc00400bd70 con 0x7fc01c0d1870 2021-02-10T11:37:27.276 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:27.258 7fc021a83700 1 -- 172.21.2.102:0/1043946583 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc01c0d1870 msgr2=0x7fc01c1b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:27.276 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:27.258 7fc021a83700 1 --2- 172.21.2.102:0/1043946583 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc01c0d1870 0x7fc01c1b03f0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0x7fc0040069d0 tx=0x7fc004009e40).stop 2021-02-10T11:37:27.276 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:27.258 7fc021a83700 1 -- 172.21.2.102:0/1043946583 shutdown_connections 2021-02-10T11:37:27.276 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:27.258 7fc021a83700 1 --2- 172.21.2.102:0/1043946583 >> v2:172.21.2.113:3300/0 conn(0x7fc01c0f0650 0x7fc01c0cfb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:27.277 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:27.258 7fc021a83700 1 --2- 172.21.2.102:0/1043946583 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc01c0d1870 0x7fc01c1b03f0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:27.277 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:27.258 7fc021a83700 1 -- 172.21.2.102:0/1043946583 >> 172.21.2.102:0/1043946583 conn(0x7fc01c0ce450 msgr2=0x7fc01c0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:27.277 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:27.259 7fc021a83700 1 -- 172.21.2.102:0/1043946583 shutdown_connections 2021-02-10T11:37:27.277 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:27.259 7fc021a83700 1 -- 172.21.2.102:0/1043946583 wait complete. 2021-02-10T11:37:27.278 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:27.273 7fc021a83700 1 Processor -- start 2021-02-10T11:37:27.278 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:27.273 7fc021a83700 1 -- start start 2021-02-10T11:37:27.278 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:27.273 7fc021a83700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc01c094310 0x7fc01c0ebe70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:27.278 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:27.273 7fc021a83700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc01c094710 0x7fc01c0bca30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:27.279 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:27.273 7fc021a83700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc01c0ec490 0x7fc01c0bd050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:27.279 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:27.273 7fc021a83700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc01c093d90 con 0x7fc01c094310 2021-02-10T11:37:27.279 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:27.273 7fc021a83700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc01c0bd670 con 0x7fc01c0ec490 2021-02-10T11:37:27.279 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:27.273 7fc021a83700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc01c0939f0 con 0x7fc01c094710 2021-02-10T11:37:27.279 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:27.274 7fc01b7fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc01c094310 0x7fc01c0ebe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:27.279 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:27.274 7fc01b7fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc01c094310 0x7fc01c0ebe70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:38652/0 (socket says 172.21.2.102:38652) 2021-02-10T11:37:27.279 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:27.274 7fc01b7fe700 1 -- 172.21.2.102:0/958880817 learned_addr learned my addr 172.21.2.102:0/958880817 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:37:27.279 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:27.274 7fc01b7fe700 1 -- 172.21.2.102:0/958880817 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc01c094710 msgr2=0x7fc01c0bca30 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:37:27.279 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:27.274 7fc01b7fe700 1 --2- 172.21.2.102:0/958880817 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc01c094710 0x7fc01c0bca30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:27.279 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:27.274 7fc01b7fe700 1 -- 172.21.2.102:0/958880817 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc01c0ec490 msgr2=0x7fc01c0bd050 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:37:27.279 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:27.274 7fc01b7fe700 1 --2- 172.21.2.102:0/958880817 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc01c0ec490 0x7fc01c0bd050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:27.279 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:27.274 7fc01b7fe700 1 -- 172.21.2.102:0/958880817 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc01c0bda80 con 0x7fc01c094310 2021-02-10T11:37:27.279 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:27.274 7fc01b7fe700 1 --2- 172.21.2.102:0/958880817 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc01c094310 0x7fc01c0ebe70 secure :-1 s=READY pgs=2950 cs=0 l=1 rev1=1 rx=0x7fc0040099e0 tx=0x7fc00400bd20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:27.280 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:27.274 7fc013fff700 1 -- 172.21.2.102:0/958880817 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc004006240 con 0x7fc01c094310 2021-02-10T11:37:27.280 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:27.274 7fc013fff700 1 -- 172.21.2.102:0/958880817 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc00400be60 con 0x7fc01c094310 2021-02-10T11:37:27.280 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:27.274 7fc013fff700 1 -- 172.21.2.102:0/958880817 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc004005660 con 0x7fc01c094310 2021-02-10T11:37:27.280 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:27.274 7fc021a83700 1 -- 172.21.2.102:0/958880817 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc01c0bda80 con 0x7fc01c094310 2021-02-10T11:37:27.280 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:27.274 7fc021a83700 1 -- 172.21.2.102:0/958880817 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc01c1a76d0 con 0x7fc01c094310 2021-02-10T11:37:27.280 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:27.275 7fc013fff700 1 -- 172.21.2.102:0/958880817 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fc004005660 con 0x7fc01c094310 2021-02-10T11:37:27.280 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:27.275 7fc013fff700 1 --2- 172.21.2.102:0/958880817 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc0000186c0 0x7fc00001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:27.280 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:27.275 7fc013fff700 1 -- 172.21.2.102:0/958880817 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(165..165 src has 1..165) v4 ==== 5675+0+0 (secure 0 0 0) 0x7fc004032980 con 0x7fc01c094310 2021-02-10T11:37:27.280 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:27.275 7fc01affd700 1 --2- 172.21.2.102:0/958880817 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc0000186c0 0x7fc00001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:27.280 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:27.276 7fc01affd700 1 --2- 172.21.2.102:0/958880817 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc0000186c0 0x7fc00001abc0 crc :-1 s=READY pgs=1294 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:27.308 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:27.306 7fc021a83700 1 -- 172.21.2.102:0/958880817 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc01c0ebc40 con 0x7fc01c094310 2021-02-10T11:37:27.310 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:27.308 7fc013fff700 1 -- 172.21.2.102:0/958880817 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fc004020ef0 con 0x7fc01c094310 2021-02-10T11:37:27.438 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:27.436 7fc021a83700 1 -- 172.21.2.102:0/958880817 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fc01c0ebc40 con 0x7fc01c094310 2021-02-10T11:37:28.182 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:28.179 7fc013fff700 1 -- 172.21.2.102:0/958880817 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v144) v1 ==== 104+0+0 (secure 0 0 0) 0x7fc004020ef0 con 0x7fc01c094310 2021-02-10T11:37:28.208 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:28.206 7fc021a83700 1 -- 172.21.2.102:0/958880817 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc0000186c0 msgr2=0x7fc00001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:28.208 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:28.206 7fc021a83700 1 --2- 172.21.2.102:0/958880817 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc0000186c0 0x7fc00001abc0 crc :-1 s=READY pgs=1294 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:28.209 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:28.206 7fc021a83700 1 -- 172.21.2.102:0/958880817 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc01c094310 msgr2=0x7fc01c0ebe70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:28.209 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:28.206 7fc021a83700 1 --2- 172.21.2.102:0/958880817 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc01c094310 0x7fc01c0ebe70 secure :-1 s=READY pgs=2950 cs=0 l=1 rev1=1 rx=0x7fc0040099e0 tx=0x7fc00400bd20).stop 2021-02-10T11:37:28.209 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:28.207 7fc021a83700 1 -- 172.21.2.102:0/958880817 shutdown_connections 2021-02-10T11:37:28.209 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:28.207 7fc021a83700 1 --2- 172.21.2.102:0/958880817 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc0000186c0 0x7fc00001abc0 unknown :-1 s=CLOSED pgs=1294 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:28.209 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:28.207 7fc021a83700 1 --2- 172.21.2.102:0/958880817 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc01c0ec490 0x7fc01c0bd050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:28.210 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:28.207 7fc021a83700 1 --2- 172.21.2.102:0/958880817 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc01c094710 0x7fc01c0bca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:28.210 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:28.207 7fc021a83700 1 --2- 172.21.2.102:0/958880817 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc01c094310 0x7fc01c0ebe70 unknown :-1 s=CLOSED pgs=2950 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:28.210 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:28.207 7fc021a83700 1 -- 172.21.2.102:0/958880817 >> 172.21.2.102:0/958880817 conn(0x7fc01c08fa10 msgr2=0x7fc01c0bfa60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:28.210 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:28.207 7fc021a83700 1 -- 172.21.2.102:0/958880817 shutdown_connections 2021-02-10T11:37:28.210 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:28.207 7fc021a83700 1 -- 172.21.2.102:0/958880817 wait complete. 2021-02-10T11:37:28.217 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool delete cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2021-02-10T11:37:28.652 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:28.633 7ffb10109700 1 Processor -- start 2021-02-10T11:37:28.652 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:28.633 7ffb10109700 1 -- start start 2021-02-10T11:37:28.652 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:28.634 7ffb10109700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ffb080d1880 0x7ffb081b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:28.652 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:28.634 7ffb10109700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7ffb081b0af0 0x7ffb08108950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:28.652 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:28.634 7ffb10109700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7ffb080d1c80 0x7ffb08108f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:28.653 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:28.634 7ffb10109700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb080e5f00 con 0x7ffb081b0af0 2021-02-10T11:37:28.653 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:28.634 7ffb10109700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb08109a70 con 0x7ffb080d1c80 2021-02-10T11:37:28.653 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:28.634 7ffb10109700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb08088070 con 0x7ffb080d1880 2021-02-10T11:37:28.653 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:28.634 7ffb0d724700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7ffb081b0af0 0x7ffb08108950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:28.653 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:28.634 7ffb0d724700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7ffb081b0af0 0x7ffb08108950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:38660/0 (socket says 172.21.2.102:38660) 2021-02-10T11:37:28.653 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:28.634 7ffb0d724700 1 -- 172.21.2.102:0/4065828607 learned_addr learned my addr 172.21.2.102:0/4065828607 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:37:28.653 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:28.634 7ffb0d724700 1 -- 172.21.2.102:0/4065828607 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ffb080d1880 msgr2=0x7ffb081b04d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:37:28.653 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:28.634 7ffb0d724700 1 --2- 172.21.2.102:0/4065828607 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ffb080d1880 0x7ffb081b04d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:28.653 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:28.634 7ffb0d724700 1 -- 172.21.2.102:0/4065828607 >> v2:172.21.2.113:3300/0 conn(0x7ffb080d1c80 msgr2=0x7ffb08108f70 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:37:28.653 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:28.634 7ffb0d724700 1 --2- 172.21.2.102:0/4065828607 >> v2:172.21.2.113:3300/0 conn(0x7ffb080d1c80 0x7ffb08108f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:28.653 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:28.634 7ffb0d724700 1 -- 172.21.2.102:0/4065828607 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb08109590 con 0x7ffb081b0af0 2021-02-10T11:37:28.653 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:28.634 7ffb0d724700 1 --2- 172.21.2.102:0/4065828607 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ffb081b0af0 0x7ffb08108950 secure :-1 s=READY pgs=2951 cs=0 l=1 rev1=1 rx=0x7ffb04006700 tx=0x7ffb04009b00).ready entity=mon.0 client_cookie=bcfed8a10a87d097 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:28.653 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:28.635 7ffb0cf23700 1 -- 172.21.2.102:0/4065828607 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ffb0400c0c0 con 0x7ffb081b0af0 2021-02-10T11:37:28.654 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:28.635 7ffb0cf23700 1 -- 172.21.2.102:0/4065828607 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb0400c340 con 0x7ffb081b0af0 2021-02-10T11:37:28.654 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:28.635 7ffb0cf23700 1 -- 172.21.2.102:0/4065828607 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ffb0400c820 con 0x7ffb081b0af0 2021-02-10T11:37:28.654 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:28.635 7ffb10109700 1 -- 172.21.2.102:0/4065828607 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ffb081b0af0 msgr2=0x7ffb08108950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:28.654 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:28.635 7ffb10109700 1 --2- 172.21.2.102:0/4065828607 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ffb081b0af0 0x7ffb08108950 secure :-1 s=READY pgs=2951 cs=0 l=1 rev1=1 rx=0x7ffb04006700 tx=0x7ffb04009b00).stop 2021-02-10T11:37:28.654 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:28.635 7ffb10109700 1 -- 172.21.2.102:0/4065828607 shutdown_connections 2021-02-10T11:37:28.654 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:28.635 7ffb10109700 1 --2- 172.21.2.102:0/4065828607 >> v2:172.21.2.113:3300/0 conn(0x7ffb080d1c80 0x7ffb08108f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:28.654 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:28.635 7ffb10109700 1 --2- 172.21.2.102:0/4065828607 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ffb081b0af0 0x7ffb08108950 unknown :-1 s=CLOSED pgs=2951 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:28.654 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:28.635 7ffb10109700 1 --2- 172.21.2.102:0/4065828607 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ffb080d1880 0x7ffb081b04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:28.654 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:28.635 7ffb10109700 1 -- 172.21.2.102:0/4065828607 >> 172.21.2.102:0/4065828607 conn(0x7ffb080ce430 msgr2=0x7ffb080d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:28.654 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:28.636 7ffb10109700 1 -- 172.21.2.102:0/4065828607 shutdown_connections 2021-02-10T11:37:28.654 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:28.636 7ffb10109700 1 -- 172.21.2.102:0/4065828607 wait complete. 2021-02-10T11:37:28.654 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:28.650 7ffb10109700 1 Processor -- start 2021-02-10T11:37:28.655 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:28.650 7ffb10109700 1 -- start start 2021-02-10T11:37:28.656 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:28.650 7ffb10109700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ffb08096710 0x7ffb08096f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:28.656 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:28.650 7ffb10109700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ffb0819c360 0x7ffb080ad520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:28.656 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:28.650 7ffb10109700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ffb080adb40 0x7ffb0819cef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:28.656 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:28.650 7ffb10109700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb0819c970 con 0x7ffb080adb40 2021-02-10T11:37:28.656 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:28.650 7ffb10109700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb08096b10 con 0x7ffb0819c360 2021-02-10T11:37:28.657 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:28.650 7ffb10109700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb08082c60 con 0x7ffb08096710 2021-02-10T11:37:28.657 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:28.651 7ffb0e726700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ffb080adb40 0x7ffb0819cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:28.657 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:28.651 7ffb0e726700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ffb080adb40 0x7ffb0819cef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:38666/0 (socket says 172.21.2.102:38666) 2021-02-10T11:37:28.657 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:28.651 7ffb0e726700 1 -- 172.21.2.102:0/469187531 learned_addr learned my addr 172.21.2.102:0/469187531 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:37:28.657 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:28.651 7ffb0df25700 1 --2- 172.21.2.102:0/469187531 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ffb08096710 0x7ffb08096f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:28.657 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:28.651 7ffb0d724700 1 --2- 172.21.2.102:0/469187531 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ffb0819c360 0x7ffb080ad520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:28.657 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:28.651 7ffb0e726700 1 -- 172.21.2.102:0/469187531 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ffb08096710 msgr2=0x7ffb08096f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:28.657 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:28.651 7ffb0e726700 1 --2- 172.21.2.102:0/469187531 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ffb08096710 0x7ffb08096f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:28.657 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:28.651 7ffb0e726700 1 -- 172.21.2.102:0/469187531 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ffb0819c360 msgr2=0x7ffb080ad520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:28.657 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:28.651 7ffb0e726700 1 --2- 172.21.2.102:0/469187531 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ffb0819c360 0x7ffb080ad520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:28.657 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:28.651 7ffb0e726700 1 -- 172.21.2.102:0/469187531 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb08097590 con 0x7ffb080adb40 2021-02-10T11:37:28.657 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:28.651 7ffb0e726700 1 --2- 172.21.2.102:0/469187531 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ffb080adb40 0x7ffb0819cef0 secure :-1 s=READY pgs=2952 cs=0 l=1 rev1=1 rx=0x7ffb00009960 tx=0x7ffb0000bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:28.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:28.651 7ffafa7fc700 1 -- 172.21.2.102:0/469187531 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ffb00003030 con 0x7ffb080adb40 2021-02-10T11:37:28.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:28.651 7ffafa7fc700 1 -- 172.21.2.102:0/469187531 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb00001310 con 0x7ffb080adb40 2021-02-10T11:37:28.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:28.651 7ffb10109700 1 -- 172.21.2.102:0/469187531 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb08097590 con 0x7ffb080adb40 2021-02-10T11:37:28.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:28.651 7ffafa7fc700 1 -- 172.21.2.102:0/469187531 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ffb00001830 con 0x7ffb080adb40 2021-02-10T11:37:28.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:28.652 7ffb10109700 1 -- 172.21.2.102:0/469187531 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb08097590 con 0x7ffb080adb40 2021-02-10T11:37:28.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:28.652 7ffafa7fc700 1 -- 172.21.2.102:0/469187531 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ffb000014f0 con 0x7ffb080adb40 2021-02-10T11:37:28.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:28.652 7ffafa7fc700 1 --2- 172.21.2.102:0/469187531 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ffaf0018730 0x7ffaf001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:28.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:28.652 7ffafa7fc700 1 -- 172.21.2.102:0/469187531 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(165..165 src has 1..165) v4 ==== 5675+0+0 (secure 0 0 0) 0x7ffb00032a50 con 0x7ffb080adb40 2021-02-10T11:37:28.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:28.652 7ffb0df25700 1 --2- 172.21.2.102:0/469187531 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ffaf0018730 0x7ffaf001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:28.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:28.653 7ffb0df25700 1 --2- 172.21.2.102:0/469187531 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ffaf0018730 0x7ffaf001ac10 crc :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:28.686 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:28.683 7ffb10109700 1 -- 172.21.2.102:0/469187531 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb08109a70 con 0x7ffb080adb40 2021-02-10T11:37:28.688 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:28.686 7ffafa7fc700 1 -- 172.21.2.102:0/469187531 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ffb00001520 con 0x7ffb080adb40 2021-02-10T11:37:28.816 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:28.814 7ffb10109700 1 -- 172.21.2.102:0/469187531 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"pool2": "cephfs_metadata", "prefix": "osd pool delete", "pool": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7ffb08109a70 con 0x7ffb080adb40 2021-02-10T11:37:29.185 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:29.183 7ffafa7fc700 1 -- 172.21.2.102:0/469187531 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"pool2": "cephfs_metadata", "prefix": "osd pool delete", "pool": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v166) v1 ==== 185+0+0 (secure 0 0 0) 0x7ffb00001310 con 0x7ffb080adb40 2021-02-10T11:37:29.232 INFO:teuthology.orchestra.run.gibba002.stderr:pool 'cephfs_metadata' removed 2021-02-10T11:37:29.234 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:29.232 7ffb10109700 1 -- 172.21.2.102:0/469187531 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ffaf0018730 msgr2=0x7ffaf001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:29.234 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:29.232 7ffb10109700 1 --2- 172.21.2.102:0/469187531 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ffaf0018730 0x7ffaf001ac10 crc :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:29.234 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:29.232 7ffb10109700 1 -- 172.21.2.102:0/469187531 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ffb080adb40 msgr2=0x7ffb0819cef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:29.234 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:29.232 7ffb10109700 1 --2- 172.21.2.102:0/469187531 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ffb080adb40 0x7ffb0819cef0 secure :-1 s=READY pgs=2952 cs=0 l=1 rev1=1 rx=0x7ffb00009960 tx=0x7ffb0000bcf0).stop 2021-02-10T11:37:29.234 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:29.232 7ffb10109700 1 -- 172.21.2.102:0/469187531 shutdown_connections 2021-02-10T11:37:29.235 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:29.232 7ffb10109700 1 --2- 172.21.2.102:0/469187531 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ffb080adb40 0x7ffb0819cef0 unknown :-1 s=CLOSED pgs=2952 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:29.235 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:29.232 7ffb10109700 1 --2- 172.21.2.102:0/469187531 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ffaf0018730 0x7ffaf001ac10 unknown :-1 s=CLOSED pgs=1295 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:29.235 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:29.232 7ffb10109700 1 --2- 172.21.2.102:0/469187531 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ffb0819c360 0x7ffb080ad520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:29.235 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:29.232 7ffb10109700 1 --2- 172.21.2.102:0/469187531 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ffb08096710 0x7ffb08096f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:29.235 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:29.232 7ffb10109700 1 -- 172.21.2.102:0/469187531 >> 172.21.2.102:0/469187531 conn(0x7ffb0819e990 msgr2=0x7ffb081918e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:29.235 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:29.232 7ffb10109700 1 -- 172.21.2.102:0/469187531 shutdown_connections 2021-02-10T11:37:29.235 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:29.232 7ffb10109700 1 -- 172.21.2.102:0/469187531 wait complete. 2021-02-10T11:37:29.243 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool delete cephfs_data cephfs_data --yes-i-really-really-mean-it 2021-02-10T11:37:29.679 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:29.660 7f3a747f8700 1 Processor -- start 2021-02-10T11:37:29.679 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:29.660 7f3a747f8700 1 -- start start 2021-02-10T11:37:29.679 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:29.661 7f3a747f8700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f3a6c0d1c80 0x7f3a6c0cef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:29.680 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:29.661 7f3a747f8700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a6c0d0180 con 0x7f3a6c0d1880 2021-02-10T11:37:29.680 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:29.661 7f3a747f8700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a6c0fe1d0 con 0x7f3a6c0cf590 2021-02-10T11:37:29.680 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:29.661 7f3a747f8700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a6c1b0cc0 con 0x7f3a6c0d1c80 2021-02-10T11:37:29.680 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:29.661 7f3a71e13700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f3a6c0d1c80 0x7f3a6c0cef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:29.680 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:29.661 7f3a72614700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f3a6c0d1880 0x7f3a6c1b04d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:59130/0 (socket says 172.21.2.102:59130) 2021-02-10T11:37:29.680 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:29.661 7f3a71e13700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f3a6c0d1c80 0x7f3a6c0cef70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:38674/0 (socket says 172.21.2.102:38674) 2021-02-10T11:37:29.680 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:29.661 7f3a72614700 1 -- 172.21.2.102:0/3070383806 learned_addr learned my addr 172.21.2.102:0/3070383806 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:37:29.680 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:29.661 7f3a71e13700 1 -- 172.21.2.102:0/3070383806 >> v1:172.21.2.113:6789/0 conn(0x7f3a6c0cf590 legacy=0x7f3a6c0cf990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:37:29.680 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:29.661 7f3a71e13700 1 -- 172.21.2.102:0/3070383806 >> v1:172.21.2.102:6789/0 conn(0x7f3a6c0d1880 legacy=0x7f3a6c1b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:29.680 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:29.661 7f3a71e13700 1 -- 172.21.2.102:0/3070383806 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a6c109a70 con 0x7f3a6c0d1c80 2021-02-10T11:37:29.681 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:29.661 7f3a71e13700 1 --2- 172.21.2.102:0/3070383806 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3a6c0d1c80 0x7f3a6c0cef70 secure :-1 s=READY pgs=2954 cs=0 l=1 rev1=1 rx=0x7f3a5c006650 tx=0x7f3a5c009a80).ready entity=mon.0 client_cookie=366fb649e6934a87 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:29.681 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:29.662 7f3a71612700 1 -- 172.21.2.102:0/3070383806 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3a5c0009b0 con 0x7f3a6c0d1c80 2021-02-10T11:37:29.681 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:29.662 7f3a71612700 1 -- 172.21.2.102:0/3070383806 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a5c00c0c0 con 0x7f3a6c0d1c80 2021-02-10T11:37:29.681 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:29.662 7f3a71612700 1 -- 172.21.2.102:0/3070383806 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3a5c00c5e0 con 0x7f3a6c0d1c80 2021-02-10T11:37:29.681 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:29.662 7f3a747f8700 1 -- 172.21.2.102:0/3070383806 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3a6c0d1c80 msgr2=0x7f3a6c0cef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:29.681 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:29.662 7f3a747f8700 1 --2- 172.21.2.102:0/3070383806 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3a6c0d1c80 0x7f3a6c0cef70 secure :-1 s=READY pgs=2954 cs=0 l=1 rev1=1 rx=0x7f3a5c006650 tx=0x7f3a5c009a80).stop 2021-02-10T11:37:29.681 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:29.663 7f3a747f8700 1 -- 172.21.2.102:0/3070383806 shutdown_connections 2021-02-10T11:37:29.681 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:29.663 7f3a747f8700 1 --2- 172.21.2.102:0/3070383806 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3a6c0d1c80 0x7f3a6c0cef70 unknown :-1 s=CLOSED pgs=2954 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:29.681 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:29.663 7f3a747f8700 1 -- 172.21.2.102:0/3070383806 >> 172.21.2.102:0/3070383806 conn(0x7f3a6c0ce430 msgr2=0x7f3a6c0d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:29.681 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:29.663 7f3a747f8700 1 -- 172.21.2.102:0/3070383806 shutdown_connections 2021-02-10T11:37:29.681 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:29.663 7f3a747f8700 1 -- 172.21.2.102:0/3070383806 wait complete. 2021-02-10T11:37:29.683 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:29.677 7f3a747f8700 1 Processor -- start 2021-02-10T11:37:29.683 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:29.677 7f3a747f8700 1 -- start start 2021-02-10T11:37:29.683 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:29.678 7f3a747f8700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3a6c097820 0x7f3a6c097c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:29.683 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:29.678 7f3a747f8700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3a6c08ac80 0x7f3a6c08b080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:29.684 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:29.678 7f3a747f8700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3a6c08bf00 0x7f3a6c08c300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:29.684 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:29.678 7f3a747f8700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a6c08b6a0 con 0x7f3a6c08bf00 2021-02-10T11:37:29.684 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:29.678 7f3a747f8700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a6c09a360 con 0x7f3a6c097820 2021-02-10T11:37:29.684 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:29.678 7f3a747f8700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a6c09a580 con 0x7f3a6c08ac80 2021-02-10T11:37:29.684 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:29.678 7f3a72e15700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3a6c08bf00 0x7f3a6c08c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:29.684 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:29.678 7f3a72e15700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3a6c08bf00 0x7f3a6c08c300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:38680/0 (socket says 172.21.2.102:38680) 2021-02-10T11:37:29.684 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:29.678 7f3a72e15700 1 -- 172.21.2.102:0/3090205719 learned_addr learned my addr 172.21.2.102:0/3090205719 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:37:29.684 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:29.678 7f3a72614700 1 --2- 172.21.2.102:0/3090205719 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3a6c097820 0x7f3a6c097c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:29.684 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:29.678 7f3a71e13700 1 --2- 172.21.2.102:0/3090205719 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3a6c08ac80 0x7f3a6c08b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:29.684 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:29.679 7f3a72e15700 1 -- 172.21.2.102:0/3090205719 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3a6c08ac80 msgr2=0x7f3a6c08b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:29.685 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:29.679 7f3a72e15700 1 --2- 172.21.2.102:0/3090205719 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3a6c08ac80 0x7f3a6c08b080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:29.685 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:29.679 7f3a72e15700 1 -- 172.21.2.102:0/3090205719 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3a6c097820 msgr2=0x7f3a6c097c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:29.685 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:29.679 7f3a72e15700 1 --2- 172.21.2.102:0/3090205719 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3a6c097820 0x7f3a6c097c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:29.685 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:29.679 7f3a72e15700 1 -- 172.21.2.102:0/3090205719 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a6c08c920 con 0x7f3a6c08bf00 2021-02-10T11:37:29.685 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:29.679 7f3a72e15700 1 --2- 172.21.2.102:0/3090205719 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3a6c08bf00 0x7f3a6c08c300 secure :-1 s=READY pgs=2955 cs=0 l=1 rev1=1 rx=0x7f3a64009940 tx=0x7f3a6400bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:29.685 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:29.679 7f3a62ffd700 1 -- 172.21.2.102:0/3090205719 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3a64000f90 con 0x7f3a6c08bf00 2021-02-10T11:37:29.685 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:29.679 7f3a62ffd700 1 -- 172.21.2.102:0/3090205719 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a6400c2f0 con 0x7f3a6c08bf00 2021-02-10T11:37:29.685 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:29.679 7f3a747f8700 1 -- 172.21.2.102:0/3090205719 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a6c08c920 con 0x7f3a6c08bf00 2021-02-10T11:37:29.685 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:29.679 7f3a747f8700 1 -- 172.21.2.102:0/3090205719 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a6c08ceb0 con 0x7f3a6c08bf00 2021-02-10T11:37:29.685 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:29.679 7f3a62ffd700 1 -- 172.21.2.102:0/3090205719 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3a6400c810 con 0x7f3a6c08bf00 2021-02-10T11:37:29.685 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:29.680 7f3a62ffd700 1 -- 172.21.2.102:0/3090205719 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f3a64001220 con 0x7f3a6c08bf00 2021-02-10T11:37:29.686 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:29.680 7f3a62ffd700 1 --2- 172.21.2.102:0/3090205719 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3a54018730 0x7f3a5401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:29.686 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:29.680 7f3a62ffd700 1 -- 172.21.2.102:0/3090205719 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(166..166 src has 1..166) v4 ==== 5237+0+0 (secure 0 0 0) 0x7f3a640329d0 con 0x7f3a6c08bf00 2021-02-10T11:37:29.686 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:29.680 7f3a72614700 1 --2- 172.21.2.102:0/3090205719 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3a54018730 0x7f3a5401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:29.686 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:29.680 7f3a72614700 1 --2- 172.21.2.102:0/3090205719 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3a54018730 0x7f3a5401ac10 crc :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:29.712 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:29.711 7f3a747f8700 1 -- 172.21.2.102:0/3090205719 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a6c080340 con 0x7f3a6c08bf00 2021-02-10T11:37:29.715 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:29.713 7f3a62ffd700 1 -- 172.21.2.102:0/3090205719 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f3a64000f40 con 0x7f3a6c08bf00 2021-02-10T11:37:29.844 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:29.842 7f3a747f8700 1 -- 172.21.2.102:0/3090205719 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"pool2": "cephfs_data", "prefix": "osd pool delete", "pool": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f3a6c088070 con 0x7f3a6c08bf00 2021-02-10T11:37:30.189 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:30.187 7f3a62ffd700 1 -- 172.21.2.102:0/3090205719 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"pool2": "cephfs_data", "prefix": "osd pool delete", "pool": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v167) v1 ==== 173+0+0 (secure 0 0 0) 0x7f3a6401f860 con 0x7f3a6c08bf00 2021-02-10T11:37:30.211 INFO:teuthology.orchestra.run.gibba002.stderr:pool 'cephfs_data' removed 2021-02-10T11:37:30.213 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:30.211 7f3a747f8700 1 -- 172.21.2.102:0/3090205719 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3a54018730 msgr2=0x7f3a5401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:30.214 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:30.211 7f3a747f8700 1 --2- 172.21.2.102:0/3090205719 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3a54018730 0x7f3a5401ac10 crc :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:30.214 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:30.211 7f3a747f8700 1 -- 172.21.2.102:0/3090205719 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3a6c08bf00 msgr2=0x7f3a6c08c300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:30.214 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:30.211 7f3a747f8700 1 --2- 172.21.2.102:0/3090205719 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3a6c08bf00 0x7f3a6c08c300 secure :-1 s=READY pgs=2955 cs=0 l=1 rev1=1 rx=0x7f3a64009940 tx=0x7f3a6400bcf0).stop 2021-02-10T11:37:30.214 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:30.211 7f3a747f8700 1 -- 172.21.2.102:0/3090205719 shutdown_connections 2021-02-10T11:37:30.214 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:30.211 7f3a747f8700 1 --2- 172.21.2.102:0/3090205719 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3a6c08bf00 0x7f3a6c08c300 unknown :-1 s=CLOSED pgs=2955 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:30.214 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:30.211 7f3a747f8700 1 --2- 172.21.2.102:0/3090205719 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3a6c08ac80 0x7f3a6c08b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:30.214 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:30.211 7f3a747f8700 1 --2- 172.21.2.102:0/3090205719 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3a54018730 0x7f3a5401ac10 unknown :-1 s=CLOSED pgs=1296 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:30.214 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:30.211 7f3a747f8700 1 --2- 172.21.2.102:0/3090205719 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3a6c097820 0x7f3a6c097c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:30.214 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:30.211 7f3a747f8700 1 -- 172.21.2.102:0/3090205719 >> 172.21.2.102:0/3090205719 conn(0x7f3a6c09fb50 msgr2=0x7f3a6c0fed60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:30.214 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:30.211 7f3a747f8700 1 -- 172.21.2.102:0/3090205719 shutdown_connections 2021-02-10T11:37:30.214 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:30.212 7f3a747f8700 1 -- 172.21.2.102:0/3090205719 wait complete. 2021-02-10T11:37:30.223 INFO:tasks.ceph.mds.a:Restarting daemon 2021-02-10T11:37:30.223 INFO:tasks.ceph.mds.a:Stopping old one... 2021-02-10T11:37:30.223 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2021-02-10T11:37:30.223 INFO:teuthology.orchestra.run:waiting for 300 2021-02-10T11:37:30.223 INFO:tasks.ceph.mds.b:Restarting daemon 2021-02-10T11:37:30.223 INFO:tasks.ceph.mds.b:Stopping old one... 2021-02-10T11:37:30.223 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2021-02-10T11:37:30.224 INFO:teuthology.orchestra.run:waiting for 300 2021-02-10T11:37:30.229 INFO:tasks.ceph.mds.a:Stopped 2021-02-10T11:37:30.229 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2021-02-10T11:37:30.230 INFO:tasks.ceph.mds.b:Stopped 2021-02-10T11:37:30.230 DEBUG:teuthology.orchestra.run.gibba013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2021-02-10T11:37:30.270 INFO:tasks.ceph.mds.a:Started 2021-02-10T11:37:30.271 DEBUG:teuthology.parallel:result is None 2021-02-10T11:37:30.273 INFO:tasks.ceph.mds.b:Started 2021-02-10T11:37:30.273 DEBUG:teuthology.parallel:result is None 2021-02-10T11:37:30.273 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd blacklist clear 2021-02-10T11:37:30.677 INFO:tasks.ceph.mds.b.gibba013.stdout:starting mds.b at 2021-02-10T11:37:30.681 INFO:tasks.ceph.mds.a.gibba002.stdout:starting mds.a at 2021-02-10T11:37:30.700 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:30.681 7f091b191700 1 Processor -- start 2021-02-10T11:37:30.701 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:30.682 7f091b191700 1 -- start start 2021-02-10T11:37:30.701 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:30.682 7f091b191700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f09140d1860 0x7f09141b03e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:30.701 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:30.682 7f091b191700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f09141b0a00 0x7f09140cf320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:30.701 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:30.682 7f091b191700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09140f0b00 con 0x7f09140d1c60 2021-02-10T11:37:30.701 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:30.682 7f091b191700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09140e5ec0 con 0x7f09140d1860 2021-02-10T11:37:30.701 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:30.682 7f091b191700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09140f0640 con 0x7f09141b0a00 2021-02-10T11:37:30.702 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:30.682 7f0918fad700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f09140d1860 0x7f09141b03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:30.702 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:30.682 7f0918fad700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f09140d1860 0x7f09141b03e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:38696/0 (socket says 172.21.2.102:38696) 2021-02-10T11:37:30.702 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:30.682 7f0918fad700 1 -- 172.21.2.102:0/3586269357 learned_addr learned my addr 172.21.2.102:0/3586269357 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:37:30.702 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:30.682 7f0913fff700 1 --2- 172.21.2.102:0/3586269357 >> v2:172.21.2.113:3300/0 conn(0x7f09141b0a00 0x7f09140cf320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:30.702 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:30.683 7f0918fad700 1 -- 172.21.2.102:0/3586269357 >> v2:172.21.2.113:3300/0 conn(0x7f09141b0a00 msgr2=0x7f09140cf320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:37:30.702 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:30.683 7f0918fad700 1 --2- 172.21.2.102:0/3586269357 >> v2:172.21.2.113:3300/0 conn(0x7f09141b0a00 0x7f09140cf320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:30.702 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:30.683 7f0918fad700 1 -- 172.21.2.102:0/3586269357 >> v1:172.21.2.102:6789/0 conn(0x7f09140d1c60 legacy=0x7f09140cf940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:30.702 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:30.683 7f09137fe700 1 -- 172.21.2.102:0/3586269357 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2394374544 0 0) 0x7f08fc000c10 con 0x7f09140d1c60 2021-02-10T11:37:30.702 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:30.683 7f0918fad700 1 -- 172.21.2.102:0/3586269357 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09140f0830 con 0x7f09140d1860 2021-02-10T11:37:30.702 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:30.683 7f0913fff700 1 --2- 172.21.2.102:0/3586269357 >> v2:172.21.2.113:3300/0 conn(0x7f09141b0a00 0x7f09140cf320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:37:30.702 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:30.683 7f0918fad700 1 --2- 172.21.2.102:0/3586269357 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f09140d1860 0x7f09141b03e0 secure :-1 s=READY pgs=2960 cs=0 l=1 rev1=1 rx=0x7f090c006a80 tx=0x7f090c009f00).ready entity=mon.0 client_cookie=e0596410980f1688 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:30.702 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:30.683 7f09137fe700 1 -- 172.21.2.102:0/3586269357 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f090c00b5c0 con 0x7f09140d1860 2021-02-10T11:37:30.703 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:30.683 7f09137fe700 1 -- 172.21.2.102:0/3586269357 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f090c00b800 con 0x7f09140d1860 2021-02-10T11:37:30.703 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:30.683 7f09137fe700 1 -- 172.21.2.102:0/3586269357 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f090c00bd20 con 0x7f09140d1860 2021-02-10T11:37:30.703 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:30.683 7f091b191700 1 -- 172.21.2.102:0/3586269357 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f09140d1860 msgr2=0x7f09141b03e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:30.703 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:30.683 7f091b191700 1 --2- 172.21.2.102:0/3586269357 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f09140d1860 0x7f09141b03e0 secure :-1 s=READY pgs=2960 cs=0 l=1 rev1=1 rx=0x7f090c006a80 tx=0x7f090c009f00).stop 2021-02-10T11:37:30.703 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:30.684 7f091b191700 1 -- 172.21.2.102:0/3586269357 shutdown_connections 2021-02-10T11:37:30.703 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:30.684 7f091b191700 1 --2- 172.21.2.102:0/3586269357 >> v2:172.21.2.113:3300/0 conn(0x7f09141b0a00 0x7f09140cf320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:30.703 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:30.684 7f091b191700 1 --2- 172.21.2.102:0/3586269357 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f09140d1860 0x7f09141b03e0 unknown :-1 s=CLOSED pgs=2960 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:30.703 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:30.684 7f091b191700 1 -- 172.21.2.102:0/3586269357 >> 172.21.2.102:0/3586269357 conn(0x7f09140ce440 msgr2=0x7f09140d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:30.703 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:30.684 7f091b191700 1 -- 172.21.2.102:0/3586269357 shutdown_connections 2021-02-10T11:37:30.703 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:30.684 7f091b191700 1 -- 172.21.2.102:0/3586269357 wait complete. 2021-02-10T11:37:30.705 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:30.699 7f091b191700 1 Processor -- start 2021-02-10T11:37:30.705 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:30.699 7f091b191700 1 -- start start 2021-02-10T11:37:30.705 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:30.699 7f091b191700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f09141a35c0 0x7f09141a39c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:30.705 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:30.699 7f091b191700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f091409d0e0 0x7f091409d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:30.705 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:30.699 7f091b191700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f091409db00 0x7f091409df00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:30.705 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:30.699 7f091b191700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f091409cb60 con 0x7f09141a35c0 2021-02-10T11:37:30.705 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:30.699 7f091b191700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09140c3730 con 0x7f091409d0e0 2021-02-10T11:37:30.705 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:30.699 7f091b191700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09140c3910 con 0x7f091409db00 2021-02-10T11:37:30.705 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:30.699 7f0918fad700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f09141a35c0 0x7f09141a39c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:30.705 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:30.699 7f0918fad700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f09141a35c0 0x7f09141a39c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:38706/0 (socket says 172.21.2.102:38706) 2021-02-10T11:37:30.705 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:30.699 7f0918fad700 1 -- 172.21.2.102:0/3618343276 learned_addr learned my addr 172.21.2.102:0/3618343276 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:37:30.705 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:30.700 7f09197ae700 1 --2- 172.21.2.102:0/3618343276 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f091409db00 0x7f091409df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:30.706 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:30.700 7f0918fad700 1 -- 172.21.2.102:0/3618343276 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f091409db00 msgr2=0x7f091409df00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:30.706 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:30.700 7f0918fad700 1 --2- 172.21.2.102:0/3618343276 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f091409db00 0x7f091409df00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:30.706 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:30.700 7f0913fff700 1 --2- 172.21.2.102:0/3618343276 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f091409d0e0 0x7f091409d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:30.706 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:30.700 7f0918fad700 1 -- 172.21.2.102:0/3618343276 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f091409d0e0 msgr2=0x7f091409d4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:30.706 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:30.700 7f0918fad700 1 --2- 172.21.2.102:0/3618343276 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f091409d0e0 0x7f091409d4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:30.706 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:30.700 7f0918fad700 1 -- 172.21.2.102:0/3618343276 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0914093e60 con 0x7f09141a35c0 2021-02-10T11:37:30.706 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:30.700 7f0913fff700 1 --2- 172.21.2.102:0/3618343276 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f091409d0e0 0x7f091409d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:37:30.706 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:30.700 7f0918fad700 1 --2- 172.21.2.102:0/3618343276 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f09141a35c0 0x7f09141a39c0 secure :-1 s=READY pgs=2961 cs=0 l=1 rev1=1 rx=0x7f090c00b910 tx=0x7f090c00bcb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:30.706 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:30.700 7f09117fa700 1 -- 172.21.2.102:0/3618343276 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f090c00c2f0 con 0x7f09141a35c0 2021-02-10T11:37:30.706 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:30.700 7f09117fa700 1 -- 172.21.2.102:0/3618343276 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f090c00c4f0 con 0x7f09141a35c0 2021-02-10T11:37:30.706 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:30.700 7f09117fa700 1 -- 172.21.2.102:0/3618343276 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f090c00c8c0 con 0x7f09141a35c0 2021-02-10T11:37:30.706 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:30.700 7f091b191700 1 -- 172.21.2.102:0/3618343276 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0914093e60 con 0x7f09141a35c0 2021-02-10T11:37:30.706 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:30.700 7f091b191700 1 -- 172.21.2.102:0/3618343276 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09140943f0 con 0x7f09141a35c0 2021-02-10T11:37:30.707 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:30.701 7f09117fa700 1 -- 172.21.2.102:0/3618343276 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f090c001010 con 0x7f09141a35c0 2021-02-10T11:37:30.707 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:30.701 7f09117fa700 1 --2- 172.21.2.102:0/3618343276 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f08f80186c0 0x7f08f801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:30.707 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:30.701 7f09117fa700 1 -- 172.21.2.102:0/3618343276 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(167..167 src has 1..167) v4 ==== 4839+0+0 (secure 0 0 0) 0x7f090c032680 con 0x7f09141a35c0 2021-02-10T11:37:30.707 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:30.701 7f0913fff700 1 --2- 172.21.2.102:0/3618343276 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f08f80186c0 0x7f08f801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:30.707 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:30.701 7f0913fff700 1 --2- 172.21.2.102:0/3618343276 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f08f80186c0 0x7f08f801abc0 crc :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:30.733 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:30.731 7f091b191700 1 -- 172.21.2.102:0/3618343276 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09140ebbe0 con 0x7f09141a35c0 2021-02-10T11:37:30.736 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:30.734 7f09117fa700 1 -- 172.21.2.102:0/3618343276 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f090c000d30 con 0x7f09141a35c0 2021-02-10T11:37:30.876 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:30.874 7f091b191700 1 -- 172.21.2.102:0/3618343276 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd blacklist clear"} v 0) v1 -- 0x7f09140f0b00 con 0x7f09141a35c0 2021-02-10T11:37:31.194 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:31.192 7f09117fa700 1 -- 172.21.2.102:0/3618343276 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd blacklist clear"}]=0 removed all blacklist entries v168) v1 ==== 97+0+0 (secure 0 0 0) 0x7f090c001010 con 0x7f09141a35c0 2021-02-10T11:37:31.242 INFO:teuthology.orchestra.run.gibba002.stderr: removed all blacklist entries 2021-02-10T11:37:31.244 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:31.241 7f091b191700 1 -- 172.21.2.102:0/3618343276 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f08f80186c0 msgr2=0x7f08f801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:31.244 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:31.241 7f091b191700 1 --2- 172.21.2.102:0/3618343276 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f08f80186c0 0x7f08f801abc0 crc :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:31.244 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:31.241 7f091b191700 1 -- 172.21.2.102:0/3618343276 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f09141a35c0 msgr2=0x7f09141a39c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:31.244 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:31.241 7f091b191700 1 --2- 172.21.2.102:0/3618343276 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f09141a35c0 0x7f09141a39c0 secure :-1 s=READY pgs=2961 cs=0 l=1 rev1=1 rx=0x7f090c00b910 tx=0x7f090c00bcb0).stop 2021-02-10T11:37:31.244 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:31.242 7f091b191700 1 -- 172.21.2.102:0/3618343276 shutdown_connections 2021-02-10T11:37:31.244 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:31.242 7f091b191700 1 --2- 172.21.2.102:0/3618343276 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f091409db00 0x7f091409df00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:31.244 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:31.242 7f091b191700 1 --2- 172.21.2.102:0/3618343276 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f08f80186c0 0x7f08f801abc0 unknown :-1 s=CLOSED pgs=1297 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:31.244 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:31.242 7f091b191700 1 --2- 172.21.2.102:0/3618343276 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f091409d0e0 0x7f091409d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:31.244 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:31.242 7f091b191700 1 --2- 172.21.2.102:0/3618343276 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f09141a35c0 0x7f09141a39c0 unknown :-1 s=CLOSED pgs=2961 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:31.244 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:31.242 7f091b191700 1 -- 172.21.2.102:0/3618343276 >> 172.21.2.102:0/3618343276 conn(0x7f09140a6b00 msgr2=0x7f0914095700 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:31.244 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:31.242 7f091b191700 1 -- 172.21.2.102:0/3618343276 shutdown_connections 2021-02-10T11:37:31.245 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:31.242 7f091b191700 1 -- 172.21.2.102:0/3618343276 wait complete. 2021-02-10T11:37:31.252 INFO:tasks.cephfs.cephfs_test_case:['0', '1', '2', '3'] 2021-02-10T11:37:31.253 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth ls --format=json-pretty 2021-02-10T11:37:31.683 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:31.664 7f041ed9e700 1 Processor -- start 2021-02-10T11:37:31.683 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:31.664 7f041ed9e700 1 -- start start 2021-02-10T11:37:31.683 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:31.665 7f041ed9e700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f04180d1870 0x7f04181b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:31.683 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:31.665 7f041ed9e700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f04181b0a10 0x7f04180cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:31.683 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:31.665 7f041ed9e700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04180f0b10 con 0x7f04180d1c70 2021-02-10T11:37:31.683 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:31.665 7f041ed9e700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04180fe230 con 0x7f04181b0a10 2021-02-10T11:37:31.683 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:31.665 7f041ed9e700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04180e5ed0 con 0x7f04180d1870 2021-02-10T11:37:31.683 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:31.665 7f0417fff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f04181b0a10 0x7f04180cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:31.683 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:31.665 7f041d3bb700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f04180d1c70 0x7f04180cf950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:59166/0 (socket says 172.21.2.102:59166) 2021-02-10T11:37:31.684 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:31.665 7f041d3bb700 1 -- 172.21.2.102:0/279131364 learned_addr learned my addr 172.21.2.102:0/279131364 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:37:31.684 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:31.665 7f0417fff700 1 -- 172.21.2.102:0/279131364 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f04180d1870 msgr2=0x7f04181b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:37:31.684 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:31.665 7f0417fff700 1 --2- 172.21.2.102:0/279131364 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f04180d1870 0x7f04181b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:31.684 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:31.665 7f04177fe700 1 -- 172.21.2.102:0/279131364 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2833061944 0 0) 0x7f0410001180 con 0x7f04180d1c70 2021-02-10T11:37:31.684 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:31.665 7f0417fff700 1 -- 172.21.2.102:0/279131364 >> v1:172.21.2.102:6789/0 conn(0x7f04180d1c70 legacy=0x7f04180cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:31.684 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:31.665 7f0417fff700 1 -- 172.21.2.102:0/279131364 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04180f0650 con 0x7f04181b0a10 2021-02-10T11:37:31.684 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:31.665 7f0417fff700 1 --2- 172.21.2.102:0/279131364 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f04181b0a10 0x7f04180cf330 secure :-1 s=READY pgs=2963 cs=0 l=1 rev1=1 rx=0x7f04080069d0 tx=0x7f0408009e40).ready entity=mon.0 client_cookie=3fc932cbe527d7b2 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:31.684 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:31.666 7f04177fe700 1 -- 172.21.2.102:0/279131364 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f040800b510 con 0x7f04181b0a10 2021-02-10T11:37:31.684 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:31.666 7f04177fe700 1 -- 172.21.2.102:0/279131364 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f040800b890 con 0x7f04181b0a10 2021-02-10T11:37:31.684 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:31.666 7f04177fe700 1 -- 172.21.2.102:0/279131364 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f040800bdb0 con 0x7f04181b0a10 2021-02-10T11:37:31.684 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:31.666 7f041ed9e700 1 -- 172.21.2.102:0/279131364 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f04181b0a10 msgr2=0x7f04180cf330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:31.684 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:31.666 7f041ed9e700 1 --2- 172.21.2.102:0/279131364 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f04181b0a10 0x7f04180cf330 secure :-1 s=READY pgs=2963 cs=0 l=1 rev1=1 rx=0x7f04080069d0 tx=0x7f0408009e40).stop 2021-02-10T11:37:31.685 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:31.666 7f041ed9e700 1 -- 172.21.2.102:0/279131364 shutdown_connections 2021-02-10T11:37:31.685 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:31.666 7f041ed9e700 1 --2- 172.21.2.102:0/279131364 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f04181b0a10 0x7f04180cf330 unknown :-1 s=CLOSED pgs=2963 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:31.685 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:31.666 7f041ed9e700 1 --2- 172.21.2.102:0/279131364 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f04180d1870 0x7f04181b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:31.685 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:31.666 7f041ed9e700 1 -- 172.21.2.102:0/279131364 >> 172.21.2.102:0/279131364 conn(0x7f04180ce450 msgr2=0x7f04180d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:31.685 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:31.666 7f041ed9e700 1 -- 172.21.2.102:0/279131364 shutdown_connections 2021-02-10T11:37:31.685 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:31.666 7f041ed9e700 1 -- 172.21.2.102:0/279131364 wait complete. 2021-02-10T11:37:31.686 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:31.681 7f041ed9e700 1 Processor -- start 2021-02-10T11:37:31.686 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:31.681 7f041ed9e700 1 -- start start 2021-02-10T11:37:31.686 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:31.681 7f041ed9e700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f04180b0ad0 0x7f0418091f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:31.687 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:31.681 7f041ed9e700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f04181a8850 0x7f04181a8c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:31.687 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:31.681 7f041ed9e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f04181a6fa0 0x7f04181a73a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:31.687 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:31.681 7f041ed9e700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04181abfe0 con 0x7f04181a6fa0 2021-02-10T11:37:31.687 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:31.681 7f041ed9e700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04181a9c40 con 0x7f04180b0ad0 2021-02-10T11:37:31.687 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:31.681 7f041ed9e700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04181a9730 con 0x7f04181a8850 2021-02-10T11:37:31.687 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:31.682 7f041d3bb700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f04181a6fa0 0x7f04181a73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:31.687 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:31.682 7f041d3bb700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f04181a6fa0 0x7f04181a73a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:38716/0 (socket says 172.21.2.102:38716) 2021-02-10T11:37:31.687 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:31.682 7f041d3bb700 1 -- 172.21.2.102:0/922368282 learned_addr learned my addr 172.21.2.102:0/922368282 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:37:31.687 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:31.682 7f041d3bb700 1 -- 172.21.2.102:0/922368282 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f04181a8850 msgr2=0x7f04181a8c50 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:37:31.687 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:31.682 7f041d3bb700 1 --2- 172.21.2.102:0/922368282 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f04181a8850 0x7f04181a8c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:31.687 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:31.682 7f041d3bb700 1 -- 172.21.2.102:0/922368282 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f04180b0ad0 msgr2=0x7f0418091f30 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:37:31.687 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:31.682 7f041d3bb700 1 --2- 172.21.2.102:0/922368282 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f04180b0ad0 0x7f0418091f30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:31.687 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:31.682 7f041d3bb700 1 -- 172.21.2.102:0/922368282 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04181a9270 con 0x7f04181a6fa0 2021-02-10T11:37:31.688 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:31.682 7f041d3bb700 1 --2- 172.21.2.102:0/922368282 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f04181a6fa0 0x7f04181a73a0 secure :-1 s=READY pgs=2964 cs=0 l=1 rev1=1 rx=0x7f0410007fc0 tx=0x7f041000a8f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:31.688 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:31.682 7f04157fa700 1 -- 172.21.2.102:0/922368282 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f041000ace0 con 0x7f04181a6fa0 2021-02-10T11:37:31.688 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:31.682 7f04157fa700 1 -- 172.21.2.102:0/922368282 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0410007030 con 0x7f04181a6fa0 2021-02-10T11:37:31.688 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:31.682 7f041ed9e700 1 -- 172.21.2.102:0/922368282 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04181a9270 con 0x7f04181a6fa0 2021-02-10T11:37:31.688 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:31.682 7f04157fa700 1 -- 172.21.2.102:0/922368282 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0410004620 con 0x7f04181a6fa0 2021-02-10T11:37:31.688 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:31.682 7f041ed9e700 1 -- 172.21.2.102:0/922368282 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04181a7cb0 con 0x7f04181a6fa0 2021-02-10T11:37:31.688 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:31.683 7f04157fa700 1 -- 172.21.2.102:0/922368282 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f04100043a0 con 0x7f04181a6fa0 2021-02-10T11:37:31.688 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:31.683 7f04157fa700 1 --2- 172.21.2.102:0/922368282 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f03fc0186c0 0x7f03fc01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:31.688 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:31.683 7f04157fa700 1 -- 172.21.2.102:0/922368282 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(168..168 src has 1..168) v4 ==== 4753+0+0 (secure 0 0 0) 0x7f0410032260 con 0x7f04181a6fa0 2021-02-10T11:37:31.688 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:31.683 7f041cbba700 1 --2- 172.21.2.102:0/922368282 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f03fc0186c0 0x7f03fc01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:31.688 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:31.683 7f041cbba700 1 --2- 172.21.2.102:0/922368282 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f03fc0186c0 0x7f03fc01abc0 crc :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:31.717 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:31.714 7f041ed9e700 1 -- 172.21.2.102:0/922368282 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04181a9710 con 0x7f04181a6fa0 2021-02-10T11:37:31.719 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:31.717 7f04157fa700 1 -- 172.21.2.102:0/922368282 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f0410030dc0 con 0x7f04181a6fa0 2021-02-10T11:37:31.851 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:31.848 7f041ed9e700 1 -- 172.21.2.102:0/922368282 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f04181a9710 con 0x7f04181a6fa0 2021-02-10T11:37:31.851 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:31.849 7f04157fa700 1 -- 172.21.2.102:0/922368282 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v92) v1 ==== 80+0+4513 (secure 0 0 0) 0x7f04100043a0 con 0x7f04181a6fa0 2021-02-10T11:37:31.851 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:37:31.852 INFO:teuthology.orchestra.run.gibba002.stdout:{ 2021-02-10T11:37:31.852 INFO:teuthology.orchestra.run.gibba002.stdout: "auth_dump": [ 2021-02-10T11:37:31.852 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:37:31.852 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "mds.a", 2021-02-10T11:37:31.852 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAevSNgsgyEARAAvr0cUjjCoEHLxaZSZ2HtVA==", 2021-02-10T11:37:31.852 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:37:31.852 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow", 2021-02-10T11:37:31.852 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:37:31.852 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:37:31.852 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:37:31.852 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:37:31.853 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:37:31.853 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:37:31.853 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "mds.b", 2021-02-10T11:37:31.853 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAevSNgwOFMMRAALfb1fbX56ao1TU8ClnTW1A==", 2021-02-10T11:37:31.853 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:37:31.853 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow", 2021-02-10T11:37:31.853 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:37:31.853 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:37:31.853 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:37:31.853 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:37:31.853 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:37:31.853 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:37:31.854 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.0", 2021-02-10T11:37:31.854 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAmvSNgj+bMMBAAD3HUD6fLiMAg+++iNzLJKQ==", 2021-02-10T11:37:31.854 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:37:31.854 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:37:31.854 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:37:31.854 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:37:31.854 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:37:31.854 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:37:31.854 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:37:31.854 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.1", 2021-02-10T11:37:31.855 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQApvSNgnvc+MRAAAOxqlNwiyLHqNITOrYeFKQ==", 2021-02-10T11:37:31.855 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:37:31.855 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:37:31.855 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:37:31.855 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:37:31.855 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:37:31.855 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:37:31.855 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:37:31.855 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.2", 2021-02-10T11:37:31.855 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAsvSNgZ4SbMRAA76IOYWfbiAuBoM/4k81GxA==", 2021-02-10T11:37:31.856 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:37:31.856 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:37:31.856 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:37:31.856 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:37:31.856 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:37:31.856 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:37:31.856 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:37:31.856 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.3", 2021-02-10T11:37:31.856 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAvvSNgFk3rMRAAWCmKKyw0LDQ+UlAHZAITOw==", 2021-02-10T11:37:31.856 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:37:31.856 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:37:31.856 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:37:31.857 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:37:31.857 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:37:31.857 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:37:31.857 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:37:31.857 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.4", 2021-02-10T11:37:31.857 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQA4vSNgpKcNJBAApJMmvj5YxVHPDOmvmQONOg==", 2021-02-10T11:37:31.857 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:37:31.857 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:37:31.857 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:37:31.857 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:37:31.857 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:37:31.857 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:37:31.858 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:37:31.858 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.5", 2021-02-10T11:37:31.858 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQA7vSNgf2ujJBAAfkuYpyb3WTzc8V5kw+Wktg==", 2021-02-10T11:37:31.858 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:37:31.858 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:37:31.858 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:37:31.858 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:37:31.858 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:37:31.858 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:37:31.858 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:37:31.858 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.6", 2021-02-10T11:37:31.859 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQA+vSNgqw9uJRAAFMU0R+UxU9C3o/9byb3e0g==", 2021-02-10T11:37:31.859 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:37:31.859 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:37:31.859 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:37:31.859 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:37:31.859 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:37:31.859 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:37:31.859 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:37:31.859 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.7", 2021-02-10T11:37:31.859 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQBBvSNgvWSrJRAA2KaCHmLpzovRgYzLvyofTQ==", 2021-02-10T11:37:31.859 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:37:31.859 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:37:31.859 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:37:31.859 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:37:31.860 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:37:31.860 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:37:31.860 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:37:31.860 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "client.0", 2021-02-10T11:37:31.860 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAfvSNgZd9XIBAAvyrvl8fdKuHnJZfczfRoFw==", 2021-02-10T11:37:31.860 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:37:31.860 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow", 2021-02-10T11:37:31.860 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow r", 2021-02-10T11:37:31.860 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:37:31.860 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:37:31.860 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:37:31.860 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:37:31.860 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "client.1", 2021-02-10T11:37:31.861 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAfvSNgSyfXNhAAIduPnF7Vd/6HSkJ0MlcMSA==", 2021-02-10T11:37:31.861 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:37:31.861 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow", 2021-02-10T11:37:31.861 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow r", 2021-02-10T11:37:31.861 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:37:31.861 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:37:31.861 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:37:31.861 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:37:31.861 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "client.2", 2021-02-10T11:37:31.861 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAgvSNgiIG/ERAA9ZTamaTI7ZsPSnxBqjplCg==", 2021-02-10T11:37:31.861 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:37:31.861 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow", 2021-02-10T11:37:31.861 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow r", 2021-02-10T11:37:31.862 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:37:31.862 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:37:31.862 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:37:31.862 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:37:31.862 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "client.3", 2021-02-10T11:37:31.862 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAgvSNgOWpOKBAAlXsEB04Q/GvtL06PqRMLeA==", 2021-02-10T11:37:31.862 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:37:31.862 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow", 2021-02-10T11:37:31.862 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow r", 2021-02-10T11:37:31.862 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:37:31.862 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:37:31.863 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:37:31.863 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:37:31.863 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "client.admin", 2021-02-10T11:37:31.863 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAavSNgO3GQBhAAL+NVnYskQRKVWLgyDPY/xw==", 2021-02-10T11:37:31.863 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:37:31.863 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow *", 2021-02-10T11:37:31.863 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:37:31.863 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:37:31.863 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:37:31.863 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:37:31.863 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:37:31.863 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:37:31.863 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "mgr.x", 2021-02-10T11:37:31.864 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAdvSNgXZZgJhAA7dtBSGzH+DsBwnddZpOBJA==", 2021-02-10T11:37:31.864 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:37:31.864 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow *", 2021-02-10T11:37:31.864 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow profile mgr", 2021-02-10T11:37:31.864 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:37:31.864 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:37:31.864 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:37:31.864 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:37:31.864 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "mgr.y", 2021-02-10T11:37:31.864 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAdvSNgVqQFDRAAwo6YEdWZCAr5GdOWVL/Qvg==", 2021-02-10T11:37:31.864 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:37:31.864 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow *", 2021-02-10T11:37:31.864 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow profile mgr", 2021-02-10T11:37:31.865 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:37:31.865 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:37:31.865 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:37:31.865 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:37:31.865 INFO:teuthology.orchestra.run.gibba002.stdout:} 2021-02-10T11:37:31.866 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:31.851 7f041ed9e700 1 -- 172.21.2.102:0/922368282 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f03fc0186c0 msgr2=0x7f03fc01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:31.866 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:31.851 7f041ed9e700 1 --2- 172.21.2.102:0/922368282 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f03fc0186c0 0x7f03fc01abc0 crc :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:31.866 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:31.851 7f041ed9e700 1 -- 172.21.2.102:0/922368282 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f04181a6fa0 msgr2=0x7f04181a73a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:31.866 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:31.851 7f041ed9e700 1 --2- 172.21.2.102:0/922368282 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f04181a6fa0 0x7f04181a73a0 secure :-1 s=READY pgs=2964 cs=0 l=1 rev1=1 rx=0x7f0410007fc0 tx=0x7f041000a8f0).stop 2021-02-10T11:37:31.866 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:31.851 7f041ed9e700 1 -- 172.21.2.102:0/922368282 shutdown_connections 2021-02-10T11:37:31.866 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:31.851 7f041ed9e700 1 --2- 172.21.2.102:0/922368282 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f04181a6fa0 0x7f04181a73a0 unknown :-1 s=CLOSED pgs=2964 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:31.866 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:31.851 7f041ed9e700 1 --2- 172.21.2.102:0/922368282 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f04181a8850 0x7f04181a8c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:31.866 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:31.851 7f041ed9e700 1 --2- 172.21.2.102:0/922368282 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f03fc0186c0 0x7f03fc01abc0 unknown :-1 s=CLOSED pgs=1298 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:31.867 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:31.851 7f041ed9e700 1 --2- 172.21.2.102:0/922368282 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f04180b0ad0 0x7f0418091f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:31.867 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:31.851 7f041ed9e700 1 -- 172.21.2.102:0/922368282 >> 172.21.2.102:0/922368282 conn(0x7f041819f8f0 msgr2=0x7f04181a5f80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:31.867 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:31.851 7f041ed9e700 1 -- 172.21.2.102:0/922368282 shutdown_connections 2021-02-10T11:37:31.867 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:31.851 7f041ed9e700 1 -- 172.21.2.102:0/922368282 wait complete. 2021-02-10T11:37:31.867 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:37:32.297 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:32.279 7fb839f4f700 1 Processor -- start 2021-02-10T11:37:32.298 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:32.279 7fb839f4f700 1 -- start start 2021-02-10T11:37:32.298 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:32.280 7fb839f4f700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fb8340d0340 0x7fb8340ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:32.298 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:32.280 7fb839f4f700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb8340d1c70 0x7fb8340efc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:32.298 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:32.280 7fb839f4f700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8340f0b10 con 0x7fb8340d1870 2021-02-10T11:37:32.298 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:32.280 7fb839f4f700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8340e5ed0 con 0x7fb8340d0340 2021-02-10T11:37:32.298 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:32.280 7fb839f4f700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8341b0be0 con 0x7fb8340d1c70 2021-02-10T11:37:32.298 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:32.280 7fb832ffd700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fb8340d0340 0x7fb8340ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:32.298 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:32.280 7fb832ffd700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fb8340d0340 0x7fb8340ef640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:38724/0 (socket says 172.21.2.102:38724) 2021-02-10T11:37:32.298 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:32.280 7fb832ffd700 1 -- 172.21.2.102:0/1085162605 learned_addr learned my addr 172.21.2.102:0/1085162605 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:37:32.298 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:32.280 7fb832ffd700 1 -- 172.21.2.102:0/1085162605 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb8340d1c70 msgr2=0x7fb8340efc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:37:32.299 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:32.280 7fb832ffd700 1 --2- 172.21.2.102:0/1085162605 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb8340d1c70 0x7fb8340efc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:32.299 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:32.280 7fb832ffd700 1 -- 172.21.2.102:0/1085162605 >> v1:172.21.2.113:6789/0 conn(0x7fb8340d1870 legacy=0x7fb8341b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:37:32.299 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:32.280 7fb832ffd700 1 -- 172.21.2.102:0/1085162605 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8340f20a0 con 0x7fb8340d0340 2021-02-10T11:37:32.299 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:32.280 7fb832ffd700 1 --2- 172.21.2.102:0/1085162605 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb8340d0340 0x7fb8340ef640 secure :-1 s=READY pgs=2965 cs=0 l=1 rev1=1 rx=0x7fb81c006700 tx=0x7fb81c009b80).ready entity=mon.0 client_cookie=dce308e80732be20 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:32.299 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:32.280 7fb8327fc700 1 -- 172.21.2.102:0/1085162605 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb81c00c0c0 con 0x7fb8340d0340 2021-02-10T11:37:32.299 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:32.280 7fb8327fc700 1 -- 172.21.2.102:0/1085162605 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb81c00c3c0 con 0x7fb8340d0340 2021-02-10T11:37:32.299 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:32.280 7fb8327fc700 1 -- 172.21.2.102:0/1085162605 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb81c00c820 con 0x7fb8340d0340 2021-02-10T11:37:32.299 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:32.280 7fb839f4f700 1 -- 172.21.2.102:0/1085162605 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb8340d0340 msgr2=0x7fb8340ef640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:32.299 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:32.280 7fb839f4f700 1 --2- 172.21.2.102:0/1085162605 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb8340d0340 0x7fb8340ef640 secure :-1 s=READY pgs=2965 cs=0 l=1 rev1=1 rx=0x7fb81c006700 tx=0x7fb81c009b80).stop 2021-02-10T11:37:32.299 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:32.281 7fb839f4f700 1 -- 172.21.2.102:0/1085162605 shutdown_connections 2021-02-10T11:37:32.299 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:32.281 7fb839f4f700 1 --2- 172.21.2.102:0/1085162605 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb8340d1c70 0x7fb8340efc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:32.299 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:32.281 7fb839f4f700 1 --2- 172.21.2.102:0/1085162605 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb8340d0340 0x7fb8340ef640 unknown :-1 s=CLOSED pgs=2965 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:32.299 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:32.281 7fb839f4f700 1 -- 172.21.2.102:0/1085162605 >> 172.21.2.102:0/1085162605 conn(0x7fb8340ce450 msgr2=0x7fb8340d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:32.300 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:32.281 7fb839f4f700 1 -- 172.21.2.102:0/1085162605 shutdown_connections 2021-02-10T11:37:32.300 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:32.281 7fb839f4f700 1 -- 172.21.2.102:0/1085162605 wait complete. 2021-02-10T11:37:32.301 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:32.296 7fb839f4f700 1 Processor -- start 2021-02-10T11:37:32.301 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:32.296 7fb839f4f700 1 -- start start 2021-02-10T11:37:32.301 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:32.296 7fb839f4f700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb8340b4f30 0x7fb83418b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:32.301 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:32.296 7fb839f4f700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb834093f40 0x7fb834094340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:32.302 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:32.296 7fb839f4f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb834094960 0x7fb834187e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:32.302 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:32.296 7fb839f4f700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb834094e90 con 0x7fb834094960 2021-02-10T11:37:32.302 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:32.296 7fb839f4f700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb83418bff0 con 0x7fb834093f40 2021-02-10T11:37:32.302 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:32.296 7fb839f4f700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb83408f780 con 0x7fb8340b4f30 2021-02-10T11:37:32.302 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:32.297 7fb833fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb834094960 0x7fb834187e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:32.302 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:32.297 7fb833fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb834094960 0x7fb834187e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:38728/0 (socket says 172.21.2.102:38728) 2021-02-10T11:37:32.302 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:32.297 7fb833fff700 1 -- 172.21.2.102:0/3115884408 learned_addr learned my addr 172.21.2.102:0/3115884408 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:37:32.302 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:32.297 7fb832ffd700 1 --2- 172.21.2.102:0/3115884408 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb834093f40 0x7fb834094340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:32.302 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:32.297 7fb8337fe700 1 --2- 172.21.2.102:0/3115884408 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb8340b4f30 0x7fb83418b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:32.302 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:32.297 7fb832ffd700 1 -- 172.21.2.102:0/3115884408 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb8340b4f30 msgr2=0x7fb83418b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:32.302 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:32.297 7fb832ffd700 1 --2- 172.21.2.102:0/3115884408 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb8340b4f30 0x7fb83418b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:32.303 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:32.297 7fb832ffd700 1 -- 172.21.2.102:0/3115884408 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb834094960 msgr2=0x7fb834187e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:32.303 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:32.297 7fb832ffd700 1 --2- 172.21.2.102:0/3115884408 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb834094960 0x7fb834187e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:32.303 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:32.297 7fb832ffd700 1 -- 172.21.2.102:0/3115884408 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb834188470 con 0x7fb834093f40 2021-02-10T11:37:32.303 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:32.297 7fb833fff700 1 --2- 172.21.2.102:0/3115884408 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb834094960 0x7fb834187e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:37:32.303 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:32.297 7fb832ffd700 1 --2- 172.21.2.102:0/3115884408 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb834093f40 0x7fb834094340 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 rx=0x7fb81c0096d0 tx=0x7fb81c00b8c0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:32.303 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:32.297 7fb82bfff700 1 -- 172.21.2.102:0/3115884408 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb81c002030 con 0x7fb834093f40 2021-02-10T11:37:32.303 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:32.297 7fb82bfff700 1 -- 172.21.2.102:0/3115884408 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb81c006390 con 0x7fb834093f40 2021-02-10T11:37:32.303 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:32.297 7fb839f4f700 1 -- 172.21.2.102:0/3115884408 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb834188470 con 0x7fb834093f40 2021-02-10T11:37:32.303 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:32.297 7fb839f4f700 1 -- 172.21.2.102:0/3115884408 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb834188470 con 0x7fb834093f40 2021-02-10T11:37:32.303 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:32.297 7fb82bfff700 1 -- 172.21.2.102:0/3115884408 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb81c002630 con 0x7fb834093f40 2021-02-10T11:37:32.303 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:32.298 7fb82bfff700 1 -- 172.21.2.102:0/3115884408 <== mon.1 v2:172.21.2.113:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fb81c0052e0 con 0x7fb834093f40 2021-02-10T11:37:32.303 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:32.298 7fb82bfff700 1 --2- 172.21.2.102:0/3115884408 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb8180186c0 0x7fb81801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:32.303 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:32.298 7fb82bfff700 1 -- 172.21.2.102:0/3115884408 <== mon.1 v2:172.21.2.113:3300/0 5 ==== osd_map(168..168 src has 1..168) v4 ==== 4753+0+0 (secure 0 0 0) 0x7fb81c0323f0 con 0x7fb834093f40 2021-02-10T11:37:32.304 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:32.298 7fb8337fe700 1 --2- 172.21.2.102:0/3115884408 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb8180186c0 0x7fb81801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:32.304 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:32.298 7fb8337fe700 1 --2- 172.21.2.102:0/3115884408 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb8180186c0 0x7fb81801abc0 crc :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:32.331 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:32.329 7fb839f4f700 1 -- 172.21.2.102:0/3115884408 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb83418bff0 con 0x7fb834093f40 2021-02-10T11:37:32.335 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:32.333 7fb82bfff700 1 -- 172.21.2.102:0/3115884408 <== mon.1 v2:172.21.2.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fb81c000c90 con 0x7fb834093f40 2021-02-10T11:37:32.462 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:32.460 7fb839f4f700 1 -- 172.21.2.102:0/3115884408 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb83418bff0 con 0x7fb834093f40 2021-02-10T11:37:32.463 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:32.462 7fb82bfff700 1 -- 172.21.2.102:0/3115884408 <== mon.1 v2:172.21.2.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v168) v1 ==== 74+0+11154 (secure 0 0 0) 0x7fb81c000f20 con 0x7fb834093f40 2021-02-10T11:37:32.466 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:37:32.466 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":168,"fsid":"68db30ea-dbce-4aef-a6df-f01c6fea0513","created":"2021-02-10 11:03:06.331557","modified":"2021-02-10 11:37:31.187610","last_up_change":"2021-02-10 11:03:32.690562","last_in_change":"2021-02-10 11:03:32.690562","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":35,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:03:37.322519","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"be691b4d-a039-408a-af80-9d53f02b6f1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6818","nonce":36088},{"type":"v1","addr":"172.21.2.102:6820","nonce":36088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6822","nonce":36088},{"type":"v1","addr":"172.21.2.102:6824","nonce":36088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6830","nonce":36088},{"type":"v1","addr":"172.21.2.102:6832","nonce":36088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6826","nonce":36088},{"type":"v1","addr":"172.21.2.102:6828","nonce":36088}]},"public_addr":"172.21.2.102:6820/36088","cluster_addr":"172.21.2.102:6824/36088","heartbeat_back_addr":"172.21.2.102:6832/36088","heartbeat_front_addr":"172.21.2.102:6828/36088","state":["exists","up"]},{"osd":1,"uuid":"64e3ccc8-9c69-4d7c-865e-af8e83822635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6819","nonce":36087},{"type":"v1","addr":"172.21.2.102:6821","nonce":36087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6823","nonce":36087},{"type":"v1","addr":"172.21.2.102:6825","nonce":36087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6831","nonce":36087},{"type":"v1","addr":"172.21.2.102:6833","nonce":36087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6827","nonce":36087},{"type":"v1","addr":"172.21.2.102:6829","nonce":36087}]},"public_addr":"172.21.2.102:6821/36087","cluster_addr":"172.21.2.102:6825/36087","heartbeat_back_addr":"172.21.2.102:6833/36087","heartbeat_front_addr":"172.21.2.102:6829/36087","state":["exists","up"]},{"osd":2,"uuid":"153700af-747f-481f-bdf4-d6072d3a842f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6810","nonce":36089},{"type":"v1","addr":"172.21.2.102:6811","nonce":36089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6812","nonce":36089},{"type":"v1","addr":"172.21.2.102:6813","nonce":36089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6816","nonce":36089},{"type":"v1","addr":"172.21.2.102:6817","nonce":36089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6814","nonce":36089},{"type":"v1","addr":"172.21.2.102:6815","nonce":36089}]},"public_addr":"172.21.2.102:6811/36089","cluster_addr":"172.21.2.102:6813/36089","heartbeat_back_addr":"172.21.2.102:6817/36089","heartbeat_front_addr":"172.21.2.102:6815/36089","state":["exists","up"]},{"osd":3,"uuid":"cbc678f4-83a9-45e6-8830-2f7c232b1ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6802","nonce":36090},{"type":"v1","addr":"172.21.2.102:6803","nonce":36090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6804","nonce":36090},{"type":"v1","addr":"172.21.2.102:6805","nonce":36090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6808","nonce":36090},{"type":"v1","addr":"172.21.2.102:6809","nonce":36090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6806","nonce":36090},{"type":"v1","addr":"172.21.2.102:6807","nonce":36090}]},"public_addr":"172.21.2.102:6803/36090","cluster_addr":"172.21.2.102:6805/36090","heartbeat_back_addr":"172.21.2.102:6809/36090","heartbeat_front_addr":"172.21.2.102:6807/36090","state":["exists","up"]},{"osd":4,"uuid":"c01e4008-b258-4f10-afd2-9e7f37214351","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6808","nonce":36368},{"type":"v1","addr":"172.21.2.113:6811","nonce":36368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6814","nonce":36368},{"type":"v1","addr":"172.21.2.113:6818","nonce":36368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6826","nonce":36368},{"type":"v1","addr":"172.21.2.113:6828","nonce":36368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6820","nonce":36368},{"type":"v1","addr":"172.21.2.113:6824","nonce":36368}]},"public_addr":"172.21.2.113:6811/36368","cluster_addr":"172.21.2.113:6818/36368","heartbeat_back_addr":"172.21.2.113:6828/36368","heartbeat_front_addr":"172.21.2.113:6824/36368","state":["exists","up"]},{"osd":5,"uuid":"551f6007-daf8-4a98-8ee0-c2b16bcccb5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6802","nonce":36366},{"type":"v1","addr":"172.21.2.113:6805","nonce":36366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6807","nonce":36366},{"type":"v1","addr":"172.21.2.113:6810","nonce":36366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6819","nonce":36366},{"type":"v1","addr":"172.21.2.113:6822","nonce":36366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6812","nonce":36366},{"type":"v1","addr":"172.21.2.113:6815","nonce":36366}]},"public_addr":"172.21.2.113:6805/36366","cluster_addr":"172.21.2.113:6810/36366","heartbeat_back_addr":"172.21.2.113:6822/36366","heartbeat_front_addr":"172.21.2.113:6815/36366","state":["exists","up"]},{"osd":6,"uuid":"bf67c040-0e50-48be-b102-c3943bc9ebef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6800","nonce":36367},{"type":"v1","addr":"172.21.2.113:6801","nonce":36367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6803","nonce":36367},{"type":"v1","addr":"172.21.2.113:6804","nonce":36367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6813","nonce":36367},{"type":"v1","addr":"172.21.2.113:6816","nonce":36367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6806","nonce":36367},{"type":"v1","addr":"172.21.2.113:6809","nonce":36367}]},"public_addr":"172.21.2.113:6801/36367","cluster_addr":"172.21.2.113:6804/36367","heartbeat_back_addr":"172.21.2.113:6816/36367","heartbeat_front_addr":"172.21.2.113:6809/36367","state":["exists","up"]},{"osd":7,"uuid":"f5461f80-018d-428f-be89-2eb2bcbe7bb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":159,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6817","nonce":36365},{"type":"v1","addr":"172.21.2.113:6821","nonce":36365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6823","nonce":36365},{"type":"v1","addr":"172.21.2.113:6825","nonce":36365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6830","nonce":36365},{"type":"v1","addr":"172.21.2.113:6831","nonce":36365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6827","nonce":36365},{"type":"v1","addr":"172.21.2.113:6829","nonce":36365}]},"public_addr":"172.21.2.113:6821/36365","cluster_addr":"172.21.2.113:6825/36365","heartbeat_back_addr":"172.21.2.113:6831/36365","heartbeat_front_addr":"172.21.2.113:6829/36365","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:37:32.467 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:32.465 7fb839f4f700 1 -- 172.21.2.102:0/3115884408 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb8180186c0 msgr2=0x7fb81801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:32.467 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:32.465 7fb839f4f700 1 --2- 172.21.2.102:0/3115884408 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb8180186c0 0x7fb81801abc0 crc :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:32.467 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:32.465 7fb839f4f700 1 -- 172.21.2.102:0/3115884408 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb834093f40 msgr2=0x7fb834094340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:32.467 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:32.465 7fb839f4f700 1 --2- 172.21.2.102:0/3115884408 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb834093f40 0x7fb834094340 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 rx=0x7fb81c0096d0 tx=0x7fb81c00b8c0).stop 2021-02-10T11:37:32.467 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:32.465 7fb839f4f700 1 -- 172.21.2.102:0/3115884408 shutdown_connections 2021-02-10T11:37:32.467 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:32.465 7fb839f4f700 1 --2- 172.21.2.102:0/3115884408 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb834094960 0x7fb834187e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:32.467 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:32.465 7fb839f4f700 1 --2- 172.21.2.102:0/3115884408 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb8180186c0 0x7fb81801abc0 unknown :-1 s=CLOSED pgs=1299 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:32.467 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:32.465 7fb839f4f700 1 --2- 172.21.2.102:0/3115884408 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb834093f40 0x7fb834094340 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:32.467 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:32.465 7fb839f4f700 1 --2- 172.21.2.102:0/3115884408 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb8340b4f30 0x7fb83418b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:32.468 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:32.465 7fb839f4f700 1 -- 172.21.2.102:0/3115884408 >> 172.21.2.102:0/3115884408 conn(0x7fb83419f8f0 msgr2=0x7fb8341a5f80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:32.468 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:32.465 7fb839f4f700 1 -- 172.21.2.102:0/3115884408 shutdown_connections 2021-02-10T11:37:32.468 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:32.465 7fb839f4f700 1 -- 172.21.2.102:0/3115884408 wait complete. 2021-02-10T11:37:32.476 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:03:37.322519', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2021-02-10T11:37:32.476 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:37:32.910 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:32.891 7fd1c46ac700 1 Processor -- start 2021-02-10T11:37:32.910 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:32.891 7fd1c46ac700 1 -- start start 2021-02-10T11:37:32.910 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:32.892 7fd1c46ac700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fd1bc0d1870 0x7fd1bc1b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:32.911 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:32.892 7fd1c46ac700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd1bc0cff70 0x7fd1bc109130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:32.911 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:32.892 7fd1c46ac700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1bc109a60 con 0x7fd1bc1b0ae0 2021-02-10T11:37:32.911 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:32.892 7fd1c46ac700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1bc109750 con 0x7fd1bc0d1870 2021-02-10T11:37:32.911 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:32.892 7fd1c46ac700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1bc088070 con 0x7fd1bc0cff70 2021-02-10T11:37:32.911 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:32.892 7fd1c24c8700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fd1bc0d1870 0x7fd1bc1b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:32.911 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:32.892 7fd1c24c8700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fd1bc0d1870 0x7fd1bc1b04c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:38736/0 (socket says 172.21.2.102:38736) 2021-02-10T11:37:32.911 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:32.892 7fd1c24c8700 1 -- 172.21.2.102:0/2487392707 learned_addr learned my addr 172.21.2.102:0/2487392707 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:37:32.911 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:32.892 7fd1c24c8700 1 -- 172.21.2.102:0/2487392707 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd1bc0cff70 msgr2=0x7fd1bc109130 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:37:32.911 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:32.892 7fd1c24c8700 1 --2- 172.21.2.102:0/2487392707 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd1bc0cff70 0x7fd1bc109130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:32.911 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:32.892 7fd1c24c8700 1 -- 172.21.2.102:0/2487392707 >> v1:172.21.2.113:6789/0 conn(0x7fd1bc1b0ae0 legacy=0x7fd1bc108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:37:32.911 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:32.892 7fd1c24c8700 1 -- 172.21.2.102:0/2487392707 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1bc0d1c70 con 0x7fd1bc0d1870 2021-02-10T11:37:32.911 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:32.892 7fd1c24c8700 1 --2- 172.21.2.102:0/2487392707 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd1bc0d1870 0x7fd1bc1b04c0 secure :-1 s=READY pgs=2966 cs=0 l=1 rev1=1 rx=0x7fd1b8006700 tx=0x7fd1b8009b80).ready entity=mon.0 client_cookie=c459f122c867b55e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:32.912 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:32.893 7fd1c14c6700 1 -- 172.21.2.102:0/2487392707 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd1b800c0c0 con 0x7fd1bc0d1870 2021-02-10T11:37:32.912 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:32.893 7fd1c14c6700 1 -- 172.21.2.102:0/2487392707 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1b800c440 con 0x7fd1bc0d1870 2021-02-10T11:37:32.912 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:32.893 7fd1c14c6700 1 -- 172.21.2.102:0/2487392707 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd1b800c8a0 con 0x7fd1bc0d1870 2021-02-10T11:37:32.912 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:32.893 7fd1c46ac700 1 -- 172.21.2.102:0/2487392707 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd1bc0d1870 msgr2=0x7fd1bc1b04c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:32.912 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:32.893 7fd1c46ac700 1 --2- 172.21.2.102:0/2487392707 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd1bc0d1870 0x7fd1bc1b04c0 secure :-1 s=READY pgs=2966 cs=0 l=1 rev1=1 rx=0x7fd1b8006700 tx=0x7fd1b8009b80).stop 2021-02-10T11:37:32.912 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:32.893 7fd1c46ac700 1 -- 172.21.2.102:0/2487392707 shutdown_connections 2021-02-10T11:37:32.912 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:32.893 7fd1c46ac700 1 --2- 172.21.2.102:0/2487392707 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd1bc0cff70 0x7fd1bc109130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:32.912 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:32.893 7fd1c46ac700 1 --2- 172.21.2.102:0/2487392707 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd1bc0d1870 0x7fd1bc1b04c0 unknown :-1 s=CLOSED pgs=2966 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:32.912 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:32.893 7fd1c46ac700 1 -- 172.21.2.102:0/2487392707 >> 172.21.2.102:0/2487392707 conn(0x7fd1bc0ce420 msgr2=0x7fd1bc0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:32.912 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:32.894 7fd1c46ac700 1 -- 172.21.2.102:0/2487392707 shutdown_connections 2021-02-10T11:37:32.912 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:32.894 7fd1c46ac700 1 -- 172.21.2.102:0/2487392707 wait complete. 2021-02-10T11:37:32.912 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:32.908 7fd1c46ac700 1 Processor -- start 2021-02-10T11:37:32.912 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:32.908 7fd1c46ac700 1 -- start start 2021-02-10T11:37:32.914 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:32.908 7fd1c46ac700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd1bc0c3820 0x7fd1bc084cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:32.914 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:32.908 7fd1c46ac700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd1bc0a0290 0x7fd1bc0852d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:32.914 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:32.908 7fd1c46ac700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd1bc086270 0x7fd1bc086670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:32.914 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:32.908 7fd1c46ac700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1bc0c0700 con 0x7fd1bc0c3820 2021-02-10T11:37:32.914 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:32.908 7fd1c46ac700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1bc0858f0 con 0x7fd1bc086270 2021-02-10T11:37:32.914 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:32.908 7fd1c46ac700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1bc085b10 con 0x7fd1bc0a0290 2021-02-10T11:37:32.914 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:32.909 7fd1c24c8700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd1bc0c3820 0x7fd1bc084cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:32.914 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:32.909 7fd1c24c8700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd1bc0c3820 0x7fd1bc084cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:38740/0 (socket says 172.21.2.102:38740) 2021-02-10T11:37:32.914 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:32.909 7fd1c24c8700 1 -- 172.21.2.102:0/2764026712 learned_addr learned my addr 172.21.2.102:0/2764026712 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:37:32.915 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:32.909 7fd1c24c8700 1 -- 172.21.2.102:0/2764026712 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd1bc0a0290 msgr2=0x7fd1bc0852d0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:37:32.915 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:32.909 7fd1c24c8700 1 --2- 172.21.2.102:0/2764026712 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd1bc0a0290 0x7fd1bc0852d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:32.915 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:32.909 7fd1c24c8700 1 -- 172.21.2.102:0/2764026712 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd1bc086270 msgr2=0x7fd1bc086670 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:37:32.915 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:32.909 7fd1c24c8700 1 --2- 172.21.2.102:0/2764026712 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd1bc086270 0x7fd1bc086670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:32.915 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:32.909 7fd1c24c8700 1 -- 172.21.2.102:0/2764026712 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1bc086c90 con 0x7fd1bc0c3820 2021-02-10T11:37:32.915 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:32.909 7fd1c24c8700 1 --2- 172.21.2.102:0/2764026712 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd1bc0c3820 0x7fd1bc084cb0 secure :-1 s=READY pgs=2967 cs=0 l=1 rev1=1 rx=0x7fd1b800b5e0 tx=0x7fd1b800b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:32.915 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:32.909 7fd1aeffd700 1 -- 172.21.2.102:0/2764026712 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd1b800bd70 con 0x7fd1bc0c3820 2021-02-10T11:37:32.915 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:32.909 7fd1aeffd700 1 -- 172.21.2.102:0/2764026712 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1b8002030 con 0x7fd1bc0c3820 2021-02-10T11:37:32.915 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:32.909 7fd1c46ac700 1 -- 172.21.2.102:0/2764026712 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1bc086c90 con 0x7fd1bc0c3820 2021-02-10T11:37:32.915 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:32.909 7fd1c46ac700 1 -- 172.21.2.102:0/2764026712 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1bc1a2f00 con 0x7fd1bc0c3820 2021-02-10T11:37:32.915 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:32.909 7fd1aeffd700 1 -- 172.21.2.102:0/2764026712 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd1b8002550 con 0x7fd1bc0c3820 2021-02-10T11:37:32.915 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:32.910 7fd1aeffd700 1 -- 172.21.2.102:0/2764026712 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd1b8000c90 con 0x7fd1bc0c3820 2021-02-10T11:37:32.915 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:32.910 7fd1aeffd700 1 --2- 172.21.2.102:0/2764026712 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd1a4018730 0x7fd1a401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:32.915 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:32.910 7fd1aeffd700 1 -- 172.21.2.102:0/2764026712 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(168..168 src has 1..168) v4 ==== 4753+0+0 (secure 0 0 0) 0x7fd1b80324e0 con 0x7fd1bc0c3820 2021-02-10T11:37:32.916 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:32.910 7fd1c1cc7700 1 --2- 172.21.2.102:0/2764026712 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd1a4018730 0x7fd1a401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:32.916 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:32.910 7fd1c1cc7700 1 --2- 172.21.2.102:0/2764026712 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd1a4018730 0x7fd1a401ac10 crc :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:32.943 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:32.941 7fd1c46ac700 1 -- 172.21.2.102:0/2764026712 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1bc082a70 con 0x7fd1bc0c3820 2021-02-10T11:37:32.946 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:32.944 7fd1aeffd700 1 -- 172.21.2.102:0/2764026712 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd1b8000940 con 0x7fd1bc0c3820 2021-02-10T11:37:33.073 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:33.070 7fd1c46ac700 1 -- 172.21.2.102:0/2764026712 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7fd1bc082a70 con 0x7fd1bc0c3820 2021-02-10T11:37:33.073 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:33.071 7fd1aeffd700 1 -- 172.21.2.102:0/2764026712 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v168) v1 ==== 92+0+10 (secure 0 0 0) 0x7fd1b8000c90 con 0x7fd1bc0c3820 2021-02-10T11:37:33.073 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:37:33.075 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:33.073 7fd1c46ac700 1 -- 172.21.2.102:0/2764026712 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd1a4018730 msgr2=0x7fd1a401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:33.075 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:33.073 7fd1c46ac700 1 --2- 172.21.2.102:0/2764026712 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd1a4018730 0x7fd1a401ac10 crc :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:33.075 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:33.073 7fd1c46ac700 1 -- 172.21.2.102:0/2764026712 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd1bc0c3820 msgr2=0x7fd1bc084cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:33.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:33.073 7fd1c46ac700 1 --2- 172.21.2.102:0/2764026712 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd1bc0c3820 0x7fd1bc084cb0 secure :-1 s=READY pgs=2967 cs=0 l=1 rev1=1 rx=0x7fd1b800b5e0 tx=0x7fd1b800b980).stop 2021-02-10T11:37:33.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:33.073 7fd1c46ac700 1 -- 172.21.2.102:0/2764026712 shutdown_connections 2021-02-10T11:37:33.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:33.073 7fd1c46ac700 1 --2- 172.21.2.102:0/2764026712 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd1a4018730 0x7fd1a401ac10 unknown :-1 s=CLOSED pgs=1300 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:33.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:33.073 7fd1c46ac700 1 --2- 172.21.2.102:0/2764026712 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd1bc086270 0x7fd1bc086670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:33.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:33.073 7fd1c46ac700 1 --2- 172.21.2.102:0/2764026712 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd1bc0a0290 0x7fd1bc0852d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:33.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:33.073 7fd1c46ac700 1 --2- 172.21.2.102:0/2764026712 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd1bc0c3820 0x7fd1bc084cb0 unknown :-1 s=CLOSED pgs=2967 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:33.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:33.073 7fd1c46ac700 1 -- 172.21.2.102:0/2764026712 >> 172.21.2.102:0/2764026712 conn(0x7fd1bc0872c0 msgr2=0x7fd1bc095700 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:33.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:33.073 7fd1c46ac700 1 -- 172.21.2.102:0/2764026712 shutdown_connections 2021-02-10T11:37:33.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:33.073 7fd1c46ac700 1 -- 172.21.2.102:0/2764026712 wait complete. 2021-02-10T11:37:33.085 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --format=json-pretty osd lspools 2021-02-10T11:37:33.513 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:33.494 7f31dc4f9700 1 Processor -- start 2021-02-10T11:37:33.513 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:33.494 7f31dc4f9700 1 -- start start 2021-02-10T11:37:33.513 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:33.495 7f31dc4f9700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f31d40d0340 0x7f31d40ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:33.514 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:33.495 7f31dc4f9700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f31d40d1c70 0x7f31d40efc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:33.514 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:33.495 7f31dc4f9700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31d40f0b10 con 0x7f31d40d1870 2021-02-10T11:37:33.514 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:33.495 7f31dc4f9700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31d40e5ed0 con 0x7f31d40d0340 2021-02-10T11:37:33.514 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:33.495 7f31dc4f9700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31d41b0be0 con 0x7f31d40d1c70 2021-02-10T11:37:33.514 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:33.495 7f31d9b14700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f31d40d0340 0x7f31d40ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:33.514 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:33.495 7f31d9b14700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f31d40d0340 0x7f31d40ef640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:38748/0 (socket says 172.21.2.102:38748) 2021-02-10T11:37:33.514 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:33.495 7f31d9b14700 1 -- 172.21.2.102:0/3276315038 learned_addr learned my addr 172.21.2.102:0/3276315038 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:37:33.514 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:33.495 7f31d9b14700 1 -- 172.21.2.102:0/3276315038 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f31d40d1c70 msgr2=0x7f31d40efc60 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:37:33.514 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:33.495 7f31d9b14700 1 --2- 172.21.2.102:0/3276315038 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f31d40d1c70 0x7f31d40efc60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:33.514 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:33.495 7f31d9b14700 1 -- 172.21.2.102:0/3276315038 >> v1:172.21.2.113:6789/0 conn(0x7f31d40d1870 legacy=0x7f31d41b03f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:37:33.514 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:33.495 7f31d9b14700 1 -- 172.21.2.102:0/3276315038 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31d40f20a0 con 0x7f31d40d0340 2021-02-10T11:37:33.514 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:33.496 7f31d9b14700 1 --2- 172.21.2.102:0/3276315038 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f31d40d0340 0x7f31d40ef640 secure :-1 s=READY pgs=2968 cs=0 l=1 rev1=1 rx=0x7f31d0006700 tx=0x7f31d0009b80).ready entity=mon.0 client_cookie=4105c4eb3f707639 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:33.515 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:33.496 7f31d9313700 1 -- 172.21.2.102:0/3276315038 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f31d000c0c0 con 0x7f31d40d0340 2021-02-10T11:37:33.515 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:33.496 7f31d9313700 1 -- 172.21.2.102:0/3276315038 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31d000c400 con 0x7f31d40d0340 2021-02-10T11:37:33.515 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:33.496 7f31d9313700 1 -- 172.21.2.102:0/3276315038 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f31d000c8e0 con 0x7f31d40d0340 2021-02-10T11:37:33.515 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:33.496 7f31dc4f9700 1 -- 172.21.2.102:0/3276315038 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f31d40d0340 msgr2=0x7f31d40ef640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:33.515 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:33.496 7f31dc4f9700 1 --2- 172.21.2.102:0/3276315038 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f31d40d0340 0x7f31d40ef640 secure :-1 s=READY pgs=2968 cs=0 l=1 rev1=1 rx=0x7f31d0006700 tx=0x7f31d0009b80).stop 2021-02-10T11:37:33.515 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:33.497 7f31dc4f9700 1 -- 172.21.2.102:0/3276315038 shutdown_connections 2021-02-10T11:37:33.515 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:33.497 7f31dc4f9700 1 --2- 172.21.2.102:0/3276315038 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f31d40d1c70 0x7f31d40efc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:33.515 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:33.497 7f31dc4f9700 1 --2- 172.21.2.102:0/3276315038 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f31d40d0340 0x7f31d40ef640 unknown :-1 s=CLOSED pgs=2968 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:33.515 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:33.497 7f31dc4f9700 1 -- 172.21.2.102:0/3276315038 >> 172.21.2.102:0/3276315038 conn(0x7f31d40ce450 msgr2=0x7f31d40d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:33.515 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:33.497 7f31dc4f9700 1 -- 172.21.2.102:0/3276315038 shutdown_connections 2021-02-10T11:37:33.515 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:33.497 7f31dc4f9700 1 -- 172.21.2.102:0/3276315038 wait complete. 2021-02-10T11:37:33.515 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:33.511 7f31dc4f9700 1 Processor -- start 2021-02-10T11:37:33.515 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:33.511 7f31dc4f9700 1 -- start start 2021-02-10T11:37:33.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:33.512 7f31dc4f9700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f31d40b4f30 0x7f31d418b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:33.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:33.512 7f31dc4f9700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f31d4093f40 0x7f31d4094340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:33.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:33.512 7f31dc4f9700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f31d4094960 0x7f31d4187e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:33.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:33.512 7f31dc4f9700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31d4094e90 con 0x7f31d4094960 2021-02-10T11:37:33.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:33.512 7f31dc4f9700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31d418bff0 con 0x7f31d4093f40 2021-02-10T11:37:33.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:33.512 7f31dc4f9700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31d408f780 con 0x7f31d40b4f30 2021-02-10T11:37:33.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:33.512 7f31dab16700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f31d4094960 0x7f31d4187e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:33.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:33.512 7f31dab16700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f31d4094960 0x7f31d4187e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:38758/0 (socket says 172.21.2.102:38758) 2021-02-10T11:37:33.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:33.512 7f31dab16700 1 -- 172.21.2.102:0/1360659742 learned_addr learned my addr 172.21.2.102:0/1360659742 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:37:33.518 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:33.512 7f31dab16700 1 -- 172.21.2.102:0/1360659742 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f31d40b4f30 msgr2=0x7f31d418b9d0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:37:33.518 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:33.512 7f31dab16700 1 --2- 172.21.2.102:0/1360659742 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f31d40b4f30 0x7f31d418b9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:33.518 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:33.512 7f31dab16700 1 -- 172.21.2.102:0/1360659742 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f31d4093f40 msgr2=0x7f31d4094340 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:37:33.518 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:33.512 7f31dab16700 1 --2- 172.21.2.102:0/1360659742 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f31d4093f40 0x7f31d4094340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:33.518 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:33.512 7f31dab16700 1 -- 172.21.2.102:0/1360659742 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31d4188470 con 0x7f31d4094960 2021-02-10T11:37:33.518 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:33.513 7f31dab16700 1 --2- 172.21.2.102:0/1360659742 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f31d4094960 0x7f31d4187e50 secure :-1 s=READY pgs=2969 cs=0 l=1 rev1=1 rx=0x7f31cc009960 tx=0x7f31cc00bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:33.518 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:33.513 7f31c6ffd700 1 -- 172.21.2.102:0/1360659742 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f31cc000f90 con 0x7f31d4094960 2021-02-10T11:37:33.518 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:33.513 7f31c6ffd700 1 -- 172.21.2.102:0/1360659742 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31cc0030c0 con 0x7f31d4094960 2021-02-10T11:37:33.518 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:33.513 7f31dc4f9700 1 -- 172.21.2.102:0/1360659742 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31d4188470 con 0x7f31d4094960 2021-02-10T11:37:33.518 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:33.513 7f31c6ffd700 1 -- 172.21.2.102:0/1360659742 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f31cc001500 con 0x7f31d4094960 2021-02-10T11:37:33.518 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:33.513 7f31dc4f9700 1 -- 172.21.2.102:0/1360659742 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31d4188a00 con 0x7f31d4094960 2021-02-10T11:37:33.518 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:33.514 7f31c6ffd700 1 -- 172.21.2.102:0/1360659742 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f31cc001310 con 0x7f31d4094960 2021-02-10T11:37:33.518 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:33.514 7f31c6ffd700 1 --2- 172.21.2.102:0/1360659742 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f31bc0186c0 0x7f31bc01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:33.519 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:33.514 7f31c6ffd700 1 -- 172.21.2.102:0/1360659742 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(168..168 src has 1..168) v4 ==== 4753+0+0 (secure 0 0 0) 0x7f31cc0328a0 con 0x7f31d4094960 2021-02-10T11:37:33.519 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:33.514 7f31da315700 1 --2- 172.21.2.102:0/1360659742 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f31bc0186c0 0x7f31bc01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:33.519 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:33.514 7f31da315700 1 --2- 172.21.2.102:0/1360659742 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f31bc0186c0 0x7f31bc01abc0 crc :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:33.547 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:33.545 7f31dc4f9700 1 -- 172.21.2.102:0/1360659742 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31d418bff0 con 0x7f31d4094960 2021-02-10T11:37:33.550 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:33.548 7f31c6ffd700 1 -- 172.21.2.102:0/1360659742 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f31cc000f90 con 0x7f31d4094960 2021-02-10T11:37:33.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:33.671 7f31dc4f9700 1 -- 172.21.2.102:0/1360659742 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f31d418bff0 con 0x7f31d4094960 2021-02-10T11:37:33.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:33.671 7f31c6ffd700 1 -- 172.21.2.102:0/1360659742 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v168) v1 ==== 84+0+64 (secure 0 0 0) 0x7f31cc0030a0 con 0x7f31d4094960 2021-02-10T11:37:33.674 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:37:33.674 INFO:teuthology.orchestra.run.gibba002.stdout:[ 2021-02-10T11:37:33.674 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:37:33.674 INFO:teuthology.orchestra.run.gibba002.stdout: "poolnum": 1, 2021-02-10T11:37:33.674 INFO:teuthology.orchestra.run.gibba002.stdout: "poolname": "rbd" 2021-02-10T11:37:33.675 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:37:33.675 INFO:teuthology.orchestra.run.gibba002.stdout:] 2021-02-10T11:37:33.676 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:33.673 7f31dc4f9700 1 -- 172.21.2.102:0/1360659742 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f31bc0186c0 msgr2=0x7f31bc01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:33.676 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:33.673 7f31dc4f9700 1 --2- 172.21.2.102:0/1360659742 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f31bc0186c0 0x7f31bc01abc0 crc :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:33.676 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:33.673 7f31dc4f9700 1 -- 172.21.2.102:0/1360659742 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f31d4094960 msgr2=0x7f31d4187e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:33.676 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:33.673 7f31dc4f9700 1 --2- 172.21.2.102:0/1360659742 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f31d4094960 0x7f31d4187e50 secure :-1 s=READY pgs=2969 cs=0 l=1 rev1=1 rx=0x7f31cc009960 tx=0x7f31cc00bcf0).stop 2021-02-10T11:37:33.676 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:33.674 7f31dc4f9700 1 -- 172.21.2.102:0/1360659742 shutdown_connections 2021-02-10T11:37:33.676 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:33.674 7f31dc4f9700 1 --2- 172.21.2.102:0/1360659742 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f31d4094960 0x7f31d4187e50 unknown :-1 s=CLOSED pgs=2969 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:33.676 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:33.674 7f31dc4f9700 1 --2- 172.21.2.102:0/1360659742 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f31bc0186c0 0x7f31bc01abc0 unknown :-1 s=CLOSED pgs=1301 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:33.676 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:33.674 7f31dc4f9700 1 --2- 172.21.2.102:0/1360659742 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f31d4093f40 0x7f31d4094340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:33.676 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:33.674 7f31dc4f9700 1 --2- 172.21.2.102:0/1360659742 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f31d40b4f30 0x7f31d418b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:33.677 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:33.674 7f31dc4f9700 1 -- 172.21.2.102:0/1360659742 >> 172.21.2.102:0/1360659742 conn(0x7f31d419f8f0 msgr2=0x7f31d41a5f80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:33.677 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:33.674 7f31dc4f9700 1 -- 172.21.2.102:0/1360659742 shutdown_connections 2021-02-10T11:37:33.677 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:33.674 7f31dc4f9700 1 -- 172.21.2.102:0/1360659742 wait complete. 2021-02-10T11:37:33.684 INFO:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2021-02-10T11:37:33.684 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool create cephfs_metadata 8 2021-02-10T11:37:34.117 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:34.098 7f87fe3d3700 1 Processor -- start 2021-02-10T11:37:34.117 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:34.098 7f87fe3d3700 1 -- start start 2021-02-10T11:37:34.117 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:34.099 7f87fe3d3700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f87f80d1870 0x7f87f81b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:34.117 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:34.099 7f87fe3d3700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f87f81b0ae0 0x7f87f8108940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:34.117 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:34.099 7f87fe3d3700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f87f80d1c70 0x7f87f8108f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:34.117 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:34.099 7f87fe3d3700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87f80e5ef0 con 0x7f87f80d1c70 2021-02-10T11:37:34.117 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:34.099 7f87fe3d3700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87f8109a60 con 0x7f87f81b0ae0 2021-02-10T11:37:34.117 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:34.099 7f87fe3d3700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87f8088070 con 0x7f87f80d1870 2021-02-10T11:37:34.117 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:34.099 7f87fc9f0700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f87f80d1c70 0x7f87f8108f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:34.118 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:34.099 7f87fc9f0700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f87f80d1c70 0x7f87f8108f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:38762/0 (socket says 172.21.2.102:38762) 2021-02-10T11:37:34.118 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:34.099 7f87fc9f0700 1 -- 172.21.2.102:0/1852916156 learned_addr learned my addr 172.21.2.102:0/1852916156 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:37:34.118 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:34.099 7f87fc9f0700 1 -- 172.21.2.102:0/1852916156 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f87f80d1870 msgr2=0x7f87f81b04c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:37:34.118 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:34.099 7f87fc9f0700 1 --2- 172.21.2.102:0/1852916156 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f87f80d1870 0x7f87f81b04c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:34.118 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:34.099 7f87fc9f0700 1 -- 172.21.2.102:0/1852916156 >> v2:172.21.2.113:3300/0 conn(0x7f87f81b0ae0 msgr2=0x7f87f8108940 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:37:34.118 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:34.099 7f87fc9f0700 1 --2- 172.21.2.102:0/1852916156 >> v2:172.21.2.113:3300/0 conn(0x7f87f81b0ae0 0x7f87f8108940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:34.118 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:34.099 7f87fc9f0700 1 -- 172.21.2.102:0/1852916156 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87f8109580 con 0x7f87f80d1c70 2021-02-10T11:37:34.118 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:34.100 7f87fc9f0700 1 --2- 172.21.2.102:0/1852916156 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f87f80d1c70 0x7f87f8108f60 secure :-1 s=READY pgs=2970 cs=0 l=1 rev1=1 rx=0x7f87f0006f00 tx=0x7f87f0009f10).ready entity=mon.0 client_cookie=5c4faa1df315ebf5 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:34.118 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:34.100 7f87f6ffd700 1 -- 172.21.2.102:0/1852916156 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f87f000b3b0 con 0x7f87f80d1c70 2021-02-10T11:37:34.118 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:34.100 7f87f6ffd700 1 -- 172.21.2.102:0/1852916156 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87f000b670 con 0x7f87f80d1c70 2021-02-10T11:37:34.119 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:34.100 7f87f6ffd700 1 -- 172.21.2.102:0/1852916156 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f87f000bb90 con 0x7f87f80d1c70 2021-02-10T11:37:34.119 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:34.100 7f87fe3d3700 1 -- 172.21.2.102:0/1852916156 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f87f80d1c70 msgr2=0x7f87f8108f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:34.119 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:34.100 7f87fe3d3700 1 --2- 172.21.2.102:0/1852916156 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f87f80d1c70 0x7f87f8108f60 secure :-1 s=READY pgs=2970 cs=0 l=1 rev1=1 rx=0x7f87f0006f00 tx=0x7f87f0009f10).stop 2021-02-10T11:37:34.119 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:34.100 7f87fe3d3700 1 -- 172.21.2.102:0/1852916156 shutdown_connections 2021-02-10T11:37:34.119 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:34.100 7f87fe3d3700 1 --2- 172.21.2.102:0/1852916156 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f87f80d1c70 0x7f87f8108f60 unknown :-1 s=CLOSED pgs=2970 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:34.119 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:34.100 7f87fe3d3700 1 --2- 172.21.2.102:0/1852916156 >> v2:172.21.2.113:3300/0 conn(0x7f87f81b0ae0 0x7f87f8108940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:34.119 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:34.100 7f87fe3d3700 1 --2- 172.21.2.102:0/1852916156 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f87f80d1870 0x7f87f81b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:34.119 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:34.100 7f87fe3d3700 1 -- 172.21.2.102:0/1852916156 >> 172.21.2.102:0/1852916156 conn(0x7f87f80ce420 msgr2=0x7f87f80d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:34.119 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:34.100 7f87fe3d3700 1 -- 172.21.2.102:0/1852916156 shutdown_connections 2021-02-10T11:37:34.119 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:34.100 7f87fe3d3700 1 -- 172.21.2.102:0/1852916156 wait complete. 2021-02-10T11:37:34.121 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:34.115 7f87fe3d3700 1 Processor -- start 2021-02-10T11:37:34.121 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:34.115 7f87fe3d3700 1 -- start start 2021-02-10T11:37:34.121 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:34.116 7f87fe3d3700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f87f80a73c0 0x7f87f80a6b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:34.122 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:34.116 7f87fe3d3700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f87f80add40 0x7f87f80ae140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:34.122 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:34.116 7f87fe3d3700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f87f80ae760 0x7f87f80af180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:34.122 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:34.116 7f87fe3d3700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87f80aec00 con 0x7f87f80ae760 2021-02-10T11:37:34.122 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:34.116 7f87fe3d3700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87f80875d0 con 0x7f87f80a73c0 2021-02-10T11:37:34.122 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:34.116 7f87fe3d3700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87f80af7a0 con 0x7f87f80add40 2021-02-10T11:37:34.122 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:34.116 7f87fc9f0700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f87f80ae760 0x7f87f80af180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:34.122 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:34.116 7f87fc9f0700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f87f80ae760 0x7f87f80af180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:38768/0 (socket says 172.21.2.102:38768) 2021-02-10T11:37:34.122 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:34.116 7f87fc9f0700 1 -- 172.21.2.102:0/497986995 learned_addr learned my addr 172.21.2.102:0/497986995 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:37:34.122 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:34.116 7f87fc9f0700 1 -- 172.21.2.102:0/497986995 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f87f80add40 msgr2=0x7f87f80ae140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:34.122 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:34.116 7f87fc9f0700 1 --2- 172.21.2.102:0/497986995 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f87f80add40 0x7f87f80ae140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:34.122 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:34.116 7f87fc9f0700 1 -- 172.21.2.102:0/497986995 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f87f80a73c0 msgr2=0x7f87f80a6b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:34.123 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:34.116 7f87fc9f0700 1 --2- 172.21.2.102:0/497986995 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f87f80a73c0 0x7f87f80a6b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:34.123 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:34.116 7f87fc9f0700 1 -- 172.21.2.102:0/497986995 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87f80af9c0 con 0x7f87f80ae760 2021-02-10T11:37:34.123 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:34.116 7f87fc9f0700 1 --2- 172.21.2.102:0/497986995 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f87f80ae760 0x7f87f80af180 secure :-1 s=READY pgs=2971 cs=0 l=1 rev1=1 rx=0x7f87f000b940 tx=0x7f87f000bce0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:34.123 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:34.117 7f87f4ff9700 1 -- 172.21.2.102:0/497986995 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f87f0000cc0 con 0x7f87f80ae760 2021-02-10T11:37:34.123 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:34.117 7f87f4ff9700 1 -- 172.21.2.102:0/497986995 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87f0005820 con 0x7f87f80ae760 2021-02-10T11:37:34.123 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:34.117 7f87f4ff9700 1 -- 172.21.2.102:0/497986995 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f87f0005c70 con 0x7f87f80ae760 2021-02-10T11:37:34.123 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:34.117 7f87fe3d3700 1 -- 172.21.2.102:0/497986995 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87f80af9c0 con 0x7f87f80ae760 2021-02-10T11:37:34.123 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:34.117 7f87fe3d3700 1 -- 172.21.2.102:0/497986995 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87f80af9c0 con 0x7f87f80ae760 2021-02-10T11:37:34.123 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:34.117 7f87f4ff9700 1 -- 172.21.2.102:0/497986995 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f87f0005a00 con 0x7f87f80ae760 2021-02-10T11:37:34.123 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:34.118 7f87f4ff9700 1 --2- 172.21.2.102:0/497986995 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f87d80186c0 0x7f87d801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:34.123 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:34.118 7f87f4ff9700 1 -- 172.21.2.102:0/497986995 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(168..168 src has 1..168) v4 ==== 4753+0+0 (secure 0 0 0) 0x7f87f00325a0 con 0x7f87f80ae760 2021-02-10T11:37:34.123 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:34.118 7f87f7fff700 1 --2- 172.21.2.102:0/497986995 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f87d80186c0 0x7f87d801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:34.123 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:34.118 7f87f7fff700 1 --2- 172.21.2.102:0/497986995 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f87d80186c0 0x7f87d801abc0 crc :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:34.151 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:34.149 7f87fe3d3700 1 -- 172.21.2.102:0/497986995 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87f8082980 con 0x7f87f80ae760 2021-02-10T11:37:34.199 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:34.151 7f87f4ff9700 1 -- 172.21.2.102:0/497986995 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f87f0021810 con 0x7f87f80ae760 2021-02-10T11:37:34.281 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:34.279 7f87fe3d3700 1 -- 172.21.2.102:0/497986995 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"} v 0) v1 -- 0x7f87f8109a60 con 0x7f87f80ae760 2021-02-10T11:37:34.689 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:34.686 7f87f4ff9700 1 -- 172.21.2.102:0/497986995 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"}]=0 pool 'cephfs_metadata' created v169) v1 ==== 133+0+0 (secure 0 0 0) 0x7f87f0021810 con 0x7f87f80ae760 2021-02-10T11:37:34.697 INFO:teuthology.orchestra.run.gibba002.stderr:pool 'cephfs_metadata' created 2021-02-10T11:37:34.700 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:34.697 7f87fe3d3700 1 -- 172.21.2.102:0/497986995 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f87d80186c0 msgr2=0x7f87d801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:34.700 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:34.697 7f87fe3d3700 1 --2- 172.21.2.102:0/497986995 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f87d80186c0 0x7f87d801abc0 crc :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:34.700 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:34.697 7f87fe3d3700 1 -- 172.21.2.102:0/497986995 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f87f80ae760 msgr2=0x7f87f80af180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:34.700 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:34.697 7f87fe3d3700 1 --2- 172.21.2.102:0/497986995 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f87f80ae760 0x7f87f80af180 secure :-1 s=READY pgs=2971 cs=0 l=1 rev1=1 rx=0x7f87f000b940 tx=0x7f87f000bce0).stop 2021-02-10T11:37:34.700 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:34.697 7f87fe3d3700 1 -- 172.21.2.102:0/497986995 shutdown_connections 2021-02-10T11:37:34.700 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:34.697 7f87fe3d3700 1 --2- 172.21.2.102:0/497986995 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f87f80ae760 0x7f87f80af180 unknown :-1 s=CLOSED pgs=2971 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:34.700 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:34.697 7f87fe3d3700 1 --2- 172.21.2.102:0/497986995 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f87f80add40 0x7f87f80ae140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:34.700 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:34.697 7f87fe3d3700 1 --2- 172.21.2.102:0/497986995 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f87d80186c0 0x7f87d801abc0 unknown :-1 s=CLOSED pgs=1302 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:34.700 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:34.697 7f87fe3d3700 1 --2- 172.21.2.102:0/497986995 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f87f80a73c0 0x7f87f80a6b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:34.700 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:34.697 7f87fe3d3700 1 -- 172.21.2.102:0/497986995 >> 172.21.2.102:0/497986995 conn(0x7f87f80be980 msgr2=0x7f87f8092cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:34.700 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:34.697 7f87fe3d3700 1 -- 172.21.2.102:0/497986995 shutdown_connections 2021-02-10T11:37:34.701 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:34.698 7f87fe3d3700 1 -- 172.21.2.102:0/497986995 wait complete. 2021-02-10T11:37:34.708 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool create cephfs_data 8 2021-02-10T11:37:35.149 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:35.130 7fa547f73700 1 Processor -- start 2021-02-10T11:37:35.149 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:35.130 7fa547f73700 1 -- start start 2021-02-10T11:37:35.149 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:35.131 7fa547f73700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fa5400d1870 0x7fa5401b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:35.149 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:35.131 7fa547f73700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fa5400cff70 0x7fa540109130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:35.149 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:35.131 7fa547f73700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5400fe1c0 con 0x7fa5401b0ae0 2021-02-10T11:37:35.149 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:35.131 7fa547f73700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa540109ad0 con 0x7fa5400cff70 2021-02-10T11:37:35.149 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:35.131 7fa547f73700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa540109750 con 0x7fa5400d1870 2021-02-10T11:37:35.149 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:35.131 7fa546590700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fa5400cff70 0x7fa540109130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:35.150 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:35.131 7fa546590700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fa5400cff70 0x7fa540109130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:38774/0 (socket says 172.21.2.102:38774) 2021-02-10T11:37:35.150 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:35.131 7fa546590700 1 -- 172.21.2.102:0/1993583731 learned_addr learned my addr 172.21.2.102:0/1993583731 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:37:35.150 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:35.131 7fa546590700 1 -- 172.21.2.102:0/1993583731 >> v2:172.21.2.113:3300/0 conn(0x7fa5400d1870 msgr2=0x7fa5401b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:37:35.150 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:35.131 7fa546590700 1 --2- 172.21.2.102:0/1993583731 >> v2:172.21.2.113:3300/0 conn(0x7fa5400d1870 0x7fa5401b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:35.150 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:35.131 7fa546590700 1 -- 172.21.2.102:0/1993583731 >> v1:172.21.2.113:6789/0 conn(0x7fa5401b0ae0 legacy=0x7fa540108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:37:35.150 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:35.131 7fa546590700 1 -- 172.21.2.102:0/1993583731 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa540088070 con 0x7fa5400cff70 2021-02-10T11:37:35.150 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:35.131 7fa546590700 1 --2- 172.21.2.102:0/1993583731 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa5400cff70 0x7fa540109130 secure :-1 s=READY pgs=2972 cs=0 l=1 rev1=1 rx=0x7fa538006f00 tx=0x7fa538009f10).ready entity=mon.0 client_cookie=14c4ca15344bc1ac server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:35.150 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:35.132 7fa544d8d700 1 -- 172.21.2.102:0/1993583731 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa53800b3b0 con 0x7fa5400cff70 2021-02-10T11:37:35.150 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:35.132 7fa544d8d700 1 -- 172.21.2.102:0/1993583731 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa538000d30 con 0x7fa5400cff70 2021-02-10T11:37:35.150 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:35.132 7fa544d8d700 1 -- 172.21.2.102:0/1993583731 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa53800b920 con 0x7fa5400cff70 2021-02-10T11:37:35.150 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:35.132 7fa547f73700 1 -- 172.21.2.102:0/1993583731 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa5400cff70 msgr2=0x7fa540109130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:35.150 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:35.132 7fa547f73700 1 --2- 172.21.2.102:0/1993583731 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa5400cff70 0x7fa540109130 secure :-1 s=READY pgs=2972 cs=0 l=1 rev1=1 rx=0x7fa538006f00 tx=0x7fa538009f10).stop 2021-02-10T11:37:35.151 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:35.132 7fa547f73700 1 -- 172.21.2.102:0/1993583731 shutdown_connections 2021-02-10T11:37:35.151 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:35.132 7fa547f73700 1 --2- 172.21.2.102:0/1993583731 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa5400cff70 0x7fa540109130 unknown :-1 s=CLOSED pgs=2972 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:35.151 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:35.132 7fa547f73700 1 --2- 172.21.2.102:0/1993583731 >> v2:172.21.2.113:3300/0 conn(0x7fa5400d1870 0x7fa5401b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:35.151 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:35.132 7fa547f73700 1 -- 172.21.2.102:0/1993583731 >> 172.21.2.102:0/1993583731 conn(0x7fa5400ce420 msgr2=0x7fa5400d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:35.151 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:35.133 7fa547f73700 1 -- 172.21.2.102:0/1993583731 shutdown_connections 2021-02-10T11:37:35.151 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:35.133 7fa547f73700 1 -- 172.21.2.102:0/1993583731 wait complete. 2021-02-10T11:37:35.152 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:35.147 7fa547f73700 1 Processor -- start 2021-02-10T11:37:35.152 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:35.147 7fa547f73700 1 -- start start 2021-02-10T11:37:35.152 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:35.148 7fa547f73700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa5400c3820 0x7fa5400a7950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:35.153 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:35.148 7fa547f73700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa5401a44c0 0x7fa5401a48c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:35.153 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:35.148 7fa547f73700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa5400863d0 0x7fa5400867d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:35.153 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:35.148 7fa547f73700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa540085ed0 con 0x7fa5400863d0 2021-02-10T11:37:35.153 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:35.148 7fa547f73700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5400c0740 con 0x7fa5401a44c0 2021-02-10T11:37:35.153 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:35.148 7fa547f73700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa540086df0 con 0x7fa5400c3820 2021-02-10T11:37:35.153 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:35.148 7fa546590700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa5400863d0 0x7fa5400867d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:35.153 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:35.148 7fa546590700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa5400863d0 0x7fa5400867d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:38780/0 (socket says 172.21.2.102:38780) 2021-02-10T11:37:35.153 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:35.148 7fa546590700 1 -- 172.21.2.102:0/1828366555 learned_addr learned my addr 172.21.2.102:0/1828366555 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:37:35.153 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:35.148 7fa546590700 1 -- 172.21.2.102:0/1828366555 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa5400c3820 msgr2=0x7fa5400a7950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:35.153 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:35.148 7fa546590700 1 --2- 172.21.2.102:0/1828366555 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa5400c3820 0x7fa5400a7950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:35.153 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:35.148 7fa546590700 1 -- 172.21.2.102:0/1828366555 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa5401a44c0 msgr2=0x7fa5401a48c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:35.154 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:35.148 7fa546590700 1 --2- 172.21.2.102:0/1828366555 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa5401a44c0 0x7fa5401a48c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:35.154 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:35.148 7fa546590700 1 -- 172.21.2.102:0/1828366555 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa540087010 con 0x7fa5400863d0 2021-02-10T11:37:35.154 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:35.148 7fa546590700 1 --2- 172.21.2.102:0/1828366555 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa5400863d0 0x7fa5400867d0 secure :-1 s=READY pgs=2973 cs=0 l=1 rev1=1 rx=0x7fa538006f30 tx=0x7fa538002d20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:35.154 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:35.148 7fa5327fc700 1 -- 172.21.2.102:0/1828366555 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa53800c300 con 0x7fa5400863d0 2021-02-10T11:37:35.154 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:35.149 7fa5327fc700 1 -- 172.21.2.102:0/1828366555 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa53800c580 con 0x7fa5400863d0 2021-02-10T11:37:35.154 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:35.149 7fa547f73700 1 -- 172.21.2.102:0/1828366555 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa540087010 con 0x7fa5400863d0 2021-02-10T11:37:35.154 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:35.149 7fa5327fc700 1 -- 172.21.2.102:0/1828366555 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa53800ca60 con 0x7fa5400863d0 2021-02-10T11:37:35.154 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:35.149 7fa547f73700 1 -- 172.21.2.102:0/1828366555 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa540087010 con 0x7fa5400863d0 2021-02-10T11:37:35.154 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:35.149 7fa5327fc700 1 -- 172.21.2.102:0/1828366555 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa538006ba0 con 0x7fa5400863d0 2021-02-10T11:37:35.154 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:35.149 7fa5327fc700 1 --2- 172.21.2.102:0/1828366555 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa5280186c0 0x7fa52801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:35.155 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:35.149 7fa5327fc700 1 -- 172.21.2.102:0/1828366555 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(169..169 src has 1..169) v4 ==== 5107+0+0 (secure 0 0 0) 0x7fa538032740 con 0x7fa5400863d0 2021-02-10T11:37:35.155 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:35.150 7fa545d8f700 1 --2- 172.21.2.102:0/1828366555 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa5280186c0 0x7fa52801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:35.155 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:35.150 7fa545d8f700 1 --2- 172.21.2.102:0/1828366555 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa5280186c0 0x7fa52801abc0 crc :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:35.183 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:35.181 7fa547f73700 1 -- 172.21.2.102:0/1828366555 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5400fe1c0 con 0x7fa5400863d0 2021-02-10T11:37:35.185 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:35.183 7fa5327fc700 1 -- 172.21.2.102:0/1828366555 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fa538000cc0 con 0x7fa5400863d0 2021-02-10T11:37:35.319 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:35.317 7fa547f73700 1 -- 172.21.2.102:0/1828366555 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_data"} v 0) v1 -- 0x7fa5400fe1c0 con 0x7fa5400863d0 2021-02-10T11:37:35.695 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:35.693 7fa5327fc700 1 -- 172.21.2.102:0/1828366555 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_data"}]=0 pool 'cephfs_data' created v170) v1 ==== 125+0+0 (secure 0 0 0) 0x7fa538006ba0 con 0x7fa5400863d0 2021-02-10T11:37:35.735 INFO:teuthology.orchestra.run.gibba002.stderr:pool 'cephfs_data' created 2021-02-10T11:37:35.737 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:35.735 7fa547f73700 1 -- 172.21.2.102:0/1828366555 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa5280186c0 msgr2=0x7fa52801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:35.737 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:35.735 7fa547f73700 1 --2- 172.21.2.102:0/1828366555 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa5280186c0 0x7fa52801abc0 crc :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:35.737 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:35.735 7fa547f73700 1 -- 172.21.2.102:0/1828366555 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa5400863d0 msgr2=0x7fa5400867d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:35.737 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:35.735 7fa547f73700 1 --2- 172.21.2.102:0/1828366555 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa5400863d0 0x7fa5400867d0 secure :-1 s=READY pgs=2973 cs=0 l=1 rev1=1 rx=0x7fa538006f30 tx=0x7fa538002d20).stop 2021-02-10T11:37:35.737 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:35.735 7fa547f73700 1 -- 172.21.2.102:0/1828366555 shutdown_connections 2021-02-10T11:37:35.737 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:35.735 7fa547f73700 1 --2- 172.21.2.102:0/1828366555 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa5400863d0 0x7fa5400867d0 unknown :-1 s=CLOSED pgs=2973 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:35.738 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:35.735 7fa547f73700 1 --2- 172.21.2.102:0/1828366555 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa5280186c0 0x7fa52801abc0 unknown :-1 s=CLOSED pgs=1303 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:35.738 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:35.735 7fa547f73700 1 --2- 172.21.2.102:0/1828366555 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa5401a44c0 0x7fa5401a48c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:35.738 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:35.735 7fa547f73700 1 --2- 172.21.2.102:0/1828366555 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa5400c3820 0x7fa5400a7950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:35.738 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:35.735 7fa547f73700 1 -- 172.21.2.102:0/1828366555 >> 172.21.2.102:0/1828366555 conn(0x7fa5400872c0 msgr2=0x7fa540095700 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:35.738 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:35.735 7fa547f73700 1 -- 172.21.2.102:0/1828366555 shutdown_connections 2021-02-10T11:37:35.738 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:35.735 7fa547f73700 1 -- 172.21.2.102:0/1828366555 wait complete. 2021-02-10T11:37:35.746 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2021-02-10T11:37:36.180 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:36.161 7f0bea9b9700 1 Processor -- start 2021-02-10T11:37:36.180 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:36.161 7f0bea9b9700 1 -- start start 2021-02-10T11:37:36.181 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:36.162 7f0bea9b9700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f0be40d1c80 0x7f0be40cef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:36.181 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:36.162 7f0bea9b9700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f0be40cf590 0x7f0be40cf990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:36.181 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:36.162 7f0bea9b9700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0be40fe1d0 con 0x7f0be40d1880 2021-02-10T11:37:36.181 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:36.162 7f0bea9b9700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0be41b0cc0 con 0x7f0be40cf590 2021-02-10T11:37:36.181 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:36.162 7f0bea9b9700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0be4109a70 con 0x7f0be40d1c80 2021-02-10T11:37:36.181 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:36.162 7f0be37fe700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f0be40d1c80 0x7f0be40cef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:36.181 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:36.162 7f0be8fd6700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f0be40cf590 0x7f0be40cf990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:36.181 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:36.162 7f0be37fe700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f0be40d1c80 0x7f0be40cef70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.113:3300/0 says I am v2:172.21.2.102:42218/0 (socket says 172.21.2.102:42218) 2021-02-10T11:37:36.181 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:36.162 7f0be37fe700 1 -- 172.21.2.102:0/1850950931 learned_addr learned my addr 172.21.2.102:0/1850950931 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:37:36.181 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:36.162 7f0be8fd6700 1 -- 172.21.2.102:0/1850950931 >> v2:172.21.2.113:3300/0 conn(0x7f0be40d1c80 msgr2=0x7f0be40cef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:37:36.181 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:36.163 7f0be8fd6700 1 --2- 172.21.2.102:0/1850950931 >> v2:172.21.2.113:3300/0 conn(0x7f0be40d1c80 0x7f0be40cef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:36.181 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:36.163 7f0be8fd6700 1 -- 172.21.2.102:0/1850950931 >> v1:172.21.2.102:6789/0 conn(0x7f0be40d1880 legacy=0x7f0be41b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:36.182 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:36.163 7f0be8fd6700 1 -- 172.21.2.102:0/1850950931 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0be4088070 con 0x7f0be40cf590 2021-02-10T11:37:36.182 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:36.163 7f0be37fe700 1 --2- 172.21.2.102:0/1850950931 >> v2:172.21.2.113:3300/0 conn(0x7f0be40d1c80 0x7f0be40cef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:37:36.182 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:36.163 7f0be8fd6700 1 --2- 172.21.2.102:0/1850950931 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0be40cf590 0x7f0be40cf990 secure :-1 s=READY pgs=2975 cs=0 l=1 rev1=1 rx=0x7f0bdc006e30 tx=0x7f0bdc009e60).ready entity=mon.0 client_cookie=cfe907f558725338 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:36.182 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:36.163 7f0be2ffd700 1 -- 172.21.2.102:0/1850950931 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0bdc00b300 con 0x7f0be40cf590 2021-02-10T11:37:36.182 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:36.163 7f0be2ffd700 1 -- 172.21.2.102:0/1850950931 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bdc00b580 con 0x7f0be40cf590 2021-02-10T11:37:36.182 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:36.163 7f0be2ffd700 1 -- 172.21.2.102:0/1850950931 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0bdc00ba60 con 0x7f0be40cf590 2021-02-10T11:37:36.182 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:36.163 7f0bea9b9700 1 -- 172.21.2.102:0/1850950931 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0be40cf590 msgr2=0x7f0be40cf990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:36.182 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:36.163 7f0bea9b9700 1 --2- 172.21.2.102:0/1850950931 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0be40cf590 0x7f0be40cf990 secure :-1 s=READY pgs=2975 cs=0 l=1 rev1=1 rx=0x7f0bdc006e30 tx=0x7f0bdc009e60).stop 2021-02-10T11:37:36.182 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:36.164 7f0bea9b9700 1 -- 172.21.2.102:0/1850950931 shutdown_connections 2021-02-10T11:37:36.182 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:36.164 7f0bea9b9700 1 --2- 172.21.2.102:0/1850950931 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0be40cf590 0x7f0be40cf990 unknown :-1 s=CLOSED pgs=2975 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:36.182 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:36.164 7f0bea9b9700 1 --2- 172.21.2.102:0/1850950931 >> v2:172.21.2.113:3300/0 conn(0x7f0be40d1c80 0x7f0be40cef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:36.182 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:36.164 7f0bea9b9700 1 -- 172.21.2.102:0/1850950931 >> 172.21.2.102:0/1850950931 conn(0x7f0be40ce430 msgr2=0x7f0be40d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:36.183 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:36.164 7f0bea9b9700 1 -- 172.21.2.102:0/1850950931 shutdown_connections 2021-02-10T11:37:36.183 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:36.164 7f0bea9b9700 1 -- 172.21.2.102:0/1850950931 wait complete. 2021-02-10T11:37:36.184 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:36.179 7f0bea9b9700 1 Processor -- start 2021-02-10T11:37:36.184 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:36.179 7f0bea9b9700 1 -- start start 2021-02-10T11:37:36.185 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:36.179 7f0bea9b9700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0be41a5af0 0x7f0be40b17a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:36.185 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:36.179 7f0bea9b9700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0be41a6030 0x7f0be40b1dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:36.185 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:36.179 7f0bea9b9700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0be40b23e0 0x7f0be4099e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:36.185 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:36.179 7f0bea9b9700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0be418e0e0 con 0x7f0be41a5af0 2021-02-10T11:37:36.185 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:36.179 7f0bea9b9700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0be409a470 con 0x7f0be40b23e0 2021-02-10T11:37:36.185 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:36.179 7f0bea9b9700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0be409a690 con 0x7f0be41a6030 2021-02-10T11:37:36.185 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:36.179 7f0be3fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0be41a5af0 0x7f0be40b17a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:36.185 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:36.179 7f0be3fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0be41a5af0 0x7f0be40b17a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:38792/0 (socket says 172.21.2.102:38792) 2021-02-10T11:37:36.185 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:36.179 7f0be3fff700 1 -- 172.21.2.102:0/32785812 learned_addr learned my addr 172.21.2.102:0/32785812 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:37:36.185 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:36.179 7f0be8fd6700 1 --2- 172.21.2.102:0/32785812 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0be40b23e0 0x7f0be4099e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:36.185 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:36.180 7f0be37fe700 1 --2- 172.21.2.102:0/32785812 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0be41a6030 0x7f0be40b1dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:36.185 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:36.180 7f0be8fd6700 1 -- 172.21.2.102:0/32785812 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0be41a6030 msgr2=0x7f0be40b1dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:36.186 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:36.180 7f0be8fd6700 1 --2- 172.21.2.102:0/32785812 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0be41a6030 0x7f0be40b1dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:36.186 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:36.180 7f0be8fd6700 1 -- 172.21.2.102:0/32785812 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0be41a5af0 msgr2=0x7f0be40b17a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:36.186 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:36.180 7f0be8fd6700 1 --2- 172.21.2.102:0/32785812 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0be41a5af0 0x7f0be40b17a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:36.186 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:36.180 7f0be8fd6700 1 -- 172.21.2.102:0/32785812 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0be409a8b0 con 0x7f0be40b23e0 2021-02-10T11:37:36.186 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:36.180 7f0be8fd6700 1 --2- 172.21.2.102:0/32785812 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0be40b23e0 0x7f0be4099e50 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 rx=0x7f0bdc0099f0 tx=0x7f0bdc00bce0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:36.186 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:36.180 7f0be0ff9700 1 -- 172.21.2.102:0/32785812 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0bdc009380 con 0x7f0be40b23e0 2021-02-10T11:37:36.186 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:36.180 7f0be0ff9700 1 -- 172.21.2.102:0/32785812 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bdc009090 con 0x7f0be40b23e0 2021-02-10T11:37:36.186 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:36.180 7f0bea9b9700 1 -- 172.21.2.102:0/32785812 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0be409a8b0 con 0x7f0be40b23e0 2021-02-10T11:37:36.186 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:36.180 7f0be0ff9700 1 -- 172.21.2.102:0/32785812 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0bdc009380 con 0x7f0be40b23e0 2021-02-10T11:37:36.186 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:36.180 7f0bea9b9700 1 -- 172.21.2.102:0/32785812 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0be409a8b0 con 0x7f0be40b23e0 2021-02-10T11:37:36.186 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:36.181 7f0be0ff9700 1 -- 172.21.2.102:0/32785812 <== mon.1 v2:172.21.2.113:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f0bdc0069c0 con 0x7f0be40b23e0 2021-02-10T11:37:36.186 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:36.181 7f0be0ff9700 1 --2- 172.21.2.102:0/32785812 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0bc4018700 0x7f0bc401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:36.187 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:36.181 7f0be0ff9700 1 -- 172.21.2.102:0/32785812 <== mon.1 v2:172.21.2.113:3300/0 5 ==== osd_map(170..170 src has 1..170) v4 ==== 5457+0+0 (secure 0 0 0) 0x7f0bdc032a10 con 0x7f0be40b23e0 2021-02-10T11:37:36.187 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:36.181 7f0be3fff700 1 --2- 172.21.2.102:0/32785812 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0bc4018700 0x7f0bc401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:36.187 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:36.181 7f0be3fff700 1 --2- 172.21.2.102:0/32785812 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0bc4018700 0x7f0bc401abc0 crc :-1 s=READY pgs=1306 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:36.213 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:36.212 7f0bea9b9700 1 -- 172.21.2.102:0/32785812 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0be4082a70 con 0x7f0be40b23e0 2021-02-10T11:37:36.217 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:36.215 7f0be0ff9700 1 -- 172.21.2.102:0/32785812 <== mon.1 v2:172.21.2.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f0bdc00be20 con 0x7f0be40b23e0 2021-02-10T11:37:36.348 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:36.346 7f0bea9b9700 1 -- 172.21.2.102:0/32785812 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_command({"data": "cephfs_data", "prefix": "fs new", "force": true, "fs_name": "cephfs", "metadata": "cephfs_metadata"} v 0) v1 -- 0x7f0be4082a70 con 0x7f0be40b23e0 2021-02-10T11:37:36.352 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:36.350 7f0be0ff9700 1 -- 172.21.2.102:0/32785812 <== mon.1 v2:172.21.2.113:3300/0 7 ==== mon_command_ack([{"data": "cephfs_data", "prefix": "fs new", "force": true, "fs_name": "cephfs", "metadata": "cephfs_metadata"}]=0 new fs with metadata pool 36 and data pool 37 v146) v1 ==== 189+0+0 (secure 0 0 0) 0x7f0bdc006ce0 con 0x7f0be40b23e0 2021-02-10T11:37:36.355 INFO:teuthology.orchestra.run.gibba002.stderr:new fs with metadata pool 36 and data pool 37 2021-02-10T11:37:36.357 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:36.355 7f0bea9b9700 1 -- 172.21.2.102:0/32785812 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0bc4018700 msgr2=0x7f0bc401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:36.357 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:36.355 7f0bea9b9700 1 --2- 172.21.2.102:0/32785812 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0bc4018700 0x7f0bc401abc0 crc :-1 s=READY pgs=1306 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:36.358 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:36.355 7f0bea9b9700 1 -- 172.21.2.102:0/32785812 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0be40b23e0 msgr2=0x7f0be4099e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:36.358 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:36.355 7f0bea9b9700 1 --2- 172.21.2.102:0/32785812 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0be40b23e0 0x7f0be4099e50 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 rx=0x7f0bdc0099f0 tx=0x7f0bdc00bce0).stop 2021-02-10T11:37:36.358 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:36.355 7f0bea9b9700 1 -- 172.21.2.102:0/32785812 shutdown_connections 2021-02-10T11:37:36.358 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:36.355 7f0bea9b9700 1 --2- 172.21.2.102:0/32785812 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0bc4018700 0x7f0bc401abc0 unknown :-1 s=CLOSED pgs=1306 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:36.358 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:36.355 7f0bea9b9700 1 --2- 172.21.2.102:0/32785812 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0be40b23e0 0x7f0be4099e50 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:36.358 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:36.355 7f0bea9b9700 1 --2- 172.21.2.102:0/32785812 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0be41a6030 0x7f0be40b1dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:36.358 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:36.355 7f0bea9b9700 1 --2- 172.21.2.102:0/32785812 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0be41a5af0 0x7f0be40b17a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:36.358 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:36.355 7f0bea9b9700 1 -- 172.21.2.102:0/32785812 >> 172.21.2.102:0/32785812 conn(0x7f0be40873b0 msgr2=0x7f0be41b2b20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:36.358 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:36.355 7f0bea9b9700 1 -- 172.21.2.102:0/32785812 shutdown_connections 2021-02-10T11:37:36.358 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:36.355 7f0bea9b9700 1 -- 172.21.2.102:0/32785812 wait complete. 2021-02-10T11:37:36.366 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:37:36.794 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:36.776 7f726a5f0700 1 Processor -- start 2021-02-10T11:37:36.794 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:36.776 7f726a5f0700 1 -- start start 2021-02-10T11:37:36.794 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:36.777 7f726a5f0700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f72640d0340 0x7f72640ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:36.794 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:36.777 7f726a5f0700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f72640d1c70 0x7f72640efc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:36.795 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:36.777 7f726a5f0700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72640f0b10 con 0x7f72640d1870 2021-02-10T11:37:36.795 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:36.777 7f726a5f0700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72640e5ed0 con 0x7f72640d1c70 2021-02-10T11:37:36.795 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:36.777 7f726a5f0700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72641b0be0 con 0x7f72640d0340 2021-02-10T11:37:36.795 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:36.777 7f7263fff700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f72640d1870 0x7f72641b03f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:59256/0 (socket says 172.21.2.102:59256) 2021-02-10T11:37:36.795 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:36.777 7f7263fff700 1 -- 172.21.2.102:0/2016863033 learned_addr learned my addr 172.21.2.102:0/2016863033 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:37:36.795 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:36.777 7f7268c0d700 1 --2- 172.21.2.102:0/2016863033 >> v2:172.21.2.102:3300/0 conn(0x7f72640d1c70 0x7f72640efc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:36.795 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:36.777 7f7262ffd700 1 -- 172.21.2.102:0/2016863033 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1934072056 0 0) 0x7f725c001500 con 0x7f72640d1870 2021-02-10T11:37:36.795 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:36.777 7f7262ffd700 1 -- 172.21.2.102:0/2016863033 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f724c002fe0 con 0x7f72640d1870 2021-02-10T11:37:36.795 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:36.777 7f7268c0d700 1 -- 172.21.2.102:0/2016863033 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f72640d0340 msgr2=0x7f72640ef640 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:37:36.795 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:36.777 7f7268c0d700 1 --2- 172.21.2.102:0/2016863033 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f72640d0340 0x7f72640ef640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:36.795 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:36.777 7f7268c0d700 1 -- 172.21.2.102:0/2016863033 >> v1:172.21.2.102:6789/0 conn(0x7f72640d1870 legacy=0x7f72641b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:36.795 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:36.777 7f7268c0d700 1 -- 172.21.2.102:0/2016863033 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72640f20a0 con 0x7f72640d1c70 2021-02-10T11:37:36.795 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:36.777 7f7268c0d700 1 --2- 172.21.2.102:0/2016863033 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f72640d1c70 0x7f72640efc60 secure :-1 s=READY pgs=2977 cs=0 l=1 rev1=1 rx=0x7f7254006ab0 tx=0x7f7254009b40).ready entity=mon.0 client_cookie=1aa8bf188e912394 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:36.796 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:36.777 7f7262ffd700 1 -- 172.21.2.102:0/2016863033 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f72540022e0 con 0x7f72640d1c70 2021-02-10T11:37:36.796 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:36.778 7f7262ffd700 1 -- 172.21.2.102:0/2016863033 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7254002660 con 0x7f72640d1c70 2021-02-10T11:37:36.796 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:36.778 7f7262ffd700 1 -- 172.21.2.102:0/2016863033 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7254002b40 con 0x7f72640d1c70 2021-02-10T11:37:36.796 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:36.778 7f726a5f0700 1 -- 172.21.2.102:0/2016863033 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f72640d1c70 msgr2=0x7f72640efc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:36.796 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:36.778 7f726a5f0700 1 --2- 172.21.2.102:0/2016863033 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f72640d1c70 0x7f72640efc60 secure :-1 s=READY pgs=2977 cs=0 l=1 rev1=1 rx=0x7f7254006ab0 tx=0x7f7254009b40).stop 2021-02-10T11:37:36.796 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:36.778 7f726a5f0700 1 -- 172.21.2.102:0/2016863033 shutdown_connections 2021-02-10T11:37:36.796 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:36.778 7f726a5f0700 1 --2- 172.21.2.102:0/2016863033 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f72640d1c70 0x7f72640efc60 unknown :-1 s=CLOSED pgs=2977 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:36.796 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:36.778 7f726a5f0700 1 --2- 172.21.2.102:0/2016863033 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f72640d0340 0x7f72640ef640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:36.796 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:36.778 7f726a5f0700 1 -- 172.21.2.102:0/2016863033 >> 172.21.2.102:0/2016863033 conn(0x7f72640ce450 msgr2=0x7f72640d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:36.796 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:36.778 7f726a5f0700 1 -- 172.21.2.102:0/2016863033 shutdown_connections 2021-02-10T11:37:36.796 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:36.778 7f726a5f0700 1 -- 172.21.2.102:0/2016863033 wait complete. 2021-02-10T11:37:36.798 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:36.792 7f726a5f0700 1 Processor -- start 2021-02-10T11:37:36.798 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:36.792 7f726a5f0700 1 -- start start 2021-02-10T11:37:36.798 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:36.793 7f726a5f0700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f72640ada90 0x7f72640ade90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:36.798 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:36.793 7f726a5f0700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7264099d20 0x7f726409a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:36.798 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:36.793 7f726a5f0700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f726409a740 0x7f7264097bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:36.798 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:36.793 7f726a5f0700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72640997a0 con 0x7f726409a740 2021-02-10T11:37:36.798 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:36.793 7f726a5f0700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f726418f0c0 con 0x7f7264099d20 2021-02-10T11:37:36.798 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:36.793 7f726a5f0700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f726409ab40 con 0x7f72640ada90 2021-02-10T11:37:36.799 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:36.793 7f7268c0d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f726409a740 0x7f7264097bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:36.799 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:36.793 7f7268c0d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f726409a740 0x7f7264097bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:38808/0 (socket says 172.21.2.102:38808) 2021-02-10T11:37:36.799 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:36.793 7f7268c0d700 1 -- 172.21.2.102:0/1205937456 learned_addr learned my addr 172.21.2.102:0/1205937456 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:37:36.799 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:36.793 7f72637fe700 1 --2- 172.21.2.102:0/1205937456 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7264099d20 0x7f726409a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:36.799 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:36.793 7f7263fff700 1 --2- 172.21.2.102:0/1205937456 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f72640ada90 0x7f72640ade90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:36.799 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:36.794 7f7268c0d700 1 -- 172.21.2.102:0/1205937456 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f72640ada90 msgr2=0x7f72640ade90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:36.799 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:36.794 7f7268c0d700 1 --2- 172.21.2.102:0/1205937456 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f72640ada90 0x7f72640ade90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:36.799 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:36.794 7f7268c0d700 1 -- 172.21.2.102:0/1205937456 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7264099d20 msgr2=0x7f726409a120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:36.799 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:36.794 7f7268c0d700 1 --2- 172.21.2.102:0/1205937456 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7264099d20 0x7f726409a120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:36.799 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:36.794 7f7268c0d700 1 -- 172.21.2.102:0/1205937456 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f726409ad60 con 0x7f726409a740 2021-02-10T11:37:36.799 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:36.794 7f7268c0d700 1 --2- 172.21.2.102:0/1205937456 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f726409a740 0x7f7264097bc0 secure :-1 s=READY pgs=2978 cs=0 l=1 rev1=1 rx=0x7f725400b5f0 tx=0x7f725400b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:36.799 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:36.794 7f7260ff9700 1 -- 172.21.2.102:0/1205937456 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f725400bd80 con 0x7f726409a740 2021-02-10T11:37:36.800 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:36.794 7f7260ff9700 1 -- 172.21.2.102:0/1205937456 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7254006020 con 0x7f726409a740 2021-02-10T11:37:36.800 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:36.794 7f7260ff9700 1 -- 172.21.2.102:0/1205937456 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f72540054a0 con 0x7f726409a740 2021-02-10T11:37:36.800 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:36.794 7f726a5f0700 1 -- 172.21.2.102:0/1205937456 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f726409ad60 con 0x7f726409a740 2021-02-10T11:37:36.800 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:36.794 7f726a5f0700 1 -- 172.21.2.102:0/1205937456 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72640984d0 con 0x7f726409a740 2021-02-10T11:37:36.800 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:36.795 7f7260ff9700 1 -- 172.21.2.102:0/1205937456 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f7254006020 con 0x7f726409a740 2021-02-10T11:37:36.800 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:36.795 7f7260ff9700 1 --2- 172.21.2.102:0/1205937456 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f72440186c0 0x7f724401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:36.800 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:36.795 7f7260ff9700 1 -- 172.21.2.102:0/1205937456 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(171..171 src has 1..171) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f7254032560 con 0x7f726409a740 2021-02-10T11:37:36.800 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:36.795 7f7263fff700 1 --2- 172.21.2.102:0/1205937456 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f72440186c0 0x7f724401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:36.800 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:36.795 7f7263fff700 1 --2- 172.21.2.102:0/1205937456 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f72440186c0 0x7f724401abc0 crc :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:36.829 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:36.827 7f726a5f0700 1 -- 172.21.2.102:0/1205937456 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72640ebc00 con 0x7f726409a740 2021-02-10T11:37:36.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:36.829 7f7260ff9700 1 -- 172.21.2.102:0/1205937456 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f72540093c0 con 0x7f726409a740 2021-02-10T11:37:36.954 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:36.952 7f726a5f0700 1 -- 172.21.2.102:0/1205937456 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f72640f0b10 con 0x7f726409a740 2021-02-10T11:37:36.955 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:36.953 7f7260ff9700 1 -- 172.21.2.102:0/1205937456 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v171) v1 ==== 74+0+14020 (secure 0 0 0) 0x7f7254006000 con 0x7f726409a740 2021-02-10T11:37:36.955 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:37:36.955 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":171,"fsid":"68db30ea-dbce-4aef-a6df-f01c6fea0513","created":"2021-02-10 11:03:06.331557","modified":"2021-02-10 11:37:36.348032","last_up_change":"2021-02-10 11:03:32.690562","last_in_change":"2021-02-10 11:03:32.690562","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":37,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:03:37.322519","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":36,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:37:34.298334","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"171","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":37,"pool_name":"cephfs_data","create_time":"2021-02-10 11:37:35.337127","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"171","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"be691b4d-a039-408a-af80-9d53f02b6f1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6818","nonce":36088},{"type":"v1","addr":"172.21.2.102:6820","nonce":36088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6822","nonce":36088},{"type":"v1","addr":"172.21.2.102:6824","nonce":36088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6830","nonce":36088},{"type":"v1","addr":"172.21.2.102:6832","nonce":36088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6826","nonce":36088},{"type":"v1","addr":"172.21.2.102:6828","nonce":36088}]},"public_addr":"172.21.2.102:6820/36088","cluster_addr":"172.21.2.102:6824/36088","heartbeat_back_addr":"172.21.2.102:6832/36088","heartbeat_front_addr":"172.21.2.102:6828/36088","state":["exists","up"]},{"osd":1,"uuid":"64e3ccc8-9c69-4d7c-865e-af8e83822635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6819","nonce":36087},{"type":"v1","addr":"172.21.2.102:6821","nonce":36087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6823","nonce":36087},{"type":"v1","addr":"172.21.2.102:6825","nonce":36087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6831","nonce":36087},{"type":"v1","addr":"172.21.2.102:6833","nonce":36087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6827","nonce":36087},{"type":"v1","addr":"172.21.2.102:6829","nonce":36087}]},"public_addr":"172.21.2.102:6821/36087","cluster_addr":"172.21.2.102:6825/36087","heartbeat_back_addr":"172.21.2.102:6833/36087","heartbeat_front_addr":"172.21.2.102:6829/36087","state":["exists","up"]},{"osd":2,"uuid":"153700af-747f-481f-bdf4-d6072d3a842f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6810","nonce":36089},{"type":"v1","addr":"172.21.2.102:6811","nonce":36089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6812","nonce":36089},{"type":"v1","addr":"172.21.2.102:6813","nonce":36089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6816","nonce":36089},{"type":"v1","addr":"172.21.2.102:6817","nonce":36089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6814","nonce":36089},{"type":"v1","addr":"172.21.2.102:6815","nonce":36089}]},"public_addr":"172.21.2.102:6811/36089","cluster_addr":"172.21.2.102:6813/36089","heartbeat_back_addr":"172.21.2.102:6817/36089","heartbeat_front_addr":"172.21.2.102:6815/36089","state":["exists","up"]},{"osd":3,"uuid":"cbc678f4-83a9-45e6-8830-2f7c232b1ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6802","nonce":36090},{"type":"v1","addr":"172.21.2.102:6803","nonce":36090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6804","nonce":36090},{"type":"v1","addr":"172.21.2.102:6805","nonce":36090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6808","nonce":36090},{"type":"v1","addr":"172.21.2.102:6809","nonce":36090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6806","nonce":36090},{"type":"v1","addr":"172.21.2.102:6807","nonce":36090}]},"public_addr":"172.21.2.102:6803/36090","cluster_addr":"172.21.2.102:6805/36090","heartbeat_back_addr":"172.21.2.102:6809/36090","heartbeat_front_addr":"172.21.2.102:6807/36090","state":["exists","up"]},{"osd":4,"uuid":"c01e4008-b258-4f10-afd2-9e7f37214351","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6808","nonce":36368},{"type":"v1","addr":"172.21.2.113:6811","nonce":36368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6814","nonce":36368},{"type":"v1","addr":"172.21.2.113:6818","nonce":36368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6826","nonce":36368},{"type":"v1","addr":"172.21.2.113:6828","nonce":36368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6820","nonce":36368},{"type":"v1","addr":"172.21.2.113:6824","nonce":36368}]},"public_addr":"172.21.2.113:6811/36368","cluster_addr":"172.21.2.113:6818/36368","heartbeat_back_addr":"172.21.2.113:6828/36368","heartbeat_front_addr":"172.21.2.113:6824/36368","state":["exists","up"]},{"osd":5,"uuid":"551f6007-daf8-4a98-8ee0-c2b16bcccb5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6802","nonce":36366},{"type":"v1","addr":"172.21.2.113:6805","nonce":36366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6807","nonce":36366},{"type":"v1","addr":"172.21.2.113:6810","nonce":36366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6819","nonce":36366},{"type":"v1","addr":"172.21.2.113:6822","nonce":36366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6812","nonce":36366},{"type":"v1","addr":"172.21.2.113:6815","nonce":36366}]},"public_addr":"172.21.2.113:6805/36366","cluster_addr":"172.21.2.113:6810/36366","heartbeat_back_addr":"172.21.2.113:6822/36366","heartbeat_front_addr":"172.21.2.113:6815/36366","state":["exists","up"]},{"osd":6,"uuid":"bf67c040-0e50-48be-b102-c3943bc9ebef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6800","nonce":36367},{"type":"v1","addr":"172.21.2.113:6801","nonce":36367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6803","nonce":36367},{"type":"v1","addr":"172.21.2.113:6804","nonce":36367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6813","nonce":36367},{"type":"v1","addr":"172.21.2.113:6816","nonce":36367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6806","nonce":36367},{"type":"v1","addr":"172.21.2.113:6809","nonce":36367}]},"public_addr":"172.21.2.113:6801/36367","cluster_addr":"172.21.2.113:6804/36367","heartbeat_back_addr":"172.21.2.113:6816/36367","heartbeat_front_addr":"172.21.2.113:6809/36367","state":["exists","up"]},{"osd":7,"uuid":"f5461f80-018d-428f-be89-2eb2bcbe7bb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":169,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6817","nonce":36365},{"type":"v1","addr":"172.21.2.113:6821","nonce":36365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6823","nonce":36365},{"type":"v1","addr":"172.21.2.113:6825","nonce":36365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6830","nonce":36365},{"type":"v1","addr":"172.21.2.113:6831","nonce":36365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6827","nonce":36365},{"type":"v1","addr":"172.21.2.113:6829","nonce":36365}]},"public_addr":"172.21.2.113:6821/36365","cluster_addr":"172.21.2.113:6825/36365","heartbeat_back_addr":"172.21.2.113:6831/36365","heartbeat_front_addr":"172.21.2.113:6829/36365","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:37:36.957 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:36.954 7f726a5f0700 1 -- 172.21.2.102:0/1205937456 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f72440186c0 msgr2=0x7f724401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:36.957 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:36.954 7f726a5f0700 1 --2- 172.21.2.102:0/1205937456 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f72440186c0 0x7f724401abc0 crc :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:36.957 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:36.954 7f726a5f0700 1 -- 172.21.2.102:0/1205937456 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f726409a740 msgr2=0x7f7264097bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:36.957 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:36.954 7f726a5f0700 1 --2- 172.21.2.102:0/1205937456 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f726409a740 0x7f7264097bc0 secure :-1 s=READY pgs=2978 cs=0 l=1 rev1=1 rx=0x7f725400b5f0 tx=0x7f725400b990).stop 2021-02-10T11:37:36.957 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:36.954 7f726a5f0700 1 -- 172.21.2.102:0/1205937456 shutdown_connections 2021-02-10T11:37:36.957 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:36.954 7f726a5f0700 1 --2- 172.21.2.102:0/1205937456 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f726409a740 0x7f7264097bc0 unknown :-1 s=CLOSED pgs=2978 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:36.957 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:36.954 7f726a5f0700 1 --2- 172.21.2.102:0/1205937456 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f72440186c0 0x7f724401abc0 unknown :-1 s=CLOSED pgs=1307 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:36.957 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:36.954 7f726a5f0700 1 --2- 172.21.2.102:0/1205937456 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7264099d20 0x7f726409a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:36.957 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:36.954 7f726a5f0700 1 --2- 172.21.2.102:0/1205937456 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f72640ada90 0x7f72640ade90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:36.957 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:36.954 7f726a5f0700 1 -- 172.21.2.102:0/1205937456 >> 172.21.2.102:0/1205937456 conn(0x7f72640f0550 msgr2=0x7f7264191730 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:36.957 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:36.955 7f726a5f0700 1 -- 172.21.2.102:0/1205937456 shutdown_connections 2021-02-10T11:37:36.957 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:36.955 7f726a5f0700 1 -- 172.21.2.102:0/1205937456 wait complete. 2021-02-10T11:37:36.965 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:37:37.395 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:37.377 7f1797326700 1 Processor -- start 2021-02-10T11:37:37.396 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:37.377 7f1797326700 1 -- start start 2021-02-10T11:37:37.396 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:37.377 7f1797326700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f17900d0340 0x7f17900ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:37.396 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:37.377 7f1797326700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f17900d1c70 0x7f17900efc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:37.396 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:37.377 7f1797326700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17900f0b10 con 0x7f17900d1870 2021-02-10T11:37:37.396 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:37.377 7f1797326700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17900e5ed0 con 0x7f17900d0340 2021-02-10T11:37:37.396 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:37.377 7f1797326700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17901b0be0 con 0x7f17900d1c70 2021-02-10T11:37:37.396 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:37.378 7f1794941700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f17900d0340 0x7f17900ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:37.396 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:37.378 7f1795142700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f17900d1870 0x7f17901b03f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:59270/0 (socket says 172.21.2.102:59270) 2021-02-10T11:37:37.396 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:37.378 7f1795142700 1 -- 172.21.2.102:0/1559845380 learned_addr learned my addr 172.21.2.102:0/1559845380 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:37:37.396 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:37.378 7f1795943700 1 --2- 172.21.2.102:0/1559845380 >> v2:172.21.2.113:3300/0 conn(0x7f17900d1c70 0x7f17900efc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:37.396 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:37.378 7f1794941700 1 -- 172.21.2.102:0/1559845380 >> v2:172.21.2.113:3300/0 conn(0x7f17900d1c70 msgr2=0x7f17900efc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:37:37.396 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:37.378 7f1794941700 1 --2- 172.21.2.102:0/1559845380 >> v2:172.21.2.113:3300/0 conn(0x7f17900d1c70 0x7f17900efc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:37.397 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:37.378 7f1794941700 1 -- 172.21.2.102:0/1559845380 >> v1:172.21.2.102:6789/0 conn(0x7f17900d1870 legacy=0x7f17901b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:37.397 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:37.378 7f1794941700 1 -- 172.21.2.102:0/1559845380 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17900f20a0 con 0x7f17900d0340 2021-02-10T11:37:37.397 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:37.378 7f1794941700 1 --2- 172.21.2.102:0/1559845380 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f17900d0340 0x7f17900ef640 secure :-1 s=READY pgs=2980 cs=0 l=1 rev1=1 rx=0x7f1784006650 tx=0x7f1784009a80).ready entity=mon.0 client_cookie=6fa3d7ec1cdbe8fa server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:37.397 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:37.378 7f1783fff700 1 -- 172.21.2.102:0/1559845380 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f178400c0c0 con 0x7f17900d0340 2021-02-10T11:37:37.397 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:37.378 7f1783fff700 1 -- 172.21.2.102:0/1559845380 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f178400c340 con 0x7f17900d0340 2021-02-10T11:37:37.397 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:37.378 7f1783fff700 1 -- 172.21.2.102:0/1559845380 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f178400c860 con 0x7f17900d0340 2021-02-10T11:37:37.397 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:37.378 7f1797326700 1 -- 172.21.2.102:0/1559845380 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f17900d0340 msgr2=0x7f17900ef640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:37.397 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:37.378 7f1797326700 1 --2- 172.21.2.102:0/1559845380 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f17900d0340 0x7f17900ef640 secure :-1 s=READY pgs=2980 cs=0 l=1 rev1=1 rx=0x7f1784006650 tx=0x7f1784009a80).stop 2021-02-10T11:37:37.397 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:37.379 7f1797326700 1 -- 172.21.2.102:0/1559845380 shutdown_connections 2021-02-10T11:37:37.397 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:37.379 7f1797326700 1 --2- 172.21.2.102:0/1559845380 >> v2:172.21.2.113:3300/0 conn(0x7f17900d1c70 0x7f17900efc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:37.397 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:37.379 7f1797326700 1 --2- 172.21.2.102:0/1559845380 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f17900d0340 0x7f17900ef640 unknown :-1 s=CLOSED pgs=2980 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:37.397 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:37.379 7f1797326700 1 -- 172.21.2.102:0/1559845380 >> 172.21.2.102:0/1559845380 conn(0x7f17900ce450 msgr2=0x7f17900d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:37.397 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:37.379 7f1797326700 1 -- 172.21.2.102:0/1559845380 shutdown_connections 2021-02-10T11:37:37.398 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:37.379 7f1797326700 1 -- 172.21.2.102:0/1559845380 wait complete. 2021-02-10T11:37:37.399 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:37.394 7f1797326700 1 Processor -- start 2021-02-10T11:37:37.399 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:37.394 7f1797326700 1 -- start start 2021-02-10T11:37:37.399 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:37.394 7f1797326700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f17900ada90 0x7f17900ade90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:37.400 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:37.394 7f1797326700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1790099d20 0x7f179009a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:37.400 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:37.394 7f1797326700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f179009a740 0x7f1790097bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:37.400 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:37.394 7f1797326700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17900997a0 con 0x7f1790099d20 2021-02-10T11:37:37.400 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:37.394 7f1797326700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f179018f0c0 con 0x7f17900ada90 2021-02-10T11:37:37.400 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:37.394 7f1797326700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f179009ab40 con 0x7f179009a740 2021-02-10T11:37:37.400 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:37.394 7f1795142700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f17900ada90 0x7f17900ade90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:37.400 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:37.395 7f1795142700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f17900ada90 0x7f17900ade90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.113:3300/0 says I am v2:172.21.2.102:42252/0 (socket says 172.21.2.102:42252) 2021-02-10T11:37:37.400 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:37.395 7f1795142700 1 -- 172.21.2.102:0/2949098108 learned_addr learned my addr 172.21.2.102:0/2949098108 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:37:37.400 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:37.395 7f1794941700 1 --2- 172.21.2.102:0/2949098108 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1790099d20 0x7f179009a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:37.400 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:37.395 7f1795943700 1 --2- 172.21.2.102:0/2949098108 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f179009a740 0x7f1790097bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:37.400 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:37.395 7f1795142700 1 -- 172.21.2.102:0/2949098108 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f179009a740 msgr2=0x7f1790097bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:37.400 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:37.395 7f1795142700 1 --2- 172.21.2.102:0/2949098108 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f179009a740 0x7f1790097bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:37.400 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:37.395 7f1795142700 1 -- 172.21.2.102:0/2949098108 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1790099d20 msgr2=0x7f179009a120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:37.401 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:37.395 7f1795142700 1 --2- 172.21.2.102:0/2949098108 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1790099d20 0x7f179009a120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:37.401 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:37.395 7f1795142700 1 -- 172.21.2.102:0/2949098108 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f179009ad60 con 0x7f17900ada90 2021-02-10T11:37:37.401 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:37.395 7f1795142700 1 --2- 172.21.2.102:0/2949098108 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f17900ada90 0x7f17900ade90 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 rx=0x7f178c00a570 tx=0x7f178c00a910).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:37.401 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:37.395 7f1781ffb700 1 -- 172.21.2.102:0/2949098108 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f178c00ad00 con 0x7f17900ada90 2021-02-10T11:37:37.401 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:37.395 7f1781ffb700 1 -- 172.21.2.102:0/2949098108 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f178c001180 con 0x7f17900ada90 2021-02-10T11:37:37.401 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:37.395 7f1797326700 1 -- 172.21.2.102:0/2949098108 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f179009ad60 con 0x7f17900ada90 2021-02-10T11:37:37.401 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:37.395 7f1797326700 1 -- 172.21.2.102:0/2949098108 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f179009ad60 con 0x7f17900ada90 2021-02-10T11:37:37.401 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:37.395 7f1781ffb700 1 -- 172.21.2.102:0/2949098108 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f178c004620 con 0x7f17900ada90 2021-02-10T11:37:37.401 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:37.396 7f1781ffb700 1 -- 172.21.2.102:0/2949098108 <== mon.1 v2:172.21.2.113:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f178c0043a0 con 0x7f17900ada90 2021-02-10T11:37:37.401 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:37.396 7f1781ffb700 1 --2- 172.21.2.102:0/2949098108 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1770018730 0x7f177001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:37.401 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:37.396 7f1781ffb700 1 -- 172.21.2.102:0/2949098108 <== mon.1 v2:172.21.2.113:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f178c0327b0 con 0x7f17900ada90 2021-02-10T11:37:37.401 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:37.396 7f1794941700 1 --2- 172.21.2.102:0/2949098108 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1770018730 0x7f177001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:37.401 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:37.396 7f1794941700 1 --2- 172.21.2.102:0/2949098108 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1770018730 0x7f177001abc0 crc :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:37.429 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:37.427 7f1797326700 1 -- 172.21.2.102:0/2949098108 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17900ebc00 con 0x7f17900ada90 2021-02-10T11:37:37.432 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:37.430 7f1781ffb700 1 -- 172.21.2.102:0/2949098108 <== mon.1 v2:172.21.2.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f178c000c90 con 0x7f17900ada90 2021-02-10T11:37:37.558 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:37.556 7f1797326700 1 -- 172.21.2.102:0/2949098108 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f17900ebc00 con 0x7f17900ada90 2021-02-10T11:37:37.559 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:37.557 7f1781ffb700 1 -- 172.21.2.102:0/2949098108 <== mon.1 v2:172.21.2.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v172) v1 ==== 74+0+14007 (secure 0 0 0) 0x7f178c001180 con 0x7f17900ada90 2021-02-10T11:37:37.559 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:37:37.559 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":172,"fsid":"68db30ea-dbce-4aef-a6df-f01c6fea0513","created":"2021-02-10 11:03:06.331557","modified":"2021-02-10 11:37:37.349944","last_up_change":"2021-02-10 11:03:32.690562","last_in_change":"2021-02-10 11:03:32.690562","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":37,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:03:37.322519","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":36,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:37:34.298334","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"171","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":37,"pool_name":"cephfs_data","create_time":"2021-02-10 11:37:35.337127","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"be691b4d-a039-408a-af80-9d53f02b6f1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6818","nonce":36088},{"type":"v1","addr":"172.21.2.102:6820","nonce":36088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6822","nonce":36088},{"type":"v1","addr":"172.21.2.102:6824","nonce":36088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6830","nonce":36088},{"type":"v1","addr":"172.21.2.102:6832","nonce":36088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6826","nonce":36088},{"type":"v1","addr":"172.21.2.102:6828","nonce":36088}]},"public_addr":"172.21.2.102:6820/36088","cluster_addr":"172.21.2.102:6824/36088","heartbeat_back_addr":"172.21.2.102:6832/36088","heartbeat_front_addr":"172.21.2.102:6828/36088","state":["exists","up"]},{"osd":1,"uuid":"64e3ccc8-9c69-4d7c-865e-af8e83822635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6819","nonce":36087},{"type":"v1","addr":"172.21.2.102:6821","nonce":36087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6823","nonce":36087},{"type":"v1","addr":"172.21.2.102:6825","nonce":36087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6831","nonce":36087},{"type":"v1","addr":"172.21.2.102:6833","nonce":36087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6827","nonce":36087},{"type":"v1","addr":"172.21.2.102:6829","nonce":36087}]},"public_addr":"172.21.2.102:6821/36087","cluster_addr":"172.21.2.102:6825/36087","heartbeat_back_addr":"172.21.2.102:6833/36087","heartbeat_front_addr":"172.21.2.102:6829/36087","state":["exists","up"]},{"osd":2,"uuid":"153700af-747f-481f-bdf4-d6072d3a842f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6810","nonce":36089},{"type":"v1","addr":"172.21.2.102:6811","nonce":36089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6812","nonce":36089},{"type":"v1","addr":"172.21.2.102:6813","nonce":36089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6816","nonce":36089},{"type":"v1","addr":"172.21.2.102:6817","nonce":36089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6814","nonce":36089},{"type":"v1","addr":"172.21.2.102:6815","nonce":36089}]},"public_addr":"172.21.2.102:6811/36089","cluster_addr":"172.21.2.102:6813/36089","heartbeat_back_addr":"172.21.2.102:6817/36089","heartbeat_front_addr":"172.21.2.102:6815/36089","state":["exists","up"]},{"osd":3,"uuid":"cbc678f4-83a9-45e6-8830-2f7c232b1ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6802","nonce":36090},{"type":"v1","addr":"172.21.2.102:6803","nonce":36090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6804","nonce":36090},{"type":"v1","addr":"172.21.2.102:6805","nonce":36090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6808","nonce":36090},{"type":"v1","addr":"172.21.2.102:6809","nonce":36090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6806","nonce":36090},{"type":"v1","addr":"172.21.2.102:6807","nonce":36090}]},"public_addr":"172.21.2.102:6803/36090","cluster_addr":"172.21.2.102:6805/36090","heartbeat_back_addr":"172.21.2.102:6809/36090","heartbeat_front_addr":"172.21.2.102:6807/36090","state":["exists","up"]},{"osd":4,"uuid":"c01e4008-b258-4f10-afd2-9e7f37214351","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6808","nonce":36368},{"type":"v1","addr":"172.21.2.113:6811","nonce":36368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6814","nonce":36368},{"type":"v1","addr":"172.21.2.113:6818","nonce":36368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6826","nonce":36368},{"type":"v1","addr":"172.21.2.113:6828","nonce":36368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6820","nonce":36368},{"type":"v1","addr":"172.21.2.113:6824","nonce":36368}]},"public_addr":"172.21.2.113:6811/36368","cluster_addr":"172.21.2.113:6818/36368","heartbeat_back_addr":"172.21.2.113:6828/36368","heartbeat_front_addr":"172.21.2.113:6824/36368","state":["exists","up"]},{"osd":5,"uuid":"551f6007-daf8-4a98-8ee0-c2b16bcccb5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6802","nonce":36366},{"type":"v1","addr":"172.21.2.113:6805","nonce":36366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6807","nonce":36366},{"type":"v1","addr":"172.21.2.113:6810","nonce":36366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6819","nonce":36366},{"type":"v1","addr":"172.21.2.113:6822","nonce":36366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6812","nonce":36366},{"type":"v1","addr":"172.21.2.113:6815","nonce":36366}]},"public_addr":"172.21.2.113:6805/36366","cluster_addr":"172.21.2.113:6810/36366","heartbeat_back_addr":"172.21.2.113:6822/36366","heartbeat_front_addr":"172.21.2.113:6815/36366","state":["exists","up"]},{"osd":6,"uuid":"bf67c040-0e50-48be-b102-c3943bc9ebef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6800","nonce":36367},{"type":"v1","addr":"172.21.2.113:6801","nonce":36367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6803","nonce":36367},{"type":"v1","addr":"172.21.2.113:6804","nonce":36367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6813","nonce":36367},{"type":"v1","addr":"172.21.2.113:6816","nonce":36367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6806","nonce":36367},{"type":"v1","addr":"172.21.2.113:6809","nonce":36367}]},"public_addr":"172.21.2.113:6801/36367","cluster_addr":"172.21.2.113:6804/36367","heartbeat_back_addr":"172.21.2.113:6816/36367","heartbeat_front_addr":"172.21.2.113:6809/36367","state":["exists","up"]},{"osd":7,"uuid":"f5461f80-018d-428f-be89-2eb2bcbe7bb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":169,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6817","nonce":36365},{"type":"v1","addr":"172.21.2.113:6821","nonce":36365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6823","nonce":36365},{"type":"v1","addr":"172.21.2.113:6825","nonce":36365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6830","nonce":36365},{"type":"v1","addr":"172.21.2.113:6831","nonce":36365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6827","nonce":36365},{"type":"v1","addr":"172.21.2.113:6829","nonce":36365}]},"public_addr":"172.21.2.113:6821/36365","cluster_addr":"172.21.2.113:6825/36365","heartbeat_back_addr":"172.21.2.113:6831/36365","heartbeat_front_addr":"172.21.2.113:6829/36365","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:37:37.560 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:37.558 7f1797326700 1 -- 172.21.2.102:0/2949098108 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1770018730 msgr2=0x7f177001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:37.560 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:37.558 7f1797326700 1 --2- 172.21.2.102:0/2949098108 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1770018730 0x7f177001abc0 crc :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:37.560 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:37.558 7f1797326700 1 -- 172.21.2.102:0/2949098108 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f17900ada90 msgr2=0x7f17900ade90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:37.560 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:37.558 7f1797326700 1 --2- 172.21.2.102:0/2949098108 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f17900ada90 0x7f17900ade90 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 rx=0x7f178c00a570 tx=0x7f178c00a910).stop 2021-02-10T11:37:37.561 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:37.558 7f1797326700 1 -- 172.21.2.102:0/2949098108 shutdown_connections 2021-02-10T11:37:37.561 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:37.558 7f1797326700 1 --2- 172.21.2.102:0/2949098108 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f179009a740 0x7f1790097bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:37.561 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:37.558 7f1797326700 1 --2- 172.21.2.102:0/2949098108 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1790099d20 0x7f179009a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:37.561 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:37.558 7f1797326700 1 --2- 172.21.2.102:0/2949098108 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1770018730 0x7f177001abc0 unknown :-1 s=CLOSED pgs=1308 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:37.561 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:37.558 7f1797326700 1 --2- 172.21.2.102:0/2949098108 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f17900ada90 0x7f17900ade90 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:37.561 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:37.558 7f1797326700 1 -- 172.21.2.102:0/2949098108 >> 172.21.2.102:0/2949098108 conn(0x7f17900f0550 msgr2=0x7f1790191730 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:37.561 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:37.558 7f1797326700 1 -- 172.21.2.102:0/2949098108 shutdown_connections 2021-02-10T11:37:37.561 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:37.559 7f1797326700 1 -- 172.21.2.102:0/2949098108 wait complete. 2021-02-10T11:37:37.569 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2021-02-10T11:37:38.003 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:37.984 7ff062139700 1 Processor -- start 2021-02-10T11:37:38.003 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:37.984 7ff062139700 1 -- start start 2021-02-10T11:37:38.003 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:37.985 7ff062139700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7ff05c0d1870 0x7ff05c1b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:38.003 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:37.985 7ff062139700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7ff05c1b0ae0 0x7ff05c108940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:38.003 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:37.985 7ff062139700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff05c0d1c70 0x7ff05c108f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:38.003 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:37.985 7ff062139700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff05c0fe1c0 con 0x7ff05c1b0ae0 2021-02-10T11:37:38.003 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:37.985 7ff062139700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff05c109a60 con 0x7ff05c0d1870 2021-02-10T11:37:38.003 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:37.985 7ff062139700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff05c088070 con 0x7ff05c0d1c70 2021-02-10T11:37:38.003 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:37.985 7ff05affd700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7ff05c1b0ae0 0x7ff05c108940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:38.003 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:37.985 7ff05affd700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7ff05c1b0ae0 0x7ff05c108940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:38828/0 (socket says 172.21.2.102:38828) 2021-02-10T11:37:38.004 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:37.985 7ff05affd700 1 -- 172.21.2.102:0/665615657 learned_addr learned my addr 172.21.2.102:0/665615657 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:37:38.004 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:37.985 7ff05b7fe700 1 --2- 172.21.2.102:0/665615657 >> v2:172.21.2.113:3300/0 conn(0x7ff05c0d1870 0x7ff05c1b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:38.004 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:37.985 7ff05bfff700 1 --2- 172.21.2.102:0/665615657 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff05c0d1c70 0x7ff05c108f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:38.004 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:37.985 7ff05affd700 1 -- 172.21.2.102:0/665615657 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff05c0d1c70 msgr2=0x7ff05c108f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:37:38.004 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:37.985 7ff05affd700 1 --2- 172.21.2.102:0/665615657 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff05c0d1c70 0x7ff05c108f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:38.004 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:37.985 7ff05affd700 1 -- 172.21.2.102:0/665615657 >> v2:172.21.2.113:3300/0 conn(0x7ff05c0d1870 msgr2=0x7ff05c1b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:37:38.004 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:37.985 7ff05affd700 1 --2- 172.21.2.102:0/665615657 >> v2:172.21.2.113:3300/0 conn(0x7ff05c0d1870 0x7ff05c1b04c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:38.004 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:37.985 7ff05affd700 1 -- 172.21.2.102:0/665615657 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff05c109580 con 0x7ff05c1b0ae0 2021-02-10T11:37:38.004 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:37.986 7ff05bfff700 1 --2- 172.21.2.102:0/665615657 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff05c0d1c70 0x7ff05c108f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:37:38.004 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:37.986 7ff05affd700 1 --2- 172.21.2.102:0/665615657 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff05c1b0ae0 0x7ff05c108940 secure :-1 s=READY pgs=2981 cs=0 l=1 rev1=1 rx=0x7ff044006700 tx=0x7ff044009b80).ready entity=mon.0 client_cookie=6f25239521f34252 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:38.004 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:37.986 7ff05a7fc700 1 -- 172.21.2.102:0/665615657 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff04400c0c0 con 0x7ff05c1b0ae0 2021-02-10T11:37:38.004 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:37.986 7ff05a7fc700 1 -- 172.21.2.102:0/665615657 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff04400c340 con 0x7ff05c1b0ae0 2021-02-10T11:37:38.004 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:37.986 7ff05a7fc700 1 -- 172.21.2.102:0/665615657 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff04400c820 con 0x7ff05c1b0ae0 2021-02-10T11:37:38.005 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:37.986 7ff062139700 1 -- 172.21.2.102:0/665615657 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff05c1b0ae0 msgr2=0x7ff05c108940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:38.005 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:37.986 7ff062139700 1 --2- 172.21.2.102:0/665615657 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff05c1b0ae0 0x7ff05c108940 secure :-1 s=READY pgs=2981 cs=0 l=1 rev1=1 rx=0x7ff044006700 tx=0x7ff044009b80).stop 2021-02-10T11:37:38.005 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:37.986 7ff062139700 1 -- 172.21.2.102:0/665615657 shutdown_connections 2021-02-10T11:37:38.005 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:37.986 7ff062139700 1 --2- 172.21.2.102:0/665615657 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff05c0d1c70 0x7ff05c108f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:38.005 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:37.986 7ff062139700 1 --2- 172.21.2.102:0/665615657 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff05c1b0ae0 0x7ff05c108940 unknown :-1 s=CLOSED pgs=2981 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:38.005 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:37.986 7ff062139700 1 --2- 172.21.2.102:0/665615657 >> v2:172.21.2.113:3300/0 conn(0x7ff05c0d1870 0x7ff05c1b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:38.005 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:37.986 7ff062139700 1 -- 172.21.2.102:0/665615657 >> 172.21.2.102:0/665615657 conn(0x7ff05c0ce420 msgr2=0x7ff05c0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:38.005 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:37.986 7ff062139700 1 -- 172.21.2.102:0/665615657 shutdown_connections 2021-02-10T11:37:38.005 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:37.986 7ff062139700 1 -- 172.21.2.102:0/665615657 wait complete. 2021-02-10T11:37:38.007 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:38.001 7ff062139700 1 Processor -- start 2021-02-10T11:37:38.007 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:38.001 7ff062139700 1 -- start start 2021-02-10T11:37:38.007 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:38.001 7ff062139700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff05c0a7950 0x7ff05c0b8df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:38.007 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:38.001 7ff062139700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff05c1a7490 0x7ff05c1a7890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:38.007 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:38.001 7ff062139700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff05c1a7eb0 0x7ff05c0ac630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:38.007 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:38.001 7ff062139700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff05c0a7fb0 con 0x7ff05c1a7490 2021-02-10T11:37:38.008 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:38.001 7ff062139700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff05c0acc50 con 0x7ff05c1a7eb0 2021-02-10T11:37:38.008 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:38.001 7ff062139700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff05c0ace30 con 0x7ff05c0a7950 2021-02-10T11:37:38.008 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:38.002 7ff05affd700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff05c1a7490 0x7ff05c1a7890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:38.008 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:38.002 7ff05affd700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff05c1a7490 0x7ff05c1a7890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:38832/0 (socket says 172.21.2.102:38832) 2021-02-10T11:37:38.008 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:38.002 7ff05affd700 1 -- 172.21.2.102:0/831224807 learned_addr learned my addr 172.21.2.102:0/831224807 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:37:38.008 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:38.002 7ff05b7fe700 1 --2- 172.21.2.102:0/831224807 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff05c0a7950 0x7ff05c0b8df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:38.008 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:38.002 7ff05bfff700 1 --2- 172.21.2.102:0/831224807 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff05c1a7eb0 0x7ff05c0ac630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:38.008 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:38.002 7ff05affd700 1 -- 172.21.2.102:0/831224807 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff05c0a7950 msgr2=0x7ff05c0b8df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:38.008 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:38.002 7ff05affd700 1 --2- 172.21.2.102:0/831224807 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff05c0a7950 0x7ff05c0b8df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:38.008 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:38.002 7ff05affd700 1 -- 172.21.2.102:0/831224807 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff05c1a7eb0 msgr2=0x7ff05c0ac630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:38.008 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:38.002 7ff05affd700 1 --2- 172.21.2.102:0/831224807 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff05c1a7eb0 0x7ff05c0ac630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:38.008 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:38.002 7ff05affd700 1 -- 172.21.2.102:0/831224807 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff05c0ad050 con 0x7ff05c1a7490 2021-02-10T11:37:38.008 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:38.002 7ff05affd700 1 --2- 172.21.2.102:0/831224807 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff05c1a7490 0x7ff05c1a7890 secure :-1 s=READY pgs=2982 cs=0 l=1 rev1=1 rx=0x7ff044000bd0 tx=0x7ff04400b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:38.009 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:38.002 7ff053fff700 1 -- 172.21.2.102:0/831224807 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff044009100 con 0x7ff05c1a7490 2021-02-10T11:37:38.009 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:38.002 7ff053fff700 1 -- 172.21.2.102:0/831224807 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0440052e0 con 0x7ff05c1a7490 2021-02-10T11:37:38.009 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:38.002 7ff062139700 1 -- 172.21.2.102:0/831224807 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff05c0ad050 con 0x7ff05c1a7490 2021-02-10T11:37:38.009 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:38.002 7ff062139700 1 -- 172.21.2.102:0/831224807 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff05c0ad050 con 0x7ff05c1a7490 2021-02-10T11:37:38.009 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:38.002 7ff053fff700 1 -- 172.21.2.102:0/831224807 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff0440057c0 con 0x7ff05c1a7490 2021-02-10T11:37:38.009 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:38.003 7ff053fff700 1 -- 172.21.2.102:0/831224807 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ff044006030 con 0x7ff05c1a7490 2021-02-10T11:37:38.009 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:38.003 7ff053fff700 1 --2- 172.21.2.102:0/831224807 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff0400186c0 0x7ff04001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:38.009 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:38.003 7ff053fff700 1 -- 172.21.2.102:0/831224807 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 5573+0+0 (secure 0 0 0) 0x7ff044032590 con 0x7ff05c1a7490 2021-02-10T11:37:38.009 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:38.003 7ff05b7fe700 1 --2- 172.21.2.102:0/831224807 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff0400186c0 0x7ff04001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:38.009 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:38.003 7ff05b7fe700 1 --2- 172.21.2.102:0/831224807 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff0400186c0 0x7ff04001abc0 crc :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:38.037 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:38.035 7ff062139700 1 -- 172.21.2.102:0/831224807 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff05c082a70 con 0x7ff05c1a7490 2021-02-10T11:37:38.039 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:38.037 7ff053fff700 1 -- 172.21.2.102:0/831224807 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ff04402cf10 con 0x7ff05c1a7490 2021-02-10T11:37:38.177 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:38.175 7ff062139700 1 -- 172.21.2.102:0/831224807 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"} v 0) v1 -- 0x7ff05c082a70 con 0x7ff05c1a7490 2021-02-10T11:37:38.360 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:38.358 7ff053fff700 1 -- 172.21.2.102:0/831224807 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"}]=0 v149) v1 ==== 118+0+0 (secure 0 0 0) 0x7ff04402cf10 con 0x7ff05c1a7490 2021-02-10T11:37:38.392 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:38.390 7ff062139700 1 -- 172.21.2.102:0/831224807 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff0400186c0 msgr2=0x7ff04001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:38.392 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:38.390 7ff062139700 1 --2- 172.21.2.102:0/831224807 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff0400186c0 0x7ff04001abc0 crc :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:38.393 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:38.390 7ff062139700 1 -- 172.21.2.102:0/831224807 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff05c1a7490 msgr2=0x7ff05c1a7890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:38.393 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:38.390 7ff062139700 1 --2- 172.21.2.102:0/831224807 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff05c1a7490 0x7ff05c1a7890 secure :-1 s=READY pgs=2982 cs=0 l=1 rev1=1 rx=0x7ff044000bd0 tx=0x7ff04400b960).stop 2021-02-10T11:37:38.393 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:38.390 7ff062139700 1 -- 172.21.2.102:0/831224807 shutdown_connections 2021-02-10T11:37:38.393 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:38.390 7ff062139700 1 --2- 172.21.2.102:0/831224807 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff0400186c0 0x7ff04001abc0 unknown :-1 s=CLOSED pgs=1309 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:38.393 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:38.390 7ff062139700 1 --2- 172.21.2.102:0/831224807 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff05c1a7eb0 0x7ff05c0ac630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:38.393 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:38.390 7ff062139700 1 --2- 172.21.2.102:0/831224807 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff05c1a7490 0x7ff05c1a7890 unknown :-1 s=CLOSED pgs=2982 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:38.393 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:38.390 7ff062139700 1 --2- 172.21.2.102:0/831224807 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff05c0a7950 0x7ff05c0b8df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:38.393 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:38.390 7ff062139700 1 -- 172.21.2.102:0/831224807 >> 172.21.2.102:0/831224807 conn(0x7ff05c0a02d0 msgr2=0x7ff05c09f200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:38.393 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:38.390 7ff062139700 1 -- 172.21.2.102:0/831224807 shutdown_connections 2021-02-10T11:37:38.394 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:38.391 7ff062139700 1 -- 172.21.2.102:0/831224807 wait complete. 2021-02-10T11:37:38.401 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:37:38.830 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:38.811 7feb53f8a700 1 Processor -- start 2021-02-10T11:37:38.830 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:38.811 7feb53f8a700 1 -- start start 2021-02-10T11:37:38.830 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:38.811 7feb53f8a700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7feb4c0d0340 0x7feb4c0ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:38.830 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:38.811 7feb53f8a700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7feb4c0d1c70 0x7feb4c0efc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:38.830 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:38.811 7feb53f8a700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb4c0f0b10 con 0x7feb4c0d1870 2021-02-10T11:37:38.830 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:38.811 7feb53f8a700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb4c0e5ed0 con 0x7feb4c0d0340 2021-02-10T11:37:38.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:38.811 7feb53f8a700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb4c1b0be0 con 0x7feb4c0d1c70 2021-02-10T11:37:38.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:38.811 7feb51da6700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7feb4c0d1870 0x7feb4c1b03f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:59296/0 (socket says 172.21.2.102:59296) 2021-02-10T11:37:38.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:38.811 7feb51da6700 1 -- 172.21.2.102:0/940268105 learned_addr learned my addr 172.21.2.102:0/940268105 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:37:38.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:38.811 7feb515a5700 1 --2- 172.21.2.102:0/940268105 >> v2:172.21.2.113:3300/0 conn(0x7feb4c0d0340 0x7feb4c0ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:38.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:38.812 7feb525a7700 1 --2- 172.21.2.102:0/940268105 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7feb4c0d1c70 0x7feb4c0efc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:38.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:38.812 7feb50da4700 1 -- 172.21.2.102:0/940268105 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 655073130 0 0) 0x7feb480010b0 con 0x7feb4c0d1870 2021-02-10T11:37:38.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:38.812 7feb50da4700 1 -- 172.21.2.102:0/940268105 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb38002fe0 con 0x7feb4c0d1870 2021-02-10T11:37:38.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:38.812 7feb515a5700 1 -- 172.21.2.102:0/940268105 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7feb4c0d1c70 msgr2=0x7feb4c0efc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:37:38.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:38.812 7feb515a5700 1 --2- 172.21.2.102:0/940268105 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7feb4c0d1c70 0x7feb4c0efc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:38.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:38.812 7feb515a5700 1 -- 172.21.2.102:0/940268105 >> v1:172.21.2.102:6789/0 conn(0x7feb4c0d1870 legacy=0x7feb4c1b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:38.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:38.812 7feb515a5700 1 -- 172.21.2.102:0/940268105 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb4c0f20a0 con 0x7feb4c0d0340 2021-02-10T11:37:38.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:38.812 7feb515a5700 1 --2- 172.21.2.102:0/940268105 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7feb4c0d0340 0x7feb4c0ef640 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 rx=0x7feb40006650 tx=0x7feb40009a80).ready entity=mon.1 client_cookie=7aee8004947556ca server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:38.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:38.813 7feb50da4700 1 -- 172.21.2.102:0/940268105 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7feb4000c0c0 con 0x7feb4c0d0340 2021-02-10T11:37:38.832 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:38.813 7feb50da4700 1 -- 172.21.2.102:0/940268105 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb4000c340 con 0x7feb4c0d0340 2021-02-10T11:37:38.832 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:38.813 7feb50da4700 1 -- 172.21.2.102:0/940268105 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7feb4000c860 con 0x7feb4c0d0340 2021-02-10T11:37:38.832 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:38.813 7feb53f8a700 1 -- 172.21.2.102:0/940268105 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7feb4c0d0340 msgr2=0x7feb4c0ef640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:38.832 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:38.813 7feb53f8a700 1 --2- 172.21.2.102:0/940268105 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7feb4c0d0340 0x7feb4c0ef640 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 rx=0x7feb40006650 tx=0x7feb40009a80).stop 2021-02-10T11:37:38.832 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:38.813 7feb53f8a700 1 -- 172.21.2.102:0/940268105 shutdown_connections 2021-02-10T11:37:38.832 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:38.813 7feb53f8a700 1 --2- 172.21.2.102:0/940268105 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7feb4c0d0340 0x7feb4c0ef640 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:38.832 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:38.813 7feb53f8a700 1 --2- 172.21.2.102:0/940268105 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7feb4c0d1c70 0x7feb4c0efc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:38.832 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:38.813 7feb53f8a700 1 -- 172.21.2.102:0/940268105 >> 172.21.2.102:0/940268105 conn(0x7feb4c0ce450 msgr2=0x7feb4c0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:38.832 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:38.814 7feb53f8a700 1 -- 172.21.2.102:0/940268105 shutdown_connections 2021-02-10T11:37:38.832 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:38.814 7feb53f8a700 1 -- 172.21.2.102:0/940268105 wait complete. 2021-02-10T11:37:38.833 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:38.828 7feb53f8a700 1 Processor -- start 2021-02-10T11:37:38.834 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:38.828 7feb53f8a700 1 -- start start 2021-02-10T11:37:38.834 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:38.829 7feb53f8a700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7feb4c0f0280 0x7feb4c0a2310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:38.834 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:38.829 7feb53f8a700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7feb4c0f0680 0x7feb4c0a2930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:38.834 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:38.829 7feb53f8a700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7feb4c0ec180 0x7feb4c0ec580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:38.834 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:38.829 7feb53f8a700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb4c0efd00 con 0x7feb4c0f0280 2021-02-10T11:37:38.834 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:38.829 7feb53f8a700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb4c0ecba0 con 0x7feb4c0ec180 2021-02-10T11:37:38.834 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:38.829 7feb53f8a700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb4c0ecd80 con 0x7feb4c0f0680 2021-02-10T11:37:38.834 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:38.829 7feb51da6700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7feb4c0f0280 0x7feb4c0a2310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:38.834 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:38.829 7feb51da6700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7feb4c0f0280 0x7feb4c0a2310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:38846/0 (socket says 172.21.2.102:38846) 2021-02-10T11:37:38.834 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:38.829 7feb51da6700 1 -- 172.21.2.102:0/858892711 learned_addr learned my addr 172.21.2.102:0/858892711 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:37:38.834 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:38.829 7feb51da6700 1 -- 172.21.2.102:0/858892711 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7feb4c0f0680 msgr2=0x7feb4c0a2930 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:37:38.835 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:38.829 7feb51da6700 1 --2- 172.21.2.102:0/858892711 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7feb4c0f0680 0x7feb4c0a2930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:38.835 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:38.829 7feb51da6700 1 -- 172.21.2.102:0/858892711 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7feb4c0ec180 msgr2=0x7feb4c0ec580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:37:38.835 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:38.829 7feb51da6700 1 --2- 172.21.2.102:0/858892711 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7feb4c0ec180 0x7feb4c0ec580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:38.835 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:38.829 7feb51da6700 1 -- 172.21.2.102:0/858892711 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb4c0ecfa0 con 0x7feb4c0f0280 2021-02-10T11:37:38.835 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:38.829 7feb51da6700 1 --2- 172.21.2.102:0/858892711 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7feb4c0f0280 0x7feb4c0a2310 secure :-1 s=READY pgs=2984 cs=0 l=1 rev1=1 rx=0x7feb48001c60 tx=0x7feb4800a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:38.835 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:38.829 7feb3e7fc700 1 -- 172.21.2.102:0/858892711 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7feb4800acc0 con 0x7feb4c0f0280 2021-02-10T11:37:38.835 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:38.829 7feb3e7fc700 1 -- 172.21.2.102:0/858892711 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb48004350 con 0x7feb4c0f0280 2021-02-10T11:37:38.835 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:38.829 7feb3e7fc700 1 -- 172.21.2.102:0/858892711 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7feb48004830 con 0x7feb4c0f0280 2021-02-10T11:37:38.835 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:38.829 7feb53f8a700 1 -- 172.21.2.102:0/858892711 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb4c0ecfa0 con 0x7feb4c0f0280 2021-02-10T11:37:38.835 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:38.830 7feb53f8a700 1 -- 172.21.2.102:0/858892711 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb4c0ecfa0 con 0x7feb4c0f0280 2021-02-10T11:37:38.835 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:38.830 7feb3e7fc700 1 -- 172.21.2.102:0/858892711 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7feb48004530 con 0x7feb4c0f0280 2021-02-10T11:37:38.835 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:38.830 7feb3e7fc700 1 --2- 172.21.2.102:0/858892711 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7feb340186c0 0x7feb3401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:38.835 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:38.830 7feb3e7fc700 1 -- 172.21.2.102:0/858892711 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 5573+0+0 (secure 0 0 0) 0x7feb480323e0 con 0x7feb4c0f0280 2021-02-10T11:37:38.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:38.830 7feb515a5700 1 --2- 172.21.2.102:0/858892711 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7feb340186c0 0x7feb3401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:38.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:38.831 7feb515a5700 1 --2- 172.21.2.102:0/858892711 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7feb340186c0 0x7feb3401abc0 crc :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:38.863 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:38.861 7feb53f8a700 1 -- 172.21.2.102:0/858892711 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb4c0ebc60 con 0x7feb4c0f0280 2021-02-10T11:37:38.866 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:38.864 7feb3e7fc700 1 -- 172.21.2.102:0/858892711 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7feb48004140 con 0x7feb4c0f0280 2021-02-10T11:37:38.992 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:38.990 7feb53f8a700 1 -- 172.21.2.102:0/858892711 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7feb4c0ebc60 con 0x7feb4c0f0280 2021-02-10T11:37:38.993 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:38.991 7feb3e7fc700 1 -- 172.21.2.102:0/858892711 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 149 v149) v1 ==== 95+0+2108 (secure 0 0 0) 0x7feb48004410 con 0x7feb4c0f0280 2021-02-10T11:37:38.993 INFO:teuthology.orchestra.run.gibba002.stderr:dumped fsmap epoch 149 2021-02-10T11:37:38.994 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:37:38.994 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":149,"default_fscid":18,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":12732,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.102:6835/3244979225","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6834","nonce":3244979225},{"type":"v1","addr":"172.21.2.102:6835","nonce":3244979225}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":145}],"filesystems":[{"mdsmap":{"epoch":149,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:37:36.348132","modified":"2021-02-10 11:37:38.355444","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8665},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8665":{"gid":8665,"name":"b","rank":0,"incarnation":146,"state":"up:active","state_seq":3,"addr":"172.21.2.113:6833/1669982519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6832","nonce":1669982519},{"type":"v1","addr":"172.21.2.113:6833","nonce":1669982519}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[37],"metadata_pool":36,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":18}]} 2021-02-10T11:37:38.995 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:38.993 7feb53f8a700 1 -- 172.21.2.102:0/858892711 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7feb340186c0 msgr2=0x7feb3401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:38.995 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:38.993 7feb53f8a700 1 --2- 172.21.2.102:0/858892711 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7feb340186c0 0x7feb3401abc0 crc :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:38.995 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:38.993 7feb53f8a700 1 -- 172.21.2.102:0/858892711 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7feb4c0f0280 msgr2=0x7feb4c0a2310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:38.995 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:38.993 7feb53f8a700 1 --2- 172.21.2.102:0/858892711 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7feb4c0f0280 0x7feb4c0a2310 secure :-1 s=READY pgs=2984 cs=0 l=1 rev1=1 rx=0x7feb48001c60 tx=0x7feb4800a8d0).stop 2021-02-10T11:37:38.995 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:38.993 7feb53f8a700 1 -- 172.21.2.102:0/858892711 shutdown_connections 2021-02-10T11:37:38.995 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:38.993 7feb53f8a700 1 --2- 172.21.2.102:0/858892711 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7feb340186c0 0x7feb3401abc0 unknown :-1 s=CLOSED pgs=1310 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:38.995 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:38.993 7feb53f8a700 1 --2- 172.21.2.102:0/858892711 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7feb4c0ec180 0x7feb4c0ec580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:38.995 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:38.993 7feb53f8a700 1 --2- 172.21.2.102:0/858892711 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7feb4c0f0680 0x7feb4c0a2930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:38.995 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:38.993 7feb53f8a700 1 --2- 172.21.2.102:0/858892711 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7feb4c0f0280 0x7feb4c0a2310 unknown :-1 s=CLOSED pgs=2984 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:38.996 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:38.993 7feb53f8a700 1 -- 172.21.2.102:0/858892711 >> 172.21.2.102:0/858892711 conn(0x7feb4c0c2130 msgr2=0x7feb4c09e3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:38.996 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:38.993 7feb53f8a700 1 -- 172.21.2.102:0/858892711 shutdown_connections 2021-02-10T11:37:38.996 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:38.993 7feb53f8a700 1 -- 172.21.2.102:0/858892711 wait complete. 2021-02-10T11:37:39.004 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:37:39.434 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:39.416 7f1fde452700 1 Processor -- start 2021-02-10T11:37:39.434 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:39.416 7f1fde452700 1 -- start start 2021-02-10T11:37:39.434 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:39.417 7f1fde452700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f1fd80d1870 0x7f1fd81b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:39.434 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:39.417 7f1fde452700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f1fd80f0650 0x7f1fd80cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:39.434 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:39.417 7f1fde452700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fd80f0b10 con 0x7f1fd81b0a10 2021-02-10T11:37:39.434 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:39.417 7f1fde452700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1fd80e5ed0 con 0x7f1fd80f0650 2021-02-10T11:37:39.435 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:39.417 7f1fde452700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1fd80d1c70 con 0x7f1fd80d1870 2021-02-10T11:37:39.435 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:39.417 7f1fdca6f700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f1fd80f0650 0x7f1fd80cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:39.435 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:39.417 7f1fdca6f700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f1fd80f0650 0x7f1fd80cfb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:38858/0 (socket says 172.21.2.102:38858) 2021-02-10T11:37:39.435 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:39.417 7f1fd77fe700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f1fd81b0a10 0x7f1fd80cf330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:59310/0 (socket says 172.21.2.102:59310) 2021-02-10T11:37:39.435 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:39.417 7f1fdca6f700 1 -- 172.21.2.102:0/3366105303 learned_addr learned my addr 172.21.2.102:0/3366105303 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:37:39.435 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:39.417 7f1fdca6f700 1 -- 172.21.2.102:0/3366105303 >> v2:172.21.2.113:3300/0 conn(0x7f1fd80d1870 msgr2=0x7f1fd81b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:37:39.435 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:39.417 7f1fdca6f700 1 --2- 172.21.2.102:0/3366105303 >> v2:172.21.2.113:3300/0 conn(0x7f1fd80d1870 0x7f1fd81b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:39.435 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:39.417 7f1fdca6f700 1 -- 172.21.2.102:0/3366105303 >> v1:172.21.2.102:6789/0 conn(0x7f1fd81b0a10 legacy=0x7f1fd80cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:39.435 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:39.417 7f1fdca6f700 1 -- 172.21.2.102:0/3366105303 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fd80d1e90 con 0x7f1fd80f0650 2021-02-10T11:37:39.435 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:39.417 7f1fdca6f700 1 --2- 172.21.2.102:0/3366105303 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1fd80f0650 0x7f1fd80cfb20 secure :-1 s=READY pgs=2986 cs=0 l=1 rev1=1 rx=0x7f1fd0006f00 tx=0x7f1fd0009f10).ready entity=mon.0 client_cookie=e984fdb38e47f20e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:39.435 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:39.417 7f1fd6ffd700 1 -- 172.21.2.102:0/3366105303 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1fd000b3b0 con 0x7f1fd80f0650 2021-02-10T11:37:39.435 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:39.417 7f1fd6ffd700 1 -- 172.21.2.102:0/3366105303 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fd000b630 con 0x7f1fd80f0650 2021-02-10T11:37:39.435 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:39.417 7f1fd6ffd700 1 -- 172.21.2.102:0/3366105303 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1fd000bb50 con 0x7f1fd80f0650 2021-02-10T11:37:39.436 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:39.417 7f1fde452700 1 -- 172.21.2.102:0/3366105303 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1fd80f0650 msgr2=0x7f1fd80cfb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:39.436 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:39.417 7f1fde452700 1 --2- 172.21.2.102:0/3366105303 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1fd80f0650 0x7f1fd80cfb20 secure :-1 s=READY pgs=2986 cs=0 l=1 rev1=1 rx=0x7f1fd0006f00 tx=0x7f1fd0009f10).stop 2021-02-10T11:37:39.436 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:39.418 7f1fde452700 1 -- 172.21.2.102:0/3366105303 shutdown_connections 2021-02-10T11:37:39.436 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:39.418 7f1fde452700 1 --2- 172.21.2.102:0/3366105303 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1fd80f0650 0x7f1fd80cfb20 unknown :-1 s=CLOSED pgs=2986 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:39.436 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:39.418 7f1fde452700 1 --2- 172.21.2.102:0/3366105303 >> v2:172.21.2.113:3300/0 conn(0x7f1fd80d1870 0x7f1fd81b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:39.436 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:39.418 7f1fde452700 1 -- 172.21.2.102:0/3366105303 >> 172.21.2.102:0/3366105303 conn(0x7f1fd80ce450 msgr2=0x7f1fd80d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:39.436 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:39.418 7f1fde452700 1 -- 172.21.2.102:0/3366105303 shutdown_connections 2021-02-10T11:37:39.436 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:39.418 7f1fde452700 1 -- 172.21.2.102:0/3366105303 wait complete. 2021-02-10T11:37:39.438 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:39.432 7f1fde452700 1 Processor -- start 2021-02-10T11:37:39.438 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:39.432 7f1fde452700 1 -- start start 2021-02-10T11:37:39.438 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:39.433 7f1fde452700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1fd80b0ad0 0x7f1fd8091f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:39.438 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:39.433 7f1fde452700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1fd81a8850 0x7f1fd81a8c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:39.438 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:39.433 7f1fde452700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1fd81a6fa0 0x7f1fd81a73a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:39.438 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:39.433 7f1fde452700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fd81abfe0 con 0x7f1fd81a8850 2021-02-10T11:37:39.438 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:39.433 7f1fde452700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fd81a9c40 con 0x7f1fd80b0ad0 2021-02-10T11:37:39.438 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:39.433 7f1fde452700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fd81a9730 con 0x7f1fd81a6fa0 2021-02-10T11:37:39.438 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:39.433 7f1fd77fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1fd81a8850 0x7f1fd81a8c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:39.439 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:39.433 7f1fd77fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1fd81a8850 0x7f1fd81a8c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:38864/0 (socket says 172.21.2.102:38864) 2021-02-10T11:37:39.439 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:39.433 7f1fd77fe700 1 -- 172.21.2.102:0/3755753446 learned_addr learned my addr 172.21.2.102:0/3755753446 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:37:39.439 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:39.434 7f1fd77fe700 1 -- 172.21.2.102:0/3755753446 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1fd81a6fa0 msgr2=0x7f1fd81a73a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:39.439 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:39.434 7f1fd77fe700 1 --2- 172.21.2.102:0/3755753446 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1fd81a6fa0 0x7f1fd81a73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:39.439 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:39.434 7f1fd77fe700 1 -- 172.21.2.102:0/3755753446 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1fd80b0ad0 msgr2=0x7f1fd8091f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:39.439 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:39.434 7f1fd77fe700 1 --2- 172.21.2.102:0/3755753446 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1fd80b0ad0 0x7f1fd8091f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:39.439 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:39.434 7f1fd77fe700 1 -- 172.21.2.102:0/3755753446 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fd81a9270 con 0x7f1fd81a8850 2021-02-10T11:37:39.439 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:39.434 7f1fd77fe700 1 --2- 172.21.2.102:0/3755753446 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1fd81a8850 0x7f1fd81a8c50 secure :-1 s=READY pgs=2987 cs=0 l=1 rev1=1 rx=0x7f1fcc001c60 tx=0x7f1fcc00a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:39.439 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:39.434 7f1fd4ff9700 1 -- 172.21.2.102:0/3755753446 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1fcc00acc0 con 0x7f1fd81a8850 2021-02-10T11:37:39.439 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:39.434 7f1fd4ff9700 1 -- 172.21.2.102:0/3755753446 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fcc004140 con 0x7f1fd81a8850 2021-02-10T11:37:39.439 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:39.434 7f1fd4ff9700 1 -- 172.21.2.102:0/3755753446 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1fcc004620 con 0x7f1fd81a8850 2021-02-10T11:37:39.439 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:39.434 7f1fde452700 1 -- 172.21.2.102:0/3755753446 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fd81a9270 con 0x7f1fd81a8850 2021-02-10T11:37:39.439 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:39.434 7f1fde452700 1 -- 172.21.2.102:0/3755753446 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fd81a9270 con 0x7f1fd81a8850 2021-02-10T11:37:39.439 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:39.435 7f1fd4ff9700 1 -- 172.21.2.102:0/3755753446 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1fcc000d90 con 0x7f1fd81a8850 2021-02-10T11:37:39.440 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:39.435 7f1fd4ff9700 1 --2- 172.21.2.102:0/3755753446 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1fb80186c0 0x7f1fb801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:39.440 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:39.435 7f1fd4ff9700 1 -- 172.21.2.102:0/3755753446 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f1fcc0327c0 con 0x7f1fd81a8850 2021-02-10T11:37:39.440 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:39.435 7f1fd7fff700 1 --2- 172.21.2.102:0/3755753446 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1fb80186c0 0x7f1fb801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:39.440 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:39.435 7f1fd7fff700 1 --2- 172.21.2.102:0/3755753446 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1fb80186c0 0x7f1fb801abc0 crc :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:39.468 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:39.466 7f1fde452700 1 -- 172.21.2.102:0/3755753446 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fd81abfe0 con 0x7f1fd81a8850 2021-02-10T11:37:39.471 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:39.469 7f1fd4ff9700 1 -- 172.21.2.102:0/3755753446 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f1fcc0009b0 con 0x7f1fd81a8850 2021-02-10T11:37:39.595 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:39.593 7f1fde452700 1 -- 172.21.2.102:0/3755753446 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1fd81abfe0 con 0x7f1fd81a8850 2021-02-10T11:37:39.595 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:39.594 7f1fd4ff9700 1 -- 172.21.2.102:0/3755753446 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v172) v1 ==== 74+0+14007 (secure 0 0 0) 0x7f1fcc000d90 con 0x7f1fd81a8850 2021-02-10T11:37:39.596 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:37:39.596 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":172,"fsid":"68db30ea-dbce-4aef-a6df-f01c6fea0513","created":"2021-02-10 11:03:06.331557","modified":"2021-02-10 11:37:37.349944","last_up_change":"2021-02-10 11:03:32.690562","last_in_change":"2021-02-10 11:03:32.690562","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":37,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:03:37.322519","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":36,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:37:34.298334","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"171","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":37,"pool_name":"cephfs_data","create_time":"2021-02-10 11:37:35.337127","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"be691b4d-a039-408a-af80-9d53f02b6f1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6818","nonce":36088},{"type":"v1","addr":"172.21.2.102:6820","nonce":36088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6822","nonce":36088},{"type":"v1","addr":"172.21.2.102:6824","nonce":36088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6830","nonce":36088},{"type":"v1","addr":"172.21.2.102:6832","nonce":36088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6826","nonce":36088},{"type":"v1","addr":"172.21.2.102:6828","nonce":36088}]},"public_addr":"172.21.2.102:6820/36088","cluster_addr":"172.21.2.102:6824/36088","heartbeat_back_addr":"172.21.2.102:6832/36088","heartbeat_front_addr":"172.21.2.102:6828/36088","state":["exists","up"]},{"osd":1,"uuid":"64e3ccc8-9c69-4d7c-865e-af8e83822635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6819","nonce":36087},{"type":"v1","addr":"172.21.2.102:6821","nonce":36087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6823","nonce":36087},{"type":"v1","addr":"172.21.2.102:6825","nonce":36087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6831","nonce":36087},{"type":"v1","addr":"172.21.2.102:6833","nonce":36087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6827","nonce":36087},{"type":"v1","addr":"172.21.2.102:6829","nonce":36087}]},"public_addr":"172.21.2.102:6821/36087","cluster_addr":"172.21.2.102:6825/36087","heartbeat_back_addr":"172.21.2.102:6833/36087","heartbeat_front_addr":"172.21.2.102:6829/36087","state":["exists","up"]},{"osd":2,"uuid":"153700af-747f-481f-bdf4-d6072d3a842f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6810","nonce":36089},{"type":"v1","addr":"172.21.2.102:6811","nonce":36089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6812","nonce":36089},{"type":"v1","addr":"172.21.2.102:6813","nonce":36089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6816","nonce":36089},{"type":"v1","addr":"172.21.2.102:6817","nonce":36089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6814","nonce":36089},{"type":"v1","addr":"172.21.2.102:6815","nonce":36089}]},"public_addr":"172.21.2.102:6811/36089","cluster_addr":"172.21.2.102:6813/36089","heartbeat_back_addr":"172.21.2.102:6817/36089","heartbeat_front_addr":"172.21.2.102:6815/36089","state":["exists","up"]},{"osd":3,"uuid":"cbc678f4-83a9-45e6-8830-2f7c232b1ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6802","nonce":36090},{"type":"v1","addr":"172.21.2.102:6803","nonce":36090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6804","nonce":36090},{"type":"v1","addr":"172.21.2.102:6805","nonce":36090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6808","nonce":36090},{"type":"v1","addr":"172.21.2.102:6809","nonce":36090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6806","nonce":36090},{"type":"v1","addr":"172.21.2.102:6807","nonce":36090}]},"public_addr":"172.21.2.102:6803/36090","cluster_addr":"172.21.2.102:6805/36090","heartbeat_back_addr":"172.21.2.102:6809/36090","heartbeat_front_addr":"172.21.2.102:6807/36090","state":["exists","up"]},{"osd":4,"uuid":"c01e4008-b258-4f10-afd2-9e7f37214351","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6808","nonce":36368},{"type":"v1","addr":"172.21.2.113:6811","nonce":36368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6814","nonce":36368},{"type":"v1","addr":"172.21.2.113:6818","nonce":36368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6826","nonce":36368},{"type":"v1","addr":"172.21.2.113:6828","nonce":36368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6820","nonce":36368},{"type":"v1","addr":"172.21.2.113:6824","nonce":36368}]},"public_addr":"172.21.2.113:6811/36368","cluster_addr":"172.21.2.113:6818/36368","heartbeat_back_addr":"172.21.2.113:6828/36368","heartbeat_front_addr":"172.21.2.113:6824/36368","state":["exists","up"]},{"osd":5,"uuid":"551f6007-daf8-4a98-8ee0-c2b16bcccb5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6802","nonce":36366},{"type":"v1","addr":"172.21.2.113:6805","nonce":36366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6807","nonce":36366},{"type":"v1","addr":"172.21.2.113:6810","nonce":36366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6819","nonce":36366},{"type":"v1","addr":"172.21.2.113:6822","nonce":36366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6812","nonce":36366},{"type":"v1","addr":"172.21.2.113:6815","nonce":36366}]},"public_addr":"172.21.2.113:6805/36366","cluster_addr":"172.21.2.113:6810/36366","heartbeat_back_addr":"172.21.2.113:6822/36366","heartbeat_front_addr":"172.21.2.113:6815/36366","state":["exists","up"]},{"osd":6,"uuid":"bf67c040-0e50-48be-b102-c3943bc9ebef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6800","nonce":36367},{"type":"v1","addr":"172.21.2.113:6801","nonce":36367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6803","nonce":36367},{"type":"v1","addr":"172.21.2.113:6804","nonce":36367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6813","nonce":36367},{"type":"v1","addr":"172.21.2.113:6816","nonce":36367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6806","nonce":36367},{"type":"v1","addr":"172.21.2.113:6809","nonce":36367}]},"public_addr":"172.21.2.113:6801/36367","cluster_addr":"172.21.2.113:6804/36367","heartbeat_back_addr":"172.21.2.113:6816/36367","heartbeat_front_addr":"172.21.2.113:6809/36367","state":["exists","up"]},{"osd":7,"uuid":"f5461f80-018d-428f-be89-2eb2bcbe7bb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":169,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6817","nonce":36365},{"type":"v1","addr":"172.21.2.113:6821","nonce":36365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6823","nonce":36365},{"type":"v1","addr":"172.21.2.113:6825","nonce":36365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6830","nonce":36365},{"type":"v1","addr":"172.21.2.113:6831","nonce":36365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6827","nonce":36365},{"type":"v1","addr":"172.21.2.113:6829","nonce":36365}]},"public_addr":"172.21.2.113:6821/36365","cluster_addr":"172.21.2.113:6825/36365","heartbeat_back_addr":"172.21.2.113:6831/36365","heartbeat_front_addr":"172.21.2.113:6829/36365","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:37:39.597 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:39.595 7f1fde452700 1 -- 172.21.2.102:0/3755753446 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1fb80186c0 msgr2=0x7f1fb801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:39.597 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:39.595 7f1fde452700 1 --2- 172.21.2.102:0/3755753446 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1fb80186c0 0x7f1fb801abc0 crc :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:39.597 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:39.595 7f1fde452700 1 -- 172.21.2.102:0/3755753446 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1fd81a8850 msgr2=0x7f1fd81a8c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:39.597 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:39.595 7f1fde452700 1 --2- 172.21.2.102:0/3755753446 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1fd81a8850 0x7f1fd81a8c50 secure :-1 s=READY pgs=2987 cs=0 l=1 rev1=1 rx=0x7f1fcc001c60 tx=0x7f1fcc00a8d0).stop 2021-02-10T11:37:39.597 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:39.595 7f1fde452700 1 -- 172.21.2.102:0/3755753446 shutdown_connections 2021-02-10T11:37:39.597 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:39.595 7f1fde452700 1 --2- 172.21.2.102:0/3755753446 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1fd81a6fa0 0x7f1fd81a73a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:39.598 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:39.595 7f1fde452700 1 --2- 172.21.2.102:0/3755753446 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1fd81a8850 0x7f1fd81a8c50 unknown :-1 s=CLOSED pgs=2987 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:39.598 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:39.595 7f1fde452700 1 --2- 172.21.2.102:0/3755753446 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1fb80186c0 0x7f1fb801abc0 unknown :-1 s=CLOSED pgs=1311 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:39.598 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:39.595 7f1fde452700 1 --2- 172.21.2.102:0/3755753446 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1fd80b0ad0 0x7f1fd8091f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:39.598 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:39.595 7f1fde452700 1 -- 172.21.2.102:0/3755753446 >> 172.21.2.102:0/3755753446 conn(0x7f1fd819f8f0 msgr2=0x7f1fd81a5f80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:39.598 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:39.596 7f1fde452700 1 -- 172.21.2.102:0/3755753446 shutdown_connections 2021-02-10T11:37:39.598 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:39.596 7f1fde452700 1 -- 172.21.2.102:0/3755753446 wait complete. 2021-02-10T11:37:39.606 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:37:40.047 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:40.029 7ffa1d700700 1 Processor -- start 2021-02-10T11:37:40.047 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:40.029 7ffa1d700700 1 -- start start 2021-02-10T11:37:40.048 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:40.030 7ffa1d700700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ffa180d1860 0x7ffa181b04b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:40.048 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:40.030 7ffa1d700700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7ffa180cff60 0x7ffa18109120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:40.048 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:40.030 7ffa1d700700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa18109a50 con 0x7ffa181b0ad0 2021-02-10T11:37:40.048 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:40.030 7ffa1d700700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa18109740 con 0x7ffa180cff60 2021-02-10T11:37:40.048 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:40.030 7ffa1d700700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa18088070 con 0x7ffa180d1860 2021-02-10T11:37:40.048 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:40.030 7ffa17ac0700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7ffa180cff60 0x7ffa18109120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:40.048 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:40.030 7ffa17ac0700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7ffa180cff60 0x7ffa18109120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:38868/0 (socket says 172.21.2.102:38868) 2021-02-10T11:37:40.048 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:40.030 7ffa17ac0700 1 -- 172.21.2.102:0/3020214796 learned_addr learned my addr 172.21.2.102:0/3020214796 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:37:40.048 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:40.030 7ffa17ac0700 1 -- 172.21.2.102:0/3020214796 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ffa180d1860 msgr2=0x7ffa181b04b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:37:40.048 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:40.030 7ffa17ac0700 1 --2- 172.21.2.102:0/3020214796 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ffa180d1860 0x7ffa181b04b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:40.048 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:40.030 7ffa17ac0700 1 -- 172.21.2.102:0/3020214796 >> v1:172.21.2.113:6789/0 conn(0x7ffa181b0ad0 legacy=0x7ffa18108930 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:37:40.048 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:40.030 7ffa17ac0700 1 -- 172.21.2.102:0/3020214796 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa18088260 con 0x7ffa180cff60 2021-02-10T11:37:40.049 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:40.030 7ffa17ac0700 1 --2- 172.21.2.102:0/3020214796 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ffa180cff60 0x7ffa18109120 secure :-1 s=READY pgs=2988 cs=0 l=1 rev1=1 rx=0x7ffa10006f00 tx=0x7ffa10009f10).ready entity=mon.0 client_cookie=483dca65006f80e5 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:40.049 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:40.031 7ffa162bd700 1 -- 172.21.2.102:0/3020214796 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ffa1000b3b0 con 0x7ffa180cff60 2021-02-10T11:37:40.049 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:40.031 7ffa162bd700 1 -- 172.21.2.102:0/3020214796 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa1000b730 con 0x7ffa180cff60 2021-02-10T11:37:40.049 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:40.031 7ffa162bd700 1 -- 172.21.2.102:0/3020214796 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ffa1000bc50 con 0x7ffa180cff60 2021-02-10T11:37:40.049 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:40.031 7ffa1d700700 1 -- 172.21.2.102:0/3020214796 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ffa180cff60 msgr2=0x7ffa18109120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:40.049 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:40.031 7ffa1d700700 1 --2- 172.21.2.102:0/3020214796 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ffa180cff60 0x7ffa18109120 secure :-1 s=READY pgs=2988 cs=0 l=1 rev1=1 rx=0x7ffa10006f00 tx=0x7ffa10009f10).stop 2021-02-10T11:37:40.049 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:40.031 7ffa1d700700 1 -- 172.21.2.102:0/3020214796 shutdown_connections 2021-02-10T11:37:40.049 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:40.031 7ffa1d700700 1 --2- 172.21.2.102:0/3020214796 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ffa180cff60 0x7ffa18109120 unknown :-1 s=CLOSED pgs=2988 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:40.049 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:40.031 7ffa1d700700 1 --2- 172.21.2.102:0/3020214796 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ffa180d1860 0x7ffa181b04b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:40.049 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:40.031 7ffa1d700700 1 -- 172.21.2.102:0/3020214796 >> 172.21.2.102:0/3020214796 conn(0x7ffa180ce410 msgr2=0x7ffa180d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:40.049 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:40.031 7ffa1d700700 1 -- 172.21.2.102:0/3020214796 shutdown_connections 2021-02-10T11:37:40.049 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:40.031 7ffa1d700700 1 -- 172.21.2.102:0/3020214796 wait complete. 2021-02-10T11:37:40.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:40.046 7ffa1d700700 1 Processor -- start 2021-02-10T11:37:40.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:40.046 7ffa1d700700 1 -- start start 2021-02-10T11:37:40.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:40.046 7ffa1d700700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ffa18099860 0x7ffa18194f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:40.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:40.046 7ffa1d700700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ffa18099c60 0x7ffa18195540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:40.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:40.046 7ffa1d700700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ffa1818a540 0x7ffa1818a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:40.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:40.046 7ffa1d700700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa18189fc0 con 0x7ffa18099c60 2021-02-10T11:37:40.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:40.046 7ffa1d700700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa18090820 con 0x7ffa1818a540 2021-02-10T11:37:40.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:40.046 7ffa1d700700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa18090a00 con 0x7ffa18099860 2021-02-10T11:37:40.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:40.046 7ffa16abe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ffa18099c60 0x7ffa18195540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:40.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:40.046 7ffa16abe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ffa18099c60 0x7ffa18195540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:38874/0 (socket says 172.21.2.102:38874) 2021-02-10T11:37:40.052 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:40.046 7ffa16abe700 1 -- 172.21.2.102:0/3737801156 learned_addr learned my addr 172.21.2.102:0/3737801156 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:37:40.052 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:40.047 7ffa16abe700 1 -- 172.21.2.102:0/3737801156 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ffa18099860 msgr2=0x7ffa18194f20 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:37:40.052 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:40.047 7ffa16abe700 1 --2- 172.21.2.102:0/3737801156 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ffa18099860 0x7ffa18194f20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:40.052 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:40.047 7ffa16abe700 1 -- 172.21.2.102:0/3737801156 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ffa1818a540 msgr2=0x7ffa1818a940 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:37:40.052 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:40.047 7ffa16abe700 1 --2- 172.21.2.102:0/3737801156 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ffa1818a540 0x7ffa1818a940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:40.052 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:40.047 7ffa16abe700 1 -- 172.21.2.102:0/3737801156 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa1818af60 con 0x7ffa18099c60 2021-02-10T11:37:40.052 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:40.047 7ffa16abe700 1 --2- 172.21.2.102:0/3737801156 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ffa18099c60 0x7ffa18195540 secure :-1 s=READY pgs=2989 cs=0 l=1 rev1=1 rx=0x7ffa0800b5b0 tx=0x7ffa0800b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:40.052 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:40.047 7ffa0ffff700 1 -- 172.21.2.102:0/3737801156 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ffa0800bd40 con 0x7ffa18099c60 2021-02-10T11:37:40.052 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:40.047 7ffa0ffff700 1 -- 172.21.2.102:0/3737801156 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa08002080 con 0x7ffa18099c60 2021-02-10T11:37:40.052 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:40.047 7ffa1d700700 1 -- 172.21.2.102:0/3737801156 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa1818af60 con 0x7ffa18099c60 2021-02-10T11:37:40.052 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:40.047 7ffa1d700700 1 -- 172.21.2.102:0/3737801156 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa1818b4f0 con 0x7ffa18099c60 2021-02-10T11:37:40.052 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:40.047 7ffa0ffff700 1 -- 172.21.2.102:0/3737801156 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ffa080025a0 con 0x7ffa18099c60 2021-02-10T11:37:40.052 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:40.048 7ffa0ffff700 1 -- 172.21.2.102:0/3737801156 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ffa08002260 con 0x7ffa18099c60 2021-02-10T11:37:40.053 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:40.048 7ffa0ffff700 1 --2- 172.21.2.102:0/3737801156 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff9fc0186c0 0x7ff9fc01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:40.053 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:40.048 7ffa0ffff700 1 -- 172.21.2.102:0/3737801156 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 5573+0+0 (secure 0 0 0) 0x7ffa08032580 con 0x7ffa18099c60 2021-02-10T11:37:40.053 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:40.048 7ffa172bf700 1 --2- 172.21.2.102:0/3737801156 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff9fc0186c0 0x7ff9fc01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:40.053 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:40.048 7ffa172bf700 1 --2- 172.21.2.102:0/3737801156 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff9fc0186c0 0x7ff9fc01abc0 crc :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:40.082 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:40.080 7ffa1d700700 1 -- 172.21.2.102:0/3737801156 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9f0002fd0 con 0x7ffa18099c60 2021-02-10T11:37:40.085 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:40.083 7ffa0ffff700 1 -- 172.21.2.102:0/3737801156 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ffa08025fb0 con 0x7ffa18099c60 2021-02-10T11:37:40.214 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:40.212 7ffa1d700700 1 -- 172.21.2.102:0/3737801156 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7ff9f0002ef0 con 0x7ffa18099c60 2021-02-10T11:37:40.218 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:40.216 7ffa0ffff700 1 -- 172.21.2.102:0/3737801156 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.0 v93) v1 ==== 184+0+0 (secure 0 0 0) 0x7ffa08025fb0 con 0x7ffa18099c60 2021-02-10T11:37:40.221 INFO:teuthology.orchestra.run.gibba002.stderr:updated caps for client.0 2021-02-10T11:37:40.223 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:40.220 7ffa1d700700 1 -- 172.21.2.102:0/3737801156 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff9fc0186c0 msgr2=0x7ff9fc01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:40.223 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:40.220 7ffa1d700700 1 --2- 172.21.2.102:0/3737801156 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff9fc0186c0 0x7ff9fc01abc0 crc :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:40.223 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:40.220 7ffa1d700700 1 -- 172.21.2.102:0/3737801156 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ffa18099c60 msgr2=0x7ffa18195540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:40.223 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:40.220 7ffa1d700700 1 --2- 172.21.2.102:0/3737801156 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ffa18099c60 0x7ffa18195540 secure :-1 s=READY pgs=2989 cs=0 l=1 rev1=1 rx=0x7ffa0800b5b0 tx=0x7ffa0800b950).stop 2021-02-10T11:37:40.223 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:40.221 7ffa1d700700 1 -- 172.21.2.102:0/3737801156 shutdown_connections 2021-02-10T11:37:40.223 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:40.221 7ffa1d700700 1 --2- 172.21.2.102:0/3737801156 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff9fc0186c0 0x7ff9fc01abc0 unknown :-1 s=CLOSED pgs=1312 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:40.223 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:40.221 7ffa1d700700 1 --2- 172.21.2.102:0/3737801156 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ffa1818a540 0x7ffa1818a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:40.223 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:40.221 7ffa1d700700 1 --2- 172.21.2.102:0/3737801156 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ffa18099c60 0x7ffa18195540 unknown :-1 s=CLOSED pgs=2989 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:40.223 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:40.221 7ffa1d700700 1 --2- 172.21.2.102:0/3737801156 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ffa18099860 0x7ffa18194f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:40.223 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:40.221 7ffa1d700700 1 -- 172.21.2.102:0/3737801156 >> 172.21.2.102:0/3737801156 conn(0x7ffa1819f820 msgr2=0x7ffa181a5cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:40.224 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:40.221 7ffa1d700700 1 -- 172.21.2.102:0/3737801156 shutdown_connections 2021-02-10T11:37:40.224 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:40.221 7ffa1d700700 1 -- 172.21.2.102:0/3737801156 wait complete. 2021-02-10T11:37:40.231 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.1 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:37:40.674 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:40.655 7f43e6326700 1 Processor -- start 2021-02-10T11:37:40.674 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:40.655 7f43e6326700 1 -- start start 2021-02-10T11:37:40.674 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:40.655 7f43e6326700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f43e00d1880 0x7f43e01b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:40.675 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:40.655 7f43e6326700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f43e00cff80 0x7f43e0109140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:40.675 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:40.655 7f43e6326700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43e0109a70 con 0x7f43e01b0af0 2021-02-10T11:37:40.675 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:40.655 7f43e6326700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43e0109760 con 0x7f43e00cff80 2021-02-10T11:37:40.675 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:40.655 7f43e6326700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43e0088070 con 0x7f43e00d1880 2021-02-10T11:37:40.675 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:40.656 7f43e4943700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f43e00cff80 0x7f43e0109140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:40.675 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:40.656 7f43e4943700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f43e00cff80 0x7f43e0109140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.113:3300/0 says I am v2:172.21.2.102:42310/0 (socket says 172.21.2.102:42310) 2021-02-10T11:37:40.675 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:40.656 7f43df7fe700 1 --1- >> v1:172.21.2.113:6789/0 conn(0x7f43e01b0af0 0x7f43e0108950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.113:6789/0 says I am v1:172.21.2.102:38428/0 (socket says 172.21.2.102:38428) 2021-02-10T11:37:40.675 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:40.656 7f43e4943700 1 -- 172.21.2.102:0/1387833892 learned_addr learned my addr 172.21.2.102:0/1387833892 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:37:40.675 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:40.656 7f43dffff700 1 --2- 172.21.2.102:0/1387833892 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f43e00d1880 0x7f43e01b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:40.675 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:40.656 7f43e4943700 1 -- 172.21.2.102:0/1387833892 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f43e00d1880 msgr2=0x7f43e01b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:37:40.675 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:40.656 7f43e4943700 1 --2- 172.21.2.102:0/1387833892 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f43e00d1880 0x7f43e01b04d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:40.675 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:40.656 7f43e4943700 1 -- 172.21.2.102:0/1387833892 >> v1:172.21.2.113:6789/0 conn(0x7f43e01b0af0 legacy=0x7f43e0108950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:40.676 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:40.656 7f43e4943700 1 -- 172.21.2.102:0/1387833892 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43e0088260 con 0x7f43e00cff80 2021-02-10T11:37:40.676 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:40.657 7f43e4943700 1 --2- 172.21.2.102:0/1387833892 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f43e00cff80 0x7f43e0109140 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 rx=0x7f43d8006aa0 tx=0x7f43d8009ea0).ready entity=mon.1 client_cookie=7c36b55de6a14ed4 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:40.676 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:40.657 7f43deffd700 1 -- 172.21.2.102:0/1387833892 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f43d800b510 con 0x7f43e00cff80 2021-02-10T11:37:40.676 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:40.657 7f43deffd700 1 -- 172.21.2.102:0/1387833892 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43d8003030 con 0x7f43e00cff80 2021-02-10T11:37:40.676 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:40.657 7f43deffd700 1 -- 172.21.2.102:0/1387833892 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f43d8000f90 con 0x7f43e00cff80 2021-02-10T11:37:40.676 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:40.657 7f43e6326700 1 -- 172.21.2.102:0/1387833892 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f43e00cff80 msgr2=0x7f43e0109140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:40.676 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:40.657 7f43e6326700 1 --2- 172.21.2.102:0/1387833892 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f43e00cff80 0x7f43e0109140 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 rx=0x7f43d8006aa0 tx=0x7f43d8009ea0).stop 2021-02-10T11:37:40.676 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:40.658 7f43e6326700 1 -- 172.21.2.102:0/1387833892 shutdown_connections 2021-02-10T11:37:40.676 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:40.658 7f43e6326700 1 --2- 172.21.2.102:0/1387833892 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f43e00cff80 0x7f43e0109140 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:40.676 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:40.658 7f43e6326700 1 --2- 172.21.2.102:0/1387833892 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f43e00d1880 0x7f43e01b04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:40.676 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:40.658 7f43e6326700 1 -- 172.21.2.102:0/1387833892 >> 172.21.2.102:0/1387833892 conn(0x7f43e00ce430 msgr2=0x7f43e00d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:40.676 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:40.658 7f43e6326700 1 -- 172.21.2.102:0/1387833892 shutdown_connections 2021-02-10T11:37:40.677 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:40.658 7f43e6326700 1 -- 172.21.2.102:0/1387833892 wait complete. 2021-02-10T11:37:40.678 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:40.673 7f43e6326700 1 Processor -- start 2021-02-10T11:37:40.678 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:40.673 7f43e6326700 1 -- start start 2021-02-10T11:37:40.678 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:40.673 7f43e6326700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f43e01a49b0 0x7f43e008aa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:40.678 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:40.673 7f43e6326700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f43e008b0a0 0x7f43e008b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:40.678 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:40.673 7f43e6326700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f43e008bac0 0x7f43e00bf1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:40.678 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:40.673 7f43e6326700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43e00b2550 con 0x7f43e008bac0 2021-02-10T11:37:40.678 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:40.673 7f43e6326700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43e00bf7d0 con 0x7f43e008b0a0 2021-02-10T11:37:40.678 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:40.673 7f43e6326700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43e00bf9f0 con 0x7f43e01a49b0 2021-02-10T11:37:40.678 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:40.673 7f43e4943700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f43e008bac0 0x7f43e00bf1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:40.679 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:40.673 7f43e4943700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f43e008bac0 0x7f43e00bf1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:38884/0 (socket says 172.21.2.102:38884) 2021-02-10T11:37:40.679 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:40.673 7f43e4943700 1 -- 172.21.2.102:0/1254592198 learned_addr learned my addr 172.21.2.102:0/1254592198 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:37:40.679 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:40.674 7f43e4943700 1 -- 172.21.2.102:0/1254592198 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f43e01a49b0 msgr2=0x7f43e008aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:40.679 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:40.674 7f43e4943700 1 --2- 172.21.2.102:0/1254592198 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f43e01a49b0 0x7f43e008aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:40.679 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:40.674 7f43e4943700 1 -- 172.21.2.102:0/1254592198 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f43e008b0a0 msgr2=0x7f43e008b4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:40.679 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:40.674 7f43e4943700 1 --2- 172.21.2.102:0/1254592198 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f43e008b0a0 0x7f43e008b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:40.679 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:40.674 7f43e4943700 1 -- 172.21.2.102:0/1254592198 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43e00bfc10 con 0x7f43e008bac0 2021-02-10T11:37:40.679 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:40.674 7f43e4943700 1 --2- 172.21.2.102:0/1254592198 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f43e008bac0 0x7f43e00bf1b0 secure :-1 s=READY pgs=2990 cs=0 l=1 rev1=1 rx=0x7f43d800b9a0 tx=0x7f43d800bd40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:40.679 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:40.674 7f43dcff9700 1 -- 172.21.2.102:0/1254592198 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f43d8009450 con 0x7f43e008bac0 2021-02-10T11:37:40.679 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:40.674 7f43dcff9700 1 -- 172.21.2.102:0/1254592198 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43d8006030 con 0x7f43e008bac0 2021-02-10T11:37:40.679 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:40.674 7f43dcff9700 1 -- 172.21.2.102:0/1254592198 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f43d8006550 con 0x7f43e008bac0 2021-02-10T11:37:40.680 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:40.674 7f43e6326700 1 -- 172.21.2.102:0/1254592198 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43e00bfc10 con 0x7f43e008bac0 2021-02-10T11:37:40.680 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:40.674 7f43e6326700 1 -- 172.21.2.102:0/1254592198 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43e00bfc10 con 0x7f43e008bac0 2021-02-10T11:37:40.680 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:40.675 7f43dcff9700 1 -- 172.21.2.102:0/1254592198 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f43d8006210 con 0x7f43e008bac0 2021-02-10T11:37:40.680 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:40.675 7f43dcff9700 1 --2- 172.21.2.102:0/1254592198 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f43c00186c0 0x7f43c001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:40.680 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:40.675 7f43dcff9700 1 -- 172.21.2.102:0/1254592198 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f43d80328f0 con 0x7f43e008bac0 2021-02-10T11:37:40.680 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:40.675 7f43dffff700 1 --2- 172.21.2.102:0/1254592198 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f43c00186c0 0x7f43c001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:40.680 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:40.675 7f43dffff700 1 --2- 172.21.2.102:0/1254592198 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f43c00186c0 0x7f43c001abc0 crc :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:40.707 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:40.705 7f43e6326700 1 -- 172.21.2.102:0/1254592198 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43e008a540 con 0x7f43e008bac0 2021-02-10T11:37:40.709 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:40.707 7f43dcff9700 1 -- 172.21.2.102:0/1254592198 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f43d802ff90 con 0x7f43e008bac0 2021-02-10T11:37:40.840 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:40.838 7f43e6326700 1 -- 172.21.2.102:0/1254592198 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f43e008a540 con 0x7f43e008bac0 2021-02-10T11:37:40.844 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:40.842 7f43dcff9700 1 -- 172.21.2.102:0/1254592198 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.1 v94) v1 ==== 184+0+0 (secure 0 0 0) 0x7f43d8006210 con 0x7f43e008bac0 2021-02-10T11:37:40.847 INFO:teuthology.orchestra.run.gibba002.stderr:updated caps for client.1 2021-02-10T11:37:40.849 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:40.847 7f43e6326700 1 -- 172.21.2.102:0/1254592198 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f43c00186c0 msgr2=0x7f43c001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:40.849 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:40.847 7f43e6326700 1 --2- 172.21.2.102:0/1254592198 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f43c00186c0 0x7f43c001abc0 crc :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:40.849 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:40.847 7f43e6326700 1 -- 172.21.2.102:0/1254592198 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f43e008bac0 msgr2=0x7f43e00bf1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:40.849 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:40.847 7f43e6326700 1 --2- 172.21.2.102:0/1254592198 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f43e008bac0 0x7f43e00bf1b0 secure :-1 s=READY pgs=2990 cs=0 l=1 rev1=1 rx=0x7f43d800b9a0 tx=0x7f43d800bd40).stop 2021-02-10T11:37:40.849 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:40.847 7f43e6326700 1 -- 172.21.2.102:0/1254592198 shutdown_connections 2021-02-10T11:37:40.850 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:40.847 7f43e6326700 1 --2- 172.21.2.102:0/1254592198 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f43e008bac0 0x7f43e00bf1b0 unknown :-1 s=CLOSED pgs=2990 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:40.850 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:40.847 7f43e6326700 1 --2- 172.21.2.102:0/1254592198 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f43c00186c0 0x7f43c001abc0 unknown :-1 s=CLOSED pgs=1313 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:40.850 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:40.847 7f43e6326700 1 --2- 172.21.2.102:0/1254592198 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f43e008b0a0 0x7f43e008b4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:40.850 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:40.847 7f43e6326700 1 --2- 172.21.2.102:0/1254592198 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f43e01a49b0 0x7f43e008aa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:40.850 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:40.847 7f43e6326700 1 -- 172.21.2.102:0/1254592198 >> 172.21.2.102:0/1254592198 conn(0x7f43e00bd7a0 msgr2=0x7f43e00b5900 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:40.850 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:40.847 7f43e6326700 1 -- 172.21.2.102:0/1254592198 shutdown_connections 2021-02-10T11:37:40.850 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:40.847 7f43e6326700 1 -- 172.21.2.102:0/1254592198 wait complete. 2021-02-10T11:37:40.858 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.2 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:37:41.292 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:41.273 7fb235b84700 1 Processor -- start 2021-02-10T11:37:41.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:41.273 7fb235b84700 1 -- start start 2021-02-10T11:37:41.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:41.274 7fb235b84700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb2300d1860 0x7fb2301b04b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:41.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:41.274 7fb235b84700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fb2301b0ad0 0x7fb230108930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:41.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:41.274 7fb235b84700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb230109a50 con 0x7fb2300d1c60 2021-02-10T11:37:41.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:41.274 7fb235b84700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2300e5ee0 con 0x7fb2301b0ad0 2021-02-10T11:37:41.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:41.274 7fb235b84700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb230109740 con 0x7fb2300d1860 2021-02-10T11:37:41.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:41.274 7fb22effd700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fb2301b0ad0 0x7fb230108930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:41.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:41.274 7fb22effd700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fb2301b0ad0 0x7fb230108930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:38892/0 (socket says 172.21.2.102:38892) 2021-02-10T11:37:41.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:41.274 7fb22effd700 1 -- 172.21.2.102:0/3240508491 learned_addr learned my addr 172.21.2.102:0/3240508491 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:37:41.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:41.274 7fb22f7fe700 1 --2- 172.21.2.102:0/3240508491 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb2300d1860 0x7fb2301b04b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:41.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:41.274 7fb22effd700 1 -- 172.21.2.102:0/3240508491 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb2300d1860 msgr2=0x7fb2301b04b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:37:41.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:41.274 7fb22effd700 1 --2- 172.21.2.102:0/3240508491 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb2300d1860 0x7fb2301b04b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:41.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:41.274 7fb22effd700 1 -- 172.21.2.102:0/3240508491 >> v1:172.21.2.102:6789/0 conn(0x7fb2300d1c60 legacy=0x7fb230108f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:41.294 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:41.274 7fb22effd700 1 -- 172.21.2.102:0/3240508491 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb230088070 con 0x7fb2301b0ad0 2021-02-10T11:37:41.294 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:41.275 7fb22effd700 1 --2- 172.21.2.102:0/3240508491 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb2301b0ad0 0x7fb230108930 secure :-1 s=READY pgs=2992 cs=0 l=1 rev1=1 rx=0x7fb218006a80 tx=0x7fb218009f00).ready entity=mon.0 client_cookie=9d74d0d1caa23fc2 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:41.294 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:41.275 7fb22e7fc700 1 -- 172.21.2.102:0/3240508491 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb21800b5c0 con 0x7fb2301b0ad0 2021-02-10T11:37:41.294 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:41.275 7fb22e7fc700 1 -- 172.21.2.102:0/3240508491 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb21800b8c0 con 0x7fb2301b0ad0 2021-02-10T11:37:41.294 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:41.275 7fb22e7fc700 1 -- 172.21.2.102:0/3240508491 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb21800bde0 con 0x7fb2301b0ad0 2021-02-10T11:37:41.294 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:41.275 7fb235b84700 1 -- 172.21.2.102:0/3240508491 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb2301b0ad0 msgr2=0x7fb230108930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:41.294 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:41.275 7fb235b84700 1 --2- 172.21.2.102:0/3240508491 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb2301b0ad0 0x7fb230108930 secure :-1 s=READY pgs=2992 cs=0 l=1 rev1=1 rx=0x7fb218006a80 tx=0x7fb218009f00).stop 2021-02-10T11:37:41.294 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:41.276 7fb235b84700 1 -- 172.21.2.102:0/3240508491 shutdown_connections 2021-02-10T11:37:41.294 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:41.276 7fb235b84700 1 --2- 172.21.2.102:0/3240508491 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb2301b0ad0 0x7fb230108930 unknown :-1 s=CLOSED pgs=2992 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:41.294 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:41.276 7fb235b84700 1 --2- 172.21.2.102:0/3240508491 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb2300d1860 0x7fb2301b04b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:41.294 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:41.276 7fb235b84700 1 -- 172.21.2.102:0/3240508491 >> 172.21.2.102:0/3240508491 conn(0x7fb2300ce410 msgr2=0x7fb2300d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:41.294 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:41.276 7fb235b84700 1 -- 172.21.2.102:0/3240508491 shutdown_connections 2021-02-10T11:37:41.295 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:41.276 7fb235b84700 1 -- 172.21.2.102:0/3240508491 wait complete. 2021-02-10T11:37:41.295 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:41.290 7fb235b84700 1 Processor -- start 2021-02-10T11:37:41.295 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:41.290 7fb235b84700 1 -- start start 2021-02-10T11:37:41.296 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:41.291 7fb235b84700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb230083190 0x7fb230083670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:41.296 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:41.291 7fb235b84700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb2300acd60 0x7fb2301a2fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:41.296 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:41.291 7fb235b84700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb2300ad160 0x7fb2301a35c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:41.296 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:41.291 7fb235b84700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2300ad750 con 0x7fb2300ad160 2021-02-10T11:37:41.296 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:41.291 7fb235b84700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb23009f620 con 0x7fb230083190 2021-02-10T11:37:41.296 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:41.291 7fb235b84700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2301a3be0 con 0x7fb2300acd60 2021-02-10T11:37:41.296 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:41.291 7fb22ffff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb2300ad160 0x7fb2301a35c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:41.296 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:41.291 7fb22ffff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb2300ad160 0x7fb2301a35c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:38898/0 (socket says 172.21.2.102:38898) 2021-02-10T11:37:41.296 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:41.291 7fb22ffff700 1 -- 172.21.2.102:0/1946877079 learned_addr learned my addr 172.21.2.102:0/1946877079 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:37:41.296 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:41.291 7fb22ffff700 1 -- 172.21.2.102:0/1946877079 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb2300acd60 msgr2=0x7fb2301a2fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:41.297 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:41.291 7fb22ffff700 1 --2- 172.21.2.102:0/1946877079 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb2300acd60 0x7fb2301a2fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:41.297 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:41.291 7fb22ffff700 1 -- 172.21.2.102:0/1946877079 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb230083190 msgr2=0x7fb230083670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:41.297 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:41.291 7fb22ffff700 1 --2- 172.21.2.102:0/1946877079 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb230083190 0x7fb230083670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:41.297 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:41.291 7fb22ffff700 1 -- 172.21.2.102:0/1946877079 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb23008e890 con 0x7fb2300ad160 2021-02-10T11:37:41.297 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:41.291 7fb22ffff700 1 --2- 172.21.2.102:0/1946877079 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb2300ad160 0x7fb2301a35c0 secure :-1 s=READY pgs=2993 cs=0 l=1 rev1=1 rx=0x7fb22800b530 tx=0x7fb22800b8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:41.297 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:41.292 7fb227fff700 1 -- 172.21.2.102:0/1946877079 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb228006030 con 0x7fb2300ad160 2021-02-10T11:37:41.298 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:41.292 7fb227fff700 1 -- 172.21.2.102:0/1946877079 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2280062f0 con 0x7fb2300ad160 2021-02-10T11:37:41.298 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:41.292 7fb227fff700 1 -- 172.21.2.102:0/1946877079 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb228005770 con 0x7fb2300ad160 2021-02-10T11:37:41.298 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:41.292 7fb235b84700 1 -- 172.21.2.102:0/1946877079 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb23008e890 con 0x7fb2300ad160 2021-02-10T11:37:41.298 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:41.292 7fb235b84700 1 -- 172.21.2.102:0/1946877079 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb23008ee20 con 0x7fb2300ad160 2021-02-10T11:37:41.298 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:41.292 7fb227fff700 1 -- 172.21.2.102:0/1946877079 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fb228005700 con 0x7fb2300ad160 2021-02-10T11:37:41.298 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:41.292 7fb227fff700 1 --2- 172.21.2.102:0/1946877079 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb2140186c0 0x7fb21401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:41.299 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:41.292 7fb227fff700 1 -- 172.21.2.102:0/1946877079 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fb2280323e0 con 0x7fb2300ad160 2021-02-10T11:37:41.299 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:41.292 7fb22f7fe700 1 --2- 172.21.2.102:0/1946877079 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb2140186c0 0x7fb21401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:41.299 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:41.293 7fb22f7fe700 1 --2- 172.21.2.102:0/1946877079 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb2140186c0 0x7fb21401abc0 crc :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:41.325 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:41.323 7fb235b84700 1 -- 172.21.2.102:0/1946877079 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb230082980 con 0x7fb2300ad160 2021-02-10T11:37:41.328 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:41.326 7fb227fff700 1 -- 172.21.2.102:0/1946877079 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fb228026f50 con 0x7fb2300ad160 2021-02-10T11:37:41.461 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:41.458 7fb235b84700 1 -- 172.21.2.102:0/1946877079 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7fb230109a50 con 0x7fb2300ad160 2021-02-10T11:37:41.464 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:41.462 7fb227fff700 1 -- 172.21.2.102:0/1946877079 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.2 v95) v1 ==== 184+0+0 (secure 0 0 0) 0x7fb228005700 con 0x7fb2300ad160 2021-02-10T11:37:41.468 INFO:teuthology.orchestra.run.gibba002.stderr:updated caps for client.2 2021-02-10T11:37:41.470 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:41.467 7fb235b84700 1 -- 172.21.2.102:0/1946877079 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb2140186c0 msgr2=0x7fb21401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:41.470 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:41.467 7fb235b84700 1 --2- 172.21.2.102:0/1946877079 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb2140186c0 0x7fb21401abc0 crc :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:41.470 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:41.467 7fb235b84700 1 -- 172.21.2.102:0/1946877079 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb2300ad160 msgr2=0x7fb2301a35c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:41.470 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:41.467 7fb235b84700 1 --2- 172.21.2.102:0/1946877079 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb2300ad160 0x7fb2301a35c0 secure :-1 s=READY pgs=2993 cs=0 l=1 rev1=1 rx=0x7fb22800b530 tx=0x7fb22800b8d0).stop 2021-02-10T11:37:41.470 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:41.468 7fb235b84700 1 -- 172.21.2.102:0/1946877079 shutdown_connections 2021-02-10T11:37:41.470 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:41.468 7fb235b84700 1 --2- 172.21.2.102:0/1946877079 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb2300ad160 0x7fb2301a35c0 unknown :-1 s=CLOSED pgs=2993 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:41.470 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:41.468 7fb235b84700 1 --2- 172.21.2.102:0/1946877079 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb2300acd60 0x7fb2301a2fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:41.470 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:41.468 7fb235b84700 1 --2- 172.21.2.102:0/1946877079 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb2140186c0 0x7fb21401abc0 unknown :-1 s=CLOSED pgs=1314 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:41.470 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:41.468 7fb235b84700 1 --2- 172.21.2.102:0/1946877079 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb230083190 0x7fb230083670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:41.470 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:41.468 7fb235b84700 1 -- 172.21.2.102:0/1946877079 >> 172.21.2.102:0/1946877079 conn(0x7fb23009fb10 msgr2=0x7fb230092630 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:41.471 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:41.468 7fb235b84700 1 -- 172.21.2.102:0/1946877079 shutdown_connections 2021-02-10T11:37:41.471 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:41.468 7fb235b84700 1 -- 172.21.2.102:0/1946877079 wait complete. 2021-02-10T11:37:41.479 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.3 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:37:41.919 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:41.901 7fb350d40700 1 Processor -- start 2021-02-10T11:37:41.920 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:41.901 7fb350d40700 1 -- start start 2021-02-10T11:37:41.920 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:41.902 7fb350d40700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fb34c0d1880 0x7fb34c1b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:41.920 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:41.902 7fb350d40700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb34c1b0af0 0x7fb34c108950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:41.920 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:41.902 7fb350d40700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fb34c0d1c80 0x7fb34c108f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:41.920 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:41.902 7fb350d40700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb34c0fe1d0 con 0x7fb34c0d1c80 2021-02-10T11:37:41.920 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:41.902 7fb350d40700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb34c109a70 con 0x7fb34c0d1880 2021-02-10T11:37:41.920 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:41.902 7fb350d40700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb34c088070 con 0x7fb34c1b0af0 2021-02-10T11:37:41.920 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:41.902 7fb34b314700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fb34c0d1c80 0x7fb34c108f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:41.920 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:41.902 7fb34b314700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fb34c0d1c80 0x7fb34c108f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:38910/0 (socket says 172.21.2.102:38910) 2021-02-10T11:37:41.920 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:41.902 7fb34b314700 1 -- 172.21.2.102:0/3804937950 learned_addr learned my addr 172.21.2.102:0/3804937950 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:37:41.920 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:41.902 7fb34b314700 1 -- 172.21.2.102:0/3804937950 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb34c1b0af0 msgr2=0x7fb34c108950 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:37:41.921 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:41.902 7fb34b314700 1 --2- 172.21.2.102:0/3804937950 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb34c1b0af0 0x7fb34c108950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:41.921 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:41.902 7fb34b314700 1 -- 172.21.2.102:0/3804937950 >> v2:172.21.2.113:3300/0 conn(0x7fb34c0d1880 msgr2=0x7fb34c1b04d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:37:41.921 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:41.902 7fb34b314700 1 --2- 172.21.2.102:0/3804937950 >> v2:172.21.2.113:3300/0 conn(0x7fb34c0d1880 0x7fb34c1b04d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:41.921 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:41.902 7fb34b314700 1 -- 172.21.2.102:0/3804937950 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb34c109590 con 0x7fb34c0d1c80 2021-02-10T11:37:41.921 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:41.902 7fb34b314700 1 --2- 172.21.2.102:0/3804937950 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb34c0d1c80 0x7fb34c108f70 secure :-1 s=READY pgs=2994 cs=0 l=1 rev1=1 rx=0x7fb344006f00 tx=0x7fb344009f10).ready entity=mon.0 client_cookie=170e477778b32858 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:41.921 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:41.903 7fb349b11700 1 -- 172.21.2.102:0/3804937950 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb34400b3b0 con 0x7fb34c0d1c80 2021-02-10T11:37:41.921 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:41.903 7fb349b11700 1 -- 172.21.2.102:0/3804937950 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb34400b3b0 con 0x7fb34c0d1c80 2021-02-10T11:37:41.921 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:41.903 7fb349b11700 1 -- 172.21.2.102:0/3804937950 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb34400b670 con 0x7fb34c0d1c80 2021-02-10T11:37:41.921 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:41.903 7fb350d40700 1 -- 172.21.2.102:0/3804937950 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb34c0d1c80 msgr2=0x7fb34c108f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:41.921 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:41.903 7fb350d40700 1 --2- 172.21.2.102:0/3804937950 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb34c0d1c80 0x7fb34c108f70 secure :-1 s=READY pgs=2994 cs=0 l=1 rev1=1 rx=0x7fb344006f00 tx=0x7fb344009f10).stop 2021-02-10T11:37:41.921 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:41.903 7fb350d40700 1 -- 172.21.2.102:0/3804937950 shutdown_connections 2021-02-10T11:37:41.921 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:41.903 7fb350d40700 1 --2- 172.21.2.102:0/3804937950 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb34c1b0af0 0x7fb34c108950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:41.921 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:41.903 7fb350d40700 1 --2- 172.21.2.102:0/3804937950 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb34c0d1c80 0x7fb34c108f70 unknown :-1 s=CLOSED pgs=2994 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:41.922 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:41.903 7fb350d40700 1 --2- 172.21.2.102:0/3804937950 >> v2:172.21.2.113:3300/0 conn(0x7fb34c0d1880 0x7fb34c1b04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:41.922 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:41.903 7fb350d40700 1 -- 172.21.2.102:0/3804937950 >> 172.21.2.102:0/3804937950 conn(0x7fb34c0ce430 msgr2=0x7fb34c0d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:41.922 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:41.903 7fb350d40700 1 -- 172.21.2.102:0/3804937950 shutdown_connections 2021-02-10T11:37:41.922 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:41.903 7fb350d40700 1 -- 172.21.2.102:0/3804937950 wait complete. 2021-02-10T11:37:41.923 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:41.918 7fb350d40700 1 Processor -- start 2021-02-10T11:37:41.923 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:41.918 7fb350d40700 1 -- start start 2021-02-10T11:37:41.923 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:41.918 7fb350d40700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb34c0aedc0 0x7fb34c095430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:41.924 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:41.918 7fb350d40700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb34c09ac10 0x7fb34c095a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:41.924 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:41.918 7fb350d40700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb34c1af300 0x7fb34c099030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:41.924 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:41.918 7fb350d40700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb34c1aed80 con 0x7fb34c0aedc0 2021-02-10T11:37:41.924 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:41.918 7fb350d40700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb34c086a60 con 0x7fb34c1af300 2021-02-10T11:37:41.924 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:41.918 7fb350d40700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb34c096070 con 0x7fb34c09ac10 2021-02-10T11:37:41.924 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:41.918 7fb34ab13700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb34c0aedc0 0x7fb34c095430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:41.924 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:41.918 7fb34ab13700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb34c0aedc0 0x7fb34c095430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:38914/0 (socket says 172.21.2.102:38914) 2021-02-10T11:37:41.924 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:41.918 7fb34ab13700 1 -- 172.21.2.102:0/2856584221 learned_addr learned my addr 172.21.2.102:0/2856584221 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:37:41.924 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:41.919 7fb34ab13700 1 -- 172.21.2.102:0/2856584221 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb34c09ac10 msgr2=0x7fb34c095a50 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:37:41.924 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:41.919 7fb34ab13700 1 --2- 172.21.2.102:0/2856584221 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb34c09ac10 0x7fb34c095a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:41.924 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:41.919 7fb34ab13700 1 -- 172.21.2.102:0/2856584221 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb34c1af300 msgr2=0x7fb34c099030 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:37:41.924 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:41.919 7fb34ab13700 1 --2- 172.21.2.102:0/2856584221 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb34c1af300 0x7fb34c099030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:41.925 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:41.919 7fb34ab13700 1 -- 172.21.2.102:0/2856584221 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb34c099650 con 0x7fb34c0aedc0 2021-02-10T11:37:41.925 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:41.919 7fb34ab13700 1 --2- 172.21.2.102:0/2856584221 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb34c0aedc0 0x7fb34c095430 secure :-1 s=READY pgs=2995 cs=0 l=1 rev1=1 rx=0x7fb33c00b5c0 tx=0x7fb33c00b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:41.925 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:41.919 7fb3437fe700 1 -- 172.21.2.102:0/2856584221 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb33c00bd50 con 0x7fb34c0aedc0 2021-02-10T11:37:41.925 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:41.919 7fb3437fe700 1 -- 172.21.2.102:0/2856584221 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb33c006030 con 0x7fb34c0aedc0 2021-02-10T11:37:41.925 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:41.919 7fb350d40700 1 -- 172.21.2.102:0/2856584221 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb34c099650 con 0x7fb34c0aedc0 2021-02-10T11:37:41.925 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:41.919 7fb350d40700 1 -- 172.21.2.102:0/2856584221 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb34c099be0 con 0x7fb34c0aedc0 2021-02-10T11:37:41.925 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:41.919 7fb3437fe700 1 -- 172.21.2.102:0/2856584221 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb33c0054d0 con 0x7fb34c0aedc0 2021-02-10T11:37:41.925 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:41.920 7fb3437fe700 1 -- 172.21.2.102:0/2856584221 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fb33c006210 con 0x7fb34c0aedc0 2021-02-10T11:37:41.925 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:41.920 7fb3437fe700 1 --2- 172.21.2.102:0/2856584221 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb330018730 0x7fb33001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:41.925 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:41.920 7fb3437fe700 1 -- 172.21.2.102:0/2856584221 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fb33c0327d0 con 0x7fb34c0aedc0 2021-02-10T11:37:41.925 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:41.920 7fb34a312700 1 --2- 172.21.2.102:0/2856584221 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb330018730 0x7fb33001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:41.925 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:41.920 7fb34a312700 1 --2- 172.21.2.102:0/2856584221 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb330018730 0x7fb33001ac10 crc :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:41.953 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:41.951 7fb350d40700 1 -- 172.21.2.102:0/2856584221 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb34c0fe1d0 con 0x7fb34c0aedc0 2021-02-10T11:37:41.956 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:41.954 7fb3437fe700 1 -- 172.21.2.102:0/2856584221 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fb33c000960 con 0x7fb34c0aedc0 2021-02-10T11:37:42.086 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:42.084 7fb350d40700 1 -- 172.21.2.102:0/2856584221 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.3", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7fb34c0fe1d0 con 0x7fb34c0aedc0 2021-02-10T11:37:42.090 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:42.088 7fb3437fe700 1 -- 172.21.2.102:0/2856584221 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.3", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.3 v96) v1 ==== 184+0+0 (secure 0 0 0) 0x7fb33c006030 con 0x7fb34c0aedc0 2021-02-10T11:37:42.093 INFO:teuthology.orchestra.run.gibba002.stderr:updated caps for client.3 2021-02-10T11:37:42.095 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:42.093 7fb350d40700 1 -- 172.21.2.102:0/2856584221 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb330018730 msgr2=0x7fb33001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:42.095 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:42.093 7fb350d40700 1 --2- 172.21.2.102:0/2856584221 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb330018730 0x7fb33001ac10 crc :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:42.095 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:42.093 7fb350d40700 1 -- 172.21.2.102:0/2856584221 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb34c0aedc0 msgr2=0x7fb34c095430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:42.095 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:42.093 7fb350d40700 1 --2- 172.21.2.102:0/2856584221 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb34c0aedc0 0x7fb34c095430 secure :-1 s=READY pgs=2995 cs=0 l=1 rev1=1 rx=0x7fb33c00b5c0 tx=0x7fb33c00b960).stop 2021-02-10T11:37:42.096 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:42.093 7fb350d40700 1 -- 172.21.2.102:0/2856584221 shutdown_connections 2021-02-10T11:37:42.096 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:42.093 7fb350d40700 1 --2- 172.21.2.102:0/2856584221 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb330018730 0x7fb33001ac10 unknown :-1 s=CLOSED pgs=1315 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:42.096 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:42.093 7fb350d40700 1 --2- 172.21.2.102:0/2856584221 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb34c1af300 0x7fb34c099030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:42.096 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:42.093 7fb350d40700 1 --2- 172.21.2.102:0/2856584221 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb34c09ac10 0x7fb34c095a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:42.096 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:42.093 7fb350d40700 1 --2- 172.21.2.102:0/2856584221 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb34c0aedc0 0x7fb34c095430 unknown :-1 s=CLOSED pgs=2995 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:42.096 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:42.093 7fb350d40700 1 -- 172.21.2.102:0/2856584221 >> 172.21.2.102:0/2856584221 conn(0x7fb34c0845b0 msgr2=0x7fb34c1ac340 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:42.096 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:42.093 7fb350d40700 1 -- 172.21.2.102:0/2856584221 shutdown_connections 2021-02-10T11:37:42.096 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:42.093 7fb350d40700 1 -- 172.21.2.102:0/2856584221 wait complete. 2021-02-10T11:37:42.103 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:37:42.532 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:42.514 7fc82a06a700 1 Processor -- start 2021-02-10T11:37:42.533 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:42.514 7fc82a06a700 1 -- start start 2021-02-10T11:37:42.533 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:42.514 7fc82a06a700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fc8240d1870 0x7fc8241b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:42.533 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:42.514 7fc82a06a700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fc8241b0a10 0x7fc8240cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:42.533 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:42.514 7fc82a06a700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc8240d1c70 0x7fc8240cf950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:42.533 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:42.514 7fc82a06a700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8240e5ed0 con 0x7fc8240d1870 2021-02-10T11:37:42.533 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:42.514 7fc82a06a700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8240f0b10 con 0x7fc8241b0a10 2021-02-10T11:37:42.533 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:42.514 7fc82a06a700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8240cff70 con 0x7fc8240d1c70 2021-02-10T11:37:42.533 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:42.515 7fc8237fe700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fc8240d1870 0x7fc8241b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:42.533 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:42.515 7fc8237fe700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fc8240d1870 0x7fc8241b03f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:38920/0 (socket says 172.21.2.102:38920) 2021-02-10T11:37:42.533 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:42.515 7fc8237fe700 1 -- 172.21.2.102:0/3067071358 learned_addr learned my addr 172.21.2.102:0/3067071358 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:37:42.533 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:42.515 7fc8237fe700 1 -- 172.21.2.102:0/3067071358 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc8240d1c70 msgr2=0x7fc8240cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:37:42.534 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:42.515 7fc8237fe700 1 --2- 172.21.2.102:0/3067071358 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc8240d1c70 0x7fc8240cf950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:42.534 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:42.515 7fc8237fe700 1 -- 172.21.2.102:0/3067071358 >> v2:172.21.2.113:3300/0 conn(0x7fc8241b0a10 msgr2=0x7fc8240cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:37:42.534 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:42.515 7fc8237fe700 1 --2- 172.21.2.102:0/3067071358 >> v2:172.21.2.113:3300/0 conn(0x7fc8241b0a10 0x7fc8240cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:42.534 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:42.515 7fc8237fe700 1 -- 172.21.2.102:0/3067071358 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8240d0150 con 0x7fc8240d1870 2021-02-10T11:37:42.534 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:42.515 7fc8237fe700 1 --2- 172.21.2.102:0/3067071358 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc8240d1870 0x7fc8241b03f0 secure :-1 s=READY pgs=2996 cs=0 l=1 rev1=1 rx=0x7fc80c006700 tx=0x7fc80c009b80).ready entity=mon.0 client_cookie=6545d39c8179f0dd server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:42.534 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:42.515 7fc8227fc700 1 -- 172.21.2.102:0/3067071358 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc80c00c0c0 con 0x7fc8240d1870 2021-02-10T11:37:42.534 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:42.515 7fc8227fc700 1 -- 172.21.2.102:0/3067071358 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc80c00c340 con 0x7fc8240d1870 2021-02-10T11:37:42.534 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:42.515 7fc8227fc700 1 -- 172.21.2.102:0/3067071358 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc80c00c860 con 0x7fc8240d1870 2021-02-10T11:37:42.534 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:42.515 7fc82a06a700 1 -- 172.21.2.102:0/3067071358 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc8240d1870 msgr2=0x7fc8241b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:42.534 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:42.515 7fc82a06a700 1 --2- 172.21.2.102:0/3067071358 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc8240d1870 0x7fc8241b03f0 secure :-1 s=READY pgs=2996 cs=0 l=1 rev1=1 rx=0x7fc80c006700 tx=0x7fc80c009b80).stop 2021-02-10T11:37:42.534 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:42.516 7fc82a06a700 1 -- 172.21.2.102:0/3067071358 shutdown_connections 2021-02-10T11:37:42.534 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:42.516 7fc82a06a700 1 --2- 172.21.2.102:0/3067071358 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc8240d1c70 0x7fc8240cf950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:42.535 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:42.516 7fc82a06a700 1 --2- 172.21.2.102:0/3067071358 >> v2:172.21.2.113:3300/0 conn(0x7fc8241b0a10 0x7fc8240cf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:42.535 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:42.516 7fc82a06a700 1 --2- 172.21.2.102:0/3067071358 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc8240d1870 0x7fc8241b03f0 unknown :-1 s=CLOSED pgs=2996 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:42.535 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:42.516 7fc82a06a700 1 -- 172.21.2.102:0/3067071358 >> 172.21.2.102:0/3067071358 conn(0x7fc8240ce450 msgr2=0x7fc8240d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:42.535 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:42.516 7fc82a06a700 1 -- 172.21.2.102:0/3067071358 shutdown_connections 2021-02-10T11:37:42.535 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:42.516 7fc82a06a700 1 -- 172.21.2.102:0/3067071358 wait complete. 2021-02-10T11:37:42.537 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:42.531 7fc82a06a700 1 Processor -- start 2021-02-10T11:37:42.537 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:42.531 7fc82a06a700 1 -- start start 2021-02-10T11:37:42.537 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:42.531 7fc82a06a700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc8241b8ad0 0x7fc824187e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:42.537 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:42.531 7fc82a06a700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc8241b8ed0 0x7fc8240b1af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:42.537 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:42.531 7fc82a06a700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc8240b2110 0x7fc8241b7a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:42.537 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:42.531 7fc82a06a700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc824188470 con 0x7fc8241b8ed0 2021-02-10T11:37:42.537 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:42.531 7fc82a06a700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc82418be10 con 0x7fc8240b2110 2021-02-10T11:37:42.537 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:42.531 7fc82a06a700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc82418c030 con 0x7fc8241b8ad0 2021-02-10T11:37:42.537 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:42.532 7fc822ffd700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc8241b8ed0 0x7fc8240b1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:42.537 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:42.532 7fc822ffd700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc8241b8ed0 0x7fc8240b1af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:38930/0 (socket says 172.21.2.102:38930) 2021-02-10T11:37:42.537 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:42.532 7fc822ffd700 1 -- 172.21.2.102:0/1265907765 learned_addr learned my addr 172.21.2.102:0/1265907765 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:37:42.537 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:42.532 7fc822ffd700 1 -- 172.21.2.102:0/1265907765 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc8241b8ad0 msgr2=0x7fc824187e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:42.538 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:42.532 7fc822ffd700 1 --2- 172.21.2.102:0/1265907765 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc8241b8ad0 0x7fc824187e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:42.538 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:42.532 7fc822ffd700 1 -- 172.21.2.102:0/1265907765 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc8240b2110 msgr2=0x7fc8241b7a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:42.538 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:42.532 7fc823fff700 1 --2- 172.21.2.102:0/1265907765 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc8240b2110 0x7fc8241b7a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:42.538 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:42.532 7fc822ffd700 1 --2- 172.21.2.102:0/1265907765 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc8240b2110 0x7fc8241b7a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:42.538 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:42.532 7fc822ffd700 1 -- 172.21.2.102:0/1265907765 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8240b2510 con 0x7fc8241b8ed0 2021-02-10T11:37:42.538 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:42.532 7fc823fff700 1 --2- 172.21.2.102:0/1265907765 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc8240b2110 0x7fc8241b7a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:37:42.538 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:42.532 7fc822ffd700 1 --2- 172.21.2.102:0/1265907765 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc8241b8ed0 0x7fc8240b1af0 secure :-1 s=READY pgs=2997 cs=0 l=1 rev1=1 rx=0x7fc81400b530 tx=0x7fc81400b8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:42.538 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:42.533 7fc81bfff700 1 -- 172.21.2.102:0/1265907765 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc8140009b0 con 0x7fc8241b8ed0 2021-02-10T11:37:42.538 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:42.533 7fc81bfff700 1 -- 172.21.2.102:0/1265907765 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc81400bcc0 con 0x7fc8241b8ed0 2021-02-10T11:37:42.538 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:42.533 7fc81bfff700 1 -- 172.21.2.102:0/1265907765 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc814000c90 con 0x7fc8241b8ed0 2021-02-10T11:37:42.538 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:42.533 7fc82a06a700 1 -- 172.21.2.102:0/1265907765 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8240b2510 con 0x7fc8241b8ed0 2021-02-10T11:37:42.538 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:42.533 7fc82a06a700 1 -- 172.21.2.102:0/1265907765 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8241b8020 con 0x7fc8241b8ed0 2021-02-10T11:37:42.539 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:42.533 7fc81bfff700 1 -- 172.21.2.102:0/1265907765 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fc814000c90 con 0x7fc8241b8ed0 2021-02-10T11:37:42.539 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:42.533 7fc81bfff700 1 --2- 172.21.2.102:0/1265907765 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc8080186c0 0x7fc80801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:42.539 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:42.533 7fc81bfff700 1 -- 172.21.2.102:0/1265907765 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fc814032570 con 0x7fc8241b8ed0 2021-02-10T11:37:42.539 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:42.534 7fc8237fe700 1 --2- 172.21.2.102:0/1265907765 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc8080186c0 0x7fc80801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:42.539 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:42.534 7fc8237fe700 1 --2- 172.21.2.102:0/1265907765 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc8080186c0 0x7fc80801abc0 crc :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:42.566 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:42.564 7fc82a06a700 1 -- 172.21.2.102:0/1265907765 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8241b88b0 con 0x7fc8241b8ed0 2021-02-10T11:37:42.569 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:42.567 7fc81bfff700 1 -- 172.21.2.102:0/1265907765 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fc814000960 con 0x7fc8241b8ed0 2021-02-10T11:37:42.698 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:42.696 7fc82a06a700 1 -- 172.21.2.102:0/1265907765 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc8241b88b0 con 0x7fc8241b8ed0 2021-02-10T11:37:42.699 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:42.697 7fc81bfff700 1 -- 172.21.2.102:0/1265907765 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 149 v149) v1 ==== 95+0+2108 (secure 0 0 0) 0x7fc814000c90 con 0x7fc8241b8ed0 2021-02-10T11:37:42.699 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:37:42.699 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":149,"default_fscid":18,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":12732,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.102:6835/3244979225","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6834","nonce":3244979225},{"type":"v1","addr":"172.21.2.102:6835","nonce":3244979225}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":145}],"filesystems":[{"mdsmap":{"epoch":149,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:37:36.348132","modified":"2021-02-10 11:37:38.355444","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8665},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8665":{"gid":8665,"name":"b","rank":0,"incarnation":146,"state":"up:active","state_seq":3,"addr":"172.21.2.113:6833/1669982519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6832","nonce":1669982519},{"type":"v1","addr":"172.21.2.113:6833","nonce":1669982519}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[37],"metadata_pool":36,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":18}]} 2021-02-10T11:37:42.699 INFO:teuthology.orchestra.run.gibba002.stderr:dumped fsmap epoch 149 2021-02-10T11:37:42.700 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:42.698 7fc82a06a700 1 -- 172.21.2.102:0/1265907765 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc8080186c0 msgr2=0x7fc80801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:42.700 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:42.698 7fc82a06a700 1 --2- 172.21.2.102:0/1265907765 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc8080186c0 0x7fc80801abc0 crc :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:42.701 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:42.698 7fc82a06a700 1 -- 172.21.2.102:0/1265907765 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc8241b8ed0 msgr2=0x7fc8240b1af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:42.701 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:42.698 7fc82a06a700 1 --2- 172.21.2.102:0/1265907765 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc8241b8ed0 0x7fc8240b1af0 secure :-1 s=READY pgs=2997 cs=0 l=1 rev1=1 rx=0x7fc81400b530 tx=0x7fc81400b8d0).stop 2021-02-10T11:37:42.701 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:42.699 7fc82a06a700 1 -- 172.21.2.102:0/1265907765 shutdown_connections 2021-02-10T11:37:42.701 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:42.699 7fc82a06a700 1 --2- 172.21.2.102:0/1265907765 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc8080186c0 0x7fc80801abc0 unknown :-1 s=CLOSED pgs=1316 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:42.701 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:42.699 7fc82a06a700 1 --2- 172.21.2.102:0/1265907765 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc8240b2110 0x7fc8241b7a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:42.701 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:42.699 7fc82a06a700 1 --2- 172.21.2.102:0/1265907765 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc8241b8ed0 0x7fc8240b1af0 unknown :-1 s=CLOSED pgs=2997 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:42.701 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:42.699 7fc82a06a700 1 --2- 172.21.2.102:0/1265907765 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc8241b8ad0 0x7fc824187e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:42.701 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:42.699 7fc82a06a700 1 -- 172.21.2.102:0/1265907765 >> 172.21.2.102:0/1265907765 conn(0x7fc8240ed810 msgr2=0x7fc824094980 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:42.702 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:42.699 7fc82a06a700 1 -- 172.21.2.102:0/1265907765 shutdown_connections 2021-02-10T11:37:42.702 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:42.699 7fc82a06a700 1 -- 172.21.2.102:0/1265907765 wait complete. 2021-02-10T11:37:42.911 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 149, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:37:36.348132', 'modified': '2021-02-10 11:37:38.355444', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8665}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8665': {'gid': 8665, 'name': 'b', 'rank': 0, 'incarnation': 146, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.113:6833/1669982519', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.113:6832', 'nonce': 1669982519}, {'type': 'v1', 'addr': '172.21.2.113:6833', 'nonce': 1669982519}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [37], 'metadata_pool': 36, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:37:42.911 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:37:42.911 DEBUG:teuthology.orchestra.run.gibba013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:37:43.309 INFO:teuthology.orchestra.run.gibba013.stdout:{ 2021-02-10T11:37:43.310 INFO:teuthology.orchestra.run.gibba013.stdout: "cluster_fsid": "68db30ea-dbce-4aef-a6df-f01c6fea0513", 2021-02-10T11:37:43.310 INFO:teuthology.orchestra.run.gibba013.stdout: "whoami": 0, 2021-02-10T11:37:43.310 INFO:teuthology.orchestra.run.gibba013.stdout: "id": 8665, 2021-02-10T11:37:43.310 INFO:teuthology.orchestra.run.gibba013.stdout: "want_state": "up:active", 2021-02-10T11:37:43.310 INFO:teuthology.orchestra.run.gibba013.stdout: "state": "up:active", 2021-02-10T11:37:43.310 INFO:teuthology.orchestra.run.gibba013.stdout: "rank_uptime": 6.9584676600000002, 2021-02-10T11:37:43.310 INFO:teuthology.orchestra.run.gibba013.stdout: "mdsmap_epoch": 149, 2021-02-10T11:37:43.310 INFO:teuthology.orchestra.run.gibba013.stdout: "osdmap_epoch": 172, 2021-02-10T11:37:43.310 INFO:teuthology.orchestra.run.gibba013.stdout: "osdmap_epoch_barrier": 172, 2021-02-10T11:37:43.311 INFO:teuthology.orchestra.run.gibba013.stdout: "uptime": 12.631041339999999 2021-02-10T11:37:43.311 INFO:teuthology.orchestra.run.gibba013.stdout:} 2021-02-10T11:37:43.314 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "68db30ea-dbce-4aef-a6df-f01c6fea0513", "whoami": 0, "id": 8665, "want_state": "up:active", "state": "up:active", "rank_uptime": 6.9584676600000002, "mdsmap_epoch": 149, "osdmap_epoch": 172, "osdmap_epoch_barrier": 172, "uptime": 12.631041339999999 } 2021-02-10T11:37:43.314 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:37:43.743 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:43.725 7f7e23016700 1 Processor -- start 2021-02-10T11:37:43.743 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:43.725 7f7e23016700 1 -- start start 2021-02-10T11:37:43.743 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:43.725 7f7e23016700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f7e1c1b0870 0x7f7e1c0f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:43.744 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:43.725 7f7e23016700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e1c1b0c70 con 0x7f7e1c1b0470 2021-02-10T11:37:43.744 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:43.725 7f7e23016700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e1c0f0b50 con 0x7f7e1c0efe00 2021-02-10T11:37:43.744 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:43.725 7f7e23016700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e1c0e5ea0 con 0x7f7e1c1b0870 2021-02-10T11:37:43.744 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:43.726 7f7e1bfff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f7e1c1b0870 0x7f7e1c0f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:43.744 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:43.726 7f7e1bfff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f7e1c1b0870 0x7f7e1c0f20e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:38934/0 (socket says 172.21.2.102:38934) 2021-02-10T11:37:43.744 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:43.726 7f7e1bfff700 1 -- 172.21.2.102:0/3671963580 learned_addr learned my addr 172.21.2.102:0/3671963580 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:37:43.744 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:43.726 7f7e1bfff700 1 -- 172.21.2.102:0/3671963580 >> v1:172.21.2.113:6789/0 conn(0x7f7e1c0efe00 legacy=0x7f7e1c0f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:43.744 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:43.726 7f7e1bfff700 1 -- 172.21.2.102:0/3671963580 >> v1:172.21.2.102:6789/0 conn(0x7f7e1c1b0470 legacy=0x7f7e1c0ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:43.744 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:43.726 7f7e1bfff700 1 -- 172.21.2.102:0/3671963580 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e1c0e52e0 con 0x7f7e1c1b0870 2021-02-10T11:37:43.744 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:43.726 7f7e1bfff700 1 --2- 172.21.2.102:0/3671963580 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7e1c1b0870 0x7f7e1c0f20e0 secure :-1 s=READY pgs=2999 cs=0 l=1 rev1=1 rx=0x7f7e0c006700 tx=0x7f7e0c009b80).ready entity=mon.0 client_cookie=a9d6de738bf4234a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:43.744 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:43.726 7f7e1b7fe700 1 -- 172.21.2.102:0/3671963580 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7e0c00c0c0 con 0x7f7e1c1b0870 2021-02-10T11:37:43.745 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:43.726 7f7e1b7fe700 1 -- 172.21.2.102:0/3671963580 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e0c00c440 con 0x7f7e1c1b0870 2021-02-10T11:37:43.745 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:43.726 7f7e1b7fe700 1 -- 172.21.2.102:0/3671963580 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7e0c00c920 con 0x7f7e1c1b0870 2021-02-10T11:37:43.745 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:43.726 7f7e23016700 1 -- 172.21.2.102:0/3671963580 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7e1c1b0870 msgr2=0x7f7e1c0f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:43.745 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:43.726 7f7e23016700 1 --2- 172.21.2.102:0/3671963580 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7e1c1b0870 0x7f7e1c0f20e0 secure :-1 s=READY pgs=2999 cs=0 l=1 rev1=1 rx=0x7f7e0c006700 tx=0x7f7e0c009b80).stop 2021-02-10T11:37:43.745 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:43.727 7f7e23016700 1 -- 172.21.2.102:0/3671963580 shutdown_connections 2021-02-10T11:37:43.745 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:43.727 7f7e23016700 1 --2- 172.21.2.102:0/3671963580 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7e1c1b0870 0x7f7e1c0f20e0 unknown :-1 s=CLOSED pgs=2999 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:43.745 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:43.727 7f7e23016700 1 -- 172.21.2.102:0/3671963580 >> 172.21.2.102:0/3671963580 conn(0x7f7e1c0ce3e0 msgr2=0x7f7e1c0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:43.745 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:43.727 7f7e23016700 1 -- 172.21.2.102:0/3671963580 shutdown_connections 2021-02-10T11:37:43.745 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:43.727 7f7e23016700 1 -- 172.21.2.102:0/3671963580 wait complete. 2021-02-10T11:37:43.747 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:43.741 7f7e23016700 1 Processor -- start 2021-02-10T11:37:43.747 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:43.741 7f7e23016700 1 -- start start 2021-02-10T11:37:43.747 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:43.742 7f7e23016700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7e1c097170 0x7f7e1c0a71e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:43.747 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:43.742 7f7e23016700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7e1c0961b0 0x7f7e1c0965b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:43.747 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:43.742 7f7e23016700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7e1c096bd0 0x7f7e1c0a7e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:43.747 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:43.742 7f7e23016700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e1c0a8470 con 0x7f7e1c0961b0 2021-02-10T11:37:43.747 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:43.742 7f7e23016700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e1c0a8650 con 0x7f7e1c097170 2021-02-10T11:37:43.748 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:43.742 7f7e23016700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e1c0a8870 con 0x7f7e1c096bd0 2021-02-10T11:37:43.748 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:43.742 7f7e1bfff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7e1c0961b0 0x7f7e1c0965b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:43.748 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:43.742 7f7e1bfff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7e1c0961b0 0x7f7e1c0965b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:38940/0 (socket says 172.21.2.102:38940) 2021-02-10T11:37:43.748 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:43.742 7f7e1bfff700 1 -- 172.21.2.102:0/3527605082 learned_addr learned my addr 172.21.2.102:0/3527605082 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:37:43.748 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:43.742 7f7e20e32700 1 --2- 172.21.2.102:0/3527605082 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7e1c097170 0x7f7e1c0a71e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:43.748 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:43.742 7f7e21633700 1 --2- 172.21.2.102:0/3527605082 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7e1c096bd0 0x7f7e1c0a7e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:43.748 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:43.743 7f7e1bfff700 1 -- 172.21.2.102:0/3527605082 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7e1c096bd0 msgr2=0x7f7e1c0a7e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:43.748 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:43.743 7f7e1bfff700 1 --2- 172.21.2.102:0/3527605082 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7e1c096bd0 0x7f7e1c0a7e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:43.748 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:43.743 7f7e1bfff700 1 -- 172.21.2.102:0/3527605082 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7e1c097170 msgr2=0x7f7e1c0a71e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:43.748 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:43.743 7f7e1bfff700 1 --2- 172.21.2.102:0/3527605082 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7e1c097170 0x7f7e1c0a71e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:43.748 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:43.743 7f7e1bfff700 1 -- 172.21.2.102:0/3527605082 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e1c0a8a90 con 0x7f7e1c0961b0 2021-02-10T11:37:43.748 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:43.743 7f7e1bfff700 1 --2- 172.21.2.102:0/3527605082 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7e1c0961b0 0x7f7e1c0965b0 secure :-1 s=READY pgs=3000 cs=0 l=1 rev1=1 rx=0x7f7e0c00b650 tx=0x7f7e0c00b9f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:43.748 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:43.743 7f7e197fa700 1 -- 172.21.2.102:0/3527605082 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7e0c00bde0 con 0x7f7e1c0961b0 2021-02-10T11:37:43.749 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:43.743 7f7e197fa700 1 -- 172.21.2.102:0/3527605082 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e0c000c90 con 0x7f7e1c0961b0 2021-02-10T11:37:43.749 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:43.743 7f7e23016700 1 -- 172.21.2.102:0/3527605082 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e1c0a8a90 con 0x7f7e1c0961b0 2021-02-10T11:37:43.749 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:43.743 7f7e23016700 1 -- 172.21.2.102:0/3527605082 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e1c0a9020 con 0x7f7e1c0961b0 2021-02-10T11:37:43.749 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:43.743 7f7e197fa700 1 -- 172.21.2.102:0/3527605082 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7e0c002220 con 0x7f7e1c0961b0 2021-02-10T11:37:43.749 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:43.744 7f7e197fa700 1 -- 172.21.2.102:0/3527605082 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f7e0c007000 con 0x7f7e1c0961b0 2021-02-10T11:37:43.749 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:43.744 7f7e197fa700 1 --2- 172.21.2.102:0/3527605082 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7e000186c0 0x7f7e0001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:43.749 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:43.744 7f7e197fa700 1 -- 172.21.2.102:0/3527605082 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f7e0c0327c0 con 0x7f7e1c0961b0 2021-02-10T11:37:43.749 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:43.744 7f7e20e32700 1 --2- 172.21.2.102:0/3527605082 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7e000186c0 0x7f7e0001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:43.749 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:43.744 7f7e20e32700 1 --2- 172.21.2.102:0/3527605082 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7e000186c0 0x7f7e0001abc0 crc :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:43.777 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:43.775 7f7e23016700 1 -- 172.21.2.102:0/3527605082 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e1c0a7920 con 0x7f7e1c0961b0 2021-02-10T11:37:43.779 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:43.777 7f7e197fa700 1 -- 172.21.2.102:0/3527605082 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f7e0c0009b0 con 0x7f7e1c0961b0 2021-02-10T11:37:43.903 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:43.901 7f7e23016700 1 -- 172.21.2.102:0/3527605082 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7e1c0a7920 con 0x7f7e1c0961b0 2021-02-10T11:37:43.904 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:43.902 7f7e197fa700 1 -- 172.21.2.102:0/3527605082 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v172) v1 ==== 74+0+14007 (secure 0 0 0) 0x7f7e0c0009b0 con 0x7f7e1c0961b0 2021-02-10T11:37:43.904 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:37:43.904 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":172,"fsid":"68db30ea-dbce-4aef-a6df-f01c6fea0513","created":"2021-02-10 11:03:06.331557","modified":"2021-02-10 11:37:37.349944","last_up_change":"2021-02-10 11:03:32.690562","last_in_change":"2021-02-10 11:03:32.690562","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":37,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:03:37.322519","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":36,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:37:34.298334","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"171","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":37,"pool_name":"cephfs_data","create_time":"2021-02-10 11:37:35.337127","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"be691b4d-a039-408a-af80-9d53f02b6f1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6818","nonce":36088},{"type":"v1","addr":"172.21.2.102:6820","nonce":36088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6822","nonce":36088},{"type":"v1","addr":"172.21.2.102:6824","nonce":36088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6830","nonce":36088},{"type":"v1","addr":"172.21.2.102:6832","nonce":36088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6826","nonce":36088},{"type":"v1","addr":"172.21.2.102:6828","nonce":36088}]},"public_addr":"172.21.2.102:6820/36088","cluster_addr":"172.21.2.102:6824/36088","heartbeat_back_addr":"172.21.2.102:6832/36088","heartbeat_front_addr":"172.21.2.102:6828/36088","state":["exists","up"]},{"osd":1,"uuid":"64e3ccc8-9c69-4d7c-865e-af8e83822635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6819","nonce":36087},{"type":"v1","addr":"172.21.2.102:6821","nonce":36087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6823","nonce":36087},{"type":"v1","addr":"172.21.2.102:6825","nonce":36087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6831","nonce":36087},{"type":"v1","addr":"172.21.2.102:6833","nonce":36087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6827","nonce":36087},{"type":"v1","addr":"172.21.2.102:6829","nonce":36087}]},"public_addr":"172.21.2.102:6821/36087","cluster_addr":"172.21.2.102:6825/36087","heartbeat_back_addr":"172.21.2.102:6833/36087","heartbeat_front_addr":"172.21.2.102:6829/36087","state":["exists","up"]},{"osd":2,"uuid":"153700af-747f-481f-bdf4-d6072d3a842f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6810","nonce":36089},{"type":"v1","addr":"172.21.2.102:6811","nonce":36089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6812","nonce":36089},{"type":"v1","addr":"172.21.2.102:6813","nonce":36089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6816","nonce":36089},{"type":"v1","addr":"172.21.2.102:6817","nonce":36089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6814","nonce":36089},{"type":"v1","addr":"172.21.2.102:6815","nonce":36089}]},"public_addr":"172.21.2.102:6811/36089","cluster_addr":"172.21.2.102:6813/36089","heartbeat_back_addr":"172.21.2.102:6817/36089","heartbeat_front_addr":"172.21.2.102:6815/36089","state":["exists","up"]},{"osd":3,"uuid":"cbc678f4-83a9-45e6-8830-2f7c232b1ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6802","nonce":36090},{"type":"v1","addr":"172.21.2.102:6803","nonce":36090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6804","nonce":36090},{"type":"v1","addr":"172.21.2.102:6805","nonce":36090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6808","nonce":36090},{"type":"v1","addr":"172.21.2.102:6809","nonce":36090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6806","nonce":36090},{"type":"v1","addr":"172.21.2.102:6807","nonce":36090}]},"public_addr":"172.21.2.102:6803/36090","cluster_addr":"172.21.2.102:6805/36090","heartbeat_back_addr":"172.21.2.102:6809/36090","heartbeat_front_addr":"172.21.2.102:6807/36090","state":["exists","up"]},{"osd":4,"uuid":"c01e4008-b258-4f10-afd2-9e7f37214351","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6808","nonce":36368},{"type":"v1","addr":"172.21.2.113:6811","nonce":36368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6814","nonce":36368},{"type":"v1","addr":"172.21.2.113:6818","nonce":36368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6826","nonce":36368},{"type":"v1","addr":"172.21.2.113:6828","nonce":36368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6820","nonce":36368},{"type":"v1","addr":"172.21.2.113:6824","nonce":36368}]},"public_addr":"172.21.2.113:6811/36368","cluster_addr":"172.21.2.113:6818/36368","heartbeat_back_addr":"172.21.2.113:6828/36368","heartbeat_front_addr":"172.21.2.113:6824/36368","state":["exists","up"]},{"osd":5,"uuid":"551f6007-daf8-4a98-8ee0-c2b16bcccb5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6802","nonce":36366},{"type":"v1","addr":"172.21.2.113:6805","nonce":36366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6807","nonce":36366},{"type":"v1","addr":"172.21.2.113:6810","nonce":36366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6819","nonce":36366},{"type":"v1","addr":"172.21.2.113:6822","nonce":36366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6812","nonce":36366},{"type":"v1","addr":"172.21.2.113:6815","nonce":36366}]},"public_addr":"172.21.2.113:6805/36366","cluster_addr":"172.21.2.113:6810/36366","heartbeat_back_addr":"172.21.2.113:6822/36366","heartbeat_front_addr":"172.21.2.113:6815/36366","state":["exists","up"]},{"osd":6,"uuid":"bf67c040-0e50-48be-b102-c3943bc9ebef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6800","nonce":36367},{"type":"v1","addr":"172.21.2.113:6801","nonce":36367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6803","nonce":36367},{"type":"v1","addr":"172.21.2.113:6804","nonce":36367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6813","nonce":36367},{"type":"v1","addr":"172.21.2.113:6816","nonce":36367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6806","nonce":36367},{"type":"v1","addr":"172.21.2.113:6809","nonce":36367}]},"public_addr":"172.21.2.113:6801/36367","cluster_addr":"172.21.2.113:6804/36367","heartbeat_back_addr":"172.21.2.113:6816/36367","heartbeat_front_addr":"172.21.2.113:6809/36367","state":["exists","up"]},{"osd":7,"uuid":"f5461f80-018d-428f-be89-2eb2bcbe7bb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":169,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6817","nonce":36365},{"type":"v1","addr":"172.21.2.113:6821","nonce":36365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6823","nonce":36365},{"type":"v1","addr":"172.21.2.113:6825","nonce":36365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6830","nonce":36365},{"type":"v1","addr":"172.21.2.113:6831","nonce":36365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6827","nonce":36365},{"type":"v1","addr":"172.21.2.113:6829","nonce":36365}]},"public_addr":"172.21.2.113:6821/36365","cluster_addr":"172.21.2.113:6825/36365","heartbeat_back_addr":"172.21.2.113:6831/36365","heartbeat_front_addr":"172.21.2.113:6829/36365","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:37:43.905 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:43.903 7f7e23016700 1 -- 172.21.2.102:0/3527605082 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7e000186c0 msgr2=0x7f7e0001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:43.905 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:43.903 7f7e23016700 1 --2- 172.21.2.102:0/3527605082 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7e000186c0 0x7f7e0001abc0 crc :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:43.905 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:43.903 7f7e23016700 1 -- 172.21.2.102:0/3527605082 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7e1c0961b0 msgr2=0x7f7e1c0965b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:43.906 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:43.903 7f7e23016700 1 --2- 172.21.2.102:0/3527605082 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7e1c0961b0 0x7f7e1c0965b0 secure :-1 s=READY pgs=3000 cs=0 l=1 rev1=1 rx=0x7f7e0c00b650 tx=0x7f7e0c00b9f0).stop 2021-02-10T11:37:43.906 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:43.904 7f7e23016700 1 -- 172.21.2.102:0/3527605082 shutdown_connections 2021-02-10T11:37:43.906 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:43.904 7f7e23016700 1 --2- 172.21.2.102:0/3527605082 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7e1c096bd0 0x7f7e1c0a7e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:43.906 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:43.904 7f7e23016700 1 --2- 172.21.2.102:0/3527605082 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7e1c0961b0 0x7f7e1c0965b0 unknown :-1 s=CLOSED pgs=3000 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:43.907 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:43.904 7f7e23016700 1 --2- 172.21.2.102:0/3527605082 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7e000186c0 0x7f7e0001abc0 unknown :-1 s=CLOSED pgs=1317 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:43.907 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:43.904 7f7e23016700 1 --2- 172.21.2.102:0/3527605082 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7e1c097170 0x7f7e1c0a71e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:43.907 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:43.904 7f7e23016700 1 -- 172.21.2.102:0/3527605082 >> 172.21.2.102:0/3527605082 conn(0x7f7e1c0c3bd0 msgr2=0x7f7e1c089210 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:43.907 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:43.904 7f7e23016700 1 -- 172.21.2.102:0/3527605082 shutdown_connections 2021-02-10T11:37:43.907 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:43.904 7f7e23016700 1 -- 172.21.2.102:0/3527605082 wait complete. 2021-02-10T11:37:43.915 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:03:37.322519', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 36, 'pool_name': 'cephfs_metadata', 'create_time': '2021-02-10 11:37:34.298334', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '171', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 37, 'pool_name': 'cephfs_data', 'create_time': '2021-02-10 11:37:35.337127', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '172', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-02-10T11:37:43.915 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:37:44.076 DEBUG:teuthology.orchestra.run.gibba002:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:37:44.079 DEBUG:teuthology.orchestra.run.gibba013:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:37:44.349 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:44.330 7f0ff8b74700 1 Processor -- start 2021-02-10T11:37:44.349 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:44.330 7f0ff8b74700 1 -- start start 2021-02-10T11:37:44.349 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:44.330 7f0ff8b74700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0ff40d1870 0x7f0ff41b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:44.349 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:44.330 7f0ff8b74700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f0ff40cff70 0x7f0ff4109130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:44.349 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:44.330 7f0ff8b74700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ff4109a60 con 0x7f0ff41b0ae0 2021-02-10T11:37:44.349 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:44.330 7f0ff8b74700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ff4109750 con 0x7f0ff40cff70 2021-02-10T11:37:44.349 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:44.330 7f0ff8b74700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ff4088070 con 0x7f0ff40d1870 2021-02-10T11:37:44.350 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:44.330 7f0ff305f700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f0ff40cff70 0x7f0ff4109130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:44.350 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:44.330 7f0ff305f700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f0ff40cff70 0x7f0ff4109130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:38952/0 (socket says 172.21.2.102:38952) 2021-02-10T11:37:44.350 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:44.330 7f0ff305f700 1 -- 172.21.2.102:0/3567225216 learned_addr learned my addr 172.21.2.102:0/3567225216 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:37:44.350 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:44.331 7f0ff285e700 1 --2- 172.21.2.102:0/3567225216 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0ff40d1870 0x7f0ff41b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:44.350 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:44.331 7f0ff305f700 1 -- 172.21.2.102:0/3567225216 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0ff40d1870 msgr2=0x7f0ff41b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:37:44.350 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:44.331 7f0ff305f700 1 --2- 172.21.2.102:0/3567225216 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0ff40d1870 0x7f0ff41b04c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:44.350 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:44.331 7f0ff305f700 1 -- 172.21.2.102:0/3567225216 >> v1:172.21.2.113:6789/0 conn(0x7f0ff41b0ae0 legacy=0x7f0ff4108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:44.350 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:44.331 7f0ff305f700 1 -- 172.21.2.102:0/3567225216 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ff4088260 con 0x7f0ff40cff70 2021-02-10T11:37:44.350 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:44.331 7f0ff285e700 1 --2- 172.21.2.102:0/3567225216 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0ff40d1870 0x7f0ff41b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:37:44.350 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:44.331 7f0ff305f700 1 --2- 172.21.2.102:0/3567225216 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0ff40cff70 0x7f0ff4109130 secure :-1 s=READY pgs=3001 cs=0 l=1 rev1=1 rx=0x7f0fec006f00 tx=0x7f0fec009f10).ready entity=mon.0 client_cookie=3d9739fcb063fc9c server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:44.350 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:44.331 7f0ff185c700 1 -- 172.21.2.102:0/3567225216 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0fec00b3b0 con 0x7f0ff40cff70 2021-02-10T11:37:44.350 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:44.331 7f0ff185c700 1 -- 172.21.2.102:0/3567225216 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fec00b630 con 0x7f0ff40cff70 2021-02-10T11:37:44.351 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:44.331 7f0ff185c700 1 -- 172.21.2.102:0/3567225216 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0fec00bb50 con 0x7f0ff40cff70 2021-02-10T11:37:44.351 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:44.331 7f0ff8b74700 1 -- 172.21.2.102:0/3567225216 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0ff40cff70 msgr2=0x7f0ff4109130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:44.351 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:44.331 7f0ff8b74700 1 --2- 172.21.2.102:0/3567225216 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0ff40cff70 0x7f0ff4109130 secure :-1 s=READY pgs=3001 cs=0 l=1 rev1=1 rx=0x7f0fec006f00 tx=0x7f0fec009f10).stop 2021-02-10T11:37:44.351 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:44.332 7f0ff8b74700 1 -- 172.21.2.102:0/3567225216 shutdown_connections 2021-02-10T11:37:44.351 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:44.332 7f0ff8b74700 1 --2- 172.21.2.102:0/3567225216 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0ff40cff70 0x7f0ff4109130 unknown :-1 s=CLOSED pgs=3001 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:44.351 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:44.332 7f0ff8b74700 1 --2- 172.21.2.102:0/3567225216 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0ff40d1870 0x7f0ff41b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:44.351 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:44.332 7f0ff8b74700 1 -- 172.21.2.102:0/3567225216 >> 172.21.2.102:0/3567225216 conn(0x7f0ff40ce420 msgr2=0x7f0ff40d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:44.351 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:44.332 7f0ff8b74700 1 -- 172.21.2.102:0/3567225216 shutdown_connections 2021-02-10T11:37:44.351 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:44.332 7f0ff8b74700 1 -- 172.21.2.102:0/3567225216 wait complete. 2021-02-10T11:37:44.351 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:44.347 7f0ff8b74700 1 Processor -- start 2021-02-10T11:37:44.351 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:44.347 7f0ff8b74700 1 -- start start 2021-02-10T11:37:44.354 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:44.347 7f0ff8b74700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0ff40abd60 0x7f0ff40ac160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:44.354 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:44.347 7f0ff8b74700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0ff40ac780 0x7f0ff40acb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:44.354 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:44.347 7f0ff8b74700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0ff4083910 0x7f0ff4083d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:44.354 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:44.347 7f0ff8b74700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ff4087180 con 0x7f0ff4083910 2021-02-10T11:37:44.354 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:44.347 7f0ff8b74700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ff418fcd0 con 0x7f0ff40abd60 2021-02-10T11:37:44.354 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:44.347 7f0ff8b74700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ff4084330 con 0x7f0ff40ac780 2021-02-10T11:37:44.354 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:44.348 7f0ff305f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0ff4083910 0x7f0ff4083d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:44.354 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:44.348 7f0ff285e700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0ff40abd60 0x7f0ff40ac160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:44.354 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:44.348 7f0ff305f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0ff4083910 0x7f0ff4083d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:38958/0 (socket says 172.21.2.102:38958) 2021-02-10T11:37:44.354 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:44.348 7f0ff305f700 1 -- 172.21.2.102:0/3410723389 learned_addr learned my addr 172.21.2.102:0/3410723389 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:37:44.354 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:44.348 7f0ff205d700 1 --2- 172.21.2.102:0/3410723389 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0ff40ac780 0x7f0ff40acb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:44.355 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:44.348 7f0ff305f700 1 -- 172.21.2.102:0/3410723389 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0ff40ac780 msgr2=0x7f0ff40acb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:44.355 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:44.348 7f0ff305f700 1 --2- 172.21.2.102:0/3410723389 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0ff40ac780 0x7f0ff40acb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:44.355 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:44.348 7f0ff305f700 1 -- 172.21.2.102:0/3410723389 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0ff40abd60 msgr2=0x7f0ff40ac160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:44.355 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:44.348 7f0ff305f700 1 --2- 172.21.2.102:0/3410723389 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0ff40abd60 0x7f0ff40ac160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:44.355 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:44.348 7f0ff305f700 1 -- 172.21.2.102:0/3410723389 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ff4084550 con 0x7f0ff4083910 2021-02-10T11:37:44.355 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:44.348 7f0ff285e700 1 --2- 172.21.2.102:0/3410723389 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0ff40abd60 0x7f0ff40ac160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:37:44.355 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:44.348 7f0ff305f700 1 --2- 172.21.2.102:0/3410723389 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0ff4083910 0x7f0ff4083d10 secure :-1 s=READY pgs=3002 cs=0 l=1 rev1=1 rx=0x7f0fec009390 tx=0x7f0fec00bd40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:44.355 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:44.348 7f0feb7fe700 1 -- 172.21.2.102:0/3410723389 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0fec003030 con 0x7f0ff4083910 2021-02-10T11:37:44.356 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:44.348 7f0feb7fe700 1 -- 172.21.2.102:0/3410723389 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fec0068e0 con 0x7f0ff4083910 2021-02-10T11:37:44.356 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:44.348 7f0ff8b74700 1 -- 172.21.2.102:0/3410723389 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ff4084550 con 0x7f0ff4083910 2021-02-10T11:37:44.356 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:44.348 7f0ff8b74700 1 -- 172.21.2.102:0/3410723389 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ff4084550 con 0x7f0ff4083910 2021-02-10T11:37:44.356 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:44.348 7f0feb7fe700 1 -- 172.21.2.102:0/3410723389 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0fec00c570 con 0x7f0ff4083910 2021-02-10T11:37:44.356 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:44.348 7f0feb7fe700 1 -- 172.21.2.102:0/3410723389 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f0fec005820 con 0x7f0ff4083910 2021-02-10T11:37:44.357 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:44.349 7f0feb7fe700 1 --2- 172.21.2.102:0/3410723389 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0fd8018730 0x7f0fd801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:44.357 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:44.349 7f0feb7fe700 1 -- 172.21.2.102:0/3410723389 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f0fec0329c0 con 0x7f0ff4083910 2021-02-10T11:37:44.357 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:44.349 7f0ff285e700 1 --2- 172.21.2.102:0/3410723389 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0fd8018730 0x7f0fd801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:44.357 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:44.349 7f0ff285e700 1 --2- 172.21.2.102:0/3410723389 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0fd8018730 0x7f0fd801ac10 crc :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:44.382 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:44.380 7f0ff8b74700 1 -- 172.21.2.102:0/3410723389 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ff4082980 con 0x7f0ff4083910 2021-02-10T11:37:44.384 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:44.382 7f0feb7fe700 1 -- 172.21.2.102:0/3410723389 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f0fec006c80 con 0x7f0ff4083910 2021-02-10T11:37:44.514 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:44.512 7f0ff8b74700 1 -- 172.21.2.102:0/3410723389 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f0ff4109a60 con 0x7f0ff4083910 2021-02-10T11:37:44.514 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:44.512 7f0feb7fe700 1 -- 172.21.2.102:0/3410723389 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v172) v1 ==== 92+0+10 (secure 0 0 0) 0x7f0fec005820 con 0x7f0ff4083910 2021-02-10T11:37:44.515 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:37:44.516 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:44.514 7f0ff8b74700 1 -- 172.21.2.102:0/3410723389 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0fd8018730 msgr2=0x7f0fd801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:44.516 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:44.514 7f0ff8b74700 1 --2- 172.21.2.102:0/3410723389 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0fd8018730 0x7f0fd801ac10 crc :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:44.516 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:44.514 7f0ff8b74700 1 -- 172.21.2.102:0/3410723389 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0ff4083910 msgr2=0x7f0ff4083d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:44.516 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:44.514 7f0ff8b74700 1 --2- 172.21.2.102:0/3410723389 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0ff4083910 0x7f0ff4083d10 secure :-1 s=READY pgs=3002 cs=0 l=1 rev1=1 rx=0x7f0fec009390 tx=0x7f0fec00bd40).stop 2021-02-10T11:37:44.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:44.514 7f0ff8b74700 1 -- 172.21.2.102:0/3410723389 shutdown_connections 2021-02-10T11:37:44.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:44.514 7f0ff8b74700 1 --2- 172.21.2.102:0/3410723389 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0ff4083910 0x7f0ff4083d10 unknown :-1 s=CLOSED pgs=3002 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:44.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:44.514 7f0ff8b74700 1 --2- 172.21.2.102:0/3410723389 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0ff40ac780 0x7f0ff40acb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:44.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:44.514 7f0ff8b74700 1 --2- 172.21.2.102:0/3410723389 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0fd8018730 0x7f0fd801ac10 unknown :-1 s=CLOSED pgs=1318 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:44.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:44.514 7f0ff8b74700 1 --2- 172.21.2.102:0/3410723389 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0ff40abd60 0x7f0ff40ac160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:44.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:44.514 7f0ff8b74700 1 -- 172.21.2.102:0/3410723389 >> 172.21.2.102:0/3410723389 conn(0x7f0ff40a2b30 msgr2=0x7f0ff409ce60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:44.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:44.514 7f0ff8b74700 1 -- 172.21.2.102:0/3410723389 shutdown_connections 2021-02-10T11:37:44.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:44.515 7f0ff8b74700 1 -- 172.21.2.102:0/3410723389 wait complete. 2021-02-10T11:37:44.525 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-02-10T11:37:44.959 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:44.940 7f6f55d01700 1 Processor -- start 2021-02-10T11:37:44.959 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:44.940 7f6f55d01700 1 -- start start 2021-02-10T11:37:44.959 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:44.941 7f6f55d01700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f6f500d1c70 0x7f6f500cef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:44.959 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:44.941 7f6f55d01700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6f500cf580 0x7f6f500cf980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:44.959 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:44.941 7f6f55d01700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f500fe1c0 con 0x7f6f500d1870 2021-02-10T11:37:44.959 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:44.941 7f6f55d01700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f501b0cb0 con 0x7f6f500d1c70 2021-02-10T11:37:44.960 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:44.941 7f6f55d01700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f50109a60 con 0x7f6f500cf580 2021-02-10T11:37:44.960 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:44.941 7f6f4effd700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f6f500d1c70 0x7f6f500cef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:44.960 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:44.941 7f6f4effd700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f6f500d1c70 0x7f6f500cef60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.113:3300/0 says I am v2:172.21.2.102:42392/0 (socket says 172.21.2.102:42392) 2021-02-10T11:37:44.960 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:44.941 7f6f4effd700 1 -- 172.21.2.102:0/4058650993 learned_addr learned my addr 172.21.2.102:0/4058650993 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:37:44.960 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:44.941 7f6f4ffff700 1 --2- 172.21.2.102:0/4058650993 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6f500cf580 0x7f6f500cf980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:44.960 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:44.941 7f6f4effd700 1 -- 172.21.2.102:0/4058650993 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6f500cf580 msgr2=0x7f6f500cf980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:37:44.960 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:44.941 7f6f4effd700 1 --2- 172.21.2.102:0/4058650993 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6f500cf580 0x7f6f500cf980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:44.960 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:44.941 7f6f4effd700 1 -- 172.21.2.102:0/4058650993 >> v1:172.21.2.113:6789/0 conn(0x7f6f500d1870 legacy=0x7f6f501b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:44.960 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:44.941 7f6f4effd700 1 -- 172.21.2.102:0/4058650993 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f50088070 con 0x7f6f500d1c70 2021-02-10T11:37:44.960 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:44.942 7f6f4effd700 1 --2- 172.21.2.102:0/4058650993 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6f500d1c70 0x7f6f500cef60 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 rx=0x7f6f40006700 tx=0x7f6f40009b80).ready entity=mon.1 client_cookie=6a433d081fac718e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:44.960 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:44.942 7f6f4e7fc700 1 -- 172.21.2.102:0/4058650993 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6f4000c0c0 con 0x7f6f500d1c70 2021-02-10T11:37:44.961 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:44.942 7f6f4e7fc700 1 -- 172.21.2.102:0/4058650993 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f4000c340 con 0x7f6f500d1c70 2021-02-10T11:37:44.961 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:44.942 7f6f4e7fc700 1 -- 172.21.2.102:0/4058650993 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6f4000c860 con 0x7f6f500d1c70 2021-02-10T11:37:44.961 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:44.942 7f6f55d01700 1 -- 172.21.2.102:0/4058650993 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6f500d1c70 msgr2=0x7f6f500cef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:44.961 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:44.942 7f6f55d01700 1 --2- 172.21.2.102:0/4058650993 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6f500d1c70 0x7f6f500cef60 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 rx=0x7f6f40006700 tx=0x7f6f40009b80).stop 2021-02-10T11:37:44.961 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:44.943 7f6f55d01700 1 -- 172.21.2.102:0/4058650993 shutdown_connections 2021-02-10T11:37:44.961 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:44.943 7f6f55d01700 1 --2- 172.21.2.102:0/4058650993 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6f500cf580 0x7f6f500cf980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:44.961 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:44.943 7f6f55d01700 1 --2- 172.21.2.102:0/4058650993 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6f500d1c70 0x7f6f500cef60 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:44.961 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:44.943 7f6f55d01700 1 -- 172.21.2.102:0/4058650993 >> 172.21.2.102:0/4058650993 conn(0x7f6f500ce420 msgr2=0x7f6f500d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:44.961 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:44.943 7f6f55d01700 1 -- 172.21.2.102:0/4058650993 shutdown_connections 2021-02-10T11:37:44.961 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:44.943 7f6f55d01700 1 -- 172.21.2.102:0/4058650993 wait complete. 2021-02-10T11:37:44.963 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:44.957 7f6f55d01700 1 Processor -- start 2021-02-10T11:37:44.963 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:44.957 7f6f55d01700 1 -- start start 2021-02-10T11:37:44.963 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:44.958 7f6f55d01700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6f50191b10 0x7f6f500ae640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:44.963 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:44.958 7f6f55d01700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6f50191f10 0x7f6f5019ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:44.963 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:44.958 7f6f55d01700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6f501a05f0 0x7f6f50082ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:44.963 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:44.958 7f6f55d01700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f50192440 con 0x7f6f50191b10 2021-02-10T11:37:44.963 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:44.958 7f6f55d01700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f501916e0 con 0x7f6f501a05f0 2021-02-10T11:37:44.963 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:44.958 7f6f55d01700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f50191900 con 0x7f6f50191f10 2021-02-10T11:37:44.963 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:44.958 7f6f4ffff700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6f501a05f0 0x7f6f50082ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:44.963 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:44.958 7f6f4f7fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6f50191b10 0x7f6f500ae640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:44.964 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:44.958 7f6f4ffff700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6f501a05f0 0x7f6f50082ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.113:3300/0 says I am v2:172.21.2.102:42402/0 (socket says 172.21.2.102:42402) 2021-02-10T11:37:44.964 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:44.958 7f6f4ffff700 1 -- 172.21.2.102:0/2096245949 learned_addr learned my addr 172.21.2.102:0/2096245949 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:37:44.964 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:44.958 7f6f4effd700 1 --2- 172.21.2.102:0/2096245949 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6f50191f10 0x7f6f5019ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:44.964 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:44.959 7f6f4ffff700 1 -- 172.21.2.102:0/2096245949 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6f50191f10 msgr2=0x7f6f5019ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:44.964 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:44.959 7f6f4ffff700 1 --2- 172.21.2.102:0/2096245949 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6f50191f10 0x7f6f5019ffd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:44.964 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:44.959 7f6f4ffff700 1 -- 172.21.2.102:0/2096245949 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6f50191b10 msgr2=0x7f6f500ae640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:44.964 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:44.959 7f6f4ffff700 1 --2- 172.21.2.102:0/2096245949 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6f50191b10 0x7f6f500ae640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:44.964 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:44.959 7f6f4ffff700 1 -- 172.21.2.102:0/2096245949 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f500834f0 con 0x7f6f501a05f0 2021-02-10T11:37:44.964 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:44.959 7f6f4ffff700 1 --2- 172.21.2.102:0/2096245949 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6f501a05f0 0x7f6f50082ed0 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 rx=0x7f6f4800b940 tx=0x7f6f4800bce0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:44.964 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:44.959 7f6f47fff700 1 -- 172.21.2.102:0/2096245949 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6f480066f0 con 0x7f6f501a05f0 2021-02-10T11:37:44.964 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:44.959 7f6f47fff700 1 -- 172.21.2.102:0/2096245949 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f48006a90 con 0x7f6f501a05f0 2021-02-10T11:37:44.964 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:44.959 7f6f55d01700 1 -- 172.21.2.102:0/2096245949 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f500834f0 con 0x7f6f501a05f0 2021-02-10T11:37:44.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:44.959 7f6f47fff700 1 -- 172.21.2.102:0/2096245949 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6f480066f0 con 0x7f6f501a05f0 2021-02-10T11:37:44.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:44.959 7f6f55d01700 1 -- 172.21.2.102:0/2096245949 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f500834f0 con 0x7f6f501a05f0 2021-02-10T11:37:44.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:44.959 7f6f47fff700 1 -- 172.21.2.102:0/2096245949 <== mon.1 v2:172.21.2.113:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f6f480056b0 con 0x7f6f501a05f0 2021-02-10T11:37:44.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:44.960 7f6f47fff700 1 --2- 172.21.2.102:0/2096245949 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6f340186c0 0x7f6f3401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:44.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:44.960 7f6f47fff700 1 -- 172.21.2.102:0/2096245949 <== mon.1 v2:172.21.2.113:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f6f480327b0 con 0x7f6f501a05f0 2021-02-10T11:37:44.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:44.960 7f6f4f7fe700 1 --2- 172.21.2.102:0/2096245949 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6f340186c0 0x7f6f3401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:44.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:44.960 7f6f4f7fe700 1 --2- 172.21.2.102:0/2096245949 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6f340186c0 0x7f6f3401abc0 crc :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:44.992 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:44.990 7f6f55d01700 1 -- 172.21.2.102:0/2096245949 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f50192440 con 0x7f6f501a05f0 2021-02-10T11:37:44.995 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:44.993 7f6f47fff700 1 -- 172.21.2.102:0/2096245949 <== mon.1 v2:172.21.2.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f6f4800be20 con 0x7f6f501a05f0 2021-02-10T11:37:45.125 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:45.123 7f6f55d01700 1 -- 172.21.2.102:0/2096245949 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7f6f50192440 con 0x7f6f501a05f0 2021-02-10T11:37:45.126 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:45.124 7f6f47fff700 1 -- 172.21.2.102:0/2096245949 <== mon.1 v2:172.21.2.113:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v172) v1 ==== 104+0+10 (secure 0 0 0) 0x7f6f480056b0 con 0x7f6f501a05f0 2021-02-10T11:37:45.128 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:37:45.130 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:45.128 7f6f55d01700 1 -- 172.21.2.102:0/2096245949 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6f340186c0 msgr2=0x7f6f3401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:45.131 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:45.128 7f6f55d01700 1 --2- 172.21.2.102:0/2096245949 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6f340186c0 0x7f6f3401abc0 crc :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:45.131 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:45.128 7f6f55d01700 1 -- 172.21.2.102:0/2096245949 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6f501a05f0 msgr2=0x7f6f50082ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:45.131 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:45.128 7f6f55d01700 1 --2- 172.21.2.102:0/2096245949 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6f501a05f0 0x7f6f50082ed0 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 rx=0x7f6f4800b940 tx=0x7f6f4800bce0).stop 2021-02-10T11:37:45.131 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:45.128 7f6f55d01700 1 -- 172.21.2.102:0/2096245949 shutdown_connections 2021-02-10T11:37:45.131 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:45.128 7f6f55d01700 1 --2- 172.21.2.102:0/2096245949 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6f340186c0 0x7f6f3401abc0 unknown :-1 s=CLOSED pgs=1319 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:45.131 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:45.128 7f6f55d01700 1 --2- 172.21.2.102:0/2096245949 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6f501a05f0 0x7f6f50082ed0 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:45.131 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:45.128 7f6f55d01700 1 --2- 172.21.2.102:0/2096245949 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6f50191f10 0x7f6f5019ffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:45.131 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:45.128 7f6f55d01700 1 --2- 172.21.2.102:0/2096245949 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6f50191b10 0x7f6f500ae640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:45.131 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:45.128 7f6f55d01700 1 -- 172.21.2.102:0/2096245949 >> 172.21.2.102:0/2096245949 conn(0x7f6f500a02d0 msgr2=0x7f6f5009f200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:45.132 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:45.129 7f6f55d01700 1 -- 172.21.2.102:0/2096245949 shutdown_connections 2021-02-10T11:37:45.132 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:45.129 7f6f55d01700 1 -- 172.21.2.102:0/2096245949 wait complete. 2021-02-10T11:37:45.139 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-02-10T11:37:45.571 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:45.553 7fc7d9dd3700 1 Processor -- start 2021-02-10T11:37:45.571 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:45.553 7fc7d9dd3700 1 -- start start 2021-02-10T11:37:45.572 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:45.553 7fc7d9dd3700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc7d40d1870 0x7fc7d41b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:45.572 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:45.553 7fc7d9dd3700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fc7d41b0ae0 0x7fc7d4108940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:45.572 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:45.553 7fc7d9dd3700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7d4109a60 con 0x7fc7d40d1c70 2021-02-10T11:37:45.572 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:45.553 7fc7d9dd3700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7d40e5ef0 con 0x7fc7d41b0ae0 2021-02-10T11:37:45.572 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:45.553 7fc7d9dd3700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7d4109750 con 0x7fc7d40d1870 2021-02-10T11:37:45.572 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:45.554 7fc7d2ffd700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fc7d41b0ae0 0x7fc7d4108940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:45.572 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:45.554 7fc7d3fff700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fc7d40d1c70 0x7fc7d4108f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:59432/0 (socket says 172.21.2.102:59432) 2021-02-10T11:37:45.572 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:45.554 7fc7d3fff700 1 -- 172.21.2.102:0/2339803992 learned_addr learned my addr 172.21.2.102:0/2339803992 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:37:45.572 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:45.554 7fc7d2ffd700 1 -- 172.21.2.102:0/2339803992 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc7d40d1870 msgr2=0x7fc7d41b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:37:45.572 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:45.554 7fc7d2ffd700 1 --2- 172.21.2.102:0/2339803992 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc7d40d1870 0x7fc7d41b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:45.572 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:45.554 7fc7d2ffd700 1 -- 172.21.2.102:0/2339803992 >> v1:172.21.2.102:6789/0 conn(0x7fc7d40d1c70 legacy=0x7fc7d4108f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:45.572 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:45.554 7fc7d2ffd700 1 -- 172.21.2.102:0/2339803992 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7d4088070 con 0x7fc7d41b0ae0 2021-02-10T11:37:45.572 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:45.554 7fc7d2ffd700 1 --2- 172.21.2.102:0/2339803992 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc7d41b0ae0 0x7fc7d4108940 secure :-1 s=READY pgs=3004 cs=0 l=1 rev1=1 rx=0x7fc7c4006650 tx=0x7fc7c4009a80).ready entity=mon.0 client_cookie=2dc2e1b8f3d79cb7 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:45.573 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:45.554 7fc7d27fc700 1 -- 172.21.2.102:0/2339803992 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc7c40009b0 con 0x7fc7d41b0ae0 2021-02-10T11:37:45.573 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:45.554 7fc7d27fc700 1 -- 172.21.2.102:0/2339803992 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7c400c0c0 con 0x7fc7d41b0ae0 2021-02-10T11:37:45.573 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:45.555 7fc7d27fc700 1 -- 172.21.2.102:0/2339803992 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc7c400c510 con 0x7fc7d41b0ae0 2021-02-10T11:37:45.573 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:45.555 7fc7d9dd3700 1 -- 172.21.2.102:0/2339803992 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc7d41b0ae0 msgr2=0x7fc7d4108940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:45.573 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:45.555 7fc7d9dd3700 1 --2- 172.21.2.102:0/2339803992 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc7d41b0ae0 0x7fc7d4108940 secure :-1 s=READY pgs=3004 cs=0 l=1 rev1=1 rx=0x7fc7c4006650 tx=0x7fc7c4009a80).stop 2021-02-10T11:37:45.573 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:45.555 7fc7d9dd3700 1 -- 172.21.2.102:0/2339803992 shutdown_connections 2021-02-10T11:37:45.573 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:45.555 7fc7d9dd3700 1 --2- 172.21.2.102:0/2339803992 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc7d41b0ae0 0x7fc7d4108940 unknown :-1 s=CLOSED pgs=3004 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:45.573 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:45.555 7fc7d9dd3700 1 --2- 172.21.2.102:0/2339803992 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc7d40d1870 0x7fc7d41b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:45.573 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:45.555 7fc7d9dd3700 1 -- 172.21.2.102:0/2339803992 >> 172.21.2.102:0/2339803992 conn(0x7fc7d40ce420 msgr2=0x7fc7d40d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:45.573 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:45.555 7fc7d9dd3700 1 -- 172.21.2.102:0/2339803992 shutdown_connections 2021-02-10T11:37:45.573 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:45.555 7fc7d9dd3700 1 -- 172.21.2.102:0/2339803992 wait complete. 2021-02-10T11:37:45.575 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:45.570 7fc7d9dd3700 1 Processor -- start 2021-02-10T11:37:45.575 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:45.570 7fc7d9dd3700 1 -- start start 2021-02-10T11:37:45.575 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:45.570 7fc7d9dd3700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc7d40c3820 0x7fc7d40a7950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:45.575 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:45.570 7fc7d9dd3700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc7d41a44c0 0x7fc7d41a48c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:45.575 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:45.570 7fc7d9dd3700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc7d40863d0 0x7fc7d40867d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:45.575 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:45.570 7fc7d9dd3700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7d4085ed0 con 0x7fc7d40863d0 2021-02-10T11:37:45.575 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:45.570 7fc7d9dd3700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7d40c0740 con 0x7fc7d40c3820 2021-02-10T11:37:45.575 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:45.570 7fc7d9dd3700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7d4086df0 con 0x7fc7d41a44c0 2021-02-10T11:37:45.575 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:45.570 7fc7d3fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc7d40863d0 0x7fc7d40867d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:45.575 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:45.570 7fc7d3fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc7d40863d0 0x7fc7d40867d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:38984/0 (socket says 172.21.2.102:38984) 2021-02-10T11:37:45.576 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:45.570 7fc7d3fff700 1 -- 172.21.2.102:0/1534920411 learned_addr learned my addr 172.21.2.102:0/1534920411 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:37:45.576 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:45.571 7fc7d3fff700 1 -- 172.21.2.102:0/1534920411 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc7d41a44c0 msgr2=0x7fc7d41a48c0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:37:45.576 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:45.571 7fc7d3fff700 1 --2- 172.21.2.102:0/1534920411 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc7d41a44c0 0x7fc7d41a48c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:45.576 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:45.571 7fc7d3fff700 1 -- 172.21.2.102:0/1534920411 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc7d40c3820 msgr2=0x7fc7d40a7950 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:37:45.576 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:45.571 7fc7d3fff700 1 --2- 172.21.2.102:0/1534920411 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc7d40c3820 0x7fc7d40a7950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:45.576 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:45.571 7fc7d3fff700 1 -- 172.21.2.102:0/1534920411 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7d4087010 con 0x7fc7d40863d0 2021-02-10T11:37:45.576 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:45.571 7fc7d3fff700 1 --2- 172.21.2.102:0/1534920411 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc7d40863d0 0x7fc7d40867d0 secure :-1 s=READY pgs=3005 cs=0 l=1 rev1=1 rx=0x7fc7cc00a940 tx=0x7fc7cc00ace0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:45.576 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:45.571 7fc7cbfff700 1 -- 172.21.2.102:0/1534920411 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc7cc001270 con 0x7fc7d40863d0 2021-02-10T11:37:45.576 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:45.571 7fc7cbfff700 1 -- 172.21.2.102:0/1534920411 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7cc007cb0 con 0x7fc7d40863d0 2021-02-10T11:37:45.576 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:45.571 7fc7d9dd3700 1 -- 172.21.2.102:0/1534920411 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7d4087010 con 0x7fc7d40863d0 2021-02-10T11:37:45.576 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:45.571 7fc7cbfff700 1 -- 172.21.2.102:0/1534920411 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc7cc004b20 con 0x7fc7d40863d0 2021-02-10T11:37:45.576 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:45.571 7fc7d9dd3700 1 -- 172.21.2.102:0/1534920411 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7d4087010 con 0x7fc7d40863d0 2021-02-10T11:37:45.576 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:45.572 7fc7cbfff700 1 -- 172.21.2.102:0/1534920411 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fc7cc00ae20 con 0x7fc7d40863d0 2021-02-10T11:37:45.576 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:45.572 7fc7cbfff700 1 --2- 172.21.2.102:0/1534920411 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc7b80186c0 0x7fc7b801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:45.576 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:45.572 7fc7cbfff700 1 -- 172.21.2.102:0/1534920411 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fc7cc032760 con 0x7fc7d40863d0 2021-02-10T11:37:45.577 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:45.572 7fc7d37fe700 1 --2- 172.21.2.102:0/1534920411 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc7b80186c0 0x7fc7b801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:45.577 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:45.572 7fc7d37fe700 1 --2- 172.21.2.102:0/1534920411 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc7b80186c0 0x7fc7b801abc0 crc :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:45.605 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:45.603 7fc7d9dd3700 1 -- 172.21.2.102:0/1534920411 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7d40fe1c0 con 0x7fc7d40863d0 2021-02-10T11:37:45.608 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:45.606 7fc7cbfff700 1 -- 172.21.2.102:0/1534920411 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fc7cc000d30 con 0x7fc7d40863d0 2021-02-10T11:37:45.736 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:45.733 7fc7d9dd3700 1 -- 172.21.2.102:0/1534920411 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7fc7d40fe1c0 con 0x7fc7d40863d0 2021-02-10T11:37:45.736 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:45.734 7fc7cbfff700 1 -- 172.21.2.102:0/1534920411 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v172) v1 ==== 100+0+10 (secure 0 0 0) 0x7fc7cc00ae20 con 0x7fc7d40863d0 2021-02-10T11:37:45.737 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:37:45.738 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:45.735 7fc7d9dd3700 1 -- 172.21.2.102:0/1534920411 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc7b80186c0 msgr2=0x7fc7b801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:45.738 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:45.735 7fc7d9dd3700 1 --2- 172.21.2.102:0/1534920411 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc7b80186c0 0x7fc7b801abc0 crc :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:45.739 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:45.735 7fc7d9dd3700 1 -- 172.21.2.102:0/1534920411 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc7d40863d0 msgr2=0x7fc7d40867d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:45.739 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:45.735 7fc7d9dd3700 1 --2- 172.21.2.102:0/1534920411 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc7d40863d0 0x7fc7d40867d0 secure :-1 s=READY pgs=3005 cs=0 l=1 rev1=1 rx=0x7fc7cc00a940 tx=0x7fc7cc00ace0).stop 2021-02-10T11:37:45.739 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:45.736 7fc7d9dd3700 1 -- 172.21.2.102:0/1534920411 shutdown_connections 2021-02-10T11:37:45.739 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:45.736 7fc7d9dd3700 1 --2- 172.21.2.102:0/1534920411 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc7d40863d0 0x7fc7d40867d0 unknown :-1 s=CLOSED pgs=3005 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:45.739 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:45.736 7fc7d9dd3700 1 --2- 172.21.2.102:0/1534920411 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc7d41a44c0 0x7fc7d41a48c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:45.740 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:45.736 7fc7d9dd3700 1 --2- 172.21.2.102:0/1534920411 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc7b80186c0 0x7fc7b801abc0 unknown :-1 s=CLOSED pgs=1320 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:45.740 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:45.736 7fc7d9dd3700 1 --2- 172.21.2.102:0/1534920411 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc7d40c3820 0x7fc7d40a7950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:45.740 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:45.736 7fc7d9dd3700 1 -- 172.21.2.102:0/1534920411 >> 172.21.2.102:0/1534920411 conn(0x7fc7d40872c0 msgr2=0x7fc7d4095700 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:45.740 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:45.736 7fc7d9dd3700 1 -- 172.21.2.102:0/1534920411 shutdown_connections 2021-02-10T11:37:45.740 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:45.736 7fc7d9dd3700 1 -- 172.21.2.102:0/1534920411 wait complete. 2021-02-10T11:37:45.747 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-02-10T11:37:45.747 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:37:46.176 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:46.158 7fde69df0700 1 Processor -- start 2021-02-10T11:37:46.177 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:46.158 7fde69df0700 1 -- start start 2021-02-10T11:37:46.177 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:46.158 7fde69df0700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fde640d1870 0x7fde641b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:46.177 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:46.158 7fde69df0700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fde641b0a10 0x7fde640cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:46.177 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:46.158 7fde69df0700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde640f0b10 con 0x7fde640d1c70 2021-02-10T11:37:46.177 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:46.158 7fde69df0700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde640e5ed0 con 0x7fde641b0a10 2021-02-10T11:37:46.177 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:46.158 7fde69df0700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde640f0650 con 0x7fde640d1870 2021-02-10T11:37:46.177 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:46.158 7fde62ffd700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fde641b0a10 0x7fde640cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:46.177 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:46.159 7fde62ffd700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fde641b0a10 0x7fde640cf330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:38990/0 (socket says 172.21.2.102:38990) 2021-02-10T11:37:46.177 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:46.159 7fde62ffd700 1 -- 172.21.2.102:0/1437972500 learned_addr learned my addr 172.21.2.102:0/1437972500 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:37:46.177 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:46.159 7fde637fe700 1 --2- 172.21.2.102:0/1437972500 >> v2:172.21.2.113:3300/0 conn(0x7fde640d1870 0x7fde641b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:46.177 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:46.159 7fde62ffd700 1 -- 172.21.2.102:0/1437972500 >> v2:172.21.2.113:3300/0 conn(0x7fde640d1870 msgr2=0x7fde641b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:37:46.178 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:46.159 7fde62ffd700 1 --2- 172.21.2.102:0/1437972500 >> v2:172.21.2.113:3300/0 conn(0x7fde640d1870 0x7fde641b03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:46.178 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:46.159 7fde62ffd700 1 -- 172.21.2.102:0/1437972500 >> v1:172.21.2.113:6789/0 conn(0x7fde640d1c70 legacy=0x7fde640cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:37:46.178 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:46.159 7fde62ffd700 1 -- 172.21.2.102:0/1437972500 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde640f0840 con 0x7fde641b0a10 2021-02-10T11:37:46.178 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:46.159 7fde62ffd700 1 --2- 172.21.2.102:0/1437972500 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fde641b0a10 0x7fde640cf330 secure :-1 s=READY pgs=3006 cs=0 l=1 rev1=1 rx=0x7fde58006700 tx=0x7fde58009b80).ready entity=mon.0 client_cookie=675119945d89d300 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:46.178 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:46.159 7fde627fc700 1 -- 172.21.2.102:0/1437972500 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fde5800c0c0 con 0x7fde641b0a10 2021-02-10T11:37:46.178 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:46.159 7fde627fc700 1 -- 172.21.2.102:0/1437972500 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde5800c340 con 0x7fde641b0a10 2021-02-10T11:37:46.178 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:46.159 7fde627fc700 1 -- 172.21.2.102:0/1437972500 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fde5800c820 con 0x7fde641b0a10 2021-02-10T11:37:46.178 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:46.159 7fde69df0700 1 -- 172.21.2.102:0/1437972500 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fde641b0a10 msgr2=0x7fde640cf330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:46.178 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:46.159 7fde69df0700 1 --2- 172.21.2.102:0/1437972500 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fde641b0a10 0x7fde640cf330 secure :-1 s=READY pgs=3006 cs=0 l=1 rev1=1 rx=0x7fde58006700 tx=0x7fde58009b80).stop 2021-02-10T11:37:46.178 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:46.160 7fde69df0700 1 -- 172.21.2.102:0/1437972500 shutdown_connections 2021-02-10T11:37:46.178 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:46.160 7fde69df0700 1 --2- 172.21.2.102:0/1437972500 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fde641b0a10 0x7fde640cf330 unknown :-1 s=CLOSED pgs=3006 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:46.178 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:46.160 7fde69df0700 1 --2- 172.21.2.102:0/1437972500 >> v2:172.21.2.113:3300/0 conn(0x7fde640d1870 0x7fde641b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:46.179 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:46.160 7fde69df0700 1 -- 172.21.2.102:0/1437972500 >> 172.21.2.102:0/1437972500 conn(0x7fde640ce450 msgr2=0x7fde640d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:46.179 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:46.160 7fde69df0700 1 -- 172.21.2.102:0/1437972500 shutdown_connections 2021-02-10T11:37:46.179 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:46.160 7fde69df0700 1 -- 172.21.2.102:0/1437972500 wait complete. 2021-02-10T11:37:46.180 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:46.175 7fde69df0700 1 Processor -- start 2021-02-10T11:37:46.180 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:46.175 7fde69df0700 1 -- start start 2021-02-10T11:37:46.180 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:46.175 7fde69df0700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fde64195650 0x7fde640b5560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:46.181 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:46.175 7fde69df0700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fde640b5b80 0x7fde641a2e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:46.181 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:46.175 7fde69df0700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fde640acfa0 0x7fde641a3440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:46.181 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:46.175 7fde69df0700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde640b6010 con 0x7fde64195650 2021-02-10T11:37:46.181 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:46.175 7fde69df0700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde641a3a60 con 0x7fde640acfa0 2021-02-10T11:37:46.181 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:46.175 7fde69df0700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde641a3c40 con 0x7fde640b5b80 2021-02-10T11:37:46.181 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:46.175 7fde637fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fde64195650 0x7fde640b5560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:46.181 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:46.175 7fde637fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fde64195650 0x7fde640b5560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:38998/0 (socket says 172.21.2.102:38998) 2021-02-10T11:37:46.181 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:46.175 7fde637fe700 1 -- 172.21.2.102:0/3466133519 learned_addr learned my addr 172.21.2.102:0/3466133519 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:37:46.181 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:46.176 7fde637fe700 1 -- 172.21.2.102:0/3466133519 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fde640b5b80 msgr2=0x7fde641a2e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:46.181 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:46.176 7fde637fe700 1 --2- 172.21.2.102:0/3466133519 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fde640b5b80 0x7fde641a2e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:46.181 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:46.176 7fde637fe700 1 -- 172.21.2.102:0/3466133519 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fde640acfa0 msgr2=0x7fde641a3440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:46.181 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:46.176 7fde637fe700 1 --2- 172.21.2.102:0/3466133519 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fde640acfa0 0x7fde641a3440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:46.182 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:46.176 7fde637fe700 1 -- 172.21.2.102:0/3466133519 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde640bcae0 con 0x7fde64195650 2021-02-10T11:37:46.182 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:46.176 7fde637fe700 1 --2- 172.21.2.102:0/3466133519 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fde64195650 0x7fde640b5560 secure :-1 s=READY pgs=3007 cs=0 l=1 rev1=1 rx=0x7fde5400b530 tx=0x7fde5400b8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:46.182 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:46.176 7fde4bfff700 1 -- 172.21.2.102:0/3466133519 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fde5400bcc0 con 0x7fde64195650 2021-02-10T11:37:46.182 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:46.176 7fde4bfff700 1 -- 172.21.2.102:0/3466133519 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde54002030 con 0x7fde64195650 2021-02-10T11:37:46.182 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:46.176 7fde4bfff700 1 -- 172.21.2.102:0/3466133519 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fde54002550 con 0x7fde64195650 2021-02-10T11:37:46.182 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:46.176 7fde69df0700 1 -- 172.21.2.102:0/3466133519 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde640bcae0 con 0x7fde64195650 2021-02-10T11:37:46.182 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:46.176 7fde69df0700 1 -- 172.21.2.102:0/3466133519 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde640bd070 con 0x7fde64195650 2021-02-10T11:37:46.182 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:46.177 7fde4bfff700 1 -- 172.21.2.102:0/3466133519 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fde54000c90 con 0x7fde64195650 2021-02-10T11:37:46.182 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:46.177 7fde4bfff700 1 --2- 172.21.2.102:0/3466133519 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fde440186c0 0x7fde4401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:46.182 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:46.177 7fde4bfff700 1 -- 172.21.2.102:0/3466133519 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fde54032570 con 0x7fde64195650 2021-02-10T11:37:46.182 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:46.177 7fde62ffd700 1 --2- 172.21.2.102:0/3466133519 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fde440186c0 0x7fde4401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:46.182 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:46.177 7fde62ffd700 1 --2- 172.21.2.102:0/3466133519 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fde440186c0 0x7fde4401abc0 crc :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:46.210 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:46.208 7fde69df0700 1 -- 172.21.2.102:0/3466133519 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde640f0ab0 con 0x7fde64195650 2021-02-10T11:37:46.213 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:46.211 7fde4bfff700 1 -- 172.21.2.102:0/3466133519 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fde54000960 con 0x7fde64195650 2021-02-10T11:37:46.342 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:46.340 7fde69df0700 1 -- 172.21.2.102:0/3466133519 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fde640f0ab0 con 0x7fde64195650 2021-02-10T11:37:46.342 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:46.340 7fde4bfff700 1 -- 172.21.2.102:0/3466133519 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 149 v149) v1 ==== 95+0+2108 (secure 0 0 0) 0x7fde54000c90 con 0x7fde64195650 2021-02-10T11:37:46.343 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:37:46.343 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":149,"default_fscid":18,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":12732,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.102:6835/3244979225","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6834","nonce":3244979225},{"type":"v1","addr":"172.21.2.102:6835","nonce":3244979225}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":145}],"filesystems":[{"mdsmap":{"epoch":149,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:37:36.348132","modified":"2021-02-10 11:37:38.355444","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8665},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8665":{"gid":8665,"name":"b","rank":0,"incarnation":146,"state":"up:active","state_seq":3,"addr":"172.21.2.113:6833/1669982519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6832","nonce":1669982519},{"type":"v1","addr":"172.21.2.113:6833","nonce":1669982519}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[37],"metadata_pool":36,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":18}]} 2021-02-10T11:37:46.343 INFO:teuthology.orchestra.run.gibba002.stderr:dumped fsmap epoch 149 2021-02-10T11:37:46.344 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:46.342 7fde69df0700 1 -- 172.21.2.102:0/3466133519 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fde440186c0 msgr2=0x7fde4401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:46.344 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:46.342 7fde69df0700 1 --2- 172.21.2.102:0/3466133519 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fde440186c0 0x7fde4401abc0 crc :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:46.344 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:46.342 7fde69df0700 1 -- 172.21.2.102:0/3466133519 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fde64195650 msgr2=0x7fde640b5560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:46.344 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:46.342 7fde69df0700 1 --2- 172.21.2.102:0/3466133519 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fde64195650 0x7fde640b5560 secure :-1 s=READY pgs=3007 cs=0 l=1 rev1=1 rx=0x7fde5400b530 tx=0x7fde5400b8d0).stop 2021-02-10T11:37:46.344 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:46.342 7fde69df0700 1 -- 172.21.2.102:0/3466133519 shutdown_connections 2021-02-10T11:37:46.344 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:46.342 7fde69df0700 1 --2- 172.21.2.102:0/3466133519 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fde440186c0 0x7fde4401abc0 unknown :-1 s=CLOSED pgs=1321 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:46.345 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:46.342 7fde69df0700 1 --2- 172.21.2.102:0/3466133519 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fde640acfa0 0x7fde641a3440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:46.345 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:46.342 7fde69df0700 1 --2- 172.21.2.102:0/3466133519 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fde640b5b80 0x7fde641a2e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:46.345 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:46.342 7fde69df0700 1 --2- 172.21.2.102:0/3466133519 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fde64195650 0x7fde640b5560 unknown :-1 s=CLOSED pgs=3007 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:46.345 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:46.342 7fde69df0700 1 -- 172.21.2.102:0/3466133519 >> 172.21.2.102:0/3466133519 conn(0x7fde640b3040 msgr2=0x7fde640c3250 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:46.345 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:46.343 7fde69df0700 1 -- 172.21.2.102:0/3466133519 shutdown_connections 2021-02-10T11:37:46.345 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:46.343 7fde69df0700 1 -- 172.21.2.102:0/3466133519 wait complete. 2021-02-10T11:37:46.554 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 149, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:37:36.348132', 'modified': '2021-02-10 11:37:38.355444', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8665}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8665': {'gid': 8665, 'name': 'b', 'rank': 0, 'incarnation': 146, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.113:6833/1669982519', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.113:6832', 'nonce': 1669982519}, {'type': 'v1', 'addr': '172.21.2.113:6833', 'nonce': 1669982519}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [37], 'metadata_pool': 36, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:37:46.555 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:37:46.555 DEBUG:teuthology.orchestra.run.gibba013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:37:46.955 INFO:teuthology.orchestra.run.gibba013.stdout:{ 2021-02-10T11:37:46.955 INFO:teuthology.orchestra.run.gibba013.stdout: "cluster_fsid": "68db30ea-dbce-4aef-a6df-f01c6fea0513", 2021-02-10T11:37:46.955 INFO:teuthology.orchestra.run.gibba013.stdout: "whoami": 0, 2021-02-10T11:37:46.955 INFO:teuthology.orchestra.run.gibba013.stdout: "id": 8665, 2021-02-10T11:37:46.955 INFO:teuthology.orchestra.run.gibba013.stdout: "want_state": "up:active", 2021-02-10T11:37:46.955 INFO:teuthology.orchestra.run.gibba013.stdout: "state": "up:active", 2021-02-10T11:37:46.955 INFO:teuthology.orchestra.run.gibba013.stdout: "rank_uptime": 10.603742963, 2021-02-10T11:37:46.955 INFO:teuthology.orchestra.run.gibba013.stdout: "mdsmap_epoch": 149, 2021-02-10T11:37:46.955 INFO:teuthology.orchestra.run.gibba013.stdout: "osdmap_epoch": 172, 2021-02-10T11:37:46.955 INFO:teuthology.orchestra.run.gibba013.stdout: "osdmap_epoch_barrier": 172, 2021-02-10T11:37:46.955 INFO:teuthology.orchestra.run.gibba013.stdout: "uptime": 16.276317289000001 2021-02-10T11:37:46.956 INFO:teuthology.orchestra.run.gibba013.stdout:} 2021-02-10T11:37:46.959 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "68db30ea-dbce-4aef-a6df-f01c6fea0513", "whoami": 0, "id": 8665, "want_state": "up:active", "state": "up:active", "rank_uptime": 10.603742963, "mdsmap_epoch": 149, "osdmap_epoch": 172, "osdmap_epoch_barrier": 172, "uptime": 16.276317289000001 } 2021-02-10T11:37:46.959 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2021-02-10T11:37:46.959 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2021-02-10T11:37:46.960 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@gibba002.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2021-02-10T11:37:46.960 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:37:46.960 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec mkdir -p /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:37:47.296 DEBUG:teuthology.orchestra.run.gibba002:> sudo modprobe fuse 2021-02-10T11:37:47.642 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:37:47.642 DEBUG:teuthology.orchestra.run.gibba002:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:37:47.987 INFO:teuthology.orchestra.run.gibba002.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:37:47.987 INFO:teuthology.orchestra.run.gibba002.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:37:47.988 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:37:47.988 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:37:47.988 DEBUG:teuthology.orchestra.run.gibba002:> ls /sys/fs/fuse/connections 2021-02-10T11:37:48.325 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2021-02-10T11:37:48.326 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:37:48.329 DEBUG:teuthology.orchestra.run.gibba002:> sudo modprobe fuse 2021-02-10T11:37:48.672 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:37:48.672 DEBUG:teuthology.orchestra.run.gibba002:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:37:48.739 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba002.stderr:ceph-fuse[208373]: starting ceph client 2021-02-10T11:37:48.739 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba002.stderr:2021-02-10 11:37:48.736 7f2182486f80 -1 init, newargv = 0x55e7c05c2160 newargc=9 2021-02-10T11:37:48.745 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba002.stderr:ceph-fuse[208373]: starting fuse 2021-02-10T11:37:49.016 INFO:teuthology.orchestra.run.gibba002.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:37:49.016 INFO:teuthology.orchestra.run.gibba002.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:37:49.017 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:37:49.017 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:37:49.017 DEBUG:teuthology.orchestra.run.gibba002:> ls /sys/fs/fuse/connections 2021-02-10T11:37:49.351 INFO:teuthology.orchestra.run.gibba002.stdout:43 2021-02-10T11:37:49.351 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2021-02-10T11:37:49.352 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:37:49.352 DEBUG:teuthology.orchestra.run.gibba002:> sudo python3 -c ' 2021-02-10T11:37:49.352 DEBUG:teuthology.orchestra.run.gibba002:> import glob 2021-02-10T11:37:49.352 DEBUG:teuthology.orchestra.run.gibba002:> import re 2021-02-10T11:37:49.352 DEBUG:teuthology.orchestra.run.gibba002:> import os 2021-02-10T11:37:49.352 DEBUG:teuthology.orchestra.run.gibba002:> import subprocess 2021-02-10T11:37:49.352 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:37:49.352 DEBUG:teuthology.orchestra.run.gibba002:> def find_socket(client_name): 2021-02-10T11:37:49.352 DEBUG:teuthology.orchestra.run.gibba002:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2021-02-10T11:37:49.352 DEBUG:teuthology.orchestra.run.gibba002:> files = glob.glob(asok_path) 2021-02-10T11:37:49.352 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:37:49.352 DEBUG:teuthology.orchestra.run.gibba002:> # Given a non-glob path, it better be there 2021-02-10T11:37:49.353 DEBUG:teuthology.orchestra.run.gibba002:> if "*" not in asok_path: 2021-02-10T11:37:49.353 DEBUG:teuthology.orchestra.run.gibba002:> assert(len(files) == 1) 2021-02-10T11:37:49.353 DEBUG:teuthology.orchestra.run.gibba002:> return files[0] 2021-02-10T11:37:49.353 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:37:49.353 DEBUG:teuthology.orchestra.run.gibba002:> for f in files: 2021-02-10T11:37:49.353 DEBUG:teuthology.orchestra.run.gibba002:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2021-02-10T11:37:49.353 DEBUG:teuthology.orchestra.run.gibba002:> if os.path.exists("/proc/{0}".format(pid)): 2021-02-10T11:37:49.353 DEBUG:teuthology.orchestra.run.gibba002:> return f 2021-02-10T11:37:49.353 DEBUG:teuthology.orchestra.run.gibba002:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2021-02-10T11:37:49.353 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:37:49.353 DEBUG:teuthology.orchestra.run.gibba002:> print(find_socket("client.0")) 2021-02-10T11:37:49.353 DEBUG:teuthology.orchestra.run.gibba002:> ' 2021-02-10T11:37:49.715 INFO:teuthology.orchestra.run.gibba002.stdout:/var/run/ceph/ceph-client.0.208373.asok 2021-02-10T11:37:49.718 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.208373.asok 2021-02-10T11:37:49.718 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:37:49.718 DEBUG:teuthology.orchestra.run.gibba002:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.208373.asok status 2021-02-10T11:37:50.109 INFO:teuthology.orchestra.run.gibba002.stdout:{ 2021-02-10T11:37:50.109 INFO:teuthology.orchestra.run.gibba002.stdout: "metadata": { 2021-02-10T11:37:50.109 INFO:teuthology.orchestra.run.gibba002.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:37:50.110 INFO:teuthology.orchestra.run.gibba002.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:37:50.110 INFO:teuthology.orchestra.run.gibba002.stdout: "entity_id": "0", 2021-02-10T11:37:50.110 INFO:teuthology.orchestra.run.gibba002.stdout: "hostname": "gibba002", 2021-02-10T11:37:50.110 INFO:teuthology.orchestra.run.gibba002.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2021-02-10T11:37:50.110 INFO:teuthology.orchestra.run.gibba002.stdout: "pid": "208373", 2021-02-10T11:37:50.110 INFO:teuthology.orchestra.run.gibba002.stdout: "root": "/" 2021-02-10T11:37:50.110 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:37:50.110 INFO:teuthology.orchestra.run.gibba002.stdout: "dentry_count": 0, 2021-02-10T11:37:50.110 INFO:teuthology.orchestra.run.gibba002.stdout: "dentry_pinned_count": 0, 2021-02-10T11:37:50.110 INFO:teuthology.orchestra.run.gibba002.stdout: "id": 12894, 2021-02-10T11:37:50.110 INFO:teuthology.orchestra.run.gibba002.stdout: "inst": { 2021-02-10T11:37:50.111 INFO:teuthology.orchestra.run.gibba002.stdout: "name": { 2021-02-10T11:37:50.111 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "client", 2021-02-10T11:37:50.111 INFO:teuthology.orchestra.run.gibba002.stdout: "num": 12894 2021-02-10T11:37:50.111 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:37:50.111 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": { 2021-02-10T11:37:50.111 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:37:50.111 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:0", 2021-02-10T11:37:50.111 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 268572943 2021-02-10T11:37:50.111 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:37:50.111 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:37:50.111 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": { 2021-02-10T11:37:50.111 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:37:50.112 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:0", 2021-02-10T11:37:50.112 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 268572943 2021-02-10T11:37:50.112 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:37:50.112 INFO:teuthology.orchestra.run.gibba002.stdout: "inst_str": "client.12894 172.21.2.102:0/268572943", 2021-02-10T11:37:50.112 INFO:teuthology.orchestra.run.gibba002.stdout: "addr_str": "172.21.2.102:0/268572943", 2021-02-10T11:37:50.112 INFO:teuthology.orchestra.run.gibba002.stdout: "inode_count": 1, 2021-02-10T11:37:50.112 INFO:teuthology.orchestra.run.gibba002.stdout: "mds_epoch": 149, 2021-02-10T11:37:50.112 INFO:teuthology.orchestra.run.gibba002.stdout: "osd_epoch": 172, 2021-02-10T11:37:50.112 INFO:teuthology.orchestra.run.gibba002.stdout: "osd_epoch_barrier": 0, 2021-02-10T11:37:50.112 INFO:teuthology.orchestra.run.gibba002.stdout: "blacklisted": false 2021-02-10T11:37:50.112 INFO:teuthology.orchestra.run.gibba002.stdout:} 2021-02-10T11:37:50.113 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:37:50.114 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:37:50.114 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:37:50.451 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:37:50.451 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:37:50.451 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:37:50.452 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:37:50.798 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:37:51.228 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:51.209 7f51a365c700 1 Processor -- start 2021-02-10T11:37:51.228 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:51.209 7f51a365c700 1 -- start start 2021-02-10T11:37:51.228 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:51.210 7f51a365c700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f519c0d1870 0x7f519c1b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:51.228 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:51.210 7f51a365c700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f519c0f0650 0x7f519c0cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:51.229 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:51.210 7f51a365c700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f519c0f0b10 con 0x7f519c1b0a10 2021-02-10T11:37:51.229 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:51.210 7f51a365c700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f519c0e5ed0 con 0x7f519c0d1870 2021-02-10T11:37:51.229 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:51.210 7f51a365c700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f519c0d1c70 con 0x7f519c0f0650 2021-02-10T11:37:51.229 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:51.210 7f51a1478700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f519c0d1870 0x7f519c1b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:51.229 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:51.210 7f51a1478700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f519c0d1870 0x7f519c1b03f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:39018/0 (socket says 172.21.2.102:39018) 2021-02-10T11:37:51.229 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:51.210 7f51a1478700 1 -- 172.21.2.102:0/2596200459 learned_addr learned my addr 172.21.2.102:0/2596200459 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:37:51.229 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:51.210 7f51a1c79700 1 --2- 172.21.2.102:0/2596200459 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f519c0f0650 0x7f519c0cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:51.229 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:51.210 7f51a1478700 1 -- 172.21.2.102:0/2596200459 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f519c0f0650 msgr2=0x7f519c0cfb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:37:51.229 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:51.210 7f51a1478700 1 --2- 172.21.2.102:0/2596200459 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f519c0f0650 0x7f519c0cfb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:51.229 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:51.210 7f51a1478700 1 -- 172.21.2.102:0/2596200459 >> v1:172.21.2.113:6789/0 conn(0x7f519c1b0a10 legacy=0x7f519c0cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:51.229 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:51.210 7f51a1478700 1 -- 172.21.2.102:0/2596200459 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f519c0d1e90 con 0x7f519c0d1870 2021-02-10T11:37:51.229 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:51.210 7f51a1478700 1 --2- 172.21.2.102:0/2596200459 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f519c0d1870 0x7f519c1b03f0 secure :-1 s=READY pgs=3010 cs=0 l=1 rev1=1 rx=0x7f5190006700 tx=0x7f5190009b80).ready entity=mon.0 client_cookie=46364e3a95bbdb07 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:51.229 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:51.211 7f518ffff700 1 -- 172.21.2.102:0/2596200459 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f519000c0c0 con 0x7f519c0d1870 2021-02-10T11:37:51.229 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:51.211 7f518ffff700 1 -- 172.21.2.102:0/2596200459 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f519000c340 con 0x7f519c0d1870 2021-02-10T11:37:51.230 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:51.211 7f518ffff700 1 -- 172.21.2.102:0/2596200459 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f519000c860 con 0x7f519c0d1870 2021-02-10T11:37:51.230 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:51.211 7f51a365c700 1 -- 172.21.2.102:0/2596200459 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f519c0d1870 msgr2=0x7f519c1b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:51.230 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:51.211 7f51a365c700 1 --2- 172.21.2.102:0/2596200459 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f519c0d1870 0x7f519c1b03f0 secure :-1 s=READY pgs=3010 cs=0 l=1 rev1=1 rx=0x7f5190006700 tx=0x7f5190009b80).stop 2021-02-10T11:37:51.230 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:51.211 7f51a365c700 1 -- 172.21.2.102:0/2596200459 shutdown_connections 2021-02-10T11:37:51.230 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:51.211 7f51a365c700 1 --2- 172.21.2.102:0/2596200459 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f519c0f0650 0x7f519c0cfb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:51.230 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:51.211 7f51a365c700 1 --2- 172.21.2.102:0/2596200459 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f519c0d1870 0x7f519c1b03f0 unknown :-1 s=CLOSED pgs=3010 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:51.230 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:51.211 7f51a365c700 1 -- 172.21.2.102:0/2596200459 >> 172.21.2.102:0/2596200459 conn(0x7f519c0ce450 msgr2=0x7f519c0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:51.230 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:51.212 7f51a365c700 1 -- 172.21.2.102:0/2596200459 shutdown_connections 2021-02-10T11:37:51.230 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:51.212 7f51a365c700 1 -- 172.21.2.102:0/2596200459 wait complete. 2021-02-10T11:37:51.232 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:51.226 7f51a365c700 1 Processor -- start 2021-02-10T11:37:51.232 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:51.226 7f51a365c700 1 -- start start 2021-02-10T11:37:51.232 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:51.227 7f51a365c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f519c0ada90 0x7f519c0ade90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:51.232 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:51.227 7f51a365c700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f519c099d20 0x7f519c09a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:51.232 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:51.227 7f51a365c700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f519c09a740 0x7f519c097bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:51.232 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:51.227 7f51a365c700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f519c0997a0 con 0x7f519c0ada90 2021-02-10T11:37:51.232 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:51.227 7f51a365c700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f519c18f0c0 con 0x7f519c099d20 2021-02-10T11:37:51.232 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:51.227 7f51a365c700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f519c09ab40 con 0x7f519c09a740 2021-02-10T11:37:51.232 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:51.227 7f51a1478700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f519c0ada90 0x7f519c0ade90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:51.233 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:51.227 7f51a1478700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f519c0ada90 0x7f519c0ade90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:39022/0 (socket says 172.21.2.102:39022) 2021-02-10T11:37:51.233 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:51.227 7f51a1478700 1 -- 172.21.2.102:0/2230746287 learned_addr learned my addr 172.21.2.102:0/2230746287 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:37:51.233 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:51.227 7f51a0c77700 1 --2- 172.21.2.102:0/2230746287 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f519c099d20 0x7f519c09a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:51.233 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:51.227 7f51a1c79700 1 --2- 172.21.2.102:0/2230746287 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f519c09a740 0x7f519c097bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:51.233 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:51.227 7f51a1478700 1 -- 172.21.2.102:0/2230746287 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f519c09a740 msgr2=0x7f519c097bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:51.233 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:51.227 7f51a1478700 1 --2- 172.21.2.102:0/2230746287 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f519c09a740 0x7f519c097bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:51.233 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:51.228 7f51a1478700 1 -- 172.21.2.102:0/2230746287 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f519c099d20 msgr2=0x7f519c09a120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:51.233 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:51.228 7f51a1478700 1 --2- 172.21.2.102:0/2230746287 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f519c099d20 0x7f519c09a120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:51.233 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:51.228 7f51a1478700 1 -- 172.21.2.102:0/2230746287 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f519c09ad60 con 0x7f519c0ada90 2021-02-10T11:37:51.233 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:51.228 7f51a0c77700 1 --2- 172.21.2.102:0/2230746287 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f519c099d20 0x7f519c09a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:37:51.233 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:51.228 7f51a1478700 1 --2- 172.21.2.102:0/2230746287 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f519c0ada90 0x7f519c0ade90 secure :-1 s=READY pgs=3011 cs=0 l=1 rev1=1 rx=0x7f519000b580 tx=0x7f519000b920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:51.233 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:51.228 7f518dffb700 1 -- 172.21.2.102:0/2230746287 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f519000bd10 con 0x7f519c0ada90 2021-02-10T11:37:51.233 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:51.228 7f518dffb700 1 -- 172.21.2.102:0/2230746287 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5190002030 con 0x7f519c0ada90 2021-02-10T11:37:51.234 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:51.228 7f51a365c700 1 -- 172.21.2.102:0/2230746287 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f519c09ad60 con 0x7f519c0ada90 2021-02-10T11:37:51.234 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:51.228 7f51a365c700 1 -- 172.21.2.102:0/2230746287 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f519c0984d0 con 0x7f519c0ada90 2021-02-10T11:37:51.234 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:51.228 7f518dffb700 1 -- 172.21.2.102:0/2230746287 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5190002480 con 0x7f519c0ada90 2021-02-10T11:37:51.234 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:51.229 7f518dffb700 1 -- 172.21.2.102:0/2230746287 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f5190000c90 con 0x7f519c0ada90 2021-02-10T11:37:51.234 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:51.229 7f518dffb700 1 --2- 172.21.2.102:0/2230746287 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f517c0186c0 0x7f517c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:51.234 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:51.229 7f518dffb700 1 -- 172.21.2.102:0/2230746287 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f5190032530 con 0x7f519c0ada90 2021-02-10T11:37:51.234 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:51.229 7f51a0c77700 1 --2- 172.21.2.102:0/2230746287 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f517c0186c0 0x7f517c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:51.234 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:51.229 7f51a0c77700 1 --2- 172.21.2.102:0/2230746287 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f517c0186c0 0x7f517c01abc0 crc :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:51.261 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:51.259 7f51a365c700 1 -- 172.21.2.102:0/2230746287 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f519c0ebc00 con 0x7f519c0ada90 2021-02-10T11:37:51.264 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:51.262 7f518dffb700 1 -- 172.21.2.102:0/2230746287 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f51900009b0 con 0x7f519c0ada90 2021-02-10T11:37:51.387 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:51.385 7f51a365c700 1 -- 172.21.2.102:0/2230746287 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f519c0ebc00 con 0x7f519c0ada90 2021-02-10T11:37:51.388 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:51.386 7f518dffb700 1 -- 172.21.2.102:0/2230746287 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v172) v1 ==== 74+0+14007 (secure 0 0 0) 0x7f5190000c90 con 0x7f519c0ada90 2021-02-10T11:37:51.388 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:37:51.388 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":172,"fsid":"68db30ea-dbce-4aef-a6df-f01c6fea0513","created":"2021-02-10 11:03:06.331557","modified":"2021-02-10 11:37:37.349944","last_up_change":"2021-02-10 11:03:32.690562","last_in_change":"2021-02-10 11:03:32.690562","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":37,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:03:37.322519","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":36,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:37:34.298334","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"171","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":37,"pool_name":"cephfs_data","create_time":"2021-02-10 11:37:35.337127","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"172","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"be691b4d-a039-408a-af80-9d53f02b6f1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6818","nonce":36088},{"type":"v1","addr":"172.21.2.102:6820","nonce":36088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6822","nonce":36088},{"type":"v1","addr":"172.21.2.102:6824","nonce":36088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6830","nonce":36088},{"type":"v1","addr":"172.21.2.102:6832","nonce":36088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6826","nonce":36088},{"type":"v1","addr":"172.21.2.102:6828","nonce":36088}]},"public_addr":"172.21.2.102:6820/36088","cluster_addr":"172.21.2.102:6824/36088","heartbeat_back_addr":"172.21.2.102:6832/36088","heartbeat_front_addr":"172.21.2.102:6828/36088","state":["exists","up"]},{"osd":1,"uuid":"64e3ccc8-9c69-4d7c-865e-af8e83822635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6819","nonce":36087},{"type":"v1","addr":"172.21.2.102:6821","nonce":36087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6823","nonce":36087},{"type":"v1","addr":"172.21.2.102:6825","nonce":36087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6831","nonce":36087},{"type":"v1","addr":"172.21.2.102:6833","nonce":36087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6827","nonce":36087},{"type":"v1","addr":"172.21.2.102:6829","nonce":36087}]},"public_addr":"172.21.2.102:6821/36087","cluster_addr":"172.21.2.102:6825/36087","heartbeat_back_addr":"172.21.2.102:6833/36087","heartbeat_front_addr":"172.21.2.102:6829/36087","state":["exists","up"]},{"osd":2,"uuid":"153700af-747f-481f-bdf4-d6072d3a842f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6810","nonce":36089},{"type":"v1","addr":"172.21.2.102:6811","nonce":36089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6812","nonce":36089},{"type":"v1","addr":"172.21.2.102:6813","nonce":36089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6816","nonce":36089},{"type":"v1","addr":"172.21.2.102:6817","nonce":36089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6814","nonce":36089},{"type":"v1","addr":"172.21.2.102:6815","nonce":36089}]},"public_addr":"172.21.2.102:6811/36089","cluster_addr":"172.21.2.102:6813/36089","heartbeat_back_addr":"172.21.2.102:6817/36089","heartbeat_front_addr":"172.21.2.102:6815/36089","state":["exists","up"]},{"osd":3,"uuid":"cbc678f4-83a9-45e6-8830-2f7c232b1ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6802","nonce":36090},{"type":"v1","addr":"172.21.2.102:6803","nonce":36090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6804","nonce":36090},{"type":"v1","addr":"172.21.2.102:6805","nonce":36090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6808","nonce":36090},{"type":"v1","addr":"172.21.2.102:6809","nonce":36090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6806","nonce":36090},{"type":"v1","addr":"172.21.2.102:6807","nonce":36090}]},"public_addr":"172.21.2.102:6803/36090","cluster_addr":"172.21.2.102:6805/36090","heartbeat_back_addr":"172.21.2.102:6809/36090","heartbeat_front_addr":"172.21.2.102:6807/36090","state":["exists","up"]},{"osd":4,"uuid":"c01e4008-b258-4f10-afd2-9e7f37214351","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6808","nonce":36368},{"type":"v1","addr":"172.21.2.113:6811","nonce":36368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6814","nonce":36368},{"type":"v1","addr":"172.21.2.113:6818","nonce":36368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6826","nonce":36368},{"type":"v1","addr":"172.21.2.113:6828","nonce":36368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6820","nonce":36368},{"type":"v1","addr":"172.21.2.113:6824","nonce":36368}]},"public_addr":"172.21.2.113:6811/36368","cluster_addr":"172.21.2.113:6818/36368","heartbeat_back_addr":"172.21.2.113:6828/36368","heartbeat_front_addr":"172.21.2.113:6824/36368","state":["exists","up"]},{"osd":5,"uuid":"551f6007-daf8-4a98-8ee0-c2b16bcccb5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6802","nonce":36366},{"type":"v1","addr":"172.21.2.113:6805","nonce":36366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6807","nonce":36366},{"type":"v1","addr":"172.21.2.113:6810","nonce":36366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6819","nonce":36366},{"type":"v1","addr":"172.21.2.113:6822","nonce":36366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6812","nonce":36366},{"type":"v1","addr":"172.21.2.113:6815","nonce":36366}]},"public_addr":"172.21.2.113:6805/36366","cluster_addr":"172.21.2.113:6810/36366","heartbeat_back_addr":"172.21.2.113:6822/36366","heartbeat_front_addr":"172.21.2.113:6815/36366","state":["exists","up"]},{"osd":6,"uuid":"bf67c040-0e50-48be-b102-c3943bc9ebef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6800","nonce":36367},{"type":"v1","addr":"172.21.2.113:6801","nonce":36367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6803","nonce":36367},{"type":"v1","addr":"172.21.2.113:6804","nonce":36367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6813","nonce":36367},{"type":"v1","addr":"172.21.2.113:6816","nonce":36367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6806","nonce":36367},{"type":"v1","addr":"172.21.2.113:6809","nonce":36367}]},"public_addr":"172.21.2.113:6801/36367","cluster_addr":"172.21.2.113:6804/36367","heartbeat_back_addr":"172.21.2.113:6816/36367","heartbeat_front_addr":"172.21.2.113:6809/36367","state":["exists","up"]},{"osd":7,"uuid":"f5461f80-018d-428f-be89-2eb2bcbe7bb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":169,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6817","nonce":36365},{"type":"v1","addr":"172.21.2.113:6821","nonce":36365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6823","nonce":36365},{"type":"v1","addr":"172.21.2.113:6825","nonce":36365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6830","nonce":36365},{"type":"v1","addr":"172.21.2.113:6831","nonce":36365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6827","nonce":36365},{"type":"v1","addr":"172.21.2.113:6829","nonce":36365}]},"public_addr":"172.21.2.113:6821/36365","cluster_addr":"172.21.2.113:6825/36365","heartbeat_back_addr":"172.21.2.113:6831/36365","heartbeat_front_addr":"172.21.2.113:6829/36365","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:37:51.389 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:51.387 7f51a365c700 1 -- 172.21.2.102:0/2230746287 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f517c0186c0 msgr2=0x7f517c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:51.390 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:51.387 7f51a365c700 1 --2- 172.21.2.102:0/2230746287 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f517c0186c0 0x7f517c01abc0 crc :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:51.390 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:51.387 7f51a365c700 1 -- 172.21.2.102:0/2230746287 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f519c0ada90 msgr2=0x7f519c0ade90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:51.390 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:51.387 7f51a365c700 1 --2- 172.21.2.102:0/2230746287 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f519c0ada90 0x7f519c0ade90 secure :-1 s=READY pgs=3011 cs=0 l=1 rev1=1 rx=0x7f519000b580 tx=0x7f519000b920).stop 2021-02-10T11:37:51.390 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:51.388 7f51a365c700 1 -- 172.21.2.102:0/2230746287 shutdown_connections 2021-02-10T11:37:51.390 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:51.388 7f51a365c700 1 --2- 172.21.2.102:0/2230746287 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f519c09a740 0x7f519c097bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:51.390 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:51.388 7f51a365c700 1 --2- 172.21.2.102:0/2230746287 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f517c0186c0 0x7f517c01abc0 unknown :-1 s=CLOSED pgs=1322 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:51.391 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:51.388 7f51a365c700 1 --2- 172.21.2.102:0/2230746287 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f519c099d20 0x7f519c09a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:51.391 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:51.388 7f51a365c700 1 --2- 172.21.2.102:0/2230746287 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f519c0ada90 0x7f519c0ade90 unknown :-1 s=CLOSED pgs=3011 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:51.391 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:51.388 7f51a365c700 1 -- 172.21.2.102:0/2230746287 >> 172.21.2.102:0/2230746287 conn(0x7f519c0f0550 msgr2=0x7f519c191730 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:51.391 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:51.388 7f51a365c700 1 -- 172.21.2.102:0/2230746287 shutdown_connections 2021-02-10T11:37:51.391 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:51.388 7f51a365c700 1 -- 172.21.2.102:0/2230746287 wait complete. 2021-02-10T11:37:51.398 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:03:37.322519', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 36, 'pool_name': 'cephfs_metadata', 'create_time': '2021-02-10 11:37:34.298334', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '171', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 37, 'pool_name': 'cephfs_data', 'create_time': '2021-02-10 11:37:35.337127', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '172', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-02-10T11:37:51.399 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:37:51.832 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:51.813 7facc43f8700 1 Processor -- start 2021-02-10T11:37:51.832 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:51.813 7facc43f8700 1 -- start start 2021-02-10T11:37:51.833 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:51.813 7facc43f8700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7facbc0d1870 0x7facbc1b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:51.833 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:51.813 7facc43f8700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7facbc0cff70 0x7facbc109130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:51.833 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:51.813 7facc43f8700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facbc0fe1c0 con 0x7facbc1b0ae0 2021-02-10T11:37:51.833 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:51.813 7facc43f8700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7facbc109ad0 con 0x7facbc0cff70 2021-02-10T11:37:51.833 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:51.813 7facc43f8700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7facbc109750 con 0x7facbc0d1870 2021-02-10T11:37:51.833 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:51.814 7facc2a15700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7facbc0cff70 0x7facbc109130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:51.833 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:51.814 7facc2a15700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7facbc0cff70 0x7facbc109130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:39034/0 (socket says 172.21.2.102:39034) 2021-02-10T11:37:51.833 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:51.814 7facc2a15700 1 -- 172.21.2.102:0/1428007482 learned_addr learned my addr 172.21.2.102:0/1428007482 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:37:51.833 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:51.814 7facc2a15700 1 -- 172.21.2.102:0/1428007482 >> v2:172.21.2.113:3300/0 conn(0x7facbc0d1870 msgr2=0x7facbc1b04c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:37:51.833 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:51.814 7facc2a15700 1 --2- 172.21.2.102:0/1428007482 >> v2:172.21.2.113:3300/0 conn(0x7facbc0d1870 0x7facbc1b04c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:51.834 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:51.814 7facc2a15700 1 -- 172.21.2.102:0/1428007482 >> v1:172.21.2.113:6789/0 conn(0x7facbc1b0ae0 legacy=0x7facbc108940 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:37:51.834 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:51.814 7facc2a15700 1 -- 172.21.2.102:0/1428007482 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facbc088070 con 0x7facbc0cff70 2021-02-10T11:37:51.834 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:51.814 7facc2a15700 1 --2- 172.21.2.102:0/1428007482 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7facbc0cff70 0x7facbc109130 secure :-1 s=READY pgs=3012 cs=0 l=1 rev1=1 rx=0x7facb8006b80 tx=0x7facb8009b90).ready entity=mon.0 client_cookie=7cf2320c0da0a586 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:51.834 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:51.814 7facc1212700 1 -- 172.21.2.102:0/1428007482 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7facb80022e0 con 0x7facbc0cff70 2021-02-10T11:37:51.834 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:51.814 7facc1212700 1 -- 172.21.2.102:0/1428007482 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facb80022e0 con 0x7facbc0cff70 2021-02-10T11:37:51.834 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:51.814 7facc1212700 1 -- 172.21.2.102:0/1428007482 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7facb8002560 con 0x7facbc0cff70 2021-02-10T11:37:51.834 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:51.814 7facc43f8700 1 -- 172.21.2.102:0/1428007482 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7facbc0cff70 msgr2=0x7facbc109130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:51.834 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:51.814 7facc43f8700 1 --2- 172.21.2.102:0/1428007482 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7facbc0cff70 0x7facbc109130 secure :-1 s=READY pgs=3012 cs=0 l=1 rev1=1 rx=0x7facb8006b80 tx=0x7facb8009b90).stop 2021-02-10T11:37:51.834 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:51.815 7facc43f8700 1 -- 172.21.2.102:0/1428007482 shutdown_connections 2021-02-10T11:37:51.834 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:51.815 7facc43f8700 1 --2- 172.21.2.102:0/1428007482 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7facbc0cff70 0x7facbc109130 unknown :-1 s=CLOSED pgs=3012 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:51.835 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:51.815 7facc43f8700 1 --2- 172.21.2.102:0/1428007482 >> v2:172.21.2.113:3300/0 conn(0x7facbc0d1870 0x7facbc1b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:51.835 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:51.815 7facc43f8700 1 -- 172.21.2.102:0/1428007482 >> 172.21.2.102:0/1428007482 conn(0x7facbc0ce420 msgr2=0x7facbc0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:51.835 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:51.815 7facc43f8700 1 -- 172.21.2.102:0/1428007482 shutdown_connections 2021-02-10T11:37:51.835 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:51.815 7facc43f8700 1 -- 172.21.2.102:0/1428007482 wait complete. 2021-02-10T11:37:51.835 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:51.830 7facc43f8700 1 Processor -- start 2021-02-10T11:37:51.835 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:51.830 7facc43f8700 1 -- start start 2021-02-10T11:37:51.837 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:51.830 7facc43f8700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7facbc0c3820 0x7facbc0a7950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:51.837 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:51.830 7facc43f8700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7facbc1a44c0 0x7facbc1a48c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:51.837 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:51.830 7facc43f8700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7facbc0863d0 0x7facbc0867d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:51.837 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:51.830 7facc43f8700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facbc085ed0 con 0x7facbc1a44c0 2021-02-10T11:37:51.837 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:51.830 7facc43f8700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facbc0c0740 con 0x7facbc0863d0 2021-02-10T11:37:51.837 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:51.830 7facc43f8700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facbc086df0 con 0x7facbc0c3820 2021-02-10T11:37:51.837 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:51.831 7facc1a13700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7facbc1a44c0 0x7facbc1a48c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:51.838 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:51.831 7facc1a13700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7facbc1a44c0 0x7facbc1a48c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:39038/0 (socket says 172.21.2.102:39038) 2021-02-10T11:37:51.838 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:51.831 7facc1a13700 1 -- 172.21.2.102:0/572759979 learned_addr learned my addr 172.21.2.102:0/572759979 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:37:51.838 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:51.831 7facc2a15700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7facbc0863d0 0x7facbc0867d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:51.838 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:51.831 7facc2214700 1 --2- 172.21.2.102:0/572759979 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7facbc0c3820 0x7facbc0a7950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:51.838 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:51.831 7facc1a13700 1 -- 172.21.2.102:0/572759979 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7facbc0c3820 msgr2=0x7facbc0a7950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:51.838 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:51.831 7facc1a13700 1 --2- 172.21.2.102:0/572759979 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7facbc0c3820 0x7facbc0a7950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:51.838 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:51.831 7facc1a13700 1 -- 172.21.2.102:0/572759979 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7facbc0863d0 msgr2=0x7facbc0867d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:51.838 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:51.831 7facc1a13700 1 --2- 172.21.2.102:0/572759979 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7facbc0863d0 0x7facbc0867d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:51.838 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:51.831 7facc1a13700 1 -- 172.21.2.102:0/572759979 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facbc087010 con 0x7facbc1a44c0 2021-02-10T11:37:51.839 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:51.831 7facc2a15700 1 --2- 172.21.2.102:0/572759979 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7facbc0863d0 0x7facbc0867d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:37:51.839 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:51.831 7facc1a13700 1 --2- 172.21.2.102:0/572759979 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7facbc1a44c0 0x7facbc1a48c0 secure :-1 s=READY pgs=3013 cs=0 l=1 rev1=1 rx=0x7facb4009940 tx=0x7facb400bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:51.839 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:51.832 7facb2ffd700 1 -- 172.21.2.102:0/572759979 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7facb4000fe0 con 0x7facbc1a44c0 2021-02-10T11:37:51.839 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:51.832 7facb2ffd700 1 -- 172.21.2.102:0/572759979 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facb400c2f0 con 0x7facbc1a44c0 2021-02-10T11:37:51.839 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:51.832 7facb2ffd700 1 -- 172.21.2.102:0/572759979 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7facb400c810 con 0x7facbc1a44c0 2021-02-10T11:37:51.839 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:51.832 7facc43f8700 1 -- 172.21.2.102:0/572759979 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7facbc087010 con 0x7facbc1a44c0 2021-02-10T11:37:51.839 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:51.832 7facc43f8700 1 -- 172.21.2.102:0/572759979 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7facbc083000 con 0x7facbc1a44c0 2021-02-10T11:37:51.839 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:51.832 7facb2ffd700 1 -- 172.21.2.102:0/572759979 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7facb4001230 con 0x7facbc1a44c0 2021-02-10T11:37:51.839 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:51.832 7facb2ffd700 1 --2- 172.21.2.102:0/572759979 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7faca40186c0 0x7faca401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:51.839 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:51.832 7facb2ffd700 1 -- 172.21.2.102:0/572759979 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 5573+0+0 (secure 0 0 0) 0x7facb4032980 con 0x7facbc1a44c0 2021-02-10T11:37:51.840 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:51.832 7facc2214700 1 --2- 172.21.2.102:0/572759979 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7faca40186c0 0x7faca401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:51.840 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:51.833 7facc2214700 1 --2- 172.21.2.102:0/572759979 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7faca40186c0 0x7faca401abc0 crc :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:51.865 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:51.863 7facc43f8700 1 -- 172.21.2.102:0/572759979 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7facbc0fe1c0 con 0x7facbc1a44c0 2021-02-10T11:37:51.868 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:51.866 7facb2ffd700 1 -- 172.21.2.102:0/572759979 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7facb4000f90 con 0x7facbc1a44c0 2021-02-10T11:37:51.995 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:51.993 7facc43f8700 1 -- 172.21.2.102:0/572759979 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7facbc0fe1c0 con 0x7facbc1a44c0 2021-02-10T11:37:51.996 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:51.994 7facb2ffd700 1 -- 172.21.2.102:0/572759979 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v172) v1 ==== 92+0+10 (secure 0 0 0) 0x7facb4001230 con 0x7facbc1a44c0 2021-02-10T11:37:51.996 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:37:51.998 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:51.996 7facc43f8700 1 -- 172.21.2.102:0/572759979 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7faca40186c0 msgr2=0x7faca401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:51.998 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:51.996 7facc43f8700 1 --2- 172.21.2.102:0/572759979 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7faca40186c0 0x7faca401abc0 crc :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:51.998 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:51.996 7facc43f8700 1 -- 172.21.2.102:0/572759979 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7facbc1a44c0 msgr2=0x7facbc1a48c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:51.998 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:51.996 7facc43f8700 1 --2- 172.21.2.102:0/572759979 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7facbc1a44c0 0x7facbc1a48c0 secure :-1 s=READY pgs=3013 cs=0 l=1 rev1=1 rx=0x7facb4009940 tx=0x7facb400bcf0).stop 2021-02-10T11:37:51.998 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:51.996 7facc43f8700 1 -- 172.21.2.102:0/572759979 shutdown_connections 2021-02-10T11:37:51.999 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:51.996 7facc43f8700 1 --2- 172.21.2.102:0/572759979 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7faca40186c0 0x7faca401abc0 unknown :-1 s=CLOSED pgs=1323 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:51.999 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:51.996 7facc43f8700 1 --2- 172.21.2.102:0/572759979 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7facbc0863d0 0x7facbc0867d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:51.999 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:51.996 7facc43f8700 1 --2- 172.21.2.102:0/572759979 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7facbc1a44c0 0x7facbc1a48c0 unknown :-1 s=CLOSED pgs=3013 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:51.999 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:51.996 7facc43f8700 1 --2- 172.21.2.102:0/572759979 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7facbc0c3820 0x7facbc0a7950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:51.999 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:51.996 7facc43f8700 1 -- 172.21.2.102:0/572759979 >> 172.21.2.102:0/572759979 conn(0x7facbc0872c0 msgr2=0x7facbc095700 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:51.999 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:51.996 7facc43f8700 1 -- 172.21.2.102:0/572759979 shutdown_connections 2021-02-10T11:37:51.999 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:51.996 7facc43f8700 1 -- 172.21.2.102:0/572759979 wait complete. 2021-02-10T11:37:52.006 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-02-10T11:37:52.439 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:52.419 7f889450b700 1 Processor -- start 2021-02-10T11:37:52.439 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:52.419 7f889450b700 1 -- start start 2021-02-10T11:37:52.439 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:52.420 7f889450b700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f888c0d1c60 0x7f888c0cef50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:52.440 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:52.420 7f889450b700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f888c0cf570 0x7f888c0cf970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:52.440 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:52.420 7f889450b700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f888c0fe1b0 con 0x7f888c0d1860 2021-02-10T11:37:52.440 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:52.420 7f889450b700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f888c1b0ca0 con 0x7f888c0cf570 2021-02-10T11:37:52.440 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:52.420 7f889450b700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f888c109a50 con 0x7f888c0d1c60 2021-02-10T11:37:52.440 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:52.420 7f8892327700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f888c0d1860 0x7f888c1b04b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:59500/0 (socket says 172.21.2.102:59500) 2021-02-10T11:37:52.440 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:52.420 7f8892327700 1 -- 172.21.2.102:0/1325976868 learned_addr learned my addr 172.21.2.102:0/1325976868 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:37:52.440 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:52.420 7f8891325700 1 -- 172.21.2.102:0/1325976868 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2065100955 0 0) 0x7f88880010b0 con 0x7f888c0d1860 2021-02-10T11:37:52.440 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:52.421 7f8891325700 1 -- 172.21.2.102:0/1325976868 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8878002fe0 con 0x7f888c0d1860 2021-02-10T11:37:52.440 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:52.421 7f8891325700 1 -- 172.21.2.102:0/1325976868 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2501063303 0 0) 0x7f88880010b0 con 0x7f888c0d1860 2021-02-10T11:37:52.440 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:52.421 7f8891325700 1 -- 172.21.2.102:0/1325976868 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f888c0d1c60 msgr2=0x7f888c0cef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:37:52.440 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:52.421 7f8891325700 1 --2- 172.21.2.102:0/1325976868 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f888c0d1c60 0x7f888c0cef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:52.440 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:52.421 7f8891325700 1 -- 172.21.2.102:0/1325976868 >> v2:172.21.2.113:3300/0 conn(0x7f888c0cf570 msgr2=0x7f888c0cf970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:37:52.441 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:52.421 7f8891325700 1 --2- 172.21.2.102:0/1325976868 >> v2:172.21.2.113:3300/0 conn(0x7f888c0cf570 0x7f888c0cf970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:52.441 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:52.421 7f8891325700 1 -- 172.21.2.102:0/1325976868 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f888c088070 con 0x7f888c0d1860 2021-02-10T11:37:52.441 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:52.421 7f8891325700 1 -- 172.21.2.102:0/1325976868 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7f88880015b0 con 0x7f888c0d1860 2021-02-10T11:37:52.441 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:52.421 7f8891325700 1 -- 172.21.2.102:0/1325976868 >> v1:172.21.2.102:6789/0 conn(0x7f888c0d1860 legacy=0x7f888c1b04b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:52.441 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:52.421 7f8891325700 1 --2- 172.21.2.102:0/1325976868 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8878003b10 0x7f8878006010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:52.441 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:52.421 7f8891325700 1 --2- 172.21.2.102:0/1325976868 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f88780067a0 0x7f8878008cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:52.441 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:52.421 7f8891325700 1 --2- 172.21.2.102:0/1325976868 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f88780095c0 0x7f887800ba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:52.441 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:52.421 7f8891325700 1 -- 172.21.2.102:0/1325976868 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f887800c2c0 con 0x7f8878003b10 2021-02-10T11:37:52.441 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:52.421 7f8891325700 1 -- 172.21.2.102:0/1325976868 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f887800c4e0 con 0x7f88780067a0 2021-02-10T11:37:52.441 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:52.421 7f8891325700 1 -- 172.21.2.102:0/1325976868 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f887800c700 con 0x7f88780095c0 2021-02-10T11:37:52.441 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:52.421 7f8892327700 1 --2- 172.21.2.102:0/1325976868 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8878003b10 0x7f8878006010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:52.441 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:52.421 7f8892b28700 1 --2- 172.21.2.102:0/1325976868 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f88780095c0 0x7f887800ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:52.441 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:52.421 7f8891b26700 1 --2- 172.21.2.102:0/1325976868 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f88780067a0 0x7f8878008cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:52.441 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:52.421 7f8892327700 1 -- 172.21.2.102:0/1325976868 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f88780095c0 msgr2=0x7f887800ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:37:52.442 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:52.421 7f8892327700 1 --2- 172.21.2.102:0/1325976868 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f88780095c0 0x7f887800ba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:52.442 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:52.421 7f8892327700 1 -- 172.21.2.102:0/1325976868 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f88780067a0 msgr2=0x7f8878008cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:37:52.442 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:52.421 7f8892327700 1 --2- 172.21.2.102:0/1325976868 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f88780067a0 0x7f8878008cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:52.442 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:52.422 7f8892327700 1 -- 172.21.2.102:0/1325976868 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f887800c9a0 con 0x7f8878003b10 2021-02-10T11:37:52.442 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:52.422 7f8892327700 1 --2- 172.21.2.102:0/1325976868 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8878003b10 0x7f8878006010 secure :-1 s=READY pgs=3015 cs=0 l=1 rev1=1 rx=0x7f88880017d0 tx=0x7f8888007d30).ready entity=mon.0 client_cookie=9a56fffd222721a3 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:52.442 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:52.422 7f8891325700 1 -- 172.21.2.102:0/1325976868 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f888804e320 con 0x7f8878003b10 2021-02-10T11:37:52.442 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:52.422 7f8891325700 1 -- 172.21.2.102:0/1325976868 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f888804e660 con 0x7f8878003b10 2021-02-10T11:37:52.442 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:52.422 7f8891325700 1 -- 172.21.2.102:0/1325976868 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f888804eb40 con 0x7f8878003b10 2021-02-10T11:37:52.442 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:52.422 7f889450b700 1 -- 172.21.2.102:0/1325976868 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8878003b10 msgr2=0x7f8878006010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:52.442 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:52.422 7f889450b700 1 --2- 172.21.2.102:0/1325976868 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8878003b10 0x7f8878006010 secure :-1 s=READY pgs=3015 cs=0 l=1 rev1=1 rx=0x7f88880017d0 tx=0x7f8888007d30).stop 2021-02-10T11:37:52.442 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:52.422 7f8892b28700 1 -- 172.21.2.102:0/1325976868 reap_dead start 2021-02-10T11:37:52.442 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:52.422 7f889450b700 1 -- 172.21.2.102:0/1325976868 shutdown_connections 2021-02-10T11:37:52.443 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:52.422 7f889450b700 1 -- 172.21.2.102:0/1325976868 >> 172.21.2.102:0/1325976868 conn(0x7f888c0ce410 msgr2=0x7f888c0d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:52.443 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:52.423 7f889450b700 1 -- 172.21.2.102:0/1325976868 shutdown_connections 2021-02-10T11:37:52.443 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:52.423 7f889450b700 1 -- 172.21.2.102:0/1325976868 wait complete. 2021-02-10T11:37:52.443 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:52.437 7f889450b700 1 Processor -- start 2021-02-10T11:37:52.443 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:52.437 7f889450b700 1 -- start start 2021-02-10T11:37:52.445 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:52.437 7f889450b700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f888c0adf50 0x7f888c0ae350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:52.445 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:52.437 7f889450b700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f888c0ae970 0x7f888c0aed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:52.445 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:52.437 7f889450b700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f888c0982c0 0x7f888c0986c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:52.445 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:52.437 7f889450b700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f888c09f450 con 0x7f888c0982c0 2021-02-10T11:37:52.445 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:52.437 7f889450b700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f888c098ce0 con 0x7f888c0ae970 2021-02-10T11:37:52.445 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:52.437 7f889450b700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f888c098ed0 con 0x7f888c0adf50 2021-02-10T11:37:52.445 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:52.438 7f8892b28700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f888c0982c0 0x7f888c0986c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:52.445 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:52.438 7f8892b28700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f888c0982c0 0x7f888c0986c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:39058/0 (socket says 172.21.2.102:39058) 2021-02-10T11:37:52.445 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:52.438 7f8892b28700 1 -- 172.21.2.102:0/2488240106 learned_addr learned my addr 172.21.2.102:0/2488240106 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:37:52.445 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:52.438 7f8891b26700 1 --2- 172.21.2.102:0/2488240106 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f888c0ae970 0x7f888c0aed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:52.446 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:52.438 7f8892327700 1 --2- 172.21.2.102:0/2488240106 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f888c0adf50 0x7f888c0ae350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:52.446 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:52.438 7f8892b28700 1 -- 172.21.2.102:0/2488240106 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f888c0adf50 msgr2=0x7f888c0ae350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:52.446 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:52.438 7f8892b28700 1 --2- 172.21.2.102:0/2488240106 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f888c0adf50 0x7f888c0ae350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:52.446 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:52.438 7f8892b28700 1 -- 172.21.2.102:0/2488240106 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f888c0ae970 msgr2=0x7f888c0aed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:52.446 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:52.438 7f8892b28700 1 --2- 172.21.2.102:0/2488240106 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f888c0ae970 0x7f888c0aed70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:52.446 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:52.438 7f8892b28700 1 -- 172.21.2.102:0/2488240106 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f888c0990f0 con 0x7f888c0982c0 2021-02-10T11:37:52.446 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:52.438 7f8892327700 1 --2- 172.21.2.102:0/2488240106 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f888c0adf50 0x7f888c0ae350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:37:52.446 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:52.438 7f8892b28700 1 --2- 172.21.2.102:0/2488240106 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f888c0982c0 0x7f888c0986c0 secure :-1 s=READY pgs=3016 cs=0 l=1 rev1=1 rx=0x7f88840090f0 tx=0x7f888400bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:52.446 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:52.438 7f887effd700 1 -- 172.21.2.102:0/2488240106 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8884009160 con 0x7f888c0982c0 2021-02-10T11:37:52.446 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:52.438 7f887effd700 1 -- 172.21.2.102:0/2488240106 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8884003020 con 0x7f888c0982c0 2021-02-10T11:37:52.446 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:52.438 7f889450b700 1 -- 172.21.2.102:0/2488240106 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f888c0990f0 con 0x7f888c0982c0 2021-02-10T11:37:52.446 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:52.438 7f889450b700 1 -- 172.21.2.102:0/2488240106 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f888c099680 con 0x7f888c0982c0 2021-02-10T11:37:52.447 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:52.438 7f887effd700 1 -- 172.21.2.102:0/2488240106 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f888400c4e0 con 0x7f888c0982c0 2021-02-10T11:37:52.447 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:52.439 7f887effd700 1 -- 172.21.2.102:0/2488240106 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f8884009fd0 con 0x7f888c0982c0 2021-02-10T11:37:52.447 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:52.439 7f887effd700 1 --2- 172.21.2.102:0/2488240106 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f88740186d0 0x7f887401abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:52.447 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:52.439 7f887effd700 1 -- 172.21.2.102:0/2488240106 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f88840329f0 con 0x7f888c0982c0 2021-02-10T11:37:52.447 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:52.439 7f8892327700 1 --2- 172.21.2.102:0/2488240106 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f88740186d0 0x7f887401abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:52.447 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:52.439 7f8892327700 1 --2- 172.21.2.102:0/2488240106 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f88740186d0 0x7f887401abd0 crc :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:52.472 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:52.470 7f889450b700 1 -- 172.21.2.102:0/2488240106 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f888c082a20 con 0x7f888c0982c0 2021-02-10T11:37:52.474 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:52.472 7f887effd700 1 -- 172.21.2.102:0/2488240106 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f8884009160 con 0x7f888c0982c0 2021-02-10T11:37:52.603 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:52.601 7f889450b700 1 -- 172.21.2.102:0/2488240106 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7f888c082a20 con 0x7f888c0982c0 2021-02-10T11:37:52.603 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:52.601 7f887effd700 1 -- 172.21.2.102:0/2488240106 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v172) v1 ==== 104+0+10 (secure 0 0 0) 0x7f8884003000 con 0x7f888c0982c0 2021-02-10T11:37:52.604 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:37:52.606 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:52.603 7f889450b700 1 -- 172.21.2.102:0/2488240106 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f88740186d0 msgr2=0x7f887401abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:52.606 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:52.603 7f889450b700 1 --2- 172.21.2.102:0/2488240106 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f88740186d0 0x7f887401abd0 crc :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:52.606 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:52.603 7f889450b700 1 -- 172.21.2.102:0/2488240106 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f888c0982c0 msgr2=0x7f888c0986c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:52.606 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:52.603 7f889450b700 1 --2- 172.21.2.102:0/2488240106 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f888c0982c0 0x7f888c0986c0 secure :-1 s=READY pgs=3016 cs=0 l=1 rev1=1 rx=0x7f88840090f0 tx=0x7f888400bd10).stop 2021-02-10T11:37:52.606 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:52.603 7f889450b700 1 -- 172.21.2.102:0/2488240106 shutdown_connections 2021-02-10T11:37:52.606 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:52.603 7f889450b700 1 --2- 172.21.2.102:0/2488240106 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f888c0982c0 0x7f888c0986c0 unknown :-1 s=CLOSED pgs=3016 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:52.606 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:52.604 7f889450b700 1 --2- 172.21.2.102:0/2488240106 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f88740186d0 0x7f887401abd0 unknown :-1 s=CLOSED pgs=1324 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:52.606 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:52.604 7f889450b700 1 --2- 172.21.2.102:0/2488240106 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f888c0ae970 0x7f888c0aed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:52.606 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:52.604 7f889450b700 1 --2- 172.21.2.102:0/2488240106 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f888c0adf50 0x7f888c0ae350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:52.606 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:52.604 7f889450b700 1 -- 172.21.2.102:0/2488240106 >> 172.21.2.102:0/2488240106 conn(0x7f888c0bee90 msgr2=0x7f888c0bb880 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:52.607 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:52.604 7f889450b700 1 -- 172.21.2.102:0/2488240106 shutdown_connections 2021-02-10T11:37:52.607 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:52.604 7f889450b700 1 -- 172.21.2.102:0/2488240106 wait complete. 2021-02-10T11:37:52.614 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-02-10T11:37:53.049 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:53.029 7f3ac1a06700 1 Processor -- start 2021-02-10T11:37:53.049 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:53.029 7f3ac1a06700 1 -- start start 2021-02-10T11:37:53.049 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:53.029 7f3ac1a06700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3abc10afe0 0x7f3abc0cf740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:53.049 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:53.029 7f3ac1a06700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3abc0fe1b0 con 0x7f3abc0d1c60 2021-02-10T11:37:53.049 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:53.029 7f3ac1a06700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3abc1b0ca0 con 0x7f3abc0d1860 2021-02-10T11:37:53.049 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:53.029 7f3ac1a06700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3abc109a50 con 0x7f3abc10afe0 2021-02-10T11:37:53.049 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:53.029 7f3abaffd700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f3abc0d1c60 0x7f3abc0cef50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:59520/0 (socket says 172.21.2.102:59520) 2021-02-10T11:37:53.049 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:53.029 7f3abaffd700 1 -- 172.21.2.102:0/2492709864 learned_addr learned my addr 172.21.2.102:0/2492709864 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:37:53.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:53.030 7f3aba7fc700 1 -- 172.21.2.102:0/2492709864 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3375446748 0 0) 0x7f3aac0009b0 con 0x7f3abc0d1c60 2021-02-10T11:37:53.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:53.030 7f3aba7fc700 1 -- 172.21.2.102:0/2492709864 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3aa8002fe0 con 0x7f3abc0d1c60 2021-02-10T11:37:53.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:53.030 7f3abbfff700 1 --2- 172.21.2.102:0/2492709864 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3abc10afe0 0x7f3abc0cf740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:53.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:53.030 7f3aba7fc700 1 -- 172.21.2.102:0/2492709864 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2994655054 0 0) 0x7f3aac001010 con 0x7f3abc0d1c60 2021-02-10T11:37:53.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:53.030 7f3aba7fc700 1 -- 172.21.2.102:0/2492709864 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3abc10afe0 msgr2=0x7f3abc0cf740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:37:53.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:53.030 7f3aba7fc700 1 --2- 172.21.2.102:0/2492709864 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3abc10afe0 0x7f3abc0cf740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:53.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:53.030 7f3aba7fc700 1 -- 172.21.2.102:0/2492709864 >> v1:172.21.2.113:6789/0 conn(0x7f3abc0d1860 legacy=0x7f3abc1b04b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:53.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:53.030 7f3aba7fc700 1 -- 172.21.2.102:0/2492709864 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3abc10b3e0 con 0x7f3abc0d1c60 2021-02-10T11:37:53.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:53.030 7f3aba7fc700 1 -- 172.21.2.102:0/2492709864 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7f3aac001590 con 0x7f3abc0d1c60 2021-02-10T11:37:53.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:53.030 7f3aba7fc700 1 -- 172.21.2.102:0/2492709864 >> v1:172.21.2.102:6789/0 conn(0x7f3abc0d1c60 legacy=0x7f3abc0cef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:53.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:53.030 7f3aba7fc700 1 --2- 172.21.2.102:0/2492709864 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3aa8003b90 0x7f3aa8005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:53.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:53.030 7f3aba7fc700 1 --2- 172.21.2.102:0/2492709864 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3aa8006770 0x7f3aa8008c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:53.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:53.030 7f3aba7fc700 1 --2- 172.21.2.102:0/2492709864 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3aa8009510 0x7f3aa800ba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:53.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:53.030 7f3aba7fc700 1 -- 172.21.2.102:0/2492709864 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3aa800c2e0 con 0x7f3aa8009510 2021-02-10T11:37:53.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:53.030 7f3aba7fc700 1 -- 172.21.2.102:0/2492709864 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3aa800c500 con 0x7f3aa8006770 2021-02-10T11:37:53.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:53.030 7f3aba7fc700 1 -- 172.21.2.102:0/2492709864 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3aa800c720 con 0x7f3aa8003b90 2021-02-10T11:37:53.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:53.030 7f3abbfff700 1 --2- 172.21.2.102:0/2492709864 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3aa8009510 0x7f3aa800ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:53.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:53.031 7f3abaffd700 1 --2- 172.21.2.102:0/2492709864 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3aa8006770 0x7f3aa8008c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:53.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:53.031 7f3abb7fe700 1 --2- 172.21.2.102:0/2492709864 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3aa8003b90 0x7f3aa8005fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:53.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:53.031 7f3abbfff700 1 -- 172.21.2.102:0/2492709864 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3aa8003b90 msgr2=0x7f3aa8005fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:37:53.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:53.031 7f3abbfff700 1 --2- 172.21.2.102:0/2492709864 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3aa8003b90 0x7f3aa8005fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:53.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:53.031 7f3abbfff700 1 -- 172.21.2.102:0/2492709864 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3aa8006770 msgr2=0x7f3aa8008c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:37:53.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:53.031 7f3abbfff700 1 --2- 172.21.2.102:0/2492709864 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3aa8006770 0x7f3aa8008c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:53.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:53.031 7f3abbfff700 1 -- 172.21.2.102:0/2492709864 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3aa8050990 con 0x7f3aa8009510 2021-02-10T11:37:53.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:53.031 7f3abbfff700 1 --2- 172.21.2.102:0/2492709864 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3aa8009510 0x7f3aa800ba30 secure :-1 s=READY pgs=3018 cs=0 l=1 rev1=1 rx=0x7f3ab4006fc0 tx=0x7f3ab4009df0).ready entity=mon.0 client_cookie=73765250ec54fd18 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:53.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:53.031 7f3aba7fc700 1 -- 172.21.2.102:0/2492709864 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3ab400b300 con 0x7f3aa8009510 2021-02-10T11:37:53.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:53.031 7f3aba7fc700 1 -- 172.21.2.102:0/2492709864 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ab4006a50 con 0x7f3aa8009510 2021-02-10T11:37:53.052 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:53.031 7f3aba7fc700 1 -- 172.21.2.102:0/2492709864 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3ab400b730 con 0x7f3aa8009510 2021-02-10T11:37:53.052 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:53.031 7f3ac1a06700 1 -- 172.21.2.102:0/2492709864 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3aa8009510 msgr2=0x7f3aa800ba30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:53.052 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:53.031 7f3ac1a06700 1 --2- 172.21.2.102:0/2492709864 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3aa8009510 0x7f3aa800ba30 secure :-1 s=READY pgs=3018 cs=0 l=1 rev1=1 rx=0x7f3ab4006fc0 tx=0x7f3ab4009df0).stop 2021-02-10T11:37:53.052 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:53.031 7f3abbfff700 1 -- 172.21.2.102:0/2492709864 reap_dead start 2021-02-10T11:37:53.052 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:53.032 7f3ac1a06700 1 -- 172.21.2.102:0/2492709864 shutdown_connections 2021-02-10T11:37:53.052 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:53.032 7f3ac1a06700 1 -- 172.21.2.102:0/2492709864 >> 172.21.2.102:0/2492709864 conn(0x7f3abc0ce410 msgr2=0x7f3abc0d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:53.052 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:53.032 7f3ac1a06700 1 -- 172.21.2.102:0/2492709864 shutdown_connections 2021-02-10T11:37:53.052 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:53.032 7f3ac1a06700 1 -- 172.21.2.102:0/2492709864 wait complete. 2021-02-10T11:37:53.052 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:53.047 7f3ac1a06700 1 Processor -- start 2021-02-10T11:37:53.052 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:53.047 7f3ac1a06700 1 -- start start 2021-02-10T11:37:53.053 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:53.047 7f3ac1a06700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3abc0b8100 0x7f3abc083440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:53.053 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:53.047 7f3ac1a06700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3abc0b8500 0x7f3abc083a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:53.053 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:53.047 7f3ac1a06700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3abc0846f0 0x7f3abc084af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:53.053 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:53.047 7f3ac1a06700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3abc1a1720 con 0x7f3abc0846f0 2021-02-10T11:37:53.054 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:53.047 7f3ac1a06700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3abc189e70 con 0x7f3abc0b8100 2021-02-10T11:37:53.054 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:53.047 7f3ac1a06700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3abc084170 con 0x7f3abc0b8500 2021-02-10T11:37:53.054 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:53.047 7f3abbfff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3abc0846f0 0x7f3abc084af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:53.054 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:53.047 7f3abbfff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3abc0846f0 0x7f3abc084af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:39078/0 (socket says 172.21.2.102:39078) 2021-02-10T11:37:53.054 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:53.047 7f3abbfff700 1 -- 172.21.2.102:0/3564052965 learned_addr learned my addr 172.21.2.102:0/3564052965 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:37:53.054 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:53.048 7f3abbfff700 1 -- 172.21.2.102:0/3564052965 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3abc0b8500 msgr2=0x7f3abc083a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:53.054 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:53.048 7f3abbfff700 1 --2- 172.21.2.102:0/3564052965 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3abc0b8500 0x7f3abc083a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:53.054 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:53.048 7f3abbfff700 1 -- 172.21.2.102:0/3564052965 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3abc0b8100 msgr2=0x7f3abc083440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:53.054 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:53.048 7f3abbfff700 1 --2- 172.21.2.102:0/3564052965 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3abc0b8100 0x7f3abc083440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:53.054 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:53.048 7f3abbfff700 1 -- 172.21.2.102:0/3564052965 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3abc085110 con 0x7f3abc0846f0 2021-02-10T11:37:53.054 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:53.048 7f3abbfff700 1 --2- 172.21.2.102:0/3564052965 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3abc0846f0 0x7f3abc084af0 secure :-1 s=READY pgs=3019 cs=0 l=1 rev1=1 rx=0x7f3ab4009fc0 tx=0x7f3ab400bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:53.054 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:53.048 7f3ab3fff700 1 -- 172.21.2.102:0/3564052965 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3ab40090d0 con 0x7f3abc0846f0 2021-02-10T11:37:53.054 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:53.048 7f3ab3fff700 1 -- 172.21.2.102:0/3564052965 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ab4001070 con 0x7f3abc0846f0 2021-02-10T11:37:53.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:53.048 7f3ab3fff700 1 -- 172.21.2.102:0/3564052965 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3ab4001590 con 0x7f3abc0846f0 2021-02-10T11:37:53.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:53.048 7f3ac1a06700 1 -- 172.21.2.102:0/3564052965 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3abc085110 con 0x7f3abc0846f0 2021-02-10T11:37:53.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:53.048 7f3ac1a06700 1 -- 172.21.2.102:0/3564052965 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3abc085660 con 0x7f3abc0846f0 2021-02-10T11:37:53.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:53.049 7f3ab3fff700 1 -- 172.21.2.102:0/3564052965 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f3ab4001250 con 0x7f3abc0846f0 2021-02-10T11:37:53.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:53.049 7f3ab3fff700 1 --2- 172.21.2.102:0/3564052965 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3aa00186c0 0x7f3aa001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:53.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:53.049 7f3ab3fff700 1 -- 172.21.2.102:0/3564052965 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f3ab40328e0 con 0x7f3abc0846f0 2021-02-10T11:37:53.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:53.049 7f3abb7fe700 1 --2- 172.21.2.102:0/3564052965 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3aa00186c0 0x7f3aa001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:53.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:53.049 7f3abb7fe700 1 --2- 172.21.2.102:0/3564052965 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3aa00186c0 0x7f3aa001abc0 crc :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:53.081 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:53.079 7f3ac1a06700 1 -- 172.21.2.102:0/3564052965 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3abc1a1720 con 0x7f3abc0846f0 2021-02-10T11:37:53.083 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:53.081 7f3ab3fff700 1 -- 172.21.2.102:0/3564052965 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f3ab4006720 con 0x7f3abc0846f0 2021-02-10T11:37:53.210 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:53.208 7f3ac1a06700 1 -- 172.21.2.102:0/3564052965 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7f3abc189e70 con 0x7f3abc0846f0 2021-02-10T11:37:53.211 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:53.209 7f3ab3fff700 1 -- 172.21.2.102:0/3564052965 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v172) v1 ==== 100+0+10 (secure 0 0 0) 0x7f3ab401f680 con 0x7f3abc0846f0 2021-02-10T11:37:53.211 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:37:53.213 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:53.210 7f3ac1a06700 1 -- 172.21.2.102:0/3564052965 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3aa00186c0 msgr2=0x7f3aa001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:53.213 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:53.210 7f3ac1a06700 1 --2- 172.21.2.102:0/3564052965 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3aa00186c0 0x7f3aa001abc0 crc :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:53.213 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:53.210 7f3ac1a06700 1 -- 172.21.2.102:0/3564052965 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3abc0846f0 msgr2=0x7f3abc084af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:53.213 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:53.210 7f3ac1a06700 1 --2- 172.21.2.102:0/3564052965 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3abc0846f0 0x7f3abc084af0 secure :-1 s=READY pgs=3019 cs=0 l=1 rev1=1 rx=0x7f3ab4009fc0 tx=0x7f3ab400bcf0).stop 2021-02-10T11:37:53.213 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:53.211 7f3ac1a06700 1 -- 172.21.2.102:0/3564052965 shutdown_connections 2021-02-10T11:37:53.213 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:53.211 7f3ac1a06700 1 --2- 172.21.2.102:0/3564052965 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3abc0846f0 0x7f3abc084af0 unknown :-1 s=CLOSED pgs=3019 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:53.213 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:53.211 7f3ac1a06700 1 --2- 172.21.2.102:0/3564052965 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3abc0b8500 0x7f3abc083a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:53.213 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:53.211 7f3ac1a06700 1 --2- 172.21.2.102:0/3564052965 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3aa00186c0 0x7f3aa001abc0 unknown :-1 s=CLOSED pgs=1325 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:53.214 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:53.211 7f3ac1a06700 1 --2- 172.21.2.102:0/3564052965 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3abc0b8100 0x7f3abc083440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:53.214 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:53.211 7f3ac1a06700 1 -- 172.21.2.102:0/3564052965 >> 172.21.2.102:0/3564052965 conn(0x7f3abc0bee90 msgr2=0x7f3abc09f200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:53.214 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:53.211 7f3ac1a06700 1 -- 172.21.2.102:0/3564052965 shutdown_connections 2021-02-10T11:37:53.214 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:53.211 7f3ac1a06700 1 -- 172.21.2.102:0/3564052965 wait complete. 2021-02-10T11:37:53.222 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-02-10T11:37:53.222 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:37:53.651 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:53.632 7f3ba08e4700 1 Processor -- start 2021-02-10T11:37:53.651 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:53.632 7f3ba08e4700 1 -- start start 2021-02-10T11:37:53.651 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:53.632 7f3ba08e4700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f3b9c1b0470 0x7f3b9c1b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:53.651 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:53.632 7f3ba08e4700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f3b9c0f20e0 0x7f3b9c0f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:53.651 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:53.632 7f3ba08e4700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b9c0f0b50 con 0x7f3b9c0ef610 2021-02-10T11:37:53.651 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:53.632 7f3ba08e4700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b9c0f0820 con 0x7f3b9c1b0470 2021-02-10T11:37:53.651 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:53.632 7f3ba08e4700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b9c0f24e0 con 0x7f3b9c0f20e0 2021-02-10T11:37:53.651 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:53.633 7f3b99e1b700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f3b9c0ef610 0x7f3b9c0efa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:59540/0 (socket says 172.21.2.102:59540) 2021-02-10T11:37:53.652 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:53.633 7f3b99e1b700 1 -- 172.21.2.102:0/4217020519 learned_addr learned my addr 172.21.2.102:0/4217020519 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:37:53.652 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:53.633 7f3b9a61c700 1 --2- 172.21.2.102:0/4217020519 >> v2:172.21.2.102:3300/0 conn(0x7f3b9c1b0470 0x7f3b9c1b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:53.652 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:53.633 7f3b9ae1d700 1 --2- 172.21.2.102:0/4217020519 >> v2:172.21.2.113:3300/0 conn(0x7f3b9c0f20e0 0x7f3b9c0f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:53.652 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:53.633 7f3b9a61c700 1 -- 172.21.2.102:0/4217020519 >> v2:172.21.2.113:3300/0 conn(0x7f3b9c0f20e0 msgr2=0x7f3b9c0f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:37:53.652 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:53.633 7f3b9a61c700 1 --2- 172.21.2.102:0/4217020519 >> v2:172.21.2.113:3300/0 conn(0x7f3b9c0f20e0 0x7f3b9c0f0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:53.652 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:53.633 7f3b9a61c700 1 -- 172.21.2.102:0/4217020519 >> v1:172.21.2.102:6789/0 conn(0x7f3b9c0ef610 legacy=0x7f3b9c0efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:53.652 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:53.633 7f3b9961a700 1 -- 172.21.2.102:0/4217020519 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 871040646 0 0) 0x7f3b840010b0 con 0x7f3b9c0ef610 2021-02-10T11:37:53.652 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:53.633 7f3b9a61c700 1 -- 172.21.2.102:0/4217020519 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b9c0e52e0 con 0x7f3b9c1b0470 2021-02-10T11:37:53.652 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:53.633 7f3b9a61c700 1 --2- 172.21.2.102:0/4217020519 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3b9c1b0470 0x7f3b9c1b0870 secure :-1 s=READY pgs=3021 cs=0 l=1 rev1=1 rx=0x7f3b8c006650 tx=0x7f3b8c009a00).ready entity=mon.0 client_cookie=f55092bc24646c1c server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:53.652 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:53.633 7f3b9961a700 1 -- 172.21.2.102:0/4217020519 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3b8c0009b0 con 0x7f3b9c1b0470 2021-02-10T11:37:53.652 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:53.633 7f3b9961a700 1 -- 172.21.2.102:0/4217020519 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b8c00c0c0 con 0x7f3b9c1b0470 2021-02-10T11:37:53.652 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:53.633 7f3b9961a700 1 -- 172.21.2.102:0/4217020519 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3b8c00c5a0 con 0x7f3b9c1b0470 2021-02-10T11:37:53.652 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:53.633 7f3ba08e4700 1 -- 172.21.2.102:0/4217020519 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3b9c1b0470 msgr2=0x7f3b9c1b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:53.653 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:53.633 7f3ba08e4700 1 --2- 172.21.2.102:0/4217020519 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3b9c1b0470 0x7f3b9c1b0870 secure :-1 s=READY pgs=3021 cs=0 l=1 rev1=1 rx=0x7f3b8c006650 tx=0x7f3b8c009a00).stop 2021-02-10T11:37:53.653 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:53.634 7f3ba08e4700 1 -- 172.21.2.102:0/4217020519 shutdown_connections 2021-02-10T11:37:53.653 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:53.634 7f3ba08e4700 1 --2- 172.21.2.102:0/4217020519 >> v2:172.21.2.113:3300/0 conn(0x7f3b9c0f20e0 0x7f3b9c0f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:53.653 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:53.634 7f3ba08e4700 1 --2- 172.21.2.102:0/4217020519 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3b9c1b0470 0x7f3b9c1b0870 unknown :-1 s=CLOSED pgs=3021 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:53.653 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:53.634 7f3ba08e4700 1 -- 172.21.2.102:0/4217020519 >> 172.21.2.102:0/4217020519 conn(0x7f3b9c0ce3e0 msgr2=0x7f3b9c0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:53.653 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:53.634 7f3ba08e4700 1 -- 172.21.2.102:0/4217020519 shutdown_connections 2021-02-10T11:37:53.653 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:53.634 7f3ba08e4700 1 -- 172.21.2.102:0/4217020519 wait complete. 2021-02-10T11:37:53.653 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:53.649 7f3ba08e4700 1 Processor -- start 2021-02-10T11:37:53.653 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:53.649 7f3ba08e4700 1 -- start start 2021-02-10T11:37:53.655 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:53.649 7f3ba08e4700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3b9c1af8a0 0x7f3b9c198e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:53.655 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:53.649 7f3ba08e4700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3b9c1af050 0x7f3b9c0aa3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:53.655 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:53.649 7f3ba08e4700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3b9c197d00 0x7f3b9c198100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:53.655 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:53.649 7f3ba08e4700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b9c1988e0 con 0x7f3b9c1af8a0 2021-02-10T11:37:53.655 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:53.649 7f3ba08e4700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b9c1af450 con 0x7f3b9c197d00 2021-02-10T11:37:53.655 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:53.649 7f3ba08e4700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b9c0aaa10 con 0x7f3b9c1af050 2021-02-10T11:37:53.655 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:53.649 7f3b9a61c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3b9c1af8a0 0x7f3b9c198e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:53.655 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:53.650 7f3b9a61c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3b9c1af8a0 0x7f3b9c198e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:39088/0 (socket says 172.21.2.102:39088) 2021-02-10T11:37:53.656 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:53.650 7f3b9a61c700 1 -- 172.21.2.102:0/1921562735 learned_addr learned my addr 172.21.2.102:0/1921562735 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:37:53.656 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:53.650 7f3b99e1b700 1 --2- 172.21.2.102:0/1921562735 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3b9c1af050 0x7f3b9c0aa3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:53.656 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:53.650 7f3b9ae1d700 1 --2- 172.21.2.102:0/1921562735 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3b9c197d00 0x7f3b9c198100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:53.656 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:53.650 7f3b9a61c700 1 -- 172.21.2.102:0/1921562735 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3b9c1af050 msgr2=0x7f3b9c0aa3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:53.656 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:53.650 7f3b9a61c700 1 --2- 172.21.2.102:0/1921562735 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3b9c1af050 0x7f3b9c0aa3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:53.656 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:53.650 7f3b9a61c700 1 -- 172.21.2.102:0/1921562735 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3b9c197d00 msgr2=0x7f3b9c198100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:53.656 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:53.650 7f3b9a61c700 1 --2- 172.21.2.102:0/1921562735 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3b9c197d00 0x7f3b9c198100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:53.656 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:53.650 7f3b9a61c700 1 -- 172.21.2.102:0/1921562735 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b9c19fec0 con 0x7f3b9c1af8a0 2021-02-10T11:37:53.656 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:53.650 7f3b9a61c700 1 --2- 172.21.2.102:0/1921562735 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3b9c1af8a0 0x7f3b9c198e60 secure :-1 s=READY pgs=3022 cs=0 l=1 rev1=1 rx=0x7f3b8c00b640 tx=0x7f3b8c00b9e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:53.656 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:53.650 7f3b92ffd700 1 -- 172.21.2.102:0/1921562735 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3b8c00bdd0 con 0x7f3b9c1af8a0 2021-02-10T11:37:53.656 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:53.650 7f3b92ffd700 1 -- 172.21.2.102:0/1921562735 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b8c002030 con 0x7f3b9c1af8a0 2021-02-10T11:37:53.656 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:53.650 7f3ba08e4700 1 -- 172.21.2.102:0/1921562735 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b9c19fec0 con 0x7f3b9c1af8a0 2021-02-10T11:37:53.656 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:53.650 7f3b92ffd700 1 -- 172.21.2.102:0/1921562735 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3b8c0024d0 con 0x7f3b9c1af8a0 2021-02-10T11:37:53.656 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:53.650 7f3ba08e4700 1 -- 172.21.2.102:0/1921562735 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b9c19fec0 con 0x7f3b9c1af8a0 2021-02-10T11:37:53.657 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:53.651 7f3b92ffd700 1 -- 172.21.2.102:0/1921562735 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f3b8c006a10 con 0x7f3b9c1af8a0 2021-02-10T11:37:53.657 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:53.651 7f3b92ffd700 1 --2- 172.21.2.102:0/1921562735 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3b80018730 0x7f3b8001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:53.657 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:53.651 7f3b92ffd700 1 -- 172.21.2.102:0/1921562735 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f3b8c032640 con 0x7f3b9c1af8a0 2021-02-10T11:37:53.657 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:53.651 7f3b99e1b700 1 --2- 172.21.2.102:0/1921562735 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3b80018730 0x7f3b8001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:53.657 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:53.652 7f3b99e1b700 1 --2- 172.21.2.102:0/1921562735 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3b80018730 0x7f3b8001ac10 crc :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:53.684 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:53.682 7f3ba08e4700 1 -- 172.21.2.102:0/1921562735 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b9c105a20 con 0x7f3b9c1af8a0 2021-02-10T11:37:53.687 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:53.685 7f3b92ffd700 1 -- 172.21.2.102:0/1921562735 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f3b8c01f500 con 0x7f3b9c1af8a0 2021-02-10T11:37:53.815 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:53.813 7f3ba08e4700 1 -- 172.21.2.102:0/1921562735 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3b9c0f0b50 con 0x7f3b9c1af8a0 2021-02-10T11:37:53.816 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:53.813 7f3b92ffd700 1 -- 172.21.2.102:0/1921562735 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 149 v149) v1 ==== 95+0+2108 (secure 0 0 0) 0x7f3b8c00bdd0 con 0x7f3b9c1af8a0 2021-02-10T11:37:53.816 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:37:53.816 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":149,"default_fscid":18,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":12732,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.102:6835/3244979225","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6834","nonce":3244979225},{"type":"v1","addr":"172.21.2.102:6835","nonce":3244979225}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":145}],"filesystems":[{"mdsmap":{"epoch":149,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:37:36.348132","modified":"2021-02-10 11:37:38.355444","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8665},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8665":{"gid":8665,"name":"b","rank":0,"incarnation":146,"state":"up:active","state_seq":3,"addr":"172.21.2.113:6833/1669982519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6832","nonce":1669982519},{"type":"v1","addr":"172.21.2.113:6833","nonce":1669982519}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[37],"metadata_pool":36,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":18}]} 2021-02-10T11:37:53.817 INFO:teuthology.orchestra.run.gibba002.stderr:dumped fsmap epoch 149 2021-02-10T11:37:53.818 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:53.815 7f3ba08e4700 1 -- 172.21.2.102:0/1921562735 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3b80018730 msgr2=0x7f3b8001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:53.818 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:53.815 7f3ba08e4700 1 --2- 172.21.2.102:0/1921562735 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3b80018730 0x7f3b8001ac10 crc :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:53.818 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:53.815 7f3ba08e4700 1 -- 172.21.2.102:0/1921562735 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3b9c1af8a0 msgr2=0x7f3b9c198e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:53.818 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:53.815 7f3ba08e4700 1 --2- 172.21.2.102:0/1921562735 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3b9c1af8a0 0x7f3b9c198e60 secure :-1 s=READY pgs=3022 cs=0 l=1 rev1=1 rx=0x7f3b8c00b640 tx=0x7f3b8c00b9e0).stop 2021-02-10T11:37:53.818 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:53.816 7f3ba08e4700 1 -- 172.21.2.102:0/1921562735 shutdown_connections 2021-02-10T11:37:53.818 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:53.816 7f3ba08e4700 1 --2- 172.21.2.102:0/1921562735 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3b80018730 0x7f3b8001ac10 unknown :-1 s=CLOSED pgs=1326 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:53.818 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:53.816 7f3ba08e4700 1 --2- 172.21.2.102:0/1921562735 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3b9c197d00 0x7f3b9c198100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:53.818 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:53.816 7f3ba08e4700 1 --2- 172.21.2.102:0/1921562735 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3b9c1af050 0x7f3b9c0aa3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:53.819 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:53.816 7f3ba08e4700 1 --2- 172.21.2.102:0/1921562735 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3b9c1af8a0 0x7f3b9c198e60 unknown :-1 s=CLOSED pgs=3022 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:53.819 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:53.816 7f3ba08e4700 1 -- 172.21.2.102:0/1921562735 >> 172.21.2.102:0/1921562735 conn(0x7f3b9c1a37a0 msgr2=0x7f3b9c18c230 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:53.819 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:53.816 7f3ba08e4700 1 -- 172.21.2.102:0/1921562735 shutdown_connections 2021-02-10T11:37:53.819 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:53.816 7f3ba08e4700 1 -- 172.21.2.102:0/1921562735 wait complete. 2021-02-10T11:37:54.028 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 149, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:37:36.348132', 'modified': '2021-02-10 11:37:38.355444', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8665}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8665': {'gid': 8665, 'name': 'b', 'rank': 0, 'incarnation': 146, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.113:6833/1669982519', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.113:6832', 'nonce': 1669982519}, {'type': 'v1', 'addr': '172.21.2.113:6833', 'nonce': 1669982519}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [37], 'metadata_pool': 36, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:37:54.028 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:37:54.028 DEBUG:teuthology.orchestra.run.gibba013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:37:54.427 INFO:teuthology.orchestra.run.gibba013.stdout:{ 2021-02-10T11:37:54.427 INFO:teuthology.orchestra.run.gibba013.stdout: "cluster_fsid": "68db30ea-dbce-4aef-a6df-f01c6fea0513", 2021-02-10T11:37:54.427 INFO:teuthology.orchestra.run.gibba013.stdout: "whoami": 0, 2021-02-10T11:37:54.427 INFO:teuthology.orchestra.run.gibba013.stdout: "id": 8665, 2021-02-10T11:37:54.427 INFO:teuthology.orchestra.run.gibba013.stdout: "want_state": "up:active", 2021-02-10T11:37:54.428 INFO:teuthology.orchestra.run.gibba013.stdout: "state": "up:active", 2021-02-10T11:37:54.428 INFO:teuthology.orchestra.run.gibba013.stdout: "rank_uptime": 18.076054156000001, 2021-02-10T11:37:54.428 INFO:teuthology.orchestra.run.gibba013.stdout: "mdsmap_epoch": 149, 2021-02-10T11:37:54.428 INFO:teuthology.orchestra.run.gibba013.stdout: "osdmap_epoch": 172, 2021-02-10T11:37:54.428 INFO:teuthology.orchestra.run.gibba013.stdout: "osdmap_epoch_barrier": 172, 2021-02-10T11:37:54.428 INFO:teuthology.orchestra.run.gibba013.stdout: "uptime": 23.748628978999999 2021-02-10T11:37:54.428 INFO:teuthology.orchestra.run.gibba013.stdout:} 2021-02-10T11:37:54.431 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "68db30ea-dbce-4aef-a6df-f01c6fea0513", "whoami": 0, "id": 8665, "want_state": "up:active", "state": "up:active", "rank_uptime": 18.076054156000001, "mdsmap_epoch": 149, "osdmap_epoch": 172, "osdmap_epoch_barrier": 172, "uptime": 23.748628978999999 } 2021-02-10T11:37:54.431 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2021-02-10T11:37:54.431 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2021-02-10T11:37:54.431 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.1 at ubuntu@gibba002.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.1... 2021-02-10T11:37:54.431 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:37:54.432 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec mkdir -p /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:37:54.766 DEBUG:teuthology.orchestra.run.gibba002:> sudo modprobe fuse 2021-02-10T11:37:55.111 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:37:55.111 DEBUG:teuthology.orchestra.run.gibba002:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:37:55.455 INFO:teuthology.orchestra.run.gibba002.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:37:55.455 INFO:teuthology.orchestra.run.gibba002.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:37:55.456 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:37:55.456 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:37:55.456 DEBUG:teuthology.orchestra.run.gibba002:> ls /sys/fs/fuse/connections 2021-02-10T11:37:55.792 INFO:teuthology.orchestra.run.gibba002.stdout:43 2021-02-10T11:37:55.792 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [43] 2021-02-10T11:37:55.792 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.1 /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:37:55.795 DEBUG:teuthology.orchestra.run.gibba002:> sudo modprobe fuse 2021-02-10T11:37:56.140 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:37:56.141 DEBUG:teuthology.orchestra.run.gibba002:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:37:56.207 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba002.stderr:ceph-fuse[209013]: starting ceph client 2021-02-10T11:37:56.207 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba002.stderr:2021-02-10 11:37:56.205 7fab87583f80 -1 init, newargv = 0x563cb5e34dc0 newargc=9 2021-02-10T11:37:56.213 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba002.stderr:ceph-fuse[209013]: starting fuse 2021-02-10T11:37:56.483 INFO:teuthology.orchestra.run.gibba002.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:37:56.484 INFO:teuthology.orchestra.run.gibba002.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:37:56.484 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:37:56.485 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:37:56.485 DEBUG:teuthology.orchestra.run.gibba002:> ls /sys/fs/fuse/connections 2021-02-10T11:37:56.820 INFO:teuthology.orchestra.run.gibba002.stdout:43 2021-02-10T11:37:56.820 INFO:teuthology.orchestra.run.gibba002.stdout:44 2021-02-10T11:37:56.821 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43, 44] 2021-02-10T11:37:56.821 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:37:56.821 DEBUG:teuthology.orchestra.run.gibba002:> sudo python3 -c ' 2021-02-10T11:37:56.821 DEBUG:teuthology.orchestra.run.gibba002:> import glob 2021-02-10T11:37:56.821 DEBUG:teuthology.orchestra.run.gibba002:> import re 2021-02-10T11:37:56.821 DEBUG:teuthology.orchestra.run.gibba002:> import os 2021-02-10T11:37:56.821 DEBUG:teuthology.orchestra.run.gibba002:> import subprocess 2021-02-10T11:37:56.821 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:37:56.821 DEBUG:teuthology.orchestra.run.gibba002:> def find_socket(client_name): 2021-02-10T11:37:56.821 DEBUG:teuthology.orchestra.run.gibba002:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2021-02-10T11:37:56.821 DEBUG:teuthology.orchestra.run.gibba002:> files = glob.glob(asok_path) 2021-02-10T11:37:56.821 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:37:56.822 DEBUG:teuthology.orchestra.run.gibba002:> # Given a non-glob path, it better be there 2021-02-10T11:37:56.822 DEBUG:teuthology.orchestra.run.gibba002:> if "*" not in asok_path: 2021-02-10T11:37:56.822 DEBUG:teuthology.orchestra.run.gibba002:> assert(len(files) == 1) 2021-02-10T11:37:56.822 DEBUG:teuthology.orchestra.run.gibba002:> return files[0] 2021-02-10T11:37:56.822 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:37:56.822 DEBUG:teuthology.orchestra.run.gibba002:> for f in files: 2021-02-10T11:37:56.822 DEBUG:teuthology.orchestra.run.gibba002:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2021-02-10T11:37:56.822 DEBUG:teuthology.orchestra.run.gibba002:> if os.path.exists("/proc/{0}".format(pid)): 2021-02-10T11:37:56.822 DEBUG:teuthology.orchestra.run.gibba002:> return f 2021-02-10T11:37:56.822 DEBUG:teuthology.orchestra.run.gibba002:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2021-02-10T11:37:56.822 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:37:56.822 DEBUG:teuthology.orchestra.run.gibba002:> print(find_socket("client.1")) 2021-02-10T11:37:56.822 DEBUG:teuthology.orchestra.run.gibba002:> ' 2021-02-10T11:37:57.185 INFO:teuthology.orchestra.run.gibba002.stdout:/var/run/ceph/ceph-client.1.209013.asok 2021-02-10T11:37:57.188 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.209013.asok 2021-02-10T11:37:57.188 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:37:57.188 DEBUG:teuthology.orchestra.run.gibba002:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.209013.asok status 2021-02-10T11:37:57.579 INFO:teuthology.orchestra.run.gibba002.stdout:{ 2021-02-10T11:37:57.580 INFO:teuthology.orchestra.run.gibba002.stdout: "metadata": { 2021-02-10T11:37:57.580 INFO:teuthology.orchestra.run.gibba002.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:37:57.580 INFO:teuthology.orchestra.run.gibba002.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:37:57.580 INFO:teuthology.orchestra.run.gibba002.stdout: "entity_id": "1", 2021-02-10T11:37:57.580 INFO:teuthology.orchestra.run.gibba002.stdout: "hostname": "gibba002", 2021-02-10T11:37:57.580 INFO:teuthology.orchestra.run.gibba002.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2021-02-10T11:37:57.580 INFO:teuthology.orchestra.run.gibba002.stdout: "pid": "209013", 2021-02-10T11:37:57.580 INFO:teuthology.orchestra.run.gibba002.stdout: "root": "/" 2021-02-10T11:37:57.580 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:37:57.580 INFO:teuthology.orchestra.run.gibba002.stdout: "dentry_count": 0, 2021-02-10T11:37:57.580 INFO:teuthology.orchestra.run.gibba002.stdout: "dentry_pinned_count": 0, 2021-02-10T11:37:57.581 INFO:teuthology.orchestra.run.gibba002.stdout: "id": 12936, 2021-02-10T11:37:57.581 INFO:teuthology.orchestra.run.gibba002.stdout: "inst": { 2021-02-10T11:37:57.581 INFO:teuthology.orchestra.run.gibba002.stdout: "name": { 2021-02-10T11:37:57.581 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "client", 2021-02-10T11:37:57.581 INFO:teuthology.orchestra.run.gibba002.stdout: "num": 12936 2021-02-10T11:37:57.581 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:37:57.581 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": { 2021-02-10T11:37:57.581 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:37:57.581 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:0", 2021-02-10T11:37:57.581 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 2257335328 2021-02-10T11:37:57.581 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:37:57.581 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:37:57.582 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": { 2021-02-10T11:37:57.582 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:37:57.582 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:0", 2021-02-10T11:37:57.582 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 2257335328 2021-02-10T11:37:57.582 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:37:57.582 INFO:teuthology.orchestra.run.gibba002.stdout: "inst_str": "client.12936 172.21.2.102:0/2257335328", 2021-02-10T11:37:57.582 INFO:teuthology.orchestra.run.gibba002.stdout: "addr_str": "172.21.2.102:0/2257335328", 2021-02-10T11:37:57.582 INFO:teuthology.orchestra.run.gibba002.stdout: "inode_count": 1, 2021-02-10T11:37:57.582 INFO:teuthology.orchestra.run.gibba002.stdout: "mds_epoch": 149, 2021-02-10T11:37:57.582 INFO:teuthology.orchestra.run.gibba002.stdout: "osd_epoch": 172, 2021-02-10T11:37:57.582 INFO:teuthology.orchestra.run.gibba002.stdout: "osd_epoch_barrier": 0, 2021-02-10T11:37:57.582 INFO:teuthology.orchestra.run.gibba002.stdout: "blacklisted": false 2021-02-10T11:37:57.582 INFO:teuthology.orchestra.run.gibba002.stdout:} 2021-02-10T11:37:57.584 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:37:57.584 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:37:57.584 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:37:57.920 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:37:57.920 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2021-02-10T11:37:57.920 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:37:57.920 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:37:58.268 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2021-02-10T11:37:58.700 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:58.681 7f8567caf700 1 Processor -- start 2021-02-10T11:37:58.700 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:58.681 7f8567caf700 1 -- start start 2021-02-10T11:37:58.700 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:58.682 7f8567caf700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f85600d1880 0x7f85601b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:58.700 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:58.682 7f8567caf700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f85601b0af0 0x7f8560108950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:58.700 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:58.682 7f8567caf700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85600fe1d0 con 0x7f85600d1c80 2021-02-10T11:37:58.700 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:58.682 7f8567caf700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8560109a70 con 0x7f85601b0af0 2021-02-10T11:37:58.700 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:58.682 7f8567caf700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8560109760 con 0x7f85600d1880 2021-02-10T11:37:58.700 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:58.682 7f85652ca700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f85601b0af0 0x7f8560108950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:58.700 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:58.682 7f85652ca700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f85601b0af0 0x7f8560108950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:39110/0 (socket says 172.21.2.102:39110) 2021-02-10T11:37:58.701 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:58.682 7f85652ca700 1 -- 172.21.2.102:0/3109608726 learned_addr learned my addr 172.21.2.102:0/3109608726 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:37:58.701 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:58.682 7f8565acb700 1 --2- 172.21.2.102:0/3109608726 >> v2:172.21.2.113:3300/0 conn(0x7f85600d1880 0x7f85601b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:58.701 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:58.683 7f85652ca700 1 -- 172.21.2.102:0/3109608726 >> v2:172.21.2.113:3300/0 conn(0x7f85600d1880 msgr2=0x7f85601b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:37:58.701 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:58.683 7f85652ca700 1 --2- 172.21.2.102:0/3109608726 >> v2:172.21.2.113:3300/0 conn(0x7f85600d1880 0x7f85601b04d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:58.701 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:58.683 7f85652ca700 1 -- 172.21.2.102:0/3109608726 >> v1:172.21.2.113:6789/0 conn(0x7f85600d1c80 legacy=0x7f8560108f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:37:58.701 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:58.683 7f85652ca700 1 -- 172.21.2.102:0/3109608726 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8560088070 con 0x7f85601b0af0 2021-02-10T11:37:58.701 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:58.683 7f8565acb700 1 --2- 172.21.2.102:0/3109608726 >> v2:172.21.2.113:3300/0 conn(0x7f85600d1880 0x7f85601b04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:37:58.701 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:58.683 7f85652ca700 1 --2- 172.21.2.102:0/3109608726 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f85601b0af0 0x7f8560108950 secure :-1 s=READY pgs=3026 cs=0 l=1 rev1=1 rx=0x7f8554006700 tx=0x7f8554009b80).ready entity=mon.0 client_cookie=666fc8904ff35af1 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:58.701 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:58.683 7f8564ac9700 1 -- 172.21.2.102:0/3109608726 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f855400c0c0 con 0x7f85601b0af0 2021-02-10T11:37:58.701 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:58.683 7f8564ac9700 1 -- 172.21.2.102:0/3109608726 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f855400c400 con 0x7f85601b0af0 2021-02-10T11:37:58.701 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:58.683 7f8564ac9700 1 -- 172.21.2.102:0/3109608726 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f855400c8e0 con 0x7f85601b0af0 2021-02-10T11:37:58.701 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:58.683 7f8567caf700 1 -- 172.21.2.102:0/3109608726 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f85601b0af0 msgr2=0x7f8560108950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:58.702 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:58.683 7f8567caf700 1 --2- 172.21.2.102:0/3109608726 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f85601b0af0 0x7f8560108950 secure :-1 s=READY pgs=3026 cs=0 l=1 rev1=1 rx=0x7f8554006700 tx=0x7f8554009b80).stop 2021-02-10T11:37:58.702 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:58.683 7f8567caf700 1 -- 172.21.2.102:0/3109608726 shutdown_connections 2021-02-10T11:37:58.702 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:58.683 7f8567caf700 1 --2- 172.21.2.102:0/3109608726 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f85601b0af0 0x7f8560108950 unknown :-1 s=CLOSED pgs=3026 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:58.702 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:58.683 7f8567caf700 1 --2- 172.21.2.102:0/3109608726 >> v2:172.21.2.113:3300/0 conn(0x7f85600d1880 0x7f85601b04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:58.702 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:58.683 7f8567caf700 1 -- 172.21.2.102:0/3109608726 >> 172.21.2.102:0/3109608726 conn(0x7f85600ce430 msgr2=0x7f85600d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:58.702 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:58.684 7f8567caf700 1 -- 172.21.2.102:0/3109608726 shutdown_connections 2021-02-10T11:37:58.702 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:58.684 7f8567caf700 1 -- 172.21.2.102:0/3109608726 wait complete. 2021-02-10T11:37:58.703 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:58.698 7f8567caf700 1 Processor -- start 2021-02-10T11:37:58.704 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:58.698 7f8567caf700 1 -- start start 2021-02-10T11:37:58.704 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:58.698 7f8567caf700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8560094030 0x7f85600aa4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:58.704 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:58.698 7f8567caf700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f85600aaad0 0x7f856019ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:58.704 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:58.698 7f8567caf700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8560093650 0x7f856019c0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:58.704 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:58.698 7f8567caf700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85601b91c0 con 0x7f85600aaad0 2021-02-10T11:37:58.704 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:58.698 7f8567caf700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f856019c6c0 con 0x7f8560094030 2021-02-10T11:37:58.704 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:58.698 7f8567caf700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85601b7f70 con 0x7f8560093650 2021-02-10T11:37:58.704 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:58.699 7f85652ca700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f85600aaad0 0x7f856019ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:58.704 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:58.699 7f85652ca700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f85600aaad0 0x7f856019ba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:39116/0 (socket says 172.21.2.102:39116) 2021-02-10T11:37:58.704 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:58.699 7f85652ca700 1 -- 172.21.2.102:0/2668147366 learned_addr learned my addr 172.21.2.102:0/2668147366 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:37:58.704 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:58.699 7f85652ca700 1 -- 172.21.2.102:0/2668147366 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8560093650 msgr2=0x7f856019c0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:58.704 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:58.699 7f85652ca700 1 --2- 172.21.2.102:0/2668147366 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8560093650 0x7f856019c0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:58.704 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:58.699 7f85652ca700 1 -- 172.21.2.102:0/2668147366 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8560094030 msgr2=0x7f85600aa4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:58.705 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:58.699 7f85652ca700 1 --2- 172.21.2.102:0/2668147366 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8560094030 0x7f85600aa4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:58.705 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:58.699 7f85652ca700 1 -- 172.21.2.102:0/2668147366 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85601b8190 con 0x7f85600aaad0 2021-02-10T11:37:58.705 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:58.699 7f85652ca700 1 --2- 172.21.2.102:0/2668147366 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f85600aaad0 0x7f856019ba80 secure :-1 s=READY pgs=3027 cs=0 l=1 rev1=1 rx=0x7f8554006420 tx=0x7f855400b8e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:58.705 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:58.699 7f85527fc700 1 -- 172.21.2.102:0/2668147366 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8554009440 con 0x7f85600aaad0 2021-02-10T11:37:58.705 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:58.699 7f85527fc700 1 -- 172.21.2.102:0/2668147366 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f855400bcd0 con 0x7f85600aaad0 2021-02-10T11:37:58.705 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:58.699 7f85527fc700 1 -- 172.21.2.102:0/2668147366 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f85540054d0 con 0x7f85600aaad0 2021-02-10T11:37:58.705 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:58.699 7f8567caf700 1 -- 172.21.2.102:0/2668147366 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85601b8190 con 0x7f85600aaad0 2021-02-10T11:37:58.705 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:58.699 7f8567caf700 1 -- 172.21.2.102:0/2668147366 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85601b8190 con 0x7f85600aaad0 2021-02-10T11:37:58.705 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:58.700 7f85527fc700 1 -- 172.21.2.102:0/2668147366 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f85540052e0 con 0x7f85600aaad0 2021-02-10T11:37:58.705 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:58.700 7f85527fc700 1 --2- 172.21.2.102:0/2668147366 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f85480186c0 0x7f854801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:58.705 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:58.700 7f85527fc700 1 -- 172.21.2.102:0/2668147366 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f8554032510 con 0x7f85600aaad0 2021-02-10T11:37:58.705 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:58.700 7f8565acb700 1 --2- 172.21.2.102:0/2668147366 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f85480186c0 0x7f854801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:58.705 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:58.701 7f8565acb700 1 --2- 172.21.2.102:0/2668147366 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f85480186c0 0x7f854801abc0 crc :-1 s=READY pgs=1327 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:58.734 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:58.732 7f8567caf700 1 -- 172.21.2.102:0/2668147366 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8560082aa0 con 0x7f85600aaad0 2021-02-10T11:37:58.736 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:58.734 7f85527fc700 1 -- 172.21.2.102:0/2668147366 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f855402c330 con 0x7f85600aaad0 2021-02-10T11:37:58.872 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:58.869 7f8567caf700 1 -- 172.21.2.102:0/2668147366 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f8560082aa0 con 0x7f85600aaad0 2021-02-10T11:37:59.181 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:59.179 7f85527fc700 1 -- 172.21.2.102:0/2668147366 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v150) v1 ==== 136+0+0 (secure 0 0 0) 0x7f855402cff0 con 0x7f85600aaad0 2021-02-10T11:37:59.190 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:59.188 7f8567caf700 1 -- 172.21.2.102:0/2668147366 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f85480186c0 msgr2=0x7f854801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:59.190 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:59.188 7f8567caf700 1 --2- 172.21.2.102:0/2668147366 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f85480186c0 0x7f854801abc0 crc :-1 s=READY pgs=1327 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:59.191 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:59.188 7f8567caf700 1 -- 172.21.2.102:0/2668147366 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f85600aaad0 msgr2=0x7f856019ba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:59.191 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:59.188 7f8567caf700 1 --2- 172.21.2.102:0/2668147366 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f85600aaad0 0x7f856019ba80 secure :-1 s=READY pgs=3027 cs=0 l=1 rev1=1 rx=0x7f8554006420 tx=0x7f855400b8e0).stop 2021-02-10T11:37:59.191 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:59.188 7f8567caf700 1 -- 172.21.2.102:0/2668147366 shutdown_connections 2021-02-10T11:37:59.191 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:59.188 7f8567caf700 1 --2- 172.21.2.102:0/2668147366 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8560093650 0x7f856019c0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:59.191 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:59.188 7f8567caf700 1 --2- 172.21.2.102:0/2668147366 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f85600aaad0 0x7f856019ba80 unknown :-1 s=CLOSED pgs=3027 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:59.191 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:59.188 7f8567caf700 1 --2- 172.21.2.102:0/2668147366 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f85480186c0 0x7f854801abc0 unknown :-1 s=CLOSED pgs=1327 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:59.191 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:59.188 7f8567caf700 1 --2- 172.21.2.102:0/2668147366 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8560094030 0x7f85600aa4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:59.191 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:59.188 7f8567caf700 1 -- 172.21.2.102:0/2668147366 >> 172.21.2.102:0/2668147366 conn(0x7f85600846b0 msgr2=0x7f8560094950 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:59.191 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:59.188 7f8567caf700 1 -- 172.21.2.102:0/2668147366 shutdown_connections 2021-02-10T11:37:59.191 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:59.188 7f8567caf700 1 -- 172.21.2.102:0/2668147366 wait complete. 2021-02-10T11:37:59.199 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs volume ls 2021-02-10T11:37:59.630 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:59.610 7f1552cdb700 1 Processor -- start 2021-02-10T11:37:59.630 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:59.610 7f1552cdb700 1 -- start start 2021-02-10T11:37:59.630 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:59.611 7f1552cdb700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f154c0d1870 0x7f154c1b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:59.630 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:59.611 7f1552cdb700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f154c0f0650 0x7f154c0cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:59.630 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:59.611 7f1552cdb700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f154c0f0b10 con 0x7f154c1b0a10 2021-02-10T11:37:59.630 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:59.611 7f1552cdb700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f154c0e5ed0 con 0x7f154c0d1870 2021-02-10T11:37:59.630 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:59.611 7f1552cdb700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f154c0d1c70 con 0x7f154c0f0650 2021-02-10T11:37:59.631 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:59.611 7f15512f8700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f154c0f0650 0x7f154c0cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:59.631 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:59.611 7f1550af7700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f154c0d1870 0x7f154c1b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:59.631 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:59.611 7f1550af7700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f154c0d1870 0x7f154c1b03f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.113:3300/0 says I am v2:172.21.2.102:42552/0 (socket says 172.21.2.102:42552) 2021-02-10T11:37:59.631 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:59.611 7f15512f8700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f154c0f0650 0x7f154c0cfb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.113:3301/0 says I am v2:172.21.2.102:44658/0 (socket says 172.21.2.102:44658) 2021-02-10T11:37:59.631 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:59.611 7f1550af7700 1 -- 172.21.2.102:0/90119131 learned_addr learned my addr 172.21.2.102:0/90119131 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:37:59.631 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:59.611 7f154bfff700 1 --1- 172.21.2.102:0/90119131 >> v1:172.21.2.113:6789/0 conn(0x7f154c1b0a10 0x7f154c0cf330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.113:6789/0 says I am v1:172.21.2.102:38674/0 (socket says 172.21.2.102:38674) 2021-02-10T11:37:59.631 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:59.612 7f154b7fe700 1 -- 172.21.2.102:0/90119131 <== mon.1 v1:172.21.2.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3211616904 0 0) 0x7f153c0010b0 con 0x7f154c1b0a10 2021-02-10T11:37:59.631 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:59.612 7f1550af7700 1 -- 172.21.2.102:0/90119131 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f154c0f0650 msgr2=0x7f154c0cfb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:37:59.631 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:59.612 7f1550af7700 1 --2- 172.21.2.102:0/90119131 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f154c0f0650 0x7f154c0cfb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:59.631 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:59.612 7f1550af7700 1 -- 172.21.2.102:0/90119131 >> v1:172.21.2.113:6789/0 conn(0x7f154c1b0a10 legacy=0x7f154c0cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:59.631 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:59.612 7f1550af7700 1 -- 172.21.2.102:0/90119131 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f154c0d1e90 con 0x7f154c0d1870 2021-02-10T11:37:59.631 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:59.612 7f15512f8700 1 --2- 172.21.2.102:0/90119131 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f154c0f0650 0x7f154c0cfb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:37:59.631 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:59.612 7f1550af7700 1 --2- 172.21.2.102:0/90119131 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f154c0d1870 0x7f154c1b03f0 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 rx=0x7f1534006700 tx=0x7f1534009b80).ready entity=mon.1 client_cookie=8006bc377444dbc3 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:59.631 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:59.613 7f154b7fe700 1 -- 172.21.2.102:0/90119131 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f153400c0c0 con 0x7f154c0d1870 2021-02-10T11:37:59.632 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:59.613 7f154b7fe700 1 -- 172.21.2.102:0/90119131 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f153400c340 con 0x7f154c0d1870 2021-02-10T11:37:59.632 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:59.613 7f154b7fe700 1 -- 172.21.2.102:0/90119131 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f153400c860 con 0x7f154c0d1870 2021-02-10T11:37:59.632 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:59.613 7f1552cdb700 1 -- 172.21.2.102:0/90119131 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f154c0d1870 msgr2=0x7f154c1b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:59.632 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:59.613 7f1552cdb700 1 --2- 172.21.2.102:0/90119131 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f154c0d1870 0x7f154c1b03f0 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 rx=0x7f1534006700 tx=0x7f1534009b80).stop 2021-02-10T11:37:59.632 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:59.613 7f1552cdb700 1 -- 172.21.2.102:0/90119131 shutdown_connections 2021-02-10T11:37:59.632 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:59.613 7f1552cdb700 1 --2- 172.21.2.102:0/90119131 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f154c0f0650 0x7f154c0cfb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:59.632 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:59.613 7f1552cdb700 1 --2- 172.21.2.102:0/90119131 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f154c0d1870 0x7f154c1b03f0 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:59.632 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:59.613 7f1552cdb700 1 -- 172.21.2.102:0/90119131 >> 172.21.2.102:0/90119131 conn(0x7f154c0ce450 msgr2=0x7f154c0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:59.632 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:59.613 7f1552cdb700 1 -- 172.21.2.102:0/90119131 shutdown_connections 2021-02-10T11:37:59.632 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:59.613 7f1552cdb700 1 -- 172.21.2.102:0/90119131 wait complete. 2021-02-10T11:37:59.632 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:59.628 7f1552cdb700 1 Processor -- start 2021-02-10T11:37:59.632 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:59.628 7f1552cdb700 1 -- start start 2021-02-10T11:37:59.634 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:59.628 7f1552cdb700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f154c1a06a0 0x7f154c09f380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:59.634 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:59.628 7f1552cdb700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f154c09f9a0 0x7f154c08b560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:59.634 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:59.628 7f1552cdb700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f154c08bb80 0x7f154c08bf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:59.634 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:59.628 7f1552cdb700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f154c08c780 con 0x7f154c09f9a0 2021-02-10T11:37:59.634 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:59.628 7f1552cdb700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f154c09fda0 con 0x7f154c1a06a0 2021-02-10T11:37:59.634 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:59.628 7f1552cdb700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f154c08cd00 con 0x7f154c08bb80 2021-02-10T11:37:59.634 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:59.629 7f154bfff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f154c09f9a0 0x7f154c08b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:59.634 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:59.629 7f154bfff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f154c09f9a0 0x7f154c08b560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:39130/0 (socket says 172.21.2.102:39130) 2021-02-10T11:37:59.634 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:59.629 7f154bfff700 1 -- 172.21.2.102:0/3133475398 learned_addr learned my addr 172.21.2.102:0/3133475398 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:37:59.635 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:59.629 7f154bfff700 1 -- 172.21.2.102:0/3133475398 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f154c08bb80 msgr2=0x7f154c08bf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:59.635 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:59.629 7f154bfff700 1 --2- 172.21.2.102:0/3133475398 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f154c08bb80 0x7f154c08bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:59.635 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:59.629 7f154bfff700 1 -- 172.21.2.102:0/3133475398 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f154c1a06a0 msgr2=0x7f154c09f380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:59.635 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:59.629 7f154bfff700 1 --2- 172.21.2.102:0/3133475398 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f154c1a06a0 0x7f154c09f380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:59.635 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:59.629 7f154bfff700 1 -- 172.21.2.102:0/3133475398 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f154c08cf20 con 0x7f154c09f9a0 2021-02-10T11:37:59.635 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:59.629 7f154bfff700 1 --2- 172.21.2.102:0/3133475398 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f154c09f9a0 0x7f154c08b560 secure :-1 s=READY pgs=3028 cs=0 l=1 rev1=1 rx=0x7f153c001c10 tx=0x7f153c00a8f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:59.635 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:59.629 7f15497fa700 1 -- 172.21.2.102:0/3133475398 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f153c0009b0 con 0x7f154c09f9a0 2021-02-10T11:37:59.635 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:59.629 7f15497fa700 1 -- 172.21.2.102:0/3133475398 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f153c0012c0 con 0x7f154c09f9a0 2021-02-10T11:37:59.635 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:59.629 7f1552cdb700 1 -- 172.21.2.102:0/3133475398 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f154c08cf20 con 0x7f154c09f9a0 2021-02-10T11:37:59.635 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:59.629 7f1552cdb700 1 -- 172.21.2.102:0/3133475398 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f154c08d4b0 con 0x7f154c09f9a0 2021-02-10T11:37:59.635 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:59.629 7f15497fa700 1 -- 172.21.2.102:0/3133475398 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f153c004350 con 0x7f154c09f9a0 2021-02-10T11:37:59.635 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:59.630 7f15497fa700 1 -- 172.21.2.102:0/3133475398 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f153c004140 con 0x7f154c09f9a0 2021-02-10T11:37:59.635 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:59.630 7f15497fa700 1 --2- 172.21.2.102:0/3133475398 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f15300186c0 0x7f153001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:59.636 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:59.630 7f15497fa700 1 -- 172.21.2.102:0/3133475398 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f153c032500 con 0x7f154c09f9a0 2021-02-10T11:37:59.636 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:59.630 7f1550af7700 1 --2- 172.21.2.102:0/3133475398 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f15300186c0 0x7f153001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:59.636 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:59.630 7f1550af7700 1 --2- 172.21.2.102:0/3133475398 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f15300186c0 0x7f153001abc0 crc :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:59.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:59.660 7f1552cdb700 1 -- 172.21.2.102:0/3133475398 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f154c0ebc50 con 0x7f154c09f9a0 2021-02-10T11:37:59.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:59.663 7f15497fa700 1 -- 172.21.2.102:0/3133475398 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f153c0009b0 con 0x7f154c09f9a0 2021-02-10T11:37:59.800 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:59.798 7f1552cdb700 1 -- 172.21.2.102:0/3133475398 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs volume ls", "target": ["mgr", ""]}) v1 -- 0x7f154c09fda0 con 0x7f15300186c0 2021-02-10T11:37:59.800 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:59.798 7f15497fa700 1 -- 172.21.2.102:0/3133475398 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+41 (crc 0 0 0) 0x7f1534006e10 con 0x7f15300186c0 2021-02-10T11:37:59.801 INFO:teuthology.orchestra.run.gibba002.stdout:[ 2021-02-10T11:37:59.801 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:37:59.801 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "cephfs" 2021-02-10T11:37:59.801 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:37:59.801 INFO:teuthology.orchestra.run.gibba002.stdout:] 2021-02-10T11:37:59.802 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:59.800 7f1552cdb700 1 -- 172.21.2.102:0/3133475398 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f15300186c0 msgr2=0x7f153001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:59.802 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:59.800 7f1552cdb700 1 --2- 172.21.2.102:0/3133475398 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f15300186c0 0x7f153001abc0 crc :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:59.802 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:59.800 7f1552cdb700 1 -- 172.21.2.102:0/3133475398 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f154c09f9a0 msgr2=0x7f154c08b560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:59.802 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:59.800 7f1552cdb700 1 --2- 172.21.2.102:0/3133475398 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f154c09f9a0 0x7f154c08b560 secure :-1 s=READY pgs=3028 cs=0 l=1 rev1=1 rx=0x7f153c001c10 tx=0x7f153c00a8f0).stop 2021-02-10T11:37:59.803 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:59.800 7f1552cdb700 1 -- 172.21.2.102:0/3133475398 shutdown_connections 2021-02-10T11:37:59.803 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:59.800 7f1552cdb700 1 --2- 172.21.2.102:0/3133475398 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f154c08bb80 0x7f154c08bf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:59.803 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:59.800 7f1552cdb700 1 --2- 172.21.2.102:0/3133475398 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f154c09f9a0 0x7f154c08b560 unknown :-1 s=CLOSED pgs=3028 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:59.803 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:59.800 7f1552cdb700 1 --2- 172.21.2.102:0/3133475398 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f15300186c0 0x7f153001abc0 unknown :-1 s=CLOSED pgs=1328 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:59.803 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:59.800 7f1552cdb700 1 --2- 172.21.2.102:0/3133475398 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f154c1a06a0 0x7f154c09f380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:59.803 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:59.800 7f1552cdb700 1 -- 172.21.2.102:0/3133475398 >> 172.21.2.102:0/3133475398 conn(0x7f154c0b83b0 msgr2=0x7f154c0a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:59.803 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:59.801 7f1552cdb700 1 -- 172.21.2.102:0/3133475398 shutdown_connections 2021-02-10T11:37:59.803 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:37:59.801 7f1552cdb700 1 -- 172.21.2.102:0/3133475398 wait complete. 2021-02-10T11:37:59.811 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config set mon mon_allow_pool_delete True 2021-02-10T11:38:00.244 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:00.225 7f7024e6f700 1 Processor -- start 2021-02-10T11:38:00.244 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:00.225 7f7024e6f700 1 -- start start 2021-02-10T11:38:00.244 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:00.225 7f7024e6f700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f70200d1870 0x7f70201b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:00.244 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:00.225 7f7024e6f700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f70201b0ae0 0x7f7020108940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:00.244 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:00.225 7f7024e6f700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f70200d1c70 0x7f7020108f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:00.244 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:00.225 7f7024e6f700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70200fe1c0 con 0x7f70200d1c70 2021-02-10T11:38:00.244 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:00.225 7f7024e6f700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7020109a60 con 0x7f70200d1870 2021-02-10T11:38:00.245 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:00.226 7f7024e6f700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7020088070 con 0x7f70201b0ae0 2021-02-10T11:38:00.245 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:00.226 7f701f314700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f70200d1c70 0x7f7020108f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:00.245 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:00.226 7f701f314700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f70200d1c70 0x7f7020108f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:39140/0 (socket says 172.21.2.102:39140) 2021-02-10T11:38:00.246 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:00.226 7f701f314700 1 -- 172.21.2.102:0/4217220802 learned_addr learned my addr 172.21.2.102:0/4217220802 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:38:00.246 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:00.226 7f701eb13700 1 --2- 172.21.2.102:0/4217220802 >> v2:172.21.2.113:3300/0 conn(0x7f70200d1870 0x7f70201b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:00.246 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:00.226 7f701e312700 1 --2- 172.21.2.102:0/4217220802 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f70201b0ae0 0x7f7020108940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:00.246 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:00.226 7f701f314700 1 -- 172.21.2.102:0/4217220802 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f70201b0ae0 msgr2=0x7f7020108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:00.246 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:00.226 7f701f314700 1 --2- 172.21.2.102:0/4217220802 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f70201b0ae0 0x7f7020108940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:00.246 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:00.226 7f701f314700 1 -- 172.21.2.102:0/4217220802 >> v2:172.21.2.113:3300/0 conn(0x7f70200d1870 msgr2=0x7f70201b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:00.246 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:00.226 7f701f314700 1 --2- 172.21.2.102:0/4217220802 >> v2:172.21.2.113:3300/0 conn(0x7f70200d1870 0x7f70201b04c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:00.246 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:00.226 7f701f314700 1 -- 172.21.2.102:0/4217220802 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7020109580 con 0x7f70200d1c70 2021-02-10T11:38:00.246 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:00.226 7f701f314700 1 --2- 172.21.2.102:0/4217220802 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f70200d1c70 0x7f7020108f60 secure :-1 s=READY pgs=3029 cs=0 l=1 rev1=1 rx=0x7f7018006f00 tx=0x7f7018009f10).ready entity=mon.0 client_cookie=dfd4754deb7448a1 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:00.246 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:00.226 7f701db11700 1 -- 172.21.2.102:0/4217220802 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f701800b3b0 con 0x7f70200d1c70 2021-02-10T11:38:00.246 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:00.226 7f701db11700 1 -- 172.21.2.102:0/4217220802 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f701800b630 con 0x7f70200d1c70 2021-02-10T11:38:00.246 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:00.227 7f701db11700 1 -- 172.21.2.102:0/4217220802 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f701800bb50 con 0x7f70200d1c70 2021-02-10T11:38:00.247 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:00.227 7f7024e6f700 1 -- 172.21.2.102:0/4217220802 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f70200d1c70 msgr2=0x7f7020108f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:00.247 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:00.227 7f7024e6f700 1 --2- 172.21.2.102:0/4217220802 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f70200d1c70 0x7f7020108f60 secure :-1 s=READY pgs=3029 cs=0 l=1 rev1=1 rx=0x7f7018006f00 tx=0x7f7018009f10).stop 2021-02-10T11:38:00.247 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:00.227 7f7024e6f700 1 -- 172.21.2.102:0/4217220802 shutdown_connections 2021-02-10T11:38:00.247 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:00.227 7f7024e6f700 1 --2- 172.21.2.102:0/4217220802 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f70201b0ae0 0x7f7020108940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:00.247 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:00.227 7f7024e6f700 1 --2- 172.21.2.102:0/4217220802 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f70200d1c70 0x7f7020108f60 unknown :-1 s=CLOSED pgs=3029 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:00.247 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:00.227 7f7024e6f700 1 --2- 172.21.2.102:0/4217220802 >> v2:172.21.2.113:3300/0 conn(0x7f70200d1870 0x7f70201b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:00.247 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:00.227 7f7024e6f700 1 -- 172.21.2.102:0/4217220802 >> 172.21.2.102:0/4217220802 conn(0x7f70200ce420 msgr2=0x7f70200d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:00.247 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:00.227 7f7024e6f700 1 -- 172.21.2.102:0/4217220802 shutdown_connections 2021-02-10T11:38:00.247 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:00.227 7f7024e6f700 1 -- 172.21.2.102:0/4217220802 wait complete. 2021-02-10T11:38:00.249 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:00.242 7f7024e6f700 1 Processor -- start 2021-02-10T11:38:00.249 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:00.242 7f7024e6f700 1 -- start start 2021-02-10T11:38:00.249 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:00.243 7f7024e6f700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7020085bc0 0x7f70200863e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:00.249 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:00.243 7f7024e6f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f70200954c0 0x7f7020096300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:00.249 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:00.243 7f7024e6f700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f70200958c0 0x7f7020096920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:00.249 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:00.243 7f7024e6f700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7020095d80 con 0x7f70200954c0 2021-02-10T11:38:00.249 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:00.243 7f7024e6f700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70200bfa60 con 0x7f70200958c0 2021-02-10T11:38:00.250 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:00.243 7f7024e6f700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7020085fc0 con 0x7f7020085bc0 2021-02-10T11:38:00.250 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:00.243 7f701e312700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f70200954c0 0x7f7020096300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:00.250 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:00.243 7f701eb13700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7020085bc0 0x7f70200863e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:00.250 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:00.243 7f701eb13700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7020085bc0 0x7f70200863e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.113:3301/0 says I am v2:172.21.2.102:44680/0 (socket says 172.21.2.102:44680) 2021-02-10T11:38:00.250 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:00.243 7f701eb13700 1 -- 172.21.2.102:0/1804713347 learned_addr learned my addr 172.21.2.102:0/1804713347 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:38:00.250 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:00.243 7f701f314700 1 --2- 172.21.2.102:0/1804713347 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f70200958c0 0x7f7020096920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:00.250 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:00.243 7f701eb13700 1 -- 172.21.2.102:0/1804713347 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f70200958c0 msgr2=0x7f7020096920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:00.250 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:00.243 7f701eb13700 1 --2- 172.21.2.102:0/1804713347 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f70200958c0 0x7f7020096920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:00.250 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:00.243 7f701eb13700 1 -- 172.21.2.102:0/1804713347 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f70200954c0 msgr2=0x7f7020096300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:00.250 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:00.243 7f701eb13700 1 --2- 172.21.2.102:0/1804713347 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f70200954c0 0x7f7020096300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:00.250 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:00.243 7f701eb13700 1 -- 172.21.2.102:0/1804713347 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7020086c10 con 0x7f7020085bc0 2021-02-10T11:38:00.250 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:00.243 7f701eb13700 1 --2- 172.21.2.102:0/1804713347 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7020085bc0 0x7f70200863e0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 rx=0x7f7008006f60 tx=0x7f700800b970).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:00.251 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:00.244 7f70177fe700 1 -- 172.21.2.102:0/1804713347 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f700800bd60 con 0x7f7020085bc0 2021-02-10T11:38:00.251 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:00.244 7f70177fe700 1 -- 172.21.2.102:0/1804713347 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7008000c90 con 0x7f7020085bc0 2021-02-10T11:38:00.251 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:00.244 7f70177fe700 1 -- 172.21.2.102:0/1804713347 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7008002220 con 0x7f7020085bc0 2021-02-10T11:38:00.251 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:00.244 7f7024e6f700 1 -- 172.21.2.102:0/1804713347 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7020086c10 con 0x7f7020085bc0 2021-02-10T11:38:00.251 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:00.244 7f7024e6f700 1 -- 172.21.2.102:0/1804713347 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7020086c10 con 0x7f7020085bc0 2021-02-10T11:38:00.251 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:00.244 7f70177fe700 1 -- 172.21.2.102:0/1804713347 <== mon.2 v2:172.21.2.113:3301/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f7008002020 con 0x7f7020085bc0 2021-02-10T11:38:00.251 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:00.244 7f70177fe700 1 --2- 172.21.2.102:0/1804713347 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f70040186c0 0x7f700401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:00.251 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:00.244 7f70177fe700 1 -- 172.21.2.102:0/1804713347 <== mon.2 v2:172.21.2.113:3301/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f7008032550 con 0x7f7020085bc0 2021-02-10T11:38:00.251 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:00.244 7f701e312700 1 --2- 172.21.2.102:0/1804713347 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f70040186c0 0x7f700401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:00.251 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:00.245 7f701e312700 1 --2- 172.21.2.102:0/1804713347 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f70040186c0 0x7f700401abc0 crc :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:00.277 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:00.275 7f7024e6f700 1 -- 172.21.2.102:0/1804713347 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70200fe1c0 con 0x7f7020085bc0 2021-02-10T11:38:00.280 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:00.278 7f70177fe700 1 -- 172.21.2.102:0/1804713347 <== mon.2 v2:172.21.2.113:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f7008006c20 con 0x7f7020085bc0 2021-02-10T11:38:00.434 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:00.431 7f7024e6f700 1 -- 172.21.2.102:0/1804713347 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f70200fe1c0 con 0x7f7020085bc0 2021-02-10T11:38:00.435 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:00.433 7f70177fe700 1 -- 172.21.2.102:0/1804713347 <== mon.2 v2:172.21.2.113:3301/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v42)=0 v42) v1 ==== 122+0+0 (secure 0 0 0) 0x7f7008002000 con 0x7f7020085bc0 2021-02-10T11:38:00.438 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:00.436 7f7024e6f700 1 -- 172.21.2.102:0/1804713347 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f70040186c0 msgr2=0x7f700401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:00.439 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:00.436 7f7024e6f700 1 --2- 172.21.2.102:0/1804713347 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f70040186c0 0x7f700401abc0 crc :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:00.439 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:00.436 7f7024e6f700 1 -- 172.21.2.102:0/1804713347 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7020085bc0 msgr2=0x7f70200863e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:00.439 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:00.436 7f7024e6f700 1 --2- 172.21.2.102:0/1804713347 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7020085bc0 0x7f70200863e0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 rx=0x7f7008006f60 tx=0x7f700800b970).stop 2021-02-10T11:38:00.439 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:00.436 7f7024e6f700 1 -- 172.21.2.102:0/1804713347 shutdown_connections 2021-02-10T11:38:00.439 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:00.436 7f7024e6f700 1 --2- 172.21.2.102:0/1804713347 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f70040186c0 0x7f700401abc0 unknown :-1 s=CLOSED pgs=1329 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:00.439 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:00.436 7f7024e6f700 1 --2- 172.21.2.102:0/1804713347 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f70200958c0 0x7f7020096920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:00.439 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:00.436 7f7024e6f700 1 --2- 172.21.2.102:0/1804713347 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f70200954c0 0x7f7020096300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:00.439 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:00.436 7f7024e6f700 1 --2- 172.21.2.102:0/1804713347 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7020085bc0 0x7f70200863e0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:00.439 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:00.436 7f7024e6f700 1 -- 172.21.2.102:0/1804713347 >> 172.21.2.102:0/1804713347 conn(0x7f70200bef00 msgr2=0x7f702009d1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:00.439 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:00.436 7f7024e6f700 1 -- 172.21.2.102:0/1804713347 shutdown_connections 2021-02-10T11:38:00.439 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:00.436 7f7024e6f700 1 -- 172.21.2.102:0/1804713347 wait complete. 2021-02-10T11:38:00.447 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolumegroup create cephfs group_0000000000799856 2021-02-10T11:38:00.879 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:00.860 7fe9deb92700 1 Processor -- start 2021-02-10T11:38:00.879 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:00.860 7fe9deb92700 1 -- start start 2021-02-10T11:38:00.879 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:00.861 7fe9deb92700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fe9d80d1c70 0x7fe9d80cef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:00.880 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:00.861 7fe9deb92700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe9d80cf580 0x7fe9d80cf980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:00.880 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:00.861 7fe9deb92700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9d80fe1c0 con 0x7fe9d80d1870 2021-02-10T11:38:00.880 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:00.861 7fe9deb92700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9d81b0cb0 con 0x7fe9d80d1c70 2021-02-10T11:38:00.880 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:00.861 7fe9deb92700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9d8109a60 con 0x7fe9d80cf580 2021-02-10T11:38:00.880 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:00.861 7fe9d7fff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fe9d80d1c70 0x7fe9d80cef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:00.880 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:00.861 7fe9d7fff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fe9d80d1c70 0x7fe9d80cef60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:39152/0 (socket says 172.21.2.102:39152) 2021-02-10T11:38:00.880 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:00.861 7fe9d7fff700 1 -- 172.21.2.102:0/83825594 learned_addr learned my addr 172.21.2.102:0/83825594 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:38:00.880 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:00.861 7fe9dc9ae700 1 --1- 172.21.2.102:0/83825594 >> v1:172.21.2.102:6789/0 conn(0x7fe9d80d1870 0x7fe9d81b04c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:59606/0 (socket says 172.21.2.102:59606) 2021-02-10T11:38:00.880 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:00.862 7fe9d7fff700 1 -- 172.21.2.102:0/83825594 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe9d80cf580 msgr2=0x7fe9d80cf980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:00.880 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:00.862 7fe9d7fff700 1 --2- 172.21.2.102:0/83825594 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe9d80cf580 0x7fe9d80cf980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:00.880 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:00.862 7fe9d7fff700 1 -- 172.21.2.102:0/83825594 >> v1:172.21.2.102:6789/0 conn(0x7fe9d80d1870 legacy=0x7fe9d81b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:00.880 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:00.862 7fe9d7fff700 1 -- 172.21.2.102:0/83825594 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9d8088070 con 0x7fe9d80d1c70 2021-02-10T11:38:00.881 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:00.862 7fe9d7fff700 1 --2- 172.21.2.102:0/83825594 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe9d80d1c70 0x7fe9d80cef60 secure :-1 s=READY pgs=3031 cs=0 l=1 rev1=1 rx=0x7fe9c8006a80 tx=0x7fe9c8009f00).ready entity=mon.0 client_cookie=8c2fd75d810b74de server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:00.881 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:00.862 7fe9d77fe700 1 -- 172.21.2.102:0/83825594 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe9c800b5c0 con 0x7fe9d80d1c70 2021-02-10T11:38:00.881 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:00.862 7fe9d77fe700 1 -- 172.21.2.102:0/83825594 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9c800b840 con 0x7fe9d80d1c70 2021-02-10T11:38:00.881 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:00.862 7fe9d77fe700 1 -- 172.21.2.102:0/83825594 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe9c800bd60 con 0x7fe9d80d1c70 2021-02-10T11:38:00.881 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:00.862 7fe9deb92700 1 -- 172.21.2.102:0/83825594 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe9d80d1c70 msgr2=0x7fe9d80cef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:00.881 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:00.862 7fe9deb92700 1 --2- 172.21.2.102:0/83825594 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe9d80d1c70 0x7fe9d80cef60 secure :-1 s=READY pgs=3031 cs=0 l=1 rev1=1 rx=0x7fe9c8006a80 tx=0x7fe9c8009f00).stop 2021-02-10T11:38:00.881 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:00.863 7fe9deb92700 1 -- 172.21.2.102:0/83825594 shutdown_connections 2021-02-10T11:38:00.881 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:00.863 7fe9deb92700 1 --2- 172.21.2.102:0/83825594 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe9d80d1c70 0x7fe9d80cef60 unknown :-1 s=CLOSED pgs=3031 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:00.881 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:00.863 7fe9deb92700 1 --2- 172.21.2.102:0/83825594 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe9d80cf580 0x7fe9d80cf980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:00.881 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:00.863 7fe9deb92700 1 -- 172.21.2.102:0/83825594 >> 172.21.2.102:0/83825594 conn(0x7fe9d80ce420 msgr2=0x7fe9d80d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:00.881 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:00.863 7fe9deb92700 1 -- 172.21.2.102:0/83825594 shutdown_connections 2021-02-10T11:38:00.881 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:00.863 7fe9deb92700 1 -- 172.21.2.102:0/83825594 wait complete. 2021-02-10T11:38:00.883 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:00.877 7fe9deb92700 1 Processor -- start 2021-02-10T11:38:00.883 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:00.878 7fe9deb92700 1 -- start start 2021-02-10T11:38:00.883 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:00.878 7fe9deb92700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe9d80c2370 0x7fe9d818f870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:00.883 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:00.878 7fe9deb92700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe9d8084cb0 0x7fe9d81a7470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:00.883 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:00.878 7fe9deb92700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe9d81a7a90 0x7fe9d8085700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:00.883 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:00.878 7fe9deb92700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9d80be840 con 0x7fe9d8084cb0 2021-02-10T11:38:00.883 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:00.878 7fe9deb92700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9d80c29a0 con 0x7fe9d81a7a90 2021-02-10T11:38:00.883 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:00.878 7fe9deb92700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9d81a7e90 con 0x7fe9d80c2370 2021-02-10T11:38:00.883 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:00.878 7fe9d7fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe9d8084cb0 0x7fe9d81a7470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:00.884 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:00.878 7fe9d7fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe9d8084cb0 0x7fe9d81a7470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:39160/0 (socket says 172.21.2.102:39160) 2021-02-10T11:38:00.884 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:00.878 7fe9d7fff700 1 -- 172.21.2.102:0/2972301055 learned_addr learned my addr 172.21.2.102:0/2972301055 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:38:00.884 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:00.879 7fe9d7fff700 1 -- 172.21.2.102:0/2972301055 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe9d80c2370 msgr2=0x7fe9d818f870 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:38:00.884 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:00.879 7fe9d7fff700 1 --2- 172.21.2.102:0/2972301055 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe9d80c2370 0x7fe9d818f870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:00.884 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:00.879 7fe9d7fff700 1 -- 172.21.2.102:0/2972301055 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe9d81a7a90 msgr2=0x7fe9d8085700 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:38:00.884 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:00.879 7fe9d7fff700 1 --2- 172.21.2.102:0/2972301055 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe9d81a7a90 0x7fe9d8085700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:00.884 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:00.879 7fe9d7fff700 1 -- 172.21.2.102:0/2972301055 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9d81a80b0 con 0x7fe9d8084cb0 2021-02-10T11:38:00.884 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:00.879 7fe9d7fff700 1 --2- 172.21.2.102:0/2972301055 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe9d8084cb0 0x7fe9d81a7470 secure :-1 s=READY pgs=3032 cs=0 l=1 rev1=1 rx=0x7fe9c800b9c0 tx=0x7fe9c800bd60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:00.884 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:00.879 7fe9d57fa700 1 -- 172.21.2.102:0/2972301055 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe9c8006240 con 0x7fe9d8084cb0 2021-02-10T11:38:00.884 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:00.879 7fe9d57fa700 1 -- 172.21.2.102:0/2972301055 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9c80064c0 con 0x7fe9d8084cb0 2021-02-10T11:38:00.884 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:00.879 7fe9deb92700 1 -- 172.21.2.102:0/2972301055 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9d81a80b0 con 0x7fe9d8084cb0 2021-02-10T11:38:00.884 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:00.879 7fe9deb92700 1 -- 172.21.2.102:0/2972301055 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9d8086010 con 0x7fe9d8084cb0 2021-02-10T11:38:00.884 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:00.879 7fe9d57fa700 1 -- 172.21.2.102:0/2972301055 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe9c8005850 con 0x7fe9d8084cb0 2021-02-10T11:38:00.885 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:00.880 7fe9d57fa700 1 -- 172.21.2.102:0/2972301055 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fe9c8006030 con 0x7fe9d8084cb0 2021-02-10T11:38:00.885 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:00.880 7fe9d57fa700 1 --2- 172.21.2.102:0/2972301055 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe9bc0186c0 0x7fe9bc01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:00.885 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:00.880 7fe9d57fa700 1 -- 172.21.2.102:0/2972301055 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fe9c8032950 con 0x7fe9d8084cb0 2021-02-10T11:38:00.885 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:00.880 7fe9dc9ae700 1 --2- 172.21.2.102:0/2972301055 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe9bc0186c0 0x7fe9bc01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:00.885 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:00.880 7fe9dc9ae700 1 --2- 172.21.2.102:0/2972301055 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe9bc0186c0 0x7fe9bc01abc0 crc :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:00.913 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:00.911 7fe9deb92700 1 -- 172.21.2.102:0/2972301055 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9b0002fd0 con 0x7fe9d8084cb0 2021-02-10T11:38:00.915 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:00.913 7fe9d57fa700 1 -- 172.21.2.102:0/2972301055 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fe9c801e960 con 0x7fe9d8084cb0 2021-02-10T11:38:01.045 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:01.043 7fe9deb92700 1 -- 172.21.2.102:0/2972301055 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "target": ["mgr", ""], "group_name": "group_0000000000799856"}) v1 -- 0x7fe9b0002f80 con 0x7fe9bc0186c0 2021-02-10T11:38:01.049 INFO:tasks.ceph.mgr.y.gibba002.stderr:2021-02-10 11:38:01.047 7fd94bc53700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:38:01.049 INFO:tasks.ceph.mgr.y.gibba002.stderr:2021-02-10 11:38:01.047 7fd94bc53700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:38:01.049 INFO:tasks.ceph.mgr.y.gibba002.stderr:2021-02-10 11:38:01.047 7fd94bc53700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:38:01.049 INFO:tasks.ceph.mgr.y.gibba002.stderr:2021-02-10 11:38:01.047 7fd94bc53700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:38:01.049 INFO:tasks.ceph.mgr.y.gibba002.stderr:2021-02-10 11:38:01.047 7fd94bc53700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:38:01.063 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:01.061 7fe9d57fa700 1 -- 172.21.2.102:0/2972301055 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fe9c0007ca0 con 0x7fe9bc0186c0 2021-02-10T11:38:01.078 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:01.076 7fe9deb92700 1 -- 172.21.2.102:0/2972301055 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe9bc0186c0 msgr2=0x7fe9bc01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:01.078 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:01.076 7fe9deb92700 1 --2- 172.21.2.102:0/2972301055 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe9bc0186c0 0x7fe9bc01abc0 crc :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:01.078 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:01.076 7fe9deb92700 1 -- 172.21.2.102:0/2972301055 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe9d8084cb0 msgr2=0x7fe9d81a7470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:01.078 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:01.076 7fe9deb92700 1 --2- 172.21.2.102:0/2972301055 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe9d8084cb0 0x7fe9d81a7470 secure :-1 s=READY pgs=3032 cs=0 l=1 rev1=1 rx=0x7fe9c800b9c0 tx=0x7fe9c800bd60).stop 2021-02-10T11:38:01.079 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:01.076 7fe9deb92700 1 -- 172.21.2.102:0/2972301055 shutdown_connections 2021-02-10T11:38:01.079 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:01.076 7fe9deb92700 1 --2- 172.21.2.102:0/2972301055 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe9bc0186c0 0x7fe9bc01abc0 unknown :-1 s=CLOSED pgs=1330 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:01.079 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:01.076 7fe9deb92700 1 --2- 172.21.2.102:0/2972301055 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe9d81a7a90 0x7fe9d8085700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:01.079 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:01.076 7fe9deb92700 1 --2- 172.21.2.102:0/2972301055 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe9d8084cb0 0x7fe9d81a7470 unknown :-1 s=CLOSED pgs=3032 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:01.079 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:01.076 7fe9deb92700 1 --2- 172.21.2.102:0/2972301055 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe9d80c2370 0x7fe9d818f870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:01.079 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:01.076 7fe9deb92700 1 -- 172.21.2.102:0/2972301055 >> 172.21.2.102:0/2972301055 conn(0x7fe9d80b5dc0 msgr2=0x7fe9d80c2d90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:01.079 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:01.076 7fe9deb92700 1 -- 172.21.2.102:0/2972301055 shutdown_connections 2021-02-10T11:38:01.079 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:01.076 7fe9deb92700 1 -- 172.21.2.102:0/2972301055 wait complete. 2021-02-10T11:38:01.087 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000492282 --group_name group_0000000000799856 2021-02-10T11:38:01.520 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:01.500 7f7ef0c75700 1 Processor -- start 2021-02-10T11:38:01.520 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:01.500 7f7ef0c75700 1 -- start start 2021-02-10T11:38:01.520 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:01.500 7f7ef0c75700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7eec0d1880 0x7f7eec1b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:01.520 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:01.500 7f7ef0c75700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f7eec0cff80 0x7f7eec109140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:01.520 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:01.500 7f7ef0c75700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7eec109a70 con 0x7f7eec1b0af0 2021-02-10T11:38:01.520 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:01.500 7f7ef0c75700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7eec109760 con 0x7f7eec0cff80 2021-02-10T11:38:01.520 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:01.500 7f7ef0c75700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7eec088070 con 0x7f7eec0d1880 2021-02-10T11:38:01.521 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:01.502 7f7eeb05f700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f7eec0cff80 0x7f7eec109140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:01.521 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:01.502 7f7eea85e700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7eec0d1880 0x7f7eec1b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:01.521 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:01.502 7f7eeb05f700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f7eec0cff80 0x7f7eec109140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.113:3300/0 says I am v2:172.21.2.102:42612/0 (socket says 172.21.2.102:42612) 2021-02-10T11:38:01.521 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:01.502 7f7eea85e700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7eec0d1880 0x7f7eec1b04d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.113:3301/0 says I am v2:172.21.2.102:44714/0 (socket says 172.21.2.102:44714) 2021-02-10T11:38:01.521 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:01.502 7f7eeb05f700 1 -- 172.21.2.102:0/3476314094 learned_addr learned my addr 172.21.2.102:0/3476314094 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:38:01.521 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:01.502 7f7eea85e700 1 -- 172.21.2.102:0/3476314094 >> v2:172.21.2.113:3300/0 conn(0x7f7eec0cff80 msgr2=0x7f7eec109140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:01.521 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:01.502 7f7eea85e700 1 --2- 172.21.2.102:0/3476314094 >> v2:172.21.2.113:3300/0 conn(0x7f7eec0cff80 0x7f7eec109140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:01.521 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:01.502 7f7eea85e700 1 -- 172.21.2.102:0/3476314094 >> v1:172.21.2.113:6789/0 conn(0x7f7eec1b0af0 legacy=0x7f7eec108950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:01.521 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:01.502 7f7eea85e700 1 -- 172.21.2.102:0/3476314094 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7eec088260 con 0x7f7eec0d1880 2021-02-10T11:38:01.521 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:01.502 7f7eeb05f700 1 --2- 172.21.2.102:0/3476314094 >> v2:172.21.2.113:3300/0 conn(0x7f7eec0cff80 0x7f7eec109140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:38:01.521 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:01.502 7f7eea85e700 1 --2- 172.21.2.102:0/3476314094 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7eec0d1880 0x7f7eec1b04d0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 rx=0x7f7ed4006650 tx=0x7f7ed4009a80).ready entity=mon.2 client_cookie=a25f203e9d02014f server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:01.521 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:01.503 7f7ee985c700 1 -- 172.21.2.102:0/3476314094 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7ed40009b0 con 0x7f7eec0d1880 2021-02-10T11:38:01.521 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:01.503 7f7ee985c700 1 -- 172.21.2.102:0/3476314094 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ed400c0c0 con 0x7f7eec0d1880 2021-02-10T11:38:01.522 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:01.503 7f7ee985c700 1 -- 172.21.2.102:0/3476314094 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7ed400c5e0 con 0x7f7eec0d1880 2021-02-10T11:38:01.522 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:01.503 7f7ef0c75700 1 -- 172.21.2.102:0/3476314094 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7eec0d1880 msgr2=0x7f7eec1b04d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:01.522 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:01.503 7f7ef0c75700 1 --2- 172.21.2.102:0/3476314094 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7eec0d1880 0x7f7eec1b04d0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 rx=0x7f7ed4006650 tx=0x7f7ed4009a80).stop 2021-02-10T11:38:01.522 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:01.504 7f7ef0c75700 1 -- 172.21.2.102:0/3476314094 shutdown_connections 2021-02-10T11:38:01.522 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:01.504 7f7ef0c75700 1 --2- 172.21.2.102:0/3476314094 >> v2:172.21.2.113:3300/0 conn(0x7f7eec0cff80 0x7f7eec109140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:01.522 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:01.504 7f7ef0c75700 1 --2- 172.21.2.102:0/3476314094 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7eec0d1880 0x7f7eec1b04d0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:01.522 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:01.504 7f7ef0c75700 1 -- 172.21.2.102:0/3476314094 >> 172.21.2.102:0/3476314094 conn(0x7f7eec0ce430 msgr2=0x7f7eec0d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:01.522 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:01.504 7f7ef0c75700 1 -- 172.21.2.102:0/3476314094 shutdown_connections 2021-02-10T11:38:01.522 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:01.504 7f7ef0c75700 1 -- 172.21.2.102:0/3476314094 wait complete. 2021-02-10T11:38:01.524 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:01.518 7f7ef0c75700 1 Processor -- start 2021-02-10T11:38:01.524 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:01.518 7f7ef0c75700 1 -- start start 2021-02-10T11:38:01.524 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:01.519 7f7ef0c75700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7eec18bca0 0x7f7eec187e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:01.524 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:01.519 7f7ef0c75700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7eec18afd0 0x7f7eec18b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:01.524 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:01.519 7f7ef0c75700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7eec0b1950 0x7f7eec0b1d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:01.524 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:01.519 7f7ef0c75700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7eec18b9f0 con 0x7f7eec18bca0 2021-02-10T11:38:01.524 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:01.519 7f7ef0c75700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7eec1884a0 con 0x7f7eec0b1950 2021-02-10T11:38:01.524 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:01.519 7f7ef0c75700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7eec0b2370 con 0x7f7eec18afd0 2021-02-10T11:38:01.524 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:01.519 7f7eea85e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7eec18bca0 0x7f7eec187e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:01.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:01.519 7f7eea85e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7eec18bca0 0x7f7eec187e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:39184/0 (socket says 172.21.2.102:39184) 2021-02-10T11:38:01.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:01.519 7f7eea85e700 1 -- 172.21.2.102:0/642419348 learned_addr learned my addr 172.21.2.102:0/642419348 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:38:01.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:01.519 7f7eeb05f700 1 --2- 172.21.2.102:0/642419348 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7eec0b1950 0x7f7eec0b1d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:01.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:01.519 7f7eea85e700 1 -- 172.21.2.102:0/642419348 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7eec18afd0 msgr2=0x7f7eec18b3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:01.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:01.519 7f7eea05d700 1 --2- 172.21.2.102:0/642419348 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7eec18afd0 0x7f7eec18b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:01.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:01.519 7f7eea85e700 1 --2- 172.21.2.102:0/642419348 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7eec18afd0 0x7f7eec18b3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:01.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:01.519 7f7eea85e700 1 -- 172.21.2.102:0/642419348 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7eec0b1950 msgr2=0x7f7eec0b1d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:01.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:01.519 7f7eea85e700 1 --2- 172.21.2.102:0/642419348 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7eec0b1950 0x7f7eec0b1d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:01.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:01.519 7f7eea85e700 1 -- 172.21.2.102:0/642419348 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7eec0b2590 con 0x7f7eec18bca0 2021-02-10T11:38:01.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:01.520 7f7eeb05f700 1 --2- 172.21.2.102:0/642419348 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7eec0b1950 0x7f7eec0b1d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:38:01.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:01.520 7f7eea05d700 1 --2- 172.21.2.102:0/642419348 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7eec18afd0 0x7f7eec18b3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:38:01.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:01.520 7f7eea85e700 1 --2- 172.21.2.102:0/642419348 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7eec18bca0 0x7f7eec187e80 secure :-1 s=READY pgs=3035 cs=0 l=1 rev1=1 rx=0x7f7ed4009630 tx=0x7f7ed400b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:01.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:01.520 7f7ee37fe700 1 -- 172.21.2.102:0/642419348 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7ed40093a0 con 0x7f7eec18bca0 2021-02-10T11:38:01.526 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:01.520 7f7ee37fe700 1 -- 172.21.2.102:0/642419348 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ed400bd70 con 0x7f7eec18bca0 2021-02-10T11:38:01.526 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:01.520 7f7ef0c75700 1 -- 172.21.2.102:0/642419348 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7eec0b2590 con 0x7f7eec18bca0 2021-02-10T11:38:01.526 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:01.520 7f7ef0c75700 1 -- 172.21.2.102:0/642419348 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7eec189040 con 0x7f7eec18bca0 2021-02-10T11:38:01.526 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:01.520 7f7ee37fe700 1 -- 172.21.2.102:0/642419348 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7ed4005560 con 0x7f7eec18bca0 2021-02-10T11:38:01.526 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:01.521 7f7ee37fe700 1 -- 172.21.2.102:0/642419348 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f7ed40052e0 con 0x7f7eec18bca0 2021-02-10T11:38:01.526 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:01.521 7f7ee37fe700 1 --2- 172.21.2.102:0/642419348 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7ed0018730 0x7f7ed001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:01.526 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:01.521 7f7ee37fe700 1 -- 172.21.2.102:0/642419348 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f7ed4032800 con 0x7f7eec18bca0 2021-02-10T11:38:01.526 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:01.521 7f7eea05d700 1 --2- 172.21.2.102:0/642419348 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7ed0018730 0x7f7ed001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:01.526 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:01.521 7f7eea05d700 1 --2- 172.21.2.102:0/642419348 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7ed0018730 0x7f7ed001ac10 crc :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:01.553 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:01.550 7f7ef0c75700 1 -- 172.21.2.102:0/642419348 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7eec18b9f0 con 0x7f7eec18bca0 2021-02-10T11:38:01.555 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:01.553 7f7ee37fe700 1 -- 172.21.2.102:0/642419348 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f7ed401f540 con 0x7f7eec18bca0 2021-02-10T11:38:01.681 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:01.679 7f7ef0c75700 1 -- 172.21.2.102:0/642419348 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000492282", "target": ["mgr", ""], "group_name": "group_0000000000799856"}) v1 -- 0x7f7eec189040 con 0x7f7ed0018730 2021-02-10T11:38:01.709 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:01.706 7f7ee37fe700 1 -- 172.21.2.102:0/642419348 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f7edc006b10 con 0x7f7ed0018730 2021-02-10T11:38:01.715 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:01.712 7f7ef0c75700 1 -- 172.21.2.102:0/642419348 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7ed0018730 msgr2=0x7f7ed001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:01.715 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:01.712 7f7ef0c75700 1 --2- 172.21.2.102:0/642419348 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7ed0018730 0x7f7ed001ac10 crc :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:01.715 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:01.712 7f7ef0c75700 1 -- 172.21.2.102:0/642419348 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7eec18bca0 msgr2=0x7f7eec187e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:01.715 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:01.712 7f7ef0c75700 1 --2- 172.21.2.102:0/642419348 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7eec18bca0 0x7f7eec187e80 secure :-1 s=READY pgs=3035 cs=0 l=1 rev1=1 rx=0x7f7ed4009630 tx=0x7f7ed400b980).stop 2021-02-10T11:38:01.715 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:01.712 7f7ef0c75700 1 -- 172.21.2.102:0/642419348 shutdown_connections 2021-02-10T11:38:01.715 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:01.712 7f7ef0c75700 1 --2- 172.21.2.102:0/642419348 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7ed0018730 0x7f7ed001ac10 unknown :-1 s=CLOSED pgs=1331 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:01.715 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:01.712 7f7ef0c75700 1 --2- 172.21.2.102:0/642419348 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7eec0b1950 0x7f7eec0b1d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:01.715 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:01.712 7f7ef0c75700 1 --2- 172.21.2.102:0/642419348 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7eec18afd0 0x7f7eec18b3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:01.715 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:01.712 7f7ef0c75700 1 --2- 172.21.2.102:0/642419348 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7eec18bca0 0x7f7eec187e80 unknown :-1 s=CLOSED pgs=3035 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:01.716 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:01.712 7f7ef0c75700 1 -- 172.21.2.102:0/642419348 >> 172.21.2.102:0/642419348 conn(0x7f7eec18e870 msgr2=0x7f7eec18ecc0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:01.716 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:01.713 7f7ef0c75700 1 -- 172.21.2.102:0/642419348 shutdown_connections 2021-02-10T11:38:01.716 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:01.713 7f7ef0c75700 1 -- 172.21.2.102:0/642419348 wait complete. 2021-02-10T11:38:01.723 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume authorize cephfs subvolume_0000000000492282 guest1 --group_name group_0000000000799856 --tenant_id tenant1 2021-02-10T11:38:02.164 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:02.146 7f07f0c1d700 1 Processor -- start 2021-02-10T11:38:02.164 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:02.146 7f07f0c1d700 1 -- start start 2021-02-10T11:38:02.165 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:02.146 7f07f0c1d700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f07ec10a040 0x7f07ec10a440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:02.165 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:02.146 7f07f0c1d700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f07ec10aa60 0x7f07ec10ae60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:02.165 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:02.146 7f07f0c1d700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07ec1b0d40 con 0x7f07ec0e53a0 2021-02-10T11:38:02.165 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:02.146 7f07f0c1d700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07ec0e5f80 con 0x7f07ec10a040 2021-02-10T11:38:02.165 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:02.146 7f07f0c1d700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07ec109ac0 con 0x7f07ec10aa60 2021-02-10T11:38:02.165 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:02.147 7f07ea05d700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f07ec10a040 0x7f07ec10a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:02.165 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:02.147 7f07ea05d700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f07ec10a040 0x7f07ec10a440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:39194/0 (socket says 172.21.2.102:39194) 2021-02-10T11:38:02.165 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:02.147 7f07ea05d700 1 -- 172.21.2.102:0/2065557841 learned_addr learned my addr 172.21.2.102:0/2065557841 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:38:02.165 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:02.147 7f07ea05d700 1 -- 172.21.2.102:0/2065557841 >> v2:172.21.2.113:3300/0 conn(0x7f07ec10aa60 msgr2=0x7f07ec10ae60 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:38:02.165 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:02.147 7f07ea05d700 1 --2- 172.21.2.102:0/2065557841 >> v2:172.21.2.113:3300/0 conn(0x7f07ec10aa60 0x7f07ec10ae60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:02.165 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:02.147 7f07ea05d700 1 -- 172.21.2.102:0/2065557841 >> v1:172.21.2.113:6789/0 conn(0x7f07ec0e53a0 legacy=0x7f07ec1b0550 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:38:02.165 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:02.147 7f07ea05d700 1 -- 172.21.2.102:0/2065557841 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07ec0e57a0 con 0x7f07ec10a040 2021-02-10T11:38:02.165 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:02.147 7f07ea05d700 1 --2- 172.21.2.102:0/2065557841 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f07ec10a040 0x7f07ec10a440 secure :-1 s=READY pgs=3036 cs=0 l=1 rev1=1 rx=0x7f07dc006700 tx=0x7f07dc009b80).ready entity=mon.0 client_cookie=79c3446d1e77954c server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:02.166 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:02.147 7f07e985c700 1 -- 172.21.2.102:0/2065557841 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f07dc00c0c0 con 0x7f07ec10a040 2021-02-10T11:38:02.166 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:02.147 7f07e985c700 1 -- 172.21.2.102:0/2065557841 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07dc00c440 con 0x7f07ec10a040 2021-02-10T11:38:02.166 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:02.147 7f07e985c700 1 -- 172.21.2.102:0/2065557841 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f07dc00c960 con 0x7f07ec10a040 2021-02-10T11:38:02.166 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:02.147 7f07f0c1d700 1 -- 172.21.2.102:0/2065557841 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f07ec10a040 msgr2=0x7f07ec10a440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:02.166 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:02.147 7f07f0c1d700 1 --2- 172.21.2.102:0/2065557841 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f07ec10a040 0x7f07ec10a440 secure :-1 s=READY pgs=3036 cs=0 l=1 rev1=1 rx=0x7f07dc006700 tx=0x7f07dc009b80).stop 2021-02-10T11:38:02.166 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:02.148 7f07f0c1d700 1 -- 172.21.2.102:0/2065557841 shutdown_connections 2021-02-10T11:38:02.166 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:02.148 7f07f0c1d700 1 --2- 172.21.2.102:0/2065557841 >> v2:172.21.2.113:3300/0 conn(0x7f07ec10aa60 0x7f07ec10ae60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:02.166 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:02.148 7f07f0c1d700 1 --2- 172.21.2.102:0/2065557841 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f07ec10a040 0x7f07ec10a440 unknown :-1 s=CLOSED pgs=3036 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:02.166 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:02.148 7f07f0c1d700 1 -- 172.21.2.102:0/2065557841 >> 172.21.2.102:0/2065557841 conn(0x7f07ec0ce4b0 msgr2=0x7f07ec0e45d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:02.166 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:02.148 7f07f0c1d700 1 -- 172.21.2.102:0/2065557841 shutdown_connections 2021-02-10T11:38:02.166 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:02.148 7f07f0c1d700 1 -- 172.21.2.102:0/2065557841 wait complete. 2021-02-10T11:38:02.168 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:02.163 7f07f0c1d700 1 Processor -- start 2021-02-10T11:38:02.168 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:02.163 7f07f0c1d700 1 -- start start 2021-02-10T11:38:02.168 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:02.163 7f07f0c1d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f07ec1a8820 0x7f07ec1a8c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:02.168 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:02.163 7f07f0c1d700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f07ec08c6d0 0x7f07ec08cad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:02.168 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:02.163 7f07f0c1d700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f07ec08d0f0 0x7f07ec08d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:02.168 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:02.163 7f07f0c1d700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07ec1a93a0 con 0x7f07ec1a8820 2021-02-10T11:38:02.168 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:02.163 7f07f0c1d700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07ec1a9920 con 0x7f07ec08d0f0 2021-02-10T11:38:02.168 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:02.163 7f07f0c1d700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07ec1a9b00 con 0x7f07ec08c6d0 2021-02-10T11:38:02.168 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:02.163 7f07ea85e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f07ec1a8820 0x7f07ec1a8c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:02.168 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:02.163 7f07ea85e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f07ec1a8820 0x7f07ec1a8c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:39198/0 (socket says 172.21.2.102:39198) 2021-02-10T11:38:02.168 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:02.163 7f07ea85e700 1 -- 172.21.2.102:0/2969556936 learned_addr learned my addr 172.21.2.102:0/2969556936 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:38:02.169 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:02.164 7f07ea85e700 1 -- 172.21.2.102:0/2969556936 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f07ec08c6d0 msgr2=0x7f07ec08cad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:02.169 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:02.164 7f07ea85e700 1 --2- 172.21.2.102:0/2969556936 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f07ec08c6d0 0x7f07ec08cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:02.169 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:02.164 7f07ea85e700 1 -- 172.21.2.102:0/2969556936 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f07ec08d0f0 msgr2=0x7f07ec08d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:02.169 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:02.164 7f07ea85e700 1 --2- 172.21.2.102:0/2969556936 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f07ec08d0f0 0x7f07ec08d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:02.169 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:02.164 7f07ea85e700 1 -- 172.21.2.102:0/2969556936 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07ec1a9d20 con 0x7f07ec1a8820 2021-02-10T11:38:02.169 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:02.164 7f07ea85e700 1 --2- 172.21.2.102:0/2969556936 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f07ec1a8820 0x7f07ec1a8c20 secure :-1 s=READY pgs=3037 cs=0 l=1 rev1=1 rx=0x7f07d400b5b0 tx=0x7f07d400b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:02.169 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:02.164 7f07e37fe700 1 -- 172.21.2.102:0/2969556936 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f07d400bd40 con 0x7f07ec1a8820 2021-02-10T11:38:02.169 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:02.164 7f07e37fe700 1 -- 172.21.2.102:0/2969556936 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07d4006240 con 0x7f07ec1a8820 2021-02-10T11:38:02.169 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:02.164 7f07f0c1d700 1 -- 172.21.2.102:0/2969556936 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07ec1a9d20 con 0x7f07ec1a8820 2021-02-10T11:38:02.169 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:02.164 7f07f0c1d700 1 -- 172.21.2.102:0/2969556936 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07ec1a9d20 con 0x7f07ec1a8820 2021-02-10T11:38:02.169 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:02.164 7f07e37fe700 1 -- 172.21.2.102:0/2969556936 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f07d4005720 con 0x7f07ec1a8820 2021-02-10T11:38:02.169 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:02.165 7f07e37fe700 1 -- 172.21.2.102:0/2969556936 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f07d40054a0 con 0x7f07ec1a8820 2021-02-10T11:38:02.169 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:02.165 7f07e37fe700 1 --2- 172.21.2.102:0/2969556936 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f07d0018730 0x7f07d001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:02.170 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:02.165 7f07e37fe700 1 -- 172.21.2.102:0/2969556936 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f07d40327f0 con 0x7f07ec1a8820 2021-02-10T11:38:02.170 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:02.165 7f07ea05d700 1 --2- 172.21.2.102:0/2969556936 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f07d0018730 0x7f07d001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:02.170 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:02.165 7f07ea05d700 1 --2- 172.21.2.102:0/2969556936 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f07d0018730 0x7f07d001ac10 crc :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:02.198 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:02.196 7f07f0c1d700 1 -- 172.21.2.102:0/2969556936 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07ec103810 con 0x7f07ec1a8820 2021-02-10T11:38:02.201 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:02.199 7f07e37fe700 1 -- 172.21.2.102:0/2969556936 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f07d40009b0 con 0x7f07ec1a8820 2021-02-10T11:38:02.325 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:02.323 7f07f0c1d700 1 -- 172.21.2.102:0/2969556936 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume authorize", "vol_name": "cephfs", "sub_name": "subvolume_0000000000492282", "target": ["mgr", ""], "tenant_id": "tenant1", "auth_id": "guest1", "group_name": "group_0000000000799856"}) v1 -- 0x7f07ec103810 con 0x7f07d0018730 2021-02-10T11:38:02.331 INFO:tasks.ceph.mgr.y.gibba002.stderr:Creating meta for ID guest1 with tenant tenant1 2021-02-10T11:38:02.357 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:02.355 7f07e37fe700 1 -- 172.21.2.102:0/2969556936 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+41 (crc 0 0 0) 0x7f07dc000970 con 0x7f07d0018730 2021-02-10T11:38:02.389 INFO:teuthology.orchestra.run.gibba002.stdout:AQCaxSNguXVOFBAAVrxotu2XTizDtpcjZqY/Hg== 2021-02-10T11:38:02.391 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:02.389 7f07f0c1d700 1 -- 172.21.2.102:0/2969556936 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f07d0018730 msgr2=0x7f07d001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:02.391 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:02.389 7f07f0c1d700 1 --2- 172.21.2.102:0/2969556936 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f07d0018730 0x7f07d001ac10 crc :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:02.391 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:02.389 7f07f0c1d700 1 -- 172.21.2.102:0/2969556936 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f07ec1a8820 msgr2=0x7f07ec1a8c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:02.391 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:02.389 7f07f0c1d700 1 --2- 172.21.2.102:0/2969556936 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f07ec1a8820 0x7f07ec1a8c20 secure :-1 s=READY pgs=3037 cs=0 l=1 rev1=1 rx=0x7f07d400b5b0 tx=0x7f07d400b950).stop 2021-02-10T11:38:02.391 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:02.389 7f07f0c1d700 1 -- 172.21.2.102:0/2969556936 shutdown_connections 2021-02-10T11:38:02.391 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:02.389 7f07f0c1d700 1 --2- 172.21.2.102:0/2969556936 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f07d0018730 0x7f07d001ac10 unknown :-1 s=CLOSED pgs=1332 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:02.391 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:02.389 7f07f0c1d700 1 --2- 172.21.2.102:0/2969556936 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f07ec08d0f0 0x7f07ec08d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:02.392 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:02.389 7f07f0c1d700 1 --2- 172.21.2.102:0/2969556936 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f07ec08c6d0 0x7f07ec08cad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:02.392 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:02.389 7f07f0c1d700 1 --2- 172.21.2.102:0/2969556936 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f07ec1a8820 0x7f07ec1a8c20 unknown :-1 s=CLOSED pgs=3037 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:02.392 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:02.389 7f07f0c1d700 1 -- 172.21.2.102:0/2969556936 >> 172.21.2.102:0/2969556936 conn(0x7f07ec18cb60 msgr2=0x7f07ec199f40 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:02.392 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:02.389 7f07f0c1d700 1 -- 172.21.2.102:0/2969556936 shutdown_connections 2021-02-10T11:38:02.392 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:02.389 7f07f0c1d700 1 -- 172.21.2.102:0/2969556936 wait complete. 2021-02-10T11:38:02.399 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.1 && sudo ls volumes 2021-02-10T11:38:02.749 INFO:teuthology.orchestra.run.gibba002.stdout:group_0000000000799856 2021-02-10T11:38:02.749 INFO:teuthology.orchestra.run.gibba002.stdout:_group_0000000000799856:subvolume_0000000000492282.meta 2021-02-10T11:38:02.749 INFO:teuthology.orchestra.run.gibba002.stdout:$guest1.meta 2021-02-10T11:38:02.751 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo sudo cat '/home/ubuntu/cephtest/mnt.0/volumes/$guest1.meta' 2021-02-10T11:38:03.118 INFO:teuthology.orchestra.run.gibba002.stdout:{"compat_version": 1, "tenant_id": "tenant1", "version": 5, "dirty": false, "subvolumes": {"group_0000000000799856/subvolume_0000000000492282": {"access_level": "rw", "dirty": false}}} 2021-02-10T11:38:03.118 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.1 && sudo sed -i s/false/true/g 'volumes/$guest1.meta' 2021-02-10T11:38:03.482 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume authorize cephfs subvolume_0000000000492282 guest1 --group_name group_0000000000799856 --tenant_id tenant1 2021-02-10T11:38:03.924 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:03.905 7f1fea565700 1 Processor -- start 2021-02-10T11:38:03.925 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:03.905 7f1fea565700 1 -- start start 2021-02-10T11:38:03.925 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:03.905 7f1fea565700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1fe40e53a0 0x7f1fe41b0550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:03.925 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:03.905 7f1fea565700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fe4109ac0 con 0x7f1fe41b0b70 2021-02-10T11:38:03.925 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:03.905 7f1fea565700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fe40e57a0 con 0x7f1fe410b050 2021-02-10T11:38:03.925 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:03.905 7f1fea565700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fe40fe2b0 con 0x7f1fe40e53a0 2021-02-10T11:38:03.925 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:03.905 7f1fe37fe700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f1fe41b0b70 0x7f1fe41089d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:59668/0 (socket says 172.21.2.102:59668) 2021-02-10T11:38:03.925 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:03.906 7f1fe37fe700 1 -- 172.21.2.102:0/252343584 learned_addr learned my addr 172.21.2.102:0/252343584 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:38:03.925 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:03.906 7f1fe2ffd700 1 -- 172.21.2.102:0/252343584 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2952351806 0 0) 0x7f1fd0001180 con 0x7f1fe41b0b70 2021-02-10T11:38:03.925 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:03.906 7f1fe2ffd700 1 -- 172.21.2.102:0/252343584 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1fc8002fe0 con 0x7f1fe41b0b70 2021-02-10T11:38:03.925 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:03.906 7f1fe2ffd700 1 -- 172.21.2.102:0/252343584 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 293057433 0 0) 0x7f1fd00010e0 con 0x7f1fe41b0b70 2021-02-10T11:38:03.925 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:03.906 7f1fe2ffd700 1 -- 172.21.2.102:0/252343584 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1fe40e53a0 msgr2=0x7f1fe41b0550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:03.925 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:03.906 7f1fe2ffd700 1 --2- 172.21.2.102:0/252343584 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1fe40e53a0 0x7f1fe41b0550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:03.926 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:03.906 7f1fe2ffd700 1 -- 172.21.2.102:0/252343584 >> v1:172.21.2.113:6789/0 conn(0x7f1fe410b050 legacy=0x7f1fe41091c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:03.926 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:03.906 7f1fe2ffd700 1 -- 172.21.2.102:0/252343584 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fe40e5f80 con 0x7f1fe41b0b70 2021-02-10T11:38:03.926 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:03.906 7f1fe2ffd700 1 -- 172.21.2.102:0/252343584 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7f1fd0001600 con 0x7f1fe41b0b70 2021-02-10T11:38:03.926 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:03.906 7f1fe2ffd700 1 -- 172.21.2.102:0/252343584 >> v1:172.21.2.102:6789/0 conn(0x7f1fe41b0b70 legacy=0x7f1fe41089d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:03.926 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:03.906 7f1fe2ffd700 1 --2- 172.21.2.102:0/252343584 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1fc8003ac0 0x7f1fc8005ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:03.926 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:03.906 7f1fe2ffd700 1 --2- 172.21.2.102:0/252343584 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1fc8006790 0x7f1fc8008c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:03.926 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:03.906 7f1fe2ffd700 1 --2- 172.21.2.102:0/252343584 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1fc8009500 0x7f1fc800ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:03.926 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:03.906 7f1fe2ffd700 1 -- 172.21.2.102:0/252343584 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fc800c2d0 con 0x7f1fc8009500 2021-02-10T11:38:03.926 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:03.906 7f1fe2ffd700 1 -- 172.21.2.102:0/252343584 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fc800c4f0 con 0x7f1fc8003ac0 2021-02-10T11:38:03.926 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:03.906 7f1fe2ffd700 1 -- 172.21.2.102:0/252343584 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fc800c710 con 0x7f1fc8006790 2021-02-10T11:38:03.926 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:03.906 7f1fe8b82700 1 --2- 172.21.2.102:0/252343584 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1fc8009500 0x7f1fc800ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:03.926 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:03.906 7f1fe37fe700 1 --2- 172.21.2.102:0/252343584 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1fc8006790 0x7f1fc8008c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:03.926 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:03.906 7f1fe3fff700 1 --2- 172.21.2.102:0/252343584 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1fc8003ac0 0x7f1fc8005ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:03.927 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:03.907 7f1fe8b82700 1 -- 172.21.2.102:0/252343584 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1fc8006790 msgr2=0x7f1fc8008c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:03.927 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:03.907 7f1fe8b82700 1 --2- 172.21.2.102:0/252343584 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1fc8006790 0x7f1fc8008c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:03.927 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:03.907 7f1fe8b82700 1 -- 172.21.2.102:0/252343584 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1fc8003ac0 msgr2=0x7f1fc8005ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:03.927 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:03.907 7f1fe8b82700 1 --2- 172.21.2.102:0/252343584 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1fc8003ac0 0x7f1fc8005ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:03.927 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:03.907 7f1fe8b82700 1 -- 172.21.2.102:0/252343584 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fc800c9b0 con 0x7f1fc8009500 2021-02-10T11:38:03.927 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:03.907 7f1fe8b82700 1 --2- 172.21.2.102:0/252343584 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1fc8009500 0x7f1fc800ba20 secure :-1 s=READY pgs=3039 cs=0 l=1 rev1=1 rx=0x7f1fdc006e30 tx=0x7f1fdc04de60).ready entity=mon.0 client_cookie=36fe339ef9e604ab server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:03.927 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:03.907 7f1fe2ffd700 1 -- 172.21.2.102:0/252343584 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1fdc006a50 con 0x7f1fc8009500 2021-02-10T11:38:03.927 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:03.907 7f1fe2ffd700 1 -- 172.21.2.102:0/252343584 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fdc001400 con 0x7f1fc8009500 2021-02-10T11:38:03.927 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:03.907 7f1fe2ffd700 1 -- 172.21.2.102:0/252343584 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1fdc001860 con 0x7f1fc8009500 2021-02-10T11:38:03.927 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:03.907 7f1fea565700 1 -- 172.21.2.102:0/252343584 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1fc8009500 msgr2=0x7f1fc800ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:03.927 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:03.907 7f1fea565700 1 --2- 172.21.2.102:0/252343584 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1fc8009500 0x7f1fc800ba20 secure :-1 s=READY pgs=3039 cs=0 l=1 rev1=1 rx=0x7f1fdc006e30 tx=0x7f1fdc04de60).stop 2021-02-10T11:38:03.927 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:03.907 7f1fe8b82700 1 -- 172.21.2.102:0/252343584 reap_dead start 2021-02-10T11:38:03.927 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:03.908 7f1fea565700 1 -- 172.21.2.102:0/252343584 shutdown_connections 2021-02-10T11:38:03.927 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:03.908 7f1fea565700 1 -- 172.21.2.102:0/252343584 >> 172.21.2.102:0/252343584 conn(0x7f1fe40ce4b0 msgr2=0x7f1fe40e45d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:03.928 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:03.908 7f1fea565700 1 -- 172.21.2.102:0/252343584 shutdown_connections 2021-02-10T11:38:03.928 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:03.908 7f1fea565700 1 -- 172.21.2.102:0/252343584 wait complete. 2021-02-10T11:38:03.928 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:03.922 7f1fea565700 1 Processor -- start 2021-02-10T11:38:03.928 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:03.922 7f1fea565700 1 -- start start 2021-02-10T11:38:03.929 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:03.923 7f1fea565700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1fe41a0090 0x7f1fe4107530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:03.929 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:03.923 7f1fea565700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1fe41a0490 0x7f1fe4107b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:03.930 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:03.923 7f1fea565700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1fe4090f70 0x7f1fe4091370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:03.930 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:03.923 7f1fea565700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fe40a9140 con 0x7f1fe41a0490 2021-02-10T11:38:03.930 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:03.923 7f1fea565700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fe40909f0 con 0x7f1fe41a0090 2021-02-10T11:38:03.930 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:03.923 7f1fea565700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fe40a8c80 con 0x7f1fe4090f70 2021-02-10T11:38:03.930 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:03.923 7f1fe37fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1fe41a0490 0x7f1fe4107b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:03.930 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:03.923 7f1fe3fff700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1fe41a0090 0x7f1fe4107530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:03.930 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:03.923 7f1fe37fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1fe41a0490 0x7f1fe4107b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:39226/0 (socket says 172.21.2.102:39226) 2021-02-10T11:38:03.930 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:03.923 7f1fe37fe700 1 -- 172.21.2.102:0/722765934 learned_addr learned my addr 172.21.2.102:0/722765934 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:38:03.930 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:03.923 7f1fe8b82700 1 --2- 172.21.2.102:0/722765934 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1fe4090f70 0x7f1fe4091370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:03.930 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:03.924 7f1fe37fe700 1 -- 172.21.2.102:0/722765934 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1fe4090f70 msgr2=0x7f1fe4091370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:03.930 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:03.924 7f1fe37fe700 1 --2- 172.21.2.102:0/722765934 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1fe4090f70 0x7f1fe4091370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:03.930 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:03.924 7f1fe37fe700 1 -- 172.21.2.102:0/722765934 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1fe41a0090 msgr2=0x7f1fe4107530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:03.931 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:03.924 7f1fe37fe700 1 --2- 172.21.2.102:0/722765934 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1fe41a0090 0x7f1fe4107530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:03.931 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:03.924 7f1fe37fe700 1 -- 172.21.2.102:0/722765934 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fe4108170 con 0x7f1fe41a0490 2021-02-10T11:38:03.931 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:03.924 7f1fe37fe700 1 --2- 172.21.2.102:0/722765934 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1fe41a0490 0x7f1fe4107b50 secure :-1 s=READY pgs=3040 cs=0 l=1 rev1=1 rx=0x7f1fd000a5d0 tx=0x7f1fd000a970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:03.931 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:03.924 7f1fe0ff9700 1 -- 172.21.2.102:0/722765934 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1fd0001c40 con 0x7f1fe41a0490 2021-02-10T11:38:03.931 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:03.924 7f1fe0ff9700 1 -- 172.21.2.102:0/722765934 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fd000ad60 con 0x7f1fe41a0490 2021-02-10T11:38:03.931 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:03.924 7f1fe0ff9700 1 -- 172.21.2.102:0/722765934 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1fd00045b0 con 0x7f1fe41a0490 2021-02-10T11:38:03.931 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:03.924 7f1fea565700 1 -- 172.21.2.102:0/722765934 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fe4108170 con 0x7f1fe41a0490 2021-02-10T11:38:03.931 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:03.924 7f1fea565700 1 -- 172.21.2.102:0/722765934 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fe4091c80 con 0x7f1fe41a0490 2021-02-10T11:38:03.931 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:03.925 7f1fe0ff9700 1 -- 172.21.2.102:0/722765934 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1fd00043a0 con 0x7f1fe41a0490 2021-02-10T11:38:03.931 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:03.925 7f1fe0ff9700 1 --2- 172.21.2.102:0/722765934 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1fc00186c0 0x7f1fc001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:03.931 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:03.925 7f1fe0ff9700 1 -- 172.21.2.102:0/722765934 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f1fd0032540 con 0x7f1fe41a0490 2021-02-10T11:38:03.931 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:03.925 7f1fe3fff700 1 --2- 172.21.2.102:0/722765934 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1fc00186c0 0x7f1fc001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:03.931 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:03.925 7f1fe3fff700 1 --2- 172.21.2.102:0/722765934 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1fc00186c0 0x7f1fc001abc0 crc :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:03.957 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:03.955 7f1fea565700 1 -- 172.21.2.102:0/722765934 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fe4109ac0 con 0x7f1fe41a0490 2021-02-10T11:38:03.960 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:03.958 7f1fe0ff9700 1 -- 172.21.2.102:0/722765934 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f1fd0001c40 con 0x7f1fe41a0490 2021-02-10T11:38:04.086 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:04.084 7f1fea565700 1 -- 172.21.2.102:0/722765934 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume authorize", "vol_name": "cephfs", "sub_name": "subvolume_0000000000492282", "target": ["mgr", ""], "tenant_id": "tenant1", "auth_id": "guest1", "group_name": "group_0000000000799856"}) v1 -- 0x7f1fe4091c80 con 0x7f1fc00186c0 2021-02-10T11:38:04.148 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:04.146 7f1fe0ff9700 1 -- 172.21.2.102:0/722765934 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+41 (crc 0 0 0) 0x7f1fd4000940 con 0x7f1fc00186c0 2021-02-10T11:38:04.149 INFO:teuthology.orchestra.run.gibba002.stdout:AQCaxSNguXVOFBAAVrxotu2XTizDtpcjZqY/Hg== 2021-02-10T11:38:04.151 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:04.149 7f1fea565700 1 -- 172.21.2.102:0/722765934 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1fc00186c0 msgr2=0x7f1fc001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:04.151 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:04.149 7f1fea565700 1 --2- 172.21.2.102:0/722765934 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1fc00186c0 0x7f1fc001abc0 crc :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:04.151 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:04.149 7f1fea565700 1 -- 172.21.2.102:0/722765934 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1fe41a0490 msgr2=0x7f1fe4107b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:04.151 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:04.149 7f1fea565700 1 --2- 172.21.2.102:0/722765934 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1fe41a0490 0x7f1fe4107b50 secure :-1 s=READY pgs=3040 cs=0 l=1 rev1=1 rx=0x7f1fd000a5d0 tx=0x7f1fd000a970).stop 2021-02-10T11:38:04.151 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:04.149 7f1fea565700 1 -- 172.21.2.102:0/722765934 shutdown_connections 2021-02-10T11:38:04.152 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:04.149 7f1fea565700 1 --2- 172.21.2.102:0/722765934 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1fe4090f70 0x7f1fe4091370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:04.152 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:04.149 7f1fea565700 1 --2- 172.21.2.102:0/722765934 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1fe41a0490 0x7f1fe4107b50 unknown :-1 s=CLOSED pgs=3040 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:04.152 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:04.149 7f1fea565700 1 --2- 172.21.2.102:0/722765934 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1fc00186c0 0x7f1fc001abc0 unknown :-1 s=CLOSED pgs=1333 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:04.152 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:04.149 7f1fea565700 1 --2- 172.21.2.102:0/722765934 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1fe41a0090 0x7f1fe4107530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:04.152 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:04.149 7f1fea565700 1 -- 172.21.2.102:0/722765934 >> 172.21.2.102:0/722765934 conn(0x7f1fe409e470 msgr2=0x7f1fe40b5960 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:04.152 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:04.149 7f1fea565700 1 -- 172.21.2.102:0/722765934 shutdown_connections 2021-02-10T11:38:04.152 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:04.149 7f1fea565700 1 -- 172.21.2.102:0/722765934 wait complete. 2021-02-10T11:38:04.159 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo sudo cat '/home/ubuntu/cephtest/mnt.0/volumes/$guest1.meta' 2021-02-10T11:38:04.522 INFO:teuthology.orchestra.run.gibba002.stdout:{"tenant_id": "tenant1", "version": 5, "compat_version": 1, "subvolumes": {"group_0000000000799856/subvolume_0000000000492282": {"access_level": "rw", "dirty": false}}, "dirty": false} 2021-02-10T11:38:04.523 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume deauthorize cephfs subvolume_0000000000492282 guest1 --group_name group_0000000000799856 2021-02-10T11:38:04.956 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:04.938 7f27773e6700 1 Processor -- start 2021-02-10T11:38:04.957 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:04.938 7f27773e6700 1 -- start start 2021-02-10T11:38:04.957 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:04.938 7f27773e6700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f27700d1890 0x7f27701b03d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:04.957 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:04.938 7f27773e6700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f27701b09f0 0x7f27701089a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:04.957 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:04.938 7f27773e6700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f27700d1c90 0x7f2770108fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:04.957 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:04.938 7f27773e6700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f277010b560 con 0x7f27701b09f0 2021-02-10T11:38:04.957 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:04.938 7f27773e6700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27700e5f50 con 0x7f27700d1c90 2021-02-10T11:38:04.957 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:04.938 7f27773e6700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2770109a60 con 0x7f27700d1890 2021-02-10T11:38:04.957 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:04.939 7f2774a01700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f27701b09f0 0x7f27701089a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:04.957 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:04.939 7f2774a01700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f27701b09f0 0x7f27701089a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:39234/0 (socket says 172.21.2.102:39234) 2021-02-10T11:38:04.957 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:04.939 7f2774a01700 1 -- 172.21.2.102:0/676824365 learned_addr learned my addr 172.21.2.102:0/676824365 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:38:04.957 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:04.939 7f2775202700 1 --2- 172.21.2.102:0/676824365 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f27700d1890 0x7f27701b03d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:04.958 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:04.939 7f2775a03700 1 --2- 172.21.2.102:0/676824365 >> v2:172.21.2.113:3300/0 conn(0x7f27700d1c90 0x7f2770108fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:04.958 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:04.939 7f2774a01700 1 -- 172.21.2.102:0/676824365 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f27700d1890 msgr2=0x7f27701b03d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:04.958 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:04.939 7f2774a01700 1 --2- 172.21.2.102:0/676824365 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f27700d1890 0x7f27701b03d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:04.958 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:04.939 7f2774a01700 1 -- 172.21.2.102:0/676824365 >> v2:172.21.2.113:3300/0 conn(0x7f27700d1c90 msgr2=0x7f2770108fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:04.958 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:04.939 7f2774a01700 1 --2- 172.21.2.102:0/676824365 >> v2:172.21.2.113:3300/0 conn(0x7f27700d1c90 0x7f2770108fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:04.958 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:04.939 7f2774a01700 1 -- 172.21.2.102:0/676824365 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2770088070 con 0x7f27701b09f0 2021-02-10T11:38:04.958 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:04.939 7f2775a03700 1 --2- 172.21.2.102:0/676824365 >> v2:172.21.2.113:3300/0 conn(0x7f27700d1c90 0x7f2770108fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:38:04.958 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:04.939 7f2774a01700 1 --2- 172.21.2.102:0/676824365 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f27701b09f0 0x7f27701089a0 secure :-1 s=READY pgs=3041 cs=0 l=1 rev1=1 rx=0x7f276c006700 tx=0x7f276c009b00).ready entity=mon.0 client_cookie=2fc6ac31798bcf1d server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:04.958 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:04.939 7f2763fff700 1 -- 172.21.2.102:0/676824365 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f276c00c0c0 con 0x7f27701b09f0 2021-02-10T11:38:04.958 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:04.939 7f2763fff700 1 -- 172.21.2.102:0/676824365 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f276c00c400 con 0x7f27701b09f0 2021-02-10T11:38:04.958 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:04.939 7f2763fff700 1 -- 172.21.2.102:0/676824365 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f276c00c860 con 0x7f27701b09f0 2021-02-10T11:38:04.958 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:04.939 7f27773e6700 1 -- 172.21.2.102:0/676824365 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f27701b09f0 msgr2=0x7f27701089a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:04.958 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:04.939 7f27773e6700 1 --2- 172.21.2.102:0/676824365 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f27701b09f0 0x7f27701089a0 secure :-1 s=READY pgs=3041 cs=0 l=1 rev1=1 rx=0x7f276c006700 tx=0x7f276c009b00).stop 2021-02-10T11:38:04.959 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:04.940 7f27773e6700 1 -- 172.21.2.102:0/676824365 shutdown_connections 2021-02-10T11:38:04.959 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:04.940 7f27773e6700 1 --2- 172.21.2.102:0/676824365 >> v2:172.21.2.113:3300/0 conn(0x7f27700d1c90 0x7f2770108fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:04.959 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:04.940 7f27773e6700 1 --2- 172.21.2.102:0/676824365 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f27701b09f0 0x7f27701089a0 unknown :-1 s=CLOSED pgs=3041 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:04.959 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:04.940 7f27773e6700 1 --2- 172.21.2.102:0/676824365 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f27700d1890 0x7f27701b03d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:04.959 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:04.940 7f27773e6700 1 -- 172.21.2.102:0/676824365 >> 172.21.2.102:0/676824365 conn(0x7f27700fdee0 msgr2=0x7f27700ce450 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:04.959 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:04.940 7f27773e6700 1 -- 172.21.2.102:0/676824365 shutdown_connections 2021-02-10T11:38:04.959 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:04.940 7f27773e6700 1 -- 172.21.2.102:0/676824365 wait complete. 2021-02-10T11:38:04.961 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:04.955 7f27773e6700 1 Processor -- start 2021-02-10T11:38:04.961 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:04.955 7f27773e6700 1 -- start start 2021-02-10T11:38:04.961 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:04.955 7f27773e6700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f27700b4a50 0x7f277018ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:04.961 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:04.955 7f27773e6700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f277018f300 0x7f277018f700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:04.961 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:04.955 7f27773e6700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f277018fd20 0x7f277018aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:04.961 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:04.955 7f27773e6700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f277018e760 con 0x7f27700b4a50 2021-02-10T11:38:04.961 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:04.955 7f27773e6700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f277019e250 con 0x7f277018f300 2021-02-10T11:38:04.961 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:04.955 7f27773e6700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f277019e430 con 0x7f277018fd20 2021-02-10T11:38:04.961 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:04.955 7f2775202700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f27700b4a50 0x7f277018ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:04.961 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:04.955 7f2775202700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f27700b4a50 0x7f277018ece0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:39238/0 (socket says 172.21.2.102:39238) 2021-02-10T11:38:04.961 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:04.955 7f2775202700 1 -- 172.21.2.102:0/2257421792 learned_addr learned my addr 172.21.2.102:0/2257421792 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:38:04.961 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:04.956 7f2775202700 1 -- 172.21.2.102:0/2257421792 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f277018fd20 msgr2=0x7f277018aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:04.961 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:04.956 7f2775202700 1 --2- 172.21.2.102:0/2257421792 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f277018fd20 0x7f277018aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:04.962 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:04.956 7f2775202700 1 -- 172.21.2.102:0/2257421792 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f277018f300 msgr2=0x7f277018f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:04.962 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:04.956 7f2775202700 1 --2- 172.21.2.102:0/2257421792 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f277018f300 0x7f277018f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:04.962 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:04.956 7f2775202700 1 -- 172.21.2.102:0/2257421792 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f277018b4d0 con 0x7f27700b4a50 2021-02-10T11:38:04.962 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:04.956 7f2775202700 1 --2- 172.21.2.102:0/2257421792 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f27700b4a50 0x7f277018ece0 secure :-1 s=READY pgs=3042 cs=0 l=1 rev1=1 rx=0x7f276400b580 tx=0x7f276400b920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:04.962 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:04.956 7f2761ffb700 1 -- 172.21.2.102:0/2257421792 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f276400bd10 con 0x7f27700b4a50 2021-02-10T11:38:04.962 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:04.956 7f2761ffb700 1 -- 172.21.2.102:0/2257421792 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2764000c90 con 0x7f27700b4a50 2021-02-10T11:38:04.962 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:04.956 7f2761ffb700 1 -- 172.21.2.102:0/2257421792 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f27640022b0 con 0x7f27700b4a50 2021-02-10T11:38:04.962 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:04.956 7f27773e6700 1 -- 172.21.2.102:0/2257421792 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f277018b4d0 con 0x7f27700b4a50 2021-02-10T11:38:04.962 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:04.956 7f27773e6700 1 -- 172.21.2.102:0/2257421792 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f277018ba60 con 0x7f27700b4a50 2021-02-10T11:38:04.962 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:04.957 7f2761ffb700 1 -- 172.21.2.102:0/2257421792 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f2764006c90 con 0x7f27700b4a50 2021-02-10T11:38:04.962 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:04.957 7f2761ffb700 1 --2- 172.21.2.102:0/2257421792 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f27500186c0 0x7f275001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:04.962 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:04.957 7f2761ffb700 1 -- 172.21.2.102:0/2257421792 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f2764032410 con 0x7f27700b4a50 2021-02-10T11:38:04.962 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:04.957 7f2774a01700 1 --2- 172.21.2.102:0/2257421792 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f27500186c0 0x7f275001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:04.963 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:04.957 7f2774a01700 1 --2- 172.21.2.102:0/2257421792 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f27500186c0 0x7f275001abc0 crc :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:04.990 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:04.988 7f27773e6700 1 -- 172.21.2.102:0/2257421792 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2770109a60 con 0x7f27700b4a50 2021-02-10T11:38:04.993 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:04.991 7f2761ffb700 1 -- 172.21.2.102:0/2257421792 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f2764006820 con 0x7f27700b4a50 2021-02-10T11:38:05.119 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:05.117 7f27773e6700 1 -- 172.21.2.102:0/2257421792 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume deauthorize", "vol_name": "cephfs", "sub_name": "subvolume_0000000000492282", "target": ["mgr", ""], "auth_id": "guest1", "group_name": "group_0000000000799856"}) v1 -- 0x7f277019e250 con 0x7f27500186c0 2021-02-10T11:38:05.155 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:05.153 7f2761ffb700 1 -- 172.21.2.102:0/2257421792 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f276c006390 con 0x7f27500186c0 2021-02-10T11:38:05.184 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:05.182 7f27773e6700 1 -- 172.21.2.102:0/2257421792 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f27500186c0 msgr2=0x7f275001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:05.185 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:05.182 7f27773e6700 1 --2- 172.21.2.102:0/2257421792 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f27500186c0 0x7f275001abc0 crc :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:05.185 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:05.182 7f27773e6700 1 -- 172.21.2.102:0/2257421792 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f27700b4a50 msgr2=0x7f277018ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:05.185 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:05.182 7f27773e6700 1 --2- 172.21.2.102:0/2257421792 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f27700b4a50 0x7f277018ece0 secure :-1 s=READY pgs=3042 cs=0 l=1 rev1=1 rx=0x7f276400b580 tx=0x7f276400b920).stop 2021-02-10T11:38:05.185 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:05.182 7f27773e6700 1 -- 172.21.2.102:0/2257421792 shutdown_connections 2021-02-10T11:38:05.185 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:05.182 7f27773e6700 1 --2- 172.21.2.102:0/2257421792 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f277018fd20 0x7f277018aeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:05.185 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:05.182 7f27773e6700 1 --2- 172.21.2.102:0/2257421792 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f27500186c0 0x7f275001abc0 unknown :-1 s=CLOSED pgs=1334 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:05.185 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:05.182 7f27773e6700 1 --2- 172.21.2.102:0/2257421792 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f277018f300 0x7f277018f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:05.185 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:05.182 7f27773e6700 1 --2- 172.21.2.102:0/2257421792 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f27700b4a50 0x7f277018ece0 unknown :-1 s=CLOSED pgs=3042 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:05.185 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:05.182 7f27773e6700 1 -- 172.21.2.102:0/2257421792 >> 172.21.2.102:0/2257421792 conn(0x7f277019f220 msgr2=0x7f27701ae400 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:05.185 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:05.182 7f27773e6700 1 -- 172.21.2.102:0/2257421792 shutdown_connections 2021-02-10T11:38:05.186 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:05.182 7f27773e6700 1 -- 172.21.2.102:0/2257421792 wait complete. 2021-02-10T11:38:05.192 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:38:05.192 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:38:05.192 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:38:05.530 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:38:05.530 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2021-02-10T11:38:05.530 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@gibba002.front.sepia.ceph.com... 2021-02-10T11:38:05.530 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-02-10T11:38:05.531 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:38:05.884 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba002.stderr:ceph-fuse[209013]: fuse finished with error 0 and tester_r 0 2021-02-10T11:38:05.885 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:38:05.886 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:38:05.886 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:38:06.220 INFO:teuthology.orchestra.run.gibba002.stdout:ext2/ext3 2021-02-10T11:38:06.220 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2021-02-10T11:38:06.220 INFO:teuthology.orchestra.run:waiting for 900 2021-02-10T11:38:06.221 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:38:06.221 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:38:06.557 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth rm client.guest1 2021-02-10T11:38:06.986 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:06.967 7f679d88d700 1 Processor -- start 2021-02-10T11:38:06.986 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:06.967 7f679d88d700 1 -- start start 2021-02-10T11:38:06.986 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:06.967 7f679d88d700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f67980d0340 0x7f67980ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:06.986 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:06.967 7f679d88d700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67980f0b10 con 0x7f67980d1870 2021-02-10T11:38:06.986 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:06.967 7f679d88d700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67981b0be0 con 0x7f67980d1c70 2021-02-10T11:38:06.986 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:06.967 7f679d88d700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67980e5ed0 con 0x7f67980d0340 2021-02-10T11:38:06.986 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:06.967 7f6797574700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f67980d1870 0x7f67981b03f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:59702/0 (socket says 172.21.2.102:59702) 2021-02-10T11:38:06.986 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:06.967 7f6797574700 1 -- 172.21.2.102:0/346761639 learned_addr learned my addr 172.21.2.102:0/346761639 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:38:06.986 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:06.967 7f6796d73700 1 --2- 172.21.2.102:0/346761639 >> v2:172.21.2.102:3300/0 conn(0x7f67980d0340 0x7f67980ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:06.986 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:06.968 7f6796d73700 1 -- 172.21.2.102:0/346761639 >> v1:172.21.2.113:6789/0 conn(0x7f67980d1c70 legacy=0x7f67980efc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:06.987 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:06.968 7f6796d73700 1 -- 172.21.2.102:0/346761639 >> v1:172.21.2.102:6789/0 conn(0x7f67980d1870 legacy=0x7f67981b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:06.987 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:06.968 7f6796d73700 1 -- 172.21.2.102:0/346761639 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67980f0450 con 0x7f67980d0340 2021-02-10T11:38:06.987 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:06.968 7f6796d73700 1 --2- 172.21.2.102:0/346761639 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f67980d0340 0x7f67980ef640 secure :-1 s=READY pgs=3044 cs=0 l=1 rev1=1 rx=0x7f6780006650 tx=0x7f6780009a00).ready entity=mon.0 client_cookie=f754c575df9045f2 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:06.987 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:06.968 7f6796572700 1 -- 172.21.2.102:0/346761639 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6780009df0 con 0x7f67980d0340 2021-02-10T11:38:06.987 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:06.968 7f6796572700 1 -- 172.21.2.102:0/346761639 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67800009b0 con 0x7f67980d0340 2021-02-10T11:38:06.987 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:06.968 7f6796572700 1 -- 172.21.2.102:0/346761639 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f678000c2b0 con 0x7f67980d0340 2021-02-10T11:38:06.987 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:06.968 7f679d88d700 1 -- 172.21.2.102:0/346761639 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f67980d0340 msgr2=0x7f67980ef640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:06.987 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:06.968 7f679d88d700 1 --2- 172.21.2.102:0/346761639 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f67980d0340 0x7f67980ef640 secure :-1 s=READY pgs=3044 cs=0 l=1 rev1=1 rx=0x7f6780006650 tx=0x7f6780009a00).stop 2021-02-10T11:38:06.987 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:06.969 7f679d88d700 1 -- 172.21.2.102:0/346761639 shutdown_connections 2021-02-10T11:38:06.987 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:06.969 7f679d88d700 1 --2- 172.21.2.102:0/346761639 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f67980d0340 0x7f67980ef640 unknown :-1 s=CLOSED pgs=3044 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:06.987 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:06.969 7f679d88d700 1 -- 172.21.2.102:0/346761639 >> 172.21.2.102:0/346761639 conn(0x7f67980ce450 msgr2=0x7f67980d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:06.987 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:06.969 7f679d88d700 1 -- 172.21.2.102:0/346761639 shutdown_connections 2021-02-10T11:38:06.987 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:06.969 7f679d88d700 1 -- 172.21.2.102:0/346761639 wait complete. 2021-02-10T11:38:06.988 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:06.984 7f679d88d700 1 Processor -- start 2021-02-10T11:38:06.989 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:06.984 7f679d88d700 1 -- start start 2021-02-10T11:38:06.989 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:06.984 7f679d88d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f67980b1130 0x7f67980b1530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:06.989 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:06.984 7f679d88d700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f67980b1b50 0x7f67980bb030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:06.989 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:06.984 7f679d88d700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f67980bb650 0x7f67980bba50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:06.989 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:06.984 7f679d88d700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f679808baa0 con 0x7f67980b1130 2021-02-10T11:38:06.989 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:06.984 7f679d88d700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67980bc070 con 0x7f67980bb650 2021-02-10T11:38:06.989 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:06.984 7f679d88d700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f679808c020 con 0x7f67980b1b50 2021-02-10T11:38:06.989 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:06.984 7f6797574700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f67980b1130 0x7f67980b1530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:06.989 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:06.985 7f6797574700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f67980b1130 0x7f67980b1530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:39252/0 (socket says 172.21.2.102:39252) 2021-02-10T11:38:06.989 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:06.985 7f6797574700 1 -- 172.21.2.102:0/442097052 learned_addr learned my addr 172.21.2.102:0/442097052 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:38:06.990 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:06.985 7f6797574700 1 -- 172.21.2.102:0/442097052 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f67980b1b50 msgr2=0x7f67980bb030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:06.990 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:06.985 7f6797574700 1 --2- 172.21.2.102:0/442097052 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f67980b1b50 0x7f67980bb030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:06.990 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:06.985 7f6797574700 1 -- 172.21.2.102:0/442097052 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f67980bb650 msgr2=0x7f67980bba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:06.990 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:06.985 7f6797574700 1 --2- 172.21.2.102:0/442097052 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f67980bb650 0x7f67980bba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:06.990 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:06.985 7f6797574700 1 -- 172.21.2.102:0/442097052 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f679808c240 con 0x7f67980b1130 2021-02-10T11:38:06.990 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:06.985 7f6797574700 1 --2- 172.21.2.102:0/442097052 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f67980b1130 0x7f67980b1530 secure :-1 s=READY pgs=3045 cs=0 l=1 rev1=1 rx=0x7f6788001410 tx=0x7f678800a950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:06.990 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:06.985 7f678ffff700 1 -- 172.21.2.102:0/442097052 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f678800ad40 con 0x7f67980b1130 2021-02-10T11:38:06.990 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:06.985 7f678ffff700 1 -- 172.21.2.102:0/442097052 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6788000c90 con 0x7f67980b1130 2021-02-10T11:38:06.990 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:06.985 7f678ffff700 1 -- 172.21.2.102:0/442097052 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6788004620 con 0x7f67980b1130 2021-02-10T11:38:06.990 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:06.985 7f679d88d700 1 -- 172.21.2.102:0/442097052 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f679808c240 con 0x7f67980b1130 2021-02-10T11:38:06.990 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:06.985 7f679d88d700 1 -- 172.21.2.102:0/442097052 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f679808c7d0 con 0x7f67980b1130 2021-02-10T11:38:06.990 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:06.986 7f678ffff700 1 -- 172.21.2.102:0/442097052 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f6788007070 con 0x7f67980b1130 2021-02-10T11:38:06.990 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:06.986 7f678ffff700 1 --2- 172.21.2.102:0/442097052 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f677c0186c0 0x7f677c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:06.991 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:06.986 7f678ffff700 1 -- 172.21.2.102:0/442097052 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f6788032560 con 0x7f67980b1130 2021-02-10T11:38:06.991 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:06.986 7f6796d73700 1 --2- 172.21.2.102:0/442097052 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f677c0186c0 0x7f677c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:06.991 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:06.986 7f6796d73700 1 --2- 172.21.2.102:0/442097052 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f677c0186c0 0x7f677c01abc0 crc :-1 s=READY pgs=1335 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:07.020 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:07.017 7f679d88d700 1 -- 172.21.2.102:0/442097052 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67980ffa00 con 0x7f67980b1130 2021-02-10T11:38:07.022 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:07.020 7f678ffff700 1 -- 172.21.2.102:0/442097052 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f6788000ef0 con 0x7f67980b1130 2021-02-10T11:38:07.151 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:07.149 7f679d88d700 1 -- 172.21.2.102:0/442097052 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "auth rm", "entity": "client.guest1"} v 0) v1 -- 0x7f67980f06f0 con 0x7f67980b1130 2021-02-10T11:38:07.151 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:07.149 7f678ffff700 1 -- 172.21.2.102:0/442097052 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "auth rm", "entity": "client.guest1"}]=0 entity client.guest1 does not exist v99) v1 ==== 117+0+0 (secure 0 0 0) 0x7f6788007050 con 0x7f67980b1130 2021-02-10T11:38:07.152 INFO:teuthology.orchestra.run.gibba002.stderr:entity client.guest1 does not exist 2021-02-10T11:38:07.153 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:07.151 7f679d88d700 1 -- 172.21.2.102:0/442097052 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f677c0186c0 msgr2=0x7f677c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:07.153 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:07.151 7f679d88d700 1 --2- 172.21.2.102:0/442097052 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f677c0186c0 0x7f677c01abc0 crc :-1 s=READY pgs=1335 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:07.153 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:07.151 7f679d88d700 1 -- 172.21.2.102:0/442097052 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f67980b1130 msgr2=0x7f67980b1530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:07.153 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:07.151 7f679d88d700 1 --2- 172.21.2.102:0/442097052 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f67980b1130 0x7f67980b1530 secure :-1 s=READY pgs=3045 cs=0 l=1 rev1=1 rx=0x7f6788001410 tx=0x7f678800a950).stop 2021-02-10T11:38:07.153 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:07.151 7f679d88d700 1 -- 172.21.2.102:0/442097052 shutdown_connections 2021-02-10T11:38:07.154 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:07.151 7f679d88d700 1 --2- 172.21.2.102:0/442097052 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f677c0186c0 0x7f677c01abc0 unknown :-1 s=CLOSED pgs=1335 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:07.154 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:07.151 7f679d88d700 1 --2- 172.21.2.102:0/442097052 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f67980bb650 0x7f67980bba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:07.154 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:07.151 7f679d88d700 1 --2- 172.21.2.102:0/442097052 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f67980b1b50 0x7f67980bb030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:07.154 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:07.151 7f679d88d700 1 --2- 172.21.2.102:0/442097052 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f67980b1130 0x7f67980b1530 unknown :-1 s=CLOSED pgs=3045 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:07.154 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:07.151 7f679d88d700 1 -- 172.21.2.102:0/442097052 >> 172.21.2.102:0/442097052 conn(0x7f67980a74c0 msgr2=0x7f67980bc7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:07.154 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:07.152 7f679d88d700 1 -- 172.21.2.102:0/442097052 shutdown_connections 2021-02-10T11:38:07.154 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:07.152 7f679d88d700 1 -- 172.21.2.102:0/442097052 wait complete. 2021-02-10T11:38:07.162 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000492282 --group_name group_0000000000799856 2021-02-10T11:38:07.595 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:07.577 7fc24d12b700 1 Processor -- start 2021-02-10T11:38:07.596 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:07.577 7fc24d12b700 1 -- start start 2021-02-10T11:38:07.596 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:07.577 7fc24d12b700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc2480d1870 0x7fc2481b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:07.596 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:07.577 7fc24d12b700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fc2480cff70 0x7fc248109130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:07.596 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:07.577 7fc24d12b700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc248109a60 con 0x7fc2481b0ae0 2021-02-10T11:38:07.596 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:07.577 7fc24d12b700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc248109750 con 0x7fc2480cff70 2021-02-10T11:38:07.596 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:07.577 7fc24d12b700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc248088070 con 0x7fc2480d1870 2021-02-10T11:38:07.596 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:07.578 7fc24659c700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fc2481b0ae0 0x7fc248108940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:59718/0 (socket says 172.21.2.102:59718) 2021-02-10T11:38:07.596 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:07.578 7fc24659c700 1 -- 172.21.2.102:0/1716715425 learned_addr learned my addr 172.21.2.102:0/1716715425 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:38:07.596 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:07.578 7fc24759e700 1 --2- 172.21.2.102:0/1716715425 >> v2:172.21.2.102:3300/0 conn(0x7fc2480cff70 0x7fc248109130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:07.596 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:07.578 7fc24759e700 1 -- 172.21.2.102:0/1716715425 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc2480d1870 msgr2=0x7fc2481b04c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:38:07.596 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:07.578 7fc24759e700 1 --2- 172.21.2.102:0/1716715425 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc2480d1870 0x7fc2481b04c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:07.596 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:07.578 7fc24759e700 1 -- 172.21.2.102:0/1716715425 >> v1:172.21.2.102:6789/0 conn(0x7fc2481b0ae0 legacy=0x7fc248108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:07.597 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:07.578 7fc24759e700 1 -- 172.21.2.102:0/1716715425 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc248088260 con 0x7fc2480cff70 2021-02-10T11:38:07.597 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:07.578 7fc24759e700 1 --2- 172.21.2.102:0/1716715425 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc2480cff70 0x7fc248109130 secure :-1 s=READY pgs=3047 cs=0 l=1 rev1=1 rx=0x7fc240006e30 tx=0x7fc240009e60).ready entity=mon.0 client_cookie=a27cd53f59379512 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:07.597 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:07.578 7fc245d9b700 1 -- 172.21.2.102:0/1716715425 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc240006a50 con 0x7fc2480cff70 2021-02-10T11:38:07.597 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:07.578 7fc245d9b700 1 -- 172.21.2.102:0/1716715425 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc24000b300 con 0x7fc2480cff70 2021-02-10T11:38:07.597 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:07.578 7fc245d9b700 1 -- 172.21.2.102:0/1716715425 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc24000b820 con 0x7fc2480cff70 2021-02-10T11:38:07.597 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:07.578 7fc24d12b700 1 -- 172.21.2.102:0/1716715425 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc2480cff70 msgr2=0x7fc248109130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:07.597 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:07.578 7fc24d12b700 1 --2- 172.21.2.102:0/1716715425 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc2480cff70 0x7fc248109130 secure :-1 s=READY pgs=3047 cs=0 l=1 rev1=1 rx=0x7fc240006e30 tx=0x7fc240009e60).stop 2021-02-10T11:38:07.597 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:07.579 7fc24d12b700 1 -- 172.21.2.102:0/1716715425 shutdown_connections 2021-02-10T11:38:07.597 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:07.579 7fc24d12b700 1 --2- 172.21.2.102:0/1716715425 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc2480cff70 0x7fc248109130 unknown :-1 s=CLOSED pgs=3047 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:07.597 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:07.579 7fc24d12b700 1 --2- 172.21.2.102:0/1716715425 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc2480d1870 0x7fc2481b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:07.597 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:07.579 7fc24d12b700 1 -- 172.21.2.102:0/1716715425 >> 172.21.2.102:0/1716715425 conn(0x7fc2480ce420 msgr2=0x7fc2480d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:07.597 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:07.579 7fc24d12b700 1 -- 172.21.2.102:0/1716715425 shutdown_connections 2021-02-10T11:38:07.597 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:07.579 7fc24d12b700 1 -- 172.21.2.102:0/1716715425 wait complete. 2021-02-10T11:38:07.599 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:07.594 7fc24d12b700 1 Processor -- start 2021-02-10T11:38:07.599 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:07.594 7fc24d12b700 1 -- start start 2021-02-10T11:38:07.599 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:07.594 7fc24d12b700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc2481aad00 0x7fc2480ac0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:07.599 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:07.594 7fc24d12b700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc2480ac6c0 0x7fc248187e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:07.599 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:07.594 7fc24d12b700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc248188490 0x7fc2481910c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:07.599 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:07.594 7fc24d12b700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc248188890 con 0x7fc248188490 2021-02-10T11:38:07.599 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:07.594 7fc24d12b700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc248188ab0 con 0x7fc2481aad00 2021-02-10T11:38:07.599 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:07.594 7fc24d12b700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2481916e0 con 0x7fc2480ac6c0 2021-02-10T11:38:07.600 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:07.594 7fc24759e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc248188490 0x7fc2481910c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:07.600 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:07.594 7fc24759e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc248188490 0x7fc2481910c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:39270/0 (socket says 172.21.2.102:39270) 2021-02-10T11:38:07.600 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:07.594 7fc24759e700 1 -- 172.21.2.102:0/624222964 learned_addr learned my addr 172.21.2.102:0/624222964 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:38:07.600 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:07.595 7fc24759e700 1 -- 172.21.2.102:0/624222964 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc2480ac6c0 msgr2=0x7fc248187e70 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:38:07.600 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:07.595 7fc24759e700 1 --2- 172.21.2.102:0/624222964 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc2480ac6c0 0x7fc248187e70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:07.600 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:07.595 7fc24759e700 1 -- 172.21.2.102:0/624222964 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc2481aad00 msgr2=0x7fc2480ac0a0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:38:07.600 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:07.595 7fc24759e700 1 --2- 172.21.2.102:0/624222964 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc2481aad00 0x7fc2480ac0a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:07.600 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:07.595 7fc24759e700 1 -- 172.21.2.102:0/624222964 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc248191900 con 0x7fc248188490 2021-02-10T11:38:07.600 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:07.595 7fc24759e700 1 --2- 172.21.2.102:0/624222964 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc248188490 0x7fc2481910c0 secure :-1 s=READY pgs=3048 cs=0 l=1 rev1=1 rx=0x7fc24000b970 tx=0x7fc24000bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:07.600 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:07.595 7fc23f7fe700 1 -- 172.21.2.102:0/624222964 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc240009380 con 0x7fc248188490 2021-02-10T11:38:07.600 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:07.595 7fc23f7fe700 1 -- 172.21.2.102:0/624222964 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc240005820 con 0x7fc248188490 2021-02-10T11:38:07.600 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:07.595 7fc23f7fe700 1 -- 172.21.2.102:0/624222964 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc240005d40 con 0x7fc248188490 2021-02-10T11:38:07.600 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:07.595 7fc24d12b700 1 -- 172.21.2.102:0/624222964 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc248191900 con 0x7fc248188490 2021-02-10T11:38:07.601 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:07.595 7fc24d12b700 1 -- 172.21.2.102:0/624222964 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc248191e90 con 0x7fc248188490 2021-02-10T11:38:07.601 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:07.596 7fc23f7fe700 1 -- 172.21.2.102:0/624222964 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fc240005cd0 con 0x7fc248188490 2021-02-10T11:38:07.601 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:07.596 7fc23f7fe700 1 --2- 172.21.2.102:0/624222964 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc22c0186c0 0x7fc22c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:07.601 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:07.596 7fc23f7fe700 1 -- 172.21.2.102:0/624222964 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fc240032920 con 0x7fc248188490 2021-02-10T11:38:07.601 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:07.596 7fc246d9d700 1 --2- 172.21.2.102:0/624222964 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc22c0186c0 0x7fc22c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:07.601 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:07.596 7fc246d9d700 1 --2- 172.21.2.102:0/624222964 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc22c0186c0 0x7fc22c01abc0 crc :-1 s=READY pgs=1336 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:07.629 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:07.627 7fc24d12b700 1 -- 172.21.2.102:0/624222964 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc248188ab0 con 0x7fc248188490 2021-02-10T11:38:07.632 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:07.630 7fc23f7fe700 1 -- 172.21.2.102:0/624222964 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fc240005ac0 con 0x7fc248188490 2021-02-10T11:38:07.760 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:07.758 7fc24d12b700 1 -- 172.21.2.102:0/624222964 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000492282", "target": ["mgr", ""], "group_name": "group_0000000000799856"}) v1 -- 0x7fc248188890 con 0x7fc22c0186c0 2021-02-10T11:38:07.770 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:07.768 7fc23f7fe700 1 -- 172.21.2.102:0/624222964 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fc2300009b0 con 0x7fc22c0186c0 2021-02-10T11:38:07.777 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:07.775 7fc24d12b700 1 -- 172.21.2.102:0/624222964 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc22c0186c0 msgr2=0x7fc22c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:07.778 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:07.775 7fc24d12b700 1 --2- 172.21.2.102:0/624222964 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc22c0186c0 0x7fc22c01abc0 crc :-1 s=READY pgs=1336 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:07.778 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:07.775 7fc24d12b700 1 -- 172.21.2.102:0/624222964 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc248188490 msgr2=0x7fc2481910c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:07.778 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:07.775 7fc24d12b700 1 --2- 172.21.2.102:0/624222964 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc248188490 0x7fc2481910c0 secure :-1 s=READY pgs=3048 cs=0 l=1 rev1=1 rx=0x7fc24000b970 tx=0x7fc24000bd10).stop 2021-02-10T11:38:07.778 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:07.775 7fc24d12b700 1 -- 172.21.2.102:0/624222964 shutdown_connections 2021-02-10T11:38:07.778 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:07.775 7fc24d12b700 1 --2- 172.21.2.102:0/624222964 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc248188490 0x7fc2481910c0 unknown :-1 s=CLOSED pgs=3048 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:07.778 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:07.775 7fc24d12b700 1 --2- 172.21.2.102:0/624222964 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc2480ac6c0 0x7fc248187e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:07.778 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:07.775 7fc24d12b700 1 --2- 172.21.2.102:0/624222964 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc22c0186c0 0x7fc22c01abc0 unknown :-1 s=CLOSED pgs=1336 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:07.778 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:07.775 7fc24d12b700 1 --2- 172.21.2.102:0/624222964 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc2481aad00 0x7fc2480ac0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:07.778 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:07.775 7fc24d12b700 1 -- 172.21.2.102:0/624222964 >> 172.21.2.102:0/624222964 conn(0x7fc248094020 msgr2=0x7fc2481922b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:07.778 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:07.775 7fc24d12b700 1 -- 172.21.2.102:0/624222964 shutdown_connections 2021-02-10T11:38:07.778 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:07.775 7fc24d12b700 1 -- 172.21.2.102:0/624222964 wait complete. 2021-02-10T11:38:07.787 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolumegroup rm cephfs group_0000000000799856 2021-02-10T11:38:08.219 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:08.200 7f33af509700 1 Processor -- start 2021-02-10T11:38:08.219 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:08.200 7f33af509700 1 -- start start 2021-02-10T11:38:08.219 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:08.201 7f33af509700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f33a80d1860 0x7f33a81b04b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:08.220 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:08.201 7f33af509700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f33a81b0ad0 0x7f33a8108930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:08.220 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:08.201 7f33af509700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33a80fe1b0 con 0x7f33a80d1c60 2021-02-10T11:38:08.220 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:08.201 7f33af509700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33a8109a50 con 0x7f33a80d1860 2021-02-10T11:38:08.220 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:08.201 7f33af509700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33a8109740 con 0x7f33a81b0ad0 2021-02-10T11:38:08.220 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:08.201 7f33adb26700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f33a80d1c60 0x7f33a8108f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:59730/0 (socket says 172.21.2.102:59730) 2021-02-10T11:38:08.220 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:08.201 7f33adb26700 1 -- 172.21.2.102:0/2200483643 learned_addr learned my addr 172.21.2.102:0/2200483643 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:38:08.220 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:08.201 7f33ad325700 1 --2- 172.21.2.102:0/2200483643 >> v2:172.21.2.102:3300/0 conn(0x7f33a80d1860 0x7f33a81b04b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:08.220 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:08.201 7f33acb24700 1 --2- 172.21.2.102:0/2200483643 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f33a81b0ad0 0x7f33a8108930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:08.220 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:08.201 7f339bfff700 1 -- 172.21.2.102:0/2200483643 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3199085908 0 0) 0x7f33a0001500 con 0x7f33a80d1c60 2021-02-10T11:38:08.220 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:08.201 7f339bfff700 1 -- 172.21.2.102:0/2200483643 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3394002fe0 con 0x7f33a80d1c60 2021-02-10T11:38:08.220 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:08.202 7f33ad325700 1 -- 172.21.2.102:0/2200483643 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f33a81b0ad0 msgr2=0x7f33a8108930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:08.220 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:08.202 7f33ad325700 1 --2- 172.21.2.102:0/2200483643 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f33a81b0ad0 0x7f33a8108930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:08.220 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:08.202 7f33ad325700 1 -- 172.21.2.102:0/2200483643 >> v1:172.21.2.102:6789/0 conn(0x7f33a80d1c60 legacy=0x7f33a8108f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:08.221 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:08.202 7f33ad325700 1 -- 172.21.2.102:0/2200483643 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33a8088070 con 0x7f33a80d1860 2021-02-10T11:38:08.221 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:08.202 7f33ad325700 1 --2- 172.21.2.102:0/2200483643 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f33a80d1860 0x7f33a81b04b0 secure :-1 s=READY pgs=3050 cs=0 l=1 rev1=1 rx=0x7f33a4006650 tx=0x7f33a4009a80).ready entity=mon.0 client_cookie=bfb3bd3ae3ccb5e2 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:08.221 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:08.202 7f339bfff700 1 -- 172.21.2.102:0/2200483643 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f33a40009b0 con 0x7f33a80d1860 2021-02-10T11:38:08.221 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:08.202 7f339bfff700 1 -- 172.21.2.102:0/2200483643 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33a400c0c0 con 0x7f33a80d1860 2021-02-10T11:38:08.221 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:08.202 7f339bfff700 1 -- 172.21.2.102:0/2200483643 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f33a400c510 con 0x7f33a80d1860 2021-02-10T11:38:08.221 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:08.202 7f33af509700 1 -- 172.21.2.102:0/2200483643 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f33a80d1860 msgr2=0x7f33a81b04b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:08.221 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:08.202 7f33af509700 1 --2- 172.21.2.102:0/2200483643 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f33a80d1860 0x7f33a81b04b0 secure :-1 s=READY pgs=3050 cs=0 l=1 rev1=1 rx=0x7f33a4006650 tx=0x7f33a4009a80).stop 2021-02-10T11:38:08.221 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:08.202 7f33af509700 1 -- 172.21.2.102:0/2200483643 shutdown_connections 2021-02-10T11:38:08.221 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:08.202 7f33af509700 1 --2- 172.21.2.102:0/2200483643 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f33a81b0ad0 0x7f33a8108930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:08.221 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:08.202 7f33af509700 1 --2- 172.21.2.102:0/2200483643 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f33a80d1860 0x7f33a81b04b0 unknown :-1 s=CLOSED pgs=3050 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:08.221 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:08.202 7f33af509700 1 -- 172.21.2.102:0/2200483643 >> 172.21.2.102:0/2200483643 conn(0x7f33a80ce410 msgr2=0x7f33a80d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:08.221 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:08.203 7f33af509700 1 -- 172.21.2.102:0/2200483643 shutdown_connections 2021-02-10T11:38:08.221 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:08.203 7f33af509700 1 -- 172.21.2.102:0/2200483643 wait complete. 2021-02-10T11:38:08.223 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:08.217 7f33af509700 1 Processor -- start 2021-02-10T11:38:08.223 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:08.217 7f33af509700 1 -- start start 2021-02-10T11:38:08.223 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:08.218 7f33af509700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f33a80adf50 0x7f33a80ae350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:08.223 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:08.218 7f33af509700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f33a80ae970 0x7f33a80aed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:08.223 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:08.218 7f33af509700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f33a80982c0 0x7f33a80986c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:08.223 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:08.218 7f33af509700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33a809f450 con 0x7f33a80982c0 2021-02-10T11:38:08.223 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:08.218 7f33af509700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33a8098ce0 con 0x7f33a80ae970 2021-02-10T11:38:08.223 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:08.218 7f33af509700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33a8098ed0 con 0x7f33a80adf50 2021-02-10T11:38:08.224 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:08.218 7f33adb26700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f33a80982c0 0x7f33a80986c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:08.224 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:08.218 7f33adb26700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f33a80982c0 0x7f33a80986c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:39284/0 (socket says 172.21.2.102:39284) 2021-02-10T11:38:08.224 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:08.218 7f33adb26700 1 -- 172.21.2.102:0/3698837340 learned_addr learned my addr 172.21.2.102:0/3698837340 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:38:08.224 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:08.219 7f33adb26700 1 -- 172.21.2.102:0/3698837340 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f33a80adf50 msgr2=0x7f33a80ae350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:08.224 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:08.219 7f33adb26700 1 --2- 172.21.2.102:0/3698837340 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f33a80adf50 0x7f33a80ae350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:08.224 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:08.219 7f33adb26700 1 -- 172.21.2.102:0/3698837340 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f33a80ae970 msgr2=0x7f33a80aed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:08.224 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:08.219 7f33adb26700 1 --2- 172.21.2.102:0/3698837340 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f33a80ae970 0x7f33a80aed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:08.224 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:08.219 7f33adb26700 1 -- 172.21.2.102:0/3698837340 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33a80990f0 con 0x7f33a80982c0 2021-02-10T11:38:08.224 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:08.219 7f33adb26700 1 --2- 172.21.2.102:0/3698837340 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f33a80982c0 0x7f33a80986c0 secure :-1 s=READY pgs=3051 cs=0 l=1 rev1=1 rx=0x7f33a0001c10 tx=0x7f33a000aca0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:08.224 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:08.219 7f3399ffb700 1 -- 172.21.2.102:0/3698837340 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f33a0007ce0 con 0x7f33a80982c0 2021-02-10T11:38:08.224 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:08.219 7f3399ffb700 1 -- 172.21.2.102:0/3698837340 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33a00070d0 con 0x7f33a80982c0 2021-02-10T11:38:08.224 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:08.219 7f3399ffb700 1 -- 172.21.2.102:0/3698837340 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f33a0004930 con 0x7f33a80982c0 2021-02-10T11:38:08.224 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:08.219 7f33af509700 1 -- 172.21.2.102:0/3698837340 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33a80990f0 con 0x7f33a80982c0 2021-02-10T11:38:08.224 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:08.219 7f33af509700 1 -- 172.21.2.102:0/3698837340 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33a8099680 con 0x7f33a80982c0 2021-02-10T11:38:08.225 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:08.220 7f3399ffb700 1 -- 172.21.2.102:0/3698837340 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f33a00072b0 con 0x7f33a80982c0 2021-02-10T11:38:08.225 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:08.220 7f3399ffb700 1 --2- 172.21.2.102:0/3698837340 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f33880186c0 0x7f338801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:08.225 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:08.220 7f3399ffb700 1 -- 172.21.2.102:0/3698837340 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f33a0032950 con 0x7f33a80982c0 2021-02-10T11:38:08.225 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:08.220 7f33ad325700 1 --2- 172.21.2.102:0/3698837340 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f33880186c0 0x7f338801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:08.225 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:08.220 7f33ad325700 1 --2- 172.21.2.102:0/3698837340 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f33880186c0 0x7f338801abc0 crc :-1 s=READY pgs=1337 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:08.252 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:08.250 7f33af509700 1 -- 172.21.2.102:0/3698837340 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33a8082a20 con 0x7f33a80982c0 2021-02-10T11:38:08.255 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:08.253 7f3399ffb700 1 -- 172.21.2.102:0/3698837340 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f33a0000f70 con 0x7f33a80982c0 2021-02-10T11:38:08.384 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:08.382 7f33af509700 1 -- 172.21.2.102:0/3698837340 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolumegroup rm", "vol_name": "cephfs", "target": ["mgr", ""], "group_name": "group_0000000000799856"}) v1 -- 0x7f33a8099680 con 0x7f33880186c0 2021-02-10T11:38:08.388 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:08.386 7f3399ffb700 1 -- 172.21.2.102:0/3698837340 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f33a4000940 con 0x7f33880186c0 2021-02-10T11:38:08.393 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:08.391 7f33af509700 1 -- 172.21.2.102:0/3698837340 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f33880186c0 msgr2=0x7f338801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:08.393 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:08.391 7f33af509700 1 --2- 172.21.2.102:0/3698837340 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f33880186c0 0x7f338801abc0 crc :-1 s=READY pgs=1337 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:08.393 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:08.391 7f33af509700 1 -- 172.21.2.102:0/3698837340 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f33a80982c0 msgr2=0x7f33a80986c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:08.393 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:08.391 7f33af509700 1 --2- 172.21.2.102:0/3698837340 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f33a80982c0 0x7f33a80986c0 secure :-1 s=READY pgs=3051 cs=0 l=1 rev1=1 rx=0x7f33a0001c10 tx=0x7f33a000aca0).stop 2021-02-10T11:38:08.394 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:08.391 7f33af509700 1 -- 172.21.2.102:0/3698837340 shutdown_connections 2021-02-10T11:38:08.394 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:08.391 7f33af509700 1 --2- 172.21.2.102:0/3698837340 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f33a80982c0 0x7f33a80986c0 unknown :-1 s=CLOSED pgs=3051 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:08.394 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:08.391 7f33af509700 1 --2- 172.21.2.102:0/3698837340 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f33880186c0 0x7f338801abc0 unknown :-1 s=CLOSED pgs=1337 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:08.394 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:08.391 7f33af509700 1 --2- 172.21.2.102:0/3698837340 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f33a80ae970 0x7f33a80aed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:08.394 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:08.391 7f33af509700 1 --2- 172.21.2.102:0/3698837340 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f33a80adf50 0x7f33a80ae350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:08.394 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:08.391 7f33af509700 1 -- 172.21.2.102:0/3698837340 >> 172.21.2.102:0/3698837340 conn(0x7f33a80bee90 msgr2=0x7f33a80bb880 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:08.394 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:08.391 7f33af509700 1 -- 172.21.2.102:0/3698837340 shutdown_connections 2021-02-10T11:38:08.394 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:08.391 7f33af509700 1 -- 172.21.2.102:0/3698837340 wait complete. 2021-02-10T11:38:08.402 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2021-02-10T11:38:08.402 DEBUG:teuthology.orchestra.run.gibba002:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:38:08.747 DEBUG:teuthology.orchestra.run.gibba013:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:38:09.090 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2021-02-10T11:38:09.090 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:38:09.091 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:38:09.091 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:38:09.427 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:38:09.427 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:38:09.427 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@gibba002.front.sepia.ceph.com... 2021-02-10T11:38:09.428 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-02-10T11:38:09.428 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:38:09.779 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba002.stderr:ceph-fuse[208373]: fuse finished with error 0 and tester_r 0 2021-02-10T11:38:09.781 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:38:09.781 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:38:09.781 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:38:10.116 INFO:teuthology.orchestra.run.gibba002.stdout:ext2/ext3 2021-02-10T11:38:10.117 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2021-02-10T11:38:10.117 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:38:10.117 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:38:10.453 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:38:10.453 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:38:10.453 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:38:10.789 INFO:teuthology.orchestra.run.gibba002.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.1’: No such file or directory 2021-02-10T11:38:10.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:38:10.790 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2021-02-10T11:38:10.790 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:38:10.790 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:38:11.126 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:38:11.126 DEBUG:teuthology.orchestra.run.gibba013:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:38:11.126 DEBUG:teuthology.orchestra.run.gibba013:> ' -- /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:38:11.462 INFO:teuthology.orchestra.run.gibba013.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.2’: No such file or directory 2021-02-10T11:38:11.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:38:11.462 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2021-02-10T11:38:11.462 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:38:11.462 DEBUG:teuthology.orchestra.run.gibba013:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:38:11.797 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:38:11.798 DEBUG:teuthology.orchestra.run.gibba013:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:38:11.798 DEBUG:teuthology.orchestra.run.gibba013:> ' -- /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:38:12.131 INFO:teuthology.orchestra.run.gibba013.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.3’: No such file or directory 2021-02-10T11:38:12.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:38:12.132 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.3 2021-02-10T11:38:12.132 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:38:12.132 DEBUG:teuthology.orchestra.run.gibba013:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:38:12.466 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config rm mon mon_allow_pool_delete 2021-02-10T11:38:12.901 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:12.883 7fb9de268700 1 Processor -- start 2021-02-10T11:38:12.902 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:12.883 7fb9de268700 1 -- start start 2021-02-10T11:38:12.902 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:12.883 7fb9de268700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fb9d80d1860 0x7fb9d81b03e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:12.902 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:12.883 7fb9de268700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fb9d81b0a00 0x7fb9d80cf320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:12.902 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:12.883 7fb9de268700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9d80f0b00 con 0x7fb9d80d1c60 2021-02-10T11:38:12.902 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:12.883 7fb9de268700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9d80e5ec0 con 0x7fb9d81b0a00 2021-02-10T11:38:12.902 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:12.883 7fb9de268700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9d80f0640 con 0x7fb9d80d1860 2021-02-10T11:38:12.902 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:12.883 7fb9d77fe700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fb9d81b0a00 0x7fb9d80cf320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:12.902 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:12.884 7fb9d77fe700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fb9d81b0a00 0x7fb9d80cf320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:39288/0 (socket says 172.21.2.102:39288) 2021-02-10T11:38:12.902 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:12.884 7fb9d77fe700 1 -- 172.21.2.102:0/3288097703 learned_addr learned my addr 172.21.2.102:0/3288097703 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:38:12.902 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:12.884 7fb9d77fe700 1 -- 172.21.2.102:0/3288097703 >> v2:172.21.2.113:3300/0 conn(0x7fb9d80d1860 msgr2=0x7fb9d81b03e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:12.903 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:12.884 7fb9d77fe700 1 --2- 172.21.2.102:0/3288097703 >> v2:172.21.2.113:3300/0 conn(0x7fb9d80d1860 0x7fb9d81b03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:12.903 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:12.884 7fb9d77fe700 1 -- 172.21.2.102:0/3288097703 >> v1:172.21.2.113:6789/0 conn(0x7fb9d80d1c60 legacy=0x7fb9d80cf940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:12.903 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:12.884 7fb9d77fe700 1 -- 172.21.2.102:0/3288097703 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9d80f0830 con 0x7fb9d81b0a00 2021-02-10T11:38:12.903 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:12.884 7fb9d77fe700 1 --2- 172.21.2.102:0/3288097703 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb9d81b0a00 0x7fb9d80cf320 secure :-1 s=READY pgs=3052 cs=0 l=1 rev1=1 rx=0x7fb9c4006700 tx=0x7fb9c4009b80).ready entity=mon.0 client_cookie=2c95681d5777e20f server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:12.903 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:12.884 7fb9d6ffd700 1 -- 172.21.2.102:0/3288097703 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb9c400c0c0 con 0x7fb9d81b0a00 2021-02-10T11:38:12.903 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:12.884 7fb9d6ffd700 1 -- 172.21.2.102:0/3288097703 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9c400c340 con 0x7fb9d81b0a00 2021-02-10T11:38:12.903 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:12.884 7fb9d6ffd700 1 -- 172.21.2.102:0/3288097703 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb9c400c7a0 con 0x7fb9d81b0a00 2021-02-10T11:38:12.903 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:12.884 7fb9de268700 1 -- 172.21.2.102:0/3288097703 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb9d81b0a00 msgr2=0x7fb9d80cf320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:12.903 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:12.884 7fb9de268700 1 --2- 172.21.2.102:0/3288097703 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb9d81b0a00 0x7fb9d80cf320 secure :-1 s=READY pgs=3052 cs=0 l=1 rev1=1 rx=0x7fb9c4006700 tx=0x7fb9c4009b80).stop 2021-02-10T11:38:12.903 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:12.885 7fb9de268700 1 -- 172.21.2.102:0/3288097703 shutdown_connections 2021-02-10T11:38:12.903 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:12.885 7fb9de268700 1 --2- 172.21.2.102:0/3288097703 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb9d81b0a00 0x7fb9d80cf320 unknown :-1 s=CLOSED pgs=3052 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:12.903 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:12.885 7fb9de268700 1 --2- 172.21.2.102:0/3288097703 >> v2:172.21.2.113:3300/0 conn(0x7fb9d80d1860 0x7fb9d81b03e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:12.903 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:12.885 7fb9de268700 1 -- 172.21.2.102:0/3288097703 >> 172.21.2.102:0/3288097703 conn(0x7fb9d80ce440 msgr2=0x7fb9d80d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:12.904 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:12.885 7fb9de268700 1 -- 172.21.2.102:0/3288097703 shutdown_connections 2021-02-10T11:38:12.904 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:12.885 7fb9de268700 1 -- 172.21.2.102:0/3288097703 wait complete. 2021-02-10T11:38:12.905 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:12.900 7fb9de268700 1 Processor -- start 2021-02-10T11:38:12.905 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:12.900 7fb9de268700 1 -- start start 2021-02-10T11:38:12.905 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:12.900 7fb9de268700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb9d80a4650 0x7fb9d80a4a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:12.905 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:12.900 7fb9de268700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb9d80a51d0 0x7fb9d80a55d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:12.905 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:12.900 7fb9de268700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb9d80a5bf0 0x7fb9d80a0510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:12.906 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:12.900 7fb9de268700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9d80a6100 con 0x7fb9d80a4650 2021-02-10T11:38:12.906 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:12.900 7fb9de268700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9d80a0b30 con 0x7fb9d80a5bf0 2021-02-10T11:38:12.906 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:12.900 7fb9de268700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9d80a0d50 con 0x7fb9d80a51d0 2021-02-10T11:38:12.906 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:12.901 7fb9d7fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb9d80a4650 0x7fb9d80a4a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:12.906 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:12.901 7fb9d7fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb9d80a4650 0x7fb9d80a4a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:39294/0 (socket says 172.21.2.102:39294) 2021-02-10T11:38:12.906 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:12.901 7fb9d7fff700 1 -- 172.21.2.102:0/4187558410 learned_addr learned my addr 172.21.2.102:0/4187558410 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:38:12.906 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:12.901 7fb9d7fff700 1 -- 172.21.2.102:0/4187558410 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb9d80a51d0 msgr2=0x7fb9d80a55d0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:38:12.906 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:12.901 7fb9d7fff700 1 --2- 172.21.2.102:0/4187558410 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb9d80a51d0 0x7fb9d80a55d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:12.906 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:12.901 7fb9d7fff700 1 -- 172.21.2.102:0/4187558410 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb9d80a5bf0 msgr2=0x7fb9d80a0510 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:38:12.906 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:12.901 7fb9d7fff700 1 --2- 172.21.2.102:0/4187558410 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb9d80a5bf0 0x7fb9d80a0510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:12.906 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:12.901 7fb9d7fff700 1 -- 172.21.2.102:0/4187558410 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9d80a0f70 con 0x7fb9d80a4650 2021-02-10T11:38:12.906 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:12.901 7fb9d7fff700 1 --2- 172.21.2.102:0/4187558410 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb9d80a4650 0x7fb9d80a4a50 secure :-1 s=READY pgs=3053 cs=0 l=1 rev1=1 rx=0x7fb9c8006f00 tx=0x7fb9c800b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:12.906 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:12.901 7fb9d4ff9700 1 -- 172.21.2.102:0/4187558410 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb9c8006b90 con 0x7fb9d80a4650 2021-02-10T11:38:12.907 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:12.901 7fb9d4ff9700 1 -- 172.21.2.102:0/4187558410 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9c800bd30 con 0x7fb9d80a4650 2021-02-10T11:38:12.907 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:12.901 7fb9de268700 1 -- 172.21.2.102:0/4187558410 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9d80a0f70 con 0x7fb9d80a4650 2021-02-10T11:38:12.907 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:12.901 7fb9de268700 1 -- 172.21.2.102:0/4187558410 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9d80a1500 con 0x7fb9d80a4650 2021-02-10T11:38:12.907 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:12.901 7fb9d4ff9700 1 -- 172.21.2.102:0/4187558410 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb9c80020d0 con 0x7fb9d80a4650 2021-02-10T11:38:12.907 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:12.902 7fb9d4ff9700 1 -- 172.21.2.102:0/4187558410 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fb9c8007000 con 0x7fb9d80a4650 2021-02-10T11:38:12.907 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:12.902 7fb9d4ff9700 1 --2- 172.21.2.102:0/4187558410 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb9b40186c0 0x7fb9b401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:12.907 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:12.902 7fb9d4ff9700 1 -- 172.21.2.102:0/4187558410 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fb9c8032560 con 0x7fb9d80a4650 2021-02-10T11:38:12.907 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:12.902 7fb9d77fe700 1 --2- 172.21.2.102:0/4187558410 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb9b40186c0 0x7fb9b401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:12.907 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:12.903 7fb9d77fe700 1 --2- 172.21.2.102:0/4187558410 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb9b40186c0 0x7fb9b401abc0 crc :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:12.935 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:12.933 7fb9de268700 1 -- 172.21.2.102:0/4187558410 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9d80ee000 con 0x7fb9d80a4650 2021-02-10T11:38:12.938 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:12.936 7fb9d4ff9700 1 -- 172.21.2.102:0/4187558410 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fb9c8000940 con 0x7fb9d80a4650 2021-02-10T11:38:13.087 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:13.085 7fb9de268700 1 -- 172.21.2.102:0/4187558410 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fb9d80ce440 con 0x7fb9d80a4650 2021-02-10T11:38:13.091 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:13.089 7fb9d4ff9700 1 -- 172.21.2.102:0/4187558410 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v43) v1 ==== 104+0+0 (secure 0 0 0) 0x7fb9c8007000 con 0x7fb9d80a4650 2021-02-10T11:38:13.096 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:13.094 7fb9de268700 1 -- 172.21.2.102:0/4187558410 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb9b40186c0 msgr2=0x7fb9b401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:13.096 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:13.094 7fb9de268700 1 --2- 172.21.2.102:0/4187558410 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb9b40186c0 0x7fb9b401abc0 crc :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:13.096 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:13.094 7fb9de268700 1 -- 172.21.2.102:0/4187558410 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb9d80a4650 msgr2=0x7fb9d80a4a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:13.096 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:13.094 7fb9de268700 1 --2- 172.21.2.102:0/4187558410 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb9d80a4650 0x7fb9d80a4a50 secure :-1 s=READY pgs=3053 cs=0 l=1 rev1=1 rx=0x7fb9c8006f00 tx=0x7fb9c800b940).stop 2021-02-10T11:38:13.096 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:13.094 7fb9de268700 1 -- 172.21.2.102:0/4187558410 shutdown_connections 2021-02-10T11:38:13.096 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:13.094 7fb9de268700 1 --2- 172.21.2.102:0/4187558410 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb9b40186c0 0x7fb9b401abc0 unknown :-1 s=CLOSED pgs=1338 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:13.096 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:13.094 7fb9de268700 1 --2- 172.21.2.102:0/4187558410 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb9d80a5bf0 0x7fb9d80a0510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:13.097 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:13.094 7fb9de268700 1 --2- 172.21.2.102:0/4187558410 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb9d80a51d0 0x7fb9d80a55d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:13.097 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:13.094 7fb9de268700 1 --2- 172.21.2.102:0/4187558410 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb9d80a4650 0x7fb9d80a4a50 unknown :-1 s=CLOSED pgs=3053 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:13.097 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:13.094 7fb9de268700 1 -- 172.21.2.102:0/4187558410 >> 172.21.2.102:0/4187558410 conn(0x7fb9d808efb0 msgr2=0x7fb9d808e690 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:13.097 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:13.094 7fb9de268700 1 -- 172.21.2.102:0/4187558410 shutdown_connections 2021-02-10T11:38:13.097 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:13.094 7fb9de268700 1 -- 172.21.2.102:0/4187558410 wait complete. 2021-02-10T11:38:13.105 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestVolumes.test_recover_auth_metadata_during_authorize' 2021-02-10T11:38:13.535 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:13.515 7fd0c3b0c700 1 Processor -- start 2021-02-10T11:38:13.535 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:13.515 7fd0c3b0c700 1 -- start start 2021-02-10T11:38:13.536 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:13.516 7fd0c3b0c700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fd0bc0d1870 0x7fd0bc1b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:13.536 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:13.516 7fd0c3b0c700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd0bc0f0650 0x7fd0bc0cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:13.536 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:13.516 7fd0c3b0c700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0bc0f0b10 con 0x7fd0bc1b0a10 2021-02-10T11:38:13.536 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:13.516 7fd0c3b0c700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0bc0e5ed0 con 0x7fd0bc0d1870 2021-02-10T11:38:13.536 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:13.516 7fd0c3b0c700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0bc0d1c70 con 0x7fd0bc0f0650 2021-02-10T11:38:13.536 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:13.517 7fd0c2129700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd0bc0f0650 0x7fd0bc0cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:13.536 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:13.517 7fd0c1928700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fd0bc0d1870 0x7fd0bc1b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:13.536 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:13.517 7fd0c1127700 1 --1- >> v1:172.21.2.113:6789/0 conn(0x7fd0bc1b0a10 0x7fd0bc0cf330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.113:6789/0 says I am v1:172.21.2.102:38850/0 (socket says 172.21.2.102:38850) 2021-02-10T11:38:13.536 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:13.517 7fd0c1127700 1 -- 172.21.2.102:0/3182466560 learned_addr learned my addr 172.21.2.102:0/3182466560 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:38:13.536 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:13.518 7fd0c2129700 1 -- 172.21.2.102:0/3182466560 >> v2:172.21.2.113:3300/0 conn(0x7fd0bc0d1870 msgr2=0x7fd0bc1b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:13.536 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:13.518 7fd0c2129700 1 --2- 172.21.2.102:0/3182466560 >> v2:172.21.2.113:3300/0 conn(0x7fd0bc0d1870 0x7fd0bc1b03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:13.536 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:13.518 7fd0c2129700 1 -- 172.21.2.102:0/3182466560 >> v1:172.21.2.113:6789/0 conn(0x7fd0bc1b0a10 legacy=0x7fd0bc0cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:13.537 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:13.518 7fd0c0926700 1 -- 172.21.2.102:0/3182466560 <== mon.1 v1:172.21.2.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3423335343 0 0) 0x7fd0b80010b0 con 0x7fd0bc1b0a10 2021-02-10T11:38:13.537 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:13.518 7fd0c2129700 1 -- 172.21.2.102:0/3182466560 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0bc0d1e90 con 0x7fd0bc0f0650 2021-02-10T11:38:13.537 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:13.518 7fd0c2129700 1 --2- 172.21.2.102:0/3182466560 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd0bc0f0650 0x7fd0bc0cfb20 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 rx=0x7fd0b4006e50 tx=0x7fd0b4009df0).ready entity=mon.2 client_cookie=a79e4420709f0c05 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:13.537 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:13.518 7fd0c0926700 1 -- 172.21.2.102:0/3182466560 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd0b400b510 con 0x7fd0bc0f0650 2021-02-10T11:38:13.537 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:13.518 7fd0c0926700 1 -- 172.21.2.102:0/3182466560 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0b4003030 con 0x7fd0bc0f0650 2021-02-10T11:38:13.537 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:13.518 7fd0c0926700 1 -- 172.21.2.102:0/3182466560 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd0b4000f90 con 0x7fd0bc0f0650 2021-02-10T11:38:13.537 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:13.518 7fd0c3b0c700 1 -- 172.21.2.102:0/3182466560 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd0bc0f0650 msgr2=0x7fd0bc0cfb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:13.537 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:13.518 7fd0c3b0c700 1 --2- 172.21.2.102:0/3182466560 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd0bc0f0650 0x7fd0bc0cfb20 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 rx=0x7fd0b4006e50 tx=0x7fd0b4009df0).stop 2021-02-10T11:38:13.537 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:13.519 7fd0c3b0c700 1 -- 172.21.2.102:0/3182466560 shutdown_connections 2021-02-10T11:38:13.537 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:13.519 7fd0c3b0c700 1 --2- 172.21.2.102:0/3182466560 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd0bc0f0650 0x7fd0bc0cfb20 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:13.537 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:13.519 7fd0c3b0c700 1 --2- 172.21.2.102:0/3182466560 >> v2:172.21.2.113:3300/0 conn(0x7fd0bc0d1870 0x7fd0bc1b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:13.537 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:13.519 7fd0c3b0c700 1 -- 172.21.2.102:0/3182466560 >> 172.21.2.102:0/3182466560 conn(0x7fd0bc0ce450 msgr2=0x7fd0bc0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:13.537 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:13.519 7fd0c3b0c700 1 -- 172.21.2.102:0/3182466560 shutdown_connections 2021-02-10T11:38:13.538 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:13.519 7fd0c3b0c700 1 -- 172.21.2.102:0/3182466560 wait complete. 2021-02-10T11:38:13.539 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:13.534 7fd0c3b0c700 1 Processor -- start 2021-02-10T11:38:13.539 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:13.534 7fd0c3b0c700 1 -- start start 2021-02-10T11:38:13.539 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:13.534 7fd0c3b0c700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd0bc195650 0x7fd0bc0b5560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:13.539 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:13.534 7fd0c3b0c700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd0bc0b5b80 0x7fd0bc1a2e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:13.540 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:13.534 7fd0c3b0c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd0bc0acfa0 0x7fd0bc1a3440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:13.540 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:13.534 7fd0c3b0c700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0bc0b6010 con 0x7fd0bc0acfa0 2021-02-10T11:38:13.540 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:13.534 7fd0c3b0c700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0bc1a3a60 con 0x7fd0bc0b5b80 2021-02-10T11:38:13.540 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:13.534 7fd0c3b0c700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0bc1a3c40 con 0x7fd0bc195650 2021-02-10T11:38:13.540 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:13.534 7fd0c2129700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd0bc0acfa0 0x7fd0bc1a3440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:13.540 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:13.534 7fd0c2129700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd0bc0acfa0 0x7fd0bc1a3440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:39308/0 (socket says 172.21.2.102:39308) 2021-02-10T11:38:13.540 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:13.534 7fd0c2129700 1 -- 172.21.2.102:0/2479230776 learned_addr learned my addr 172.21.2.102:0/2479230776 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:38:13.540 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:13.535 7fd0c2129700 1 -- 172.21.2.102:0/2479230776 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd0bc195650 msgr2=0x7fd0bc0b5560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:13.540 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:13.535 7fd0c2129700 1 --2- 172.21.2.102:0/2479230776 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd0bc195650 0x7fd0bc0b5560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:13.540 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:13.535 7fd0c2129700 1 -- 172.21.2.102:0/2479230776 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd0bc0b5b80 msgr2=0x7fd0bc1a2e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:13.540 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:13.535 7fd0c2129700 1 --2- 172.21.2.102:0/2479230776 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd0bc0b5b80 0x7fd0bc1a2e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:13.540 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:13.535 7fd0c2129700 1 -- 172.21.2.102:0/2479230776 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0bc0bcae0 con 0x7fd0bc0acfa0 2021-02-10T11:38:13.540 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:13.535 7fd0c2129700 1 --2- 172.21.2.102:0/2479230776 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd0bc0acfa0 0x7fd0bc1a3440 secure :-1 s=READY pgs=3054 cs=0 l=1 rev1=1 rx=0x7fd0b4006fa0 tx=0x7fd0b400bd90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:13.541 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:13.535 7fd0ae7fc700 1 -- 172.21.2.102:0/2479230776 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd0b40093d0 con 0x7fd0bc0acfa0 2021-02-10T11:38:13.541 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:13.535 7fd0ae7fc700 1 -- 172.21.2.102:0/2479230776 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0b40082e0 con 0x7fd0bc0acfa0 2021-02-10T11:38:13.541 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:13.535 7fd0c3b0c700 1 -- 172.21.2.102:0/2479230776 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0bc0bcae0 con 0x7fd0bc0acfa0 2021-02-10T11:38:13.541 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:13.535 7fd0ae7fc700 1 -- 172.21.2.102:0/2479230776 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd0b4008800 con 0x7fd0bc0acfa0 2021-02-10T11:38:13.541 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:13.535 7fd0c3b0c700 1 -- 172.21.2.102:0/2479230776 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0bc0bcae0 con 0x7fd0bc0acfa0 2021-02-10T11:38:13.541 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:13.536 7fd0ae7fc700 1 -- 172.21.2.102:0/2479230776 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd0b40084c0 con 0x7fd0bc0acfa0 2021-02-10T11:38:13.541 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:13.536 7fd0ae7fc700 1 --2- 172.21.2.102:0/2479230776 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd0a40186c0 0x7fd0a401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:13.541 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:13.536 7fd0ae7fc700 1 -- 172.21.2.102:0/2479230776 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fd0b4032b70 con 0x7fd0bc0acfa0 2021-02-10T11:38:13.541 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:13.536 7fd0c1928700 1 --2- 172.21.2.102:0/2479230776 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd0a40186c0 0x7fd0a401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:13.541 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:13.536 7fd0c1928700 1 --2- 172.21.2.102:0/2479230776 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd0a40186c0 0x7fd0a401abc0 crc :-1 s=READY pgs=1339 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:13.568 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:13.566 7fd0c3b0c700 1 -- 172.21.2.102:0/2479230776 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0bc0b6010 con 0x7fd0bc0acfa0 2021-02-10T11:38:13.571 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:13.569 7fd0ae7fc700 1 -- 172.21.2.102:0/2479230776 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd0b4008090 con 0x7fd0bc0acfa0 2021-02-10T11:38:13.703 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:13.701 7fd0c3b0c700 1 -- 172.21.2.102:0/2479230776 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestVolumes.test_recover_auth_metadata_during_authorize"]} v 0) v1 -- 0x7fd0bc0b6010 con 0x7fd0bc0acfa0 2021-02-10T11:38:14.187 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:14.185 7fd0ae7fc700 1 -- 172.21.2.102:0/2479230776 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestVolumes.test_recover_auth_metadata_during_authorize"]}]=0 v838) v1 ==== 160+0+0 (secure 0 0 0) 0x7fd0b40082a0 con 0x7fd0bc0acfa0 2021-02-10T11:38:14.224 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:14.221 7fd0c3b0c700 1 -- 172.21.2.102:0/2479230776 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd0a40186c0 msgr2=0x7fd0a401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:14.224 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:14.221 7fd0c3b0c700 1 --2- 172.21.2.102:0/2479230776 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd0a40186c0 0x7fd0a401abc0 crc :-1 s=READY pgs=1339 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:14.225 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:14.221 7fd0c3b0c700 1 -- 172.21.2.102:0/2479230776 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd0bc0acfa0 msgr2=0x7fd0bc1a3440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:14.225 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:14.221 7fd0c3b0c700 1 --2- 172.21.2.102:0/2479230776 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd0bc0acfa0 0x7fd0bc1a3440 secure :-1 s=READY pgs=3054 cs=0 l=1 rev1=1 rx=0x7fd0b4006fa0 tx=0x7fd0b400bd90).stop 2021-02-10T11:38:14.225 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:14.221 7fd0c3b0c700 1 -- 172.21.2.102:0/2479230776 shutdown_connections 2021-02-10T11:38:14.225 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:14.221 7fd0c3b0c700 1 --2- 172.21.2.102:0/2479230776 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd0bc0acfa0 0x7fd0bc1a3440 unknown :-1 s=CLOSED pgs=3054 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:14.225 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:14.221 7fd0c3b0c700 1 --2- 172.21.2.102:0/2479230776 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd0a40186c0 0x7fd0a401abc0 unknown :-1 s=CLOSED pgs=1339 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:14.226 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:14.221 7fd0c3b0c700 1 --2- 172.21.2.102:0/2479230776 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd0bc0b5b80 0x7fd0bc1a2e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:14.226 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:14.221 7fd0c3b0c700 1 --2- 172.21.2.102:0/2479230776 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd0bc195650 0x7fd0bc0b5560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:14.226 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:14.221 7fd0c3b0c700 1 -- 172.21.2.102:0/2479230776 >> 172.21.2.102:0/2479230776 conn(0x7fd0bc0b3040 msgr2=0x7fd0bc0c3250 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:14.226 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:14.221 7fd0c3b0c700 1 -- 172.21.2.102:0/2479230776 shutdown_connections 2021-02-10T11:38:14.226 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:14.221 7fd0c3b0c700 1 -- 172.21.2.102:0/2479230776 wait complete. 2021-02-10T11:38:14.232 INFO:tasks.cephfs_test_runner:test_recover_auth_metadata_during_authorize (tasks.cephfs.test_volumes.TestVolumes) ... ok 2021-02-10T11:38:14.232 INFO:tasks.cephfs_test_runner:Starting test: test_recover_auth_metadata_during_deauthorize (tasks.cephfs.test_volumes.TestVolumes) 2021-02-10T11:38:14.233 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestVolumes.test_recover_auth_metadata_during_deauthorize' 2021-02-10T11:38:14.447 DEBUG:teuthology.orchestra.run.gibba002:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:38:14.451 DEBUG:teuthology.orchestra.run.gibba013:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:38:14.660 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:14.642 7f517f456700 1 Processor -- start 2021-02-10T11:38:14.661 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:14.642 7f517f456700 1 -- start start 2021-02-10T11:38:14.661 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:14.643 7f517f456700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f51780d1870 0x7f51781b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:14.661 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:14.643 7f517f456700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f51780f0650 0x7f51780cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:14.661 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:14.643 7f517f456700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51780f0b10 con 0x7f51781b0a10 2021-02-10T11:38:14.661 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:14.643 7f517f456700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51780e5ed0 con 0x7f51780d1870 2021-02-10T11:38:14.661 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:14.643 7f517f456700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51780d1c70 con 0x7f51780f0650 2021-02-10T11:38:14.661 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:14.643 7f517d272700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f51780d1870 0x7f51781b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:14.661 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:14.643 7f517d272700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f51780d1870 0x7f51781b03f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:39312/0 (socket says 172.21.2.102:39312) 2021-02-10T11:38:14.661 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:14.643 7f517d272700 1 -- 172.21.2.102:0/1840608153 learned_addr learned my addr 172.21.2.102:0/1840608153 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:38:14.661 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:14.643 7f517d272700 1 -- 172.21.2.102:0/1840608153 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f51780f0650 msgr2=0x7f51780cfb20 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:38:14.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:14.643 7f517d272700 1 --2- 172.21.2.102:0/1840608153 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f51780f0650 0x7f51780cfb20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:14.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:14.643 7f517d272700 1 -- 172.21.2.102:0/1840608153 >> v1:172.21.2.113:6789/0 conn(0x7f51781b0a10 legacy=0x7f51780cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:14.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:14.644 7f517d272700 1 -- 172.21.2.102:0/1840608153 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51780d1e90 con 0x7f51780d1870 2021-02-10T11:38:14.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:14.644 7f517d272700 1 --2- 172.21.2.102:0/1840608153 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f51780d1870 0x7f51781b03f0 secure :-1 s=READY pgs=3055 cs=0 l=1 rev1=1 rx=0x7f5168006700 tx=0x7f5168009b00).ready entity=mon.0 client_cookie=611b7b90222e137a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:14.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:14.644 7f516ffff700 1 -- 172.21.2.102:0/1840608153 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f516800c0c0 con 0x7f51780d1870 2021-02-10T11:38:14.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:14.644 7f516ffff700 1 -- 172.21.2.102:0/1840608153 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f516800c400 con 0x7f51780d1870 2021-02-10T11:38:14.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:14.644 7f516ffff700 1 -- 172.21.2.102:0/1840608153 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f516800c8e0 con 0x7f51780d1870 2021-02-10T11:38:14.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:14.644 7f517f456700 1 -- 172.21.2.102:0/1840608153 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f51780d1870 msgr2=0x7f51781b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:14.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:14.644 7f517f456700 1 --2- 172.21.2.102:0/1840608153 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f51780d1870 0x7f51781b03f0 secure :-1 s=READY pgs=3055 cs=0 l=1 rev1=1 rx=0x7f5168006700 tx=0x7f5168009b00).stop 2021-02-10T11:38:14.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:14.644 7f517f456700 1 -- 172.21.2.102:0/1840608153 shutdown_connections 2021-02-10T11:38:14.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:14.644 7f517f456700 1 --2- 172.21.2.102:0/1840608153 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f51780f0650 0x7f51780cfb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:14.663 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:14.644 7f517f456700 1 --2- 172.21.2.102:0/1840608153 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f51780d1870 0x7f51781b03f0 unknown :-1 s=CLOSED pgs=3055 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:14.663 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:14.644 7f517f456700 1 -- 172.21.2.102:0/1840608153 >> 172.21.2.102:0/1840608153 conn(0x7f51780ce450 msgr2=0x7f51780d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:14.663 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:14.644 7f517f456700 1 -- 172.21.2.102:0/1840608153 shutdown_connections 2021-02-10T11:38:14.663 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:14.644 7f517f456700 1 -- 172.21.2.102:0/1840608153 wait complete. 2021-02-10T11:38:14.663 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:14.659 7f517f456700 1 Processor -- start 2021-02-10T11:38:14.663 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:14.659 7f517f456700 1 -- start start 2021-02-10T11:38:14.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:14.659 7f517f456700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f51780ae970 0x7f5178094bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:14.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:14.659 7f517f456700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f51780951d0 0x7f51780eeb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:14.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:14.659 7f517f456700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f51780ef140 0x7f51780ef540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:14.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:14.659 7f517f456700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f517819ccb0 con 0x7f51780ef140 2021-02-10T11:38:14.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:14.659 7f517f456700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f517819d230 con 0x7f51780951d0 2021-02-10T11:38:14.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:14.659 7f517f456700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f517819d410 con 0x7f51780ae970 2021-02-10T11:38:14.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:14.659 7f517da73700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f51780ef140 0x7f51780ef540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:14.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:14.659 7f517da73700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f51780ef140 0x7f51780ef540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:39314/0 (socket says 172.21.2.102:39314) 2021-02-10T11:38:14.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:14.659 7f517da73700 1 -- 172.21.2.102:0/366600096 learned_addr learned my addr 172.21.2.102:0/366600096 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:38:14.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:14.659 7f517da73700 1 -- 172.21.2.102:0/366600096 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f51780ae970 msgr2=0x7f5178094bb0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:38:14.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:14.660 7f517da73700 1 --2- 172.21.2.102:0/366600096 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f51780ae970 0x7f5178094bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:14.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:14.660 7f517da73700 1 -- 172.21.2.102:0/366600096 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f51780951d0 msgr2=0x7f51780eeb20 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:38:14.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:14.660 7f517da73700 1 --2- 172.21.2.102:0/366600096 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f51780951d0 0x7f51780eeb20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:14.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:14.660 7f517da73700 1 -- 172.21.2.102:0/366600096 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f517819d630 con 0x7f51780ef140 2021-02-10T11:38:14.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:14.660 7f517da73700 1 --2- 172.21.2.102:0/366600096 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f51780ef140 0x7f51780ef540 secure :-1 s=READY pgs=3056 cs=0 l=1 rev1=1 rx=0x7f517000b5c0 tx=0x7f517000b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:14.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:14.660 7f516dffb700 1 -- 172.21.2.102:0/366600096 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f517000bd50 con 0x7f51780ef140 2021-02-10T11:38:14.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:14.660 7f516dffb700 1 -- 172.21.2.102:0/366600096 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5170006030 con 0x7f51780ef140 2021-02-10T11:38:14.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:14.660 7f517f456700 1 -- 172.21.2.102:0/366600096 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f517819d630 con 0x7f51780ef140 2021-02-10T11:38:14.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:14.660 7f517f456700 1 -- 172.21.2.102:0/366600096 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f517819d630 con 0x7f51780ef140 2021-02-10T11:38:14.666 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:14.660 7f516dffb700 1 -- 172.21.2.102:0/366600096 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f51700054d0 con 0x7f51780ef140 2021-02-10T11:38:14.666 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:14.660 7f516dffb700 1 -- 172.21.2.102:0/366600096 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f5170006210 con 0x7f51780ef140 2021-02-10T11:38:14.666 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:14.661 7f516dffb700 1 --2- 172.21.2.102:0/366600096 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f51580186d0 0x7f515801abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:14.666 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:14.661 7f516dffb700 1 -- 172.21.2.102:0/366600096 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f51700327a0 con 0x7f51780ef140 2021-02-10T11:38:14.666 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:14.661 7f517d272700 1 --2- 172.21.2.102:0/366600096 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f51580186d0 0x7f515801abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:14.666 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:14.661 7f517f456700 1 -- 172.21.2.102:0/366600096 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51780edce0 con 0x7f51780ef140 2021-02-10T11:38:14.666 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:14.661 7f517d272700 1 --2- 172.21.2.102:0/366600096 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f51580186d0 0x7f515801abd0 crc :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:14.666 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:14.664 7f516dffb700 1 -- 172.21.2.102:0/366600096 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f5170006b80 con 0x7f51780ef140 2021-02-10T11:38:14.803 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:14.800 7f517f456700 1 -- 172.21.2.102:0/366600096 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestVolumes.test_recover_auth_metadata_during_deauthorize"]} v 0) v1 -- 0x7f51780edce0 con 0x7f51780ef140 2021-02-10T11:38:15.190 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:15.187 7f516dffb700 1 -- 172.21.2.102:0/366600096 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestVolumes.test_recover_auth_metadata_during_deauthorize"]}]=0 v839) v1 ==== 165+0+0 (secure 0 0 0) 0x7f5170006030 con 0x7f51780ef140 2021-02-10T11:38:15.221 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:15.219 7f517f456700 1 -- 172.21.2.102:0/366600096 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f51580186d0 msgr2=0x7f515801abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:15.221 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:15.219 7f517f456700 1 --2- 172.21.2.102:0/366600096 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f51580186d0 0x7f515801abd0 crc :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:15.221 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:15.219 7f517f456700 1 -- 172.21.2.102:0/366600096 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f51780ef140 msgr2=0x7f51780ef540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:15.222 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:15.219 7f517f456700 1 --2- 172.21.2.102:0/366600096 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f51780ef140 0x7f51780ef540 secure :-1 s=READY pgs=3056 cs=0 l=1 rev1=1 rx=0x7f517000b5c0 tx=0x7f517000b960).stop 2021-02-10T11:38:15.223 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:15.219 7f517f456700 1 -- 172.21.2.102:0/366600096 shutdown_connections 2021-02-10T11:38:15.223 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:15.219 7f517f456700 1 --2- 172.21.2.102:0/366600096 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f51780ef140 0x7f51780ef540 unknown :-1 s=CLOSED pgs=3056 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:15.223 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:15.219 7f517f456700 1 --2- 172.21.2.102:0/366600096 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f51580186d0 0x7f515801abd0 unknown :-1 s=CLOSED pgs=1340 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:15.223 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:15.219 7f517f456700 1 --2- 172.21.2.102:0/366600096 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f51780951d0 0x7f51780eeb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:15.223 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:15.219 7f517f456700 1 --2- 172.21.2.102:0/366600096 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f51780ae970 0x7f5178094bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:15.224 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:15.219 7f517f456700 1 -- 172.21.2.102:0/366600096 >> 172.21.2.102:0/366600096 conn(0x7f51781b1e40 msgr2=0x7f51781a6560 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:15.224 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:15.219 7f517f456700 1 -- 172.21.2.102:0/366600096 shutdown_connections 2021-02-10T11:38:15.224 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:15.219 7f517f456700 1 -- 172.21.2.102:0/366600096 wait complete. 2021-02-10T11:38:15.230 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config set mon mon_allow_pool_delete True 2021-02-10T11:38:15.663 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:15.645 7f4868fca700 1 Processor -- start 2021-02-10T11:38:15.663 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:15.645 7f4868fca700 1 -- start start 2021-02-10T11:38:15.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:15.645 7f4868fca700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f48640d1860 0x7f48641b04b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:15.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:15.645 7f4868fca700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f48640cff60 0x7f4864109120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:15.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:15.645 7f4868fca700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4864109a50 con 0x7f48641b0ad0 2021-02-10T11:38:15.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:15.645 7f4868fca700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4864109740 con 0x7f48640cff60 2021-02-10T11:38:15.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:15.645 7f4868fca700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4864088070 con 0x7f48640d1860 2021-02-10T11:38:15.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:15.646 7f486359e700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f48640cff60 0x7f4864109120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:15.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:15.646 7f486359e700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f48640cff60 0x7f4864109120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:39326/0 (socket says 172.21.2.102:39326) 2021-02-10T11:38:15.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:15.646 7f486359e700 1 -- 172.21.2.102:0/875684477 learned_addr learned my addr 172.21.2.102:0/875684477 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:38:15.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:15.646 7f486359e700 1 -- 172.21.2.102:0/875684477 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f48640d1860 msgr2=0x7f48641b04b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:38:15.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:15.646 7f486359e700 1 --2- 172.21.2.102:0/875684477 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f48640d1860 0x7f48641b04b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:15.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:15.646 7f486359e700 1 -- 172.21.2.102:0/875684477 >> v1:172.21.2.113:6789/0 conn(0x7f48641b0ad0 legacy=0x7f4864108930 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:38:15.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:15.646 7f486359e700 1 -- 172.21.2.102:0/875684477 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4864088260 con 0x7f48640cff60 2021-02-10T11:38:15.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:15.646 7f486359e700 1 --2- 172.21.2.102:0/875684477 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f48640cff60 0x7f4864109120 secure :-1 s=READY pgs=3057 cs=0 l=1 rev1=1 rx=0x7f485c006f00 tx=0x7f485c009f10).ready entity=mon.0 client_cookie=3e89be671040c3de server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:15.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:15.646 7f4861d9b700 1 -- 172.21.2.102:0/875684477 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f485c003030 con 0x7f48640cff60 2021-02-10T11:38:15.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:15.646 7f4861d9b700 1 -- 172.21.2.102:0/875684477 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f485c00b3b0 con 0x7f48640cff60 2021-02-10T11:38:15.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:15.646 7f4861d9b700 1 -- 172.21.2.102:0/875684477 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f485c00b8d0 con 0x7f48640cff60 2021-02-10T11:38:15.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:15.646 7f4868fca700 1 -- 172.21.2.102:0/875684477 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f48640cff60 msgr2=0x7f4864109120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:15.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:15.646 7f4868fca700 1 --2- 172.21.2.102:0/875684477 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f48640cff60 0x7f4864109120 secure :-1 s=READY pgs=3057 cs=0 l=1 rev1=1 rx=0x7f485c006f00 tx=0x7f485c009f10).stop 2021-02-10T11:38:15.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:15.647 7f4868fca700 1 -- 172.21.2.102:0/875684477 shutdown_connections 2021-02-10T11:38:15.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:15.647 7f4868fca700 1 --2- 172.21.2.102:0/875684477 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f48640cff60 0x7f4864109120 unknown :-1 s=CLOSED pgs=3057 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:15.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:15.647 7f4868fca700 1 --2- 172.21.2.102:0/875684477 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f48640d1860 0x7f48641b04b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:15.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:15.647 7f4868fca700 1 -- 172.21.2.102:0/875684477 >> 172.21.2.102:0/875684477 conn(0x7f48640ce410 msgr2=0x7f48640d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:15.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:15.647 7f4868fca700 1 -- 172.21.2.102:0/875684477 shutdown_connections 2021-02-10T11:38:15.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:15.647 7f4868fca700 1 -- 172.21.2.102:0/875684477 wait complete. 2021-02-10T11:38:15.667 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:15.662 7f4868fca700 1 Processor -- start 2021-02-10T11:38:15.667 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:15.662 7f4868fca700 1 -- start start 2021-02-10T11:38:15.667 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:15.662 7f4868fca700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4864199ed0 0x7f48641a7c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:15.667 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:15.662 7f4868fca700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f48641a8260 0x7f48640c1ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:15.667 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:15.662 7f4868fca700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f48640c22c0 0x7f48640c26c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:15.667 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:15.662 7f4868fca700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48640c17a0 con 0x7f48640c22c0 2021-02-10T11:38:15.667 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:15.662 7f4868fca700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48640b8c70 con 0x7f48641a8260 2021-02-10T11:38:15.667 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:15.662 7f4868fca700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48640c2ce0 con 0x7f4864199ed0 2021-02-10T11:38:15.667 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:15.662 7f486359e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f48640c22c0 0x7f48640c26c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:15.667 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:15.662 7f486359e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f48640c22c0 0x7f48640c26c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:39332/0 (socket says 172.21.2.102:39332) 2021-02-10T11:38:15.667 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:15.662 7f486359e700 1 -- 172.21.2.102:0/1779664681 learned_addr learned my addr 172.21.2.102:0/1779664681 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:38:15.668 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:15.663 7f486359e700 1 -- 172.21.2.102:0/1779664681 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4864199ed0 msgr2=0x7f48641a7c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:15.668 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:15.663 7f486359e700 1 --2- 172.21.2.102:0/1779664681 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4864199ed0 0x7f48641a7c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:15.668 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:15.663 7f486359e700 1 -- 172.21.2.102:0/1779664681 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f48641a8260 msgr2=0x7f48640c1ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:15.668 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:15.663 7f486359e700 1 --2- 172.21.2.102:0/1779664681 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f48641a8260 0x7f48640c1ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:15.668 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:15.663 7f486359e700 1 -- 172.21.2.102:0/1779664681 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48640b7200 con 0x7f48640c22c0 2021-02-10T11:38:15.668 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:15.663 7f486359e700 1 --2- 172.21.2.102:0/1779664681 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f48640c22c0 0x7f48640c26c0 secure :-1 s=READY pgs=3058 cs=0 l=1 rev1=1 rx=0x7f485c00b940 tx=0x7f485c00bce0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:15.668 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:15.663 7f485b7fe700 1 -- 172.21.2.102:0/1779664681 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f485c009450 con 0x7f48640c22c0 2021-02-10T11:38:15.668 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:15.663 7f485b7fe700 1 -- 172.21.2.102:0/1779664681 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f485c003030 con 0x7f48640c22c0 2021-02-10T11:38:15.668 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:15.663 7f485b7fe700 1 -- 172.21.2.102:0/1779664681 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f485c00c570 con 0x7f48640c22c0 2021-02-10T11:38:15.668 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:15.663 7f4868fca700 1 -- 172.21.2.102:0/1779664681 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48640b7200 con 0x7f48640c22c0 2021-02-10T11:38:15.668 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:15.663 7f4868fca700 1 -- 172.21.2.102:0/1779664681 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48640b7200 con 0x7f48640c22c0 2021-02-10T11:38:15.668 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:15.664 7f485b7fe700 1 -- 172.21.2.102:0/1779664681 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f485c001270 con 0x7f48640c22c0 2021-02-10T11:38:15.668 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:15.664 7f485b7fe700 1 --2- 172.21.2.102:0/1779664681 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f48480186d0 0x7f484801abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:15.669 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:15.664 7f485b7fe700 1 -- 172.21.2.102:0/1779664681 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f485c032b70 con 0x7f48640c22c0 2021-02-10T11:38:15.669 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:15.664 7f4862d9d700 1 --2- 172.21.2.102:0/1779664681 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f48480186d0 0x7f484801abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:15.669 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:15.664 7f4862d9d700 1 --2- 172.21.2.102:0/1779664681 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f48480186d0 0x7f484801abd0 crc :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:15.697 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:15.695 7f4868fca700 1 -- 172.21.2.102:0/1779664681 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48640fe1b0 con 0x7f48640c22c0 2021-02-10T11:38:15.700 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:15.698 7f485b7fe700 1 -- 172.21.2.102:0/1779664681 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f485c00be20 con 0x7f48640c22c0 2021-02-10T11:38:15.849 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:15.847 7f4868fca700 1 -- 172.21.2.102:0/1779664681 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f48640fe1b0 con 0x7f48640c22c0 2021-02-10T11:38:15.852 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:15.850 7f485b7fe700 1 -- 172.21.2.102:0/1779664681 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v44)=0 v44) v1 ==== 122+0+0 (secure 0 0 0) 0x7f485c001270 con 0x7f48640c22c0 2021-02-10T11:38:15.858 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:15.855 7f4868fca700 1 -- 172.21.2.102:0/1779664681 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f48480186d0 msgr2=0x7f484801abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:15.858 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:15.855 7f4868fca700 1 --2- 172.21.2.102:0/1779664681 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f48480186d0 0x7f484801abd0 crc :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:15.858 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:15.855 7f4868fca700 1 -- 172.21.2.102:0/1779664681 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f48640c22c0 msgr2=0x7f48640c26c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:15.858 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:15.855 7f4868fca700 1 --2- 172.21.2.102:0/1779664681 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f48640c22c0 0x7f48640c26c0 secure :-1 s=READY pgs=3058 cs=0 l=1 rev1=1 rx=0x7f485c00b940 tx=0x7f485c00bce0).stop 2021-02-10T11:38:15.858 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:15.856 7f4868fca700 1 -- 172.21.2.102:0/1779664681 shutdown_connections 2021-02-10T11:38:15.858 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:15.856 7f4868fca700 1 --2- 172.21.2.102:0/1779664681 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f48640c22c0 0x7f48640c26c0 unknown :-1 s=CLOSED pgs=3058 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:15.858 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:15.856 7f4868fca700 1 --2- 172.21.2.102:0/1779664681 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f48480186d0 0x7f484801abd0 unknown :-1 s=CLOSED pgs=1341 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:15.858 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:15.856 7f4868fca700 1 --2- 172.21.2.102:0/1779664681 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f48641a8260 0x7f48640c1ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:15.858 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:15.856 7f4868fca700 1 --2- 172.21.2.102:0/1779664681 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4864199ed0 0x7f48641a7c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:15.859 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:15.856 7f4868fca700 1 -- 172.21.2.102:0/1779664681 >> 172.21.2.102:0/1779664681 conn(0x7f486409c130 msgr2=0x7f4864095700 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:15.859 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:15.856 7f4868fca700 1 -- 172.21.2.102:0/1779664681 shutdown_connections 2021-02-10T11:38:15.859 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:15.856 7f4868fca700 1 -- 172.21.2.102:0/1779664681 wait complete. 2021-02-10T11:38:15.866 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2021-02-10T11:38:15.866 DEBUG:teuthology.orchestra.run.gibba002:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:38:16.213 DEBUG:teuthology.orchestra.run.gibba013:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:38:16.562 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2021-02-10T11:38:16.562 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:38:16.562 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:38:16.563 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:38:16.898 INFO:teuthology.orchestra.run.gibba002.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.0’: No such file or directory 2021-02-10T11:38:16.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:38:16.898 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.0 2021-02-10T11:38:16.899 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:38:16.899 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:38:16.899 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:38:17.235 INFO:teuthology.orchestra.run.gibba002.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.1’: No such file or directory 2021-02-10T11:38:17.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:38:17.235 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2021-02-10T11:38:17.235 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:38:17.235 DEBUG:teuthology.orchestra.run.gibba013:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:38:17.235 DEBUG:teuthology.orchestra.run.gibba013:> ' -- /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:38:17.571 INFO:teuthology.orchestra.run.gibba013.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.2’: No such file or directory 2021-02-10T11:38:17.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:38:17.571 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2021-02-10T11:38:17.571 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:38:17.572 DEBUG:teuthology.orchestra.run.gibba013:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:38:17.572 DEBUG:teuthology.orchestra.run.gibba013:> ' -- /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:38:17.907 INFO:teuthology.orchestra.run.gibba013.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.3’: No such file or directory 2021-02-10T11:38:17.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:38:17.907 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.3 2021-02-10T11:38:17.908 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json-pretty 2021-02-10T11:38:18.338 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:18.320 7f980aea7700 1 Processor -- start 2021-02-10T11:38:18.339 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:18.320 7f980aea7700 1 -- start start 2021-02-10T11:38:18.339 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:18.320 7f980aea7700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f98040d1c70 0x7f98040f20a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:18.339 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:18.320 7f980aea7700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98040f0b10 con 0x7f98040d0340 2021-02-10T11:38:18.339 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:18.320 7f980aea7700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98041b0be0 con 0x7f98040d1870 2021-02-10T11:38:18.339 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:18.320 7f980aea7700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98040e5ed0 con 0x7f98040d1c70 2021-02-10T11:38:18.339 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:18.321 7f98094c4700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f98040d1c70 0x7f98040f20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:18.339 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:18.321 7f98094c4700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f98040d1c70 0x7f98040f20a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:39340/0 (socket says 172.21.2.102:39340) 2021-02-10T11:38:18.339 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:18.321 7f98094c4700 1 -- 172.21.2.102:0/3285182855 learned_addr learned my addr 172.21.2.102:0/3285182855 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:38:18.339 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:18.321 7f98094c4700 1 -- 172.21.2.102:0/3285182855 >> v1:172.21.2.113:6789/0 conn(0x7f98040d1870 legacy=0x7f98041b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:18.339 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:18.321 7f98094c4700 1 -- 172.21.2.102:0/3285182855 >> v1:172.21.2.102:6789/0 conn(0x7f98040d0340 legacy=0x7f98040ef640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:18.339 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:18.321 7f98094c4700 1 -- 172.21.2.102:0/3285182855 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98040efe30 con 0x7f98040d1c70 2021-02-10T11:38:18.340 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:18.321 7f98094c4700 1 --2- 172.21.2.102:0/3285182855 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f98040d1c70 0x7f98040f20a0 secure :-1 s=READY pgs=3060 cs=0 l=1 rev1=1 rx=0x7f97fc006f00 tx=0x7f97fc009f10).ready entity=mon.0 client_cookie=329b4f879a980ae0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:18.340 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:18.321 7f98037fe700 1 -- 172.21.2.102:0/3285182855 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f97fc00b3b0 con 0x7f98040d1c70 2021-02-10T11:38:18.340 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:18.321 7f98037fe700 1 -- 172.21.2.102:0/3285182855 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97fc00b730 con 0x7f98040d1c70 2021-02-10T11:38:18.340 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:18.321 7f98037fe700 1 -- 172.21.2.102:0/3285182855 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f97fc00bc50 con 0x7f98040d1c70 2021-02-10T11:38:18.340 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:18.321 7f980aea7700 1 -- 172.21.2.102:0/3285182855 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f98040d1c70 msgr2=0x7f98040f20a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:18.340 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:18.321 7f980aea7700 1 --2- 172.21.2.102:0/3285182855 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f98040d1c70 0x7f98040f20a0 secure :-1 s=READY pgs=3060 cs=0 l=1 rev1=1 rx=0x7f97fc006f00 tx=0x7f97fc009f10).stop 2021-02-10T11:38:18.340 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:18.322 7f980aea7700 1 -- 172.21.2.102:0/3285182855 shutdown_connections 2021-02-10T11:38:18.340 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:18.322 7f980aea7700 1 --2- 172.21.2.102:0/3285182855 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f98040d1c70 0x7f98040f20a0 unknown :-1 s=CLOSED pgs=3060 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:18.340 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:18.322 7f980aea7700 1 -- 172.21.2.102:0/3285182855 >> 172.21.2.102:0/3285182855 conn(0x7f98040ce450 msgr2=0x7f98040d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:18.340 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:18.322 7f980aea7700 1 -- 172.21.2.102:0/3285182855 shutdown_connections 2021-02-10T11:38:18.340 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:18.322 7f980aea7700 1 -- 172.21.2.102:0/3285182855 wait complete. 2021-02-10T11:38:18.342 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:18.337 7f980aea7700 1 Processor -- start 2021-02-10T11:38:18.342 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:18.337 7f980aea7700 1 -- start start 2021-02-10T11:38:18.342 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:18.337 7f980aea7700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f98041c6350 0x7f980409a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:18.342 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:18.337 7f980aea7700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f98041c6750 0x7f98040b17a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:18.342 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:18.337 7f980aea7700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f98040b1dc0 0x7f980409bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:18.342 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:18.337 7f980aea7700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98041c6b50 con 0x7f98041c6750 2021-02-10T11:38:18.342 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:18.337 7f980aea7700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98040b21c0 con 0x7f98040b1dc0 2021-02-10T11:38:18.342 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:18.337 7f980aea7700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f980409af60 con 0x7f98041c6350 2021-02-10T11:38:18.342 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:18.337 7f9803fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f98041c6750 0x7f98040b17a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:18.342 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:18.337 7f9803fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f98041c6750 0x7f98040b17a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:39344/0 (socket says 172.21.2.102:39344) 2021-02-10T11:38:18.342 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:18.337 7f9803fff700 1 -- 172.21.2.102:0/1287315175 learned_addr learned my addr 172.21.2.102:0/1287315175 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:38:18.343 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:18.338 7f9803fff700 1 -- 172.21.2.102:0/1287315175 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f98041c6350 msgr2=0x7f980409a830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:18.343 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:18.338 7f9803fff700 1 --2- 172.21.2.102:0/1287315175 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f98041c6350 0x7f980409a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:18.343 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:18.338 7f9803fff700 1 -- 172.21.2.102:0/1287315175 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f98040b1dc0 msgr2=0x7f980409bfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:18.343 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:18.338 7f9803fff700 1 --2- 172.21.2.102:0/1287315175 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f98040b1dc0 0x7f980409bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:18.343 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:18.338 7f9803fff700 1 -- 172.21.2.102:0/1287315175 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98040f2c00 con 0x7f98041c6750 2021-02-10T11:38:18.343 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:18.338 7f9803fff700 1 --2- 172.21.2.102:0/1287315175 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f98041c6750 0x7f98040b17a0 secure :-1 s=READY pgs=3061 cs=0 l=1 rev1=1 rx=0x7f97f4006ed0 tx=0x7f97f400b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:18.343 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:18.338 7f98017fa700 1 -- 172.21.2.102:0/1287315175 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f97f4006020 con 0x7f98041c6750 2021-02-10T11:38:18.343 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:18.338 7f98017fa700 1 -- 172.21.2.102:0/1287315175 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97f40009b0 con 0x7f98041c6750 2021-02-10T11:38:18.343 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:18.338 7f98017fa700 1 -- 172.21.2.102:0/1287315175 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f97f40054b0 con 0x7f98041c6750 2021-02-10T11:38:18.343 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:18.338 7f980aea7700 1 -- 172.21.2.102:0/1287315175 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98040f2c00 con 0x7f98041c6750 2021-02-10T11:38:18.343 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:18.338 7f980aea7700 1 -- 172.21.2.102:0/1287315175 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f980409c8d0 con 0x7f98041c6750 2021-02-10T11:38:18.343 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:18.339 7f98017fa700 1 -- 172.21.2.102:0/1287315175 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f97f4006020 con 0x7f98041c6750 2021-02-10T11:38:18.344 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:18.339 7f98017fa700 1 --2- 172.21.2.102:0/1287315175 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f97e80186c0 0x7f97e801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:18.344 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:18.339 7f98017fa700 1 -- 172.21.2.102:0/1287315175 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f97f40325a0 con 0x7f98041c6750 2021-02-10T11:38:18.344 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:18.339 7f9808cc3700 1 --2- 172.21.2.102:0/1287315175 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f97e80186c0 0x7f97e801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:18.344 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:18.339 7f9808cc3700 1 --2- 172.21.2.102:0/1287315175 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f97e80186c0 0x7f97e801abc0 crc :-1 s=READY pgs=1342 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:18.372 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:18.370 7f980aea7700 1 -- 172.21.2.102:0/1287315175 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f980409c8d0 con 0x7f98041c6750 2021-02-10T11:38:18.375 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:18.373 7f98017fa700 1 -- 172.21.2.102:0/1287315175 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f97f4030d50 con 0x7f98041c6750 2021-02-10T11:38:18.498 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:18.496 7f980aea7700 1 -- 172.21.2.102:0/1287315175 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7f980409c8d0 con 0x7f98041c6750 2021-02-10T11:38:18.499 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:18.497 7f98017fa700 1 -- 172.21.2.102:0/1287315175 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v172) v1 ==== 81+0+30374 (secure 0 0 0) 0x7f97f4006000 con 0x7f98041c6750 2021-02-10T11:38:18.500 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:38:18.500 INFO:teuthology.orchestra.run.gibba002.stdout:{ 2021-02-10T11:38:18.500 INFO:teuthology.orchestra.run.gibba002.stdout: "epoch": 172, 2021-02-10T11:38:18.500 INFO:teuthology.orchestra.run.gibba002.stdout: "fsid": "68db30ea-dbce-4aef-a6df-f01c6fea0513", 2021-02-10T11:38:18.500 INFO:teuthology.orchestra.run.gibba002.stdout: "created": "2021-02-10 11:03:06.331557", 2021-02-10T11:38:18.500 INFO:teuthology.orchestra.run.gibba002.stdout: "modified": "2021-02-10 11:37:37.349944", 2021-02-10T11:38:18.500 INFO:teuthology.orchestra.run.gibba002.stdout: "last_up_change": "2021-02-10 11:03:32.690562", 2021-02-10T11:38:18.500 INFO:teuthology.orchestra.run.gibba002.stdout: "last_in_change": "2021-02-10 11:03:32.690562", 2021-02-10T11:38:18.500 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2021-02-10T11:38:18.500 INFO:teuthology.orchestra.run.gibba002.stdout: "flags_num": 5799936, 2021-02-10T11:38:18.500 INFO:teuthology.orchestra.run.gibba002.stdout: "flags_set": [ 2021-02-10T11:38:18.501 INFO:teuthology.orchestra.run.gibba002.stdout: "pglog_hardlimit", 2021-02-10T11:38:18.501 INFO:teuthology.orchestra.run.gibba002.stdout: "purged_snapdirs", 2021-02-10T11:38:18.501 INFO:teuthology.orchestra.run.gibba002.stdout: "recovery_deletes", 2021-02-10T11:38:18.501 INFO:teuthology.orchestra.run.gibba002.stdout: "sortbitwise" 2021-02-10T11:38:18.501 INFO:teuthology.orchestra.run.gibba002.stdout: ], 2021-02-10T11:38:18.501 INFO:teuthology.orchestra.run.gibba002.stdout: "crush_version": 4, 2021-02-10T11:38:18.501 INFO:teuthology.orchestra.run.gibba002.stdout: "full_ratio": 0.94999998807907104, 2021-02-10T11:38:18.501 INFO:teuthology.orchestra.run.gibba002.stdout: "backfillfull_ratio": 0.89999997615814209, 2021-02-10T11:38:18.501 INFO:teuthology.orchestra.run.gibba002.stdout: "nearfull_ratio": 0.85000002384185791, 2021-02-10T11:38:18.501 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_snapshot": "", 2021-02-10T11:38:18.501 INFO:teuthology.orchestra.run.gibba002.stdout: "pool_max": 37, 2021-02-10T11:38:18.501 INFO:teuthology.orchestra.run.gibba002.stdout: "max_osd": 8, 2021-02-10T11:38:18.501 INFO:teuthology.orchestra.run.gibba002.stdout: "require_min_compat_client": "jewel", 2021-02-10T11:38:18.502 INFO:teuthology.orchestra.run.gibba002.stdout: "min_compat_client": "jewel", 2021-02-10T11:38:18.502 INFO:teuthology.orchestra.run.gibba002.stdout: "require_osd_release": "nautilus", 2021-02-10T11:38:18.502 INFO:teuthology.orchestra.run.gibba002.stdout: "pools": [ 2021-02-10T11:38:18.502 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:38:18.502 INFO:teuthology.orchestra.run.gibba002.stdout: "pool": 1, 2021-02-10T11:38:18.502 INFO:teuthology.orchestra.run.gibba002.stdout: "pool_name": "rbd", 2021-02-10T11:38:18.502 INFO:teuthology.orchestra.run.gibba002.stdout: "create_time": "2021-02-10 11:03:37.322519", 2021-02-10T11:38:18.502 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 1, 2021-02-10T11:38:18.502 INFO:teuthology.orchestra.run.gibba002.stdout: "flags_names": "hashpspool", 2021-02-10T11:38:18.502 INFO:teuthology.orchestra.run.gibba002.stdout: "type": 1, 2021-02-10T11:38:18.502 INFO:teuthology.orchestra.run.gibba002.stdout: "size": 2, 2021-02-10T11:38:18.502 INFO:teuthology.orchestra.run.gibba002.stdout: "min_size": 1, 2021-02-10T11:38:18.502 INFO:teuthology.orchestra.run.gibba002.stdout: "crush_rule": 0, 2021-02-10T11:38:18.503 INFO:teuthology.orchestra.run.gibba002.stdout: "object_hash": 2, 2021-02-10T11:38:18.503 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:38:18.503 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num": 8, 2021-02-10T11:38:18.503 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_placement_num": 8, 2021-02-10T11:38:18.503 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_placement_num_target": 8, 2021-02-10T11:38:18.503 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num_target": 8, 2021-02-10T11:38:18.503 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num_pending": 8, 2021-02-10T11:38:18.503 INFO:teuthology.orchestra.run.gibba002.stdout: "last_pg_merge_meta": { 2021-02-10T11:38:18.503 INFO:teuthology.orchestra.run.gibba002.stdout: "source_pgid": "0.0", 2021-02-10T11:38:18.503 INFO:teuthology.orchestra.run.gibba002.stdout: "ready_epoch": 0, 2021-02-10T11:38:18.503 INFO:teuthology.orchestra.run.gibba002.stdout: "last_epoch_started": 0, 2021-02-10T11:38:18.503 INFO:teuthology.orchestra.run.gibba002.stdout: "last_epoch_clean": 0, 2021-02-10T11:38:18.503 INFO:teuthology.orchestra.run.gibba002.stdout: "source_version": "0'0", 2021-02-10T11:38:18.504 INFO:teuthology.orchestra.run.gibba002.stdout: "target_version": "0'0" 2021-02-10T11:38:18.504 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:38:18.504 INFO:teuthology.orchestra.run.gibba002.stdout: "last_change": "16", 2021-02-10T11:38:18.504 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend": "0", 2021-02-10T11:38:18.504 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:38:18.504 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:38:18.504 INFO:teuthology.orchestra.run.gibba002.stdout: "auid": 0, 2021-02-10T11:38:18.504 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:38:18.504 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_seq": 0, 2021-02-10T11:38:18.504 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_epoch": 0, 2021-02-10T11:38:18.504 INFO:teuthology.orchestra.run.gibba002.stdout: "pool_snaps": [], 2021-02-10T11:38:18.504 INFO:teuthology.orchestra.run.gibba002.stdout: "removed_snaps": "[]", 2021-02-10T11:38:18.504 INFO:teuthology.orchestra.run.gibba002.stdout: "quota_max_bytes": 0, 2021-02-10T11:38:18.505 INFO:teuthology.orchestra.run.gibba002.stdout: "quota_max_objects": 0, 2021-02-10T11:38:18.505 INFO:teuthology.orchestra.run.gibba002.stdout: "tiers": [], 2021-02-10T11:38:18.505 INFO:teuthology.orchestra.run.gibba002.stdout: "tier_of": -1, 2021-02-10T11:38:18.505 INFO:teuthology.orchestra.run.gibba002.stdout: "read_tier": -1, 2021-02-10T11:38:18.505 INFO:teuthology.orchestra.run.gibba002.stdout: "write_tier": -1, 2021-02-10T11:38:18.505 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_mode": "none", 2021-02-10T11:38:18.505 INFO:teuthology.orchestra.run.gibba002.stdout: "target_max_bytes": 0, 2021-02-10T11:38:18.505 INFO:teuthology.orchestra.run.gibba002.stdout: "target_max_objects": 0, 2021-02-10T11:38:18.505 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:38:18.505 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:38:18.505 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:38:18.505 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_min_flush_age": 0, 2021-02-10T11:38:18.506 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_min_evict_age": 0, 2021-02-10T11:38:18.506 INFO:teuthology.orchestra.run.gibba002.stdout: "erasure_code_profile": "", 2021-02-10T11:38:18.506 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_params": { 2021-02-10T11:38:18.506 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "none" 2021-02-10T11:38:18.506 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:38:18.506 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_period": 0, 2021-02-10T11:38:18.506 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_count": 0, 2021-02-10T11:38:18.506 INFO:teuthology.orchestra.run.gibba002.stdout: "use_gmt_hitset": true, 2021-02-10T11:38:18.506 INFO:teuthology.orchestra.run.gibba002.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:38:18.506 INFO:teuthology.orchestra.run.gibba002.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:38:18.506 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:38:18.506 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:38:18.507 INFO:teuthology.orchestra.run.gibba002.stdout: "grade_table": [], 2021-02-10T11:38:18.507 INFO:teuthology.orchestra.run.gibba002.stdout: "stripe_width": 0, 2021-02-10T11:38:18.507 INFO:teuthology.orchestra.run.gibba002.stdout: "expected_num_objects": 0, 2021-02-10T11:38:18.507 INFO:teuthology.orchestra.run.gibba002.stdout: "fast_read": false, 2021-02-10T11:38:18.507 INFO:teuthology.orchestra.run.gibba002.stdout: "options": {}, 2021-02-10T11:38:18.507 INFO:teuthology.orchestra.run.gibba002.stdout: "application_metadata": { 2021-02-10T11:38:18.507 INFO:teuthology.orchestra.run.gibba002.stdout: "rbd": {} 2021-02-10T11:38:18.507 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:38:18.507 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:38:18.507 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:38:18.507 INFO:teuthology.orchestra.run.gibba002.stdout: "pool": 36, 2021-02-10T11:38:18.507 INFO:teuthology.orchestra.run.gibba002.stdout: "pool_name": "cephfs_metadata", 2021-02-10T11:38:18.508 INFO:teuthology.orchestra.run.gibba002.stdout: "create_time": "2021-02-10 11:37:34.298334", 2021-02-10T11:38:18.508 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 1, 2021-02-10T11:38:18.508 INFO:teuthology.orchestra.run.gibba002.stdout: "flags_names": "hashpspool", 2021-02-10T11:38:18.508 INFO:teuthology.orchestra.run.gibba002.stdout: "type": 1, 2021-02-10T11:38:18.508 INFO:teuthology.orchestra.run.gibba002.stdout: "size": 2, 2021-02-10T11:38:18.508 INFO:teuthology.orchestra.run.gibba002.stdout: "min_size": 1, 2021-02-10T11:38:18.508 INFO:teuthology.orchestra.run.gibba002.stdout: "crush_rule": 0, 2021-02-10T11:38:18.508 INFO:teuthology.orchestra.run.gibba002.stdout: "object_hash": 2, 2021-02-10T11:38:18.508 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:38:18.508 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num": 8, 2021-02-10T11:38:18.509 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_placement_num": 8, 2021-02-10T11:38:18.509 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_placement_num_target": 8, 2021-02-10T11:38:18.509 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num_target": 8, 2021-02-10T11:38:18.509 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num_pending": 8, 2021-02-10T11:38:18.509 INFO:teuthology.orchestra.run.gibba002.stdout: "last_pg_merge_meta": { 2021-02-10T11:38:18.509 INFO:teuthology.orchestra.run.gibba002.stdout: "source_pgid": "0.0", 2021-02-10T11:38:18.509 INFO:teuthology.orchestra.run.gibba002.stdout: "ready_epoch": 0, 2021-02-10T11:38:18.509 INFO:teuthology.orchestra.run.gibba002.stdout: "last_epoch_started": 0, 2021-02-10T11:38:18.509 INFO:teuthology.orchestra.run.gibba002.stdout: "last_epoch_clean": 0, 2021-02-10T11:38:18.509 INFO:teuthology.orchestra.run.gibba002.stdout: "source_version": "0'0", 2021-02-10T11:38:18.509 INFO:teuthology.orchestra.run.gibba002.stdout: "target_version": "0'0" 2021-02-10T11:38:18.509 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:38:18.510 INFO:teuthology.orchestra.run.gibba002.stdout: "last_change": "171", 2021-02-10T11:38:18.510 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend": "0", 2021-02-10T11:38:18.510 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:38:18.510 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:38:18.510 INFO:teuthology.orchestra.run.gibba002.stdout: "auid": 0, 2021-02-10T11:38:18.510 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:38:18.510 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_seq": 0, 2021-02-10T11:38:18.510 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_epoch": 0, 2021-02-10T11:38:18.510 INFO:teuthology.orchestra.run.gibba002.stdout: "pool_snaps": [], 2021-02-10T11:38:18.510 INFO:teuthology.orchestra.run.gibba002.stdout: "removed_snaps": "[]", 2021-02-10T11:38:18.510 INFO:teuthology.orchestra.run.gibba002.stdout: "quota_max_bytes": 0, 2021-02-10T11:38:18.510 INFO:teuthology.orchestra.run.gibba002.stdout: "quota_max_objects": 0, 2021-02-10T11:38:18.511 INFO:teuthology.orchestra.run.gibba002.stdout: "tiers": [], 2021-02-10T11:38:18.511 INFO:teuthology.orchestra.run.gibba002.stdout: "tier_of": -1, 2021-02-10T11:38:18.511 INFO:teuthology.orchestra.run.gibba002.stdout: "read_tier": -1, 2021-02-10T11:38:18.511 INFO:teuthology.orchestra.run.gibba002.stdout: "write_tier": -1, 2021-02-10T11:38:18.511 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_mode": "none", 2021-02-10T11:38:18.511 INFO:teuthology.orchestra.run.gibba002.stdout: "target_max_bytes": 0, 2021-02-10T11:38:18.511 INFO:teuthology.orchestra.run.gibba002.stdout: "target_max_objects": 0, 2021-02-10T11:38:18.511 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:38:18.511 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:38:18.511 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:38:18.511 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_min_flush_age": 0, 2021-02-10T11:38:18.511 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_min_evict_age": 0, 2021-02-10T11:38:18.512 INFO:teuthology.orchestra.run.gibba002.stdout: "erasure_code_profile": "", 2021-02-10T11:38:18.512 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_params": { 2021-02-10T11:38:18.512 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "none" 2021-02-10T11:38:18.512 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:38:18.512 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_period": 0, 2021-02-10T11:38:18.512 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_count": 0, 2021-02-10T11:38:18.512 INFO:teuthology.orchestra.run.gibba002.stdout: "use_gmt_hitset": true, 2021-02-10T11:38:18.512 INFO:teuthology.orchestra.run.gibba002.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:38:18.512 INFO:teuthology.orchestra.run.gibba002.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:38:18.512 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:38:18.512 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:38:18.512 INFO:teuthology.orchestra.run.gibba002.stdout: "grade_table": [], 2021-02-10T11:38:18.513 INFO:teuthology.orchestra.run.gibba002.stdout: "stripe_width": 0, 2021-02-10T11:38:18.513 INFO:teuthology.orchestra.run.gibba002.stdout: "expected_num_objects": 0, 2021-02-10T11:38:18.513 INFO:teuthology.orchestra.run.gibba002.stdout: "fast_read": false, 2021-02-10T11:38:18.513 INFO:teuthology.orchestra.run.gibba002.stdout: "options": { 2021-02-10T11:38:18.513 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_autoscale_bias": 4, 2021-02-10T11:38:18.513 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num_min": 16, 2021-02-10T11:38:18.513 INFO:teuthology.orchestra.run.gibba002.stdout: "recovery_priority": 5 2021-02-10T11:38:18.513 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:38:18.513 INFO:teuthology.orchestra.run.gibba002.stdout: "application_metadata": { 2021-02-10T11:38:18.513 INFO:teuthology.orchestra.run.gibba002.stdout: "cephfs": { 2021-02-10T11:38:18.513 INFO:teuthology.orchestra.run.gibba002.stdout: "metadata": "cephfs" 2021-02-10T11:38:18.513 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:38:18.513 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:38:18.514 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:38:18.514 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:38:18.514 INFO:teuthology.orchestra.run.gibba002.stdout: "pool": 37, 2021-02-10T11:38:18.514 INFO:teuthology.orchestra.run.gibba002.stdout: "pool_name": "cephfs_data", 2021-02-10T11:38:18.514 INFO:teuthology.orchestra.run.gibba002.stdout: "create_time": "2021-02-10 11:37:35.337127", 2021-02-10T11:38:18.514 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 1, 2021-02-10T11:38:18.514 INFO:teuthology.orchestra.run.gibba002.stdout: "flags_names": "hashpspool", 2021-02-10T11:38:18.514 INFO:teuthology.orchestra.run.gibba002.stdout: "type": 1, 2021-02-10T11:38:18.514 INFO:teuthology.orchestra.run.gibba002.stdout: "size": 2, 2021-02-10T11:38:18.514 INFO:teuthology.orchestra.run.gibba002.stdout: "min_size": 1, 2021-02-10T11:38:18.514 INFO:teuthology.orchestra.run.gibba002.stdout: "crush_rule": 0, 2021-02-10T11:38:18.514 INFO:teuthology.orchestra.run.gibba002.stdout: "object_hash": 2, 2021-02-10T11:38:18.514 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:38:18.515 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num": 8, 2021-02-10T11:38:18.515 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_placement_num": 8, 2021-02-10T11:38:18.515 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_placement_num_target": 8, 2021-02-10T11:38:18.515 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num_target": 8, 2021-02-10T11:38:18.515 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num_pending": 8, 2021-02-10T11:38:18.515 INFO:teuthology.orchestra.run.gibba002.stdout: "last_pg_merge_meta": { 2021-02-10T11:38:18.515 INFO:teuthology.orchestra.run.gibba002.stdout: "source_pgid": "0.0", 2021-02-10T11:38:18.515 INFO:teuthology.orchestra.run.gibba002.stdout: "ready_epoch": 0, 2021-02-10T11:38:18.515 INFO:teuthology.orchestra.run.gibba002.stdout: "last_epoch_started": 0, 2021-02-10T11:38:18.515 INFO:teuthology.orchestra.run.gibba002.stdout: "last_epoch_clean": 0, 2021-02-10T11:38:18.515 INFO:teuthology.orchestra.run.gibba002.stdout: "source_version": "0'0", 2021-02-10T11:38:18.515 INFO:teuthology.orchestra.run.gibba002.stdout: "target_version": "0'0" 2021-02-10T11:38:18.515 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:38:18.515 INFO:teuthology.orchestra.run.gibba002.stdout: "last_change": "172", 2021-02-10T11:38:18.516 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend": "0", 2021-02-10T11:38:18.516 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:38:18.516 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:38:18.516 INFO:teuthology.orchestra.run.gibba002.stdout: "auid": 0, 2021-02-10T11:38:18.516 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:38:18.516 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_seq": 0, 2021-02-10T11:38:18.516 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_epoch": 0, 2021-02-10T11:38:18.516 INFO:teuthology.orchestra.run.gibba002.stdout: "pool_snaps": [], 2021-02-10T11:38:18.516 INFO:teuthology.orchestra.run.gibba002.stdout: "removed_snaps": "[]", 2021-02-10T11:38:18.516 INFO:teuthology.orchestra.run.gibba002.stdout: "quota_max_bytes": 0, 2021-02-10T11:38:18.516 INFO:teuthology.orchestra.run.gibba002.stdout: "quota_max_objects": 0, 2021-02-10T11:38:18.516 INFO:teuthology.orchestra.run.gibba002.stdout: "tiers": [], 2021-02-10T11:38:18.516 INFO:teuthology.orchestra.run.gibba002.stdout: "tier_of": -1, 2021-02-10T11:38:18.517 INFO:teuthology.orchestra.run.gibba002.stdout: "read_tier": -1, 2021-02-10T11:38:18.517 INFO:teuthology.orchestra.run.gibba002.stdout: "write_tier": -1, 2021-02-10T11:38:18.517 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_mode": "none", 2021-02-10T11:38:18.517 INFO:teuthology.orchestra.run.gibba002.stdout: "target_max_bytes": 0, 2021-02-10T11:38:18.517 INFO:teuthology.orchestra.run.gibba002.stdout: "target_max_objects": 0, 2021-02-10T11:38:18.517 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:38:18.517 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:38:18.517 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:38:18.517 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_min_flush_age": 0, 2021-02-10T11:38:18.517 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_min_evict_age": 0, 2021-02-10T11:38:18.517 INFO:teuthology.orchestra.run.gibba002.stdout: "erasure_code_profile": "", 2021-02-10T11:38:18.517 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_params": { 2021-02-10T11:38:18.518 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "none" 2021-02-10T11:38:18.518 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:38:18.518 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_period": 0, 2021-02-10T11:38:18.518 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_count": 0, 2021-02-10T11:38:18.518 INFO:teuthology.orchestra.run.gibba002.stdout: "use_gmt_hitset": true, 2021-02-10T11:38:18.518 INFO:teuthology.orchestra.run.gibba002.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:38:18.518 INFO:teuthology.orchestra.run.gibba002.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:38:18.518 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:38:18.518 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:38:18.518 INFO:teuthology.orchestra.run.gibba002.stdout: "grade_table": [], 2021-02-10T11:38:18.518 INFO:teuthology.orchestra.run.gibba002.stdout: "stripe_width": 0, 2021-02-10T11:38:18.518 INFO:teuthology.orchestra.run.gibba002.stdout: "expected_num_objects": 0, 2021-02-10T11:38:18.519 INFO:teuthology.orchestra.run.gibba002.stdout: "fast_read": false, 2021-02-10T11:38:18.519 INFO:teuthology.orchestra.run.gibba002.stdout: "options": {}, 2021-02-10T11:38:18.519 INFO:teuthology.orchestra.run.gibba002.stdout: "application_metadata": { 2021-02-10T11:38:18.519 INFO:teuthology.orchestra.run.gibba002.stdout: "cephfs": { 2021-02-10T11:38:18.519 INFO:teuthology.orchestra.run.gibba002.stdout: "data": "cephfs" 2021-02-10T11:38:18.519 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:38:18.519 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:38:18.519 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:38:18.519 INFO:teuthology.orchestra.run.gibba002.stdout: ], 2021-02-10T11:38:18.519 INFO:teuthology.orchestra.run.gibba002.stdout: "osds": [ 2021-02-10T11:38:18.519 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:38:18.519 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 0, 2021-02-10T11:38:18.520 INFO:teuthology.orchestra.run.gibba002.stdout: "uuid": "be691b4d-a039-408a-af80-9d53f02b6f1f", 2021-02-10T11:38:18.520 INFO:teuthology.orchestra.run.gibba002.stdout: "up": 1, 2021-02-10T11:38:18.520 INFO:teuthology.orchestra.run.gibba002.stdout: "in": 1, 2021-02-10T11:38:18.520 INFO:teuthology.orchestra.run.gibba002.stdout: "weight": 1, 2021-02-10T11:38:18.520 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_affinity": 1, 2021-02-10T11:38:18.520 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_begin": 0, 2021-02-10T11:38:18.520 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_end": 0, 2021-02-10T11:38:18.520 INFO:teuthology.orchestra.run.gibba002.stdout: "up_from": 13, 2021-02-10T11:38:18.520 INFO:teuthology.orchestra.run.gibba002.stdout: "up_thru": 170, 2021-02-10T11:38:18.520 INFO:teuthology.orchestra.run.gibba002.stdout: "down_at": 0, 2021-02-10T11:38:18.520 INFO:teuthology.orchestra.run.gibba002.stdout: "lost_at": 0, 2021-02-10T11:38:18.520 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addrs": { 2021-02-10T11:38:18.520 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:38:18.521 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:38:18.521 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:38:18.521 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6818", 2021-02-10T11:38:18.521 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36088 2021-02-10T11:38:18.521 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:38:18.521 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:38:18.521 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:38:18.521 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6820", 2021-02-10T11:38:18.521 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36088 2021-02-10T11:38:18.521 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:38:18.521 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:38:18.521 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:38:18.521 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addrs": { 2021-02-10T11:38:18.522 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:38:18.522 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:38:18.522 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:38:18.522 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6822", 2021-02-10T11:38:18.522 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36088 2021-02-10T11:38:18.522 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:38:18.522 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:38:18.522 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:38:18.522 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6824", 2021-02-10T11:38:18.522 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36088 2021-02-10T11:38:18.522 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:38:18.522 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:38:18.523 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:38:18.523 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addrs": { 2021-02-10T11:38:18.523 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:38:18.523 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:38:18.523 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:38:18.523 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6830", 2021-02-10T11:38:18.523 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36088 2021-02-10T11:38:18.523 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:38:18.523 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:38:18.523 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:38:18.523 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6832", 2021-02-10T11:38:18.523 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36088 2021-02-10T11:38:18.524 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:38:18.524 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:38:18.524 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:38:18.524 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addrs": { 2021-02-10T11:38:18.524 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:38:18.524 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:38:18.524 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:38:18.524 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6826", 2021-02-10T11:38:18.524 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36088 2021-02-10T11:38:18.524 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:38:18.524 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:38:18.524 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:38:18.524 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6828", 2021-02-10T11:38:18.525 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36088 2021-02-10T11:38:18.525 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:38:18.525 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:38:18.525 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:38:18.525 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addr": "172.21.2.102:6820/36088", 2021-02-10T11:38:18.525 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addr": "172.21.2.102:6824/36088", 2021-02-10T11:38:18.525 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addr": "172.21.2.102:6832/36088", 2021-02-10T11:38:18.525 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addr": "172.21.2.102:6828/36088", 2021-02-10T11:38:18.525 INFO:teuthology.orchestra.run.gibba002.stdout: "state": [ 2021-02-10T11:38:18.525 INFO:teuthology.orchestra.run.gibba002.stdout: "exists", 2021-02-10T11:38:18.525 INFO:teuthology.orchestra.run.gibba002.stdout: "up" 2021-02-10T11:38:18.525 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:38:18.526 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:38:18.526 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:38:18.526 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 1, 2021-02-10T11:38:18.526 INFO:teuthology.orchestra.run.gibba002.stdout: "uuid": "64e3ccc8-9c69-4d7c-865e-af8e83822635", 2021-02-10T11:38:18.526 INFO:teuthology.orchestra.run.gibba002.stdout: "up": 1, 2021-02-10T11:38:18.526 INFO:teuthology.orchestra.run.gibba002.stdout: "in": 1, 2021-02-10T11:38:18.526 INFO:teuthology.orchestra.run.gibba002.stdout: "weight": 1, 2021-02-10T11:38:18.526 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_affinity": 1, 2021-02-10T11:38:18.526 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_begin": 0, 2021-02-10T11:38:18.526 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_end": 0, 2021-02-10T11:38:18.526 INFO:teuthology.orchestra.run.gibba002.stdout: "up_from": 13, 2021-02-10T11:38:18.526 INFO:teuthology.orchestra.run.gibba002.stdout: "up_thru": 170, 2021-02-10T11:38:18.527 INFO:teuthology.orchestra.run.gibba002.stdout: "down_at": 0, 2021-02-10T11:38:18.527 INFO:teuthology.orchestra.run.gibba002.stdout: "lost_at": 0, 2021-02-10T11:38:18.527 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addrs": { 2021-02-10T11:38:18.527 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:38:18.527 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:38:18.527 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:38:18.527 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6819", 2021-02-10T11:38:18.527 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36087 2021-02-10T11:38:18.527 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:38:18.527 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:38:18.527 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:38:18.527 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6821", 2021-02-10T11:38:18.527 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36087 2021-02-10T11:38:18.528 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:38:18.528 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:38:18.528 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:38:18.528 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addrs": { 2021-02-10T11:38:18.528 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:38:18.528 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:38:18.528 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:38:18.528 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6823", 2021-02-10T11:38:18.528 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36087 2021-02-10T11:38:18.528 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:38:18.528 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:38:18.528 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:38:18.528 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6825", 2021-02-10T11:38:18.529 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36087 2021-02-10T11:38:18.529 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:38:18.529 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:38:18.529 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:38:18.529 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addrs": { 2021-02-10T11:38:18.529 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:38:18.529 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:38:18.529 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:38:18.529 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6831", 2021-02-10T11:38:18.529 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36087 2021-02-10T11:38:18.529 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:38:18.529 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:38:18.529 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:38:18.529 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6833", 2021-02-10T11:38:18.530 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36087 2021-02-10T11:38:18.530 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:38:18.530 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:38:18.530 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:38:18.530 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addrs": { 2021-02-10T11:38:18.530 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:38:18.530 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:38:18.530 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:38:18.530 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6827", 2021-02-10T11:38:18.530 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36087 2021-02-10T11:38:18.530 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:38:18.530 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:38:18.530 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:38:18.531 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6829", 2021-02-10T11:38:18.531 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36087 2021-02-10T11:38:18.531 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:38:18.531 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:38:18.531 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:38:18.531 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addr": "172.21.2.102:6821/36087", 2021-02-10T11:38:18.531 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addr": "172.21.2.102:6825/36087", 2021-02-10T11:38:18.531 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addr": "172.21.2.102:6833/36087", 2021-02-10T11:38:18.531 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addr": "172.21.2.102:6829/36087", 2021-02-10T11:38:18.531 INFO:teuthology.orchestra.run.gibba002.stdout: "state": [ 2021-02-10T11:38:18.531 INFO:teuthology.orchestra.run.gibba002.stdout: "exists", 2021-02-10T11:38:18.532 INFO:teuthology.orchestra.run.gibba002.stdout: "up" 2021-02-10T11:38:18.532 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:38:18.532 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:38:18.532 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:38:18.532 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 2, 2021-02-10T11:38:18.532 INFO:teuthology.orchestra.run.gibba002.stdout: "uuid": "153700af-747f-481f-bdf4-d6072d3a842f", 2021-02-10T11:38:18.532 INFO:teuthology.orchestra.run.gibba002.stdout: "up": 1, 2021-02-10T11:38:18.532 INFO:teuthology.orchestra.run.gibba002.stdout: "in": 1, 2021-02-10T11:38:18.532 INFO:teuthology.orchestra.run.gibba002.stdout: "weight": 1, 2021-02-10T11:38:18.532 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_affinity": 1, 2021-02-10T11:38:18.532 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_begin": 0, 2021-02-10T11:38:18.532 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_end": 0, 2021-02-10T11:38:18.533 INFO:teuthology.orchestra.run.gibba002.stdout: "up_from": 13, 2021-02-10T11:38:18.533 INFO:teuthology.orchestra.run.gibba002.stdout: "up_thru": 170, 2021-02-10T11:38:18.533 INFO:teuthology.orchestra.run.gibba002.stdout: "down_at": 0, 2021-02-10T11:38:18.533 INFO:teuthology.orchestra.run.gibba002.stdout: "lost_at": 0, 2021-02-10T11:38:18.533 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addrs": { 2021-02-10T11:38:18.533 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:38:18.533 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:38:18.533 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:38:18.533 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6810", 2021-02-10T11:38:18.533 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36089 2021-02-10T11:38:18.533 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:38:18.533 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:38:18.534 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:38:18.534 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6811", 2021-02-10T11:38:18.534 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36089 2021-02-10T11:38:18.534 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:38:18.534 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:38:18.534 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:38:18.534 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addrs": { 2021-02-10T11:38:18.534 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:38:18.534 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:38:18.534 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:38:18.534 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6812", 2021-02-10T11:38:18.534 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36089 2021-02-10T11:38:18.534 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:38:18.535 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:38:18.535 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:38:18.535 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6813", 2021-02-10T11:38:18.535 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36089 2021-02-10T11:38:18.535 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:38:18.535 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:38:18.535 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:38:18.535 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addrs": { 2021-02-10T11:38:18.535 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:38:18.535 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:38:18.535 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:38:18.535 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6816", 2021-02-10T11:38:18.535 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36089 2021-02-10T11:38:18.536 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:38:18.536 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:38:18.536 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:38:18.536 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6817", 2021-02-10T11:38:18.536 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36089 2021-02-10T11:38:18.536 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:38:18.536 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:38:18.536 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:38:18.536 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addrs": { 2021-02-10T11:38:18.536 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:38:18.536 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:38:18.536 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:38:18.537 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6814", 2021-02-10T11:38:18.537 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36089 2021-02-10T11:38:18.537 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:38:18.537 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:38:18.537 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:38:18.537 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6815", 2021-02-10T11:38:18.537 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36089 2021-02-10T11:38:18.537 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:38:18.537 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:38:18.537 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:38:18.537 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addr": "172.21.2.102:6811/36089", 2021-02-10T11:38:18.537 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addr": "172.21.2.102:6813/36089", 2021-02-10T11:38:18.537 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addr": "172.21.2.102:6817/36089", 2021-02-10T11:38:18.537 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addr": "172.21.2.102:6815/36089", 2021-02-10T11:38:18.538 INFO:teuthology.orchestra.run.gibba002.stdout: "state": [ 2021-02-10T11:38:18.538 INFO:teuthology.orchestra.run.gibba002.stdout: "exists", 2021-02-10T11:38:18.538 INFO:teuthology.orchestra.run.gibba002.stdout: "up" 2021-02-10T11:38:18.538 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:38:18.538 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:38:18.538 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:38:18.538 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 3, 2021-02-10T11:38:18.538 INFO:teuthology.orchestra.run.gibba002.stdout: "uuid": "cbc678f4-83a9-45e6-8830-2f7c232b1ea7", 2021-02-10T11:38:18.538 INFO:teuthology.orchestra.run.gibba002.stdout: "up": 1, 2021-02-10T11:38:18.538 INFO:teuthology.orchestra.run.gibba002.stdout: "in": 1, 2021-02-10T11:38:18.538 INFO:teuthology.orchestra.run.gibba002.stdout: "weight": 1, 2021-02-10T11:38:18.538 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_affinity": 1, 2021-02-10T11:38:18.538 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_begin": 0, 2021-02-10T11:38:18.539 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_end": 0, 2021-02-10T11:38:18.539 INFO:teuthology.orchestra.run.gibba002.stdout: "up_from": 13, 2021-02-10T11:38:18.539 INFO:teuthology.orchestra.run.gibba002.stdout: "up_thru": 170, 2021-02-10T11:38:18.539 INFO:teuthology.orchestra.run.gibba002.stdout: "down_at": 0, 2021-02-10T11:38:18.539 INFO:teuthology.orchestra.run.gibba002.stdout: "lost_at": 0, 2021-02-10T11:38:18.539 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addrs": { 2021-02-10T11:38:18.539 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:38:18.539 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:38:18.539 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:38:18.539 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6802", 2021-02-10T11:38:18.539 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36090 2021-02-10T11:38:18.539 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:38:18.539 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:38:18.539 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:38:18.540 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6803", 2021-02-10T11:38:18.540 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36090 2021-02-10T11:38:18.540 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:38:18.540 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:38:18.540 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:38:18.540 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addrs": { 2021-02-10T11:38:18.540 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:38:18.540 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:38:18.540 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:38:18.540 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6804", 2021-02-10T11:38:18.540 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36090 2021-02-10T11:38:18.540 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:38:18.541 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:38:18.541 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:38:18.541 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6805", 2021-02-10T11:38:18.541 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36090 2021-02-10T11:38:18.541 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:38:18.541 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:38:18.541 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:38:18.541 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addrs": { 2021-02-10T11:38:18.541 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:38:18.541 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:38:18.541 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:38:18.541 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6808", 2021-02-10T11:38:18.541 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36090 2021-02-10T11:38:18.541 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:38:18.542 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:38:18.542 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:38:18.542 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6809", 2021-02-10T11:38:18.542 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36090 2021-02-10T11:38:18.542 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:38:18.542 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:38:18.542 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:38:18.542 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addrs": { 2021-02-10T11:38:18.542 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:38:18.542 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:38:18.542 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:38:18.542 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6806", 2021-02-10T11:38:18.543 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36090 2021-02-10T11:38:18.543 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:38:18.543 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:38:18.543 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:38:18.543 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6807", 2021-02-10T11:38:18.543 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36090 2021-02-10T11:38:18.543 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:38:18.543 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:38:18.543 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:38:18.543 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addr": "172.21.2.102:6803/36090", 2021-02-10T11:38:18.543 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addr": "172.21.2.102:6805/36090", 2021-02-10T11:38:18.543 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addr": "172.21.2.102:6809/36090", 2021-02-10T11:38:18.543 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addr": "172.21.2.102:6807/36090", 2021-02-10T11:38:18.543 INFO:teuthology.orchestra.run.gibba002.stdout: "state": [ 2021-02-10T11:38:18.544 INFO:teuthology.orchestra.run.gibba002.stdout: "exists", 2021-02-10T11:38:18.544 INFO:teuthology.orchestra.run.gibba002.stdout: "up" 2021-02-10T11:38:18.544 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:38:18.544 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:38:18.544 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:38:18.544 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 4, 2021-02-10T11:38:18.544 INFO:teuthology.orchestra.run.gibba002.stdout: "uuid": "c01e4008-b258-4f10-afd2-9e7f37214351", 2021-02-10T11:38:18.544 INFO:teuthology.orchestra.run.gibba002.stdout: "up": 1, 2021-02-10T11:38:18.544 INFO:teuthology.orchestra.run.gibba002.stdout: "in": 1, 2021-02-10T11:38:18.544 INFO:teuthology.orchestra.run.gibba002.stdout: "weight": 1, 2021-02-10T11:38:18.544 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_affinity": 1, 2021-02-10T11:38:18.544 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_begin": 0, 2021-02-10T11:38:18.544 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_end": 0, 2021-02-10T11:38:18.545 INFO:teuthology.orchestra.run.gibba002.stdout: "up_from": 13, 2021-02-10T11:38:18.545 INFO:teuthology.orchestra.run.gibba002.stdout: "up_thru": 170, 2021-02-10T11:38:18.545 INFO:teuthology.orchestra.run.gibba002.stdout: "down_at": 0, 2021-02-10T11:38:18.545 INFO:teuthology.orchestra.run.gibba002.stdout: "lost_at": 0, 2021-02-10T11:38:18.545 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addrs": { 2021-02-10T11:38:18.545 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:38:18.545 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:38:18.545 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:38:18.545 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6808", 2021-02-10T11:38:18.545 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36368 2021-02-10T11:38:18.545 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:38:18.545 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:38:18.545 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:38:18.545 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6811", 2021-02-10T11:38:18.546 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36368 2021-02-10T11:38:18.546 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:38:18.546 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:38:18.546 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:38:18.546 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addrs": { 2021-02-10T11:38:18.546 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:38:18.546 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:38:18.546 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:38:18.546 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6814", 2021-02-10T11:38:18.546 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36368 2021-02-10T11:38:18.546 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:38:18.546 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:38:18.546 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:38:18.547 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6818", 2021-02-10T11:38:18.547 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36368 2021-02-10T11:38:18.547 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:38:18.547 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:38:18.547 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:38:18.547 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addrs": { 2021-02-10T11:38:18.547 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:38:18.547 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:38:18.547 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:38:18.547 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6826", 2021-02-10T11:38:18.547 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36368 2021-02-10T11:38:18.547 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:38:18.547 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:38:18.547 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:38:18.548 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6828", 2021-02-10T11:38:18.548 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36368 2021-02-10T11:38:18.548 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:38:18.548 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:38:18.548 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:38:18.548 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addrs": { 2021-02-10T11:38:18.548 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:38:18.548 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:38:18.548 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:38:18.548 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6820", 2021-02-10T11:38:18.548 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36368 2021-02-10T11:38:18.548 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:38:18.548 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:38:18.549 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:38:18.549 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6824", 2021-02-10T11:38:18.549 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36368 2021-02-10T11:38:18.549 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:38:18.549 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:38:18.549 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:38:18.549 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addr": "172.21.2.113:6811/36368", 2021-02-10T11:38:18.549 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addr": "172.21.2.113:6818/36368", 2021-02-10T11:38:18.549 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addr": "172.21.2.113:6828/36368", 2021-02-10T11:38:18.549 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addr": "172.21.2.113:6824/36368", 2021-02-10T11:38:18.549 INFO:teuthology.orchestra.run.gibba002.stdout: "state": [ 2021-02-10T11:38:18.549 INFO:teuthology.orchestra.run.gibba002.stdout: "exists", 2021-02-10T11:38:18.549 INFO:teuthology.orchestra.run.gibba002.stdout: "up" 2021-02-10T11:38:18.549 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:38:18.550 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:38:18.550 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:38:18.550 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 5, 2021-02-10T11:38:18.550 INFO:teuthology.orchestra.run.gibba002.stdout: "uuid": "551f6007-daf8-4a98-8ee0-c2b16bcccb5e", 2021-02-10T11:38:18.550 INFO:teuthology.orchestra.run.gibba002.stdout: "up": 1, 2021-02-10T11:38:18.550 INFO:teuthology.orchestra.run.gibba002.stdout: "in": 1, 2021-02-10T11:38:18.550 INFO:teuthology.orchestra.run.gibba002.stdout: "weight": 1, 2021-02-10T11:38:18.550 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_affinity": 1, 2021-02-10T11:38:18.550 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_begin": 0, 2021-02-10T11:38:18.550 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_end": 0, 2021-02-10T11:38:18.550 INFO:teuthology.orchestra.run.gibba002.stdout: "up_from": 13, 2021-02-10T11:38:18.550 INFO:teuthology.orchestra.run.gibba002.stdout: "up_thru": 170, 2021-02-10T11:38:18.551 INFO:teuthology.orchestra.run.gibba002.stdout: "down_at": 0, 2021-02-10T11:38:18.551 INFO:teuthology.orchestra.run.gibba002.stdout: "lost_at": 0, 2021-02-10T11:38:18.551 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addrs": { 2021-02-10T11:38:18.551 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:38:18.551 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:38:18.551 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:38:18.551 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6802", 2021-02-10T11:38:18.551 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36366 2021-02-10T11:38:18.551 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:38:18.551 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:38:18.551 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:38:18.551 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6805", 2021-02-10T11:38:18.551 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36366 2021-02-10T11:38:18.551 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:38:18.552 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:38:18.552 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:38:18.552 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addrs": { 2021-02-10T11:38:18.552 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:38:18.552 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:38:18.552 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:38:18.552 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6807", 2021-02-10T11:38:18.552 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36366 2021-02-10T11:38:18.552 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:38:18.552 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:38:18.552 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:38:18.553 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6810", 2021-02-10T11:38:18.553 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36366 2021-02-10T11:38:18.553 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:38:18.553 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:38:18.553 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:38:18.553 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addrs": { 2021-02-10T11:38:18.553 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:38:18.553 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:38:18.553 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:38:18.553 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6819", 2021-02-10T11:38:18.553 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36366 2021-02-10T11:38:18.553 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:38:18.553 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:38:18.554 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:38:18.554 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6822", 2021-02-10T11:38:18.554 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36366 2021-02-10T11:38:18.554 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:38:18.554 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:38:18.554 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:38:18.554 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addrs": { 2021-02-10T11:38:18.554 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:38:18.554 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:38:18.554 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:38:18.554 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6812", 2021-02-10T11:38:18.554 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36366 2021-02-10T11:38:18.555 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:38:18.555 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:38:18.555 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:38:18.555 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6815", 2021-02-10T11:38:18.555 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36366 2021-02-10T11:38:18.555 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:38:18.555 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:38:18.555 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:38:18.555 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addr": "172.21.2.113:6805/36366", 2021-02-10T11:38:18.555 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addr": "172.21.2.113:6810/36366", 2021-02-10T11:38:18.555 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addr": "172.21.2.113:6822/36366", 2021-02-10T11:38:18.555 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addr": "172.21.2.113:6815/36366", 2021-02-10T11:38:18.555 INFO:teuthology.orchestra.run.gibba002.stdout: "state": [ 2021-02-10T11:38:18.555 INFO:teuthology.orchestra.run.gibba002.stdout: "exists", 2021-02-10T11:38:18.556 INFO:teuthology.orchestra.run.gibba002.stdout: "up" 2021-02-10T11:38:18.556 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:38:18.556 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:38:18.556 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:38:18.556 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 6, 2021-02-10T11:38:18.556 INFO:teuthology.orchestra.run.gibba002.stdout: "uuid": "bf67c040-0e50-48be-b102-c3943bc9ebef", 2021-02-10T11:38:18.556 INFO:teuthology.orchestra.run.gibba002.stdout: "up": 1, 2021-02-10T11:38:18.556 INFO:teuthology.orchestra.run.gibba002.stdout: "in": 1, 2021-02-10T11:38:18.556 INFO:teuthology.orchestra.run.gibba002.stdout: "weight": 1, 2021-02-10T11:38:18.556 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_affinity": 1, 2021-02-10T11:38:18.556 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_begin": 0, 2021-02-10T11:38:18.556 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_end": 0, 2021-02-10T11:38:18.556 INFO:teuthology.orchestra.run.gibba002.stdout: "up_from": 13, 2021-02-10T11:38:18.556 INFO:teuthology.orchestra.run.gibba002.stdout: "up_thru": 170, 2021-02-10T11:38:18.557 INFO:teuthology.orchestra.run.gibba002.stdout: "down_at": 0, 2021-02-10T11:38:18.557 INFO:teuthology.orchestra.run.gibba002.stdout: "lost_at": 0, 2021-02-10T11:38:18.557 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addrs": { 2021-02-10T11:38:18.557 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:38:18.557 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:38:18.557 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:38:18.557 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6800", 2021-02-10T11:38:18.557 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36367 2021-02-10T11:38:18.557 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:38:18.557 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:38:18.557 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:38:18.557 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6801", 2021-02-10T11:38:18.557 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36367 2021-02-10T11:38:18.558 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:38:18.558 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:38:18.558 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:38:18.558 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addrs": { 2021-02-10T11:38:18.558 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:38:18.558 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:38:18.558 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:38:18.558 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6803", 2021-02-10T11:38:18.558 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36367 2021-02-10T11:38:18.558 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:38:18.558 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:38:18.558 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:38:18.558 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6804", 2021-02-10T11:38:18.558 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36367 2021-02-10T11:38:18.559 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:38:18.559 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:38:18.559 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:38:18.559 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addrs": { 2021-02-10T11:38:18.559 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:38:18.559 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:38:18.559 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:38:18.559 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6813", 2021-02-10T11:38:18.559 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36367 2021-02-10T11:38:18.559 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:38:18.559 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:38:18.559 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:38:18.559 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6816", 2021-02-10T11:38:18.559 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36367 2021-02-10T11:38:18.560 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:38:18.560 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:38:18.560 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:38:18.560 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addrs": { 2021-02-10T11:38:18.560 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:38:18.560 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:38:18.560 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:38:18.560 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6806", 2021-02-10T11:38:18.560 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36367 2021-02-10T11:38:18.560 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:38:18.560 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:38:18.560 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:38:18.561 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6809", 2021-02-10T11:38:18.561 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36367 2021-02-10T11:38:18.561 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:38:18.561 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:38:18.561 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:38:18.561 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addr": "172.21.2.113:6801/36367", 2021-02-10T11:38:18.561 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addr": "172.21.2.113:6804/36367", 2021-02-10T11:38:18.561 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addr": "172.21.2.113:6816/36367", 2021-02-10T11:38:18.561 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addr": "172.21.2.113:6809/36367", 2021-02-10T11:38:18.561 INFO:teuthology.orchestra.run.gibba002.stdout: "state": [ 2021-02-10T11:38:18.561 INFO:teuthology.orchestra.run.gibba002.stdout: "exists", 2021-02-10T11:38:18.561 INFO:teuthology.orchestra.run.gibba002.stdout: "up" 2021-02-10T11:38:18.561 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:38:18.562 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:38:18.562 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:38:18.562 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 7, 2021-02-10T11:38:18.562 INFO:teuthology.orchestra.run.gibba002.stdout: "uuid": "f5461f80-018d-428f-be89-2eb2bcbe7bb6", 2021-02-10T11:38:18.562 INFO:teuthology.orchestra.run.gibba002.stdout: "up": 1, 2021-02-10T11:38:18.562 INFO:teuthology.orchestra.run.gibba002.stdout: "in": 1, 2021-02-10T11:38:18.562 INFO:teuthology.orchestra.run.gibba002.stdout: "weight": 1, 2021-02-10T11:38:18.562 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_affinity": 1, 2021-02-10T11:38:18.562 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_begin": 0, 2021-02-10T11:38:18.562 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_end": 0, 2021-02-10T11:38:18.562 INFO:teuthology.orchestra.run.gibba002.stdout: "up_from": 13, 2021-02-10T11:38:18.562 INFO:teuthology.orchestra.run.gibba002.stdout: "up_thru": 169, 2021-02-10T11:38:18.562 INFO:teuthology.orchestra.run.gibba002.stdout: "down_at": 0, 2021-02-10T11:38:18.563 INFO:teuthology.orchestra.run.gibba002.stdout: "lost_at": 0, 2021-02-10T11:38:18.563 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addrs": { 2021-02-10T11:38:18.563 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:38:18.563 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:38:18.563 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:38:18.563 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6817", 2021-02-10T11:38:18.563 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36365 2021-02-10T11:38:18.563 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:38:18.563 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:38:18.563 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:38:18.563 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6821", 2021-02-10T11:38:18.563 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36365 2021-02-10T11:38:18.563 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:38:18.564 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:38:18.564 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:38:18.564 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addrs": { 2021-02-10T11:38:18.564 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:38:18.564 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:38:18.564 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:38:18.564 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6823", 2021-02-10T11:38:18.564 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36365 2021-02-10T11:38:18.564 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:38:18.564 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:38:18.564 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:38:18.564 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6825", 2021-02-10T11:38:18.564 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36365 2021-02-10T11:38:18.564 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:38:18.565 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:38:18.565 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:38:18.565 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addrs": { 2021-02-10T11:38:18.565 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:38:18.565 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:38:18.565 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:38:18.565 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6830", 2021-02-10T11:38:18.565 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36365 2021-02-10T11:38:18.565 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:38:18.565 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:38:18.565 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:38:18.565 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6831", 2021-02-10T11:38:18.565 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36365 2021-02-10T11:38:18.565 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:38:18.566 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:38:18.566 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:38:18.566 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addrs": { 2021-02-10T11:38:18.566 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:38:18.566 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:38:18.566 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:38:18.566 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6827", 2021-02-10T11:38:18.566 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36365 2021-02-10T11:38:18.566 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:38:18.566 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:38:18.566 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:38:18.566 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6829", 2021-02-10T11:38:18.567 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36365 2021-02-10T11:38:18.567 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:38:18.567 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:38:18.567 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:38:18.567 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addr": "172.21.2.113:6821/36365", 2021-02-10T11:38:18.567 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addr": "172.21.2.113:6825/36365", 2021-02-10T11:38:18.567 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addr": "172.21.2.113:6831/36365", 2021-02-10T11:38:18.567 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addr": "172.21.2.113:6829/36365", 2021-02-10T11:38:18.567 INFO:teuthology.orchestra.run.gibba002.stdout: "state": [ 2021-02-10T11:38:18.567 INFO:teuthology.orchestra.run.gibba002.stdout: "exists", 2021-02-10T11:38:18.567 INFO:teuthology.orchestra.run.gibba002.stdout: "up" 2021-02-10T11:38:18.567 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:38:18.567 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:38:18.567 INFO:teuthology.orchestra.run.gibba002.stdout: ], 2021-02-10T11:38:18.568 INFO:teuthology.orchestra.run.gibba002.stdout: "osd_xinfo": [ 2021-02-10T11:38:18.568 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:38:18.568 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 0, 2021-02-10T11:38:18.568 INFO:teuthology.orchestra.run.gibba002.stdout: "down_stamp": "0.000000", 2021-02-10T11:38:18.568 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_probability": 0, 2021-02-10T11:38:18.568 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_interval": 0, 2021-02-10T11:38:18.568 INFO:teuthology.orchestra.run.gibba002.stdout: "features": 4611087854035861503, 2021-02-10T11:38:18.568 INFO:teuthology.orchestra.run.gibba002.stdout: "old_weight": 0 2021-02-10T11:38:18.568 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:38:18.568 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:38:18.568 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 1, 2021-02-10T11:38:18.568 INFO:teuthology.orchestra.run.gibba002.stdout: "down_stamp": "0.000000", 2021-02-10T11:38:18.568 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_probability": 0, 2021-02-10T11:38:18.569 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_interval": 0, 2021-02-10T11:38:18.569 INFO:teuthology.orchestra.run.gibba002.stdout: "features": 4611087854035861503, 2021-02-10T11:38:18.569 INFO:teuthology.orchestra.run.gibba002.stdout: "old_weight": 0 2021-02-10T11:38:18.569 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:38:18.569 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:38:18.569 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 2, 2021-02-10T11:38:18.569 INFO:teuthology.orchestra.run.gibba002.stdout: "down_stamp": "0.000000", 2021-02-10T11:38:18.569 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_probability": 0, 2021-02-10T11:38:18.569 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_interval": 0, 2021-02-10T11:38:18.569 INFO:teuthology.orchestra.run.gibba002.stdout: "features": 4611087854035861503, 2021-02-10T11:38:18.569 INFO:teuthology.orchestra.run.gibba002.stdout: "old_weight": 0 2021-02-10T11:38:18.569 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:38:18.569 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:38:18.570 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 3, 2021-02-10T11:38:18.570 INFO:teuthology.orchestra.run.gibba002.stdout: "down_stamp": "0.000000", 2021-02-10T11:38:18.570 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_probability": 0, 2021-02-10T11:38:18.570 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_interval": 0, 2021-02-10T11:38:18.570 INFO:teuthology.orchestra.run.gibba002.stdout: "features": 4611087854035861503, 2021-02-10T11:38:18.570 INFO:teuthology.orchestra.run.gibba002.stdout: "old_weight": 0 2021-02-10T11:38:18.570 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:38:18.570 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:38:18.570 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 4, 2021-02-10T11:38:18.570 INFO:teuthology.orchestra.run.gibba002.stdout: "down_stamp": "0.000000", 2021-02-10T11:38:18.570 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_probability": 0, 2021-02-10T11:38:18.570 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_interval": 0, 2021-02-10T11:38:18.571 INFO:teuthology.orchestra.run.gibba002.stdout: "features": 4611087854035861503, 2021-02-10T11:38:18.571 INFO:teuthology.orchestra.run.gibba002.stdout: "old_weight": 0 2021-02-10T11:38:18.571 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:38:18.571 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:38:18.571 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 5, 2021-02-10T11:38:18.571 INFO:teuthology.orchestra.run.gibba002.stdout: "down_stamp": "0.000000", 2021-02-10T11:38:18.571 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_probability": 0, 2021-02-10T11:38:18.571 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_interval": 0, 2021-02-10T11:38:18.571 INFO:teuthology.orchestra.run.gibba002.stdout: "features": 4611087854035861503, 2021-02-10T11:38:18.571 INFO:teuthology.orchestra.run.gibba002.stdout: "old_weight": 0 2021-02-10T11:38:18.571 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:38:18.571 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:38:18.571 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 6, 2021-02-10T11:38:18.572 INFO:teuthology.orchestra.run.gibba002.stdout: "down_stamp": "0.000000", 2021-02-10T11:38:18.572 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_probability": 0, 2021-02-10T11:38:18.572 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_interval": 0, 2021-02-10T11:38:18.572 INFO:teuthology.orchestra.run.gibba002.stdout: "features": 4611087854035861503, 2021-02-10T11:38:18.572 INFO:teuthology.orchestra.run.gibba002.stdout: "old_weight": 0 2021-02-10T11:38:18.572 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:38:18.572 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:38:18.572 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 7, 2021-02-10T11:38:18.572 INFO:teuthology.orchestra.run.gibba002.stdout: "down_stamp": "0.000000", 2021-02-10T11:38:18.572 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_probability": 0, 2021-02-10T11:38:18.572 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_interval": 0, 2021-02-10T11:38:18.573 INFO:teuthology.orchestra.run.gibba002.stdout: "features": 4611087854035861503, 2021-02-10T11:38:18.573 INFO:teuthology.orchestra.run.gibba002.stdout: "old_weight": 0 2021-02-10T11:38:18.573 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:38:18.573 INFO:teuthology.orchestra.run.gibba002.stdout: ], 2021-02-10T11:38:18.573 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_upmap": [], 2021-02-10T11:38:18.573 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_upmap_items": [], 2021-02-10T11:38:18.573 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_temp": [], 2021-02-10T11:38:18.573 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_temp": [], 2021-02-10T11:38:18.573 INFO:teuthology.orchestra.run.gibba002.stdout: "blacklist": {}, 2021-02-10T11:38:18.573 INFO:teuthology.orchestra.run.gibba002.stdout: "erasure_code_profiles": { 2021-02-10T11:38:18.573 INFO:teuthology.orchestra.run.gibba002.stdout: "default": { 2021-02-10T11:38:18.573 INFO:teuthology.orchestra.run.gibba002.stdout: "crush-failure-domain": "osd", 2021-02-10T11:38:18.573 INFO:teuthology.orchestra.run.gibba002.stdout: "k": "2", 2021-02-10T11:38:18.573 INFO:teuthology.orchestra.run.gibba002.stdout: "m": "1", 2021-02-10T11:38:18.574 INFO:teuthology.orchestra.run.gibba002.stdout: "plugin": "jerasure", 2021-02-10T11:38:18.574 INFO:teuthology.orchestra.run.gibba002.stdout: "ruleset-failure-domain": "osd", 2021-02-10T11:38:18.574 INFO:teuthology.orchestra.run.gibba002.stdout: "technique": "reed_sol_van" 2021-02-10T11:38:18.574 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:38:18.574 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:38:18.574 INFO:teuthology.orchestra.run.gibba002.stdout: "removed_snaps_queue": [], 2021-02-10T11:38:18.574 INFO:teuthology.orchestra.run.gibba002.stdout: "new_removed_snaps": [], 2021-02-10T11:38:18.574 INFO:teuthology.orchestra.run.gibba002.stdout: "new_purged_snaps": [], 2021-02-10T11:38:18.574 INFO:teuthology.orchestra.run.gibba002.stdout: "crush_node_flags": {}, 2021-02-10T11:38:18.574 INFO:teuthology.orchestra.run.gibba002.stdout: "device_class_flags": {} 2021-02-10T11:38:18.574 INFO:teuthology.orchestra.run.gibba002.stdout:} 2021-02-10T11:38:18.576 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:18.498 7f980aea7700 1 -- 172.21.2.102:0/1287315175 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f97e80186c0 msgr2=0x7f97e801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:18.576 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:18.498 7f980aea7700 1 --2- 172.21.2.102:0/1287315175 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f97e80186c0 0x7f97e801abc0 crc :-1 s=READY pgs=1342 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:18.576 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:18.498 7f980aea7700 1 -- 172.21.2.102:0/1287315175 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f98041c6750 msgr2=0x7f98040b17a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:18.576 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:18.498 7f980aea7700 1 --2- 172.21.2.102:0/1287315175 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f98041c6750 0x7f98040b17a0 secure :-1 s=READY pgs=3061 cs=0 l=1 rev1=1 rx=0x7f97f4006ed0 tx=0x7f97f400b960).stop 2021-02-10T11:38:18.576 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:18.499 7f980aea7700 1 -- 172.21.2.102:0/1287315175 shutdown_connections 2021-02-10T11:38:18.576 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:18.499 7f980aea7700 1 --2- 172.21.2.102:0/1287315175 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f97e80186c0 0x7f97e801abc0 unknown :-1 s=CLOSED pgs=1342 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:18.576 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:18.499 7f980aea7700 1 --2- 172.21.2.102:0/1287315175 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f98040b1dc0 0x7f980409bfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:18.576 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:18.499 7f980aea7700 1 --2- 172.21.2.102:0/1287315175 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f98041c6750 0x7f98040b17a0 unknown :-1 s=CLOSED pgs=3061 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:18.576 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:18.499 7f980aea7700 1 --2- 172.21.2.102:0/1287315175 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f98041c6350 0x7f980409a830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:18.576 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:18.499 7f980aea7700 1 -- 172.21.2.102:0/1287315175 >> 172.21.2.102:0/1287315175 conn(0x7f98040a0450 msgr2=0x7f98040ec020 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:18.577 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:18.499 7f980aea7700 1 -- 172.21.2.102:0/1287315175 shutdown_connections 2021-02-10T11:38:18.577 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:18.499 7f980aea7700 1 -- 172.21.2.102:0/1287315175 wait complete. 2021-02-10T11:38:18.577 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:38:19.005 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:18.987 7fe7ead19700 1 Processor -- start 2021-02-10T11:38:19.005 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:18.987 7fe7ead19700 1 -- start start 2021-02-10T11:38:19.005 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:18.987 7fe7ead19700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fe7e40d0340 0x7fe7e40ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:19.005 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:18.987 7fe7ead19700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe7e40d1c70 0x7fe7e40efc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:19.005 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:18.987 7fe7ead19700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7e40f0b10 con 0x7fe7e40d1870 2021-02-10T11:38:19.005 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:18.987 7fe7ead19700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7e40e5ed0 con 0x7fe7e40d0340 2021-02-10T11:38:19.005 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:18.987 7fe7ead19700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7e41b0be0 con 0x7fe7e40d1c70 2021-02-10T11:38:19.005 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:18.987 7fe7e8b35700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fe7e40d1870 0x7fe7e41b03f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:59806/0 (socket says 172.21.2.102:59806) 2021-02-10T11:38:19.005 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:18.987 7fe7e8b35700 1 -- 172.21.2.102:0/3570966240 learned_addr learned my addr 172.21.2.102:0/3570966240 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:38:19.006 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:18.987 7fe7e3fff700 1 --2- 172.21.2.102:0/3570966240 >> v2:172.21.2.102:3300/0 conn(0x7fe7e40d0340 0x7fe7e40ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:19.006 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:18.987 7fe7e37fe700 1 -- 172.21.2.102:0/3570966240 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3456763499 0 0) 0x7fe7cc001180 con 0x7fe7e40d1870 2021-02-10T11:38:19.006 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:18.988 7fe7e37fe700 1 -- 172.21.2.102:0/3570966240 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7d0002fe0 con 0x7fe7e40d1870 2021-02-10T11:38:19.006 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:18.988 7fe7e3fff700 1 -- 172.21.2.102:0/3570966240 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe7e40d1c70 msgr2=0x7fe7e40efc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:19.006 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:18.988 7fe7e3fff700 1 --2- 172.21.2.102:0/3570966240 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe7e40d1c70 0x7fe7e40efc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:19.006 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:18.988 7fe7e3fff700 1 -- 172.21.2.102:0/3570966240 >> v1:172.21.2.102:6789/0 conn(0x7fe7e40d1870 legacy=0x7fe7e41b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:19.006 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:18.988 7fe7e3fff700 1 -- 172.21.2.102:0/3570966240 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7e40f20a0 con 0x7fe7e40d0340 2021-02-10T11:38:19.006 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:18.988 7fe7e3fff700 1 --2- 172.21.2.102:0/3570966240 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe7e40d0340 0x7fe7e40ef640 secure :-1 s=READY pgs=3063 cs=0 l=1 rev1=1 rx=0x7fe7dc006650 tx=0x7fe7dc009a80).ready entity=mon.0 client_cookie=460196b10c937092 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:19.006 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:18.988 7fe7e37fe700 1 -- 172.21.2.102:0/3570966240 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe7dc00c0c0 con 0x7fe7e40d0340 2021-02-10T11:38:19.006 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:18.988 7fe7e37fe700 1 -- 172.21.2.102:0/3570966240 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7dc00c340 con 0x7fe7e40d0340 2021-02-10T11:38:19.006 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:18.988 7fe7e37fe700 1 -- 172.21.2.102:0/3570966240 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe7dc00c7a0 con 0x7fe7e40d0340 2021-02-10T11:38:19.006 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:18.988 7fe7ead19700 1 -- 172.21.2.102:0/3570966240 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe7e40d0340 msgr2=0x7fe7e40ef640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:19.006 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:18.988 7fe7ead19700 1 --2- 172.21.2.102:0/3570966240 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe7e40d0340 0x7fe7e40ef640 secure :-1 s=READY pgs=3063 cs=0 l=1 rev1=1 rx=0x7fe7dc006650 tx=0x7fe7dc009a80).stop 2021-02-10T11:38:19.007 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:18.988 7fe7ead19700 1 -- 172.21.2.102:0/3570966240 shutdown_connections 2021-02-10T11:38:19.007 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:18.988 7fe7ead19700 1 --2- 172.21.2.102:0/3570966240 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe7e40d0340 0x7fe7e40ef640 unknown :-1 s=CLOSED pgs=3063 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:19.007 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:18.988 7fe7ead19700 1 --2- 172.21.2.102:0/3570966240 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe7e40d1c70 0x7fe7e40efc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:19.007 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:18.988 7fe7ead19700 1 -- 172.21.2.102:0/3570966240 >> 172.21.2.102:0/3570966240 conn(0x7fe7e40ce450 msgr2=0x7fe7e40d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:19.007 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:18.988 7fe7ead19700 1 -- 172.21.2.102:0/3570966240 shutdown_connections 2021-02-10T11:38:19.007 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:18.988 7fe7ead19700 1 -- 172.21.2.102:0/3570966240 wait complete. 2021-02-10T11:38:19.009 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:19.003 7fe7ead19700 1 Processor -- start 2021-02-10T11:38:19.009 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:19.003 7fe7ead19700 1 -- start start 2021-02-10T11:38:19.009 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:19.003 7fe7ead19700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe7e40b1ea0 0x7fe7e40b22a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:19.009 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:19.003 7fe7ead19700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe7e40ebe30 0x7fe7e40ec230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:19.009 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:19.003 7fe7ead19700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe7e40ec850 0x7fe7e40ecc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:19.009 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:19.003 7fe7ead19700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7e409ae10 con 0x7fe7e40ec850 2021-02-10T11:38:19.009 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:19.003 7fe7ead19700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7e41afe10 con 0x7fe7e40ebe30 2021-02-10T11:38:19.009 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:19.003 7fe7ead19700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7e41af800 con 0x7fe7e40b1ea0 2021-02-10T11:38:19.009 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:19.004 7fe7e9336700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe7e40ec850 0x7fe7e40ecc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:19.009 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:19.004 7fe7e9336700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe7e40ec850 0x7fe7e40ecc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:39356/0 (socket says 172.21.2.102:39356) 2021-02-10T11:38:19.010 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:19.004 7fe7e9336700 1 -- 172.21.2.102:0/2696214012 learned_addr learned my addr 172.21.2.102:0/2696214012 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:38:19.010 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:19.004 7fe7e9336700 1 -- 172.21.2.102:0/2696214012 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe7e40b1ea0 msgr2=0x7fe7e40b22a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:19.010 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:19.004 7fe7e9336700 1 --2- 172.21.2.102:0/2696214012 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe7e40b1ea0 0x7fe7e40b22a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:19.010 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:19.004 7fe7e9336700 1 -- 172.21.2.102:0/2696214012 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe7e40ebe30 msgr2=0x7fe7e40ec230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:19.010 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:19.004 7fe7e9336700 1 --2- 172.21.2.102:0/2696214012 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe7e40ebe30 0x7fe7e40ec230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:19.011 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:19.004 7fe7e9336700 1 -- 172.21.2.102:0/2696214012 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7e40ed850 con 0x7fe7e40ec850 2021-02-10T11:38:19.011 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:19.004 7fe7e9336700 1 --2- 172.21.2.102:0/2696214012 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe7e40ec850 0x7fe7e40ecc50 secure :-1 s=READY pgs=3064 cs=0 l=1 rev1=1 rx=0x7fe7d4009980 tx=0x7fe7d400bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:19.011 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:19.004 7fe7e17fa700 1 -- 172.21.2.102:0/2696214012 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe7d4005870 con 0x7fe7e40ec850 2021-02-10T11:38:19.011 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:19.004 7fe7e17fa700 1 -- 172.21.2.102:0/2696214012 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7d4005b30 con 0x7fe7e40ec850 2021-02-10T11:38:19.011 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:19.004 7fe7ead19700 1 -- 172.21.2.102:0/2696214012 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7e40ed850 con 0x7fe7e40ec850 2021-02-10T11:38:19.011 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:19.004 7fe7e17fa700 1 -- 172.21.2.102:0/2696214012 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe7d4006030 con 0x7fe7e40ec850 2021-02-10T11:38:19.011 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:19.004 7fe7ead19700 1 -- 172.21.2.102:0/2696214012 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7e40ed850 con 0x7fe7e40ec850 2021-02-10T11:38:19.011 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:19.005 7fe7e17fa700 1 -- 172.21.2.102:0/2696214012 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fe7d4005d10 con 0x7fe7e40ec850 2021-02-10T11:38:19.011 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:19.005 7fe7e17fa700 1 --2- 172.21.2.102:0/2696214012 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe7c8018730 0x7fe7c801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:19.011 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:19.005 7fe7e17fa700 1 -- 172.21.2.102:0/2696214012 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fe7d40329c0 con 0x7fe7e40ec850 2021-02-10T11:38:19.011 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:19.005 7fe7e8b35700 1 --2- 172.21.2.102:0/2696214012 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe7c8018730 0x7fe7c801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:19.011 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:19.006 7fe7e8b35700 1 --2- 172.21.2.102:0/2696214012 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe7c8018730 0x7fe7c801abc0 crc :-1 s=READY pgs=1343 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:19.039 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:19.037 7fe7ead19700 1 -- 172.21.2.102:0/2696214012 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7e40ebc00 con 0x7fe7e40ec850 2021-02-10T11:38:19.041 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:19.039 7fe7e17fa700 1 -- 172.21.2.102:0/2696214012 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fe7d400be10 con 0x7fe7e40ec850 2021-02-10T11:38:19.170 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:19.167 7fe7ead19700 1 -- 172.21.2.102:0/2696214012 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe7e40f0b10 con 0x7fe7e40ec850 2021-02-10T11:38:19.170 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:19.168 7fe7e17fa700 1 -- 172.21.2.102:0/2696214012 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 150 v150) v1 ==== 95+0+2108 (secure 0 0 0) 0x7fe7d4006f20 con 0x7fe7e40ec850 2021-02-10T11:38:19.171 INFO:teuthology.orchestra.run.gibba002.stderr:dumped fsmap epoch 150 2021-02-10T11:38:19.171 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:38:19.171 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":150,"default_fscid":18,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":12732,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.102:6835/3244979225","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6834","nonce":3244979225},{"type":"v1","addr":"172.21.2.102:6835","nonce":3244979225}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":145}],"filesystems":[{"mdsmap":{"epoch":149,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:37:36.348132","modified":"2021-02-10 11:37:38.355444","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8665},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8665":{"gid":8665,"name":"b","rank":0,"incarnation":146,"state":"up:active","state_seq":3,"addr":"172.21.2.113:6833/1669982519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6832","nonce":1669982519},{"type":"v1","addr":"172.21.2.113:6833","nonce":1669982519}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[37],"metadata_pool":36,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":18}]} 2021-02-10T11:38:19.172 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:19.170 7fe7ead19700 1 -- 172.21.2.102:0/2696214012 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe7c8018730 msgr2=0x7fe7c801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:19.172 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:19.170 7fe7ead19700 1 --2- 172.21.2.102:0/2696214012 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe7c8018730 0x7fe7c801abc0 crc :-1 s=READY pgs=1343 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:19.172 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:19.170 7fe7ead19700 1 -- 172.21.2.102:0/2696214012 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe7e40ec850 msgr2=0x7fe7e40ecc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:19.173 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:19.170 7fe7ead19700 1 --2- 172.21.2.102:0/2696214012 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe7e40ec850 0x7fe7e40ecc50 secure :-1 s=READY pgs=3064 cs=0 l=1 rev1=1 rx=0x7fe7d4009980 tx=0x7fe7d400bcd0).stop 2021-02-10T11:38:19.173 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:19.170 7fe7ead19700 1 -- 172.21.2.102:0/2696214012 shutdown_connections 2021-02-10T11:38:19.173 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:19.170 7fe7ead19700 1 --2- 172.21.2.102:0/2696214012 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe7e40ec850 0x7fe7e40ecc50 unknown :-1 s=CLOSED pgs=3064 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:19.173 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:19.170 7fe7ead19700 1 --2- 172.21.2.102:0/2696214012 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe7c8018730 0x7fe7c801abc0 unknown :-1 s=CLOSED pgs=1343 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:19.173 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:19.170 7fe7ead19700 1 --2- 172.21.2.102:0/2696214012 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe7e40ebe30 0x7fe7e40ec230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:19.173 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:19.170 7fe7ead19700 1 --2- 172.21.2.102:0/2696214012 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe7e40b1ea0 0x7fe7e40b22a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:19.173 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:19.170 7fe7ead19700 1 -- 172.21.2.102:0/2696214012 >> 172.21.2.102:0/2696214012 conn(0x7fe7e419f120 msgr2=0x7fe7e419e480 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:19.174 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:19.170 7fe7ead19700 1 -- 172.21.2.102:0/2696214012 shutdown_connections 2021-02-10T11:38:19.174 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:19.170 7fe7ead19700 1 -- 172.21.2.102:0/2696214012 wait complete. 2021-02-10T11:38:19.181 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs fail cephfs 2021-02-10T11:38:19.612 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:19.590 7f279e148700 1 Processor -- start 2021-02-10T11:38:19.613 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:19.591 7f279e148700 1 -- start start 2021-02-10T11:38:19.613 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:19.591 7f279e148700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f27980d1870 0x7f27981b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:19.613 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:19.591 7f279e148700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f27980f0650 0x7f27980cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:19.613 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:19.591 7f279e148700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27980f0b10 con 0x7f27981b0a10 2021-02-10T11:38:19.613 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:19.591 7f279e148700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27980fe230 con 0x7f27980f0650 2021-02-10T11:38:19.614 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:19.591 7f279e148700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27980e5ed0 con 0x7f27980d1870 2021-02-10T11:38:19.614 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:19.591 7f2797fff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f27980f0650 0x7f27980cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:19.614 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:19.591 7f2797fff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f27980f0650 0x7f27980cfb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:39364/0 (socket says 172.21.2.102:39364) 2021-02-10T11:38:19.614 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:19.591 7f2797fff700 1 -- 172.21.2.102:0/3824454946 learned_addr learned my addr 172.21.2.102:0/3824454946 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:38:19.614 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:19.591 7f2796ffd700 1 --1- 172.21.2.102:0/3824454946 >> v1:172.21.2.102:6789/0 conn(0x7f27981b0a10 0x7f27980cf330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:59820/0 (socket says 172.21.2.102:59820) 2021-02-10T11:38:19.615 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:19.592 7f2797fff700 1 -- 172.21.2.102:0/3824454946 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f27980d1870 msgr2=0x7f27981b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:19.615 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:19.592 7f2797fff700 1 --2- 172.21.2.102:0/3824454946 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f27980d1870 0x7f27981b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:19.615 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:19.592 7f2797fff700 1 -- 172.21.2.102:0/3824454946 >> v1:172.21.2.102:6789/0 conn(0x7f27981b0a10 legacy=0x7f27980cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:19.615 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:19.592 7f2797fff700 1 -- 172.21.2.102:0/3824454946 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27980f20a0 con 0x7f27980f0650 2021-02-10T11:38:19.615 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:19.592 7f2797fff700 1 --2- 172.21.2.102:0/3824454946 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f27980f0650 0x7f27980cfb20 secure :-1 s=READY pgs=3066 cs=0 l=1 rev1=1 rx=0x7f2790006f00 tx=0x7f2790009f10).ready entity=mon.0 client_cookie=34b90b306e37c6e6 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:19.615 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:19.592 7f27967fc700 1 -- 172.21.2.102:0/3824454946 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f279000b3b0 con 0x7f27980f0650 2021-02-10T11:38:19.616 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:19.592 7f27967fc700 1 -- 172.21.2.102:0/3824454946 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f279000b630 con 0x7f27980f0650 2021-02-10T11:38:19.616 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:19.592 7f27967fc700 1 -- 172.21.2.102:0/3824454946 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f279000bb50 con 0x7f27980f0650 2021-02-10T11:38:19.616 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:19.592 7f279e148700 1 -- 172.21.2.102:0/3824454946 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f27980f0650 msgr2=0x7f27980cfb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:19.616 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:19.592 7f279e148700 1 --2- 172.21.2.102:0/3824454946 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f27980f0650 0x7f27980cfb20 secure :-1 s=READY pgs=3066 cs=0 l=1 rev1=1 rx=0x7f2790006f00 tx=0x7f2790009f10).stop 2021-02-10T11:38:19.616 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:19.593 7f279e148700 1 -- 172.21.2.102:0/3824454946 shutdown_connections 2021-02-10T11:38:19.617 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:19.593 7f279e148700 1 --2- 172.21.2.102:0/3824454946 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f27980f0650 0x7f27980cfb20 unknown :-1 s=CLOSED pgs=3066 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:19.617 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:19.593 7f279e148700 1 --2- 172.21.2.102:0/3824454946 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f27980d1870 0x7f27981b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:19.617 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:19.593 7f279e148700 1 -- 172.21.2.102:0/3824454946 >> 172.21.2.102:0/3824454946 conn(0x7f27980ce450 msgr2=0x7f27980d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:19.617 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:19.593 7f279e148700 1 -- 172.21.2.102:0/3824454946 shutdown_connections 2021-02-10T11:38:19.617 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:19.593 7f279e148700 1 -- 172.21.2.102:0/3824454946 wait complete. 2021-02-10T11:38:19.617 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:19.607 7f279e148700 1 Processor -- start 2021-02-10T11:38:19.618 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:19.607 7f279e148700 1 -- start start 2021-02-10T11:38:19.618 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:19.608 7f279e148700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f27980bdda0 0x7f279818b250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:19.618 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:19.608 7f279e148700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f279818b870 0x7f27980c0030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:19.618 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:19.608 7f279e148700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f279818bc70 0x7f27980c0650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:19.618 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:19.608 7f279e148700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f279819d390 con 0x7f27980bdda0 2021-02-10T11:38:19.618 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:19.608 7f279e148700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27980bd740 con 0x7f279818b870 2021-02-10T11:38:19.619 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:19.608 7f279e148700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27980be1a0 con 0x7f279818bc70 2021-02-10T11:38:19.619 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:19.608 7f27977fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f27980bdda0 0x7f279818b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:19.619 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:19.608 7f27977fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f27980bdda0 0x7f279818b250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:39372/0 (socket says 172.21.2.102:39372) 2021-02-10T11:38:19.619 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:19.608 7f27977fe700 1 -- 172.21.2.102:0/1295004042 learned_addr learned my addr 172.21.2.102:0/1295004042 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:38:19.619 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:19.608 7f2796ffd700 1 --2- 172.21.2.102:0/1295004042 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f279818b870 0x7f27980c0030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:19.620 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:19.608 7f27977fe700 1 -- 172.21.2.102:0/1295004042 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f279818bc70 msgr2=0x7f27980c0650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:19.620 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:19.608 7f2797fff700 1 --2- 172.21.2.102:0/1295004042 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f279818bc70 0x7f27980c0650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:19.620 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:19.608 7f27977fe700 1 --2- 172.21.2.102:0/1295004042 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f279818bc70 0x7f27980c0650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:19.620 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:19.608 7f27977fe700 1 -- 172.21.2.102:0/1295004042 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f279818b870 msgr2=0x7f27980c0030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:19.620 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:19.608 7f27977fe700 1 --2- 172.21.2.102:0/1295004042 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f279818b870 0x7f27980c0030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:19.621 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:19.608 7f27977fe700 1 -- 172.21.2.102:0/1295004042 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27980c0c70 con 0x7f27980bdda0 2021-02-10T11:38:19.621 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:19.609 7f27977fe700 1 --2- 172.21.2.102:0/1295004042 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f27980bdda0 0x7f279818b250 secure :-1 s=READY pgs=3067 cs=0 l=1 rev1=1 rx=0x7f278800b530 tx=0x7f278800b8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:19.621 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:19.609 7f278ffff700 1 -- 172.21.2.102:0/1295004042 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f278800bcc0 con 0x7f27980bdda0 2021-02-10T11:38:19.621 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:19.609 7f278ffff700 1 -- 172.21.2.102:0/1295004042 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2788000c90 con 0x7f27980bdda0 2021-02-10T11:38:19.621 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:19.609 7f278ffff700 1 -- 172.21.2.102:0/1295004042 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f27880022a0 con 0x7f27980bdda0 2021-02-10T11:38:19.621 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:19.609 7f279e148700 1 -- 172.21.2.102:0/1295004042 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27980c0c70 con 0x7f27980bdda0 2021-02-10T11:38:19.622 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:19.609 7f279e148700 1 -- 172.21.2.102:0/1295004042 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27980c1200 con 0x7f27980bdda0 2021-02-10T11:38:19.622 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:19.610 7f278ffff700 1 -- 172.21.2.102:0/1295004042 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f2788005310 con 0x7f27980bdda0 2021-02-10T11:38:19.622 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:19.610 7f278ffff700 1 --2- 172.21.2.102:0/1295004042 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f277c0186c0 0x7f277c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:19.622 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:19.610 7f278ffff700 1 -- 172.21.2.102:0/1295004042 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f2788032500 con 0x7f27980bdda0 2021-02-10T11:38:19.622 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:19.610 7f2796ffd700 1 --2- 172.21.2.102:0/1295004042 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f277c0186c0 0x7f277c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:19.623 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:19.610 7f2796ffd700 1 --2- 172.21.2.102:0/1295004042 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f277c0186c0 0x7f277c01abc0 crc :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:19.643 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:19.641 7f279e148700 1 -- 172.21.2.102:0/1295004042 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27980ee0b0 con 0x7f27980bdda0 2021-02-10T11:38:19.645 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:19.643 7f278ffff700 1 -- 172.21.2.102:0/1295004042 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f2788000940 con 0x7f27980bdda0 2021-02-10T11:38:19.774 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:19.772 7f279e148700 1 -- 172.21.2.102:0/1295004042 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f27980ee0b0 con 0x7f27980bdda0 2021-02-10T11:38:20.181 INFO:tasks.ceph.mon.a.gibba002.stderr:2021-02-10 11:38:20.180 7f8dd2755700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-02-10T11:38:20.185 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:20.183 7f278ffff700 1 -- 172.21.2.102:0/1295004042 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v151) v1 ==== 161+0+0 (secure 0 0 0) 0x7f2788005310 con 0x7f27980bdda0 2021-02-10T11:38:20.191 INFO:teuthology.orchestra.run.gibba002.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2021-02-10T11:38:20.193 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:20.190 7f279e148700 1 -- 172.21.2.102:0/1295004042 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f277c0186c0 msgr2=0x7f277c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:20.193 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:20.190 7f279e148700 1 --2- 172.21.2.102:0/1295004042 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f277c0186c0 0x7f277c01abc0 crc :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:20.193 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:20.190 7f279e148700 1 -- 172.21.2.102:0/1295004042 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f27980bdda0 msgr2=0x7f279818b250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:20.193 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:20.190 7f279e148700 1 --2- 172.21.2.102:0/1295004042 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f27980bdda0 0x7f279818b250 secure :-1 s=READY pgs=3067 cs=0 l=1 rev1=1 rx=0x7f278800b530 tx=0x7f278800b8d0).stop 2021-02-10T11:38:20.193 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:20.191 7f279e148700 1 -- 172.21.2.102:0/1295004042 shutdown_connections 2021-02-10T11:38:20.193 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:20.191 7f279e148700 1 --2- 172.21.2.102:0/1295004042 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f279818bc70 0x7f27980c0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:20.193 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:20.191 7f279e148700 1 --2- 172.21.2.102:0/1295004042 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f277c0186c0 0x7f277c01abc0 unknown :-1 s=CLOSED pgs=1344 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:20.193 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:20.191 7f279e148700 1 --2- 172.21.2.102:0/1295004042 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f279818b870 0x7f27980c0030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:20.193 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:20.191 7f279e148700 1 --2- 172.21.2.102:0/1295004042 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f27980bdda0 0x7f279818b250 unknown :-1 s=CLOSED pgs=3067 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:20.193 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:20.191 7f279e148700 1 -- 172.21.2.102:0/1295004042 >> 172.21.2.102:0/1295004042 conn(0x7f27980a6eb0 msgr2=0x7f2798098ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:20.194 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:20.191 7f279e148700 1 -- 172.21.2.102:0/1295004042 shutdown_connections 2021-02-10T11:38:20.194 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:20.191 7f279e148700 1 -- 172.21.2.102:0/1295004042 wait complete. 2021-02-10T11:38:20.201 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:38:20.237 INFO:tasks.ceph.mds.b.gibba013.stdout:starting mds.b at 2021-02-10T11:38:20.632 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:20.612 7f779bae1700 1 Processor -- start 2021-02-10T11:38:20.632 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:20.612 7f779bae1700 1 -- start start 2021-02-10T11:38:20.632 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:20.612 7f779bae1700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f77940d0340 0x7f77940ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:20.632 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:20.612 7f779bae1700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f77940d1c70 0x7f77940efc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:20.632 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:20.612 7f779bae1700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77940f0b10 con 0x7f77940d1870 2021-02-10T11:38:20.632 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:20.612 7f779bae1700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77940e5ed0 con 0x7f77940d0340 2021-02-10T11:38:20.632 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:20.612 7f779bae1700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77941b0be0 con 0x7f77940d1c70 2021-02-10T11:38:20.632 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:20.613 7f779a0fe700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f77940d1c70 0x7f77940efc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:20.632 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:20.613 7f77990fc700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f77940d0340 0x7f77940ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:20.632 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:20.613 7f77998fd700 1 --1- >> v1:172.21.2.113:6789/0 conn(0x7f77940d1870 0x7f77941b03f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.113:6789/0 says I am v1:172.21.2.102:38924/0 (socket says 172.21.2.102:38924) 2021-02-10T11:38:20.632 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:20.613 7f77998fd700 1 -- 172.21.2.102:0/1575122942 learned_addr learned my addr 172.21.2.102:0/1575122942 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:38:20.633 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:20.615 7f77990fc700 1 -- 172.21.2.102:0/1575122942 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f77940d1c70 msgr2=0x7f77940efc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:20.633 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:20.615 7f77990fc700 1 --2- 172.21.2.102:0/1575122942 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f77940d1c70 0x7f77940efc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:20.633 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:20.615 7f77990fc700 1 -- 172.21.2.102:0/1575122942 >> v1:172.21.2.113:6789/0 conn(0x7f77940d1870 legacy=0x7f77941b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:20.633 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:20.615 7f77990fc700 1 -- 172.21.2.102:0/1575122942 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77940f20a0 con 0x7f77940d0340 2021-02-10T11:38:20.633 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:20.615 7f779a0fe700 1 --2- 172.21.2.102:0/1575122942 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f77940d1c70 0x7f77940efc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:38:20.633 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:20.615 7f77990fc700 1 --2- 172.21.2.102:0/1575122942 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f77940d0340 0x7f77940ef640 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 rx=0x7f7788006650 tx=0x7f7788009a80).ready entity=mon.1 client_cookie=a45269502bb916de server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:20.633 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:20.615 7f77988fb700 1 -- 172.21.2.102:0/1575122942 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f778800c0c0 con 0x7f77940d0340 2021-02-10T11:38:20.633 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:20.615 7f77988fb700 1 -- 172.21.2.102:0/1575122942 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f778800c340 con 0x7f77940d0340 2021-02-10T11:38:20.633 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:20.615 7f77988fb700 1 -- 172.21.2.102:0/1575122942 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f778800c860 con 0x7f77940d0340 2021-02-10T11:38:20.633 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:20.615 7f779bae1700 1 -- 172.21.2.102:0/1575122942 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f77940d0340 msgr2=0x7f77940ef640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:20.633 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:20.615 7f779bae1700 1 --2- 172.21.2.102:0/1575122942 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f77940d0340 0x7f77940ef640 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 rx=0x7f7788006650 tx=0x7f7788009a80).stop 2021-02-10T11:38:20.633 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:20.615 7f779bae1700 1 -- 172.21.2.102:0/1575122942 shutdown_connections 2021-02-10T11:38:20.633 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:20.615 7f779bae1700 1 --2- 172.21.2.102:0/1575122942 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f77940d1c70 0x7f77940efc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:20.634 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:20.615 7f779bae1700 1 --2- 172.21.2.102:0/1575122942 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f77940d0340 0x7f77940ef640 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:20.634 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:20.615 7f779bae1700 1 -- 172.21.2.102:0/1575122942 >> 172.21.2.102:0/1575122942 conn(0x7f77940ce450 msgr2=0x7f77940d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:20.634 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:20.615 7f779bae1700 1 -- 172.21.2.102:0/1575122942 shutdown_connections 2021-02-10T11:38:20.634 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:20.615 7f779bae1700 1 -- 172.21.2.102:0/1575122942 wait complete. 2021-02-10T11:38:20.635 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:20.630 7f779bae1700 1 Processor -- start 2021-02-10T11:38:20.635 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:20.630 7f779bae1700 1 -- start start 2021-02-10T11:38:20.635 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:20.630 7f779bae1700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f779418d870 0x7f77940b19b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:20.635 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:20.630 7f779bae1700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f77940b1fd0 0x7f7794099ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:20.635 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:20.630 7f779bae1700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f77940b23d0 0x7f779409a510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:20.635 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:20.630 7f779bae1700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f779418dc70 con 0x7f779418d870 2021-02-10T11:38:20.635 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:20.630 7f779bae1700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7794199e60 con 0x7f77940b1fd0 2021-02-10T11:38:20.636 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:20.630 7f779bae1700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f779409ab30 con 0x7f77940b23d0 2021-02-10T11:38:20.636 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:20.631 7f77998fd700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f779418d870 0x7f77940b19b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:20.636 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:20.631 7f77998fd700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f779418d870 0x7f77940b19b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:39386/0 (socket says 172.21.2.102:39386) 2021-02-10T11:38:20.636 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:20.631 7f77998fd700 1 -- 172.21.2.102:0/1091402997 learned_addr learned my addr 172.21.2.102:0/1091402997 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:38:20.636 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:20.631 7f77998fd700 1 -- 172.21.2.102:0/1091402997 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f77940b23d0 msgr2=0x7f779409a510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:20.636 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:20.631 7f77998fd700 1 --2- 172.21.2.102:0/1091402997 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f77940b23d0 0x7f779409a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:20.636 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:20.631 7f77998fd700 1 -- 172.21.2.102:0/1091402997 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f77940b1fd0 msgr2=0x7f7794099ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:20.636 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:20.631 7f77998fd700 1 --2- 172.21.2.102:0/1091402997 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f77940b1fd0 0x7f7794099ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:20.636 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:20.631 7f77998fd700 1 -- 172.21.2.102:0/1091402997 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f779409ad50 con 0x7f779418d870 2021-02-10T11:38:20.636 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:20.631 7f77998fd700 1 --2- 172.21.2.102:0/1091402997 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f779418d870 0x7f77940b19b0 secure :-1 s=READY pgs=3070 cs=0 l=1 rev1=1 rx=0x7f7790001c60 tx=0x7f779000a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:20.636 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:20.631 7f77867fc700 1 -- 172.21.2.102:0/1091402997 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f779000acc0 con 0x7f779418d870 2021-02-10T11:38:20.636 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:20.631 7f77867fc700 1 -- 172.21.2.102:0/1091402997 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7790004140 con 0x7f779418d870 2021-02-10T11:38:20.636 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:20.631 7f77867fc700 1 -- 172.21.2.102:0/1091402997 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7790004660 con 0x7f779418d870 2021-02-10T11:38:20.637 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:20.631 7f779bae1700 1 -- 172.21.2.102:0/1091402997 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f779409ad50 con 0x7f779418d870 2021-02-10T11:38:20.637 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:20.631 7f779bae1700 1 -- 172.21.2.102:0/1091402997 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f779409ad50 con 0x7f779418d870 2021-02-10T11:38:20.637 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:20.632 7f77867fc700 1 -- 172.21.2.102:0/1091402997 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f7790000da0 con 0x7f779418d870 2021-02-10T11:38:20.637 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:20.632 7f77867fc700 1 --2- 172.21.2.102:0/1091402997 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f777c0186c0 0x7f777c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:20.637 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:20.632 7f77867fc700 1 -- 172.21.2.102:0/1091402997 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(173..173 src has 1..173) v4 ==== 5659+0+0 (secure 0 0 0) 0x7f7790032610 con 0x7f779418d870 2021-02-10T11:38:20.637 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:20.632 7f77990fc700 1 --2- 172.21.2.102:0/1091402997 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f777c0186c0 0x7f777c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:20.637 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:20.632 7f77990fc700 1 --2- 172.21.2.102:0/1091402997 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f777c0186c0 0x7f777c01abc0 crc :-1 s=READY pgs=1345 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:20.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:20.662 7f779bae1700 1 -- 172.21.2.102:0/1091402997 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77940999c0 con 0x7f779418d870 2021-02-10T11:38:20.666 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:20.664 7f77867fc700 1 -- 172.21.2.102:0/1091402997 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f77900009b0 con 0x7f779418d870 2021-02-10T11:38:20.795 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:20.793 7f779bae1700 1 -- 172.21.2.102:0/1091402997 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f77940999c0 con 0x7f779418d870 2021-02-10T11:38:20.795 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:20.793 7f77867fc700 1 -- 172.21.2.102:0/1091402997 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 151 v151) v1 ==== 95+0+1763 (secure 0 0 0) 0x7f7790000d80 con 0x7f779418d870 2021-02-10T11:38:20.796 INFO:teuthology.orchestra.run.gibba002.stderr:dumped fsmap epoch 151 2021-02-10T11:38:20.796 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:38:20.797 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":151,"default_fscid":18,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":12732,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.102:6835/3244979225","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6834","nonce":3244979225},{"type":"v1","addr":"172.21.2.102:6835","nonce":3244979225}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":145}],"filesystems":[{"mdsmap":{"epoch":151,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:37:36.348132","modified":"2021-02-10 11:38:20.180790","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":173,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[37],"metadata_pool":36,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":18}]} 2021-02-10T11:38:20.797 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:20.796 7f779bae1700 1 -- 172.21.2.102:0/1091402997 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f777c0186c0 msgr2=0x7f777c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:20.798 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:20.796 7f779bae1700 1 --2- 172.21.2.102:0/1091402997 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f777c0186c0 0x7f777c01abc0 crc :-1 s=READY pgs=1345 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:20.798 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:20.796 7f779bae1700 1 -- 172.21.2.102:0/1091402997 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f779418d870 msgr2=0x7f77940b19b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:20.798 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:20.796 7f779bae1700 1 --2- 172.21.2.102:0/1091402997 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f779418d870 0x7f77940b19b0 secure :-1 s=READY pgs=3070 cs=0 l=1 rev1=1 rx=0x7f7790001c60 tx=0x7f779000a8d0).stop 2021-02-10T11:38:20.798 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:20.796 7f779bae1700 1 -- 172.21.2.102:0/1091402997 shutdown_connections 2021-02-10T11:38:20.799 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:20.796 7f779bae1700 1 --2- 172.21.2.102:0/1091402997 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f77940b23d0 0x7f779409a510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:20.799 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:20.796 7f779bae1700 1 --2- 172.21.2.102:0/1091402997 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f777c0186c0 0x7f777c01abc0 unknown :-1 s=CLOSED pgs=1345 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:20.799 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:20.796 7f779bae1700 1 --2- 172.21.2.102:0/1091402997 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f77940b1fd0 0x7f7794099ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:20.799 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:20.796 7f779bae1700 1 --2- 172.21.2.102:0/1091402997 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f779418d870 0x7f77940b19b0 unknown :-1 s=CLOSED pgs=3070 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:20.799 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:20.796 7f779bae1700 1 -- 172.21.2.102:0/1091402997 >> 172.21.2.102:0/1091402997 conn(0x7f77940f0550 msgr2=0x7f7794191730 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:20.799 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:20.796 7f779bae1700 1 -- 172.21.2.102:0/1091402997 shutdown_connections 2021-02-10T11:38:20.799 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:20.796 7f779bae1700 1 -- 172.21.2.102:0/1091402997 wait complete. 2021-02-10T11:38:20.807 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2021-02-10T11:38:21.240 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:21.222 7f8a1fa9f700 1 Processor -- start 2021-02-10T11:38:21.240 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:21.222 7f8a1fa9f700 1 -- start start 2021-02-10T11:38:21.240 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:21.222 7f8a1fa9f700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f8a180d0340 0x7f8a180ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:21.241 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:21.222 7f8a1fa9f700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a180f0b10 con 0x7f8a180d1870 2021-02-10T11:38:21.241 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:21.222 7f8a1fa9f700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a181b0be0 con 0x7f8a180d1c70 2021-02-10T11:38:21.241 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:21.222 7f8a1fa9f700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a180e5ed0 con 0x7f8a180d0340 2021-02-10T11:38:21.241 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:21.223 7f8a1d0ba700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f8a180d0340 0x7f8a180ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:21.241 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:21.223 7f8a1d8bb700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f8a180d1870 0x7f8a181b03f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:59846/0 (socket says 172.21.2.102:59846) 2021-02-10T11:38:21.241 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:21.223 7f8a1d8bb700 1 -- 172.21.2.102:0/1233915124 learned_addr learned my addr 172.21.2.102:0/1233915124 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:38:21.241 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:21.223 7f8a1d0ba700 1 -- 172.21.2.102:0/1233915124 >> v1:172.21.2.113:6789/0 conn(0x7f8a180d1c70 legacy=0x7f8a180efc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:21.241 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:21.223 7f8a1d0ba700 1 -- 172.21.2.102:0/1233915124 >> v1:172.21.2.102:6789/0 conn(0x7f8a180d1870 legacy=0x7f8a181b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:21.241 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:21.223 7f8a1d0ba700 1 -- 172.21.2.102:0/1233915124 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a180f0450 con 0x7f8a180d0340 2021-02-10T11:38:21.241 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:21.223 7f8a1d0ba700 1 --2- 172.21.2.102:0/1233915124 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8a180d0340 0x7f8a180ef640 secure :-1 s=READY pgs=3072 cs=0 l=1 rev1=1 rx=0x7f8a0c006650 tx=0x7f8a0c009a80).ready entity=mon.0 client_cookie=c443780a41f12ab2 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:21.241 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:21.223 7f8a1c8b9700 1 -- 172.21.2.102:0/1233915124 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8a0c0009b0 con 0x7f8a180d0340 2021-02-10T11:38:21.241 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:21.223 7f8a1c8b9700 1 -- 172.21.2.102:0/1233915124 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a0c00c0c0 con 0x7f8a180d0340 2021-02-10T11:38:21.242 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:21.223 7f8a1c8b9700 1 -- 172.21.2.102:0/1233915124 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8a0c00c520 con 0x7f8a180d0340 2021-02-10T11:38:21.242 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:21.223 7f8a1fa9f700 1 -- 172.21.2.102:0/1233915124 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8a180d0340 msgr2=0x7f8a180ef640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:21.242 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:21.223 7f8a1fa9f700 1 --2- 172.21.2.102:0/1233915124 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8a180d0340 0x7f8a180ef640 secure :-1 s=READY pgs=3072 cs=0 l=1 rev1=1 rx=0x7f8a0c006650 tx=0x7f8a0c009a80).stop 2021-02-10T11:38:21.242 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:21.224 7f8a1fa9f700 1 -- 172.21.2.102:0/1233915124 shutdown_connections 2021-02-10T11:38:21.242 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:21.224 7f8a1fa9f700 1 --2- 172.21.2.102:0/1233915124 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8a180d0340 0x7f8a180ef640 unknown :-1 s=CLOSED pgs=3072 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:21.242 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:21.224 7f8a1fa9f700 1 -- 172.21.2.102:0/1233915124 >> 172.21.2.102:0/1233915124 conn(0x7f8a180ce450 msgr2=0x7f8a180d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:21.242 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:21.224 7f8a1fa9f700 1 -- 172.21.2.102:0/1233915124 shutdown_connections 2021-02-10T11:38:21.242 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:21.224 7f8a1fa9f700 1 -- 172.21.2.102:0/1233915124 wait complete. 2021-02-10T11:38:21.243 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:21.238 7f8a1fa9f700 1 Processor -- start 2021-02-10T11:38:21.244 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:21.239 7f8a1fa9f700 1 -- start start 2021-02-10T11:38:21.244 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:21.239 7f8a1fa9f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8a18095eb0 0x7f8a180999a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:21.244 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:21.239 7f8a1fa9f700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8a180989f0 0x7f8a18095380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:21.244 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:21.239 7f8a1fa9f700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8a180959a0 0x7f8a180a7b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:21.244 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:21.239 7f8a1fa9f700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a18099420 con 0x7f8a18095eb0 2021-02-10T11:38:21.244 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:21.239 7f8a1fa9f700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a18099fc0 con 0x7f8a180959a0 2021-02-10T11:38:21.244 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:21.239 7f8a1fa9f700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a18098df0 con 0x7f8a180989f0 2021-02-10T11:38:21.244 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:21.239 7f8a1d8bb700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8a18095eb0 0x7f8a180999a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:21.244 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:21.239 7f8a1d8bb700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8a18095eb0 0x7f8a180999a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:39398/0 (socket says 172.21.2.102:39398) 2021-02-10T11:38:21.244 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:21.239 7f8a1d8bb700 1 -- 172.21.2.102:0/2738666907 learned_addr learned my addr 172.21.2.102:0/2738666907 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:38:21.244 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:21.239 7f8a1d8bb700 1 -- 172.21.2.102:0/2738666907 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8a180989f0 msgr2=0x7f8a18095380 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:38:21.244 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:21.239 7f8a1d8bb700 1 --2- 172.21.2.102:0/2738666907 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8a180989f0 0x7f8a18095380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:21.244 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:21.239 7f8a1d8bb700 1 -- 172.21.2.102:0/2738666907 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8a180959a0 msgr2=0x7f8a180a7b60 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:38:21.245 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:21.239 7f8a1d8bb700 1 --2- 172.21.2.102:0/2738666907 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8a180959a0 0x7f8a180a7b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:21.245 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:21.239 7f8a1d8bb700 1 -- 172.21.2.102:0/2738666907 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a18099010 con 0x7f8a18095eb0 2021-02-10T11:38:21.245 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:21.240 7f8a1d8bb700 1 --2- 172.21.2.102:0/2738666907 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8a18095eb0 0x7f8a180999a0 secure :-1 s=READY pgs=3073 cs=0 l=1 rev1=1 rx=0x7f8a14001410 tx=0x7f8a1400a950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:21.245 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:21.240 7f8a0a7fc700 1 -- 172.21.2.102:0/2738666907 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8a1400ad40 con 0x7f8a18095eb0 2021-02-10T11:38:21.245 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:21.240 7f8a0a7fc700 1 -- 172.21.2.102:0/2738666907 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a14000c90 con 0x7f8a18095eb0 2021-02-10T11:38:21.245 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:21.240 7f8a0a7fc700 1 -- 172.21.2.102:0/2738666907 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8a14004590 con 0x7f8a18095eb0 2021-02-10T11:38:21.245 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:21.240 7f8a1fa9f700 1 -- 172.21.2.102:0/2738666907 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a18099010 con 0x7f8a18095eb0 2021-02-10T11:38:21.245 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:21.240 7f8a1fa9f700 1 -- 172.21.2.102:0/2738666907 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a18099010 con 0x7f8a18095eb0 2021-02-10T11:38:21.245 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:21.241 7f8a0a7fc700 1 -- 172.21.2.102:0/2738666907 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f8a140043a0 con 0x7f8a18095eb0 2021-02-10T11:38:21.246 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:21.241 7f8a0a7fc700 1 --2- 172.21.2.102:0/2738666907 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8a000186c0 0x7f8a0001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:21.246 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:21.241 7f8a0a7fc700 1 -- 172.21.2.102:0/2738666907 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(173..173 src has 1..173) v4 ==== 5659+0+0 (secure 0 0 0) 0x7f8a140325a0 con 0x7f8a18095eb0 2021-02-10T11:38:21.246 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:21.241 7f8a1d0ba700 1 --2- 172.21.2.102:0/2738666907 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8a000186c0 0x7f8a0001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:21.246 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:21.241 7f8a1d0ba700 1 --2- 172.21.2.102:0/2738666907 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8a000186c0 0x7f8a0001abc0 crc :-1 s=READY pgs=1346 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:21.275 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:21.272 7f8a1fa9f700 1 -- 172.21.2.102:0/2738666907 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a180ffa10 con 0x7f8a18095eb0 2021-02-10T11:38:21.277 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:21.275 7f8a0a7fc700 1 -- 172.21.2.102:0/2738666907 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f8a14030d00 con 0x7f8a18095eb0 2021-02-10T11:38:21.403 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:21.401 7f8a1fa9f700 1 -- 172.21.2.102:0/2738666907 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f8a180f06f0 con 0x7f8a18095eb0 2021-02-10T11:38:22.196 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:22.194 7f8a0a7fc700 1 -- 172.21.2.102:0/2738666907 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v152) v1 ==== 104+0+0 (secure 0 0 0) 0x7f8a140043a0 con 0x7f8a18095eb0 2021-02-10T11:38:22.224 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:22.222 7f8a1fa9f700 1 -- 172.21.2.102:0/2738666907 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8a000186c0 msgr2=0x7f8a0001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:22.224 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:22.222 7f8a1fa9f700 1 --2- 172.21.2.102:0/2738666907 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8a000186c0 0x7f8a0001abc0 crc :-1 s=READY pgs=1346 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:22.224 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:22.222 7f8a1fa9f700 1 -- 172.21.2.102:0/2738666907 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8a18095eb0 msgr2=0x7f8a180999a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:22.224 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:22.222 7f8a1fa9f700 1 --2- 172.21.2.102:0/2738666907 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8a18095eb0 0x7f8a180999a0 secure :-1 s=READY pgs=3073 cs=0 l=1 rev1=1 rx=0x7f8a14001410 tx=0x7f8a1400a950).stop 2021-02-10T11:38:22.225 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:22.222 7f8a1fa9f700 1 -- 172.21.2.102:0/2738666907 shutdown_connections 2021-02-10T11:38:22.225 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:22.222 7f8a1fa9f700 1 --2- 172.21.2.102:0/2738666907 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8a000186c0 0x7f8a0001abc0 unknown :-1 s=CLOSED pgs=1346 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:22.225 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:22.222 7f8a1fa9f700 1 --2- 172.21.2.102:0/2738666907 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8a180959a0 0x7f8a180a7b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:22.225 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:22.222 7f8a1fa9f700 1 --2- 172.21.2.102:0/2738666907 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8a180989f0 0x7f8a18095380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:22.225 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:22.222 7f8a1fa9f700 1 --2- 172.21.2.102:0/2738666907 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8a18095eb0 0x7f8a180999a0 unknown :-1 s=CLOSED pgs=3073 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:22.225 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:22.222 7f8a1fa9f700 1 -- 172.21.2.102:0/2738666907 >> 172.21.2.102:0/2738666907 conn(0x7f8a180bf100 msgr2=0x7f8a180a1ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:22.225 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:22.222 7f8a1fa9f700 1 -- 172.21.2.102:0/2738666907 shutdown_connections 2021-02-10T11:38:22.225 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:22.222 7f8a1fa9f700 1 -- 172.21.2.102:0/2738666907 wait complete. 2021-02-10T11:38:22.233 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool delete cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2021-02-10T11:38:22.666 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:22.648 7fcc54778700 1 Processor -- start 2021-02-10T11:38:22.667 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:22.648 7fcc54778700 1 -- start start 2021-02-10T11:38:22.667 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:22.648 7fcc54778700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcc4c0d1880 0x7fcc4c1b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:22.667 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:22.648 7fcc54778700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fcc4c1b0af0 0x7fcc4c108950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:22.667 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:22.648 7fcc54778700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc4c109a70 con 0x7fcc4c0d1c80 2021-02-10T11:38:22.667 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:22.648 7fcc54778700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc4c0e5f00 con 0x7fcc4c1b0af0 2021-02-10T11:38:22.667 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:22.648 7fcc54778700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc4c109760 con 0x7fcc4c0d1880 2021-02-10T11:38:22.667 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:22.649 7fcc51d93700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fcc4c1b0af0 0x7fcc4c108950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:22.667 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:22.649 7fcc51d93700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fcc4c1b0af0 0x7fcc4c108950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:39404/0 (socket says 172.21.2.102:39404) 2021-02-10T11:38:22.668 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:22.649 7fcc51d93700 1 -- 172.21.2.102:0/4087671112 learned_addr learned my addr 172.21.2.102:0/4087671112 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:38:22.668 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:22.649 7fcc51d93700 1 -- 172.21.2.102:0/4087671112 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcc4c0d1880 msgr2=0x7fcc4c1b04d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:38:22.668 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:22.649 7fcc51d93700 1 --2- 172.21.2.102:0/4087671112 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcc4c0d1880 0x7fcc4c1b04d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:22.668 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:22.649 7fcc51d93700 1 -- 172.21.2.102:0/4087671112 >> v1:172.21.2.113:6789/0 conn(0x7fcc4c0d1c80 legacy=0x7fcc4c108f70 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:38:22.668 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:22.649 7fcc51d93700 1 -- 172.21.2.102:0/4087671112 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc4c088070 con 0x7fcc4c1b0af0 2021-02-10T11:38:22.668 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:22.649 7fcc51d93700 1 --2- 172.21.2.102:0/4087671112 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcc4c1b0af0 0x7fcc4c108950 secure :-1 s=READY pgs=3074 cs=0 l=1 rev1=1 rx=0x7fcc48006700 tx=0x7fcc48009b00).ready entity=mon.0 client_cookie=dc0d56226ef1ffae server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:22.668 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:22.649 7fcc51592700 1 -- 172.21.2.102:0/4087671112 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcc4800c0c0 con 0x7fcc4c1b0af0 2021-02-10T11:38:22.668 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:22.649 7fcc51592700 1 -- 172.21.2.102:0/4087671112 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc4800c340 con 0x7fcc4c1b0af0 2021-02-10T11:38:22.668 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:22.649 7fcc51592700 1 -- 172.21.2.102:0/4087671112 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcc4800c820 con 0x7fcc4c1b0af0 2021-02-10T11:38:22.668 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:22.649 7fcc54778700 1 -- 172.21.2.102:0/4087671112 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcc4c1b0af0 msgr2=0x7fcc4c108950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:22.668 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:22.649 7fcc54778700 1 --2- 172.21.2.102:0/4087671112 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcc4c1b0af0 0x7fcc4c108950 secure :-1 s=READY pgs=3074 cs=0 l=1 rev1=1 rx=0x7fcc48006700 tx=0x7fcc48009b00).stop 2021-02-10T11:38:22.668 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:22.650 7fcc54778700 1 -- 172.21.2.102:0/4087671112 shutdown_connections 2021-02-10T11:38:22.669 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:22.650 7fcc54778700 1 --2- 172.21.2.102:0/4087671112 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcc4c1b0af0 0x7fcc4c108950 unknown :-1 s=CLOSED pgs=3074 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:22.669 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:22.650 7fcc54778700 1 --2- 172.21.2.102:0/4087671112 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcc4c0d1880 0x7fcc4c1b04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:22.669 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:22.650 7fcc54778700 1 -- 172.21.2.102:0/4087671112 >> 172.21.2.102:0/4087671112 conn(0x7fcc4c0ce430 msgr2=0x7fcc4c0d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:22.669 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:22.650 7fcc54778700 1 -- 172.21.2.102:0/4087671112 shutdown_connections 2021-02-10T11:38:22.669 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:22.650 7fcc54778700 1 -- 172.21.2.102:0/4087671112 wait complete. 2021-02-10T11:38:22.671 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:22.665 7fcc54778700 1 Processor -- start 2021-02-10T11:38:22.671 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:22.665 7fcc54778700 1 -- start start 2021-02-10T11:38:22.671 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:22.665 7fcc54778700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcc4c19e5b0 0x7fcc4c1afc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:22.671 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:22.665 7fcc54778700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fcc4c19e9b0 0x7fcc4c1b0dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:22.671 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:22.665 7fcc54778700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcc4c1b0250 0x7fcc4c1b13e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:22.671 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:22.665 7fcc54778700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc4c195bd0 con 0x7fcc4c1b0250 2021-02-10T11:38:22.671 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:22.665 7fcc54778700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc4c1b0840 con 0x7fcc4c19e9b0 2021-02-10T11:38:22.672 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:22.665 7fcc54778700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc4c19eed0 con 0x7fcc4c19e5b0 2021-02-10T11:38:22.672 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:22.666 7fcc52d95700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcc4c1b0250 0x7fcc4c1b13e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:22.672 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:22.666 7fcc52d95700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcc4c1b0250 0x7fcc4c1b13e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:39412/0 (socket says 172.21.2.102:39412) 2021-02-10T11:38:22.672 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:22.666 7fcc52d95700 1 -- 172.21.2.102:0/3409492078 learned_addr learned my addr 172.21.2.102:0/3409492078 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:38:22.672 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:22.666 7fcc51d93700 1 --2- 172.21.2.102:0/3409492078 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fcc4c19e9b0 0x7fcc4c1b0dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:22.672 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:22.666 7fcc52594700 1 --2- 172.21.2.102:0/3409492078 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcc4c19e5b0 0x7fcc4c1afc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:22.672 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:22.666 7fcc52d95700 1 -- 172.21.2.102:0/3409492078 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcc4c19e5b0 msgr2=0x7fcc4c1afc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:22.672 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:22.666 7fcc52d95700 1 --2- 172.21.2.102:0/3409492078 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcc4c19e5b0 0x7fcc4c1afc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:22.672 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:22.666 7fcc52d95700 1 -- 172.21.2.102:0/3409492078 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fcc4c19e9b0 msgr2=0x7fcc4c1b0dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:22.672 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:22.666 7fcc52d95700 1 --2- 172.21.2.102:0/3409492078 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fcc4c19e9b0 0x7fcc4c1b0dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:22.672 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:22.666 7fcc52d95700 1 -- 172.21.2.102:0/3409492078 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc4c19f0b0 con 0x7fcc4c1b0250 2021-02-10T11:38:22.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:22.666 7fcc52594700 1 --2- 172.21.2.102:0/3409492078 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcc4c19e5b0 0x7fcc4c1afc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:38:22.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:22.666 7fcc52d95700 1 --2- 172.21.2.102:0/3409492078 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcc4c1b0250 0x7fcc4c1b13e0 secure :-1 s=READY pgs=3075 cs=0 l=1 rev1=1 rx=0x7fcc44009940 tx=0x7fcc4400bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:22.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:22.666 7fcc3effd700 1 -- 172.21.2.102:0/3409492078 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcc44003030 con 0x7fcc4c1b0250 2021-02-10T11:38:22.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:22.667 7fcc3effd700 1 -- 172.21.2.102:0/3409492078 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc44001270 con 0x7fcc4c1b0250 2021-02-10T11:38:22.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:22.667 7fcc3effd700 1 -- 172.21.2.102:0/3409492078 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcc44001790 con 0x7fcc4c1b0250 2021-02-10T11:38:22.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:22.667 7fcc54778700 1 -- 172.21.2.102:0/3409492078 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc4c19f0b0 con 0x7fcc4c1b0250 2021-02-10T11:38:22.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:22.667 7fcc54778700 1 -- 172.21.2.102:0/3409492078 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc4c1b1cf0 con 0x7fcc4c1b0250 2021-02-10T11:38:22.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:22.667 7fcc3effd700 1 -- 172.21.2.102:0/3409492078 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fcc44001450 con 0x7fcc4c1b0250 2021-02-10T11:38:22.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:22.667 7fcc3effd700 1 --2- 172.21.2.102:0/3409492078 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcc340186c0 0x7fcc3401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:22.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:22.667 7fcc3effd700 1 -- 172.21.2.102:0/3409492078 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(173..173 src has 1..173) v4 ==== 5659+0+0 (secure 0 0 0) 0x7fcc44032800 con 0x7fcc4c1b0250 2021-02-10T11:38:22.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:22.667 7fcc52594700 1 --2- 172.21.2.102:0/3409492078 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcc340186c0 0x7fcc3401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:22.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:22.668 7fcc52594700 1 --2- 172.21.2.102:0/3409492078 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcc340186c0 0x7fcc3401abc0 crc :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:22.701 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:22.699 7fcc54778700 1 -- 172.21.2.102:0/3409492078 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc4c082aa0 con 0x7fcc4c1b0250 2021-02-10T11:38:22.703 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:22.701 7fcc3effd700 1 -- 172.21.2.102:0/3409492078 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fcc4400be30 con 0x7fcc4c1b0250 2021-02-10T11:38:22.829 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:22.827 7fcc54778700 1 -- 172.21.2.102:0/3409492078 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"pool2": "cephfs_metadata", "prefix": "osd pool delete", "pool": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fcc4c195bd0 con 0x7fcc4c1b0250 2021-02-10T11:38:23.200 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:23.198 7fcc3effd700 1 -- 172.21.2.102:0/3409492078 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"pool2": "cephfs_metadata", "prefix": "osd pool delete", "pool": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v174) v1 ==== 185+0+0 (secure 0 0 0) 0x7fcc44001430 con 0x7fcc4c1b0250 2021-02-10T11:38:23.245 INFO:teuthology.orchestra.run.gibba002.stderr:pool 'cephfs_metadata' removed 2021-02-10T11:38:23.248 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:23.245 7fcc54778700 1 -- 172.21.2.102:0/3409492078 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcc340186c0 msgr2=0x7fcc3401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:23.248 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:23.245 7fcc54778700 1 --2- 172.21.2.102:0/3409492078 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcc340186c0 0x7fcc3401abc0 crc :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:23.248 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:23.245 7fcc54778700 1 -- 172.21.2.102:0/3409492078 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcc4c1b0250 msgr2=0x7fcc4c1b13e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:23.248 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:23.245 7fcc54778700 1 --2- 172.21.2.102:0/3409492078 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcc4c1b0250 0x7fcc4c1b13e0 secure :-1 s=READY pgs=3075 cs=0 l=1 rev1=1 rx=0x7fcc44009940 tx=0x7fcc4400bcf0).stop 2021-02-10T11:38:23.248 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:23.245 7fcc54778700 1 -- 172.21.2.102:0/3409492078 shutdown_connections 2021-02-10T11:38:23.248 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:23.245 7fcc54778700 1 --2- 172.21.2.102:0/3409492078 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcc4c1b0250 0x7fcc4c1b13e0 unknown :-1 s=CLOSED pgs=3075 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:23.248 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:23.245 7fcc54778700 1 --2- 172.21.2.102:0/3409492078 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcc340186c0 0x7fcc3401abc0 unknown :-1 s=CLOSED pgs=1347 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:23.248 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:23.245 7fcc54778700 1 --2- 172.21.2.102:0/3409492078 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fcc4c19e9b0 0x7fcc4c1b0dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:23.248 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:23.245 7fcc54778700 1 --2- 172.21.2.102:0/3409492078 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcc4c19e5b0 0x7fcc4c1afc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:23.248 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:23.245 7fcc54778700 1 -- 172.21.2.102:0/3409492078 >> 172.21.2.102:0/3409492078 conn(0x7fcc4c19f940 msgr2=0x7fcc4c18c290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:23.249 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:23.245 7fcc54778700 1 -- 172.21.2.102:0/3409492078 shutdown_connections 2021-02-10T11:38:23.249 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:23.246 7fcc54778700 1 -- 172.21.2.102:0/3409492078 wait complete. 2021-02-10T11:38:23.256 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool delete cephfs_data cephfs_data --yes-i-really-really-mean-it 2021-02-10T11:38:23.688 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:23.670 7f2ce200d700 1 Processor -- start 2021-02-10T11:38:23.689 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:23.670 7f2ce200d700 1 -- start start 2021-02-10T11:38:23.689 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:23.671 7f2ce200d700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2cdc0d1880 0x7f2cdc1b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:23.689 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:23.671 7f2ce200d700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f2cdc1b0af0 0x7f2cdc108950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:23.689 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:23.671 7f2ce200d700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cdc109a70 con 0x7f2cdc0d1c80 2021-02-10T11:38:23.689 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:23.671 7f2ce200d700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2cdc0e5f00 con 0x7f2cdc1b0af0 2021-02-10T11:38:23.689 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:23.671 7f2ce200d700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cdc109760 con 0x7f2cdc0d1880 2021-02-10T11:38:23.689 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:23.671 7f2cdaffd700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f2cdc1b0af0 0x7f2cdc108950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:23.689 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:23.671 7f2cdaffd700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f2cdc1b0af0 0x7f2cdc108950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:39418/0 (socket says 172.21.2.102:39418) 2021-02-10T11:38:23.689 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:23.671 7f2cdaffd700 1 -- 172.21.2.102:0/2509429655 learned_addr learned my addr 172.21.2.102:0/2509429655 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:38:23.689 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:23.671 7f2cdb7fe700 1 --2- 172.21.2.102:0/2509429655 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2cdc0d1880 0x7f2cdc1b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:23.689 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:23.671 7f2cdaffd700 1 -- 172.21.2.102:0/2509429655 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2cdc0d1880 msgr2=0x7f2cdc1b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:23.690 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:23.671 7f2cdaffd700 1 --2- 172.21.2.102:0/2509429655 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2cdc0d1880 0x7f2cdc1b04d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:23.690 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:23.671 7f2cdaffd700 1 -- 172.21.2.102:0/2509429655 >> v1:172.21.2.113:6789/0 conn(0x7f2cdc0d1c80 legacy=0x7f2cdc108f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:23.690 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:23.671 7f2cdaffd700 1 -- 172.21.2.102:0/2509429655 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cdc088070 con 0x7f2cdc1b0af0 2021-02-10T11:38:23.690 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:23.671 7f2cdaffd700 1 --2- 172.21.2.102:0/2509429655 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2cdc1b0af0 0x7f2cdc108950 secure :-1 s=READY pgs=3076 cs=0 l=1 rev1=1 rx=0x7f2cd0006700 tx=0x7f2cd0009b80).ready entity=mon.0 client_cookie=9193fec1d0274af3 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:23.690 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:23.672 7f2cda7fc700 1 -- 172.21.2.102:0/2509429655 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2cd000c0c0 con 0x7f2cdc1b0af0 2021-02-10T11:38:23.690 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:23.672 7f2cda7fc700 1 -- 172.21.2.102:0/2509429655 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cd000c0c0 con 0x7f2cdc1b0af0 2021-02-10T11:38:23.690 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:23.672 7f2cda7fc700 1 -- 172.21.2.102:0/2509429655 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2cd000c400 con 0x7f2cdc1b0af0 2021-02-10T11:38:23.690 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:23.672 7f2ce200d700 1 -- 172.21.2.102:0/2509429655 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2cdc1b0af0 msgr2=0x7f2cdc108950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:23.690 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:23.672 7f2ce200d700 1 --2- 172.21.2.102:0/2509429655 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2cdc1b0af0 0x7f2cdc108950 secure :-1 s=READY pgs=3076 cs=0 l=1 rev1=1 rx=0x7f2cd0006700 tx=0x7f2cd0009b80).stop 2021-02-10T11:38:23.690 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:23.672 7f2ce200d700 1 -- 172.21.2.102:0/2509429655 shutdown_connections 2021-02-10T11:38:23.690 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:23.672 7f2ce200d700 1 --2- 172.21.2.102:0/2509429655 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2cdc1b0af0 0x7f2cdc108950 unknown :-1 s=CLOSED pgs=3076 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:23.690 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:23.672 7f2ce200d700 1 --2- 172.21.2.102:0/2509429655 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2cdc0d1880 0x7f2cdc1b04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:23.691 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:23.672 7f2ce200d700 1 -- 172.21.2.102:0/2509429655 >> 172.21.2.102:0/2509429655 conn(0x7f2cdc0ce430 msgr2=0x7f2cdc0d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:23.691 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:23.672 7f2ce200d700 1 -- 172.21.2.102:0/2509429655 shutdown_connections 2021-02-10T11:38:23.691 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:23.672 7f2ce200d700 1 -- 172.21.2.102:0/2509429655 wait complete. 2021-02-10T11:38:23.693 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:23.687 7f2ce200d700 1 Processor -- start 2021-02-10T11:38:23.693 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:23.687 7f2ce200d700 1 -- start start 2021-02-10T11:38:23.693 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:23.687 7f2ce200d700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2cdc09a690 0x7f2cdc09aa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:23.693 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:23.687 7f2ce200d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2cdc1b4160 0x7f2cdc1b4560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:23.693 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:23.687 7f2ce200d700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2cdc1b7dd0 0x7f2cdc1b81d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:23.693 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:23.687 7f2ce200d700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cdc1b4b80 con 0x7f2cdc1b4160 2021-02-10T11:38:23.693 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:23.687 7f2ce200d700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cdc1b87f0 con 0x7f2cdc1b7dd0 2021-02-10T11:38:23.693 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:23.687 7f2ce200d700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cdc1b8a10 con 0x7f2cdc09a690 2021-02-10T11:38:23.693 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:23.687 7f2cdb7fe700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2cdc09a690 0x7f2cdc09aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:23.693 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:23.687 7f2cdaffd700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2cdc1b4160 0x7f2cdc1b4560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:23.694 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:23.687 7f2cdaffd700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2cdc1b4160 0x7f2cdc1b4560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:39424/0 (socket says 172.21.2.102:39424) 2021-02-10T11:38:23.694 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:23.687 7f2cdaffd700 1 -- 172.21.2.102:0/888138183 learned_addr learned my addr 172.21.2.102:0/888138183 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:38:23.694 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:23.688 7f2cdbfff700 1 --2- 172.21.2.102:0/888138183 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2cdc1b7dd0 0x7f2cdc1b81d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:23.694 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:23.688 7f2cdaffd700 1 -- 172.21.2.102:0/888138183 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2cdc09a690 msgr2=0x7f2cdc09aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:23.694 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:23.688 7f2cdaffd700 1 --2- 172.21.2.102:0/888138183 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2cdc09a690 0x7f2cdc09aa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:23.694 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:23.688 7f2cdaffd700 1 -- 172.21.2.102:0/888138183 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2cdc1b7dd0 msgr2=0x7f2cdc1b81d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:23.694 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:23.688 7f2cdaffd700 1 --2- 172.21.2.102:0/888138183 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2cdc1b7dd0 0x7f2cdc1b81d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:23.694 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:23.688 7f2cdaffd700 1 -- 172.21.2.102:0/888138183 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cdc1b8c30 con 0x7f2cdc1b4160 2021-02-10T11:38:23.694 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:23.688 7f2cdb7fe700 1 --2- 172.21.2.102:0/888138183 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2cdc09a690 0x7f2cdc09aa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:38:23.694 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:23.688 7f2cdbfff700 1 --2- 172.21.2.102:0/888138183 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2cdc1b7dd0 0x7f2cdc1b81d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:38:23.694 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:23.688 7f2cdaffd700 1 --2- 172.21.2.102:0/888138183 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2cdc1b4160 0x7f2cdc1b4560 secure :-1 s=READY pgs=3077 cs=0 l=1 rev1=1 rx=0x7f2cd0006420 tx=0x7f2cd000b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:23.694 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:23.688 7f2cc3fff700 1 -- 172.21.2.102:0/888138183 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2cd00090d0 con 0x7f2cdc1b4160 2021-02-10T11:38:23.694 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:23.688 7f2cc3fff700 1 -- 172.21.2.102:0/888138183 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cd000bd50 con 0x7f2cdc1b4160 2021-02-10T11:38:23.695 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:23.688 7f2ce200d700 1 -- 172.21.2.102:0/888138183 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2cdc1b8c30 con 0x7f2cdc1b4160 2021-02-10T11:38:23.695 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:23.688 7f2cc3fff700 1 -- 172.21.2.102:0/888138183 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2cd00054d0 con 0x7f2cdc1b4160 2021-02-10T11:38:23.695 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:23.689 7f2ce200d700 1 -- 172.21.2.102:0/888138183 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2cdc1b8c30 con 0x7f2cdc1b4160 2021-02-10T11:38:23.695 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:23.689 7f2cc3fff700 1 -- 172.21.2.102:0/888138183 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f2cd0009440 con 0x7f2cdc1b4160 2021-02-10T11:38:23.695 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:23.689 7f2cc3fff700 1 --2- 172.21.2.102:0/888138183 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2cbc0186c0 0x7f2cbc01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:23.695 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:23.689 7f2cc3fff700 1 -- 172.21.2.102:0/888138183 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(174..174 src has 1..174) v4 ==== 5221+0+0 (secure 0 0 0) 0x7f2cd0032480 con 0x7f2cdc1b4160 2021-02-10T11:38:23.695 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:23.689 7f2cdb7fe700 1 --2- 172.21.2.102:0/888138183 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2cbc0186c0 0x7f2cbc01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:23.695 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:23.690 7f2cdb7fe700 1 --2- 172.21.2.102:0/888138183 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2cbc0186c0 0x7f2cbc01abc0 crc :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:23.723 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:23.721 7f2ce200d700 1 -- 172.21.2.102:0/888138183 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2cdc09a160 con 0x7f2cdc1b4160 2021-02-10T11:38:23.725 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:23.723 7f2cc3fff700 1 -- 172.21.2.102:0/888138183 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f2cd0009030 con 0x7f2cdc1b4160 2021-02-10T11:38:23.851 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:23.849 7f2ce200d700 1 -- 172.21.2.102:0/888138183 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"pool2": "cephfs_data", "prefix": "osd pool delete", "pool": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f2cdc09a160 con 0x7f2cdc1b4160 2021-02-10T11:38:24.207 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:24.204 7f2cc3fff700 1 -- 172.21.2.102:0/888138183 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"pool2": "cephfs_data", "prefix": "osd pool delete", "pool": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v175) v1 ==== 173+0+0 (secure 0 0 0) 0x7f2cd0009440 con 0x7f2cdc1b4160 2021-02-10T11:38:24.217 INFO:teuthology.orchestra.run.gibba002.stderr:pool 'cephfs_data' removed 2021-02-10T11:38:24.219 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:24.217 7f2ce200d700 1 -- 172.21.2.102:0/888138183 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2cbc0186c0 msgr2=0x7f2cbc01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:24.219 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:24.217 7f2ce200d700 1 --2- 172.21.2.102:0/888138183 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2cbc0186c0 0x7f2cbc01abc0 crc :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:24.219 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:24.217 7f2ce200d700 1 -- 172.21.2.102:0/888138183 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2cdc1b4160 msgr2=0x7f2cdc1b4560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:24.219 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:24.217 7f2ce200d700 1 --2- 172.21.2.102:0/888138183 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2cdc1b4160 0x7f2cdc1b4560 secure :-1 s=READY pgs=3077 cs=0 l=1 rev1=1 rx=0x7f2cd0006420 tx=0x7f2cd000b960).stop 2021-02-10T11:38:24.220 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:24.217 7f2ce200d700 1 -- 172.21.2.102:0/888138183 shutdown_connections 2021-02-10T11:38:24.220 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:24.217 7f2ce200d700 1 --2- 172.21.2.102:0/888138183 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2cbc0186c0 0x7f2cbc01abc0 unknown :-1 s=CLOSED pgs=1348 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:24.220 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:24.217 7f2ce200d700 1 --2- 172.21.2.102:0/888138183 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2cdc1b7dd0 0x7f2cdc1b81d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:24.220 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:24.217 7f2ce200d700 1 --2- 172.21.2.102:0/888138183 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2cdc1b4160 0x7f2cdc1b4560 unknown :-1 s=CLOSED pgs=3077 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:24.220 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:24.217 7f2ce200d700 1 --2- 172.21.2.102:0/888138183 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2cdc09a690 0x7f2cdc09aa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:24.220 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:24.217 7f2ce200d700 1 -- 172.21.2.102:0/888138183 >> 172.21.2.102:0/888138183 conn(0x7f2cdc18e870 msgr2=0x7f2cdc18ecc0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:24.220 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:24.217 7f2ce200d700 1 -- 172.21.2.102:0/888138183 shutdown_connections 2021-02-10T11:38:24.220 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:24.217 7f2ce200d700 1 -- 172.21.2.102:0/888138183 wait complete. 2021-02-10T11:38:24.227 INFO:tasks.ceph.mds.a:Restarting daemon 2021-02-10T11:38:24.228 INFO:tasks.ceph.mds.a:Stopping old one... 2021-02-10T11:38:24.228 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2021-02-10T11:38:24.228 INFO:teuthology.orchestra.run:waiting for 300 2021-02-10T11:38:24.228 INFO:tasks.ceph.mds.b:Restarting daemon 2021-02-10T11:38:24.228 INFO:tasks.ceph.mds.b:Stopping old one... 2021-02-10T11:38:24.228 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2021-02-10T11:38:24.228 INFO:teuthology.orchestra.run:waiting for 300 2021-02-10T11:38:24.234 INFO:tasks.ceph.mds.a:Stopped 2021-02-10T11:38:24.234 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2021-02-10T11:38:24.235 INFO:tasks.ceph.mds.b:Stopped 2021-02-10T11:38:24.235 DEBUG:teuthology.orchestra.run.gibba013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2021-02-10T11:38:24.275 INFO:tasks.ceph.mds.a:Started 2021-02-10T11:38:24.276 DEBUG:teuthology.parallel:result is None 2021-02-10T11:38:24.278 INFO:tasks.ceph.mds.b:Started 2021-02-10T11:38:24.278 DEBUG:teuthology.parallel:result is None 2021-02-10T11:38:24.278 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd blacklist clear 2021-02-10T11:38:24.682 INFO:tasks.ceph.mds.b.gibba013.stdout:starting mds.b at 2021-02-10T11:38:24.683 INFO:tasks.ceph.mds.a.gibba002.stdout:starting mds.a at 2021-02-10T11:38:24.705 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:24.687 7f2466955700 1 Processor -- start 2021-02-10T11:38:24.705 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:24.687 7f2466955700 1 -- start start 2021-02-10T11:38:24.706 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:24.688 7f2466955700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f24600d0340 0x7f24600ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:24.706 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:24.688 7f2466955700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f24600d1c70 0x7f24600efc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:24.706 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:24.688 7f2466955700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24600f0b10 con 0x7f24600d1870 2021-02-10T11:38:24.706 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:24.688 7f2466955700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24600e5ed0 con 0x7f24600d1c70 2021-02-10T11:38:24.706 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:24.688 7f2466955700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24601b0be0 con 0x7f24600d0340 2021-02-10T11:38:24.706 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:24.688 7f2464f72700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f24600d1c70 0x7f24600efc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:24.706 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:24.688 7f2464f72700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f24600d1c70 0x7f24600efc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:39444/0 (socket says 172.21.2.102:39444) 2021-02-10T11:38:24.706 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:24.688 7f2464f72700 1 -- 172.21.2.102:0/3333101778 learned_addr learned my addr 172.21.2.102:0/3333101778 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:38:24.706 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:24.688 7f245f7fe700 1 --2- 172.21.2.102:0/3333101778 >> v2:172.21.2.113:3300/0 conn(0x7f24600d0340 0x7f24600ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:24.706 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:24.688 7f2464f72700 1 -- 172.21.2.102:0/3333101778 >> v2:172.21.2.113:3300/0 conn(0x7f24600d0340 msgr2=0x7f24600ef640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:24.706 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:24.688 7f2464f72700 1 --2- 172.21.2.102:0/3333101778 >> v2:172.21.2.113:3300/0 conn(0x7f24600d0340 0x7f24600ef640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:24.706 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:24.688 7f2464f72700 1 -- 172.21.2.102:0/3333101778 >> v1:172.21.2.113:6789/0 conn(0x7f24600d1870 legacy=0x7f24601b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:24.707 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:24.688 7f2464f72700 1 -- 172.21.2.102:0/3333101778 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24600d3360 con 0x7f24600d1c70 2021-02-10T11:38:24.707 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:24.688 7f2464f72700 1 --2- 172.21.2.102:0/3333101778 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f24600d1c70 0x7f24600efc60 secure :-1 s=READY pgs=3079 cs=0 l=1 rev1=1 rx=0x7f2458006f00 tx=0x7f2458009e90).ready entity=mon.0 client_cookie=becf4594b8882cfc server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:24.707 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:24.688 7f245effd700 1 -- 172.21.2.102:0/3333101778 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f245800b300 con 0x7f24600d1c70 2021-02-10T11:38:24.707 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:24.688 7f245effd700 1 -- 172.21.2.102:0/3333101778 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f245800b580 con 0x7f24600d1c70 2021-02-10T11:38:24.707 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:24.688 7f245effd700 1 -- 172.21.2.102:0/3333101778 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f245800b9d0 con 0x7f24600d1c70 2021-02-10T11:38:24.707 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:24.688 7f2466955700 1 -- 172.21.2.102:0/3333101778 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f24600d1c70 msgr2=0x7f24600efc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:24.707 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:24.688 7f2466955700 1 --2- 172.21.2.102:0/3333101778 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f24600d1c70 0x7f24600efc60 secure :-1 s=READY pgs=3079 cs=0 l=1 rev1=1 rx=0x7f2458006f00 tx=0x7f2458009e90).stop 2021-02-10T11:38:24.707 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:24.689 7f2466955700 1 -- 172.21.2.102:0/3333101778 shutdown_connections 2021-02-10T11:38:24.707 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:24.689 7f2466955700 1 --2- 172.21.2.102:0/3333101778 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f24600d1c70 0x7f24600efc60 unknown :-1 s=CLOSED pgs=3079 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:24.707 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:24.689 7f2466955700 1 --2- 172.21.2.102:0/3333101778 >> v2:172.21.2.113:3300/0 conn(0x7f24600d0340 0x7f24600ef640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:24.708 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:24.689 7f2466955700 1 -- 172.21.2.102:0/3333101778 >> 172.21.2.102:0/3333101778 conn(0x7f24600ce450 msgr2=0x7f24600d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:24.708 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:24.689 7f2466955700 1 -- 172.21.2.102:0/3333101778 shutdown_connections 2021-02-10T11:38:24.708 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:24.689 7f2466955700 1 -- 172.21.2.102:0/3333101778 wait complete. 2021-02-10T11:38:24.709 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:24.704 7f2466955700 1 Processor -- start 2021-02-10T11:38:24.710 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:24.704 7f2466955700 1 -- start start 2021-02-10T11:38:24.710 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:24.704 7f2466955700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f24600a2fb0 0x7f24600a33b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:24.710 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:24.704 7f2466955700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f24600a39d0 0x7f24600a4860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:24.710 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:24.704 7f2466955700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f24600a3dd0 0x7f24600a4e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:24.710 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:24.704 7f2466955700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24600a2620 con 0x7f24600a39d0 2021-02-10T11:38:24.710 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:24.704 7f2466955700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24600a2840 con 0x7f24600a2fb0 2021-02-10T11:38:24.710 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:24.704 7f2466955700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24600a2b80 con 0x7f24600a3dd0 2021-02-10T11:38:24.710 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:24.704 7f245f7fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f24600a39d0 0x7f24600a4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:24.710 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:24.704 7f245f7fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f24600a39d0 0x7f24600a4860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:39448/0 (socket says 172.21.2.102:39448) 2021-02-10T11:38:24.710 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:24.704 7f245f7fe700 1 -- 172.21.2.102:0/3832921075 learned_addr learned my addr 172.21.2.102:0/3832921075 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:38:24.710 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:24.705 7f2464f72700 1 --2- 172.21.2.102:0/3832921075 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f24600a3dd0 0x7f24600a4e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:24.711 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:24.705 7f245ffff700 1 --2- 172.21.2.102:0/3832921075 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f24600a2fb0 0x7f24600a33b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:24.711 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:24.705 7f245f7fe700 1 -- 172.21.2.102:0/3832921075 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f24600a3dd0 msgr2=0x7f24600a4e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:24.711 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:24.705 7f245f7fe700 1 --2- 172.21.2.102:0/3832921075 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f24600a3dd0 0x7f24600a4e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:24.711 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:24.705 7f245f7fe700 1 -- 172.21.2.102:0/3832921075 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f24600a2fb0 msgr2=0x7f24600a33b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:24.711 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:24.705 7f245f7fe700 1 --2- 172.21.2.102:0/3832921075 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f24600a2fb0 0x7f24600a33b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:24.711 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:24.705 7f245f7fe700 1 -- 172.21.2.102:0/3832921075 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24600a54a0 con 0x7f24600a39d0 2021-02-10T11:38:24.711 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:24.705 7f2464f72700 1 --2- 172.21.2.102:0/3832921075 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f24600a3dd0 0x7f24600a4e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:38:24.711 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:24.705 7f245f7fe700 1 --2- 172.21.2.102:0/3832921075 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f24600a39d0 0x7f24600a4860 secure :-1 s=READY pgs=3080 cs=0 l=1 rev1=1 rx=0x7f2454006f10 tx=0x7f245400b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:24.711 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:24.705 7f245cff9700 1 -- 172.21.2.102:0/3832921075 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f245400bd80 con 0x7f24600a39d0 2021-02-10T11:38:24.711 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:24.705 7f245cff9700 1 -- 172.21.2.102:0/3832921075 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2454000c90 con 0x7f24600a39d0 2021-02-10T11:38:24.711 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:24.705 7f245cff9700 1 -- 172.21.2.102:0/3832921075 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f24540022b0 con 0x7f24600a39d0 2021-02-10T11:38:24.711 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:24.705 7f2466955700 1 -- 172.21.2.102:0/3832921075 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24600a54a0 con 0x7f24600a39d0 2021-02-10T11:38:24.711 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:24.705 7f2466955700 1 -- 172.21.2.102:0/3832921075 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24600a54a0 con 0x7f24600a39d0 2021-02-10T11:38:24.711 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:24.706 7f245cff9700 1 -- 172.21.2.102:0/3832921075 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f24540052e0 con 0x7f24600a39d0 2021-02-10T11:38:24.712 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:24.706 7f245cff9700 1 --2- 172.21.2.102:0/3832921075 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f24400186c0 0x7f244001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:24.712 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:24.706 7f245cff9700 1 -- 172.21.2.102:0/3832921075 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(175..175 src has 1..175) v4 ==== 4839+0+0 (secure 0 0 0) 0x7f2454032260 con 0x7f24600a39d0 2021-02-10T11:38:24.712 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:24.706 7f245ffff700 1 --2- 172.21.2.102:0/3832921075 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f24400186c0 0x7f244001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:24.712 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:24.706 7f245ffff700 1 --2- 172.21.2.102:0/3832921075 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f24400186c0 0x7f244001abc0 crc :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:24.739 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:24.737 7f2466955700 1 -- 172.21.2.102:0/3832921075 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24600ebc50 con 0x7f24600a39d0 2021-02-10T11:38:24.741 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:24.739 7f245cff9700 1 -- 172.21.2.102:0/3832921075 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f245402d370 con 0x7f24600a39d0 2021-02-10T11:38:24.882 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:24.880 7f2466955700 1 -- 172.21.2.102:0/3832921075 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd blacklist clear"} v 0) v1 -- 0x7f24600a5d20 con 0x7f24600a39d0 2021-02-10T11:38:25.210 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:25.207 7f245cff9700 1 -- 172.21.2.102:0/3832921075 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd blacklist clear"}]=0 removed all blacklist entries v176) v1 ==== 97+0+0 (secure 0 0 0) 0x7f245402d370 con 0x7f24600a39d0 2021-02-10T11:38:25.248 INFO:teuthology.orchestra.run.gibba002.stderr: removed all blacklist entries 2021-02-10T11:38:25.250 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:25.248 7f2466955700 1 -- 172.21.2.102:0/3832921075 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f24400186c0 msgr2=0x7f244001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:25.251 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:25.248 7f2466955700 1 --2- 172.21.2.102:0/3832921075 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f24400186c0 0x7f244001abc0 crc :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:25.251 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:25.248 7f2466955700 1 -- 172.21.2.102:0/3832921075 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f24600a39d0 msgr2=0x7f24600a4860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:25.251 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:25.248 7f2466955700 1 --2- 172.21.2.102:0/3832921075 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f24600a39d0 0x7f24600a4860 secure :-1 s=READY pgs=3080 cs=0 l=1 rev1=1 rx=0x7f2454006f10 tx=0x7f245400b990).stop 2021-02-10T11:38:25.251 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:25.248 7f2466955700 1 -- 172.21.2.102:0/3832921075 shutdown_connections 2021-02-10T11:38:25.251 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:25.248 7f2466955700 1 --2- 172.21.2.102:0/3832921075 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f24600a3dd0 0x7f24600a4e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:25.251 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:25.248 7f2466955700 1 --2- 172.21.2.102:0/3832921075 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f24600a39d0 0x7f24600a4860 unknown :-1 s=CLOSED pgs=3080 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:25.251 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:25.248 7f2466955700 1 --2- 172.21.2.102:0/3832921075 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f24400186c0 0x7f244001abc0 unknown :-1 s=CLOSED pgs=1349 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:25.251 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:25.248 7f2466955700 1 --2- 172.21.2.102:0/3832921075 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f24600a2fb0 0x7f24600a33b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:25.251 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:25.248 7f2466955700 1 -- 172.21.2.102:0/3832921075 >> 172.21.2.102:0/3832921075 conn(0x7f246008f030 msgr2=0x7f24600b5e80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:25.251 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:25.248 7f2466955700 1 -- 172.21.2.102:0/3832921075 shutdown_connections 2021-02-10T11:38:25.251 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:25.248 7f2466955700 1 -- 172.21.2.102:0/3832921075 wait complete. 2021-02-10T11:38:25.258 INFO:tasks.cephfs.cephfs_test_case:['0', '1', '2', '3'] 2021-02-10T11:38:25.259 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth ls --format=json-pretty 2021-02-10T11:38:25.688 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:25.670 7f9a20146700 1 Processor -- start 2021-02-10T11:38:25.688 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:25.670 7f9a20146700 1 -- start start 2021-02-10T11:38:25.688 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:25.671 7f9a20146700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f9a180d1870 0x7f9a181b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:25.689 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:25.671 7f9a20146700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f9a181b0a10 0x7f9a180cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:25.689 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:25.671 7f9a20146700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9a180d1c70 0x7f9a180cf950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:25.689 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:25.671 7f9a20146700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a180e5ed0 con 0x7f9a180d1870 2021-02-10T11:38:25.689 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:25.671 7f9a20146700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a180f0b10 con 0x7f9a181b0a10 2021-02-10T11:38:25.689 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:25.671 7f9a20146700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a180cff70 con 0x7f9a180d1c70 2021-02-10T11:38:25.689 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:25.671 7f9a1df62700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f9a180d1870 0x7f9a181b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:25.689 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:25.671 7f9a1df62700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f9a180d1870 0x7f9a181b03f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:39454/0 (socket says 172.21.2.102:39454) 2021-02-10T11:38:25.689 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:25.671 7f9a1df62700 1 -- 172.21.2.102:0/1292218816 learned_addr learned my addr 172.21.2.102:0/1292218816 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:38:25.689 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:25.671 7f9a1df62700 1 -- 172.21.2.102:0/1292218816 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9a180d1c70 msgr2=0x7f9a180cf950 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:38:25.689 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:25.671 7f9a1df62700 1 --2- 172.21.2.102:0/1292218816 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9a180d1c70 0x7f9a180cf950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:25.689 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:25.671 7f9a1df62700 1 -- 172.21.2.102:0/1292218816 >> v2:172.21.2.113:3300/0 conn(0x7f9a181b0a10 msgr2=0x7f9a180cf330 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:38:25.690 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:25.671 7f9a1df62700 1 --2- 172.21.2.102:0/1292218816 >> v2:172.21.2.113:3300/0 conn(0x7f9a181b0a10 0x7f9a180cf330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:25.690 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:25.671 7f9a1df62700 1 -- 172.21.2.102:0/1292218816 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a180d0150 con 0x7f9a180d1870 2021-02-10T11:38:25.690 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:25.671 7f9a1df62700 1 --2- 172.21.2.102:0/1292218816 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9a180d1870 0x7f9a181b03f0 secure :-1 s=READY pgs=3081 cs=0 l=1 rev1=1 rx=0x7f9a08006a80 tx=0x7f9a08009f00).ready entity=mon.0 client_cookie=3b69dc20584258a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:25.690 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:25.671 7f9a1cf60700 1 -- 172.21.2.102:0/1292218816 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9a0800b5c0 con 0x7f9a180d1870 2021-02-10T11:38:25.690 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:25.671 7f9a1cf60700 1 -- 172.21.2.102:0/1292218816 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a0800b800 con 0x7f9a180d1870 2021-02-10T11:38:25.690 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:25.671 7f9a1cf60700 1 -- 172.21.2.102:0/1292218816 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9a08000d30 con 0x7f9a180d1870 2021-02-10T11:38:25.690 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:25.671 7f9a20146700 1 -- 172.21.2.102:0/1292218816 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9a180d1870 msgr2=0x7f9a181b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:25.690 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:25.671 7f9a20146700 1 --2- 172.21.2.102:0/1292218816 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9a180d1870 0x7f9a181b03f0 secure :-1 s=READY pgs=3081 cs=0 l=1 rev1=1 rx=0x7f9a08006a80 tx=0x7f9a08009f00).stop 2021-02-10T11:38:25.690 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:25.672 7f9a20146700 1 -- 172.21.2.102:0/1292218816 shutdown_connections 2021-02-10T11:38:25.690 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:25.672 7f9a20146700 1 --2- 172.21.2.102:0/1292218816 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9a180d1c70 0x7f9a180cf950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:25.691 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:25.672 7f9a20146700 1 --2- 172.21.2.102:0/1292218816 >> v2:172.21.2.113:3300/0 conn(0x7f9a181b0a10 0x7f9a180cf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:25.691 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:25.672 7f9a20146700 1 --2- 172.21.2.102:0/1292218816 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9a180d1870 0x7f9a181b03f0 unknown :-1 s=CLOSED pgs=3081 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:25.691 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:25.672 7f9a20146700 1 -- 172.21.2.102:0/1292218816 >> 172.21.2.102:0/1292218816 conn(0x7f9a180ce450 msgr2=0x7f9a180d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:25.691 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:25.672 7f9a20146700 1 -- 172.21.2.102:0/1292218816 shutdown_connections 2021-02-10T11:38:25.691 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:25.672 7f9a20146700 1 -- 172.21.2.102:0/1292218816 wait complete. 2021-02-10T11:38:25.692 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:25.687 7f9a20146700 1 Processor -- start 2021-02-10T11:38:25.692 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:25.687 7f9a20146700 1 -- start start 2021-02-10T11:38:25.693 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:25.687 7f9a20146700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9a181b7cf0 0x7f9a1819cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:25.693 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:25.687 7f9a20146700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9a181b80f0 0x7f9a181b84f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:25.693 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:25.687 7f9a20146700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f9a1819da20 0x7f9a1819de20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:25.693 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:25.687 7f9a20146700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a180a77c0 con 0x7f9a181b80f0 2021-02-10T11:38:25.693 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:25.687 7f9a20146700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a1819d4a0 con 0x7f9a1819da20 2021-02-10T11:38:25.693 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:25.687 7f9a20146700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a180954f0 con 0x7f9a181b7cf0 2021-02-10T11:38:25.693 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:25.687 7f9a1d761700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9a181b80f0 0x7f9a181b84f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:25.693 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:25.687 7f9a1d761700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9a181b80f0 0x7f9a181b84f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:39462/0 (socket says 172.21.2.102:39462) 2021-02-10T11:38:25.693 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:25.687 7f9a1d761700 1 -- 172.21.2.102:0/1731666678 learned_addr learned my addr 172.21.2.102:0/1731666678 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:38:25.693 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:25.688 7f9a1d761700 1 -- 172.21.2.102:0/1731666678 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9a181b7cf0 msgr2=0x7f9a1819cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:25.693 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:25.688 7f9a1d761700 1 --2- 172.21.2.102:0/1731666678 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9a181b7cf0 0x7f9a1819cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:25.693 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:25.688 7f9a1d761700 1 -- 172.21.2.102:0/1731666678 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f9a1819da20 msgr2=0x7f9a1819de20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:25.693 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:25.688 7f9a1d761700 1 --2- 172.21.2.102:0/1731666678 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f9a1819da20 0x7f9a1819de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:25.694 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:25.688 7f9a1d761700 1 -- 172.21.2.102:0/1731666678 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a181b8d60 con 0x7f9a181b80f0 2021-02-10T11:38:25.694 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:25.688 7f9a1d761700 1 --2- 172.21.2.102:0/1731666678 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9a181b80f0 0x7f9a181b84f0 secure :-1 s=READY pgs=3082 cs=0 l=1 rev1=1 rx=0x7f9a0c00b5c0 tx=0x7f9a0c00b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:25.694 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:25.688 7f9a167fc700 1 -- 172.21.2.102:0/1731666678 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9a0c0009b0 con 0x7f9a181b80f0 2021-02-10T11:38:25.694 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:25.688 7f9a167fc700 1 -- 172.21.2.102:0/1731666678 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a0c00bd50 con 0x7f9a181b80f0 2021-02-10T11:38:25.694 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:25.688 7f9a167fc700 1 -- 172.21.2.102:0/1731666678 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9a0c0062d0 con 0x7f9a181b80f0 2021-02-10T11:38:25.694 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:25.688 7f9a20146700 1 -- 172.21.2.102:0/1731666678 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a181b8d60 con 0x7f9a181b80f0 2021-02-10T11:38:25.694 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:25.688 7f9a20146700 1 -- 172.21.2.102:0/1731666678 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a181b8d60 con 0x7f9a181b80f0 2021-02-10T11:38:25.694 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:25.689 7f9a167fc700 1 -- 172.21.2.102:0/1731666678 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f9a0c006030 con 0x7f9a181b80f0 2021-02-10T11:38:25.694 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:25.689 7f9a167fc700 1 --2- 172.21.2.102:0/1731666678 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f99f80186c0 0x7f99f801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:25.694 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:25.689 7f9a167fc700 1 -- 172.21.2.102:0/1731666678 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(176..176 src has 1..176) v4 ==== 4753+0+0 (secure 0 0 0) 0x7f9a0c032220 con 0x7f9a181b80f0 2021-02-10T11:38:25.694 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:25.689 7f9a1df62700 1 --2- 172.21.2.102:0/1731666678 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f99f80186c0 0x7f99f801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:25.694 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:25.689 7f9a1df62700 1 --2- 172.21.2.102:0/1731666678 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f99f80186c0 0x7f99f801abc0 crc :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:25.723 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:25.721 7f9a20146700 1 -- 172.21.2.102:0/1731666678 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a180ebc00 con 0x7f9a181b80f0 2021-02-10T11:38:25.725 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:25.724 7f9a167fc700 1 -- 172.21.2.102:0/1731666678 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f9a0c000c40 con 0x7f9a181b80f0 2021-02-10T11:38:25.856 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:25.854 7f9a20146700 1 -- 172.21.2.102:0/1731666678 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f9a180f0b10 con 0x7f9a181b80f0 2021-02-10T11:38:25.857 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:25.854 7f9a167fc700 1 -- 172.21.2.102:0/1731666678 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v99) v1 ==== 80+0+4513 (secure 0 0 0) 0x7f9a0c01a400 con 0x7f9a181b80f0 2021-02-10T11:38:25.857 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:38:25.857 INFO:teuthology.orchestra.run.gibba002.stdout:{ 2021-02-10T11:38:25.857 INFO:teuthology.orchestra.run.gibba002.stdout: "auth_dump": [ 2021-02-10T11:38:25.858 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:38:25.858 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "mds.a", 2021-02-10T11:38:25.858 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAevSNgsgyEARAAvr0cUjjCoEHLxaZSZ2HtVA==", 2021-02-10T11:38:25.858 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:38:25.858 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow", 2021-02-10T11:38:25.858 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:38:25.858 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:38:25.858 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:38:25.858 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:38:25.858 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:38:25.858 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:38:25.859 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "mds.b", 2021-02-10T11:38:25.859 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAevSNgwOFMMRAALfb1fbX56ao1TU8ClnTW1A==", 2021-02-10T11:38:25.859 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:38:25.859 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow", 2021-02-10T11:38:25.859 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:38:25.859 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:38:25.859 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:38:25.859 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:38:25.859 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:38:25.859 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:38:25.859 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.0", 2021-02-10T11:38:25.859 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAmvSNgj+bMMBAAD3HUD6fLiMAg+++iNzLJKQ==", 2021-02-10T11:38:25.859 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:38:25.860 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:38:25.860 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:38:25.860 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:38:25.860 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:38:25.860 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:38:25.860 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:38:25.860 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.1", 2021-02-10T11:38:25.860 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQApvSNgnvc+MRAAAOxqlNwiyLHqNITOrYeFKQ==", 2021-02-10T11:38:25.860 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:38:25.860 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:38:25.860 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:38:25.860 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:38:25.861 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:38:25.861 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:38:25.861 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:38:25.861 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.2", 2021-02-10T11:38:25.861 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAsvSNgZ4SbMRAA76IOYWfbiAuBoM/4k81GxA==", 2021-02-10T11:38:25.861 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:38:25.861 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:38:25.861 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:38:25.861 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:38:25.861 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:38:25.861 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:38:25.861 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:38:25.861 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.3", 2021-02-10T11:38:25.862 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAvvSNgFk3rMRAAWCmKKyw0LDQ+UlAHZAITOw==", 2021-02-10T11:38:25.862 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:38:25.862 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:38:25.862 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:38:25.862 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:38:25.862 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:38:25.862 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:38:25.862 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:38:25.862 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.4", 2021-02-10T11:38:25.862 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQA4vSNgpKcNJBAApJMmvj5YxVHPDOmvmQONOg==", 2021-02-10T11:38:25.862 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:38:25.862 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:38:25.862 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:38:25.863 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:38:25.863 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:38:25.863 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:38:25.863 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:38:25.863 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.5", 2021-02-10T11:38:25.863 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQA7vSNgf2ujJBAAfkuYpyb3WTzc8V5kw+Wktg==", 2021-02-10T11:38:25.863 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:38:25.863 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:38:25.863 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:38:25.863 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:38:25.863 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:38:25.863 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:38:25.863 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:38:25.864 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.6", 2021-02-10T11:38:25.864 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQA+vSNgqw9uJRAAFMU0R+UxU9C3o/9byb3e0g==", 2021-02-10T11:38:25.864 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:38:25.864 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:38:25.864 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:38:25.864 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:38:25.864 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:38:25.864 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:38:25.864 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:38:25.864 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.7", 2021-02-10T11:38:25.864 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQBBvSNgvWSrJRAA2KaCHmLpzovRgYzLvyofTQ==", 2021-02-10T11:38:25.864 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:38:25.864 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:38:25.864 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:38:25.865 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:38:25.865 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:38:25.865 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:38:25.865 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:38:25.865 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "client.0", 2021-02-10T11:38:25.865 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAfvSNgZd9XIBAAvyrvl8fdKuHnJZfczfRoFw==", 2021-02-10T11:38:25.865 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:38:25.865 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow", 2021-02-10T11:38:25.865 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow r", 2021-02-10T11:38:25.865 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:38:25.865 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:38:25.865 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:38:25.865 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:38:25.865 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "client.1", 2021-02-10T11:38:25.866 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAfvSNgSyfXNhAAIduPnF7Vd/6HSkJ0MlcMSA==", 2021-02-10T11:38:25.866 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:38:25.866 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow", 2021-02-10T11:38:25.866 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow r", 2021-02-10T11:38:25.866 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:38:25.866 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:38:25.866 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:38:25.866 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:38:25.866 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "client.2", 2021-02-10T11:38:25.866 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAgvSNgiIG/ERAA9ZTamaTI7ZsPSnxBqjplCg==", 2021-02-10T11:38:25.866 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:38:25.866 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow", 2021-02-10T11:38:25.867 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow r", 2021-02-10T11:38:25.867 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:38:25.867 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:38:25.867 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:38:25.867 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:38:25.867 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "client.3", 2021-02-10T11:38:25.867 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAgvSNgOWpOKBAAlXsEB04Q/GvtL06PqRMLeA==", 2021-02-10T11:38:25.867 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:38:25.867 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow", 2021-02-10T11:38:25.867 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow r", 2021-02-10T11:38:25.868 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:38:25.868 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:38:25.868 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:38:25.868 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:38:25.868 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "client.admin", 2021-02-10T11:38:25.868 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAavSNgO3GQBhAAL+NVnYskQRKVWLgyDPY/xw==", 2021-02-10T11:38:25.868 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:38:25.868 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow *", 2021-02-10T11:38:25.868 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:38:25.868 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:38:25.868 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:38:25.868 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:38:25.868 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:38:25.868 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:38:25.869 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "mgr.x", 2021-02-10T11:38:25.869 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAdvSNgXZZgJhAA7dtBSGzH+DsBwnddZpOBJA==", 2021-02-10T11:38:25.869 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:38:25.869 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow *", 2021-02-10T11:38:25.869 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow profile mgr", 2021-02-10T11:38:25.869 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:38:25.869 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:38:25.869 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:38:25.869 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:38:25.869 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "mgr.y", 2021-02-10T11:38:25.869 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAdvSNgVqQFDRAAwo6YEdWZCAr5GdOWVL/Qvg==", 2021-02-10T11:38:25.869 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:38:25.869 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow *", 2021-02-10T11:38:25.869 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow profile mgr", 2021-02-10T11:38:25.870 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:38:25.870 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:38:25.870 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:38:25.870 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:38:25.870 INFO:teuthology.orchestra.run.gibba002.stdout:} 2021-02-10T11:38:25.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:25.856 7f9a20146700 1 -- 172.21.2.102:0/1731666678 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f99f80186c0 msgr2=0x7f99f801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:25.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:25.856 7f9a20146700 1 --2- 172.21.2.102:0/1731666678 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f99f80186c0 0x7f99f801abc0 crc :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:25.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:25.856 7f9a20146700 1 -- 172.21.2.102:0/1731666678 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9a181b80f0 msgr2=0x7f9a181b84f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:25.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:25.856 7f9a20146700 1 --2- 172.21.2.102:0/1731666678 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9a181b80f0 0x7f9a181b84f0 secure :-1 s=READY pgs=3082 cs=0 l=1 rev1=1 rx=0x7f9a0c00b5c0 tx=0x7f9a0c00b960).stop 2021-02-10T11:38:25.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:25.857 7f9a20146700 1 -- 172.21.2.102:0/1731666678 shutdown_connections 2021-02-10T11:38:25.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:25.857 7f9a20146700 1 --2- 172.21.2.102:0/1731666678 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f99f80186c0 0x7f99f801abc0 unknown :-1 s=CLOSED pgs=1350 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:25.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:25.857 7f9a20146700 1 --2- 172.21.2.102:0/1731666678 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f9a1819da20 0x7f9a1819de20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:25.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:25.857 7f9a20146700 1 --2- 172.21.2.102:0/1731666678 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9a181b80f0 0x7f9a181b84f0 unknown :-1 s=CLOSED pgs=3082 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:25.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:25.857 7f9a20146700 1 --2- 172.21.2.102:0/1731666678 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9a181b7cf0 0x7f9a1819cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:25.872 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:25.857 7f9a20146700 1 -- 172.21.2.102:0/1731666678 >> 172.21.2.102:0/1731666678 conn(0x7f9a180ed810 msgr2=0x7f9a18094980 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:25.872 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:25.857 7f9a20146700 1 -- 172.21.2.102:0/1731666678 shutdown_connections 2021-02-10T11:38:25.872 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:25.857 7f9a20146700 1 -- 172.21.2.102:0/1731666678 wait complete. 2021-02-10T11:38:25.872 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:38:26.303 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:26.285 7f7d422c8700 1 Processor -- start 2021-02-10T11:38:26.304 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:26.285 7f7d422c8700 1 -- start start 2021-02-10T11:38:26.304 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:26.286 7f7d422c8700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f7d3c0d1c70 0x7f7d3c0f20a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:26.304 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:26.286 7f7d422c8700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d3c0f0b10 con 0x7f7d3c0d1870 2021-02-10T11:38:26.304 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:26.286 7f7d422c8700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d3c1b0be0 con 0x7f7d3c0d0340 2021-02-10T11:38:26.304 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:26.286 7f7d422c8700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d3c0e5ed0 con 0x7f7d3c0d1c70 2021-02-10T11:38:26.304 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:26.286 7f7d3bfff700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f7d3c0d1870 0x7f7d3c1b03f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:59924/0 (socket says 172.21.2.102:59924) 2021-02-10T11:38:26.304 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:26.286 7f7d3bfff700 1 -- 172.21.2.102:0/1514533064 learned_addr learned my addr 172.21.2.102:0/1514533064 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:38:26.304 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:26.286 7f7d408e5700 1 --2- 172.21.2.102:0/1514533064 >> v2:172.21.2.113:3300/0 conn(0x7f7d3c0d1c70 0x7f7d3c0f20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:26.304 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:26.286 7f7d3affd700 1 -- 172.21.2.102:0/1514533064 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1058723090 0 0) 0x7f7d2c000ce0 con 0x7f7d3c0d1870 2021-02-10T11:38:26.305 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:26.286 7f7d3affd700 1 -- 172.21.2.102:0/1514533064 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d24002fe0 con 0x7f7d3c0d1870 2021-02-10T11:38:26.305 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:26.286 7f7d408e5700 1 -- 172.21.2.102:0/1514533064 >> v1:172.21.2.113:6789/0 conn(0x7f7d3c0d0340 legacy=0x7f7d3c0ef640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:26.305 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:26.286 7f7d408e5700 1 -- 172.21.2.102:0/1514533064 >> v1:172.21.2.102:6789/0 conn(0x7f7d3c0d1870 legacy=0x7f7d3c1b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:26.305 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:26.286 7f7d408e5700 1 -- 172.21.2.102:0/1514533064 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d3c0efe30 con 0x7f7d3c0d1c70 2021-02-10T11:38:26.305 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:26.286 7f7d408e5700 1 --2- 172.21.2.102:0/1514533064 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7d3c0d1c70 0x7f7d3c0f20a0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 rx=0x7f7d34006e50 tx=0x7f7d34009e70).ready entity=mon.1 client_cookie=46029f0394220fec server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:26.305 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:26.287 7f7d3affd700 1 -- 172.21.2.102:0/1514533064 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7d3400b510 con 0x7f7d3c0d1c70 2021-02-10T11:38:26.305 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:26.287 7f7d3affd700 1 -- 172.21.2.102:0/1514533064 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d34006790 con 0x7f7d3c0d1c70 2021-02-10T11:38:26.305 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:26.287 7f7d3affd700 1 -- 172.21.2.102:0/1514533064 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7d3400b940 con 0x7f7d3c0d1c70 2021-02-10T11:38:26.305 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:26.287 7f7d422c8700 1 -- 172.21.2.102:0/1514533064 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7d3c0d1c70 msgr2=0x7f7d3c0f20a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:26.305 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:26.287 7f7d422c8700 1 --2- 172.21.2.102:0/1514533064 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7d3c0d1c70 0x7f7d3c0f20a0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 rx=0x7f7d34006e50 tx=0x7f7d34009e70).stop 2021-02-10T11:38:26.305 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:26.287 7f7d422c8700 1 -- 172.21.2.102:0/1514533064 shutdown_connections 2021-02-10T11:38:26.305 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:26.287 7f7d422c8700 1 --2- 172.21.2.102:0/1514533064 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7d3c0d1c70 0x7f7d3c0f20a0 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:26.306 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:26.287 7f7d422c8700 1 -- 172.21.2.102:0/1514533064 >> 172.21.2.102:0/1514533064 conn(0x7f7d3c0ce450 msgr2=0x7f7d3c0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:26.306 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:26.287 7f7d422c8700 1 -- 172.21.2.102:0/1514533064 shutdown_connections 2021-02-10T11:38:26.306 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:26.287 7f7d422c8700 1 -- 172.21.2.102:0/1514533064 wait complete. 2021-02-10T11:38:26.307 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:26.302 7f7d422c8700 1 Processor -- start 2021-02-10T11:38:26.308 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:26.302 7f7d422c8700 1 -- start start 2021-02-10T11:38:26.308 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:26.302 7f7d422c8700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7d3c1947b0 0x7f7d3c0a69a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:26.308 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:26.302 7f7d422c8700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7d3c0a6fc0 0x7f7d3c1a15d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:26.308 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:26.302 7f7d422c8700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7d3c1a1bf0 0x7f7d3c1a1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:26.308 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:26.302 7f7d422c8700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d3c0a73c0 con 0x7f7d3c1a1bf0 2021-02-10T11:38:26.308 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:26.302 7f7d422c8700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d3c18f170 con 0x7f7d3c0a6fc0 2021-02-10T11:38:26.308 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:26.302 7f7d422c8700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d3c194bb0 con 0x7f7d3c1947b0 2021-02-10T11:38:26.308 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:26.303 7f7d408e5700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7d3c1a1bf0 0x7f7d3c1a1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:26.308 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:26.303 7f7d408e5700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7d3c1a1bf0 0x7f7d3c1a1ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:39472/0 (socket says 172.21.2.102:39472) 2021-02-10T11:38:26.308 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:26.303 7f7d408e5700 1 -- 172.21.2.102:0/578926368 learned_addr learned my addr 172.21.2.102:0/578926368 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:38:26.308 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:26.303 7f7d408e5700 1 -- 172.21.2.102:0/578926368 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7d3c1947b0 msgr2=0x7f7d3c0a69a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:26.309 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:26.303 7f7d408e5700 1 --2- 172.21.2.102:0/578926368 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7d3c1947b0 0x7f7d3c0a69a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:26.309 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:26.303 7f7d408e5700 1 -- 172.21.2.102:0/578926368 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7d3c0a6fc0 msgr2=0x7f7d3c1a15d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:26.309 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:26.303 7f7d408e5700 1 --2- 172.21.2.102:0/578926368 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7d3c0a6fc0 0x7f7d3c1a15d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:26.309 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:26.303 7f7d408e5700 1 -- 172.21.2.102:0/578926368 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d3c18ec70 con 0x7f7d3c1a1bf0 2021-02-10T11:38:26.309 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:26.303 7f7d408e5700 1 --2- 172.21.2.102:0/578926368 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7d3c1a1bf0 0x7f7d3c1a1ff0 secure :-1 s=READY pgs=3084 cs=0 l=1 rev1=1 rx=0x7f7d34009a90 tx=0x7f7d3400bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:26.309 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:26.303 7f7d38ff9700 1 -- 172.21.2.102:0/578926368 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7d34002ef0 con 0x7f7d3c1a1bf0 2021-02-10T11:38:26.309 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:26.303 7f7d38ff9700 1 -- 172.21.2.102:0/578926368 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d340093a0 con 0x7f7d3c1a1bf0 2021-02-10T11:38:26.309 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:26.303 7f7d422c8700 1 -- 172.21.2.102:0/578926368 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d3c18ec70 con 0x7f7d3c1a1bf0 2021-02-10T11:38:26.309 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:26.303 7f7d422c8700 1 -- 172.21.2.102:0/578926368 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d3c18ec70 con 0x7f7d3c1a1bf0 2021-02-10T11:38:26.309 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:26.303 7f7d38ff9700 1 -- 172.21.2.102:0/578926368 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7d3400c4e0 con 0x7f7d3c1a1bf0 2021-02-10T11:38:26.309 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:26.304 7f7d38ff9700 1 -- 172.21.2.102:0/578926368 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f7d340056b0 con 0x7f7d3c1a1bf0 2021-02-10T11:38:26.309 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:26.304 7f7d38ff9700 1 --2- 172.21.2.102:0/578926368 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7d1c0186c0 0x7f7d1c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:26.309 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:26.304 7f7d38ff9700 1 -- 172.21.2.102:0/578926368 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(176..176 src has 1..176) v4 ==== 4753+0+0 (secure 0 0 0) 0x7f7d340325d0 con 0x7f7d3c1a1bf0 2021-02-10T11:38:26.310 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:26.304 7f7d3bfff700 1 --2- 172.21.2.102:0/578926368 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7d1c0186c0 0x7f7d1c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:26.310 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:26.304 7f7d3bfff700 1 --2- 172.21.2.102:0/578926368 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7d1c0186c0 0x7f7d1c01abc0 crc :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:26.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:26.335 7f7d422c8700 1 -- 172.21.2.102:0/578926368 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d3c0ee0e0 con 0x7f7d3c1a1bf0 2021-02-10T11:38:26.339 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:26.338 7f7d38ff9700 1 -- 172.21.2.102:0/578926368 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f7d3402cff0 con 0x7f7d3c1a1bf0 2021-02-10T11:38:26.466 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:26.464 7f7d422c8700 1 -- 172.21.2.102:0/578926368 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7d3c0ce450 con 0x7f7d3c1a1bf0 2021-02-10T11:38:26.466 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:26.464 7f7d38ff9700 1 -- 172.21.2.102:0/578926368 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v176) v1 ==== 74+0+11154 (secure 0 0 0) 0x7f7d3402cff0 con 0x7f7d3c1a1bf0 2021-02-10T11:38:26.467 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:38:26.467 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":176,"fsid":"68db30ea-dbce-4aef-a6df-f01c6fea0513","created":"2021-02-10 11:03:06.331557","modified":"2021-02-10 11:38:25.204749","last_up_change":"2021-02-10 11:03:32.690562","last_in_change":"2021-02-10 11:03:32.690562","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":37,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:03:37.322519","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"be691b4d-a039-408a-af80-9d53f02b6f1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6818","nonce":36088},{"type":"v1","addr":"172.21.2.102:6820","nonce":36088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6822","nonce":36088},{"type":"v1","addr":"172.21.2.102:6824","nonce":36088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6830","nonce":36088},{"type":"v1","addr":"172.21.2.102:6832","nonce":36088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6826","nonce":36088},{"type":"v1","addr":"172.21.2.102:6828","nonce":36088}]},"public_addr":"172.21.2.102:6820/36088","cluster_addr":"172.21.2.102:6824/36088","heartbeat_back_addr":"172.21.2.102:6832/36088","heartbeat_front_addr":"172.21.2.102:6828/36088","state":["exists","up"]},{"osd":1,"uuid":"64e3ccc8-9c69-4d7c-865e-af8e83822635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6819","nonce":36087},{"type":"v1","addr":"172.21.2.102:6821","nonce":36087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6823","nonce":36087},{"type":"v1","addr":"172.21.2.102:6825","nonce":36087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6831","nonce":36087},{"type":"v1","addr":"172.21.2.102:6833","nonce":36087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6827","nonce":36087},{"type":"v1","addr":"172.21.2.102:6829","nonce":36087}]},"public_addr":"172.21.2.102:6821/36087","cluster_addr":"172.21.2.102:6825/36087","heartbeat_back_addr":"172.21.2.102:6833/36087","heartbeat_front_addr":"172.21.2.102:6829/36087","state":["exists","up"]},{"osd":2,"uuid":"153700af-747f-481f-bdf4-d6072d3a842f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6810","nonce":36089},{"type":"v1","addr":"172.21.2.102:6811","nonce":36089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6812","nonce":36089},{"type":"v1","addr":"172.21.2.102:6813","nonce":36089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6816","nonce":36089},{"type":"v1","addr":"172.21.2.102:6817","nonce":36089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6814","nonce":36089},{"type":"v1","addr":"172.21.2.102:6815","nonce":36089}]},"public_addr":"172.21.2.102:6811/36089","cluster_addr":"172.21.2.102:6813/36089","heartbeat_back_addr":"172.21.2.102:6817/36089","heartbeat_front_addr":"172.21.2.102:6815/36089","state":["exists","up"]},{"osd":3,"uuid":"cbc678f4-83a9-45e6-8830-2f7c232b1ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6802","nonce":36090},{"type":"v1","addr":"172.21.2.102:6803","nonce":36090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6804","nonce":36090},{"type":"v1","addr":"172.21.2.102:6805","nonce":36090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6808","nonce":36090},{"type":"v1","addr":"172.21.2.102:6809","nonce":36090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6806","nonce":36090},{"type":"v1","addr":"172.21.2.102:6807","nonce":36090}]},"public_addr":"172.21.2.102:6803/36090","cluster_addr":"172.21.2.102:6805/36090","heartbeat_back_addr":"172.21.2.102:6809/36090","heartbeat_front_addr":"172.21.2.102:6807/36090","state":["exists","up"]},{"osd":4,"uuid":"c01e4008-b258-4f10-afd2-9e7f37214351","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6808","nonce":36368},{"type":"v1","addr":"172.21.2.113:6811","nonce":36368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6814","nonce":36368},{"type":"v1","addr":"172.21.2.113:6818","nonce":36368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6826","nonce":36368},{"type":"v1","addr":"172.21.2.113:6828","nonce":36368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6820","nonce":36368},{"type":"v1","addr":"172.21.2.113:6824","nonce":36368}]},"public_addr":"172.21.2.113:6811/36368","cluster_addr":"172.21.2.113:6818/36368","heartbeat_back_addr":"172.21.2.113:6828/36368","heartbeat_front_addr":"172.21.2.113:6824/36368","state":["exists","up"]},{"osd":5,"uuid":"551f6007-daf8-4a98-8ee0-c2b16bcccb5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6802","nonce":36366},{"type":"v1","addr":"172.21.2.113:6805","nonce":36366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6807","nonce":36366},{"type":"v1","addr":"172.21.2.113:6810","nonce":36366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6819","nonce":36366},{"type":"v1","addr":"172.21.2.113:6822","nonce":36366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6812","nonce":36366},{"type":"v1","addr":"172.21.2.113:6815","nonce":36366}]},"public_addr":"172.21.2.113:6805/36366","cluster_addr":"172.21.2.113:6810/36366","heartbeat_back_addr":"172.21.2.113:6822/36366","heartbeat_front_addr":"172.21.2.113:6815/36366","state":["exists","up"]},{"osd":6,"uuid":"bf67c040-0e50-48be-b102-c3943bc9ebef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6800","nonce":36367},{"type":"v1","addr":"172.21.2.113:6801","nonce":36367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6803","nonce":36367},{"type":"v1","addr":"172.21.2.113:6804","nonce":36367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6813","nonce":36367},{"type":"v1","addr":"172.21.2.113:6816","nonce":36367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6806","nonce":36367},{"type":"v1","addr":"172.21.2.113:6809","nonce":36367}]},"public_addr":"172.21.2.113:6801/36367","cluster_addr":"172.21.2.113:6804/36367","heartbeat_back_addr":"172.21.2.113:6816/36367","heartbeat_front_addr":"172.21.2.113:6809/36367","state":["exists","up"]},{"osd":7,"uuid":"f5461f80-018d-428f-be89-2eb2bcbe7bb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":169,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6817","nonce":36365},{"type":"v1","addr":"172.21.2.113:6821","nonce":36365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6823","nonce":36365},{"type":"v1","addr":"172.21.2.113:6825","nonce":36365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6830","nonce":36365},{"type":"v1","addr":"172.21.2.113:6831","nonce":36365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6827","nonce":36365},{"type":"v1","addr":"172.21.2.113:6829","nonce":36365}]},"public_addr":"172.21.2.113:6821/36365","cluster_addr":"172.21.2.113:6825/36365","heartbeat_back_addr":"172.21.2.113:6831/36365","heartbeat_front_addr":"172.21.2.113:6829/36365","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:38:26.468 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:26.467 7f7d422c8700 1 -- 172.21.2.102:0/578926368 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7d1c0186c0 msgr2=0x7f7d1c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:26.468 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:26.467 7f7d422c8700 1 --2- 172.21.2.102:0/578926368 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7d1c0186c0 0x7f7d1c01abc0 crc :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:26.468 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:26.467 7f7d422c8700 1 -- 172.21.2.102:0/578926368 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7d3c1a1bf0 msgr2=0x7f7d3c1a1ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:26.468 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:26.467 7f7d422c8700 1 --2- 172.21.2.102:0/578926368 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7d3c1a1bf0 0x7f7d3c1a1ff0 secure :-1 s=READY pgs=3084 cs=0 l=1 rev1=1 rx=0x7f7d34009a90 tx=0x7f7d3400bcf0).stop 2021-02-10T11:38:26.468 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:26.467 7f7d422c8700 1 -- 172.21.2.102:0/578926368 shutdown_connections 2021-02-10T11:38:26.468 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:26.467 7f7d422c8700 1 --2- 172.21.2.102:0/578926368 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7d3c1a1bf0 0x7f7d3c1a1ff0 unknown :-1 s=CLOSED pgs=3084 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:26.468 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:26.467 7f7d422c8700 1 --2- 172.21.2.102:0/578926368 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7d1c0186c0 0x7f7d1c01abc0 unknown :-1 s=CLOSED pgs=1351 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:26.469 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:26.467 7f7d422c8700 1 --2- 172.21.2.102:0/578926368 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7d3c0a6fc0 0x7f7d3c1a15d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:26.469 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:26.467 7f7d422c8700 1 --2- 172.21.2.102:0/578926368 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7d3c1947b0 0x7f7d3c0a69a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:26.469 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:26.467 7f7d422c8700 1 -- 172.21.2.102:0/578926368 >> 172.21.2.102:0/578926368 conn(0x7f7d3c1a38c0 msgr2=0x7f7d3c1a2f00 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:26.469 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:26.467 7f7d422c8700 1 -- 172.21.2.102:0/578926368 shutdown_connections 2021-02-10T11:38:26.469 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:26.467 7f7d422c8700 1 -- 172.21.2.102:0/578926368 wait complete. 2021-02-10T11:38:26.477 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:03:37.322519', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2021-02-10T11:38:26.477 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:38:26.911 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:26.891 7fdd0c549700 1 Processor -- start 2021-02-10T11:38:26.911 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:26.891 7fdd0c549700 1 -- start start 2021-02-10T11:38:26.911 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:26.892 7fdd0c549700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdd040d1870 0x7fdd041b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:26.911 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:26.892 7fdd0c549700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fdd040cff70 0x7fdd04109130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:26.911 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:26.892 7fdd0c549700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd04109a60 con 0x7fdd041b0ae0 2021-02-10T11:38:26.911 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:26.892 7fdd0c549700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd04109750 con 0x7fdd040cff70 2021-02-10T11:38:26.911 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:26.892 7fdd0c549700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd04088070 con 0x7fdd040d1870 2021-02-10T11:38:26.911 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:26.892 7fdd09b64700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fdd041b0ae0 0x7fdd04108940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:59936/0 (socket says 172.21.2.102:59936) 2021-02-10T11:38:26.912 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:26.892 7fdd09b64700 1 -- 172.21.2.102:0/1175503069 learned_addr learned my addr 172.21.2.102:0/1175503069 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:38:26.912 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:26.892 7fdd09363700 1 -- 172.21.2.102:0/1175503069 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3371368281 0 0) 0x7fdcf80010b0 con 0x7fdd041b0ae0 2021-02-10T11:38:26.912 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:26.892 7fdd09363700 1 -- 172.21.2.102:0/1175503069 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdcf0002fe0 con 0x7fdd041b0ae0 2021-02-10T11:38:26.912 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:26.893 7fdd09363700 1 -- 172.21.2.102:0/1175503069 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1211803237 0 0) 0x7fdcf80010b0 con 0x7fdd041b0ae0 2021-02-10T11:38:26.912 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:26.893 7fdd09363700 1 -- 172.21.2.102:0/1175503069 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdd040d1870 msgr2=0x7fdd041b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:26.912 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:26.893 7fdd09363700 1 --2- 172.21.2.102:0/1175503069 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdd040d1870 0x7fdd041b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:26.912 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:26.893 7fdd09363700 1 -- 172.21.2.102:0/1175503069 >> v2:172.21.2.113:3300/0 conn(0x7fdd040cff70 msgr2=0x7fdd04109130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:26.912 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:26.893 7fdd09363700 1 --2- 172.21.2.102:0/1175503069 >> v2:172.21.2.113:3300/0 conn(0x7fdd040cff70 0x7fdd04109130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:26.912 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:26.893 7fdd09363700 1 -- 172.21.2.102:0/1175503069 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd04088260 con 0x7fdd041b0ae0 2021-02-10T11:38:26.912 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:26.893 7fdd09363700 1 -- 172.21.2.102:0/1175503069 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7fdcf80015b0 con 0x7fdd041b0ae0 2021-02-10T11:38:26.912 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:26.893 7fdd09363700 1 -- 172.21.2.102:0/1175503069 >> v1:172.21.2.102:6789/0 conn(0x7fdd041b0ae0 legacy=0x7fdd04108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:26.912 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:26.893 7fdd09363700 1 --2- 172.21.2.102:0/1175503069 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fdcf0003ac0 0x7fdcf0005f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:26.912 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:26.893 7fdd09363700 1 --2- 172.21.2.102:0/1175503069 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdcf0006780 0x7fdcf0008c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:26.913 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:26.893 7fdd09363700 1 --2- 172.21.2.102:0/1175503069 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdcf0009580 0x7fdcf000ba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:26.913 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:26.893 7fdd09363700 1 -- 172.21.2.102:0/1175503069 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcf000c2c0 con 0x7fdcf0006780 2021-02-10T11:38:26.913 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:26.893 7fdd09363700 1 -- 172.21.2.102:0/1175503069 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcf000c4e0 con 0x7fdcf0003ac0 2021-02-10T11:38:26.913 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:26.893 7fdd09363700 1 -- 172.21.2.102:0/1175503069 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdcf000c700 con 0x7fdcf0009580 2021-02-10T11:38:26.913 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:26.893 7fdd09b64700 1 --2- 172.21.2.102:0/1175503069 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdcf0006780 0x7fdcf0008c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:26.913 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:26.893 7fdd0ab66700 1 --2- 172.21.2.102:0/1175503069 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdcf0009580 0x7fdcf000ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:26.913 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:26.893 7fdd09b64700 1 -- 172.21.2.102:0/1175503069 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdcf0009580 msgr2=0x7fdcf000ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:26.913 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:26.893 7fdd09b64700 1 --2- 172.21.2.102:0/1175503069 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdcf0009580 0x7fdcf000ba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:26.913 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:26.893 7fdd09b64700 1 -- 172.21.2.102:0/1175503069 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fdcf0003ac0 msgr2=0x7fdcf0005f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:26.913 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:26.893 7fdd09b64700 1 --2- 172.21.2.102:0/1175503069 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fdcf0003ac0 0x7fdcf0005f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:26.913 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:26.893 7fdd09b64700 1 -- 172.21.2.102:0/1175503069 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcf000c9a0 con 0x7fdcf0006780 2021-02-10T11:38:26.913 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:26.893 7fdd0ab66700 1 --2- 172.21.2.102:0/1175503069 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdcf0009580 0x7fdcf000ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:38:26.914 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:26.893 7fdd09b64700 1 --2- 172.21.2.102:0/1175503069 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdcf0006780 0x7fdcf0008c80 secure :-1 s=READY pgs=3086 cs=0 l=1 rev1=1 rx=0x7fdcf80017d0 tx=0x7fdcf8007d30).ready entity=mon.0 client_cookie=2d0417ae1b4e172 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:26.914 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:26.893 7fdd09363700 1 -- 172.21.2.102:0/1175503069 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdcf8001010 con 0x7fdcf0006780 2021-02-10T11:38:26.914 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:26.893 7fdd09363700 1 -- 172.21.2.102:0/1175503069 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcf804e320 con 0x7fdcf0006780 2021-02-10T11:38:26.914 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:26.893 7fdd09363700 1 -- 172.21.2.102:0/1175503069 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdcf8001010 con 0x7fdcf0006780 2021-02-10T11:38:26.914 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:26.893 7fdd0c549700 1 -- 172.21.2.102:0/1175503069 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdcf0006780 msgr2=0x7fdcf0008c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:26.914 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:26.893 7fdd0c549700 1 --2- 172.21.2.102:0/1175503069 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdcf0006780 0x7fdcf0008c80 secure :-1 s=READY pgs=3086 cs=0 l=1 rev1=1 rx=0x7fdcf80017d0 tx=0x7fdcf8007d30).stop 2021-02-10T11:38:26.914 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:26.894 7fdd0ab66700 1 -- 172.21.2.102:0/1175503069 reap_dead start 2021-02-10T11:38:26.914 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:26.894 7fdd0c549700 1 -- 172.21.2.102:0/1175503069 shutdown_connections 2021-02-10T11:38:26.914 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:26.894 7fdd0c549700 1 -- 172.21.2.102:0/1175503069 >> 172.21.2.102:0/1175503069 conn(0x7fdd040ce420 msgr2=0x7fdd040d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:26.914 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:26.894 7fdd0c549700 1 -- 172.21.2.102:0/1175503069 shutdown_connections 2021-02-10T11:38:26.914 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:26.894 7fdd0c549700 1 -- 172.21.2.102:0/1175503069 wait complete. 2021-02-10T11:38:26.914 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:26.909 7fdd0c549700 1 Processor -- start 2021-02-10T11:38:26.915 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:26.909 7fdd0c549700 1 -- start start 2021-02-10T11:38:26.916 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:26.909 7fdd0c549700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdd040a2580 0x7fdd040a1cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:26.916 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:26.909 7fdd0c549700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdd04094c40 0x7fdd041a75e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:26.916 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:26.909 7fdd0c549700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fdd04095040 0x7fdd041a7c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:26.916 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:26.909 7fdd0c549700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd04188180 con 0x7fdd04094c40 2021-02-10T11:38:26.916 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:26.909 7fdd0c549700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd04187e70 con 0x7fdd04095040 2021-02-10T11:38:26.916 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:26.909 7fdd0c549700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd041a8220 con 0x7fdd040a2580 2021-02-10T11:38:26.916 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:26.909 7fdd09b64700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdd04094c40 0x7fdd041a75e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:26.916 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:26.909 7fdd09b64700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdd04094c40 0x7fdd041a75e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:39492/0 (socket says 172.21.2.102:39492) 2021-02-10T11:38:26.916 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:26.909 7fdd09b64700 1 -- 172.21.2.102:0/1844507084 learned_addr learned my addr 172.21.2.102:0/1844507084 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:38:26.916 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:26.909 7fdd09b64700 1 -- 172.21.2.102:0/1844507084 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdd040a2580 msgr2=0x7fdd040a1cf0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:38:26.917 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:26.909 7fdd09b64700 1 --2- 172.21.2.102:0/1844507084 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdd040a2580 0x7fdd040a1cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:26.917 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:26.910 7fdd09b64700 1 -- 172.21.2.102:0/1844507084 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fdd04095040 msgr2=0x7fdd041a7c00 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:38:26.917 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:26.910 7fdd09b64700 1 --2- 172.21.2.102:0/1844507084 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fdd04095040 0x7fdd041a7c00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:26.917 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:26.910 7fdd09b64700 1 -- 172.21.2.102:0/1844507084 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd04188700 con 0x7fdd04094c40 2021-02-10T11:38:26.917 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:26.910 7fdd09b64700 1 --2- 172.21.2.102:0/1844507084 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdd04094c40 0x7fdd041a75e0 secure :-1 s=READY pgs=3087 cs=0 l=1 rev1=1 rx=0x7fdcf804f520 tx=0x7fdcf804f8c0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:26.917 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:26.910 7fdcf6ffd700 1 -- 172.21.2.102:0/1844507084 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdcf804fcb0 con 0x7fdd04094c40 2021-02-10T11:38:26.917 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:26.910 7fdcf6ffd700 1 -- 172.21.2.102:0/1844507084 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcf8050070 con 0x7fdd04094c40 2021-02-10T11:38:26.917 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:26.910 7fdcf6ffd700 1 -- 172.21.2.102:0/1844507084 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdcf8050590 con 0x7fdd04094c40 2021-02-10T11:38:26.917 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:26.910 7fdd0c549700 1 -- 172.21.2.102:0/1844507084 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd04188700 con 0x7fdd04094c40 2021-02-10T11:38:26.917 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:26.910 7fdd0c549700 1 -- 172.21.2.102:0/1844507084 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd04188c90 con 0x7fdd04094c40 2021-02-10T11:38:26.917 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:26.911 7fdcf6ffd700 1 -- 172.21.2.102:0/1844507084 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fdcf804fcb0 con 0x7fdd04094c40 2021-02-10T11:38:26.917 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:26.911 7fdcf6ffd700 1 --2- 172.21.2.102:0/1844507084 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fdcec0186c0 0x7fdcec01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:26.917 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:26.911 7fdcf6ffd700 1 -- 172.21.2.102:0/1844507084 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(176..176 src has 1..176) v4 ==== 4753+0+0 (secure 0 0 0) 0x7fdcf8076250 con 0x7fdd04094c40 2021-02-10T11:38:26.918 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:26.911 7fdd0a365700 1 --2- 172.21.2.102:0/1844507084 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fdcec0186c0 0x7fdcec01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:26.918 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:26.911 7fdd0a365700 1 --2- 172.21.2.102:0/1844507084 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fdcec0186c0 0x7fdcec01abc0 crc :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:26.944 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:26.942 7fdd0c549700 1 -- 172.21.2.102:0/1844507084 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd04082980 con 0x7fdd04094c40 2021-02-10T11:38:26.947 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:26.946 7fdcf6ffd700 1 -- 172.21.2.102:0/1844507084 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fdcf80009b0 con 0x7fdd04094c40 2021-02-10T11:38:27.075 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:27.072 7fdd0c549700 1 -- 172.21.2.102:0/1844507084 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7fdd04109a60 con 0x7fdd04094c40 2021-02-10T11:38:27.075 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:27.074 7fdcf6ffd700 1 -- 172.21.2.102:0/1844507084 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v176) v1 ==== 92+0+10 (secure 0 0 0) 0x7fdcf804fcb0 con 0x7fdd04094c40 2021-02-10T11:38:27.075 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:38:27.077 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:27.075 7fdd0c549700 1 -- 172.21.2.102:0/1844507084 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fdcec0186c0 msgr2=0x7fdcec01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:27.077 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:27.075 7fdd0c549700 1 --2- 172.21.2.102:0/1844507084 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fdcec0186c0 0x7fdcec01abc0 crc :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:27.078 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:27.075 7fdd0c549700 1 -- 172.21.2.102:0/1844507084 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdd04094c40 msgr2=0x7fdd041a75e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:27.078 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:27.075 7fdd0c549700 1 --2- 172.21.2.102:0/1844507084 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdd04094c40 0x7fdd041a75e0 secure :-1 s=READY pgs=3087 cs=0 l=1 rev1=1 rx=0x7fdcf804f520 tx=0x7fdcf804f8c0).stop 2021-02-10T11:38:27.078 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:27.075 7fdd0c549700 1 -- 172.21.2.102:0/1844507084 shutdown_connections 2021-02-10T11:38:27.078 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:27.075 7fdd0c549700 1 --2- 172.21.2.102:0/1844507084 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fdcec0186c0 0x7fdcec01abc0 unknown :-1 s=CLOSED pgs=1352 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:27.078 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:27.075 7fdd0c549700 1 --2- 172.21.2.102:0/1844507084 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fdd04095040 0x7fdd041a7c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:27.078 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:27.075 7fdd0c549700 1 --2- 172.21.2.102:0/1844507084 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdd04094c40 0x7fdd041a75e0 unknown :-1 s=CLOSED pgs=3087 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:27.078 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:27.075 7fdd0c549700 1 --2- 172.21.2.102:0/1844507084 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdd040a2580 0x7fdd040a1cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:27.078 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:27.075 7fdd0c549700 1 -- 172.21.2.102:0/1844507084 >> 172.21.2.102:0/1844507084 conn(0x7fdd040a2b30 msgr2=0x7fdd0409ce60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:27.078 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:27.075 7fdd0c549700 1 -- 172.21.2.102:0/1844507084 shutdown_connections 2021-02-10T11:38:27.078 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:27.075 7fdd0c549700 1 -- 172.21.2.102:0/1844507084 wait complete. 2021-02-10T11:38:27.086 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --format=json-pretty osd lspools 2021-02-10T11:38:27.515 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:27.495 7fba668bf700 1 Processor -- start 2021-02-10T11:38:27.515 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:27.495 7fba668bf700 1 -- start start 2021-02-10T11:38:27.515 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:27.496 7fba668bf700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fba600d1870 0x7fba601b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:27.515 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:27.496 7fba668bf700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fba600f0650 0x7fba600cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:27.515 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:27.496 7fba668bf700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba600f0b10 con 0x7fba601b0a10 2021-02-10T11:38:27.515 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:27.496 7fba668bf700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba600e5ed0 con 0x7fba600d1870 2021-02-10T11:38:27.515 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:27.496 7fba668bf700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba600d1c70 con 0x7fba600f0650 2021-02-10T11:38:27.516 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:27.498 7fba5ffff700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fba600d1870 0x7fba601b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:27.516 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:27.498 7fba64edc700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fba600f0650 0x7fba600cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:27.516 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:27.498 7fba5ffff700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fba600d1870 0x7fba601b03f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.113:3300/0 says I am v2:172.21.2.102:42928/0 (socket says 172.21.2.102:42928) 2021-02-10T11:38:27.516 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:27.498 7fba5ffff700 1 -- 172.21.2.102:0/126762325 learned_addr learned my addr 172.21.2.102:0/126762325 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:38:27.516 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:27.498 7fba5f7fe700 1 --1- >> v1:172.21.2.113:6789/0 conn(0x7fba601b0a10 0x7fba600cf330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.113:6789/0 says I am v1:172.21.2.102:39048/0 (socket says 172.21.2.102:39048) 2021-02-10T11:38:27.516 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:27.498 7fba5ffff700 1 -- 172.21.2.102:0/126762325 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fba600f0650 msgr2=0x7fba600cfb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:27.516 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:27.498 7fba5ffff700 1 --2- 172.21.2.102:0/126762325 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fba600f0650 0x7fba600cfb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:27.516 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:27.498 7fba5ffff700 1 -- 172.21.2.102:0/126762325 >> v1:172.21.2.113:6789/0 conn(0x7fba601b0a10 legacy=0x7fba600cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:27.516 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:27.498 7fba5ffff700 1 -- 172.21.2.102:0/126762325 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba600d1e90 con 0x7fba600d1870 2021-02-10T11:38:27.516 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:27.498 7fba5ffff700 1 --2- 172.21.2.102:0/126762325 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fba600d1870 0x7fba601b03f0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 rx=0x7fba50006700 tx=0x7fba50009b80).ready entity=mon.1 client_cookie=d47774198af4064c server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:27.516 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:27.498 7fba5effd700 1 -- 172.21.2.102:0/126762325 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fba5000c0c0 con 0x7fba600d1870 2021-02-10T11:38:27.516 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:27.498 7fba5effd700 1 -- 172.21.2.102:0/126762325 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba5000c440 con 0x7fba600d1870 2021-02-10T11:38:27.516 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:27.498 7fba5effd700 1 -- 172.21.2.102:0/126762325 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fba5000c960 con 0x7fba600d1870 2021-02-10T11:38:27.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:27.498 7fba668bf700 1 -- 172.21.2.102:0/126762325 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fba600d1870 msgr2=0x7fba601b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:27.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:27.498 7fba668bf700 1 --2- 172.21.2.102:0/126762325 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fba600d1870 0x7fba601b03f0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 rx=0x7fba50006700 tx=0x7fba50009b80).stop 2021-02-10T11:38:27.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:27.499 7fba668bf700 1 -- 172.21.2.102:0/126762325 shutdown_connections 2021-02-10T11:38:27.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:27.499 7fba668bf700 1 --2- 172.21.2.102:0/126762325 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fba600f0650 0x7fba600cfb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:27.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:27.499 7fba668bf700 1 --2- 172.21.2.102:0/126762325 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fba600d1870 0x7fba601b03f0 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:27.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:27.499 7fba668bf700 1 -- 172.21.2.102:0/126762325 >> 172.21.2.102:0/126762325 conn(0x7fba600ce450 msgr2=0x7fba600d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:27.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:27.499 7fba668bf700 1 -- 172.21.2.102:0/126762325 shutdown_connections 2021-02-10T11:38:27.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:27.499 7fba668bf700 1 -- 172.21.2.102:0/126762325 wait complete. 2021-02-10T11:38:27.519 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:27.513 7fba668bf700 1 Processor -- start 2021-02-10T11:38:27.519 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:27.513 7fba668bf700 1 -- start start 2021-02-10T11:38:27.519 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:27.514 7fba668bf700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fba601942b0 0x7fba600b3550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:27.519 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:27.514 7fba668bf700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fba60195070 0x7fba60195470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:27.519 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:27.514 7fba668bf700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fba600bb8e0 0x7fba600bbce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:27.519 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:27.514 7fba668bf700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba601946b0 con 0x7fba601942b0 2021-02-10T11:38:27.519 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:27.514 7fba668bf700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba600b3b70 con 0x7fba60195070 2021-02-10T11:38:27.519 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:27.514 7fba668bf700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba600b3d90 con 0x7fba600bb8e0 2021-02-10T11:38:27.519 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:27.514 7fba5ffff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fba601942b0 0x7fba600b3550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:27.520 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:27.514 7fba5ffff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fba601942b0 0x7fba600b3550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:39502/0 (socket says 172.21.2.102:39502) 2021-02-10T11:38:27.520 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:27.514 7fba5ffff700 1 -- 172.21.2.102:0/1815358145 learned_addr learned my addr 172.21.2.102:0/1815358145 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:38:27.520 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:27.514 7fba5ffff700 1 -- 172.21.2.102:0/1815358145 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fba600bb8e0 msgr2=0x7fba600bbce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:27.520 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:27.514 7fba5ffff700 1 --2- 172.21.2.102:0/1815358145 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fba600bb8e0 0x7fba600bbce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:27.520 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:27.514 7fba5ffff700 1 -- 172.21.2.102:0/1815358145 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fba60195070 msgr2=0x7fba60195470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:27.520 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:27.514 7fba5ffff700 1 --2- 172.21.2.102:0/1815358145 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fba60195070 0x7fba60195470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:27.520 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:27.514 7fba5ffff700 1 -- 172.21.2.102:0/1815358145 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba601a49e0 con 0x7fba601942b0 2021-02-10T11:38:27.520 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:27.516 7fba5ffff700 1 --2- 172.21.2.102:0/1815358145 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fba601942b0 0x7fba600b3550 secure :-1 s=READY pgs=3088 cs=0 l=1 rev1=1 rx=0x7fba5000b5e0 tx=0x7fba5000b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:27.520 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:27.516 7fba5cff9700 1 -- 172.21.2.102:0/1815358145 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fba5000bd70 con 0x7fba601942b0 2021-02-10T11:38:27.520 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:27.516 7fba5cff9700 1 -- 172.21.2.102:0/1815358145 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba50002030 con 0x7fba601942b0 2021-02-10T11:38:27.520 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:27.516 7fba5cff9700 1 -- 172.21.2.102:0/1815358145 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fba50002550 con 0x7fba601942b0 2021-02-10T11:38:27.521 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:27.516 7fba668bf700 1 -- 172.21.2.102:0/1815358145 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba601a49e0 con 0x7fba601942b0 2021-02-10T11:38:27.521 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:27.516 7fba668bf700 1 -- 172.21.2.102:0/1815358145 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba600bc5f0 con 0x7fba601942b0 2021-02-10T11:38:27.521 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:27.516 7fba5cff9700 1 -- 172.21.2.102:0/1815358145 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fba50000c90 con 0x7fba601942b0 2021-02-10T11:38:27.521 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:27.517 7fba5cff9700 1 --2- 172.21.2.102:0/1815358145 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fba400186c0 0x7fba4001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:27.521 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:27.517 7fba5cff9700 1 -- 172.21.2.102:0/1815358145 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(176..176 src has 1..176) v4 ==== 4753+0+0 (secure 0 0 0) 0x7fba500322c0 con 0x7fba601942b0 2021-02-10T11:38:27.521 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:27.517 7fba5f7fe700 1 --2- 172.21.2.102:0/1815358145 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fba400186c0 0x7fba4001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:27.521 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:27.517 7fba5f7fe700 1 --2- 172.21.2.102:0/1815358145 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fba400186c0 0x7fba4001abc0 crc :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:27.548 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:27.546 7fba668bf700 1 -- 172.21.2.102:0/1815358145 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba600bc5f0 con 0x7fba601942b0 2021-02-10T11:38:27.550 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:27.548 7fba5cff9700 1 -- 172.21.2.102:0/1815358145 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fba5001b310 con 0x7fba601942b0 2021-02-10T11:38:27.674 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:27.673 7fba668bf700 1 -- 172.21.2.102:0/1815358145 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fba600bc5f0 con 0x7fba601942b0 2021-02-10T11:38:27.674 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:27.673 7fba5cff9700 1 -- 172.21.2.102:0/1815358145 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v176) v1 ==== 84+0+64 (secure 0 0 0) 0x7fba5001b310 con 0x7fba601942b0 2021-02-10T11:38:27.675 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:38:27.675 INFO:teuthology.orchestra.run.gibba002.stdout:[ 2021-02-10T11:38:27.675 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:38:27.675 INFO:teuthology.orchestra.run.gibba002.stdout: "poolnum": 1, 2021-02-10T11:38:27.675 INFO:teuthology.orchestra.run.gibba002.stdout: "poolname": "rbd" 2021-02-10T11:38:27.675 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:38:27.675 INFO:teuthology.orchestra.run.gibba002.stdout:] 2021-02-10T11:38:27.676 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:27.675 7fba668bf700 1 -- 172.21.2.102:0/1815358145 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fba400186c0 msgr2=0x7fba4001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:27.676 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:27.675 7fba668bf700 1 --2- 172.21.2.102:0/1815358145 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fba400186c0 0x7fba4001abc0 crc :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:27.676 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:27.675 7fba668bf700 1 -- 172.21.2.102:0/1815358145 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fba601942b0 msgr2=0x7fba600b3550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:27.677 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:27.675 7fba668bf700 1 --2- 172.21.2.102:0/1815358145 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fba601942b0 0x7fba600b3550 secure :-1 s=READY pgs=3088 cs=0 l=1 rev1=1 rx=0x7fba5000b5e0 tx=0x7fba5000b980).stop 2021-02-10T11:38:27.677 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:27.675 7fba668bf700 1 -- 172.21.2.102:0/1815358145 shutdown_connections 2021-02-10T11:38:27.677 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:27.675 7fba668bf700 1 --2- 172.21.2.102:0/1815358145 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fba600bb8e0 0x7fba600bbce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:27.677 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:27.675 7fba668bf700 1 --2- 172.21.2.102:0/1815358145 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fba400186c0 0x7fba4001abc0 unknown :-1 s=CLOSED pgs=1353 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:27.677 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:27.675 7fba668bf700 1 --2- 172.21.2.102:0/1815358145 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fba60195070 0x7fba60195470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:27.677 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:27.675 7fba668bf700 1 --2- 172.21.2.102:0/1815358145 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fba601942b0 0x7fba600b3550 unknown :-1 s=CLOSED pgs=3088 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:27.677 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:27.675 7fba668bf700 1 -- 172.21.2.102:0/1815358145 >> 172.21.2.102:0/1815358145 conn(0x7fba600b3040 msgr2=0x7fba600c3250 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:27.677 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:27.676 7fba668bf700 1 -- 172.21.2.102:0/1815358145 shutdown_connections 2021-02-10T11:38:27.678 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:27.676 7fba668bf700 1 -- 172.21.2.102:0/1815358145 wait complete. 2021-02-10T11:38:27.685 INFO:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2021-02-10T11:38:27.685 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool create cephfs_metadata 8 2021-02-10T11:38:28.119 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:28.100 7f1fc2b79700 1 Processor -- start 2021-02-10T11:38:28.119 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:28.100 7f1fc2b79700 1 -- start start 2021-02-10T11:38:28.119 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:28.100 7f1fc2b79700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f1fbc0d1870 0x7f1fbc1b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:28.119 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:28.100 7f1fc2b79700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1fbc1b0ae0 0x7f1fbc108940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:28.119 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:28.100 7f1fc2b79700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fbc0fe1c0 con 0x7f1fbc0d1c70 2021-02-10T11:38:28.119 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:28.100 7f1fc2b79700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1fbc109a60 con 0x7f1fbc0d1870 2021-02-10T11:38:28.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:28.100 7f1fc2b79700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fbc109750 con 0x7f1fbc1b0ae0 2021-02-10T11:38:28.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:28.101 7f1fc1196700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f1fbc0d1c70 0x7f1fbc108f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:59970/0 (socket says 172.21.2.102:59970) 2021-02-10T11:38:28.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:28.101 7f1fc1196700 1 -- 172.21.2.102:0/3498771579 learned_addr learned my addr 172.21.2.102:0/3498771579 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:38:28.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:28.101 7f1fbb7fe700 1 -- 172.21.2.102:0/3498771579 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2339582823 0 0) 0x7f1fb4001180 con 0x7f1fbc0d1c70 2021-02-10T11:38:28.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:28.101 7f1fbb7fe700 1 -- 172.21.2.102:0/3498771579 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1fa8002fe0 con 0x7f1fbc0d1c70 2021-02-10T11:38:28.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:28.101 7f1fbb7fe700 1 -- 172.21.2.102:0/3498771579 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1052979112 0 0) 0x7f1fb4001180 con 0x7f1fbc0d1c70 2021-02-10T11:38:28.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:28.101 7f1fbb7fe700 1 -- 172.21.2.102:0/3498771579 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1fbc1b0ae0 msgr2=0x7f1fbc108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:28.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:28.101 7f1fbb7fe700 1 --2- 172.21.2.102:0/3498771579 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1fbc1b0ae0 0x7f1fbc108940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:28.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:28.101 7f1fbb7fe700 1 -- 172.21.2.102:0/3498771579 >> v2:172.21.2.113:3300/0 conn(0x7f1fbc0d1870 msgr2=0x7f1fbc1b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:28.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:28.101 7f1fc0995700 1 --2- 172.21.2.102:0/3498771579 >> v2:172.21.2.113:3300/0 conn(0x7f1fbc0d1870 0x7f1fbc1b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:28.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:28.101 7f1fbb7fe700 1 --2- 172.21.2.102:0/3498771579 >> v2:172.21.2.113:3300/0 conn(0x7f1fbc0d1870 0x7f1fbc1b04c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:28.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:28.101 7f1fbb7fe700 1 -- 172.21.2.102:0/3498771579 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fbc088070 con 0x7f1fbc0d1c70 2021-02-10T11:38:28.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:28.101 7f1fbb7fe700 1 -- 172.21.2.102:0/3498771579 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7f1fb4001680 con 0x7f1fbc0d1c70 2021-02-10T11:38:28.121 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:28.101 7f1fbb7fe700 1 -- 172.21.2.102:0/3498771579 >> v1:172.21.2.102:6789/0 conn(0x7f1fbc0d1c70 legacy=0x7f1fbc108f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:28.121 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:28.101 7f1fbb7fe700 1 --2- 172.21.2.102:0/3498771579 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1fa8003ae0 0x7f1fa8005f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:28.121 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:28.101 7f1fbb7fe700 1 --2- 172.21.2.102:0/3498771579 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1fa80067f0 0x7f1fa8008cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:28.121 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:28.101 7f1fbb7fe700 1 --2- 172.21.2.102:0/3498771579 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1fa8009590 0x7f1fa800bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:28.121 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:28.101 7f1fbb7fe700 1 -- 172.21.2.102:0/3498771579 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fa800c360 con 0x7f1fa80067f0 2021-02-10T11:38:28.121 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:28.101 7f1fbb7fe700 1 -- 172.21.2.102:0/3498771579 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fa800c580 con 0x7f1fa8009590 2021-02-10T11:38:28.121 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:28.101 7f1fbb7fe700 1 -- 172.21.2.102:0/3498771579 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fa800c7a0 con 0x7f1fa8003ae0 2021-02-10T11:38:28.121 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:28.102 7f1fc0995700 1 --2- 172.21.2.102:0/3498771579 >> v2:172.21.2.113:3300/0 conn(0x7f1fbc0d1870 0x7f1fbc1b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:38:28.121 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:28.102 7f1fbbfff700 1 --2- 172.21.2.102:0/3498771579 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1fa80067f0 0x7f1fa8008cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:28.121 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:28.102 7f1fc1196700 1 --2- 172.21.2.102:0/3498771579 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1fa8009590 0x7f1fa800bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:28.121 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:28.102 7f1fbbfff700 1 -- 172.21.2.102:0/3498771579 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1fa8003ae0 msgr2=0x7f1fa8005f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:28.121 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:28.102 7f1fbbfff700 1 --2- 172.21.2.102:0/3498771579 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1fa8003ae0 0x7f1fa8005f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:28.121 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:28.102 7f1fbbfff700 1 -- 172.21.2.102:0/3498771579 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1fa8009590 msgr2=0x7f1fa800bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:28.122 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:28.102 7f1fbbfff700 1 --2- 172.21.2.102:0/3498771579 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1fa8009590 0x7f1fa800bab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:28.122 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:28.102 7f1fbbfff700 1 -- 172.21.2.102:0/3498771579 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fa800ca40 con 0x7f1fa80067f0 2021-02-10T11:38:28.122 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:28.102 7f1fbbfff700 1 --2- 172.21.2.102:0/3498771579 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1fa80067f0 0x7f1fa8008cf0 secure :-1 s=READY pgs=3090 cs=0 l=1 rev1=1 rx=0x7f1fac006e30 tx=0x7f1fac009e60).ready entity=mon.0 client_cookie=5a35298a47a144aa server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:28.122 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:28.102 7f1fbb7fe700 1 -- 172.21.2.102:0/3498771579 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1fac00b510 con 0x7f1fa80067f0 2021-02-10T11:38:28.122 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:28.102 7f1fbb7fe700 1 -- 172.21.2.102:0/3498771579 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fac006770 con 0x7f1fa80067f0 2021-02-10T11:38:28.122 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:28.102 7f1fbb7fe700 1 -- 172.21.2.102:0/3498771579 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1fac00b850 con 0x7f1fa80067f0 2021-02-10T11:38:28.122 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:28.102 7f1fc2b79700 1 -- 172.21.2.102:0/3498771579 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1fa80067f0 msgr2=0x7f1fa8008cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:28.122 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:28.102 7f1fc2b79700 1 --2- 172.21.2.102:0/3498771579 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1fa80067f0 0x7f1fa8008cf0 secure :-1 s=READY pgs=3090 cs=0 l=1 rev1=1 rx=0x7f1fac006e30 tx=0x7f1fac009e60).stop 2021-02-10T11:38:28.122 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:28.102 7f1fc1196700 1 -- 172.21.2.102:0/3498771579 reap_dead start 2021-02-10T11:38:28.122 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:28.103 7f1fc2b79700 1 -- 172.21.2.102:0/3498771579 shutdown_connections 2021-02-10T11:38:28.122 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:28.103 7f1fc2b79700 1 -- 172.21.2.102:0/3498771579 >> 172.21.2.102:0/3498771579 conn(0x7f1fbc0ce420 msgr2=0x7f1fbc0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:28.122 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:28.103 7f1fc2b79700 1 -- 172.21.2.102:0/3498771579 shutdown_connections 2021-02-10T11:38:28.122 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:28.103 7f1fc2b79700 1 -- 172.21.2.102:0/3498771579 wait complete. 2021-02-10T11:38:28.123 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:28.118 7f1fc2b79700 1 Processor -- start 2021-02-10T11:38:28.123 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:28.118 7f1fc2b79700 1 -- start start 2021-02-10T11:38:28.124 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:28.118 7f1fc2b79700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1fbc1a79a0 0x7f1fbc1a7da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:28.124 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:28.118 7f1fc2b79700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1fbc0bdb40 0x7f1fbc0bdf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:28.124 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:28.118 7f1fc2b79700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1fbc082ca0 0x7f1fbc0830a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:28.124 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:28.118 7f1fc2b79700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fbc19d990 con 0x7f1fbc0bdb40 2021-02-10T11:38:28.124 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:28.118 7f1fc2b79700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fbc19dbb0 con 0x7f1fbc1a79a0 2021-02-10T11:38:28.124 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:28.118 7f1fc2b79700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fbc0836c0 con 0x7f1fbc082ca0 2021-02-10T11:38:28.124 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:28.119 7f1fbbfff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1fbc0bdb40 0x7f1fbc0bdf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:28.124 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:28.119 7f1fbbfff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1fbc0bdb40 0x7f1fbc0bdf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:39522/0 (socket says 172.21.2.102:39522) 2021-02-10T11:38:28.124 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:28.119 7f1fbbfff700 1 -- 172.21.2.102:0/316614251 learned_addr learned my addr 172.21.2.102:0/316614251 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:38:28.124 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:28.119 7f1fbbfff700 1 -- 172.21.2.102:0/316614251 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1fbc082ca0 msgr2=0x7f1fbc0830a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:28.124 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:28.119 7f1fbbfff700 1 --2- 172.21.2.102:0/316614251 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1fbc082ca0 0x7f1fbc0830a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:28.125 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:28.119 7f1fbbfff700 1 -- 172.21.2.102:0/316614251 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1fbc1a79a0 msgr2=0x7f1fbc1a7da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:28.125 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:28.119 7f1fbbfff700 1 --2- 172.21.2.102:0/316614251 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1fbc1a79a0 0x7f1fbc1a7da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:28.125 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:28.119 7f1fbbfff700 1 -- 172.21.2.102:0/316614251 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fbc0838e0 con 0x7f1fbc0bdb40 2021-02-10T11:38:28.125 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:28.119 7f1fbbfff700 1 --2- 172.21.2.102:0/316614251 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1fbc0bdb40 0x7f1fbc0bdf40 secure :-1 s=READY pgs=3091 cs=0 l=1 rev1=1 rx=0x7f1fac006c40 tx=0x7f1fac00bd40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:28.125 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:28.119 7f1fb97fa700 1 -- 172.21.2.102:0/316614251 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1fac00c2f0 con 0x7f1fbc0bdb40 2021-02-10T11:38:28.125 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:28.119 7f1fb97fa700 1 -- 172.21.2.102:0/316614251 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fac00c530 con 0x7f1fbc0bdb40 2021-02-10T11:38:28.125 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:28.119 7f1fb97fa700 1 -- 172.21.2.102:0/316614251 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1fac00c980 con 0x7f1fbc0bdb40 2021-02-10T11:38:28.125 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:28.119 7f1fc2b79700 1 -- 172.21.2.102:0/316614251 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fbc0838e0 con 0x7f1fbc0bdb40 2021-02-10T11:38:28.125 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:28.119 7f1fc2b79700 1 -- 172.21.2.102:0/316614251 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fbc084420 con 0x7f1fbc0bdb40 2021-02-10T11:38:28.125 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:28.120 7f1fb97fa700 1 -- 172.21.2.102:0/316614251 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1fac000cc0 con 0x7f1fbc0bdb40 2021-02-10T11:38:28.125 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:28.120 7f1fb97fa700 1 --2- 172.21.2.102:0/316614251 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1fa00186c0 0x7f1fa001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:28.125 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:28.120 7f1fb97fa700 1 -- 172.21.2.102:0/316614251 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(176..176 src has 1..176) v4 ==== 4753+0+0 (secure 0 0 0) 0x7f1fac032600 con 0x7f1fbc0bdb40 2021-02-10T11:38:28.125 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:28.120 7f1fc0995700 1 --2- 172.21.2.102:0/316614251 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1fa00186c0 0x7f1fa001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:28.126 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:28.120 7f1fc0995700 1 --2- 172.21.2.102:0/316614251 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1fa00186c0 0x7f1fa001abc0 crc :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:28.151 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:28.150 7f1fc2b79700 1 -- 172.21.2.102:0/316614251 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fbc082a70 con 0x7f1fbc0bdb40 2021-02-10T11:38:28.153 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:28.152 7f1fb97fa700 1 -- 172.21.2.102:0/316614251 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f1fac009750 con 0x7f1fbc0bdb40 2021-02-10T11:38:28.281 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:28.280 7f1fc2b79700 1 -- 172.21.2.102:0/316614251 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"} v 0) v1 -- 0x7f1fbc0838e0 con 0x7f1fbc0bdb40 2021-02-10T11:38:28.694 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:28.692 7f1fb97fa700 1 -- 172.21.2.102:0/316614251 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"}]=0 pool 'cephfs_metadata' created v177) v1 ==== 133+0+0 (secure 0 0 0) 0x7f1fac000cc0 con 0x7f1fbc0bdb40 2021-02-10T11:38:28.697 INFO:teuthology.orchestra.run.gibba002.stderr:pool 'cephfs_metadata' created 2021-02-10T11:38:28.698 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:28.697 7f1fc2b79700 1 -- 172.21.2.102:0/316614251 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1fa00186c0 msgr2=0x7f1fa001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:28.699 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:28.697 7f1fc2b79700 1 --2- 172.21.2.102:0/316614251 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1fa00186c0 0x7f1fa001abc0 crc :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:28.699 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:28.697 7f1fc2b79700 1 -- 172.21.2.102:0/316614251 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1fbc0bdb40 msgr2=0x7f1fbc0bdf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:28.699 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:28.697 7f1fc2b79700 1 --2- 172.21.2.102:0/316614251 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1fbc0bdb40 0x7f1fbc0bdf40 secure :-1 s=READY pgs=3091 cs=0 l=1 rev1=1 rx=0x7f1fac006c40 tx=0x7f1fac00bd40).stop 2021-02-10T11:38:28.699 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:28.698 7f1fc2b79700 1 -- 172.21.2.102:0/316614251 shutdown_connections 2021-02-10T11:38:28.699 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:28.698 7f1fc2b79700 1 --2- 172.21.2.102:0/316614251 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1fbc082ca0 0x7f1fbc0830a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:28.699 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:28.698 7f1fc2b79700 1 --2- 172.21.2.102:0/316614251 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1fbc0bdb40 0x7f1fbc0bdf40 unknown :-1 s=CLOSED pgs=3091 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:28.700 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:28.698 7f1fc2b79700 1 --2- 172.21.2.102:0/316614251 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1fa00186c0 0x7f1fa001abc0 unknown :-1 s=CLOSED pgs=1354 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:28.700 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:28.698 7f1fc2b79700 1 --2- 172.21.2.102:0/316614251 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1fbc1a79a0 0x7f1fbc1a7da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:28.700 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:28.698 7f1fc2b79700 1 -- 172.21.2.102:0/316614251 >> 172.21.2.102:0/316614251 conn(0x7f1fbc0bef00 msgr2=0x7f1fbc09f200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:28.700 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:28.698 7f1fc2b79700 1 -- 172.21.2.102:0/316614251 shutdown_connections 2021-02-10T11:38:28.700 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:28.698 7f1fc2b79700 1 -- 172.21.2.102:0/316614251 wait complete. 2021-02-10T11:38:28.708 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool create cephfs_data 8 2021-02-10T11:38:29.142 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:29.123 7fb272e21700 1 Processor -- start 2021-02-10T11:38:29.142 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:29.123 7fb272e21700 1 -- start start 2021-02-10T11:38:29.142 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:29.124 7fb272e21700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fb26c0d1870 0x7fb26c1b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:29.142 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:29.124 7fb272e21700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb26c0fe1c0 con 0x7fb26c0cff70 2021-02-10T11:38:29.142 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:29.124 7fb272e21700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb26c109a60 con 0x7fb26c1b0ae0 2021-02-10T11:38:29.142 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:29.124 7fb272e21700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb26c088070 con 0x7fb26c0d1870 2021-02-10T11:38:29.142 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:29.124 7fb27143e700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fb26c0cff70 0x7fb26c109130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:59986/0 (socket says 172.21.2.102:59986) 2021-02-10T11:38:29.142 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:29.124 7fb27143e700 1 -- 172.21.2.102:0/3919204217 learned_addr learned my addr 172.21.2.102:0/3919204217 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:38:29.143 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:29.124 7fb26b7fe700 1 -- 172.21.2.102:0/3919204217 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3178211476 0 0) 0x7fb264001180 con 0x7fb26c0cff70 2021-02-10T11:38:29.143 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:29.124 7fb26b7fe700 1 -- 172.21.2.102:0/3919204217 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb258002fe0 con 0x7fb26c0cff70 2021-02-10T11:38:29.143 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:29.124 7fb270c3d700 1 --2- 172.21.2.102:0/3919204217 >> v2:172.21.2.113:3300/0 conn(0x7fb26c0d1870 0x7fb26c1b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:29.143 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:29.124 7fb26b7fe700 1 -- 172.21.2.102:0/3919204217 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 562181163 0 0) 0x7fb2640010e0 con 0x7fb26c0cff70 2021-02-10T11:38:29.143 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:29.125 7fb26b7fe700 1 -- 172.21.2.102:0/3919204217 >> v2:172.21.2.113:3300/0 conn(0x7fb26c0d1870 msgr2=0x7fb26c1b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:29.143 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:29.125 7fb26b7fe700 1 --2- 172.21.2.102:0/3919204217 >> v2:172.21.2.113:3300/0 conn(0x7fb26c0d1870 0x7fb26c1b04c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:29.143 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:29.125 7fb26b7fe700 1 -- 172.21.2.102:0/3919204217 >> v1:172.21.2.113:6789/0 conn(0x7fb26c1b0ae0 legacy=0x7fb26c108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:29.143 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:29.125 7fb26b7fe700 1 -- 172.21.2.102:0/3919204217 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb26c088260 con 0x7fb26c0cff70 2021-02-10T11:38:29.143 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:29.125 7fb26b7fe700 1 -- 172.21.2.102:0/3919204217 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7fb264001600 con 0x7fb26c0cff70 2021-02-10T11:38:29.143 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:29.125 7fb26b7fe700 1 -- 172.21.2.102:0/3919204217 >> v1:172.21.2.102:6789/0 conn(0x7fb26c0cff70 legacy=0x7fb26c109130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:29.143 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:29.125 7fb26b7fe700 1 --2- 172.21.2.102:0/3919204217 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb258003b20 0x7fb258006000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:29.143 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:29.125 7fb26b7fe700 1 --2- 172.21.2.102:0/3919204217 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb258006820 0x7fb258008d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:29.143 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:29.125 7fb26b7fe700 1 --2- 172.21.2.102:0/3919204217 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb258009620 0x7fb25800bb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:29.144 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:29.125 7fb26b7fe700 1 -- 172.21.2.102:0/3919204217 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb25800c390 con 0x7fb258003b20 2021-02-10T11:38:29.144 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:29.125 7fb26b7fe700 1 -- 172.21.2.102:0/3919204217 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb25800c5b0 con 0x7fb258009620 2021-02-10T11:38:29.144 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:29.125 7fb26b7fe700 1 -- 172.21.2.102:0/3919204217 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb25800c7d0 con 0x7fb258006820 2021-02-10T11:38:29.144 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:29.125 7fb270c3d700 1 --2- 172.21.2.102:0/3919204217 >> v2:172.21.2.113:3300/0 conn(0x7fb26c0d1870 0x7fb26c1b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:38:29.144 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:29.125 7fb270c3d700 1 --2- 172.21.2.102:0/3919204217 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb258003b20 0x7fb258006000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:29.144 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:29.125 7fb26bfff700 1 --2- 172.21.2.102:0/3919204217 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb258006820 0x7fb258008d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:29.144 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:29.125 7fb27143e700 1 --2- 172.21.2.102:0/3919204217 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb258009620 0x7fb25800bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:29.144 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:29.125 7fb270c3d700 1 -- 172.21.2.102:0/3919204217 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb258006820 msgr2=0x7fb258008d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:29.144 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:29.125 7fb270c3d700 1 --2- 172.21.2.102:0/3919204217 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb258006820 0x7fb258008d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:29.144 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:29.125 7fb270c3d700 1 -- 172.21.2.102:0/3919204217 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb258009620 msgr2=0x7fb25800bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:29.144 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:29.125 7fb270c3d700 1 --2- 172.21.2.102:0/3919204217 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb258009620 0x7fb25800bb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:29.144 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:29.125 7fb270c3d700 1 -- 172.21.2.102:0/3919204217 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb25800ca70 con 0x7fb258003b20 2021-02-10T11:38:29.144 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:29.125 7fb27143e700 1 -- 172.21.2.102:0/3919204217 reap_dead start 2021-02-10T11:38:29.145 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:29.125 7fb270c3d700 1 --2- 172.21.2.102:0/3919204217 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb258003b20 0x7fb258006000 secure :-1 s=READY pgs=3093 cs=0 l=1 rev1=1 rx=0x7fb254006cf0 tx=0x7fb25404dd20).ready entity=mon.0 client_cookie=cf35121e0d9ffbb4 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:29.145 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:29.125 7fb26b7fe700 1 -- 172.21.2.102:0/3919204217 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb2540065d0 con 0x7fb258003b20 2021-02-10T11:38:29.145 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:29.125 7fb26b7fe700 1 -- 172.21.2.102:0/3919204217 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb254006910 con 0x7fb258003b20 2021-02-10T11:38:29.145 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:29.125 7fb26b7fe700 1 -- 172.21.2.102:0/3919204217 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb2540020d0 con 0x7fb258003b20 2021-02-10T11:38:29.145 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:29.125 7fb272e21700 1 -- 172.21.2.102:0/3919204217 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb258003b20 msgr2=0x7fb258006000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:29.145 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:29.125 7fb272e21700 1 --2- 172.21.2.102:0/3919204217 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb258003b20 0x7fb258006000 secure :-1 s=READY pgs=3093 cs=0 l=1 rev1=1 rx=0x7fb254006cf0 tx=0x7fb25404dd20).stop 2021-02-10T11:38:29.145 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:29.126 7fb272e21700 1 -- 172.21.2.102:0/3919204217 shutdown_connections 2021-02-10T11:38:29.145 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:29.126 7fb272e21700 1 --2- 172.21.2.102:0/3919204217 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb258003b20 0x7fb258006000 unknown :-1 s=CLOSED pgs=3093 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:29.145 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:29.126 7fb272e21700 1 -- 172.21.2.102:0/3919204217 >> 172.21.2.102:0/3919204217 conn(0x7fb26c0ce420 msgr2=0x7fb26c0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:29.145 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:29.126 7fb272e21700 1 -- 172.21.2.102:0/3919204217 shutdown_connections 2021-02-10T11:38:29.145 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:29.126 7fb272e21700 1 -- 172.21.2.102:0/3919204217 wait complete. 2021-02-10T11:38:29.145 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:29.141 7fb272e21700 1 Processor -- start 2021-02-10T11:38:29.145 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:29.141 7fb272e21700 1 -- start start 2021-02-10T11:38:29.147 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:29.141 7fb272e21700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb26c1a79a0 0x7fb26c1a7da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:29.147 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:29.141 7fb272e21700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb26c0bdb40 0x7fb26c0bdf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:29.147 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:29.141 7fb272e21700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb26c082ca0 0x7fb26c0830a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:29.147 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:29.141 7fb272e21700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb26c19d990 con 0x7fb26c082ca0 2021-02-10T11:38:29.147 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:29.141 7fb272e21700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb26c19dbb0 con 0x7fb26c1a79a0 2021-02-10T11:38:29.147 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:29.141 7fb272e21700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb26c0836c0 con 0x7fb26c0bdb40 2021-02-10T11:38:29.147 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:29.141 7fb27143e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb26c082ca0 0x7fb26c0830a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:29.147 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:29.141 7fb27143e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb26c082ca0 0x7fb26c0830a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:39540/0 (socket says 172.21.2.102:39540) 2021-02-10T11:38:29.147 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:29.141 7fb27143e700 1 -- 172.21.2.102:0/144212247 learned_addr learned my addr 172.21.2.102:0/144212247 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:38:29.148 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:29.142 7fb26bfff700 1 --2- 172.21.2.102:0/144212247 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb26c0bdb40 0x7fb26c0bdf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:29.148 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:29.142 7fb270c3d700 1 --2- 172.21.2.102:0/144212247 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb26c1a79a0 0x7fb26c1a7da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:29.148 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:29.142 7fb27143e700 1 -- 172.21.2.102:0/144212247 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb26c0bdb40 msgr2=0x7fb26c0bdf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:29.148 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:29.142 7fb27143e700 1 --2- 172.21.2.102:0/144212247 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb26c0bdb40 0x7fb26c0bdf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:29.148 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:29.142 7fb27143e700 1 -- 172.21.2.102:0/144212247 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb26c1a79a0 msgr2=0x7fb26c1a7da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:29.148 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:29.142 7fb27143e700 1 --2- 172.21.2.102:0/144212247 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb26c1a79a0 0x7fb26c1a7da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:29.148 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:29.142 7fb27143e700 1 -- 172.21.2.102:0/144212247 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb26c0838e0 con 0x7fb26c082ca0 2021-02-10T11:38:29.148 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:29.142 7fb270c3d700 1 --2- 172.21.2.102:0/144212247 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb26c1a79a0 0x7fb26c1a7da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:38:29.148 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:29.142 7fb27143e700 1 --2- 172.21.2.102:0/144212247 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb26c082ca0 0x7fb26c0830a0 secure :-1 s=READY pgs=3094 cs=0 l=1 rev1=1 rx=0x7fb26400a570 tx=0x7fb26400a910).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:29.148 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:29.142 7fb2697fa700 1 -- 172.21.2.102:0/144212247 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb2640070d0 con 0x7fb26c082ca0 2021-02-10T11:38:29.148 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:29.142 7fb2697fa700 1 -- 172.21.2.102:0/144212247 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb26400ad00 con 0x7fb26c082ca0 2021-02-10T11:38:29.148 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:29.142 7fb2697fa700 1 -- 172.21.2.102:0/144212247 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb264004620 con 0x7fb26c082ca0 2021-02-10T11:38:29.148 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:29.142 7fb272e21700 1 -- 172.21.2.102:0/144212247 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb26c0838e0 con 0x7fb26c082ca0 2021-02-10T11:38:29.149 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:29.142 7fb272e21700 1 -- 172.21.2.102:0/144212247 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb26c084420 con 0x7fb26c082ca0 2021-02-10T11:38:29.149 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:29.143 7fb2697fa700 1 -- 172.21.2.102:0/144212247 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fb2640043a0 con 0x7fb26c082ca0 2021-02-10T11:38:29.149 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:29.143 7fb2697fa700 1 --2- 172.21.2.102:0/144212247 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb2500186c0 0x7fb25001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:29.149 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:29.143 7fb2697fa700 1 -- 172.21.2.102:0/144212247 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(177..177 src has 1..177) v4 ==== 5107+0+0 (secure 0 0 0) 0x7fb264032310 con 0x7fb26c082ca0 2021-02-10T11:38:29.149 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:29.143 7fb270c3d700 1 --2- 172.21.2.102:0/144212247 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb2500186c0 0x7fb25001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:29.149 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:29.143 7fb270c3d700 1 --2- 172.21.2.102:0/144212247 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb2500186c0 0x7fb25001abc0 crc :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:29.175 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:29.174 7fb272e21700 1 -- 172.21.2.102:0/144212247 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb26c082a70 con 0x7fb26c082ca0 2021-02-10T11:38:29.177 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:29.176 7fb2697fa700 1 -- 172.21.2.102:0/144212247 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fb264022490 con 0x7fb26c082ca0 2021-02-10T11:38:29.309 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:29.308 7fb272e21700 1 -- 172.21.2.102:0/144212247 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_data"} v 0) v1 -- 0x7fb26c082a70 con 0x7fb26c082ca0 2021-02-10T11:38:29.700 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:29.699 7fb2697fa700 1 -- 172.21.2.102:0/144212247 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_data"}]=0 pool 'cephfs_data' created v178) v1 ==== 125+0+0 (secure 0 0 0) 0x7fb264022490 con 0x7fb26c082ca0 2021-02-10T11:38:29.724 INFO:teuthology.orchestra.run.gibba002.stderr:pool 'cephfs_data' created 2021-02-10T11:38:29.726 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:29.725 7fb272e21700 1 -- 172.21.2.102:0/144212247 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb2500186c0 msgr2=0x7fb25001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:29.726 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:29.725 7fb272e21700 1 --2- 172.21.2.102:0/144212247 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb2500186c0 0x7fb25001abc0 crc :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:29.726 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:29.725 7fb272e21700 1 -- 172.21.2.102:0/144212247 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb26c082ca0 msgr2=0x7fb26c0830a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:29.726 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:29.725 7fb272e21700 1 --2- 172.21.2.102:0/144212247 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb26c082ca0 0x7fb26c0830a0 secure :-1 s=READY pgs=3094 cs=0 l=1 rev1=1 rx=0x7fb26400a570 tx=0x7fb26400a910).stop 2021-02-10T11:38:29.727 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:29.725 7fb272e21700 1 -- 172.21.2.102:0/144212247 shutdown_connections 2021-02-10T11:38:29.727 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:29.725 7fb272e21700 1 --2- 172.21.2.102:0/144212247 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb26c082ca0 0x7fb26c0830a0 unknown :-1 s=CLOSED pgs=3094 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:29.727 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:29.725 7fb272e21700 1 --2- 172.21.2.102:0/144212247 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb26c0bdb40 0x7fb26c0bdf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:29.727 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:29.725 7fb272e21700 1 --2- 172.21.2.102:0/144212247 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb2500186c0 0x7fb25001abc0 unknown :-1 s=CLOSED pgs=1355 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:29.727 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:29.725 7fb272e21700 1 --2- 172.21.2.102:0/144212247 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb26c1a79a0 0x7fb26c1a7da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:29.727 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:29.725 7fb272e21700 1 -- 172.21.2.102:0/144212247 >> 172.21.2.102:0/144212247 conn(0x7fb26c0bef00 msgr2=0x7fb26c09f200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:29.727 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:29.726 7fb272e21700 1 -- 172.21.2.102:0/144212247 shutdown_connections 2021-02-10T11:38:29.727 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:29.726 7fb272e21700 1 -- 172.21.2.102:0/144212247 wait complete. 2021-02-10T11:38:29.735 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2021-02-10T11:38:30.168 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:30.150 7f0f4d730700 1 Processor -- start 2021-02-10T11:38:30.168 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:30.150 7f0f4d730700 1 -- start start 2021-02-10T11:38:30.168 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:30.151 7f0f4d730700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f0f480d1c80 0x7f0f480cef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:30.168 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:30.151 7f0f4d730700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f480d0180 con 0x7f0f480d1880 2021-02-10T11:38:30.168 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:30.151 7f0f4d730700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f480fe1d0 con 0x7f0f480cf590 2021-02-10T11:38:30.168 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:30.151 7f0f4d730700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f481b0cc0 con 0x7f0f480d1c80 2021-02-10T11:38:30.168 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:30.151 7f0f46d48700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f0f480d1c80 0x7f0f480cef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:30.168 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:30.151 7f0f46d48700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f0f480d1c80 0x7f0f480cef70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:39550/0 (socket says 172.21.2.102:39550) 2021-02-10T11:38:30.168 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:30.151 7f0f46d48700 1 -- 172.21.2.102:0/4118277962 learned_addr learned my addr 172.21.2.102:0/4118277962 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:38:30.168 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:30.151 7f0f46d48700 1 -- 172.21.2.102:0/4118277962 >> v1:172.21.2.113:6789/0 conn(0x7f0f480cf590 legacy=0x7f0f480cf990 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:38:30.169 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:30.151 7f0f46d48700 1 -- 172.21.2.102:0/4118277962 >> v1:172.21.2.102:6789/0 conn(0x7f0f480d1880 legacy=0x7f0f481b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:30.169 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:30.151 7f0f46d48700 1 -- 172.21.2.102:0/4118277962 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f48109a70 con 0x7f0f480d1c80 2021-02-10T11:38:30.169 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:30.152 7f0f46d48700 1 --2- 172.21.2.102:0/4118277962 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0f480d1c80 0x7f0f480cef70 secure :-1 s=READY pgs=3096 cs=0 l=1 rev1=1 rx=0x7f0f40006a80 tx=0x7f0f40009f00).ready entity=mon.0 client_cookie=78796de17655eb81 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:30.169 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:30.152 7f0f46547700 1 -- 172.21.2.102:0/4118277962 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0f4000b5c0 con 0x7f0f480d1c80 2021-02-10T11:38:30.169 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:30.152 7f0f46547700 1 -- 172.21.2.102:0/4118277962 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f4000b8c0 con 0x7f0f480d1c80 2021-02-10T11:38:30.169 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:30.152 7f0f46547700 1 -- 172.21.2.102:0/4118277962 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0f4000bda0 con 0x7f0f480d1c80 2021-02-10T11:38:30.169 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:30.152 7f0f4d730700 1 -- 172.21.2.102:0/4118277962 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0f480d1c80 msgr2=0x7f0f480cef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:30.169 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:30.152 7f0f4d730700 1 --2- 172.21.2.102:0/4118277962 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0f480d1c80 0x7f0f480cef70 secure :-1 s=READY pgs=3096 cs=0 l=1 rev1=1 rx=0x7f0f40006a80 tx=0x7f0f40009f00).stop 2021-02-10T11:38:30.169 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:30.152 7f0f4d730700 1 -- 172.21.2.102:0/4118277962 shutdown_connections 2021-02-10T11:38:30.169 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:30.152 7f0f4d730700 1 --2- 172.21.2.102:0/4118277962 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0f480d1c80 0x7f0f480cef70 unknown :-1 s=CLOSED pgs=3096 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:30.169 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:30.152 7f0f4d730700 1 -- 172.21.2.102:0/4118277962 >> 172.21.2.102:0/4118277962 conn(0x7f0f480ce430 msgr2=0x7f0f480d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:30.169 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:30.152 7f0f4d730700 1 -- 172.21.2.102:0/4118277962 shutdown_connections 2021-02-10T11:38:30.169 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:30.152 7f0f4d730700 1 -- 172.21.2.102:0/4118277962 wait complete. 2021-02-10T11:38:30.172 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:30.167 7f0f4d730700 1 Processor -- start 2021-02-10T11:38:30.172 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:30.167 7f0f4d730700 1 -- start start 2021-02-10T11:38:30.172 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:30.168 7f0f4d730700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0f48097820 0x7f0f48097c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:30.172 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:30.168 7f0f4d730700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0f4808ac80 0x7f0f4808b080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:30.172 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:30.168 7f0f4d730700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0f4808bf00 0x7f0f4808c300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:30.172 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:30.168 7f0f4d730700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f4808b6a0 con 0x7f0f48097820 2021-02-10T11:38:30.172 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:30.168 7f0f4d730700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f4809a360 con 0x7f0f4808bf00 2021-02-10T11:38:30.172 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:30.168 7f0f4d730700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f4809a580 con 0x7f0f4808ac80 2021-02-10T11:38:30.172 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:30.168 7f0f47549700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0f48097820 0x7f0f48097c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:30.172 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:30.168 7f0f47549700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0f48097820 0x7f0f48097c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:39554/0 (socket says 172.21.2.102:39554) 2021-02-10T11:38:30.172 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:30.168 7f0f47549700 1 -- 172.21.2.102:0/2215419244 learned_addr learned my addr 172.21.2.102:0/2215419244 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:38:30.173 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:30.168 7f0f46d48700 1 --2- 172.21.2.102:0/2215419244 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0f4808ac80 0x7f0f4808b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:30.173 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:30.168 7f0f47d4a700 1 --2- 172.21.2.102:0/2215419244 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0f4808bf00 0x7f0f4808c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:30.173 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:30.168 7f0f47549700 1 -- 172.21.2.102:0/2215419244 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0f4808ac80 msgr2=0x7f0f4808b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:30.173 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:30.168 7f0f47549700 1 --2- 172.21.2.102:0/2215419244 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0f4808ac80 0x7f0f4808b080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:30.173 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:30.168 7f0f47549700 1 -- 172.21.2.102:0/2215419244 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0f4808bf00 msgr2=0x7f0f4808c300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:30.173 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:30.168 7f0f47549700 1 --2- 172.21.2.102:0/2215419244 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0f4808bf00 0x7f0f4808c300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:30.173 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:30.168 7f0f47549700 1 -- 172.21.2.102:0/2215419244 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f4808c920 con 0x7f0f48097820 2021-02-10T11:38:30.173 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:30.168 7f0f47d4a700 1 --2- 172.21.2.102:0/2215419244 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0f4808bf00 0x7f0f4808c300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:38:30.173 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:30.169 7f0f47549700 1 --2- 172.21.2.102:0/2215419244 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0f48097820 0x7f0f48097c20 secure :-1 s=READY pgs=3097 cs=0 l=1 rev1=1 rx=0x7f0f3c006f00 tx=0x7f0f3c00b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:30.173 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:30.169 7f0f2ffff700 1 -- 172.21.2.102:0/2215419244 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0f3c007070 con 0x7f0f48097820 2021-02-10T11:38:30.173 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:30.169 7f0f2ffff700 1 -- 172.21.2.102:0/2215419244 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f3c007270 con 0x7f0f48097820 2021-02-10T11:38:30.173 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:30.169 7f0f4d730700 1 -- 172.21.2.102:0/2215419244 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f4808c920 con 0x7f0f48097820 2021-02-10T11:38:30.173 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:30.169 7f0f4d730700 1 -- 172.21.2.102:0/2215419244 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f4808c920 con 0x7f0f48097820 2021-02-10T11:38:30.174 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:30.169 7f0f2ffff700 1 -- 172.21.2.102:0/2215419244 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0f3c007750 con 0x7f0f48097820 2021-02-10T11:38:30.174 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:30.169 7f0f2ffff700 1 -- 172.21.2.102:0/2215419244 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f0f3c0055c0 con 0x7f0f48097820 2021-02-10T11:38:30.174 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:30.170 7f0f2ffff700 1 --2- 172.21.2.102:0/2215419244 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0f280186d0 0x7f0f2801abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:30.174 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:30.170 7f0f2ffff700 1 -- 172.21.2.102:0/2215419244 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(178..178 src has 1..178) v4 ==== 5457+0+0 (secure 0 0 0) 0x7f0f3c032620 con 0x7f0f48097820 2021-02-10T11:38:30.174 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:30.170 7f0f46d48700 1 --2- 172.21.2.102:0/2215419244 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0f280186d0 0x7f0f2801abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:30.174 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:30.170 7f0f46d48700 1 --2- 172.21.2.102:0/2215419244 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0f280186d0 0x7f0f2801abd0 crc :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:30.202 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:30.201 7f0f4d730700 1 -- 172.21.2.102:0/2215419244 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f48080340 con 0x7f0f48097820 2021-02-10T11:38:30.204 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:30.203 7f0f2ffff700 1 -- 172.21.2.102:0/2215419244 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f0f3c006b90 con 0x7f0f48097820 2021-02-10T11:38:30.336 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:30.334 7f0f4d730700 1 -- 172.21.2.102:0/2215419244 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"data": "cephfs_data", "prefix": "fs new", "force": true, "fs_name": "cephfs", "metadata": "cephfs_metadata"} v 0) v1 -- 0x7f0f48080340 con 0x7f0f48097820 2021-02-10T11:38:30.339 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:30.338 7f0f2ffff700 1 -- 172.21.2.102:0/2215419244 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"data": "cephfs_data", "prefix": "fs new", "force": true, "fs_name": "cephfs", "metadata": "cephfs_metadata"}]=0 new fs with metadata pool 38 and data pool 39 v154) v1 ==== 189+0+0 (secure 0 0 0) 0x7f0f3c007270 con 0x7f0f48097820 2021-02-10T11:38:30.343 INFO:teuthology.orchestra.run.gibba002.stderr:new fs with metadata pool 38 and data pool 39 2021-02-10T11:38:30.344 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:30.343 7f0f4d730700 1 -- 172.21.2.102:0/2215419244 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0f280186d0 msgr2=0x7f0f2801abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:30.345 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:30.343 7f0f4d730700 1 --2- 172.21.2.102:0/2215419244 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0f280186d0 0x7f0f2801abd0 crc :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:30.345 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:30.343 7f0f4d730700 1 -- 172.21.2.102:0/2215419244 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0f48097820 msgr2=0x7f0f48097c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:30.345 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:30.343 7f0f4d730700 1 --2- 172.21.2.102:0/2215419244 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0f48097820 0x7f0f48097c20 secure :-1 s=READY pgs=3097 cs=0 l=1 rev1=1 rx=0x7f0f3c006f00 tx=0x7f0f3c00b940).stop 2021-02-10T11:38:30.345 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:30.343 7f0f4d730700 1 -- 172.21.2.102:0/2215419244 shutdown_connections 2021-02-10T11:38:30.345 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:30.343 7f0f4d730700 1 --2- 172.21.2.102:0/2215419244 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0f280186d0 0x7f0f2801abd0 unknown :-1 s=CLOSED pgs=1358 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:30.345 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:30.343 7f0f4d730700 1 --2- 172.21.2.102:0/2215419244 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0f4808bf00 0x7f0f4808c300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:30.345 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:30.343 7f0f4d730700 1 --2- 172.21.2.102:0/2215419244 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0f4808ac80 0x7f0f4808b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:30.345 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:30.343 7f0f4d730700 1 --2- 172.21.2.102:0/2215419244 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0f48097820 0x7f0f48097c20 unknown :-1 s=CLOSED pgs=3097 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:30.345 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:30.343 7f0f4d730700 1 -- 172.21.2.102:0/2215419244 >> 172.21.2.102:0/2215419244 conn(0x7f0f4809fb50 msgr2=0x7f0f480fed60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:30.345 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:30.343 7f0f4d730700 1 -- 172.21.2.102:0/2215419244 shutdown_connections 2021-02-10T11:38:30.345 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:30.344 7f0f4d730700 1 -- 172.21.2.102:0/2215419244 wait complete. 2021-02-10T11:38:30.353 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:38:30.783 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:30.765 7f5645c8b700 1 Processor -- start 2021-02-10T11:38:30.783 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:30.765 7f5645c8b700 1 -- start start 2021-02-10T11:38:30.783 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:30.765 7f5645c8b700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f56400d0340 0x7f56400ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:30.783 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:30.765 7f5645c8b700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f56400d1c70 0x7f56400efc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:30.783 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:30.765 7f5645c8b700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56400f0b10 con 0x7f56400d1870 2021-02-10T11:38:30.783 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:30.765 7f5645c8b700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56400e5ed0 con 0x7f56400d0340 2021-02-10T11:38:30.783 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:30.765 7f5645c8b700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56401b0be0 con 0x7f56400d1c70 2021-02-10T11:38:30.783 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:30.766 7f563effd700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f56400d0340 0x7f56400ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:30.783 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:30.766 7f563effd700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f56400d0340 0x7f56400ef640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:39562/0 (socket says 172.21.2.102:39562) 2021-02-10T11:38:30.783 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:30.766 7f563effd700 1 -- 172.21.2.102:0/4069418178 learned_addr learned my addr 172.21.2.102:0/4069418178 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:38:30.783 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:30.766 7f563effd700 1 -- 172.21.2.102:0/4069418178 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f56400d1c70 msgr2=0x7f56400efc60 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:38:30.784 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:30.766 7f563effd700 1 --2- 172.21.2.102:0/4069418178 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f56400d1c70 0x7f56400efc60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:30.784 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:30.766 7f563effd700 1 -- 172.21.2.102:0/4069418178 >> v1:172.21.2.113:6789/0 conn(0x7f56400d1870 legacy=0x7f56401b03f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:38:30.784 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:30.766 7f563effd700 1 -- 172.21.2.102:0/4069418178 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56400f20a0 con 0x7f56400d0340 2021-02-10T11:38:30.784 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:30.766 7f563effd700 1 --2- 172.21.2.102:0/4069418178 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f56400d0340 0x7f56400ef640 secure :-1 s=READY pgs=3098 cs=0 l=1 rev1=1 rx=0x7f5630006700 tx=0x7f5630009b00).ready entity=mon.0 client_cookie=b5ae319727bd6232 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:30.784 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:30.766 7f563e7fc700 1 -- 172.21.2.102:0/4069418178 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f563000c0c0 con 0x7f56400d0340 2021-02-10T11:38:30.784 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:30.766 7f563e7fc700 1 -- 172.21.2.102:0/4069418178 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f563000c400 con 0x7f56400d0340 2021-02-10T11:38:30.784 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:30.766 7f563e7fc700 1 -- 172.21.2.102:0/4069418178 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f563000c8e0 con 0x7f56400d0340 2021-02-10T11:38:30.784 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:30.766 7f5645c8b700 1 -- 172.21.2.102:0/4069418178 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f56400d0340 msgr2=0x7f56400ef640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:30.784 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:30.766 7f5645c8b700 1 --2- 172.21.2.102:0/4069418178 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f56400d0340 0x7f56400ef640 secure :-1 s=READY pgs=3098 cs=0 l=1 rev1=1 rx=0x7f5630006700 tx=0x7f5630009b00).stop 2021-02-10T11:38:30.784 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:30.767 7f5645c8b700 1 -- 172.21.2.102:0/4069418178 shutdown_connections 2021-02-10T11:38:30.784 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:30.767 7f5645c8b700 1 --2- 172.21.2.102:0/4069418178 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f56400d1c70 0x7f56400efc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:30.784 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:30.767 7f5645c8b700 1 --2- 172.21.2.102:0/4069418178 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f56400d0340 0x7f56400ef640 unknown :-1 s=CLOSED pgs=3098 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:30.785 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:30.767 7f5645c8b700 1 -- 172.21.2.102:0/4069418178 >> 172.21.2.102:0/4069418178 conn(0x7f56400ce450 msgr2=0x7f56400d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:30.785 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:30.767 7f5645c8b700 1 -- 172.21.2.102:0/4069418178 shutdown_connections 2021-02-10T11:38:30.785 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:30.767 7f5645c8b700 1 -- 172.21.2.102:0/4069418178 wait complete. 2021-02-10T11:38:30.786 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:30.782 7f5645c8b700 1 Processor -- start 2021-02-10T11:38:30.786 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:30.782 7f5645c8b700 1 -- start start 2021-02-10T11:38:30.786 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:30.782 7f5645c8b700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f56400b0ad0 0x7f5640091f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:30.786 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:30.782 7f5645c8b700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f56401a8850 0x7f56401a8c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:30.786 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:30.782 7f5645c8b700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f56401a6fa0 0x7f56401a73a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:30.786 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:30.782 7f5645c8b700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56401abfe0 con 0x7f56401a8850 2021-02-10T11:38:30.787 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:30.782 7f5645c8b700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56401a9c40 con 0x7f56400b0ad0 2021-02-10T11:38:30.787 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:30.782 7f5645c8b700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56401a9730 con 0x7f56401a6fa0 2021-02-10T11:38:30.787 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:30.783 7f563effd700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f56401a8850 0x7f56401a8c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:30.787 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:30.783 7f563effd700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f56401a8850 0x7f56401a8c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:39568/0 (socket says 172.21.2.102:39568) 2021-02-10T11:38:30.787 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:30.783 7f563effd700 1 -- 172.21.2.102:0/429357679 learned_addr learned my addr 172.21.2.102:0/429357679 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:38:30.787 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:30.783 7f563effd700 1 -- 172.21.2.102:0/429357679 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f56401a6fa0 msgr2=0x7f56401a73a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:30.787 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:30.783 7f563effd700 1 --2- 172.21.2.102:0/429357679 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f56401a6fa0 0x7f56401a73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:30.787 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:30.783 7f563effd700 1 -- 172.21.2.102:0/429357679 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f56400b0ad0 msgr2=0x7f5640091f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:30.787 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:30.783 7f563effd700 1 --2- 172.21.2.102:0/429357679 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f56400b0ad0 0x7f5640091f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:30.787 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:30.783 7f563effd700 1 -- 172.21.2.102:0/429357679 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56401a9270 con 0x7f56401a8850 2021-02-10T11:38:30.787 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:30.783 7f563effd700 1 --2- 172.21.2.102:0/429357679 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f56401a8850 0x7f56401a8c50 secure :-1 s=READY pgs=3099 cs=0 l=1 rev1=1 rx=0x7f563000b660 tx=0x7f563000ba00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:30.787 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:30.783 7f5637fff700 1 -- 172.21.2.102:0/429357679 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5630009440 con 0x7f56401a8850 2021-02-10T11:38:30.788 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:30.783 7f5637fff700 1 -- 172.21.2.102:0/429357679 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56300090c0 con 0x7f56401a8850 2021-02-10T11:38:30.788 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:30.783 7f5637fff700 1 -- 172.21.2.102:0/429357679 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5630005370 con 0x7f56401a8850 2021-02-10T11:38:30.788 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:30.783 7f5645c8b700 1 -- 172.21.2.102:0/429357679 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56401a9270 con 0x7f56401a8850 2021-02-10T11:38:30.788 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:30.783 7f5645c8b700 1 -- 172.21.2.102:0/429357679 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56401a9270 con 0x7f56401a8850 2021-02-10T11:38:30.788 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:30.784 7f5637fff700 1 -- 172.21.2.102:0/429357679 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f56300062c0 con 0x7f56401a8850 2021-02-10T11:38:30.788 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:30.784 7f5637fff700 1 --2- 172.21.2.102:0/429357679 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f56240186c0 0x7f562401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:30.788 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:30.784 7f5637fff700 1 -- 172.21.2.102:0/429357679 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(179..179 src has 1..179) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f5630032650 con 0x7f56401a8850 2021-02-10T11:38:30.788 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:30.784 7f563f7fe700 1 --2- 172.21.2.102:0/429357679 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f56240186c0 0x7f562401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:30.788 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:30.785 7f563f7fe700 1 --2- 172.21.2.102:0/429357679 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f56240186c0 0x7f562401abc0 crc :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:30.816 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:30.815 7f5645c8b700 1 -- 172.21.2.102:0/429357679 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56401abfe0 con 0x7f56401a8850 2021-02-10T11:38:30.819 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:30.818 7f5637fff700 1 -- 172.21.2.102:0/429357679 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f56300090c0 con 0x7f56401a8850 2021-02-10T11:38:30.945 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:30.943 7f5645c8b700 1 -- 172.21.2.102:0/429357679 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f56401abfe0 con 0x7f56401a8850 2021-02-10T11:38:30.945 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:30.944 7f5637fff700 1 -- 172.21.2.102:0/429357679 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v179) v1 ==== 74+0+14020 (secure 0 0 0) 0x7f56300090c0 con 0x7f56401a8850 2021-02-10T11:38:30.946 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:38:30.946 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":179,"fsid":"68db30ea-dbce-4aef-a6df-f01c6fea0513","created":"2021-02-10 11:03:06.331557","modified":"2021-02-10 11:38:30.335437","last_up_change":"2021-02-10 11:03:32.690562","last_in_change":"2021-02-10 11:03:32.690562","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":39,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:03:37.322519","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":38,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:38:28.298788","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"179","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":39,"pool_name":"cephfs_data","create_time":"2021-02-10 11:38:29.326316","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"179","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"be691b4d-a039-408a-af80-9d53f02b6f1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":178,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6818","nonce":36088},{"type":"v1","addr":"172.21.2.102:6820","nonce":36088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6822","nonce":36088},{"type":"v1","addr":"172.21.2.102:6824","nonce":36088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6830","nonce":36088},{"type":"v1","addr":"172.21.2.102:6832","nonce":36088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6826","nonce":36088},{"type":"v1","addr":"172.21.2.102:6828","nonce":36088}]},"public_addr":"172.21.2.102:6820/36088","cluster_addr":"172.21.2.102:6824/36088","heartbeat_back_addr":"172.21.2.102:6832/36088","heartbeat_front_addr":"172.21.2.102:6828/36088","state":["exists","up"]},{"osd":1,"uuid":"64e3ccc8-9c69-4d7c-865e-af8e83822635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":178,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6819","nonce":36087},{"type":"v1","addr":"172.21.2.102:6821","nonce":36087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6823","nonce":36087},{"type":"v1","addr":"172.21.2.102:6825","nonce":36087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6831","nonce":36087},{"type":"v1","addr":"172.21.2.102:6833","nonce":36087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6827","nonce":36087},{"type":"v1","addr":"172.21.2.102:6829","nonce":36087}]},"public_addr":"172.21.2.102:6821/36087","cluster_addr":"172.21.2.102:6825/36087","heartbeat_back_addr":"172.21.2.102:6833/36087","heartbeat_front_addr":"172.21.2.102:6829/36087","state":["exists","up"]},{"osd":2,"uuid":"153700af-747f-481f-bdf4-d6072d3a842f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":178,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6810","nonce":36089},{"type":"v1","addr":"172.21.2.102:6811","nonce":36089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6812","nonce":36089},{"type":"v1","addr":"172.21.2.102:6813","nonce":36089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6816","nonce":36089},{"type":"v1","addr":"172.21.2.102:6817","nonce":36089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6814","nonce":36089},{"type":"v1","addr":"172.21.2.102:6815","nonce":36089}]},"public_addr":"172.21.2.102:6811/36089","cluster_addr":"172.21.2.102:6813/36089","heartbeat_back_addr":"172.21.2.102:6817/36089","heartbeat_front_addr":"172.21.2.102:6815/36089","state":["exists","up"]},{"osd":3,"uuid":"cbc678f4-83a9-45e6-8830-2f7c232b1ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":178,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6802","nonce":36090},{"type":"v1","addr":"172.21.2.102:6803","nonce":36090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6804","nonce":36090},{"type":"v1","addr":"172.21.2.102:6805","nonce":36090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6808","nonce":36090},{"type":"v1","addr":"172.21.2.102:6809","nonce":36090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6806","nonce":36090},{"type":"v1","addr":"172.21.2.102:6807","nonce":36090}]},"public_addr":"172.21.2.102:6803/36090","cluster_addr":"172.21.2.102:6805/36090","heartbeat_back_addr":"172.21.2.102:6809/36090","heartbeat_front_addr":"172.21.2.102:6807/36090","state":["exists","up"]},{"osd":4,"uuid":"c01e4008-b258-4f10-afd2-9e7f37214351","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":178,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6808","nonce":36368},{"type":"v1","addr":"172.21.2.113:6811","nonce":36368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6814","nonce":36368},{"type":"v1","addr":"172.21.2.113:6818","nonce":36368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6826","nonce":36368},{"type":"v1","addr":"172.21.2.113:6828","nonce":36368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6820","nonce":36368},{"type":"v1","addr":"172.21.2.113:6824","nonce":36368}]},"public_addr":"172.21.2.113:6811/36368","cluster_addr":"172.21.2.113:6818/36368","heartbeat_back_addr":"172.21.2.113:6828/36368","heartbeat_front_addr":"172.21.2.113:6824/36368","state":["exists","up"]},{"osd":5,"uuid":"551f6007-daf8-4a98-8ee0-c2b16bcccb5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":178,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6802","nonce":36366},{"type":"v1","addr":"172.21.2.113:6805","nonce":36366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6807","nonce":36366},{"type":"v1","addr":"172.21.2.113:6810","nonce":36366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6819","nonce":36366},{"type":"v1","addr":"172.21.2.113:6822","nonce":36366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6812","nonce":36366},{"type":"v1","addr":"172.21.2.113:6815","nonce":36366}]},"public_addr":"172.21.2.113:6805/36366","cluster_addr":"172.21.2.113:6810/36366","heartbeat_back_addr":"172.21.2.113:6822/36366","heartbeat_front_addr":"172.21.2.113:6815/36366","state":["exists","up"]},{"osd":6,"uuid":"bf67c040-0e50-48be-b102-c3943bc9ebef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6800","nonce":36367},{"type":"v1","addr":"172.21.2.113:6801","nonce":36367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6803","nonce":36367},{"type":"v1","addr":"172.21.2.113:6804","nonce":36367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6813","nonce":36367},{"type":"v1","addr":"172.21.2.113:6816","nonce":36367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6806","nonce":36367},{"type":"v1","addr":"172.21.2.113:6809","nonce":36367}]},"public_addr":"172.21.2.113:6801/36367","cluster_addr":"172.21.2.113:6804/36367","heartbeat_back_addr":"172.21.2.113:6816/36367","heartbeat_front_addr":"172.21.2.113:6809/36367","state":["exists","up"]},{"osd":7,"uuid":"f5461f80-018d-428f-be89-2eb2bcbe7bb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":178,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6817","nonce":36365},{"type":"v1","addr":"172.21.2.113:6821","nonce":36365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6823","nonce":36365},{"type":"v1","addr":"172.21.2.113:6825","nonce":36365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6830","nonce":36365},{"type":"v1","addr":"172.21.2.113:6831","nonce":36365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6827","nonce":36365},{"type":"v1","addr":"172.21.2.113:6829","nonce":36365}]},"public_addr":"172.21.2.113:6821/36365","cluster_addr":"172.21.2.113:6825/36365","heartbeat_back_addr":"172.21.2.113:6831/36365","heartbeat_front_addr":"172.21.2.113:6829/36365","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:38:30.947 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:30.945 7f5645c8b700 1 -- 172.21.2.102:0/429357679 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f56240186c0 msgr2=0x7f562401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:30.947 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:30.945 7f5645c8b700 1 --2- 172.21.2.102:0/429357679 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f56240186c0 0x7f562401abc0 crc :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:30.947 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:30.946 7f5645c8b700 1 -- 172.21.2.102:0/429357679 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f56401a8850 msgr2=0x7f56401a8c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:30.947 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:30.946 7f5645c8b700 1 --2- 172.21.2.102:0/429357679 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f56401a8850 0x7f56401a8c50 secure :-1 s=READY pgs=3099 cs=0 l=1 rev1=1 rx=0x7f563000b660 tx=0x7f563000ba00).stop 2021-02-10T11:38:30.947 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:30.946 7f5645c8b700 1 -- 172.21.2.102:0/429357679 shutdown_connections 2021-02-10T11:38:30.947 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:30.946 7f5645c8b700 1 --2- 172.21.2.102:0/429357679 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f56401a6fa0 0x7f56401a73a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:30.947 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:30.946 7f5645c8b700 1 --2- 172.21.2.102:0/429357679 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f56401a8850 0x7f56401a8c50 unknown :-1 s=CLOSED pgs=3099 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:30.947 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:30.946 7f5645c8b700 1 --2- 172.21.2.102:0/429357679 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f56240186c0 0x7f562401abc0 unknown :-1 s=CLOSED pgs=1359 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:30.947 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:30.946 7f5645c8b700 1 --2- 172.21.2.102:0/429357679 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f56400b0ad0 0x7f5640091f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:30.947 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:30.946 7f5645c8b700 1 -- 172.21.2.102:0/429357679 >> 172.21.2.102:0/429357679 conn(0x7f564019f8f0 msgr2=0x7f56401a5f80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:30.948 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:30.946 7f5645c8b700 1 -- 172.21.2.102:0/429357679 shutdown_connections 2021-02-10T11:38:30.948 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:30.946 7f5645c8b700 1 -- 172.21.2.102:0/429357679 wait complete. 2021-02-10T11:38:30.956 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:38:31.386 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:31.368 7fe7e9bdd700 1 Processor -- start 2021-02-10T11:38:31.386 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:31.368 7fe7e9bdd700 1 -- start start 2021-02-10T11:38:31.386 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:31.368 7fe7e9bdd700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe7e40d1870 0x7fe7e41b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:31.386 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:31.368 7fe7e9bdd700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7e40f0b10 con 0x7fe7e41b0a10 2021-02-10T11:38:31.386 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:31.368 7fe7e9bdd700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7e40d0310 con 0x7fe7e40f0650 2021-02-10T11:38:31.386 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:31.368 7fe7e9bdd700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7e40e5ed0 con 0x7fe7e40d1870 2021-02-10T11:38:31.386 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:31.368 7fe7e2ffd700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fe7e41b0a10 0x7fe7e40cf330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:60032/0 (socket says 172.21.2.102:60032) 2021-02-10T11:38:31.386 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:31.369 7fe7e2ffd700 1 -- 172.21.2.102:0/766371505 learned_addr learned my addr 172.21.2.102:0/766371505 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:38:31.386 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:31.369 7fe7e37fe700 1 --2- 172.21.2.102:0/766371505 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe7e40d1870 0x7fe7e41b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:31.387 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:31.369 7fe7e27fc700 1 -- 172.21.2.102:0/766371505 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3193359716 0 0) 0x7fe7cc0009b0 con 0x7fe7e41b0a10 2021-02-10T11:38:31.387 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:31.369 7fe7e27fc700 1 -- 172.21.2.102:0/766371505 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7d0002fe0 con 0x7fe7e41b0a10 2021-02-10T11:38:31.387 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:31.369 7fe7e27fc700 1 -- 172.21.2.102:0/766371505 <== mon.1 v1:172.21.2.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1652571079 0 0) 0x7fe7dc001400 con 0x7fe7e40f0650 2021-02-10T11:38:31.387 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:31.369 7fe7e27fc700 1 -- 172.21.2.102:0/766371505 --> v1:172.21.2.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7d0004370 con 0x7fe7e40f0650 2021-02-10T11:38:31.387 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:31.369 7fe7e37fe700 1 -- 172.21.2.102:0/766371505 >> v1:172.21.2.113:6789/0 conn(0x7fe7e40f0650 legacy=0x7fe7e40cfb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:31.387 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:31.369 7fe7e37fe700 1 -- 172.21.2.102:0/766371505 >> v1:172.21.2.102:6789/0 conn(0x7fe7e41b0a10 legacy=0x7fe7e40cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:31.387 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:31.369 7fe7e37fe700 1 -- 172.21.2.102:0/766371505 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7e40d0530 con 0x7fe7e40d1870 2021-02-10T11:38:31.387 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:31.369 7fe7e37fe700 1 --2- 172.21.2.102:0/766371505 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe7e40d1870 0x7fe7e41b03f0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 rx=0x7fe7d4006650 tx=0x7fe7d4009a80).ready entity=mon.2 client_cookie=40a82a22be2d1e64 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:31.387 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:31.370 7fe7e27fc700 1 -- 172.21.2.102:0/766371505 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe7d40009b0 con 0x7fe7e40d1870 2021-02-10T11:38:31.387 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:31.370 7fe7e27fc700 1 -- 172.21.2.102:0/766371505 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7d400c0c0 con 0x7fe7e40d1870 2021-02-10T11:38:31.387 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:31.370 7fe7e27fc700 1 -- 172.21.2.102:0/766371505 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe7d400c520 con 0x7fe7e40d1870 2021-02-10T11:38:31.387 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:31.370 7fe7e9bdd700 1 -- 172.21.2.102:0/766371505 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe7e40d1870 msgr2=0x7fe7e41b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:31.387 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:31.370 7fe7e9bdd700 1 --2- 172.21.2.102:0/766371505 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe7e40d1870 0x7fe7e41b03f0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 rx=0x7fe7d4006650 tx=0x7fe7d4009a80).stop 2021-02-10T11:38:31.388 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:31.370 7fe7e9bdd700 1 -- 172.21.2.102:0/766371505 shutdown_connections 2021-02-10T11:38:31.388 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:31.370 7fe7e9bdd700 1 --2- 172.21.2.102:0/766371505 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe7e40d1870 0x7fe7e41b03f0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:31.388 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:31.370 7fe7e9bdd700 1 -- 172.21.2.102:0/766371505 >> 172.21.2.102:0/766371505 conn(0x7fe7e40ce450 msgr2=0x7fe7e40d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:31.388 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:31.370 7fe7e9bdd700 1 -- 172.21.2.102:0/766371505 shutdown_connections 2021-02-10T11:38:31.388 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:31.370 7fe7e9bdd700 1 -- 172.21.2.102:0/766371505 wait complete. 2021-02-10T11:38:31.390 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:31.385 7fe7e9bdd700 1 Processor -- start 2021-02-10T11:38:31.390 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:31.385 7fe7e9bdd700 1 -- start start 2021-02-10T11:38:31.390 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:31.385 7fe7e9bdd700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe7e418e930 0x7fe7e40aa4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:31.390 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:31.385 7fe7e9bdd700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe7e40aaaf0 0x7fe7e4093700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:31.390 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:31.385 7fe7e9bdd700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe7e418ed30 0x7fe7e4093d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:31.390 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:31.385 7fe7e9bdd700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7e40944e0 con 0x7fe7e40aaaf0 2021-02-10T11:38:31.390 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:31.385 7fe7e9bdd700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7e408a590 con 0x7fe7e418e930 2021-02-10T11:38:31.390 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:31.385 7fe7e9bdd700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7e408a770 con 0x7fe7e418ed30 2021-02-10T11:38:31.390 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:31.386 7fe7e2ffd700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe7e40aaaf0 0x7fe7e4093700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:31.390 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:31.386 7fe7e2ffd700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe7e40aaaf0 0x7fe7e4093700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:39582/0 (socket says 172.21.2.102:39582) 2021-02-10T11:38:31.391 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:31.386 7fe7e2ffd700 1 -- 172.21.2.102:0/3007688930 learned_addr learned my addr 172.21.2.102:0/3007688930 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:38:31.391 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:31.386 7fe7e37fe700 1 --2- 172.21.2.102:0/3007688930 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe7e418e930 0x7fe7e40aa4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:31.391 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:31.386 7fe7e3fff700 1 --2- 172.21.2.102:0/3007688930 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe7e418ed30 0x7fe7e4093d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:31.391 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:31.386 7fe7e2ffd700 1 -- 172.21.2.102:0/3007688930 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe7e418ed30 msgr2=0x7fe7e4093d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:31.391 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:31.386 7fe7e2ffd700 1 --2- 172.21.2.102:0/3007688930 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe7e418ed30 0x7fe7e4093d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:31.391 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:31.386 7fe7e2ffd700 1 -- 172.21.2.102:0/3007688930 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe7e418e930 msgr2=0x7fe7e40aa4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:31.391 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:31.386 7fe7e2ffd700 1 --2- 172.21.2.102:0/3007688930 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe7e418e930 0x7fe7e40aa4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:31.391 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:31.386 7fe7e2ffd700 1 -- 172.21.2.102:0/3007688930 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7e408a990 con 0x7fe7e40aaaf0 2021-02-10T11:38:31.391 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:31.386 7fe7e37fe700 1 --2- 172.21.2.102:0/3007688930 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe7e418e930 0x7fe7e40aa4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:38:31.391 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:31.386 7fe7e2ffd700 1 --2- 172.21.2.102:0/3007688930 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe7e40aaaf0 0x7fe7e4093700 secure :-1 s=READY pgs=3101 cs=0 l=1 rev1=1 rx=0x7fe7cc001c60 tx=0x7fe7cc00a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:31.391 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:31.386 7fe7dbfff700 1 -- 172.21.2.102:0/3007688930 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe7cc00acc0 con 0x7fe7e40aaaf0 2021-02-10T11:38:31.391 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:31.386 7fe7dbfff700 1 -- 172.21.2.102:0/3007688930 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7cc004140 con 0x7fe7e40aaaf0 2021-02-10T11:38:31.392 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:31.386 7fe7dbfff700 1 -- 172.21.2.102:0/3007688930 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe7cc004620 con 0x7fe7e40aaaf0 2021-02-10T11:38:31.392 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:31.386 7fe7e9bdd700 1 -- 172.21.2.102:0/3007688930 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7e408a990 con 0x7fe7e40aaaf0 2021-02-10T11:38:31.392 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:31.386 7fe7e9bdd700 1 -- 172.21.2.102:0/3007688930 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7e408a990 con 0x7fe7e40aaaf0 2021-02-10T11:38:31.392 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:31.387 7fe7dbfff700 1 -- 172.21.2.102:0/3007688930 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fe7cc007f30 con 0x7fe7e40aaaf0 2021-02-10T11:38:31.392 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:31.387 7fe7dbfff700 1 --2- 172.21.2.102:0/3007688930 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe7c8018700 0x7fe7c801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:31.392 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:31.387 7fe7dbfff700 1 -- 172.21.2.102:0/3007688930 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(180..180 src has 1..180) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fe7cc032620 con 0x7fe7e40aaaf0 2021-02-10T11:38:31.392 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:31.387 7fe7e37fe700 1 --2- 172.21.2.102:0/3007688930 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe7c8018700 0x7fe7c801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:31.392 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:31.387 7fe7e37fe700 1 --2- 172.21.2.102:0/3007688930 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe7c8018700 0x7fe7c801abc0 crc :-1 s=READY pgs=1360 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:31.419 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:31.418 7fe7e9bdd700 1 -- 172.21.2.102:0/3007688930 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7e408a530 con 0x7fe7e40aaaf0 2021-02-10T11:38:31.421 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:31.420 7fe7dbfff700 1 -- 172.21.2.102:0/3007688930 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fe7cc000e20 con 0x7fe7e40aaaf0 2021-02-10T11:38:31.548 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:31.546 7fe7e9bdd700 1 -- 172.21.2.102:0/3007688930 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe7e408a530 con 0x7fe7e40aaaf0 2021-02-10T11:38:31.548 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:31.547 7fe7dbfff700 1 -- 172.21.2.102:0/3007688930 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v180) v1 ==== 74+0+14007 (secure 0 0 0) 0x7fe7cc007f30 con 0x7fe7e40aaaf0 2021-02-10T11:38:31.549 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:38:31.549 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":180,"fsid":"68db30ea-dbce-4aef-a6df-f01c6fea0513","created":"2021-02-10 11:03:06.331557","modified":"2021-02-10 11:38:31.337750","last_up_change":"2021-02-10 11:03:32.690562","last_in_change":"2021-02-10 11:03:32.690562","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":39,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:03:37.322519","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":38,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:38:28.298788","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"179","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":39,"pool_name":"cephfs_data","create_time":"2021-02-10 11:38:29.326316","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"180","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"be691b4d-a039-408a-af80-9d53f02b6f1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":178,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6818","nonce":36088},{"type":"v1","addr":"172.21.2.102:6820","nonce":36088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6822","nonce":36088},{"type":"v1","addr":"172.21.2.102:6824","nonce":36088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6830","nonce":36088},{"type":"v1","addr":"172.21.2.102:6832","nonce":36088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6826","nonce":36088},{"type":"v1","addr":"172.21.2.102:6828","nonce":36088}]},"public_addr":"172.21.2.102:6820/36088","cluster_addr":"172.21.2.102:6824/36088","heartbeat_back_addr":"172.21.2.102:6832/36088","heartbeat_front_addr":"172.21.2.102:6828/36088","state":["exists","up"]},{"osd":1,"uuid":"64e3ccc8-9c69-4d7c-865e-af8e83822635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":178,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6819","nonce":36087},{"type":"v1","addr":"172.21.2.102:6821","nonce":36087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6823","nonce":36087},{"type":"v1","addr":"172.21.2.102:6825","nonce":36087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6831","nonce":36087},{"type":"v1","addr":"172.21.2.102:6833","nonce":36087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6827","nonce":36087},{"type":"v1","addr":"172.21.2.102:6829","nonce":36087}]},"public_addr":"172.21.2.102:6821/36087","cluster_addr":"172.21.2.102:6825/36087","heartbeat_back_addr":"172.21.2.102:6833/36087","heartbeat_front_addr":"172.21.2.102:6829/36087","state":["exists","up"]},{"osd":2,"uuid":"153700af-747f-481f-bdf4-d6072d3a842f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":178,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6810","nonce":36089},{"type":"v1","addr":"172.21.2.102:6811","nonce":36089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6812","nonce":36089},{"type":"v1","addr":"172.21.2.102:6813","nonce":36089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6816","nonce":36089},{"type":"v1","addr":"172.21.2.102:6817","nonce":36089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6814","nonce":36089},{"type":"v1","addr":"172.21.2.102:6815","nonce":36089}]},"public_addr":"172.21.2.102:6811/36089","cluster_addr":"172.21.2.102:6813/36089","heartbeat_back_addr":"172.21.2.102:6817/36089","heartbeat_front_addr":"172.21.2.102:6815/36089","state":["exists","up"]},{"osd":3,"uuid":"cbc678f4-83a9-45e6-8830-2f7c232b1ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":178,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6802","nonce":36090},{"type":"v1","addr":"172.21.2.102:6803","nonce":36090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6804","nonce":36090},{"type":"v1","addr":"172.21.2.102:6805","nonce":36090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6808","nonce":36090},{"type":"v1","addr":"172.21.2.102:6809","nonce":36090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6806","nonce":36090},{"type":"v1","addr":"172.21.2.102:6807","nonce":36090}]},"public_addr":"172.21.2.102:6803/36090","cluster_addr":"172.21.2.102:6805/36090","heartbeat_back_addr":"172.21.2.102:6809/36090","heartbeat_front_addr":"172.21.2.102:6807/36090","state":["exists","up"]},{"osd":4,"uuid":"c01e4008-b258-4f10-afd2-9e7f37214351","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":178,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6808","nonce":36368},{"type":"v1","addr":"172.21.2.113:6811","nonce":36368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6814","nonce":36368},{"type":"v1","addr":"172.21.2.113:6818","nonce":36368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6826","nonce":36368},{"type":"v1","addr":"172.21.2.113:6828","nonce":36368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6820","nonce":36368},{"type":"v1","addr":"172.21.2.113:6824","nonce":36368}]},"public_addr":"172.21.2.113:6811/36368","cluster_addr":"172.21.2.113:6818/36368","heartbeat_back_addr":"172.21.2.113:6828/36368","heartbeat_front_addr":"172.21.2.113:6824/36368","state":["exists","up"]},{"osd":5,"uuid":"551f6007-daf8-4a98-8ee0-c2b16bcccb5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":178,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6802","nonce":36366},{"type":"v1","addr":"172.21.2.113:6805","nonce":36366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6807","nonce":36366},{"type":"v1","addr":"172.21.2.113:6810","nonce":36366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6819","nonce":36366},{"type":"v1","addr":"172.21.2.113:6822","nonce":36366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6812","nonce":36366},{"type":"v1","addr":"172.21.2.113:6815","nonce":36366}]},"public_addr":"172.21.2.113:6805/36366","cluster_addr":"172.21.2.113:6810/36366","heartbeat_back_addr":"172.21.2.113:6822/36366","heartbeat_front_addr":"172.21.2.113:6815/36366","state":["exists","up"]},{"osd":6,"uuid":"bf67c040-0e50-48be-b102-c3943bc9ebef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6800","nonce":36367},{"type":"v1","addr":"172.21.2.113:6801","nonce":36367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6803","nonce":36367},{"type":"v1","addr":"172.21.2.113:6804","nonce":36367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6813","nonce":36367},{"type":"v1","addr":"172.21.2.113:6816","nonce":36367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6806","nonce":36367},{"type":"v1","addr":"172.21.2.113:6809","nonce":36367}]},"public_addr":"172.21.2.113:6801/36367","cluster_addr":"172.21.2.113:6804/36367","heartbeat_back_addr":"172.21.2.113:6816/36367","heartbeat_front_addr":"172.21.2.113:6809/36367","state":["exists","up"]},{"osd":7,"uuid":"f5461f80-018d-428f-be89-2eb2bcbe7bb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":178,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6817","nonce":36365},{"type":"v1","addr":"172.21.2.113:6821","nonce":36365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6823","nonce":36365},{"type":"v1","addr":"172.21.2.113:6825","nonce":36365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6830","nonce":36365},{"type":"v1","addr":"172.21.2.113:6831","nonce":36365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6827","nonce":36365},{"type":"v1","addr":"172.21.2.113:6829","nonce":36365}]},"public_addr":"172.21.2.113:6821/36365","cluster_addr":"172.21.2.113:6825/36365","heartbeat_back_addr":"172.21.2.113:6831/36365","heartbeat_front_addr":"172.21.2.113:6829/36365","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:38:31.550 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:31.549 7fe7e9bdd700 1 -- 172.21.2.102:0/3007688930 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe7c8018700 msgr2=0x7fe7c801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:31.550 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:31.549 7fe7e9bdd700 1 --2- 172.21.2.102:0/3007688930 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe7c8018700 0x7fe7c801abc0 crc :-1 s=READY pgs=1360 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:31.550 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:31.549 7fe7e9bdd700 1 -- 172.21.2.102:0/3007688930 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe7e40aaaf0 msgr2=0x7fe7e4093700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:31.550 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:31.549 7fe7e9bdd700 1 --2- 172.21.2.102:0/3007688930 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe7e40aaaf0 0x7fe7e4093700 secure :-1 s=READY pgs=3101 cs=0 l=1 rev1=1 rx=0x7fe7cc001c60 tx=0x7fe7cc00a8d0).stop 2021-02-10T11:38:31.550 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:31.549 7fe7e9bdd700 1 -- 172.21.2.102:0/3007688930 shutdown_connections 2021-02-10T11:38:31.550 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:31.549 7fe7e9bdd700 1 --2- 172.21.2.102:0/3007688930 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe7e418ed30 0x7fe7e4093d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:31.550 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:31.549 7fe7e9bdd700 1 --2- 172.21.2.102:0/3007688930 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe7e40aaaf0 0x7fe7e4093700 unknown :-1 s=CLOSED pgs=3101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:31.550 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:31.549 7fe7e9bdd700 1 --2- 172.21.2.102:0/3007688930 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe7c8018700 0x7fe7c801abc0 unknown :-1 s=CLOSED pgs=1360 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:31.551 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:31.549 7fe7e9bdd700 1 --2- 172.21.2.102:0/3007688930 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe7e418e930 0x7fe7e40aa4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:31.551 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:31.549 7fe7e9bdd700 1 -- 172.21.2.102:0/3007688930 >> 172.21.2.102:0/3007688930 conn(0x7fe7e419f8f0 msgr2=0x7fe7e418c260 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:31.551 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:31.549 7fe7e9bdd700 1 -- 172.21.2.102:0/3007688930 shutdown_connections 2021-02-10T11:38:31.551 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:31.549 7fe7e9bdd700 1 -- 172.21.2.102:0/3007688930 wait complete. 2021-02-10T11:38:31.559 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2021-02-10T11:38:31.994 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:31.975 7f3af30c4700 1 Processor -- start 2021-02-10T11:38:31.995 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:31.975 7f3af30c4700 1 -- start start 2021-02-10T11:38:31.995 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:31.976 7f3af30c4700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f3aec0d1870 0x7f3aec1b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:31.995 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:31.976 7f3af30c4700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3aec0fe1c0 con 0x7f3aec1b0ae0 2021-02-10T11:38:31.995 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:31.976 7f3af30c4700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3aec109a60 con 0x7f3aec0cff70 2021-02-10T11:38:31.995 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:31.976 7f3af30c4700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3aec088070 con 0x7f3aec0d1870 2021-02-10T11:38:31.995 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:31.976 7f3aebfff700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f3aec1b0ae0 0x7f3aec108940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:60044/0 (socket says 172.21.2.102:60044) 2021-02-10T11:38:31.995 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:31.976 7f3aebfff700 1 -- 172.21.2.102:0/3409212355 learned_addr learned my addr 172.21.2.102:0/3409212355 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:38:31.995 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:31.976 7f3aeb7fe700 1 -- 172.21.2.102:0/3409212355 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 634739190 0 0) 0x7f3adc001060 con 0x7f3aec1b0ae0 2021-02-10T11:38:31.995 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:31.976 7f3aeb7fe700 1 -- 172.21.2.102:0/3409212355 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ad8002fe0 con 0x7f3aec1b0ae0 2021-02-10T11:38:31.995 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:31.976 7f3af0ee0700 1 --2- 172.21.2.102:0/3409212355 >> v2:172.21.2.113:3300/0 conn(0x7f3aec0d1870 0x7f3aec1b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:31.995 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:31.977 7f3aeb7fe700 1 -- 172.21.2.102:0/3409212355 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4003102064 0 0) 0x7f3adc001460 con 0x7f3aec1b0ae0 2021-02-10T11:38:31.996 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:31.977 7f3aeb7fe700 1 -- 172.21.2.102:0/3409212355 >> v2:172.21.2.113:3300/0 conn(0x7f3aec0d1870 msgr2=0x7f3aec1b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:31.996 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:31.977 7f3aeb7fe700 1 --2- 172.21.2.102:0/3409212355 >> v2:172.21.2.113:3300/0 conn(0x7f3aec0d1870 0x7f3aec1b04c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:31.996 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:31.977 7f3aeb7fe700 1 -- 172.21.2.102:0/3409212355 >> v1:172.21.2.113:6789/0 conn(0x7f3aec0cff70 legacy=0x7f3aec109130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:31.996 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:31.977 7f3aeb7fe700 1 -- 172.21.2.102:0/3409212355 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3aec088260 con 0x7f3aec1b0ae0 2021-02-10T11:38:31.996 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:31.977 7f3aeb7fe700 1 -- 172.21.2.102:0/3409212355 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7f3adc0019e0 con 0x7f3aec1b0ae0 2021-02-10T11:38:31.996 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:31.977 7f3aeb7fe700 1 -- 172.21.2.102:0/3409212355 >> v1:172.21.2.102:6789/0 conn(0x7f3aec1b0ae0 legacy=0x7f3aec108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:31.996 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:31.977 7f3aeb7fe700 1 --2- 172.21.2.102:0/3409212355 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3ad8003a40 0x7f3ad8005f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:31.996 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:31.977 7f3aeb7fe700 1 --2- 172.21.2.102:0/3409212355 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3ad8006800 0x7f3ad8008d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:31.996 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:31.977 7f3aeb7fe700 1 --2- 172.21.2.102:0/3409212355 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3ad80095a0 0x7f3ad800bac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:31.996 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:31.977 7f3aeb7fe700 1 -- 172.21.2.102:0/3409212355 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ad800c310 con 0x7f3ad80095a0 2021-02-10T11:38:31.996 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:31.977 7f3aeb7fe700 1 -- 172.21.2.102:0/3409212355 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ad800c530 con 0x7f3ad8003a40 2021-02-10T11:38:31.996 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:31.977 7f3aeb7fe700 1 -- 172.21.2.102:0/3409212355 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ad800c750 con 0x7f3ad8006800 2021-02-10T11:38:31.997 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:31.977 7f3af16e1700 1 --2- 172.21.2.102:0/3409212355 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3ad80095a0 0x7f3ad800bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:31.997 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:31.977 7f3af0ee0700 1 --2- 172.21.2.102:0/3409212355 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3ad8003a40 0x7f3ad8005f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:31.997 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:31.977 7f3aebfff700 1 --2- 172.21.2.102:0/3409212355 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3ad8006800 0x7f3ad8008d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:31.997 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:31.977 7f3af16e1700 1 -- 172.21.2.102:0/3409212355 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3ad8006800 msgr2=0x7f3ad8008d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:31.997 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:31.977 7f3af16e1700 1 --2- 172.21.2.102:0/3409212355 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3ad8006800 0x7f3ad8008d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:31.997 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:31.977 7f3af16e1700 1 -- 172.21.2.102:0/3409212355 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3ad8003a40 msgr2=0x7f3ad8005f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:31.997 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:31.977 7f3af16e1700 1 --2- 172.21.2.102:0/3409212355 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3ad8003a40 0x7f3ad8005f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:31.997 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:31.977 7f3af16e1700 1 -- 172.21.2.102:0/3409212355 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ad800c9f0 con 0x7f3ad80095a0 2021-02-10T11:38:31.997 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:31.977 7f3af16e1700 1 -- 172.21.2.102:0/3409212355 reap_dead start 2021-02-10T11:38:31.997 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:31.978 7f3af16e1700 1 --2- 172.21.2.102:0/3409212355 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3ad80095a0 0x7f3ad800bac0 secure :-1 s=READY pgs=3103 cs=0 l=1 rev1=1 rx=0x7f3ae4006660 tx=0x7f3ae4009a90).ready entity=mon.0 client_cookie=476fca96dafce2e0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:31.997 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:31.978 7f3aeb7fe700 1 -- 172.21.2.102:0/3409212355 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3ae400c0c0 con 0x7f3ad80095a0 2021-02-10T11:38:31.997 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:31.978 7f3aeb7fe700 1 -- 172.21.2.102:0/3409212355 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ae4000e00 con 0x7f3ad80095a0 2021-02-10T11:38:31.998 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:31.978 7f3aeb7fe700 1 -- 172.21.2.102:0/3409212355 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3ae400c530 con 0x7f3ad80095a0 2021-02-10T11:38:31.998 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:31.978 7f3af30c4700 1 -- 172.21.2.102:0/3409212355 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3ad80095a0 msgr2=0x7f3ad800bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:31.998 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:31.978 7f3af30c4700 1 --2- 172.21.2.102:0/3409212355 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3ad80095a0 0x7f3ad800bac0 secure :-1 s=READY pgs=3103 cs=0 l=1 rev1=1 rx=0x7f3ae4006660 tx=0x7f3ae4009a90).stop 2021-02-10T11:38:31.998 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:31.978 7f3af30c4700 1 -- 172.21.2.102:0/3409212355 shutdown_connections 2021-02-10T11:38:31.998 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:31.978 7f3af30c4700 1 --2- 172.21.2.102:0/3409212355 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3ad80095a0 0x7f3ad800bac0 unknown :-1 s=CLOSED pgs=3103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:31.998 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:31.978 7f3af30c4700 1 -- 172.21.2.102:0/3409212355 >> 172.21.2.102:0/3409212355 conn(0x7f3aec0ce420 msgr2=0x7f3aec0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:31.998 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:31.978 7f3af30c4700 1 -- 172.21.2.102:0/3409212355 shutdown_connections 2021-02-10T11:38:31.998 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:31.978 7f3af30c4700 1 -- 172.21.2.102:0/3409212355 wait complete. 2021-02-10T11:38:31.998 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:31.993 7f3af30c4700 1 Processor -- start 2021-02-10T11:38:31.998 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:31.993 7f3af30c4700 1 -- start start 2021-02-10T11:38:32.000 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:31.994 7f3af30c4700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3aec09bcc0 0x7f3aec09c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:32.000 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:31.994 7f3af30c4700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3aec0a0820 0x7f3aec0a0c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:32.000 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:31.994 7f3af30c4700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3aec0a1240 0x7f3aec0858d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:32.000 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:31.994 7f3af30c4700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3aec0a66b0 con 0x7f3aec0a0820 2021-02-10T11:38:32.000 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:31.994 7f3af30c4700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3aec1a49b0 con 0x7f3aec0a1240 2021-02-10T11:38:32.000 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:31.994 7f3af30c4700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3aec1a4b90 con 0x7f3aec09bcc0 2021-02-10T11:38:32.000 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:31.994 7f3aebfff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3aec0a0820 0x7f3aec0a0c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:32.000 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:31.994 7f3aebfff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3aec0a0820 0x7f3aec0a0c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:39604/0 (socket says 172.21.2.102:39604) 2021-02-10T11:38:32.000 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:31.994 7f3aebfff700 1 -- 172.21.2.102:0/853476717 learned_addr learned my addr 172.21.2.102:0/853476717 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:38:32.000 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:31.994 7f3aebfff700 1 -- 172.21.2.102:0/853476717 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3aec09bcc0 msgr2=0x7f3aec09c0c0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:38:32.000 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:31.994 7f3aebfff700 1 --2- 172.21.2.102:0/853476717 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3aec09bcc0 0x7f3aec09c0c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:32.001 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:31.994 7f3aebfff700 1 -- 172.21.2.102:0/853476717 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3aec0a1240 msgr2=0x7f3aec0858d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:32.001 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:31.994 7f3aebfff700 1 --2- 172.21.2.102:0/853476717 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3aec0a1240 0x7f3aec0858d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:32.001 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:31.994 7f3aebfff700 1 -- 172.21.2.102:0/853476717 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3aec0a1640 con 0x7f3aec0a0820 2021-02-10T11:38:32.001 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:31.995 7f3aebfff700 1 --2- 172.21.2.102:0/853476717 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3aec0a0820 0x7f3aec0a0c20 secure :-1 s=READY pgs=3104 cs=0 l=1 rev1=1 rx=0x7f3adc0014f0 tx=0x7f3adc00ac10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:32.001 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:31.995 7f3ae97fa700 1 -- 172.21.2.102:0/853476717 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3adc0070d0 con 0x7f3aec0a0820 2021-02-10T11:38:32.001 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:31.995 7f3ae97fa700 1 -- 172.21.2.102:0/853476717 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3adc001150 con 0x7f3aec0a0820 2021-02-10T11:38:32.001 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:31.995 7f3ae97fa700 1 -- 172.21.2.102:0/853476717 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3adc007500 con 0x7f3aec0a0820 2021-02-10T11:38:32.001 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:31.995 7f3af30c4700 1 -- 172.21.2.102:0/853476717 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3aec0a1640 con 0x7f3aec0a0820 2021-02-10T11:38:32.001 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:31.995 7f3af30c4700 1 -- 172.21.2.102:0/853476717 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3aec0861e0 con 0x7f3aec0a0820 2021-02-10T11:38:32.001 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:31.995 7f3ae97fa700 1 -- 172.21.2.102:0/853476717 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f3adc004720 con 0x7f3aec0a0820 2021-02-10T11:38:32.001 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:31.995 7f3ae97fa700 1 --2- 172.21.2.102:0/853476717 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3ad00186c0 0x7f3ad001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:32.001 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:31.995 7f3ae97fa700 1 -- 172.21.2.102:0/853476717 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(180..180 src has 1..180) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f3adc0328c0 con 0x7f3aec0a0820 2021-02-10T11:38:32.001 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:31.996 7f3af0ee0700 1 --2- 172.21.2.102:0/853476717 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3ad00186c0 0x7f3ad001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:32.002 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:31.996 7f3af0ee0700 1 --2- 172.21.2.102:0/853476717 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3ad00186c0 0x7f3ad001abc0 crc :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:32.027 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:32.026 7f3af30c4700 1 -- 172.21.2.102:0/853476717 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3aec086a70 con 0x7f3aec0a0820 2021-02-10T11:38:32.029 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:32.028 7f3ae97fa700 1 -- 172.21.2.102:0/853476717 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f3adc0133c0 con 0x7f3aec0a0820 2021-02-10T11:38:32.168 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:32.167 7f3af30c4700 1 -- 172.21.2.102:0/853476717 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"} v 0) v1 -- 0x7f3aec086a70 con 0x7f3aec0a0820 2021-02-10T11:38:32.347 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:32.346 7f3ae97fa700 1 -- 172.21.2.102:0/853476717 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"}]=0 v157) v1 ==== 118+0+0 (secure 0 0 0) 0x7f3adc0133c0 con 0x7f3aec0a0820 2021-02-10T11:38:32.386 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:32.384 7f3af30c4700 1 -- 172.21.2.102:0/853476717 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3ad00186c0 msgr2=0x7f3ad001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:32.386 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:32.384 7f3af30c4700 1 --2- 172.21.2.102:0/853476717 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3ad00186c0 0x7f3ad001abc0 crc :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:32.386 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:32.384 7f3af30c4700 1 -- 172.21.2.102:0/853476717 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3aec0a0820 msgr2=0x7f3aec0a0c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:32.386 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:32.384 7f3af30c4700 1 --2- 172.21.2.102:0/853476717 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3aec0a0820 0x7f3aec0a0c20 secure :-1 s=READY pgs=3104 cs=0 l=1 rev1=1 rx=0x7f3adc0014f0 tx=0x7f3adc00ac10).stop 2021-02-10T11:38:32.386 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:32.384 7f3af30c4700 1 -- 172.21.2.102:0/853476717 shutdown_connections 2021-02-10T11:38:32.387 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:32.384 7f3af30c4700 1 --2- 172.21.2.102:0/853476717 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3ad00186c0 0x7f3ad001abc0 unknown :-1 s=CLOSED pgs=1361 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:32.387 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:32.384 7f3af30c4700 1 --2- 172.21.2.102:0/853476717 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3aec0a1240 0x7f3aec0858d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:32.387 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:32.384 7f3af30c4700 1 --2- 172.21.2.102:0/853476717 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3aec0a0820 0x7f3aec0a0c20 unknown :-1 s=CLOSED pgs=3104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:32.387 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:32.384 7f3af30c4700 1 --2- 172.21.2.102:0/853476717 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3aec09bcc0 0x7f3aec09c0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:32.387 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:32.384 7f3af30c4700 1 -- 172.21.2.102:0/853476717 >> 172.21.2.102:0/853476717 conn(0x7f3aec0bef00 msgr2=0x7f3aec09d1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:32.387 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:32.384 7f3af30c4700 1 -- 172.21.2.102:0/853476717 shutdown_connections 2021-02-10T11:38:32.387 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:32.384 7f3af30c4700 1 -- 172.21.2.102:0/853476717 wait complete. 2021-02-10T11:38:32.394 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:38:32.823 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:32.805 7f15dc009700 1 Processor -- start 2021-02-10T11:38:32.824 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:32.806 7f15dc009700 1 -- start start 2021-02-10T11:38:32.824 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:32.806 7f15dc009700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f15d40d1870 0x7f15d41b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:32.824 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:32.806 7f15dc009700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f15d41b0a10 0x7f15d40cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:32.824 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:32.806 7f15dc009700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15d40f0b10 con 0x7f15d40d1c70 2021-02-10T11:38:32.824 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:32.806 7f15dc009700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15d40fe230 con 0x7f15d41b0a10 2021-02-10T11:38:32.825 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:32.806 7f15dc009700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15d40e5ed0 con 0x7f15d40d1870 2021-02-10T11:38:32.825 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:32.806 7f15d9624700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f15d41b0a10 0x7f15d40cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:32.825 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:32.806 7f15da626700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f15d40d1c70 0x7f15d40cf950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:60064/0 (socket says 172.21.2.102:60064) 2021-02-10T11:38:32.825 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:32.806 7f15da626700 1 -- 172.21.2.102:0/1194640718 learned_addr learned my addr 172.21.2.102:0/1194640718 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:38:32.825 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:32.807 7f15d9624700 1 -- 172.21.2.102:0/1194640718 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f15d40d1870 msgr2=0x7f15d41b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:32.825 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:32.807 7f15d9624700 1 --2- 172.21.2.102:0/1194640718 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f15d40d1870 0x7f15d41b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:32.826 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:32.807 7f15d9624700 1 -- 172.21.2.102:0/1194640718 >> v1:172.21.2.102:6789/0 conn(0x7f15d40d1c70 legacy=0x7f15d40cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:32.826 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:32.807 7f15d9624700 1 -- 172.21.2.102:0/1194640718 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15d40f0650 con 0x7f15d41b0a10 2021-02-10T11:38:32.826 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:32.807 7f15d8e23700 1 -- 172.21.2.102:0/1194640718 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 455204236 0 0) 0x7f15cc001500 con 0x7f15d40d1c70 2021-02-10T11:38:32.826 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:32.807 7f15d9624700 1 --2- 172.21.2.102:0/1194640718 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f15d41b0a10 0x7f15d40cf330 secure :-1 s=READY pgs=3106 cs=0 l=1 rev1=1 rx=0x7f15c8006650 tx=0x7f15c8009a80).ready entity=mon.0 client_cookie=c0f4797d60d96b8b server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:32.826 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:32.807 7f15d8e23700 1 -- 172.21.2.102:0/1194640718 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f15c80009b0 con 0x7f15d41b0a10 2021-02-10T11:38:32.826 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:32.807 7f15d8e23700 1 -- 172.21.2.102:0/1194640718 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15c800c0c0 con 0x7f15d41b0a10 2021-02-10T11:38:32.827 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:32.807 7f15d8e23700 1 -- 172.21.2.102:0/1194640718 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f15c800c520 con 0x7f15d41b0a10 2021-02-10T11:38:32.827 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:32.807 7f15dc009700 1 -- 172.21.2.102:0/1194640718 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f15d41b0a10 msgr2=0x7f15d40cf330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:32.827 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:32.807 7f15dc009700 1 --2- 172.21.2.102:0/1194640718 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f15d41b0a10 0x7f15d40cf330 secure :-1 s=READY pgs=3106 cs=0 l=1 rev1=1 rx=0x7f15c8006650 tx=0x7f15c8009a80).stop 2021-02-10T11:38:32.827 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:32.808 7f15dc009700 1 -- 172.21.2.102:0/1194640718 shutdown_connections 2021-02-10T11:38:32.827 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:32.808 7f15dc009700 1 --2- 172.21.2.102:0/1194640718 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f15d41b0a10 0x7f15d40cf330 unknown :-1 s=CLOSED pgs=3106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:32.827 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:32.808 7f15dc009700 1 --2- 172.21.2.102:0/1194640718 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f15d40d1870 0x7f15d41b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:32.827 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:32.808 7f15dc009700 1 -- 172.21.2.102:0/1194640718 >> 172.21.2.102:0/1194640718 conn(0x7f15d40ce450 msgr2=0x7f15d40d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:32.827 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:32.808 7f15dc009700 1 -- 172.21.2.102:0/1194640718 shutdown_connections 2021-02-10T11:38:32.828 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:32.808 7f15dc009700 1 -- 172.21.2.102:0/1194640718 wait complete. 2021-02-10T11:38:32.828 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:32.822 7f15dc009700 1 Processor -- start 2021-02-10T11:38:32.828 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:32.822 7f15dc009700 1 -- start start 2021-02-10T11:38:32.830 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:32.823 7f15dc009700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f15d41b8ad0 0x7f15d4187e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:32.830 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:32.823 7f15dc009700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f15d41b8ed0 0x7f15d40b1af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:32.830 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:32.823 7f15dc009700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f15d40b2110 0x7f15d41b7a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:32.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:32.823 7f15dc009700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15d4188470 con 0x7f15d41b8ad0 2021-02-10T11:38:32.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:32.823 7f15dc009700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15d418be10 con 0x7f15d41b8ed0 2021-02-10T11:38:32.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:32.823 7f15dc009700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15d418c030 con 0x7f15d40b2110 2021-02-10T11:38:32.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:32.823 7f15d9e25700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f15d41b8ad0 0x7f15d4187e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:32.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:32.823 7f15d9e25700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f15d41b8ad0 0x7f15d4187e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:39614/0 (socket says 172.21.2.102:39614) 2021-02-10T11:38:32.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:32.823 7f15d9e25700 1 -- 172.21.2.102:0/2368623388 learned_addr learned my addr 172.21.2.102:0/2368623388 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:38:32.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:32.823 7f15d9e25700 1 -- 172.21.2.102:0/2368623388 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f15d40b2110 msgr2=0x7f15d41b7a00 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:38:32.832 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:32.823 7f15d9e25700 1 --2- 172.21.2.102:0/2368623388 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f15d40b2110 0x7f15d41b7a00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:32.832 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:32.823 7f15d9e25700 1 -- 172.21.2.102:0/2368623388 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f15d41b8ed0 msgr2=0x7f15d40b1af0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:38:32.832 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:32.823 7f15d9e25700 1 --2- 172.21.2.102:0/2368623388 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f15d41b8ed0 0x7f15d40b1af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:32.832 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:32.823 7f15d9e25700 1 -- 172.21.2.102:0/2368623388 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15d40b2510 con 0x7f15d41b8ad0 2021-02-10T11:38:32.832 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:32.823 7f15d9e25700 1 --2- 172.21.2.102:0/2368623388 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f15d41b8ad0 0x7f15d4187e50 secure :-1 s=READY pgs=3107 cs=0 l=1 rev1=1 rx=0x7f15d0006f00 tx=0x7f15d000b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:32.832 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:32.823 7f15c67fc700 1 -- 172.21.2.102:0/2368623388 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f15d0006b90 con 0x7f15d41b8ad0 2021-02-10T11:38:32.832 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:32.823 7f15c67fc700 1 -- 172.21.2.102:0/2368623388 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15d000bd30 con 0x7f15d41b8ad0 2021-02-10T11:38:32.833 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:32.823 7f15dc009700 1 -- 172.21.2.102:0/2368623388 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15d40b2510 con 0x7f15d41b8ad0 2021-02-10T11:38:32.833 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:32.824 7f15c67fc700 1 -- 172.21.2.102:0/2368623388 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f15d00020d0 con 0x7f15d41b8ad0 2021-02-10T11:38:32.833 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:32.824 7f15dc009700 1 -- 172.21.2.102:0/2368623388 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15d40b2510 con 0x7f15d41b8ad0 2021-02-10T11:38:32.833 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:32.824 7f15c67fc700 1 -- 172.21.2.102:0/2368623388 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f15d0002240 con 0x7f15d41b8ad0 2021-02-10T11:38:32.833 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:32.824 7f15c67fc700 1 --2- 172.21.2.102:0/2368623388 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f15bc018730 0x7f15bc01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:32.833 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:32.824 7f15c67fc700 1 -- 172.21.2.102:0/2368623388 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(180..180 src has 1..180) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f15d00325f0 con 0x7f15d41b8ad0 2021-02-10T11:38:32.833 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:32.824 7f15d9624700 1 --2- 172.21.2.102:0/2368623388 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f15bc018730 0x7f15bc01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:32.834 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:32.824 7f15d9624700 1 --2- 172.21.2.102:0/2368623388 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f15bc018730 0x7f15bc01ac10 crc :-1 s=READY pgs=1362 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:32.857 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:32.856 7f15dc009700 1 -- 172.21.2.102:0/2368623388 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15d40ebc00 con 0x7f15d41b8ad0 2021-02-10T11:38:32.859 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:32.858 7f15c67fc700 1 -- 172.21.2.102:0/2368623388 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f15d0002030 con 0x7f15d41b8ad0 2021-02-10T11:38:32.989 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:32.988 7f15dc009700 1 -- 172.21.2.102:0/2368623388 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f15d40ebc00 con 0x7f15d41b8ad0 2021-02-10T11:38:32.990 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:32.988 7f15c67fc700 1 -- 172.21.2.102:0/2368623388 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 157 v157) v1 ==== 95+0+2107 (secure 0 0 0) 0x7f15d000bd30 con 0x7f15d41b8ad0 2021-02-10T11:38:32.991 INFO:teuthology.orchestra.run.gibba002.stderr:dumped fsmap epoch 157 2021-02-10T11:38:32.991 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:38:32.991 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":157,"default_fscid":19,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8815,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.102:6835/3396946217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6834","nonce":3396946217},{"type":"v1","addr":"172.21.2.102:6835","nonce":3396946217}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":153}],"filesystems":[{"mdsmap":{"epoch":157,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:38:30.335525","modified":"2021-02-10 11:38:32.341928","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8812},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8812":{"gid":8812,"name":"b","rank":0,"incarnation":154,"state":"up:active","state_seq":3,"addr":"172.21.2.113:6833/1513291768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6832","nonce":1513291768},{"type":"v1","addr":"172.21.2.113:6833","nonce":1513291768}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[39],"metadata_pool":38,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":19}]} 2021-02-10T11:38:32.993 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:32.990 7f15dc009700 1 -- 172.21.2.102:0/2368623388 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f15bc018730 msgr2=0x7f15bc01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:32.993 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:32.990 7f15dc009700 1 --2- 172.21.2.102:0/2368623388 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f15bc018730 0x7f15bc01ac10 crc :-1 s=READY pgs=1362 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:32.993 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:32.990 7f15dc009700 1 -- 172.21.2.102:0/2368623388 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f15d41b8ad0 msgr2=0x7f15d4187e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:32.993 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:32.990 7f15dc009700 1 --2- 172.21.2.102:0/2368623388 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f15d41b8ad0 0x7f15d4187e50 secure :-1 s=READY pgs=3107 cs=0 l=1 rev1=1 rx=0x7f15d0006f00 tx=0x7f15d000b940).stop 2021-02-10T11:38:32.993 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:32.991 7f15dc009700 1 -- 172.21.2.102:0/2368623388 shutdown_connections 2021-02-10T11:38:32.994 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:32.991 7f15dc009700 1 --2- 172.21.2.102:0/2368623388 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f15d40b2110 0x7f15d41b7a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:32.994 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:32.991 7f15dc009700 1 --2- 172.21.2.102:0/2368623388 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f15bc018730 0x7f15bc01ac10 unknown :-1 s=CLOSED pgs=1362 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:32.994 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:32.991 7f15dc009700 1 --2- 172.21.2.102:0/2368623388 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f15d41b8ed0 0x7f15d40b1af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:32.994 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:32.991 7f15dc009700 1 --2- 172.21.2.102:0/2368623388 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f15d41b8ad0 0x7f15d4187e50 unknown :-1 s=CLOSED pgs=3107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:32.994 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:32.991 7f15dc009700 1 -- 172.21.2.102:0/2368623388 >> 172.21.2.102:0/2368623388 conn(0x7f15d40ed810 msgr2=0x7f15d4094980 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:32.994 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:32.991 7f15dc009700 1 -- 172.21.2.102:0/2368623388 shutdown_connections 2021-02-10T11:38:32.995 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:32.991 7f15dc009700 1 -- 172.21.2.102:0/2368623388 wait complete. 2021-02-10T11:38:33.001 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:38:33.432 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:33.413 7fe8a1696700 1 Processor -- start 2021-02-10T11:38:33.432 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:33.413 7fe8a1696700 1 -- start start 2021-02-10T11:38:33.432 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:33.414 7fe8a1696700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe89c0d1870 0x7fe89c1b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:33.432 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:33.414 7fe8a1696700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fe89c0f0650 0x7fe89c0cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:33.432 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:33.414 7fe8a1696700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe89c0f0b10 con 0x7fe89c1b0a10 2021-02-10T11:38:33.432 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:33.414 7fe8a1696700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe89c0fe230 con 0x7fe89c0f0650 2021-02-10T11:38:33.433 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:33.414 7fe8a1696700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe89c0e5ed0 con 0x7fe89c0d1870 2021-02-10T11:38:33.433 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:33.415 7fe89bac0700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fe89c0f0650 0x7fe89c0cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:33.434 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:33.415 7fe89b2bf700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe89c0d1870 0x7fe89c1b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:33.434 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:33.415 7fe89aabe700 1 --1- >> v1:172.21.2.113:6789/0 conn(0x7fe89c1b0a10 0x7fe89c0cf330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.113:6789/0 says I am v1:172.21.2.102:39168/0 (socket says 172.21.2.102:39168) 2021-02-10T11:38:33.434 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:33.415 7fe89aabe700 1 -- 172.21.2.102:0/2699630317 learned_addr learned my addr 172.21.2.102:0/2699630317 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:38:33.434 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:33.415 7fe89bac0700 1 -- 172.21.2.102:0/2699630317 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe89c0d1870 msgr2=0x7fe89c1b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:33.434 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:33.415 7fe89bac0700 1 --2- 172.21.2.102:0/2699630317 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe89c0d1870 0x7fe89c1b03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:33.434 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:33.415 7fe89bac0700 1 -- 172.21.2.102:0/2699630317 >> v1:172.21.2.113:6789/0 conn(0x7fe89c1b0a10 legacy=0x7fe89c0cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:33.434 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:33.415 7fe89bac0700 1 -- 172.21.2.102:0/2699630317 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe89c0f20a0 con 0x7fe89c0f0650 2021-02-10T11:38:33.434 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:33.415 7fe89bac0700 1 --2- 172.21.2.102:0/2699630317 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe89c0f0650 0x7fe89c0cfb20 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 rx=0x7fe894006e50 tx=0x7fe894009e70).ready entity=mon.1 client_cookie=e6cd97e36c2b1125 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:33.434 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:33.416 7fe89a2bd700 1 -- 172.21.2.102:0/2699630317 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe89400b510 con 0x7fe89c0f0650 2021-02-10T11:38:33.434 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:33.416 7fe89a2bd700 1 -- 172.21.2.102:0/2699630317 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe894006790 con 0x7fe89c0f0650 2021-02-10T11:38:33.435 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:33.416 7fe89a2bd700 1 -- 172.21.2.102:0/2699630317 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe89400b940 con 0x7fe89c0f0650 2021-02-10T11:38:33.435 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:33.416 7fe8a1696700 1 -- 172.21.2.102:0/2699630317 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe89c0f0650 msgr2=0x7fe89c0cfb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:33.435 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:33.416 7fe8a1696700 1 --2- 172.21.2.102:0/2699630317 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe89c0f0650 0x7fe89c0cfb20 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 rx=0x7fe894006e50 tx=0x7fe894009e70).stop 2021-02-10T11:38:33.435 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:33.416 7fe8a1696700 1 -- 172.21.2.102:0/2699630317 shutdown_connections 2021-02-10T11:38:33.435 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:33.416 7fe8a1696700 1 --2- 172.21.2.102:0/2699630317 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe89c0f0650 0x7fe89c0cfb20 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:33.435 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:33.416 7fe8a1696700 1 --2- 172.21.2.102:0/2699630317 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe89c0d1870 0x7fe89c1b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:33.435 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:33.417 7fe8a1696700 1 -- 172.21.2.102:0/2699630317 >> 172.21.2.102:0/2699630317 conn(0x7fe89c0ce450 msgr2=0x7fe89c0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:33.435 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:33.417 7fe8a1696700 1 -- 172.21.2.102:0/2699630317 shutdown_connections 2021-02-10T11:38:33.435 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:33.417 7fe8a1696700 1 -- 172.21.2.102:0/2699630317 wait complete. 2021-02-10T11:38:33.437 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:33.431 7fe8a1696700 1 Processor -- start 2021-02-10T11:38:33.437 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:33.431 7fe8a1696700 1 -- start start 2021-02-10T11:38:33.437 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:33.431 7fe8a1696700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe89c0ada90 0x7fe89c0ade90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:33.437 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:33.431 7fe8a1696700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe89c099d20 0x7fe89c09a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:33.437 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:33.431 7fe8a1696700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe89c09a740 0x7fe89c097bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:33.438 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:33.431 7fe8a1696700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe89c0997a0 con 0x7fe89c099d20 2021-02-10T11:38:33.438 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:33.431 7fe8a1696700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe89c18f0c0 con 0x7fe89c09a740 2021-02-10T11:38:33.438 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:33.431 7fe8a1696700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe89c09ab40 con 0x7fe89c0ada90 2021-02-10T11:38:33.438 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:33.432 7fe89aabe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe89c099d20 0x7fe89c09a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:33.438 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:33.432 7fe89aabe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe89c099d20 0x7fe89c09a120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:39628/0 (socket says 172.21.2.102:39628) 2021-02-10T11:38:33.438 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:33.432 7fe89aabe700 1 -- 172.21.2.102:0/1945169593 learned_addr learned my addr 172.21.2.102:0/1945169593 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:38:33.438 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:33.432 7fe89aabe700 1 -- 172.21.2.102:0/1945169593 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe89c0ada90 msgr2=0x7fe89c0ade90 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:38:33.438 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:33.432 7fe89aabe700 1 --2- 172.21.2.102:0/1945169593 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe89c0ada90 0x7fe89c0ade90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:33.438 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:33.432 7fe89aabe700 1 -- 172.21.2.102:0/1945169593 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe89c09a740 msgr2=0x7fe89c097bc0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:38:33.439 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:33.432 7fe89aabe700 1 --2- 172.21.2.102:0/1945169593 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe89c09a740 0x7fe89c097bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:33.439 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:33.432 7fe89aabe700 1 -- 172.21.2.102:0/1945169593 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe89c09ad60 con 0x7fe89c099d20 2021-02-10T11:38:33.439 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:33.432 7fe89aabe700 1 --2- 172.21.2.102:0/1945169593 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe89c099d20 0x7fe89c09a120 secure :-1 s=READY pgs=3108 cs=0 l=1 rev1=1 rx=0x7fe884001c60 tx=0x7fe88400a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:33.439 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:33.432 7fe893fff700 1 -- 172.21.2.102:0/1945169593 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe88400acc0 con 0x7fe89c099d20 2021-02-10T11:38:33.439 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:33.432 7fe893fff700 1 -- 172.21.2.102:0/1945169593 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe884004140 con 0x7fe89c099d20 2021-02-10T11:38:33.439 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:33.432 7fe8a1696700 1 -- 172.21.2.102:0/1945169593 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe89c09ad60 con 0x7fe89c099d20 2021-02-10T11:38:33.439 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:33.432 7fe893fff700 1 -- 172.21.2.102:0/1945169593 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe8840045a0 con 0x7fe89c099d20 2021-02-10T11:38:33.439 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:33.432 7fe8a1696700 1 -- 172.21.2.102:0/1945169593 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe89c09ad60 con 0x7fe89c099d20 2021-02-10T11:38:33.439 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:33.433 7fe893fff700 1 -- 172.21.2.102:0/1945169593 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fe884000ce0 con 0x7fe89c099d20 2021-02-10T11:38:33.439 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:33.433 7fe893fff700 1 --2- 172.21.2.102:0/1945169593 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe880018730 0x7fe88001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:33.439 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:33.433 7fe893fff700 1 -- 172.21.2.102:0/1945169593 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(180..180 src has 1..180) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fe884032640 con 0x7fe89c099d20 2021-02-10T11:38:33.439 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:33.433 7fe89b2bf700 1 --2- 172.21.2.102:0/1945169593 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe880018730 0x7fe88001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:33.440 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:33.433 7fe89b2bf700 1 --2- 172.21.2.102:0/1945169593 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe880018730 0x7fe88001ac10 crc :-1 s=READY pgs=1363 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:33.465 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:33.464 7fe8a1696700 1 -- 172.21.2.102:0/1945169593 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe89c0ebc00 con 0x7fe89c099d20 2021-02-10T11:38:33.467 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:33.466 7fe893fff700 1 -- 172.21.2.102:0/1945169593 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fe8840009b0 con 0x7fe89c099d20 2021-02-10T11:38:33.592 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:33.590 7fe8a1696700 1 -- 172.21.2.102:0/1945169593 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe89c0ebc00 con 0x7fe89c099d20 2021-02-10T11:38:33.592 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:33.591 7fe893fff700 1 -- 172.21.2.102:0/1945169593 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v180) v1 ==== 74+0+14007 (secure 0 0 0) 0x7fe884000ce0 con 0x7fe89c099d20 2021-02-10T11:38:33.593 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:38:33.593 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":180,"fsid":"68db30ea-dbce-4aef-a6df-f01c6fea0513","created":"2021-02-10 11:03:06.331557","modified":"2021-02-10 11:38:31.337750","last_up_change":"2021-02-10 11:03:32.690562","last_in_change":"2021-02-10 11:03:32.690562","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":39,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:03:37.322519","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":38,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:38:28.298788","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"179","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":39,"pool_name":"cephfs_data","create_time":"2021-02-10 11:38:29.326316","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"180","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"be691b4d-a039-408a-af80-9d53f02b6f1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":178,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6818","nonce":36088},{"type":"v1","addr":"172.21.2.102:6820","nonce":36088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6822","nonce":36088},{"type":"v1","addr":"172.21.2.102:6824","nonce":36088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6830","nonce":36088},{"type":"v1","addr":"172.21.2.102:6832","nonce":36088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6826","nonce":36088},{"type":"v1","addr":"172.21.2.102:6828","nonce":36088}]},"public_addr":"172.21.2.102:6820/36088","cluster_addr":"172.21.2.102:6824/36088","heartbeat_back_addr":"172.21.2.102:6832/36088","heartbeat_front_addr":"172.21.2.102:6828/36088","state":["exists","up"]},{"osd":1,"uuid":"64e3ccc8-9c69-4d7c-865e-af8e83822635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":178,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6819","nonce":36087},{"type":"v1","addr":"172.21.2.102:6821","nonce":36087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6823","nonce":36087},{"type":"v1","addr":"172.21.2.102:6825","nonce":36087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6831","nonce":36087},{"type":"v1","addr":"172.21.2.102:6833","nonce":36087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6827","nonce":36087},{"type":"v1","addr":"172.21.2.102:6829","nonce":36087}]},"public_addr":"172.21.2.102:6821/36087","cluster_addr":"172.21.2.102:6825/36087","heartbeat_back_addr":"172.21.2.102:6833/36087","heartbeat_front_addr":"172.21.2.102:6829/36087","state":["exists","up"]},{"osd":2,"uuid":"153700af-747f-481f-bdf4-d6072d3a842f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":178,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6810","nonce":36089},{"type":"v1","addr":"172.21.2.102:6811","nonce":36089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6812","nonce":36089},{"type":"v1","addr":"172.21.2.102:6813","nonce":36089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6816","nonce":36089},{"type":"v1","addr":"172.21.2.102:6817","nonce":36089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6814","nonce":36089},{"type":"v1","addr":"172.21.2.102:6815","nonce":36089}]},"public_addr":"172.21.2.102:6811/36089","cluster_addr":"172.21.2.102:6813/36089","heartbeat_back_addr":"172.21.2.102:6817/36089","heartbeat_front_addr":"172.21.2.102:6815/36089","state":["exists","up"]},{"osd":3,"uuid":"cbc678f4-83a9-45e6-8830-2f7c232b1ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":178,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6802","nonce":36090},{"type":"v1","addr":"172.21.2.102:6803","nonce":36090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6804","nonce":36090},{"type":"v1","addr":"172.21.2.102:6805","nonce":36090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6808","nonce":36090},{"type":"v1","addr":"172.21.2.102:6809","nonce":36090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6806","nonce":36090},{"type":"v1","addr":"172.21.2.102:6807","nonce":36090}]},"public_addr":"172.21.2.102:6803/36090","cluster_addr":"172.21.2.102:6805/36090","heartbeat_back_addr":"172.21.2.102:6809/36090","heartbeat_front_addr":"172.21.2.102:6807/36090","state":["exists","up"]},{"osd":4,"uuid":"c01e4008-b258-4f10-afd2-9e7f37214351","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":178,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6808","nonce":36368},{"type":"v1","addr":"172.21.2.113:6811","nonce":36368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6814","nonce":36368},{"type":"v1","addr":"172.21.2.113:6818","nonce":36368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6826","nonce":36368},{"type":"v1","addr":"172.21.2.113:6828","nonce":36368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6820","nonce":36368},{"type":"v1","addr":"172.21.2.113:6824","nonce":36368}]},"public_addr":"172.21.2.113:6811/36368","cluster_addr":"172.21.2.113:6818/36368","heartbeat_back_addr":"172.21.2.113:6828/36368","heartbeat_front_addr":"172.21.2.113:6824/36368","state":["exists","up"]},{"osd":5,"uuid":"551f6007-daf8-4a98-8ee0-c2b16bcccb5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":178,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6802","nonce":36366},{"type":"v1","addr":"172.21.2.113:6805","nonce":36366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6807","nonce":36366},{"type":"v1","addr":"172.21.2.113:6810","nonce":36366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6819","nonce":36366},{"type":"v1","addr":"172.21.2.113:6822","nonce":36366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6812","nonce":36366},{"type":"v1","addr":"172.21.2.113:6815","nonce":36366}]},"public_addr":"172.21.2.113:6805/36366","cluster_addr":"172.21.2.113:6810/36366","heartbeat_back_addr":"172.21.2.113:6822/36366","heartbeat_front_addr":"172.21.2.113:6815/36366","state":["exists","up"]},{"osd":6,"uuid":"bf67c040-0e50-48be-b102-c3943bc9ebef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6800","nonce":36367},{"type":"v1","addr":"172.21.2.113:6801","nonce":36367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6803","nonce":36367},{"type":"v1","addr":"172.21.2.113:6804","nonce":36367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6813","nonce":36367},{"type":"v1","addr":"172.21.2.113:6816","nonce":36367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6806","nonce":36367},{"type":"v1","addr":"172.21.2.113:6809","nonce":36367}]},"public_addr":"172.21.2.113:6801/36367","cluster_addr":"172.21.2.113:6804/36367","heartbeat_back_addr":"172.21.2.113:6816/36367","heartbeat_front_addr":"172.21.2.113:6809/36367","state":["exists","up"]},{"osd":7,"uuid":"f5461f80-018d-428f-be89-2eb2bcbe7bb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":178,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6817","nonce":36365},{"type":"v1","addr":"172.21.2.113:6821","nonce":36365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6823","nonce":36365},{"type":"v1","addr":"172.21.2.113:6825","nonce":36365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6830","nonce":36365},{"type":"v1","addr":"172.21.2.113:6831","nonce":36365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6827","nonce":36365},{"type":"v1","addr":"172.21.2.113:6829","nonce":36365}]},"public_addr":"172.21.2.113:6821/36365","cluster_addr":"172.21.2.113:6825/36365","heartbeat_back_addr":"172.21.2.113:6831/36365","heartbeat_front_addr":"172.21.2.113:6829/36365","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:38:33.594 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:33.593 7fe8a1696700 1 -- 172.21.2.102:0/1945169593 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe880018730 msgr2=0x7fe88001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:33.594 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:33.593 7fe8a1696700 1 --2- 172.21.2.102:0/1945169593 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe880018730 0x7fe88001ac10 crc :-1 s=READY pgs=1363 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:33.594 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:33.593 7fe8a1696700 1 -- 172.21.2.102:0/1945169593 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe89c099d20 msgr2=0x7fe89c09a120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:33.595 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:33.593 7fe8a1696700 1 --2- 172.21.2.102:0/1945169593 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe89c099d20 0x7fe89c09a120 secure :-1 s=READY pgs=3108 cs=0 l=1 rev1=1 rx=0x7fe884001c60 tx=0x7fe88400a8d0).stop 2021-02-10T11:38:33.595 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:33.593 7fe8a1696700 1 -- 172.21.2.102:0/1945169593 shutdown_connections 2021-02-10T11:38:33.595 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:33.593 7fe8a1696700 1 --2- 172.21.2.102:0/1945169593 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe880018730 0x7fe88001ac10 unknown :-1 s=CLOSED pgs=1363 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:33.595 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:33.593 7fe8a1696700 1 --2- 172.21.2.102:0/1945169593 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe89c09a740 0x7fe89c097bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:33.595 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:33.593 7fe8a1696700 1 --2- 172.21.2.102:0/1945169593 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe89c099d20 0x7fe89c09a120 unknown :-1 s=CLOSED pgs=3108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:33.595 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:33.593 7fe8a1696700 1 --2- 172.21.2.102:0/1945169593 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe89c0ada90 0x7fe89c0ade90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:33.595 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:33.593 7fe8a1696700 1 -- 172.21.2.102:0/1945169593 >> 172.21.2.102:0/1945169593 conn(0x7fe89c0f0550 msgr2=0x7fe89c191730 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:33.595 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:33.593 7fe8a1696700 1 -- 172.21.2.102:0/1945169593 shutdown_connections 2021-02-10T11:38:33.595 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:33.593 7fe8a1696700 1 -- 172.21.2.102:0/1945169593 wait complete. 2021-02-10T11:38:33.603 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:38:34.045 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:34.026 7fb6ca6be700 1 Processor -- start 2021-02-10T11:38:34.045 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:34.026 7fb6ca6be700 1 -- start start 2021-02-10T11:38:34.045 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:34.027 7fb6ca6be700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fb6c40d1880 0x7fb6c41b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:34.045 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:34.027 7fb6ca6be700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb6c41b0af0 0x7fb6c4108950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:34.045 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:34.027 7fb6ca6be700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6c40fe1d0 con 0x7fb6c40d1c80 2021-02-10T11:38:34.045 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:34.027 7fb6ca6be700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6c4109a70 con 0x7fb6c40d1880 2021-02-10T11:38:34.045 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:34.027 7fb6ca6be700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6c4109760 con 0x7fb6c41b0af0 2021-02-10T11:38:34.045 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:34.027 7fb6c3fff700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fb6c40d1880 0x7fb6c41b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:34.045 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:34.027 7fb6c37fe700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb6c41b0af0 0x7fb6c4108950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:34.046 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:34.027 7fb6c8cdb700 1 --1- >> v1:172.21.2.113:6789/0 conn(0x7fb6c40d1c80 0x7fb6c4108f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.113:6789/0 says I am v1:172.21.2.102:39180/0 (socket says 172.21.2.102:39180) 2021-02-10T11:38:34.046 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:34.027 7fb6c8cdb700 1 -- 172.21.2.102:0/1015627761 learned_addr learned my addr 172.21.2.102:0/1015627761 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:38:34.046 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:34.028 7fb6c3fff700 1 -- 172.21.2.102:0/1015627761 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb6c41b0af0 msgr2=0x7fb6c4108950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:34.046 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:34.028 7fb6c3fff700 1 --2- 172.21.2.102:0/1015627761 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb6c41b0af0 0x7fb6c4108950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:34.046 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:34.028 7fb6c3fff700 1 -- 172.21.2.102:0/1015627761 >> v1:172.21.2.113:6789/0 conn(0x7fb6c40d1c80 legacy=0x7fb6c4108f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:34.046 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:34.028 7fb6c2ffd700 1 -- 172.21.2.102:0/1015627761 <== mon.1 v1:172.21.2.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1489946378 0 0) 0x7fb6bc000d30 con 0x7fb6c40d1c80 2021-02-10T11:38:34.046 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:34.028 7fb6c3fff700 1 -- 172.21.2.102:0/1015627761 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6c4088070 con 0x7fb6c40d1880 2021-02-10T11:38:34.046 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:34.028 7fb6c3fff700 1 --2- 172.21.2.102:0/1015627761 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb6c40d1880 0x7fb6c41b04d0 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 rx=0x7fb6b4006650 tx=0x7fb6b4009a00).ready entity=mon.1 client_cookie=3602e0b595d28faa server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:34.046 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:34.028 7fb6c2ffd700 1 -- 172.21.2.102:0/1015627761 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb6b4009df0 con 0x7fb6c40d1880 2021-02-10T11:38:34.046 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:34.028 7fb6c2ffd700 1 -- 172.21.2.102:0/1015627761 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6b40009b0 con 0x7fb6c40d1880 2021-02-10T11:38:34.046 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:34.028 7fb6c2ffd700 1 -- 172.21.2.102:0/1015627761 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb6b400c2b0 con 0x7fb6c40d1880 2021-02-10T11:38:34.047 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:34.028 7fb6ca6be700 1 -- 172.21.2.102:0/1015627761 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb6c40d1880 msgr2=0x7fb6c41b04d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:34.047 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:34.028 7fb6ca6be700 1 --2- 172.21.2.102:0/1015627761 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb6c40d1880 0x7fb6c41b04d0 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 rx=0x7fb6b4006650 tx=0x7fb6b4009a00).stop 2021-02-10T11:38:34.047 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:34.029 7fb6ca6be700 1 -- 172.21.2.102:0/1015627761 shutdown_connections 2021-02-10T11:38:34.047 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:34.029 7fb6ca6be700 1 --2- 172.21.2.102:0/1015627761 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb6c41b0af0 0x7fb6c4108950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:34.047 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:34.029 7fb6ca6be700 1 --2- 172.21.2.102:0/1015627761 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb6c40d1880 0x7fb6c41b04d0 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:34.047 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:34.029 7fb6ca6be700 1 -- 172.21.2.102:0/1015627761 >> 172.21.2.102:0/1015627761 conn(0x7fb6c40ce430 msgr2=0x7fb6c40d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:34.047 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:34.029 7fb6ca6be700 1 -- 172.21.2.102:0/1015627761 shutdown_connections 2021-02-10T11:38:34.047 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:34.029 7fb6ca6be700 1 -- 172.21.2.102:0/1015627761 wait complete. 2021-02-10T11:38:34.049 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:34.044 7fb6ca6be700 1 Processor -- start 2021-02-10T11:38:34.049 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:34.044 7fb6ca6be700 1 -- start start 2021-02-10T11:38:34.049 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:34.044 7fb6ca6be700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb6c419a070 0x7fb6c408c780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:34.049 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:34.044 7fb6ca6be700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb6c419a470 0x7fb6c408cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:34.049 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:34.044 7fb6ca6be700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb6c4095520 0x7fb6c4095920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:34.049 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:34.044 7fb6ca6be700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6c4096900 con 0x7fb6c4095520 2021-02-10T11:38:34.049 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:34.044 7fb6ca6be700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6c408d4c0 con 0x7fb6c419a470 2021-02-10T11:38:34.049 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:34.044 7fb6ca6be700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6c419e9e0 con 0x7fb6c419a070 2021-02-10T11:38:34.049 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:34.045 7fb6c8cdb700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb6c4095520 0x7fb6c4095920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:34.049 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:34.045 7fb6c8cdb700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb6c4095520 0x7fb6c4095920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:39638/0 (socket says 172.21.2.102:39638) 2021-02-10T11:38:34.049 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:34.045 7fb6c8cdb700 1 -- 172.21.2.102:0/100917290 learned_addr learned my addr 172.21.2.102:0/100917290 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:38:34.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:34.045 7fb6c3fff700 1 --2- 172.21.2.102:0/100917290 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb6c419a070 0x7fb6c408c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:34.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:34.045 7fb6c37fe700 1 --2- 172.21.2.102:0/100917290 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb6c419a470 0x7fb6c408cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:34.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:34.045 7fb6c8cdb700 1 -- 172.21.2.102:0/100917290 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb6c419a070 msgr2=0x7fb6c408c780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:34.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:34.045 7fb6c8cdb700 1 --2- 172.21.2.102:0/100917290 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb6c419a070 0x7fb6c408c780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:34.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:34.045 7fb6c8cdb700 1 -- 172.21.2.102:0/100917290 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb6c419a470 msgr2=0x7fb6c408cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:34.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:34.045 7fb6c8cdb700 1 --2- 172.21.2.102:0/100917290 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb6c419a470 0x7fb6c408cda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:34.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:34.045 7fb6c8cdb700 1 -- 172.21.2.102:0/100917290 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6c4095f40 con 0x7fb6c4095520 2021-02-10T11:38:34.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:34.045 7fb6c3fff700 1 --2- 172.21.2.102:0/100917290 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb6c419a070 0x7fb6c408c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:38:34.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:34.045 7fb6c8cdb700 1 --2- 172.21.2.102:0/100917290 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb6c4095520 0x7fb6c4095920 secure :-1 s=READY pgs=3109 cs=0 l=1 rev1=1 rx=0x7fb6bc00a920 tx=0x7fb6bc00acc0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:34.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:34.046 7fb6c0ff9700 1 -- 172.21.2.102:0/100917290 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb6bc007ca0 con 0x7fb6c4095520 2021-02-10T11:38:34.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:34.046 7fb6c0ff9700 1 -- 172.21.2.102:0/100917290 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6bc0070d0 con 0x7fb6c4095520 2021-02-10T11:38:34.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:34.046 7fb6c0ff9700 1 -- 172.21.2.102:0/100917290 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb6bc0048b0 con 0x7fb6c4095520 2021-02-10T11:38:34.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:34.046 7fb6ca6be700 1 -- 172.21.2.102:0/100917290 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6c4095f40 con 0x7fb6c4095520 2021-02-10T11:38:34.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:34.046 7fb6ca6be700 1 -- 172.21.2.102:0/100917290 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6c4095f40 con 0x7fb6c4095520 2021-02-10T11:38:34.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:34.046 7fb6c0ff9700 1 -- 172.21.2.102:0/100917290 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fb6bc0072b0 con 0x7fb6c4095520 2021-02-10T11:38:34.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:34.046 7fb6c0ff9700 1 --2- 172.21.2.102:0/100917290 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb6a40186c0 0x7fb6a401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:34.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:34.046 7fb6c0ff9700 1 -- 172.21.2.102:0/100917290 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(180..180 src has 1..180) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fb6bc032b50 con 0x7fb6c4095520 2021-02-10T11:38:34.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:34.047 7fb6c3fff700 1 --2- 172.21.2.102:0/100917290 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb6a40186c0 0x7fb6a401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:34.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:34.047 7fb6c3fff700 1 --2- 172.21.2.102:0/100917290 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb6a40186c0 0x7fb6a401abc0 crc :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:34.078 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:34.077 7fb6ca6be700 1 -- 172.21.2.102:0/100917290 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6c4082980 con 0x7fb6c4095520 2021-02-10T11:38:34.080 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:34.079 7fb6c0ff9700 1 -- 172.21.2.102:0/100917290 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fb6bc007ca0 con 0x7fb6c4095520 2021-02-10T11:38:34.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:34.211 7fb6ca6be700 1 -- 172.21.2.102:0/100917290 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7fb6c4082980 con 0x7fb6c4095520 2021-02-10T11:38:34.216 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:34.214 7fb6c0ff9700 1 -- 172.21.2.102:0/100917290 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.0 v100) v1 ==== 184+0+0 (secure 0 0 0) 0x7fb6bc0070d0 con 0x7fb6c4095520 2021-02-10T11:38:34.216 INFO:teuthology.orchestra.run.gibba002.stderr:updated caps for client.0 2021-02-10T11:38:34.217 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:34.216 7fb6ca6be700 1 -- 172.21.2.102:0/100917290 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb6a40186c0 msgr2=0x7fb6a401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:34.217 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:34.216 7fb6ca6be700 1 --2- 172.21.2.102:0/100917290 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb6a40186c0 0x7fb6a401abc0 crc :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:34.218 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:34.216 7fb6ca6be700 1 -- 172.21.2.102:0/100917290 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb6c4095520 msgr2=0x7fb6c4095920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:34.218 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:34.216 7fb6ca6be700 1 --2- 172.21.2.102:0/100917290 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb6c4095520 0x7fb6c4095920 secure :-1 s=READY pgs=3109 cs=0 l=1 rev1=1 rx=0x7fb6bc00a920 tx=0x7fb6bc00acc0).stop 2021-02-10T11:38:34.218 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:34.216 7fb6ca6be700 1 -- 172.21.2.102:0/100917290 shutdown_connections 2021-02-10T11:38:34.218 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:34.216 7fb6ca6be700 1 --2- 172.21.2.102:0/100917290 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb6c4095520 0x7fb6c4095920 unknown :-1 s=CLOSED pgs=3109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:34.218 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:34.216 7fb6ca6be700 1 --2- 172.21.2.102:0/100917290 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb6a40186c0 0x7fb6a401abc0 unknown :-1 s=CLOSED pgs=1364 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:34.218 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:34.216 7fb6ca6be700 1 --2- 172.21.2.102:0/100917290 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb6c419a470 0x7fb6c408cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:34.218 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:34.216 7fb6ca6be700 1 --2- 172.21.2.102:0/100917290 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb6c419a070 0x7fb6c408c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:34.218 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:34.216 7fb6ca6be700 1 -- 172.21.2.102:0/100917290 >> 172.21.2.102:0/100917290 conn(0x7fb6c41a6190 msgr2=0x7fb6c40949b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:34.219 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:34.217 7fb6ca6be700 1 -- 172.21.2.102:0/100917290 shutdown_connections 2021-02-10T11:38:34.219 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:34.217 7fb6ca6be700 1 -- 172.21.2.102:0/100917290 wait complete. 2021-02-10T11:38:34.226 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.1 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:38:34.660 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:34.641 7fc95316d700 1 Processor -- start 2021-02-10T11:38:34.660 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:34.641 7fc95316d700 1 -- start start 2021-02-10T11:38:34.660 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:34.642 7fc95316d700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fc94c0d1880 0x7fc94c1b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:34.660 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:34.642 7fc95316d700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc94c0cff80 0x7fc94c109140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:34.660 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:34.642 7fc95316d700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc94c109a70 con 0x7fc94c1b0af0 2021-02-10T11:38:34.660 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:34.642 7fc95316d700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc94c109760 con 0x7fc94c0d1880 2021-02-10T11:38:34.660 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:34.642 7fc95316d700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc94c088070 con 0x7fc94c0cff80 2021-02-10T11:38:34.660 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:34.642 7fc94bfff700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fc94c1b0af0 0x7fc94c108950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:60102/0 (socket says 172.21.2.102:60102) 2021-02-10T11:38:34.660 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:34.642 7fc94bfff700 1 -- 172.21.2.102:0/2320773922 learned_addr learned my addr 172.21.2.102:0/2320773922 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:38:34.660 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:34.642 7fc950f89700 1 --2- 172.21.2.102:0/2320773922 >> v2:172.21.2.113:3300/0 conn(0x7fc94c0d1880 0x7fc94c1b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:34.660 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:34.642 7fc95178a700 1 --2- 172.21.2.102:0/2320773922 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc94c0cff80 0x7fc94c109140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:34.661 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:34.643 7fc94b7fe700 1 -- 172.21.2.102:0/2320773922 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1505714868 0 0) 0x7fc93c001060 con 0x7fc94c1b0af0 2021-02-10T11:38:34.661 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:34.643 7fc94b7fe700 1 -- 172.21.2.102:0/2320773922 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc938002fe0 con 0x7fc94c1b0af0 2021-02-10T11:38:34.661 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:34.643 7fc950f89700 1 -- 172.21.2.102:0/2320773922 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc94c0cff80 msgr2=0x7fc94c109140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:34.661 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:34.643 7fc950f89700 1 --2- 172.21.2.102:0/2320773922 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc94c0cff80 0x7fc94c109140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:34.661 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:34.643 7fc950f89700 1 -- 172.21.2.102:0/2320773922 >> v1:172.21.2.102:6789/0 conn(0x7fc94c1b0af0 legacy=0x7fc94c108950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:34.661 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:34.643 7fc950f89700 1 -- 172.21.2.102:0/2320773922 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc94c0d1c80 con 0x7fc94c0d1880 2021-02-10T11:38:34.661 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:34.643 7fc95178a700 1 --2- 172.21.2.102:0/2320773922 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc94c0cff80 0x7fc94c109140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:38:34.661 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:34.643 7fc950f89700 1 --2- 172.21.2.102:0/2320773922 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc94c0d1880 0x7fc94c1b04d0 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 rx=0x7fc934006650 tx=0x7fc934009a80).ready entity=mon.1 client_cookie=7e78398a14344b82 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:34.661 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:34.643 7fc94b7fe700 1 -- 172.21.2.102:0/2320773922 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc9340009b0 con 0x7fc94c0d1880 2021-02-10T11:38:34.661 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:34.643 7fc94b7fe700 1 -- 172.21.2.102:0/2320773922 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc93400c0c0 con 0x7fc94c0d1880 2021-02-10T11:38:34.661 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:34.643 7fc94b7fe700 1 -- 172.21.2.102:0/2320773922 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc93400c5e0 con 0x7fc94c0d1880 2021-02-10T11:38:34.661 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:34.643 7fc95316d700 1 -- 172.21.2.102:0/2320773922 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc94c0d1880 msgr2=0x7fc94c1b04d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:34.661 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:34.643 7fc95316d700 1 --2- 172.21.2.102:0/2320773922 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc94c0d1880 0x7fc94c1b04d0 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 rx=0x7fc934006650 tx=0x7fc934009a80).stop 2021-02-10T11:38:34.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:34.644 7fc95316d700 1 -- 172.21.2.102:0/2320773922 shutdown_connections 2021-02-10T11:38:34.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:34.644 7fc95316d700 1 --2- 172.21.2.102:0/2320773922 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc94c0cff80 0x7fc94c109140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:34.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:34.644 7fc95316d700 1 --2- 172.21.2.102:0/2320773922 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc94c0d1880 0x7fc94c1b04d0 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:34.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:34.644 7fc95316d700 1 -- 172.21.2.102:0/2320773922 >> 172.21.2.102:0/2320773922 conn(0x7fc94c0ce430 msgr2=0x7fc94c0d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:34.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:34.644 7fc95316d700 1 -- 172.21.2.102:0/2320773922 shutdown_connections 2021-02-10T11:38:34.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:34.644 7fc95316d700 1 -- 172.21.2.102:0/2320773922 wait complete. 2021-02-10T11:38:34.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:34.659 7fc95316d700 1 Processor -- start 2021-02-10T11:38:34.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:34.659 7fc95316d700 1 -- start start 2021-02-10T11:38:34.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:34.659 7fc95316d700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc94c0abd30 0x7fc94c082bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:34.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:34.659 7fc95316d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc94c0831d0 0x7fc94c08a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:34.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:34.659 7fc95316d700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc94c0ac130 0x7fc94c08aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:34.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:34.659 7fc95316d700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc94c083700 con 0x7fc94c0831d0 2021-02-10T11:38:34.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:34.659 7fc95316d700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc94c08b0c0 con 0x7fc94c0abd30 2021-02-10T11:38:34.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:34.659 7fc95316d700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc94c08b2a0 con 0x7fc94c0ac130 2021-02-10T11:38:34.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:34.660 7fc94bfff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc94c0831d0 0x7fc94c08a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:34.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:34.660 7fc94bfff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc94c0831d0 0x7fc94c08a480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:39652/0 (socket says 172.21.2.102:39652) 2021-02-10T11:38:34.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:34.660 7fc94bfff700 1 -- 172.21.2.102:0/3834125159 learned_addr learned my addr 172.21.2.102:0/3834125159 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:38:34.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:34.660 7fc950f89700 1 --2- 172.21.2.102:0/3834125159 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc94c0abd30 0x7fc94c082bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:34.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:34.660 7fc95178a700 1 --2- 172.21.2.102:0/3834125159 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc94c0ac130 0x7fc94c08aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:34.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:34.660 7fc950f89700 1 -- 172.21.2.102:0/3834125159 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc94c0ac130 msgr2=0x7fc94c08aaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:34.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:34.660 7fc950f89700 1 --2- 172.21.2.102:0/3834125159 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc94c0ac130 0x7fc94c08aaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:34.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:34.660 7fc950f89700 1 -- 172.21.2.102:0/3834125159 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc94c0831d0 msgr2=0x7fc94c08a480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:34.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:34.660 7fc950f89700 1 --2- 172.21.2.102:0/3834125159 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc94c0831d0 0x7fc94c08a480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:34.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:34.660 7fc950f89700 1 -- 172.21.2.102:0/3834125159 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc94c08b5f0 con 0x7fc94c0abd30 2021-02-10T11:38:34.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:34.660 7fc94bfff700 1 --2- 172.21.2.102:0/3834125159 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc94c0831d0 0x7fc94c08a480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:38:34.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:34.660 7fc950f89700 1 --2- 172.21.2.102:0/3834125159 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc94c0abd30 0x7fc94c082bb0 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 rx=0x7fc93400b530 tx=0x7fc93400b8d0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:34.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:34.661 7fc9497fa700 1 -- 172.21.2.102:0/3834125159 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc934009290 con 0x7fc94c0abd30 2021-02-10T11:38:34.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:34.661 7fc9497fa700 1 -- 172.21.2.102:0/3834125159 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc934002030 con 0x7fc94c0abd30 2021-02-10T11:38:34.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:34.661 7fc9497fa700 1 -- 172.21.2.102:0/3834125159 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc934002550 con 0x7fc94c0abd30 2021-02-10T11:38:34.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:34.661 7fc95316d700 1 -- 172.21.2.102:0/3834125159 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc94c08b5f0 con 0x7fc94c0abd30 2021-02-10T11:38:34.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:34.661 7fc95316d700 1 -- 172.21.2.102:0/3834125159 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc94c08b5f0 con 0x7fc94c0abd30 2021-02-10T11:38:34.666 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:34.661 7fc9497fa700 1 -- 172.21.2.102:0/3834125159 <== mon.1 v2:172.21.2.113:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fc934000c90 con 0x7fc94c0abd30 2021-02-10T11:38:34.666 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:34.661 7fc9497fa700 1 --2- 172.21.2.102:0/3834125159 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc9300186c0 0x7fc93001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:34.666 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:34.662 7fc9497fa700 1 -- 172.21.2.102:0/3834125159 <== mon.1 v2:172.21.2.113:3300/0 5 ==== osd_map(180..180 src has 1..180) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fc934032480 con 0x7fc94c0abd30 2021-02-10T11:38:34.666 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:34.662 7fc94bfff700 1 --2- 172.21.2.102:0/3834125159 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc9300186c0 0x7fc93001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:34.666 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:34.662 7fc94bfff700 1 --2- 172.21.2.102:0/3834125159 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc9300186c0 0x7fc93001abc0 crc :-1 s=READY pgs=1365 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:34.693 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:34.692 7fc95316d700 1 -- 172.21.2.102:0/3834125159 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc94c082980 con 0x7fc94c0abd30 2021-02-10T11:38:34.695 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:34.694 7fc9497fa700 1 -- 172.21.2.102:0/3834125159 <== mon.1 v2:172.21.2.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fc934006020 con 0x7fc94c0abd30 2021-02-10T11:38:34.827 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:34.825 7fc95316d700 1 -- 172.21.2.102:0/3834125159 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7fc94c082980 con 0x7fc94c0abd30 2021-02-10T11:38:34.831 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:34.829 7fc9497fa700 1 -- 172.21.2.102:0/3834125159 <== mon.1 v2:172.21.2.113:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.1 v101) v1 ==== 184+0+0 (secure 0 0 0) 0x7fc934000c90 con 0x7fc94c0abd30 2021-02-10T11:38:34.834 INFO:teuthology.orchestra.run.gibba002.stderr:updated caps for client.1 2021-02-10T11:38:34.835 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:34.834 7fc95316d700 1 -- 172.21.2.102:0/3834125159 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc9300186c0 msgr2=0x7fc93001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:34.835 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:34.834 7fc95316d700 1 --2- 172.21.2.102:0/3834125159 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc9300186c0 0x7fc93001abc0 crc :-1 s=READY pgs=1365 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:34.835 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:34.834 7fc95316d700 1 -- 172.21.2.102:0/3834125159 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc94c0abd30 msgr2=0x7fc94c082bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:34.835 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:34.834 7fc95316d700 1 --2- 172.21.2.102:0/3834125159 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc94c0abd30 0x7fc94c082bb0 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 rx=0x7fc93400b530 tx=0x7fc93400b8d0).stop 2021-02-10T11:38:34.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:34.834 7fc95316d700 1 -- 172.21.2.102:0/3834125159 shutdown_connections 2021-02-10T11:38:34.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:34.834 7fc95316d700 1 --2- 172.21.2.102:0/3834125159 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc94c0ac130 0x7fc94c08aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:34.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:34.834 7fc95316d700 1 --2- 172.21.2.102:0/3834125159 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc94c0831d0 0x7fc94c08a480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:34.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:34.834 7fc95316d700 1 --2- 172.21.2.102:0/3834125159 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc9300186c0 0x7fc93001abc0 unknown :-1 s=CLOSED pgs=1365 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:34.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:34.834 7fc95316d700 1 --2- 172.21.2.102:0/3834125159 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc94c0abd30 0x7fc94c082bb0 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:34.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:34.834 7fc95316d700 1 -- 172.21.2.102:0/3834125159 >> 172.21.2.102:0/3834125159 conn(0x7fc94c0afc60 msgr2=0x7fc94c0b50d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:34.837 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:34.834 7fc95316d700 1 -- 172.21.2.102:0/3834125159 shutdown_connections 2021-02-10T11:38:34.837 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:34.835 7fc95316d700 1 -- 172.21.2.102:0/3834125159 wait complete. 2021-02-10T11:38:34.844 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.2 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:38:35.287 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:35.268 7f6e9ca51700 1 Processor -- start 2021-02-10T11:38:35.287 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:35.268 7f6e9ca51700 1 -- start start 2021-02-10T11:38:35.287 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:35.268 7f6e9ca51700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f6e980d1c80 0x7f6e980cef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:35.287 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:35.268 7f6e9ca51700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6e980cf590 0x7f6e980cf990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:35.287 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:35.268 7f6e9ca51700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e980fe1d0 con 0x7f6e980d1880 2021-02-10T11:38:35.287 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:35.268 7f6e9ca51700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e981b0cc0 con 0x7f6e980d1c80 2021-02-10T11:38:35.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:35.268 7f6e9ca51700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e98109a70 con 0x7f6e980cf590 2021-02-10T11:38:35.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:35.269 7f6e9705f700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6e980cf590 0x7f6e980cf990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:35.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:35.269 7f6e9605d700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f6e980d1c80 0x7f6e980cef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:35.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:35.269 7f6e9685e700 1 --1- >> v1:172.21.2.113:6789/0 conn(0x7f6e980d1880 0x7f6e981b04d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.113:6789/0 says I am v1:172.21.2.102:39206/0 (socket says 172.21.2.102:39206) 2021-02-10T11:38:35.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:35.269 7f6e9685e700 1 -- 172.21.2.102:0/4047366356 learned_addr learned my addr 172.21.2.102:0/4047366356 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:38:35.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:35.270 7f6e9605d700 1 -- 172.21.2.102:0/4047366356 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6e980cf590 msgr2=0x7f6e980cf990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:35.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:35.270 7f6e9605d700 1 --2- 172.21.2.102:0/4047366356 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6e980cf590 0x7f6e980cf990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:35.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:35.270 7f6e9605d700 1 -- 172.21.2.102:0/4047366356 >> v1:172.21.2.113:6789/0 conn(0x7f6e980d1880 legacy=0x7f6e981b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:35.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:35.270 7f6e9585c700 1 -- 172.21.2.102:0/4047366356 <== mon.1 v1:172.21.2.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1728621576 0 0) 0x7f6e880010c0 con 0x7f6e980d1880 2021-02-10T11:38:35.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:35.270 7f6e9605d700 1 -- 172.21.2.102:0/4047366356 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e98088070 con 0x7f6e980d1c80 2021-02-10T11:38:35.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:35.270 7f6e9705f700 1 --2- 172.21.2.102:0/4047366356 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6e980cf590 0x7f6e980cf990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:38:35.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:35.270 7f6e9605d700 1 --2- 172.21.2.102:0/4047366356 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6e980d1c80 0x7f6e980cef70 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 rx=0x7f6e8c006650 tx=0x7f6e8c009a80).ready entity=mon.1 client_cookie=5113d8e197b96f72 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:35.289 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:35.271 7f6e9585c700 1 -- 172.21.2.102:0/4047366356 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6e8c0009b0 con 0x7f6e980d1c80 2021-02-10T11:38:35.289 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:35.271 7f6e9585c700 1 -- 172.21.2.102:0/4047366356 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e8c00c0c0 con 0x7f6e980d1c80 2021-02-10T11:38:35.289 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:35.271 7f6e9585c700 1 -- 172.21.2.102:0/4047366356 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6e8c00c5e0 con 0x7f6e980d1c80 2021-02-10T11:38:35.289 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:35.271 7f6e9ca51700 1 -- 172.21.2.102:0/4047366356 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6e980d1c80 msgr2=0x7f6e980cef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:35.289 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:35.271 7f6e9ca51700 1 --2- 172.21.2.102:0/4047366356 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6e980d1c80 0x7f6e980cef70 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 rx=0x7f6e8c006650 tx=0x7f6e8c009a80).stop 2021-02-10T11:38:35.289 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:35.272 7f6e9ca51700 1 -- 172.21.2.102:0/4047366356 shutdown_connections 2021-02-10T11:38:35.289 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:35.272 7f6e9ca51700 1 --2- 172.21.2.102:0/4047366356 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6e980cf590 0x7f6e980cf990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:35.289 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:35.272 7f6e9ca51700 1 --2- 172.21.2.102:0/4047366356 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6e980d1c80 0x7f6e980cef70 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:35.289 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:35.272 7f6e9ca51700 1 -- 172.21.2.102:0/4047366356 >> 172.21.2.102:0/4047366356 conn(0x7f6e980ce430 msgr2=0x7f6e980d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:35.289 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:35.272 7f6e9ca51700 1 -- 172.21.2.102:0/4047366356 shutdown_connections 2021-02-10T11:38:35.289 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:35.272 7f6e9ca51700 1 -- 172.21.2.102:0/4047366356 wait complete. 2021-02-10T11:38:35.291 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:35.286 7f6e9ca51700 1 Processor -- start 2021-02-10T11:38:35.291 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:35.286 7f6e9ca51700 1 -- start start 2021-02-10T11:38:35.291 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:35.287 7f6e9ca51700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6e981a0580 0x7f6e98086fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:35.292 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:35.287 7f6e9ca51700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6e980b66b0 0x7f6e980b6ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:35.292 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:35.287 7f6e9ca51700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6e98082bb0 0x7f6e98082fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:35.292 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:35.287 7f6e9ca51700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e980875d0 con 0x7f6e98082bb0 2021-02-10T11:38:35.292 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:35.287 7f6e9ca51700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e980835d0 con 0x7f6e981a0580 2021-02-10T11:38:35.292 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:35.287 7f6e9ca51700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e980837f0 con 0x7f6e980b66b0 2021-02-10T11:38:35.292 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:35.287 7f6e9705f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6e98082bb0 0x7f6e98082fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:35.292 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:35.287 7f6e9705f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6e98082bb0 0x7f6e98082fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:39664/0 (socket says 172.21.2.102:39664) 2021-02-10T11:38:35.292 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:35.287 7f6e9705f700 1 -- 172.21.2.102:0/2480470337 learned_addr learned my addr 172.21.2.102:0/2480470337 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:38:35.292 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:35.287 7f6e9705f700 1 -- 172.21.2.102:0/2480470337 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6e980b66b0 msgr2=0x7f6e980b6ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:35.292 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:35.287 7f6e9705f700 1 --2- 172.21.2.102:0/2480470337 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6e980b66b0 0x7f6e980b6ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:35.292 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:35.287 7f6e9705f700 1 -- 172.21.2.102:0/2480470337 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6e981a0580 msgr2=0x7f6e98086fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:35.292 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:35.287 7f6e9705f700 1 --2- 172.21.2.102:0/2480470337 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6e981a0580 0x7f6e98086fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:35.292 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:35.287 7f6e9705f700 1 -- 172.21.2.102:0/2480470337 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e98083a10 con 0x7f6e98082bb0 2021-02-10T11:38:35.292 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:35.288 7f6e9705f700 1 --2- 172.21.2.102:0/2480470337 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6e98082bb0 0x7f6e98082fb0 secure :-1 s=READY pgs=3111 cs=0 l=1 rev1=1 rx=0x7f6e9000b970 tx=0x7f6e9000bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:35.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:35.288 7f6e877fe700 1 -- 172.21.2.102:0/2480470337 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6e90000f90 con 0x7f6e98082bb0 2021-02-10T11:38:35.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:35.288 7f6e877fe700 1 -- 172.21.2.102:0/2480470337 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e90003030 con 0x7f6e98082bb0 2021-02-10T11:38:35.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:35.288 7f6e9ca51700 1 -- 172.21.2.102:0/2480470337 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e98083a10 con 0x7f6e98082bb0 2021-02-10T11:38:35.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:35.288 7f6e9ca51700 1 -- 172.21.2.102:0/2480470337 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e98083fa0 con 0x7f6e98082bb0 2021-02-10T11:38:35.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:35.288 7f6e877fe700 1 -- 172.21.2.102:0/2480470337 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6e900015f0 con 0x7f6e98082bb0 2021-02-10T11:38:35.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:35.288 7f6e877fe700 1 -- 172.21.2.102:0/2480470337 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f6e900056b0 con 0x7f6e98082bb0 2021-02-10T11:38:35.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:35.289 7f6e877fe700 1 --2- 172.21.2.102:0/2480470337 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6e7c018730 0x7f6e7c01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:35.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:35.289 7f6e877fe700 1 -- 172.21.2.102:0/2480470337 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(180..180 src has 1..180) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f6e90032b30 con 0x7f6e98082bb0 2021-02-10T11:38:35.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:35.289 7f6e9685e700 1 --2- 172.21.2.102:0/2480470337 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6e7c018730 0x7f6e7c01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:35.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:35.289 7f6e9685e700 1 --2- 172.21.2.102:0/2480470337 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6e7c018730 0x7f6e7c01ac10 crc :-1 s=READY pgs=1366 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:35.319 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:35.318 7f6e9ca51700 1 -- 172.21.2.102:0/2480470337 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e98082980 con 0x7f6e98082bb0 2021-02-10T11:38:35.322 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:35.321 7f6e877fe700 1 -- 172.21.2.102:0/2480470337 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f6e90000f40 con 0x7f6e98082bb0 2021-02-10T11:38:35.453 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:35.452 7f6e9ca51700 1 -- 172.21.2.102:0/2480470337 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f6e98082980 con 0x7f6e98082bb0 2021-02-10T11:38:35.457 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:35.456 7f6e877fe700 1 -- 172.21.2.102:0/2480470337 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.2 v102) v1 ==== 184+0+0 (secure 0 0 0) 0x7f6e900056b0 con 0x7f6e98082bb0 2021-02-10T11:38:35.460 INFO:teuthology.orchestra.run.gibba002.stderr:updated caps for client.2 2021-02-10T11:38:35.462 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:35.461 7f6e9ca51700 1 -- 172.21.2.102:0/2480470337 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6e7c018730 msgr2=0x7f6e7c01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:35.462 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:35.461 7f6e9ca51700 1 --2- 172.21.2.102:0/2480470337 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6e7c018730 0x7f6e7c01ac10 crc :-1 s=READY pgs=1366 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:35.463 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:35.461 7f6e9ca51700 1 -- 172.21.2.102:0/2480470337 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6e98082bb0 msgr2=0x7f6e98082fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:35.463 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:35.461 7f6e9ca51700 1 --2- 172.21.2.102:0/2480470337 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6e98082bb0 0x7f6e98082fb0 secure :-1 s=READY pgs=3111 cs=0 l=1 rev1=1 rx=0x7f6e9000b970 tx=0x7f6e9000bd10).stop 2021-02-10T11:38:35.463 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:35.461 7f6e9ca51700 1 -- 172.21.2.102:0/2480470337 shutdown_connections 2021-02-10T11:38:35.463 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:35.461 7f6e9ca51700 1 --2- 172.21.2.102:0/2480470337 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6e98082bb0 0x7f6e98082fb0 unknown :-1 s=CLOSED pgs=3111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:35.463 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:35.461 7f6e9ca51700 1 --2- 172.21.2.102:0/2480470337 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6e980b66b0 0x7f6e980b6ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:35.463 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:35.461 7f6e9ca51700 1 --2- 172.21.2.102:0/2480470337 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6e7c018730 0x7f6e7c01ac10 unknown :-1 s=CLOSED pgs=1366 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:35.463 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:35.461 7f6e9ca51700 1 --2- 172.21.2.102:0/2480470337 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6e981a0580 0x7f6e98086fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:35.463 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:35.461 7f6e9ca51700 1 -- 172.21.2.102:0/2480470337 >> 172.21.2.102:0/2480470337 conn(0x7f6e980afc60 msgr2=0x7f6e980b50d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:35.463 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:35.461 7f6e9ca51700 1 -- 172.21.2.102:0/2480470337 shutdown_connections 2021-02-10T11:38:35.463 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:35.461 7f6e9ca51700 1 -- 172.21.2.102:0/2480470337 wait complete. 2021-02-10T11:38:35.471 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.3 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:38:35.911 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:35.893 7ff46762a700 1 Processor -- start 2021-02-10T11:38:35.928 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:35.893 7ff46762a700 1 -- start start 2021-02-10T11:38:35.928 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:35.894 7ff46762a700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7ff4600d1880 0x7ff4601b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:35.928 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:35.894 7ff46762a700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff4600cff80 0x7ff460109140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:35.928 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:35.894 7ff46762a700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff460109a70 con 0x7ff4601b0af0 2021-02-10T11:38:35.928 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:35.894 7ff46762a700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff460109760 con 0x7ff4600d1880 2021-02-10T11:38:35.928 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:35.894 7ff46762a700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff460088070 con 0x7ff4600cff80 2021-02-10T11:38:35.928 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:35.894 7ff465446700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7ff4600d1880 0x7ff4601b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:35.929 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:35.894 7ff464c45700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7ff4601b0af0 0x7ff460108950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:60124/0 (socket says 172.21.2.102:60124) 2021-02-10T11:38:35.929 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:35.894 7ff464c45700 1 -- 172.21.2.102:0/1381867485 learned_addr learned my addr 172.21.2.102:0/1381867485 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:38:35.929 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:35.894 7ff465446700 1 -- 172.21.2.102:0/1381867485 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff4600cff80 msgr2=0x7ff460109140 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:38:35.929 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:35.894 7ff465446700 1 --2- 172.21.2.102:0/1381867485 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff4600cff80 0x7ff460109140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:35.929 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:35.894 7ff465446700 1 -- 172.21.2.102:0/1381867485 >> v1:172.21.2.102:6789/0 conn(0x7ff4601b0af0 legacy=0x7ff460108950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:35.929 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:35.895 7ff465446700 1 -- 172.21.2.102:0/1381867485 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4600d1c80 con 0x7ff4600d1880 2021-02-10T11:38:35.929 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:35.895 7ff465446700 1 --2- 172.21.2.102:0/1381867485 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff4600d1880 0x7ff4601b04d0 secure :-1 s=READY pgs=3113 cs=0 l=1 rev1=1 rx=0x7ff454006650 tx=0x7ff454009a00).ready entity=mon.0 client_cookie=d8cebdeff6f82641 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:35.929 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:35.895 7ff453fff700 1 -- 172.21.2.102:0/1381867485 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff4540009b0 con 0x7ff4600d1880 2021-02-10T11:38:35.929 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:35.895 7ff453fff700 1 -- 172.21.2.102:0/1381867485 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff45400c0c0 con 0x7ff4600d1880 2021-02-10T11:38:35.929 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:35.895 7ff453fff700 1 -- 172.21.2.102:0/1381867485 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff45400c5a0 con 0x7ff4600d1880 2021-02-10T11:38:35.929 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:35.895 7ff46762a700 1 -- 172.21.2.102:0/1381867485 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff4600d1880 msgr2=0x7ff4601b04d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:35.929 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:35.895 7ff46762a700 1 --2- 172.21.2.102:0/1381867485 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff4600d1880 0x7ff4601b04d0 secure :-1 s=READY pgs=3113 cs=0 l=1 rev1=1 rx=0x7ff454006650 tx=0x7ff454009a00).stop 2021-02-10T11:38:35.929 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:35.896 7ff46762a700 1 -- 172.21.2.102:0/1381867485 shutdown_connections 2021-02-10T11:38:35.930 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:35.896 7ff46762a700 1 --2- 172.21.2.102:0/1381867485 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff4600cff80 0x7ff460109140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:35.930 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:35.896 7ff46762a700 1 --2- 172.21.2.102:0/1381867485 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff4600d1880 0x7ff4601b04d0 unknown :-1 s=CLOSED pgs=3113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:35.930 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:35.896 7ff46762a700 1 -- 172.21.2.102:0/1381867485 >> 172.21.2.102:0/1381867485 conn(0x7ff4600ce430 msgr2=0x7ff4600d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:35.930 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:35.896 7ff46762a700 1 -- 172.21.2.102:0/1381867485 shutdown_connections 2021-02-10T11:38:35.930 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:35.896 7ff46762a700 1 -- 172.21.2.102:0/1381867485 wait complete. 2021-02-10T11:38:35.931 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:35.910 7ff46762a700 1 Processor -- start 2021-02-10T11:38:35.932 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:35.910 7ff46762a700 1 -- start start 2021-02-10T11:38:35.932 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:35.911 7ff46762a700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff4601a68f0 0x7ff46019fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:35.932 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:35.911 7ff46762a700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff4601a5e00 0x7ff4601a02c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:35.932 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:35.911 7ff46762a700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff4601a08e0 0x7ff4601aae20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:35.932 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:35.911 7ff46762a700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4601a6370 con 0x7ff4601a68f0 2021-02-10T11:38:35.932 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:35.911 7ff46762a700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4601a3960 con 0x7ff4601a5e00 2021-02-10T11:38:35.932 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:35.911 7ff46762a700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4601ab440 con 0x7ff4601a08e0 2021-02-10T11:38:35.932 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:35.911 7ff465446700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff4601a68f0 0x7ff46019fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:35.932 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:35.911 7ff465446700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff4601a68f0 0x7ff46019fca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:39676/0 (socket says 172.21.2.102:39676) 2021-02-10T11:38:35.932 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:35.911 7ff465446700 1 -- 172.21.2.102:0/1413574796 learned_addr learned my addr 172.21.2.102:0/1413574796 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:38:35.932 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:35.912 7ff465446700 1 -- 172.21.2.102:0/1413574796 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff4601a08e0 msgr2=0x7ff4601aae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:35.932 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:35.912 7ff465446700 1 --2- 172.21.2.102:0/1413574796 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff4601a08e0 0x7ff4601aae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:35.933 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:35.912 7ff465446700 1 -- 172.21.2.102:0/1413574796 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff4601a5e00 msgr2=0x7ff4601a02c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:35.933 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:35.912 7ff465446700 1 --2- 172.21.2.102:0/1413574796 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff4601a5e00 0x7ff4601a02c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:35.933 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:35.912 7ff465446700 1 -- 172.21.2.102:0/1413574796 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4601ab660 con 0x7ff4601a68f0 2021-02-10T11:38:35.933 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:35.912 7ff465446700 1 --2- 172.21.2.102:0/1413574796 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff4601a68f0 0x7ff46019fca0 secure :-1 s=READY pgs=3114 cs=0 l=1 rev1=1 rx=0x7ff45400b590 tx=0x7ff45400b930).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:35.933 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:35.912 7ff451ffb700 1 -- 172.21.2.102:0/1413574796 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff45400bd20 con 0x7ff4601a68f0 2021-02-10T11:38:35.933 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:35.912 7ff451ffb700 1 -- 172.21.2.102:0/1413574796 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff454000c90 con 0x7ff4601a68f0 2021-02-10T11:38:35.933 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:35.912 7ff451ffb700 1 -- 172.21.2.102:0/1413574796 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff4540022b0 con 0x7ff4601a68f0 2021-02-10T11:38:35.933 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:35.912 7ff46762a700 1 -- 172.21.2.102:0/1413574796 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4601ab660 con 0x7ff4601a68f0 2021-02-10T11:38:35.933 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:35.912 7ff46762a700 1 -- 172.21.2.102:0/1413574796 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4601abbf0 con 0x7ff4601a68f0 2021-02-10T11:38:35.933 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:35.912 7ff451ffb700 1 -- 172.21.2.102:0/1413574796 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ff4540052e0 con 0x7ff4601a68f0 2021-02-10T11:38:35.933 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:35.913 7ff451ffb700 1 --2- 172.21.2.102:0/1413574796 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff4400186d0 0x7ff44001abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:35.933 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:35.913 7ff451ffb700 1 -- 172.21.2.102:0/1413574796 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(180..180 src has 1..180) v4 ==== 5573+0+0 (secure 0 0 0) 0x7ff454032610 con 0x7ff4601a68f0 2021-02-10T11:38:35.933 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:35.913 7ff464c45700 1 --2- 172.21.2.102:0/1413574796 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff4400186d0 0x7ff44001abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:35.934 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:35.913 7ff464c45700 1 --2- 172.21.2.102:0/1413574796 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff4400186d0 0x7ff44001abd0 crc :-1 s=READY pgs=1367 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:35.945 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:35.944 7ff46762a700 1 -- 172.21.2.102:0/1413574796 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4600829a0 con 0x7ff4601a68f0 2021-02-10T11:38:35.948 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:35.947 7ff451ffb700 1 -- 172.21.2.102:0/1413574796 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ff4540009b0 con 0x7ff4601a68f0 2021-02-10T11:38:36.078 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:36.077 7ff46762a700 1 -- 172.21.2.102:0/1413574796 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.3", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7ff4600829a0 con 0x7ff4601a68f0 2021-02-10T11:38:36.082 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:36.081 7ff451ffb700 1 -- 172.21.2.102:0/1413574796 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.3", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.3 v103) v1 ==== 184+0+0 (secure 0 0 0) 0x7ff4540052e0 con 0x7ff4601a68f0 2021-02-10T11:38:36.085 INFO:teuthology.orchestra.run.gibba002.stderr:updated caps for client.3 2021-02-10T11:38:36.086 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:36.085 7ff46762a700 1 -- 172.21.2.102:0/1413574796 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff4400186d0 msgr2=0x7ff44001abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:36.086 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:36.085 7ff46762a700 1 --2- 172.21.2.102:0/1413574796 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff4400186d0 0x7ff44001abd0 crc :-1 s=READY pgs=1367 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:36.086 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:36.085 7ff46762a700 1 -- 172.21.2.102:0/1413574796 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff4601a68f0 msgr2=0x7ff46019fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:36.087 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:36.085 7ff46762a700 1 --2- 172.21.2.102:0/1413574796 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff4601a68f0 0x7ff46019fca0 secure :-1 s=READY pgs=3114 cs=0 l=1 rev1=1 rx=0x7ff45400b590 tx=0x7ff45400b930).stop 2021-02-10T11:38:36.087 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:36.086 7ff46762a700 1 -- 172.21.2.102:0/1413574796 shutdown_connections 2021-02-10T11:38:36.087 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:36.086 7ff46762a700 1 --2- 172.21.2.102:0/1413574796 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff4601a08e0 0x7ff4601aae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:36.087 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:36.086 7ff46762a700 1 --2- 172.21.2.102:0/1413574796 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff4400186d0 0x7ff44001abd0 unknown :-1 s=CLOSED pgs=1367 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:36.087 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:36.086 7ff46762a700 1 --2- 172.21.2.102:0/1413574796 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff4601a5e00 0x7ff4601a02c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:36.087 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:36.086 7ff46762a700 1 --2- 172.21.2.102:0/1413574796 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff4601a68f0 0x7ff46019fca0 unknown :-1 s=CLOSED pgs=3114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:36.087 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:36.086 7ff46762a700 1 -- 172.21.2.102:0/1413574796 >> 172.21.2.102:0/1413574796 conn(0x7ff46019f880 msgr2=0x7ff4601a71a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:36.088 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:36.086 7ff46762a700 1 -- 172.21.2.102:0/1413574796 shutdown_connections 2021-02-10T11:38:36.088 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:36.086 7ff46762a700 1 -- 172.21.2.102:0/1413574796 wait complete. 2021-02-10T11:38:36.095 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:38:36.524 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:36.506 7f623cc90700 1 Processor -- start 2021-02-10T11:38:36.524 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:36.506 7f623cc90700 1 -- start start 2021-02-10T11:38:36.524 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:36.507 7f623cc90700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f62380d1870 0x7f62381b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:36.524 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:36.507 7f623cc90700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f62380f0650 0x7f62380cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:36.524 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:36.507 7f623cc90700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62380f0b10 con 0x7f62381b0a10 2021-02-10T11:38:36.524 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:36.507 7f623cc90700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62380e5ed0 con 0x7f62380d1870 2021-02-10T11:38:36.524 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:36.507 7f623cc90700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62380d1c70 con 0x7f62380f0650 2021-02-10T11:38:36.524 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:36.507 7f623685e700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f62380d1870 0x7f62381b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:36.524 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:36.507 7f623685e700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f62380d1870 0x7f62381b03f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:39682/0 (socket says 172.21.2.102:39682) 2021-02-10T11:38:36.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:36.507 7f623685e700 1 -- 172.21.2.102:0/601841239 learned_addr learned my addr 172.21.2.102:0/601841239 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:38:36.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:36.507 7f623605d700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f62381b0a10 0x7f62380cf330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:60140/0 (socket says 172.21.2.102:60140) 2021-02-10T11:38:36.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:36.507 7f623685e700 1 -- 172.21.2.102:0/601841239 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f62380f0650 msgr2=0x7f62380cfb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:36.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:36.507 7f623685e700 1 --2- 172.21.2.102:0/601841239 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f62380f0650 0x7f62380cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:36.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:36.507 7f623685e700 1 -- 172.21.2.102:0/601841239 >> v1:172.21.2.102:6789/0 conn(0x7f62381b0a10 legacy=0x7f62380cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:36.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:36.507 7f623685e700 1 -- 172.21.2.102:0/601841239 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62380d1e90 con 0x7f62380d1870 2021-02-10T11:38:36.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:36.507 7f623685e700 1 --2- 172.21.2.102:0/601841239 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f62380d1870 0x7f62381b03f0 secure :-1 s=READY pgs=3116 cs=0 l=1 rev1=1 rx=0x7f6228006700 tx=0x7f6228009b00).ready entity=mon.0 client_cookie=45db95cc0f21ff39 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:36.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:36.508 7f623585c700 1 -- 172.21.2.102:0/601841239 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f62280009b0 con 0x7f62380d1870 2021-02-10T11:38:36.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:36.508 7f623585c700 1 -- 172.21.2.102:0/601841239 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f622800c0c0 con 0x7f62380d1870 2021-02-10T11:38:36.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:36.508 7f623585c700 1 -- 172.21.2.102:0/601841239 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f622800c550 con 0x7f62380d1870 2021-02-10T11:38:36.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:36.508 7f623cc90700 1 -- 172.21.2.102:0/601841239 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f62380d1870 msgr2=0x7f62381b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:36.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:36.508 7f623cc90700 1 --2- 172.21.2.102:0/601841239 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f62380d1870 0x7f62381b03f0 secure :-1 s=READY pgs=3116 cs=0 l=1 rev1=1 rx=0x7f6228006700 tx=0x7f6228009b00).stop 2021-02-10T11:38:36.526 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:36.508 7f623cc90700 1 -- 172.21.2.102:0/601841239 shutdown_connections 2021-02-10T11:38:36.526 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:36.508 7f623cc90700 1 --2- 172.21.2.102:0/601841239 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f62380f0650 0x7f62380cfb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:36.526 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:36.508 7f623cc90700 1 --2- 172.21.2.102:0/601841239 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f62380d1870 0x7f62381b03f0 unknown :-1 s=CLOSED pgs=3116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:36.526 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:36.508 7f623cc90700 1 -- 172.21.2.102:0/601841239 >> 172.21.2.102:0/601841239 conn(0x7f62380ce450 msgr2=0x7f62380d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:36.526 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:36.508 7f623cc90700 1 -- 172.21.2.102:0/601841239 shutdown_connections 2021-02-10T11:38:36.526 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:36.508 7f623cc90700 1 -- 172.21.2.102:0/601841239 wait complete. 2021-02-10T11:38:36.526 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:36.523 7f623cc90700 1 Processor -- start 2021-02-10T11:38:36.526 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:36.523 7f623cc90700 1 -- start start 2021-02-10T11:38:36.528 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:36.523 7f623cc90700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f623808a450 0x7f6238095810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:36.528 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:36.523 7f623cc90700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f62381a2f60 0x7f623808afd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:36.528 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:36.523 7f623cc90700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f62381a3360 0x7f623808b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:36.528 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:36.523 7f623cc90700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f623808a850 con 0x7f62381a3360 2021-02-10T11:38:36.528 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:36.523 7f623cc90700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62380a7750 con 0x7f62381a2f60 2021-02-10T11:38:36.529 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:36.523 7f623cc90700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f623808aac0 con 0x7f623808a450 2021-02-10T11:38:36.529 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:36.523 7f623705f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f62381a3360 0x7f623808b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:36.529 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:36.523 7f623705f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f62381a3360 0x7f623808b5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:39688/0 (socket says 172.21.2.102:39688) 2021-02-10T11:38:36.529 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:36.523 7f623705f700 1 -- 172.21.2.102:0/1004682927 learned_addr learned my addr 172.21.2.102:0/1004682927 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:38:36.529 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:36.523 7f623705f700 1 -- 172.21.2.102:0/1004682927 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f623808a450 msgr2=0x7f6238095810 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:38:36.529 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:36.523 7f623705f700 1 --2- 172.21.2.102:0/1004682927 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f623808a450 0x7f6238095810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:36.529 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:36.523 7f623705f700 1 -- 172.21.2.102:0/1004682927 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f62381a2f60 msgr2=0x7f623808afd0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:38:36.529 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:36.523 7f623705f700 1 --2- 172.21.2.102:0/1004682927 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f62381a2f60 0x7f623808afd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:36.529 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:36.523 7f623705f700 1 -- 172.21.2.102:0/1004682927 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6238095fa0 con 0x7f62381a3360 2021-02-10T11:38:36.529 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:36.524 7f623705f700 1 --2- 172.21.2.102:0/1004682927 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f62381a3360 0x7f623808b5f0 secure :-1 s=READY pgs=3117 cs=0 l=1 rev1=1 rx=0x7f6230009980 tx=0x7f623000bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:36.529 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:36.524 7f622f7fe700 1 -- 172.21.2.102:0/1004682927 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6230005af0 con 0x7f62381a3360 2021-02-10T11:38:36.529 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:36.524 7f622f7fe700 1 -- 172.21.2.102:0/1004682927 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6230005d70 con 0x7f62381a3360 2021-02-10T11:38:36.529 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:36.524 7f623cc90700 1 -- 172.21.2.102:0/1004682927 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6238095fa0 con 0x7f62381a3360 2021-02-10T11:38:36.529 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:36.524 7f622f7fe700 1 -- 172.21.2.102:0/1004682927 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6230006220 con 0x7f62381a3360 2021-02-10T11:38:36.530 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:36.524 7f623cc90700 1 -- 172.21.2.102:0/1004682927 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6238095fa0 con 0x7f62381a3360 2021-02-10T11:38:36.530 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:36.524 7f622f7fe700 1 -- 172.21.2.102:0/1004682927 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f6230005870 con 0x7f62381a3360 2021-02-10T11:38:36.530 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:36.524 7f622f7fe700 1 --2- 172.21.2.102:0/1004682927 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f621c0186d0 0x7f621c01abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:36.530 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:36.524 7f622f7fe700 1 -- 172.21.2.102:0/1004682927 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(180..180 src has 1..180) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f62300329c0 con 0x7f62381a3360 2021-02-10T11:38:36.530 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:36.525 7f623685e700 1 --2- 172.21.2.102:0/1004682927 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f621c0186d0 0x7f621c01abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:36.530 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:36.525 7f623685e700 1 --2- 172.21.2.102:0/1004682927 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f621c0186d0 0x7f621c01abd0 crc :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:36.558 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:36.557 7f623cc90700 1 -- 172.21.2.102:0/1004682927 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62380a7710 con 0x7f62381a3360 2021-02-10T11:38:36.560 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:36.559 7f622f7fe700 1 -- 172.21.2.102:0/1004682927 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f6230005d50 con 0x7f62381a3360 2021-02-10T11:38:36.689 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:36.687 7f623cc90700 1 -- 172.21.2.102:0/1004682927 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f62380a7710 con 0x7f62381a3360 2021-02-10T11:38:36.689 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:36.688 7f622f7fe700 1 -- 172.21.2.102:0/1004682927 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 157 v157) v1 ==== 95+0+2107 (secure 0 0 0) 0x7f6230005d50 con 0x7f62381a3360 2021-02-10T11:38:36.689 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:38:36.690 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":157,"default_fscid":19,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8815,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.102:6835/3396946217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6834","nonce":3396946217},{"type":"v1","addr":"172.21.2.102:6835","nonce":3396946217}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":153}],"filesystems":[{"mdsmap":{"epoch":157,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:38:30.335525","modified":"2021-02-10 11:38:32.341928","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8812},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8812":{"gid":8812,"name":"b","rank":0,"incarnation":154,"state":"up:active","state_seq":3,"addr":"172.21.2.113:6833/1513291768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6832","nonce":1513291768},{"type":"v1","addr":"172.21.2.113:6833","nonce":1513291768}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[39],"metadata_pool":38,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":19}]} 2021-02-10T11:38:36.690 INFO:teuthology.orchestra.run.gibba002.stderr:dumped fsmap epoch 157 2021-02-10T11:38:36.691 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:36.690 7f623cc90700 1 -- 172.21.2.102:0/1004682927 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f621c0186d0 msgr2=0x7f621c01abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:36.691 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:36.690 7f623cc90700 1 --2- 172.21.2.102:0/1004682927 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f621c0186d0 0x7f621c01abd0 crc :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:36.691 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:36.690 7f623cc90700 1 -- 172.21.2.102:0/1004682927 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f62381a3360 msgr2=0x7f623808b5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:36.691 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:36.690 7f623cc90700 1 --2- 172.21.2.102:0/1004682927 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f62381a3360 0x7f623808b5f0 secure :-1 s=READY pgs=3117 cs=0 l=1 rev1=1 rx=0x7f6230009980 tx=0x7f623000bcd0).stop 2021-02-10T11:38:36.692 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:36.690 7f623cc90700 1 -- 172.21.2.102:0/1004682927 shutdown_connections 2021-02-10T11:38:36.692 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:36.690 7f623cc90700 1 --2- 172.21.2.102:0/1004682927 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f62381a3360 0x7f623808b5f0 unknown :-1 s=CLOSED pgs=3117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:36.692 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:36.690 7f623cc90700 1 --2- 172.21.2.102:0/1004682927 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f621c0186d0 0x7f621c01abd0 unknown :-1 s=CLOSED pgs=1368 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:36.692 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:36.690 7f623cc90700 1 --2- 172.21.2.102:0/1004682927 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f62381a2f60 0x7f623808afd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:36.692 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:36.690 7f623cc90700 1 --2- 172.21.2.102:0/1004682927 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f623808a450 0x7f6238095810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:36.692 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:36.690 7f623cc90700 1 -- 172.21.2.102:0/1004682927 >> 172.21.2.102:0/1004682927 conn(0x7f62381b9b90 msgr2=0x7f623818c260 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:36.692 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:36.690 7f623cc90700 1 -- 172.21.2.102:0/1004682927 shutdown_connections 2021-02-10T11:38:36.692 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:36.690 7f623cc90700 1 -- 172.21.2.102:0/1004682927 wait complete. 2021-02-10T11:38:36.902 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 157, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:38:30.335525', 'modified': '2021-02-10 11:38:32.341928', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8812}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8812': {'gid': 8812, 'name': 'b', 'rank': 0, 'incarnation': 154, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.113:6833/1513291768', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.113:6832', 'nonce': 1513291768}, {'type': 'v1', 'addr': '172.21.2.113:6833', 'nonce': 1513291768}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [39], 'metadata_pool': 38, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:38:36.902 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:38:36.902 DEBUG:teuthology.orchestra.run.gibba013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:38:37.300 INFO:teuthology.orchestra.run.gibba013.stdout:{ 2021-02-10T11:38:37.300 INFO:teuthology.orchestra.run.gibba013.stdout: "cluster_fsid": "68db30ea-dbce-4aef-a6df-f01c6fea0513", 2021-02-10T11:38:37.300 INFO:teuthology.orchestra.run.gibba013.stdout: "whoami": 0, 2021-02-10T11:38:37.300 INFO:teuthology.orchestra.run.gibba013.stdout: "id": 8812, 2021-02-10T11:38:37.300 INFO:teuthology.orchestra.run.gibba013.stdout: "want_state": "up:active", 2021-02-10T11:38:37.300 INFO:teuthology.orchestra.run.gibba013.stdout: "state": "up:active", 2021-02-10T11:38:37.301 INFO:teuthology.orchestra.run.gibba013.stdout: "rank_uptime": 6.961025867, 2021-02-10T11:38:37.301 INFO:teuthology.orchestra.run.gibba013.stdout: "mdsmap_epoch": 157, 2021-02-10T11:38:37.301 INFO:teuthology.orchestra.run.gibba013.stdout: "osdmap_epoch": 180, 2021-02-10T11:38:37.301 INFO:teuthology.orchestra.run.gibba013.stdout: "osdmap_epoch_barrier": 180, 2021-02-10T11:38:37.301 INFO:teuthology.orchestra.run.gibba013.stdout: "uptime": 12.616264038000001 2021-02-10T11:38:37.301 INFO:teuthology.orchestra.run.gibba013.stdout:} 2021-02-10T11:38:37.304 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "68db30ea-dbce-4aef-a6df-f01c6fea0513", "whoami": 0, "id": 8812, "want_state": "up:active", "state": "up:active", "rank_uptime": 6.961025867, "mdsmap_epoch": 157, "osdmap_epoch": 180, "osdmap_epoch_barrier": 180, "uptime": 12.616264038000001 } 2021-02-10T11:38:37.305 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:38:37.737 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:37.718 7fad40d9f700 1 Processor -- start 2021-02-10T11:38:37.737 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:37.718 7fad40d9f700 1 -- start start 2021-02-10T11:38:37.737 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:37.718 7fad40d9f700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fad3c0d1870 0x7fad3c1b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:37.737 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:37.718 7fad40d9f700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fad3c1b0a10 0x7fad3c0cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:37.737 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:37.718 7fad40d9f700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad3c0f0b10 con 0x7fad3c0d1c70 2021-02-10T11:38:37.737 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:37.718 7fad40d9f700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad3c0fe230 con 0x7fad3c1b0a10 2021-02-10T11:38:37.737 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:37.718 7fad40d9f700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad3c0e5ed0 con 0x7fad3c0d1870 2021-02-10T11:38:37.737 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:37.719 7fad3b314700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fad3c0d1c70 0x7fad3c0cf950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:60156/0 (socket says 172.21.2.102:60156) 2021-02-10T11:38:37.737 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:37.719 7fad3b314700 1 -- 172.21.2.102:0/298453663 learned_addr learned my addr 172.21.2.102:0/298453663 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:38:37.737 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:37.719 7fad39b11700 1 -- 172.21.2.102:0/298453663 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2283335954 0 0) 0x7fad34001500 con 0x7fad3c0d1c70 2021-02-10T11:38:37.737 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:37.719 7fad39b11700 1 -- 172.21.2.102:0/298453663 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad28002fe0 con 0x7fad3c0d1c70 2021-02-10T11:38:37.738 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:37.719 7fad3a312700 1 --2- 172.21.2.102:0/298453663 >> v2:172.21.2.113:3300/0 conn(0x7fad3c1b0a10 0x7fad3c0cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:37.738 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:37.719 7fad3ab13700 1 --2- 172.21.2.102:0/298453663 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fad3c0d1870 0x7fad3c1b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:37.738 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:37.719 7fad39b11700 1 -- 172.21.2.102:0/298453663 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2322338455 0 0) 0x7fad34001460 con 0x7fad3c0d1c70 2021-02-10T11:38:37.738 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:37.719 7fad39b11700 1 -- 172.21.2.102:0/298453663 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fad3c0d1870 msgr2=0x7fad3c1b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:37.738 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:37.719 7fad39b11700 1 --2- 172.21.2.102:0/298453663 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fad3c0d1870 0x7fad3c1b03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:37.738 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:37.719 7fad39b11700 1 -- 172.21.2.102:0/298453663 >> v2:172.21.2.113:3300/0 conn(0x7fad3c1b0a10 msgr2=0x7fad3c0cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:37.738 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:37.719 7fad39b11700 1 --2- 172.21.2.102:0/298453663 >> v2:172.21.2.113:3300/0 conn(0x7fad3c1b0a10 0x7fad3c0cf330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:37.738 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:37.719 7fad39b11700 1 -- 172.21.2.102:0/298453663 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad3c0f0650 con 0x7fad3c0d1c70 2021-02-10T11:38:37.738 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:37.719 7fad39b11700 1 -- 172.21.2.102:0/298453663 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7fad34001980 con 0x7fad3c0d1c70 2021-02-10T11:38:37.738 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:37.719 7fad39b11700 1 -- 172.21.2.102:0/298453663 >> v1:172.21.2.102:6789/0 conn(0x7fad3c0d1c70 legacy=0x7fad3c0cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:37.738 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:37.719 7fad39b11700 1 --2- 172.21.2.102:0/298453663 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fad28003ac0 0x7fad28005ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:37.738 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:37.720 7fad39b11700 1 --2- 172.21.2.102:0/298453663 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fad2804a760 0x7fad2804cc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:37.739 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:37.720 7fad39b11700 1 --2- 172.21.2.102:0/298453663 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fad2804d520 0x7fad2804fa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:37.739 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:37.720 7fad39b11700 1 -- 172.21.2.102:0/298453663 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad280502f0 con 0x7fad2804a760 2021-02-10T11:38:37.739 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:37.720 7fad39b11700 1 -- 172.21.2.102:0/298453663 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad28050510 con 0x7fad2804d520 2021-02-10T11:38:37.739 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:37.720 7fad39b11700 1 -- 172.21.2.102:0/298453663 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad28050730 con 0x7fad28003ac0 2021-02-10T11:38:37.739 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:37.720 7fad3a312700 1 --2- 172.21.2.102:0/298453663 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fad2804a760 0x7fad2804cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:37.739 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:37.720 7fad3ab13700 1 --2- 172.21.2.102:0/298453663 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fad28003ac0 0x7fad28005ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:37.739 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:37.720 7fad3b314700 1 --2- 172.21.2.102:0/298453663 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fad2804d520 0x7fad2804fa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:37.739 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:37.720 7fad3a312700 1 -- 172.21.2.102:0/298453663 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fad28003ac0 msgr2=0x7fad28005ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:37.739 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:37.720 7fad3a312700 1 --2- 172.21.2.102:0/298453663 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fad28003ac0 0x7fad28005ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:37.739 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:37.720 7fad3a312700 1 -- 172.21.2.102:0/298453663 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fad2804d520 msgr2=0x7fad2804fa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:37.739 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:37.720 7fad3a312700 1 --2- 172.21.2.102:0/298453663 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fad2804d520 0x7fad2804fa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:37.739 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:37.720 7fad3a312700 1 -- 172.21.2.102:0/298453663 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad280509d0 con 0x7fad2804a760 2021-02-10T11:38:37.739 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:37.720 7fad3a312700 1 --2- 172.21.2.102:0/298453663 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fad2804a760 0x7fad2804cc80 secure :-1 s=READY pgs=3119 cs=0 l=1 rev1=1 rx=0x7fad2400c870 tx=0x7fad2400cc10).ready entity=mon.0 client_cookie=fa48eedeab54fc54 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:37.739 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:37.720 7fad39b11700 1 -- 172.21.2.102:0/298453663 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fad240090d0 con 0x7fad2804a760 2021-02-10T11:38:37.740 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:37.720 7fad39b11700 1 -- 172.21.2.102:0/298453663 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad24009310 con 0x7fad2804a760 2021-02-10T11:38:37.740 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:37.720 7fad39b11700 1 -- 172.21.2.102:0/298453663 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fad24009760 con 0x7fad2804a760 2021-02-10T11:38:37.740 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:37.721 7fad40d9f700 1 -- 172.21.2.102:0/298453663 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fad2804a760 msgr2=0x7fad2804cc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:37.740 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:37.721 7fad40d9f700 1 --2- 172.21.2.102:0/298453663 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fad2804a760 0x7fad2804cc80 secure :-1 s=READY pgs=3119 cs=0 l=1 rev1=1 rx=0x7fad2400c870 tx=0x7fad2400cc10).stop 2021-02-10T11:38:37.740 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:37.721 7fad3b314700 1 -- 172.21.2.102:0/298453663 reap_dead start 2021-02-10T11:38:37.740 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:37.721 7fad40d9f700 1 -- 172.21.2.102:0/298453663 shutdown_connections 2021-02-10T11:38:37.740 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:37.721 7fad40d9f700 1 -- 172.21.2.102:0/298453663 >> 172.21.2.102:0/298453663 conn(0x7fad3c0ce450 msgr2=0x7fad3c0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:37.740 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:37.721 7fad40d9f700 1 -- 172.21.2.102:0/298453663 shutdown_connections 2021-02-10T11:38:37.740 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:37.721 7fad40d9f700 1 -- 172.21.2.102:0/298453663 wait complete. 2021-02-10T11:38:37.741 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:37.736 7fad40d9f700 1 Processor -- start 2021-02-10T11:38:37.742 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:37.736 7fad40d9f700 1 -- start start 2021-02-10T11:38:37.742 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:37.736 7fad40d9f700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fad3c0b25d0 0x7fad3c0a8d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:37.742 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:37.736 7fad40d9f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fad3c0a9340 0x7fad3c0a63b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:37.742 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:37.736 7fad40d9f700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fad3c0a69d0 0x7fad3c0a6dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:37.742 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:37.736 7fad40d9f700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad3c0b29d0 con 0x7fad3c0a9340 2021-02-10T11:38:37.742 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:37.736 7fad40d9f700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad3c0a73f0 con 0x7fad3c0a69d0 2021-02-10T11:38:37.742 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:37.736 7fad40d9f700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad3c0a7870 con 0x7fad3c0b25d0 2021-02-10T11:38:37.742 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:37.737 7fad3a312700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fad3c0a9340 0x7fad3c0a63b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:37.742 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:37.737 7fad3a312700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fad3c0a9340 0x7fad3c0a63b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:39710/0 (socket says 172.21.2.102:39710) 2021-02-10T11:38:37.743 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:37.737 7fad3a312700 1 -- 172.21.2.102:0/1364402988 learned_addr learned my addr 172.21.2.102:0/1364402988 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:38:37.743 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:37.737 7fad3a312700 1 -- 172.21.2.102:0/1364402988 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fad3c0b25d0 msgr2=0x7fad3c0a8d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:37.743 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:37.737 7fad3a312700 1 --2- 172.21.2.102:0/1364402988 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fad3c0b25d0 0x7fad3c0a8d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:37.743 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:37.737 7fad3a312700 1 -- 172.21.2.102:0/1364402988 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fad3c0a69d0 msgr2=0x7fad3c0a6dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:37.743 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:37.737 7fad3a312700 1 --2- 172.21.2.102:0/1364402988 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fad3c0a69d0 0x7fad3c0a6dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:37.743 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:37.737 7fad3a312700 1 -- 172.21.2.102:0/1364402988 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad3c0a7a90 con 0x7fad3c0a9340 2021-02-10T11:38:37.743 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:37.737 7fad3a312700 1 --2- 172.21.2.102:0/1364402988 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fad3c0a9340 0x7fad3c0a63b0 secure :-1 s=READY pgs=3120 cs=0 l=1 rev1=1 rx=0x7fad2400b170 tx=0x7fad2400b510).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:37.743 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:37.737 7fad337fe700 1 -- 172.21.2.102:0/1364402988 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fad2400b900 con 0x7fad3c0a9340 2021-02-10T11:38:37.743 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:37.737 7fad337fe700 1 -- 172.21.2.102:0/1364402988 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad2400bbc0 con 0x7fad3c0a9340 2021-02-10T11:38:37.743 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:37.737 7fad337fe700 1 -- 172.21.2.102:0/1364402988 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fad2400c730 con 0x7fad3c0a9340 2021-02-10T11:38:37.744 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:37.737 7fad40d9f700 1 -- 172.21.2.102:0/1364402988 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad3c0a7a90 con 0x7fad3c0a9340 2021-02-10T11:38:37.744 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:37.737 7fad40d9f700 1 -- 172.21.2.102:0/1364402988 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad3c0a8020 con 0x7fad3c0a9340 2021-02-10T11:38:37.744 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:37.738 7fad337fe700 1 -- 172.21.2.102:0/1364402988 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fad24002020 con 0x7fad3c0a9340 2021-02-10T11:38:37.744 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:37.738 7fad337fe700 1 --2- 172.21.2.102:0/1364402988 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fad200186c0 0x7fad2001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:37.744 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:37.738 7fad337fe700 1 -- 172.21.2.102:0/1364402988 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(180..180 src has 1..180) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fad240325a0 con 0x7fad3c0a9340 2021-02-10T11:38:37.744 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:37.738 7fad3ab13700 1 --2- 172.21.2.102:0/1364402988 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fad200186c0 0x7fad2001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:37.744 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:37.738 7fad3ab13700 1 --2- 172.21.2.102:0/1364402988 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fad200186c0 0x7fad2001abc0 crc :-1 s=READY pgs=1369 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:37.769 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:37.768 7fad40d9f700 1 -- 172.21.2.102:0/1364402988 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad3c0ebc80 con 0x7fad3c0a9340 2021-02-10T11:38:37.771 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:37.770 7fad337fe700 1 -- 172.21.2.102:0/1364402988 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fad240069a0 con 0x7fad3c0a9340 2021-02-10T11:38:37.898 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:37.896 7fad40d9f700 1 -- 172.21.2.102:0/1364402988 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fad3c0f0b10 con 0x7fad3c0a9340 2021-02-10T11:38:37.899 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:37.897 7fad337fe700 1 -- 172.21.2.102:0/1364402988 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v180) v1 ==== 74+0+14007 (secure 0 0 0) 0x7fad24001fe0 con 0x7fad3c0a9340 2021-02-10T11:38:37.899 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:38:37.899 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":180,"fsid":"68db30ea-dbce-4aef-a6df-f01c6fea0513","created":"2021-02-10 11:03:06.331557","modified":"2021-02-10 11:38:31.337750","last_up_change":"2021-02-10 11:03:32.690562","last_in_change":"2021-02-10 11:03:32.690562","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":39,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:03:37.322519","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":38,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:38:28.298788","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"179","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":39,"pool_name":"cephfs_data","create_time":"2021-02-10 11:38:29.326316","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"180","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"be691b4d-a039-408a-af80-9d53f02b6f1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":178,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6818","nonce":36088},{"type":"v1","addr":"172.21.2.102:6820","nonce":36088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6822","nonce":36088},{"type":"v1","addr":"172.21.2.102:6824","nonce":36088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6830","nonce":36088},{"type":"v1","addr":"172.21.2.102:6832","nonce":36088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6826","nonce":36088},{"type":"v1","addr":"172.21.2.102:6828","nonce":36088}]},"public_addr":"172.21.2.102:6820/36088","cluster_addr":"172.21.2.102:6824/36088","heartbeat_back_addr":"172.21.2.102:6832/36088","heartbeat_front_addr":"172.21.2.102:6828/36088","state":["exists","up"]},{"osd":1,"uuid":"64e3ccc8-9c69-4d7c-865e-af8e83822635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":178,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6819","nonce":36087},{"type":"v1","addr":"172.21.2.102:6821","nonce":36087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6823","nonce":36087},{"type":"v1","addr":"172.21.2.102:6825","nonce":36087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6831","nonce":36087},{"type":"v1","addr":"172.21.2.102:6833","nonce":36087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6827","nonce":36087},{"type":"v1","addr":"172.21.2.102:6829","nonce":36087}]},"public_addr":"172.21.2.102:6821/36087","cluster_addr":"172.21.2.102:6825/36087","heartbeat_back_addr":"172.21.2.102:6833/36087","heartbeat_front_addr":"172.21.2.102:6829/36087","state":["exists","up"]},{"osd":2,"uuid":"153700af-747f-481f-bdf4-d6072d3a842f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":178,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6810","nonce":36089},{"type":"v1","addr":"172.21.2.102:6811","nonce":36089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6812","nonce":36089},{"type":"v1","addr":"172.21.2.102:6813","nonce":36089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6816","nonce":36089},{"type":"v1","addr":"172.21.2.102:6817","nonce":36089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6814","nonce":36089},{"type":"v1","addr":"172.21.2.102:6815","nonce":36089}]},"public_addr":"172.21.2.102:6811/36089","cluster_addr":"172.21.2.102:6813/36089","heartbeat_back_addr":"172.21.2.102:6817/36089","heartbeat_front_addr":"172.21.2.102:6815/36089","state":["exists","up"]},{"osd":3,"uuid":"cbc678f4-83a9-45e6-8830-2f7c232b1ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":178,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6802","nonce":36090},{"type":"v1","addr":"172.21.2.102:6803","nonce":36090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6804","nonce":36090},{"type":"v1","addr":"172.21.2.102:6805","nonce":36090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6808","nonce":36090},{"type":"v1","addr":"172.21.2.102:6809","nonce":36090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6806","nonce":36090},{"type":"v1","addr":"172.21.2.102:6807","nonce":36090}]},"public_addr":"172.21.2.102:6803/36090","cluster_addr":"172.21.2.102:6805/36090","heartbeat_back_addr":"172.21.2.102:6809/36090","heartbeat_front_addr":"172.21.2.102:6807/36090","state":["exists","up"]},{"osd":4,"uuid":"c01e4008-b258-4f10-afd2-9e7f37214351","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":178,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6808","nonce":36368},{"type":"v1","addr":"172.21.2.113:6811","nonce":36368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6814","nonce":36368},{"type":"v1","addr":"172.21.2.113:6818","nonce":36368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6826","nonce":36368},{"type":"v1","addr":"172.21.2.113:6828","nonce":36368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6820","nonce":36368},{"type":"v1","addr":"172.21.2.113:6824","nonce":36368}]},"public_addr":"172.21.2.113:6811/36368","cluster_addr":"172.21.2.113:6818/36368","heartbeat_back_addr":"172.21.2.113:6828/36368","heartbeat_front_addr":"172.21.2.113:6824/36368","state":["exists","up"]},{"osd":5,"uuid":"551f6007-daf8-4a98-8ee0-c2b16bcccb5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":178,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6802","nonce":36366},{"type":"v1","addr":"172.21.2.113:6805","nonce":36366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6807","nonce":36366},{"type":"v1","addr":"172.21.2.113:6810","nonce":36366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6819","nonce":36366},{"type":"v1","addr":"172.21.2.113:6822","nonce":36366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6812","nonce":36366},{"type":"v1","addr":"172.21.2.113:6815","nonce":36366}]},"public_addr":"172.21.2.113:6805/36366","cluster_addr":"172.21.2.113:6810/36366","heartbeat_back_addr":"172.21.2.113:6822/36366","heartbeat_front_addr":"172.21.2.113:6815/36366","state":["exists","up"]},{"osd":6,"uuid":"bf67c040-0e50-48be-b102-c3943bc9ebef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6800","nonce":36367},{"type":"v1","addr":"172.21.2.113:6801","nonce":36367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6803","nonce":36367},{"type":"v1","addr":"172.21.2.113:6804","nonce":36367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6813","nonce":36367},{"type":"v1","addr":"172.21.2.113:6816","nonce":36367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6806","nonce":36367},{"type":"v1","addr":"172.21.2.113:6809","nonce":36367}]},"public_addr":"172.21.2.113:6801/36367","cluster_addr":"172.21.2.113:6804/36367","heartbeat_back_addr":"172.21.2.113:6816/36367","heartbeat_front_addr":"172.21.2.113:6809/36367","state":["exists","up"]},{"osd":7,"uuid":"f5461f80-018d-428f-be89-2eb2bcbe7bb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":178,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6817","nonce":36365},{"type":"v1","addr":"172.21.2.113:6821","nonce":36365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6823","nonce":36365},{"type":"v1","addr":"172.21.2.113:6825","nonce":36365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6830","nonce":36365},{"type":"v1","addr":"172.21.2.113:6831","nonce":36365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6827","nonce":36365},{"type":"v1","addr":"172.21.2.113:6829","nonce":36365}]},"public_addr":"172.21.2.113:6821/36365","cluster_addr":"172.21.2.113:6825/36365","heartbeat_back_addr":"172.21.2.113:6831/36365","heartbeat_front_addr":"172.21.2.113:6829/36365","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:38:37.900 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:37.899 7fad40d9f700 1 -- 172.21.2.102:0/1364402988 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fad200186c0 msgr2=0x7fad2001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:37.901 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:37.899 7fad40d9f700 1 --2- 172.21.2.102:0/1364402988 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fad200186c0 0x7fad2001abc0 crc :-1 s=READY pgs=1369 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:37.901 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:37.899 7fad40d9f700 1 -- 172.21.2.102:0/1364402988 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fad3c0a9340 msgr2=0x7fad3c0a63b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:37.901 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:37.899 7fad40d9f700 1 --2- 172.21.2.102:0/1364402988 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fad3c0a9340 0x7fad3c0a63b0 secure :-1 s=READY pgs=3120 cs=0 l=1 rev1=1 rx=0x7fad2400b170 tx=0x7fad2400b510).stop 2021-02-10T11:38:37.901 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:37.899 7fad40d9f700 1 -- 172.21.2.102:0/1364402988 shutdown_connections 2021-02-10T11:38:37.901 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:37.899 7fad40d9f700 1 --2- 172.21.2.102:0/1364402988 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fad200186c0 0x7fad2001abc0 unknown :-1 s=CLOSED pgs=1369 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:37.901 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:37.899 7fad40d9f700 1 --2- 172.21.2.102:0/1364402988 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fad3c0a69d0 0x7fad3c0a6dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:37.902 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:37.899 7fad40d9f700 1 --2- 172.21.2.102:0/1364402988 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fad3c0a9340 0x7fad3c0a63b0 unknown :-1 s=CLOSED pgs=3120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:37.902 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:37.899 7fad40d9f700 1 --2- 172.21.2.102:0/1364402988 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fad3c0b25d0 0x7fad3c0a8d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:37.902 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:37.899 7fad40d9f700 1 -- 172.21.2.102:0/1364402988 >> 172.21.2.102:0/1364402988 conn(0x7fad3c0bbb50 msgr2=0x7fad3c0ad8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:37.902 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:37.899 7fad40d9f700 1 -- 172.21.2.102:0/1364402988 shutdown_connections 2021-02-10T11:38:37.902 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:37.899 7fad40d9f700 1 -- 172.21.2.102:0/1364402988 wait complete. 2021-02-10T11:38:37.909 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:03:37.322519', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 38, 'pool_name': 'cephfs_metadata', 'create_time': '2021-02-10 11:38:28.298788', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '179', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 39, 'pool_name': 'cephfs_data', 'create_time': '2021-02-10 11:38:29.326316', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '180', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-02-10T11:38:37.910 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:38:38.345 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:38.327 7fe124d02700 1 Processor -- start 2021-02-10T11:38:38.345 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:38.327 7fe124d02700 1 -- start start 2021-02-10T11:38:38.345 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:38.327 7fe124d02700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe1200d1c70 0x7fe1200cef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:38.345 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:38.327 7fe124d02700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1200d0170 con 0x7fe1200d1870 2021-02-10T11:38:38.345 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:38.327 7fe124d02700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1200fe1c0 con 0x7fe1200cf580 2021-02-10T11:38:38.346 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:38.327 7fe124d02700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1201b0cb0 con 0x7fe1200d1c70 2021-02-10T11:38:38.346 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:38.327 7fe11eb13700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fe1200d1870 0x7fe1201b04c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:60172/0 (socket says 172.21.2.102:60172) 2021-02-10T11:38:38.346 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:38.327 7fe11eb13700 1 -- 172.21.2.102:0/754594745 learned_addr learned my addr 172.21.2.102:0/754594745 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:38:38.346 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:38.328 7fe11db11700 1 -- 172.21.2.102:0/754594745 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3426417047 0 0) 0x7fe108001180 con 0x7fe1200d1870 2021-02-10T11:38:38.346 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:38.328 7fe11db11700 1 -- 172.21.2.102:0/754594745 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe10c002fe0 con 0x7fe1200d1870 2021-02-10T11:38:38.346 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:38.328 7fe11e312700 1 --2- 172.21.2.102:0/754594745 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe1200d1c70 0x7fe1200cef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:38.346 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:38.328 7fe11db11700 1 -- 172.21.2.102:0/754594745 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3365060751 0 0) 0x7fe108001180 con 0x7fe1200d1870 2021-02-10T11:38:38.346 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:38.328 7fe11db11700 1 -- 172.21.2.102:0/754594745 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe1200d1c70 msgr2=0x7fe1200cef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:38.346 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:38.328 7fe11db11700 1 --2- 172.21.2.102:0/754594745 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe1200d1c70 0x7fe1200cef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:38.347 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:38.328 7fe11db11700 1 -- 172.21.2.102:0/754594745 >> v1:172.21.2.113:6789/0 conn(0x7fe1200cf580 legacy=0x7fe1200cf980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:38.347 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:38.328 7fe11db11700 1 -- 172.21.2.102:0/754594745 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe120109a60 con 0x7fe1200d1870 2021-02-10T11:38:38.347 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:38.328 7fe11db11700 1 -- 172.21.2.102:0/754594745 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7fe108001680 con 0x7fe1200d1870 2021-02-10T11:38:38.347 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:38.328 7fe11db11700 1 -- 172.21.2.102:0/754594745 >> v1:172.21.2.102:6789/0 conn(0x7fe1200d1870 legacy=0x7fe1201b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:38.347 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:38.328 7fe11db11700 1 --2- 172.21.2.102:0/754594745 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe10c003ac0 0x7fe10c005f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:38.347 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:38.328 7fe11db11700 1 --2- 172.21.2.102:0/754594745 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe10c006800 0x7fe10c008cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:38.347 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:38.328 7fe11db11700 1 --2- 172.21.2.102:0/754594745 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe10c009590 0x7fe10c00bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:38.347 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:38.328 7fe11db11700 1 -- 172.21.2.102:0/754594745 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe10c00c2e0 con 0x7fe10c003ac0 2021-02-10T11:38:38.347 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:38.328 7fe11db11700 1 -- 172.21.2.102:0/754594745 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe10c00c500 con 0x7fe10c009590 2021-02-10T11:38:38.347 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:38.328 7fe11db11700 1 -- 172.21.2.102:0/754594745 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe10c00c720 con 0x7fe10c006800 2021-02-10T11:38:38.347 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:38.328 7fe11eb13700 1 --2- 172.21.2.102:0/754594745 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe10c003ac0 0x7fe10c005f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:38.348 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:38.328 7fe11f314700 1 --2- 172.21.2.102:0/754594745 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe10c009590 0x7fe10c00bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:38.348 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:38.328 7fe11eb13700 1 -- 172.21.2.102:0/754594745 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe10c006800 msgr2=0x7fe10c008cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:38.348 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:38.328 7fe11eb13700 1 --2- 172.21.2.102:0/754594745 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe10c006800 0x7fe10c008cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:38.348 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:38.328 7fe11eb13700 1 -- 172.21.2.102:0/754594745 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe10c009590 msgr2=0x7fe10c00bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:38.348 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:38.328 7fe11eb13700 1 --2- 172.21.2.102:0/754594745 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe10c009590 0x7fe10c00bab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:38.348 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:38.328 7fe11eb13700 1 -- 172.21.2.102:0/754594745 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe10c050990 con 0x7fe10c003ac0 2021-02-10T11:38:38.348 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:38.329 7fe11eb13700 1 --2- 172.21.2.102:0/754594745 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe10c003ac0 0x7fe10c005f80 secure :-1 s=READY pgs=3122 cs=0 l=1 rev1=1 rx=0x7fe1080078f0 tx=0x7fe108007c90).ready entity=mon.0 client_cookie=e7630921905087aa server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:38.348 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:38.329 7fe11db11700 1 -- 172.21.2.102:0/754594745 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe1080070c0 con 0x7fe10c003ac0 2021-02-10T11:38:38.348 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:38.329 7fe11db11700 1 -- 172.21.2.102:0/754594745 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1080043a0 con 0x7fe10c003ac0 2021-02-10T11:38:38.348 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:38.329 7fe11db11700 1 -- 172.21.2.102:0/754594745 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe1080048c0 con 0x7fe10c003ac0 2021-02-10T11:38:38.348 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:38.329 7fe124d02700 1 -- 172.21.2.102:0/754594745 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe10c003ac0 msgr2=0x7fe10c005f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:38.348 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:38.329 7fe124d02700 1 --2- 172.21.2.102:0/754594745 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe10c003ac0 0x7fe10c005f80 secure :-1 s=READY pgs=3122 cs=0 l=1 rev1=1 rx=0x7fe1080078f0 tx=0x7fe108007c90).stop 2021-02-10T11:38:38.348 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:38.329 7fe11f314700 1 -- 172.21.2.102:0/754594745 reap_dead start 2021-02-10T11:38:38.349 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:38.330 7fe124d02700 1 -- 172.21.2.102:0/754594745 shutdown_connections 2021-02-10T11:38:38.349 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:38.330 7fe124d02700 1 -- 172.21.2.102:0/754594745 >> 172.21.2.102:0/754594745 conn(0x7fe1200ce420 msgr2=0x7fe1200d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:38.349 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:38.330 7fe124d02700 1 -- 172.21.2.102:0/754594745 shutdown_connections 2021-02-10T11:38:38.349 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:38.330 7fe124d02700 1 -- 172.21.2.102:0/754594745 wait complete. 2021-02-10T11:38:38.350 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:38.344 7fe124d02700 1 Processor -- start 2021-02-10T11:38:38.351 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:38.344 7fe124d02700 1 -- start start 2021-02-10T11:38:38.351 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:38.345 7fe124d02700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe12009bcc0 0x7fe12009c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:38.351 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:38.345 7fe124d02700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe1200a0820 0x7fe1200a0c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:38.351 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:38.345 7fe124d02700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe1200a1240 0x7fe1200858d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:38.351 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:38.345 7fe124d02700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1200a66b0 con 0x7fe1200a1240 2021-02-10T11:38:38.351 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:38.345 7fe124d02700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1201a49b0 con 0x7fe1200a0820 2021-02-10T11:38:38.351 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:38.345 7fe124d02700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1201a4b90 con 0x7fe12009bcc0 2021-02-10T11:38:38.351 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:38.345 7fe11f314700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe1200a1240 0x7fe1200858d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:38.351 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:38.345 7fe11f314700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe1200a1240 0x7fe1200858d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:39732/0 (socket says 172.21.2.102:39732) 2021-02-10T11:38:38.351 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:38.345 7fe11f314700 1 -- 172.21.2.102:0/2481606279 learned_addr learned my addr 172.21.2.102:0/2481606279 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:38:38.351 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:38.345 7fe11eb13700 1 --2- 172.21.2.102:0/2481606279 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe12009bcc0 0x7fe12009c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:38.351 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:38.345 7fe11e312700 1 --2- 172.21.2.102:0/2481606279 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe1200a0820 0x7fe1200a0c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:38.352 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:38.345 7fe11f314700 1 -- 172.21.2.102:0/2481606279 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe12009bcc0 msgr2=0x7fe12009c0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:38.352 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:38.345 7fe11f314700 1 --2- 172.21.2.102:0/2481606279 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe12009bcc0 0x7fe12009c0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:38.352 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:38.345 7fe11f314700 1 -- 172.21.2.102:0/2481606279 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe1200a0820 msgr2=0x7fe1200a0c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:38.352 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:38.345 7fe11f314700 1 --2- 172.21.2.102:0/2481606279 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe1200a0820 0x7fe1200a0c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:38.352 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:38.345 7fe11f314700 1 -- 172.21.2.102:0/2481606279 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1200a1640 con 0x7fe1200a1240 2021-02-10T11:38:38.352 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:38.346 7fe11f314700 1 --2- 172.21.2.102:0/2481606279 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe1200a1240 0x7fe1200858d0 secure :-1 s=READY pgs=3123 cs=0 l=1 rev1=1 rx=0x7fe118006900 tx=0x7fe11800bca0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:38.352 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:38.346 7fe1177fe700 1 -- 172.21.2.102:0/2481606279 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe118006ba0 con 0x7fe1200a1240 2021-02-10T11:38:38.352 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:38.346 7fe1177fe700 1 -- 172.21.2.102:0/2481606279 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe118006030 con 0x7fe1200a1240 2021-02-10T11:38:38.352 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:38.346 7fe1177fe700 1 -- 172.21.2.102:0/2481606279 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe118006500 con 0x7fe1200a1240 2021-02-10T11:38:38.352 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:38.346 7fe124d02700 1 -- 172.21.2.102:0/2481606279 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1200a1640 con 0x7fe1200a1240 2021-02-10T11:38:38.352 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:38.346 7fe124d02700 1 -- 172.21.2.102:0/2481606279 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1200861e0 con 0x7fe1200a1240 2021-02-10T11:38:38.352 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:38.347 7fe1177fe700 1 -- 172.21.2.102:0/2481606279 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fe118006210 con 0x7fe1200a1240 2021-02-10T11:38:38.352 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:38.347 7fe1177fe700 1 --2- 172.21.2.102:0/2481606279 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe104018730 0x7fe10401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:38.353 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:38.347 7fe1177fe700 1 -- 172.21.2.102:0/2481606279 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(180..180 src has 1..180) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fe118032b70 con 0x7fe1200a1240 2021-02-10T11:38:38.353 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:38.347 7fe11eb13700 1 --2- 172.21.2.102:0/2481606279 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe104018730 0x7fe10401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:38.353 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:38.347 7fe11eb13700 1 --2- 172.21.2.102:0/2481606279 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe104018730 0x7fe10401abc0 crc :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:38.378 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:38.377 7fe124d02700 1 -- 172.21.2.102:0/2481606279 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1200fe1c0 con 0x7fe1200a1240 2021-02-10T11:38:38.380 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:38.379 7fe1177fe700 1 -- 172.21.2.102:0/2481606279 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fe118006520 con 0x7fe1200a1240 2021-02-10T11:38:38.509 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:38.508 7fe124d02700 1 -- 172.21.2.102:0/2481606279 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7fe1200fe1c0 con 0x7fe1200a1240 2021-02-10T11:38:38.510 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:38.509 7fe1177fe700 1 -- 172.21.2.102:0/2481606279 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v180) v1 ==== 92+0+10 (secure 0 0 0) 0x7fe118007f00 con 0x7fe1200a1240 2021-02-10T11:38:38.510 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:38:38.512 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:38.510 7fe124d02700 1 -- 172.21.2.102:0/2481606279 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe104018730 msgr2=0x7fe10401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:38.512 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:38.510 7fe124d02700 1 --2- 172.21.2.102:0/2481606279 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe104018730 0x7fe10401abc0 crc :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:38.512 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:38.510 7fe124d02700 1 -- 172.21.2.102:0/2481606279 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe1200a1240 msgr2=0x7fe1200858d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:38.512 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:38.510 7fe124d02700 1 --2- 172.21.2.102:0/2481606279 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe1200a1240 0x7fe1200858d0 secure :-1 s=READY pgs=3123 cs=0 l=1 rev1=1 rx=0x7fe118006900 tx=0x7fe11800bca0).stop 2021-02-10T11:38:38.512 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:38.511 7fe124d02700 1 -- 172.21.2.102:0/2481606279 shutdown_connections 2021-02-10T11:38:38.512 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:38.511 7fe124d02700 1 --2- 172.21.2.102:0/2481606279 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe1200a1240 0x7fe1200858d0 unknown :-1 s=CLOSED pgs=3123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:38.512 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:38.511 7fe124d02700 1 --2- 172.21.2.102:0/2481606279 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe104018730 0x7fe10401abc0 unknown :-1 s=CLOSED pgs=1370 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:38.513 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:38.511 7fe124d02700 1 --2- 172.21.2.102:0/2481606279 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe1200a0820 0x7fe1200a0c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:38.513 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:38.511 7fe124d02700 1 --2- 172.21.2.102:0/2481606279 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe12009bcc0 0x7fe12009c0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:38.513 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:38.511 7fe124d02700 1 -- 172.21.2.102:0/2481606279 >> 172.21.2.102:0/2481606279 conn(0x7fe1200bef00 msgr2=0x7fe12009d1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:38.513 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:38.511 7fe124d02700 1 -- 172.21.2.102:0/2481606279 shutdown_connections 2021-02-10T11:38:38.513 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:38.511 7fe124d02700 1 -- 172.21.2.102:0/2481606279 wait complete. 2021-02-10T11:38:38.520 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-02-10T11:38:38.953 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:38.935 7fcac8894700 1 Processor -- start 2021-02-10T11:38:38.953 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:38.935 7fcac8894700 1 -- start start 2021-02-10T11:38:38.953 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:38.935 7fcac8894700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fcac40d1870 0x7fcac41b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:38.953 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:38.935 7fcac8894700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fcac41b0ae0 0x7fcac4108940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:38.953 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:38.935 7fcac8894700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcac40d1c70 0x7fcac4108f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:38.953 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:38.935 7fcac8894700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcac40fe1c0 con 0x7fcac40d1870 2021-02-10T11:38:38.953 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:38.935 7fcac8894700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcac4109a60 con 0x7fcac41b0ae0 2021-02-10T11:38:38.953 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:38.935 7fcac8894700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcac4088070 con 0x7fcac40d1c70 2021-02-10T11:38:38.953 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:38.936 7fcac261c700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fcac40d1870 0x7fcac41b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:38.954 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:38.936 7fcac261c700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fcac40d1870 0x7fcac41b04c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:39736/0 (socket says 172.21.2.102:39736) 2021-02-10T11:38:38.954 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:38.936 7fcac261c700 1 -- 172.21.2.102:0/1829062033 learned_addr learned my addr 172.21.2.102:0/1829062033 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:38:38.954 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:38.936 7fcac261c700 1 -- 172.21.2.102:0/1829062033 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcac40d1c70 msgr2=0x7fcac4108f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:38.954 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:38.936 7fcac261c700 1 --2- 172.21.2.102:0/1829062033 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcac40d1c70 0x7fcac4108f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:38.954 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:38.936 7fcac261c700 1 -- 172.21.2.102:0/1829062033 >> v2:172.21.2.113:3300/0 conn(0x7fcac41b0ae0 msgr2=0x7fcac4108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:38.954 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:38.936 7fcac261c700 1 --2- 172.21.2.102:0/1829062033 >> v2:172.21.2.113:3300/0 conn(0x7fcac41b0ae0 0x7fcac4108940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:38.954 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:38.936 7fcac261c700 1 -- 172.21.2.102:0/1829062033 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcac4109580 con 0x7fcac40d1870 2021-02-10T11:38:38.954 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:38.936 7fcac261c700 1 --2- 172.21.2.102:0/1829062033 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcac40d1870 0x7fcac41b04c0 secure :-1 s=READY pgs=3124 cs=0 l=1 rev1=1 rx=0x7fcaac006700 tx=0x7fcaac009b80).ready entity=mon.0 client_cookie=3e7e929d559dbd6e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:38.954 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:38.937 7fcac161a700 1 -- 172.21.2.102:0/1829062033 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcaac00c0c0 con 0x7fcac40d1870 2021-02-10T11:38:38.954 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:38.937 7fcac161a700 1 -- 172.21.2.102:0/1829062033 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcaac00c400 con 0x7fcac40d1870 2021-02-10T11:38:38.954 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:38.937 7fcac161a700 1 -- 172.21.2.102:0/1829062033 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcaac00c920 con 0x7fcac40d1870 2021-02-10T11:38:38.954 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:38.937 7fcac8894700 1 -- 172.21.2.102:0/1829062033 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcac40d1870 msgr2=0x7fcac41b04c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:38.954 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:38.937 7fcac8894700 1 --2- 172.21.2.102:0/1829062033 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcac40d1870 0x7fcac41b04c0 secure :-1 s=READY pgs=3124 cs=0 l=1 rev1=1 rx=0x7fcaac006700 tx=0x7fcaac009b80).stop 2021-02-10T11:38:38.955 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:38.937 7fcac8894700 1 -- 172.21.2.102:0/1829062033 shutdown_connections 2021-02-10T11:38:38.955 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:38.937 7fcac8894700 1 --2- 172.21.2.102:0/1829062033 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcac40d1c70 0x7fcac4108f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:38.955 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:38.937 7fcac8894700 1 --2- 172.21.2.102:0/1829062033 >> v2:172.21.2.113:3300/0 conn(0x7fcac41b0ae0 0x7fcac4108940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:38.955 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:38.937 7fcac8894700 1 --2- 172.21.2.102:0/1829062033 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcac40d1870 0x7fcac41b04c0 unknown :-1 s=CLOSED pgs=3124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:38.955 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:38.937 7fcac8894700 1 -- 172.21.2.102:0/1829062033 >> 172.21.2.102:0/1829062033 conn(0x7fcac40ce420 msgr2=0x7fcac40d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:38.955 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:38.938 7fcac8894700 1 -- 172.21.2.102:0/1829062033 shutdown_connections 2021-02-10T11:38:38.955 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:38.938 7fcac8894700 1 -- 172.21.2.102:0/1829062033 wait complete. 2021-02-10T11:38:38.955 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:38.952 7fcac8894700 1 Processor -- start 2021-02-10T11:38:38.955 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:38.952 7fcac8894700 1 -- start start 2021-02-10T11:38:38.957 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:38.952 7fcac8894700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fcac40c3820 0x7fcac4084cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:38.957 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:38.952 7fcac8894700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcac40a0290 0x7fcac40852d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:38.957 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:38.952 7fcac8894700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcac4086270 0x7fcac4086670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:38.957 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:38.952 7fcac8894700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcac40c0700 con 0x7fcac40a0290 2021-02-10T11:38:38.957 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:38.952 7fcac8894700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcac40858f0 con 0x7fcac40c3820 2021-02-10T11:38:38.957 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:38.952 7fcac8894700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcac4085b10 con 0x7fcac4086270 2021-02-10T11:38:38.957 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:38.952 7fcac1e1b700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcac40a0290 0x7fcac40852d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:38.957 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:38.952 7fcac1e1b700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcac40a0290 0x7fcac40852d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:39744/0 (socket says 172.21.2.102:39744) 2021-02-10T11:38:38.957 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:38.952 7fcac1e1b700 1 -- 172.21.2.102:0/3109575554 learned_addr learned my addr 172.21.2.102:0/3109575554 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:38:38.957 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:38.953 7fcac1e1b700 1 -- 172.21.2.102:0/3109575554 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcac4086270 msgr2=0x7fcac4086670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:38.957 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:38.953 7fcac1e1b700 1 --2- 172.21.2.102:0/3109575554 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcac4086270 0x7fcac4086670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:38.957 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:38.953 7fcac1e1b700 1 -- 172.21.2.102:0/3109575554 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fcac40c3820 msgr2=0x7fcac4084cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:38.957 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:38.953 7fcac1e1b700 1 --2- 172.21.2.102:0/3109575554 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fcac40c3820 0x7fcac4084cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:38.958 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:38.953 7fcac1e1b700 1 -- 172.21.2.102:0/3109575554 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcac4086c90 con 0x7fcac40a0290 2021-02-10T11:38:38.958 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:38.953 7fcac1e1b700 1 --2- 172.21.2.102:0/3109575554 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcac40a0290 0x7fcac40852d0 secure :-1 s=READY pgs=3125 cs=0 l=1 rev1=1 rx=0x7fcab4006f00 tx=0x7fcab400b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:38.958 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:38.953 7fcabaffd700 1 -- 172.21.2.102:0/3109575554 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcab400bd30 con 0x7fcac40a0290 2021-02-10T11:38:38.958 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:38.953 7fcabaffd700 1 -- 172.21.2.102:0/3109575554 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcab4000c90 con 0x7fcac40a0290 2021-02-10T11:38:38.958 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:38.953 7fcabaffd700 1 -- 172.21.2.102:0/3109575554 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcab40022a0 con 0x7fcac40a0290 2021-02-10T11:38:38.958 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:38.953 7fcac8894700 1 -- 172.21.2.102:0/3109575554 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcac4086c90 con 0x7fcac40a0290 2021-02-10T11:38:38.958 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:38.953 7fcac8894700 1 -- 172.21.2.102:0/3109575554 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcac41a2f00 con 0x7fcac40a0290 2021-02-10T11:38:38.958 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:38.954 7fcabaffd700 1 -- 172.21.2.102:0/3109575554 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fcab4005310 con 0x7fcac40a0290 2021-02-10T11:38:38.958 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:38.954 7fcabaffd700 1 --2- 172.21.2.102:0/3109575554 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcaa80186d0 0x7fcaa801abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:38.958 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:38.954 7fcabaffd700 1 -- 172.21.2.102:0/3109575554 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(180..180 src has 1..180) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fcab4032620 con 0x7fcac40a0290 2021-02-10T11:38:38.958 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:38.954 7fcac261c700 1 --2- 172.21.2.102:0/3109575554 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcaa80186d0 0x7fcaa801abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:38.958 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:38.954 7fcac261c700 1 --2- 172.21.2.102:0/3109575554 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcaa80186d0 0x7fcaa801abd0 crc :-1 s=READY pgs=1371 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:38.986 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:38.985 7fcac8894700 1 -- 172.21.2.102:0/3109575554 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcac4082a70 con 0x7fcac40a0290 2021-02-10T11:38:38.989 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:38.988 7fcabaffd700 1 -- 172.21.2.102:0/3109575554 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fcab4000940 con 0x7fcac40a0290 2021-02-10T11:38:39.116 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:39.115 7fcac8894700 1 -- 172.21.2.102:0/3109575554 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7fcac4082a70 con 0x7fcac40a0290 2021-02-10T11:38:39.117 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:39.116 7fcabaffd700 1 -- 172.21.2.102:0/3109575554 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v180) v1 ==== 104+0+10 (secure 0 0 0) 0x7fcab400bd30 con 0x7fcac40a0290 2021-02-10T11:38:39.117 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:38:39.119 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:39.117 7fcac8894700 1 -- 172.21.2.102:0/3109575554 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcaa80186d0 msgr2=0x7fcaa801abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:39.119 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:39.117 7fcac8894700 1 --2- 172.21.2.102:0/3109575554 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcaa80186d0 0x7fcaa801abd0 crc :-1 s=READY pgs=1371 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:39.119 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:39.117 7fcac8894700 1 -- 172.21.2.102:0/3109575554 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcac40a0290 msgr2=0x7fcac40852d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:39.119 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:39.117 7fcac8894700 1 --2- 172.21.2.102:0/3109575554 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcac40a0290 0x7fcac40852d0 secure :-1 s=READY pgs=3125 cs=0 l=1 rev1=1 rx=0x7fcab4006f00 tx=0x7fcab400b940).stop 2021-02-10T11:38:39.119 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:39.117 7fcac8894700 1 -- 172.21.2.102:0/3109575554 shutdown_connections 2021-02-10T11:38:39.119 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:39.117 7fcac8894700 1 --2- 172.21.2.102:0/3109575554 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcac4086270 0x7fcac4086670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:39.119 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:39.117 7fcac8894700 1 --2- 172.21.2.102:0/3109575554 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcac40a0290 0x7fcac40852d0 unknown :-1 s=CLOSED pgs=3125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:39.119 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:39.117 7fcac8894700 1 --2- 172.21.2.102:0/3109575554 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcaa80186d0 0x7fcaa801abd0 unknown :-1 s=CLOSED pgs=1371 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:39.119 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:39.117 7fcac8894700 1 --2- 172.21.2.102:0/3109575554 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fcac40c3820 0x7fcac4084cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:39.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:39.117 7fcac8894700 1 -- 172.21.2.102:0/3109575554 >> 172.21.2.102:0/3109575554 conn(0x7fcac40872c0 msgr2=0x7fcac4095700 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:39.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:39.118 7fcac8894700 1 -- 172.21.2.102:0/3109575554 shutdown_connections 2021-02-10T11:38:39.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:39.118 7fcac8894700 1 -- 172.21.2.102:0/3109575554 wait complete. 2021-02-10T11:38:39.127 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-02-10T11:38:39.560 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:39.543 7facf18a7700 1 Processor -- start 2021-02-10T11:38:39.561 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:39.543 7facf18a7700 1 -- start start 2021-02-10T11:38:39.561 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:39.543 7facf18a7700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7facec0d1c70 0x7facec0cef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:39.561 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:39.543 7facf18a7700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7facec0cf580 0x7facec0cf980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:39.561 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:39.543 7facf18a7700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facec0fe1c0 con 0x7facec0d1870 2021-02-10T11:38:39.561 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:39.543 7facf18a7700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7facec1b0cb0 con 0x7facec0cf580 2021-02-10T11:38:39.561 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:39.543 7facf18a7700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facec109a60 con 0x7facec0d1c70 2021-02-10T11:38:39.561 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:39.544 7facead73700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7facec0d1c70 0x7facec0cef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:39.561 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:39.544 7facebd75700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7facec0cf580 0x7facec0cf980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:39.561 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:39.544 7faceb574700 1 --1- >> v1:172.21.2.113:6789/0 conn(0x7facec0d1870 0x7facec1b04c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.113:6789/0 says I am v1:172.21.2.102:39298/0 (socket says 172.21.2.102:39298) 2021-02-10T11:38:39.561 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:39.544 7faceb574700 1 -- 172.21.2.102:0/2504055784 learned_addr learned my addr 172.21.2.102:0/2504055784 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:38:39.561 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:39.544 7facead73700 1 -- 172.21.2.102:0/2504055784 >> v2:172.21.2.113:3300/0 conn(0x7facec0cf580 msgr2=0x7facec0cf980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:39.561 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:39.544 7facead73700 1 --2- 172.21.2.102:0/2504055784 >> v2:172.21.2.113:3300/0 conn(0x7facec0cf580 0x7facec0cf980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:39.562 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:39.544 7facead73700 1 -- 172.21.2.102:0/2504055784 >> v1:172.21.2.113:6789/0 conn(0x7facec0d1870 legacy=0x7facec1b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:39.562 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:39.544 7facead73700 1 -- 172.21.2.102:0/2504055784 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facec088070 con 0x7facec0d1c70 2021-02-10T11:38:39.562 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:39.544 7facebd75700 1 --2- 172.21.2.102:0/2504055784 >> v2:172.21.2.113:3300/0 conn(0x7facec0cf580 0x7facec0cf980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:38:39.562 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:39.544 7facead73700 1 --2- 172.21.2.102:0/2504055784 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7facec0d1c70 0x7facec0cef60 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 rx=0x7facdc006650 tx=0x7facdc009a80).ready entity=mon.2 client_cookie=b9ce9700ecfff512 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:39.562 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:39.544 7facea572700 1 -- 172.21.2.102:0/2504055784 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7facdc00c0c0 con 0x7facec0d1c70 2021-02-10T11:38:39.562 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:39.544 7facea572700 1 -- 172.21.2.102:0/2504055784 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facdc00c340 con 0x7facec0d1c70 2021-02-10T11:38:39.562 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:39.545 7facf18a7700 1 -- 172.21.2.102:0/2504055784 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7facec0d1c70 msgr2=0x7facec0cef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:39.562 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:39.545 7facf18a7700 1 --2- 172.21.2.102:0/2504055784 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7facec0d1c70 0x7facec0cef60 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 rx=0x7facdc006650 tx=0x7facdc009a80).stop 2021-02-10T11:38:39.562 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:39.545 7facf18a7700 1 -- 172.21.2.102:0/2504055784 shutdown_connections 2021-02-10T11:38:39.563 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:39.545 7facf18a7700 1 --2- 172.21.2.102:0/2504055784 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7facec0d1c70 0x7facec0cef60 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:39.563 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:39.545 7facf18a7700 1 --2- 172.21.2.102:0/2504055784 >> v2:172.21.2.113:3300/0 conn(0x7facec0cf580 0x7facec0cf980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:39.563 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:39.545 7facf18a7700 1 -- 172.21.2.102:0/2504055784 >> 172.21.2.102:0/2504055784 conn(0x7facec0ce420 msgr2=0x7facec0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:39.563 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:39.545 7facf18a7700 1 -- 172.21.2.102:0/2504055784 shutdown_connections 2021-02-10T11:38:39.563 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:39.545 7facf18a7700 1 -- 172.21.2.102:0/2504055784 wait complete. 2021-02-10T11:38:39.565 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:39.560 7facf18a7700 1 Processor -- start 2021-02-10T11:38:39.565 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:39.560 7facf18a7700 1 -- start start 2021-02-10T11:38:39.565 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:39.560 7facf18a7700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7facec1a79a0 0x7facec1a7da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:39.565 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:39.560 7facf18a7700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7facec0bdb40 0x7facec0bdf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:39.565 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:39.560 7facf18a7700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7facec082ca0 0x7facec0830a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:39.565 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:39.560 7facf18a7700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facec19d990 con 0x7facec1a79a0 2021-02-10T11:38:39.565 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:39.560 7facf18a7700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facec19dbb0 con 0x7facec082ca0 2021-02-10T11:38:39.565 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:39.560 7facf18a7700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facec0836c0 con 0x7facec0bdb40 2021-02-10T11:38:39.565 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:39.560 7faceb574700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7facec1a79a0 0x7facec1a7da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:39.565 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:39.560 7faceb574700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7facec1a79a0 0x7facec1a7da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:39756/0 (socket says 172.21.2.102:39756) 2021-02-10T11:38:39.565 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:39.560 7faceb574700 1 -- 172.21.2.102:0/374562551 learned_addr learned my addr 172.21.2.102:0/374562551 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:38:39.565 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:39.561 7faceb574700 1 -- 172.21.2.102:0/374562551 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7facec0bdb40 msgr2=0x7facec0bdf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:39.566 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:39.561 7faceb574700 1 --2- 172.21.2.102:0/374562551 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7facec0bdb40 0x7facec0bdf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:39.566 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:39.561 7faceb574700 1 -- 172.21.2.102:0/374562551 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7facec082ca0 msgr2=0x7facec0830a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:39.566 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:39.561 7faceb574700 1 --2- 172.21.2.102:0/374562551 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7facec082ca0 0x7facec0830a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:39.566 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:39.561 7faceb574700 1 -- 172.21.2.102:0/374562551 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facec0838e0 con 0x7facec1a79a0 2021-02-10T11:38:39.566 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:39.561 7faceb574700 1 --2- 172.21.2.102:0/374562551 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7facec1a79a0 0x7facec1a7da0 secure :-1 s=READY pgs=3126 cs=0 l=1 rev1=1 rx=0x7facd40013a0 tx=0x7facd400a8a0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:39.566 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:39.561 7face3fff700 1 -- 172.21.2.102:0/374562551 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7facd40070d0 con 0x7facec1a79a0 2021-02-10T11:38:39.566 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:39.561 7face3fff700 1 -- 172.21.2.102:0/374562551 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facd400ac90 con 0x7facec1a79a0 2021-02-10T11:38:39.566 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:39.561 7face3fff700 1 -- 172.21.2.102:0/374562551 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7facd40045a0 con 0x7facec1a79a0 2021-02-10T11:38:39.566 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:39.561 7facf18a7700 1 -- 172.21.2.102:0/374562551 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7facec0838e0 con 0x7facec1a79a0 2021-02-10T11:38:39.566 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:39.561 7facf18a7700 1 -- 172.21.2.102:0/374562551 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7facec084420 con 0x7facec1a79a0 2021-02-10T11:38:39.566 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:39.562 7face3fff700 1 -- 172.21.2.102:0/374562551 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7facd4008000 con 0x7facec1a79a0 2021-02-10T11:38:39.566 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:39.562 7face3fff700 1 --2- 172.21.2.102:0/374562551 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7facd00186c0 0x7facd001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:39.566 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:39.562 7face3fff700 1 -- 172.21.2.102:0/374562551 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(180..180 src has 1..180) v4 ==== 5573+0+0 (secure 0 0 0) 0x7facd4032420 con 0x7facec1a79a0 2021-02-10T11:38:39.567 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:39.562 7facead73700 1 --2- 172.21.2.102:0/374562551 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7facd00186c0 0x7facd001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:39.567 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:39.562 7facead73700 1 --2- 172.21.2.102:0/374562551 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7facd00186c0 0x7facd001abc0 crc :-1 s=READY pgs=1372 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:39.595 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:39.593 7facf18a7700 1 -- 172.21.2.102:0/374562551 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7facec082a70 con 0x7facec1a79a0 2021-02-10T11:38:39.597 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:39.596 7face3fff700 1 -- 172.21.2.102:0/374562551 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7facd4030d50 con 0x7facec1a79a0 2021-02-10T11:38:39.724 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:39.723 7facf18a7700 1 -- 172.21.2.102:0/374562551 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7facec082a70 con 0x7facec1a79a0 2021-02-10T11:38:39.725 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:39.723 7face3fff700 1 -- 172.21.2.102:0/374562551 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v180) v1 ==== 100+0+10 (secure 0 0 0) 0x7facd4008000 con 0x7facec1a79a0 2021-02-10T11:38:39.725 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:38:39.727 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:39.725 7facf18a7700 1 -- 172.21.2.102:0/374562551 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7facd00186c0 msgr2=0x7facd001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:39.727 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:39.725 7facf18a7700 1 --2- 172.21.2.102:0/374562551 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7facd00186c0 0x7facd001abc0 crc :-1 s=READY pgs=1372 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:39.727 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:39.725 7facf18a7700 1 -- 172.21.2.102:0/374562551 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7facec1a79a0 msgr2=0x7facec1a7da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:39.727 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:39.725 7facf18a7700 1 --2- 172.21.2.102:0/374562551 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7facec1a79a0 0x7facec1a7da0 secure :-1 s=READY pgs=3126 cs=0 l=1 rev1=1 rx=0x7facd40013a0 tx=0x7facd400a8a0).stop 2021-02-10T11:38:39.727 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:39.725 7facf18a7700 1 -- 172.21.2.102:0/374562551 shutdown_connections 2021-02-10T11:38:39.727 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:39.725 7facf18a7700 1 --2- 172.21.2.102:0/374562551 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7facd00186c0 0x7facd001abc0 unknown :-1 s=CLOSED pgs=1372 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:39.727 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:39.725 7facf18a7700 1 --2- 172.21.2.102:0/374562551 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7facec082ca0 0x7facec0830a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:39.727 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:39.725 7facf18a7700 1 --2- 172.21.2.102:0/374562551 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7facec0bdb40 0x7facec0bdf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:39.727 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:39.725 7facf18a7700 1 --2- 172.21.2.102:0/374562551 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7facec1a79a0 0x7facec1a7da0 unknown :-1 s=CLOSED pgs=3126 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:39.727 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:39.725 7facf18a7700 1 -- 172.21.2.102:0/374562551 >> 172.21.2.102:0/374562551 conn(0x7facec0bef00 msgr2=0x7facec09f200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:39.728 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:39.726 7facf18a7700 1 -- 172.21.2.102:0/374562551 shutdown_connections 2021-02-10T11:38:39.728 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:39.726 7facf18a7700 1 -- 172.21.2.102:0/374562551 wait complete. 2021-02-10T11:38:39.735 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-02-10T11:38:39.736 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:38:40.164 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:40.146 7f5838f16700 1 Processor -- start 2021-02-10T11:38:40.165 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:40.146 7f5838f16700 1 -- start start 2021-02-10T11:38:40.165 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:40.147 7f5838f16700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f58340d1870 0x7f58341b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:40.165 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:40.147 7f5838f16700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f58340f0650 0x7f58340cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:40.165 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:40.147 7f5838f16700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58340f0b10 con 0x7f58341b0a10 2021-02-10T11:38:40.165 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:40.147 7f5838f16700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58340e5ed0 con 0x7f58340d1870 2021-02-10T11:38:40.165 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:40.147 7f5838f16700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58340d1c70 con 0x7f58340f0650 2021-02-10T11:38:40.165 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:40.147 7f5832b13700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f58340d1870 0x7f58341b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:40.165 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:40.147 7f5832b13700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f58340d1870 0x7f58341b03f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:39762/0 (socket says 172.21.2.102:39762) 2021-02-10T11:38:40.166 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:40.147 7f5832b13700 1 -- 172.21.2.102:0/4142242775 learned_addr learned my addr 172.21.2.102:0/4142242775 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:38:40.166 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:40.147 7f5832b13700 1 -- 172.21.2.102:0/4142242775 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f58340f0650 msgr2=0x7f58340cfb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:40.166 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:40.147 7f5832b13700 1 --2- 172.21.2.102:0/4142242775 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f58340f0650 0x7f58340cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:40.166 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:40.147 7f5832b13700 1 -- 172.21.2.102:0/4142242775 >> v1:172.21.2.113:6789/0 conn(0x7f58341b0a10 legacy=0x7f58340cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:40.166 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:40.147 7f5832b13700 1 -- 172.21.2.102:0/4142242775 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58340d1e90 con 0x7f58340d1870 2021-02-10T11:38:40.166 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:40.148 7f5832b13700 1 --2- 172.21.2.102:0/4142242775 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f58340d1870 0x7f58341b03f0 secure :-1 s=READY pgs=3127 cs=0 l=1 rev1=1 rx=0x7f581c006700 tx=0x7f581c009b80).ready entity=mon.0 client_cookie=6ed79439db2c40b5 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:40.166 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:40.148 7f5831b11700 1 -- 172.21.2.102:0/4142242775 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f581c00c0c0 con 0x7f58340d1870 2021-02-10T11:38:40.166 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:40.148 7f5831b11700 1 -- 172.21.2.102:0/4142242775 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f581c00c380 con 0x7f58340d1870 2021-02-10T11:38:40.166 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:40.148 7f5831b11700 1 -- 172.21.2.102:0/4142242775 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f581c00c8a0 con 0x7f58340d1870 2021-02-10T11:38:40.166 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:40.148 7f5838f16700 1 -- 172.21.2.102:0/4142242775 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f58340d1870 msgr2=0x7f58341b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:40.166 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:40.148 7f5838f16700 1 --2- 172.21.2.102:0/4142242775 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f58340d1870 0x7f58341b03f0 secure :-1 s=READY pgs=3127 cs=0 l=1 rev1=1 rx=0x7f581c006700 tx=0x7f581c009b80).stop 2021-02-10T11:38:40.167 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:40.149 7f5838f16700 1 -- 172.21.2.102:0/4142242775 shutdown_connections 2021-02-10T11:38:40.167 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:40.149 7f5838f16700 1 --2- 172.21.2.102:0/4142242775 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f58340f0650 0x7f58340cfb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:40.167 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:40.149 7f5838f16700 1 --2- 172.21.2.102:0/4142242775 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f58340d1870 0x7f58341b03f0 unknown :-1 s=CLOSED pgs=3127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:40.167 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:40.149 7f5838f16700 1 -- 172.21.2.102:0/4142242775 >> 172.21.2.102:0/4142242775 conn(0x7f58340ce450 msgr2=0x7f58340d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:40.167 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:40.149 7f5838f16700 1 -- 172.21.2.102:0/4142242775 shutdown_connections 2021-02-10T11:38:40.167 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:40.149 7f5838f16700 1 -- 172.21.2.102:0/4142242775 wait complete. 2021-02-10T11:38:40.167 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:40.163 7f5838f16700 1 Processor -- start 2021-02-10T11:38:40.167 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:40.163 7f5838f16700 1 -- start start 2021-02-10T11:38:40.169 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:40.164 7f5838f16700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f58340b0ad0 0x7f5834091f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:40.169 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:40.164 7f5838f16700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f58341a8850 0x7f58341a8c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:40.169 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:40.164 7f5838f16700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f58341a6fa0 0x7f58341a73a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:40.169 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:40.164 7f5838f16700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58341abfe0 con 0x7f58341a6fa0 2021-02-10T11:38:40.169 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:40.164 7f5838f16700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58341a9c40 con 0x7f58341a8850 2021-02-10T11:38:40.169 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:40.164 7f5838f16700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58341a9730 con 0x7f58340b0ad0 2021-02-10T11:38:40.170 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:40.164 7f5833314700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f58341a6fa0 0x7f58341a73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:40.170 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:40.164 7f5833314700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f58341a6fa0 0x7f58341a73a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:39770/0 (socket says 172.21.2.102:39770) 2021-02-10T11:38:40.170 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:40.164 7f5833314700 1 -- 172.21.2.102:0/1270241383 learned_addr learned my addr 172.21.2.102:0/1270241383 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:38:40.170 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:40.164 7f5832312700 1 --2- 172.21.2.102:0/1270241383 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f58341a8850 0x7f58341a8c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:40.170 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:40.165 7f5833314700 1 -- 172.21.2.102:0/1270241383 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f58340b0ad0 msgr2=0x7f5834091f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:40.170 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:40.165 7f5833314700 1 --2- 172.21.2.102:0/1270241383 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f58340b0ad0 0x7f5834091f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:40.170 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:40.165 7f5833314700 1 -- 172.21.2.102:0/1270241383 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f58341a8850 msgr2=0x7f58341a8c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:40.170 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:40.165 7f5833314700 1 --2- 172.21.2.102:0/1270241383 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f58341a8850 0x7f58341a8c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:40.170 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:40.165 7f5833314700 1 -- 172.21.2.102:0/1270241383 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58341a9270 con 0x7f58341a6fa0 2021-02-10T11:38:40.170 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:40.165 7f5833314700 1 --2- 172.21.2.102:0/1270241383 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f58341a6fa0 0x7f58341a73a0 secure :-1 s=READY pgs=3128 cs=0 l=1 rev1=1 rx=0x7f582c009980 tx=0x7f582c00bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:40.170 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:40.165 7f582b7fe700 1 -- 172.21.2.102:0/1270241383 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f582c005870 con 0x7f58341a6fa0 2021-02-10T11:38:40.171 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:40.165 7f582b7fe700 1 -- 172.21.2.102:0/1270241383 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f582c005b30 con 0x7f58341a6fa0 2021-02-10T11:38:40.171 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:40.165 7f582b7fe700 1 -- 172.21.2.102:0/1270241383 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f582c006030 con 0x7f58341a6fa0 2021-02-10T11:38:40.171 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:40.165 7f5838f16700 1 -- 172.21.2.102:0/1270241383 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58341a9270 con 0x7f58341a6fa0 2021-02-10T11:38:40.171 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:40.165 7f5838f16700 1 -- 172.21.2.102:0/1270241383 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58341a7cb0 con 0x7f58341a6fa0 2021-02-10T11:38:40.171 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:40.166 7f582b7fe700 1 -- 172.21.2.102:0/1270241383 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f582c005d10 con 0x7f58341a6fa0 2021-02-10T11:38:40.171 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:40.166 7f582b7fe700 1 --2- 172.21.2.102:0/1270241383 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f58180186c0 0x7f581801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:40.171 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:40.166 7f582b7fe700 1 -- 172.21.2.102:0/1270241383 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(180..180 src has 1..180) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f582c032930 con 0x7f58341a6fa0 2021-02-10T11:38:40.172 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:40.166 7f5832b13700 1 --2- 172.21.2.102:0/1270241383 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f58180186c0 0x7f581801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:40.172 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:40.166 7f5832b13700 1 --2- 172.21.2.102:0/1270241383 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f58180186c0 0x7f581801abc0 crc :-1 s=READY pgs=1373 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:40.198 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:40.197 7f5838f16700 1 -- 172.21.2.102:0/1270241383 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58341abfe0 con 0x7f58341a6fa0 2021-02-10T11:38:40.200 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:40.199 7f582b7fe700 1 -- 172.21.2.102:0/1270241383 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f582c00be10 con 0x7f58341a6fa0 2021-02-10T11:38:40.327 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:40.326 7f5838f16700 1 -- 172.21.2.102:0/1270241383 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f58341abfe0 con 0x7f58341a6fa0 2021-02-10T11:38:40.328 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:40.327 7f582b7fe700 1 -- 172.21.2.102:0/1270241383 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 157 v157) v1 ==== 95+0+2107 (secure 0 0 0) 0x7f582c006f20 con 0x7f58341a6fa0 2021-02-10T11:38:40.329 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:38:40.329 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":157,"default_fscid":19,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8815,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.102:6835/3396946217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6834","nonce":3396946217},{"type":"v1","addr":"172.21.2.102:6835","nonce":3396946217}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":153}],"filesystems":[{"mdsmap":{"epoch":157,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:38:30.335525","modified":"2021-02-10 11:38:32.341928","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8812},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8812":{"gid":8812,"name":"b","rank":0,"incarnation":154,"state":"up:active","state_seq":3,"addr":"172.21.2.113:6833/1513291768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6832","nonce":1513291768},{"type":"v1","addr":"172.21.2.113:6833","nonce":1513291768}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[39],"metadata_pool":38,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":19}]} 2021-02-10T11:38:40.329 INFO:teuthology.orchestra.run.gibba002.stderr:dumped fsmap epoch 157 2021-02-10T11:38:40.330 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:40.329 7f5838f16700 1 -- 172.21.2.102:0/1270241383 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f58180186c0 msgr2=0x7f581801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:40.330 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:40.329 7f5838f16700 1 --2- 172.21.2.102:0/1270241383 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f58180186c0 0x7f581801abc0 crc :-1 s=READY pgs=1373 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:40.330 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:40.329 7f5838f16700 1 -- 172.21.2.102:0/1270241383 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f58341a6fa0 msgr2=0x7f58341a73a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:40.331 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:40.329 7f5838f16700 1 --2- 172.21.2.102:0/1270241383 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f58341a6fa0 0x7f58341a73a0 secure :-1 s=READY pgs=3128 cs=0 l=1 rev1=1 rx=0x7f582c009980 tx=0x7f582c00bcd0).stop 2021-02-10T11:38:40.331 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:40.329 7f5838f16700 1 -- 172.21.2.102:0/1270241383 shutdown_connections 2021-02-10T11:38:40.331 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:40.329 7f5838f16700 1 --2- 172.21.2.102:0/1270241383 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f58341a6fa0 0x7f58341a73a0 unknown :-1 s=CLOSED pgs=3128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:40.331 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:40.329 7f5838f16700 1 --2- 172.21.2.102:0/1270241383 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f58180186c0 0x7f581801abc0 unknown :-1 s=CLOSED pgs=1373 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:40.331 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:40.329 7f5838f16700 1 --2- 172.21.2.102:0/1270241383 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f58341a8850 0x7f58341a8c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:40.331 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:40.329 7f5838f16700 1 --2- 172.21.2.102:0/1270241383 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f58340b0ad0 0x7f5834091f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:40.331 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:40.329 7f5838f16700 1 -- 172.21.2.102:0/1270241383 >> 172.21.2.102:0/1270241383 conn(0x7f583419f8f0 msgr2=0x7f58341a5f80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:40.331 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:40.329 7f5838f16700 1 -- 172.21.2.102:0/1270241383 shutdown_connections 2021-02-10T11:38:40.331 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:40.329 7f5838f16700 1 -- 172.21.2.102:0/1270241383 wait complete. 2021-02-10T11:38:40.541 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 157, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:38:30.335525', 'modified': '2021-02-10 11:38:32.341928', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8812}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8812': {'gid': 8812, 'name': 'b', 'rank': 0, 'incarnation': 154, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.113:6833/1513291768', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.113:6832', 'nonce': 1513291768}, {'type': 'v1', 'addr': '172.21.2.113:6833', 'nonce': 1513291768}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [39], 'metadata_pool': 38, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:38:40.541 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:38:40.541 DEBUG:teuthology.orchestra.run.gibba013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:38:40.939 INFO:teuthology.orchestra.run.gibba013.stdout:{ 2021-02-10T11:38:40.939 INFO:teuthology.orchestra.run.gibba013.stdout: "cluster_fsid": "68db30ea-dbce-4aef-a6df-f01c6fea0513", 2021-02-10T11:38:40.939 INFO:teuthology.orchestra.run.gibba013.stdout: "whoami": 0, 2021-02-10T11:38:40.939 INFO:teuthology.orchestra.run.gibba013.stdout: "id": 8812, 2021-02-10T11:38:40.939 INFO:teuthology.orchestra.run.gibba013.stdout: "want_state": "up:active", 2021-02-10T11:38:40.939 INFO:teuthology.orchestra.run.gibba013.stdout: "state": "up:active", 2021-02-10T11:38:40.939 INFO:teuthology.orchestra.run.gibba013.stdout: "rank_uptime": 10.599916587999999, 2021-02-10T11:38:40.939 INFO:teuthology.orchestra.run.gibba013.stdout: "mdsmap_epoch": 157, 2021-02-10T11:38:40.940 INFO:teuthology.orchestra.run.gibba013.stdout: "osdmap_epoch": 180, 2021-02-10T11:38:40.940 INFO:teuthology.orchestra.run.gibba013.stdout: "osdmap_epoch_barrier": 180, 2021-02-10T11:38:40.940 INFO:teuthology.orchestra.run.gibba013.stdout: "uptime": 16.255157709999999 2021-02-10T11:38:40.940 INFO:teuthology.orchestra.run.gibba013.stdout:} 2021-02-10T11:38:40.943 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "68db30ea-dbce-4aef-a6df-f01c6fea0513", "whoami": 0, "id": 8812, "want_state": "up:active", "state": "up:active", "rank_uptime": 10.599916587999999, "mdsmap_epoch": 157, "osdmap_epoch": 180, "osdmap_epoch_barrier": 180, "uptime": 16.255157709999999 } 2021-02-10T11:38:40.943 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2021-02-10T11:38:40.944 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2021-02-10T11:38:40.944 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@gibba002.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2021-02-10T11:38:40.944 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:38:40.944 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec mkdir -p /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:38:41.280 DEBUG:teuthology.orchestra.run.gibba002:> sudo modprobe fuse 2021-02-10T11:38:41.626 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:38:41.626 DEBUG:teuthology.orchestra.run.gibba002:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:38:41.970 INFO:teuthology.orchestra.run.gibba002.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:38:41.970 INFO:teuthology.orchestra.run.gibba002.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:38:41.971 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:38:41.971 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:38:41.971 DEBUG:teuthology.orchestra.run.gibba002:> ls /sys/fs/fuse/connections 2021-02-10T11:38:42.306 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2021-02-10T11:38:42.307 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:38:42.309 DEBUG:teuthology.orchestra.run.gibba002:> sudo modprobe fuse 2021-02-10T11:38:42.654 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:38:42.654 DEBUG:teuthology.orchestra.run.gibba002:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:38:42.720 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba002.stderr:ceph-fuse[212521]: starting ceph client 2021-02-10T11:38:42.721 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba002.stderr:2021-02-10 11:38:42.719 7f239c469f80 -1 init, newargv = 0x55ccc32ed1a0 newargc=9 2021-02-10T11:38:42.726 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba002.stderr:ceph-fuse[212521]: starting fuse 2021-02-10T11:38:42.997 INFO:teuthology.orchestra.run.gibba002.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:38:42.998 INFO:teuthology.orchestra.run.gibba002.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:38:42.998 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:38:42.999 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:38:42.999 DEBUG:teuthology.orchestra.run.gibba002:> ls /sys/fs/fuse/connections 2021-02-10T11:38:43.333 INFO:teuthology.orchestra.run.gibba002.stdout:43 2021-02-10T11:38:43.334 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2021-02-10T11:38:43.334 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:38:43.334 DEBUG:teuthology.orchestra.run.gibba002:> sudo python3 -c ' 2021-02-10T11:38:43.334 DEBUG:teuthology.orchestra.run.gibba002:> import glob 2021-02-10T11:38:43.334 DEBUG:teuthology.orchestra.run.gibba002:> import re 2021-02-10T11:38:43.334 DEBUG:teuthology.orchestra.run.gibba002:> import os 2021-02-10T11:38:43.334 DEBUG:teuthology.orchestra.run.gibba002:> import subprocess 2021-02-10T11:38:43.335 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:38:43.335 DEBUG:teuthology.orchestra.run.gibba002:> def find_socket(client_name): 2021-02-10T11:38:43.335 DEBUG:teuthology.orchestra.run.gibba002:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2021-02-10T11:38:43.335 DEBUG:teuthology.orchestra.run.gibba002:> files = glob.glob(asok_path) 2021-02-10T11:38:43.335 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:38:43.335 DEBUG:teuthology.orchestra.run.gibba002:> # Given a non-glob path, it better be there 2021-02-10T11:38:43.335 DEBUG:teuthology.orchestra.run.gibba002:> if "*" not in asok_path: 2021-02-10T11:38:43.335 DEBUG:teuthology.orchestra.run.gibba002:> assert(len(files) == 1) 2021-02-10T11:38:43.335 DEBUG:teuthology.orchestra.run.gibba002:> return files[0] 2021-02-10T11:38:43.335 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:38:43.335 DEBUG:teuthology.orchestra.run.gibba002:> for f in files: 2021-02-10T11:38:43.335 DEBUG:teuthology.orchestra.run.gibba002:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2021-02-10T11:38:43.335 DEBUG:teuthology.orchestra.run.gibba002:> if os.path.exists("/proc/{0}".format(pid)): 2021-02-10T11:38:43.335 DEBUG:teuthology.orchestra.run.gibba002:> return f 2021-02-10T11:38:43.335 DEBUG:teuthology.orchestra.run.gibba002:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2021-02-10T11:38:43.336 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:38:43.336 DEBUG:teuthology.orchestra.run.gibba002:> print(find_socket("client.0")) 2021-02-10T11:38:43.336 DEBUG:teuthology.orchestra.run.gibba002:> ' 2021-02-10T11:38:43.698 INFO:teuthology.orchestra.run.gibba002.stdout:/var/run/ceph/ceph-client.0.212521.asok 2021-02-10T11:38:43.701 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.212521.asok 2021-02-10T11:38:43.701 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:38:43.701 DEBUG:teuthology.orchestra.run.gibba002:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.212521.asok status 2021-02-10T11:38:44.094 INFO:teuthology.orchestra.run.gibba002.stdout:{ 2021-02-10T11:38:44.094 INFO:teuthology.orchestra.run.gibba002.stdout: "metadata": { 2021-02-10T11:38:44.094 INFO:teuthology.orchestra.run.gibba002.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:38:44.094 INFO:teuthology.orchestra.run.gibba002.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:38:44.094 INFO:teuthology.orchestra.run.gibba002.stdout: "entity_id": "0", 2021-02-10T11:38:44.094 INFO:teuthology.orchestra.run.gibba002.stdout: "hostname": "gibba002", 2021-02-10T11:38:44.094 INFO:teuthology.orchestra.run.gibba002.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2021-02-10T11:38:44.094 INFO:teuthology.orchestra.run.gibba002.stdout: "pid": "212521", 2021-02-10T11:38:44.094 INFO:teuthology.orchestra.run.gibba002.stdout: "root": "/" 2021-02-10T11:38:44.095 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:38:44.095 INFO:teuthology.orchestra.run.gibba002.stdout: "dentry_count": 0, 2021-02-10T11:38:44.095 INFO:teuthology.orchestra.run.gibba002.stdout: "dentry_pinned_count": 0, 2021-02-10T11:38:44.095 INFO:teuthology.orchestra.run.gibba002.stdout: "id": 13248, 2021-02-10T11:38:44.095 INFO:teuthology.orchestra.run.gibba002.stdout: "inst": { 2021-02-10T11:38:44.095 INFO:teuthology.orchestra.run.gibba002.stdout: "name": { 2021-02-10T11:38:44.095 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "client", 2021-02-10T11:38:44.095 INFO:teuthology.orchestra.run.gibba002.stdout: "num": 13248 2021-02-10T11:38:44.095 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:38:44.095 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": { 2021-02-10T11:38:44.095 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:38:44.095 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:0", 2021-02-10T11:38:44.095 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 1690812145 2021-02-10T11:38:44.095 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:38:44.096 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:38:44.096 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": { 2021-02-10T11:38:44.096 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:38:44.096 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:0", 2021-02-10T11:38:44.096 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 1690812145 2021-02-10T11:38:44.096 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:38:44.096 INFO:teuthology.orchestra.run.gibba002.stdout: "inst_str": "client.13248 172.21.2.102:0/1690812145", 2021-02-10T11:38:44.096 INFO:teuthology.orchestra.run.gibba002.stdout: "addr_str": "172.21.2.102:0/1690812145", 2021-02-10T11:38:44.096 INFO:teuthology.orchestra.run.gibba002.stdout: "inode_count": 1, 2021-02-10T11:38:44.096 INFO:teuthology.orchestra.run.gibba002.stdout: "mds_epoch": 157, 2021-02-10T11:38:44.096 INFO:teuthology.orchestra.run.gibba002.stdout: "osd_epoch": 180, 2021-02-10T11:38:44.096 INFO:teuthology.orchestra.run.gibba002.stdout: "osd_epoch_barrier": 0, 2021-02-10T11:38:44.096 INFO:teuthology.orchestra.run.gibba002.stdout: "blacklisted": false 2021-02-10T11:38:44.096 INFO:teuthology.orchestra.run.gibba002.stdout:} 2021-02-10T11:38:44.098 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:38:44.098 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:38:44.098 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:38:44.434 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:38:44.434 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:38:44.434 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:38:44.435 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:38:44.783 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:38:44.836 DEBUG:teuthology.orchestra.run.gibba002:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:38:44.838 DEBUG:teuthology.orchestra.run.gibba013:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:38:45.214 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:45.197 7fd963402700 1 Processor -- start 2021-02-10T11:38:45.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:45.197 7fd963402700 1 -- start start 2021-02-10T11:38:45.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:45.197 7fd963402700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd95c0d0340 0x7fd95c0ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:45.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:45.197 7fd963402700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fd95c0d1c70 0x7fd95c0efc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:45.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:45.197 7fd963402700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd95c0f0b10 con 0x7fd95c0d1870 2021-02-10T11:38:45.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:45.197 7fd963402700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd95c0e5ed0 con 0x7fd95c0d1c70 2021-02-10T11:38:45.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:45.197 7fd963402700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd95c1b0be0 con 0x7fd95c0d0340 2021-02-10T11:38:45.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:45.198 7fd961a1f700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fd95c0d1c70 0x7fd95c0efc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:45.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:45.198 7fd961a1f700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fd95c0d1c70 0x7fd95c0efc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:39790/0 (socket says 172.21.2.102:39790) 2021-02-10T11:38:45.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:45.198 7fd961a1f700 1 -- 172.21.2.102:0/2640190873 learned_addr learned my addr 172.21.2.102:0/2640190873 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:38:45.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:45.198 7fd96121e700 1 --1- 172.21.2.102:0/2640190873 >> v1:172.21.2.102:6789/0 conn(0x7fd95c0d1870 0x7fd95c1b03f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:60242/0 (socket says 172.21.2.102:60242) 2021-02-10T11:38:45.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:45.198 7fd960a1d700 1 --2- 172.21.2.102:0/2640190873 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd95c0d0340 0x7fd95c0ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:45.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:45.198 7fd961a1f700 1 -- 172.21.2.102:0/2640190873 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd95c0d0340 msgr2=0x7fd95c0ef640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:45.216 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:45.198 7fd961a1f700 1 --2- 172.21.2.102:0/2640190873 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd95c0d0340 0x7fd95c0ef640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:45.216 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:45.198 7fd961a1f700 1 -- 172.21.2.102:0/2640190873 >> v1:172.21.2.102:6789/0 conn(0x7fd95c0d1870 legacy=0x7fd95c1b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:45.216 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:45.198 7fd961a1f700 1 -- 172.21.2.102:0/2640190873 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd95c0f20a0 con 0x7fd95c0d1c70 2021-02-10T11:38:45.216 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:45.198 7fd960a1d700 1 --2- 172.21.2.102:0/2640190873 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd95c0d0340 0x7fd95c0ef640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:38:45.216 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:45.198 7fd961a1f700 1 --2- 172.21.2.102:0/2640190873 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd95c0d1c70 0x7fd95c0efc60 secure :-1 s=READY pgs=3133 cs=0 l=1 rev1=1 rx=0x7fd954006f00 tx=0x7fd954009f10).ready entity=mon.0 client_cookie=327bb0e5634749ec server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:45.216 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:45.198 7fd94ffff700 1 -- 172.21.2.102:0/2640190873 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd95400b3b0 con 0x7fd95c0d1c70 2021-02-10T11:38:45.216 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:45.198 7fd94ffff700 1 -- 172.21.2.102:0/2640190873 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd954000d30 con 0x7fd95c0d1c70 2021-02-10T11:38:45.216 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:45.198 7fd94ffff700 1 -- 172.21.2.102:0/2640190873 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd95400b7e0 con 0x7fd95c0d1c70 2021-02-10T11:38:45.216 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:45.198 7fd963402700 1 -- 172.21.2.102:0/2640190873 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd95c0d1c70 msgr2=0x7fd95c0efc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:45.216 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:45.198 7fd963402700 1 --2- 172.21.2.102:0/2640190873 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd95c0d1c70 0x7fd95c0efc60 secure :-1 s=READY pgs=3133 cs=0 l=1 rev1=1 rx=0x7fd954006f00 tx=0x7fd954009f10).stop 2021-02-10T11:38:45.216 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:45.199 7fd963402700 1 -- 172.21.2.102:0/2640190873 shutdown_connections 2021-02-10T11:38:45.216 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:45.199 7fd963402700 1 --2- 172.21.2.102:0/2640190873 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd95c0d1c70 0x7fd95c0efc60 unknown :-1 s=CLOSED pgs=3133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:45.216 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:45.199 7fd963402700 1 --2- 172.21.2.102:0/2640190873 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd95c0d0340 0x7fd95c0ef640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:45.216 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:45.199 7fd963402700 1 -- 172.21.2.102:0/2640190873 >> 172.21.2.102:0/2640190873 conn(0x7fd95c0ce450 msgr2=0x7fd95c0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:45.217 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:45.199 7fd963402700 1 -- 172.21.2.102:0/2640190873 shutdown_connections 2021-02-10T11:38:45.217 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:45.199 7fd963402700 1 -- 172.21.2.102:0/2640190873 wait complete. 2021-02-10T11:38:45.218 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:45.214 7fd963402700 1 Processor -- start 2021-02-10T11:38:45.218 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:45.214 7fd963402700 1 -- start start 2021-02-10T11:38:45.218 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:45.214 7fd963402700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd95c094040 0x7fd95c093370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:45.218 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:45.214 7fd963402700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd95c093990 0x7fd95c0ad700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:45.218 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:45.214 7fd963402700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd95c0add20 0x7fd95c1a2770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:45.218 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:45.214 7fd963402700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd95c093d90 con 0x7fd95c094040 2021-02-10T11:38:45.218 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:45.214 7fd963402700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd95c1a1e50 con 0x7fd95c093990 2021-02-10T11:38:45.218 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:45.214 7fd963402700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd95c094590 con 0x7fd95c0add20 2021-02-10T11:38:45.219 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:45.215 7fd96121e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd95c094040 0x7fd95c093370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:45.219 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:45.215 7fd96121e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd95c094040 0x7fd95c093370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:39792/0 (socket says 172.21.2.102:39792) 2021-02-10T11:38:45.219 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:45.215 7fd96121e700 1 -- 172.21.2.102:0/3636758900 learned_addr learned my addr 172.21.2.102:0/3636758900 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:38:45.219 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:45.215 7fd96121e700 1 -- 172.21.2.102:0/3636758900 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd95c0add20 msgr2=0x7fd95c1a2770 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:38:45.219 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:45.215 7fd96121e700 1 --2- 172.21.2.102:0/3636758900 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd95c0add20 0x7fd95c1a2770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:45.219 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:45.215 7fd96121e700 1 -- 172.21.2.102:0/3636758900 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd95c093990 msgr2=0x7fd95c0ad700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:45.219 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:45.215 7fd96121e700 1 --2- 172.21.2.102:0/3636758900 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd95c093990 0x7fd95c0ad700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:45.219 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:45.215 7fd96121e700 1 -- 172.21.2.102:0/3636758900 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd95c0ae120 con 0x7fd95c094040 2021-02-10T11:38:45.219 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:45.215 7fd96121e700 1 --2- 172.21.2.102:0/3636758900 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd95c094040 0x7fd95c093370 secure :-1 s=READY pgs=3134 cs=0 l=1 rev1=1 rx=0x7fd958001c60 tx=0x7fd95800a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:45.219 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:45.215 7fd94dffb700 1 -- 172.21.2.102:0/3636758900 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd95800acc0 con 0x7fd95c094040 2021-02-10T11:38:45.219 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:45.215 7fd94dffb700 1 -- 172.21.2.102:0/3636758900 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd958004140 con 0x7fd95c094040 2021-02-10T11:38:45.219 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:45.215 7fd963402700 1 -- 172.21.2.102:0/3636758900 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd95c0ae120 con 0x7fd95c094040 2021-02-10T11:38:45.219 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:45.215 7fd963402700 1 -- 172.21.2.102:0/3636758900 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd95c1a2d90 con 0x7fd95c094040 2021-02-10T11:38:45.220 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:45.215 7fd94dffb700 1 -- 172.21.2.102:0/3636758900 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd9580045d0 con 0x7fd95c094040 2021-02-10T11:38:45.220 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:45.216 7fd94dffb700 1 -- 172.21.2.102:0/3636758900 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd958004320 con 0x7fd95c094040 2021-02-10T11:38:45.220 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:45.216 7fd94dffb700 1 --2- 172.21.2.102:0/3636758900 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd93c0186c0 0x7fd93c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:45.220 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:45.216 7fd94dffb700 1 -- 172.21.2.102:0/3636758900 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(180..180 src has 1..180) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fd9580325b0 con 0x7fd95c094040 2021-02-10T11:38:45.220 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:45.216 7fd960a1d700 1 --2- 172.21.2.102:0/3636758900 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd93c0186c0 0x7fd93c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:45.220 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:45.216 7fd960a1d700 1 --2- 172.21.2.102:0/3636758900 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd93c0186c0 0x7fd93c01abc0 crc :-1 s=READY pgs=1374 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:45.248 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:45.247 7fd963402700 1 -- 172.21.2.102:0/3636758900 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd95c0ebc60 con 0x7fd95c094040 2021-02-10T11:38:45.251 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:45.250 7fd94dffb700 1 -- 172.21.2.102:0/3636758900 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd958028320 con 0x7fd95c094040 2021-02-10T11:38:45.377 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:45.376 7fd963402700 1 -- 172.21.2.102:0/3636758900 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd95c0f0b10 con 0x7fd95c094040 2021-02-10T11:38:45.377 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:45.376 7fd94dffb700 1 -- 172.21.2.102:0/3636758900 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v180) v1 ==== 74+0+14007 (secure 0 0 0) 0x7fd958028ec0 con 0x7fd95c094040 2021-02-10T11:38:45.378 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:38:45.378 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":180,"fsid":"68db30ea-dbce-4aef-a6df-f01c6fea0513","created":"2021-02-10 11:03:06.331557","modified":"2021-02-10 11:38:31.337750","last_up_change":"2021-02-10 11:03:32.690562","last_in_change":"2021-02-10 11:03:32.690562","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":39,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:03:37.322519","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":38,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:38:28.298788","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"179","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":39,"pool_name":"cephfs_data","create_time":"2021-02-10 11:38:29.326316","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"180","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"be691b4d-a039-408a-af80-9d53f02b6f1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":178,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6818","nonce":36088},{"type":"v1","addr":"172.21.2.102:6820","nonce":36088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6822","nonce":36088},{"type":"v1","addr":"172.21.2.102:6824","nonce":36088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6830","nonce":36088},{"type":"v1","addr":"172.21.2.102:6832","nonce":36088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6826","nonce":36088},{"type":"v1","addr":"172.21.2.102:6828","nonce":36088}]},"public_addr":"172.21.2.102:6820/36088","cluster_addr":"172.21.2.102:6824/36088","heartbeat_back_addr":"172.21.2.102:6832/36088","heartbeat_front_addr":"172.21.2.102:6828/36088","state":["exists","up"]},{"osd":1,"uuid":"64e3ccc8-9c69-4d7c-865e-af8e83822635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":178,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6819","nonce":36087},{"type":"v1","addr":"172.21.2.102:6821","nonce":36087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6823","nonce":36087},{"type":"v1","addr":"172.21.2.102:6825","nonce":36087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6831","nonce":36087},{"type":"v1","addr":"172.21.2.102:6833","nonce":36087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6827","nonce":36087},{"type":"v1","addr":"172.21.2.102:6829","nonce":36087}]},"public_addr":"172.21.2.102:6821/36087","cluster_addr":"172.21.2.102:6825/36087","heartbeat_back_addr":"172.21.2.102:6833/36087","heartbeat_front_addr":"172.21.2.102:6829/36087","state":["exists","up"]},{"osd":2,"uuid":"153700af-747f-481f-bdf4-d6072d3a842f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":178,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6810","nonce":36089},{"type":"v1","addr":"172.21.2.102:6811","nonce":36089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6812","nonce":36089},{"type":"v1","addr":"172.21.2.102:6813","nonce":36089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6816","nonce":36089},{"type":"v1","addr":"172.21.2.102:6817","nonce":36089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6814","nonce":36089},{"type":"v1","addr":"172.21.2.102:6815","nonce":36089}]},"public_addr":"172.21.2.102:6811/36089","cluster_addr":"172.21.2.102:6813/36089","heartbeat_back_addr":"172.21.2.102:6817/36089","heartbeat_front_addr":"172.21.2.102:6815/36089","state":["exists","up"]},{"osd":3,"uuid":"cbc678f4-83a9-45e6-8830-2f7c232b1ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":178,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6802","nonce":36090},{"type":"v1","addr":"172.21.2.102:6803","nonce":36090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6804","nonce":36090},{"type":"v1","addr":"172.21.2.102:6805","nonce":36090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6808","nonce":36090},{"type":"v1","addr":"172.21.2.102:6809","nonce":36090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6806","nonce":36090},{"type":"v1","addr":"172.21.2.102:6807","nonce":36090}]},"public_addr":"172.21.2.102:6803/36090","cluster_addr":"172.21.2.102:6805/36090","heartbeat_back_addr":"172.21.2.102:6809/36090","heartbeat_front_addr":"172.21.2.102:6807/36090","state":["exists","up"]},{"osd":4,"uuid":"c01e4008-b258-4f10-afd2-9e7f37214351","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":178,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6808","nonce":36368},{"type":"v1","addr":"172.21.2.113:6811","nonce":36368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6814","nonce":36368},{"type":"v1","addr":"172.21.2.113:6818","nonce":36368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6826","nonce":36368},{"type":"v1","addr":"172.21.2.113:6828","nonce":36368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6820","nonce":36368},{"type":"v1","addr":"172.21.2.113:6824","nonce":36368}]},"public_addr":"172.21.2.113:6811/36368","cluster_addr":"172.21.2.113:6818/36368","heartbeat_back_addr":"172.21.2.113:6828/36368","heartbeat_front_addr":"172.21.2.113:6824/36368","state":["exists","up"]},{"osd":5,"uuid":"551f6007-daf8-4a98-8ee0-c2b16bcccb5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":178,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6802","nonce":36366},{"type":"v1","addr":"172.21.2.113:6805","nonce":36366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6807","nonce":36366},{"type":"v1","addr":"172.21.2.113:6810","nonce":36366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6819","nonce":36366},{"type":"v1","addr":"172.21.2.113:6822","nonce":36366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6812","nonce":36366},{"type":"v1","addr":"172.21.2.113:6815","nonce":36366}]},"public_addr":"172.21.2.113:6805/36366","cluster_addr":"172.21.2.113:6810/36366","heartbeat_back_addr":"172.21.2.113:6822/36366","heartbeat_front_addr":"172.21.2.113:6815/36366","state":["exists","up"]},{"osd":6,"uuid":"bf67c040-0e50-48be-b102-c3943bc9ebef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6800","nonce":36367},{"type":"v1","addr":"172.21.2.113:6801","nonce":36367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6803","nonce":36367},{"type":"v1","addr":"172.21.2.113:6804","nonce":36367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6813","nonce":36367},{"type":"v1","addr":"172.21.2.113:6816","nonce":36367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6806","nonce":36367},{"type":"v1","addr":"172.21.2.113:6809","nonce":36367}]},"public_addr":"172.21.2.113:6801/36367","cluster_addr":"172.21.2.113:6804/36367","heartbeat_back_addr":"172.21.2.113:6816/36367","heartbeat_front_addr":"172.21.2.113:6809/36367","state":["exists","up"]},{"osd":7,"uuid":"f5461f80-018d-428f-be89-2eb2bcbe7bb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":178,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6817","nonce":36365},{"type":"v1","addr":"172.21.2.113:6821","nonce":36365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6823","nonce":36365},{"type":"v1","addr":"172.21.2.113:6825","nonce":36365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6830","nonce":36365},{"type":"v1","addr":"172.21.2.113:6831","nonce":36365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6827","nonce":36365},{"type":"v1","addr":"172.21.2.113:6829","nonce":36365}]},"public_addr":"172.21.2.113:6821/36365","cluster_addr":"172.21.2.113:6825/36365","heartbeat_back_addr":"172.21.2.113:6831/36365","heartbeat_front_addr":"172.21.2.113:6829/36365","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:38:45.379 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:45.378 7fd963402700 1 -- 172.21.2.102:0/3636758900 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd93c0186c0 msgr2=0x7fd93c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:45.379 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:45.378 7fd963402700 1 --2- 172.21.2.102:0/3636758900 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd93c0186c0 0x7fd93c01abc0 crc :-1 s=READY pgs=1374 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:45.379 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:45.378 7fd963402700 1 -- 172.21.2.102:0/3636758900 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd95c094040 msgr2=0x7fd95c093370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:45.379 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:45.378 7fd963402700 1 --2- 172.21.2.102:0/3636758900 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd95c094040 0x7fd95c093370 secure :-1 s=READY pgs=3134 cs=0 l=1 rev1=1 rx=0x7fd958001c60 tx=0x7fd95800a8d0).stop 2021-02-10T11:38:45.380 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:45.378 7fd963402700 1 -- 172.21.2.102:0/3636758900 shutdown_connections 2021-02-10T11:38:45.380 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:45.378 7fd963402700 1 --2- 172.21.2.102:0/3636758900 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd95c0add20 0x7fd95c1a2770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:45.380 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:45.378 7fd963402700 1 --2- 172.21.2.102:0/3636758900 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd93c0186c0 0x7fd93c01abc0 unknown :-1 s=CLOSED pgs=1374 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:45.380 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:45.378 7fd963402700 1 --2- 172.21.2.102:0/3636758900 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd95c093990 0x7fd95c0ad700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:45.380 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:45.378 7fd963402700 1 --2- 172.21.2.102:0/3636758900 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd95c094040 0x7fd95c093370 unknown :-1 s=CLOSED pgs=3134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:45.380 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:45.378 7fd963402700 1 -- 172.21.2.102:0/3636758900 >> 172.21.2.102:0/3636758900 conn(0x7fd95c19f8a0 msgr2=0x7fd95c094980 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:45.380 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:45.379 7fd963402700 1 -- 172.21.2.102:0/3636758900 shutdown_connections 2021-02-10T11:38:45.381 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:45.379 7fd963402700 1 -- 172.21.2.102:0/3636758900 wait complete. 2021-02-10T11:38:45.388 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:03:37.322519', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 38, 'pool_name': 'cephfs_metadata', 'create_time': '2021-02-10 11:38:28.298788', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '179', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 39, 'pool_name': 'cephfs_data', 'create_time': '2021-02-10 11:38:29.326316', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '180', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-02-10T11:38:45.388 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:38:45.821 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:45.804 7fb94b67c700 1 Processor -- start 2021-02-10T11:38:45.821 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:45.804 7fb94b67c700 1 -- start start 2021-02-10T11:38:45.822 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:45.804 7fb94b67c700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fb9440d1870 0x7fb9441b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:45.822 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:45.804 7fb94b67c700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fb9441b0ae0 0x7fb944108940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:45.822 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:45.804 7fb94b67c700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb9440d1c70 0x7fb944108f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:45.822 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:45.805 7fb94b67c700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9440fe1c0 con 0x7fb9440d1870 2021-02-10T11:38:45.822 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:45.805 7fb94b67c700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb944109a60 con 0x7fb9441b0ae0 2021-02-10T11:38:45.822 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:45.805 7fb94b67c700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb944088070 con 0x7fb9440d1c70 2021-02-10T11:38:45.822 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:45.805 7fb949498700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fb9440d1870 0x7fb9441b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:45.822 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:45.805 7fb949498700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fb9440d1870 0x7fb9441b04c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:39800/0 (socket says 172.21.2.102:39800) 2021-02-10T11:38:45.822 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:45.805 7fb949498700 1 -- 172.21.2.102:0/1984061904 learned_addr learned my addr 172.21.2.102:0/1984061904 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:38:45.822 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:45.805 7fb949498700 1 -- 172.21.2.102:0/1984061904 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb9440d1c70 msgr2=0x7fb944108f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:45.822 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:45.805 7fb949498700 1 --2- 172.21.2.102:0/1984061904 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb9440d1c70 0x7fb944108f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:45.822 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:45.805 7fb949498700 1 -- 172.21.2.102:0/1984061904 >> v2:172.21.2.113:3300/0 conn(0x7fb9441b0ae0 msgr2=0x7fb944108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:45.822 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:45.805 7fb949498700 1 --2- 172.21.2.102:0/1984061904 >> v2:172.21.2.113:3300/0 conn(0x7fb9441b0ae0 0x7fb944108940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:45.823 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:45.805 7fb949498700 1 -- 172.21.2.102:0/1984061904 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb944109580 con 0x7fb9440d1870 2021-02-10T11:38:45.823 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:45.805 7fb949498700 1 --2- 172.21.2.102:0/1984061904 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb9440d1870 0x7fb9441b04c0 secure :-1 s=READY pgs=3135 cs=0 l=1 rev1=1 rx=0x7fb934006700 tx=0x7fb934009b80).ready entity=mon.0 client_cookie=21153900ec4e60e0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:45.823 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:45.805 7fb93bfff700 1 -- 172.21.2.102:0/1984061904 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb93400c0c0 con 0x7fb9440d1870 2021-02-10T11:38:45.823 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:45.806 7fb93bfff700 1 -- 172.21.2.102:0/1984061904 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb93400c340 con 0x7fb9440d1870 2021-02-10T11:38:45.823 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:45.806 7fb93bfff700 1 -- 172.21.2.102:0/1984061904 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb93400c7a0 con 0x7fb9440d1870 2021-02-10T11:38:45.823 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:45.806 7fb94b67c700 1 -- 172.21.2.102:0/1984061904 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb9440d1870 msgr2=0x7fb9441b04c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:45.823 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:45.806 7fb94b67c700 1 --2- 172.21.2.102:0/1984061904 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb9440d1870 0x7fb9441b04c0 secure :-1 s=READY pgs=3135 cs=0 l=1 rev1=1 rx=0x7fb934006700 tx=0x7fb934009b80).stop 2021-02-10T11:38:45.823 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:45.806 7fb94b67c700 1 -- 172.21.2.102:0/1984061904 shutdown_connections 2021-02-10T11:38:45.823 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:45.806 7fb94b67c700 1 --2- 172.21.2.102:0/1984061904 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb9440d1c70 0x7fb944108f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:45.823 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:45.806 7fb94b67c700 1 --2- 172.21.2.102:0/1984061904 >> v2:172.21.2.113:3300/0 conn(0x7fb9441b0ae0 0x7fb944108940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:45.823 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:45.806 7fb94b67c700 1 --2- 172.21.2.102:0/1984061904 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb9440d1870 0x7fb9441b04c0 unknown :-1 s=CLOSED pgs=3135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:45.823 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:45.806 7fb94b67c700 1 -- 172.21.2.102:0/1984061904 >> 172.21.2.102:0/1984061904 conn(0x7fb9440ce420 msgr2=0x7fb9440d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:45.823 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:45.806 7fb94b67c700 1 -- 172.21.2.102:0/1984061904 shutdown_connections 2021-02-10T11:38:45.824 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:45.806 7fb94b67c700 1 -- 172.21.2.102:0/1984061904 wait complete. 2021-02-10T11:38:45.825 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:45.821 7fb94b67c700 1 Processor -- start 2021-02-10T11:38:45.825 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:45.821 7fb94b67c700 1 -- start start 2021-02-10T11:38:45.825 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:45.821 7fb94b67c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb9440c3820 0x7fb9440a7950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:45.825 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:45.821 7fb94b67c700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb9441a44c0 0x7fb9441a48c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:45.825 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:45.821 7fb94b67c700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb9440863d0 0x7fb9440867d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:45.825 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:45.821 7fb94b67c700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb944085ed0 con 0x7fb9440c3820 2021-02-10T11:38:45.825 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:45.821 7fb94b67c700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9440c0740 con 0x7fb9441a44c0 2021-02-10T11:38:45.825 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:45.821 7fb94b67c700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb944086df0 con 0x7fb9440863d0 2021-02-10T11:38:45.825 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:45.821 7fb949498700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb9440c3820 0x7fb9440a7950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:45.826 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:45.821 7fb949498700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb9440c3820 0x7fb9440a7950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:39806/0 (socket says 172.21.2.102:39806) 2021-02-10T11:38:45.826 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:45.821 7fb949498700 1 -- 172.21.2.102:0/3636074554 learned_addr learned my addr 172.21.2.102:0/3636074554 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:38:45.826 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:45.822 7fb949498700 1 -- 172.21.2.102:0/3636074554 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb9440863d0 msgr2=0x7fb9440867d0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:38:45.826 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:45.822 7fb949498700 1 --2- 172.21.2.102:0/3636074554 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb9440863d0 0x7fb9440867d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:45.826 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:45.822 7fb949498700 1 -- 172.21.2.102:0/3636074554 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb9441a44c0 msgr2=0x7fb9441a48c0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:38:45.826 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:45.822 7fb949498700 1 --2- 172.21.2.102:0/3636074554 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb9441a44c0 0x7fb9441a48c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:45.826 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:45.822 7fb949498700 1 -- 172.21.2.102:0/3636074554 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb944087010 con 0x7fb9440c3820 2021-02-10T11:38:45.826 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:45.822 7fb949498700 1 --2- 172.21.2.102:0/3636074554 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb9440c3820 0x7fb9440a7950 secure :-1 s=READY pgs=3136 cs=0 l=1 rev1=1 rx=0x7fb93400b5e0 tx=0x7fb93400b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:45.826 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:45.822 7fb939ffb700 1 -- 172.21.2.102:0/3636074554 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb93400bd70 con 0x7fb9440c3820 2021-02-10T11:38:45.826 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:45.822 7fb939ffb700 1 -- 172.21.2.102:0/3636074554 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb934002030 con 0x7fb9440c3820 2021-02-10T11:38:45.826 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:45.822 7fb94b67c700 1 -- 172.21.2.102:0/3636074554 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb944087010 con 0x7fb9440c3820 2021-02-10T11:38:45.826 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:45.822 7fb94b67c700 1 -- 172.21.2.102:0/3636074554 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb944083000 con 0x7fb9440c3820 2021-02-10T11:38:45.826 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:45.822 7fb939ffb700 1 -- 172.21.2.102:0/3636074554 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb934002550 con 0x7fb9440c3820 2021-02-10T11:38:45.827 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:45.823 7fb939ffb700 1 -- 172.21.2.102:0/3636074554 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fb934000c90 con 0x7fb9440c3820 2021-02-10T11:38:45.827 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:45.823 7fb939ffb700 1 --2- 172.21.2.102:0/3636074554 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb9240186c0 0x7fb92401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:45.827 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:45.823 7fb939ffb700 1 -- 172.21.2.102:0/3636074554 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(180..180 src has 1..180) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fb934032430 con 0x7fb9440c3820 2021-02-10T11:38:45.827 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:45.823 7fb948c97700 1 --2- 172.21.2.102:0/3636074554 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb9240186c0 0x7fb92401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:45.827 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:45.823 7fb948c97700 1 --2- 172.21.2.102:0/3636074554 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb9240186c0 0x7fb92401abc0 crc :-1 s=READY pgs=1375 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:45.856 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:45.854 7fb94b67c700 1 -- 172.21.2.102:0/3636074554 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb944083000 con 0x7fb9440c3820 2021-02-10T11:38:45.858 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:45.857 7fb939ffb700 1 -- 172.21.2.102:0/3636074554 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fb934028ec0 con 0x7fb9440c3820 2021-02-10T11:38:45.985 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:45.984 7fb94b67c700 1 -- 172.21.2.102:0/3636074554 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7fb944083000 con 0x7fb9440c3820 2021-02-10T11:38:45.986 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:45.984 7fb939ffb700 1 -- 172.21.2.102:0/3636074554 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v180) v1 ==== 92+0+10 (secure 0 0 0) 0x7fb934028ec0 con 0x7fb9440c3820 2021-02-10T11:38:45.986 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:38:45.987 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:45.986 7fb94b67c700 1 -- 172.21.2.102:0/3636074554 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb9240186c0 msgr2=0x7fb92401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:45.987 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:45.986 7fb94b67c700 1 --2- 172.21.2.102:0/3636074554 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb9240186c0 0x7fb92401abc0 crc :-1 s=READY pgs=1375 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:45.987 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:45.986 7fb94b67c700 1 -- 172.21.2.102:0/3636074554 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb9440c3820 msgr2=0x7fb9440a7950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:45.988 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:45.986 7fb94b67c700 1 --2- 172.21.2.102:0/3636074554 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb9440c3820 0x7fb9440a7950 secure :-1 s=READY pgs=3136 cs=0 l=1 rev1=1 rx=0x7fb93400b5e0 tx=0x7fb93400b980).stop 2021-02-10T11:38:45.988 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:45.987 7fb94b67c700 1 -- 172.21.2.102:0/3636074554 shutdown_connections 2021-02-10T11:38:45.988 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:45.987 7fb94b67c700 1 --2- 172.21.2.102:0/3636074554 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb9440863d0 0x7fb9440867d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:45.988 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:45.987 7fb94b67c700 1 --2- 172.21.2.102:0/3636074554 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb9240186c0 0x7fb92401abc0 unknown :-1 s=CLOSED pgs=1375 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:45.988 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:45.987 7fb94b67c700 1 --2- 172.21.2.102:0/3636074554 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb9441a44c0 0x7fb9441a48c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:45.988 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:45.987 7fb94b67c700 1 --2- 172.21.2.102:0/3636074554 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb9440c3820 0x7fb9440a7950 unknown :-1 s=CLOSED pgs=3136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:45.989 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:45.987 7fb94b67c700 1 -- 172.21.2.102:0/3636074554 >> 172.21.2.102:0/3636074554 conn(0x7fb9440872c0 msgr2=0x7fb944095700 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:45.989 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:45.987 7fb94b67c700 1 -- 172.21.2.102:0/3636074554 shutdown_connections 2021-02-10T11:38:45.989 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:45.987 7fb94b67c700 1 -- 172.21.2.102:0/3636074554 wait complete. 2021-02-10T11:38:45.996 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-02-10T11:38:46.428 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:46.411 7f639dbfe700 1 Processor -- start 2021-02-10T11:38:46.429 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:46.411 7f639dbfe700 1 -- start start 2021-02-10T11:38:46.429 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:46.412 7f639dbfe700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f63980d1870 0x7f63981b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:46.429 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:46.412 7f639dbfe700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f63981b0ae0 0x7f6398108940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:46.429 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:46.412 7f639dbfe700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63980fe1c0 con 0x7f63980d1c70 2021-02-10T11:38:46.429 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:46.412 7f639dbfe700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6398109a60 con 0x7f63980d1870 2021-02-10T11:38:46.429 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:46.412 7f639dbfe700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6398109750 con 0x7f63981b0ae0 2021-02-10T11:38:46.429 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:46.412 7f63977fe700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f63980d1870 0x7f63981b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:46.429 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:46.412 7f63977fe700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f63980d1870 0x7f63981b04c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:39814/0 (socket says 172.21.2.102:39814) 2021-02-10T11:38:46.430 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:46.412 7f63977fe700 1 -- 172.21.2.102:0/3866819938 learned_addr learned my addr 172.21.2.102:0/3866819938 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:38:46.430 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:46.412 7f6397fff700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f63980d1c70 0x7f6398108f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:60270/0 (socket says 172.21.2.102:60270) 2021-02-10T11:38:46.430 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:46.412 7f63977fe700 1 -- 172.21.2.102:0/3866819938 >> v2:172.21.2.113:3300/0 conn(0x7f63981b0ae0 msgr2=0x7f6398108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:46.430 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:46.412 7f63977fe700 1 --2- 172.21.2.102:0/3866819938 >> v2:172.21.2.113:3300/0 conn(0x7f63981b0ae0 0x7f6398108940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:46.430 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:46.412 7f63977fe700 1 -- 172.21.2.102:0/3866819938 >> v1:172.21.2.102:6789/0 conn(0x7f63980d1c70 legacy=0x7f6398108f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:46.430 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:46.412 7f63967fc700 1 -- 172.21.2.102:0/3866819938 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 375240845 0 0) 0x7f6390001480 con 0x7f63980d1c70 2021-02-10T11:38:46.430 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:46.413 7f63977fe700 1 -- 172.21.2.102:0/3866819938 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6398088070 con 0x7f63980d1870 2021-02-10T11:38:46.430 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:46.413 7f63977fe700 1 --2- 172.21.2.102:0/3866819938 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f63980d1870 0x7f63981b04c0 secure :-1 s=READY pgs=3138 cs=0 l=1 rev1=1 rx=0x7f6388006700 tx=0x7f6388009b80).ready entity=mon.0 client_cookie=1b8a10ad1c71f88e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:46.430 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:46.413 7f63967fc700 1 -- 172.21.2.102:0/3866819938 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f638800c0c0 con 0x7f63980d1870 2021-02-10T11:38:46.431 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:46.413 7f63967fc700 1 -- 172.21.2.102:0/3866819938 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f638800c440 con 0x7f63980d1870 2021-02-10T11:38:46.431 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:46.413 7f63967fc700 1 -- 172.21.2.102:0/3866819938 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f638800c920 con 0x7f63980d1870 2021-02-10T11:38:46.431 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:46.413 7f639dbfe700 1 -- 172.21.2.102:0/3866819938 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f63980d1870 msgr2=0x7f63981b04c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:46.431 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:46.413 7f639dbfe700 1 --2- 172.21.2.102:0/3866819938 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f63980d1870 0x7f63981b04c0 secure :-1 s=READY pgs=3138 cs=0 l=1 rev1=1 rx=0x7f6388006700 tx=0x7f6388009b80).stop 2021-02-10T11:38:46.431 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:46.413 7f639dbfe700 1 -- 172.21.2.102:0/3866819938 shutdown_connections 2021-02-10T11:38:46.431 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:46.413 7f639dbfe700 1 --2- 172.21.2.102:0/3866819938 >> v2:172.21.2.113:3300/0 conn(0x7f63981b0ae0 0x7f6398108940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:46.431 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:46.413 7f639dbfe700 1 --2- 172.21.2.102:0/3866819938 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f63980d1870 0x7f63981b04c0 unknown :-1 s=CLOSED pgs=3138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:46.431 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:46.413 7f639dbfe700 1 -- 172.21.2.102:0/3866819938 >> 172.21.2.102:0/3866819938 conn(0x7f63980ce420 msgr2=0x7f63980d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:46.431 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:46.413 7f639dbfe700 1 -- 172.21.2.102:0/3866819938 shutdown_connections 2021-02-10T11:38:46.431 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:46.413 7f639dbfe700 1 -- 172.21.2.102:0/3866819938 wait complete. 2021-02-10T11:38:46.433 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:46.428 7f639dbfe700 1 Processor -- start 2021-02-10T11:38:46.433 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:46.428 7f639dbfe700 1 -- start start 2021-02-10T11:38:46.433 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:46.428 7f639dbfe700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f63980c2370 0x7f639818f870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:46.433 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:46.428 7f639dbfe700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6398084cb0 0x7f63981a7470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:46.433 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:46.428 7f639dbfe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f63981a7a90 0x7f6398085700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:46.433 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:46.428 7f639dbfe700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63980be840 con 0x7f63981a7a90 2021-02-10T11:38:46.433 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:46.428 7f639dbfe700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63980c29a0 con 0x7f6398084cb0 2021-02-10T11:38:46.433 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:46.428 7f639dbfe700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63981a7e90 con 0x7f63980c2370 2021-02-10T11:38:46.434 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:46.428 7f6397fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f63981a7a90 0x7f6398085700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:46.434 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:46.428 7f6397fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f63981a7a90 0x7f6398085700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:39822/0 (socket says 172.21.2.102:39822) 2021-02-10T11:38:46.434 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:46.428 7f6397fff700 1 -- 172.21.2.102:0/2362762698 learned_addr learned my addr 172.21.2.102:0/2362762698 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:38:46.434 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:46.429 7f6397fff700 1 -- 172.21.2.102:0/2362762698 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f63980c2370 msgr2=0x7f639818f870 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:38:46.434 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:46.429 7f6397fff700 1 --2- 172.21.2.102:0/2362762698 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f63980c2370 0x7f639818f870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:46.434 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:46.429 7f6397fff700 1 -- 172.21.2.102:0/2362762698 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6398084cb0 msgr2=0x7f63981a7470 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:38:46.434 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:46.429 7f6397fff700 1 --2- 172.21.2.102:0/2362762698 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6398084cb0 0x7f63981a7470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:46.434 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:46.429 7f6397fff700 1 -- 172.21.2.102:0/2362762698 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63981a80b0 con 0x7f63981a7a90 2021-02-10T11:38:46.434 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:46.429 7f6397fff700 1 --2- 172.21.2.102:0/2362762698 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f63981a7a90 0x7f6398085700 secure :-1 s=READY pgs=3139 cs=0 l=1 rev1=1 rx=0x7f6390007e80 tx=0x7f639000aca0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:46.434 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:46.429 7f638ffff700 1 -- 172.21.2.102:0/2362762698 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6390007020 con 0x7f63981a7a90 2021-02-10T11:38:46.434 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:46.429 7f638ffff700 1 -- 172.21.2.102:0/2362762698 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6390007220 con 0x7f63981a7a90 2021-02-10T11:38:46.435 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:46.429 7f638ffff700 1 -- 172.21.2.102:0/2362762698 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6390004720 con 0x7f63981a7a90 2021-02-10T11:38:46.435 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:46.429 7f639dbfe700 1 -- 172.21.2.102:0/2362762698 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63981a80b0 con 0x7f63981a7a90 2021-02-10T11:38:46.435 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:46.429 7f639dbfe700 1 -- 172.21.2.102:0/2362762698 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6398086010 con 0x7f63981a7a90 2021-02-10T11:38:46.435 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:46.430 7f638ffff700 1 -- 172.21.2.102:0/2362762698 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f6390007400 con 0x7f63981a7a90 2021-02-10T11:38:46.435 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:46.430 7f638ffff700 1 --2- 172.21.2.102:0/2362762698 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f637c0186c0 0x7f637c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:46.435 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:46.430 7f638ffff700 1 -- 172.21.2.102:0/2362762698 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(180..180 src has 1..180) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f6390032930 con 0x7f63981a7a90 2021-02-10T11:38:46.435 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:46.430 7f63977fe700 1 --2- 172.21.2.102:0/2362762698 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f637c0186c0 0x7f637c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:46.435 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:46.430 7f63977fe700 1 --2- 172.21.2.102:0/2362762698 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f637c0186c0 0x7f637c01abc0 crc :-1 s=READY pgs=1376 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:46.463 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:46.461 7f639dbfe700 1 -- 172.21.2.102:0/2362762698 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6398082a70 con 0x7f63981a7a90 2021-02-10T11:38:46.465 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:46.464 7f638ffff700 1 -- 172.21.2.102:0/2362762698 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f6390020f00 con 0x7f63981a7a90 2021-02-10T11:38:46.595 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:46.594 7f639dbfe700 1 -- 172.21.2.102:0/2362762698 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7f6398082a70 con 0x7f63981a7a90 2021-02-10T11:38:46.596 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:46.595 7f638ffff700 1 -- 172.21.2.102:0/2362762698 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v180) v1 ==== 104+0+10 (secure 0 0 0) 0x7f6390020f00 con 0x7f63981a7a90 2021-02-10T11:38:46.596 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:38:46.598 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:46.597 7f639dbfe700 1 -- 172.21.2.102:0/2362762698 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f637c0186c0 msgr2=0x7f637c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:46.598 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:46.597 7f639dbfe700 1 --2- 172.21.2.102:0/2362762698 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f637c0186c0 0x7f637c01abc0 crc :-1 s=READY pgs=1376 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:46.598 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:46.597 7f639dbfe700 1 -- 172.21.2.102:0/2362762698 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f63981a7a90 msgr2=0x7f6398085700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:46.598 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:46.597 7f639dbfe700 1 --2- 172.21.2.102:0/2362762698 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f63981a7a90 0x7f6398085700 secure :-1 s=READY pgs=3139 cs=0 l=1 rev1=1 rx=0x7f6390007e80 tx=0x7f639000aca0).stop 2021-02-10T11:38:46.598 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:46.597 7f639dbfe700 1 -- 172.21.2.102:0/2362762698 shutdown_connections 2021-02-10T11:38:46.598 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:46.597 7f639dbfe700 1 --2- 172.21.2.102:0/2362762698 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f63981a7a90 0x7f6398085700 unknown :-1 s=CLOSED pgs=3139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:46.599 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:46.597 7f639dbfe700 1 --2- 172.21.2.102:0/2362762698 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f637c0186c0 0x7f637c01abc0 unknown :-1 s=CLOSED pgs=1376 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:46.599 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:46.597 7f639dbfe700 1 --2- 172.21.2.102:0/2362762698 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6398084cb0 0x7f63981a7470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:46.599 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:46.597 7f639dbfe700 1 --2- 172.21.2.102:0/2362762698 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f63980c2370 0x7f639818f870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:46.599 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:46.597 7f639dbfe700 1 -- 172.21.2.102:0/2362762698 >> 172.21.2.102:0/2362762698 conn(0x7f63980b5dc0 msgr2=0x7f63980c2d90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:46.599 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:46.597 7f639dbfe700 1 -- 172.21.2.102:0/2362762698 shutdown_connections 2021-02-10T11:38:46.599 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:46.597 7f639dbfe700 1 -- 172.21.2.102:0/2362762698 wait complete. 2021-02-10T11:38:46.606 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-02-10T11:38:47.039 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:47.022 7f2bc2fd0700 1 Processor -- start 2021-02-10T11:38:47.040 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:47.022 7f2bc2fd0700 1 -- start start 2021-02-10T11:38:47.040 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:47.023 7f2bc2fd0700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f2bbc0d1870 0x7f2bbc1b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:47.040 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:47.023 7f2bc2fd0700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f2bbc1b0ae0 0x7f2bbc108940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:47.040 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:47.023 7f2bc2fd0700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bbc0fe1c0 con 0x7f2bbc0d1c70 2021-02-10T11:38:47.040 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:47.023 7f2bc2fd0700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2bbc109a60 con 0x7f2bbc1b0ae0 2021-02-10T11:38:47.040 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:47.023 7f2bc2fd0700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2bbc109750 con 0x7f2bbc0d1870 2021-02-10T11:38:47.040 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:47.023 7f2bbbfff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f2bbc1b0ae0 0x7f2bbc108940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:47.040 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:47.023 7f2bbbfff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f2bbc1b0ae0 0x7f2bbc108940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:39828/0 (socket says 172.21.2.102:39828) 2021-02-10T11:38:47.040 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:47.023 7f2bbbfff700 1 -- 172.21.2.102:0/1427904029 learned_addr learned my addr 172.21.2.102:0/1427904029 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:38:47.041 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:47.023 7f2bbbfff700 1 -- 172.21.2.102:0/1427904029 >> v2:172.21.2.113:3300/0 conn(0x7f2bbc0d1870 msgr2=0x7f2bbc1b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:47.041 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:47.023 7f2bbbfff700 1 --2- 172.21.2.102:0/1427904029 >> v2:172.21.2.113:3300/0 conn(0x7f2bbc0d1870 0x7f2bbc1b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:47.041 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:47.023 7f2bbbfff700 1 -- 172.21.2.102:0/1427904029 >> v1:172.21.2.113:6789/0 conn(0x7f2bbc0d1c70 legacy=0x7f2bbc108f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:47.041 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:47.023 7f2bbbfff700 1 -- 172.21.2.102:0/1427904029 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bbc088070 con 0x7f2bbc1b0ae0 2021-02-10T11:38:47.041 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:47.023 7f2bbbfff700 1 --2- 172.21.2.102:0/1427904029 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2bbc1b0ae0 0x7f2bbc108940 secure :-1 s=READY pgs=3140 cs=0 l=1 rev1=1 rx=0x7f2bac006700 tx=0x7f2bac009b00).ready entity=mon.0 client_cookie=4fd39bd45bea2168 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:47.041 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:47.023 7f2bbb7fe700 1 -- 172.21.2.102:0/1427904029 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2bac00c0c0 con 0x7f2bbc1b0ae0 2021-02-10T11:38:47.041 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:47.024 7f2bbb7fe700 1 -- 172.21.2.102:0/1427904029 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bac00c400 con 0x7f2bbc1b0ae0 2021-02-10T11:38:47.041 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:47.024 7f2bbb7fe700 1 -- 172.21.2.102:0/1427904029 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2bac00c8e0 con 0x7f2bbc1b0ae0 2021-02-10T11:38:47.041 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:47.024 7f2bc2fd0700 1 -- 172.21.2.102:0/1427904029 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2bbc1b0ae0 msgr2=0x7f2bbc108940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:47.041 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:47.024 7f2bc2fd0700 1 --2- 172.21.2.102:0/1427904029 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2bbc1b0ae0 0x7f2bbc108940 secure :-1 s=READY pgs=3140 cs=0 l=1 rev1=1 rx=0x7f2bac006700 tx=0x7f2bac009b00).stop 2021-02-10T11:38:47.041 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:47.024 7f2bc2fd0700 1 -- 172.21.2.102:0/1427904029 shutdown_connections 2021-02-10T11:38:47.041 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:47.024 7f2bc2fd0700 1 --2- 172.21.2.102:0/1427904029 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2bbc1b0ae0 0x7f2bbc108940 unknown :-1 s=CLOSED pgs=3140 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:47.042 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:47.024 7f2bc2fd0700 1 --2- 172.21.2.102:0/1427904029 >> v2:172.21.2.113:3300/0 conn(0x7f2bbc0d1870 0x7f2bbc1b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:47.042 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:47.024 7f2bc2fd0700 1 -- 172.21.2.102:0/1427904029 >> 172.21.2.102:0/1427904029 conn(0x7f2bbc0ce420 msgr2=0x7f2bbc0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:47.042 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:47.024 7f2bc2fd0700 1 -- 172.21.2.102:0/1427904029 shutdown_connections 2021-02-10T11:38:47.042 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:47.024 7f2bc2fd0700 1 -- 172.21.2.102:0/1427904029 wait complete. 2021-02-10T11:38:47.043 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:47.039 7f2bc2fd0700 1 Processor -- start 2021-02-10T11:38:47.043 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:47.039 7f2bc2fd0700 1 -- start start 2021-02-10T11:38:47.043 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:47.039 7f2bc2fd0700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2bbc0c3820 0x7f2bbc084cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:47.043 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:47.039 7f2bc2fd0700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2bbc0a0290 0x7f2bbc0852d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:47.044 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:47.039 7f2bc2fd0700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2bbc086270 0x7f2bbc086670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:47.044 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:47.039 7f2bc2fd0700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bbc0c0700 con 0x7f2bbc086270 2021-02-10T11:38:47.044 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:47.039 7f2bc2fd0700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bbc0858f0 con 0x7f2bbc0a0290 2021-02-10T11:38:47.044 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:47.039 7f2bc2fd0700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2bbc085b10 con 0x7f2bbc0c3820 2021-02-10T11:38:47.044 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:47.040 7f2bc15ed700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2bbc086270 0x7f2bbc086670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:47.044 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:47.040 7f2bc15ed700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2bbc086270 0x7f2bbc086670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:39832/0 (socket says 172.21.2.102:39832) 2021-02-10T11:38:47.044 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:47.040 7f2bc15ed700 1 -- 172.21.2.102:0/778460972 learned_addr learned my addr 172.21.2.102:0/778460972 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:38:47.044 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:47.040 7f2bc15ed700 1 -- 172.21.2.102:0/778460972 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2bbc0c3820 msgr2=0x7f2bbc084cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:47.044 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:47.040 7f2bc15ed700 1 --2- 172.21.2.102:0/778460972 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2bbc0c3820 0x7f2bbc084cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:47.044 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:47.040 7f2bc15ed700 1 -- 172.21.2.102:0/778460972 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2bbc0a0290 msgr2=0x7f2bbc0852d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:47.044 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:47.040 7f2bc15ed700 1 --2- 172.21.2.102:0/778460972 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2bbc0a0290 0x7f2bbc0852d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:47.044 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:47.040 7f2bc15ed700 1 -- 172.21.2.102:0/778460972 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bbc086c90 con 0x7f2bbc086270 2021-02-10T11:38:47.044 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:47.040 7f2bc15ed700 1 --2- 172.21.2.102:0/778460972 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2bbc086270 0x7f2bbc086670 secure :-1 s=READY pgs=3141 cs=0 l=1 rev1=1 rx=0x7f2bb4009940 tx=0x7f2bb400bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:47.045 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:47.040 7f2bb97fa700 1 -- 172.21.2.102:0/778460972 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2bb4000f90 con 0x7f2bbc086270 2021-02-10T11:38:47.045 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:47.040 7f2bb97fa700 1 -- 172.21.2.102:0/778460972 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bb400c2f0 con 0x7f2bbc086270 2021-02-10T11:38:47.045 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:47.040 7f2bb97fa700 1 -- 172.21.2.102:0/778460972 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2bb400c750 con 0x7f2bbc086270 2021-02-10T11:38:47.045 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:47.040 7f2bc2fd0700 1 -- 172.21.2.102:0/778460972 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2bbc086c90 con 0x7f2bbc086270 2021-02-10T11:38:47.045 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:47.040 7f2bc2fd0700 1 -- 172.21.2.102:0/778460972 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2bbc1a2f00 con 0x7f2bbc086270 2021-02-10T11:38:47.045 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:47.041 7f2bb97fa700 1 -- 172.21.2.102:0/778460972 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f2bb4001220 con 0x7f2bbc086270 2021-02-10T11:38:47.045 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:47.041 7f2bb97fa700 1 --2- 172.21.2.102:0/778460972 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2ba00186c0 0x7f2ba001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:47.045 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:47.041 7f2bb97fa700 1 -- 172.21.2.102:0/778460972 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(180..180 src has 1..180) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f2bb4032950 con 0x7f2bbc086270 2021-02-10T11:38:47.045 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:47.041 7f2bc0dec700 1 --2- 172.21.2.102:0/778460972 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2ba00186c0 0x7f2ba001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:47.045 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:47.041 7f2bc0dec700 1 --2- 172.21.2.102:0/778460972 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2ba00186c0 0x7f2ba001abc0 crc :-1 s=READY pgs=1377 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:47.073 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:47.072 7f2bc2fd0700 1 -- 172.21.2.102:0/778460972 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2bbc082a70 con 0x7f2bbc086270 2021-02-10T11:38:47.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:47.075 7f2bb97fa700 1 -- 172.21.2.102:0/778460972 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f2bb4002ef0 con 0x7f2bbc086270 2021-02-10T11:38:47.203 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:47.202 7f2bc2fd0700 1 -- 172.21.2.102:0/778460972 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7f2bbc082a70 con 0x7f2bbc086270 2021-02-10T11:38:47.204 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:47.202 7f2bb97fa700 1 -- 172.21.2.102:0/778460972 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v180) v1 ==== 100+0+10 (secure 0 0 0) 0x7f2bb4000f40 con 0x7f2bbc086270 2021-02-10T11:38:47.204 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:38:47.205 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:47.204 7f2bc2fd0700 1 -- 172.21.2.102:0/778460972 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2ba00186c0 msgr2=0x7f2ba001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:47.205 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:47.204 7f2bc2fd0700 1 --2- 172.21.2.102:0/778460972 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2ba00186c0 0x7f2ba001abc0 crc :-1 s=READY pgs=1377 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:47.205 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:47.204 7f2bc2fd0700 1 -- 172.21.2.102:0/778460972 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2bbc086270 msgr2=0x7f2bbc086670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:47.206 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:47.204 7f2bc2fd0700 1 --2- 172.21.2.102:0/778460972 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2bbc086270 0x7f2bbc086670 secure :-1 s=READY pgs=3141 cs=0 l=1 rev1=1 rx=0x7f2bb4009940 tx=0x7f2bb400bcf0).stop 2021-02-10T11:38:47.206 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:47.204 7f2bc2fd0700 1 -- 172.21.2.102:0/778460972 shutdown_connections 2021-02-10T11:38:47.206 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:47.205 7f2bc2fd0700 1 --2- 172.21.2.102:0/778460972 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2bbc086270 0x7f2bbc086670 unknown :-1 s=CLOSED pgs=3141 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:47.206 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:47.205 7f2bc2fd0700 1 --2- 172.21.2.102:0/778460972 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2ba00186c0 0x7f2ba001abc0 unknown :-1 s=CLOSED pgs=1377 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:47.206 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:47.205 7f2bc2fd0700 1 --2- 172.21.2.102:0/778460972 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2bbc0a0290 0x7f2bbc0852d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:47.207 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:47.205 7f2bc2fd0700 1 --2- 172.21.2.102:0/778460972 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2bbc0c3820 0x7f2bbc084cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:47.207 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:47.205 7f2bc2fd0700 1 -- 172.21.2.102:0/778460972 >> 172.21.2.102:0/778460972 conn(0x7f2bbc0872c0 msgr2=0x7f2bbc095700 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:47.207 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:47.205 7f2bc2fd0700 1 -- 172.21.2.102:0/778460972 shutdown_connections 2021-02-10T11:38:47.207 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:47.205 7f2bc2fd0700 1 -- 172.21.2.102:0/778460972 wait complete. 2021-02-10T11:38:47.214 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-02-10T11:38:47.214 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:38:47.644 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:47.625 7f27c81d4700 1 Processor -- start 2021-02-10T11:38:47.644 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:47.625 7f27c81d4700 1 -- start start 2021-02-10T11:38:47.644 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:47.626 7f27c81d4700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f27c00d1870 0x7f27c01b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:47.644 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:47.626 7f27c81d4700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f27c01b0a10 0x7f27c00cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:47.644 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:47.626 7f27c81d4700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27c00f0b10 con 0x7f27c00d1c70 2021-02-10T11:38:47.645 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:47.626 7f27c81d4700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27c00e5ed0 con 0x7f27c00d1870 2021-02-10T11:38:47.645 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:47.626 7f27c81d4700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27c00f0650 con 0x7f27c01b0a10 2021-02-10T11:38:47.645 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:47.626 7f27c5ff0700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f27c00d1870 0x7f27c01b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:47.645 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:47.626 7f27c5ff0700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f27c00d1870 0x7f27c01b03f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:39840/0 (socket says 172.21.2.102:39840) 2021-02-10T11:38:47.645 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:47.626 7f27c5ff0700 1 -- 172.21.2.102:0/3621613986 learned_addr learned my addr 172.21.2.102:0/3621613986 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:38:47.645 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:47.626 7f27c5ff0700 1 -- 172.21.2.102:0/3621613986 >> v2:172.21.2.113:3300/0 conn(0x7f27c01b0a10 msgr2=0x7f27c00cf330 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:38:47.645 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:47.626 7f27c5ff0700 1 --2- 172.21.2.102:0/3621613986 >> v2:172.21.2.113:3300/0 conn(0x7f27c01b0a10 0x7f27c00cf330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:47.645 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:47.626 7f27c5ff0700 1 -- 172.21.2.102:0/3621613986 >> v1:172.21.2.113:6789/0 conn(0x7f27c00d1c70 legacy=0x7f27c00cf950 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:38:47.645 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:47.626 7f27c5ff0700 1 -- 172.21.2.102:0/3621613986 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27c00f0840 con 0x7f27c00d1870 2021-02-10T11:38:47.645 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:47.626 7f27c5ff0700 1 --2- 172.21.2.102:0/3621613986 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f27c00d1870 0x7f27c01b03f0 secure :-1 s=READY pgs=3142 cs=0 l=1 rev1=1 rx=0x7f27bc006700 tx=0x7f27bc009b00).ready entity=mon.0 client_cookie=3d52f06f2b0350ef server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:47.645 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:47.627 7f27c4fee700 1 -- 172.21.2.102:0/3621613986 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f27bc00c0c0 con 0x7f27c00d1870 2021-02-10T11:38:47.645 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:47.627 7f27c4fee700 1 -- 172.21.2.102:0/3621613986 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27bc00c340 con 0x7f27c00d1870 2021-02-10T11:38:47.646 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:47.627 7f27c4fee700 1 -- 172.21.2.102:0/3621613986 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f27bc00c820 con 0x7f27c00d1870 2021-02-10T11:38:47.646 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:47.627 7f27c81d4700 1 -- 172.21.2.102:0/3621613986 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f27c00d1870 msgr2=0x7f27c01b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:47.646 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:47.627 7f27c81d4700 1 --2- 172.21.2.102:0/3621613986 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f27c00d1870 0x7f27c01b03f0 secure :-1 s=READY pgs=3142 cs=0 l=1 rev1=1 rx=0x7f27bc006700 tx=0x7f27bc009b00).stop 2021-02-10T11:38:47.646 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:47.627 7f27c81d4700 1 -- 172.21.2.102:0/3621613986 shutdown_connections 2021-02-10T11:38:47.646 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:47.627 7f27c81d4700 1 --2- 172.21.2.102:0/3621613986 >> v2:172.21.2.113:3300/0 conn(0x7f27c01b0a10 0x7f27c00cf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:47.646 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:47.627 7f27c81d4700 1 --2- 172.21.2.102:0/3621613986 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f27c00d1870 0x7f27c01b03f0 unknown :-1 s=CLOSED pgs=3142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:47.646 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:47.627 7f27c81d4700 1 -- 172.21.2.102:0/3621613986 >> 172.21.2.102:0/3621613986 conn(0x7f27c00ce450 msgr2=0x7f27c00d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:47.646 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:47.628 7f27c81d4700 1 -- 172.21.2.102:0/3621613986 shutdown_connections 2021-02-10T11:38:47.646 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:47.628 7f27c81d4700 1 -- 172.21.2.102:0/3621613986 wait complete. 2021-02-10T11:38:47.648 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:47.642 7f27c81d4700 1 Processor -- start 2021-02-10T11:38:47.648 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:47.642 7f27c81d4700 1 -- start start 2021-02-10T11:38:47.648 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:47.643 7f27c81d4700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f27c0099d50 0x7f27c01b31b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:47.648 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:47.643 7f27c81d4700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f27c0187e50 0x7f27c0188250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:47.648 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:47.643 7f27c81d4700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f27c01b12a0 0x7f27c01b16a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:47.648 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:47.643 7f27c81d4700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27c0188870 con 0x7f27c0099d50 2021-02-10T11:38:47.648 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:47.643 7f27c81d4700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27c01b37d0 con 0x7f27c01b12a0 2021-02-10T11:38:47.648 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:47.643 7f27c81d4700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27c01b1cc0 con 0x7f27c0187e50 2021-02-10T11:38:47.648 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:47.643 7f27c5ff0700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f27c0099d50 0x7f27c01b31b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:47.648 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:47.643 7f27c5ff0700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f27c0099d50 0x7f27c01b31b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:39844/0 (socket says 172.21.2.102:39844) 2021-02-10T11:38:47.648 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:47.643 7f27c5ff0700 1 -- 172.21.2.102:0/3056918768 learned_addr learned my addr 172.21.2.102:0/3056918768 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:38:47.649 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:47.643 7f27c5ff0700 1 -- 172.21.2.102:0/3056918768 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f27c0187e50 msgr2=0x7f27c0188250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:47.649 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:47.643 7f27c5ff0700 1 --2- 172.21.2.102:0/3056918768 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f27c0187e50 0x7f27c0188250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:47.649 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:47.643 7f27c5ff0700 1 -- 172.21.2.102:0/3056918768 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f27c01b12a0 msgr2=0x7f27c01b16a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:47.649 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:47.643 7f27c5ff0700 1 --2- 172.21.2.102:0/3056918768 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f27c01b12a0 0x7f27c01b16a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:47.649 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:47.643 7f27c5ff0700 1 -- 172.21.2.102:0/3056918768 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27c01b1ee0 con 0x7f27c0099d50 2021-02-10T11:38:47.649 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:47.644 7f27c5ff0700 1 --2- 172.21.2.102:0/3056918768 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f27c0099d50 0x7f27c01b31b0 secure :-1 s=READY pgs=3143 cs=0 l=1 rev1=1 rx=0x7f27bc006f20 tx=0x7f27bc00b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:47.649 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:47.644 7f27b27fc700 1 -- 172.21.2.102:0/3056918768 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f27bc00bd80 con 0x7f27c0099d50 2021-02-10T11:38:47.649 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:47.644 7f27b27fc700 1 -- 172.21.2.102:0/3056918768 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27bc000c90 con 0x7f27c0099d50 2021-02-10T11:38:47.649 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:47.644 7f27b27fc700 1 -- 172.21.2.102:0/3056918768 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f27bc0022b0 con 0x7f27c0099d50 2021-02-10T11:38:47.649 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:47.644 7f27c81d4700 1 -- 172.21.2.102:0/3056918768 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27c01b1ee0 con 0x7f27c0099d50 2021-02-10T11:38:47.649 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:47.644 7f27c81d4700 1 -- 172.21.2.102:0/3056918768 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27c01b1ee0 con 0x7f27c0099d50 2021-02-10T11:38:47.649 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:47.645 7f27b27fc700 1 -- 172.21.2.102:0/3056918768 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f27bc009100 con 0x7f27c0099d50 2021-02-10T11:38:47.649 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:47.645 7f27b27fc700 1 --2- 172.21.2.102:0/3056918768 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f27a80186c0 0x7f27a801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:47.650 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:47.645 7f27b27fc700 1 -- 172.21.2.102:0/3056918768 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(180..180 src has 1..180) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f27bc0325b0 con 0x7f27c0099d50 2021-02-10T11:38:47.650 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:47.645 7f27c57ef700 1 --2- 172.21.2.102:0/3056918768 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f27a80186c0 0x7f27a801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:47.650 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:47.645 7f27c57ef700 1 --2- 172.21.2.102:0/3056918768 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f27a80186c0 0x7f27a801abc0 crc :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:47.677 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:47.675 7f27c81d4700 1 -- 172.21.2.102:0/3056918768 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27c009a1e0 con 0x7f27c0099d50 2021-02-10T11:38:47.680 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:47.678 7f27b27fc700 1 -- 172.21.2.102:0/3056918768 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f27bc006390 con 0x7f27c0099d50 2021-02-10T11:38:47.808 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:47.806 7f27c81d4700 1 -- 172.21.2.102:0/3056918768 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f27c009a1e0 con 0x7f27c0099d50 2021-02-10T11:38:47.808 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:47.806 7f27b27fc700 1 -- 172.21.2.102:0/3056918768 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 157 v157) v1 ==== 95+0+2107 (secure 0 0 0) 0x7f27bc00bd80 con 0x7f27c0099d50 2021-02-10T11:38:47.809 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:38:47.809 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":157,"default_fscid":19,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8815,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.102:6835/3396946217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6834","nonce":3396946217},{"type":"v1","addr":"172.21.2.102:6835","nonce":3396946217}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":153}],"filesystems":[{"mdsmap":{"epoch":157,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:38:30.335525","modified":"2021-02-10 11:38:32.341928","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8812},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8812":{"gid":8812,"name":"b","rank":0,"incarnation":154,"state":"up:active","state_seq":3,"addr":"172.21.2.113:6833/1513291768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6832","nonce":1513291768},{"type":"v1","addr":"172.21.2.113:6833","nonce":1513291768}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[39],"metadata_pool":38,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":19}]} 2021-02-10T11:38:47.809 INFO:teuthology.orchestra.run.gibba002.stderr:dumped fsmap epoch 157 2021-02-10T11:38:47.811 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:47.808 7f27c81d4700 1 -- 172.21.2.102:0/3056918768 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f27a80186c0 msgr2=0x7f27a801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:47.811 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:47.808 7f27c81d4700 1 --2- 172.21.2.102:0/3056918768 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f27a80186c0 0x7f27a801abc0 crc :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:47.811 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:47.808 7f27c81d4700 1 -- 172.21.2.102:0/3056918768 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f27c0099d50 msgr2=0x7f27c01b31b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:47.811 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:47.808 7f27c81d4700 1 --2- 172.21.2.102:0/3056918768 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f27c0099d50 0x7f27c01b31b0 secure :-1 s=READY pgs=3143 cs=0 l=1 rev1=1 rx=0x7f27bc006f20 tx=0x7f27bc00b990).stop 2021-02-10T11:38:47.811 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:47.808 7f27c81d4700 1 -- 172.21.2.102:0/3056918768 shutdown_connections 2021-02-10T11:38:47.811 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:47.808 7f27c81d4700 1 --2- 172.21.2.102:0/3056918768 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f27a80186c0 0x7f27a801abc0 unknown :-1 s=CLOSED pgs=1378 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:47.811 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:47.808 7f27c81d4700 1 --2- 172.21.2.102:0/3056918768 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f27c01b12a0 0x7f27c01b16a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:47.811 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:47.808 7f27c81d4700 1 --2- 172.21.2.102:0/3056918768 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f27c0187e50 0x7f27c0188250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:47.811 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:47.808 7f27c81d4700 1 --2- 172.21.2.102:0/3056918768 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f27c0099d50 0x7f27c01b31b0 unknown :-1 s=CLOSED pgs=3143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:47.811 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:47.808 7f27c81d4700 1 -- 172.21.2.102:0/3056918768 >> 172.21.2.102:0/3056918768 conn(0x7f27c019f8c0 msgr2=0x7f27c01a5fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:47.811 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:47.808 7f27c81d4700 1 -- 172.21.2.102:0/3056918768 shutdown_connections 2021-02-10T11:38:47.812 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:47.808 7f27c81d4700 1 -- 172.21.2.102:0/3056918768 wait complete. 2021-02-10T11:38:48.020 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 157, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:38:30.335525', 'modified': '2021-02-10 11:38:32.341928', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8812}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8812': {'gid': 8812, 'name': 'b', 'rank': 0, 'incarnation': 154, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.113:6833/1513291768', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.113:6832', 'nonce': 1513291768}, {'type': 'v1', 'addr': '172.21.2.113:6833', 'nonce': 1513291768}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [39], 'metadata_pool': 38, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:38:48.020 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:38:48.021 DEBUG:teuthology.orchestra.run.gibba013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:38:48.420 INFO:teuthology.orchestra.run.gibba013.stdout:{ 2021-02-10T11:38:48.421 INFO:teuthology.orchestra.run.gibba013.stdout: "cluster_fsid": "68db30ea-dbce-4aef-a6df-f01c6fea0513", 2021-02-10T11:38:48.421 INFO:teuthology.orchestra.run.gibba013.stdout: "whoami": 0, 2021-02-10T11:38:48.421 INFO:teuthology.orchestra.run.gibba013.stdout: "id": 8812, 2021-02-10T11:38:48.421 INFO:teuthology.orchestra.run.gibba013.stdout: "want_state": "up:active", 2021-02-10T11:38:48.421 INFO:teuthology.orchestra.run.gibba013.stdout: "state": "up:active", 2021-02-10T11:38:48.421 INFO:teuthology.orchestra.run.gibba013.stdout: "rank_uptime": 18.081625539000001, 2021-02-10T11:38:48.421 INFO:teuthology.orchestra.run.gibba013.stdout: "mdsmap_epoch": 157, 2021-02-10T11:38:48.421 INFO:teuthology.orchestra.run.gibba013.stdout: "osdmap_epoch": 180, 2021-02-10T11:38:48.421 INFO:teuthology.orchestra.run.gibba013.stdout: "osdmap_epoch_barrier": 180, 2021-02-10T11:38:48.421 INFO:teuthology.orchestra.run.gibba013.stdout: "uptime": 23.736866779 2021-02-10T11:38:48.421 INFO:teuthology.orchestra.run.gibba013.stdout:} 2021-02-10T11:38:48.425 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "68db30ea-dbce-4aef-a6df-f01c6fea0513", "whoami": 0, "id": 8812, "want_state": "up:active", "state": "up:active", "rank_uptime": 18.081625539000001, "mdsmap_epoch": 157, "osdmap_epoch": 180, "osdmap_epoch_barrier": 180, "uptime": 23.736866779 } 2021-02-10T11:38:48.425 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2021-02-10T11:38:48.425 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2021-02-10T11:38:48.425 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.1 at ubuntu@gibba002.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.1... 2021-02-10T11:38:48.425 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:38:48.425 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec mkdir -p /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:38:48.762 DEBUG:teuthology.orchestra.run.gibba002:> sudo modprobe fuse 2021-02-10T11:38:49.107 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:38:49.107 DEBUG:teuthology.orchestra.run.gibba002:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:38:49.452 INFO:teuthology.orchestra.run.gibba002.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:38:49.452 INFO:teuthology.orchestra.run.gibba002.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:38:49.453 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:38:49.453 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:38:49.453 DEBUG:teuthology.orchestra.run.gibba002:> ls /sys/fs/fuse/connections 2021-02-10T11:38:49.789 INFO:teuthology.orchestra.run.gibba002.stdout:43 2021-02-10T11:38:49.790 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [43] 2021-02-10T11:38:49.790 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.1 /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:38:49.793 DEBUG:teuthology.orchestra.run.gibba002:> sudo modprobe fuse 2021-02-10T11:38:50.138 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:38:50.138 DEBUG:teuthology.orchestra.run.gibba002:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:38:50.204 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba002.stderr:ceph-fuse[213188]: starting ceph client 2021-02-10T11:38:50.205 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba002.stderr:2021-02-10 11:38:50.202 7f251e1e9f80 -1 init, newargv = 0x55b0389a0220 newargc=9 2021-02-10T11:38:50.209 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba002.stderr:ceph-fuse[213188]: starting fuse 2021-02-10T11:38:50.481 INFO:teuthology.orchestra.run.gibba002.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:38:50.481 INFO:teuthology.orchestra.run.gibba002.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:38:50.482 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:38:50.482 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:38:50.482 DEBUG:teuthology.orchestra.run.gibba002:> ls /sys/fs/fuse/connections 2021-02-10T11:38:50.818 INFO:teuthology.orchestra.run.gibba002.stdout:43 2021-02-10T11:38:50.819 INFO:teuthology.orchestra.run.gibba002.stdout:44 2021-02-10T11:38:50.819 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43, 44] 2021-02-10T11:38:50.819 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:38:50.819 DEBUG:teuthology.orchestra.run.gibba002:> sudo python3 -c ' 2021-02-10T11:38:50.819 DEBUG:teuthology.orchestra.run.gibba002:> import glob 2021-02-10T11:38:50.819 DEBUG:teuthology.orchestra.run.gibba002:> import re 2021-02-10T11:38:50.819 DEBUG:teuthology.orchestra.run.gibba002:> import os 2021-02-10T11:38:50.819 DEBUG:teuthology.orchestra.run.gibba002:> import subprocess 2021-02-10T11:38:50.820 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:38:50.820 DEBUG:teuthology.orchestra.run.gibba002:> def find_socket(client_name): 2021-02-10T11:38:50.820 DEBUG:teuthology.orchestra.run.gibba002:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2021-02-10T11:38:50.820 DEBUG:teuthology.orchestra.run.gibba002:> files = glob.glob(asok_path) 2021-02-10T11:38:50.820 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:38:50.820 DEBUG:teuthology.orchestra.run.gibba002:> # Given a non-glob path, it better be there 2021-02-10T11:38:50.820 DEBUG:teuthology.orchestra.run.gibba002:> if "*" not in asok_path: 2021-02-10T11:38:50.820 DEBUG:teuthology.orchestra.run.gibba002:> assert(len(files) == 1) 2021-02-10T11:38:50.820 DEBUG:teuthology.orchestra.run.gibba002:> return files[0] 2021-02-10T11:38:50.820 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:38:50.820 DEBUG:teuthology.orchestra.run.gibba002:> for f in files: 2021-02-10T11:38:50.820 DEBUG:teuthology.orchestra.run.gibba002:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2021-02-10T11:38:50.820 DEBUG:teuthology.orchestra.run.gibba002:> if os.path.exists("/proc/{0}".format(pid)): 2021-02-10T11:38:50.820 DEBUG:teuthology.orchestra.run.gibba002:> return f 2021-02-10T11:38:50.820 DEBUG:teuthology.orchestra.run.gibba002:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2021-02-10T11:38:50.821 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:38:50.821 DEBUG:teuthology.orchestra.run.gibba002:> print(find_socket("client.1")) 2021-02-10T11:38:50.821 DEBUG:teuthology.orchestra.run.gibba002:> ' 2021-02-10T11:38:51.184 INFO:teuthology.orchestra.run.gibba002.stdout:/var/run/ceph/ceph-client.1.213188.asok 2021-02-10T11:38:51.187 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.213188.asok 2021-02-10T11:38:51.187 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:38:51.187 DEBUG:teuthology.orchestra.run.gibba002:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.213188.asok status 2021-02-10T11:38:51.579 INFO:teuthology.orchestra.run.gibba002.stdout:{ 2021-02-10T11:38:51.579 INFO:teuthology.orchestra.run.gibba002.stdout: "metadata": { 2021-02-10T11:38:51.580 INFO:teuthology.orchestra.run.gibba002.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:38:51.580 INFO:teuthology.orchestra.run.gibba002.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:38:51.580 INFO:teuthology.orchestra.run.gibba002.stdout: "entity_id": "1", 2021-02-10T11:38:51.580 INFO:teuthology.orchestra.run.gibba002.stdout: "hostname": "gibba002", 2021-02-10T11:38:51.580 INFO:teuthology.orchestra.run.gibba002.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2021-02-10T11:38:51.580 INFO:teuthology.orchestra.run.gibba002.stdout: "pid": "213188", 2021-02-10T11:38:51.580 INFO:teuthology.orchestra.run.gibba002.stdout: "root": "/" 2021-02-10T11:38:51.580 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:38:51.580 INFO:teuthology.orchestra.run.gibba002.stdout: "dentry_count": 0, 2021-02-10T11:38:51.580 INFO:teuthology.orchestra.run.gibba002.stdout: "dentry_pinned_count": 0, 2021-02-10T11:38:51.580 INFO:teuthology.orchestra.run.gibba002.stdout: "id": 13290, 2021-02-10T11:38:51.580 INFO:teuthology.orchestra.run.gibba002.stdout: "inst": { 2021-02-10T11:38:51.581 INFO:teuthology.orchestra.run.gibba002.stdout: "name": { 2021-02-10T11:38:51.581 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "client", 2021-02-10T11:38:51.581 INFO:teuthology.orchestra.run.gibba002.stdout: "num": 13290 2021-02-10T11:38:51.581 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:38:51.581 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": { 2021-02-10T11:38:51.581 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:38:51.581 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:0", 2021-02-10T11:38:51.581 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 2983720308 2021-02-10T11:38:51.581 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:38:51.581 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:38:51.581 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": { 2021-02-10T11:38:51.581 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:38:51.581 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:0", 2021-02-10T11:38:51.582 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 2983720308 2021-02-10T11:38:51.582 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:38:51.582 INFO:teuthology.orchestra.run.gibba002.stdout: "inst_str": "client.13290 172.21.2.102:0/2983720308", 2021-02-10T11:38:51.582 INFO:teuthology.orchestra.run.gibba002.stdout: "addr_str": "172.21.2.102:0/2983720308", 2021-02-10T11:38:51.582 INFO:teuthology.orchestra.run.gibba002.stdout: "inode_count": 1, 2021-02-10T11:38:51.582 INFO:teuthology.orchestra.run.gibba002.stdout: "mds_epoch": 157, 2021-02-10T11:38:51.582 INFO:teuthology.orchestra.run.gibba002.stdout: "osd_epoch": 180, 2021-02-10T11:38:51.582 INFO:teuthology.orchestra.run.gibba002.stdout: "osd_epoch_barrier": 0, 2021-02-10T11:38:51.582 INFO:teuthology.orchestra.run.gibba002.stdout: "blacklisted": false 2021-02-10T11:38:51.582 INFO:teuthology.orchestra.run.gibba002.stdout:} 2021-02-10T11:38:51.583 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:38:51.583 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:38:51.584 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:38:51.921 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:38:51.921 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2021-02-10T11:38:51.921 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:38:51.921 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:38:52.267 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2021-02-10T11:38:52.702 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:52.682 7f12b3f72700 1 Processor -- start 2021-02-10T11:38:52.703 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:52.682 7f12b3f72700 1 -- start start 2021-02-10T11:38:52.703 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:52.683 7f12b3f72700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f12ac0d1880 0x7f12ac1b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:52.703 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:52.683 7f12b3f72700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12ac0fe1d0 con 0x7f12ac0cff80 2021-02-10T11:38:52.703 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:52.683 7f12b3f72700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12ac109a70 con 0x7f12ac1b0af0 2021-02-10T11:38:52.703 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:52.683 7f12b3f72700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12ac088070 con 0x7f12ac0d1880 2021-02-10T11:38:52.703 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:52.683 7f12b258f700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f12ac0cff80 0x7f12ac109140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:60320/0 (socket says 172.21.2.102:60320) 2021-02-10T11:38:52.703 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:52.683 7f12b258f700 1 -- 172.21.2.102:0/3975895203 learned_addr learned my addr 172.21.2.102:0/3975895203 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:38:52.703 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:52.683 7f12b0d8c700 1 -- 172.21.2.102:0/3975895203 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2955943322 0 0) 0x7f12a4001500 con 0x7f12ac0cff80 2021-02-10T11:38:52.703 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:52.684 7f12b0d8c700 1 -- 172.21.2.102:0/3975895203 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1298002fe0 con 0x7f12ac0cff80 2021-02-10T11:38:52.703 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:52.684 7f12b1d8e700 1 --2- 172.21.2.102:0/3975895203 >> v2:172.21.2.113:3300/0 conn(0x7f12ac0d1880 0x7f12ac1b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:52.703 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:52.684 7f12b0d8c700 1 -- 172.21.2.102:0/3975895203 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3925942584 0 0) 0x7f12a4001460 con 0x7f12ac0cff80 2021-02-10T11:38:52.704 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:52.684 7f12b0d8c700 1 -- 172.21.2.102:0/3975895203 >> v2:172.21.2.113:3300/0 conn(0x7f12ac0d1880 msgr2=0x7f12ac1b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:52.704 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:52.684 7f12b0d8c700 1 --2- 172.21.2.102:0/3975895203 >> v2:172.21.2.113:3300/0 conn(0x7f12ac0d1880 0x7f12ac1b04d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:52.704 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:52.684 7f12b0d8c700 1 -- 172.21.2.102:0/3975895203 >> v1:172.21.2.113:6789/0 conn(0x7f12ac1b0af0 legacy=0x7f12ac108950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:52.704 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:52.684 7f12b0d8c700 1 -- 172.21.2.102:0/3975895203 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12ac088260 con 0x7f12ac0cff80 2021-02-10T11:38:52.704 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:52.684 7f12b0d8c700 1 -- 172.21.2.102:0/3975895203 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7f12a4001980 con 0x7f12ac0cff80 2021-02-10T11:38:52.704 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:52.684 7f12b0d8c700 1 -- 172.21.2.102:0/3975895203 >> v1:172.21.2.102:6789/0 conn(0x7f12ac0cff80 legacy=0x7f12ac109140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:52.704 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:52.684 7f12b0d8c700 1 --2- 172.21.2.102:0/3975895203 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1298003ae0 0x7f1298006000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:52.704 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:52.684 7f12b0d8c700 1 --2- 172.21.2.102:0/3975895203 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1298006820 0x7f1298008d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:52.704 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:52.684 7f12b0d8c700 1 --2- 172.21.2.102:0/3975895203 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1298009620 0x7f129800bb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:52.704 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:52.684 7f12b0d8c700 1 -- 172.21.2.102:0/3975895203 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f129800c310 con 0x7f1298006820 2021-02-10T11:38:52.704 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:52.684 7f12b0d8c700 1 -- 172.21.2.102:0/3975895203 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f129800c530 con 0x7f1298009620 2021-02-10T11:38:52.704 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:52.684 7f12b0d8c700 1 -- 172.21.2.102:0/3975895203 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f129800c750 con 0x7f1298003ae0 2021-02-10T11:38:52.704 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:52.684 7f12b258f700 1 --2- 172.21.2.102:0/3975895203 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1298009620 0x7f129800bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:52.705 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:52.684 7f12b158d700 1 --2- 172.21.2.102:0/3975895203 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1298006820 0x7f1298008d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:52.705 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:52.684 7f12b1d8e700 1 --2- 172.21.2.102:0/3975895203 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1298003ae0 0x7f1298006000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:52.705 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:52.684 7f12b158d700 1 -- 172.21.2.102:0/3975895203 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1298003ae0 msgr2=0x7f1298006000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:52.705 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:52.684 7f12b158d700 1 --2- 172.21.2.102:0/3975895203 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1298003ae0 0x7f1298006000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:52.705 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:52.684 7f12b158d700 1 -- 172.21.2.102:0/3975895203 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1298009620 msgr2=0x7f129800bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:52.705 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:52.684 7f12b158d700 1 --2- 172.21.2.102:0/3975895203 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1298009620 0x7f129800bb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:52.705 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:52.684 7f12b158d700 1 -- 172.21.2.102:0/3975895203 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12980509c0 con 0x7f1298006820 2021-02-10T11:38:52.705 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:52.684 7f12b258f700 1 -- 172.21.2.102:0/3975895203 reap_dead start 2021-02-10T11:38:52.705 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:52.685 7f12b158d700 1 --2- 172.21.2.102:0/3975895203 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1298006820 0x7f1298008d40 secure :-1 s=READY pgs=3147 cs=0 l=1 rev1=1 rx=0x7f12a0006ac0 tx=0x7f12a0009a70).ready entity=mon.0 client_cookie=3736ac5e7002eac0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:52.705 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:52.685 7f12b0d8c700 1 -- 172.21.2.102:0/3975895203 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f12a0007470 con 0x7f1298006820 2021-02-10T11:38:52.705 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:52.685 7f12b0d8c700 1 -- 172.21.2.102:0/3975895203 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12a0007770 con 0x7f1298006820 2021-02-10T11:38:52.705 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:52.685 7f12b0d8c700 1 -- 172.21.2.102:0/3975895203 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f12a0007c90 con 0x7f1298006820 2021-02-10T11:38:52.706 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:52.685 7f12b3f72700 1 -- 172.21.2.102:0/3975895203 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1298006820 msgr2=0x7f1298008d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:52.706 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:52.685 7f12b3f72700 1 --2- 172.21.2.102:0/3975895203 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1298006820 0x7f1298008d40 secure :-1 s=READY pgs=3147 cs=0 l=1 rev1=1 rx=0x7f12a0006ac0 tx=0x7f12a0009a70).stop 2021-02-10T11:38:52.706 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:52.685 7f12b3f72700 1 -- 172.21.2.102:0/3975895203 shutdown_connections 2021-02-10T11:38:52.706 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:52.685 7f12b3f72700 1 --2- 172.21.2.102:0/3975895203 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1298006820 0x7f1298008d40 unknown :-1 s=CLOSED pgs=3147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:52.706 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:52.685 7f12b3f72700 1 -- 172.21.2.102:0/3975895203 >> 172.21.2.102:0/3975895203 conn(0x7f12ac0ce430 msgr2=0x7f12ac0d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:52.706 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:52.686 7f12b3f72700 1 -- 172.21.2.102:0/3975895203 shutdown_connections 2021-02-10T11:38:52.706 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:52.686 7f12b3f72700 1 -- 172.21.2.102:0/3975895203 wait complete. 2021-02-10T11:38:52.706 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:52.700 7f12b3f72700 1 Processor -- start 2021-02-10T11:38:52.706 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:52.700 7f12b3f72700 1 -- start start 2021-02-10T11:38:52.708 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:52.701 7f12b3f72700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f12ac188400 0x7f12ac188800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:52.708 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:52.701 7f12b3f72700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f12ac188e20 0x7f12ac189220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:52.708 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:52.701 7f12b3f72700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f12ac09cbc0 0x7f12ac09cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:52.708 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:52.701 7f12b3f72700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12ac187e80 con 0x7f12ac188e20 2021-02-10T11:38:52.708 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:52.701 7f12b3f72700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12ac0b27c0 con 0x7f12ac09cbc0 2021-02-10T11:38:52.708 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:52.701 7f12b3f72700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12ac0b29a0 con 0x7f12ac188400 2021-02-10T11:38:52.708 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:52.701 7f12b158d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f12ac188e20 0x7f12ac189220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:52.708 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:52.701 7f12b158d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f12ac188e20 0x7f12ac189220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:39874/0 (socket says 172.21.2.102:39874) 2021-02-10T11:38:52.708 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:52.701 7f12b158d700 1 -- 172.21.2.102:0/1689676396 learned_addr learned my addr 172.21.2.102:0/1689676396 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:38:52.708 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:52.701 7f12b158d700 1 -- 172.21.2.102:0/1689676396 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f12ac188400 msgr2=0x7f12ac188800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:52.708 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:52.701 7f12b158d700 1 --2- 172.21.2.102:0/1689676396 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f12ac188400 0x7f12ac188800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:52.708 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:52.701 7f12b158d700 1 -- 172.21.2.102:0/1689676396 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f12ac09cbc0 msgr2=0x7f12ac09cfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:52.709 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:52.701 7f12b158d700 1 --2- 172.21.2.102:0/1689676396 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f12ac09cbc0 0x7f12ac09cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:52.709 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:52.701 7f12b158d700 1 -- 172.21.2.102:0/1689676396 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12ac09d5e0 con 0x7f12ac188e20 2021-02-10T11:38:52.709 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:52.701 7f12b158d700 1 --2- 172.21.2.102:0/1689676396 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f12ac188e20 0x7f12ac189220 secure :-1 s=READY pgs=3148 cs=0 l=1 rev1=1 rx=0x7f12a000b6b0 tx=0x7f12a000d910).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:52.709 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:52.702 7f129e7fc700 1 -- 172.21.2.102:0/1689676396 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f12a0006b50 con 0x7f12ac188e20 2021-02-10T11:38:52.709 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:52.702 7f129e7fc700 1 -- 172.21.2.102:0/1689676396 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12a000b4d0 con 0x7f12ac188e20 2021-02-10T11:38:52.709 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:52.702 7f12b3f72700 1 -- 172.21.2.102:0/1689676396 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12ac09d5e0 con 0x7f12ac188e20 2021-02-10T11:38:52.709 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:52.702 7f129e7fc700 1 -- 172.21.2.102:0/1689676396 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f12a0009890 con 0x7f12ac188e20 2021-02-10T11:38:52.709 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:52.702 7f12b3f72700 1 -- 172.21.2.102:0/1689676396 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12ac09d5e0 con 0x7f12ac188e20 2021-02-10T11:38:52.709 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:52.702 7f129e7fc700 1 -- 172.21.2.102:0/1689676396 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f12a0002050 con 0x7f12ac188e20 2021-02-10T11:38:52.709 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:52.702 7f129e7fc700 1 --2- 172.21.2.102:0/1689676396 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1294018730 0x7f129401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:52.709 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:52.702 7f129e7fc700 1 -- 172.21.2.102:0/1689676396 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(180..180 src has 1..180) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f12a0032620 con 0x7f12ac188e20 2021-02-10T11:38:52.709 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:52.703 7f12b1d8e700 1 --2- 172.21.2.102:0/1689676396 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1294018730 0x7f129401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:52.709 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:52.703 7f12b1d8e700 1 --2- 172.21.2.102:0/1689676396 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1294018730 0x7f129401abc0 crc :-1 s=READY pgs=1379 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:52.735 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:52.733 7f12b3f72700 1 -- 172.21.2.102:0/1689676396 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12ac0b29a0 con 0x7f12ac188e20 2021-02-10T11:38:52.737 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:52.735 7f129e7fc700 1 -- 172.21.2.102:0/1689676396 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f12a0006b50 con 0x7f12ac188e20 2021-02-10T11:38:52.873 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:52.871 7f12b3f72700 1 -- 172.21.2.102:0/1689676396 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f12ac0b29a0 con 0x7f12ac188e20 2021-02-10T11:38:53.193 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:53.190 7f129e7fc700 1 -- 172.21.2.102:0/1689676396 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v158) v1 ==== 136+0+0 (secure 0 0 0) 0x7f12a0002050 con 0x7f12ac188e20 2021-02-10T11:38:53.242 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:53.239 7f12b3f72700 1 -- 172.21.2.102:0/1689676396 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1294018730 msgr2=0x7f129401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:53.242 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:53.239 7f12b3f72700 1 --2- 172.21.2.102:0/1689676396 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1294018730 0x7f129401abc0 crc :-1 s=READY pgs=1379 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:53.242 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:53.239 7f12b3f72700 1 -- 172.21.2.102:0/1689676396 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f12ac188e20 msgr2=0x7f12ac189220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:53.242 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:53.239 7f12b3f72700 1 --2- 172.21.2.102:0/1689676396 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f12ac188e20 0x7f12ac189220 secure :-1 s=READY pgs=3148 cs=0 l=1 rev1=1 rx=0x7f12a000b6b0 tx=0x7f12a000d910).stop 2021-02-10T11:38:53.242 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:53.240 7f12b3f72700 1 -- 172.21.2.102:0/1689676396 shutdown_connections 2021-02-10T11:38:53.242 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:53.240 7f12b3f72700 1 --2- 172.21.2.102:0/1689676396 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1294018730 0x7f129401abc0 unknown :-1 s=CLOSED pgs=1379 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:53.243 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:53.240 7f12b3f72700 1 --2- 172.21.2.102:0/1689676396 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f12ac09cbc0 0x7f12ac09cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:53.243 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:53.240 7f12b3f72700 1 --2- 172.21.2.102:0/1689676396 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f12ac188e20 0x7f12ac189220 unknown :-1 s=CLOSED pgs=3148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:53.243 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:53.240 7f12b3f72700 1 --2- 172.21.2.102:0/1689676396 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f12ac188400 0x7f12ac188800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:53.243 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:53.240 7f12b3f72700 1 -- 172.21.2.102:0/1689676396 >> 172.21.2.102:0/1689676396 conn(0x7f12ac0c2cd0 msgr2=0x7f12ac0b5950 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:53.243 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:53.240 7f12b3f72700 1 -- 172.21.2.102:0/1689676396 shutdown_connections 2021-02-10T11:38:53.243 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:53.240 7f12b3f72700 1 -- 172.21.2.102:0/1689676396 wait complete. 2021-02-10T11:38:53.249 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs volume ls 2021-02-10T11:38:53.678 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:53.660 7f768568f700 1 Processor -- start 2021-02-10T11:38:53.678 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:53.660 7f768568f700 1 -- start start 2021-02-10T11:38:53.679 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:53.660 7f768568f700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f76800d0340 0x7f76800ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:53.679 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:53.660 7f768568f700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f76800d1c70 0x7f76800efc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:53.679 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:53.660 7f768568f700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76800f0b10 con 0x7f76800d1870 2021-02-10T11:38:53.679 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:53.660 7f768568f700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76800e5ed0 con 0x7f76800d1c70 2021-02-10T11:38:53.679 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:53.660 7f768568f700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76801b0be0 con 0x7f76800d0340 2021-02-10T11:38:53.679 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:53.660 7f767fac0700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f76800d1c70 0x7f76800efc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:53.679 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:53.660 7f767fac0700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f76800d1c70 0x7f76800efc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:39886/0 (socket says 172.21.2.102:39886) 2021-02-10T11:38:53.679 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:53.660 7f767fac0700 1 -- 172.21.2.102:0/2407149554 learned_addr learned my addr 172.21.2.102:0/2407149554 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:38:53.679 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:53.661 7f767fac0700 1 -- 172.21.2.102:0/2407149554 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f76800d0340 msgr2=0x7f76800ef640 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:38:53.679 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:53.661 7f767fac0700 1 --2- 172.21.2.102:0/2407149554 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f76800d0340 0x7f76800ef640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:53.679 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:53.661 7f767fac0700 1 -- 172.21.2.102:0/2407149554 >> v1:172.21.2.113:6789/0 conn(0x7f76800d1870 legacy=0x7f76801b03f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:38:53.679 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:53.661 7f767fac0700 1 -- 172.21.2.102:0/2407149554 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76800f20a0 con 0x7f76800d1c70 2021-02-10T11:38:53.680 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:53.661 7f767fac0700 1 --2- 172.21.2.102:0/2407149554 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f76800d1c70 0x7f76800efc60 secure :-1 s=READY pgs=3149 cs=0 l=1 rev1=1 rx=0x7f7678006f00 tx=0x7f7678009f10).ready entity=mon.0 client_cookie=ad6ab4763b334b2e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:53.680 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:53.661 7f767e2bd700 1 -- 172.21.2.102:0/2407149554 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f767800b3b0 con 0x7f76800d1c70 2021-02-10T11:38:53.680 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:53.661 7f767e2bd700 1 -- 172.21.2.102:0/2407149554 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f767800b630 con 0x7f76800d1c70 2021-02-10T11:38:53.680 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:53.661 7f767e2bd700 1 -- 172.21.2.102:0/2407149554 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f767800bb50 con 0x7f76800d1c70 2021-02-10T11:38:53.680 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:53.661 7f768568f700 1 -- 172.21.2.102:0/2407149554 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f76800d1c70 msgr2=0x7f76800efc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:53.680 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:53.661 7f768568f700 1 --2- 172.21.2.102:0/2407149554 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f76800d1c70 0x7f76800efc60 secure :-1 s=READY pgs=3149 cs=0 l=1 rev1=1 rx=0x7f7678006f00 tx=0x7f7678009f10).stop 2021-02-10T11:38:53.680 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:53.662 7f768568f700 1 -- 172.21.2.102:0/2407149554 shutdown_connections 2021-02-10T11:38:53.680 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:53.662 7f768568f700 1 --2- 172.21.2.102:0/2407149554 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f76800d0340 0x7f76800ef640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:53.680 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:53.662 7f768568f700 1 --2- 172.21.2.102:0/2407149554 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f76800d1c70 0x7f76800efc60 unknown :-1 s=CLOSED pgs=3149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:53.680 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:53.662 7f768568f700 1 -- 172.21.2.102:0/2407149554 >> 172.21.2.102:0/2407149554 conn(0x7f76800ce450 msgr2=0x7f76800d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:53.680 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:53.662 7f768568f700 1 -- 172.21.2.102:0/2407149554 shutdown_connections 2021-02-10T11:38:53.680 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:53.662 7f768568f700 1 -- 172.21.2.102:0/2407149554 wait complete. 2021-02-10T11:38:53.680 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:53.676 7f768568f700 1 Processor -- start 2021-02-10T11:38:53.681 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:53.676 7f768568f700 1 -- start start 2021-02-10T11:38:53.682 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:53.677 7f768568f700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f76801908b0 0x7f76801a4440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:53.682 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:53.677 7f768568f700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f76800bd920 0x7f76800bdd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:53.682 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:53.677 7f768568f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f768008d510 0x7f768008d910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:53.682 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:53.677 7f768568f700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f768008cf90 con 0x7f768008d510 2021-02-10T11:38:53.683 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:53.677 7f768568f700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76800be340 con 0x7f76801908b0 2021-02-10T11:38:53.683 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:53.677 7f768568f700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76800be5f0 con 0x7f76800bd920 2021-02-10T11:38:53.683 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:53.677 7f767fac0700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f768008d510 0x7f768008d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:53.683 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:53.677 7f767fac0700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f768008d510 0x7f768008d910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:39892/0 (socket says 172.21.2.102:39892) 2021-02-10T11:38:53.683 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:53.677 7f767fac0700 1 -- 172.21.2.102:0/2843726928 learned_addr learned my addr 172.21.2.102:0/2843726928 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:38:53.683 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:53.677 7f767eabe700 1 --2- 172.21.2.102:0/2843726928 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f76800bd920 0x7f76800bdd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:53.683 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:53.677 7f767f2bf700 1 --2- 172.21.2.102:0/2843726928 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f76801908b0 0x7f76801a4440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:53.683 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:53.677 7f767fac0700 1 -- 172.21.2.102:0/2843726928 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f76800bd920 msgr2=0x7f76800bdd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:53.683 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:53.677 7f767fac0700 1 --2- 172.21.2.102:0/2843726928 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f76800bd920 0x7f76800bdd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:53.683 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:53.677 7f767fac0700 1 -- 172.21.2.102:0/2843726928 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f76801908b0 msgr2=0x7f76801a4440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:53.683 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:53.677 7f767fac0700 1 --2- 172.21.2.102:0/2843726928 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f76801908b0 0x7f76801a4440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:53.683 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:53.677 7f767fac0700 1 -- 172.21.2.102:0/2843726928 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76801a4a60 con 0x7f768008d510 2021-02-10T11:38:53.683 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:53.677 7f767eabe700 1 --2- 172.21.2.102:0/2843726928 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f76800bd920 0x7f76800bdd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:38:53.684 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:53.677 7f767fac0700 1 --2- 172.21.2.102:0/2843726928 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f768008d510 0x7f768008d910 secure :-1 s=READY pgs=3150 cs=0 l=1 rev1=1 rx=0x7f76780099f0 tx=0x7f767800bcc0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:53.684 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:53.678 7f7667fff700 1 -- 172.21.2.102:0/2843726928 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f767800be00 con 0x7f768008d510 2021-02-10T11:38:53.684 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:53.678 7f7667fff700 1 -- 172.21.2.102:0/2843726928 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7678000cc0 con 0x7f768008d510 2021-02-10T11:38:53.684 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:53.678 7f7667fff700 1 -- 172.21.2.102:0/2843726928 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7678005a10 con 0x7f768008d510 2021-02-10T11:38:53.684 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:53.678 7f768568f700 1 -- 172.21.2.102:0/2843726928 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76801a4a60 con 0x7f768008d510 2021-02-10T11:38:53.684 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:53.678 7f768568f700 1 -- 172.21.2.102:0/2843726928 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f768008df30 con 0x7f768008d510 2021-02-10T11:38:53.684 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:53.678 7f7667fff700 1 -- 172.21.2.102:0/2843726928 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f7678005820 con 0x7f768008d510 2021-02-10T11:38:53.684 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:53.678 7f7667fff700 1 --2- 172.21.2.102:0/2843726928 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f76600186c0 0x7f766001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:53.684 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:53.678 7f7667fff700 1 -- 172.21.2.102:0/2843726928 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(180..180 src has 1..180) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f7678032b70 con 0x7f768008d510 2021-02-10T11:38:53.684 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:53.678 7f767f2bf700 1 --2- 172.21.2.102:0/2843726928 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f76600186c0 0x7f766001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:53.684 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:53.679 7f767f2bf700 1 --2- 172.21.2.102:0/2843726928 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f76600186c0 0x7f766001abc0 crc :-1 s=READY pgs=1380 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:53.713 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:53.710 7f768568f700 1 -- 172.21.2.102:0/2843726928 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76800ebc50 con 0x7f768008d510 2021-02-10T11:38:53.715 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:53.713 7f7667fff700 1 -- 172.21.2.102:0/2843726928 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f767800be00 con 0x7f768008d510 2021-02-10T11:38:53.850 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:53.848 7f768568f700 1 -- 172.21.2.102:0/2843726928 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs volume ls", "target": ["mgr", ""]}) v1 -- 0x7f768008df30 con 0x7f76600186c0 2021-02-10T11:38:53.850 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:53.848 7f7667fff700 1 -- 172.21.2.102:0/2843726928 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+41 (crc 0 0 0) 0x7f7670006040 con 0x7f76600186c0 2021-02-10T11:38:53.851 INFO:teuthology.orchestra.run.gibba002.stdout:[ 2021-02-10T11:38:53.851 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:38:53.851 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "cephfs" 2021-02-10T11:38:53.851 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:38:53.851 INFO:teuthology.orchestra.run.gibba002.stdout:] 2021-02-10T11:38:53.852 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:53.850 7f768568f700 1 -- 172.21.2.102:0/2843726928 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f76600186c0 msgr2=0x7f766001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:53.852 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:53.850 7f768568f700 1 --2- 172.21.2.102:0/2843726928 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f76600186c0 0x7f766001abc0 crc :-1 s=READY pgs=1380 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:53.852 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:53.850 7f768568f700 1 -- 172.21.2.102:0/2843726928 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f768008d510 msgr2=0x7f768008d910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:53.852 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:53.850 7f768568f700 1 --2- 172.21.2.102:0/2843726928 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f768008d510 0x7f768008d910 secure :-1 s=READY pgs=3150 cs=0 l=1 rev1=1 rx=0x7f76780099f0 tx=0x7f767800bcc0).stop 2021-02-10T11:38:53.852 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:53.850 7f768568f700 1 -- 172.21.2.102:0/2843726928 shutdown_connections 2021-02-10T11:38:53.852 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:53.850 7f768568f700 1 --2- 172.21.2.102:0/2843726928 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f768008d510 0x7f768008d910 unknown :-1 s=CLOSED pgs=3150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:53.852 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:53.850 7f768568f700 1 --2- 172.21.2.102:0/2843726928 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f76800bd920 0x7f76800bdd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:53.852 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:53.850 7f768568f700 1 --2- 172.21.2.102:0/2843726928 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f76600186c0 0x7f766001abc0 unknown :-1 s=CLOSED pgs=1380 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:53.853 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:53.850 7f768568f700 1 --2- 172.21.2.102:0/2843726928 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f76801908b0 0x7f76801a4440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:53.853 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:53.850 7f768568f700 1 -- 172.21.2.102:0/2843726928 >> 172.21.2.102:0/2843726928 conn(0x7f768008f030 msgr2=0x7f76800b5e80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:53.853 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:53.850 7f768568f700 1 -- 172.21.2.102:0/2843726928 shutdown_connections 2021-02-10T11:38:53.853 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:53.850 7f768568f700 1 -- 172.21.2.102:0/2843726928 wait complete. 2021-02-10T11:38:53.860 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config set mon mon_allow_pool_delete True 2021-02-10T11:38:54.295 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:54.276 7fee82493700 1 Processor -- start 2021-02-10T11:38:54.295 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:54.276 7fee82493700 1 -- start start 2021-02-10T11:38:54.295 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:54.277 7fee82493700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fee7c0d1870 0x7fee7c1b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:54.295 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:54.277 7fee82493700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fee7c1b0ae0 0x7fee7c108940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:54.295 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:54.277 7fee82493700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee7c0fe1c0 con 0x7fee7c0d1c70 2021-02-10T11:38:54.296 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:54.277 7fee82493700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee7c109a60 con 0x7fee7c1b0ae0 2021-02-10T11:38:54.296 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:54.277 7fee82493700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee7c109750 con 0x7fee7c0d1870 2021-02-10T11:38:54.296 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:54.277 7fee7b7fe700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fee7c1b0ae0 0x7fee7c108940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:54.296 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:54.277 7fee7b7fe700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fee7c1b0ae0 0x7fee7c108940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:39900/0 (socket says 172.21.2.102:39900) 2021-02-10T11:38:54.296 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:54.277 7fee7b7fe700 1 -- 172.21.2.102:0/3245994383 learned_addr learned my addr 172.21.2.102:0/3245994383 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:38:54.296 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:54.277 7fee7bfff700 1 --2- 172.21.2.102:0/3245994383 >> v2:172.21.2.113:3300/0 conn(0x7fee7c0d1870 0x7fee7c1b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:54.296 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:54.277 7fee7b7fe700 1 -- 172.21.2.102:0/3245994383 >> v2:172.21.2.113:3300/0 conn(0x7fee7c0d1870 msgr2=0x7fee7c1b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:54.296 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:54.277 7fee7b7fe700 1 --2- 172.21.2.102:0/3245994383 >> v2:172.21.2.113:3300/0 conn(0x7fee7c0d1870 0x7fee7c1b04c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:54.296 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:54.278 7fee7b7fe700 1 -- 172.21.2.102:0/3245994383 >> v1:172.21.2.113:6789/0 conn(0x7fee7c0d1c70 legacy=0x7fee7c108f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:54.296 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:54.278 7fee7b7fe700 1 -- 172.21.2.102:0/3245994383 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee7c088070 con 0x7fee7c1b0ae0 2021-02-10T11:38:54.296 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:54.278 7fee7bfff700 1 --2- 172.21.2.102:0/3245994383 >> v2:172.21.2.113:3300/0 conn(0x7fee7c0d1870 0x7fee7c1b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:38:54.296 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:54.278 7fee7b7fe700 1 --2- 172.21.2.102:0/3245994383 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fee7c1b0ae0 0x7fee7c108940 secure :-1 s=READY pgs=3151 cs=0 l=1 rev1=1 rx=0x7fee70006700 tx=0x7fee70009b80).ready entity=mon.0 client_cookie=c3ebfa5a6355c578 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:54.296 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:54.278 7fee7affd700 1 -- 172.21.2.102:0/3245994383 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fee70006340 con 0x7fee7c1b0ae0 2021-02-10T11:38:54.297 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:54.278 7fee7affd700 1 -- 172.21.2.102:0/3245994383 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee7000c0c0 con 0x7fee7c1b0ae0 2021-02-10T11:38:54.297 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:54.278 7fee7affd700 1 -- 172.21.2.102:0/3245994383 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fee7000c600 con 0x7fee7c1b0ae0 2021-02-10T11:38:54.297 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:54.278 7fee82493700 1 -- 172.21.2.102:0/3245994383 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fee7c1b0ae0 msgr2=0x7fee7c108940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:54.297 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:54.278 7fee82493700 1 --2- 172.21.2.102:0/3245994383 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fee7c1b0ae0 0x7fee7c108940 secure :-1 s=READY pgs=3151 cs=0 l=1 rev1=1 rx=0x7fee70006700 tx=0x7fee70009b80).stop 2021-02-10T11:38:54.297 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:54.278 7fee82493700 1 -- 172.21.2.102:0/3245994383 shutdown_connections 2021-02-10T11:38:54.297 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:54.279 7fee82493700 1 --2- 172.21.2.102:0/3245994383 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fee7c1b0ae0 0x7fee7c108940 unknown :-1 s=CLOSED pgs=3151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:54.297 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:54.279 7fee82493700 1 --2- 172.21.2.102:0/3245994383 >> v2:172.21.2.113:3300/0 conn(0x7fee7c0d1870 0x7fee7c1b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:54.297 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:54.279 7fee82493700 1 -- 172.21.2.102:0/3245994383 >> 172.21.2.102:0/3245994383 conn(0x7fee7c0ce420 msgr2=0x7fee7c0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:54.297 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:54.279 7fee82493700 1 -- 172.21.2.102:0/3245994383 shutdown_connections 2021-02-10T11:38:54.297 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:54.279 7fee82493700 1 -- 172.21.2.102:0/3245994383 wait complete. 2021-02-10T11:38:54.299 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:54.293 7fee82493700 1 Processor -- start 2021-02-10T11:38:54.299 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:54.293 7fee82493700 1 -- start start 2021-02-10T11:38:54.299 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:54.294 7fee82493700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fee7c0c2370 0x7fee7c18f870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:54.299 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:54.294 7fee82493700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fee7c084cb0 0x7fee7c1a7470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:54.299 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:54.294 7fee82493700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fee7c1a7a90 0x7fee7c085700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:54.299 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:54.294 7fee82493700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee7c0be840 con 0x7fee7c084cb0 2021-02-10T11:38:54.299 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:54.294 7fee82493700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee7c0c29a0 con 0x7fee7c1a7a90 2021-02-10T11:38:54.300 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:54.294 7fee82493700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee7c1a7e90 con 0x7fee7c0c2370 2021-02-10T11:38:54.300 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:54.294 7fee7b7fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fee7c084cb0 0x7fee7c1a7470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:54.300 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:54.294 7fee7b7fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fee7c084cb0 0x7fee7c1a7470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:39904/0 (socket says 172.21.2.102:39904) 2021-02-10T11:38:54.300 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:54.294 7fee7b7fe700 1 -- 172.21.2.102:0/2359595865 learned_addr learned my addr 172.21.2.102:0/2359595865 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:38:54.300 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:54.294 7fee7bfff700 1 --2- 172.21.2.102:0/2359595865 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fee7c0c2370 0x7fee7c18f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:54.300 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:54.294 7fee80ab0700 1 --2- 172.21.2.102:0/2359595865 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fee7c1a7a90 0x7fee7c085700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:54.300 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:54.294 7fee7b7fe700 1 -- 172.21.2.102:0/2359595865 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fee7c0c2370 msgr2=0x7fee7c18f870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:54.300 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:54.294 7fee7b7fe700 1 --2- 172.21.2.102:0/2359595865 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fee7c0c2370 0x7fee7c18f870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:54.300 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:54.294 7fee7b7fe700 1 -- 172.21.2.102:0/2359595865 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fee7c1a7a90 msgr2=0x7fee7c085700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:54.301 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:54.294 7fee7b7fe700 1 --2- 172.21.2.102:0/2359595865 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fee7c1a7a90 0x7fee7c085700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:54.301 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:54.294 7fee7b7fe700 1 -- 172.21.2.102:0/2359595865 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee7c1a80b0 con 0x7fee7c084cb0 2021-02-10T11:38:54.301 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:54.295 7fee7b7fe700 1 --2- 172.21.2.102:0/2359595865 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fee7c084cb0 0x7fee7c1a7470 secure :-1 s=READY pgs=3152 cs=0 l=1 rev1=1 rx=0x7fee70009720 tx=0x7fee7000b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:54.301 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:54.295 7fee78ff9700 1 -- 172.21.2.102:0/2359595865 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fee700090c0 con 0x7fee7c084cb0 2021-02-10T11:38:54.301 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:54.295 7fee78ff9700 1 -- 172.21.2.102:0/2359595865 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee70002030 con 0x7fee7c084cb0 2021-02-10T11:38:54.301 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:54.295 7fee78ff9700 1 -- 172.21.2.102:0/2359595865 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fee700024c0 con 0x7fee7c084cb0 2021-02-10T11:38:54.301 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:54.295 7fee82493700 1 -- 172.21.2.102:0/2359595865 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee7c1a80b0 con 0x7fee7c084cb0 2021-02-10T11:38:54.302 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:54.295 7fee82493700 1 -- 172.21.2.102:0/2359595865 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee7c086010 con 0x7fee7c084cb0 2021-02-10T11:38:54.302 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:54.295 7fee78ff9700 1 -- 172.21.2.102:0/2359595865 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fee70000c90 con 0x7fee7c084cb0 2021-02-10T11:38:54.302 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:54.296 7fee78ff9700 1 --2- 172.21.2.102:0/2359595865 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fee5c0186c0 0x7fee5c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:54.302 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:54.296 7fee78ff9700 1 -- 172.21.2.102:0/2359595865 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(180..180 src has 1..180) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fee70032540 con 0x7fee7c084cb0 2021-02-10T11:38:54.302 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:54.296 7fee7bfff700 1 --2- 172.21.2.102:0/2359595865 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fee5c0186c0 0x7fee5c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:54.302 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:54.296 7fee7bfff700 1 --2- 172.21.2.102:0/2359595865 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fee5c0186c0 0x7fee5c01abc0 crc :-1 s=READY pgs=1381 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:54.329 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:54.327 7fee82493700 1 -- 172.21.2.102:0/2359595865 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee7c082a70 con 0x7fee7c084cb0 2021-02-10T11:38:54.331 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:54.329 7fee78ff9700 1 -- 172.21.2.102:0/2359595865 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fee70001f10 con 0x7fee7c084cb0 2021-02-10T11:38:54.481 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:54.478 7fee82493700 1 -- 172.21.2.102:0/2359595865 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fee7c082a70 con 0x7fee7c084cb0 2021-02-10T11:38:54.481 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:54.479 7fee78ff9700 1 -- 172.21.2.102:0/2359595865 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v44)=0 v44) v1 ==== 122+0+0 (secure 0 0 0) 0x7fee70001f10 con 0x7fee7c084cb0 2021-02-10T11:38:54.483 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:54.481 7fee82493700 1 -- 172.21.2.102:0/2359595865 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fee5c0186c0 msgr2=0x7fee5c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:54.483 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:54.481 7fee82493700 1 --2- 172.21.2.102:0/2359595865 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fee5c0186c0 0x7fee5c01abc0 crc :-1 s=READY pgs=1381 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:54.483 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:54.481 7fee82493700 1 -- 172.21.2.102:0/2359595865 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fee7c084cb0 msgr2=0x7fee7c1a7470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:54.483 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:54.481 7fee82493700 1 --2- 172.21.2.102:0/2359595865 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fee7c084cb0 0x7fee7c1a7470 secure :-1 s=READY pgs=3152 cs=0 l=1 rev1=1 rx=0x7fee70009720 tx=0x7fee7000b960).stop 2021-02-10T11:38:54.483 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:54.481 7fee82493700 1 -- 172.21.2.102:0/2359595865 shutdown_connections 2021-02-10T11:38:54.483 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:54.481 7fee82493700 1 --2- 172.21.2.102:0/2359595865 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fee5c0186c0 0x7fee5c01abc0 unknown :-1 s=CLOSED pgs=1381 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:54.483 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:54.481 7fee82493700 1 --2- 172.21.2.102:0/2359595865 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fee7c1a7a90 0x7fee7c085700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:54.483 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:54.481 7fee82493700 1 --2- 172.21.2.102:0/2359595865 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fee7c084cb0 0x7fee7c1a7470 unknown :-1 s=CLOSED pgs=3152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:54.484 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:54.481 7fee82493700 1 --2- 172.21.2.102:0/2359595865 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fee7c0c2370 0x7fee7c18f870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:54.484 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:54.481 7fee82493700 1 -- 172.21.2.102:0/2359595865 >> 172.21.2.102:0/2359595865 conn(0x7fee7c0b5dc0 msgr2=0x7fee7c0c2d90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:54.484 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:54.481 7fee82493700 1 -- 172.21.2.102:0/2359595865 shutdown_connections 2021-02-10T11:38:54.484 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:54.481 7fee82493700 1 -- 172.21.2.102:0/2359595865 wait complete. 2021-02-10T11:38:54.492 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolumegroup create cephfs group_0000000000453502 2021-02-10T11:38:54.927 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:54.907 7fc04276d700 1 Processor -- start 2021-02-10T11:38:54.927 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:54.907 7fc04276d700 1 -- start start 2021-02-10T11:38:54.927 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:54.908 7fc04276d700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fc03c0d1870 0x7fc03c1b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:54.927 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:54.908 7fc04276d700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc03c0cff70 0x7fc03c109130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:54.928 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:54.908 7fc04276d700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc03c109a60 con 0x7fc03c1b0ae0 2021-02-10T11:38:54.928 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:54.908 7fc04276d700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc03c109750 con 0x7fc03c0d1870 2021-02-10T11:38:54.928 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:54.908 7fc04276d700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc03c088070 con 0x7fc03c0cff70 2021-02-10T11:38:54.928 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:54.908 7fc03b7fe700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fc03c1b0ae0 0x7fc03c108940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:60368/0 (socket says 172.21.2.102:60368) 2021-02-10T11:38:54.928 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:54.908 7fc03b7fe700 1 -- 172.21.2.102:0/385630106 learned_addr learned my addr 172.21.2.102:0/385630106 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:38:54.928 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:54.908 7fc03affd700 1 -- 172.21.2.102:0/385630106 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2412568590 0 0) 0x7fc0300009b0 con 0x7fc03c1b0ae0 2021-02-10T11:38:54.928 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:54.908 7fc03affd700 1 -- 172.21.2.102:0/385630106 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc024002fe0 con 0x7fc03c1b0ae0 2021-02-10T11:38:54.928 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:54.908 7fc03bfff700 1 --2- 172.21.2.102:0/385630106 >> v2:172.21.2.113:3300/0 conn(0x7fc03c0d1870 0x7fc03c1b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:54.931 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:54.908 7fc040d8a700 1 --2- 172.21.2.102:0/385630106 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc03c0cff70 0x7fc03c109130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:54.931 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:54.909 7fc03affd700 1 -- 172.21.2.102:0/385630106 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 527664850 0 0) 0x7fc030001010 con 0x7fc03c1b0ae0 2021-02-10T11:38:54.931 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:54.909 7fc03affd700 1 -- 172.21.2.102:0/385630106 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc03c0cff70 msgr2=0x7fc03c109130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:54.931 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:54.909 7fc03affd700 1 --2- 172.21.2.102:0/385630106 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc03c0cff70 0x7fc03c109130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:54.931 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:54.909 7fc03affd700 1 -- 172.21.2.102:0/385630106 >> v2:172.21.2.113:3300/0 conn(0x7fc03c0d1870 msgr2=0x7fc03c1b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:54.931 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:54.909 7fc03affd700 1 --2- 172.21.2.102:0/385630106 >> v2:172.21.2.113:3300/0 conn(0x7fc03c0d1870 0x7fc03c1b04c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:54.931 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:54.909 7fc03affd700 1 -- 172.21.2.102:0/385630106 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc03c0d1c70 con 0x7fc03c1b0ae0 2021-02-10T11:38:54.931 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:54.909 7fc03affd700 1 -- 172.21.2.102:0/385630106 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7fc030001590 con 0x7fc03c1b0ae0 2021-02-10T11:38:54.931 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:54.909 7fc03affd700 1 -- 172.21.2.102:0/385630106 >> v1:172.21.2.102:6789/0 conn(0x7fc03c1b0ae0 legacy=0x7fc03c108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:54.931 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:54.909 7fc03affd700 1 --2- 172.21.2.102:0/385630106 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc024003b90 0x7fc024006000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:54.931 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:54.909 7fc03affd700 1 --2- 172.21.2.102:0/385630106 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc02404a830 0x7fc02404cc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:54.932 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:54.909 7fc03affd700 1 --2- 172.21.2.102:0/385630106 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc02404d520 0x7fc02404fa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:54.932 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:54.909 7fc03affd700 1 -- 172.21.2.102:0/385630106 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0240502f0 con 0x7fc02404a830 2021-02-10T11:38:54.932 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:54.909 7fc03affd700 1 -- 172.21.2.102:0/385630106 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc024050510 con 0x7fc02404d520 2021-02-10T11:38:54.932 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:54.909 7fc03affd700 1 -- 172.21.2.102:0/385630106 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc024050730 con 0x7fc024003b90 2021-02-10T11:38:54.932 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:54.909 7fc03b7fe700 1 --2- 172.21.2.102:0/385630106 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc02404a830 0x7fc02404cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:54.932 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:54.909 7fc03bfff700 1 --2- 172.21.2.102:0/385630106 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc024003b90 0x7fc024006000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:54.932 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:54.909 7fc040d8a700 1 --2- 172.21.2.102:0/385630106 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc02404d520 0x7fc02404fa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:54.932 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:54.909 7fc03b7fe700 1 -- 172.21.2.102:0/385630106 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc024003b90 msgr2=0x7fc024006000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:54.932 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:54.909 7fc03b7fe700 1 --2- 172.21.2.102:0/385630106 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc024003b90 0x7fc024006000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:54.932 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:54.909 7fc03b7fe700 1 -- 172.21.2.102:0/385630106 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc02404d520 msgr2=0x7fc02404fa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:54.932 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:54.909 7fc03b7fe700 1 --2- 172.21.2.102:0/385630106 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc02404d520 0x7fc02404fa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:54.932 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:54.909 7fc03b7fe700 1 -- 172.21.2.102:0/385630106 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0240509d0 con 0x7fc02404a830 2021-02-10T11:38:54.932 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:54.909 7fc03b7fe700 1 --2- 172.21.2.102:0/385630106 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc02404a830 0x7fc02404cc80 secure :-1 s=READY pgs=3154 cs=0 l=1 rev1=1 rx=0x7fc0300017d0 tx=0x7fc030007d30).ready entity=mon.0 client_cookie=90f4df2e34ff6d87 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:54.933 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:54.909 7fc03affd700 1 -- 172.21.2.102:0/385630106 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc03000a310 con 0x7fc02404a830 2021-02-10T11:38:54.933 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:54.909 7fc03affd700 1 -- 172.21.2.102:0/385630106 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0300013a0 con 0x7fc02404a830 2021-02-10T11:38:54.933 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:54.909 7fc03affd700 1 -- 172.21.2.102:0/385630106 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc0300009b0 con 0x7fc02404a830 2021-02-10T11:38:54.933 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:54.909 7fc04276d700 1 -- 172.21.2.102:0/385630106 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc02404a830 msgr2=0x7fc02404cc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:54.933 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:54.909 7fc04276d700 1 --2- 172.21.2.102:0/385630106 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc02404a830 0x7fc02404cc80 secure :-1 s=READY pgs=3154 cs=0 l=1 rev1=1 rx=0x7fc0300017d0 tx=0x7fc030007d30).stop 2021-02-10T11:38:54.933 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:54.909 7fc040d8a700 1 -- 172.21.2.102:0/385630106 reap_dead start 2021-02-10T11:38:54.933 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:54.910 7fc04276d700 1 -- 172.21.2.102:0/385630106 shutdown_connections 2021-02-10T11:38:54.933 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:54.910 7fc04276d700 1 -- 172.21.2.102:0/385630106 >> 172.21.2.102:0/385630106 conn(0x7fc03c0ce420 msgr2=0x7fc03c0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:54.933 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:54.910 7fc04276d700 1 -- 172.21.2.102:0/385630106 shutdown_connections 2021-02-10T11:38:54.933 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:54.910 7fc04276d700 1 -- 172.21.2.102:0/385630106 wait complete. 2021-02-10T11:38:54.933 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:54.925 7fc04276d700 1 Processor -- start 2021-02-10T11:38:54.933 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:54.925 7fc04276d700 1 -- start start 2021-02-10T11:38:54.935 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:54.925 7fc04276d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc03c085bc0 0x7fc03c0863e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:54.935 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:54.925 7fc04276d700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc03c0954c0 0x7fc03c096300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:54.935 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:54.925 7fc04276d700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc03c0958c0 0x7fc03c096920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:54.935 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:54.925 7fc04276d700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc03c095d80 con 0x7fc03c085bc0 2021-02-10T11:38:54.935 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:54.925 7fc04276d700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc03c0bfa60 con 0x7fc03c0954c0 2021-02-10T11:38:54.936 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:54.925 7fc04276d700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc03c085fc0 con 0x7fc03c0958c0 2021-02-10T11:38:54.936 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:54.926 7fc03bfff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc03c085bc0 0x7fc03c0863e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:54.936 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:54.926 7fc03bfff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc03c085bc0 0x7fc03c0863e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:39922/0 (socket says 172.21.2.102:39922) 2021-02-10T11:38:54.936 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:54.926 7fc03b7fe700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc03c0954c0 0x7fc03c096300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:54.936 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:54.926 7fc03bfff700 1 -- 172.21.2.102:0/1962285320 learned_addr learned my addr 172.21.2.102:0/1962285320 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:38:54.936 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:54.926 7fc040d8a700 1 --2- 172.21.2.102:0/1962285320 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc03c0958c0 0x7fc03c096920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:54.936 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:54.926 7fc03b7fe700 1 -- 172.21.2.102:0/1962285320 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc03c0958c0 msgr2=0x7fc03c096920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:54.936 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:54.926 7fc03b7fe700 1 --2- 172.21.2.102:0/1962285320 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc03c0958c0 0x7fc03c096920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:54.936 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:54.926 7fc03b7fe700 1 -- 172.21.2.102:0/1962285320 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc03c085bc0 msgr2=0x7fc03c0863e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:54.936 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:54.926 7fc03b7fe700 1 --2- 172.21.2.102:0/1962285320 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc03c085bc0 0x7fc03c0863e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:54.936 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:54.926 7fc03b7fe700 1 -- 172.21.2.102:0/1962285320 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc03c086c10 con 0x7fc03c0954c0 2021-02-10T11:38:54.936 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:54.926 7fc03bfff700 1 --2- 172.21.2.102:0/1962285320 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc03c085bc0 0x7fc03c0863e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:38:54.936 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:54.926 7fc03b7fe700 1 --2- 172.21.2.102:0/1962285320 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc03c0954c0 0x7fc03c096300 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 rx=0x7fc030001200 tx=0x7fc03000a820).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:54.937 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:54.926 7fc038ff9700 1 -- 172.21.2.102:0/1962285320 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc0300009b0 con 0x7fc03c0954c0 2021-02-10T11:38:54.937 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:54.927 7fc038ff9700 1 -- 172.21.2.102:0/1962285320 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc03000ac10 con 0x7fc03c0954c0 2021-02-10T11:38:54.937 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:54.927 7fc038ff9700 1 -- 172.21.2.102:0/1962285320 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc030004330 con 0x7fc03c0954c0 2021-02-10T11:38:54.937 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:54.927 7fc04276d700 1 -- 172.21.2.102:0/1962285320 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc03c086c10 con 0x7fc03c0954c0 2021-02-10T11:38:54.937 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:54.927 7fc04276d700 1 -- 172.21.2.102:0/1962285320 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc03c086c10 con 0x7fc03c0954c0 2021-02-10T11:38:54.937 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:54.927 7fc038ff9700 1 -- 172.21.2.102:0/1962285320 <== mon.1 v2:172.21.2.113:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fc0300075a0 con 0x7fc03c0954c0 2021-02-10T11:38:54.937 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:54.927 7fc038ff9700 1 --2- 172.21.2.102:0/1962285320 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc01c0186c0 0x7fc01c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:54.937 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:54.927 7fc038ff9700 1 -- 172.21.2.102:0/1962285320 <== mon.1 v2:172.21.2.113:3300/0 5 ==== osd_map(180..180 src has 1..180) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fc030032480 con 0x7fc03c0954c0 2021-02-10T11:38:54.937 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:54.927 7fc03bfff700 1 --2- 172.21.2.102:0/1962285320 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc01c0186c0 0x7fc01c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:54.937 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:54.928 7fc03bfff700 1 --2- 172.21.2.102:0/1962285320 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc01c0186c0 0x7fc01c01abc0 crc :-1 s=READY pgs=1382 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:54.960 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:54.957 7fc04276d700 1 -- 172.21.2.102:0/1962285320 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc03c0fe1c0 con 0x7fc03c0954c0 2021-02-10T11:38:54.963 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:54.961 7fc038ff9700 1 -- 172.21.2.102:0/1962285320 <== mon.1 v2:172.21.2.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fc030030c30 con 0x7fc03c0954c0 2021-02-10T11:38:55.096 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:55.094 7fc04276d700 1 -- 172.21.2.102:0/1962285320 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "target": ["mgr", ""], "group_name": "group_0000000000453502"}) v1 -- 0x7fc03c095d80 con 0x7fc01c0186c0 2021-02-10T11:38:55.100 INFO:tasks.ceph.mgr.y.gibba002.stderr:2021-02-10 11:38:55.098 7fd94bc53700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:38:55.100 INFO:tasks.ceph.mgr.y.gibba002.stderr:2021-02-10 11:38:55.098 7fd94bc53700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:38:55.100 INFO:tasks.ceph.mgr.y.gibba002.stderr:2021-02-10 11:38:55.098 7fd94bc53700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:38:55.100 INFO:tasks.ceph.mgr.y.gibba002.stderr:2021-02-10 11:38:55.098 7fd94bc53700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:38:55.100 INFO:tasks.ceph.mgr.y.gibba002.stderr:2021-02-10 11:38:55.098 7fd94bc53700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:38:55.115 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:55.113 7fc038ff9700 1 -- 172.21.2.102:0/1962285320 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fc02c006400 con 0x7fc01c0186c0 2021-02-10T11:38:55.129 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:55.127 7fc04276d700 1 -- 172.21.2.102:0/1962285320 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc01c0186c0 msgr2=0x7fc01c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:55.129 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:55.127 7fc04276d700 1 --2- 172.21.2.102:0/1962285320 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc01c0186c0 0x7fc01c01abc0 crc :-1 s=READY pgs=1382 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:55.129 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:55.127 7fc04276d700 1 -- 172.21.2.102:0/1962285320 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc03c0954c0 msgr2=0x7fc03c096300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:55.129 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:55.127 7fc04276d700 1 --2- 172.21.2.102:0/1962285320 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc03c0954c0 0x7fc03c096300 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 rx=0x7fc030001200 tx=0x7fc03000a820).stop 2021-02-10T11:38:55.130 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:55.127 7fc04276d700 1 -- 172.21.2.102:0/1962285320 shutdown_connections 2021-02-10T11:38:55.130 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:55.127 7fc04276d700 1 --2- 172.21.2.102:0/1962285320 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc03c0958c0 0x7fc03c096920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:55.130 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:55.127 7fc04276d700 1 --2- 172.21.2.102:0/1962285320 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc01c0186c0 0x7fc01c01abc0 unknown :-1 s=CLOSED pgs=1382 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:55.131 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:55.127 7fc04276d700 1 --2- 172.21.2.102:0/1962285320 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc03c0954c0 0x7fc03c096300 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:55.131 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:55.127 7fc04276d700 1 --2- 172.21.2.102:0/1962285320 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc03c085bc0 0x7fc03c0863e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:55.131 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:55.127 7fc04276d700 1 -- 172.21.2.102:0/1962285320 >> 172.21.2.102:0/1962285320 conn(0x7fc03c0bef00 msgr2=0x7fc03c09d1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:55.131 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:55.127 7fc04276d700 1 -- 172.21.2.102:0/1962285320 shutdown_connections 2021-02-10T11:38:55.131 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:55.127 7fc04276d700 1 -- 172.21.2.102:0/1962285320 wait complete. 2021-02-10T11:38:55.137 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000423388 --group_name group_0000000000453502 2021-02-10T11:38:55.570 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:55.551 7fd87ab39700 1 Processor -- start 2021-02-10T11:38:55.571 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:55.551 7fd87ab39700 1 -- start start 2021-02-10T11:38:55.571 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:55.552 7fd87ab39700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fd8740d1880 0x7fd8741b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:55.571 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:55.552 7fd87ab39700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fd8740cff80 0x7fd874109140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:55.571 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:55.552 7fd87ab39700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8740fe1d0 con 0x7fd8741b0af0 2021-02-10T11:38:55.571 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:55.552 7fd87ab39700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd874109ae0 con 0x7fd8740d1880 2021-02-10T11:38:55.571 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:55.552 7fd87ab39700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd874109760 con 0x7fd8740cff80 2021-02-10T11:38:55.571 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:55.552 7fd878955700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fd8740d1880 0x7fd8741b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:55.571 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:55.552 7fd873fff700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fd8741b0af0 0x7fd874108950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:60398/0 (socket says 172.21.2.102:60398) 2021-02-10T11:38:55.571 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:55.552 7fd873fff700 1 -- 172.21.2.102:0/4293927721 learned_addr learned my addr 172.21.2.102:0/4293927721 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:38:55.571 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:55.552 7fd878955700 1 --2- 172.21.2.102:0/4293927721 >> v2:172.21.2.102:3300/0 conn(0x7fd8740d1880 0x7fd8741b04d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:39944/0 (socket says 172.21.2.102:39944) 2021-02-10T11:38:55.571 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:55.553 7fd878955700 1 -- 172.21.2.102:0/4293927721 >> v2:172.21.2.113:3300/0 conn(0x7fd8740cff80 msgr2=0x7fd874109140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:55.572 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:55.553 7fd879156700 1 --2- 172.21.2.102:0/4293927721 >> v2:172.21.2.113:3300/0 conn(0x7fd8740cff80 0x7fd874109140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:55.572 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:55.553 7fd878955700 1 --2- 172.21.2.102:0/4293927721 >> v2:172.21.2.113:3300/0 conn(0x7fd8740cff80 0x7fd874109140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:55.572 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:55.553 7fd8737fe700 1 -- 172.21.2.102:0/4293927721 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 661522767 0 0) 0x7fd864001060 con 0x7fd8741b0af0 2021-02-10T11:38:55.572 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:55.553 7fd878955700 1 -- 172.21.2.102:0/4293927721 >> v1:172.21.2.102:6789/0 conn(0x7fd8741b0af0 legacy=0x7fd874108950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:55.572 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:55.553 7fd878955700 1 -- 172.21.2.102:0/4293927721 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd874088070 con 0x7fd8740d1880 2021-02-10T11:38:55.572 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:55.553 7fd879156700 1 --2- 172.21.2.102:0/4293927721 >> v2:172.21.2.113:3300/0 conn(0x7fd8740cff80 0x7fd874109140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:38:55.572 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:55.553 7fd878955700 1 --2- 172.21.2.102:0/4293927721 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd8740d1880 0x7fd8741b04d0 secure :-1 s=READY pgs=3158 cs=0 l=1 rev1=1 rx=0x7fd85c006650 tx=0x7fd85c009a80).ready entity=mon.0 client_cookie=618fb5a43d7f19b9 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:55.572 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:55.553 7fd8737fe700 1 -- 172.21.2.102:0/4293927721 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd85c00c0c0 con 0x7fd8740d1880 2021-02-10T11:38:55.572 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:55.553 7fd8737fe700 1 -- 172.21.2.102:0/4293927721 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd85c00c440 con 0x7fd8740d1880 2021-02-10T11:38:55.572 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:55.553 7fd8737fe700 1 -- 172.21.2.102:0/4293927721 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd85c00c960 con 0x7fd8740d1880 2021-02-10T11:38:55.572 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:55.553 7fd87ab39700 1 -- 172.21.2.102:0/4293927721 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd8740d1880 msgr2=0x7fd8741b04d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:55.572 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:55.553 7fd87ab39700 1 --2- 172.21.2.102:0/4293927721 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd8740d1880 0x7fd8741b04d0 secure :-1 s=READY pgs=3158 cs=0 l=1 rev1=1 rx=0x7fd85c006650 tx=0x7fd85c009a80).stop 2021-02-10T11:38:55.572 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:55.554 7fd87ab39700 1 -- 172.21.2.102:0/4293927721 shutdown_connections 2021-02-10T11:38:55.573 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:55.554 7fd87ab39700 1 --2- 172.21.2.102:0/4293927721 >> v2:172.21.2.113:3300/0 conn(0x7fd8740cff80 0x7fd874109140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:55.573 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:55.554 7fd87ab39700 1 --2- 172.21.2.102:0/4293927721 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd8740d1880 0x7fd8741b04d0 unknown :-1 s=CLOSED pgs=3158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:55.573 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:55.554 7fd87ab39700 1 -- 172.21.2.102:0/4293927721 >> 172.21.2.102:0/4293927721 conn(0x7fd8740ce430 msgr2=0x7fd8740d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:55.573 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:55.554 7fd87ab39700 1 -- 172.21.2.102:0/4293927721 shutdown_connections 2021-02-10T11:38:55.573 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:55.554 7fd87ab39700 1 -- 172.21.2.102:0/4293927721 wait complete. 2021-02-10T11:38:55.574 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:55.569 7fd87ab39700 1 Processor -- start 2021-02-10T11:38:55.574 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:55.569 7fd87ab39700 1 -- start start 2021-02-10T11:38:55.575 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:55.569 7fd87ab39700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd8741a4a10 0x7fd874187e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:55.575 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:55.569 7fd87ab39700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd8740913a0 0x7fd8740a8e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:55.575 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:55.569 7fd87ab39700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd8740a9470 0x7fd8740a7170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:55.575 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:55.569 7fd87ab39700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8740a8950 con 0x7fd8740a9470 2021-02-10T11:38:55.575 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:55.569 7fd87ab39700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8740917a0 con 0x7fd8740913a0 2021-02-10T11:38:55.575 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:55.569 7fd87ab39700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8740a9870 con 0x7fd8741a4a10 2021-02-10T11:38:55.575 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:55.569 7fd879156700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd8740a9470 0x7fd8740a7170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:55.575 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:55.569 7fd879156700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd8740a9470 0x7fd8740a7170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:39950/0 (socket says 172.21.2.102:39950) 2021-02-10T11:38:55.575 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:55.569 7fd879156700 1 -- 172.21.2.102:0/3906600220 learned_addr learned my addr 172.21.2.102:0/3906600220 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:38:55.575 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:55.570 7fd879156700 1 -- 172.21.2.102:0/3906600220 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd8741a4a10 msgr2=0x7fd874187e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:55.575 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:55.570 7fd879156700 1 --2- 172.21.2.102:0/3906600220 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd8741a4a10 0x7fd874187e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:55.575 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:55.570 7fd879156700 1 -- 172.21.2.102:0/3906600220 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd8740913a0 msgr2=0x7fd8740a8e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:55.575 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:55.570 7fd879156700 1 --2- 172.21.2.102:0/3906600220 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd8740913a0 0x7fd8740a8e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:55.575 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:55.570 7fd879156700 1 -- 172.21.2.102:0/3906600220 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8740a9a90 con 0x7fd8740a9470 2021-02-10T11:38:55.576 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:55.570 7fd879156700 1 --2- 172.21.2.102:0/3906600220 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd8740a9470 0x7fd8740a7170 secure :-1 s=READY pgs=3159 cs=0 l=1 rev1=1 rx=0x7fd86c006f10 tx=0x7fd86c00b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:55.576 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:55.570 7fd8717fa700 1 -- 172.21.2.102:0/3906600220 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd86c00bd80 con 0x7fd8740a9470 2021-02-10T11:38:55.576 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:55.570 7fd8717fa700 1 -- 172.21.2.102:0/3906600220 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd86c006410 con 0x7fd8740a9470 2021-02-10T11:38:55.576 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:55.570 7fd8717fa700 1 -- 172.21.2.102:0/3906600220 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd86c002220 con 0x7fd8740a9470 2021-02-10T11:38:55.576 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:55.570 7fd87ab39700 1 -- 172.21.2.102:0/3906600220 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8740a9a90 con 0x7fd8740a9470 2021-02-10T11:38:55.576 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:55.570 7fd87ab39700 1 -- 172.21.2.102:0/3906600220 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8740a7a80 con 0x7fd8740a9470 2021-02-10T11:38:55.576 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:55.571 7fd8717fa700 1 -- 172.21.2.102:0/3906600220 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd86c00a030 con 0x7fd8740a9470 2021-02-10T11:38:55.576 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:55.571 7fd8717fa700 1 --2- 172.21.2.102:0/3906600220 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd8580186c0 0x7fd85801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:55.576 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:55.571 7fd8717fa700 1 -- 172.21.2.102:0/3906600220 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(180..180 src has 1..180) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fd86c032560 con 0x7fd8740a9470 2021-02-10T11:38:55.576 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:55.571 7fd878955700 1 --2- 172.21.2.102:0/3906600220 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd8580186c0 0x7fd85801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:55.576 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:55.571 7fd878955700 1 --2- 172.21.2.102:0/3906600220 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd8580186c0 0x7fd85801abc0 crc :-1 s=READY pgs=1383 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:55.604 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:55.602 7fd87ab39700 1 -- 172.21.2.102:0/3906600220 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd874082aa0 con 0x7fd8740a9470 2021-02-10T11:38:55.607 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:55.605 7fd8717fa700 1 -- 172.21.2.102:0/3906600220 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd86c000bc0 con 0x7fd8740a9470 2021-02-10T11:38:55.734 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:55.731 7fd87ab39700 1 -- 172.21.2.102:0/3906600220 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000423388", "target": ["mgr", ""], "group_name": "group_0000000000453502"}) v1 -- 0x7fd8740917a0 con 0x7fd8580186c0 2021-02-10T11:38:55.760 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:55.758 7fd8717fa700 1 -- 172.21.2.102:0/3906600220 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fd85c006e70 con 0x7fd8580186c0 2021-02-10T11:38:55.767 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:55.765 7fd87ab39700 1 -- 172.21.2.102:0/3906600220 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd8580186c0 msgr2=0x7fd85801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:55.767 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:55.765 7fd87ab39700 1 --2- 172.21.2.102:0/3906600220 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd8580186c0 0x7fd85801abc0 crc :-1 s=READY pgs=1383 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:55.767 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:55.765 7fd87ab39700 1 -- 172.21.2.102:0/3906600220 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd8740a9470 msgr2=0x7fd8740a7170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:55.767 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:55.765 7fd87ab39700 1 --2- 172.21.2.102:0/3906600220 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd8740a9470 0x7fd8740a7170 secure :-1 s=READY pgs=3159 cs=0 l=1 rev1=1 rx=0x7fd86c006f10 tx=0x7fd86c00b990).stop 2021-02-10T11:38:55.767 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:55.765 7fd87ab39700 1 -- 172.21.2.102:0/3906600220 shutdown_connections 2021-02-10T11:38:55.768 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:55.765 7fd87ab39700 1 --2- 172.21.2.102:0/3906600220 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd8740a9470 0x7fd8740a7170 unknown :-1 s=CLOSED pgs=3159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:55.768 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:55.765 7fd87ab39700 1 --2- 172.21.2.102:0/3906600220 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd8580186c0 0x7fd85801abc0 unknown :-1 s=CLOSED pgs=1383 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:55.768 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:55.765 7fd87ab39700 1 --2- 172.21.2.102:0/3906600220 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd8740913a0 0x7fd8740a8e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:55.768 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:55.765 7fd87ab39700 1 --2- 172.21.2.102:0/3906600220 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd8741a4a10 0x7fd874187e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:55.768 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:55.765 7fd87ab39700 1 -- 172.21.2.102:0/3906600220 >> 172.21.2.102:0/3906600220 conn(0x7fd874087360 msgr2=0x7fd8740b5980 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:55.768 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:55.765 7fd87ab39700 1 -- 172.21.2.102:0/3906600220 shutdown_connections 2021-02-10T11:38:55.768 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:55.765 7fd87ab39700 1 -- 172.21.2.102:0/3906600220 wait complete. 2021-02-10T11:38:55.775 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000423389 --group_name group_0000000000453502 2021-02-10T11:38:56.208 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:56.189 7fe68cad6700 1 Processor -- start 2021-02-10T11:38:56.209 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:56.189 7fe68cad6700 1 -- start start 2021-02-10T11:38:56.209 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:56.190 7fe68cad6700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fe6880d1c80 0x7fe6880cef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:56.209 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:56.190 7fe68cad6700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6880d0180 con 0x7fe6880cf590 2021-02-10T11:38:56.209 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:56.190 7fe68cad6700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6880fe1d0 con 0x7fe6880d1880 2021-02-10T11:38:56.209 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:56.190 7fe68cad6700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6881b0cc0 con 0x7fe6880d1c80 2021-02-10T11:38:56.209 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:56.190 7fe68605d700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fe6880d1c80 0x7fe6880cef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:56.209 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:56.190 7fe68605d700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fe6880d1c80 0x7fe6880cef70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:39960/0 (socket says 172.21.2.102:39960) 2021-02-10T11:38:56.209 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:56.190 7fe68705f700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fe6880cf590 0x7fe6880cf990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:60418/0 (socket says 172.21.2.102:60418) 2021-02-10T11:38:56.209 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:56.190 7fe68605d700 1 -- 172.21.2.102:0/1735865852 learned_addr learned my addr 172.21.2.102:0/1735865852 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:38:56.209 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:56.191 7fe68605d700 1 -- 172.21.2.102:0/1735865852 >> v1:172.21.2.113:6789/0 conn(0x7fe6880d1880 legacy=0x7fe6881b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:56.209 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:56.191 7fe68605d700 1 -- 172.21.2.102:0/1735865852 >> v1:172.21.2.102:6789/0 conn(0x7fe6880cf590 legacy=0x7fe6880cf990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:56.209 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:56.191 7fe68605d700 1 -- 172.21.2.102:0/1735865852 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe688109a70 con 0x7fe6880d1c80 2021-02-10T11:38:56.210 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:56.191 7fe68605d700 1 --2- 172.21.2.102:0/1735865852 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe6880d1c80 0x7fe6880cef70 secure :-1 s=READY pgs=3161 cs=0 l=1 rev1=1 rx=0x7fe670006700 tx=0x7fe670009b80).ready entity=mon.0 client_cookie=224a69a7fd942909 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:56.210 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:56.191 7fe68585c700 1 -- 172.21.2.102:0/1735865852 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe67000c0c0 con 0x7fe6880d1c80 2021-02-10T11:38:56.210 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:56.191 7fe68585c700 1 -- 172.21.2.102:0/1735865852 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe67000c340 con 0x7fe6880d1c80 2021-02-10T11:38:56.210 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:56.191 7fe68585c700 1 -- 172.21.2.102:0/1735865852 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe67000c860 con 0x7fe6880d1c80 2021-02-10T11:38:56.210 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:56.191 7fe68cad6700 1 -- 172.21.2.102:0/1735865852 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe6880d1c80 msgr2=0x7fe6880cef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:56.210 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:56.191 7fe68cad6700 1 --2- 172.21.2.102:0/1735865852 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe6880d1c80 0x7fe6880cef70 secure :-1 s=READY pgs=3161 cs=0 l=1 rev1=1 rx=0x7fe670006700 tx=0x7fe670009b80).stop 2021-02-10T11:38:56.210 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:56.192 7fe68cad6700 1 -- 172.21.2.102:0/1735865852 shutdown_connections 2021-02-10T11:38:56.210 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:56.192 7fe68cad6700 1 --2- 172.21.2.102:0/1735865852 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe6880d1c80 0x7fe6880cef70 unknown :-1 s=CLOSED pgs=3161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:56.210 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:56.192 7fe68cad6700 1 -- 172.21.2.102:0/1735865852 >> 172.21.2.102:0/1735865852 conn(0x7fe6880ce430 msgr2=0x7fe6880d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:56.210 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:56.192 7fe68cad6700 1 -- 172.21.2.102:0/1735865852 shutdown_connections 2021-02-10T11:38:56.210 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:56.192 7fe68cad6700 1 -- 172.21.2.102:0/1735865852 wait complete. 2021-02-10T11:38:56.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:56.207 7fe68cad6700 1 Processor -- start 2021-02-10T11:38:56.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:56.207 7fe68cad6700 1 -- start start 2021-02-10T11:38:56.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:56.207 7fe68cad6700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe688097820 0x7fe688097c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:56.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:56.207 7fe68cad6700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe68808ac80 0x7fe68808b080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:56.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:56.207 7fe68cad6700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe68808bf00 0x7fe68808c300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:56.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:56.207 7fe68cad6700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe68808b6a0 con 0x7fe68808bf00 2021-02-10T11:38:56.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:56.207 7fe68cad6700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe68809a360 con 0x7fe688097820 2021-02-10T11:38:56.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:56.207 7fe68cad6700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe68809a580 con 0x7fe68808ac80 2021-02-10T11:38:56.213 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:56.207 7fe68705f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe68808bf00 0x7fe68808c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:56.213 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:56.207 7fe68705f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe68808bf00 0x7fe68808c300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:39966/0 (socket says 172.21.2.102:39966) 2021-02-10T11:38:56.213 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:56.207 7fe68705f700 1 -- 172.21.2.102:0/3981829683 learned_addr learned my addr 172.21.2.102:0/3981829683 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:38:56.213 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:56.208 7fe68605d700 1 --2- 172.21.2.102:0/3981829683 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe68808ac80 0x7fe68808b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:56.213 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:56.208 7fe68685e700 1 --2- 172.21.2.102:0/3981829683 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe688097820 0x7fe688097c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:56.213 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:56.208 7fe68705f700 1 -- 172.21.2.102:0/3981829683 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe68808ac80 msgr2=0x7fe68808b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:56.213 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:56.208 7fe68705f700 1 --2- 172.21.2.102:0/3981829683 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe68808ac80 0x7fe68808b080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:56.213 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:56.208 7fe68705f700 1 -- 172.21.2.102:0/3981829683 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe688097820 msgr2=0x7fe688097c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:56.213 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:56.208 7fe68705f700 1 --2- 172.21.2.102:0/3981829683 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe688097820 0x7fe688097c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:56.213 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:56.208 7fe68705f700 1 -- 172.21.2.102:0/3981829683 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe68808c920 con 0x7fe68808bf00 2021-02-10T11:38:56.213 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:56.208 7fe68685e700 1 --2- 172.21.2.102:0/3981829683 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe688097820 0x7fe688097c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:38:56.213 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:56.208 7fe68705f700 1 --2- 172.21.2.102:0/3981829683 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe68808bf00 0x7fe68808c300 secure :-1 s=READY pgs=3162 cs=0 l=1 rev1=1 rx=0x7fe680007f90 tx=0x7fe68000ac50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:56.213 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:56.208 7fe67f7fe700 1 -- 172.21.2.102:0/3981829683 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe680000ec0 con 0x7fe68808bf00 2021-02-10T11:38:56.214 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:56.208 7fe67f7fe700 1 -- 172.21.2.102:0/3981829683 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe680007020 con 0x7fe68808bf00 2021-02-10T11:38:56.214 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:56.208 7fe67f7fe700 1 -- 172.21.2.102:0/3981829683 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe680007480 con 0x7fe68808bf00 2021-02-10T11:38:56.214 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:56.208 7fe68cad6700 1 -- 172.21.2.102:0/3981829683 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe68808c920 con 0x7fe68808bf00 2021-02-10T11:38:56.214 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:56.208 7fe68cad6700 1 -- 172.21.2.102:0/3981829683 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe68808c920 con 0x7fe68808bf00 2021-02-10T11:38:56.214 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:56.209 7fe67f7fe700 1 -- 172.21.2.102:0/3981829683 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fe680007200 con 0x7fe68808bf00 2021-02-10T11:38:56.214 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:56.209 7fe67f7fe700 1 --2- 172.21.2.102:0/3981829683 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe66c018730 0x7fe66c01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:56.214 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:56.209 7fe67f7fe700 1 -- 172.21.2.102:0/3981829683 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(180..180 src has 1..180) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fe680032990 con 0x7fe68808bf00 2021-02-10T11:38:56.214 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:56.209 7fe68685e700 1 --2- 172.21.2.102:0/3981829683 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe66c018730 0x7fe66c01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:56.214 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:56.210 7fe68685e700 1 --2- 172.21.2.102:0/3981829683 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe66c018730 0x7fe66c01ac10 crc :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:56.242 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:56.240 7fe68cad6700 1 -- 172.21.2.102:0/3981829683 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe688080340 con 0x7fe68808bf00 2021-02-10T11:38:56.244 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:56.242 7fe67f7fe700 1 -- 172.21.2.102:0/3981829683 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fe680007d00 con 0x7fe68808bf00 2021-02-10T11:38:56.372 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:56.370 7fe68cad6700 1 -- 172.21.2.102:0/3981829683 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000423389", "target": ["mgr", ""], "group_name": "group_0000000000453502"}) v1 -- 0x7fe688080340 con 0x7fe66c018730 2021-02-10T11:38:56.392 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:56.390 7fe67f7fe700 1 -- 172.21.2.102:0/3981829683 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fe678006bc0 con 0x7fe66c018730 2021-02-10T11:38:56.405 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:56.403 7fe68cad6700 1 -- 172.21.2.102:0/3981829683 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe66c018730 msgr2=0x7fe66c01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:56.405 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:56.403 7fe68cad6700 1 --2- 172.21.2.102:0/3981829683 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe66c018730 0x7fe66c01ac10 crc :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:56.405 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:56.403 7fe68cad6700 1 -- 172.21.2.102:0/3981829683 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe68808bf00 msgr2=0x7fe68808c300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:56.405 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:56.403 7fe68cad6700 1 --2- 172.21.2.102:0/3981829683 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe68808bf00 0x7fe68808c300 secure :-1 s=READY pgs=3162 cs=0 l=1 rev1=1 rx=0x7fe680007f90 tx=0x7fe68000ac50).stop 2021-02-10T11:38:56.406 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:56.403 7fe68cad6700 1 -- 172.21.2.102:0/3981829683 shutdown_connections 2021-02-10T11:38:56.406 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:56.403 7fe68cad6700 1 --2- 172.21.2.102:0/3981829683 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe68808bf00 0x7fe68808c300 unknown :-1 s=CLOSED pgs=3162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:56.406 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:56.403 7fe68cad6700 1 --2- 172.21.2.102:0/3981829683 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe68808ac80 0x7fe68808b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:56.406 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:56.403 7fe68cad6700 1 --2- 172.21.2.102:0/3981829683 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe66c018730 0x7fe66c01ac10 unknown :-1 s=CLOSED pgs=1384 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:56.406 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:56.403 7fe68cad6700 1 --2- 172.21.2.102:0/3981829683 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe688097820 0x7fe688097c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:56.406 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:56.403 7fe68cad6700 1 -- 172.21.2.102:0/3981829683 >> 172.21.2.102:0/3981829683 conn(0x7fe68809fb50 msgr2=0x7fe6880fed60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:56.406 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:56.403 7fe68cad6700 1 -- 172.21.2.102:0/3981829683 shutdown_connections 2021-02-10T11:38:56.407 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:56.403 7fe68cad6700 1 -- 172.21.2.102:0/3981829683 wait complete. 2021-02-10T11:38:56.414 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume authorize cephfs subvolume_0000000000423388 guest1 --group_name group_0000000000453502 --tenant_id tenant1 2021-02-10T11:38:56.877 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:56.858 7fd6716a6700 1 Processor -- start 2021-02-10T11:38:56.877 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:56.858 7fd6716a6700 1 -- start start 2021-02-10T11:38:56.878 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:56.859 7fd6716a6700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd66c0e53a0 0x7fd66c1b0550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:56.878 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:56.859 7fd6716a6700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fd66c10b050 0x7fd66c1091c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:56.878 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:56.859 7fd6716a6700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd66c109ac0 con 0x7fd66c1b0b70 2021-02-10T11:38:56.878 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:56.859 7fd6716a6700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd66c0fe2b0 con 0x7fd66c10b050 2021-02-10T11:38:56.878 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:56.859 7fd6716a6700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd66c0e5f80 con 0x7fd66c0e53a0 2021-02-10T11:38:56.878 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:56.859 7fd66aabe700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fd66c1b0b70 0x7fd66c1089d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:60428/0 (socket says 172.21.2.102:60428) 2021-02-10T11:38:56.878 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:56.859 7fd66aabe700 1 -- 172.21.2.102:0/2973685451 learned_addr learned my addr 172.21.2.102:0/2973685451 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:38:56.878 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:56.859 7fd66bac0700 1 --2- 172.21.2.102:0/2973685451 >> v2:172.21.2.113:3300/0 conn(0x7fd66c10b050 0x7fd66c1091c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:56.878 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:56.859 7fd66b2bf700 1 --2- 172.21.2.102:0/2973685451 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd66c0e53a0 0x7fd66c1b0550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:56.878 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:56.860 7fd66bac0700 1 -- 172.21.2.102:0/2973685451 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd66c0e53a0 msgr2=0x7fd66c1b0550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:56.878 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:56.860 7fd66bac0700 1 --2- 172.21.2.102:0/2973685451 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd66c0e53a0 0x7fd66c1b0550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:56.879 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:56.860 7fd66bac0700 1 -- 172.21.2.102:0/2973685451 >> v1:172.21.2.102:6789/0 conn(0x7fd66c1b0b70 legacy=0x7fd66c1089d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:56.879 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:56.860 7fd66bac0700 1 -- 172.21.2.102:0/2973685451 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd66c0e57a0 con 0x7fd66c10b050 2021-02-10T11:38:56.879 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:56.860 7fd66b2bf700 1 --2- 172.21.2.102:0/2973685451 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd66c0e53a0 0x7fd66c1b0550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:38:56.879 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:56.860 7fd66bac0700 1 --2- 172.21.2.102:0/2973685451 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd66c10b050 0x7fd66c1091c0 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 rx=0x7fd65c006e50 tx=0x7fd65c009e70).ready entity=mon.1 client_cookie=1e67bfdfe12d7797 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:56.879 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:56.860 7fd66a2bd700 1 -- 172.21.2.102:0/2973685451 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd65c00b510 con 0x7fd66c10b050 2021-02-10T11:38:56.879 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:56.860 7fd66a2bd700 1 -- 172.21.2.102:0/2973685451 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd65c006790 con 0x7fd66c10b050 2021-02-10T11:38:56.879 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:56.860 7fd66a2bd700 1 -- 172.21.2.102:0/2973685451 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd65c00b940 con 0x7fd66c10b050 2021-02-10T11:38:56.879 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:56.860 7fd6716a6700 1 -- 172.21.2.102:0/2973685451 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd66c10b050 msgr2=0x7fd66c1091c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:56.879 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:56.860 7fd6716a6700 1 --2- 172.21.2.102:0/2973685451 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd66c10b050 0x7fd66c1091c0 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 rx=0x7fd65c006e50 tx=0x7fd65c009e70).stop 2021-02-10T11:38:56.879 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:56.861 7fd6716a6700 1 -- 172.21.2.102:0/2973685451 shutdown_connections 2021-02-10T11:38:56.879 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:56.861 7fd6716a6700 1 --2- 172.21.2.102:0/2973685451 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd66c10b050 0x7fd66c1091c0 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:56.879 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:56.861 7fd6716a6700 1 --2- 172.21.2.102:0/2973685451 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd66c0e53a0 0x7fd66c1b0550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:56.879 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:56.861 7fd6716a6700 1 -- 172.21.2.102:0/2973685451 >> 172.21.2.102:0/2973685451 conn(0x7fd66c0ce4b0 msgr2=0x7fd66c0e45d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:56.880 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:56.861 7fd6716a6700 1 -- 172.21.2.102:0/2973685451 shutdown_connections 2021-02-10T11:38:56.880 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:56.861 7fd6716a6700 1 -- 172.21.2.102:0/2973685451 wait complete. 2021-02-10T11:38:56.881 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:56.875 7fd6716a6700 1 Processor -- start 2021-02-10T11:38:56.881 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:56.876 7fd6716a6700 1 -- start start 2021-02-10T11:38:56.881 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:56.876 7fd6716a6700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd66c19a540 0x7fd66c0b1fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:56.881 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:56.876 7fd6716a6700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd66c0b2600 0x7fd66c187f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:56.882 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:56.876 7fd6716a6700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd66c191390 0x7fd66c188520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:56.882 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:56.876 7fd6716a6700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd66c190e10 con 0x7fd66c0b2600 2021-02-10T11:38:56.882 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:56.876 7fd6716a6700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd66c19a0a0 con 0x7fd66c19a540 2021-02-10T11:38:56.882 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:56.876 7fd6716a6700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd66c19a280 con 0x7fd66c191390 2021-02-10T11:38:56.882 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:56.876 7fd66aabe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd66c0b2600 0x7fd66c187f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:56.882 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:56.876 7fd66aabe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd66c0b2600 0x7fd66c187f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:39978/0 (socket says 172.21.2.102:39978) 2021-02-10T11:38:56.882 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:56.876 7fd66aabe700 1 -- 172.21.2.102:0/2142929162 learned_addr learned my addr 172.21.2.102:0/2142929162 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:38:56.882 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:56.876 7fd66bac0700 1 --2- 172.21.2.102:0/2142929162 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd66c191390 0x7fd66c188520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:56.882 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:56.876 7fd66b2bf700 1 --2- 172.21.2.102:0/2142929162 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd66c19a540 0x7fd66c0b1fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:56.882 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:56.877 7fd66aabe700 1 -- 172.21.2.102:0/2142929162 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd66c191390 msgr2=0x7fd66c188520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:56.882 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:56.877 7fd66aabe700 1 --2- 172.21.2.102:0/2142929162 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd66c191390 0x7fd66c188520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:56.882 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:56.877 7fd66aabe700 1 -- 172.21.2.102:0/2142929162 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd66c19a540 msgr2=0x7fd66c0b1fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:56.883 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:56.877 7fd66aabe700 1 --2- 172.21.2.102:0/2142929162 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd66c19a540 0x7fd66c0b1fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:56.883 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:56.877 7fd66aabe700 1 -- 172.21.2.102:0/2142929162 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd66c188b40 con 0x7fd66c0b2600 2021-02-10T11:38:56.883 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:56.877 7fd66aabe700 1 --2- 172.21.2.102:0/2142929162 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd66c0b2600 0x7fd66c187f00 secure :-1 s=READY pgs=3164 cs=0 l=1 rev1=1 rx=0x7fd654001c60 tx=0x7fd65400a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:56.883 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:56.877 7fd663fff700 1 -- 172.21.2.102:0/2142929162 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd6540010b0 con 0x7fd66c0b2600 2021-02-10T11:38:56.883 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:56.877 7fd663fff700 1 -- 172.21.2.102:0/2142929162 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd654004140 con 0x7fd66c0b2600 2021-02-10T11:38:56.883 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:56.877 7fd663fff700 1 -- 172.21.2.102:0/2142929162 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd654004660 con 0x7fd66c0b2600 2021-02-10T11:38:56.883 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:56.877 7fd6716a6700 1 -- 172.21.2.102:0/2142929162 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd66c188b40 con 0x7fd66c0b2600 2021-02-10T11:38:56.883 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:56.877 7fd6716a6700 1 -- 172.21.2.102:0/2142929162 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd66c188b40 con 0x7fd66c0b2600 2021-02-10T11:38:56.883 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:56.877 7fd663fff700 1 -- 172.21.2.102:0/2142929162 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd654001090 con 0x7fd66c0b2600 2021-02-10T11:38:56.883 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:56.878 7fd663fff700 1 --2- 172.21.2.102:0/2142929162 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd6500186c0 0x7fd65001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:56.883 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:56.878 7fd663fff700 1 -- 172.21.2.102:0/2142929162 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(180..180 src has 1..180) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fd654032630 con 0x7fd66c0b2600 2021-02-10T11:38:56.883 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:56.878 7fd66b2bf700 1 --2- 172.21.2.102:0/2142929162 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd6500186c0 0x7fd65001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:56.883 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:56.878 7fd66b2bf700 1 --2- 172.21.2.102:0/2142929162 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd6500186c0 0x7fd65001abc0 crc :-1 s=READY pgs=1385 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:56.911 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:56.909 7fd6716a6700 1 -- 172.21.2.102:0/2142929162 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd66c109ac0 con 0x7fd66c0b2600 2021-02-10T11:38:56.914 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:56.911 7fd663fff700 1 -- 172.21.2.102:0/2142929162 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd654000980 con 0x7fd66c0b2600 2021-02-10T11:38:57.040 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:57.038 7fd6716a6700 1 -- 172.21.2.102:0/2142929162 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume authorize", "vol_name": "cephfs", "sub_name": "subvolume_0000000000423388", "target": ["mgr", ""], "tenant_id": "tenant1", "auth_id": "guest1", "group_name": "group_0000000000453502"}) v1 -- 0x7fd66c109ac0 con 0x7fd6500186c0 2021-02-10T11:38:57.045 INFO:tasks.ceph.mgr.y.gibba002.stderr:Creating meta for ID guest1 with tenant tenant1 2021-02-10T11:38:57.071 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:57.069 7fd663fff700 1 -- 172.21.2.102:0/2142929162 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+41 (crc 0 0 0) 0x7fd664006a30 con 0x7fd6500186c0 2021-02-10T11:38:57.072 INFO:teuthology.orchestra.run.gibba002.stdout:AQDRxSNgVANEAxAAC9CI+KxcAkDYGiy4SDqOTw== 2021-02-10T11:38:57.073 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:57.071 7fd6716a6700 1 -- 172.21.2.102:0/2142929162 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd6500186c0 msgr2=0x7fd65001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:57.073 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:57.071 7fd6716a6700 1 --2- 172.21.2.102:0/2142929162 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd6500186c0 0x7fd65001abc0 crc :-1 s=READY pgs=1385 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:57.073 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:57.071 7fd6716a6700 1 -- 172.21.2.102:0/2142929162 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd66c0b2600 msgr2=0x7fd66c187f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:57.073 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:57.071 7fd6716a6700 1 --2- 172.21.2.102:0/2142929162 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd66c0b2600 0x7fd66c187f00 secure :-1 s=READY pgs=3164 cs=0 l=1 rev1=1 rx=0x7fd654001c60 tx=0x7fd65400a8d0).stop 2021-02-10T11:38:57.074 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:57.071 7fd6716a6700 1 -- 172.21.2.102:0/2142929162 shutdown_connections 2021-02-10T11:38:57.074 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:57.071 7fd6716a6700 1 --2- 172.21.2.102:0/2142929162 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd66c191390 0x7fd66c188520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:57.074 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:57.071 7fd6716a6700 1 --2- 172.21.2.102:0/2142929162 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd66c0b2600 0x7fd66c187f00 unknown :-1 s=CLOSED pgs=3164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:57.074 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:57.071 7fd6716a6700 1 --2- 172.21.2.102:0/2142929162 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd6500186c0 0x7fd65001abc0 unknown :-1 s=CLOSED pgs=1385 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:57.074 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:57.071 7fd6716a6700 1 --2- 172.21.2.102:0/2142929162 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd66c19a540 0x7fd66c0b1fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:57.074 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:57.071 7fd6716a6700 1 -- 172.21.2.102:0/2142929162 >> 172.21.2.102:0/2142929162 conn(0x7fd66c19f8e0 msgr2=0x7fd66c19aa50 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:57.074 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:57.071 7fd6716a6700 1 -- 172.21.2.102:0/2142929162 shutdown_connections 2021-02-10T11:38:57.075 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:57.071 7fd6716a6700 1 -- 172.21.2.102:0/2142929162 wait complete. 2021-02-10T11:38:57.082 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.1 && sudo ls volumes 2021-02-10T11:38:57.431 INFO:teuthology.orchestra.run.gibba002.stdout:group_0000000000453502 2021-02-10T11:38:57.432 INFO:teuthology.orchestra.run.gibba002.stdout:_group_0000000000453502:subvolume_0000000000423388.meta 2021-02-10T11:38:57.432 INFO:teuthology.orchestra.run.gibba002.stdout:_group_0000000000453502:subvolume_0000000000423389.meta 2021-02-10T11:38:57.432 INFO:teuthology.orchestra.run.gibba002.stdout:$guest1.meta 2021-02-10T11:38:57.433 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo sudo cat '/home/ubuntu/cephtest/mnt.0/volumes/$guest1.meta' 2021-02-10T11:38:57.799 INFO:teuthology.orchestra.run.gibba002.stdout:{"compat_version": 1, "tenant_id": "tenant1", "version": 5, "dirty": false, "subvolumes": {"group_0000000000453502/subvolume_0000000000423388": {"access_level": "rw", "dirty": false}}} 2021-02-10T11:38:57.799 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume authorize cephfs subvolume_0000000000423389 guest1 --group_name group_0000000000453502 --tenant_id tenant1 2021-02-10T11:38:58.240 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:58.221 7f03fcf59700 1 Processor -- start 2021-02-10T11:38:58.241 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:58.222 7f03fcf59700 1 -- start start 2021-02-10T11:38:58.241 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:58.222 7f03fcf59700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f03f80e53a0 0x7f03f81b0550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:58.241 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:58.222 7f03fcf59700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f03f81b0b70 0x7f03f81089d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:58.241 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:58.222 7f03fcf59700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03f8109ac0 con 0x7f03f8108ff0 2021-02-10T11:38:58.241 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:58.222 7f03fcf59700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03f80e57a0 con 0x7f03f80e53a0 2021-02-10T11:38:58.241 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:58.222 7f03fcf59700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03f8088070 con 0x7f03f81b0b70 2021-02-10T11:38:58.241 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:58.222 7f03f6b13700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f03f80e53a0 0x7f03f81b0550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:58.241 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:58.222 7f03f6b13700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f03f80e53a0 0x7f03f81b0550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:39988/0 (socket says 172.21.2.102:39988) 2021-02-10T11:38:58.241 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:58.222 7f03f6b13700 1 -- 172.21.2.102:0/833342120 learned_addr learned my addr 172.21.2.102:0/833342120 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:38:58.241 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:58.223 7f03f6b13700 1 -- 172.21.2.102:0/833342120 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f03f81b0b70 msgr2=0x7f03f81089d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:58.241 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:58.223 7f03f6b13700 1 --2- 172.21.2.102:0/833342120 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f03f81b0b70 0x7f03f81089d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:58.241 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:58.223 7f03f6b13700 1 -- 172.21.2.102:0/833342120 >> v1:172.21.2.102:6789/0 conn(0x7f03f8108ff0 legacy=0x7f03f810a040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:58.242 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:58.223 7f03f6b13700 1 -- 172.21.2.102:0/833342120 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03f81093f0 con 0x7f03f80e53a0 2021-02-10T11:38:58.242 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:58.223 7f03f6b13700 1 --2- 172.21.2.102:0/833342120 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f03f80e53a0 0x7f03f81b0550 secure :-1 s=READY pgs=3166 cs=0 l=1 rev1=1 rx=0x7f03e8006700 tx=0x7f03e8009b80).ready entity=mon.0 client_cookie=bb931fea81df06e1 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:58.242 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:58.223 7f03f5b11700 1 -- 172.21.2.102:0/833342120 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f03e800c0c0 con 0x7f03f80e53a0 2021-02-10T11:38:58.242 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:58.223 7f03f5b11700 1 -- 172.21.2.102:0/833342120 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03e800c340 con 0x7f03f80e53a0 2021-02-10T11:38:58.242 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:58.223 7f03f5b11700 1 -- 172.21.2.102:0/833342120 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f03e800c860 con 0x7f03f80e53a0 2021-02-10T11:38:58.242 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:58.223 7f03fcf59700 1 -- 172.21.2.102:0/833342120 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f03f80e53a0 msgr2=0x7f03f81b0550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:58.242 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:58.223 7f03fcf59700 1 --2- 172.21.2.102:0/833342120 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f03f80e53a0 0x7f03f81b0550 secure :-1 s=READY pgs=3166 cs=0 l=1 rev1=1 rx=0x7f03e8006700 tx=0x7f03e8009b80).stop 2021-02-10T11:38:58.242 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:58.224 7f03fcf59700 1 -- 172.21.2.102:0/833342120 shutdown_connections 2021-02-10T11:38:58.242 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:58.224 7f03fcf59700 1 --2- 172.21.2.102:0/833342120 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f03f81b0b70 0x7f03f81089d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:58.242 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:58.224 7f03fcf59700 1 --2- 172.21.2.102:0/833342120 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f03f80e53a0 0x7f03f81b0550 unknown :-1 s=CLOSED pgs=3166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:58.242 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:58.224 7f03fcf59700 1 -- 172.21.2.102:0/833342120 >> 172.21.2.102:0/833342120 conn(0x7f03f80ce4b0 msgr2=0x7f03f80e45d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:58.242 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:58.224 7f03fcf59700 1 -- 172.21.2.102:0/833342120 shutdown_connections 2021-02-10T11:38:58.243 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:58.224 7f03fcf59700 1 -- 172.21.2.102:0/833342120 wait complete. 2021-02-10T11:38:58.244 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:58.239 7f03fcf59700 1 Processor -- start 2021-02-10T11:38:58.244 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:58.239 7f03fcf59700 1 -- start start 2021-02-10T11:38:58.244 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:58.239 7f03fcf59700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f03f81a8820 0x7f03f81a8c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:58.244 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:58.239 7f03fcf59700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f03f808c6d0 0x7f03f808cad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:58.244 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:58.239 7f03fcf59700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f03f808d0f0 0x7f03f808d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:58.244 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:58.239 7f03fcf59700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03f81a93a0 con 0x7f03f808c6d0 2021-02-10T11:38:58.244 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:58.239 7f03fcf59700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03f81a9920 con 0x7f03f81a8820 2021-02-10T11:38:58.244 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:58.239 7f03fcf59700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03f81a9b00 con 0x7f03f808d0f0 2021-02-10T11:38:58.244 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:58.239 7f03f6312700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f03f808c6d0 0x7f03f808cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:58.245 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:58.239 7f03f6312700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f03f808c6d0 0x7f03f808cad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:39996/0 (socket says 172.21.2.102:39996) 2021-02-10T11:38:58.245 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:58.239 7f03f6312700 1 -- 172.21.2.102:0/3323574788 learned_addr learned my addr 172.21.2.102:0/3323574788 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:38:58.245 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:58.240 7f03f6312700 1 -- 172.21.2.102:0/3323574788 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f03f808d0f0 msgr2=0x7f03f808d4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:38:58.245 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:58.240 7f03f6312700 1 --2- 172.21.2.102:0/3323574788 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f03f808d0f0 0x7f03f808d4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:58.245 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:58.240 7f03f6312700 1 -- 172.21.2.102:0/3323574788 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f03f81a8820 msgr2=0x7f03f81a8c20 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:38:58.245 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:58.240 7f03f6312700 1 --2- 172.21.2.102:0/3323574788 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f03f81a8820 0x7f03f81a8c20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:58.245 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:58.240 7f03f6312700 1 -- 172.21.2.102:0/3323574788 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03f81a9d20 con 0x7f03f808c6d0 2021-02-10T11:38:58.245 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:58.240 7f03f6312700 1 --2- 172.21.2.102:0/3323574788 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f03f808c6d0 0x7f03f808cad0 secure :-1 s=READY pgs=3167 cs=0 l=1 rev1=1 rx=0x7f03e0006f00 tx=0x7f03e000b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:58.245 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:58.240 7f03ef7fe700 1 -- 172.21.2.102:0/3323574788 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f03e000bd30 con 0x7f03f808c6d0 2021-02-10T11:38:58.245 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:58.240 7f03ef7fe700 1 -- 172.21.2.102:0/3323574788 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03e0000c90 con 0x7f03f808c6d0 2021-02-10T11:38:58.245 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:58.240 7f03ef7fe700 1 -- 172.21.2.102:0/3323574788 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f03e00022a0 con 0x7f03f808c6d0 2021-02-10T11:38:58.245 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:58.240 7f03fcf59700 1 -- 172.21.2.102:0/3323574788 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03f81a9d20 con 0x7f03f808c6d0 2021-02-10T11:38:58.245 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:58.240 7f03fcf59700 1 -- 172.21.2.102:0/3323574788 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03f81aa2b0 con 0x7f03f808c6d0 2021-02-10T11:38:58.245 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:58.241 7f03ef7fe700 1 -- 172.21.2.102:0/3323574788 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f03e0005310 con 0x7f03f808c6d0 2021-02-10T11:38:58.246 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:58.241 7f03ef7fe700 1 --2- 172.21.2.102:0/3323574788 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f03dc0186c0 0x7f03dc01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:58.246 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:58.241 7f03ef7fe700 1 -- 172.21.2.102:0/3323574788 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(180..180 src has 1..180) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f03e00323e0 con 0x7f03f808c6d0 2021-02-10T11:38:58.246 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:58.241 7f03f6b13700 1 --2- 172.21.2.102:0/3323574788 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f03dc0186c0 0x7f03dc01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:58.246 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:58.241 7f03f6b13700 1 --2- 172.21.2.102:0/3323574788 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f03dc0186c0 0x7f03dc01abc0 crc :-1 s=READY pgs=1386 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:58.274 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:58.272 7f03fcf59700 1 -- 172.21.2.102:0/3323574788 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03f8103810 con 0x7f03f808c6d0 2021-02-10T11:38:58.276 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:58.274 7f03ef7fe700 1 -- 172.21.2.102:0/3323574788 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f03e0000960 con 0x7f03f808c6d0 2021-02-10T11:38:58.402 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:58.400 7f03fcf59700 1 -- 172.21.2.102:0/3323574788 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume authorize", "vol_name": "cephfs", "sub_name": "subvolume_0000000000423389", "target": ["mgr", ""], "tenant_id": "tenant1", "auth_id": "guest1", "group_name": "group_0000000000453502"}) v1 -- 0x7f03f81a93a0 con 0x7f03dc0186c0 2021-02-10T11:38:58.439 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:58.436 7f03ef7fe700 1 -- 172.21.2.102:0/3323574788 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+41 (crc 0 0 0) 0x7f03e8009100 con 0x7f03dc0186c0 2021-02-10T11:38:58.465 INFO:teuthology.orchestra.run.gibba002.stdout:AQDRxSNgVANEAxAAC9CI+KxcAkDYGiy4SDqOTw== 2021-02-10T11:38:58.468 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:58.465 7f03fcf59700 1 -- 172.21.2.102:0/3323574788 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f03dc0186c0 msgr2=0x7f03dc01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:58.468 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:58.465 7f03fcf59700 1 --2- 172.21.2.102:0/3323574788 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f03dc0186c0 0x7f03dc01abc0 crc :-1 s=READY pgs=1386 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:58.468 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:58.465 7f03fcf59700 1 -- 172.21.2.102:0/3323574788 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f03f808c6d0 msgr2=0x7f03f808cad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:58.468 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:58.465 7f03fcf59700 1 --2- 172.21.2.102:0/3323574788 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f03f808c6d0 0x7f03f808cad0 secure :-1 s=READY pgs=3167 cs=0 l=1 rev1=1 rx=0x7f03e0006f00 tx=0x7f03e000b940).stop 2021-02-10T11:38:58.468 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:58.465 7f03fcf59700 1 -- 172.21.2.102:0/3323574788 shutdown_connections 2021-02-10T11:38:58.468 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:58.465 7f03fcf59700 1 --2- 172.21.2.102:0/3323574788 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f03f808d0f0 0x7f03f808d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:58.468 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:58.465 7f03fcf59700 1 --2- 172.21.2.102:0/3323574788 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f03f808c6d0 0x7f03f808cad0 unknown :-1 s=CLOSED pgs=3167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:58.468 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:58.465 7f03fcf59700 1 --2- 172.21.2.102:0/3323574788 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f03dc0186c0 0x7f03dc01abc0 unknown :-1 s=CLOSED pgs=1386 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:58.468 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:58.465 7f03fcf59700 1 --2- 172.21.2.102:0/3323574788 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f03f81a8820 0x7f03f81a8c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:58.468 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:58.465 7f03fcf59700 1 -- 172.21.2.102:0/3323574788 >> 172.21.2.102:0/3323574788 conn(0x7f03f818cb60 msgr2=0x7f03f8199f40 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:58.469 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:58.466 7f03fcf59700 1 -- 172.21.2.102:0/3323574788 shutdown_connections 2021-02-10T11:38:58.469 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:58.466 7f03fcf59700 1 -- 172.21.2.102:0/3323574788 wait complete. 2021-02-10T11:38:58.476 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.1 && sudo sed -i s/false/true/g 'volumes/$guest1.meta' 2021-02-10T11:38:58.847 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume deauthorize cephfs subvolume_0000000000423389 guest1 --group_name group_0000000000453502 2021-02-10T11:38:59.281 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:59.262 7f3a7087f700 1 Processor -- start 2021-02-10T11:38:59.282 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:59.262 7f3a7087f700 1 -- start start 2021-02-10T11:38:59.282 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:59.262 7f3a7087f700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3a6c0d1c90 0x7f3a6c0cef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:59.282 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:59.262 7f3a7087f700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f3a6c0cf5b0 0x7f3a6c0cf9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:59.282 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:59.262 7f3a7087f700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a6c109a60 con 0x7f3a6c0d1890 2021-02-10T11:38:59.282 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:59.262 7f3a7087f700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a6c0e5f50 con 0x7f3a6c0cf5b0 2021-02-10T11:38:59.282 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:59.262 7f3a7087f700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a6c1b0bc0 con 0x7f3a6c0d1c90 2021-02-10T11:38:59.282 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:59.263 7f3a6ae1d700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f3a6c0cf5b0 0x7f3a6c0cf9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:59.282 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:59.263 7f3a69e1b700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3a6c0d1c90 0x7f3a6c0cef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:59.282 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:59.263 7f3a6a61c700 1 --1- >> v1:172.21.2.113:6789/0 conn(0x7f3a6c0d1890 0x7f3a6c1b03d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.113:6789/0 says I am v1:172.21.2.102:39556/0 (socket says 172.21.2.102:39556) 2021-02-10T11:38:59.282 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:59.263 7f3a6a61c700 1 -- 172.21.2.102:0/3516499765 learned_addr learned my addr 172.21.2.102:0/3516499765 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:38:59.282 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:59.263 7f3a6ae1d700 1 -- 172.21.2.102:0/3516499765 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3a6c0d1c90 msgr2=0x7f3a6c0cef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:59.283 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:59.264 7f3a6ae1d700 1 --2- 172.21.2.102:0/3516499765 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3a6c0d1c90 0x7f3a6c0cef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:59.283 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:59.264 7f3a6ae1d700 1 -- 172.21.2.102:0/3516499765 >> v1:172.21.2.113:6789/0 conn(0x7f3a6c0d1890 legacy=0x7f3a6c1b03d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:59.283 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:59.264 7f3a6ae1d700 1 -- 172.21.2.102:0/3516499765 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a6c088070 con 0x7f3a6c0cf5b0 2021-02-10T11:38:59.283 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:59.264 7f3a6ae1d700 1 --2- 172.21.2.102:0/3516499765 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3a6c0cf5b0 0x7f3a6c0cf9b0 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 rx=0x7f3a64006e50 tx=0x7f3a64009e70).ready entity=mon.1 client_cookie=527082466731170a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:59.283 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:59.264 7f3a6961a700 1 -- 172.21.2.102:0/3516499765 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3a6400b510 con 0x7f3a6c0cf5b0 2021-02-10T11:38:59.283 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:59.264 7f3a6961a700 1 -- 172.21.2.102:0/3516499765 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a64006790 con 0x7f3a6c0cf5b0 2021-02-10T11:38:59.283 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:59.264 7f3a6961a700 1 -- 172.21.2.102:0/3516499765 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3a6400b940 con 0x7f3a6c0cf5b0 2021-02-10T11:38:59.283 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:59.264 7f3a7087f700 1 -- 172.21.2.102:0/3516499765 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3a6c0cf5b0 msgr2=0x7f3a6c0cf9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:59.283 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:59.264 7f3a7087f700 1 --2- 172.21.2.102:0/3516499765 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3a6c0cf5b0 0x7f3a6c0cf9b0 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 rx=0x7f3a64006e50 tx=0x7f3a64009e70).stop 2021-02-10T11:38:59.283 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:59.265 7f3a7087f700 1 -- 172.21.2.102:0/3516499765 shutdown_connections 2021-02-10T11:38:59.283 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:59.265 7f3a7087f700 1 --2- 172.21.2.102:0/3516499765 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3a6c0d1c90 0x7f3a6c0cef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:59.283 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:59.265 7f3a7087f700 1 --2- 172.21.2.102:0/3516499765 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3a6c0cf5b0 0x7f3a6c0cf9b0 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:59.283 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:59.265 7f3a7087f700 1 -- 172.21.2.102:0/3516499765 >> 172.21.2.102:0/3516499765 conn(0x7f3a6c0fdee0 msgr2=0x7f3a6c0ce450 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:59.284 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:59.265 7f3a7087f700 1 -- 172.21.2.102:0/3516499765 shutdown_connections 2021-02-10T11:38:59.284 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:59.265 7f3a7087f700 1 -- 172.21.2.102:0/3516499765 wait complete. 2021-02-10T11:38:59.285 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:59.280 7f3a7087f700 1 Processor -- start 2021-02-10T11:38:59.285 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:59.280 7f3a7087f700 1 -- start start 2021-02-10T11:38:59.285 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:59.280 7f3a7087f700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3a6c0aa2a0 0x7f3a6c0aa6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:59.285 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:59.280 7f3a7087f700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3a6c095450 0x7f3a6c1887d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:59.285 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:59.280 7f3a7087f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3a6c095850 0x7f3a6c188df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:59.286 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:59.280 7f3a7087f700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a6c188250 con 0x7f3a6c095850 2021-02-10T11:38:59.286 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:59.280 7f3a7087f700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a6c07d740 con 0x7f3a6c0aa2a0 2021-02-10T11:38:59.286 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:59.280 7f3a7087f700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a6c187ed0 con 0x7f3a6c095450 2021-02-10T11:38:59.286 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:59.280 7f3a6ae1d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3a6c095850 0x7f3a6c188df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:59.286 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:59.280 7f3a6ae1d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3a6c095850 0x7f3a6c188df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:40014/0 (socket says 172.21.2.102:40014) 2021-02-10T11:38:59.286 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:59.280 7f3a6ae1d700 1 -- 172.21.2.102:0/758296060 learned_addr learned my addr 172.21.2.102:0/758296060 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:38:59.286 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:59.281 7f3a6ae1d700 1 -- 172.21.2.102:0/758296060 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3a6c095450 msgr2=0x7f3a6c1887d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:59.286 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:59.281 7f3a6ae1d700 1 --2- 172.21.2.102:0/758296060 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3a6c095450 0x7f3a6c1887d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:59.286 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:59.281 7f3a6ae1d700 1 -- 172.21.2.102:0/758296060 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3a6c0aa2a0 msgr2=0x7f3a6c0aa6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:59.286 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:59.281 7f3a6ae1d700 1 --2- 172.21.2.102:0/758296060 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3a6c0aa2a0 0x7f3a6c0aa6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:59.286 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:59.281 7f3a6ae1d700 1 -- 172.21.2.102:0/758296060 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a6c07d230 con 0x7f3a6c095850 2021-02-10T11:38:59.286 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:59.281 7f3a6ae1d700 1 --2- 172.21.2.102:0/758296060 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3a6c095850 0x7f3a6c188df0 secure :-1 s=READY pgs=3168 cs=0 l=1 rev1=1 rx=0x7f3a64006c50 tx=0x7f3a6400bd20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:59.287 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:59.281 7f3a62ffd700 1 -- 172.21.2.102:0/758296060 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3a640093d0 con 0x7f3a6c095850 2021-02-10T11:38:59.287 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:59.281 7f3a62ffd700 1 -- 172.21.2.102:0/758296060 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a6400be60 con 0x7f3a6c095850 2021-02-10T11:38:59.287 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:59.281 7f3a62ffd700 1 -- 172.21.2.102:0/758296060 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3a6400c570 con 0x7f3a6c095850 2021-02-10T11:38:59.287 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:59.281 7f3a7087f700 1 -- 172.21.2.102:0/758296060 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a6c07d230 con 0x7f3a6c095850 2021-02-10T11:38:59.287 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:59.281 7f3a7087f700 1 -- 172.21.2.102:0/758296060 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a6c189410 con 0x7f3a6c095850 2021-02-10T11:38:59.287 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:59.282 7f3a62ffd700 1 -- 172.21.2.102:0/758296060 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f3a640093d0 con 0x7f3a6c095850 2021-02-10T11:38:59.287 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:59.282 7f3a62ffd700 1 --2- 172.21.2.102:0/758296060 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3a500186c0 0x7f3a5001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:59.287 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:59.282 7f3a62ffd700 1 -- 172.21.2.102:0/758296060 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(180..180 src has 1..180) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f3a64032950 con 0x7f3a6c095850 2021-02-10T11:38:59.287 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:59.282 7f3a6a61c700 1 --2- 172.21.2.102:0/758296060 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3a500186c0 0x7f3a5001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:59.287 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:59.282 7f3a6a61c700 1 --2- 172.21.2.102:0/758296060 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3a500186c0 0x7f3a5001abc0 crc :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:59.314 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:59.312 7f3a7087f700 1 -- 172.21.2.102:0/758296060 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a6c105720 con 0x7f3a6c095850 2021-02-10T11:38:59.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:59.315 7f3a62ffd700 1 -- 172.21.2.102:0/758296060 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f3a6402dfc0 con 0x7f3a6c095850 2021-02-10T11:38:59.443 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:59.441 7f3a7087f700 1 -- 172.21.2.102:0/758296060 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume deauthorize", "vol_name": "cephfs", "sub_name": "subvolume_0000000000423389", "target": ["mgr", ""], "auth_id": "guest1", "group_name": "group_0000000000453502"}) v1 -- 0x7f3a6c109a60 con 0x7f3a500186c0 2021-02-10T11:38:59.510 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:59.508 7f3a62ffd700 1 -- 172.21.2.102:0/758296060 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f3a5c007b30 con 0x7f3a500186c0 2021-02-10T11:38:59.559 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:59.556 7f3a7087f700 1 -- 172.21.2.102:0/758296060 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3a500186c0 msgr2=0x7f3a5001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:59.559 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:59.556 7f3a7087f700 1 --2- 172.21.2.102:0/758296060 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3a500186c0 0x7f3a5001abc0 crc :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:59.559 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:59.556 7f3a7087f700 1 -- 172.21.2.102:0/758296060 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3a6c095850 msgr2=0x7f3a6c188df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:59.559 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:59.556 7f3a7087f700 1 --2- 172.21.2.102:0/758296060 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3a6c095850 0x7f3a6c188df0 secure :-1 s=READY pgs=3168 cs=0 l=1 rev1=1 rx=0x7f3a64006c50 tx=0x7f3a6400bd20).stop 2021-02-10T11:38:59.559 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:59.556 7f3a7087f700 1 -- 172.21.2.102:0/758296060 shutdown_connections 2021-02-10T11:38:59.559 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:59.556 7f3a7087f700 1 --2- 172.21.2.102:0/758296060 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3a6c095850 0x7f3a6c188df0 unknown :-1 s=CLOSED pgs=3168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:59.559 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:59.556 7f3a7087f700 1 --2- 172.21.2.102:0/758296060 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3a6c095450 0x7f3a6c1887d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:59.559 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:59.556 7f3a7087f700 1 --2- 172.21.2.102:0/758296060 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3a500186c0 0x7f3a5001abc0 unknown :-1 s=CLOSED pgs=1387 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:59.559 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:59.556 7f3a7087f700 1 --2- 172.21.2.102:0/758296060 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3a6c0aa2a0 0x7f3a6c0aa6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:59.559 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:59.556 7f3a7087f700 1 -- 172.21.2.102:0/758296060 >> 172.21.2.102:0/758296060 conn(0x7f3a6c19b460 msgr2=0x7f3a6c107960 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:59.559 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:59.557 7f3a7087f700 1 -- 172.21.2.102:0/758296060 shutdown_connections 2021-02-10T11:38:59.559 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:38:59.557 7f3a7087f700 1 -- 172.21.2.102:0/758296060 wait complete. 2021-02-10T11:38:59.566 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo sudo cat '/home/ubuntu/cephtest/mnt.0/volumes/$guest1.meta' 2021-02-10T11:38:59.927 INFO:teuthology.orchestra.run.gibba002.stdout:{"tenant_id": "tenant1", "version": 5, "dirty": false, "compat_version": 1, "subvolumes": {"group_0000000000453502/subvolume_0000000000423388": {"access_level": "rw", "dirty": false}}} 2021-02-10T11:38:59.927 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume deauthorize cephfs subvolume_0000000000423388 guest1 --group_name group_0000000000453502 2021-02-10T11:39:00.362 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:00.342 7fd334fcb700 1 Processor -- start 2021-02-10T11:39:00.362 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:00.342 7fd334fcb700 1 -- start start 2021-02-10T11:39:00.362 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:00.343 7fd334fcb700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fd3300eb220 0x7fd3300eb620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:00.362 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:00.343 7fd334fcb700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd33010a550 con 0x7fd3300dce80 2021-02-10T11:39:00.362 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:00.343 7fd334fcb700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3300881d0 con 0x7fd33010aad0 2021-02-10T11:39:00.362 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:00.343 7fd334fcb700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3300d1ce0 con 0x7fd3300eb220 2021-02-10T11:39:00.362 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:00.343 7fd32659c700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fd3300eb220 0x7fd3300eb620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:00.362 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:00.343 7fd32659c700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fd3300eb220 0x7fd3300eb620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:40024/0 (socket says 172.21.2.102:40024) 2021-02-10T11:39:00.362 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:00.343 7fd32659c700 1 -- 172.21.2.102:0/3656286789 learned_addr learned my addr 172.21.2.102:0/3656286789 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:39:00.363 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:00.343 7fd32ed9d700 1 --1- 172.21.2.102:0/3656286789 >> v1:172.21.2.102:6789/0 conn(0x7fd3300dce80 0x7fd3300d14f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:60480/0 (socket says 172.21.2.102:60480) 2021-02-10T11:39:00.363 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:00.344 7fd32e59c700 1 -- 172.21.2.102:0/3656286789 <== mon.1 v1:172.21.2.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3812256889 0 0) 0x7fd3280009b0 con 0x7fd33010aad0 2021-02-10T11:39:00.363 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:00.344 7fd32659c700 1 -- 172.21.2.102:0/3656286789 >> v1:172.21.2.113:6789/0 conn(0x7fd33010aad0 legacy=0x7fd33010aed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:00.363 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:00.344 7fd32659c700 1 -- 172.21.2.102:0/3656286789 >> v1:172.21.2.102:6789/0 conn(0x7fd3300dce80 legacy=0x7fd3300d14f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:00.363 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:00.344 7fd32659c700 1 -- 172.21.2.102:0/3656286789 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3300d1f60 con 0x7fd3300eb220 2021-02-10T11:39:00.363 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:00.344 7fd32659c700 1 --2- 172.21.2.102:0/3656286789 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd3300eb220 0x7fd3300eb620 secure :-1 s=READY pgs=3170 cs=0 l=1 rev1=1 rx=0x7fd318006700 tx=0x7fd318009b00).ready entity=mon.0 client_cookie=c58a05f2baf42ab6 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:00.363 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:00.344 7fd32e59c700 1 -- 172.21.2.102:0/3656286789 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd31800c0c0 con 0x7fd3300eb220 2021-02-10T11:39:00.363 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:00.344 7fd32e59c700 1 -- 172.21.2.102:0/3656286789 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd31800c400 con 0x7fd3300eb220 2021-02-10T11:39:00.363 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:00.344 7fd32e59c700 1 -- 172.21.2.102:0/3656286789 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd31800c8e0 con 0x7fd3300eb220 2021-02-10T11:39:00.363 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:00.344 7fd334fcb700 1 -- 172.21.2.102:0/3656286789 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd3300eb220 msgr2=0x7fd3300eb620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:00.363 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:00.344 7fd334fcb700 1 --2- 172.21.2.102:0/3656286789 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd3300eb220 0x7fd3300eb620 secure :-1 s=READY pgs=3170 cs=0 l=1 rev1=1 rx=0x7fd318006700 tx=0x7fd318009b00).stop 2021-02-10T11:39:00.363 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:00.345 7fd334fcb700 1 -- 172.21.2.102:0/3656286789 shutdown_connections 2021-02-10T11:39:00.363 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:00.345 7fd334fcb700 1 --2- 172.21.2.102:0/3656286789 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd3300eb220 0x7fd3300eb620 unknown :-1 s=CLOSED pgs=3170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:00.364 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:00.345 7fd334fcb700 1 -- 172.21.2.102:0/3656286789 >> 172.21.2.102:0/3656286789 conn(0x7fd3300ea630 msgr2=0x7fd3300d0800 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:00.364 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:00.345 7fd334fcb700 1 -- 172.21.2.102:0/3656286789 shutdown_connections 2021-02-10T11:39:00.364 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:00.345 7fd334fcb700 1 -- 172.21.2.102:0/3656286789 wait complete. 2021-02-10T11:39:00.365 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:00.360 7fd334fcb700 1 Processor -- start 2021-02-10T11:39:00.365 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:00.360 7fd334fcb700 1 -- start start 2021-02-10T11:39:00.366 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:00.360 7fd334fcb700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd3300aca10 0x7fd3300fb110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:00.366 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:00.360 7fd334fcb700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd3300fb730 0x7fd3300fbb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:00.366 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:00.360 7fd334fcb700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd3300fc150 0x7fd3300f9580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:00.366 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:00.360 7fd334fcb700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd33008b3b0 con 0x7fd3300aca10 2021-02-10T11:39:00.366 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:00.360 7fd334fcb700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3300fab90 con 0x7fd3300fb730 2021-02-10T11:39:00.366 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:00.360 7fd334fcb700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3301ab700 con 0x7fd3300fc150 2021-02-10T11:39:00.366 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:00.361 7fd32ed9d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd3300aca10 0x7fd3300fb110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:00.366 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:00.361 7fd32ed9d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd3300aca10 0x7fd3300fb110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:40028/0 (socket says 172.21.2.102:40028) 2021-02-10T11:39:00.366 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:00.361 7fd32ed9d700 1 -- 172.21.2.102:0/4092478137 learned_addr learned my addr 172.21.2.102:0/4092478137 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:39:00.366 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:00.361 7fd32659c700 1 --2- 172.21.2.102:0/4092478137 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd3300fb730 0x7fd3300fbb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:00.366 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:00.361 7fd32f59e700 1 --2- 172.21.2.102:0/4092478137 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd3300fc150 0x7fd3300f9580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:00.366 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:00.361 7fd32ed9d700 1 -- 172.21.2.102:0/4092478137 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd3300fc150 msgr2=0x7fd3300f9580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:00.367 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:00.361 7fd32ed9d700 1 --2- 172.21.2.102:0/4092478137 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd3300fc150 0x7fd3300f9580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:00.367 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:00.361 7fd32ed9d700 1 -- 172.21.2.102:0/4092478137 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd3300fb730 msgr2=0x7fd3300fbb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:00.367 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:00.361 7fd32ed9d700 1 --2- 172.21.2.102:0/4092478137 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd3300fb730 0x7fd3300fbb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:00.367 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:00.361 7fd32ed9d700 1 -- 172.21.2.102:0/4092478137 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3300f9ba0 con 0x7fd3300aca10 2021-02-10T11:39:00.367 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:00.361 7fd32659c700 1 --2- 172.21.2.102:0/4092478137 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd3300fb730 0x7fd3300fbb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:39:00.367 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:00.361 7fd32ed9d700 1 --2- 172.21.2.102:0/4092478137 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd3300aca10 0x7fd3300fb110 secure :-1 s=READY pgs=3171 cs=0 l=1 rev1=1 rx=0x7fd320009960 tx=0x7fd32000bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:00.367 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:00.361 7fd327fff700 1 -- 172.21.2.102:0/4092478137 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd3200062f0 con 0x7fd3300aca10 2021-02-10T11:39:00.367 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:00.361 7fd327fff700 1 -- 172.21.2.102:0/4092478137 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3200065b0 con 0x7fd3300aca10 2021-02-10T11:39:00.367 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:00.361 7fd327fff700 1 -- 172.21.2.102:0/4092478137 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd320004910 con 0x7fd3300aca10 2021-02-10T11:39:00.367 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:00.361 7fd334fcb700 1 -- 172.21.2.102:0/4092478137 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3300f9ba0 con 0x7fd3300aca10 2021-02-10T11:39:00.367 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:00.361 7fd334fcb700 1 -- 172.21.2.102:0/4092478137 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3300fa130 con 0x7fd3300aca10 2021-02-10T11:39:00.367 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:00.362 7fd327fff700 1 -- 172.21.2.102:0/4092478137 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd320009f40 con 0x7fd3300aca10 2021-02-10T11:39:00.367 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:00.362 7fd327fff700 1 --2- 172.21.2.102:0/4092478137 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd3140186c0 0x7fd31401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:00.368 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:00.362 7fd327fff700 1 -- 172.21.2.102:0/4092478137 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(180..180 src has 1..180) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fd320032b70 con 0x7fd3300aca10 2021-02-10T11:39:00.368 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:00.362 7fd32659c700 1 --2- 172.21.2.102:0/4092478137 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd3140186c0 0x7fd31401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:00.368 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:00.362 7fd32659c700 1 --2- 172.21.2.102:0/4092478137 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd3140186c0 0x7fd31401abc0 crc :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:00.394 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:00.392 7fd334fcb700 1 -- 172.21.2.102:0/4092478137 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3300f9350 con 0x7fd3300aca10 2021-02-10T11:39:00.397 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:00.394 7fd327fff700 1 -- 172.21.2.102:0/4092478137 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd320006f00 con 0x7fd3300aca10 2021-02-10T11:39:00.523 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:00.520 7fd334fcb700 1 -- 172.21.2.102:0/4092478137 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume deauthorize", "vol_name": "cephfs", "sub_name": "subvolume_0000000000423388", "target": ["mgr", ""], "auth_id": "guest1", "group_name": "group_0000000000453502"}) v1 -- 0x7fd3300fa130 con 0x7fd3140186c0 2021-02-10T11:39:00.559 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:00.557 7fd327fff700 1 -- 172.21.2.102:0/4092478137 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fd318000c90 con 0x7fd3140186c0 2021-02-10T11:39:00.588 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:00.586 7fd334fcb700 1 -- 172.21.2.102:0/4092478137 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd3140186c0 msgr2=0x7fd31401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:00.588 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:00.586 7fd334fcb700 1 --2- 172.21.2.102:0/4092478137 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd3140186c0 0x7fd31401abc0 crc :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:00.588 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:00.586 7fd334fcb700 1 -- 172.21.2.102:0/4092478137 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd3300aca10 msgr2=0x7fd3300fb110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:00.588 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:00.586 7fd334fcb700 1 --2- 172.21.2.102:0/4092478137 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd3300aca10 0x7fd3300fb110 secure :-1 s=READY pgs=3171 cs=0 l=1 rev1=1 rx=0x7fd320009960 tx=0x7fd32000bcf0).stop 2021-02-10T11:39:00.588 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:00.586 7fd334fcb700 1 -- 172.21.2.102:0/4092478137 shutdown_connections 2021-02-10T11:39:00.588 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:00.586 7fd334fcb700 1 --2- 172.21.2.102:0/4092478137 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd3300fc150 0x7fd3300f9580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:00.588 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:00.586 7fd334fcb700 1 --2- 172.21.2.102:0/4092478137 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd3140186c0 0x7fd31401abc0 unknown :-1 s=CLOSED pgs=1388 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:00.588 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:00.586 7fd334fcb700 1 --2- 172.21.2.102:0/4092478137 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd3300fb730 0x7fd3300fbb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:00.589 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:00.586 7fd334fcb700 1 --2- 172.21.2.102:0/4092478137 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd3300aca10 0x7fd3300fb110 unknown :-1 s=CLOSED pgs=3171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:00.589 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:00.586 7fd334fcb700 1 -- 172.21.2.102:0/4092478137 >> 172.21.2.102:0/4092478137 conn(0x7fd3301aef40 msgr2=0x7fd330195e90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:00.589 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:00.586 7fd334fcb700 1 -- 172.21.2.102:0/4092478137 shutdown_connections 2021-02-10T11:39:00.589 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:00.586 7fd334fcb700 1 -- 172.21.2.102:0/4092478137 wait complete. 2021-02-10T11:39:00.596 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:39:00.596 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:39:00.597 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:39:00.932 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:39:00.933 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2021-02-10T11:39:00.933 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@gibba002.front.sepia.ceph.com... 2021-02-10T11:39:00.933 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-02-10T11:39:00.933 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:39:01.287 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba002.stderr:ceph-fuse[213188]: fuse finished with error 0 and tester_r 0 2021-02-10T11:39:01.289 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:39:01.289 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:39:01.289 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:39:01.624 INFO:teuthology.orchestra.run.gibba002.stdout:ext2/ext3 2021-02-10T11:39:01.625 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2021-02-10T11:39:01.625 INFO:teuthology.orchestra.run:waiting for 900 2021-02-10T11:39:01.625 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:39:01.625 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:39:01.962 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth rm client.guest1 2021-02-10T11:39:02.392 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:02.373 7f40ba4e0700 1 Processor -- start 2021-02-10T11:39:02.392 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:02.373 7f40ba4e0700 1 -- start start 2021-02-10T11:39:02.392 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:02.373 7f40ba4e0700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f40b40d1870 0x7f40b41b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:02.392 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:02.373 7f40ba4e0700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f40b40f0650 0x7f40b40cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:02.392 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:02.373 7f40ba4e0700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40b40f0b10 con 0x7f40b41b0a10 2021-02-10T11:39:02.392 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:02.373 7f40ba4e0700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40b40e5ed0 con 0x7f40b40f0650 2021-02-10T11:39:02.392 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:02.373 7f40ba4e0700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40b40d1c70 con 0x7f40b40d1870 2021-02-10T11:39:02.392 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:02.373 7f40b8afd700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f40b40f0650 0x7f40b40cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:02.392 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:02.373 7f40b8afd700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f40b40f0650 0x7f40b40cfb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:40034/0 (socket says 172.21.2.102:40034) 2021-02-10T11:39:02.392 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:02.373 7f40b8afd700 1 -- 172.21.2.102:0/3783267632 learned_addr learned my addr 172.21.2.102:0/3783267632 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:39:02.393 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:02.373 7f40b37fe700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f40b41b0a10 0x7f40b40cf330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:60494/0 (socket says 172.21.2.102:60494) 2021-02-10T11:39:02.393 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:02.374 7f40b3fff700 1 --2- 172.21.2.102:0/3783267632 >> v2:172.21.2.113:3300/0 conn(0x7f40b40d1870 0x7f40b41b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:02.393 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:02.374 7f40b8afd700 1 -- 172.21.2.102:0/3783267632 >> v2:172.21.2.113:3300/0 conn(0x7f40b40d1870 msgr2=0x7f40b41b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:02.393 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:02.374 7f40b8afd700 1 --2- 172.21.2.102:0/3783267632 >> v2:172.21.2.113:3300/0 conn(0x7f40b40d1870 0x7f40b41b03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:02.393 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:02.374 7f40b8afd700 1 -- 172.21.2.102:0/3783267632 >> v1:172.21.2.102:6789/0 conn(0x7f40b41b0a10 legacy=0x7f40b40cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:02.393 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:02.374 7f40b8afd700 1 -- 172.21.2.102:0/3783267632 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40b40d1e90 con 0x7f40b40f0650 2021-02-10T11:39:02.393 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:02.374 7f40b3fff700 1 --2- 172.21.2.102:0/3783267632 >> v2:172.21.2.113:3300/0 conn(0x7f40b40d1870 0x7f40b41b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:39:02.393 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:02.374 7f40b8afd700 1 --2- 172.21.2.102:0/3783267632 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f40b40f0650 0x7f40b40cfb20 secure :-1 s=READY pgs=3173 cs=0 l=1 rev1=1 rx=0x7f40ac006f00 tx=0x7f40ac009f10).ready entity=mon.0 client_cookie=618868417ec953b9 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:02.393 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:02.374 7f40b2ffd700 1 -- 172.21.2.102:0/3783267632 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f40ac00b3b0 con 0x7f40b40f0650 2021-02-10T11:39:02.393 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:02.374 7f40b2ffd700 1 -- 172.21.2.102:0/3783267632 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40ac000d30 con 0x7f40b40f0650 2021-02-10T11:39:02.393 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:02.374 7f40b2ffd700 1 -- 172.21.2.102:0/3783267632 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f40ac00b970 con 0x7f40b40f0650 2021-02-10T11:39:02.393 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:02.374 7f40ba4e0700 1 -- 172.21.2.102:0/3783267632 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f40b40f0650 msgr2=0x7f40b40cfb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:02.393 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:02.374 7f40ba4e0700 1 --2- 172.21.2.102:0/3783267632 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f40b40f0650 0x7f40b40cfb20 secure :-1 s=READY pgs=3173 cs=0 l=1 rev1=1 rx=0x7f40ac006f00 tx=0x7f40ac009f10).stop 2021-02-10T11:39:02.394 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:02.375 7f40ba4e0700 1 -- 172.21.2.102:0/3783267632 shutdown_connections 2021-02-10T11:39:02.394 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:02.375 7f40ba4e0700 1 --2- 172.21.2.102:0/3783267632 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f40b40f0650 0x7f40b40cfb20 unknown :-1 s=CLOSED pgs=3173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:02.394 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:02.375 7f40ba4e0700 1 --2- 172.21.2.102:0/3783267632 >> v2:172.21.2.113:3300/0 conn(0x7f40b40d1870 0x7f40b41b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:02.394 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:02.375 7f40ba4e0700 1 -- 172.21.2.102:0/3783267632 >> 172.21.2.102:0/3783267632 conn(0x7f40b40ce450 msgr2=0x7f40b40d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:02.394 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:02.375 7f40ba4e0700 1 -- 172.21.2.102:0/3783267632 shutdown_connections 2021-02-10T11:39:02.394 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:02.375 7f40ba4e0700 1 -- 172.21.2.102:0/3783267632 wait complete. 2021-02-10T11:39:02.395 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:02.390 7f40ba4e0700 1 Processor -- start 2021-02-10T11:39:02.396 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:02.390 7f40ba4e0700 1 -- start start 2021-02-10T11:39:02.396 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:02.390 7f40ba4e0700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f40b40ae800 0x7f40b4090b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:02.396 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:02.390 7f40ba4e0700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f40b4091190 0x7f40b408a420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:02.396 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:02.390 7f40ba4e0700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f40b408aa40 0x7f40b408ae40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:02.396 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:02.390 7f40ba4e0700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40b4091590 con 0x7f40b40ae800 2021-02-10T11:39:02.396 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:02.390 7f40ba4e0700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40b408b710 con 0x7f40b4091190 2021-02-10T11:39:02.396 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:02.390 7f40ba4e0700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40b408b460 con 0x7f40b408aa40 2021-02-10T11:39:02.396 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:02.391 7f40b3fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f40b40ae800 0x7f40b4090b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:02.397 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:02.391 7f40b3fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f40b40ae800 0x7f40b4090b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:40040/0 (socket says 172.21.2.102:40040) 2021-02-10T11:39:02.397 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:02.391 7f40b3fff700 1 -- 172.21.2.102:0/2110026865 learned_addr learned my addr 172.21.2.102:0/2110026865 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:39:02.397 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:02.391 7f40b8afd700 1 --2- 172.21.2.102:0/2110026865 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f40b408aa40 0x7f40b408ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:02.397 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:02.391 7f40b37fe700 1 --2- 172.21.2.102:0/2110026865 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f40b4091190 0x7f40b408a420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:02.397 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:02.391 7f40b3fff700 1 -- 172.21.2.102:0/2110026865 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f40b408aa40 msgr2=0x7f40b408ae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:02.397 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:02.391 7f40b3fff700 1 --2- 172.21.2.102:0/2110026865 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f40b408aa40 0x7f40b408ae40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:02.397 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:02.391 7f40b3fff700 1 -- 172.21.2.102:0/2110026865 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f40b4091190 msgr2=0x7f40b408a420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:02.397 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:02.391 7f40b3fff700 1 --2- 172.21.2.102:0/2110026865 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f40b4091190 0x7f40b408a420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:02.397 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:02.391 7f40b3fff700 1 -- 172.21.2.102:0/2110026865 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40b40aec00 con 0x7f40b40ae800 2021-02-10T11:39:02.397 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:02.391 7f40b3fff700 1 --2- 172.21.2.102:0/2110026865 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f40b40ae800 0x7f40b4090b70 secure :-1 s=READY pgs=3174 cs=0 l=1 rev1=1 rx=0x7f40a4006f20 tx=0x7f40a400b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:02.397 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:02.391 7f40b0ff9700 1 -- 172.21.2.102:0/2110026865 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f40a40009b0 con 0x7f40b40ae800 2021-02-10T11:39:02.397 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:02.391 7f40b0ff9700 1 -- 172.21.2.102:0/2110026865 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40a400bd80 con 0x7f40b40ae800 2021-02-10T11:39:02.397 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:02.391 7f40ba4e0700 1 -- 172.21.2.102:0/2110026865 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40b40aec00 con 0x7f40b40ae800 2021-02-10T11:39:02.398 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:02.391 7f40b0ff9700 1 -- 172.21.2.102:0/2110026865 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f40a40022b0 con 0x7f40b40ae800 2021-02-10T11:39:02.398 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:02.391 7f40ba4e0700 1 -- 172.21.2.102:0/2110026865 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40b408f990 con 0x7f40b40ae800 2021-02-10T11:39:02.398 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:02.392 7f40b0ff9700 1 -- 172.21.2.102:0/2110026865 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f40a4000c90 con 0x7f40b40ae800 2021-02-10T11:39:02.398 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:02.392 7f40b0ff9700 1 --2- 172.21.2.102:0/2110026865 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f40940186c0 0x7f409401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:02.398 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:02.392 7f40b0ff9700 1 -- 172.21.2.102:0/2110026865 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(180..180 src has 1..180) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f40a4032580 con 0x7f40b40ae800 2021-02-10T11:39:02.398 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:02.392 7f40b37fe700 1 --2- 172.21.2.102:0/2110026865 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f40940186c0 0x7f409401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:02.398 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:02.392 7f40b37fe700 1 --2- 172.21.2.102:0/2110026865 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f40940186c0 0x7f409401abc0 crc :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:02.425 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:02.423 7f40ba4e0700 1 -- 172.21.2.102:0/2110026865 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40b408f990 con 0x7f40b40ae800 2021-02-10T11:39:02.428 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:02.425 7f40b0ff9700 1 -- 172.21.2.102:0/2110026865 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f40a40009b0 con 0x7f40b40ae800 2021-02-10T11:39:02.561 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:02.559 7f40ba4e0700 1 -- 172.21.2.102:0/2110026865 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "auth rm", "entity": "client.guest1"} v 0) v1 -- 0x7f40b408f990 con 0x7f40b40ae800 2021-02-10T11:39:02.561 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:02.559 7f40b0ff9700 1 -- 172.21.2.102:0/2110026865 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "auth rm", "entity": "client.guest1"}]=0 entity client.guest1 does not exist v107) v1 ==== 117+0+0 (secure 0 0 0) 0x7f40a4000c90 con 0x7f40b40ae800 2021-02-10T11:39:02.562 INFO:teuthology.orchestra.run.gibba002.stderr:entity client.guest1 does not exist 2021-02-10T11:39:02.563 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:02.561 7f40ba4e0700 1 -- 172.21.2.102:0/2110026865 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f40940186c0 msgr2=0x7f409401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:02.563 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:02.561 7f40ba4e0700 1 --2- 172.21.2.102:0/2110026865 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f40940186c0 0x7f409401abc0 crc :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:02.563 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:02.561 7f40ba4e0700 1 -- 172.21.2.102:0/2110026865 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f40b40ae800 msgr2=0x7f40b4090b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:02.563 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:02.561 7f40ba4e0700 1 --2- 172.21.2.102:0/2110026865 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f40b40ae800 0x7f40b4090b70 secure :-1 s=READY pgs=3174 cs=0 l=1 rev1=1 rx=0x7f40a4006f20 tx=0x7f40a400b990).stop 2021-02-10T11:39:02.563 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:02.561 7f40ba4e0700 1 -- 172.21.2.102:0/2110026865 shutdown_connections 2021-02-10T11:39:02.564 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:02.561 7f40ba4e0700 1 --2- 172.21.2.102:0/2110026865 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f40b408aa40 0x7f40b408ae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:02.564 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:02.561 7f40ba4e0700 1 --2- 172.21.2.102:0/2110026865 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f40940186c0 0x7f409401abc0 unknown :-1 s=CLOSED pgs=1389 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:02.564 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:02.561 7f40ba4e0700 1 --2- 172.21.2.102:0/2110026865 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f40b4091190 0x7f40b408a420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:02.564 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:02.561 7f40ba4e0700 1 --2- 172.21.2.102:0/2110026865 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f40b40ae800 0x7f40b4090b70 unknown :-1 s=CLOSED pgs=3174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:02.564 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:02.561 7f40ba4e0700 1 -- 172.21.2.102:0/2110026865 >> 172.21.2.102:0/2110026865 conn(0x7f40b40a0260 msgr2=0x7f40b4091840 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:02.564 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:02.561 7f40ba4e0700 1 -- 172.21.2.102:0/2110026865 shutdown_connections 2021-02-10T11:39:02.564 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:02.561 7f40ba4e0700 1 -- 172.21.2.102:0/2110026865 wait complete. 2021-02-10T11:39:02.572 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000423388 --group_name group_0000000000453502 2021-02-10T11:39:03.006 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:02.987 7f52cbc07700 1 Processor -- start 2021-02-10T11:39:03.006 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:02.987 7f52cbc07700 1 -- start start 2021-02-10T11:39:03.006 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:02.987 7f52cbc07700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f52c40d1880 0x7f52c41b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:03.006 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:02.987 7f52cbc07700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f52c41b0af0 0x7f52c4108950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:03.007 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:02.987 7f52cbc07700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52c4109a70 con 0x7f52c40d1c80 2021-02-10T11:39:03.007 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:02.987 7f52cbc07700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52c40e5f00 con 0x7f52c41b0af0 2021-02-10T11:39:03.007 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:02.987 7f52cbc07700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52c4109760 con 0x7f52c40d1880 2021-02-10T11:39:03.007 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:02.988 7f52c9a23700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f52c40d1880 0x7f52c41b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:03.007 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:02.988 7f52ca224700 1 --1- >> v1:172.21.2.113:6789/0 conn(0x7f52c40d1c80 0x7f52c4108f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.113:6789/0 says I am v1:172.21.2.102:39596/0 (socket says 172.21.2.102:39596) 2021-02-10T11:39:03.007 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:02.988 7f52c9222700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f52c41b0af0 0x7f52c4108950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:03.007 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:02.988 7f52ca224700 1 -- 172.21.2.102:0/3890613388 learned_addr learned my addr 172.21.2.102:0/3890613388 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:39:03.007 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:02.989 7f52c9a23700 1 -- 172.21.2.102:0/3890613388 >> v2:172.21.2.113:3300/0 conn(0x7f52c41b0af0 msgr2=0x7f52c4108950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:03.007 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:02.989 7f52c9a23700 1 --2- 172.21.2.102:0/3890613388 >> v2:172.21.2.113:3300/0 conn(0x7f52c41b0af0 0x7f52c4108950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:03.007 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:02.989 7f52c9a23700 1 -- 172.21.2.102:0/3890613388 >> v1:172.21.2.113:6789/0 conn(0x7f52c40d1c80 legacy=0x7f52c4108f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:03.007 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:02.989 7f52c9a23700 1 -- 172.21.2.102:0/3890613388 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52c4088070 con 0x7f52c40d1880 2021-02-10T11:39:03.007 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:02.989 7f52c9a23700 1 --2- 172.21.2.102:0/3890613388 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f52c40d1880 0x7f52c41b04d0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 rx=0x7f52b8006650 tx=0x7f52b8009a80).ready entity=mon.2 client_cookie=f5caed3be412159a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:03.008 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:02.989 7f52c8a21700 1 -- 172.21.2.102:0/3890613388 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f52b80009b0 con 0x7f52c40d1880 2021-02-10T11:39:03.008 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:02.989 7f52c8a21700 1 -- 172.21.2.102:0/3890613388 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52b800c0c0 con 0x7f52c40d1880 2021-02-10T11:39:03.008 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:02.989 7f52c8a21700 1 -- 172.21.2.102:0/3890613388 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f52b800c5e0 con 0x7f52c40d1880 2021-02-10T11:39:03.008 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:02.989 7f52cbc07700 1 -- 172.21.2.102:0/3890613388 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f52c40d1880 msgr2=0x7f52c41b04d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:03.008 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:02.989 7f52cbc07700 1 --2- 172.21.2.102:0/3890613388 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f52c40d1880 0x7f52c41b04d0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 rx=0x7f52b8006650 tx=0x7f52b8009a80).stop 2021-02-10T11:39:03.008 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:02.990 7f52cbc07700 1 -- 172.21.2.102:0/3890613388 shutdown_connections 2021-02-10T11:39:03.008 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:02.990 7f52cbc07700 1 --2- 172.21.2.102:0/3890613388 >> v2:172.21.2.113:3300/0 conn(0x7f52c41b0af0 0x7f52c4108950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:03.008 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:02.990 7f52cbc07700 1 --2- 172.21.2.102:0/3890613388 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f52c40d1880 0x7f52c41b04d0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:03.008 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:02.990 7f52cbc07700 1 -- 172.21.2.102:0/3890613388 >> 172.21.2.102:0/3890613388 conn(0x7f52c40ce430 msgr2=0x7f52c40d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:03.008 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:02.990 7f52cbc07700 1 -- 172.21.2.102:0/3890613388 shutdown_connections 2021-02-10T11:39:03.008 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:02.990 7f52cbc07700 1 -- 172.21.2.102:0/3890613388 wait complete. 2021-02-10T11:39:03.010 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:03.004 7f52cbc07700 1 Processor -- start 2021-02-10T11:39:03.010 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:03.004 7f52cbc07700 1 -- start start 2021-02-10T11:39:03.010 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:03.005 7f52cbc07700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f52c41a0580 0x7f52c4082c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:03.010 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:03.005 7f52cbc07700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f52c40832a0 0x7f52c40836a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:03.010 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:03.005 7f52cbc07700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f52c40b66d0 0x7f52c40b6ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:03.010 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:03.005 7f52cbc07700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52c40ba540 con 0x7f52c41a0580 2021-02-10T11:39:03.010 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:03.005 7f52cbc07700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52c40baac0 con 0x7f52c40b66d0 2021-02-10T11:39:03.010 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:03.005 7f52cbc07700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52c40b70f0 con 0x7f52c40832a0 2021-02-10T11:39:03.011 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:03.005 7f52c9a23700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f52c41a0580 0x7f52c4082c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:03.011 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:03.005 7f52c9a23700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f52c41a0580 0x7f52c4082c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:40054/0 (socket says 172.21.2.102:40054) 2021-02-10T11:39:03.011 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:03.005 7f52c9a23700 1 -- 172.21.2.102:0/84362276 learned_addr learned my addr 172.21.2.102:0/84362276 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:39:03.011 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:03.005 7f52c9222700 1 --2- 172.21.2.102:0/84362276 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f52c40832a0 0x7f52c40836a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:03.011 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:03.005 7f52c9a23700 1 -- 172.21.2.102:0/84362276 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f52c40832a0 msgr2=0x7f52c40836a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:03.011 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:03.005 7f52c9a23700 1 --2- 172.21.2.102:0/84362276 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f52c40832a0 0x7f52c40836a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:03.011 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:03.005 7f52c9a23700 1 -- 172.21.2.102:0/84362276 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f52c40b66d0 msgr2=0x7f52c40b6ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:03.011 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:03.005 7f52c9a23700 1 --2- 172.21.2.102:0/84362276 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f52c40b66d0 0x7f52c40b6ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:03.011 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:03.005 7f52c9a23700 1 -- 172.21.2.102:0/84362276 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52c40b7310 con 0x7f52c41a0580 2021-02-10T11:39:03.011 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:03.005 7f52c9222700 1 --2- 172.21.2.102:0/84362276 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f52c40832a0 0x7f52c40836a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:39:03.011 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:03.005 7f52c9a23700 1 --2- 172.21.2.102:0/84362276 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f52c41a0580 0x7f52c4082c80 secure :-1 s=READY pgs=3175 cs=0 l=1 rev1=1 rx=0x7f52b8009630 tx=0x7f52b800b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:03.011 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:03.006 7f52b67fc700 1 -- 172.21.2.102:0/84362276 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f52b80093a0 con 0x7f52c41a0580 2021-02-10T11:39:03.011 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:03.006 7f52b67fc700 1 -- 172.21.2.102:0/84362276 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52b800bd70 con 0x7f52c41a0580 2021-02-10T11:39:03.012 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:03.006 7f52b67fc700 1 -- 172.21.2.102:0/84362276 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f52b80054d0 con 0x7f52c41a0580 2021-02-10T11:39:03.012 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:03.006 7f52cbc07700 1 -- 172.21.2.102:0/84362276 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52c40b7310 con 0x7f52c41a0580 2021-02-10T11:39:03.012 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:03.006 7f52cbc07700 1 -- 172.21.2.102:0/84362276 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52c40b7310 con 0x7f52c41a0580 2021-02-10T11:39:03.012 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:03.006 7f52b67fc700 1 -- 172.21.2.102:0/84362276 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f52b8000c90 con 0x7f52c41a0580 2021-02-10T11:39:03.012 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:03.007 7f52b67fc700 1 --2- 172.21.2.102:0/84362276 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f52ac0186c0 0x7f52ac01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:03.012 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:03.007 7f52b67fc700 1 -- 172.21.2.102:0/84362276 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(180..180 src has 1..180) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f52b8032560 con 0x7f52c41a0580 2021-02-10T11:39:03.012 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:03.007 7f52c9222700 1 --2- 172.21.2.102:0/84362276 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f52ac0186c0 0x7f52ac01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:03.012 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:03.007 7f52c9222700 1 --2- 172.21.2.102:0/84362276 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f52ac0186c0 0x7f52ac01abc0 crc :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:03.039 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:03.037 7f52cbc07700 1 -- 172.21.2.102:0/84362276 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52c4082a50 con 0x7f52c41a0580 2021-02-10T11:39:03.042 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:03.040 7f52b67fc700 1 -- 172.21.2.102:0/84362276 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f52b8030d10 con 0x7f52c41a0580 2021-02-10T11:39:03.169 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:03.167 7f52cbc07700 1 -- 172.21.2.102:0/84362276 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000423388", "target": ["mgr", ""], "group_name": "group_0000000000453502"}) v1 -- 0x7f52c4082a50 con 0x7f52ac0186c0 2021-02-10T11:39:03.178 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:03.176 7f52b67fc700 1 -- 172.21.2.102:0/84362276 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f52c0006b30 con 0x7f52ac0186c0 2021-02-10T11:39:03.186 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:03.183 7f52cbc07700 1 -- 172.21.2.102:0/84362276 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f52ac0186c0 msgr2=0x7f52ac01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:03.186 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:03.183 7f52cbc07700 1 --2- 172.21.2.102:0/84362276 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f52ac0186c0 0x7f52ac01abc0 crc :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:03.186 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:03.183 7f52cbc07700 1 -- 172.21.2.102:0/84362276 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f52c41a0580 msgr2=0x7f52c4082c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:03.186 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:03.183 7f52cbc07700 1 --2- 172.21.2.102:0/84362276 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f52c41a0580 0x7f52c4082c80 secure :-1 s=READY pgs=3175 cs=0 l=1 rev1=1 rx=0x7f52b8009630 tx=0x7f52b800b980).stop 2021-02-10T11:39:03.186 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:03.184 7f52cbc07700 1 -- 172.21.2.102:0/84362276 shutdown_connections 2021-02-10T11:39:03.186 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:03.184 7f52cbc07700 1 --2- 172.21.2.102:0/84362276 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f52ac0186c0 0x7f52ac01abc0 unknown :-1 s=CLOSED pgs=1390 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:03.186 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:03.184 7f52cbc07700 1 --2- 172.21.2.102:0/84362276 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f52c40b66d0 0x7f52c40b6ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:03.187 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:03.184 7f52cbc07700 1 --2- 172.21.2.102:0/84362276 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f52c40832a0 0x7f52c40836a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:03.187 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:03.184 7f52cbc07700 1 --2- 172.21.2.102:0/84362276 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f52c41a0580 0x7f52c4082c80 unknown :-1 s=CLOSED pgs=3175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:03.187 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:03.184 7f52cbc07700 1 -- 172.21.2.102:0/84362276 >> 172.21.2.102:0/84362276 conn(0x7f52c40c20f0 msgr2=0x7f52c409e410 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:03.187 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:03.184 7f52cbc07700 1 -- 172.21.2.102:0/84362276 shutdown_connections 2021-02-10T11:39:03.187 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:03.184 7f52cbc07700 1 -- 172.21.2.102:0/84362276 wait complete. 2021-02-10T11:39:03.194 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000423389 --group_name group_0000000000453502 2021-02-10T11:39:03.625 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:03.607 7f3cbecf1700 1 Processor -- start 2021-02-10T11:39:03.625 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:03.608 7f3cbecf1700 1 -- start start 2021-02-10T11:39:03.626 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:03.608 7f3cbecf1700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f3cb80d1880 0x7f3cb81b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:03.626 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:03.608 7f3cbecf1700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cb80fe1d0 con 0x7f3cb81b0af0 2021-02-10T11:39:03.626 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:03.608 7f3cbecf1700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cb8109a70 con 0x7f3cb80cff80 2021-02-10T11:39:03.626 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:03.608 7f3cbecf1700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3cb8088070 con 0x7f3cb80d1880 2021-02-10T11:39:03.626 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:03.608 7f3cbcb0d700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f3cb80d1880 0x7f3cb81b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:03.626 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:03.608 7f3cbcb0d700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f3cb80d1880 0x7f3cb81b04d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:40062/0 (socket says 172.21.2.102:40062) 2021-02-10T11:39:03.626 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:03.608 7f3cb7fff700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f3cb81b0af0 0x7f3cb8108950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:60522/0 (socket says 172.21.2.102:60522) 2021-02-10T11:39:03.626 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:03.608 7f3cbcb0d700 1 -- 172.21.2.102:0/1573518635 learned_addr learned my addr 172.21.2.102:0/1573518635 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:39:03.626 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:03.608 7f3cbcb0d700 1 -- 172.21.2.102:0/1573518635 >> v1:172.21.2.113:6789/0 conn(0x7f3cb80cff80 legacy=0x7f3cb8109140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:03.626 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:03.608 7f3cb77fe700 1 -- 172.21.2.102:0/1573518635 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1526618795 0 0) 0x7f3cac0009b0 con 0x7f3cb81b0af0 2021-02-10T11:39:03.627 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:03.608 7f3cbcb0d700 1 -- 172.21.2.102:0/1573518635 >> v1:172.21.2.102:6789/0 conn(0x7f3cb81b0af0 legacy=0x7f3cb8108950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:03.627 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:03.608 7f3cbcb0d700 1 -- 172.21.2.102:0/1573518635 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cb8088260 con 0x7f3cb80d1880 2021-02-10T11:39:03.627 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:03.609 7f3cbcb0d700 1 --2- 172.21.2.102:0/1573518635 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3cb80d1880 0x7f3cb81b04d0 secure :-1 s=READY pgs=3177 cs=0 l=1 rev1=1 rx=0x7f3ca8006700 tx=0x7f3ca8009b00).ready entity=mon.0 client_cookie=1c19b2e94606dc server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:03.627 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:03.609 7f3cb77fe700 1 -- 172.21.2.102:0/1573518635 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3ca800c0c0 con 0x7f3cb80d1880 2021-02-10T11:39:03.627 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:03.609 7f3cb77fe700 1 -- 172.21.2.102:0/1573518635 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ca800c340 con 0x7f3cb80d1880 2021-02-10T11:39:03.627 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:03.609 7f3cb77fe700 1 -- 172.21.2.102:0/1573518635 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3ca800c7a0 con 0x7f3cb80d1880 2021-02-10T11:39:03.627 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:03.609 7f3cbecf1700 1 -- 172.21.2.102:0/1573518635 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3cb80d1880 msgr2=0x7f3cb81b04d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:03.627 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:03.609 7f3cbecf1700 1 --2- 172.21.2.102:0/1573518635 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3cb80d1880 0x7f3cb81b04d0 secure :-1 s=READY pgs=3177 cs=0 l=1 rev1=1 rx=0x7f3ca8006700 tx=0x7f3ca8009b00).stop 2021-02-10T11:39:03.627 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:03.609 7f3cbecf1700 1 -- 172.21.2.102:0/1573518635 shutdown_connections 2021-02-10T11:39:03.627 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:03.609 7f3cbecf1700 1 --2- 172.21.2.102:0/1573518635 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3cb80d1880 0x7f3cb81b04d0 unknown :-1 s=CLOSED pgs=3177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:03.627 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:03.609 7f3cbecf1700 1 -- 172.21.2.102:0/1573518635 >> 172.21.2.102:0/1573518635 conn(0x7f3cb80ce430 msgr2=0x7f3cb80d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:03.627 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:03.609 7f3cbecf1700 1 -- 172.21.2.102:0/1573518635 shutdown_connections 2021-02-10T11:39:03.628 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:03.609 7f3cbecf1700 1 -- 172.21.2.102:0/1573518635 wait complete. 2021-02-10T11:39:03.629 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:03.624 7f3cbecf1700 1 Processor -- start 2021-02-10T11:39:03.629 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:03.624 7f3cbecf1700 1 -- start start 2021-02-10T11:39:03.629 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:03.624 7f3cbecf1700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3cb809ef20 0x7f3cb8187e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:03.629 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:03.624 7f3cbecf1700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3cb81884a0 0x7f3cb81888a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:03.629 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:03.624 7f3cbecf1700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3cb8189840 0x7f3cb8189c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:03.629 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:03.624 7f3cbecf1700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cb81b3c10 con 0x7f3cb81884a0 2021-02-10T11:39:03.630 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:03.624 7f3cbecf1700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cb81964f0 con 0x7f3cb8189840 2021-02-10T11:39:03.630 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:03.624 7f3cbecf1700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3cb8188ec0 con 0x7f3cb809ef20 2021-02-10T11:39:03.630 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:03.624 7f3cb7fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3cb81884a0 0x7f3cb81888a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:03.630 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:03.624 7f3cb7fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3cb81884a0 0x7f3cb81888a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:40070/0 (socket says 172.21.2.102:40070) 2021-02-10T11:39:03.630 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:03.624 7f3cb7fff700 1 -- 172.21.2.102:0/4058371800 learned_addr learned my addr 172.21.2.102:0/4058371800 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:39:03.630 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:03.625 7f3cb7fff700 1 -- 172.21.2.102:0/4058371800 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3cb809ef20 msgr2=0x7f3cb8187e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:03.630 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:03.625 7f3cb7fff700 1 --2- 172.21.2.102:0/4058371800 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3cb809ef20 0x7f3cb8187e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:03.630 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:03.625 7f3cb7fff700 1 -- 172.21.2.102:0/4058371800 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3cb8189840 msgr2=0x7f3cb8189c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:03.630 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:03.625 7f3cb7fff700 1 --2- 172.21.2.102:0/4058371800 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3cb8189840 0x7f3cb8189c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:03.630 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:03.625 7f3cb7fff700 1 -- 172.21.2.102:0/4058371800 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cb809f320 con 0x7f3cb81884a0 2021-02-10T11:39:03.630 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:03.625 7f3cb7fff700 1 --2- 172.21.2.102:0/4058371800 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3cb81884a0 0x7f3cb81888a0 secure :-1 s=READY pgs=3178 cs=0 l=1 rev1=1 rx=0x7f3cac001c60 tx=0x7f3cac00a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:03.630 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:03.625 7f3cb57fa700 1 -- 172.21.2.102:0/4058371800 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3cac00acc0 con 0x7f3cb81884a0 2021-02-10T11:39:03.631 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:03.625 7f3cb57fa700 1 -- 172.21.2.102:0/4058371800 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cac004140 con 0x7f3cb81884a0 2021-02-10T11:39:03.631 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:03.625 7f3cb57fa700 1 -- 172.21.2.102:0/4058371800 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3cac0045e0 con 0x7f3cb81884a0 2021-02-10T11:39:03.631 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:03.625 7f3cbecf1700 1 -- 172.21.2.102:0/4058371800 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3cb809f320 con 0x7f3cb81884a0 2021-02-10T11:39:03.631 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:03.625 7f3cbecf1700 1 -- 172.21.2.102:0/4058371800 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3cb818a550 con 0x7f3cb81884a0 2021-02-10T11:39:03.631 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:03.626 7f3cb57fa700 1 -- 172.21.2.102:0/4058371800 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f3cac004320 con 0x7f3cb81884a0 2021-02-10T11:39:03.631 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:03.626 7f3cb57fa700 1 --2- 172.21.2.102:0/4058371800 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3c980186c0 0x7f3c9801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:03.631 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:03.626 7f3cb57fa700 1 -- 172.21.2.102:0/4058371800 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(180..180 src has 1..180) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f3cac032520 con 0x7f3cb81884a0 2021-02-10T11:39:03.631 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:03.626 7f3cbcb0d700 1 --2- 172.21.2.102:0/4058371800 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3c980186c0 0x7f3c9801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:03.631 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:03.626 7f3cbcb0d700 1 --2- 172.21.2.102:0/4058371800 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3c980186c0 0x7f3c9801abc0 crc :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:03.660 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:03.658 7f3cbecf1700 1 -- 172.21.2.102:0/4058371800 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3cb8082ad0 con 0x7f3cb81884a0 2021-02-10T11:39:03.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:03.660 7f3cb57fa700 1 -- 172.21.2.102:0/4058371800 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f3cac000dd0 con 0x7f3cb81884a0 2021-02-10T11:39:03.789 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:03.787 7f3cbecf1700 1 -- 172.21.2.102:0/4058371800 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000423389", "target": ["mgr", ""], "group_name": "group_0000000000453502"}) v1 -- 0x7f3cb818a550 con 0x7f3c980186c0 2021-02-10T11:39:03.798 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:03.796 7f3cb57fa700 1 -- 172.21.2.102:0/4058371800 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f3ca8006390 con 0x7f3c980186c0 2021-02-10T11:39:03.806 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:03.804 7f3cbecf1700 1 -- 172.21.2.102:0/4058371800 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3c980186c0 msgr2=0x7f3c9801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:03.806 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:03.804 7f3cbecf1700 1 --2- 172.21.2.102:0/4058371800 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3c980186c0 0x7f3c9801abc0 crc :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:03.806 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:03.804 7f3cbecf1700 1 -- 172.21.2.102:0/4058371800 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3cb81884a0 msgr2=0x7f3cb81888a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:03.807 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:03.804 7f3cbecf1700 1 --2- 172.21.2.102:0/4058371800 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3cb81884a0 0x7f3cb81888a0 secure :-1 s=READY pgs=3178 cs=0 l=1 rev1=1 rx=0x7f3cac001c60 tx=0x7f3cac00a8d0).stop 2021-02-10T11:39:03.807 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:03.805 7f3cbecf1700 1 -- 172.21.2.102:0/4058371800 shutdown_connections 2021-02-10T11:39:03.807 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:03.805 7f3cbecf1700 1 --2- 172.21.2.102:0/4058371800 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3c980186c0 0x7f3c9801abc0 unknown :-1 s=CLOSED pgs=1391 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:03.807 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:03.805 7f3cbecf1700 1 --2- 172.21.2.102:0/4058371800 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3cb8189840 0x7f3cb8189c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:03.807 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:03.805 7f3cbecf1700 1 --2- 172.21.2.102:0/4058371800 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3cb81884a0 0x7f3cb81888a0 unknown :-1 s=CLOSED pgs=3178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:03.807 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:03.805 7f3cbecf1700 1 --2- 172.21.2.102:0/4058371800 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3cb809ef20 0x7f3cb8187e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:03.808 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:03.805 7f3cbecf1700 1 -- 172.21.2.102:0/4058371800 >> 172.21.2.102:0/4058371800 conn(0x7f3cb819f970 msgr2=0x7f3cb81a61d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:03.808 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:03.805 7f3cbecf1700 1 -- 172.21.2.102:0/4058371800 shutdown_connections 2021-02-10T11:39:03.808 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:03.805 7f3cbecf1700 1 -- 172.21.2.102:0/4058371800 wait complete. 2021-02-10T11:39:03.815 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolumegroup rm cephfs group_0000000000453502 2021-02-10T11:39:04.246 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:04.228 7f3d05c46700 1 Processor -- start 2021-02-10T11:39:04.247 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:04.228 7f3d05c46700 1 -- start start 2021-02-10T11:39:04.247 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:04.228 7f3d05c46700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f3d000d1870 0x7f3d001b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:04.247 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:04.228 7f3d05c46700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f3d001b0ae0 0x7f3d00108940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:04.247 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:04.228 7f3d05c46700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d000fe1c0 con 0x7f3d000d1c70 2021-02-10T11:39:04.247 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:04.228 7f3d05c46700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d00109a60 con 0x7f3d001b0ae0 2021-02-10T11:39:04.247 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:04.228 7f3d05c46700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d00109750 con 0x7f3d000d1870 2021-02-10T11:39:04.247 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:04.229 7f3cfffff700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f3d000d1c70 0x7f3d00108f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:60536/0 (socket says 172.21.2.102:60536) 2021-02-10T11:39:04.247 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:04.229 7f3cfffff700 1 -- 172.21.2.102:0/1015465664 learned_addr learned my addr 172.21.2.102:0/1015465664 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:39:04.247 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:04.229 7f3cfeffd700 1 --2- 172.21.2.102:0/1015465664 >> v2:172.21.2.102:3300/0 conn(0x7f3d001b0ae0 0x7f3d00108940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:04.247 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:04.229 7f3cff7fe700 1 --2- 172.21.2.102:0/1015465664 >> v2:172.21.2.113:3300/0 conn(0x7f3d000d1870 0x7f3d001b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:04.247 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:04.229 7f3cfeffd700 1 -- 172.21.2.102:0/1015465664 >> v2:172.21.2.113:3300/0 conn(0x7f3d000d1870 msgr2=0x7f3d001b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:04.247 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:04.229 7f3cfeffd700 1 --2- 172.21.2.102:0/1015465664 >> v2:172.21.2.113:3300/0 conn(0x7f3d000d1870 0x7f3d001b04c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:04.248 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:04.229 7f3cfeffd700 1 -- 172.21.2.102:0/1015465664 >> v1:172.21.2.102:6789/0 conn(0x7f3d000d1c70 legacy=0x7f3d00108f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:04.248 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:04.229 7f3cfeffd700 1 -- 172.21.2.102:0/1015465664 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d00088070 con 0x7f3d001b0ae0 2021-02-10T11:39:04.248 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:04.229 7f3cff7fe700 1 --2- 172.21.2.102:0/1015465664 >> v2:172.21.2.113:3300/0 conn(0x7f3d000d1870 0x7f3d001b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:39:04.248 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:04.229 7f3cfeffd700 1 --2- 172.21.2.102:0/1015465664 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3d001b0ae0 0x7f3d00108940 secure :-1 s=READY pgs=3180 cs=0 l=1 rev1=1 rx=0x7f3cf0006650 tx=0x7f3cf0009a80).ready entity=mon.0 client_cookie=59781bfe530cb570 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:04.248 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:04.229 7f3cfe7fc700 1 -- 172.21.2.102:0/1015465664 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3cf000c0c0 con 0x7f3d001b0ae0 2021-02-10T11:39:04.248 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:04.229 7f3cfe7fc700 1 -- 172.21.2.102:0/1015465664 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cf000c340 con 0x7f3d001b0ae0 2021-02-10T11:39:04.248 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:04.229 7f3cfe7fc700 1 -- 172.21.2.102:0/1015465664 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3cf000c860 con 0x7f3d001b0ae0 2021-02-10T11:39:04.248 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:04.230 7f3d05c46700 1 -- 172.21.2.102:0/1015465664 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3d001b0ae0 msgr2=0x7f3d00108940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:04.248 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:04.230 7f3d05c46700 1 --2- 172.21.2.102:0/1015465664 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3d001b0ae0 0x7f3d00108940 secure :-1 s=READY pgs=3180 cs=0 l=1 rev1=1 rx=0x7f3cf0006650 tx=0x7f3cf0009a80).stop 2021-02-10T11:39:04.248 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:04.230 7f3d05c46700 1 -- 172.21.2.102:0/1015465664 shutdown_connections 2021-02-10T11:39:04.248 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:04.230 7f3d05c46700 1 --2- 172.21.2.102:0/1015465664 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3d001b0ae0 0x7f3d00108940 unknown :-1 s=CLOSED pgs=3180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:04.248 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:04.230 7f3d05c46700 1 --2- 172.21.2.102:0/1015465664 >> v2:172.21.2.113:3300/0 conn(0x7f3d000d1870 0x7f3d001b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:04.249 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:04.230 7f3d05c46700 1 -- 172.21.2.102:0/1015465664 >> 172.21.2.102:0/1015465664 conn(0x7f3d000ce420 msgr2=0x7f3d000d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:04.249 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:04.230 7f3d05c46700 1 -- 172.21.2.102:0/1015465664 shutdown_connections 2021-02-10T11:39:04.249 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:04.230 7f3d05c46700 1 -- 172.21.2.102:0/1015465664 wait complete. 2021-02-10T11:39:04.250 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:04.245 7f3d05c46700 1 Processor -- start 2021-02-10T11:39:04.250 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:04.245 7f3d05c46700 1 -- start start 2021-02-10T11:39:04.251 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:04.245 7f3d05c46700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3d00191b10 0x7f3d000ae640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:04.251 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:04.245 7f3d05c46700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3d00191f10 0x7f3d0019ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:04.251 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:04.245 7f3d05c46700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3d001a05f0 0x7f3d00082ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:04.251 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:04.245 7f3d05c46700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d00192440 con 0x7f3d00191b10 2021-02-10T11:39:04.251 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:04.245 7f3d05c46700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d001916e0 con 0x7f3d00191f10 2021-02-10T11:39:04.251 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:04.245 7f3d05c46700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d00191900 con 0x7f3d001a05f0 2021-02-10T11:39:04.251 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:04.246 7f3cff7fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3d00191b10 0x7f3d000ae640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:04.251 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:04.246 7f3cff7fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3d00191b10 0x7f3d000ae640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:40084/0 (socket says 172.21.2.102:40084) 2021-02-10T11:39:04.251 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:04.246 7f3cff7fe700 1 -- 172.21.2.102:0/746849918 learned_addr learned my addr 172.21.2.102:0/746849918 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:39:04.251 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:04.246 7f3cfeffd700 1 --2- 172.21.2.102:0/746849918 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3d00191f10 0x7f3d0019ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:04.251 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:04.246 7f3cfffff700 1 --2- 172.21.2.102:0/746849918 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3d001a05f0 0x7f3d00082ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:04.251 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:04.246 7f3cff7fe700 1 -- 172.21.2.102:0/746849918 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3d001a05f0 msgr2=0x7f3d00082ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:04.251 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:04.246 7f3cff7fe700 1 --2- 172.21.2.102:0/746849918 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3d001a05f0 0x7f3d00082ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:04.252 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:04.246 7f3cff7fe700 1 -- 172.21.2.102:0/746849918 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3d00191f10 msgr2=0x7f3d0019ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:04.252 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:04.246 7f3cff7fe700 1 --2- 172.21.2.102:0/746849918 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3d00191f10 0x7f3d0019ffd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:04.252 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:04.246 7f3cff7fe700 1 -- 172.21.2.102:0/746849918 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d000834f0 con 0x7f3d00191b10 2021-02-10T11:39:04.252 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:04.246 7f3cff7fe700 1 --2- 172.21.2.102:0/746849918 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3d00191b10 0x7f3d000ae640 secure :-1 s=READY pgs=3181 cs=0 l=1 rev1=1 rx=0x7f3ce800b580 tx=0x7f3ce800b920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:04.252 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:04.246 7f3cf7fff700 1 -- 172.21.2.102:0/746849918 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3ce800bd10 con 0x7f3d00191b10 2021-02-10T11:39:04.252 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:04.246 7f3cf7fff700 1 -- 172.21.2.102:0/746849918 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ce8000c90 con 0x7f3d00191b10 2021-02-10T11:39:04.252 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:04.246 7f3d05c46700 1 -- 172.21.2.102:0/746849918 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d000834f0 con 0x7f3d00191b10 2021-02-10T11:39:04.252 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:04.246 7f3cf7fff700 1 -- 172.21.2.102:0/746849918 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3ce80022b0 con 0x7f3d00191b10 2021-02-10T11:39:04.252 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:04.246 7f3d05c46700 1 -- 172.21.2.102:0/746849918 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d00083a80 con 0x7f3d00191b10 2021-02-10T11:39:04.252 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:04.247 7f3cf7fff700 1 -- 172.21.2.102:0/746849918 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f3ce800a000 con 0x7f3d00191b10 2021-02-10T11:39:04.252 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:04.247 7f3cf7fff700 1 --2- 172.21.2.102:0/746849918 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3ce40186c0 0x7f3ce401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:04.252 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:04.247 7f3cf7fff700 1 -- 172.21.2.102:0/746849918 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(180..180 src has 1..180) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f3ce80327b0 con 0x7f3d00191b10 2021-02-10T11:39:04.253 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:04.247 7f3cfeffd700 1 --2- 172.21.2.102:0/746849918 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3ce40186c0 0x7f3ce401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:04.253 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:04.247 7f3cfeffd700 1 --2- 172.21.2.102:0/746849918 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3ce40186c0 0x7f3ce401abc0 crc :-1 s=READY pgs=1392 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:04.280 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:04.278 7f3d05c46700 1 -- 172.21.2.102:0/746849918 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d00191900 con 0x7f3d00191b10 2021-02-10T11:39:04.283 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:04.281 7f3cf7fff700 1 -- 172.21.2.102:0/746849918 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f3ce8000940 con 0x7f3d00191b10 2021-02-10T11:39:04.411 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:04.409 7f3d05c46700 1 -- 172.21.2.102:0/746849918 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolumegroup rm", "vol_name": "cephfs", "target": ["mgr", ""], "group_name": "group_0000000000453502"}) v1 -- 0x7f3d00083a80 con 0x7f3ce40186c0 2021-02-10T11:39:04.414 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:04.412 7f3cf7fff700 1 -- 172.21.2.102:0/746849918 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f3cf0000c90 con 0x7f3ce40186c0 2021-02-10T11:39:04.420 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:04.417 7f3d05c46700 1 -- 172.21.2.102:0/746849918 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3ce40186c0 msgr2=0x7f3ce401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:04.420 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:04.417 7f3d05c46700 1 --2- 172.21.2.102:0/746849918 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3ce40186c0 0x7f3ce401abc0 crc :-1 s=READY pgs=1392 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:04.420 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:04.417 7f3d05c46700 1 -- 172.21.2.102:0/746849918 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3d00191b10 msgr2=0x7f3d000ae640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:04.420 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:04.417 7f3d05c46700 1 --2- 172.21.2.102:0/746849918 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3d00191b10 0x7f3d000ae640 secure :-1 s=READY pgs=3181 cs=0 l=1 rev1=1 rx=0x7f3ce800b580 tx=0x7f3ce800b920).stop 2021-02-10T11:39:04.420 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:04.418 7f3d05c46700 1 -- 172.21.2.102:0/746849918 shutdown_connections 2021-02-10T11:39:04.420 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:04.418 7f3d05c46700 1 --2- 172.21.2.102:0/746849918 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3d001a05f0 0x7f3d00082ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:04.420 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:04.418 7f3d05c46700 1 --2- 172.21.2.102:0/746849918 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3ce40186c0 0x7f3ce401abc0 unknown :-1 s=CLOSED pgs=1392 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:04.421 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:04.418 7f3d05c46700 1 --2- 172.21.2.102:0/746849918 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3d00191f10 0x7f3d0019ffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:04.421 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:04.418 7f3d05c46700 1 --2- 172.21.2.102:0/746849918 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3d00191b10 0x7f3d000ae640 unknown :-1 s=CLOSED pgs=3181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:04.421 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:04.418 7f3d05c46700 1 -- 172.21.2.102:0/746849918 >> 172.21.2.102:0/746849918 conn(0x7f3d000a02d0 msgr2=0x7f3d0009f200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:04.421 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:04.418 7f3d05c46700 1 -- 172.21.2.102:0/746849918 shutdown_connections 2021-02-10T11:39:04.421 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:04.418 7f3d05c46700 1 -- 172.21.2.102:0/746849918 wait complete. 2021-02-10T11:39:04.428 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2021-02-10T11:39:04.428 DEBUG:teuthology.orchestra.run.gibba002:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:39:04.775 DEBUG:teuthology.orchestra.run.gibba013:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:39:05.119 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2021-02-10T11:39:05.120 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:39:05.120 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:39:05.120 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:39:05.457 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:39:05.457 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:39:05.458 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@gibba002.front.sepia.ceph.com... 2021-02-10T11:39:05.458 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-02-10T11:39:05.458 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:39:05.809 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba002.stderr:ceph-fuse[212521]: fuse finished with error 0 and tester_r 0 2021-02-10T11:39:05.810 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:39:05.811 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:39:05.811 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:39:06.145 INFO:teuthology.orchestra.run.gibba002.stdout:ext2/ext3 2021-02-10T11:39:06.145 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2021-02-10T11:39:06.146 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:39:06.146 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:39:06.482 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:39:06.482 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:39:06.482 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:39:06.819 INFO:teuthology.orchestra.run.gibba002.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.1’: No such file or directory 2021-02-10T11:39:10.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:39:10.428 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2021-02-10T11:39:10.429 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:39:10.429 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:39:10.766 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:39:10.766 DEBUG:teuthology.orchestra.run.gibba013:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:39:10.766 DEBUG:teuthology.orchestra.run.gibba013:> ' -- /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:39:11.102 INFO:teuthology.orchestra.run.gibba013.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.2’: No such file or directory 2021-02-10T11:39:11.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:39:11.102 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2021-02-10T11:39:11.103 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:39:11.103 DEBUG:teuthology.orchestra.run.gibba013:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:39:11.436 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:39:11.437 DEBUG:teuthology.orchestra.run.gibba013:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:39:11.437 DEBUG:teuthology.orchestra.run.gibba013:> ' -- /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:39:11.775 INFO:teuthology.orchestra.run.gibba013.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.3’: No such file or directory 2021-02-10T11:39:11.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:39:11.775 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.3 2021-02-10T11:39:11.776 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:39:11.776 DEBUG:teuthology.orchestra.run.gibba013:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:39:12.120 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config rm mon mon_allow_pool_delete 2021-02-10T11:39:12.552 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:12.533 7f9cde5a0700 1 Processor -- start 2021-02-10T11:39:12.552 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:12.533 7f9cde5a0700 1 -- start start 2021-02-10T11:39:12.552 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:12.533 7f9cde5a0700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f9cd80d0340 0x7f9cd80ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:12.552 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:12.533 7f9cde5a0700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9cd80f0b10 con 0x7f9cd80d1c70 2021-02-10T11:39:12.552 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:12.533 7f9cde5a0700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9cd81b0be0 con 0x7f9cd80d1870 2021-02-10T11:39:12.552 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:12.533 7f9cde5a0700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9cd80e5ed0 con 0x7f9cd80d0340 2021-02-10T11:39:12.552 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:12.533 7f9cd77fe700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f9cd80d0340 0x7f9cd80ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:12.553 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:12.533 7f9cdcbbd700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f9cd80d1c70 0x7f9cd80efc60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:60546/0 (socket says 172.21.2.102:60546) 2021-02-10T11:39:12.553 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:12.533 7f9cdcbbd700 1 -- 172.21.2.102:0/258035871 learned_addr learned my addr 172.21.2.102:0/258035871 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:39:12.553 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:12.534 7f9cd77fe700 1 -- 172.21.2.102:0/258035871 >> v1:172.21.2.113:6789/0 conn(0x7f9cd80d1870 legacy=0x7f9cd81b03f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:39:12.553 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:12.534 7f9cd77fe700 1 -- 172.21.2.102:0/258035871 >> v1:172.21.2.102:6789/0 conn(0x7f9cd80d1c70 legacy=0x7f9cd80efc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:12.553 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:12.534 7f9cd77fe700 1 -- 172.21.2.102:0/258035871 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cd80f0450 con 0x7f9cd80d0340 2021-02-10T11:39:12.553 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:12.534 7f9cd77fe700 1 --2- 172.21.2.102:0/258035871 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9cd80d0340 0x7f9cd80ef640 secure :-1 s=READY pgs=3183 cs=0 l=1 rev1=1 rx=0x7f9ccc006650 tx=0x7f9ccc009a80).ready entity=mon.0 client_cookie=145dda9969fe136a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:12.553 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:12.534 7f9cd6ffd700 1 -- 172.21.2.102:0/258035871 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9ccc00c0c0 con 0x7f9cd80d0340 2021-02-10T11:39:12.553 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:12.534 7f9cd6ffd700 1 -- 172.21.2.102:0/258035871 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ccc00c340 con 0x7f9cd80d0340 2021-02-10T11:39:12.553 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:12.534 7f9cd6ffd700 1 -- 172.21.2.102:0/258035871 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9ccc00c820 con 0x7f9cd80d0340 2021-02-10T11:39:12.553 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:12.534 7f9cde5a0700 1 -- 172.21.2.102:0/258035871 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9cd80d0340 msgr2=0x7f9cd80ef640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:12.553 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:12.534 7f9cde5a0700 1 --2- 172.21.2.102:0/258035871 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9cd80d0340 0x7f9cd80ef640 secure :-1 s=READY pgs=3183 cs=0 l=1 rev1=1 rx=0x7f9ccc006650 tx=0x7f9ccc009a80).stop 2021-02-10T11:39:12.553 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:12.535 7f9cde5a0700 1 -- 172.21.2.102:0/258035871 shutdown_connections 2021-02-10T11:39:12.553 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:12.535 7f9cde5a0700 1 --2- 172.21.2.102:0/258035871 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9cd80d0340 0x7f9cd80ef640 unknown :-1 s=CLOSED pgs=3183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:12.554 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:12.535 7f9cde5a0700 1 -- 172.21.2.102:0/258035871 >> 172.21.2.102:0/258035871 conn(0x7f9cd80ce450 msgr2=0x7f9cd80d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:12.554 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:12.535 7f9cde5a0700 1 -- 172.21.2.102:0/258035871 shutdown_connections 2021-02-10T11:39:12.554 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:12.535 7f9cde5a0700 1 -- 172.21.2.102:0/258035871 wait complete. 2021-02-10T11:39:12.555 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:12.550 7f9cde5a0700 1 Processor -- start 2021-02-10T11:39:12.556 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:12.550 7f9cde5a0700 1 -- start start 2021-02-10T11:39:12.556 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:12.551 7f9cde5a0700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9cd81b00f0 0x7f9cd81b04f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:12.556 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:12.551 7f9cde5a0700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f9cd81b0b10 0x7f9cd81b0f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:12.556 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:12.551 7f9cde5a0700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9cd80943f0 0x7f9cd80947f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:12.556 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:12.551 7f9cde5a0700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cd80a1f20 con 0x7f9cd80943f0 2021-02-10T11:39:12.556 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:12.551 7f9cde5a0700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cd8094e10 con 0x7f9cd81b0b10 2021-02-10T11:39:12.556 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:12.551 7f9cde5a0700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9cd8094ff0 con 0x7f9cd81b00f0 2021-02-10T11:39:12.556 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:12.551 7f9cdcbbd700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9cd80943f0 0x7f9cd80947f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:12.556 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:12.551 7f9cdcbbd700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9cd80943f0 0x7f9cd80947f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:40098/0 (socket says 172.21.2.102:40098) 2021-02-10T11:39:12.556 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:12.551 7f9cdcbbd700 1 -- 172.21.2.102:0/4152418166 learned_addr learned my addr 172.21.2.102:0/4152418166 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:39:12.556 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:12.551 7f9cd77fe700 1 --2- 172.21.2.102:0/4152418166 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f9cd81b0b10 0x7f9cd81b0f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:12.556 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:12.551 7f9cd7fff700 1 --2- 172.21.2.102:0/4152418166 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9cd81b00f0 0x7f9cd81b04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:12.557 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:12.551 7f9cdcbbd700 1 -- 172.21.2.102:0/4152418166 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9cd81b00f0 msgr2=0x7f9cd81b04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:12.557 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:12.551 7f9cdcbbd700 1 --2- 172.21.2.102:0/4152418166 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9cd81b00f0 0x7f9cd81b04f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:12.557 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:12.551 7f9cdcbbd700 1 -- 172.21.2.102:0/4152418166 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f9cd81b0b10 msgr2=0x7f9cd81b0f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:12.557 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:12.551 7f9cdcbbd700 1 --2- 172.21.2.102:0/4152418166 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f9cd81b0b10 0x7f9cd81b0f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:12.557 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:12.551 7f9cdcbbd700 1 -- 172.21.2.102:0/4152418166 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cd8095210 con 0x7f9cd80943f0 2021-02-10T11:39:12.557 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:12.552 7f9cdcbbd700 1 --2- 172.21.2.102:0/4152418166 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9cd80943f0 0x7f9cd80947f0 secure :-1 s=READY pgs=3184 cs=0 l=1 rev1=1 rx=0x7f9cd000a940 tx=0x7f9cd000ace0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:12.557 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:12.552 7f9cd4ff9700 1 -- 172.21.2.102:0/4152418166 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9cd00070d0 con 0x7f9cd80943f0 2021-02-10T11:39:12.557 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:12.552 7f9cd4ff9700 1 -- 172.21.2.102:0/4152418166 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cd0000f90 con 0x7f9cd80943f0 2021-02-10T11:39:12.557 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:12.552 7f9cd4ff9700 1 -- 172.21.2.102:0/4152418166 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9cd0004720 con 0x7f9cd80943f0 2021-02-10T11:39:12.557 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:12.552 7f9cde5a0700 1 -- 172.21.2.102:0/4152418166 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9cd8095210 con 0x7f9cd80943f0 2021-02-10T11:39:12.557 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:12.552 7f9cde5a0700 1 -- 172.21.2.102:0/4152418166 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9cd8095210 con 0x7f9cd80943f0 2021-02-10T11:39:12.557 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:12.553 7f9cd4ff9700 1 -- 172.21.2.102:0/4152418166 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f9cd0007350 con 0x7f9cd80943f0 2021-02-10T11:39:12.557 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:12.553 7f9cd4ff9700 1 --2- 172.21.2.102:0/4152418166 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f9cb80186c0 0x7f9cb801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:12.558 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:12.553 7f9cd4ff9700 1 -- 172.21.2.102:0/4152418166 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(180..180 src has 1..180) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f9cd00329d0 con 0x7f9cd80943f0 2021-02-10T11:39:12.558 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:12.553 7f9cd7fff700 1 --2- 172.21.2.102:0/4152418166 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f9cb80186c0 0x7f9cb801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:12.558 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:12.553 7f9cd7fff700 1 --2- 172.21.2.102:0/4152418166 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f9cb80186c0 0x7f9cb801abc0 crc :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:12.585 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:12.583 7f9cde5a0700 1 -- 172.21.2.102:0/4152418166 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9cd80ffa40 con 0x7f9cd80943f0 2021-02-10T11:39:12.588 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:12.586 7f9cd4ff9700 1 -- 172.21.2.102:0/4152418166 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f9cd0007cb0 con 0x7f9cd80943f0 2021-02-10T11:39:12.734 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:12.732 7f9cde5a0700 1 -- 172.21.2.102:0/4152418166 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f9cd80f06f0 con 0x7f9cd80943f0 2021-02-10T11:39:12.738 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:12.736 7f9cd4ff9700 1 -- 172.21.2.102:0/4152418166 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v45) v1 ==== 104+0+0 (secure 0 0 0) 0x7f9cd0007350 con 0x7f9cd80943f0 2021-02-10T11:39:12.743 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:12.741 7f9cde5a0700 1 -- 172.21.2.102:0/4152418166 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f9cb80186c0 msgr2=0x7f9cb801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:12.743 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:12.741 7f9cde5a0700 1 --2- 172.21.2.102:0/4152418166 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f9cb80186c0 0x7f9cb801abc0 crc :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:12.743 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:12.741 7f9cde5a0700 1 -- 172.21.2.102:0/4152418166 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9cd80943f0 msgr2=0x7f9cd80947f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:12.743 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:12.741 7f9cde5a0700 1 --2- 172.21.2.102:0/4152418166 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9cd80943f0 0x7f9cd80947f0 secure :-1 s=READY pgs=3184 cs=0 l=1 rev1=1 rx=0x7f9cd000a940 tx=0x7f9cd000ace0).stop 2021-02-10T11:39:12.743 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:12.741 7f9cde5a0700 1 -- 172.21.2.102:0/4152418166 shutdown_connections 2021-02-10T11:39:12.743 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:12.741 7f9cde5a0700 1 --2- 172.21.2.102:0/4152418166 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9cd80943f0 0x7f9cd80947f0 unknown :-1 s=CLOSED pgs=3184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:12.744 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:12.741 7f9cde5a0700 1 --2- 172.21.2.102:0/4152418166 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f9cb80186c0 0x7f9cb801abc0 unknown :-1 s=CLOSED pgs=1393 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:12.744 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:12.741 7f9cde5a0700 1 --2- 172.21.2.102:0/4152418166 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f9cd81b0b10 0x7f9cd81b0f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:12.744 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:12.741 7f9cde5a0700 1 --2- 172.21.2.102:0/4152418166 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9cd81b00f0 0x7f9cd81b04f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:12.744 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:12.741 7f9cde5a0700 1 -- 172.21.2.102:0/4152418166 >> 172.21.2.102:0/4152418166 conn(0x7f9cd80bd580 msgr2=0x7f9cd8098cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:12.744 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:12.741 7f9cde5a0700 1 -- 172.21.2.102:0/4152418166 shutdown_connections 2021-02-10T11:39:12.744 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:12.741 7f9cde5a0700 1 -- 172.21.2.102:0/4152418166 wait complete. 2021-02-10T11:39:12.751 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestVolumes.test_recover_auth_metadata_during_deauthorize' 2021-02-10T11:39:13.184 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:13.164 7fc73e8df700 1 Processor -- start 2021-02-10T11:39:13.184 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:13.164 7fc73e8df700 1 -- start start 2021-02-10T11:39:13.185 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:13.165 7fc73e8df700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fc7380f05f0 0x7fc7380f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:13.185 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:13.165 7fc73e8df700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7381b0c70 con 0x7fc7381b0470 2021-02-10T11:39:13.185 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:13.165 7fc73e8df700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7380f0b50 con 0x7fc7381b0870 2021-02-10T11:39:13.185 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:13.165 7fc73e8df700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7380e5ea0 con 0x7fc7380f05f0 2021-02-10T11:39:13.185 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:13.165 7fc737fff700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fc7381b0470 0x7fc7380ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:60560/0 (socket says 172.21.2.102:60560) 2021-02-10T11:39:13.185 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:13.165 7fc737fff700 1 -- 172.21.2.102:0/490455949 learned_addr learned my addr 172.21.2.102:0/490455949 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:39:13.185 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:13.165 7fc736ffd700 1 -- 172.21.2.102:0/490455949 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1588579108 0 0) 0x7fc728001180 con 0x7fc7381b0470 2021-02-10T11:39:13.185 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:13.165 7fc736ffd700 1 -- 172.21.2.102:0/490455949 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc720002fe0 con 0x7fc7381b0470 2021-02-10T11:39:13.185 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:13.166 7fc73cefc700 1 --2- 172.21.2.102:0/490455949 >> v2:172.21.2.113:3300/0 conn(0x7fc7380f05f0 0x7fc7380f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:13.185 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:13.166 7fc736ffd700 1 -- 172.21.2.102:0/490455949 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 531294915 0 0) 0x7fc728001180 con 0x7fc7381b0470 2021-02-10T11:39:13.185 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:13.166 7fc736ffd700 1 -- 172.21.2.102:0/490455949 >> v2:172.21.2.113:3300/0 conn(0x7fc7380f05f0 msgr2=0x7fc7380f20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:13.185 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:13.166 7fc736ffd700 1 --2- 172.21.2.102:0/490455949 >> v2:172.21.2.113:3300/0 conn(0x7fc7380f05f0 0x7fc7380f20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:13.186 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:13.166 7fc736ffd700 1 -- 172.21.2.102:0/490455949 >> v1:172.21.2.113:6789/0 conn(0x7fc7381b0870 legacy=0x7fc7380efe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:13.186 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:13.166 7fc736ffd700 1 -- 172.21.2.102:0/490455949 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7380e52e0 con 0x7fc7381b0470 2021-02-10T11:39:13.186 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:13.166 7fc736ffd700 1 -- 172.21.2.102:0/490455949 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7fc728001680 con 0x7fc7381b0470 2021-02-10T11:39:13.186 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:13.166 7fc736ffd700 1 -- 172.21.2.102:0/490455949 >> v1:172.21.2.102:6789/0 conn(0x7fc7381b0470 legacy=0x7fc7380ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:13.186 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:13.166 7fc736ffd700 1 --2- 172.21.2.102:0/490455949 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc720003a60 0x7fc720005ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:13.186 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:13.166 7fc736ffd700 1 --2- 172.21.2.102:0/490455949 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc720006790 0x7fc720008c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:13.186 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:13.166 7fc736ffd700 1 --2- 172.21.2.102:0/490455949 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc720009590 0x7fc72000bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:13.186 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:13.166 7fc736ffd700 1 -- 172.21.2.102:0/490455949 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc72000c300 con 0x7fc720009590 2021-02-10T11:39:13.186 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:13.166 7fc736ffd700 1 -- 172.21.2.102:0/490455949 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc72000c520 con 0x7fc720003a60 2021-02-10T11:39:13.186 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:13.166 7fc736ffd700 1 -- 172.21.2.102:0/490455949 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc72000c740 con 0x7fc720006790 2021-02-10T11:39:13.186 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:13.166 7fc737fff700 1 --2- 172.21.2.102:0/490455949 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc720003a60 0x7fc720005ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:13.186 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:13.166 7fc73cefc700 1 --2- 172.21.2.102:0/490455949 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc720009590 0x7fc72000bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:13.187 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:13.166 7fc7377fe700 1 --2- 172.21.2.102:0/490455949 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc720006790 0x7fc720008c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:13.187 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:13.166 7fc73cefc700 1 -- 172.21.2.102:0/490455949 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc720006790 msgr2=0x7fc720008c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:13.187 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:13.166 7fc73cefc700 1 --2- 172.21.2.102:0/490455949 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc720006790 0x7fc720008c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:13.187 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:13.166 7fc73cefc700 1 -- 172.21.2.102:0/490455949 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc720003a60 msgr2=0x7fc720005ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:13.187 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:13.166 7fc73cefc700 1 --2- 172.21.2.102:0/490455949 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc720003a60 0x7fc720005ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:13.187 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:13.166 7fc73cefc700 1 -- 172.21.2.102:0/490455949 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc72000c9e0 con 0x7fc720009590 2021-02-10T11:39:13.187 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:13.167 7fc73cefc700 1 -- 172.21.2.102:0/490455949 reap_dead start 2021-02-10T11:39:13.187 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:13.167 7fc73cefc700 1 --2- 172.21.2.102:0/490455949 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc720009590 0x7fc72000bab0 secure :-1 s=READY pgs=3186 cs=0 l=1 rev1=1 rx=0x7fc7300090f0 tx=0x7fc73000b050).ready entity=mon.0 client_cookie=539ca3a353055420 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:13.187 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:13.167 7fc736ffd700 1 -- 172.21.2.102:0/490455949 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc730006bd0 con 0x7fc720009590 2021-02-10T11:39:13.187 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:13.167 7fc736ffd700 1 -- 172.21.2.102:0/490455949 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc73000b440 con 0x7fc720009590 2021-02-10T11:39:13.187 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:13.167 7fc736ffd700 1 -- 172.21.2.102:0/490455949 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc73000b8f0 con 0x7fc720009590 2021-02-10T11:39:13.187 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:13.167 7fc73e8df700 1 -- 172.21.2.102:0/490455949 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc720009590 msgr2=0x7fc72000bab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:13.187 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:13.167 7fc73e8df700 1 --2- 172.21.2.102:0/490455949 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc720009590 0x7fc72000bab0 secure :-1 s=READY pgs=3186 cs=0 l=1 rev1=1 rx=0x7fc7300090f0 tx=0x7fc73000b050).stop 2021-02-10T11:39:13.188 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:13.167 7fc73e8df700 1 -- 172.21.2.102:0/490455949 shutdown_connections 2021-02-10T11:39:13.188 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:13.167 7fc73e8df700 1 --2- 172.21.2.102:0/490455949 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc720009590 0x7fc72000bab0 unknown :-1 s=CLOSED pgs=3186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:13.188 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:13.167 7fc73e8df700 1 -- 172.21.2.102:0/490455949 >> 172.21.2.102:0/490455949 conn(0x7fc7380ce3e0 msgr2=0x7fc7380e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:13.188 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:13.168 7fc73e8df700 1 -- 172.21.2.102:0/490455949 shutdown_connections 2021-02-10T11:39:13.188 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:13.168 7fc73e8df700 1 -- 172.21.2.102:0/490455949 wait complete. 2021-02-10T11:39:13.188 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:13.182 7fc73e8df700 1 Processor -- start 2021-02-10T11:39:13.188 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:13.182 7fc73e8df700 1 -- start start 2021-02-10T11:39:13.190 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:13.183 7fc73e8df700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc738108ce0 0x7fc73809da50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:13.190 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:13.183 7fc73e8df700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc7381090e0 0x7fc73809e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:13.190 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:13.183 7fc73e8df700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc73809eee0 0x7fc73809f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:13.190 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:13.183 7fc73e8df700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7381094e0 con 0x7fc738108ce0 2021-02-10T11:39:13.190 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:13.183 7fc73e8df700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc73809e960 con 0x7fc73809eee0 2021-02-10T11:39:13.190 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:13.183 7fc73e8df700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc73809e690 con 0x7fc7381090e0 2021-02-10T11:39:13.190 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:13.183 7fc737fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc738108ce0 0x7fc73809da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:13.190 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:13.183 7fc737fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc738108ce0 0x7fc73809da50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:40116/0 (socket says 172.21.2.102:40116) 2021-02-10T11:39:13.190 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:13.183 7fc737fff700 1 -- 172.21.2.102:0/3363319927 learned_addr learned my addr 172.21.2.102:0/3363319927 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:39:13.190 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:13.183 7fc73cefc700 1 --2- 172.21.2.102:0/3363319927 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc73809eee0 0x7fc73809f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:13.191 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:13.183 7fc7377fe700 1 --2- 172.21.2.102:0/3363319927 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc7381090e0 0x7fc73809e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:13.191 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:13.183 7fc737fff700 1 -- 172.21.2.102:0/3363319927 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc7381090e0 msgr2=0x7fc73809e070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:13.191 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:13.183 7fc737fff700 1 --2- 172.21.2.102:0/3363319927 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc7381090e0 0x7fc73809e070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:13.191 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:13.183 7fc737fff700 1 -- 172.21.2.102:0/3363319927 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc73809eee0 msgr2=0x7fc73809f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:13.191 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:13.183 7fc737fff700 1 --2- 172.21.2.102:0/3363319927 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc73809eee0 0x7fc73809f2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:13.191 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:13.183 7fc737fff700 1 -- 172.21.2.102:0/3363319927 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc73809f900 con 0x7fc738108ce0 2021-02-10T11:39:13.191 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:13.183 7fc7377fe700 1 --2- 172.21.2.102:0/3363319927 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc7381090e0 0x7fc73809e070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:39:13.191 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:13.183 7fc737fff700 1 --2- 172.21.2.102:0/3363319927 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc738108ce0 0x7fc73809da50 secure :-1 s=READY pgs=3187 cs=0 l=1 rev1=1 rx=0x7fc72804e5d0 tx=0x7fc72804e970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:13.191 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:13.184 7fc734ff9700 1 -- 172.21.2.102:0/3363319927 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc728001180 con 0x7fc738108ce0 2021-02-10T11:39:13.191 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:13.184 7fc734ff9700 1 -- 172.21.2.102:0/3363319927 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc72804ed60 con 0x7fc738108ce0 2021-02-10T11:39:13.191 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:13.184 7fc73e8df700 1 -- 172.21.2.102:0/3363319927 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc73809f900 con 0x7fc738108ce0 2021-02-10T11:39:13.191 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:13.184 7fc73e8df700 1 -- 172.21.2.102:0/3363319927 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc73809fe90 con 0x7fc738108ce0 2021-02-10T11:39:13.191 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:13.184 7fc734ff9700 1 -- 172.21.2.102:0/3363319927 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc728004620 con 0x7fc738108ce0 2021-02-10T11:39:13.192 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:13.184 7fc734ff9700 1 -- 172.21.2.102:0/3363319927 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fc728001180 con 0x7fc738108ce0 2021-02-10T11:39:13.192 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:13.184 7fc734ff9700 1 --2- 172.21.2.102:0/3363319927 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc7180186c0 0x7fc71801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:13.192 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:13.184 7fc734ff9700 1 -- 172.21.2.102:0/3363319927 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(180..180 src has 1..180) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fc728076750 con 0x7fc738108ce0 2021-02-10T11:39:13.192 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:13.184 7fc7377fe700 1 --2- 172.21.2.102:0/3363319927 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc7180186c0 0x7fc71801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:13.192 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:13.185 7fc7377fe700 1 --2- 172.21.2.102:0/3363319927 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc7180186c0 0x7fc71801abc0 crc :-1 s=READY pgs=1394 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:13.217 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:13.214 7fc73e8df700 1 -- 172.21.2.102:0/3363319927 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7380f0b50 con 0x7fc738108ce0 2021-02-10T11:39:13.219 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:13.217 7fc734ff9700 1 -- 172.21.2.102:0/3363319927 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fc7280060d0 con 0x7fc738108ce0 2021-02-10T11:39:13.353 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:13.351 7fc73e8df700 1 -- 172.21.2.102:0/3363319927 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestVolumes.test_recover_auth_metadata_during_deauthorize"]} v 0) v1 -- 0x7fc7380f0b50 con 0x7fc738108ce0 2021-02-10T11:39:14.197 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:14.195 7fc734ff9700 1 -- 172.21.2.102:0/3363319927 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestVolumes.test_recover_auth_metadata_during_deauthorize"]}]=0 v877) v1 ==== 162+0+0 (secure 0 0 0) 0x7fc7280634b0 con 0x7fc738108ce0 2021-02-10T11:39:14.223 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:14.220 7fc73e8df700 1 -- 172.21.2.102:0/3363319927 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc7180186c0 msgr2=0x7fc71801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:14.223 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:14.220 7fc73e8df700 1 --2- 172.21.2.102:0/3363319927 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc7180186c0 0x7fc71801abc0 crc :-1 s=READY pgs=1394 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:14.223 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:14.220 7fc73e8df700 1 -- 172.21.2.102:0/3363319927 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc738108ce0 msgr2=0x7fc73809da50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:14.223 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:14.220 7fc73e8df700 1 --2- 172.21.2.102:0/3363319927 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc738108ce0 0x7fc73809da50 secure :-1 s=READY pgs=3187 cs=0 l=1 rev1=1 rx=0x7fc72804e5d0 tx=0x7fc72804e970).stop 2021-02-10T11:39:14.223 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:14.221 7fc73e8df700 1 -- 172.21.2.102:0/3363319927 shutdown_connections 2021-02-10T11:39:14.223 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:14.221 7fc73e8df700 1 --2- 172.21.2.102:0/3363319927 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc7180186c0 0x7fc71801abc0 unknown :-1 s=CLOSED pgs=1394 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:14.223 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:14.221 7fc73e8df700 1 --2- 172.21.2.102:0/3363319927 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc73809eee0 0x7fc73809f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:14.223 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:14.221 7fc73e8df700 1 --2- 172.21.2.102:0/3363319927 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc7381090e0 0x7fc73809e070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:14.223 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:14.221 7fc73e8df700 1 --2- 172.21.2.102:0/3363319927 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc738108ce0 0x7fc73809da50 unknown :-1 s=CLOSED pgs=3187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:14.223 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:14.221 7fc73e8df700 1 -- 172.21.2.102:0/3363319927 >> 172.21.2.102:0/3363319927 conn(0x7fc7380bc6c0 msgr2=0x7fc7380b1f70 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:14.224 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:14.221 7fc73e8df700 1 -- 172.21.2.102:0/3363319927 shutdown_connections 2021-02-10T11:39:14.224 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:14.221 7fc73e8df700 1 -- 172.21.2.102:0/3363319927 wait complete. 2021-02-10T11:39:14.231 INFO:tasks.cephfs_test_runner:test_recover_auth_metadata_during_deauthorize (tasks.cephfs.test_volumes.TestVolumes) ... ok 2021-02-10T11:39:14.231 INFO:tasks.cephfs_test_runner:Starting test: test_subvolume_authorized_list (tasks.cephfs.test_volumes.TestVolumes) 2021-02-10T11:39:14.232 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestVolumes.test_subvolume_authorized_list' 2021-02-10T11:39:14.660 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:14.640 7f0668193700 1 Processor -- start 2021-02-10T11:39:14.660 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:14.640 7f0668193700 1 -- start start 2021-02-10T11:39:14.660 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:14.641 7f0668193700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f06600d0340 0x7f06600ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:14.660 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:14.641 7f0668193700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f06600d1c70 0x7f06600efc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:14.660 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:14.641 7f0668193700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06600f0b10 con 0x7f06600d1870 2021-02-10T11:39:14.660 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:14.641 7f0668193700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06600e5ed0 con 0x7f06600d0340 2021-02-10T11:39:14.660 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:14.641 7f0668193700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06601b0be0 con 0x7f06600d1c70 2021-02-10T11:39:14.660 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:14.641 7f06657ae700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f06600d0340 0x7f06600ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:14.660 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:14.641 7f06667b0700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f06600d1c70 0x7f06600efc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:14.661 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:14.642 7f06657ae700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f06600d0340 0x7f06600ef640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.113:3300/0 says I am v2:172.21.2.102:43554/0 (socket says 172.21.2.102:43554) 2021-02-10T11:39:14.661 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:14.642 7f06667b0700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f06600d1c70 0x7f06600efc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.113:3301/0 says I am v2:172.21.2.102:45664/0 (socket says 172.21.2.102:45664) 2021-02-10T11:39:14.661 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:14.642 7f06657ae700 1 -- 172.21.2.102:0/1665737310 learned_addr learned my addr 172.21.2.102:0/1665737310 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:39:14.661 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:14.642 7f06657ae700 1 -- 172.21.2.102:0/1665737310 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f06600d1c70 msgr2=0x7f06600efc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:14.661 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:14.642 7f06657ae700 1 --2- 172.21.2.102:0/1665737310 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f06600d1c70 0x7f06600efc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:14.661 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:14.642 7f06657ae700 1 -- 172.21.2.102:0/1665737310 >> v1:172.21.2.113:6789/0 conn(0x7f06600d1870 legacy=0x7f06601b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:14.661 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:14.642 7f06657ae700 1 -- 172.21.2.102:0/1665737310 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06600f20a0 con 0x7f06600d0340 2021-02-10T11:39:14.661 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:14.642 7f06667b0700 1 --2- 172.21.2.102:0/1665737310 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f06600d1c70 0x7f06600efc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:39:14.661 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:14.642 7f06657ae700 1 --2- 172.21.2.102:0/1665737310 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f06600d0340 0x7f06600ef640 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 rx=0x7f0654006700 tx=0x7f0654009b80).ready entity=mon.1 client_cookie=a04b7cc6f6ae567d server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:14.661 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:14.643 7f0664fad700 1 -- 172.21.2.102:0/1665737310 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f065400c0c0 con 0x7f06600d0340 2021-02-10T11:39:14.661 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:14.643 7f0664fad700 1 -- 172.21.2.102:0/1665737310 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f065400c340 con 0x7f06600d0340 2021-02-10T11:39:14.661 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:14.643 7f0664fad700 1 -- 172.21.2.102:0/1665737310 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f065400c860 con 0x7f06600d0340 2021-02-10T11:39:14.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:14.643 7f0668193700 1 -- 172.21.2.102:0/1665737310 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f06600d0340 msgr2=0x7f06600ef640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:14.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:14.643 7f0668193700 1 --2- 172.21.2.102:0/1665737310 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f06600d0340 0x7f06600ef640 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 rx=0x7f0654006700 tx=0x7f0654009b80).stop 2021-02-10T11:39:14.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:14.643 7f0668193700 1 -- 172.21.2.102:0/1665737310 shutdown_connections 2021-02-10T11:39:14.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:14.643 7f0668193700 1 --2- 172.21.2.102:0/1665737310 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f06600d1c70 0x7f06600efc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:14.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:14.643 7f0668193700 1 --2- 172.21.2.102:0/1665737310 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f06600d0340 0x7f06600ef640 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:14.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:14.643 7f0668193700 1 -- 172.21.2.102:0/1665737310 >> 172.21.2.102:0/1665737310 conn(0x7f06600ce450 msgr2=0x7f06600d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:14.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:14.644 7f0668193700 1 -- 172.21.2.102:0/1665737310 shutdown_connections 2021-02-10T11:39:14.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:14.644 7f0668193700 1 -- 172.21.2.102:0/1665737310 wait complete. 2021-02-10T11:39:14.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:14.658 7f0668193700 1 Processor -- start 2021-02-10T11:39:14.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:14.658 7f0668193700 1 -- start start 2021-02-10T11:39:14.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:14.658 7f0668193700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f066018d870 0x7f06600b19b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:14.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:14.658 7f0668193700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f06600b1fd0 0x7f0660099ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:14.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:14.658 7f0668193700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f06600b23d0 0x7f066009a510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:14.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:14.659 7f0668193700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f066018dc70 con 0x7f06600b1fd0 2021-02-10T11:39:14.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:14.659 7f0668193700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0660199e60 con 0x7f06600b23d0 2021-02-10T11:39:14.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:14.659 7f0668193700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f066009ab30 con 0x7f066018d870 2021-02-10T11:39:14.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:14.659 7f06657ae700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f06600b1fd0 0x7f0660099ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:14.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:14.659 7f06657ae700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f06600b1fd0 0x7f0660099ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:40130/0 (socket says 172.21.2.102:40130) 2021-02-10T11:39:14.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:14.659 7f06657ae700 1 -- 172.21.2.102:0/1032420182 learned_addr learned my addr 172.21.2.102:0/1032420182 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:39:14.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:14.659 7f0665faf700 1 --2- 172.21.2.102:0/1032420182 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f066018d870 0x7f06600b19b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:14.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:14.659 7f06667b0700 1 --2- 172.21.2.102:0/1032420182 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f06600b23d0 0x7f066009a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:14.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:14.659 7f06657ae700 1 -- 172.21.2.102:0/1032420182 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f066018d870 msgr2=0x7f06600b19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:14.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:14.659 7f06657ae700 1 --2- 172.21.2.102:0/1032420182 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f066018d870 0x7f06600b19b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:14.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:14.659 7f06657ae700 1 -- 172.21.2.102:0/1032420182 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f06600b23d0 msgr2=0x7f066009a510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:14.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:14.659 7f06657ae700 1 --2- 172.21.2.102:0/1032420182 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f06600b23d0 0x7f066009a510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:14.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:14.659 7f06657ae700 1 -- 172.21.2.102:0/1032420182 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f066009ad50 con 0x7f06600b1fd0 2021-02-10T11:39:14.666 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:14.659 7f06657ae700 1 --2- 172.21.2.102:0/1032420182 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f06600b1fd0 0x7f0660099ef0 secure :-1 s=READY pgs=3188 cs=0 l=1 rev1=1 rx=0x7f065400b5e0 tx=0x7f065400b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:14.666 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:14.659 7f06527fc700 1 -- 172.21.2.102:0/1032420182 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f065400bd70 con 0x7f06600b1fd0 2021-02-10T11:39:14.666 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:14.659 7f06527fc700 1 -- 172.21.2.102:0/1032420182 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0654002030 con 0x7f06600b1fd0 2021-02-10T11:39:14.666 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:14.659 7f0668193700 1 -- 172.21.2.102:0/1032420182 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f066009ad50 con 0x7f06600b1fd0 2021-02-10T11:39:14.666 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:14.660 7f06527fc700 1 -- 172.21.2.102:0/1032420182 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0654002550 con 0x7f06600b1fd0 2021-02-10T11:39:14.666 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:14.660 7f0668193700 1 -- 172.21.2.102:0/1032420182 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f066009ad50 con 0x7f06600b1fd0 2021-02-10T11:39:14.666 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:14.660 7f06527fc700 1 -- 172.21.2.102:0/1032420182 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f0654002210 con 0x7f06600b1fd0 2021-02-10T11:39:14.666 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:14.660 7f06527fc700 1 --2- 172.21.2.102:0/1032420182 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0648018730 0x7f064801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:14.666 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:14.660 7f06527fc700 1 -- 172.21.2.102:0/1032420182 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(180..180 src has 1..180) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f0654032620 con 0x7f06600b1fd0 2021-02-10T11:39:14.666 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:14.660 7f0665faf700 1 --2- 172.21.2.102:0/1032420182 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0648018730 0x7f064801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:14.666 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:14.661 7f0665faf700 1 --2- 172.21.2.102:0/1032420182 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0648018730 0x7f064801ac10 crc :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:14.693 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:14.690 7f0668193700 1 -- 172.21.2.102:0/1032420182 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06600999c0 con 0x7f06600b1fd0 2021-02-10T11:39:14.695 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:14.693 7f06527fc700 1 -- 172.21.2.102:0/1032420182 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f0654002240 con 0x7f06600b1fd0 2021-02-10T11:39:14.828 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:14.826 7f0668193700 1 -- 172.21.2.102:0/1032420182 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestVolumes.test_subvolume_authorized_list"]} v 0) v1 -- 0x7f06600999c0 con 0x7f06600b1fd0 2021-02-10T11:39:15.195 DEBUG:teuthology.orchestra.run.gibba002:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:39:15.198 DEBUG:teuthology.orchestra.run.gibba013:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:39:15.199 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:15.197 7f06527fc700 1 -- 172.21.2.102:0/1032420182 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestVolumes.test_subvolume_authorized_list"]}]=0 v878) v1 ==== 150+0+0 (secure 0 0 0) 0x7f065401f4e0 con 0x7f06600b1fd0 2021-02-10T11:39:15.247 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:15.245 7f0668193700 1 -- 172.21.2.102:0/1032420182 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0648018730 msgr2=0x7f064801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:15.247 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:15.245 7f0668193700 1 --2- 172.21.2.102:0/1032420182 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0648018730 0x7f064801ac10 crc :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:15.248 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:15.245 7f0668193700 1 -- 172.21.2.102:0/1032420182 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f06600b1fd0 msgr2=0x7f0660099ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:15.248 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:15.245 7f0668193700 1 --2- 172.21.2.102:0/1032420182 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f06600b1fd0 0x7f0660099ef0 secure :-1 s=READY pgs=3188 cs=0 l=1 rev1=1 rx=0x7f065400b5e0 tx=0x7f065400b980).stop 2021-02-10T11:39:15.248 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:15.245 7f0668193700 1 -- 172.21.2.102:0/1032420182 shutdown_connections 2021-02-10T11:39:15.248 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:15.245 7f0668193700 1 --2- 172.21.2.102:0/1032420182 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0648018730 0x7f064801ac10 unknown :-1 s=CLOSED pgs=1395 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:15.248 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:15.245 7f0668193700 1 --2- 172.21.2.102:0/1032420182 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f06600b23d0 0x7f066009a510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:15.248 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:15.245 7f0668193700 1 --2- 172.21.2.102:0/1032420182 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f06600b1fd0 0x7f0660099ef0 unknown :-1 s=CLOSED pgs=3188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:15.248 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:15.245 7f0668193700 1 --2- 172.21.2.102:0/1032420182 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f066018d870 0x7f06600b19b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:15.248 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:15.245 7f0668193700 1 -- 172.21.2.102:0/1032420182 >> 172.21.2.102:0/1032420182 conn(0x7f06600f0550 msgr2=0x7f0660191730 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:15.248 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:15.245 7f0668193700 1 -- 172.21.2.102:0/1032420182 shutdown_connections 2021-02-10T11:39:15.248 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:15.245 7f0668193700 1 -- 172.21.2.102:0/1032420182 wait complete. 2021-02-10T11:39:15.256 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config set mon mon_allow_pool_delete True 2021-02-10T11:39:15.633 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:15.612 7f06ffd64700 1 Processor -- start 2021-02-10T11:39:15.633 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:15.612 7f06ffd64700 1 -- start start 2021-02-10T11:39:15.633 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:15.613 7f06ffd64700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f06f80d1860 0x7f06f81b04b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:15.633 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:15.613 7f06ffd64700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f06f80cff60 0x7f06f8109120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:15.633 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:15.613 7f06ffd64700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06f8109a50 con 0x7f06f81b0ad0 2021-02-10T11:39:15.634 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:15.613 7f06ffd64700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06f8109740 con 0x7f06f80cff60 2021-02-10T11:39:15.634 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:15.613 7f06ffd64700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06f8088070 con 0x7f06f80d1860 2021-02-10T11:39:15.634 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:15.613 7f06fd37f700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f06f81b0ad0 0x7f06f8108930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:60594/0 (socket says 172.21.2.102:60594) 2021-02-10T11:39:15.634 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:15.613 7f06fd37f700 1 -- 172.21.2.102:0/3188195619 learned_addr learned my addr 172.21.2.102:0/3188195619 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:39:15.634 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:15.614 7f06fcb7e700 1 -- 172.21.2.102:0/3188195619 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2484373803 0 0) 0x7f06ec0009b0 con 0x7f06f81b0ad0 2021-02-10T11:39:15.634 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:15.614 7f06fcb7e700 1 -- 172.21.2.102:0/3188195619 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f06e4002fe0 con 0x7f06f81b0ad0 2021-02-10T11:39:15.634 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:15.614 7f06fe381700 1 --2- 172.21.2.102:0/3188195619 >> v2:172.21.2.113:3300/0 conn(0x7f06f80cff60 0x7f06f8109120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:15.634 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:15.614 7f06fdb80700 1 --2- 172.21.2.102:0/3188195619 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f06f80d1860 0x7f06f81b04b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:15.634 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:15.614 7f06fcb7e700 1 -- 172.21.2.102:0/3188195619 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4052690912 0 0) 0x7f06ec001010 con 0x7f06f81b0ad0 2021-02-10T11:39:15.634 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:15.614 7f06fcb7e700 1 -- 172.21.2.102:0/3188195619 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f06f80d1860 msgr2=0x7f06f81b04b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:15.634 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:15.614 7f06fcb7e700 1 --2- 172.21.2.102:0/3188195619 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f06f80d1860 0x7f06f81b04b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:15.634 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:15.614 7f06fcb7e700 1 -- 172.21.2.102:0/3188195619 >> v2:172.21.2.113:3300/0 conn(0x7f06f80cff60 msgr2=0x7f06f8109120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:15.635 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:15.614 7f06fcb7e700 1 --2- 172.21.2.102:0/3188195619 >> v2:172.21.2.113:3300/0 conn(0x7f06f80cff60 0x7f06f8109120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:15.635 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:15.614 7f06fcb7e700 1 -- 172.21.2.102:0/3188195619 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06f8088260 con 0x7f06f81b0ad0 2021-02-10T11:39:15.635 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:15.614 7f06fcb7e700 1 -- 172.21.2.102:0/3188195619 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7f06ec001590 con 0x7f06f81b0ad0 2021-02-10T11:39:15.635 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:15.614 7f06fcb7e700 1 -- 172.21.2.102:0/3188195619 >> v1:172.21.2.102:6789/0 conn(0x7f06f81b0ad0 legacy=0x7f06f8108930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:15.635 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:15.614 7f06fcb7e700 1 --2- 172.21.2.102:0/3188195619 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f06e4003ac0 0x7f06e4005ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:15.635 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:15.614 7f06fcb7e700 1 --2- 172.21.2.102:0/3188195619 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f06e404a7a0 0x7f06e404cca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:15.635 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:15.614 7f06fcb7e700 1 --2- 172.21.2.102:0/3188195619 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f06e404d5a0 0x7f06e404fa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:15.635 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:15.614 7f06fcb7e700 1 -- 172.21.2.102:0/3188195619 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06e40502e0 con 0x7f06e404a7a0 2021-02-10T11:39:15.635 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:15.614 7f06fcb7e700 1 -- 172.21.2.102:0/3188195619 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06e4050500 con 0x7f06e4003ac0 2021-02-10T11:39:15.635 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:15.614 7f06fcb7e700 1 -- 172.21.2.102:0/3188195619 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06e4050720 con 0x7f06e404d5a0 2021-02-10T11:39:15.635 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:15.614 7f06fd37f700 1 --2- 172.21.2.102:0/3188195619 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f06e404a7a0 0x7f06e404cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:15.635 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:15.614 7f06fdb80700 1 --2- 172.21.2.102:0/3188195619 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f06e4003ac0 0x7f06e4005ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:15.635 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:15.614 7f06fe381700 1 --2- 172.21.2.102:0/3188195619 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f06e404d5a0 0x7f06e404fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:15.636 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:15.614 7f06fd37f700 1 -- 172.21.2.102:0/3188195619 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f06e404d5a0 msgr2=0x7f06e404fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:15.636 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:15.614 7f06fd37f700 1 --2- 172.21.2.102:0/3188195619 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f06e404d5a0 0x7f06e404fa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:15.636 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:15.614 7f06fd37f700 1 -- 172.21.2.102:0/3188195619 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f06e4003ac0 msgr2=0x7f06e4005ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:15.636 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:15.614 7f06fd37f700 1 --2- 172.21.2.102:0/3188195619 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f06e4003ac0 0x7f06e4005ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:15.636 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:15.614 7f06fd37f700 1 -- 172.21.2.102:0/3188195619 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06e40509c0 con 0x7f06e404a7a0 2021-02-10T11:39:15.636 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:15.615 7f06fd37f700 1 --2- 172.21.2.102:0/3188195619 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f06e404a7a0 0x7f06e404cca0 secure :-1 s=READY pgs=3190 cs=0 l=1 rev1=1 rx=0x7f06ec0017d0 tx=0x7f06ec007d30).ready entity=mon.0 client_cookie=b5daf1219ab09fe9 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:15.636 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:15.615 7f06fcb7e700 1 -- 172.21.2.102:0/3188195619 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f06ec00a310 con 0x7f06e404a7a0 2021-02-10T11:39:15.636 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:15.615 7f06fcb7e700 1 -- 172.21.2.102:0/3188195619 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06ec0013a0 con 0x7f06e404a7a0 2021-02-10T11:39:15.636 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:15.615 7f06fcb7e700 1 -- 172.21.2.102:0/3188195619 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f06ec00a840 con 0x7f06e404a7a0 2021-02-10T11:39:15.636 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:15.615 7f06ffd64700 1 -- 172.21.2.102:0/3188195619 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f06e404a7a0 msgr2=0x7f06e404cca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:15.636 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:15.615 7f06ffd64700 1 --2- 172.21.2.102:0/3188195619 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f06e404a7a0 0x7f06e404cca0 secure :-1 s=READY pgs=3190 cs=0 l=1 rev1=1 rx=0x7f06ec0017d0 tx=0x7f06ec007d30).stop 2021-02-10T11:39:15.636 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:15.615 7f06fe381700 1 -- 172.21.2.102:0/3188195619 reap_dead start 2021-02-10T11:39:15.637 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:15.616 7f06ffd64700 1 -- 172.21.2.102:0/3188195619 shutdown_connections 2021-02-10T11:39:15.637 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:15.616 7f06ffd64700 1 -- 172.21.2.102:0/3188195619 >> 172.21.2.102:0/3188195619 conn(0x7f06f80ce410 msgr2=0x7f06f80d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:15.637 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:15.616 7f06ffd64700 1 -- 172.21.2.102:0/3188195619 shutdown_connections 2021-02-10T11:39:15.637 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:15.616 7f06ffd64700 1 -- 172.21.2.102:0/3188195619 wait complete. 2021-02-10T11:39:15.637 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:15.631 7f06ffd64700 1 Processor -- start 2021-02-10T11:39:15.637 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:15.631 7f06ffd64700 1 -- start start 2021-02-10T11:39:15.638 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:15.631 7f06ffd64700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f06f8094d20 0x7f06f80a3e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:15.639 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:15.631 7f06ffd64700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f06f80a4460 0x7f06f80a4860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:15.639 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:15.631 7f06ffd64700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f06f80bf9e0 0x7f06f80bfde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:15.639 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:15.631 7f06ffd64700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06f80a7560 con 0x7f06f80bf9e0 2021-02-10T11:39:15.639 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:15.631 7f06ffd64700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06f80b0bc0 con 0x7f06f8094d20 2021-02-10T11:39:15.639 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:15.631 7f06ffd64700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06f8095120 con 0x7f06f80a4460 2021-02-10T11:39:15.639 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:15.631 7f06fe381700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f06f80bf9e0 0x7f06f80bfde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:15.639 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:15.631 7f06fe381700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f06f80bf9e0 0x7f06f80bfde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:40148/0 (socket says 172.21.2.102:40148) 2021-02-10T11:39:15.639 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:15.631 7f06fe381700 1 -- 172.21.2.102:0/4016751828 learned_addr learned my addr 172.21.2.102:0/4016751828 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:39:15.639 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:15.631 7f06fdb80700 1 --2- 172.21.2.102:0/4016751828 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f06f8094d20 0x7f06f80a3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:15.639 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:15.632 7f06fd37f700 1 --2- 172.21.2.102:0/4016751828 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f06f80a4460 0x7f06f80a4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:15.639 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:15.632 7f06fe381700 1 -- 172.21.2.102:0/4016751828 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f06f80a4460 msgr2=0x7f06f80a4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:15.639 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:15.632 7f06fe381700 1 --2- 172.21.2.102:0/4016751828 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f06f80a4460 0x7f06f80a4860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:15.640 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:15.632 7f06fe381700 1 -- 172.21.2.102:0/4016751828 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f06f8094d20 msgr2=0x7f06f80a3e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:15.640 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:15.632 7f06fe381700 1 --2- 172.21.2.102:0/4016751828 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f06f8094d20 0x7f06f80a3e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:15.640 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:15.632 7f06fe381700 1 -- 172.21.2.102:0/4016751828 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06f80c0400 con 0x7f06f80bf9e0 2021-02-10T11:39:15.640 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:15.632 7f06fe381700 1 --2- 172.21.2.102:0/4016751828 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f06f80bf9e0 0x7f06f80bfde0 secure :-1 s=READY pgs=3191 cs=0 l=1 rev1=1 rx=0x7f06f0006fa0 tx=0x7f06f000bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:15.640 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:15.632 7f06ea7fc700 1 -- 172.21.2.102:0/4016751828 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f06f0000f60 con 0x7f06f80bf9e0 2021-02-10T11:39:15.640 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:15.632 7f06ea7fc700 1 -- 172.21.2.102:0/4016751828 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06f000c2f0 con 0x7f06f80bf9e0 2021-02-10T11:39:15.640 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:15.632 7f06ffd64700 1 -- 172.21.2.102:0/4016751828 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06f80c0400 con 0x7f06f80bf9e0 2021-02-10T11:39:15.640 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:15.632 7f06ea7fc700 1 -- 172.21.2.102:0/4016751828 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f06f000c750 con 0x7f06f80bf9e0 2021-02-10T11:39:15.640 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:15.632 7f06ffd64700 1 -- 172.21.2.102:0/4016751828 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06f80c0400 con 0x7f06f80bf9e0 2021-02-10T11:39:15.640 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:15.633 7f06ea7fc700 1 -- 172.21.2.102:0/4016751828 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f06f00011f0 con 0x7f06f80bf9e0 2021-02-10T11:39:15.640 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:15.633 7f06ea7fc700 1 --2- 172.21.2.102:0/4016751828 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f06e0018700 0x7f06e001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:15.640 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:15.633 7f06ea7fc700 1 -- 172.21.2.102:0/4016751828 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(180..180 src has 1..180) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f06f00329a0 con 0x7f06f80bf9e0 2021-02-10T11:39:15.640 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:15.633 7f06fdb80700 1 --2- 172.21.2.102:0/4016751828 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f06e0018700 0x7f06e001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:15.641 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:15.633 7f06fdb80700 1 --2- 172.21.2.102:0/4016751828 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f06e0018700 0x7f06e001abc0 crc :-1 s=READY pgs=1396 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:15.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:15.663 7f06ffd64700 1 -- 172.21.2.102:0/4016751828 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06f8082920 con 0x7f06f80bf9e0 2021-02-10T11:39:15.667 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:15.665 7f06ea7fc700 1 -- 172.21.2.102:0/4016751828 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f06f000be30 con 0x7f06f80bf9e0 2021-02-10T11:39:15.819 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:15.816 7f06ffd64700 1 -- 172.21.2.102:0/4016751828 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f06f8109a50 con 0x7f06f80bf9e0 2021-02-10T11:39:15.823 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:15.821 7f06ea7fc700 1 -- 172.21.2.102:0/4016751828 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v46)=0 v46) v1 ==== 122+0+0 (secure 0 0 0) 0x7f06f00011f0 con 0x7f06f80bf9e0 2021-02-10T11:39:15.828 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:15.826 7f06ffd64700 1 -- 172.21.2.102:0/4016751828 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f06e0018700 msgr2=0x7f06e001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:15.828 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:15.826 7f06ffd64700 1 --2- 172.21.2.102:0/4016751828 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f06e0018700 0x7f06e001abc0 crc :-1 s=READY pgs=1396 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:15.828 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:15.826 7f06ffd64700 1 -- 172.21.2.102:0/4016751828 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f06f80bf9e0 msgr2=0x7f06f80bfde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:15.828 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:15.826 7f06ffd64700 1 --2- 172.21.2.102:0/4016751828 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f06f80bf9e0 0x7f06f80bfde0 secure :-1 s=READY pgs=3191 cs=0 l=1 rev1=1 rx=0x7f06f0006fa0 tx=0x7f06f000bcf0).stop 2021-02-10T11:39:15.829 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:15.826 7f06ffd64700 1 -- 172.21.2.102:0/4016751828 shutdown_connections 2021-02-10T11:39:15.829 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:15.826 7f06ffd64700 1 --2- 172.21.2.102:0/4016751828 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f06f80bf9e0 0x7f06f80bfde0 unknown :-1 s=CLOSED pgs=3191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:15.829 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:15.826 7f06ffd64700 1 --2- 172.21.2.102:0/4016751828 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f06f80a4460 0x7f06f80a4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:15.829 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:15.826 7f06ffd64700 1 --2- 172.21.2.102:0/4016751828 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f06e0018700 0x7f06e001abc0 unknown :-1 s=CLOSED pgs=1396 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:15.829 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:15.826 7f06ffd64700 1 --2- 172.21.2.102:0/4016751828 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f06f8094d20 0x7f06f80a3e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:15.829 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:15.826 7f06ffd64700 1 -- 172.21.2.102:0/4016751828 >> 172.21.2.102:0/4016751828 conn(0x7f06f80c3e40 msgr2=0x7f06f80916a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:15.829 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:15.826 7f06ffd64700 1 -- 172.21.2.102:0/4016751828 shutdown_connections 2021-02-10T11:39:15.829 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:15.826 7f06ffd64700 1 -- 172.21.2.102:0/4016751828 wait complete. 2021-02-10T11:39:15.837 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2021-02-10T11:39:15.837 DEBUG:teuthology.orchestra.run.gibba002:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:39:16.184 DEBUG:teuthology.orchestra.run.gibba013:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:39:16.531 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2021-02-10T11:39:16.531 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:39:16.531 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:39:16.531 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:39:16.866 INFO:teuthology.orchestra.run.gibba002.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.0’: No such file or directory 2021-02-10T11:39:16.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:39:16.867 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.0 2021-02-10T11:39:16.867 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:39:16.867 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:39:16.867 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:39:17.223 INFO:teuthology.orchestra.run.gibba002.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.1’: No such file or directory 2021-02-10T11:39:17.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:39:17.224 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2021-02-10T11:39:17.224 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:39:17.224 DEBUG:teuthology.orchestra.run.gibba013:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:39:17.224 DEBUG:teuthology.orchestra.run.gibba013:> ' -- /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:39:17.559 INFO:teuthology.orchestra.run.gibba013.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.2’: No such file or directory 2021-02-10T11:39:17.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:39:17.559 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2021-02-10T11:39:17.560 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:39:17.560 DEBUG:teuthology.orchestra.run.gibba013:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:39:17.560 DEBUG:teuthology.orchestra.run.gibba013:> ' -- /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:39:17.894 INFO:teuthology.orchestra.run.gibba013.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.3’: No such file or directory 2021-02-10T11:39:17.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:39:17.894 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.3 2021-02-10T11:39:17.894 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json-pretty 2021-02-10T11:39:18.324 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:18.305 7f2e0cd0c700 1 Processor -- start 2021-02-10T11:39:18.324 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:18.305 7f2e0cd0c700 1 -- start start 2021-02-10T11:39:18.324 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:18.306 7f2e0cd0c700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f2e080d1870 0x7f2e081b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:18.324 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:18.306 7f2e0cd0c700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2e080f0650 0x7f2e080cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:18.324 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:18.306 7f2e0cd0c700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e080f0b10 con 0x7f2e081b0a10 2021-02-10T11:39:18.325 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:18.306 7f2e0cd0c700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e080e5ed0 con 0x7f2e080d1870 2021-02-10T11:39:18.325 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:18.306 7f2e0cd0c700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e080d1c70 con 0x7f2e080f0650 2021-02-10T11:39:18.325 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:18.306 7f2e06b13700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f2e080d1870 0x7f2e081b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:18.325 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:18.306 7f2e06b13700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f2e080d1870 0x7f2e081b03f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:40156/0 (socket says 172.21.2.102:40156) 2021-02-10T11:39:18.325 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:18.306 7f2e06b13700 1 -- 172.21.2.102:0/4205518972 learned_addr learned my addr 172.21.2.102:0/4205518972 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:39:18.325 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:18.306 7f2e06b13700 1 -- 172.21.2.102:0/4205518972 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2e080f0650 msgr2=0x7f2e080cfb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:18.325 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:18.306 7f2e06b13700 1 --2- 172.21.2.102:0/4205518972 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2e080f0650 0x7f2e080cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:18.325 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:18.306 7f2e06b13700 1 -- 172.21.2.102:0/4205518972 >> v1:172.21.2.102:6789/0 conn(0x7f2e081b0a10 legacy=0x7f2e080cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:18.325 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:18.306 7f2e06b13700 1 -- 172.21.2.102:0/4205518972 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e080d1e90 con 0x7f2e080d1870 2021-02-10T11:39:18.325 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:18.307 7f2e06b13700 1 --2- 172.21.2.102:0/4205518972 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2e080d1870 0x7f2e081b03f0 secure :-1 s=READY pgs=3193 cs=0 l=1 rev1=1 rx=0x7f2df0006700 tx=0x7f2df0009b00).ready entity=mon.0 client_cookie=16d8a0faf4ff52a4 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:18.325 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:18.307 7f2e05b11700 1 -- 172.21.2.102:0/4205518972 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2df000c0c0 con 0x7f2e080d1870 2021-02-10T11:39:18.325 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:18.307 7f2e05b11700 1 -- 172.21.2.102:0/4205518972 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2df000c340 con 0x7f2e080d1870 2021-02-10T11:39:18.325 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:18.307 7f2e05b11700 1 -- 172.21.2.102:0/4205518972 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2df000c820 con 0x7f2e080d1870 2021-02-10T11:39:18.326 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:18.307 7f2e0cd0c700 1 -- 172.21.2.102:0/4205518972 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2e080d1870 msgr2=0x7f2e081b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:18.326 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:18.307 7f2e0cd0c700 1 --2- 172.21.2.102:0/4205518972 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2e080d1870 0x7f2e081b03f0 secure :-1 s=READY pgs=3193 cs=0 l=1 rev1=1 rx=0x7f2df0006700 tx=0x7f2df0009b00).stop 2021-02-10T11:39:18.326 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:18.307 7f2e0cd0c700 1 -- 172.21.2.102:0/4205518972 shutdown_connections 2021-02-10T11:39:18.326 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:18.307 7f2e0cd0c700 1 --2- 172.21.2.102:0/4205518972 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2e080f0650 0x7f2e080cfb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:18.326 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:18.307 7f2e0cd0c700 1 --2- 172.21.2.102:0/4205518972 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2e080d1870 0x7f2e081b03f0 unknown :-1 s=CLOSED pgs=3193 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:18.326 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:18.307 7f2e0cd0c700 1 -- 172.21.2.102:0/4205518972 >> 172.21.2.102:0/4205518972 conn(0x7f2e080ce450 msgr2=0x7f2e080d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:18.326 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:18.308 7f2e0cd0c700 1 -- 172.21.2.102:0/4205518972 shutdown_connections 2021-02-10T11:39:18.326 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:18.308 7f2e0cd0c700 1 -- 172.21.2.102:0/4205518972 wait complete. 2021-02-10T11:39:18.328 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:18.322 7f2e0cd0c700 1 Processor -- start 2021-02-10T11:39:18.328 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:18.322 7f2e0cd0c700 1 -- start start 2021-02-10T11:39:18.328 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:18.323 7f2e0cd0c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2e080b4f30 0x7f2e0818b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:18.328 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:18.323 7f2e0cd0c700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2e08093f40 0x7f2e08094340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:18.328 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:18.323 7f2e0cd0c700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2e08094960 0x7f2e08187e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:18.328 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:18.323 7f2e0cd0c700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e08094e90 con 0x7f2e080b4f30 2021-02-10T11:39:18.328 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:18.323 7f2e0cd0c700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e0818bff0 con 0x7f2e08093f40 2021-02-10T11:39:18.328 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:18.323 7f2e0cd0c700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e0808f780 con 0x7f2e08094960 2021-02-10T11:39:18.328 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:18.323 7f2e06b13700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2e080b4f30 0x7f2e0818b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:18.329 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:18.323 7f2e06b13700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2e080b4f30 0x7f2e0818b9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:40166/0 (socket says 172.21.2.102:40166) 2021-02-10T11:39:18.329 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:18.323 7f2e06b13700 1 -- 172.21.2.102:0/1124693840 learned_addr learned my addr 172.21.2.102:0/1124693840 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:39:18.329 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:18.323 7f2e06b13700 1 -- 172.21.2.102:0/1124693840 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2e08094960 msgr2=0x7f2e08187e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:18.329 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:18.323 7f2e07314700 1 --2- 172.21.2.102:0/1124693840 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2e08094960 0x7f2e08187e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:18.329 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:18.323 7f2e06b13700 1 --2- 172.21.2.102:0/1124693840 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2e08094960 0x7f2e08187e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:18.329 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:18.323 7f2e06312700 1 --2- 172.21.2.102:0/1124693840 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2e08093f40 0x7f2e08094340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:18.329 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:18.323 7f2e06b13700 1 -- 172.21.2.102:0/1124693840 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2e08093f40 msgr2=0x7f2e08094340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:18.329 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:18.323 7f2e06b13700 1 --2- 172.21.2.102:0/1124693840 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2e08093f40 0x7f2e08094340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:18.329 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:18.324 7f2e06b13700 1 -- 172.21.2.102:0/1124693840 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e08188470 con 0x7f2e080b4f30 2021-02-10T11:39:18.329 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:18.324 7f2e06b13700 1 --2- 172.21.2.102:0/1124693840 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2e080b4f30 0x7f2e0818b9d0 secure :-1 s=READY pgs=3194 cs=0 l=1 rev1=1 rx=0x7f2df0006f20 tx=0x7f2df000b910).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:18.329 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:18.324 7f2dff7fe700 1 -- 172.21.2.102:0/1124693840 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2df000bd00 con 0x7f2e080b4f30 2021-02-10T11:39:18.329 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:18.324 7f2dff7fe700 1 -- 172.21.2.102:0/1124693840 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2df0000c90 con 0x7f2e080b4f30 2021-02-10T11:39:18.330 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:18.324 7f2dff7fe700 1 -- 172.21.2.102:0/1124693840 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2df00022b0 con 0x7f2e080b4f30 2021-02-10T11:39:18.330 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:18.324 7f2e0cd0c700 1 -- 172.21.2.102:0/1124693840 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e08188470 con 0x7f2e080b4f30 2021-02-10T11:39:18.330 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:18.324 7f2e0cd0c700 1 -- 172.21.2.102:0/1124693840 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e08188470 con 0x7f2e080b4f30 2021-02-10T11:39:18.330 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:18.325 7f2dff7fe700 1 -- 172.21.2.102:0/1124693840 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f2df0002020 con 0x7f2e080b4f30 2021-02-10T11:39:18.330 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:18.325 7f2dff7fe700 1 --2- 172.21.2.102:0/1124693840 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2dec0186c0 0x7f2dec01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:18.330 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:18.325 7f2dff7fe700 1 -- 172.21.2.102:0/1124693840 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(180..180 src has 1..180) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f2df00327a0 con 0x7f2e080b4f30 2021-02-10T11:39:18.330 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:18.325 7f2e06312700 1 --2- 172.21.2.102:0/1124693840 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2dec0186c0 0x7f2dec01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:18.330 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:18.325 7f2e06312700 1 --2- 172.21.2.102:0/1124693840 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2dec0186c0 0x7f2dec01abc0 crc :-1 s=READY pgs=1397 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:18.358 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:18.356 7f2e0cd0c700 1 -- 172.21.2.102:0/1124693840 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e080ce450 con 0x7f2e080b4f30 2021-02-10T11:39:18.360 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:18.358 7f2dff7fe700 1 -- 172.21.2.102:0/1124693840 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f2df0006390 con 0x7f2e080b4f30 2021-02-10T11:39:18.485 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:18.483 7f2e0cd0c700 1 -- 172.21.2.102:0/1124693840 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7f2e080ce450 con 0x7f2e080b4f30 2021-02-10T11:39:18.486 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:18.484 7f2dff7fe700 1 -- 172.21.2.102:0/1124693840 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v180) v1 ==== 81+0+30374 (secure 0 0 0) 0x7f2df001f4e0 con 0x7f2e080b4f30 2021-02-10T11:39:18.487 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:39:18.487 INFO:teuthology.orchestra.run.gibba002.stdout:{ 2021-02-10T11:39:18.487 INFO:teuthology.orchestra.run.gibba002.stdout: "epoch": 180, 2021-02-10T11:39:18.487 INFO:teuthology.orchestra.run.gibba002.stdout: "fsid": "68db30ea-dbce-4aef-a6df-f01c6fea0513", 2021-02-10T11:39:18.487 INFO:teuthology.orchestra.run.gibba002.stdout: "created": "2021-02-10 11:03:06.331557", 2021-02-10T11:39:18.487 INFO:teuthology.orchestra.run.gibba002.stdout: "modified": "2021-02-10 11:38:31.337750", 2021-02-10T11:39:18.487 INFO:teuthology.orchestra.run.gibba002.stdout: "last_up_change": "2021-02-10 11:03:32.690562", 2021-02-10T11:39:18.487 INFO:teuthology.orchestra.run.gibba002.stdout: "last_in_change": "2021-02-10 11:03:32.690562", 2021-02-10T11:39:18.487 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2021-02-10T11:39:18.487 INFO:teuthology.orchestra.run.gibba002.stdout: "flags_num": 5799936, 2021-02-10T11:39:18.487 INFO:teuthology.orchestra.run.gibba002.stdout: "flags_set": [ 2021-02-10T11:39:18.487 INFO:teuthology.orchestra.run.gibba002.stdout: "pglog_hardlimit", 2021-02-10T11:39:18.488 INFO:teuthology.orchestra.run.gibba002.stdout: "purged_snapdirs", 2021-02-10T11:39:18.488 INFO:teuthology.orchestra.run.gibba002.stdout: "recovery_deletes", 2021-02-10T11:39:18.488 INFO:teuthology.orchestra.run.gibba002.stdout: "sortbitwise" 2021-02-10T11:39:18.488 INFO:teuthology.orchestra.run.gibba002.stdout: ], 2021-02-10T11:39:18.488 INFO:teuthology.orchestra.run.gibba002.stdout: "crush_version": 4, 2021-02-10T11:39:18.488 INFO:teuthology.orchestra.run.gibba002.stdout: "full_ratio": 0.94999998807907104, 2021-02-10T11:39:18.488 INFO:teuthology.orchestra.run.gibba002.stdout: "backfillfull_ratio": 0.89999997615814209, 2021-02-10T11:39:18.488 INFO:teuthology.orchestra.run.gibba002.stdout: "nearfull_ratio": 0.85000002384185791, 2021-02-10T11:39:18.488 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_snapshot": "", 2021-02-10T11:39:18.488 INFO:teuthology.orchestra.run.gibba002.stdout: "pool_max": 39, 2021-02-10T11:39:18.488 INFO:teuthology.orchestra.run.gibba002.stdout: "max_osd": 8, 2021-02-10T11:39:18.488 INFO:teuthology.orchestra.run.gibba002.stdout: "require_min_compat_client": "jewel", 2021-02-10T11:39:18.488 INFO:teuthology.orchestra.run.gibba002.stdout: "min_compat_client": "jewel", 2021-02-10T11:39:18.489 INFO:teuthology.orchestra.run.gibba002.stdout: "require_osd_release": "nautilus", 2021-02-10T11:39:18.489 INFO:teuthology.orchestra.run.gibba002.stdout: "pools": [ 2021-02-10T11:39:18.489 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:39:18.489 INFO:teuthology.orchestra.run.gibba002.stdout: "pool": 1, 2021-02-10T11:39:18.489 INFO:teuthology.orchestra.run.gibba002.stdout: "pool_name": "rbd", 2021-02-10T11:39:18.489 INFO:teuthology.orchestra.run.gibba002.stdout: "create_time": "2021-02-10 11:03:37.322519", 2021-02-10T11:39:18.489 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 1, 2021-02-10T11:39:18.489 INFO:teuthology.orchestra.run.gibba002.stdout: "flags_names": "hashpspool", 2021-02-10T11:39:18.489 INFO:teuthology.orchestra.run.gibba002.stdout: "type": 1, 2021-02-10T11:39:18.489 INFO:teuthology.orchestra.run.gibba002.stdout: "size": 2, 2021-02-10T11:39:18.489 INFO:teuthology.orchestra.run.gibba002.stdout: "min_size": 1, 2021-02-10T11:39:18.489 INFO:teuthology.orchestra.run.gibba002.stdout: "crush_rule": 0, 2021-02-10T11:39:18.489 INFO:teuthology.orchestra.run.gibba002.stdout: "object_hash": 2, 2021-02-10T11:39:18.489 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:39:18.490 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num": 8, 2021-02-10T11:39:18.490 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_placement_num": 8, 2021-02-10T11:39:18.490 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_placement_num_target": 8, 2021-02-10T11:39:18.490 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num_target": 8, 2021-02-10T11:39:18.490 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num_pending": 8, 2021-02-10T11:39:18.490 INFO:teuthology.orchestra.run.gibba002.stdout: "last_pg_merge_meta": { 2021-02-10T11:39:18.490 INFO:teuthology.orchestra.run.gibba002.stdout: "source_pgid": "0.0", 2021-02-10T11:39:18.490 INFO:teuthology.orchestra.run.gibba002.stdout: "ready_epoch": 0, 2021-02-10T11:39:18.490 INFO:teuthology.orchestra.run.gibba002.stdout: "last_epoch_started": 0, 2021-02-10T11:39:18.490 INFO:teuthology.orchestra.run.gibba002.stdout: "last_epoch_clean": 0, 2021-02-10T11:39:18.490 INFO:teuthology.orchestra.run.gibba002.stdout: "source_version": "0'0", 2021-02-10T11:39:18.490 INFO:teuthology.orchestra.run.gibba002.stdout: "target_version": "0'0" 2021-02-10T11:39:18.490 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:39:18.491 INFO:teuthology.orchestra.run.gibba002.stdout: "last_change": "16", 2021-02-10T11:39:18.491 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend": "0", 2021-02-10T11:39:18.491 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:39:18.491 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:39:18.491 INFO:teuthology.orchestra.run.gibba002.stdout: "auid": 0, 2021-02-10T11:39:18.491 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:39:18.491 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_seq": 0, 2021-02-10T11:39:18.491 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_epoch": 0, 2021-02-10T11:39:18.491 INFO:teuthology.orchestra.run.gibba002.stdout: "pool_snaps": [], 2021-02-10T11:39:18.491 INFO:teuthology.orchestra.run.gibba002.stdout: "removed_snaps": "[]", 2021-02-10T11:39:18.491 INFO:teuthology.orchestra.run.gibba002.stdout: "quota_max_bytes": 0, 2021-02-10T11:39:18.491 INFO:teuthology.orchestra.run.gibba002.stdout: "quota_max_objects": 0, 2021-02-10T11:39:18.491 INFO:teuthology.orchestra.run.gibba002.stdout: "tiers": [], 2021-02-10T11:39:18.492 INFO:teuthology.orchestra.run.gibba002.stdout: "tier_of": -1, 2021-02-10T11:39:18.492 INFO:teuthology.orchestra.run.gibba002.stdout: "read_tier": -1, 2021-02-10T11:39:18.492 INFO:teuthology.orchestra.run.gibba002.stdout: "write_tier": -1, 2021-02-10T11:39:18.492 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_mode": "none", 2021-02-10T11:39:18.492 INFO:teuthology.orchestra.run.gibba002.stdout: "target_max_bytes": 0, 2021-02-10T11:39:18.492 INFO:teuthology.orchestra.run.gibba002.stdout: "target_max_objects": 0, 2021-02-10T11:39:18.492 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:39:18.492 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:39:18.492 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:39:18.492 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_min_flush_age": 0, 2021-02-10T11:39:18.492 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_min_evict_age": 0, 2021-02-10T11:39:18.492 INFO:teuthology.orchestra.run.gibba002.stdout: "erasure_code_profile": "", 2021-02-10T11:39:18.492 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_params": { 2021-02-10T11:39:18.492 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "none" 2021-02-10T11:39:18.493 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:39:18.493 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_period": 0, 2021-02-10T11:39:18.493 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_count": 0, 2021-02-10T11:39:18.493 INFO:teuthology.orchestra.run.gibba002.stdout: "use_gmt_hitset": true, 2021-02-10T11:39:18.493 INFO:teuthology.orchestra.run.gibba002.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:39:18.493 INFO:teuthology.orchestra.run.gibba002.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:39:18.493 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:39:18.493 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:39:18.493 INFO:teuthology.orchestra.run.gibba002.stdout: "grade_table": [], 2021-02-10T11:39:18.493 INFO:teuthology.orchestra.run.gibba002.stdout: "stripe_width": 0, 2021-02-10T11:39:18.493 INFO:teuthology.orchestra.run.gibba002.stdout: "expected_num_objects": 0, 2021-02-10T11:39:18.493 INFO:teuthology.orchestra.run.gibba002.stdout: "fast_read": false, 2021-02-10T11:39:18.493 INFO:teuthology.orchestra.run.gibba002.stdout: "options": {}, 2021-02-10T11:39:18.493 INFO:teuthology.orchestra.run.gibba002.stdout: "application_metadata": { 2021-02-10T11:39:18.494 INFO:teuthology.orchestra.run.gibba002.stdout: "rbd": {} 2021-02-10T11:39:18.494 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:39:18.494 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:39:18.494 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:39:18.494 INFO:teuthology.orchestra.run.gibba002.stdout: "pool": 38, 2021-02-10T11:39:18.494 INFO:teuthology.orchestra.run.gibba002.stdout: "pool_name": "cephfs_metadata", 2021-02-10T11:39:18.494 INFO:teuthology.orchestra.run.gibba002.stdout: "create_time": "2021-02-10 11:38:28.298788", 2021-02-10T11:39:18.494 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 1, 2021-02-10T11:39:18.494 INFO:teuthology.orchestra.run.gibba002.stdout: "flags_names": "hashpspool", 2021-02-10T11:39:18.494 INFO:teuthology.orchestra.run.gibba002.stdout: "type": 1, 2021-02-10T11:39:18.494 INFO:teuthology.orchestra.run.gibba002.stdout: "size": 2, 2021-02-10T11:39:18.494 INFO:teuthology.orchestra.run.gibba002.stdout: "min_size": 1, 2021-02-10T11:39:18.494 INFO:teuthology.orchestra.run.gibba002.stdout: "crush_rule": 0, 2021-02-10T11:39:18.495 INFO:teuthology.orchestra.run.gibba002.stdout: "object_hash": 2, 2021-02-10T11:39:18.495 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:39:18.495 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num": 8, 2021-02-10T11:39:18.495 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_placement_num": 8, 2021-02-10T11:39:18.495 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_placement_num_target": 8, 2021-02-10T11:39:18.495 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num_target": 8, 2021-02-10T11:39:18.495 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num_pending": 8, 2021-02-10T11:39:18.495 INFO:teuthology.orchestra.run.gibba002.stdout: "last_pg_merge_meta": { 2021-02-10T11:39:18.495 INFO:teuthology.orchestra.run.gibba002.stdout: "source_pgid": "0.0", 2021-02-10T11:39:18.495 INFO:teuthology.orchestra.run.gibba002.stdout: "ready_epoch": 0, 2021-02-10T11:39:18.495 INFO:teuthology.orchestra.run.gibba002.stdout: "last_epoch_started": 0, 2021-02-10T11:39:18.495 INFO:teuthology.orchestra.run.gibba002.stdout: "last_epoch_clean": 0, 2021-02-10T11:39:18.495 INFO:teuthology.orchestra.run.gibba002.stdout: "source_version": "0'0", 2021-02-10T11:39:18.495 INFO:teuthology.orchestra.run.gibba002.stdout: "target_version": "0'0" 2021-02-10T11:39:18.496 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:39:18.496 INFO:teuthology.orchestra.run.gibba002.stdout: "last_change": "179", 2021-02-10T11:39:18.496 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend": "0", 2021-02-10T11:39:18.496 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:39:18.496 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:39:18.496 INFO:teuthology.orchestra.run.gibba002.stdout: "auid": 0, 2021-02-10T11:39:18.496 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:39:18.496 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_seq": 0, 2021-02-10T11:39:18.496 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_epoch": 0, 2021-02-10T11:39:18.496 INFO:teuthology.orchestra.run.gibba002.stdout: "pool_snaps": [], 2021-02-10T11:39:18.496 INFO:teuthology.orchestra.run.gibba002.stdout: "removed_snaps": "[]", 2021-02-10T11:39:18.496 INFO:teuthology.orchestra.run.gibba002.stdout: "quota_max_bytes": 0, 2021-02-10T11:39:18.496 INFO:teuthology.orchestra.run.gibba002.stdout: "quota_max_objects": 0, 2021-02-10T11:39:18.496 INFO:teuthology.orchestra.run.gibba002.stdout: "tiers": [], 2021-02-10T11:39:18.497 INFO:teuthology.orchestra.run.gibba002.stdout: "tier_of": -1, 2021-02-10T11:39:18.497 INFO:teuthology.orchestra.run.gibba002.stdout: "read_tier": -1, 2021-02-10T11:39:18.497 INFO:teuthology.orchestra.run.gibba002.stdout: "write_tier": -1, 2021-02-10T11:39:18.497 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_mode": "none", 2021-02-10T11:39:18.497 INFO:teuthology.orchestra.run.gibba002.stdout: "target_max_bytes": 0, 2021-02-10T11:39:18.497 INFO:teuthology.orchestra.run.gibba002.stdout: "target_max_objects": 0, 2021-02-10T11:39:18.497 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:39:18.497 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:39:18.497 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:39:18.497 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_min_flush_age": 0, 2021-02-10T11:39:18.497 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_min_evict_age": 0, 2021-02-10T11:39:18.497 INFO:teuthology.orchestra.run.gibba002.stdout: "erasure_code_profile": "", 2021-02-10T11:39:18.497 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_params": { 2021-02-10T11:39:18.497 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "none" 2021-02-10T11:39:18.498 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:39:18.498 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_period": 0, 2021-02-10T11:39:18.498 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_count": 0, 2021-02-10T11:39:18.498 INFO:teuthology.orchestra.run.gibba002.stdout: "use_gmt_hitset": true, 2021-02-10T11:39:18.498 INFO:teuthology.orchestra.run.gibba002.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:39:18.498 INFO:teuthology.orchestra.run.gibba002.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:39:18.498 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:39:18.498 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:39:18.498 INFO:teuthology.orchestra.run.gibba002.stdout: "grade_table": [], 2021-02-10T11:39:18.498 INFO:teuthology.orchestra.run.gibba002.stdout: "stripe_width": 0, 2021-02-10T11:39:18.498 INFO:teuthology.orchestra.run.gibba002.stdout: "expected_num_objects": 0, 2021-02-10T11:39:18.498 INFO:teuthology.orchestra.run.gibba002.stdout: "fast_read": false, 2021-02-10T11:39:18.498 INFO:teuthology.orchestra.run.gibba002.stdout: "options": { 2021-02-10T11:39:18.499 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_autoscale_bias": 4, 2021-02-10T11:39:18.499 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num_min": 16, 2021-02-10T11:39:18.499 INFO:teuthology.orchestra.run.gibba002.stdout: "recovery_priority": 5 2021-02-10T11:39:18.499 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:39:18.499 INFO:teuthology.orchestra.run.gibba002.stdout: "application_metadata": { 2021-02-10T11:39:18.499 INFO:teuthology.orchestra.run.gibba002.stdout: "cephfs": { 2021-02-10T11:39:18.499 INFO:teuthology.orchestra.run.gibba002.stdout: "metadata": "cephfs" 2021-02-10T11:39:18.499 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:39:18.499 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:39:18.499 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:39:18.499 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:39:18.499 INFO:teuthology.orchestra.run.gibba002.stdout: "pool": 39, 2021-02-10T11:39:18.499 INFO:teuthology.orchestra.run.gibba002.stdout: "pool_name": "cephfs_data", 2021-02-10T11:39:18.499 INFO:teuthology.orchestra.run.gibba002.stdout: "create_time": "2021-02-10 11:38:29.326316", 2021-02-10T11:39:18.500 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 1, 2021-02-10T11:39:18.500 INFO:teuthology.orchestra.run.gibba002.stdout: "flags_names": "hashpspool", 2021-02-10T11:39:18.500 INFO:teuthology.orchestra.run.gibba002.stdout: "type": 1, 2021-02-10T11:39:18.500 INFO:teuthology.orchestra.run.gibba002.stdout: "size": 2, 2021-02-10T11:39:18.500 INFO:teuthology.orchestra.run.gibba002.stdout: "min_size": 1, 2021-02-10T11:39:18.500 INFO:teuthology.orchestra.run.gibba002.stdout: "crush_rule": 0, 2021-02-10T11:39:18.500 INFO:teuthology.orchestra.run.gibba002.stdout: "object_hash": 2, 2021-02-10T11:39:18.500 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:39:18.500 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num": 8, 2021-02-10T11:39:18.500 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_placement_num": 8, 2021-02-10T11:39:18.500 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_placement_num_target": 8, 2021-02-10T11:39:18.500 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num_target": 8, 2021-02-10T11:39:18.500 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num_pending": 8, 2021-02-10T11:39:18.500 INFO:teuthology.orchestra.run.gibba002.stdout: "last_pg_merge_meta": { 2021-02-10T11:39:18.501 INFO:teuthology.orchestra.run.gibba002.stdout: "source_pgid": "0.0", 2021-02-10T11:39:18.501 INFO:teuthology.orchestra.run.gibba002.stdout: "ready_epoch": 0, 2021-02-10T11:39:18.501 INFO:teuthology.orchestra.run.gibba002.stdout: "last_epoch_started": 0, 2021-02-10T11:39:18.501 INFO:teuthology.orchestra.run.gibba002.stdout: "last_epoch_clean": 0, 2021-02-10T11:39:18.501 INFO:teuthology.orchestra.run.gibba002.stdout: "source_version": "0'0", 2021-02-10T11:39:18.501 INFO:teuthology.orchestra.run.gibba002.stdout: "target_version": "0'0" 2021-02-10T11:39:18.501 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:39:18.501 INFO:teuthology.orchestra.run.gibba002.stdout: "last_change": "180", 2021-02-10T11:39:18.501 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend": "0", 2021-02-10T11:39:18.501 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:39:18.501 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:39:18.501 INFO:teuthology.orchestra.run.gibba002.stdout: "auid": 0, 2021-02-10T11:39:18.501 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:39:18.501 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_seq": 0, 2021-02-10T11:39:18.502 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_epoch": 0, 2021-02-10T11:39:18.502 INFO:teuthology.orchestra.run.gibba002.stdout: "pool_snaps": [], 2021-02-10T11:39:18.502 INFO:teuthology.orchestra.run.gibba002.stdout: "removed_snaps": "[]", 2021-02-10T11:39:18.502 INFO:teuthology.orchestra.run.gibba002.stdout: "quota_max_bytes": 0, 2021-02-10T11:39:18.502 INFO:teuthology.orchestra.run.gibba002.stdout: "quota_max_objects": 0, 2021-02-10T11:39:18.502 INFO:teuthology.orchestra.run.gibba002.stdout: "tiers": [], 2021-02-10T11:39:18.502 INFO:teuthology.orchestra.run.gibba002.stdout: "tier_of": -1, 2021-02-10T11:39:18.502 INFO:teuthology.orchestra.run.gibba002.stdout: "read_tier": -1, 2021-02-10T11:39:18.502 INFO:teuthology.orchestra.run.gibba002.stdout: "write_tier": -1, 2021-02-10T11:39:18.502 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_mode": "none", 2021-02-10T11:39:18.502 INFO:teuthology.orchestra.run.gibba002.stdout: "target_max_bytes": 0, 2021-02-10T11:39:18.502 INFO:teuthology.orchestra.run.gibba002.stdout: "target_max_objects": 0, 2021-02-10T11:39:18.502 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:39:18.503 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:39:18.503 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:39:18.503 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_min_flush_age": 0, 2021-02-10T11:39:18.503 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_min_evict_age": 0, 2021-02-10T11:39:18.503 INFO:teuthology.orchestra.run.gibba002.stdout: "erasure_code_profile": "", 2021-02-10T11:39:18.503 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_params": { 2021-02-10T11:39:18.503 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "none" 2021-02-10T11:39:18.503 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:39:18.503 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_period": 0, 2021-02-10T11:39:18.503 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_count": 0, 2021-02-10T11:39:18.503 INFO:teuthology.orchestra.run.gibba002.stdout: "use_gmt_hitset": true, 2021-02-10T11:39:18.503 INFO:teuthology.orchestra.run.gibba002.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:39:18.503 INFO:teuthology.orchestra.run.gibba002.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:39:18.503 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:39:18.504 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:39:18.504 INFO:teuthology.orchestra.run.gibba002.stdout: "grade_table": [], 2021-02-10T11:39:18.504 INFO:teuthology.orchestra.run.gibba002.stdout: "stripe_width": 0, 2021-02-10T11:39:18.504 INFO:teuthology.orchestra.run.gibba002.stdout: "expected_num_objects": 0, 2021-02-10T11:39:18.504 INFO:teuthology.orchestra.run.gibba002.stdout: "fast_read": false, 2021-02-10T11:39:18.504 INFO:teuthology.orchestra.run.gibba002.stdout: "options": {}, 2021-02-10T11:39:18.504 INFO:teuthology.orchestra.run.gibba002.stdout: "application_metadata": { 2021-02-10T11:39:18.504 INFO:teuthology.orchestra.run.gibba002.stdout: "cephfs": { 2021-02-10T11:39:18.504 INFO:teuthology.orchestra.run.gibba002.stdout: "data": "cephfs" 2021-02-10T11:39:18.504 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:39:18.504 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:39:18.504 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:39:18.504 INFO:teuthology.orchestra.run.gibba002.stdout: ], 2021-02-10T11:39:18.504 INFO:teuthology.orchestra.run.gibba002.stdout: "osds": [ 2021-02-10T11:39:18.505 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:39:18.505 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 0, 2021-02-10T11:39:18.505 INFO:teuthology.orchestra.run.gibba002.stdout: "uuid": "be691b4d-a039-408a-af80-9d53f02b6f1f", 2021-02-10T11:39:18.505 INFO:teuthology.orchestra.run.gibba002.stdout: "up": 1, 2021-02-10T11:39:18.505 INFO:teuthology.orchestra.run.gibba002.stdout: "in": 1, 2021-02-10T11:39:18.505 INFO:teuthology.orchestra.run.gibba002.stdout: "weight": 1, 2021-02-10T11:39:18.505 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_affinity": 1, 2021-02-10T11:39:18.505 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_begin": 0, 2021-02-10T11:39:18.505 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_end": 0, 2021-02-10T11:39:18.505 INFO:teuthology.orchestra.run.gibba002.stdout: "up_from": 13, 2021-02-10T11:39:18.505 INFO:teuthology.orchestra.run.gibba002.stdout: "up_thru": 178, 2021-02-10T11:39:18.505 INFO:teuthology.orchestra.run.gibba002.stdout: "down_at": 0, 2021-02-10T11:39:18.505 INFO:teuthology.orchestra.run.gibba002.stdout: "lost_at": 0, 2021-02-10T11:39:18.505 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addrs": { 2021-02-10T11:39:18.506 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:39:18.506 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:39:18.506 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:39:18.506 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6818", 2021-02-10T11:39:18.506 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36088 2021-02-10T11:39:18.506 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:39:18.506 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:39:18.506 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:39:18.506 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6820", 2021-02-10T11:39:18.506 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36088 2021-02-10T11:39:18.506 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:39:18.506 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:39:18.506 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:39:18.507 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addrs": { 2021-02-10T11:39:18.507 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:39:18.507 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:39:18.507 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:39:18.507 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6822", 2021-02-10T11:39:18.507 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36088 2021-02-10T11:39:18.507 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:39:18.507 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:39:18.507 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:39:18.507 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6824", 2021-02-10T11:39:18.507 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36088 2021-02-10T11:39:18.507 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:39:18.507 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:39:18.507 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:39:18.508 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addrs": { 2021-02-10T11:39:18.508 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:39:18.508 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:39:18.508 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:39:18.508 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6830", 2021-02-10T11:39:18.508 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36088 2021-02-10T11:39:18.508 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:39:18.508 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:39:18.508 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:39:18.508 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6832", 2021-02-10T11:39:18.508 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36088 2021-02-10T11:39:18.508 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:39:18.508 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:39:18.508 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:39:18.509 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addrs": { 2021-02-10T11:39:18.509 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:39:18.509 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:39:18.509 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:39:18.509 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6826", 2021-02-10T11:39:18.509 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36088 2021-02-10T11:39:18.509 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:39:18.509 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:39:18.509 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:39:18.509 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6828", 2021-02-10T11:39:18.509 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36088 2021-02-10T11:39:18.509 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:39:18.509 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:39:18.509 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:39:18.510 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addr": "172.21.2.102:6820/36088", 2021-02-10T11:39:18.510 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addr": "172.21.2.102:6824/36088", 2021-02-10T11:39:18.510 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addr": "172.21.2.102:6832/36088", 2021-02-10T11:39:18.510 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addr": "172.21.2.102:6828/36088", 2021-02-10T11:39:18.510 INFO:teuthology.orchestra.run.gibba002.stdout: "state": [ 2021-02-10T11:39:18.510 INFO:teuthology.orchestra.run.gibba002.stdout: "exists", 2021-02-10T11:39:18.510 INFO:teuthology.orchestra.run.gibba002.stdout: "up" 2021-02-10T11:39:18.510 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:39:18.510 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:39:18.510 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:39:18.510 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 1, 2021-02-10T11:39:18.510 INFO:teuthology.orchestra.run.gibba002.stdout: "uuid": "64e3ccc8-9c69-4d7c-865e-af8e83822635", 2021-02-10T11:39:18.510 INFO:teuthology.orchestra.run.gibba002.stdout: "up": 1, 2021-02-10T11:39:18.511 INFO:teuthology.orchestra.run.gibba002.stdout: "in": 1, 2021-02-10T11:39:18.511 INFO:teuthology.orchestra.run.gibba002.stdout: "weight": 1, 2021-02-10T11:39:18.511 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_affinity": 1, 2021-02-10T11:39:18.511 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_begin": 0, 2021-02-10T11:39:18.511 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_end": 0, 2021-02-10T11:39:18.511 INFO:teuthology.orchestra.run.gibba002.stdout: "up_from": 13, 2021-02-10T11:39:18.511 INFO:teuthology.orchestra.run.gibba002.stdout: "up_thru": 178, 2021-02-10T11:39:18.511 INFO:teuthology.orchestra.run.gibba002.stdout: "down_at": 0, 2021-02-10T11:39:18.511 INFO:teuthology.orchestra.run.gibba002.stdout: "lost_at": 0, 2021-02-10T11:39:18.511 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addrs": { 2021-02-10T11:39:18.511 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:39:18.511 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:39:18.511 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:39:18.511 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6819", 2021-02-10T11:39:18.512 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36087 2021-02-10T11:39:18.512 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:39:18.512 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:39:18.512 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:39:18.512 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6821", 2021-02-10T11:39:18.512 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36087 2021-02-10T11:39:18.512 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:39:18.512 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:39:18.512 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:39:18.512 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addrs": { 2021-02-10T11:39:18.512 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:39:18.512 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:39:18.512 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:39:18.512 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6823", 2021-02-10T11:39:18.513 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36087 2021-02-10T11:39:18.513 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:39:18.513 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:39:18.513 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:39:18.513 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6825", 2021-02-10T11:39:18.513 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36087 2021-02-10T11:39:18.513 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:39:18.513 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:39:18.513 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:39:18.513 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addrs": { 2021-02-10T11:39:18.513 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:39:18.513 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:39:18.513 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:39:18.513 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6831", 2021-02-10T11:39:18.514 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36087 2021-02-10T11:39:18.514 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:39:18.514 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:39:18.514 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:39:18.514 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6833", 2021-02-10T11:39:18.514 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36087 2021-02-10T11:39:18.514 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:39:18.514 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:39:18.514 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:39:18.514 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addrs": { 2021-02-10T11:39:18.514 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:39:18.514 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:39:18.514 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:39:18.514 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6827", 2021-02-10T11:39:18.515 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36087 2021-02-10T11:39:18.515 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:39:18.515 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:39:18.515 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:39:18.515 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6829", 2021-02-10T11:39:18.515 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36087 2021-02-10T11:39:18.515 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:39:18.515 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:39:18.515 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:39:18.515 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addr": "172.21.2.102:6821/36087", 2021-02-10T11:39:18.515 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addr": "172.21.2.102:6825/36087", 2021-02-10T11:39:18.515 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addr": "172.21.2.102:6833/36087", 2021-02-10T11:39:18.515 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addr": "172.21.2.102:6829/36087", 2021-02-10T11:39:18.515 INFO:teuthology.orchestra.run.gibba002.stdout: "state": [ 2021-02-10T11:39:18.516 INFO:teuthology.orchestra.run.gibba002.stdout: "exists", 2021-02-10T11:39:18.516 INFO:teuthology.orchestra.run.gibba002.stdout: "up" 2021-02-10T11:39:18.516 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:39:18.516 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:39:18.516 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:39:18.516 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 2, 2021-02-10T11:39:18.516 INFO:teuthology.orchestra.run.gibba002.stdout: "uuid": "153700af-747f-481f-bdf4-d6072d3a842f", 2021-02-10T11:39:18.516 INFO:teuthology.orchestra.run.gibba002.stdout: "up": 1, 2021-02-10T11:39:18.516 INFO:teuthology.orchestra.run.gibba002.stdout: "in": 1, 2021-02-10T11:39:18.516 INFO:teuthology.orchestra.run.gibba002.stdout: "weight": 1, 2021-02-10T11:39:18.516 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_affinity": 1, 2021-02-10T11:39:18.516 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_begin": 0, 2021-02-10T11:39:18.516 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_end": 0, 2021-02-10T11:39:18.516 INFO:teuthology.orchestra.run.gibba002.stdout: "up_from": 13, 2021-02-10T11:39:18.517 INFO:teuthology.orchestra.run.gibba002.stdout: "up_thru": 178, 2021-02-10T11:39:18.517 INFO:teuthology.orchestra.run.gibba002.stdout: "down_at": 0, 2021-02-10T11:39:18.517 INFO:teuthology.orchestra.run.gibba002.stdout: "lost_at": 0, 2021-02-10T11:39:18.517 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addrs": { 2021-02-10T11:39:18.517 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:39:18.517 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:39:18.517 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:39:18.517 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6810", 2021-02-10T11:39:18.517 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36089 2021-02-10T11:39:18.517 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:39:18.517 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:39:18.517 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:39:18.517 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6811", 2021-02-10T11:39:18.517 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36089 2021-02-10T11:39:18.518 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:39:18.518 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:39:18.518 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:39:18.518 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addrs": { 2021-02-10T11:39:18.518 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:39:18.518 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:39:18.518 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:39:18.518 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6812", 2021-02-10T11:39:18.518 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36089 2021-02-10T11:39:18.518 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:39:18.518 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:39:18.518 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:39:18.519 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6813", 2021-02-10T11:39:18.519 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36089 2021-02-10T11:39:18.519 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:39:18.519 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:39:18.519 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:39:18.519 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addrs": { 2021-02-10T11:39:18.519 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:39:18.519 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:39:18.519 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:39:18.519 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6816", 2021-02-10T11:39:18.519 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36089 2021-02-10T11:39:18.519 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:39:18.519 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:39:18.519 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:39:18.520 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6817", 2021-02-10T11:39:18.520 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36089 2021-02-10T11:39:18.520 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:39:18.520 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:39:18.520 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:39:18.520 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addrs": { 2021-02-10T11:39:18.520 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:39:18.520 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:39:18.520 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:39:18.520 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6814", 2021-02-10T11:39:18.520 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36089 2021-02-10T11:39:18.520 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:39:18.520 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:39:18.520 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:39:18.521 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6815", 2021-02-10T11:39:18.521 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36089 2021-02-10T11:39:18.521 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:39:18.521 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:39:18.521 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:39:18.521 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addr": "172.21.2.102:6811/36089", 2021-02-10T11:39:18.521 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addr": "172.21.2.102:6813/36089", 2021-02-10T11:39:18.521 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addr": "172.21.2.102:6817/36089", 2021-02-10T11:39:18.521 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addr": "172.21.2.102:6815/36089", 2021-02-10T11:39:18.521 INFO:teuthology.orchestra.run.gibba002.stdout: "state": [ 2021-02-10T11:39:18.521 INFO:teuthology.orchestra.run.gibba002.stdout: "exists", 2021-02-10T11:39:18.522 INFO:teuthology.orchestra.run.gibba002.stdout: "up" 2021-02-10T11:39:18.522 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:39:18.522 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:39:18.522 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:39:18.522 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 3, 2021-02-10T11:39:18.522 INFO:teuthology.orchestra.run.gibba002.stdout: "uuid": "cbc678f4-83a9-45e6-8830-2f7c232b1ea7", 2021-02-10T11:39:18.522 INFO:teuthology.orchestra.run.gibba002.stdout: "up": 1, 2021-02-10T11:39:18.522 INFO:teuthology.orchestra.run.gibba002.stdout: "in": 1, 2021-02-10T11:39:18.522 INFO:teuthology.orchestra.run.gibba002.stdout: "weight": 1, 2021-02-10T11:39:18.522 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_affinity": 1, 2021-02-10T11:39:18.522 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_begin": 0, 2021-02-10T11:39:18.522 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_end": 0, 2021-02-10T11:39:18.523 INFO:teuthology.orchestra.run.gibba002.stdout: "up_from": 13, 2021-02-10T11:39:18.523 INFO:teuthology.orchestra.run.gibba002.stdout: "up_thru": 178, 2021-02-10T11:39:18.523 INFO:teuthology.orchestra.run.gibba002.stdout: "down_at": 0, 2021-02-10T11:39:18.523 INFO:teuthology.orchestra.run.gibba002.stdout: "lost_at": 0, 2021-02-10T11:39:18.523 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addrs": { 2021-02-10T11:39:18.523 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:39:18.523 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:39:18.523 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:39:18.523 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6802", 2021-02-10T11:39:18.523 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36090 2021-02-10T11:39:18.523 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:39:18.523 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:39:18.523 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:39:18.523 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6803", 2021-02-10T11:39:18.524 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36090 2021-02-10T11:39:18.524 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:39:18.524 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:39:18.524 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:39:18.524 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addrs": { 2021-02-10T11:39:18.524 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:39:18.524 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:39:18.524 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:39:18.524 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6804", 2021-02-10T11:39:18.524 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36090 2021-02-10T11:39:18.524 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:39:18.524 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:39:18.524 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:39:18.525 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6805", 2021-02-10T11:39:18.525 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36090 2021-02-10T11:39:18.525 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:39:18.525 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:39:18.525 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:39:18.525 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addrs": { 2021-02-10T11:39:18.525 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:39:18.525 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:39:18.525 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:39:18.525 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6808", 2021-02-10T11:39:18.525 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36090 2021-02-10T11:39:18.525 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:39:18.525 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:39:18.526 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:39:18.526 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6809", 2021-02-10T11:39:18.526 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36090 2021-02-10T11:39:18.526 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:39:18.526 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:39:18.526 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:39:18.526 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addrs": { 2021-02-10T11:39:18.526 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:39:18.526 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:39:18.526 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:39:18.526 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6806", 2021-02-10T11:39:18.526 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36090 2021-02-10T11:39:18.527 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:39:18.527 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:39:18.527 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:39:18.527 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6807", 2021-02-10T11:39:18.527 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36090 2021-02-10T11:39:18.527 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:39:18.527 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:39:18.527 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:39:18.527 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addr": "172.21.2.102:6803/36090", 2021-02-10T11:39:18.527 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addr": "172.21.2.102:6805/36090", 2021-02-10T11:39:18.527 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addr": "172.21.2.102:6809/36090", 2021-02-10T11:39:18.527 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addr": "172.21.2.102:6807/36090", 2021-02-10T11:39:18.527 INFO:teuthology.orchestra.run.gibba002.stdout: "state": [ 2021-02-10T11:39:18.527 INFO:teuthology.orchestra.run.gibba002.stdout: "exists", 2021-02-10T11:39:18.528 INFO:teuthology.orchestra.run.gibba002.stdout: "up" 2021-02-10T11:39:18.528 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:39:18.528 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:39:18.528 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:39:18.528 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 4, 2021-02-10T11:39:18.528 INFO:teuthology.orchestra.run.gibba002.stdout: "uuid": "c01e4008-b258-4f10-afd2-9e7f37214351", 2021-02-10T11:39:18.528 INFO:teuthology.orchestra.run.gibba002.stdout: "up": 1, 2021-02-10T11:39:18.528 INFO:teuthology.orchestra.run.gibba002.stdout: "in": 1, 2021-02-10T11:39:18.528 INFO:teuthology.orchestra.run.gibba002.stdout: "weight": 1, 2021-02-10T11:39:18.528 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_affinity": 1, 2021-02-10T11:39:18.528 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_begin": 0, 2021-02-10T11:39:18.528 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_end": 0, 2021-02-10T11:39:18.528 INFO:teuthology.orchestra.run.gibba002.stdout: "up_from": 13, 2021-02-10T11:39:18.529 INFO:teuthology.orchestra.run.gibba002.stdout: "up_thru": 178, 2021-02-10T11:39:18.529 INFO:teuthology.orchestra.run.gibba002.stdout: "down_at": 0, 2021-02-10T11:39:18.529 INFO:teuthology.orchestra.run.gibba002.stdout: "lost_at": 0, 2021-02-10T11:39:18.529 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addrs": { 2021-02-10T11:39:18.529 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:39:18.529 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:39:18.529 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:39:18.529 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6808", 2021-02-10T11:39:18.529 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36368 2021-02-10T11:39:18.529 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:39:18.529 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:39:18.529 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:39:18.529 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6811", 2021-02-10T11:39:18.529 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36368 2021-02-10T11:39:18.530 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:39:18.530 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:39:18.530 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:39:18.530 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addrs": { 2021-02-10T11:39:18.530 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:39:18.530 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:39:18.530 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:39:18.530 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6814", 2021-02-10T11:39:18.530 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36368 2021-02-10T11:39:18.530 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:39:18.530 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:39:18.530 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:39:18.530 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6818", 2021-02-10T11:39:18.531 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36368 2021-02-10T11:39:18.531 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:39:18.531 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:39:18.531 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:39:18.531 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addrs": { 2021-02-10T11:39:18.531 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:39:18.531 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:39:18.531 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:39:18.531 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6826", 2021-02-10T11:39:18.531 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36368 2021-02-10T11:39:18.531 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:39:18.531 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:39:18.531 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:39:18.531 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6828", 2021-02-10T11:39:18.532 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36368 2021-02-10T11:39:18.532 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:39:18.532 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:39:18.532 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:39:18.532 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addrs": { 2021-02-10T11:39:18.532 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:39:18.532 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:39:18.532 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:39:18.532 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6820", 2021-02-10T11:39:18.532 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36368 2021-02-10T11:39:18.532 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:39:18.532 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:39:18.532 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:39:18.532 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6824", 2021-02-10T11:39:18.533 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36368 2021-02-10T11:39:18.533 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:39:18.533 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:39:18.533 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:39:18.533 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addr": "172.21.2.113:6811/36368", 2021-02-10T11:39:18.533 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addr": "172.21.2.113:6818/36368", 2021-02-10T11:39:18.533 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addr": "172.21.2.113:6828/36368", 2021-02-10T11:39:18.533 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addr": "172.21.2.113:6824/36368", 2021-02-10T11:39:18.533 INFO:teuthology.orchestra.run.gibba002.stdout: "state": [ 2021-02-10T11:39:18.533 INFO:teuthology.orchestra.run.gibba002.stdout: "exists", 2021-02-10T11:39:18.533 INFO:teuthology.orchestra.run.gibba002.stdout: "up" 2021-02-10T11:39:18.533 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:39:18.533 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:39:18.534 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:39:18.534 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 5, 2021-02-10T11:39:18.534 INFO:teuthology.orchestra.run.gibba002.stdout: "uuid": "551f6007-daf8-4a98-8ee0-c2b16bcccb5e", 2021-02-10T11:39:18.534 INFO:teuthology.orchestra.run.gibba002.stdout: "up": 1, 2021-02-10T11:39:18.534 INFO:teuthology.orchestra.run.gibba002.stdout: "in": 1, 2021-02-10T11:39:18.534 INFO:teuthology.orchestra.run.gibba002.stdout: "weight": 1, 2021-02-10T11:39:18.534 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_affinity": 1, 2021-02-10T11:39:18.534 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_begin": 0, 2021-02-10T11:39:18.534 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_end": 0, 2021-02-10T11:39:18.534 INFO:teuthology.orchestra.run.gibba002.stdout: "up_from": 13, 2021-02-10T11:39:18.534 INFO:teuthology.orchestra.run.gibba002.stdout: "up_thru": 178, 2021-02-10T11:39:18.534 INFO:teuthology.orchestra.run.gibba002.stdout: "down_at": 0, 2021-02-10T11:39:18.534 INFO:teuthology.orchestra.run.gibba002.stdout: "lost_at": 0, 2021-02-10T11:39:18.534 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addrs": { 2021-02-10T11:39:18.535 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:39:18.535 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:39:18.535 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:39:18.535 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6802", 2021-02-10T11:39:18.535 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36366 2021-02-10T11:39:18.535 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:39:18.535 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:39:18.535 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:39:18.535 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6805", 2021-02-10T11:39:18.535 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36366 2021-02-10T11:39:18.535 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:39:18.535 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:39:18.535 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:39:18.535 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addrs": { 2021-02-10T11:39:18.536 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:39:18.536 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:39:18.536 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:39:18.536 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6807", 2021-02-10T11:39:18.536 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36366 2021-02-10T11:39:18.536 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:39:18.536 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:39:18.536 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:39:18.536 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6810", 2021-02-10T11:39:18.536 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36366 2021-02-10T11:39:18.536 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:39:18.536 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:39:18.536 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:39:18.536 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addrs": { 2021-02-10T11:39:18.537 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:39:18.537 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:39:18.537 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:39:18.537 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6819", 2021-02-10T11:39:18.537 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36366 2021-02-10T11:39:18.537 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:39:18.537 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:39:18.537 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:39:18.537 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6822", 2021-02-10T11:39:18.537 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36366 2021-02-10T11:39:18.537 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:39:18.538 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:39:18.538 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:39:18.538 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addrs": { 2021-02-10T11:39:18.538 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:39:18.538 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:39:18.538 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:39:18.538 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6812", 2021-02-10T11:39:18.538 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36366 2021-02-10T11:39:18.538 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:39:18.539 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:39:18.539 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:39:18.539 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6815", 2021-02-10T11:39:18.539 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36366 2021-02-10T11:39:18.539 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:39:18.539 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:39:18.539 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:39:18.539 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addr": "172.21.2.113:6805/36366", 2021-02-10T11:39:18.539 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addr": "172.21.2.113:6810/36366", 2021-02-10T11:39:18.540 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addr": "172.21.2.113:6822/36366", 2021-02-10T11:39:18.540 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addr": "172.21.2.113:6815/36366", 2021-02-10T11:39:18.540 INFO:teuthology.orchestra.run.gibba002.stdout: "state": [ 2021-02-10T11:39:18.540 INFO:teuthology.orchestra.run.gibba002.stdout: "exists", 2021-02-10T11:39:18.540 INFO:teuthology.orchestra.run.gibba002.stdout: "up" 2021-02-10T11:39:18.540 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:39:18.540 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:39:18.541 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:39:18.541 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 6, 2021-02-10T11:39:18.541 INFO:teuthology.orchestra.run.gibba002.stdout: "uuid": "bf67c040-0e50-48be-b102-c3943bc9ebef", 2021-02-10T11:39:18.541 INFO:teuthology.orchestra.run.gibba002.stdout: "up": 1, 2021-02-10T11:39:18.541 INFO:teuthology.orchestra.run.gibba002.stdout: "in": 1, 2021-02-10T11:39:18.541 INFO:teuthology.orchestra.run.gibba002.stdout: "weight": 1, 2021-02-10T11:39:18.541 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_affinity": 1, 2021-02-10T11:39:18.541 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_begin": 0, 2021-02-10T11:39:18.541 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_end": 0, 2021-02-10T11:39:18.542 INFO:teuthology.orchestra.run.gibba002.stdout: "up_from": 13, 2021-02-10T11:39:18.542 INFO:teuthology.orchestra.run.gibba002.stdout: "up_thru": 177, 2021-02-10T11:39:18.542 INFO:teuthology.orchestra.run.gibba002.stdout: "down_at": 0, 2021-02-10T11:39:18.542 INFO:teuthology.orchestra.run.gibba002.stdout: "lost_at": 0, 2021-02-10T11:39:18.542 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addrs": { 2021-02-10T11:39:18.542 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:39:18.542 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:39:18.542 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:39:18.542 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6800", 2021-02-10T11:39:18.542 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36367 2021-02-10T11:39:18.542 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:39:18.542 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:39:18.542 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:39:18.543 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6801", 2021-02-10T11:39:18.543 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36367 2021-02-10T11:39:18.543 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:39:18.543 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:39:18.543 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:39:18.543 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addrs": { 2021-02-10T11:39:18.543 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:39:18.543 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:39:18.543 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:39:18.543 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6803", 2021-02-10T11:39:18.544 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36367 2021-02-10T11:39:18.544 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:39:18.544 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:39:18.544 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:39:18.544 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6804", 2021-02-10T11:39:18.544 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36367 2021-02-10T11:39:18.544 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:39:18.545 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:39:18.545 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:39:18.545 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addrs": { 2021-02-10T11:39:18.545 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:39:18.545 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:39:18.545 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:39:18.545 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6813", 2021-02-10T11:39:18.545 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36367 2021-02-10T11:39:18.545 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:39:18.545 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:39:18.545 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:39:18.546 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6816", 2021-02-10T11:39:18.546 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36367 2021-02-10T11:39:18.546 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:39:18.546 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:39:18.546 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:39:18.546 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addrs": { 2021-02-10T11:39:18.546 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:39:18.546 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:39:18.546 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:39:18.546 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6806", 2021-02-10T11:39:18.546 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36367 2021-02-10T11:39:18.546 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:39:18.547 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:39:18.547 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:39:18.547 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6809", 2021-02-10T11:39:18.547 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36367 2021-02-10T11:39:18.547 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:39:18.547 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:39:18.547 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:39:18.547 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addr": "172.21.2.113:6801/36367", 2021-02-10T11:39:18.547 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addr": "172.21.2.113:6804/36367", 2021-02-10T11:39:18.547 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addr": "172.21.2.113:6816/36367", 2021-02-10T11:39:18.547 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addr": "172.21.2.113:6809/36367", 2021-02-10T11:39:18.547 INFO:teuthology.orchestra.run.gibba002.stdout: "state": [ 2021-02-10T11:39:18.548 INFO:teuthology.orchestra.run.gibba002.stdout: "exists", 2021-02-10T11:39:18.548 INFO:teuthology.orchestra.run.gibba002.stdout: "up" 2021-02-10T11:39:18.548 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:39:18.548 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:39:18.548 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:39:18.548 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 7, 2021-02-10T11:39:18.548 INFO:teuthology.orchestra.run.gibba002.stdout: "uuid": "f5461f80-018d-428f-be89-2eb2bcbe7bb6", 2021-02-10T11:39:18.548 INFO:teuthology.orchestra.run.gibba002.stdout: "up": 1, 2021-02-10T11:39:18.549 INFO:teuthology.orchestra.run.gibba002.stdout: "in": 1, 2021-02-10T11:39:18.549 INFO:teuthology.orchestra.run.gibba002.stdout: "weight": 1, 2021-02-10T11:39:18.549 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_affinity": 1, 2021-02-10T11:39:18.549 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_begin": 0, 2021-02-10T11:39:18.549 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_end": 0, 2021-02-10T11:39:18.549 INFO:teuthology.orchestra.run.gibba002.stdout: "up_from": 13, 2021-02-10T11:39:18.549 INFO:teuthology.orchestra.run.gibba002.stdout: "up_thru": 178, 2021-02-10T11:39:18.550 INFO:teuthology.orchestra.run.gibba002.stdout: "down_at": 0, 2021-02-10T11:39:18.550 INFO:teuthology.orchestra.run.gibba002.stdout: "lost_at": 0, 2021-02-10T11:39:18.550 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addrs": { 2021-02-10T11:39:18.550 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:39:18.550 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:39:18.550 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:39:18.550 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6817", 2021-02-10T11:39:18.550 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36365 2021-02-10T11:39:18.550 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:39:18.550 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:39:18.550 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:39:18.550 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6821", 2021-02-10T11:39:18.551 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36365 2021-02-10T11:39:18.551 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:39:18.551 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:39:18.551 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:39:18.551 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addrs": { 2021-02-10T11:39:18.551 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:39:18.551 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:39:18.551 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:39:18.551 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6823", 2021-02-10T11:39:18.551 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36365 2021-02-10T11:39:18.551 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:39:18.551 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:39:18.551 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:39:18.552 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6825", 2021-02-10T11:39:18.552 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36365 2021-02-10T11:39:18.552 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:39:18.552 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:39:18.552 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:39:18.552 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addrs": { 2021-02-10T11:39:18.552 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:39:18.552 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:39:18.552 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:39:18.552 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6830", 2021-02-10T11:39:18.552 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36365 2021-02-10T11:39:18.553 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:39:18.553 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:39:18.553 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:39:18.553 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6831", 2021-02-10T11:39:18.553 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36365 2021-02-10T11:39:18.553 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:39:18.553 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:39:18.553 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:39:18.554 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addrs": { 2021-02-10T11:39:18.554 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:39:18.554 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:39:18.554 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:39:18.554 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6827", 2021-02-10T11:39:18.554 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36365 2021-02-10T11:39:18.554 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:39:18.555 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:39:18.555 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:39:18.555 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6829", 2021-02-10T11:39:18.555 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36365 2021-02-10T11:39:18.555 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:39:18.555 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:39:18.555 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:39:18.555 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addr": "172.21.2.113:6821/36365", 2021-02-10T11:39:18.555 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addr": "172.21.2.113:6825/36365", 2021-02-10T11:39:18.555 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addr": "172.21.2.113:6831/36365", 2021-02-10T11:39:18.555 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addr": "172.21.2.113:6829/36365", 2021-02-10T11:39:18.555 INFO:teuthology.orchestra.run.gibba002.stdout: "state": [ 2021-02-10T11:39:18.555 INFO:teuthology.orchestra.run.gibba002.stdout: "exists", 2021-02-10T11:39:18.556 INFO:teuthology.orchestra.run.gibba002.stdout: "up" 2021-02-10T11:39:18.556 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:39:18.556 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:39:18.556 INFO:teuthology.orchestra.run.gibba002.stdout: ], 2021-02-10T11:39:18.556 INFO:teuthology.orchestra.run.gibba002.stdout: "osd_xinfo": [ 2021-02-10T11:39:18.556 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:39:18.556 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 0, 2021-02-10T11:39:18.556 INFO:teuthology.orchestra.run.gibba002.stdout: "down_stamp": "0.000000", 2021-02-10T11:39:18.556 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_probability": 0, 2021-02-10T11:39:18.556 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_interval": 0, 2021-02-10T11:39:18.556 INFO:teuthology.orchestra.run.gibba002.stdout: "features": 4611087854035861503, 2021-02-10T11:39:18.556 INFO:teuthology.orchestra.run.gibba002.stdout: "old_weight": 0 2021-02-10T11:39:18.556 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:39:18.556 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:39:18.557 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 1, 2021-02-10T11:39:18.557 INFO:teuthology.orchestra.run.gibba002.stdout: "down_stamp": "0.000000", 2021-02-10T11:39:18.557 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_probability": 0, 2021-02-10T11:39:18.557 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_interval": 0, 2021-02-10T11:39:18.557 INFO:teuthology.orchestra.run.gibba002.stdout: "features": 4611087854035861503, 2021-02-10T11:39:18.557 INFO:teuthology.orchestra.run.gibba002.stdout: "old_weight": 0 2021-02-10T11:39:18.557 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:39:18.557 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:39:18.557 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 2, 2021-02-10T11:39:18.557 INFO:teuthology.orchestra.run.gibba002.stdout: "down_stamp": "0.000000", 2021-02-10T11:39:18.557 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_probability": 0, 2021-02-10T11:39:18.557 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_interval": 0, 2021-02-10T11:39:18.557 INFO:teuthology.orchestra.run.gibba002.stdout: "features": 4611087854035861503, 2021-02-10T11:39:18.557 INFO:teuthology.orchestra.run.gibba002.stdout: "old_weight": 0 2021-02-10T11:39:18.558 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:39:18.558 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:39:18.558 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 3, 2021-02-10T11:39:18.558 INFO:teuthology.orchestra.run.gibba002.stdout: "down_stamp": "0.000000", 2021-02-10T11:39:18.558 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_probability": 0, 2021-02-10T11:39:18.558 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_interval": 0, 2021-02-10T11:39:18.558 INFO:teuthology.orchestra.run.gibba002.stdout: "features": 4611087854035861503, 2021-02-10T11:39:18.558 INFO:teuthology.orchestra.run.gibba002.stdout: "old_weight": 0 2021-02-10T11:39:18.558 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:39:18.558 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:39:18.558 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 4, 2021-02-10T11:39:18.558 INFO:teuthology.orchestra.run.gibba002.stdout: "down_stamp": "0.000000", 2021-02-10T11:39:18.558 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_probability": 0, 2021-02-10T11:39:18.559 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_interval": 0, 2021-02-10T11:39:18.559 INFO:teuthology.orchestra.run.gibba002.stdout: "features": 4611087854035861503, 2021-02-10T11:39:18.559 INFO:teuthology.orchestra.run.gibba002.stdout: "old_weight": 0 2021-02-10T11:39:18.559 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:39:18.559 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:39:18.559 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 5, 2021-02-10T11:39:18.559 INFO:teuthology.orchestra.run.gibba002.stdout: "down_stamp": "0.000000", 2021-02-10T11:39:18.559 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_probability": 0, 2021-02-10T11:39:18.559 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_interval": 0, 2021-02-10T11:39:18.559 INFO:teuthology.orchestra.run.gibba002.stdout: "features": 4611087854035861503, 2021-02-10T11:39:18.559 INFO:teuthology.orchestra.run.gibba002.stdout: "old_weight": 0 2021-02-10T11:39:18.559 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:39:18.559 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:39:18.559 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 6, 2021-02-10T11:39:18.560 INFO:teuthology.orchestra.run.gibba002.stdout: "down_stamp": "0.000000", 2021-02-10T11:39:18.560 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_probability": 0, 2021-02-10T11:39:18.560 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_interval": 0, 2021-02-10T11:39:18.560 INFO:teuthology.orchestra.run.gibba002.stdout: "features": 4611087854035861503, 2021-02-10T11:39:18.560 INFO:teuthology.orchestra.run.gibba002.stdout: "old_weight": 0 2021-02-10T11:39:18.560 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:39:18.560 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:39:18.560 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 7, 2021-02-10T11:39:18.560 INFO:teuthology.orchestra.run.gibba002.stdout: "down_stamp": "0.000000", 2021-02-10T11:39:18.560 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_probability": 0, 2021-02-10T11:39:18.560 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_interval": 0, 2021-02-10T11:39:18.560 INFO:teuthology.orchestra.run.gibba002.stdout: "features": 4611087854035861503, 2021-02-10T11:39:18.560 INFO:teuthology.orchestra.run.gibba002.stdout: "old_weight": 0 2021-02-10T11:39:18.560 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:39:18.561 INFO:teuthology.orchestra.run.gibba002.stdout: ], 2021-02-10T11:39:18.561 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_upmap": [], 2021-02-10T11:39:18.561 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_upmap_items": [], 2021-02-10T11:39:18.561 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_temp": [], 2021-02-10T11:39:18.561 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_temp": [], 2021-02-10T11:39:18.561 INFO:teuthology.orchestra.run.gibba002.stdout: "blacklist": {}, 2021-02-10T11:39:18.561 INFO:teuthology.orchestra.run.gibba002.stdout: "erasure_code_profiles": { 2021-02-10T11:39:18.561 INFO:teuthology.orchestra.run.gibba002.stdout: "default": { 2021-02-10T11:39:18.561 INFO:teuthology.orchestra.run.gibba002.stdout: "crush-failure-domain": "osd", 2021-02-10T11:39:18.561 INFO:teuthology.orchestra.run.gibba002.stdout: "k": "2", 2021-02-10T11:39:18.561 INFO:teuthology.orchestra.run.gibba002.stdout: "m": "1", 2021-02-10T11:39:18.561 INFO:teuthology.orchestra.run.gibba002.stdout: "plugin": "jerasure", 2021-02-10T11:39:18.561 INFO:teuthology.orchestra.run.gibba002.stdout: "ruleset-failure-domain": "osd", 2021-02-10T11:39:18.561 INFO:teuthology.orchestra.run.gibba002.stdout: "technique": "reed_sol_van" 2021-02-10T11:39:18.562 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:39:18.562 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:39:18.562 INFO:teuthology.orchestra.run.gibba002.stdout: "removed_snaps_queue": [], 2021-02-10T11:39:18.562 INFO:teuthology.orchestra.run.gibba002.stdout: "new_removed_snaps": [], 2021-02-10T11:39:18.562 INFO:teuthology.orchestra.run.gibba002.stdout: "new_purged_snaps": [], 2021-02-10T11:39:18.562 INFO:teuthology.orchestra.run.gibba002.stdout: "crush_node_flags": {}, 2021-02-10T11:39:18.562 INFO:teuthology.orchestra.run.gibba002.stdout: "device_class_flags": {} 2021-02-10T11:39:18.562 INFO:teuthology.orchestra.run.gibba002.stdout:} 2021-02-10T11:39:18.564 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:18.485 7f2e0cd0c700 1 -- 172.21.2.102:0/1124693840 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2dec0186c0 msgr2=0x7f2dec01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:18.564 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:18.485 7f2e0cd0c700 1 --2- 172.21.2.102:0/1124693840 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2dec0186c0 0x7f2dec01abc0 crc :-1 s=READY pgs=1397 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:18.564 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:18.485 7f2e0cd0c700 1 -- 172.21.2.102:0/1124693840 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2e080b4f30 msgr2=0x7f2e0818b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:18.564 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:18.485 7f2e0cd0c700 1 --2- 172.21.2.102:0/1124693840 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2e080b4f30 0x7f2e0818b9d0 secure :-1 s=READY pgs=3194 cs=0 l=1 rev1=1 rx=0x7f2df0006f20 tx=0x7f2df000b910).stop 2021-02-10T11:39:18.564 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:18.486 7f2e0cd0c700 1 -- 172.21.2.102:0/1124693840 shutdown_connections 2021-02-10T11:39:18.564 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:18.486 7f2e0cd0c700 1 --2- 172.21.2.102:0/1124693840 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2e08094960 0x7f2e08187e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:18.564 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:18.486 7f2e0cd0c700 1 --2- 172.21.2.102:0/1124693840 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2dec0186c0 0x7f2dec01abc0 unknown :-1 s=CLOSED pgs=1397 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:18.564 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:18.486 7f2e0cd0c700 1 --2- 172.21.2.102:0/1124693840 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2e08093f40 0x7f2e08094340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:18.564 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:18.486 7f2e0cd0c700 1 --2- 172.21.2.102:0/1124693840 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2e080b4f30 0x7f2e0818b9d0 unknown :-1 s=CLOSED pgs=3194 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:18.564 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:18.486 7f2e0cd0c700 1 -- 172.21.2.102:0/1124693840 >> 172.21.2.102:0/1124693840 conn(0x7f2e0819f8f0 msgr2=0x7f2e081a5f80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:18.564 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:18.486 7f2e0cd0c700 1 -- 172.21.2.102:0/1124693840 shutdown_connections 2021-02-10T11:39:18.564 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:18.486 7f2e0cd0c700 1 -- 172.21.2.102:0/1124693840 wait complete. 2021-02-10T11:39:18.565 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:39:18.996 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:18.976 7f92fa973700 1 Processor -- start 2021-02-10T11:39:18.996 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:18.976 7f92fa973700 1 -- start start 2021-02-10T11:39:18.996 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:18.976 7f92fa973700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f92f40d0340 0x7f92f40ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:18.996 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:18.976 7f92fa973700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92f40f0b10 con 0x7f92f40d1870 2021-02-10T11:39:18.996 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:18.976 7f92fa973700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92f41b0be0 con 0x7f92f40d1c70 2021-02-10T11:39:18.996 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:18.976 7f92fa973700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92f40e5ed0 con 0x7f92f40d0340 2021-02-10T11:39:18.996 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:18.977 7f92f3fff700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f92f40d1870 0x7f92f41b03f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:60626/0 (socket says 172.21.2.102:60626) 2021-02-10T11:39:18.996 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:18.977 7f92f3fff700 1 -- 172.21.2.102:0/4134270250 learned_addr learned my addr 172.21.2.102:0/4134270250 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:39:18.996 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:18.977 7f92f2ffd700 1 -- 172.21.2.102:0/4134270250 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3947366596 0 0) 0x7f92e40010b0 con 0x7f92f40d1870 2021-02-10T11:39:18.997 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:18.977 7f92f2ffd700 1 -- 172.21.2.102:0/4134270250 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92dc002fe0 con 0x7f92f40d1870 2021-02-10T11:39:18.997 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:18.977 7f92f2ffd700 1 -- 172.21.2.102:0/4134270250 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2725576918 0 0) 0x7f92e40010b0 con 0x7f92f40d1870 2021-02-10T11:39:18.997 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:18.977 7f92f37fe700 1 --2- 172.21.2.102:0/4134270250 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f92f40d0340 0x7f92f40ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:18.997 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:18.977 7f92f2ffd700 1 -- 172.21.2.102:0/4134270250 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f92f40d0340 msgr2=0x7f92f40ef640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:18.997 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:18.977 7f92f2ffd700 1 --2- 172.21.2.102:0/4134270250 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f92f40d0340 0x7f92f40ef640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:18.997 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:18.977 7f92f2ffd700 1 -- 172.21.2.102:0/4134270250 >> v1:172.21.2.113:6789/0 conn(0x7f92f40d1c70 legacy=0x7f92f40efc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:18.997 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:18.977 7f92f2ffd700 1 -- 172.21.2.102:0/4134270250 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92f40f0450 con 0x7f92f40d1870 2021-02-10T11:39:18.997 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:18.977 7f92f2ffd700 1 -- 172.21.2.102:0/4134270250 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7f92e40015b0 con 0x7f92f40d1870 2021-02-10T11:39:18.997 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:18.977 7f92f2ffd700 1 -- 172.21.2.102:0/4134270250 >> v1:172.21.2.102:6789/0 conn(0x7f92f40d1870 legacy=0x7f92f41b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:18.997 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:18.977 7f92f2ffd700 1 --2- 172.21.2.102:0/4134270250 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f92dc003b70 0x7f92dc006000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:18.997 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:18.977 7f92f2ffd700 1 --2- 172.21.2.102:0/4134270250 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f92dc006820 0x7f92dc008d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:18.997 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:18.977 7f92f2ffd700 1 --2- 172.21.2.102:0/4134270250 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f92dc0095c0 0x7f92dc00bac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:18.997 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:18.977 7f92f2ffd700 1 -- 172.21.2.102:0/4134270250 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92dc00c370 con 0x7f92dc0095c0 2021-02-10T11:39:18.997 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:18.977 7f92f2ffd700 1 -- 172.21.2.102:0/4134270250 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92dc00c590 con 0x7f92dc006820 2021-02-10T11:39:18.998 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:18.977 7f92f2ffd700 1 -- 172.21.2.102:0/4134270250 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92dc00c7b0 con 0x7f92dc003b70 2021-02-10T11:39:18.998 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:18.977 7f92f37fe700 1 --2- 172.21.2.102:0/4134270250 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f92f40d0340 0x7f92f40ef640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:39:18.998 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:18.978 7f92f8f90700 1 --2- 172.21.2.102:0/4134270250 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f92dc0095c0 0x7f92dc00bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:18.998 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:18.978 7f92f3fff700 1 --2- 172.21.2.102:0/4134270250 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f92dc003b70 0x7f92dc006000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:18.998 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:18.978 7f92f8f90700 1 -- 172.21.2.102:0/4134270250 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f92dc003b70 msgr2=0x7f92dc006000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:18.998 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:18.978 7f92f8f90700 1 --2- 172.21.2.102:0/4134270250 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f92dc003b70 0x7f92dc006000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:18.998 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:18.978 7f92f8f90700 1 -- 172.21.2.102:0/4134270250 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f92dc006820 msgr2=0x7f92dc008d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:18.998 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:18.978 7f92f8f90700 1 --2- 172.21.2.102:0/4134270250 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f92dc006820 0x7f92dc008d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:18.998 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:18.978 7f92f8f90700 1 -- 172.21.2.102:0/4134270250 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92dc00ca50 con 0x7f92dc0095c0 2021-02-10T11:39:18.998 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:18.978 7f92f8f90700 1 --2- 172.21.2.102:0/4134270250 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f92dc0095c0 0x7f92dc00bac0 secure :-1 s=READY pgs=3196 cs=0 l=1 rev1=1 rx=0x7f92ec006e30 tx=0x7f92ec04de60).ready entity=mon.0 client_cookie=9bd8f8ee941cad7e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:18.998 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:18.978 7f92f2ffd700 1 -- 172.21.2.102:0/4134270250 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f92ec006a50 con 0x7f92dc0095c0 2021-02-10T11:39:18.998 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:18.978 7f92f2ffd700 1 -- 172.21.2.102:0/4134270250 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92ec001400 con 0x7f92dc0095c0 2021-02-10T11:39:18.999 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:18.978 7f92f2ffd700 1 -- 172.21.2.102:0/4134270250 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f92ec0018e0 con 0x7f92dc0095c0 2021-02-10T11:39:18.999 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:18.978 7f92fa973700 1 -- 172.21.2.102:0/4134270250 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f92dc0095c0 msgr2=0x7f92dc00bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:18.999 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:18.978 7f92fa973700 1 --2- 172.21.2.102:0/4134270250 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f92dc0095c0 0x7f92dc00bac0 secure :-1 s=READY pgs=3196 cs=0 l=1 rev1=1 rx=0x7f92ec006e30 tx=0x7f92ec04de60).stop 2021-02-10T11:39:18.999 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:18.978 7f92f8f90700 1 -- 172.21.2.102:0/4134270250 reap_dead start 2021-02-10T11:39:18.999 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:18.979 7f92fa973700 1 -- 172.21.2.102:0/4134270250 shutdown_connections 2021-02-10T11:39:18.999 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:18.979 7f92fa973700 1 -- 172.21.2.102:0/4134270250 >> 172.21.2.102:0/4134270250 conn(0x7f92f40ce450 msgr2=0x7f92f40d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:18.999 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:18.979 7f92fa973700 1 -- 172.21.2.102:0/4134270250 shutdown_connections 2021-02-10T11:39:18.999 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:18.979 7f92fa973700 1 -- 172.21.2.102:0/4134270250 wait complete. 2021-02-10T11:39:18.999 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:18.994 7f92fa973700 1 Processor -- start 2021-02-10T11:39:18.999 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:18.994 7f92fa973700 1 -- start start 2021-02-10T11:39:19.000 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:18.994 7f92fa973700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f92f4195650 0x7f92f40b5560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:19.000 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:18.994 7f92fa973700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f92f40b5b80 0x7f92f41a2e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:19.001 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:18.994 7f92fa973700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f92f40acfa0 0x7f92f41a3440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:19.001 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:18.994 7f92fa973700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92f40b6010 con 0x7f92f4195650 2021-02-10T11:39:19.001 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:18.994 7f92fa973700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92f41a3a60 con 0x7f92f40b5b80 2021-02-10T11:39:19.001 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:18.994 7f92fa973700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92f41a3c40 con 0x7f92f40acfa0 2021-02-10T11:39:19.001 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:18.994 7f92f3fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f92f4195650 0x7f92f40b5560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:19.001 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:18.994 7f92f3fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f92f4195650 0x7f92f40b5560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:40182/0 (socket says 172.21.2.102:40182) 2021-02-10T11:39:19.001 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:18.994 7f92f3fff700 1 -- 172.21.2.102:0/483885424 learned_addr learned my addr 172.21.2.102:0/483885424 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:39:19.001 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:18.995 7f92f3fff700 1 -- 172.21.2.102:0/483885424 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f92f40acfa0 msgr2=0x7f92f41a3440 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:39:19.001 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:18.995 7f92f3fff700 1 --2- 172.21.2.102:0/483885424 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f92f40acfa0 0x7f92f41a3440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:19.001 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:18.995 7f92f3fff700 1 -- 172.21.2.102:0/483885424 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f92f40b5b80 msgr2=0x7f92f41a2e20 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:39:19.001 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:18.995 7f92f3fff700 1 --2- 172.21.2.102:0/483885424 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f92f40b5b80 0x7f92f41a2e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:19.001 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:18.995 7f92f3fff700 1 -- 172.21.2.102:0/483885424 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92f40bcae0 con 0x7f92f4195650 2021-02-10T11:39:19.001 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:18.995 7f92f3fff700 1 --2- 172.21.2.102:0/483885424 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f92f4195650 0x7f92f40b5560 secure :-1 s=READY pgs=3197 cs=0 l=1 rev1=1 rx=0x7f92e4001660 tx=0x7f92e400a930).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:19.002 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:18.995 7f92f0ff9700 1 -- 172.21.2.102:0/483885424 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f92e400ad20 con 0x7f92f4195650 2021-02-10T11:39:19.002 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:18.995 7f92f0ff9700 1 -- 172.21.2.102:0/483885424 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92e40016a0 con 0x7f92f4195650 2021-02-10T11:39:19.002 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:18.995 7f92fa973700 1 -- 172.21.2.102:0/483885424 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92f40bcae0 con 0x7f92f4195650 2021-02-10T11:39:19.002 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:18.995 7f92f0ff9700 1 -- 172.21.2.102:0/483885424 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f92e40043c0 con 0x7f92f4195650 2021-02-10T11:39:19.002 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:18.995 7f92fa973700 1 -- 172.21.2.102:0/483885424 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92f40bcae0 con 0x7f92f4195650 2021-02-10T11:39:19.002 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:18.996 7f92f0ff9700 1 -- 172.21.2.102:0/483885424 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f92e4004140 con 0x7f92f4195650 2021-02-10T11:39:19.002 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:18.996 7f92f0ff9700 1 --2- 172.21.2.102:0/483885424 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f92d40186c0 0x7f92d401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:19.002 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:18.996 7f92f0ff9700 1 -- 172.21.2.102:0/483885424 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(180..180 src has 1..180) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f92e4032530 con 0x7f92f4195650 2021-02-10T11:39:19.002 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:18.996 7f92f37fe700 1 --2- 172.21.2.102:0/483885424 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f92d40186c0 0x7f92d401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:19.002 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:18.996 7f92f37fe700 1 --2- 172.21.2.102:0/483885424 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f92d40186c0 0x7f92d401abc0 crc :-1 s=READY pgs=1398 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:19.029 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:19.026 7f92fa973700 1 -- 172.21.2.102:0/483885424 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92f40b6010 con 0x7f92f4195650 2021-02-10T11:39:19.031 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:19.029 7f92f0ff9700 1 -- 172.21.2.102:0/483885424 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f92e4025cd0 con 0x7f92f4195650 2021-02-10T11:39:19.160 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:19.157 7f92fa973700 1 -- 172.21.2.102:0/483885424 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f92f40b6010 con 0x7f92f4195650 2021-02-10T11:39:19.160 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:19.158 7f92f0ff9700 1 -- 172.21.2.102:0/483885424 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 158 v158) v1 ==== 95+0+2107 (secure 0 0 0) 0x7f92e4025cd0 con 0x7f92f4195650 2021-02-10T11:39:19.161 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:39:19.161 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":158,"default_fscid":19,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8815,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.102:6835/3396946217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6834","nonce":3396946217},{"type":"v1","addr":"172.21.2.102:6835","nonce":3396946217}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":153}],"filesystems":[{"mdsmap":{"epoch":157,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:38:30.335525","modified":"2021-02-10 11:38:32.341928","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8812},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8812":{"gid":8812,"name":"b","rank":0,"incarnation":154,"state":"up:active","state_seq":3,"addr":"172.21.2.113:6833/1513291768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6832","nonce":1513291768},{"type":"v1","addr":"172.21.2.113:6833","nonce":1513291768}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[39],"metadata_pool":38,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":19}]} 2021-02-10T11:39:19.161 INFO:teuthology.orchestra.run.gibba002.stderr:dumped fsmap epoch 158 2021-02-10T11:39:19.162 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:19.160 7f92fa973700 1 -- 172.21.2.102:0/483885424 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f92d40186c0 msgr2=0x7f92d401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:19.162 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:19.160 7f92fa973700 1 --2- 172.21.2.102:0/483885424 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f92d40186c0 0x7f92d401abc0 crc :-1 s=READY pgs=1398 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:19.162 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:19.160 7f92fa973700 1 -- 172.21.2.102:0/483885424 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f92f4195650 msgr2=0x7f92f40b5560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:19.162 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:19.160 7f92fa973700 1 --2- 172.21.2.102:0/483885424 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f92f4195650 0x7f92f40b5560 secure :-1 s=READY pgs=3197 cs=0 l=1 rev1=1 rx=0x7f92e4001660 tx=0x7f92e400a930).stop 2021-02-10T11:39:19.162 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:19.160 7f92fa973700 1 -- 172.21.2.102:0/483885424 shutdown_connections 2021-02-10T11:39:19.162 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:19.160 7f92fa973700 1 --2- 172.21.2.102:0/483885424 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f92f40acfa0 0x7f92f41a3440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:19.162 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:19.160 7f92fa973700 1 --2- 172.21.2.102:0/483885424 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f92d40186c0 0x7f92d401abc0 unknown :-1 s=CLOSED pgs=1398 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:19.162 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:19.160 7f92fa973700 1 --2- 172.21.2.102:0/483885424 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f92f40b5b80 0x7f92f41a2e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:19.163 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:19.160 7f92fa973700 1 --2- 172.21.2.102:0/483885424 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f92f4195650 0x7f92f40b5560 unknown :-1 s=CLOSED pgs=3197 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:19.163 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:19.160 7f92fa973700 1 -- 172.21.2.102:0/483885424 >> 172.21.2.102:0/483885424 conn(0x7f92f40b3040 msgr2=0x7f92f40c3250 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:19.163 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:19.160 7f92fa973700 1 -- 172.21.2.102:0/483885424 shutdown_connections 2021-02-10T11:39:19.163 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:19.160 7f92fa973700 1 -- 172.21.2.102:0/483885424 wait complete. 2021-02-10T11:39:19.171 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs fail cephfs 2021-02-10T11:39:19.600 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:19.582 7fd11678b700 1 Processor -- start 2021-02-10T11:39:19.601 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:19.582 7fd11678b700 1 -- start start 2021-02-10T11:39:19.601 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:19.583 7fd11678b700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fd1100d0340 0x7fd1100ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:19.601 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:19.583 7fd11678b700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fd1100d1c70 0x7fd1100efc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:19.601 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:19.583 7fd11678b700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1100f0b10 con 0x7fd1100d1870 2021-02-10T11:39:19.601 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:19.583 7fd11678b700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1100e5ed0 con 0x7fd1100d0340 2021-02-10T11:39:19.601 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:19.583 7fd11678b700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1101b0be0 con 0x7fd1100d1c70 2021-02-10T11:39:19.601 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:19.583 7fd10f7fe700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fd1100d0340 0x7fd1100ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:19.601 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:19.583 7fd10ffff700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fd1100d1870 0x7fd1101b03f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:60644/0 (socket says 172.21.2.102:60644) 2021-02-10T11:39:19.601 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:19.583 7fd10ffff700 1 -- 172.21.2.102:0/266994656 learned_addr learned my addr 172.21.2.102:0/266994656 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:39:19.601 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:19.583 7fd10f7fe700 1 -- 172.21.2.102:0/266994656 >> v2:172.21.2.113:3300/0 conn(0x7fd1100d1c70 msgr2=0x7fd1100efc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:19.601 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:19.583 7fd10f7fe700 1 --2- 172.21.2.102:0/266994656 >> v2:172.21.2.113:3300/0 conn(0x7fd1100d1c70 0x7fd1100efc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:19.601 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:19.583 7fd10f7fe700 1 -- 172.21.2.102:0/266994656 >> v1:172.21.2.102:6789/0 conn(0x7fd1100d1870 legacy=0x7fd1101b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:19.601 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:19.583 7fd10effd700 1 -- 172.21.2.102:0/266994656 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3299853962 0 0) 0x7fd1000009b0 con 0x7fd1100d1870 2021-02-10T11:39:19.602 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:19.583 7fd10f7fe700 1 -- 172.21.2.102:0/266994656 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1100f20a0 con 0x7fd1100d0340 2021-02-10T11:39:19.602 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:19.583 7fd10f7fe700 1 --2- 172.21.2.102:0/266994656 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd1100d0340 0x7fd1100ef640 secure :-1 s=READY pgs=3199 cs=0 l=1 rev1=1 rx=0x7fd104006650 tx=0x7fd104009a80).ready entity=mon.0 client_cookie=5a7c379db85f5116 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:19.602 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:19.583 7fd10effd700 1 -- 172.21.2.102:0/266994656 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd1040009b0 con 0x7fd1100d0340 2021-02-10T11:39:19.602 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:19.583 7fd10effd700 1 -- 172.21.2.102:0/266994656 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd10400c0c0 con 0x7fd1100d0340 2021-02-10T11:39:19.602 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:19.583 7fd10effd700 1 -- 172.21.2.102:0/266994656 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd10400c510 con 0x7fd1100d0340 2021-02-10T11:39:19.602 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:19.583 7fd11678b700 1 -- 172.21.2.102:0/266994656 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd1100d0340 msgr2=0x7fd1100ef640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:19.602 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:19.584 7fd11678b700 1 --2- 172.21.2.102:0/266994656 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd1100d0340 0x7fd1100ef640 secure :-1 s=READY pgs=3199 cs=0 l=1 rev1=1 rx=0x7fd104006650 tx=0x7fd104009a80).stop 2021-02-10T11:39:19.602 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:19.584 7fd11678b700 1 -- 172.21.2.102:0/266994656 shutdown_connections 2021-02-10T11:39:19.602 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:19.584 7fd11678b700 1 --2- 172.21.2.102:0/266994656 >> v2:172.21.2.113:3300/0 conn(0x7fd1100d1c70 0x7fd1100efc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:19.602 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:19.584 7fd11678b700 1 --2- 172.21.2.102:0/266994656 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd1100d0340 0x7fd1100ef640 unknown :-1 s=CLOSED pgs=3199 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:19.602 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:19.584 7fd11678b700 1 -- 172.21.2.102:0/266994656 >> 172.21.2.102:0/266994656 conn(0x7fd1100ce450 msgr2=0x7fd1100d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:19.602 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:19.584 7fd11678b700 1 -- 172.21.2.102:0/266994656 shutdown_connections 2021-02-10T11:39:19.602 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:19.584 7fd11678b700 1 -- 172.21.2.102:0/266994656 wait complete. 2021-02-10T11:39:19.604 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:19.599 7fd11678b700 1 Processor -- start 2021-02-10T11:39:19.604 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:19.599 7fd11678b700 1 -- start start 2021-02-10T11:39:19.604 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:19.599 7fd11678b700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd1100a7930 0x7fd1100be420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:19.604 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:19.599 7fd11678b700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd1100bb140 0x7fd1101a4420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:19.604 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:19.599 7fd11678b700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd1101a4a40 0x7fd1100bf0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:19.604 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:19.599 7fd11678b700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1100ba2c0 con 0x7fd1101a4a40 2021-02-10T11:39:19.604 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:19.599 7fd11678b700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1100bad20 con 0x7fd1100a7930 2021-02-10T11:39:19.604 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:19.599 7fd11678b700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1100baf40 con 0x7fd1100bb140 2021-02-10T11:39:19.605 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:19.599 7fd114da8700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd1101a4a40 0x7fd1100bf0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:19.605 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:19.599 7fd114da8700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd1101a4a40 0x7fd1100bf0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:40196/0 (socket says 172.21.2.102:40196) 2021-02-10T11:39:19.605 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:19.599 7fd114da8700 1 -- 172.21.2.102:0/629653911 learned_addr learned my addr 172.21.2.102:0/629653911 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:39:19.605 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:19.600 7fd114da8700 1 -- 172.21.2.102:0/629653911 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd1100bb140 msgr2=0x7fd1101a4420 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:39:19.605 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:19.600 7fd114da8700 1 --2- 172.21.2.102:0/629653911 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd1100bb140 0x7fd1101a4420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:19.605 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:19.600 7fd114da8700 1 -- 172.21.2.102:0/629653911 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd1100a7930 msgr2=0x7fd1100be420 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:39:19.605 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:19.600 7fd114da8700 1 --2- 172.21.2.102:0/629653911 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd1100a7930 0x7fd1100be420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:19.605 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:19.600 7fd114da8700 1 -- 172.21.2.102:0/629653911 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1100bb540 con 0x7fd1101a4a40 2021-02-10T11:39:19.605 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:19.600 7fd114da8700 1 --2- 172.21.2.102:0/629653911 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd1101a4a40 0x7fd1100bf0d0 secure :-1 s=READY pgs=3200 cs=0 l=1 rev1=1 rx=0x7fd108009980 tx=0x7fd10800bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:19.605 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:19.600 7fd10cff9700 1 -- 172.21.2.102:0/629653911 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd108005870 con 0x7fd1101a4a40 2021-02-10T11:39:19.605 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:19.600 7fd10cff9700 1 -- 172.21.2.102:0/629653911 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd108005af0 con 0x7fd1101a4a40 2021-02-10T11:39:19.605 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:19.600 7fd10cff9700 1 -- 172.21.2.102:0/629653911 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd108006030 con 0x7fd1101a4a40 2021-02-10T11:39:19.605 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:19.600 7fd11678b700 1 -- 172.21.2.102:0/629653911 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1100bb540 con 0x7fd1101a4a40 2021-02-10T11:39:19.605 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:19.600 7fd11678b700 1 -- 172.21.2.102:0/629653911 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1100bb540 con 0x7fd1101a4a40 2021-02-10T11:39:19.606 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:19.600 7fd10cff9700 1 -- 172.21.2.102:0/629653911 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd108005cd0 con 0x7fd1101a4a40 2021-02-10T11:39:19.606 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:19.601 7fd10cff9700 1 --2- 172.21.2.102:0/629653911 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd0f00186c0 0x7fd0f001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:19.606 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:19.601 7fd10cff9700 1 -- 172.21.2.102:0/629653911 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(180..180 src has 1..180) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fd108032900 con 0x7fd1101a4a40 2021-02-10T11:39:19.606 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:19.601 7fd10ffff700 1 --2- 172.21.2.102:0/629653911 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd0f00186c0 0x7fd0f001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:19.606 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:19.601 7fd10ffff700 1 --2- 172.21.2.102:0/629653911 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd0f00186c0 0x7fd0f001abc0 crc :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:19.635 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:19.633 7fd11678b700 1 -- 172.21.2.102:0/629653911 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1100bffc0 con 0x7fd1101a4a40 2021-02-10T11:39:19.637 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:19.635 7fd10cff9700 1 -- 172.21.2.102:0/629653911 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd10802df70 con 0x7fd1101a4a40 2021-02-10T11:39:19.765 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:19.763 7fd11678b700 1 -- 172.21.2.102:0/629653911 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fd1100ba2c0 con 0x7fd1101a4a40 2021-02-10T11:39:20.192 INFO:tasks.ceph.mon.a.gibba002.stderr:2021-02-10 11:39:20.191 7f8dd2755700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-02-10T11:39:20.197 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:20.195 7fd10cff9700 1 -- 172.21.2.102:0/629653911 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v159) v1 ==== 161+0+0 (secure 0 0 0) 0x7fd108006f20 con 0x7fd1101a4a40 2021-02-10T11:39:20.232 INFO:teuthology.orchestra.run.gibba002.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2021-02-10T11:39:20.234 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:20.232 7fd11678b700 1 -- 172.21.2.102:0/629653911 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd0f00186c0 msgr2=0x7fd0f001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:20.234 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:20.232 7fd11678b700 1 --2- 172.21.2.102:0/629653911 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd0f00186c0 0x7fd0f001abc0 crc :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:20.234 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:20.232 7fd11678b700 1 -- 172.21.2.102:0/629653911 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd1101a4a40 msgr2=0x7fd1100bf0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:20.234 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:20.232 7fd11678b700 1 --2- 172.21.2.102:0/629653911 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd1101a4a40 0x7fd1100bf0d0 secure :-1 s=READY pgs=3200 cs=0 l=1 rev1=1 rx=0x7fd108009980 tx=0x7fd10800bcd0).stop 2021-02-10T11:39:20.234 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:20.232 7fd11678b700 1 -- 172.21.2.102:0/629653911 shutdown_connections 2021-02-10T11:39:20.235 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:20.232 7fd11678b700 1 --2- 172.21.2.102:0/629653911 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd1101a4a40 0x7fd1100bf0d0 unknown :-1 s=CLOSED pgs=3200 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:20.235 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:20.232 7fd11678b700 1 --2- 172.21.2.102:0/629653911 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd1100bb140 0x7fd1101a4420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:20.235 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:20.232 7fd11678b700 1 --2- 172.21.2.102:0/629653911 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd0f00186c0 0x7fd0f001abc0 unknown :-1 s=CLOSED pgs=1399 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:20.235 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:20.232 7fd11678b700 1 --2- 172.21.2.102:0/629653911 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd1100a7930 0x7fd1100be420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:20.235 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:20.232 7fd11678b700 1 -- 172.21.2.102:0/629653911 >> 172.21.2.102:0/629653911 conn(0x7fd11018d1f0 msgr2=0x7fd110094c40 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:20.235 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:20.232 7fd11678b700 1 -- 172.21.2.102:0/629653911 shutdown_connections 2021-02-10T11:39:20.235 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:20.232 7fd11678b700 1 -- 172.21.2.102:0/629653911 wait complete. 2021-02-10T11:39:20.243 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:39:20.249 INFO:tasks.ceph.mds.b.gibba013.stdout:starting mds.b at 2021-02-10T11:39:20.671 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:20.652 7f3f54755700 1 Processor -- start 2021-02-10T11:39:20.671 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:20.652 7f3f54755700 1 -- start start 2021-02-10T11:39:20.671 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:20.653 7f3f54755700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f3f4c1b0470 0x7f3f4c1b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:20.671 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:20.653 7f3f54755700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f4c0f0b50 con 0x7f3f4c0f20e0 2021-02-10T11:39:20.671 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:20.653 7f3f54755700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f4c0f24e0 con 0x7f3f4c0ef610 2021-02-10T11:39:20.671 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:20.653 7f3f54755700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f4c0e5ea0 con 0x7f3f4c1b0470 2021-02-10T11:39:20.671 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:20.653 7f3f52571700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f3f4c1b0470 0x7f3f4c1b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:20.671 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:20.653 7f3f52571700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f3f4c1b0470 0x7f3f4c1b0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:40202/0 (socket says 172.21.2.102:40202) 2021-02-10T11:39:20.671 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:20.653 7f3f52571700 1 -- 172.21.2.102:0/171989423 learned_addr learned my addr 172.21.2.102:0/171989423 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:39:20.672 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:20.653 7f3f52d72700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f3f4c0f20e0 0x7f3f4c0f0200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:60658/0 (socket says 172.21.2.102:60658) 2021-02-10T11:39:20.672 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:20.653 7f3f52571700 1 -- 172.21.2.102:0/171989423 >> v1:172.21.2.113:6789/0 conn(0x7f3f4c0ef610 legacy=0x7f3f4c0efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:20.672 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:20.653 7f3f52571700 1 -- 172.21.2.102:0/171989423 >> v1:172.21.2.102:6789/0 conn(0x7f3f4c0f20e0 legacy=0x7f3f4c0f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:20.672 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:20.653 7f3f52571700 1 -- 172.21.2.102:0/171989423 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f4c0e52e0 con 0x7f3f4c1b0470 2021-02-10T11:39:20.672 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:20.654 7f3f52571700 1 --2- 172.21.2.102:0/171989423 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3f4c1b0470 0x7f3f4c1b0870 secure :-1 s=READY pgs=3203 cs=0 l=1 rev1=1 rx=0x7f3f40006700 tx=0x7f3f40009b80).ready entity=mon.0 client_cookie=d68347262f5eb128 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:20.672 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:20.654 7f3f5156f700 1 -- 172.21.2.102:0/171989423 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3f4000c0c0 con 0x7f3f4c1b0470 2021-02-10T11:39:20.672 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:20.654 7f3f5156f700 1 -- 172.21.2.102:0/171989423 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f4000c340 con 0x7f3f4c1b0470 2021-02-10T11:39:20.672 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:20.654 7f3f5156f700 1 -- 172.21.2.102:0/171989423 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3f4000c860 con 0x7f3f4c1b0470 2021-02-10T11:39:20.672 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:20.654 7f3f54755700 1 -- 172.21.2.102:0/171989423 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3f4c1b0470 msgr2=0x7f3f4c1b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:20.672 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:20.654 7f3f54755700 1 --2- 172.21.2.102:0/171989423 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3f4c1b0470 0x7f3f4c1b0870 secure :-1 s=READY pgs=3203 cs=0 l=1 rev1=1 rx=0x7f3f40006700 tx=0x7f3f40009b80).stop 2021-02-10T11:39:20.672 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:20.654 7f3f54755700 1 -- 172.21.2.102:0/171989423 shutdown_connections 2021-02-10T11:39:20.672 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:20.654 7f3f54755700 1 --2- 172.21.2.102:0/171989423 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3f4c1b0470 0x7f3f4c1b0870 unknown :-1 s=CLOSED pgs=3203 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:20.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:20.654 7f3f54755700 1 -- 172.21.2.102:0/171989423 >> 172.21.2.102:0/171989423 conn(0x7f3f4c0ce3e0 msgr2=0x7f3f4c0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:20.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:20.654 7f3f54755700 1 -- 172.21.2.102:0/171989423 shutdown_connections 2021-02-10T11:39:20.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:20.654 7f3f54755700 1 -- 172.21.2.102:0/171989423 wait complete. 2021-02-10T11:39:20.674 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:20.669 7f3f54755700 1 Processor -- start 2021-02-10T11:39:20.674 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:20.669 7f3f54755700 1 -- start start 2021-02-10T11:39:20.675 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:20.669 7f3f54755700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3f4c097170 0x7f3f4c0a71e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:20.675 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:20.669 7f3f54755700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3f4c0961b0 0x7f3f4c0965b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:20.675 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:20.669 7f3f54755700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3f4c096bd0 0x7f3f4c0a7e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:20.675 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:20.669 7f3f54755700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f4c0a8470 con 0x7f3f4c097170 2021-02-10T11:39:20.675 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:20.669 7f3f54755700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f4c0a8650 con 0x7f3f4c096bd0 2021-02-10T11:39:20.675 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:20.669 7f3f54755700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f4c0a8870 con 0x7f3f4c0961b0 2021-02-10T11:39:20.675 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:20.670 7f3f52571700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3f4c097170 0x7f3f4c0a71e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:20.675 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:20.670 7f3f52571700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3f4c097170 0x7f3f4c0a71e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:40206/0 (socket says 172.21.2.102:40206) 2021-02-10T11:39:20.675 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:20.670 7f3f52571700 1 -- 172.21.2.102:0/2063555402 learned_addr learned my addr 172.21.2.102:0/2063555402 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:39:20.675 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:20.670 7f3f52571700 1 -- 172.21.2.102:0/2063555402 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3f4c0961b0 msgr2=0x7f3f4c0965b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:20.675 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:20.670 7f3f52571700 1 --2- 172.21.2.102:0/2063555402 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3f4c0961b0 0x7f3f4c0965b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:20.675 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:20.670 7f3f52571700 1 -- 172.21.2.102:0/2063555402 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3f4c096bd0 msgr2=0x7f3f4c0a7e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:20.676 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:20.670 7f3f52571700 1 --2- 172.21.2.102:0/2063555402 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3f4c096bd0 0x7f3f4c0a7e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:20.676 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:20.670 7f3f52571700 1 -- 172.21.2.102:0/2063555402 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f4c0a8a90 con 0x7f3f4c097170 2021-02-10T11:39:20.676 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:20.670 7f3f52571700 1 --2- 172.21.2.102:0/2063555402 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3f4c097170 0x7f3f4c0a71e0 secure :-1 s=READY pgs=3204 cs=0 l=1 rev1=1 rx=0x7f3f4000b650 tx=0x7f3f4000b9f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:20.676 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:20.670 7f3f3effd700 1 -- 172.21.2.102:0/2063555402 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3f4000bde0 con 0x7f3f4c097170 2021-02-10T11:39:20.676 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:20.670 7f3f3effd700 1 -- 172.21.2.102:0/2063555402 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f40000c90 con 0x7f3f4c097170 2021-02-10T11:39:20.676 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:20.670 7f3f3effd700 1 -- 172.21.2.102:0/2063555402 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3f400022b0 con 0x7f3f4c097170 2021-02-10T11:39:20.676 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:20.670 7f3f54755700 1 -- 172.21.2.102:0/2063555402 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f4c0a8a90 con 0x7f3f4c097170 2021-02-10T11:39:20.676 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:20.671 7f3f54755700 1 -- 172.21.2.102:0/2063555402 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f4c0a9020 con 0x7f3f4c097170 2021-02-10T11:39:20.676 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:20.671 7f3f3effd700 1 -- 172.21.2.102:0/2063555402 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f3f40009f90 con 0x7f3f4c097170 2021-02-10T11:39:20.676 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:20.671 7f3f3effd700 1 --2- 172.21.2.102:0/2063555402 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3f340186c0 0x7f3f3401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:20.676 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:20.671 7f3f3effd700 1 -- 172.21.2.102:0/2063555402 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(181..181 src has 1..181) v4 ==== 5659+0+0 (secure 0 0 0) 0x7f3f400325b0 con 0x7f3f4c097170 2021-02-10T11:39:20.676 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:20.671 7f3f51d70700 1 --2- 172.21.2.102:0/2063555402 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3f340186c0 0x7f3f3401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:20.676 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:20.672 7f3f51d70700 1 --2- 172.21.2.102:0/2063555402 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3f340186c0 0x7f3f3401abc0 crc :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:20.705 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:20.703 7f3f54755700 1 -- 172.21.2.102:0/2063555402 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f4c0a7920 con 0x7f3f4c097170 2021-02-10T11:39:20.707 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:20.705 7f3f3effd700 1 -- 172.21.2.102:0/2063555402 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f3f400009b0 con 0x7f3f4c097170 2021-02-10T11:39:20.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:20.833 7f3f54755700 1 -- 172.21.2.102:0/2063555402 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3f4c0a7920 con 0x7f3f4c097170 2021-02-10T11:39:20.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:20.834 7f3f3effd700 1 -- 172.21.2.102:0/2063555402 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 159 v159) v1 ==== 95+0+1762 (secure 0 0 0) 0x7f3f4000bde0 con 0x7f3f4c097170 2021-02-10T11:39:20.837 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:39:20.837 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":159,"default_fscid":19,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8815,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.102:6835/3396946217","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6834","nonce":3396946217},{"type":"v1","addr":"172.21.2.102:6835","nonce":3396946217}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":153}],"filesystems":[{"mdsmap":{"epoch":159,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:38:30.335525","modified":"2021-02-10 11:39:20.191881","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":181,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[39],"metadata_pool":38,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":19}]} 2021-02-10T11:39:20.837 INFO:teuthology.orchestra.run.gibba002.stderr:dumped fsmap epoch 159 2021-02-10T11:39:20.838 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:20.836 7f3f54755700 1 -- 172.21.2.102:0/2063555402 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3f340186c0 msgr2=0x7f3f3401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:20.838 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:20.836 7f3f54755700 1 --2- 172.21.2.102:0/2063555402 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3f340186c0 0x7f3f3401abc0 crc :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:20.839 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:20.836 7f3f54755700 1 -- 172.21.2.102:0/2063555402 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3f4c097170 msgr2=0x7f3f4c0a71e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:20.839 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:20.836 7f3f54755700 1 --2- 172.21.2.102:0/2063555402 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3f4c097170 0x7f3f4c0a71e0 secure :-1 s=READY pgs=3204 cs=0 l=1 rev1=1 rx=0x7f3f4000b650 tx=0x7f3f4000b9f0).stop 2021-02-10T11:39:20.839 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:20.836 7f3f54755700 1 -- 172.21.2.102:0/2063555402 shutdown_connections 2021-02-10T11:39:20.839 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:20.836 7f3f54755700 1 --2- 172.21.2.102:0/2063555402 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3f340186c0 0x7f3f3401abc0 unknown :-1 s=CLOSED pgs=1400 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:20.839 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:20.836 7f3f54755700 1 --2- 172.21.2.102:0/2063555402 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3f4c096bd0 0x7f3f4c0a7e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:20.839 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:20.836 7f3f54755700 1 --2- 172.21.2.102:0/2063555402 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3f4c0961b0 0x7f3f4c0965b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:20.839 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:20.836 7f3f54755700 1 --2- 172.21.2.102:0/2063555402 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3f4c097170 0x7f3f4c0a71e0 unknown :-1 s=CLOSED pgs=3204 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:20.839 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:20.836 7f3f54755700 1 -- 172.21.2.102:0/2063555402 >> 172.21.2.102:0/2063555402 conn(0x7f3f4c0c3bd0 msgr2=0x7f3f4c089210 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:20.839 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:20.836 7f3f54755700 1 -- 172.21.2.102:0/2063555402 shutdown_connections 2021-02-10T11:39:20.839 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:20.836 7f3f54755700 1 -- 172.21.2.102:0/2063555402 wait complete. 2021-02-10T11:39:20.847 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2021-02-10T11:39:21.281 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:21.262 7f9e0c599700 1 Processor -- start 2021-02-10T11:39:21.281 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:21.262 7f9e0c599700 1 -- start start 2021-02-10T11:39:21.281 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:21.263 7f9e0c599700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f9e040d1870 0x7f9e041b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:21.282 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:21.263 7f9e0c599700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f9e041b0a10 0x7f9e040cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:21.282 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:21.263 7f9e0c599700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9e040d1c70 0x7f9e040cf950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:21.282 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:21.263 7f9e0c599700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e040e5ed0 con 0x7f9e041b0a10 2021-02-10T11:39:21.282 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:21.263 7f9e0c599700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e040f0b10 con 0x7f9e040d1870 2021-02-10T11:39:21.282 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:21.263 7f9e0c599700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e040cff70 con 0x7f9e040d1c70 2021-02-10T11:39:21.282 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:21.263 7f9e09bb4700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f9e041b0a10 0x7f9e040cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:21.282 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:21.263 7f9e09bb4700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f9e041b0a10 0x7f9e040cf330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:40214/0 (socket says 172.21.2.102:40214) 2021-02-10T11:39:21.282 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:21.263 7f9e09bb4700 1 -- 172.21.2.102:0/1711472657 learned_addr learned my addr 172.21.2.102:0/1711472657 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:39:21.282 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:21.263 7f9e0a3b5700 1 --2- 172.21.2.102:0/1711472657 >> v2:172.21.2.113:3300/0 conn(0x7f9e040d1870 0x7f9e041b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:21.282 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:21.263 7f9e09bb4700 1 -- 172.21.2.102:0/1711472657 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9e040d1c70 msgr2=0x7f9e040cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:21.282 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:21.263 7f9e09bb4700 1 --2- 172.21.2.102:0/1711472657 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9e040d1c70 0x7f9e040cf950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:21.282 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:21.263 7f9e09bb4700 1 -- 172.21.2.102:0/1711472657 >> v2:172.21.2.113:3300/0 conn(0x7f9e040d1870 msgr2=0x7f9e041b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:21.283 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:21.263 7f9e09bb4700 1 --2- 172.21.2.102:0/1711472657 >> v2:172.21.2.113:3300/0 conn(0x7f9e040d1870 0x7f9e041b03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:21.283 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:21.263 7f9e09bb4700 1 -- 172.21.2.102:0/1711472657 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e040d0150 con 0x7f9e041b0a10 2021-02-10T11:39:21.283 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:21.263 7f9e0a3b5700 1 --2- 172.21.2.102:0/1711472657 >> v2:172.21.2.113:3300/0 conn(0x7f9e040d1870 0x7f9e041b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:39:21.283 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:21.263 7f9e09bb4700 1 --2- 172.21.2.102:0/1711472657 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9e041b0a10 0x7f9e040cf330 secure :-1 s=READY pgs=3205 cs=0 l=1 rev1=1 rx=0x7f9df8006700 tx=0x7f9df8009b80).ready entity=mon.0 client_cookie=73844efb98ab45ef server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:21.283 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:21.264 7f9e093b3700 1 -- 172.21.2.102:0/1711472657 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9df800c0c0 con 0x7f9e041b0a10 2021-02-10T11:39:21.283 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:21.264 7f9e093b3700 1 -- 172.21.2.102:0/1711472657 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9df800c340 con 0x7f9e041b0a10 2021-02-10T11:39:21.283 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:21.264 7f9e093b3700 1 -- 172.21.2.102:0/1711472657 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9df800c860 con 0x7f9e041b0a10 2021-02-10T11:39:21.283 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:21.264 7f9e0c599700 1 -- 172.21.2.102:0/1711472657 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9e041b0a10 msgr2=0x7f9e040cf330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:21.283 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:21.264 7f9e0c599700 1 --2- 172.21.2.102:0/1711472657 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9e041b0a10 0x7f9e040cf330 secure :-1 s=READY pgs=3205 cs=0 l=1 rev1=1 rx=0x7f9df8006700 tx=0x7f9df8009b80).stop 2021-02-10T11:39:21.283 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:21.265 7f9e0c599700 1 -- 172.21.2.102:0/1711472657 shutdown_connections 2021-02-10T11:39:21.283 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:21.265 7f9e0c599700 1 --2- 172.21.2.102:0/1711472657 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9e040d1c70 0x7f9e040cf950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:21.283 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:21.265 7f9e0c599700 1 --2- 172.21.2.102:0/1711472657 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9e041b0a10 0x7f9e040cf330 unknown :-1 s=CLOSED pgs=3205 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:21.283 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:21.265 7f9e0c599700 1 --2- 172.21.2.102:0/1711472657 >> v2:172.21.2.113:3300/0 conn(0x7f9e040d1870 0x7f9e041b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:21.284 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:21.265 7f9e0c599700 1 -- 172.21.2.102:0/1711472657 >> 172.21.2.102:0/1711472657 conn(0x7f9e040ce450 msgr2=0x7f9e040d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:21.284 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:21.265 7f9e0c599700 1 -- 172.21.2.102:0/1711472657 shutdown_connections 2021-02-10T11:39:21.284 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:21.265 7f9e0c599700 1 -- 172.21.2.102:0/1711472657 wait complete. 2021-02-10T11:39:21.285 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:21.279 7f9e0c599700 1 Processor -- start 2021-02-10T11:39:21.285 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:21.279 7f9e0c599700 1 -- start start 2021-02-10T11:39:21.285 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:21.280 7f9e0c599700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f9e04188430 0x7f9e041a44c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:21.285 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:21.280 7f9e0c599700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9e04188830 0x7f9e04190c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:21.285 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:21.280 7f9e0c599700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9e041a4ae0 0x7f9e04191240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:21.286 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:21.280 7f9e0c599700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e04188d50 con 0x7f9e041a4ae0 2021-02-10T11:39:21.286 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:21.280 7f9e0c599700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e04191fa0 con 0x7f9e04188430 2021-02-10T11:39:21.286 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:21.280 7f9e0c599700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e041921c0 con 0x7f9e04188830 2021-02-10T11:39:21.286 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:21.280 7f9e0abb6700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9e041a4ae0 0x7f9e04191240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:21.286 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:21.280 7f9e0abb6700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9e041a4ae0 0x7f9e04191240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:40218/0 (socket says 172.21.2.102:40218) 2021-02-10T11:39:21.286 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:21.280 7f9e0abb6700 1 -- 172.21.2.102:0/329979390 learned_addr learned my addr 172.21.2.102:0/329979390 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:39:21.286 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:21.281 7f9e0abb6700 1 -- 172.21.2.102:0/329979390 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9e04188830 msgr2=0x7f9e04190c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:21.286 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:21.281 7f9e0abb6700 1 --2- 172.21.2.102:0/329979390 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9e04188830 0x7f9e04190c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:21.286 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:21.281 7f9e0abb6700 1 -- 172.21.2.102:0/329979390 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f9e04188430 msgr2=0x7f9e041a44c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:21.286 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:21.281 7f9e0abb6700 1 --2- 172.21.2.102:0/329979390 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f9e04188430 0x7f9e041a44c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:21.286 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:21.281 7f9e0abb6700 1 -- 172.21.2.102:0/329979390 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e041923e0 con 0x7f9e041a4ae0 2021-02-10T11:39:21.286 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:21.281 7f9e0abb6700 1 --2- 172.21.2.102:0/329979390 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9e041a4ae0 0x7f9e04191240 secure :-1 s=READY pgs=3206 cs=0 l=1 rev1=1 rx=0x7f9dfc009980 tx=0x7f9dfc00bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:21.286 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:21.281 7f9df6ffd700 1 -- 172.21.2.102:0/329979390 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9dfc003020 con 0x7f9e041a4ae0 2021-02-10T11:39:21.287 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:21.281 7f9df6ffd700 1 -- 172.21.2.102:0/329979390 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9dfc005a80 con 0x7f9e041a4ae0 2021-02-10T11:39:21.287 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:21.281 7f9df6ffd700 1 -- 172.21.2.102:0/329979390 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9dfc006030 con 0x7f9e041a4ae0 2021-02-10T11:39:21.287 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:21.281 7f9e0c599700 1 -- 172.21.2.102:0/329979390 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e041923e0 con 0x7f9e041a4ae0 2021-02-10T11:39:21.287 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:21.281 7f9e0c599700 1 -- 172.21.2.102:0/329979390 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e041923e0 con 0x7f9e041a4ae0 2021-02-10T11:39:21.287 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:21.282 7f9df6ffd700 1 -- 172.21.2.102:0/329979390 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f9dfc005ce0 con 0x7f9e041a4ae0 2021-02-10T11:39:21.287 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:21.282 7f9df6ffd700 1 --2- 172.21.2.102:0/329979390 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f9dec0186c0 0x7f9dec01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:21.287 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:21.282 7f9df6ffd700 1 -- 172.21.2.102:0/329979390 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(181..181 src has 1..181) v4 ==== 5659+0+0 (secure 0 0 0) 0x7f9dfc032bf0 con 0x7f9e041a4ae0 2021-02-10T11:39:21.287 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:21.282 7f9e0a3b5700 1 --2- 172.21.2.102:0/329979390 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f9dec0186c0 0x7f9dec01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:21.287 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:21.282 7f9e0a3b5700 1 --2- 172.21.2.102:0/329979390 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f9dec0186c0 0x7f9dec01abc0 crc :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:21.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:21.313 7f9e0c599700 1 -- 172.21.2.102:0/329979390 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e040ebc40 con 0x7f9e041a4ae0 2021-02-10T11:39:21.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:21.315 7f9df6ffd700 1 -- 172.21.2.102:0/329979390 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f9dfc003110 con 0x7f9e041a4ae0 2021-02-10T11:39:21.445 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:21.443 7f9e0c599700 1 -- 172.21.2.102:0/329979390 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f9e040ebc40 con 0x7f9e041a4ae0 2021-02-10T11:39:22.207 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:22.204 7f9df6ffd700 1 -- 172.21.2.102:0/329979390 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v160) v1 ==== 104+0+0 (secure 0 0 0) 0x7f9dfc005870 con 0x7f9e041a4ae0 2021-02-10T11:39:22.214 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:22.212 7f9e0c599700 1 -- 172.21.2.102:0/329979390 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f9dec0186c0 msgr2=0x7f9dec01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:22.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:22.212 7f9e0c599700 1 --2- 172.21.2.102:0/329979390 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f9dec0186c0 0x7f9dec01abc0 crc :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:22.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:22.212 7f9e0c599700 1 -- 172.21.2.102:0/329979390 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9e041a4ae0 msgr2=0x7f9e04191240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:22.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:22.212 7f9e0c599700 1 --2- 172.21.2.102:0/329979390 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9e041a4ae0 0x7f9e04191240 secure :-1 s=READY pgs=3206 cs=0 l=1 rev1=1 rx=0x7f9dfc009980 tx=0x7f9dfc00bcd0).stop 2021-02-10T11:39:22.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:22.212 7f9e0c599700 1 -- 172.21.2.102:0/329979390 shutdown_connections 2021-02-10T11:39:22.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:22.212 7f9e0c599700 1 --2- 172.21.2.102:0/329979390 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9e041a4ae0 0x7f9e04191240 unknown :-1 s=CLOSED pgs=3206 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:22.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:22.212 7f9e0c599700 1 --2- 172.21.2.102:0/329979390 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9e04188830 0x7f9e04190c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:22.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:22.212 7f9e0c599700 1 --2- 172.21.2.102:0/329979390 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f9dec0186c0 0x7f9dec01abc0 unknown :-1 s=CLOSED pgs=1401 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:22.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:22.212 7f9e0c599700 1 --2- 172.21.2.102:0/329979390 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f9e04188430 0x7f9e041a44c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:22.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:22.212 7f9e0c599700 1 -- 172.21.2.102:0/329979390 >> 172.21.2.102:0/329979390 conn(0x7f9e040c3130 msgr2=0x7f9e04095700 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:22.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:22.213 7f9e0c599700 1 -- 172.21.2.102:0/329979390 shutdown_connections 2021-02-10T11:39:22.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:22.213 7f9e0c599700 1 -- 172.21.2.102:0/329979390 wait complete. 2021-02-10T11:39:22.223 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool delete cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2021-02-10T11:39:22.656 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:22.636 7fb4860bc700 1 Processor -- start 2021-02-10T11:39:22.656 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:22.636 7fb4860bc700 1 -- start start 2021-02-10T11:39:22.656 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:22.637 7fb4860bc700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb4800d1c80 0x7fb4800cef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:22.656 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:22.637 7fb4860bc700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fb4800cf590 0x7fb4800cf990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:22.656 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:22.637 7fb4860bc700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4800fe1d0 con 0x7fb4800d1880 2021-02-10T11:39:22.656 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:22.637 7fb4860bc700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4801b0cc0 con 0x7fb4800cf590 2021-02-10T11:39:22.656 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:22.637 7fb4860bc700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb480109a70 con 0x7fb4800d1c80 2021-02-10T11:39:22.657 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:22.637 7fb47f7fe700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fb4800d1880 0x7fb4801b04d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:60682/0 (socket says 172.21.2.102:60682) 2021-02-10T11:39:22.657 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:22.637 7fb47f7fe700 1 -- 172.21.2.102:0/485831017 learned_addr learned my addr 172.21.2.102:0/485831017 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:39:22.657 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:22.637 7fb47effd700 1 -- 172.21.2.102:0/485831017 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2013435977 0 0) 0x7fb468000ce0 con 0x7fb4800d1880 2021-02-10T11:39:22.657 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:22.637 7fb47effd700 1 -- 172.21.2.102:0/485831017 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb46c002fe0 con 0x7fb4800d1880 2021-02-10T11:39:22.657 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:22.637 7fb47effd700 1 -- 172.21.2.102:0/485831017 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2244741089 0 0) 0x7fb4680010e0 con 0x7fb4800d1880 2021-02-10T11:39:22.657 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:22.637 7fb47ffff700 1 --2- 172.21.2.102:0/485831017 >> v2:172.21.2.113:3300/0 conn(0x7fb4800cf590 0x7fb4800cf990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:22.657 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:22.637 7fb47effd700 1 -- 172.21.2.102:0/485831017 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb4800d1c80 msgr2=0x7fb4800cef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:22.657 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:22.637 7fb476ffd700 1 --2- 172.21.2.102:0/485831017 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb4800d1c80 0x7fb4800cef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:22.657 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:22.638 7fb47effd700 1 --2- 172.21.2.102:0/485831017 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb4800d1c80 0x7fb4800cef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:22.657 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:22.638 7fb47effd700 1 -- 172.21.2.102:0/485831017 >> v2:172.21.2.113:3300/0 conn(0x7fb4800cf590 msgr2=0x7fb4800cf990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:22.657 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:22.638 7fb47effd700 1 --2- 172.21.2.102:0/485831017 >> v2:172.21.2.113:3300/0 conn(0x7fb4800cf590 0x7fb4800cf990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:22.657 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:22.638 7fb47effd700 1 -- 172.21.2.102:0/485831017 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb480088070 con 0x7fb4800d1880 2021-02-10T11:39:22.657 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:22.638 7fb47effd700 1 -- 172.21.2.102:0/485831017 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7fb468001660 con 0x7fb4800d1880 2021-02-10T11:39:22.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:22.638 7fb47ffff700 1 --2- 172.21.2.102:0/485831017 >> v2:172.21.2.113:3300/0 conn(0x7fb4800cf590 0x7fb4800cf990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:39:22.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:22.638 7fb476ffd700 1 --2- 172.21.2.102:0/485831017 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb4800d1c80 0x7fb4800cef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:39:22.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:22.638 7fb47effd700 1 -- 172.21.2.102:0/485831017 >> v1:172.21.2.102:6789/0 conn(0x7fb4800d1880 legacy=0x7fb4801b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:22.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:22.638 7fb47effd700 1 --2- 172.21.2.102:0/485831017 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb46c003a60 0x7fb46c005ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:22.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:22.638 7fb47effd700 1 --2- 172.21.2.102:0/485831017 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb46c006790 0x7fb46c008c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:22.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:22.638 7fb47effd700 1 --2- 172.21.2.102:0/485831017 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb46c009520 0x7fb46c00ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:22.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:22.638 7fb47effd700 1 -- 172.21.2.102:0/485831017 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb46c00c2d0 con 0x7fb46c009520 2021-02-10T11:39:22.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:22.638 7fb47effd700 1 -- 172.21.2.102:0/485831017 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb46c00c4f0 con 0x7fb46c003a60 2021-02-10T11:39:22.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:22.638 7fb47effd700 1 -- 172.21.2.102:0/485831017 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb46c0506b0 con 0x7fb46c006790 2021-02-10T11:39:22.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:22.638 7fb47ffff700 1 --2- 172.21.2.102:0/485831017 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb46c009520 0x7fb46c00ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:22.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:22.638 7fb47f7fe700 1 --2- 172.21.2.102:0/485831017 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb46c003a60 0x7fb46c005ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:22.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:22.638 7fb476ffd700 1 --2- 172.21.2.102:0/485831017 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb46c006790 0x7fb46c008c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:22.659 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:22.638 7fb47ffff700 1 -- 172.21.2.102:0/485831017 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb46c006790 msgr2=0x7fb46c008c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:22.659 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:22.638 7fb47ffff700 1 --2- 172.21.2.102:0/485831017 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb46c006790 0x7fb46c008c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:22.659 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:22.638 7fb47ffff700 1 -- 172.21.2.102:0/485831017 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb46c003a60 msgr2=0x7fb46c005ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:22.659 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:22.638 7fb47ffff700 1 --2- 172.21.2.102:0/485831017 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb46c003a60 0x7fb46c005ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:22.659 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:22.638 7fb47ffff700 1 -- 172.21.2.102:0/485831017 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb46c050950 con 0x7fb46c009520 2021-02-10T11:39:22.659 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:22.638 7fb47ffff700 1 --2- 172.21.2.102:0/485831017 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb46c009520 0x7fb46c00ba20 secure :-1 s=READY pgs=3208 cs=0 l=1 rev1=1 rx=0x7fb478006ce0 tx=0x7fb47800b300).ready entity=mon.0 client_cookie=973b1a83f5a99cd3 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:22.659 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:22.638 7fb47effd700 1 -- 172.21.2.102:0/485831017 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb47800b6f0 con 0x7fb46c009520 2021-02-10T11:39:22.659 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:22.639 7fb47effd700 1 -- 172.21.2.102:0/485831017 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb478003030 con 0x7fb46c009520 2021-02-10T11:39:22.659 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:22.639 7fb47effd700 1 -- 172.21.2.102:0/485831017 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb47800bbf0 con 0x7fb46c009520 2021-02-10T11:39:22.659 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:22.639 7fb4860bc700 1 -- 172.21.2.102:0/485831017 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb46c009520 msgr2=0x7fb46c00ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:22.659 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:22.639 7fb4860bc700 1 --2- 172.21.2.102:0/485831017 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb46c009520 0x7fb46c00ba20 secure :-1 s=READY pgs=3208 cs=0 l=1 rev1=1 rx=0x7fb478006ce0 tx=0x7fb47800b300).stop 2021-02-10T11:39:22.659 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:22.639 7fb47ffff700 1 -- 172.21.2.102:0/485831017 reap_dead start 2021-02-10T11:39:22.659 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:22.639 7fb4860bc700 1 -- 172.21.2.102:0/485831017 shutdown_connections 2021-02-10T11:39:22.660 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:22.639 7fb4860bc700 1 -- 172.21.2.102:0/485831017 >> 172.21.2.102:0/485831017 conn(0x7fb4800ce430 msgr2=0x7fb4800d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:22.660 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:22.639 7fb4860bc700 1 -- 172.21.2.102:0/485831017 shutdown_connections 2021-02-10T11:39:22.660 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:22.639 7fb4860bc700 1 -- 172.21.2.102:0/485831017 wait complete. 2021-02-10T11:39:22.660 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:22.654 7fb4860bc700 1 Processor -- start 2021-02-10T11:39:22.660 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:22.654 7fb4860bc700 1 -- start start 2021-02-10T11:39:22.661 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:22.654 7fb4860bc700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb480188400 0x7fb480188800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:22.661 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:22.654 7fb4860bc700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb480188e20 0x7fb480189220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:22.661 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:22.654 7fb4860bc700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb48009cbc0 0x7fb48009cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:22.661 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:22.654 7fb4860bc700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb480187e80 con 0x7fb480188e20 2021-02-10T11:39:22.661 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:22.654 7fb4860bc700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4800b27c0 con 0x7fb480188400 2021-02-10T11:39:22.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:22.654 7fb4860bc700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4800b29a0 con 0x7fb48009cbc0 2021-02-10T11:39:22.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:22.654 7fb476ffd700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb480188e20 0x7fb480189220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:22.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:22.655 7fb476ffd700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb480188e20 0x7fb480189220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:40238/0 (socket says 172.21.2.102:40238) 2021-02-10T11:39:22.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:22.655 7fb476ffd700 1 -- 172.21.2.102:0/1286329254 learned_addr learned my addr 172.21.2.102:0/1286329254 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:39:22.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:22.655 7fb476ffd700 1 -- 172.21.2.102:0/1286329254 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb48009cbc0 msgr2=0x7fb48009cfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:22.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:22.655 7fb476ffd700 1 --2- 172.21.2.102:0/1286329254 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb48009cbc0 0x7fb48009cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:22.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:22.655 7fb476ffd700 1 -- 172.21.2.102:0/1286329254 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb480188400 msgr2=0x7fb480188800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:22.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:22.655 7fb476ffd700 1 --2- 172.21.2.102:0/1286329254 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb480188400 0x7fb480188800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:22.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:22.655 7fb476ffd700 1 -- 172.21.2.102:0/1286329254 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb48009d5e0 con 0x7fb480188e20 2021-02-10T11:39:22.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:22.655 7fb476ffd700 1 --2- 172.21.2.102:0/1286329254 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb480188e20 0x7fb480189220 secure :-1 s=READY pgs=3209 cs=0 l=1 rev1=1 rx=0x7fb470006f10 tx=0x7fb47000b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:22.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:22.655 7fb47cff9700 1 -- 172.21.2.102:0/1286329254 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb47000bd80 con 0x7fb480188e20 2021-02-10T11:39:22.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:22.655 7fb47cff9700 1 -- 172.21.2.102:0/1286329254 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb470000d20 con 0x7fb480188e20 2021-02-10T11:39:22.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:22.655 7fb47cff9700 1 -- 172.21.2.102:0/1286329254 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb470002220 con 0x7fb480188e20 2021-02-10T11:39:22.663 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:22.655 7fb4860bc700 1 -- 172.21.2.102:0/1286329254 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb48009d5e0 con 0x7fb480188e20 2021-02-10T11:39:22.663 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:22.655 7fb4860bc700 1 -- 172.21.2.102:0/1286329254 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb48009db70 con 0x7fb480188e20 2021-02-10T11:39:22.663 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:22.656 7fb47cff9700 1 -- 172.21.2.102:0/1286329254 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fb470009fe0 con 0x7fb480188e20 2021-02-10T11:39:22.663 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:22.656 7fb47cff9700 1 --2- 172.21.2.102:0/1286329254 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb4640186c0 0x7fb46401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:22.663 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:22.656 7fb47cff9700 1 -- 172.21.2.102:0/1286329254 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(181..181 src has 1..181) v4 ==== 5659+0+0 (secure 0 0 0) 0x7fb470032440 con 0x7fb480188e20 2021-02-10T11:39:22.663 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:22.656 7fb47f7fe700 1 --2- 172.21.2.102:0/1286329254 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb4640186c0 0x7fb46401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:22.663 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:22.657 7fb47f7fe700 1 --2- 172.21.2.102:0/1286329254 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb4640186c0 0x7fb46401abc0 crc :-1 s=READY pgs=1402 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:22.688 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:22.686 7fb4860bc700 1 -- 172.21.2.102:0/1286329254 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb480082a70 con 0x7fb480188e20 2021-02-10T11:39:22.691 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:22.689 7fb47cff9700 1 -- 172.21.2.102:0/1286329254 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fb470000f00 con 0x7fb480188e20 2021-02-10T11:39:22.819 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:22.817 7fb4860bc700 1 -- 172.21.2.102:0/1286329254 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"pool2": "cephfs_metadata", "prefix": "osd pool delete", "pool": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fb480082a70 con 0x7fb480188e20 2021-02-10T11:39:23.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:23.209 7fb47cff9700 1 -- 172.21.2.102:0/1286329254 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"pool2": "cephfs_metadata", "prefix": "osd pool delete", "pool": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v182) v1 ==== 185+0+0 (secure 0 0 0) 0x7fb470000940 con 0x7fb480188e20 2021-02-10T11:39:23.236 INFO:teuthology.orchestra.run.gibba002.stderr:pool 'cephfs_metadata' removed 2021-02-10T11:39:23.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:23.235 7fb4860bc700 1 -- 172.21.2.102:0/1286329254 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb4640186c0 msgr2=0x7fb46401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:23.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:23.235 7fb4860bc700 1 --2- 172.21.2.102:0/1286329254 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb4640186c0 0x7fb46401abc0 crc :-1 s=READY pgs=1402 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:23.238 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:23.235 7fb4860bc700 1 -- 172.21.2.102:0/1286329254 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb480188e20 msgr2=0x7fb480189220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:23.238 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:23.235 7fb4860bc700 1 --2- 172.21.2.102:0/1286329254 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb480188e20 0x7fb480189220 secure :-1 s=READY pgs=3209 cs=0 l=1 rev1=1 rx=0x7fb470006f10 tx=0x7fb47000b990).stop 2021-02-10T11:39:23.238 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:23.235 7fb4860bc700 1 -- 172.21.2.102:0/1286329254 shutdown_connections 2021-02-10T11:39:23.238 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:23.235 7fb4860bc700 1 --2- 172.21.2.102:0/1286329254 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb48009cbc0 0x7fb48009cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:23.238 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:23.235 7fb4860bc700 1 --2- 172.21.2.102:0/1286329254 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb480188e20 0x7fb480189220 unknown :-1 s=CLOSED pgs=3209 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:23.238 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:23.235 7fb4860bc700 1 --2- 172.21.2.102:0/1286329254 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb4640186c0 0x7fb46401abc0 unknown :-1 s=CLOSED pgs=1402 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:23.238 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:23.235 7fb4860bc700 1 --2- 172.21.2.102:0/1286329254 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb480188400 0x7fb480188800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:23.238 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:23.235 7fb4860bc700 1 -- 172.21.2.102:0/1286329254 >> 172.21.2.102:0/1286329254 conn(0x7fb4800c2cd0 msgr2=0x7fb4800b5950 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:23.239 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:23.236 7fb4860bc700 1 -- 172.21.2.102:0/1286329254 shutdown_connections 2021-02-10T11:39:23.239 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:23.236 7fb4860bc700 1 -- 172.21.2.102:0/1286329254 wait complete. 2021-02-10T11:39:23.246 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool delete cephfs_data cephfs_data --yes-i-really-really-mean-it 2021-02-10T11:39:23.680 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:23.660 7f1b0dc4b700 1 Processor -- start 2021-02-10T11:39:23.680 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:23.660 7f1b0dc4b700 1 -- start start 2021-02-10T11:39:23.680 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:23.661 7f1b0dc4b700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1b0810b000 0x7f1b080cf760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:23.680 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:23.661 7f1b0dc4b700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b080fe1d0 con 0x7f1b080d1880 2021-02-10T11:39:23.680 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:23.661 7f1b0dc4b700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b081b0cc0 con 0x7f1b080d1c80 2021-02-10T11:39:23.680 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:23.661 7f1b0dc4b700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b08109a70 con 0x7f1b0810b000 2021-02-10T11:39:23.680 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:23.661 7f1b077fe700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f1b080d1880 0x7f1b081b04d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:60700/0 (socket says 172.21.2.102:60700) 2021-02-10T11:39:23.680 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:23.661 7f1b077fe700 1 -- 172.21.2.102:0/2213257412 learned_addr learned my addr 172.21.2.102:0/2213257412 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:39:23.681 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:23.661 7f1b07fff700 1 --2- 172.21.2.102:0/2213257412 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1b0810b000 0x7f1b080cf760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:23.681 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:23.661 7f1b067fc700 1 -- 172.21.2.102:0/2213257412 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1549319614 0 0) 0x7f1af80010b0 con 0x7f1b080d1880 2021-02-10T11:39:23.681 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:23.662 7f1b067fc700 1 -- 172.21.2.102:0/2213257412 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1af0002fe0 con 0x7f1b080d1880 2021-02-10T11:39:23.681 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:23.662 7f1b067fc700 1 -- 172.21.2.102:0/2213257412 <== mon.1 v1:172.21.2.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3187031176 0 0) 0x7f1afc000ff0 con 0x7f1b080d1c80 2021-02-10T11:39:23.681 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:23.662 7f1b067fc700 1 -- 172.21.2.102:0/2213257412 --> v1:172.21.2.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1af0003210 con 0x7f1b080d1c80 2021-02-10T11:39:23.681 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:23.662 7f1b067fc700 1 -- 172.21.2.102:0/2213257412 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2923121762 0 0) 0x7f1af80010b0 con 0x7f1b080d1880 2021-02-10T11:39:23.681 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:23.662 7f1b07fff700 1 -- 172.21.2.102:0/2213257412 >> v1:172.21.2.113:6789/0 conn(0x7f1b080d1c80 legacy=0x7f1b080cef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:23.681 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:23.662 7f1b07fff700 1 -- 172.21.2.102:0/2213257412 >> v1:172.21.2.102:6789/0 conn(0x7f1b080d1880 legacy=0x7f1b081b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:23.681 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:23.662 7f1b07fff700 1 -- 172.21.2.102:0/2213257412 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b0810b400 con 0x7f1b0810b000 2021-02-10T11:39:23.681 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:23.662 7f1b07fff700 1 --2- 172.21.2.102:0/2213257412 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1b0810b000 0x7f1b080cf760 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 rx=0x7f1b00006e50 tx=0x7f1b00009df0).ready entity=mon.2 client_cookie=d137161baa3fdd8a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:23.681 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:23.663 7f1b067fc700 1 -- 172.21.2.102:0/2213257412 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1b0000b510 con 0x7f1b0810b000 2021-02-10T11:39:23.681 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:23.663 7f1b067fc700 1 -- 172.21.2.102:0/2213257412 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b00003030 con 0x7f1b0810b000 2021-02-10T11:39:23.682 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:23.663 7f1b067fc700 1 -- 172.21.2.102:0/2213257412 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1b0000b7d0 con 0x7f1b0810b000 2021-02-10T11:39:23.682 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:23.663 7f1b0dc4b700 1 -- 172.21.2.102:0/2213257412 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1b0810b000 msgr2=0x7f1b080cf760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:23.682 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:23.663 7f1b0dc4b700 1 --2- 172.21.2.102:0/2213257412 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1b0810b000 0x7f1b080cf760 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 rx=0x7f1b00006e50 tx=0x7f1b00009df0).stop 2021-02-10T11:39:23.682 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:23.664 7f1b0dc4b700 1 -- 172.21.2.102:0/2213257412 shutdown_connections 2021-02-10T11:39:23.682 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:23.664 7f1b0dc4b700 1 --2- 172.21.2.102:0/2213257412 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1b0810b000 0x7f1b080cf760 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:23.682 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:23.664 7f1b0dc4b700 1 -- 172.21.2.102:0/2213257412 >> 172.21.2.102:0/2213257412 conn(0x7f1b080ce430 msgr2=0x7f1b080d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:23.682 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:23.664 7f1b0dc4b700 1 -- 172.21.2.102:0/2213257412 shutdown_connections 2021-02-10T11:39:23.682 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:23.664 7f1b0dc4b700 1 -- 172.21.2.102:0/2213257412 wait complete. 2021-02-10T11:39:23.683 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:23.678 7f1b0dc4b700 1 Processor -- start 2021-02-10T11:39:23.684 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:23.678 7f1b0dc4b700 1 -- start start 2021-02-10T11:39:23.684 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:23.679 7f1b0dc4b700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1b08188400 0x7f1b08188800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:23.684 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:23.679 7f1b0dc4b700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1b08188e20 0x7f1b08189220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:23.684 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:23.679 7f1b0dc4b700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1b0809cbc0 0x7f1b0809cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:23.684 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:23.679 7f1b0dc4b700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b08187e80 con 0x7f1b08188400 2021-02-10T11:39:23.684 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:23.679 7f1b0dc4b700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b080b27c0 con 0x7f1b08188e20 2021-02-10T11:39:23.684 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:23.679 7f1b0dc4b700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b080b29a0 con 0x7f1b0809cbc0 2021-02-10T11:39:23.684 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:23.679 7f1b077fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1b08188400 0x7f1b08188800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:23.684 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:23.679 7f1b077fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1b08188400 0x7f1b08188800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:40250/0 (socket says 172.21.2.102:40250) 2021-02-10T11:39:23.684 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:23.679 7f1b077fe700 1 -- 172.21.2.102:0/2826799065 learned_addr learned my addr 172.21.2.102:0/2826799065 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:39:23.684 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:23.679 7f1b077fe700 1 -- 172.21.2.102:0/2826799065 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1b0809cbc0 msgr2=0x7f1b0809cfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:23.685 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:23.679 7f1b077fe700 1 --2- 172.21.2.102:0/2826799065 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1b0809cbc0 0x7f1b0809cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:23.685 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:23.679 7f1b077fe700 1 -- 172.21.2.102:0/2826799065 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1b08188e20 msgr2=0x7f1b08189220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:23.685 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:23.679 7f1b077fe700 1 --2- 172.21.2.102:0/2826799065 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1b08188e20 0x7f1b08189220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:23.685 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:23.679 7f1b077fe700 1 -- 172.21.2.102:0/2826799065 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b0809d5e0 con 0x7f1b08188400 2021-02-10T11:39:23.685 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:23.679 7f1b077fe700 1 --2- 172.21.2.102:0/2826799065 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1b08188400 0x7f1b08188800 secure :-1 s=READY pgs=3211 cs=0 l=1 rev1=1 rx=0x7f1af8001c60 tx=0x7f1af800a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:23.685 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:23.680 7f1aeffff700 1 -- 172.21.2.102:0/2826799065 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1af800acc0 con 0x7f1b08188400 2021-02-10T11:39:23.685 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:23.680 7f1aeffff700 1 -- 172.21.2.102:0/2826799065 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1af8004140 con 0x7f1b08188400 2021-02-10T11:39:23.685 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:23.680 7f1b0dc4b700 1 -- 172.21.2.102:0/2826799065 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b0809d5e0 con 0x7f1b08188400 2021-02-10T11:39:23.685 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:23.680 7f1aeffff700 1 -- 172.21.2.102:0/2826799065 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1af8004660 con 0x7f1b08188400 2021-02-10T11:39:23.685 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:23.680 7f1b0dc4b700 1 -- 172.21.2.102:0/2826799065 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b0809db70 con 0x7f1b08188400 2021-02-10T11:39:23.685 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:23.680 7f1aeffff700 1 -- 172.21.2.102:0/2826799065 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1af8004320 con 0x7f1b08188400 2021-02-10T11:39:23.685 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:23.680 7f1aeffff700 1 --2- 172.21.2.102:0/2826799065 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1ae80186d0 0x7f1ae801abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:23.685 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:23.680 7f1aeffff700 1 -- 172.21.2.102:0/2826799065 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(182..182 src has 1..182) v4 ==== 5221+0+0 (secure 0 0 0) 0x7f1af8032630 con 0x7f1b08188400 2021-02-10T11:39:23.686 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:23.681 7f1b06ffd700 1 --2- 172.21.2.102:0/2826799065 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1ae80186d0 0x7f1ae801abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:23.686 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:23.681 7f1b06ffd700 1 --2- 172.21.2.102:0/2826799065 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1ae80186d0 0x7f1ae801abd0 crc :-1 s=READY pgs=1403 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:23.713 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:23.711 7f1b0dc4b700 1 -- 172.21.2.102:0/2826799065 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b08082a70 con 0x7f1b08188400 2021-02-10T11:39:23.715 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:23.713 7f1aeffff700 1 -- 172.21.2.102:0/2826799065 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f1af8000e20 con 0x7f1b08188400 2021-02-10T11:39:23.843 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:23.840 7f1b0dc4b700 1 -- 172.21.2.102:0/2826799065 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"pool2": "cephfs_data", "prefix": "osd pool delete", "pool": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f1b08082a70 con 0x7f1b08188400 2021-02-10T11:39:24.216 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:24.213 7f1aeffff700 1 -- 172.21.2.102:0/2826799065 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"pool2": "cephfs_data", "prefix": "osd pool delete", "pool": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v183) v1 ==== 173+0+0 (secure 0 0 0) 0x7f1af8004140 con 0x7f1b08188400 2021-02-10T11:39:24.260 INFO:teuthology.orchestra.run.gibba002.stderr:pool 'cephfs_data' removed 2021-02-10T11:39:24.262 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:24.259 7f1b0dc4b700 1 -- 172.21.2.102:0/2826799065 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1ae80186d0 msgr2=0x7f1ae801abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:24.262 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:24.259 7f1b0dc4b700 1 --2- 172.21.2.102:0/2826799065 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1ae80186d0 0x7f1ae801abd0 crc :-1 s=READY pgs=1403 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:24.262 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:24.259 7f1b0dc4b700 1 -- 172.21.2.102:0/2826799065 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1b08188400 msgr2=0x7f1b08188800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:24.262 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:24.259 7f1b0dc4b700 1 --2- 172.21.2.102:0/2826799065 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1b08188400 0x7f1b08188800 secure :-1 s=READY pgs=3211 cs=0 l=1 rev1=1 rx=0x7f1af8001c60 tx=0x7f1af800a8d0).stop 2021-02-10T11:39:24.262 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:24.260 7f1b0dc4b700 1 -- 172.21.2.102:0/2826799065 shutdown_connections 2021-02-10T11:39:24.262 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:24.260 7f1b0dc4b700 1 --2- 172.21.2.102:0/2826799065 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1b0809cbc0 0x7f1b0809cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:24.262 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:24.260 7f1b0dc4b700 1 --2- 172.21.2.102:0/2826799065 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1ae80186d0 0x7f1ae801abd0 unknown :-1 s=CLOSED pgs=1403 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:24.262 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:24.260 7f1b0dc4b700 1 --2- 172.21.2.102:0/2826799065 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1b08188e20 0x7f1b08189220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:24.262 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:24.260 7f1b0dc4b700 1 --2- 172.21.2.102:0/2826799065 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1b08188400 0x7f1b08188800 unknown :-1 s=CLOSED pgs=3211 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:24.262 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:24.260 7f1b0dc4b700 1 -- 172.21.2.102:0/2826799065 >> 172.21.2.102:0/2826799065 conn(0x7f1b080c2cd0 msgr2=0x7f1b080b5950 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:24.263 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:24.260 7f1b0dc4b700 1 -- 172.21.2.102:0/2826799065 shutdown_connections 2021-02-10T11:39:24.263 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:24.260 7f1b0dc4b700 1 -- 172.21.2.102:0/2826799065 wait complete. 2021-02-10T11:39:24.271 INFO:tasks.ceph.mds.a:Restarting daemon 2021-02-10T11:39:24.271 INFO:tasks.ceph.mds.a:Stopping old one... 2021-02-10T11:39:24.271 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2021-02-10T11:39:24.271 INFO:teuthology.orchestra.run:waiting for 300 2021-02-10T11:39:24.271 INFO:tasks.ceph.mds.b:Restarting daemon 2021-02-10T11:39:24.272 INFO:tasks.ceph.mds.b:Stopping old one... 2021-02-10T11:39:24.272 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2021-02-10T11:39:24.272 INFO:teuthology.orchestra.run:waiting for 300 2021-02-10T11:39:24.277 INFO:tasks.ceph.mds.a:Stopped 2021-02-10T11:39:24.277 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2021-02-10T11:39:24.278 INFO:tasks.ceph.mds.b:Stopped 2021-02-10T11:39:24.278 DEBUG:teuthology.orchestra.run.gibba013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2021-02-10T11:39:24.319 INFO:tasks.ceph.mds.a:Started 2021-02-10T11:39:24.320 DEBUG:teuthology.parallel:result is None 2021-02-10T11:39:24.320 INFO:tasks.ceph.mds.b:Started 2021-02-10T11:39:24.320 DEBUG:teuthology.parallel:result is None 2021-02-10T11:39:24.321 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd blacklist clear 2021-02-10T11:39:24.725 INFO:tasks.ceph.mds.b.gibba013.stdout:starting mds.b at 2021-02-10T11:39:24.729 INFO:tasks.ceph.mds.a.gibba002.stdout:starting mds.a at 2021-02-10T11:39:24.748 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:24.729 7fd2b2176700 1 Processor -- start 2021-02-10T11:39:24.748 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:24.729 7fd2b2176700 1 -- start start 2021-02-10T11:39:24.748 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:24.730 7fd2b2176700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fd2ac0d1870 0x7fd2ac1b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:24.748 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:24.730 7fd2b2176700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd2ac0f0650 0x7fd2ac0cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:24.748 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:24.730 7fd2b2176700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2ac0f0b10 con 0x7fd2ac1b0a10 2021-02-10T11:39:24.748 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:24.730 7fd2b2176700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2ac0e5ed0 con 0x7fd2ac0d1870 2021-02-10T11:39:24.748 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:24.730 7fd2b2176700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2ac0d1c70 con 0x7fd2ac0f0650 2021-02-10T11:39:24.748 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:24.730 7fd2aaffd700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fd2ac1b0a10 0x7fd2ac0cf330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:60724/0 (socket says 172.21.2.102:60724) 2021-02-10T11:39:24.749 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:24.730 7fd2aaffd700 1 -- 172.21.2.102:0/407117149 learned_addr learned my addr 172.21.2.102:0/407117149 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:39:24.749 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:24.730 7fd2abfff700 1 --2- 172.21.2.102:0/407117149 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd2ac0f0650 0x7fd2ac0cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:24.749 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:24.730 7fd2ab7fe700 1 --2- 172.21.2.102:0/407117149 >> v2:172.21.2.113:3300/0 conn(0x7fd2ac0d1870 0x7fd2ac1b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:24.749 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:24.730 7fd2aa7fc700 1 -- 172.21.2.102:0/407117149 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 816560187 0 0) 0x7fd2a0001060 con 0x7fd2ac1b0a10 2021-02-10T11:39:24.749 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:24.730 7fd2abfff700 1 -- 172.21.2.102:0/407117149 >> v2:172.21.2.113:3300/0 conn(0x7fd2ac0d1870 msgr2=0x7fd2ac1b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:24.749 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:24.730 7fd2abfff700 1 --2- 172.21.2.102:0/407117149 >> v2:172.21.2.113:3300/0 conn(0x7fd2ac0d1870 0x7fd2ac1b03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:24.749 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:24.730 7fd2abfff700 1 -- 172.21.2.102:0/407117149 >> v1:172.21.2.102:6789/0 conn(0x7fd2ac1b0a10 legacy=0x7fd2ac0cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:24.749 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:24.730 7fd2abfff700 1 -- 172.21.2.102:0/407117149 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2ac0d1e90 con 0x7fd2ac0f0650 2021-02-10T11:39:24.749 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:24.730 7fd2ab7fe700 1 --2- 172.21.2.102:0/407117149 >> v2:172.21.2.113:3300/0 conn(0x7fd2ac0d1870 0x7fd2ac1b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:39:24.749 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:24.731 7fd2abfff700 1 --2- 172.21.2.102:0/407117149 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd2ac0f0650 0x7fd2ac0cfb20 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 rx=0x7fd2a4006ad0 tx=0x7fd2a4009ad0).ready entity=mon.2 client_cookie=32fec2619b1ccc4a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:24.749 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:24.731 7fd2aa7fc700 1 -- 172.21.2.102:0/407117149 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd2a40022e0 con 0x7fd2ac0f0650 2021-02-10T11:39:24.749 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:24.731 7fd2aa7fc700 1 -- 172.21.2.102:0/407117149 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2a4006410 con 0x7fd2ac0f0650 2021-02-10T11:39:24.750 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:24.731 7fd2aa7fc700 1 -- 172.21.2.102:0/407117149 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd2a4002620 con 0x7fd2ac0f0650 2021-02-10T11:39:24.750 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:24.731 7fd2b2176700 1 -- 172.21.2.102:0/407117149 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd2ac0f0650 msgr2=0x7fd2ac0cfb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:24.750 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:24.731 7fd2b2176700 1 --2- 172.21.2.102:0/407117149 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd2ac0f0650 0x7fd2ac0cfb20 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 rx=0x7fd2a4006ad0 tx=0x7fd2a4009ad0).stop 2021-02-10T11:39:24.750 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:24.731 7fd2b2176700 1 -- 172.21.2.102:0/407117149 shutdown_connections 2021-02-10T11:39:24.750 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:24.731 7fd2b2176700 1 --2- 172.21.2.102:0/407117149 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd2ac0f0650 0x7fd2ac0cfb20 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:24.750 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:24.731 7fd2b2176700 1 --2- 172.21.2.102:0/407117149 >> v2:172.21.2.113:3300/0 conn(0x7fd2ac0d1870 0x7fd2ac1b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:24.750 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:24.731 7fd2b2176700 1 -- 172.21.2.102:0/407117149 >> 172.21.2.102:0/407117149 conn(0x7fd2ac0ce450 msgr2=0x7fd2ac0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:24.750 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:24.731 7fd2b2176700 1 -- 172.21.2.102:0/407117149 shutdown_connections 2021-02-10T11:39:24.750 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:24.731 7fd2b2176700 1 -- 172.21.2.102:0/407117149 wait complete. 2021-02-10T11:39:24.752 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:24.746 7fd2b2176700 1 Processor -- start 2021-02-10T11:39:24.752 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:24.746 7fd2b2176700 1 -- start start 2021-02-10T11:39:24.752 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:24.747 7fd2b2176700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd2ac0c1760 0x7fd2ac0c1b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:24.752 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:24.747 7fd2b2176700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd2ac0c2180 0x7fd2ac0bc940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:24.752 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:24.747 7fd2b2176700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd2ac0bcf60 0x7fd2ac0bd360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:24.752 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:24.747 7fd2b2176700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2ac0bc440 con 0x7fd2ac0c2180 2021-02-10T11:39:24.752 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:24.747 7fd2b2176700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2ac0bc060 con 0x7fd2ac0bcf60 2021-02-10T11:39:24.752 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:24.747 7fd2b2176700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2ac0c2ca0 con 0x7fd2ac0c1760 2021-02-10T11:39:24.752 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:24.747 7fd2aaffd700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd2ac0c2180 0x7fd2ac0bc940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:24.752 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:24.747 7fd2aaffd700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd2ac0c2180 0x7fd2ac0bc940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:40274/0 (socket says 172.21.2.102:40274) 2021-02-10T11:39:24.752 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:24.747 7fd2aaffd700 1 -- 172.21.2.102:0/16399093 learned_addr learned my addr 172.21.2.102:0/16399093 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:39:24.752 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:24.747 7fd2ab7fe700 1 --2- 172.21.2.102:0/16399093 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd2ac0c1760 0x7fd2ac0c1b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:24.753 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:24.747 7fd2abfff700 1 --2- 172.21.2.102:0/16399093 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd2ac0bcf60 0x7fd2ac0bd360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:24.753 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:24.747 7fd2aaffd700 1 -- 172.21.2.102:0/16399093 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd2ac0c1760 msgr2=0x7fd2ac0c1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:24.753 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:24.747 7fd2aaffd700 1 --2- 172.21.2.102:0/16399093 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd2ac0c1760 0x7fd2ac0c1b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:24.753 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:24.747 7fd2aaffd700 1 -- 172.21.2.102:0/16399093 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd2ac0bcf60 msgr2=0x7fd2ac0bd360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:24.753 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:24.747 7fd2aaffd700 1 --2- 172.21.2.102:0/16399093 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd2ac0bcf60 0x7fd2ac0bd360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:24.753 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:24.747 7fd2aaffd700 1 -- 172.21.2.102:0/16399093 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2ac0c27b0 con 0x7fd2ac0c2180 2021-02-10T11:39:24.753 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:24.748 7fd2aaffd700 1 --2- 172.21.2.102:0/16399093 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd2ac0c2180 0x7fd2ac0bc940 secure :-1 s=READY pgs=3214 cs=0 l=1 rev1=1 rx=0x7fd2a000a940 tx=0x7fd2a000ace0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:24.753 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:24.748 7fd293fff700 1 -- 172.21.2.102:0/16399093 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd2a0007020 con 0x7fd2ac0c2180 2021-02-10T11:39:24.753 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:24.748 7fd293fff700 1 -- 172.21.2.102:0/16399093 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2a0001250 con 0x7fd2ac0c2180 2021-02-10T11:39:24.753 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:24.748 7fd2b2176700 1 -- 172.21.2.102:0/16399093 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2ac0c27b0 con 0x7fd2ac0c2180 2021-02-10T11:39:24.753 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:24.748 7fd2b2176700 1 -- 172.21.2.102:0/16399093 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2ac0c27b0 con 0x7fd2ac0c2180 2021-02-10T11:39:24.753 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:24.748 7fd293fff700 1 -- 172.21.2.102:0/16399093 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd2a0004720 con 0x7fd2ac0c2180 2021-02-10T11:39:24.753 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:24.748 7fd293fff700 1 -- 172.21.2.102:0/16399093 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd2a000ae20 con 0x7fd2ac0c2180 2021-02-10T11:39:24.754 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:24.748 7fd293fff700 1 --2- 172.21.2.102:0/16399093 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd28c0186d0 0x7fd28c01abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:24.754 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:24.748 7fd293fff700 1 -- 172.21.2.102:0/16399093 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(183..183 src has 1..183) v4 ==== 4839+0+0 (secure 0 0 0) 0x7fd2a0032780 con 0x7fd2ac0c2180 2021-02-10T11:39:24.754 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:24.748 7fd2ab7fe700 1 --2- 172.21.2.102:0/16399093 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd28c0186d0 0x7fd28c01abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:24.754 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:24.749 7fd2ab7fe700 1 --2- 172.21.2.102:0/16399093 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd28c0186d0 0x7fd28c01abd0 crc :-1 s=READY pgs=1404 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:24.781 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:24.779 7fd2b2176700 1 -- 172.21.2.102:0/16399093 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2ac0ebc90 con 0x7fd2ac0c2180 2021-02-10T11:39:24.783 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:24.781 7fd293fff700 1 -- 172.21.2.102:0/16399093 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd2a0001250 con 0x7fd2ac0c2180 2021-02-10T11:39:24.920 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:24.918 7fd2b2176700 1 -- 172.21.2.102:0/16399093 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd blacklist clear"} v 0) v1 -- 0x7fd2ac0f0b10 con 0x7fd2ac0c2180 2021-02-10T11:39:25.257 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:25.255 7fd293fff700 1 -- 172.21.2.102:0/16399093 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd blacklist clear"}]=0 removed all blacklist entries v184) v1 ==== 97+0+0 (secure 0 0 0) 0x7fd2a0001250 con 0x7fd2ac0c2180 2021-02-10T11:39:25.286 INFO:teuthology.orchestra.run.gibba002.stderr: removed all blacklist entries 2021-02-10T11:39:25.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:25.286 7fd2b2176700 1 -- 172.21.2.102:0/16399093 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd28c0186d0 msgr2=0x7fd28c01abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:25.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:25.286 7fd2b2176700 1 --2- 172.21.2.102:0/16399093 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd28c0186d0 0x7fd28c01abd0 crc :-1 s=READY pgs=1404 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:25.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:25.286 7fd2b2176700 1 -- 172.21.2.102:0/16399093 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd2ac0c2180 msgr2=0x7fd2ac0bc940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:25.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:25.286 7fd2b2176700 1 --2- 172.21.2.102:0/16399093 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd2ac0c2180 0x7fd2ac0bc940 secure :-1 s=READY pgs=3214 cs=0 l=1 rev1=1 rx=0x7fd2a000a940 tx=0x7fd2a000ace0).stop 2021-02-10T11:39:25.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:25.286 7fd2b2176700 1 -- 172.21.2.102:0/16399093 shutdown_connections 2021-02-10T11:39:25.289 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:25.286 7fd2b2176700 1 --2- 172.21.2.102:0/16399093 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd28c0186d0 0x7fd28c01abd0 unknown :-1 s=CLOSED pgs=1404 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:25.289 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:25.286 7fd2b2176700 1 --2- 172.21.2.102:0/16399093 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd2ac0bcf60 0x7fd2ac0bd360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:25.289 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:25.286 7fd2b2176700 1 --2- 172.21.2.102:0/16399093 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd2ac0c2180 0x7fd2ac0bc940 unknown :-1 s=CLOSED pgs=3214 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:25.289 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:25.286 7fd2b2176700 1 --2- 172.21.2.102:0/16399093 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd2ac0c1760 0x7fd2ac0c1b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:25.289 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:25.286 7fd2b2176700 1 -- 172.21.2.102:0/16399093 >> 172.21.2.102:0/16399093 conn(0x7fd2ac0aeb90 msgr2=0x7fd2ac0af100 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:25.289 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:25.286 7fd2b2176700 1 -- 172.21.2.102:0/16399093 shutdown_connections 2021-02-10T11:39:25.289 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:25.286 7fd2b2176700 1 -- 172.21.2.102:0/16399093 wait complete. 2021-02-10T11:39:25.296 INFO:tasks.cephfs.cephfs_test_case:['0', '1', '2', '3'] 2021-02-10T11:39:25.297 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth ls --format=json-pretty 2021-02-10T11:39:25.726 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:25.707 7f2642547700 1 Processor -- start 2021-02-10T11:39:25.726 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:25.707 7f2642547700 1 -- start start 2021-02-10T11:39:25.726 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:25.708 7f2642547700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f263c0d0340 0x7f263c0ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:25.726 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:25.708 7f2642547700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f263c0d1c70 0x7f263c0efc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:25.726 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:25.708 7f2642547700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f263c0f0b10 con 0x7f263c0d1870 2021-02-10T11:39:25.726 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:25.708 7f2642547700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f263c0e5ed0 con 0x7f263c0d0340 2021-02-10T11:39:25.726 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:25.708 7f2642547700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f263c1b0be0 con 0x7f263c0d1c70 2021-02-10T11:39:25.727 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:25.708 7f263b7fe700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f263c0d0340 0x7f263c0ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:25.727 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:25.708 7f263bfff700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f263c0d1870 0x7f263c1b03f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:60738/0 (socket says 172.21.2.102:60738) 2021-02-10T11:39:25.727 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:25.708 7f263bfff700 1 -- 172.21.2.102:0/4276569888 learned_addr learned my addr 172.21.2.102:0/4276569888 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:39:25.727 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:25.708 7f263b7fe700 1 -- 172.21.2.102:0/4276569888 >> v2:172.21.2.113:3300/0 conn(0x7f263c0d1c70 msgr2=0x7f263c0efc60 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:39:25.727 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:25.708 7f263b7fe700 1 --2- 172.21.2.102:0/4276569888 >> v2:172.21.2.113:3300/0 conn(0x7f263c0d1c70 0x7f263c0efc60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:25.727 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:25.708 7f263b7fe700 1 -- 172.21.2.102:0/4276569888 >> v1:172.21.2.102:6789/0 conn(0x7f263c0d1870 legacy=0x7f263c1b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:25.727 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:25.708 7f263b7fe700 1 -- 172.21.2.102:0/4276569888 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f263c0f20a0 con 0x7f263c0d0340 2021-02-10T11:39:25.727 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:25.708 7f263b7fe700 1 --2- 172.21.2.102:0/4276569888 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f263c0d0340 0x7f263c0ef640 secure :-1 s=READY pgs=3216 cs=0 l=1 rev1=1 rx=0x7f2630006650 tx=0x7f2630009a80).ready entity=mon.0 client_cookie=8e88af6a25485063 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:25.727 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:25.709 7f263affd700 1 -- 172.21.2.102:0/4276569888 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f263000c0c0 con 0x7f263c0d0340 2021-02-10T11:39:25.727 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:25.709 7f263affd700 1 -- 172.21.2.102:0/4276569888 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f263000c340 con 0x7f263c0d0340 2021-02-10T11:39:25.727 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:25.709 7f263affd700 1 -- 172.21.2.102:0/4276569888 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f263000c860 con 0x7f263c0d0340 2021-02-10T11:39:25.727 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:25.709 7f2642547700 1 -- 172.21.2.102:0/4276569888 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f263c0d0340 msgr2=0x7f263c0ef640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:25.728 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:25.709 7f2642547700 1 --2- 172.21.2.102:0/4276569888 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f263c0d0340 0x7f263c0ef640 secure :-1 s=READY pgs=3216 cs=0 l=1 rev1=1 rx=0x7f2630006650 tx=0x7f2630009a80).stop 2021-02-10T11:39:25.728 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:25.709 7f2642547700 1 -- 172.21.2.102:0/4276569888 shutdown_connections 2021-02-10T11:39:25.728 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:25.709 7f2642547700 1 --2- 172.21.2.102:0/4276569888 >> v2:172.21.2.113:3300/0 conn(0x7f263c0d1c70 0x7f263c0efc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:25.728 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:25.709 7f2642547700 1 --2- 172.21.2.102:0/4276569888 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f263c0d0340 0x7f263c0ef640 unknown :-1 s=CLOSED pgs=3216 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:25.728 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:25.709 7f2642547700 1 -- 172.21.2.102:0/4276569888 >> 172.21.2.102:0/4276569888 conn(0x7f263c0ce450 msgr2=0x7f263c0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:25.728 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:25.709 7f2642547700 1 -- 172.21.2.102:0/4276569888 shutdown_connections 2021-02-10T11:39:25.728 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:25.709 7f2642547700 1 -- 172.21.2.102:0/4276569888 wait complete. 2021-02-10T11:39:25.728 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:25.724 7f2642547700 1 Processor -- start 2021-02-10T11:39:25.728 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:25.724 7f2642547700 1 -- start start 2021-02-10T11:39:25.729 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:25.724 7f2642547700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f263c18f0a0 0x7f263c196ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:25.729 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:25.724 7f2642547700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f263c1970d0 0x7f263c1974d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:25.730 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:25.724 7f2642547700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f263c094650 0x7f263c094a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:25.730 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:25.724 7f2642547700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f263c095bd0 con 0x7f263c18f0a0 2021-02-10T11:39:25.730 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:25.724 7f2642547700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f263c0ad7c0 con 0x7f263c1970d0 2021-02-10T11:39:25.730 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:25.724 7f2642547700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f263c18ec70 con 0x7f263c094650 2021-02-10T11:39:25.730 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:25.724 7f263bfff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f263c18f0a0 0x7f263c196ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:25.730 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:25.724 7f263bfff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f263c18f0a0 0x7f263c196ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:40288/0 (socket says 172.21.2.102:40288) 2021-02-10T11:39:25.730 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:25.724 7f263bfff700 1 -- 172.21.2.102:0/1718327558 learned_addr learned my addr 172.21.2.102:0/1718327558 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:39:25.730 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:25.725 7f263bfff700 1 -- 172.21.2.102:0/1718327558 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f263c094650 msgr2=0x7f263c094a50 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:39:25.730 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:25.725 7f263bfff700 1 --2- 172.21.2.102:0/1718327558 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f263c094650 0x7f263c094a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:25.730 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:25.725 7f263bfff700 1 -- 172.21.2.102:0/1718327558 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f263c1970d0 msgr2=0x7f263c1974d0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:39:25.730 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:25.725 7f263bfff700 1 --2- 172.21.2.102:0/1718327558 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f263c1970d0 0x7f263c1974d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:25.730 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:25.725 7f263bfff700 1 -- 172.21.2.102:0/1718327558 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f263c095070 con 0x7f263c18f0a0 2021-02-10T11:39:25.731 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:25.725 7f263bfff700 1 --2- 172.21.2.102:0/1718327558 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f263c18f0a0 0x7f263c196ab0 secure :-1 s=READY pgs=3217 cs=0 l=1 rev1=1 rx=0x7f262c001410 tx=0x7f262c00a950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:25.731 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:25.725 7f2638ff9700 1 -- 172.21.2.102:0/1718327558 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f262c00ad40 con 0x7f263c18f0a0 2021-02-10T11:39:25.731 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:25.725 7f2638ff9700 1 -- 172.21.2.102:0/1718327558 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f262c000c90 con 0x7f263c18f0a0 2021-02-10T11:39:25.731 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:25.725 7f2638ff9700 1 -- 172.21.2.102:0/1718327558 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f262c004590 con 0x7f263c18f0a0 2021-02-10T11:39:25.731 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:25.725 7f2642547700 1 -- 172.21.2.102:0/1718327558 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f263c095070 con 0x7f263c18f0a0 2021-02-10T11:39:25.731 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:25.725 7f2642547700 1 -- 172.21.2.102:0/1718327558 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f263c095070 con 0x7f263c18f0a0 2021-02-10T11:39:25.731 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:25.726 7f2638ff9700 1 -- 172.21.2.102:0/1718327558 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f262c0043a0 con 0x7f263c18f0a0 2021-02-10T11:39:25.731 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:25.726 7f2638ff9700 1 --2- 172.21.2.102:0/1718327558 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f261c0186c0 0x7f261c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:25.731 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:25.726 7f2638ff9700 1 -- 172.21.2.102:0/1718327558 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(184..184 src has 1..184) v4 ==== 4753+0+0 (secure 0 0 0) 0x7f262c032100 con 0x7f263c18f0a0 2021-02-10T11:39:25.731 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:25.726 7f263b7fe700 1 --2- 172.21.2.102:0/1718327558 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f261c0186c0 0x7f261c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:25.731 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:25.726 7f263b7fe700 1 --2- 172.21.2.102:0/1718327558 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f261c0186c0 0x7f261c01abc0 crc :-1 s=READY pgs=1405 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:25.760 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:25.758 7f2642547700 1 -- 172.21.2.102:0/1718327558 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f263c0ad790 con 0x7f263c18f0a0 2021-02-10T11:39:25.762 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:25.760 7f2638ff9700 1 -- 172.21.2.102:0/1718327558 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f262c025fe0 con 0x7f263c18f0a0 2021-02-10T11:39:25.893 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:25.891 7f2642547700 1 -- 172.21.2.102:0/1718327558 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f263c0ad790 con 0x7f263c18f0a0 2021-02-10T11:39:25.894 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:25.892 7f2638ff9700 1 -- 172.21.2.102:0/1718327558 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v107) v1 ==== 80+0+4513 (secure 0 0 0) 0x7f262c025fe0 con 0x7f263c18f0a0 2021-02-10T11:39:25.894 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:39:25.894 INFO:teuthology.orchestra.run.gibba002.stdout:{ 2021-02-10T11:39:25.895 INFO:teuthology.orchestra.run.gibba002.stdout: "auth_dump": [ 2021-02-10T11:39:25.895 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:39:25.895 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "mds.a", 2021-02-10T11:39:25.895 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAevSNgsgyEARAAvr0cUjjCoEHLxaZSZ2HtVA==", 2021-02-10T11:39:25.895 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:39:25.895 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow", 2021-02-10T11:39:25.895 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:39:25.895 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:39:25.896 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:39:25.896 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:39:25.896 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:39:25.896 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:39:25.896 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "mds.b", 2021-02-10T11:39:25.896 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAevSNgwOFMMRAALfb1fbX56ao1TU8ClnTW1A==", 2021-02-10T11:39:25.896 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:39:25.896 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow", 2021-02-10T11:39:25.896 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:39:25.896 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:39:25.896 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:39:25.896 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:39:25.896 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:39:25.897 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:39:25.897 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.0", 2021-02-10T11:39:25.897 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAmvSNgj+bMMBAAD3HUD6fLiMAg+++iNzLJKQ==", 2021-02-10T11:39:25.897 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:39:25.897 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:39:25.897 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:39:25.897 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:39:25.897 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:39:25.897 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:39:25.897 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:39:25.897 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.1", 2021-02-10T11:39:25.897 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQApvSNgnvc+MRAAAOxqlNwiyLHqNITOrYeFKQ==", 2021-02-10T11:39:25.897 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:39:25.897 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:39:25.898 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:39:25.898 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:39:25.898 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:39:25.898 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:39:25.898 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:39:25.898 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.2", 2021-02-10T11:39:25.898 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAsvSNgZ4SbMRAA76IOYWfbiAuBoM/4k81GxA==", 2021-02-10T11:39:25.898 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:39:25.898 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:39:25.898 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:39:25.898 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:39:25.898 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:39:25.898 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:39:25.899 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:39:25.899 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.3", 2021-02-10T11:39:25.899 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAvvSNgFk3rMRAAWCmKKyw0LDQ+UlAHZAITOw==", 2021-02-10T11:39:25.899 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:39:25.899 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:39:25.899 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:39:25.899 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:39:25.899 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:39:25.899 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:39:25.899 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:39:25.899 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.4", 2021-02-10T11:39:25.899 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQA4vSNgpKcNJBAApJMmvj5YxVHPDOmvmQONOg==", 2021-02-10T11:39:25.899 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:39:25.900 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:39:25.900 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:39:25.900 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:39:25.900 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:39:25.900 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:39:25.900 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:39:25.900 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.5", 2021-02-10T11:39:25.900 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQA7vSNgf2ujJBAAfkuYpyb3WTzc8V5kw+Wktg==", 2021-02-10T11:39:25.900 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:39:25.900 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:39:25.900 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:39:25.900 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:39:25.900 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:39:25.900 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:39:25.901 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:39:25.901 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.6", 2021-02-10T11:39:25.901 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQA+vSNgqw9uJRAAFMU0R+UxU9C3o/9byb3e0g==", 2021-02-10T11:39:25.901 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:39:25.901 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:39:25.901 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:39:25.901 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:39:25.901 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:39:25.901 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:39:25.901 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:39:25.901 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.7", 2021-02-10T11:39:25.901 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQBBvSNgvWSrJRAA2KaCHmLpzovRgYzLvyofTQ==", 2021-02-10T11:39:25.901 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:39:25.902 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:39:25.902 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:39:25.902 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:39:25.902 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:39:25.902 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:39:25.902 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:39:25.903 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "client.0", 2021-02-10T11:39:25.903 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAfvSNgZd9XIBAAvyrvl8fdKuHnJZfczfRoFw==", 2021-02-10T11:39:25.903 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:39:25.903 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow", 2021-02-10T11:39:25.903 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow r", 2021-02-10T11:39:25.903 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:39:25.903 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:39:25.903 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:39:25.903 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:39:25.903 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "client.1", 2021-02-10T11:39:25.903 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAfvSNgSyfXNhAAIduPnF7Vd/6HSkJ0MlcMSA==", 2021-02-10T11:39:25.904 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:39:25.904 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow", 2021-02-10T11:39:25.904 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow r", 2021-02-10T11:39:25.904 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:39:25.904 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:39:25.904 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:39:25.904 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:39:25.904 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "client.2", 2021-02-10T11:39:25.904 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAgvSNgiIG/ERAA9ZTamaTI7ZsPSnxBqjplCg==", 2021-02-10T11:39:25.904 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:39:25.904 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow", 2021-02-10T11:39:25.904 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow r", 2021-02-10T11:39:25.904 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:39:25.904 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:39:25.905 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:39:25.905 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:39:25.905 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "client.3", 2021-02-10T11:39:25.905 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAgvSNgOWpOKBAAlXsEB04Q/GvtL06PqRMLeA==", 2021-02-10T11:39:25.906 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:39:25.906 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow", 2021-02-10T11:39:25.906 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow r", 2021-02-10T11:39:25.906 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:39:25.906 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:39:25.906 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:39:25.906 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:39:25.906 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "client.admin", 2021-02-10T11:39:25.906 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAavSNgO3GQBhAAL+NVnYskQRKVWLgyDPY/xw==", 2021-02-10T11:39:25.906 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:39:25.906 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow *", 2021-02-10T11:39:25.907 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:39:25.907 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:39:25.907 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:39:25.907 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:39:25.907 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:39:25.907 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:39:25.907 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "mgr.x", 2021-02-10T11:39:25.907 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAdvSNgXZZgJhAA7dtBSGzH+DsBwnddZpOBJA==", 2021-02-10T11:39:25.907 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:39:25.907 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow *", 2021-02-10T11:39:25.907 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow profile mgr", 2021-02-10T11:39:25.907 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:39:25.907 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:39:25.908 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:39:25.908 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:39:25.908 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "mgr.y", 2021-02-10T11:39:25.908 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAdvSNgVqQFDRAAwo6YEdWZCAr5GdOWVL/Qvg==", 2021-02-10T11:39:25.908 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:39:25.908 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow *", 2021-02-10T11:39:25.908 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow profile mgr", 2021-02-10T11:39:25.908 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:39:25.908 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:39:25.908 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:39:25.908 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:39:25.908 INFO:teuthology.orchestra.run.gibba002.stdout:} 2021-02-10T11:39:25.909 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:25.893 7f2642547700 1 -- 172.21.2.102:0/1718327558 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f261c0186c0 msgr2=0x7f261c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:25.910 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:25.893 7f2642547700 1 --2- 172.21.2.102:0/1718327558 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f261c0186c0 0x7f261c01abc0 crc :-1 s=READY pgs=1405 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:25.910 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:25.893 7f2642547700 1 -- 172.21.2.102:0/1718327558 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f263c18f0a0 msgr2=0x7f263c196ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:25.910 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:25.893 7f2642547700 1 --2- 172.21.2.102:0/1718327558 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f263c18f0a0 0x7f263c196ab0 secure :-1 s=READY pgs=3217 cs=0 l=1 rev1=1 rx=0x7f262c001410 tx=0x7f262c00a950).stop 2021-02-10T11:39:25.910 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:25.894 7f2642547700 1 -- 172.21.2.102:0/1718327558 shutdown_connections 2021-02-10T11:39:25.910 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:25.894 7f2642547700 1 --2- 172.21.2.102:0/1718327558 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f263c094650 0x7f263c094a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:25.910 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:25.894 7f2642547700 1 --2- 172.21.2.102:0/1718327558 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f261c0186c0 0x7f261c01abc0 unknown :-1 s=CLOSED pgs=1405 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:25.910 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:25.894 7f2642547700 1 --2- 172.21.2.102:0/1718327558 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f263c1970d0 0x7f263c1974d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:25.910 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:25.894 7f2642547700 1 --2- 172.21.2.102:0/1718327558 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f263c18f0a0 0x7f263c196ab0 unknown :-1 s=CLOSED pgs=3217 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:25.910 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:25.894 7f2642547700 1 -- 172.21.2.102:0/1718327558 >> 172.21.2.102:0/1718327558 conn(0x7f263c1a38c0 msgr2=0x7f263c1a2f00 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:25.910 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:25.894 7f2642547700 1 -- 172.21.2.102:0/1718327558 shutdown_connections 2021-02-10T11:39:25.910 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:25.894 7f2642547700 1 -- 172.21.2.102:0/1718327558 wait complete. 2021-02-10T11:39:25.911 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:39:26.341 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:26.322 7fa11ab7a700 1 Processor -- start 2021-02-10T11:39:26.341 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:26.322 7fa11ab7a700 1 -- start start 2021-02-10T11:39:26.341 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:26.322 7fa11ab7a700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa1140d1870 0x7fa1141b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:26.341 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:26.322 7fa11ab7a700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fa1140f0650 0x7fa1140cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:26.341 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:26.322 7fa11ab7a700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1140f0b10 con 0x7fa1141b0a10 2021-02-10T11:39:26.341 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:26.322 7fa11ab7a700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1140fe230 con 0x7fa1140f0650 2021-02-10T11:39:26.341 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:26.322 7fa11ab7a700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1140e5ed0 con 0x7fa1140d1870 2021-02-10T11:39:26.341 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:26.323 7fa113fff700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fa1141b0a10 0x7fa1140cf330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:60752/0 (socket says 172.21.2.102:60752) 2021-02-10T11:39:26.342 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:26.323 7fa113fff700 1 -- 172.21.2.102:0/3442865702 learned_addr learned my addr 172.21.2.102:0/3442865702 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:39:26.342 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:26.323 7fa119197700 1 --2- 172.21.2.102:0/3442865702 >> v2:172.21.2.102:3300/0 conn(0x7fa1140f0650 0x7fa1140cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:26.342 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:26.323 7fa118996700 1 --2- 172.21.2.102:0/3442865702 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa1140d1870 0x7fa1141b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:26.342 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:26.323 7fa119197700 1 -- 172.21.2.102:0/3442865702 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa1140d1870 msgr2=0x7fa1141b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:26.342 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:26.323 7fa119197700 1 --2- 172.21.2.102:0/3442865702 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa1140d1870 0x7fa1141b03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:26.342 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:26.323 7fa119197700 1 -- 172.21.2.102:0/3442865702 >> v1:172.21.2.102:6789/0 conn(0x7fa1141b0a10 legacy=0x7fa1140cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:26.342 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:26.323 7fa119197700 1 -- 172.21.2.102:0/3442865702 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1140f20a0 con 0x7fa1140f0650 2021-02-10T11:39:26.342 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:26.323 7fa119197700 1 --2- 172.21.2.102:0/3442865702 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa1140f0650 0x7fa1140cfb20 secure :-1 s=READY pgs=3219 cs=0 l=1 rev1=1 rx=0x7fa10c006e30 tx=0x7fa10c009e60).ready entity=mon.0 client_cookie=d0a9b243a7497889 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:26.342 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:26.323 7fa1137fe700 1 -- 172.21.2.102:0/3442865702 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa10c006a50 con 0x7fa1140f0650 2021-02-10T11:39:26.342 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:26.323 7fa1137fe700 1 -- 172.21.2.102:0/3442865702 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa10c00b300 con 0x7fa1140f0650 2021-02-10T11:39:26.342 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:26.323 7fa1137fe700 1 -- 172.21.2.102:0/3442865702 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa10c00b7e0 con 0x7fa1140f0650 2021-02-10T11:39:26.342 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:26.323 7fa11ab7a700 1 -- 172.21.2.102:0/3442865702 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa1140f0650 msgr2=0x7fa1140cfb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:26.343 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:26.323 7fa11ab7a700 1 --2- 172.21.2.102:0/3442865702 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa1140f0650 0x7fa1140cfb20 secure :-1 s=READY pgs=3219 cs=0 l=1 rev1=1 rx=0x7fa10c006e30 tx=0x7fa10c009e60).stop 2021-02-10T11:39:26.343 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:26.324 7fa11ab7a700 1 -- 172.21.2.102:0/3442865702 shutdown_connections 2021-02-10T11:39:26.343 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:26.324 7fa11ab7a700 1 --2- 172.21.2.102:0/3442865702 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa1140f0650 0x7fa1140cfb20 unknown :-1 s=CLOSED pgs=3219 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:26.343 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:26.324 7fa11ab7a700 1 --2- 172.21.2.102:0/3442865702 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa1140d1870 0x7fa1141b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:26.343 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:26.324 7fa11ab7a700 1 -- 172.21.2.102:0/3442865702 >> 172.21.2.102:0/3442865702 conn(0x7fa1140ce450 msgr2=0x7fa1140d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:26.343 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:26.324 7fa11ab7a700 1 -- 172.21.2.102:0/3442865702 shutdown_connections 2021-02-10T11:39:26.343 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:26.324 7fa11ab7a700 1 -- 172.21.2.102:0/3442865702 wait complete. 2021-02-10T11:39:26.343 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:26.339 7fa11ab7a700 1 Processor -- start 2021-02-10T11:39:26.343 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:26.339 7fa11ab7a700 1 -- start start 2021-02-10T11:39:26.345 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:26.339 7fa11ab7a700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa114195650 0x7fa1140b5560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:26.345 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:26.339 7fa11ab7a700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa1140b5b80 0x7fa1141a2e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:26.345 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:26.339 7fa11ab7a700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa1140acfa0 0x7fa1141a3440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:26.345 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:26.339 7fa11ab7a700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1140b6010 con 0x7fa1140b5b80 2021-02-10T11:39:26.345 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:26.339 7fa11ab7a700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1141a3a60 con 0x7fa1140acfa0 2021-02-10T11:39:26.345 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:26.339 7fa11ab7a700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1141a3c40 con 0x7fa114195650 2021-02-10T11:39:26.345 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:26.339 7fa113fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa1140b5b80 0x7fa1141a2e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:26.345 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:26.339 7fa113fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa1140b5b80 0x7fa1141a2e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:40302/0 (socket says 172.21.2.102:40302) 2021-02-10T11:39:26.345 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:26.339 7fa113fff700 1 -- 172.21.2.102:0/4241729106 learned_addr learned my addr 172.21.2.102:0/4241729106 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:39:26.345 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:26.340 7fa113fff700 1 -- 172.21.2.102:0/4241729106 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa114195650 msgr2=0x7fa1140b5560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:26.345 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:26.340 7fa113fff700 1 --2- 172.21.2.102:0/4241729106 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa114195650 0x7fa1140b5560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:26.346 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:26.340 7fa113fff700 1 -- 172.21.2.102:0/4241729106 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa1140acfa0 msgr2=0x7fa1141a3440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:26.346 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:26.340 7fa113fff700 1 --2- 172.21.2.102:0/4241729106 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa1140acfa0 0x7fa1141a3440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:26.346 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:26.340 7fa113fff700 1 -- 172.21.2.102:0/4241729106 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1140bcae0 con 0x7fa1140b5b80 2021-02-10T11:39:26.346 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:26.340 7fa113fff700 1 --2- 172.21.2.102:0/4241729106 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa1140b5b80 0x7fa1141a2e20 secure :-1 s=READY pgs=3220 cs=0 l=1 rev1=1 rx=0x7fa104001410 tx=0x7fa10400a950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:26.346 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:26.340 7fa1117fa700 1 -- 172.21.2.102:0/4241729106 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa10400ad40 con 0x7fa1140b5b80 2021-02-10T11:39:26.346 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:26.340 7fa1117fa700 1 -- 172.21.2.102:0/4241729106 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1040043a0 con 0x7fa1140b5b80 2021-02-10T11:39:26.346 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:26.340 7fa1117fa700 1 -- 172.21.2.102:0/4241729106 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa1040047b0 con 0x7fa1140b5b80 2021-02-10T11:39:26.346 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:26.340 7fa11ab7a700 1 -- 172.21.2.102:0/4241729106 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1140bcae0 con 0x7fa1140b5b80 2021-02-10T11:39:26.346 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:26.340 7fa11ab7a700 1 -- 172.21.2.102:0/4241729106 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1140bcae0 con 0x7fa1140b5b80 2021-02-10T11:39:26.346 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:26.341 7fa1117fa700 1 -- 172.21.2.102:0/4241729106 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa104004580 con 0x7fa1140b5b80 2021-02-10T11:39:26.346 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:26.341 7fa1117fa700 1 --2- 172.21.2.102:0/4241729106 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa0f80186c0 0x7fa0f801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:26.347 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:26.341 7fa1117fa700 1 -- 172.21.2.102:0/4241729106 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(184..184 src has 1..184) v4 ==== 4753+0+0 (secure 0 0 0) 0x7fa1040322b0 con 0x7fa1140b5b80 2021-02-10T11:39:26.347 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:26.341 7fa118996700 1 --2- 172.21.2.102:0/4241729106 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa0f80186c0 0x7fa0f801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:26.347 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:26.341 7fa118996700 1 --2- 172.21.2.102:0/4241729106 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa0f80186c0 0x7fa0f801abc0 crc :-1 s=READY pgs=1406 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:26.374 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:26.372 7fa11ab7a700 1 -- 172.21.2.102:0/4241729106 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1141a3c40 con 0x7fa1140b5b80 2021-02-10T11:39:26.377 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:26.375 7fa1117fa700 1 -- 172.21.2.102:0/4241729106 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fa104030d90 con 0x7fa1140b5b80 2021-02-10T11:39:26.501 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:26.498 7fa11ab7a700 1 -- 172.21.2.102:0/4241729106 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa1141a3c40 con 0x7fa1140b5b80 2021-02-10T11:39:26.501 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:26.499 7fa1117fa700 1 -- 172.21.2.102:0/4241729106 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v184) v1 ==== 74+0+11154 (secure 0 0 0) 0x7fa104004560 con 0x7fa1140b5b80 2021-02-10T11:39:26.502 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:39:26.502 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":184,"fsid":"68db30ea-dbce-4aef-a6df-f01c6fea0513","created":"2021-02-10 11:03:06.331557","modified":"2021-02-10 11:39:25.253860","last_up_change":"2021-02-10 11:03:32.690562","last_in_change":"2021-02-10 11:03:32.690562","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":39,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:03:37.322519","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"be691b4d-a039-408a-af80-9d53f02b6f1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":178,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6818","nonce":36088},{"type":"v1","addr":"172.21.2.102:6820","nonce":36088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6822","nonce":36088},{"type":"v1","addr":"172.21.2.102:6824","nonce":36088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6830","nonce":36088},{"type":"v1","addr":"172.21.2.102:6832","nonce":36088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6826","nonce":36088},{"type":"v1","addr":"172.21.2.102:6828","nonce":36088}]},"public_addr":"172.21.2.102:6820/36088","cluster_addr":"172.21.2.102:6824/36088","heartbeat_back_addr":"172.21.2.102:6832/36088","heartbeat_front_addr":"172.21.2.102:6828/36088","state":["exists","up"]},{"osd":1,"uuid":"64e3ccc8-9c69-4d7c-865e-af8e83822635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":178,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6819","nonce":36087},{"type":"v1","addr":"172.21.2.102:6821","nonce":36087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6823","nonce":36087},{"type":"v1","addr":"172.21.2.102:6825","nonce":36087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6831","nonce":36087},{"type":"v1","addr":"172.21.2.102:6833","nonce":36087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6827","nonce":36087},{"type":"v1","addr":"172.21.2.102:6829","nonce":36087}]},"public_addr":"172.21.2.102:6821/36087","cluster_addr":"172.21.2.102:6825/36087","heartbeat_back_addr":"172.21.2.102:6833/36087","heartbeat_front_addr":"172.21.2.102:6829/36087","state":["exists","up"]},{"osd":2,"uuid":"153700af-747f-481f-bdf4-d6072d3a842f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":178,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6810","nonce":36089},{"type":"v1","addr":"172.21.2.102:6811","nonce":36089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6812","nonce":36089},{"type":"v1","addr":"172.21.2.102:6813","nonce":36089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6816","nonce":36089},{"type":"v1","addr":"172.21.2.102:6817","nonce":36089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6814","nonce":36089},{"type":"v1","addr":"172.21.2.102:6815","nonce":36089}]},"public_addr":"172.21.2.102:6811/36089","cluster_addr":"172.21.2.102:6813/36089","heartbeat_back_addr":"172.21.2.102:6817/36089","heartbeat_front_addr":"172.21.2.102:6815/36089","state":["exists","up"]},{"osd":3,"uuid":"cbc678f4-83a9-45e6-8830-2f7c232b1ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":178,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6802","nonce":36090},{"type":"v1","addr":"172.21.2.102:6803","nonce":36090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6804","nonce":36090},{"type":"v1","addr":"172.21.2.102:6805","nonce":36090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6808","nonce":36090},{"type":"v1","addr":"172.21.2.102:6809","nonce":36090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6806","nonce":36090},{"type":"v1","addr":"172.21.2.102:6807","nonce":36090}]},"public_addr":"172.21.2.102:6803/36090","cluster_addr":"172.21.2.102:6805/36090","heartbeat_back_addr":"172.21.2.102:6809/36090","heartbeat_front_addr":"172.21.2.102:6807/36090","state":["exists","up"]},{"osd":4,"uuid":"c01e4008-b258-4f10-afd2-9e7f37214351","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":178,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6808","nonce":36368},{"type":"v1","addr":"172.21.2.113:6811","nonce":36368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6814","nonce":36368},{"type":"v1","addr":"172.21.2.113:6818","nonce":36368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6826","nonce":36368},{"type":"v1","addr":"172.21.2.113:6828","nonce":36368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6820","nonce":36368},{"type":"v1","addr":"172.21.2.113:6824","nonce":36368}]},"public_addr":"172.21.2.113:6811/36368","cluster_addr":"172.21.2.113:6818/36368","heartbeat_back_addr":"172.21.2.113:6828/36368","heartbeat_front_addr":"172.21.2.113:6824/36368","state":["exists","up"]},{"osd":5,"uuid":"551f6007-daf8-4a98-8ee0-c2b16bcccb5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":178,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6802","nonce":36366},{"type":"v1","addr":"172.21.2.113:6805","nonce":36366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6807","nonce":36366},{"type":"v1","addr":"172.21.2.113:6810","nonce":36366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6819","nonce":36366},{"type":"v1","addr":"172.21.2.113:6822","nonce":36366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6812","nonce":36366},{"type":"v1","addr":"172.21.2.113:6815","nonce":36366}]},"public_addr":"172.21.2.113:6805/36366","cluster_addr":"172.21.2.113:6810/36366","heartbeat_back_addr":"172.21.2.113:6822/36366","heartbeat_front_addr":"172.21.2.113:6815/36366","state":["exists","up"]},{"osd":6,"uuid":"bf67c040-0e50-48be-b102-c3943bc9ebef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6800","nonce":36367},{"type":"v1","addr":"172.21.2.113:6801","nonce":36367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6803","nonce":36367},{"type":"v1","addr":"172.21.2.113:6804","nonce":36367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6813","nonce":36367},{"type":"v1","addr":"172.21.2.113:6816","nonce":36367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6806","nonce":36367},{"type":"v1","addr":"172.21.2.113:6809","nonce":36367}]},"public_addr":"172.21.2.113:6801/36367","cluster_addr":"172.21.2.113:6804/36367","heartbeat_back_addr":"172.21.2.113:6816/36367","heartbeat_front_addr":"172.21.2.113:6809/36367","state":["exists","up"]},{"osd":7,"uuid":"f5461f80-018d-428f-be89-2eb2bcbe7bb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":178,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6817","nonce":36365},{"type":"v1","addr":"172.21.2.113:6821","nonce":36365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6823","nonce":36365},{"type":"v1","addr":"172.21.2.113:6825","nonce":36365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6830","nonce":36365},{"type":"v1","addr":"172.21.2.113:6831","nonce":36365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6827","nonce":36365},{"type":"v1","addr":"172.21.2.113:6829","nonce":36365}]},"public_addr":"172.21.2.113:6821/36365","cluster_addr":"172.21.2.113:6825/36365","heartbeat_back_addr":"172.21.2.113:6831/36365","heartbeat_front_addr":"172.21.2.113:6829/36365","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:39:26.503 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:26.500 7fa11ab7a700 1 -- 172.21.2.102:0/4241729106 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa0f80186c0 msgr2=0x7fa0f801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:26.503 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:26.500 7fa11ab7a700 1 --2- 172.21.2.102:0/4241729106 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa0f80186c0 0x7fa0f801abc0 crc :-1 s=READY pgs=1406 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:26.504 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:26.500 7fa11ab7a700 1 -- 172.21.2.102:0/4241729106 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa1140b5b80 msgr2=0x7fa1141a2e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:26.504 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:26.500 7fa11ab7a700 1 --2- 172.21.2.102:0/4241729106 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa1140b5b80 0x7fa1141a2e20 secure :-1 s=READY pgs=3220 cs=0 l=1 rev1=1 rx=0x7fa104001410 tx=0x7fa10400a950).stop 2021-02-10T11:39:26.504 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:26.501 7fa11ab7a700 1 -- 172.21.2.102:0/4241729106 shutdown_connections 2021-02-10T11:39:26.504 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:26.501 7fa11ab7a700 1 --2- 172.21.2.102:0/4241729106 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa0f80186c0 0x7fa0f801abc0 unknown :-1 s=CLOSED pgs=1406 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:26.504 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:26.501 7fa11ab7a700 1 --2- 172.21.2.102:0/4241729106 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa1140acfa0 0x7fa1141a3440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:26.504 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:26.501 7fa11ab7a700 1 --2- 172.21.2.102:0/4241729106 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa1140b5b80 0x7fa1141a2e20 unknown :-1 s=CLOSED pgs=3220 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:26.504 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:26.501 7fa11ab7a700 1 --2- 172.21.2.102:0/4241729106 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa114195650 0x7fa1140b5560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:26.504 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:26.501 7fa11ab7a700 1 -- 172.21.2.102:0/4241729106 >> 172.21.2.102:0/4241729106 conn(0x7fa1140b3040 msgr2=0x7fa1140c3250 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:26.504 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:26.501 7fa11ab7a700 1 -- 172.21.2.102:0/4241729106 shutdown_connections 2021-02-10T11:39:26.505 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:26.501 7fa11ab7a700 1 -- 172.21.2.102:0/4241729106 wait complete. 2021-02-10T11:39:26.512 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:03:37.322519', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2021-02-10T11:39:26.513 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:39:26.947 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:26.927 7fe650f2b700 1 Processor -- start 2021-02-10T11:39:26.947 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:26.927 7fe650f2b700 1 -- start start 2021-02-10T11:39:26.947 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:26.928 7fe650f2b700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fe64c0d1870 0x7fe64c1b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:26.947 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:26.928 7fe650f2b700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe64c0fe1c0 con 0x7fe64c1b0ae0 2021-02-10T11:39:26.948 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:26.928 7fe650f2b700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe64c109a60 con 0x7fe64c0cff70 2021-02-10T11:39:26.948 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:26.928 7fe650f2b700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe64c088070 con 0x7fe64c0d1870 2021-02-10T11:39:26.948 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:26.928 7fe64a312700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fe64c1b0ae0 0x7fe64c108940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:60766/0 (socket says 172.21.2.102:60766) 2021-02-10T11:39:26.948 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:26.928 7fe64a312700 1 -- 172.21.2.102:0/1982814597 learned_addr learned my addr 172.21.2.102:0/1982814597 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:39:26.948 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:26.929 7fe649b11700 1 -- 172.21.2.102:0/1982814597 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3486964648 0 0) 0x7fe63c000ce0 con 0x7fe64c1b0ae0 2021-02-10T11:39:26.948 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:26.929 7fe649b11700 1 -- 172.21.2.102:0/1982814597 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe638002fe0 con 0x7fe64c1b0ae0 2021-02-10T11:39:26.948 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:26.929 7fe64ab13700 1 --2- 172.21.2.102:0/1982814597 >> v2:172.21.2.113:3300/0 conn(0x7fe64c0d1870 0x7fe64c1b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:26.948 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:26.929 7fe649b11700 1 -- 172.21.2.102:0/1982814597 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1292811328 0 0) 0x7fe63c0010e0 con 0x7fe64c1b0ae0 2021-02-10T11:39:26.948 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:26.929 7fe649b11700 1 -- 172.21.2.102:0/1982814597 >> v2:172.21.2.113:3300/0 conn(0x7fe64c0d1870 msgr2=0x7fe64c1b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:26.948 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:26.929 7fe649b11700 1 --2- 172.21.2.102:0/1982814597 >> v2:172.21.2.113:3300/0 conn(0x7fe64c0d1870 0x7fe64c1b04c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:26.948 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:26.929 7fe649b11700 1 -- 172.21.2.102:0/1982814597 >> v1:172.21.2.113:6789/0 conn(0x7fe64c0cff70 legacy=0x7fe64c109130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:26.948 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:26.929 7fe649b11700 1 -- 172.21.2.102:0/1982814597 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe64c088260 con 0x7fe64c1b0ae0 2021-02-10T11:39:26.949 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:26.929 7fe649b11700 1 -- 172.21.2.102:0/1982814597 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7fe63c001660 con 0x7fe64c1b0ae0 2021-02-10T11:39:26.949 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:26.929 7fe649b11700 1 -- 172.21.2.102:0/1982814597 >> v1:172.21.2.102:6789/0 conn(0x7fe64c1b0ae0 legacy=0x7fe64c108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:26.949 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:26.929 7fe649b11700 1 --2- 172.21.2.102:0/1982814597 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe638003ac0 0x7fe638005ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:26.949 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:26.929 7fe649b11700 1 --2- 172.21.2.102:0/1982814597 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe638006770 0x7fe638008c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:26.949 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:26.929 7fe649b11700 1 --2- 172.21.2.102:0/1982814597 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe638009590 0x7fe63800bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:26.949 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:26.929 7fe649b11700 1 -- 172.21.2.102:0/1982814597 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe63800c300 con 0x7fe638009590 2021-02-10T11:39:26.949 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:26.929 7fe649b11700 1 -- 172.21.2.102:0/1982814597 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe63800c520 con 0x7fe638006770 2021-02-10T11:39:26.949 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:26.929 7fe649b11700 1 -- 172.21.2.102:0/1982814597 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe63800c740 con 0x7fe638003ac0 2021-02-10T11:39:26.949 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:26.929 7fe64b314700 1 --2- 172.21.2.102:0/1982814597 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe638009590 0x7fe63800bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:26.949 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:26.929 7fe64a312700 1 --2- 172.21.2.102:0/1982814597 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe638006770 0x7fe638008c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:26.949 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:26.929 7fe64ab13700 1 --2- 172.21.2.102:0/1982814597 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe638003ac0 0x7fe638005ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:26.949 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:26.929 7fe64b314700 1 -- 172.21.2.102:0/1982814597 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe638003ac0 msgr2=0x7fe638005ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:26.949 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:26.929 7fe64b314700 1 --2- 172.21.2.102:0/1982814597 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe638003ac0 0x7fe638005ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:26.950 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:26.929 7fe64b314700 1 -- 172.21.2.102:0/1982814597 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe638006770 msgr2=0x7fe638008c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:26.950 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:26.929 7fe64b314700 1 --2- 172.21.2.102:0/1982814597 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe638006770 0x7fe638008c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:26.950 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:26.929 7fe64b314700 1 -- 172.21.2.102:0/1982814597 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6380509b0 con 0x7fe638009590 2021-02-10T11:39:26.950 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:26.930 7fe64b314700 1 -- 172.21.2.102:0/1982814597 reap_dead start 2021-02-10T11:39:26.950 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:26.930 7fe64b314700 1 --2- 172.21.2.102:0/1982814597 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe638009590 0x7fe63800bab0 secure :-1 s=READY pgs=3222 cs=0 l=1 rev1=1 rx=0x7fe644006e40 tx=0x7fe644009db0).ready entity=mon.0 client_cookie=85f1f20e06cf76fc server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:26.950 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:26.930 7fe649b11700 1 -- 172.21.2.102:0/1982814597 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe644006a60 con 0x7fe638009590 2021-02-10T11:39:26.950 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:26.930 7fe649b11700 1 -- 172.21.2.102:0/1982814597 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe64400b300 con 0x7fe638009590 2021-02-10T11:39:26.950 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:26.930 7fe649b11700 1 -- 172.21.2.102:0/1982814597 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe64400b800 con 0x7fe638009590 2021-02-10T11:39:26.950 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:26.930 7fe650f2b700 1 -- 172.21.2.102:0/1982814597 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe638009590 msgr2=0x7fe63800bab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:26.951 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:26.930 7fe650f2b700 1 --2- 172.21.2.102:0/1982814597 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe638009590 0x7fe63800bab0 secure :-1 s=READY pgs=3222 cs=0 l=1 rev1=1 rx=0x7fe644006e40 tx=0x7fe644009db0).stop 2021-02-10T11:39:26.951 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:26.930 7fe650f2b700 1 -- 172.21.2.102:0/1982814597 shutdown_connections 2021-02-10T11:39:26.951 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:26.930 7fe650f2b700 1 --2- 172.21.2.102:0/1982814597 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe638009590 0x7fe63800bab0 unknown :-1 s=CLOSED pgs=3222 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:26.951 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:26.930 7fe650f2b700 1 -- 172.21.2.102:0/1982814597 >> 172.21.2.102:0/1982814597 conn(0x7fe64c0ce420 msgr2=0x7fe64c0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:26.951 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:26.930 7fe650f2b700 1 -- 172.21.2.102:0/1982814597 shutdown_connections 2021-02-10T11:39:26.951 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:26.930 7fe650f2b700 1 -- 172.21.2.102:0/1982814597 wait complete. 2021-02-10T11:39:26.951 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:26.945 7fe650f2b700 1 Processor -- start 2021-02-10T11:39:26.951 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:26.945 7fe650f2b700 1 -- start start 2021-02-10T11:39:26.952 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:26.945 7fe650f2b700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe64c085bc0 0x7fe64c0863e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:26.953 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:26.945 7fe650f2b700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe64c0954c0 0x7fe64c096300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:26.953 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:26.945 7fe650f2b700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe64c0958c0 0x7fe64c096920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:26.953 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:26.945 7fe650f2b700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe64c095d80 con 0x7fe64c085bc0 2021-02-10T11:39:26.953 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:26.945 7fe650f2b700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe64c0bfa60 con 0x7fe64c0958c0 2021-02-10T11:39:26.953 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:26.945 7fe650f2b700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe64c085fc0 con 0x7fe64c0954c0 2021-02-10T11:39:26.953 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:26.946 7fe64ab13700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe64c085bc0 0x7fe64c0863e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:26.953 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:26.946 7fe64ab13700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe64c085bc0 0x7fe64c0863e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:40320/0 (socket says 172.21.2.102:40320) 2021-02-10T11:39:26.953 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:26.946 7fe64ab13700 1 -- 172.21.2.102:0/1949892225 learned_addr learned my addr 172.21.2.102:0/1949892225 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:39:26.953 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:26.946 7fe64ab13700 1 -- 172.21.2.102:0/1949892225 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe64c0954c0 msgr2=0x7fe64c096300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:26.953 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:26.946 7fe64ab13700 1 --2- 172.21.2.102:0/1949892225 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe64c0954c0 0x7fe64c096300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:26.954 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:26.946 7fe64ab13700 1 -- 172.21.2.102:0/1949892225 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe64c0958c0 msgr2=0x7fe64c096920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:26.954 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:26.946 7fe64ab13700 1 --2- 172.21.2.102:0/1949892225 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe64c0958c0 0x7fe64c096920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:26.954 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:26.946 7fe64ab13700 1 -- 172.21.2.102:0/1949892225 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe64c086c10 con 0x7fe64c085bc0 2021-02-10T11:39:26.954 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:26.946 7fe64ab13700 1 --2- 172.21.2.102:0/1949892225 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe64c085bc0 0x7fe64c0863e0 secure :-1 s=READY pgs=3223 cs=0 l=1 rev1=1 rx=0x7fe634006f60 tx=0x7fe63400b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:26.955 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:26.946 7fe6437fe700 1 -- 172.21.2.102:0/1949892225 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe63400bd60 con 0x7fe64c085bc0 2021-02-10T11:39:26.955 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:26.946 7fe6437fe700 1 -- 172.21.2.102:0/1949892225 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe634000c90 con 0x7fe64c085bc0 2021-02-10T11:39:26.955 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:26.946 7fe6437fe700 1 -- 172.21.2.102:0/1949892225 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe6340022b0 con 0x7fe64c085bc0 2021-02-10T11:39:26.955 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:26.946 7fe650f2b700 1 -- 172.21.2.102:0/1949892225 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe64c086c10 con 0x7fe64c085bc0 2021-02-10T11:39:26.955 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:26.947 7fe650f2b700 1 -- 172.21.2.102:0/1949892225 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe64c097230 con 0x7fe64c085bc0 2021-02-10T11:39:26.955 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:26.947 7fe6437fe700 1 -- 172.21.2.102:0/1949892225 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fe634006b00 con 0x7fe64c085bc0 2021-02-10T11:39:26.955 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:26.947 7fe6437fe700 1 --2- 172.21.2.102:0/1949892225 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe6300186c0 0x7fe63001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:26.955 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:26.947 7fe6437fe700 1 -- 172.21.2.102:0/1949892225 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(184..184 src has 1..184) v4 ==== 4753+0+0 (secure 0 0 0) 0x7fe634032090 con 0x7fe64c085bc0 2021-02-10T11:39:26.955 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:26.947 7fe64a312700 1 --2- 172.21.2.102:0/1949892225 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe6300186c0 0x7fe63001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:26.955 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:26.948 7fe64a312700 1 --2- 172.21.2.102:0/1949892225 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe6300186c0 0x7fe63001abc0 crc :-1 s=READY pgs=1407 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:26.981 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:26.978 7fe650f2b700 1 -- 172.21.2.102:0/1949892225 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe64c0fe1c0 con 0x7fe64c085bc0 2021-02-10T11:39:26.983 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:26.981 7fe6437fe700 1 -- 172.21.2.102:0/1949892225 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fe63402fbf0 con 0x7fe64c085bc0 2021-02-10T11:39:27.112 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:27.109 7fe650f2b700 1 -- 172.21.2.102:0/1949892225 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7fe64c0fe1c0 con 0x7fe64c085bc0 2021-02-10T11:39:27.112 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:27.110 7fe6437fe700 1 -- 172.21.2.102:0/1949892225 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v184) v1 ==== 92+0+10 (secure 0 0 0) 0x7fe63402fbf0 con 0x7fe64c085bc0 2021-02-10T11:39:27.113 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:39:27.114 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:27.112 7fe650f2b700 1 -- 172.21.2.102:0/1949892225 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe6300186c0 msgr2=0x7fe63001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:27.115 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:27.112 7fe650f2b700 1 --2- 172.21.2.102:0/1949892225 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe6300186c0 0x7fe63001abc0 crc :-1 s=READY pgs=1407 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:27.115 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:27.112 7fe650f2b700 1 -- 172.21.2.102:0/1949892225 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe64c085bc0 msgr2=0x7fe64c0863e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:27.115 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:27.112 7fe650f2b700 1 --2- 172.21.2.102:0/1949892225 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe64c085bc0 0x7fe64c0863e0 secure :-1 s=READY pgs=3223 cs=0 l=1 rev1=1 rx=0x7fe634006f60 tx=0x7fe63400b970).stop 2021-02-10T11:39:27.115 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:27.112 7fe650f2b700 1 -- 172.21.2.102:0/1949892225 shutdown_connections 2021-02-10T11:39:27.115 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:27.112 7fe650f2b700 1 --2- 172.21.2.102:0/1949892225 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe6300186c0 0x7fe63001abc0 unknown :-1 s=CLOSED pgs=1407 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:27.115 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:27.112 7fe650f2b700 1 --2- 172.21.2.102:0/1949892225 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe64c0958c0 0x7fe64c096920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:27.115 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:27.112 7fe650f2b700 1 --2- 172.21.2.102:0/1949892225 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe64c0954c0 0x7fe64c096300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:27.115 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:27.112 7fe650f2b700 1 --2- 172.21.2.102:0/1949892225 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe64c085bc0 0x7fe64c0863e0 unknown :-1 s=CLOSED pgs=3223 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:27.115 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:27.112 7fe650f2b700 1 -- 172.21.2.102:0/1949892225 >> 172.21.2.102:0/1949892225 conn(0x7fe64c0bef00 msgr2=0x7fe64c09d1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:27.115 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:27.112 7fe650f2b700 1 -- 172.21.2.102:0/1949892225 shutdown_connections 2021-02-10T11:39:27.115 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:27.112 7fe650f2b700 1 -- 172.21.2.102:0/1949892225 wait complete. 2021-02-10T11:39:27.124 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --format=json-pretty osd lspools 2021-02-10T11:39:27.553 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:27.534 7f220b78c700 1 Processor -- start 2021-02-10T11:39:27.553 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:27.534 7f220b78c700 1 -- start start 2021-02-10T11:39:27.554 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:27.535 7f220b78c700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f22040d1c70 0x7f22040f20a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:27.554 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:27.535 7f220b78c700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22040f0b10 con 0x7f22040d1870 2021-02-10T11:39:27.554 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:27.535 7f220b78c700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22041b0be0 con 0x7f22040d0340 2021-02-10T11:39:27.554 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:27.535 7f220b78c700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22040e5ed0 con 0x7f22040d1c70 2021-02-10T11:39:27.554 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:27.535 7f2209da9700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f22040d1c70 0x7f22040f20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:27.554 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:27.535 7f22095a8700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f22040d1870 0x7f22041b03f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:60784/0 (socket says 172.21.2.102:60784) 2021-02-10T11:39:27.554 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:27.535 7f22095a8700 1 -- 172.21.2.102:0/445158889 learned_addr learned my addr 172.21.2.102:0/445158889 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:39:27.554 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:27.535 7f2209da9700 1 -- 172.21.2.102:0/445158889 >> v1:172.21.2.113:6789/0 conn(0x7f22040d0340 legacy=0x7f22040ef640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:27.554 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:27.535 7f2209da9700 1 -- 172.21.2.102:0/445158889 >> v1:172.21.2.102:6789/0 conn(0x7f22040d1870 legacy=0x7f22041b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:27.554 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:27.536 7f2209da9700 1 -- 172.21.2.102:0/445158889 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22040efe30 con 0x7f22040d1c70 2021-02-10T11:39:27.554 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:27.535 7f21f7fff700 1 -- 172.21.2.102:0/445158889 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1882000017 0 0) 0x7f21f8001180 con 0x7f22040d1870 2021-02-10T11:39:27.554 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:27.536 7f2209da9700 1 --2- 172.21.2.102:0/445158889 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f22040d1c70 0x7f22040f20a0 secure :-1 s=READY pgs=3225 cs=0 l=1 rev1=1 rx=0x7f21fc006e30 tx=0x7f21fc009de0).ready entity=mon.0 client_cookie=839babb5b3dfd581 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:27.555 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:27.536 7f21f7fff700 1 -- 172.21.2.102:0/445158889 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f21fc00b300 con 0x7f22040d1c70 2021-02-10T11:39:27.555 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:27.536 7f21f7fff700 1 -- 172.21.2.102:0/445158889 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21fc00b680 con 0x7f22040d1c70 2021-02-10T11:39:27.555 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:27.536 7f21f7fff700 1 -- 172.21.2.102:0/445158889 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f21fc00bb60 con 0x7f22040d1c70 2021-02-10T11:39:27.555 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:27.536 7f220b78c700 1 -- 172.21.2.102:0/445158889 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f22040d1c70 msgr2=0x7f22040f20a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:27.555 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:27.536 7f220b78c700 1 --2- 172.21.2.102:0/445158889 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f22040d1c70 0x7f22040f20a0 secure :-1 s=READY pgs=3225 cs=0 l=1 rev1=1 rx=0x7f21fc006e30 tx=0x7f21fc009de0).stop 2021-02-10T11:39:27.555 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:27.537 7f220b78c700 1 -- 172.21.2.102:0/445158889 shutdown_connections 2021-02-10T11:39:27.555 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:27.537 7f220b78c700 1 --2- 172.21.2.102:0/445158889 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f22040d1c70 0x7f22040f20a0 unknown :-1 s=CLOSED pgs=3225 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:27.555 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:27.537 7f220b78c700 1 -- 172.21.2.102:0/445158889 >> 172.21.2.102:0/445158889 conn(0x7f22040ce450 msgr2=0x7f22040d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:27.555 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:27.537 7f220b78c700 1 -- 172.21.2.102:0/445158889 shutdown_connections 2021-02-10T11:39:27.555 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:27.537 7f220b78c700 1 -- 172.21.2.102:0/445158889 wait complete. 2021-02-10T11:39:27.557 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:27.551 7f220b78c700 1 Processor -- start 2021-02-10T11:39:27.557 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:27.551 7f220b78c700 1 -- start start 2021-02-10T11:39:27.557 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:27.552 7f220b78c700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f22041947b0 0x7f22040a69a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:27.557 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:27.552 7f220b78c700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f22040a6fc0 0x7f22041a15d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:27.557 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:27.552 7f220b78c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f22041a1bf0 0x7f22041a1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:27.557 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:27.552 7f220b78c700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22040a73c0 con 0x7f22041a1bf0 2021-02-10T11:39:27.557 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:27.552 7f220b78c700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f220418f170 con 0x7f22041947b0 2021-02-10T11:39:27.557 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:27.552 7f220b78c700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2204194bb0 con 0x7f22040a6fc0 2021-02-10T11:39:27.557 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:27.552 7f2209da9700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f22041a1bf0 0x7f22041a1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:27.557 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:27.552 7f2209da9700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f22041a1bf0 0x7f22041a1ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:40338/0 (socket says 172.21.2.102:40338) 2021-02-10T11:39:27.557 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:27.552 7f2209da9700 1 -- 172.21.2.102:0/92898347 learned_addr learned my addr 172.21.2.102:0/92898347 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:39:27.557 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:27.553 7f2209da9700 1 -- 172.21.2.102:0/92898347 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f22040a6fc0 msgr2=0x7f22041a15d0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:39:27.558 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:27.553 7f2209da9700 1 --2- 172.21.2.102:0/92898347 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f22040a6fc0 0x7f22041a15d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:27.558 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:27.553 7f2209da9700 1 -- 172.21.2.102:0/92898347 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f22041947b0 msgr2=0x7f22040a69a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:27.558 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:27.553 7f2209da9700 1 --2- 172.21.2.102:0/92898347 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f22041947b0 0x7f22040a69a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:27.558 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:27.553 7f2209da9700 1 -- 172.21.2.102:0/92898347 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f220418ec70 con 0x7f22041a1bf0 2021-02-10T11:39:27.558 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:27.553 7f2209da9700 1 --2- 172.21.2.102:0/92898347 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f22041a1bf0 0x7f22041a1ff0 secure :-1 s=READY pgs=3226 cs=0 l=1 rev1=1 rx=0x7f21fc00b970 tx=0x7f21fc00bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:27.558 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:27.553 7f21f5ffb700 1 -- 172.21.2.102:0/92898347 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f21fc009380 con 0x7f22041a1bf0 2021-02-10T11:39:27.558 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:27.553 7f21f5ffb700 1 -- 172.21.2.102:0/92898347 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21fc005820 con 0x7f22041a1bf0 2021-02-10T11:39:27.558 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:27.553 7f21f5ffb700 1 -- 172.21.2.102:0/92898347 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f21fc005d40 con 0x7f22041a1bf0 2021-02-10T11:39:27.558 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:27.553 7f220b78c700 1 -- 172.21.2.102:0/92898347 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f220418ec70 con 0x7f22041a1bf0 2021-02-10T11:39:27.558 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:27.553 7f220b78c700 1 -- 172.21.2.102:0/92898347 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22041a2900 con 0x7f22041a1bf0 2021-02-10T11:39:27.558 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:27.554 7f21f5ffb700 1 -- 172.21.2.102:0/92898347 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f21fc005cd0 con 0x7f22041a1bf0 2021-02-10T11:39:27.558 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:27.554 7f21f5ffb700 1 --2- 172.21.2.102:0/92898347 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f21e40186c0 0x7f21e401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:27.558 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:27.554 7f21f5ffb700 1 -- 172.21.2.102:0/92898347 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(184..184 src has 1..184) v4 ==== 4753+0+0 (secure 0 0 0) 0x7f21fc032600 con 0x7f22041a1bf0 2021-02-10T11:39:27.559 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:27.554 7f22095a8700 1 --2- 172.21.2.102:0/92898347 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f21e40186c0 0x7f21e401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:27.559 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:27.554 7f22095a8700 1 --2- 172.21.2.102:0/92898347 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f21e40186c0 0x7f21e401abc0 crc :-1 s=READY pgs=1408 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:27.587 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:27.585 7f220b78c700 1 -- 172.21.2.102:0/92898347 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22040ee0e0 con 0x7f22041a1bf0 2021-02-10T11:39:27.589 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:27.587 7f21f5ffb700 1 -- 172.21.2.102:0/92898347 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f21fc005ac0 con 0x7f22041a1bf0 2021-02-10T11:39:27.712 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:27.710 7f220b78c700 1 -- 172.21.2.102:0/92898347 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f22040ce450 con 0x7f22041a1bf0 2021-02-10T11:39:27.713 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:27.710 7f21f5ffb700 1 -- 172.21.2.102:0/92898347 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v184) v1 ==== 84+0+64 (secure 0 0 0) 0x7f21fc005cd0 con 0x7f22041a1bf0 2021-02-10T11:39:27.713 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:39:27.713 INFO:teuthology.orchestra.run.gibba002.stdout:[ 2021-02-10T11:39:27.713 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:39:27.713 INFO:teuthology.orchestra.run.gibba002.stdout: "poolnum": 1, 2021-02-10T11:39:27.713 INFO:teuthology.orchestra.run.gibba002.stdout: "poolname": "rbd" 2021-02-10T11:39:27.714 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:39:27.714 INFO:teuthology.orchestra.run.gibba002.stdout:] 2021-02-10T11:39:27.714 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:27.712 7f220b78c700 1 -- 172.21.2.102:0/92898347 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f21e40186c0 msgr2=0x7f21e401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:27.714 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:27.712 7f220b78c700 1 --2- 172.21.2.102:0/92898347 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f21e40186c0 0x7f21e401abc0 crc :-1 s=READY pgs=1408 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:27.715 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:27.712 7f220b78c700 1 -- 172.21.2.102:0/92898347 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f22041a1bf0 msgr2=0x7f22041a1ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:27.715 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:27.712 7f220b78c700 1 --2- 172.21.2.102:0/92898347 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f22041a1bf0 0x7f22041a1ff0 secure :-1 s=READY pgs=3226 cs=0 l=1 rev1=1 rx=0x7f21fc00b970 tx=0x7f21fc00bd10).stop 2021-02-10T11:39:27.715 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:27.713 7f220b78c700 1 -- 172.21.2.102:0/92898347 shutdown_connections 2021-02-10T11:39:27.715 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:27.713 7f220b78c700 1 --2- 172.21.2.102:0/92898347 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f22041a1bf0 0x7f22041a1ff0 unknown :-1 s=CLOSED pgs=3226 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:27.715 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:27.713 7f220b78c700 1 --2- 172.21.2.102:0/92898347 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f22040a6fc0 0x7f22041a15d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:27.716 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:27.713 7f220b78c700 1 --2- 172.21.2.102:0/92898347 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f21e40186c0 0x7f21e401abc0 unknown :-1 s=CLOSED pgs=1408 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:27.716 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:27.713 7f220b78c700 1 --2- 172.21.2.102:0/92898347 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f22041947b0 0x7f22040a69a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:27.716 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:27.713 7f220b78c700 1 -- 172.21.2.102:0/92898347 >> 172.21.2.102:0/92898347 conn(0x7f22041a38c0 msgr2=0x7f22041a2f00 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:27.716 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:27.713 7f220b78c700 1 -- 172.21.2.102:0/92898347 shutdown_connections 2021-02-10T11:39:27.716 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:27.713 7f220b78c700 1 -- 172.21.2.102:0/92898347 wait complete. 2021-02-10T11:39:27.723 INFO:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2021-02-10T11:39:27.724 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool create cephfs_metadata 8 2021-02-10T11:39:28.157 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:28.138 7f1b3e238700 1 Processor -- start 2021-02-10T11:39:28.157 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:28.139 7f1b3e238700 1 -- start start 2021-02-10T11:39:28.157 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:28.139 7f1b3e238700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f1b380d1870 0x7f1b381b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:28.158 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:28.139 7f1b3e238700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1b380cff70 0x7f1b38109130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:28.158 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:28.139 7f1b3e238700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b38109a60 con 0x7f1b381b0ae0 2021-02-10T11:39:28.158 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:28.139 7f1b3e238700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b38109750 con 0x7f1b380d1870 2021-02-10T11:39:28.158 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:28.139 7f1b3e238700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b38088070 con 0x7f1b380cff70 2021-02-10T11:39:28.158 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:28.139 7f1b37fff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f1b380d1870 0x7f1b381b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:28.158 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:28.139 7f1b37fff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f1b380d1870 0x7f1b381b04c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:40342/0 (socket says 172.21.2.102:40342) 2021-02-10T11:39:28.158 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:28.139 7f1b37fff700 1 -- 172.21.2.102:0/2771803166 learned_addr learned my addr 172.21.2.102:0/2771803166 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:39:28.158 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:28.140 7f1b37fff700 1 -- 172.21.2.102:0/2771803166 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1b380cff70 msgr2=0x7f1b38109130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:28.158 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:28.140 7f1b37fff700 1 --2- 172.21.2.102:0/2771803166 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1b380cff70 0x7f1b38109130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:28.158 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:28.140 7f1b37fff700 1 -- 172.21.2.102:0/2771803166 >> v1:172.21.2.113:6789/0 conn(0x7f1b381b0ae0 legacy=0x7f1b38108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:28.158 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:28.140 7f1b37fff700 1 -- 172.21.2.102:0/2771803166 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b380d1c70 con 0x7f1b380d1870 2021-02-10T11:39:28.159 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:28.140 7f1b37fff700 1 --2- 172.21.2.102:0/2771803166 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1b380d1870 0x7f1b381b04c0 secure :-1 s=READY pgs=3227 cs=0 l=1 rev1=1 rx=0x7f1b28006700 tx=0x7f1b28009b00).ready entity=mon.0 client_cookie=ec4a65c65f9bd0a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:28.159 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:28.140 7f1b36ffd700 1 -- 172.21.2.102:0/2771803166 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1b2800c0c0 con 0x7f1b380d1870 2021-02-10T11:39:28.159 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:28.140 7f1b36ffd700 1 -- 172.21.2.102:0/2771803166 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b2800c400 con 0x7f1b380d1870 2021-02-10T11:39:28.159 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:28.140 7f1b36ffd700 1 -- 172.21.2.102:0/2771803166 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1b2800c920 con 0x7f1b380d1870 2021-02-10T11:39:28.159 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:28.140 7f1b3e238700 1 -- 172.21.2.102:0/2771803166 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1b380d1870 msgr2=0x7f1b381b04c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:28.159 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:28.140 7f1b3e238700 1 --2- 172.21.2.102:0/2771803166 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1b380d1870 0x7f1b381b04c0 secure :-1 s=READY pgs=3227 cs=0 l=1 rev1=1 rx=0x7f1b28006700 tx=0x7f1b28009b00).stop 2021-02-10T11:39:28.159 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:28.141 7f1b3e238700 1 -- 172.21.2.102:0/2771803166 shutdown_connections 2021-02-10T11:39:28.159 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:28.141 7f1b3e238700 1 --2- 172.21.2.102:0/2771803166 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1b380cff70 0x7f1b38109130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:28.159 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:28.141 7f1b3e238700 1 --2- 172.21.2.102:0/2771803166 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1b380d1870 0x7f1b381b04c0 unknown :-1 s=CLOSED pgs=3227 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:28.159 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:28.141 7f1b3e238700 1 -- 172.21.2.102:0/2771803166 >> 172.21.2.102:0/2771803166 conn(0x7f1b380ce420 msgr2=0x7f1b380d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:28.159 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:28.141 7f1b3e238700 1 -- 172.21.2.102:0/2771803166 shutdown_connections 2021-02-10T11:39:28.159 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:28.141 7f1b3e238700 1 -- 172.21.2.102:0/2771803166 wait complete. 2021-02-10T11:39:28.161 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:28.155 7f1b3e238700 1 Processor -- start 2021-02-10T11:39:28.161 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:28.156 7f1b3e238700 1 -- start start 2021-02-10T11:39:28.161 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:28.156 7f1b3e238700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1b380c3820 0x7f1b38084cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:28.161 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:28.156 7f1b3e238700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1b380a0290 0x7f1b380852d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:28.162 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:28.156 7f1b3e238700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1b38086270 0x7f1b38086670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:28.162 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:28.156 7f1b3e238700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b380c0700 con 0x7f1b380c3820 2021-02-10T11:39:28.162 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:28.156 7f1b3e238700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b380858f0 con 0x7f1b38086270 2021-02-10T11:39:28.162 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:28.156 7f1b3e238700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b38085b10 con 0x7f1b380a0290 2021-02-10T11:39:28.162 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:28.156 7f1b37fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1b380c3820 0x7f1b38084cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:28.162 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:28.156 7f1b37fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1b380c3820 0x7f1b38084cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:40344/0 (socket says 172.21.2.102:40344) 2021-02-10T11:39:28.162 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:28.156 7f1b37fff700 1 -- 172.21.2.102:0/2088563873 learned_addr learned my addr 172.21.2.102:0/2088563873 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:39:28.162 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:28.157 7f1b37fff700 1 -- 172.21.2.102:0/2088563873 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1b380a0290 msgr2=0x7f1b380852d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:28.162 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:28.157 7f1b3c855700 1 --2- 172.21.2.102:0/2088563873 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1b38086270 0x7f1b38086670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:28.162 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:28.157 7f1b37fff700 1 --2- 172.21.2.102:0/2088563873 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1b380a0290 0x7f1b380852d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:28.162 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:28.157 7f1b37fff700 1 -- 172.21.2.102:0/2088563873 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1b38086270 msgr2=0x7f1b38086670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:28.162 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:28.157 7f1b37fff700 1 --2- 172.21.2.102:0/2088563873 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1b38086270 0x7f1b38086670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:28.163 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:28.157 7f1b37fff700 1 -- 172.21.2.102:0/2088563873 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b38086c90 con 0x7f1b380c3820 2021-02-10T11:39:28.163 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:28.157 7f1b3c855700 1 --2- 172.21.2.102:0/2088563873 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1b38086270 0x7f1b38086670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:39:28.163 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:28.157 7f1b37fff700 1 --2- 172.21.2.102:0/2088563873 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1b380c3820 0x7f1b38084cb0 secure :-1 s=READY pgs=3228 cs=0 l=1 rev1=1 rx=0x7f1b2800b590 tx=0x7f1b2800b930).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:28.163 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:28.157 7f1b34ff9700 1 -- 172.21.2.102:0/2088563873 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1b28006390 con 0x7f1b380c3820 2021-02-10T11:39:28.163 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:28.157 7f1b34ff9700 1 -- 172.21.2.102:0/2088563873 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b2800bd20 con 0x7f1b380c3820 2021-02-10T11:39:28.163 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:28.157 7f1b34ff9700 1 -- 172.21.2.102:0/2088563873 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1b28005560 con 0x7f1b380c3820 2021-02-10T11:39:28.163 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:28.157 7f1b3e238700 1 -- 172.21.2.102:0/2088563873 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b38086c90 con 0x7f1b380c3820 2021-02-10T11:39:28.163 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:28.157 7f1b3e238700 1 -- 172.21.2.102:0/2088563873 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b38086c90 con 0x7f1b380c3820 2021-02-10T11:39:28.163 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:28.158 7f1b34ff9700 1 -- 172.21.2.102:0/2088563873 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1b280062d0 con 0x7f1b380c3820 2021-02-10T11:39:28.163 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:28.158 7f1b34ff9700 1 --2- 172.21.2.102:0/2088563873 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1b180186c0 0x7f1b1801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:28.163 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:28.158 7f1b34ff9700 1 -- 172.21.2.102:0/2088563873 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(184..184 src has 1..184) v4 ==== 4753+0+0 (secure 0 0 0) 0x7f1b28032250 con 0x7f1b380c3820 2021-02-10T11:39:28.163 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:28.158 7f1b377fe700 1 --2- 172.21.2.102:0/2088563873 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1b180186c0 0x7f1b1801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:28.163 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:28.159 7f1b377fe700 1 --2- 172.21.2.102:0/2088563873 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1b180186c0 0x7f1b1801abc0 crc :-1 s=READY pgs=1409 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:28.191 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:28.189 7f1b3e238700 1 -- 172.21.2.102:0/2088563873 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b38082a70 con 0x7f1b380c3820 2021-02-10T11:39:28.194 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:28.192 7f1b34ff9700 1 -- 172.21.2.102:0/2088563873 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f1b280250d0 con 0x7f1b380c3820 2021-02-10T11:39:28.323 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:28.321 7f1b3e238700 1 -- 172.21.2.102:0/2088563873 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"} v 0) v1 -- 0x7f1b38082a70 con 0x7f1b380c3820 2021-02-10T11:39:28.734 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:28.732 7f1b34ff9700 1 -- 172.21.2.102:0/2088563873 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"}]=0 pool 'cephfs_metadata' created v185) v1 ==== 133+0+0 (secure 0 0 0) 0x7f1b280250d0 con 0x7f1b380c3820 2021-02-10T11:39:28.739 INFO:teuthology.orchestra.run.gibba002.stderr:pool 'cephfs_metadata' created 2021-02-10T11:39:28.740 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:28.738 7f1b3e238700 1 -- 172.21.2.102:0/2088563873 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1b180186c0 msgr2=0x7f1b1801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:28.740 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:28.738 7f1b3e238700 1 --2- 172.21.2.102:0/2088563873 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1b180186c0 0x7f1b1801abc0 crc :-1 s=READY pgs=1409 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:28.741 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:28.738 7f1b3e238700 1 -- 172.21.2.102:0/2088563873 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1b380c3820 msgr2=0x7f1b38084cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:28.741 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:28.738 7f1b3e238700 1 --2- 172.21.2.102:0/2088563873 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1b380c3820 0x7f1b38084cb0 secure :-1 s=READY pgs=3228 cs=0 l=1 rev1=1 rx=0x7f1b2800b590 tx=0x7f1b2800b930).stop 2021-02-10T11:39:28.741 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:28.738 7f1b3e238700 1 -- 172.21.2.102:0/2088563873 shutdown_connections 2021-02-10T11:39:28.741 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:28.738 7f1b3e238700 1 --2- 172.21.2.102:0/2088563873 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1b180186c0 0x7f1b1801abc0 unknown :-1 s=CLOSED pgs=1409 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:28.741 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:28.738 7f1b3e238700 1 --2- 172.21.2.102:0/2088563873 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1b38086270 0x7f1b38086670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:28.741 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:28.738 7f1b3e238700 1 --2- 172.21.2.102:0/2088563873 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1b380a0290 0x7f1b380852d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:28.741 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:28.738 7f1b3e238700 1 --2- 172.21.2.102:0/2088563873 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1b380c3820 0x7f1b38084cb0 unknown :-1 s=CLOSED pgs=3228 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:28.741 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:28.738 7f1b3e238700 1 -- 172.21.2.102:0/2088563873 >> 172.21.2.102:0/2088563873 conn(0x7f1b380872c0 msgr2=0x7f1b38095700 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:28.741 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:28.738 7f1b3e238700 1 -- 172.21.2.102:0/2088563873 shutdown_connections 2021-02-10T11:39:28.741 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:28.738 7f1b3e238700 1 -- 172.21.2.102:0/2088563873 wait complete. 2021-02-10T11:39:28.749 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool create cephfs_data 8 2021-02-10T11:39:29.181 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:29.162 7f73d4af5700 1 Processor -- start 2021-02-10T11:39:29.182 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:29.162 7f73d4af5700 1 -- start start 2021-02-10T11:39:29.182 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:29.163 7f73d4af5700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f73d00d1870 0x7f73d01b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:29.182 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:29.163 7f73d4af5700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f73d01b0ae0 0x7f73d0108940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:29.182 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:29.163 7f73d4af5700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f73d00d1c70 0x7f73d0108f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:29.182 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:29.163 7f73d4af5700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73d00fe1c0 con 0x7f73d01b0ae0 2021-02-10T11:39:29.182 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:29.163 7f73d4af5700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73d0109a60 con 0x7f73d00d1870 2021-02-10T11:39:29.182 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:29.163 7f73d4af5700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73d0088070 con 0x7f73d00d1c70 2021-02-10T11:39:29.182 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:29.163 7f73ce05d700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f73d01b0ae0 0x7f73d0108940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:29.182 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:29.163 7f73ce05d700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f73d01b0ae0 0x7f73d0108940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:40352/0 (socket says 172.21.2.102:40352) 2021-02-10T11:39:29.182 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:29.163 7f73ce05d700 1 -- 172.21.2.102:0/2280220119 learned_addr learned my addr 172.21.2.102:0/2280220119 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:39:29.183 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:29.163 7f73ce85e700 1 --2- 172.21.2.102:0/2280220119 >> v2:172.21.2.113:3300/0 conn(0x7f73d00d1870 0x7f73d01b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:29.183 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:29.163 7f73cf05f700 1 --2- 172.21.2.102:0/2280220119 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f73d00d1c70 0x7f73d0108f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:29.183 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:29.164 7f73ce05d700 1 -- 172.21.2.102:0/2280220119 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f73d00d1c70 msgr2=0x7f73d0108f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:29.183 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:29.164 7f73ce05d700 1 --2- 172.21.2.102:0/2280220119 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f73d00d1c70 0x7f73d0108f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:29.183 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:29.164 7f73ce05d700 1 -- 172.21.2.102:0/2280220119 >> v2:172.21.2.113:3300/0 conn(0x7f73d00d1870 msgr2=0x7f73d01b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:29.183 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:29.164 7f73ce05d700 1 --2- 172.21.2.102:0/2280220119 >> v2:172.21.2.113:3300/0 conn(0x7f73d00d1870 0x7f73d01b04c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:29.183 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:29.164 7f73ce05d700 1 -- 172.21.2.102:0/2280220119 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73d0109580 con 0x7f73d01b0ae0 2021-02-10T11:39:29.183 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:29.164 7f73ce05d700 1 --2- 172.21.2.102:0/2280220119 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f73d01b0ae0 0x7f73d0108940 secure :-1 s=READY pgs=3229 cs=0 l=1 rev1=1 rx=0x7f73c4006700 tx=0x7f73c4009b80).ready entity=mon.0 client_cookie=94025e2c7ee261ba server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:29.183 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:29.164 7f73cd85c700 1 -- 172.21.2.102:0/2280220119 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f73c400c0c0 con 0x7f73d01b0ae0 2021-02-10T11:39:29.183 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:29.164 7f73cd85c700 1 -- 172.21.2.102:0/2280220119 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73c400c440 con 0x7f73d01b0ae0 2021-02-10T11:39:29.183 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:29.164 7f73cd85c700 1 -- 172.21.2.102:0/2280220119 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f73c400c960 con 0x7f73d01b0ae0 2021-02-10T11:39:29.183 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:29.164 7f73d4af5700 1 -- 172.21.2.102:0/2280220119 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f73d01b0ae0 msgr2=0x7f73d0108940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:29.183 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:29.164 7f73d4af5700 1 --2- 172.21.2.102:0/2280220119 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f73d01b0ae0 0x7f73d0108940 secure :-1 s=READY pgs=3229 cs=0 l=1 rev1=1 rx=0x7f73c4006700 tx=0x7f73c4009b80).stop 2021-02-10T11:39:29.183 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:29.165 7f73d4af5700 1 -- 172.21.2.102:0/2280220119 shutdown_connections 2021-02-10T11:39:29.184 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:29.165 7f73d4af5700 1 --2- 172.21.2.102:0/2280220119 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f73d00d1c70 0x7f73d0108f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:29.184 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:29.165 7f73d4af5700 1 --2- 172.21.2.102:0/2280220119 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f73d01b0ae0 0x7f73d0108940 unknown :-1 s=CLOSED pgs=3229 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:29.184 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:29.165 7f73d4af5700 1 --2- 172.21.2.102:0/2280220119 >> v2:172.21.2.113:3300/0 conn(0x7f73d00d1870 0x7f73d01b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:29.184 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:29.165 7f73d4af5700 1 -- 172.21.2.102:0/2280220119 >> 172.21.2.102:0/2280220119 conn(0x7f73d00ce420 msgr2=0x7f73d00d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:29.184 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:29.165 7f73d4af5700 1 -- 172.21.2.102:0/2280220119 shutdown_connections 2021-02-10T11:39:29.184 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:29.165 7f73d4af5700 1 -- 172.21.2.102:0/2280220119 wait complete. 2021-02-10T11:39:29.185 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:29.180 7f73d4af5700 1 Processor -- start 2021-02-10T11:39:29.186 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:29.180 7f73d4af5700 1 -- start start 2021-02-10T11:39:29.186 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:29.180 7f73d4af5700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f73d01a79a0 0x7f73d01a7da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:29.186 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:29.180 7f73d4af5700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f73d00bdb40 0x7f73d00bdf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:29.186 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:29.180 7f73d4af5700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f73d0082ca0 0x7f73d00830a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:29.186 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:29.180 7f73d4af5700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73d019d990 con 0x7f73d01a79a0 2021-02-10T11:39:29.186 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:29.180 7f73d4af5700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73d019dbb0 con 0x7f73d00bdb40 2021-02-10T11:39:29.186 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:29.180 7f73d4af5700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73d00836c0 con 0x7f73d0082ca0 2021-02-10T11:39:29.186 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:29.180 7f73ce85e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f73d01a79a0 0x7f73d01a7da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:29.186 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:29.180 7f73ce85e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f73d01a79a0 0x7f73d01a7da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:40356/0 (socket says 172.21.2.102:40356) 2021-02-10T11:39:29.186 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:29.180 7f73ce85e700 1 -- 172.21.2.102:0/2392549105 learned_addr learned my addr 172.21.2.102:0/2392549105 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:39:29.186 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:29.181 7f73ce85e700 1 -- 172.21.2.102:0/2392549105 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f73d0082ca0 msgr2=0x7f73d00830a0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:39:29.186 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:29.181 7f73ce85e700 1 --2- 172.21.2.102:0/2392549105 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f73d0082ca0 0x7f73d00830a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:29.186 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:29.181 7f73ce85e700 1 -- 172.21.2.102:0/2392549105 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f73d00bdb40 msgr2=0x7f73d00bdf40 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:39:29.187 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:29.181 7f73ce85e700 1 --2- 172.21.2.102:0/2392549105 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f73d00bdb40 0x7f73d00bdf40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:29.187 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:29.181 7f73ce85e700 1 -- 172.21.2.102:0/2392549105 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73d00838e0 con 0x7f73d01a79a0 2021-02-10T11:39:29.187 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:29.181 7f73ce85e700 1 --2- 172.21.2.102:0/2392549105 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f73d01a79a0 0x7f73d01a7da0 secure :-1 s=READY pgs=3230 cs=0 l=1 rev1=1 rx=0x7f73c0006f60 tx=0x7f73c000b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:29.187 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:29.181 7f73bf7fe700 1 -- 172.21.2.102:0/2392549105 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f73c000bd60 con 0x7f73d01a79a0 2021-02-10T11:39:29.187 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:29.181 7f73bf7fe700 1 -- 172.21.2.102:0/2392549105 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73c0000c90 con 0x7f73d01a79a0 2021-02-10T11:39:29.187 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:29.181 7f73bf7fe700 1 -- 172.21.2.102:0/2392549105 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f73c00022b0 con 0x7f73d01a79a0 2021-02-10T11:39:29.187 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:29.181 7f73d4af5700 1 -- 172.21.2.102:0/2392549105 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73d00838e0 con 0x7f73d01a79a0 2021-02-10T11:39:29.187 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:29.181 7f73d4af5700 1 -- 172.21.2.102:0/2392549105 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73d00838e0 con 0x7f73d01a79a0 2021-02-10T11:39:29.187 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:29.182 7f73bf7fe700 1 -- 172.21.2.102:0/2392549105 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f73c0006b00 con 0x7f73d01a79a0 2021-02-10T11:39:29.187 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:29.182 7f73bf7fe700 1 --2- 172.21.2.102:0/2392549105 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f73b40186c0 0x7f73b401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:29.187 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:29.182 7f73bf7fe700 1 -- 172.21.2.102:0/2392549105 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(186..186 src has 1..186) v4 ==== 5107+0+0 (secure 0 0 0) 0x7f73c0032380 con 0x7f73d01a79a0 2021-02-10T11:39:29.187 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:29.182 7f73ce05d700 1 --2- 172.21.2.102:0/2392549105 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f73b40186c0 0x7f73b401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:29.187 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:29.182 7f73ce05d700 1 --2- 172.21.2.102:0/2392549105 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f73b40186c0 0x7f73b401abc0 crc :-1 s=READY pgs=1410 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:29.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:29.213 7f73d4af5700 1 -- 172.21.2.102:0/2392549105 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73d0082a70 con 0x7f73d01a79a0 2021-02-10T11:39:29.218 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:29.216 7f73bf7fe700 1 -- 172.21.2.102:0/2392549105 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f73c0019430 con 0x7f73d01a79a0 2021-02-10T11:39:29.350 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:29.348 7f73d4af5700 1 -- 172.21.2.102:0/2392549105 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_data"} v 0) v1 -- 0x7f73d0082a70 con 0x7f73d01a79a0 2021-02-10T11:39:29.743 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:29.740 7f73bf7fe700 1 -- 172.21.2.102:0/2392549105 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_data"}]=0 pool 'cephfs_data' created v187) v1 ==== 125+0+0 (secure 0 0 0) 0x7f73c0019430 con 0x7f73d01a79a0 2021-02-10T11:39:29.767 INFO:teuthology.orchestra.run.gibba002.stderr:pool 'cephfs_data' created 2021-02-10T11:39:29.768 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:29.766 7f73d4af5700 1 -- 172.21.2.102:0/2392549105 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f73b40186c0 msgr2=0x7f73b401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:29.769 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:29.766 7f73d4af5700 1 --2- 172.21.2.102:0/2392549105 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f73b40186c0 0x7f73b401abc0 crc :-1 s=READY pgs=1410 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:29.769 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:29.766 7f73d4af5700 1 -- 172.21.2.102:0/2392549105 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f73d01a79a0 msgr2=0x7f73d01a7da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:29.769 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:29.766 7f73d4af5700 1 --2- 172.21.2.102:0/2392549105 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f73d01a79a0 0x7f73d01a7da0 secure :-1 s=READY pgs=3230 cs=0 l=1 rev1=1 rx=0x7f73c0006f60 tx=0x7f73c000b970).stop 2021-02-10T11:39:29.769 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:29.766 7f73d4af5700 1 -- 172.21.2.102:0/2392549105 shutdown_connections 2021-02-10T11:39:29.769 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:29.766 7f73d4af5700 1 --2- 172.21.2.102:0/2392549105 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f73d0082ca0 0x7f73d00830a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:29.769 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:29.766 7f73d4af5700 1 --2- 172.21.2.102:0/2392549105 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f73b40186c0 0x7f73b401abc0 unknown :-1 s=CLOSED pgs=1410 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:29.769 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:29.766 7f73d4af5700 1 --2- 172.21.2.102:0/2392549105 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f73d00bdb40 0x7f73d00bdf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:29.769 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:29.766 7f73d4af5700 1 --2- 172.21.2.102:0/2392549105 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f73d01a79a0 0x7f73d01a7da0 unknown :-1 s=CLOSED pgs=3230 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:29.769 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:29.766 7f73d4af5700 1 -- 172.21.2.102:0/2392549105 >> 172.21.2.102:0/2392549105 conn(0x7f73d00bef00 msgr2=0x7f73d009f200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:29.769 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:29.766 7f73d4af5700 1 -- 172.21.2.102:0/2392549105 shutdown_connections 2021-02-10T11:39:29.769 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:29.766 7f73d4af5700 1 -- 172.21.2.102:0/2392549105 wait complete. 2021-02-10T11:39:29.777 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2021-02-10T11:39:30.210 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:30.191 7f5ef8746700 1 Processor -- start 2021-02-10T11:39:30.211 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:30.191 7f5ef8746700 1 -- start start 2021-02-10T11:39:30.211 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:30.192 7f5ef8746700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5ef00d1880 0x7f5ef01b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:30.211 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:30.192 7f5ef8746700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f5ef00cff80 0x7f5ef0109140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:30.211 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:30.192 7f5ef8746700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ef0109a70 con 0x7f5ef01b0af0 2021-02-10T11:39:30.211 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:30.192 7f5ef8746700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ef0109760 con 0x7f5ef00cff80 2021-02-10T11:39:30.211 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:30.192 7f5ef8746700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ef0088070 con 0x7f5ef00d1880 2021-02-10T11:39:30.211 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:30.192 7f5ef6562700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5ef00d1880 0x7f5ef01b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:30.211 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:30.192 7f5ef5d61700 1 --1- >> v1:172.21.2.113:6789/0 conn(0x7f5ef01b0af0 0x7f5ef0108950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.113:6789/0 says I am v1:172.21.2.102:39914/0 (socket says 172.21.2.102:39914) 2021-02-10T11:39:30.211 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:30.192 7f5ef6562700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5ef00d1880 0x7f5ef01b04d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.113:3301/0 says I am v2:172.21.2.102:45902/0 (socket says 172.21.2.102:45902) 2021-02-10T11:39:30.211 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:30.192 7f5ef5d61700 1 -- 172.21.2.102:0/1054906311 learned_addr learned my addr 172.21.2.102:0/1054906311 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:39:30.211 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:30.192 7f5ef6d63700 1 --2- 172.21.2.102:0/1054906311 >> v2:172.21.2.113:3300/0 conn(0x7f5ef00cff80 0x7f5ef0109140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:30.211 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:30.193 7f5ef6562700 1 -- 172.21.2.102:0/1054906311 >> v2:172.21.2.113:3300/0 conn(0x7f5ef00cff80 msgr2=0x7f5ef0109140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:30.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:30.193 7f5ef6562700 1 --2- 172.21.2.102:0/1054906311 >> v2:172.21.2.113:3300/0 conn(0x7f5ef00cff80 0x7f5ef0109140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:30.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:30.193 7f5ef6562700 1 -- 172.21.2.102:0/1054906311 >> v1:172.21.2.113:6789/0 conn(0x7f5ef01b0af0 legacy=0x7f5ef0108950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:30.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:30.193 7f5ef6562700 1 -- 172.21.2.102:0/1054906311 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ef0088260 con 0x7f5ef00d1880 2021-02-10T11:39:30.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:30.193 7f5ef6562700 1 --2- 172.21.2.102:0/1054906311 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5ef00d1880 0x7f5ef01b04d0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 rx=0x7f5ee4006650 tx=0x7f5ee4009a80).ready entity=mon.2 client_cookie=d0e537c1adce4e3 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:30.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:30.193 7f5ef5560700 1 -- 172.21.2.102:0/1054906311 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5ee400c0c0 con 0x7f5ef00d1880 2021-02-10T11:39:30.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:30.193 7f5ef5560700 1 -- 172.21.2.102:0/1054906311 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ee400c340 con 0x7f5ef00d1880 2021-02-10T11:39:30.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:30.193 7f5ef5560700 1 -- 172.21.2.102:0/1054906311 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5ee400c860 con 0x7f5ef00d1880 2021-02-10T11:39:30.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:30.193 7f5ef8746700 1 -- 172.21.2.102:0/1054906311 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5ef00d1880 msgr2=0x7f5ef01b04d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:30.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:30.193 7f5ef8746700 1 --2- 172.21.2.102:0/1054906311 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5ef00d1880 0x7f5ef01b04d0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 rx=0x7f5ee4006650 tx=0x7f5ee4009a80).stop 2021-02-10T11:39:30.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:30.194 7f5ef8746700 1 -- 172.21.2.102:0/1054906311 shutdown_connections 2021-02-10T11:39:30.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:30.194 7f5ef8746700 1 --2- 172.21.2.102:0/1054906311 >> v2:172.21.2.113:3300/0 conn(0x7f5ef00cff80 0x7f5ef0109140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:30.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:30.194 7f5ef8746700 1 --2- 172.21.2.102:0/1054906311 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5ef00d1880 0x7f5ef01b04d0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:30.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:30.194 7f5ef8746700 1 -- 172.21.2.102:0/1054906311 >> 172.21.2.102:0/1054906311 conn(0x7f5ef00ce430 msgr2=0x7f5ef00d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:30.213 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:30.194 7f5ef8746700 1 -- 172.21.2.102:0/1054906311 shutdown_connections 2021-02-10T11:39:30.213 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:30.194 7f5ef8746700 1 -- 172.21.2.102:0/1054906311 wait complete. 2021-02-10T11:39:30.214 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:30.209 7f5ef8746700 1 Processor -- start 2021-02-10T11:39:30.214 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:30.209 7f5ef8746700 1 -- start start 2021-02-10T11:39:30.214 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:30.209 7f5ef8746700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5ef009a690 0x7f5ef009aa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:30.214 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:30.209 7f5ef8746700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5ef01b4160 0x7f5ef01b4560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:30.214 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:30.209 7f5ef8746700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5ef01b7dd0 0x7f5ef01b81d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:30.214 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:30.209 7f5ef8746700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ef01b4b80 con 0x7f5ef01b7dd0 2021-02-10T11:39:30.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:30.209 7f5ef8746700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ef01b87f0 con 0x7f5ef01b4160 2021-02-10T11:39:30.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:30.209 7f5ef8746700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ef01b8a10 con 0x7f5ef009a690 2021-02-10T11:39:30.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:30.209 7f5ef6d63700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5ef01b7dd0 0x7f5ef01b81d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:30.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:30.209 7f5ef6d63700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5ef01b7dd0 0x7f5ef01b81d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:40376/0 (socket says 172.21.2.102:40376) 2021-02-10T11:39:30.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:30.209 7f5ef6d63700 1 -- 172.21.2.102:0/3071447966 learned_addr learned my addr 172.21.2.102:0/3071447966 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:39:30.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:30.210 7f5ef5d61700 1 --2- 172.21.2.102:0/3071447966 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5ef01b4160 0x7f5ef01b4560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:30.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:30.210 7f5ef6d63700 1 -- 172.21.2.102:0/3071447966 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5ef009a690 msgr2=0x7f5ef009aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:30.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:30.210 7f5ef6d63700 1 --2- 172.21.2.102:0/3071447966 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5ef009a690 0x7f5ef009aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:30.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:30.210 7f5ef6d63700 1 -- 172.21.2.102:0/3071447966 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5ef01b4160 msgr2=0x7f5ef01b4560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:30.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:30.210 7f5ef6d63700 1 --2- 172.21.2.102:0/3071447966 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5ef01b4160 0x7f5ef01b4560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:30.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:30.210 7f5ef6d63700 1 -- 172.21.2.102:0/3071447966 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ef01b8c30 con 0x7f5ef01b7dd0 2021-02-10T11:39:30.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:30.210 7f5ef6d63700 1 --2- 172.21.2.102:0/3071447966 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5ef01b7dd0 0x7f5ef01b81d0 secure :-1 s=READY pgs=3231 cs=0 l=1 rev1=1 rx=0x7f5ee800b970 tx=0x7f5ee800bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:30.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:30.210 7f5ee2ffd700 1 -- 172.21.2.102:0/3071447966 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5ee8000f90 con 0x7f5ef01b7dd0 2021-02-10T11:39:30.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:30.210 7f5ee2ffd700 1 -- 172.21.2.102:0/3071447966 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ee8003030 con 0x7f5ef01b7dd0 2021-02-10T11:39:30.216 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:30.210 7f5ee2ffd700 1 -- 172.21.2.102:0/3071447966 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5ee8001400 con 0x7f5ef01b7dd0 2021-02-10T11:39:30.216 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:30.210 7f5ef8746700 1 -- 172.21.2.102:0/3071447966 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ef01b8c30 con 0x7f5ef01b7dd0 2021-02-10T11:39:30.216 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:30.210 7f5ef8746700 1 -- 172.21.2.102:0/3071447966 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ef01b91c0 con 0x7f5ef01b7dd0 2021-02-10T11:39:30.216 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:30.211 7f5ee2ffd700 1 -- 172.21.2.102:0/3071447966 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f5ee80011f0 con 0x7f5ef01b7dd0 2021-02-10T11:39:30.216 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:30.211 7f5ee2ffd700 1 --2- 172.21.2.102:0/3071447966 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5ed80186c0 0x7f5ed801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:30.216 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:30.211 7f5ee2ffd700 1 -- 172.21.2.102:0/3071447966 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(187..187 src has 1..187) v4 ==== 5457+0+0 (secure 0 0 0) 0x7f5ee80326d0 con 0x7f5ef01b7dd0 2021-02-10T11:39:30.216 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:30.211 7f5ef6562700 1 --2- 172.21.2.102:0/3071447966 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5ed80186c0 0x7f5ed801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:30.216 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:30.212 7f5ef6562700 1 --2- 172.21.2.102:0/3071447966 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5ed80186c0 0x7f5ed801abc0 crc :-1 s=READY pgs=1413 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:30.244 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:30.242 7f5ef8746700 1 -- 172.21.2.102:0/3071447966 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ef009a160 con 0x7f5ef01b7dd0 2021-02-10T11:39:30.246 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:30.244 7f5ee2ffd700 1 -- 172.21.2.102:0/3071447966 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f5ee8000f40 con 0x7f5ef01b7dd0 2021-02-10T11:39:30.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:30.373 7f5ef8746700 1 -- 172.21.2.102:0/3071447966 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"data": "cephfs_data", "prefix": "fs new", "force": true, "fs_name": "cephfs", "metadata": "cephfs_metadata"} v 0) v1 -- 0x7f5ef009a160 con 0x7f5ef01b7dd0 2021-02-10T11:39:30.381 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:30.379 7f5ee2ffd700 1 -- 172.21.2.102:0/3071447966 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"data": "cephfs_data", "prefix": "fs new", "force": true, "fs_name": "cephfs", "metadata": "cephfs_metadata"}]=0 new fs with metadata pool 40 and data pool 41 v163) v1 ==== 189+0+0 (secure 0 0 0) 0x7f5ee80015a0 con 0x7f5ef01b7dd0 2021-02-10T11:39:30.383 INFO:teuthology.orchestra.run.gibba002.stderr:new fs with metadata pool 40 and data pool 41 2021-02-10T11:39:30.385 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:30.383 7f5ef8746700 1 -- 172.21.2.102:0/3071447966 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5ed80186c0 msgr2=0x7f5ed801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:30.385 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:30.383 7f5ef8746700 1 --2- 172.21.2.102:0/3071447966 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5ed80186c0 0x7f5ed801abc0 crc :-1 s=READY pgs=1413 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:30.385 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:30.383 7f5ef8746700 1 -- 172.21.2.102:0/3071447966 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5ef01b7dd0 msgr2=0x7f5ef01b81d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:30.385 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:30.383 7f5ef8746700 1 --2- 172.21.2.102:0/3071447966 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5ef01b7dd0 0x7f5ef01b81d0 secure :-1 s=READY pgs=3231 cs=0 l=1 rev1=1 rx=0x7f5ee800b970 tx=0x7f5ee800bd10).stop 2021-02-10T11:39:30.386 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:30.383 7f5ef8746700 1 -- 172.21.2.102:0/3071447966 shutdown_connections 2021-02-10T11:39:30.386 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:30.383 7f5ef8746700 1 --2- 172.21.2.102:0/3071447966 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5ef01b7dd0 0x7f5ef01b81d0 unknown :-1 s=CLOSED pgs=3231 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:30.386 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:30.383 7f5ef8746700 1 --2- 172.21.2.102:0/3071447966 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5ed80186c0 0x7f5ed801abc0 unknown :-1 s=CLOSED pgs=1413 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:30.386 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:30.383 7f5ef8746700 1 --2- 172.21.2.102:0/3071447966 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5ef01b4160 0x7f5ef01b4560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:30.386 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:30.383 7f5ef8746700 1 --2- 172.21.2.102:0/3071447966 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5ef009a690 0x7f5ef009aa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:30.386 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:30.383 7f5ef8746700 1 -- 172.21.2.102:0/3071447966 >> 172.21.2.102:0/3071447966 conn(0x7f5ef018e870 msgr2=0x7f5ef018ecc0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:30.386 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:30.383 7f5ef8746700 1 -- 172.21.2.102:0/3071447966 shutdown_connections 2021-02-10T11:39:30.386 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:30.383 7f5ef8746700 1 -- 172.21.2.102:0/3071447966 wait complete. 2021-02-10T11:39:30.394 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:39:30.823 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:30.805 7f97e954a700 1 Processor -- start 2021-02-10T11:39:30.824 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:30.805 7f97e954a700 1 -- start start 2021-02-10T11:39:30.824 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:30.806 7f97e954a700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f97e40d1870 0x7f97e41b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:30.824 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:30.806 7f97e954a700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f97e40f0650 0x7f97e40cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:30.824 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:30.806 7f97e954a700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97e40f0b10 con 0x7f97e41b0a10 2021-02-10T11:39:30.824 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:30.806 7f97e954a700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97e40fe230 con 0x7f97e40f0650 2021-02-10T11:39:30.824 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:30.806 7f97e954a700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97e40e5ed0 con 0x7f97e40d1870 2021-02-10T11:39:30.824 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:30.806 7f97e3ac0700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f97e40f0650 0x7f97e40cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:30.824 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:30.806 7f97e3ac0700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f97e40f0650 0x7f97e40cfb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:40390/0 (socket says 172.21.2.102:40390) 2021-02-10T11:39:30.824 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:30.806 7f97e3ac0700 1 -- 172.21.2.102:0/2557219359 learned_addr learned my addr 172.21.2.102:0/2557219359 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:39:30.824 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:30.806 7f97e2abe700 1 --1- 172.21.2.102:0/2557219359 >> v1:172.21.2.102:6789/0 conn(0x7f97e41b0a10 0x7f97e40cf330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:60846/0 (socket says 172.21.2.102:60846) 2021-02-10T11:39:30.824 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:30.806 7f97e32bf700 1 --2- 172.21.2.102:0/2557219359 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f97e40d1870 0x7f97e41b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:30.825 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:30.806 7f97e3ac0700 1 -- 172.21.2.102:0/2557219359 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f97e40d1870 msgr2=0x7f97e41b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:30.825 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:30.806 7f97e3ac0700 1 --2- 172.21.2.102:0/2557219359 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f97e40d1870 0x7f97e41b03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:30.825 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:30.806 7f97e3ac0700 1 -- 172.21.2.102:0/2557219359 >> v1:172.21.2.102:6789/0 conn(0x7f97e41b0a10 legacy=0x7f97e40cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:30.825 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:30.806 7f97e3ac0700 1 -- 172.21.2.102:0/2557219359 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97e40f20a0 con 0x7f97e40f0650 2021-02-10T11:39:30.825 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:30.806 7f97e3ac0700 1 --2- 172.21.2.102:0/2557219359 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f97e40f0650 0x7f97e40cfb20 secure :-1 s=READY pgs=3233 cs=0 l=1 rev1=1 rx=0x7f97dc006f00 tx=0x7f97dc009f10).ready entity=mon.0 client_cookie=14ad71ac3b4e783a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:30.825 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:30.806 7f97e22bd700 1 -- 172.21.2.102:0/2557219359 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f97dc00b3b0 con 0x7f97e40f0650 2021-02-10T11:39:30.825 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:30.807 7f97e22bd700 1 -- 172.21.2.102:0/2557219359 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97dc00b630 con 0x7f97e40f0650 2021-02-10T11:39:30.825 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:30.807 7f97e22bd700 1 -- 172.21.2.102:0/2557219359 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f97dc00bb10 con 0x7f97e40f0650 2021-02-10T11:39:30.825 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:30.807 7f97e954a700 1 -- 172.21.2.102:0/2557219359 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f97e40f0650 msgr2=0x7f97e40cfb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:30.825 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:30.807 7f97e954a700 1 --2- 172.21.2.102:0/2557219359 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f97e40f0650 0x7f97e40cfb20 secure :-1 s=READY pgs=3233 cs=0 l=1 rev1=1 rx=0x7f97dc006f00 tx=0x7f97dc009f10).stop 2021-02-10T11:39:30.825 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:30.807 7f97e954a700 1 -- 172.21.2.102:0/2557219359 shutdown_connections 2021-02-10T11:39:30.826 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:30.807 7f97e954a700 1 --2- 172.21.2.102:0/2557219359 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f97e40f0650 0x7f97e40cfb20 unknown :-1 s=CLOSED pgs=3233 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:30.826 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:30.807 7f97e954a700 1 --2- 172.21.2.102:0/2557219359 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f97e40d1870 0x7f97e41b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:30.826 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:30.807 7f97e954a700 1 -- 172.21.2.102:0/2557219359 >> 172.21.2.102:0/2557219359 conn(0x7f97e40ce450 msgr2=0x7f97e40d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:30.826 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:30.807 7f97e954a700 1 -- 172.21.2.102:0/2557219359 shutdown_connections 2021-02-10T11:39:30.826 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:30.807 7f97e954a700 1 -- 172.21.2.102:0/2557219359 wait complete. 2021-02-10T11:39:30.827 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:30.822 7f97e954a700 1 Processor -- start 2021-02-10T11:39:30.827 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:30.822 7f97e954a700 1 -- start start 2021-02-10T11:39:30.827 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:30.822 7f97e954a700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f97e41a4a60 0x7f97e40c1c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:30.827 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:30.822 7f97e954a700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f97e40c2290 0x7f97e40c2690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:30.828 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:30.822 7f97e954a700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f97e40c2cb0 0x7f97e40bfd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:30.828 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:30.822 7f97e954a700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97e40f05b0 con 0x7f97e41a4a60 2021-02-10T11:39:30.828 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:30.822 7f97e954a700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97e40f07d0 con 0x7f97e40c2cb0 2021-02-10T11:39:30.828 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:30.822 7f97e954a700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97e40c0340 con 0x7f97e40c2290 2021-02-10T11:39:30.828 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:30.822 7f97e32bf700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f97e41a4a60 0x7f97e40c1c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:30.828 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:30.823 7f97e32bf700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f97e41a4a60 0x7f97e40c1c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:40394/0 (socket says 172.21.2.102:40394) 2021-02-10T11:39:30.828 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:30.823 7f97e32bf700 1 -- 172.21.2.102:0/888532909 learned_addr learned my addr 172.21.2.102:0/888532909 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:39:30.828 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:30.823 7f97e32bf700 1 -- 172.21.2.102:0/888532909 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f97e40c2290 msgr2=0x7f97e40c2690 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:39:30.828 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:30.823 7f97e32bf700 1 --2- 172.21.2.102:0/888532909 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f97e40c2290 0x7f97e40c2690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:30.828 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:30.823 7f97e32bf700 1 -- 172.21.2.102:0/888532909 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f97e40c2cb0 msgr2=0x7f97e40bfd20 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:39:30.828 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:30.823 7f97e32bf700 1 --2- 172.21.2.102:0/888532909 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f97e40c2cb0 0x7f97e40bfd20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:30.828 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:30.823 7f97e32bf700 1 -- 172.21.2.102:0/888532909 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97e40c0560 con 0x7f97e41a4a60 2021-02-10T11:39:30.829 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:30.823 7f97e32bf700 1 --2- 172.21.2.102:0/888532909 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f97e41a4a60 0x7f97e40c1c70 secure :-1 s=READY pgs=3234 cs=0 l=1 rev1=1 rx=0x7f97d4006f60 tx=0x7f97d400b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:30.829 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:30.823 7f97dbfff700 1 -- 172.21.2.102:0/888532909 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f97d400bd60 con 0x7f97e41a4a60 2021-02-10T11:39:30.829 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:30.823 7f97dbfff700 1 -- 172.21.2.102:0/888532909 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97d4000c90 con 0x7f97e41a4a60 2021-02-10T11:39:30.829 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:30.823 7f97e954a700 1 -- 172.21.2.102:0/888532909 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97e40c0560 con 0x7f97e41a4a60 2021-02-10T11:39:30.829 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:30.823 7f97e954a700 1 -- 172.21.2.102:0/888532909 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97e40c0af0 con 0x7f97e41a4a60 2021-02-10T11:39:30.829 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:30.823 7f97dbfff700 1 -- 172.21.2.102:0/888532909 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f97d4002220 con 0x7f97e41a4a60 2021-02-10T11:39:30.829 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:30.824 7f97dbfff700 1 -- 172.21.2.102:0/888532909 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f97d40052e0 con 0x7f97e41a4a60 2021-02-10T11:39:30.829 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:30.824 7f97dbfff700 1 --2- 172.21.2.102:0/888532909 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f97c80186c0 0x7f97c801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:30.829 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:30.824 7f97dbfff700 1 -- 172.21.2.102:0/888532909 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f97d4032550 con 0x7f97e41a4a60 2021-02-10T11:39:30.829 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:30.824 7f97e2abe700 1 --2- 172.21.2.102:0/888532909 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f97c80186c0 0x7f97c801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:30.829 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:30.824 7f97e2abe700 1 --2- 172.21.2.102:0/888532909 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f97c80186c0 0x7f97c801abc0 crc :-1 s=READY pgs=1414 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:30.858 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:30.856 7f97e954a700 1 -- 172.21.2.102:0/888532909 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97e40f0af0 con 0x7f97e41a4a60 2021-02-10T11:39:30.860 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:30.858 7f97dbfff700 1 -- 172.21.2.102:0/888532909 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f97d402dee0 con 0x7f97e41a4a60 2021-02-10T11:39:30.985 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:30.982 7f97e954a700 1 -- 172.21.2.102:0/888532909 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f97e40f0af0 con 0x7f97e41a4a60 2021-02-10T11:39:30.986 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:30.983 7f97dbfff700 1 -- 172.21.2.102:0/888532909 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v188) v1 ==== 74+0+14020 (secure 0 0 0) 0x7f97d402dee0 con 0x7f97e41a4a60 2021-02-10T11:39:30.986 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:39:30.986 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":188,"fsid":"68db30ea-dbce-4aef-a6df-f01c6fea0513","created":"2021-02-10 11:03:06.331557","modified":"2021-02-10 11:39:30.375378","last_up_change":"2021-02-10 11:03:32.690562","last_in_change":"2021-02-10 11:03:32.690562","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":41,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:03:37.322519","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":40,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:39:28.340428","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"188","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":41,"pool_name":"cephfs_data","create_time":"2021-02-10 11:39:29.367624","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"188","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"be691b4d-a039-408a-af80-9d53f02b6f1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":185,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6818","nonce":36088},{"type":"v1","addr":"172.21.2.102:6820","nonce":36088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6822","nonce":36088},{"type":"v1","addr":"172.21.2.102:6824","nonce":36088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6830","nonce":36088},{"type":"v1","addr":"172.21.2.102:6832","nonce":36088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6826","nonce":36088},{"type":"v1","addr":"172.21.2.102:6828","nonce":36088}]},"public_addr":"172.21.2.102:6820/36088","cluster_addr":"172.21.2.102:6824/36088","heartbeat_back_addr":"172.21.2.102:6832/36088","heartbeat_front_addr":"172.21.2.102:6828/36088","state":["exists","up"]},{"osd":1,"uuid":"64e3ccc8-9c69-4d7c-865e-af8e83822635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":185,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6819","nonce":36087},{"type":"v1","addr":"172.21.2.102:6821","nonce":36087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6823","nonce":36087},{"type":"v1","addr":"172.21.2.102:6825","nonce":36087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6831","nonce":36087},{"type":"v1","addr":"172.21.2.102:6833","nonce":36087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6827","nonce":36087},{"type":"v1","addr":"172.21.2.102:6829","nonce":36087}]},"public_addr":"172.21.2.102:6821/36087","cluster_addr":"172.21.2.102:6825/36087","heartbeat_back_addr":"172.21.2.102:6833/36087","heartbeat_front_addr":"172.21.2.102:6829/36087","state":["exists","up"]},{"osd":2,"uuid":"153700af-747f-481f-bdf4-d6072d3a842f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":187,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6810","nonce":36089},{"type":"v1","addr":"172.21.2.102:6811","nonce":36089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6812","nonce":36089},{"type":"v1","addr":"172.21.2.102:6813","nonce":36089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6816","nonce":36089},{"type":"v1","addr":"172.21.2.102:6817","nonce":36089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6814","nonce":36089},{"type":"v1","addr":"172.21.2.102:6815","nonce":36089}]},"public_addr":"172.21.2.102:6811/36089","cluster_addr":"172.21.2.102:6813/36089","heartbeat_back_addr":"172.21.2.102:6817/36089","heartbeat_front_addr":"172.21.2.102:6815/36089","state":["exists","up"]},{"osd":3,"uuid":"cbc678f4-83a9-45e6-8830-2f7c232b1ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":187,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6802","nonce":36090},{"type":"v1","addr":"172.21.2.102:6803","nonce":36090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6804","nonce":36090},{"type":"v1","addr":"172.21.2.102:6805","nonce":36090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6808","nonce":36090},{"type":"v1","addr":"172.21.2.102:6809","nonce":36090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6806","nonce":36090},{"type":"v1","addr":"172.21.2.102:6807","nonce":36090}]},"public_addr":"172.21.2.102:6803/36090","cluster_addr":"172.21.2.102:6805/36090","heartbeat_back_addr":"172.21.2.102:6809/36090","heartbeat_front_addr":"172.21.2.102:6807/36090","state":["exists","up"]},{"osd":4,"uuid":"c01e4008-b258-4f10-afd2-9e7f37214351","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":187,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6808","nonce":36368},{"type":"v1","addr":"172.21.2.113:6811","nonce":36368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6814","nonce":36368},{"type":"v1","addr":"172.21.2.113:6818","nonce":36368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6826","nonce":36368},{"type":"v1","addr":"172.21.2.113:6828","nonce":36368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6820","nonce":36368},{"type":"v1","addr":"172.21.2.113:6824","nonce":36368}]},"public_addr":"172.21.2.113:6811/36368","cluster_addr":"172.21.2.113:6818/36368","heartbeat_back_addr":"172.21.2.113:6828/36368","heartbeat_front_addr":"172.21.2.113:6824/36368","state":["exists","up"]},{"osd":5,"uuid":"551f6007-daf8-4a98-8ee0-c2b16bcccb5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":178,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6802","nonce":36366},{"type":"v1","addr":"172.21.2.113:6805","nonce":36366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6807","nonce":36366},{"type":"v1","addr":"172.21.2.113:6810","nonce":36366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6819","nonce":36366},{"type":"v1","addr":"172.21.2.113:6822","nonce":36366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6812","nonce":36366},{"type":"v1","addr":"172.21.2.113:6815","nonce":36366}]},"public_addr":"172.21.2.113:6805/36366","cluster_addr":"172.21.2.113:6810/36366","heartbeat_back_addr":"172.21.2.113:6822/36366","heartbeat_front_addr":"172.21.2.113:6815/36366","state":["exists","up"]},{"osd":6,"uuid":"bf67c040-0e50-48be-b102-c3943bc9ebef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":187,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6800","nonce":36367},{"type":"v1","addr":"172.21.2.113:6801","nonce":36367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6803","nonce":36367},{"type":"v1","addr":"172.21.2.113:6804","nonce":36367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6813","nonce":36367},{"type":"v1","addr":"172.21.2.113:6816","nonce":36367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6806","nonce":36367},{"type":"v1","addr":"172.21.2.113:6809","nonce":36367}]},"public_addr":"172.21.2.113:6801/36367","cluster_addr":"172.21.2.113:6804/36367","heartbeat_back_addr":"172.21.2.113:6816/36367","heartbeat_front_addr":"172.21.2.113:6809/36367","state":["exists","up"]},{"osd":7,"uuid":"f5461f80-018d-428f-be89-2eb2bcbe7bb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":187,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6817","nonce":36365},{"type":"v1","addr":"172.21.2.113:6821","nonce":36365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6823","nonce":36365},{"type":"v1","addr":"172.21.2.113:6825","nonce":36365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6830","nonce":36365},{"type":"v1","addr":"172.21.2.113:6831","nonce":36365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6827","nonce":36365},{"type":"v1","addr":"172.21.2.113:6829","nonce":36365}]},"public_addr":"172.21.2.113:6821/36365","cluster_addr":"172.21.2.113:6825/36365","heartbeat_back_addr":"172.21.2.113:6831/36365","heartbeat_front_addr":"172.21.2.113:6829/36365","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:39:30.987 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:30.984 7f97e954a700 1 -- 172.21.2.102:0/888532909 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f97c80186c0 msgr2=0x7f97c801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:30.987 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:30.984 7f97e954a700 1 --2- 172.21.2.102:0/888532909 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f97c80186c0 0x7f97c801abc0 crc :-1 s=READY pgs=1414 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:30.987 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:30.984 7f97e954a700 1 -- 172.21.2.102:0/888532909 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f97e41a4a60 msgr2=0x7f97e40c1c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:30.987 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:30.984 7f97e954a700 1 --2- 172.21.2.102:0/888532909 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f97e41a4a60 0x7f97e40c1c70 secure :-1 s=READY pgs=3234 cs=0 l=1 rev1=1 rx=0x7f97d4006f60 tx=0x7f97d400b970).stop 2021-02-10T11:39:30.987 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:30.985 7f97e954a700 1 -- 172.21.2.102:0/888532909 shutdown_connections 2021-02-10T11:39:30.987 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:30.985 7f97e954a700 1 --2- 172.21.2.102:0/888532909 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f97c80186c0 0x7f97c801abc0 unknown :-1 s=CLOSED pgs=1414 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:30.987 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:30.985 7f97e954a700 1 --2- 172.21.2.102:0/888532909 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f97e40c2cb0 0x7f97e40bfd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:30.987 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:30.985 7f97e954a700 1 --2- 172.21.2.102:0/888532909 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f97e40c2290 0x7f97e40c2690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:30.987 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:30.985 7f97e954a700 1 --2- 172.21.2.102:0/888532909 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f97e41a4a60 0x7f97e40c1c70 unknown :-1 s=CLOSED pgs=3234 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:30.987 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:30.985 7f97e954a700 1 -- 172.21.2.102:0/888532909 >> 172.21.2.102:0/888532909 conn(0x7f97e40ebe30 msgr2=0x7f97e40bf200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:30.988 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:30.985 7f97e954a700 1 -- 172.21.2.102:0/888532909 shutdown_connections 2021-02-10T11:39:30.988 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:30.985 7f97e954a700 1 -- 172.21.2.102:0/888532909 wait complete. 2021-02-10T11:39:30.996 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:39:31.427 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:31.408 7f59de6f9700 1 Processor -- start 2021-02-10T11:39:31.427 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:31.408 7f59de6f9700 1 -- start start 2021-02-10T11:39:31.427 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:31.409 7f59de6f9700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f59d80d1870 0x7f59d81b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:31.427 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:31.409 7f59de6f9700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f59d80f0650 0x7f59d80cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:31.427 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:31.409 7f59de6f9700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59d80f0b10 con 0x7f59d81b0a10 2021-02-10T11:39:31.427 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:31.409 7f59de6f9700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59d80fe230 con 0x7f59d80f0650 2021-02-10T11:39:31.428 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:31.409 7f59de6f9700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59d80e5ed0 con 0x7f59d80d1870 2021-02-10T11:39:31.428 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:31.409 7f59dcd16700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f59d80f0650 0x7f59d80cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:31.428 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:31.409 7f59dcd16700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f59d80f0650 0x7f59d80cfb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.113:3300/0 says I am v2:172.21.2.102:43834/0 (socket says 172.21.2.102:43834) 2021-02-10T11:39:31.428 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:31.409 7f59dcd16700 1 -- 172.21.2.102:0/613190081 learned_addr learned my addr 172.21.2.102:0/613190081 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:39:31.428 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:31.409 7f59d7fff700 1 --2- 172.21.2.102:0/613190081 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f59d80d1870 0x7f59d81b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:31.428 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:31.410 7f59dcd16700 1 -- 172.21.2.102:0/613190081 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f59d80d1870 msgr2=0x7f59d81b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:31.428 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:31.410 7f59dcd16700 1 --2- 172.21.2.102:0/613190081 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f59d80d1870 0x7f59d81b03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:31.428 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:31.410 7f59dcd16700 1 -- 172.21.2.102:0/613190081 >> v1:172.21.2.113:6789/0 conn(0x7f59d81b0a10 legacy=0x7f59d80cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:31.428 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:31.410 7f59dcd16700 1 -- 172.21.2.102:0/613190081 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59d80f20a0 con 0x7f59d80f0650 2021-02-10T11:39:31.428 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:31.410 7f59dcd16700 1 --2- 172.21.2.102:0/613190081 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f59d80f0650 0x7f59d80cfb20 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 rx=0x7f59d0006aa0 tx=0x7f59d0009f20).ready entity=mon.1 client_cookie=db71028d42fc96fc server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:31.428 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:31.410 7f59d6ffd700 1 -- 172.21.2.102:0/613190081 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f59d000b5c0 con 0x7f59d80f0650 2021-02-10T11:39:31.429 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:31.410 7f59d6ffd700 1 -- 172.21.2.102:0/613190081 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59d0003030 con 0x7f59d80f0650 2021-02-10T11:39:31.429 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:31.410 7f59d6ffd700 1 -- 172.21.2.102:0/613190081 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f59d000bb40 con 0x7f59d80f0650 2021-02-10T11:39:31.429 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:31.410 7f59de6f9700 1 -- 172.21.2.102:0/613190081 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f59d80f0650 msgr2=0x7f59d80cfb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:31.429 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:31.410 7f59de6f9700 1 --2- 172.21.2.102:0/613190081 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f59d80f0650 0x7f59d80cfb20 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 rx=0x7f59d0006aa0 tx=0x7f59d0009f20).stop 2021-02-10T11:39:31.429 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:31.410 7f59de6f9700 1 -- 172.21.2.102:0/613190081 shutdown_connections 2021-02-10T11:39:31.429 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:31.410 7f59de6f9700 1 --2- 172.21.2.102:0/613190081 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f59d80f0650 0x7f59d80cfb20 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:31.429 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:31.410 7f59de6f9700 1 --2- 172.21.2.102:0/613190081 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f59d80d1870 0x7f59d81b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:31.429 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:31.411 7f59de6f9700 1 -- 172.21.2.102:0/613190081 >> 172.21.2.102:0/613190081 conn(0x7f59d80ce450 msgr2=0x7f59d80d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:31.429 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:31.411 7f59de6f9700 1 -- 172.21.2.102:0/613190081 shutdown_connections 2021-02-10T11:39:31.429 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:31.411 7f59de6f9700 1 -- 172.21.2.102:0/613190081 wait complete. 2021-02-10T11:39:31.431 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:31.425 7f59de6f9700 1 Processor -- start 2021-02-10T11:39:31.431 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:31.425 7f59de6f9700 1 -- start start 2021-02-10T11:39:31.431 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:31.426 7f59de6f9700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f59d8195650 0x7f59d80b5560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:31.431 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:31.426 7f59de6f9700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f59d80b5b80 0x7f59d81a2e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:31.431 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:31.426 7f59de6f9700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f59d80acfa0 0x7f59d81a3440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:31.431 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:31.426 7f59de6f9700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59d80b6010 con 0x7f59d80acfa0 2021-02-10T11:39:31.432 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:31.426 7f59de6f9700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59d81a3a60 con 0x7f59d80b5b80 2021-02-10T11:39:31.432 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:31.426 7f59de6f9700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59d81a3c40 con 0x7f59d8195650 2021-02-10T11:39:31.432 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:31.426 7f59dcd16700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f59d80acfa0 0x7f59d81a3440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:31.432 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:31.426 7f59dcd16700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f59d80acfa0 0x7f59d81a3440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:40410/0 (socket says 172.21.2.102:40410) 2021-02-10T11:39:31.432 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:31.426 7f59dcd16700 1 -- 172.21.2.102:0/401947941 learned_addr learned my addr 172.21.2.102:0/401947941 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:39:31.432 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:31.426 7f59dcd16700 1 -- 172.21.2.102:0/401947941 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f59d8195650 msgr2=0x7f59d80b5560 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:39:31.432 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:31.426 7f59dcd16700 1 --2- 172.21.2.102:0/401947941 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f59d8195650 0x7f59d80b5560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:31.432 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:31.426 7f59dcd16700 1 -- 172.21.2.102:0/401947941 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f59d80b5b80 msgr2=0x7f59d81a2e20 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:39:31.432 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:31.426 7f59dcd16700 1 --2- 172.21.2.102:0/401947941 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f59d80b5b80 0x7f59d81a2e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:31.432 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:31.426 7f59dcd16700 1 -- 172.21.2.102:0/401947941 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59d80bcae0 con 0x7f59d80acfa0 2021-02-10T11:39:31.433 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:31.426 7f59dcd16700 1 --2- 172.21.2.102:0/401947941 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f59d80acfa0 0x7f59d81a3440 secure :-1 s=READY pgs=3235 cs=0 l=1 rev1=1 rx=0x7f59d0009f70 tx=0x7f59d000bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:31.433 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:31.426 7f59d4ff9700 1 -- 172.21.2.102:0/401947941 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f59d0009140 con 0x7f59d80acfa0 2021-02-10T11:39:31.433 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:31.426 7f59d4ff9700 1 -- 172.21.2.102:0/401947941 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59d0006990 con 0x7f59d80acfa0 2021-02-10T11:39:31.433 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:31.426 7f59de6f9700 1 -- 172.21.2.102:0/401947941 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59d80bcae0 con 0x7f59d80acfa0 2021-02-10T11:39:31.433 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:31.426 7f59de6f9700 1 -- 172.21.2.102:0/401947941 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59d80bd070 con 0x7f59d80acfa0 2021-02-10T11:39:31.433 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:31.426 7f59d4ff9700 1 -- 172.21.2.102:0/401947941 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f59d00012e0 con 0x7f59d80acfa0 2021-02-10T11:39:31.433 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:31.427 7f59d4ff9700 1 -- 172.21.2.102:0/401947941 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f59d000a000 con 0x7f59d80acfa0 2021-02-10T11:39:31.433 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:31.427 7f59d4ff9700 1 --2- 172.21.2.102:0/401947941 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f59b80186c0 0x7f59b801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:31.433 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:31.427 7f59d4ff9700 1 -- 172.21.2.102:0/401947941 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(189..189 src has 1..189) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f59d00328b0 con 0x7f59d80acfa0 2021-02-10T11:39:31.433 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:31.427 7f59d7fff700 1 --2- 172.21.2.102:0/401947941 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f59b80186c0 0x7f59b801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:31.433 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:31.427 7f59d7fff700 1 --2- 172.21.2.102:0/401947941 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f59b80186c0 0x7f59b801abc0 crc :-1 s=READY pgs=1415 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:31.461 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:31.459 7f59de6f9700 1 -- 172.21.2.102:0/401947941 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59d80f0ab0 con 0x7f59d80acfa0 2021-02-10T11:39:31.463 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:31.461 7f59d4ff9700 1 -- 172.21.2.102:0/401947941 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f59d0009450 con 0x7f59d80acfa0 2021-02-10T11:39:31.587 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:31.584 7f59de6f9700 1 -- 172.21.2.102:0/401947941 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f59d80f0ab0 con 0x7f59d80acfa0 2021-02-10T11:39:31.587 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:31.585 7f59d4ff9700 1 -- 172.21.2.102:0/401947941 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v189) v1 ==== 74+0+14007 (secure 0 0 0) 0x7f59d000a000 con 0x7f59d80acfa0 2021-02-10T11:39:31.588 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:39:31.588 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":189,"fsid":"68db30ea-dbce-4aef-a6df-f01c6fea0513","created":"2021-02-10 11:03:06.331557","modified":"2021-02-10 11:39:31.377604","last_up_change":"2021-02-10 11:03:32.690562","last_in_change":"2021-02-10 11:03:32.690562","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":41,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:03:37.322519","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":40,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:39:28.340428","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"188","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":41,"pool_name":"cephfs_data","create_time":"2021-02-10 11:39:29.367624","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"189","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"be691b4d-a039-408a-af80-9d53f02b6f1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":185,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6818","nonce":36088},{"type":"v1","addr":"172.21.2.102:6820","nonce":36088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6822","nonce":36088},{"type":"v1","addr":"172.21.2.102:6824","nonce":36088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6830","nonce":36088},{"type":"v1","addr":"172.21.2.102:6832","nonce":36088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6826","nonce":36088},{"type":"v1","addr":"172.21.2.102:6828","nonce":36088}]},"public_addr":"172.21.2.102:6820/36088","cluster_addr":"172.21.2.102:6824/36088","heartbeat_back_addr":"172.21.2.102:6832/36088","heartbeat_front_addr":"172.21.2.102:6828/36088","state":["exists","up"]},{"osd":1,"uuid":"64e3ccc8-9c69-4d7c-865e-af8e83822635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":185,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6819","nonce":36087},{"type":"v1","addr":"172.21.2.102:6821","nonce":36087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6823","nonce":36087},{"type":"v1","addr":"172.21.2.102:6825","nonce":36087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6831","nonce":36087},{"type":"v1","addr":"172.21.2.102:6833","nonce":36087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6827","nonce":36087},{"type":"v1","addr":"172.21.2.102:6829","nonce":36087}]},"public_addr":"172.21.2.102:6821/36087","cluster_addr":"172.21.2.102:6825/36087","heartbeat_back_addr":"172.21.2.102:6833/36087","heartbeat_front_addr":"172.21.2.102:6829/36087","state":["exists","up"]},{"osd":2,"uuid":"153700af-747f-481f-bdf4-d6072d3a842f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":187,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6810","nonce":36089},{"type":"v1","addr":"172.21.2.102:6811","nonce":36089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6812","nonce":36089},{"type":"v1","addr":"172.21.2.102:6813","nonce":36089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6816","nonce":36089},{"type":"v1","addr":"172.21.2.102:6817","nonce":36089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6814","nonce":36089},{"type":"v1","addr":"172.21.2.102:6815","nonce":36089}]},"public_addr":"172.21.2.102:6811/36089","cluster_addr":"172.21.2.102:6813/36089","heartbeat_back_addr":"172.21.2.102:6817/36089","heartbeat_front_addr":"172.21.2.102:6815/36089","state":["exists","up"]},{"osd":3,"uuid":"cbc678f4-83a9-45e6-8830-2f7c232b1ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":187,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6802","nonce":36090},{"type":"v1","addr":"172.21.2.102:6803","nonce":36090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6804","nonce":36090},{"type":"v1","addr":"172.21.2.102:6805","nonce":36090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6808","nonce":36090},{"type":"v1","addr":"172.21.2.102:6809","nonce":36090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6806","nonce":36090},{"type":"v1","addr":"172.21.2.102:6807","nonce":36090}]},"public_addr":"172.21.2.102:6803/36090","cluster_addr":"172.21.2.102:6805/36090","heartbeat_back_addr":"172.21.2.102:6809/36090","heartbeat_front_addr":"172.21.2.102:6807/36090","state":["exists","up"]},{"osd":4,"uuid":"c01e4008-b258-4f10-afd2-9e7f37214351","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":187,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6808","nonce":36368},{"type":"v1","addr":"172.21.2.113:6811","nonce":36368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6814","nonce":36368},{"type":"v1","addr":"172.21.2.113:6818","nonce":36368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6826","nonce":36368},{"type":"v1","addr":"172.21.2.113:6828","nonce":36368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6820","nonce":36368},{"type":"v1","addr":"172.21.2.113:6824","nonce":36368}]},"public_addr":"172.21.2.113:6811/36368","cluster_addr":"172.21.2.113:6818/36368","heartbeat_back_addr":"172.21.2.113:6828/36368","heartbeat_front_addr":"172.21.2.113:6824/36368","state":["exists","up"]},{"osd":5,"uuid":"551f6007-daf8-4a98-8ee0-c2b16bcccb5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":178,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6802","nonce":36366},{"type":"v1","addr":"172.21.2.113:6805","nonce":36366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6807","nonce":36366},{"type":"v1","addr":"172.21.2.113:6810","nonce":36366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6819","nonce":36366},{"type":"v1","addr":"172.21.2.113:6822","nonce":36366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6812","nonce":36366},{"type":"v1","addr":"172.21.2.113:6815","nonce":36366}]},"public_addr":"172.21.2.113:6805/36366","cluster_addr":"172.21.2.113:6810/36366","heartbeat_back_addr":"172.21.2.113:6822/36366","heartbeat_front_addr":"172.21.2.113:6815/36366","state":["exists","up"]},{"osd":6,"uuid":"bf67c040-0e50-48be-b102-c3943bc9ebef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":187,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6800","nonce":36367},{"type":"v1","addr":"172.21.2.113:6801","nonce":36367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6803","nonce":36367},{"type":"v1","addr":"172.21.2.113:6804","nonce":36367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6813","nonce":36367},{"type":"v1","addr":"172.21.2.113:6816","nonce":36367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6806","nonce":36367},{"type":"v1","addr":"172.21.2.113:6809","nonce":36367}]},"public_addr":"172.21.2.113:6801/36367","cluster_addr":"172.21.2.113:6804/36367","heartbeat_back_addr":"172.21.2.113:6816/36367","heartbeat_front_addr":"172.21.2.113:6809/36367","state":["exists","up"]},{"osd":7,"uuid":"f5461f80-018d-428f-be89-2eb2bcbe7bb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":187,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6817","nonce":36365},{"type":"v1","addr":"172.21.2.113:6821","nonce":36365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6823","nonce":36365},{"type":"v1","addr":"172.21.2.113:6825","nonce":36365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6830","nonce":36365},{"type":"v1","addr":"172.21.2.113:6831","nonce":36365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6827","nonce":36365},{"type":"v1","addr":"172.21.2.113:6829","nonce":36365}]},"public_addr":"172.21.2.113:6821/36365","cluster_addr":"172.21.2.113:6825/36365","heartbeat_back_addr":"172.21.2.113:6831/36365","heartbeat_front_addr":"172.21.2.113:6829/36365","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:39:31.589 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:31.587 7f59de6f9700 1 -- 172.21.2.102:0/401947941 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f59b80186c0 msgr2=0x7f59b801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:31.589 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:31.587 7f59de6f9700 1 --2- 172.21.2.102:0/401947941 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f59b80186c0 0x7f59b801abc0 crc :-1 s=READY pgs=1415 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:31.589 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:31.587 7f59de6f9700 1 -- 172.21.2.102:0/401947941 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f59d80acfa0 msgr2=0x7f59d81a3440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:31.589 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:31.587 7f59de6f9700 1 --2- 172.21.2.102:0/401947941 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f59d80acfa0 0x7f59d81a3440 secure :-1 s=READY pgs=3235 cs=0 l=1 rev1=1 rx=0x7f59d0009f70 tx=0x7f59d000bcf0).stop 2021-02-10T11:39:31.590 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:31.587 7f59de6f9700 1 -- 172.21.2.102:0/401947941 shutdown_connections 2021-02-10T11:39:31.590 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:31.587 7f59de6f9700 1 --2- 172.21.2.102:0/401947941 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f59d80acfa0 0x7f59d81a3440 unknown :-1 s=CLOSED pgs=3235 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:31.590 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:31.587 7f59de6f9700 1 --2- 172.21.2.102:0/401947941 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f59b80186c0 0x7f59b801abc0 unknown :-1 s=CLOSED pgs=1415 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:31.590 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:31.587 7f59de6f9700 1 --2- 172.21.2.102:0/401947941 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f59d80b5b80 0x7f59d81a2e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:31.590 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:31.587 7f59de6f9700 1 --2- 172.21.2.102:0/401947941 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f59d8195650 0x7f59d80b5560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:31.590 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:31.587 7f59de6f9700 1 -- 172.21.2.102:0/401947941 >> 172.21.2.102:0/401947941 conn(0x7f59d80b3040 msgr2=0x7f59d80c3250 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:31.591 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:31.588 7f59de6f9700 1 -- 172.21.2.102:0/401947941 shutdown_connections 2021-02-10T11:39:31.591 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:31.588 7f59de6f9700 1 -- 172.21.2.102:0/401947941 wait complete. 2021-02-10T11:39:31.598 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2021-02-10T11:39:32.032 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:32.012 7f6feb36e700 1 Processor -- start 2021-02-10T11:39:32.032 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:32.012 7f6feb36e700 1 -- start start 2021-02-10T11:39:32.032 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:32.013 7f6feb36e700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6fe40d1860 0x7f6fe41b04b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:32.033 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:32.013 7f6feb36e700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f6fe41b0ad0 0x7f6fe4108930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:32.033 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:32.013 7f6feb36e700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fe4109a50 con 0x7f6fe40d1c60 2021-02-10T11:39:32.033 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:32.013 7f6feb36e700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6fe40e5ee0 con 0x7f6fe41b0ad0 2021-02-10T11:39:32.033 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:32.013 7f6feb36e700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fe4109740 con 0x7f6fe40d1860 2021-02-10T11:39:32.033 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:32.013 7f6fe998b700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f6fe40d1c60 0x7f6fe4108f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:60872/0 (socket says 172.21.2.102:60872) 2021-02-10T11:39:32.033 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:32.013 7f6fe998b700 1 -- 172.21.2.102:0/1499636332 learned_addr learned my addr 172.21.2.102:0/1499636332 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:39:32.033 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:32.013 7f6fd7fff700 1 -- 172.21.2.102:0/1499636332 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3713443774 0 0) 0x7f6fdc001500 con 0x7f6fe40d1c60 2021-02-10T11:39:32.033 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:32.013 7f6fd7fff700 1 -- 172.21.2.102:0/1499636332 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6fd0002fe0 con 0x7f6fe40d1c60 2021-02-10T11:39:32.033 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:32.013 7f6fe8989700 1 --2- 172.21.2.102:0/1499636332 >> v2:172.21.2.113:3300/0 conn(0x7f6fe41b0ad0 0x7f6fe4108930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:32.033 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:32.014 7f6fe918a700 1 --2- 172.21.2.102:0/1499636332 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6fe40d1860 0x7f6fe41b04b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:32.033 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:32.014 7f6fd7fff700 1 -- 172.21.2.102:0/1499636332 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3925944086 0 0) 0x7f6fdc001460 con 0x7f6fe40d1c60 2021-02-10T11:39:32.033 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:32.014 7f6fd7fff700 1 -- 172.21.2.102:0/1499636332 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6fe40d1860 msgr2=0x7f6fe41b04b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:32.033 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:32.014 7f6fd7fff700 1 --2- 172.21.2.102:0/1499636332 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6fe40d1860 0x7f6fe41b04b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:32.033 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:32.014 7f6fd7fff700 1 -- 172.21.2.102:0/1499636332 >> v2:172.21.2.113:3300/0 conn(0x7f6fe41b0ad0 msgr2=0x7f6fe4108930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:32.034 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:32.014 7f6fd7fff700 1 --2- 172.21.2.102:0/1499636332 >> v2:172.21.2.113:3300/0 conn(0x7f6fe41b0ad0 0x7f6fe4108930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:32.034 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:32.014 7f6fd7fff700 1 -- 172.21.2.102:0/1499636332 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fe4088070 con 0x7f6fe40d1c60 2021-02-10T11:39:32.034 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:32.014 7f6fd7fff700 1 -- 172.21.2.102:0/1499636332 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7f6fdc001980 con 0x7f6fe40d1c60 2021-02-10T11:39:32.034 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:32.014 7f6fd7fff700 1 -- 172.21.2.102:0/1499636332 >> v1:172.21.2.102:6789/0 conn(0x7f6fe40d1c60 legacy=0x7f6fe4108f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:32.034 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:32.014 7f6fd7fff700 1 --2- 172.21.2.102:0/1499636332 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6fd0003a60 0x7f6fd0005f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:32.034 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:32.014 7f6fd7fff700 1 --2- 172.21.2.102:0/1499636332 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6fd00067b0 0x7f6fd0008cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:32.034 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:32.014 7f6fd7fff700 1 --2- 172.21.2.102:0/1499636332 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6fd00095b0 0x7f6fd000baa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:32.034 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:32.014 7f6fd7fff700 1 -- 172.21.2.102:0/1499636332 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fd000c2f0 con 0x7f6fd0003a60 2021-02-10T11:39:32.034 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:32.014 7f6fd7fff700 1 -- 172.21.2.102:0/1499636332 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fd00504e0 con 0x7f6fd00067b0 2021-02-10T11:39:32.034 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:32.014 7f6fd7fff700 1 -- 172.21.2.102:0/1499636332 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fd0050700 con 0x7f6fd00095b0 2021-02-10T11:39:32.034 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:32.014 7f6fe918a700 1 --2- 172.21.2.102:0/1499636332 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6fd0003a60 0x7f6fd0005f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:32.034 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:32.014 7f6fe8989700 1 --2- 172.21.2.102:0/1499636332 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6fd00067b0 0x7f6fd0008cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:32.035 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:32.014 7f6fe998b700 1 --2- 172.21.2.102:0/1499636332 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6fd00095b0 0x7f6fd000baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:32.035 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:32.014 7f6fe918a700 1 -- 172.21.2.102:0/1499636332 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6fd00095b0 msgr2=0x7f6fd000baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:32.035 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:32.014 7f6fe918a700 1 --2- 172.21.2.102:0/1499636332 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6fd00095b0 0x7f6fd000baa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:32.035 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:32.014 7f6fe918a700 1 -- 172.21.2.102:0/1499636332 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6fd00067b0 msgr2=0x7f6fd0008cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:32.035 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:32.014 7f6fe918a700 1 --2- 172.21.2.102:0/1499636332 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6fd00067b0 0x7f6fd0008cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:32.035 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:32.014 7f6fe918a700 1 -- 172.21.2.102:0/1499636332 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fd00509a0 con 0x7f6fd0003a60 2021-02-10T11:39:32.035 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:32.014 7f6fe918a700 1 --2- 172.21.2.102:0/1499636332 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6fd0003a60 0x7f6fd0005f10 secure :-1 s=READY pgs=3237 cs=0 l=1 rev1=1 rx=0x7f6fd8006a90 tx=0x7f6fd8009a90).ready entity=mon.0 client_cookie=892b8f91f1118e73 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:32.035 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:32.014 7f6fd7fff700 1 -- 172.21.2.102:0/1499636332 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6fd800c0c0 con 0x7f6fd0003a60 2021-02-10T11:39:32.035 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:32.014 7f6fd7fff700 1 -- 172.21.2.102:0/1499636332 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fd800c3c0 con 0x7f6fd0003a60 2021-02-10T11:39:32.035 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:32.014 7f6fd7fff700 1 -- 172.21.2.102:0/1499636332 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6fd800c820 con 0x7f6fd0003a60 2021-02-10T11:39:32.035 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:32.014 7f6feb36e700 1 -- 172.21.2.102:0/1499636332 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6fd0003a60 msgr2=0x7f6fd0005f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:32.035 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:32.014 7f6feb36e700 1 --2- 172.21.2.102:0/1499636332 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6fd0003a60 0x7f6fd0005f10 secure :-1 s=READY pgs=3237 cs=0 l=1 rev1=1 rx=0x7f6fd8006a90 tx=0x7f6fd8009a90).stop 2021-02-10T11:39:32.035 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:32.014 7f6fe998b700 1 -- 172.21.2.102:0/1499636332 reap_dead start 2021-02-10T11:39:32.035 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:32.015 7f6feb36e700 1 -- 172.21.2.102:0/1499636332 shutdown_connections 2021-02-10T11:39:32.036 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:32.015 7f6feb36e700 1 -- 172.21.2.102:0/1499636332 >> 172.21.2.102:0/1499636332 conn(0x7f6fe40ce410 msgr2=0x7f6fe40d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:32.036 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:32.015 7f6feb36e700 1 -- 172.21.2.102:0/1499636332 shutdown_connections 2021-02-10T11:39:32.036 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:32.015 7f6feb36e700 1 -- 172.21.2.102:0/1499636332 wait complete. 2021-02-10T11:39:32.036 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:32.030 7f6feb36e700 1 Processor -- start 2021-02-10T11:39:32.036 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:32.030 7f6feb36e700 1 -- start start 2021-02-10T11:39:32.037 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:32.030 7f6feb36e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6fe40b8100 0x7f6fe4083440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:32.037 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:32.030 7f6feb36e700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6fe40b8500 0x7f6fe4083a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:32.037 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:32.030 7f6feb36e700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6fe40846f0 0x7f6fe4084af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:32.038 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:32.030 7f6feb36e700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fe41a1720 con 0x7f6fe40b8100 2021-02-10T11:39:32.038 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:32.030 7f6feb36e700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fe4189e70 con 0x7f6fe40846f0 2021-02-10T11:39:32.038 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:32.030 7f6feb36e700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fe4084170 con 0x7f6fe40b8500 2021-02-10T11:39:32.038 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:32.031 7f6fe918a700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6fe40b8100 0x7f6fe4083440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:32.038 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:32.031 7f6fe918a700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6fe40b8100 0x7f6fe4083440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:40426/0 (socket says 172.21.2.102:40426) 2021-02-10T11:39:32.038 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:32.031 7f6fe918a700 1 -- 172.21.2.102:0/1179874455 learned_addr learned my addr 172.21.2.102:0/1179874455 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:39:32.038 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:32.031 7f6fe918a700 1 -- 172.21.2.102:0/1179874455 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6fe40b8500 msgr2=0x7f6fe4083a60 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:39:32.038 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:32.031 7f6fe918a700 1 --2- 172.21.2.102:0/1179874455 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6fe40b8500 0x7f6fe4083a60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:32.038 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:32.031 7f6fe918a700 1 -- 172.21.2.102:0/1179874455 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6fe40846f0 msgr2=0x7f6fe4084af0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:39:32.038 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:32.031 7f6fe918a700 1 --2- 172.21.2.102:0/1179874455 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6fe40846f0 0x7f6fe4084af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:32.038 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:32.031 7f6fe918a700 1 -- 172.21.2.102:0/1179874455 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fe4085110 con 0x7f6fe40b8100 2021-02-10T11:39:32.038 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:32.031 7f6fe918a700 1 --2- 172.21.2.102:0/1179874455 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6fe40b8100 0x7f6fe4083440 secure :-1 s=READY pgs=3238 cs=0 l=1 rev1=1 rx=0x7f6fd8006cd0 tx=0x7f6fd800ba10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:32.039 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:32.031 7f6fd5ffb700 1 -- 172.21.2.102:0/1179874455 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6fd8002030 con 0x7f6fe40b8100 2021-02-10T11:39:32.039 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:32.031 7f6fd5ffb700 1 -- 172.21.2.102:0/1179874455 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fd8000940 con 0x7f6fe40b8100 2021-02-10T11:39:32.039 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:32.031 7f6feb36e700 1 -- 172.21.2.102:0/1179874455 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fe4085110 con 0x7f6fe40b8100 2021-02-10T11:39:32.039 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:32.031 7f6feb36e700 1 -- 172.21.2.102:0/1179874455 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fe4085660 con 0x7f6fe40b8100 2021-02-10T11:39:32.039 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:32.031 7f6fd5ffb700 1 -- 172.21.2.102:0/1179874455 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6fd80023b0 con 0x7f6fe40b8100 2021-02-10T11:39:32.039 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:32.032 7f6fd5ffb700 1 -- 172.21.2.102:0/1179874455 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f6fd8000c90 con 0x7f6fe40b8100 2021-02-10T11:39:32.039 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:32.032 7f6fd5ffb700 1 --2- 172.21.2.102:0/1179874455 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6fc40186c0 0x7f6fc401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:32.039 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:32.032 7f6fd5ffb700 1 -- 172.21.2.102:0/1179874455 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(189..189 src has 1..189) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f6fd80323d0 con 0x7f6fe40b8100 2021-02-10T11:39:32.039 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:32.032 7f6fe8989700 1 --2- 172.21.2.102:0/1179874455 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6fc40186c0 0x7f6fc401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:32.039 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:32.032 7f6fe8989700 1 --2- 172.21.2.102:0/1179874455 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6fc40186c0 0x7f6fc401abc0 crc :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:32.064 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:32.062 7f6feb36e700 1 -- 172.21.2.102:0/1179874455 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fe41a1720 con 0x7f6fe40b8100 2021-02-10T11:39:32.068 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:32.065 7f6fd5ffb700 1 -- 172.21.2.102:0/1179874455 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f6fd8025860 con 0x7f6fe40b8100 2021-02-10T11:39:32.207 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:32.204 7f6feb36e700 1 -- 172.21.2.102:0/1179874455 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"} v 0) v1 -- 0x7f6fe4189e70 con 0x7f6fe40b8100 2021-02-10T11:39:32.385 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:32.383 7f6fd5ffb700 1 -- 172.21.2.102:0/1179874455 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"}]=0 v165) v1 ==== 118+0+0 (secure 0 0 0) 0x7f6fd8025860 con 0x7f6fe40b8100 2021-02-10T11:39:32.422 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:32.420 7f6feb36e700 1 -- 172.21.2.102:0/1179874455 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6fc40186c0 msgr2=0x7f6fc401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:32.422 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:32.420 7f6feb36e700 1 --2- 172.21.2.102:0/1179874455 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6fc40186c0 0x7f6fc401abc0 crc :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:32.422 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:32.420 7f6feb36e700 1 -- 172.21.2.102:0/1179874455 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6fe40b8100 msgr2=0x7f6fe4083440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:32.422 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:32.420 7f6feb36e700 1 --2- 172.21.2.102:0/1179874455 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6fe40b8100 0x7f6fe4083440 secure :-1 s=READY pgs=3238 cs=0 l=1 rev1=1 rx=0x7f6fd8006cd0 tx=0x7f6fd800ba10).stop 2021-02-10T11:39:32.423 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:32.420 7f6feb36e700 1 -- 172.21.2.102:0/1179874455 shutdown_connections 2021-02-10T11:39:32.423 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:32.420 7f6feb36e700 1 --2- 172.21.2.102:0/1179874455 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6fc40186c0 0x7f6fc401abc0 unknown :-1 s=CLOSED pgs=1416 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:32.423 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:32.420 7f6feb36e700 1 --2- 172.21.2.102:0/1179874455 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6fe40846f0 0x7f6fe4084af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:32.423 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:32.420 7f6feb36e700 1 --2- 172.21.2.102:0/1179874455 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6fe40b8500 0x7f6fe4083a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:32.423 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:32.420 7f6feb36e700 1 --2- 172.21.2.102:0/1179874455 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6fe40b8100 0x7f6fe4083440 unknown :-1 s=CLOSED pgs=3238 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:32.423 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:32.420 7f6feb36e700 1 -- 172.21.2.102:0/1179874455 >> 172.21.2.102:0/1179874455 conn(0x7f6fe40bee90 msgr2=0x7f6fe409f200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:32.423 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:32.420 7f6feb36e700 1 -- 172.21.2.102:0/1179874455 shutdown_connections 2021-02-10T11:39:32.423 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:32.420 7f6feb36e700 1 -- 172.21.2.102:0/1179874455 wait complete. 2021-02-10T11:39:32.431 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:39:32.860 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:32.841 7f5acbf50700 1 Processor -- start 2021-02-10T11:39:32.860 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:32.841 7f5acbf50700 1 -- start start 2021-02-10T11:39:32.860 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:32.842 7f5acbf50700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f5ac41b0870 0x7f5ac40f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:32.860 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:32.842 7f5acbf50700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5ac40efe00 0x7f5ac40f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:32.860 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:32.842 7f5acbf50700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ac41b0c70 con 0x7f5ac41b0470 2021-02-10T11:39:32.860 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:32.842 7f5acbf50700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ac40f0b50 con 0x7f5ac41b0870 2021-02-10T11:39:32.860 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:32.842 7f5acbf50700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ac40f0820 con 0x7f5ac40efe00 2021-02-10T11:39:32.860 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:32.842 7f5ac9d6c700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f5ac41b0470 0x7f5ac40ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:60890/0 (socket says 172.21.2.102:60890) 2021-02-10T11:39:32.860 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:32.842 7f5ac9d6c700 1 -- 172.21.2.102:0/3290968714 learned_addr learned my addr 172.21.2.102:0/3290968714 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:39:32.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:32.842 7f5ac956b700 1 --2- 172.21.2.102:0/3290968714 >> v2:172.21.2.113:3300/0 conn(0x7f5ac41b0870 0x7f5ac40f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:32.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:32.842 7f5aca56d700 1 --2- 172.21.2.102:0/3290968714 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5ac40efe00 0x7f5ac40f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:32.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:32.842 7f5ac956b700 1 -- 172.21.2.102:0/3290968714 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5ac40efe00 msgr2=0x7f5ac40f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:32.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:32.842 7f5ac8d6a700 1 -- 172.21.2.102:0/3290968714 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 194428397 0 0) 0x7f5ac0001180 con 0x7f5ac41b0470 2021-02-10T11:39:32.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:32.842 7f5ac956b700 1 --2- 172.21.2.102:0/3290968714 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5ac40efe00 0x7f5ac40f0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:32.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:32.842 7f5ac956b700 1 -- 172.21.2.102:0/3290968714 >> v1:172.21.2.102:6789/0 conn(0x7f5ac41b0470 legacy=0x7f5ac40ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:32.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:32.842 7f5ac956b700 1 -- 172.21.2.102:0/3290968714 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ac40e52e0 con 0x7f5ac41b0870 2021-02-10T11:39:32.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:32.843 7f5ac956b700 1 --2- 172.21.2.102:0/3290968714 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5ac41b0870 0x7f5ac40f20e0 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 rx=0x7f5ab8006650 tx=0x7f5ab8009a80).ready entity=mon.1 client_cookie=d213443e95811d8a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:32.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:32.843 7f5ac8d6a700 1 -- 172.21.2.102:0/3290968714 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5ab8006390 con 0x7f5ac41b0870 2021-02-10T11:39:32.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:32.843 7f5ac8d6a700 1 -- 172.21.2.102:0/3290968714 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ab80009b0 con 0x7f5ac41b0870 2021-02-10T11:39:32.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:32.843 7f5ac8d6a700 1 -- 172.21.2.102:0/3290968714 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5ab800c2b0 con 0x7f5ac41b0870 2021-02-10T11:39:32.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:32.843 7f5acbf50700 1 -- 172.21.2.102:0/3290968714 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5ac41b0870 msgr2=0x7f5ac40f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:32.862 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:32.843 7f5acbf50700 1 --2- 172.21.2.102:0/3290968714 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5ac41b0870 0x7f5ac40f20e0 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 rx=0x7f5ab8006650 tx=0x7f5ab8009a80).stop 2021-02-10T11:39:32.862 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:32.843 7f5acbf50700 1 -- 172.21.2.102:0/3290968714 shutdown_connections 2021-02-10T11:39:32.862 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:32.843 7f5acbf50700 1 --2- 172.21.2.102:0/3290968714 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5ac41b0870 0x7f5ac40f20e0 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:32.862 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:32.843 7f5acbf50700 1 --2- 172.21.2.102:0/3290968714 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5ac40efe00 0x7f5ac40f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:32.862 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:32.843 7f5acbf50700 1 -- 172.21.2.102:0/3290968714 >> 172.21.2.102:0/3290968714 conn(0x7f5ac40ce3e0 msgr2=0x7f5ac40e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:32.862 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:32.843 7f5acbf50700 1 -- 172.21.2.102:0/3290968714 shutdown_connections 2021-02-10T11:39:32.862 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:32.843 7f5acbf50700 1 -- 172.21.2.102:0/3290968714 wait complete. 2021-02-10T11:39:32.863 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:32.858 7f5acbf50700 1 Processor -- start 2021-02-10T11:39:32.863 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:32.858 7f5acbf50700 1 -- start start 2021-02-10T11:39:32.864 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:32.858 7f5acbf50700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5ac40aed00 0x7f5ac4096400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:32.864 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:32.858 7f5acbf50700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5ac418ef20 0x7f5ac40955e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:32.864 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:32.858 7f5acbf50700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5ac4095c00 0x7f5ac4093240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:32.864 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:32.858 7f5acbf50700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ac4109ad0 con 0x7f5ac418ef20 2021-02-10T11:39:32.864 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:32.858 7f5acbf50700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ac4096a20 con 0x7f5ac40aed00 2021-02-10T11:39:32.864 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:32.858 7f5acbf50700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ac4096000 con 0x7f5ac4095c00 2021-02-10T11:39:32.864 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:32.858 7f5ac956b700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5ac418ef20 0x7f5ac40955e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:32.864 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:32.859 7f5ac956b700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5ac418ef20 0x7f5ac40955e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:40438/0 (socket says 172.21.2.102:40438) 2021-02-10T11:39:32.864 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:32.859 7f5ac956b700 1 -- 172.21.2.102:0/829055403 learned_addr learned my addr 172.21.2.102:0/829055403 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:39:32.864 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:32.859 7f5ac956b700 1 -- 172.21.2.102:0/829055403 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5ac4095c00 msgr2=0x7f5ac4093240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:32.864 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:32.859 7f5ac956b700 1 --2- 172.21.2.102:0/829055403 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5ac4095c00 0x7f5ac4093240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:32.864 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:32.859 7f5ac956b700 1 -- 172.21.2.102:0/829055403 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5ac40aed00 msgr2=0x7f5ac4096400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:32.864 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:32.859 7f5ac956b700 1 --2- 172.21.2.102:0/829055403 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5ac40aed00 0x7f5ac4096400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:32.865 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:32.859 7f5ac956b700 1 -- 172.21.2.102:0/829055403 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ac4093860 con 0x7f5ac418ef20 2021-02-10T11:39:32.865 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:32.859 7f5ac956b700 1 --2- 172.21.2.102:0/829055403 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5ac418ef20 0x7f5ac40955e0 secure :-1 s=READY pgs=3240 cs=0 l=1 rev1=1 rx=0x7f5ab8009630 tx=0x7f5ab800b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:32.865 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:32.859 7f5ab67fc700 1 -- 172.21.2.102:0/829055403 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5ab80093a0 con 0x7f5ac418ef20 2021-02-10T11:39:32.865 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:32.859 7f5ab67fc700 1 -- 172.21.2.102:0/829055403 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ab800bd70 con 0x7f5ac418ef20 2021-02-10T11:39:32.865 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:32.859 7f5acbf50700 1 -- 172.21.2.102:0/829055403 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ac4093860 con 0x7f5ac418ef20 2021-02-10T11:39:32.865 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:32.859 7f5ab67fc700 1 -- 172.21.2.102:0/829055403 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5ab8005560 con 0x7f5ac418ef20 2021-02-10T11:39:32.865 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:32.859 7f5acbf50700 1 -- 172.21.2.102:0/829055403 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ac4093860 con 0x7f5ac418ef20 2021-02-10T11:39:32.865 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:32.860 7f5ab67fc700 1 -- 172.21.2.102:0/829055403 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f5ab80052e0 con 0x7f5ac418ef20 2021-02-10T11:39:32.865 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:32.860 7f5ab67fc700 1 --2- 172.21.2.102:0/829055403 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5aac018730 0x7f5aac01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:32.865 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:32.860 7f5ab67fc700 1 -- 172.21.2.102:0/829055403 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(189..189 src has 1..189) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f5ab8032650 con 0x7f5ac418ef20 2021-02-10T11:39:32.865 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:32.860 7f5ac9d6c700 1 --2- 172.21.2.102:0/829055403 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5aac018730 0x7f5aac01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:32.865 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:32.860 7f5ac9d6c700 1 --2- 172.21.2.102:0/829055403 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5aac018730 0x7f5aac01ac10 crc :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:32.894 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:32.892 7f5acbf50700 1 -- 172.21.2.102:0/829055403 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ac4105a20 con 0x7f5ac418ef20 2021-02-10T11:39:32.896 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:32.894 7f5ab67fc700 1 -- 172.21.2.102:0/829055403 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f5ab8007ee0 con 0x7f5ac418ef20 2021-02-10T11:39:33.025 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:33.023 7f5acbf50700 1 -- 172.21.2.102:0/829055403 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5ac40f0b50 con 0x7f5ac418ef20 2021-02-10T11:39:33.026 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:33.024 7f5ab67fc700 1 -- 172.21.2.102:0/829055403 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 165 v165) v1 ==== 95+0+2104 (secure 0 0 0) 0x7f5ab800bd70 con 0x7f5ac418ef20 2021-02-10T11:39:33.026 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:39:33.027 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":165,"default_fscid":20,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8998,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.113:6833/479366648","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6832","nonce":479366648},{"type":"v1","addr":"172.21.2.113:6833","nonce":479366648}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":162}],"filesystems":[{"mdsmap":{"epoch":165,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:39:30.375466","modified":"2021-02-10 11:39:32.379360","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7481},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7481":{"gid":7481,"name":"a","rank":0,"incarnation":163,"state":"up:active","state_seq":3,"addr":"172.21.2.102:6835/4290051997","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6834","nonce":4290051997},{"type":"v1","addr":"172.21.2.102:6835","nonce":4290051997}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[41],"metadata_pool":40,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":20}]} 2021-02-10T11:39:33.027 INFO:teuthology.orchestra.run.gibba002.stderr:dumped fsmap epoch 165 2021-02-10T11:39:33.028 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:33.025 7f5acbf50700 1 -- 172.21.2.102:0/829055403 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5aac018730 msgr2=0x7f5aac01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:33.028 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:33.025 7f5acbf50700 1 --2- 172.21.2.102:0/829055403 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5aac018730 0x7f5aac01ac10 crc :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:33.028 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:33.025 7f5acbf50700 1 -- 172.21.2.102:0/829055403 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5ac418ef20 msgr2=0x7f5ac40955e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:33.028 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:33.025 7f5acbf50700 1 --2- 172.21.2.102:0/829055403 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5ac418ef20 0x7f5ac40955e0 secure :-1 s=READY pgs=3240 cs=0 l=1 rev1=1 rx=0x7f5ab8009630 tx=0x7f5ab800b980).stop 2021-02-10T11:39:33.028 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:33.026 7f5acbf50700 1 -- 172.21.2.102:0/829055403 shutdown_connections 2021-02-10T11:39:33.028 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:33.026 7f5acbf50700 1 --2- 172.21.2.102:0/829055403 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5ac4095c00 0x7f5ac4093240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:33.028 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:33.026 7f5acbf50700 1 --2- 172.21.2.102:0/829055403 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5ac418ef20 0x7f5ac40955e0 unknown :-1 s=CLOSED pgs=3240 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:33.028 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:33.026 7f5acbf50700 1 --2- 172.21.2.102:0/829055403 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5aac018730 0x7f5aac01ac10 unknown :-1 s=CLOSED pgs=1417 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:33.028 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:33.026 7f5acbf50700 1 --2- 172.21.2.102:0/829055403 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5ac40aed00 0x7f5ac4096400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:33.028 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:33.026 7f5acbf50700 1 -- 172.21.2.102:0/829055403 >> 172.21.2.102:0/829055403 conn(0x7f5ac41a37a0 msgr2=0x7f5ac419d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:33.029 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:33.026 7f5acbf50700 1 -- 172.21.2.102:0/829055403 shutdown_connections 2021-02-10T11:39:33.029 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:33.026 7f5acbf50700 1 -- 172.21.2.102:0/829055403 wait complete. 2021-02-10T11:39:33.036 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:39:33.467 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:33.447 7f36954b1700 1 Processor -- start 2021-02-10T11:39:33.468 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:33.447 7f36954b1700 1 -- start start 2021-02-10T11:39:33.468 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:33.448 7f36954b1700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f36900d1c70 0x7f36900f20a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:33.468 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:33.448 7f36954b1700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36900f0b10 con 0x7f36900d1870 2021-02-10T11:39:33.468 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:33.448 7f36954b1700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36901b0be0 con 0x7f36900d0340 2021-02-10T11:39:33.468 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:33.448 7f36954b1700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36900e5ed0 con 0x7f36900d1c70 2021-02-10T11:39:33.468 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:33.448 7f368f2bf700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f36900d1870 0x7f36901b03f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:60902/0 (socket says 172.21.2.102:60902) 2021-02-10T11:39:33.468 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:33.448 7f368f2bf700 1 -- 172.21.2.102:0/548762079 learned_addr learned my addr 172.21.2.102:0/548762079 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:39:33.468 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:33.449 7f368e2bd700 1 -- 172.21.2.102:0/548762079 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1354521431 0 0) 0x7f3680000ce0 con 0x7f36900d1870 2021-02-10T11:39:33.468 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:33.449 7f368fac0700 1 --2- 172.21.2.102:0/548762079 >> v2:172.21.2.113:3300/0 conn(0x7f36900d1c70 0x7f36900f20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:33.468 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:33.449 7f368e2bd700 1 -- 172.21.2.102:0/548762079 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3678002fe0 con 0x7f36900d1870 2021-02-10T11:39:33.469 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:33.449 7f368e2bd700 1 -- 172.21.2.102:0/548762079 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1352078631 0 0) 0x7f3680001180 con 0x7f36900d1870 2021-02-10T11:39:33.469 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:33.449 7f368e2bd700 1 -- 172.21.2.102:0/548762079 >> v2:172.21.2.113:3300/0 conn(0x7f36900d1c70 msgr2=0x7f36900f20a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:33.469 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:33.449 7f368e2bd700 1 --2- 172.21.2.102:0/548762079 >> v2:172.21.2.113:3300/0 conn(0x7f36900d1c70 0x7f36900f20a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:33.469 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:33.449 7f368e2bd700 1 -- 172.21.2.102:0/548762079 >> v1:172.21.2.113:6789/0 conn(0x7f36900d0340 legacy=0x7f36900ef640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:33.469 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:33.449 7f368e2bd700 1 -- 172.21.2.102:0/548762079 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36900efe30 con 0x7f36900d1870 2021-02-10T11:39:33.469 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:33.449 7f368e2bd700 1 -- 172.21.2.102:0/548762079 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7f3680001680 con 0x7f36900d1870 2021-02-10T11:39:33.469 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:33.449 7f368e2bd700 1 -- 172.21.2.102:0/548762079 >> v1:172.21.2.102:6789/0 conn(0x7f36900d1870 legacy=0x7f36901b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:33.469 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:33.449 7f368e2bd700 1 --2- 172.21.2.102:0/548762079 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3678003ac0 0x7f3678005f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:33.469 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:33.449 7f368e2bd700 1 --2- 172.21.2.102:0/548762079 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3678006780 0x7f3678008ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:33.469 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:33.449 7f368e2bd700 1 --2- 172.21.2.102:0/548762079 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3678009580 0x7f367800baa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:33.469 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:33.449 7f368e2bd700 1 -- 172.21.2.102:0/548762079 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f367800c2f0 con 0x7f3678003ac0 2021-02-10T11:39:33.469 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:33.449 7f368e2bd700 1 -- 172.21.2.102:0/548762079 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f367800c510 con 0x7f3678009580 2021-02-10T11:39:33.470 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:33.449 7f368e2bd700 1 -- 172.21.2.102:0/548762079 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f367800c730 con 0x7f3678006780 2021-02-10T11:39:33.470 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:33.449 7f368f2bf700 1 --2- 172.21.2.102:0/548762079 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3678003ac0 0x7f3678005f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:33.470 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:33.450 7f368fac0700 1 --2- 172.21.2.102:0/548762079 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3678009580 0x7f367800baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:33.470 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:33.450 7f368f2bf700 1 -- 172.21.2.102:0/548762079 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3678006780 msgr2=0x7f3678008ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:33.470 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:33.450 7f368eabe700 1 --2- 172.21.2.102:0/548762079 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3678006780 0x7f3678008ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:33.470 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:33.450 7f368f2bf700 1 --2- 172.21.2.102:0/548762079 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3678006780 0x7f3678008ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:33.470 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:33.450 7f368f2bf700 1 -- 172.21.2.102:0/548762079 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3678009580 msgr2=0x7f367800baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:33.470 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:33.450 7f368f2bf700 1 --2- 172.21.2.102:0/548762079 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3678009580 0x7f367800baa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:33.470 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:33.450 7f368f2bf700 1 -- 172.21.2.102:0/548762079 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3678050970 con 0x7f3678003ac0 2021-02-10T11:39:33.470 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:33.450 7f368fac0700 1 --2- 172.21.2.102:0/548762079 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3678009580 0x7f367800baa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:39:33.470 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:33.450 7f368fac0700 1 -- 172.21.2.102:0/548762079 reap_dead start 2021-02-10T11:39:33.470 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:33.450 7f368eabe700 1 --2- 172.21.2.102:0/548762079 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3678006780 0x7f3678008ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:39:33.471 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:33.450 7f368f2bf700 1 --2- 172.21.2.102:0/548762079 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3678003ac0 0x7f3678005f60 secure :-1 s=READY pgs=3242 cs=0 l=1 rev1=1 rx=0x7f36800014e0 tx=0x7f3680007d20).ready entity=mon.0 client_cookie=8ad04fbcb906dbeb server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:33.471 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:33.450 7f368e2bd700 1 -- 172.21.2.102:0/548762079 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f36800070c0 con 0x7f3678003ac0 2021-02-10T11:39:33.471 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:33.450 7f368e2bd700 1 -- 172.21.2.102:0/548762079 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36800043a0 con 0x7f3678003ac0 2021-02-10T11:39:33.471 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:33.450 7f368e2bd700 1 -- 172.21.2.102:0/548762079 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3680004840 con 0x7f3678003ac0 2021-02-10T11:39:33.471 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:33.450 7f36954b1700 1 -- 172.21.2.102:0/548762079 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3678003ac0 msgr2=0x7f3678005f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:33.471 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:33.450 7f36954b1700 1 --2- 172.21.2.102:0/548762079 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3678003ac0 0x7f3678005f60 secure :-1 s=READY pgs=3242 cs=0 l=1 rev1=1 rx=0x7f36800014e0 tx=0x7f3680007d20).stop 2021-02-10T11:39:33.471 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:33.451 7f36954b1700 1 -- 172.21.2.102:0/548762079 shutdown_connections 2021-02-10T11:39:33.471 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:33.451 7f36954b1700 1 --2- 172.21.2.102:0/548762079 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3678003ac0 0x7f3678005f60 unknown :-1 s=CLOSED pgs=3242 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:33.471 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:33.451 7f36954b1700 1 -- 172.21.2.102:0/548762079 >> 172.21.2.102:0/548762079 conn(0x7f36900ce450 msgr2=0x7f36900d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:33.471 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:33.451 7f36954b1700 1 -- 172.21.2.102:0/548762079 shutdown_connections 2021-02-10T11:39:33.471 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:33.451 7f36954b1700 1 -- 172.21.2.102:0/548762079 wait complete. 2021-02-10T11:39:33.471 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:33.465 7f36954b1700 1 Processor -- start 2021-02-10T11:39:33.471 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:33.465 7f36954b1700 1 -- start start 2021-02-10T11:39:33.473 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:33.466 7f36954b1700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f369009c870 0x7f36900a0500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:33.473 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:33.466 7f36954b1700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f36900a0b20 0x7f36900ba6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:33.473 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:33.466 7f36954b1700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f36900a0f20 0x7f36900bacd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:33.473 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:33.466 7f36954b1700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f369009cc70 con 0x7f369009c870 2021-02-10T11:39:33.473 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:33.466 7f36954b1700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36900a0060 con 0x7f36900a0f20 2021-02-10T11:39:33.474 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:33.466 7f36954b1700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36900bb2f0 con 0x7f36900a0b20 2021-02-10T11:39:33.474 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:33.466 7f368f2bf700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f369009c870 0x7f36900a0500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:33.474 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:33.466 7f368f2bf700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f369009c870 0x7f36900a0500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:40460/0 (socket says 172.21.2.102:40460) 2021-02-10T11:39:33.474 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:33.466 7f368f2bf700 1 -- 172.21.2.102:0/3216321036 learned_addr learned my addr 172.21.2.102:0/3216321036 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:39:33.474 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:33.466 7f368fac0700 1 --2- 172.21.2.102:0/3216321036 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f36900a0f20 0x7f36900bacd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:33.474 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:33.467 7f368eabe700 1 --2- 172.21.2.102:0/3216321036 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f36900a0b20 0x7f36900ba6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:33.474 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:33.467 7f368f2bf700 1 -- 172.21.2.102:0/3216321036 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f36900a0b20 msgr2=0x7f36900ba6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:33.474 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:33.467 7f368f2bf700 1 --2- 172.21.2.102:0/3216321036 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f36900a0b20 0x7f36900ba6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:33.474 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:33.467 7f368f2bf700 1 -- 172.21.2.102:0/3216321036 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f36900a0f20 msgr2=0x7f36900bacd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:33.474 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:33.467 7f368f2bf700 1 --2- 172.21.2.102:0/3216321036 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f36900a0f20 0x7f36900bacd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:33.474 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:33.467 7f368f2bf700 1 -- 172.21.2.102:0/3216321036 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36900ebe90 con 0x7f369009c870 2021-02-10T11:39:33.474 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:33.467 7f368eabe700 1 --2- 172.21.2.102:0/3216321036 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f36900a0b20 0x7f36900ba6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:39:33.474 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:33.467 7f368f2bf700 1 --2- 172.21.2.102:0/3216321036 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f369009c870 0x7f36900a0500 secure :-1 s=READY pgs=3243 cs=0 l=1 rev1=1 rx=0x7f3680001c20 tx=0x7f368000a970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:33.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:33.467 7f3677fff700 1 -- 172.21.2.102:0/3216321036 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3680001c60 con 0x7f369009c870 2021-02-10T11:39:33.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:33.467 7f3677fff700 1 -- 172.21.2.102:0/3216321036 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f368000ad60 con 0x7f369009c870 2021-02-10T11:39:33.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:33.467 7f3677fff700 1 -- 172.21.2.102:0/3216321036 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f36800043a0 con 0x7f369009c870 2021-02-10T11:39:33.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:33.467 7f36954b1700 1 -- 172.21.2.102:0/3216321036 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36900ebe90 con 0x7f369009c870 2021-02-10T11:39:33.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:33.467 7f36954b1700 1 -- 172.21.2.102:0/3216321036 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36900ec420 con 0x7f369009c870 2021-02-10T11:39:33.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:33.468 7f3677fff700 1 -- 172.21.2.102:0/3216321036 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f36800043a0 con 0x7f369009c870 2021-02-10T11:39:33.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:33.468 7f3677fff700 1 --2- 172.21.2.102:0/3216321036 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f36700186c0 0x7f367001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:33.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:33.468 7f3677fff700 1 -- 172.21.2.102:0/3216321036 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(189..189 src has 1..189) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f3680032550 con 0x7f369009c870 2021-02-10T11:39:33.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:33.468 7f368eabe700 1 --2- 172.21.2.102:0/3216321036 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f36700186c0 0x7f367001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:33.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:33.468 7f368eabe700 1 --2- 172.21.2.102:0/3216321036 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f36700186c0 0x7f367001abc0 crc :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:33.500 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:33.498 7f36954b1700 1 -- 172.21.2.102:0/3216321036 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36900eccb0 con 0x7f369009c870 2021-02-10T11:39:33.502 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:33.500 7f3677fff700 1 -- 172.21.2.102:0/3216321036 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f3680000cb0 con 0x7f369009c870 2021-02-10T11:39:33.627 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:33.624 7f36954b1700 1 -- 172.21.2.102:0/3216321036 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f36900eccb0 con 0x7f369009c870 2021-02-10T11:39:33.627 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:33.625 7f3677fff700 1 -- 172.21.2.102:0/3216321036 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v189) v1 ==== 74+0+14007 (secure 0 0 0) 0x7f36800043a0 con 0x7f369009c870 2021-02-10T11:39:33.628 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:39:33.628 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":189,"fsid":"68db30ea-dbce-4aef-a6df-f01c6fea0513","created":"2021-02-10 11:03:06.331557","modified":"2021-02-10 11:39:31.377604","last_up_change":"2021-02-10 11:03:32.690562","last_in_change":"2021-02-10 11:03:32.690562","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":41,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:03:37.322519","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":40,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:39:28.340428","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"188","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":41,"pool_name":"cephfs_data","create_time":"2021-02-10 11:39:29.367624","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"189","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"be691b4d-a039-408a-af80-9d53f02b6f1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":185,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6818","nonce":36088},{"type":"v1","addr":"172.21.2.102:6820","nonce":36088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6822","nonce":36088},{"type":"v1","addr":"172.21.2.102:6824","nonce":36088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6830","nonce":36088},{"type":"v1","addr":"172.21.2.102:6832","nonce":36088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6826","nonce":36088},{"type":"v1","addr":"172.21.2.102:6828","nonce":36088}]},"public_addr":"172.21.2.102:6820/36088","cluster_addr":"172.21.2.102:6824/36088","heartbeat_back_addr":"172.21.2.102:6832/36088","heartbeat_front_addr":"172.21.2.102:6828/36088","state":["exists","up"]},{"osd":1,"uuid":"64e3ccc8-9c69-4d7c-865e-af8e83822635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":185,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6819","nonce":36087},{"type":"v1","addr":"172.21.2.102:6821","nonce":36087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6823","nonce":36087},{"type":"v1","addr":"172.21.2.102:6825","nonce":36087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6831","nonce":36087},{"type":"v1","addr":"172.21.2.102:6833","nonce":36087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6827","nonce":36087},{"type":"v1","addr":"172.21.2.102:6829","nonce":36087}]},"public_addr":"172.21.2.102:6821/36087","cluster_addr":"172.21.2.102:6825/36087","heartbeat_back_addr":"172.21.2.102:6833/36087","heartbeat_front_addr":"172.21.2.102:6829/36087","state":["exists","up"]},{"osd":2,"uuid":"153700af-747f-481f-bdf4-d6072d3a842f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":187,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6810","nonce":36089},{"type":"v1","addr":"172.21.2.102:6811","nonce":36089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6812","nonce":36089},{"type":"v1","addr":"172.21.2.102:6813","nonce":36089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6816","nonce":36089},{"type":"v1","addr":"172.21.2.102:6817","nonce":36089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6814","nonce":36089},{"type":"v1","addr":"172.21.2.102:6815","nonce":36089}]},"public_addr":"172.21.2.102:6811/36089","cluster_addr":"172.21.2.102:6813/36089","heartbeat_back_addr":"172.21.2.102:6817/36089","heartbeat_front_addr":"172.21.2.102:6815/36089","state":["exists","up"]},{"osd":3,"uuid":"cbc678f4-83a9-45e6-8830-2f7c232b1ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":187,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6802","nonce":36090},{"type":"v1","addr":"172.21.2.102:6803","nonce":36090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6804","nonce":36090},{"type":"v1","addr":"172.21.2.102:6805","nonce":36090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6808","nonce":36090},{"type":"v1","addr":"172.21.2.102:6809","nonce":36090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6806","nonce":36090},{"type":"v1","addr":"172.21.2.102:6807","nonce":36090}]},"public_addr":"172.21.2.102:6803/36090","cluster_addr":"172.21.2.102:6805/36090","heartbeat_back_addr":"172.21.2.102:6809/36090","heartbeat_front_addr":"172.21.2.102:6807/36090","state":["exists","up"]},{"osd":4,"uuid":"c01e4008-b258-4f10-afd2-9e7f37214351","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":187,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6808","nonce":36368},{"type":"v1","addr":"172.21.2.113:6811","nonce":36368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6814","nonce":36368},{"type":"v1","addr":"172.21.2.113:6818","nonce":36368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6826","nonce":36368},{"type":"v1","addr":"172.21.2.113:6828","nonce":36368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6820","nonce":36368},{"type":"v1","addr":"172.21.2.113:6824","nonce":36368}]},"public_addr":"172.21.2.113:6811/36368","cluster_addr":"172.21.2.113:6818/36368","heartbeat_back_addr":"172.21.2.113:6828/36368","heartbeat_front_addr":"172.21.2.113:6824/36368","state":["exists","up"]},{"osd":5,"uuid":"551f6007-daf8-4a98-8ee0-c2b16bcccb5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":178,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6802","nonce":36366},{"type":"v1","addr":"172.21.2.113:6805","nonce":36366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6807","nonce":36366},{"type":"v1","addr":"172.21.2.113:6810","nonce":36366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6819","nonce":36366},{"type":"v1","addr":"172.21.2.113:6822","nonce":36366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6812","nonce":36366},{"type":"v1","addr":"172.21.2.113:6815","nonce":36366}]},"public_addr":"172.21.2.113:6805/36366","cluster_addr":"172.21.2.113:6810/36366","heartbeat_back_addr":"172.21.2.113:6822/36366","heartbeat_front_addr":"172.21.2.113:6815/36366","state":["exists","up"]},{"osd":6,"uuid":"bf67c040-0e50-48be-b102-c3943bc9ebef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":187,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6800","nonce":36367},{"type":"v1","addr":"172.21.2.113:6801","nonce":36367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6803","nonce":36367},{"type":"v1","addr":"172.21.2.113:6804","nonce":36367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6813","nonce":36367},{"type":"v1","addr":"172.21.2.113:6816","nonce":36367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6806","nonce":36367},{"type":"v1","addr":"172.21.2.113:6809","nonce":36367}]},"public_addr":"172.21.2.113:6801/36367","cluster_addr":"172.21.2.113:6804/36367","heartbeat_back_addr":"172.21.2.113:6816/36367","heartbeat_front_addr":"172.21.2.113:6809/36367","state":["exists","up"]},{"osd":7,"uuid":"f5461f80-018d-428f-be89-2eb2bcbe7bb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":187,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6817","nonce":36365},{"type":"v1","addr":"172.21.2.113:6821","nonce":36365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6823","nonce":36365},{"type":"v1","addr":"172.21.2.113:6825","nonce":36365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6830","nonce":36365},{"type":"v1","addr":"172.21.2.113:6831","nonce":36365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6827","nonce":36365},{"type":"v1","addr":"172.21.2.113:6829","nonce":36365}]},"public_addr":"172.21.2.113:6821/36365","cluster_addr":"172.21.2.113:6825/36365","heartbeat_back_addr":"172.21.2.113:6831/36365","heartbeat_front_addr":"172.21.2.113:6829/36365","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:39:33.629 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:33.627 7f36954b1700 1 -- 172.21.2.102:0/3216321036 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f36700186c0 msgr2=0x7f367001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:33.629 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:33.627 7f36954b1700 1 --2- 172.21.2.102:0/3216321036 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f36700186c0 0x7f367001abc0 crc :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:33.629 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:33.627 7f36954b1700 1 -- 172.21.2.102:0/3216321036 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f369009c870 msgr2=0x7f36900a0500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:33.629 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:33.627 7f36954b1700 1 --2- 172.21.2.102:0/3216321036 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f369009c870 0x7f36900a0500 secure :-1 s=READY pgs=3243 cs=0 l=1 rev1=1 rx=0x7f3680001c20 tx=0x7f368000a970).stop 2021-02-10T11:39:33.629 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:33.627 7f36954b1700 1 -- 172.21.2.102:0/3216321036 shutdown_connections 2021-02-10T11:39:33.629 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:33.627 7f36954b1700 1 --2- 172.21.2.102:0/3216321036 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f36700186c0 0x7f367001abc0 unknown :-1 s=CLOSED pgs=1418 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:33.630 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:33.627 7f36954b1700 1 --2- 172.21.2.102:0/3216321036 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f36900a0f20 0x7f36900bacd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:33.630 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:33.627 7f36954b1700 1 --2- 172.21.2.102:0/3216321036 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f36900a0b20 0x7f36900ba6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:33.630 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:33.627 7f36954b1700 1 --2- 172.21.2.102:0/3216321036 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f369009c870 0x7f36900a0500 unknown :-1 s=CLOSED pgs=3243 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:33.630 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:33.627 7f36954b1700 1 -- 172.21.2.102:0/3216321036 >> 172.21.2.102:0/3216321036 conn(0x7f36900a8740 msgr2=0x7f36900b50c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:33.630 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:33.627 7f36954b1700 1 -- 172.21.2.102:0/3216321036 shutdown_connections 2021-02-10T11:39:33.630 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:33.627 7f36954b1700 1 -- 172.21.2.102:0/3216321036 wait complete. 2021-02-10T11:39:33.638 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:39:34.073 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.052 7f908eae7700 1 Processor -- start 2021-02-10T11:39:34.073 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.052 7f908eae7700 1 -- start start 2021-02-10T11:39:34.073 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.053 7f908eae7700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f908810afe0 0x7f90880cf740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:34.073 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.053 7f908eae7700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90880fe1b0 con 0x7f90880d1860 2021-02-10T11:39:34.073 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.053 7f908eae7700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90881b0ca0 con 0x7f90880d1c60 2021-02-10T11:39:34.073 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.053 7f908eae7700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9088109a50 con 0x7f908810afe0 2021-02-10T11:39:34.073 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.053 7f908c903700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f90880d1860 0x7f90881b04b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:60922/0 (socket says 172.21.2.102:60922) 2021-02-10T11:39:34.073 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.053 7f908c903700 1 -- 172.21.2.102:0/4067747399 learned_addr learned my addr 172.21.2.102:0/4067747399 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:39:34.073 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.053 7f90877fe700 1 -- 172.21.2.102:0/4067747399 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 831141983 0 0) 0x7f9070000ce0 con 0x7f90880d1860 2021-02-10T11:39:34.073 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.053 7f90877fe700 1 -- 172.21.2.102:0/4067747399 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9074002fe0 con 0x7f90880d1860 2021-02-10T11:39:34.074 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.054 7f908d104700 1 --2- 172.21.2.102:0/4067747399 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f908810afe0 0x7f90880cf740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:34.074 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.054 7f90877fe700 1 -- 172.21.2.102:0/4067747399 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3372170439 0 0) 0x7f90700010e0 con 0x7f90880d1860 2021-02-10T11:39:34.074 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.054 7f90877fe700 1 -- 172.21.2.102:0/4067747399 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f908810afe0 msgr2=0x7f90880cf740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:34.074 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.054 7f90877fe700 1 --2- 172.21.2.102:0/4067747399 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f908810afe0 0x7f90880cf740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:34.074 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.054 7f90877fe700 1 -- 172.21.2.102:0/4067747399 >> v1:172.21.2.113:6789/0 conn(0x7f90880d1c60 legacy=0x7f90880cef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:34.074 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.054 7f90877fe700 1 -- 172.21.2.102:0/4067747399 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f908810b3e0 con 0x7f90880d1860 2021-02-10T11:39:34.074 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.054 7f90877fe700 1 -- 172.21.2.102:0/4067747399 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7f9070001660 con 0x7f90880d1860 2021-02-10T11:39:34.074 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.054 7f90877fe700 1 -- 172.21.2.102:0/4067747399 >> v1:172.21.2.102:6789/0 conn(0x7f90880d1860 legacy=0x7f90881b04b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:34.074 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.054 7f90877fe700 1 --2- 172.21.2.102:0/4067747399 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9074003aa0 0x7f9074005ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:34.074 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.054 7f90877fe700 1 --2- 172.21.2.102:0/4067747399 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f9074006790 0x7f9074008c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:34.074 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.054 7f90877fe700 1 --2- 172.21.2.102:0/4067747399 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9074009590 0x7f907400ba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:34.075 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.054 7f90877fe700 1 -- 172.21.2.102:0/4067747399 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f907400c2d0 con 0x7f9074003aa0 2021-02-10T11:39:34.075 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.054 7f90877fe700 1 -- 172.21.2.102:0/4067747399 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f907400c4f0 con 0x7f9074006790 2021-02-10T11:39:34.075 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.054 7f90877fe700 1 -- 172.21.2.102:0/4067747399 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f907400c710 con 0x7f9074009590 2021-02-10T11:39:34.075 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.054 7f908c903700 1 --2- 172.21.2.102:0/4067747399 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9074003aa0 0x7f9074005ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:34.075 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.054 7f9087fff700 1 --2- 172.21.2.102:0/4067747399 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f9074006790 0x7f9074008c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:34.075 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.054 7f908d104700 1 --2- 172.21.2.102:0/4067747399 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9074009590 0x7f907400ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:34.075 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.055 7f908c903700 1 -- 172.21.2.102:0/4067747399 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9074009590 msgr2=0x7f907400ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:34.075 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.055 7f908c903700 1 --2- 172.21.2.102:0/4067747399 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9074009590 0x7f907400ba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:34.075 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.055 7f908c903700 1 -- 172.21.2.102:0/4067747399 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f9074006790 msgr2=0x7f9074008c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:34.075 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.055 7f908c903700 1 --2- 172.21.2.102:0/4067747399 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f9074006790 0x7f9074008c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:34.075 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.055 7f908c903700 1 -- 172.21.2.102:0/4067747399 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f907400c9b0 con 0x7f9074003aa0 2021-02-10T11:39:34.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.055 7f908c903700 1 --2- 172.21.2.102:0/4067747399 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9074003aa0 0x7f9074005ef0 secure :-1 s=READY pgs=3245 cs=0 l=1 rev1=1 rx=0x7f907004b690 tx=0x7f9070001bf0).ready entity=mon.0 client_cookie=840fdf434bd8255f server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:34.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.055 7f90877fe700 1 -- 172.21.2.102:0/4067747399 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9070000ce0 con 0x7f9074003aa0 2021-02-10T11:39:34.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.055 7f90877fe700 1 -- 172.21.2.102:0/4067747399 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f907004e310 con 0x7f9074003aa0 2021-02-10T11:39:34.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.055 7f90877fe700 1 -- 172.21.2.102:0/4067747399 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f907004e870 con 0x7f9074003aa0 2021-02-10T11:39:34.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.055 7f908eae7700 1 -- 172.21.2.102:0/4067747399 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9074003aa0 msgr2=0x7f9074005ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:34.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.055 7f908eae7700 1 --2- 172.21.2.102:0/4067747399 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9074003aa0 0x7f9074005ef0 secure :-1 s=READY pgs=3245 cs=0 l=1 rev1=1 rx=0x7f907004b690 tx=0x7f9070001bf0).stop 2021-02-10T11:39:34.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.055 7f908d104700 1 -- 172.21.2.102:0/4067747399 reap_dead start 2021-02-10T11:39:34.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.055 7f908eae7700 1 -- 172.21.2.102:0/4067747399 shutdown_connections 2021-02-10T11:39:34.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.055 7f908eae7700 1 -- 172.21.2.102:0/4067747399 >> 172.21.2.102:0/4067747399 conn(0x7f90880ce410 msgr2=0x7f90880d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:34.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.055 7f908eae7700 1 -- 172.21.2.102:0/4067747399 shutdown_connections 2021-02-10T11:39:34.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.056 7f908eae7700 1 -- 172.21.2.102:0/4067747399 wait complete. 2021-02-10T11:39:34.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.070 7f908eae7700 1 Processor -- start 2021-02-10T11:39:34.077 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.070 7f908eae7700 1 -- start start 2021-02-10T11:39:34.078 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.071 7f908eae7700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f90880b5fb0 0x7f90880a81f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:34.078 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.071 7f908eae7700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f90880a8810 0x7f90880a2540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:34.078 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.071 7f908eae7700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f90880a8c10 0x7f90880a2b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:34.079 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.071 7f908eae7700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90880b63b0 con 0x7f90880a8810 2021-02-10T11:39:34.079 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.071 7f908eae7700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90880a9210 con 0x7f90880a8c10 2021-02-10T11:39:34.079 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.071 7f908eae7700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90880b2650 con 0x7f90880b5fb0 2021-02-10T11:39:34.079 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.071 7f9087fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f90880a8810 0x7f90880a2540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:34.079 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.071 7f9087fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f90880a8810 0x7f90880a2540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:40478/0 (socket says 172.21.2.102:40478) 2021-02-10T11:39:34.079 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.071 7f9087fff700 1 -- 172.21.2.102:0/3785127651 learned_addr learned my addr 172.21.2.102:0/3785127651 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:39:34.079 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.071 7f908d104700 1 --2- 172.21.2.102:0/3785127651 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f90880a8c10 0x7f90880a2b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:34.079 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.071 7f908c903700 1 --2- 172.21.2.102:0/3785127651 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f90880b5fb0 0x7f90880a81f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:34.079 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.072 7f9087fff700 1 -- 172.21.2.102:0/3785127651 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f90880b5fb0 msgr2=0x7f90880a81f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:34.079 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.072 7f9087fff700 1 --2- 172.21.2.102:0/3785127651 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f90880b5fb0 0x7f90880a81f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:34.079 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.072 7f9087fff700 1 -- 172.21.2.102:0/3785127651 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f90880a8c10 msgr2=0x7f90880a2b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:34.079 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.072 7f9087fff700 1 --2- 172.21.2.102:0/3785127651 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f90880a8c10 0x7f90880a2b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:34.080 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.072 7f9087fff700 1 -- 172.21.2.102:0/3785127651 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90880a3180 con 0x7f90880a8810 2021-02-10T11:39:34.080 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.072 7f908d104700 1 --2- 172.21.2.102:0/3785127651 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f90880a8c10 0x7f90880a2b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:39:34.080 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.072 7f908c903700 1 --2- 172.21.2.102:0/3785127651 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f90880b5fb0 0x7f90880a81f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:39:34.080 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.072 7f9087fff700 1 --2- 172.21.2.102:0/3785127651 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f90880a8810 0x7f90880a2540 secure :-1 s=READY pgs=3246 cs=0 l=1 rev1=1 rx=0x7f907800b610 tx=0x7f907800b9b0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:34.080 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.072 7f90857fa700 1 -- 172.21.2.102:0/3785127651 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9078000960 con 0x7f90880a8810 2021-02-10T11:39:34.080 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.072 7f90857fa700 1 -- 172.21.2.102:0/3785127651 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f907800bda0 con 0x7f90880a8810 2021-02-10T11:39:34.080 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.072 7f90857fa700 1 -- 172.21.2.102:0/3785127651 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f90780022b0 con 0x7f90880a8810 2021-02-10T11:39:34.080 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.072 7f908eae7700 1 -- 172.21.2.102:0/3785127651 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90880a3180 con 0x7f90880a8810 2021-02-10T11:39:34.080 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.072 7f908eae7700 1 -- 172.21.2.102:0/3785127651 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90880a3710 con 0x7f90880a8810 2021-02-10T11:39:34.080 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.073 7f90857fa700 1 -- 172.21.2.102:0/3785127651 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f9078000ca0 con 0x7f90880a8810 2021-02-10T11:39:34.080 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.073 7f90857fa700 1 --2- 172.21.2.102:0/3785127651 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f906c0186c0 0x7f906c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:34.081 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.073 7f90857fa700 1 -- 172.21.2.102:0/3785127651 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(189..189 src has 1..189) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f9078032570 con 0x7f90880a8810 2021-02-10T11:39:34.081 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.073 7f908c903700 1 --2- 172.21.2.102:0/3785127651 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f906c0186c0 0x7f906c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:34.081 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.073 7f908c903700 1 --2- 172.21.2.102:0/3785127651 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f906c0186c0 0x7f906c01abc0 crc :-1 s=READY pgs=1419 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:34.105 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.103 7f908eae7700 1 -- 172.21.2.102:0/3785127651 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90880fe1b0 con 0x7f90880a8810 2021-02-10T11:39:34.107 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.105 7f90857fa700 1 -- 172.21.2.102:0/3785127651 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f90780174e0 con 0x7f90880a8810 2021-02-10T11:39:34.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.234 7f908eae7700 1 -- 172.21.2.102:0/3785127651 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f90880fe1b0 con 0x7f90880a8810 2021-02-10T11:39:34.240 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.237 7f90857fa700 1 -- 172.21.2.102:0/3785127651 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.0 v108) v1 ==== 184+0+0 (secure 0 0 0) 0x7f90780174e0 con 0x7f90880a8810 2021-02-10T11:39:34.240 INFO:teuthology.orchestra.run.gibba002.stderr:updated caps for client.0 2021-02-10T11:39:34.241 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.239 7f908eae7700 1 -- 172.21.2.102:0/3785127651 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f906c0186c0 msgr2=0x7f906c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:34.241 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.239 7f908eae7700 1 --2- 172.21.2.102:0/3785127651 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f906c0186c0 0x7f906c01abc0 crc :-1 s=READY pgs=1419 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:34.241 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.239 7f908eae7700 1 -- 172.21.2.102:0/3785127651 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f90880a8810 msgr2=0x7f90880a2540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:34.241 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.239 7f908eae7700 1 --2- 172.21.2.102:0/3785127651 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f90880a8810 0x7f90880a2540 secure :-1 s=READY pgs=3246 cs=0 l=1 rev1=1 rx=0x7f907800b610 tx=0x7f907800b9b0).stop 2021-02-10T11:39:34.242 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.239 7f908eae7700 1 -- 172.21.2.102:0/3785127651 shutdown_connections 2021-02-10T11:39:34.242 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.239 7f908eae7700 1 --2- 172.21.2.102:0/3785127651 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f906c0186c0 0x7f906c01abc0 unknown :-1 s=CLOSED pgs=1419 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:34.242 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.239 7f908eae7700 1 --2- 172.21.2.102:0/3785127651 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f90880a8c10 0x7f90880a2b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:34.242 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.239 7f908eae7700 1 --2- 172.21.2.102:0/3785127651 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f90880a8810 0x7f90880a2540 unknown :-1 s=CLOSED pgs=3246 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:34.242 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.239 7f908eae7700 1 --2- 172.21.2.102:0/3785127651 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f90880b5fb0 0x7f90880a81f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:34.242 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.239 7f908eae7700 1 -- 172.21.2.102:0/3785127651 >> 172.21.2.102:0/3785127651 conn(0x7f908809e660 msgr2=0x7f90880a9550 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:34.242 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.239 7f908eae7700 1 -- 172.21.2.102:0/3785127651 shutdown_connections 2021-02-10T11:39:34.242 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.239 7f908eae7700 1 -- 172.21.2.102:0/3785127651 wait complete. 2021-02-10T11:39:34.250 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.1 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:39:34.692 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.672 7f4d21515700 1 Processor -- start 2021-02-10T11:39:34.692 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.672 7f4d21515700 1 -- start start 2021-02-10T11:39:34.692 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.672 7f4d21515700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4d1c0d1c80 0x7f4d1c0cef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:34.692 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.672 7f4d21515700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d1c0d0180 con 0x7f4d1c0d1880 2021-02-10T11:39:34.692 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.672 7f4d21515700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d1c0fe1d0 con 0x7f4d1c0cf590 2021-02-10T11:39:34.693 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.672 7f4d21515700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d1c1b0cc0 con 0x7f4d1c0d1c80 2021-02-10T11:39:34.693 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.673 7f4d1b2bf700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f4d1c0d1880 0x7f4d1c1b04d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:60942/0 (socket says 172.21.2.102:60942) 2021-02-10T11:39:34.693 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.673 7f4d1b2bf700 1 -- 172.21.2.102:0/2352923173 learned_addr learned my addr 172.21.2.102:0/2352923173 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:39:34.693 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.673 7f4d1a2bd700 1 -- 172.21.2.102:0/2352923173 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 409642372 0 0) 0x7f4d040010b0 con 0x7f4d1c0d1880 2021-02-10T11:39:34.693 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.673 7f4d1a2bd700 1 -- 172.21.2.102:0/2352923173 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d08002fe0 con 0x7f4d1c0d1880 2021-02-10T11:39:34.693 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.673 7f4d1aabe700 1 --2- 172.21.2.102:0/2352923173 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4d1c0d1c80 0x7f4d1c0cef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:34.693 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.673 7f4d1a2bd700 1 -- 172.21.2.102:0/2352923173 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3700299643 0 0) 0x7f4d040010b0 con 0x7f4d1c0d1880 2021-02-10T11:39:34.693 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.673 7f4d1a2bd700 1 -- 172.21.2.102:0/2352923173 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4d1c0d1c80 msgr2=0x7f4d1c0cef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:34.693 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.673 7f4d1a2bd700 1 --2- 172.21.2.102:0/2352923173 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4d1c0d1c80 0x7f4d1c0cef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:34.693 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.673 7f4d1a2bd700 1 -- 172.21.2.102:0/2352923173 >> v1:172.21.2.113:6789/0 conn(0x7f4d1c0cf590 legacy=0x7f4d1c0cf990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:34.693 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.673 7f4d1a2bd700 1 -- 172.21.2.102:0/2352923173 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d1c109a70 con 0x7f4d1c0d1880 2021-02-10T11:39:34.693 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.673 7f4d1a2bd700 1 -- 172.21.2.102:0/2352923173 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7f4d040015b0 con 0x7f4d1c0d1880 2021-02-10T11:39:34.694 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.673 7f4d1a2bd700 1 -- 172.21.2.102:0/2352923173 >> v1:172.21.2.102:6789/0 conn(0x7f4d1c0d1880 legacy=0x7f4d1c1b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:34.694 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.673 7f4d1a2bd700 1 --2- 172.21.2.102:0/2352923173 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4d08003af0 0x7f4d08005f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:34.694 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.673 7f4d1a2bd700 1 --2- 172.21.2.102:0/2352923173 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4d080067a0 0x7f4d08008ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:34.694 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.673 7f4d1a2bd700 1 --2- 172.21.2.102:0/2352923173 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4d08009580 0x7f4d0800ba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:34.694 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.673 7f4d1a2bd700 1 -- 172.21.2.102:0/2352923173 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d0800c2c0 con 0x7f4d080067a0 2021-02-10T11:39:34.694 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.673 7f4d1a2bd700 1 -- 172.21.2.102:0/2352923173 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d0800c4e0 con 0x7f4d08003af0 2021-02-10T11:39:34.694 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.673 7f4d1a2bd700 1 -- 172.21.2.102:0/2352923173 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d0800c700 con 0x7f4d08009580 2021-02-10T11:39:34.694 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.674 7f4d1aabe700 1 --2- 172.21.2.102:0/2352923173 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4d080067a0 0x7f4d08008ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:34.694 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.674 7f4d1b2bf700 1 --2- 172.21.2.102:0/2352923173 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4d08003af0 0x7f4d08005f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:34.694 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.674 7f4d1bac0700 1 --2- 172.21.2.102:0/2352923173 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4d08009580 0x7f4d0800ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:34.694 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.674 7f4d1aabe700 1 -- 172.21.2.102:0/2352923173 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4d08009580 msgr2=0x7f4d0800ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:34.694 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.674 7f4d1aabe700 1 --2- 172.21.2.102:0/2352923173 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4d08009580 0x7f4d0800ba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:34.695 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.674 7f4d1aabe700 1 -- 172.21.2.102:0/2352923173 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4d08003af0 msgr2=0x7f4d08005f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:34.695 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.674 7f4d1aabe700 1 --2- 172.21.2.102:0/2352923173 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4d08003af0 0x7f4d08005f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:34.695 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.674 7f4d1aabe700 1 -- 172.21.2.102:0/2352923173 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d0800c9a0 con 0x7f4d080067a0 2021-02-10T11:39:34.695 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.674 7f4d1aabe700 1 --2- 172.21.2.102:0/2352923173 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4d080067a0 0x7f4d08008ca0 secure :-1 s=READY pgs=3248 cs=0 l=1 rev1=1 rx=0x7f4d0c006a90 tx=0x7f4d0c009a90).ready entity=mon.0 client_cookie=ddee8cf2391849bc server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:34.695 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.674 7f4d1a2bd700 1 -- 172.21.2.102:0/2352923173 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4d0c00c0c0 con 0x7f4d080067a0 2021-02-10T11:39:34.695 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.674 7f4d1a2bd700 1 -- 172.21.2.102:0/2352923173 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d0c00c440 con 0x7f4d080067a0 2021-02-10T11:39:34.695 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.674 7f4d1a2bd700 1 -- 172.21.2.102:0/2352923173 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4d0c00c0c0 con 0x7f4d080067a0 2021-02-10T11:39:34.695 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.674 7f4d21515700 1 -- 172.21.2.102:0/2352923173 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4d080067a0 msgr2=0x7f4d08008ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:34.695 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.674 7f4d21515700 1 --2- 172.21.2.102:0/2352923173 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4d080067a0 0x7f4d08008ca0 secure :-1 s=READY pgs=3248 cs=0 l=1 rev1=1 rx=0x7f4d0c006a90 tx=0x7f4d0c009a90).stop 2021-02-10T11:39:34.695 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.674 7f4d1bac0700 1 -- 172.21.2.102:0/2352923173 reap_dead start 2021-02-10T11:39:34.695 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.675 7f4d21515700 1 -- 172.21.2.102:0/2352923173 shutdown_connections 2021-02-10T11:39:34.695 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.675 7f4d21515700 1 -- 172.21.2.102:0/2352923173 >> 172.21.2.102:0/2352923173 conn(0x7f4d1c0ce430 msgr2=0x7f4d1c0d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:34.695 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.675 7f4d21515700 1 -- 172.21.2.102:0/2352923173 shutdown_connections 2021-02-10T11:39:34.696 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.675 7f4d21515700 1 -- 172.21.2.102:0/2352923173 wait complete. 2021-02-10T11:39:34.696 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.690 7f4d21515700 1 Processor -- start 2021-02-10T11:39:34.696 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.690 7f4d21515700 1 -- start start 2021-02-10T11:39:34.698 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.690 7f4d21515700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4d1c1a0580 0x7f4d1c086fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:34.698 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.690 7f4d21515700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4d1c0b66b0 0x7f4d1c0b6ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:34.698 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.690 7f4d21515700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4d1c082bb0 0x7f4d1c082fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:34.698 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.690 7f4d21515700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d1c0875d0 con 0x7f4d1c0b66b0 2021-02-10T11:39:34.698 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.690 7f4d21515700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d1c0835d0 con 0x7f4d1c1a0580 2021-02-10T11:39:34.698 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.690 7f4d21515700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d1c0837f0 con 0x7f4d1c082bb0 2021-02-10T11:39:34.698 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.690 7f4d1aabe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4d1c0b66b0 0x7f4d1c0b6ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:34.698 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.690 7f4d1aabe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4d1c0b66b0 0x7f4d1c0b6ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:40496/0 (socket says 172.21.2.102:40496) 2021-02-10T11:39:34.698 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.690 7f4d1aabe700 1 -- 172.21.2.102:0/4081818973 learned_addr learned my addr 172.21.2.102:0/4081818973 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:39:34.698 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.691 7f4d1aabe700 1 -- 172.21.2.102:0/4081818973 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4d1c082bb0 msgr2=0x7f4d1c082fb0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:39:34.698 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.691 7f4d1aabe700 1 --2- 172.21.2.102:0/4081818973 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4d1c082bb0 0x7f4d1c082fb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:34.698 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.691 7f4d1aabe700 1 -- 172.21.2.102:0/4081818973 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4d1c1a0580 msgr2=0x7f4d1c086fb0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:39:34.699 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.691 7f4d1aabe700 1 --2- 172.21.2.102:0/4081818973 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4d1c1a0580 0x7f4d1c086fb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:34.699 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.691 7f4d1aabe700 1 -- 172.21.2.102:0/4081818973 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d1c083a10 con 0x7f4d1c0b66b0 2021-02-10T11:39:34.699 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.691 7f4d1aabe700 1 --2- 172.21.2.102:0/4081818973 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4d1c0b66b0 0x7f4d1c0b6ab0 secure :-1 s=READY pgs=3249 cs=0 l=1 rev1=1 rx=0x7f4d0c006710 tx=0x7f4d0c00b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:34.699 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.691 7f4d13fff700 1 -- 172.21.2.102:0/4081818973 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4d0c000c90 con 0x7f4d1c0b66b0 2021-02-10T11:39:34.699 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.691 7f4d13fff700 1 -- 172.21.2.102:0/4081818973 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d0c002030 con 0x7f4d1c0b66b0 2021-02-10T11:39:34.699 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.691 7f4d13fff700 1 -- 172.21.2.102:0/4081818973 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4d0c002400 con 0x7f4d1c0b66b0 2021-02-10T11:39:34.699 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.691 7f4d21515700 1 -- 172.21.2.102:0/4081818973 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d1c083a10 con 0x7f4d1c0b66b0 2021-02-10T11:39:34.699 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.691 7f4d21515700 1 -- 172.21.2.102:0/4081818973 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d1c083a10 con 0x7f4d1c0b66b0 2021-02-10T11:39:34.699 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.692 7f4d13fff700 1 -- 172.21.2.102:0/4081818973 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4d0c000c90 con 0x7f4d1c0b66b0 2021-02-10T11:39:34.699 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.692 7f4d13fff700 1 --2- 172.21.2.102:0/4081818973 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4d000186c0 0x7f4d0001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:34.699 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.692 7f4d13fff700 1 -- 172.21.2.102:0/4081818973 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(189..189 src has 1..189) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f4d0c032500 con 0x7f4d1c0b66b0 2021-02-10T11:39:34.699 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.692 7f4d1b2bf700 1 --2- 172.21.2.102:0/4081818973 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4d000186c0 0x7f4d0001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:34.699 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.692 7f4d1b2bf700 1 --2- 172.21.2.102:0/4081818973 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4d000186c0 0x7f4d0001abc0 crc :-1 s=READY pgs=1420 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:34.724 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.722 7f4d21515700 1 -- 172.21.2.102:0/4081818973 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d1c0875d0 con 0x7f4d1c0b66b0 2021-02-10T11:39:34.727 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.725 7f4d13fff700 1 -- 172.21.2.102:0/4081818973 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f4d0c009020 con 0x7f4d1c0b66b0 2021-02-10T11:39:34.858 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.856 7f4d21515700 1 -- 172.21.2.102:0/4081818973 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f4d1c0875d0 con 0x7f4d1c0b66b0 2021-02-10T11:39:34.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.859 7f4d13fff700 1 -- 172.21.2.102:0/4081818973 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.1 v109) v1 ==== 184+0+0 (secure 0 0 0) 0x7f4d0c01f4b0 con 0x7f4d1c0b66b0 2021-02-10T11:39:34.861 INFO:teuthology.orchestra.run.gibba002.stderr:updated caps for client.1 2021-02-10T11:39:34.862 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.860 7f4d21515700 1 -- 172.21.2.102:0/4081818973 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4d000186c0 msgr2=0x7f4d0001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:34.863 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.860 7f4d21515700 1 --2- 172.21.2.102:0/4081818973 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4d000186c0 0x7f4d0001abc0 crc :-1 s=READY pgs=1420 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:34.863 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.860 7f4d21515700 1 -- 172.21.2.102:0/4081818973 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4d1c0b66b0 msgr2=0x7f4d1c0b6ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:34.863 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.860 7f4d21515700 1 --2- 172.21.2.102:0/4081818973 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4d1c0b66b0 0x7f4d1c0b6ab0 secure :-1 s=READY pgs=3249 cs=0 l=1 rev1=1 rx=0x7f4d0c006710 tx=0x7f4d0c00b960).stop 2021-02-10T11:39:34.863 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.861 7f4d21515700 1 -- 172.21.2.102:0/4081818973 shutdown_connections 2021-02-10T11:39:34.863 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.861 7f4d21515700 1 --2- 172.21.2.102:0/4081818973 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4d1c082bb0 0x7f4d1c082fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:34.863 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.861 7f4d21515700 1 --2- 172.21.2.102:0/4081818973 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4d1c0b66b0 0x7f4d1c0b6ab0 unknown :-1 s=CLOSED pgs=3249 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:34.864 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.861 7f4d21515700 1 --2- 172.21.2.102:0/4081818973 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4d000186c0 0x7f4d0001abc0 unknown :-1 s=CLOSED pgs=1420 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:34.864 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.861 7f4d21515700 1 --2- 172.21.2.102:0/4081818973 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4d1c1a0580 0x7f4d1c086fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:34.864 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.861 7f4d21515700 1 -- 172.21.2.102:0/4081818973 >> 172.21.2.102:0/4081818973 conn(0x7f4d1c0afc60 msgr2=0x7f4d1c0b50d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:34.864 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.861 7f4d21515700 1 -- 172.21.2.102:0/4081818973 shutdown_connections 2021-02-10T11:39:34.864 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:34.861 7f4d21515700 1 -- 172.21.2.102:0/4081818973 wait complete. 2021-02-10T11:39:34.872 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.2 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:39:35.313 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:35.294 7f5d7f852700 1 Processor -- start 2021-02-10T11:39:35.313 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:35.294 7f5d7f852700 1 -- start start 2021-02-10T11:39:35.313 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:35.295 7f5d7f852700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f5d780d1880 0x7f5d781b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:35.313 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:35.295 7f5d7f852700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5d781b0af0 0x7f5d78108950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:35.313 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:35.295 7f5d7f852700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d780fe1d0 con 0x7f5d780d1c80 2021-02-10T11:39:35.313 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:35.295 7f5d7f852700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d78109a70 con 0x7f5d780d1880 2021-02-10T11:39:35.313 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:35.295 7f5d7f852700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d78109760 con 0x7f5d781b0af0 2021-02-10T11:39:35.313 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:35.295 7f5d7d66e700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f5d780d1880 0x7f5d781b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:35.314 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:35.295 7f5d7d66e700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f5d780d1880 0x7f5d781b04d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:40504/0 (socket says 172.21.2.102:40504) 2021-02-10T11:39:35.314 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:35.295 7f5d7d66e700 1 -- 172.21.2.102:0/4086478961 learned_addr learned my addr 172.21.2.102:0/4086478961 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:39:35.314 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:35.295 7f5d7de6f700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f5d780d1c80 0x7f5d78108f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:60960/0 (socket says 172.21.2.102:60960) 2021-02-10T11:39:35.314 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:35.295 7f5d7ce6d700 1 --2- 172.21.2.102:0/4086478961 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5d781b0af0 0x7f5d78108950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:35.314 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:35.295 7f5d7d66e700 1 -- 172.21.2.102:0/4086478961 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5d781b0af0 msgr2=0x7f5d78108950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:35.314 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:35.295 7f5d7d66e700 1 --2- 172.21.2.102:0/4086478961 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5d781b0af0 0x7f5d78108950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:35.314 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:35.295 7f5d7d66e700 1 -- 172.21.2.102:0/4086478961 >> v1:172.21.2.102:6789/0 conn(0x7f5d780d1c80 legacy=0x7f5d78108f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:35.314 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:35.295 7f5d7d66e700 1 -- 172.21.2.102:0/4086478961 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d78088070 con 0x7f5d780d1880 2021-02-10T11:39:35.314 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:35.295 7f5d7ce6d700 1 --2- 172.21.2.102:0/4086478961 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5d781b0af0 0x7f5d78108950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:39:35.314 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:35.296 7f5d7d66e700 1 --2- 172.21.2.102:0/4086478961 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5d780d1880 0x7f5d781b04d0 secure :-1 s=READY pgs=3251 cs=0 l=1 rev1=1 rx=0x7f5d6c006700 tx=0x7f5d6c009b80).ready entity=mon.0 client_cookie=6c402cb72170492e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:35.314 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:35.296 7f5d6bfff700 1 -- 172.21.2.102:0/4086478961 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5d6c00c0c0 con 0x7f5d780d1880 2021-02-10T11:39:35.314 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:35.296 7f5d6bfff700 1 -- 172.21.2.102:0/4086478961 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d6c00c440 con 0x7f5d780d1880 2021-02-10T11:39:35.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:35.296 7f5d6bfff700 1 -- 172.21.2.102:0/4086478961 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5d6c00c960 con 0x7f5d780d1880 2021-02-10T11:39:35.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:35.296 7f5d7f852700 1 -- 172.21.2.102:0/4086478961 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5d780d1880 msgr2=0x7f5d781b04d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:35.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:35.296 7f5d7f852700 1 --2- 172.21.2.102:0/4086478961 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5d780d1880 0x7f5d781b04d0 secure :-1 s=READY pgs=3251 cs=0 l=1 rev1=1 rx=0x7f5d6c006700 tx=0x7f5d6c009b80).stop 2021-02-10T11:39:35.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:35.296 7f5d7f852700 1 -- 172.21.2.102:0/4086478961 shutdown_connections 2021-02-10T11:39:35.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:35.297 7f5d7f852700 1 --2- 172.21.2.102:0/4086478961 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5d781b0af0 0x7f5d78108950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:35.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:35.297 7f5d7f852700 1 --2- 172.21.2.102:0/4086478961 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5d780d1880 0x7f5d781b04d0 unknown :-1 s=CLOSED pgs=3251 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:35.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:35.297 7f5d7f852700 1 -- 172.21.2.102:0/4086478961 >> 172.21.2.102:0/4086478961 conn(0x7f5d780ce430 msgr2=0x7f5d780d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:35.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:35.297 7f5d7f852700 1 -- 172.21.2.102:0/4086478961 shutdown_connections 2021-02-10T11:39:35.315 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:35.297 7f5d7f852700 1 -- 172.21.2.102:0/4086478961 wait complete. 2021-02-10T11:39:35.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:35.311 7f5d7f852700 1 Processor -- start 2021-02-10T11:39:35.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:35.311 7f5d7f852700 1 -- start start 2021-02-10T11:39:35.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:35.312 7f5d7f852700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5d7818e010 0x7f5d780a8300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:35.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:35.312 7f5d7f852700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5d780a8920 0x7f5d78083e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:35.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:35.312 7f5d7f852700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5d78084490 0x7f5d7808a740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:35.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:35.312 7f5d7f852700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d78091ba0 con 0x7f5d780a8920 2021-02-10T11:39:35.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:35.312 7f5d7f852700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d780914c0 con 0x7f5d78084490 2021-02-10T11:39:35.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:35.312 7f5d7f852700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d7808ad60 con 0x7f5d7818e010 2021-02-10T11:39:35.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:35.312 7f5d7ce6d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5d780a8920 0x7f5d78083e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:35.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:35.312 7f5d7ce6d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5d780a8920 0x7f5d78083e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:40508/0 (socket says 172.21.2.102:40508) 2021-02-10T11:39:35.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:35.312 7f5d7ce6d700 1 -- 172.21.2.102:0/1509280841 learned_addr learned my addr 172.21.2.102:0/1509280841 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:39:35.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:35.312 7f5d7ce6d700 1 -- 172.21.2.102:0/1509280841 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5d7818e010 msgr2=0x7f5d780a8300 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:39:35.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:35.312 7f5d7ce6d700 1 --2- 172.21.2.102:0/1509280841 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5d7818e010 0x7f5d780a8300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:35.318 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:35.312 7f5d7ce6d700 1 -- 172.21.2.102:0/1509280841 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5d78084490 msgr2=0x7f5d7808a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:35.318 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:35.312 7f5d7ce6d700 1 --2- 172.21.2.102:0/1509280841 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5d78084490 0x7f5d7808a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:35.318 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:35.312 7f5d7ce6d700 1 -- 172.21.2.102:0/1509280841 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d7808af80 con 0x7f5d780a8920 2021-02-10T11:39:35.318 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:35.312 7f5d7ce6d700 1 --2- 172.21.2.102:0/1509280841 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5d780a8920 0x7f5d78083e70 secure :-1 s=READY pgs=3252 cs=0 l=1 rev1=1 rx=0x7f5d7400b580 tx=0x7f5d7400b920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:35.318 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:35.312 7f5d69ffb700 1 -- 172.21.2.102:0/1509280841 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5d7400bd10 con 0x7f5d780a8920 2021-02-10T11:39:35.318 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:35.312 7f5d69ffb700 1 -- 172.21.2.102:0/1509280841 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d74000c90 con 0x7f5d780a8920 2021-02-10T11:39:35.318 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:35.313 7f5d69ffb700 1 -- 172.21.2.102:0/1509280841 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5d740022b0 con 0x7f5d780a8920 2021-02-10T11:39:35.318 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:35.313 7f5d7f852700 1 -- 172.21.2.102:0/1509280841 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d7808af80 con 0x7f5d780a8920 2021-02-10T11:39:35.318 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:35.313 7f5d7f852700 1 -- 172.21.2.102:0/1509280841 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d7808b510 con 0x7f5d780a8920 2021-02-10T11:39:35.318 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:35.313 7f5d69ffb700 1 -- 172.21.2.102:0/1509280841 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f5d7400a000 con 0x7f5d780a8920 2021-02-10T11:39:35.318 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:35.313 7f5d69ffb700 1 --2- 172.21.2.102:0/1509280841 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5d580186c0 0x7f5d5801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:35.318 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:35.313 7f5d69ffb700 1 -- 172.21.2.102:0/1509280841 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(189..189 src has 1..189) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f5d740323d0 con 0x7f5d780a8920 2021-02-10T11:39:35.318 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:35.313 7f5d7d66e700 1 --2- 172.21.2.102:0/1509280841 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5d580186c0 0x7f5d5801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:35.319 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:35.314 7f5d7d66e700 1 --2- 172.21.2.102:0/1509280841 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5d580186c0 0x7f5d5801abc0 crc :-1 s=READY pgs=1421 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:35.347 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:35.344 7f5d7f852700 1 -- 172.21.2.102:0/1509280841 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d78082980 con 0x7f5d780a8920 2021-02-10T11:39:35.349 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:35.347 7f5d69ffb700 1 -- 172.21.2.102:0/1509280841 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f5d74000940 con 0x7f5d780a8920 2021-02-10T11:39:35.481 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:35.479 7f5d7f852700 1 -- 172.21.2.102:0/1509280841 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f5d78082980 con 0x7f5d780a8920 2021-02-10T11:39:35.484 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:35.482 7f5d69ffb700 1 -- 172.21.2.102:0/1509280841 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.2 v110) v1 ==== 184+0+0 (secure 0 0 0) 0x7f5d7401f4d0 con 0x7f5d780a8920 2021-02-10T11:39:35.488 INFO:teuthology.orchestra.run.gibba002.stderr:updated caps for client.2 2021-02-10T11:39:35.490 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:35.488 7f5d7f852700 1 -- 172.21.2.102:0/1509280841 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5d580186c0 msgr2=0x7f5d5801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:35.490 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:35.488 7f5d7f852700 1 --2- 172.21.2.102:0/1509280841 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5d580186c0 0x7f5d5801abc0 crc :-1 s=READY pgs=1421 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:35.490 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:35.488 7f5d7f852700 1 -- 172.21.2.102:0/1509280841 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5d780a8920 msgr2=0x7f5d78083e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:35.490 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:35.488 7f5d7f852700 1 --2- 172.21.2.102:0/1509280841 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5d780a8920 0x7f5d78083e70 secure :-1 s=READY pgs=3252 cs=0 l=1 rev1=1 rx=0x7f5d7400b580 tx=0x7f5d7400b920).stop 2021-02-10T11:39:35.490 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:35.488 7f5d7f852700 1 -- 172.21.2.102:0/1509280841 shutdown_connections 2021-02-10T11:39:35.490 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:35.488 7f5d7f852700 1 --2- 172.21.2.102:0/1509280841 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5d580186c0 0x7f5d5801abc0 unknown :-1 s=CLOSED pgs=1421 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:35.491 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:35.488 7f5d7f852700 1 --2- 172.21.2.102:0/1509280841 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5d78084490 0x7f5d7808a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:35.491 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:35.488 7f5d7f852700 1 --2- 172.21.2.102:0/1509280841 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5d780a8920 0x7f5d78083e70 unknown :-1 s=CLOSED pgs=3252 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:35.491 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:35.488 7f5d7f852700 1 --2- 172.21.2.102:0/1509280841 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5d7818e010 0x7f5d780a8300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:35.491 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:35.488 7f5d7f852700 1 -- 172.21.2.102:0/1509280841 >> 172.21.2.102:0/1509280841 conn(0x7f5d7819c5a0 msgr2=0x7f5d78082bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:35.491 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:35.488 7f5d7f852700 1 -- 172.21.2.102:0/1509280841 shutdown_connections 2021-02-10T11:39:35.491 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:35.488 7f5d7f852700 1 -- 172.21.2.102:0/1509280841 wait complete. 2021-02-10T11:39:35.499 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.3 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:39:35.942 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:35.922 7fc377a54700 1 Processor -- start 2021-02-10T11:39:35.942 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:35.922 7fc377a54700 1 -- start start 2021-02-10T11:39:35.942 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:35.923 7fc377a54700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc3700d1880 0x7fc3701b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:35.942 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:35.923 7fc377a54700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fc3700cff80 0x7fc370109140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:35.943 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:35.923 7fc377a54700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc370109a70 con 0x7fc3701b0af0 2021-02-10T11:39:35.943 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:35.923 7fc377a54700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc370109760 con 0x7fc3700cff80 2021-02-10T11:39:35.943 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:35.923 7fc377a54700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc370088070 con 0x7fc3700d1880 2021-02-10T11:39:35.943 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:35.923 7fc37506f700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fc3701b0af0 0x7fc370108950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:60972/0 (socket says 172.21.2.102:60972) 2021-02-10T11:39:35.943 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:35.923 7fc37506f700 1 -- 172.21.2.102:0/2816879847 learned_addr learned my addr 172.21.2.102:0/2816879847 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:39:35.943 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:35.923 7fc37486e700 1 -- 172.21.2.102:0/2816879847 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2073308835 0 0) 0x7fc3600010b0 con 0x7fc3701b0af0 2021-02-10T11:39:35.943 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:35.923 7fc37486e700 1 -- 172.21.2.102:0/2816879847 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc35c002fe0 con 0x7fc3701b0af0 2021-02-10T11:39:35.943 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:35.923 7fc376071700 1 --2- 172.21.2.102:0/2816879847 >> v2:172.21.2.113:3300/0 conn(0x7fc3700cff80 0x7fc370109140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:35.943 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:35.923 7fc375870700 1 --2- 172.21.2.102:0/2816879847 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc3700d1880 0x7fc3701b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:35.943 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:35.924 7fc37486e700 1 -- 172.21.2.102:0/2816879847 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1904740531 0 0) 0x7fc3600010b0 con 0x7fc3701b0af0 2021-02-10T11:39:35.943 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:35.924 7fc37486e700 1 -- 172.21.2.102:0/2816879847 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc3700d1880 msgr2=0x7fc3701b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:35.943 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:35.924 7fc37486e700 1 --2- 172.21.2.102:0/2816879847 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc3700d1880 0x7fc3701b04d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:35.943 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:35.924 7fc37486e700 1 -- 172.21.2.102:0/2816879847 >> v2:172.21.2.113:3300/0 conn(0x7fc3700cff80 msgr2=0x7fc370109140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:35.943 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:35.924 7fc37486e700 1 --2- 172.21.2.102:0/2816879847 >> v2:172.21.2.113:3300/0 conn(0x7fc3700cff80 0x7fc370109140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:35.944 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:35.924 7fc37486e700 1 -- 172.21.2.102:0/2816879847 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc370088260 con 0x7fc3701b0af0 2021-02-10T11:39:35.944 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:35.924 7fc37486e700 1 -- 172.21.2.102:0/2816879847 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7fc3600015b0 con 0x7fc3701b0af0 2021-02-10T11:39:35.944 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:35.924 7fc37486e700 1 -- 172.21.2.102:0/2816879847 >> v1:172.21.2.102:6789/0 conn(0x7fc3701b0af0 legacy=0x7fc370108950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:35.944 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:35.924 7fc37486e700 1 --2- 172.21.2.102:0/2816879847 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc35c003b10 0x7fc35c005fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:35.944 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:35.924 7fc37486e700 1 --2- 172.21.2.102:0/2816879847 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc35c04a780 0x7fc35c04cc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:35.944 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:35.924 7fc37486e700 1 --2- 172.21.2.102:0/2816879847 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc35c04d500 0x7fc35c04fa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:35.944 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:35.924 7fc37486e700 1 -- 172.21.2.102:0/2816879847 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc35c0502d0 con 0x7fc35c04a780 2021-02-10T11:39:35.944 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:35.924 7fc37486e700 1 -- 172.21.2.102:0/2816879847 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc35c0504f0 con 0x7fc35c04d500 2021-02-10T11:39:35.944 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:35.924 7fc37486e700 1 -- 172.21.2.102:0/2816879847 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc35c050710 con 0x7fc35c003b10 2021-02-10T11:39:35.944 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:35.924 7fc37506f700 1 --2- 172.21.2.102:0/2816879847 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc35c04a780 0x7fc35c04cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:35.944 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:35.924 7fc375870700 1 --2- 172.21.2.102:0/2816879847 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc35c003b10 0x7fc35c005fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:35.944 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:35.924 7fc376071700 1 --2- 172.21.2.102:0/2816879847 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc35c04d500 0x7fc35c04fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:35.944 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:35.924 7fc37506f700 1 -- 172.21.2.102:0/2816879847 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc35c003b10 msgr2=0x7fc35c005fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:35.944 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:35.924 7fc37506f700 1 --2- 172.21.2.102:0/2816879847 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc35c003b10 0x7fc35c005fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:35.945 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:35.924 7fc37506f700 1 -- 172.21.2.102:0/2816879847 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc35c04d500 msgr2=0x7fc35c04fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:35.945 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:35.924 7fc37506f700 1 --2- 172.21.2.102:0/2816879847 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc35c04d500 0x7fc35c04fa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:35.945 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:35.924 7fc37506f700 1 -- 172.21.2.102:0/2816879847 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc35c0509b0 con 0x7fc35c04a780 2021-02-10T11:39:35.945 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:35.924 7fc37506f700 1 --2- 172.21.2.102:0/2816879847 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc35c04a780 0x7fc35c04cc60 secure :-1 s=READY pgs=3254 cs=0 l=1 rev1=1 rx=0x7fc3600017d0 tx=0x7fc360007d30).ready entity=mon.0 client_cookie=96567acb35d80714 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:35.945 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:35.924 7fc37486e700 1 -- 172.21.2.102:0/2816879847 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc3600013a0 con 0x7fc35c04a780 2021-02-10T11:39:35.945 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:35.924 7fc37486e700 1 -- 172.21.2.102:0/2816879847 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc36000a310 con 0x7fc35c04a780 2021-02-10T11:39:35.945 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:35.924 7fc37486e700 1 -- 172.21.2.102:0/2816879847 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc3600009b0 con 0x7fc35c04a780 2021-02-10T11:39:35.945 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:35.924 7fc377a54700 1 -- 172.21.2.102:0/2816879847 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc35c04a780 msgr2=0x7fc35c04cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:35.945 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:35.924 7fc377a54700 1 --2- 172.21.2.102:0/2816879847 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc35c04a780 0x7fc35c04cc60 secure :-1 s=READY pgs=3254 cs=0 l=1 rev1=1 rx=0x7fc3600017d0 tx=0x7fc360007d30).stop 2021-02-10T11:39:35.945 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:35.925 7fc376071700 1 -- 172.21.2.102:0/2816879847 reap_dead start 2021-02-10T11:39:35.945 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:35.925 7fc377a54700 1 -- 172.21.2.102:0/2816879847 shutdown_connections 2021-02-10T11:39:35.945 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:35.925 7fc377a54700 1 -- 172.21.2.102:0/2816879847 >> 172.21.2.102:0/2816879847 conn(0x7fc3700ce430 msgr2=0x7fc3700d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:35.945 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:35.925 7fc377a54700 1 -- 172.21.2.102:0/2816879847 shutdown_connections 2021-02-10T11:39:35.945 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:35.925 7fc377a54700 1 -- 172.21.2.102:0/2816879847 wait complete. 2021-02-10T11:39:35.946 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:35.940 7fc377a54700 1 Processor -- start 2021-02-10T11:39:35.946 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:35.940 7fc377a54700 1 -- start start 2021-02-10T11:39:35.947 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:35.940 7fc377a54700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc370085300 0x7fc3700bf1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:35.947 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:35.940 7fc377a54700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc3700bf7d0 0x7fc370085fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:35.947 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:35.940 7fc377a54700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc3700bfbd0 0x7fc3700865d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:35.947 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:35.940 7fc377a54700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc370084d80 con 0x7fc3700bfbd0 2021-02-10T11:39:35.948 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:35.940 7fc377a54700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc370085700 con 0x7fc3700bf7d0 2021-02-10T11:39:35.948 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:35.940 7fc377a54700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc370086bf0 con 0x7fc370085300 2021-02-10T11:39:35.948 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:35.941 7fc376071700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc3700bfbd0 0x7fc3700865d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:35.948 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:35.941 7fc376071700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc3700bfbd0 0x7fc3700865d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:40532/0 (socket says 172.21.2.102:40532) 2021-02-10T11:39:35.948 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:35.941 7fc376071700 1 -- 172.21.2.102:0/236899524 learned_addr learned my addr 172.21.2.102:0/236899524 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:39:35.948 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:35.941 7fc375870700 1 --2- 172.21.2.102:0/236899524 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc370085300 0x7fc3700bf1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:35.948 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:35.941 7fc37506f700 1 --2- 172.21.2.102:0/236899524 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc3700bf7d0 0x7fc370085fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:35.948 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:35.941 7fc376071700 1 -- 172.21.2.102:0/236899524 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc370085300 msgr2=0x7fc3700bf1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:35.948 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:35.941 7fc376071700 1 --2- 172.21.2.102:0/236899524 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc370085300 0x7fc3700bf1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:35.948 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:35.941 7fc376071700 1 -- 172.21.2.102:0/236899524 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc3700bf7d0 msgr2=0x7fc370085fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:35.948 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:35.941 7fc376071700 1 --2- 172.21.2.102:0/236899524 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc3700bf7d0 0x7fc370085fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:35.948 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:35.941 7fc376071700 1 -- 172.21.2.102:0/236899524 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc370086e10 con 0x7fc3700bfbd0 2021-02-10T11:39:35.948 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:35.941 7fc376071700 1 --2- 172.21.2.102:0/236899524 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc3700bfbd0 0x7fc3700865d0 secure :-1 s=READY pgs=3255 cs=0 l=1 rev1=1 rx=0x7fc3680099e0 tx=0x7fc36800bd20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:35.949 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:35.941 7fc3667fc700 1 -- 172.21.2.102:0/236899524 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc368001400 con 0x7fc3700bfbd0 2021-02-10T11:39:35.949 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:35.941 7fc3667fc700 1 -- 172.21.2.102:0/236899524 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc36800be60 con 0x7fc3700bfbd0 2021-02-10T11:39:35.949 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:35.941 7fc377a54700 1 -- 172.21.2.102:0/236899524 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc370086e10 con 0x7fc3700bfbd0 2021-02-10T11:39:35.949 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:35.941 7fc3667fc700 1 -- 172.21.2.102:0/236899524 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc368001700 con 0x7fc3700bfbd0 2021-02-10T11:39:35.949 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:35.941 7fc377a54700 1 -- 172.21.2.102:0/236899524 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc370086e10 con 0x7fc3700bfbd0 2021-02-10T11:39:35.949 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:35.942 7fc3667fc700 1 -- 172.21.2.102:0/236899524 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fc368006a20 con 0x7fc3700bfbd0 2021-02-10T11:39:35.949 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:35.942 7fc3667fc700 1 --2- 172.21.2.102:0/236899524 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc358018730 0x7fc35801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:35.949 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:35.942 7fc3667fc700 1 -- 172.21.2.102:0/236899524 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(189..189 src has 1..189) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fc3680329b0 con 0x7fc3700bfbd0 2021-02-10T11:39:35.949 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:35.942 7fc375870700 1 --2- 172.21.2.102:0/236899524 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc358018730 0x7fc35801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:35.949 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:35.942 7fc375870700 1 --2- 172.21.2.102:0/236899524 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc358018730 0x7fc35801ac10 crc :-1 s=READY pgs=1422 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:35.974 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:35.972 7fc377a54700 1 -- 172.21.2.102:0/236899524 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3700829a0 con 0x7fc3700bfbd0 2021-02-10T11:39:35.977 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:35.975 7fc3667fc700 1 -- 172.21.2.102:0/236899524 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fc368001bd0 con 0x7fc3700bfbd0 2021-02-10T11:39:36.107 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:36.105 7fc377a54700 1 -- 172.21.2.102:0/236899524 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.3", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7fc370109a70 con 0x7fc3700bfbd0 2021-02-10T11:39:36.111 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:36.109 7fc3667fc700 1 -- 172.21.2.102:0/236899524 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.3", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.3 v111) v1 ==== 184+0+0 (secure 0 0 0) 0x7fc368006a20 con 0x7fc3700bfbd0 2021-02-10T11:39:36.114 INFO:teuthology.orchestra.run.gibba002.stderr:updated caps for client.3 2021-02-10T11:39:36.116 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:36.114 7fc377a54700 1 -- 172.21.2.102:0/236899524 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc358018730 msgr2=0x7fc35801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:36.116 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:36.114 7fc377a54700 1 --2- 172.21.2.102:0/236899524 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc358018730 0x7fc35801ac10 crc :-1 s=READY pgs=1422 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:36.116 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:36.114 7fc377a54700 1 -- 172.21.2.102:0/236899524 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc3700bfbd0 msgr2=0x7fc3700865d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:36.116 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:36.114 7fc377a54700 1 --2- 172.21.2.102:0/236899524 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc3700bfbd0 0x7fc3700865d0 secure :-1 s=READY pgs=3255 cs=0 l=1 rev1=1 rx=0x7fc3680099e0 tx=0x7fc36800bd20).stop 2021-02-10T11:39:36.116 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:36.114 7fc377a54700 1 -- 172.21.2.102:0/236899524 shutdown_connections 2021-02-10T11:39:36.116 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:36.114 7fc377a54700 1 --2- 172.21.2.102:0/236899524 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc3700bfbd0 0x7fc3700865d0 unknown :-1 s=CLOSED pgs=3255 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:36.116 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:36.114 7fc377a54700 1 --2- 172.21.2.102:0/236899524 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc358018730 0x7fc35801ac10 unknown :-1 s=CLOSED pgs=1422 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:36.116 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:36.114 7fc377a54700 1 --2- 172.21.2.102:0/236899524 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc3700bf7d0 0x7fc370085fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:36.117 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:36.114 7fc377a54700 1 --2- 172.21.2.102:0/236899524 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc370085300 0x7fc3700bf1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:36.117 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:36.114 7fc377a54700 1 -- 172.21.2.102:0/236899524 >> 172.21.2.102:0/236899524 conn(0x7fc3700bd7a0 msgr2=0x7fc3700b5900 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:36.117 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:36.114 7fc377a54700 1 -- 172.21.2.102:0/236899524 shutdown_connections 2021-02-10T11:39:36.117 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:36.114 7fc377a54700 1 -- 172.21.2.102:0/236899524 wait complete. 2021-02-10T11:39:36.124 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:39:36.554 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:36.536 7fd95f666700 1 Processor -- start 2021-02-10T11:39:36.555 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:36.536 7fd95f666700 1 -- start start 2021-02-10T11:39:36.555 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:36.537 7fd95f666700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fd9581b0470 0x7fd9581b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:36.555 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:36.537 7fd95f666700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fd9580ef610 0x7fd9580efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:36.555 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:36.537 7fd95f666700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd9580f0030 0x7fd9580f0430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:36.555 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:36.537 7fd95f666700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9580e5ea0 con 0x7fd9580ef610 2021-02-10T11:39:36.555 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:36.537 7fd95f666700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9580f0b50 con 0x7fd9581b0470 2021-02-10T11:39:36.555 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:36.537 7fd95f666700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9580e52e0 con 0x7fd9580f0030 2021-02-10T11:39:36.555 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:36.537 7fd95cc81700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fd9580ef610 0x7fd9580efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:36.555 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:36.537 7fd95cc81700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fd9580ef610 0x7fd9580efa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:40536/0 (socket says 172.21.2.102:40536) 2021-02-10T11:39:36.555 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:36.537 7fd95cc81700 1 -- 172.21.2.102:0/6812930 learned_addr learned my addr 172.21.2.102:0/6812930 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:39:36.555 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:36.537 7fd95cc81700 1 -- 172.21.2.102:0/6812930 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd9580f0030 msgr2=0x7fd9580f0430 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:39:36.555 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:36.537 7fd95cc81700 1 --2- 172.21.2.102:0/6812930 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd9580f0030 0x7fd9580f0430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:36.556 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:36.537 7fd95cc81700 1 -- 172.21.2.102:0/6812930 >> v2:172.21.2.113:3300/0 conn(0x7fd9581b0470 msgr2=0x7fd9581b0870 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:39:36.556 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:36.537 7fd95cc81700 1 --2- 172.21.2.102:0/6812930 >> v2:172.21.2.113:3300/0 conn(0x7fd9581b0470 0x7fd9581b0870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:36.556 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:36.537 7fd95cc81700 1 -- 172.21.2.102:0/6812930 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9580f10d0 con 0x7fd9580ef610 2021-02-10T11:39:36.556 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:36.537 7fd95cc81700 1 --2- 172.21.2.102:0/6812930 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd9580ef610 0x7fd9580efa10 secure :-1 s=READY pgs=3256 cs=0 l=1 rev1=1 rx=0x7fd94c006700 tx=0x7fd94c009b80).ready entity=mon.0 client_cookie=6fdd60229e312c76 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:36.556 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:36.537 7fd94bfff700 1 -- 172.21.2.102:0/6812930 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd94c00c0c0 con 0x7fd9580ef610 2021-02-10T11:39:36.556 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:36.538 7fd94bfff700 1 -- 172.21.2.102:0/6812930 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd94c00c440 con 0x7fd9580ef610 2021-02-10T11:39:36.556 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:36.538 7fd94bfff700 1 -- 172.21.2.102:0/6812930 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd94c00c920 con 0x7fd9580ef610 2021-02-10T11:39:36.556 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:36.538 7fd95f666700 1 -- 172.21.2.102:0/6812930 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd9580ef610 msgr2=0x7fd9580efa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:36.556 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:36.538 7fd95f666700 1 --2- 172.21.2.102:0/6812930 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd9580ef610 0x7fd9580efa10 secure :-1 s=READY pgs=3256 cs=0 l=1 rev1=1 rx=0x7fd94c006700 tx=0x7fd94c009b80).stop 2021-02-10T11:39:36.556 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:36.538 7fd95f666700 1 -- 172.21.2.102:0/6812930 shutdown_connections 2021-02-10T11:39:36.556 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:36.538 7fd95f666700 1 --2- 172.21.2.102:0/6812930 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd9580f0030 0x7fd9580f0430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:36.556 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:36.538 7fd95f666700 1 --2- 172.21.2.102:0/6812930 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd9580ef610 0x7fd9580efa10 unknown :-1 s=CLOSED pgs=3256 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:36.556 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:36.538 7fd95f666700 1 --2- 172.21.2.102:0/6812930 >> v2:172.21.2.113:3300/0 conn(0x7fd9581b0470 0x7fd9581b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:36.556 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:36.538 7fd95f666700 1 -- 172.21.2.102:0/6812930 >> 172.21.2.102:0/6812930 conn(0x7fd9580ce3e0 msgr2=0x7fd9580e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:36.557 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:36.538 7fd95f666700 1 -- 172.21.2.102:0/6812930 shutdown_connections 2021-02-10T11:39:36.557 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:36.538 7fd95f666700 1 -- 172.21.2.102:0/6812930 wait complete. 2021-02-10T11:39:36.558 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:36.553 7fd95f666700 1 Processor -- start 2021-02-10T11:39:36.558 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:36.553 7fd95f666700 1 -- start start 2021-02-10T11:39:36.558 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:36.553 7fd95f666700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd9580b1f20 0x7fd9580b2320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:36.558 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:36.553 7fd95f666700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd9581ad760 0x7fd9581adb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:36.558 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:36.553 7fd95f666700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd9581ae180 0x7fd9581ae580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:36.558 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:36.553 7fd95f666700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9581ad1e0 con 0x7fd9581ad760 2021-02-10T11:39:36.558 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:36.553 7fd95f666700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd95819cb70 con 0x7fd9580b1f20 2021-02-10T11:39:36.559 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:36.553 7fd95f666700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd95819cea0 con 0x7fd9581ae180 2021-02-10T11:39:36.559 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:36.553 7fd95cc81700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd9581ad760 0x7fd9581adb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:36.559 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:36.553 7fd95cc81700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd9581ad760 0x7fd9581adb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:40542/0 (socket says 172.21.2.102:40542) 2021-02-10T11:39:36.559 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:36.553 7fd95cc81700 1 -- 172.21.2.102:0/1313592109 learned_addr learned my addr 172.21.2.102:0/1313592109 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:39:36.559 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:36.554 7fd95cc81700 1 -- 172.21.2.102:0/1313592109 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd9581ae180 msgr2=0x7fd9581ae580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:39:36.559 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:36.554 7fd95cc81700 1 --2- 172.21.2.102:0/1313592109 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd9581ae180 0x7fd9581ae580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:36.559 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:36.554 7fd95cc81700 1 -- 172.21.2.102:0/1313592109 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd9580b1f20 msgr2=0x7fd9580b2320 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:39:36.559 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:36.554 7fd95cc81700 1 --2- 172.21.2.102:0/1313592109 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd9580b1f20 0x7fd9580b2320 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:36.559 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:36.554 7fd95cc81700 1 -- 172.21.2.102:0/1313592109 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd95819d0c0 con 0x7fd9581ad760 2021-02-10T11:39:36.559 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:36.554 7fd95cc81700 1 --2- 172.21.2.102:0/1313592109 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd9581ad760 0x7fd9581adb60 secure :-1 s=READY pgs=3257 cs=0 l=1 rev1=1 rx=0x7fd94c00b5e0 tx=0x7fd94c00b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:36.559 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:36.554 7fd949ffb700 1 -- 172.21.2.102:0/1313592109 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd94c00bd70 con 0x7fd9581ad760 2021-02-10T11:39:36.559 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:36.554 7fd949ffb700 1 -- 172.21.2.102:0/1313592109 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd94c002030 con 0x7fd9581ad760 2021-02-10T11:39:36.559 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:36.554 7fd949ffb700 1 -- 172.21.2.102:0/1313592109 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd94c002550 con 0x7fd9581ad760 2021-02-10T11:39:36.559 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:36.554 7fd95f666700 1 -- 172.21.2.102:0/1313592109 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd95819d0c0 con 0x7fd9581ad760 2021-02-10T11:39:36.560 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:36.554 7fd95f666700 1 -- 172.21.2.102:0/1313592109 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd95819d650 con 0x7fd9581ad760 2021-02-10T11:39:36.560 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:36.555 7fd949ffb700 1 -- 172.21.2.102:0/1313592109 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd94c000c90 con 0x7fd9581ad760 2021-02-10T11:39:36.560 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:36.555 7fd949ffb700 1 --2- 172.21.2.102:0/1313592109 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd9380186c0 0x7fd93801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:36.560 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:36.555 7fd949ffb700 1 -- 172.21.2.102:0/1313592109 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(189..189 src has 1..189) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fd94c032580 con 0x7fd9581ad760 2021-02-10T11:39:36.560 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:36.555 7fd95d482700 1 --2- 172.21.2.102:0/1313592109 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd9380186c0 0x7fd93801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:36.560 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:36.555 7fd95d482700 1 --2- 172.21.2.102:0/1313592109 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd9380186c0 0x7fd93801abc0 crc :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:36.589 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:36.587 7fd95f666700 1 -- 172.21.2.102:0/1313592109 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9580f0b50 con 0x7fd9581ad760 2021-02-10T11:39:36.592 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:36.589 7fd949ffb700 1 -- 172.21.2.102:0/1313592109 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd94c020fa0 con 0x7fd9581ad760 2021-02-10T11:39:36.720 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:36.718 7fd95f666700 1 -- 172.21.2.102:0/1313592109 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd95819caf0 con 0x7fd9581ad760 2021-02-10T11:39:36.721 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:36.719 7fd949ffb700 1 -- 172.21.2.102:0/1313592109 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 165 v165) v1 ==== 95+0+2104 (secure 0 0 0) 0x7fd94c020fa0 con 0x7fd9581ad760 2021-02-10T11:39:36.721 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:39:36.721 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":165,"default_fscid":20,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8998,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.113:6833/479366648","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6832","nonce":479366648},{"type":"v1","addr":"172.21.2.113:6833","nonce":479366648}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":162}],"filesystems":[{"mdsmap":{"epoch":165,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:39:30.375466","modified":"2021-02-10 11:39:32.379360","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7481},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7481":{"gid":7481,"name":"a","rank":0,"incarnation":163,"state":"up:active","state_seq":3,"addr":"172.21.2.102:6835/4290051997","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6834","nonce":4290051997},{"type":"v1","addr":"172.21.2.102:6835","nonce":4290051997}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[41],"metadata_pool":40,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":20}]} 2021-02-10T11:39:36.722 INFO:teuthology.orchestra.run.gibba002.stderr:dumped fsmap epoch 165 2021-02-10T11:39:36.722 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:36.720 7fd95f666700 1 -- 172.21.2.102:0/1313592109 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd9380186c0 msgr2=0x7fd93801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:36.722 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:36.720 7fd95f666700 1 --2- 172.21.2.102:0/1313592109 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd9380186c0 0x7fd93801abc0 crc :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:36.723 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:36.720 7fd95f666700 1 -- 172.21.2.102:0/1313592109 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd9581ad760 msgr2=0x7fd9581adb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:36.723 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:36.720 7fd95f666700 1 --2- 172.21.2.102:0/1313592109 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd9581ad760 0x7fd9581adb60 secure :-1 s=READY pgs=3257 cs=0 l=1 rev1=1 rx=0x7fd94c00b5e0 tx=0x7fd94c00b980).stop 2021-02-10T11:39:36.723 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:36.721 7fd95f666700 1 -- 172.21.2.102:0/1313592109 shutdown_connections 2021-02-10T11:39:36.723 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:36.721 7fd95f666700 1 --2- 172.21.2.102:0/1313592109 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd9581ae180 0x7fd9581ae580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:36.724 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:36.721 7fd95f666700 1 --2- 172.21.2.102:0/1313592109 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd9581ad760 0x7fd9581adb60 unknown :-1 s=CLOSED pgs=3257 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:36.724 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:36.721 7fd95f666700 1 --2- 172.21.2.102:0/1313592109 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd9380186c0 0x7fd93801abc0 unknown :-1 s=CLOSED pgs=1423 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:36.724 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:36.721 7fd95f666700 1 --2- 172.21.2.102:0/1313592109 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd9580b1f20 0x7fd9580b2320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:36.724 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:36.721 7fd95f666700 1 -- 172.21.2.102:0/1313592109 >> 172.21.2.102:0/1313592109 conn(0x7fd958109a10 msgr2=0x7fd95809aa50 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:36.724 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:36.721 7fd95f666700 1 -- 172.21.2.102:0/1313592109 shutdown_connections 2021-02-10T11:39:36.724 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:36.721 7fd95f666700 1 -- 172.21.2.102:0/1313592109 wait complete. 2021-02-10T11:39:36.933 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 165, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:39:30.375466', 'modified': '2021-02-10 11:39:32.379360', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7481}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7481': {'gid': 7481, 'name': 'a', 'rank': 0, 'incarnation': 163, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.102:6835/4290051997', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.102:6834', 'nonce': 4290051997}, {'type': 'v1', 'addr': '172.21.2.102:6835', 'nonce': 4290051997}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [41], 'metadata_pool': 40, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:39:36.933 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:39:36.933 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok status 2021-02-10T11:39:37.335 INFO:teuthology.orchestra.run.gibba002.stdout:{ 2021-02-10T11:39:37.619 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_fsid": "68db30ea-dbce-4aef-a6df-f01c6fea0513", 2021-02-10T11:39:37.619 INFO:teuthology.orchestra.run.gibba002.stdout: "whoami": 0, 2021-02-10T11:39:37.619 INFO:teuthology.orchestra.run.gibba002.stdout: "id": 7481, 2021-02-10T11:39:37.619 INFO:teuthology.orchestra.run.gibba002.stdout: "want_state": "up:active", 2021-02-10T11:39:37.619 INFO:teuthology.orchestra.run.gibba002.stdout: "state": "up:active", 2021-02-10T11:39:37.619 INFO:teuthology.orchestra.run.gibba002.stdout: "rank_uptime": 6.9553298420000003, 2021-02-10T11:39:37.619 INFO:teuthology.orchestra.run.gibba002.stdout: "mdsmap_epoch": 165, 2021-02-10T11:39:37.620 INFO:teuthology.orchestra.run.gibba002.stdout: "osdmap_epoch": 189, 2021-02-10T11:39:37.620 INFO:teuthology.orchestra.run.gibba002.stdout: "osdmap_epoch_barrier": 189, 2021-02-10T11:39:37.620 INFO:teuthology.orchestra.run.gibba002.stdout: "uptime": 12.604973490000001 2021-02-10T11:39:37.620 INFO:teuthology.orchestra.run.gibba002.stdout:} 2021-02-10T11:39:37.621 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "68db30ea-dbce-4aef-a6df-f01c6fea0513", "whoami": 0, "id": 7481, "want_state": "up:active", "state": "up:active", "rank_uptime": 6.9553298420000003, "mdsmap_epoch": 165, "osdmap_epoch": 189, "osdmap_epoch_barrier": 189, "uptime": 12.604973490000001 } 2021-02-10T11:39:37.621 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:39:38.053 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:38.032 7fa420c3f700 1 Processor -- start 2021-02-10T11:39:38.053 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:38.032 7fa420c3f700 1 -- start start 2021-02-10T11:39:38.053 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:38.033 7fa420c3f700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fa41c0d0340 0x7fa41c0ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:38.053 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:38.033 7fa420c3f700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa41c0d1c70 0x7fa41c0efc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:38.054 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:38.033 7fa420c3f700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa41c0f0b10 con 0x7fa41c0d1870 2021-02-10T11:39:38.054 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:38.033 7fa420c3f700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa41c0e5ed0 con 0x7fa41c0d0340 2021-02-10T11:39:38.054 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:38.033 7fa420c3f700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa41c1b0be0 con 0x7fa41c0d1c70 2021-02-10T11:39:38.054 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:38.033 7fa41a85e700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fa41c0d1870 0x7fa41c1b03f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:32772/0 (socket says 172.21.2.102:32772) 2021-02-10T11:39:38.054 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:38.033 7fa41a85e700 1 -- 172.21.2.102:0/321313044 learned_addr learned my addr 172.21.2.102:0/321313044 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:39:38.054 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:38.033 7fa41985c700 1 -- 172.21.2.102:0/321313044 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1221365632 0 0) 0x7fa404000ce0 con 0x7fa41c0d1870 2021-02-10T11:39:38.054 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:38.033 7fa41985c700 1 -- 172.21.2.102:0/321313044 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa408002fe0 con 0x7fa41c0d1870 2021-02-10T11:39:38.054 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:38.034 7fa41a05d700 1 --2- 172.21.2.102:0/321313044 >> v2:172.21.2.113:3300/0 conn(0x7fa41c0d0340 0x7fa41c0ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:38.054 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:38.034 7fa41b05f700 1 --2- 172.21.2.102:0/321313044 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa41c0d1c70 0x7fa41c0efc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:38.054 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:38.034 7fa41985c700 1 -- 172.21.2.102:0/321313044 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3334537198 0 0) 0x7fa404001180 con 0x7fa41c0d1870 2021-02-10T11:39:38.054 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:38.034 7fa41985c700 1 -- 172.21.2.102:0/321313044 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa41c0d1c70 msgr2=0x7fa41c0efc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:38.054 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:38.034 7fa41985c700 1 --2- 172.21.2.102:0/321313044 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa41c0d1c70 0x7fa41c0efc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:38.054 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:38.034 7fa41985c700 1 -- 172.21.2.102:0/321313044 >> v2:172.21.2.113:3300/0 conn(0x7fa41c0d0340 msgr2=0x7fa41c0ef640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:38.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:38.034 7fa41985c700 1 --2- 172.21.2.102:0/321313044 >> v2:172.21.2.113:3300/0 conn(0x7fa41c0d0340 0x7fa41c0ef640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:38.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:38.034 7fa41985c700 1 -- 172.21.2.102:0/321313044 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa41c0f20a0 con 0x7fa41c0d1870 2021-02-10T11:39:38.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:38.034 7fa41985c700 1 -- 172.21.2.102:0/321313044 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7fa404001680 con 0x7fa41c0d1870 2021-02-10T11:39:38.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:38.034 7fa41985c700 1 -- 172.21.2.102:0/321313044 >> v1:172.21.2.102:6789/0 conn(0x7fa41c0d1870 legacy=0x7fa41c1b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:38.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:38.034 7fa41985c700 1 --2- 172.21.2.102:0/321313044 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa408003a60 0x7fa408005eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:38.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:38.034 7fa41985c700 1 --2- 172.21.2.102:0/321313044 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa408006750 0x7fa408008c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:38.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:38.034 7fa41985c700 1 --2- 172.21.2.102:0/321313044 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa4080094f0 0x7fa40800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:38.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:38.034 7fa41985c700 1 -- 172.21.2.102:0/321313044 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa40800c2c0 con 0x7fa4080094f0 2021-02-10T11:39:38.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:38.034 7fa41985c700 1 -- 172.21.2.102:0/321313044 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4080504b0 con 0x7fa408006750 2021-02-10T11:39:38.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:38.034 7fa41985c700 1 -- 172.21.2.102:0/321313044 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4080506d0 con 0x7fa408003a60 2021-02-10T11:39:38.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:38.034 7fa41b05f700 1 --2- 172.21.2.102:0/321313044 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa4080094f0 0x7fa40800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:38.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:38.034 7fa41a05d700 1 --2- 172.21.2.102:0/321313044 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa408006750 0x7fa408008c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:38.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:38.034 7fa41a85e700 1 --2- 172.21.2.102:0/321313044 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa408003a60 0x7fa408005eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:38.056 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:38.034 7fa41a05d700 1 -- 172.21.2.102:0/321313044 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa408003a60 msgr2=0x7fa408005eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:38.056 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:38.034 7fa41a05d700 1 --2- 172.21.2.102:0/321313044 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa408003a60 0x7fa408005eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:38.056 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:38.034 7fa41a05d700 1 -- 172.21.2.102:0/321313044 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa4080094f0 msgr2=0x7fa40800ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:38.056 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:38.034 7fa41a05d700 1 --2- 172.21.2.102:0/321313044 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa4080094f0 0x7fa40800ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:38.056 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:38.034 7fa41a05d700 1 -- 172.21.2.102:0/321313044 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa408050970 con 0x7fa408006750 2021-02-10T11:39:38.056 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:38.034 7fa41b05f700 1 --2- 172.21.2.102:0/321313044 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa4080094f0 0x7fa40800ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:39:38.056 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:38.035 7fa41a05d700 1 --2- 172.21.2.102:0/321313044 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa408006750 0x7fa408008c70 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 rx=0x7fa40c006360 tx=0x7fa40c009c30).ready entity=mon.1 client_cookie=3d1f4ced2f6a8726 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:38.056 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:38.035 7fa41985c700 1 -- 172.21.2.102:0/321313044 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa40c0009b0 con 0x7fa408006750 2021-02-10T11:39:38.056 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:38.035 7fa41985c700 1 -- 172.21.2.102:0/321313044 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa40c000d20 con 0x7fa408006750 2021-02-10T11:39:38.056 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:38.035 7fa41985c700 1 -- 172.21.2.102:0/321313044 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa40c002220 con 0x7fa408006750 2021-02-10T11:39:38.056 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:38.035 7fa420c3f700 1 -- 172.21.2.102:0/321313044 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa408006750 msgr2=0x7fa408008c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:38.056 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:38.035 7fa420c3f700 1 --2- 172.21.2.102:0/321313044 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa408006750 0x7fa408008c70 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 rx=0x7fa40c006360 tx=0x7fa40c009c30).stop 2021-02-10T11:39:38.056 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:38.035 7fa41b05f700 1 -- 172.21.2.102:0/321313044 reap_dead start 2021-02-10T11:39:38.057 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:38.036 7fa420c3f700 1 -- 172.21.2.102:0/321313044 shutdown_connections 2021-02-10T11:39:38.057 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:38.036 7fa420c3f700 1 -- 172.21.2.102:0/321313044 >> 172.21.2.102:0/321313044 conn(0x7fa41c0ce450 msgr2=0x7fa41c0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:38.057 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:38.036 7fa420c3f700 1 -- 172.21.2.102:0/321313044 shutdown_connections 2021-02-10T11:39:38.057 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:38.036 7fa420c3f700 1 -- 172.21.2.102:0/321313044 wait complete. 2021-02-10T11:39:38.057 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:38.051 7fa420c3f700 1 Processor -- start 2021-02-10T11:39:38.057 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:38.051 7fa420c3f700 1 -- start start 2021-02-10T11:39:38.059 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:38.051 7fa420c3f700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa41c0f0280 0x7fa41c0a2310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:38.059 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:38.051 7fa420c3f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa41c0f0680 0x7fa41c0a2930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:38.059 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:38.051 7fa420c3f700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa41c0ec180 0x7fa41c0ec580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:38.059 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:38.051 7fa420c3f700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa41c0efd00 con 0x7fa41c0f0680 2021-02-10T11:39:38.059 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:38.051 7fa420c3f700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa41c0ecba0 con 0x7fa41c0f0280 2021-02-10T11:39:38.059 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:38.051 7fa420c3f700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa41c0ecd80 con 0x7fa41c0ec180 2021-02-10T11:39:38.059 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:38.051 7fa41a05d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa41c0f0680 0x7fa41c0a2930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:38.059 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:38.051 7fa41a05d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa41c0f0680 0x7fa41c0a2930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:40558/0 (socket says 172.21.2.102:40558) 2021-02-10T11:39:38.059 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:38.051 7fa41a05d700 1 -- 172.21.2.102:0/3443393309 learned_addr learned my addr 172.21.2.102:0/3443393309 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:39:38.060 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:38.051 7fa41b05f700 1 --2- 172.21.2.102:0/3443393309 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa41c0ec180 0x7fa41c0ec580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:38.060 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:38.052 7fa41a85e700 1 --2- 172.21.2.102:0/3443393309 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa41c0f0280 0x7fa41c0a2310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:38.060 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:38.052 7fa41a05d700 1 -- 172.21.2.102:0/3443393309 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa41c0ec180 msgr2=0x7fa41c0ec580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:38.060 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:38.052 7fa41a05d700 1 --2- 172.21.2.102:0/3443393309 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa41c0ec180 0x7fa41c0ec580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:38.060 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:38.052 7fa41a05d700 1 -- 172.21.2.102:0/3443393309 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa41c0f0280 msgr2=0x7fa41c0a2310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:38.060 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:38.052 7fa41a05d700 1 --2- 172.21.2.102:0/3443393309 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa41c0f0280 0x7fa41c0a2310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:38.060 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:38.052 7fa41a05d700 1 -- 172.21.2.102:0/3443393309 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa41c0ecfa0 con 0x7fa41c0f0680 2021-02-10T11:39:38.060 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:38.052 7fa41a05d700 1 --2- 172.21.2.102:0/3443393309 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa41c0f0680 0x7fa41c0a2930 secure :-1 s=READY pgs=3259 cs=0 l=1 rev1=1 rx=0x7fa40c006980 tx=0x7fa40c00b9b0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:38.060 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:38.052 7fa4137fe700 1 -- 172.21.2.102:0/3443393309 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa40c00bda0 con 0x7fa41c0f0680 2021-02-10T11:39:38.060 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:38.052 7fa4137fe700 1 -- 172.21.2.102:0/3443393309 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa40c000c90 con 0x7fa41c0f0680 2021-02-10T11:39:38.060 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:38.052 7fa420c3f700 1 -- 172.21.2.102:0/3443393309 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa41c0ecfa0 con 0x7fa41c0f0680 2021-02-10T11:39:38.060 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:38.052 7fa4137fe700 1 -- 172.21.2.102:0/3443393309 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa40c0022b0 con 0x7fa41c0f0680 2021-02-10T11:39:38.060 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:38.052 7fa420c3f700 1 -- 172.21.2.102:0/3443393309 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa41c0ed530 con 0x7fa41c0f0680 2021-02-10T11:39:38.060 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:38.053 7fa4137fe700 1 -- 172.21.2.102:0/3443393309 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa40c009fd0 con 0x7fa41c0f0680 2021-02-10T11:39:38.061 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:38.053 7fa4137fe700 1 --2- 172.21.2.102:0/3443393309 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa400018730 0x7fa40001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:38.061 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:38.053 7fa4137fe700 1 -- 172.21.2.102:0/3443393309 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(189..189 src has 1..189) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fa40c032670 con 0x7fa41c0f0680 2021-02-10T11:39:38.061 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:38.053 7fa41a85e700 1 --2- 172.21.2.102:0/3443393309 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa400018730 0x7fa40001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:38.061 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:38.053 7fa41a85e700 1 --2- 172.21.2.102:0/3443393309 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa400018730 0x7fa40001ac10 crc :-1 s=READY pgs=1424 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:38.084 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:38.082 7fa420c3f700 1 -- 172.21.2.102:0/3443393309 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa41c0ebc60 con 0x7fa41c0f0680 2021-02-10T11:39:38.087 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:38.085 7fa4137fe700 1 -- 172.21.2.102:0/3443393309 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fa40c006cb0 con 0x7fa41c0f0680 2021-02-10T11:39:38.211 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:38.209 7fa420c3f700 1 -- 172.21.2.102:0/3443393309 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa41c0ebc60 con 0x7fa41c0f0680 2021-02-10T11:39:38.212 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:38.209 7fa4137fe700 1 -- 172.21.2.102:0/3443393309 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v189) v1 ==== 74+0+14007 (secure 0 0 0) 0x7fa40c009fd0 con 0x7fa41c0f0680 2021-02-10T11:39:38.213 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:39:38.213 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":189,"fsid":"68db30ea-dbce-4aef-a6df-f01c6fea0513","created":"2021-02-10 11:03:06.331557","modified":"2021-02-10 11:39:31.377604","last_up_change":"2021-02-10 11:03:32.690562","last_in_change":"2021-02-10 11:03:32.690562","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":41,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:03:37.322519","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":40,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:39:28.340428","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"188","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":41,"pool_name":"cephfs_data","create_time":"2021-02-10 11:39:29.367624","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"189","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"be691b4d-a039-408a-af80-9d53f02b6f1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":185,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6818","nonce":36088},{"type":"v1","addr":"172.21.2.102:6820","nonce":36088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6822","nonce":36088},{"type":"v1","addr":"172.21.2.102:6824","nonce":36088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6830","nonce":36088},{"type":"v1","addr":"172.21.2.102:6832","nonce":36088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6826","nonce":36088},{"type":"v1","addr":"172.21.2.102:6828","nonce":36088}]},"public_addr":"172.21.2.102:6820/36088","cluster_addr":"172.21.2.102:6824/36088","heartbeat_back_addr":"172.21.2.102:6832/36088","heartbeat_front_addr":"172.21.2.102:6828/36088","state":["exists","up"]},{"osd":1,"uuid":"64e3ccc8-9c69-4d7c-865e-af8e83822635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":185,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6819","nonce":36087},{"type":"v1","addr":"172.21.2.102:6821","nonce":36087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6823","nonce":36087},{"type":"v1","addr":"172.21.2.102:6825","nonce":36087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6831","nonce":36087},{"type":"v1","addr":"172.21.2.102:6833","nonce":36087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6827","nonce":36087},{"type":"v1","addr":"172.21.2.102:6829","nonce":36087}]},"public_addr":"172.21.2.102:6821/36087","cluster_addr":"172.21.2.102:6825/36087","heartbeat_back_addr":"172.21.2.102:6833/36087","heartbeat_front_addr":"172.21.2.102:6829/36087","state":["exists","up"]},{"osd":2,"uuid":"153700af-747f-481f-bdf4-d6072d3a842f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":187,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6810","nonce":36089},{"type":"v1","addr":"172.21.2.102:6811","nonce":36089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6812","nonce":36089},{"type":"v1","addr":"172.21.2.102:6813","nonce":36089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6816","nonce":36089},{"type":"v1","addr":"172.21.2.102:6817","nonce":36089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6814","nonce":36089},{"type":"v1","addr":"172.21.2.102:6815","nonce":36089}]},"public_addr":"172.21.2.102:6811/36089","cluster_addr":"172.21.2.102:6813/36089","heartbeat_back_addr":"172.21.2.102:6817/36089","heartbeat_front_addr":"172.21.2.102:6815/36089","state":["exists","up"]},{"osd":3,"uuid":"cbc678f4-83a9-45e6-8830-2f7c232b1ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":187,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6802","nonce":36090},{"type":"v1","addr":"172.21.2.102:6803","nonce":36090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6804","nonce":36090},{"type":"v1","addr":"172.21.2.102:6805","nonce":36090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6808","nonce":36090},{"type":"v1","addr":"172.21.2.102:6809","nonce":36090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6806","nonce":36090},{"type":"v1","addr":"172.21.2.102:6807","nonce":36090}]},"public_addr":"172.21.2.102:6803/36090","cluster_addr":"172.21.2.102:6805/36090","heartbeat_back_addr":"172.21.2.102:6809/36090","heartbeat_front_addr":"172.21.2.102:6807/36090","state":["exists","up"]},{"osd":4,"uuid":"c01e4008-b258-4f10-afd2-9e7f37214351","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":187,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6808","nonce":36368},{"type":"v1","addr":"172.21.2.113:6811","nonce":36368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6814","nonce":36368},{"type":"v1","addr":"172.21.2.113:6818","nonce":36368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6826","nonce":36368},{"type":"v1","addr":"172.21.2.113:6828","nonce":36368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6820","nonce":36368},{"type":"v1","addr":"172.21.2.113:6824","nonce":36368}]},"public_addr":"172.21.2.113:6811/36368","cluster_addr":"172.21.2.113:6818/36368","heartbeat_back_addr":"172.21.2.113:6828/36368","heartbeat_front_addr":"172.21.2.113:6824/36368","state":["exists","up"]},{"osd":5,"uuid":"551f6007-daf8-4a98-8ee0-c2b16bcccb5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":178,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6802","nonce":36366},{"type":"v1","addr":"172.21.2.113:6805","nonce":36366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6807","nonce":36366},{"type":"v1","addr":"172.21.2.113:6810","nonce":36366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6819","nonce":36366},{"type":"v1","addr":"172.21.2.113:6822","nonce":36366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6812","nonce":36366},{"type":"v1","addr":"172.21.2.113:6815","nonce":36366}]},"public_addr":"172.21.2.113:6805/36366","cluster_addr":"172.21.2.113:6810/36366","heartbeat_back_addr":"172.21.2.113:6822/36366","heartbeat_front_addr":"172.21.2.113:6815/36366","state":["exists","up"]},{"osd":6,"uuid":"bf67c040-0e50-48be-b102-c3943bc9ebef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":187,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6800","nonce":36367},{"type":"v1","addr":"172.21.2.113:6801","nonce":36367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6803","nonce":36367},{"type":"v1","addr":"172.21.2.113:6804","nonce":36367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6813","nonce":36367},{"type":"v1","addr":"172.21.2.113:6816","nonce":36367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6806","nonce":36367},{"type":"v1","addr":"172.21.2.113:6809","nonce":36367}]},"public_addr":"172.21.2.113:6801/36367","cluster_addr":"172.21.2.113:6804/36367","heartbeat_back_addr":"172.21.2.113:6816/36367","heartbeat_front_addr":"172.21.2.113:6809/36367","state":["exists","up"]},{"osd":7,"uuid":"f5461f80-018d-428f-be89-2eb2bcbe7bb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":187,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6817","nonce":36365},{"type":"v1","addr":"172.21.2.113:6821","nonce":36365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6823","nonce":36365},{"type":"v1","addr":"172.21.2.113:6825","nonce":36365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6830","nonce":36365},{"type":"v1","addr":"172.21.2.113:6831","nonce":36365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6827","nonce":36365},{"type":"v1","addr":"172.21.2.113:6829","nonce":36365}]},"public_addr":"172.21.2.113:6821/36365","cluster_addr":"172.21.2.113:6825/36365","heartbeat_back_addr":"172.21.2.113:6831/36365","heartbeat_front_addr":"172.21.2.113:6829/36365","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:39:38.213 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:38.211 7fa420c3f700 1 -- 172.21.2.102:0/3443393309 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa400018730 msgr2=0x7fa40001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:38.214 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:38.211 7fa420c3f700 1 --2- 172.21.2.102:0/3443393309 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa400018730 0x7fa40001ac10 crc :-1 s=READY pgs=1424 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:38.214 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:38.211 7fa420c3f700 1 -- 172.21.2.102:0/3443393309 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa41c0f0680 msgr2=0x7fa41c0a2930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:38.214 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:38.211 7fa420c3f700 1 --2- 172.21.2.102:0/3443393309 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa41c0f0680 0x7fa41c0a2930 secure :-1 s=READY pgs=3259 cs=0 l=1 rev1=1 rx=0x7fa40c006980 tx=0x7fa40c00b9b0).stop 2021-02-10T11:39:38.214 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:38.211 7fa420c3f700 1 -- 172.21.2.102:0/3443393309 shutdown_connections 2021-02-10T11:39:38.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:38.211 7fa420c3f700 1 --2- 172.21.2.102:0/3443393309 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa41c0ec180 0x7fa41c0ec580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:38.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:38.211 7fa420c3f700 1 --2- 172.21.2.102:0/3443393309 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa41c0f0680 0x7fa41c0a2930 unknown :-1 s=CLOSED pgs=3259 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:38.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:38.211 7fa420c3f700 1 --2- 172.21.2.102:0/3443393309 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa400018730 0x7fa40001ac10 unknown :-1 s=CLOSED pgs=1424 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:38.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:38.211 7fa420c3f700 1 --2- 172.21.2.102:0/3443393309 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa41c0f0280 0x7fa41c0a2310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:38.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:38.211 7fa420c3f700 1 -- 172.21.2.102:0/3443393309 >> 172.21.2.102:0/3443393309 conn(0x7fa41c0c2130 msgr2=0x7fa41c09e3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:38.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:38.211 7fa420c3f700 1 -- 172.21.2.102:0/3443393309 shutdown_connections 2021-02-10T11:39:38.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:38.211 7fa420c3f700 1 -- 172.21.2.102:0/3443393309 wait complete. 2021-02-10T11:39:38.222 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:03:37.322519', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 40, 'pool_name': 'cephfs_metadata', 'create_time': '2021-02-10 11:39:28.340428', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '188', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 41, 'pool_name': 'cephfs_data', 'create_time': '2021-02-10 11:39:29.367624', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '189', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-02-10T11:39:38.223 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:39:38.655 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:38.636 7fb87734a700 1 Processor -- start 2021-02-10T11:39:38.655 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:38.636 7fb87734a700 1 -- start start 2021-02-10T11:39:38.655 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:38.637 7fb87734a700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fb8700d1870 0x7fb8701b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:38.655 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:38.637 7fb87734a700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fb8701b0ae0 0x7fb870108940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:38.655 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:38.637 7fb87734a700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8700fe1c0 con 0x7fb8700d1c70 2021-02-10T11:39:38.655 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:38.637 7fb87734a700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb870109a60 con 0x7fb8700d1870 2021-02-10T11:39:38.655 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:38.637 7fb87734a700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb870109750 con 0x7fb8701b0ae0 2021-02-10T11:39:38.655 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:38.637 7fb875166700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fb8700d1870 0x7fb8701b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:38.655 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:38.637 7fb875166700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fb8700d1870 0x7fb8701b04c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:40566/0 (socket says 172.21.2.102:40566) 2021-02-10T11:39:38.656 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:38.637 7fb875166700 1 -- 172.21.2.102:0/2200596467 learned_addr learned my addr 172.21.2.102:0/2200596467 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:39:38.656 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:38.637 7fb875166700 1 -- 172.21.2.102:0/2200596467 >> v2:172.21.2.113:3300/0 conn(0x7fb8701b0ae0 msgr2=0x7fb870108940 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:39:38.656 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:38.637 7fb875166700 1 --2- 172.21.2.102:0/2200596467 >> v2:172.21.2.113:3300/0 conn(0x7fb8701b0ae0 0x7fb870108940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:38.656 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:38.637 7fb875166700 1 -- 172.21.2.102:0/2200596467 >> v1:172.21.2.113:6789/0 conn(0x7fb8700d1c70 legacy=0x7fb870108f60 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:39:38.656 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:38.637 7fb875166700 1 -- 172.21.2.102:0/2200596467 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb870088070 con 0x7fb8700d1870 2021-02-10T11:39:38.656 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:38.637 7fb875166700 1 --2- 172.21.2.102:0/2200596467 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb8700d1870 0x7fb8701b04c0 secure :-1 s=READY pgs=3260 cs=0 l=1 rev1=1 rx=0x7fb864006700 tx=0x7fb864009b00).ready entity=mon.0 client_cookie=bb6bf2bb6a6cea23 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:38.656 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:38.638 7fb863fff700 1 -- 172.21.2.102:0/2200596467 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb86400c0c0 con 0x7fb8700d1870 2021-02-10T11:39:38.656 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:38.638 7fb863fff700 1 -- 172.21.2.102:0/2200596467 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb86400c400 con 0x7fb8700d1870 2021-02-10T11:39:38.656 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:38.638 7fb863fff700 1 -- 172.21.2.102:0/2200596467 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb86400c920 con 0x7fb8700d1870 2021-02-10T11:39:38.656 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:38.638 7fb87734a700 1 -- 172.21.2.102:0/2200596467 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb8700d1870 msgr2=0x7fb8701b04c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:38.656 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:38.638 7fb87734a700 1 --2- 172.21.2.102:0/2200596467 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb8700d1870 0x7fb8701b04c0 secure :-1 s=READY pgs=3260 cs=0 l=1 rev1=1 rx=0x7fb864006700 tx=0x7fb864009b00).stop 2021-02-10T11:39:38.656 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:38.638 7fb87734a700 1 -- 172.21.2.102:0/2200596467 shutdown_connections 2021-02-10T11:39:38.656 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:38.638 7fb87734a700 1 --2- 172.21.2.102:0/2200596467 >> v2:172.21.2.113:3300/0 conn(0x7fb8701b0ae0 0x7fb870108940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:38.656 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:38.638 7fb87734a700 1 --2- 172.21.2.102:0/2200596467 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb8700d1870 0x7fb8701b04c0 unknown :-1 s=CLOSED pgs=3260 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:38.657 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:38.638 7fb87734a700 1 -- 172.21.2.102:0/2200596467 >> 172.21.2.102:0/2200596467 conn(0x7fb8700ce420 msgr2=0x7fb8700d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:38.657 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:38.639 7fb87734a700 1 -- 172.21.2.102:0/2200596467 shutdown_connections 2021-02-10T11:39:38.657 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:38.639 7fb87734a700 1 -- 172.21.2.102:0/2200596467 wait complete. 2021-02-10T11:39:38.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:38.653 7fb87734a700 1 Processor -- start 2021-02-10T11:39:38.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:38.653 7fb87734a700 1 -- start start 2021-02-10T11:39:38.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:38.653 7fb87734a700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb8700c2370 0x7fb87018f870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:38.659 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:38.653 7fb87734a700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb870084cb0 0x7fb8701a7470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:38.659 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:38.653 7fb87734a700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb8701a7a90 0x7fb870085700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:38.659 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:38.653 7fb87734a700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8700be840 con 0x7fb8701a7a90 2021-02-10T11:39:38.659 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:38.653 7fb87734a700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8700c29a0 con 0x7fb870084cb0 2021-02-10T11:39:38.659 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:38.653 7fb87734a700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8701a7e90 con 0x7fb8700c2370 2021-02-10T11:39:38.659 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:38.653 7fb875967700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb8701a7a90 0x7fb870085700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:38.659 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:38.654 7fb875967700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb8701a7a90 0x7fb870085700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:40572/0 (socket says 172.21.2.102:40572) 2021-02-10T11:39:38.659 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:38.654 7fb875967700 1 -- 172.21.2.102:0/2265375002 learned_addr learned my addr 172.21.2.102:0/2265375002 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:39:38.659 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:38.654 7fb875967700 1 -- 172.21.2.102:0/2265375002 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb8700c2370 msgr2=0x7fb87018f870 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:39:38.659 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:38.654 7fb875967700 1 --2- 172.21.2.102:0/2265375002 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb8700c2370 0x7fb87018f870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:38.659 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:38.654 7fb875967700 1 -- 172.21.2.102:0/2265375002 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb870084cb0 msgr2=0x7fb8701a7470 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:39:38.659 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:38.654 7fb875967700 1 --2- 172.21.2.102:0/2265375002 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb870084cb0 0x7fb8701a7470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:38.659 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:38.654 7fb875967700 1 -- 172.21.2.102:0/2265375002 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8701a80b0 con 0x7fb8701a7a90 2021-02-10T11:39:38.659 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:38.654 7fb875967700 1 --2- 172.21.2.102:0/2265375002 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb8701a7a90 0x7fb870085700 secure :-1 s=READY pgs=3261 cs=0 l=1 rev1=1 rx=0x7fb868009940 tx=0x7fb86800bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:38.660 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:38.654 7fb861ffb700 1 -- 172.21.2.102:0/2265375002 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb868003030 con 0x7fb8701a7a90 2021-02-10T11:39:38.660 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:38.654 7fb861ffb700 1 -- 172.21.2.102:0/2265375002 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb868000d30 con 0x7fb8701a7a90 2021-02-10T11:39:38.660 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:38.654 7fb861ffb700 1 -- 172.21.2.102:0/2265375002 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb8680014f0 con 0x7fb8701a7a90 2021-02-10T11:39:38.660 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:38.654 7fb87734a700 1 -- 172.21.2.102:0/2265375002 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8701a80b0 con 0x7fb8701a7a90 2021-02-10T11:39:38.660 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:38.654 7fb87734a700 1 -- 172.21.2.102:0/2265375002 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8701a80b0 con 0x7fb8701a7a90 2021-02-10T11:39:38.660 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:38.655 7fb861ffb700 1 -- 172.21.2.102:0/2265375002 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fb868001270 con 0x7fb8701a7a90 2021-02-10T11:39:38.660 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:38.655 7fb861ffb700 1 --2- 172.21.2.102:0/2265375002 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb8500186c0 0x7fb85001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:38.660 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:38.655 7fb861ffb700 1 -- 172.21.2.102:0/2265375002 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(189..189 src has 1..189) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fb868032990 con 0x7fb8701a7a90 2021-02-10T11:39:38.660 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:38.655 7fb875166700 1 --2- 172.21.2.102:0/2265375002 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb8500186c0 0x7fb85001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:38.660 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:38.655 7fb875166700 1 --2- 172.21.2.102:0/2265375002 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb8500186c0 0x7fb85001abc0 crc :-1 s=READY pgs=1425 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:38.689 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:38.687 7fb87734a700 1 -- 172.21.2.102:0/2265375002 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8700be840 con 0x7fb8701a7a90 2021-02-10T11:39:38.692 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:38.690 7fb861ffb700 1 -- 172.21.2.102:0/2265375002 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fb86800be30 con 0x7fb8701a7a90 2021-02-10T11:39:38.820 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:38.818 7fb87734a700 1 -- 172.21.2.102:0/2265375002 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7fb8700be840 con 0x7fb8701a7a90 2021-02-10T11:39:38.820 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:38.818 7fb861ffb700 1 -- 172.21.2.102:0/2265375002 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v189) v1 ==== 92+0+10 (secure 0 0 0) 0x7fb86801f8f0 con 0x7fb8701a7a90 2021-02-10T11:39:38.821 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:39:38.822 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:38.820 7fb87734a700 1 -- 172.21.2.102:0/2265375002 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb8500186c0 msgr2=0x7fb85001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:38.822 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:38.820 7fb87734a700 1 --2- 172.21.2.102:0/2265375002 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb8500186c0 0x7fb85001abc0 crc :-1 s=READY pgs=1425 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:38.822 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:38.820 7fb87734a700 1 -- 172.21.2.102:0/2265375002 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb8701a7a90 msgr2=0x7fb870085700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:38.822 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:38.820 7fb87734a700 1 --2- 172.21.2.102:0/2265375002 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb8701a7a90 0x7fb870085700 secure :-1 s=READY pgs=3261 cs=0 l=1 rev1=1 rx=0x7fb868009940 tx=0x7fb86800bcf0).stop 2021-02-10T11:39:38.823 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:38.820 7fb87734a700 1 -- 172.21.2.102:0/2265375002 shutdown_connections 2021-02-10T11:39:38.823 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:38.820 7fb87734a700 1 --2- 172.21.2.102:0/2265375002 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb8701a7a90 0x7fb870085700 unknown :-1 s=CLOSED pgs=3261 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:38.823 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:38.820 7fb87734a700 1 --2- 172.21.2.102:0/2265375002 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb8500186c0 0x7fb85001abc0 unknown :-1 s=CLOSED pgs=1425 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:38.823 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:38.820 7fb87734a700 1 --2- 172.21.2.102:0/2265375002 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb870084cb0 0x7fb8701a7470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:38.823 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:38.820 7fb87734a700 1 --2- 172.21.2.102:0/2265375002 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb8700c2370 0x7fb87018f870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:38.823 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:38.820 7fb87734a700 1 -- 172.21.2.102:0/2265375002 >> 172.21.2.102:0/2265375002 conn(0x7fb8700b5dc0 msgr2=0x7fb8700c2d90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:38.824 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:38.821 7fb87734a700 1 -- 172.21.2.102:0/2265375002 shutdown_connections 2021-02-10T11:39:38.824 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:38.821 7fb87734a700 1 -- 172.21.2.102:0/2265375002 wait complete. 2021-02-10T11:39:38.831 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-02-10T11:39:39.266 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:39.247 7f0ddfdbe700 1 Processor -- start 2021-02-10T11:39:39.266 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:39.247 7f0ddfdbe700 1 -- start start 2021-02-10T11:39:39.266 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:39.247 7f0ddfdbe700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f0dd80d1870 0x7f0dd81b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:39.266 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:39.247 7f0ddfdbe700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f0dd81b0ae0 0x7f0dd8108940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:39.266 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:39.247 7f0ddfdbe700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0dd80fe1c0 con 0x7f0dd80d1c70 2021-02-10T11:39:39.266 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:39.247 7f0ddfdbe700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0dd8109a60 con 0x7f0dd80d1870 2021-02-10T11:39:39.267 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:39.247 7f0ddfdbe700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0dd8109750 con 0x7f0dd81b0ae0 2021-02-10T11:39:39.267 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:39.248 7f0dddbda700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f0dd80d1870 0x7f0dd81b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:39.267 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:39.248 7f0ddd3d9700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f0dd81b0ae0 0x7f0dd8108940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:39.267 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:39.248 7f0dddbda700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f0dd80d1870 0x7f0dd81b04c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:40576/0 (socket says 172.21.2.102:40576) 2021-02-10T11:39:39.267 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:39.248 7f0ddd3d9700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f0dd81b0ae0 0x7f0dd8108940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.113:3300/0 says I am v2:172.21.2.102:44006/0 (socket says 172.21.2.102:44006) 2021-02-10T11:39:39.267 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:39.248 7f0dddbda700 1 -- 172.21.2.102:0/3367012589 learned_addr learned my addr 172.21.2.102:0/3367012589 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:39:39.267 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:39.248 7f0dde3db700 1 --1- 172.21.2.102:0/3367012589 >> v1:172.21.2.102:6789/0 conn(0x7f0dd80d1c70 0x7f0dd8108f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:32800/0 (socket says 172.21.2.102:32800) 2021-02-10T11:39:39.267 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:39.248 7f0dddbda700 1 -- 172.21.2.102:0/3367012589 >> v2:172.21.2.113:3300/0 conn(0x7f0dd81b0ae0 msgr2=0x7f0dd8108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:39.267 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:39.248 7f0dddbda700 1 --2- 172.21.2.102:0/3367012589 >> v2:172.21.2.113:3300/0 conn(0x7f0dd81b0ae0 0x7f0dd8108940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:39.267 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:39.248 7f0dddbda700 1 -- 172.21.2.102:0/3367012589 >> v1:172.21.2.102:6789/0 conn(0x7f0dd80d1c70 legacy=0x7f0dd8108f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:39.267 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:39.248 7f0dddbda700 1 -- 172.21.2.102:0/3367012589 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dd8088070 con 0x7f0dd80d1870 2021-02-10T11:39:39.267 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:39.248 7f0ddd3d9700 1 --2- 172.21.2.102:0/3367012589 >> v2:172.21.2.113:3300/0 conn(0x7f0dd81b0ae0 0x7f0dd8108940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:39:39.268 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:39.248 7f0dddbda700 1 --2- 172.21.2.102:0/3367012589 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0dd80d1870 0x7f0dd81b04c0 secure :-1 s=READY pgs=3263 cs=0 l=1 rev1=1 rx=0x7f0dd4006700 tx=0x7f0dd4009b80).ready entity=mon.0 client_cookie=3dd958d9994bffc2 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:39.268 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:39.249 7f0ddcbd8700 1 -- 172.21.2.102:0/3367012589 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0dd4006340 con 0x7f0dd80d1870 2021-02-10T11:39:39.268 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:39.249 7f0ddcbd8700 1 -- 172.21.2.102:0/3367012589 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0dd400c0c0 con 0x7f0dd80d1870 2021-02-10T11:39:39.268 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:39.249 7f0ddcbd8700 1 -- 172.21.2.102:0/3367012589 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0dd400c5c0 con 0x7f0dd80d1870 2021-02-10T11:39:39.268 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:39.249 7f0ddfdbe700 1 -- 172.21.2.102:0/3367012589 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0dd80d1870 msgr2=0x7f0dd81b04c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:39.268 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:39.249 7f0ddfdbe700 1 --2- 172.21.2.102:0/3367012589 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0dd80d1870 0x7f0dd81b04c0 secure :-1 s=READY pgs=3263 cs=0 l=1 rev1=1 rx=0x7f0dd4006700 tx=0x7f0dd4009b80).stop 2021-02-10T11:39:39.268 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:39.249 7f0ddfdbe700 1 -- 172.21.2.102:0/3367012589 shutdown_connections 2021-02-10T11:39:39.268 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:39.249 7f0ddfdbe700 1 --2- 172.21.2.102:0/3367012589 >> v2:172.21.2.113:3300/0 conn(0x7f0dd81b0ae0 0x7f0dd8108940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:39.268 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:39.249 7f0ddfdbe700 1 --2- 172.21.2.102:0/3367012589 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0dd80d1870 0x7f0dd81b04c0 unknown :-1 s=CLOSED pgs=3263 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:39.268 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:39.249 7f0ddfdbe700 1 -- 172.21.2.102:0/3367012589 >> 172.21.2.102:0/3367012589 conn(0x7f0dd80ce420 msgr2=0x7f0dd80d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:39.268 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:39.250 7f0ddfdbe700 1 -- 172.21.2.102:0/3367012589 shutdown_connections 2021-02-10T11:39:39.268 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:39.250 7f0ddfdbe700 1 -- 172.21.2.102:0/3367012589 wait complete. 2021-02-10T11:39:39.270 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:39.264 7f0ddfdbe700 1 Processor -- start 2021-02-10T11:39:39.270 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:39.264 7f0ddfdbe700 1 -- start start 2021-02-10T11:39:39.270 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:39.265 7f0ddfdbe700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0dd8085bc0 0x7f0dd80863e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:39.270 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:39.265 7f0ddfdbe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0dd80954c0 0x7f0dd8096300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:39.270 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:39.265 7f0ddfdbe700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0dd80958c0 0x7f0dd8096920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:39.270 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:39.265 7f0ddfdbe700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dd8095d80 con 0x7f0dd80954c0 2021-02-10T11:39:39.271 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:39.265 7f0ddfdbe700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dd80bfa60 con 0x7f0dd80958c0 2021-02-10T11:39:39.271 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:39.265 7f0ddfdbe700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0dd8085fc0 con 0x7f0dd8085bc0 2021-02-10T11:39:39.271 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:39.265 7f0ddd3d9700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0dd80954c0 0x7f0dd8096300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:39.271 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:39.265 7f0ddd3d9700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0dd80954c0 0x7f0dd8096300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:40578/0 (socket says 172.21.2.102:40578) 2021-02-10T11:39:39.271 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:39.265 7f0ddd3d9700 1 -- 172.21.2.102:0/121102699 learned_addr learned my addr 172.21.2.102:0/121102699 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:39:39.271 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:39.265 7f0dddbda700 1 --2- 172.21.2.102:0/121102699 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0dd8085bc0 0x7f0dd80863e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:39.271 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:39.265 7f0dde3db700 1 --2- 172.21.2.102:0/121102699 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0dd80958c0 0x7f0dd8096920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:39.271 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:39.265 7f0ddd3d9700 1 -- 172.21.2.102:0/121102699 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0dd8085bc0 msgr2=0x7f0dd80863e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:39.271 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:39.265 7f0ddd3d9700 1 --2- 172.21.2.102:0/121102699 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0dd8085bc0 0x7f0dd80863e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:39.271 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:39.265 7f0ddd3d9700 1 -- 172.21.2.102:0/121102699 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0dd80958c0 msgr2=0x7f0dd8096920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:39.271 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:39.265 7f0ddd3d9700 1 --2- 172.21.2.102:0/121102699 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0dd80958c0 0x7f0dd8096920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:39.271 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:39.265 7f0ddd3d9700 1 -- 172.21.2.102:0/121102699 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dd8086c10 con 0x7f0dd80954c0 2021-02-10T11:39:39.271 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:39.265 7f0dddbda700 1 --2- 172.21.2.102:0/121102699 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0dd8085bc0 0x7f0dd80863e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:39:39.272 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:39.265 7f0dde3db700 1 --2- 172.21.2.102:0/121102699 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0dd80958c0 0x7f0dd8096920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:39:39.272 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:39.266 7f0ddd3d9700 1 --2- 172.21.2.102:0/121102699 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0dd80954c0 0x7f0dd8096300 secure :-1 s=READY pgs=3264 cs=0 l=1 rev1=1 rx=0x7f0dc8006f40 tx=0x7f0dc800b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:39.272 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:39.266 7f0dce7fc700 1 -- 172.21.2.102:0/121102699 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0dc800bd80 con 0x7f0dd80954c0 2021-02-10T11:39:39.272 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:39.266 7f0dce7fc700 1 -- 172.21.2.102:0/121102699 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0dc8000c90 con 0x7f0dd80954c0 2021-02-10T11:39:39.272 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:39.266 7f0ddfdbe700 1 -- 172.21.2.102:0/121102699 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0dd8086c10 con 0x7f0dd80954c0 2021-02-10T11:39:39.272 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:39.266 7f0dce7fc700 1 -- 172.21.2.102:0/121102699 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0dc8002220 con 0x7f0dd80954c0 2021-02-10T11:39:39.272 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:39.266 7f0ddfdbe700 1 -- 172.21.2.102:0/121102699 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0dd8086c10 con 0x7f0dd80954c0 2021-02-10T11:39:39.272 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:39.267 7f0dce7fc700 1 -- 172.21.2.102:0/121102699 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f0dc800a000 con 0x7f0dd80954c0 2021-02-10T11:39:39.272 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:39.267 7f0dce7fc700 1 --2- 172.21.2.102:0/121102699 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0dc0018730 0x7f0dc001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:39.272 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:39.267 7f0dce7fc700 1 -- 172.21.2.102:0/121102699 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(189..189 src has 1..189) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f0dc8032790 con 0x7f0dd80954c0 2021-02-10T11:39:39.272 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:39.267 7f0dddbda700 1 --2- 172.21.2.102:0/121102699 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0dc0018730 0x7f0dc001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:39.272 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:39.267 7f0dddbda700 1 --2- 172.21.2.102:0/121102699 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0dc0018730 0x7f0dc001ac10 crc :-1 s=READY pgs=1426 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:39.299 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:39.297 7f0ddfdbe700 1 -- 172.21.2.102:0/121102699 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0dd80fe1c0 con 0x7f0dd80954c0 2021-02-10T11:39:39.302 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:39.300 7f0dce7fc700 1 -- 172.21.2.102:0/121102699 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f0dc8000c90 con 0x7f0dd80954c0 2021-02-10T11:39:39.431 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:39.428 7f0ddfdbe700 1 -- 172.21.2.102:0/121102699 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7f0dd80fe1c0 con 0x7f0dd80954c0 2021-02-10T11:39:39.431 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:39.429 7f0dce7fc700 1 -- 172.21.2.102:0/121102699 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v189) v1 ==== 104+0+10 (secure 0 0 0) 0x7f0dc800a000 con 0x7f0dd80954c0 2021-02-10T11:39:39.432 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:39:39.433 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:39.431 7f0ddfdbe700 1 -- 172.21.2.102:0/121102699 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0dc0018730 msgr2=0x7f0dc001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:39.433 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:39.431 7f0ddfdbe700 1 --2- 172.21.2.102:0/121102699 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0dc0018730 0x7f0dc001ac10 crc :-1 s=READY pgs=1426 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:39.434 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:39.431 7f0ddfdbe700 1 -- 172.21.2.102:0/121102699 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0dd80954c0 msgr2=0x7f0dd8096300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:39.434 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:39.431 7f0ddfdbe700 1 --2- 172.21.2.102:0/121102699 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0dd80954c0 0x7f0dd8096300 secure :-1 s=READY pgs=3264 cs=0 l=1 rev1=1 rx=0x7f0dc8006f40 tx=0x7f0dc800b990).stop 2021-02-10T11:39:39.434 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:39.431 7f0ddfdbe700 1 -- 172.21.2.102:0/121102699 shutdown_connections 2021-02-10T11:39:39.434 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:39.431 7f0ddfdbe700 1 --2- 172.21.2.102:0/121102699 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0dc0018730 0x7f0dc001ac10 unknown :-1 s=CLOSED pgs=1426 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:39.434 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:39.431 7f0ddfdbe700 1 --2- 172.21.2.102:0/121102699 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0dd80958c0 0x7f0dd8096920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:39.434 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:39.431 7f0ddfdbe700 1 --2- 172.21.2.102:0/121102699 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0dd80954c0 0x7f0dd8096300 unknown :-1 s=CLOSED pgs=3264 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:39.434 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:39.431 7f0ddfdbe700 1 --2- 172.21.2.102:0/121102699 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0dd8085bc0 0x7f0dd80863e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:39.434 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:39.431 7f0ddfdbe700 1 -- 172.21.2.102:0/121102699 >> 172.21.2.102:0/121102699 conn(0x7f0dd80bef00 msgr2=0x7f0dd809d1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:39.434 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:39.431 7f0ddfdbe700 1 -- 172.21.2.102:0/121102699 shutdown_connections 2021-02-10T11:39:39.434 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:39.431 7f0ddfdbe700 1 -- 172.21.2.102:0/121102699 wait complete. 2021-02-10T11:39:39.441 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-02-10T11:39:39.876 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:39.855 7f68fcb2a700 1 Processor -- start 2021-02-10T11:39:39.877 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:39.856 7f68fcb2a700 1 -- start start 2021-02-10T11:39:39.877 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:39.856 7f68fcb2a700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f68f80d1870 0x7f68f81b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:39.877 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:39.856 7f68fcb2a700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f68f80cff70 0x7f68f8109130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:39.877 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:39.856 7f68fcb2a700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68f8109a60 con 0x7f68f81b0ae0 2021-02-10T11:39:39.877 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:39.856 7f68fcb2a700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68f8109750 con 0x7f68f80d1870 2021-02-10T11:39:39.877 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:39.856 7f68fcb2a700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68f8088070 con 0x7f68f80cff70 2021-02-10T11:39:39.877 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:39.859 7f68f685e700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f68f80d1870 0x7f68f81b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:39.877 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:39.859 7f68f705f700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f68f80cff70 0x7f68f8109130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:39.877 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:39.859 7f68f685e700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f68f80d1870 0x7f68f81b04c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.113:3300/0 says I am v2:172.21.2.102:44016/0 (socket says 172.21.2.102:44016) 2021-02-10T11:39:39.877 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:39.859 7f68f705f700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f68f80cff70 0x7f68f8109130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.113:3301/0 says I am v2:172.21.2.102:46122/0 (socket says 172.21.2.102:46122) 2021-02-10T11:39:39.878 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:39.859 7f68f685e700 1 -- 172.21.2.102:0/1731906058 learned_addr learned my addr 172.21.2.102:0/1731906058 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:39:39.878 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:39.859 7f68f605d700 1 --1- 172.21.2.102:0/1731906058 >> v1:172.21.2.113:6789/0 conn(0x7f68f81b0ae0 0x7f68f8108940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.113:6789/0 says I am v1:172.21.2.102:40134/0 (socket says 172.21.2.102:40134) 2021-02-10T11:39:39.878 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:39.859 7f68f585c700 1 -- 172.21.2.102:0/1731906058 <== mon.1 v1:172.21.2.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1276533528 0 0) 0x7f68ec0010c0 con 0x7f68f81b0ae0 2021-02-10T11:39:39.878 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:39.859 7f68f585c700 1 -- 172.21.2.102:0/1731906058 --> v1:172.21.2.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68e0002fe0 con 0x7f68f81b0ae0 2021-02-10T11:39:39.878 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:39.859 7f68f685e700 1 -- 172.21.2.102:0/1731906058 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f68f80cff70 msgr2=0x7f68f8109130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:39.878 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:39.859 7f68f685e700 1 --2- 172.21.2.102:0/1731906058 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f68f80cff70 0x7f68f8109130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:39.878 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:39.859 7f68f685e700 1 -- 172.21.2.102:0/1731906058 >> v1:172.21.2.113:6789/0 conn(0x7f68f81b0ae0 legacy=0x7f68f8108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:39.878 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:39.859 7f68f685e700 1 -- 172.21.2.102:0/1731906058 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68f80d1c70 con 0x7f68f80d1870 2021-02-10T11:39:39.878 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:39.859 7f68f705f700 1 --2- 172.21.2.102:0/1731906058 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f68f80cff70 0x7f68f8109130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:39:39.878 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:39.859 7f68f685e700 1 --2- 172.21.2.102:0/1731906058 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f68f80d1870 0x7f68f81b04c0 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 rx=0x7f68e8006700 tx=0x7f68e8009b80).ready entity=mon.1 client_cookie=67ac2fef944f1c51 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:39.878 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:39.859 7f68f585c700 1 -- 172.21.2.102:0/1731906058 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f68e800c0c0 con 0x7f68f80d1870 2021-02-10T11:39:39.878 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:39.859 7f68f585c700 1 -- 172.21.2.102:0/1731906058 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68e800c440 con 0x7f68f80d1870 2021-02-10T11:39:39.879 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:39.859 7f68f585c700 1 -- 172.21.2.102:0/1731906058 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f68e800c960 con 0x7f68f80d1870 2021-02-10T11:39:39.879 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:39.859 7f68fcb2a700 1 -- 172.21.2.102:0/1731906058 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f68f80d1870 msgr2=0x7f68f81b04c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:39.879 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:39.859 7f68fcb2a700 1 --2- 172.21.2.102:0/1731906058 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f68f80d1870 0x7f68f81b04c0 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 rx=0x7f68e8006700 tx=0x7f68e8009b80).stop 2021-02-10T11:39:39.879 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:39.860 7f68fcb2a700 1 -- 172.21.2.102:0/1731906058 shutdown_connections 2021-02-10T11:39:39.879 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:39.860 7f68fcb2a700 1 --2- 172.21.2.102:0/1731906058 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f68f80cff70 0x7f68f8109130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:39.879 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:39.860 7f68fcb2a700 1 --2- 172.21.2.102:0/1731906058 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f68f80d1870 0x7f68f81b04c0 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:39.879 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:39.860 7f68fcb2a700 1 -- 172.21.2.102:0/1731906058 >> 172.21.2.102:0/1731906058 conn(0x7f68f80ce420 msgr2=0x7f68f80d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:39.879 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:39.860 7f68fcb2a700 1 -- 172.21.2.102:0/1731906058 shutdown_connections 2021-02-10T11:39:39.879 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:39.860 7f68fcb2a700 1 -- 172.21.2.102:0/1731906058 wait complete. 2021-02-10T11:39:39.881 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:39.875 7f68fcb2a700 1 Processor -- start 2021-02-10T11:39:39.881 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:39.875 7f68fcb2a700 1 -- start start 2021-02-10T11:39:39.881 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:39.875 7f68fcb2a700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f68f81a79a0 0x7f68f81a7da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:39.881 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:39.875 7f68fcb2a700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f68f80bdb40 0x7f68f80bdf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:39.881 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:39.875 7f68fcb2a700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f68f8082ca0 0x7f68f80830a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:39.881 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:39.875 7f68fcb2a700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68f819d990 con 0x7f68f8082ca0 2021-02-10T11:39:39.881 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:39.875 7f68fcb2a700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68f819dbb0 con 0x7f68f80bdb40 2021-02-10T11:39:39.881 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:39.875 7f68fcb2a700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68f80836c0 con 0x7f68f81a79a0 2021-02-10T11:39:39.881 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:39.875 7f68f705f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f68f8082ca0 0x7f68f80830a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:39.882 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:39.875 7f68f705f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f68f8082ca0 0x7f68f80830a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:40588/0 (socket says 172.21.2.102:40588) 2021-02-10T11:39:39.882 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:39.875 7f68f705f700 1 -- 172.21.2.102:0/2681540197 learned_addr learned my addr 172.21.2.102:0/2681540197 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:39:39.882 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:39.876 7f68f605d700 1 --2- 172.21.2.102:0/2681540197 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f68f80bdb40 0x7f68f80bdf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:39.882 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:39.876 7f68f685e700 1 --2- 172.21.2.102:0/2681540197 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f68f81a79a0 0x7f68f81a7da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:39.882 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:39.876 7f68f705f700 1 -- 172.21.2.102:0/2681540197 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f68f81a79a0 msgr2=0x7f68f81a7da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:39.882 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:39.876 7f68f705f700 1 --2- 172.21.2.102:0/2681540197 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f68f81a79a0 0x7f68f81a7da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:39.882 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:39.876 7f68f705f700 1 -- 172.21.2.102:0/2681540197 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f68f80bdb40 msgr2=0x7f68f80bdf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:39.882 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:39.876 7f68f705f700 1 --2- 172.21.2.102:0/2681540197 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f68f80bdb40 0x7f68f80bdf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:39.882 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:39.876 7f68f705f700 1 -- 172.21.2.102:0/2681540197 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68f80838e0 con 0x7f68f8082ca0 2021-02-10T11:39:39.882 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:39.876 7f68f685e700 1 --2- 172.21.2.102:0/2681540197 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f68f81a79a0 0x7f68f81a7da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:39:39.882 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:39.876 7f68f705f700 1 --2- 172.21.2.102:0/2681540197 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f68f8082ca0 0x7f68f80830a0 secure :-1 s=READY pgs=3265 cs=0 l=1 rev1=1 rx=0x7f68f000b970 tx=0x7f68f000bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:39.882 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:39.876 7f68e77fe700 1 -- 172.21.2.102:0/2681540197 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f68f0000f90 con 0x7f68f8082ca0 2021-02-10T11:39:39.883 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:39.876 7f68fcb2a700 1 -- 172.21.2.102:0/2681540197 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68f80838e0 con 0x7f68f8082ca0 2021-02-10T11:39:39.883 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:39.876 7f68fcb2a700 1 -- 172.21.2.102:0/2681540197 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68f8084420 con 0x7f68f8082ca0 2021-02-10T11:39:39.883 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:39.876 7f68e77fe700 1 -- 172.21.2.102:0/2681540197 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68f0003030 con 0x7f68f8082ca0 2021-02-10T11:39:39.883 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:39.876 7f68e77fe700 1 -- 172.21.2.102:0/2681540197 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f68f0000f90 con 0x7f68f8082ca0 2021-02-10T11:39:39.883 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:39.877 7f68e77fe700 1 -- 172.21.2.102:0/2681540197 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f68f00011f0 con 0x7f68f8082ca0 2021-02-10T11:39:39.883 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:39.877 7f68e77fe700 1 --2- 172.21.2.102:0/2681540197 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f68dc018700 0x7f68dc01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:39.883 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:39.877 7f68e77fe700 1 -- 172.21.2.102:0/2681540197 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(189..189 src has 1..189) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f68f0032b30 con 0x7f68f8082ca0 2021-02-10T11:39:39.883 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:39.877 7f68f685e700 1 --2- 172.21.2.102:0/2681540197 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f68dc018700 0x7f68dc01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:39.883 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:39.877 7f68f685e700 1 --2- 172.21.2.102:0/2681540197 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f68dc018700 0x7f68dc01abc0 crc :-1 s=READY pgs=1427 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:39.908 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:39.906 7f68fcb2a700 1 -- 172.21.2.102:0/2681540197 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68f8082a70 con 0x7f68f8082ca0 2021-02-10T11:39:39.911 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:39.909 7f68e77fe700 1 -- 172.21.2.102:0/2681540197 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f68f00066f0 con 0x7f68f8082ca0 2021-02-10T11:39:40.039 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:40.036 7f68fcb2a700 1 -- 172.21.2.102:0/2681540197 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7f68f80838e0 con 0x7f68f8082ca0 2021-02-10T11:39:40.039 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:40.037 7f68e77fe700 1 -- 172.21.2.102:0/2681540197 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v189) v1 ==== 100+0+10 (secure 0 0 0) 0x7f68f0000e70 con 0x7f68f8082ca0 2021-02-10T11:39:40.040 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:39:40.041 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:40.039 7f68fcb2a700 1 -- 172.21.2.102:0/2681540197 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f68dc018700 msgr2=0x7f68dc01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:40.042 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:40.039 7f68fcb2a700 1 --2- 172.21.2.102:0/2681540197 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f68dc018700 0x7f68dc01abc0 crc :-1 s=READY pgs=1427 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:40.042 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:40.039 7f68fcb2a700 1 -- 172.21.2.102:0/2681540197 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f68f8082ca0 msgr2=0x7f68f80830a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:40.042 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:40.039 7f68fcb2a700 1 --2- 172.21.2.102:0/2681540197 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f68f8082ca0 0x7f68f80830a0 secure :-1 s=READY pgs=3265 cs=0 l=1 rev1=1 rx=0x7f68f000b970 tx=0x7f68f000bd10).stop 2021-02-10T11:39:40.042 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:40.039 7f68fcb2a700 1 -- 172.21.2.102:0/2681540197 shutdown_connections 2021-02-10T11:39:40.042 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:40.039 7f68fcb2a700 1 --2- 172.21.2.102:0/2681540197 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f68f8082ca0 0x7f68f80830a0 unknown :-1 s=CLOSED pgs=3265 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:40.042 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:40.039 7f68fcb2a700 1 --2- 172.21.2.102:0/2681540197 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f68dc018700 0x7f68dc01abc0 unknown :-1 s=CLOSED pgs=1427 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:40.042 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:40.039 7f68fcb2a700 1 --2- 172.21.2.102:0/2681540197 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f68f80bdb40 0x7f68f80bdf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:40.042 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:40.039 7f68fcb2a700 1 --2- 172.21.2.102:0/2681540197 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f68f81a79a0 0x7f68f81a7da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:40.042 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:40.039 7f68fcb2a700 1 -- 172.21.2.102:0/2681540197 >> 172.21.2.102:0/2681540197 conn(0x7f68f80bef00 msgr2=0x7f68f809f200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:40.042 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:40.039 7f68fcb2a700 1 -- 172.21.2.102:0/2681540197 shutdown_connections 2021-02-10T11:39:40.042 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:40.039 7f68fcb2a700 1 -- 172.21.2.102:0/2681540197 wait complete. 2021-02-10T11:39:40.050 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-02-10T11:39:40.050 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:39:40.482 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:40.462 7f3842d4e700 1 Processor -- start 2021-02-10T11:39:40.482 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:40.462 7f3842d4e700 1 -- start start 2021-02-10T11:39:40.482 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:40.463 7f3842d4e700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f383c0d0340 0x7f383c0ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:40.482 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:40.463 7f3842d4e700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f383c0f0b10 con 0x7f383c0d1870 2021-02-10T11:39:40.482 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:40.463 7f3842d4e700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f383c1b0be0 con 0x7f383c0d1c70 2021-02-10T11:39:40.482 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:40.463 7f3842d4e700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f383c0e5ed0 con 0x7f383c0d0340 2021-02-10T11:39:40.482 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:40.463 7f383bfff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f383c0d0340 0x7f383c0ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:40.482 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:40.463 7f3840b6a700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f383c0d1870 0x7f383c1b03f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:32820/0 (socket says 172.21.2.102:32820) 2021-02-10T11:39:40.483 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:40.463 7f383bfff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f383c0d0340 0x7f383c0ef640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:40596/0 (socket says 172.21.2.102:40596) 2021-02-10T11:39:40.483 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:40.463 7f3840b6a700 1 -- 172.21.2.102:0/2667144238 learned_addr learned my addr 172.21.2.102:0/2667144238 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:39:40.483 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:40.463 7f383bfff700 1 -- 172.21.2.102:0/2667144238 >> v1:172.21.2.113:6789/0 conn(0x7f383c0d1c70 legacy=0x7f383c0efc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:40.483 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:40.464 7f383bfff700 1 -- 172.21.2.102:0/2667144238 >> v1:172.21.2.102:6789/0 conn(0x7f383c0d1870 legacy=0x7f383c1b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:40.483 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:40.464 7f383b7fe700 1 -- 172.21.2.102:0/2667144238 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4048743099 0 0) 0x7f3824000ce0 con 0x7f383c0d1870 2021-02-10T11:39:40.483 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:40.464 7f383bfff700 1 -- 172.21.2.102:0/2667144238 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f383c0f0450 con 0x7f383c0d0340 2021-02-10T11:39:40.483 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:40.464 7f383bfff700 1 --2- 172.21.2.102:0/2667144238 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f383c0d0340 0x7f383c0ef640 secure :-1 s=READY pgs=3267 cs=0 l=1 rev1=1 rx=0x7f382c006650 tx=0x7f382c009a00).ready entity=mon.0 client_cookie=5efa3c00810b01aa server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:40.483 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:40.464 7f383b7fe700 1 -- 172.21.2.102:0/2667144238 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f382c009df0 con 0x7f383c0d0340 2021-02-10T11:39:40.483 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:40.464 7f383b7fe700 1 -- 172.21.2.102:0/2667144238 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f382c0009b0 con 0x7f383c0d0340 2021-02-10T11:39:40.483 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:40.464 7f383b7fe700 1 -- 172.21.2.102:0/2667144238 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f382c00c2b0 con 0x7f383c0d0340 2021-02-10T11:39:40.483 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:40.464 7f3842d4e700 1 -- 172.21.2.102:0/2667144238 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f383c0d0340 msgr2=0x7f383c0ef640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:40.483 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:40.464 7f3842d4e700 1 --2- 172.21.2.102:0/2667144238 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f383c0d0340 0x7f383c0ef640 secure :-1 s=READY pgs=3267 cs=0 l=1 rev1=1 rx=0x7f382c006650 tx=0x7f382c009a00).stop 2021-02-10T11:39:40.483 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:40.465 7f3842d4e700 1 -- 172.21.2.102:0/2667144238 shutdown_connections 2021-02-10T11:39:40.484 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:40.465 7f3842d4e700 1 --2- 172.21.2.102:0/2667144238 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f383c0d0340 0x7f383c0ef640 unknown :-1 s=CLOSED pgs=3267 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:40.484 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:40.465 7f3842d4e700 1 -- 172.21.2.102:0/2667144238 >> 172.21.2.102:0/2667144238 conn(0x7f383c0ce450 msgr2=0x7f383c0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:40.484 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:40.465 7f3842d4e700 1 -- 172.21.2.102:0/2667144238 shutdown_connections 2021-02-10T11:39:40.484 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:40.465 7f3842d4e700 1 -- 172.21.2.102:0/2667144238 wait complete. 2021-02-10T11:39:40.485 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:40.480 7f3842d4e700 1 Processor -- start 2021-02-10T11:39:40.485 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:40.480 7f3842d4e700 1 -- start start 2021-02-10T11:39:40.485 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:40.480 7f3842d4e700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f383c18f0a0 0x7f383c196ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:40.486 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:40.480 7f3842d4e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f383c1970d0 0x7f383c1974d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:40.486 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:40.480 7f3842d4e700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f383c094650 0x7f383c094a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:40.486 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:40.480 7f3842d4e700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f383c095bd0 con 0x7f383c1970d0 2021-02-10T11:39:40.486 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:40.480 7f3842d4e700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f383c0ad7c0 con 0x7f383c18f0a0 2021-02-10T11:39:40.486 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:40.480 7f3842d4e700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f383c18ec70 con 0x7f383c094650 2021-02-10T11:39:40.486 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:40.481 7f383bfff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f383c1970d0 0x7f383c1974d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:40.486 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:40.481 7f383bfff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f383c1970d0 0x7f383c1974d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:40600/0 (socket says 172.21.2.102:40600) 2021-02-10T11:39:40.486 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:40.481 7f383bfff700 1 -- 172.21.2.102:0/3829834678 learned_addr learned my addr 172.21.2.102:0/3829834678 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:39:40.486 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:40.481 7f383bfff700 1 -- 172.21.2.102:0/3829834678 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f383c094650 msgr2=0x7f383c094a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:40.486 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:40.481 7f383bfff700 1 --2- 172.21.2.102:0/3829834678 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f383c094650 0x7f383c094a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:40.487 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:40.481 7f383bfff700 1 -- 172.21.2.102:0/3829834678 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f383c18f0a0 msgr2=0x7f383c196ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:40.487 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:40.481 7f383bfff700 1 --2- 172.21.2.102:0/3829834678 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f383c18f0a0 0x7f383c196ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:40.487 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:40.481 7f383bfff700 1 -- 172.21.2.102:0/3829834678 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f383c095070 con 0x7f383c1970d0 2021-02-10T11:39:40.487 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:40.481 7f383bfff700 1 --2- 172.21.2.102:0/3829834678 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f383c1970d0 0x7f383c1974d0 secure :-1 s=READY pgs=3268 cs=0 l=1 rev1=1 rx=0x7f382c0096a0 tx=0x7f382c00b8e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:40.487 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:40.481 7f38397fa700 1 -- 172.21.2.102:0/3829834678 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f382c0093a0 con 0x7f383c1970d0 2021-02-10T11:39:40.487 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:40.481 7f38397fa700 1 -- 172.21.2.102:0/3829834678 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f382c0067f0 con 0x7f383c1970d0 2021-02-10T11:39:40.487 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:40.481 7f3842d4e700 1 -- 172.21.2.102:0/3829834678 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f383c095070 con 0x7f383c1970d0 2021-02-10T11:39:40.487 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:40.481 7f3842d4e700 1 -- 172.21.2.102:0/3829834678 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f383c095600 con 0x7f383c1970d0 2021-02-10T11:39:40.487 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:40.481 7f38397fa700 1 -- 172.21.2.102:0/3829834678 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f382c002220 con 0x7f383c1970d0 2021-02-10T11:39:40.487 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:40.482 7f38397fa700 1 -- 172.21.2.102:0/3829834678 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f382c000c90 con 0x7f383c1970d0 2021-02-10T11:39:40.488 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:40.482 7f38397fa700 1 --2- 172.21.2.102:0/3829834678 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f38200186c0 0x7f382001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:40.488 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:40.482 7f38397fa700 1 -- 172.21.2.102:0/3829834678 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(189..189 src has 1..189) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f382c032520 con 0x7f383c1970d0 2021-02-10T11:39:40.488 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:40.482 7f3840b6a700 1 --2- 172.21.2.102:0/3829834678 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f38200186c0 0x7f382001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:40.488 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:40.482 7f3840b6a700 1 --2- 172.21.2.102:0/3829834678 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f38200186c0 0x7f382001abc0 crc :-1 s=READY pgs=1428 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:40.515 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:40.513 7f3842d4e700 1 -- 172.21.2.102:0/3829834678 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f383c0ee0e0 con 0x7f383c1970d0 2021-02-10T11:39:40.517 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:40.515 7f38397fa700 1 -- 172.21.2.102:0/3829834678 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f382c0093a0 con 0x7f383c1970d0 2021-02-10T11:39:40.648 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:40.645 7f3842d4e700 1 -- 172.21.2.102:0/3829834678 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f383c0ce450 con 0x7f383c1970d0 2021-02-10T11:39:40.648 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:40.646 7f38397fa700 1 -- 172.21.2.102:0/3829834678 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 165 v165) v1 ==== 95+0+2104 (secure 0 0 0) 0x7f382c000c90 con 0x7f383c1970d0 2021-02-10T11:39:40.649 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:39:40.649 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":165,"default_fscid":20,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8998,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.113:6833/479366648","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6832","nonce":479366648},{"type":"v1","addr":"172.21.2.113:6833","nonce":479366648}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":162}],"filesystems":[{"mdsmap":{"epoch":165,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:39:30.375466","modified":"2021-02-10 11:39:32.379360","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7481},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7481":{"gid":7481,"name":"a","rank":0,"incarnation":163,"state":"up:active","state_seq":3,"addr":"172.21.2.102:6835/4290051997","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6834","nonce":4290051997},{"type":"v1","addr":"172.21.2.102:6835","nonce":4290051997}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[41],"metadata_pool":40,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":20}]} 2021-02-10T11:39:40.649 INFO:teuthology.orchestra.run.gibba002.stderr:dumped fsmap epoch 165 2021-02-10T11:39:40.650 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:40.648 7f3842d4e700 1 -- 172.21.2.102:0/3829834678 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f38200186c0 msgr2=0x7f382001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:40.650 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:40.648 7f3842d4e700 1 --2- 172.21.2.102:0/3829834678 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f38200186c0 0x7f382001abc0 crc :-1 s=READY pgs=1428 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:40.650 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:40.648 7f3842d4e700 1 -- 172.21.2.102:0/3829834678 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f383c1970d0 msgr2=0x7f383c1974d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:40.650 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:40.648 7f3842d4e700 1 --2- 172.21.2.102:0/3829834678 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f383c1970d0 0x7f383c1974d0 secure :-1 s=READY pgs=3268 cs=0 l=1 rev1=1 rx=0x7f382c0096a0 tx=0x7f382c00b8e0).stop 2021-02-10T11:39:40.651 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:40.648 7f3842d4e700 1 -- 172.21.2.102:0/3829834678 shutdown_connections 2021-02-10T11:39:40.651 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:40.648 7f3842d4e700 1 --2- 172.21.2.102:0/3829834678 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f383c094650 0x7f383c094a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:40.651 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:40.648 7f3842d4e700 1 --2- 172.21.2.102:0/3829834678 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f383c1970d0 0x7f383c1974d0 unknown :-1 s=CLOSED pgs=3268 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:40.651 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:40.648 7f3842d4e700 1 --2- 172.21.2.102:0/3829834678 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f38200186c0 0x7f382001abc0 unknown :-1 s=CLOSED pgs=1428 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:40.651 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:40.648 7f3842d4e700 1 --2- 172.21.2.102:0/3829834678 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f383c18f0a0 0x7f383c196ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:40.651 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:40.648 7f3842d4e700 1 -- 172.21.2.102:0/3829834678 >> 172.21.2.102:0/3829834678 conn(0x7f383c1a38c0 msgr2=0x7f383c1a2f00 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:40.651 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:40.648 7f3842d4e700 1 -- 172.21.2.102:0/3829834678 shutdown_connections 2021-02-10T11:39:40.651 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:40.648 7f3842d4e700 1 -- 172.21.2.102:0/3829834678 wait complete. 2021-02-10T11:39:40.861 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 165, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:39:30.375466', 'modified': '2021-02-10 11:39:32.379360', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7481}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7481': {'gid': 7481, 'name': 'a', 'rank': 0, 'incarnation': 163, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.102:6835/4290051997', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.102:6834', 'nonce': 4290051997}, {'type': 'v1', 'addr': '172.21.2.102:6835', 'nonce': 4290051997}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [41], 'metadata_pool': 40, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:39:40.861 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:39:40.861 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok status 2021-02-10T11:39:41.263 INFO:teuthology.orchestra.run.gibba002.stdout:{ 2021-02-10T11:39:41.264 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_fsid": "68db30ea-dbce-4aef-a6df-f01c6fea0513", 2021-02-10T11:39:41.264 INFO:teuthology.orchestra.run.gibba002.stdout: "whoami": 0, 2021-02-10T11:39:41.264 INFO:teuthology.orchestra.run.gibba002.stdout: "id": 7481, 2021-02-10T11:39:41.264 INFO:teuthology.orchestra.run.gibba002.stdout: "want_state": "up:active", 2021-02-10T11:39:41.264 INFO:teuthology.orchestra.run.gibba002.stdout: "state": "up:active", 2021-02-10T11:39:41.264 INFO:teuthology.orchestra.run.gibba002.stdout: "rank_uptime": 10.883951570000001, 2021-02-10T11:39:41.264 INFO:teuthology.orchestra.run.gibba002.stdout: "mdsmap_epoch": 165, 2021-02-10T11:39:41.264 INFO:teuthology.orchestra.run.gibba002.stdout: "osdmap_epoch": 189, 2021-02-10T11:39:41.264 INFO:teuthology.orchestra.run.gibba002.stdout: "osdmap_epoch_barrier": 189, 2021-02-10T11:39:41.264 INFO:teuthology.orchestra.run.gibba002.stdout: "uptime": 16.533595837 2021-02-10T11:39:41.264 INFO:teuthology.orchestra.run.gibba002.stdout:} 2021-02-10T11:39:41.268 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "68db30ea-dbce-4aef-a6df-f01c6fea0513", "whoami": 0, "id": 7481, "want_state": "up:active", "state": "up:active", "rank_uptime": 10.883951570000001, "mdsmap_epoch": 165, "osdmap_epoch": 189, "osdmap_epoch_barrier": 189, "uptime": 16.533595837 } 2021-02-10T11:39:41.268 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2021-02-10T11:39:41.268 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2021-02-10T11:39:41.268 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@gibba002.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2021-02-10T11:39:41.268 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:39:41.269 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec mkdir -p /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:39:41.605 DEBUG:teuthology.orchestra.run.gibba002:> sudo modprobe fuse 2021-02-10T11:39:41.951 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:39:41.951 DEBUG:teuthology.orchestra.run.gibba002:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:39:42.295 INFO:teuthology.orchestra.run.gibba002.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:39:42.295 INFO:teuthology.orchestra.run.gibba002.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:39:42.296 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:39:42.296 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:39:42.296 DEBUG:teuthology.orchestra.run.gibba002:> ls /sys/fs/fuse/connections 2021-02-10T11:39:42.631 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2021-02-10T11:39:42.632 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:39:42.634 DEBUG:teuthology.orchestra.run.gibba002:> sudo modprobe fuse 2021-02-10T11:39:42.977 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:39:42.978 DEBUG:teuthology.orchestra.run.gibba002:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:39:43.045 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba002.stderr:ceph-fuse[216940]: starting ceph client 2021-02-10T11:39:43.045 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba002.stderr:2021-02-10 11:39:43.042 7fdeb87bbf80 -1 init, newargv = 0x55c0c6fb21a0 newargc=9 2021-02-10T11:39:43.050 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba002.stderr:ceph-fuse[216940]: starting fuse 2021-02-10T11:39:43.321 INFO:teuthology.orchestra.run.gibba002.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:39:43.321 INFO:teuthology.orchestra.run.gibba002.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:39:43.322 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:39:43.322 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:39:43.322 DEBUG:teuthology.orchestra.run.gibba002:> ls /sys/fs/fuse/connections 2021-02-10T11:39:43.657 INFO:teuthology.orchestra.run.gibba002.stdout:43 2021-02-10T11:39:43.658 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2021-02-10T11:39:43.658 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:39:43.658 DEBUG:teuthology.orchestra.run.gibba002:> sudo python3 -c ' 2021-02-10T11:39:43.658 DEBUG:teuthology.orchestra.run.gibba002:> import glob 2021-02-10T11:39:43.658 DEBUG:teuthology.orchestra.run.gibba002:> import re 2021-02-10T11:39:43.658 DEBUG:teuthology.orchestra.run.gibba002:> import os 2021-02-10T11:39:43.658 DEBUG:teuthology.orchestra.run.gibba002:> import subprocess 2021-02-10T11:39:43.658 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:39:43.658 DEBUG:teuthology.orchestra.run.gibba002:> def find_socket(client_name): 2021-02-10T11:39:43.659 DEBUG:teuthology.orchestra.run.gibba002:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2021-02-10T11:39:43.659 DEBUG:teuthology.orchestra.run.gibba002:> files = glob.glob(asok_path) 2021-02-10T11:39:43.659 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:39:43.659 DEBUG:teuthology.orchestra.run.gibba002:> # Given a non-glob path, it better be there 2021-02-10T11:39:43.659 DEBUG:teuthology.orchestra.run.gibba002:> if "*" not in asok_path: 2021-02-10T11:39:43.659 DEBUG:teuthology.orchestra.run.gibba002:> assert(len(files) == 1) 2021-02-10T11:39:43.659 DEBUG:teuthology.orchestra.run.gibba002:> return files[0] 2021-02-10T11:39:43.659 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:39:43.659 DEBUG:teuthology.orchestra.run.gibba002:> for f in files: 2021-02-10T11:39:43.659 DEBUG:teuthology.orchestra.run.gibba002:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2021-02-10T11:39:43.659 DEBUG:teuthology.orchestra.run.gibba002:> if os.path.exists("/proc/{0}".format(pid)): 2021-02-10T11:39:43.659 DEBUG:teuthology.orchestra.run.gibba002:> return f 2021-02-10T11:39:43.659 DEBUG:teuthology.orchestra.run.gibba002:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2021-02-10T11:39:43.659 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:39:43.659 DEBUG:teuthology.orchestra.run.gibba002:> print(find_socket("client.0")) 2021-02-10T11:39:43.660 DEBUG:teuthology.orchestra.run.gibba002:> ' 2021-02-10T11:39:44.022 INFO:teuthology.orchestra.run.gibba002.stdout:/var/run/ceph/ceph-client.0.216940.asok 2021-02-10T11:39:44.025 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.216940.asok 2021-02-10T11:39:44.025 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:39:44.025 DEBUG:teuthology.orchestra.run.gibba002:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.216940.asok status 2021-02-10T11:39:44.416 INFO:teuthology.orchestra.run.gibba002.stdout:{ 2021-02-10T11:39:44.417 INFO:teuthology.orchestra.run.gibba002.stdout: "metadata": { 2021-02-10T11:39:44.417 INFO:teuthology.orchestra.run.gibba002.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:39:44.417 INFO:teuthology.orchestra.run.gibba002.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:39:44.417 INFO:teuthology.orchestra.run.gibba002.stdout: "entity_id": "0", 2021-02-10T11:39:44.417 INFO:teuthology.orchestra.run.gibba002.stdout: "hostname": "gibba002", 2021-02-10T11:39:44.417 INFO:teuthology.orchestra.run.gibba002.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2021-02-10T11:39:44.417 INFO:teuthology.orchestra.run.gibba002.stdout: "pid": "216940", 2021-02-10T11:39:44.417 INFO:teuthology.orchestra.run.gibba002.stdout: "root": "/" 2021-02-10T11:39:44.417 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:39:44.417 INFO:teuthology.orchestra.run.gibba002.stdout: "dentry_count": 0, 2021-02-10T11:39:44.417 INFO:teuthology.orchestra.run.gibba002.stdout: "dentry_pinned_count": 0, 2021-02-10T11:39:44.418 INFO:teuthology.orchestra.run.gibba002.stdout: "id": 13641, 2021-02-10T11:39:44.418 INFO:teuthology.orchestra.run.gibba002.stdout: "inst": { 2021-02-10T11:39:44.418 INFO:teuthology.orchestra.run.gibba002.stdout: "name": { 2021-02-10T11:39:44.418 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "client", 2021-02-10T11:39:44.418 INFO:teuthology.orchestra.run.gibba002.stdout: "num": 13641 2021-02-10T11:39:44.418 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:39:44.418 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": { 2021-02-10T11:39:44.418 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:39:44.418 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:0", 2021-02-10T11:39:44.418 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 211065895 2021-02-10T11:39:44.418 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:39:44.418 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:39:44.418 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": { 2021-02-10T11:39:44.419 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:39:44.419 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:0", 2021-02-10T11:39:44.419 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 211065895 2021-02-10T11:39:44.419 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:39:44.419 INFO:teuthology.orchestra.run.gibba002.stdout: "inst_str": "client.13641 172.21.2.102:0/211065895", 2021-02-10T11:39:44.419 INFO:teuthology.orchestra.run.gibba002.stdout: "addr_str": "172.21.2.102:0/211065895", 2021-02-10T11:39:44.419 INFO:teuthology.orchestra.run.gibba002.stdout: "inode_count": 1, 2021-02-10T11:39:44.419 INFO:teuthology.orchestra.run.gibba002.stdout: "mds_epoch": 165, 2021-02-10T11:39:44.419 INFO:teuthology.orchestra.run.gibba002.stdout: "osd_epoch": 189, 2021-02-10T11:39:44.419 INFO:teuthology.orchestra.run.gibba002.stdout: "osd_epoch_barrier": 0, 2021-02-10T11:39:44.419 INFO:teuthology.orchestra.run.gibba002.stdout: "blacklisted": false 2021-02-10T11:39:44.419 INFO:teuthology.orchestra.run.gibba002.stdout:} 2021-02-10T11:39:44.420 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:39:44.421 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:39:44.421 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:39:44.757 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:39:44.758 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:39:44.759 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:39:44.759 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:39:45.105 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:39:45.534 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:45.515 7f8608a42700 1 Processor -- start 2021-02-10T11:39:45.534 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:45.515 7f8608a42700 1 -- start start 2021-02-10T11:39:45.534 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:45.516 7f8608a42700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f86040d0340 0x7f86040ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:45.534 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:45.516 7f8608a42700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f86040d1c70 0x7f86040efc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:45.534 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:45.516 7f8608a42700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86040f0b10 con 0x7f86040d1870 2021-02-10T11:39:45.534 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:45.516 7f8608a42700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86040e5ed0 con 0x7f86040d0340 2021-02-10T11:39:45.535 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:45.516 7f8608a42700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86041b0be0 con 0x7f86040d1c70 2021-02-10T11:39:45.535 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:45.516 7f860205d700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f86040d0340 0x7f86040ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:45.535 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:45.516 7f860205d700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f86040d0340 0x7f86040ef640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:40630/0 (socket says 172.21.2.102:40630) 2021-02-10T11:39:45.535 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:45.516 7f860205d700 1 -- 172.21.2.102:0/931151641 learned_addr learned my addr 172.21.2.102:0/931151641 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:39:45.535 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:45.516 7f860205d700 1 -- 172.21.2.102:0/931151641 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f86040d1c70 msgr2=0x7f86040efc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:45.535 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:45.516 7f860205d700 1 --2- 172.21.2.102:0/931151641 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f86040d1c70 0x7f86040efc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:45.535 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:45.516 7f860205d700 1 -- 172.21.2.102:0/931151641 >> v1:172.21.2.113:6789/0 conn(0x7f86040d1870 legacy=0x7f86041b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:45.535 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:45.516 7f860205d700 1 -- 172.21.2.102:0/931151641 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86040f20a0 con 0x7f86040d0340 2021-02-10T11:39:45.535 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:45.516 7f860205d700 1 --2- 172.21.2.102:0/931151641 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f86040d0340 0x7f86040ef640 secure :-1 s=READY pgs=3272 cs=0 l=1 rev1=1 rx=0x7f85f4006700 tx=0x7f85f4009b80).ready entity=mon.0 client_cookie=50bf3e1db4126673 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:45.535 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:45.517 7f860185c700 1 -- 172.21.2.102:0/931151641 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f85f400c0c0 con 0x7f86040d0340 2021-02-10T11:39:45.535 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:45.517 7f860185c700 1 -- 172.21.2.102:0/931151641 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85f400c3c0 con 0x7f86040d0340 2021-02-10T11:39:45.535 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:45.517 7f860185c700 1 -- 172.21.2.102:0/931151641 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f85f400c8e0 con 0x7f86040d0340 2021-02-10T11:39:45.535 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:45.517 7f8608a42700 1 -- 172.21.2.102:0/931151641 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f86040d0340 msgr2=0x7f86040ef640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:45.536 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:45.517 7f8608a42700 1 --2- 172.21.2.102:0/931151641 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f86040d0340 0x7f86040ef640 secure :-1 s=READY pgs=3272 cs=0 l=1 rev1=1 rx=0x7f85f4006700 tx=0x7f85f4009b80).stop 2021-02-10T11:39:45.536 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:45.517 7f8608a42700 1 -- 172.21.2.102:0/931151641 shutdown_connections 2021-02-10T11:39:45.536 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:45.517 7f8608a42700 1 --2- 172.21.2.102:0/931151641 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f86040d1c70 0x7f86040efc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:45.536 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:45.517 7f8608a42700 1 --2- 172.21.2.102:0/931151641 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f86040d0340 0x7f86040ef640 unknown :-1 s=CLOSED pgs=3272 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:45.536 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:45.517 7f8608a42700 1 -- 172.21.2.102:0/931151641 >> 172.21.2.102:0/931151641 conn(0x7f86040ce450 msgr2=0x7f86040d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:45.536 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:45.518 7f8608a42700 1 -- 172.21.2.102:0/931151641 shutdown_connections 2021-02-10T11:39:45.536 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:45.518 7f8608a42700 1 -- 172.21.2.102:0/931151641 wait complete. 2021-02-10T11:39:45.538 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:45.532 7f8608a42700 1 Processor -- start 2021-02-10T11:39:45.538 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:45.532 7f8608a42700 1 -- start start 2021-02-10T11:39:45.538 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:45.533 7f8608a42700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f86040b0ad0 0x7f8604091f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:45.539 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:45.533 7f8608a42700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f86041a8850 0x7f86041a8c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:45.539 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:45.533 7f8608a42700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f86041a6fa0 0x7f86041a73a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:45.539 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:45.533 7f8608a42700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86041abfe0 con 0x7f86041a8850 2021-02-10T11:39:45.539 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:45.533 7f8608a42700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86041a9c40 con 0x7f86040b0ad0 2021-02-10T11:39:45.539 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:45.533 7f8608a42700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86041a9730 con 0x7f86041a6fa0 2021-02-10T11:39:45.539 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:45.533 7f860205d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f86041a8850 0x7f86041a8c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:45.539 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:45.533 7f860205d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f86041a8850 0x7f86041a8c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:40636/0 (socket says 172.21.2.102:40636) 2021-02-10T11:39:45.539 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:45.533 7f860205d700 1 -- 172.21.2.102:0/4149394325 learned_addr learned my addr 172.21.2.102:0/4149394325 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:39:45.539 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:45.533 7f860285e700 1 --2- 172.21.2.102:0/4149394325 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f86040b0ad0 0x7f8604091f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:45.540 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:45.533 7f860305f700 1 --2- 172.21.2.102:0/4149394325 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f86041a6fa0 0x7f86041a73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:45.540 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:45.533 7f860205d700 1 -- 172.21.2.102:0/4149394325 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f86041a6fa0 msgr2=0x7f86041a73a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:45.540 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:45.533 7f860205d700 1 --2- 172.21.2.102:0/4149394325 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f86041a6fa0 0x7f86041a73a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:45.540 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:45.533 7f860205d700 1 -- 172.21.2.102:0/4149394325 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f86040b0ad0 msgr2=0x7f8604091f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:45.540 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:45.533 7f860205d700 1 --2- 172.21.2.102:0/4149394325 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f86040b0ad0 0x7f8604091f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:45.540 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:45.533 7f860205d700 1 -- 172.21.2.102:0/4149394325 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86041a9270 con 0x7f86041a8850 2021-02-10T11:39:45.540 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:45.533 7f860205d700 1 --2- 172.21.2.102:0/4149394325 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f86041a8850 0x7f86041a8c50 secure :-1 s=READY pgs=3273 cs=0 l=1 rev1=1 rx=0x7f85f400b650 tx=0x7f85f400b9f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:45.540 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:45.534 7f85fb7fe700 1 -- 172.21.2.102:0/4149394325 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f85f400bde0 con 0x7f86041a8850 2021-02-10T11:39:45.540 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:45.534 7f85fb7fe700 1 -- 172.21.2.102:0/4149394325 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85f4002030 con 0x7f86041a8850 2021-02-10T11:39:45.540 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:45.534 7f85fb7fe700 1 -- 172.21.2.102:0/4149394325 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f85f4002550 con 0x7f86041a8850 2021-02-10T11:39:45.540 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:45.534 7f8608a42700 1 -- 172.21.2.102:0/4149394325 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86041a9270 con 0x7f86041a8850 2021-02-10T11:39:45.541 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:45.534 7f8608a42700 1 -- 172.21.2.102:0/4149394325 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86041a7cb0 con 0x7f86041a8850 2021-02-10T11:39:45.541 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:45.535 7f85fb7fe700 1 -- 172.21.2.102:0/4149394325 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f85f4000c90 con 0x7f86041a8850 2021-02-10T11:39:45.541 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:45.535 7f85fb7fe700 1 --2- 172.21.2.102:0/4149394325 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f85e80186c0 0x7f85e801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:45.541 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:45.535 7f85fb7fe700 1 -- 172.21.2.102:0/4149394325 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(189..189 src has 1..189) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f85f4032590 con 0x7f86041a8850 2021-02-10T11:39:45.541 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:45.535 7f860285e700 1 --2- 172.21.2.102:0/4149394325 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f85e80186c0 0x7f85e801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:45.541 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:45.535 7f860285e700 1 --2- 172.21.2.102:0/4149394325 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f85e80186c0 0x7f85e801abc0 crc :-1 s=READY pgs=1429 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:45.568 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:45.565 7f8608a42700 1 -- 172.21.2.102:0/4149394325 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86040ce450 con 0x7f86041a8850 2021-02-10T11:39:45.570 DEBUG:teuthology.orchestra.run.gibba002:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:39:45.571 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:45.568 7f85fb7fe700 1 -- 172.21.2.102:0/4149394325 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f85f4008a80 con 0x7f86041a8850 2021-02-10T11:39:45.573 DEBUG:teuthology.orchestra.run.gibba013:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:39:45.698 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:45.696 7f8608a42700 1 -- 172.21.2.102:0/4149394325 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f86040ce450 con 0x7f86041a8850 2021-02-10T11:39:45.699 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:45.697 7f85fb7fe700 1 -- 172.21.2.102:0/4149394325 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v189) v1 ==== 74+0+14007 (secure 0 0 0) 0x7f85f4008a80 con 0x7f86041a8850 2021-02-10T11:39:45.700 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:39:45.700 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":189,"fsid":"68db30ea-dbce-4aef-a6df-f01c6fea0513","created":"2021-02-10 11:03:06.331557","modified":"2021-02-10 11:39:31.377604","last_up_change":"2021-02-10 11:03:32.690562","last_in_change":"2021-02-10 11:03:32.690562","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":41,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:03:37.322519","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":40,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:39:28.340428","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"188","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":41,"pool_name":"cephfs_data","create_time":"2021-02-10 11:39:29.367624","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"189","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"be691b4d-a039-408a-af80-9d53f02b6f1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":185,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6818","nonce":36088},{"type":"v1","addr":"172.21.2.102:6820","nonce":36088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6822","nonce":36088},{"type":"v1","addr":"172.21.2.102:6824","nonce":36088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6830","nonce":36088},{"type":"v1","addr":"172.21.2.102:6832","nonce":36088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6826","nonce":36088},{"type":"v1","addr":"172.21.2.102:6828","nonce":36088}]},"public_addr":"172.21.2.102:6820/36088","cluster_addr":"172.21.2.102:6824/36088","heartbeat_back_addr":"172.21.2.102:6832/36088","heartbeat_front_addr":"172.21.2.102:6828/36088","state":["exists","up"]},{"osd":1,"uuid":"64e3ccc8-9c69-4d7c-865e-af8e83822635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":185,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6819","nonce":36087},{"type":"v1","addr":"172.21.2.102:6821","nonce":36087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6823","nonce":36087},{"type":"v1","addr":"172.21.2.102:6825","nonce":36087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6831","nonce":36087},{"type":"v1","addr":"172.21.2.102:6833","nonce":36087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6827","nonce":36087},{"type":"v1","addr":"172.21.2.102:6829","nonce":36087}]},"public_addr":"172.21.2.102:6821/36087","cluster_addr":"172.21.2.102:6825/36087","heartbeat_back_addr":"172.21.2.102:6833/36087","heartbeat_front_addr":"172.21.2.102:6829/36087","state":["exists","up"]},{"osd":2,"uuid":"153700af-747f-481f-bdf4-d6072d3a842f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":187,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6810","nonce":36089},{"type":"v1","addr":"172.21.2.102:6811","nonce":36089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6812","nonce":36089},{"type":"v1","addr":"172.21.2.102:6813","nonce":36089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6816","nonce":36089},{"type":"v1","addr":"172.21.2.102:6817","nonce":36089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6814","nonce":36089},{"type":"v1","addr":"172.21.2.102:6815","nonce":36089}]},"public_addr":"172.21.2.102:6811/36089","cluster_addr":"172.21.2.102:6813/36089","heartbeat_back_addr":"172.21.2.102:6817/36089","heartbeat_front_addr":"172.21.2.102:6815/36089","state":["exists","up"]},{"osd":3,"uuid":"cbc678f4-83a9-45e6-8830-2f7c232b1ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":187,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6802","nonce":36090},{"type":"v1","addr":"172.21.2.102:6803","nonce":36090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6804","nonce":36090},{"type":"v1","addr":"172.21.2.102:6805","nonce":36090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6808","nonce":36090},{"type":"v1","addr":"172.21.2.102:6809","nonce":36090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6806","nonce":36090},{"type":"v1","addr":"172.21.2.102:6807","nonce":36090}]},"public_addr":"172.21.2.102:6803/36090","cluster_addr":"172.21.2.102:6805/36090","heartbeat_back_addr":"172.21.2.102:6809/36090","heartbeat_front_addr":"172.21.2.102:6807/36090","state":["exists","up"]},{"osd":4,"uuid":"c01e4008-b258-4f10-afd2-9e7f37214351","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":187,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6808","nonce":36368},{"type":"v1","addr":"172.21.2.113:6811","nonce":36368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6814","nonce":36368},{"type":"v1","addr":"172.21.2.113:6818","nonce":36368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6826","nonce":36368},{"type":"v1","addr":"172.21.2.113:6828","nonce":36368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6820","nonce":36368},{"type":"v1","addr":"172.21.2.113:6824","nonce":36368}]},"public_addr":"172.21.2.113:6811/36368","cluster_addr":"172.21.2.113:6818/36368","heartbeat_back_addr":"172.21.2.113:6828/36368","heartbeat_front_addr":"172.21.2.113:6824/36368","state":["exists","up"]},{"osd":5,"uuid":"551f6007-daf8-4a98-8ee0-c2b16bcccb5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":178,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6802","nonce":36366},{"type":"v1","addr":"172.21.2.113:6805","nonce":36366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6807","nonce":36366},{"type":"v1","addr":"172.21.2.113:6810","nonce":36366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6819","nonce":36366},{"type":"v1","addr":"172.21.2.113:6822","nonce":36366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6812","nonce":36366},{"type":"v1","addr":"172.21.2.113:6815","nonce":36366}]},"public_addr":"172.21.2.113:6805/36366","cluster_addr":"172.21.2.113:6810/36366","heartbeat_back_addr":"172.21.2.113:6822/36366","heartbeat_front_addr":"172.21.2.113:6815/36366","state":["exists","up"]},{"osd":6,"uuid":"bf67c040-0e50-48be-b102-c3943bc9ebef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":187,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6800","nonce":36367},{"type":"v1","addr":"172.21.2.113:6801","nonce":36367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6803","nonce":36367},{"type":"v1","addr":"172.21.2.113:6804","nonce":36367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6813","nonce":36367},{"type":"v1","addr":"172.21.2.113:6816","nonce":36367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6806","nonce":36367},{"type":"v1","addr":"172.21.2.113:6809","nonce":36367}]},"public_addr":"172.21.2.113:6801/36367","cluster_addr":"172.21.2.113:6804/36367","heartbeat_back_addr":"172.21.2.113:6816/36367","heartbeat_front_addr":"172.21.2.113:6809/36367","state":["exists","up"]},{"osd":7,"uuid":"f5461f80-018d-428f-be89-2eb2bcbe7bb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":187,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6817","nonce":36365},{"type":"v1","addr":"172.21.2.113:6821","nonce":36365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6823","nonce":36365},{"type":"v1","addr":"172.21.2.113:6825","nonce":36365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6830","nonce":36365},{"type":"v1","addr":"172.21.2.113:6831","nonce":36365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6827","nonce":36365},{"type":"v1","addr":"172.21.2.113:6829","nonce":36365}]},"public_addr":"172.21.2.113:6821/36365","cluster_addr":"172.21.2.113:6825/36365","heartbeat_back_addr":"172.21.2.113:6831/36365","heartbeat_front_addr":"172.21.2.113:6829/36365","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:39:45.701 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:45.698 7f8608a42700 1 -- 172.21.2.102:0/4149394325 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f85e80186c0 msgr2=0x7f85e801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:45.701 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:45.698 7f8608a42700 1 --2- 172.21.2.102:0/4149394325 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f85e80186c0 0x7f85e801abc0 crc :-1 s=READY pgs=1429 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:45.701 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:45.698 7f8608a42700 1 -- 172.21.2.102:0/4149394325 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f86041a8850 msgr2=0x7f86041a8c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:45.701 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:45.698 7f8608a42700 1 --2- 172.21.2.102:0/4149394325 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f86041a8850 0x7f86041a8c50 secure :-1 s=READY pgs=3273 cs=0 l=1 rev1=1 rx=0x7f85f400b650 tx=0x7f85f400b9f0).stop 2021-02-10T11:39:45.701 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:45.699 7f8608a42700 1 -- 172.21.2.102:0/4149394325 shutdown_connections 2021-02-10T11:39:45.701 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:45.699 7f8608a42700 1 --2- 172.21.2.102:0/4149394325 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f86041a6fa0 0x7f86041a73a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:45.701 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:45.699 7f8608a42700 1 --2- 172.21.2.102:0/4149394325 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f86041a8850 0x7f86041a8c50 unknown :-1 s=CLOSED pgs=3273 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:45.701 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:45.699 7f8608a42700 1 --2- 172.21.2.102:0/4149394325 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f85e80186c0 0x7f85e801abc0 unknown :-1 s=CLOSED pgs=1429 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:45.702 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:45.699 7f8608a42700 1 --2- 172.21.2.102:0/4149394325 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f86040b0ad0 0x7f8604091f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:45.702 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:45.699 7f8608a42700 1 -- 172.21.2.102:0/4149394325 >> 172.21.2.102:0/4149394325 conn(0x7f860419f8f0 msgr2=0x7f86041a5f80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:45.702 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:45.699 7f8608a42700 1 -- 172.21.2.102:0/4149394325 shutdown_connections 2021-02-10T11:39:45.702 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:45.699 7f8608a42700 1 -- 172.21.2.102:0/4149394325 wait complete. 2021-02-10T11:39:45.709 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:03:37.322519', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 40, 'pool_name': 'cephfs_metadata', 'create_time': '2021-02-10 11:39:28.340428', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '188', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 41, 'pool_name': 'cephfs_data', 'create_time': '2021-02-10 11:39:29.367624', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '189', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-02-10T11:39:45.709 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:39:46.141 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:46.123 7fc39ec9d700 1 Processor -- start 2021-02-10T11:39:46.141 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:46.123 7fc39ec9d700 1 -- start start 2021-02-10T11:39:46.142 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:46.123 7fc39ec9d700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fc3980d1870 0x7fc3981b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:46.142 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:46.123 7fc39ec9d700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fc3981b0ae0 0x7fc398108940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:46.142 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:46.123 7fc39ec9d700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc3980d1c70 0x7fc398108f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:46.142 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:46.123 7fc39ec9d700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3980fe1c0 con 0x7fc3981b0ae0 2021-02-10T11:39:46.142 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:46.123 7fc39ec9d700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc398109a60 con 0x7fc3980d1870 2021-02-10T11:39:46.142 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:46.123 7fc39ec9d700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc398088070 con 0x7fc3980d1c70 2021-02-10T11:39:46.143 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:46.123 7fc397fff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fc3981b0ae0 0x7fc398108940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:46.143 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:46.123 7fc397fff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fc3981b0ae0 0x7fc398108940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:40642/0 (socket says 172.21.2.102:40642) 2021-02-10T11:39:46.143 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:46.123 7fc397fff700 1 -- 172.21.2.102:0/3655149285 learned_addr learned my addr 172.21.2.102:0/3655149285 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:39:46.143 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:46.124 7fc397fff700 1 -- 172.21.2.102:0/3655149285 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc3980d1c70 msgr2=0x7fc398108f60 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:39:46.143 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:46.124 7fc397fff700 1 --2- 172.21.2.102:0/3655149285 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc3980d1c70 0x7fc398108f60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:46.143 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:46.124 7fc397fff700 1 -- 172.21.2.102:0/3655149285 >> v2:172.21.2.113:3300/0 conn(0x7fc3980d1870 msgr2=0x7fc3981b04c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:39:46.143 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:46.124 7fc397fff700 1 --2- 172.21.2.102:0/3655149285 >> v2:172.21.2.113:3300/0 conn(0x7fc3980d1870 0x7fc3981b04c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:46.144 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:46.124 7fc397fff700 1 -- 172.21.2.102:0/3655149285 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc398109580 con 0x7fc3981b0ae0 2021-02-10T11:39:46.144 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:46.124 7fc397fff700 1 --2- 172.21.2.102:0/3655149285 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc3981b0ae0 0x7fc398108940 secure :-1 s=READY pgs=3274 cs=0 l=1 rev1=1 rx=0x7fc388006700 tx=0x7fc388009b80).ready entity=mon.0 client_cookie=accdd520c791fb56 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:46.144 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:46.124 7fc3977fe700 1 -- 172.21.2.102:0/3655149285 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc38800c0c0 con 0x7fc3981b0ae0 2021-02-10T11:39:46.144 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:46.124 7fc3977fe700 1 -- 172.21.2.102:0/3655149285 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc38800c400 con 0x7fc3981b0ae0 2021-02-10T11:39:46.144 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:46.124 7fc3977fe700 1 -- 172.21.2.102:0/3655149285 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc38800c8e0 con 0x7fc3981b0ae0 2021-02-10T11:39:46.144 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:46.124 7fc39ec9d700 1 -- 172.21.2.102:0/3655149285 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc3981b0ae0 msgr2=0x7fc398108940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:46.144 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:46.124 7fc39ec9d700 1 --2- 172.21.2.102:0/3655149285 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc3981b0ae0 0x7fc398108940 secure :-1 s=READY pgs=3274 cs=0 l=1 rev1=1 rx=0x7fc388006700 tx=0x7fc388009b80).stop 2021-02-10T11:39:46.145 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:46.124 7fc39ec9d700 1 -- 172.21.2.102:0/3655149285 shutdown_connections 2021-02-10T11:39:46.145 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:46.124 7fc39ec9d700 1 --2- 172.21.2.102:0/3655149285 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc3980d1c70 0x7fc398108f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:46.145 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:46.124 7fc39ec9d700 1 --2- 172.21.2.102:0/3655149285 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc3981b0ae0 0x7fc398108940 unknown :-1 s=CLOSED pgs=3274 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:46.145 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:46.124 7fc39ec9d700 1 --2- 172.21.2.102:0/3655149285 >> v2:172.21.2.113:3300/0 conn(0x7fc3980d1870 0x7fc3981b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:46.145 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:46.124 7fc39ec9d700 1 -- 172.21.2.102:0/3655149285 >> 172.21.2.102:0/3655149285 conn(0x7fc3980ce420 msgr2=0x7fc3980d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:46.145 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:46.125 7fc39ec9d700 1 -- 172.21.2.102:0/3655149285 shutdown_connections 2021-02-10T11:39:46.145 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:46.125 7fc39ec9d700 1 -- 172.21.2.102:0/3655149285 wait complete. 2021-02-10T11:39:46.145 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:46.139 7fc39ec9d700 1 Processor -- start 2021-02-10T11:39:46.146 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:46.139 7fc39ec9d700 1 -- start start 2021-02-10T11:39:46.148 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:46.139 7fc39ec9d700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc3980c3820 0x7fc398084cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:46.149 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:46.139 7fc39ec9d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc3980a0290 0x7fc3980852d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:46.149 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:46.139 7fc39ec9d700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc398086270 0x7fc398086670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:46.149 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:46.139 7fc39ec9d700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3980c0700 con 0x7fc3980a0290 2021-02-10T11:39:46.149 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:46.139 7fc39ec9d700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3980858f0 con 0x7fc3980c3820 2021-02-10T11:39:46.149 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:46.139 7fc39ec9d700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc398085b10 con 0x7fc398086270 2021-02-10T11:39:46.149 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:46.140 7fc397fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc3980a0290 0x7fc3980852d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:46.149 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:46.140 7fc397fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc3980a0290 0x7fc3980852d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:40648/0 (socket says 172.21.2.102:40648) 2021-02-10T11:39:46.150 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:46.140 7fc397fff700 1 -- 172.21.2.102:0/363235941 learned_addr learned my addr 172.21.2.102:0/363235941 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:39:46.150 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:46.140 7fc397fff700 1 -- 172.21.2.102:0/363235941 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc398086270 msgr2=0x7fc398086670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:46.150 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:46.140 7fc397fff700 1 --2- 172.21.2.102:0/363235941 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc398086270 0x7fc398086670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:46.150 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:46.140 7fc397fff700 1 -- 172.21.2.102:0/363235941 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc3980c3820 msgr2=0x7fc398084cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:46.150 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:46.140 7fc39cab9700 1 --2- 172.21.2.102:0/363235941 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc3980c3820 0x7fc398084cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:46.150 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:46.140 7fc397fff700 1 --2- 172.21.2.102:0/363235941 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc3980c3820 0x7fc398084cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:46.150 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:46.140 7fc397fff700 1 -- 172.21.2.102:0/363235941 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc398086c90 con 0x7fc3980a0290 2021-02-10T11:39:46.150 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:46.140 7fc39cab9700 1 --2- 172.21.2.102:0/363235941 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc3980c3820 0x7fc398084cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:39:46.150 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:46.140 7fc397fff700 1 --2- 172.21.2.102:0/363235941 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc3980a0290 0x7fc3980852d0 secure :-1 s=READY pgs=3275 cs=0 l=1 rev1=1 rx=0x7fc38800b580 tx=0x7fc38800b920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:46.150 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:46.141 7fc3957fa700 1 -- 172.21.2.102:0/363235941 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc38800bd10 con 0x7fc3980a0290 2021-02-10T11:39:46.150 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:46.141 7fc3957fa700 1 -- 172.21.2.102:0/363235941 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3880052e0 con 0x7fc3980a0290 2021-02-10T11:39:46.150 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:46.141 7fc3957fa700 1 -- 172.21.2.102:0/363235941 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc388005800 con 0x7fc3980a0290 2021-02-10T11:39:46.151 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:46.141 7fc39ec9d700 1 -- 172.21.2.102:0/363235941 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc398086c90 con 0x7fc3980a0290 2021-02-10T11:39:46.151 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:46.141 7fc39ec9d700 1 -- 172.21.2.102:0/363235941 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc398086c90 con 0x7fc3980a0290 2021-02-10T11:39:46.151 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:46.141 7fc3957fa700 1 -- 172.21.2.102:0/363235941 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fc388006030 con 0x7fc3980a0290 2021-02-10T11:39:46.151 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:46.141 7fc3957fa700 1 --2- 172.21.2.102:0/363235941 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc37c0186c0 0x7fc37c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:46.151 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:46.141 7fc3957fa700 1 -- 172.21.2.102:0/363235941 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(189..189 src has 1..189) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fc388032460 con 0x7fc3980a0290 2021-02-10T11:39:46.151 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:46.142 7fc39cab9700 1 --2- 172.21.2.102:0/363235941 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc37c0186c0 0x7fc37c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:46.151 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:46.142 7fc39cab9700 1 --2- 172.21.2.102:0/363235941 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc37c0186c0 0x7fc37c01abc0 crc :-1 s=READY pgs=1430 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:46.175 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:46.173 7fc39ec9d700 1 -- 172.21.2.102:0/363235941 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc398082a70 con 0x7fc3980a0290 2021-02-10T11:39:46.177 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:46.175 7fc3957fa700 1 -- 172.21.2.102:0/363235941 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fc388018cf0 con 0x7fc3980a0290 2021-02-10T11:39:46.307 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:46.305 7fc39ec9d700 1 -- 172.21.2.102:0/363235941 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7fc398082a70 con 0x7fc3980a0290 2021-02-10T11:39:46.308 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:46.305 7fc3957fa700 1 -- 172.21.2.102:0/363235941 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v189) v1 ==== 92+0+10 (secure 0 0 0) 0x7fc388018cf0 con 0x7fc3980a0290 2021-02-10T11:39:46.308 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:39:46.310 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:46.307 7fc39ec9d700 1 -- 172.21.2.102:0/363235941 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc37c0186c0 msgr2=0x7fc37c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:46.310 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:46.307 7fc39ec9d700 1 --2- 172.21.2.102:0/363235941 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc37c0186c0 0x7fc37c01abc0 crc :-1 s=READY pgs=1430 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:46.310 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:46.307 7fc39ec9d700 1 -- 172.21.2.102:0/363235941 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc3980a0290 msgr2=0x7fc3980852d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:46.310 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:46.307 7fc39ec9d700 1 --2- 172.21.2.102:0/363235941 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc3980a0290 0x7fc3980852d0 secure :-1 s=READY pgs=3275 cs=0 l=1 rev1=1 rx=0x7fc38800b580 tx=0x7fc38800b920).stop 2021-02-10T11:39:46.311 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:46.308 7fc39ec9d700 1 -- 172.21.2.102:0/363235941 shutdown_connections 2021-02-10T11:39:46.311 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:46.308 7fc39ec9d700 1 --2- 172.21.2.102:0/363235941 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc398086270 0x7fc398086670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:46.311 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:46.308 7fc39ec9d700 1 --2- 172.21.2.102:0/363235941 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc3980a0290 0x7fc3980852d0 unknown :-1 s=CLOSED pgs=3275 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:46.311 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:46.308 7fc39ec9d700 1 --2- 172.21.2.102:0/363235941 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc37c0186c0 0x7fc37c01abc0 unknown :-1 s=CLOSED pgs=1430 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:46.311 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:46.308 7fc39ec9d700 1 --2- 172.21.2.102:0/363235941 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc3980c3820 0x7fc398084cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:46.311 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:46.308 7fc39ec9d700 1 -- 172.21.2.102:0/363235941 >> 172.21.2.102:0/363235941 conn(0x7fc3980872c0 msgr2=0x7fc398095700 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:46.311 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:46.308 7fc39ec9d700 1 -- 172.21.2.102:0/363235941 shutdown_connections 2021-02-10T11:39:46.312 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:46.308 7fc39ec9d700 1 -- 172.21.2.102:0/363235941 wait complete. 2021-02-10T11:39:46.319 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-02-10T11:39:46.751 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:46.731 7ff691bc5700 1 Processor -- start 2021-02-10T11:39:46.751 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:46.732 7ff691bc5700 1 -- start start 2021-02-10T11:39:46.751 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:46.732 7ff691bc5700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7ff68c0d1870 0x7ff68c1b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:46.751 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:46.732 7ff691bc5700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff68c0fe1c0 con 0x7ff68c1b0ae0 2021-02-10T11:39:46.751 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:46.732 7ff691bc5700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff68c109a60 con 0x7ff68c0cff70 2021-02-10T11:39:46.751 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:46.732 7ff691bc5700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff68c088070 con 0x7ff68c0d1870 2021-02-10T11:39:46.752 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:46.732 7ff68affd700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7ff68c1b0ae0 0x7ff68c108940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:32880/0 (socket says 172.21.2.102:32880) 2021-02-10T11:39:46.752 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:46.732 7ff68affd700 1 -- 172.21.2.102:0/927795019 learned_addr learned my addr 172.21.2.102:0/927795019 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:39:46.752 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:46.732 7ff68b7fe700 1 --2- 172.21.2.102:0/927795019 >> v2:172.21.2.113:3300/0 conn(0x7ff68c0d1870 0x7ff68c1b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:46.752 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:46.733 7ff68a7fc700 1 -- 172.21.2.102:0/927795019 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3564488350 0 0) 0x7ff67c0010b0 con 0x7ff68c1b0ae0 2021-02-10T11:39:46.752 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:46.733 7ff68a7fc700 1 -- 172.21.2.102:0/927795019 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff678002fe0 con 0x7ff68c1b0ae0 2021-02-10T11:39:46.752 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:46.733 7ff68b7fe700 1 -- 172.21.2.102:0/927795019 >> v1:172.21.2.113:6789/0 conn(0x7ff68c0cff70 legacy=0x7ff68c109130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:46.752 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:46.733 7ff68b7fe700 1 -- 172.21.2.102:0/927795019 >> v1:172.21.2.102:6789/0 conn(0x7ff68c1b0ae0 legacy=0x7ff68c108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:46.752 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:46.733 7ff68b7fe700 1 -- 172.21.2.102:0/927795019 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff68c088260 con 0x7ff68c0d1870 2021-02-10T11:39:46.752 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:46.733 7ff68b7fe700 1 --2- 172.21.2.102:0/927795019 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff68c0d1870 0x7ff68c1b04c0 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 rx=0x7ff674006650 tx=0x7ff674009a80).ready entity=mon.1 client_cookie=489b107b564305a0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:46.752 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:46.733 7ff68a7fc700 1 -- 172.21.2.102:0/927795019 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff6740009b0 con 0x7ff68c0d1870 2021-02-10T11:39:46.752 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:46.733 7ff68a7fc700 1 -- 172.21.2.102:0/927795019 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff67400c0c0 con 0x7ff68c0d1870 2021-02-10T11:39:46.752 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:46.733 7ff68a7fc700 1 -- 172.21.2.102:0/927795019 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff67400c5a0 con 0x7ff68c0d1870 2021-02-10T11:39:46.752 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:46.733 7ff691bc5700 1 -- 172.21.2.102:0/927795019 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff68c0d1870 msgr2=0x7ff68c1b04c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:46.753 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:46.733 7ff691bc5700 1 --2- 172.21.2.102:0/927795019 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff68c0d1870 0x7ff68c1b04c0 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 rx=0x7ff674006650 tx=0x7ff674009a80).stop 2021-02-10T11:39:46.753 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:46.734 7ff691bc5700 1 -- 172.21.2.102:0/927795019 shutdown_connections 2021-02-10T11:39:46.753 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:46.734 7ff691bc5700 1 --2- 172.21.2.102:0/927795019 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff68c0d1870 0x7ff68c1b04c0 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:46.753 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:46.734 7ff691bc5700 1 -- 172.21.2.102:0/927795019 >> 172.21.2.102:0/927795019 conn(0x7ff68c0ce420 msgr2=0x7ff68c0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:46.753 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:46.734 7ff691bc5700 1 -- 172.21.2.102:0/927795019 shutdown_connections 2021-02-10T11:39:46.753 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:46.734 7ff691bc5700 1 -- 172.21.2.102:0/927795019 wait complete. 2021-02-10T11:39:46.753 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:46.749 7ff691bc5700 1 Processor -- start 2021-02-10T11:39:46.753 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:46.749 7ff691bc5700 1 -- start start 2021-02-10T11:39:46.755 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:46.749 7ff691bc5700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff68c0c2370 0x7ff68c18f870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:46.755 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:46.749 7ff691bc5700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff68c084cb0 0x7ff68c1a7470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:46.755 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:46.749 7ff691bc5700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff68c1a7a90 0x7ff68c085700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:46.755 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:46.749 7ff691bc5700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff68c0be840 con 0x7ff68c084cb0 2021-02-10T11:39:46.755 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:46.749 7ff691bc5700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff68c0c29a0 con 0x7ff68c1a7a90 2021-02-10T11:39:46.755 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:46.749 7ff691bc5700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff68c1a7e90 con 0x7ff68c0c2370 2021-02-10T11:39:46.755 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:46.749 7ff68affd700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff68c084cb0 0x7ff68c1a7470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:46.755 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:46.749 7ff68affd700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff68c084cb0 0x7ff68c1a7470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:40664/0 (socket says 172.21.2.102:40664) 2021-02-10T11:39:46.755 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:46.749 7ff68affd700 1 -- 172.21.2.102:0/3664441953 learned_addr learned my addr 172.21.2.102:0/3664441953 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:39:46.755 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:46.749 7ff68bfff700 1 --2- 172.21.2.102:0/3664441953 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff68c1a7a90 0x7ff68c085700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:46.756 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:46.749 7ff68b7fe700 1 --2- 172.21.2.102:0/3664441953 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff68c0c2370 0x7ff68c18f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:46.756 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:46.750 7ff68affd700 1 -- 172.21.2.102:0/3664441953 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff68c0c2370 msgr2=0x7ff68c18f870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:46.756 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:46.750 7ff68affd700 1 --2- 172.21.2.102:0/3664441953 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff68c0c2370 0x7ff68c18f870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:46.756 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:46.750 7ff68affd700 1 -- 172.21.2.102:0/3664441953 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff68c1a7a90 msgr2=0x7ff68c085700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:46.756 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:46.750 7ff68affd700 1 --2- 172.21.2.102:0/3664441953 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff68c1a7a90 0x7ff68c085700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:46.756 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:46.750 7ff68affd700 1 -- 172.21.2.102:0/3664441953 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff68c1a80b0 con 0x7ff68c084cb0 2021-02-10T11:39:46.756 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:46.750 7ff68affd700 1 --2- 172.21.2.102:0/3664441953 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff68c084cb0 0x7ff68c1a7470 secure :-1 s=READY pgs=3277 cs=0 l=1 rev1=1 rx=0x7ff67c001c60 tx=0x7ff67c00a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:46.756 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:46.750 7ff683fff700 1 -- 172.21.2.102:0/3664441953 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff67c00acc0 con 0x7ff68c084cb0 2021-02-10T11:39:46.756 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:46.751 7ff683fff700 1 -- 172.21.2.102:0/3664441953 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff67c004350 con 0x7ff68c084cb0 2021-02-10T11:39:46.756 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:46.751 7ff683fff700 1 -- 172.21.2.102:0/3664441953 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff67c004830 con 0x7ff68c084cb0 2021-02-10T11:39:46.757 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:46.751 7ff691bc5700 1 -- 172.21.2.102:0/3664441953 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff68c1a80b0 con 0x7ff68c084cb0 2021-02-10T11:39:46.757 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:46.751 7ff691bc5700 1 -- 172.21.2.102:0/3664441953 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff68c1a80b0 con 0x7ff68c084cb0 2021-02-10T11:39:46.757 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:46.751 7ff683fff700 1 -- 172.21.2.102:0/3664441953 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ff67c004530 con 0x7ff68c084cb0 2021-02-10T11:39:46.757 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:46.751 7ff683fff700 1 --2- 172.21.2.102:0/3664441953 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff6700186c0 0x7ff67001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:46.757 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:46.751 7ff683fff700 1 -- 172.21.2.102:0/3664441953 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(189..189 src has 1..189) v4 ==== 5573+0+0 (secure 0 0 0) 0x7ff67c0327b0 con 0x7ff68c084cb0 2021-02-10T11:39:46.757 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:46.752 7ff68b7fe700 1 --2- 172.21.2.102:0/3664441953 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff6700186c0 0x7ff67001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:46.757 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:46.752 7ff68b7fe700 1 --2- 172.21.2.102:0/3664441953 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff6700186c0 0x7ff67001abc0 crc :-1 s=READY pgs=1431 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:46.784 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:46.782 7ff691bc5700 1 -- 172.21.2.102:0/3664441953 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff68c0be840 con 0x7ff68c084cb0 2021-02-10T11:39:46.786 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:46.784 7ff683fff700 1 -- 172.21.2.102:0/3664441953 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ff67c004140 con 0x7ff68c084cb0 2021-02-10T11:39:46.915 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:46.912 7ff691bc5700 1 -- 172.21.2.102:0/3664441953 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7ff68c0be840 con 0x7ff68c084cb0 2021-02-10T11:39:46.915 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:46.912 7ff683fff700 1 -- 172.21.2.102:0/3664441953 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v189) v1 ==== 104+0+10 (secure 0 0 0) 0x7ff67c00acc0 con 0x7ff68c084cb0 2021-02-10T11:39:46.915 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:39:46.917 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:46.914 7ff691bc5700 1 -- 172.21.2.102:0/3664441953 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff6700186c0 msgr2=0x7ff67001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:46.917 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:46.914 7ff691bc5700 1 --2- 172.21.2.102:0/3664441953 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff6700186c0 0x7ff67001abc0 crc :-1 s=READY pgs=1431 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:46.917 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:46.914 7ff691bc5700 1 -- 172.21.2.102:0/3664441953 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff68c084cb0 msgr2=0x7ff68c1a7470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:46.917 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:46.914 7ff691bc5700 1 --2- 172.21.2.102:0/3664441953 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff68c084cb0 0x7ff68c1a7470 secure :-1 s=READY pgs=3277 cs=0 l=1 rev1=1 rx=0x7ff67c001c60 tx=0x7ff67c00a8d0).stop 2021-02-10T11:39:46.917 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:46.915 7ff691bc5700 1 -- 172.21.2.102:0/3664441953 shutdown_connections 2021-02-10T11:39:46.917 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:46.915 7ff691bc5700 1 --2- 172.21.2.102:0/3664441953 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff6700186c0 0x7ff67001abc0 unknown :-1 s=CLOSED pgs=1431 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:46.917 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:46.915 7ff691bc5700 1 --2- 172.21.2.102:0/3664441953 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff68c1a7a90 0x7ff68c085700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:46.917 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:46.915 7ff691bc5700 1 --2- 172.21.2.102:0/3664441953 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff68c084cb0 0x7ff68c1a7470 unknown :-1 s=CLOSED pgs=3277 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:46.918 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:46.915 7ff691bc5700 1 --2- 172.21.2.102:0/3664441953 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff68c0c2370 0x7ff68c18f870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:46.918 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:46.915 7ff691bc5700 1 -- 172.21.2.102:0/3664441953 >> 172.21.2.102:0/3664441953 conn(0x7ff68c0b5dc0 msgr2=0x7ff68c0c2d90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:46.918 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:46.915 7ff691bc5700 1 -- 172.21.2.102:0/3664441953 shutdown_connections 2021-02-10T11:39:46.918 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:46.915 7ff691bc5700 1 -- 172.21.2.102:0/3664441953 wait complete. 2021-02-10T11:39:46.925 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-02-10T11:39:47.359 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:47.340 7f0d4be94700 1 Processor -- start 2021-02-10T11:39:47.360 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:47.340 7f0d4be94700 1 -- start start 2021-02-10T11:39:47.360 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:47.341 7f0d4be94700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f0d440d1870 0x7f0d441b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:47.360 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:47.341 7f0d4be94700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0d441b0ae0 0x7f0d44108940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:47.360 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:47.341 7f0d4be94700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d440fe1c0 con 0x7f0d440d1c70 2021-02-10T11:39:47.360 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:47.341 7f0d4be94700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d44109a60 con 0x7f0d440d1870 2021-02-10T11:39:47.360 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:47.341 7f0d4be94700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d44109750 con 0x7f0d441b0ae0 2021-02-10T11:39:47.360 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:47.341 7f0d4a4b1700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f0d440d1c70 0x7f0d44108f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:32892/0 (socket says 172.21.2.102:32892) 2021-02-10T11:39:47.360 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:47.341 7f0d4a4b1700 1 -- 172.21.2.102:0/871060919 learned_addr learned my addr 172.21.2.102:0/871060919 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:39:47.360 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:47.341 7f0d494af700 1 --2- 172.21.2.102:0/871060919 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0d441b0ae0 0x7f0d44108940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:47.360 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:47.341 7f0d49cb0700 1 --2- 172.21.2.102:0/871060919 >> v2:172.21.2.113:3300/0 conn(0x7f0d440d1870 0x7f0d441b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:47.360 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:47.341 7f0d48cae700 1 -- 172.21.2.102:0/871060919 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 469135729 0 0) 0x7f0d3c001500 con 0x7f0d440d1c70 2021-02-10T11:39:47.360 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:47.341 7f0d48cae700 1 -- 172.21.2.102:0/871060919 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d30002fe0 con 0x7f0d440d1c70 2021-02-10T11:39:47.361 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:47.342 7f0d494af700 1 -- 172.21.2.102:0/871060919 >> v2:172.21.2.113:3300/0 conn(0x7f0d440d1870 msgr2=0x7f0d441b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:47.361 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:47.342 7f0d494af700 1 --2- 172.21.2.102:0/871060919 >> v2:172.21.2.113:3300/0 conn(0x7f0d440d1870 0x7f0d441b04c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:47.361 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:47.342 7f0d494af700 1 -- 172.21.2.102:0/871060919 >> v1:172.21.2.102:6789/0 conn(0x7f0d440d1c70 legacy=0x7f0d44108f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:47.361 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:47.342 7f0d494af700 1 -- 172.21.2.102:0/871060919 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d44088070 con 0x7f0d441b0ae0 2021-02-10T11:39:47.361 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:47.342 7f0d494af700 1 --2- 172.21.2.102:0/871060919 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0d441b0ae0 0x7f0d44108940 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 rx=0x7f0d38006650 tx=0x7f0d38009a00).ready entity=mon.2 client_cookie=6bc6b0d453ed91b9 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:47.361 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:47.342 7f0d48cae700 1 -- 172.21.2.102:0/871060919 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0d380009b0 con 0x7f0d441b0ae0 2021-02-10T11:39:47.361 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:47.342 7f0d48cae700 1 -- 172.21.2.102:0/871060919 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d3800c0c0 con 0x7f0d441b0ae0 2021-02-10T11:39:47.361 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:47.342 7f0d48cae700 1 -- 172.21.2.102:0/871060919 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0d3800c5e0 con 0x7f0d441b0ae0 2021-02-10T11:39:47.361 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:47.342 7f0d4be94700 1 -- 172.21.2.102:0/871060919 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0d441b0ae0 msgr2=0x7f0d44108940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:47.361 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:47.342 7f0d4be94700 1 --2- 172.21.2.102:0/871060919 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0d441b0ae0 0x7f0d44108940 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 rx=0x7f0d38006650 tx=0x7f0d38009a00).stop 2021-02-10T11:39:47.361 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:47.343 7f0d4be94700 1 -- 172.21.2.102:0/871060919 shutdown_connections 2021-02-10T11:39:47.361 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:47.343 7f0d4be94700 1 --2- 172.21.2.102:0/871060919 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0d441b0ae0 0x7f0d44108940 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:47.361 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:47.343 7f0d4be94700 1 --2- 172.21.2.102:0/871060919 >> v2:172.21.2.113:3300/0 conn(0x7f0d440d1870 0x7f0d441b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:47.362 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:47.343 7f0d4be94700 1 -- 172.21.2.102:0/871060919 >> 172.21.2.102:0/871060919 conn(0x7f0d440ce420 msgr2=0x7f0d440d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:47.362 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:47.343 7f0d4be94700 1 -- 172.21.2.102:0/871060919 shutdown_connections 2021-02-10T11:39:47.362 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:47.343 7f0d4be94700 1 -- 172.21.2.102:0/871060919 wait complete. 2021-02-10T11:39:47.363 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:47.358 7f0d4be94700 1 Processor -- start 2021-02-10T11:39:47.363 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:47.358 7f0d4be94700 1 -- start start 2021-02-10T11:39:47.363 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:47.358 7f0d4be94700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0d4409bcc0 0x7f0d4409c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:47.363 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:47.358 7f0d4be94700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0d440a0820 0x7f0d440a0c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:47.364 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:47.358 7f0d4be94700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0d440a1240 0x7f0d440858d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:47.364 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:47.358 7f0d4be94700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d440a66b0 con 0x7f0d440a0820 2021-02-10T11:39:47.364 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:47.358 7f0d4be94700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d441a49b0 con 0x7f0d4409bcc0 2021-02-10T11:39:47.364 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:47.358 7f0d4be94700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d441a4b90 con 0x7f0d440a1240 2021-02-10T11:39:47.364 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:47.358 7f0d494af700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0d440a0820 0x7f0d440a0c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:47.364 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:47.358 7f0d494af700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0d440a0820 0x7f0d440a0c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:40672/0 (socket says 172.21.2.102:40672) 2021-02-10T11:39:47.364 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:47.358 7f0d494af700 1 -- 172.21.2.102:0/226909413 learned_addr learned my addr 172.21.2.102:0/226909413 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:39:47.364 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:47.359 7f0d494af700 1 -- 172.21.2.102:0/226909413 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0d440a1240 msgr2=0x7f0d440858d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:47.364 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:47.359 7f0d494af700 1 --2- 172.21.2.102:0/226909413 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0d440a1240 0x7f0d440858d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:47.364 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:47.359 7f0d494af700 1 -- 172.21.2.102:0/226909413 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0d4409bcc0 msgr2=0x7f0d4409c0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:47.364 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:47.359 7f0d494af700 1 --2- 172.21.2.102:0/226909413 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0d4409bcc0 0x7f0d4409c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:47.364 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:47.359 7f0d494af700 1 -- 172.21.2.102:0/226909413 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d440a1640 con 0x7f0d440a0820 2021-02-10T11:39:47.364 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:47.359 7f0d494af700 1 --2- 172.21.2.102:0/226909413 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0d440a0820 0x7f0d440a0c20 secure :-1 s=READY pgs=3279 cs=0 l=1 rev1=1 rx=0x7f0d380096d0 tx=0x7f0d3800b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:47.364 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:47.359 7f0d367fc700 1 -- 172.21.2.102:0/226909413 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0d38002030 con 0x7f0d440a0820 2021-02-10T11:39:47.365 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:47.359 7f0d367fc700 1 -- 172.21.2.102:0/226909413 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d38000940 con 0x7f0d440a0820 2021-02-10T11:39:47.365 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:47.359 7f0d367fc700 1 -- 172.21.2.102:0/226909413 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0d380024e0 con 0x7f0d440a0820 2021-02-10T11:39:47.365 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:47.359 7f0d4be94700 1 -- 172.21.2.102:0/226909413 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d440a1640 con 0x7f0d440a0820 2021-02-10T11:39:47.365 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:47.359 7f0d4be94700 1 -- 172.21.2.102:0/226909413 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d440a1640 con 0x7f0d440a0820 2021-02-10T11:39:47.365 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:47.360 7f0d367fc700 1 -- 172.21.2.102:0/226909413 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f0d38000c90 con 0x7f0d440a0820 2021-02-10T11:39:47.365 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:47.360 7f0d367fc700 1 --2- 172.21.2.102:0/226909413 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0d2c0186c0 0x7f0d2c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:47.365 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:47.360 7f0d367fc700 1 -- 172.21.2.102:0/226909413 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(189..189 src has 1..189) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f0d38032560 con 0x7f0d440a0820 2021-02-10T11:39:47.365 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:47.360 7f0d49cb0700 1 --2- 172.21.2.102:0/226909413 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0d2c0186c0 0x7f0d2c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:47.365 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:47.360 7f0d49cb0700 1 --2- 172.21.2.102:0/226909413 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0d2c0186c0 0x7f0d2c01abc0 crc :-1 s=READY pgs=1432 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:47.393 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:47.391 7f0d4be94700 1 -- 172.21.2.102:0/226909413 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d440fe1c0 con 0x7f0d440a0820 2021-02-10T11:39:47.396 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:47.394 7f0d367fc700 1 -- 172.21.2.102:0/226909413 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f0d38025230 con 0x7f0d440a0820 2021-02-10T11:39:47.523 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:47.521 7f0d4be94700 1 -- 172.21.2.102:0/226909413 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7f0d440fe1c0 con 0x7f0d440a0820 2021-02-10T11:39:47.523 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:47.521 7f0d367fc700 1 -- 172.21.2.102:0/226909413 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v189) v1 ==== 100+0+10 (secure 0 0 0) 0x7f0d38025230 con 0x7f0d440a0820 2021-02-10T11:39:47.524 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:39:47.526 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:47.523 7f0d4be94700 1 -- 172.21.2.102:0/226909413 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0d2c0186c0 msgr2=0x7f0d2c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:47.526 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:47.523 7f0d4be94700 1 --2- 172.21.2.102:0/226909413 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0d2c0186c0 0x7f0d2c01abc0 crc :-1 s=READY pgs=1432 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:47.526 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:47.523 7f0d4be94700 1 -- 172.21.2.102:0/226909413 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0d440a0820 msgr2=0x7f0d440a0c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:47.526 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:47.523 7f0d4be94700 1 --2- 172.21.2.102:0/226909413 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0d440a0820 0x7f0d440a0c20 secure :-1 s=READY pgs=3279 cs=0 l=1 rev1=1 rx=0x7f0d380096d0 tx=0x7f0d3800b990).stop 2021-02-10T11:39:47.526 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:47.523 7f0d4be94700 1 -- 172.21.2.102:0/226909413 shutdown_connections 2021-02-10T11:39:47.526 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:47.523 7f0d4be94700 1 --2- 172.21.2.102:0/226909413 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0d440a1240 0x7f0d440858d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:47.526 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:47.523 7f0d4be94700 1 --2- 172.21.2.102:0/226909413 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0d440a0820 0x7f0d440a0c20 unknown :-1 s=CLOSED pgs=3279 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:47.526 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:47.523 7f0d4be94700 1 --2- 172.21.2.102:0/226909413 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0d2c0186c0 0x7f0d2c01abc0 unknown :-1 s=CLOSED pgs=1432 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:47.526 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:47.523 7f0d4be94700 1 --2- 172.21.2.102:0/226909413 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0d4409bcc0 0x7f0d4409c0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:47.526 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:47.523 7f0d4be94700 1 -- 172.21.2.102:0/226909413 >> 172.21.2.102:0/226909413 conn(0x7f0d440bef00 msgr2=0x7f0d4409d1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:47.527 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:47.523 7f0d4be94700 1 -- 172.21.2.102:0/226909413 shutdown_connections 2021-02-10T11:39:47.527 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:47.523 7f0d4be94700 1 -- 172.21.2.102:0/226909413 wait complete. 2021-02-10T11:39:47.534 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-02-10T11:39:47.534 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:39:47.963 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:47.944 7f12699ca700 1 Processor -- start 2021-02-10T11:39:47.964 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:47.944 7f12699ca700 1 -- start start 2021-02-10T11:39:47.964 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:47.944 7f12699ca700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f12640d1870 0x7f12641b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:47.964 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:47.944 7f12699ca700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12640f0b10 con 0x7f12641b0a10 2021-02-10T11:39:47.964 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:47.944 7f12699ca700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12640d0310 con 0x7f12640f0650 2021-02-10T11:39:47.964 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:47.944 7f12699ca700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12640d0530 con 0x7f12640d1870 2021-02-10T11:39:47.964 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:47.945 7f1263574700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f12640d1870 0x7f12641b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:47.964 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:47.945 7f1262d73700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f12641b0a10 0x7f12640cf330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:32904/0 (socket says 172.21.2.102:32904) 2021-02-10T11:39:47.964 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:47.945 7f1262d73700 1 -- 172.21.2.102:0/2608040771 learned_addr learned my addr 172.21.2.102:0/2608040771 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:39:47.964 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:47.945 7f1263574700 1 -- 172.21.2.102:0/2608040771 >> v1:172.21.2.113:6789/0 conn(0x7f12640f0650 legacy=0x7f12640cfb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:47.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:47.945 7f1263574700 1 -- 172.21.2.102:0/2608040771 >> v1:172.21.2.102:6789/0 conn(0x7f12641b0a10 legacy=0x7f12640cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:47.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:47.945 7f1263574700 1 -- 172.21.2.102:0/2608040771 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12640d1c70 con 0x7f12640d1870 2021-02-10T11:39:47.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:47.945 7f1263574700 1 --2- 172.21.2.102:0/2608040771 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f12640d1870 0x7f12641b03f0 secure :-1 s=READY pgs=3281 cs=0 l=1 rev1=1 rx=0x7f124c006650 tx=0x7f124c009a80).ready entity=mon.0 client_cookie=80c2c367624c8064 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:47.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:47.946 7f1262572700 1 -- 172.21.2.102:0/2608040771 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f124c00c0c0 con 0x7f12640d1870 2021-02-10T11:39:47.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:47.946 7f1262572700 1 -- 172.21.2.102:0/2608040771 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f124c00c340 con 0x7f12640d1870 2021-02-10T11:39:47.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:47.946 7f1262572700 1 -- 172.21.2.102:0/2608040771 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f124c00c860 con 0x7f12640d1870 2021-02-10T11:39:47.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:47.946 7f12699ca700 1 -- 172.21.2.102:0/2608040771 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f12640d1870 msgr2=0x7f12641b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:47.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:47.946 7f12699ca700 1 --2- 172.21.2.102:0/2608040771 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f12640d1870 0x7f12641b03f0 secure :-1 s=READY pgs=3281 cs=0 l=1 rev1=1 rx=0x7f124c006650 tx=0x7f124c009a80).stop 2021-02-10T11:39:47.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:47.946 7f12699ca700 1 -- 172.21.2.102:0/2608040771 shutdown_connections 2021-02-10T11:39:47.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:47.946 7f12699ca700 1 --2- 172.21.2.102:0/2608040771 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f12640d1870 0x7f12641b03f0 unknown :-1 s=CLOSED pgs=3281 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:47.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:47.946 7f12699ca700 1 -- 172.21.2.102:0/2608040771 >> 172.21.2.102:0/2608040771 conn(0x7f12640ce450 msgr2=0x7f12640d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:47.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:47.947 7f12699ca700 1 -- 172.21.2.102:0/2608040771 shutdown_connections 2021-02-10T11:39:47.965 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:47.947 7f12699ca700 1 -- 172.21.2.102:0/2608040771 wait complete. 2021-02-10T11:39:47.966 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:47.961 7f12699ca700 1 Processor -- start 2021-02-10T11:39:47.966 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:47.961 7f12699ca700 1 -- start start 2021-02-10T11:39:47.967 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:47.962 7f12699ca700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f126409bca0 0x7f126409c0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:47.967 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:47.962 7f12699ca700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f126409c6c0 0x7f126409cac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:47.967 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:47.962 7f12699ca700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f126408c1a0 0x7f126408c5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:47.967 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:47.962 7f12699ca700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f126409b720 con 0x7f126409c6c0 2021-02-10T11:39:47.967 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:47.962 7f12699ca700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12640abcd0 con 0x7f126409bca0 2021-02-10T11:39:47.967 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:47.962 7f12699ca700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f126408cbc0 con 0x7f126408c1a0 2021-02-10T11:39:47.967 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:47.962 7f1262d73700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f126409c6c0 0x7f126409cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:47.967 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:47.962 7f1262d73700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f126409c6c0 0x7f126409cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:40684/0 (socket says 172.21.2.102:40684) 2021-02-10T11:39:47.967 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:47.962 7f1262d73700 1 -- 172.21.2.102:0/1626889215 learned_addr learned my addr 172.21.2.102:0/1626889215 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:39:47.967 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:47.962 7f1262d73700 1 -- 172.21.2.102:0/1626889215 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f126408c1a0 msgr2=0x7f126408c5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:47.968 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:47.962 7f1262d73700 1 --2- 172.21.2.102:0/1626889215 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f126408c1a0 0x7f126408c5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:47.968 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:47.962 7f1262d73700 1 -- 172.21.2.102:0/1626889215 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f126409bca0 msgr2=0x7f126409c0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:47.968 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:47.962 7f1262d73700 1 --2- 172.21.2.102:0/1626889215 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f126409bca0 0x7f126409c0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:47.968 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:47.962 7f1262d73700 1 -- 172.21.2.102:0/1626889215 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f126408cde0 con 0x7f126409c6c0 2021-02-10T11:39:47.968 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:47.962 7f1262d73700 1 --2- 172.21.2.102:0/1626889215 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f126409c6c0 0x7f126409cac0 secure :-1 s=READY pgs=3282 cs=0 l=1 rev1=1 rx=0x7f125400a4c0 tx=0x7f125400a860).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:47.968 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:47.963 7f125bfff700 1 -- 172.21.2.102:0/1626889215 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f125400ac50 con 0x7f126409c6c0 2021-02-10T11:39:47.968 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:47.963 7f125bfff700 1 -- 172.21.2.102:0/1626889215 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1254004140 con 0x7f126409c6c0 2021-02-10T11:39:47.968 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:47.963 7f12699ca700 1 -- 172.21.2.102:0/1626889215 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f126408cde0 con 0x7f126409c6c0 2021-02-10T11:39:47.968 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:47.963 7f125bfff700 1 -- 172.21.2.102:0/1626889215 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f12540045a0 con 0x7f126409c6c0 2021-02-10T11:39:47.968 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:47.963 7f12699ca700 1 -- 172.21.2.102:0/1626889215 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f126408d370 con 0x7f126409c6c0 2021-02-10T11:39:47.968 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:47.963 7f125bfff700 1 -- 172.21.2.102:0/1626889215 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1254004320 con 0x7f126409c6c0 2021-02-10T11:39:47.968 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:47.963 7f125bfff700 1 --2- 172.21.2.102:0/1626889215 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1248018730 0x7f124801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:47.968 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:47.963 7f125bfff700 1 -- 172.21.2.102:0/1626889215 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(189..189 src has 1..189) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f1254032610 con 0x7f126409c6c0 2021-02-10T11:39:47.968 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:47.963 7f1263574700 1 --2- 172.21.2.102:0/1626889215 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1248018730 0x7f124801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:47.969 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:47.964 7f1263574700 1 --2- 172.21.2.102:0/1626889215 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1248018730 0x7f124801ac10 crc :-1 s=READY pgs=1433 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:47.997 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:47.994 7f12699ca700 1 -- 172.21.2.102:0/1626889215 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12640d1ee0 con 0x7f126409c6c0 2021-02-10T11:39:47.999 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:47.997 7f125bfff700 1 -- 172.21.2.102:0/1626889215 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f1254000940 con 0x7f126409c6c0 2021-02-10T11:39:48.126 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:48.124 7f12699ca700 1 -- 172.21.2.102:0/1626889215 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f12640d1ee0 con 0x7f126409c6c0 2021-02-10T11:39:48.127 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:48.124 7f125bfff700 1 -- 172.21.2.102:0/1626889215 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 165 v165) v1 ==== 95+0+2104 (secure 0 0 0) 0x7f125400ac50 con 0x7f126409c6c0 2021-02-10T11:39:48.127 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:39:48.127 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":165,"default_fscid":20,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8998,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.113:6833/479366648","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6832","nonce":479366648},{"type":"v1","addr":"172.21.2.113:6833","nonce":479366648}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":162}],"filesystems":[{"mdsmap":{"epoch":165,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:39:30.375466","modified":"2021-02-10 11:39:32.379360","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7481},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7481":{"gid":7481,"name":"a","rank":0,"incarnation":163,"state":"up:active","state_seq":3,"addr":"172.21.2.102:6835/4290051997","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6834","nonce":4290051997},{"type":"v1","addr":"172.21.2.102:6835","nonce":4290051997}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[41],"metadata_pool":40,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":20}]} 2021-02-10T11:39:48.127 INFO:teuthology.orchestra.run.gibba002.stderr:dumped fsmap epoch 165 2021-02-10T11:39:48.128 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:48.126 7f12699ca700 1 -- 172.21.2.102:0/1626889215 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1248018730 msgr2=0x7f124801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:48.129 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:48.126 7f12699ca700 1 --2- 172.21.2.102:0/1626889215 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1248018730 0x7f124801ac10 crc :-1 s=READY pgs=1433 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:48.129 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:48.126 7f12699ca700 1 -- 172.21.2.102:0/1626889215 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f126409c6c0 msgr2=0x7f126409cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:48.129 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:48.126 7f12699ca700 1 --2- 172.21.2.102:0/1626889215 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f126409c6c0 0x7f126409cac0 secure :-1 s=READY pgs=3282 cs=0 l=1 rev1=1 rx=0x7f125400a4c0 tx=0x7f125400a860).stop 2021-02-10T11:39:48.129 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:48.126 7f12699ca700 1 -- 172.21.2.102:0/1626889215 shutdown_connections 2021-02-10T11:39:48.129 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:48.126 7f12699ca700 1 --2- 172.21.2.102:0/1626889215 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f126408c1a0 0x7f126408c5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:48.129 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:48.126 7f12699ca700 1 --2- 172.21.2.102:0/1626889215 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f126409c6c0 0x7f126409cac0 unknown :-1 s=CLOSED pgs=3282 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:48.129 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:48.126 7f12699ca700 1 --2- 172.21.2.102:0/1626889215 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1248018730 0x7f124801ac10 unknown :-1 s=CLOSED pgs=1433 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:48.129 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:48.126 7f12699ca700 1 --2- 172.21.2.102:0/1626889215 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f126409bca0 0x7f126409c0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:48.129 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:48.126 7f12699ca700 1 -- 172.21.2.102:0/1626889215 >> 172.21.2.102:0/1626889215 conn(0x7f12640a0450 msgr2=0x7f12640f0460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:48.129 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:48.127 7f12699ca700 1 -- 172.21.2.102:0/1626889215 shutdown_connections 2021-02-10T11:39:48.129 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:48.127 7f12699ca700 1 -- 172.21.2.102:0/1626889215 wait complete. 2021-02-10T11:39:48.338 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 165, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:39:30.375466', 'modified': '2021-02-10 11:39:32.379360', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7481}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7481': {'gid': 7481, 'name': 'a', 'rank': 0, 'incarnation': 163, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.102:6835/4290051997', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.102:6834', 'nonce': 4290051997}, {'type': 'v1', 'addr': '172.21.2.102:6835', 'nonce': 4290051997}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [41], 'metadata_pool': 40, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:39:48.338 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:39:48.339 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok status 2021-02-10T11:39:48.741 INFO:teuthology.orchestra.run.gibba002.stdout:{ 2021-02-10T11:39:48.742 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_fsid": "68db30ea-dbce-4aef-a6df-f01c6fea0513", 2021-02-10T11:39:48.742 INFO:teuthology.orchestra.run.gibba002.stdout: "whoami": 0, 2021-02-10T11:39:48.742 INFO:teuthology.orchestra.run.gibba002.stdout: "id": 7481, 2021-02-10T11:39:48.742 INFO:teuthology.orchestra.run.gibba002.stdout: "want_state": "up:active", 2021-02-10T11:39:48.742 INFO:teuthology.orchestra.run.gibba002.stdout: "state": "up:active", 2021-02-10T11:39:48.742 INFO:teuthology.orchestra.run.gibba002.stdout: "rank_uptime": 18.361846577000001, 2021-02-10T11:39:48.742 INFO:teuthology.orchestra.run.gibba002.stdout: "mdsmap_epoch": 165, 2021-02-10T11:39:48.742 INFO:teuthology.orchestra.run.gibba002.stdout: "osdmap_epoch": 189, 2021-02-10T11:39:48.743 INFO:teuthology.orchestra.run.gibba002.stdout: "osdmap_epoch_barrier": 189, 2021-02-10T11:39:48.743 INFO:teuthology.orchestra.run.gibba002.stdout: "uptime": 24.011492052000001 2021-02-10T11:39:48.743 INFO:teuthology.orchestra.run.gibba002.stdout:} 2021-02-10T11:39:48.746 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "68db30ea-dbce-4aef-a6df-f01c6fea0513", "whoami": 0, "id": 7481, "want_state": "up:active", "state": "up:active", "rank_uptime": 18.361846577000001, "mdsmap_epoch": 165, "osdmap_epoch": 189, "osdmap_epoch_barrier": 189, "uptime": 24.011492052000001 } 2021-02-10T11:39:48.747 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2021-02-10T11:39:48.747 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2021-02-10T11:39:48.747 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.1 at ubuntu@gibba002.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.1... 2021-02-10T11:39:48.747 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:39:48.748 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec mkdir -p /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:39:49.086 DEBUG:teuthology.orchestra.run.gibba002:> sudo modprobe fuse 2021-02-10T11:39:49.431 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:39:49.431 DEBUG:teuthology.orchestra.run.gibba002:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:39:49.775 INFO:teuthology.orchestra.run.gibba002.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:39:49.775 INFO:teuthology.orchestra.run.gibba002.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:39:49.776 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:39:49.776 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:39:49.776 DEBUG:teuthology.orchestra.run.gibba002:> ls /sys/fs/fuse/connections 2021-02-10T11:39:50.110 INFO:teuthology.orchestra.run.gibba002.stdout:43 2021-02-10T11:39:50.111 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [43] 2021-02-10T11:39:50.111 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.1 /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:39:50.114 DEBUG:teuthology.orchestra.run.gibba002:> sudo modprobe fuse 2021-02-10T11:39:50.458 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:39:50.458 DEBUG:teuthology.orchestra.run.gibba002:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:39:50.524 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba002.stderr:ceph-fuse[217640]: starting ceph client 2021-02-10T11:39:50.525 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba002.stderr:2021-02-10 11:39:50.522 7f30ee70cf80 -1 init, newargv = 0x55c6a3a99ab0 newargc=9 2021-02-10T11:39:50.529 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba002.stderr:ceph-fuse[217640]: starting fuse 2021-02-10T11:39:50.801 INFO:teuthology.orchestra.run.gibba002.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:39:50.801 INFO:teuthology.orchestra.run.gibba002.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:39:50.802 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:39:50.802 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:39:50.802 DEBUG:teuthology.orchestra.run.gibba002:> ls /sys/fs/fuse/connections 2021-02-10T11:39:51.137 INFO:teuthology.orchestra.run.gibba002.stdout:43 2021-02-10T11:39:51.138 INFO:teuthology.orchestra.run.gibba002.stdout:44 2021-02-10T11:39:51.138 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43, 44] 2021-02-10T11:39:51.138 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:39:51.138 DEBUG:teuthology.orchestra.run.gibba002:> sudo python3 -c ' 2021-02-10T11:39:51.138 DEBUG:teuthology.orchestra.run.gibba002:> import glob 2021-02-10T11:39:51.138 DEBUG:teuthology.orchestra.run.gibba002:> import re 2021-02-10T11:39:51.138 DEBUG:teuthology.orchestra.run.gibba002:> import os 2021-02-10T11:39:51.139 DEBUG:teuthology.orchestra.run.gibba002:> import subprocess 2021-02-10T11:39:51.139 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:39:51.139 DEBUG:teuthology.orchestra.run.gibba002:> def find_socket(client_name): 2021-02-10T11:39:51.139 DEBUG:teuthology.orchestra.run.gibba002:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2021-02-10T11:39:51.139 DEBUG:teuthology.orchestra.run.gibba002:> files = glob.glob(asok_path) 2021-02-10T11:39:51.139 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:39:51.139 DEBUG:teuthology.orchestra.run.gibba002:> # Given a non-glob path, it better be there 2021-02-10T11:39:51.139 DEBUG:teuthology.orchestra.run.gibba002:> if "*" not in asok_path: 2021-02-10T11:39:51.139 DEBUG:teuthology.orchestra.run.gibba002:> assert(len(files) == 1) 2021-02-10T11:39:51.139 DEBUG:teuthology.orchestra.run.gibba002:> return files[0] 2021-02-10T11:39:51.139 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:39:51.139 DEBUG:teuthology.orchestra.run.gibba002:> for f in files: 2021-02-10T11:39:51.139 DEBUG:teuthology.orchestra.run.gibba002:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2021-02-10T11:39:51.139 DEBUG:teuthology.orchestra.run.gibba002:> if os.path.exists("/proc/{0}".format(pid)): 2021-02-10T11:39:51.139 DEBUG:teuthology.orchestra.run.gibba002:> return f 2021-02-10T11:39:51.139 DEBUG:teuthology.orchestra.run.gibba002:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2021-02-10T11:39:51.140 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:39:51.140 DEBUG:teuthology.orchestra.run.gibba002:> print(find_socket("client.1")) 2021-02-10T11:39:51.140 DEBUG:teuthology.orchestra.run.gibba002:> ' 2021-02-10T11:39:51.502 INFO:teuthology.orchestra.run.gibba002.stdout:/var/run/ceph/ceph-client.1.217640.asok 2021-02-10T11:39:51.504 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.217640.asok 2021-02-10T11:39:51.504 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:39:51.505 DEBUG:teuthology.orchestra.run.gibba002:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.217640.asok status 2021-02-10T11:39:51.895 INFO:teuthology.orchestra.run.gibba002.stdout:{ 2021-02-10T11:39:51.895 INFO:teuthology.orchestra.run.gibba002.stdout: "metadata": { 2021-02-10T11:39:51.895 INFO:teuthology.orchestra.run.gibba002.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:39:51.895 INFO:teuthology.orchestra.run.gibba002.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:39:51.895 INFO:teuthology.orchestra.run.gibba002.stdout: "entity_id": "1", 2021-02-10T11:39:51.895 INFO:teuthology.orchestra.run.gibba002.stdout: "hostname": "gibba002", 2021-02-10T11:39:51.895 INFO:teuthology.orchestra.run.gibba002.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2021-02-10T11:39:51.895 INFO:teuthology.orchestra.run.gibba002.stdout: "pid": "217640", 2021-02-10T11:39:51.896 INFO:teuthology.orchestra.run.gibba002.stdout: "root": "/" 2021-02-10T11:39:51.896 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:39:51.896 INFO:teuthology.orchestra.run.gibba002.stdout: "dentry_count": 0, 2021-02-10T11:39:51.896 INFO:teuthology.orchestra.run.gibba002.stdout: "dentry_pinned_count": 0, 2021-02-10T11:39:51.896 INFO:teuthology.orchestra.run.gibba002.stdout: "id": 13680, 2021-02-10T11:39:51.896 INFO:teuthology.orchestra.run.gibba002.stdout: "inst": { 2021-02-10T11:39:51.896 INFO:teuthology.orchestra.run.gibba002.stdout: "name": { 2021-02-10T11:39:51.896 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "client", 2021-02-10T11:39:51.896 INFO:teuthology.orchestra.run.gibba002.stdout: "num": 13680 2021-02-10T11:39:51.896 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:39:51.896 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": { 2021-02-10T11:39:51.896 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:39:51.897 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:0", 2021-02-10T11:39:51.897 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 3332606390 2021-02-10T11:39:51.897 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:39:51.897 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:39:51.897 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": { 2021-02-10T11:39:51.897 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:39:51.897 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:0", 2021-02-10T11:39:51.897 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 3332606390 2021-02-10T11:39:51.897 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:39:51.897 INFO:teuthology.orchestra.run.gibba002.stdout: "inst_str": "client.13680 172.21.2.102:0/3332606390", 2021-02-10T11:39:51.897 INFO:teuthology.orchestra.run.gibba002.stdout: "addr_str": "172.21.2.102:0/3332606390", 2021-02-10T11:39:51.897 INFO:teuthology.orchestra.run.gibba002.stdout: "inode_count": 1, 2021-02-10T11:39:51.897 INFO:teuthology.orchestra.run.gibba002.stdout: "mds_epoch": 165, 2021-02-10T11:39:51.897 INFO:teuthology.orchestra.run.gibba002.stdout: "osd_epoch": 189, 2021-02-10T11:39:51.898 INFO:teuthology.orchestra.run.gibba002.stdout: "osd_epoch_barrier": 0, 2021-02-10T11:39:51.898 INFO:teuthology.orchestra.run.gibba002.stdout: "blacklisted": false 2021-02-10T11:39:51.898 INFO:teuthology.orchestra.run.gibba002.stdout:} 2021-02-10T11:39:51.899 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:39:51.899 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:39:51.899 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:39:52.236 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:39:52.236 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2021-02-10T11:39:52.237 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:39:52.237 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:39:52.584 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2021-02-10T11:39:53.016 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:52.998 7f148ae09700 1 Processor -- start 2021-02-10T11:39:53.017 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:52.998 7f148ae09700 1 -- start start 2021-02-10T11:39:53.017 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:52.999 7f148ae09700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f14840d1c80 0x7f14840cef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:53.017 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:52.999 7f148ae09700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f14840cf590 0x7f14840cf990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:53.017 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:52.999 7f148ae09700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14840fe1d0 con 0x7f14840d1880 2021-02-10T11:39:53.017 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:52.999 7f148ae09700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14841b0cc0 con 0x7f14840cf590 2021-02-10T11:39:53.017 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:52.999 7f148ae09700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1484109a70 con 0x7f14840d1c80 2021-02-10T11:39:53.017 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:52.999 7f1489426700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f14840cf590 0x7f14840cf990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:53.017 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:52.999 7f1489426700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f14840cf590 0x7f14840cf990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:40708/0 (socket says 172.21.2.102:40708) 2021-02-10T11:39:53.017 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:52.999 7f1489426700 1 -- 172.21.2.102:0/2073360886 learned_addr learned my addr 172.21.2.102:0/2073360886 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:39:53.017 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:52.999 7f1489426700 1 -- 172.21.2.102:0/2073360886 >> v2:172.21.2.113:3300/0 conn(0x7f14840d1c80 msgr2=0x7f14840cef70 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:39:53.018 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:52.999 7f1489426700 1 --2- 172.21.2.102:0/2073360886 >> v2:172.21.2.113:3300/0 conn(0x7f14840d1c80 0x7f14840cef70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:53.018 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:52.999 7f1489426700 1 -- 172.21.2.102:0/2073360886 >> v1:172.21.2.113:6789/0 conn(0x7f14840d1880 legacy=0x7f14841b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:53.018 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:52.999 7f1489426700 1 -- 172.21.2.102:0/2073360886 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1484088070 con 0x7f14840cf590 2021-02-10T11:39:53.018 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:52.999 7f1489426700 1 --2- 172.21.2.102:0/2073360886 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f14840cf590 0x7f14840cf990 secure :-1 s=READY pgs=3285 cs=0 l=1 rev1=1 rx=0x7f147c006b80 tx=0x7f147c009b90).ready entity=mon.0 client_cookie=5415d57bfa8f86b1 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:53.018 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:53.000 7f14837fe700 1 -- 172.21.2.102:0/2073360886 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f147c0022e0 con 0x7f14840cf590 2021-02-10T11:39:53.018 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:53.000 7f14837fe700 1 -- 172.21.2.102:0/2073360886 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f147c002620 con 0x7f14840cf590 2021-02-10T11:39:53.018 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:53.000 7f14837fe700 1 -- 172.21.2.102:0/2073360886 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f147c002b00 con 0x7f14840cf590 2021-02-10T11:39:53.018 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:53.000 7f148ae09700 1 -- 172.21.2.102:0/2073360886 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f14840cf590 msgr2=0x7f14840cf990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:53.018 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:53.000 7f148ae09700 1 --2- 172.21.2.102:0/2073360886 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f14840cf590 0x7f14840cf990 secure :-1 s=READY pgs=3285 cs=0 l=1 rev1=1 rx=0x7f147c006b80 tx=0x7f147c009b90).stop 2021-02-10T11:39:53.018 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:53.000 7f148ae09700 1 -- 172.21.2.102:0/2073360886 shutdown_connections 2021-02-10T11:39:53.018 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:53.000 7f148ae09700 1 --2- 172.21.2.102:0/2073360886 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f14840cf590 0x7f14840cf990 unknown :-1 s=CLOSED pgs=3285 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:53.018 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:53.000 7f148ae09700 1 --2- 172.21.2.102:0/2073360886 >> v2:172.21.2.113:3300/0 conn(0x7f14840d1c80 0x7f14840cef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:53.019 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:53.000 7f148ae09700 1 -- 172.21.2.102:0/2073360886 >> 172.21.2.102:0/2073360886 conn(0x7f14840ce430 msgr2=0x7f14840d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:53.019 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:53.000 7f148ae09700 1 -- 172.21.2.102:0/2073360886 shutdown_connections 2021-02-10T11:39:53.019 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:53.000 7f148ae09700 1 -- 172.21.2.102:0/2073360886 wait complete. 2021-02-10T11:39:53.020 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:53.015 7f148ae09700 1 Processor -- start 2021-02-10T11:39:53.020 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:53.015 7f148ae09700 1 -- start start 2021-02-10T11:39:53.020 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:53.015 7f148ae09700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f148419e5b0 0x7f14841afc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:53.021 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:53.015 7f148ae09700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f148419e9b0 0x7f14841b0dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:53.021 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:53.015 7f148ae09700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f14841b0250 0x7f14841b13e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:53.021 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:53.015 7f148ae09700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1484195bd0 con 0x7f148419e9b0 2021-02-10T11:39:53.021 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:53.015 7f148ae09700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14841b0840 con 0x7f148419e5b0 2021-02-10T11:39:53.021 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:53.015 7f148ae09700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f148419eed0 con 0x7f14841b0250 2021-02-10T11:39:53.021 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:53.015 7f1483fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f148419e9b0 0x7f14841b0dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:53.021 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:53.015 7f1483fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f148419e9b0 0x7f14841b0dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:40712/0 (socket says 172.21.2.102:40712) 2021-02-10T11:39:53.021 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:53.015 7f1483fff700 1 -- 172.21.2.102:0/3825525047 learned_addr learned my addr 172.21.2.102:0/3825525047 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:39:53.021 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:53.016 7f1483fff700 1 -- 172.21.2.102:0/3825525047 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f14841b0250 msgr2=0x7f14841b13e0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:39:53.021 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:53.016 7f1483fff700 1 --2- 172.21.2.102:0/3825525047 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f14841b0250 0x7f14841b13e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:53.021 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:53.016 7f1483fff700 1 -- 172.21.2.102:0/3825525047 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f148419e5b0 msgr2=0x7f14841afc30 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:39:53.021 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:53.016 7f1483fff700 1 --2- 172.21.2.102:0/3825525047 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f148419e5b0 0x7f14841afc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:53.021 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:53.016 7f1483fff700 1 -- 172.21.2.102:0/3825525047 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f148419f0b0 con 0x7f148419e9b0 2021-02-10T11:39:53.022 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:53.016 7f1483fff700 1 --2- 172.21.2.102:0/3825525047 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f148419e9b0 0x7f14841b0dc0 secure :-1 s=READY pgs=3286 cs=0 l=1 rev1=1 rx=0x7f1474009960 tx=0x7f147400bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:53.022 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:53.016 7f14817fa700 1 -- 172.21.2.102:0/3825525047 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1474003030 con 0x7f148419e9b0 2021-02-10T11:39:53.022 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:53.016 7f14817fa700 1 -- 172.21.2.102:0/3825525047 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1474001310 con 0x7f148419e9b0 2021-02-10T11:39:53.022 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:53.016 7f14817fa700 1 -- 172.21.2.102:0/3825525047 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1474001830 con 0x7f148419e9b0 2021-02-10T11:39:53.022 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:53.016 7f148ae09700 1 -- 172.21.2.102:0/3825525047 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f148419f0b0 con 0x7f148419e9b0 2021-02-10T11:39:53.022 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:53.016 7f148ae09700 1 -- 172.21.2.102:0/3825525047 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14841b1cf0 con 0x7f148419e9b0 2021-02-10T11:39:53.022 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:53.018 7f14817fa700 1 -- 172.21.2.102:0/3825525047 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f14740014f0 con 0x7f148419e9b0 2021-02-10T11:39:53.022 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:53.018 7f14817fa700 1 --2- 172.21.2.102:0/3825525047 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f14680186c0 0x7f146801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:53.022 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:53.018 7f14817fa700 1 -- 172.21.2.102:0/3825525047 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(189..189 src has 1..189) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f1474032920 con 0x7f148419e9b0 2021-02-10T11:39:53.022 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:53.018 7f1488c25700 1 --2- 172.21.2.102:0/3825525047 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f14680186c0 0x7f146801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:53.022 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:53.018 7f1488c25700 1 --2- 172.21.2.102:0/3825525047 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f14680186c0 0x7f146801abc0 crc :-1 s=READY pgs=1434 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:53.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:53.049 7f148ae09700 1 -- 172.21.2.102:0/3825525047 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14841b0840 con 0x7f148419e9b0 2021-02-10T11:39:53.053 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:53.051 7f14817fa700 1 -- 172.21.2.102:0/3825525047 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f1474000ce0 con 0x7f148419e9b0 2021-02-10T11:39:53.188 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:53.186 7f148ae09700 1 -- 172.21.2.102:0/3825525047 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f14841b0840 con 0x7f148419e9b0 2021-02-10T11:39:53.205 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:53.204 7f14817fa700 1 -- 172.21.2.102:0/3825525047 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v166) v1 ==== 136+0+0 (secure 0 0 0) 0x7f1474000cc0 con 0x7f148419e9b0 2021-02-10T11:39:53.222 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:53.220 7f148ae09700 1 -- 172.21.2.102:0/3825525047 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f14680186c0 msgr2=0x7f146801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:53.222 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:53.220 7f148ae09700 1 --2- 172.21.2.102:0/3825525047 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f14680186c0 0x7f146801abc0 crc :-1 s=READY pgs=1434 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:53.222 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:53.220 7f148ae09700 1 -- 172.21.2.102:0/3825525047 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f148419e9b0 msgr2=0x7f14841b0dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:53.222 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:53.220 7f148ae09700 1 --2- 172.21.2.102:0/3825525047 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f148419e9b0 0x7f14841b0dc0 secure :-1 s=READY pgs=3286 cs=0 l=1 rev1=1 rx=0x7f1474009960 tx=0x7f147400bcf0).stop 2021-02-10T11:39:53.223 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:53.220 7f148ae09700 1 -- 172.21.2.102:0/3825525047 shutdown_connections 2021-02-10T11:39:53.223 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:53.220 7f148ae09700 1 --2- 172.21.2.102:0/3825525047 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f14841b0250 0x7f14841b13e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:53.223 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:53.220 7f148ae09700 1 --2- 172.21.2.102:0/3825525047 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f148419e9b0 0x7f14841b0dc0 unknown :-1 s=CLOSED pgs=3286 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:53.223 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:53.220 7f148ae09700 1 --2- 172.21.2.102:0/3825525047 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f14680186c0 0x7f146801abc0 unknown :-1 s=CLOSED pgs=1434 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:53.223 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:53.220 7f148ae09700 1 --2- 172.21.2.102:0/3825525047 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f148419e5b0 0x7f14841afc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:53.224 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:53.220 7f148ae09700 1 -- 172.21.2.102:0/3825525047 >> 172.21.2.102:0/3825525047 conn(0x7f148419f940 msgr2=0x7f148418c290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:53.224 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:53.221 7f148ae09700 1 -- 172.21.2.102:0/3825525047 shutdown_connections 2021-02-10T11:39:53.224 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:53.221 7f148ae09700 1 -- 172.21.2.102:0/3825525047 wait complete. 2021-02-10T11:39:53.230 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs volume ls 2021-02-10T11:39:53.659 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:53.639 7f373b8db700 1 Processor -- start 2021-02-10T11:39:53.659 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:53.639 7f373b8db700 1 -- start start 2021-02-10T11:39:53.659 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:53.640 7f373b8db700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f37340d0340 0x7f37340ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:53.659 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:53.640 7f373b8db700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37340f0b10 con 0x7f37340d1c70 2021-02-10T11:39:53.659 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:53.640 7f373b8db700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37341b0be0 con 0x7f37340d1870 2021-02-10T11:39:53.660 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:53.640 7f373b8db700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37340e5ed0 con 0x7f37340d0340 2021-02-10T11:39:53.660 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:53.640 7f3739ef8700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f37340d1c70 0x7f37340efc60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:32942/0 (socket says 172.21.2.102:32942) 2021-02-10T11:39:53.660 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:53.640 7f3739ef8700 1 -- 172.21.2.102:0/1668936449 learned_addr learned my addr 172.21.2.102:0/1668936449 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:39:53.660 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:53.640 7f3727fff700 1 -- 172.21.2.102:0/1668936449 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4224371330 0 0) 0x7f372c001500 con 0x7f37340d1c70 2021-02-10T11:39:53.660 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:53.640 7f3727fff700 1 -- 172.21.2.102:0/1668936449 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3720002fe0 con 0x7f37340d1c70 2021-02-10T11:39:53.660 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:53.641 7f3738ef6700 1 --2- 172.21.2.102:0/1668936449 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f37340d0340 0x7f37340ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:53.660 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:53.641 7f3727fff700 1 -- 172.21.2.102:0/1668936449 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1599099041 0 0) 0x7f372c001460 con 0x7f37340d1c70 2021-02-10T11:39:53.660 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:53.641 7f3727fff700 1 -- 172.21.2.102:0/1668936449 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f37340d0340 msgr2=0x7f37340ef640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:53.660 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:53.641 7f3727fff700 1 --2- 172.21.2.102:0/1668936449 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f37340d0340 0x7f37340ef640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:53.660 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:53.641 7f3727fff700 1 -- 172.21.2.102:0/1668936449 >> v1:172.21.2.113:6789/0 conn(0x7f37340d1870 legacy=0x7f37341b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:53.660 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:53.641 7f3727fff700 1 -- 172.21.2.102:0/1668936449 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37340f0450 con 0x7f37340d1c70 2021-02-10T11:39:53.660 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:53.641 7f3727fff700 1 -- 172.21.2.102:0/1668936449 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7f372c001980 con 0x7f37340d1c70 2021-02-10T11:39:53.660 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:53.641 7f3727fff700 1 -- 172.21.2.102:0/1668936449 >> v1:172.21.2.102:6789/0 conn(0x7f37340d1c70 legacy=0x7f37340efc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:53.660 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:53.641 7f3727fff700 1 --2- 172.21.2.102:0/1668936449 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3720003b10 0x7f3720005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:53.661 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:53.641 7f3727fff700 1 --2- 172.21.2.102:0/1668936449 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f37200067b0 0x7f3720008c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:53.661 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:53.641 7f3727fff700 1 --2- 172.21.2.102:0/1668936449 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3720009500 0x7f372000ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:53.661 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:53.641 7f3727fff700 1 -- 172.21.2.102:0/1668936449 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f372000c2d0 con 0x7f3720009500 2021-02-10T11:39:53.661 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:53.641 7f3727fff700 1 -- 172.21.2.102:0/1668936449 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f372000c4f0 con 0x7f3720003b10 2021-02-10T11:39:53.661 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:53.641 7f3727fff700 1 -- 172.21.2.102:0/1668936449 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f372000c710 con 0x7f37200067b0 2021-02-10T11:39:53.661 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:53.641 7f3739ef8700 1 --2- 172.21.2.102:0/1668936449 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3720009500 0x7f372000ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:53.661 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:53.641 7f3738ef6700 1 --2- 172.21.2.102:0/1668936449 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f37200067b0 0x7f3720008c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:53.661 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:53.641 7f37396f7700 1 --2- 172.21.2.102:0/1668936449 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3720003b10 0x7f3720005fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:53.661 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:53.641 7f3739ef8700 1 -- 172.21.2.102:0/1668936449 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f37200067b0 msgr2=0x7f3720008c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:53.661 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:53.641 7f3739ef8700 1 --2- 172.21.2.102:0/1668936449 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f37200067b0 0x7f3720008c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:53.661 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:53.641 7f3739ef8700 1 -- 172.21.2.102:0/1668936449 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3720003b10 msgr2=0x7f3720005fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:53.661 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:53.641 7f3739ef8700 1 --2- 172.21.2.102:0/1668936449 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3720003b10 0x7f3720005fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:53.661 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:53.641 7f3739ef8700 1 -- 172.21.2.102:0/1668936449 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f372000c9b0 con 0x7f3720009500 2021-02-10T11:39:53.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:53.641 7f3739ef8700 1 --2- 172.21.2.102:0/1668936449 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3720009500 0x7f372000ba20 secure :-1 s=READY pgs=3288 cs=0 l=1 rev1=1 rx=0x7f372c001f00 tx=0x7f372c04e320).ready entity=mon.0 client_cookie=3886d774d73be41a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:53.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:53.642 7f3727fff700 1 -- 172.21.2.102:0/1668936449 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f372c007920 con 0x7f3720009500 2021-02-10T11:39:53.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:53.642 7f3727fff700 1 -- 172.21.2.102:0/1668936449 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f372c007d80 con 0x7f3720009500 2021-02-10T11:39:53.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:53.642 7f3727fff700 1 -- 172.21.2.102:0/1668936449 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f372c0073d0 con 0x7f3720009500 2021-02-10T11:39:53.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:53.642 7f373b8db700 1 -- 172.21.2.102:0/1668936449 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3720009500 msgr2=0x7f372000ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:53.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:53.642 7f373b8db700 1 --2- 172.21.2.102:0/1668936449 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3720009500 0x7f372000ba20 secure :-1 s=READY pgs=3288 cs=0 l=1 rev1=1 rx=0x7f372c001f00 tx=0x7f372c04e320).stop 2021-02-10T11:39:53.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:53.642 7f3739ef8700 1 -- 172.21.2.102:0/1668936449 reap_dead start 2021-02-10T11:39:53.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:53.642 7f373b8db700 1 -- 172.21.2.102:0/1668936449 shutdown_connections 2021-02-10T11:39:53.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:53.642 7f373b8db700 1 -- 172.21.2.102:0/1668936449 >> 172.21.2.102:0/1668936449 conn(0x7f37340ce450 msgr2=0x7f37340d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:53.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:53.642 7f373b8db700 1 -- 172.21.2.102:0/1668936449 shutdown_connections 2021-02-10T11:39:53.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:53.642 7f373b8db700 1 -- 172.21.2.102:0/1668936449 wait complete. 2021-02-10T11:39:53.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:53.657 7f373b8db700 1 Processor -- start 2021-02-10T11:39:53.662 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:53.657 7f373b8db700 1 -- start start 2021-02-10T11:39:53.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:53.657 7f373b8db700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f373419ffa0 0x7f37340eda10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:53.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:53.657 7f373b8db700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f37341a03a0 0x7f37340ee030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:53.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:53.657 7f373b8db700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f37340ee650 0x7f37340ebe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:53.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:53.657 7f373b8db700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37340ed490 con 0x7f37340ee650 2021-02-10T11:39:53.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:53.657 7f373b8db700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37340eea50 con 0x7f37341a03a0 2021-02-10T11:39:53.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:53.657 7f373b8db700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37340eec30 con 0x7f373419ffa0 2021-02-10T11:39:53.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:53.658 7f3739ef8700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f37340ee650 0x7f37340ebe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:53.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:53.658 7f3739ef8700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f37340ee650 0x7f37340ebe80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:40732/0 (socket says 172.21.2.102:40732) 2021-02-10T11:39:53.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:53.658 7f3739ef8700 1 -- 172.21.2.102:0/1532501073 learned_addr learned my addr 172.21.2.102:0/1532501073 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:39:53.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:53.658 7f3738ef6700 1 --2- 172.21.2.102:0/1532501073 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f37341a03a0 0x7f37340ee030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:53.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:53.658 7f37396f7700 1 --2- 172.21.2.102:0/1532501073 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f373419ffa0 0x7f37340eda10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:53.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:53.658 7f3739ef8700 1 -- 172.21.2.102:0/1532501073 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f373419ffa0 msgr2=0x7f37340eda10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:53.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:53.658 7f3739ef8700 1 --2- 172.21.2.102:0/1532501073 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f373419ffa0 0x7f37340eda10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:53.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:53.658 7f3739ef8700 1 -- 172.21.2.102:0/1532501073 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f37341a03a0 msgr2=0x7f37340ee030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:53.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:53.658 7f3739ef8700 1 --2- 172.21.2.102:0/1532501073 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f37341a03a0 0x7f37340ee030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:53.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:53.658 7f3739ef8700 1 -- 172.21.2.102:0/1532501073 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37340ec4a0 con 0x7f37340ee650 2021-02-10T11:39:53.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:53.658 7f37396f7700 1 --2- 172.21.2.102:0/1532501073 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f373419ffa0 0x7f37340eda10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:39:53.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:53.658 7f3739ef8700 1 --2- 172.21.2.102:0/1532501073 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f37340ee650 0x7f37340ebe80 secure :-1 s=READY pgs=3289 cs=0 l=1 rev1=1 rx=0x7f372c04f8c0 tx=0x7f372c04fc60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:53.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:53.658 7f3725ffb700 1 -- 172.21.2.102:0/1532501073 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f372c001950 con 0x7f37340ee650 2021-02-10T11:39:53.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:53.658 7f3725ffb700 1 -- 172.21.2.102:0/1532501073 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f372c007030 con 0x7f37340ee650 2021-02-10T11:39:53.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:53.658 7f3725ffb700 1 -- 172.21.2.102:0/1532501073 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f372c007480 con 0x7f37340ee650 2021-02-10T11:39:53.666 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:53.658 7f373b8db700 1 -- 172.21.2.102:0/1532501073 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37340ec4a0 con 0x7f37340ee650 2021-02-10T11:39:53.666 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:53.659 7f373b8db700 1 -- 172.21.2.102:0/1532501073 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37340eca30 con 0x7f37340ee650 2021-02-10T11:39:53.666 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:53.659 7f3725ffb700 1 -- 172.21.2.102:0/1532501073 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f372c007210 con 0x7f37340ee650 2021-02-10T11:39:53.666 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:53.659 7f3725ffb700 1 --2- 172.21.2.102:0/1532501073 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f37140186c0 0x7f371401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:53.666 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:53.659 7f3725ffb700 1 -- 172.21.2.102:0/1532501073 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(189..189 src has 1..189) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f372c076920 con 0x7f37340ee650 2021-02-10T11:39:53.666 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:53.659 7f37396f7700 1 --2- 172.21.2.102:0/1532501073 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f37140186c0 0x7f371401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:53.666 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:53.661 7f37396f7700 1 --2- 172.21.2.102:0/1532501073 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f37140186c0 0x7f371401abc0 crc :-1 s=READY pgs=1435 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:53.692 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:53.691 7f373b8db700 1 -- 172.21.2.102:0/1532501073 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37340ebc50 con 0x7f37340ee650 2021-02-10T11:39:53.694 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:53.693 7f3725ffb700 1 -- 172.21.2.102:0/1532501073 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f372c001d50 con 0x7f37340ee650 2021-02-10T11:39:53.835 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:53.831 7f373b8db700 1 -- 172.21.2.102:0/1532501073 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs volume ls", "target": ["mgr", ""]}) v1 -- 0x7f37340eca30 con 0x7f37140186c0 2021-02-10T11:39:53.835 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:53.831 7f3725ffb700 1 -- 172.21.2.102:0/1532501073 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+41 (crc 0 0 0) 0x7f3730006c00 con 0x7f37140186c0 2021-02-10T11:39:53.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:53.833 7f373b8db700 1 -- 172.21.2.102:0/1532501073 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f37140186c0 msgr2=0x7f371401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:53.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:53.833 7f373b8db700 1 --2- 172.21.2.102:0/1532501073 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f37140186c0 0x7f371401abc0 crc :-1 s=READY pgs=1435 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:53.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:53.833 7f373b8db700 1 -- 172.21.2.102:0/1532501073 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f37340ee650 msgr2=0x7f37340ebe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:53.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:53.833 7f373b8db700 1 --2- 172.21.2.102:0/1532501073 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f37340ee650 0x7f37340ebe80 secure :-1 s=READY pgs=3289 cs=0 l=1 rev1=1 rx=0x7f372c04f8c0 tx=0x7f372c04fc60).stop 2021-02-10T11:39:53.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:53.833 7f373b8db700 1 -- 172.21.2.102:0/1532501073 shutdown_connections 2021-02-10T11:39:53.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:53.833 7f373b8db700 1 --2- 172.21.2.102:0/1532501073 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f37340ee650 0x7f37340ebe80 unknown :-1 s=CLOSED pgs=3289 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:53.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:53.833 7f373b8db700 1 --2- 172.21.2.102:0/1532501073 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f37140186c0 0x7f371401abc0 unknown :-1 s=CLOSED pgs=1435 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:53.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:53.833 7f373b8db700 1 --2- 172.21.2.102:0/1532501073 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f37341a03a0 0x7f37340ee030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:53.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:53.833 7f373b8db700 1 --2- 172.21.2.102:0/1532501073 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f373419ffa0 0x7f37340eda10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:53.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:53.833 7f373b8db700 1 -- 172.21.2.102:0/1532501073 >> 172.21.2.102:0/1532501073 conn(0x7f3734094d90 msgr2=0x7f373408a420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:53.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:53.833 7f373b8db700 1 -- 172.21.2.102:0/1532501073 shutdown_connections 2021-02-10T11:39:53.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:53.833 7f373b8db700 1 -- 172.21.2.102:0/1532501073 wait complete. 2021-02-10T11:39:53.837 INFO:teuthology.orchestra.run.gibba002.stdout:[ 2021-02-10T11:39:53.837 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:39:53.837 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "cephfs" 2021-02-10T11:39:53.837 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:39:53.837 INFO:teuthology.orchestra.run.gibba002.stdout:] 2021-02-10T11:39:53.842 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config set mon mon_allow_pool_delete True 2021-02-10T11:39:54.276 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:54.258 7f6ce651b700 1 Processor -- start 2021-02-10T11:39:54.276 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:54.258 7f6ce651b700 1 -- start start 2021-02-10T11:39:54.276 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:54.259 7f6ce651b700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f6ce00d1c70 0x7f6ce00cef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:54.276 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:54.259 7f6ce651b700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f6ce00cf580 0x7f6ce00cf980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:54.276 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:54.259 7f6ce651b700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ce00fe1c0 con 0x7f6ce00d1870 2021-02-10T11:39:54.276 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:54.259 7f6ce651b700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ce01b0cb0 con 0x7f6ce00d1c70 2021-02-10T11:39:54.276 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:54.259 7f6ce651b700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ce0109a60 con 0x7f6ce00cf580 2021-02-10T11:39:54.276 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:54.259 7f6cdf7fe700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f6ce00d1c70 0x7f6ce00cef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:54.277 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:54.259 7f6cdf7fe700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f6ce00d1c70 0x7f6ce00cef60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:40736/0 (socket says 172.21.2.102:40736) 2021-02-10T11:39:54.277 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:54.259 7f6cdf7fe700 1 -- 172.21.2.102:0/1892359534 learned_addr learned my addr 172.21.2.102:0/1892359534 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:39:54.277 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:54.259 7f6cdf7fe700 1 -- 172.21.2.102:0/1892359534 >> v2:172.21.2.113:3300/0 conn(0x7f6ce00cf580 msgr2=0x7f6ce00cf980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:54.277 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:54.259 7f6cdf7fe700 1 --2- 172.21.2.102:0/1892359534 >> v2:172.21.2.113:3300/0 conn(0x7f6ce00cf580 0x7f6ce00cf980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:54.277 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:54.259 7f6cdf7fe700 1 -- 172.21.2.102:0/1892359534 >> v1:172.21.2.113:6789/0 conn(0x7f6ce00d1870 legacy=0x7f6ce01b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:54.277 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:54.259 7f6cdf7fe700 1 -- 172.21.2.102:0/1892359534 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ce0088070 con 0x7f6ce00d1c70 2021-02-10T11:39:54.277 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:54.259 7f6cdf7fe700 1 --2- 172.21.2.102:0/1892359534 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6ce00d1c70 0x7f6ce00cef60 secure :-1 s=READY pgs=3290 cs=0 l=1 rev1=1 rx=0x7f6ccc006700 tx=0x7f6ccc009b80).ready entity=mon.0 client_cookie=23fd487e83505848 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:54.277 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:54.260 7f6cdeffd700 1 -- 172.21.2.102:0/1892359534 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6ccc00c0c0 con 0x7f6ce00d1c70 2021-02-10T11:39:54.277 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:54.260 7f6cdeffd700 1 -- 172.21.2.102:0/1892359534 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ccc00c440 con 0x7f6ce00d1c70 2021-02-10T11:39:54.277 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:54.260 7f6cdeffd700 1 -- 172.21.2.102:0/1892359534 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6ccc0009b0 con 0x7f6ce00d1c70 2021-02-10T11:39:54.277 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:54.260 7f6ce651b700 1 -- 172.21.2.102:0/1892359534 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6ce00d1c70 msgr2=0x7f6ce00cef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:54.277 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:54.260 7f6ce651b700 1 --2- 172.21.2.102:0/1892359534 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6ce00d1c70 0x7f6ce00cef60 secure :-1 s=READY pgs=3290 cs=0 l=1 rev1=1 rx=0x7f6ccc006700 tx=0x7f6ccc009b80).stop 2021-02-10T11:39:54.277 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:54.260 7f6ce651b700 1 -- 172.21.2.102:0/1892359534 shutdown_connections 2021-02-10T11:39:54.278 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:54.260 7f6ce651b700 1 --2- 172.21.2.102:0/1892359534 >> v2:172.21.2.113:3300/0 conn(0x7f6ce00cf580 0x7f6ce00cf980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:54.278 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:54.260 7f6ce651b700 1 --2- 172.21.2.102:0/1892359534 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6ce00d1c70 0x7f6ce00cef60 unknown :-1 s=CLOSED pgs=3290 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:54.278 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:54.260 7f6ce651b700 1 -- 172.21.2.102:0/1892359534 >> 172.21.2.102:0/1892359534 conn(0x7f6ce00ce420 msgr2=0x7f6ce00d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:54.278 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:54.260 7f6ce651b700 1 -- 172.21.2.102:0/1892359534 shutdown_connections 2021-02-10T11:39:54.278 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:54.260 7f6ce651b700 1 -- 172.21.2.102:0/1892359534 wait complete. 2021-02-10T11:39:54.278 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:54.275 7f6ce651b700 1 Processor -- start 2021-02-10T11:39:54.278 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:54.275 7f6ce651b700 1 -- start start 2021-02-10T11:39:54.280 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:54.275 7f6ce651b700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6ce00c3820 0x7f6ce0084cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:54.280 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:54.275 7f6ce651b700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6ce00a0290 0x7f6ce00852d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:54.280 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:54.275 7f6ce651b700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6ce0086270 0x7f6ce0086670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:54.280 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:54.275 7f6ce651b700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ce00c0700 con 0x7f6ce0086270 2021-02-10T11:39:54.280 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:54.275 7f6ce651b700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ce00858f0 con 0x7f6ce00c3820 2021-02-10T11:39:54.280 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:54.275 7f6ce651b700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ce0085b10 con 0x7f6ce00a0290 2021-02-10T11:39:54.280 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:54.275 7f6ce4b38700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6ce0086270 0x7f6ce0086670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:54.280 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:54.275 7f6ce4b38700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6ce0086270 0x7f6ce0086670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:40740/0 (socket says 172.21.2.102:40740) 2021-02-10T11:39:54.280 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:54.275 7f6ce4b38700 1 -- 172.21.2.102:0/4049514835 learned_addr learned my addr 172.21.2.102:0/4049514835 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:39:54.280 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:54.276 7f6ce4b38700 1 -- 172.21.2.102:0/4049514835 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6ce00a0290 msgr2=0x7f6ce00852d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:54.280 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:54.276 7f6ce4b38700 1 --2- 172.21.2.102:0/4049514835 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6ce00a0290 0x7f6ce00852d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:54.281 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:54.276 7f6ce4b38700 1 -- 172.21.2.102:0/4049514835 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6ce00c3820 msgr2=0x7f6ce0084cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:54.281 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:54.276 7f6ce4b38700 1 --2- 172.21.2.102:0/4049514835 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6ce00c3820 0x7f6ce0084cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:54.281 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:54.276 7f6ce4b38700 1 -- 172.21.2.102:0/4049514835 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ce0086c90 con 0x7f6ce0086270 2021-02-10T11:39:54.281 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:54.276 7f6ce4b38700 1 --2- 172.21.2.102:0/4049514835 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6ce0086270 0x7f6ce0086670 secure :-1 s=READY pgs=3291 cs=0 l=1 rev1=1 rx=0x7f6cd800b8c0 tx=0x7f6cd800bc60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:54.281 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:54.276 7f6cdcff9700 1 -- 172.21.2.102:0/4049514835 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6cd8005870 con 0x7f6ce0086270 2021-02-10T11:39:54.281 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:54.276 7f6cdcff9700 1 -- 172.21.2.102:0/4049514835 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cd8005b30 con 0x7f6ce0086270 2021-02-10T11:39:54.281 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:54.276 7f6ce651b700 1 -- 172.21.2.102:0/4049514835 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ce0086c90 con 0x7f6ce0086270 2021-02-10T11:39:54.281 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:54.276 7f6cdcff9700 1 -- 172.21.2.102:0/4049514835 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6cd8006030 con 0x7f6ce0086270 2021-02-10T11:39:54.281 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:54.276 7f6ce651b700 1 -- 172.21.2.102:0/4049514835 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ce0086c90 con 0x7f6ce0086270 2021-02-10T11:39:54.281 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:54.277 7f6cdcff9700 1 -- 172.21.2.102:0/4049514835 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f6cd8005d10 con 0x7f6ce0086270 2021-02-10T11:39:54.281 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:54.277 7f6cdcff9700 1 --2- 172.21.2.102:0/4049514835 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6cbc0186d0 0x7f6cbc01abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:54.281 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:54.277 7f6cdcff9700 1 -- 172.21.2.102:0/4049514835 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(189..189 src has 1..189) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f6cd80329f0 con 0x7f6ce0086270 2021-02-10T11:39:54.281 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:54.277 7f6cdffff700 1 --2- 172.21.2.102:0/4049514835 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6cbc0186d0 0x7f6cbc01abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:54.281 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:54.277 7f6cdffff700 1 --2- 172.21.2.102:0/4049514835 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6cbc0186d0 0x7f6cbc01abd0 crc :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:54.310 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:54.309 7f6ce651b700 1 -- 172.21.2.102:0/4049514835 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ce0082a70 con 0x7f6ce0086270 2021-02-10T11:39:54.312 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:54.311 7f6cdcff9700 1 -- 172.21.2.102:0/4049514835 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f6cd8005870 con 0x7f6ce0086270 2021-02-10T11:39:54.463 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:54.461 7f6ce651b700 1 -- 172.21.2.102:0/4049514835 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f6ce0082a70 con 0x7f6ce0086270 2021-02-10T11:39:54.463 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:54.461 7f6cdcff9700 1 -- 172.21.2.102:0/4049514835 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v46)=0 v46) v1 ==== 122+0+0 (secure 0 0 0) 0x7f6cd8006f20 con 0x7f6ce0086270 2021-02-10T11:39:54.465 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:54.463 7f6ce651b700 1 -- 172.21.2.102:0/4049514835 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6cbc0186d0 msgr2=0x7f6cbc01abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:54.465 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:54.463 7f6ce651b700 1 --2- 172.21.2.102:0/4049514835 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6cbc0186d0 0x7f6cbc01abd0 crc :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:54.465 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:54.463 7f6ce651b700 1 -- 172.21.2.102:0/4049514835 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6ce0086270 msgr2=0x7f6ce0086670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:54.465 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:54.463 7f6ce651b700 1 --2- 172.21.2.102:0/4049514835 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6ce0086270 0x7f6ce0086670 secure :-1 s=READY pgs=3291 cs=0 l=1 rev1=1 rx=0x7f6cd800b8c0 tx=0x7f6cd800bc60).stop 2021-02-10T11:39:54.465 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:54.464 7f6ce651b700 1 -- 172.21.2.102:0/4049514835 shutdown_connections 2021-02-10T11:39:54.465 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:54.464 7f6ce651b700 1 --2- 172.21.2.102:0/4049514835 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6ce0086270 0x7f6ce0086670 unknown :-1 s=CLOSED pgs=3291 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:54.465 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:54.464 7f6ce651b700 1 --2- 172.21.2.102:0/4049514835 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6ce00a0290 0x7f6ce00852d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:54.465 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:54.464 7f6ce651b700 1 --2- 172.21.2.102:0/4049514835 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6cbc0186d0 0x7f6cbc01abd0 unknown :-1 s=CLOSED pgs=1436 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:54.466 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:54.464 7f6ce651b700 1 --2- 172.21.2.102:0/4049514835 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6ce00c3820 0x7f6ce0084cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:54.466 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:54.464 7f6ce651b700 1 -- 172.21.2.102:0/4049514835 >> 172.21.2.102:0/4049514835 conn(0x7f6ce00872c0 msgr2=0x7f6ce0095700 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:54.466 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:54.464 7f6ce651b700 1 -- 172.21.2.102:0/4049514835 shutdown_connections 2021-02-10T11:39:54.466 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:54.464 7f6ce651b700 1 -- 172.21.2.102:0/4049514835 wait complete. 2021-02-10T11:39:54.473 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolumegroup create cephfs group_0000000000599689 2021-02-10T11:39:54.905 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:54.887 7efcbf90b700 1 Processor -- start 2021-02-10T11:39:54.906 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:54.887 7efcbf90b700 1 -- start start 2021-02-10T11:39:54.906 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:54.888 7efcbf90b700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7efcb810aff0 0x7efcb80cf750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:54.906 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:54.888 7efcbf90b700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcb80fe1c0 con 0x7efcb80d1c70 2021-02-10T11:39:54.906 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:54.888 7efcbf90b700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcb81b0cb0 con 0x7efcb80d1870 2021-02-10T11:39:54.906 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:54.888 7efcbf90b700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efcb8109a60 con 0x7efcb810aff0 2021-02-10T11:39:54.906 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:54.888 7efcbcf26700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7efcb80d1c70 0x7efcb80cef60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:32970/0 (socket says 172.21.2.102:32970) 2021-02-10T11:39:54.906 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:54.888 7efcbcf26700 1 -- 172.21.2.102:0/2813700745 learned_addr learned my addr 172.21.2.102:0/2813700745 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:39:54.906 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:54.888 7efcbdf28700 1 --2- 172.21.2.102:0/2813700745 >> v2:172.21.2.102:3300/0 conn(0x7efcb810aff0 0x7efcb80cf750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:54.906 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:54.888 7efcbdf28700 1 -- 172.21.2.102:0/2813700745 >> v1:172.21.2.113:6789/0 conn(0x7efcb80d1870 legacy=0x7efcb81b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:54.906 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:54.888 7efcbdf28700 1 -- 172.21.2.102:0/2813700745 >> v1:172.21.2.102:6789/0 conn(0x7efcb80d1c70 legacy=0x7efcb80cef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:54.906 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:54.889 7efcabfff700 1 -- 172.21.2.102:0/2813700745 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 425576589 0 0) 0x7efcb40010b0 con 0x7efcb80d1c70 2021-02-10T11:39:54.907 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:54.889 7efcbdf28700 1 -- 172.21.2.102:0/2813700745 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcb810b3f0 con 0x7efcb810aff0 2021-02-10T11:39:54.907 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:54.889 7efcbdf28700 1 --2- 172.21.2.102:0/2813700745 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efcb810aff0 0x7efcb80cf750 secure :-1 s=READY pgs=3293 cs=0 l=1 rev1=1 rx=0x7efcb0006e30 tx=0x7efcb0009e60).ready entity=mon.0 client_cookie=87c805ba9ac26e3b server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:54.907 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:54.889 7efcabfff700 1 -- 172.21.2.102:0/2813700745 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efcb0006a50 con 0x7efcb810aff0 2021-02-10T11:39:54.907 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:54.889 7efcabfff700 1 -- 172.21.2.102:0/2813700745 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcb000b300 con 0x7efcb810aff0 2021-02-10T11:39:54.907 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:54.889 7efcabfff700 1 -- 172.21.2.102:0/2813700745 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efcb000b820 con 0x7efcb810aff0 2021-02-10T11:39:54.907 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:54.889 7efcbf90b700 1 -- 172.21.2.102:0/2813700745 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efcb810aff0 msgr2=0x7efcb80cf750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:54.907 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:54.889 7efcbf90b700 1 --2- 172.21.2.102:0/2813700745 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efcb810aff0 0x7efcb80cf750 secure :-1 s=READY pgs=3293 cs=0 l=1 rev1=1 rx=0x7efcb0006e30 tx=0x7efcb0009e60).stop 2021-02-10T11:39:54.907 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:54.890 7efcbf90b700 1 -- 172.21.2.102:0/2813700745 shutdown_connections 2021-02-10T11:39:54.907 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:54.890 7efcbf90b700 1 --2- 172.21.2.102:0/2813700745 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efcb810aff0 0x7efcb80cf750 unknown :-1 s=CLOSED pgs=3293 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:54.907 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:54.890 7efcbf90b700 1 -- 172.21.2.102:0/2813700745 >> 172.21.2.102:0/2813700745 conn(0x7efcb80ce420 msgr2=0x7efcb80d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:54.907 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:54.890 7efcbf90b700 1 -- 172.21.2.102:0/2813700745 shutdown_connections 2021-02-10T11:39:54.907 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:54.890 7efcbf90b700 1 -- 172.21.2.102:0/2813700745 wait complete. 2021-02-10T11:39:54.909 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:54.904 7efcbf90b700 1 Processor -- start 2021-02-10T11:39:54.909 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:54.905 7efcbf90b700 1 -- start start 2021-02-10T11:39:54.909 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:54.905 7efcbf90b700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efcb80c3820 0x7efcb80a7950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:54.909 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:54.905 7efcbf90b700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7efcb81a44c0 0x7efcb81a48c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:54.909 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:54.905 7efcbf90b700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7efcb80863d0 0x7efcb80867d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:54.909 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:54.905 7efcbf90b700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcb8085ed0 con 0x7efcb80c3820 2021-02-10T11:39:54.909 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:54.905 7efcbf90b700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcb80c0740 con 0x7efcb80863d0 2021-02-10T11:39:54.909 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:54.905 7efcbf90b700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcb8086df0 con 0x7efcb81a44c0 2021-02-10T11:39:54.909 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:54.905 7efcbd727700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efcb80c3820 0x7efcb80a7950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:54.909 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:54.905 7efcbd727700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efcb80c3820 0x7efcb80a7950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:40752/0 (socket says 172.21.2.102:40752) 2021-02-10T11:39:54.910 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:54.905 7efcbd727700 1 -- 172.21.2.102:0/4168883059 learned_addr learned my addr 172.21.2.102:0/4168883059 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:39:54.910 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:54.906 7efcbd727700 1 -- 172.21.2.102:0/4168883059 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7efcb81a44c0 msgr2=0x7efcb81a48c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:54.910 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:54.906 7efcbd727700 1 --2- 172.21.2.102:0/4168883059 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7efcb81a44c0 0x7efcb81a48c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:54.910 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:54.906 7efcbd727700 1 -- 172.21.2.102:0/4168883059 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7efcb80863d0 msgr2=0x7efcb80867d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:54.910 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:54.906 7efcbd727700 1 --2- 172.21.2.102:0/4168883059 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7efcb80863d0 0x7efcb80867d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:54.910 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:54.906 7efcbd727700 1 -- 172.21.2.102:0/4168883059 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcb8087010 con 0x7efcb80c3820 2021-02-10T11:39:54.910 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:54.906 7efcbd727700 1 --2- 172.21.2.102:0/4168883059 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efcb80c3820 0x7efcb80a7950 secure :-1 s=READY pgs=3294 cs=0 l=1 rev1=1 rx=0x7efcac00b5b0 tx=0x7efcac00b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:54.910 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:54.906 7efca9ffb700 1 -- 172.21.2.102:0/4168883059 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efcac00bd40 con 0x7efcb80c3820 2021-02-10T11:39:54.910 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:54.906 7efca9ffb700 1 -- 172.21.2.102:0/4168883059 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcac000e20 con 0x7efcb80c3820 2021-02-10T11:39:54.910 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:54.906 7efca9ffb700 1 -- 172.21.2.102:0/4168883059 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efcac0054a0 con 0x7efcb80c3820 2021-02-10T11:39:54.910 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:54.906 7efcbf90b700 1 -- 172.21.2.102:0/4168883059 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efcb8087010 con 0x7efcb80c3820 2021-02-10T11:39:54.910 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:54.906 7efcbf90b700 1 -- 172.21.2.102:0/4168883059 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efcb8087010 con 0x7efcb80c3820 2021-02-10T11:39:54.910 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:54.906 7efca9ffb700 1 -- 172.21.2.102:0/4168883059 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7efcac006030 con 0x7efcb80c3820 2021-02-10T11:39:54.911 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:54.907 7efca9ffb700 1 --2- 172.21.2.102:0/4168883059 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7efc980186c0 0x7efc9801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:54.911 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:54.907 7efca9ffb700 1 -- 172.21.2.102:0/4168883059 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(189..189 src has 1..189) v4 ==== 5573+0+0 (secure 0 0 0) 0x7efcac032590 con 0x7efcb80c3820 2021-02-10T11:39:54.911 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:54.907 7efcbcf26700 1 --2- 172.21.2.102:0/4168883059 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7efc980186c0 0x7efc9801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:54.911 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:54.907 7efcbcf26700 1 --2- 172.21.2.102:0/4168883059 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7efc980186c0 0x7efc9801abc0 crc :-1 s=READY pgs=1437 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:54.939 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:54.938 7efcbf90b700 1 -- 172.21.2.102:0/4168883059 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efcb80fe1c0 con 0x7efcb80c3820 2021-02-10T11:39:54.941 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:54.940 7efca9ffb700 1 -- 172.21.2.102:0/4168883059 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7efcac02def0 con 0x7efcb80c3820 2021-02-10T11:39:55.069 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:55.068 7efcbf90b700 1 -- 172.21.2.102:0/4168883059 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "target": ["mgr", ""], "group_name": "group_0000000000599689"}) v1 -- 0x7efcb80fe1c0 con 0x7efc980186c0 2021-02-10T11:39:55.073 INFO:tasks.ceph.mgr.y.gibba002.stderr:2021-02-10 11:39:55.072 7fd94bc53700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:39:55.073 INFO:tasks.ceph.mgr.y.gibba002.stderr:2021-02-10 11:39:55.072 7fd94bc53700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:39:55.073 INFO:tasks.ceph.mgr.y.gibba002.stderr:2021-02-10 11:39:55.072 7fd94bc53700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:39:55.073 INFO:tasks.ceph.mgr.y.gibba002.stderr:2021-02-10 11:39:55.072 7fd94bc53700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:39:55.073 INFO:tasks.ceph.mgr.y.gibba002.stderr:2021-02-10 11:39:55.072 7fd94bc53700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:39:55.087 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:55.086 7efca9ffb700 1 -- 172.21.2.102:0/4168883059 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7efcb4000940 con 0x7efc980186c0 2021-02-10T11:39:55.102 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:55.101 7efcbf90b700 1 -- 172.21.2.102:0/4168883059 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7efc980186c0 msgr2=0x7efc9801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:55.102 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:55.101 7efcbf90b700 1 --2- 172.21.2.102:0/4168883059 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7efc980186c0 0x7efc9801abc0 crc :-1 s=READY pgs=1437 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:55.102 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:55.101 7efcbf90b700 1 -- 172.21.2.102:0/4168883059 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efcb80c3820 msgr2=0x7efcb80a7950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:55.102 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:55.101 7efcbf90b700 1 --2- 172.21.2.102:0/4168883059 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efcb80c3820 0x7efcb80a7950 secure :-1 s=READY pgs=3294 cs=0 l=1 rev1=1 rx=0x7efcac00b5b0 tx=0x7efcac00b950).stop 2021-02-10T11:39:55.103 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:55.101 7efcbf90b700 1 -- 172.21.2.102:0/4168883059 shutdown_connections 2021-02-10T11:39:55.103 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:55.101 7efcbf90b700 1 --2- 172.21.2.102:0/4168883059 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7efc980186c0 0x7efc9801abc0 unknown :-1 s=CLOSED pgs=1437 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:55.103 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:55.101 7efcbf90b700 1 --2- 172.21.2.102:0/4168883059 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7efcb80863d0 0x7efcb80867d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:55.103 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:55.101 7efcbf90b700 1 --2- 172.21.2.102:0/4168883059 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7efcb81a44c0 0x7efcb81a48c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:55.103 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:55.101 7efcbf90b700 1 --2- 172.21.2.102:0/4168883059 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7efcb80c3820 0x7efcb80a7950 unknown :-1 s=CLOSED pgs=3294 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:55.103 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:55.101 7efcbf90b700 1 -- 172.21.2.102:0/4168883059 >> 172.21.2.102:0/4168883059 conn(0x7efcb80872c0 msgr2=0x7efcb8095700 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:55.103 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:55.101 7efcbf90b700 1 -- 172.21.2.102:0/4168883059 shutdown_connections 2021-02-10T11:39:55.103 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:55.101 7efcbf90b700 1 -- 172.21.2.102:0/4168883059 wait complete. 2021-02-10T11:39:55.111 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000518037 --group_name group_0000000000599689 2021-02-10T11:39:55.545 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:55.526 7fc07eb8a700 1 Processor -- start 2021-02-10T11:39:55.545 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:55.526 7fc07eb8a700 1 -- start start 2021-02-10T11:39:55.545 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:55.526 7fc07eb8a700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fc0780d1c80 0x7fc0780cef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:55.545 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:55.526 7fc07eb8a700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc0780cf590 0x7fc0780cf990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:55.545 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:55.526 7fc07eb8a700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0780fe1d0 con 0x7fc0780d1880 2021-02-10T11:39:55.545 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:55.526 7fc07eb8a700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0781b0cc0 con 0x7fc0780d1c80 2021-02-10T11:39:55.545 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:55.526 7fc07eb8a700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc078109a70 con 0x7fc0780cf590 2021-02-10T11:39:55.545 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:55.527 7fc07c9a6700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fc0780d1880 0x7fc0781b04d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:33000/0 (socket says 172.21.2.102:33000) 2021-02-10T11:39:55.546 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:55.527 7fc07c9a6700 1 -- 172.21.2.102:0/3416536137 learned_addr learned my addr 172.21.2.102:0/3416536137 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:39:55.546 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:55.527 7fc0777fe700 1 -- 172.21.2.102:0/3416536137 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2332990987 0 0) 0x7fc0600009b0 con 0x7fc0780d1880 2021-02-10T11:39:55.546 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:55.527 7fc0777fe700 1 -- 172.21.2.102:0/3416536137 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc064002fe0 con 0x7fc0780d1880 2021-02-10T11:39:55.546 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:55.527 7fc077fff700 1 --2- 172.21.2.102:0/3416536137 >> v2:172.21.2.113:3300/0 conn(0x7fc0780d1c80 0x7fc0780cef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:55.546 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:55.527 7fc0777fe700 1 -- 172.21.2.102:0/3416536137 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2851479771 0 0) 0x7fc060001010 con 0x7fc0780d1880 2021-02-10T11:39:55.546 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:55.527 7fc07d1a7700 1 --2- 172.21.2.102:0/3416536137 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc0780cf590 0x7fc0780cf990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:55.546 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:55.527 7fc0777fe700 1 -- 172.21.2.102:0/3416536137 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc0780cf590 msgr2=0x7fc0780cf990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:55.546 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:55.527 7fc0777fe700 1 --2- 172.21.2.102:0/3416536137 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc0780cf590 0x7fc0780cf990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:55.546 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:55.527 7fc0777fe700 1 -- 172.21.2.102:0/3416536137 >> v2:172.21.2.113:3300/0 conn(0x7fc0780d1c80 msgr2=0x7fc0780cef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:55.546 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:55.527 7fc0777fe700 1 --2- 172.21.2.102:0/3416536137 >> v2:172.21.2.113:3300/0 conn(0x7fc0780d1c80 0x7fc0780cef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:55.546 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:55.527 7fc0777fe700 1 -- 172.21.2.102:0/3416536137 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc078088070 con 0x7fc0780d1880 2021-02-10T11:39:55.546 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:55.527 7fc0777fe700 1 -- 172.21.2.102:0/3416536137 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7fc060001530 con 0x7fc0780d1880 2021-02-10T11:39:55.547 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:55.527 7fc0777fe700 1 -- 172.21.2.102:0/3416536137 >> v1:172.21.2.102:6789/0 conn(0x7fc0780d1880 legacy=0x7fc0781b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:55.547 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:55.527 7fc0777fe700 1 --2- 172.21.2.102:0/3416536137 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc064003af0 0x7fc064006010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:55.547 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:55.527 7fc0777fe700 1 --2- 172.21.2.102:0/3416536137 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc064006830 0x7fc064008d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:55.547 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:55.527 7fc0777fe700 1 --2- 172.21.2.102:0/3416536137 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc0640095c0 0x7fc06400bac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:55.547 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:55.527 7fc0777fe700 1 -- 172.21.2.102:0/3416536137 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc06400c370 con 0x7fc064003af0 2021-02-10T11:39:55.547 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:55.527 7fc0777fe700 1 -- 172.21.2.102:0/3416536137 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc06400c590 con 0x7fc0640095c0 2021-02-10T11:39:55.547 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:55.527 7fc0777fe700 1 -- 172.21.2.102:0/3416536137 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0640507c0 con 0x7fc064006830 2021-02-10T11:39:55.547 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:55.527 7fc07c9a6700 1 --2- 172.21.2.102:0/3416536137 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc064003af0 0x7fc064006010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:55.547 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:55.527 7fc07d1a7700 1 --2- 172.21.2.102:0/3416536137 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc0780cf590 0x7fc0780cf990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:39:55.547 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:55.527 7fc077fff700 1 --2- 172.21.2.102:0/3416536137 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc064006830 0x7fc064008d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:55.547 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:55.528 7fc07d1a7700 1 --2- 172.21.2.102:0/3416536137 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc0640095c0 0x7fc06400bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:55.547 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:55.528 7fc07c9a6700 1 -- 172.21.2.102:0/3416536137 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc064006830 msgr2=0x7fc064008d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:55.547 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:55.528 7fc07c9a6700 1 --2- 172.21.2.102:0/3416536137 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc064006830 0x7fc064008d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:55.548 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:55.528 7fc07c9a6700 1 -- 172.21.2.102:0/3416536137 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc0640095c0 msgr2=0x7fc06400bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:55.548 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:55.528 7fc07c9a6700 1 --2- 172.21.2.102:0/3416536137 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc0640095c0 0x7fc06400bac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:55.548 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:55.528 7fc07c9a6700 1 -- 172.21.2.102:0/3416536137 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc064050a20 con 0x7fc064003af0 2021-02-10T11:39:55.548 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:55.528 7fc07c9a6700 1 --2- 172.21.2.102:0/3416536137 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc064003af0 0x7fc064006010 secure :-1 s=READY pgs=3298 cs=0 l=1 rev1=1 rx=0x7fc0600017e0 tx=0x7fc060007d40).ready entity=mon.0 client_cookie=95ea3915b0ed27ae server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:55.548 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:55.528 7fc0777fe700 1 -- 172.21.2.102:0/3416536137 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc0600009b0 con 0x7fc064003af0 2021-02-10T11:39:55.548 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:55.528 7fc0777fe700 1 -- 172.21.2.102:0/3416536137 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc060001240 con 0x7fc064003af0 2021-02-10T11:39:55.548 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:55.528 7fc0777fe700 1 -- 172.21.2.102:0/3416536137 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc06000a520 con 0x7fc064003af0 2021-02-10T11:39:55.548 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:55.528 7fc07eb8a700 1 -- 172.21.2.102:0/3416536137 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc064003af0 msgr2=0x7fc064006010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:55.548 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:55.528 7fc07eb8a700 1 --2- 172.21.2.102:0/3416536137 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc064003af0 0x7fc064006010 secure :-1 s=READY pgs=3298 cs=0 l=1 rev1=1 rx=0x7fc0600017e0 tx=0x7fc060007d40).stop 2021-02-10T11:39:55.548 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:55.528 7fc07d1a7700 1 -- 172.21.2.102:0/3416536137 reap_dead start 2021-02-10T11:39:55.548 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:55.529 7fc07eb8a700 1 -- 172.21.2.102:0/3416536137 shutdown_connections 2021-02-10T11:39:55.548 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:55.529 7fc07eb8a700 1 -- 172.21.2.102:0/3416536137 >> 172.21.2.102:0/3416536137 conn(0x7fc0780ce430 msgr2=0x7fc0780d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:55.549 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:55.529 7fc07eb8a700 1 -- 172.21.2.102:0/3416536137 shutdown_connections 2021-02-10T11:39:55.549 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:55.529 7fc07eb8a700 1 -- 172.21.2.102:0/3416536137 wait complete. 2021-02-10T11:39:55.549 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:55.544 7fc07eb8a700 1 Processor -- start 2021-02-10T11:39:55.549 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:55.544 7fc07eb8a700 1 -- start start 2021-02-10T11:39:55.550 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:55.544 7fc07eb8a700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc0781a3570 0x7fc0781a3970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:55.550 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:55.544 7fc07eb8a700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc07809c9f0 0x7fc07809cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:55.550 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:55.544 7fc07eb8a700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc07809d410 0x7fc078098170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:55.551 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:55.544 7fc07eb8a700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc07809c470 con 0x7fc0781a3570 2021-02-10T11:39:55.551 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:55.544 7fc07eb8a700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0780b2840 con 0x7fc07809c9f0 2021-02-10T11:39:55.551 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:55.544 7fc07eb8a700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0780b2a20 con 0x7fc07809d410 2021-02-10T11:39:55.551 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:55.544 7fc07c9a6700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc0781a3570 0x7fc0781a3970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:55.551 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:55.544 7fc07c9a6700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc0781a3570 0x7fc0781a3970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:40790/0 (socket says 172.21.2.102:40790) 2021-02-10T11:39:55.551 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:55.544 7fc07c9a6700 1 -- 172.21.2.102:0/1120117418 learned_addr learned my addr 172.21.2.102:0/1120117418 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:39:55.551 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:55.545 7fc07c9a6700 1 -- 172.21.2.102:0/1120117418 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc07809d410 msgr2=0x7fc078098170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:55.551 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:55.545 7fc07c9a6700 1 --2- 172.21.2.102:0/1120117418 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc07809d410 0x7fc078098170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:55.551 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:55.545 7fc07c9a6700 1 -- 172.21.2.102:0/1120117418 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc07809c9f0 msgr2=0x7fc07809cdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:55.551 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:55.545 7fc07c9a6700 1 --2- 172.21.2.102:0/1120117418 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc07809c9f0 0x7fc07809cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:55.552 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:55.545 7fc07c9a6700 1 -- 172.21.2.102:0/1120117418 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc07809d880 con 0x7fc0781a3570 2021-02-10T11:39:55.552 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:55.545 7fc07c9a6700 1 --2- 172.21.2.102:0/1120117418 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc0781a3570 0x7fc0781a3970 secure :-1 s=READY pgs=3299 cs=0 l=1 rev1=1 rx=0x7fc060001660 tx=0x7fc06000a900).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:55.552 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:55.545 7fc0757fa700 1 -- 172.21.2.102:0/1120117418 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc06000acf0 con 0x7fc0781a3570 2021-02-10T11:39:55.552 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:55.545 7fc0757fa700 1 -- 172.21.2.102:0/1120117418 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0600009b0 con 0x7fc0781a3570 2021-02-10T11:39:55.552 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:55.545 7fc0757fa700 1 -- 172.21.2.102:0/1120117418 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc060004540 con 0x7fc0781a3570 2021-02-10T11:39:55.552 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:55.545 7fc07eb8a700 1 -- 172.21.2.102:0/1120117418 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc07809d880 con 0x7fc0781a3570 2021-02-10T11:39:55.552 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:55.545 7fc07eb8a700 1 -- 172.21.2.102:0/1120117418 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc078098a80 con 0x7fc0781a3570 2021-02-10T11:39:55.552 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:55.546 7fc0757fa700 1 -- 172.21.2.102:0/1120117418 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fc060000df0 con 0x7fc0781a3570 2021-02-10T11:39:55.552 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:55.546 7fc0757fa700 1 --2- 172.21.2.102:0/1120117418 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc05c0186c0 0x7fc05c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:55.552 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:55.546 7fc0757fa700 1 -- 172.21.2.102:0/1120117418 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(189..189 src has 1..189) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fc060032410 con 0x7fc0781a3570 2021-02-10T11:39:55.552 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:55.546 7fc077fff700 1 --2- 172.21.2.102:0/1120117418 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc05c0186c0 0x7fc05c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:55.552 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:55.546 7fc077fff700 1 --2- 172.21.2.102:0/1120117418 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc05c0186c0 0x7fc05c01abc0 crc :-1 s=READY pgs=1438 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:55.578 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:55.576 7fc07eb8a700 1 -- 172.21.2.102:0/1120117418 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc078082a70 con 0x7fc0781a3570 2021-02-10T11:39:55.580 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:55.579 7fc0757fa700 1 -- 172.21.2.102:0/1120117418 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fc0600258a0 con 0x7fc0781a3570 2021-02-10T11:39:55.705 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:55.704 7fc07eb8a700 1 -- 172.21.2.102:0/1120117418 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000518037", "target": ["mgr", ""], "group_name": "group_0000000000599689"}) v1 -- 0x7fc07809c470 con 0x7fc05c0186c0 2021-02-10T11:39:55.729 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:55.728 7fc0757fa700 1 -- 172.21.2.102:0/1120117418 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fc068000d30 con 0x7fc05c0186c0 2021-02-10T11:39:55.738 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:55.737 7fc07eb8a700 1 -- 172.21.2.102:0/1120117418 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc05c0186c0 msgr2=0x7fc05c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:55.738 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:55.737 7fc07eb8a700 1 --2- 172.21.2.102:0/1120117418 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc05c0186c0 0x7fc05c01abc0 crc :-1 s=READY pgs=1438 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:55.738 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:55.737 7fc07eb8a700 1 -- 172.21.2.102:0/1120117418 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc0781a3570 msgr2=0x7fc0781a3970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:55.738 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:55.737 7fc07eb8a700 1 --2- 172.21.2.102:0/1120117418 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc0781a3570 0x7fc0781a3970 secure :-1 s=READY pgs=3299 cs=0 l=1 rev1=1 rx=0x7fc060001660 tx=0x7fc06000a900).stop 2021-02-10T11:39:55.739 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:55.737 7fc07eb8a700 1 -- 172.21.2.102:0/1120117418 shutdown_connections 2021-02-10T11:39:55.739 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:55.737 7fc07eb8a700 1 --2- 172.21.2.102:0/1120117418 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc07809d410 0x7fc078098170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:55.739 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:55.737 7fc07eb8a700 1 --2- 172.21.2.102:0/1120117418 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc05c0186c0 0x7fc05c01abc0 unknown :-1 s=CLOSED pgs=1438 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:55.739 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:55.737 7fc07eb8a700 1 --2- 172.21.2.102:0/1120117418 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc07809c9f0 0x7fc07809cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:55.739 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:55.737 7fc07eb8a700 1 --2- 172.21.2.102:0/1120117418 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc0781a3570 0x7fc0781a3970 unknown :-1 s=CLOSED pgs=3299 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:55.739 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:55.737 7fc07eb8a700 1 -- 172.21.2.102:0/1120117418 >> 172.21.2.102:0/1120117418 conn(0x7fc0780c2cd0 msgr2=0x7fc0780b5950 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:55.739 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:55.737 7fc07eb8a700 1 -- 172.21.2.102:0/1120117418 shutdown_connections 2021-02-10T11:39:55.739 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:55.737 7fc07eb8a700 1 -- 172.21.2.102:0/1120117418 wait complete. 2021-02-10T11:39:55.747 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume authorize cephfs subvolume_0000000000518037 alice --group_name group_0000000000599689 2021-02-10T11:39:56.180 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:56.161 7f5668a8c700 1 Processor -- start 2021-02-10T11:39:56.180 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:56.161 7f5668a8c700 1 -- start start 2021-02-10T11:39:56.180 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:56.162 7f5668a8c700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f56640eb220 0x7f56640eb620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:56.180 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:56.162 7f5668a8c700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f566410aad0 0x7f566410aed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:56.181 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:56.162 7f5668a8c700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f566410b4f0 con 0x7f56640dce80 2021-02-10T11:39:56.181 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:56.162 7f5668a8c700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f566410a550 con 0x7f566410aad0 2021-02-10T11:39:56.181 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:56.162 7f5668a8c700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56640881d0 con 0x7f56640eb220 2021-02-10T11:39:56.181 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:56.163 7f566305f700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f566410aad0 0x7f566410aed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:56.181 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:56.163 7f566305f700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f566410aad0 0x7f566410aed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.113:3300/0 says I am v2:172.21.2.102:44226/0 (socket says 172.21.2.102:44226) 2021-02-10T11:39:56.181 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:56.163 7f566305f700 1 -- 172.21.2.102:0/2564291025 learned_addr learned my addr 172.21.2.102:0/2564291025 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:39:56.181 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:56.163 7f566205d700 1 --2- 172.21.2.102:0/2564291025 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f56640eb220 0x7f56640eb620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:56.181 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:56.163 7f566305f700 1 -- 172.21.2.102:0/2564291025 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f56640eb220 msgr2=0x7f56640eb620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:56.181 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:56.163 7f566305f700 1 --2- 172.21.2.102:0/2564291025 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f56640eb220 0x7f56640eb620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:56.181 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:56.163 7f566305f700 1 -- 172.21.2.102:0/2564291025 >> v1:172.21.2.113:6789/0 conn(0x7f56640dce80 legacy=0x7f56640d14f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:56.181 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:56.163 7f566305f700 1 -- 172.21.2.102:0/2564291025 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56640d1f30 con 0x7f566410aad0 2021-02-10T11:39:56.181 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:56.163 7f566205d700 1 --2- 172.21.2.102:0/2564291025 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f56640eb220 0x7f56640eb620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:39:56.181 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:56.164 7f566305f700 1 --2- 172.21.2.102:0/2564291025 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f566410aad0 0x7f566410aed0 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 rx=0x7f565c006aa0 tx=0x7f565c009f20).ready entity=mon.1 client_cookie=fa66375bafe3e794 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:56.182 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:56.164 7f566185c700 1 -- 172.21.2.102:0/2564291025 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f565c00b5c0 con 0x7f566410aad0 2021-02-10T11:39:56.182 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:56.164 7f566185c700 1 -- 172.21.2.102:0/2564291025 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f565c003030 con 0x7f566410aad0 2021-02-10T11:39:56.182 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:56.164 7f566185c700 1 -- 172.21.2.102:0/2564291025 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f565c00bab0 con 0x7f566410aad0 2021-02-10T11:39:56.182 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:56.164 7f5668a8c700 1 -- 172.21.2.102:0/2564291025 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f566410aad0 msgr2=0x7f566410aed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:56.182 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:56.164 7f5668a8c700 1 --2- 172.21.2.102:0/2564291025 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f566410aad0 0x7f566410aed0 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 rx=0x7f565c006aa0 tx=0x7f565c009f20).stop 2021-02-10T11:39:56.182 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:56.165 7f5668a8c700 1 -- 172.21.2.102:0/2564291025 shutdown_connections 2021-02-10T11:39:56.182 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:56.165 7f5668a8c700 1 --2- 172.21.2.102:0/2564291025 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f56640eb220 0x7f56640eb620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:56.182 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:56.165 7f5668a8c700 1 --2- 172.21.2.102:0/2564291025 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f566410aad0 0x7f566410aed0 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:56.182 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:56.165 7f5668a8c700 1 -- 172.21.2.102:0/2564291025 >> 172.21.2.102:0/2564291025 conn(0x7f56640ea630 msgr2=0x7f56640d0800 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:56.182 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:56.165 7f5668a8c700 1 -- 172.21.2.102:0/2564291025 shutdown_connections 2021-02-10T11:39:56.182 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:56.165 7f5668a8c700 1 -- 172.21.2.102:0/2564291025 wait complete. 2021-02-10T11:39:56.184 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:56.179 7f5668a8c700 1 Processor -- start 2021-02-10T11:39:56.184 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:56.179 7f5668a8c700 1 -- start start 2021-02-10T11:39:56.184 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:56.180 7f5668a8c700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5664105910 0x7f56640b2ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:56.184 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:56.180 7f5668a8c700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5664097640 0x7f56641a7500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:56.184 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:56.180 7f5668a8c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5664097a40 0x7f56641a7b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:56.184 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:56.180 7f5668a8c700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5664107950 con 0x7f5664097a40 2021-02-10T11:39:56.184 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:56.180 7f5668a8c700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5664105e50 con 0x7f5664097640 2021-02-10T11:39:56.184 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:56.180 7f5668a8c700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56641a8140 con 0x7f5664105910 2021-02-10T11:39:56.184 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:56.180 7f566305f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5664097a40 0x7f56641a7b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:56.184 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:56.180 7f566305f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5664097a40 0x7f56641a7b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:40800/0 (socket says 172.21.2.102:40800) 2021-02-10T11:39:56.185 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:56.180 7f566305f700 1 -- 172.21.2.102:0/243667098 learned_addr learned my addr 172.21.2.102:0/243667098 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:39:56.185 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:56.180 7f566305f700 1 -- 172.21.2.102:0/243667098 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5664105910 msgr2=0x7f56640b2ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:56.185 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:56.180 7f566305f700 1 --2- 172.21.2.102:0/243667098 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5664105910 0x7f56640b2ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:56.185 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:56.180 7f566305f700 1 -- 172.21.2.102:0/243667098 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5664097640 msgr2=0x7f56641a7500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:56.185 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:56.180 7f566305f700 1 --2- 172.21.2.102:0/243667098 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5664097640 0x7f56641a7500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:56.185 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:56.180 7f566305f700 1 -- 172.21.2.102:0/243667098 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56641a8320 con 0x7f5664097a40 2021-02-10T11:39:56.185 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:56.180 7f566305f700 1 --2- 172.21.2.102:0/243667098 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5664097a40 0x7f56641a7b20 secure :-1 s=READY pgs=3300 cs=0 l=1 rev1=1 rx=0x7f565c00b970 tx=0x7f565c00bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:56.185 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:56.181 7f56537fe700 1 -- 172.21.2.102:0/243667098 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f565c009450 con 0x7f5664097a40 2021-02-10T11:39:56.185 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:56.181 7f56537fe700 1 -- 172.21.2.102:0/243667098 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f565c00c2f0 con 0x7f5664097a40 2021-02-10T11:39:56.185 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:56.181 7f56537fe700 1 -- 172.21.2.102:0/243667098 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f565c00c780 con 0x7f5664097a40 2021-02-10T11:39:56.185 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:56.181 7f5668a8c700 1 -- 172.21.2.102:0/243667098 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56641a8320 con 0x7f5664097a40 2021-02-10T11:39:56.185 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:56.181 7f5668a8c700 1 -- 172.21.2.102:0/243667098 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56641066c0 con 0x7f5664097a40 2021-02-10T11:39:56.185 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:56.181 7f56537fe700 1 -- 172.21.2.102:0/243667098 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f565c009fb0 con 0x7f5664097a40 2021-02-10T11:39:56.185 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:56.181 7f56537fe700 1 --2- 172.21.2.102:0/243667098 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f56480186c0 0x7f564801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:56.186 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:56.181 7f56537fe700 1 -- 172.21.2.102:0/243667098 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(189..189 src has 1..189) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f565c0328d0 con 0x7f5664097a40 2021-02-10T11:39:56.186 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:56.182 7f566285e700 1 --2- 172.21.2.102:0/243667098 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f56480186c0 0x7f564801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:56.186 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:56.182 7f566285e700 1 --2- 172.21.2.102:0/243667098 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f56480186c0 0x7f564801abc0 crc :-1 s=READY pgs=1439 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:56.213 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:56.212 7f5668a8c700 1 -- 172.21.2.102:0/243667098 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56640f93d0 con 0x7f5664097a40 2021-02-10T11:39:56.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:56.214 7f56537fe700 1 -- 172.21.2.102:0/243667098 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f565c009450 con 0x7f5664097a40 2021-02-10T11:39:56.342 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:56.341 7f5668a8c700 1 -- 172.21.2.102:0/243667098 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume authorize", "vol_name": "cephfs", "sub_name": "subvolume_0000000000518037", "target": ["mgr", ""], "auth_id": "alice", "group_name": "group_0000000000599689"}) v1 -- 0x7f56641066c0 con 0x7f56480186c0 2021-02-10T11:39:56.346 INFO:tasks.ceph.mgr.y.gibba002.stderr:Creating meta for ID alice with tenant None 2021-02-10T11:39:56.373 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:56.372 7f56537fe700 1 -- 172.21.2.102:0/243667098 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f5654006e40 con 0x7f56480186c0 2021-02-10T11:39:56.375 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:56.374 7f5668a8c700 1 -- 172.21.2.102:0/243667098 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f56480186c0 msgr2=0x7f564801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:56.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:56.374 7f5668a8c700 1 --2- 172.21.2.102:0/243667098 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f56480186c0 0x7f564801abc0 crc :-1 s=READY pgs=1439 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:56.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:56.374 7f5668a8c700 1 -- 172.21.2.102:0/243667098 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5664097a40 msgr2=0x7f56641a7b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:56.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:56.374 7f5668a8c700 1 --2- 172.21.2.102:0/243667098 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5664097a40 0x7f56641a7b20 secure :-1 s=READY pgs=3300 cs=0 l=1 rev1=1 rx=0x7f565c00b970 tx=0x7f565c00bd10).stop 2021-02-10T11:39:56.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:56.374 7f5668a8c700 1 -- 172.21.2.102:0/243667098 shutdown_connections 2021-02-10T11:39:56.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:56.374 7f5668a8c700 1 --2- 172.21.2.102:0/243667098 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5664097a40 0x7f56641a7b20 unknown :-1 s=CLOSED pgs=3300 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:56.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:56.374 7f5668a8c700 1 --2- 172.21.2.102:0/243667098 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f56480186c0 0x7f564801abc0 unknown :-1 s=CLOSED pgs=1439 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:56.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:56.374 7f5668a8c700 1 --2- 172.21.2.102:0/243667098 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5664097640 0x7f56641a7500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:56.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:56.374 7f5668a8c700 1 --2- 172.21.2.102:0/243667098 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5664105910 0x7f56640b2ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:56.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:56.374 7f5668a8c700 1 -- 172.21.2.102:0/243667098 >> 172.21.2.102:0/243667098 conn(0x7f56640bfbf0 msgr2=0x7f56640a1650 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:56.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:56.374 7f5668a8c700 1 -- 172.21.2.102:0/243667098 shutdown_connections 2021-02-10T11:39:56.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:56.374 7f5668a8c700 1 -- 172.21.2.102:0/243667098 wait complete. 2021-02-10T11:39:56.383 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume authorize cephfs subvolume_0000000000518037 guest1 --group_name group_0000000000599689 2021-02-10T11:39:56.814 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:56.797 7f54e925d700 1 Processor -- start 2021-02-10T11:39:56.815 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:56.797 7f54e925d700 1 -- start start 2021-02-10T11:39:56.815 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:56.798 7f54e925d700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f54e40dce80 0x7f54e40d14f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:56.815 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:56.798 7f54e925d700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f54e410b2c0 0x7f54e40eb620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:56.815 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:56.798 7f54e925d700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54e410a550 con 0x7f54e40eb220 2021-02-10T11:39:56.815 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:56.798 7f54e925d700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54e40881d0 con 0x7f54e410b2c0 2021-02-10T11:39:56.815 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:56.798 7f54e925d700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54e40d1f30 con 0x7f54e40dce80 2021-02-10T11:39:56.815 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:56.798 7f54e359e700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f54e410b2c0 0x7f54e40eb620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:56.815 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:56.798 7f54e359e700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f54e410b2c0 0x7f54e40eb620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:40812/0 (socket says 172.21.2.102:40812) 2021-02-10T11:39:56.815 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:56.798 7f54e359e700 1 -- 172.21.2.102:0/1486446448 learned_addr learned my addr 172.21.2.102:0/1486446448 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:39:56.815 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:56.798 7f54e259c700 1 --1- 172.21.2.102:0/1486446448 >> v1:172.21.2.102:6789/0 conn(0x7f54e40eb220 0x7f54e410aad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:33040/0 (socket says 172.21.2.102:33040) 2021-02-10T11:39:56.815 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:56.798 7f54e359e700 1 -- 172.21.2.102:0/1486446448 >> v2:172.21.2.113:3300/0 conn(0x7f54e40dce80 msgr2=0x7f54e40d14f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:39:56.816 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:56.798 7f54e359e700 1 --2- 172.21.2.102:0/1486446448 >> v2:172.21.2.113:3300/0 conn(0x7f54e40dce80 0x7f54e40d14f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:56.816 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:56.798 7f54e359e700 1 -- 172.21.2.102:0/1486446448 >> v1:172.21.2.102:6789/0 conn(0x7f54e40eb220 legacy=0x7f54e410aad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:56.816 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:56.798 7f54e359e700 1 -- 172.21.2.102:0/1486446448 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54e40dd280 con 0x7f54e410b2c0 2021-02-10T11:39:56.816 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:56.798 7f54e359e700 1 --2- 172.21.2.102:0/1486446448 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f54e410b2c0 0x7f54e40eb620 secure :-1 s=READY pgs=3302 cs=0 l=1 rev1=1 rx=0x7f54dc006f00 tx=0x7f54dc009f10).ready entity=mon.0 client_cookie=b8bf74db6026e4a5 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:56.816 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:56.798 7f54e1d9b700 1 -- 172.21.2.102:0/1486446448 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f54dc00b3b0 con 0x7f54e410b2c0 2021-02-10T11:39:56.816 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:56.799 7f54e1d9b700 1 -- 172.21.2.102:0/1486446448 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54dc00b630 con 0x7f54e410b2c0 2021-02-10T11:39:56.816 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:56.799 7f54e1d9b700 1 -- 172.21.2.102:0/1486446448 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f54dc00bb10 con 0x7f54e410b2c0 2021-02-10T11:39:56.816 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:56.799 7f54e925d700 1 -- 172.21.2.102:0/1486446448 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f54e410b2c0 msgr2=0x7f54e40eb620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:56.816 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:56.799 7f54e925d700 1 --2- 172.21.2.102:0/1486446448 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f54e410b2c0 0x7f54e40eb620 secure :-1 s=READY pgs=3302 cs=0 l=1 rev1=1 rx=0x7f54dc006f00 tx=0x7f54dc009f10).stop 2021-02-10T11:39:56.816 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:56.799 7f54e925d700 1 -- 172.21.2.102:0/1486446448 shutdown_connections 2021-02-10T11:39:56.816 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:56.799 7f54e925d700 1 --2- 172.21.2.102:0/1486446448 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f54e410b2c0 0x7f54e40eb620 unknown :-1 s=CLOSED pgs=3302 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:56.816 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:56.799 7f54e925d700 1 --2- 172.21.2.102:0/1486446448 >> v2:172.21.2.113:3300/0 conn(0x7f54e40dce80 0x7f54e40d14f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:56.816 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:56.799 7f54e925d700 1 -- 172.21.2.102:0/1486446448 >> 172.21.2.102:0/1486446448 conn(0x7f54e40ea630 msgr2=0x7f54e40d0800 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:56.816 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:56.799 7f54e925d700 1 -- 172.21.2.102:0/1486446448 shutdown_connections 2021-02-10T11:39:56.817 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:56.799 7f54e925d700 1 -- 172.21.2.102:0/1486446448 wait complete. 2021-02-10T11:39:56.818 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:56.814 7f54e925d700 1 Processor -- start 2021-02-10T11:39:56.818 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:56.814 7f54e925d700 1 -- start start 2021-02-10T11:39:56.818 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:56.814 7f54e925d700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f54e4105010 0x7f54e40944f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:56.818 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:56.814 7f54e925d700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f54e4105410 0x7f54e419a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:56.818 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:56.814 7f54e925d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f54e419a6f0 0x7f54e4103860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:56.819 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:56.814 7f54e925d700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54e4199bd0 con 0x7f54e419a6f0 2021-02-10T11:39:56.819 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:56.814 7f54e925d700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54e41b5240 con 0x7f54e4105410 2021-02-10T11:39:56.819 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:56.814 7f54e925d700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54e41b5430 con 0x7f54e4105010 2021-02-10T11:39:56.819 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:56.814 7f54e359e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f54e419a6f0 0x7f54e4103860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:56.819 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:56.814 7f54e359e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f54e419a6f0 0x7f54e4103860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:40820/0 (socket says 172.21.2.102:40820) 2021-02-10T11:39:56.819 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:56.814 7f54e359e700 1 -- 172.21.2.102:0/4294724068 learned_addr learned my addr 172.21.2.102:0/4294724068 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:39:56.819 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:56.814 7f54e359e700 1 -- 172.21.2.102:0/4294724068 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f54e4105010 msgr2=0x7f54e40944f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:56.819 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:56.814 7f54e359e700 1 --2- 172.21.2.102:0/4294724068 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f54e4105010 0x7f54e40944f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:56.819 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:56.815 7f54e359e700 1 -- 172.21.2.102:0/4294724068 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f54e4105410 msgr2=0x7f54e419a0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:56.819 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:56.815 7f54e359e700 1 --2- 172.21.2.102:0/4294724068 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f54e4105410 0x7f54e419a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:56.819 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:56.815 7f54e359e700 1 -- 172.21.2.102:0/4294724068 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54e4094b40 con 0x7f54e419a6f0 2021-02-10T11:39:56.819 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:56.815 7f54e359e700 1 --2- 172.21.2.102:0/4294724068 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f54e419a6f0 0x7f54e4103860 secure :-1 s=READY pgs=3303 cs=0 l=1 rev1=1 rx=0x7f54dc009390 tx=0x7f54dc00bd40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:56.819 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:56.815 7f54db7fe700 1 -- 172.21.2.102:0/4294724068 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f54dc003030 con 0x7f54e419a6f0 2021-02-10T11:39:56.820 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:56.815 7f54db7fe700 1 -- 172.21.2.102:0/4294724068 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54dc0068e0 con 0x7f54e419a6f0 2021-02-10T11:39:56.820 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:56.815 7f54db7fe700 1 -- 172.21.2.102:0/4294724068 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f54dc00c4e0 con 0x7f54e419a6f0 2021-02-10T11:39:56.820 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:56.815 7f54e925d700 1 -- 172.21.2.102:0/4294724068 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54e4094b40 con 0x7f54e419a6f0 2021-02-10T11:39:56.820 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:56.815 7f54e925d700 1 -- 172.21.2.102:0/4294724068 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54e4094b40 con 0x7f54e419a6f0 2021-02-10T11:39:56.820 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:56.816 7f54db7fe700 1 -- 172.21.2.102:0/4294724068 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f54dc001270 con 0x7f54e419a6f0 2021-02-10T11:39:56.820 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:56.816 7f54db7fe700 1 --2- 172.21.2.102:0/4294724068 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f54c80186c0 0x7f54c801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:56.820 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:56.816 7f54db7fe700 1 -- 172.21.2.102:0/4294724068 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(189..189 src has 1..189) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f54dc0328c0 con 0x7f54e419a6f0 2021-02-10T11:39:56.820 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:56.816 7f54e2d9d700 1 --2- 172.21.2.102:0/4294724068 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f54c80186c0 0x7f54c801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:56.820 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:56.816 7f54e2d9d700 1 --2- 172.21.2.102:0/4294724068 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f54c80186c0 0x7f54c801abc0 crc :-1 s=READY pgs=1440 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:56.849 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:56.848 7f54e925d700 1 -- 172.21.2.102:0/4294724068 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54e40f9360 con 0x7f54e419a6f0 2021-02-10T11:39:56.852 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:56.851 7f54db7fe700 1 -- 172.21.2.102:0/4294724068 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f54dc006c80 con 0x7f54e419a6f0 2021-02-10T11:39:56.977 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:56.976 7f54e925d700 1 -- 172.21.2.102:0/4294724068 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume authorize", "vol_name": "cephfs", "sub_name": "subvolume_0000000000518037", "target": ["mgr", ""], "auth_id": "guest1", "group_name": "group_0000000000599689"}) v1 -- 0x7f54e40f9360 con 0x7f54c80186c0 2021-02-10T11:39:56.982 INFO:tasks.ceph.mgr.y.gibba002.stderr:Creating meta for ID guest1 with tenant None 2021-02-10T11:39:57.014 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:57.012 7f54db7fe700 1 -- 172.21.2.102:0/4294724068 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f54cc006e40 con 0x7f54c80186c0 2021-02-10T11:39:57.043 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:57.041 7f54e925d700 1 -- 172.21.2.102:0/4294724068 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f54c80186c0 msgr2=0x7f54c801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:57.043 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:57.041 7f54e925d700 1 --2- 172.21.2.102:0/4294724068 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f54c80186c0 0x7f54c801abc0 crc :-1 s=READY pgs=1440 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:57.043 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:57.041 7f54e925d700 1 -- 172.21.2.102:0/4294724068 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f54e419a6f0 msgr2=0x7f54e4103860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:57.043 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:57.041 7f54e925d700 1 --2- 172.21.2.102:0/4294724068 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f54e419a6f0 0x7f54e4103860 secure :-1 s=READY pgs=3303 cs=0 l=1 rev1=1 rx=0x7f54dc009390 tx=0x7f54dc00bd40).stop 2021-02-10T11:39:57.043 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:57.041 7f54e925d700 1 -- 172.21.2.102:0/4294724068 shutdown_connections 2021-02-10T11:39:57.044 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:57.041 7f54e925d700 1 --2- 172.21.2.102:0/4294724068 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f54e419a6f0 0x7f54e4103860 unknown :-1 s=CLOSED pgs=3303 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:57.044 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:57.041 7f54e925d700 1 --2- 172.21.2.102:0/4294724068 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f54c80186c0 0x7f54c801abc0 unknown :-1 s=CLOSED pgs=1440 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:57.044 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:57.041 7f54e925d700 1 --2- 172.21.2.102:0/4294724068 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f54e4105410 0x7f54e419a0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:57.044 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:57.041 7f54e925d700 1 --2- 172.21.2.102:0/4294724068 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f54e4105010 0x7f54e40944f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:57.044 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:57.041 7f54e925d700 1 -- 172.21.2.102:0/4294724068 >> 172.21.2.102:0/4294724068 conn(0x7f54e41b0d40 msgr2=0x7f54e41a7230 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:57.044 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:57.041 7f54e925d700 1 -- 172.21.2.102:0/4294724068 shutdown_connections 2021-02-10T11:39:57.044 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:57.041 7f54e925d700 1 -- 172.21.2.102:0/4294724068 wait complete. 2021-02-10T11:39:57.051 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume authorize cephfs subvolume_0000000000518037 guest2 --group_name group_0000000000599689 --access_level r 2021-02-10T11:39:57.491 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:57.473 7f920ff09700 1 Processor -- start 2021-02-10T11:39:57.491 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:57.473 7f920ff09700 1 -- start start 2021-02-10T11:39:57.491 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:57.473 7f920ff09700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f920810a040 0x7f920810a440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:57.491 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:57.473 7f920ff09700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f920810aa60 0x7f920810ae60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:57.491 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:57.473 7f920ff09700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92081b0d40 con 0x7f92080e53a0 2021-02-10T11:39:57.491 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:57.473 7f920ff09700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9208109ac0 con 0x7f920810aa60 2021-02-10T11:39:57.491 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:57.473 7f920ff09700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92080e57a0 con 0x7f920810a040 2021-02-10T11:39:57.492 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:57.474 7f920dd25700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f92080e53a0 0x7f92081b0550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:33050/0 (socket says 172.21.2.102:33050) 2021-02-10T11:39:57.492 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:57.474 7f920dd25700 1 -- 172.21.2.102:0/2350985429 learned_addr learned my addr 172.21.2.102:0/2350985429 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:39:57.492 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:57.474 7f920e526700 1 --2- 172.21.2.102:0/2350985429 >> v2:172.21.2.102:3300/0 conn(0x7f920810aa60 0x7f920810ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:57.492 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:57.474 7f920d524700 1 --2- 172.21.2.102:0/2350985429 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f920810a040 0x7f920810a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:57.492 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:57.474 7f920cd23700 1 -- 172.21.2.102:0/2350985429 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3461395124 0 0) 0x7f91fc000ce0 con 0x7f92080e53a0 2021-02-10T11:39:57.492 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:57.474 7f920cd23700 1 -- 172.21.2.102:0/2350985429 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91f4002fe0 con 0x7f92080e53a0 2021-02-10T11:39:57.492 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:57.474 7f920e526700 1 -- 172.21.2.102:0/2350985429 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f920810a040 msgr2=0x7f920810a440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:57.492 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:57.474 7f920e526700 1 --2- 172.21.2.102:0/2350985429 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f920810a040 0x7f920810a440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:57.492 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:57.474 7f920e526700 1 -- 172.21.2.102:0/2350985429 >> v1:172.21.2.102:6789/0 conn(0x7f92080e53a0 legacy=0x7f92081b0550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:57.492 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:57.474 7f920e526700 1 -- 172.21.2.102:0/2350985429 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f920810b480 con 0x7f920810aa60 2021-02-10T11:39:57.492 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:57.474 7f920e526700 1 --2- 172.21.2.102:0/2350985429 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f920810aa60 0x7f920810ae60 secure :-1 s=READY pgs=3305 cs=0 l=1 rev1=1 rx=0x7f9200006e30 tx=0x7f9200009e60).ready entity=mon.0 client_cookie=b9833c94c7f28d8d server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:57.492 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:57.475 7f920cd23700 1 -- 172.21.2.102:0/2350985429 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9200006a50 con 0x7f920810aa60 2021-02-10T11:39:57.492 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:57.475 7f920cd23700 1 -- 172.21.2.102:0/2350985429 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f920000b300 con 0x7f920810aa60 2021-02-10T11:39:57.493 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:57.475 7f920cd23700 1 -- 172.21.2.102:0/2350985429 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f920000b7e0 con 0x7f920810aa60 2021-02-10T11:39:57.493 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:57.475 7f920ff09700 1 -- 172.21.2.102:0/2350985429 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f920810aa60 msgr2=0x7f920810ae60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:57.493 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:57.475 7f920ff09700 1 --2- 172.21.2.102:0/2350985429 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f920810aa60 0x7f920810ae60 secure :-1 s=READY pgs=3305 cs=0 l=1 rev1=1 rx=0x7f9200006e30 tx=0x7f9200009e60).stop 2021-02-10T11:39:57.493 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:57.475 7f920ff09700 1 -- 172.21.2.102:0/2350985429 shutdown_connections 2021-02-10T11:39:57.493 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:57.475 7f920ff09700 1 --2- 172.21.2.102:0/2350985429 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f920810aa60 0x7f920810ae60 unknown :-1 s=CLOSED pgs=3305 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:57.493 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:57.475 7f920ff09700 1 --2- 172.21.2.102:0/2350985429 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f920810a040 0x7f920810a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:57.493 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:57.475 7f920ff09700 1 -- 172.21.2.102:0/2350985429 >> 172.21.2.102:0/2350985429 conn(0x7f92080ce4b0 msgr2=0x7f92080e45d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:57.493 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:57.475 7f920ff09700 1 -- 172.21.2.102:0/2350985429 shutdown_connections 2021-02-10T11:39:57.493 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:57.475 7f920ff09700 1 -- 172.21.2.102:0/2350985429 wait complete. 2021-02-10T11:39:57.495 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:57.490 7f920ff09700 1 Processor -- start 2021-02-10T11:39:57.495 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:57.490 7f920ff09700 1 -- start start 2021-02-10T11:39:57.495 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:57.490 7f920ff09700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f92080c27d0 0x7f92080a7ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:57.495 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:57.490 7f920ff09700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f920818fce0 0x7f92080b62a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:57.495 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:57.490 7f920ff09700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f92081900e0 0x7f92080b68c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:57.496 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:57.490 7f920ff09700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92080b7280 con 0x7f92080c27d0 2021-02-10T11:39:57.496 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:57.490 7f920ff09700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92080c2250 con 0x7f92081900e0 2021-02-10T11:39:57.496 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:57.490 7f920ff09700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92080a8100 con 0x7f920818fce0 2021-02-10T11:39:57.496 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:57.491 7f920dd25700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f92080c27d0 0x7f92080a7ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:57.496 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:57.491 7f920dd25700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f92080c27d0 0x7f92080a7ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:40834/0 (socket says 172.21.2.102:40834) 2021-02-10T11:39:57.496 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:57.491 7f920dd25700 1 -- 172.21.2.102:0/4189496493 learned_addr learned my addr 172.21.2.102:0/4189496493 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:39:57.496 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:57.491 7f920d524700 1 --2- 172.21.2.102:0/4189496493 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f920818fce0 0x7f92080b62a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:57.496 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:57.491 7f920e526700 1 --2- 172.21.2.102:0/4189496493 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f92081900e0 0x7f92080b68c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:57.496 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:57.491 7f920dd25700 1 -- 172.21.2.102:0/4189496493 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f920818fce0 msgr2=0x7f92080b62a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:57.496 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:57.491 7f920dd25700 1 --2- 172.21.2.102:0/4189496493 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f920818fce0 0x7f92080b62a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:57.496 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:57.491 7f920dd25700 1 -- 172.21.2.102:0/4189496493 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f92081900e0 msgr2=0x7f92080b68c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:57.496 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:57.491 7f920dd25700 1 --2- 172.21.2.102:0/4189496493 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f92081900e0 0x7f92080b68c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:57.497 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:57.491 7f920dd25700 1 -- 172.21.2.102:0/4189496493 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92080b6ee0 con 0x7f92080c27d0 2021-02-10T11:39:57.497 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:57.491 7f920d524700 1 --2- 172.21.2.102:0/4189496493 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f920818fce0 0x7f92080b62a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:39:57.497 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:57.491 7f920dd25700 1 --2- 172.21.2.102:0/4189496493 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f92080c27d0 0x7f92080a7ae0 secure :-1 s=READY pgs=3306 cs=0 l=1 rev1=1 rx=0x7f91fc00b5c0 tx=0x7f91fc00b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:57.497 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:57.491 7f91fa7fc700 1 -- 172.21.2.102:0/4189496493 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f91fc00bd50 con 0x7f92080c27d0 2021-02-10T11:39:57.497 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:57.491 7f91fa7fc700 1 -- 172.21.2.102:0/4189496493 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91fc006280 con 0x7f92080c27d0 2021-02-10T11:39:57.497 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:57.491 7f91fa7fc700 1 -- 172.21.2.102:0/4189496493 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f91fc004620 con 0x7f92080c27d0 2021-02-10T11:39:57.497 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:57.491 7f920ff09700 1 -- 172.21.2.102:0/4189496493 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92080b6ee0 con 0x7f92080c27d0 2021-02-10T11:39:57.498 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:57.492 7f920ff09700 1 -- 172.21.2.102:0/4189496493 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92080b6ee0 con 0x7f92080c27d0 2021-02-10T11:39:57.498 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:57.492 7f91fa7fc700 1 -- 172.21.2.102:0/4189496493 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f91fc0043a0 con 0x7f92080c27d0 2021-02-10T11:39:57.498 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:57.492 7f91fa7fc700 1 --2- 172.21.2.102:0/4189496493 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f91f00186c0 0x7f91f001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:57.498 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:57.492 7f91fa7fc700 1 -- 172.21.2.102:0/4189496493 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(189..189 src has 1..189) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f91fc032540 con 0x7f92080c27d0 2021-02-10T11:39:57.498 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:57.493 7f920d524700 1 --2- 172.21.2.102:0/4189496493 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f91f00186c0 0x7f91f001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:57.498 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:57.493 7f920d524700 1 --2- 172.21.2.102:0/4189496493 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f91f00186c0 0x7f91f001abc0 crc :-1 s=READY pgs=1441 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:57.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:57.523 7f920ff09700 1 -- 172.21.2.102:0/4189496493 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9208103750 con 0x7f92080c27d0 2021-02-10T11:39:57.527 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:57.526 7f91fa7fc700 1 -- 172.21.2.102:0/4189496493 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f91fc006050 con 0x7f92080c27d0 2021-02-10T11:39:57.651 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:57.650 7f920ff09700 1 -- 172.21.2.102:0/4189496493 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"access_level": "r", "prefix": "fs subvolume authorize", "vol_name": "cephfs", "sub_name": "subvolume_0000000000518037", "target": ["mgr", ""], "auth_id": "guest2", "group_name": "group_0000000000599689"}) v1 -- 0x7f92080c2250 con 0x7f91f00186c0 2021-02-10T11:39:57.656 INFO:tasks.ceph.mgr.y.gibba002.stderr:Creating meta for ID guest2 with tenant None 2021-02-10T11:39:57.689 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:57.688 7f91fa7fc700 1 -- 172.21.2.102:0/4189496493 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f92040009b0 con 0x7f91f00186c0 2021-02-10T11:39:57.717 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:57.715 7f920ff09700 1 -- 172.21.2.102:0/4189496493 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f91f00186c0 msgr2=0x7f91f001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:57.717 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:57.715 7f920ff09700 1 --2- 172.21.2.102:0/4189496493 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f91f00186c0 0x7f91f001abc0 crc :-1 s=READY pgs=1441 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:57.717 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:57.715 7f920ff09700 1 -- 172.21.2.102:0/4189496493 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f92080c27d0 msgr2=0x7f92080a7ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:57.717 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:57.715 7f920ff09700 1 --2- 172.21.2.102:0/4189496493 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f92080c27d0 0x7f92080a7ae0 secure :-1 s=READY pgs=3306 cs=0 l=1 rev1=1 rx=0x7f91fc00b5c0 tx=0x7f91fc00b960).stop 2021-02-10T11:39:57.717 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:57.715 7f920ff09700 1 -- 172.21.2.102:0/4189496493 shutdown_connections 2021-02-10T11:39:57.717 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:57.715 7f920ff09700 1 --2- 172.21.2.102:0/4189496493 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f91f00186c0 0x7f91f001abc0 unknown :-1 s=CLOSED pgs=1441 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:57.717 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:57.715 7f920ff09700 1 --2- 172.21.2.102:0/4189496493 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f92081900e0 0x7f92080b68c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:57.717 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:57.715 7f920ff09700 1 --2- 172.21.2.102:0/4189496493 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f920818fce0 0x7f92080b62a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:57.717 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:57.715 7f920ff09700 1 --2- 172.21.2.102:0/4189496493 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f92080c27d0 0x7f92080a7ae0 unknown :-1 s=CLOSED pgs=3306 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:57.717 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:57.715 7f920ff09700 1 -- 172.21.2.102:0/4189496493 >> 172.21.2.102:0/4189496493 conn(0x7f92080a8450 msgr2=0x7f92080979f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:57.718 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:57.715 7f920ff09700 1 -- 172.21.2.102:0/4189496493 shutdown_connections 2021-02-10T11:39:57.718 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:57.716 7f920ff09700 1 -- 172.21.2.102:0/4189496493 wait complete. 2021-02-10T11:39:57.725 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume authorized_list cephfs subvolume_0000000000518037 --group_name group_0000000000599689 2021-02-10T11:39:58.159 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:58.141 7f6c64c2d700 1 Processor -- start 2021-02-10T11:39:58.160 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:58.141 7f6c64c2d700 1 -- start start 2021-02-10T11:39:58.160 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:58.142 7f6c64c2d700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f6c600d1870 0x7f6c601b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:58.160 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:58.142 7f6c64c2d700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6c601b0ae0 0x7f6c60108940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:58.160 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:58.142 7f6c64c2d700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f6c600d1c70 0x7f6c60108f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:58.160 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:58.142 7f6c64c2d700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c600fe1c0 con 0x7f6c600d1870 2021-02-10T11:39:58.160 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:58.142 7f6c64c2d700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c60109a60 con 0x7f6c600d1c70 2021-02-10T11:39:58.160 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:58.142 7f6c64c2d700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c60088070 con 0x7f6c601b0ae0 2021-02-10T11:39:58.160 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:58.142 7f6c5e85e700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f6c600d1870 0x7f6c601b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:58.160 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:58.142 7f6c5e85e700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f6c600d1870 0x7f6c601b04c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:40840/0 (socket says 172.21.2.102:40840) 2021-02-10T11:39:58.160 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:58.142 7f6c5e85e700 1 -- 172.21.2.102:0/1700492852 learned_addr learned my addr 172.21.2.102:0/1700492852 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:39:58.160 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:58.142 7f6c5e85e700 1 -- 172.21.2.102:0/1700492852 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6c601b0ae0 msgr2=0x7f6c60108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:58.160 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:58.142 7f6c5e85e700 1 --2- 172.21.2.102:0/1700492852 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6c601b0ae0 0x7f6c60108940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:58.161 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:58.142 7f6c5e85e700 1 -- 172.21.2.102:0/1700492852 >> v2:172.21.2.113:3300/0 conn(0x7f6c600d1c70 msgr2=0x7f6c60108f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:58.161 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:58.142 7f6c5e85e700 1 --2- 172.21.2.102:0/1700492852 >> v2:172.21.2.113:3300/0 conn(0x7f6c600d1c70 0x7f6c60108f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:58.161 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:58.142 7f6c5e85e700 1 -- 172.21.2.102:0/1700492852 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c60109580 con 0x7f6c600d1870 2021-02-10T11:39:58.161 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:58.143 7f6c5e85e700 1 --2- 172.21.2.102:0/1700492852 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6c600d1870 0x7f6c601b04c0 secure :-1 s=READY pgs=3307 cs=0 l=1 rev1=1 rx=0x7f6c48006700 tx=0x7f6c48009b80).ready entity=mon.0 client_cookie=ec4416b7a3523c07 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:58.161 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:58.143 7f6c5d85c700 1 -- 172.21.2.102:0/1700492852 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6c4800c0c0 con 0x7f6c600d1870 2021-02-10T11:39:58.161 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:58.143 7f6c5d85c700 1 -- 172.21.2.102:0/1700492852 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c4800c440 con 0x7f6c600d1870 2021-02-10T11:39:58.161 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:58.143 7f6c5d85c700 1 -- 172.21.2.102:0/1700492852 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6c4800c8a0 con 0x7f6c600d1870 2021-02-10T11:39:58.161 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:58.143 7f6c64c2d700 1 -- 172.21.2.102:0/1700492852 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6c600d1870 msgr2=0x7f6c601b04c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:58.161 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:58.143 7f6c64c2d700 1 --2- 172.21.2.102:0/1700492852 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6c600d1870 0x7f6c601b04c0 secure :-1 s=READY pgs=3307 cs=0 l=1 rev1=1 rx=0x7f6c48006700 tx=0x7f6c48009b80).stop 2021-02-10T11:39:58.161 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:58.144 7f6c64c2d700 1 -- 172.21.2.102:0/1700492852 shutdown_connections 2021-02-10T11:39:58.161 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:58.144 7f6c64c2d700 1 --2- 172.21.2.102:0/1700492852 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6c601b0ae0 0x7f6c60108940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:58.162 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:58.144 7f6c64c2d700 1 --2- 172.21.2.102:0/1700492852 >> v2:172.21.2.113:3300/0 conn(0x7f6c600d1c70 0x7f6c60108f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:58.162 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:58.144 7f6c64c2d700 1 --2- 172.21.2.102:0/1700492852 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6c600d1870 0x7f6c601b04c0 unknown :-1 s=CLOSED pgs=3307 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:58.162 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:58.144 7f6c64c2d700 1 -- 172.21.2.102:0/1700492852 >> 172.21.2.102:0/1700492852 conn(0x7f6c600ce420 msgr2=0x7f6c600d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:58.162 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:58.144 7f6c64c2d700 1 -- 172.21.2.102:0/1700492852 shutdown_connections 2021-02-10T11:39:58.162 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:58.144 7f6c64c2d700 1 -- 172.21.2.102:0/1700492852 wait complete. 2021-02-10T11:39:58.163 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:58.159 7f6c64c2d700 1 Processor -- start 2021-02-10T11:39:58.163 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:58.159 7f6c64c2d700 1 -- start start 2021-02-10T11:39:58.164 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:58.159 7f6c64c2d700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6c6019fb30 0x7f6c6019ff30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:58.164 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:58.159 7f6c64c2d700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6c6009a0d0 0x7f6c6009a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:58.164 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:58.159 7f6c64c2d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6c6009aaf0 0x7f6c601a0ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:58.164 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:58.159 7f6c64c2d700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c601a0920 con 0x7f6c6009aaf0 2021-02-10T11:39:58.164 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:58.159 7f6c64c2d700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c6009aef0 con 0x7f6c6019fb30 2021-02-10T11:39:58.164 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:58.159 7f6c64c2d700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c601a0550 con 0x7f6c6009a0d0 2021-02-10T11:39:58.164 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:58.159 7f6c5f05f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6c6009aaf0 0x7f6c601a0ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:58.164 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:58.159 7f6c5f05f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6c6009aaf0 0x7f6c601a0ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:40848/0 (socket says 172.21.2.102:40848) 2021-02-10T11:39:58.164 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:58.159 7f6c5f05f700 1 -- 172.21.2.102:0/1653042153 learned_addr learned my addr 172.21.2.102:0/1653042153 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:39:58.164 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:58.160 7f6c5f05f700 1 -- 172.21.2.102:0/1653042153 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6c6009a0d0 msgr2=0x7f6c6009a4d0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:39:58.164 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:58.160 7f6c5f05f700 1 --2- 172.21.2.102:0/1653042153 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6c6009a0d0 0x7f6c6009a4d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:58.164 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:58.160 7f6c5f05f700 1 -- 172.21.2.102:0/1653042153 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6c6019fb30 msgr2=0x7f6c6019ff30 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:39:58.164 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:58.160 7f6c5f05f700 1 --2- 172.21.2.102:0/1653042153 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6c6019fb30 0x7f6c6019ff30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:58.165 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:58.160 7f6c5f05f700 1 -- 172.21.2.102:0/1653042153 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c601a14c0 con 0x7f6c6009aaf0 2021-02-10T11:39:58.165 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:58.160 7f6c5f05f700 1 --2- 172.21.2.102:0/1653042153 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6c6009aaf0 0x7f6c601a0ea0 secure :-1 s=READY pgs=3308 cs=0 l=1 rev1=1 rx=0x7f6c58009980 tx=0x7f6c5800bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:58.165 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:58.160 7f6c577fe700 1 -- 172.21.2.102:0/1653042153 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6c58005870 con 0x7f6c6009aaf0 2021-02-10T11:39:58.165 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:58.160 7f6c577fe700 1 -- 172.21.2.102:0/1653042153 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c58005b30 con 0x7f6c6009aaf0 2021-02-10T11:39:58.165 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:58.160 7f6c577fe700 1 -- 172.21.2.102:0/1653042153 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6c58006030 con 0x7f6c6009aaf0 2021-02-10T11:39:58.165 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:58.160 7f6c64c2d700 1 -- 172.21.2.102:0/1653042153 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c601a14c0 con 0x7f6c6009aaf0 2021-02-10T11:39:58.165 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:58.160 7f6c64c2d700 1 -- 172.21.2.102:0/1653042153 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c601a14c0 con 0x7f6c6009aaf0 2021-02-10T11:39:58.165 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:58.161 7f6c577fe700 1 -- 172.21.2.102:0/1653042153 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f6c58006030 con 0x7f6c6009aaf0 2021-02-10T11:39:58.165 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:58.161 7f6c577fe700 1 --2- 172.21.2.102:0/1653042153 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6c440186c0 0x7f6c4401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:58.165 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:58.161 7f6c577fe700 1 -- 172.21.2.102:0/1653042153 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(189..189 src has 1..189) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f6c58032b40 con 0x7f6c6009aaf0 2021-02-10T11:39:58.165 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:58.161 7f6c5e85e700 1 --2- 172.21.2.102:0/1653042153 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6c440186c0 0x7f6c4401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:58.165 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:58.161 7f6c5e85e700 1 --2- 172.21.2.102:0/1653042153 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6c440186c0 0x7f6c4401abc0 crc :-1 s=READY pgs=1442 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:58.193 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:58.192 7f6c64c2d700 1 -- 172.21.2.102:0/1653042153 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c600825d0 con 0x7f6c6009aaf0 2021-02-10T11:39:58.195 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:58.194 7f6c577fe700 1 -- 172.21.2.102:0/1653042153 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f6c5800be10 con 0x7f6c6009aaf0 2021-02-10T11:39:58.321 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:58.320 7f6c64c2d700 1 -- 172.21.2.102:0/1653042153 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume authorized_list", "vol_name": "cephfs", "sub_name": "subvolume_0000000000518037", "target": ["mgr", ""], "group_name": "group_0000000000599689"}) v1 -- 0x7f6c601a0920 con 0x7f6c440186c0 2021-02-10T11:39:58.325 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:58.324 7f6c577fe700 1 -- 172.21.2.102:0/1653042153 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+111 (crc 0 0 0) 0x7f6c48009100 con 0x7f6c440186c0 2021-02-10T11:39:58.328 INFO:teuthology.orchestra.run.gibba002.stdout:[ 2021-02-10T11:39:58.328 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:39:58.328 INFO:teuthology.orchestra.run.gibba002.stdout: "guest2": "r" 2021-02-10T11:39:58.328 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:39:58.328 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:39:58.329 INFO:teuthology.orchestra.run.gibba002.stdout: "alice": "rw" 2021-02-10T11:39:58.329 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:39:58.329 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:39:58.329 INFO:teuthology.orchestra.run.gibba002.stdout: "guest1": "rw" 2021-02-10T11:39:58.329 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:39:58.329 INFO:teuthology.orchestra.run.gibba002.stdout:] 2021-02-10T11:39:58.330 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:58.328 7f6c64c2d700 1 -- 172.21.2.102:0/1653042153 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6c440186c0 msgr2=0x7f6c4401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:58.330 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:58.328 7f6c64c2d700 1 --2- 172.21.2.102:0/1653042153 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6c440186c0 0x7f6c4401abc0 crc :-1 s=READY pgs=1442 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:58.330 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:58.328 7f6c64c2d700 1 -- 172.21.2.102:0/1653042153 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6c6009aaf0 msgr2=0x7f6c601a0ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:58.330 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:58.328 7f6c64c2d700 1 --2- 172.21.2.102:0/1653042153 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6c6009aaf0 0x7f6c601a0ea0 secure :-1 s=READY pgs=3308 cs=0 l=1 rev1=1 rx=0x7f6c58009980 tx=0x7f6c5800bcd0).stop 2021-02-10T11:39:58.330 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:58.329 7f6c64c2d700 1 -- 172.21.2.102:0/1653042153 shutdown_connections 2021-02-10T11:39:58.330 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:58.329 7f6c64c2d700 1 --2- 172.21.2.102:0/1653042153 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6c6009aaf0 0x7f6c601a0ea0 unknown :-1 s=CLOSED pgs=3308 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:58.330 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:58.329 7f6c64c2d700 1 --2- 172.21.2.102:0/1653042153 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6c6009a0d0 0x7f6c6009a4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:58.330 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:58.329 7f6c64c2d700 1 --2- 172.21.2.102:0/1653042153 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6c440186c0 0x7f6c4401abc0 unknown :-1 s=CLOSED pgs=1442 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:58.330 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:58.329 7f6c64c2d700 1 --2- 172.21.2.102:0/1653042153 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6c6019fb30 0x7f6c6019ff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:58.330 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:58.329 7f6c64c2d700 1 -- 172.21.2.102:0/1653042153 >> 172.21.2.102:0/1653042153 conn(0x7f6c601b3560 msgr2=0x7f6c601b90e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:58.331 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:58.329 7f6c64c2d700 1 -- 172.21.2.102:0/1653042153 shutdown_connections 2021-02-10T11:39:58.331 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:58.329 7f6c64c2d700 1 -- 172.21.2.102:0/1653042153 wait complete. 2021-02-10T11:39:58.338 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume deauthorize cephfs subvolume_0000000000518037 alice --group_name group_0000000000599689 2021-02-10T11:39:58.774 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:58.757 7fdf8710e700 1 Processor -- start 2021-02-10T11:39:58.775 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:58.757 7fdf8710e700 1 -- start start 2021-02-10T11:39:58.775 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:58.757 7fdf8710e700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fdf800dce80 0x7fdf800d14f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:58.775 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:58.757 7fdf8710e700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdf800eb220 0x7fdf800eb620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:58.775 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:58.757 7fdf8710e700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf8010a550 con 0x7fdf8010aad0 2021-02-10T11:39:58.775 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:58.757 7fdf8710e700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf80088240 con 0x7fdf800dce80 2021-02-10T11:39:58.775 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:58.757 7fdf8710e700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf800d1f30 con 0x7fdf800eb220 2021-02-10T11:39:58.775 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:58.757 7fdf84f2a700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fdf800dce80 0x7fdf800d14f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:58.775 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:58.758 7fdf7ffff700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdf800eb220 0x7fdf800eb620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:58.775 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:58.758 7fdf84f2a700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fdf800dce80 0x7fdf800d14f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.113:3300/0 says I am v2:172.21.2.102:44284/0 (socket says 172.21.2.102:44284) 2021-02-10T11:39:58.775 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:58.758 7fdf84f2a700 1 -- 172.21.2.102:0/3902409445 learned_addr learned my addr 172.21.2.102:0/3902409445 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:39:58.775 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:58.758 7fdf7ffff700 1 -- 172.21.2.102:0/3902409445 >> v2:172.21.2.113:3300/0 conn(0x7fdf800dce80 msgr2=0x7fdf800d14f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:58.775 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:58.758 7fdf7ffff700 1 --2- 172.21.2.102:0/3902409445 >> v2:172.21.2.113:3300/0 conn(0x7fdf800dce80 0x7fdf800d14f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:58.775 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:58.758 7fdf7ffff700 1 -- 172.21.2.102:0/3902409445 >> v1:172.21.2.102:6789/0 conn(0x7fdf8010aad0 legacy=0x7fdf8010aed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:58.776 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:58.758 7fdf7ffff700 1 -- 172.21.2.102:0/3902409445 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf800dd280 con 0x7fdf800eb220 2021-02-10T11:39:58.776 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:58.758 7fdf84f2a700 1 --2- 172.21.2.102:0/3902409445 >> v2:172.21.2.113:3300/0 conn(0x7fdf800dce80 0x7fdf800d14f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:39:58.776 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:58.758 7fdf7ffff700 1 --2- 172.21.2.102:0/3902409445 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdf800eb220 0x7fdf800eb620 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 rx=0x7fdf6c006650 tx=0x7fdf6c009a80).ready entity=mon.2 client_cookie=956db7e1d6d23226 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:58.776 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:58.758 7fdf7f7fe700 1 -- 172.21.2.102:0/3902409445 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdf6c006390 con 0x7fdf800eb220 2021-02-10T11:39:58.776 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:58.758 7fdf7f7fe700 1 -- 172.21.2.102:0/3902409445 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf6c0009b0 con 0x7fdf800eb220 2021-02-10T11:39:58.776 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:58.758 7fdf7f7fe700 1 -- 172.21.2.102:0/3902409445 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdf6c00c2b0 con 0x7fdf800eb220 2021-02-10T11:39:58.776 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:58.758 7fdf8710e700 1 -- 172.21.2.102:0/3902409445 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdf800eb220 msgr2=0x7fdf800eb620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:58.776 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:58.758 7fdf8710e700 1 --2- 172.21.2.102:0/3902409445 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdf800eb220 0x7fdf800eb620 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 rx=0x7fdf6c006650 tx=0x7fdf6c009a80).stop 2021-02-10T11:39:58.776 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:58.759 7fdf8710e700 1 -- 172.21.2.102:0/3902409445 shutdown_connections 2021-02-10T11:39:58.776 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:58.759 7fdf8710e700 1 --2- 172.21.2.102:0/3902409445 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdf800eb220 0x7fdf800eb620 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:58.776 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:58.759 7fdf8710e700 1 --2- 172.21.2.102:0/3902409445 >> v2:172.21.2.113:3300/0 conn(0x7fdf800dce80 0x7fdf800d14f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:58.776 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:58.759 7fdf8710e700 1 -- 172.21.2.102:0/3902409445 >> 172.21.2.102:0/3902409445 conn(0x7fdf800ea630 msgr2=0x7fdf800d0800 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:58.776 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:58.759 7fdf8710e700 1 -- 172.21.2.102:0/3902409445 shutdown_connections 2021-02-10T11:39:58.777 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:58.759 7fdf8710e700 1 -- 172.21.2.102:0/3902409445 wait complete. 2021-02-10T11:39:58.778 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:58.773 7fdf8710e700 1 Processor -- start 2021-02-10T11:39:58.779 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:58.774 7fdf8710e700 1 -- start start 2021-02-10T11:39:58.779 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:58.774 7fdf8710e700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdf801a3580 0x7fdf801a3980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:58.779 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:58.774 7fdf8710e700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fdf80099280 0x7fdf800924d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:58.779 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:58.774 7fdf8710e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdf80099680 0x7fdf80092af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:58.779 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:58.774 7fdf8710e700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf80109b70 con 0x7fdf80099680 2021-02-10T11:39:58.779 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:58.774 7fdf8710e700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf80096b80 con 0x7fdf80099280 2021-02-10T11:39:58.779 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:58.774 7fdf8710e700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf80093750 con 0x7fdf801a3580 2021-02-10T11:39:58.779 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:58.774 7fdf8572b700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdf80099680 0x7fdf80092af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:58.779 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:58.774 7fdf8572b700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdf80099680 0x7fdf80092af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:40864/0 (socket says 172.21.2.102:40864) 2021-02-10T11:39:58.779 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:58.774 7fdf8572b700 1 -- 172.21.2.102:0/3730025656 learned_addr learned my addr 172.21.2.102:0/3730025656 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:39:58.779 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:58.774 7fdf7ffff700 1 --2- 172.21.2.102:0/3730025656 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fdf80099280 0x7fdf800924d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:58.779 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:58.774 7fdf84f2a700 1 --2- 172.21.2.102:0/3730025656 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdf801a3580 0x7fdf801a3980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:58.779 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:58.775 7fdf8572b700 1 -- 172.21.2.102:0/3730025656 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdf801a3580 msgr2=0x7fdf801a3980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:58.780 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:58.775 7fdf8572b700 1 --2- 172.21.2.102:0/3730025656 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdf801a3580 0x7fdf801a3980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:58.780 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:58.775 7fdf8572b700 1 -- 172.21.2.102:0/3730025656 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fdf80099280 msgr2=0x7fdf800924d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:58.780 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:58.775 7fdf8572b700 1 --2- 172.21.2.102:0/3730025656 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fdf80099280 0x7fdf800924d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:58.780 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:58.775 7fdf8572b700 1 -- 172.21.2.102:0/3730025656 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf80093970 con 0x7fdf80099680 2021-02-10T11:39:58.780 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:58.775 7fdf8572b700 1 --2- 172.21.2.102:0/3730025656 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdf80099680 0x7fdf80092af0 secure :-1 s=READY pgs=3310 cs=0 l=1 rev1=1 rx=0x7fdf78009980 tx=0x7fdf7800bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:58.780 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:58.775 7fdf7d7fa700 1 -- 172.21.2.102:0/3730025656 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdf78005870 con 0x7fdf80099680 2021-02-10T11:39:58.780 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:58.775 7fdf7d7fa700 1 -- 172.21.2.102:0/3730025656 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf78005a70 con 0x7fdf80099680 2021-02-10T11:39:58.780 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:58.775 7fdf7d7fa700 1 -- 172.21.2.102:0/3730025656 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdf78006030 con 0x7fdf80099680 2021-02-10T11:39:58.780 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:58.775 7fdf8710e700 1 -- 172.21.2.102:0/3730025656 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf80093970 con 0x7fdf80099680 2021-02-10T11:39:58.780 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:58.775 7fdf8710e700 1 -- 172.21.2.102:0/3730025656 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf80093970 con 0x7fdf80099680 2021-02-10T11:39:58.780 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:58.776 7fdf7d7fa700 1 -- 172.21.2.102:0/3730025656 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fdf78005c50 con 0x7fdf80099680 2021-02-10T11:39:58.780 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:58.776 7fdf7d7fa700 1 --2- 172.21.2.102:0/3730025656 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fdf5c0186c0 0x7fdf5c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:58.780 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:58.776 7fdf7d7fa700 1 -- 172.21.2.102:0/3730025656 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(189..189 src has 1..189) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fdf78032920 con 0x7fdf80099680 2021-02-10T11:39:58.781 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:58.776 7fdf84f2a700 1 --2- 172.21.2.102:0/3730025656 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fdf5c0186c0 0x7fdf5c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:58.781 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:58.776 7fdf84f2a700 1 --2- 172.21.2.102:0/3730025656 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fdf5c0186c0 0x7fdf5c01abc0 crc :-1 s=READY pgs=1443 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:58.808 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:58.807 7fdf8710e700 1 -- 172.21.2.102:0/3730025656 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf80109b70 con 0x7fdf80099680 2021-02-10T11:39:58.810 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:58.809 7fdf7d7fa700 1 -- 172.21.2.102:0/3730025656 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fdf7800be10 con 0x7fdf80099680 2021-02-10T11:39:58.937 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:58.936 7fdf8710e700 1 -- 172.21.2.102:0/3730025656 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume deauthorize", "vol_name": "cephfs", "sub_name": "subvolume_0000000000518037", "target": ["mgr", ""], "auth_id": "alice", "group_name": "group_0000000000599689"}) v1 -- 0x7fdf80096b80 con 0x7fdf5c0186c0 2021-02-10T11:39:58.968 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:58.967 7fdf7d7fa700 1 -- 172.21.2.102:0/3730025656 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fdf700009b0 con 0x7fdf5c0186c0 2021-02-10T11:39:58.970 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:58.969 7fdf8710e700 1 -- 172.21.2.102:0/3730025656 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fdf5c0186c0 msgr2=0x7fdf5c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:58.970 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:58.969 7fdf8710e700 1 --2- 172.21.2.102:0/3730025656 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fdf5c0186c0 0x7fdf5c01abc0 crc :-1 s=READY pgs=1443 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:58.970 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:58.969 7fdf8710e700 1 -- 172.21.2.102:0/3730025656 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdf80099680 msgr2=0x7fdf80092af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:58.970 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:58.969 7fdf8710e700 1 --2- 172.21.2.102:0/3730025656 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdf80099680 0x7fdf80092af0 secure :-1 s=READY pgs=3310 cs=0 l=1 rev1=1 rx=0x7fdf78009980 tx=0x7fdf7800bcd0).stop 2021-02-10T11:39:58.971 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:58.969 7fdf8710e700 1 -- 172.21.2.102:0/3730025656 shutdown_connections 2021-02-10T11:39:58.971 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:58.969 7fdf8710e700 1 --2- 172.21.2.102:0/3730025656 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fdf80099680 0x7fdf80092af0 unknown :-1 s=CLOSED pgs=3310 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:58.971 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:58.969 7fdf8710e700 1 --2- 172.21.2.102:0/3730025656 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fdf5c0186c0 0x7fdf5c01abc0 unknown :-1 s=CLOSED pgs=1443 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:58.971 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:58.969 7fdf8710e700 1 --2- 172.21.2.102:0/3730025656 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fdf80099280 0x7fdf800924d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:58.971 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:58.969 7fdf8710e700 1 --2- 172.21.2.102:0/3730025656 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fdf801a3580 0x7fdf801a3980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:58.971 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:58.969 7fdf8710e700 1 -- 172.21.2.102:0/3730025656 >> 172.21.2.102:0/3730025656 conn(0x7fdf80109df0 msgr2=0x7fdf800fc8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:58.971 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:58.969 7fdf8710e700 1 -- 172.21.2.102:0/3730025656 shutdown_connections 2021-02-10T11:39:58.971 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:58.969 7fdf8710e700 1 -- 172.21.2.102:0/3730025656 wait complete. 2021-02-10T11:39:58.978 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume deauthorize cephfs subvolume_0000000000518037 guest1 --group_name group_0000000000599689 2021-02-10T11:39:59.411 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:59.394 7f66c634c700 1 Processor -- start 2021-02-10T11:39:59.411 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:59.394 7f66c634c700 1 -- start start 2021-02-10T11:39:59.411 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:59.394 7f66c634c700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f66c00d1890 0x7f66c01b03d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:59.411 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:59.394 7f66c634c700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f66c01b09f0 0x7f66c01089a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:59.411 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:59.394 7f66c634c700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66c0109a60 con 0x7f66c00d1c90 2021-02-10T11:39:59.411 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:59.394 7f66c634c700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66c00e5f50 con 0x7f66c00d1890 2021-02-10T11:39:59.411 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:59.394 7f66c634c700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66c01097b0 con 0x7f66c01b09f0 2021-02-10T11:39:59.411 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:59.394 7f66bffff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f66c00d1890 0x7f66c01b03d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:59.412 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:59.394 7f66bffff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f66c00d1890 0x7f66c01b03d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:40868/0 (socket says 172.21.2.102:40868) 2021-02-10T11:39:59.412 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:59.394 7f66bffff700 1 -- 172.21.2.102:0/1179508224 learned_addr learned my addr 172.21.2.102:0/1179508224 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:39:59.412 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:59.395 7f66bffff700 1 -- 172.21.2.102:0/1179508224 >> v2:172.21.2.113:3300/0 conn(0x7f66c01b09f0 msgr2=0x7f66c01089a0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:39:59.412 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:59.395 7f66bffff700 1 --2- 172.21.2.102:0/1179508224 >> v2:172.21.2.113:3300/0 conn(0x7f66c01b09f0 0x7f66c01089a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:59.412 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:59.395 7f66bffff700 1 -- 172.21.2.102:0/1179508224 >> v1:172.21.2.113:6789/0 conn(0x7f66c00d1c90 legacy=0x7f66c0108fc0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:39:59.412 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:59.395 7f66bffff700 1 -- 172.21.2.102:0/1179508224 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66c0088070 con 0x7f66c00d1890 2021-02-10T11:39:59.412 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:59.395 7f66bffff700 1 --2- 172.21.2.102:0/1179508224 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f66c00d1890 0x7f66c01b03d0 secure :-1 s=READY pgs=3311 cs=0 l=1 rev1=1 rx=0x7f66b0006a80 tx=0x7f66b0009f00).ready entity=mon.0 client_cookie=8b220e7ea5454653 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:59.412 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:59.395 7f66beffd700 1 -- 172.21.2.102:0/1179508224 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f66b0000d30 con 0x7f66c00d1890 2021-02-10T11:39:59.412 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:59.395 7f66beffd700 1 -- 172.21.2.102:0/1179508224 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66b000b5c0 con 0x7f66c00d1890 2021-02-10T11:39:59.412 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:59.395 7f66beffd700 1 -- 172.21.2.102:0/1179508224 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f66b000bae0 con 0x7f66c00d1890 2021-02-10T11:39:59.412 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:59.395 7f66c634c700 1 -- 172.21.2.102:0/1179508224 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f66c00d1890 msgr2=0x7f66c01b03d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:59.412 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:59.395 7f66c634c700 1 --2- 172.21.2.102:0/1179508224 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f66c00d1890 0x7f66c01b03d0 secure :-1 s=READY pgs=3311 cs=0 l=1 rev1=1 rx=0x7f66b0006a80 tx=0x7f66b0009f00).stop 2021-02-10T11:39:59.412 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:59.395 7f66c634c700 1 -- 172.21.2.102:0/1179508224 shutdown_connections 2021-02-10T11:39:59.413 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:59.395 7f66c634c700 1 --2- 172.21.2.102:0/1179508224 >> v2:172.21.2.113:3300/0 conn(0x7f66c01b09f0 0x7f66c01089a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:59.413 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:59.395 7f66c634c700 1 --2- 172.21.2.102:0/1179508224 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f66c00d1890 0x7f66c01b03d0 unknown :-1 s=CLOSED pgs=3311 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:59.413 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:59.395 7f66c634c700 1 -- 172.21.2.102:0/1179508224 >> 172.21.2.102:0/1179508224 conn(0x7f66c00fdee0 msgr2=0x7f66c00ce450 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:59.413 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:59.396 7f66c634c700 1 -- 172.21.2.102:0/1179508224 shutdown_connections 2021-02-10T11:39:59.413 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:59.396 7f66c634c700 1 -- 172.21.2.102:0/1179508224 wait complete. 2021-02-10T11:39:59.415 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:59.410 7f66c634c700 1 Processor -- start 2021-02-10T11:39:59.415 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:59.410 7f66c634c700 1 -- start start 2021-02-10T11:39:59.415 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:59.411 7f66c634c700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f66c008f080 0x7f66c0190af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:59.415 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:59.411 7f66c634c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f66c00b3910 0x7f66c00b3d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:59.415 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:59.411 7f66c634c700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f66c0187ed0 0x7f66c01882d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:59.415 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:59.411 7f66c634c700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66c0191110 con 0x7f66c00b3910 2021-02-10T11:39:59.415 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:59.411 7f66c634c700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66c0190570 con 0x7f66c0187ed0 2021-02-10T11:39:59.415 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:59.411 7f66c634c700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66c01888f0 con 0x7f66c008f080 2021-02-10T11:39:59.415 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:59.411 7f66bf7fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f66c00b3910 0x7f66c00b3d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:59.415 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:59.411 7f66bf7fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f66c00b3910 0x7f66c00b3d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:40878/0 (socket says 172.21.2.102:40878) 2021-02-10T11:39:59.416 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:59.411 7f66bf7fe700 1 -- 172.21.2.102:0/2825128306 learned_addr learned my addr 172.21.2.102:0/2825128306 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:39:59.416 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:59.411 7f66c4969700 1 --2- 172.21.2.102:0/2825128306 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f66c0187ed0 0x7f66c01882d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:59.416 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:59.411 7f66bf7fe700 1 -- 172.21.2.102:0/2825128306 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f66c008f080 msgr2=0x7f66c0190af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:59.416 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:59.411 7f66bf7fe700 1 --2- 172.21.2.102:0/2825128306 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f66c008f080 0x7f66c0190af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:59.416 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:59.411 7f66bf7fe700 1 -- 172.21.2.102:0/2825128306 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f66c0187ed0 msgr2=0x7f66c01882d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:59.416 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:59.411 7f66bf7fe700 1 --2- 172.21.2.102:0/2825128306 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f66c0187ed0 0x7f66c01882d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:59.416 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:59.411 7f66bf7fe700 1 -- 172.21.2.102:0/2825128306 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66c018ee20 con 0x7f66c00b3910 2021-02-10T11:39:59.416 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:59.411 7f66c4969700 1 --2- 172.21.2.102:0/2825128306 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f66c0187ed0 0x7f66c01882d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:39:59.416 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:59.411 7f66bf7fe700 1 --2- 172.21.2.102:0/2825128306 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f66c00b3910 0x7f66c00b3d10 secure :-1 s=READY pgs=3312 cs=0 l=1 rev1=1 rx=0x7f66a800b5c0 tx=0x7f66a800b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:59.416 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:59.411 7f66bcff9700 1 -- 172.21.2.102:0/2825128306 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f66a800bd50 con 0x7f66c00b3910 2021-02-10T11:39:59.416 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:59.412 7f66bcff9700 1 -- 172.21.2.102:0/2825128306 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66a8006030 con 0x7f66c00b3910 2021-02-10T11:39:59.416 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:59.412 7f66c634c700 1 -- 172.21.2.102:0/2825128306 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66c018ee20 con 0x7f66c00b3910 2021-02-10T11:39:59.416 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:59.412 7f66bcff9700 1 -- 172.21.2.102:0/2825128306 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f66a80054d0 con 0x7f66c00b3910 2021-02-10T11:39:59.416 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:59.412 7f66c634c700 1 -- 172.21.2.102:0/2825128306 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66c018ee20 con 0x7f66c00b3910 2021-02-10T11:39:59.417 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:59.412 7f66bcff9700 1 -- 172.21.2.102:0/2825128306 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f66a8006210 con 0x7f66c00b3910 2021-02-10T11:39:59.417 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:59.412 7f66bcff9700 1 --2- 172.21.2.102:0/2825128306 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f66a4018730 0x7f66a401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:59.417 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:59.412 7f66bcff9700 1 -- 172.21.2.102:0/2825128306 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(189..189 src has 1..189) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f66a8032610 con 0x7f66c00b3910 2021-02-10T11:39:59.417 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:59.413 7f66bffff700 1 --2- 172.21.2.102:0/2825128306 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f66a4018730 0x7f66a401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:59.417 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:59.413 7f66bffff700 1 --2- 172.21.2.102:0/2825128306 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f66a4018730 0x7f66a401ac10 crc :-1 s=READY pgs=1444 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:59.445 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:59.444 7f66c634c700 1 -- 172.21.2.102:0/2825128306 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66c00ff360 con 0x7f66c00b3910 2021-02-10T11:39:59.447 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:59.446 7f66bcff9700 1 -- 172.21.2.102:0/2825128306 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f66a8006b80 con 0x7f66c00b3910 2021-02-10T11:39:59.575 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:59.573 7f66c634c700 1 -- 172.21.2.102:0/2825128306 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume deauthorize", "vol_name": "cephfs", "sub_name": "subvolume_0000000000518037", "target": ["mgr", ""], "auth_id": "guest1", "group_name": "group_0000000000599689"}) v1 -- 0x7f66c01888f0 con 0x7f66a4018730 2021-02-10T11:39:59.605 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:59.604 7f66bcff9700 1 -- 172.21.2.102:0/2825128306 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f66b0006710 con 0x7f66a4018730 2021-02-10T11:39:59.608 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:59.606 7f66c634c700 1 -- 172.21.2.102:0/2825128306 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f66a4018730 msgr2=0x7f66a401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:59.608 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:59.606 7f66c634c700 1 --2- 172.21.2.102:0/2825128306 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f66a4018730 0x7f66a401ac10 crc :-1 s=READY pgs=1444 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:59.608 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:59.606 7f66c634c700 1 -- 172.21.2.102:0/2825128306 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f66c00b3910 msgr2=0x7f66c00b3d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:59.608 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:59.606 7f66c634c700 1 --2- 172.21.2.102:0/2825128306 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f66c00b3910 0x7f66c00b3d10 secure :-1 s=READY pgs=3312 cs=0 l=1 rev1=1 rx=0x7f66a800b5c0 tx=0x7f66a800b960).stop 2021-02-10T11:39:59.608 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:59.607 7f66c634c700 1 -- 172.21.2.102:0/2825128306 shutdown_connections 2021-02-10T11:39:59.608 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:59.607 7f66c634c700 1 --2- 172.21.2.102:0/2825128306 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f66a4018730 0x7f66a401ac10 unknown :-1 s=CLOSED pgs=1444 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:59.608 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:59.607 7f66c634c700 1 --2- 172.21.2.102:0/2825128306 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f66c0187ed0 0x7f66c01882d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:59.608 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:59.607 7f66c634c700 1 --2- 172.21.2.102:0/2825128306 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f66c00b3910 0x7f66c00b3d10 unknown :-1 s=CLOSED pgs=3312 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:59.609 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:59.607 7f66c634c700 1 --2- 172.21.2.102:0/2825128306 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f66c008f080 0x7f66c0190af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:59.609 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:59.607 7f66c634c700 1 -- 172.21.2.102:0/2825128306 >> 172.21.2.102:0/2825128306 conn(0x7f66c01aee50 msgr2=0x7f66c019dba0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:59.609 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:59.607 7f66c634c700 1 -- 172.21.2.102:0/2825128306 shutdown_connections 2021-02-10T11:39:59.609 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:39:59.607 7f66c634c700 1 -- 172.21.2.102:0/2825128306 wait complete. 2021-02-10T11:39:59.616 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume deauthorize cephfs subvolume_0000000000518037 guest2 --group_name group_0000000000599689 2021-02-10T11:40:00.049 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:00.030 7f304d5f8700 1 Processor -- start 2021-02-10T11:40:00.049 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:00.030 7f304d5f8700 1 -- start start 2021-02-10T11:40:00.049 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:00.031 7f304d5f8700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f304810aff0 0x7f30480cf780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:00.049 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:00.031 7f304d5f8700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3048109a60 con 0x7f30480d1c90 2021-02-10T11:40:00.049 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:00.031 7f304d5f8700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f304810b3f0 con 0x7f30480d1890 2021-02-10T11:40:00.049 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:00.031 7f304d5f8700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30480e5f50 con 0x7f304810aff0 2021-02-10T11:40:00.049 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:00.031 7f3046abe700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f30480d1c90 0x7f30480cef90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:33106/0 (socket says 172.21.2.102:33106) 2021-02-10T11:40:00.049 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:00.031 7f3046abe700 1 -- 172.21.2.102:0/4031686476 learned_addr learned my addr 172.21.2.102:0/4031686476 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:40:00.049 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:00.031 7f30462bd700 1 -- 172.21.2.102:0/4031686476 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2095344271 0 0) 0x7f30380010b0 con 0x7f30480d1c90 2021-02-10T11:40:00.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:00.031 7f30462bd700 1 -- 172.21.2.102:0/4031686476 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3034002fe0 con 0x7f30480d1c90 2021-02-10T11:40:00.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:00.032 7f3047ac0700 1 --2- 172.21.2.102:0/4031686476 >> v2:172.21.2.113:3300/0 conn(0x7f304810aff0 0x7f30480cf780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:00.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:00.032 7f30462bd700 1 -- 172.21.2.102:0/4031686476 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1711899203 0 0) 0x7f30380010b0 con 0x7f30480d1c90 2021-02-10T11:40:00.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:00.032 7f30462bd700 1 -- 172.21.2.102:0/4031686476 >> v2:172.21.2.113:3300/0 conn(0x7f304810aff0 msgr2=0x7f30480cf780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:00.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:00.032 7f30462bd700 1 --2- 172.21.2.102:0/4031686476 >> v2:172.21.2.113:3300/0 conn(0x7f304810aff0 0x7f30480cf780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:00.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:00.032 7f30462bd700 1 -- 172.21.2.102:0/4031686476 >> v1:172.21.2.113:6789/0 conn(0x7f30480d1890 legacy=0x7f30481b03d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:00.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:00.032 7f30462bd700 1 -- 172.21.2.102:0/4031686476 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30481b0bc0 con 0x7f30480d1c90 2021-02-10T11:40:00.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:00.032 7f30462bd700 1 -- 172.21.2.102:0/4031686476 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7f30380015b0 con 0x7f30480d1c90 2021-02-10T11:40:00.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:00.032 7f30462bd700 1 -- 172.21.2.102:0/4031686476 >> v1:172.21.2.102:6789/0 conn(0x7f30480d1c90 legacy=0x7f30480cef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:00.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:00.032 7f30462bd700 1 --2- 172.21.2.102:0/4031686476 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3034003aa0 0x7f3034005ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:00.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:00.032 7f30462bd700 1 --2- 172.21.2.102:0/4031686476 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3034006790 0x7f3034008c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:00.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:00.032 7f30462bd700 1 --2- 172.21.2.102:0/4031686476 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3034009590 0x7f303400bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:00.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:00.032 7f30462bd700 1 -- 172.21.2.102:0/4031686476 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f303400c300 con 0x7f3034009590 2021-02-10T11:40:00.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:00.032 7f30462bd700 1 -- 172.21.2.102:0/4031686476 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f303400c520 con 0x7f3034003aa0 2021-02-10T11:40:00.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:00.032 7f30462bd700 1 -- 172.21.2.102:0/4031686476 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f303400c740 con 0x7f3034006790 2021-02-10T11:40:00.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:00.032 7f3047ac0700 1 --2- 172.21.2.102:0/4031686476 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3034009590 0x7f303400bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:00.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:00.032 7f30472bf700 1 --2- 172.21.2.102:0/4031686476 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3034003aa0 0x7f3034005ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:00.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:00.032 7f3046abe700 1 --2- 172.21.2.102:0/4031686476 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3034006790 0x7f3034008c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:00.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:00.032 7f3047ac0700 1 -- 172.21.2.102:0/4031686476 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3034006790 msgr2=0x7f3034008c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:00.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:00.032 7f3047ac0700 1 --2- 172.21.2.102:0/4031686476 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3034006790 0x7f3034008c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:00.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:00.032 7f3047ac0700 1 -- 172.21.2.102:0/4031686476 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3034003aa0 msgr2=0x7f3034005ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:00.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:00.032 7f3047ac0700 1 --2- 172.21.2.102:0/4031686476 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3034003aa0 0x7f3034005ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:00.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:00.032 7f3047ac0700 1 -- 172.21.2.102:0/4031686476 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f303400c9e0 con 0x7f3034009590 2021-02-10T11:40:00.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:00.032 7f3046abe700 1 --2- 172.21.2.102:0/4031686476 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3034006790 0x7f3034008c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:40:00.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:00.032 7f30472bf700 1 --2- 172.21.2.102:0/4031686476 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3034003aa0 0x7f3034005ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:40:00.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:00.032 7f3047ac0700 1 -- 172.21.2.102:0/4031686476 reap_dead start 2021-02-10T11:40:00.052 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:00.032 7f3047ac0700 1 --2- 172.21.2.102:0/4031686476 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3034009590 0x7f303400bab0 secure :-1 s=READY pgs=3314 cs=0 l=1 rev1=1 rx=0x7f304004dc80 tx=0x7f304004f300).ready entity=mon.0 client_cookie=d2e22c0a31ed41b1 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:00.052 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:00.032 7f30462bd700 1 -- 172.21.2.102:0/4031686476 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f30400068b0 con 0x7f3034009590 2021-02-10T11:40:00.052 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:00.032 7f30462bd700 1 -- 172.21.2.102:0/4031686476 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3040003030 con 0x7f3034009590 2021-02-10T11:40:00.052 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:00.032 7f30462bd700 1 -- 172.21.2.102:0/4031686476 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f304004f970 con 0x7f3034009590 2021-02-10T11:40:00.052 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:00.033 7f304d5f8700 1 -- 172.21.2.102:0/4031686476 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3034009590 msgr2=0x7f303400bab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:00.052 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:00.033 7f304d5f8700 1 --2- 172.21.2.102:0/4031686476 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3034009590 0x7f303400bab0 secure :-1 s=READY pgs=3314 cs=0 l=1 rev1=1 rx=0x7f304004dc80 tx=0x7f304004f300).stop 2021-02-10T11:40:00.052 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:00.033 7f304d5f8700 1 -- 172.21.2.102:0/4031686476 shutdown_connections 2021-02-10T11:40:00.052 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:00.033 7f304d5f8700 1 --2- 172.21.2.102:0/4031686476 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3034009590 0x7f303400bab0 unknown :-1 s=CLOSED pgs=3314 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:00.052 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:00.033 7f304d5f8700 1 -- 172.21.2.102:0/4031686476 >> 172.21.2.102:0/4031686476 conn(0x7f30480fdee0 msgr2=0x7f30480ce450 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:00.052 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:00.033 7f304d5f8700 1 -- 172.21.2.102:0/4031686476 shutdown_connections 2021-02-10T11:40:00.052 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:00.033 7f304d5f8700 1 -- 172.21.2.102:0/4031686476 wait complete. 2021-02-10T11:40:00.052 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:00.048 7f304d5f8700 1 Processor -- start 2021-02-10T11:40:00.052 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:00.048 7f304d5f8700 1 -- start start 2021-02-10T11:40:00.054 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:00.048 7f304d5f8700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f304819f870 0x7f30480b34b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:00.054 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:00.048 7f304d5f8700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f30480b3ad0 0x7f3048096380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:00.054 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:00.048 7f304d5f8700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f30480969a0 0x7f30481a0510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:00.054 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:00.048 7f304d5f8700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f304818ac40 con 0x7f30480969a0 2021-02-10T11:40:00.054 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:00.048 7f304d5f8700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f304819fc70 con 0x7f304819f870 2021-02-10T11:40:00.054 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:00.048 7f304d5f8700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f304818a6d0 con 0x7f30480b3ad0 2021-02-10T11:40:00.054 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:00.048 7f3047ac0700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f30480969a0 0x7f30481a0510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:00.054 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:00.048 7f3047ac0700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f30480969a0 0x7f30481a0510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:40896/0 (socket says 172.21.2.102:40896) 2021-02-10T11:40:00.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:00.048 7f3047ac0700 1 -- 172.21.2.102:0/758616359 learned_addr learned my addr 172.21.2.102:0/758616359 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:40:00.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:00.048 7f30472bf700 1 --2- 172.21.2.102:0/758616359 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f304819f870 0x7f30480b34b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:00.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:00.048 7f3046abe700 1 --2- 172.21.2.102:0/758616359 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f30480b3ad0 0x7f3048096380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:00.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:00.049 7f3047ac0700 1 -- 172.21.2.102:0/758616359 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f30480b3ad0 msgr2=0x7f3048096380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:00.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:00.049 7f3047ac0700 1 --2- 172.21.2.102:0/758616359 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f30480b3ad0 0x7f3048096380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:00.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:00.049 7f3047ac0700 1 -- 172.21.2.102:0/758616359 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f304819f870 msgr2=0x7f30480b34b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:00.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:00.049 7f3047ac0700 1 --2- 172.21.2.102:0/758616359 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f304819f870 0x7f30480b34b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:00.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:00.049 7f3047ac0700 1 -- 172.21.2.102:0/758616359 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3048096da0 con 0x7f30480969a0 2021-02-10T11:40:00.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:00.049 7f30472bf700 1 --2- 172.21.2.102:0/758616359 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f304819f870 0x7f30480b34b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:40:00.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:00.049 7f3047ac0700 1 --2- 172.21.2.102:0/758616359 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f30480969a0 0x7f30481a0510 secure :-1 s=READY pgs=3315 cs=0 l=1 rev1=1 rx=0x7f3040050910 tx=0x7f3040050cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:00.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:00.049 7f303ffff700 1 -- 172.21.2.102:0/758616359 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f304004b510 con 0x7f30480969a0 2021-02-10T11:40:00.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:00.049 7f303ffff700 1 -- 172.21.2.102:0/758616359 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3040006d60 con 0x7f30480969a0 2021-02-10T11:40:00.055 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:00.049 7f304d5f8700 1 -- 172.21.2.102:0/758616359 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3048096da0 con 0x7f30480969a0 2021-02-10T11:40:00.056 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:00.049 7f303ffff700 1 -- 172.21.2.102:0/758616359 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f304004bb10 con 0x7f30480969a0 2021-02-10T11:40:00.056 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:00.049 7f304d5f8700 1 -- 172.21.2.102:0/758616359 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30481a0b30 con 0x7f30480969a0 2021-02-10T11:40:00.056 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:00.050 7f303ffff700 1 -- 172.21.2.102:0/758616359 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f304004b300 con 0x7f30480969a0 2021-02-10T11:40:00.056 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:00.050 7f303ffff700 1 --2- 172.21.2.102:0/758616359 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f302c0186c0 0x7f302c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:00.056 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:00.050 7f303ffff700 1 -- 172.21.2.102:0/758616359 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(189..189 src has 1..189) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f3040076b30 con 0x7f30480969a0 2021-02-10T11:40:00.056 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:00.050 7f30472bf700 1 --2- 172.21.2.102:0/758616359 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f302c0186c0 0x7f302c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:00.056 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:00.050 7f30472bf700 1 --2- 172.21.2.102:0/758616359 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f302c0186c0 0x7f302c01abc0 crc :-1 s=READY pgs=1445 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:00.082 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:00.081 7f304d5f8700 1 -- 172.21.2.102:0/758616359 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f304807e3b0 con 0x7f30480969a0 2021-02-10T11:40:00.085 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:00.083 7f303ffff700 1 -- 172.21.2.102:0/758616359 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f304004d520 con 0x7f30480969a0 2021-02-10T11:40:00.211 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:00.210 7f304d5f8700 1 -- 172.21.2.102:0/758616359 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume deauthorize", "vol_name": "cephfs", "sub_name": "subvolume_0000000000518037", "target": ["mgr", ""], "auth_id": "guest2", "group_name": "group_0000000000599689"}) v1 -- 0x7f3048109a60 con 0x7f302c0186c0 2021-02-10T11:40:00.244 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:00.243 7f303ffff700 1 -- 172.21.2.102:0/758616359 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f30300068c0 con 0x7f302c0186c0 2021-02-10T11:40:00.276 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:00.275 7f304d5f8700 1 -- 172.21.2.102:0/758616359 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f302c0186c0 msgr2=0x7f302c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:00.277 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:00.275 7f304d5f8700 1 --2- 172.21.2.102:0/758616359 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f302c0186c0 0x7f302c01abc0 crc :-1 s=READY pgs=1445 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:00.277 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:00.275 7f304d5f8700 1 -- 172.21.2.102:0/758616359 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f30480969a0 msgr2=0x7f30481a0510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:00.277 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:00.275 7f304d5f8700 1 --2- 172.21.2.102:0/758616359 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f30480969a0 0x7f30481a0510 secure :-1 s=READY pgs=3315 cs=0 l=1 rev1=1 rx=0x7f3040050910 tx=0x7f3040050cb0).stop 2021-02-10T11:40:00.277 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:00.275 7f304d5f8700 1 -- 172.21.2.102:0/758616359 shutdown_connections 2021-02-10T11:40:00.277 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:00.275 7f304d5f8700 1 --2- 172.21.2.102:0/758616359 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f30480969a0 0x7f30481a0510 unknown :-1 s=CLOSED pgs=3315 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:00.277 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:00.275 7f304d5f8700 1 --2- 172.21.2.102:0/758616359 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f30480b3ad0 0x7f3048096380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:00.277 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:00.275 7f304d5f8700 1 --2- 172.21.2.102:0/758616359 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f302c0186c0 0x7f302c01abc0 unknown :-1 s=CLOSED pgs=1445 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:00.277 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:00.275 7f304d5f8700 1 --2- 172.21.2.102:0/758616359 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f304819f870 0x7f30480b34b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:00.277 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:00.275 7f304d5f8700 1 -- 172.21.2.102:0/758616359 >> 172.21.2.102:0/758616359 conn(0x7f30481abcc0 msgr2=0x7f30481ae430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:00.277 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:00.275 7f304d5f8700 1 -- 172.21.2.102:0/758616359 shutdown_connections 2021-02-10T11:40:00.277 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:00.275 7f304d5f8700 1 -- 172.21.2.102:0/758616359 wait complete. 2021-02-10T11:40:00.285 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000518037 --group_name group_0000000000599689 2021-02-10T11:40:00.718 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:00.699 7fe2df842700 1 Processor -- start 2021-02-10T11:40:00.718 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:00.699 7fe2df842700 1 -- start start 2021-02-10T11:40:00.718 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:00.700 7fe2df842700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fe2d80d1870 0x7fe2d81b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:00.718 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:00.700 7fe2df842700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fe2d81b0ae0 0x7fe2d8108940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:00.719 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:00.700 7fe2df842700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2d80fe1c0 con 0x7fe2d80d1c70 2021-02-10T11:40:00.719 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:00.700 7fe2df842700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2d8109a60 con 0x7fe2d80d1870 2021-02-10T11:40:00.719 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:00.700 7fe2df842700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2d8109750 con 0x7fe2d81b0ae0 2021-02-10T11:40:00.719 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:00.700 7fe2dd65e700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fe2d80d1870 0x7fe2d81b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:00.719 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:00.700 7fe2dce5d700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fe2d81b0ae0 0x7fe2d8108940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:00.719 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:00.700 7fe2dde5f700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fe2d80d1c70 0x7fe2d8108f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:33124/0 (socket says 172.21.2.102:33124) 2021-02-10T11:40:00.719 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:00.700 7fe2dde5f700 1 -- 172.21.2.102:0/2347537818 learned_addr learned my addr 172.21.2.102:0/2347537818 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:40:00.719 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:00.701 7fe2dce5d700 1 -- 172.21.2.102:0/2347537818 >> v2:172.21.2.102:3300/0 conn(0x7fe2d80d1870 msgr2=0x7fe2d81b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:00.719 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:00.701 7fe2dce5d700 1 --2- 172.21.2.102:0/2347537818 >> v2:172.21.2.102:3300/0 conn(0x7fe2d80d1870 0x7fe2d81b04c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:00.719 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:00.701 7fe2dce5d700 1 -- 172.21.2.102:0/2347537818 >> v1:172.21.2.102:6789/0 conn(0x7fe2d80d1c70 legacy=0x7fe2d8108f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:00.719 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:00.701 7fe2dce5d700 1 -- 172.21.2.102:0/2347537818 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2d8088070 con 0x7fe2d81b0ae0 2021-02-10T11:40:00.719 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:00.701 7fe2dd65e700 1 --2- 172.21.2.102:0/2347537818 >> v2:172.21.2.102:3300/0 conn(0x7fe2d80d1870 0x7fe2d81b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:40:00.719 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:00.701 7fe2cbfff700 1 -- 172.21.2.102:0/2347537818 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3887433690 0 0) 0x7fe2d0000f90 con 0x7fe2d80d1c70 2021-02-10T11:40:00.720 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:00.701 7fe2dce5d700 1 --2- 172.21.2.102:0/2347537818 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe2d81b0ae0 0x7fe2d8108940 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 rx=0x7fe2d4006650 tx=0x7fe2d4009a80).ready entity=mon.1 client_cookie=ce856507f545b0a4 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:00.720 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:00.701 7fe2cbfff700 1 -- 172.21.2.102:0/2347537818 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe2d400c0c0 con 0x7fe2d81b0ae0 2021-02-10T11:40:00.720 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:00.701 7fe2cbfff700 1 -- 172.21.2.102:0/2347537818 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2d400c340 con 0x7fe2d81b0ae0 2021-02-10T11:40:00.720 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:00.701 7fe2cbfff700 1 -- 172.21.2.102:0/2347537818 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe2d400c820 con 0x7fe2d81b0ae0 2021-02-10T11:40:00.720 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:00.701 7fe2df842700 1 -- 172.21.2.102:0/2347537818 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe2d81b0ae0 msgr2=0x7fe2d8108940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:00.720 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:00.701 7fe2df842700 1 --2- 172.21.2.102:0/2347537818 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe2d81b0ae0 0x7fe2d8108940 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 rx=0x7fe2d4006650 tx=0x7fe2d4009a80).stop 2021-02-10T11:40:00.720 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:00.702 7fe2df842700 1 -- 172.21.2.102:0/2347537818 shutdown_connections 2021-02-10T11:40:00.720 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:00.702 7fe2df842700 1 --2- 172.21.2.102:0/2347537818 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe2d81b0ae0 0x7fe2d8108940 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:00.720 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:00.702 7fe2df842700 1 --2- 172.21.2.102:0/2347537818 >> v2:172.21.2.102:3300/0 conn(0x7fe2d80d1870 0x7fe2d81b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:00.720 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:00.702 7fe2df842700 1 -- 172.21.2.102:0/2347537818 >> 172.21.2.102:0/2347537818 conn(0x7fe2d80ce420 msgr2=0x7fe2d80d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:00.720 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:00.702 7fe2df842700 1 -- 172.21.2.102:0/2347537818 shutdown_connections 2021-02-10T11:40:00.720 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:00.702 7fe2df842700 1 -- 172.21.2.102:0/2347537818 wait complete. 2021-02-10T11:40:00.721 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:00.717 7fe2df842700 1 Processor -- start 2021-02-10T11:40:00.721 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:00.717 7fe2df842700 1 -- start start 2021-02-10T11:40:00.722 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:00.717 7fe2df842700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe2d81a05a0 0x7fe2d80a8c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:00.722 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:00.717 7fe2df842700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe2d80b5090 0x7fe2d80b5490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:00.722 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:00.717 7fe2df842700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe2d80b5ab0 0x7fe2d80b68d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:00.722 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:00.717 7fe2df842700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2d80b6090 con 0x7fe2d80b5ab0 2021-02-10T11:40:00.723 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:00.717 7fe2df842700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2d80a9240 con 0x7fe2d80b5090 2021-02-10T11:40:00.723 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:00.717 7fe2df842700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2d80b2a30 con 0x7fe2d81a05a0 2021-02-10T11:40:00.723 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:00.718 7fe2dde5f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe2d80b5ab0 0x7fe2d80b68d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:00.723 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:00.718 7fe2dde5f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe2d80b5ab0 0x7fe2d80b68d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:40908/0 (socket says 172.21.2.102:40908) 2021-02-10T11:40:00.723 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:00.718 7fe2dde5f700 1 -- 172.21.2.102:0/2197427093 learned_addr learned my addr 172.21.2.102:0/2197427093 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:40:00.723 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:00.718 7fe2dde5f700 1 -- 172.21.2.102:0/2197427093 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe2d81a05a0 msgr2=0x7fe2d80a8c20 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:40:00.723 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:00.718 7fe2dde5f700 1 --2- 172.21.2.102:0/2197427093 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe2d81a05a0 0x7fe2d80a8c20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:00.723 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:00.718 7fe2dde5f700 1 -- 172.21.2.102:0/2197427093 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe2d80b5090 msgr2=0x7fe2d80b5490 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:40:00.723 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:00.718 7fe2dde5f700 1 --2- 172.21.2.102:0/2197427093 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe2d80b5090 0x7fe2d80b5490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:00.723 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:00.718 7fe2dde5f700 1 -- 172.21.2.102:0/2197427093 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2d80b6ef0 con 0x7fe2d80b5ab0 2021-02-10T11:40:00.723 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:00.718 7fe2dde5f700 1 --2- 172.21.2.102:0/2197427093 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe2d80b5ab0 0x7fe2d80b68d0 secure :-1 s=READY pgs=3317 cs=0 l=1 rev1=1 rx=0x7fe2d000a900 tx=0x7fe2d000aca0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:00.723 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:00.718 7fe2c9ffb700 1 -- 172.21.2.102:0/2197427093 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe2d0000f90 con 0x7fe2d80b5ab0 2021-02-10T11:40:00.723 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:00.718 7fe2c9ffb700 1 -- 172.21.2.102:0/2197427093 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2d0007020 con 0x7fe2d80b5ab0 2021-02-10T11:40:00.724 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:00.718 7fe2df842700 1 -- 172.21.2.102:0/2197427093 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2d80b6ef0 con 0x7fe2d80b5ab0 2021-02-10T11:40:00.724 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:00.718 7fe2df842700 1 -- 172.21.2.102:0/2197427093 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2d80b6ef0 con 0x7fe2d80b5ab0 2021-02-10T11:40:00.724 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:00.718 7fe2c9ffb700 1 -- 172.21.2.102:0/2197427093 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe2d0007470 con 0x7fe2d80b5ab0 2021-02-10T11:40:00.724 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:00.719 7fe2c9ffb700 1 -- 172.21.2.102:0/2197427093 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fe2d0000d10 con 0x7fe2d80b5ab0 2021-02-10T11:40:00.724 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:00.719 7fe2c9ffb700 1 --2- 172.21.2.102:0/2197427093 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe2b8018730 0x7fe2b801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:00.724 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:00.719 7fe2c9ffb700 1 -- 172.21.2.102:0/2197427093 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(189..189 src has 1..189) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fe2d0032b80 con 0x7fe2d80b5ab0 2021-02-10T11:40:00.724 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:00.719 7fe2dd65e700 1 --2- 172.21.2.102:0/2197427093 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe2b8018730 0x7fe2b801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:00.724 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:00.719 7fe2dd65e700 1 --2- 172.21.2.102:0/2197427093 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe2b8018730 0x7fe2b801ac10 crc :-1 s=READY pgs=1446 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:00.751 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:00.749 7fe2df842700 1 -- 172.21.2.102:0/2197427093 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2d80825c0 con 0x7fe2d80b5ab0 2021-02-10T11:40:00.753 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:00.752 7fe2c9ffb700 1 -- 172.21.2.102:0/2197427093 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fe2d0007c90 con 0x7fe2d80b5ab0 2021-02-10T11:40:00.879 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:00.878 7fe2df842700 1 -- 172.21.2.102:0/2197427093 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000518037", "target": ["mgr", ""], "group_name": "group_0000000000599689"}) v1 -- 0x7fe2d80825c0 con 0x7fe2b8018730 2021-02-10T11:40:00.890 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:00.889 7fe2c9ffb700 1 -- 172.21.2.102:0/2197427093 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fe2cc0009b0 con 0x7fe2b8018730 2021-02-10T11:40:00.896 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:00.895 7fe2df842700 1 -- 172.21.2.102:0/2197427093 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe2b8018730 msgr2=0x7fe2b801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:00.896 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:00.895 7fe2df842700 1 --2- 172.21.2.102:0/2197427093 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe2b8018730 0x7fe2b801ac10 crc :-1 s=READY pgs=1446 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:00.896 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:00.895 7fe2df842700 1 -- 172.21.2.102:0/2197427093 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe2d80b5ab0 msgr2=0x7fe2d80b68d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:00.896 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:00.895 7fe2df842700 1 --2- 172.21.2.102:0/2197427093 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe2d80b5ab0 0x7fe2d80b68d0 secure :-1 s=READY pgs=3317 cs=0 l=1 rev1=1 rx=0x7fe2d000a900 tx=0x7fe2d000aca0).stop 2021-02-10T11:40:00.897 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:00.895 7fe2df842700 1 -- 172.21.2.102:0/2197427093 shutdown_connections 2021-02-10T11:40:00.897 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:00.895 7fe2df842700 1 --2- 172.21.2.102:0/2197427093 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe2d80b5ab0 0x7fe2d80b68d0 unknown :-1 s=CLOSED pgs=3317 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:00.897 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:00.895 7fe2df842700 1 --2- 172.21.2.102:0/2197427093 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe2b8018730 0x7fe2b801ac10 unknown :-1 s=CLOSED pgs=1446 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:00.897 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:00.895 7fe2df842700 1 --2- 172.21.2.102:0/2197427093 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe2d80b5090 0x7fe2d80b5490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:00.897 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:00.895 7fe2df842700 1 --2- 172.21.2.102:0/2197427093 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe2d81a05a0 0x7fe2d80a8c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:00.897 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:00.895 7fe2df842700 1 -- 172.21.2.102:0/2197427093 >> 172.21.2.102:0/2197427093 conn(0x7fe2d80bc130 msgr2=0x7fe2d80a9630 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:00.897 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:00.895 7fe2df842700 1 -- 172.21.2.102:0/2197427093 shutdown_connections 2021-02-10T11:40:00.897 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:00.895 7fe2df842700 1 -- 172.21.2.102:0/2197427093 wait complete. 2021-02-10T11:40:00.905 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolumegroup rm cephfs group_0000000000599689 2021-02-10T11:40:01.338 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:01.321 7f713bb4a700 1 Processor -- start 2021-02-10T11:40:01.338 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:01.321 7f713bb4a700 1 -- start start 2021-02-10T11:40:01.339 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:01.322 7f713bb4a700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f71340d1870 0x7f71341b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:01.339 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:01.322 7f713bb4a700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f71341b0ae0 0x7f7134108940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:01.339 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:01.322 7f713bb4a700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f71340d1c70 0x7f7134108f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:01.339 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:01.322 7f713bb4a700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71340fe1c0 con 0x7f71340d1870 2021-02-10T11:40:01.339 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:01.322 7f713bb4a700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7134109a60 con 0x7f71340d1c70 2021-02-10T11:40:01.339 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:01.322 7f713bb4a700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7134088070 con 0x7f71341b0ae0 2021-02-10T11:40:01.339 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:01.322 7f7139966700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f71340d1870 0x7f71341b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:01.339 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:01.322 7f7139966700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f71340d1870 0x7f71341b04c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:40912/0 (socket says 172.21.2.102:40912) 2021-02-10T11:40:01.339 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:01.322 7f7139966700 1 -- 172.21.2.102:0/760210065 learned_addr learned my addr 172.21.2.102:0/760210065 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:40:01.339 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:01.322 7f7139966700 1 -- 172.21.2.102:0/760210065 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f71341b0ae0 msgr2=0x7f7134108940 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:40:01.339 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:01.322 7f7139966700 1 --2- 172.21.2.102:0/760210065 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f71341b0ae0 0x7f7134108940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:01.339 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:01.322 7f7139966700 1 -- 172.21.2.102:0/760210065 >> v2:172.21.2.113:3300/0 conn(0x7f71340d1c70 msgr2=0x7f7134108f60 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:40:01.339 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:01.322 7f7139966700 1 --2- 172.21.2.102:0/760210065 >> v2:172.21.2.113:3300/0 conn(0x7f71340d1c70 0x7f7134108f60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:01.340 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:01.322 7f7139966700 1 -- 172.21.2.102:0/760210065 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7134109580 con 0x7f71340d1870 2021-02-10T11:40:01.340 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:01.322 7f7139966700 1 --2- 172.21.2.102:0/760210065 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f71340d1870 0x7f71341b04c0 secure :-1 s=READY pgs=3318 cs=0 l=1 rev1=1 rx=0x7f7124006700 tx=0x7f7124009b00).ready entity=mon.0 client_cookie=5f2ac42b64c05f4a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:01.340 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:01.322 7f7138964700 1 -- 172.21.2.102:0/760210065 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f712400c0c0 con 0x7f71340d1870 2021-02-10T11:40:01.340 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:01.322 7f7138964700 1 -- 172.21.2.102:0/760210065 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f712400c340 con 0x7f71340d1870 2021-02-10T11:40:01.340 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:01.323 7f7138964700 1 -- 172.21.2.102:0/760210065 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f712400c820 con 0x7f71340d1870 2021-02-10T11:40:01.340 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:01.323 7f713bb4a700 1 -- 172.21.2.102:0/760210065 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f71340d1870 msgr2=0x7f71341b04c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:01.340 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:01.323 7f713bb4a700 1 --2- 172.21.2.102:0/760210065 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f71340d1870 0x7f71341b04c0 secure :-1 s=READY pgs=3318 cs=0 l=1 rev1=1 rx=0x7f7124006700 tx=0x7f7124009b00).stop 2021-02-10T11:40:01.340 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:01.323 7f713bb4a700 1 -- 172.21.2.102:0/760210065 shutdown_connections 2021-02-10T11:40:01.340 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:01.323 7f713bb4a700 1 --2- 172.21.2.102:0/760210065 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f71341b0ae0 0x7f7134108940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:01.340 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:01.323 7f713bb4a700 1 --2- 172.21.2.102:0/760210065 >> v2:172.21.2.113:3300/0 conn(0x7f71340d1c70 0x7f7134108f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:01.340 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:01.323 7f713bb4a700 1 --2- 172.21.2.102:0/760210065 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f71340d1870 0x7f71341b04c0 unknown :-1 s=CLOSED pgs=3318 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:01.340 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:01.323 7f713bb4a700 1 -- 172.21.2.102:0/760210065 >> 172.21.2.102:0/760210065 conn(0x7f71340ce420 msgr2=0x7f71340d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:01.340 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:01.323 7f713bb4a700 1 -- 172.21.2.102:0/760210065 shutdown_connections 2021-02-10T11:40:01.341 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:01.323 7f713bb4a700 1 -- 172.21.2.102:0/760210065 wait complete. 2021-02-10T11:40:01.342 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:01.337 7f713bb4a700 1 Processor -- start 2021-02-10T11:40:01.342 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:01.338 7f713bb4a700 1 -- start start 2021-02-10T11:40:01.342 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:01.338 7f713bb4a700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f71340c3820 0x7f7134084cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:01.342 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:01.338 7f713bb4a700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f71340a0290 0x7f71340852d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:01.342 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:01.338 7f713bb4a700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7134086270 0x7f7134086670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:01.342 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:01.338 7f713bb4a700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71340c0700 con 0x7f71340a0290 2021-02-10T11:40:01.342 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:01.338 7f713bb4a700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71340858f0 con 0x7f7134086270 2021-02-10T11:40:01.342 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:01.338 7f713bb4a700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7134085b10 con 0x7f71340c3820 2021-02-10T11:40:01.342 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:01.338 7f7139165700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f71340a0290 0x7f71340852d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:01.343 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:01.338 7f7139165700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f71340a0290 0x7f71340852d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:40922/0 (socket says 172.21.2.102:40922) 2021-02-10T11:40:01.343 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:01.338 7f7139165700 1 -- 172.21.2.102:0/2839414780 learned_addr learned my addr 172.21.2.102:0/2839414780 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:40:01.343 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:01.338 7f7139165700 1 -- 172.21.2.102:0/2839414780 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f71340c3820 msgr2=0x7f7134084cb0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:40:01.343 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:01.338 7f7139165700 1 --2- 172.21.2.102:0/2839414780 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f71340c3820 0x7f7134084cb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:01.343 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:01.338 7f7139165700 1 -- 172.21.2.102:0/2839414780 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7134086270 msgr2=0x7f7134086670 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:40:01.343 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:01.338 7f7139165700 1 --2- 172.21.2.102:0/2839414780 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7134086270 0x7f7134086670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:01.343 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:01.338 7f7139165700 1 -- 172.21.2.102:0/2839414780 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7134086c90 con 0x7f71340a0290 2021-02-10T11:40:01.343 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:01.339 7f7139165700 1 --2- 172.21.2.102:0/2839414780 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f71340a0290 0x7f71340852d0 secure :-1 s=READY pgs=3319 cs=0 l=1 rev1=1 rx=0x7f713000b560 tx=0x7f713000b900).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:01.343 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:01.339 7f712a7fc700 1 -- 172.21.2.102:0/2839414780 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f713000bcf0 con 0x7f71340a0290 2021-02-10T11:40:01.343 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:01.339 7f712a7fc700 1 -- 172.21.2.102:0/2839414780 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7130006030 con 0x7f71340a0290 2021-02-10T11:40:01.343 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:01.339 7f713bb4a700 1 -- 172.21.2.102:0/2839414780 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7134086c90 con 0x7f71340a0290 2021-02-10T11:40:01.343 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:01.339 7f712a7fc700 1 -- 172.21.2.102:0/2839414780 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f71300054d0 con 0x7f71340a0290 2021-02-10T11:40:01.343 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:01.339 7f713bb4a700 1 -- 172.21.2.102:0/2839414780 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7134086c90 con 0x7f71340a0290 2021-02-10T11:40:01.344 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:01.340 7f712a7fc700 1 -- 172.21.2.102:0/2839414780 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f7130006210 con 0x7f71340a0290 2021-02-10T11:40:01.344 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:01.340 7f712a7fc700 1 --2- 172.21.2.102:0/2839414780 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f711c018730 0x7f711c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:01.344 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:01.340 7f712a7fc700 1 -- 172.21.2.102:0/2839414780 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(189..189 src has 1..189) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f71300327d0 con 0x7f71340a0290 2021-02-10T11:40:01.344 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:01.340 7f7139966700 1 --2- 172.21.2.102:0/2839414780 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f711c018730 0x7f711c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:01.344 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:01.340 7f7139966700 1 --2- 172.21.2.102:0/2839414780 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f711c018730 0x7f711c01abc0 crc :-1 s=READY pgs=1447 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:01.373 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:01.372 7f713bb4a700 1 -- 172.21.2.102:0/2839414780 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7134082a70 con 0x7f71340a0290 2021-02-10T11:40:01.375 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:01.374 7f712a7fc700 1 -- 172.21.2.102:0/2839414780 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f7130006b80 con 0x7f71340a0290 2021-02-10T11:40:01.506 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:01.505 7f713bb4a700 1 -- 172.21.2.102:0/2839414780 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolumegroup rm", "vol_name": "cephfs", "target": ["mgr", ""], "group_name": "group_0000000000599689"}) v1 -- 0x7f7134082a70 con 0x7f711c018730 2021-02-10T11:40:01.509 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:01.508 7f712a7fc700 1 -- 172.21.2.102:0/2839414780 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f7124009390 con 0x7f711c018730 2021-02-10T11:40:01.510 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:01.509 7f713bb4a700 1 -- 172.21.2.102:0/2839414780 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f711c018730 msgr2=0x7f711c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:01.510 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:01.509 7f713bb4a700 1 --2- 172.21.2.102:0/2839414780 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f711c018730 0x7f711c01abc0 crc :-1 s=READY pgs=1447 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:01.511 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:01.509 7f713bb4a700 1 -- 172.21.2.102:0/2839414780 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f71340a0290 msgr2=0x7f71340852d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:01.511 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:01.509 7f713bb4a700 1 --2- 172.21.2.102:0/2839414780 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f71340a0290 0x7f71340852d0 secure :-1 s=READY pgs=3319 cs=0 l=1 rev1=1 rx=0x7f713000b560 tx=0x7f713000b900).stop 2021-02-10T11:40:01.511 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:01.509 7f713bb4a700 1 -- 172.21.2.102:0/2839414780 shutdown_connections 2021-02-10T11:40:01.511 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:01.509 7f713bb4a700 1 --2- 172.21.2.102:0/2839414780 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f711c018730 0x7f711c01abc0 unknown :-1 s=CLOSED pgs=1447 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:01.511 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:01.509 7f713bb4a700 1 --2- 172.21.2.102:0/2839414780 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7134086270 0x7f7134086670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:01.511 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:01.509 7f713bb4a700 1 --2- 172.21.2.102:0/2839414780 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f71340a0290 0x7f71340852d0 unknown :-1 s=CLOSED pgs=3319 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:01.511 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:01.509 7f713bb4a700 1 --2- 172.21.2.102:0/2839414780 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f71340c3820 0x7f7134084cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:01.511 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:01.509 7f713bb4a700 1 -- 172.21.2.102:0/2839414780 >> 172.21.2.102:0/2839414780 conn(0x7f71340872c0 msgr2=0x7f7134095700 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:01.511 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:01.509 7f713bb4a700 1 -- 172.21.2.102:0/2839414780 shutdown_connections 2021-02-10T11:40:01.511 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:01.509 7f713bb4a700 1 -- 172.21.2.102:0/2839414780 wait complete. 2021-02-10T11:40:01.518 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2021-02-10T11:40:01.518 DEBUG:teuthology.orchestra.run.gibba002:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:40:01.864 DEBUG:teuthology.orchestra.run.gibba013:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:40:02.214 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2021-02-10T11:40:02.215 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:40:02.215 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:40:02.215 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:40:02.554 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:40:02.554 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:40:02.555 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@gibba002.front.sepia.ceph.com... 2021-02-10T11:40:02.555 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-02-10T11:40:02.555 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:40:02.909 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba002.stderr:ceph-fuse[216940]: fuse finished with error 0 and tester_r 0 2021-02-10T11:40:02.911 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:40:02.911 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:40:02.911 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:40:03.254 INFO:teuthology.orchestra.run.gibba002.stdout:ext2/ext3 2021-02-10T11:40:03.255 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2021-02-10T11:40:03.255 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:40:03.255 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:40:03.591 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:40:03.592 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:40:03.592 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:40:03.928 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:40:03.928 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2021-02-10T11:40:03.929 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@gibba002.front.sepia.ceph.com... 2021-02-10T11:40:03.929 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-02-10T11:40:03.929 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:40:04.284 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba002.stderr:ceph-fuse[217640]: fuse finished with error 0 and tester_r 0 2021-02-10T11:40:04.285 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:40:04.286 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:40:04.286 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:40:04.620 INFO:teuthology.orchestra.run.gibba002.stdout:ext2/ext3 2021-02-10T11:40:04.621 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2021-02-10T11:40:04.621 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:40:04.621 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:40:04.956 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:40:04.957 DEBUG:teuthology.orchestra.run.gibba013:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:40:04.957 DEBUG:teuthology.orchestra.run.gibba013:> ' -- /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:40:05.298 INFO:teuthology.orchestra.run.gibba013.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.2’: No such file or directory 2021-02-10T11:40:05.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:40:05.298 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2021-02-10T11:40:05.299 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:40:05.299 DEBUG:teuthology.orchestra.run.gibba013:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:40:05.634 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:40:05.635 DEBUG:teuthology.orchestra.run.gibba013:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:40:05.635 DEBUG:teuthology.orchestra.run.gibba013:> ' -- /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:40:05.970 INFO:teuthology.orchestra.run.gibba013.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.3’: No such file or directory 2021-02-10T11:40:05.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:40:05.970 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.3 2021-02-10T11:40:05.970 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:40:05.971 DEBUG:teuthology.orchestra.run.gibba013:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:40:06.305 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config rm mon mon_allow_pool_delete 2021-02-10T11:40:06.740 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:06.720 7f2cd7fc1700 1 Processor -- start 2021-02-10T11:40:06.740 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:06.721 7f2cd7fc1700 1 -- start start 2021-02-10T11:40:06.740 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:06.721 7f2cd7fc1700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f2cd00d1870 0x7f2cd01b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:06.740 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:06.721 7f2cd7fc1700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cd00f0b10 con 0x7f2cd01b0a10 2021-02-10T11:40:06.740 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:06.721 7f2cd7fc1700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cd00d0310 con 0x7f2cd00f0650 2021-02-10T11:40:06.740 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:06.721 7f2cd7fc1700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2cd00d0530 con 0x7f2cd00d1870 2021-02-10T11:40:06.741 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:06.721 7f2cd55dc700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f2cd01b0a10 0x7f2cd00cf330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:33150/0 (socket says 172.21.2.102:33150) 2021-02-10T11:40:06.741 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:06.721 7f2cd55dc700 1 -- 172.21.2.102:0/3782336881 learned_addr learned my addr 172.21.2.102:0/3782336881 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:40:06.741 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:06.722 7f2cd4ddb700 1 -- 172.21.2.102:0/3782336881 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2526767931 0 0) 0x7f2ccc0010b0 con 0x7f2cd01b0a10 2021-02-10T11:40:06.741 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:06.722 7f2cd4ddb700 1 -- 172.21.2.102:0/3782336881 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2cbc002fe0 con 0x7f2cd01b0a10 2021-02-10T11:40:06.741 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:06.722 7f2cd5ddd700 1 --2- 172.21.2.102:0/3782336881 >> v2:172.21.2.113:3300/0 conn(0x7f2cd00d1870 0x7f2cd01b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:06.741 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:06.722 7f2cd4ddb700 1 -- 172.21.2.102:0/3782336881 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3980014666 0 0) 0x7f2ccc0010b0 con 0x7f2cd01b0a10 2021-02-10T11:40:06.741 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:06.722 7f2cd4ddb700 1 -- 172.21.2.102:0/3782336881 >> v2:172.21.2.113:3300/0 conn(0x7f2cd00d1870 msgr2=0x7f2cd01b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:06.741 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:06.722 7f2cd4ddb700 1 --2- 172.21.2.102:0/3782336881 >> v2:172.21.2.113:3300/0 conn(0x7f2cd00d1870 0x7f2cd01b03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:06.741 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:06.722 7f2cd4ddb700 1 -- 172.21.2.102:0/3782336881 >> v1:172.21.2.113:6789/0 conn(0x7f2cd00f0650 legacy=0x7f2cd00cfb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:06.741 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:06.722 7f2cd4ddb700 1 -- 172.21.2.102:0/3782336881 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cd00d1c70 con 0x7f2cd01b0a10 2021-02-10T11:40:06.741 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:06.722 7f2cd4ddb700 1 -- 172.21.2.102:0/3782336881 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7f2ccc0015b0 con 0x7f2cd01b0a10 2021-02-10T11:40:06.741 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:06.722 7f2cd4ddb700 1 -- 172.21.2.102:0/3782336881 >> v1:172.21.2.102:6789/0 conn(0x7f2cd01b0a10 legacy=0x7f2cd00cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:06.741 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:06.722 7f2cd4ddb700 1 --2- 172.21.2.102:0/3782336881 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2cbc003a60 0x7f2cbc005f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:06.742 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:06.722 7f2cd4ddb700 1 --2- 172.21.2.102:0/3782336881 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2cbc006780 0x7f2cbc008ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:06.742 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:06.722 7f2cd4ddb700 1 --2- 172.21.2.102:0/3782336881 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2cbc009580 0x7f2cbc00baa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:06.742 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:06.722 7f2cd4ddb700 1 -- 172.21.2.102:0/3782336881 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cbc00c2f0 con 0x7f2cbc009580 2021-02-10T11:40:06.742 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:06.722 7f2cd4ddb700 1 -- 172.21.2.102:0/3782336881 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cbc00c510 con 0x7f2cbc006780 2021-02-10T11:40:06.742 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:06.722 7f2cd4ddb700 1 -- 172.21.2.102:0/3782336881 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cbc00c730 con 0x7f2cbc003a60 2021-02-10T11:40:06.742 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:06.722 7f2cd65de700 1 --2- 172.21.2.102:0/3782336881 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2cbc009580 0x7f2cbc00baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:06.742 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:06.722 7f2cd55dc700 1 --2- 172.21.2.102:0/3782336881 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2cbc006780 0x7f2cbc008ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:06.742 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:06.722 7f2cd5ddd700 1 --2- 172.21.2.102:0/3782336881 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2cbc003a60 0x7f2cbc005f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:06.742 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:06.723 7f2cd65de700 1 -- 172.21.2.102:0/3782336881 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2cbc003a60 msgr2=0x7f2cbc005f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:06.742 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:06.723 7f2cd65de700 1 --2- 172.21.2.102:0/3782336881 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2cbc003a60 0x7f2cbc005f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:06.742 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:06.723 7f2cd65de700 1 -- 172.21.2.102:0/3782336881 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2cbc006780 msgr2=0x7f2cbc008ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:06.743 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:06.723 7f2cd65de700 1 --2- 172.21.2.102:0/3782336881 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2cbc006780 0x7f2cbc008ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:06.743 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:06.723 7f2cd65de700 1 -- 172.21.2.102:0/3782336881 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cbc0509a0 con 0x7f2cbc009580 2021-02-10T11:40:06.743 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:06.723 7f2cd65de700 1 -- 172.21.2.102:0/3782336881 reap_dead start 2021-02-10T11:40:06.743 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:06.723 7f2cd65de700 1 --2- 172.21.2.102:0/3782336881 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2cbc009580 0x7f2cbc00baa0 secure :-1 s=READY pgs=3321 cs=0 l=1 rev1=1 rx=0x7f2cc8006e30 tx=0x7f2cc8009dd0).ready entity=mon.0 client_cookie=8b82ca8f417814fb server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:06.743 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:06.723 7f2cd4ddb700 1 -- 172.21.2.102:0/3782336881 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2cc8001400 con 0x7f2cbc009580 2021-02-10T11:40:06.743 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:06.723 7f2cd4ddb700 1 -- 172.21.2.102:0/3782336881 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cc8006a70 con 0x7f2cbc009580 2021-02-10T11:40:06.743 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:06.723 7f2cd4ddb700 1 -- 172.21.2.102:0/3782336881 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2cc8001640 con 0x7f2cbc009580 2021-02-10T11:40:06.743 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:06.723 7f2cd7fc1700 1 -- 172.21.2.102:0/3782336881 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2cbc009580 msgr2=0x7f2cbc00baa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:06.743 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:06.723 7f2cd7fc1700 1 --2- 172.21.2.102:0/3782336881 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2cbc009580 0x7f2cbc00baa0 secure :-1 s=READY pgs=3321 cs=0 l=1 rev1=1 rx=0x7f2cc8006e30 tx=0x7f2cc8009dd0).stop 2021-02-10T11:40:06.743 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:06.724 7f2cd7fc1700 1 -- 172.21.2.102:0/3782336881 shutdown_connections 2021-02-10T11:40:06.743 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:06.724 7f2cd7fc1700 1 --2- 172.21.2.102:0/3782336881 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2cbc009580 0x7f2cbc00baa0 unknown :-1 s=CLOSED pgs=3321 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:06.743 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:06.724 7f2cd7fc1700 1 -- 172.21.2.102:0/3782336881 >> 172.21.2.102:0/3782336881 conn(0x7f2cd00ce450 msgr2=0x7f2cd00d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:06.743 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:06.724 7f2cd7fc1700 1 -- 172.21.2.102:0/3782336881 shutdown_connections 2021-02-10T11:40:06.744 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:06.724 7f2cd7fc1700 1 -- 172.21.2.102:0/3782336881 wait complete. 2021-02-10T11:40:06.745 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:06.739 7f2cd7fc1700 1 Processor -- start 2021-02-10T11:40:06.746 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:06.740 7f2cd7fc1700 1 -- start start 2021-02-10T11:40:06.746 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:06.740 7f2cd7fc1700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2cd0094310 0x7f2cd00ebe70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:06.746 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:06.740 7f2cd7fc1700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2cd0094710 0x7f2cd00bca30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:06.746 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:06.740 7f2cd7fc1700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2cd00ec490 0x7f2cd00bd050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:06.746 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:06.740 7f2cd7fc1700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cd0093d90 con 0x7f2cd0094710 2021-02-10T11:40:06.746 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:06.740 7f2cd7fc1700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cd00bd670 con 0x7f2cd00ec490 2021-02-10T11:40:06.746 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:06.740 7f2cd7fc1700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cd00939f0 con 0x7f2cd0094310 2021-02-10T11:40:06.746 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:06.740 7f2cd55dc700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2cd0094710 0x7f2cd00bca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:06.746 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:06.740 7f2cd55dc700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2cd0094710 0x7f2cd00bca30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:40940/0 (socket says 172.21.2.102:40940) 2021-02-10T11:40:06.746 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:06.740 7f2cd55dc700 1 -- 172.21.2.102:0/610061923 learned_addr learned my addr 172.21.2.102:0/610061923 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:40:06.746 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:06.740 7f2cd5ddd700 1 --2- 172.21.2.102:0/610061923 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2cd0094310 0x7f2cd00ebe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:06.746 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:06.740 7f2cd65de700 1 --2- 172.21.2.102:0/610061923 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2cd00ec490 0x7f2cd00bd050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:06.747 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:06.741 7f2cd55dc700 1 -- 172.21.2.102:0/610061923 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2cd0094310 msgr2=0x7f2cd00ebe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:06.747 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:06.741 7f2cd55dc700 1 --2- 172.21.2.102:0/610061923 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2cd0094310 0x7f2cd00ebe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:06.747 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:06.741 7f2cd55dc700 1 -- 172.21.2.102:0/610061923 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2cd00ec490 msgr2=0x7f2cd00bd050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:06.747 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:06.741 7f2cd55dc700 1 --2- 172.21.2.102:0/610061923 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2cd00ec490 0x7f2cd00bd050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:06.747 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:06.741 7f2cd55dc700 1 -- 172.21.2.102:0/610061923 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cd00bda80 con 0x7f2cd0094710 2021-02-10T11:40:06.747 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:06.741 7f2cd55dc700 1 --2- 172.21.2.102:0/610061923 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2cd0094710 0x7f2cd00bca30 secure :-1 s=READY pgs=3322 cs=0 l=1 rev1=1 rx=0x7f2ccc007650 tx=0x7f2ccc00a900).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:06.747 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:06.741 7f2cc27fc700 1 -- 172.21.2.102:0/610061923 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2ccc00acf0 con 0x7f2cd0094710 2021-02-10T11:40:06.747 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:06.741 7f2cc27fc700 1 -- 172.21.2.102:0/610061923 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ccc004350 con 0x7f2cd0094710 2021-02-10T11:40:06.747 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:06.741 7f2cd7fc1700 1 -- 172.21.2.102:0/610061923 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2cd00bda80 con 0x7f2cd0094710 2021-02-10T11:40:06.747 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:06.741 7f2cc27fc700 1 -- 172.21.2.102:0/610061923 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2ccc004870 con 0x7f2cd0094710 2021-02-10T11:40:06.747 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:06.741 7f2cd7fc1700 1 -- 172.21.2.102:0/610061923 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2cd00bda80 con 0x7f2cd0094710 2021-02-10T11:40:06.747 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:06.742 7f2cc27fc700 1 -- 172.21.2.102:0/610061923 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f2ccc004530 con 0x7f2cd0094710 2021-02-10T11:40:06.747 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:06.742 7f2cc27fc700 1 --2- 172.21.2.102:0/610061923 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2cb8018730 0x7f2cb801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:06.748 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:06.742 7f2cc27fc700 1 -- 172.21.2.102:0/610061923 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(189..189 src has 1..189) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f2ccc0327b0 con 0x7f2cd0094710 2021-02-10T11:40:06.748 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:06.742 7f2cd5ddd700 1 --2- 172.21.2.102:0/610061923 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2cb8018730 0x7f2cb801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:06.748 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:06.742 7f2cd5ddd700 1 --2- 172.21.2.102:0/610061923 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2cb8018730 0x7f2cb801ac10 crc :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:06.773 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:06.771 7f2cd7fc1700 1 -- 172.21.2.102:0/610061923 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2cd00ebc40 con 0x7f2cd0094710 2021-02-10T11:40:06.775 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:06.774 7f2cc27fc700 1 -- 172.21.2.102:0/610061923 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f2ccc004140 con 0x7f2cd0094710 2021-02-10T11:40:06.927 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:06.926 7f2cd7fc1700 1 -- 172.21.2.102:0/610061923 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f2cd00f0b10 con 0x7f2cd0094710 2021-02-10T11:40:06.931 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:06.930 7f2cc27fc700 1 -- 172.21.2.102:0/610061923 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v47) v1 ==== 104+0+0 (secure 0 0 0) 0x7f2ccc004350 con 0x7f2cd0094710 2021-02-10T11:40:06.937 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:06.935 7f2cd7fc1700 1 -- 172.21.2.102:0/610061923 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2cb8018730 msgr2=0x7f2cb801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:06.937 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:06.935 7f2cd7fc1700 1 --2- 172.21.2.102:0/610061923 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2cb8018730 0x7f2cb801ac10 crc :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:06.937 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:06.935 7f2cd7fc1700 1 -- 172.21.2.102:0/610061923 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2cd0094710 msgr2=0x7f2cd00bca30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:06.937 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:06.935 7f2cd7fc1700 1 --2- 172.21.2.102:0/610061923 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2cd0094710 0x7f2cd00bca30 secure :-1 s=READY pgs=3322 cs=0 l=1 rev1=1 rx=0x7f2ccc007650 tx=0x7f2ccc00a900).stop 2021-02-10T11:40:06.937 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:06.935 7f2cd7fc1700 1 -- 172.21.2.102:0/610061923 shutdown_connections 2021-02-10T11:40:06.937 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:06.935 7f2cd7fc1700 1 --2- 172.21.2.102:0/610061923 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2cb8018730 0x7f2cb801ac10 unknown :-1 s=CLOSED pgs=1448 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:06.937 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:06.935 7f2cd7fc1700 1 --2- 172.21.2.102:0/610061923 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2cd00ec490 0x7f2cd00bd050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:06.937 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:06.935 7f2cd7fc1700 1 --2- 172.21.2.102:0/610061923 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2cd0094710 0x7f2cd00bca30 unknown :-1 s=CLOSED pgs=3322 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:06.937 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:06.935 7f2cd7fc1700 1 --2- 172.21.2.102:0/610061923 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2cd0094310 0x7f2cd00ebe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:06.937 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:06.935 7f2cd7fc1700 1 -- 172.21.2.102:0/610061923 >> 172.21.2.102:0/610061923 conn(0x7f2cd008fa10 msgr2=0x7f2cd00bfa60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:06.937 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:06.935 7f2cd7fc1700 1 -- 172.21.2.102:0/610061923 shutdown_connections 2021-02-10T11:40:06.938 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:06.935 7f2cd7fc1700 1 -- 172.21.2.102:0/610061923 wait complete. 2021-02-10T11:40:06.945 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestVolumes.test_subvolume_authorized_list' 2021-02-10T11:40:07.498 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:07.357 7fdedd2df700 1 Processor -- start 2021-02-10T11:40:07.499 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:07.357 7fdedd2df700 1 -- start start 2021-02-10T11:40:07.499 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:07.358 7fdedd2df700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fded80d1870 0x7fded81b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:07.499 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:07.358 7fdedd2df700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fded80f0650 0x7fded80cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:07.499 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:07.358 7fdedd2df700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fded80f0b10 con 0x7fded81b0a10 2021-02-10T11:40:07.499 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:07.358 7fdedd2df700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fded80e5ed0 con 0x7fded80f0650 2021-02-10T11:40:07.499 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:07.358 7fdedd2df700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fded80d1c70 con 0x7fded80d1870 2021-02-10T11:40:07.499 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:07.358 7fded787e700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fded80f0650 0x7fded80cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:07.499 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:07.358 7fded787e700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fded80f0650 0x7fded80cfb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:40946/0 (socket says 172.21.2.102:40946) 2021-02-10T11:40:07.499 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:07.358 7fded787e700 1 -- 172.21.2.102:0/1554967887 learned_addr learned my addr 172.21.2.102:0/1554967887 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:40:07.499 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:07.358 7fded707d700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fded80d1870 0x7fded81b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:07.499 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:07.358 7fded787e700 1 -- 172.21.2.102:0/1554967887 >> v2:172.21.2.113:3300/0 conn(0x7fded80d1870 msgr2=0x7fded81b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:07.499 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:07.358 7fded787e700 1 --2- 172.21.2.102:0/1554967887 >> v2:172.21.2.113:3300/0 conn(0x7fded80d1870 0x7fded81b03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:07.500 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:07.358 7fded787e700 1 -- 172.21.2.102:0/1554967887 >> v1:172.21.2.113:6789/0 conn(0x7fded81b0a10 legacy=0x7fded80cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:07.500 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:07.358 7fded787e700 1 -- 172.21.2.102:0/1554967887 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fded80d1e90 con 0x7fded80f0650 2021-02-10T11:40:07.500 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:07.358 7fded787e700 1 --2- 172.21.2.102:0/1554967887 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fded80f0650 0x7fded80cfb20 secure :-1 s=READY pgs=3323 cs=0 l=1 rev1=1 rx=0x7fdec8006b80 tx=0x7fdec8009b90).ready entity=mon.0 client_cookie=2a8b096af20328de server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:07.500 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:07.358 7fded607b700 1 -- 172.21.2.102:0/1554967887 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdec80022e0 con 0x7fded80f0650 2021-02-10T11:40:07.500 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:07.358 7fded607b700 1 -- 172.21.2.102:0/1554967887 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdec8002620 con 0x7fded80f0650 2021-02-10T11:40:07.500 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:07.358 7fded607b700 1 -- 172.21.2.102:0/1554967887 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdec8002b00 con 0x7fded80f0650 2021-02-10T11:40:07.500 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:07.359 7fdedd2df700 1 -- 172.21.2.102:0/1554967887 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fded80f0650 msgr2=0x7fded80cfb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:07.500 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:07.359 7fdedd2df700 1 --2- 172.21.2.102:0/1554967887 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fded80f0650 0x7fded80cfb20 secure :-1 s=READY pgs=3323 cs=0 l=1 rev1=1 rx=0x7fdec8006b80 tx=0x7fdec8009b90).stop 2021-02-10T11:40:07.500 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:07.359 7fdedd2df700 1 -- 172.21.2.102:0/1554967887 shutdown_connections 2021-02-10T11:40:07.500 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:07.359 7fdedd2df700 1 --2- 172.21.2.102:0/1554967887 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fded80f0650 0x7fded80cfb20 unknown :-1 s=CLOSED pgs=3323 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:07.500 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:07.359 7fdedd2df700 1 --2- 172.21.2.102:0/1554967887 >> v2:172.21.2.113:3300/0 conn(0x7fded80d1870 0x7fded81b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:07.500 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:07.359 7fdedd2df700 1 -- 172.21.2.102:0/1554967887 >> 172.21.2.102:0/1554967887 conn(0x7fded80ce450 msgr2=0x7fded80d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:07.500 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:07.359 7fdedd2df700 1 -- 172.21.2.102:0/1554967887 shutdown_connections 2021-02-10T11:40:07.501 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:07.359 7fdedd2df700 1 -- 172.21.2.102:0/1554967887 wait complete. 2021-02-10T11:40:07.502 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:07.498 7fdedd2df700 1 Processor -- start 2021-02-10T11:40:07.502 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:07.498 7fdedd2df700 1 -- start start 2021-02-10T11:40:07.502 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:07.498 7fdedd2df700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fded80c0bb0 0x7fded809f270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:07.502 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:07.498 7fdedd2df700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fded809c370 0x7fded809c770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:07.502 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:07.498 7fdedd2df700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fded809cd90 0x7fded80c17d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:07.503 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:07.498 7fdedd2df700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fded80c1250 con 0x7fded809cd90 2021-02-10T11:40:07.503 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:07.498 7fdedd2df700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fded80c0fb0 con 0x7fded80c0bb0 2021-02-10T11:40:07.503 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:07.498 7fdedd2df700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fded809d190 con 0x7fded809c370 2021-02-10T11:40:07.503 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:07.499 7fded787e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fded809cd90 0x7fded80c17d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:07.503 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:07.499 7fded787e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fded809cd90 0x7fded80c17d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:40952/0 (socket says 172.21.2.102:40952) 2021-02-10T11:40:07.503 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:07.499 7fded787e700 1 -- 172.21.2.102:0/1034095870 learned_addr learned my addr 172.21.2.102:0/1034095870 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:40:07.503 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:07.499 7fded787e700 1 -- 172.21.2.102:0/1034095870 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fded809c370 msgr2=0x7fded809c770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:07.503 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:07.499 7fded787e700 1 --2- 172.21.2.102:0/1034095870 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fded809c370 0x7fded809c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:07.503 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:07.499 7fded787e700 1 -- 172.21.2.102:0/1034095870 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fded80c0bb0 msgr2=0x7fded809f270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:07.503 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:07.499 7fded787e700 1 --2- 172.21.2.102:0/1034095870 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fded80c0bb0 0x7fded809f270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:07.503 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:07.499 7fded787e700 1 -- 172.21.2.102:0/1034095870 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fded809d3b0 con 0x7fded809cd90 2021-02-10T11:40:07.503 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:07.499 7fded787e700 1 --2- 172.21.2.102:0/1034095870 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fded809cd90 0x7fded80c17d0 secure :-1 s=READY pgs=3324 cs=0 l=1 rev1=1 rx=0x7fdec8009640 tx=0x7fdec800b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:07.503 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:07.500 7fdebffff700 1 -- 172.21.2.102:0/1034095870 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdec8009100 con 0x7fded809cd90 2021-02-10T11:40:07.504 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:07.500 7fdebffff700 1 -- 172.21.2.102:0/1034095870 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdec800bde0 con 0x7fded809cd90 2021-02-10T11:40:07.504 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:07.500 7fdebffff700 1 -- 172.21.2.102:0/1034095870 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdec8006470 con 0x7fded809cd90 2021-02-10T11:40:07.504 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:07.500 7fdedd2df700 1 -- 172.21.2.102:0/1034095870 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fded809d3b0 con 0x7fded809cd90 2021-02-10T11:40:07.504 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:07.500 7fdedd2df700 1 -- 172.21.2.102:0/1034095870 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fded80c2110 con 0x7fded809cd90 2021-02-10T11:40:07.504 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:07.500 7fdebffff700 1 -- 172.21.2.102:0/1034095870 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fdec8006030 con 0x7fded809cd90 2021-02-10T11:40:07.504 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:07.500 7fdebffff700 1 --2- 172.21.2.102:0/1034095870 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fdeb80186d0 0x7fdeb801abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:07.504 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:07.500 7fdebffff700 1 -- 172.21.2.102:0/1034095870 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(189..189 src has 1..189) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fdec80327a0 con 0x7fded809cd90 2021-02-10T11:40:07.504 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:07.501 7fded707d700 1 --2- 172.21.2.102:0/1034095870 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fdeb80186d0 0x7fdeb801abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:07.504 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:07.501 7fded707d700 1 --2- 172.21.2.102:0/1034095870 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fdeb80186d0 0x7fdeb801abd0 crc :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:07.509 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:07.508 7fdedd2df700 1 -- 172.21.2.102:0/1034095870 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fded80ebc00 con 0x7fded809cd90 2021-02-10T11:40:07.512 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:07.511 7fdebffff700 1 -- 172.21.2.102:0/1034095870 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fdec8009100 con 0x7fded809cd90 2021-02-10T11:40:07.645 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:07.644 7fdedd2df700 1 -- 172.21.2.102:0/1034095870 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestVolumes.test_subvolume_authorized_list"]} v 0) v1 -- 0x7fded80f0b10 con 0x7fded809cd90 2021-02-10T11:40:08.208 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:08.207 7fdebffff700 1 -- 172.21.2.102:0/1034095870 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestVolumes.test_subvolume_authorized_list"]}]=0 v915) v1 ==== 147+0+0 (secure 0 0 0) 0x7fdec8006030 con 0x7fded809cd90 2021-02-10T11:40:08.214 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:08.213 7fdedd2df700 1 -- 172.21.2.102:0/1034095870 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fdeb80186d0 msgr2=0x7fdeb801abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:08.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:08.213 7fdedd2df700 1 --2- 172.21.2.102:0/1034095870 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fdeb80186d0 0x7fdeb801abd0 crc :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:08.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:08.213 7fdedd2df700 1 -- 172.21.2.102:0/1034095870 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fded809cd90 msgr2=0x7fded80c17d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:08.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:08.213 7fdedd2df700 1 --2- 172.21.2.102:0/1034095870 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fded809cd90 0x7fded80c17d0 secure :-1 s=READY pgs=3324 cs=0 l=1 rev1=1 rx=0x7fdec8009640 tx=0x7fdec800b960).stop 2021-02-10T11:40:08.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:08.213 7fdedd2df700 1 -- 172.21.2.102:0/1034095870 shutdown_connections 2021-02-10T11:40:08.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:08.213 7fdedd2df700 1 --2- 172.21.2.102:0/1034095870 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fded809cd90 0x7fded80c17d0 unknown :-1 s=CLOSED pgs=3324 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:08.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:08.213 7fdedd2df700 1 --2- 172.21.2.102:0/1034095870 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fded809c370 0x7fded809c770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:08.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:08.213 7fdedd2df700 1 --2- 172.21.2.102:0/1034095870 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fdeb80186d0 0x7fdeb801abd0 unknown :-1 s=CLOSED pgs=1449 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:08.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:08.213 7fdedd2df700 1 --2- 172.21.2.102:0/1034095870 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fded80c0bb0 0x7fded809f270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:08.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:08.213 7fdedd2df700 1 -- 172.21.2.102:0/1034095870 >> 172.21.2.102:0/1034095870 conn(0x7fded8097ac0 msgr2=0x7fded80c2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:08.216 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:08.214 7fdedd2df700 1 -- 172.21.2.102:0/1034095870 shutdown_connections 2021-02-10T11:40:08.216 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:08.214 7fdedd2df700 1 -- 172.21.2.102:0/1034095870 wait complete. 2021-02-10T11:40:08.223 INFO:tasks.cephfs_test_runner:test_subvolume_authorized_list (tasks.cephfs.test_volumes.TestVolumes) ... ok 2021-02-10T11:40:08.223 INFO:tasks.cephfs_test_runner:Starting test: test_subvolume_clone_in_progress_getpath (tasks.cephfs.test_volumes.TestVolumes) 2021-02-10T11:40:08.224 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestVolumes.test_subvolume_clone_in_progress_getpath' 2021-02-10T11:40:08.653 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:08.635 7fd01d0dd700 1 Processor -- start 2021-02-10T11:40:08.653 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:08.635 7fd01d0dd700 1 -- start start 2021-02-10T11:40:08.653 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:08.636 7fd01d0dd700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fd0180d1c70 0x7fd0180f20a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:08.653 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:08.636 7fd01d0dd700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0180f0b10 con 0x7fd0180d1870 2021-02-10T11:40:08.654 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:08.636 7fd01d0dd700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0181b0be0 con 0x7fd0180d0340 2021-02-10T11:40:08.654 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:08.636 7fd01d0dd700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0180e5ed0 con 0x7fd0180d1c70 2021-02-10T11:40:08.654 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:08.636 7fd01759e700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fd0180d1c70 0x7fd0180f20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:08.654 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:08.636 7fd016d9d700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fd0180d1870 0x7fd0181b03f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:33180/0 (socket says 172.21.2.102:33180) 2021-02-10T11:40:08.654 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:08.636 7fd016d9d700 1 -- 172.21.2.102:0/1501585056 learned_addr learned my addr 172.21.2.102:0/1501585056 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:40:08.654 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:08.636 7fd01759e700 1 -- 172.21.2.102:0/1501585056 >> v1:172.21.2.113:6789/0 conn(0x7fd0180d0340 legacy=0x7fd0180ef640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:08.654 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:08.636 7fd01759e700 1 -- 172.21.2.102:0/1501585056 >> v1:172.21.2.102:6789/0 conn(0x7fd0180d1870 legacy=0x7fd0181b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:08.654 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:08.636 7fd015d9b700 1 -- 172.21.2.102:0/1501585056 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3221127456 0 0) 0x7fd000000ce0 con 0x7fd0180d1870 2021-02-10T11:40:08.654 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:08.637 7fd01759e700 1 -- 172.21.2.102:0/1501585056 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0180efe30 con 0x7fd0180d1c70 2021-02-10T11:40:08.654 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:08.637 7fd01759e700 1 --2- 172.21.2.102:0/1501585056 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd0180d1c70 0x7fd0180f20a0 secure :-1 s=READY pgs=3326 cs=0 l=1 rev1=1 rx=0x7fd010006e30 tx=0x7fd010009de0).ready entity=mon.0 client_cookie=a7ebe06d3e095ff1 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:08.654 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:08.637 7fd015d9b700 1 -- 172.21.2.102:0/1501585056 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd01000b300 con 0x7fd0180d1c70 2021-02-10T11:40:08.654 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:08.637 7fd015d9b700 1 -- 172.21.2.102:0/1501585056 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd01000b580 con 0x7fd0180d1c70 2021-02-10T11:40:08.655 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:08.637 7fd015d9b700 1 -- 172.21.2.102:0/1501585056 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd01000baa0 con 0x7fd0180d1c70 2021-02-10T11:40:08.655 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:08.637 7fd01d0dd700 1 -- 172.21.2.102:0/1501585056 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd0180d1c70 msgr2=0x7fd0180f20a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:08.655 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:08.637 7fd01d0dd700 1 --2- 172.21.2.102:0/1501585056 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd0180d1c70 0x7fd0180f20a0 secure :-1 s=READY pgs=3326 cs=0 l=1 rev1=1 rx=0x7fd010006e30 tx=0x7fd010009de0).stop 2021-02-10T11:40:08.655 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:08.638 7fd01d0dd700 1 -- 172.21.2.102:0/1501585056 shutdown_connections 2021-02-10T11:40:08.655 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:08.638 7fd01d0dd700 1 --2- 172.21.2.102:0/1501585056 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd0180d1c70 0x7fd0180f20a0 unknown :-1 s=CLOSED pgs=3326 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:08.655 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:08.638 7fd01d0dd700 1 -- 172.21.2.102:0/1501585056 >> 172.21.2.102:0/1501585056 conn(0x7fd0180ce450 msgr2=0x7fd0180d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:08.655 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:08.638 7fd01d0dd700 1 -- 172.21.2.102:0/1501585056 shutdown_connections 2021-02-10T11:40:08.655 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:08.638 7fd01d0dd700 1 -- 172.21.2.102:0/1501585056 wait complete. 2021-02-10T11:40:08.656 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:08.652 7fd01d0dd700 1 Processor -- start 2021-02-10T11:40:08.657 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:08.652 7fd01d0dd700 1 -- start start 2021-02-10T11:40:08.657 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:08.653 7fd01d0dd700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd01819a0e0 0x7fd01809cb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:08.657 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:08.653 7fd01d0dd700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd01818aee0 0x7fd01818b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:08.657 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:08.653 7fd01d0dd700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd01818b900 0x7fd018187e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:08.657 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:08.653 7fd01d0dd700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd01809d1a0 con 0x7fd01818aee0 2021-02-10T11:40:08.657 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:08.653 7fd01d0dd700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd01819a4e0 con 0x7fd01819a0e0 2021-02-10T11:40:08.657 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:08.653 7fd01d0dd700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0181ab160 con 0x7fd01818b900 2021-02-10T11:40:08.657 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:08.653 7fd01659c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd01818aee0 0x7fd01818b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:08.657 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:08.653 7fd01659c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd01818aee0 0x7fd01818b2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:40964/0 (socket says 172.21.2.102:40964) 2021-02-10T11:40:08.657 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:08.653 7fd01659c700 1 -- 172.21.2.102:0/615300648 learned_addr learned my addr 172.21.2.102:0/615300648 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:40:08.657 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:08.653 7fd01659c700 1 -- 172.21.2.102:0/615300648 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd01818b900 msgr2=0x7fd018187e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:08.657 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:08.653 7fd01659c700 1 --2- 172.21.2.102:0/615300648 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd01818b900 0x7fd018187e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:08.657 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:08.653 7fd01659c700 1 -- 172.21.2.102:0/615300648 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd01819a0e0 msgr2=0x7fd01809cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:08.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:08.653 7fd01659c700 1 --2- 172.21.2.102:0/615300648 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd01819a0e0 0x7fd01809cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:08.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:08.653 7fd01659c700 1 -- 172.21.2.102:0/615300648 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd01818bd00 con 0x7fd01818aee0 2021-02-10T11:40:08.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:08.653 7fd01659c700 1 --2- 172.21.2.102:0/615300648 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd01818aee0 0x7fd01818b2e0 secure :-1 s=READY pgs=3327 cs=0 l=1 rev1=1 rx=0x7fd00800b530 tx=0x7fd00800b8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:08.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:08.654 7fd00f7fe700 1 -- 172.21.2.102:0/615300648 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd00800bcc0 con 0x7fd01818aee0 2021-02-10T11:40:08.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:08.654 7fd00f7fe700 1 -- 172.21.2.102:0/615300648 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd008000e20 con 0x7fd01818aee0 2021-02-10T11:40:08.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:08.654 7fd00f7fe700 1 -- 172.21.2.102:0/615300648 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd008006210 con 0x7fd01818aee0 2021-02-10T11:40:08.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:08.654 7fd01d0dd700 1 -- 172.21.2.102:0/615300648 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd01818bd00 con 0x7fd01818aee0 2021-02-10T11:40:08.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:08.654 7fd01d0dd700 1 -- 172.21.2.102:0/615300648 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd018188760 con 0x7fd01818aee0 2021-02-10T11:40:08.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:08.654 7fd00f7fe700 1 -- 172.21.2.102:0/615300648 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd008006020 con 0x7fd01818aee0 2021-02-10T11:40:08.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:08.655 7fd00f7fe700 1 --2- 172.21.2.102:0/615300648 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcffc0186c0 0x7fcffc01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:08.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:08.655 7fd00f7fe700 1 -- 172.21.2.102:0/615300648 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(189..189 src has 1..189) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fd0080325a0 con 0x7fd01818aee0 2021-02-10T11:40:08.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:08.655 7fd016d9d700 1 --2- 172.21.2.102:0/615300648 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcffc0186c0 0x7fcffc01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:08.658 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:08.655 7fd016d9d700 1 --2- 172.21.2.102:0/615300648 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcffc0186c0 0x7fcffc01abc0 crc :-1 s=READY pgs=1450 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:08.687 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:08.686 7fd01d0dd700 1 -- 172.21.2.102:0/615300648 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd018188ff0 con 0x7fd01818aee0 2021-02-10T11:40:08.689 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:08.688 7fd00f7fe700 1 -- 172.21.2.102:0/615300648 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd008000940 con 0x7fd01818aee0 2021-02-10T11:40:08.823 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:08.821 7fd01d0dd700 1 -- 172.21.2.102:0/615300648 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestVolumes.test_subvolume_clone_in_progress_getpath"]} v 0) v1 -- 0x7fd018188ff0 con 0x7fd01818aee0 2021-02-10T11:40:09.211 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:09.210 7fd00f7fe700 1 -- 172.21.2.102:0/615300648 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestVolumes.test_subvolume_clone_in_progress_getpath"]}]=0 v916) v1 ==== 160+0+0 (secure 0 0 0) 0x7fd00801f550 con 0x7fd01818aee0 2021-02-10T11:40:09.241 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:09.240 7fd01d0dd700 1 -- 172.21.2.102:0/615300648 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcffc0186c0 msgr2=0x7fcffc01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:09.241 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:09.240 7fd01d0dd700 1 --2- 172.21.2.102:0/615300648 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcffc0186c0 0x7fcffc01abc0 crc :-1 s=READY pgs=1450 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:09.242 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:09.240 7fd01d0dd700 1 -- 172.21.2.102:0/615300648 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd01818aee0 msgr2=0x7fd01818b2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:09.242 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:09.240 7fd01d0dd700 1 --2- 172.21.2.102:0/615300648 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd01818aee0 0x7fd01818b2e0 secure :-1 s=READY pgs=3327 cs=0 l=1 rev1=1 rx=0x7fd00800b530 tx=0x7fd00800b8d0).stop 2021-02-10T11:40:09.242 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:09.240 7fd01d0dd700 1 -- 172.21.2.102:0/615300648 shutdown_connections 2021-02-10T11:40:09.242 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:09.240 7fd01d0dd700 1 --2- 172.21.2.102:0/615300648 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd01818b900 0x7fd018187e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:09.242 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:09.240 7fd01d0dd700 1 --2- 172.21.2.102:0/615300648 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd01818aee0 0x7fd01818b2e0 unknown :-1 s=CLOSED pgs=3327 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:09.242 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:09.240 7fd01d0dd700 1 --2- 172.21.2.102:0/615300648 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcffc0186c0 0x7fcffc01abc0 unknown :-1 s=CLOSED pgs=1450 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:09.242 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:09.240 7fd01d0dd700 1 --2- 172.21.2.102:0/615300648 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd01819a0e0 0x7fd01809cb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:09.242 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:09.240 7fd01d0dd700 1 -- 172.21.2.102:0/615300648 >> 172.21.2.102:0/615300648 conn(0x7fd018093640 msgr2=0x7fd01818eb00 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:09.242 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:09.240 7fd01d0dd700 1 -- 172.21.2.102:0/615300648 shutdown_connections 2021-02-10T11:40:09.242 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:09.240 7fd01d0dd700 1 -- 172.21.2.102:0/615300648 wait complete. 2021-02-10T11:40:09.249 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config set mon mon_allow_pool_delete True 2021-02-10T11:40:09.684 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:09.665 7fde521a8700 1 Processor -- start 2021-02-10T11:40:09.684 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:09.665 7fde521a8700 1 -- start start 2021-02-10T11:40:09.684 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:09.666 7fde521a8700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fde4c0d1860 0x7fde4c1b04b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:09.684 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:09.666 7fde521a8700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fde4c1b0ad0 0x7fde4c108930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:09.684 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:09.666 7fde521a8700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fde4c0d1c60 0x7fde4c108f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:09.684 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:09.666 7fde521a8700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde4c0e5ee0 con 0x7fde4c1b0ad0 2021-02-10T11:40:09.684 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:09.666 7fde521a8700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde4c109a50 con 0x7fde4c0d1c60 2021-02-10T11:40:09.684 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:09.666 7fde521a8700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde4c088070 con 0x7fde4c0d1860 2021-02-10T11:40:09.684 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:09.666 7fde4affd700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fde4c1b0ad0 0x7fde4c108930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:09.685 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:09.666 7fde4affd700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fde4c1b0ad0 0x7fde4c108930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:40970/0 (socket says 172.21.2.102:40970) 2021-02-10T11:40:09.685 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:09.666 7fde4affd700 1 -- 172.21.2.102:0/2518338960 learned_addr learned my addr 172.21.2.102:0/2518338960 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:40:09.685 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:09.667 7fde4affd700 1 -- 172.21.2.102:0/2518338960 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fde4c0d1860 msgr2=0x7fde4c1b04b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:09.685 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:09.667 7fde4affd700 1 --2- 172.21.2.102:0/2518338960 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fde4c0d1860 0x7fde4c1b04b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:09.685 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:09.667 7fde4affd700 1 -- 172.21.2.102:0/2518338960 >> v2:172.21.2.113:3300/0 conn(0x7fde4c0d1c60 msgr2=0x7fde4c108f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:09.685 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:09.667 7fde4bfff700 1 --2- 172.21.2.102:0/2518338960 >> v2:172.21.2.113:3300/0 conn(0x7fde4c0d1c60 0x7fde4c108f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:09.685 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:09.667 7fde4affd700 1 --2- 172.21.2.102:0/2518338960 >> v2:172.21.2.113:3300/0 conn(0x7fde4c0d1c60 0x7fde4c108f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:09.685 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:09.667 7fde4affd700 1 -- 172.21.2.102:0/2518338960 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde4c109570 con 0x7fde4c1b0ad0 2021-02-10T11:40:09.685 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:09.667 7fde4bfff700 1 --2- 172.21.2.102:0/2518338960 >> v2:172.21.2.113:3300/0 conn(0x7fde4c0d1c60 0x7fde4c108f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:40:09.685 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:09.667 7fde4affd700 1 --2- 172.21.2.102:0/2518338960 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fde4c1b0ad0 0x7fde4c108930 secure :-1 s=READY pgs=3328 cs=0 l=1 rev1=1 rx=0x7fde40006700 tx=0x7fde40009b00).ready entity=mon.0 client_cookie=46a9944cb3e952b1 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:09.685 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:09.667 7fde4a7fc700 1 -- 172.21.2.102:0/2518338960 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fde4000c0c0 con 0x7fde4c1b0ad0 2021-02-10T11:40:09.685 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:09.667 7fde4a7fc700 1 -- 172.21.2.102:0/2518338960 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde4000c340 con 0x7fde4c1b0ad0 2021-02-10T11:40:09.685 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:09.667 7fde4a7fc700 1 -- 172.21.2.102:0/2518338960 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fde4000c820 con 0x7fde4c1b0ad0 2021-02-10T11:40:09.686 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:09.667 7fde521a8700 1 -- 172.21.2.102:0/2518338960 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fde4c1b0ad0 msgr2=0x7fde4c108930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:09.686 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:09.668 7fde521a8700 1 --2- 172.21.2.102:0/2518338960 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fde4c1b0ad0 0x7fde4c108930 secure :-1 s=READY pgs=3328 cs=0 l=1 rev1=1 rx=0x7fde40006700 tx=0x7fde40009b00).stop 2021-02-10T11:40:09.686 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:09.668 7fde521a8700 1 -- 172.21.2.102:0/2518338960 shutdown_connections 2021-02-10T11:40:09.686 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:09.668 7fde521a8700 1 --2- 172.21.2.102:0/2518338960 >> v2:172.21.2.113:3300/0 conn(0x7fde4c0d1c60 0x7fde4c108f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:09.686 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:09.668 7fde521a8700 1 --2- 172.21.2.102:0/2518338960 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fde4c1b0ad0 0x7fde4c108930 unknown :-1 s=CLOSED pgs=3328 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:09.686 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:09.668 7fde521a8700 1 --2- 172.21.2.102:0/2518338960 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fde4c0d1860 0x7fde4c1b04b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:09.686 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:09.668 7fde521a8700 1 -- 172.21.2.102:0/2518338960 >> 172.21.2.102:0/2518338960 conn(0x7fde4c0ce410 msgr2=0x7fde4c0d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:09.686 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:09.668 7fde521a8700 1 -- 172.21.2.102:0/2518338960 shutdown_connections 2021-02-10T11:40:09.686 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:09.668 7fde521a8700 1 -- 172.21.2.102:0/2518338960 wait complete. 2021-02-10T11:40:09.687 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:09.683 7fde521a8700 1 Processor -- start 2021-02-10T11:40:09.688 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:09.683 7fde521a8700 1 -- start start 2021-02-10T11:40:09.688 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:09.683 7fde521a8700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fde4c1a7b50 0x7fde4c1a7f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:09.688 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:09.683 7fde521a8700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fde4c0ba230 0x7fde4c0ba630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:09.688 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:09.683 7fde521a8700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fde4c0963c0 0x7fde4c0967c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:09.688 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:09.683 7fde521a8700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde4c097850 con 0x7fde4c0963c0 2021-02-10T11:40:09.688 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:09.683 7fde521a8700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde4c096de0 con 0x7fde4c1a7b50 2021-02-10T11:40:09.688 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:09.683 7fde521a8700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde4c096fc0 con 0x7fde4c0ba230 2021-02-10T11:40:09.688 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:09.684 7fde4bfff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fde4c0963c0 0x7fde4c0967c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:09.688 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:09.684 7fde4bfff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fde4c0963c0 0x7fde4c0967c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:40980/0 (socket says 172.21.2.102:40980) 2021-02-10T11:40:09.688 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:09.684 7fde4bfff700 1 -- 172.21.2.102:0/2099954775 learned_addr learned my addr 172.21.2.102:0/2099954775 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:40:09.688 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:09.684 7fde4b7fe700 1 --2- 172.21.2.102:0/2099954775 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fde4c1a7b50 0x7fde4c1a7f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:09.688 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:09.684 7fde4affd700 1 --2- 172.21.2.102:0/2099954775 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fde4c0ba230 0x7fde4c0ba630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:09.688 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:09.684 7fde4bfff700 1 -- 172.21.2.102:0/2099954775 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fde4c0ba230 msgr2=0x7fde4c0ba630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:09.689 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:09.684 7fde4bfff700 1 --2- 172.21.2.102:0/2099954775 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fde4c0ba230 0x7fde4c0ba630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:09.689 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:09.684 7fde4bfff700 1 -- 172.21.2.102:0/2099954775 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fde4c1a7b50 msgr2=0x7fde4c1a7f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:09.689 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:09.684 7fde4bfff700 1 --2- 172.21.2.102:0/2099954775 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fde4c1a7b50 0x7fde4c1a7f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:09.689 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:09.684 7fde4bfff700 1 -- 172.21.2.102:0/2099954775 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde4c0971e0 con 0x7fde4c0963c0 2021-02-10T11:40:09.689 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:09.684 7fde4bfff700 1 --2- 172.21.2.102:0/2099954775 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fde4c0963c0 0x7fde4c0967c0 secure :-1 s=READY pgs=3329 cs=0 l=1 rev1=1 rx=0x7fde440099a0 tx=0x7fde4400bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:09.689 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:09.684 7fde33fff700 1 -- 172.21.2.102:0/2099954775 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fde44006030 con 0x7fde4c0963c0 2021-02-10T11:40:09.689 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:09.684 7fde33fff700 1 -- 172.21.2.102:0/2099954775 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde44006230 con 0x7fde4c0963c0 2021-02-10T11:40:09.689 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:09.684 7fde33fff700 1 -- 172.21.2.102:0/2099954775 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fde440056b0 con 0x7fde4c0963c0 2021-02-10T11:40:09.689 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:09.684 7fde521a8700 1 -- 172.21.2.102:0/2099954775 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde4c0971e0 con 0x7fde4c0963c0 2021-02-10T11:40:09.689 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:09.684 7fde521a8700 1 -- 172.21.2.102:0/2099954775 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde4c097dd0 con 0x7fde4c0963c0 2021-02-10T11:40:09.689 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:09.685 7fde33fff700 1 -- 172.21.2.102:0/2099954775 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fde44006950 con 0x7fde4c0963c0 2021-02-10T11:40:09.689 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:09.685 7fde33fff700 1 --2- 172.21.2.102:0/2099954775 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fde2c0186c0 0x7fde2c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:09.689 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:09.685 7fde33fff700 1 -- 172.21.2.102:0/2099954775 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(189..189 src has 1..189) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fde44032740 con 0x7fde4c0963c0 2021-02-10T11:40:09.689 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:09.685 7fde4b7fe700 1 --2- 172.21.2.102:0/2099954775 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fde2c0186c0 0x7fde2c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:09.690 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:09.685 7fde4b7fe700 1 --2- 172.21.2.102:0/2099954775 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fde2c0186c0 0x7fde2c01abc0 crc :-1 s=READY pgs=1451 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:09.717 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:09.716 7fde521a8700 1 -- 172.21.2.102:0/2099954775 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde4c082920 con 0x7fde4c0963c0 2021-02-10T11:40:09.719 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:09.718 7fde33fff700 1 -- 172.21.2.102:0/2099954775 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fde4400be30 con 0x7fde4c0963c0 2021-02-10T11:40:09.869 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:09.868 7fde521a8700 1 -- 172.21.2.102:0/2099954775 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fde4c109a50 con 0x7fde4c0963c0 2021-02-10T11:40:09.873 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:09.872 7fde33fff700 1 -- 172.21.2.102:0/2099954775 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v48)=0 v48) v1 ==== 122+0+0 (secure 0 0 0) 0x7fde44006950 con 0x7fde4c0963c0 2021-02-10T11:40:09.878 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:09.877 7fde521a8700 1 -- 172.21.2.102:0/2099954775 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fde2c0186c0 msgr2=0x7fde2c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:09.878 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:09.877 7fde521a8700 1 --2- 172.21.2.102:0/2099954775 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fde2c0186c0 0x7fde2c01abc0 crc :-1 s=READY pgs=1451 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:09.878 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:09.877 7fde521a8700 1 -- 172.21.2.102:0/2099954775 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fde4c0963c0 msgr2=0x7fde4c0967c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:09.878 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:09.877 7fde521a8700 1 --2- 172.21.2.102:0/2099954775 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fde4c0963c0 0x7fde4c0967c0 secure :-1 s=READY pgs=3329 cs=0 l=1 rev1=1 rx=0x7fde440099a0 tx=0x7fde4400bcf0).stop 2021-02-10T11:40:09.879 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:09.877 7fde521a8700 1 -- 172.21.2.102:0/2099954775 shutdown_connections 2021-02-10T11:40:09.879 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:09.877 7fde521a8700 1 --2- 172.21.2.102:0/2099954775 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fde4c0963c0 0x7fde4c0967c0 unknown :-1 s=CLOSED pgs=3329 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:09.879 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:09.877 7fde521a8700 1 --2- 172.21.2.102:0/2099954775 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fde4c0ba230 0x7fde4c0ba630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:09.879 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:09.877 7fde521a8700 1 --2- 172.21.2.102:0/2099954775 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fde2c0186c0 0x7fde2c01abc0 unknown :-1 s=CLOSED pgs=1451 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:09.879 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:09.877 7fde521a8700 1 --2- 172.21.2.102:0/2099954775 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fde4c1a7b50 0x7fde4c1a7f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:09.879 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:09.877 7fde521a8700 1 -- 172.21.2.102:0/2099954775 >> 172.21.2.102:0/2099954775 conn(0x7fde4c095eb0 msgr2=0x7fde4c0a6ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:09.879 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:09.877 7fde521a8700 1 -- 172.21.2.102:0/2099954775 shutdown_connections 2021-02-10T11:40:09.879 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:09.877 7fde521a8700 1 -- 172.21.2.102:0/2099954775 wait complete. 2021-02-10T11:40:09.887 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2021-02-10T11:40:09.887 DEBUG:teuthology.orchestra.run.gibba002:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:40:10.233 DEBUG:teuthology.orchestra.run.gibba013:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:40:10.578 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2021-02-10T11:40:10.578 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:40:10.579 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:40:10.579 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:40:10.915 INFO:teuthology.orchestra.run.gibba002.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.0’: No such file or directory 2021-02-10T11:40:10.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:40:10.916 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.0 2021-02-10T11:40:10.916 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:40:10.916 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:40:10.916 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:40:11.251 INFO:teuthology.orchestra.run.gibba002.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.1’: No such file or directory 2021-02-10T11:40:11.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:40:11.252 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2021-02-10T11:40:11.252 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:40:11.252 DEBUG:teuthology.orchestra.run.gibba013:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:40:11.252 DEBUG:teuthology.orchestra.run.gibba013:> ' -- /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:40:11.587 INFO:teuthology.orchestra.run.gibba013.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.2’: No such file or directory 2021-02-10T11:40:11.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:40:11.587 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2021-02-10T11:40:11.587 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:40:11.587 DEBUG:teuthology.orchestra.run.gibba013:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:40:11.587 DEBUG:teuthology.orchestra.run.gibba013:> ' -- /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:40:11.921 INFO:teuthology.orchestra.run.gibba013.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.3’: No such file or directory 2021-02-10T11:40:11.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:40:11.921 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.3 2021-02-10T11:40:11.921 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json-pretty 2021-02-10T11:40:12.350 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:12.333 7f09be1ad700 1 Processor -- start 2021-02-10T11:40:12.350 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:12.333 7f09be1ad700 1 -- start start 2021-02-10T11:40:12.351 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:12.333 7f09be1ad700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f09b81b0870 0x7f09b80f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:12.351 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:12.333 7f09be1ad700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f09b80efe00 0x7f09b80f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:12.351 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:12.333 7f09be1ad700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09b81b0c70 con 0x7f09b81b0470 2021-02-10T11:40:12.351 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:12.333 7f09be1ad700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09b80f0b50 con 0x7f09b80efe00 2021-02-10T11:40:12.351 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:12.333 7f09be1ad700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09b80f0820 con 0x7f09b81b0870 2021-02-10T11:40:12.351 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:12.333 7f09b7fff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f09b80efe00 0x7f09b80f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:12.351 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:12.333 7f09b7fff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f09b80efe00 0x7f09b80f0200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:40988/0 (socket says 172.21.2.102:40988) 2021-02-10T11:40:12.351 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:12.333 7f09b77fe700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f09b81b0470 0x7f09b80ef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:33208/0 (socket says 172.21.2.102:33208) 2021-02-10T11:40:12.351 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:12.333 7f09b7fff700 1 -- 172.21.2.102:0/4159773832 learned_addr learned my addr 172.21.2.102:0/4159773832 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:40:12.351 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:12.334 7f09b6ffd700 1 --2- 172.21.2.102:0/4159773832 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f09b81b0870 0x7f09b80f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:12.352 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:12.334 7f09b7fff700 1 -- 172.21.2.102:0/4159773832 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f09b81b0870 msgr2=0x7f09b80f20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:12.352 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:12.334 7f09b7fff700 1 --2- 172.21.2.102:0/4159773832 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f09b81b0870 0x7f09b80f20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:12.352 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:12.334 7f09b7fff700 1 -- 172.21.2.102:0/4159773832 >> v1:172.21.2.102:6789/0 conn(0x7f09b81b0470 legacy=0x7f09b80ef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:12.352 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:12.334 7f09b7fff700 1 -- 172.21.2.102:0/4159773832 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09b80e52e0 con 0x7f09b80efe00 2021-02-10T11:40:12.352 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:12.334 7f09b7fff700 1 --2- 172.21.2.102:0/4159773832 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f09b80efe00 0x7f09b80f0200 secure :-1 s=READY pgs=3331 cs=0 l=1 rev1=1 rx=0x7f09b0006f00 tx=0x7f09b0009f10).ready entity=mon.0 client_cookie=9f787d93c11977e7 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:12.352 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:12.334 7f09b67fc700 1 -- 172.21.2.102:0/4159773832 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f09b000b3b0 con 0x7f09b80efe00 2021-02-10T11:40:12.352 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:12.334 7f09b67fc700 1 -- 172.21.2.102:0/4159773832 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09b000b630 con 0x7f09b80efe00 2021-02-10T11:40:12.352 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:12.334 7f09b67fc700 1 -- 172.21.2.102:0/4159773832 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f09b000bb50 con 0x7f09b80efe00 2021-02-10T11:40:12.352 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:12.334 7f09be1ad700 1 -- 172.21.2.102:0/4159773832 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f09b80efe00 msgr2=0x7f09b80f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:12.352 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:12.334 7f09be1ad700 1 --2- 172.21.2.102:0/4159773832 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f09b80efe00 0x7f09b80f0200 secure :-1 s=READY pgs=3331 cs=0 l=1 rev1=1 rx=0x7f09b0006f00 tx=0x7f09b0009f10).stop 2021-02-10T11:40:12.352 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:12.335 7f09be1ad700 1 -- 172.21.2.102:0/4159773832 shutdown_connections 2021-02-10T11:40:12.353 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:12.335 7f09be1ad700 1 --2- 172.21.2.102:0/4159773832 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f09b80efe00 0x7f09b80f0200 unknown :-1 s=CLOSED pgs=3331 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:12.353 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:12.335 7f09be1ad700 1 --2- 172.21.2.102:0/4159773832 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f09b81b0870 0x7f09b80f20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:12.353 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:12.335 7f09be1ad700 1 -- 172.21.2.102:0/4159773832 >> 172.21.2.102:0/4159773832 conn(0x7f09b80ce3e0 msgr2=0x7f09b80e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:12.353 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:12.335 7f09be1ad700 1 -- 172.21.2.102:0/4159773832 shutdown_connections 2021-02-10T11:40:12.353 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:12.335 7f09be1ad700 1 -- 172.21.2.102:0/4159773832 wait complete. 2021-02-10T11:40:12.354 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:12.349 7f09be1ad700 1 Processor -- start 2021-02-10T11:40:12.355 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:12.349 7f09be1ad700 1 -- start start 2021-02-10T11:40:12.355 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:12.350 7f09be1ad700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f09b80aed00 0x7f09b8096400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:12.355 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:12.350 7f09be1ad700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f09b818ef20 0x7f09b80955e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:12.355 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:12.350 7f09be1ad700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f09b8095c00 0x7f09b8093240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:12.355 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:12.350 7f09be1ad700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09b8109ad0 con 0x7f09b818ef20 2021-02-10T11:40:12.355 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:12.350 7f09be1ad700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09b8096a20 con 0x7f09b8095c00 2021-02-10T11:40:12.355 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:12.350 7f09be1ad700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09b8096000 con 0x7f09b80aed00 2021-02-10T11:40:12.355 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:12.350 7f09b6ffd700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f09b818ef20 0x7f09b80955e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:12.355 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:12.350 7f09b6ffd700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f09b818ef20 0x7f09b80955e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:40994/0 (socket says 172.21.2.102:40994) 2021-02-10T11:40:12.355 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:12.350 7f09b6ffd700 1 -- 172.21.2.102:0/664533238 learned_addr learned my addr 172.21.2.102:0/664533238 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:40:12.355 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:12.350 7f09b6ffd700 1 -- 172.21.2.102:0/664533238 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f09b80aed00 msgr2=0x7f09b8096400 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:40:12.355 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:12.350 7f09b6ffd700 1 --2- 172.21.2.102:0/664533238 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f09b80aed00 0x7f09b8096400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:12.356 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:12.350 7f09b6ffd700 1 -- 172.21.2.102:0/664533238 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f09b8095c00 msgr2=0x7f09b8093240 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:40:12.356 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:12.350 7f09b6ffd700 1 --2- 172.21.2.102:0/664533238 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f09b8095c00 0x7f09b8093240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:12.356 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:12.350 7f09b6ffd700 1 -- 172.21.2.102:0/664533238 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09b8093860 con 0x7f09b818ef20 2021-02-10T11:40:12.356 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:12.350 7f09b6ffd700 1 --2- 172.21.2.102:0/664533238 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f09b818ef20 0x7f09b80955e0 secure :-1 s=READY pgs=3332 cs=0 l=1 rev1=1 rx=0x7f09a8006f00 tx=0x7f09a800b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:12.356 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:12.351 7f09affff700 1 -- 172.21.2.102:0/664533238 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f09a800bd30 con 0x7f09b818ef20 2021-02-10T11:40:12.356 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:12.351 7f09be1ad700 1 -- 172.21.2.102:0/664533238 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09b8093860 con 0x7f09b818ef20 2021-02-10T11:40:12.356 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:12.351 7f09affff700 1 -- 172.21.2.102:0/664533238 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09a800bd30 con 0x7f09b818ef20 2021-02-10T11:40:12.356 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:12.351 7f09affff700 1 -- 172.21.2.102:0/664533238 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f09a8002030 con 0x7f09b818ef20 2021-02-10T11:40:12.356 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:12.351 7f09be1ad700 1 -- 172.21.2.102:0/664533238 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09b8093df0 con 0x7f09b818ef20 2021-02-10T11:40:12.356 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:12.351 7f09affff700 1 -- 172.21.2.102:0/664533238 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f09a8000c90 con 0x7f09b818ef20 2021-02-10T11:40:12.356 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:12.351 7f09affff700 1 --2- 172.21.2.102:0/664533238 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f099c0186c0 0x7f099c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:12.357 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:12.351 7f09affff700 1 -- 172.21.2.102:0/664533238 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(189..189 src has 1..189) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f09a8032420 con 0x7f09b818ef20 2021-02-10T11:40:12.357 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:12.351 7f09b77fe700 1 --2- 172.21.2.102:0/664533238 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f099c0186c0 0x7f099c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:12.357 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:12.352 7f09b77fe700 1 --2- 172.21.2.102:0/664533238 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f099c0186c0 0x7f099c01abc0 crc :-1 s=READY pgs=1452 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:12.384 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:12.383 7f09be1ad700 1 -- 172.21.2.102:0/664533238 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09b8105a20 con 0x7f09b818ef20 2021-02-10T11:40:12.386 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:12.385 7f09affff700 1 -- 172.21.2.102:0/664533238 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f09a8000940 con 0x7f09b818ef20 2021-02-10T11:40:12.511 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:12.510 7f09be1ad700 1 -- 172.21.2.102:0/664533238 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7f09b80f0b50 con 0x7f09b818ef20 2021-02-10T11:40:12.512 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:12.510 7f09affff700 1 -- 172.21.2.102:0/664533238 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v189) v1 ==== 81+0+30374 (secure 0 0 0) 0x7f09a8030d50 con 0x7f09b818ef20 2021-02-10T11:40:12.512 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:40:12.512 INFO:teuthology.orchestra.run.gibba002.stdout:{ 2021-02-10T11:40:12.513 INFO:teuthology.orchestra.run.gibba002.stdout: "epoch": 189, 2021-02-10T11:40:12.513 INFO:teuthology.orchestra.run.gibba002.stdout: "fsid": "68db30ea-dbce-4aef-a6df-f01c6fea0513", 2021-02-10T11:40:12.513 INFO:teuthology.orchestra.run.gibba002.stdout: "created": "2021-02-10 11:03:06.331557", 2021-02-10T11:40:12.513 INFO:teuthology.orchestra.run.gibba002.stdout: "modified": "2021-02-10 11:39:31.377604", 2021-02-10T11:40:12.513 INFO:teuthology.orchestra.run.gibba002.stdout: "last_up_change": "2021-02-10 11:03:32.690562", 2021-02-10T11:40:12.513 INFO:teuthology.orchestra.run.gibba002.stdout: "last_in_change": "2021-02-10 11:03:32.690562", 2021-02-10T11:40:12.513 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2021-02-10T11:40:12.513 INFO:teuthology.orchestra.run.gibba002.stdout: "flags_num": 5799936, 2021-02-10T11:40:12.513 INFO:teuthology.orchestra.run.gibba002.stdout: "flags_set": [ 2021-02-10T11:40:12.514 INFO:teuthology.orchestra.run.gibba002.stdout: "pglog_hardlimit", 2021-02-10T11:40:12.514 INFO:teuthology.orchestra.run.gibba002.stdout: "purged_snapdirs", 2021-02-10T11:40:12.514 INFO:teuthology.orchestra.run.gibba002.stdout: "recovery_deletes", 2021-02-10T11:40:12.514 INFO:teuthology.orchestra.run.gibba002.stdout: "sortbitwise" 2021-02-10T11:40:12.514 INFO:teuthology.orchestra.run.gibba002.stdout: ], 2021-02-10T11:40:12.514 INFO:teuthology.orchestra.run.gibba002.stdout: "crush_version": 4, 2021-02-10T11:40:12.514 INFO:teuthology.orchestra.run.gibba002.stdout: "full_ratio": 0.94999998807907104, 2021-02-10T11:40:12.514 INFO:teuthology.orchestra.run.gibba002.stdout: "backfillfull_ratio": 0.89999997615814209, 2021-02-10T11:40:12.514 INFO:teuthology.orchestra.run.gibba002.stdout: "nearfull_ratio": 0.85000002384185791, 2021-02-10T11:40:12.514 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_snapshot": "", 2021-02-10T11:40:12.515 INFO:teuthology.orchestra.run.gibba002.stdout: "pool_max": 41, 2021-02-10T11:40:12.515 INFO:teuthology.orchestra.run.gibba002.stdout: "max_osd": 8, 2021-02-10T11:40:12.515 INFO:teuthology.orchestra.run.gibba002.stdout: "require_min_compat_client": "jewel", 2021-02-10T11:40:12.515 INFO:teuthology.orchestra.run.gibba002.stdout: "min_compat_client": "jewel", 2021-02-10T11:40:12.515 INFO:teuthology.orchestra.run.gibba002.stdout: "require_osd_release": "nautilus", 2021-02-10T11:40:12.515 INFO:teuthology.orchestra.run.gibba002.stdout: "pools": [ 2021-02-10T11:40:12.515 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:40:12.515 INFO:teuthology.orchestra.run.gibba002.stdout: "pool": 1, 2021-02-10T11:40:12.515 INFO:teuthology.orchestra.run.gibba002.stdout: "pool_name": "rbd", 2021-02-10T11:40:12.515 INFO:teuthology.orchestra.run.gibba002.stdout: "create_time": "2021-02-10 11:03:37.322519", 2021-02-10T11:40:12.516 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 1, 2021-02-10T11:40:12.516 INFO:teuthology.orchestra.run.gibba002.stdout: "flags_names": "hashpspool", 2021-02-10T11:40:12.516 INFO:teuthology.orchestra.run.gibba002.stdout: "type": 1, 2021-02-10T11:40:12.516 INFO:teuthology.orchestra.run.gibba002.stdout: "size": 2, 2021-02-10T11:40:12.516 INFO:teuthology.orchestra.run.gibba002.stdout: "min_size": 1, 2021-02-10T11:40:12.516 INFO:teuthology.orchestra.run.gibba002.stdout: "crush_rule": 0, 2021-02-10T11:40:12.516 INFO:teuthology.orchestra.run.gibba002.stdout: "object_hash": 2, 2021-02-10T11:40:12.517 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:40:12.517 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num": 8, 2021-02-10T11:40:12.517 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_placement_num": 8, 2021-02-10T11:40:12.517 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_placement_num_target": 8, 2021-02-10T11:40:12.517 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num_target": 8, 2021-02-10T11:40:12.517 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num_pending": 8, 2021-02-10T11:40:12.517 INFO:teuthology.orchestra.run.gibba002.stdout: "last_pg_merge_meta": { 2021-02-10T11:40:12.517 INFO:teuthology.orchestra.run.gibba002.stdout: "source_pgid": "0.0", 2021-02-10T11:40:12.517 INFO:teuthology.orchestra.run.gibba002.stdout: "ready_epoch": 0, 2021-02-10T11:40:12.517 INFO:teuthology.orchestra.run.gibba002.stdout: "last_epoch_started": 0, 2021-02-10T11:40:12.518 INFO:teuthology.orchestra.run.gibba002.stdout: "last_epoch_clean": 0, 2021-02-10T11:40:12.518 INFO:teuthology.orchestra.run.gibba002.stdout: "source_version": "0'0", 2021-02-10T11:40:12.518 INFO:teuthology.orchestra.run.gibba002.stdout: "target_version": "0'0" 2021-02-10T11:40:12.518 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:40:12.518 INFO:teuthology.orchestra.run.gibba002.stdout: "last_change": "16", 2021-02-10T11:40:12.518 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend": "0", 2021-02-10T11:40:12.518 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:40:12.518 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:40:12.518 INFO:teuthology.orchestra.run.gibba002.stdout: "auid": 0, 2021-02-10T11:40:12.519 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:40:12.519 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_seq": 0, 2021-02-10T11:40:12.519 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_epoch": 0, 2021-02-10T11:40:12.519 INFO:teuthology.orchestra.run.gibba002.stdout: "pool_snaps": [], 2021-02-10T11:40:12.519 INFO:teuthology.orchestra.run.gibba002.stdout: "removed_snaps": "[]", 2021-02-10T11:40:12.519 INFO:teuthology.orchestra.run.gibba002.stdout: "quota_max_bytes": 0, 2021-02-10T11:40:12.519 INFO:teuthology.orchestra.run.gibba002.stdout: "quota_max_objects": 0, 2021-02-10T11:40:12.519 INFO:teuthology.orchestra.run.gibba002.stdout: "tiers": [], 2021-02-10T11:40:12.519 INFO:teuthology.orchestra.run.gibba002.stdout: "tier_of": -1, 2021-02-10T11:40:12.519 INFO:teuthology.orchestra.run.gibba002.stdout: "read_tier": -1, 2021-02-10T11:40:12.520 INFO:teuthology.orchestra.run.gibba002.stdout: "write_tier": -1, 2021-02-10T11:40:12.520 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_mode": "none", 2021-02-10T11:40:12.520 INFO:teuthology.orchestra.run.gibba002.stdout: "target_max_bytes": 0, 2021-02-10T11:40:12.520 INFO:teuthology.orchestra.run.gibba002.stdout: "target_max_objects": 0, 2021-02-10T11:40:12.520 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:40:12.520 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:40:12.520 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:40:12.520 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_min_flush_age": 0, 2021-02-10T11:40:12.520 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_min_evict_age": 0, 2021-02-10T11:40:12.520 INFO:teuthology.orchestra.run.gibba002.stdout: "erasure_code_profile": "", 2021-02-10T11:40:12.521 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_params": { 2021-02-10T11:40:12.521 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "none" 2021-02-10T11:40:12.521 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:40:12.521 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_period": 0, 2021-02-10T11:40:12.521 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_count": 0, 2021-02-10T11:40:12.521 INFO:teuthology.orchestra.run.gibba002.stdout: "use_gmt_hitset": true, 2021-02-10T11:40:12.521 INFO:teuthology.orchestra.run.gibba002.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:40:12.521 INFO:teuthology.orchestra.run.gibba002.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:40:12.521 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:40:12.521 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:40:12.522 INFO:teuthology.orchestra.run.gibba002.stdout: "grade_table": [], 2021-02-10T11:40:12.522 INFO:teuthology.orchestra.run.gibba002.stdout: "stripe_width": 0, 2021-02-10T11:40:12.522 INFO:teuthology.orchestra.run.gibba002.stdout: "expected_num_objects": 0, 2021-02-10T11:40:12.522 INFO:teuthology.orchestra.run.gibba002.stdout: "fast_read": false, 2021-02-10T11:40:12.522 INFO:teuthology.orchestra.run.gibba002.stdout: "options": {}, 2021-02-10T11:40:12.522 INFO:teuthology.orchestra.run.gibba002.stdout: "application_metadata": { 2021-02-10T11:40:12.522 INFO:teuthology.orchestra.run.gibba002.stdout: "rbd": {} 2021-02-10T11:40:12.522 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:40:12.522 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:40:12.522 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:40:12.523 INFO:teuthology.orchestra.run.gibba002.stdout: "pool": 40, 2021-02-10T11:40:12.523 INFO:teuthology.orchestra.run.gibba002.stdout: "pool_name": "cephfs_metadata", 2021-02-10T11:40:12.523 INFO:teuthology.orchestra.run.gibba002.stdout: "create_time": "2021-02-10 11:39:28.340428", 2021-02-10T11:40:12.523 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 1, 2021-02-10T11:40:12.523 INFO:teuthology.orchestra.run.gibba002.stdout: "flags_names": "hashpspool", 2021-02-10T11:40:12.523 INFO:teuthology.orchestra.run.gibba002.stdout: "type": 1, 2021-02-10T11:40:12.523 INFO:teuthology.orchestra.run.gibba002.stdout: "size": 2, 2021-02-10T11:40:12.523 INFO:teuthology.orchestra.run.gibba002.stdout: "min_size": 1, 2021-02-10T11:40:12.523 INFO:teuthology.orchestra.run.gibba002.stdout: "crush_rule": 0, 2021-02-10T11:40:12.524 INFO:teuthology.orchestra.run.gibba002.stdout: "object_hash": 2, 2021-02-10T11:40:12.524 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:40:12.524 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num": 8, 2021-02-10T11:40:12.524 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_placement_num": 8, 2021-02-10T11:40:12.524 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_placement_num_target": 8, 2021-02-10T11:40:12.524 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num_target": 8, 2021-02-10T11:40:12.524 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num_pending": 8, 2021-02-10T11:40:12.524 INFO:teuthology.orchestra.run.gibba002.stdout: "last_pg_merge_meta": { 2021-02-10T11:40:12.524 INFO:teuthology.orchestra.run.gibba002.stdout: "source_pgid": "0.0", 2021-02-10T11:40:12.524 INFO:teuthology.orchestra.run.gibba002.stdout: "ready_epoch": 0, 2021-02-10T11:40:12.525 INFO:teuthology.orchestra.run.gibba002.stdout: "last_epoch_started": 0, 2021-02-10T11:40:12.525 INFO:teuthology.orchestra.run.gibba002.stdout: "last_epoch_clean": 0, 2021-02-10T11:40:12.525 INFO:teuthology.orchestra.run.gibba002.stdout: "source_version": "0'0", 2021-02-10T11:40:12.525 INFO:teuthology.orchestra.run.gibba002.stdout: "target_version": "0'0" 2021-02-10T11:40:12.525 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:40:12.525 INFO:teuthology.orchestra.run.gibba002.stdout: "last_change": "188", 2021-02-10T11:40:12.525 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend": "0", 2021-02-10T11:40:12.525 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:40:12.525 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:40:12.525 INFO:teuthology.orchestra.run.gibba002.stdout: "auid": 0, 2021-02-10T11:40:12.525 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:40:12.526 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_seq": 0, 2021-02-10T11:40:12.526 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_epoch": 0, 2021-02-10T11:40:12.526 INFO:teuthology.orchestra.run.gibba002.stdout: "pool_snaps": [], 2021-02-10T11:40:12.526 INFO:teuthology.orchestra.run.gibba002.stdout: "removed_snaps": "[]", 2021-02-10T11:40:12.526 INFO:teuthology.orchestra.run.gibba002.stdout: "quota_max_bytes": 0, 2021-02-10T11:40:12.526 INFO:teuthology.orchestra.run.gibba002.stdout: "quota_max_objects": 0, 2021-02-10T11:40:12.526 INFO:teuthology.orchestra.run.gibba002.stdout: "tiers": [], 2021-02-10T11:40:12.526 INFO:teuthology.orchestra.run.gibba002.stdout: "tier_of": -1, 2021-02-10T11:40:12.526 INFO:teuthology.orchestra.run.gibba002.stdout: "read_tier": -1, 2021-02-10T11:40:12.527 INFO:teuthology.orchestra.run.gibba002.stdout: "write_tier": -1, 2021-02-10T11:40:12.527 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_mode": "none", 2021-02-10T11:40:12.527 INFO:teuthology.orchestra.run.gibba002.stdout: "target_max_bytes": 0, 2021-02-10T11:40:12.527 INFO:teuthology.orchestra.run.gibba002.stdout: "target_max_objects": 0, 2021-02-10T11:40:12.527 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:40:12.527 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:40:12.527 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:40:12.527 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_min_flush_age": 0, 2021-02-10T11:40:12.527 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_min_evict_age": 0, 2021-02-10T11:40:12.527 INFO:teuthology.orchestra.run.gibba002.stdout: "erasure_code_profile": "", 2021-02-10T11:40:12.528 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_params": { 2021-02-10T11:40:12.528 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "none" 2021-02-10T11:40:12.528 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:40:12.528 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_period": 0, 2021-02-10T11:40:12.528 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_count": 0, 2021-02-10T11:40:12.528 INFO:teuthology.orchestra.run.gibba002.stdout: "use_gmt_hitset": true, 2021-02-10T11:40:12.528 INFO:teuthology.orchestra.run.gibba002.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:40:12.528 INFO:teuthology.orchestra.run.gibba002.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:40:12.528 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:40:12.528 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:40:12.529 INFO:teuthology.orchestra.run.gibba002.stdout: "grade_table": [], 2021-02-10T11:40:12.529 INFO:teuthology.orchestra.run.gibba002.stdout: "stripe_width": 0, 2021-02-10T11:40:12.529 INFO:teuthology.orchestra.run.gibba002.stdout: "expected_num_objects": 0, 2021-02-10T11:40:12.529 INFO:teuthology.orchestra.run.gibba002.stdout: "fast_read": false, 2021-02-10T11:40:12.529 INFO:teuthology.orchestra.run.gibba002.stdout: "options": { 2021-02-10T11:40:12.529 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_autoscale_bias": 4, 2021-02-10T11:40:12.529 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num_min": 16, 2021-02-10T11:40:12.529 INFO:teuthology.orchestra.run.gibba002.stdout: "recovery_priority": 5 2021-02-10T11:40:12.529 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:40:12.529 INFO:teuthology.orchestra.run.gibba002.stdout: "application_metadata": { 2021-02-10T11:40:12.530 INFO:teuthology.orchestra.run.gibba002.stdout: "cephfs": { 2021-02-10T11:40:12.530 INFO:teuthology.orchestra.run.gibba002.stdout: "metadata": "cephfs" 2021-02-10T11:40:12.530 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:40:12.530 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:40:12.530 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:40:12.530 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:40:12.530 INFO:teuthology.orchestra.run.gibba002.stdout: "pool": 41, 2021-02-10T11:40:12.530 INFO:teuthology.orchestra.run.gibba002.stdout: "pool_name": "cephfs_data", 2021-02-10T11:40:12.530 INFO:teuthology.orchestra.run.gibba002.stdout: "create_time": "2021-02-10 11:39:29.367624", 2021-02-10T11:40:12.531 INFO:teuthology.orchestra.run.gibba002.stdout: "flags": 1, 2021-02-10T11:40:12.531 INFO:teuthology.orchestra.run.gibba002.stdout: "flags_names": "hashpspool", 2021-02-10T11:40:12.531 INFO:teuthology.orchestra.run.gibba002.stdout: "type": 1, 2021-02-10T11:40:12.531 INFO:teuthology.orchestra.run.gibba002.stdout: "size": 2, 2021-02-10T11:40:12.531 INFO:teuthology.orchestra.run.gibba002.stdout: "min_size": 1, 2021-02-10T11:40:12.531 INFO:teuthology.orchestra.run.gibba002.stdout: "crush_rule": 0, 2021-02-10T11:40:12.531 INFO:teuthology.orchestra.run.gibba002.stdout: "object_hash": 2, 2021-02-10T11:40:12.531 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:40:12.531 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num": 8, 2021-02-10T11:40:12.532 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_placement_num": 8, 2021-02-10T11:40:12.532 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_placement_num_target": 8, 2021-02-10T11:40:12.532 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num_target": 8, 2021-02-10T11:40:12.532 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_num_pending": 8, 2021-02-10T11:40:12.532 INFO:teuthology.orchestra.run.gibba002.stdout: "last_pg_merge_meta": { 2021-02-10T11:40:12.532 INFO:teuthology.orchestra.run.gibba002.stdout: "source_pgid": "0.0", 2021-02-10T11:40:12.532 INFO:teuthology.orchestra.run.gibba002.stdout: "ready_epoch": 0, 2021-02-10T11:40:12.533 INFO:teuthology.orchestra.run.gibba002.stdout: "last_epoch_started": 0, 2021-02-10T11:40:12.533 INFO:teuthology.orchestra.run.gibba002.stdout: "last_epoch_clean": 0, 2021-02-10T11:40:12.533 INFO:teuthology.orchestra.run.gibba002.stdout: "source_version": "0'0", 2021-02-10T11:40:12.533 INFO:teuthology.orchestra.run.gibba002.stdout: "target_version": "0'0" 2021-02-10T11:40:12.533 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:40:12.533 INFO:teuthology.orchestra.run.gibba002.stdout: "last_change": "189", 2021-02-10T11:40:12.533 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend": "0", 2021-02-10T11:40:12.533 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:40:12.533 INFO:teuthology.orchestra.run.gibba002.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:40:12.533 INFO:teuthology.orchestra.run.gibba002.stdout: "auid": 0, 2021-02-10T11:40:12.533 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:40:12.534 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_seq": 0, 2021-02-10T11:40:12.534 INFO:teuthology.orchestra.run.gibba002.stdout: "snap_epoch": 0, 2021-02-10T11:40:12.534 INFO:teuthology.orchestra.run.gibba002.stdout: "pool_snaps": [], 2021-02-10T11:40:12.534 INFO:teuthology.orchestra.run.gibba002.stdout: "removed_snaps": "[]", 2021-02-10T11:40:12.534 INFO:teuthology.orchestra.run.gibba002.stdout: "quota_max_bytes": 0, 2021-02-10T11:40:12.534 INFO:teuthology.orchestra.run.gibba002.stdout: "quota_max_objects": 0, 2021-02-10T11:40:12.534 INFO:teuthology.orchestra.run.gibba002.stdout: "tiers": [], 2021-02-10T11:40:12.534 INFO:teuthology.orchestra.run.gibba002.stdout: "tier_of": -1, 2021-02-10T11:40:12.534 INFO:teuthology.orchestra.run.gibba002.stdout: "read_tier": -1, 2021-02-10T11:40:12.534 INFO:teuthology.orchestra.run.gibba002.stdout: "write_tier": -1, 2021-02-10T11:40:12.535 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_mode": "none", 2021-02-10T11:40:12.535 INFO:teuthology.orchestra.run.gibba002.stdout: "target_max_bytes": 0, 2021-02-10T11:40:12.535 INFO:teuthology.orchestra.run.gibba002.stdout: "target_max_objects": 0, 2021-02-10T11:40:12.535 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:40:12.535 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:40:12.535 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:40:12.535 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_min_flush_age": 0, 2021-02-10T11:40:12.535 INFO:teuthology.orchestra.run.gibba002.stdout: "cache_min_evict_age": 0, 2021-02-10T11:40:12.535 INFO:teuthology.orchestra.run.gibba002.stdout: "erasure_code_profile": "", 2021-02-10T11:40:12.535 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_params": { 2021-02-10T11:40:12.536 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "none" 2021-02-10T11:40:12.536 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:40:12.536 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_period": 0, 2021-02-10T11:40:12.536 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_count": 0, 2021-02-10T11:40:12.536 INFO:teuthology.orchestra.run.gibba002.stdout: "use_gmt_hitset": true, 2021-02-10T11:40:12.536 INFO:teuthology.orchestra.run.gibba002.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:40:12.536 INFO:teuthology.orchestra.run.gibba002.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:40:12.536 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:40:12.536 INFO:teuthology.orchestra.run.gibba002.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:40:12.536 INFO:teuthology.orchestra.run.gibba002.stdout: "grade_table": [], 2021-02-10T11:40:12.536 INFO:teuthology.orchestra.run.gibba002.stdout: "stripe_width": 0, 2021-02-10T11:40:12.536 INFO:teuthology.orchestra.run.gibba002.stdout: "expected_num_objects": 0, 2021-02-10T11:40:12.537 INFO:teuthology.orchestra.run.gibba002.stdout: "fast_read": false, 2021-02-10T11:40:12.537 INFO:teuthology.orchestra.run.gibba002.stdout: "options": {}, 2021-02-10T11:40:12.537 INFO:teuthology.orchestra.run.gibba002.stdout: "application_metadata": { 2021-02-10T11:40:12.537 INFO:teuthology.orchestra.run.gibba002.stdout: "cephfs": { 2021-02-10T11:40:12.537 INFO:teuthology.orchestra.run.gibba002.stdout: "data": "cephfs" 2021-02-10T11:40:12.537 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:40:12.537 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:40:12.537 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:40:12.537 INFO:teuthology.orchestra.run.gibba002.stdout: ], 2021-02-10T11:40:12.538 INFO:teuthology.orchestra.run.gibba002.stdout: "osds": [ 2021-02-10T11:40:12.538 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:40:12.538 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 0, 2021-02-10T11:40:12.538 INFO:teuthology.orchestra.run.gibba002.stdout: "uuid": "be691b4d-a039-408a-af80-9d53f02b6f1f", 2021-02-10T11:40:12.538 INFO:teuthology.orchestra.run.gibba002.stdout: "up": 1, 2021-02-10T11:40:12.538 INFO:teuthology.orchestra.run.gibba002.stdout: "in": 1, 2021-02-10T11:40:12.539 INFO:teuthology.orchestra.run.gibba002.stdout: "weight": 1, 2021-02-10T11:40:12.539 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_affinity": 1, 2021-02-10T11:40:12.539 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_begin": 0, 2021-02-10T11:40:12.539 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_end": 0, 2021-02-10T11:40:12.539 INFO:teuthology.orchestra.run.gibba002.stdout: "up_from": 13, 2021-02-10T11:40:12.539 INFO:teuthology.orchestra.run.gibba002.stdout: "up_thru": 185, 2021-02-10T11:40:12.539 INFO:teuthology.orchestra.run.gibba002.stdout: "down_at": 0, 2021-02-10T11:40:12.539 INFO:teuthology.orchestra.run.gibba002.stdout: "lost_at": 0, 2021-02-10T11:40:12.539 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addrs": { 2021-02-10T11:40:12.539 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:40:12.539 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:40:12.540 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:40:12.540 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6818", 2021-02-10T11:40:12.540 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36088 2021-02-10T11:40:12.540 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:40:12.540 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:40:12.540 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:40:12.540 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6820", 2021-02-10T11:40:12.540 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36088 2021-02-10T11:40:12.540 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:40:12.540 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:40:12.541 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:40:12.541 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addrs": { 2021-02-10T11:40:12.541 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:40:12.541 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:40:12.541 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:40:12.541 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6822", 2021-02-10T11:40:12.541 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36088 2021-02-10T11:40:12.541 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:40:12.541 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:40:12.541 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:40:12.542 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6824", 2021-02-10T11:40:12.543 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36088 2021-02-10T11:40:12.543 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:40:12.543 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:40:12.543 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:40:12.543 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addrs": { 2021-02-10T11:40:12.543 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:40:12.543 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:40:12.543 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:40:12.544 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6830", 2021-02-10T11:40:12.544 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36088 2021-02-10T11:40:12.544 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:40:12.544 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:40:12.544 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:40:12.544 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6832", 2021-02-10T11:40:12.544 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36088 2021-02-10T11:40:12.544 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:40:12.544 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:40:12.544 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:40:12.545 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addrs": { 2021-02-10T11:40:12.545 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:40:12.545 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:40:12.545 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:40:12.545 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6826", 2021-02-10T11:40:12.545 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36088 2021-02-10T11:40:12.545 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:40:12.545 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:40:12.545 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:40:12.545 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6828", 2021-02-10T11:40:12.545 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36088 2021-02-10T11:40:12.546 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:40:12.546 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:40:12.546 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:40:12.546 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addr": "172.21.2.102:6820/36088", 2021-02-10T11:40:12.546 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addr": "172.21.2.102:6824/36088", 2021-02-10T11:40:12.546 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addr": "172.21.2.102:6832/36088", 2021-02-10T11:40:12.546 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addr": "172.21.2.102:6828/36088", 2021-02-10T11:40:12.546 INFO:teuthology.orchestra.run.gibba002.stdout: "state": [ 2021-02-10T11:40:12.546 INFO:teuthology.orchestra.run.gibba002.stdout: "exists", 2021-02-10T11:40:12.547 INFO:teuthology.orchestra.run.gibba002.stdout: "up" 2021-02-10T11:40:12.547 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:40:12.547 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:40:12.547 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:40:12.547 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 1, 2021-02-10T11:40:12.547 INFO:teuthology.orchestra.run.gibba002.stdout: "uuid": "64e3ccc8-9c69-4d7c-865e-af8e83822635", 2021-02-10T11:40:12.547 INFO:teuthology.orchestra.run.gibba002.stdout: "up": 1, 2021-02-10T11:40:12.547 INFO:teuthology.orchestra.run.gibba002.stdout: "in": 1, 2021-02-10T11:40:12.547 INFO:teuthology.orchestra.run.gibba002.stdout: "weight": 1, 2021-02-10T11:40:12.548 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_affinity": 1, 2021-02-10T11:40:12.548 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_begin": 0, 2021-02-10T11:40:12.548 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_end": 0, 2021-02-10T11:40:12.548 INFO:teuthology.orchestra.run.gibba002.stdout: "up_from": 13, 2021-02-10T11:40:12.548 INFO:teuthology.orchestra.run.gibba002.stdout: "up_thru": 185, 2021-02-10T11:40:12.548 INFO:teuthology.orchestra.run.gibba002.stdout: "down_at": 0, 2021-02-10T11:40:12.548 INFO:teuthology.orchestra.run.gibba002.stdout: "lost_at": 0, 2021-02-10T11:40:12.548 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addrs": { 2021-02-10T11:40:12.548 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:40:12.548 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:40:12.549 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:40:12.549 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6819", 2021-02-10T11:40:12.549 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36087 2021-02-10T11:40:12.549 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:40:12.549 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:40:12.549 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:40:12.549 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6821", 2021-02-10T11:40:12.549 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36087 2021-02-10T11:40:12.549 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:40:12.549 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:40:12.549 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:40:12.550 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addrs": { 2021-02-10T11:40:12.550 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:40:12.550 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:40:12.550 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:40:12.550 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6823", 2021-02-10T11:40:12.550 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36087 2021-02-10T11:40:12.550 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:40:12.550 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:40:12.550 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:40:12.551 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6825", 2021-02-10T11:40:12.551 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36087 2021-02-10T11:40:12.551 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:40:12.551 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:40:12.551 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:40:12.551 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addrs": { 2021-02-10T11:40:12.551 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:40:12.551 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:40:12.551 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:40:12.551 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6831", 2021-02-10T11:40:12.552 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36087 2021-02-10T11:40:12.552 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:40:12.552 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:40:12.552 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:40:12.552 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6833", 2021-02-10T11:40:12.552 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36087 2021-02-10T11:40:12.552 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:40:12.552 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:40:12.552 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:40:12.552 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addrs": { 2021-02-10T11:40:12.553 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:40:12.553 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:40:12.553 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:40:12.553 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6827", 2021-02-10T11:40:12.553 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36087 2021-02-10T11:40:12.553 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:40:12.553 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:40:12.553 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:40:12.553 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6829", 2021-02-10T11:40:12.554 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36087 2021-02-10T11:40:12.554 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:40:12.554 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:40:12.554 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:40:12.554 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addr": "172.21.2.102:6821/36087", 2021-02-10T11:40:12.554 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addr": "172.21.2.102:6825/36087", 2021-02-10T11:40:12.554 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addr": "172.21.2.102:6833/36087", 2021-02-10T11:40:12.554 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addr": "172.21.2.102:6829/36087", 2021-02-10T11:40:12.554 INFO:teuthology.orchestra.run.gibba002.stdout: "state": [ 2021-02-10T11:40:12.554 INFO:teuthology.orchestra.run.gibba002.stdout: "exists", 2021-02-10T11:40:12.555 INFO:teuthology.orchestra.run.gibba002.stdout: "up" 2021-02-10T11:40:12.555 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:40:12.555 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:40:12.555 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:40:12.555 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 2, 2021-02-10T11:40:12.555 INFO:teuthology.orchestra.run.gibba002.stdout: "uuid": "153700af-747f-481f-bdf4-d6072d3a842f", 2021-02-10T11:40:12.555 INFO:teuthology.orchestra.run.gibba002.stdout: "up": 1, 2021-02-10T11:40:12.555 INFO:teuthology.orchestra.run.gibba002.stdout: "in": 1, 2021-02-10T11:40:12.555 INFO:teuthology.orchestra.run.gibba002.stdout: "weight": 1, 2021-02-10T11:40:12.555 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_affinity": 1, 2021-02-10T11:40:12.556 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_begin": 0, 2021-02-10T11:40:12.556 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_end": 0, 2021-02-10T11:40:12.556 INFO:teuthology.orchestra.run.gibba002.stdout: "up_from": 13, 2021-02-10T11:40:12.556 INFO:teuthology.orchestra.run.gibba002.stdout: "up_thru": 187, 2021-02-10T11:40:12.556 INFO:teuthology.orchestra.run.gibba002.stdout: "down_at": 0, 2021-02-10T11:40:12.556 INFO:teuthology.orchestra.run.gibba002.stdout: "lost_at": 0, 2021-02-10T11:40:12.556 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addrs": { 2021-02-10T11:40:12.556 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:40:12.556 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:40:12.556 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:40:12.557 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6810", 2021-02-10T11:40:12.557 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36089 2021-02-10T11:40:12.557 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:40:12.557 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:40:12.557 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:40:12.557 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6811", 2021-02-10T11:40:12.557 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36089 2021-02-10T11:40:12.557 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:40:12.557 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:40:12.557 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:40:12.557 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addrs": { 2021-02-10T11:40:12.558 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:40:12.558 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:40:12.558 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:40:12.558 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6812", 2021-02-10T11:40:12.558 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36089 2021-02-10T11:40:12.558 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:40:12.558 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:40:12.558 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:40:12.558 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6813", 2021-02-10T11:40:12.559 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36089 2021-02-10T11:40:12.559 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:40:12.559 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:40:12.559 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:40:12.559 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addrs": { 2021-02-10T11:40:12.559 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:40:12.559 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:40:12.559 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:40:12.559 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6816", 2021-02-10T11:40:12.559 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36089 2021-02-10T11:40:12.560 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:40:12.560 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:40:12.560 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:40:12.560 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6817", 2021-02-10T11:40:12.560 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36089 2021-02-10T11:40:12.560 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:40:12.560 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:40:12.560 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:40:12.560 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addrs": { 2021-02-10T11:40:12.560 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:40:12.561 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:40:12.561 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:40:12.561 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6814", 2021-02-10T11:40:12.561 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36089 2021-02-10T11:40:12.561 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:40:12.561 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:40:12.561 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:40:12.561 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6815", 2021-02-10T11:40:12.561 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36089 2021-02-10T11:40:12.561 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:40:12.562 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:40:12.562 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:40:12.562 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addr": "172.21.2.102:6811/36089", 2021-02-10T11:40:12.562 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addr": "172.21.2.102:6813/36089", 2021-02-10T11:40:12.562 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addr": "172.21.2.102:6817/36089", 2021-02-10T11:40:12.562 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addr": "172.21.2.102:6815/36089", 2021-02-10T11:40:12.562 INFO:teuthology.orchestra.run.gibba002.stdout: "state": [ 2021-02-10T11:40:12.562 INFO:teuthology.orchestra.run.gibba002.stdout: "exists", 2021-02-10T11:40:12.562 INFO:teuthology.orchestra.run.gibba002.stdout: "up" 2021-02-10T11:40:12.562 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:40:12.563 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:40:12.563 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:40:12.563 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 3, 2021-02-10T11:40:12.563 INFO:teuthology.orchestra.run.gibba002.stdout: "uuid": "cbc678f4-83a9-45e6-8830-2f7c232b1ea7", 2021-02-10T11:40:12.563 INFO:teuthology.orchestra.run.gibba002.stdout: "up": 1, 2021-02-10T11:40:12.563 INFO:teuthology.orchestra.run.gibba002.stdout: "in": 1, 2021-02-10T11:40:12.563 INFO:teuthology.orchestra.run.gibba002.stdout: "weight": 1, 2021-02-10T11:40:12.563 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_affinity": 1, 2021-02-10T11:40:12.563 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_begin": 0, 2021-02-10T11:40:12.563 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_end": 0, 2021-02-10T11:40:12.564 INFO:teuthology.orchestra.run.gibba002.stdout: "up_from": 13, 2021-02-10T11:40:12.564 INFO:teuthology.orchestra.run.gibba002.stdout: "up_thru": 187, 2021-02-10T11:40:12.564 INFO:teuthology.orchestra.run.gibba002.stdout: "down_at": 0, 2021-02-10T11:40:12.564 INFO:teuthology.orchestra.run.gibba002.stdout: "lost_at": 0, 2021-02-10T11:40:12.564 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addrs": { 2021-02-10T11:40:12.564 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:40:12.564 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:40:12.564 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:40:12.564 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6802", 2021-02-10T11:40:12.564 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36090 2021-02-10T11:40:12.565 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:40:12.565 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:40:12.565 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:40:12.565 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6803", 2021-02-10T11:40:12.565 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36090 2021-02-10T11:40:12.565 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:40:12.565 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:40:12.565 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:40:12.565 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addrs": { 2021-02-10T11:40:12.565 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:40:12.566 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:40:12.566 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:40:12.566 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6804", 2021-02-10T11:40:12.566 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36090 2021-02-10T11:40:12.566 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:40:12.566 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:40:12.566 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:40:12.566 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6805", 2021-02-10T11:40:12.566 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36090 2021-02-10T11:40:12.566 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:40:12.567 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:40:12.567 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:40:12.567 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addrs": { 2021-02-10T11:40:12.567 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:40:12.567 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:40:12.567 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:40:12.567 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6808", 2021-02-10T11:40:12.567 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36090 2021-02-10T11:40:12.567 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:40:12.567 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:40:12.567 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:40:12.568 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6809", 2021-02-10T11:40:12.568 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36090 2021-02-10T11:40:12.568 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:40:12.568 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:40:12.568 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:40:12.568 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addrs": { 2021-02-10T11:40:12.568 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:40:12.568 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:40:12.568 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:40:12.568 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6806", 2021-02-10T11:40:12.569 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36090 2021-02-10T11:40:12.569 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:40:12.569 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:40:12.569 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:40:12.569 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:6807", 2021-02-10T11:40:12.569 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36090 2021-02-10T11:40:12.569 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:40:12.569 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:40:12.570 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:40:12.570 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addr": "172.21.2.102:6803/36090", 2021-02-10T11:40:12.570 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addr": "172.21.2.102:6805/36090", 2021-02-10T11:40:12.570 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addr": "172.21.2.102:6809/36090", 2021-02-10T11:40:12.570 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addr": "172.21.2.102:6807/36090", 2021-02-10T11:40:12.570 INFO:teuthology.orchestra.run.gibba002.stdout: "state": [ 2021-02-10T11:40:12.571 INFO:teuthology.orchestra.run.gibba002.stdout: "exists", 2021-02-10T11:40:12.571 INFO:teuthology.orchestra.run.gibba002.stdout: "up" 2021-02-10T11:40:12.571 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:40:12.571 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:40:12.571 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:40:12.571 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 4, 2021-02-10T11:40:12.571 INFO:teuthology.orchestra.run.gibba002.stdout: "uuid": "c01e4008-b258-4f10-afd2-9e7f37214351", 2021-02-10T11:40:12.571 INFO:teuthology.orchestra.run.gibba002.stdout: "up": 1, 2021-02-10T11:40:12.571 INFO:teuthology.orchestra.run.gibba002.stdout: "in": 1, 2021-02-10T11:40:12.571 INFO:teuthology.orchestra.run.gibba002.stdout: "weight": 1, 2021-02-10T11:40:12.572 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_affinity": 1, 2021-02-10T11:40:12.572 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_begin": 0, 2021-02-10T11:40:12.572 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_end": 0, 2021-02-10T11:40:12.572 INFO:teuthology.orchestra.run.gibba002.stdout: "up_from": 13, 2021-02-10T11:40:12.572 INFO:teuthology.orchestra.run.gibba002.stdout: "up_thru": 187, 2021-02-10T11:40:12.573 INFO:teuthology.orchestra.run.gibba002.stdout: "down_at": 0, 2021-02-10T11:40:12.573 INFO:teuthology.orchestra.run.gibba002.stdout: "lost_at": 0, 2021-02-10T11:40:12.573 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addrs": { 2021-02-10T11:40:12.573 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:40:12.573 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:40:12.573 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:40:12.573 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6808", 2021-02-10T11:40:12.573 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36368 2021-02-10T11:40:12.574 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:40:12.574 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:40:12.574 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:40:12.574 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6811", 2021-02-10T11:40:12.574 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36368 2021-02-10T11:40:12.574 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:40:12.574 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:40:12.574 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:40:12.574 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addrs": { 2021-02-10T11:40:12.574 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:40:12.574 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:40:12.574 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:40:12.574 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6814", 2021-02-10T11:40:12.575 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36368 2021-02-10T11:40:12.575 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:40:12.575 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:40:12.575 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:40:12.575 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6818", 2021-02-10T11:40:12.575 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36368 2021-02-10T11:40:12.575 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:40:12.575 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:40:12.575 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:40:12.575 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addrs": { 2021-02-10T11:40:12.575 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:40:12.575 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:40:12.575 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:40:12.576 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6826", 2021-02-10T11:40:12.576 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36368 2021-02-10T11:40:12.576 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:40:12.576 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:40:12.576 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:40:12.576 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6828", 2021-02-10T11:40:12.576 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36368 2021-02-10T11:40:12.576 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:40:12.576 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:40:12.576 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:40:12.576 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addrs": { 2021-02-10T11:40:12.576 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:40:12.576 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:40:12.577 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:40:12.577 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6820", 2021-02-10T11:40:12.577 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36368 2021-02-10T11:40:12.577 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:40:12.577 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:40:12.577 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:40:12.577 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6824", 2021-02-10T11:40:12.577 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36368 2021-02-10T11:40:12.577 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:40:12.577 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:40:12.577 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:40:12.577 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addr": "172.21.2.113:6811/36368", 2021-02-10T11:40:12.578 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addr": "172.21.2.113:6818/36368", 2021-02-10T11:40:12.578 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addr": "172.21.2.113:6828/36368", 2021-02-10T11:40:12.578 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addr": "172.21.2.113:6824/36368", 2021-02-10T11:40:12.578 INFO:teuthology.orchestra.run.gibba002.stdout: "state": [ 2021-02-10T11:40:12.578 INFO:teuthology.orchestra.run.gibba002.stdout: "exists", 2021-02-10T11:40:12.578 INFO:teuthology.orchestra.run.gibba002.stdout: "up" 2021-02-10T11:40:12.579 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:40:12.579 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:40:12.579 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:40:12.579 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 5, 2021-02-10T11:40:12.579 INFO:teuthology.orchestra.run.gibba002.stdout: "uuid": "551f6007-daf8-4a98-8ee0-c2b16bcccb5e", 2021-02-10T11:40:12.579 INFO:teuthology.orchestra.run.gibba002.stdout: "up": 1, 2021-02-10T11:40:12.579 INFO:teuthology.orchestra.run.gibba002.stdout: "in": 1, 2021-02-10T11:40:12.579 INFO:teuthology.orchestra.run.gibba002.stdout: "weight": 1, 2021-02-10T11:40:12.579 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_affinity": 1, 2021-02-10T11:40:12.579 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_begin": 0, 2021-02-10T11:40:12.579 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_end": 0, 2021-02-10T11:40:12.579 INFO:teuthology.orchestra.run.gibba002.stdout: "up_from": 13, 2021-02-10T11:40:12.579 INFO:teuthology.orchestra.run.gibba002.stdout: "up_thru": 178, 2021-02-10T11:40:12.580 INFO:teuthology.orchestra.run.gibba002.stdout: "down_at": 0, 2021-02-10T11:40:12.580 INFO:teuthology.orchestra.run.gibba002.stdout: "lost_at": 0, 2021-02-10T11:40:12.580 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addrs": { 2021-02-10T11:40:12.580 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:40:12.580 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:40:12.580 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:40:12.580 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6802", 2021-02-10T11:40:12.580 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36366 2021-02-10T11:40:12.580 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:40:12.580 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:40:12.580 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:40:12.580 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6805", 2021-02-10T11:40:12.580 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36366 2021-02-10T11:40:12.581 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:40:12.581 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:40:12.581 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:40:12.581 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addrs": { 2021-02-10T11:40:12.581 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:40:12.581 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:40:12.581 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:40:12.581 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6807", 2021-02-10T11:40:12.581 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36366 2021-02-10T11:40:12.581 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:40:12.581 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:40:12.581 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:40:12.581 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6810", 2021-02-10T11:40:12.581 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36366 2021-02-10T11:40:12.582 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:40:12.582 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:40:12.582 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:40:12.582 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addrs": { 2021-02-10T11:40:12.582 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:40:12.582 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:40:12.582 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:40:12.582 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6819", 2021-02-10T11:40:12.582 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36366 2021-02-10T11:40:12.582 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:40:12.582 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:40:12.582 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:40:12.582 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6822", 2021-02-10T11:40:12.583 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36366 2021-02-10T11:40:12.583 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:40:12.583 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:40:12.583 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:40:12.583 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addrs": { 2021-02-10T11:40:12.583 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:40:12.583 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:40:12.583 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:40:12.583 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6812", 2021-02-10T11:40:12.583 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36366 2021-02-10T11:40:12.583 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:40:12.583 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:40:12.583 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:40:12.583 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6815", 2021-02-10T11:40:12.584 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36366 2021-02-10T11:40:12.584 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:40:12.584 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:40:12.584 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:40:12.584 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addr": "172.21.2.113:6805/36366", 2021-02-10T11:40:12.584 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addr": "172.21.2.113:6810/36366", 2021-02-10T11:40:12.584 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addr": "172.21.2.113:6822/36366", 2021-02-10T11:40:12.584 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addr": "172.21.2.113:6815/36366", 2021-02-10T11:40:12.584 INFO:teuthology.orchestra.run.gibba002.stdout: "state": [ 2021-02-10T11:40:12.584 INFO:teuthology.orchestra.run.gibba002.stdout: "exists", 2021-02-10T11:40:12.584 INFO:teuthology.orchestra.run.gibba002.stdout: "up" 2021-02-10T11:40:12.584 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:40:12.584 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:40:12.584 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:40:12.585 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 6, 2021-02-10T11:40:12.585 INFO:teuthology.orchestra.run.gibba002.stdout: "uuid": "bf67c040-0e50-48be-b102-c3943bc9ebef", 2021-02-10T11:40:12.585 INFO:teuthology.orchestra.run.gibba002.stdout: "up": 1, 2021-02-10T11:40:12.585 INFO:teuthology.orchestra.run.gibba002.stdout: "in": 1, 2021-02-10T11:40:12.585 INFO:teuthology.orchestra.run.gibba002.stdout: "weight": 1, 2021-02-10T11:40:12.585 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_affinity": 1, 2021-02-10T11:40:12.585 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_begin": 0, 2021-02-10T11:40:12.585 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_end": 0, 2021-02-10T11:40:12.585 INFO:teuthology.orchestra.run.gibba002.stdout: "up_from": 13, 2021-02-10T11:40:12.585 INFO:teuthology.orchestra.run.gibba002.stdout: "up_thru": 187, 2021-02-10T11:40:12.585 INFO:teuthology.orchestra.run.gibba002.stdout: "down_at": 0, 2021-02-10T11:40:12.585 INFO:teuthology.orchestra.run.gibba002.stdout: "lost_at": 0, 2021-02-10T11:40:12.585 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addrs": { 2021-02-10T11:40:12.586 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:40:12.586 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:40:12.586 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:40:12.586 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6800", 2021-02-10T11:40:12.586 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36367 2021-02-10T11:40:12.586 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:40:12.586 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:40:12.586 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:40:12.586 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6801", 2021-02-10T11:40:12.586 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36367 2021-02-10T11:40:12.586 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:40:12.586 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:40:12.586 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:40:12.587 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addrs": { 2021-02-10T11:40:12.587 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:40:12.587 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:40:12.587 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:40:12.587 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6803", 2021-02-10T11:40:12.587 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36367 2021-02-10T11:40:12.587 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:40:12.587 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:40:12.587 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:40:12.587 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6804", 2021-02-10T11:40:12.587 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36367 2021-02-10T11:40:12.587 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:40:12.587 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:40:12.587 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:40:12.588 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addrs": { 2021-02-10T11:40:12.588 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:40:12.588 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:40:12.588 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:40:12.588 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6813", 2021-02-10T11:40:12.588 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36367 2021-02-10T11:40:12.588 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:40:12.588 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:40:12.588 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:40:12.588 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6816", 2021-02-10T11:40:12.588 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36367 2021-02-10T11:40:12.588 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:40:12.588 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:40:12.589 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:40:12.589 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addrs": { 2021-02-10T11:40:12.589 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:40:12.589 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:40:12.589 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:40:12.589 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6806", 2021-02-10T11:40:12.589 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36367 2021-02-10T11:40:12.589 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:40:12.589 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:40:12.589 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:40:12.589 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6809", 2021-02-10T11:40:12.589 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36367 2021-02-10T11:40:12.589 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:40:12.590 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:40:12.590 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:40:12.590 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addr": "172.21.2.113:6801/36367", 2021-02-10T11:40:12.590 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addr": "172.21.2.113:6804/36367", 2021-02-10T11:40:12.590 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addr": "172.21.2.113:6816/36367", 2021-02-10T11:40:12.590 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addr": "172.21.2.113:6809/36367", 2021-02-10T11:40:12.590 INFO:teuthology.orchestra.run.gibba002.stdout: "state": [ 2021-02-10T11:40:12.590 INFO:teuthology.orchestra.run.gibba002.stdout: "exists", 2021-02-10T11:40:12.590 INFO:teuthology.orchestra.run.gibba002.stdout: "up" 2021-02-10T11:40:12.590 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:40:12.590 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:40:12.590 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:40:12.590 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 7, 2021-02-10T11:40:12.591 INFO:teuthology.orchestra.run.gibba002.stdout: "uuid": "f5461f80-018d-428f-be89-2eb2bcbe7bb6", 2021-02-10T11:40:12.591 INFO:teuthology.orchestra.run.gibba002.stdout: "up": 1, 2021-02-10T11:40:12.591 INFO:teuthology.orchestra.run.gibba002.stdout: "in": 1, 2021-02-10T11:40:12.591 INFO:teuthology.orchestra.run.gibba002.stdout: "weight": 1, 2021-02-10T11:40:12.591 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_affinity": 1, 2021-02-10T11:40:12.591 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_begin": 0, 2021-02-10T11:40:12.591 INFO:teuthology.orchestra.run.gibba002.stdout: "last_clean_end": 0, 2021-02-10T11:40:12.591 INFO:teuthology.orchestra.run.gibba002.stdout: "up_from": 13, 2021-02-10T11:40:12.591 INFO:teuthology.orchestra.run.gibba002.stdout: "up_thru": 187, 2021-02-10T11:40:12.591 INFO:teuthology.orchestra.run.gibba002.stdout: "down_at": 0, 2021-02-10T11:40:12.591 INFO:teuthology.orchestra.run.gibba002.stdout: "lost_at": 0, 2021-02-10T11:40:12.591 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addrs": { 2021-02-10T11:40:12.592 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:40:12.592 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:40:12.592 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:40:12.592 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6817", 2021-02-10T11:40:12.592 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36365 2021-02-10T11:40:12.592 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:40:12.592 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:40:12.592 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:40:12.592 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6821", 2021-02-10T11:40:12.592 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36365 2021-02-10T11:40:12.592 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:40:12.592 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:40:12.592 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:40:12.592 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addrs": { 2021-02-10T11:40:12.593 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:40:12.593 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:40:12.593 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:40:12.593 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6823", 2021-02-10T11:40:12.593 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36365 2021-02-10T11:40:12.593 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:40:12.593 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:40:12.593 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:40:12.593 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6825", 2021-02-10T11:40:12.593 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36365 2021-02-10T11:40:12.593 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:40:12.593 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:40:12.593 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:40:12.594 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addrs": { 2021-02-10T11:40:12.594 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:40:12.594 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:40:12.594 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:40:12.594 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6830", 2021-02-10T11:40:12.594 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36365 2021-02-10T11:40:12.594 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:40:12.594 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:40:12.594 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:40:12.594 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6831", 2021-02-10T11:40:12.594 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36365 2021-02-10T11:40:12.594 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:40:12.594 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:40:12.594 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:40:12.595 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addrs": { 2021-02-10T11:40:12.595 INFO:teuthology.orchestra.run.gibba002.stdout: "addrvec": [ 2021-02-10T11:40:12.595 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:40:12.595 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v2", 2021-02-10T11:40:12.595 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6827", 2021-02-10T11:40:12.595 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36365 2021-02-10T11:40:12.595 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:40:12.595 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:40:12.595 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:40:12.595 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.113:6829", 2021-02-10T11:40:12.595 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 36365 2021-02-10T11:40:12.595 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:40:12.595 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:40:12.596 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:40:12.596 INFO:teuthology.orchestra.run.gibba002.stdout: "public_addr": "172.21.2.113:6821/36365", 2021-02-10T11:40:12.596 INFO:teuthology.orchestra.run.gibba002.stdout: "cluster_addr": "172.21.2.113:6825/36365", 2021-02-10T11:40:12.596 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_back_addr": "172.21.2.113:6831/36365", 2021-02-10T11:40:12.596 INFO:teuthology.orchestra.run.gibba002.stdout: "heartbeat_front_addr": "172.21.2.113:6829/36365", 2021-02-10T11:40:12.596 INFO:teuthology.orchestra.run.gibba002.stdout: "state": [ 2021-02-10T11:40:12.596 INFO:teuthology.orchestra.run.gibba002.stdout: "exists", 2021-02-10T11:40:12.596 INFO:teuthology.orchestra.run.gibba002.stdout: "up" 2021-02-10T11:40:12.596 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:40:12.596 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:40:12.596 INFO:teuthology.orchestra.run.gibba002.stdout: ], 2021-02-10T11:40:12.596 INFO:teuthology.orchestra.run.gibba002.stdout: "osd_xinfo": [ 2021-02-10T11:40:12.596 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:40:12.596 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 0, 2021-02-10T11:40:12.597 INFO:teuthology.orchestra.run.gibba002.stdout: "down_stamp": "0.000000", 2021-02-10T11:40:12.597 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_probability": 0, 2021-02-10T11:40:12.597 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_interval": 0, 2021-02-10T11:40:12.597 INFO:teuthology.orchestra.run.gibba002.stdout: "features": 4611087854035861503, 2021-02-10T11:40:12.597 INFO:teuthology.orchestra.run.gibba002.stdout: "old_weight": 0 2021-02-10T11:40:12.597 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:40:12.597 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:40:12.597 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 1, 2021-02-10T11:40:12.597 INFO:teuthology.orchestra.run.gibba002.stdout: "down_stamp": "0.000000", 2021-02-10T11:40:12.597 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_probability": 0, 2021-02-10T11:40:12.597 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_interval": 0, 2021-02-10T11:40:12.597 INFO:teuthology.orchestra.run.gibba002.stdout: "features": 4611087854035861503, 2021-02-10T11:40:12.597 INFO:teuthology.orchestra.run.gibba002.stdout: "old_weight": 0 2021-02-10T11:40:12.597 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:40:12.598 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:40:12.598 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 2, 2021-02-10T11:40:12.598 INFO:teuthology.orchestra.run.gibba002.stdout: "down_stamp": "0.000000", 2021-02-10T11:40:12.598 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_probability": 0, 2021-02-10T11:40:12.598 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_interval": 0, 2021-02-10T11:40:12.598 INFO:teuthology.orchestra.run.gibba002.stdout: "features": 4611087854035861503, 2021-02-10T11:40:12.598 INFO:teuthology.orchestra.run.gibba002.stdout: "old_weight": 0 2021-02-10T11:40:12.598 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:40:12.598 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:40:12.598 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 3, 2021-02-10T11:40:12.598 INFO:teuthology.orchestra.run.gibba002.stdout: "down_stamp": "0.000000", 2021-02-10T11:40:12.598 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_probability": 0, 2021-02-10T11:40:12.598 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_interval": 0, 2021-02-10T11:40:12.599 INFO:teuthology.orchestra.run.gibba002.stdout: "features": 4611087854035861503, 2021-02-10T11:40:12.599 INFO:teuthology.orchestra.run.gibba002.stdout: "old_weight": 0 2021-02-10T11:40:12.599 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:40:12.599 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:40:12.599 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 4, 2021-02-10T11:40:12.599 INFO:teuthology.orchestra.run.gibba002.stdout: "down_stamp": "0.000000", 2021-02-10T11:40:12.599 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_probability": 0, 2021-02-10T11:40:12.599 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_interval": 0, 2021-02-10T11:40:12.599 INFO:teuthology.orchestra.run.gibba002.stdout: "features": 4611087854035861503, 2021-02-10T11:40:12.599 INFO:teuthology.orchestra.run.gibba002.stdout: "old_weight": 0 2021-02-10T11:40:12.599 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:40:12.599 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:40:12.599 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 5, 2021-02-10T11:40:12.599 INFO:teuthology.orchestra.run.gibba002.stdout: "down_stamp": "0.000000", 2021-02-10T11:40:12.600 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_probability": 0, 2021-02-10T11:40:12.600 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_interval": 0, 2021-02-10T11:40:12.600 INFO:teuthology.orchestra.run.gibba002.stdout: "features": 4611087854035861503, 2021-02-10T11:40:12.600 INFO:teuthology.orchestra.run.gibba002.stdout: "old_weight": 0 2021-02-10T11:40:12.600 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:40:12.600 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:40:12.600 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 6, 2021-02-10T11:40:12.600 INFO:teuthology.orchestra.run.gibba002.stdout: "down_stamp": "0.000000", 2021-02-10T11:40:12.600 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_probability": 0, 2021-02-10T11:40:12.600 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_interval": 0, 2021-02-10T11:40:12.600 INFO:teuthology.orchestra.run.gibba002.stdout: "features": 4611087854035861503, 2021-02-10T11:40:12.600 INFO:teuthology.orchestra.run.gibba002.stdout: "old_weight": 0 2021-02-10T11:40:12.600 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:40:12.601 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:40:12.601 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": 7, 2021-02-10T11:40:12.601 INFO:teuthology.orchestra.run.gibba002.stdout: "down_stamp": "0.000000", 2021-02-10T11:40:12.601 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_probability": 0, 2021-02-10T11:40:12.601 INFO:teuthology.orchestra.run.gibba002.stdout: "laggy_interval": 0, 2021-02-10T11:40:12.601 INFO:teuthology.orchestra.run.gibba002.stdout: "features": 4611087854035861503, 2021-02-10T11:40:12.601 INFO:teuthology.orchestra.run.gibba002.stdout: "old_weight": 0 2021-02-10T11:40:12.601 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:40:12.601 INFO:teuthology.orchestra.run.gibba002.stdout: ], 2021-02-10T11:40:12.601 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_upmap": [], 2021-02-10T11:40:12.601 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_upmap_items": [], 2021-02-10T11:40:12.601 INFO:teuthology.orchestra.run.gibba002.stdout: "pg_temp": [], 2021-02-10T11:40:12.601 INFO:teuthology.orchestra.run.gibba002.stdout: "primary_temp": [], 2021-02-10T11:40:12.602 INFO:teuthology.orchestra.run.gibba002.stdout: "blacklist": {}, 2021-02-10T11:40:12.602 INFO:teuthology.orchestra.run.gibba002.stdout: "erasure_code_profiles": { 2021-02-10T11:40:12.602 INFO:teuthology.orchestra.run.gibba002.stdout: "default": { 2021-02-10T11:40:12.603 INFO:teuthology.orchestra.run.gibba002.stdout: "crush-failure-domain": "osd", 2021-02-10T11:40:12.603 INFO:teuthology.orchestra.run.gibba002.stdout: "k": "2", 2021-02-10T11:40:12.603 INFO:teuthology.orchestra.run.gibba002.stdout: "m": "1", 2021-02-10T11:40:12.603 INFO:teuthology.orchestra.run.gibba002.stdout: "plugin": "jerasure", 2021-02-10T11:40:12.603 INFO:teuthology.orchestra.run.gibba002.stdout: "ruleset-failure-domain": "osd", 2021-02-10T11:40:12.603 INFO:teuthology.orchestra.run.gibba002.stdout: "technique": "reed_sol_van" 2021-02-10T11:40:12.603 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:40:12.604 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:40:12.604 INFO:teuthology.orchestra.run.gibba002.stdout: "removed_snaps_queue": [], 2021-02-10T11:40:12.604 INFO:teuthology.orchestra.run.gibba002.stdout: "new_removed_snaps": [], 2021-02-10T11:40:12.604 INFO:teuthology.orchestra.run.gibba002.stdout: "new_purged_snaps": [], 2021-02-10T11:40:12.604 INFO:teuthology.orchestra.run.gibba002.stdout: "crush_node_flags": {}, 2021-02-10T11:40:12.604 INFO:teuthology.orchestra.run.gibba002.stdout: "device_class_flags": {} 2021-02-10T11:40:12.604 INFO:teuthology.orchestra.run.gibba002.stdout:} 2021-02-10T11:40:12.605 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:12.512 7f09be1ad700 1 -- 172.21.2.102:0/664533238 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f099c0186c0 msgr2=0x7f099c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:12.605 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:12.512 7f09be1ad700 1 --2- 172.21.2.102:0/664533238 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f099c0186c0 0x7f099c01abc0 crc :-1 s=READY pgs=1452 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:12.605 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:12.512 7f09be1ad700 1 -- 172.21.2.102:0/664533238 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f09b818ef20 msgr2=0x7f09b80955e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:12.605 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:12.512 7f09be1ad700 1 --2- 172.21.2.102:0/664533238 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f09b818ef20 0x7f09b80955e0 secure :-1 s=READY pgs=3332 cs=0 l=1 rev1=1 rx=0x7f09a8006f00 tx=0x7f09a800b940).stop 2021-02-10T11:40:12.606 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:12.512 7f09be1ad700 1 -- 172.21.2.102:0/664533238 shutdown_connections 2021-02-10T11:40:12.606 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:12.512 7f09be1ad700 1 --2- 172.21.2.102:0/664533238 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f099c0186c0 0x7f099c01abc0 unknown :-1 s=CLOSED pgs=1452 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:12.606 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:12.512 7f09be1ad700 1 --2- 172.21.2.102:0/664533238 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f09b8095c00 0x7f09b8093240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:12.606 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:12.512 7f09be1ad700 1 --2- 172.21.2.102:0/664533238 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f09b818ef20 0x7f09b80955e0 unknown :-1 s=CLOSED pgs=3332 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:12.606 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:12.512 7f09be1ad700 1 --2- 172.21.2.102:0/664533238 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f09b80aed00 0x7f09b8096400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:12.606 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:12.512 7f09be1ad700 1 -- 172.21.2.102:0/664533238 >> 172.21.2.102:0/664533238 conn(0x7f09b81a37a0 msgr2=0x7f09b819d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:12.606 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:12.512 7f09be1ad700 1 -- 172.21.2.102:0/664533238 shutdown_connections 2021-02-10T11:40:12.606 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:12.512 7f09be1ad700 1 -- 172.21.2.102:0/664533238 wait complete. 2021-02-10T11:40:12.607 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:40:13.035 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:13.017 7f8e6b8c3700 1 Processor -- start 2021-02-10T11:40:13.035 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:13.018 7f8e6b8c3700 1 -- start start 2021-02-10T11:40:13.035 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:13.018 7f8e6b8c3700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f8e640d1870 0x7f8e641b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:13.035 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:13.018 7f8e6b8c3700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f8e641b0a10 0x7f8e640cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:13.035 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:13.018 7f8e6b8c3700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8e640d1c70 0x7f8e640cf950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:13.036 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:13.018 7f8e6b8c3700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e640e5ed0 con 0x7f8e640d1870 2021-02-10T11:40:13.036 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:13.018 7f8e6b8c3700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e640f0b10 con 0x7f8e641b0a10 2021-02-10T11:40:13.036 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:13.018 7f8e6b8c3700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e640cff70 con 0x7f8e640d1c70 2021-02-10T11:40:13.036 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:13.018 7f8e696df700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f8e640d1870 0x7f8e641b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:13.036 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:13.018 7f8e696df700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f8e640d1870 0x7f8e641b03f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:40998/0 (socket says 172.21.2.102:40998) 2021-02-10T11:40:13.036 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:13.018 7f8e696df700 1 -- 172.21.2.102:0/3518458581 learned_addr learned my addr 172.21.2.102:0/3518458581 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:40:13.036 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:13.019 7f8e696df700 1 -- 172.21.2.102:0/3518458581 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8e640d1c70 msgr2=0x7f8e640cf950 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:40:13.036 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:13.019 7f8e696df700 1 --2- 172.21.2.102:0/3518458581 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8e640d1c70 0x7f8e640cf950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:13.036 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:13.019 7f8e696df700 1 -- 172.21.2.102:0/3518458581 >> v2:172.21.2.113:3300/0 conn(0x7f8e641b0a10 msgr2=0x7f8e640cf330 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:40:13.036 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:13.019 7f8e696df700 1 --2- 172.21.2.102:0/3518458581 >> v2:172.21.2.113:3300/0 conn(0x7f8e641b0a10 0x7f8e640cf330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:13.036 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:13.019 7f8e696df700 1 -- 172.21.2.102:0/3518458581 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e640d0150 con 0x7f8e640d1870 2021-02-10T11:40:13.036 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:13.019 7f8e696df700 1 --2- 172.21.2.102:0/3518458581 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8e640d1870 0x7f8e641b03f0 secure :-1 s=READY pgs=3333 cs=0 l=1 rev1=1 rx=0x7f8e58006700 tx=0x7f8e58009b80).ready entity=mon.0 client_cookie=a06d768b635016ed server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:13.037 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:13.019 7f8e57fff700 1 -- 172.21.2.102:0/3518458581 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8e5800c0c0 con 0x7f8e640d1870 2021-02-10T11:40:13.037 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:13.019 7f8e57fff700 1 -- 172.21.2.102:0/3518458581 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e5800c340 con 0x7f8e640d1870 2021-02-10T11:40:13.037 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:13.019 7f8e57fff700 1 -- 172.21.2.102:0/3518458581 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8e5800c860 con 0x7f8e640d1870 2021-02-10T11:40:13.037 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:13.019 7f8e6b8c3700 1 -- 172.21.2.102:0/3518458581 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8e640d1870 msgr2=0x7f8e641b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:13.037 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:13.019 7f8e6b8c3700 1 --2- 172.21.2.102:0/3518458581 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8e640d1870 0x7f8e641b03f0 secure :-1 s=READY pgs=3333 cs=0 l=1 rev1=1 rx=0x7f8e58006700 tx=0x7f8e58009b80).stop 2021-02-10T11:40:13.037 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:13.020 7f8e6b8c3700 1 -- 172.21.2.102:0/3518458581 shutdown_connections 2021-02-10T11:40:13.037 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:13.020 7f8e6b8c3700 1 --2- 172.21.2.102:0/3518458581 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8e640d1c70 0x7f8e640cf950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:13.037 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:13.020 7f8e6b8c3700 1 --2- 172.21.2.102:0/3518458581 >> v2:172.21.2.113:3300/0 conn(0x7f8e641b0a10 0x7f8e640cf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:13.037 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:13.020 7f8e6b8c3700 1 --2- 172.21.2.102:0/3518458581 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8e640d1870 0x7f8e641b03f0 unknown :-1 s=CLOSED pgs=3333 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:13.037 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:13.020 7f8e6b8c3700 1 -- 172.21.2.102:0/3518458581 >> 172.21.2.102:0/3518458581 conn(0x7f8e640ce450 msgr2=0x7f8e640d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:13.037 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:13.020 7f8e6b8c3700 1 -- 172.21.2.102:0/3518458581 shutdown_connections 2021-02-10T11:40:13.037 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:13.020 7f8e6b8c3700 1 -- 172.21.2.102:0/3518458581 wait complete. 2021-02-10T11:40:13.039 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:13.034 7f8e6b8c3700 1 Processor -- start 2021-02-10T11:40:13.039 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:13.034 7f8e6b8c3700 1 -- start start 2021-02-10T11:40:13.039 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:13.035 7f8e6b8c3700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8e64093060 0x7f8e6419a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:13.039 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:13.035 7f8e6b8c3700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8e6419aeb0 0x7f8e640b1990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:13.039 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:13.035 7f8e6b8c3700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8e6419b2b0 0x7f8e640b1fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:13.039 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:13.035 7f8e6b8c3700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e640efd70 con 0x7f8e6419b2b0 2021-02-10T11:40:13.039 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:13.035 7f8e6b8c3700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e6419b8a0 con 0x7f8e64093060 2021-02-10T11:40:13.039 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:13.035 7f8e6b8c3700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e640b25d0 con 0x7f8e6419aeb0 2021-02-10T11:40:13.039 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:13.035 7f8e69ee0700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8e6419b2b0 0x7f8e640b1fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:13.039 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:13.035 7f8e69ee0700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8e6419b2b0 0x7f8e640b1fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:41006/0 (socket says 172.21.2.102:41006) 2021-02-10T11:40:13.039 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:13.035 7f8e69ee0700 1 -- 172.21.2.102:0/462082160 learned_addr learned my addr 172.21.2.102:0/462082160 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:40:13.039 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:13.035 7f8e69ee0700 1 -- 172.21.2.102:0/462082160 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8e6419aeb0 msgr2=0x7f8e640b1990 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:40:13.040 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:13.035 7f8e69ee0700 1 --2- 172.21.2.102:0/462082160 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8e6419aeb0 0x7f8e640b1990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:13.040 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:13.035 7f8e69ee0700 1 -- 172.21.2.102:0/462082160 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8e64093060 msgr2=0x7f8e6419a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:40:13.040 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:13.035 7f8e69ee0700 1 --2- 172.21.2.102:0/462082160 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8e64093060 0x7f8e6419a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:13.040 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:13.035 7f8e69ee0700 1 -- 172.21.2.102:0/462082160 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e6418e410 con 0x7f8e6419b2b0 2021-02-10T11:40:13.040 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:13.036 7f8e69ee0700 1 --2- 172.21.2.102:0/462082160 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8e6419b2b0 0x7f8e640b1fb0 secure :-1 s=READY pgs=3334 cs=0 l=1 rev1=1 rx=0x7f8e5c009960 tx=0x7f8e5c00bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:13.040 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:13.036 7f8e55ffb700 1 -- 172.21.2.102:0/462082160 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8e5c003030 con 0x7f8e6419b2b0 2021-02-10T11:40:13.040 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:13.036 7f8e55ffb700 1 -- 172.21.2.102:0/462082160 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e5c001310 con 0x7f8e6419b2b0 2021-02-10T11:40:13.040 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:13.036 7f8e55ffb700 1 -- 172.21.2.102:0/462082160 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8e5c001830 con 0x7f8e6419b2b0 2021-02-10T11:40:13.040 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:13.036 7f8e6b8c3700 1 -- 172.21.2.102:0/462082160 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e6418e410 con 0x7f8e6419b2b0 2021-02-10T11:40:13.040 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:13.036 7f8e6b8c3700 1 -- 172.21.2.102:0/462082160 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e6418e9a0 con 0x7f8e6419b2b0 2021-02-10T11:40:13.040 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:13.036 7f8e55ffb700 1 -- 172.21.2.102:0/462082160 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f8e5c0014f0 con 0x7f8e6419b2b0 2021-02-10T11:40:13.040 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:13.036 7f8e55ffb700 1 --2- 172.21.2.102:0/462082160 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8e440186c0 0x7f8e4401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:13.040 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:13.036 7f8e55ffb700 1 -- 172.21.2.102:0/462082160 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(189..189 src has 1..189) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f8e5c032950 con 0x7f8e6419b2b0 2021-02-10T11:40:13.041 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:13.037 7f8e696df700 1 --2- 172.21.2.102:0/462082160 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8e440186c0 0x7f8e4401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:13.041 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:13.037 7f8e696df700 1 --2- 172.21.2.102:0/462082160 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8e440186c0 0x7f8e4401abc0 crc :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:13.069 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:13.068 7f8e6b8c3700 1 -- 172.21.2.102:0/462082160 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e6418f230 con 0x7f8e6419b2b0 2021-02-10T11:40:13.071 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:13.070 7f8e55ffb700 1 -- 172.21.2.102:0/462082160 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f8e5c000f40 con 0x7f8e6419b2b0 2021-02-10T11:40:13.200 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:13.198 7f8e6b8c3700 1 -- 172.21.2.102:0/462082160 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8e6418f230 con 0x7f8e6419b2b0 2021-02-10T11:40:13.200 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:13.199 7f8e55ffb700 1 -- 172.21.2.102:0/462082160 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 166 v166) v1 ==== 95+0+2104 (secure 0 0 0) 0x7f8e5c0014f0 con 0x7f8e6419b2b0 2021-02-10T11:40:13.201 INFO:teuthology.orchestra.run.gibba002.stderr:dumped fsmap epoch 166 2021-02-10T11:40:13.201 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:40:13.201 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":166,"default_fscid":20,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8998,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.113:6833/479366648","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6832","nonce":479366648},{"type":"v1","addr":"172.21.2.113:6833","nonce":479366648}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":162}],"filesystems":[{"mdsmap":{"epoch":165,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:39:30.375466","modified":"2021-02-10 11:39:32.379360","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7481},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7481":{"gid":7481,"name":"a","rank":0,"incarnation":163,"state":"up:active","state_seq":3,"addr":"172.21.2.102:6835/4290051997","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6834","nonce":4290051997},{"type":"v1","addr":"172.21.2.102:6835","nonce":4290051997}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[41],"metadata_pool":40,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":20}]} 2021-02-10T11:40:13.202 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:13.201 7f8e6b8c3700 1 -- 172.21.2.102:0/462082160 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8e440186c0 msgr2=0x7f8e4401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:13.202 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:13.201 7f8e6b8c3700 1 --2- 172.21.2.102:0/462082160 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8e440186c0 0x7f8e4401abc0 crc :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:13.202 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:13.201 7f8e6b8c3700 1 -- 172.21.2.102:0/462082160 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8e6419b2b0 msgr2=0x7f8e640b1fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:13.203 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:13.201 7f8e6b8c3700 1 --2- 172.21.2.102:0/462082160 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8e6419b2b0 0x7f8e640b1fb0 secure :-1 s=READY pgs=3334 cs=0 l=1 rev1=1 rx=0x7f8e5c009960 tx=0x7f8e5c00bcf0).stop 2021-02-10T11:40:13.203 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:13.201 7f8e6b8c3700 1 -- 172.21.2.102:0/462082160 shutdown_connections 2021-02-10T11:40:13.203 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:13.201 7f8e6b8c3700 1 --2- 172.21.2.102:0/462082160 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8e6419b2b0 0x7f8e640b1fb0 unknown :-1 s=CLOSED pgs=3334 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:13.203 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:13.201 7f8e6b8c3700 1 --2- 172.21.2.102:0/462082160 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8e6419aeb0 0x7f8e640b1990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:13.203 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:13.201 7f8e6b8c3700 1 --2- 172.21.2.102:0/462082160 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8e440186c0 0x7f8e4401abc0 unknown :-1 s=CLOSED pgs=1453 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:13.203 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:13.201 7f8e6b8c3700 1 --2- 172.21.2.102:0/462082160 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8e64093060 0x7f8e6419a890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:13.204 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:13.201 7f8e6b8c3700 1 -- 172.21.2.102:0/462082160 >> 172.21.2.102:0/462082160 conn(0x7f8e6419f880 msgr2=0x7f8e64199a80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:13.204 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:13.201 7f8e6b8c3700 1 -- 172.21.2.102:0/462082160 shutdown_connections 2021-02-10T11:40:13.204 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:13.202 7f8e6b8c3700 1 -- 172.21.2.102:0/462082160 wait complete. 2021-02-10T11:40:13.212 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs fail cephfs 2021-02-10T11:40:13.643 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:13.625 7feb769b9700 1 Processor -- start 2021-02-10T11:40:13.643 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:13.625 7feb769b9700 1 -- start start 2021-02-10T11:40:13.643 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:13.625 7feb769b9700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7feb700d1860 0x7feb701b03e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:13.644 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:13.625 7feb769b9700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7feb701b0a00 0x7feb700cf320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:13.644 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:13.626 7feb769b9700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb700f0b00 con 0x7feb700d1c60 2021-02-10T11:40:13.644 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:13.626 7feb769b9700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb700fe220 con 0x7feb701b0a00 2021-02-10T11:40:13.644 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:13.626 7feb769b9700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb700e5ec0 con 0x7feb700d1860 2021-02-10T11:40:13.644 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:13.626 7feb6f7fe700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7feb701b0a00 0x7feb700cf320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:13.644 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:13.626 7feb74fd6700 1 --1- >> v1:172.21.2.113:6789/0 conn(0x7feb700d1c60 0x7feb700cf940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.113:6789/0 says I am v1:172.21.2.102:40562/0 (socket says 172.21.2.102:40562) 2021-02-10T11:40:13.644 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:13.626 7feb6f7fe700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7feb701b0a00 0x7feb700cf320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.113:3300/0 says I am v2:172.21.2.102:44442/0 (socket says 172.21.2.102:44442) 2021-02-10T11:40:13.644 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:13.626 7feb74fd6700 1 -- 172.21.2.102:0/702439899 learned_addr learned my addr 172.21.2.102:0/702439899 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:40:13.644 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:13.626 7feb6ffff700 1 --2- 172.21.2.102:0/702439899 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7feb700d1860 0x7feb701b03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:13.644 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:13.626 7feb6f7fe700 1 -- 172.21.2.102:0/702439899 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7feb700d1860 msgr2=0x7feb701b03e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:13.644 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:13.626 7feb6f7fe700 1 --2- 172.21.2.102:0/702439899 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7feb700d1860 0x7feb701b03e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:13.644 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:13.626 7feb6f7fe700 1 -- 172.21.2.102:0/702439899 >> v1:172.21.2.113:6789/0 conn(0x7feb700d1c60 legacy=0x7feb700cf940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:13.645 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:13.626 7feb6f7fe700 1 -- 172.21.2.102:0/702439899 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb700f0640 con 0x7feb701b0a00 2021-02-10T11:40:13.645 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:13.627 7feb6f7fe700 1 --2- 172.21.2.102:0/702439899 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7feb701b0a00 0x7feb700cf320 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 rx=0x7feb64006650 tx=0x7feb64009a80).ready entity=mon.1 client_cookie=33f0fe9fd5f632d4 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:13.645 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:13.627 7feb6effd700 1 -- 172.21.2.102:0/702439899 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7feb6400c0c0 con 0x7feb701b0a00 2021-02-10T11:40:13.645 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:13.627 7feb6effd700 1 -- 172.21.2.102:0/702439899 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb6400c340 con 0x7feb701b0a00 2021-02-10T11:40:13.645 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:13.627 7feb6effd700 1 -- 172.21.2.102:0/702439899 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7feb6400c860 con 0x7feb701b0a00 2021-02-10T11:40:13.645 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:13.627 7feb769b9700 1 -- 172.21.2.102:0/702439899 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7feb701b0a00 msgr2=0x7feb700cf320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:13.645 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:13.627 7feb769b9700 1 --2- 172.21.2.102:0/702439899 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7feb701b0a00 0x7feb700cf320 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 rx=0x7feb64006650 tx=0x7feb64009a80).stop 2021-02-10T11:40:13.645 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:13.628 7feb769b9700 1 -- 172.21.2.102:0/702439899 shutdown_connections 2021-02-10T11:40:13.645 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:13.628 7feb769b9700 1 --2- 172.21.2.102:0/702439899 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7feb701b0a00 0x7feb700cf320 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:13.646 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:13.628 7feb769b9700 1 --2- 172.21.2.102:0/702439899 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7feb700d1860 0x7feb701b03e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:13.646 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:13.628 7feb769b9700 1 -- 172.21.2.102:0/702439899 >> 172.21.2.102:0/702439899 conn(0x7feb700ce440 msgr2=0x7feb700d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:13.646 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:13.628 7feb769b9700 1 -- 172.21.2.102:0/702439899 shutdown_connections 2021-02-10T11:40:13.646 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:13.628 7feb769b9700 1 -- 172.21.2.102:0/702439899 wait complete. 2021-02-10T11:40:13.647 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:13.642 7feb769b9700 1 Processor -- start 2021-02-10T11:40:13.647 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:13.642 7feb769b9700 1 -- start start 2021-02-10T11:40:13.647 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:13.643 7feb769b9700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7feb700bc9f0 0x7feb700bcdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:13.647 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:13.643 7feb769b9700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7feb700bd410 0x7feb700bd810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:13.647 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:13.643 7feb769b9700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7feb700c1440 0x7feb700c1840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:13.648 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:13.643 7feb769b9700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb700bc470 con 0x7feb700c1440 2021-02-10T11:40:13.648 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:13.643 7feb769b9700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb700b9cd0 con 0x7feb700bc9f0 2021-02-10T11:40:13.648 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:13.643 7feb769b9700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb700c1e60 con 0x7feb700bd410 2021-02-10T11:40:13.648 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:13.643 7feb74fd6700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7feb700c1440 0x7feb700c1840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:13.648 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:13.643 7feb74fd6700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7feb700c1440 0x7feb700c1840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:41020/0 (socket says 172.21.2.102:41020) 2021-02-10T11:40:13.648 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:13.643 7feb74fd6700 1 -- 172.21.2.102:0/3496247459 learned_addr learned my addr 172.21.2.102:0/3496247459 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:40:13.648 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:13.643 7feb6ffff700 1 --2- 172.21.2.102:0/3496247459 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7feb700bc9f0 0x7feb700bcdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:13.648 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:13.643 7feb6f7fe700 1 --2- 172.21.2.102:0/3496247459 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7feb700bd410 0x7feb700bd810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:13.648 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:13.643 7feb74fd6700 1 -- 172.21.2.102:0/3496247459 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7feb700bd410 msgr2=0x7feb700bd810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:13.648 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:13.643 7feb74fd6700 1 --2- 172.21.2.102:0/3496247459 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7feb700bd410 0x7feb700bd810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:13.648 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:13.643 7feb74fd6700 1 -- 172.21.2.102:0/3496247459 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7feb700bc9f0 msgr2=0x7feb700bcdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:13.648 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:13.643 7feb74fd6700 1 --2- 172.21.2.102:0/3496247459 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7feb700bc9f0 0x7feb700bcdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:13.648 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:13.643 7feb74fd6700 1 -- 172.21.2.102:0/3496247459 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb700c2080 con 0x7feb700c1440 2021-02-10T11:40:13.649 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:13.643 7feb6f7fe700 1 --2- 172.21.2.102:0/3496247459 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7feb700bd410 0x7feb700bd810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:40:13.649 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:13.644 7feb74fd6700 1 --2- 172.21.2.102:0/3496247459 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7feb700c1440 0x7feb700c1840 secure :-1 s=READY pgs=3335 cs=0 l=1 rev1=1 rx=0x7feb68007fc0 tx=0x7feb6800ac80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:13.649 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:13.644 7feb6cff9700 1 -- 172.21.2.102:0/3496247459 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7feb68007020 con 0x7feb700c1440 2021-02-10T11:40:13.649 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:13.644 7feb6cff9700 1 -- 172.21.2.102:0/3496247459 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb68007220 con 0x7feb700c1440 2021-02-10T11:40:13.649 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:13.644 7feb769b9700 1 -- 172.21.2.102:0/3496247459 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb700c2080 con 0x7feb700c1440 2021-02-10T11:40:13.649 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:13.644 7feb6cff9700 1 -- 172.21.2.102:0/3496247459 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7feb680046a0 con 0x7feb700c1440 2021-02-10T11:40:13.649 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:13.644 7feb769b9700 1 -- 172.21.2.102:0/3496247459 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb700c2610 con 0x7feb700c1440 2021-02-10T11:40:13.649 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:13.644 7feb6cff9700 1 -- 172.21.2.102:0/3496247459 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7feb680046a0 con 0x7feb700c1440 2021-02-10T11:40:13.649 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:13.645 7feb6cff9700 1 --2- 172.21.2.102:0/3496247459 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7feb500186c0 0x7feb5001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:13.649 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:13.645 7feb6cff9700 1 -- 172.21.2.102:0/3496247459 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(189..189 src has 1..189) v4 ==== 5573+0+0 (secure 0 0 0) 0x7feb68032750 con 0x7feb700c1440 2021-02-10T11:40:13.649 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:13.645 7feb6ffff700 1 --2- 172.21.2.102:0/3496247459 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7feb500186c0 0x7feb5001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:13.649 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:13.645 7feb6ffff700 1 --2- 172.21.2.102:0/3496247459 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7feb500186c0 0x7feb5001abc0 crc :-1 s=READY pgs=1454 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:13.677 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:13.675 7feb769b9700 1 -- 172.21.2.102:0/3496247459 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb700ebbe0 con 0x7feb700c1440 2021-02-10T11:40:13.679 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:13.678 7feb6cff9700 1 -- 172.21.2.102:0/3496247459 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7feb680011f0 con 0x7feb700c1440 2021-02-10T11:40:13.807 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:13.806 7feb769b9700 1 -- 172.21.2.102:0/3496247459 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7feb700f0b00 con 0x7feb700c1440 2021-02-10T11:40:14.208 INFO:tasks.ceph.mon.a.gibba002.stderr:2021-02-10 11:40:14.205 7f8dd2755700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-02-10T11:40:14.214 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:14.212 7feb6cff9700 1 -- 172.21.2.102:0/3496247459 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v167) v1 ==== 161+0+0 (secure 0 0 0) 0x7feb680046a0 con 0x7feb700c1440 2021-02-10T11:40:14.222 INFO:teuthology.orchestra.run.gibba002.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2021-02-10T11:40:14.224 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:14.222 7feb769b9700 1 -- 172.21.2.102:0/3496247459 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7feb500186c0 msgr2=0x7feb5001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:14.224 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:14.222 7feb769b9700 1 --2- 172.21.2.102:0/3496247459 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7feb500186c0 0x7feb5001abc0 crc :-1 s=READY pgs=1454 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:14.224 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:14.222 7feb769b9700 1 -- 172.21.2.102:0/3496247459 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7feb700c1440 msgr2=0x7feb700c1840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:14.224 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:14.222 7feb769b9700 1 --2- 172.21.2.102:0/3496247459 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7feb700c1440 0x7feb700c1840 secure :-1 s=READY pgs=3335 cs=0 l=1 rev1=1 rx=0x7feb68007fc0 tx=0x7feb6800ac80).stop 2021-02-10T11:40:14.224 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:14.222 7feb769b9700 1 -- 172.21.2.102:0/3496247459 shutdown_connections 2021-02-10T11:40:14.224 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:14.222 7feb769b9700 1 --2- 172.21.2.102:0/3496247459 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7feb700c1440 0x7feb700c1840 unknown :-1 s=CLOSED pgs=3335 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:14.225 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:14.222 7feb769b9700 1 --2- 172.21.2.102:0/3496247459 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7feb700bd410 0x7feb700bd810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:14.225 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:14.222 7feb769b9700 1 --2- 172.21.2.102:0/3496247459 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7feb500186c0 0x7feb5001abc0 unknown :-1 s=CLOSED pgs=1454 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:14.225 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:14.222 7feb769b9700 1 --2- 172.21.2.102:0/3496247459 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7feb700bc9f0 0x7feb700bcdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:14.225 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:14.222 7feb769b9700 1 -- 172.21.2.102:0/3496247459 >> 172.21.2.102:0/3496247459 conn(0x7feb700c2c50 msgr2=0x7feb70095700 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:14.225 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:14.222 7feb769b9700 1 -- 172.21.2.102:0/3496247459 shutdown_connections 2021-02-10T11:40:14.225 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:14.222 7feb769b9700 1 -- 172.21.2.102:0/3496247459 wait complete. 2021-02-10T11:40:14.233 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:40:14.269 INFO:tasks.ceph.mds.a.gibba002.stdout:starting mds.a at 2021-02-10T11:40:14.663 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:14.643 7f6e0d954700 1 Processor -- start 2021-02-10T11:40:14.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:14.643 7f6e0d954700 1 -- start start 2021-02-10T11:40:14.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:14.643 7f6e0d954700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f6e080d1870 0x7f6e081b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:14.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:14.643 7f6e0d954700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6e081b0a10 0x7f6e080cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:14.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:14.643 7f6e0d954700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e080f0b10 con 0x7f6e080d1c70 2021-02-10T11:40:14.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:14.643 7f6e0d954700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e080e5ed0 con 0x7f6e080d1870 2021-02-10T11:40:14.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:14.643 7f6e0d954700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e080f0650 con 0x7f6e081b0a10 2021-02-10T11:40:14.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:14.644 7f6e07574700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f6e080d1870 0x7f6e081b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:14.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:14.644 7f6e07574700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f6e080d1870 0x7f6e081b03f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.113:3300/0 says I am v2:172.21.2.102:44466/0 (socket says 172.21.2.102:44466) 2021-02-10T11:40:14.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:14.644 7f6e07574700 1 -- 172.21.2.102:0/3119023106 learned_addr learned my addr 172.21.2.102:0/3119023106 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:40:14.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:14.644 7f6e06d73700 1 --2- 172.21.2.102:0/3119023106 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6e081b0a10 0x7f6e080cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:14.664 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:14.645 7f6e07574700 1 -- 172.21.2.102:0/3119023106 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6e081b0a10 msgr2=0x7f6e080cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:14.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:14.645 7f6e07574700 1 --2- 172.21.2.102:0/3119023106 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6e081b0a10 0x7f6e080cf330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:14.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:14.645 7f6e07574700 1 -- 172.21.2.102:0/3119023106 >> v1:172.21.2.113:6789/0 conn(0x7f6e080d1c70 legacy=0x7f6e080cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:14.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:14.645 7f6e07574700 1 -- 172.21.2.102:0/3119023106 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e080f0870 con 0x7f6e080d1870 2021-02-10T11:40:14.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:14.645 7f6e06d73700 1 --2- 172.21.2.102:0/3119023106 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6e081b0a10 0x7f6e080cf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:40:14.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:14.646 7f6e07574700 1 --2- 172.21.2.102:0/3119023106 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6e080d1870 0x7f6e081b03f0 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 rx=0x7f6df0006a80 tx=0x7f6df0009f00).ready entity=mon.1 client_cookie=b6c3c40201e6773d server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:14.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:14.646 7f6e06572700 1 -- 172.21.2.102:0/3119023106 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6df00066c0 con 0x7f6e080d1870 2021-02-10T11:40:14.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:14.646 7f6e06572700 1 -- 172.21.2.102:0/3119023106 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6df000b5c0 con 0x7f6e080d1870 2021-02-10T11:40:14.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:14.646 7f6e06572700 1 -- 172.21.2.102:0/3119023106 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6df000ba10 con 0x7f6e080d1870 2021-02-10T11:40:14.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:14.646 7f6e0d954700 1 -- 172.21.2.102:0/3119023106 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6e080d1870 msgr2=0x7f6e081b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:14.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:14.646 7f6e0d954700 1 --2- 172.21.2.102:0/3119023106 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6e080d1870 0x7f6e081b03f0 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 rx=0x7f6df0006a80 tx=0x7f6df0009f00).stop 2021-02-10T11:40:14.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:14.647 7f6e0d954700 1 -- 172.21.2.102:0/3119023106 shutdown_connections 2021-02-10T11:40:14.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:14.647 7f6e0d954700 1 --2- 172.21.2.102:0/3119023106 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6e081b0a10 0x7f6e080cf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:14.665 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:14.647 7f6e0d954700 1 --2- 172.21.2.102:0/3119023106 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6e080d1870 0x7f6e081b03f0 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:14.666 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:14.647 7f6e0d954700 1 -- 172.21.2.102:0/3119023106 >> 172.21.2.102:0/3119023106 conn(0x7f6e080ce450 msgr2=0x7f6e080d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:14.666 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:14.647 7f6e0d954700 1 -- 172.21.2.102:0/3119023106 shutdown_connections 2021-02-10T11:40:14.666 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:14.647 7f6e0d954700 1 -- 172.21.2.102:0/3119023106 wait complete. 2021-02-10T11:40:14.668 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:14.662 7f6e0d954700 1 Processor -- start 2021-02-10T11:40:14.668 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:14.662 7f6e0d954700 1 -- start start 2021-02-10T11:40:14.668 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:14.662 7f6e0d954700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6e080ada90 0x7f6e080ade90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:14.668 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:14.662 7f6e0d954700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6e08099d20 0x7f6e0809a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:14.668 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:14.662 7f6e0d954700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6e0809a740 0x7f6e08097bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:14.668 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:14.662 7f6e0d954700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e080997a0 con 0x7f6e0809a740 2021-02-10T11:40:14.668 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:14.662 7f6e0d954700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e0818f0c0 con 0x7f6e080ada90 2021-02-10T11:40:14.668 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:14.662 7f6e0d954700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e0809ab40 con 0x7f6e08099d20 2021-02-10T11:40:14.668 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:14.663 7f6e07d75700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6e0809a740 0x7f6e08097bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:14.668 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:14.663 7f6e07d75700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6e0809a740 0x7f6e08097bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:41040/0 (socket says 172.21.2.102:41040) 2021-02-10T11:40:14.669 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:14.663 7f6e07d75700 1 -- 172.21.2.102:0/1355544794 learned_addr learned my addr 172.21.2.102:0/1355544794 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:40:14.669 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:14.663 7f6e07574700 1 --2- 172.21.2.102:0/1355544794 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6e080ada90 0x7f6e080ade90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:14.669 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:14.663 7f6e06d73700 1 --2- 172.21.2.102:0/1355544794 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6e08099d20 0x7f6e0809a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:14.669 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:14.663 7f6e07574700 1 -- 172.21.2.102:0/1355544794 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6e08099d20 msgr2=0x7f6e0809a120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:14.669 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:14.663 7f6e07574700 1 --2- 172.21.2.102:0/1355544794 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6e08099d20 0x7f6e0809a120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:14.669 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:14.663 7f6e07574700 1 -- 172.21.2.102:0/1355544794 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6e0809a740 msgr2=0x7f6e08097bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:14.669 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:14.663 7f6e07574700 1 --2- 172.21.2.102:0/1355544794 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6e0809a740 0x7f6e08097bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:14.669 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:14.663 7f6e07574700 1 -- 172.21.2.102:0/1355544794 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e0809ad60 con 0x7f6e080ada90 2021-02-10T11:40:14.669 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:14.663 7f6e07d75700 1 --2- 172.21.2.102:0/1355544794 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6e0809a740 0x7f6e08097bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:40:14.669 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:14.663 7f6e07574700 1 --2- 172.21.2.102:0/1355544794 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6e080ada90 0x7f6e080ade90 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 rx=0x7f6df0009960 tx=0x7f6df000bc70).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:14.669 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:14.664 7f6dfffff700 1 -- 172.21.2.102:0/1355544794 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6df000c2f0 con 0x7f6e080ada90 2021-02-10T11:40:14.669 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:14.664 7f6dfffff700 1 -- 172.21.2.102:0/1355544794 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6df000c5f0 con 0x7f6e080ada90 2021-02-10T11:40:14.669 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:14.664 7f6dfffff700 1 -- 172.21.2.102:0/1355544794 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6df000cb10 con 0x7f6e080ada90 2021-02-10T11:40:14.670 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:14.664 7f6e0d954700 1 -- 172.21.2.102:0/1355544794 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e0809ad60 con 0x7f6e080ada90 2021-02-10T11:40:14.670 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:14.664 7f6e0d954700 1 -- 172.21.2.102:0/1355544794 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e0809ad60 con 0x7f6e080ada90 2021-02-10T11:40:14.670 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:14.665 7f6dfffff700 1 -- 172.21.2.102:0/1355544794 <== mon.1 v2:172.21.2.113:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f6df0001010 con 0x7f6e080ada90 2021-02-10T11:40:14.670 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:14.665 7f6dfffff700 1 --2- 172.21.2.102:0/1355544794 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6de00186c0 0x7f6de001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:14.670 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:14.665 7f6dfffff700 1 -- 172.21.2.102:0/1355544794 <== mon.1 v2:172.21.2.113:3300/0 5 ==== osd_map(190..190 src has 1..190) v4 ==== 5659+0+0 (secure 0 0 0) 0x7f6df0032840 con 0x7f6e080ada90 2021-02-10T11:40:14.670 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:14.665 7f6e06d73700 1 --2- 172.21.2.102:0/1355544794 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6de00186c0 0x7f6de001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:14.670 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:14.665 7f6e06d73700 1 --2- 172.21.2.102:0/1355544794 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6de00186c0 0x7f6de001abc0 crc :-1 s=READY pgs=1455 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:14.696 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:14.693 7f6e0d954700 1 -- 172.21.2.102:0/1355544794 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e080ebc00 con 0x7f6e080ada90 2021-02-10T11:40:14.700 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:14.698 7f6dfffff700 1 -- 172.21.2.102:0/1355544794 <== mon.1 v2:172.21.2.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f6df000bdb0 con 0x7f6e080ada90 2021-02-10T11:40:14.832 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:14.830 7f6e0d954700 1 -- 172.21.2.102:0/1355544794 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6e080ebc00 con 0x7f6e080ada90 2021-02-10T11:40:14.833 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:14.830 7f6dfffff700 1 -- 172.21.2.102:0/1355544794 <== mon.1 v2:172.21.2.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 167 v167) v1 ==== 95+0+1759 (secure 0 0 0) 0x7f6df0001010 con 0x7f6e080ada90 2021-02-10T11:40:14.833 INFO:teuthology.orchestra.run.gibba002.stderr:dumped fsmap epoch 167 2021-02-10T11:40:14.833 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:40:14.833 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":167,"default_fscid":20,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8998,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.113:6833/479366648","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6832","nonce":479366648},{"type":"v1","addr":"172.21.2.113:6833","nonce":479366648}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":162}],"filesystems":[{"mdsmap":{"epoch":167,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:39:30.375466","modified":"2021-02-10 11:40:14.207001","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":190,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[41],"metadata_pool":40,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":20}]} 2021-02-10T11:40:14.835 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:14.832 7f6e0d954700 1 -- 172.21.2.102:0/1355544794 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6de00186c0 msgr2=0x7f6de001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:14.835 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:14.832 7f6e0d954700 1 --2- 172.21.2.102:0/1355544794 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6de00186c0 0x7f6de001abc0 crc :-1 s=READY pgs=1455 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:14.835 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:14.832 7f6e0d954700 1 -- 172.21.2.102:0/1355544794 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6e080ada90 msgr2=0x7f6e080ade90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:14.835 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:14.832 7f6e0d954700 1 --2- 172.21.2.102:0/1355544794 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6e080ada90 0x7f6e080ade90 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 rx=0x7f6df0009960 tx=0x7f6df000bc70).stop 2021-02-10T11:40:14.835 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:14.832 7f6e0d954700 1 -- 172.21.2.102:0/1355544794 shutdown_connections 2021-02-10T11:40:14.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:14.832 7f6e0d954700 1 --2- 172.21.2.102:0/1355544794 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6e0809a740 0x7f6e08097bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:14.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:14.832 7f6e0d954700 1 --2- 172.21.2.102:0/1355544794 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6e08099d20 0x7f6e0809a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:14.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:14.832 7f6e0d954700 1 --2- 172.21.2.102:0/1355544794 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6de00186c0 0x7f6de001abc0 unknown :-1 s=CLOSED pgs=1455 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:14.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:14.832 7f6e0d954700 1 --2- 172.21.2.102:0/1355544794 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6e080ada90 0x7f6e080ade90 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:14.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:14.832 7f6e0d954700 1 -- 172.21.2.102:0/1355544794 >> 172.21.2.102:0/1355544794 conn(0x7f6e080f0550 msgr2=0x7f6e08191730 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:14.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:14.833 7f6e0d954700 1 -- 172.21.2.102:0/1355544794 shutdown_connections 2021-02-10T11:40:14.836 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:14.833 7f6e0d954700 1 -- 172.21.2.102:0/1355544794 wait complete. 2021-02-10T11:40:14.843 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2021-02-10T11:40:15.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:15.298 7f1191a7f700 1 Processor -- start 2021-02-10T11:40:15.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:15.298 7f1191a7f700 1 -- start start 2021-02-10T11:40:15.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:15.298 7f1191a7f700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f118c0d1c70 0x7f118c0f20a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:15.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:15.298 7f1191a7f700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f118c0f0b10 con 0x7f118c0d1870 2021-02-10T11:40:15.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:15.298 7f1191a7f700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f118c1b0be0 con 0x7f118c0d0340 2021-02-10T11:40:15.318 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:15.298 7f1191a7f700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f118c0e5ed0 con 0x7f118c0d1c70 2021-02-10T11:40:15.318 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:15.298 7f118b7fe700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f118c0d1870 0x7f118c1b03f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:33272/0 (socket says 172.21.2.102:33272) 2021-02-10T11:40:15.318 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:15.298 7f118b7fe700 1 -- 172.21.2.102:0/282468841 learned_addr learned my addr 172.21.2.102:0/282468841 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:40:15.318 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:15.299 7f118bfff700 1 --2- 172.21.2.102:0/282468841 >> v2:172.21.2.113:3300/0 conn(0x7f118c0d1c70 0x7f118c0f20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:15.318 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:15.299 7f118a7fc700 1 -- 172.21.2.102:0/282468841 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3410643572 0 0) 0x7f1184000ce0 con 0x7f118c0d1870 2021-02-10T11:40:15.318 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:15.299 7f118a7fc700 1 -- 172.21.2.102:0/282468841 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1178002fe0 con 0x7f118c0d1870 2021-02-10T11:40:15.318 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:15.299 7f118bfff700 1 -- 172.21.2.102:0/282468841 >> v1:172.21.2.113:6789/0 conn(0x7f118c0d0340 legacy=0x7f118c0ef640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:15.318 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:15.299 7f118bfff700 1 -- 172.21.2.102:0/282468841 >> v1:172.21.2.102:6789/0 conn(0x7f118c0d1870 legacy=0x7f118c1b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:15.318 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:15.299 7f118bfff700 1 -- 172.21.2.102:0/282468841 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f118c0d3360 con 0x7f118c0d1c70 2021-02-10T11:40:15.318 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:15.299 7f118bfff700 1 --2- 172.21.2.102:0/282468841 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f118c0d1c70 0x7f118c0f20a0 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 rx=0x7f117c006e50 tx=0x7f117c009e70).ready entity=mon.1 client_cookie=15ceb59814ad9df5 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:15.318 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:15.299 7f118a7fc700 1 -- 172.21.2.102:0/282468841 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f117c006a20 con 0x7f118c0d1c70 2021-02-10T11:40:15.318 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:15.299 7f118a7fc700 1 -- 172.21.2.102:0/282468841 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f117c006790 con 0x7f118c0d1c70 2021-02-10T11:40:15.319 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:15.299 7f118a7fc700 1 -- 172.21.2.102:0/282468841 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f117c00b700 con 0x7f118c0d1c70 2021-02-10T11:40:15.319 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:15.299 7f1191a7f700 1 -- 172.21.2.102:0/282468841 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f118c0d1c70 msgr2=0x7f118c0f20a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:15.319 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:15.299 7f1191a7f700 1 --2- 172.21.2.102:0/282468841 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f118c0d1c70 0x7f118c0f20a0 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 rx=0x7f117c006e50 tx=0x7f117c009e70).stop 2021-02-10T11:40:15.319 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:15.300 7f1191a7f700 1 -- 172.21.2.102:0/282468841 shutdown_connections 2021-02-10T11:40:15.319 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:15.300 7f1191a7f700 1 --2- 172.21.2.102:0/282468841 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f118c0d1c70 0x7f118c0f20a0 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:15.319 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:15.300 7f1191a7f700 1 -- 172.21.2.102:0/282468841 >> 172.21.2.102:0/282468841 conn(0x7f118c0ce450 msgr2=0x7f118c0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:15.319 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:15.300 7f1191a7f700 1 -- 172.21.2.102:0/282468841 shutdown_connections 2021-02-10T11:40:15.319 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:15.300 7f1191a7f700 1 -- 172.21.2.102:0/282468841 wait complete. 2021-02-10T11:40:15.319 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:15.315 7f1191a7f700 1 Processor -- start 2021-02-10T11:40:15.319 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:15.315 7f1191a7f700 1 -- start start 2021-02-10T11:40:15.321 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:15.315 7f1191a7f700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f118c09bf50 0x7f118c09c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:15.321 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:15.315 7f1191a7f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f118c09c350 0x7f118c0ac5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:15.321 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:15.315 7f1191a7f700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f118c09cfc0 0x7f118c0acbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:15.321 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:15.315 7f1191a7f700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f118c0ad210 con 0x7f118c09c350 2021-02-10T11:40:15.321 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:15.315 7f1191a7f700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f118c0ad400 con 0x7f118c09bf50 2021-02-10T11:40:15.321 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:15.315 7f1191a7f700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f118c0ad620 con 0x7f118c09cfc0 2021-02-10T11:40:15.321 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:15.315 7f118affd700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f118c09c350 0x7f118c0ac5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:15.321 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:15.316 7f118b7fe700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f118c09bf50 0x7f118c09c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:15.321 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:15.316 7f118bfff700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f118c09cfc0 0x7f118c0acbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:15.321 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:15.316 7f118affd700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f118c09c350 0x7f118c0ac5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:41054/0 (socket says 172.21.2.102:41054) 2021-02-10T11:40:15.321 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:15.316 7f118affd700 1 -- 172.21.2.102:0/3470002769 learned_addr learned my addr 172.21.2.102:0/3470002769 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:40:15.322 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:15.316 7f118affd700 1 -- 172.21.2.102:0/3470002769 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f118c09cfc0 msgr2=0x7f118c0acbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:15.322 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:15.316 7f118affd700 1 --2- 172.21.2.102:0/3470002769 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f118c09cfc0 0x7f118c0acbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:15.322 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:15.316 7f118affd700 1 -- 172.21.2.102:0/3470002769 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f118c09bf50 msgr2=0x7f118c09c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:15.322 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:15.316 7f118affd700 1 --2- 172.21.2.102:0/3470002769 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f118c09bf50 0x7f118c09c9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:15.322 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:15.316 7f118affd700 1 -- 172.21.2.102:0/3470002769 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f118c0ada80 con 0x7f118c09c350 2021-02-10T11:40:15.322 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:15.316 7f118affd700 1 --2- 172.21.2.102:0/3470002769 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f118c09c350 0x7f118c0ac5d0 secure :-1 s=READY pgs=3340 cs=0 l=1 rev1=1 rx=0x7f117400b5b0 tx=0x7f117400b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:15.322 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:15.316 7f1183fff700 1 -- 172.21.2.102:0/3470002769 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f117400bd40 con 0x7f118c09c350 2021-02-10T11:40:15.322 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:15.316 7f1183fff700 1 -- 172.21.2.102:0/3470002769 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1174006020 con 0x7f118c09c350 2021-02-10T11:40:15.322 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:15.316 7f1183fff700 1 -- 172.21.2.102:0/3470002769 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f11740054a0 con 0x7f118c09c350 2021-02-10T11:40:15.323 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:15.316 7f1191a7f700 1 -- 172.21.2.102:0/3470002769 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f118c0ada80 con 0x7f118c09c350 2021-02-10T11:40:15.323 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:15.316 7f1191a7f700 1 -- 172.21.2.102:0/3470002769 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f118c0ada80 con 0x7f118c09c350 2021-02-10T11:40:15.323 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:15.317 7f1183fff700 1 -- 172.21.2.102:0/3470002769 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1174006200 con 0x7f118c09c350 2021-02-10T11:40:15.323 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:15.317 7f1183fff700 1 --2- 172.21.2.102:0/3470002769 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1170018730 0x7f117001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:15.323 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:15.317 7f1183fff700 1 -- 172.21.2.102:0/3470002769 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(190..190 src has 1..190) v4 ==== 5659+0+0 (secure 0 0 0) 0x7f1174032680 con 0x7f118c09c350 2021-02-10T11:40:15.323 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:15.317 7f118b7fe700 1 --2- 172.21.2.102:0/3470002769 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1170018730 0x7f117001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:15.323 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:15.317 7f118b7fe700 1 --2- 172.21.2.102:0/3470002769 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1170018730 0x7f117001ac10 crc :-1 s=READY pgs=1456 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:15.350 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:15.348 7f1191a7f700 1 -- 172.21.2.102:0/3470002769 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f118c0ce450 con 0x7f118c09c350 2021-02-10T11:40:15.353 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:15.350 7f1183fff700 1 -- 172.21.2.102:0/3470002769 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f117400bd40 con 0x7f118c09c350 2021-02-10T11:40:15.482 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:15.479 7f1191a7f700 1 -- 172.21.2.102:0/3470002769 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f118c0ce450 con 0x7f118c09c350 2021-02-10T11:40:15.968 DEBUG:teuthology.orchestra.run.gibba002:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:40:15.971 DEBUG:teuthology.orchestra.run.gibba013:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:40:16.219 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:16.217 7f1183fff700 1 -- 172.21.2.102:0/3470002769 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v168) v1 ==== 104+0+0 (secure 0 0 0) 0x7f1174006020 con 0x7f118c09c350 2021-02-10T11:40:16.253 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:16.249 7f1191a7f700 1 -- 172.21.2.102:0/3470002769 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1170018730 msgr2=0x7f117001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:16.253 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:16.249 7f1191a7f700 1 --2- 172.21.2.102:0/3470002769 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1170018730 0x7f117001ac10 crc :-1 s=READY pgs=1456 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:16.253 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:16.249 7f1191a7f700 1 -- 172.21.2.102:0/3470002769 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f118c09c350 msgr2=0x7f118c0ac5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:16.253 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:16.249 7f1191a7f700 1 --2- 172.21.2.102:0/3470002769 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f118c09c350 0x7f118c0ac5d0 secure :-1 s=READY pgs=3340 cs=0 l=1 rev1=1 rx=0x7f117400b5b0 tx=0x7f117400b950).stop 2021-02-10T11:40:16.253 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:16.249 7f1191a7f700 1 -- 172.21.2.102:0/3470002769 shutdown_connections 2021-02-10T11:40:16.254 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:16.249 7f1191a7f700 1 --2- 172.21.2.102:0/3470002769 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f118c09cfc0 0x7f118c0acbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:16.254 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:16.249 7f1191a7f700 1 --2- 172.21.2.102:0/3470002769 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f118c09c350 0x7f118c0ac5d0 unknown :-1 s=CLOSED pgs=3340 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:16.254 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:16.249 7f1191a7f700 1 --2- 172.21.2.102:0/3470002769 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1170018730 0x7f117001ac10 unknown :-1 s=CLOSED pgs=1456 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:16.254 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:16.249 7f1191a7f700 1 --2- 172.21.2.102:0/3470002769 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f118c09bf50 0x7f118c09c9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:16.254 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:16.249 7f1191a7f700 1 -- 172.21.2.102:0/3470002769 >> 172.21.2.102:0/3470002769 conn(0x7f118c0a6620 msgr2=0x7f118c0b17a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:16.255 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:16.250 7f1191a7f700 1 -- 172.21.2.102:0/3470002769 shutdown_connections 2021-02-10T11:40:16.255 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:16.250 7f1191a7f700 1 -- 172.21.2.102:0/3470002769 wait complete. 2021-02-10T11:40:16.261 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool delete cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2021-02-10T11:40:16.697 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:16.675 7f5a45d9e700 1 Processor -- start 2021-02-10T11:40:16.697 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:16.675 7f5a45d9e700 1 -- start start 2021-02-10T11:40:16.697 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:16.676 7f5a45d9e700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f5a400d1880 0x7f5a401b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:16.697 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:16.676 7f5a45d9e700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5a401b0af0 0x7f5a40108950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:16.698 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:16.676 7f5a45d9e700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a400fe1d0 con 0x7f5a400d1c80 2021-02-10T11:40:16.698 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:16.676 7f5a45d9e700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a40109a70 con 0x7f5a400d1880 2021-02-10T11:40:16.698 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:16.676 7f5a45d9e700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a40109760 con 0x7f5a401b0af0 2021-02-10T11:40:16.698 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:16.676 7f5a3f7fe700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f5a400d1880 0x7f5a401b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:16.698 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:16.676 7f5a3ffff700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f5a400d1c80 0x7f5a40108f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:33286/0 (socket says 172.21.2.102:33286) 2021-02-10T11:40:16.698 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:16.676 7f5a3ffff700 1 -- 172.21.2.102:0/2637371565 learned_addr learned my addr 172.21.2.102:0/2637371565 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:40:16.699 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:16.676 7f5a3f7fe700 1 -- 172.21.2.102:0/2637371565 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5a401b0af0 msgr2=0x7f5a40108950 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:40:16.699 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:16.676 7f5a3f7fe700 1 --2- 172.21.2.102:0/2637371565 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5a401b0af0 0x7f5a40108950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:16.699 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:16.676 7f5a3f7fe700 1 -- 172.21.2.102:0/2637371565 >> v1:172.21.2.102:6789/0 conn(0x7f5a400d1c80 legacy=0x7f5a40108f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:16.699 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:16.676 7f5a3f7fe700 1 -- 172.21.2.102:0/2637371565 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a40088070 con 0x7f5a400d1880 2021-02-10T11:40:16.699 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:16.676 7f5a3f7fe700 1 --2- 172.21.2.102:0/2637371565 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5a400d1880 0x7f5a401b04d0 secure :-1 s=READY pgs=3342 cs=0 l=1 rev1=1 rx=0x7f5a30006650 tx=0x7f5a30009a80).ready entity=mon.0 client_cookie=c30067d093410c48 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:16.699 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:16.677 7f5a3e7fc700 1 -- 172.21.2.102:0/2637371565 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5a300009b0 con 0x7f5a400d1880 2021-02-10T11:40:16.700 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:16.677 7f5a3e7fc700 1 -- 172.21.2.102:0/2637371565 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a3000c0c0 con 0x7f5a400d1880 2021-02-10T11:40:16.700 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:16.677 7f5a3e7fc700 1 -- 172.21.2.102:0/2637371565 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5a3000c5e0 con 0x7f5a400d1880 2021-02-10T11:40:16.700 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:16.677 7f5a45d9e700 1 -- 172.21.2.102:0/2637371565 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5a400d1880 msgr2=0x7f5a401b04d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:16.700 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:16.677 7f5a45d9e700 1 --2- 172.21.2.102:0/2637371565 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5a400d1880 0x7f5a401b04d0 secure :-1 s=READY pgs=3342 cs=0 l=1 rev1=1 rx=0x7f5a30006650 tx=0x7f5a30009a80).stop 2021-02-10T11:40:16.700 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:16.677 7f5a45d9e700 1 -- 172.21.2.102:0/2637371565 shutdown_connections 2021-02-10T11:40:16.701 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:16.677 7f5a45d9e700 1 --2- 172.21.2.102:0/2637371565 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5a401b0af0 0x7f5a40108950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:16.701 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:16.677 7f5a45d9e700 1 --2- 172.21.2.102:0/2637371565 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5a400d1880 0x7f5a401b04d0 unknown :-1 s=CLOSED pgs=3342 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:16.701 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:16.677 7f5a45d9e700 1 -- 172.21.2.102:0/2637371565 >> 172.21.2.102:0/2637371565 conn(0x7f5a400ce430 msgr2=0x7f5a400d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:16.701 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:16.677 7f5a45d9e700 1 -- 172.21.2.102:0/2637371565 shutdown_connections 2021-02-10T11:40:16.701 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:16.677 7f5a45d9e700 1 -- 172.21.2.102:0/2637371565 wait complete. 2021-02-10T11:40:16.701 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:16.692 7f5a45d9e700 1 Processor -- start 2021-02-10T11:40:16.702 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:16.692 7f5a45d9e700 1 -- start start 2021-02-10T11:40:16.702 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:16.692 7f5a45d9e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5a4009ef20 0x7f5a4018d430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:16.702 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:16.692 7f5a45d9e700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5a4008eae0 0x7f5a4008eee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:16.702 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:16.692 7f5a45d9e700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5a4008b0e0 0x7f5a4008b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:16.702 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:16.692 7f5a45d9e700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a4008f500 con 0x7f5a4009ef20 2021-02-10T11:40:16.703 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:16.692 7f5a45d9e700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a4009f320 con 0x7f5a4008b0e0 2021-02-10T11:40:16.703 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:16.692 7f5a45d9e700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a4009f540 con 0x7f5a4008eae0 2021-02-10T11:40:16.703 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:16.693 7f5a3f7fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5a4009ef20 0x7f5a4018d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:16.703 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:16.693 7f5a3f7fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5a4009ef20 0x7f5a4018d430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:41070/0 (socket says 172.21.2.102:41070) 2021-02-10T11:40:16.703 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:16.693 7f5a3f7fe700 1 -- 172.21.2.102:0/3259570660 learned_addr learned my addr 172.21.2.102:0/3259570660 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:40:16.703 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:16.693 7f5a3f7fe700 1 -- 172.21.2.102:0/3259570660 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5a4008eae0 msgr2=0x7f5a4008eee0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:40:16.703 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:16.693 7f5a3f7fe700 1 --2- 172.21.2.102:0/3259570660 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5a4008eae0 0x7f5a4008eee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:16.704 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:16.693 7f5a3f7fe700 1 -- 172.21.2.102:0/3259570660 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5a4008b0e0 msgr2=0x7f5a4008b4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:40:16.704 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:16.693 7f5a3f7fe700 1 --2- 172.21.2.102:0/3259570660 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5a4008b0e0 0x7f5a4008b4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:16.704 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:16.693 7f5a3f7fe700 1 -- 172.21.2.102:0/3259570660 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a4008bb00 con 0x7f5a4009ef20 2021-02-10T11:40:16.704 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:16.693 7f5a3f7fe700 1 --2- 172.21.2.102:0/3259570660 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5a4009ef20 0x7f5a4018d430 secure :-1 s=READY pgs=3343 cs=0 l=1 rev1=1 rx=0x7f5a30009630 tx=0x7f5a3000b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:16.704 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:16.693 7f5a27fff700 1 -- 172.21.2.102:0/3259570660 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5a300093a0 con 0x7f5a4009ef20 2021-02-10T11:40:16.705 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:16.693 7f5a27fff700 1 -- 172.21.2.102:0/3259570660 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a3000bd70 con 0x7f5a4009ef20 2021-02-10T11:40:16.705 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:16.693 7f5a45d9e700 1 -- 172.21.2.102:0/3259570660 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a4008bb00 con 0x7f5a4009ef20 2021-02-10T11:40:16.705 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:16.693 7f5a45d9e700 1 -- 172.21.2.102:0/3259570660 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a4008c090 con 0x7f5a4009ef20 2021-02-10T11:40:16.705 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:16.693 7f5a27fff700 1 -- 172.21.2.102:0/3259570660 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5a30005560 con 0x7f5a4009ef20 2021-02-10T11:40:16.705 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:16.694 7f5a27fff700 1 -- 172.21.2.102:0/3259570660 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f5a3000a000 con 0x7f5a4009ef20 2021-02-10T11:40:16.706 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:16.694 7f5a27fff700 1 --2- 172.21.2.102:0/3259570660 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5a200186c0 0x7f5a2001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:16.706 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:16.694 7f5a27fff700 1 -- 172.21.2.102:0/3259570660 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(190..190 src has 1..190) v4 ==== 5659+0+0 (secure 0 0 0) 0x7f5a30032850 con 0x7f5a4009ef20 2021-02-10T11:40:16.706 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:16.694 7f5a3effd700 1 --2- 172.21.2.102:0/3259570660 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5a200186c0 0x7f5a2001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:16.706 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:16.694 7f5a3effd700 1 --2- 172.21.2.102:0/3259570660 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5a200186c0 0x7f5a2001abc0 crc :-1 s=READY pgs=1457 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:16.728 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:16.726 7f5a45d9e700 1 -- 172.21.2.102:0/3259570660 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a4008e5b0 con 0x7f5a4009ef20 2021-02-10T11:40:16.731 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:16.729 7f5a27fff700 1 -- 172.21.2.102:0/3259570660 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f5a30009340 con 0x7f5a4009ef20 2021-02-10T11:40:16.857 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:16.855 7f5a45d9e700 1 -- 172.21.2.102:0/3259570660 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"pool2": "cephfs_metadata", "prefix": "osd pool delete", "pool": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f5a4008e5b0 con 0x7f5a4009ef20 2021-02-10T11:40:17.223 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:17.220 7f5a27fff700 1 -- 172.21.2.102:0/3259570660 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"pool2": "cephfs_metadata", "prefix": "osd pool delete", "pool": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v191) v1 ==== 185+0+0 (secure 0 0 0) 0x7f5a3001f540 con 0x7f5a4009ef20 2021-02-10T11:40:17.223 INFO:teuthology.orchestra.run.gibba002.stderr:pool 'cephfs_metadata' removed 2021-02-10T11:40:17.226 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:17.223 7f5a45d9e700 1 -- 172.21.2.102:0/3259570660 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5a200186c0 msgr2=0x7f5a2001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:17.226 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:17.223 7f5a45d9e700 1 --2- 172.21.2.102:0/3259570660 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5a200186c0 0x7f5a2001abc0 crc :-1 s=READY pgs=1457 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:17.226 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:17.223 7f5a45d9e700 1 -- 172.21.2.102:0/3259570660 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5a4009ef20 msgr2=0x7f5a4018d430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:17.226 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:17.223 7f5a45d9e700 1 --2- 172.21.2.102:0/3259570660 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5a4009ef20 0x7f5a4018d430 secure :-1 s=READY pgs=3343 cs=0 l=1 rev1=1 rx=0x7f5a30009630 tx=0x7f5a3000b980).stop 2021-02-10T11:40:17.226 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:17.223 7f5a45d9e700 1 -- 172.21.2.102:0/3259570660 shutdown_connections 2021-02-10T11:40:17.226 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:17.223 7f5a45d9e700 1 --2- 172.21.2.102:0/3259570660 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5a200186c0 0x7f5a2001abc0 unknown :-1 s=CLOSED pgs=1457 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:17.226 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:17.223 7f5a45d9e700 1 --2- 172.21.2.102:0/3259570660 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5a4008b0e0 0x7f5a4008b4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:17.226 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:17.223 7f5a45d9e700 1 --2- 172.21.2.102:0/3259570660 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5a4008eae0 0x7f5a4008eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:17.226 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:17.223 7f5a45d9e700 1 --2- 172.21.2.102:0/3259570660 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5a4009ef20 0x7f5a4018d430 unknown :-1 s=CLOSED pgs=3343 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:17.226 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:17.223 7f5a45d9e700 1 -- 172.21.2.102:0/3259570660 >> 172.21.2.102:0/3259570660 conn(0x7f5a4019f970 msgr2=0x7f5a401a61d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:17.227 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:17.223 7f5a45d9e700 1 -- 172.21.2.102:0/3259570660 shutdown_connections 2021-02-10T11:40:17.227 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:17.223 7f5a45d9e700 1 -- 172.21.2.102:0/3259570660 wait complete. 2021-02-10T11:40:17.235 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool delete cephfs_data cephfs_data --yes-i-really-really-mean-it 2021-02-10T11:40:17.666 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:17.647 7f849decd700 1 Processor -- start 2021-02-10T11:40:17.666 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:17.647 7f849decd700 1 -- start start 2021-02-10T11:40:17.666 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:17.648 7f849decd700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f84980d1880 0x7f84981b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:17.667 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:17.648 7f849decd700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84980fe1d0 con 0x7f84980cff80 2021-02-10T11:40:17.667 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:17.648 7f849decd700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8498109a70 con 0x7f84981b0af0 2021-02-10T11:40:17.667 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:17.648 7f849decd700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8498088070 con 0x7f84980d1880 2021-02-10T11:40:17.667 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:17.648 7f84977fe700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f84980d1880 0x7f84981b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:17.667 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:17.648 7f8497fff700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f84980cff80 0x7f8498109140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:33300/0 (socket says 172.21.2.102:33300) 2021-02-10T11:40:17.667 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:17.648 7f8497fff700 1 -- 172.21.2.102:0/1880317817 learned_addr learned my addr 172.21.2.102:0/1880317817 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:40:17.667 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:17.648 7f84977fe700 1 -- 172.21.2.102:0/1880317817 >> v1:172.21.2.113:6789/0 conn(0x7f84981b0af0 legacy=0x7f8498108950 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:40:17.667 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:17.648 7f84977fe700 1 -- 172.21.2.102:0/1880317817 >> v1:172.21.2.102:6789/0 conn(0x7f84980cff80 legacy=0x7f8498109140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:17.667 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:17.648 7f84967fc700 1 -- 172.21.2.102:0/1880317817 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2490869248 0 0) 0x7f8490001500 con 0x7f84980cff80 2021-02-10T11:40:17.667 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:17.648 7f84977fe700 1 -- 172.21.2.102:0/1880317817 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8498088260 con 0x7f84980d1880 2021-02-10T11:40:17.668 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:17.649 7f84977fe700 1 --2- 172.21.2.102:0/1880317817 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f84980d1880 0x7f84981b04d0 secure :-1 s=READY pgs=3345 cs=0 l=1 rev1=1 rx=0x7f8480006650 tx=0x7f8480009a80).ready entity=mon.0 client_cookie=78e8724ae5af2ef2 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:17.668 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:17.649 7f84967fc700 1 -- 172.21.2.102:0/1880317817 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f84800009b0 con 0x7f84980d1880 2021-02-10T11:40:17.668 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:17.649 7f84967fc700 1 -- 172.21.2.102:0/1880317817 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f848000c0c0 con 0x7f84980d1880 2021-02-10T11:40:17.668 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:17.649 7f84967fc700 1 -- 172.21.2.102:0/1880317817 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f848000c520 con 0x7f84980d1880 2021-02-10T11:40:17.668 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:17.649 7f849decd700 1 -- 172.21.2.102:0/1880317817 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f84980d1880 msgr2=0x7f84981b04d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:17.668 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:17.649 7f849decd700 1 --2- 172.21.2.102:0/1880317817 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f84980d1880 0x7f84981b04d0 secure :-1 s=READY pgs=3345 cs=0 l=1 rev1=1 rx=0x7f8480006650 tx=0x7f8480009a80).stop 2021-02-10T11:40:17.668 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:17.649 7f849decd700 1 -- 172.21.2.102:0/1880317817 shutdown_connections 2021-02-10T11:40:17.668 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:17.649 7f849decd700 1 --2- 172.21.2.102:0/1880317817 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f84980d1880 0x7f84981b04d0 unknown :-1 s=CLOSED pgs=3345 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:17.668 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:17.649 7f849decd700 1 -- 172.21.2.102:0/1880317817 >> 172.21.2.102:0/1880317817 conn(0x7f84980ce430 msgr2=0x7f84980d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:17.668 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:17.650 7f849decd700 1 -- 172.21.2.102:0/1880317817 shutdown_connections 2021-02-10T11:40:17.668 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:17.650 7f849decd700 1 -- 172.21.2.102:0/1880317817 wait complete. 2021-02-10T11:40:17.670 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:17.664 7f849decd700 1 Processor -- start 2021-02-10T11:40:17.671 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:17.664 7f849decd700 1 -- start start 2021-02-10T11:40:17.671 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:17.665 7f849decd700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f849808fb40 0x7f8498195720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:17.671 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:17.665 7f849decd700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f849809b4f0 0x7f849809b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:17.671 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:17.665 7f849decd700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f849809bf10 0x7f84981940a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:17.671 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:17.665 7f849decd700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84981951a0 con 0x7f849809b4f0 2021-02-10T11:40:17.671 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:17.665 7f849decd700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8498094990 con 0x7f849808fb40 2021-02-10T11:40:17.671 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:17.665 7f849decd700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84981946c0 con 0x7f849809bf10 2021-02-10T11:40:17.671 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:17.665 7f8496ffd700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f849809b4f0 0x7f849809b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:17.671 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:17.665 7f84977fe700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f849808fb40 0x7f8498195720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:17.671 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:17.665 7f8496ffd700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f849809b4f0 0x7f849809b8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:41086/0 (socket says 172.21.2.102:41086) 2021-02-10T11:40:17.671 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:17.665 7f8496ffd700 1 -- 172.21.2.102:0/1452838328 learned_addr learned my addr 172.21.2.102:0/1452838328 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:40:17.672 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:17.665 7f8497fff700 1 --2- 172.21.2.102:0/1452838328 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f849809bf10 0x7f84981940a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:17.672 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:17.665 7f8496ffd700 1 -- 172.21.2.102:0/1452838328 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f849809bf10 msgr2=0x7f84981940a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:17.672 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:17.665 7f8496ffd700 1 --2- 172.21.2.102:0/1452838328 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f849809bf10 0x7f84981940a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:17.672 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:17.665 7f8496ffd700 1 -- 172.21.2.102:0/1452838328 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f849808fb40 msgr2=0x7f8498195720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:17.672 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:17.665 7f8496ffd700 1 --2- 172.21.2.102:0/1452838328 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f849808fb40 0x7f8498195720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:17.672 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:17.665 7f8496ffd700 1 -- 172.21.2.102:0/1452838328 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84981948e0 con 0x7f849809b4f0 2021-02-10T11:40:17.672 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:17.666 7f84977fe700 1 --2- 172.21.2.102:0/1452838328 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f849808fb40 0x7f8498195720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:40:17.672 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:17.666 7f8496ffd700 1 --2- 172.21.2.102:0/1452838328 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f849809b4f0 0x7f849809b8f0 secure :-1 s=READY pgs=3346 cs=0 l=1 rev1=1 rx=0x7f848800b5b0 tx=0x7f848800b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:17.672 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:17.666 7f848ffff700 1 -- 172.21.2.102:0/1452838328 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f848800bd40 con 0x7f849809b4f0 2021-02-10T11:40:17.672 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:17.666 7f848ffff700 1 -- 172.21.2.102:0/1452838328 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8488006020 con 0x7f849809b4f0 2021-02-10T11:40:17.672 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:17.666 7f849decd700 1 -- 172.21.2.102:0/1452838328 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84981948e0 con 0x7f849809b4f0 2021-02-10T11:40:17.672 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:17.666 7f848ffff700 1 -- 172.21.2.102:0/1452838328 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f84880054a0 con 0x7f849809b4f0 2021-02-10T11:40:17.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:17.666 7f849decd700 1 -- 172.21.2.102:0/1452838328 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8498194e70 con 0x7f849809b4f0 2021-02-10T11:40:17.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:17.667 7f848ffff700 1 -- 172.21.2.102:0/1452838328 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f8488006200 con 0x7f849809b4f0 2021-02-10T11:40:17.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:17.667 7f848ffff700 1 --2- 172.21.2.102:0/1452838328 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f847c0186c0 0x7f847c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:17.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:17.667 7f848ffff700 1 -- 172.21.2.102:0/1452838328 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(191..191 src has 1..191) v4 ==== 5221+0+0 (secure 0 0 0) 0x7f84880322c0 con 0x7f849809b4f0 2021-02-10T11:40:17.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:17.667 7f84977fe700 1 --2- 172.21.2.102:0/1452838328 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f847c0186c0 0x7f847c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:17.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:17.668 7f84977fe700 1 --2- 172.21.2.102:0/1452838328 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f847c0186c0 0x7f847c01abc0 crc :-1 s=READY pgs=1458 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:17.700 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:17.698 7f849decd700 1 -- 172.21.2.102:0/1452838328 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8498082ab0 con 0x7f849809b4f0 2021-02-10T11:40:17.704 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:17.700 7f848ffff700 1 -- 172.21.2.102:0/1452838328 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f84880264c0 con 0x7f849809b4f0 2021-02-10T11:40:17.829 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:17.826 7f849decd700 1 -- 172.21.2.102:0/1452838328 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"pool2": "cephfs_data", "prefix": "osd pool delete", "pool": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f8498082ab0 con 0x7f849809b4f0 2021-02-10T11:40:18.229 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:18.226 7f848ffff700 1 -- 172.21.2.102:0/1452838328 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"pool2": "cephfs_data", "prefix": "osd pool delete", "pool": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v192) v1 ==== 173+0+0 (secure 0 0 0) 0x7f84880264c0 con 0x7f849809b4f0 2021-02-10T11:40:18.245 INFO:teuthology.orchestra.run.gibba002.stderr:pool 'cephfs_data' removed 2021-02-10T11:40:18.247 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:18.245 7f849decd700 1 -- 172.21.2.102:0/1452838328 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f847c0186c0 msgr2=0x7f847c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:18.247 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:18.245 7f849decd700 1 --2- 172.21.2.102:0/1452838328 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f847c0186c0 0x7f847c01abc0 crc :-1 s=READY pgs=1458 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:18.247 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:18.245 7f849decd700 1 -- 172.21.2.102:0/1452838328 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f849809b4f0 msgr2=0x7f849809b8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:18.247 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:18.245 7f849decd700 1 --2- 172.21.2.102:0/1452838328 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f849809b4f0 0x7f849809b8f0 secure :-1 s=READY pgs=3346 cs=0 l=1 rev1=1 rx=0x7f848800b5b0 tx=0x7f848800b950).stop 2021-02-10T11:40:18.247 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:18.245 7f849decd700 1 -- 172.21.2.102:0/1452838328 shutdown_connections 2021-02-10T11:40:18.248 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:18.245 7f849decd700 1 --2- 172.21.2.102:0/1452838328 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f849809bf10 0x7f84981940a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:18.248 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:18.245 7f849decd700 1 --2- 172.21.2.102:0/1452838328 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f849809b4f0 0x7f849809b8f0 unknown :-1 s=CLOSED pgs=3346 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:18.248 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:18.245 7f849decd700 1 --2- 172.21.2.102:0/1452838328 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f847c0186c0 0x7f847c01abc0 unknown :-1 s=CLOSED pgs=1458 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:18.248 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:18.245 7f849decd700 1 --2- 172.21.2.102:0/1452838328 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f849808fb40 0x7f8498195720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:18.248 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:18.245 7f849decd700 1 -- 172.21.2.102:0/1452838328 >> 172.21.2.102:0/1452838328 conn(0x7f849809afc0 msgr2=0x7f84980953e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:18.248 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:18.245 7f849decd700 1 -- 172.21.2.102:0/1452838328 shutdown_connections 2021-02-10T11:40:18.248 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:18.245 7f849decd700 1 -- 172.21.2.102:0/1452838328 wait complete. 2021-02-10T11:40:18.256 INFO:tasks.ceph.mds.a:Restarting daemon 2021-02-10T11:40:18.256 INFO:tasks.ceph.mds.a:Stopping old one... 2021-02-10T11:40:18.256 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2021-02-10T11:40:18.256 INFO:teuthology.orchestra.run:waiting for 300 2021-02-10T11:40:18.256 INFO:tasks.ceph.mds.b:Restarting daemon 2021-02-10T11:40:18.256 INFO:tasks.ceph.mds.b:Stopping old one... 2021-02-10T11:40:18.257 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2021-02-10T11:40:18.257 INFO:teuthology.orchestra.run:waiting for 300 2021-02-10T11:40:18.262 INFO:tasks.ceph.mds.a:Stopped 2021-02-10T11:40:18.263 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2021-02-10T11:40:18.263 INFO:tasks.ceph.mds.b:Stopped 2021-02-10T11:40:18.264 DEBUG:teuthology.orchestra.run.gibba013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2021-02-10T11:40:18.307 INFO:tasks.ceph.mds.a:Started 2021-02-10T11:40:18.307 DEBUG:teuthology.parallel:result is None 2021-02-10T11:40:18.309 INFO:tasks.ceph.mds.b:Started 2021-02-10T11:40:18.309 DEBUG:teuthology.parallel:result is None 2021-02-10T11:40:18.310 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd blacklist clear 2021-02-10T11:40:18.714 INFO:tasks.ceph.mds.b.gibba013.stdout:starting mds.b at 2021-02-10T11:40:18.715 INFO:tasks.ceph.mds.a.gibba002.stdout:starting mds.a at 2021-02-10T11:40:18.735 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:18.716 7f89c08a1700 1 Processor -- start 2021-02-10T11:40:18.735 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:18.716 7f89c08a1700 1 -- start start 2021-02-10T11:40:18.735 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:18.716 7f89c08a1700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f89bc0d1870 0x7f89bc1b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:18.735 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:18.716 7f89c08a1700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f89bc1b0a10 0x7f89bc0cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:18.736 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:18.716 7f89c08a1700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89bc0f0b10 con 0x7f89bc0d1c70 2021-02-10T11:40:18.736 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:18.716 7f89c08a1700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89bc0e5ed0 con 0x7f89bc0d1870 2021-02-10T11:40:18.736 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:18.716 7f89c08a1700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89bc0f0650 con 0x7f89bc1b0a10 2021-02-10T11:40:18.736 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:18.717 7f89ba61c700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f89bc0d1870 0x7f89bc1b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:18.736 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:18.717 7f89ba61c700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f89bc0d1870 0x7f89bc1b03f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:41100/0 (socket says 172.21.2.102:41100) 2021-02-10T11:40:18.736 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:18.717 7f89ba61c700 1 -- 172.21.2.102:0/1667268417 learned_addr learned my addr 172.21.2.102:0/1667268417 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:40:18.736 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:18.717 7f89b9e1b700 1 --2- 172.21.2.102:0/1667268417 >> v2:172.21.2.113:3300/0 conn(0x7f89bc1b0a10 0x7f89bc0cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:18.736 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:18.717 7f89ba61c700 1 -- 172.21.2.102:0/1667268417 >> v2:172.21.2.113:3300/0 conn(0x7f89bc1b0a10 msgr2=0x7f89bc0cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:18.736 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:18.717 7f89ba61c700 1 --2- 172.21.2.102:0/1667268417 >> v2:172.21.2.113:3300/0 conn(0x7f89bc1b0a10 0x7f89bc0cf330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:18.736 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:18.717 7f89ba61c700 1 -- 172.21.2.102:0/1667268417 >> v1:172.21.2.113:6789/0 conn(0x7f89bc0d1c70 legacy=0x7f89bc0cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:18.736 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:18.717 7f89ba61c700 1 -- 172.21.2.102:0/1667268417 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89bc0f0840 con 0x7f89bc0d1870 2021-02-10T11:40:18.737 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:18.717 7f89ba61c700 1 --2- 172.21.2.102:0/1667268417 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f89bc0d1870 0x7f89bc1b03f0 secure :-1 s=READY pgs=3350 cs=0 l=1 rev1=1 rx=0x7f89b0006700 tx=0x7f89b0009b80).ready entity=mon.0 client_cookie=2131510354b69936 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:18.737 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:18.717 7f89b961a700 1 -- 172.21.2.102:0/1667268417 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f89b000c0c0 con 0x7f89bc0d1870 2021-02-10T11:40:18.737 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:18.717 7f89b961a700 1 -- 172.21.2.102:0/1667268417 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89b000c380 con 0x7f89bc0d1870 2021-02-10T11:40:18.737 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:18.717 7f89b961a700 1 -- 172.21.2.102:0/1667268417 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f89b000c8c0 con 0x7f89bc0d1870 2021-02-10T11:40:18.737 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:18.717 7f89c08a1700 1 -- 172.21.2.102:0/1667268417 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f89bc0d1870 msgr2=0x7f89bc1b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:18.737 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:18.717 7f89c08a1700 1 --2- 172.21.2.102:0/1667268417 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f89bc0d1870 0x7f89bc1b03f0 secure :-1 s=READY pgs=3350 cs=0 l=1 rev1=1 rx=0x7f89b0006700 tx=0x7f89b0009b80).stop 2021-02-10T11:40:18.737 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:18.718 7f89c08a1700 1 -- 172.21.2.102:0/1667268417 shutdown_connections 2021-02-10T11:40:18.737 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:18.718 7f89c08a1700 1 --2- 172.21.2.102:0/1667268417 >> v2:172.21.2.113:3300/0 conn(0x7f89bc1b0a10 0x7f89bc0cf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:18.737 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:18.718 7f89c08a1700 1 --2- 172.21.2.102:0/1667268417 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f89bc0d1870 0x7f89bc1b03f0 unknown :-1 s=CLOSED pgs=3350 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:18.737 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:18.718 7f89c08a1700 1 -- 172.21.2.102:0/1667268417 >> 172.21.2.102:0/1667268417 conn(0x7f89bc0ce450 msgr2=0x7f89bc0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:18.737 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:18.718 7f89c08a1700 1 -- 172.21.2.102:0/1667268417 shutdown_connections 2021-02-10T11:40:18.738 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:18.718 7f89c08a1700 1 -- 172.21.2.102:0/1667268417 wait complete. 2021-02-10T11:40:18.739 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:18.733 7f89c08a1700 1 Processor -- start 2021-02-10T11:40:18.739 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:18.733 7f89c08a1700 1 -- start start 2021-02-10T11:40:18.739 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:18.734 7f89c08a1700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f89bc0c0f50 0x7f89bc1a4420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:18.739 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:18.734 7f89c08a1700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f89bc1a4a40 0x7f89bc0afac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:18.739 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:18.734 7f89c08a1700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f89bc0c37f0 0x7f89bc0b0c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:18.740 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:18.734 7f89c08a1700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89bc0c13d0 con 0x7f89bc0c0f50 2021-02-10T11:40:18.740 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:18.734 7f89c08a1700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89bc0b00e0 con 0x7f89bc1a4a40 2021-02-10T11:40:18.740 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:18.734 7f89c08a1700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89bc0b0300 con 0x7f89bc0c37f0 2021-02-10T11:40:18.740 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:18.734 7f89ba61c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f89bc0c0f50 0x7f89bc1a4420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:18.740 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:18.734 7f89ba61c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f89bc0c0f50 0x7f89bc1a4420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:41108/0 (socket says 172.21.2.102:41108) 2021-02-10T11:40:18.740 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:18.734 7f89ba61c700 1 -- 172.21.2.102:0/2342709744 learned_addr learned my addr 172.21.2.102:0/2342709744 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:40:18.740 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:18.734 7f89bae1d700 1 --2- 172.21.2.102:0/2342709744 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f89bc0c37f0 0x7f89bc0b0c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:18.740 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:18.734 7f89b9e1b700 1 --2- 172.21.2.102:0/2342709744 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f89bc1a4a40 0x7f89bc0afac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:18.740 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:18.734 7f89ba61c700 1 -- 172.21.2.102:0/2342709744 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f89bc0c37f0 msgr2=0x7f89bc0b0c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:18.740 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:18.734 7f89ba61c700 1 --2- 172.21.2.102:0/2342709744 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f89bc0c37f0 0x7f89bc0b0c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:18.740 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:18.734 7f89ba61c700 1 -- 172.21.2.102:0/2342709744 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f89bc1a4a40 msgr2=0x7f89bc0afac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:18.740 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:18.734 7f89ba61c700 1 --2- 172.21.2.102:0/2342709744 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f89bc1a4a40 0x7f89bc0afac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:18.741 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:18.734 7f89ba61c700 1 -- 172.21.2.102:0/2342709744 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89bc0b1220 con 0x7f89bc0c0f50 2021-02-10T11:40:18.741 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:18.734 7f89ba61c700 1 --2- 172.21.2.102:0/2342709744 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f89bc0c0f50 0x7f89bc1a4420 secure :-1 s=READY pgs=3351 cs=0 l=1 rev1=1 rx=0x7f89b000b5e0 tx=0x7f89b000b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:18.741 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:18.735 7f89aaffd700 1 -- 172.21.2.102:0/2342709744 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f89b000bd70 con 0x7f89bc0c0f50 2021-02-10T11:40:18.741 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:18.735 7f89aaffd700 1 -- 172.21.2.102:0/2342709744 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89b0009440 con 0x7f89bc0c0f50 2021-02-10T11:40:18.741 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:18.735 7f89aaffd700 1 -- 172.21.2.102:0/2342709744 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f89b0002300 con 0x7f89bc0c0f50 2021-02-10T11:40:18.741 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:18.735 7f89c08a1700 1 -- 172.21.2.102:0/2342709744 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89bc0b1220 con 0x7f89bc0c0f50 2021-02-10T11:40:18.741 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:18.735 7f89c08a1700 1 -- 172.21.2.102:0/2342709744 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89bc0b17b0 con 0x7f89bc0c0f50 2021-02-10T11:40:18.741 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:18.735 7f89aaffd700 1 -- 172.21.2.102:0/2342709744 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f89b0000c90 con 0x7f89bc0c0f50 2021-02-10T11:40:18.741 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:18.735 7f89aaffd700 1 --2- 172.21.2.102:0/2342709744 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f89a00186c0 0x7f89a001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:18.741 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:18.735 7f89aaffd700 1 -- 172.21.2.102:0/2342709744 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(192..192 src has 1..192) v4 ==== 4839+0+0 (secure 0 0 0) 0x7f89b00322f0 con 0x7f89bc0c0f50 2021-02-10T11:40:18.741 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:18.735 7f89b9e1b700 1 --2- 172.21.2.102:0/2342709744 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f89a00186c0 0x7f89a001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:18.741 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:18.736 7f89b9e1b700 1 --2- 172.21.2.102:0/2342709744 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f89a00186c0 0x7f89a001abc0 crc :-1 s=READY pgs=1459 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:18.767 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:18.765 7f89c08a1700 1 -- 172.21.2.102:0/2342709744 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89bc0ebc90 con 0x7f89bc0c0f50 2021-02-10T11:40:18.770 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:18.768 7f89aaffd700 1 -- 172.21.2.102:0/2342709744 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f89b00256f0 con 0x7f89bc0c0f50 2021-02-10T11:40:18.910 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:18.907 7f89c08a1700 1 -- 172.21.2.102:0/2342709744 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd blacklist clear"} v 0) v1 -- 0x7f89bc0f0b10 con 0x7f89bc0c0f50 2021-02-10T11:40:19.231 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:19.229 7f89aaffd700 1 -- 172.21.2.102:0/2342709744 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd blacklist clear"}]=0 removed all blacklist entries v193) v1 ==== 97+0+0 (secure 0 0 0) 0x7f89b00256f0 con 0x7f89bc0c0f50 2021-02-10T11:40:19.281 INFO:teuthology.orchestra.run.gibba002.stderr: removed all blacklist entries 2021-02-10T11:40:19.282 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:19.276 7f89c08a1700 1 -- 172.21.2.102:0/2342709744 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f89a00186c0 msgr2=0x7f89a001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:19.282 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:19.277 7f89c08a1700 1 --2- 172.21.2.102:0/2342709744 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f89a00186c0 0x7f89a001abc0 crc :-1 s=READY pgs=1459 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:19.282 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:19.277 7f89c08a1700 1 -- 172.21.2.102:0/2342709744 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f89bc0c0f50 msgr2=0x7f89bc1a4420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:19.282 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:19.277 7f89c08a1700 1 --2- 172.21.2.102:0/2342709744 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f89bc0c0f50 0x7f89bc1a4420 secure :-1 s=READY pgs=3351 cs=0 l=1 rev1=1 rx=0x7f89b000b5e0 tx=0x7f89b000b980).stop 2021-02-10T11:40:19.282 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:19.277 7f89c08a1700 1 -- 172.21.2.102:0/2342709744 shutdown_connections 2021-02-10T11:40:19.282 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:19.277 7f89c08a1700 1 --2- 172.21.2.102:0/2342709744 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f89bc0c37f0 0x7f89bc0b0c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:19.282 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:19.277 7f89c08a1700 1 --2- 172.21.2.102:0/2342709744 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f89a00186c0 0x7f89a001abc0 unknown :-1 s=CLOSED pgs=1459 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:19.282 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:19.277 7f89c08a1700 1 --2- 172.21.2.102:0/2342709744 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f89bc1a4a40 0x7f89bc0afac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:19.282 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:19.277 7f89c08a1700 1 --2- 172.21.2.102:0/2342709744 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f89bc0c0f50 0x7f89bc1a4420 unknown :-1 s=CLOSED pgs=3351 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:19.283 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:19.277 7f89c08a1700 1 -- 172.21.2.102:0/2342709744 >> 172.21.2.102:0/2342709744 conn(0x7f89bc1900d0 msgr2=0x7f89bc09c9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:19.283 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:19.277 7f89c08a1700 1 -- 172.21.2.102:0/2342709744 shutdown_connections 2021-02-10T11:40:19.283 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:19.277 7f89c08a1700 1 -- 172.21.2.102:0/2342709744 wait complete. 2021-02-10T11:40:19.287 INFO:tasks.cephfs.cephfs_test_case:['0', '1', '2', '3'] 2021-02-10T11:40:19.287 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth ls --format=json-pretty 2021-02-10T11:40:19.720 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:19.701 7f77d773e700 1 Processor -- start 2021-02-10T11:40:19.720 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:19.701 7f77d773e700 1 -- start start 2021-02-10T11:40:19.720 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:19.702 7f77d773e700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f77d00d1870 0x7f77d01b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:19.720 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:19.702 7f77d773e700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f77d00f0650 0x7f77d00cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:19.720 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:19.702 7f77d773e700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77d00f0b10 con 0x7f77d01b0a10 2021-02-10T11:40:19.721 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:19.702 7f77d773e700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77d00fe230 con 0x7f77d00f0650 2021-02-10T11:40:19.721 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:19.702 7f77d773e700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77d00e5ed0 con 0x7f77d00d1870 2021-02-10T11:40:19.721 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:19.702 7f77d555a700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f77d00d1870 0x7f77d01b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:19.721 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:19.702 7f77d555a700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f77d00d1870 0x7f77d01b03f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.113:3301/0 says I am v2:172.21.2.102:46650/0 (socket says 172.21.2.102:46650) 2021-02-10T11:40:19.721 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:19.702 7f77d555a700 1 -- 172.21.2.102:0/4242868366 learned_addr learned my addr 172.21.2.102:0/4242868366 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:40:19.721 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:19.702 7f77d5d5b700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f77d00f0650 0x7f77d00cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:19.721 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:19.702 7f77d4d59700 1 --1- 172.21.2.102:0/4242868366 >> v1:172.21.2.102:6789/0 conn(0x7f77d01b0a10 0x7f77d00cf330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:33338/0 (socket says 172.21.2.102:33338) 2021-02-10T11:40:19.721 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:19.702 7f77d555a700 1 -- 172.21.2.102:0/4242868366 >> v2:172.21.2.113:3300/0 conn(0x7f77d00f0650 msgr2=0x7f77d00cfb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:19.721 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:19.702 7f77d555a700 1 --2- 172.21.2.102:0/4242868366 >> v2:172.21.2.113:3300/0 conn(0x7f77d00f0650 0x7f77d00cfb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:19.721 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:19.702 7f77d555a700 1 -- 172.21.2.102:0/4242868366 >> v1:172.21.2.102:6789/0 conn(0x7f77d01b0a10 legacy=0x7f77d00cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:19.721 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:19.702 7f77d555a700 1 -- 172.21.2.102:0/4242868366 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77d00f20a0 con 0x7f77d00d1870 2021-02-10T11:40:19.721 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:19.703 7f77d555a700 1 --2- 172.21.2.102:0/4242868366 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f77d00d1870 0x7f77d01b03f0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 rx=0x7f77c8006a80 tx=0x7f77c8009e80).ready entity=mon.2 client_cookie=dfc117e67d0f5222 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:19.721 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:19.703 7f77c7fff700 1 -- 172.21.2.102:0/4242868366 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f77c800b510 con 0x7f77d00d1870 2021-02-10T11:40:19.721 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:19.703 7f77c7fff700 1 -- 172.21.2.102:0/4242868366 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77c800b7d0 con 0x7f77d00d1870 2021-02-10T11:40:19.722 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:19.703 7f77d773e700 1 -- 172.21.2.102:0/4242868366 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f77d00d1870 msgr2=0x7f77d01b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:19.722 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:19.703 7f77d773e700 1 --2- 172.21.2.102:0/4242868366 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f77d00d1870 0x7f77d01b03f0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 rx=0x7f77c8006a80 tx=0x7f77c8009e80).stop 2021-02-10T11:40:19.722 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:19.703 7f77c7fff700 1 -- 172.21.2.102:0/4242868366 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f77c8000d30 con 0x7f77d00d1870 2021-02-10T11:40:19.722 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:19.703 7f77d773e700 1 -- 172.21.2.102:0/4242868366 shutdown_connections 2021-02-10T11:40:19.722 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:19.703 7f77d773e700 1 --2- 172.21.2.102:0/4242868366 >> v2:172.21.2.113:3300/0 conn(0x7f77d00f0650 0x7f77d00cfb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:19.722 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:19.703 7f77d773e700 1 --2- 172.21.2.102:0/4242868366 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f77d00d1870 0x7f77d01b03f0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:19.722 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:19.703 7f77d773e700 1 -- 172.21.2.102:0/4242868366 >> 172.21.2.102:0/4242868366 conn(0x7f77d00ce450 msgr2=0x7f77d00d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:19.722 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:19.703 7f77d773e700 1 -- 172.21.2.102:0/4242868366 shutdown_connections 2021-02-10T11:40:19.722 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:19.703 7f77d773e700 1 -- 172.21.2.102:0/4242868366 wait complete. 2021-02-10T11:40:19.724 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:19.718 7f77d773e700 1 Processor -- start 2021-02-10T11:40:19.724 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:19.718 7f77d773e700 1 -- start start 2021-02-10T11:40:19.724 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:19.719 7f77d773e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f77d00ada90 0x7f77d00ade90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:19.724 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:19.719 7f77d773e700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f77d0099d20 0x7f77d009a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:19.724 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:19.719 7f77d773e700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f77d009a740 0x7f77d0097bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:19.724 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:19.719 7f77d773e700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77d00997a0 con 0x7f77d00ada90 2021-02-10T11:40:19.724 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:19.719 7f77d773e700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77d018f0c0 con 0x7f77d0099d20 2021-02-10T11:40:19.724 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:19.719 7f77d773e700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77d009ab40 con 0x7f77d009a740 2021-02-10T11:40:19.724 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:19.719 7f77d555a700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f77d00ada90 0x7f77d00ade90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:19.724 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:19.719 7f77d555a700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f77d00ada90 0x7f77d00ade90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:41118/0 (socket says 172.21.2.102:41118) 2021-02-10T11:40:19.725 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:19.719 7f77d555a700 1 -- 172.21.2.102:0/1503502427 learned_addr learned my addr 172.21.2.102:0/1503502427 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:40:19.725 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:19.719 7f77d5d5b700 1 --2- 172.21.2.102:0/1503502427 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f77d009a740 0x7f77d0097bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:19.725 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:19.719 7f77d4d59700 1 --2- 172.21.2.102:0/1503502427 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f77d0099d20 0x7f77d009a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:19.725 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:19.719 7f77d555a700 1 -- 172.21.2.102:0/1503502427 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f77d009a740 msgr2=0x7f77d0097bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:19.725 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:19.719 7f77d555a700 1 --2- 172.21.2.102:0/1503502427 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f77d009a740 0x7f77d0097bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:19.725 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:19.719 7f77d555a700 1 -- 172.21.2.102:0/1503502427 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f77d0099d20 msgr2=0x7f77d009a120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:19.725 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:19.719 7f77d555a700 1 --2- 172.21.2.102:0/1503502427 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f77d0099d20 0x7f77d009a120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:19.725 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:19.719 7f77d555a700 1 -- 172.21.2.102:0/1503502427 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77d009ad60 con 0x7f77d00ada90 2021-02-10T11:40:19.725 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:19.720 7f77d555a700 1 --2- 172.21.2.102:0/1503502427 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f77d00ada90 0x7f77d00ade90 secure :-1 s=READY pgs=3353 cs=0 l=1 rev1=1 rx=0x7f77c8009f60 tx=0x7f77c800bd20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:19.725 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:19.720 7f77c5ffb700 1 -- 172.21.2.102:0/1503502427 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f77c8006e30 con 0x7f77d00ada90 2021-02-10T11:40:19.725 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:19.720 7f77c5ffb700 1 -- 172.21.2.102:0/1503502427 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77c800be60 con 0x7f77d00ada90 2021-02-10T11:40:19.725 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:19.720 7f77c5ffb700 1 -- 172.21.2.102:0/1503502427 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f77c800c570 con 0x7f77d00ada90 2021-02-10T11:40:19.725 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:19.720 7f77d773e700 1 -- 172.21.2.102:0/1503502427 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77d009ad60 con 0x7f77d00ada90 2021-02-10T11:40:19.726 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:19.720 7f77d773e700 1 -- 172.21.2.102:0/1503502427 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77d00984d0 con 0x7f77d00ada90 2021-02-10T11:40:19.726 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:19.721 7f77c5ffb700 1 -- 172.21.2.102:0/1503502427 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f77c8001010 con 0x7f77d00ada90 2021-02-10T11:40:19.726 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:19.721 7f77c5ffb700 1 --2- 172.21.2.102:0/1503502427 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f77b00186c0 0x7f77b001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:19.726 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:19.721 7f77c5ffb700 1 -- 172.21.2.102:0/1503502427 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(193..193 src has 1..193) v4 ==== 4753+0+0 (secure 0 0 0) 0x7f77c80325e0 con 0x7f77d00ada90 2021-02-10T11:40:19.726 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:19.721 7f77d4d59700 1 --2- 172.21.2.102:0/1503502427 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f77b00186c0 0x7f77b001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:19.726 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:19.721 7f77d4d59700 1 --2- 172.21.2.102:0/1503502427 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f77b00186c0 0x7f77b001abc0 crc :-1 s=READY pgs=1460 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:19.753 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:19.751 7f77d773e700 1 -- 172.21.2.102:0/1503502427 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77d00ebc00 con 0x7f77d00ada90 2021-02-10T11:40:19.756 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:19.753 7f77c5ffb700 1 -- 172.21.2.102:0/1503502427 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f77c8009810 con 0x7f77d00ada90 2021-02-10T11:40:19.888 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:19.885 7f77d773e700 1 -- 172.21.2.102:0/1503502427 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f77d00f0b10 con 0x7f77d00ada90 2021-02-10T11:40:19.888 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:19.886 7f77c5ffb700 1 -- 172.21.2.102:0/1503502427 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v117) v1 ==== 80+0+4513 (secure 0 0 0) 0x7f77c801f650 con 0x7f77d00ada90 2021-02-10T11:40:19.889 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:40:19.889 INFO:teuthology.orchestra.run.gibba002.stdout:{ 2021-02-10T11:40:19.889 INFO:teuthology.orchestra.run.gibba002.stdout: "auth_dump": [ 2021-02-10T11:40:19.889 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:40:19.889 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "mds.a", 2021-02-10T11:40:19.889 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAevSNgsgyEARAAvr0cUjjCoEHLxaZSZ2HtVA==", 2021-02-10T11:40:19.889 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:40:19.889 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow", 2021-02-10T11:40:19.889 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:40:19.889 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:40:19.889 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:40:19.890 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:40:19.890 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:40:19.890 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:40:19.890 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "mds.b", 2021-02-10T11:40:19.890 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAevSNgwOFMMRAALfb1fbX56ao1TU8ClnTW1A==", 2021-02-10T11:40:19.890 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:40:19.890 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow", 2021-02-10T11:40:19.890 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:40:19.890 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:40:19.890 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:40:19.890 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:40:19.890 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:40:19.891 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:40:19.891 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.0", 2021-02-10T11:40:19.891 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAmvSNgj+bMMBAAD3HUD6fLiMAg+++iNzLJKQ==", 2021-02-10T11:40:19.891 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:40:19.891 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:40:19.891 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:40:19.891 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:40:19.891 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:40:19.891 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:40:19.891 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:40:19.891 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.1", 2021-02-10T11:40:19.891 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQApvSNgnvc+MRAAAOxqlNwiyLHqNITOrYeFKQ==", 2021-02-10T11:40:19.891 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:40:19.892 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:40:19.892 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:40:19.892 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:40:19.892 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:40:19.892 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:40:19.892 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:40:19.892 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.2", 2021-02-10T11:40:19.892 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAsvSNgZ4SbMRAA76IOYWfbiAuBoM/4k81GxA==", 2021-02-10T11:40:19.892 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:40:19.892 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:40:19.892 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:40:19.892 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:40:19.892 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:40:19.893 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:40:19.893 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:40:19.893 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.3", 2021-02-10T11:40:19.893 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAvvSNgFk3rMRAAWCmKKyw0LDQ+UlAHZAITOw==", 2021-02-10T11:40:19.893 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:40:19.893 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:40:19.893 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:40:19.893 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:40:19.893 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:40:19.893 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:40:19.893 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:40:19.893 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.4", 2021-02-10T11:40:19.894 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQA4vSNgpKcNJBAApJMmvj5YxVHPDOmvmQONOg==", 2021-02-10T11:40:19.894 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:40:19.894 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:40:19.894 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:40:19.894 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:40:19.894 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:40:19.894 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:40:19.894 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:40:19.894 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.5", 2021-02-10T11:40:19.894 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQA7vSNgf2ujJBAAfkuYpyb3WTzc8V5kw+Wktg==", 2021-02-10T11:40:19.894 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:40:19.894 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:40:19.895 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:40:19.895 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:40:19.895 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:40:19.895 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:40:19.895 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:40:19.895 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.6", 2021-02-10T11:40:19.895 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQA+vSNgqw9uJRAAFMU0R+UxU9C3o/9byb3e0g==", 2021-02-10T11:40:19.895 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:40:19.895 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:40:19.895 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:40:19.895 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:40:19.895 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:40:19.895 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:40:19.896 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:40:19.896 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "osd.7", 2021-02-10T11:40:19.896 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQBBvSNgvWSrJRAA2KaCHmLpzovRgYzLvyofTQ==", 2021-02-10T11:40:19.896 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:40:19.896 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:40:19.896 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:40:19.896 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:40:19.896 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:40:19.896 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:40:19.896 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:40:19.896 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "client.0", 2021-02-10T11:40:19.896 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAfvSNgZd9XIBAAvyrvl8fdKuHnJZfczfRoFw==", 2021-02-10T11:40:19.896 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:40:19.897 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow", 2021-02-10T11:40:19.897 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow r", 2021-02-10T11:40:19.897 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:40:19.897 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:40:19.897 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:40:19.897 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:40:19.897 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "client.1", 2021-02-10T11:40:19.897 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAfvSNgSyfXNhAAIduPnF7Vd/6HSkJ0MlcMSA==", 2021-02-10T11:40:19.897 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:40:19.897 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow", 2021-02-10T11:40:19.897 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow r", 2021-02-10T11:40:19.897 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:40:19.897 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:40:19.898 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:40:19.898 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:40:19.898 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "client.2", 2021-02-10T11:40:19.898 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAgvSNgiIG/ERAA9ZTamaTI7ZsPSnxBqjplCg==", 2021-02-10T11:40:19.898 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:40:19.898 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow", 2021-02-10T11:40:19.898 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow r", 2021-02-10T11:40:19.898 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:40:19.898 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:40:19.898 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:40:19.898 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:40:19.898 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "client.3", 2021-02-10T11:40:19.899 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAgvSNgOWpOKBAAlXsEB04Q/GvtL06PqRMLeA==", 2021-02-10T11:40:19.899 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:40:19.899 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow", 2021-02-10T11:40:19.899 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow r", 2021-02-10T11:40:19.899 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:40:19.899 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:40:19.899 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:40:19.899 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:40:19.899 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "client.admin", 2021-02-10T11:40:19.899 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAavSNgO3GQBhAAL+NVnYskQRKVWLgyDPY/xw==", 2021-02-10T11:40:19.899 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:40:19.899 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow *", 2021-02-10T11:40:19.899 INFO:teuthology.orchestra.run.gibba002.stdout: "mgr": "allow *", 2021-02-10T11:40:19.900 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow *", 2021-02-10T11:40:19.900 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:40:19.900 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:40:19.900 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:40:19.900 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:40:19.900 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "mgr.x", 2021-02-10T11:40:19.900 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAdvSNgXZZgJhAA7dtBSGzH+DsBwnddZpOBJA==", 2021-02-10T11:40:19.900 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:40:19.900 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow *", 2021-02-10T11:40:19.900 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow profile mgr", 2021-02-10T11:40:19.900 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:40:19.900 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:40:19.900 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:40:19.901 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:40:19.901 INFO:teuthology.orchestra.run.gibba002.stdout: "entity": "mgr.y", 2021-02-10T11:40:19.901 INFO:teuthology.orchestra.run.gibba002.stdout: "key": "AQAdvSNgVqQFDRAAwo6YEdWZCAr5GdOWVL/Qvg==", 2021-02-10T11:40:19.901 INFO:teuthology.orchestra.run.gibba002.stdout: "caps": { 2021-02-10T11:40:19.901 INFO:teuthology.orchestra.run.gibba002.stdout: "mds": "allow *", 2021-02-10T11:40:19.901 INFO:teuthology.orchestra.run.gibba002.stdout: "mon": "allow profile mgr", 2021-02-10T11:40:19.901 INFO:teuthology.orchestra.run.gibba002.stdout: "osd": "allow *" 2021-02-10T11:40:19.901 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:40:19.901 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:40:19.901 INFO:teuthology.orchestra.run.gibba002.stdout: ] 2021-02-10T11:40:19.901 INFO:teuthology.orchestra.run.gibba002.stdout:} 2021-02-10T11:40:19.903 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:19.887 7f77d773e700 1 -- 172.21.2.102:0/1503502427 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f77b00186c0 msgr2=0x7f77b001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:19.903 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:19.888 7f77d773e700 1 --2- 172.21.2.102:0/1503502427 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f77b00186c0 0x7f77b001abc0 crc :-1 s=READY pgs=1460 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:19.903 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:19.888 7f77d773e700 1 -- 172.21.2.102:0/1503502427 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f77d00ada90 msgr2=0x7f77d00ade90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:19.903 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:19.888 7f77d773e700 1 --2- 172.21.2.102:0/1503502427 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f77d00ada90 0x7f77d00ade90 secure :-1 s=READY pgs=3353 cs=0 l=1 rev1=1 rx=0x7f77c8009f60 tx=0x7f77c800bd20).stop 2021-02-10T11:40:19.903 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:19.888 7f77d773e700 1 -- 172.21.2.102:0/1503502427 shutdown_connections 2021-02-10T11:40:19.903 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:19.888 7f77d773e700 1 --2- 172.21.2.102:0/1503502427 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f77d009a740 0x7f77d0097bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:19.903 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:19.888 7f77d773e700 1 --2- 172.21.2.102:0/1503502427 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f77b00186c0 0x7f77b001abc0 unknown :-1 s=CLOSED pgs=1460 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:19.903 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:19.888 7f77d773e700 1 --2- 172.21.2.102:0/1503502427 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f77d0099d20 0x7f77d009a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:19.904 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:19.888 7f77d773e700 1 --2- 172.21.2.102:0/1503502427 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f77d00ada90 0x7f77d00ade90 unknown :-1 s=CLOSED pgs=3353 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:19.904 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:19.888 7f77d773e700 1 -- 172.21.2.102:0/1503502427 >> 172.21.2.102:0/1503502427 conn(0x7f77d00f0550 msgr2=0x7f77d0191730 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:19.904 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:19.888 7f77d773e700 1 -- 172.21.2.102:0/1503502427 shutdown_connections 2021-02-10T11:40:19.904 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:19.888 7f77d773e700 1 -- 172.21.2.102:0/1503502427 wait complete. 2021-02-10T11:40:19.904 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:40:20.336 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:20.317 7f69d0afb700 1 Processor -- start 2021-02-10T11:40:20.336 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:20.317 7f69d0afb700 1 -- start start 2021-02-10T11:40:20.336 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:20.317 7f69d0afb700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f69cc0d1870 0x7f69cc1b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:20.336 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:20.317 7f69d0afb700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f69cc0f0650 0x7f69cc0cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:20.336 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:20.317 7f69d0afb700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69cc0f0b10 con 0x7f69cc1b0a10 2021-02-10T11:40:20.336 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:20.317 7f69d0afb700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69cc0e5ed0 con 0x7f69cc0d1870 2021-02-10T11:40:20.336 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:20.317 7f69d0afb700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69cc0d1c70 con 0x7f69cc0f0650 2021-02-10T11:40:20.336 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:20.317 7f69ca05d700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f69cc1b0a10 0x7f69cc0cf330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:33352/0 (socket says 172.21.2.102:33352) 2021-02-10T11:40:20.336 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:20.317 7f69ca05d700 1 -- 172.21.2.102:0/1628361321 learned_addr learned my addr 172.21.2.102:0/1628361321 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:40:20.336 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:20.318 7f69ca85e700 1 --2- 172.21.2.102:0/1628361321 >> v2:172.21.2.113:3300/0 conn(0x7f69cc0d1870 0x7f69cc1b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:20.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:20.318 7f69cb05f700 1 --2- 172.21.2.102:0/1628361321 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f69cc0f0650 0x7f69cc0cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:20.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:20.318 7f69c985c700 1 -- 172.21.2.102:0/1628361321 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4183806593 0 0) 0x7f69b40009b0 con 0x7f69cc1b0a10 2021-02-10T11:40:20.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:20.318 7f69c985c700 1 -- 172.21.2.102:0/1628361321 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69b8002fe0 con 0x7f69cc1b0a10 2021-02-10T11:40:20.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:20.318 7f69c985c700 1 -- 172.21.2.102:0/1628361321 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2045483863 0 0) 0x7f69b4001010 con 0x7f69cc1b0a10 2021-02-10T11:40:20.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:20.318 7f69ca85e700 1 -- 172.21.2.102:0/1628361321 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f69cc0f0650 msgr2=0x7f69cc0cfb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:20.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:20.318 7f69ca85e700 1 --2- 172.21.2.102:0/1628361321 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f69cc0f0650 0x7f69cc0cfb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:20.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:20.318 7f69ca85e700 1 -- 172.21.2.102:0/1628361321 >> v1:172.21.2.102:6789/0 conn(0x7f69cc1b0a10 legacy=0x7f69cc0cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:20.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:20.318 7f69ca85e700 1 -- 172.21.2.102:0/1628361321 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69cc0d1e90 con 0x7f69cc0d1870 2021-02-10T11:40:20.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:20.318 7f69cb05f700 1 --2- 172.21.2.102:0/1628361321 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f69cc0f0650 0x7f69cc0cfb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:40:20.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:20.318 7f69ca85e700 1 --2- 172.21.2.102:0/1628361321 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f69cc0d1870 0x7f69cc1b03f0 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 rx=0x7f69bc006650 tx=0x7f69bc009a80).ready entity=mon.1 client_cookie=ef07a78302f23c30 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:20.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:20.318 7f69c985c700 1 -- 172.21.2.102:0/1628361321 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f69bc006390 con 0x7f69cc0d1870 2021-02-10T11:40:20.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:20.319 7f69c985c700 1 -- 172.21.2.102:0/1628361321 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69bc0009b0 con 0x7f69cc0d1870 2021-02-10T11:40:20.337 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:20.319 7f69c985c700 1 -- 172.21.2.102:0/1628361321 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f69bc00c2b0 con 0x7f69cc0d1870 2021-02-10T11:40:20.338 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:20.319 7f69d0afb700 1 -- 172.21.2.102:0/1628361321 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f69cc0d1870 msgr2=0x7f69cc1b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:20.338 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:20.319 7f69d0afb700 1 --2- 172.21.2.102:0/1628361321 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f69cc0d1870 0x7f69cc1b03f0 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 rx=0x7f69bc006650 tx=0x7f69bc009a80).stop 2021-02-10T11:40:20.338 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:20.319 7f69d0afb700 1 -- 172.21.2.102:0/1628361321 shutdown_connections 2021-02-10T11:40:20.338 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:20.319 7f69d0afb700 1 --2- 172.21.2.102:0/1628361321 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f69cc0f0650 0x7f69cc0cfb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:20.338 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:20.319 7f69d0afb700 1 --2- 172.21.2.102:0/1628361321 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f69cc0d1870 0x7f69cc1b03f0 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:20.338 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:20.319 7f69d0afb700 1 -- 172.21.2.102:0/1628361321 >> 172.21.2.102:0/1628361321 conn(0x7f69cc0ce450 msgr2=0x7f69cc0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:20.338 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:20.319 7f69d0afb700 1 -- 172.21.2.102:0/1628361321 shutdown_connections 2021-02-10T11:40:20.338 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:20.319 7f69d0afb700 1 -- 172.21.2.102:0/1628361321 wait complete. 2021-02-10T11:40:20.340 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:20.334 7f69d0afb700 1 Processor -- start 2021-02-10T11:40:20.340 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:20.334 7f69d0afb700 1 -- start start 2021-02-10T11:40:20.340 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:20.334 7f69d0afb700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f69cc1942b0 0x7f69cc0b3550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:20.340 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:20.334 7f69d0afb700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f69cc195070 0x7f69cc195470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:20.340 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:20.334 7f69d0afb700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f69cc0bb8e0 0x7f69cc0bbce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:20.340 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:20.334 7f69d0afb700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69cc1946b0 con 0x7f69cc0bb8e0 2021-02-10T11:40:20.340 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:20.334 7f69d0afb700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69cc0b3b70 con 0x7f69cc195070 2021-02-10T11:40:20.340 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:20.334 7f69d0afb700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69cc0b3d90 con 0x7f69cc1942b0 2021-02-10T11:40:20.340 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:20.335 7f69cb05f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f69cc0bb8e0 0x7f69cc0bbce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:20.340 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:20.335 7f69ca85e700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f69cc1942b0 0x7f69cc0b3550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:20.340 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:20.335 7f69cb05f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f69cc0bb8e0 0x7f69cc0bbce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:41132/0 (socket says 172.21.2.102:41132) 2021-02-10T11:40:20.340 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:20.335 7f69cb05f700 1 -- 172.21.2.102:0/84308434 learned_addr learned my addr 172.21.2.102:0/84308434 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:40:20.340 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:20.335 7f69ca05d700 1 --2- 172.21.2.102:0/84308434 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f69cc195070 0x7f69cc195470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:20.341 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:20.335 7f69cb05f700 1 -- 172.21.2.102:0/84308434 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f69cc1942b0 msgr2=0x7f69cc0b3550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:20.341 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:20.335 7f69cb05f700 1 --2- 172.21.2.102:0/84308434 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f69cc1942b0 0x7f69cc0b3550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:20.341 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:20.335 7f69cb05f700 1 -- 172.21.2.102:0/84308434 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f69cc195070 msgr2=0x7f69cc195470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:20.341 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:20.335 7f69cb05f700 1 --2- 172.21.2.102:0/84308434 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f69cc195070 0x7f69cc195470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:20.341 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:20.335 7f69cb05f700 1 -- 172.21.2.102:0/84308434 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69cc1a49e0 con 0x7f69cc0bb8e0 2021-02-10T11:40:20.341 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:20.335 7f69cb05f700 1 --2- 172.21.2.102:0/84308434 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f69cc0bb8e0 0x7f69cc0bbce0 secure :-1 s=READY pgs=3355 cs=0 l=1 rev1=1 rx=0x7f69c400b970 tx=0x7f69c400bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:20.341 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:20.335 7f69c37fe700 1 -- 172.21.2.102:0/84308434 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f69c4000f90 con 0x7f69cc0bb8e0 2021-02-10T11:40:20.341 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:20.335 7f69c37fe700 1 -- 172.21.2.102:0/84308434 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69c4003030 con 0x7f69cc0bb8e0 2021-02-10T11:40:20.341 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:20.335 7f69c37fe700 1 -- 172.21.2.102:0/84308434 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f69c4001400 con 0x7f69cc0bb8e0 2021-02-10T11:40:20.341 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:20.335 7f69d0afb700 1 -- 172.21.2.102:0/84308434 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69cc1a49e0 con 0x7f69cc0bb8e0 2021-02-10T11:40:20.341 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:20.335 7f69d0afb700 1 -- 172.21.2.102:0/84308434 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69cc1a49e0 con 0x7f69cc0bb8e0 2021-02-10T11:40:20.341 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:20.336 7f69c37fe700 1 -- 172.21.2.102:0/84308434 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f69c4000df0 con 0x7f69cc0bb8e0 2021-02-10T11:40:20.341 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:20.336 7f69c37fe700 1 --2- 172.21.2.102:0/84308434 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f69b00186c0 0x7f69b001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:20.342 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:20.336 7f69c37fe700 1 -- 172.21.2.102:0/84308434 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(193..193 src has 1..193) v4 ==== 4753+0+0 (secure 0 0 0) 0x7f69c4032440 con 0x7f69cc0bb8e0 2021-02-10T11:40:20.342 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:20.336 7f69ca85e700 1 --2- 172.21.2.102:0/84308434 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f69b00186c0 0x7f69b001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:20.342 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:20.336 7f69ca85e700 1 --2- 172.21.2.102:0/84308434 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f69b00186c0 0x7f69b001abc0 crc :-1 s=READY pgs=1461 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:20.369 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:20.367 7f69d0afb700 1 -- 172.21.2.102:0/84308434 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69cc0bc5f0 con 0x7f69cc0bb8e0 2021-02-10T11:40:20.371 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:20.369 7f69c37fe700 1 -- 172.21.2.102:0/84308434 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f69c40066f0 con 0x7f69cc0bb8e0 2021-02-10T11:40:20.494 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:20.492 7f69d0afb700 1 -- 172.21.2.102:0/84308434 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f69cc0bc5f0 con 0x7f69cc0bb8e0 2021-02-10T11:40:20.495 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:20.493 7f69c37fe700 1 -- 172.21.2.102:0/84308434 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v193) v1 ==== 74+0+11154 (secure 0 0 0) 0x7f69c4000df0 con 0x7f69cc0bb8e0 2021-02-10T11:40:20.495 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:40:20.495 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":193,"fsid":"68db30ea-dbce-4aef-a6df-f01c6fea0513","created":"2021-02-10 11:03:06.331557","modified":"2021-02-10 11:40:19.226436","last_up_change":"2021-02-10 11:03:32.690562","last_in_change":"2021-02-10 11:03:32.690562","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":41,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:03:37.322519","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"be691b4d-a039-408a-af80-9d53f02b6f1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":185,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6818","nonce":36088},{"type":"v1","addr":"172.21.2.102:6820","nonce":36088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6822","nonce":36088},{"type":"v1","addr":"172.21.2.102:6824","nonce":36088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6830","nonce":36088},{"type":"v1","addr":"172.21.2.102:6832","nonce":36088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6826","nonce":36088},{"type":"v1","addr":"172.21.2.102:6828","nonce":36088}]},"public_addr":"172.21.2.102:6820/36088","cluster_addr":"172.21.2.102:6824/36088","heartbeat_back_addr":"172.21.2.102:6832/36088","heartbeat_front_addr":"172.21.2.102:6828/36088","state":["exists","up"]},{"osd":1,"uuid":"64e3ccc8-9c69-4d7c-865e-af8e83822635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":185,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6819","nonce":36087},{"type":"v1","addr":"172.21.2.102:6821","nonce":36087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6823","nonce":36087},{"type":"v1","addr":"172.21.2.102:6825","nonce":36087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6831","nonce":36087},{"type":"v1","addr":"172.21.2.102:6833","nonce":36087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6827","nonce":36087},{"type":"v1","addr":"172.21.2.102:6829","nonce":36087}]},"public_addr":"172.21.2.102:6821/36087","cluster_addr":"172.21.2.102:6825/36087","heartbeat_back_addr":"172.21.2.102:6833/36087","heartbeat_front_addr":"172.21.2.102:6829/36087","state":["exists","up"]},{"osd":2,"uuid":"153700af-747f-481f-bdf4-d6072d3a842f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":187,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6810","nonce":36089},{"type":"v1","addr":"172.21.2.102:6811","nonce":36089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6812","nonce":36089},{"type":"v1","addr":"172.21.2.102:6813","nonce":36089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6816","nonce":36089},{"type":"v1","addr":"172.21.2.102:6817","nonce":36089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6814","nonce":36089},{"type":"v1","addr":"172.21.2.102:6815","nonce":36089}]},"public_addr":"172.21.2.102:6811/36089","cluster_addr":"172.21.2.102:6813/36089","heartbeat_back_addr":"172.21.2.102:6817/36089","heartbeat_front_addr":"172.21.2.102:6815/36089","state":["exists","up"]},{"osd":3,"uuid":"cbc678f4-83a9-45e6-8830-2f7c232b1ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":187,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6802","nonce":36090},{"type":"v1","addr":"172.21.2.102:6803","nonce":36090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6804","nonce":36090},{"type":"v1","addr":"172.21.2.102:6805","nonce":36090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6808","nonce":36090},{"type":"v1","addr":"172.21.2.102:6809","nonce":36090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6806","nonce":36090},{"type":"v1","addr":"172.21.2.102:6807","nonce":36090}]},"public_addr":"172.21.2.102:6803/36090","cluster_addr":"172.21.2.102:6805/36090","heartbeat_back_addr":"172.21.2.102:6809/36090","heartbeat_front_addr":"172.21.2.102:6807/36090","state":["exists","up"]},{"osd":4,"uuid":"c01e4008-b258-4f10-afd2-9e7f37214351","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":187,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6808","nonce":36368},{"type":"v1","addr":"172.21.2.113:6811","nonce":36368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6814","nonce":36368},{"type":"v1","addr":"172.21.2.113:6818","nonce":36368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6826","nonce":36368},{"type":"v1","addr":"172.21.2.113:6828","nonce":36368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6820","nonce":36368},{"type":"v1","addr":"172.21.2.113:6824","nonce":36368}]},"public_addr":"172.21.2.113:6811/36368","cluster_addr":"172.21.2.113:6818/36368","heartbeat_back_addr":"172.21.2.113:6828/36368","heartbeat_front_addr":"172.21.2.113:6824/36368","state":["exists","up"]},{"osd":5,"uuid":"551f6007-daf8-4a98-8ee0-c2b16bcccb5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":178,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6802","nonce":36366},{"type":"v1","addr":"172.21.2.113:6805","nonce":36366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6807","nonce":36366},{"type":"v1","addr":"172.21.2.113:6810","nonce":36366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6819","nonce":36366},{"type":"v1","addr":"172.21.2.113:6822","nonce":36366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6812","nonce":36366},{"type":"v1","addr":"172.21.2.113:6815","nonce":36366}]},"public_addr":"172.21.2.113:6805/36366","cluster_addr":"172.21.2.113:6810/36366","heartbeat_back_addr":"172.21.2.113:6822/36366","heartbeat_front_addr":"172.21.2.113:6815/36366","state":["exists","up"]},{"osd":6,"uuid":"bf67c040-0e50-48be-b102-c3943bc9ebef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":187,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6800","nonce":36367},{"type":"v1","addr":"172.21.2.113:6801","nonce":36367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6803","nonce":36367},{"type":"v1","addr":"172.21.2.113:6804","nonce":36367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6813","nonce":36367},{"type":"v1","addr":"172.21.2.113:6816","nonce":36367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6806","nonce":36367},{"type":"v1","addr":"172.21.2.113:6809","nonce":36367}]},"public_addr":"172.21.2.113:6801/36367","cluster_addr":"172.21.2.113:6804/36367","heartbeat_back_addr":"172.21.2.113:6816/36367","heartbeat_front_addr":"172.21.2.113:6809/36367","state":["exists","up"]},{"osd":7,"uuid":"f5461f80-018d-428f-be89-2eb2bcbe7bb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":187,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6817","nonce":36365},{"type":"v1","addr":"172.21.2.113:6821","nonce":36365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6823","nonce":36365},{"type":"v1","addr":"172.21.2.113:6825","nonce":36365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6830","nonce":36365},{"type":"v1","addr":"172.21.2.113:6831","nonce":36365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6827","nonce":36365},{"type":"v1","addr":"172.21.2.113:6829","nonce":36365}]},"public_addr":"172.21.2.113:6821/36365","cluster_addr":"172.21.2.113:6825/36365","heartbeat_back_addr":"172.21.2.113:6831/36365","heartbeat_front_addr":"172.21.2.113:6829/36365","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:40:20.497 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:20.494 7f69d0afb700 1 -- 172.21.2.102:0/84308434 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f69b00186c0 msgr2=0x7f69b001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:20.497 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:20.494 7f69d0afb700 1 --2- 172.21.2.102:0/84308434 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f69b00186c0 0x7f69b001abc0 crc :-1 s=READY pgs=1461 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:20.497 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:20.494 7f69d0afb700 1 -- 172.21.2.102:0/84308434 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f69cc0bb8e0 msgr2=0x7f69cc0bbce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:20.497 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:20.494 7f69d0afb700 1 --2- 172.21.2.102:0/84308434 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f69cc0bb8e0 0x7f69cc0bbce0 secure :-1 s=READY pgs=3355 cs=0 l=1 rev1=1 rx=0x7f69c400b970 tx=0x7f69c400bd10).stop 2021-02-10T11:40:20.497 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:20.495 7f69d0afb700 1 -- 172.21.2.102:0/84308434 shutdown_connections 2021-02-10T11:40:20.497 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:20.495 7f69d0afb700 1 --2- 172.21.2.102:0/84308434 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f69cc0bb8e0 0x7f69cc0bbce0 unknown :-1 s=CLOSED pgs=3355 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:20.497 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:20.495 7f69d0afb700 1 --2- 172.21.2.102:0/84308434 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f69b00186c0 0x7f69b001abc0 unknown :-1 s=CLOSED pgs=1461 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:20.497 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:20.495 7f69d0afb700 1 --2- 172.21.2.102:0/84308434 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f69cc195070 0x7f69cc195470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:20.497 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:20.495 7f69d0afb700 1 --2- 172.21.2.102:0/84308434 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f69cc1942b0 0x7f69cc0b3550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:20.498 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:20.495 7f69d0afb700 1 -- 172.21.2.102:0/84308434 >> 172.21.2.102:0/84308434 conn(0x7f69cc0b3040 msgr2=0x7f69cc0c3250 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:20.498 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:20.495 7f69d0afb700 1 -- 172.21.2.102:0/84308434 shutdown_connections 2021-02-10T11:40:20.498 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:20.495 7f69d0afb700 1 -- 172.21.2.102:0/84308434 wait complete. 2021-02-10T11:40:20.506 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:03:37.322519', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2021-02-10T11:40:20.506 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:40:20.939 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:20.920 7fbaa33c6700 1 Processor -- start 2021-02-10T11:40:20.939 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:20.920 7fbaa33c6700 1 -- start start 2021-02-10T11:40:20.939 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:20.920 7fbaa33c6700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fba9c0d1870 0x7fba9c1b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:20.939 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:20.920 7fbaa33c6700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fba9c1b0ae0 0x7fba9c108940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:20.939 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:20.920 7fbaa33c6700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba9c109a60 con 0x7fba9c0d1c70 2021-02-10T11:40:20.939 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:20.920 7fbaa33c6700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba9c0e5ef0 con 0x7fba9c1b0ae0 2021-02-10T11:40:20.939 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:20.920 7fbaa33c6700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba9c109750 con 0x7fba9c0d1870 2021-02-10T11:40:20.939 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:20.921 7fbaa19e3700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fba9c0d1c70 0x7fba9c108f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:33364/0 (socket says 172.21.2.102:33364) 2021-02-10T11:40:20.939 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:20.921 7fbaa19e3700 1 -- 172.21.2.102:0/2902780488 learned_addr learned my addr 172.21.2.102:0/2902780488 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:40:20.940 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:20.921 7fbaa09e1700 1 --2- 172.21.2.102:0/2902780488 >> v2:172.21.2.102:3300/0 conn(0x7fba9c1b0ae0 0x7fba9c108940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:20.940 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:20.921 7fbaa09e1700 1 -- 172.21.2.102:0/2902780488 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fba9c0d1870 msgr2=0x7fba9c1b04c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:40:20.940 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:20.921 7fbaa09e1700 1 --2- 172.21.2.102:0/2902780488 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fba9c0d1870 0x7fba9c1b04c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:20.940 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:20.921 7fbaa09e1700 1 -- 172.21.2.102:0/2902780488 >> v1:172.21.2.102:6789/0 conn(0x7fba9c0d1c70 legacy=0x7fba9c108f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:20.940 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:20.921 7fbaa09e1700 1 -- 172.21.2.102:0/2902780488 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba9c088070 con 0x7fba9c1b0ae0 2021-02-10T11:40:20.940 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:20.921 7fbaa09e1700 1 --2- 172.21.2.102:0/2902780488 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fba9c1b0ae0 0x7fba9c108940 secure :-1 s=READY pgs=3357 cs=0 l=1 rev1=1 rx=0x7fba90006650 tx=0x7fba90009a00).ready entity=mon.0 client_cookie=db1f5814612564ae server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:20.941 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:20.921 7fba8ffff700 1 -- 172.21.2.102:0/2902780488 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fba900009b0 con 0x7fba9c1b0ae0 2021-02-10T11:40:20.941 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:20.921 7fba8ffff700 1 -- 172.21.2.102:0/2902780488 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba9000c0c0 con 0x7fba9c1b0ae0 2021-02-10T11:40:20.941 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:20.921 7fba8ffff700 1 -- 172.21.2.102:0/2902780488 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fba9000c5a0 con 0x7fba9c1b0ae0 2021-02-10T11:40:20.941 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:20.921 7fbaa33c6700 1 -- 172.21.2.102:0/2902780488 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fba9c1b0ae0 msgr2=0x7fba9c108940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:20.941 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:20.921 7fbaa33c6700 1 --2- 172.21.2.102:0/2902780488 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fba9c1b0ae0 0x7fba9c108940 secure :-1 s=READY pgs=3357 cs=0 l=1 rev1=1 rx=0x7fba90006650 tx=0x7fba90009a00).stop 2021-02-10T11:40:20.941 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:20.922 7fbaa33c6700 1 -- 172.21.2.102:0/2902780488 shutdown_connections 2021-02-10T11:40:20.942 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:20.922 7fbaa33c6700 1 --2- 172.21.2.102:0/2902780488 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fba9c1b0ae0 0x7fba9c108940 unknown :-1 s=CLOSED pgs=3357 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:20.942 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:20.922 7fbaa33c6700 1 --2- 172.21.2.102:0/2902780488 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fba9c0d1870 0x7fba9c1b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:20.942 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:20.922 7fbaa33c6700 1 -- 172.21.2.102:0/2902780488 >> 172.21.2.102:0/2902780488 conn(0x7fba9c0ce420 msgr2=0x7fba9c0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:20.942 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:20.922 7fbaa33c6700 1 -- 172.21.2.102:0/2902780488 shutdown_connections 2021-02-10T11:40:20.942 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:20.922 7fbaa33c6700 1 -- 172.21.2.102:0/2902780488 wait complete. 2021-02-10T11:40:20.944 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:20.937 7fbaa33c6700 1 Processor -- start 2021-02-10T11:40:20.944 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:20.937 7fbaa33c6700 1 -- start start 2021-02-10T11:40:20.945 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:20.937 7fbaa33c6700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fba9c0c3820 0x7fba9c084cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:20.945 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:20.937 7fbaa33c6700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fba9c0a0290 0x7fba9c0852d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:20.945 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:20.937 7fbaa33c6700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fba9c086270 0x7fba9c086670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:20.945 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:20.937 7fbaa33c6700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba9c0c0700 con 0x7fba9c0a0290 2021-02-10T11:40:20.945 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:20.937 7fbaa33c6700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba9c0858f0 con 0x7fba9c086270 2021-02-10T11:40:20.945 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:20.937 7fbaa33c6700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba9c085b10 con 0x7fba9c0c3820 2021-02-10T11:40:20.945 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:20.938 7fbaa09e1700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fba9c0a0290 0x7fba9c0852d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:20.945 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:20.938 7fbaa09e1700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fba9c0a0290 0x7fba9c0852d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:41148/0 (socket says 172.21.2.102:41148) 2021-02-10T11:40:20.945 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:20.938 7fbaa09e1700 1 -- 172.21.2.102:0/3355692200 learned_addr learned my addr 172.21.2.102:0/3355692200 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:40:20.946 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:20.938 7fbaa09e1700 1 -- 172.21.2.102:0/3355692200 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fba9c0c3820 msgr2=0x7fba9c084cb0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:40:20.946 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:20.938 7fbaa09e1700 1 --2- 172.21.2.102:0/3355692200 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fba9c0c3820 0x7fba9c084cb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:20.946 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:20.938 7fbaa09e1700 1 -- 172.21.2.102:0/3355692200 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fba9c086270 msgr2=0x7fba9c086670 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:40:20.946 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:20.938 7fbaa09e1700 1 --2- 172.21.2.102:0/3355692200 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fba9c086270 0x7fba9c086670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:20.946 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:20.938 7fbaa09e1700 1 -- 172.21.2.102:0/3355692200 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba9c086c90 con 0x7fba9c0a0290 2021-02-10T11:40:20.946 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:20.938 7fbaa09e1700 1 --2- 172.21.2.102:0/3355692200 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fba9c0a0290 0x7fba9c0852d0 secure :-1 s=READY pgs=3358 cs=0 l=1 rev1=1 rx=0x7fba9000b5e0 tx=0x7fba9000b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:20.946 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:20.938 7fba8dffb700 1 -- 172.21.2.102:0/3355692200 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fba900009b0 con 0x7fba9c0a0290 2021-02-10T11:40:20.946 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:20.938 7fba8dffb700 1 -- 172.21.2.102:0/3355692200 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba900063b0 con 0x7fba9c0a0290 2021-02-10T11:40:20.946 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:20.938 7fba8dffb700 1 -- 172.21.2.102:0/3355692200 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fba900022e0 con 0x7fba9c0a0290 2021-02-10T11:40:20.946 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:20.938 7fbaa33c6700 1 -- 172.21.2.102:0/3355692200 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba9c086c90 con 0x7fba9c0a0290 2021-02-10T11:40:20.946 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:20.938 7fbaa33c6700 1 -- 172.21.2.102:0/3355692200 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba9c086c90 con 0x7fba9c0a0290 2021-02-10T11:40:20.946 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:20.939 7fba8dffb700 1 -- 172.21.2.102:0/3355692200 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fba90000c90 con 0x7fba9c0a0290 2021-02-10T11:40:20.946 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:20.939 7fba8dffb700 1 --2- 172.21.2.102:0/3355692200 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fba7c0186c0 0x7fba7c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:20.946 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:20.939 7fba8dffb700 1 -- 172.21.2.102:0/3355692200 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(193..193 src has 1..193) v4 ==== 4753+0+0 (secure 0 0 0) 0x7fba900320e0 con 0x7fba9c0a0290 2021-02-10T11:40:20.947 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:20.939 7fbaa11e2700 1 --2- 172.21.2.102:0/3355692200 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fba7c0186c0 0x7fba7c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:20.947 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:20.939 7fbaa11e2700 1 --2- 172.21.2.102:0/3355692200 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fba7c0186c0 0x7fba7c01abc0 crc :-1 s=READY pgs=1462 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:20.972 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:20.970 7fbaa33c6700 1 -- 172.21.2.102:0/3355692200 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba9c082a70 con 0x7fba9c0a0290 2021-02-10T11:40:20.975 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:20.973 7fba8dffb700 1 -- 172.21.2.102:0/3355692200 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fba90000940 con 0x7fba9c0a0290 2021-02-10T11:40:21.102 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:21.100 7fbaa33c6700 1 -- 172.21.2.102:0/3355692200 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7fba9c082a70 con 0x7fba9c0a0290 2021-02-10T11:40:21.102 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:21.100 7fba8dffb700 1 -- 172.21.2.102:0/3355692200 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v193) v1 ==== 92+0+10 (secure 0 0 0) 0x7fba9000bd70 con 0x7fba9c0a0290 2021-02-10T11:40:21.103 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:40:21.104 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:21.102 7fbaa33c6700 1 -- 172.21.2.102:0/3355692200 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fba7c0186c0 msgr2=0x7fba7c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:21.104 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:21.102 7fbaa33c6700 1 --2- 172.21.2.102:0/3355692200 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fba7c0186c0 0x7fba7c01abc0 crc :-1 s=READY pgs=1462 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:21.104 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:21.102 7fbaa33c6700 1 -- 172.21.2.102:0/3355692200 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fba9c0a0290 msgr2=0x7fba9c0852d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:21.104 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:21.102 7fbaa33c6700 1 --2- 172.21.2.102:0/3355692200 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fba9c0a0290 0x7fba9c0852d0 secure :-1 s=READY pgs=3358 cs=0 l=1 rev1=1 rx=0x7fba9000b5e0 tx=0x7fba9000b980).stop 2021-02-10T11:40:21.105 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:21.102 7fbaa33c6700 1 -- 172.21.2.102:0/3355692200 shutdown_connections 2021-02-10T11:40:21.105 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:21.102 7fbaa33c6700 1 --2- 172.21.2.102:0/3355692200 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fba7c0186c0 0x7fba7c01abc0 unknown :-1 s=CLOSED pgs=1462 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:21.105 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:21.102 7fbaa33c6700 1 --2- 172.21.2.102:0/3355692200 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fba9c086270 0x7fba9c086670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:21.105 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:21.102 7fbaa33c6700 1 --2- 172.21.2.102:0/3355692200 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fba9c0a0290 0x7fba9c0852d0 unknown :-1 s=CLOSED pgs=3358 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:21.105 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:21.102 7fbaa33c6700 1 --2- 172.21.2.102:0/3355692200 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fba9c0c3820 0x7fba9c084cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:21.105 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:21.102 7fbaa33c6700 1 -- 172.21.2.102:0/3355692200 >> 172.21.2.102:0/3355692200 conn(0x7fba9c0872c0 msgr2=0x7fba9c095700 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:21.105 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:21.102 7fbaa33c6700 1 -- 172.21.2.102:0/3355692200 shutdown_connections 2021-02-10T11:40:21.105 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:21.102 7fbaa33c6700 1 -- 172.21.2.102:0/3355692200 wait complete. 2021-02-10T11:40:21.113 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --format=json-pretty osd lspools 2021-02-10T11:40:21.544 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:21.524 7f6ad0179700 1 Processor -- start 2021-02-10T11:40:21.544 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:21.524 7f6ad0179700 1 -- start start 2021-02-10T11:40:21.544 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:21.525 7f6ad0179700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6ac80d1870 0x7f6ac81b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:21.545 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:21.525 7f6ad0179700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ac80f0b10 con 0x7f6ac80f0650 2021-02-10T11:40:21.545 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:21.525 7f6ad0179700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ac80d0310 con 0x7f6ac81b0a10 2021-02-10T11:40:21.545 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:21.525 7f6ad0179700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ac80e5ed0 con 0x7f6ac80d1870 2021-02-10T11:40:21.545 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:21.525 7f6ace796700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f6ac80f0650 0x7f6ac80cfb20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:33378/0 (socket says 172.21.2.102:33378) 2021-02-10T11:40:21.545 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:21.525 7f6ace796700 1 -- 172.21.2.102:0/4217100359 learned_addr learned my addr 172.21.2.102:0/4217100359 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:40:21.545 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:21.525 7f6accf93700 1 -- 172.21.2.102:0/4217100359 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1705364373 0 0) 0x7f6ac0001500 con 0x7f6ac80f0650 2021-02-10T11:40:21.545 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:21.525 7f6accf93700 1 -- 172.21.2.102:0/4217100359 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ab4002fe0 con 0x7f6ac80f0650 2021-02-10T11:40:21.545 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:21.525 7f6acdf95700 1 --2- 172.21.2.102:0/4217100359 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6ac80d1870 0x7f6ac81b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:21.545 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:21.526 7f6accf93700 1 -- 172.21.2.102:0/4217100359 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2758996192 0 0) 0x7f6ac0001460 con 0x7f6ac80f0650 2021-02-10T11:40:21.545 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:21.526 7f6accf93700 1 -- 172.21.2.102:0/4217100359 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6ac80d1870 msgr2=0x7f6ac81b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:21.545 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:21.526 7f6accf93700 1 --2- 172.21.2.102:0/4217100359 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6ac80d1870 0x7f6ac81b03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:21.545 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:21.526 7f6accf93700 1 -- 172.21.2.102:0/4217100359 >> v1:172.21.2.113:6789/0 conn(0x7f6ac81b0a10 legacy=0x7f6ac80cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:21.546 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:21.526 7f6accf93700 1 -- 172.21.2.102:0/4217100359 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ac80d0530 con 0x7f6ac80f0650 2021-02-10T11:40:21.546 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:21.526 7f6accf93700 1 -- 172.21.2.102:0/4217100359 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7f6ac0001980 con 0x7f6ac80f0650 2021-02-10T11:40:21.546 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:21.526 7f6accf93700 1 -- 172.21.2.102:0/4217100359 >> v1:172.21.2.102:6789/0 conn(0x7f6ac80f0650 legacy=0x7f6ac80cfb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:21.546 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:21.526 7f6accf93700 1 --2- 172.21.2.102:0/4217100359 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6ab4003aa0 0x7f6ab4005f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:21.546 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:21.526 7f6accf93700 1 --2- 172.21.2.102:0/4217100359 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6ab4006750 0x7f6ab4008c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:21.546 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:21.526 7f6accf93700 1 --2- 172.21.2.102:0/4217100359 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6ab40094f0 0x7f6ab400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:21.546 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:21.526 7f6accf93700 1 -- 172.21.2.102:0/4217100359 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ab400c2c0 con 0x7f6ab40094f0 2021-02-10T11:40:21.546 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:21.526 7f6accf93700 1 -- 172.21.2.102:0/4217100359 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ab400c4e0 con 0x7f6ab4006750 2021-02-10T11:40:21.546 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:21.526 7f6accf93700 1 -- 172.21.2.102:0/4217100359 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ab400c700 con 0x7f6ab4003aa0 2021-02-10T11:40:21.546 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:21.526 7f6ace796700 1 --2- 172.21.2.102:0/4217100359 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6ab40094f0 0x7f6ab400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:21.546 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:21.526 7f6acdf95700 1 --2- 172.21.2.102:0/4217100359 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6ab4003aa0 0x7f6ab4005f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:21.546 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:21.526 7f6acd794700 1 --2- 172.21.2.102:0/4217100359 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6ab4006750 0x7f6ab4008c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:21.547 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:21.526 7f6acdf95700 1 -- 172.21.2.102:0/4217100359 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6ab4006750 msgr2=0x7f6ab4008c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:21.547 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:21.526 7f6acdf95700 1 --2- 172.21.2.102:0/4217100359 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6ab4006750 0x7f6ab4008c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:21.547 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:21.526 7f6acdf95700 1 -- 172.21.2.102:0/4217100359 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6ab40094f0 msgr2=0x7f6ab400ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:21.547 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:21.526 7f6acdf95700 1 --2- 172.21.2.102:0/4217100359 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6ab40094f0 0x7f6ab400ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:21.547 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:21.526 7f6acdf95700 1 -- 172.21.2.102:0/4217100359 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ab400c9a0 con 0x7f6ab4003aa0 2021-02-10T11:40:21.547 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:21.527 7f6acdf95700 1 --2- 172.21.2.102:0/4217100359 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6ab4003aa0 0x7f6ab4005f30 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 rx=0x7f6abc006ab0 tx=0x7f6abc009b40).ready entity=mon.2 client_cookie=ecd9cf8eec2ff16c server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:21.547 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:21.527 7f6accf93700 1 -- 172.21.2.102:0/4217100359 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6abc006680 con 0x7f6ab4003aa0 2021-02-10T11:40:21.547 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:21.527 7f6accf93700 1 -- 172.21.2.102:0/4217100359 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6abc0063f0 con 0x7f6ab4003aa0 2021-02-10T11:40:21.547 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:21.527 7f6accf93700 1 -- 172.21.2.102:0/4217100359 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6abc00c2b0 con 0x7f6ab4003aa0 2021-02-10T11:40:21.547 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:21.527 7f6ad0179700 1 -- 172.21.2.102:0/4217100359 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6ab4003aa0 msgr2=0x7f6ab4005f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:21.547 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:21.527 7f6ad0179700 1 --2- 172.21.2.102:0/4217100359 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6ab4003aa0 0x7f6ab4005f30 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 rx=0x7f6abc006ab0 tx=0x7f6abc009b40).stop 2021-02-10T11:40:21.547 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:21.527 7f6ace796700 1 -- 172.21.2.102:0/4217100359 reap_dead start 2021-02-10T11:40:21.547 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:21.527 7f6ad0179700 1 -- 172.21.2.102:0/4217100359 shutdown_connections 2021-02-10T11:40:21.548 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:21.527 7f6ad0179700 1 -- 172.21.2.102:0/4217100359 >> 172.21.2.102:0/4217100359 conn(0x7f6ac80ce450 msgr2=0x7f6ac80d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:21.548 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:21.528 7f6ad0179700 1 -- 172.21.2.102:0/4217100359 shutdown_connections 2021-02-10T11:40:21.548 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:21.528 7f6ad0179700 1 -- 172.21.2.102:0/4217100359 wait complete. 2021-02-10T11:40:21.549 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:21.542 7f6ad0179700 1 Processor -- start 2021-02-10T11:40:21.549 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:21.542 7f6ad0179700 1 -- start start 2021-02-10T11:40:21.551 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:21.542 7f6ad0179700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6ac80a2560 0x7f6ac80b4fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:21.551 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:21.542 7f6ad0179700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6ac80b55e0 0x7f6ac80b59e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:21.551 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:21.542 7f6ad0179700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6ac80ee200 0x7f6ac80ee600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:21.551 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:21.542 7f6ad0179700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ac818fda0 con 0x7f6ac80a2560 2021-02-10T11:40:21.551 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:21.542 7f6ad0179700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ac80a20c0 con 0x7f6ac80b55e0 2021-02-10T11:40:21.551 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:21.542 7f6ad0179700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ac80b6000 con 0x7f6ac80ee200 2021-02-10T11:40:21.551 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:21.543 7f6acdf95700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6ac80a2560 0x7f6ac80b4fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:21.551 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:21.543 7f6acdf95700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6ac80a2560 0x7f6ac80b4fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:41166/0 (socket says 172.21.2.102:41166) 2021-02-10T11:40:21.551 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:21.543 7f6acdf95700 1 -- 172.21.2.102:0/3153717193 learned_addr learned my addr 172.21.2.102:0/3153717193 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:40:21.551 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:21.543 7f6acdf95700 1 -- 172.21.2.102:0/3153717193 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6ac80ee200 msgr2=0x7f6ac80ee600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:21.551 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:21.543 7f6acdf95700 1 --2- 172.21.2.102:0/3153717193 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6ac80ee200 0x7f6ac80ee600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:21.552 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:21.543 7f6acdf95700 1 -- 172.21.2.102:0/3153717193 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6ac80b55e0 msgr2=0x7f6ac80b59e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:21.552 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:21.543 7f6acdf95700 1 --2- 172.21.2.102:0/3153717193 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6ac80b55e0 0x7f6ac80b59e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:21.552 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:21.543 7f6acdf95700 1 -- 172.21.2.102:0/3153717193 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ac80eec20 con 0x7f6ac80a2560 2021-02-10T11:40:21.552 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:21.543 7f6acdf95700 1 --2- 172.21.2.102:0/3153717193 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6ac80a2560 0x7f6ac80b4fc0 secure :-1 s=READY pgs=3360 cs=0 l=1 rev1=1 rx=0x7f6abc009ed0 tx=0x7f6abc00b9e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:21.552 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:21.543 7f6aba7fc700 1 -- 172.21.2.102:0/3153717193 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6abc006030 con 0x7f6ac80a2560 2021-02-10T11:40:21.552 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:21.544 7f6aba7fc700 1 -- 172.21.2.102:0/3153717193 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6abc0090b0 con 0x7f6ac80a2560 2021-02-10T11:40:21.552 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:21.544 7f6ad0179700 1 -- 172.21.2.102:0/3153717193 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ac80eec20 con 0x7f6ac80a2560 2021-02-10T11:40:21.552 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:21.544 7f6aba7fc700 1 -- 172.21.2.102:0/3153717193 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6abc0052e0 con 0x7f6ac80a2560 2021-02-10T11:40:21.552 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:21.544 7f6ad0179700 1 -- 172.21.2.102:0/3153717193 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ac80eec20 con 0x7f6ac80a2560 2021-02-10T11:40:21.552 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:21.544 7f6aba7fc700 1 -- 172.21.2.102:0/3153717193 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f6abc009290 con 0x7f6ac80a2560 2021-02-10T11:40:21.552 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:21.544 7f6aba7fc700 1 --2- 172.21.2.102:0/3153717193 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6ab00186d0 0x7f6ab001abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:21.552 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:21.544 7f6aba7fc700 1 -- 172.21.2.102:0/3153717193 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(193..193 src has 1..193) v4 ==== 4753+0+0 (secure 0 0 0) 0x7f6abc032460 con 0x7f6ac80a2560 2021-02-10T11:40:21.552 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:21.544 7f6acd794700 1 --2- 172.21.2.102:0/3153717193 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6ab00186d0 0x7f6ab001abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:21.553 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:21.545 7f6acd794700 1 --2- 172.21.2.102:0/3153717193 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6ab00186d0 0x7f6ab001abd0 crc :-1 s=READY pgs=1463 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:21.577 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:21.575 7f6ad0179700 1 -- 172.21.2.102:0/3153717193 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ac80f0b10 con 0x7f6ac80a2560 2021-02-10T11:40:21.579 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:21.577 7f6aba7fc700 1 -- 172.21.2.102:0/3153717193 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f6abc0092c0 con 0x7f6ac80a2560 2021-02-10T11:40:21.703 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:21.700 7f6ad0179700 1 -- 172.21.2.102:0/3153717193 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f6ac80f0b10 con 0x7f6ac80a2560 2021-02-10T11:40:21.703 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:21.701 7f6aba7fc700 1 -- 172.21.2.102:0/3153717193 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v193) v1 ==== 84+0+64 (secure 0 0 0) 0x7f6abc0090b0 con 0x7f6ac80a2560 2021-02-10T11:40:21.703 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:40:21.703 INFO:teuthology.orchestra.run.gibba002.stdout:[ 2021-02-10T11:40:21.703 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:40:21.704 INFO:teuthology.orchestra.run.gibba002.stdout: "poolnum": 1, 2021-02-10T11:40:21.704 INFO:teuthology.orchestra.run.gibba002.stdout: "poolname": "rbd" 2021-02-10T11:40:21.704 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:40:21.704 INFO:teuthology.orchestra.run.gibba002.stdout:] 2021-02-10T11:40:21.705 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:21.702 7f6ad0179700 1 -- 172.21.2.102:0/3153717193 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6ab00186d0 msgr2=0x7f6ab001abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:21.705 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:21.702 7f6ad0179700 1 --2- 172.21.2.102:0/3153717193 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6ab00186d0 0x7f6ab001abd0 crc :-1 s=READY pgs=1463 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:21.705 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:21.703 7f6ad0179700 1 -- 172.21.2.102:0/3153717193 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6ac80a2560 msgr2=0x7f6ac80b4fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:21.705 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:21.703 7f6ad0179700 1 --2- 172.21.2.102:0/3153717193 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6ac80a2560 0x7f6ac80b4fc0 secure :-1 s=READY pgs=3360 cs=0 l=1 rev1=1 rx=0x7f6abc009ed0 tx=0x7f6abc00b9e0).stop 2021-02-10T11:40:21.705 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:21.703 7f6ad0179700 1 -- 172.21.2.102:0/3153717193 shutdown_connections 2021-02-10T11:40:21.705 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:21.703 7f6ad0179700 1 --2- 172.21.2.102:0/3153717193 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6ac80ee200 0x7f6ac80ee600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:21.706 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:21.703 7f6ad0179700 1 --2- 172.21.2.102:0/3153717193 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6ab00186d0 0x7f6ab001abd0 unknown :-1 s=CLOSED pgs=1463 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:21.706 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:21.703 7f6ad0179700 1 --2- 172.21.2.102:0/3153717193 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6ac80b55e0 0x7f6ac80b59e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:21.706 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:21.703 7f6ad0179700 1 --2- 172.21.2.102:0/3153717193 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6ac80a2560 0x7f6ac80b4fc0 unknown :-1 s=CLOSED pgs=3360 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:21.706 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:21.703 7f6ad0179700 1 -- 172.21.2.102:0/3153717193 >> 172.21.2.102:0/3153717193 conn(0x7f6ac80ac990 msgr2=0x7f6ac8091b30 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:21.706 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:21.703 7f6ad0179700 1 -- 172.21.2.102:0/3153717193 shutdown_connections 2021-02-10T11:40:21.706 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:21.703 7f6ad0179700 1 -- 172.21.2.102:0/3153717193 wait complete. 2021-02-10T11:40:21.713 INFO:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2021-02-10T11:40:21.713 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool create cephfs_metadata 8 2021-02-10T11:40:22.148 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:22.129 7ff38f5b8700 1 Processor -- start 2021-02-10T11:40:22.148 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:22.129 7ff38f5b8700 1 -- start start 2021-02-10T11:40:22.148 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:22.130 7ff38f5b8700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7ff3880d1870 0x7ff3881b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:22.148 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:22.130 7ff38f5b8700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff3881b0ae0 0x7ff388108940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:22.148 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:22.130 7ff38f5b8700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7ff3880d1c70 0x7ff388108f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:22.148 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:22.130 7ff38f5b8700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3880fe1c0 con 0x7ff3880d1c70 2021-02-10T11:40:22.148 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:22.130 7ff38f5b8700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff388109a60 con 0x7ff3880d1870 2021-02-10T11:40:22.148 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:22.130 7ff38f5b8700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff388088070 con 0x7ff3881b0ae0 2021-02-10T11:40:22.148 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:22.130 7ff38dbd5700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7ff3880d1c70 0x7ff388108f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:22.148 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:22.130 7ff38dbd5700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7ff3880d1c70 0x7ff388108f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:41178/0 (socket says 172.21.2.102:41178) 2021-02-10T11:40:22.149 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:22.130 7ff38dbd5700 1 -- 172.21.2.102:0/2865897560 learned_addr learned my addr 172.21.2.102:0/2865897560 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:40:22.149 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:22.130 7ff38dbd5700 1 -- 172.21.2.102:0/2865897560 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff3881b0ae0 msgr2=0x7ff388108940 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:40:22.149 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:22.130 7ff38dbd5700 1 --2- 172.21.2.102:0/2865897560 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff3881b0ae0 0x7ff388108940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:22.149 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:22.130 7ff38dbd5700 1 -- 172.21.2.102:0/2865897560 >> v2:172.21.2.113:3300/0 conn(0x7ff3880d1870 msgr2=0x7ff3881b04c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:40:22.149 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:22.130 7ff38dbd5700 1 --2- 172.21.2.102:0/2865897560 >> v2:172.21.2.113:3300/0 conn(0x7ff3880d1870 0x7ff3881b04c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:22.149 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:22.130 7ff38dbd5700 1 -- 172.21.2.102:0/2865897560 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff388109580 con 0x7ff3880d1c70 2021-02-10T11:40:22.149 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:22.130 7ff38dbd5700 1 --2- 172.21.2.102:0/2865897560 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff3880d1c70 0x7ff388108f60 secure :-1 s=READY pgs=3361 cs=0 l=1 rev1=1 rx=0x7ff384006b80 tx=0x7ff384009b90).ready entity=mon.0 client_cookie=a702d2deb8bb8851 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:22.149 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:22.131 7ff37ffff700 1 -- 172.21.2.102:0/2865897560 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff3840022e0 con 0x7ff3880d1c70 2021-02-10T11:40:22.149 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:22.131 7ff37ffff700 1 -- 172.21.2.102:0/2865897560 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff384002620 con 0x7ff3880d1c70 2021-02-10T11:40:22.149 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:22.131 7ff37ffff700 1 -- 172.21.2.102:0/2865897560 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff384002b00 con 0x7ff3880d1c70 2021-02-10T11:40:22.149 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:22.131 7ff38f5b8700 1 -- 172.21.2.102:0/2865897560 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff3880d1c70 msgr2=0x7ff388108f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:22.149 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:22.131 7ff38f5b8700 1 --2- 172.21.2.102:0/2865897560 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff3880d1c70 0x7ff388108f60 secure :-1 s=READY pgs=3361 cs=0 l=1 rev1=1 rx=0x7ff384006b80 tx=0x7ff384009b90).stop 2021-02-10T11:40:22.150 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:22.131 7ff38f5b8700 1 -- 172.21.2.102:0/2865897560 shutdown_connections 2021-02-10T11:40:22.150 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:22.131 7ff38f5b8700 1 --2- 172.21.2.102:0/2865897560 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff3881b0ae0 0x7ff388108940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:22.150 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:22.131 7ff38f5b8700 1 --2- 172.21.2.102:0/2865897560 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff3880d1c70 0x7ff388108f60 unknown :-1 s=CLOSED pgs=3361 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:22.150 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:22.131 7ff38f5b8700 1 --2- 172.21.2.102:0/2865897560 >> v2:172.21.2.113:3300/0 conn(0x7ff3880d1870 0x7ff3881b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:22.150 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:22.131 7ff38f5b8700 1 -- 172.21.2.102:0/2865897560 >> 172.21.2.102:0/2865897560 conn(0x7ff3880ce420 msgr2=0x7ff3880d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:22.150 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:22.132 7ff38f5b8700 1 -- 172.21.2.102:0/2865897560 shutdown_connections 2021-02-10T11:40:22.150 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:22.132 7ff38f5b8700 1 -- 172.21.2.102:0/2865897560 wait complete. 2021-02-10T11:40:22.152 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:22.146 7ff38f5b8700 1 Processor -- start 2021-02-10T11:40:22.152 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:22.146 7ff38f5b8700 1 -- start start 2021-02-10T11:40:22.152 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:22.146 7ff38f5b8700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff3880c3820 0x7ff388084cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:22.152 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:22.146 7ff38f5b8700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff3880a0290 0x7ff3880852d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:22.152 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:22.146 7ff38f5b8700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff388086270 0x7ff388086670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:22.152 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:22.146 7ff38f5b8700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3880c0700 con 0x7ff3880a0290 2021-02-10T11:40:22.152 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:22.146 7ff38f5b8700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3880858f0 con 0x7ff388086270 2021-02-10T11:40:22.152 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:22.146 7ff38f5b8700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff388085b10 con 0x7ff3880c3820 2021-02-10T11:40:22.152 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:22.146 7ff38cbd3700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff3880a0290 0x7ff3880852d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:22.152 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:22.146 7ff38cbd3700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff3880a0290 0x7ff3880852d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:41182/0 (socket says 172.21.2.102:41182) 2021-02-10T11:40:22.152 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:22.146 7ff38cbd3700 1 -- 172.21.2.102:0/1434793184 learned_addr learned my addr 172.21.2.102:0/1434793184 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:40:22.153 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:22.147 7ff38cbd3700 1 -- 172.21.2.102:0/1434793184 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff3880c3820 msgr2=0x7ff388084cb0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:40:22.153 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:22.147 7ff38cbd3700 1 --2- 172.21.2.102:0/1434793184 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff3880c3820 0x7ff388084cb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:22.153 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:22.147 7ff38cbd3700 1 -- 172.21.2.102:0/1434793184 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff388086270 msgr2=0x7ff388086670 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:40:22.153 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:22.147 7ff38cbd3700 1 --2- 172.21.2.102:0/1434793184 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff388086270 0x7ff388086670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:22.153 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:22.147 7ff38cbd3700 1 -- 172.21.2.102:0/1434793184 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff388086c90 con 0x7ff3880a0290 2021-02-10T11:40:22.153 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:22.147 7ff38cbd3700 1 --2- 172.21.2.102:0/1434793184 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff3880a0290 0x7ff3880852d0 secure :-1 s=READY pgs=3362 cs=0 l=1 rev1=1 rx=0x7ff380009960 tx=0x7ff38000bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:22.153 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:22.147 7ff37dffb700 1 -- 172.21.2.102:0/1434793184 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff380003030 con 0x7ff3880a0290 2021-02-10T11:40:22.153 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:22.147 7ff37dffb700 1 -- 172.21.2.102:0/1434793184 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff380001310 con 0x7ff3880a0290 2021-02-10T11:40:22.153 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:22.147 7ff38f5b8700 1 -- 172.21.2.102:0/1434793184 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff388086c90 con 0x7ff3880a0290 2021-02-10T11:40:22.153 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:22.147 7ff38f5b8700 1 -- 172.21.2.102:0/1434793184 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff388086c90 con 0x7ff3880a0290 2021-02-10T11:40:22.153 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:22.147 7ff37dffb700 1 -- 172.21.2.102:0/1434793184 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff3800017f0 con 0x7ff3880a0290 2021-02-10T11:40:22.153 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:22.148 7ff37dffb700 1 -- 172.21.2.102:0/1434793184 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ff380001550 con 0x7ff3880a0290 2021-02-10T11:40:22.153 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:22.148 7ff37dffb700 1 --2- 172.21.2.102:0/1434793184 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff3680186c0 0x7ff36801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:22.153 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:22.148 7ff37dffb700 1 -- 172.21.2.102:0/1434793184 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(193..193 src has 1..193) v4 ==== 4753+0+0 (secure 0 0 0) 0x7ff3800327d0 con 0x7ff3880a0290 2021-02-10T11:40:22.154 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:22.148 7ff38d3d4700 1 --2- 172.21.2.102:0/1434793184 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff3680186c0 0x7ff36801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:22.154 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:22.148 7ff38d3d4700 1 --2- 172.21.2.102:0/1434793184 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff3680186c0 0x7ff36801abc0 crc :-1 s=READY pgs=1464 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:22.182 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:22.180 7ff38f5b8700 1 -- 172.21.2.102:0/1434793184 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff388082a70 con 0x7ff3880a0290 2021-02-10T11:40:22.184 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:22.182 7ff37dffb700 1 -- 172.21.2.102:0/1434793184 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ff38000be30 con 0x7ff3880a0290 2021-02-10T11:40:22.313 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:22.311 7ff38f5b8700 1 -- 172.21.2.102:0/1434793184 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"} v 0) v1 -- 0x7ff388082a70 con 0x7ff3880a0290 2021-02-10T11:40:22.724 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:22.721 7ff37dffb700 1 -- 172.21.2.102:0/1434793184 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"}]=0 pool 'cephfs_metadata' created v194) v1 ==== 133+0+0 (secure 0 0 0) 0x7ff380001310 con 0x7ff3880a0290 2021-02-10T11:40:22.727 INFO:teuthology.orchestra.run.gibba002.stderr:pool 'cephfs_metadata' created 2021-02-10T11:40:22.729 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:22.727 7ff38f5b8700 1 -- 172.21.2.102:0/1434793184 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff3680186c0 msgr2=0x7ff36801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:22.730 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:22.727 7ff38f5b8700 1 --2- 172.21.2.102:0/1434793184 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff3680186c0 0x7ff36801abc0 crc :-1 s=READY pgs=1464 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:22.730 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:22.727 7ff38f5b8700 1 -- 172.21.2.102:0/1434793184 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff3880a0290 msgr2=0x7ff3880852d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:22.730 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:22.727 7ff38f5b8700 1 --2- 172.21.2.102:0/1434793184 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff3880a0290 0x7ff3880852d0 secure :-1 s=READY pgs=3362 cs=0 l=1 rev1=1 rx=0x7ff380009960 tx=0x7ff38000bcf0).stop 2021-02-10T11:40:22.730 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:22.727 7ff38f5b8700 1 -- 172.21.2.102:0/1434793184 shutdown_connections 2021-02-10T11:40:22.730 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:22.727 7ff38f5b8700 1 --2- 172.21.2.102:0/1434793184 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff3680186c0 0x7ff36801abc0 unknown :-1 s=CLOSED pgs=1464 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:22.730 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:22.727 7ff38f5b8700 1 --2- 172.21.2.102:0/1434793184 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff388086270 0x7ff388086670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:22.730 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:22.727 7ff38f5b8700 1 --2- 172.21.2.102:0/1434793184 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff3880a0290 0x7ff3880852d0 unknown :-1 s=CLOSED pgs=3362 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:22.730 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:22.727 7ff38f5b8700 1 --2- 172.21.2.102:0/1434793184 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff3880c3820 0x7ff388084cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:22.730 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:22.727 7ff38f5b8700 1 -- 172.21.2.102:0/1434793184 >> 172.21.2.102:0/1434793184 conn(0x7ff3880872c0 msgr2=0x7ff388095700 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:22.730 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:22.727 7ff38f5b8700 1 -- 172.21.2.102:0/1434793184 shutdown_connections 2021-02-10T11:40:22.731 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:22.727 7ff38f5b8700 1 -- 172.21.2.102:0/1434793184 wait complete. 2021-02-10T11:40:22.738 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool create cephfs_data 8 2021-02-10T11:40:23.173 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:23.153 7f1809def700 1 Processor -- start 2021-02-10T11:40:23.173 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:23.153 7f1809def700 1 -- start start 2021-02-10T11:40:23.173 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:23.154 7f1809def700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f18040d1870 0x7f18041b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:23.173 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:23.154 7f1809def700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f18041b0ae0 0x7f1804108940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:23.173 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:23.154 7f1809def700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1804109a60 con 0x7f18040d1c70 2021-02-10T11:40:23.173 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:23.154 7f1809def700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18040e5ef0 con 0x7f18041b0ae0 2021-02-10T11:40:23.174 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:23.154 7f1809def700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1804109750 con 0x7f18040d1870 2021-02-10T11:40:23.174 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:23.154 7f1803fff700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f18040d1c70 0x7f1804108f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:33412/0 (socket says 172.21.2.102:33412) 2021-02-10T11:40:23.174 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:23.154 7f1803fff700 1 -- 172.21.2.102:0/2888065522 learned_addr learned my addr 172.21.2.102:0/2888065522 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:40:23.174 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:23.155 7f18027fc700 1 -- 172.21.2.102:0/2888065522 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1632983980 0 0) 0x7f17fc001500 con 0x7f18040d1c70 2021-02-10T11:40:23.174 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:23.155 7f1802ffd700 1 --2- 172.21.2.102:0/2888065522 >> v2:172.21.2.113:3300/0 conn(0x7f18041b0ae0 0x7f1804108940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:23.174 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:23.155 7f18037fe700 1 --2- 172.21.2.102:0/2888065522 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f18040d1870 0x7f18041b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:23.174 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:23.155 7f18027fc700 1 -- 172.21.2.102:0/2888065522 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f17ec002fe0 con 0x7f18040d1c70 2021-02-10T11:40:23.174 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:23.155 7f18027fc700 1 -- 172.21.2.102:0/2888065522 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1904028855 0 0) 0x7f17fc001500 con 0x7f18040d1c70 2021-02-10T11:40:23.174 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:23.155 7f18037fe700 1 -- 172.21.2.102:0/2888065522 >> v2:172.21.2.113:3300/0 conn(0x7f18041b0ae0 msgr2=0x7f1804108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:23.174 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:23.155 7f18037fe700 1 --2- 172.21.2.102:0/2888065522 >> v2:172.21.2.113:3300/0 conn(0x7f18041b0ae0 0x7f1804108940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:23.174 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:23.155 7f18037fe700 1 -- 172.21.2.102:0/2888065522 >> v1:172.21.2.102:6789/0 conn(0x7f18040d1c70 legacy=0x7f1804108f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:23.174 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:23.155 7f18037fe700 1 -- 172.21.2.102:0/2888065522 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1804088070 con 0x7f18040d1870 2021-02-10T11:40:23.174 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:23.155 7f1802ffd700 1 --2- 172.21.2.102:0/2888065522 >> v2:172.21.2.113:3300/0 conn(0x7f18041b0ae0 0x7f1804108940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:40:23.175 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:23.155 7f18037fe700 1 --2- 172.21.2.102:0/2888065522 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f18040d1870 0x7f18041b04c0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0x7f17f4006650 tx=0x7f17f4009a80).ready entity=mon.2 client_cookie=1bc6d1ed44e5954c server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:23.175 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:23.156 7f18027fc700 1 -- 172.21.2.102:0/2888065522 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f17f40009b0 con 0x7f18040d1870 2021-02-10T11:40:23.175 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:23.156 7f18027fc700 1 -- 172.21.2.102:0/2888065522 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17f400c0c0 con 0x7f18040d1870 2021-02-10T11:40:23.175 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:23.156 7f18027fc700 1 -- 172.21.2.102:0/2888065522 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f17f400c5e0 con 0x7f18040d1870 2021-02-10T11:40:23.175 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:23.156 7f1809def700 1 -- 172.21.2.102:0/2888065522 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f18040d1870 msgr2=0x7f18041b04c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:23.175 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:23.156 7f1809def700 1 --2- 172.21.2.102:0/2888065522 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f18040d1870 0x7f18041b04c0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0x7f17f4006650 tx=0x7f17f4009a80).stop 2021-02-10T11:40:23.175 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:23.156 7f1809def700 1 -- 172.21.2.102:0/2888065522 shutdown_connections 2021-02-10T11:40:23.175 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:23.156 7f1809def700 1 --2- 172.21.2.102:0/2888065522 >> v2:172.21.2.113:3300/0 conn(0x7f18041b0ae0 0x7f1804108940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:23.175 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:23.156 7f1809def700 1 --2- 172.21.2.102:0/2888065522 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f18040d1870 0x7f18041b04c0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:23.175 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:23.156 7f1809def700 1 -- 172.21.2.102:0/2888065522 >> 172.21.2.102:0/2888065522 conn(0x7f18040ce420 msgr2=0x7f18040d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:23.175 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:23.156 7f1809def700 1 -- 172.21.2.102:0/2888065522 shutdown_connections 2021-02-10T11:40:23.175 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:23.156 7f1809def700 1 -- 172.21.2.102:0/2888065522 wait complete. 2021-02-10T11:40:23.175 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:23.171 7f1809def700 1 Processor -- start 2021-02-10T11:40:23.176 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:23.171 7f1809def700 1 -- start start 2021-02-10T11:40:23.177 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:23.171 7f1809def700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f18040a2580 0x7f18040a1cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:23.178 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:23.171 7f1809def700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1804094c40 0x7f18041a75e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:23.178 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:23.171 7f1809def700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1804095040 0x7f18041a7c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:23.178 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:23.171 7f1809def700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1804188180 con 0x7f1804094c40 2021-02-10T11:40:23.178 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:23.171 7f1809def700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1804187e70 con 0x7f18040a2580 2021-02-10T11:40:23.178 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:23.171 7f1809def700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18041a8220 con 0x7f1804095040 2021-02-10T11:40:23.178 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:23.172 7f1802ffd700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1804094c40 0x7f18041a75e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:23.178 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:23.172 7f1802ffd700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1804094c40 0x7f18041a75e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:41192/0 (socket says 172.21.2.102:41192) 2021-02-10T11:40:23.178 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:23.172 7f1802ffd700 1 -- 172.21.2.102:0/1487566899 learned_addr learned my addr 172.21.2.102:0/1487566899 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:40:23.178 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:23.172 7f18037fe700 1 --2- 172.21.2.102:0/1487566899 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f18040a2580 0x7f18040a1cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:23.178 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:23.172 7f1803fff700 1 --2- 172.21.2.102:0/1487566899 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1804095040 0x7f18041a7c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:23.178 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:23.172 7f1802ffd700 1 -- 172.21.2.102:0/1487566899 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1804095040 msgr2=0x7f18041a7c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:23.179 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:23.172 7f1802ffd700 1 --2- 172.21.2.102:0/1487566899 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1804095040 0x7f18041a7c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:23.179 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:23.172 7f1802ffd700 1 -- 172.21.2.102:0/1487566899 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f18040a2580 msgr2=0x7f18040a1cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:23.179 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:23.172 7f1802ffd700 1 --2- 172.21.2.102:0/1487566899 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f18040a2580 0x7f18040a1cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:23.179 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:23.172 7f1802ffd700 1 -- 172.21.2.102:0/1487566899 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1804188700 con 0x7f1804094c40 2021-02-10T11:40:23.179 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:23.172 7f1803fff700 1 --2- 172.21.2.102:0/1487566899 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1804095040 0x7f18041a7c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:40:23.179 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:23.172 7f18037fe700 1 --2- 172.21.2.102:0/1487566899 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f18040a2580 0x7f18040a1cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:40:23.179 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:23.172 7f1802ffd700 1 --2- 172.21.2.102:0/1487566899 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1804094c40 0x7f18041a75e0 secure :-1 s=READY pgs=3364 cs=0 l=1 rev1=1 rx=0x7f17f8006f40 tx=0x7f17f800b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:23.179 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:23.172 7f17ebfff700 1 -- 172.21.2.102:0/1487566899 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f17f800bd80 con 0x7f1804094c40 2021-02-10T11:40:23.179 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:23.172 7f17ebfff700 1 -- 172.21.2.102:0/1487566899 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17f8000c90 con 0x7f1804094c40 2021-02-10T11:40:23.179 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:23.172 7f17ebfff700 1 -- 172.21.2.102:0/1487566899 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f17f80022b0 con 0x7f1804094c40 2021-02-10T11:40:23.179 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:23.172 7f1809def700 1 -- 172.21.2.102:0/1487566899 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1804188700 con 0x7f1804094c40 2021-02-10T11:40:23.180 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:23.172 7f1809def700 1 -- 172.21.2.102:0/1487566899 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1804188700 con 0x7f1804094c40 2021-02-10T11:40:23.180 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:23.173 7f17ebfff700 1 -- 172.21.2.102:0/1487566899 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f17f8006b00 con 0x7f1804094c40 2021-02-10T11:40:23.180 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:23.173 7f17ebfff700 1 --2- 172.21.2.102:0/1487566899 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f17e40186c0 0x7f17e401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:23.180 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:23.173 7f17ebfff700 1 -- 172.21.2.102:0/1487566899 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(194..194 src has 1..194) v4 ==== 5107+0+0 (secure 0 0 0) 0x7f17f8032350 con 0x7f1804094c40 2021-02-10T11:40:23.180 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:23.173 7f18037fe700 1 --2- 172.21.2.102:0/1487566899 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f17e40186c0 0x7f17e401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:23.180 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:23.173 7f18037fe700 1 --2- 172.21.2.102:0/1487566899 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f17e40186c0 0x7f17e401abc0 crc :-1 s=READY pgs=1465 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:23.206 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:23.204 7f1809def700 1 -- 172.21.2.102:0/1487566899 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1804082980 con 0x7f1804094c40 2021-02-10T11:40:23.208 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:23.206 7f17ebfff700 1 -- 172.21.2.102:0/1487566899 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f17f8024680 con 0x7f1804094c40 2021-02-10T11:40:23.340 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:23.338 7f1809def700 1 -- 172.21.2.102:0/1487566899 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_data"} v 0) v1 -- 0x7f1804082980 con 0x7f1804094c40 2021-02-10T11:40:23.726 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:23.724 7f17ebfff700 1 -- 172.21.2.102:0/1487566899 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_data"}]=0 pool 'cephfs_data' created v195) v1 ==== 125+0+0 (secure 0 0 0) 0x7f17f8024680 con 0x7f1804094c40 2021-02-10T11:40:23.755 INFO:teuthology.orchestra.run.gibba002.stderr:pool 'cephfs_data' created 2021-02-10T11:40:23.757 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:23.755 7f1809def700 1 -- 172.21.2.102:0/1487566899 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f17e40186c0 msgr2=0x7f17e401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:23.758 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:23.755 7f1809def700 1 --2- 172.21.2.102:0/1487566899 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f17e40186c0 0x7f17e401abc0 crc :-1 s=READY pgs=1465 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:23.758 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:23.755 7f1809def700 1 -- 172.21.2.102:0/1487566899 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1804094c40 msgr2=0x7f18041a75e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:23.758 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:23.755 7f1809def700 1 --2- 172.21.2.102:0/1487566899 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1804094c40 0x7f18041a75e0 secure :-1 s=READY pgs=3364 cs=0 l=1 rev1=1 rx=0x7f17f8006f40 tx=0x7f17f800b990).stop 2021-02-10T11:40:23.758 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:23.755 7f1809def700 1 -- 172.21.2.102:0/1487566899 shutdown_connections 2021-02-10T11:40:23.758 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:23.755 7f1809def700 1 --2- 172.21.2.102:0/1487566899 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1804095040 0x7f18041a7c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:23.758 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:23.755 7f1809def700 1 --2- 172.21.2.102:0/1487566899 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1804094c40 0x7f18041a75e0 unknown :-1 s=CLOSED pgs=3364 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:23.759 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:23.755 7f1809def700 1 --2- 172.21.2.102:0/1487566899 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f17e40186c0 0x7f17e401abc0 unknown :-1 s=CLOSED pgs=1465 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:23.759 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:23.755 7f1809def700 1 --2- 172.21.2.102:0/1487566899 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f18040a2580 0x7f18040a1cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:23.759 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:23.755 7f1809def700 1 -- 172.21.2.102:0/1487566899 >> 172.21.2.102:0/1487566899 conn(0x7f18040a2b30 msgr2=0x7f180409ce60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:23.759 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:23.755 7f1809def700 1 -- 172.21.2.102:0/1487566899 shutdown_connections 2021-02-10T11:40:23.759 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:23.755 7f1809def700 1 -- 172.21.2.102:0/1487566899 wait complete. 2021-02-10T11:40:23.766 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2021-02-10T11:40:24.199 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:24.180 7f0eacef9700 1 Processor -- start 2021-02-10T11:40:24.200 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:24.181 7f0eacef9700 1 -- start start 2021-02-10T11:40:24.200 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:24.181 7f0eacef9700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0ea80d1880 0x7f0ea81b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:24.200 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:24.181 7f0eacef9700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f0ea80cff80 0x7f0ea8109140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:24.200 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:24.181 7f0eacef9700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ea8109a70 con 0x7f0ea81b0af0 2021-02-10T11:40:24.200 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:24.181 7f0eacef9700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ea8109760 con 0x7f0ea80cff80 2021-02-10T11:40:24.200 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:24.181 7f0eacef9700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ea8088070 con 0x7f0ea80d1880 2021-02-10T11:40:24.200 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:24.181 7f0ea7314700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f0ea80cff80 0x7f0ea8109140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:24.200 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:24.181 7f0ea7314700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f0ea80cff80 0x7f0ea8109140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:41206/0 (socket says 172.21.2.102:41206) 2021-02-10T11:40:24.200 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:24.181 7f0ea7314700 1 -- 172.21.2.102:0/200638091 learned_addr learned my addr 172.21.2.102:0/200638091 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:40:24.200 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:24.182 7f0ea7314700 1 -- 172.21.2.102:0/200638091 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0ea80d1880 msgr2=0x7f0ea81b04d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:40:24.200 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:24.182 7f0ea7314700 1 --2- 172.21.2.102:0/200638091 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0ea80d1880 0x7f0ea81b04d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:24.200 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:24.182 7f0ea7314700 1 -- 172.21.2.102:0/200638091 >> v1:172.21.2.113:6789/0 conn(0x7f0ea81b0af0 legacy=0x7f0ea8108950 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:40:24.201 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:24.182 7f0ea7314700 1 -- 172.21.2.102:0/200638091 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ea8088260 con 0x7f0ea80cff80 2021-02-10T11:40:24.201 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:24.182 7f0ea7314700 1 --2- 172.21.2.102:0/200638091 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0ea80cff80 0x7f0ea8109140 secure :-1 s=READY pgs=3365 cs=0 l=1 rev1=1 rx=0x7f0ea0006f00 tx=0x7f0ea0009e90).ready entity=mon.0 client_cookie=a7ed03a7d3647e5b server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:24.201 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:24.182 7f0ea5b11700 1 -- 172.21.2.102:0/200638091 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0ea000b300 con 0x7f0ea80cff80 2021-02-10T11:40:24.201 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:24.182 7f0ea5b11700 1 -- 172.21.2.102:0/200638091 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ea000b580 con 0x7f0ea80cff80 2021-02-10T11:40:24.201 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:24.182 7f0ea5b11700 1 -- 172.21.2.102:0/200638091 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0ea000b9d0 con 0x7f0ea80cff80 2021-02-10T11:40:24.201 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:24.182 7f0eacef9700 1 -- 172.21.2.102:0/200638091 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0ea80cff80 msgr2=0x7f0ea8109140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:24.201 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:24.182 7f0eacef9700 1 --2- 172.21.2.102:0/200638091 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0ea80cff80 0x7f0ea8109140 secure :-1 s=READY pgs=3365 cs=0 l=1 rev1=1 rx=0x7f0ea0006f00 tx=0x7f0ea0009e90).stop 2021-02-10T11:40:24.201 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:24.183 7f0eacef9700 1 -- 172.21.2.102:0/200638091 shutdown_connections 2021-02-10T11:40:24.201 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:24.183 7f0eacef9700 1 --2- 172.21.2.102:0/200638091 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0ea80cff80 0x7f0ea8109140 unknown :-1 s=CLOSED pgs=3365 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:24.201 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:24.183 7f0eacef9700 1 --2- 172.21.2.102:0/200638091 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0ea80d1880 0x7f0ea81b04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:24.201 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:24.183 7f0eacef9700 1 -- 172.21.2.102:0/200638091 >> 172.21.2.102:0/200638091 conn(0x7f0ea80ce430 msgr2=0x7f0ea80d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:24.201 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:24.183 7f0eacef9700 1 -- 172.21.2.102:0/200638091 shutdown_connections 2021-02-10T11:40:24.201 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:24.183 7f0eacef9700 1 -- 172.21.2.102:0/200638091 wait complete. 2021-02-10T11:40:24.203 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:24.198 7f0eacef9700 1 Processor -- start 2021-02-10T11:40:24.203 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:24.198 7f0eacef9700 1 -- start start 2021-02-10T11:40:24.204 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:24.198 7f0eacef9700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0ea809ef20 0x7f0ea8187e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:24.204 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:24.198 7f0eacef9700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0ea81884a0 0x7f0ea81888a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:24.204 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:24.198 7f0eacef9700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0ea8189840 0x7f0ea8189c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:24.204 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:24.198 7f0eacef9700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ea81b3c10 con 0x7f0ea809ef20 2021-02-10T11:40:24.204 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:24.198 7f0eacef9700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ea81964f0 con 0x7f0ea8189840 2021-02-10T11:40:24.204 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:24.198 7f0eacef9700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ea8188ec0 con 0x7f0ea81884a0 2021-02-10T11:40:24.204 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:24.198 7f0ea6b13700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0ea809ef20 0x7f0ea8187e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:24.204 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:24.198 7f0ea6b13700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0ea809ef20 0x7f0ea8187e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:41212/0 (socket says 172.21.2.102:41212) 2021-02-10T11:40:24.204 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:24.198 7f0ea6b13700 1 -- 172.21.2.102:0/1897174574 learned_addr learned my addr 172.21.2.102:0/1897174574 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:40:24.204 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:24.199 7f0ea7314700 1 --2- 172.21.2.102:0/1897174574 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0ea8189840 0x7f0ea8189c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:24.204 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:24.199 7f0ea6312700 1 --2- 172.21.2.102:0/1897174574 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0ea81884a0 0x7f0ea81888a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:24.204 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:24.199 7f0ea6b13700 1 -- 172.21.2.102:0/1897174574 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0ea81884a0 msgr2=0x7f0ea81888a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:24.204 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:24.199 7f0ea6b13700 1 --2- 172.21.2.102:0/1897174574 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0ea81884a0 0x7f0ea81888a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:24.205 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:24.199 7f0ea6b13700 1 -- 172.21.2.102:0/1897174574 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0ea8189840 msgr2=0x7f0ea8189c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:24.205 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:24.199 7f0ea6b13700 1 --2- 172.21.2.102:0/1897174574 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0ea8189840 0x7f0ea8189c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:24.205 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:24.199 7f0ea6b13700 1 -- 172.21.2.102:0/1897174574 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ea809f320 con 0x7f0ea809ef20 2021-02-10T11:40:24.205 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:24.199 7f0ea6312700 1 --2- 172.21.2.102:0/1897174574 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0ea81884a0 0x7f0ea81888a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:40:24.205 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:24.199 7f0ea6b13700 1 --2- 172.21.2.102:0/1897174574 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0ea809ef20 0x7f0ea8187e80 secure :-1 s=READY pgs=3366 cs=0 l=1 rev1=1 rx=0x7f0e9000b5c0 tx=0x7f0e9000b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:24.205 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:24.199 7f0e9f7fe700 1 -- 172.21.2.102:0/1897174574 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0e900009b0 con 0x7f0ea809ef20 2021-02-10T11:40:24.205 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:24.199 7f0e9f7fe700 1 -- 172.21.2.102:0/1897174574 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e90000cc0 con 0x7f0ea809ef20 2021-02-10T11:40:24.205 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:24.199 7f0eacef9700 1 -- 172.21.2.102:0/1897174574 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ea809f320 con 0x7f0ea809ef20 2021-02-10T11:40:24.205 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:24.199 7f0eacef9700 1 -- 172.21.2.102:0/1897174574 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ea809f320 con 0x7f0ea809ef20 2021-02-10T11:40:24.205 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:24.199 7f0e9f7fe700 1 -- 172.21.2.102:0/1897174574 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0e90006240 con 0x7f0ea809ef20 2021-02-10T11:40:24.205 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:24.200 7f0e9f7fe700 1 -- 172.21.2.102:0/1897174574 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f0e90006240 con 0x7f0ea809ef20 2021-02-10T11:40:24.205 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:24.200 7f0e9f7fe700 1 --2- 172.21.2.102:0/1897174574 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0e8c0186c0 0x7f0e8c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:24.205 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:24.200 7f0e9f7fe700 1 -- 172.21.2.102:0/1897174574 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(195..195 src has 1..195) v4 ==== 5457+0+0 (secure 0 0 0) 0x7f0e900324b0 con 0x7f0ea809ef20 2021-02-10T11:40:24.206 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:24.200 7f0ea6312700 1 --2- 172.21.2.102:0/1897174574 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0e8c0186c0 0x7f0e8c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:24.206 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:24.200 7f0ea6312700 1 --2- 172.21.2.102:0/1897174574 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0e8c0186c0 0x7f0e8c01abc0 crc :-1 s=READY pgs=1468 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:24.233 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:24.231 7f0eacef9700 1 -- 172.21.2.102:0/1897174574 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ea81b3c10 con 0x7f0ea809ef20 2021-02-10T11:40:24.236 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:24.234 7f0e9f7fe700 1 -- 172.21.2.102:0/1897174574 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f0e90006030 con 0x7f0ea809ef20 2021-02-10T11:40:24.365 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:24.363 7f0eacef9700 1 -- 172.21.2.102:0/1897174574 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"data": "cephfs_data", "prefix": "fs new", "force": true, "fs_name": "cephfs", "metadata": "cephfs_metadata"} v 0) v1 -- 0x7f0ea81b3c10 con 0x7f0ea809ef20 2021-02-10T11:40:24.369 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:24.367 7f0e9f7fe700 1 -- 172.21.2.102:0/1897174574 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"data": "cephfs_data", "prefix": "fs new", "force": true, "fs_name": "cephfs", "metadata": "cephfs_metadata"}]=0 new fs with metadata pool 42 and data pool 43 v170) v1 ==== 189+0+0 (secure 0 0 0) 0x7f0e90006240 con 0x7f0ea809ef20 2021-02-10T11:40:24.373 INFO:teuthology.orchestra.run.gibba002.stderr:new fs with metadata pool 42 and data pool 43 2021-02-10T11:40:24.375 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:24.372 7f0eacef9700 1 -- 172.21.2.102:0/1897174574 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0e8c0186c0 msgr2=0x7f0e8c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:24.375 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:24.372 7f0eacef9700 1 --2- 172.21.2.102:0/1897174574 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0e8c0186c0 0x7f0e8c01abc0 crc :-1 s=READY pgs=1468 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:24.375 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:24.372 7f0eacef9700 1 -- 172.21.2.102:0/1897174574 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0ea809ef20 msgr2=0x7f0ea8187e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:24.375 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:24.372 7f0eacef9700 1 --2- 172.21.2.102:0/1897174574 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0ea809ef20 0x7f0ea8187e80 secure :-1 s=READY pgs=3366 cs=0 l=1 rev1=1 rx=0x7f0e9000b5c0 tx=0x7f0e9000b960).stop 2021-02-10T11:40:24.375 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:24.372 7f0eacef9700 1 -- 172.21.2.102:0/1897174574 shutdown_connections 2021-02-10T11:40:24.375 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:24.372 7f0eacef9700 1 --2- 172.21.2.102:0/1897174574 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f0e8c0186c0 0x7f0e8c01abc0 unknown :-1 s=CLOSED pgs=1468 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:24.375 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:24.372 7f0eacef9700 1 --2- 172.21.2.102:0/1897174574 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f0ea8189840 0x7f0ea8189c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:24.375 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:24.372 7f0eacef9700 1 --2- 172.21.2.102:0/1897174574 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f0ea81884a0 0x7f0ea81888a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:24.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:24.372 7f0eacef9700 1 --2- 172.21.2.102:0/1897174574 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f0ea809ef20 0x7f0ea8187e80 unknown :-1 s=CLOSED pgs=3366 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:24.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:24.372 7f0eacef9700 1 -- 172.21.2.102:0/1897174574 >> 172.21.2.102:0/1897174574 conn(0x7f0ea819f970 msgr2=0x7f0ea81a61d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:24.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:24.372 7f0eacef9700 1 -- 172.21.2.102:0/1897174574 shutdown_connections 2021-02-10T11:40:24.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:24.372 7f0eacef9700 1 -- 172.21.2.102:0/1897174574 wait complete. 2021-02-10T11:40:24.383 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:40:24.811 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:24.792 7f2eb820c700 1 Processor -- start 2021-02-10T11:40:24.811 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:24.792 7f2eb820c700 1 -- start start 2021-02-10T11:40:24.811 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:24.793 7f2eb820c700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f2eb01b0470 0x7f2eb01b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:24.811 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:24.793 7f2eb820c700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2eb00ef610 0x7f2eb00efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:24.811 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:24.793 7f2eb820c700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2eb00f0b50 con 0x7f2eb00f0030 2021-02-10T11:40:24.812 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:24.793 7f2eb820c700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2eb00e52e0 con 0x7f2eb01b0470 2021-02-10T11:40:24.812 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:24.793 7f2eb820c700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2eb00e54d0 con 0x7f2eb00ef610 2021-02-10T11:40:24.812 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:24.793 7f2eb6028700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f2eb01b0470 0x7f2eb01b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:24.812 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:24.793 7f2eb6829700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f2eb00f0030 0x7f2eb00f10d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:33440/0 (socket says 172.21.2.102:33440) 2021-02-10T11:40:24.812 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:24.793 7f2eb6829700 1 -- 172.21.2.102:0/1894817757 learned_addr learned my addr 172.21.2.102:0/1894817757 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:40:24.812 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:24.793 7f2eb6028700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f2eb01b0470 0x7f2eb01b0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:41216/0 (socket says 172.21.2.102:41216) 2021-02-10T11:40:24.812 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:24.793 7f2eb5827700 1 --2- 172.21.2.102:0/1894817757 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2eb00ef610 0x7f2eb00efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:24.812 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:24.793 7f2eb6028700 1 -- 172.21.2.102:0/1894817757 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2eb00ef610 msgr2=0x7f2eb00efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:24.812 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:24.793 7f2eb6028700 1 --2- 172.21.2.102:0/1894817757 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2eb00ef610 0x7f2eb00efa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:24.812 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:24.793 7f2eb6028700 1 -- 172.21.2.102:0/1894817757 >> v1:172.21.2.102:6789/0 conn(0x7f2eb00f0030 legacy=0x7f2eb00f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:24.812 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:24.793 7f2eb6028700 1 -- 172.21.2.102:0/1894817757 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2eb00f0430 con 0x7f2eb01b0470 2021-02-10T11:40:24.812 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:24.793 7f2eb5026700 1 -- 172.21.2.102:0/1894817757 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 852101874 0 0) 0x7f2ea8000f90 con 0x7f2eb00f0030 2021-02-10T11:40:24.812 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:24.793 7f2eb6028700 1 --2- 172.21.2.102:0/1894817757 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2eb01b0470 0x7f2eb01b0870 secure :-1 s=READY pgs=3368 cs=0 l=1 rev1=1 rx=0x7f2ea4006650 tx=0x7f2ea4009a80).ready entity=mon.0 client_cookie=1a6a56208ccfe3a4 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:24.813 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:24.793 7f2eb5026700 1 -- 172.21.2.102:0/1894817757 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2ea40009b0 con 0x7f2eb01b0470 2021-02-10T11:40:24.813 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:24.793 7f2eb5026700 1 -- 172.21.2.102:0/1894817757 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ea400c0c0 con 0x7f2eb01b0470 2021-02-10T11:40:24.813 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:24.794 7f2eb5026700 1 -- 172.21.2.102:0/1894817757 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2ea400c510 con 0x7f2eb01b0470 2021-02-10T11:40:24.813 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:24.794 7f2eb820c700 1 -- 172.21.2.102:0/1894817757 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2eb01b0470 msgr2=0x7f2eb01b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:24.813 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:24.794 7f2eb820c700 1 --2- 172.21.2.102:0/1894817757 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2eb01b0470 0x7f2eb01b0870 secure :-1 s=READY pgs=3368 cs=0 l=1 rev1=1 rx=0x7f2ea4006650 tx=0x7f2ea4009a80).stop 2021-02-10T11:40:24.813 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:24.794 7f2eb820c700 1 -- 172.21.2.102:0/1894817757 shutdown_connections 2021-02-10T11:40:24.813 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:24.794 7f2eb820c700 1 --2- 172.21.2.102:0/1894817757 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2eb00ef610 0x7f2eb00efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:24.813 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:24.794 7f2eb820c700 1 --2- 172.21.2.102:0/1894817757 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2eb01b0470 0x7f2eb01b0870 unknown :-1 s=CLOSED pgs=3368 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:24.813 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:24.794 7f2eb820c700 1 -- 172.21.2.102:0/1894817757 >> 172.21.2.102:0/1894817757 conn(0x7f2eb00ce3e0 msgr2=0x7f2eb00e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:24.813 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:24.794 7f2eb820c700 1 -- 172.21.2.102:0/1894817757 shutdown_connections 2021-02-10T11:40:24.813 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:24.794 7f2eb820c700 1 -- 172.21.2.102:0/1894817757 wait complete. 2021-02-10T11:40:24.813 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:24.809 7f2eb820c700 1 Processor -- start 2021-02-10T11:40:24.813 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:24.809 7f2eb820c700 1 -- start start 2021-02-10T11:40:24.815 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:24.809 7f2eb820c700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2eb01a48e0 0x7f2eb00b65c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:24.815 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:24.809 7f2eb820c700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2eb0090bc0 0x7f2eb0090fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:24.815 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:24.809 7f2eb820c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2eb00915e0 0x7f2eb00b73e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:24.815 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:24.809 7f2eb820c700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2eb009d7b0 con 0x7f2eb00915e0 2021-02-10T11:40:24.815 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:24.809 7f2eb820c700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2eb009d2f0 con 0x7f2eb0090bc0 2021-02-10T11:40:24.815 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:24.809 7f2eb820c700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2eb00b7a00 con 0x7f2eb01a48e0 2021-02-10T11:40:24.815 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:24.809 7f2eb6028700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2eb01a48e0 0x7f2eb00b65c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:24.815 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:24.809 7f2eb6028700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2eb01a48e0 0x7f2eb00b65c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.113:3301/0 says I am v2:172.21.2.102:46756/0 (socket says 172.21.2.102:46756) 2021-02-10T11:40:24.815 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:24.809 7f2eb6829700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2eb00915e0 0x7f2eb00b73e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:24.816 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:24.809 7f2eb6028700 1 -- 172.21.2.102:0/1003297137 learned_addr learned my addr 172.21.2.102:0/1003297137 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:40:24.816 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:24.810 7f2eb5827700 1 --2- 172.21.2.102:0/1003297137 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2eb0090bc0 0x7f2eb0090fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:24.816 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:24.810 7f2eb6829700 1 -- 172.21.2.102:0/1003297137 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2eb01a48e0 msgr2=0x7f2eb00b65c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:24.816 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:24.810 7f2eb6829700 1 --2- 172.21.2.102:0/1003297137 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2eb01a48e0 0x7f2eb00b65c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:24.816 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:24.810 7f2eb6829700 1 -- 172.21.2.102:0/1003297137 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2eb0090bc0 msgr2=0x7f2eb0090fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:24.816 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:24.810 7f2eb6829700 1 --2- 172.21.2.102:0/1003297137 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2eb0090bc0 0x7f2eb0090fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:24.816 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:24.810 7f2eb6829700 1 -- 172.21.2.102:0/1003297137 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2eb00b7c20 con 0x7f2eb00915e0 2021-02-10T11:40:24.816 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:24.810 7f2eb6829700 1 --2- 172.21.2.102:0/1003297137 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2eb00915e0 0x7f2eb00b73e0 secure :-1 s=READY pgs=3369 cs=0 l=1 rev1=1 rx=0x7f2ea8007fa0 tx=0x7f2ea800aba0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:24.816 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:24.810 7f2ea2ffd700 1 -- 172.21.2.102:0/1003297137 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2ea80070d0 con 0x7f2eb00915e0 2021-02-10T11:40:24.816 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:24.810 7f2ea2ffd700 1 -- 172.21.2.102:0/1003297137 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ea8007310 con 0x7f2eb00915e0 2021-02-10T11:40:24.816 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:24.810 7f2ea2ffd700 1 -- 172.21.2.102:0/1003297137 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2ea8004930 con 0x7f2eb00915e0 2021-02-10T11:40:24.816 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:24.810 7f2eb820c700 1 -- 172.21.2.102:0/1003297137 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2eb00b7c20 con 0x7f2eb00915e0 2021-02-10T11:40:24.816 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:24.810 7f2eb820c700 1 -- 172.21.2.102:0/1003297137 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2eb00b81b0 con 0x7f2eb00915e0 2021-02-10T11:40:24.817 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:24.811 7f2ea2ffd700 1 -- 172.21.2.102:0/1003297137 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f2ea8000df0 con 0x7f2eb00915e0 2021-02-10T11:40:24.817 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:24.811 7f2ea2ffd700 1 --2- 172.21.2.102:0/1003297137 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2e980186c0 0x7f2e9801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:24.817 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:24.811 7f2ea2ffd700 1 -- 172.21.2.102:0/1003297137 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(196..196 src has 1..196) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f2ea8032820 con 0x7f2eb00915e0 2021-02-10T11:40:24.817 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:24.811 7f2eb6028700 1 --2- 172.21.2.102:0/1003297137 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2e980186c0 0x7f2e9801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:24.817 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:24.811 7f2eb6028700 1 --2- 172.21.2.102:0/1003297137 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2e980186c0 0x7f2e9801abc0 crc :-1 s=READY pgs=1469 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:24.845 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:24.843 7f2eb820c700 1 -- 172.21.2.102:0/1003297137 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2eb0105a20 con 0x7f2eb00915e0 2021-02-10T11:40:24.848 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:24.845 7f2ea2ffd700 1 -- 172.21.2.102:0/1003297137 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f2ea8004bf0 con 0x7f2eb00915e0 2021-02-10T11:40:24.974 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:24.972 7f2eb820c700 1 -- 172.21.2.102:0/1003297137 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2eb00f0b50 con 0x7f2eb00915e0 2021-02-10T11:40:24.975 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:24.973 7f2ea2ffd700 1 -- 172.21.2.102:0/1003297137 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v196) v1 ==== 74+0+14020 (secure 0 0 0) 0x7f2ea8000df0 con 0x7f2eb00915e0 2021-02-10T11:40:24.975 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:40:24.976 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":196,"fsid":"68db30ea-dbce-4aef-a6df-f01c6fea0513","created":"2021-02-10 11:03:06.331557","modified":"2021-02-10 11:40:24.364935","last_up_change":"2021-02-10 11:03:32.690562","last_in_change":"2021-02-10 11:03:32.690562","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":43,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:03:37.322519","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":42,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:40:22.330567","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"196","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":43,"pool_name":"cephfs_data","create_time":"2021-02-10 11:40:23.357509","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"196","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"be691b4d-a039-408a-af80-9d53f02b6f1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":185,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6818","nonce":36088},{"type":"v1","addr":"172.21.2.102:6820","nonce":36088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6822","nonce":36088},{"type":"v1","addr":"172.21.2.102:6824","nonce":36088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6830","nonce":36088},{"type":"v1","addr":"172.21.2.102:6832","nonce":36088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6826","nonce":36088},{"type":"v1","addr":"172.21.2.102:6828","nonce":36088}]},"public_addr":"172.21.2.102:6820/36088","cluster_addr":"172.21.2.102:6824/36088","heartbeat_back_addr":"172.21.2.102:6832/36088","heartbeat_front_addr":"172.21.2.102:6828/36088","state":["exists","up"]},{"osd":1,"uuid":"64e3ccc8-9c69-4d7c-865e-af8e83822635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6819","nonce":36087},{"type":"v1","addr":"172.21.2.102:6821","nonce":36087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6823","nonce":36087},{"type":"v1","addr":"172.21.2.102:6825","nonce":36087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6831","nonce":36087},{"type":"v1","addr":"172.21.2.102:6833","nonce":36087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6827","nonce":36087},{"type":"v1","addr":"172.21.2.102:6829","nonce":36087}]},"public_addr":"172.21.2.102:6821/36087","cluster_addr":"172.21.2.102:6825/36087","heartbeat_back_addr":"172.21.2.102:6833/36087","heartbeat_front_addr":"172.21.2.102:6829/36087","state":["exists","up"]},{"osd":2,"uuid":"153700af-747f-481f-bdf4-d6072d3a842f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6810","nonce":36089},{"type":"v1","addr":"172.21.2.102:6811","nonce":36089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6812","nonce":36089},{"type":"v1","addr":"172.21.2.102:6813","nonce":36089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6816","nonce":36089},{"type":"v1","addr":"172.21.2.102:6817","nonce":36089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6814","nonce":36089},{"type":"v1","addr":"172.21.2.102:6815","nonce":36089}]},"public_addr":"172.21.2.102:6811/36089","cluster_addr":"172.21.2.102:6813/36089","heartbeat_back_addr":"172.21.2.102:6817/36089","heartbeat_front_addr":"172.21.2.102:6815/36089","state":["exists","up"]},{"osd":3,"uuid":"cbc678f4-83a9-45e6-8830-2f7c232b1ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6802","nonce":36090},{"type":"v1","addr":"172.21.2.102:6803","nonce":36090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6804","nonce":36090},{"type":"v1","addr":"172.21.2.102:6805","nonce":36090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6808","nonce":36090},{"type":"v1","addr":"172.21.2.102:6809","nonce":36090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6806","nonce":36090},{"type":"v1","addr":"172.21.2.102:6807","nonce":36090}]},"public_addr":"172.21.2.102:6803/36090","cluster_addr":"172.21.2.102:6805/36090","heartbeat_back_addr":"172.21.2.102:6809/36090","heartbeat_front_addr":"172.21.2.102:6807/36090","state":["exists","up"]},{"osd":4,"uuid":"c01e4008-b258-4f10-afd2-9e7f37214351","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6808","nonce":36368},{"type":"v1","addr":"172.21.2.113:6811","nonce":36368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6814","nonce":36368},{"type":"v1","addr":"172.21.2.113:6818","nonce":36368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6826","nonce":36368},{"type":"v1","addr":"172.21.2.113:6828","nonce":36368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6820","nonce":36368},{"type":"v1","addr":"172.21.2.113:6824","nonce":36368}]},"public_addr":"172.21.2.113:6811/36368","cluster_addr":"172.21.2.113:6818/36368","heartbeat_back_addr":"172.21.2.113:6828/36368","heartbeat_front_addr":"172.21.2.113:6824/36368","state":["exists","up"]},{"osd":5,"uuid":"551f6007-daf8-4a98-8ee0-c2b16bcccb5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6802","nonce":36366},{"type":"v1","addr":"172.21.2.113:6805","nonce":36366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6807","nonce":36366},{"type":"v1","addr":"172.21.2.113:6810","nonce":36366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6819","nonce":36366},{"type":"v1","addr":"172.21.2.113:6822","nonce":36366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6812","nonce":36366},{"type":"v1","addr":"172.21.2.113:6815","nonce":36366}]},"public_addr":"172.21.2.113:6805/36366","cluster_addr":"172.21.2.113:6810/36366","heartbeat_back_addr":"172.21.2.113:6822/36366","heartbeat_front_addr":"172.21.2.113:6815/36366","state":["exists","up"]},{"osd":6,"uuid":"bf67c040-0e50-48be-b102-c3943bc9ebef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6800","nonce":36367},{"type":"v1","addr":"172.21.2.113:6801","nonce":36367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6803","nonce":36367},{"type":"v1","addr":"172.21.2.113:6804","nonce":36367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6813","nonce":36367},{"type":"v1","addr":"172.21.2.113:6816","nonce":36367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6806","nonce":36367},{"type":"v1","addr":"172.21.2.113:6809","nonce":36367}]},"public_addr":"172.21.2.113:6801/36367","cluster_addr":"172.21.2.113:6804/36367","heartbeat_back_addr":"172.21.2.113:6816/36367","heartbeat_front_addr":"172.21.2.113:6809/36367","state":["exists","up"]},{"osd":7,"uuid":"f5461f80-018d-428f-be89-2eb2bcbe7bb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6817","nonce":36365},{"type":"v1","addr":"172.21.2.113:6821","nonce":36365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6823","nonce":36365},{"type":"v1","addr":"172.21.2.113:6825","nonce":36365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6830","nonce":36365},{"type":"v1","addr":"172.21.2.113:6831","nonce":36365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6827","nonce":36365},{"type":"v1","addr":"172.21.2.113:6829","nonce":36365}]},"public_addr":"172.21.2.113:6821/36365","cluster_addr":"172.21.2.113:6825/36365","heartbeat_back_addr":"172.21.2.113:6831/36365","heartbeat_front_addr":"172.21.2.113:6829/36365","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:40:24.977 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:24.974 7f2eb820c700 1 -- 172.21.2.102:0/1003297137 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2e980186c0 msgr2=0x7f2e9801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:24.977 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:24.974 7f2eb820c700 1 --2- 172.21.2.102:0/1003297137 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2e980186c0 0x7f2e9801abc0 crc :-1 s=READY pgs=1469 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:24.977 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:24.974 7f2eb820c700 1 -- 172.21.2.102:0/1003297137 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2eb00915e0 msgr2=0x7f2eb00b73e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:24.977 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:24.974 7f2eb820c700 1 --2- 172.21.2.102:0/1003297137 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2eb00915e0 0x7f2eb00b73e0 secure :-1 s=READY pgs=3369 cs=0 l=1 rev1=1 rx=0x7f2ea8007fa0 tx=0x7f2ea800aba0).stop 2021-02-10T11:40:24.977 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:24.974 7f2eb820c700 1 -- 172.21.2.102:0/1003297137 shutdown_connections 2021-02-10T11:40:24.978 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:24.974 7f2eb820c700 1 --2- 172.21.2.102:0/1003297137 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2eb00915e0 0x7f2eb00b73e0 unknown :-1 s=CLOSED pgs=3369 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:24.978 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:24.974 7f2eb820c700 1 --2- 172.21.2.102:0/1003297137 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2e980186c0 0x7f2e9801abc0 unknown :-1 s=CLOSED pgs=1469 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:24.978 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:24.974 7f2eb820c700 1 --2- 172.21.2.102:0/1003297137 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2eb0090bc0 0x7f2eb0090fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:24.978 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:24.974 7f2eb820c700 1 --2- 172.21.2.102:0/1003297137 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2eb01a48e0 0x7f2eb00b65c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:24.978 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:24.974 7f2eb820c700 1 -- 172.21.2.102:0/1003297137 >> 172.21.2.102:0/1003297137 conn(0x7f2eb009cb20 msgr2=0x7f2eb00c3c80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:24.978 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:24.974 7f2eb820c700 1 -- 172.21.2.102:0/1003297137 shutdown_connections 2021-02-10T11:40:24.979 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:24.975 7f2eb820c700 1 -- 172.21.2.102:0/1003297137 wait complete. 2021-02-10T11:40:24.987 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:40:25.417 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:25.399 7f4073778700 1 Processor -- start 2021-02-10T11:40:25.418 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:25.399 7f4073778700 1 -- start start 2021-02-10T11:40:25.418 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:25.399 7f4073778700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f406c0d0340 0x7f406c0ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:25.418 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:25.399 7f4073778700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f406c0f0b10 con 0x7f406c0d1c70 2021-02-10T11:40:25.418 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:25.400 7f4073778700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f406c1b0be0 con 0x7f406c0d1870 2021-02-10T11:40:25.418 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:25.400 7f4073778700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f406c0e5ed0 con 0x7f406c0d0340 2021-02-10T11:40:25.418 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:25.400 7f4071d95700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f406c0d1c70 0x7f406c0efc60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:33456/0 (socket says 172.21.2.102:33456) 2021-02-10T11:40:25.418 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:25.400 7f4071d95700 1 -- 172.21.2.102:0/173328037 learned_addr learned my addr 172.21.2.102:0/173328037 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:40:25.418 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:25.400 7f4070d93700 1 --2- 172.21.2.102:0/173328037 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f406c0d0340 0x7f406c0ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:25.418 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:25.400 7f4070d93700 1 -- 172.21.2.102:0/173328037 >> v1:172.21.2.113:6789/0 conn(0x7f406c0d1870 legacy=0x7f406c1b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:25.418 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:25.400 7f4070d93700 1 -- 172.21.2.102:0/173328037 >> v1:172.21.2.102:6789/0 conn(0x7f406c0d1c70 legacy=0x7f406c0efc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:25.419 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:25.400 7f4070d93700 1 -- 172.21.2.102:0/173328037 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f406c0f0450 con 0x7f406c0d0340 2021-02-10T11:40:25.419 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:25.400 7f4070d93700 1 --2- 172.21.2.102:0/173328037 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f406c0d0340 0x7f406c0ef640 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 rx=0x7f4068006650 tx=0x7f4068009a00).ready entity=mon.2 client_cookie=5db168b6dddf509a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:25.419 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:25.401 7f405ffff700 1 -- 172.21.2.102:0/173328037 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f40680009b0 con 0x7f406c0d0340 2021-02-10T11:40:25.419 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:25.401 7f405ffff700 1 -- 172.21.2.102:0/173328037 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f406800c0c0 con 0x7f406c0d0340 2021-02-10T11:40:25.419 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:25.401 7f405ffff700 1 -- 172.21.2.102:0/173328037 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f406800c5a0 con 0x7f406c0d0340 2021-02-10T11:40:25.419 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:25.401 7f4073778700 1 -- 172.21.2.102:0/173328037 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f406c0d0340 msgr2=0x7f406c0ef640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:25.419 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:25.401 7f4073778700 1 --2- 172.21.2.102:0/173328037 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f406c0d0340 0x7f406c0ef640 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 rx=0x7f4068006650 tx=0x7f4068009a00).stop 2021-02-10T11:40:25.419 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:25.401 7f4073778700 1 -- 172.21.2.102:0/173328037 shutdown_connections 2021-02-10T11:40:25.419 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:25.401 7f4073778700 1 --2- 172.21.2.102:0/173328037 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f406c0d0340 0x7f406c0ef640 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:25.419 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:25.401 7f4073778700 1 -- 172.21.2.102:0/173328037 >> 172.21.2.102:0/173328037 conn(0x7f406c0ce450 msgr2=0x7f406c0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:25.419 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:25.401 7f4073778700 1 -- 172.21.2.102:0/173328037 shutdown_connections 2021-02-10T11:40:25.419 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:25.401 7f4073778700 1 -- 172.21.2.102:0/173328037 wait complete. 2021-02-10T11:40:25.421 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:25.416 7f4073778700 1 Processor -- start 2021-02-10T11:40:25.421 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:25.416 7f4073778700 1 -- start start 2021-02-10T11:40:25.421 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:25.416 7f4073778700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f406c0af040 0x7f406c0af440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:25.421 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:25.416 7f4073778700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f406c0afa60 0x7f406c0ada20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:25.421 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:25.416 7f4073778700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f406c0afe60 0x7f406c0ae040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:25.421 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:25.416 7f4073778700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f406c0923c0 con 0x7f406c0afa60 2021-02-10T11:40:25.421 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:25.416 7f4073778700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f406c0b0410 con 0x7f406c0af040 2021-02-10T11:40:25.421 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:25.416 7f4073778700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f406c091e40 con 0x7f406c0afe60 2021-02-10T11:40:25.422 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:25.416 7f4070d93700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f406c0afa60 0x7f406c0ada20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:25.422 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:25.416 7f4070d93700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f406c0afa60 0x7f406c0ada20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:41238/0 (socket says 172.21.2.102:41238) 2021-02-10T11:40:25.422 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:25.416 7f4070d93700 1 -- 172.21.2.102:0/1426666013 learned_addr learned my addr 172.21.2.102:0/1426666013 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:40:25.422 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:25.417 7f4071594700 1 --2- 172.21.2.102:0/1426666013 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f406c0af040 0x7f406c0af440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:25.422 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:25.417 7f4071d95700 1 --2- 172.21.2.102:0/1426666013 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f406c0afe60 0x7f406c0ae040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:25.422 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:25.417 7f4070d93700 1 -- 172.21.2.102:0/1426666013 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f406c0afe60 msgr2=0x7f406c0ae040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:25.422 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:25.417 7f4070d93700 1 --2- 172.21.2.102:0/1426666013 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f406c0afe60 0x7f406c0ae040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:25.422 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:25.417 7f4070d93700 1 -- 172.21.2.102:0/1426666013 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f406c0af040 msgr2=0x7f406c0af440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:25.422 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:25.417 7f4070d93700 1 --2- 172.21.2.102:0/1426666013 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f406c0af040 0x7f406c0af440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:25.422 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:25.417 7f4070d93700 1 -- 172.21.2.102:0/1426666013 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f406c08f6d0 con 0x7f406c0afa60 2021-02-10T11:40:25.422 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:25.417 7f4071d95700 1 --2- 172.21.2.102:0/1426666013 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f406c0afe60 0x7f406c0ae040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:40:25.422 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:25.417 7f4070d93700 1 --2- 172.21.2.102:0/1426666013 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f406c0afa60 0x7f406c0ada20 secure :-1 s=READY pgs=3371 cs=0 l=1 rev1=1 rx=0x7f406800b5e0 tx=0x7f406800b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:25.423 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:25.417 7f405dffb700 1 -- 172.21.2.102:0/1426666013 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4068006390 con 0x7f406c0afa60 2021-02-10T11:40:25.423 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:25.417 7f405dffb700 1 -- 172.21.2.102:0/1426666013 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40680009b0 con 0x7f406c0afa60 2021-02-10T11:40:25.423 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:25.417 7f405dffb700 1 -- 172.21.2.102:0/1426666013 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4068002220 con 0x7f406c0afa60 2021-02-10T11:40:25.423 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:25.417 7f4073778700 1 -- 172.21.2.102:0/1426666013 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f406c08f6d0 con 0x7f406c0afa60 2021-02-10T11:40:25.423 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:25.417 7f4073778700 1 -- 172.21.2.102:0/1426666013 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f406c08f6d0 con 0x7f406c0afa60 2021-02-10T11:40:25.423 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:25.418 7f405dffb700 1 -- 172.21.2.102:0/1426666013 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f406800a000 con 0x7f406c0afa60 2021-02-10T11:40:25.423 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:25.418 7f405dffb700 1 --2- 172.21.2.102:0/1426666013 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f404c0186c0 0x7f404c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:25.423 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:25.418 7f405dffb700 1 -- 172.21.2.102:0/1426666013 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f4068032750 con 0x7f406c0afa60 2021-02-10T11:40:25.423 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:25.418 7f4071594700 1 --2- 172.21.2.102:0/1426666013 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f404c0186c0 0x7f404c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:25.423 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:25.418 7f4071594700 1 --2- 172.21.2.102:0/1426666013 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f404c0186c0 0x7f404c01abc0 crc :-1 s=READY pgs=1470 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:25.452 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:25.450 7f4073778700 1 -- 172.21.2.102:0/1426666013 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f406c0e7df0 con 0x7f406c0afa60 2021-02-10T11:40:25.454 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:25.452 7f405dffb700 1 -- 172.21.2.102:0/1426666013 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f4068000c90 con 0x7f406c0afa60 2021-02-10T11:40:25.583 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:25.581 7f4073778700 1 -- 172.21.2.102:0/1426666013 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f406c0ce450 con 0x7f406c0afa60 2021-02-10T11:40:25.584 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:25.581 7f405dffb700 1 -- 172.21.2.102:0/1426666013 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v197) v1 ==== 74+0+14007 (secure 0 0 0) 0x7f40680009b0 con 0x7f406c0afa60 2021-02-10T11:40:25.584 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:40:25.585 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":197,"fsid":"68db30ea-dbce-4aef-a6df-f01c6fea0513","created":"2021-02-10 11:03:06.331557","modified":"2021-02-10 11:40:25.367346","last_up_change":"2021-02-10 11:03:32.690562","last_in_change":"2021-02-10 11:03:32.690562","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":43,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:03:37.322519","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":42,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:40:22.330567","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"196","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":43,"pool_name":"cephfs_data","create_time":"2021-02-10 11:40:23.357509","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"197","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"be691b4d-a039-408a-af80-9d53f02b6f1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":185,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6818","nonce":36088},{"type":"v1","addr":"172.21.2.102:6820","nonce":36088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6822","nonce":36088},{"type":"v1","addr":"172.21.2.102:6824","nonce":36088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6830","nonce":36088},{"type":"v1","addr":"172.21.2.102:6832","nonce":36088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6826","nonce":36088},{"type":"v1","addr":"172.21.2.102:6828","nonce":36088}]},"public_addr":"172.21.2.102:6820/36088","cluster_addr":"172.21.2.102:6824/36088","heartbeat_back_addr":"172.21.2.102:6832/36088","heartbeat_front_addr":"172.21.2.102:6828/36088","state":["exists","up"]},{"osd":1,"uuid":"64e3ccc8-9c69-4d7c-865e-af8e83822635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6819","nonce":36087},{"type":"v1","addr":"172.21.2.102:6821","nonce":36087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6823","nonce":36087},{"type":"v1","addr":"172.21.2.102:6825","nonce":36087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6831","nonce":36087},{"type":"v1","addr":"172.21.2.102:6833","nonce":36087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6827","nonce":36087},{"type":"v1","addr":"172.21.2.102:6829","nonce":36087}]},"public_addr":"172.21.2.102:6821/36087","cluster_addr":"172.21.2.102:6825/36087","heartbeat_back_addr":"172.21.2.102:6833/36087","heartbeat_front_addr":"172.21.2.102:6829/36087","state":["exists","up"]},{"osd":2,"uuid":"153700af-747f-481f-bdf4-d6072d3a842f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6810","nonce":36089},{"type":"v1","addr":"172.21.2.102:6811","nonce":36089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6812","nonce":36089},{"type":"v1","addr":"172.21.2.102:6813","nonce":36089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6816","nonce":36089},{"type":"v1","addr":"172.21.2.102:6817","nonce":36089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6814","nonce":36089},{"type":"v1","addr":"172.21.2.102:6815","nonce":36089}]},"public_addr":"172.21.2.102:6811/36089","cluster_addr":"172.21.2.102:6813/36089","heartbeat_back_addr":"172.21.2.102:6817/36089","heartbeat_front_addr":"172.21.2.102:6815/36089","state":["exists","up"]},{"osd":3,"uuid":"cbc678f4-83a9-45e6-8830-2f7c232b1ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6802","nonce":36090},{"type":"v1","addr":"172.21.2.102:6803","nonce":36090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6804","nonce":36090},{"type":"v1","addr":"172.21.2.102:6805","nonce":36090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6808","nonce":36090},{"type":"v1","addr":"172.21.2.102:6809","nonce":36090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6806","nonce":36090},{"type":"v1","addr":"172.21.2.102:6807","nonce":36090}]},"public_addr":"172.21.2.102:6803/36090","cluster_addr":"172.21.2.102:6805/36090","heartbeat_back_addr":"172.21.2.102:6809/36090","heartbeat_front_addr":"172.21.2.102:6807/36090","state":["exists","up"]},{"osd":4,"uuid":"c01e4008-b258-4f10-afd2-9e7f37214351","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6808","nonce":36368},{"type":"v1","addr":"172.21.2.113:6811","nonce":36368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6814","nonce":36368},{"type":"v1","addr":"172.21.2.113:6818","nonce":36368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6826","nonce":36368},{"type":"v1","addr":"172.21.2.113:6828","nonce":36368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6820","nonce":36368},{"type":"v1","addr":"172.21.2.113:6824","nonce":36368}]},"public_addr":"172.21.2.113:6811/36368","cluster_addr":"172.21.2.113:6818/36368","heartbeat_back_addr":"172.21.2.113:6828/36368","heartbeat_front_addr":"172.21.2.113:6824/36368","state":["exists","up"]},{"osd":5,"uuid":"551f6007-daf8-4a98-8ee0-c2b16bcccb5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6802","nonce":36366},{"type":"v1","addr":"172.21.2.113:6805","nonce":36366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6807","nonce":36366},{"type":"v1","addr":"172.21.2.113:6810","nonce":36366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6819","nonce":36366},{"type":"v1","addr":"172.21.2.113:6822","nonce":36366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6812","nonce":36366},{"type":"v1","addr":"172.21.2.113:6815","nonce":36366}]},"public_addr":"172.21.2.113:6805/36366","cluster_addr":"172.21.2.113:6810/36366","heartbeat_back_addr":"172.21.2.113:6822/36366","heartbeat_front_addr":"172.21.2.113:6815/36366","state":["exists","up"]},{"osd":6,"uuid":"bf67c040-0e50-48be-b102-c3943bc9ebef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6800","nonce":36367},{"type":"v1","addr":"172.21.2.113:6801","nonce":36367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6803","nonce":36367},{"type":"v1","addr":"172.21.2.113:6804","nonce":36367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6813","nonce":36367},{"type":"v1","addr":"172.21.2.113:6816","nonce":36367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6806","nonce":36367},{"type":"v1","addr":"172.21.2.113:6809","nonce":36367}]},"public_addr":"172.21.2.113:6801/36367","cluster_addr":"172.21.2.113:6804/36367","heartbeat_back_addr":"172.21.2.113:6816/36367","heartbeat_front_addr":"172.21.2.113:6809/36367","state":["exists","up"]},{"osd":7,"uuid":"f5461f80-018d-428f-be89-2eb2bcbe7bb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6817","nonce":36365},{"type":"v1","addr":"172.21.2.113:6821","nonce":36365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6823","nonce":36365},{"type":"v1","addr":"172.21.2.113:6825","nonce":36365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6830","nonce":36365},{"type":"v1","addr":"172.21.2.113:6831","nonce":36365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6827","nonce":36365},{"type":"v1","addr":"172.21.2.113:6829","nonce":36365}]},"public_addr":"172.21.2.113:6821/36365","cluster_addr":"172.21.2.113:6825/36365","heartbeat_back_addr":"172.21.2.113:6831/36365","heartbeat_front_addr":"172.21.2.113:6829/36365","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:40:25.586 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:25.583 7f4073778700 1 -- 172.21.2.102:0/1426666013 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f404c0186c0 msgr2=0x7f404c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:25.586 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:25.583 7f4073778700 1 --2- 172.21.2.102:0/1426666013 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f404c0186c0 0x7f404c01abc0 crc :-1 s=READY pgs=1470 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:25.586 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:25.583 7f4073778700 1 -- 172.21.2.102:0/1426666013 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f406c0afa60 msgr2=0x7f406c0ada20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:25.587 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:25.583 7f4073778700 1 --2- 172.21.2.102:0/1426666013 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f406c0afa60 0x7f406c0ada20 secure :-1 s=READY pgs=3371 cs=0 l=1 rev1=1 rx=0x7f406800b5e0 tx=0x7f406800b980).stop 2021-02-10T11:40:25.587 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:25.583 7f4073778700 1 -- 172.21.2.102:0/1426666013 shutdown_connections 2021-02-10T11:40:25.587 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:25.583 7f4073778700 1 --2- 172.21.2.102:0/1426666013 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f406c0afe60 0x7f406c0ae040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:25.587 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:25.583 7f4073778700 1 --2- 172.21.2.102:0/1426666013 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f406c0afa60 0x7f406c0ada20 unknown :-1 s=CLOSED pgs=3371 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:25.587 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:25.583 7f4073778700 1 --2- 172.21.2.102:0/1426666013 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f404c0186c0 0x7f404c01abc0 unknown :-1 s=CLOSED pgs=1470 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:25.587 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:25.583 7f4073778700 1 --2- 172.21.2.102:0/1426666013 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f406c0af040 0x7f406c0af440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:25.588 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:25.583 7f4073778700 1 -- 172.21.2.102:0/1426666013 >> 172.21.2.102:0/1426666013 conn(0x7f406c0a0030 msgr2=0x7f406c0f22b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:25.588 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:25.584 7f4073778700 1 -- 172.21.2.102:0/1426666013 shutdown_connections 2021-02-10T11:40:25.588 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:25.584 7f4073778700 1 -- 172.21.2.102:0/1426666013 wait complete. 2021-02-10T11:40:25.595 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2021-02-10T11:40:26.028 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:26.009 7f2fa8521700 1 Processor -- start 2021-02-10T11:40:26.028 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:26.009 7f2fa8521700 1 -- start start 2021-02-10T11:40:26.028 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:26.010 7f2fa8521700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f2fa00d1870 0x7f2fa01b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:26.029 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:26.010 7f2fa8521700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f2fa01b0ae0 0x7f2fa0108940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:26.029 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:26.010 7f2fa8521700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fa00fe1c0 con 0x7f2fa00d1c70 2021-02-10T11:40:26.029 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:26.010 7f2fa8521700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fa0109a60 con 0x7f2fa00d1870 2021-02-10T11:40:26.029 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:26.010 7f2fa8521700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fa0109750 con 0x7f2fa01b0ae0 2021-02-10T11:40:26.029 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:26.010 7f2fa633d700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f2fa00d1870 0x7f2fa01b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:26.029 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:26.010 7f2fa633d700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f2fa00d1870 0x7f2fa01b04c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:41242/0 (socket says 172.21.2.102:41242) 2021-02-10T11:40:26.029 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:26.010 7f2fa633d700 1 -- 172.21.2.102:0/1115215001 learned_addr learned my addr 172.21.2.102:0/1115215001 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:40:26.029 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:26.010 7f2fa633d700 1 -- 172.21.2.102:0/1115215001 >> v2:172.21.2.113:3300/0 conn(0x7f2fa01b0ae0 msgr2=0x7f2fa0108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:26.029 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:26.010 7f2fa633d700 1 --2- 172.21.2.102:0/1115215001 >> v2:172.21.2.113:3300/0 conn(0x7f2fa01b0ae0 0x7f2fa0108940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:26.029 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:26.010 7f2fa633d700 1 -- 172.21.2.102:0/1115215001 >> v1:172.21.2.113:6789/0 conn(0x7f2fa00d1c70 legacy=0x7f2fa0108f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:26.029 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:26.010 7f2fa633d700 1 -- 172.21.2.102:0/1115215001 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fa0088070 con 0x7f2fa00d1870 2021-02-10T11:40:26.029 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:26.011 7f2fa633d700 1 --2- 172.21.2.102:0/1115215001 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2fa00d1870 0x7f2fa01b04c0 secure :-1 s=READY pgs=3372 cs=0 l=1 rev1=1 rx=0x7f2f94006700 tx=0x7f2f94009b80).ready entity=mon.0 client_cookie=4d8a557533b8e8f4 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:26.029 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:26.011 7f2fa533b700 1 -- 172.21.2.102:0/1115215001 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2f9400c0c0 con 0x7f2fa00d1870 2021-02-10T11:40:26.030 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:26.011 7f2fa533b700 1 -- 172.21.2.102:0/1115215001 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f9400c340 con 0x7f2fa00d1870 2021-02-10T11:40:26.030 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:26.011 7f2fa533b700 1 -- 172.21.2.102:0/1115215001 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2f9400c860 con 0x7f2fa00d1870 2021-02-10T11:40:26.030 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:26.011 7f2fa8521700 1 -- 172.21.2.102:0/1115215001 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2fa00d1870 msgr2=0x7f2fa01b04c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:26.030 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:26.011 7f2fa8521700 1 --2- 172.21.2.102:0/1115215001 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2fa00d1870 0x7f2fa01b04c0 secure :-1 s=READY pgs=3372 cs=0 l=1 rev1=1 rx=0x7f2f94006700 tx=0x7f2f94009b80).stop 2021-02-10T11:40:26.030 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:26.012 7f2fa8521700 1 -- 172.21.2.102:0/1115215001 shutdown_connections 2021-02-10T11:40:26.030 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:26.012 7f2fa8521700 1 --2- 172.21.2.102:0/1115215001 >> v2:172.21.2.113:3300/0 conn(0x7f2fa01b0ae0 0x7f2fa0108940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:26.030 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:26.012 7f2fa8521700 1 --2- 172.21.2.102:0/1115215001 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2fa00d1870 0x7f2fa01b04c0 unknown :-1 s=CLOSED pgs=3372 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:26.030 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:26.012 7f2fa8521700 1 -- 172.21.2.102:0/1115215001 >> 172.21.2.102:0/1115215001 conn(0x7f2fa00ce420 msgr2=0x7f2fa00d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:26.030 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:26.012 7f2fa8521700 1 -- 172.21.2.102:0/1115215001 shutdown_connections 2021-02-10T11:40:26.030 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:26.012 7f2fa8521700 1 -- 172.21.2.102:0/1115215001 wait complete. 2021-02-10T11:40:26.032 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:26.026 7f2fa8521700 1 Processor -- start 2021-02-10T11:40:26.032 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:26.026 7f2fa8521700 1 -- start start 2021-02-10T11:40:26.032 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:26.027 7f2fa8521700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2fa00c3820 0x7f2fa0084cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:26.032 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:26.027 7f2fa8521700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2fa00a0290 0x7f2fa00852d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:26.032 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:26.027 7f2fa8521700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2fa0086270 0x7f2fa0086670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:26.032 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:26.027 7f2fa8521700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fa00c0700 con 0x7f2fa00a0290 2021-02-10T11:40:26.032 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:26.027 7f2fa8521700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fa00858f0 con 0x7f2fa0086270 2021-02-10T11:40:26.032 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:26.027 7f2fa8521700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fa0085b10 con 0x7f2fa00c3820 2021-02-10T11:40:26.032 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:26.027 7f2fa5b3c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2fa00a0290 0x7f2fa00852d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:26.032 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:26.027 7f2fa5b3c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2fa00a0290 0x7f2fa00852d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:41250/0 (socket says 172.21.2.102:41250) 2021-02-10T11:40:26.032 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:26.027 7f2fa5b3c700 1 -- 172.21.2.102:0/595723678 learned_addr learned my addr 172.21.2.102:0/595723678 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:40:26.033 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:26.028 7f2fa5b3c700 1 -- 172.21.2.102:0/595723678 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2fa00c3820 msgr2=0x7f2fa0084cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:26.033 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:26.028 7f2fa5b3c700 1 --2- 172.21.2.102:0/595723678 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2fa00c3820 0x7f2fa0084cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:26.033 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:26.028 7f2fa5b3c700 1 -- 172.21.2.102:0/595723678 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2fa0086270 msgr2=0x7f2fa0086670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:26.033 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:26.028 7f2fa5b3c700 1 --2- 172.21.2.102:0/595723678 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2fa0086270 0x7f2fa0086670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:26.033 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:26.028 7f2fa5b3c700 1 -- 172.21.2.102:0/595723678 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fa0086c90 con 0x7f2fa00a0290 2021-02-10T11:40:26.033 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:26.028 7f2fa5b3c700 1 --2- 172.21.2.102:0/595723678 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2fa00a0290 0x7f2fa00852d0 secure :-1 s=READY pgs=3373 cs=0 l=1 rev1=1 rx=0x7f2f9c00b530 tx=0x7f2f9c00b8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:26.033 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:26.028 7f2f92ffd700 1 -- 172.21.2.102:0/595723678 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2f9c006b90 con 0x7f2fa00a0290 2021-02-10T11:40:26.033 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:26.028 7f2f92ffd700 1 -- 172.21.2.102:0/595723678 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f9c00bcc0 con 0x7f2fa00a0290 2021-02-10T11:40:26.033 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:26.028 7f2f92ffd700 1 -- 172.21.2.102:0/595723678 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2f9c0020d0 con 0x7f2fa00a0290 2021-02-10T11:40:26.033 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:26.028 7f2fa8521700 1 -- 172.21.2.102:0/595723678 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fa0086c90 con 0x7f2fa00a0290 2021-02-10T11:40:26.033 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:26.028 7f2fa8521700 1 -- 172.21.2.102:0/595723678 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fa01a2f00 con 0x7f2fa00a0290 2021-02-10T11:40:26.033 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:26.029 7f2f92ffd700 1 -- 172.21.2.102:0/595723678 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f2f9c007000 con 0x7f2fa00a0290 2021-02-10T11:40:26.033 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:26.029 7f2f92ffd700 1 --2- 172.21.2.102:0/595723678 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2f880186c0 0x7f2f8801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:26.034 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:26.029 7f2f92ffd700 1 -- 172.21.2.102:0/595723678 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f2f9c032580 con 0x7f2fa00a0290 2021-02-10T11:40:26.034 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:26.029 7f2fa633d700 1 --2- 172.21.2.102:0/595723678 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2f880186c0 0x7f2f8801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:26.034 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:26.029 7f2fa633d700 1 --2- 172.21.2.102:0/595723678 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2f880186c0 0x7f2f8801abc0 crc :-1 s=READY pgs=1471 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:26.062 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:26.059 7f2fa8521700 1 -- 172.21.2.102:0/595723678 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fa0082a70 con 0x7f2fa00a0290 2021-02-10T11:40:26.064 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:26.062 7f2f92ffd700 1 -- 172.21.2.102:0/595723678 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f2f9c000940 con 0x7f2fa00a0290 2021-02-10T11:40:26.202 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:26.199 7f2fa8521700 1 -- 172.21.2.102:0/595723678 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"} v 0) v1 -- 0x7f2fa0082a70 con 0x7f2fa00a0290 2021-02-10T11:40:26.377 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:26.374 7f2f92ffd700 1 -- 172.21.2.102:0/595723678 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"}]=0 v173) v1 ==== 118+0+0 (secure 0 0 0) 0x7f2f9c007000 con 0x7f2fa00a0290 2021-02-10T11:40:26.419 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:26.416 7f2fa8521700 1 -- 172.21.2.102:0/595723678 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2f880186c0 msgr2=0x7f2f8801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:26.419 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:26.416 7f2fa8521700 1 --2- 172.21.2.102:0/595723678 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2f880186c0 0x7f2f8801abc0 crc :-1 s=READY pgs=1471 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:26.419 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:26.416 7f2fa8521700 1 -- 172.21.2.102:0/595723678 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2fa00a0290 msgr2=0x7f2fa00852d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:26.419 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:26.416 7f2fa8521700 1 --2- 172.21.2.102:0/595723678 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2fa00a0290 0x7f2fa00852d0 secure :-1 s=READY pgs=3373 cs=0 l=1 rev1=1 rx=0x7f2f9c00b530 tx=0x7f2f9c00b8d0).stop 2021-02-10T11:40:26.419 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:26.417 7f2fa8521700 1 -- 172.21.2.102:0/595723678 shutdown_connections 2021-02-10T11:40:26.419 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:26.417 7f2fa8521700 1 --2- 172.21.2.102:0/595723678 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2f880186c0 0x7f2f8801abc0 unknown :-1 s=CLOSED pgs=1471 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:26.420 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:26.417 7f2fa8521700 1 --2- 172.21.2.102:0/595723678 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2fa0086270 0x7f2fa0086670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:26.420 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:26.417 7f2fa8521700 1 --2- 172.21.2.102:0/595723678 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2fa00a0290 0x7f2fa00852d0 unknown :-1 s=CLOSED pgs=3373 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:26.420 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:26.417 7f2fa8521700 1 --2- 172.21.2.102:0/595723678 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2fa00c3820 0x7f2fa0084cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:26.420 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:26.417 7f2fa8521700 1 -- 172.21.2.102:0/595723678 >> 172.21.2.102:0/595723678 conn(0x7f2fa00872c0 msgr2=0x7f2fa0095700 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:26.420 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:26.417 7f2fa8521700 1 -- 172.21.2.102:0/595723678 shutdown_connections 2021-02-10T11:40:26.420 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:26.417 7f2fa8521700 1 -- 172.21.2.102:0/595723678 wait complete. 2021-02-10T11:40:26.427 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:40:26.857 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:26.838 7fcc69358700 1 Processor -- start 2021-02-10T11:40:26.857 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:26.838 7fcc69358700 1 -- start start 2021-02-10T11:40:26.857 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:26.839 7fcc69358700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fcc641b0470 0x7fcc641b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:26.857 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:26.839 7fcc69358700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fcc640ef610 0x7fcc640efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:26.857 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:26.839 7fcc69358700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcc640f0030 0x7fcc640f0430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:26.857 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:26.839 7fcc69358700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc640e5ea0 con 0x7fcc640ef610 2021-02-10T11:40:26.857 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:26.839 7fcc69358700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc640f0b50 con 0x7fcc641b0470 2021-02-10T11:40:26.857 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:26.839 7fcc69358700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc640e52e0 con 0x7fcc640f0030 2021-02-10T11:40:26.857 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:26.839 7fcc6287c700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fcc640ef610 0x7fcc640efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:26.857 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:26.839 7fcc6287c700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fcc640ef610 0x7fcc640efa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:41256/0 (socket says 172.21.2.102:41256) 2021-02-10T11:40:26.858 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:26.839 7fcc6287c700 1 -- 172.21.2.102:0/1614716227 learned_addr learned my addr 172.21.2.102:0/1614716227 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:40:26.858 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:26.839 7fcc6287c700 1 -- 172.21.2.102:0/1614716227 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcc640f0030 msgr2=0x7fcc640f0430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:26.858 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:26.839 7fcc6287c700 1 --2- 172.21.2.102:0/1614716227 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcc640f0030 0x7fcc640f0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:26.858 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:26.839 7fcc6287c700 1 -- 172.21.2.102:0/1614716227 >> v2:172.21.2.113:3300/0 conn(0x7fcc641b0470 msgr2=0x7fcc641b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:26.858 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:26.839 7fcc6287c700 1 --2- 172.21.2.102:0/1614716227 >> v2:172.21.2.113:3300/0 conn(0x7fcc641b0470 0x7fcc641b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:26.858 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:26.839 7fcc6287c700 1 -- 172.21.2.102:0/1614716227 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc640f10d0 con 0x7fcc640ef610 2021-02-10T11:40:26.858 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:26.840 7fcc6287c700 1 --2- 172.21.2.102:0/1614716227 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcc640ef610 0x7fcc640efa10 secure :-1 s=READY pgs=3374 cs=0 l=1 rev1=1 rx=0x7fcc54006700 tx=0x7fcc54009b80).ready entity=mon.0 client_cookie=f8debc206aaad44f server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:26.858 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:26.840 7fcc6207b700 1 -- 172.21.2.102:0/1614716227 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcc5400c0c0 con 0x7fcc640ef610 2021-02-10T11:40:26.858 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:26.840 7fcc6207b700 1 -- 172.21.2.102:0/1614716227 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc5400c340 con 0x7fcc640ef610 2021-02-10T11:40:26.858 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:26.840 7fcc6207b700 1 -- 172.21.2.102:0/1614716227 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcc5400c860 con 0x7fcc640ef610 2021-02-10T11:40:26.858 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:26.840 7fcc69358700 1 -- 172.21.2.102:0/1614716227 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcc640ef610 msgr2=0x7fcc640efa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:26.858 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:26.840 7fcc69358700 1 --2- 172.21.2.102:0/1614716227 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcc640ef610 0x7fcc640efa10 secure :-1 s=READY pgs=3374 cs=0 l=1 rev1=1 rx=0x7fcc54006700 tx=0x7fcc54009b80).stop 2021-02-10T11:40:26.859 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:26.840 7fcc69358700 1 -- 172.21.2.102:0/1614716227 shutdown_connections 2021-02-10T11:40:26.859 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:26.840 7fcc69358700 1 --2- 172.21.2.102:0/1614716227 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcc640f0030 0x7fcc640f0430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:26.859 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:26.840 7fcc69358700 1 --2- 172.21.2.102:0/1614716227 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcc640ef610 0x7fcc640efa10 unknown :-1 s=CLOSED pgs=3374 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:26.859 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:26.840 7fcc69358700 1 --2- 172.21.2.102:0/1614716227 >> v2:172.21.2.113:3300/0 conn(0x7fcc641b0470 0x7fcc641b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:26.859 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:26.840 7fcc69358700 1 -- 172.21.2.102:0/1614716227 >> 172.21.2.102:0/1614716227 conn(0x7fcc640ce3e0 msgr2=0x7fcc640e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:26.859 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:26.840 7fcc69358700 1 -- 172.21.2.102:0/1614716227 shutdown_connections 2021-02-10T11:40:26.859 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:26.841 7fcc69358700 1 -- 172.21.2.102:0/1614716227 wait complete. 2021-02-10T11:40:26.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:26.855 7fcc69358700 1 Processor -- start 2021-02-10T11:40:26.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:26.855 7fcc69358700 1 -- start start 2021-02-10T11:40:26.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:26.855 7fcc69358700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fcc641095b0 0x7fcc6419e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:26.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:26.855 7fcc69358700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcc641099b0 0x7fcc640933c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:26.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:26.855 7fcc69358700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcc640939e0 0x7fcc641ace00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:26.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:26.855 7fcc69358700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc64093de0 con 0x7fcc640939e0 2021-02-10T11:40:26.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:26.855 7fcc69358700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc640aed40 con 0x7fcc641095b0 2021-02-10T11:40:26.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:26.855 7fcc69358700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc641ad420 con 0x7fcc641099b0 2021-02-10T11:40:26.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:26.856 7fcc6387e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcc640939e0 0x7fcc641ace00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:26.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:26.856 7fcc6387e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcc640939e0 0x7fcc641ace00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:41262/0 (socket says 172.21.2.102:41262) 2021-02-10T11:40:26.862 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:26.856 7fcc6387e700 1 -- 172.21.2.102:0/1479761083 learned_addr learned my addr 172.21.2.102:0/1479761083 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:40:26.862 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:26.856 7fcc6387e700 1 -- 172.21.2.102:0/1479761083 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcc641099b0 msgr2=0x7fcc640933c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:26.862 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:26.856 7fcc6387e700 1 --2- 172.21.2.102:0/1479761083 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcc641099b0 0x7fcc640933c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:26.862 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:26.856 7fcc6387e700 1 -- 172.21.2.102:0/1479761083 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fcc641095b0 msgr2=0x7fcc6419e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:26.862 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:26.856 7fcc6387e700 1 --2- 172.21.2.102:0/1479761083 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fcc641095b0 0x7fcc6419e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:26.862 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:26.856 7fcc6387e700 1 -- 172.21.2.102:0/1479761083 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc641ad640 con 0x7fcc640939e0 2021-02-10T11:40:26.862 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:26.856 7fcc6387e700 1 --2- 172.21.2.102:0/1479761083 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcc640939e0 0x7fcc641ace00 secure :-1 s=READY pgs=3375 cs=0 l=1 rev1=1 rx=0x7fcc5c009980 tx=0x7fcc5c00bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:26.862 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:26.857 7fcc5bfff700 1 -- 172.21.2.102:0/1479761083 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcc5c005af0 con 0x7fcc640939e0 2021-02-10T11:40:26.862 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:26.857 7fcc5bfff700 1 -- 172.21.2.102:0/1479761083 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc5c005df0 con 0x7fcc640939e0 2021-02-10T11:40:26.862 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:26.857 7fcc5bfff700 1 -- 172.21.2.102:0/1479761083 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcc5c0062b0 con 0x7fcc640939e0 2021-02-10T11:40:26.862 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:26.857 7fcc69358700 1 -- 172.21.2.102:0/1479761083 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc641ad640 con 0x7fcc640939e0 2021-02-10T11:40:26.862 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:26.857 7fcc69358700 1 -- 172.21.2.102:0/1479761083 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc641adbd0 con 0x7fcc640939e0 2021-02-10T11:40:26.862 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:26.857 7fcc5bfff700 1 -- 172.21.2.102:0/1479761083 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fcc5c005af0 con 0x7fcc640939e0 2021-02-10T11:40:26.863 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:26.858 7fcc5bfff700 1 --2- 172.21.2.102:0/1479761083 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcc480186c0 0x7fcc4801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:26.863 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:26.858 7fcc5bfff700 1 -- 172.21.2.102:0/1479761083 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fcc5c0328f0 con 0x7fcc640939e0 2021-02-10T11:40:26.863 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:26.858 7fcc6307d700 1 --2- 172.21.2.102:0/1479761083 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcc480186c0 0x7fcc4801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:26.863 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:26.858 7fcc6307d700 1 --2- 172.21.2.102:0/1479761083 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcc480186c0 0x7fcc4801abc0 crc :-1 s=READY pgs=1472 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:26.890 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:26.888 7fcc69358700 1 -- 172.21.2.102:0/1479761083 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc640f0b50 con 0x7fcc640939e0 2021-02-10T11:40:26.893 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:26.891 7fcc5bfff700 1 -- 172.21.2.102:0/1479761083 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fcc5c008ef0 con 0x7fcc640939e0 2021-02-10T11:40:27.020 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:27.017 7fcc69358700 1 -- 172.21.2.102:0/1479761083 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcc640f0b50 con 0x7fcc640939e0 2021-02-10T11:40:27.020 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:27.018 7fcc5bfff700 1 -- 172.21.2.102:0/1479761083 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 173 v173) v1 ==== 95+0+2108 (secure 0 0 0) 0x7fcc5c006f20 con 0x7fcc640939e0 2021-02-10T11:40:27.021 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:40:27.021 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":173,"default_fscid":21,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":13872,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.102:6835/1871410080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6834","nonce":1871410080},{"type":"v1","addr":"172.21.2.102:6835","nonce":1871410080}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":169}],"filesystems":[{"mdsmap":{"epoch":173,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:40:24.365024","modified":"2021-02-10 11:40:26.372300","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7598},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7598":{"gid":7598,"name":"b","rank":0,"incarnation":170,"state":"up:active","state_seq":3,"addr":"172.21.2.113:6833/2247818770","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6832","nonce":2247818770},{"type":"v1","addr":"172.21.2.113:6833","nonce":2247818770}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[43],"metadata_pool":42,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":21}]} 2021-02-10T11:40:27.021 INFO:teuthology.orchestra.run.gibba002.stderr:dumped fsmap epoch 173 2021-02-10T11:40:27.022 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:27.020 7fcc69358700 1 -- 172.21.2.102:0/1479761083 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcc480186c0 msgr2=0x7fcc4801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:27.022 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:27.020 7fcc69358700 1 --2- 172.21.2.102:0/1479761083 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcc480186c0 0x7fcc4801abc0 crc :-1 s=READY pgs=1472 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:27.022 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:27.020 7fcc69358700 1 -- 172.21.2.102:0/1479761083 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcc640939e0 msgr2=0x7fcc641ace00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:27.022 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:27.020 7fcc69358700 1 --2- 172.21.2.102:0/1479761083 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcc640939e0 0x7fcc641ace00 secure :-1 s=READY pgs=3375 cs=0 l=1 rev1=1 rx=0x7fcc5c009980 tx=0x7fcc5c00bcd0).stop 2021-02-10T11:40:27.022 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:27.020 7fcc69358700 1 -- 172.21.2.102:0/1479761083 shutdown_connections 2021-02-10T11:40:27.022 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:27.020 7fcc69358700 1 --2- 172.21.2.102:0/1479761083 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcc640939e0 0x7fcc641ace00 unknown :-1 s=CLOSED pgs=3375 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:27.022 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:27.020 7fcc69358700 1 --2- 172.21.2.102:0/1479761083 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcc641099b0 0x7fcc640933c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:27.023 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:27.020 7fcc69358700 1 --2- 172.21.2.102:0/1479761083 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcc480186c0 0x7fcc4801abc0 unknown :-1 s=CLOSED pgs=1472 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:27.023 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:27.020 7fcc69358700 1 --2- 172.21.2.102:0/1479761083 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fcc641095b0 0x7fcc6419e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:27.023 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:27.020 7fcc69358700 1 -- 172.21.2.102:0/1479761083 >> 172.21.2.102:0/1479761083 conn(0x7fcc6419f7b0 msgr2=0x7fcc6418f380 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:27.023 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:27.020 7fcc69358700 1 -- 172.21.2.102:0/1479761083 shutdown_connections 2021-02-10T11:40:27.023 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:27.020 7fcc69358700 1 -- 172.21.2.102:0/1479761083 wait complete. 2021-02-10T11:40:27.031 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:40:27.459 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:27.441 7f208d799700 1 Processor -- start 2021-02-10T11:40:27.460 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:27.441 7f208d799700 1 -- start start 2021-02-10T11:40:27.460 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:27.442 7f208d799700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f20880d1870 0x7f20881b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:27.460 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:27.442 7f208d799700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f20881b0a10 0x7f20880cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:27.460 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:27.442 7f208d799700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20880f0b10 con 0x7f20880d1c70 2021-02-10T11:40:27.460 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:27.442 7f208d799700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20880e5ed0 con 0x7f20881b0a10 2021-02-10T11:40:27.460 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:27.442 7f208d799700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20880f0650 con 0x7f20880d1870 2021-02-10T11:40:27.460 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:27.442 7f2086d73700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f20881b0a10 0x7f20880cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:27.460 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:27.442 7f2086d73700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f20881b0a10 0x7f20880cf330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:41268/0 (socket says 172.21.2.102:41268) 2021-02-10T11:40:27.460 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:27.442 7f2086d73700 1 -- 172.21.2.102:0/1825556282 learned_addr learned my addr 172.21.2.102:0/1825556282 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:40:27.460 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:27.442 7f2087d75700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f20880d1c70 0x7f20880cf950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:33492/0 (socket says 172.21.2.102:33492) 2021-02-10T11:40:27.460 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:27.442 7f2086d73700 1 -- 172.21.2.102:0/1825556282 >> v2:172.21.2.113:3300/0 conn(0x7f20880d1870 msgr2=0x7f20881b03f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:40:27.461 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:27.442 7f2086d73700 1 --2- 172.21.2.102:0/1825556282 >> v2:172.21.2.113:3300/0 conn(0x7f20880d1870 0x7f20881b03f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:27.461 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:27.442 7f2086d73700 1 -- 172.21.2.102:0/1825556282 >> v1:172.21.2.102:6789/0 conn(0x7f20880d1c70 legacy=0x7f20880cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:27.461 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:27.442 7f2086d73700 1 -- 172.21.2.102:0/1825556282 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20880f0840 con 0x7f20881b0a10 2021-02-10T11:40:27.461 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:27.442 7f2086d73700 1 --2- 172.21.2.102:0/1825556282 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f20881b0a10 0x7f20880cf330 secure :-1 s=READY pgs=3377 cs=0 l=1 rev1=1 rx=0x7f2070006700 tx=0x7f2070009b00).ready entity=mon.0 client_cookie=f7c3ad8940615687 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:27.461 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:27.443 7f2086572700 1 -- 172.21.2.102:0/1825556282 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f207000c0c0 con 0x7f20881b0a10 2021-02-10T11:40:27.461 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:27.443 7f2086572700 1 -- 172.21.2.102:0/1825556282 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f207000c0c0 con 0x7f20881b0a10 2021-02-10T11:40:27.461 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:27.443 7f2086572700 1 -- 172.21.2.102:0/1825556282 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f207000c340 con 0x7f20881b0a10 2021-02-10T11:40:27.461 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:27.443 7f208d799700 1 -- 172.21.2.102:0/1825556282 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f20881b0a10 msgr2=0x7f20880cf330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:27.461 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:27.443 7f208d799700 1 --2- 172.21.2.102:0/1825556282 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f20881b0a10 0x7f20880cf330 secure :-1 s=READY pgs=3377 cs=0 l=1 rev1=1 rx=0x7f2070006700 tx=0x7f2070009b00).stop 2021-02-10T11:40:27.461 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:27.443 7f208d799700 1 -- 172.21.2.102:0/1825556282 shutdown_connections 2021-02-10T11:40:27.461 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:27.443 7f208d799700 1 --2- 172.21.2.102:0/1825556282 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f20881b0a10 0x7f20880cf330 unknown :-1 s=CLOSED pgs=3377 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:27.461 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:27.443 7f208d799700 1 --2- 172.21.2.102:0/1825556282 >> v2:172.21.2.113:3300/0 conn(0x7f20880d1870 0x7f20881b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:27.462 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:27.443 7f208d799700 1 -- 172.21.2.102:0/1825556282 >> 172.21.2.102:0/1825556282 conn(0x7f20880ce450 msgr2=0x7f20880d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:27.462 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:27.443 7f208d799700 1 -- 172.21.2.102:0/1825556282 shutdown_connections 2021-02-10T11:40:27.462 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:27.443 7f208d799700 1 -- 172.21.2.102:0/1825556282 wait complete. 2021-02-10T11:40:27.463 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:27.458 7f208d799700 1 Processor -- start 2021-02-10T11:40:27.463 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:27.458 7f208d799700 1 -- start start 2021-02-10T11:40:27.464 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:27.458 7f208d799700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f20881b7cf0 0x7f208819cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:27.464 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:27.458 7f208d799700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f20881b80f0 0x7f20881b84f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:27.464 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:27.458 7f208d799700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f208819da20 0x7f208819de20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:27.464 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:27.458 7f208d799700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20880a77c0 con 0x7f20881b7cf0 2021-02-10T11:40:27.464 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:27.458 7f208d799700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f208819d4a0 con 0x7f208819da20 2021-02-10T11:40:27.464 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:27.458 7f208d799700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20880954f0 con 0x7f20881b80f0 2021-02-10T11:40:27.464 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:27.458 7f2087574700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f20881b7cf0 0x7f208819cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:27.464 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:27.458 7f2086d73700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f20881b80f0 0x7f20881b84f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:27.464 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:27.458 7f2087574700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f20881b7cf0 0x7f208819cdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:41274/0 (socket says 172.21.2.102:41274) 2021-02-10T11:40:27.464 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:27.459 7f2087574700 1 -- 172.21.2.102:0/144530873 learned_addr learned my addr 172.21.2.102:0/144530873 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:40:27.464 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:27.459 7f2087d75700 1 --2- 172.21.2.102:0/144530873 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f208819da20 0x7f208819de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:27.465 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:27.459 7f2087574700 1 -- 172.21.2.102:0/144530873 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f20881b80f0 msgr2=0x7f20881b84f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:27.465 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:27.459 7f2087574700 1 --2- 172.21.2.102:0/144530873 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f20881b80f0 0x7f20881b84f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:27.465 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:27.459 7f2087574700 1 -- 172.21.2.102:0/144530873 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f208819da20 msgr2=0x7f208819de20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:27.465 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:27.459 7f2087574700 1 --2- 172.21.2.102:0/144530873 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f208819da20 0x7f208819de20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:27.465 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:27.459 7f2087574700 1 -- 172.21.2.102:0/144530873 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20881b8d60 con 0x7f20881b7cf0 2021-02-10T11:40:27.465 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:27.459 7f2086d73700 1 --2- 172.21.2.102:0/144530873 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f20881b80f0 0x7f20881b84f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:40:27.465 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:27.459 7f2087574700 1 --2- 172.21.2.102:0/144530873 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f20881b7cf0 0x7f208819cdb0 secure :-1 s=READY pgs=3378 cs=0 l=1 rev1=1 rx=0x7f207800b5c0 tx=0x7f207800b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:27.465 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:27.459 7f207ffff700 1 -- 172.21.2.102:0/144530873 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f207800bd50 con 0x7f20881b7cf0 2021-02-10T11:40:27.465 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:27.459 7f207ffff700 1 -- 172.21.2.102:0/144530873 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2078006030 con 0x7f20881b7cf0 2021-02-10T11:40:27.466 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:27.459 7f208d799700 1 -- 172.21.2.102:0/144530873 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20881b8d60 con 0x7f20881b7cf0 2021-02-10T11:40:27.466 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:27.459 7f208d799700 1 -- 172.21.2.102:0/144530873 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f208819e730 con 0x7f20881b7cf0 2021-02-10T11:40:27.466 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:27.459 7f207ffff700 1 -- 172.21.2.102:0/144530873 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f20780054d0 con 0x7f20881b7cf0 2021-02-10T11:40:27.466 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:27.460 7f207ffff700 1 -- 172.21.2.102:0/144530873 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f2078006210 con 0x7f20881b7cf0 2021-02-10T11:40:27.466 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:27.460 7f207ffff700 1 --2- 172.21.2.102:0/144530873 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f206c0186c0 0x7f206c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:27.466 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:27.460 7f207ffff700 1 -- 172.21.2.102:0/144530873 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f20780325a0 con 0x7f20881b7cf0 2021-02-10T11:40:27.466 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:27.460 7f2086d73700 1 --2- 172.21.2.102:0/144530873 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f206c0186c0 0x7f206c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:27.466 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:27.460 7f2086d73700 1 --2- 172.21.2.102:0/144530873 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f206c0186c0 0x7f206c01abc0 crc :-1 s=READY pgs=1473 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:27.494 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:27.492 7f208d799700 1 -- 172.21.2.102:0/144530873 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20880ebc00 con 0x7f20881b7cf0 2021-02-10T11:40:27.497 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:27.494 7f207ffff700 1 -- 172.21.2.102:0/144530873 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f2078000940 con 0x7f20881b7cf0 2021-02-10T11:40:27.623 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:27.620 7f208d799700 1 -- 172.21.2.102:0/144530873 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f20880f0b10 con 0x7f20881b7cf0 2021-02-10T11:40:27.623 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:27.621 7f207ffff700 1 -- 172.21.2.102:0/144530873 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v197) v1 ==== 74+0+14007 (secure 0 0 0) 0x7f2078006210 con 0x7f20881b7cf0 2021-02-10T11:40:27.624 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:40:27.624 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":197,"fsid":"68db30ea-dbce-4aef-a6df-f01c6fea0513","created":"2021-02-10 11:03:06.331557","modified":"2021-02-10 11:40:25.367346","last_up_change":"2021-02-10 11:03:32.690562","last_in_change":"2021-02-10 11:03:32.690562","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":43,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:03:37.322519","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":42,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:40:22.330567","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"196","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":43,"pool_name":"cephfs_data","create_time":"2021-02-10 11:40:23.357509","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"197","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"be691b4d-a039-408a-af80-9d53f02b6f1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":185,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6818","nonce":36088},{"type":"v1","addr":"172.21.2.102:6820","nonce":36088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6822","nonce":36088},{"type":"v1","addr":"172.21.2.102:6824","nonce":36088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6830","nonce":36088},{"type":"v1","addr":"172.21.2.102:6832","nonce":36088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6826","nonce":36088},{"type":"v1","addr":"172.21.2.102:6828","nonce":36088}]},"public_addr":"172.21.2.102:6820/36088","cluster_addr":"172.21.2.102:6824/36088","heartbeat_back_addr":"172.21.2.102:6832/36088","heartbeat_front_addr":"172.21.2.102:6828/36088","state":["exists","up"]},{"osd":1,"uuid":"64e3ccc8-9c69-4d7c-865e-af8e83822635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6819","nonce":36087},{"type":"v1","addr":"172.21.2.102:6821","nonce":36087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6823","nonce":36087},{"type":"v1","addr":"172.21.2.102:6825","nonce":36087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6831","nonce":36087},{"type":"v1","addr":"172.21.2.102:6833","nonce":36087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6827","nonce":36087},{"type":"v1","addr":"172.21.2.102:6829","nonce":36087}]},"public_addr":"172.21.2.102:6821/36087","cluster_addr":"172.21.2.102:6825/36087","heartbeat_back_addr":"172.21.2.102:6833/36087","heartbeat_front_addr":"172.21.2.102:6829/36087","state":["exists","up"]},{"osd":2,"uuid":"153700af-747f-481f-bdf4-d6072d3a842f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6810","nonce":36089},{"type":"v1","addr":"172.21.2.102:6811","nonce":36089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6812","nonce":36089},{"type":"v1","addr":"172.21.2.102:6813","nonce":36089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6816","nonce":36089},{"type":"v1","addr":"172.21.2.102:6817","nonce":36089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6814","nonce":36089},{"type":"v1","addr":"172.21.2.102:6815","nonce":36089}]},"public_addr":"172.21.2.102:6811/36089","cluster_addr":"172.21.2.102:6813/36089","heartbeat_back_addr":"172.21.2.102:6817/36089","heartbeat_front_addr":"172.21.2.102:6815/36089","state":["exists","up"]},{"osd":3,"uuid":"cbc678f4-83a9-45e6-8830-2f7c232b1ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6802","nonce":36090},{"type":"v1","addr":"172.21.2.102:6803","nonce":36090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6804","nonce":36090},{"type":"v1","addr":"172.21.2.102:6805","nonce":36090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6808","nonce":36090},{"type":"v1","addr":"172.21.2.102:6809","nonce":36090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6806","nonce":36090},{"type":"v1","addr":"172.21.2.102:6807","nonce":36090}]},"public_addr":"172.21.2.102:6803/36090","cluster_addr":"172.21.2.102:6805/36090","heartbeat_back_addr":"172.21.2.102:6809/36090","heartbeat_front_addr":"172.21.2.102:6807/36090","state":["exists","up"]},{"osd":4,"uuid":"c01e4008-b258-4f10-afd2-9e7f37214351","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6808","nonce":36368},{"type":"v1","addr":"172.21.2.113:6811","nonce":36368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6814","nonce":36368},{"type":"v1","addr":"172.21.2.113:6818","nonce":36368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6826","nonce":36368},{"type":"v1","addr":"172.21.2.113:6828","nonce":36368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6820","nonce":36368},{"type":"v1","addr":"172.21.2.113:6824","nonce":36368}]},"public_addr":"172.21.2.113:6811/36368","cluster_addr":"172.21.2.113:6818/36368","heartbeat_back_addr":"172.21.2.113:6828/36368","heartbeat_front_addr":"172.21.2.113:6824/36368","state":["exists","up"]},{"osd":5,"uuid":"551f6007-daf8-4a98-8ee0-c2b16bcccb5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6802","nonce":36366},{"type":"v1","addr":"172.21.2.113:6805","nonce":36366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6807","nonce":36366},{"type":"v1","addr":"172.21.2.113:6810","nonce":36366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6819","nonce":36366},{"type":"v1","addr":"172.21.2.113:6822","nonce":36366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6812","nonce":36366},{"type":"v1","addr":"172.21.2.113:6815","nonce":36366}]},"public_addr":"172.21.2.113:6805/36366","cluster_addr":"172.21.2.113:6810/36366","heartbeat_back_addr":"172.21.2.113:6822/36366","heartbeat_front_addr":"172.21.2.113:6815/36366","state":["exists","up"]},{"osd":6,"uuid":"bf67c040-0e50-48be-b102-c3943bc9ebef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6800","nonce":36367},{"type":"v1","addr":"172.21.2.113:6801","nonce":36367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6803","nonce":36367},{"type":"v1","addr":"172.21.2.113:6804","nonce":36367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6813","nonce":36367},{"type":"v1","addr":"172.21.2.113:6816","nonce":36367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6806","nonce":36367},{"type":"v1","addr":"172.21.2.113:6809","nonce":36367}]},"public_addr":"172.21.2.113:6801/36367","cluster_addr":"172.21.2.113:6804/36367","heartbeat_back_addr":"172.21.2.113:6816/36367","heartbeat_front_addr":"172.21.2.113:6809/36367","state":["exists","up"]},{"osd":7,"uuid":"f5461f80-018d-428f-be89-2eb2bcbe7bb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6817","nonce":36365},{"type":"v1","addr":"172.21.2.113:6821","nonce":36365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6823","nonce":36365},{"type":"v1","addr":"172.21.2.113:6825","nonce":36365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6830","nonce":36365},{"type":"v1","addr":"172.21.2.113:6831","nonce":36365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6827","nonce":36365},{"type":"v1","addr":"172.21.2.113:6829","nonce":36365}]},"public_addr":"172.21.2.113:6821/36365","cluster_addr":"172.21.2.113:6825/36365","heartbeat_back_addr":"172.21.2.113:6831/36365","heartbeat_front_addr":"172.21.2.113:6829/36365","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:40:27.625 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:27.622 7f208d799700 1 -- 172.21.2.102:0/144530873 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f206c0186c0 msgr2=0x7f206c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:27.625 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:27.622 7f208d799700 1 --2- 172.21.2.102:0/144530873 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f206c0186c0 0x7f206c01abc0 crc :-1 s=READY pgs=1473 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:27.625 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:27.622 7f208d799700 1 -- 172.21.2.102:0/144530873 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f20881b7cf0 msgr2=0x7f208819cdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:27.625 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:27.622 7f208d799700 1 --2- 172.21.2.102:0/144530873 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f20881b7cf0 0x7f208819cdb0 secure :-1 s=READY pgs=3378 cs=0 l=1 rev1=1 rx=0x7f207800b5c0 tx=0x7f207800b960).stop 2021-02-10T11:40:27.625 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:27.623 7f208d799700 1 -- 172.21.2.102:0/144530873 shutdown_connections 2021-02-10T11:40:27.626 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:27.623 7f208d799700 1 --2- 172.21.2.102:0/144530873 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f206c0186c0 0x7f206c01abc0 unknown :-1 s=CLOSED pgs=1473 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:27.626 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:27.623 7f208d799700 1 --2- 172.21.2.102:0/144530873 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f208819da20 0x7f208819de20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:27.626 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:27.623 7f208d799700 1 --2- 172.21.2.102:0/144530873 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f20881b80f0 0x7f20881b84f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:27.626 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:27.623 7f208d799700 1 --2- 172.21.2.102:0/144530873 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f20881b7cf0 0x7f208819cdb0 unknown :-1 s=CLOSED pgs=3378 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:27.626 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:27.623 7f208d799700 1 -- 172.21.2.102:0/144530873 >> 172.21.2.102:0/144530873 conn(0x7f20880ed810 msgr2=0x7f2088094980 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:27.626 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:27.623 7f208d799700 1 -- 172.21.2.102:0/144530873 shutdown_connections 2021-02-10T11:40:27.626 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:27.623 7f208d799700 1 -- 172.21.2.102:0/144530873 wait complete. 2021-02-10T11:40:27.634 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:40:28.076 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:28.058 7f3c3a951700 1 Processor -- start 2021-02-10T11:40:28.077 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:28.058 7f3c3a951700 1 -- start start 2021-02-10T11:40:28.077 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:28.058 7f3c3a951700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3c340d1880 0x7f3c341b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:28.077 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:28.058 7f3c3a951700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f3c341b0af0 0x7f3c34108950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:28.077 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:28.058 7f3c3a951700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f3c340d1c80 0x7f3c34108f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:28.077 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:28.058 7f3c3a951700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c340e5f00 con 0x7f3c340d1c80 2021-02-10T11:40:28.077 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:28.058 7f3c3a951700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c34109a70 con 0x7f3c341b0af0 2021-02-10T11:40:28.077 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:28.058 7f3c3a951700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c34088070 con 0x7f3c340d1880 2021-02-10T11:40:28.077 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:28.058 7f3c38f6e700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f3c340d1c80 0x7f3c34108f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:28.077 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:28.059 7f3c38f6e700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f3c340d1c80 0x7f3c34108f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:41280/0 (socket says 172.21.2.102:41280) 2021-02-10T11:40:28.077 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:28.059 7f3c38f6e700 1 -- 172.21.2.102:0/215029308 learned_addr learned my addr 172.21.2.102:0/215029308 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:40:28.077 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:28.059 7f3c38f6e700 1 -- 172.21.2.102:0/215029308 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3c340d1880 msgr2=0x7f3c341b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:28.078 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:28.059 7f3c38f6e700 1 --2- 172.21.2.102:0/215029308 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3c340d1880 0x7f3c341b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:28.078 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:28.059 7f3c38f6e700 1 -- 172.21.2.102:0/215029308 >> v2:172.21.2.113:3300/0 conn(0x7f3c341b0af0 msgr2=0x7f3c34108950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:28.078 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:28.059 7f3c38f6e700 1 --2- 172.21.2.102:0/215029308 >> v2:172.21.2.113:3300/0 conn(0x7f3c341b0af0 0x7f3c34108950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:28.078 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:28.059 7f3c38f6e700 1 -- 172.21.2.102:0/215029308 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c34109590 con 0x7f3c340d1c80 2021-02-10T11:40:28.078 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:28.059 7f3c38f6e700 1 --2- 172.21.2.102:0/215029308 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3c340d1c80 0x7f3c34108f70 secure :-1 s=READY pgs=3379 cs=0 l=1 rev1=1 rx=0x7f3c2c006f00 tx=0x7f3c2c009f10).ready entity=mon.0 client_cookie=538a55dfefefcf0a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:28.078 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:28.059 7f3c337fe700 1 -- 172.21.2.102:0/215029308 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3c2c00b3b0 con 0x7f3c340d1c80 2021-02-10T11:40:28.078 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:28.059 7f3c337fe700 1 -- 172.21.2.102:0/215029308 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c2c00b730 con 0x7f3c340d1c80 2021-02-10T11:40:28.078 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:28.059 7f3c337fe700 1 -- 172.21.2.102:0/215029308 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3c2c00bc50 con 0x7f3c340d1c80 2021-02-10T11:40:28.078 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:28.059 7f3c3a951700 1 -- 172.21.2.102:0/215029308 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3c340d1c80 msgr2=0x7f3c34108f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:28.078 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:28.059 7f3c3a951700 1 --2- 172.21.2.102:0/215029308 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3c340d1c80 0x7f3c34108f70 secure :-1 s=READY pgs=3379 cs=0 l=1 rev1=1 rx=0x7f3c2c006f00 tx=0x7f3c2c009f10).stop 2021-02-10T11:40:28.078 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:28.060 7f3c3a951700 1 -- 172.21.2.102:0/215029308 shutdown_connections 2021-02-10T11:40:28.078 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:28.060 7f3c3a951700 1 --2- 172.21.2.102:0/215029308 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3c340d1c80 0x7f3c34108f70 unknown :-1 s=CLOSED pgs=3379 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:28.079 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:28.060 7f3c3a951700 1 --2- 172.21.2.102:0/215029308 >> v2:172.21.2.113:3300/0 conn(0x7f3c341b0af0 0x7f3c34108950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:28.079 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:28.060 7f3c3a951700 1 --2- 172.21.2.102:0/215029308 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3c340d1880 0x7f3c341b04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:28.079 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:28.060 7f3c3a951700 1 -- 172.21.2.102:0/215029308 >> 172.21.2.102:0/215029308 conn(0x7f3c340ce430 msgr2=0x7f3c340d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:28.079 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:28.060 7f3c3a951700 1 -- 172.21.2.102:0/215029308 shutdown_connections 2021-02-10T11:40:28.079 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:28.060 7f3c3a951700 1 -- 172.21.2.102:0/215029308 wait complete. 2021-02-10T11:40:28.081 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:28.075 7f3c3a951700 1 Processor -- start 2021-02-10T11:40:28.081 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:28.075 7f3c3a951700 1 -- start start 2021-02-10T11:40:28.081 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:28.075 7f3c3a951700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3c34086dd0 0x7f3c34082b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:28.081 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:28.075 7f3c3a951700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3c340ae8f0 0x7f3c340aecf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:28.081 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:28.075 7f3c3a951700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3c341adfb0 0x7f3c341ae3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:28.081 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:28.075 7f3c3a951700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c340871d0 con 0x7f3c34086dd0 2021-02-10T11:40:28.081 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:28.075 7f3c3a951700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c341ae9d0 con 0x7f3c340ae8f0 2021-02-10T11:40:28.081 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:28.075 7f3c3a951700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c341aebf0 con 0x7f3c341adfb0 2021-02-10T11:40:28.081 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:28.075 7f3c33fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3c34086dd0 0x7f3c34082b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:28.082 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:28.075 7f3c33fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3c34086dd0 0x7f3c34082b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:41286/0 (socket says 172.21.2.102:41286) 2021-02-10T11:40:28.082 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:28.075 7f3c33fff700 1 -- 172.21.2.102:0/4239649255 learned_addr learned my addr 172.21.2.102:0/4239649255 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:40:28.082 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:28.076 7f3c38f6e700 1 --2- 172.21.2.102:0/4239649255 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3c341adfb0 0x7f3c341ae3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:28.082 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:28.076 7f3c33fff700 1 -- 172.21.2.102:0/4239649255 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3c341adfb0 msgr2=0x7f3c341ae3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:28.082 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:28.076 7f3c2b7fe700 1 --2- 172.21.2.102:0/4239649255 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3c340ae8f0 0x7f3c340aecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:28.082 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:28.076 7f3c33fff700 1 --2- 172.21.2.102:0/4239649255 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3c341adfb0 0x7f3c341ae3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:28.082 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:28.076 7f3c33fff700 1 -- 172.21.2.102:0/4239649255 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3c340ae8f0 msgr2=0x7f3c340aecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:28.082 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:28.076 7f3c33fff700 1 --2- 172.21.2.102:0/4239649255 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3c340ae8f0 0x7f3c340aecf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:28.082 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:28.076 7f3c33fff700 1 -- 172.21.2.102:0/4239649255 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c34083870 con 0x7f3c34086dd0 2021-02-10T11:40:28.082 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:28.076 7f3c38f6e700 1 --2- 172.21.2.102:0/4239649255 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3c341adfb0 0x7f3c341ae3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:40:28.082 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:28.076 7f3c2b7fe700 1 --2- 172.21.2.102:0/4239649255 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3c340ae8f0 0x7f3c340aecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:40:28.082 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:28.076 7f3c33fff700 1 --2- 172.21.2.102:0/4239649255 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3c34086dd0 0x7f3c34082b30 secure :-1 s=READY pgs=3380 cs=0 l=1 rev1=1 rx=0x7f3c24006f00 tx=0x7f3c2400b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:28.082 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:28.076 7f3c317fa700 1 -- 172.21.2.102:0/4239649255 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3c24006350 con 0x7f3c34086dd0 2021-02-10T11:40:28.083 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:28.076 7f3c317fa700 1 -- 172.21.2.102:0/4239649255 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c2400bd30 con 0x7f3c34086dd0 2021-02-10T11:40:28.083 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:28.076 7f3c317fa700 1 -- 172.21.2.102:0/4239649255 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3c240020d0 con 0x7f3c34086dd0 2021-02-10T11:40:28.083 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:28.076 7f3c3a951700 1 -- 172.21.2.102:0/4239649255 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c34083870 con 0x7f3c34086dd0 2021-02-10T11:40:28.083 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:28.076 7f3c3a951700 1 -- 172.21.2.102:0/4239649255 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c34083e00 con 0x7f3c34086dd0 2021-02-10T11:40:28.083 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:28.077 7f3c317fa700 1 -- 172.21.2.102:0/4239649255 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f3c24005310 con 0x7f3c34086dd0 2021-02-10T11:40:28.083 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:28.077 7f3c317fa700 1 --2- 172.21.2.102:0/4239649255 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3c180186c0 0x7f3c1801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:28.083 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:28.077 7f3c317fa700 1 -- 172.21.2.102:0/4239649255 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f3c24032570 con 0x7f3c34086dd0 2021-02-10T11:40:28.083 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:28.077 7f3c2b7fe700 1 --2- 172.21.2.102:0/4239649255 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3c180186c0 0x7f3c1801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:28.083 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:28.077 7f3c2b7fe700 1 --2- 172.21.2.102:0/4239649255 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3c180186c0 0x7f3c1801abc0 crc :-1 s=READY pgs=1474 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:28.110 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:28.108 7f3c3a951700 1 -- 172.21.2.102:0/4239649255 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c34082900 con 0x7f3c34086dd0 2021-02-10T11:40:28.113 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:28.111 7f3c317fa700 1 -- 172.21.2.102:0/4239649255 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f3c24006350 con 0x7f3c34086dd0 2021-02-10T11:40:28.242 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:28.239 7f3c3a951700 1 -- 172.21.2.102:0/4239649255 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f3c34109a70 con 0x7f3c34086dd0 2021-02-10T11:40:28.245 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:28.243 7f3c317fa700 1 -- 172.21.2.102:0/4239649255 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.0 v118) v1 ==== 184+0+0 (secure 0 0 0) 0x7f3c2401f4d0 con 0x7f3c34086dd0 2021-02-10T11:40:28.249 INFO:teuthology.orchestra.run.gibba002.stderr:updated caps for client.0 2021-02-10T11:40:28.250 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:28.248 7f3c3a951700 1 -- 172.21.2.102:0/4239649255 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3c180186c0 msgr2=0x7f3c1801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:28.250 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:28.248 7f3c3a951700 1 --2- 172.21.2.102:0/4239649255 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3c180186c0 0x7f3c1801abc0 crc :-1 s=READY pgs=1474 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:28.250 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:28.248 7f3c3a951700 1 -- 172.21.2.102:0/4239649255 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3c34086dd0 msgr2=0x7f3c34082b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:28.250 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:28.248 7f3c3a951700 1 --2- 172.21.2.102:0/4239649255 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3c34086dd0 0x7f3c34082b30 secure :-1 s=READY pgs=3380 cs=0 l=1 rev1=1 rx=0x7f3c24006f00 tx=0x7f3c2400b940).stop 2021-02-10T11:40:28.251 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:28.248 7f3c3a951700 1 -- 172.21.2.102:0/4239649255 shutdown_connections 2021-02-10T11:40:28.251 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:28.248 7f3c3a951700 1 --2- 172.21.2.102:0/4239649255 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f3c341adfb0 0x7f3c341ae3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:28.251 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:28.248 7f3c3a951700 1 --2- 172.21.2.102:0/4239649255 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f3c180186c0 0x7f3c1801abc0 unknown :-1 s=CLOSED pgs=1474 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:28.251 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:28.248 7f3c3a951700 1 --2- 172.21.2.102:0/4239649255 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f3c340ae8f0 0x7f3c340aecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:28.251 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:28.248 7f3c3a951700 1 --2- 172.21.2.102:0/4239649255 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f3c34086dd0 0x7f3c34082b30 unknown :-1 s=CLOSED pgs=3380 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:28.252 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:28.248 7f3c3a951700 1 -- 172.21.2.102:0/4239649255 >> 172.21.2.102:0/4239649255 conn(0x7f3c341b9600 msgr2=0x7f3c341b0a50 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:28.252 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:28.249 7f3c3a951700 1 -- 172.21.2.102:0/4239649255 shutdown_connections 2021-02-10T11:40:28.252 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:28.249 7f3c3a951700 1 -- 172.21.2.102:0/4239649255 wait complete. 2021-02-10T11:40:28.259 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.1 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:40:28.691 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:28.671 7f588c925700 1 Processor -- start 2021-02-10T11:40:28.691 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:28.671 7f588c925700 1 -- start start 2021-02-10T11:40:28.691 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:28.672 7f588c925700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f58880d1880 0x7f58881b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:28.691 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:28.672 7f588c925700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f58881b0af0 0x7f5888108950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:28.691 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:28.672 7f588c925700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5888109a70 con 0x7f58880d1c80 2021-02-10T11:40:28.691 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:28.672 7f588c925700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58880e5f00 con 0x7f58881b0af0 2021-02-10T11:40:28.691 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:28.672 7f588c925700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5888109760 con 0x7f58880d1880 2021-02-10T11:40:28.691 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:28.673 7f5886e1d700 1 --1- >> v1:172.21.2.113:6789/0 conn(0x7f58880d1c80 0x7f5888108f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.113:6789/0 says I am v1:172.21.2.102:40840/0 (socket says 172.21.2.102:40840) 2021-02-10T11:40:28.692 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:28.673 7f5886e1d700 1 -- 172.21.2.102:0/1283556531 learned_addr learned my addr 172.21.2.102:0/1283556531 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:40:28.692 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:28.673 7f588661c700 1 --2- 172.21.2.102:0/1283556531 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f58880d1880 0x7f58881b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:28.692 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:28.673 7f5885e1b700 1 --2- 172.21.2.102:0/1283556531 >> v2:172.21.2.113:3300/0 conn(0x7f58881b0af0 0x7f5888108950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:28.692 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:28.673 7f588561a700 1 -- 172.21.2.102:0/1283556531 <== mon.1 v1:172.21.2.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1124471678 0 0) 0x7f5880001510 con 0x7f58880d1c80 2021-02-10T11:40:28.692 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:28.673 7f588561a700 1 -- 172.21.2.102:0/1283556531 --> v1:172.21.2.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5874002fe0 con 0x7f58880d1c80 2021-02-10T11:40:28.692 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:28.673 7f588661c700 1 -- 172.21.2.102:0/1283556531 >> v2:172.21.2.113:3300/0 conn(0x7f58881b0af0 msgr2=0x7f5888108950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:28.692 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:28.673 7f588661c700 1 --2- 172.21.2.102:0/1283556531 >> v2:172.21.2.113:3300/0 conn(0x7f58881b0af0 0x7f5888108950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:28.692 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:28.673 7f588661c700 1 -- 172.21.2.102:0/1283556531 >> v1:172.21.2.113:6789/0 conn(0x7f58880d1c80 legacy=0x7f5888108f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:28.692 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:28.673 7f588661c700 1 -- 172.21.2.102:0/1283556531 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5888088070 con 0x7f58880d1880 2021-02-10T11:40:28.692 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:28.673 7f5885e1b700 1 --2- 172.21.2.102:0/1283556531 >> v2:172.21.2.113:3300/0 conn(0x7f58881b0af0 0x7f5888108950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:40:28.693 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:28.673 7f588661c700 1 --2- 172.21.2.102:0/1283556531 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f58880d1880 0x7f58881b04d0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 rx=0x7f5870006650 tx=0x7f5870009a80).ready entity=mon.2 client_cookie=e0ae86c8c16f480f server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:28.693 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:28.674 7f588561a700 1 -- 172.21.2.102:0/1283556531 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f58700009b0 con 0x7f58880d1880 2021-02-10T11:40:28.693 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:28.674 7f588561a700 1 -- 172.21.2.102:0/1283556531 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f587000c0c0 con 0x7f58880d1880 2021-02-10T11:40:28.693 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:28.674 7f588561a700 1 -- 172.21.2.102:0/1283556531 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f587000c5e0 con 0x7f58880d1880 2021-02-10T11:40:28.693 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:28.674 7f588c925700 1 -- 172.21.2.102:0/1283556531 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f58880d1880 msgr2=0x7f58881b04d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:28.693 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:28.674 7f588c925700 1 --2- 172.21.2.102:0/1283556531 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f58880d1880 0x7f58881b04d0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 rx=0x7f5870006650 tx=0x7f5870009a80).stop 2021-02-10T11:40:28.693 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:28.674 7f588c925700 1 -- 172.21.2.102:0/1283556531 shutdown_connections 2021-02-10T11:40:28.693 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:28.674 7f588c925700 1 --2- 172.21.2.102:0/1283556531 >> v2:172.21.2.113:3300/0 conn(0x7f58881b0af0 0x7f5888108950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:28.693 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:28.674 7f588c925700 1 --2- 172.21.2.102:0/1283556531 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f58880d1880 0x7f58881b04d0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:28.693 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:28.674 7f588c925700 1 -- 172.21.2.102:0/1283556531 >> 172.21.2.102:0/1283556531 conn(0x7f58880ce430 msgr2=0x7f58880d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:28.693 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:28.675 7f588c925700 1 -- 172.21.2.102:0/1283556531 shutdown_connections 2021-02-10T11:40:28.693 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:28.675 7f588c925700 1 -- 172.21.2.102:0/1283556531 wait complete. 2021-02-10T11:40:28.695 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:28.689 7f588c925700 1 Processor -- start 2021-02-10T11:40:28.695 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:28.689 7f588c925700 1 -- start start 2021-02-10T11:40:28.695 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:28.689 7f588c925700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5888085300 0x7f58880bf1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:28.695 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:28.689 7f588c925700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f58880bf7d0 0x7f5888085fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:28.696 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:28.689 7f588c925700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f58880bfbd0 0x7f58880865d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:28.696 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:28.689 7f588c925700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5888084d80 con 0x7f58880bf7d0 2021-02-10T11:40:28.696 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:28.689 7f588c925700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5888085700 con 0x7f5888085300 2021-02-10T11:40:28.696 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:28.689 7f588c925700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5888086bf0 con 0x7f58880bfbd0 2021-02-10T11:40:28.696 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:28.690 7f5886e1d700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f58880bfbd0 0x7f58880865d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:28.696 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:28.690 7f5885e1b700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f58880bf7d0 0x7f5888085fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:28.696 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:28.690 7f5886e1d700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f58880bfbd0 0x7f58880865d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.113:3301/0 says I am v2:172.21.2.102:46836/0 (socket says 172.21.2.102:46836) 2021-02-10T11:40:28.696 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:28.690 7f5886e1d700 1 -- 172.21.2.102:0/2485160982 learned_addr learned my addr 172.21.2.102:0/2485160982 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:40:28.696 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:28.690 7f588661c700 1 --2- 172.21.2.102:0/2485160982 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5888085300 0x7f58880bf1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:28.696 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:28.690 7f5886e1d700 1 -- 172.21.2.102:0/2485160982 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5888085300 msgr2=0x7f58880bf1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:28.696 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:28.690 7f5886e1d700 1 --2- 172.21.2.102:0/2485160982 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5888085300 0x7f58880bf1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:28.696 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:28.690 7f5886e1d700 1 -- 172.21.2.102:0/2485160982 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f58880bf7d0 msgr2=0x7f5888085fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:28.697 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:28.690 7f5886e1d700 1 --2- 172.21.2.102:0/2485160982 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f58880bf7d0 0x7f5888085fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:28.697 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:28.690 7f5886e1d700 1 -- 172.21.2.102:0/2485160982 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5888086e10 con 0x7f58880bfbd0 2021-02-10T11:40:28.697 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:28.690 7f5886e1d700 1 --2- 172.21.2.102:0/2485160982 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f58880bfbd0 0x7f58880865d0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 rx=0x7f5880007f90 tx=0x7f588000aca0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:28.697 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:28.690 7f587effd700 1 -- 172.21.2.102:0/2485160982 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f58800011f0 con 0x7f58880bfbd0 2021-02-10T11:40:28.697 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:28.690 7f587effd700 1 -- 172.21.2.102:0/2485160982 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5880007020 con 0x7f58880bfbd0 2021-02-10T11:40:28.697 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:28.690 7f588c925700 1 -- 172.21.2.102:0/2485160982 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5888086e10 con 0x7f58880bfbd0 2021-02-10T11:40:28.697 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:28.690 7f587effd700 1 -- 172.21.2.102:0/2485160982 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5880000f40 con 0x7f58880bfbd0 2021-02-10T11:40:28.697 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:28.690 7f588c925700 1 -- 172.21.2.102:0/2485160982 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5888086e10 con 0x7f58880bfbd0 2021-02-10T11:40:28.697 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:28.691 7f587effd700 1 -- 172.21.2.102:0/2485160982 <== mon.2 v2:172.21.2.113:3301/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f5880007200 con 0x7f58880bfbd0 2021-02-10T11:40:28.697 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:28.691 7f587effd700 1 --2- 172.21.2.102:0/2485160982 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f586c0186c0 0x7f586c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:28.697 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:28.691 7f587effd700 1 -- 172.21.2.102:0/2485160982 <== mon.2 v2:172.21.2.113:3301/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f5880032840 con 0x7f58880bfbd0 2021-02-10T11:40:28.697 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:28.691 7f588661c700 1 --2- 172.21.2.102:0/2485160982 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f586c0186c0 0x7f586c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:28.697 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:28.692 7f588661c700 1 --2- 172.21.2.102:0/2485160982 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f586c0186c0 0x7f586c01abc0 crc :-1 s=READY pgs=1475 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:28.724 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:28.722 7f588c925700 1 -- 172.21.2.102:0/2485160982 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58880829a0 con 0x7f58880bfbd0 2021-02-10T11:40:28.727 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:28.724 7f587effd700 1 -- 172.21.2.102:0/2485160982 <== mon.2 v2:172.21.2.113:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f5880000f20 con 0x7f58880bfbd0 2021-02-10T11:40:28.857 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:28.855 7f588c925700 1 -- 172.21.2.102:0/2485160982 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_command({"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f5888109a70 con 0x7f58880bfbd0 2021-02-10T11:40:28.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:28.859 7f587effd700 1 -- 172.21.2.102:0/2485160982 <== mon.2 v2:172.21.2.113:3301/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.1 v119) v1 ==== 184+0+0 (secure 0 0 0) 0x7f588000ade0 con 0x7f58880bfbd0 2021-02-10T11:40:28.864 INFO:teuthology.orchestra.run.gibba002.stderr:updated caps for client.1 2021-02-10T11:40:28.866 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:28.864 7f588c925700 1 -- 172.21.2.102:0/2485160982 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f586c0186c0 msgr2=0x7f586c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:28.866 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:28.864 7f588c925700 1 --2- 172.21.2.102:0/2485160982 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f586c0186c0 0x7f586c01abc0 crc :-1 s=READY pgs=1475 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:28.866 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:28.864 7f588c925700 1 -- 172.21.2.102:0/2485160982 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f58880bfbd0 msgr2=0x7f58880865d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:28.866 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:28.864 7f588c925700 1 --2- 172.21.2.102:0/2485160982 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f58880bfbd0 0x7f58880865d0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 rx=0x7f5880007f90 tx=0x7f588000aca0).stop 2021-02-10T11:40:28.867 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:28.864 7f588c925700 1 -- 172.21.2.102:0/2485160982 shutdown_connections 2021-02-10T11:40:28.867 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:28.864 7f588c925700 1 --2- 172.21.2.102:0/2485160982 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f58880bfbd0 0x7f58880865d0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:28.867 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:28.864 7f588c925700 1 --2- 172.21.2.102:0/2485160982 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f58880bf7d0 0x7f5888085fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:28.867 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:28.864 7f588c925700 1 --2- 172.21.2.102:0/2485160982 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f586c0186c0 0x7f586c01abc0 unknown :-1 s=CLOSED pgs=1475 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:28.867 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:28.864 7f588c925700 1 --2- 172.21.2.102:0/2485160982 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5888085300 0x7f58880bf1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:28.867 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:28.864 7f588c925700 1 -- 172.21.2.102:0/2485160982 >> 172.21.2.102:0/2485160982 conn(0x7f58880bd7a0 msgr2=0x7f58880b5900 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:28.867 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:28.864 7f588c925700 1 -- 172.21.2.102:0/2485160982 shutdown_connections 2021-02-10T11:40:28.867 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:28.864 7f588c925700 1 -- 172.21.2.102:0/2485160982 wait complete. 2021-02-10T11:40:28.874 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.2 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:40:29.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:29.296 7fc01187e700 1 Processor -- start 2021-02-10T11:40:29.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:29.297 7fc01187e700 1 -- start start 2021-02-10T11:40:29.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:29.297 7fc01187e700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fc00c0d1880 0x7fc00c1b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:29.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:29.297 7fc01187e700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc00c0fe1d0 con 0x7fc00c1b0af0 2021-02-10T11:40:29.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:29.297 7fc01187e700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc00c109a70 con 0x7fc00c0cff80 2021-02-10T11:40:29.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:29.297 7fc01187e700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc00c088070 con 0x7fc00c0d1880 2021-02-10T11:40:29.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:29.297 7fc00ad73700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fc00c1b0af0 0x7fc00c108950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:33532/0 (socket says 172.21.2.102:33532) 2021-02-10T11:40:29.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:29.297 7fc00ad73700 1 -- 172.21.2.102:0/2741818364 learned_addr learned my addr 172.21.2.102:0/2741818364 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:40:29.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:29.297 7fc00b574700 1 --2- 172.21.2.102:0/2741818364 >> v2:172.21.2.102:3300/0 conn(0x7fc00c0d1880 0x7fc00c1b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:29.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:29.298 7fc00a572700 1 -- 172.21.2.102:0/2741818364 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3909410188 0 0) 0x7fbffc0009b0 con 0x7fc00c1b0af0 2021-02-10T11:40:29.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:29.298 7fc00a572700 1 -- 172.21.2.102:0/2741818364 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbff8002fe0 con 0x7fc00c1b0af0 2021-02-10T11:40:29.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:29.298 7fc00b574700 1 -- 172.21.2.102:0/2741818364 >> v1:172.21.2.113:6789/0 conn(0x7fc00c0cff80 legacy=0x7fc00c109140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:29.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:29.298 7fc00b574700 1 -- 172.21.2.102:0/2741818364 >> v1:172.21.2.102:6789/0 conn(0x7fc00c1b0af0 legacy=0x7fc00c108950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:29.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:29.298 7fc00b574700 1 -- 172.21.2.102:0/2741818364 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc00c088260 con 0x7fc00c0d1880 2021-02-10T11:40:29.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:29.298 7fc00b574700 1 --2- 172.21.2.102:0/2741818364 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc00c0d1880 0x7fc00c1b04d0 secure :-1 s=READY pgs=3382 cs=0 l=1 rev1=1 rx=0x7fbff4006650 tx=0x7fbff4009a80).ready entity=mon.0 client_cookie=e898442aa6e93481 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:29.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:29.298 7fc00a572700 1 -- 172.21.2.102:0/2741818364 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbff400c0c0 con 0x7fc00c0d1880 2021-02-10T11:40:29.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:29.298 7fc00a572700 1 -- 172.21.2.102:0/2741818364 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbff400c340 con 0x7fc00c0d1880 2021-02-10T11:40:29.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:29.298 7fc00a572700 1 -- 172.21.2.102:0/2741818364 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbff400c860 con 0x7fc00c0d1880 2021-02-10T11:40:29.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:29.298 7fc01187e700 1 -- 172.21.2.102:0/2741818364 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc00c0d1880 msgr2=0x7fc00c1b04d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:29.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:29.298 7fc01187e700 1 --2- 172.21.2.102:0/2741818364 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc00c0d1880 0x7fc00c1b04d0 secure :-1 s=READY pgs=3382 cs=0 l=1 rev1=1 rx=0x7fbff4006650 tx=0x7fbff4009a80).stop 2021-02-10T11:40:29.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:29.299 7fc01187e700 1 -- 172.21.2.102:0/2741818364 shutdown_connections 2021-02-10T11:40:29.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:29.299 7fc01187e700 1 --2- 172.21.2.102:0/2741818364 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc00c0d1880 0x7fc00c1b04d0 unknown :-1 s=CLOSED pgs=3382 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:29.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:29.299 7fc01187e700 1 -- 172.21.2.102:0/2741818364 >> 172.21.2.102:0/2741818364 conn(0x7fc00c0ce430 msgr2=0x7fc00c0d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:29.318 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:29.299 7fc01187e700 1 -- 172.21.2.102:0/2741818364 shutdown_connections 2021-02-10T11:40:29.318 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:29.299 7fc01187e700 1 -- 172.21.2.102:0/2741818364 wait complete. 2021-02-10T11:40:29.319 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:29.314 7fc01187e700 1 Processor -- start 2021-02-10T11:40:29.319 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:29.314 7fc01187e700 1 -- start start 2021-02-10T11:40:29.319 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:29.314 7fc01187e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc00c1a68f0 0x7fc00c19fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:29.319 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:29.314 7fc01187e700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc00c1a5e00 0x7fc00c1a02c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:29.319 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:29.314 7fc01187e700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc00c1a08e0 0x7fc00c1aae20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:29.320 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:29.314 7fc01187e700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc00c1a6370 con 0x7fc00c1a68f0 2021-02-10T11:40:29.320 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:29.314 7fc01187e700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc00c1a3960 con 0x7fc00c1a08e0 2021-02-10T11:40:29.320 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:29.314 7fc01187e700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc00c1ab440 con 0x7fc00c1a5e00 2021-02-10T11:40:29.320 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:29.314 7fc00b574700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc00c1a68f0 0x7fc00c19fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:29.320 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:29.314 7fc00b574700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc00c1a68f0 0x7fc00c19fca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:41310/0 (socket says 172.21.2.102:41310) 2021-02-10T11:40:29.320 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:29.314 7fc00b574700 1 -- 172.21.2.102:0/2284198744 learned_addr learned my addr 172.21.2.102:0/2284198744 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:40:29.320 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:29.315 7fc00b574700 1 -- 172.21.2.102:0/2284198744 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc00c1a5e00 msgr2=0x7fc00c1a02c0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:40:29.320 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:29.315 7fc00b574700 1 --2- 172.21.2.102:0/2284198744 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc00c1a5e00 0x7fc00c1a02c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:29.320 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:29.315 7fc00b574700 1 -- 172.21.2.102:0/2284198744 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc00c1a08e0 msgr2=0x7fc00c1aae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:29.320 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:29.315 7fc00b574700 1 --2- 172.21.2.102:0/2284198744 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc00c1a08e0 0x7fc00c1aae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:29.320 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:29.315 7fc00b574700 1 -- 172.21.2.102:0/2284198744 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc00c1ab660 con 0x7fc00c1a68f0 2021-02-10T11:40:29.320 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:29.315 7fc00b574700 1 --2- 172.21.2.102:0/2284198744 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc00c1a68f0 0x7fc00c19fca0 secure :-1 s=READY pgs=3383 cs=0 l=1 rev1=1 rx=0x7fbff4009630 tx=0x7fbff400b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:29.320 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:29.315 7fc003fff700 1 -- 172.21.2.102:0/2284198744 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbff40093a0 con 0x7fc00c1a68f0 2021-02-10T11:40:29.321 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:29.315 7fc003fff700 1 -- 172.21.2.102:0/2284198744 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbff400bd70 con 0x7fc00c1a68f0 2021-02-10T11:40:29.321 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:29.315 7fc01187e700 1 -- 172.21.2.102:0/2284198744 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc00c1ab660 con 0x7fc00c1a68f0 2021-02-10T11:40:29.321 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:29.315 7fc01187e700 1 -- 172.21.2.102:0/2284198744 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc00c1abbf0 con 0x7fc00c1a68f0 2021-02-10T11:40:29.321 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:29.315 7fc003fff700 1 -- 172.21.2.102:0/2284198744 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbff4005560 con 0x7fc00c1a68f0 2021-02-10T11:40:29.321 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:29.316 7fc003fff700 1 -- 172.21.2.102:0/2284198744 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fbff400a000 con 0x7fc00c1a68f0 2021-02-10T11:40:29.321 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:29.316 7fc003fff700 1 --2- 172.21.2.102:0/2284198744 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fbff00186c0 0x7fbff001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:29.321 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:29.316 7fc003fff700 1 -- 172.21.2.102:0/2284198744 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fbff4032550 con 0x7fc00c1a68f0 2021-02-10T11:40:29.321 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:29.316 7fc00ad73700 1 --2- 172.21.2.102:0/2284198744 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fbff00186c0 0x7fbff001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:29.321 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:29.316 7fc00ad73700 1 --2- 172.21.2.102:0/2284198744 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fbff00186c0 0x7fbff001abc0 crc :-1 s=READY pgs=1476 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:29.349 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:29.347 7fc01187e700 1 -- 172.21.2.102:0/2284198744 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc00c0829a0 con 0x7fc00c1a68f0 2021-02-10T11:40:29.352 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:29.350 7fc003fff700 1 -- 172.21.2.102:0/2284198744 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fbff4030dd0 con 0x7fc00c1a68f0 2021-02-10T11:40:29.481 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:29.479 7fc01187e700 1 -- 172.21.2.102:0/2284198744 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7fc00c0829a0 con 0x7fc00c1a68f0 2021-02-10T11:40:29.485 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:29.483 7fc003fff700 1 -- 172.21.2.102:0/2284198744 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.2 v120) v1 ==== 184+0+0 (secure 0 0 0) 0x7fbff401f4e0 con 0x7fc00c1a68f0 2021-02-10T11:40:29.488 INFO:teuthology.orchestra.run.gibba002.stderr:updated caps for client.2 2021-02-10T11:40:29.490 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:29.488 7fc01187e700 1 -- 172.21.2.102:0/2284198744 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fbff00186c0 msgr2=0x7fbff001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:29.490 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:29.488 7fc01187e700 1 --2- 172.21.2.102:0/2284198744 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fbff00186c0 0x7fbff001abc0 crc :-1 s=READY pgs=1476 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:29.490 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:29.488 7fc01187e700 1 -- 172.21.2.102:0/2284198744 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc00c1a68f0 msgr2=0x7fc00c19fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:29.490 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:29.488 7fc01187e700 1 --2- 172.21.2.102:0/2284198744 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc00c1a68f0 0x7fc00c19fca0 secure :-1 s=READY pgs=3383 cs=0 l=1 rev1=1 rx=0x7fbff4009630 tx=0x7fbff400b980).stop 2021-02-10T11:40:29.491 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:29.488 7fc01187e700 1 -- 172.21.2.102:0/2284198744 shutdown_connections 2021-02-10T11:40:29.491 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:29.488 7fc01187e700 1 --2- 172.21.2.102:0/2284198744 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fbff00186c0 0x7fbff001abc0 unknown :-1 s=CLOSED pgs=1476 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:29.491 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:29.488 7fc01187e700 1 --2- 172.21.2.102:0/2284198744 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc00c1a08e0 0x7fc00c1aae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:29.491 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:29.488 7fc01187e700 1 --2- 172.21.2.102:0/2284198744 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc00c1a5e00 0x7fc00c1a02c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:29.491 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:29.488 7fc01187e700 1 --2- 172.21.2.102:0/2284198744 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc00c1a68f0 0x7fc00c19fca0 unknown :-1 s=CLOSED pgs=3383 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:29.491 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:29.488 7fc01187e700 1 -- 172.21.2.102:0/2284198744 >> 172.21.2.102:0/2284198744 conn(0x7fc00c19f880 msgr2=0x7fc00c1a71a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:29.491 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:29.488 7fc01187e700 1 -- 172.21.2.102:0/2284198744 shutdown_connections 2021-02-10T11:40:29.491 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:29.489 7fc01187e700 1 -- 172.21.2.102:0/2284198744 wait complete. 2021-02-10T11:40:29.499 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.3 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:40:29.940 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:29.921 7f26c935f700 1 Processor -- start 2021-02-10T11:40:29.941 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:29.921 7f26c935f700 1 -- start start 2021-02-10T11:40:29.941 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:29.922 7f26c935f700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f26c40d1880 0x7f26c41b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:29.941 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:29.922 7f26c935f700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f26c41b0af0 0x7f26c4108950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:29.941 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:29.922 7f26c935f700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26c4109a70 con 0x7f26c40d1c80 2021-02-10T11:40:29.941 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:29.922 7f26c935f700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26c40e5f00 con 0x7f26c41b0af0 2021-02-10T11:40:29.941 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:29.922 7f26c935f700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26c4109760 con 0x7f26c40d1880 2021-02-10T11:40:29.941 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:29.923 7f26c387e700 1 --1- >> v1:172.21.2.113:6789/0 conn(0x7f26c40d1c80 0x7f26c4108f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.113:6789/0 says I am v1:172.21.2.102:40866/0 (socket says 172.21.2.102:40866) 2021-02-10T11:40:29.941 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:29.923 7f26c387e700 1 -- 172.21.2.102:0/254466145 learned_addr learned my addr 172.21.2.102:0/254466145 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:40:29.941 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:29.923 7f26c307d700 1 --2- 172.21.2.102:0/254466145 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f26c40d1880 0x7f26c41b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:29.941 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:29.923 7f26c287c700 1 --2- 172.21.2.102:0/254466145 >> v2:172.21.2.113:3300/0 conn(0x7f26c41b0af0 0x7f26c4108950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:29.942 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:29.923 7f26c207b700 1 -- 172.21.2.102:0/254466145 <== mon.1 v1:172.21.2.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2189325554 0 0) 0x7f26bc001510 con 0x7f26c40d1c80 2021-02-10T11:40:29.942 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:29.923 7f26c207b700 1 -- 172.21.2.102:0/254466145 --> v1:172.21.2.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26ac002fe0 con 0x7f26c40d1c80 2021-02-10T11:40:29.942 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:29.923 7f26c307d700 1 -- 172.21.2.102:0/254466145 >> v2:172.21.2.113:3300/0 conn(0x7f26c41b0af0 msgr2=0x7f26c4108950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:29.942 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:29.923 7f26c307d700 1 --2- 172.21.2.102:0/254466145 >> v2:172.21.2.113:3300/0 conn(0x7f26c41b0af0 0x7f26c4108950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:29.942 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:29.923 7f26c307d700 1 -- 172.21.2.102:0/254466145 >> v1:172.21.2.113:6789/0 conn(0x7f26c40d1c80 legacy=0x7f26c4108f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:29.942 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:29.923 7f26c307d700 1 -- 172.21.2.102:0/254466145 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26c4088070 con 0x7f26c40d1880 2021-02-10T11:40:29.942 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:29.923 7f26c287c700 1 --2- 172.21.2.102:0/254466145 >> v2:172.21.2.113:3300/0 conn(0x7f26c41b0af0 0x7f26c4108950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:40:29.942 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:29.923 7f26c307d700 1 --2- 172.21.2.102:0/254466145 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f26c40d1880 0x7f26c41b04d0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 rx=0x7f26b4006650 tx=0x7f26b4009a80).ready entity=mon.2 client_cookie=e073dca52b69ef10 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:29.942 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:29.924 7f26c207b700 1 -- 172.21.2.102:0/254466145 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f26b400c0c0 con 0x7f26c40d1880 2021-02-10T11:40:29.942 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:29.924 7f26c207b700 1 -- 172.21.2.102:0/254466145 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26b400c340 con 0x7f26c40d1880 2021-02-10T11:40:29.942 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:29.924 7f26c207b700 1 -- 172.21.2.102:0/254466145 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f26b400c860 con 0x7f26c40d1880 2021-02-10T11:40:29.942 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:29.924 7f26c935f700 1 -- 172.21.2.102:0/254466145 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f26c40d1880 msgr2=0x7f26c41b04d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:29.942 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:29.924 7f26c935f700 1 --2- 172.21.2.102:0/254466145 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f26c40d1880 0x7f26c41b04d0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 rx=0x7f26b4006650 tx=0x7f26b4009a80).stop 2021-02-10T11:40:29.943 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:29.924 7f26c935f700 1 -- 172.21.2.102:0/254466145 shutdown_connections 2021-02-10T11:40:29.943 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:29.924 7f26c935f700 1 --2- 172.21.2.102:0/254466145 >> v2:172.21.2.113:3300/0 conn(0x7f26c41b0af0 0x7f26c4108950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:29.943 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:29.924 7f26c935f700 1 --2- 172.21.2.102:0/254466145 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f26c40d1880 0x7f26c41b04d0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:29.943 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:29.924 7f26c935f700 1 -- 172.21.2.102:0/254466145 >> 172.21.2.102:0/254466145 conn(0x7f26c40ce430 msgr2=0x7f26c40d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:29.943 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:29.925 7f26c935f700 1 -- 172.21.2.102:0/254466145 shutdown_connections 2021-02-10T11:40:29.943 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:29.925 7f26c935f700 1 -- 172.21.2.102:0/254466145 wait complete. 2021-02-10T11:40:29.944 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:29.939 7f26c935f700 1 Processor -- start 2021-02-10T11:40:29.944 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:29.939 7f26c935f700 1 -- start start 2021-02-10T11:40:29.944 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:29.939 7f26c935f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f26c41a49b0 0x7f26c408aa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:29.945 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:29.939 7f26c935f700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f26c408b0a0 0x7f26c408b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:29.945 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:29.939 7f26c935f700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f26c408bac0 0x7f26c40bf1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:29.945 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:29.939 7f26c935f700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26c40b2550 con 0x7f26c41a49b0 2021-02-10T11:40:29.945 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:29.939 7f26c935f700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26c40bf7d0 con 0x7f26c408b0a0 2021-02-10T11:40:29.945 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:29.939 7f26c935f700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26c40bf9f0 con 0x7f26c408bac0 2021-02-10T11:40:29.945 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:29.939 7f26c307d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f26c41a49b0 0x7f26c408aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:29.945 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:29.939 7f26c307d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f26c41a49b0 0x7f26c408aa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:41326/0 (socket says 172.21.2.102:41326) 2021-02-10T11:40:29.945 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:29.939 7f26c307d700 1 -- 172.21.2.102:0/1985133506 learned_addr learned my addr 172.21.2.102:0/1985133506 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:40:29.945 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:29.940 7f26c307d700 1 -- 172.21.2.102:0/1985133506 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f26c408bac0 msgr2=0x7f26c40bf1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:29.945 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:29.940 7f26c307d700 1 --2- 172.21.2.102:0/1985133506 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f26c408bac0 0x7f26c40bf1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:29.945 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:29.940 7f26c307d700 1 -- 172.21.2.102:0/1985133506 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f26c408b0a0 msgr2=0x7f26c408b4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:29.946 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:29.940 7f26c307d700 1 --2- 172.21.2.102:0/1985133506 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f26c408b0a0 0x7f26c408b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:29.946 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:29.940 7f26c307d700 1 -- 172.21.2.102:0/1985133506 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26c40bfc10 con 0x7f26c41a49b0 2021-02-10T11:40:29.946 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:29.940 7f26c307d700 1 --2- 172.21.2.102:0/1985133506 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f26c41a49b0 0x7f26c408aa80 secure :-1 s=READY pgs=3384 cs=0 l=1 rev1=1 rx=0x7f26b4009630 tx=0x7f26b400b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:29.946 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:29.940 7f26abfff700 1 -- 172.21.2.102:0/1985133506 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f26b40093a0 con 0x7f26c41a49b0 2021-02-10T11:40:29.946 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:29.940 7f26abfff700 1 -- 172.21.2.102:0/1985133506 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26b400bd70 con 0x7f26c41a49b0 2021-02-10T11:40:29.946 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:29.940 7f26abfff700 1 -- 172.21.2.102:0/1985133506 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f26b40054d0 con 0x7f26c41a49b0 2021-02-10T11:40:29.946 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:29.940 7f26c935f700 1 -- 172.21.2.102:0/1985133506 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26c40bfc10 con 0x7f26c41a49b0 2021-02-10T11:40:29.946 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:29.940 7f26c935f700 1 -- 172.21.2.102:0/1985133506 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26c40c01a0 con 0x7f26c41a49b0 2021-02-10T11:40:29.946 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:29.941 7f26abfff700 1 -- 172.21.2.102:0/1985133506 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f26b400a000 con 0x7f26c41a49b0 2021-02-10T11:40:29.946 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:29.941 7f26abfff700 1 --2- 172.21.2.102:0/1985133506 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f26a40186c0 0x7f26a401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:29.946 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:29.941 7f26abfff700 1 -- 172.21.2.102:0/1985133506 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f26b4032580 con 0x7f26c41a49b0 2021-02-10T11:40:29.946 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:29.941 7f26c287c700 1 --2- 172.21.2.102:0/1985133506 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f26a40186c0 0x7f26a401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:29.946 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:29.941 7f26c287c700 1 --2- 172.21.2.102:0/1985133506 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f26a40186c0 0x7f26a401abc0 crc :-1 s=READY pgs=1477 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:29.974 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:29.972 7f26c935f700 1 -- 172.21.2.102:0/1985133506 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26c408a540 con 0x7f26c41a49b0 2021-02-10T11:40:29.977 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:29.975 7f26abfff700 1 -- 172.21.2.102:0/1985133506 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f26b4009340 con 0x7f26c41a49b0 2021-02-10T11:40:30.107 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:30.105 7f26c935f700 1 -- 172.21.2.102:0/1985133506 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.3", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f26c408a540 con 0x7f26c41a49b0 2021-02-10T11:40:30.110 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:30.108 7f26abfff700 1 -- 172.21.2.102:0/1985133506 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.3", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.3 v121) v1 ==== 184+0+0 (secure 0 0 0) 0x7f26b400a330 con 0x7f26c41a49b0 2021-02-10T11:40:30.114 INFO:teuthology.orchestra.run.gibba002.stderr:updated caps for client.3 2021-02-10T11:40:30.119 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:30.114 7f26c935f700 1 -- 172.21.2.102:0/1985133506 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f26a40186c0 msgr2=0x7f26a401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:30.119 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:30.114 7f26c935f700 1 --2- 172.21.2.102:0/1985133506 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f26a40186c0 0x7f26a401abc0 crc :-1 s=READY pgs=1477 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:30.119 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:30.114 7f26c935f700 1 -- 172.21.2.102:0/1985133506 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f26c41a49b0 msgr2=0x7f26c408aa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:30.119 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:30.114 7f26c935f700 1 --2- 172.21.2.102:0/1985133506 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f26c41a49b0 0x7f26c408aa80 secure :-1 s=READY pgs=3384 cs=0 l=1 rev1=1 rx=0x7f26b4009630 tx=0x7f26b400b980).stop 2021-02-10T11:40:30.119 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:30.114 7f26c935f700 1 -- 172.21.2.102:0/1985133506 shutdown_connections 2021-02-10T11:40:30.119 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:30.114 7f26c935f700 1 --2- 172.21.2.102:0/1985133506 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f26c408bac0 0x7f26c40bf1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:30.119 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:30.114 7f26c935f700 1 --2- 172.21.2.102:0/1985133506 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f26a40186c0 0x7f26a401abc0 unknown :-1 s=CLOSED pgs=1477 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:30.119 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:30.114 7f26c935f700 1 --2- 172.21.2.102:0/1985133506 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f26c408b0a0 0x7f26c408b4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:30.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:30.114 7f26c935f700 1 --2- 172.21.2.102:0/1985133506 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f26c41a49b0 0x7f26c408aa80 unknown :-1 s=CLOSED pgs=3384 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:30.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:30.114 7f26c935f700 1 -- 172.21.2.102:0/1985133506 >> 172.21.2.102:0/1985133506 conn(0x7f26c40bd7a0 msgr2=0x7f26c40b5900 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:30.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:30.114 7f26c935f700 1 -- 172.21.2.102:0/1985133506 shutdown_connections 2021-02-10T11:40:30.120 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:30.114 7f26c935f700 1 -- 172.21.2.102:0/1985133506 wait complete. 2021-02-10T11:40:30.126 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:40:30.556 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:30.536 7f966bc43700 1 Processor -- start 2021-02-10T11:40:30.556 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:30.536 7f966bc43700 1 -- start start 2021-02-10T11:40:30.556 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:30.536 7f966bc43700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f96640d1870 0x7f96641b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:30.556 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:30.536 7f966bc43700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96640f0b10 con 0x7f96641b0a10 2021-02-10T11:40:30.556 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:30.536 7f966bc43700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96640d0310 con 0x7f96640f0650 2021-02-10T11:40:30.556 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:30.536 7f966bc43700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96640d0530 con 0x7f96640d1870 2021-02-10T11:40:30.556 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:30.537 7f966925e700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f96641b0a10 0x7f96640cf330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:33554/0 (socket says 172.21.2.102:33554) 2021-02-10T11:40:30.556 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:30.537 7f966925e700 1 -- 172.21.2.102:0/4230184933 learned_addr learned my addr 172.21.2.102:0/4230184933 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:40:30.556 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:30.537 7f9668a5d700 1 -- 172.21.2.102:0/4230184933 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2927989373 0 0) 0x7f96600010b0 con 0x7f96641b0a10 2021-02-10T11:40:30.556 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:30.537 7f9668a5d700 1 -- 172.21.2.102:0/4230184933 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9650002fe0 con 0x7f96641b0a10 2021-02-10T11:40:30.557 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:30.537 7f9669a5f700 1 --2- 172.21.2.102:0/4230184933 >> v2:172.21.2.113:3300/0 conn(0x7f96640d1870 0x7f96641b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:30.557 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:30.537 7f9668a5d700 1 -- 172.21.2.102:0/4230184933 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2283877236 0 0) 0x7f96600010b0 con 0x7f96641b0a10 2021-02-10T11:40:30.557 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:30.537 7f9668a5d700 1 -- 172.21.2.102:0/4230184933 >> v2:172.21.2.113:3300/0 conn(0x7f96640d1870 msgr2=0x7f96641b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:30.557 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:30.537 7f9668a5d700 1 --2- 172.21.2.102:0/4230184933 >> v2:172.21.2.113:3300/0 conn(0x7f96640d1870 0x7f96641b03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:30.557 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:30.537 7f9668a5d700 1 -- 172.21.2.102:0/4230184933 >> v1:172.21.2.113:6789/0 conn(0x7f96640f0650 legacy=0x7f96640cfb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:30.557 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:30.537 7f9668a5d700 1 -- 172.21.2.102:0/4230184933 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96640d1c70 con 0x7f96641b0a10 2021-02-10T11:40:30.557 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:30.537 7f9668a5d700 1 -- 172.21.2.102:0/4230184933 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7f96600015b0 con 0x7f96641b0a10 2021-02-10T11:40:30.557 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:30.537 7f9668a5d700 1 -- 172.21.2.102:0/4230184933 >> v1:172.21.2.102:6789/0 conn(0x7f96641b0a10 legacy=0x7f96640cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:30.557 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:30.537 7f9668a5d700 1 --2- 172.21.2.102:0/4230184933 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9650003a60 0x7f9650005f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:30.557 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:30.537 7f9668a5d700 1 --2- 172.21.2.102:0/4230184933 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f9650006780 0x7f9650008ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:30.557 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:30.537 7f9668a5d700 1 --2- 172.21.2.102:0/4230184933 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f96500095a0 0x7f965000bac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:30.557 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:30.537 7f9668a5d700 1 -- 172.21.2.102:0/4230184933 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f965000c2f0 con 0x7f9650003a60 2021-02-10T11:40:30.557 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:30.537 7f9668a5d700 1 -- 172.21.2.102:0/4230184933 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f965000c510 con 0x7f9650006780 2021-02-10T11:40:30.558 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:30.537 7f9668a5d700 1 -- 172.21.2.102:0/4230184933 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f965000c730 con 0x7f96500095a0 2021-02-10T11:40:30.558 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:30.538 7f9669a5f700 1 --2- 172.21.2.102:0/4230184933 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9650003a60 0x7f9650005f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:30.558 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:30.538 7f966925e700 1 --2- 172.21.2.102:0/4230184933 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f9650006780 0x7f9650008ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:30.558 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:30.538 7f966a260700 1 --2- 172.21.2.102:0/4230184933 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f96500095a0 0x7f965000bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:30.558 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:30.538 7f9669a5f700 1 -- 172.21.2.102:0/4230184933 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f96500095a0 msgr2=0x7f965000bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:30.558 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:30.538 7f9669a5f700 1 --2- 172.21.2.102:0/4230184933 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f96500095a0 0x7f965000bac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:30.558 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:30.538 7f9669a5f700 1 -- 172.21.2.102:0/4230184933 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f9650006780 msgr2=0x7f9650008ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:30.558 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:30.538 7f9669a5f700 1 --2- 172.21.2.102:0/4230184933 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f9650006780 0x7f9650008ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:30.558 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:30.538 7f9669a5f700 1 -- 172.21.2.102:0/4230184933 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96500509a0 con 0x7f9650003a60 2021-02-10T11:40:30.558 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:30.538 7f966a260700 1 -- 172.21.2.102:0/4230184933 reap_dead start 2021-02-10T11:40:30.558 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:30.538 7f9669a5f700 1 --2- 172.21.2.102:0/4230184933 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9650003a60 0x7f9650005f60 secure :-1 s=READY pgs=3386 cs=0 l=1 rev1=1 rx=0x7f965800c8f0 tx=0x7f965800cc90).ready entity=mon.0 client_cookie=ad24ab50d046b735 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:30.558 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:30.538 7f9668a5d700 1 -- 172.21.2.102:0/4230184933 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9658009310 con 0x7f9650003a60 2021-02-10T11:40:30.559 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:30.538 7f9668a5d700 1 -- 172.21.2.102:0/4230184933 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9658009690 con 0x7f9650003a60 2021-02-10T11:40:30.559 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:30.538 7f9668a5d700 1 -- 172.21.2.102:0/4230184933 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9658009bb0 con 0x7f9650003a60 2021-02-10T11:40:30.559 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:30.538 7f966bc43700 1 -- 172.21.2.102:0/4230184933 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9650003a60 msgr2=0x7f9650005f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:30.559 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:30.538 7f966bc43700 1 --2- 172.21.2.102:0/4230184933 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9650003a60 0x7f9650005f60 secure :-1 s=READY pgs=3386 cs=0 l=1 rev1=1 rx=0x7f965800c8f0 tx=0x7f965800cc90).stop 2021-02-10T11:40:30.559 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:30.539 7f966bc43700 1 -- 172.21.2.102:0/4230184933 shutdown_connections 2021-02-10T11:40:30.559 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:30.539 7f966bc43700 1 --2- 172.21.2.102:0/4230184933 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f9650003a60 0x7f9650005f60 unknown :-1 s=CLOSED pgs=3386 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:30.559 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:30.539 7f966bc43700 1 -- 172.21.2.102:0/4230184933 >> 172.21.2.102:0/4230184933 conn(0x7f96640ce450 msgr2=0x7f96640d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:30.559 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:30.539 7f966bc43700 1 -- 172.21.2.102:0/4230184933 shutdown_connections 2021-02-10T11:40:30.559 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:30.539 7f966bc43700 1 -- 172.21.2.102:0/4230184933 wait complete. 2021-02-10T11:40:30.560 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:30.554 7f966bc43700 1 Processor -- start 2021-02-10T11:40:30.561 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:30.554 7f966bc43700 1 -- start start 2021-02-10T11:40:30.561 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:30.554 7f966bc43700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9664195650 0x7f96640b5560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:30.561 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:30.554 7f966bc43700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f96640b5b80 0x7f96641a2e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:30.561 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:30.554 7f966bc43700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f96640acfa0 0x7f96641a3440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:30.561 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:30.554 7f966bc43700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96640b6010 con 0x7f96640b5b80 2021-02-10T11:40:30.561 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:30.554 7f966bc43700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96641a3a60 con 0x7f96640acfa0 2021-02-10T11:40:30.561 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:30.554 7f966bc43700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96641a3c40 con 0x7f9664195650 2021-02-10T11:40:30.561 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:30.554 7f966925e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f96640b5b80 0x7f96641a2e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:30.561 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:30.554 7f966925e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f96640b5b80 0x7f96641a2e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:41346/0 (socket says 172.21.2.102:41346) 2021-02-10T11:40:30.561 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:30.554 7f966925e700 1 -- 172.21.2.102:0/222877245 learned_addr learned my addr 172.21.2.102:0/222877245 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:40:30.561 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:30.555 7f966925e700 1 -- 172.21.2.102:0/222877245 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9664195650 msgr2=0x7f96640b5560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:30.561 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:30.555 7f966925e700 1 --2- 172.21.2.102:0/222877245 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9664195650 0x7f96640b5560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:30.562 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:30.555 7f966925e700 1 -- 172.21.2.102:0/222877245 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f96640acfa0 msgr2=0x7f96641a3440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:30.562 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:30.555 7f966925e700 1 --2- 172.21.2.102:0/222877245 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f96640acfa0 0x7f96641a3440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:30.562 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:30.555 7f966925e700 1 -- 172.21.2.102:0/222877245 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96640bcae0 con 0x7f96640b5b80 2021-02-10T11:40:30.562 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:30.555 7f966925e700 1 --2- 172.21.2.102:0/222877245 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f96640b5b80 0x7f96641a2e20 secure :-1 s=READY pgs=3387 cs=0 l=1 rev1=1 rx=0x7f9660001c00 tx=0x7f966000a8e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:30.562 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:30.555 7f96567fc700 1 -- 172.21.2.102:0/222877245 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f96600009b0 con 0x7f96640b5b80 2021-02-10T11:40:30.562 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:30.555 7f96567fc700 1 -- 172.21.2.102:0/222877245 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f966000acd0 con 0x7f96640b5b80 2021-02-10T11:40:30.562 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:30.555 7f96567fc700 1 -- 172.21.2.102:0/222877245 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9660004540 con 0x7f96640b5b80 2021-02-10T11:40:30.562 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:30.555 7f966bc43700 1 -- 172.21.2.102:0/222877245 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96640bcae0 con 0x7f96640b5b80 2021-02-10T11:40:30.562 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:30.555 7f966bc43700 1 -- 172.21.2.102:0/222877245 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96640bcae0 con 0x7f96640b5b80 2021-02-10T11:40:30.562 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:30.556 7f96567fc700 1 -- 172.21.2.102:0/222877245 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f9660004350 con 0x7f96640b5b80 2021-02-10T11:40:30.562 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:30.556 7f96567fc700 1 --2- 172.21.2.102:0/222877245 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f964c0186c0 0x7f964c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:30.562 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:30.556 7f96567fc700 1 -- 172.21.2.102:0/222877245 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f9660032510 con 0x7f96640b5b80 2021-02-10T11:40:30.562 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:30.556 7f9669a5f700 1 --2- 172.21.2.102:0/222877245 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f964c0186c0 0x7f964c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:30.563 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:30.557 7f9669a5f700 1 --2- 172.21.2.102:0/222877245 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f964c0186c0 0x7f964c01abc0 crc :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:30.589 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:30.586 7f966bc43700 1 -- 172.21.2.102:0/222877245 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96640b6010 con 0x7f96640b5b80 2021-02-10T11:40:30.591 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:30.589 7f96567fc700 1 -- 172.21.2.102:0/222877245 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f966001d9c0 con 0x7f96640b5b80 2021-02-10T11:40:30.721 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:30.719 7f966bc43700 1 -- 172.21.2.102:0/222877245 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f96640b6010 con 0x7f96640b5b80 2021-02-10T11:40:30.722 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:30.719 7f96567fc700 1 -- 172.21.2.102:0/222877245 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 173 v173) v1 ==== 95+0+2108 (secure 0 0 0) 0x7f966001d9c0 con 0x7f96640b5b80 2021-02-10T11:40:30.722 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:40:30.722 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":173,"default_fscid":21,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":13872,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.102:6835/1871410080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6834","nonce":1871410080},{"type":"v1","addr":"172.21.2.102:6835","nonce":1871410080}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":169}],"filesystems":[{"mdsmap":{"epoch":173,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:40:24.365024","modified":"2021-02-10 11:40:26.372300","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7598},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7598":{"gid":7598,"name":"b","rank":0,"incarnation":170,"state":"up:active","state_seq":3,"addr":"172.21.2.113:6833/2247818770","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6832","nonce":2247818770},{"type":"v1","addr":"172.21.2.113:6833","nonce":2247818770}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[43],"metadata_pool":42,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":21}]} 2021-02-10T11:40:30.722 INFO:teuthology.orchestra.run.gibba002.stderr:dumped fsmap epoch 173 2021-02-10T11:40:30.723 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:30.721 7f966bc43700 1 -- 172.21.2.102:0/222877245 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f964c0186c0 msgr2=0x7f964c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:30.724 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:30.721 7f966bc43700 1 --2- 172.21.2.102:0/222877245 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f964c0186c0 0x7f964c01abc0 crc :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:30.724 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:30.721 7f966bc43700 1 -- 172.21.2.102:0/222877245 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f96640b5b80 msgr2=0x7f96641a2e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:30.724 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:30.721 7f966bc43700 1 --2- 172.21.2.102:0/222877245 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f96640b5b80 0x7f96641a2e20 secure :-1 s=READY pgs=3387 cs=0 l=1 rev1=1 rx=0x7f9660001c00 tx=0x7f966000a8e0).stop 2021-02-10T11:40:30.724 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:30.722 7f966bc43700 1 -- 172.21.2.102:0/222877245 shutdown_connections 2021-02-10T11:40:30.724 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:30.722 7f966bc43700 1 --2- 172.21.2.102:0/222877245 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f964c0186c0 0x7f964c01abc0 unknown :-1 s=CLOSED pgs=1478 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:30.724 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:30.722 7f966bc43700 1 --2- 172.21.2.102:0/222877245 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f96640acfa0 0x7f96641a3440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:30.725 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:30.722 7f966bc43700 1 --2- 172.21.2.102:0/222877245 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f96640b5b80 0x7f96641a2e20 unknown :-1 s=CLOSED pgs=3387 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:30.725 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:30.722 7f966bc43700 1 --2- 172.21.2.102:0/222877245 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f9664195650 0x7f96640b5560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:30.725 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:30.722 7f966bc43700 1 -- 172.21.2.102:0/222877245 >> 172.21.2.102:0/222877245 conn(0x7f96640b3040 msgr2=0x7f96640c3250 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:30.725 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:30.722 7f966bc43700 1 -- 172.21.2.102:0/222877245 shutdown_connections 2021-02-10T11:40:30.725 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:30.722 7f966bc43700 1 -- 172.21.2.102:0/222877245 wait complete. 2021-02-10T11:40:30.934 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 173, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:40:24.365024', 'modified': '2021-02-10 11:40:26.372300', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7598}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7598': {'gid': 7598, 'name': 'b', 'rank': 0, 'incarnation': 170, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.113:6833/2247818770', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.113:6832', 'nonce': 2247818770}, {'type': 'v1', 'addr': '172.21.2.113:6833', 'nonce': 2247818770}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [43], 'metadata_pool': 42, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:40:30.934 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:40:30.934 DEBUG:teuthology.orchestra.run.gibba013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:40:31.333 INFO:teuthology.orchestra.run.gibba013.stdout:{ 2021-02-10T11:40:31.333 INFO:teuthology.orchestra.run.gibba013.stdout: "cluster_fsid": "68db30ea-dbce-4aef-a6df-f01c6fea0513", 2021-02-10T11:40:31.333 INFO:teuthology.orchestra.run.gibba013.stdout: "whoami": 0, 2021-02-10T11:40:31.333 INFO:teuthology.orchestra.run.gibba013.stdout: "id": 7598, 2021-02-10T11:40:31.333 INFO:teuthology.orchestra.run.gibba013.stdout: "want_state": "up:active", 2021-02-10T11:40:31.333 INFO:teuthology.orchestra.run.gibba013.stdout: "state": "up:active", 2021-02-10T11:40:31.333 INFO:teuthology.orchestra.run.gibba013.stdout: "rank_uptime": 6.9639773299999996, 2021-02-10T11:40:31.333 INFO:teuthology.orchestra.run.gibba013.stdout: "mdsmap_epoch": 173, 2021-02-10T11:40:31.334 INFO:teuthology.orchestra.run.gibba013.stdout: "osdmap_epoch": 197, 2021-02-10T11:40:31.334 INFO:teuthology.orchestra.run.gibba013.stdout: "osdmap_epoch_barrier": 197, 2021-02-10T11:40:31.334 INFO:teuthology.orchestra.run.gibba013.stdout: "uptime": 12.617495124 2021-02-10T11:40:31.334 INFO:teuthology.orchestra.run.gibba013.stdout:} 2021-02-10T11:40:31.337 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "68db30ea-dbce-4aef-a6df-f01c6fea0513", "whoami": 0, "id": 7598, "want_state": "up:active", "state": "up:active", "rank_uptime": 6.9639773299999996, "mdsmap_epoch": 173, "osdmap_epoch": 197, "osdmap_epoch_barrier": 197, "uptime": 12.617495124 } 2021-02-10T11:40:31.337 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:40:31.768 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:31.749 7f434741a700 1 Processor -- start 2021-02-10T11:40:31.768 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:31.749 7f434741a700 1 -- start start 2021-02-10T11:40:31.768 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:31.749 7f434741a700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f43400d0340 0x7f43400ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:31.768 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:31.749 7f434741a700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f43400d1c70 0x7f43400efc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:31.768 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:31.749 7f434741a700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43400f0b10 con 0x7f43400d1870 2021-02-10T11:40:31.768 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:31.749 7f434741a700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43400e5ed0 con 0x7f43400d1c70 2021-02-10T11:40:31.768 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:31.749 7f434741a700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43401b0be0 con 0x7f43400d0340 2021-02-10T11:40:31.768 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:31.749 7f4345a37700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f43400d1c70 0x7f43400efc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:31.769 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:31.749 7f4345236700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f43400d1870 0x7f43401b03f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:33574/0 (socket says 172.21.2.102:33574) 2021-02-10T11:40:31.769 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:31.749 7f4345236700 1 -- 172.21.2.102:0/2000606548 learned_addr learned my addr 172.21.2.102:0/2000606548 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:40:31.769 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:31.750 7f4345a37700 1 -- 172.21.2.102:0/2000606548 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f43400d0340 msgr2=0x7f43400ef640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:31.769 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:31.750 7f4345a37700 1 --2- 172.21.2.102:0/2000606548 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f43400d0340 0x7f43400ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:31.769 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:31.750 7f4345a37700 1 -- 172.21.2.102:0/2000606548 >> v1:172.21.2.102:6789/0 conn(0x7f43400d1870 legacy=0x7f43401b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:31.769 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:31.750 7f4337fff700 1 -- 172.21.2.102:0/2000606548 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 434880848 0 0) 0x7f433c001180 con 0x7f43400d1870 2021-02-10T11:40:31.769 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:31.750 7f4345a37700 1 -- 172.21.2.102:0/2000606548 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43400f20a0 con 0x7f43400d1c70 2021-02-10T11:40:31.769 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:31.750 7f4345a37700 1 --2- 172.21.2.102:0/2000606548 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f43400d1c70 0x7f43400efc60 secure :-1 s=READY pgs=3389 cs=0 l=1 rev1=1 rx=0x7f4338006e30 tx=0x7f4338009de0).ready entity=mon.0 client_cookie=2fcf19fbcc2a2b2f server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:31.769 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:31.751 7f4337fff700 1 -- 172.21.2.102:0/2000606548 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f433800b300 con 0x7f43400d1c70 2021-02-10T11:40:31.769 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:31.751 7f4337fff700 1 -- 172.21.2.102:0/2000606548 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f433800b680 con 0x7f43400d1c70 2021-02-10T11:40:31.769 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:31.751 7f4337fff700 1 -- 172.21.2.102:0/2000606548 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f433800bba0 con 0x7f43400d1c70 2021-02-10T11:40:31.769 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:31.751 7f434741a700 1 -- 172.21.2.102:0/2000606548 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f43400d1c70 msgr2=0x7f43400efc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:31.769 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:31.751 7f434741a700 1 --2- 172.21.2.102:0/2000606548 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f43400d1c70 0x7f43400efc60 secure :-1 s=READY pgs=3389 cs=0 l=1 rev1=1 rx=0x7f4338006e30 tx=0x7f4338009de0).stop 2021-02-10T11:40:31.770 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:31.751 7f434741a700 1 -- 172.21.2.102:0/2000606548 shutdown_connections 2021-02-10T11:40:31.770 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:31.751 7f434741a700 1 --2- 172.21.2.102:0/2000606548 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f43400d1c70 0x7f43400efc60 unknown :-1 s=CLOSED pgs=3389 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:31.770 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:31.751 7f434741a700 1 --2- 172.21.2.102:0/2000606548 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f43400d0340 0x7f43400ef640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:31.770 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:31.751 7f434741a700 1 -- 172.21.2.102:0/2000606548 >> 172.21.2.102:0/2000606548 conn(0x7f43400ce450 msgr2=0x7f43400d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:31.770 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:31.752 7f434741a700 1 -- 172.21.2.102:0/2000606548 shutdown_connections 2021-02-10T11:40:31.770 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:31.752 7f434741a700 1 -- 172.21.2.102:0/2000606548 wait complete. 2021-02-10T11:40:31.772 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:31.766 7f434741a700 1 Processor -- start 2021-02-10T11:40:31.772 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:31.766 7f434741a700 1 -- start start 2021-02-10T11:40:31.772 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:31.767 7f434741a700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f43401952e0 0x7f434018ed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:31.772 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:31.767 7f434741a700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f43401b8bc0 0x7f43401ac4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:31.772 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:31.767 7f434741a700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f43401acae0 0x7f4340194240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:31.772 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:31.767 7f434741a700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43401b8640 con 0x7f43401acae0 2021-02-10T11:40:31.772 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:31.767 7f434741a700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f434019e8b0 con 0x7f43401b8bc0 2021-02-10T11:40:31.772 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:31.767 7f434741a700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f434019ea90 con 0x7f43401952e0 2021-02-10T11:40:31.772 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:31.767 7f4345a37700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f43401acae0 0x7f4340194240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:31.772 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:31.767 7f4345a37700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f43401acae0 0x7f4340194240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:41358/0 (socket says 172.21.2.102:41358) 2021-02-10T11:40:31.773 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:31.767 7f4345a37700 1 -- 172.21.2.102:0/3813549929 learned_addr learned my addr 172.21.2.102:0/3813549929 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:40:31.773 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:31.767 7f4345236700 1 --2- 172.21.2.102:0/3813549929 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f43401952e0 0x7f434018ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:31.773 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:31.767 7f4344a35700 1 --2- 172.21.2.102:0/3813549929 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f43401b8bc0 0x7f43401ac4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:31.773 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:31.767 7f4345a37700 1 -- 172.21.2.102:0/3813549929 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f43401952e0 msgr2=0x7f434018ed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:31.773 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:31.767 7f4345a37700 1 --2- 172.21.2.102:0/3813549929 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f43401952e0 0x7f434018ed80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:31.773 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:31.767 7f4345a37700 1 -- 172.21.2.102:0/3813549929 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f43401b8bc0 msgr2=0x7f43401ac4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:31.773 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:31.767 7f4345a37700 1 --2- 172.21.2.102:0/3813549929 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f43401b8bc0 0x7f43401ac4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:31.773 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:31.767 7f4345a37700 1 -- 172.21.2.102:0/3813549929 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43401acee0 con 0x7f43401acae0 2021-02-10T11:40:31.773 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:31.767 7f4344a35700 1 --2- 172.21.2.102:0/3813549929 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f43401b8bc0 0x7f43401ac4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:40:31.773 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:31.767 7f4345236700 1 --2- 172.21.2.102:0/3813549929 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f43401952e0 0x7f434018ed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:40:31.773 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:31.768 7f4345a37700 1 --2- 172.21.2.102:0/3813549929 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f43401acae0 0x7f4340194240 secure :-1 s=READY pgs=3390 cs=0 l=1 rev1=1 rx=0x7f433800b970 tx=0x7f433800bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:31.773 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:31.768 7f4335ffb700 1 -- 172.21.2.102:0/3813549929 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4338009380 con 0x7f43401acae0 2021-02-10T11:40:31.773 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:31.768 7f4335ffb700 1 -- 172.21.2.102:0/3813549929 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4338001270 con 0x7f43401acae0 2021-02-10T11:40:31.773 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:31.768 7f4335ffb700 1 -- 172.21.2.102:0/3813549929 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4338001790 con 0x7f43401acae0 2021-02-10T11:40:31.774 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:31.768 7f434741a700 1 -- 172.21.2.102:0/3813549929 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43401acee0 con 0x7f43401acae0 2021-02-10T11:40:31.774 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:31.768 7f434741a700 1 -- 172.21.2.102:0/3813549929 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43401acee0 con 0x7f43401acae0 2021-02-10T11:40:31.774 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:31.769 7f4335ffb700 1 -- 172.21.2.102:0/3813549929 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4338001450 con 0x7f43401acae0 2021-02-10T11:40:31.774 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:31.769 7f4335ffb700 1 --2- 172.21.2.102:0/3813549929 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f43200186c0 0x7f432001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:31.774 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:31.769 7f4335ffb700 1 -- 172.21.2.102:0/3813549929 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f4338032b60 con 0x7f43401acae0 2021-02-10T11:40:31.774 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:31.769 7f4345236700 1 --2- 172.21.2.102:0/3813549929 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f43200186c0 0x7f432001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:31.774 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:31.769 7f4345236700 1 --2- 172.21.2.102:0/3813549929 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f43200186c0 0x7f432001abc0 crc :-1 s=READY pgs=1479 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:31.802 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:31.799 7f434741a700 1 -- 172.21.2.102:0/3813549929 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43400ebc60 con 0x7f43401acae0 2021-02-10T11:40:31.804 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:31.802 7f4335ffb700 1 -- 172.21.2.102:0/3813549929 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f4338003070 con 0x7f43401acae0 2021-02-10T11:40:31.929 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:31.927 7f434741a700 1 -- 172.21.2.102:0/3813549929 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f43400f0b10 con 0x7f43401acae0 2021-02-10T11:40:31.930 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:31.928 7f4335ffb700 1 -- 172.21.2.102:0/3813549929 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v197) v1 ==== 74+0+14007 (secure 0 0 0) 0x7f433801f880 con 0x7f43401acae0 2021-02-10T11:40:31.931 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:40:31.931 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":197,"fsid":"68db30ea-dbce-4aef-a6df-f01c6fea0513","created":"2021-02-10 11:03:06.331557","modified":"2021-02-10 11:40:25.367346","last_up_change":"2021-02-10 11:03:32.690562","last_in_change":"2021-02-10 11:03:32.690562","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":43,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:03:37.322519","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":42,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:40:22.330567","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"196","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":43,"pool_name":"cephfs_data","create_time":"2021-02-10 11:40:23.357509","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"197","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"be691b4d-a039-408a-af80-9d53f02b6f1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":185,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6818","nonce":36088},{"type":"v1","addr":"172.21.2.102:6820","nonce":36088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6822","nonce":36088},{"type":"v1","addr":"172.21.2.102:6824","nonce":36088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6830","nonce":36088},{"type":"v1","addr":"172.21.2.102:6832","nonce":36088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6826","nonce":36088},{"type":"v1","addr":"172.21.2.102:6828","nonce":36088}]},"public_addr":"172.21.2.102:6820/36088","cluster_addr":"172.21.2.102:6824/36088","heartbeat_back_addr":"172.21.2.102:6832/36088","heartbeat_front_addr":"172.21.2.102:6828/36088","state":["exists","up"]},{"osd":1,"uuid":"64e3ccc8-9c69-4d7c-865e-af8e83822635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6819","nonce":36087},{"type":"v1","addr":"172.21.2.102:6821","nonce":36087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6823","nonce":36087},{"type":"v1","addr":"172.21.2.102:6825","nonce":36087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6831","nonce":36087},{"type":"v1","addr":"172.21.2.102:6833","nonce":36087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6827","nonce":36087},{"type":"v1","addr":"172.21.2.102:6829","nonce":36087}]},"public_addr":"172.21.2.102:6821/36087","cluster_addr":"172.21.2.102:6825/36087","heartbeat_back_addr":"172.21.2.102:6833/36087","heartbeat_front_addr":"172.21.2.102:6829/36087","state":["exists","up"]},{"osd":2,"uuid":"153700af-747f-481f-bdf4-d6072d3a842f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6810","nonce":36089},{"type":"v1","addr":"172.21.2.102:6811","nonce":36089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6812","nonce":36089},{"type":"v1","addr":"172.21.2.102:6813","nonce":36089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6816","nonce":36089},{"type":"v1","addr":"172.21.2.102:6817","nonce":36089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6814","nonce":36089},{"type":"v1","addr":"172.21.2.102:6815","nonce":36089}]},"public_addr":"172.21.2.102:6811/36089","cluster_addr":"172.21.2.102:6813/36089","heartbeat_back_addr":"172.21.2.102:6817/36089","heartbeat_front_addr":"172.21.2.102:6815/36089","state":["exists","up"]},{"osd":3,"uuid":"cbc678f4-83a9-45e6-8830-2f7c232b1ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6802","nonce":36090},{"type":"v1","addr":"172.21.2.102:6803","nonce":36090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6804","nonce":36090},{"type":"v1","addr":"172.21.2.102:6805","nonce":36090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6808","nonce":36090},{"type":"v1","addr":"172.21.2.102:6809","nonce":36090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6806","nonce":36090},{"type":"v1","addr":"172.21.2.102:6807","nonce":36090}]},"public_addr":"172.21.2.102:6803/36090","cluster_addr":"172.21.2.102:6805/36090","heartbeat_back_addr":"172.21.2.102:6809/36090","heartbeat_front_addr":"172.21.2.102:6807/36090","state":["exists","up"]},{"osd":4,"uuid":"c01e4008-b258-4f10-afd2-9e7f37214351","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6808","nonce":36368},{"type":"v1","addr":"172.21.2.113:6811","nonce":36368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6814","nonce":36368},{"type":"v1","addr":"172.21.2.113:6818","nonce":36368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6826","nonce":36368},{"type":"v1","addr":"172.21.2.113:6828","nonce":36368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6820","nonce":36368},{"type":"v1","addr":"172.21.2.113:6824","nonce":36368}]},"public_addr":"172.21.2.113:6811/36368","cluster_addr":"172.21.2.113:6818/36368","heartbeat_back_addr":"172.21.2.113:6828/36368","heartbeat_front_addr":"172.21.2.113:6824/36368","state":["exists","up"]},{"osd":5,"uuid":"551f6007-daf8-4a98-8ee0-c2b16bcccb5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6802","nonce":36366},{"type":"v1","addr":"172.21.2.113:6805","nonce":36366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6807","nonce":36366},{"type":"v1","addr":"172.21.2.113:6810","nonce":36366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6819","nonce":36366},{"type":"v1","addr":"172.21.2.113:6822","nonce":36366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6812","nonce":36366},{"type":"v1","addr":"172.21.2.113:6815","nonce":36366}]},"public_addr":"172.21.2.113:6805/36366","cluster_addr":"172.21.2.113:6810/36366","heartbeat_back_addr":"172.21.2.113:6822/36366","heartbeat_front_addr":"172.21.2.113:6815/36366","state":["exists","up"]},{"osd":6,"uuid":"bf67c040-0e50-48be-b102-c3943bc9ebef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6800","nonce":36367},{"type":"v1","addr":"172.21.2.113:6801","nonce":36367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6803","nonce":36367},{"type":"v1","addr":"172.21.2.113:6804","nonce":36367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6813","nonce":36367},{"type":"v1","addr":"172.21.2.113:6816","nonce":36367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6806","nonce":36367},{"type":"v1","addr":"172.21.2.113:6809","nonce":36367}]},"public_addr":"172.21.2.113:6801/36367","cluster_addr":"172.21.2.113:6804/36367","heartbeat_back_addr":"172.21.2.113:6816/36367","heartbeat_front_addr":"172.21.2.113:6809/36367","state":["exists","up"]},{"osd":7,"uuid":"f5461f80-018d-428f-be89-2eb2bcbe7bb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6817","nonce":36365},{"type":"v1","addr":"172.21.2.113:6821","nonce":36365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6823","nonce":36365},{"type":"v1","addr":"172.21.2.113:6825","nonce":36365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6830","nonce":36365},{"type":"v1","addr":"172.21.2.113:6831","nonce":36365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6827","nonce":36365},{"type":"v1","addr":"172.21.2.113:6829","nonce":36365}]},"public_addr":"172.21.2.113:6821/36365","cluster_addr":"172.21.2.113:6825/36365","heartbeat_back_addr":"172.21.2.113:6831/36365","heartbeat_front_addr":"172.21.2.113:6829/36365","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:40:31.932 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:31.929 7f434741a700 1 -- 172.21.2.102:0/3813549929 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f43200186c0 msgr2=0x7f432001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:31.932 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:31.929 7f434741a700 1 --2- 172.21.2.102:0/3813549929 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f43200186c0 0x7f432001abc0 crc :-1 s=READY pgs=1479 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:31.932 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:31.929 7f434741a700 1 -- 172.21.2.102:0/3813549929 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f43401acae0 msgr2=0x7f4340194240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:31.932 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:31.929 7f434741a700 1 --2- 172.21.2.102:0/3813549929 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f43401acae0 0x7f4340194240 secure :-1 s=READY pgs=3390 cs=0 l=1 rev1=1 rx=0x7f433800b970 tx=0x7f433800bd10).stop 2021-02-10T11:40:31.932 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:31.930 7f434741a700 1 -- 172.21.2.102:0/3813549929 shutdown_connections 2021-02-10T11:40:31.932 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:31.930 7f434741a700 1 --2- 172.21.2.102:0/3813549929 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f43401acae0 0x7f4340194240 unknown :-1 s=CLOSED pgs=3390 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:31.933 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:31.930 7f434741a700 1 --2- 172.21.2.102:0/3813549929 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f43200186c0 0x7f432001abc0 unknown :-1 s=CLOSED pgs=1479 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:31.933 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:31.930 7f434741a700 1 --2- 172.21.2.102:0/3813549929 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f43401b8bc0 0x7f43401ac4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:31.933 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:31.930 7f434741a700 1 --2- 172.21.2.102:0/3813549929 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f43401952e0 0x7f434018ed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:31.933 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:31.930 7f434741a700 1 -- 172.21.2.102:0/3813549929 >> 172.21.2.102:0/3813549929 conn(0x7f43401ac040 msgr2=0x7f4340094980 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:31.933 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:31.930 7f434741a700 1 -- 172.21.2.102:0/3813549929 shutdown_connections 2021-02-10T11:40:31.933 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:31.930 7f434741a700 1 -- 172.21.2.102:0/3813549929 wait complete. 2021-02-10T11:40:31.940 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:03:37.322519', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 42, 'pool_name': 'cephfs_metadata', 'create_time': '2021-02-10 11:40:22.330567', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '196', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 43, 'pool_name': 'cephfs_data', 'create_time': '2021-02-10 11:40:23.357509', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '197', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-02-10T11:40:31.941 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:40:32.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:32.357 7fcd70444700 1 Processor -- start 2021-02-10T11:40:32.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:32.357 7fcd70444700 1 -- start start 2021-02-10T11:40:32.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:32.357 7fcd70444700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fcd680d1870 0x7fcd681b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:32.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:32.357 7fcd70444700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcd681b0ae0 0x7fcd68108940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:32.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:32.357 7fcd70444700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd680fe1c0 con 0x7fcd680d1c70 2021-02-10T11:40:32.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:32.357 7fcd70444700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd68109a60 con 0x7fcd680d1870 2021-02-10T11:40:32.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:32.357 7fcd70444700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd68109750 con 0x7fcd681b0ae0 2021-02-10T11:40:32.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:32.357 7fcd6e260700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fcd680d1870 0x7fcd681b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:32.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:32.357 7fcd6ea61700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fcd680d1c70 0x7fcd68108f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:33590/0 (socket says 172.21.2.102:33590) 2021-02-10T11:40:32.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:32.357 7fcd6ea61700 1 -- 172.21.2.102:0/2524048699 learned_addr learned my addr 172.21.2.102:0/2524048699 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:40:32.377 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:32.358 7fcd6e260700 1 -- 172.21.2.102:0/2524048699 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcd681b0ae0 msgr2=0x7fcd68108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:32.377 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:32.358 7fcd6e260700 1 --2- 172.21.2.102:0/2524048699 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcd681b0ae0 0x7fcd68108940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:32.377 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:32.358 7fcd6e260700 1 -- 172.21.2.102:0/2524048699 >> v1:172.21.2.102:6789/0 conn(0x7fcd680d1c70 legacy=0x7fcd68108f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:32.377 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:32.358 7fcd6e260700 1 -- 172.21.2.102:0/2524048699 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd68088070 con 0x7fcd680d1870 2021-02-10T11:40:32.377 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:32.358 7fcd6e260700 1 --2- 172.21.2.102:0/2524048699 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcd680d1870 0x7fcd681b04c0 secure :-1 s=READY pgs=3392 cs=0 l=1 rev1=1 rx=0x7fcd64006650 tx=0x7fcd64009a80).ready entity=mon.0 client_cookie=3f36ee3f61cc20ff server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:32.377 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:32.358 7fcd6d25e700 1 -- 172.21.2.102:0/2524048699 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcd6400c0c0 con 0x7fcd680d1870 2021-02-10T11:40:32.377 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:32.359 7fcd6d25e700 1 -- 172.21.2.102:0/2524048699 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd6400c340 con 0x7fcd680d1870 2021-02-10T11:40:32.377 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:32.359 7fcd6d25e700 1 -- 172.21.2.102:0/2524048699 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcd6400c860 con 0x7fcd680d1870 2021-02-10T11:40:32.377 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:32.359 7fcd70444700 1 -- 172.21.2.102:0/2524048699 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcd680d1870 msgr2=0x7fcd681b04c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:32.377 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:32.359 7fcd70444700 1 --2- 172.21.2.102:0/2524048699 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcd680d1870 0x7fcd681b04c0 secure :-1 s=READY pgs=3392 cs=0 l=1 rev1=1 rx=0x7fcd64006650 tx=0x7fcd64009a80).stop 2021-02-10T11:40:32.377 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:32.359 7fcd70444700 1 -- 172.21.2.102:0/2524048699 shutdown_connections 2021-02-10T11:40:32.377 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:32.359 7fcd70444700 1 --2- 172.21.2.102:0/2524048699 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcd681b0ae0 0x7fcd68108940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:32.377 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:32.359 7fcd70444700 1 --2- 172.21.2.102:0/2524048699 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcd680d1870 0x7fcd681b04c0 unknown :-1 s=CLOSED pgs=3392 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:32.378 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:32.359 7fcd70444700 1 -- 172.21.2.102:0/2524048699 >> 172.21.2.102:0/2524048699 conn(0x7fcd680ce420 msgr2=0x7fcd680d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:32.378 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:32.359 7fcd70444700 1 -- 172.21.2.102:0/2524048699 shutdown_connections 2021-02-10T11:40:32.378 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:32.359 7fcd70444700 1 -- 172.21.2.102:0/2524048699 wait complete. 2021-02-10T11:40:32.379 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:32.374 7fcd70444700 1 Processor -- start 2021-02-10T11:40:32.379 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:32.374 7fcd70444700 1 -- start start 2021-02-10T11:40:32.379 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:32.374 7fcd70444700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcd680c3820 0x7fcd68084cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:32.380 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:32.374 7fcd70444700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcd680a0290 0x7fcd680852d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:32.380 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:32.374 7fcd70444700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fcd68086270 0x7fcd68086670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:32.380 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:32.374 7fcd70444700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd680c0700 con 0x7fcd680a0290 2021-02-10T11:40:32.380 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:32.374 7fcd70444700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd680858f0 con 0x7fcd68086270 2021-02-10T11:40:32.380 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:32.374 7fcd70444700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd68085b10 con 0x7fcd680c3820 2021-02-10T11:40:32.380 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:32.374 7fcd6da5f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcd680a0290 0x7fcd680852d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:32.380 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:32.375 7fcd6da5f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcd680a0290 0x7fcd680852d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:41368/0 (socket says 172.21.2.102:41368) 2021-02-10T11:40:32.380 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:32.375 7fcd6da5f700 1 -- 172.21.2.102:0/4150310224 learned_addr learned my addr 172.21.2.102:0/4150310224 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:40:32.380 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:32.375 7fcd6da5f700 1 -- 172.21.2.102:0/4150310224 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcd680c3820 msgr2=0x7fcd68084cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:32.380 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:32.375 7fcd6ea61700 1 --2- 172.21.2.102:0/4150310224 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fcd68086270 0x7fcd68086670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:32.380 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:32.375 7fcd6da5f700 1 --2- 172.21.2.102:0/4150310224 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcd680c3820 0x7fcd68084cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:32.380 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:32.375 7fcd6da5f700 1 -- 172.21.2.102:0/4150310224 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fcd68086270 msgr2=0x7fcd68086670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:32.381 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:32.375 7fcd6da5f700 1 --2- 172.21.2.102:0/4150310224 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fcd68086270 0x7fcd68086670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:32.381 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:32.375 7fcd6da5f700 1 -- 172.21.2.102:0/4150310224 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd68086c90 con 0x7fcd680a0290 2021-02-10T11:40:32.381 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:32.375 7fcd6da5f700 1 --2- 172.21.2.102:0/4150310224 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcd680a0290 0x7fcd680852d0 secure :-1 s=READY pgs=3393 cs=0 l=1 rev1=1 rx=0x7fcd5800b530 tx=0x7fcd5800b8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:32.381 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:32.375 7fcd5effd700 1 -- 172.21.2.102:0/4150310224 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcd5800bcc0 con 0x7fcd680a0290 2021-02-10T11:40:32.381 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:32.375 7fcd5effd700 1 -- 172.21.2.102:0/4150310224 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd58000c90 con 0x7fcd680a0290 2021-02-10T11:40:32.381 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:32.375 7fcd5effd700 1 -- 172.21.2.102:0/4150310224 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcd580022a0 con 0x7fcd680a0290 2021-02-10T11:40:32.381 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:32.375 7fcd70444700 1 -- 172.21.2.102:0/4150310224 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd68086c90 con 0x7fcd680a0290 2021-02-10T11:40:32.381 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:32.375 7fcd70444700 1 -- 172.21.2.102:0/4150310224 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd681a2f00 con 0x7fcd680a0290 2021-02-10T11:40:32.381 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:32.376 7fcd5effd700 1 -- 172.21.2.102:0/4150310224 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fcd58005310 con 0x7fcd680a0290 2021-02-10T11:40:32.381 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:32.376 7fcd5effd700 1 --2- 172.21.2.102:0/4150310224 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcd500186c0 0x7fcd5001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:32.382 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:32.376 7fcd5effd700 1 -- 172.21.2.102:0/4150310224 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fcd58032580 con 0x7fcd680a0290 2021-02-10T11:40:32.382 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:32.376 7fcd6e260700 1 --2- 172.21.2.102:0/4150310224 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcd500186c0 0x7fcd5001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:32.382 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:32.377 7fcd6e260700 1 --2- 172.21.2.102:0/4150310224 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcd500186c0 0x7fcd5001abc0 crc :-1 s=READY pgs=1480 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:32.409 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:32.407 7fcd70444700 1 -- 172.21.2.102:0/4150310224 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd68082a70 con 0x7fcd680a0290 2021-02-10T11:40:32.412 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:32.410 7fcd5effd700 1 -- 172.21.2.102:0/4150310224 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fcd58000960 con 0x7fcd680a0290 2021-02-10T11:40:32.542 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:32.539 7fcd70444700 1 -- 172.21.2.102:0/4150310224 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7fcd68082a70 con 0x7fcd680a0290 2021-02-10T11:40:32.542 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:32.540 7fcd5effd700 1 -- 172.21.2.102:0/4150310224 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v197) v1 ==== 92+0+10 (secure 0 0 0) 0x7fcd58005310 con 0x7fcd680a0290 2021-02-10T11:40:32.543 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:40:32.544 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:32.542 7fcd70444700 1 -- 172.21.2.102:0/4150310224 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcd500186c0 msgr2=0x7fcd5001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:32.544 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:32.542 7fcd70444700 1 --2- 172.21.2.102:0/4150310224 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcd500186c0 0x7fcd5001abc0 crc :-1 s=READY pgs=1480 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:32.544 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:32.542 7fcd70444700 1 -- 172.21.2.102:0/4150310224 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcd680a0290 msgr2=0x7fcd680852d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:32.544 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:32.542 7fcd70444700 1 --2- 172.21.2.102:0/4150310224 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcd680a0290 0x7fcd680852d0 secure :-1 s=READY pgs=3393 cs=0 l=1 rev1=1 rx=0x7fcd5800b530 tx=0x7fcd5800b8d0).stop 2021-02-10T11:40:32.544 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:32.542 7fcd70444700 1 -- 172.21.2.102:0/4150310224 shutdown_connections 2021-02-10T11:40:32.545 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:32.542 7fcd70444700 1 --2- 172.21.2.102:0/4150310224 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcd500186c0 0x7fcd5001abc0 unknown :-1 s=CLOSED pgs=1480 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:32.545 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:32.542 7fcd70444700 1 --2- 172.21.2.102:0/4150310224 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fcd68086270 0x7fcd68086670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:32.545 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:32.542 7fcd70444700 1 --2- 172.21.2.102:0/4150310224 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcd680a0290 0x7fcd680852d0 unknown :-1 s=CLOSED pgs=3393 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:32.545 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:32.542 7fcd70444700 1 --2- 172.21.2.102:0/4150310224 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcd680c3820 0x7fcd68084cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:32.545 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:32.542 7fcd70444700 1 -- 172.21.2.102:0/4150310224 >> 172.21.2.102:0/4150310224 conn(0x7fcd680872c0 msgr2=0x7fcd68095700 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:32.545 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:32.542 7fcd70444700 1 -- 172.21.2.102:0/4150310224 shutdown_connections 2021-02-10T11:40:32.545 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:32.542 7fcd70444700 1 -- 172.21.2.102:0/4150310224 wait complete. 2021-02-10T11:40:32.552 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-02-10T11:40:32.987 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:32.967 7f175dab1700 1 Processor -- start 2021-02-10T11:40:32.987 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:32.968 7f175dab1700 1 -- start start 2021-02-10T11:40:32.987 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:32.968 7f175dab1700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f17580d1870 0x7f17581b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:32.987 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:32.968 7f175dab1700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f17581b0ae0 0x7f1758108940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:32.987 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:32.968 7f175dab1700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f17580d1c70 0x7f1758108f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:32.987 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:32.968 7f175dab1700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17580e5ef0 con 0x7f17581b0ae0 2021-02-10T11:40:32.987 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:32.968 7f175dab1700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1758109a60 con 0x7f17580d1c70 2021-02-10T11:40:32.987 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:32.968 7f175dab1700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1758088070 con 0x7f17580d1870 2021-02-10T11:40:32.987 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:32.968 7f1756ffd700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f17581b0ae0 0x7f1758108940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:32.987 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:32.968 7f1756ffd700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f17581b0ae0 0x7f1758108940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:41378/0 (socket says 172.21.2.102:41378) 2021-02-10T11:40:32.987 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:32.968 7f1756ffd700 1 -- 172.21.2.102:0/4193110264 learned_addr learned my addr 172.21.2.102:0/4193110264 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:40:32.988 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:32.969 7f1756ffd700 1 -- 172.21.2.102:0/4193110264 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f17580d1870 msgr2=0x7f17581b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:32.988 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:32.969 7f1756ffd700 1 --2- 172.21.2.102:0/4193110264 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f17580d1870 0x7f17581b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:32.988 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:32.969 7f1756ffd700 1 -- 172.21.2.102:0/4193110264 >> v2:172.21.2.113:3300/0 conn(0x7f17580d1c70 msgr2=0x7f1758108f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:32.988 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:32.969 7f1756ffd700 1 --2- 172.21.2.102:0/4193110264 >> v2:172.21.2.113:3300/0 conn(0x7f17580d1c70 0x7f1758108f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:32.988 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:32.969 7f1756ffd700 1 -- 172.21.2.102:0/4193110264 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1758109580 con 0x7f17581b0ae0 2021-02-10T11:40:32.988 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:32.969 7f1756ffd700 1 --2- 172.21.2.102:0/4193110264 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f17581b0ae0 0x7f1758108940 secure :-1 s=READY pgs=3394 cs=0 l=1 rev1=1 rx=0x7f174c006700 tx=0x7f174c009b80).ready entity=mon.0 client_cookie=dd18c2c514d6d0df server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:32.988 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:32.969 7f17567fc700 1 -- 172.21.2.102:0/4193110264 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f174c00c0c0 con 0x7f17581b0ae0 2021-02-10T11:40:32.988 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:32.969 7f17567fc700 1 -- 172.21.2.102:0/4193110264 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f174c00c440 con 0x7f17581b0ae0 2021-02-10T11:40:32.988 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:32.969 7f17567fc700 1 -- 172.21.2.102:0/4193110264 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f174c00c8a0 con 0x7f17581b0ae0 2021-02-10T11:40:32.988 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:32.969 7f175dab1700 1 -- 172.21.2.102:0/4193110264 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f17581b0ae0 msgr2=0x7f1758108940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:32.988 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:32.969 7f175dab1700 1 --2- 172.21.2.102:0/4193110264 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f17581b0ae0 0x7f1758108940 secure :-1 s=READY pgs=3394 cs=0 l=1 rev1=1 rx=0x7f174c006700 tx=0x7f174c009b80).stop 2021-02-10T11:40:32.988 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:32.970 7f175dab1700 1 -- 172.21.2.102:0/4193110264 shutdown_connections 2021-02-10T11:40:32.989 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:32.970 7f175dab1700 1 --2- 172.21.2.102:0/4193110264 >> v2:172.21.2.113:3300/0 conn(0x7f17580d1c70 0x7f1758108f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:32.989 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:32.970 7f175dab1700 1 --2- 172.21.2.102:0/4193110264 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f17581b0ae0 0x7f1758108940 unknown :-1 s=CLOSED pgs=3394 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:32.989 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:32.970 7f175dab1700 1 --2- 172.21.2.102:0/4193110264 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f17580d1870 0x7f17581b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:32.989 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:32.970 7f175dab1700 1 -- 172.21.2.102:0/4193110264 >> 172.21.2.102:0/4193110264 conn(0x7f17580ce420 msgr2=0x7f17580d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:32.989 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:32.970 7f175dab1700 1 -- 172.21.2.102:0/4193110264 shutdown_connections 2021-02-10T11:40:32.989 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:32.970 7f175dab1700 1 -- 172.21.2.102:0/4193110264 wait complete. 2021-02-10T11:40:32.990 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:32.985 7f175dab1700 1 Processor -- start 2021-02-10T11:40:32.990 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:32.985 7f175dab1700 1 -- start start 2021-02-10T11:40:32.990 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:32.985 7f175dab1700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f17581a7bf0 0x7f17581a7ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:32.990 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:32.985 7f175dab1700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1758195560 0x7f1758195960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:32.990 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:32.985 7f175dab1700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1758195f80 0x7f1758196380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:32.991 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:32.985 7f175dab1700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1758194fe0 con 0x7f17581a7bf0 2021-02-10T11:40:32.991 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:32.985 7f175dab1700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1758193020 con 0x7f1758195560 2021-02-10T11:40:32.991 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:32.985 7f175dab1700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1758193440 con 0x7f1758195f80 2021-02-10T11:40:32.991 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:32.985 7f17577fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f17581a7bf0 0x7f17581a7ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:32.991 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:32.986 7f17577fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f17581a7bf0 0x7f17581a7ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:41382/0 (socket says 172.21.2.102:41382) 2021-02-10T11:40:32.991 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:32.986 7f17577fe700 1 -- 172.21.2.102:0/3736156802 learned_addr learned my addr 172.21.2.102:0/3736156802 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:40:32.991 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:32.986 7f17577fe700 1 -- 172.21.2.102:0/3736156802 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1758195f80 msgr2=0x7f1758196380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:32.991 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:32.986 7f17577fe700 1 --2- 172.21.2.102:0/3736156802 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1758195f80 0x7f1758196380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:32.991 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:32.986 7f17577fe700 1 -- 172.21.2.102:0/3736156802 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1758195560 msgr2=0x7f1758195960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:32.991 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:32.986 7f17577fe700 1 --2- 172.21.2.102:0/3736156802 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1758195560 0x7f1758195960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:32.991 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:32.986 7f17577fe700 1 -- 172.21.2.102:0/3736156802 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1758193660 con 0x7f17581a7bf0 2021-02-10T11:40:32.991 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:32.986 7f17577fe700 1 --2- 172.21.2.102:0/3736156802 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f17581a7bf0 0x7f17581a7ff0 secure :-1 s=READY pgs=3395 cs=0 l=1 rev1=1 rx=0x7f1750009980 tx=0x7f175000bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:32.992 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:32.986 7f173ffff700 1 -- 172.21.2.102:0/3736156802 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f175000c2f0 con 0x7f17581a7bf0 2021-02-10T11:40:32.992 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:32.986 7f173ffff700 1 -- 172.21.2.102:0/3736156802 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f175000c570 con 0x7f17581a7bf0 2021-02-10T11:40:32.992 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:32.986 7f175dab1700 1 -- 172.21.2.102:0/3736156802 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1758193660 con 0x7f17581a7bf0 2021-02-10T11:40:32.992 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:32.986 7f173ffff700 1 -- 172.21.2.102:0/3736156802 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f175000c9d0 con 0x7f17581a7bf0 2021-02-10T11:40:32.992 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:32.986 7f175dab1700 1 -- 172.21.2.102:0/3736156802 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1758193bf0 con 0x7f17581a7bf0 2021-02-10T11:40:32.992 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:32.987 7f173ffff700 1 -- 172.21.2.102:0/3736156802 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1750001010 con 0x7f17581a7bf0 2021-02-10T11:40:32.992 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:32.987 7f173ffff700 1 --2- 172.21.2.102:0/3736156802 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f17380186c0 0x7f173801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:32.992 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:32.987 7f173ffff700 1 -- 172.21.2.102:0/3736156802 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f17500328d0 con 0x7f17581a7bf0 2021-02-10T11:40:32.992 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:32.987 7f1756ffd700 1 --2- 172.21.2.102:0/3736156802 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f17380186c0 0x7f173801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:32.992 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:32.987 7f1756ffd700 1 --2- 172.21.2.102:0/3736156802 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f17380186c0 0x7f173801abc0 crc :-1 s=READY pgs=1481 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:33.021 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:33.018 7f175dab1700 1 -- 172.21.2.102:0/3736156802 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1758082980 con 0x7f17581a7bf0 2021-02-10T11:40:33.023 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:33.021 7f173ffff700 1 -- 172.21.2.102:0/3736156802 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f1750000cc0 con 0x7f17581a7bf0 2021-02-10T11:40:33.151 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:33.149 7f175dab1700 1 -- 172.21.2.102:0/3736156802 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7f1758082980 con 0x7f17581a7bf0 2021-02-10T11:40:33.151 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:33.149 7f173ffff700 1 -- 172.21.2.102:0/3736156802 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v197) v1 ==== 104+0+10 (secure 0 0 0) 0x7f1750001010 con 0x7f17581a7bf0 2021-02-10T11:40:33.152 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:40:33.153 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:33.151 7f175dab1700 1 -- 172.21.2.102:0/3736156802 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f17380186c0 msgr2=0x7f173801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:33.153 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:33.151 7f175dab1700 1 --2- 172.21.2.102:0/3736156802 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f17380186c0 0x7f173801abc0 crc :-1 s=READY pgs=1481 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:33.153 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:33.151 7f175dab1700 1 -- 172.21.2.102:0/3736156802 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f17581a7bf0 msgr2=0x7f17581a7ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:33.154 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:33.151 7f175dab1700 1 --2- 172.21.2.102:0/3736156802 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f17581a7bf0 0x7f17581a7ff0 secure :-1 s=READY pgs=3395 cs=0 l=1 rev1=1 rx=0x7f1750009980 tx=0x7f175000bcd0).stop 2021-02-10T11:40:33.154 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:33.151 7f175dab1700 1 -- 172.21.2.102:0/3736156802 shutdown_connections 2021-02-10T11:40:33.154 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:33.151 7f175dab1700 1 --2- 172.21.2.102:0/3736156802 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1758195f80 0x7f1758196380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:33.154 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:33.151 7f175dab1700 1 --2- 172.21.2.102:0/3736156802 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f17380186c0 0x7f173801abc0 unknown :-1 s=CLOSED pgs=1481 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:33.154 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:33.151 7f175dab1700 1 --2- 172.21.2.102:0/3736156802 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1758195560 0x7f1758195960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:33.154 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:33.151 7f175dab1700 1 --2- 172.21.2.102:0/3736156802 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f17581a7bf0 0x7f17581a7ff0 unknown :-1 s=CLOSED pgs=3395 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:33.154 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:33.151 7f175dab1700 1 -- 172.21.2.102:0/3736156802 >> 172.21.2.102:0/3736156802 conn(0x7f17580a3270 msgr2=0x7f1758095970 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:33.154 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:33.151 7f175dab1700 1 -- 172.21.2.102:0/3736156802 shutdown_connections 2021-02-10T11:40:33.154 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:33.151 7f175dab1700 1 -- 172.21.2.102:0/3736156802 wait complete. 2021-02-10T11:40:33.162 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-02-10T11:40:33.595 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:33.576 7f18ffc1b700 1 Processor -- start 2021-02-10T11:40:33.595 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:33.576 7f18ffc1b700 1 -- start start 2021-02-10T11:40:33.595 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:33.576 7f18ffc1b700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f18f80d1870 0x7f18f81b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:33.595 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:33.576 7f18ffc1b700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f18f80cff70 0x7f18f8109130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:33.595 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:33.576 7f18ffc1b700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18f8109a60 con 0x7f18f81b0ae0 2021-02-10T11:40:33.596 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:33.576 7f18ffc1b700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18f8109750 con 0x7f18f80d1870 2021-02-10T11:40:33.596 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:33.576 7f18ffc1b700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18f8088070 con 0x7f18f80cff70 2021-02-10T11:40:33.596 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:33.576 7f18fd236700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f18f81b0ae0 0x7f18f8108940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:33614/0 (socket says 172.21.2.102:33614) 2021-02-10T11:40:33.596 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:33.576 7f18fda37700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f18f80d1870 0x7f18f81b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:33.596 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:33.577 7f18fd236700 1 -- 172.21.2.102:0/1607977741 learned_addr learned my addr 172.21.2.102:0/1607977741 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:40:33.596 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:33.577 7f18fe238700 1 --2- 172.21.2.102:0/1607977741 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f18f80cff70 0x7f18f8109130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:33.596 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:33.577 7f18fda37700 1 -- 172.21.2.102:0/1607977741 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f18f80cff70 msgr2=0x7f18f8109130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:33.596 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:33.577 7f18fda37700 1 --2- 172.21.2.102:0/1607977741 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f18f80cff70 0x7f18f8109130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:33.596 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:33.577 7f18fda37700 1 -- 172.21.2.102:0/1607977741 >> v1:172.21.2.102:6789/0 conn(0x7f18f81b0ae0 legacy=0x7f18f8108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:33.596 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:33.577 7f18fda37700 1 -- 172.21.2.102:0/1607977741 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18f80d1c70 con 0x7f18f80d1870 2021-02-10T11:40:33.596 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:33.577 7f18fda37700 1 --2- 172.21.2.102:0/1607977741 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f18f80d1870 0x7f18f81b04c0 secure :-1 s=READY pgs=3397 cs=0 l=1 rev1=1 rx=0x7f18ec006650 tx=0x7f18ec009a80).ready entity=mon.0 client_cookie=5ee2c48bf821dc65 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:33.596 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:33.577 7f18fca35700 1 -- 172.21.2.102:0/1607977741 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f18ec00c0c0 con 0x7f18f80d1870 2021-02-10T11:40:33.596 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:33.577 7f18fca35700 1 -- 172.21.2.102:0/1607977741 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18ec00c340 con 0x7f18f80d1870 2021-02-10T11:40:33.597 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:33.577 7f18fca35700 1 -- 172.21.2.102:0/1607977741 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f18ec00c860 con 0x7f18f80d1870 2021-02-10T11:40:33.597 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:33.577 7f18ffc1b700 1 -- 172.21.2.102:0/1607977741 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f18f80d1870 msgr2=0x7f18f81b04c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:33.597 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:33.577 7f18ffc1b700 1 --2- 172.21.2.102:0/1607977741 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f18f80d1870 0x7f18f81b04c0 secure :-1 s=READY pgs=3397 cs=0 l=1 rev1=1 rx=0x7f18ec006650 tx=0x7f18ec009a80).stop 2021-02-10T11:40:33.597 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:33.578 7f18ffc1b700 1 -- 172.21.2.102:0/1607977741 shutdown_connections 2021-02-10T11:40:33.597 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:33.578 7f18ffc1b700 1 --2- 172.21.2.102:0/1607977741 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f18f80cff70 0x7f18f8109130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:33.597 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:33.578 7f18ffc1b700 1 --2- 172.21.2.102:0/1607977741 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f18f80d1870 0x7f18f81b04c0 unknown :-1 s=CLOSED pgs=3397 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:33.597 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:33.578 7f18ffc1b700 1 -- 172.21.2.102:0/1607977741 >> 172.21.2.102:0/1607977741 conn(0x7f18f80ce420 msgr2=0x7f18f80d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:33.597 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:33.578 7f18ffc1b700 1 -- 172.21.2.102:0/1607977741 shutdown_connections 2021-02-10T11:40:33.597 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:33.578 7f18ffc1b700 1 -- 172.21.2.102:0/1607977741 wait complete. 2021-02-10T11:40:33.599 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:33.593 7f18ffc1b700 1 Processor -- start 2021-02-10T11:40:33.599 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:33.593 7f18ffc1b700 1 -- start start 2021-02-10T11:40:33.599 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:33.593 7f18ffc1b700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f18f80b04d0 0x7f18f81a7a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:33.599 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:33.593 7f18ffc1b700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f18f81a8090 0x7f18f80c23a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:33.599 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:33.594 7f18ffc1b700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f18f808d680 0x7f18f80c29c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:33.599 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:33.594 7f18ffc1b700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18f80b0a00 con 0x7f18f808d680 2021-02-10T11:40:33.599 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:33.594 7f18ffc1b700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18f80c2fe0 con 0x7f18f81a8090 2021-02-10T11:40:33.600 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:33.594 7f18ffc1b700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18f80c3200 con 0x7f18f80b04d0 2021-02-10T11:40:33.600 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:33.594 7f18fe238700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f18f808d680 0x7f18f80c29c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:33.600 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:33.594 7f18fe238700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f18f808d680 0x7f18f80c29c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:41400/0 (socket says 172.21.2.102:41400) 2021-02-10T11:40:33.600 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:33.594 7f18fe238700 1 -- 172.21.2.102:0/3300928807 learned_addr learned my addr 172.21.2.102:0/3300928807 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:40:33.600 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:33.594 7f18fda37700 1 --2- 172.21.2.102:0/3300928807 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f18f80b04d0 0x7f18f81a7a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:33.600 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:33.594 7f18fd236700 1 --2- 172.21.2.102:0/3300928807 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f18f81a8090 0x7f18f80c23a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:33.600 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:33.594 7f18fe238700 1 -- 172.21.2.102:0/3300928807 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f18f80b04d0 msgr2=0x7f18f81a7a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:33.600 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:33.594 7f18fe238700 1 --2- 172.21.2.102:0/3300928807 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f18f80b04d0 0x7f18f81a7a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:33.600 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:33.594 7f18fe238700 1 -- 172.21.2.102:0/3300928807 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f18f81a8090 msgr2=0x7f18f80c23a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:33.600 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:33.594 7f18fe238700 1 --2- 172.21.2.102:0/3300928807 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f18f81a8090 0x7f18f80c23a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:33.600 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:33.594 7f18fe238700 1 -- 172.21.2.102:0/3300928807 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18f808e080 con 0x7f18f808d680 2021-02-10T11:40:33.600 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:33.594 7f18fe238700 1 --2- 172.21.2.102:0/3300928807 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f18f808d680 0x7f18f80c29c0 secure :-1 s=READY pgs=3398 cs=0 l=1 rev1=1 rx=0x7f18f000b970 tx=0x7f18f000bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:33.601 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:33.594 7f18ea7fc700 1 -- 172.21.2.102:0/3300928807 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f18f0000f90 con 0x7f18f808d680 2021-02-10T11:40:33.601 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:33.595 7f18ea7fc700 1 -- 172.21.2.102:0/3300928807 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18f0006aa0 con 0x7f18f808d680 2021-02-10T11:40:33.601 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:33.595 7f18ea7fc700 1 -- 172.21.2.102:0/3300928807 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f18f00015f0 con 0x7f18f808d680 2021-02-10T11:40:33.601 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:33.595 7f18ffc1b700 1 -- 172.21.2.102:0/3300928807 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18f808e080 con 0x7f18f808d680 2021-02-10T11:40:33.601 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:33.595 7f18ffc1b700 1 -- 172.21.2.102:0/3300928807 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18f808e080 con 0x7f18f808d680 2021-02-10T11:40:33.601 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:33.595 7f18ea7fc700 1 -- 172.21.2.102:0/3300928807 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f18f00011f0 con 0x7f18f808d680 2021-02-10T11:40:33.601 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:33.595 7f18ea7fc700 1 --2- 172.21.2.102:0/3300928807 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f18e0018730 0x7f18e001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:33.601 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:33.596 7f18ea7fc700 1 -- 172.21.2.102:0/3300928807 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f18f00011f0 con 0x7f18f808d680 2021-02-10T11:40:33.601 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:33.596 7f18fda37700 1 --2- 172.21.2.102:0/3300928807 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f18e0018730 0x7f18e001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:33.601 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:33.596 7f18fda37700 1 --2- 172.21.2.102:0/3300928807 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f18e0018730 0x7f18e001ac10 crc :-1 s=READY pgs=1482 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:33.628 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:33.626 7f18ffc1b700 1 -- 172.21.2.102:0/3300928807 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18f8082a70 con 0x7f18f808d680 2021-02-10T11:40:33.631 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:33.629 7f18ea7fc700 1 -- 172.21.2.102:0/3300928807 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f18f0000f40 con 0x7f18f808d680 2021-02-10T11:40:33.761 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:33.759 7f18ffc1b700 1 -- 172.21.2.102:0/3300928807 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7f18f8082a70 con 0x7f18f808d680 2021-02-10T11:40:33.761 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:33.759 7f18ea7fc700 1 -- 172.21.2.102:0/3300928807 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v197) v1 ==== 100+0+10 (secure 0 0 0) 0x7f18f0006aa0 con 0x7f18f808d680 2021-02-10T11:40:33.762 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:40:33.764 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:33.761 7f18ffc1b700 1 -- 172.21.2.102:0/3300928807 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f18e0018730 msgr2=0x7f18e001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:33.764 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:33.761 7f18ffc1b700 1 --2- 172.21.2.102:0/3300928807 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f18e0018730 0x7f18e001ac10 crc :-1 s=READY pgs=1482 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:33.764 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:33.761 7f18ffc1b700 1 -- 172.21.2.102:0/3300928807 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f18f808d680 msgr2=0x7f18f80c29c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:33.764 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:33.761 7f18ffc1b700 1 --2- 172.21.2.102:0/3300928807 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f18f808d680 0x7f18f80c29c0 secure :-1 s=READY pgs=3398 cs=0 l=1 rev1=1 rx=0x7f18f000b970 tx=0x7f18f000bd10).stop 2021-02-10T11:40:33.764 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:33.761 7f18ffc1b700 1 -- 172.21.2.102:0/3300928807 shutdown_connections 2021-02-10T11:40:33.764 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:33.761 7f18ffc1b700 1 --2- 172.21.2.102:0/3300928807 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f18f808d680 0x7f18f80c29c0 unknown :-1 s=CLOSED pgs=3398 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:33.764 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:33.761 7f18ffc1b700 1 --2- 172.21.2.102:0/3300928807 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f18e0018730 0x7f18e001ac10 unknown :-1 s=CLOSED pgs=1482 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:33.764 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:33.761 7f18ffc1b700 1 --2- 172.21.2.102:0/3300928807 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f18f81a8090 0x7f18f80c23a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:33.764 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:33.761 7f18ffc1b700 1 --2- 172.21.2.102:0/3300928807 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f18f80b04d0 0x7f18f81a7a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:33.764 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:33.761 7f18ffc1b700 1 -- 172.21.2.102:0/3300928807 >> 172.21.2.102:0/3300928807 conn(0x7f18f80be530 msgr2=0x7f18f809f1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:33.764 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:33.762 7f18ffc1b700 1 -- 172.21.2.102:0/3300928807 shutdown_connections 2021-02-10T11:40:33.765 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:33.762 7f18ffc1b700 1 -- 172.21.2.102:0/3300928807 wait complete. 2021-02-10T11:40:33.772 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-02-10T11:40:33.772 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:40:34.203 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:34.183 7f8c59688700 1 Processor -- start 2021-02-10T11:40:34.203 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:34.183 7f8c59688700 1 -- start start 2021-02-10T11:40:34.203 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:34.184 7f8c59688700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f8c540d1870 0x7f8c541b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:34.203 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:34.184 7f8c59688700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f8c540f0650 0x7f8c540cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:34.203 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:34.184 7f8c59688700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c540f0b10 con 0x7f8c541b0a10 2021-02-10T11:40:34.203 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:34.184 7f8c59688700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c540e5ed0 con 0x7f8c540f0650 2021-02-10T11:40:34.203 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:34.184 7f8c59688700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c540d1c70 con 0x7f8c540d1870 2021-02-10T11:40:34.203 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:34.184 7f8c52abe700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f8c541b0a10 0x7f8c540cf330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:33628/0 (socket says 172.21.2.102:33628) 2021-02-10T11:40:34.203 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:34.184 7f8c52abe700 1 -- 172.21.2.102:0/2419659192 learned_addr learned my addr 172.21.2.102:0/2419659192 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:40:34.203 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:34.184 7f8c53ac0700 1 --2- 172.21.2.102:0/2419659192 >> v2:172.21.2.102:3300/0 conn(0x7f8c540f0650 0x7f8c540cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:34.204 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:34.184 7f8c53ac0700 1 -- 172.21.2.102:0/2419659192 >> v2:172.21.2.113:3300/0 conn(0x7f8c540d1870 msgr2=0x7f8c541b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:34.204 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:34.184 7f8c53ac0700 1 --2- 172.21.2.102:0/2419659192 >> v2:172.21.2.113:3300/0 conn(0x7f8c540d1870 0x7f8c541b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:34.204 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:34.184 7f8c53ac0700 1 -- 172.21.2.102:0/2419659192 >> v1:172.21.2.102:6789/0 conn(0x7f8c541b0a10 legacy=0x7f8c540cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:34.204 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:34.184 7f8c53ac0700 1 -- 172.21.2.102:0/2419659192 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c540d1e90 con 0x7f8c540f0650 2021-02-10T11:40:34.204 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:34.184 7f8c53ac0700 1 --2- 172.21.2.102:0/2419659192 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8c540f0650 0x7f8c540cfb20 secure :-1 s=READY pgs=3400 cs=0 l=1 rev1=1 rx=0x7f8c4c006e30 tx=0x7f8c4c009de0).ready entity=mon.0 client_cookie=aa79bfbd94522155 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:34.204 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:34.185 7f8c522bd700 1 -- 172.21.2.102:0/2419659192 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8c4c00b300 con 0x7f8c540f0650 2021-02-10T11:40:34.204 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:34.185 7f8c522bd700 1 -- 172.21.2.102:0/2419659192 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c4c00b680 con 0x7f8c540f0650 2021-02-10T11:40:34.204 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:34.185 7f8c522bd700 1 -- 172.21.2.102:0/2419659192 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8c4c00bba0 con 0x7f8c540f0650 2021-02-10T11:40:34.204 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:34.185 7f8c59688700 1 -- 172.21.2.102:0/2419659192 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8c540f0650 msgr2=0x7f8c540cfb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:34.204 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:34.185 7f8c59688700 1 --2- 172.21.2.102:0/2419659192 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8c540f0650 0x7f8c540cfb20 secure :-1 s=READY pgs=3400 cs=0 l=1 rev1=1 rx=0x7f8c4c006e30 tx=0x7f8c4c009de0).stop 2021-02-10T11:40:34.204 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:34.185 7f8c59688700 1 -- 172.21.2.102:0/2419659192 shutdown_connections 2021-02-10T11:40:34.204 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:34.185 7f8c59688700 1 --2- 172.21.2.102:0/2419659192 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8c540f0650 0x7f8c540cfb20 unknown :-1 s=CLOSED pgs=3400 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:34.204 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:34.185 7f8c59688700 1 --2- 172.21.2.102:0/2419659192 >> v2:172.21.2.113:3300/0 conn(0x7f8c540d1870 0x7f8c541b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:34.205 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:34.185 7f8c59688700 1 -- 172.21.2.102:0/2419659192 >> 172.21.2.102:0/2419659192 conn(0x7f8c540ce450 msgr2=0x7f8c540d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:34.205 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:34.186 7f8c59688700 1 -- 172.21.2.102:0/2419659192 shutdown_connections 2021-02-10T11:40:34.205 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:34.186 7f8c59688700 1 -- 172.21.2.102:0/2419659192 wait complete. 2021-02-10T11:40:34.206 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:34.201 7f8c59688700 1 Processor -- start 2021-02-10T11:40:34.206 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:34.201 7f8c59688700 1 -- start start 2021-02-10T11:40:34.206 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:34.201 7f8c59688700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8c540b4f30 0x7f8c5418b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:34.207 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:34.201 7f8c59688700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8c54093f40 0x7f8c54094340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:34.207 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:34.201 7f8c59688700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8c54094960 0x7f8c54187e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:34.207 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:34.201 7f8c59688700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c54094e90 con 0x7f8c54094960 2021-02-10T11:40:34.207 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:34.201 7f8c59688700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c5418bff0 con 0x7f8c540b4f30 2021-02-10T11:40:34.207 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:34.201 7f8c59688700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c5408f780 con 0x7f8c54093f40 2021-02-10T11:40:34.207 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:34.201 7f8c53ac0700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8c54094960 0x7f8c54187e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:34.207 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:34.202 7f8c53ac0700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8c54094960 0x7f8c54187e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:41410/0 (socket says 172.21.2.102:41410) 2021-02-10T11:40:34.207 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:34.202 7f8c53ac0700 1 -- 172.21.2.102:0/3854430904 learned_addr learned my addr 172.21.2.102:0/3854430904 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:40:34.207 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:34.202 7f8c52abe700 1 --2- 172.21.2.102:0/3854430904 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8c54093f40 0x7f8c54094340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:34.207 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:34.202 7f8c532bf700 1 --2- 172.21.2.102:0/3854430904 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8c540b4f30 0x7f8c5418b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:34.207 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:34.202 7f8c53ac0700 1 -- 172.21.2.102:0/3854430904 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8c54093f40 msgr2=0x7f8c54094340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:34.207 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:34.202 7f8c53ac0700 1 --2- 172.21.2.102:0/3854430904 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8c54093f40 0x7f8c54094340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:34.208 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:34.202 7f8c53ac0700 1 -- 172.21.2.102:0/3854430904 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8c540b4f30 msgr2=0x7f8c5418b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:34.208 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:34.202 7f8c53ac0700 1 --2- 172.21.2.102:0/3854430904 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8c540b4f30 0x7f8c5418b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:34.208 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:34.202 7f8c53ac0700 1 -- 172.21.2.102:0/3854430904 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c54188470 con 0x7f8c54094960 2021-02-10T11:40:34.208 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:34.202 7f8c532bf700 1 --2- 172.21.2.102:0/3854430904 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8c540b4f30 0x7f8c5418b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:40:34.208 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:34.202 7f8c53ac0700 1 --2- 172.21.2.102:0/3854430904 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8c54094960 0x7f8c54187e50 secure :-1 s=READY pgs=3401 cs=0 l=1 rev1=1 rx=0x7f8c4c00b970 tx=0x7f8c4c00bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:34.208 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:34.202 7f8c4bfff700 1 -- 172.21.2.102:0/3854430904 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8c4c0093d0 con 0x7f8c54094960 2021-02-10T11:40:34.208 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:34.202 7f8c4bfff700 1 -- 172.21.2.102:0/3854430904 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c4c005820 con 0x7f8c54094960 2021-02-10T11:40:34.208 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:34.202 7f8c59688700 1 -- 172.21.2.102:0/3854430904 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c54188470 con 0x7f8c54094960 2021-02-10T11:40:34.208 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:34.202 7f8c59688700 1 -- 172.21.2.102:0/3854430904 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c54188470 con 0x7f8c54094960 2021-02-10T11:40:34.208 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:34.202 7f8c4bfff700 1 -- 172.21.2.102:0/3854430904 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8c4c005d00 con 0x7f8c54094960 2021-02-10T11:40:34.208 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:34.203 7f8c4bfff700 1 -- 172.21.2.102:0/3854430904 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f8c4c005a00 con 0x7f8c54094960 2021-02-10T11:40:34.208 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:34.203 7f8c4bfff700 1 --2- 172.21.2.102:0/3854430904 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8c380186c0 0x7f8c3801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:34.208 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:34.203 7f8c4bfff700 1 -- 172.21.2.102:0/3854430904 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f8c4c032950 con 0x7f8c54094960 2021-02-10T11:40:34.208 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:34.203 7f8c532bf700 1 --2- 172.21.2.102:0/3854430904 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8c380186c0 0x7f8c3801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:34.209 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:34.203 7f8c532bf700 1 --2- 172.21.2.102:0/3854430904 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8c380186c0 0x7f8c3801abc0 crc :-1 s=READY pgs=1483 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:34.236 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:34.234 7f8c59688700 1 -- 172.21.2.102:0/3854430904 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c5418bff0 con 0x7f8c54094960 2021-02-10T11:40:34.238 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:34.236 7f8c4bfff700 1 -- 172.21.2.102:0/3854430904 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f8c4c02cfa0 con 0x7f8c54094960 2021-02-10T11:40:34.369 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:34.367 7f8c59688700 1 -- 172.21.2.102:0/3854430904 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8c5418bff0 con 0x7f8c54094960 2021-02-10T11:40:34.370 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:34.368 7f8c4bfff700 1 -- 172.21.2.102:0/3854430904 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 173 v173) v1 ==== 95+0+2108 (secure 0 0 0) 0x7f8c4c02cfa0 con 0x7f8c54094960 2021-02-10T11:40:34.370 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:40:34.371 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":173,"default_fscid":21,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":13872,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.102:6835/1871410080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6834","nonce":1871410080},{"type":"v1","addr":"172.21.2.102:6835","nonce":1871410080}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":169}],"filesystems":[{"mdsmap":{"epoch":173,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:40:24.365024","modified":"2021-02-10 11:40:26.372300","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7598},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7598":{"gid":7598,"name":"b","rank":0,"incarnation":170,"state":"up:active","state_seq":3,"addr":"172.21.2.113:6833/2247818770","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6832","nonce":2247818770},{"type":"v1","addr":"172.21.2.113:6833","nonce":2247818770}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[43],"metadata_pool":42,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":21}]} 2021-02-10T11:40:34.371 INFO:teuthology.orchestra.run.gibba002.stderr:dumped fsmap epoch 173 2021-02-10T11:40:34.372 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:34.369 7f8c59688700 1 -- 172.21.2.102:0/3854430904 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8c380186c0 msgr2=0x7f8c3801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:34.372 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:34.369 7f8c59688700 1 --2- 172.21.2.102:0/3854430904 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8c380186c0 0x7f8c3801abc0 crc :-1 s=READY pgs=1483 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:34.372 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:34.369 7f8c59688700 1 -- 172.21.2.102:0/3854430904 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8c54094960 msgr2=0x7f8c54187e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:34.372 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:34.369 7f8c59688700 1 --2- 172.21.2.102:0/3854430904 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8c54094960 0x7f8c54187e50 secure :-1 s=READY pgs=3401 cs=0 l=1 rev1=1 rx=0x7f8c4c00b970 tx=0x7f8c4c00bd10).stop 2021-02-10T11:40:34.372 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:34.370 7f8c59688700 1 -- 172.21.2.102:0/3854430904 shutdown_connections 2021-02-10T11:40:34.372 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:34.370 7f8c59688700 1 --2- 172.21.2.102:0/3854430904 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f8c54094960 0x7f8c54187e50 unknown :-1 s=CLOSED pgs=3401 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:34.372 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:34.370 7f8c59688700 1 --2- 172.21.2.102:0/3854430904 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f8c54093f40 0x7f8c54094340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:34.372 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:34.370 7f8c59688700 1 --2- 172.21.2.102:0/3854430904 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f8c380186c0 0x7f8c3801abc0 unknown :-1 s=CLOSED pgs=1483 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:34.372 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:34.370 7f8c59688700 1 --2- 172.21.2.102:0/3854430904 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f8c540b4f30 0x7f8c5418b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:34.372 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:34.370 7f8c59688700 1 -- 172.21.2.102:0/3854430904 >> 172.21.2.102:0/3854430904 conn(0x7f8c5419f8f0 msgr2=0x7f8c541a5f80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:34.373 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:34.370 7f8c59688700 1 -- 172.21.2.102:0/3854430904 shutdown_connections 2021-02-10T11:40:34.373 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:34.370 7f8c59688700 1 -- 172.21.2.102:0/3854430904 wait complete. 2021-02-10T11:40:34.583 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 173, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:40:24.365024', 'modified': '2021-02-10 11:40:26.372300', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7598}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7598': {'gid': 7598, 'name': 'b', 'rank': 0, 'incarnation': 170, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.113:6833/2247818770', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.113:6832', 'nonce': 2247818770}, {'type': 'v1', 'addr': '172.21.2.113:6833', 'nonce': 2247818770}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [43], 'metadata_pool': 42, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:40:34.583 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:40:34.583 DEBUG:teuthology.orchestra.run.gibba013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:40:34.983 INFO:teuthology.orchestra.run.gibba013.stdout:{ 2021-02-10T11:40:34.983 INFO:teuthology.orchestra.run.gibba013.stdout: "cluster_fsid": "68db30ea-dbce-4aef-a6df-f01c6fea0513", 2021-02-10T11:40:34.984 INFO:teuthology.orchestra.run.gibba013.stdout: "whoami": 0, 2021-02-10T11:40:34.984 INFO:teuthology.orchestra.run.gibba013.stdout: "id": 7598, 2021-02-10T11:40:34.984 INFO:teuthology.orchestra.run.gibba013.stdout: "want_state": "up:active", 2021-02-10T11:40:34.984 INFO:teuthology.orchestra.run.gibba013.stdout: "state": "up:active", 2021-02-10T11:40:34.984 INFO:teuthology.orchestra.run.gibba013.stdout: "rank_uptime": 10.614456633, 2021-02-10T11:40:34.984 INFO:teuthology.orchestra.run.gibba013.stdout: "mdsmap_epoch": 173, 2021-02-10T11:40:34.984 INFO:teuthology.orchestra.run.gibba013.stdout: "osdmap_epoch": 197, 2021-02-10T11:40:34.984 INFO:teuthology.orchestra.run.gibba013.stdout: "osdmap_epoch_barrier": 197, 2021-02-10T11:40:34.984 INFO:teuthology.orchestra.run.gibba013.stdout: "uptime": 16.267975777 2021-02-10T11:40:34.984 INFO:teuthology.orchestra.run.gibba013.stdout:} 2021-02-10T11:40:34.987 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "68db30ea-dbce-4aef-a6df-f01c6fea0513", "whoami": 0, "id": 7598, "want_state": "up:active", "state": "up:active", "rank_uptime": 10.614456633, "mdsmap_epoch": 173, "osdmap_epoch": 197, "osdmap_epoch_barrier": 197, "uptime": 16.267975777 } 2021-02-10T11:40:34.988 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2021-02-10T11:40:34.988 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2021-02-10T11:40:34.988 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@gibba002.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2021-02-10T11:40:34.988 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:40:34.988 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec mkdir -p /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:40:35.324 DEBUG:teuthology.orchestra.run.gibba002:> sudo modprobe fuse 2021-02-10T11:40:35.672 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:40:35.673 DEBUG:teuthology.orchestra.run.gibba002:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:40:36.016 INFO:teuthology.orchestra.run.gibba002.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:40:36.016 INFO:teuthology.orchestra.run.gibba002.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:40:36.017 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:40:36.017 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:40:36.018 DEBUG:teuthology.orchestra.run.gibba002:> ls /sys/fs/fuse/connections 2021-02-10T11:40:36.352 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2021-02-10T11:40:36.352 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:40:36.355 DEBUG:teuthology.orchestra.run.gibba002:> sudo modprobe fuse 2021-02-10T11:40:36.700 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:40:36.700 DEBUG:teuthology.orchestra.run.gibba002:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:40:36.767 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba002.stderr:ceph-fuse[221170]: starting ceph client 2021-02-10T11:40:36.768 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba002.stderr:2021-02-10 11:40:36.765 7f5384791f80 -1 init, newargv = 0x563b5ee9ed70 newargc=9 2021-02-10T11:40:36.773 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba002.stderr:ceph-fuse[221170]: starting fuse 2021-02-10T11:40:37.043 INFO:teuthology.orchestra.run.gibba002.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:40:37.043 INFO:teuthology.orchestra.run.gibba002.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:40:37.044 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:40:37.044 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:40:37.044 DEBUG:teuthology.orchestra.run.gibba002:> ls /sys/fs/fuse/connections 2021-02-10T11:40:37.379 INFO:teuthology.orchestra.run.gibba002.stdout:43 2021-02-10T11:40:37.380 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2021-02-10T11:40:37.380 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:40:37.380 DEBUG:teuthology.orchestra.run.gibba002:> sudo python3 -c ' 2021-02-10T11:40:37.380 DEBUG:teuthology.orchestra.run.gibba002:> import glob 2021-02-10T11:40:37.380 DEBUG:teuthology.orchestra.run.gibba002:> import re 2021-02-10T11:40:37.380 DEBUG:teuthology.orchestra.run.gibba002:> import os 2021-02-10T11:40:37.380 DEBUG:teuthology.orchestra.run.gibba002:> import subprocess 2021-02-10T11:40:37.380 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:40:37.380 DEBUG:teuthology.orchestra.run.gibba002:> def find_socket(client_name): 2021-02-10T11:40:37.380 DEBUG:teuthology.orchestra.run.gibba002:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2021-02-10T11:40:37.380 DEBUG:teuthology.orchestra.run.gibba002:> files = glob.glob(asok_path) 2021-02-10T11:40:37.381 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:40:37.381 DEBUG:teuthology.orchestra.run.gibba002:> # Given a non-glob path, it better be there 2021-02-10T11:40:37.381 DEBUG:teuthology.orchestra.run.gibba002:> if "*" not in asok_path: 2021-02-10T11:40:37.381 DEBUG:teuthology.orchestra.run.gibba002:> assert(len(files) == 1) 2021-02-10T11:40:37.381 DEBUG:teuthology.orchestra.run.gibba002:> return files[0] 2021-02-10T11:40:37.381 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:40:37.381 DEBUG:teuthology.orchestra.run.gibba002:> for f in files: 2021-02-10T11:40:37.381 DEBUG:teuthology.orchestra.run.gibba002:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2021-02-10T11:40:37.381 DEBUG:teuthology.orchestra.run.gibba002:> if os.path.exists("/proc/{0}".format(pid)): 2021-02-10T11:40:37.381 DEBUG:teuthology.orchestra.run.gibba002:> return f 2021-02-10T11:40:37.381 DEBUG:teuthology.orchestra.run.gibba002:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2021-02-10T11:40:37.381 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:40:37.381 DEBUG:teuthology.orchestra.run.gibba002:> print(find_socket("client.0")) 2021-02-10T11:40:37.381 DEBUG:teuthology.orchestra.run.gibba002:> ' 2021-02-10T11:40:37.745 INFO:teuthology.orchestra.run.gibba002.stdout:/var/run/ceph/ceph-client.0.221170.asok 2021-02-10T11:40:37.747 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.221170.asok 2021-02-10T11:40:37.747 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:40:37.748 DEBUG:teuthology.orchestra.run.gibba002:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.221170.asok status 2021-02-10T11:40:38.140 INFO:teuthology.orchestra.run.gibba002.stdout:{ 2021-02-10T11:40:38.141 INFO:teuthology.orchestra.run.gibba002.stdout: "metadata": { 2021-02-10T11:40:38.141 INFO:teuthology.orchestra.run.gibba002.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:40:38.141 INFO:teuthology.orchestra.run.gibba002.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:40:38.141 INFO:teuthology.orchestra.run.gibba002.stdout: "entity_id": "0", 2021-02-10T11:40:38.141 INFO:teuthology.orchestra.run.gibba002.stdout: "hostname": "gibba002", 2021-02-10T11:40:38.141 INFO:teuthology.orchestra.run.gibba002.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2021-02-10T11:40:38.141 INFO:teuthology.orchestra.run.gibba002.stdout: "pid": "221170", 2021-02-10T11:40:38.141 INFO:teuthology.orchestra.run.gibba002.stdout: "root": "/" 2021-02-10T11:40:38.141 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:40:38.141 INFO:teuthology.orchestra.run.gibba002.stdout: "dentry_count": 0, 2021-02-10T11:40:38.141 INFO:teuthology.orchestra.run.gibba002.stdout: "dentry_pinned_count": 0, 2021-02-10T11:40:38.141 INFO:teuthology.orchestra.run.gibba002.stdout: "id": 14037, 2021-02-10T11:40:38.141 INFO:teuthology.orchestra.run.gibba002.stdout: "inst": { 2021-02-10T11:40:38.142 INFO:teuthology.orchestra.run.gibba002.stdout: "name": { 2021-02-10T11:40:38.142 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "client", 2021-02-10T11:40:38.142 INFO:teuthology.orchestra.run.gibba002.stdout: "num": 14037 2021-02-10T11:40:38.142 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:40:38.142 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": { 2021-02-10T11:40:38.142 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:40:38.142 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:0", 2021-02-10T11:40:38.142 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 3333390531 2021-02-10T11:40:38.142 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:40:38.142 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:40:38.143 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": { 2021-02-10T11:40:38.143 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:40:38.143 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:0", 2021-02-10T11:40:38.143 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 3333390531 2021-02-10T11:40:38.143 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:40:38.143 INFO:teuthology.orchestra.run.gibba002.stdout: "inst_str": "client.14037 172.21.2.102:0/3333390531", 2021-02-10T11:40:38.143 INFO:teuthology.orchestra.run.gibba002.stdout: "addr_str": "172.21.2.102:0/3333390531", 2021-02-10T11:40:38.143 INFO:teuthology.orchestra.run.gibba002.stdout: "inode_count": 1, 2021-02-10T11:40:38.143 INFO:teuthology.orchestra.run.gibba002.stdout: "mds_epoch": 173, 2021-02-10T11:40:38.143 INFO:teuthology.orchestra.run.gibba002.stdout: "osd_epoch": 197, 2021-02-10T11:40:38.143 INFO:teuthology.orchestra.run.gibba002.stdout: "osd_epoch_barrier": 0, 2021-02-10T11:40:38.143 INFO:teuthology.orchestra.run.gibba002.stdout: "blacklisted": false 2021-02-10T11:40:38.143 INFO:teuthology.orchestra.run.gibba002.stdout:} 2021-02-10T11:40:38.145 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:40:38.145 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:40:38.145 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:40:38.482 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:40:38.482 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:40:38.482 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:40:38.482 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:40:38.829 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:40:39.260 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:39.240 7f1d37497700 1 Processor -- start 2021-02-10T11:40:39.260 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:39.240 7f1d37497700 1 -- start start 2021-02-10T11:40:39.260 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:39.241 7f1d37497700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f1d300d0340 0x7f1d300ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:39.260 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:39.241 7f1d37497700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d300f0b10 con 0x7f1d300d1870 2021-02-10T11:40:39.260 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:39.241 7f1d37497700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d301b0be0 con 0x7f1d300d1c70 2021-02-10T11:40:39.260 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:39.241 7f1d37497700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d300e5ed0 con 0x7f1d300d0340 2021-02-10T11:40:39.260 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:39.241 7f1d352b3700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f1d300d1870 0x7f1d301b03f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:33650/0 (socket says 172.21.2.102:33650) 2021-02-10T11:40:39.260 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:39.241 7f1d352b3700 1 -- 172.21.2.102:0/967362197 learned_addr learned my addr 172.21.2.102:0/967362197 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:40:39.260 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:39.241 7f1d34ab2700 1 --2- 172.21.2.102:0/967362197 >> v2:172.21.2.113:3300/0 conn(0x7f1d300d0340 0x7f1d300ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:39.261 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:39.242 7f1d23fff700 1 -- 172.21.2.102:0/967362197 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3873345143 0 0) 0x7f1d2c0010b0 con 0x7f1d300d1870 2021-02-10T11:40:39.261 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:39.242 7f1d23fff700 1 -- 172.21.2.102:0/967362197 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d1c002fe0 con 0x7f1d300d1870 2021-02-10T11:40:39.261 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:39.242 7f1d34ab2700 1 -- 172.21.2.102:0/967362197 >> v1:172.21.2.113:6789/0 conn(0x7f1d300d1c70 legacy=0x7f1d300efc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:39.261 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:39.242 7f1d34ab2700 1 -- 172.21.2.102:0/967362197 >> v1:172.21.2.102:6789/0 conn(0x7f1d300d1870 legacy=0x7f1d301b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:39.261 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:39.242 7f1d34ab2700 1 -- 172.21.2.102:0/967362197 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d300f0450 con 0x7f1d300d0340 2021-02-10T11:40:39.261 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:39.242 7f1d34ab2700 1 --2- 172.21.2.102:0/967362197 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1d300d0340 0x7f1d300ef640 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 rx=0x7f1d240069d0 tx=0x7f1d24009e40).ready entity=mon.1 client_cookie=b2337bbedb374631 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:39.261 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:39.242 7f1d23fff700 1 -- 172.21.2.102:0/967362197 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1d2400b510 con 0x7f1d300d0340 2021-02-10T11:40:39.261 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:39.243 7f1d23fff700 1 -- 172.21.2.102:0/967362197 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d2400b790 con 0x7f1d300d0340 2021-02-10T11:40:39.261 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:39.243 7f1d23fff700 1 -- 172.21.2.102:0/967362197 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1d2400bcb0 con 0x7f1d300d0340 2021-02-10T11:40:39.261 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:39.243 7f1d37497700 1 -- 172.21.2.102:0/967362197 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1d300d0340 msgr2=0x7f1d300ef640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:39.261 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:39.243 7f1d37497700 1 --2- 172.21.2.102:0/967362197 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1d300d0340 0x7f1d300ef640 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 rx=0x7f1d240069d0 tx=0x7f1d24009e40).stop 2021-02-10T11:40:39.261 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:39.243 7f1d37497700 1 -- 172.21.2.102:0/967362197 shutdown_connections 2021-02-10T11:40:39.262 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:39.243 7f1d37497700 1 --2- 172.21.2.102:0/967362197 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1d300d0340 0x7f1d300ef640 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:39.262 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:39.243 7f1d37497700 1 -- 172.21.2.102:0/967362197 >> 172.21.2.102:0/967362197 conn(0x7f1d300ce450 msgr2=0x7f1d300d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:39.262 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:39.243 7f1d37497700 1 -- 172.21.2.102:0/967362197 shutdown_connections 2021-02-10T11:40:39.262 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:39.244 7f1d37497700 1 -- 172.21.2.102:0/967362197 wait complete. 2021-02-10T11:40:39.263 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:39.258 7f1d37497700 1 Processor -- start 2021-02-10T11:40:39.264 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:39.258 7f1d37497700 1 -- start start 2021-02-10T11:40:39.264 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:39.259 7f1d37497700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1d3018f0a0 0x7f1d30196ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:39.264 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:39.259 7f1d37497700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1d301970d0 0x7f1d301974d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:39.264 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:39.259 7f1d37497700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1d30094650 0x7f1d30094a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:39.264 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:39.259 7f1d37497700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d30095bd0 con 0x7f1d3018f0a0 2021-02-10T11:40:39.264 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:39.259 7f1d37497700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d300ad7c0 con 0x7f1d301970d0 2021-02-10T11:40:39.264 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:39.259 7f1d37497700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d3018ec70 con 0x7f1d30094650 2021-02-10T11:40:39.264 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:39.259 7f1d352b3700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1d3018f0a0 0x7f1d30196ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:39.264 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:39.259 7f1d352b3700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1d3018f0a0 0x7f1d30196ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:41430/0 (socket says 172.21.2.102:41430) 2021-02-10T11:40:39.264 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:39.259 7f1d352b3700 1 -- 172.21.2.102:0/744495068 learned_addr learned my addr 172.21.2.102:0/744495068 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:40:39.264 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:39.259 7f1d35ab4700 1 --2- 172.21.2.102:0/744495068 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1d30094650 0x7f1d30094a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:39.264 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:39.259 7f1d34ab2700 1 --2- 172.21.2.102:0/744495068 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1d301970d0 0x7f1d301974d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:39.264 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:39.259 7f1d352b3700 1 -- 172.21.2.102:0/744495068 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1d30094650 msgr2=0x7f1d30094a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:39.265 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:39.259 7f1d352b3700 1 --2- 172.21.2.102:0/744495068 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1d30094650 0x7f1d30094a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:39.265 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:39.259 7f1d352b3700 1 -- 172.21.2.102:0/744495068 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1d301970d0 msgr2=0x7f1d301974d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:39.265 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:39.259 7f1d352b3700 1 --2- 172.21.2.102:0/744495068 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1d301970d0 0x7f1d301974d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:39.265 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:39.259 7f1d352b3700 1 -- 172.21.2.102:0/744495068 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d30095070 con 0x7f1d3018f0a0 2021-02-10T11:40:39.265 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:39.259 7f1d34ab2700 1 --2- 172.21.2.102:0/744495068 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1d301970d0 0x7f1d301974d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:40:39.265 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:39.259 7f1d352b3700 1 --2- 172.21.2.102:0/744495068 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1d3018f0a0 0x7f1d30196ab0 secure :-1 s=READY pgs=3406 cs=0 l=1 rev1=1 rx=0x7f1d2c001c60 tx=0x7f1d2c00a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:39.265 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:39.260 7f1d21ffb700 1 -- 172.21.2.102:0/744495068 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1d2c00acc0 con 0x7f1d3018f0a0 2021-02-10T11:40:39.265 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:39.260 7f1d21ffb700 1 -- 172.21.2.102:0/744495068 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d2c004140 con 0x7f1d3018f0a0 2021-02-10T11:40:39.265 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:39.260 7f1d21ffb700 1 -- 172.21.2.102:0/744495068 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1d2c004660 con 0x7f1d3018f0a0 2021-02-10T11:40:39.265 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:39.260 7f1d37497700 1 -- 172.21.2.102:0/744495068 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d30095070 con 0x7f1d3018f0a0 2021-02-10T11:40:39.265 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:39.260 7f1d37497700 1 -- 172.21.2.102:0/744495068 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d30095600 con 0x7f1d3018f0a0 2021-02-10T11:40:39.265 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:39.260 7f1d21ffb700 1 -- 172.21.2.102:0/744495068 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1d2c004320 con 0x7f1d3018f0a0 2021-02-10T11:40:39.265 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:39.260 7f1d21ffb700 1 --2- 172.21.2.102:0/744495068 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1d100186d0 0x7f1d1001abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:39.266 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:39.260 7f1d21ffb700 1 -- 172.21.2.102:0/744495068 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f1d2c0327b0 con 0x7f1d3018f0a0 2021-02-10T11:40:39.266 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:39.261 7f1d34ab2700 1 --2- 172.21.2.102:0/744495068 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1d100186d0 0x7f1d1001abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:39.266 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:39.261 7f1d34ab2700 1 --2- 172.21.2.102:0/744495068 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1d100186d0 0x7f1d1001abd0 crc :-1 s=READY pgs=1484 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:39.294 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:39.291 7f1d37497700 1 -- 172.21.2.102:0/744495068 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d300ad790 con 0x7f1d3018f0a0 2021-02-10T11:40:39.296 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:39.294 7f1d21ffb700 1 -- 172.21.2.102:0/744495068 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f1d2c000960 con 0x7f1d3018f0a0 2021-02-10T11:40:39.419 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:39.417 7f1d37497700 1 -- 172.21.2.102:0/744495068 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1d300ad790 con 0x7f1d3018f0a0 2021-02-10T11:40:39.420 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:39.418 7f1d21ffb700 1 -- 172.21.2.102:0/744495068 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v197) v1 ==== 74+0+14007 (secure 0 0 0) 0x7f1d2c004140 con 0x7f1d3018f0a0 2021-02-10T11:40:39.421 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:40:39.421 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":197,"fsid":"68db30ea-dbce-4aef-a6df-f01c6fea0513","created":"2021-02-10 11:03:06.331557","modified":"2021-02-10 11:40:25.367346","last_up_change":"2021-02-10 11:03:32.690562","last_in_change":"2021-02-10 11:03:32.690562","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":43,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:03:37.322519","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":42,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:40:22.330567","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"196","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":43,"pool_name":"cephfs_data","create_time":"2021-02-10 11:40:23.357509","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"197","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"be691b4d-a039-408a-af80-9d53f02b6f1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":185,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6818","nonce":36088},{"type":"v1","addr":"172.21.2.102:6820","nonce":36088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6822","nonce":36088},{"type":"v1","addr":"172.21.2.102:6824","nonce":36088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6830","nonce":36088},{"type":"v1","addr":"172.21.2.102:6832","nonce":36088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6826","nonce":36088},{"type":"v1","addr":"172.21.2.102:6828","nonce":36088}]},"public_addr":"172.21.2.102:6820/36088","cluster_addr":"172.21.2.102:6824/36088","heartbeat_back_addr":"172.21.2.102:6832/36088","heartbeat_front_addr":"172.21.2.102:6828/36088","state":["exists","up"]},{"osd":1,"uuid":"64e3ccc8-9c69-4d7c-865e-af8e83822635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6819","nonce":36087},{"type":"v1","addr":"172.21.2.102:6821","nonce":36087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6823","nonce":36087},{"type":"v1","addr":"172.21.2.102:6825","nonce":36087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6831","nonce":36087},{"type":"v1","addr":"172.21.2.102:6833","nonce":36087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6827","nonce":36087},{"type":"v1","addr":"172.21.2.102:6829","nonce":36087}]},"public_addr":"172.21.2.102:6821/36087","cluster_addr":"172.21.2.102:6825/36087","heartbeat_back_addr":"172.21.2.102:6833/36087","heartbeat_front_addr":"172.21.2.102:6829/36087","state":["exists","up"]},{"osd":2,"uuid":"153700af-747f-481f-bdf4-d6072d3a842f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6810","nonce":36089},{"type":"v1","addr":"172.21.2.102:6811","nonce":36089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6812","nonce":36089},{"type":"v1","addr":"172.21.2.102:6813","nonce":36089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6816","nonce":36089},{"type":"v1","addr":"172.21.2.102:6817","nonce":36089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6814","nonce":36089},{"type":"v1","addr":"172.21.2.102:6815","nonce":36089}]},"public_addr":"172.21.2.102:6811/36089","cluster_addr":"172.21.2.102:6813/36089","heartbeat_back_addr":"172.21.2.102:6817/36089","heartbeat_front_addr":"172.21.2.102:6815/36089","state":["exists","up"]},{"osd":3,"uuid":"cbc678f4-83a9-45e6-8830-2f7c232b1ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6802","nonce":36090},{"type":"v1","addr":"172.21.2.102:6803","nonce":36090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6804","nonce":36090},{"type":"v1","addr":"172.21.2.102:6805","nonce":36090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6808","nonce":36090},{"type":"v1","addr":"172.21.2.102:6809","nonce":36090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6806","nonce":36090},{"type":"v1","addr":"172.21.2.102:6807","nonce":36090}]},"public_addr":"172.21.2.102:6803/36090","cluster_addr":"172.21.2.102:6805/36090","heartbeat_back_addr":"172.21.2.102:6809/36090","heartbeat_front_addr":"172.21.2.102:6807/36090","state":["exists","up"]},{"osd":4,"uuid":"c01e4008-b258-4f10-afd2-9e7f37214351","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6808","nonce":36368},{"type":"v1","addr":"172.21.2.113:6811","nonce":36368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6814","nonce":36368},{"type":"v1","addr":"172.21.2.113:6818","nonce":36368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6826","nonce":36368},{"type":"v1","addr":"172.21.2.113:6828","nonce":36368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6820","nonce":36368},{"type":"v1","addr":"172.21.2.113:6824","nonce":36368}]},"public_addr":"172.21.2.113:6811/36368","cluster_addr":"172.21.2.113:6818/36368","heartbeat_back_addr":"172.21.2.113:6828/36368","heartbeat_front_addr":"172.21.2.113:6824/36368","state":["exists","up"]},{"osd":5,"uuid":"551f6007-daf8-4a98-8ee0-c2b16bcccb5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6802","nonce":36366},{"type":"v1","addr":"172.21.2.113:6805","nonce":36366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6807","nonce":36366},{"type":"v1","addr":"172.21.2.113:6810","nonce":36366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6819","nonce":36366},{"type":"v1","addr":"172.21.2.113:6822","nonce":36366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6812","nonce":36366},{"type":"v1","addr":"172.21.2.113:6815","nonce":36366}]},"public_addr":"172.21.2.113:6805/36366","cluster_addr":"172.21.2.113:6810/36366","heartbeat_back_addr":"172.21.2.113:6822/36366","heartbeat_front_addr":"172.21.2.113:6815/36366","state":["exists","up"]},{"osd":6,"uuid":"bf67c040-0e50-48be-b102-c3943bc9ebef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6800","nonce":36367},{"type":"v1","addr":"172.21.2.113:6801","nonce":36367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6803","nonce":36367},{"type":"v1","addr":"172.21.2.113:6804","nonce":36367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6813","nonce":36367},{"type":"v1","addr":"172.21.2.113:6816","nonce":36367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6806","nonce":36367},{"type":"v1","addr":"172.21.2.113:6809","nonce":36367}]},"public_addr":"172.21.2.113:6801/36367","cluster_addr":"172.21.2.113:6804/36367","heartbeat_back_addr":"172.21.2.113:6816/36367","heartbeat_front_addr":"172.21.2.113:6809/36367","state":["exists","up"]},{"osd":7,"uuid":"f5461f80-018d-428f-be89-2eb2bcbe7bb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6817","nonce":36365},{"type":"v1","addr":"172.21.2.113:6821","nonce":36365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6823","nonce":36365},{"type":"v1","addr":"172.21.2.113:6825","nonce":36365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6830","nonce":36365},{"type":"v1","addr":"172.21.2.113:6831","nonce":36365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6827","nonce":36365},{"type":"v1","addr":"172.21.2.113:6829","nonce":36365}]},"public_addr":"172.21.2.113:6821/36365","cluster_addr":"172.21.2.113:6825/36365","heartbeat_back_addr":"172.21.2.113:6831/36365","heartbeat_front_addr":"172.21.2.113:6829/36365","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:40:39.421 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:39.419 7f1d37497700 1 -- 172.21.2.102:0/744495068 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1d100186d0 msgr2=0x7f1d1001abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:39.422 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:39.419 7f1d37497700 1 --2- 172.21.2.102:0/744495068 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1d100186d0 0x7f1d1001abd0 crc :-1 s=READY pgs=1484 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:39.422 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:39.419 7f1d37497700 1 -- 172.21.2.102:0/744495068 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1d3018f0a0 msgr2=0x7f1d30196ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:39.422 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:39.419 7f1d37497700 1 --2- 172.21.2.102:0/744495068 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1d3018f0a0 0x7f1d30196ab0 secure :-1 s=READY pgs=3406 cs=0 l=1 rev1=1 rx=0x7f1d2c001c60 tx=0x7f1d2c00a8d0).stop 2021-02-10T11:40:39.422 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:39.420 7f1d37497700 1 -- 172.21.2.102:0/744495068 shutdown_connections 2021-02-10T11:40:39.422 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:39.420 7f1d37497700 1 --2- 172.21.2.102:0/744495068 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1d30094650 0x7f1d30094a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:39.423 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:39.420 7f1d37497700 1 --2- 172.21.2.102:0/744495068 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1d100186d0 0x7f1d1001abd0 unknown :-1 s=CLOSED pgs=1484 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:39.423 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:39.420 7f1d37497700 1 --2- 172.21.2.102:0/744495068 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1d301970d0 0x7f1d301974d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:39.423 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:39.420 7f1d37497700 1 --2- 172.21.2.102:0/744495068 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1d3018f0a0 0x7f1d30196ab0 unknown :-1 s=CLOSED pgs=3406 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:39.423 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:39.420 7f1d37497700 1 -- 172.21.2.102:0/744495068 >> 172.21.2.102:0/744495068 conn(0x7f1d301a38c0 msgr2=0x7f1d301a2f00 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:39.423 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:39.420 7f1d37497700 1 -- 172.21.2.102:0/744495068 shutdown_connections 2021-02-10T11:40:39.423 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:39.420 7f1d37497700 1 -- 172.21.2.102:0/744495068 wait complete. 2021-02-10T11:40:39.431 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:03:37.322519', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 42, 'pool_name': 'cephfs_metadata', 'create_time': '2021-02-10 11:40:22.330567', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '196', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 43, 'pool_name': 'cephfs_data', 'create_time': '2021-02-10 11:40:23.357509', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '197', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-02-10T11:40:39.431 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:40:39.864 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:39.845 7ff95d614700 1 Processor -- start 2021-02-10T11:40:39.865 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:39.845 7ff95d614700 1 -- start start 2021-02-10T11:40:39.865 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:39.845 7ff95d614700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7ff95810aff0 0x7ff9580cf750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:39.865 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:39.845 7ff95d614700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9580fe1c0 con 0x7ff9580d1870 2021-02-10T11:40:39.865 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:39.845 7ff95d614700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9581b0cb0 con 0x7ff9580d1c70 2021-02-10T11:40:39.865 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:39.845 7ff95d614700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff958109a60 con 0x7ff95810aff0 2021-02-10T11:40:39.865 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:39.846 7ff9572bf700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7ff9580d1870 0x7ff9581b04c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:33662/0 (socket says 172.21.2.102:33662) 2021-02-10T11:40:39.865 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:39.846 7ff9572bf700 1 -- 172.21.2.102:0/2891027468 learned_addr learned my addr 172.21.2.102:0/2891027468 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:40:39.865 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:39.846 7ff957ac0700 1 --2- 172.21.2.102:0/2891027468 >> v2:172.21.2.113:3300/0 conn(0x7ff95810aff0 0x7ff9580cf750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:39.865 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:39.846 7ff9562bd700 1 -- 172.21.2.102:0/2891027468 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3485625481 0 0) 0x7ff940001180 con 0x7ff9580d1870 2021-02-10T11:40:39.865 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:39.846 7ff9562bd700 1 -- 172.21.2.102:0/2891027468 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff944002fe0 con 0x7ff9580d1870 2021-02-10T11:40:39.866 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:39.846 7ff9562bd700 1 -- 172.21.2.102:0/2891027468 <== mon.1 v1:172.21.2.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 492904054 0 0) 0x7ff948001080 con 0x7ff9580d1c70 2021-02-10T11:40:39.866 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:39.846 7ff9562bd700 1 -- 172.21.2.102:0/2891027468 --> v1:172.21.2.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff944003210 con 0x7ff9580d1c70 2021-02-10T11:40:39.866 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:39.846 7ff9562bd700 1 -- 172.21.2.102:0/2891027468 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2714421396 0 0) 0x7ff940001180 con 0x7ff9580d1870 2021-02-10T11:40:39.866 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:39.846 7ff9562bd700 1 -- 172.21.2.102:0/2891027468 >> v2:172.21.2.113:3300/0 conn(0x7ff95810aff0 msgr2=0x7ff9580cf750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:39.866 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:39.846 7ff9562bd700 1 --2- 172.21.2.102:0/2891027468 >> v2:172.21.2.113:3300/0 conn(0x7ff95810aff0 0x7ff9580cf750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:39.866 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:39.846 7ff9562bd700 1 -- 172.21.2.102:0/2891027468 >> v1:172.21.2.113:6789/0 conn(0x7ff9580d1c70 legacy=0x7ff9580cef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:39.866 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:39.846 7ff9562bd700 1 -- 172.21.2.102:0/2891027468 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff95810b3f0 con 0x7ff9580d1870 2021-02-10T11:40:39.866 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:39.846 7ff9562bd700 1 -- 172.21.2.102:0/2891027468 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7ff940001680 con 0x7ff9580d1870 2021-02-10T11:40:39.866 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:39.846 7ff9562bd700 1 -- 172.21.2.102:0/2891027468 >> v1:172.21.2.102:6789/0 conn(0x7ff9580d1870 legacy=0x7ff9581b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:39.866 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:39.846 7ff9562bd700 1 --2- 172.21.2.102:0/2891027468 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff944003b90 0x7ff944005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:39.866 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:39.846 7ff9562bd700 1 --2- 172.21.2.102:0/2891027468 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff9440067a0 0x7ff944008cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:39.866 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:39.847 7ff9562bd700 1 --2- 172.21.2.102:0/2891027468 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff9440095a0 0x7ff94400bac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:39.867 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:39.847 7ff9562bd700 1 -- 172.21.2.102:0/2891027468 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff944050320 con 0x7ff9440095a0 2021-02-10T11:40:39.867 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:39.847 7ff9562bd700 1 -- 172.21.2.102:0/2891027468 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff944050500 con 0x7ff944003b90 2021-02-10T11:40:39.867 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:39.847 7ff9562bd700 1 -- 172.21.2.102:0/2891027468 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff944050720 con 0x7ff9440067a0 2021-02-10T11:40:39.867 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:39.847 7ff957ac0700 1 --2- 172.21.2.102:0/2891027468 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff9440095a0 0x7ff94400bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:39.867 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:39.847 7ff956abe700 1 --2- 172.21.2.102:0/2891027468 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff9440067a0 0x7ff944008cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:39.867 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:39.847 7ff9572bf700 1 --2- 172.21.2.102:0/2891027468 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff944003b90 0x7ff944005fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:39.867 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:39.847 7ff957ac0700 1 -- 172.21.2.102:0/2891027468 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff9440067a0 msgr2=0x7ff944008cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:39.867 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:39.847 7ff957ac0700 1 --2- 172.21.2.102:0/2891027468 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff9440067a0 0x7ff944008cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:39.867 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:39.847 7ff957ac0700 1 -- 172.21.2.102:0/2891027468 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff944003b90 msgr2=0x7ff944005fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:39.867 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:39.847 7ff957ac0700 1 --2- 172.21.2.102:0/2891027468 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff944003b90 0x7ff944005fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:39.867 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:39.847 7ff957ac0700 1 -- 172.21.2.102:0/2891027468 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9440509c0 con 0x7ff9440095a0 2021-02-10T11:40:39.867 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:39.847 7ff957ac0700 1 -- 172.21.2.102:0/2891027468 reap_dead start 2021-02-10T11:40:39.867 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:39.847 7ff957ac0700 1 --2- 172.21.2.102:0/2891027468 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff9440095a0 0x7ff94400bac0 secure :-1 s=READY pgs=3408 cs=0 l=1 rev1=1 rx=0x7ff950006a40 tx=0x7ff95000c0c0).ready entity=mon.0 client_cookie=d51b1c8ff9775cb5 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:39.868 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:39.847 7ff9562bd700 1 -- 172.21.2.102:0/2891027468 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff950006a80 con 0x7ff9440095a0 2021-02-10T11:40:39.868 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:39.847 7ff9562bd700 1 -- 172.21.2.102:0/2891027468 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff950007070 con 0x7ff9440095a0 2021-02-10T11:40:39.868 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:39.847 7ff9562bd700 1 -- 172.21.2.102:0/2891027468 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff950007590 con 0x7ff9440095a0 2021-02-10T11:40:39.868 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:39.847 7ff95d614700 1 -- 172.21.2.102:0/2891027468 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff9440095a0 msgr2=0x7ff94400bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:39.868 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:39.847 7ff95d614700 1 --2- 172.21.2.102:0/2891027468 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff9440095a0 0x7ff94400bac0 secure :-1 s=READY pgs=3408 cs=0 l=1 rev1=1 rx=0x7ff950006a40 tx=0x7ff95000c0c0).stop 2021-02-10T11:40:39.868 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:39.848 7ff95d614700 1 -- 172.21.2.102:0/2891027468 shutdown_connections 2021-02-10T11:40:39.868 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:39.848 7ff95d614700 1 --2- 172.21.2.102:0/2891027468 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff9440095a0 0x7ff94400bac0 unknown :-1 s=CLOSED pgs=3408 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:39.868 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:39.848 7ff95d614700 1 -- 172.21.2.102:0/2891027468 >> 172.21.2.102:0/2891027468 conn(0x7ff9580ce420 msgr2=0x7ff9580d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:39.868 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:39.848 7ff95d614700 1 -- 172.21.2.102:0/2891027468 shutdown_connections 2021-02-10T11:40:39.868 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:39.848 7ff95d614700 1 -- 172.21.2.102:0/2891027468 wait complete. 2021-02-10T11:40:39.870 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:39.863 7ff95d614700 1 Processor -- start 2021-02-10T11:40:39.870 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:39.863 7ff95d614700 1 -- start start 2021-02-10T11:40:39.870 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:39.863 7ff95d614700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff958085bc0 0x7ff9580863e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:39.870 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:39.863 7ff95d614700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff9580954c0 0x7ff958096300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:39.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:39.863 7ff95d614700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff9580958c0 0x7ff958096920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:39.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:39.863 7ff95d614700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff958095d80 con 0x7ff9580958c0 2021-02-10T11:40:39.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:39.863 7ff95d614700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9580bfa60 con 0x7ff958085bc0 2021-02-10T11:40:39.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:39.863 7ff95d614700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff958085fc0 con 0x7ff9580954c0 2021-02-10T11:40:39.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:39.863 7ff957ac0700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff9580958c0 0x7ff958096920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:39.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:39.863 7ff957ac0700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff9580958c0 0x7ff958096920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:41448/0 (socket says 172.21.2.102:41448) 2021-02-10T11:40:39.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:39.863 7ff957ac0700 1 -- 172.21.2.102:0/3028832045 learned_addr learned my addr 172.21.2.102:0/3028832045 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:40:39.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:39.864 7ff9572bf700 1 --2- 172.21.2.102:0/3028832045 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff958085bc0 0x7ff9580863e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:39.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:39.864 7ff956abe700 1 --2- 172.21.2.102:0/3028832045 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff9580954c0 0x7ff958096300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:39.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:39.864 7ff957ac0700 1 -- 172.21.2.102:0/3028832045 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff9580954c0 msgr2=0x7ff958096300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:39.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:39.864 7ff957ac0700 1 --2- 172.21.2.102:0/3028832045 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff9580954c0 0x7ff958096300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:39.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:39.864 7ff957ac0700 1 -- 172.21.2.102:0/3028832045 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff958085bc0 msgr2=0x7ff9580863e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:39.871 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:39.864 7ff957ac0700 1 --2- 172.21.2.102:0/3028832045 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff958085bc0 0x7ff9580863e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:39.872 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:39.864 7ff957ac0700 1 -- 172.21.2.102:0/3028832045 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff958086c10 con 0x7ff9580958c0 2021-02-10T11:40:39.872 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:39.864 7ff956abe700 1 --2- 172.21.2.102:0/3028832045 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff9580954c0 0x7ff958096300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:40:39.872 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:39.864 7ff957ac0700 1 --2- 172.21.2.102:0/3028832045 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff9580958c0 0x7ff958096920 secure :-1 s=READY pgs=3409 cs=0 l=1 rev1=1 rx=0x7ff9500099c0 tx=0x7ff95000bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:39.872 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:39.864 7ff94ffff700 1 -- 172.21.2.102:0/3028832045 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff95000c2f0 con 0x7ff9580958c0 2021-02-10T11:40:39.872 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:39.864 7ff95d614700 1 -- 172.21.2.102:0/3028832045 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff958086c10 con 0x7ff9580958c0 2021-02-10T11:40:39.872 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:39.864 7ff95d614700 1 -- 172.21.2.102:0/3028832045 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff958086c10 con 0x7ff9580958c0 2021-02-10T11:40:39.872 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:39.864 7ff94ffff700 1 -- 172.21.2.102:0/3028832045 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff95000c2f0 con 0x7ff9580958c0 2021-02-10T11:40:39.872 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:39.864 7ff94ffff700 1 -- 172.21.2.102:0/3028832045 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff95000c5b0 con 0x7ff9580958c0 2021-02-10T11:40:39.872 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:39.865 7ff94ffff700 1 -- 172.21.2.102:0/3028832045 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ff9500056b0 con 0x7ff9580958c0 2021-02-10T11:40:39.872 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:39.865 7ff94ffff700 1 --2- 172.21.2.102:0/3028832045 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff93c0186d0 0x7ff93c01abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:39.872 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:39.865 7ff94ffff700 1 -- 172.21.2.102:0/3028832045 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 5573+0+0 (secure 0 0 0) 0x7ff950032b30 con 0x7ff9580958c0 2021-02-10T11:40:39.872 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:39.865 7ff9572bf700 1 --2- 172.21.2.102:0/3028832045 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff93c0186d0 0x7ff93c01abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:39.872 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:39.865 7ff9572bf700 1 --2- 172.21.2.102:0/3028832045 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff93c0186d0 0x7ff93c01abd0 crc :-1 s=READY pgs=1485 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:39.897 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:39.895 7ff95d614700 1 -- 172.21.2.102:0/3028832045 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9580fe1c0 con 0x7ff9580958c0 2021-02-10T11:40:39.900 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:39.898 7ff94ffff700 1 -- 172.21.2.102:0/3028832045 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ff95000be30 con 0x7ff9580958c0 2021-02-10T11:40:40.026 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:40.024 7ff95d614700 1 -- 172.21.2.102:0/3028832045 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7ff9580fe1c0 con 0x7ff9580958c0 2021-02-10T11:40:40.027 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:40.024 7ff94ffff700 1 -- 172.21.2.102:0/3028832045 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v197) v1 ==== 92+0+10 (secure 0 0 0) 0x7ff95001f850 con 0x7ff9580958c0 2021-02-10T11:40:40.027 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:40:40.029 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:40.026 7ff95d614700 1 -- 172.21.2.102:0/3028832045 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff93c0186d0 msgr2=0x7ff93c01abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:40.029 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:40.026 7ff95d614700 1 --2- 172.21.2.102:0/3028832045 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff93c0186d0 0x7ff93c01abd0 crc :-1 s=READY pgs=1485 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:40.029 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:40.026 7ff95d614700 1 -- 172.21.2.102:0/3028832045 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff9580958c0 msgr2=0x7ff958096920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:40.030 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:40.026 7ff95d614700 1 --2- 172.21.2.102:0/3028832045 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff9580958c0 0x7ff958096920 secure :-1 s=READY pgs=3409 cs=0 l=1 rev1=1 rx=0x7ff9500099c0 tx=0x7ff95000bcf0).stop 2021-02-10T11:40:40.030 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:40.026 7ff95d614700 1 -- 172.21.2.102:0/3028832045 shutdown_connections 2021-02-10T11:40:40.030 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:40.026 7ff95d614700 1 --2- 172.21.2.102:0/3028832045 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff9580958c0 0x7ff958096920 unknown :-1 s=CLOSED pgs=3409 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:40.030 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:40.026 7ff95d614700 1 --2- 172.21.2.102:0/3028832045 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff9580954c0 0x7ff958096300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:40.030 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:40.026 7ff95d614700 1 --2- 172.21.2.102:0/3028832045 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff93c0186d0 0x7ff93c01abd0 unknown :-1 s=CLOSED pgs=1485 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:40.030 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:40.027 7ff95d614700 1 --2- 172.21.2.102:0/3028832045 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff958085bc0 0x7ff9580863e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:40.030 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:40.027 7ff95d614700 1 -- 172.21.2.102:0/3028832045 >> 172.21.2.102:0/3028832045 conn(0x7ff9580bef00 msgr2=0x7ff95809d1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:40.030 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:40.027 7ff95d614700 1 -- 172.21.2.102:0/3028832045 shutdown_connections 2021-02-10T11:40:40.030 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:40.027 7ff95d614700 1 -- 172.21.2.102:0/3028832045 wait complete. 2021-02-10T11:40:40.037 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-02-10T11:40:40.469 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:40.451 7fa8ada55700 1 Processor -- start 2021-02-10T11:40:40.470 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:40.451 7fa8ada55700 1 -- start start 2021-02-10T11:40:40.470 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:40.452 7fa8ada55700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa8a80d1870 0x7fa8a81b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:40.470 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:40.452 7fa8ada55700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fa8a80cff70 0x7fa8a8109130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:40.470 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:40.452 7fa8ada55700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8a8109a60 con 0x7fa8a81b0ae0 2021-02-10T11:40:40.470 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:40.452 7fa8ada55700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8a8109750 con 0x7fa8a80cff70 2021-02-10T11:40:40.470 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:40.452 7fa8ada55700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8a8088070 con 0x7fa8a80d1870 2021-02-10T11:40:40.470 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:40.452 7fa8a7fff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fa8a80cff70 0x7fa8a8109130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:40.470 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:40.452 7fa8a7fff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fa8a80cff70 0x7fa8a8109130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:41456/0 (socket says 172.21.2.102:41456) 2021-02-10T11:40:40.470 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:40.452 7fa8a7fff700 1 -- 172.21.2.102:0/53442069 learned_addr learned my addr 172.21.2.102:0/53442069 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:40:40.470 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:40.452 7fa8a7fff700 1 -- 172.21.2.102:0/53442069 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa8a80d1870 msgr2=0x7fa8a81b04c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:40:40.470 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:40.452 7fa8a7fff700 1 --2- 172.21.2.102:0/53442069 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa8a80d1870 0x7fa8a81b04c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:40.471 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:40.452 7fa8a7fff700 1 -- 172.21.2.102:0/53442069 >> v1:172.21.2.113:6789/0 conn(0x7fa8a81b0ae0 legacy=0x7fa8a8108940 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:40:40.471 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:40.452 7fa8a7fff700 1 -- 172.21.2.102:0/53442069 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8a8088260 con 0x7fa8a80cff70 2021-02-10T11:40:40.471 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:40.452 7fa8a7fff700 1 --2- 172.21.2.102:0/53442069 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa8a80cff70 0x7fa8a8109130 secure :-1 s=READY pgs=3410 cs=0 l=1 rev1=1 rx=0x7fa8a0006f00 tx=0x7fa8a0009f10).ready entity=mon.0 client_cookie=b5a122297511e3fe server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:40.471 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:40.452 7fa8a67fc700 1 -- 172.21.2.102:0/53442069 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa8a000b3b0 con 0x7fa8a80cff70 2021-02-10T11:40:40.471 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:40.452 7fa8a67fc700 1 -- 172.21.2.102:0/53442069 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8a000b630 con 0x7fa8a80cff70 2021-02-10T11:40:40.471 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:40.452 7fa8a67fc700 1 -- 172.21.2.102:0/53442069 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa8a000bb10 con 0x7fa8a80cff70 2021-02-10T11:40:40.471 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:40.452 7fa8ada55700 1 -- 172.21.2.102:0/53442069 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa8a80cff70 msgr2=0x7fa8a8109130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:40.471 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:40.452 7fa8ada55700 1 --2- 172.21.2.102:0/53442069 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa8a80cff70 0x7fa8a8109130 secure :-1 s=READY pgs=3410 cs=0 l=1 rev1=1 rx=0x7fa8a0006f00 tx=0x7fa8a0009f10).stop 2021-02-10T11:40:40.471 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:40.453 7fa8ada55700 1 -- 172.21.2.102:0/53442069 shutdown_connections 2021-02-10T11:40:40.471 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:40.453 7fa8ada55700 1 --2- 172.21.2.102:0/53442069 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa8a80cff70 0x7fa8a8109130 unknown :-1 s=CLOSED pgs=3410 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:40.471 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:40.453 7fa8ada55700 1 --2- 172.21.2.102:0/53442069 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa8a80d1870 0x7fa8a81b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:40.471 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:40.453 7fa8ada55700 1 -- 172.21.2.102:0/53442069 >> 172.21.2.102:0/53442069 conn(0x7fa8a80ce420 msgr2=0x7fa8a80d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:40.472 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:40.453 7fa8ada55700 1 -- 172.21.2.102:0/53442069 shutdown_connections 2021-02-10T11:40:40.472 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:40.453 7fa8ada55700 1 -- 172.21.2.102:0/53442069 wait complete. 2021-02-10T11:40:40.473 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:40.468 7fa8ada55700 1 Processor -- start 2021-02-10T11:40:40.473 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:40.468 7fa8ada55700 1 -- start start 2021-02-10T11:40:40.473 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:40.468 7fa8ada55700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa8a80c3820 0x7fa8a80a7950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:40.473 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:40.468 7fa8ada55700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa8a81a44c0 0x7fa8a81a48c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:40.473 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:40.468 7fa8ada55700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa8a80863d0 0x7fa8a80867d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:40.474 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:40.468 7fa8ada55700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8a8085ed0 con 0x7fa8a80c3820 2021-02-10T11:40:40.474 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:40.468 7fa8ada55700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8a80c0740 con 0x7fa8a80863d0 2021-02-10T11:40:40.474 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:40.468 7fa8ada55700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8a8086df0 con 0x7fa8a81a44c0 2021-02-10T11:40:40.474 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:40.468 7fa8a77fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa8a80c3820 0x7fa8a80a7950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:40.474 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:40.468 7fa8a77fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa8a80c3820 0x7fa8a80a7950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:41462/0 (socket says 172.21.2.102:41462) 2021-02-10T11:40:40.474 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:40.468 7fa8a77fe700 1 -- 172.21.2.102:0/2077021414 learned_addr learned my addr 172.21.2.102:0/2077021414 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:40:40.474 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:40.469 7fa8a77fe700 1 -- 172.21.2.102:0/2077021414 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa8a81a44c0 msgr2=0x7fa8a81a48c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:40.474 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:40.469 7fa8a77fe700 1 --2- 172.21.2.102:0/2077021414 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa8a81a44c0 0x7fa8a81a48c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:40.474 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:40.469 7fa8a77fe700 1 -- 172.21.2.102:0/2077021414 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa8a80863d0 msgr2=0x7fa8a80867d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:40.474 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:40.469 7fa8a7fff700 1 --2- 172.21.2.102:0/2077021414 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa8a80863d0 0x7fa8a80867d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:40.474 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:40.469 7fa8a77fe700 1 --2- 172.21.2.102:0/2077021414 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa8a80863d0 0x7fa8a80867d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:40.474 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:40.469 7fa8a77fe700 1 -- 172.21.2.102:0/2077021414 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8a8087010 con 0x7fa8a80c3820 2021-02-10T11:40:40.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:40.469 7fa8a7fff700 1 --2- 172.21.2.102:0/2077021414 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa8a80863d0 0x7fa8a80867d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:40:40.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:40.469 7fa8a77fe700 1 --2- 172.21.2.102:0/2077021414 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa8a80c3820 0x7fa8a80a7950 secure :-1 s=READY pgs=3411 cs=0 l=1 rev1=1 rx=0x7fa89800b560 tx=0x7fa89800b900).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:40.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:40.469 7fa89ffff700 1 -- 172.21.2.102:0/2077021414 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa89800bcf0 con 0x7fa8a80c3820 2021-02-10T11:40:40.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:40.469 7fa89ffff700 1 -- 172.21.2.102:0/2077021414 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa898006030 con 0x7fa8a80c3820 2021-02-10T11:40:40.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:40.469 7fa8ada55700 1 -- 172.21.2.102:0/2077021414 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8a8087010 con 0x7fa8a80c3820 2021-02-10T11:40:40.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:40.469 7fa89ffff700 1 -- 172.21.2.102:0/2077021414 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa8980054d0 con 0x7fa8a80c3820 2021-02-10T11:40:40.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:40.469 7fa8ada55700 1 -- 172.21.2.102:0/2077021414 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8a8087010 con 0x7fa8a80c3820 2021-02-10T11:40:40.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:40.470 7fa89ffff700 1 -- 172.21.2.102:0/2077021414 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa898006210 con 0x7fa8a80c3820 2021-02-10T11:40:40.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:40.470 7fa89ffff700 1 --2- 172.21.2.102:0/2077021414 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa88c018730 0x7fa88c01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:40.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:40.470 7fa89ffff700 1 -- 172.21.2.102:0/2077021414 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fa8980327d0 con 0x7fa8a80c3820 2021-02-10T11:40:40.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:40.470 7fa8a6ffd700 1 --2- 172.21.2.102:0/2077021414 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa88c018730 0x7fa88c01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:40.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:40.470 7fa8a6ffd700 1 --2- 172.21.2.102:0/2077021414 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa88c018730 0x7fa88c01ac10 crc :-1 s=READY pgs=1486 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:40.504 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:40.502 7fa8ada55700 1 -- 172.21.2.102:0/2077021414 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8a80fe1c0 con 0x7fa8a80c3820 2021-02-10T11:40:40.506 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:40.504 7fa89ffff700 1 -- 172.21.2.102:0/2077021414 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fa898006b80 con 0x7fa8a80c3820 2021-02-10T11:40:40.635 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:40.633 7fa8ada55700 1 -- 172.21.2.102:0/2077021414 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7fa8a80fe1c0 con 0x7fa8a80c3820 2021-02-10T11:40:40.636 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:40.633 7fa89ffff700 1 -- 172.21.2.102:0/2077021414 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v197) v1 ==== 104+0+10 (secure 0 0 0) 0x7fa898006b80 con 0x7fa8a80c3820 2021-02-10T11:40:40.636 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:40:40.637 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:40.635 7fa8ada55700 1 -- 172.21.2.102:0/2077021414 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa88c018730 msgr2=0x7fa88c01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:40.637 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:40.635 7fa8ada55700 1 --2- 172.21.2.102:0/2077021414 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa88c018730 0x7fa88c01ac10 crc :-1 s=READY pgs=1486 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:40.637 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:40.635 7fa8ada55700 1 -- 172.21.2.102:0/2077021414 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa8a80c3820 msgr2=0x7fa8a80a7950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:40.637 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:40.635 7fa8ada55700 1 --2- 172.21.2.102:0/2077021414 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa8a80c3820 0x7fa8a80a7950 secure :-1 s=READY pgs=3411 cs=0 l=1 rev1=1 rx=0x7fa89800b560 tx=0x7fa89800b900).stop 2021-02-10T11:40:40.638 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:40.635 7fa8ada55700 1 -- 172.21.2.102:0/2077021414 shutdown_connections 2021-02-10T11:40:40.638 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:40.635 7fa8ada55700 1 --2- 172.21.2.102:0/2077021414 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa88c018730 0x7fa88c01ac10 unknown :-1 s=CLOSED pgs=1486 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:40.638 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:40.635 7fa8ada55700 1 --2- 172.21.2.102:0/2077021414 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa8a80863d0 0x7fa8a80867d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:40.638 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:40.635 7fa8ada55700 1 --2- 172.21.2.102:0/2077021414 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa8a81a44c0 0x7fa8a81a48c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:40.638 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:40.635 7fa8ada55700 1 --2- 172.21.2.102:0/2077021414 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa8a80c3820 0x7fa8a80a7950 unknown :-1 s=CLOSED pgs=3411 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:40.638 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:40.635 7fa8ada55700 1 -- 172.21.2.102:0/2077021414 >> 172.21.2.102:0/2077021414 conn(0x7fa8a80872c0 msgr2=0x7fa8a8095700 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:40.638 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:40.635 7fa8ada55700 1 -- 172.21.2.102:0/2077021414 shutdown_connections 2021-02-10T11:40:40.639 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:40.635 7fa8ada55700 1 -- 172.21.2.102:0/2077021414 wait complete. 2021-02-10T11:40:40.646 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-02-10T11:40:41.079 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:41.059 7fa80f5e5700 1 Processor -- start 2021-02-10T11:40:41.079 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:41.060 7fa80f5e5700 1 -- start start 2021-02-10T11:40:41.079 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:41.060 7fa80f5e5700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fa8080d1c70 0x7fa8080cef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:41.079 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:41.060 7fa80f5e5700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fa8080cf580 0x7fa8080cf980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:41.079 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:41.060 7fa80f5e5700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8080fe1c0 con 0x7fa8080d1870 2021-02-10T11:40:41.080 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:41.060 7fa80f5e5700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8081b0cb0 con 0x7fa8080d1c70 2021-02-10T11:40:41.080 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:41.060 7fa80f5e5700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa808109a60 con 0x7fa8080cf580 2021-02-10T11:40:41.080 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:41.060 7fa80cc00700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fa8080d1c70 0x7fa8080cef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:41.080 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:41.060 7fa80d401700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fa8080d1870 0x7fa8081b04c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:33692/0 (socket says 172.21.2.102:33692) 2021-02-10T11:40:41.080 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:41.060 7fa80d401700 1 -- 172.21.2.102:0/2456624548 learned_addr learned my addr 172.21.2.102:0/2456624548 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:40:41.080 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:41.061 7fa80cc00700 1 -- 172.21.2.102:0/2456624548 >> v2:172.21.2.113:3300/0 conn(0x7fa8080cf580 msgr2=0x7fa8080cf980 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:40:41.080 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:41.061 7fa80cc00700 1 --2- 172.21.2.102:0/2456624548 >> v2:172.21.2.113:3300/0 conn(0x7fa8080cf580 0x7fa8080cf980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:41.080 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:41.061 7fa80cc00700 1 -- 172.21.2.102:0/2456624548 >> v1:172.21.2.102:6789/0 conn(0x7fa8080d1870 legacy=0x7fa8081b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:41.080 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:41.061 7fa80cc00700 1 -- 172.21.2.102:0/2456624548 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa808088070 con 0x7fa8080d1c70 2021-02-10T11:40:41.080 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:41.061 7fa80cc00700 1 --2- 172.21.2.102:0/2456624548 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa8080d1c70 0x7fa8080cef60 secure :-1 s=READY pgs=3413 cs=0 l=1 rev1=1 rx=0x7fa8040069d0 tx=0x7fa804009dc0).ready entity=mon.0 client_cookie=796a607c88215e46 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:41.081 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:41.061 7fa7fffff700 1 -- 172.21.2.102:0/2456624548 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa80400b510 con 0x7fa8080d1c70 2021-02-10T11:40:41.081 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:41.061 7fa7fffff700 1 -- 172.21.2.102:0/2456624548 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa80400b790 con 0x7fa8080d1c70 2021-02-10T11:40:41.081 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:41.061 7fa7fffff700 1 -- 172.21.2.102:0/2456624548 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa80400bcb0 con 0x7fa8080d1c70 2021-02-10T11:40:41.081 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:41.061 7fa80f5e5700 1 -- 172.21.2.102:0/2456624548 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa8080d1c70 msgr2=0x7fa8080cef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:41.081 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:41.061 7fa80f5e5700 1 --2- 172.21.2.102:0/2456624548 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa8080d1c70 0x7fa8080cef60 secure :-1 s=READY pgs=3413 cs=0 l=1 rev1=1 rx=0x7fa8040069d0 tx=0x7fa804009dc0).stop 2021-02-10T11:40:41.081 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:41.062 7fa80f5e5700 1 -- 172.21.2.102:0/2456624548 shutdown_connections 2021-02-10T11:40:41.081 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:41.062 7fa80f5e5700 1 --2- 172.21.2.102:0/2456624548 >> v2:172.21.2.113:3300/0 conn(0x7fa8080cf580 0x7fa8080cf980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:41.081 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:41.062 7fa80f5e5700 1 --2- 172.21.2.102:0/2456624548 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa8080d1c70 0x7fa8080cef60 unknown :-1 s=CLOSED pgs=3413 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:41.081 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:41.062 7fa80f5e5700 1 -- 172.21.2.102:0/2456624548 >> 172.21.2.102:0/2456624548 conn(0x7fa8080ce420 msgr2=0x7fa8080d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:41.081 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:41.062 7fa80f5e5700 1 -- 172.21.2.102:0/2456624548 shutdown_connections 2021-02-10T11:40:41.081 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:41.062 7fa80f5e5700 1 -- 172.21.2.102:0/2456624548 wait complete. 2021-02-10T11:40:41.082 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:41.077 7fa80f5e5700 1 Processor -- start 2021-02-10T11:40:41.082 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:41.077 7fa80f5e5700 1 -- start start 2021-02-10T11:40:41.083 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:41.077 7fa80f5e5700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa8080c3820 0x7fa8080a7950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:41.083 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:41.077 7fa80f5e5700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa8081a44c0 0x7fa8081a48c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:41.083 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:41.077 7fa80f5e5700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa8080863d0 0x7fa8080867d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:41.083 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:41.077 7fa80f5e5700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa808085ed0 con 0x7fa8080863d0 2021-02-10T11:40:41.084 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:41.077 7fa80f5e5700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8080c0740 con 0x7fa8081a44c0 2021-02-10T11:40:41.084 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:41.077 7fa80f5e5700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa808086df0 con 0x7fa8080c3820 2021-02-10T11:40:41.084 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:41.077 7fa80dc02700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa8080863d0 0x7fa8080867d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:41.084 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:41.077 7fa80dc02700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa8080863d0 0x7fa8080867d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:41478/0 (socket says 172.21.2.102:41478) 2021-02-10T11:40:41.084 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:41.077 7fa80dc02700 1 -- 172.21.2.102:0/3323691904 learned_addr learned my addr 172.21.2.102:0/3323691904 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:40:41.084 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:41.078 7fa80dc02700 1 -- 172.21.2.102:0/3323691904 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa8080c3820 msgr2=0x7fa8080a7950 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:40:41.084 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:41.078 7fa80dc02700 1 --2- 172.21.2.102:0/3323691904 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa8080c3820 0x7fa8080a7950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:41.084 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:41.078 7fa80dc02700 1 -- 172.21.2.102:0/3323691904 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa8081a44c0 msgr2=0x7fa8081a48c0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:40:41.084 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:41.078 7fa80dc02700 1 --2- 172.21.2.102:0/3323691904 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa8081a44c0 0x7fa8081a48c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:41.084 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:41.078 7fa80dc02700 1 -- 172.21.2.102:0/3323691904 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa808087010 con 0x7fa8080863d0 2021-02-10T11:40:41.084 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:41.078 7fa80dc02700 1 --2- 172.21.2.102:0/3323691904 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa8080863d0 0x7fa8080867d0 secure :-1 s=READY pgs=3414 cs=0 l=1 rev1=1 rx=0x7fa800006ea0 tx=0x7fa80000b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:41.084 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:41.078 7fa7fdffb700 1 -- 172.21.2.102:0/3323691904 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa80000bd30 con 0x7fa8080863d0 2021-02-10T11:40:41.085 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:41.078 7fa7fdffb700 1 -- 172.21.2.102:0/3323691904 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa800001070 con 0x7fa8080863d0 2021-02-10T11:40:41.085 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:41.078 7fa7fdffb700 1 -- 172.21.2.102:0/3323691904 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa800001590 con 0x7fa8080863d0 2021-02-10T11:40:41.085 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:41.078 7fa80f5e5700 1 -- 172.21.2.102:0/3323691904 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa808087010 con 0x7fa8080863d0 2021-02-10T11:40:41.085 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:41.078 7fa80f5e5700 1 -- 172.21.2.102:0/3323691904 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa808087010 con 0x7fa8080863d0 2021-02-10T11:40:41.085 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:41.079 7fa7fdffb700 1 -- 172.21.2.102:0/3323691904 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa800006a60 con 0x7fa8080863d0 2021-02-10T11:40:41.085 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:41.079 7fa7fdffb700 1 --2- 172.21.2.102:0/3323691904 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa7e80186c0 0x7fa7e801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:41.085 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:41.079 7fa7fdffb700 1 -- 172.21.2.102:0/3323691904 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fa800032560 con 0x7fa8080863d0 2021-02-10T11:40:41.085 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:41.079 7fa80d401700 1 --2- 172.21.2.102:0/3323691904 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa7e80186c0 0x7fa7e801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:41.085 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:41.079 7fa80d401700 1 --2- 172.21.2.102:0/3323691904 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa7e80186c0 0x7fa7e801abc0 crc :-1 s=READY pgs=1487 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:41.112 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:41.110 7fa80f5e5700 1 -- 172.21.2.102:0/3323691904 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8080fe1c0 con 0x7fa8080863d0 2021-02-10T11:40:41.114 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:41.112 7fa7fdffb700 1 -- 172.21.2.102:0/3323691904 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fa8000304a0 con 0x7fa8080863d0 2021-02-10T11:40:41.242 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:41.240 7fa80f5e5700 1 -- 172.21.2.102:0/3323691904 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7fa8080fe1c0 con 0x7fa8080863d0 2021-02-10T11:40:41.242 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:41.240 7fa7fdffb700 1 -- 172.21.2.102:0/3323691904 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v197) v1 ==== 100+0+10 (secure 0 0 0) 0x7fa8000304a0 con 0x7fa8080863d0 2021-02-10T11:40:41.243 INFO:teuthology.orchestra.run.gibba002.stdout:pg_num: 8 2021-02-10T11:40:41.244 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:41.242 7fa80f5e5700 1 -- 172.21.2.102:0/3323691904 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa7e80186c0 msgr2=0x7fa7e801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:41.244 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:41.242 7fa80f5e5700 1 --2- 172.21.2.102:0/3323691904 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa7e80186c0 0x7fa7e801abc0 crc :-1 s=READY pgs=1487 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:41.244 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:41.242 7fa80f5e5700 1 -- 172.21.2.102:0/3323691904 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa8080863d0 msgr2=0x7fa8080867d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:41.244 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:41.242 7fa80f5e5700 1 --2- 172.21.2.102:0/3323691904 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa8080863d0 0x7fa8080867d0 secure :-1 s=READY pgs=3414 cs=0 l=1 rev1=1 rx=0x7fa800006ea0 tx=0x7fa80000b940).stop 2021-02-10T11:40:41.245 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:41.242 7fa80f5e5700 1 -- 172.21.2.102:0/3323691904 shutdown_connections 2021-02-10T11:40:41.245 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:41.242 7fa80f5e5700 1 --2- 172.21.2.102:0/3323691904 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa8080863d0 0x7fa8080867d0 unknown :-1 s=CLOSED pgs=3414 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:41.245 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:41.242 7fa80f5e5700 1 --2- 172.21.2.102:0/3323691904 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa7e80186c0 0x7fa7e801abc0 unknown :-1 s=CLOSED pgs=1487 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:41.245 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:41.242 7fa80f5e5700 1 --2- 172.21.2.102:0/3323691904 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa8081a44c0 0x7fa8081a48c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:41.245 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:41.242 7fa80f5e5700 1 --2- 172.21.2.102:0/3323691904 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa8080c3820 0x7fa8080a7950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:41.245 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:41.242 7fa80f5e5700 1 -- 172.21.2.102:0/3323691904 >> 172.21.2.102:0/3323691904 conn(0x7fa8080872c0 msgr2=0x7fa808095700 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:41.245 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:41.242 7fa80f5e5700 1 -- 172.21.2.102:0/3323691904 shutdown_connections 2021-02-10T11:40:41.245 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:41.242 7fa80f5e5700 1 -- 172.21.2.102:0/3323691904 wait complete. 2021-02-10T11:40:41.253 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-02-10T11:40:41.253 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:40:41.682 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:41.663 7f82c1bba700 1 Processor -- start 2021-02-10T11:40:41.682 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:41.663 7f82c1bba700 1 -- start start 2021-02-10T11:40:41.682 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:41.664 7f82c1bba700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f82bc0d1870 0x7f82bc1b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:41.683 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:41.664 7f82c1bba700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f82bc0f0650 0x7f82bc0cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:41.683 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:41.664 7f82c1bba700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82bc0f0b10 con 0x7f82bc1b0a10 2021-02-10T11:40:41.683 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:41.664 7f82c1bba700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82bc0e5ed0 con 0x7f82bc0f0650 2021-02-10T11:40:41.683 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:41.664 7f82c1bba700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82bc0d1c70 con 0x7f82bc0d1870 2021-02-10T11:40:41.683 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:41.664 7f82bbfff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f82bc0f0650 0x7f82bc0cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:41.683 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:41.664 7f82bbfff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f82bc0f0650 0x7f82bc0cfb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:41482/0 (socket says 172.21.2.102:41482) 2021-02-10T11:40:41.683 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:41.664 7f82bbfff700 1 -- 172.21.2.102:0/2210023557 learned_addr learned my addr 172.21.2.102:0/2210023557 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:40:41.683 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:41.664 7f82baffd700 1 --1- 172.21.2.102:0/2210023557 >> v1:172.21.2.102:6789/0 conn(0x7f82bc1b0a10 0x7f82bc0cf330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:33706/0 (socket says 172.21.2.102:33706) 2021-02-10T11:40:41.683 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:41.664 7f82ba7fc700 1 -- 172.21.2.102:0/2210023557 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3529623636 0 0) 0x7f82b00010b0 con 0x7f82bc1b0a10 2021-02-10T11:40:41.683 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:41.664 7f82ba7fc700 1 -- 172.21.2.102:0/2210023557 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82a4002fe0 con 0x7f82bc1b0a10 2021-02-10T11:40:41.684 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:41.664 7f82bb7fe700 1 --2- 172.21.2.102:0/2210023557 >> v2:172.21.2.113:3300/0 conn(0x7f82bc0d1870 0x7f82bc1b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:41.684 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:41.664 7f82bbfff700 1 -- 172.21.2.102:0/2210023557 >> v2:172.21.2.113:3300/0 conn(0x7f82bc0d1870 msgr2=0x7f82bc1b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:41.684 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:41.664 7f82bbfff700 1 --2- 172.21.2.102:0/2210023557 >> v2:172.21.2.113:3300/0 conn(0x7f82bc0d1870 0x7f82bc1b03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:41.684 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:41.664 7f82bbfff700 1 -- 172.21.2.102:0/2210023557 >> v1:172.21.2.102:6789/0 conn(0x7f82bc1b0a10 legacy=0x7f82bc0cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:41.684 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:41.664 7f82bbfff700 1 -- 172.21.2.102:0/2210023557 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82bc0d1e90 con 0x7f82bc0f0650 2021-02-10T11:40:41.684 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:41.665 7f82bb7fe700 1 --2- 172.21.2.102:0/2210023557 >> v2:172.21.2.113:3300/0 conn(0x7f82bc0d1870 0x7f82bc1b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:40:41.684 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:41.665 7f82bbfff700 1 --2- 172.21.2.102:0/2210023557 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f82bc0f0650 0x7f82bc0cfb20 secure :-1 s=READY pgs=3416 cs=0 l=1 rev1=1 rx=0x7f82ac006b80 tx=0x7f82ac009b90).ready entity=mon.0 client_cookie=c1ad02ad0464a429 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:41.684 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:41.665 7f82ba7fc700 1 -- 172.21.2.102:0/2210023557 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f82ac0022e0 con 0x7f82bc0f0650 2021-02-10T11:40:41.684 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:41.665 7f82ba7fc700 1 -- 172.21.2.102:0/2210023557 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82ac002620 con 0x7f82bc0f0650 2021-02-10T11:40:41.684 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:41.665 7f82ba7fc700 1 -- 172.21.2.102:0/2210023557 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f82ac002a80 con 0x7f82bc0f0650 2021-02-10T11:40:41.685 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:41.665 7f82c1bba700 1 -- 172.21.2.102:0/2210023557 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f82bc0f0650 msgr2=0x7f82bc0cfb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:41.685 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:41.665 7f82c1bba700 1 --2- 172.21.2.102:0/2210023557 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f82bc0f0650 0x7f82bc0cfb20 secure :-1 s=READY pgs=3416 cs=0 l=1 rev1=1 rx=0x7f82ac006b80 tx=0x7f82ac009b90).stop 2021-02-10T11:40:41.685 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:41.666 7f82c1bba700 1 -- 172.21.2.102:0/2210023557 shutdown_connections 2021-02-10T11:40:41.685 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:41.666 7f82c1bba700 1 --2- 172.21.2.102:0/2210023557 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f82bc0f0650 0x7f82bc0cfb20 unknown :-1 s=CLOSED pgs=3416 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:41.685 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:41.666 7f82c1bba700 1 --2- 172.21.2.102:0/2210023557 >> v2:172.21.2.113:3300/0 conn(0x7f82bc0d1870 0x7f82bc1b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:41.685 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:41.666 7f82c1bba700 1 -- 172.21.2.102:0/2210023557 >> 172.21.2.102:0/2210023557 conn(0x7f82bc0ce450 msgr2=0x7f82bc0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:41.685 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:41.666 7f82c1bba700 1 -- 172.21.2.102:0/2210023557 shutdown_connections 2021-02-10T11:40:41.685 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:41.666 7f82c1bba700 1 -- 172.21.2.102:0/2210023557 wait complete. 2021-02-10T11:40:41.686 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:41.680 7f82c1bba700 1 Processor -- start 2021-02-10T11:40:41.687 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:41.680 7f82c1bba700 1 -- start start 2021-02-10T11:40:41.687 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:41.681 7f82c1bba700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f82bc0beeb0 0x7f82bc0bf2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:41.687 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:41.681 7f82c1bba700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f82bc18ef70 0x7f82bc18f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:41.687 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:41.681 7f82c1bba700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f82bc18f990 0x7f82bc0c01d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:41.687 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:41.681 7f82c1bba700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82bc0bfc50 con 0x7f82bc18f990 2021-02-10T11:40:41.687 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:41.681 7f82c1bba700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82bc18fd90 con 0x7f82bc0beeb0 2021-02-10T11:40:41.687 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:41.681 7f82c1bba700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82bc0bf8d0 con 0x7f82bc18ef70 2021-02-10T11:40:41.687 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:41.681 7f82bbfff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f82bc18f990 0x7f82bc0c01d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:41.687 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:41.681 7f82bbfff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f82bc18f990 0x7f82bc0c01d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:41488/0 (socket says 172.21.2.102:41488) 2021-02-10T11:40:41.688 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:41.681 7f82bbfff700 1 -- 172.21.2.102:0/2721127792 learned_addr learned my addr 172.21.2.102:0/2721127792 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:40:41.688 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:41.681 7f82bbfff700 1 -- 172.21.2.102:0/2721127792 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f82bc18ef70 msgr2=0x7f82bc18f370 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:40:41.688 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:41.681 7f82bbfff700 1 --2- 172.21.2.102:0/2721127792 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f82bc18ef70 0x7f82bc18f370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:41.688 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:41.681 7f82bbfff700 1 -- 172.21.2.102:0/2721127792 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f82bc0beeb0 msgr2=0x7f82bc0bf2b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:40:41.688 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:41.681 7f82bbfff700 1 --2- 172.21.2.102:0/2721127792 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f82bc0beeb0 0x7f82bc0bf2b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:41.688 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:41.681 7f82bbfff700 1 -- 172.21.2.102:0/2721127792 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82bc0c07f0 con 0x7f82bc18f990 2021-02-10T11:40:41.688 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:41.681 7f82bbfff700 1 --2- 172.21.2.102:0/2721127792 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f82bc18f990 0x7f82bc0c01d0 secure :-1 s=READY pgs=3417 cs=0 l=1 rev1=1 rx=0x7f82ac006f10 tx=0x7f82ac00b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:41.688 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:41.682 7f82a3fff700 1 -- 172.21.2.102:0/2721127792 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f82ac00bd80 con 0x7f82bc18f990 2021-02-10T11:40:41.688 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:41.682 7f82a3fff700 1 -- 172.21.2.102:0/2721127792 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82ac002030 con 0x7f82bc18f990 2021-02-10T11:40:41.688 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:41.682 7f82c1bba700 1 -- 172.21.2.102:0/2721127792 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82bc0c07f0 con 0x7f82bc18f990 2021-02-10T11:40:41.688 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:41.682 7f82c1bba700 1 -- 172.21.2.102:0/2721127792 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82bc0c0d80 con 0x7f82bc18f990 2021-02-10T11:40:41.688 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:41.682 7f82a3fff700 1 -- 172.21.2.102:0/2721127792 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f82ac002550 con 0x7f82bc18f990 2021-02-10T11:40:41.689 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:41.682 7f82a3fff700 1 -- 172.21.2.102:0/2721127792 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f82ac000940 con 0x7f82bc18f990 2021-02-10T11:40:41.689 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:41.682 7f82a3fff700 1 --2- 172.21.2.102:0/2721127792 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f829c0186c0 0x7f829c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:41.689 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:41.682 7f82a3fff700 1 -- 172.21.2.102:0/2721127792 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f82ac0325e0 con 0x7f82bc18f990 2021-02-10T11:40:41.689 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:41.683 7f82bb7fe700 1 --2- 172.21.2.102:0/2721127792 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f829c0186c0 0x7f829c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:41.689 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:41.683 7f82bb7fe700 1 --2- 172.21.2.102:0/2721127792 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f829c0186c0 0x7f829c01abc0 crc :-1 s=READY pgs=1488 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:41.716 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:41.713 7f82c1bba700 1 -- 172.21.2.102:0/2721127792 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82bc0ebc60 con 0x7f82bc18f990 2021-02-10T11:40:41.718 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:41.716 7f82a3fff700 1 -- 172.21.2.102:0/2721127792 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f82ac001f30 con 0x7f82bc18f990 2021-02-10T11:40:41.845 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:41.843 7f82c1bba700 1 -- 172.21.2.102:0/2721127792 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f82bc0f0b10 con 0x7f82bc18f990 2021-02-10T11:40:41.846 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:41.844 7f82a3fff700 1 -- 172.21.2.102:0/2721127792 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 173 v173) v1 ==== 95+0+2108 (secure 0 0 0) 0x7f82ac001f30 con 0x7f82bc18f990 2021-02-10T11:40:41.847 INFO:teuthology.orchestra.run.gibba002.stderr:dumped fsmap epoch 173 2021-02-10T11:40:41.847 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:40:41.847 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":173,"default_fscid":21,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":13872,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.102:6835/1871410080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6834","nonce":1871410080},{"type":"v1","addr":"172.21.2.102:6835","nonce":1871410080}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":169}],"filesystems":[{"mdsmap":{"epoch":173,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:40:24.365024","modified":"2021-02-10 11:40:26.372300","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7598},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7598":{"gid":7598,"name":"b","rank":0,"incarnation":170,"state":"up:active","state_seq":3,"addr":"172.21.2.113:6833/2247818770","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6832","nonce":2247818770},{"type":"v1","addr":"172.21.2.113:6833","nonce":2247818770}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[43],"metadata_pool":42,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":21}]} 2021-02-10T11:40:41.848 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:41.845 7f82c1bba700 1 -- 172.21.2.102:0/2721127792 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f829c0186c0 msgr2=0x7f829c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:41.848 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:41.845 7f82c1bba700 1 --2- 172.21.2.102:0/2721127792 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f829c0186c0 0x7f829c01abc0 crc :-1 s=READY pgs=1488 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:41.848 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:41.845 7f82c1bba700 1 -- 172.21.2.102:0/2721127792 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f82bc18f990 msgr2=0x7f82bc0c01d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:41.848 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:41.845 7f82c1bba700 1 --2- 172.21.2.102:0/2721127792 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f82bc18f990 0x7f82bc0c01d0 secure :-1 s=READY pgs=3417 cs=0 l=1 rev1=1 rx=0x7f82ac006f10 tx=0x7f82ac00b990).stop 2021-02-10T11:40:41.848 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:41.846 7f82c1bba700 1 -- 172.21.2.102:0/2721127792 shutdown_connections 2021-02-10T11:40:41.849 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:41.846 7f82c1bba700 1 --2- 172.21.2.102:0/2721127792 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f82bc18f990 0x7f82bc0c01d0 unknown :-1 s=CLOSED pgs=3417 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:41.849 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:41.846 7f82c1bba700 1 --2- 172.21.2.102:0/2721127792 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f82bc18ef70 0x7f82bc18f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:41.849 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:41.846 7f82c1bba700 1 --2- 172.21.2.102:0/2721127792 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f829c0186c0 0x7f829c01abc0 unknown :-1 s=CLOSED pgs=1488 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:41.849 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:41.846 7f82c1bba700 1 --2- 172.21.2.102:0/2721127792 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f82bc0beeb0 0x7f82bc0bf2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:41.849 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:41.846 7f82c1bba700 1 -- 172.21.2.102:0/2721127792 >> 172.21.2.102:0/2721127792 conn(0x7f82bc0c2130 msgr2=0x7f82bc09e3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:41.849 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:41.846 7f82c1bba700 1 -- 172.21.2.102:0/2721127792 shutdown_connections 2021-02-10T11:40:41.849 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:41.846 7f82c1bba700 1 -- 172.21.2.102:0/2721127792 wait complete. 2021-02-10T11:40:42.058 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 173, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:40:24.365024', 'modified': '2021-02-10 11:40:26.372300', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7598}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7598': {'gid': 7598, 'name': 'b', 'rank': 0, 'incarnation': 170, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.113:6833/2247818770', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.113:6832', 'nonce': 2247818770}, {'type': 'v1', 'addr': '172.21.2.113:6833', 'nonce': 2247818770}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [43], 'metadata_pool': 42, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:40:42.058 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:40:42.058 DEBUG:teuthology.orchestra.run.gibba013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:40:42.457 INFO:teuthology.orchestra.run.gibba013.stdout:{ 2021-02-10T11:40:42.458 INFO:teuthology.orchestra.run.gibba013.stdout: "cluster_fsid": "68db30ea-dbce-4aef-a6df-f01c6fea0513", 2021-02-10T11:40:42.458 INFO:teuthology.orchestra.run.gibba013.stdout: "whoami": 0, 2021-02-10T11:40:42.458 INFO:teuthology.orchestra.run.gibba013.stdout: "id": 7598, 2021-02-10T11:40:42.458 INFO:teuthology.orchestra.run.gibba013.stdout: "want_state": "up:active", 2021-02-10T11:40:42.458 INFO:teuthology.orchestra.run.gibba013.stdout: "state": "up:active", 2021-02-10T11:40:42.458 INFO:teuthology.orchestra.run.gibba013.stdout: "rank_uptime": 18.088530473999999, 2021-02-10T11:40:42.458 INFO:teuthology.orchestra.run.gibba013.stdout: "mdsmap_epoch": 173, 2021-02-10T11:40:42.458 INFO:teuthology.orchestra.run.gibba013.stdout: "osdmap_epoch": 197, 2021-02-10T11:40:42.458 INFO:teuthology.orchestra.run.gibba013.stdout: "osdmap_epoch_barrier": 197, 2021-02-10T11:40:42.458 INFO:teuthology.orchestra.run.gibba013.stdout: "uptime": 23.742049192 2021-02-10T11:40:42.459 INFO:teuthology.orchestra.run.gibba013.stdout:} 2021-02-10T11:40:42.462 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "68db30ea-dbce-4aef-a6df-f01c6fea0513", "whoami": 0, "id": 7598, "want_state": "up:active", "state": "up:active", "rank_uptime": 18.088530473999999, "mdsmap_epoch": 173, "osdmap_epoch": 197, "osdmap_epoch_barrier": 197, "uptime": 23.742049192 } 2021-02-10T11:40:42.462 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2021-02-10T11:40:42.462 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2021-02-10T11:40:42.462 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.1 at ubuntu@gibba002.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.1... 2021-02-10T11:40:42.462 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:40:42.462 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec mkdir -p /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:40:42.799 DEBUG:teuthology.orchestra.run.gibba002:> sudo modprobe fuse 2021-02-10T11:40:43.145 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:40:43.145 DEBUG:teuthology.orchestra.run.gibba002:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:40:43.489 INFO:teuthology.orchestra.run.gibba002.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:40:43.490 INFO:teuthology.orchestra.run.gibba002.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:40:43.490 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:40:43.491 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:40:43.491 DEBUG:teuthology.orchestra.run.gibba002:> ls /sys/fs/fuse/connections 2021-02-10T11:40:43.825 INFO:teuthology.orchestra.run.gibba002.stdout:43 2021-02-10T11:40:43.825 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [43] 2021-02-10T11:40:43.826 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.1 /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:40:43.828 DEBUG:teuthology.orchestra.run.gibba002:> sudo modprobe fuse 2021-02-10T11:40:44.175 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:40:44.175 DEBUG:teuthology.orchestra.run.gibba002:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:40:44.241 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba002.stderr:2021-02-10 11:40:44.238 7f4d7d731f80 -1 init, newargv = 0x55a0109ab220 newargc=9 2021-02-10T11:40:44.241 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba002.stderr:ceph-fuse[221810]: starting ceph client 2021-02-10T11:40:44.245 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba002.stderr:ceph-fuse[221810]: starting fuse 2021-02-10T11:40:44.519 INFO:teuthology.orchestra.run.gibba002.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:40:44.519 INFO:teuthology.orchestra.run.gibba002.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:40:44.520 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:40:44.520 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:40:44.520 DEBUG:teuthology.orchestra.run.gibba002:> ls /sys/fs/fuse/connections 2021-02-10T11:40:44.857 INFO:teuthology.orchestra.run.gibba002.stdout:43 2021-02-10T11:40:44.857 INFO:teuthology.orchestra.run.gibba002.stdout:44 2021-02-10T11:40:44.857 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43, 44] 2021-02-10T11:40:44.857 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:40:44.857 DEBUG:teuthology.orchestra.run.gibba002:> sudo python3 -c ' 2021-02-10T11:40:44.858 DEBUG:teuthology.orchestra.run.gibba002:> import glob 2021-02-10T11:40:44.858 DEBUG:teuthology.orchestra.run.gibba002:> import re 2021-02-10T11:40:44.858 DEBUG:teuthology.orchestra.run.gibba002:> import os 2021-02-10T11:40:44.858 DEBUG:teuthology.orchestra.run.gibba002:> import subprocess 2021-02-10T11:40:44.858 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:40:44.858 DEBUG:teuthology.orchestra.run.gibba002:> def find_socket(client_name): 2021-02-10T11:40:44.858 DEBUG:teuthology.orchestra.run.gibba002:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2021-02-10T11:40:44.858 DEBUG:teuthology.orchestra.run.gibba002:> files = glob.glob(asok_path) 2021-02-10T11:40:44.858 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:40:44.858 DEBUG:teuthology.orchestra.run.gibba002:> # Given a non-glob path, it better be there 2021-02-10T11:40:44.858 DEBUG:teuthology.orchestra.run.gibba002:> if "*" not in asok_path: 2021-02-10T11:40:44.859 DEBUG:teuthology.orchestra.run.gibba002:> assert(len(files) == 1) 2021-02-10T11:40:44.859 DEBUG:teuthology.orchestra.run.gibba002:> return files[0] 2021-02-10T11:40:44.859 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:40:44.859 DEBUG:teuthology.orchestra.run.gibba002:> for f in files: 2021-02-10T11:40:44.859 DEBUG:teuthology.orchestra.run.gibba002:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2021-02-10T11:40:44.859 DEBUG:teuthology.orchestra.run.gibba002:> if os.path.exists("/proc/{0}".format(pid)): 2021-02-10T11:40:44.859 DEBUG:teuthology.orchestra.run.gibba002:> return f 2021-02-10T11:40:44.859 DEBUG:teuthology.orchestra.run.gibba002:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2021-02-10T11:40:44.860 DEBUG:teuthology.orchestra.run.gibba002:> 2021-02-10T11:40:44.860 DEBUG:teuthology.orchestra.run.gibba002:> print(find_socket("client.1")) 2021-02-10T11:40:44.860 DEBUG:teuthology.orchestra.run.gibba002:> ' 2021-02-10T11:40:45.222 INFO:teuthology.orchestra.run.gibba002.stdout:/var/run/ceph/ceph-client.1.221810.asok 2021-02-10T11:40:45.225 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.221810.asok 2021-02-10T11:40:45.225 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:40:45.225 DEBUG:teuthology.orchestra.run.gibba002:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.221810.asok status 2021-02-10T11:40:45.620 INFO:teuthology.orchestra.run.gibba002.stdout:{ 2021-02-10T11:40:45.620 INFO:teuthology.orchestra.run.gibba002.stdout: "metadata": { 2021-02-10T11:40:45.620 INFO:teuthology.orchestra.run.gibba002.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:40:45.620 INFO:teuthology.orchestra.run.gibba002.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:40:45.620 INFO:teuthology.orchestra.run.gibba002.stdout: "entity_id": "1", 2021-02-10T11:40:45.620 INFO:teuthology.orchestra.run.gibba002.stdout: "hostname": "gibba002", 2021-02-10T11:40:45.620 INFO:teuthology.orchestra.run.gibba002.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2021-02-10T11:40:45.620 INFO:teuthology.orchestra.run.gibba002.stdout: "pid": "221810", 2021-02-10T11:40:45.620 INFO:teuthology.orchestra.run.gibba002.stdout: "root": "/" 2021-02-10T11:40:45.620 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:40:45.620 INFO:teuthology.orchestra.run.gibba002.stdout: "dentry_count": 0, 2021-02-10T11:40:45.621 INFO:teuthology.orchestra.run.gibba002.stdout: "dentry_pinned_count": 0, 2021-02-10T11:40:45.621 INFO:teuthology.orchestra.run.gibba002.stdout: "id": 14082, 2021-02-10T11:40:45.621 INFO:teuthology.orchestra.run.gibba002.stdout: "inst": { 2021-02-10T11:40:45.621 INFO:teuthology.orchestra.run.gibba002.stdout: "name": { 2021-02-10T11:40:45.621 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "client", 2021-02-10T11:40:45.621 INFO:teuthology.orchestra.run.gibba002.stdout: "num": 14082 2021-02-10T11:40:45.621 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:40:45.621 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": { 2021-02-10T11:40:45.621 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:40:45.621 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:0", 2021-02-10T11:40:45.621 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 3105792919 2021-02-10T11:40:45.621 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:40:45.621 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:40:45.622 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": { 2021-02-10T11:40:45.622 INFO:teuthology.orchestra.run.gibba002.stdout: "type": "v1", 2021-02-10T11:40:45.622 INFO:teuthology.orchestra.run.gibba002.stdout: "addr": "172.21.2.102:0", 2021-02-10T11:40:45.622 INFO:teuthology.orchestra.run.gibba002.stdout: "nonce": 3105792919 2021-02-10T11:40:45.622 INFO:teuthology.orchestra.run.gibba002.stdout: }, 2021-02-10T11:40:45.622 INFO:teuthology.orchestra.run.gibba002.stdout: "inst_str": "client.14082 172.21.2.102:0/3105792919", 2021-02-10T11:40:45.622 INFO:teuthology.orchestra.run.gibba002.stdout: "addr_str": "172.21.2.102:0/3105792919", 2021-02-10T11:40:45.622 INFO:teuthology.orchestra.run.gibba002.stdout: "inode_count": 1, 2021-02-10T11:40:45.622 INFO:teuthology.orchestra.run.gibba002.stdout: "mds_epoch": 173, 2021-02-10T11:40:45.622 INFO:teuthology.orchestra.run.gibba002.stdout: "osd_epoch": 197, 2021-02-10T11:40:45.622 INFO:teuthology.orchestra.run.gibba002.stdout: "osd_epoch_barrier": 0, 2021-02-10T11:40:45.622 INFO:teuthology.orchestra.run.gibba002.stdout: "blacklisted": false 2021-02-10T11:40:45.622 INFO:teuthology.orchestra.run.gibba002.stdout:} 2021-02-10T11:40:45.624 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:40:45.624 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:40:45.624 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:40:45.960 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:40:45.960 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2021-02-10T11:40:45.960 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:40:45.961 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:40:46.308 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2021-02-10T11:40:46.348 DEBUG:teuthology.orchestra.run.gibba002:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:40:46.354 DEBUG:teuthology.orchestra.run.gibba013:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:40:46.744 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:46.725 7feed50f4700 1 Processor -- start 2021-02-10T11:40:46.744 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:46.725 7feed50f4700 1 -- start start 2021-02-10T11:40:46.744 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:46.726 7feed50f4700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7feed010aff0 0x7feed00cf750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:46.745 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:46.726 7feed50f4700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feed00fe1c0 con 0x7feed00d1870 2021-02-10T11:40:46.745 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:46.726 7feed50f4700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feed01b0cb0 con 0x7feed00d1c70 2021-02-10T11:40:46.745 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:46.726 7feed50f4700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feed0109a60 con 0x7feed010aff0 2021-02-10T11:40:46.745 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:46.726 7feeced9d700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7feed00d1870 0x7feed01b04c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:33730/0 (socket says 172.21.2.102:33730) 2021-02-10T11:40:46.745 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:46.726 7feeced9d700 1 -- 172.21.2.102:0/3519602052 learned_addr learned my addr 172.21.2.102:0/3519602052 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:40:46.745 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:46.726 7feecf59e700 1 --2- 172.21.2.102:0/3519602052 >> v2:172.21.2.113:3300/0 conn(0x7feed010aff0 0x7feed00cf750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:46.745 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:46.726 7feecdd9b700 1 -- 172.21.2.102:0/3519602052 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2830843832 0 0) 0x7feec0001180 con 0x7feed00d1870 2021-02-10T11:40:46.745 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:46.726 7feecdd9b700 1 -- 172.21.2.102:0/3519602052 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feebc002fe0 con 0x7feed00d1870 2021-02-10T11:40:46.745 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:46.726 7feecdd9b700 1 -- 172.21.2.102:0/3519602052 <== mon.1 v1:172.21.2.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1417650093 0 0) 0x7feeb8000ff0 con 0x7feed00d1c70 2021-02-10T11:40:46.745 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:46.726 7feecdd9b700 1 -- 172.21.2.102:0/3519602052 --> v1:172.21.2.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feebc003210 con 0x7feed00d1c70 2021-02-10T11:40:46.745 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:46.726 7feecf59e700 1 -- 172.21.2.102:0/3519602052 >> v1:172.21.2.113:6789/0 conn(0x7feed00d1c70 legacy=0x7feed00cef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:46.746 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:46.726 7feecf59e700 1 -- 172.21.2.102:0/3519602052 >> v1:172.21.2.102:6789/0 conn(0x7feed00d1870 legacy=0x7feed01b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:46.746 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:46.726 7feecf59e700 1 -- 172.21.2.102:0/3519602052 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feed010b3f0 con 0x7feed010aff0 2021-02-10T11:40:46.746 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:46.727 7feecf59e700 1 --2- 172.21.2.102:0/3519602052 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7feed010aff0 0x7feed00cf750 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 rx=0x7feec8006e50 tx=0x7feec8009e70).ready entity=mon.1 client_cookie=c2d11c53d1f442f5 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:46.746 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:46.727 7feecdd9b700 1 -- 172.21.2.102:0/3519602052 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7feec8006a20 con 0x7feed010aff0 2021-02-10T11:40:46.746 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:46.727 7feecdd9b700 1 -- 172.21.2.102:0/3519602052 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feec8006790 con 0x7feed010aff0 2021-02-10T11:40:46.746 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:46.727 7feecdd9b700 1 -- 172.21.2.102:0/3519602052 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7feec8000f90 con 0x7feed010aff0 2021-02-10T11:40:46.746 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:46.727 7feed50f4700 1 -- 172.21.2.102:0/3519602052 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7feed010aff0 msgr2=0x7feed00cf750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:46.746 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:46.727 7feed50f4700 1 --2- 172.21.2.102:0/3519602052 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7feed010aff0 0x7feed00cf750 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 rx=0x7feec8006e50 tx=0x7feec8009e70).stop 2021-02-10T11:40:46.746 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:46.728 7feed50f4700 1 -- 172.21.2.102:0/3519602052 shutdown_connections 2021-02-10T11:40:46.746 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:46.728 7feed50f4700 1 --2- 172.21.2.102:0/3519602052 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7feed010aff0 0x7feed00cf750 unknown :-1 s=CLOSED pgs=909 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:46.747 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:46.728 7feed50f4700 1 -- 172.21.2.102:0/3519602052 >> 172.21.2.102:0/3519602052 conn(0x7feed00ce420 msgr2=0x7feed00d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:46.747 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:46.728 7feed50f4700 1 -- 172.21.2.102:0/3519602052 shutdown_connections 2021-02-10T11:40:46.747 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:46.728 7feed50f4700 1 -- 172.21.2.102:0/3519602052 wait complete. 2021-02-10T11:40:46.748 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:46.742 7feed50f4700 1 Processor -- start 2021-02-10T11:40:46.748 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:46.742 7feed50f4700 1 -- start start 2021-02-10T11:40:46.749 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:46.743 7feed50f4700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7feed018e7c0 0x7feed01aeeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:46.749 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:46.743 7feed50f4700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7feed01af4d0 0x7feed00ad560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:46.749 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:46.743 7feed50f4700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7feed00adb80 0x7feed008a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:46.749 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:46.743 7feed50f4700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feed01afcd0 con 0x7feed01af4d0 2021-02-10T11:40:46.749 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:46.743 7feed50f4700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feed018ebc0 con 0x7feed018e7c0 2021-02-10T11:40:46.749 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:46.743 7feed50f4700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feed01af8d0 con 0x7feed00adb80 2021-02-10T11:40:46.749 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:46.743 7feece59c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7feed01af4d0 0x7feed00ad560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:46.749 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:46.743 7feece59c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7feed01af4d0 0x7feed00ad560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:41516/0 (socket says 172.21.2.102:41516) 2021-02-10T11:40:46.749 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:46.743 7feece59c700 1 -- 172.21.2.102:0/500243707 learned_addr learned my addr 172.21.2.102:0/500243707 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:40:46.749 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:46.743 7feece59c700 1 -- 172.21.2.102:0/500243707 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7feed00adb80 msgr2=0x7feed008a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:46.749 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:46.743 7feece59c700 1 --2- 172.21.2.102:0/500243707 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7feed00adb80 0x7feed008a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:46.750 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:46.743 7feece59c700 1 -- 172.21.2.102:0/500243707 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7feed018e7c0 msgr2=0x7feed01aeeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:46.750 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:46.743 7feece59c700 1 --2- 172.21.2.102:0/500243707 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7feed018e7c0 0x7feed01aeeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:46.750 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:46.743 7feece59c700 1 -- 172.21.2.102:0/500243707 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feed00adf80 con 0x7feed01af4d0 2021-02-10T11:40:46.750 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:46.744 7feece59c700 1 --2- 172.21.2.102:0/500243707 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7feed01af4d0 0x7feed00ad560 secure :-1 s=READY pgs=3422 cs=0 l=1 rev1=1 rx=0x7feeb800a570 tx=0x7feeb800a910).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:46.750 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:46.744 7feec77fe700 1 -- 172.21.2.102:0/500243707 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7feeb800ad00 con 0x7feed01af4d0 2021-02-10T11:40:46.750 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:46.744 7feec77fe700 1 -- 172.21.2.102:0/500243707 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feeb800c0d0 con 0x7feed01af4d0 2021-02-10T11:40:46.750 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:46.744 7feec77fe700 1 -- 172.21.2.102:0/500243707 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7feeb800c560 con 0x7feed01af4d0 2021-02-10T11:40:46.750 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:46.744 7feed50f4700 1 -- 172.21.2.102:0/500243707 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feed00adf80 con 0x7feed01af4d0 2021-02-10T11:40:46.750 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:46.744 7feed50f4700 1 -- 172.21.2.102:0/500243707 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feed008aff0 con 0x7feed01af4d0 2021-02-10T11:40:46.751 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:46.744 7feec77fe700 1 -- 172.21.2.102:0/500243707 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7feeb8008030 con 0x7feed01af4d0 2021-02-10T11:40:46.751 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:46.744 7feec77fe700 1 --2- 172.21.2.102:0/500243707 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7feeb40186c0 0x7feeb401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:46.751 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:46.744 7feec77fe700 1 -- 172.21.2.102:0/500243707 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 5573+0+0 (secure 0 0 0) 0x7feeb80325e0 con 0x7feed01af4d0 2021-02-10T11:40:46.751 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:46.745 7feeced9d700 1 --2- 172.21.2.102:0/500243707 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7feeb40186c0 0x7feeb401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:46.751 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:46.745 7feeced9d700 1 --2- 172.21.2.102:0/500243707 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7feeb40186c0 0x7feeb401abc0 crc :-1 s=READY pgs=1489 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:46.777 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:46.775 7feed50f4700 1 -- 172.21.2.102:0/500243707 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feed00825c0 con 0x7feed01af4d0 2021-02-10T11:40:46.780 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:46.778 7feec77fe700 1 -- 172.21.2.102:0/500243707 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7feeb8028ef0 con 0x7feed01af4d0 2021-02-10T11:40:46.915 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:46.913 7feed50f4700 1 -- 172.21.2.102:0/500243707 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7feed00825c0 con 0x7feed01af4d0 2021-02-10T11:40:47.215 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:47.213 7feec77fe700 1 -- 172.21.2.102:0/500243707 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v174) v1 ==== 136+0+0 (secure 0 0 0) 0x7feeb8028ef0 con 0x7feed01af4d0 2021-02-10T11:40:47.232 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:47.230 7feed50f4700 1 -- 172.21.2.102:0/500243707 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7feeb40186c0 msgr2=0x7feeb401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:47.233 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:47.230 7feed50f4700 1 --2- 172.21.2.102:0/500243707 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7feeb40186c0 0x7feeb401abc0 crc :-1 s=READY pgs=1489 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:47.233 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:47.230 7feed50f4700 1 -- 172.21.2.102:0/500243707 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7feed01af4d0 msgr2=0x7feed00ad560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:47.233 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:47.230 7feed50f4700 1 --2- 172.21.2.102:0/500243707 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7feed01af4d0 0x7feed00ad560 secure :-1 s=READY pgs=3422 cs=0 l=1 rev1=1 rx=0x7feeb800a570 tx=0x7feeb800a910).stop 2021-02-10T11:40:47.233 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:47.230 7feed50f4700 1 -- 172.21.2.102:0/500243707 shutdown_connections 2021-02-10T11:40:47.233 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:47.230 7feed50f4700 1 --2- 172.21.2.102:0/500243707 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7feed00adb80 0x7feed008a9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:47.233 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:47.230 7feed50f4700 1 --2- 172.21.2.102:0/500243707 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7feed01af4d0 0x7feed00ad560 unknown :-1 s=CLOSED pgs=3422 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:47.233 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:47.230 7feed50f4700 1 --2- 172.21.2.102:0/500243707 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7feeb40186c0 0x7feeb401abc0 unknown :-1 s=CLOSED pgs=1489 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:47.233 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:47.230 7feed50f4700 1 --2- 172.21.2.102:0/500243707 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7feed018e7c0 0x7feed01aeeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:47.233 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:47.230 7feed50f4700 1 -- 172.21.2.102:0/500243707 >> 172.21.2.102:0/500243707 conn(0x7feed01aca10 msgr2=0x7feed01b25e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:47.233 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:47.230 7feed50f4700 1 -- 172.21.2.102:0/500243707 shutdown_connections 2021-02-10T11:40:47.233 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:47.230 7feed50f4700 1 -- 172.21.2.102:0/500243707 wait complete. 2021-02-10T11:40:47.240 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs volume ls 2021-02-10T11:40:47.669 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:47.650 7fc3ba3f4700 1 Processor -- start 2021-02-10T11:40:47.669 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:47.650 7fc3ba3f4700 1 -- start start 2021-02-10T11:40:47.670 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:47.651 7fc3ba3f4700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc3b40d1c60 0x7fc3b40f2090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:47.670 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:47.651 7fc3ba3f4700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3b40f0b00 con 0x7fc3b40d0330 2021-02-10T11:40:47.670 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:47.651 7fc3ba3f4700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3b41b0bd0 con 0x7fc3b40d1860 2021-02-10T11:40:47.670 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:47.651 7fc3ba3f4700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3b40e5ec0 con 0x7fc3b40d1c60 2021-02-10T11:40:47.670 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:47.651 7fc3b37fe700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fc3b40d0330 0x7fc3b40ef630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:33746/0 (socket says 172.21.2.102:33746) 2021-02-10T11:40:47.670 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:47.651 7fc3b37fe700 1 -- 172.21.2.102:0/1313927934 learned_addr learned my addr 172.21.2.102:0/1313927934 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:40:47.670 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:47.651 7fc3b8a11700 1 --2- 172.21.2.102:0/1313927934 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc3b40d1c60 0x7fc3b40f2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:47.670 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:47.651 7fc3b2ffd700 1 -- 172.21.2.102:0/1313927934 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 767154041 0 0) 0x7fc3a8001180 con 0x7fc3b40d0330 2021-02-10T11:40:47.670 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:47.651 7fc3b2ffd700 1 -- 172.21.2.102:0/1313927934 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc39c002fe0 con 0x7fc3b40d0330 2021-02-10T11:40:47.670 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:47.651 7fc3b2ffd700 1 -- 172.21.2.102:0/1313927934 <== mon.1 v1:172.21.2.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3952870449 0 0) 0x7fc3a4001080 con 0x7fc3b40d1860 2021-02-10T11:40:47.670 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:47.651 7fc3b2ffd700 1 -- 172.21.2.102:0/1313927934 --> v1:172.21.2.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc39c003210 con 0x7fc3b40d1860 2021-02-10T11:40:47.671 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:47.652 7fc3b8a11700 1 -- 172.21.2.102:0/1313927934 >> v1:172.21.2.113:6789/0 conn(0x7fc3b40d1860 legacy=0x7fc3b41b03e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:47.671 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:47.652 7fc3b8a11700 1 -- 172.21.2.102:0/1313927934 >> v1:172.21.2.102:6789/0 conn(0x7fc3b40d0330 legacy=0x7fc3b40ef630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:47.671 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:47.652 7fc3b8a11700 1 -- 172.21.2.102:0/1313927934 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3b40efe20 con 0x7fc3b40d1c60 2021-02-10T11:40:47.671 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:47.652 7fc3b8a11700 1 --2- 172.21.2.102:0/1313927934 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc3b40d1c60 0x7fc3b40f2090 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 rx=0x7fc3ac006e50 tx=0x7fc3ac009e70).ready entity=mon.2 client_cookie=762975ccdf45b5b5 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:47.671 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:47.652 7fc3b2ffd700 1 -- 172.21.2.102:0/1313927934 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc3ac00b510 con 0x7fc3b40d1c60 2021-02-10T11:40:47.671 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:47.652 7fc3b2ffd700 1 -- 172.21.2.102:0/1313927934 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3ac006790 con 0x7fc3b40d1c60 2021-02-10T11:40:47.671 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:47.652 7fc3b2ffd700 1 -- 172.21.2.102:0/1313927934 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc3ac00b940 con 0x7fc3b40d1c60 2021-02-10T11:40:47.671 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:47.652 7fc3ba3f4700 1 -- 172.21.2.102:0/1313927934 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc3b40d1c60 msgr2=0x7fc3b40f2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:47.671 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:47.652 7fc3ba3f4700 1 --2- 172.21.2.102:0/1313927934 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc3b40d1c60 0x7fc3b40f2090 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 rx=0x7fc3ac006e50 tx=0x7fc3ac009e70).stop 2021-02-10T11:40:47.671 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:47.653 7fc3ba3f4700 1 -- 172.21.2.102:0/1313927934 shutdown_connections 2021-02-10T11:40:47.671 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:47.653 7fc3ba3f4700 1 --2- 172.21.2.102:0/1313927934 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc3b40d1c60 0x7fc3b40f2090 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:47.671 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:47.653 7fc3ba3f4700 1 -- 172.21.2.102:0/1313927934 >> 172.21.2.102:0/1313927934 conn(0x7fc3b40ce440 msgr2=0x7fc3b40d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:47.671 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:47.653 7fc3ba3f4700 1 -- 172.21.2.102:0/1313927934 shutdown_connections 2021-02-10T11:40:47.672 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:47.653 7fc3ba3f4700 1 -- 172.21.2.102:0/1313927934 wait complete. 2021-02-10T11:40:47.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:47.667 7fc3ba3f4700 1 Processor -- start 2021-02-10T11:40:47.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:47.668 7fc3ba3f4700 1 -- start start 2021-02-10T11:40:47.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:47.668 7fc3ba3f4700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc3b41a35c0 0x7fc3b41a39c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:47.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:47.668 7fc3ba3f4700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc3b409d0e0 0x7fc3b409d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:47.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:47.668 7fc3ba3f4700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc3b409db00 0x7fc3b409df00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:47.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:47.668 7fc3ba3f4700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3b409cb60 con 0x7fc3b409db00 2021-02-10T11:40:47.674 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:47.668 7fc3ba3f4700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3b40c3730 con 0x7fc3b41a35c0 2021-02-10T11:40:47.674 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:47.668 7fc3ba3f4700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3b40c3910 con 0x7fc3b409d0e0 2021-02-10T11:40:47.674 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:47.668 7fc3b8a11700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc3b409db00 0x7fc3b409df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:47.674 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:47.668 7fc3b8a11700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc3b409db00 0x7fc3b409df00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:41526/0 (socket says 172.21.2.102:41526) 2021-02-10T11:40:47.674 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:47.668 7fc3b8a11700 1 -- 172.21.2.102:0/762509449 learned_addr learned my addr 172.21.2.102:0/762509449 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:40:47.674 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:47.669 7fc3b8a11700 1 -- 172.21.2.102:0/762509449 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc3b409d0e0 msgr2=0x7fc3b409d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:40:47.674 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:47.669 7fc3b8a11700 1 --2- 172.21.2.102:0/762509449 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc3b409d0e0 0x7fc3b409d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:47.674 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:47.669 7fc3b8a11700 1 -- 172.21.2.102:0/762509449 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc3b41a35c0 msgr2=0x7fc3b41a39c0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:40:47.674 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:47.669 7fc3b8a11700 1 --2- 172.21.2.102:0/762509449 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc3b41a35c0 0x7fc3b41a39c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:47.674 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:47.669 7fc3b8a11700 1 -- 172.21.2.102:0/762509449 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3b4093e60 con 0x7fc3b409db00 2021-02-10T11:40:47.674 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:47.669 7fc3b8a11700 1 --2- 172.21.2.102:0/762509449 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc3b409db00 0x7fc3b409df00 secure :-1 s=READY pgs=3424 cs=0 l=1 rev1=1 rx=0x7fc3ac006c50 tx=0x7fc3ac00bd20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:47.675 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:47.669 7fc3b0ff9700 1 -- 172.21.2.102:0/762509449 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc3ac0093d0 con 0x7fc3b409db00 2021-02-10T11:40:47.675 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:47.669 7fc3b0ff9700 1 -- 172.21.2.102:0/762509449 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3ac00be60 con 0x7fc3b409db00 2021-02-10T11:40:47.675 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:47.669 7fc3ba3f4700 1 -- 172.21.2.102:0/762509449 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3b4093e60 con 0x7fc3b409db00 2021-02-10T11:40:47.675 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:47.669 7fc3ba3f4700 1 -- 172.21.2.102:0/762509449 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3b4093e60 con 0x7fc3b409db00 2021-02-10T11:40:47.675 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:47.669 7fc3b0ff9700 1 -- 172.21.2.102:0/762509449 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc3ac00c570 con 0x7fc3b409db00 2021-02-10T11:40:47.675 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:47.670 7fc3b0ff9700 1 -- 172.21.2.102:0/762509449 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fc3ac0093d0 con 0x7fc3b409db00 2021-02-10T11:40:47.675 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:47.670 7fc3b0ff9700 1 --2- 172.21.2.102:0/762509449 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc3940186c0 0x7fc39401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:47.675 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:47.670 7fc3b0ff9700 1 -- 172.21.2.102:0/762509449 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fc3ac032b70 con 0x7fc3b409db00 2021-02-10T11:40:47.675 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:47.670 7fc3b3fff700 1 --2- 172.21.2.102:0/762509449 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc3940186c0 0x7fc39401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:47.675 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:47.670 7fc3b3fff700 1 --2- 172.21.2.102:0/762509449 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc3940186c0 0x7fc39401abc0 crc :-1 s=READY pgs=1490 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:47.702 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:47.700 7fc3ba3f4700 1 -- 172.21.2.102:0/762509449 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3b40ebbe0 con 0x7fc3b409db00 2021-02-10T11:40:47.705 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:47.703 7fc3b0ff9700 1 -- 172.21.2.102:0/762509449 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fc3ac000df0 con 0x7fc3b409db00 2021-02-10T11:40:47.840 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:47.838 7fc3ba3f4700 1 -- 172.21.2.102:0/762509449 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs volume ls", "target": ["mgr", ""]}) v1 -- 0x7fc3b40c3730 con 0x7fc3940186c0 2021-02-10T11:40:47.841 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:47.839 7fc3b0ff9700 1 -- 172.21.2.102:0/762509449 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+41 (crc 0 0 0) 0x7fc3a4006970 con 0x7fc3940186c0 2021-02-10T11:40:47.841 INFO:teuthology.orchestra.run.gibba002.stdout:[ 2021-02-10T11:40:47.841 INFO:teuthology.orchestra.run.gibba002.stdout: { 2021-02-10T11:40:47.842 INFO:teuthology.orchestra.run.gibba002.stdout: "name": "cephfs" 2021-02-10T11:40:47.842 INFO:teuthology.orchestra.run.gibba002.stdout: } 2021-02-10T11:40:47.842 INFO:teuthology.orchestra.run.gibba002.stdout:] 2021-02-10T11:40:47.843 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:47.840 7fc3ba3f4700 1 -- 172.21.2.102:0/762509449 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc3940186c0 msgr2=0x7fc39401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:47.843 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:47.840 7fc3ba3f4700 1 --2- 172.21.2.102:0/762509449 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc3940186c0 0x7fc39401abc0 crc :-1 s=READY pgs=1490 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:47.843 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:47.840 7fc3ba3f4700 1 -- 172.21.2.102:0/762509449 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc3b409db00 msgr2=0x7fc3b409df00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:47.843 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:47.840 7fc3ba3f4700 1 --2- 172.21.2.102:0/762509449 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc3b409db00 0x7fc3b409df00 secure :-1 s=READY pgs=3424 cs=0 l=1 rev1=1 rx=0x7fc3ac006c50 tx=0x7fc3ac00bd20).stop 2021-02-10T11:40:47.843 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:47.841 7fc3ba3f4700 1 -- 172.21.2.102:0/762509449 shutdown_connections 2021-02-10T11:40:47.844 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:47.841 7fc3ba3f4700 1 --2- 172.21.2.102:0/762509449 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc3b409db00 0x7fc3b409df00 unknown :-1 s=CLOSED pgs=3424 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:47.844 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:47.841 7fc3ba3f4700 1 --2- 172.21.2.102:0/762509449 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc3b409d0e0 0x7fc3b409d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:47.844 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:47.841 7fc3ba3f4700 1 --2- 172.21.2.102:0/762509449 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc3940186c0 0x7fc39401abc0 unknown :-1 s=CLOSED pgs=1490 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:47.844 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:47.841 7fc3ba3f4700 1 --2- 172.21.2.102:0/762509449 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc3b41a35c0 0x7fc3b41a39c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:47.844 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:47.841 7fc3ba3f4700 1 -- 172.21.2.102:0/762509449 >> 172.21.2.102:0/762509449 conn(0x7fc3b40a6b00 msgr2=0x7fc3b4095700 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:47.844 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:47.841 7fc3ba3f4700 1 -- 172.21.2.102:0/762509449 shutdown_connections 2021-02-10T11:40:47.844 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:47.841 7fc3ba3f4700 1 -- 172.21.2.102:0/762509449 wait complete. 2021-02-10T11:40:47.851 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config set mon mon_allow_pool_delete True 2021-02-10T11:40:48.286 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:48.267 7f44d5c67700 1 Processor -- start 2021-02-10T11:40:48.286 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:48.267 7f44d5c67700 1 -- start start 2021-02-10T11:40:48.286 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:48.268 7f44d5c67700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f44d00d1870 0x7f44d01b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:48.287 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:48.268 7f44d5c67700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f44d00cff70 0x7f44d0109130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:48.287 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:48.268 7f44d5c67700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44d00fe1c0 con 0x7f44d01b0ae0 2021-02-10T11:40:48.287 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:48.268 7f44d5c67700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44d0109ad0 con 0x7f44d00cff70 2021-02-10T11:40:48.287 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:48.268 7f44d5c67700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44d0109750 con 0x7f44d00d1870 2021-02-10T11:40:48.287 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:48.268 7f44cffff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f44d00cff70 0x7f44d0109130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:48.287 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:48.268 7f44cffff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f44d00cff70 0x7f44d0109130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:41538/0 (socket says 172.21.2.102:41538) 2021-02-10T11:40:48.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:48.268 7f44cffff700 1 -- 172.21.2.102:0/145007726 learned_addr learned my addr 172.21.2.102:0/145007726 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:40:48.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:48.268 7f44cffff700 1 -- 172.21.2.102:0/145007726 >> v2:172.21.2.113:3300/0 conn(0x7f44d00d1870 msgr2=0x7f44d01b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:48.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:48.268 7f44cffff700 1 --2- 172.21.2.102:0/145007726 >> v2:172.21.2.113:3300/0 conn(0x7f44d00d1870 0x7f44d01b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:48.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:48.268 7f44cffff700 1 -- 172.21.2.102:0/145007726 >> v1:172.21.2.113:6789/0 conn(0x7f44d01b0ae0 legacy=0x7f44d0108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:48.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:48.268 7f44cffff700 1 -- 172.21.2.102:0/145007726 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44d0088070 con 0x7f44d00cff70 2021-02-10T11:40:48.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:48.268 7f44cffff700 1 --2- 172.21.2.102:0/145007726 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f44d00cff70 0x7f44d0109130 secure :-1 s=READY pgs=3425 cs=0 l=1 rev1=1 rx=0x7f44c8006f00 tx=0x7f44c8009e90).ready entity=mon.0 client_cookie=50e1331c1e4173a7 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:48.289 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:48.269 7f44ce7fc700 1 -- 172.21.2.102:0/145007726 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f44c800b300 con 0x7f44d00cff70 2021-02-10T11:40:48.289 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:48.269 7f44ce7fc700 1 -- 172.21.2.102:0/145007726 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44c800b580 con 0x7f44d00cff70 2021-02-10T11:40:48.289 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:48.269 7f44ce7fc700 1 -- 172.21.2.102:0/145007726 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f44c800ba60 con 0x7f44d00cff70 2021-02-10T11:40:48.289 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:48.269 7f44d5c67700 1 -- 172.21.2.102:0/145007726 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f44d00cff70 msgr2=0x7f44d0109130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:48.289 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:48.269 7f44d5c67700 1 --2- 172.21.2.102:0/145007726 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f44d00cff70 0x7f44d0109130 secure :-1 s=READY pgs=3425 cs=0 l=1 rev1=1 rx=0x7f44c8006f00 tx=0x7f44c8009e90).stop 2021-02-10T11:40:48.290 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:48.269 7f44d5c67700 1 -- 172.21.2.102:0/145007726 shutdown_connections 2021-02-10T11:40:48.290 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:48.269 7f44d5c67700 1 --2- 172.21.2.102:0/145007726 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f44d00cff70 0x7f44d0109130 unknown :-1 s=CLOSED pgs=3425 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:48.290 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:48.269 7f44d5c67700 1 --2- 172.21.2.102:0/145007726 >> v2:172.21.2.113:3300/0 conn(0x7f44d00d1870 0x7f44d01b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:48.290 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:48.269 7f44d5c67700 1 -- 172.21.2.102:0/145007726 >> 172.21.2.102:0/145007726 conn(0x7f44d00ce420 msgr2=0x7f44d00d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:48.291 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:48.269 7f44d5c67700 1 -- 172.21.2.102:0/145007726 shutdown_connections 2021-02-10T11:40:48.291 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:48.269 7f44d5c67700 1 -- 172.21.2.102:0/145007726 wait complete. 2021-02-10T11:40:48.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:48.284 7f44d5c67700 1 Processor -- start 2021-02-10T11:40:48.293 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:48.284 7f44d5c67700 1 -- start start 2021-02-10T11:40:48.294 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:48.285 7f44d5c67700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f44d00c2370 0x7f44d018f870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:48.294 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:48.285 7f44d5c67700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f44d0084cb0 0x7f44d01a7470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:48.294 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:48.285 7f44d5c67700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f44d01a7a90 0x7f44d0085700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:48.294 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:48.285 7f44d5c67700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44d00be840 con 0x7f44d00c2370 2021-02-10T11:40:48.294 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:48.285 7f44d5c67700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44d00c29a0 con 0x7f44d01a7a90 2021-02-10T11:40:48.295 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:48.285 7f44d5c67700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44d01a7e90 con 0x7f44d0084cb0 2021-02-10T11:40:48.295 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:48.285 7f44cf7fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f44d00c2370 0x7f44d018f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:48.295 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:48.285 7f44cf7fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f44d00c2370 0x7f44d018f870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:41540/0 (socket says 172.21.2.102:41540) 2021-02-10T11:40:48.295 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:48.285 7f44cf7fe700 1 -- 172.21.2.102:0/3799744200 learned_addr learned my addr 172.21.2.102:0/3799744200 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:40:48.295 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:48.285 7f44cf7fe700 1 -- 172.21.2.102:0/3799744200 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f44d0084cb0 msgr2=0x7f44d01a7470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:48.295 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:48.285 7f44cf7fe700 1 --2- 172.21.2.102:0/3799744200 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f44d0084cb0 0x7f44d01a7470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:48.296 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:48.285 7f44cf7fe700 1 -- 172.21.2.102:0/3799744200 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f44d01a7a90 msgr2=0x7f44d0085700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:48.296 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:48.285 7f44cf7fe700 1 --2- 172.21.2.102:0/3799744200 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f44d01a7a90 0x7f44d0085700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:48.296 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:48.285 7f44cf7fe700 1 -- 172.21.2.102:0/3799744200 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44d01a80b0 con 0x7f44d00c2370 2021-02-10T11:40:48.296 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:48.285 7f44cf7fe700 1 --2- 172.21.2.102:0/3799744200 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f44d00c2370 0x7f44d018f870 secure :-1 s=READY pgs=3426 cs=0 l=1 rev1=1 rx=0x7f44c0006f00 tx=0x7f44c000b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:48.296 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:48.285 7f44b7fff700 1 -- 172.21.2.102:0/3799744200 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f44c0006b90 con 0x7f44d00c2370 2021-02-10T11:40:48.296 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:48.285 7f44b7fff700 1 -- 172.21.2.102:0/3799744200 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44c000bd30 con 0x7f44d00c2370 2021-02-10T11:40:48.297 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:48.285 7f44b7fff700 1 -- 172.21.2.102:0/3799744200 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f44c00020d0 con 0x7f44d00c2370 2021-02-10T11:40:48.297 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:48.285 7f44d5c67700 1 -- 172.21.2.102:0/3799744200 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44d01a80b0 con 0x7f44d00c2370 2021-02-10T11:40:48.297 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:48.285 7f44d5c67700 1 -- 172.21.2.102:0/3799744200 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44d0086010 con 0x7f44d00c2370 2021-02-10T11:40:48.297 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:48.286 7f44b7fff700 1 -- 172.21.2.102:0/3799744200 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f44c0007020 con 0x7f44d00c2370 2021-02-10T11:40:48.297 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:48.286 7f44b7fff700 1 --2- 172.21.2.102:0/3799744200 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f44b00186c0 0x7f44b001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:48.297 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:48.286 7f44b7fff700 1 -- 172.21.2.102:0/3799744200 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f44c0032540 con 0x7f44d00c2370 2021-02-10T11:40:48.298 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:48.286 7f44ceffd700 1 --2- 172.21.2.102:0/3799744200 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f44b00186c0 0x7f44b001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:48.298 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:48.286 7f44ceffd700 1 --2- 172.21.2.102:0/3799744200 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f44b00186c0 0x7f44b001abc0 crc :-1 s=READY pgs=1491 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:48.320 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:48.318 7f44d5c67700 1 -- 172.21.2.102:0/3799744200 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44d0082a70 con 0x7f44d00c2370 2021-02-10T11:40:48.323 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:48.321 7f44b7fff700 1 -- 172.21.2.102:0/3799744200 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f44c0025fe0 con 0x7f44d00c2370 2021-02-10T11:40:48.473 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:48.471 7f44d5c67700 1 -- 172.21.2.102:0/3799744200 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f44d0082a70 con 0x7f44d00c2370 2021-02-10T11:40:48.473 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:48.471 7f44b7fff700 1 -- 172.21.2.102:0/3799744200 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v48)=0 v48) v1 ==== 122+0+0 (secure 0 0 0) 0x7f44c0025fe0 con 0x7f44d00c2370 2021-02-10T11:40:48.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:48.473 7f44d5c67700 1 -- 172.21.2.102:0/3799744200 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f44b00186c0 msgr2=0x7f44b001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:48.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:48.473 7f44d5c67700 1 --2- 172.21.2.102:0/3799744200 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f44b00186c0 0x7f44b001abc0 crc :-1 s=READY pgs=1491 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:48.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:48.473 7f44d5c67700 1 -- 172.21.2.102:0/3799744200 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f44d00c2370 msgr2=0x7f44d018f870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:48.475 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:48.473 7f44d5c67700 1 --2- 172.21.2.102:0/3799744200 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f44d00c2370 0x7f44d018f870 secure :-1 s=READY pgs=3426 cs=0 l=1 rev1=1 rx=0x7f44c0006f00 tx=0x7f44c000b940).stop 2021-02-10T11:40:48.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:48.473 7f44d5c67700 1 -- 172.21.2.102:0/3799744200 shutdown_connections 2021-02-10T11:40:48.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:48.473 7f44d5c67700 1 --2- 172.21.2.102:0/3799744200 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f44b00186c0 0x7f44b001abc0 unknown :-1 s=CLOSED pgs=1491 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:48.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:48.473 7f44d5c67700 1 --2- 172.21.2.102:0/3799744200 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f44d01a7a90 0x7f44d0085700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:48.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:48.473 7f44d5c67700 1 --2- 172.21.2.102:0/3799744200 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f44d0084cb0 0x7f44d01a7470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:48.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:48.473 7f44d5c67700 1 --2- 172.21.2.102:0/3799744200 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f44d00c2370 0x7f44d018f870 unknown :-1 s=CLOSED pgs=3426 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:48.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:48.473 7f44d5c67700 1 -- 172.21.2.102:0/3799744200 >> 172.21.2.102:0/3799744200 conn(0x7f44d00b5dc0 msgr2=0x7f44d00c2d90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:48.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:48.473 7f44d5c67700 1 -- 172.21.2.102:0/3799744200 shutdown_connections 2021-02-10T11:40:48.476 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:48.473 7f44d5c67700 1 -- 172.21.2.102:0/3799744200 wait complete. 2021-02-10T11:40:48.483 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000115792 2021-02-10T11:40:48.916 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:48.897 7f1d3bd44700 1 Processor -- start 2021-02-10T11:40:48.916 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:48.897 7f1d3bd44700 1 -- start start 2021-02-10T11:40:48.917 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:48.898 7f1d3bd44700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1d340d1870 0x7f1d341b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:48.917 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:48.898 7f1d3bd44700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f1d341b0ae0 0x7f1d34108940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:48.917 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:48.898 7f1d3bd44700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f1d340d1c70 0x7f1d34108f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:48.917 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:48.898 7f1d3bd44700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d340e5ef0 con 0x7f1d341b0ae0 2021-02-10T11:40:48.917 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:48.898 7f1d3bd44700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d34109a60 con 0x7f1d340d1c70 2021-02-10T11:40:48.917 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:48.898 7f1d3bd44700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d34088070 con 0x7f1d340d1870 2021-02-10T11:40:48.917 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:48.898 7f1d3935f700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f1d341b0ae0 0x7f1d34108940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:48.917 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:48.898 7f1d3935f700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f1d341b0ae0 0x7f1d34108940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:41548/0 (socket says 172.21.2.102:41548) 2021-02-10T11:40:48.917 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:48.898 7f1d3935f700 1 -- 172.21.2.102:0/3530354725 learned_addr learned my addr 172.21.2.102:0/3530354725 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:40:48.917 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:48.898 7f1d3935f700 1 -- 172.21.2.102:0/3530354725 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1d340d1870 msgr2=0x7f1d341b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:48.917 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:48.898 7f1d3935f700 1 --2- 172.21.2.102:0/3530354725 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1d340d1870 0x7f1d341b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:48.917 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:48.898 7f1d3935f700 1 -- 172.21.2.102:0/3530354725 >> v2:172.21.2.113:3300/0 conn(0x7f1d340d1c70 msgr2=0x7f1d34108f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:48.918 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:48.898 7f1d3935f700 1 --2- 172.21.2.102:0/3530354725 >> v2:172.21.2.113:3300/0 conn(0x7f1d340d1c70 0x7f1d34108f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:48.918 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:48.898 7f1d3935f700 1 -- 172.21.2.102:0/3530354725 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d34109580 con 0x7f1d341b0ae0 2021-02-10T11:40:48.918 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:48.899 7f1d3935f700 1 --2- 172.21.2.102:0/3530354725 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1d341b0ae0 0x7f1d34108940 secure :-1 s=READY pgs=3427 cs=0 l=1 rev1=1 rx=0x7f1d30006700 tx=0x7f1d30009b80).ready entity=mon.0 client_cookie=fd724759bd8f8420 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:48.918 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:48.899 7f1d38b5e700 1 -- 172.21.2.102:0/3530354725 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1d3000c0c0 con 0x7f1d341b0ae0 2021-02-10T11:40:48.918 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:48.899 7f1d38b5e700 1 -- 172.21.2.102:0/3530354725 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d3000c340 con 0x7f1d341b0ae0 2021-02-10T11:40:48.918 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:48.899 7f1d38b5e700 1 -- 172.21.2.102:0/3530354725 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1d3000c860 con 0x7f1d341b0ae0 2021-02-10T11:40:48.918 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:48.899 7f1d3bd44700 1 -- 172.21.2.102:0/3530354725 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1d341b0ae0 msgr2=0x7f1d34108940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:48.918 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:48.899 7f1d3bd44700 1 --2- 172.21.2.102:0/3530354725 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1d341b0ae0 0x7f1d34108940 secure :-1 s=READY pgs=3427 cs=0 l=1 rev1=1 rx=0x7f1d30006700 tx=0x7f1d30009b80).stop 2021-02-10T11:40:48.918 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:48.899 7f1d3bd44700 1 -- 172.21.2.102:0/3530354725 shutdown_connections 2021-02-10T11:40:48.918 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:48.899 7f1d3bd44700 1 --2- 172.21.2.102:0/3530354725 >> v2:172.21.2.113:3300/0 conn(0x7f1d340d1c70 0x7f1d34108f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:48.918 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:48.899 7f1d3bd44700 1 --2- 172.21.2.102:0/3530354725 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1d341b0ae0 0x7f1d34108940 unknown :-1 s=CLOSED pgs=3427 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:48.919 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:48.899 7f1d3bd44700 1 --2- 172.21.2.102:0/3530354725 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1d340d1870 0x7f1d341b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:48.919 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:48.899 7f1d3bd44700 1 -- 172.21.2.102:0/3530354725 >> 172.21.2.102:0/3530354725 conn(0x7f1d340ce420 msgr2=0x7f1d340d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:48.919 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:48.900 7f1d3bd44700 1 -- 172.21.2.102:0/3530354725 shutdown_connections 2021-02-10T11:40:48.919 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:48.900 7f1d3bd44700 1 -- 172.21.2.102:0/3530354725 wait complete. 2021-02-10T11:40:48.920 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:48.915 7f1d3bd44700 1 Processor -- start 2021-02-10T11:40:48.920 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:48.915 7f1d3bd44700 1 -- start start 2021-02-10T11:40:48.920 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:48.915 7f1d3bd44700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1d340a73c0 0x7f1d340a6b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:48.920 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:48.915 7f1d3bd44700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1d340add40 0x7f1d340ae140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:48.921 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:48.915 7f1d3bd44700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1d340ae760 0x7f1d340af180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:48.921 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:48.915 7f1d3bd44700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d340aec00 con 0x7f1d340a73c0 2021-02-10T11:40:48.921 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:48.915 7f1d3bd44700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d340875d0 con 0x7f1d340ae760 2021-02-10T11:40:48.921 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:48.915 7f1d3bd44700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d340af7a0 con 0x7f1d340add40 2021-02-10T11:40:48.921 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:48.915 7f1d39b60700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1d340a73c0 0x7f1d340a6b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:48.921 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:48.915 7f1d39b60700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1d340a73c0 0x7f1d340a6b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:41554/0 (socket says 172.21.2.102:41554) 2021-02-10T11:40:48.921 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:48.915 7f1d39b60700 1 -- 172.21.2.102:0/289318414 learned_addr learned my addr 172.21.2.102:0/289318414 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:40:48.921 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:48.915 7f1d3a361700 1 --2- 172.21.2.102:0/289318414 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1d340ae760 0x7f1d340af180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:48.921 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:48.915 7f1d3935f700 1 --2- 172.21.2.102:0/289318414 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1d340add40 0x7f1d340ae140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:48.921 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:48.916 7f1d39b60700 1 -- 172.21.2.102:0/289318414 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1d340add40 msgr2=0x7f1d340ae140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:48.921 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:48.916 7f1d39b60700 1 --2- 172.21.2.102:0/289318414 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1d340add40 0x7f1d340ae140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:48.921 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:48.916 7f1d39b60700 1 -- 172.21.2.102:0/289318414 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1d340ae760 msgr2=0x7f1d340af180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:48.921 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:48.916 7f1d39b60700 1 --2- 172.21.2.102:0/289318414 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1d340ae760 0x7f1d340af180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:48.921 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:48.916 7f1d39b60700 1 -- 172.21.2.102:0/289318414 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d340af9c0 con 0x7f1d340a73c0 2021-02-10T11:40:48.922 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:48.916 7f1d3935f700 1 --2- 172.21.2.102:0/289318414 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1d340add40 0x7f1d340ae140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:40:48.922 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:48.916 7f1d39b60700 1 --2- 172.21.2.102:0/289318414 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1d340a73c0 0x7f1d340a6b50 secure :-1 s=READY pgs=3428 cs=0 l=1 rev1=1 rx=0x7f1d28006f00 tx=0x7f1d2800b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:48.922 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:48.916 7f1d267fc700 1 -- 172.21.2.102:0/289318414 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1d2800bd30 con 0x7f1d340a73c0 2021-02-10T11:40:48.922 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:48.916 7f1d267fc700 1 -- 172.21.2.102:0/289318414 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d28000c90 con 0x7f1d340a73c0 2021-02-10T11:40:48.922 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:48.916 7f1d267fc700 1 -- 172.21.2.102:0/289318414 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1d280022a0 con 0x7f1d340a73c0 2021-02-10T11:40:48.922 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:48.916 7f1d3bd44700 1 -- 172.21.2.102:0/289318414 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d340af9c0 con 0x7f1d340a73c0 2021-02-10T11:40:48.922 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:48.916 7f1d3bd44700 1 -- 172.21.2.102:0/289318414 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d340af9c0 con 0x7f1d340a73c0 2021-02-10T11:40:48.922 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:48.917 7f1d267fc700 1 -- 172.21.2.102:0/289318414 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1d28005310 con 0x7f1d340a73c0 2021-02-10T11:40:48.922 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:48.917 7f1d267fc700 1 --2- 172.21.2.102:0/289318414 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1d1c0186c0 0x7f1d1c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:48.922 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:48.917 7f1d267fc700 1 -- 172.21.2.102:0/289318414 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f1d28032550 con 0x7f1d340a73c0 2021-02-10T11:40:48.922 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:48.917 7f1d3935f700 1 --2- 172.21.2.102:0/289318414 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1d1c0186c0 0x7f1d1c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:48.922 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:48.917 7f1d3935f700 1 --2- 172.21.2.102:0/289318414 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1d1c0186c0 0x7f1d1c01abc0 crc :-1 s=READY pgs=1492 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:48.950 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:48.948 7f1d3bd44700 1 -- 172.21.2.102:0/289318414 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d34082980 con 0x7f1d340a73c0 2021-02-10T11:40:48.952 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:48.950 7f1d267fc700 1 -- 172.21.2.102:0/289318414 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f1d28006b90 con 0x7f1d340a73c0 2021-02-10T11:40:49.079 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:49.076 7f1d3bd44700 1 -- 172.21.2.102:0/289318414 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000115792", "target": ["mgr", ""]}) v1 -- 0x7f1d34109a60 con 0x7f1d1c0186c0 2021-02-10T11:40:49.082 INFO:tasks.ceph.mgr.y.gibba002.stderr:2021-02-10 11:40:49.080 7fd94bc53700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:40:49.083 INFO:tasks.ceph.mgr.y.gibba002.stderr:2021-02-10 11:40:49.080 7fd94bc53700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:40:49.083 INFO:tasks.ceph.mgr.y.gibba002.stderr:2021-02-10 11:40:49.080 7fd94bc53700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:40:49.083 INFO:tasks.ceph.mgr.y.gibba002.stderr:2021-02-10 11:40:49.080 7fd94bc53700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:40:49.083 INFO:tasks.ceph.mgr.y.gibba002.stderr:2021-02-10 11:40:49.080 7fd94bc53700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:40:49.117 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:49.115 7f1d267fc700 1 -- 172.21.2.102:0/289318414 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f1d300009b0 con 0x7f1d1c0186c0 2021-02-10T11:40:49.144 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:49.142 7f1d3bd44700 1 -- 172.21.2.102:0/289318414 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1d1c0186c0 msgr2=0x7f1d1c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:49.145 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:49.142 7f1d3bd44700 1 --2- 172.21.2.102:0/289318414 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1d1c0186c0 0x7f1d1c01abc0 crc :-1 s=READY pgs=1492 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:49.145 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:49.142 7f1d3bd44700 1 -- 172.21.2.102:0/289318414 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1d340a73c0 msgr2=0x7f1d340a6b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:49.145 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:49.142 7f1d3bd44700 1 --2- 172.21.2.102:0/289318414 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1d340a73c0 0x7f1d340a6b50 secure :-1 s=READY pgs=3428 cs=0 l=1 rev1=1 rx=0x7f1d28006f00 tx=0x7f1d2800b940).stop 2021-02-10T11:40:49.145 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:49.142 7f1d3bd44700 1 -- 172.21.2.102:0/289318414 shutdown_connections 2021-02-10T11:40:49.145 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:49.142 7f1d3bd44700 1 --2- 172.21.2.102:0/289318414 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1d1c0186c0 0x7f1d1c01abc0 unknown :-1 s=CLOSED pgs=1492 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:49.145 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:49.142 7f1d3bd44700 1 --2- 172.21.2.102:0/289318414 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1d340ae760 0x7f1d340af180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:49.145 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:49.142 7f1d3bd44700 1 --2- 172.21.2.102:0/289318414 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1d340add40 0x7f1d340ae140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:49.145 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:49.142 7f1d3bd44700 1 --2- 172.21.2.102:0/289318414 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1d340a73c0 0x7f1d340a6b50 unknown :-1 s=CLOSED pgs=3428 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:49.145 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:49.142 7f1d3bd44700 1 -- 172.21.2.102:0/289318414 >> 172.21.2.102:0/289318414 conn(0x7f1d340be980 msgr2=0x7f1d34092cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:49.145 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:49.142 7f1d3bd44700 1 -- 172.21.2.102:0/289318414 shutdown_connections 2021-02-10T11:40:49.145 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:49.142 7f1d3bd44700 1 -- 172.21.2.102:0/289318414 wait complete. 2021-02-10T11:40:49.153 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000115792 2021-02-10T11:40:49.588 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:49.567 7f24f0e12700 1 Processor -- start 2021-02-10T11:40:49.588 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:49.567 7f24f0e12700 1 -- start start 2021-02-10T11:40:49.588 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:49.568 7f24f0e12700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f24ec0d1870 0x7f24ec1b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:49.588 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:49.568 7f24f0e12700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f24ec1b0ae0 0x7f24ec108940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:49.588 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:49.568 7f24f0e12700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f24ec0d1c70 0x7f24ec108f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:49.588 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:49.568 7f24f0e12700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24ec0e5ef0 con 0x7f24ec0d1c70 2021-02-10T11:40:49.588 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:49.568 7f24f0e12700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24ec109a60 con 0x7f24ec1b0ae0 2021-02-10T11:40:49.588 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:49.568 7f24f0e12700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24ec088070 con 0x7f24ec0d1870 2021-02-10T11:40:49.588 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:49.568 7f24eb314700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f24ec0d1c70 0x7f24ec108f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:49.588 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:49.568 7f24eb314700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f24ec0d1c70 0x7f24ec108f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:41578/0 (socket says 172.21.2.102:41578) 2021-02-10T11:40:49.589 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:49.568 7f24eb314700 1 -- 172.21.2.102:0/3747239569 learned_addr learned my addr 172.21.2.102:0/3747239569 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:40:49.589 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:49.568 7f24eb314700 1 -- 172.21.2.102:0/3747239569 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f24ec0d1870 msgr2=0x7f24ec1b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:49.589 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:49.568 7f24eb314700 1 --2- 172.21.2.102:0/3747239569 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f24ec0d1870 0x7f24ec1b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:49.589 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:49.568 7f24ea312700 1 --2- 172.21.2.102:0/3747239569 >> v2:172.21.2.113:3300/0 conn(0x7f24ec1b0ae0 0x7f24ec108940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:49.589 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:49.568 7f24eb314700 1 -- 172.21.2.102:0/3747239569 >> v2:172.21.2.113:3300/0 conn(0x7f24ec1b0ae0 msgr2=0x7f24ec108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:49.589 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:49.568 7f24eb314700 1 --2- 172.21.2.102:0/3747239569 >> v2:172.21.2.113:3300/0 conn(0x7f24ec1b0ae0 0x7f24ec108940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:49.589 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:49.569 7f24eb314700 1 -- 172.21.2.102:0/3747239569 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24ec109580 con 0x7f24ec0d1c70 2021-02-10T11:40:49.589 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:49.569 7f24eb314700 1 --2- 172.21.2.102:0/3747239569 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f24ec0d1c70 0x7f24ec108f60 secure :-1 s=READY pgs=3431 cs=0 l=1 rev1=1 rx=0x7f24e4006f00 tx=0x7f24e4009e90).ready entity=mon.0 client_cookie=8275e9730c8da59b server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:49.589 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:49.569 7f24e9b11700 1 -- 172.21.2.102:0/3747239569 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f24e400b300 con 0x7f24ec0d1c70 2021-02-10T11:40:49.589 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:49.569 7f24e9b11700 1 -- 172.21.2.102:0/3747239569 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24e400b580 con 0x7f24ec0d1c70 2021-02-10T11:40:49.589 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:49.569 7f24e9b11700 1 -- 172.21.2.102:0/3747239569 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f24e400b9d0 con 0x7f24ec0d1c70 2021-02-10T11:40:49.589 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:49.569 7f24f0e12700 1 -- 172.21.2.102:0/3747239569 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f24ec0d1c70 msgr2=0x7f24ec108f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:49.589 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:49.569 7f24f0e12700 1 --2- 172.21.2.102:0/3747239569 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f24ec0d1c70 0x7f24ec108f60 secure :-1 s=READY pgs=3431 cs=0 l=1 rev1=1 rx=0x7f24e4006f00 tx=0x7f24e4009e90).stop 2021-02-10T11:40:49.590 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:49.570 7f24f0e12700 1 -- 172.21.2.102:0/3747239569 shutdown_connections 2021-02-10T11:40:49.590 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:49.570 7f24f0e12700 1 --2- 172.21.2.102:0/3747239569 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f24ec0d1c70 0x7f24ec108f60 unknown :-1 s=CLOSED pgs=3431 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:49.590 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:49.570 7f24f0e12700 1 --2- 172.21.2.102:0/3747239569 >> v2:172.21.2.113:3300/0 conn(0x7f24ec1b0ae0 0x7f24ec108940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:49.590 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:49.570 7f24f0e12700 1 --2- 172.21.2.102:0/3747239569 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f24ec0d1870 0x7f24ec1b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:49.590 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:49.570 7f24f0e12700 1 -- 172.21.2.102:0/3747239569 >> 172.21.2.102:0/3747239569 conn(0x7f24ec0ce420 msgr2=0x7f24ec0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:49.590 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:49.570 7f24f0e12700 1 -- 172.21.2.102:0/3747239569 shutdown_connections 2021-02-10T11:40:49.590 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:49.570 7f24f0e12700 1 -- 172.21.2.102:0/3747239569 wait complete. 2021-02-10T11:40:49.590 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:49.585 7f24f0e12700 1 Processor -- start 2021-02-10T11:40:49.590 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:49.585 7f24f0e12700 1 -- start start 2021-02-10T11:40:49.590 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:49.585 7f24f0e12700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f24ec0abd60 0x7f24ec0ac160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:49.590 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:49.585 7f24f0e12700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f24ec0ac780 0x7f24ec0acb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:49.590 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:49.585 7f24f0e12700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f24ec083910 0x7f24ec083d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:49.590 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:49.585 7f24f0e12700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24ec087180 con 0x7f24ec083910 2021-02-10T11:40:49.591 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:49.585 7f24f0e12700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24ec18fcd0 con 0x7f24ec0ac780 2021-02-10T11:40:49.591 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:49.585 7f24f0e12700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24ec084330 con 0x7f24ec0abd60 2021-02-10T11:40:49.591 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:49.585 7f24eb314700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f24ec083910 0x7f24ec083d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:49.591 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:49.585 7f24eb314700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f24ec083910 0x7f24ec083d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:41586/0 (socket says 172.21.2.102:41586) 2021-02-10T11:40:49.591 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:49.585 7f24eb314700 1 -- 172.21.2.102:0/1833524026 learned_addr learned my addr 172.21.2.102:0/1833524026 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:40:49.592 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:49.586 7f24ea312700 1 --2- 172.21.2.102:0/1833524026 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f24ec0ac780 0x7f24ec0acb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:49.592 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:49.586 7f24eab13700 1 --2- 172.21.2.102:0/1833524026 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f24ec0abd60 0x7f24ec0ac160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:49.592 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:49.586 7f24eb314700 1 -- 172.21.2.102:0/1833524026 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f24ec0abd60 msgr2=0x7f24ec0ac160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:49.592 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:49.586 7f24eb314700 1 --2- 172.21.2.102:0/1833524026 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f24ec0abd60 0x7f24ec0ac160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:49.593 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:49.586 7f24eb314700 1 -- 172.21.2.102:0/1833524026 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f24ec0ac780 msgr2=0x7f24ec0acb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:49.593 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:49.586 7f24eb314700 1 --2- 172.21.2.102:0/1833524026 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f24ec0ac780 0x7f24ec0acb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:49.593 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:49.586 7f24eb314700 1 -- 172.21.2.102:0/1833524026 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24ec084550 con 0x7f24ec083910 2021-02-10T11:40:49.593 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:49.586 7f24eb314700 1 --2- 172.21.2.102:0/1833524026 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f24ec083910 0x7f24ec083d10 secure :-1 s=READY pgs=3432 cs=0 l=1 rev1=1 rx=0x7f24e400b910 tx=0x7f24e400bcb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:49.593 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:49.586 7f24db7fe700 1 -- 172.21.2.102:0/1833524026 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f24e400bdf0 con 0x7f24ec083910 2021-02-10T11:40:49.593 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:49.586 7f24db7fe700 1 -- 172.21.2.102:0/1833524026 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24e4005820 con 0x7f24ec083910 2021-02-10T11:40:49.593 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:49.586 7f24db7fe700 1 -- 172.21.2.102:0/1833524026 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f24e4005d40 con 0x7f24ec083910 2021-02-10T11:40:49.593 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:49.586 7f24f0e12700 1 -- 172.21.2.102:0/1833524026 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24ec084550 con 0x7f24ec083910 2021-02-10T11:40:49.593 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:49.587 7f24f0e12700 1 -- 172.21.2.102:0/1833524026 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24ec084ae0 con 0x7f24ec083910 2021-02-10T11:40:49.593 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:49.587 7f24db7fe700 1 -- 172.21.2.102:0/1833524026 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f24e4005a00 con 0x7f24ec083910 2021-02-10T11:40:49.593 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:49.587 7f24db7fe700 1 --2- 172.21.2.102:0/1833524026 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f24d00186c0 0x7f24d001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:49.593 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:49.587 7f24db7fe700 1 -- 172.21.2.102:0/1833524026 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f24e40328f0 con 0x7f24ec083910 2021-02-10T11:40:49.593 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:49.587 7f24eab13700 1 --2- 172.21.2.102:0/1833524026 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f24d00186c0 0x7f24d001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:49.593 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:49.588 7f24eab13700 1 --2- 172.21.2.102:0/1833524026 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f24d00186c0 0x7f24d001abc0 crc :-1 s=READY pgs=1493 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:49.620 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:49.618 7f24f0e12700 1 -- 172.21.2.102:0/1833524026 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24ec084d90 con 0x7f24ec083910 2021-02-10T11:40:49.623 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:49.621 7f24db7fe700 1 -- 172.21.2.102:0/1833524026 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f24e4000f90 con 0x7f24ec083910 2021-02-10T11:40:49.754 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:49.751 7f24f0e12700 1 -- 172.21.2.102:0/1833524026 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000115792", "target": ["mgr", ""]}) v1 -- 0x7f24ec084ae0 con 0x7f24d00186c0 2021-02-10T11:40:49.757 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:49.754 7f24db7fe700 1 -- 172.21.2.102:0/1833524026 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f24dc000940 con 0x7f24d00186c0 2021-02-10T11:40:49.761 INFO:teuthology.orchestra.run.gibba002.stdout:/volumes/_nogroup/subvolume_0000000000115792/c18f2d18-55cf-4ba5-8b0a-11694bb4e7e6 2021-02-10T11:40:49.762 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:49.760 7f24f0e12700 1 -- 172.21.2.102:0/1833524026 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f24d00186c0 msgr2=0x7f24d001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:49.763 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:49.760 7f24f0e12700 1 --2- 172.21.2.102:0/1833524026 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f24d00186c0 0x7f24d001abc0 crc :-1 s=READY pgs=1493 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:49.763 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:49.760 7f24f0e12700 1 -- 172.21.2.102:0/1833524026 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f24ec083910 msgr2=0x7f24ec083d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:49.763 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:49.760 7f24f0e12700 1 --2- 172.21.2.102:0/1833524026 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f24ec083910 0x7f24ec083d10 secure :-1 s=READY pgs=3432 cs=0 l=1 rev1=1 rx=0x7f24e400b910 tx=0x7f24e400bcb0).stop 2021-02-10T11:40:49.763 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:49.760 7f24f0e12700 1 -- 172.21.2.102:0/1833524026 shutdown_connections 2021-02-10T11:40:49.763 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:49.760 7f24f0e12700 1 --2- 172.21.2.102:0/1833524026 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f24ec083910 0x7f24ec083d10 unknown :-1 s=CLOSED pgs=3432 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:49.763 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:49.760 7f24f0e12700 1 --2- 172.21.2.102:0/1833524026 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f24d00186c0 0x7f24d001abc0 unknown :-1 s=CLOSED pgs=1493 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:49.763 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:49.760 7f24f0e12700 1 --2- 172.21.2.102:0/1833524026 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f24ec0ac780 0x7f24ec0acb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:49.763 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:49.760 7f24f0e12700 1 --2- 172.21.2.102:0/1833524026 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f24ec0abd60 0x7f24ec0ac160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:49.763 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:49.760 7f24f0e12700 1 -- 172.21.2.102:0/1833524026 >> 172.21.2.102:0/1833524026 conn(0x7f24ec0a2b30 msgr2=0x7f24ec09ce60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:49.763 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:49.760 7f24f0e12700 1 -- 172.21.2.102:0/1833524026 shutdown_connections 2021-02-10T11:40:49.763 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:40:49.760 7f24f0e12700 1 -- 172.21.2.102:0/1833524026 wait complete. 2021-02-10T11:40:49.771 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000115792 with 64 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000115792/c18f2d18-55cf-4ba5-8b0a-11694bb4e7e6 2021-02-10T11:40:49.771 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:40:49.771 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:40:49.771 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:40:50.108 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:40:50.108 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:40:50.109 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000115792/c18f2d18-55cf-4ba5-8b0a-11694bb4e7e6/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:40:50.495 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:40:50.495 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:40:50.495 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0281196 s, 37.3 MB/s 2021-02-10T11:40:50.496 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:40:50.496 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:40:50.496 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:40:50.832 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:40:50.832 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:40:50.832 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000115792/c18f2d18-55cf-4ba5-8b0a-11694bb4e7e6/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:40:51.206 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:40:51.206 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:40:51.206 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0246541 s, 42.5 MB/s 2021-02-10T11:40:51.207 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:40:51.207 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:40:51.207 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:40:51.544 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:40:51.544 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:40:51.544 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000115792/c18f2d18-55cf-4ba5-8b0a-11694bb4e7e6/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:40:51.920 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:40:51.921 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:40:51.921 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0244449 s, 42.9 MB/s 2021-02-10T11:40:51.921 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:40:51.922 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:40:51.922 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:40:52.257 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:40:52.257 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:40:52.258 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000115792/c18f2d18-55cf-4ba5-8b0a-11694bb4e7e6/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:40:52.637 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:40:52.637 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:40:52.637 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0268407 s, 39.1 MB/s 2021-02-10T11:40:52.638 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:40:52.638 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:40:52.639 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:40:52.974 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:40:52.974 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:40:52.974 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000115792/c18f2d18-55cf-4ba5-8b0a-11694bb4e7e6/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:40:53.350 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:40:53.350 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:40:53.350 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.025227 s, 41.6 MB/s 2021-02-10T11:40:53.351 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:40:53.352 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:40:53.352 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:40:53.688 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:40:53.689 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:40:53.689 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000115792/c18f2d18-55cf-4ba5-8b0a-11694bb4e7e6/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:40:54.062 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:40:54.062 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:40:54.062 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0246423 s, 42.6 MB/s 2021-02-10T11:40:54.063 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:40:54.063 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:40:54.063 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:40:54.400 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:40:54.401 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:40:54.401 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000115792/c18f2d18-55cf-4ba5-8b0a-11694bb4e7e6/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:40:54.805 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:40:54.805 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:40:54.806 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0230853 s, 45.4 MB/s 2021-02-10T11:40:54.807 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:40:54.807 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:40:54.807 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:40:55.153 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:40:55.153 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:40:55.153 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000115792/c18f2d18-55cf-4ba5-8b0a-11694bb4e7e6/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:40:55.532 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:40:55.532 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:40:55.532 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0258992 s, 40.5 MB/s 2021-02-10T11:40:55.533 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:40:55.533 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:40:55.533 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:40:55.871 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:40:55.871 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:40:55.872 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000115792/c18f2d18-55cf-4ba5-8b0a-11694bb4e7e6/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:40:56.248 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:40:56.249 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:40:56.249 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0257909 s, 40.7 MB/s 2021-02-10T11:40:56.250 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:40:56.250 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:40:56.250 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:40:56.586 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:40:56.586 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:40:56.587 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000115792/c18f2d18-55cf-4ba5-8b0a-11694bb4e7e6/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:40:56.962 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:40:56.962 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:40:56.962 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0238617 s, 43.9 MB/s 2021-02-10T11:40:56.963 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:40:56.963 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:40:56.963 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:40:57.298 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:40:57.298 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:40:57.298 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000115792/c18f2d18-55cf-4ba5-8b0a-11694bb4e7e6/subvolume_file.10 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:40:57.677 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:40:57.677 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:40:57.677 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0243564 s, 43.1 MB/s 2021-02-10T11:40:57.678 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:40:57.678 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:40:57.678 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:40:58.014 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:40:58.014 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:40:58.015 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000115792/c18f2d18-55cf-4ba5-8b0a-11694bb4e7e6/subvolume_file.11 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:40:58.390 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:40:58.391 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:40:58.391 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0238349 s, 44.0 MB/s 2021-02-10T11:40:58.392 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:40:58.392 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:40:58.392 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:40:58.727 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:40:58.727 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:40:58.728 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000115792/c18f2d18-55cf-4ba5-8b0a-11694bb4e7e6/subvolume_file.12 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:40:59.099 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:40:59.099 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:40:59.099 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0227756 s, 46.0 MB/s 2021-02-10T11:40:59.100 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:40:59.100 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:40:59.100 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:40:59.436 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:40:59.436 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:40:59.437 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000115792/c18f2d18-55cf-4ba5-8b0a-11694bb4e7e6/subvolume_file.13 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:40:59.815 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:40:59.815 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:40:59.815 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0247643 s, 42.3 MB/s 2021-02-10T11:40:59.816 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:40:59.816 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:40:59.816 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:41:00.152 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:41:00.152 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:41:00.153 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000115792/c18f2d18-55cf-4ba5-8b0a-11694bb4e7e6/subvolume_file.14 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:41:00.529 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:41:00.530 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:41:00.530 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0236982 s, 44.2 MB/s 2021-02-10T11:41:00.531 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:41:00.531 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:41:00.531 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:41:00.866 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:41:00.866 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:41:00.866 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000115792/c18f2d18-55cf-4ba5-8b0a-11694bb4e7e6/subvolume_file.15 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:41:01.243 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:41:01.243 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:41:01.243 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0236143 s, 44.4 MB/s 2021-02-10T11:41:01.244 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:41:01.244 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:41:01.245 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:41:01.581 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:41:01.582 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:41:01.582 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000115792/c18f2d18-55cf-4ba5-8b0a-11694bb4e7e6/subvolume_file.16 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:41:01.953 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:41:01.954 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:41:01.954 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.022212 s, 47.2 MB/s 2021-02-10T11:41:01.955 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:41:01.955 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:41:01.955 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:41:02.291 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:41:02.291 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:41:02.291 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000115792/c18f2d18-55cf-4ba5-8b0a-11694bb4e7e6/subvolume_file.17 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:41:02.666 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:41:02.666 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:41:02.667 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0256893 s, 40.8 MB/s 2021-02-10T11:41:02.667 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:41:02.668 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:41:02.668 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:41:03.004 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:41:03.004 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:41:03.004 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000115792/c18f2d18-55cf-4ba5-8b0a-11694bb4e7e6/subvolume_file.18 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:41:03.378 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:41:03.378 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:41:03.378 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0225442 s, 46.5 MB/s 2021-02-10T11:41:03.379 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:41:03.379 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:41:03.379 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:41:03.715 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:41:03.715 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:41:03.716 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000115792/c18f2d18-55cf-4ba5-8b0a-11694bb4e7e6/subvolume_file.19 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:41:04.088 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:41:04.088 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:41:04.088 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0225464 s, 46.5 MB/s 2021-02-10T11:41:04.089 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:41:04.089 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:41:04.089 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:41:04.425 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:41:04.425 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:41:04.426 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000115792/c18f2d18-55cf-4ba5-8b0a-11694bb4e7e6/subvolume_file.20 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:41:04.800 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:41:04.800 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:41:04.800 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0219999 s, 47.7 MB/s 2021-02-10T11:41:04.801 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:41:04.801 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:41:04.801 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:41:05.136 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:41:05.136 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:41:05.137 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000115792/c18f2d18-55cf-4ba5-8b0a-11694bb4e7e6/subvolume_file.21 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:41:05.513 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:41:05.513 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:41:05.513 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0237506 s, 44.1 MB/s 2021-02-10T11:41:05.514 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:41:05.514 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:41:05.515 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:41:05.850 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:41:05.850 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:41:05.851 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000115792/c18f2d18-55cf-4ba5-8b0a-11694bb4e7e6/subvolume_file.22 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:41:06.227 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:41:06.228 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:41:06.228 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0238426 s, 44.0 MB/s 2021-02-10T11:41:06.229 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:41:06.229 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:41:06.229 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:41:06.564 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:41:06.564 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:41:06.565 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000115792/c18f2d18-55cf-4ba5-8b0a-11694bb4e7e6/subvolume_file.23 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:41:06.939 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:41:06.939 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:41:06.939 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0229105 s, 45.8 MB/s 2021-02-10T11:41:06.940 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:41:06.940 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:41:06.940 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:41:07.276 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:41:07.276 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:41:07.276 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000115792/c18f2d18-55cf-4ba5-8b0a-11694bb4e7e6/subvolume_file.24 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:41:07.654 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:41:07.655 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:41:07.655 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0250215 s, 41.9 MB/s 2021-02-10T11:41:07.655 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:41:07.656 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:41:07.656 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:41:07.991 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:41:07.992 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:41:07.992 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000115792/c18f2d18-55cf-4ba5-8b0a-11694bb4e7e6/subvolume_file.25 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:41:08.368 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:41:08.368 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:41:08.368 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0236442 s, 44.3 MB/s 2021-02-10T11:41:08.369 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:41:08.369 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:41:08.369 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:41:08.705 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:41:08.705 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:41:08.705 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000115792/c18f2d18-55cf-4ba5-8b0a-11694bb4e7e6/subvolume_file.26 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:41:09.079 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:41:09.079 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:41:09.079 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0237428 s, 44.2 MB/s 2021-02-10T11:41:09.080 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:41:09.080 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:41:09.080 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:41:09.415 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:41:09.415 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:41:09.416 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000115792/c18f2d18-55cf-4ba5-8b0a-11694bb4e7e6/subvolume_file.27 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:41:09.794 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:41:09.794 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:41:09.794 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.025838 s, 40.6 MB/s 2021-02-10T11:41:09.795 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:41:09.795 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:41:09.795 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:41:10.133 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:41:10.133 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:41:10.133 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000115792/c18f2d18-55cf-4ba5-8b0a-11694bb4e7e6/subvolume_file.28 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:41:10.510 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:41:10.510 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:41:10.510 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0253321 s, 41.4 MB/s 2021-02-10T11:41:10.511 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:41:10.511 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:41:10.511 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:41:10.847 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:41:10.847 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:41:10.847 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000115792/c18f2d18-55cf-4ba5-8b0a-11694bb4e7e6/subvolume_file.29 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:41:11.222 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:41:11.222 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:41:11.222 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0234068 s, 44.8 MB/s 2021-02-10T11:41:11.223 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:41:11.223 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:41:11.223 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:41:11.559 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:41:11.559 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:41:11.560 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000115792/c18f2d18-55cf-4ba5-8b0a-11694bb4e7e6/subvolume_file.30 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:41:11.935 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:41:11.935 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:41:11.935 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0227173 s, 46.2 MB/s 2021-02-10T11:41:11.936 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:41:11.936 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:41:11.937 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:41:12.273 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:41:12.273 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:41:12.274 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000115792/c18f2d18-55cf-4ba5-8b0a-11694bb4e7e6/subvolume_file.31 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:41:12.648 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:41:12.648 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:41:12.648 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.022683 s, 46.2 MB/s 2021-02-10T11:41:12.649 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:41:12.649 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:41:12.649 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:41:12.986 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:41:12.987 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:41:12.987 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000115792/c18f2d18-55cf-4ba5-8b0a-11694bb4e7e6/subvolume_file.32 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:41:13.364 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:41:13.364 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:41:13.364 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.025208 s, 41.6 MB/s 2021-02-10T11:41:13.365 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:41:13.365 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:41:13.365 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:41:13.700 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:41:13.701 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:41:13.701 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000115792/c18f2d18-55cf-4ba5-8b0a-11694bb4e7e6/subvolume_file.33 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:41:14.073 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:41:14.073 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:41:14.073 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0236719 s, 44.3 MB/s 2021-02-10T11:41:14.074 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:41:14.074 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:41:14.074 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:41:14.410 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:41:14.410 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:41:14.411 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000115792/c18f2d18-55cf-4ba5-8b0a-11694bb4e7e6/subvolume_file.34 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:41:14.785 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:41:14.785 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:41:14.785 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0227349 s, 46.1 MB/s 2021-02-10T11:41:14.786 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:41:14.786 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:41:14.786 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:41:15.121 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:41:15.121 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:41:15.121 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000115792/c18f2d18-55cf-4ba5-8b0a-11694bb4e7e6/subvolume_file.35 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:41:15.499 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:41:15.499 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:41:15.499 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0257259 s, 40.8 MB/s 2021-02-10T11:41:15.500 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:41:15.500 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:41:15.500 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:41:15.835 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:41:15.836 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:41:15.836 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000115792/c18f2d18-55cf-4ba5-8b0a-11694bb4e7e6/subvolume_file.36 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:41:16.213 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:41:16.213 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:41:16.214 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0246647 s, 42.5 MB/s 2021-02-10T11:41:16.215 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:41:16.215 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:41:16.215 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:41:16.552 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:41:16.552 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:41:16.552 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000115792/c18f2d18-55cf-4ba5-8b0a-11694bb4e7e6/subvolume_file.37 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:41:16.736 DEBUG:teuthology.orchestra.run.gibba002:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:41:16.740 DEBUG:teuthology.orchestra.run.gibba013:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:41:16.929 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:41:16.929 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:41:16.929 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0244275 s, 42.9 MB/s 2021-02-10T11:41:16.930 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:41:16.930 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:41:16.930 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:41:17.265 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:41:17.266 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:41:17.266 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000115792/c18f2d18-55cf-4ba5-8b0a-11694bb4e7e6/subvolume_file.38 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:41:17.641 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:41:17.641 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:41:17.641 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.023643 s, 44.4 MB/s 2021-02-10T11:41:17.642 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:41:17.642 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:41:17.642 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:41:17.978 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:41:17.978 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:41:17.979 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000115792/c18f2d18-55cf-4ba5-8b0a-11694bb4e7e6/subvolume_file.39 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:41:18.355 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:41:18.355 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:41:18.355 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0231949 s, 45.2 MB/s 2021-02-10T11:41:18.356 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:41:18.356 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:41:18.356 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:41:18.692 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:41:18.692 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:41:18.692 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000115792/c18f2d18-55cf-4ba5-8b0a-11694bb4e7e6/subvolume_file.40 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:41:19.066 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:41:19.066 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:41:19.066 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0235192 s, 44.6 MB/s 2021-02-10T11:41:19.068 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:41:19.068 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:41:19.068 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:41:19.405 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:41:19.405 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:41:19.406 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000115792/c18f2d18-55cf-4ba5-8b0a-11694bb4e7e6/subvolume_file.41 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:41:19.783 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:41:19.783 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:41:19.783 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0239188 s, 43.8 MB/s 2021-02-10T11:41:19.784 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:41:19.784 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:41:19.784 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:41:20.121 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:41:20.122 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:41:20.122 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000115792/c18f2d18-55cf-4ba5-8b0a-11694bb4e7e6/subvolume_file.42 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:41:20.497 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:41:20.497 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:41:20.497 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0235251 s, 44.6 MB/s 2021-02-10T11:41:20.498 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:41:20.499 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:41:20.499 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:41:20.834 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:41:20.834 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:41:20.835 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000115792/c18f2d18-55cf-4ba5-8b0a-11694bb4e7e6/subvolume_file.43 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:41:21.212 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:41:21.212 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:41:21.213 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0248065 s, 42.3 MB/s 2021-02-10T11:41:21.213 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:41:21.214 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:41:21.214 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:41:21.550 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:41:21.550 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:41:21.551 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000115792/c18f2d18-55cf-4ba5-8b0a-11694bb4e7e6/subvolume_file.44 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:41:21.926 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:41:21.926 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:41:21.927 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.023774 s, 44.1 MB/s 2021-02-10T11:41:21.927 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:41:21.928 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:41:21.928 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:41:22.264 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:41:22.264 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:41:22.264 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000115792/c18f2d18-55cf-4ba5-8b0a-11694bb4e7e6/subvolume_file.45 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:41:22.643 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:41:22.643 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:41:22.643 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0263709 s, 39.8 MB/s 2021-02-10T11:41:22.644 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:41:22.644 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:41:22.644 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:41:22.980 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:41:22.980 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:41:22.981 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000115792/c18f2d18-55cf-4ba5-8b0a-11694bb4e7e6/subvolume_file.46 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:41:23.355 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:41:23.355 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:41:23.355 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.023335 s, 44.9 MB/s 2021-02-10T11:41:23.356 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:41:23.357 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:41:23.357 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:41:23.693 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:41:23.693 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:41:23.694 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000115792/c18f2d18-55cf-4ba5-8b0a-11694bb4e7e6/subvolume_file.47 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:41:24.066 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:41:24.066 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:41:24.067 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0239123 s, 43.9 MB/s 2021-02-10T11:41:24.067 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:41:24.068 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:41:24.068 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:41:24.403 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:41:24.404 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:41:24.404 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000115792/c18f2d18-55cf-4ba5-8b0a-11694bb4e7e6/subvolume_file.48 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:41:24.779 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:41:24.779 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:41:24.780 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.022958 s, 45.7 MB/s 2021-02-10T11:41:24.781 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:41:24.781 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:41:24.782 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:41:25.118 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:41:25.119 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:41:25.119 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000115792/c18f2d18-55cf-4ba5-8b0a-11694bb4e7e6/subvolume_file.49 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:41:25.504 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:41:25.504 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:41:25.504 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0241666 s, 43.4 MB/s 2021-02-10T11:41:25.505 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:41:25.506 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:41:25.506 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:41:25.850 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:41:25.850 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:41:25.850 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000115792/c18f2d18-55cf-4ba5-8b0a-11694bb4e7e6/subvolume_file.50 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:41:26.226 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:41:26.226 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:41:26.226 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0246554 s, 42.5 MB/s 2021-02-10T11:41:26.227 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:41:26.227 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:41:26.227 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:41:26.563 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:41:26.569 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:41:26.569 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000115792/c18f2d18-55cf-4ba5-8b0a-11694bb4e7e6/subvolume_file.51 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:41:26.943 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:41:26.944 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:41:26.944 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0238272 s, 44.0 MB/s 2021-02-10T11:41:26.944 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:41:26.945 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:41:26.945 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:41:27.280 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:41:27.280 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:41:27.280 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000115792/c18f2d18-55cf-4ba5-8b0a-11694bb4e7e6/subvolume_file.52 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:41:27.656 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:41:27.656 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:41:27.657 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0236079 s, 44.4 MB/s 2021-02-10T11:41:27.657 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:41:27.658 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:41:27.658 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:41:27.993 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:41:27.993 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:41:27.993 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000115792/c18f2d18-55cf-4ba5-8b0a-11694bb4e7e6/subvolume_file.53 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:41:28.369 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:41:28.369 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:41:28.369 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.022903 s, 45.8 MB/s 2021-02-10T11:41:28.370 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:41:28.370 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:41:28.370 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:41:28.706 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:41:28.707 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:41:28.707 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000115792/c18f2d18-55cf-4ba5-8b0a-11694bb4e7e6/subvolume_file.54 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:41:29.081 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:41:29.081 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:41:29.081 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0233972 s, 44.8 MB/s 2021-02-10T11:41:29.082 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:41:29.082 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:41:29.082 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:41:29.419 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:41:29.419 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:41:29.420 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000115792/c18f2d18-55cf-4ba5-8b0a-11694bb4e7e6/subvolume_file.55 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:41:29.793 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:41:29.793 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:41:29.794 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0224342 s, 46.7 MB/s 2021-02-10T11:41:29.794 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:41:29.795 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:41:29.795 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:41:30.130 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:41:30.130 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:41:30.130 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000115792/c18f2d18-55cf-4ba5-8b0a-11694bb4e7e6/subvolume_file.56 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:41:30.506 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:41:30.506 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:41:30.506 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0248053 s, 42.3 MB/s 2021-02-10T11:41:30.507 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:41:30.507 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:41:30.507 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:41:30.843 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:41:30.844 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:41:30.844 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000115792/c18f2d18-55cf-4ba5-8b0a-11694bb4e7e6/subvolume_file.57 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:41:31.220 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:41:31.220 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:41:31.220 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0236393 s, 44.4 MB/s 2021-02-10T11:41:31.221 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:41:31.221 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:41:31.221 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:41:31.556 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:41:31.557 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:41:31.557 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000115792/c18f2d18-55cf-4ba5-8b0a-11694bb4e7e6/subvolume_file.58 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:41:31.932 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:41:31.933 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:41:31.933 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0243154 s, 43.1 MB/s 2021-02-10T11:41:31.933 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:41:31.934 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:41:31.934 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:41:32.269 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:41:32.269 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:41:32.269 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000115792/c18f2d18-55cf-4ba5-8b0a-11694bb4e7e6/subvolume_file.59 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:41:32.646 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:41:32.646 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:41:32.646 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0253659 s, 41.3 MB/s 2021-02-10T11:41:32.647 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:41:32.648 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:41:32.648 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:41:32.983 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:41:32.983 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:41:32.984 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000115792/c18f2d18-55cf-4ba5-8b0a-11694bb4e7e6/subvolume_file.60 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:41:33.359 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:41:33.359 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:41:33.360 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0238018 s, 44.1 MB/s 2021-02-10T11:41:33.361 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:41:33.361 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:41:33.361 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:41:33.698 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:41:33.698 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:41:33.699 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000115792/c18f2d18-55cf-4ba5-8b0a-11694bb4e7e6/subvolume_file.61 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:41:34.070 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:41:34.070 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:41:34.070 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0228826 s, 45.8 MB/s 2021-02-10T11:41:34.071 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:41:34.071 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:41:34.071 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:41:34.407 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:41:34.408 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:41:34.408 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000115792/c18f2d18-55cf-4ba5-8b0a-11694bb4e7e6/subvolume_file.62 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:41:34.784 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:41:34.784 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:41:34.784 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0243179 s, 43.1 MB/s 2021-02-10T11:41:34.785 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:41:34.786 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:41:34.786 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:41:35.122 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:41:35.122 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:41:35.123 DEBUG:teuthology.orchestra.run.gibba002:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000115792/c18f2d18-55cf-4ba5-8b0a-11694bb4e7e6/subvolume_file.63 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:41:35.499 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records in 2021-02-10T11:41:35.499 INFO:teuthology.orchestra.run.gibba002.stderr:1+0 records out 2021-02-10T11:41:35.499 INFO:teuthology.orchestra.run.gibba002.stderr:1048576 bytes (1.0 MB) copied, 0.0236098 s, 44.4 MB/s 2021-02-10T11:41:35.500 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume snapshot create cephfs subvolume_0000000000115792 snapshot_0000000000245284 2021-02-10T11:41:35.933 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:35.916 7f16710d8700 1 Processor -- start 2021-02-10T11:41:35.934 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:35.916 7f16710d8700 1 -- start start 2021-02-10T11:41:35.934 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:35.916 7f16710d8700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f166c0d1880 0x7f166c1b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:35.934 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:35.916 7f16710d8700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f166c1b0af0 0x7f166c108950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:35.934 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:35.916 7f16710d8700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f166c0d1c80 0x7f166c108f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:35.934 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:35.916 7f16710d8700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f166c0fe1d0 con 0x7f166c0d1880 2021-02-10T11:41:35.934 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:35.916 7f16710d8700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f166c109a70 con 0x7f166c0d1c80 2021-02-10T11:41:35.934 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:35.916 7f16710d8700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f166c088070 con 0x7f166c1b0af0 2021-02-10T11:41:35.934 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:35.916 7f166ad9d700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f166c0d1880 0x7f166c1b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:35.934 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:35.916 7f166ad9d700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f166c0d1880 0x7f166c1b04d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:41600/0 (socket says 172.21.2.102:41600) 2021-02-10T11:41:35.934 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:35.916 7f166ad9d700 1 -- 172.21.2.102:0/507885780 learned_addr learned my addr 172.21.2.102:0/507885780 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:41:35.935 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:35.916 7f166259c700 1 --2- 172.21.2.102:0/507885780 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f166c1b0af0 0x7f166c108950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:35.935 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:35.916 7f166b59e700 1 --2- 172.21.2.102:0/507885780 >> v2:172.21.2.113:3300/0 conn(0x7f166c0d1c80 0x7f166c108f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:35.935 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:35.917 7f166ad9d700 1 -- 172.21.2.102:0/507885780 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f166c1b0af0 msgr2=0x7f166c108950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:41:35.935 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:35.917 7f166ad9d700 1 --2- 172.21.2.102:0/507885780 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f166c1b0af0 0x7f166c108950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:35.935 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:35.917 7f166ad9d700 1 -- 172.21.2.102:0/507885780 >> v2:172.21.2.113:3300/0 conn(0x7f166c0d1c80 msgr2=0x7f166c108f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:41:35.935 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:35.917 7f166ad9d700 1 --2- 172.21.2.102:0/507885780 >> v2:172.21.2.113:3300/0 conn(0x7f166c0d1c80 0x7f166c108f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:35.935 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:35.917 7f166ad9d700 1 -- 172.21.2.102:0/507885780 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f166c109590 con 0x7f166c0d1880 2021-02-10T11:41:35.935 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:35.917 7f166ad9d700 1 --2- 172.21.2.102:0/507885780 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f166c0d1880 0x7f166c1b04d0 secure :-1 s=READY pgs=3433 cs=0 l=1 rev1=1 rx=0x7f165c006700 tx=0x7f165c009b80).ready entity=mon.0 client_cookie=8c48118cedd29532 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:35.935 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:35.917 7f166a59c700 1 -- 172.21.2.102:0/507885780 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f165c00c0c0 con 0x7f166c0d1880 2021-02-10T11:41:35.935 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:35.917 7f166a59c700 1 -- 172.21.2.102:0/507885780 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f165c00c340 con 0x7f166c0d1880 2021-02-10T11:41:35.935 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:35.917 7f166a59c700 1 -- 172.21.2.102:0/507885780 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f165c00c860 con 0x7f166c0d1880 2021-02-10T11:41:35.935 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:35.917 7f16710d8700 1 -- 172.21.2.102:0/507885780 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f166c0d1880 msgr2=0x7f166c1b04d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:35.936 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:35.917 7f16710d8700 1 --2- 172.21.2.102:0/507885780 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f166c0d1880 0x7f166c1b04d0 secure :-1 s=READY pgs=3433 cs=0 l=1 rev1=1 rx=0x7f165c006700 tx=0x7f165c009b80).stop 2021-02-10T11:41:35.936 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:35.918 7f16710d8700 1 -- 172.21.2.102:0/507885780 shutdown_connections 2021-02-10T11:41:35.936 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:35.918 7f16710d8700 1 --2- 172.21.2.102:0/507885780 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f166c1b0af0 0x7f166c108950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:35.936 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:35.918 7f16710d8700 1 --2- 172.21.2.102:0/507885780 >> v2:172.21.2.113:3300/0 conn(0x7f166c0d1c80 0x7f166c108f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:35.936 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:35.918 7f16710d8700 1 --2- 172.21.2.102:0/507885780 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f166c0d1880 0x7f166c1b04d0 unknown :-1 s=CLOSED pgs=3433 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:35.936 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:35.918 7f16710d8700 1 -- 172.21.2.102:0/507885780 >> 172.21.2.102:0/507885780 conn(0x7f166c0ce430 msgr2=0x7f166c0d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:35.936 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:35.918 7f16710d8700 1 -- 172.21.2.102:0/507885780 shutdown_connections 2021-02-10T11:41:35.936 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:35.918 7f16710d8700 1 -- 172.21.2.102:0/507885780 wait complete. 2021-02-10T11:41:35.938 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:35.933 7f16710d8700 1 Processor -- start 2021-02-10T11:41:35.938 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:35.933 7f16710d8700 1 -- start start 2021-02-10T11:41:35.938 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:35.933 7f16710d8700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f166c1a3570 0x7f166c1a3970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:35.938 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:35.933 7f16710d8700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f166c09c9f0 0x7f166c09cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:35.938 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:35.933 7f16710d8700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f166c09d410 0x7f166c098170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:35.938 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:35.933 7f16710d8700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f166c09c470 con 0x7f166c1a3570 2021-02-10T11:41:35.938 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:35.933 7f16710d8700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f166c0b2840 con 0x7f166c09c9f0 2021-02-10T11:41:35.938 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:35.933 7f16710d8700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f166c0b2a20 con 0x7f166c09d410 2021-02-10T11:41:35.938 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:35.933 7f166ad9d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f166c1a3570 0x7f166c1a3970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:35.939 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:35.933 7f166ad9d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f166c1a3570 0x7f166c1a3970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:41608/0 (socket says 172.21.2.102:41608) 2021-02-10T11:41:35.939 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:35.933 7f166ad9d700 1 -- 172.21.2.102:0/2412734857 learned_addr learned my addr 172.21.2.102:0/2412734857 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:41:35.939 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:35.934 7f166b59e700 1 --2- 172.21.2.102:0/2412734857 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f166c09d410 0x7f166c098170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:35.939 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:35.934 7f166259c700 1 --2- 172.21.2.102:0/2412734857 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f166c09c9f0 0x7f166c09cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:35.939 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:35.934 7f166ad9d700 1 -- 172.21.2.102:0/2412734857 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f166c09d410 msgr2=0x7f166c098170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:35.939 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:35.934 7f166ad9d700 1 --2- 172.21.2.102:0/2412734857 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f166c09d410 0x7f166c098170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:35.939 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:35.934 7f166ad9d700 1 -- 172.21.2.102:0/2412734857 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f166c09c9f0 msgr2=0x7f166c09cdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:35.939 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:35.934 7f166ad9d700 1 --2- 172.21.2.102:0/2412734857 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f166c09c9f0 0x7f166c09cdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:35.939 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:35.934 7f166ad9d700 1 -- 172.21.2.102:0/2412734857 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f166c09d880 con 0x7f166c1a3570 2021-02-10T11:41:35.939 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:35.934 7f166ad9d700 1 --2- 172.21.2.102:0/2412734857 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f166c1a3570 0x7f166c1a3970 secure :-1 s=READY pgs=3434 cs=0 l=1 rev1=1 rx=0x7f165c00b650 tx=0x7f165c00b9f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:35.939 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:35.934 7f1663fff700 1 -- 172.21.2.102:0/2412734857 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f165c00bde0 con 0x7f166c1a3570 2021-02-10T11:41:35.939 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:35.934 7f1663fff700 1 -- 172.21.2.102:0/2412734857 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f165c006500 con 0x7f166c1a3570 2021-02-10T11:41:35.939 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:35.934 7f16710d8700 1 -- 172.21.2.102:0/2412734857 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f166c09d880 con 0x7f166c1a3570 2021-02-10T11:41:35.939 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:35.934 7f1663fff700 1 -- 172.21.2.102:0/2412734857 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f165c002220 con 0x7f166c1a3570 2021-02-10T11:41:35.940 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:35.934 7f16710d8700 1 -- 172.21.2.102:0/2412734857 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f166c098a80 con 0x7f166c1a3570 2021-02-10T11:41:35.940 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:35.935 7f1663fff700 1 -- 172.21.2.102:0/2412734857 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f165c000c90 con 0x7f166c1a3570 2021-02-10T11:41:35.940 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:35.935 7f1663fff700 1 --2- 172.21.2.102:0/2412734857 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1650018730 0x7f165001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:35.940 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:35.935 7f1663fff700 1 -- 172.21.2.102:0/2412734857 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f165c032670 con 0x7f166c1a3570 2021-02-10T11:41:35.940 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:35.935 7f166259c700 1 --2- 172.21.2.102:0/2412734857 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1650018730 0x7f165001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:35.940 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:35.935 7f166259c700 1 --2- 172.21.2.102:0/2412734857 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1650018730 0x7f165001ac10 crc :-1 s=READY pgs=1494 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:35.967 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:35.965 7f16710d8700 1 -- 172.21.2.102:0/2412734857 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f166c082a70 con 0x7f166c1a3570 2021-02-10T11:41:35.969 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:35.968 7f1663fff700 1 -- 172.21.2.102:0/2412734857 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f165c006500 con 0x7f166c1a3570 2021-02-10T11:41:36.098 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:36.096 7f16710d8700 1 -- 172.21.2.102:0/2412734857 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume snapshot create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000115792", "target": ["mgr", ""], "snap_name": "snapshot_0000000000245284"}) v1 -- 0x7f166c09c470 con 0x7f1650018730 2021-02-10T11:41:36.112 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:36.111 7f1663fff700 1 -- 172.21.2.102:0/2412734857 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f1654006a60 con 0x7f1650018730 2021-02-10T11:41:36.114 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:36.113 7f16710d8700 1 -- 172.21.2.102:0/2412734857 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1650018730 msgr2=0x7f165001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:36.114 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:36.113 7f16710d8700 1 --2- 172.21.2.102:0/2412734857 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1650018730 0x7f165001ac10 crc :-1 s=READY pgs=1494 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:36.115 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:36.113 7f16710d8700 1 -- 172.21.2.102:0/2412734857 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f166c1a3570 msgr2=0x7f166c1a3970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:36.115 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:36.113 7f16710d8700 1 --2- 172.21.2.102:0/2412734857 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f166c1a3570 0x7f166c1a3970 secure :-1 s=READY pgs=3434 cs=0 l=1 rev1=1 rx=0x7f165c00b650 tx=0x7f165c00b9f0).stop 2021-02-10T11:41:36.115 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:36.113 7f16710d8700 1 -- 172.21.2.102:0/2412734857 shutdown_connections 2021-02-10T11:41:36.115 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:36.113 7f16710d8700 1 --2- 172.21.2.102:0/2412734857 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f166c09d410 0x7f166c098170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:36.115 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:36.113 7f16710d8700 1 --2- 172.21.2.102:0/2412734857 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1650018730 0x7f165001ac10 unknown :-1 s=CLOSED pgs=1494 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:36.115 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:36.113 7f16710d8700 1 --2- 172.21.2.102:0/2412734857 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f166c09c9f0 0x7f166c09cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:36.115 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:36.113 7f16710d8700 1 --2- 172.21.2.102:0/2412734857 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f166c1a3570 0x7f166c1a3970 unknown :-1 s=CLOSED pgs=3434 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:36.115 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:36.113 7f16710d8700 1 -- 172.21.2.102:0/2412734857 >> 172.21.2.102:0/2412734857 conn(0x7f166c0c2cd0 msgr2=0x7f166c0b5950 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:36.115 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:36.113 7f16710d8700 1 -- 172.21.2.102:0/2412734857 shutdown_connections 2021-02-10T11:41:36.115 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:36.113 7f16710d8700 1 -- 172.21.2.102:0/2412734857 wait complete. 2021-02-10T11:41:36.122 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume snapshot clone cephfs subvolume_0000000000115792 snapshot_0000000000245284 clone_0000000000944983 2021-02-10T11:41:36.555 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:36.538 7fd9feb43700 1 Processor -- start 2021-02-10T11:41:36.555 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:36.538 7fd9feb43700 1 -- start start 2021-02-10T11:41:36.555 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:36.538 7fd9feb43700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fd9f80d1c90 0x7fd9f80cef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:36.556 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:36.538 7fd9feb43700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9f80d01a0 con 0x7fd9f80cf5b0 2021-02-10T11:41:36.556 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:36.538 7fd9feb43700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9f8109a60 con 0x7fd9f80d1890 2021-02-10T11:41:36.556 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:36.538 7fd9feb43700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9f80e5f50 con 0x7fd9f80d1c90 2021-02-10T11:41:36.556 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:36.538 7fd9f7fff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fd9f80d1c90 0x7fd9f80cef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:36.556 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:36.538 7fd9f7fff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fd9f80d1c90 0x7fd9f80cef90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:41614/0 (socket says 172.21.2.102:41614) 2021-02-10T11:41:36.556 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:36.538 7fd9f7fff700 1 -- 172.21.2.102:0/1324278331 learned_addr learned my addr 172.21.2.102:0/1324278331 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:41:36.556 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:36.539 7fd9f7fff700 1 -- 172.21.2.102:0/1324278331 >> v1:172.21.2.113:6789/0 conn(0x7fd9f80d1890 legacy=0x7fd9f81b03d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:41:36.556 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:36.539 7fd9f7fff700 1 -- 172.21.2.102:0/1324278331 >> v1:172.21.2.102:6789/0 conn(0x7fd9f80cf5b0 legacy=0x7fd9f80cf9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:36.556 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:36.539 7fd9f7fff700 1 -- 172.21.2.102:0/1324278331 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9f81b0bc0 con 0x7fd9f80d1c90 2021-02-10T11:41:36.556 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:36.539 7fd9f7fff700 1 --2- 172.21.2.102:0/1324278331 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd9f80d1c90 0x7fd9f80cef90 secure :-1 s=READY pgs=3436 cs=0 l=1 rev1=1 rx=0x7fd9e8006700 tx=0x7fd9e8009b80).ready entity=mon.0 client_cookie=6df026ef2b594719 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:36.556 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:36.539 7fd9f77fe700 1 -- 172.21.2.102:0/1324278331 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd9e800c0c0 con 0x7fd9f80d1c90 2021-02-10T11:41:36.556 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:36.539 7fd9f77fe700 1 -- 172.21.2.102:0/1324278331 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9e800c380 con 0x7fd9f80d1c90 2021-02-10T11:41:36.556 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:36.539 7fd9f77fe700 1 -- 172.21.2.102:0/1324278331 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd9e800c7d0 con 0x7fd9f80d1c90 2021-02-10T11:41:36.557 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:36.539 7fd9feb43700 1 -- 172.21.2.102:0/1324278331 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd9f80d1c90 msgr2=0x7fd9f80cef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:36.557 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:36.539 7fd9feb43700 1 --2- 172.21.2.102:0/1324278331 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd9f80d1c90 0x7fd9f80cef90 secure :-1 s=READY pgs=3436 cs=0 l=1 rev1=1 rx=0x7fd9e8006700 tx=0x7fd9e8009b80).stop 2021-02-10T11:41:36.557 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:36.539 7fd9feb43700 1 -- 172.21.2.102:0/1324278331 shutdown_connections 2021-02-10T11:41:36.557 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:36.539 7fd9feb43700 1 --2- 172.21.2.102:0/1324278331 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd9f80d1c90 0x7fd9f80cef90 unknown :-1 s=CLOSED pgs=3436 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:36.557 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:36.539 7fd9feb43700 1 -- 172.21.2.102:0/1324278331 >> 172.21.2.102:0/1324278331 conn(0x7fd9f80fdee0 msgr2=0x7fd9f80ce450 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:36.557 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:36.539 7fd9feb43700 1 -- 172.21.2.102:0/1324278331 shutdown_connections 2021-02-10T11:41:36.557 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:36.539 7fd9feb43700 1 -- 172.21.2.102:0/1324278331 wait complete. 2021-02-10T11:41:36.557 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:36.554 7fd9feb43700 1 Processor -- start 2021-02-10T11:41:36.557 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:36.554 7fd9feb43700 1 -- start start 2021-02-10T11:41:36.558 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:36.555 7fd9feb43700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd9f80bcc60 0x7fd9f80bc470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:36.559 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:36.555 7fd9feb43700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd9f80be4d0 0x7fd9f80be8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:36.559 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:36.555 7fd9feb43700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd9f80beef0 0x7fd9f810e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:36.559 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:36.555 7fd9feb43700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9f810e050 con 0x7fd9f80be4d0 2021-02-10T11:41:36.559 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:36.555 7fd9feb43700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9f80bf2f0 con 0x7fd9f80bcc60 2021-02-10T11:41:36.559 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:36.555 7fd9feb43700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9f80bf4d0 con 0x7fd9f80beef0 2021-02-10T11:41:36.559 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:36.555 7fd9f7fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd9f80be4d0 0x7fd9f80be8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:36.559 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:36.555 7fd9f7fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd9f80be4d0 0x7fd9f80be8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:41620/0 (socket says 172.21.2.102:41620) 2021-02-10T11:41:36.559 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:36.555 7fd9f7fff700 1 -- 172.21.2.102:0/366592564 learned_addr learned my addr 172.21.2.102:0/366592564 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:41:36.559 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:36.555 7fd9f7fff700 1 -- 172.21.2.102:0/366592564 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd9f80beef0 msgr2=0x7fd9f810e5d0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:41:36.559 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:36.555 7fd9f7fff700 1 --2- 172.21.2.102:0/366592564 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd9f80beef0 0x7fd9f810e5d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:36.559 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:36.555 7fd9f7fff700 1 -- 172.21.2.102:0/366592564 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd9f80bcc60 msgr2=0x7fd9f80bc470 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:41:36.560 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:36.555 7fd9f7fff700 1 --2- 172.21.2.102:0/366592564 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd9f80bcc60 0x7fd9f80bc470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:36.560 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:36.555 7fd9f7fff700 1 -- 172.21.2.102:0/366592564 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9f810ebf0 con 0x7fd9f80be4d0 2021-02-10T11:41:36.560 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:36.555 7fd9f7fff700 1 --2- 172.21.2.102:0/366592564 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd9f80be4d0 0x7fd9f80be8d0 secure :-1 s=READY pgs=3437 cs=0 l=1 rev1=1 rx=0x7fd9e800b530 tx=0x7fd9e800b8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:36.560 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:36.555 7fd9f57fa700 1 -- 172.21.2.102:0/366592564 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd9e8009440 con 0x7fd9f80be4d0 2021-02-10T11:41:36.560 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:36.555 7fd9f57fa700 1 -- 172.21.2.102:0/366592564 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9e8006030 con 0x7fd9f80be4d0 2021-02-10T11:41:36.560 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:36.555 7fd9f57fa700 1 -- 172.21.2.102:0/366592564 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd9e80052e0 con 0x7fd9f80be4d0 2021-02-10T11:41:36.560 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:36.555 7fd9feb43700 1 -- 172.21.2.102:0/366592564 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9f810ebf0 con 0x7fd9f80be4d0 2021-02-10T11:41:36.560 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:36.556 7fd9feb43700 1 -- 172.21.2.102:0/366592564 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9f810f180 con 0x7fd9f80be4d0 2021-02-10T11:41:36.560 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:36.556 7fd9f57fa700 1 -- 172.21.2.102:0/366592564 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd9e8009440 con 0x7fd9f80be4d0 2021-02-10T11:41:36.561 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:36.556 7fd9f57fa700 1 --2- 172.21.2.102:0/366592564 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd9dc0186c0 0x7fd9dc01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:36.561 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:36.556 7fd9f57fa700 1 -- 172.21.2.102:0/366592564 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fd9e8032440 con 0x7fd9f80be4d0 2021-02-10T11:41:36.561 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:36.556 7fd9fc95f700 1 --2- 172.21.2.102:0/366592564 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd9dc0186c0 0x7fd9dc01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:36.561 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:36.557 7fd9fc95f700 1 --2- 172.21.2.102:0/366592564 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd9dc0186c0 0x7fd9dc01abc0 crc :-1 s=READY pgs=1495 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:36.589 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:36.588 7fd9feb43700 1 -- 172.21.2.102:0/366592564 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9f80e6140 con 0x7fd9f80be4d0 2021-02-10T11:41:36.591 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:36.590 7fd9f57fa700 1 -- 172.21.2.102:0/366592564 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd9e8025ff0 con 0x7fd9f80be4d0 2021-02-10T11:41:36.719 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:36.717 7fd9feb43700 1 -- 172.21.2.102:0/366592564 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "subvolume_0000000000115792", "target": ["mgr", ""], "snap_name": "snapshot_0000000000245284", "target_sub_name": "clone_0000000000944983"}) v1 -- 0x7fd9f80bd060 con 0x7fd9dc0186c0 2021-02-10T11:41:36.764 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:36.763 7fd9f57fa700 1 -- 172.21.2.102:0/366592564 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fd9e0000c60 con 0x7fd9dc0186c0 2021-02-10T11:41:36.784 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:36.782 7fd9feb43700 1 -- 172.21.2.102:0/366592564 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd9dc0186c0 msgr2=0x7fd9dc01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:36.784 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:36.782 7fd9feb43700 1 --2- 172.21.2.102:0/366592564 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd9dc0186c0 0x7fd9dc01abc0 crc :-1 s=READY pgs=1495 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:36.784 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:36.782 7fd9feb43700 1 -- 172.21.2.102:0/366592564 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd9f80be4d0 msgr2=0x7fd9f80be8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:36.784 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:36.782 7fd9feb43700 1 --2- 172.21.2.102:0/366592564 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd9f80be4d0 0x7fd9f80be8d0 secure :-1 s=READY pgs=3437 cs=0 l=1 rev1=1 rx=0x7fd9e800b530 tx=0x7fd9e800b8d0).stop 2021-02-10T11:41:36.784 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:36.783 7fd9feb43700 1 -- 172.21.2.102:0/366592564 shutdown_connections 2021-02-10T11:41:36.784 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:36.783 7fd9feb43700 1 --2- 172.21.2.102:0/366592564 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fd9f80beef0 0x7fd9f810e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:36.785 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:36.783 7fd9feb43700 1 --2- 172.21.2.102:0/366592564 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fd9f80be4d0 0x7fd9f80be8d0 unknown :-1 s=CLOSED pgs=3437 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:36.785 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:36.783 7fd9feb43700 1 --2- 172.21.2.102:0/366592564 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fd9dc0186c0 0x7fd9dc01abc0 unknown :-1 s=CLOSED pgs=1495 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:36.785 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:36.783 7fd9feb43700 1 --2- 172.21.2.102:0/366592564 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fd9f80bcc60 0x7fd9f80bc470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:36.785 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:36.783 7fd9feb43700 1 -- 172.21.2.102:0/366592564 >> 172.21.2.102:0/366592564 conn(0x7fd9f809f1e0 msgr2=0x7fd9f8104880 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:36.785 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:36.783 7fd9feb43700 1 -- 172.21.2.102:0/366592564 shutdown_connections 2021-02-10T11:41:36.785 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:36.783 7fd9feb43700 1 -- 172.21.2.102:0/366592564 wait complete. 2021-02-10T11:41:36.793 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs clone_0000000000944983 2021-02-10T11:41:37.230 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:37.212 7fc3a5a7b700 1 Processor -- start 2021-02-10T11:41:37.230 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:37.212 7fc3a5a7b700 1 -- start start 2021-02-10T11:41:37.231 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:37.212 7fc3a5a7b700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fc3a00d1c60 0x7fc3a00cef50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:37.231 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:37.212 7fc3a5a7b700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3a00d0160 con 0x7fc3a00cf570 2021-02-10T11:41:37.231 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:37.212 7fc3a5a7b700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3a00fe1b0 con 0x7fc3a00d1860 2021-02-10T11:41:37.231 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:37.212 7fc3a5a7b700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3a00d2fd0 con 0x7fc3a00d1c60 2021-02-10T11:41:37.231 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:37.212 7fc39ffff700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fc3a00cf570 0x7fc3a00cf970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:33864/0 (socket says 172.21.2.102:33864) 2021-02-10T11:41:37.231 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:37.212 7fc39ffff700 1 -- 172.21.2.102:0/372019050 learned_addr learned my addr 172.21.2.102:0/372019050 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:41:37.231 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:37.212 7fc396ffd700 1 --2- 172.21.2.102:0/372019050 >> v2:172.21.2.113:3300/0 conn(0x7fc3a00d1c60 0x7fc3a00cef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:37.231 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:37.213 7fc39effd700 1 -- 172.21.2.102:0/372019050 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2634393813 0 0) 0x7fc3980009b0 con 0x7fc3a00cf570 2021-02-10T11:41:37.231 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:37.213 7fc39effd700 1 -- 172.21.2.102:0/372019050 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc38c002fe0 con 0x7fc3a00cf570 2021-02-10T11:41:37.231 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:37.213 7fc396ffd700 1 -- 172.21.2.102:0/372019050 >> v1:172.21.2.113:6789/0 conn(0x7fc3a00d1860 legacy=0x7fc3a01b04b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:37.231 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:37.213 7fc396ffd700 1 -- 172.21.2.102:0/372019050 >> v1:172.21.2.102:6789/0 conn(0x7fc3a00cf570 legacy=0x7fc3a00cf970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:37.231 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:37.213 7fc396ffd700 1 -- 172.21.2.102:0/372019050 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3a00d3350 con 0x7fc3a00d1c60 2021-02-10T11:41:37.232 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:37.213 7fc396ffd700 1 --2- 172.21.2.102:0/372019050 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc3a00d1c60 0x7fc3a00cef50 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 rx=0x7fc388006650 tx=0x7fc388009a00).ready entity=mon.1 client_cookie=ddf92ca72be41319 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:37.232 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:37.213 7fc39effd700 1 -- 172.21.2.102:0/372019050 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc388009df0 con 0x7fc3a00d1c60 2021-02-10T11:41:37.232 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:37.213 7fc39effd700 1 -- 172.21.2.102:0/372019050 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3880009b0 con 0x7fc3a00d1c60 2021-02-10T11:41:37.232 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:37.213 7fc39effd700 1 -- 172.21.2.102:0/372019050 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc388009df0 con 0x7fc3a00d1c60 2021-02-10T11:41:37.232 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:37.213 7fc3a5a7b700 1 -- 172.21.2.102:0/372019050 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc3a00d1c60 msgr2=0x7fc3a00cef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:37.232 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:37.213 7fc3a5a7b700 1 --2- 172.21.2.102:0/372019050 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc3a00d1c60 0x7fc3a00cef50 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 rx=0x7fc388006650 tx=0x7fc388009a00).stop 2021-02-10T11:41:37.232 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:37.214 7fc3a5a7b700 1 -- 172.21.2.102:0/372019050 shutdown_connections 2021-02-10T11:41:37.232 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:37.214 7fc3a5a7b700 1 --2- 172.21.2.102:0/372019050 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc3a00d1c60 0x7fc3a00cef50 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:37.232 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:37.214 7fc3a5a7b700 1 -- 172.21.2.102:0/372019050 >> 172.21.2.102:0/372019050 conn(0x7fc3a00ce410 msgr2=0x7fc3a00d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:37.232 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:37.214 7fc3a5a7b700 1 -- 172.21.2.102:0/372019050 shutdown_connections 2021-02-10T11:41:37.232 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:37.214 7fc3a5a7b700 1 -- 172.21.2.102:0/372019050 wait complete. 2021-02-10T11:41:37.232 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:37.229 7fc3a5a7b700 1 Processor -- start 2021-02-10T11:41:37.232 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:37.229 7fc3a5a7b700 1 -- start start 2021-02-10T11:41:37.234 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:37.230 7fc3a5a7b700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc3a01a05e0 0x7fc3a018b970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:37.235 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:37.230 7fc3a5a7b700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc3a00b8780 0x7fc3a00bfa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:37.235 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:37.230 7fc3a5a7b700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc3a00c0030 0x7fc3a0189e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:37.235 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:37.230 7fc3a5a7b700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3a00b82a0 con 0x7fc3a01a05e0 2021-02-10T11:41:37.235 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:37.230 7fc3a5a7b700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3a00c0430 con 0x7fc3a00b8780 2021-02-10T11:41:37.235 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:37.230 7fc3a5a7b700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3a00c0650 con 0x7fc3a00c0030 2021-02-10T11:41:37.235 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:37.230 7fc39f7fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc3a01a05e0 0x7fc3a018b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:37.235 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:37.230 7fc39f7fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc3a01a05e0 0x7fc3a018b970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:41642/0 (socket says 172.21.2.102:41642) 2021-02-10T11:41:37.235 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:37.230 7fc39f7fe700 1 -- 172.21.2.102:0/3493375071 learned_addr learned my addr 172.21.2.102:0/3493375071 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:41:37.235 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:37.230 7fc396ffd700 1 --2- 172.21.2.102:0/3493375071 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc3a00b8780 0x7fc3a00bfa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:37.235 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:37.230 7fc39ffff700 1 --2- 172.21.2.102:0/3493375071 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc3a00c0030 0x7fc3a0189e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:37.235 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:37.230 7fc39f7fe700 1 -- 172.21.2.102:0/3493375071 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc3a00c0030 msgr2=0x7fc3a0189e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:37.235 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:37.230 7fc39f7fe700 1 --2- 172.21.2.102:0/3493375071 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc3a00c0030 0x7fc3a0189e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:37.235 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:37.230 7fc39f7fe700 1 -- 172.21.2.102:0/3493375071 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc3a00b8780 msgr2=0x7fc3a00bfa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:37.236 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:37.230 7fc39f7fe700 1 --2- 172.21.2.102:0/3493375071 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc3a00b8780 0x7fc3a00bfa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:37.236 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:37.230 7fc39f7fe700 1 -- 172.21.2.102:0/3493375071 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3a018a490 con 0x7fc3a01a05e0 2021-02-10T11:41:37.236 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:37.230 7fc396ffd700 1 --2- 172.21.2.102:0/3493375071 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc3a00b8780 0x7fc3a00bfa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:41:37.236 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:37.230 7fc39f7fe700 1 --2- 172.21.2.102:0/3493375071 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc3a01a05e0 0x7fc3a018b970 secure :-1 s=READY pgs=3439 cs=0 l=1 rev1=1 rx=0x7fc390009940 tx=0x7fc39000bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:37.236 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:37.230 7fc39cff9700 1 -- 172.21.2.102:0/3493375071 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc390000ce0 con 0x7fc3a01a05e0 2021-02-10T11:41:37.236 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:37.230 7fc39cff9700 1 -- 172.21.2.102:0/3493375071 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc39000c2f0 con 0x7fc3a01a05e0 2021-02-10T11:41:37.236 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:37.230 7fc3a5a7b700 1 -- 172.21.2.102:0/3493375071 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3a018a490 con 0x7fc3a01a05e0 2021-02-10T11:41:37.236 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:37.230 7fc39cff9700 1 -- 172.21.2.102:0/3493375071 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc39000c750 con 0x7fc3a01a05e0 2021-02-10T11:41:37.236 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:37.230 7fc3a5a7b700 1 -- 172.21.2.102:0/3493375071 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3a018a490 con 0x7fc3a01a05e0 2021-02-10T11:41:37.236 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:37.231 7fc39cff9700 1 -- 172.21.2.102:0/3493375071 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fc390001230 con 0x7fc3a01a05e0 2021-02-10T11:41:37.236 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:37.231 7fc39cff9700 1 --2- 172.21.2.102:0/3493375071 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc384018730 0x7fc38401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:37.236 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:37.231 7fc39cff9700 1 -- 172.21.2.102:0/3493375071 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fc390032b60 con 0x7fc3a01a05e0 2021-02-10T11:41:37.236 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:37.231 7fc396ffd700 1 --2- 172.21.2.102:0/3493375071 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc384018730 0x7fc38401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:37.237 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:37.231 7fc396ffd700 1 --2- 172.21.2.102:0/3493375071 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc384018730 0x7fc38401ac10 crc :-1 s=READY pgs=1496 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:37.262 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:37.261 7fc3a5a7b700 1 -- 172.21.2.102:0/3493375071 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3a0082a20 con 0x7fc3a01a05e0 2021-02-10T11:41:37.265 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:37.264 7fc39cff9700 1 -- 172.21.2.102:0/3493375071 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fc390000fe0 con 0x7fc3a01a05e0 2021-02-10T11:41:37.400 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:37.399 7fc3a5a7b700 1 -- 172.21.2.102:0/3493375071 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "clone_0000000000944983", "target": ["mgr", ""]}) v1 -- 0x7fc3a0082a20 con 0x7fc384018730 2021-02-10T11:41:37.402 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:37.401 7fc39cff9700 1 -- 172.21.2.102:0/3493375071 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+78 (crc 0 0 0) 0x7fc3880092f0 con 0x7fc384018730 2021-02-10T11:41:37.403 INFO:teuthology.orchestra.run.gibba002.stdout:/volumes/_nogroup/clone_0000000000944983/57f96ddd-bd6a-41f5-aaa3-7de4a6702f2d 2021-02-10T11:41:37.405 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:37.403 7fc3a5a7b700 1 -- 172.21.2.102:0/3493375071 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc384018730 msgr2=0x7fc38401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:37.405 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:37.403 7fc3a5a7b700 1 --2- 172.21.2.102:0/3493375071 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc384018730 0x7fc38401ac10 crc :-1 s=READY pgs=1496 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:37.405 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:37.403 7fc3a5a7b700 1 -- 172.21.2.102:0/3493375071 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc3a01a05e0 msgr2=0x7fc3a018b970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:37.405 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:37.403 7fc3a5a7b700 1 --2- 172.21.2.102:0/3493375071 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc3a01a05e0 0x7fc3a018b970 secure :-1 s=READY pgs=3439 cs=0 l=1 rev1=1 rx=0x7fc390009940 tx=0x7fc39000bcf0).stop 2021-02-10T11:41:37.405 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:37.403 7fc3a5a7b700 1 -- 172.21.2.102:0/3493375071 shutdown_connections 2021-02-10T11:41:37.405 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:37.403 7fc3a5a7b700 1 --2- 172.21.2.102:0/3493375071 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fc3a00c0030 0x7fc3a0189e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:37.405 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:37.403 7fc3a5a7b700 1 --2- 172.21.2.102:0/3493375071 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fc384018730 0x7fc38401ac10 unknown :-1 s=CLOSED pgs=1496 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:37.405 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:37.403 7fc3a5a7b700 1 --2- 172.21.2.102:0/3493375071 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fc3a00b8780 0x7fc3a00bfa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:37.406 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:37.403 7fc3a5a7b700 1 --2- 172.21.2.102:0/3493375071 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fc3a01a05e0 0x7fc3a018b970 unknown :-1 s=CLOSED pgs=3439 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:37.406 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:37.403 7fc3a5a7b700 1 -- 172.21.2.102:0/3493375071 >> 172.21.2.102:0/3493375071 conn(0x7fc3a00a6a80 msgr2=0x7fc3a00ae110 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:37.406 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:37.404 7fc3a5a7b700 1 -- 172.21.2.102:0/3493375071 shutdown_connections 2021-02-10T11:41:37.406 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:37.404 7fc3a5a7b700 1 -- 172.21.2.102:0/3493375071 wait complete. 2021-02-10T11:41:37.413 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2021-02-10T11:41:37.413 DEBUG:teuthology.orchestra.run.gibba002:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:41:37.760 DEBUG:teuthology.orchestra.run.gibba013:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:41:38.106 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2021-02-10T11:41:38.106 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:41:38.107 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:41:38.107 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:41:38.443 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:41:38.443 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:41:38.443 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@gibba002.front.sepia.ceph.com... 2021-02-10T11:41:38.443 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-02-10T11:41:38.443 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:41:38.805 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba002.stderr:ceph-fuse[221170]: fuse finished with error 0 and tester_r 0 2021-02-10T11:41:38.806 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:41:38.806 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:41:38.806 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:41:39.142 INFO:teuthology.orchestra.run.gibba002.stdout:ext2/ext3 2021-02-10T11:41:39.142 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2021-02-10T11:41:39.143 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:41:39.143 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:41:39.479 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:41:39.480 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:41:39.480 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:41:39.818 INFO:teuthology.orchestra.run.gibba002.stdout:fuseblk 2021-02-10T11:41:39.818 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2021-02-10T11:41:39.819 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@gibba002.front.sepia.ceph.com... 2021-02-10T11:41:39.819 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-02-10T11:41:39.819 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:41:40.175 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba002.stderr:ceph-fuse[221810]: fuse finished with error 0 and tester_r 0 2021-02-10T11:41:40.176 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:41:40.177 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:41:40.177 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:41:40.512 INFO:teuthology.orchestra.run.gibba002.stdout:ext2/ext3 2021-02-10T11:41:40.513 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2021-02-10T11:41:40.513 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:41:40.513 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:41:40.848 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:41:40.849 DEBUG:teuthology.orchestra.run.gibba013:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:41:40.849 DEBUG:teuthology.orchestra.run.gibba013:> ' -- /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:41:41.183 INFO:teuthology.orchestra.run.gibba013.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.2’: No such file or directory 2021-02-10T11:41:41.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:41:41.183 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2021-02-10T11:41:41.184 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:41:41.184 DEBUG:teuthology.orchestra.run.gibba013:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:41:41.519 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:41:41.519 DEBUG:teuthology.orchestra.run.gibba013:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:41:41.519 DEBUG:teuthology.orchestra.run.gibba013:> ' -- /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:41:41.852 INFO:teuthology.orchestra.run.gibba013.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.3’: No such file or directory 2021-02-10T11:41:41.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:41:41.853 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.3 2021-02-10T11:41:41.853 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:41:41.853 DEBUG:teuthology.orchestra.run.gibba013:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:41:42.188 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config rm mon mon_allow_pool_delete 2021-02-10T11:41:42.621 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:42.602 7fa24232f700 1 Processor -- start 2021-02-10T11:41:42.621 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:42.603 7fa24232f700 1 -- start start 2021-02-10T11:41:42.621 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:42.603 7fa24232f700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fa23c0d1870 0x7fa23c1b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:42.621 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:42.603 7fa24232f700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa23c0f0650 0x7fa23c0cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:42.621 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:42.603 7fa24232f700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa23c0f0b10 con 0x7fa23c1b0a10 2021-02-10T11:41:42.622 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:42.603 7fa24232f700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa23c0e5ed0 con 0x7fa23c0d1870 2021-02-10T11:41:42.622 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:42.603 7fa24232f700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa23c0d1c70 con 0x7fa23c0f0650 2021-02-10T11:41:42.622 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:42.603 7fa23bfff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fa23c0d1870 0x7fa23c1b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:42.622 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:42.603 7fa23bfff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fa23c0d1870 0x7fa23c1b03f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:41650/0 (socket says 172.21.2.102:41650) 2021-02-10T11:41:42.622 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:42.603 7fa23bfff700 1 -- 172.21.2.102:0/3722701239 learned_addr learned my addr 172.21.2.102:0/3722701239 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:41:42.622 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:42.603 7fa23b7fe700 1 --1- >> v1:172.21.2.113:6789/0 conn(0x7fa23c1b0a10 0x7fa23c0cf330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.113:6789/0 says I am v1:172.21.2.102:41198/0 (socket says 172.21.2.102:41198) 2021-02-10T11:41:42.622 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:42.603 7fa24094c700 1 --2- 172.21.2.102:0/3722701239 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa23c0f0650 0x7fa23c0cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:42.622 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:42.604 7fa23bfff700 1 -- 172.21.2.102:0/3722701239 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa23c0f0650 msgr2=0x7fa23c0cfb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:41:42.622 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:42.604 7fa23bfff700 1 --2- 172.21.2.102:0/3722701239 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa23c0f0650 0x7fa23c0cfb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:42.622 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:42.604 7fa23bfff700 1 -- 172.21.2.102:0/3722701239 >> v1:172.21.2.113:6789/0 conn(0x7fa23c1b0a10 legacy=0x7fa23c0cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:42.622 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:42.604 7fa23affd700 1 -- 172.21.2.102:0/3722701239 <== mon.1 v1:172.21.2.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 209813366 0 0) 0x7fa224000c60 con 0x7fa23c1b0a10 2021-02-10T11:41:42.623 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:42.604 7fa23bfff700 1 -- 172.21.2.102:0/3722701239 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa23c0d1e90 con 0x7fa23c0d1870 2021-02-10T11:41:42.623 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:42.604 7fa24094c700 1 --2- 172.21.2.102:0/3722701239 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa23c0f0650 0x7fa23c0cfb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:41:42.623 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:42.604 7fa23bfff700 1 --2- 172.21.2.102:0/3722701239 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa23c0d1870 0x7fa23c1b03f0 secure :-1 s=READY pgs=3440 cs=0 l=1 rev1=1 rx=0x7fa22c006a80 tx=0x7fa22c009f00).ready entity=mon.0 client_cookie=a43915fc2ec3047c server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:42.623 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:42.604 7fa23affd700 1 -- 172.21.2.102:0/3722701239 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa22c000d30 con 0x7fa23c0d1870 2021-02-10T11:41:42.623 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:42.604 7fa23affd700 1 -- 172.21.2.102:0/3722701239 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa22c00b5c0 con 0x7fa23c0d1870 2021-02-10T11:41:42.623 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:42.604 7fa23affd700 1 -- 172.21.2.102:0/3722701239 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa22c00ba10 con 0x7fa23c0d1870 2021-02-10T11:41:42.623 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:42.604 7fa24232f700 1 -- 172.21.2.102:0/3722701239 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa23c0d1870 msgr2=0x7fa23c1b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:42.623 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:42.604 7fa24232f700 1 --2- 172.21.2.102:0/3722701239 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa23c0d1870 0x7fa23c1b03f0 secure :-1 s=READY pgs=3440 cs=0 l=1 rev1=1 rx=0x7fa22c006a80 tx=0x7fa22c009f00).stop 2021-02-10T11:41:42.623 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:42.605 7fa24232f700 1 -- 172.21.2.102:0/3722701239 shutdown_connections 2021-02-10T11:41:42.623 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:42.605 7fa24232f700 1 --2- 172.21.2.102:0/3722701239 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa23c0f0650 0x7fa23c0cfb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:42.623 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:42.605 7fa24232f700 1 --2- 172.21.2.102:0/3722701239 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa23c0d1870 0x7fa23c1b03f0 unknown :-1 s=CLOSED pgs=3440 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:42.623 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:42.605 7fa24232f700 1 -- 172.21.2.102:0/3722701239 >> 172.21.2.102:0/3722701239 conn(0x7fa23c0ce450 msgr2=0x7fa23c0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:42.623 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:42.605 7fa24232f700 1 -- 172.21.2.102:0/3722701239 shutdown_connections 2021-02-10T11:41:42.623 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:42.605 7fa24232f700 1 -- 172.21.2.102:0/3722701239 wait complete. 2021-02-10T11:41:42.624 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:42.620 7fa24232f700 1 Processor -- start 2021-02-10T11:41:42.624 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:42.620 7fa24232f700 1 -- start start 2021-02-10T11:41:42.625 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:42.620 7fa24232f700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa23c094310 0x7fa23c0ebe70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:42.625 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:42.620 7fa24232f700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa23c094710 0x7fa23c0bca30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:42.625 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:42.620 7fa24232f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa23c0ec490 0x7fa23c0bd050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:42.625 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:42.620 7fa24232f700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa23c093d90 con 0x7fa23c0ec490 2021-02-10T11:41:42.625 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:42.620 7fa24232f700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa23c0bd670 con 0x7fa23c094310 2021-02-10T11:41:42.625 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:42.620 7fa24232f700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa23c0939f0 con 0x7fa23c094710 2021-02-10T11:41:42.625 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:42.621 7fa24094c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa23c0ec490 0x7fa23c0bd050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:42.626 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:42.621 7fa24094c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa23c0ec490 0x7fa23c0bd050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:41656/0 (socket says 172.21.2.102:41656) 2021-02-10T11:41:42.626 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:42.621 7fa24094c700 1 -- 172.21.2.102:0/3905812887 learned_addr learned my addr 172.21.2.102:0/3905812887 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:41:42.627 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:42.621 7fa24094c700 1 -- 172.21.2.102:0/3905812887 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa23c094710 msgr2=0x7fa23c0bca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:42.627 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:42.621 7fa24094c700 1 --2- 172.21.2.102:0/3905812887 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa23c094710 0x7fa23c0bca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:42.627 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:42.621 7fa24094c700 1 -- 172.21.2.102:0/3905812887 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa23c094310 msgr2=0x7fa23c0ebe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:42.627 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:42.621 7fa24094c700 1 --2- 172.21.2.102:0/3905812887 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa23c094310 0x7fa23c0ebe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:42.627 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:42.621 7fa24094c700 1 -- 172.21.2.102:0/3905812887 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa23c0bda80 con 0x7fa23c0ec490 2021-02-10T11:41:42.627 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:42.621 7fa24094c700 1 --2- 172.21.2.102:0/3905812887 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa23c0ec490 0x7fa23c0bd050 secure :-1 s=READY pgs=3441 cs=0 l=1 rev1=1 rx=0x7fa23400b5e0 tx=0x7fa23400b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:42.627 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:42.621 7fa238ff9700 1 -- 172.21.2.102:0/3905812887 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa23400bd70 con 0x7fa23c0ec490 2021-02-10T11:41:42.627 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:42.621 7fa238ff9700 1 -- 172.21.2.102:0/3905812887 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2340022e0 con 0x7fa23c0ec490 2021-02-10T11:41:42.627 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:42.621 7fa24232f700 1 -- 172.21.2.102:0/3905812887 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa23c0bda80 con 0x7fa23c0ec490 2021-02-10T11:41:42.627 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:42.621 7fa24232f700 1 -- 172.21.2.102:0/3905812887 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa23c1a76d0 con 0x7fa23c0ec490 2021-02-10T11:41:42.627 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:42.622 7fa238ff9700 1 -- 172.21.2.102:0/3905812887 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa234002800 con 0x7fa23c0ec490 2021-02-10T11:41:42.628 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:42.622 7fa238ff9700 1 -- 172.21.2.102:0/3905812887 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa2340024c0 con 0x7fa23c0ec490 2021-02-10T11:41:42.628 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:42.622 7fa238ff9700 1 --2- 172.21.2.102:0/3905812887 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa2200186c0 0x7fa22001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:42.628 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:42.622 7fa238ff9700 1 -- 172.21.2.102:0/3905812887 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fa2340325b0 con 0x7fa23c0ec490 2021-02-10T11:41:42.628 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:42.622 7fa23bfff700 1 --2- 172.21.2.102:0/3905812887 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa2200186c0 0x7fa22001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:42.628 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:42.623 7fa23bfff700 1 --2- 172.21.2.102:0/3905812887 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa2200186c0 0x7fa22001abc0 crc :-1 s=READY pgs=1497 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:42.654 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:42.653 7fa24232f700 1 -- 172.21.2.102:0/3905812887 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa23c0ebc40 con 0x7fa23c0ec490 2021-02-10T11:41:42.657 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:42.656 7fa238ff9700 1 -- 172.21.2.102:0/3905812887 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fa234000bc0 con 0x7fa23c0ec490 2021-02-10T11:41:42.806 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:42.805 7fa24232f700 1 -- 172.21.2.102:0/3905812887 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fa23c0f0b10 con 0x7fa23c0ec490 2021-02-10T11:41:42.810 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:42.808 7fa238ff9700 1 -- 172.21.2.102:0/3905812887 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v49) v1 ==== 104+0+0 (secure 0 0 0) 0x7fa2340023b0 con 0x7fa23c0ec490 2021-02-10T11:41:42.815 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:42.814 7fa24232f700 1 -- 172.21.2.102:0/3905812887 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa2200186c0 msgr2=0x7fa22001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:42.815 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:42.814 7fa24232f700 1 --2- 172.21.2.102:0/3905812887 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa2200186c0 0x7fa22001abc0 crc :-1 s=READY pgs=1497 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:42.815 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:42.814 7fa24232f700 1 -- 172.21.2.102:0/3905812887 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa23c0ec490 msgr2=0x7fa23c0bd050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:42.815 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:42.814 7fa24232f700 1 --2- 172.21.2.102:0/3905812887 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa23c0ec490 0x7fa23c0bd050 secure :-1 s=READY pgs=3441 cs=0 l=1 rev1=1 rx=0x7fa23400b5e0 tx=0x7fa23400b980).stop 2021-02-10T11:41:42.816 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:42.814 7fa24232f700 1 -- 172.21.2.102:0/3905812887 shutdown_connections 2021-02-10T11:41:42.816 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:42.814 7fa24232f700 1 --2- 172.21.2.102:0/3905812887 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fa23c0ec490 0x7fa23c0bd050 unknown :-1 s=CLOSED pgs=3441 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:42.816 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:42.814 7fa24232f700 1 --2- 172.21.2.102:0/3905812887 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fa23c094710 0x7fa23c0bca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:42.816 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:42.814 7fa24232f700 1 --2- 172.21.2.102:0/3905812887 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fa2200186c0 0x7fa22001abc0 unknown :-1 s=CLOSED pgs=1497 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:42.816 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:42.814 7fa24232f700 1 --2- 172.21.2.102:0/3905812887 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fa23c094310 0x7fa23c0ebe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:42.816 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:42.814 7fa24232f700 1 -- 172.21.2.102:0/3905812887 >> 172.21.2.102:0/3905812887 conn(0x7fa23c08fa10 msgr2=0x7fa23c0bfa60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:42.816 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:42.814 7fa24232f700 1 -- 172.21.2.102:0/3905812887 shutdown_connections 2021-02-10T11:41:42.816 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:42.814 7fa24232f700 1 -- 172.21.2.102:0/3905812887 wait complete. 2021-02-10T11:41:42.824 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestVolumes.test_subvolume_clone_in_progress_getpath' 2021-02-10T11:41:43.253 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:43.235 7faee5f7b700 1 Processor -- start 2021-02-10T11:41:43.253 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:43.235 7faee5f7b700 1 -- start start 2021-02-10T11:41:43.254 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:43.235 7faee5f7b700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7faee01b0470 0x7faee01b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:43.254 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:43.235 7faee5f7b700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7faee00ef610 0x7faee00efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:43.254 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:43.235 7faee5f7b700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7faee00f0030 0x7faee00f0430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:43.254 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:43.235 7faee5f7b700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faee00e5ea0 con 0x7faee00f0030 2021-02-10T11:41:43.254 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:43.235 7faee5f7b700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faee00f0b50 con 0x7faee01b0470 2021-02-10T11:41:43.254 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:43.235 7faee5f7b700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faee00e52e0 con 0x7faee00ef610 2021-02-10T11:41:43.254 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:43.235 7faedffff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7faee00f0030 0x7faee00f0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:43.254 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:43.236 7faedffff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7faee00f0030 0x7faee00f0430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:41664/0 (socket says 172.21.2.102:41664) 2021-02-10T11:41:43.254 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:43.236 7faedffff700 1 -- 172.21.2.102:0/399989823 learned_addr learned my addr 172.21.2.102:0/399989823 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:41:43.254 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:43.236 7faedffff700 1 -- 172.21.2.102:0/399989823 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7faee00ef610 msgr2=0x7faee00efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:41:43.254 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:43.236 7faedffff700 1 --2- 172.21.2.102:0/399989823 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7faee00ef610 0x7faee00efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:43.255 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:43.236 7faedffff700 1 -- 172.21.2.102:0/399989823 >> v2:172.21.2.113:3300/0 conn(0x7faee01b0470 msgr2=0x7faee01b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:41:43.255 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:43.236 7faedffff700 1 --2- 172.21.2.102:0/399989823 >> v2:172.21.2.113:3300/0 conn(0x7faee01b0470 0x7faee01b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:43.255 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:43.236 7faedffff700 1 -- 172.21.2.102:0/399989823 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faee00f10d0 con 0x7faee00f0030 2021-02-10T11:41:43.255 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:43.236 7faedffff700 1 --2- 172.21.2.102:0/399989823 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7faee00f0030 0x7faee00f0430 secure :-1 s=READY pgs=3442 cs=0 l=1 rev1=1 rx=0x7faed0006b80 tx=0x7faed0009b90).ready entity=mon.0 client_cookie=e1e545e19f0cd4e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:43.255 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:43.236 7faede7fc700 1 -- 172.21.2.102:0/399989823 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faed00022e0 con 0x7faee00f0030 2021-02-10T11:41:43.255 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:43.236 7faede7fc700 1 -- 172.21.2.102:0/399989823 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faed0002560 con 0x7faee00f0030 2021-02-10T11:41:43.255 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:43.236 7faede7fc700 1 -- 172.21.2.102:0/399989823 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faed0002a80 con 0x7faee00f0030 2021-02-10T11:41:43.255 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:43.236 7faee5f7b700 1 -- 172.21.2.102:0/399989823 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7faee00f0030 msgr2=0x7faee00f0430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:43.255 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:43.236 7faee5f7b700 1 --2- 172.21.2.102:0/399989823 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7faee00f0030 0x7faee00f0430 secure :-1 s=READY pgs=3442 cs=0 l=1 rev1=1 rx=0x7faed0006b80 tx=0x7faed0009b90).stop 2021-02-10T11:41:43.255 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:43.237 7faee5f7b700 1 -- 172.21.2.102:0/399989823 shutdown_connections 2021-02-10T11:41:43.255 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:43.237 7faee5f7b700 1 --2- 172.21.2.102:0/399989823 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7faee00ef610 0x7faee00efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:43.255 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:43.237 7faee5f7b700 1 --2- 172.21.2.102:0/399989823 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7faee00f0030 0x7faee00f0430 unknown :-1 s=CLOSED pgs=3442 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:43.255 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:43.237 7faee5f7b700 1 --2- 172.21.2.102:0/399989823 >> v2:172.21.2.113:3300/0 conn(0x7faee01b0470 0x7faee01b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:43.256 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:43.237 7faee5f7b700 1 -- 172.21.2.102:0/399989823 >> 172.21.2.102:0/399989823 conn(0x7faee00ce3e0 msgr2=0x7faee00e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:43.256 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:43.237 7faee5f7b700 1 -- 172.21.2.102:0/399989823 shutdown_connections 2021-02-10T11:41:43.256 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:43.237 7faee5f7b700 1 -- 172.21.2.102:0/399989823 wait complete. 2021-02-10T11:41:43.257 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:43.251 7faee5f7b700 1 Processor -- start 2021-02-10T11:41:43.257 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:43.252 7faee5f7b700 1 -- start start 2021-02-10T11:41:43.257 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:43.252 7faee5f7b700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7faee0188110 0x7faee0188510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:43.257 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:43.252 7faee5f7b700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7faee0188b30 0x7faee018ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:43.257 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:43.252 7faee5f7b700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7faee018f200 0x7faee018f600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:43.257 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:43.252 7faee5f7b700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faee00aba90 con 0x7faee018f200 2021-02-10T11:41:43.257 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:43.252 7faee5f7b700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faee0189190 con 0x7faee0188b30 2021-02-10T11:41:43.257 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:43.252 7faee5f7b700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faee0191620 con 0x7faee0188110 2021-02-10T11:41:43.257 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:43.252 7faedffff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7faee018f200 0x7faee018f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:43.258 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:43.252 7faedffff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7faee018f200 0x7faee018f600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:41670/0 (socket says 172.21.2.102:41670) 2021-02-10T11:41:43.258 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:43.252 7faedffff700 1 -- 172.21.2.102:0/3230190803 learned_addr learned my addr 172.21.2.102:0/3230190803 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:41:43.258 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:43.253 7faedffff700 1 -- 172.21.2.102:0/3230190803 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7faee0188110 msgr2=0x7faee0188510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:43.258 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:43.253 7faedffff700 1 --2- 172.21.2.102:0/3230190803 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7faee0188110 0x7faee0188510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:43.258 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:43.253 7faedffff700 1 -- 172.21.2.102:0/3230190803 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7faee0188b30 msgr2=0x7faee018ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:43.258 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:43.253 7faedffff700 1 --2- 172.21.2.102:0/3230190803 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7faee0188b30 0x7faee018ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:43.258 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:43.253 7faedffff700 1 -- 172.21.2.102:0/3230190803 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faee018fc20 con 0x7faee018f200 2021-02-10T11:41:43.258 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:43.253 7faedffff700 1 --2- 172.21.2.102:0/3230190803 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7faee018f200 0x7faee018f600 secure :-1 s=READY pgs=3443 cs=0 l=1 rev1=1 rx=0x7faed0009640 tx=0x7faed000b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:43.258 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:43.253 7faec7fff700 1 -- 172.21.2.102:0/3230190803 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faed000bd50 con 0x7faee018f200 2021-02-10T11:41:43.258 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:43.253 7faec7fff700 1 -- 172.21.2.102:0/3230190803 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faed0006240 con 0x7faee018f200 2021-02-10T11:41:43.258 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:43.253 7faec7fff700 1 -- 172.21.2.102:0/3230190803 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faed00054a0 con 0x7faee018f200 2021-02-10T11:41:43.258 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:43.253 7faee5f7b700 1 -- 172.21.2.102:0/3230190803 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faee018fc20 con 0x7faee018f200 2021-02-10T11:41:43.259 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:43.253 7faee5f7b700 1 -- 172.21.2.102:0/3230190803 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faee01904d0 con 0x7faee018f200 2021-02-10T11:41:43.259 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:43.254 7faec7fff700 1 -- 172.21.2.102:0/3230190803 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7faed0006420 con 0x7faee018f200 2021-02-10T11:41:43.259 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:43.254 7faec7fff700 1 --2- 172.21.2.102:0/3230190803 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7faec00186c0 0x7faec001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:43.259 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:43.254 7faec7fff700 1 -- 172.21.2.102:0/3230190803 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 5573+0+0 (secure 0 0 0) 0x7faed00325c0 con 0x7faee018f200 2021-02-10T11:41:43.259 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:43.254 7faedf7fe700 1 --2- 172.21.2.102:0/3230190803 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7faec00186c0 0x7faec001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:43.259 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:43.254 7faedf7fe700 1 --2- 172.21.2.102:0/3230190803 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7faec00186c0 0x7faec001abc0 crc :-1 s=READY pgs=1498 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:43.288 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:43.285 7faee5f7b700 1 -- 172.21.2.102:0/3230190803 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faee0189190 con 0x7faee018f200 2021-02-10T11:41:43.290 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:43.288 7faec7fff700 1 -- 172.21.2.102:0/3230190803 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7faed0009100 con 0x7faee018f200 2021-02-10T11:41:43.422 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:43.420 7faee5f7b700 1 -- 172.21.2.102:0/3230190803 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestVolumes.test_subvolume_clone_in_progress_getpath"]} v 0) v1 -- 0x7faee0189190 con 0x7faee018f200 2021-02-10T11:41:44.228 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:44.225 7faec7fff700 1 -- 172.21.2.102:0/3230190803 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestVolumes.test_subvolume_clone_in_progress_getpath"]}]=0 v953) v1 ==== 157+0+0 (secure 0 0 0) 0x7faed0006400 con 0x7faee018f200 2021-02-10T11:41:44.243 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:44.240 7faee5f7b700 1 -- 172.21.2.102:0/3230190803 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7faec00186c0 msgr2=0x7faec001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:44.244 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:44.240 7faee5f7b700 1 --2- 172.21.2.102:0/3230190803 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7faec00186c0 0x7faec001abc0 crc :-1 s=READY pgs=1498 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:44.244 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:44.240 7faee5f7b700 1 -- 172.21.2.102:0/3230190803 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7faee018f200 msgr2=0x7faee018f600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:44.244 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:44.240 7faee5f7b700 1 --2- 172.21.2.102:0/3230190803 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7faee018f200 0x7faee018f600 secure :-1 s=READY pgs=3443 cs=0 l=1 rev1=1 rx=0x7faed0009640 tx=0x7faed000b960).stop 2021-02-10T11:41:44.244 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:44.241 7faee5f7b700 1 -- 172.21.2.102:0/3230190803 shutdown_connections 2021-02-10T11:41:44.244 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:44.241 7faee5f7b700 1 --2- 172.21.2.102:0/3230190803 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7faee018f200 0x7faee018f600 unknown :-1 s=CLOSED pgs=3443 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:44.244 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:44.241 7faee5f7b700 1 --2- 172.21.2.102:0/3230190803 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7faec00186c0 0x7faec001abc0 unknown :-1 s=CLOSED pgs=1498 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:44.244 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:44.241 7faee5f7b700 1 --2- 172.21.2.102:0/3230190803 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7faee0188b30 0x7faee018ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:44.244 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:44.241 7faee5f7b700 1 --2- 172.21.2.102:0/3230190803 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7faee0188110 0x7faee0188510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:44.244 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:44.241 7faee5f7b700 1 -- 172.21.2.102:0/3230190803 >> 172.21.2.102:0/3230190803 conn(0x7faee019f7b0 msgr2=0x7faee01af800 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:44.245 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:44.241 7faee5f7b700 1 -- 172.21.2.102:0/3230190803 shutdown_connections 2021-02-10T11:41:44.245 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:44.241 7faee5f7b700 1 -- 172.21.2.102:0/3230190803 wait complete. 2021-02-10T11:41:44.316 INFO:tasks.cephfs_test_runner:test_subvolume_clone_in_progress_getpath (tasks.cephfs.test_volumes.TestVolumes) ... ERROR 2021-02-10T11:41:44.316 INFO:tasks.cephfs_test_runner: 2021-02-10T11:41:44.316 INFO:tasks.cephfs_test_runner:====================================================================== 2021-02-10T11:41:44.316 INFO:tasks.cephfs_test_runner:ERROR: test_subvolume_clone_in_progress_getpath (tasks.cephfs.test_volumes.TestVolumes) 2021-02-10T11:41:44.316 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2021-02-10T11:41:44.317 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2021-02-10T11:41:44.317 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_ceph_ceph-c_962fac13b1a781b8c7d25c1eed50d93813b8f5f0/qa/tasks/cephfs/test_volumes.py", line 4052, in test_subvolume_clone_in_progress_getpath 2021-02-10T11:41:44.317 INFO:tasks.cephfs_test_runner: raise RuntimeError("expected fetching path of an pending clone to fail") 2021-02-10T11:41:44.317 INFO:tasks.cephfs_test_runner:RuntimeError: expected fetching path of an pending clone to fail 2021-02-10T11:41:44.317 INFO:tasks.cephfs_test_runner: 2021-02-10T11:41:44.317 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2021-02-10T11:41:44.317 INFO:tasks.cephfs_test_runner:Ran 21 tests in 2217.225s 2021-02-10T11:41:44.317 INFO:tasks.cephfs_test_runner: 2021-02-10T11:41:44.317 INFO:tasks.cephfs_test_runner:FAILED (errors=1, skipped=1) 2021-02-10T11:41:44.317 INFO:tasks.cephfs_test_runner: 2021-02-10T11:41:44.317 INFO:tasks.cephfs_test_runner:====================================================================== 2021-02-10T11:41:44.317 INFO:tasks.cephfs_test_runner:ERROR: test_subvolume_clone_in_progress_getpath (tasks.cephfs.test_volumes.TestVolumes) 2021-02-10T11:41:44.317 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2021-02-10T11:41:44.317 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2021-02-10T11:41:44.318 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_ceph_ceph-c_962fac13b1a781b8c7d25c1eed50d93813b8f5f0/qa/tasks/cephfs/test_volumes.py", line 4052, in test_subvolume_clone_in_progress_getpath 2021-02-10T11:41:44.318 INFO:tasks.cephfs_test_runner: raise RuntimeError("expected fetching path of an pending clone to fail") 2021-02-10T11:41:44.318 INFO:tasks.cephfs_test_runner:RuntimeError: expected fetching path of an pending clone to fail 2021-02-10T11:41:44.318 INFO:tasks.cephfs_test_runner: 2021-02-10T11:41:44.318 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/teuthology/run_tasks.py", line 94, in run_tasks manager.__enter__() File "/usr/lib/python3.6/contextlib.py", line 81, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_962fac13b1a781b8c7d25c1eed50d93813b8f5f0/qa/tasks/cephfs_test_runner.py", line 207, in task raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests))) RuntimeError: Test failure: test_subvolume_clone_in_progress_getpath (tasks.cephfs.test_volumes.TestVolumes) 2021-02-10T11:41:44.676 ERROR:teuthology.run_tasks: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=51f28f5bae9b41f797f1b6af9e4e0682 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/teuthology/run_tasks.py", line 94, in run_tasks manager.__enter__() File "/usr/lib/python3.6/contextlib.py", line 81, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_962fac13b1a781b8c7d25c1eed50d93813b8f5f0/qa/tasks/cephfs_test_runner.py", line 207, in task raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests))) RuntimeError: Test failure: test_subvolume_clone_in_progress_getpath (tasks.cephfs.test_volumes.TestVolumes) 2021-02-10T11:41:44.678 DEBUG:teuthology.run_tasks:Unwinding manager cephfs_test_runner 2021-02-10T11:41:44.691 DEBUG:teuthology.run_tasks:Unwinding manager ceph-fuse 2021-02-10T11:41:44.714 INFO:tasks.ceph_fuse:Unmounting ceph-fuse clients... 2021-02-10T11:41:44.715 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:41:44.715 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:41:44.715 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:41:45.071 INFO:teuthology.orchestra.run.gibba002.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.0’: No such file or directory 2021-02-10T11:41:45.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:41:45.071 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.0 2021-02-10T11:41:45.072 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:41:45.072 DEBUG:teuthology.orchestra.run.gibba002:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:41:45.072 DEBUG:teuthology.orchestra.run.gibba002:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:41:45.408 INFO:teuthology.orchestra.run.gibba002.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.1’: No such file or directory 2021-02-10T11:41:45.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:41:45.409 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2021-02-10T11:41:45.409 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:41:45.410 DEBUG:teuthology.orchestra.run.gibba013:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:41:45.410 DEBUG:teuthology.orchestra.run.gibba013:> ' -- /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:41:45.746 INFO:teuthology.orchestra.run.gibba013.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.2’: No such file or directory 2021-02-10T11:41:45.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:41:45.747 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2021-02-10T11:41:45.747 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:41:45.747 DEBUG:teuthology.orchestra.run.gibba013:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:41:45.747 DEBUG:teuthology.orchestra.run.gibba013:> ' -- /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:41:46.081 INFO:teuthology.orchestra.run.gibba013.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.3’: No such file or directory 2021-02-10T11:41:46.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:41:46.082 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.3 2021-02-10T11:41:46.082 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2021-02-10T11:41:46.102 INFO:tasks.ceph.ceph_manager.ceph:Canceling any pending splits or merges... 2021-02-10T11:41:46.103 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:41:46.534 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:46.516 7f00d640a700 1 Processor -- start 2021-02-10T11:41:46.535 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:46.516 7f00d640a700 1 -- start start 2021-02-10T11:41:46.535 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:46.516 7f00d640a700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f00d00d1870 0x7f00d01b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:46.535 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:46.516 7f00d640a700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f00d00f0650 0x7f00d00cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:46.535 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:46.516 7f00d640a700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00d00f0b10 con 0x7f00d01b0a10 2021-02-10T11:41:46.535 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:46.516 7f00d640a700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00d00e5ed0 con 0x7f00d00d1870 2021-02-10T11:41:46.535 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:46.516 7f00d640a700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00d00d1c70 con 0x7f00d00f0650 2021-02-10T11:41:46.536 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:46.516 7f00cffff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f00d00d1870 0x7f00d01b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:46.536 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:46.516 7f00cffff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f00d00d1870 0x7f00d01b03f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:41674/0 (socket says 172.21.2.102:41674) 2021-02-10T11:41:46.536 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:46.516 7f00cffff700 1 -- 172.21.2.102:0/4281250609 learned_addr learned my addr 172.21.2.102:0/4281250609 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:41:46.536 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:46.517 7f00cffff700 1 -- 172.21.2.102:0/4281250609 >> v2:172.21.2.113:3300/0 conn(0x7f00d00f0650 msgr2=0x7f00d00cfb20 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:41:46.537 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:46.517 7f00cffff700 1 --2- 172.21.2.102:0/4281250609 >> v2:172.21.2.113:3300/0 conn(0x7f00d00f0650 0x7f00d00cfb20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:46.537 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:46.517 7f00cffff700 1 -- 172.21.2.102:0/4281250609 >> v1:172.21.2.113:6789/0 conn(0x7f00d01b0a10 legacy=0x7f00d00cf330 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:41:46.537 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:46.517 7f00cffff700 1 -- 172.21.2.102:0/4281250609 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00d00d1e90 con 0x7f00d00d1870 2021-02-10T11:41:46.537 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:46.517 7f00cffff700 1 --2- 172.21.2.102:0/4281250609 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f00d00d1870 0x7f00d01b03f0 secure :-1 s=READY pgs=3444 cs=0 l=1 rev1=1 rx=0x7f00c0006700 tx=0x7f00c0009b80).ready entity=mon.0 client_cookie=7251f4977bdbf7cd server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:46.537 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:46.517 7f00ceffd700 1 -- 172.21.2.102:0/4281250609 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f00c000c0c0 con 0x7f00d00d1870 2021-02-10T11:41:46.537 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:46.517 7f00ceffd700 1 -- 172.21.2.102:0/4281250609 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00c000c340 con 0x7f00d00d1870 2021-02-10T11:41:46.537 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:46.517 7f00ceffd700 1 -- 172.21.2.102:0/4281250609 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f00c000c860 con 0x7f00d00d1870 2021-02-10T11:41:46.537 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:46.517 7f00d640a700 1 -- 172.21.2.102:0/4281250609 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f00d00d1870 msgr2=0x7f00d01b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:46.538 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:46.517 7f00d640a700 1 --2- 172.21.2.102:0/4281250609 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f00d00d1870 0x7f00d01b03f0 secure :-1 s=READY pgs=3444 cs=0 l=1 rev1=1 rx=0x7f00c0006700 tx=0x7f00c0009b80).stop 2021-02-10T11:41:46.538 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:46.518 7f00d640a700 1 -- 172.21.2.102:0/4281250609 shutdown_connections 2021-02-10T11:41:46.538 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:46.518 7f00d640a700 1 --2- 172.21.2.102:0/4281250609 >> v2:172.21.2.113:3300/0 conn(0x7f00d00f0650 0x7f00d00cfb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:46.538 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:46.518 7f00d640a700 1 --2- 172.21.2.102:0/4281250609 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f00d00d1870 0x7f00d01b03f0 unknown :-1 s=CLOSED pgs=3444 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:46.538 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:46.518 7f00d640a700 1 -- 172.21.2.102:0/4281250609 >> 172.21.2.102:0/4281250609 conn(0x7f00d00ce450 msgr2=0x7f00d00d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:46.538 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:46.518 7f00d640a700 1 -- 172.21.2.102:0/4281250609 shutdown_connections 2021-02-10T11:41:46.538 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:46.518 7f00d640a700 1 -- 172.21.2.102:0/4281250609 wait complete. 2021-02-10T11:41:46.542 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:46.532 7f00d640a700 1 Processor -- start 2021-02-10T11:41:46.542 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:46.532 7f00d640a700 1 -- start start 2021-02-10T11:41:46.543 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:46.533 7f00d640a700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f00d00b0ad0 0x7f00d0091f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:46.543 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:46.533 7f00d640a700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f00d01a8850 0x7f00d01a8c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:46.543 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:46.533 7f00d640a700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f00d01a6fa0 0x7f00d01a73a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:46.543 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:46.533 7f00d640a700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00d01abfe0 con 0x7f00d00b0ad0 2021-02-10T11:41:46.543 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:46.533 7f00d640a700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00d01a9c40 con 0x7f00d01a8850 2021-02-10T11:41:46.543 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:46.533 7f00d640a700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00d01a9730 con 0x7f00d01a6fa0 2021-02-10T11:41:46.543 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:46.533 7f00cffff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f00d00b0ad0 0x7f00d0091f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:46.543 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:46.533 7f00cffff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f00d00b0ad0 0x7f00d0091f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:41680/0 (socket says 172.21.2.102:41680) 2021-02-10T11:41:46.544 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:46.533 7f00cffff700 1 -- 172.21.2.102:0/1072600817 learned_addr learned my addr 172.21.2.102:0/1072600817 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:41:46.544 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:46.533 7f00cffff700 1 -- 172.21.2.102:0/1072600817 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f00d01a6fa0 msgr2=0x7f00d01a73a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:46.544 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:46.533 7f00d4a27700 1 --2- 172.21.2.102:0/1072600817 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f00d01a6fa0 0x7f00d01a73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:46.544 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:46.533 7f00cffff700 1 --2- 172.21.2.102:0/1072600817 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f00d01a6fa0 0x7f00d01a73a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:46.544 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:46.534 7f00cffff700 1 -- 172.21.2.102:0/1072600817 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f00d01a8850 msgr2=0x7f00d01a8c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:46.544 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:46.533 7f00cf7fe700 1 --2- 172.21.2.102:0/1072600817 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f00d01a8850 0x7f00d01a8c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:46.544 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:46.534 7f00cffff700 1 --2- 172.21.2.102:0/1072600817 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f00d01a8850 0x7f00d01a8c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:46.545 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:46.534 7f00cffff700 1 -- 172.21.2.102:0/1072600817 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00d01a9270 con 0x7f00d00b0ad0 2021-02-10T11:41:46.545 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:46.534 7f00d4a27700 1 --2- 172.21.2.102:0/1072600817 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f00d01a6fa0 0x7f00d01a73a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:41:46.545 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:46.534 7f00cf7fe700 1 --2- 172.21.2.102:0/1072600817 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f00d01a8850 0x7f00d01a8c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:41:46.545 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:46.534 7f00cffff700 1 --2- 172.21.2.102:0/1072600817 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f00d00b0ad0 0x7f00d0091f30 secure :-1 s=READY pgs=3445 cs=0 l=1 rev1=1 rx=0x7f00c000b5e0 tx=0x7f00c000b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:46.545 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:46.534 7f00ccff9700 1 -- 172.21.2.102:0/1072600817 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f00c000bd70 con 0x7f00d00b0ad0 2021-02-10T11:41:46.545 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:46.534 7f00ccff9700 1 -- 172.21.2.102:0/1072600817 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00c0002030 con 0x7f00d00b0ad0 2021-02-10T11:41:46.545 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:46.534 7f00ccff9700 1 -- 172.21.2.102:0/1072600817 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f00c0002550 con 0x7f00d00b0ad0 2021-02-10T11:41:46.545 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:46.534 7f00d640a700 1 -- 172.21.2.102:0/1072600817 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00d01a9270 con 0x7f00d00b0ad0 2021-02-10T11:41:46.546 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:46.534 7f00d640a700 1 -- 172.21.2.102:0/1072600817 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00d01a7cb0 con 0x7f00d00b0ad0 2021-02-10T11:41:46.546 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:46.535 7f00ccff9700 1 -- 172.21.2.102:0/1072600817 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f00c0000c90 con 0x7f00d00b0ad0 2021-02-10T11:41:46.546 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:46.535 7f00ccff9700 1 --2- 172.21.2.102:0/1072600817 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f00b00186c0 0x7f00b001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:46.546 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:46.535 7f00ccff9700 1 -- 172.21.2.102:0/1072600817 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f00c0032580 con 0x7f00d00b0ad0 2021-02-10T11:41:46.546 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:46.535 7f00cf7fe700 1 --2- 172.21.2.102:0/1072600817 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f00b00186c0 0x7f00b001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:46.547 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:46.535 7f00cf7fe700 1 --2- 172.21.2.102:0/1072600817 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f00b00186c0 0x7f00b001abc0 crc :-1 s=READY pgs=1499 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:46.568 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:46.566 7f00d640a700 1 -- 172.21.2.102:0/1072600817 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00d00ce450 con 0x7f00d00b0ad0 2021-02-10T11:41:46.570 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:46.568 7f00ccff9700 1 -- 172.21.2.102:0/1072600817 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f00c0023990 con 0x7f00d00b0ad0 2021-02-10T11:41:46.696 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:46.693 7f00d640a700 1 -- 172.21.2.102:0/1072600817 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f00d00ce450 con 0x7f00d00b0ad0 2021-02-10T11:41:46.696 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:46.694 7f00ccff9700 1 -- 172.21.2.102:0/1072600817 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v197) v1 ==== 74+0+14007 (secure 0 0 0) 0x7f00c0023990 con 0x7f00d00b0ad0 2021-02-10T11:41:46.697 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:41:46.697 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":197,"fsid":"68db30ea-dbce-4aef-a6df-f01c6fea0513","created":"2021-02-10 11:03:06.331557","modified":"2021-02-10 11:40:25.367346","last_up_change":"2021-02-10 11:03:32.690562","last_in_change":"2021-02-10 11:03:32.690562","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":43,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:03:37.322519","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":42,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:40:22.330567","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"196","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":43,"pool_name":"cephfs_data","create_time":"2021-02-10 11:40:23.357509","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"197","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"be691b4d-a039-408a-af80-9d53f02b6f1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":185,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6818","nonce":36088},{"type":"v1","addr":"172.21.2.102:6820","nonce":36088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6822","nonce":36088},{"type":"v1","addr":"172.21.2.102:6824","nonce":36088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6830","nonce":36088},{"type":"v1","addr":"172.21.2.102:6832","nonce":36088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6826","nonce":36088},{"type":"v1","addr":"172.21.2.102:6828","nonce":36088}]},"public_addr":"172.21.2.102:6820/36088","cluster_addr":"172.21.2.102:6824/36088","heartbeat_back_addr":"172.21.2.102:6832/36088","heartbeat_front_addr":"172.21.2.102:6828/36088","state":["exists","up"]},{"osd":1,"uuid":"64e3ccc8-9c69-4d7c-865e-af8e83822635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6819","nonce":36087},{"type":"v1","addr":"172.21.2.102:6821","nonce":36087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6823","nonce":36087},{"type":"v1","addr":"172.21.2.102:6825","nonce":36087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6831","nonce":36087},{"type":"v1","addr":"172.21.2.102:6833","nonce":36087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6827","nonce":36087},{"type":"v1","addr":"172.21.2.102:6829","nonce":36087}]},"public_addr":"172.21.2.102:6821/36087","cluster_addr":"172.21.2.102:6825/36087","heartbeat_back_addr":"172.21.2.102:6833/36087","heartbeat_front_addr":"172.21.2.102:6829/36087","state":["exists","up"]},{"osd":2,"uuid":"153700af-747f-481f-bdf4-d6072d3a842f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6810","nonce":36089},{"type":"v1","addr":"172.21.2.102:6811","nonce":36089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6812","nonce":36089},{"type":"v1","addr":"172.21.2.102:6813","nonce":36089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6816","nonce":36089},{"type":"v1","addr":"172.21.2.102:6817","nonce":36089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6814","nonce":36089},{"type":"v1","addr":"172.21.2.102:6815","nonce":36089}]},"public_addr":"172.21.2.102:6811/36089","cluster_addr":"172.21.2.102:6813/36089","heartbeat_back_addr":"172.21.2.102:6817/36089","heartbeat_front_addr":"172.21.2.102:6815/36089","state":["exists","up"]},{"osd":3,"uuid":"cbc678f4-83a9-45e6-8830-2f7c232b1ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6802","nonce":36090},{"type":"v1","addr":"172.21.2.102:6803","nonce":36090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6804","nonce":36090},{"type":"v1","addr":"172.21.2.102:6805","nonce":36090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6808","nonce":36090},{"type":"v1","addr":"172.21.2.102:6809","nonce":36090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6806","nonce":36090},{"type":"v1","addr":"172.21.2.102:6807","nonce":36090}]},"public_addr":"172.21.2.102:6803/36090","cluster_addr":"172.21.2.102:6805/36090","heartbeat_back_addr":"172.21.2.102:6809/36090","heartbeat_front_addr":"172.21.2.102:6807/36090","state":["exists","up"]},{"osd":4,"uuid":"c01e4008-b258-4f10-afd2-9e7f37214351","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6808","nonce":36368},{"type":"v1","addr":"172.21.2.113:6811","nonce":36368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6814","nonce":36368},{"type":"v1","addr":"172.21.2.113:6818","nonce":36368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6826","nonce":36368},{"type":"v1","addr":"172.21.2.113:6828","nonce":36368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6820","nonce":36368},{"type":"v1","addr":"172.21.2.113:6824","nonce":36368}]},"public_addr":"172.21.2.113:6811/36368","cluster_addr":"172.21.2.113:6818/36368","heartbeat_back_addr":"172.21.2.113:6828/36368","heartbeat_front_addr":"172.21.2.113:6824/36368","state":["exists","up"]},{"osd":5,"uuid":"551f6007-daf8-4a98-8ee0-c2b16bcccb5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6802","nonce":36366},{"type":"v1","addr":"172.21.2.113:6805","nonce":36366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6807","nonce":36366},{"type":"v1","addr":"172.21.2.113:6810","nonce":36366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6819","nonce":36366},{"type":"v1","addr":"172.21.2.113:6822","nonce":36366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6812","nonce":36366},{"type":"v1","addr":"172.21.2.113:6815","nonce":36366}]},"public_addr":"172.21.2.113:6805/36366","cluster_addr":"172.21.2.113:6810/36366","heartbeat_back_addr":"172.21.2.113:6822/36366","heartbeat_front_addr":"172.21.2.113:6815/36366","state":["exists","up"]},{"osd":6,"uuid":"bf67c040-0e50-48be-b102-c3943bc9ebef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6800","nonce":36367},{"type":"v1","addr":"172.21.2.113:6801","nonce":36367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6803","nonce":36367},{"type":"v1","addr":"172.21.2.113:6804","nonce":36367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6813","nonce":36367},{"type":"v1","addr":"172.21.2.113:6816","nonce":36367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6806","nonce":36367},{"type":"v1","addr":"172.21.2.113:6809","nonce":36367}]},"public_addr":"172.21.2.113:6801/36367","cluster_addr":"172.21.2.113:6804/36367","heartbeat_back_addr":"172.21.2.113:6816/36367","heartbeat_front_addr":"172.21.2.113:6809/36367","state":["exists","up"]},{"osd":7,"uuid":"f5461f80-018d-428f-be89-2eb2bcbe7bb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6817","nonce":36365},{"type":"v1","addr":"172.21.2.113:6821","nonce":36365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6823","nonce":36365},{"type":"v1","addr":"172.21.2.113:6825","nonce":36365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6830","nonce":36365},{"type":"v1","addr":"172.21.2.113:6831","nonce":36365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6827","nonce":36365},{"type":"v1","addr":"172.21.2.113:6829","nonce":36365}]},"public_addr":"172.21.2.113:6821/36365","cluster_addr":"172.21.2.113:6825/36365","heartbeat_back_addr":"172.21.2.113:6831/36365","heartbeat_front_addr":"172.21.2.113:6829/36365","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:41:46.698 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:46.696 7f00d640a700 1 -- 172.21.2.102:0/1072600817 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f00b00186c0 msgr2=0x7f00b001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:46.698 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:46.696 7f00d640a700 1 --2- 172.21.2.102:0/1072600817 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f00b00186c0 0x7f00b001abc0 crc :-1 s=READY pgs=1499 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:46.698 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:46.696 7f00d640a700 1 -- 172.21.2.102:0/1072600817 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f00d00b0ad0 msgr2=0x7f00d0091f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:46.699 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:46.696 7f00d640a700 1 --2- 172.21.2.102:0/1072600817 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f00d00b0ad0 0x7f00d0091f30 secure :-1 s=READY pgs=3445 cs=0 l=1 rev1=1 rx=0x7f00c000b5e0 tx=0x7f00c000b980).stop 2021-02-10T11:41:46.699 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:46.696 7f00d640a700 1 -- 172.21.2.102:0/1072600817 shutdown_connections 2021-02-10T11:41:46.699 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:46.696 7f00d640a700 1 --2- 172.21.2.102:0/1072600817 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f00d01a6fa0 0x7f00d01a73a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:46.699 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:46.696 7f00d640a700 1 --2- 172.21.2.102:0/1072600817 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f00b00186c0 0x7f00b001abc0 unknown :-1 s=CLOSED pgs=1499 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:46.699 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:46.696 7f00d640a700 1 --2- 172.21.2.102:0/1072600817 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f00d01a8850 0x7f00d01a8c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:46.699 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:46.696 7f00d640a700 1 --2- 172.21.2.102:0/1072600817 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f00d00b0ad0 0x7f00d0091f30 unknown :-1 s=CLOSED pgs=3445 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:46.699 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:46.696 7f00d640a700 1 -- 172.21.2.102:0/1072600817 >> 172.21.2.102:0/1072600817 conn(0x7f00d019f8f0 msgr2=0x7f00d01a5f80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:46.699 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:46.696 7f00d640a700 1 -- 172.21.2.102:0/1072600817 shutdown_connections 2021-02-10T11:41:46.700 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:46.696 7f00d640a700 1 -- 172.21.2.102:0/1072600817 wait complete. 2021-02-10T11:41:46.707 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph pg dump --format=json 2021-02-10T11:41:47.104 DEBUG:teuthology.orchestra.run.gibba002:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:41:47.106 DEBUG:teuthology.orchestra.run.gibba013:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:41:47.139 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:47.119 7f5cac1a2700 1 Processor -- start 2021-02-10T11:41:47.139 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:47.119 7f5cac1a2700 1 -- start start 2021-02-10T11:41:47.139 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:47.120 7f5cac1a2700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5ca40d0340 0x7f5ca40ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:47.140 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:47.120 7f5cac1a2700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f5ca40d1c70 0x7f5ca40efc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:47.140 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:47.120 7f5cac1a2700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ca40f0b10 con 0x7f5ca40d1870 2021-02-10T11:41:47.140 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:47.120 7f5cac1a2700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ca40e5ed0 con 0x7f5ca40d1c70 2021-02-10T11:41:47.140 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:47.120 7f5cac1a2700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ca41b0be0 con 0x7f5ca40d0340 2021-02-10T11:41:47.140 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:47.120 7f5caa7bf700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f5ca40d1c70 0x7f5ca40efc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:47.140 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:47.120 7f5caa7bf700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f5ca40d1c70 0x7f5ca40efc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:41690/0 (socket says 172.21.2.102:41690) 2021-02-10T11:41:47.140 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:47.120 7f5caa7bf700 1 -- 172.21.2.102:0/719522474 learned_addr learned my addr 172.21.2.102:0/719522474 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:41:47.141 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:47.120 7f5ca97bd700 1 --2- 172.21.2.102:0/719522474 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5ca40d0340 0x7f5ca40ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:47.141 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:47.121 7f5caa7bf700 1 -- 172.21.2.102:0/719522474 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5ca40d0340 msgr2=0x7f5ca40ef640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:41:47.141 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:47.121 7f5caa7bf700 1 --2- 172.21.2.102:0/719522474 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5ca40d0340 0x7f5ca40ef640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:47.141 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:47.121 7f5caa7bf700 1 -- 172.21.2.102:0/719522474 >> v1:172.21.2.113:6789/0 conn(0x7f5ca40d1870 legacy=0x7f5ca41b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:47.141 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:47.121 7f5caa7bf700 1 -- 172.21.2.102:0/719522474 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ca40f20a0 con 0x7f5ca40d1c70 2021-02-10T11:41:47.141 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:47.121 7f5caa7bf700 1 --2- 172.21.2.102:0/719522474 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5ca40d1c70 0x7f5ca40efc60 secure :-1 s=READY pgs=3446 cs=0 l=1 rev1=1 rx=0x7f5c9c006f00 tx=0x7f5c9c009e90).ready entity=mon.0 client_cookie=8ffd12691f956c97 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:47.141 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:47.121 7f5ca8fbc700 1 -- 172.21.2.102:0/719522474 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5c9c00b300 con 0x7f5ca40d1c70 2021-02-10T11:41:47.141 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:47.121 7f5ca8fbc700 1 -- 172.21.2.102:0/719522474 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c9c00b640 con 0x7f5ca40d1c70 2021-02-10T11:41:47.141 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:47.121 7f5ca8fbc700 1 -- 172.21.2.102:0/719522474 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5c9c00bb20 con 0x7f5ca40d1c70 2021-02-10T11:41:47.142 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:47.121 7f5cac1a2700 1 -- 172.21.2.102:0/719522474 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5ca40d1c70 msgr2=0x7f5ca40efc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:47.142 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:47.121 7f5cac1a2700 1 --2- 172.21.2.102:0/719522474 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5ca40d1c70 0x7f5ca40efc60 secure :-1 s=READY pgs=3446 cs=0 l=1 rev1=1 rx=0x7f5c9c006f00 tx=0x7f5c9c009e90).stop 2021-02-10T11:41:47.142 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:47.122 7f5cac1a2700 1 -- 172.21.2.102:0/719522474 shutdown_connections 2021-02-10T11:41:47.142 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:47.122 7f5cac1a2700 1 --2- 172.21.2.102:0/719522474 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5ca40d0340 0x7f5ca40ef640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:47.142 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:47.122 7f5cac1a2700 1 --2- 172.21.2.102:0/719522474 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5ca40d1c70 0x7f5ca40efc60 unknown :-1 s=CLOSED pgs=3446 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:47.142 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:47.122 7f5cac1a2700 1 -- 172.21.2.102:0/719522474 >> 172.21.2.102:0/719522474 conn(0x7f5ca40ce450 msgr2=0x7f5ca40d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:47.142 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:47.122 7f5cac1a2700 1 -- 172.21.2.102:0/719522474 shutdown_connections 2021-02-10T11:41:47.142 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:47.122 7f5cac1a2700 1 -- 172.21.2.102:0/719522474 wait complete. 2021-02-10T11:41:47.144 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:47.137 7f5cac1a2700 1 Processor -- start 2021-02-10T11:41:47.144 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:47.137 7f5cac1a2700 1 -- start start 2021-02-10T11:41:47.144 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:47.138 7f5cac1a2700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5ca40ada90 0x7f5ca40ade90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:47.144 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:47.138 7f5cac1a2700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5ca4099d20 0x7f5ca409a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:47.145 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:47.138 7f5cac1a2700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5ca409a740 0x7f5ca4097bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:47.145 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:47.138 7f5cac1a2700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ca40997a0 con 0x7f5ca409a740 2021-02-10T11:41:47.145 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:47.138 7f5cac1a2700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ca418f0c0 con 0x7f5ca4099d20 2021-02-10T11:41:47.145 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:47.138 7f5cac1a2700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ca409ab40 con 0x7f5ca40ada90 2021-02-10T11:41:47.145 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:47.138 7f5caa7bf700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5ca409a740 0x7f5ca4097bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:47.145 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:47.138 7f5caa7bf700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5ca409a740 0x7f5ca4097bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:41696/0 (socket says 172.21.2.102:41696) 2021-02-10T11:41:47.145 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:47.138 7f5caa7bf700 1 -- 172.21.2.102:0/4279697665 learned_addr learned my addr 172.21.2.102:0/4279697665 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:41:47.146 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:47.138 7f5caa7bf700 1 -- 172.21.2.102:0/4279697665 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5ca40ada90 msgr2=0x7f5ca40ade90 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:41:47.146 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:47.138 7f5caa7bf700 1 --2- 172.21.2.102:0/4279697665 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5ca40ada90 0x7f5ca40ade90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:47.146 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:47.138 7f5caa7bf700 1 -- 172.21.2.102:0/4279697665 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5ca4099d20 msgr2=0x7f5ca409a120 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:41:47.146 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:47.138 7f5caa7bf700 1 --2- 172.21.2.102:0/4279697665 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5ca4099d20 0x7f5ca409a120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:47.147 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:47.138 7f5caa7bf700 1 -- 172.21.2.102:0/4279697665 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ca409ad60 con 0x7f5ca409a740 2021-02-10T11:41:47.147 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:47.138 7f5caa7bf700 1 --2- 172.21.2.102:0/4279697665 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5ca409a740 0x7f5ca4097bc0 secure :-1 s=READY pgs=3447 cs=0 l=1 rev1=1 rx=0x7f5c9c0099f0 tx=0x7f5c9c00bce0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:47.147 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:47.138 7f5c967fc700 1 -- 172.21.2.102:0/4279697665 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5c9c00be20 con 0x7f5ca409a740 2021-02-10T11:41:47.147 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:47.138 7f5c967fc700 1 -- 172.21.2.102:0/4279697665 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c9c003030 con 0x7f5ca409a740 2021-02-10T11:41:47.148 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:47.139 7f5cac1a2700 1 -- 172.21.2.102:0/4279697665 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ca409ad60 con 0x7f5ca409a740 2021-02-10T11:41:47.148 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:47.139 7f5c967fc700 1 -- 172.21.2.102:0/4279697665 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5c9c0014f0 con 0x7f5ca409a740 2021-02-10T11:41:47.148 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:47.139 7f5cac1a2700 1 -- 172.21.2.102:0/4279697665 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ca409ad60 con 0x7f5ca409a740 2021-02-10T11:41:47.148 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:47.139 7f5c967fc700 1 -- 172.21.2.102:0/4279697665 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f5c9c001270 con 0x7f5ca409a740 2021-02-10T11:41:47.148 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:47.139 7f5c967fc700 1 --2- 172.21.2.102:0/4279697665 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5c8c018730 0x7f5c8c01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:47.148 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:47.139 7f5c967fc700 1 -- 172.21.2.102:0/4279697665 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f5c9c0329f0 con 0x7f5ca409a740 2021-02-10T11:41:47.148 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:47.139 7f5ca9fbe700 1 --2- 172.21.2.102:0/4279697665 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5c8c018730 0x7f5c8c01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:47.148 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:47.140 7f5ca9fbe700 1 --2- 172.21.2.102:0/4279697665 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5c8c018730 0x7f5c8c01ac10 crc :-1 s=READY pgs=1500 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:47.172 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:47.169 7f5cac1a2700 1 -- 172.21.2.102:0/4279697665 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ca40ebc00 con 0x7f5ca409a740 2021-02-10T11:41:47.174 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:47.172 7f5c967fc700 1 -- 172.21.2.102:0/4279697665 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f5c9c000cc0 con 0x7f5ca409a740 2021-02-10T11:41:47.310 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:47.307 7f5cac1a2700 1 -- 172.21.2.102:0/4279697665 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "pg dump", "target": ["mgr", ""], "format": "json"}) v1 -- 0x7f5ca40ebc00 con 0x7f5c8c018730 2021-02-10T11:41:47.312 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:47.310 7f5c967fc700 1 -- 172.21.2.102:0/4279697665 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 dumped all) v1 ==== 18+0+100969 (crc 0 0 0) 0x7f5ca0006970 con 0x7f5c8c018730 2021-02-10T11:41:47.314 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:41:47.314 INFO:teuthology.orchestra.run.gibba002.stderr:dumped all 2021-02-10T11:41:47.316 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:47.312 7f5cac1a2700 1 -- 172.21.2.102:0/4279697665 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5c8c018730 msgr2=0x7f5c8c01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:47.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:47.312 7f5cac1a2700 1 --2- 172.21.2.102:0/4279697665 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5c8c018730 0x7f5c8c01ac10 crc :-1 s=READY pgs=1500 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:47.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:47.312 7f5cac1a2700 1 -- 172.21.2.102:0/4279697665 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5ca409a740 msgr2=0x7f5ca4097bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:47.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:47.312 7f5cac1a2700 1 --2- 172.21.2.102:0/4279697665 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5ca409a740 0x7f5ca4097bc0 secure :-1 s=READY pgs=3447 cs=0 l=1 rev1=1 rx=0x7f5c9c0099f0 tx=0x7f5c9c00bce0).stop 2021-02-10T11:41:47.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:47.313 7f5cac1a2700 1 -- 172.21.2.102:0/4279697665 shutdown_connections 2021-02-10T11:41:47.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:47.313 7f5cac1a2700 1 --2- 172.21.2.102:0/4279697665 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5ca409a740 0x7f5ca4097bc0 unknown :-1 s=CLOSED pgs=3447 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:47.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:47.313 7f5cac1a2700 1 --2- 172.21.2.102:0/4279697665 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5c8c018730 0x7f5c8c01ac10 unknown :-1 s=CLOSED pgs=1500 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:47.317 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:47.313 7f5cac1a2700 1 --2- 172.21.2.102:0/4279697665 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5ca4099d20 0x7f5ca409a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:47.318 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:47.313 7f5cac1a2700 1 --2- 172.21.2.102:0/4279697665 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5ca40ada90 0x7f5ca40ade90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:47.318 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:47.313 7f5cac1a2700 1 -- 172.21.2.102:0/4279697665 >> 172.21.2.102:0/4279697665 conn(0x7f5ca40f0550 msgr2=0x7f5ca4191730 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:47.318 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:47.313 7f5cac1a2700 1 -- 172.21.2.102:0/4279697665 shutdown_connections 2021-02-10T11:41:47.318 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:47.313 7f5cac1a2700 1 -- 172.21.2.102:0/4279697665 wait complete. 2021-02-10T11:41:47.323 INFO:teuthology.orchestra.run.gibba002.stdout:{"pg_ready":true,"pg_map":{"version":1347,"stamp":"2021-02-10 11:41:46.168024","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":135133296,"num_objects":154,"num_object_clones":1,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":154,"num_whiteouts":0,"num_read":65,"num_read_kb":65537,"num_write":426,"num_write_kb":132064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":411,"ondisk_log_size":411,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":8,"num_per_pool_osds":8,"kb":754974720,"kb_used":8672292,"kb_used_data":283616,"kb_used_omap":0,"kb_used_meta":8388608,"kb_avail":746302428,"statfs":{"total":773094113280,"available":764213686272,"internally_reserved":8589934592,"allocated":290422784,"data_stored":278737232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8589934592},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":70779026,"num_objects":70,"num_object_clones":1,"num_object_copies":140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":70,"num_whiteouts":0,"num_read":65,"num_read_kb":65537,"num_write":195,"num_write_kb":69165,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002393"},"pg_stats":[{"pgid":"43.6","version":"197'22","reported_seq":"40","reported_epoch":"197","state":"active+clean","last_fresh":"2021-02-10 11:41:37.311138","last_change":"2021-02-10 11:40:24.375639","last_active":"2021-02-10 11:41:37.311138","last_peered":"2021-02-10 11:41:37.311138","last_clean":"2021-02-10 11:41:37.311138","last_became_active":"2021-02-10 11:40:24.375377","last_became_peered":"2021-02-10 11:40:24.375377","last_unstale":"2021-02-10 11:41:37.311138","last_undegraded":"2021-02-10 11:41:37.311138","last_fullsized":"2021-02-10 11:41:37.311138","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:40:23.720923","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:40:23.720923","last_clean_scrub_stamp":"2021-02-10 11:40:23.720923","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":23068672,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":9,"num_read_kb":9216,"num_write":22,"num_write_kb":22528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"42.7","version":"197'8","reported_seq":"18","reported_epoch":"197","state":"active+clean","last_fresh":"2021-02-10 11:40:25.375917","last_change":"2021-02-10 11:40:23.729705","last_active":"2021-02-10 11:40:25.375917","last_peered":"2021-02-10 11:40:25.375917","last_clean":"2021-02-10 11:40:25.375917","last_became_active":"2021-02-10 11:40:23.729462","last_became_peered":"2021-02-10 11:40:23.729462","last_unstale":"2021-02-10 11:40:25.375917","last_undegraded":"2021-02-10 11:40:25.375917","last_fullsized":"2021-02-10 11:40:25.375917","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:40:22.718405","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:40:22.718405","last_clean_scrub_stamp":"2021-02-10 11:40:22.718405","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":566,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"43.7","version":"197'14","reported_seq":"30","reported_epoch":"197","state":"active+clean","last_fresh":"2021-02-10 11:41:37.314870","last_change":"2021-02-10 11:40:24.371366","last_active":"2021-02-10 11:41:37.314870","last_peered":"2021-02-10 11:41:37.314870","last_clean":"2021-02-10 11:41:37.314870","last_became_active":"2021-02-10 11:40:24.371106","last_became_peered":"2021-02-10 11:40:24.371106","last_unstale":"2021-02-10 11:41:37.314870","last_undegraded":"2021-02-10 11:41:37.314870","last_fullsized":"2021-02-10 11:41:37.314870","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:40:23.720923","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:40:23.720923","last_clean_scrub_stamp":"2021-02-10 11:40:23.720923","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":14680064,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":7,"num_read_kb":7168,"num_write":14,"num_write_kb":14336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"42.6","version":"197'16","reported_seq":"26","reported_epoch":"197","state":"active+clean","last_fresh":"2021-02-10 11:41:38.724076","last_change":"2021-02-10 11:40:23.729347","last_active":"2021-02-10 11:41:38.724076","last_peered":"2021-02-10 11:41:38.724076","last_clean":"2021-02-10 11:41:38.724076","last_became_active":"2021-02-10 11:40:23.729136","last_became_peered":"2021-02-10 11:40:23.729136","last_unstale":"2021-02-10 11:41:38.724076","last_undegraded":"2021-02-10 11:41:38.724076","last_fullsized":"2021-02-10 11:41:38.724076","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:40:22.718405","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:40:22.718405","last_clean_scrub_stamp":"2021-02-10 11:40:22.718405","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":35,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"43.5","version":"197'23","reported_seq":"41","reported_epoch":"197","state":"active+clean","last_fresh":"2021-02-10 11:41:37.352951","last_change":"2021-02-10 11:40:24.372300","last_active":"2021-02-10 11:41:37.352951","last_peered":"2021-02-10 11:41:37.352951","last_clean":"2021-02-10 11:41:37.352951","last_became_active":"2021-02-10 11:40:24.372091","last_became_peered":"2021-02-10 11:40:24.372091","last_unstale":"2021-02-10 11:41:37.352951","last_undegraded":"2021-02-10 11:41:37.352951","last_fullsized":"2021-02-10 11:41:37.352951","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:40:23.720923","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:40:23.720923","last_clean_scrub_stamp":"2021-02-10 11:40:23.720923","log_size":23,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16777506,"num_objects":18,"num_object_clones":1,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":10,"num_read_kb":10240,"num_write":20,"num_write_kb":16387,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"42.4","version":"197'217","reported_seq":"227","reported_epoch":"197","state":"active+clean","last_fresh":"2021-02-10 11:41:43.725395","last_change":"2021-02-10 11:40:23.728534","last_active":"2021-02-10 11:41:43.725395","last_peered":"2021-02-10 11:41:43.725395","last_clean":"2021-02-10 11:41:43.725395","last_became_active":"2021-02-10 11:40:23.728257","last_became_peered":"2021-02-10 11:40:23.728257","last_unstale":"2021-02-10 11:41:43.725395","last_undegraded":"2021-02-10 11:41:43.725395","last_fullsized":"2021-02-10 11:41:43.725395","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:40:22.718405","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:40:22.718405","last_clean_scrub_stamp":"2021-02-10 11:40:22.718405","log_size":217,"ondisk_log_size":217,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":914280,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":217,"num_write_kb":958,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"43.2","version":"197'20","reported_seq":"37","reported_epoch":"197","state":"active+clean","last_fresh":"2021-02-10 11:41:37.294458","last_change":"2021-02-10 11:40:24.372473","last_active":"2021-02-10 11:41:37.294458","last_peered":"2021-02-10 11:41:37.294458","last_clean":"2021-02-10 11:41:37.294458","last_became_active":"2021-02-10 11:40:24.372214","last_became_peered":"2021-02-10 11:40:24.372214","last_unstale":"2021-02-10 11:41:37.294458","last_undegraded":"2021-02-10 11:41:37.294458","last_fullsized":"2021-02-10 11:41:37.294458","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:40:23.720923","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:40:23.720923","last_clean_scrub_stamp":"2021-02-10 11:40:23.720923","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20971520,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":8,"num_read_kb":8192,"num_write":20,"num_write_kb":20480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"42.3","version":"197'6","reported_seq":"16","reported_epoch":"197","state":"active+clean","last_fresh":"2021-02-10 11:40:25.502736","last_change":"2021-02-10 11:40:23.728906","last_active":"2021-02-10 11:40:25.502736","last_peered":"2021-02-10 11:40:25.502736","last_clean":"2021-02-10 11:40:25.502736","last_became_active":"2021-02-10 11:40:23.728701","last_became_peered":"2021-02-10 11:40:23.728701","last_unstale":"2021-02-10 11:40:25.502736","last_undegraded":"2021-02-10 11:40:25.502736","last_fullsized":"2021-02-10 11:40:25.502736","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:40:22.718405","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:40:22.718405","last_clean_scrub_stamp":"2021-02-10 11:40:22.718405","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"43.3","version":"197'25","reported_seq":"42","reported_epoch":"197","state":"active+clean","last_fresh":"2021-02-10 11:41:37.358591","last_change":"2021-02-10 11:40:24.371035","last_active":"2021-02-10 11:41:37.358591","last_peered":"2021-02-10 11:41:37.358591","last_clean":"2021-02-10 11:41:37.358591","last_became_active":"2021-02-10 11:40:24.370863","last_became_peered":"2021-02-10 11:40:24.370863","last_unstale":"2021-02-10 11:41:37.358591","last_undegraded":"2021-02-10 11:41:37.358591","last_fullsized":"2021-02-10 11:41:37.358591","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:40:23.720923","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:40:23.720923","last_clean_scrub_stamp":"2021-02-10 11:40:23.720923","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":18874505,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":8,"num_read_kb":8192,"num_write":25,"num_write_kb":18436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"42.2","version":"196'1","reported_seq":"11","reported_epoch":"196","state":"active+clean","last_fresh":"2021-02-10 11:40:24.372892","last_change":"2021-02-10 11:40:23.728597","last_active":"2021-02-10 11:40:24.372892","last_peered":"2021-02-10 11:40:24.372892","last_clean":"2021-02-10 11:40:24.372892","last_became_active":"2021-02-10 11:40:23.728297","last_became_peered":"2021-02-10 11:40:23.728297","last_unstale":"2021-02-10 11:40:24.372892","last_undegraded":"2021-02-10 11:40:24.372892","last_fullsized":"2021-02-10 11:40:24.372892","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:40:22.718405","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:40:22.718405","last_clean_scrub_stamp":"2021-02-10 11:40:22.718405","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"43.0","version":"197'19","reported_seq":"38","reported_epoch":"197","state":"active+clean","last_fresh":"2021-02-10 11:41:37.271520","last_change":"2021-02-10 11:40:24.372861","last_active":"2021-02-10 11:41:37.271520","last_peered":"2021-02-10 11:41:37.271520","last_clean":"2021-02-10 11:41:37.271520","last_became_active":"2021-02-10 11:40:24.372579","last_became_peered":"2021-02-10 11:40:24.372579","last_unstale":"2021-02-10 11:41:37.271520","last_undegraded":"2021-02-10 11:41:37.271520","last_fullsized":"2021-02-10 11:41:37.271520","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:40:23.720923","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:40:23.720923","last_clean_scrub_stamp":"2021-02-10 11:40:23.720923","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":18874368,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":10,"num_read_kb":10240,"num_write":18,"num_write_kb":18432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"42.1","version":"197'2","reported_seq":"13","reported_epoch":"197","state":"active+clean","last_fresh":"2021-02-10 11:41:38.383325","last_change":"2021-02-10 11:40:23.728372","last_active":"2021-02-10 11:41:38.383325","last_peered":"2021-02-10 11:41:38.383325","last_clean":"2021-02-10 11:41:38.383325","last_became_active":"2021-02-10 11:40:23.728061","last_became_peered":"2021-02-10 11:40:23.728061","last_unstale":"2021-02-10 11:41:38.383325","last_undegraded":"2021-02-10 11:41:38.383325","last_fullsized":"2021-02-10 11:41:38.383325","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:40:22.718405","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:40:22.718405","last_clean_scrub_stamp":"2021-02-10 11:40:22.718405","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":101,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"43.1","version":"197'11","reported_seq":"26","reported_epoch":"197","state":"active+clean","last_fresh":"2021-02-10 11:41:37.334933","last_change":"2021-02-10 11:40:24.373836","last_active":"2021-02-10 11:41:37.334933","last_peered":"2021-02-10 11:41:37.334933","last_clean":"2021-02-10 11:41:37.334933","last_became_active":"2021-02-10 11:40:24.373558","last_became_peered":"2021-02-10 11:40:24.373558","last_unstale":"2021-02-10 11:41:37.334933","last_undegraded":"2021-02-10 11:41:37.334933","last_fullsized":"2021-02-10 11:41:37.334933","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:40:23.720923","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:40:23.720923","last_clean_scrub_stamp":"2021-02-10 11:40:23.720923","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":10485760,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":6,"num_read_kb":6144,"num_write":10,"num_write_kb":10240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"42.0","version":"196'8","reported_seq":"18","reported_epoch":"196","state":"active+clean","last_fresh":"2021-02-10 11:40:24.377584","last_change":"2021-02-10 11:40:23.729211","last_active":"2021-02-10 11:40:24.377584","last_peered":"2021-02-10 11:40:24.377584","last_clean":"2021-02-10 11:40:24.377584","last_became_active":"2021-02-10 11:40:23.728990","last_became_peered":"2021-02-10 11:40:23.728990","last_unstale":"2021-02-10 11:40:24.377584","last_undegraded":"2021-02-10 11:40:24.377584","last_fullsized":"2021-02-10 11:40:24.377584","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:40:22.718405","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:40:22.718405","last_clean_scrub_stamp":"2021-02-10 11:40:22.718405","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":"190","reported_epoch":"196","state":"active+clean","last_fresh":"2021-02-10 11:40:24.371416","last_change":"2021-02-10 11:03:38.737712","last_active":"2021-02-10 11:40:24.371416","last_peered":"2021-02-10 11:40:24.371416","last_clean":"2021-02-10 11:40:24.371416","last_became_active":"2021-02-10 11:03:38.737435","last_became_peered":"2021-02-10 11:03:38.737435","last_unstale":"2021-02-10 11:40:24.371416","last_undegraded":"2021-02-10 11:40:24.371416","last_fullsized":"2021-02-10 11:40:24.371416","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:03:37.729316","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:03:37.729316","last_clean_scrub_stamp":"2021-02-10 11:03:37.729316","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":"190","reported_epoch":"196","state":"active+clean","last_fresh":"2021-02-10 11:40:24.371275","last_change":"2021-02-10 11:03:38.738386","last_active":"2021-02-10 11:40:24.371275","last_peered":"2021-02-10 11:40:24.371275","last_clean":"2021-02-10 11:40:24.371275","last_became_active":"2021-02-10 11:03:38.738144","last_became_peered":"2021-02-10 11:03:38.738144","last_unstale":"2021-02-10 11:40:24.371275","last_undegraded":"2021-02-10 11:40:24.371275","last_fullsized":"2021-02-10 11:40:24.371275","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:03:37.729316","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:03:37.729316","last_clean_scrub_stamp":"2021-02-10 11:03:37.729316","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":"188","reported_epoch":"196","state":"active+clean","last_fresh":"2021-02-10 11:40:24.373770","last_change":"2021-02-10 11:03:38.738326","last_active":"2021-02-10 11:40:24.373770","last_peered":"2021-02-10 11:40:24.373770","last_clean":"2021-02-10 11:40:24.373770","last_became_active":"2021-02-10 11:03:38.738088","last_became_peered":"2021-02-10 11:03:38.738088","last_unstale":"2021-02-10 11:40:24.373770","last_undegraded":"2021-02-10 11:40:24.373770","last_fullsized":"2021-02-10 11:40:24.373770","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:03:37.729316","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:03:37.729316","last_clean_scrub_stamp":"2021-02-10 11:03:37.729316","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":"180","reported_epoch":"196","state":"active+clean","last_fresh":"2021-02-10 11:40:24.369187","last_change":"2021-02-10 11:03:38.740643","last_active":"2021-02-10 11:40:24.369187","last_peered":"2021-02-10 11:40:24.369187","last_clean":"2021-02-10 11:40:24.369187","last_became_active":"2021-02-10 11:03:38.740334","last_became_peered":"2021-02-10 11:03:38.740334","last_unstale":"2021-02-10 11:40:24.369187","last_undegraded":"2021-02-10 11:40:24.369187","last_fullsized":"2021-02-10 11:40:24.369187","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:03:37.729316","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:03:37.729316","last_clean_scrub_stamp":"2021-02-10 11:03:37.729316","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"180","reported_epoch":"196","state":"active+clean","last_fresh":"2021-02-10 11:40:24.372572","last_change":"2021-02-10 11:03:38.739319","last_active":"2021-02-10 11:40:24.372572","last_peered":"2021-02-10 11:40:24.372572","last_clean":"2021-02-10 11:40:24.372572","last_became_active":"2021-02-10 11:03:38.739150","last_became_peered":"2021-02-10 11:03:38.739150","last_unstale":"2021-02-10 11:40:24.372572","last_undegraded":"2021-02-10 11:40:24.372572","last_fullsized":"2021-02-10 11:40:24.372572","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:03:37.729316","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:03:37.729316","last_clean_scrub_stamp":"2021-02-10 11:03:37.729316","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"43.4","version":"197'10","reported_seq":"25","reported_epoch":"197","state":"active+clean","last_fresh":"2021-02-10 11:41:37.279158","last_change":"2021-02-10 11:40:24.371458","last_active":"2021-02-10 11:41:37.279158","last_peered":"2021-02-10 11:41:37.279158","last_clean":"2021-02-10 11:41:37.279158","last_became_active":"2021-02-10 11:40:24.371177","last_became_peered":"2021-02-10 11:40:24.371177","last_unstale":"2021-02-10 11:41:37.279158","last_undegraded":"2021-02-10 11:41:37.279158","last_fullsized":"2021-02-10 11:41:37.279158","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:40:23.720923","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:40:23.720923","last_clean_scrub_stamp":"2021-02-10 11:40:23.720923","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":10485760,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":6,"num_read_kb":6144,"num_write":10,"num_write_kb":10240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"42.5","version":"197'9","reported_seq":"19","reported_epoch":"197","state":"active+clean","last_fresh":"2021-02-10 11:41:38.383320","last_change":"2021-02-10 11:40:23.729552","last_active":"2021-02-10 11:41:38.383320","last_peered":"2021-02-10 11:41:38.383320","last_clean":"2021-02-10 11:41:38.383320","last_became_active":"2021-02-10 11:40:23.729375","last_became_peered":"2021-02-10 11:40:23.729375","last_unstale":"2021-02-10 11:41:38.383320","last_undegraded":"2021-02-10 11:41:38.383320","last_fullsized":"2021-02-10 11:41:38.383320","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:40:22.718405","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:40:22.718405","last_clean_scrub_stamp":"2021-02-10 11:40:22.718405","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":160,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":"175","reported_epoch":"196","state":"active+clean","last_fresh":"2021-02-10 11:40:24.371231","last_change":"2021-02-10 11:03:38.738931","last_active":"2021-02-10 11:40:24.371231","last_peered":"2021-02-10 11:40:24.371231","last_clean":"2021-02-10 11:40:24.371231","last_became_active":"2021-02-10 11:03:38.738675","last_became_peered":"2021-02-10 11:03:38.738675","last_unstale":"2021-02-10 11:40:24.371231","last_undegraded":"2021-02-10 11:40:24.371231","last_fullsized":"2021-02-10 11:40:24.371231","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:03:37.729316","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:03:37.729316","last_clean_scrub_stamp":"2021-02-10 11:03:37.729316","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":"180","reported_epoch":"196","state":"active+clean","last_fresh":"2021-02-10 11:40:24.372508","last_change":"2021-02-10 11:03:38.739203","last_active":"2021-02-10 11:40:24.372508","last_peered":"2021-02-10 11:40:24.372508","last_clean":"2021-02-10 11:40:24.372508","last_became_active":"2021-02-10 11:03:38.739032","last_became_peered":"2021-02-10 11:03:38.739032","last_unstale":"2021-02-10 11:40:24.372508","last_undegraded":"2021-02-10 11:40:24.372508","last_fullsized":"2021-02-10 11:40:24.372508","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:03:37.729316","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:03:37.729316","last_clean_scrub_stamp":"2021-02-10 11:03:37.729316","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":"178","reported_epoch":"196","state":"active+clean","last_fresh":"2021-02-10 11:40:24.370029","last_change":"2021-02-10 11:03:38.740597","last_active":"2021-02-10 11:40:24.370029","last_peered":"2021-02-10 11:40:24.370029","last_clean":"2021-02-10 11:40:24.370029","last_became_active":"2021-02-10 11:03:38.740308","last_became_peered":"2021-02-10 11:03:38.740308","last_unstale":"2021-02-10 11:40:24.370029","last_undegraded":"2021-02-10 11:40:24.370029","last_fullsized":"2021-02-10 11:40:24.370029","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:03:37.729316","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:03:37.729316","last_clean_scrub_stamp":"2021-02-10 11:03:37.729316","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":43,"num_pg":8,"stat_sum":{"num_bytes":134218155,"num_objects":131,"num_object_clones":1,"num_object_copies":262,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":131,"num_whiteouts":0,"num_read":64,"num_read_kb":65536,"num_write":139,"num_write_kb":131079,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":268460032,"data_stored":268436310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":144,"ondisk_log_size":144,"up":16,"acting":16,"num_store_stats":8},{"poolid":42,"num_pg":8,"stat_sum":{"num_bytes":915141,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":287,"num_write_kb":985,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1900544,"data_stored":1830282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":267,"ondisk_log_size":267,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575310,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1087952,"kb_used_data":39368,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93283888,"statfs":{"total":96636764160,"available":95522701312,"internally_reserved":1073741824,"allocated":40312832,"data_stored":38844297,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 11:40:45 2021","interfaces":[{"interface":"back","average":{"1min":0.90500000000000003,"5min":0.93500000000000005,"15min":0.94899999999999995},"min":{"1min":0.46500000000000002,"5min":0.432,"15min":0.39600000000000002},"max":{"1min":1.2689999999999999,"5min":1.4630000000000001,"15min":1.4630000000000001},"last":1.1439999999999999},{"interface":"front","average":{"1min":0.92700000000000005,"5min":0.91800000000000004,"15min":0.92800000000000005},"min":{"1min":0.39400000000000002,"5min":0.38800000000000001,"15min":0.36799999999999999},"max":{"1min":1.272,"5min":1.367,"15min":1.367},"last":1.123}]},{"osd":1,"last update":"Wed Feb 10 11:40:45 2021","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.79200000000000004,"15min":0.81499999999999995},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":1.087,"5min":1.361,"15min":1.361},"last":1.0409999999999999},{"interface":"front","average":{"1min":0.86199999999999999,"5min":0.81299999999999994,"15min":0.82199999999999995},"min":{"1min":0.47799999999999998,"5min":0.40500000000000003,"15min":0.39100000000000001},"max":{"1min":1.173,"5min":1.288,"15min":1.508},"last":0.86299999999999999}]},{"osd":2,"last update":"Wed Feb 10 11:40:45 2021","interfaces":[{"interface":"back","average":{"1min":0.81000000000000005,"5min":0.81499999999999995,"15min":0.81899999999999995},"min":{"1min":0.51600000000000001,"5min":0.38,"15min":0.38},"max":{"1min":1.2450000000000001,"5min":1.323,"15min":1.3859999999999999},"last":0.90700000000000003},{"interface":"front","average":{"1min":0.85699999999999998,"5min":0.86299999999999999,"15min":0.86599999999999999},"min":{"1min":0.57299999999999995,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1.1919999999999999,"5min":1.2370000000000001,"15min":1.329},"last":0.80000000000000004}]},{"osd":3,"last update":"Wed Feb 10 11:40:45 2021","interfaces":[{"interface":"back","average":{"1min":0.82699999999999996,"5min":0.82299999999999995,"15min":0.84099999999999997},"min":{"1min":0.50600000000000001,"5min":0.45900000000000002,"15min":0.40500000000000003},"max":{"1min":1.1659999999999999,"5min":1.272,"15min":1.4139999999999999},"last":0.78500000000000003},{"interface":"front","average":{"1min":0.89800000000000002,"5min":0.88900000000000001,"15min":0.89300000000000002},"min":{"1min":0.53600000000000003,"5min":0.47799999999999998,"15min":0.45500000000000002},"max":{"1min":1.1850000000000001,"5min":1.3009999999999999,"15min":1.389},"last":0.97299999999999998}]},{"osd":4,"last update":"Wed Feb 10 11:40:45 2021","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.81799999999999995,"15min":0.81000000000000005},"min":{"1min":0.49099999999999999,"5min":0.47099999999999997,"15min":0.46800000000000003},"max":{"1min":1.034,"5min":1.4330000000000001,"15min":1.4330000000000001},"last":0.751},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.82599999999999996,"15min":0.83399999999999996},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.98699999999999999,"5min":1.302,"15min":1.5489999999999999},"last":1.016}]},{"osd":5,"last update":"Wed Feb 10 11:40:45 2021","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.76500000000000001,"15min":0.751},"min":{"1min":0.52100000000000002,"5min":0.49299999999999999,"15min":0.46899999999999997},"max":{"1min":1.0700000000000001,"5min":1.3360000000000001,"15min":1.3360000000000001},"last":1.0900000000000001},{"interface":"front","average":{"1min":0.81599999999999995,"5min":0.78600000000000003,"15min":0.76500000000000001},"min":{"1min":0.54400000000000004,"5min":0.47699999999999998,"15min":0.44700000000000001},"max":{"1min":1.054,"5min":1.232,"15min":1.232},"last":0.92900000000000005}]},{"osd":6,"last update":"Wed Feb 10 11:40:45 2021","interfaces":[{"interface":"back","average":{"1min":0.82099999999999995,"5min":0.85499999999999998,"15min":0.83899999999999997},"min":{"1min":0.53100000000000003,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.1499999999999999,"5min":1.329,"15min":1.329},"last":1.0609999999999999},{"interface":"front","average":{"1min":0.78600000000000003,"5min":0.81799999999999995,"15min":0.79800000000000004},"min":{"1min":0.49099999999999999,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.046,"5min":1.429,"15min":1.429},"last":1.0780000000000001}]}]},{"osd":6,"up_from":13,"seq":55834575310,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1070268,"kb_used_data":21684,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93301572,"statfs":{"total":96636764160,"available":95540809728,"internally_reserved":1073741824,"allocated":22204416,"data_stored":20740190,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 11:40:54 2021","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.78100000000000003,"15min":0.79700000000000004},"min":{"1min":0.42199999999999999,"5min":0.36599999999999999,"15min":0.35599999999999998},"max":{"1min":1.1879999999999999,"5min":1.254,"15min":1.4099999999999999},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.79200000000000004,"5min":0.76100000000000001,"15min":0.79700000000000004},"min":{"1min":0.48299999999999998,"5min":0.34999999999999998,"15min":0.34799999999999998},"max":{"1min":1.1539999999999999,"5min":1.2969999999999999,"15min":1.3140000000000001},"last":0.59999999999999998}]},{"osd":1,"last update":"Wed Feb 10 11:40:54 2021","interfaces":[{"interface":"back","average":{"1min":0.86899999999999999,"5min":0.82199999999999995,"15min":0.85099999999999998},"min":{"1min":0.58499999999999996,"5min":0.41299999999999998,"15min":0.37},"max":{"1min":1.2,"5min":1.224,"15min":1.427},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.82099999999999995,"5min":0.80700000000000005,"15min":0.82799999999999996},"min":{"1min":0.47499999999999998,"5min":0.371,"15min":0.371},"max":{"1min":1.1830000000000001,"5min":1.2230000000000001,"15min":1.2809999999999999},"last":0.76300000000000001}]},{"osd":2,"last update":"Wed Feb 10 11:40:54 2021","interfaces":[{"interface":"back","average":{"1min":0.77700000000000002,"5min":0.76600000000000001,"15min":0.78800000000000003},"min":{"1min":0.44400000000000001,"5min":0.434,"15min":0.39000000000000001},"max":{"1min":1.1559999999999999,"5min":1.2569999999999999,"15min":1.2569999999999999},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.82699999999999996,"5min":0.81699999999999995,"15min":0.85599999999999998},"min":{"1min":0.45900000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":1.1499999999999999,"5min":1.3029999999999999,"15min":1.431},"last":0.63800000000000001}]},{"osd":3,"last update":"Wed Feb 10 11:40:54 2021","interfaces":[{"interface":"back","average":{"1min":0.89000000000000001,"5min":0.91100000000000003,"15min":0.94199999999999995},"min":{"1min":0.433,"5min":0.433,"15min":0.42099999999999999},"max":{"1min":1.2589999999999999,"5min":1.4319999999999999,"15min":1.4319999999999999},"last":1.0029999999999999},{"interface":"front","average":{"1min":0.88400000000000001,"5min":0.91800000000000004,"15min":0.93799999999999994},"min":{"1min":0.51700000000000002,"5min":0.499,"15min":0.49099999999999999},"max":{"1min":1.2230000000000001,"5min":1.3440000000000001,"15min":1.3999999999999999},"last":1.0760000000000001}]},{"osd":4,"last update":"Wed Feb 10 11:40:54 2021","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.70699999999999996,"15min":0.71299999999999997},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.35699999999999998},"max":{"1min":0.96099999999999997,"5min":1.099,"15min":1.163},"last":0.82699999999999996},{"interface":"front","average":{"1min":0.73899999999999999,"5min":0.72999999999999998,"15min":0.73099999999999998},"min":{"1min":0.52700000000000002,"5min":0.373,"15min":0.29699999999999999},"max":{"1min":1.119,"5min":1.119,"15min":1.2769999999999999},"last":0.93799999999999994}]},{"osd":5,"last update":"Wed Feb 10 11:40:54 2021","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.36399999999999999},"max":{"1min":1.125,"5min":1.2649999999999999,"15min":1.2649999999999999},"last":1.012},{"interface":"front","average":{"1min":0.87,"5min":0.86499999999999999,"15min":0.876},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.53000000000000003},"max":{"1min":1.198,"5min":1.198,"15min":1.468},"last":1.0269999999999999}]},{"osd":7,"last update":"Wed Feb 10 11:40:54 2021","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":0.79300000000000004,"15min":0.80400000000000005},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":1.1859999999999999,"5min":1.1859999999999999,"15min":1.452},"last":1.056},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.76500000000000001,"15min":0.78000000000000003},"min":{"1min":0.48499999999999999,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.137,"5min":1.1499999999999999,"15min":1.3220000000000001},"last":0.84799999999999998}]}]},{"osd":5,"up_from":13,"seq":55834575310,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1083844,"kb_used_data":35260,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93287996,"statfs":{"total":96636764160,"available":95526907904,"internally_reserved":1073741824,"allocated":36106240,"data_stored":34649196,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 11:41:42 2021","interfaces":[{"interface":"back","average":{"1min":0.93600000000000005,"5min":0.85799999999999998,"15min":0.88200000000000001},"min":{"1min":0.49099999999999999,"5min":0.40000000000000002,"15min":0.34300000000000003},"max":{"1min":1.113,"5min":1.222,"15min":1.2829999999999999},"last":0.877},{"interface":"front","average":{"1min":0.995,"5min":0.90900000000000003,"15min":0.92100000000000004},"min":{"1min":0.73699999999999999,"5min":0.38400000000000001,"15min":0.33100000000000002},"max":{"1min":1.2589999999999999,"5min":1.27,"15min":1.444},"last":0.89400000000000002}]},{"osd":1,"last update":"Wed Feb 10 11:41:42 2021","interfaces":[{"interface":"back","average":{"1min":0.81899999999999995,"5min":0.75900000000000001,"15min":0.78100000000000003},"min":{"1min":0.44400000000000001,"5min":0.432,"15min":0.36499999999999999},"max":{"1min":1.081,"5min":1.2030000000000001,"15min":1.3169999999999999},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.82699999999999996,"5min":0.77200000000000002,"15min":0.80200000000000005},"min":{"1min":0.42899999999999999,"5min":0.41499999999999998,"15min":0.40400000000000003},"max":{"1min":1.0049999999999999,"5min":1.0980000000000001,"15min":1.2849999999999999},"last":0.63}]},{"osd":2,"last update":"Wed Feb 10 11:41:42 2021","interfaces":[{"interface":"back","average":{"1min":0.89000000000000001,"5min":0.84899999999999998,"15min":0.89400000000000002},"min":{"1min":0.51200000000000001,"5min":0.47299999999999998,"15min":0.38400000000000001},"max":{"1min":1.202,"5min":1.258,"15min":1.534},"last":0.76800000000000002},{"interface":"front","average":{"1min":0.95099999999999996,"5min":0.91300000000000003,"15min":0.92500000000000004},"min":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.42199999999999999},"max":{"1min":1.232,"5min":1.3100000000000001,"15min":1.357},"last":0.52600000000000002}]},{"osd":3,"last update":"Wed Feb 10 11:41:42 2021","interfaces":[{"interface":"back","average":{"1min":0.93999999999999995,"5min":0.88800000000000001,"15min":0.93000000000000005},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1.2729999999999999,"5min":1.367,"15min":1.6399999999999999},"last":0.95399999999999996},{"interface":"front","average":{"1min":1,"5min":0.93000000000000005,"15min":0.95099999999999996},"min":{"1min":0.54600000000000004,"5min":0.51300000000000001,"15min":0.48399999999999999},"max":{"1min":1.252,"5min":1.252,"15min":1.3640000000000001},"last":0.91600000000000004}]},{"osd":4,"last update":"Wed Feb 10 11:41:42 2021","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.68000000000000005,"15min":0.69399999999999995},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.95099999999999996,"5min":1.002,"15min":1.127},"last":0.82499999999999996},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.69499999999999995,"15min":0.72499999999999998},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.97199999999999998,"5min":1.0620000000000001,"15min":1.3009999999999999},"last":0.59899999999999998}]},{"osd":6,"last update":"Wed Feb 10 11:41:42 2021","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.71299999999999997,"15min":0.71399999999999997},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.86399999999999999,"5min":1.125,"15min":1.139},"last":0.86399999999999999},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.75900000000000001,"15min":0.76800000000000002},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.48099999999999998},"max":{"1min":1.036,"5min":1.2,"15min":1.2170000000000001},"last":0.75800000000000001}]},{"osd":7,"last update":"Wed Feb 10 11:41:42 2021","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.78800000000000003,"15min":0.80300000000000005},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.0029999999999999,"5min":1.262,"15min":1.347},"last":0.84499999999999997},{"interface":"front","average":{"1min":0.77700000000000002,"5min":0.81399999999999995,"15min":0.81899999999999995},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1.0369999999999999,"5min":1.117,"15min":1.2669999999999999},"last":0.81299999999999994}]}]},{"osd":4,"up_from":13,"seq":55834575310,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1074360,"kb_used_data":25776,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93297480,"statfs":{"total":96636764160,"available":95536619520,"internally_reserved":1073741824,"allocated":26394624,"data_stored":24934460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 11:40:47 2021","interfaces":[{"interface":"back","average":{"1min":0.84099999999999997,"5min":0.80600000000000005,"15min":0.80200000000000005},"min":{"1min":0.47499999999999998,"5min":0.40200000000000002,"15min":0.32200000000000001},"max":{"1min":1.1850000000000001,"5min":1.266,"15min":1.3979999999999999},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.85499999999999998,"5min":0.81000000000000005,"15min":0.80900000000000005},"min":{"1min":0.48099999999999998,"5min":0.379,"15min":0.379},"max":{"1min":1.3320000000000001,"5min":1.3320000000000001,"15min":1.3320000000000001},"last":0.47899999999999998}]},{"osd":1,"last update":"Wed Feb 10 11:40:47 2021","interfaces":[{"interface":"back","average":{"1min":0.83699999999999997,"5min":0.81899999999999995,"15min":0.82599999999999996},"min":{"1min":0.53900000000000003,"5min":0.46899999999999997,"15min":0.307},"max":{"1min":1.1339999999999999,"5min":1.417,"15min":1.417},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.84299999999999997,"5min":0.79700000000000004,"15min":0.81399999999999995},"min":{"1min":0.46400000000000002,"5min":0.434,"15min":0.38300000000000001},"max":{"1min":1.2929999999999999,"5min":1.387,"15min":1.387},"last":0.64200000000000002}]},{"osd":2,"last update":"Wed Feb 10 11:40:47 2021","interfaces":[{"interface":"back","average":{"1min":0.93700000000000006,"5min":0.91400000000000003,"15min":0.92700000000000005},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1.2689999999999999,"5min":1.363,"15min":1.4239999999999999},"last":0.57999999999999996},{"interface":"front","average":{"1min":1.01,"5min":0.99099999999999999,"15min":0.98299999999999998},"min":{"1min":0.67500000000000004,"5min":0.57699999999999996,"15min":0.499},"max":{"1min":1.3979999999999999,"5min":1.571,"15min":1.571},"last":0.82799999999999996}]},{"osd":3,"last update":"Wed Feb 10 11:40:47 2021","interfaces":[{"interface":"back","average":{"1min":0.84199999999999997,"5min":0.81399999999999995,"15min":0.82499999999999996},"min":{"1min":0.52500000000000002,"5min":0.40999999999999998,"15min":0.40100000000000002},"max":{"1min":1.1379999999999999,"5min":1.4550000000000001,"15min":1.4550000000000001},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.86799999999999999,"5min":0.85099999999999998,"15min":0.86899999999999999},"min":{"1min":0.498,"5min":0.47299999999999998,"15min":0.39000000000000001},"max":{"1min":1.1200000000000001,"5min":1.2190000000000001,"15min":1.325},"last":0.46100000000000002}]},{"osd":5,"last update":"Wed Feb 10 11:40:47 2021","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.74199999999999999,"15min":0.71399999999999997},"min":{"1min":0.46300000000000002,"5min":0.437,"15min":0.39500000000000002},"max":{"1min":1.042,"5min":1.0669999999999999,"15min":1.109},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.71699999999999997,"15min":0.70999999999999996},"min":{"1min":0.497,"5min":0.44900000000000001,"15min":0.36599999999999999},"max":{"1min":0.92800000000000005,"5min":1.272,"15min":1.272},"last":0.73199999999999998}]},{"osd":6,"last update":"Wed Feb 10 11:40:47 2021","interfaces":[{"interface":"back","average":{"1min":0.77600000000000002,"5min":0.745,"15min":0.746},"min":{"1min":0.58599999999999997,"5min":0.503,"15min":0.39400000000000002},"max":{"1min":1.018,"5min":1.1919999999999999,"15min":1.254},"last":0.74399999999999999},{"interface":"front","average":{"1min":0.79300000000000004,"5min":0.82299999999999995,"15min":0.80100000000000005},"min":{"1min":0.46600000000000003,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.105,"5min":1.2350000000000001,"15min":1.262},"last":0.76000000000000001}]},{"osd":7,"last update":"Wed Feb 10 11:40:47 2021","interfaces":[{"interface":"back","average":{"1min":0.81000000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"min":{"1min":0.60099999999999998,"5min":0.55400000000000005,"15min":0.48099999999999998},"max":{"1min":0.98399999999999999,"5min":1.3240000000000001,"15min":1.3240000000000001},"last":0.79900000000000004},{"interface":"front","average":{"1min":0.874,"5min":0.86599999999999999,"15min":0.84499999999999997},"min":{"1min":0.59699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.2330000000000001,"5min":1.2330000000000001,"15min":1.3129999999999999},"last":0.68100000000000005}]}]},{"osd":0,"up_from":13,"seq":55834575309,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1061312,"kb_used_data":12728,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93310528,"statfs":{"total":96636764160,"available":95549980672,"internally_reserved":1073741824,"allocated":13033472,"data_stored":11580398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Feb 10 11:40:55 2021","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.503,"15min":0.49199999999999999},"min":{"1min":0.32900000000000001,"5min":0.32400000000000001,"15min":0.30299999999999999},"max":{"1min":0.72499999999999998,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.501,"5min":0.499,"15min":0.49199999999999999},"min":{"1min":0.36499999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.79000000000000004,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.41299999999999998}]},{"osd":2,"last update":"Wed Feb 10 11:40:55 2021","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.54700000000000004,"15min":0.54200000000000004},"min":{"1min":0.379,"5min":0.318,"15min":0.318},"max":{"1min":0.83799999999999997,"5min":0.94799999999999995,"15min":1.0249999999999999},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54300000000000004,"15min":0.55100000000000005},"min":{"1min":0.41999999999999998,"5min":0.40600000000000003,"15min":0.35599999999999998},"max":{"1min":0.747,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.59899999999999998}]},{"osd":3,"last update":"Wed Feb 10 11:40:55 2021","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.52000000000000002,"15min":0.51200000000000001},"min":{"1min":0.371,"5min":0.34799999999999998,"15min":0.33900000000000002},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.84499999999999997},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61099999999999999},"min":{"1min":0.499,"5min":0.45100000000000001,"15min":0.36299999999999999},"max":{"1min":0.83299999999999996,"5min":0.878,"15min":0.97599999999999998},"last":0.52300000000000002}]},{"osd":4,"last update":"Wed Feb 10 11:40:55 2021","interfaces":[{"interface":"back","average":{"1min":0.871,"5min":1.0029999999999999,"15min":0.98199999999999998},"min":{"1min":0.499,"5min":0.48799999999999999,"15min":0.41199999999999998},"max":{"1min":1.1859999999999999,"5min":1.5189999999999999,"15min":1.53},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.88800000000000001,"5min":0.99299999999999999,"15min":0.98399999999999999},"min":{"1min":0.56100000000000005,"5min":0.498,"15min":0.498},"max":{"1min":1.224,"5min":1.4390000000000001,"15min":1.4710000000000001},"last":0.68600000000000005}]},{"osd":5,"last update":"Wed Feb 10 11:40:55 2021","interfaces":[{"interface":"back","average":{"1min":0.93400000000000005,"5min":1.0209999999999999,"15min":1.008},"min":{"1min":0.626,"5min":0.59599999999999997,"15min":0.48099999999999998},"max":{"1min":1.353,"5min":1.353,"15min":1.484},"last":0.747},{"interface":"front","average":{"1min":0.91900000000000004,"5min":1.0209999999999999,"15min":1.002},"min":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.51700000000000002},"max":{"1min":1.3049999999999999,"5min":1.4059999999999999,"15min":1.4359999999999999},"last":0.76300000000000001}]},{"osd":6,"last update":"Wed Feb 10 11:40:55 2021","interfaces":[{"interface":"back","average":{"1min":0.95999999999999996,"5min":1.0409999999999999,"15min":1.016},"min":{"1min":0.66500000000000004,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.3169999999999999,"5min":1.417,"15min":1.417},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.95399999999999996,"5min":1.0449999999999999,"15min":1.024},"min":{"1min":0.629,"5min":0.629,"15min":0.55100000000000005},"max":{"1min":1.2350000000000001,"5min":1.4870000000000001,"15min":1.4870000000000001},"last":0.83199999999999996}]},{"osd":7,"last update":"Wed Feb 10 11:40:55 2021","interfaces":[{"interface":"back","average":{"1min":0.97899999999999998,"5min":1.0589999999999999,"15min":1.038},"min":{"1min":0.71099999999999997,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":1.286,"5min":1.726,"15min":1.726},"last":0.91400000000000003},{"interface":"front","average":{"1min":0.96599999999999997,"5min":1.026,"15min":1.0069999999999999},"min":{"1min":0.66600000000000004,"5min":0.62,"15min":0.60499999999999998},"max":{"1min":1.2010000000000001,"5min":1.6080000000000001,"15min":1.6080000000000001},"last":0.81599999999999995}]}]},{"osd":1,"up_from":13,"seq":55834575310,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1092044,"kb_used_data":43456,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93279796,"statfs":{"total":96636764160,"available":95518511104,"internally_reserved":1073741824,"allocated":44498944,"data_stored":43038035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 11:40:58 2021","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.60199999999999998,"15min":0.60299999999999998},"min":{"1min":0.33700000000000002,"5min":0.32600000000000001,"15min":0.32200000000000001},"max":{"1min":1.0169999999999999,"5min":1.056,"15min":1.119},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.60399999999999998,"15min":0.62},"min":{"1min":0.41299999999999998,"5min":0.40500000000000003,"15min":0.316},"max":{"1min":0.88500000000000001,"5min":1.038,"15min":1.038},"last":0.53700000000000003}]},{"osd":2,"last update":"Wed Feb 10 11:40:58 2021","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.48599999999999999,"15min":0.48799999999999999},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.30499999999999999},"max":{"1min":0.64000000000000001,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.55500000000000005,"15min":0.55400000000000005},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.77100000000000002,"5min":1.1399999999999999,"15min":1.1399999999999999},"last":0.49099999999999999}]},{"osd":3,"last update":"Wed Feb 10 11:40:58 2021","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.60799999999999998,"15min":0.60899999999999999},"min":{"1min":0.38400000000000001,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.94199999999999995,"5min":1.244,"15min":1.244},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.35099999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.71699999999999997,"5min":1.155,"15min":1.155},"last":0.45400000000000001}]},{"osd":4,"last update":"Wed Feb 10 11:40:58 2021","interfaces":[{"interface":"back","average":{"1min":0.89100000000000001,"5min":0.95399999999999996,"15min":0.96699999999999997},"min":{"1min":0.47799999999999998,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.3220000000000001,"5min":1.5229999999999999,"15min":1.5229999999999999},"last":1.218},{"interface":"front","average":{"1min":0.92800000000000005,"5min":0.99099999999999999,"15min":0.997},"min":{"1min":0.47599999999999998,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.2709999999999999,"5min":1.508,"15min":1.508},"last":1.028}]},{"osd":5,"last update":"Wed Feb 10 11:40:58 2021","interfaces":[{"interface":"back","average":{"1min":0.91600000000000004,"5min":0.997,"15min":0.98899999999999999},"min":{"1min":0.502,"5min":0.496,"15min":0.496},"max":{"1min":1.2430000000000001,"5min":1.5329999999999999,"15min":1.6279999999999999},"last":1.0840000000000001},{"interface":"front","average":{"1min":0.93300000000000005,"5min":0.98399999999999999,"15min":0.998},"min":{"1min":0.57099999999999995,"5min":0.56999999999999995,"15min":0.46899999999999997},"max":{"1min":1.276,"5min":1.413,"15min":1.5760000000000001},"last":1.155}]},{"osd":6,"last update":"Wed Feb 10 11:40:58 2021","interfaces":[{"interface":"back","average":{"1min":0.93200000000000005,"5min":1.0149999999999999,"15min":1.0049999999999999},"min":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"max":{"1min":1.2410000000000001,"5min":1.3740000000000001,"15min":1.4059999999999999},"last":1.137},{"interface":"front","average":{"1min":0.94799999999999995,"5min":1.0289999999999999,"15min":1.0269999999999999},"min":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":1.2989999999999999,"5min":1.429,"15min":1.456},"last":1.129}]},{"osd":7,"last update":"Wed Feb 10 11:40:58 2021","interfaces":[{"interface":"back","average":{"1min":0.96199999999999997,"5min":1.036,"15min":1.038},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":1.298,"5min":1.472,"15min":1.786},"last":1.0009999999999999},{"interface":"front","average":{"1min":0.93700000000000006,"5min":0.99199999999999999,"15min":1.0109999999999999},"min":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.56599999999999995},"max":{"1min":1.284,"5min":1.415,"15min":1.494},"last":1.0569999999999999}]}]},{"osd":2,"up_from":13,"seq":55834575309,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1122768,"kb_used_data":74184,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93249072,"statfs":{"total":96636764160,"available":95487049728,"internally_reserved":1073741824,"allocated":75964416,"data_stored":74495787,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 11:40:41 2021","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.63100000000000001,"15min":0.626},"min":{"1min":0.35399999999999998,"5min":0.32500000000000001,"15min":0.29299999999999998},"max":{"1min":1.038,"5min":1.038,"15min":1.103},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.63100000000000001,"15min":0.624},"min":{"1min":0.45700000000000002,"5min":0.34599999999999997,"15min":0.28699999999999998},"max":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.69499999999999995}]},{"osd":1,"last update":"Wed Feb 10 11:40:41 2021","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.53700000000000003,"15min":0.52900000000000003},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.30599999999999999},"max":{"1min":0.70399999999999996,"5min":0.78700000000000003,"15min":1.0509999999999999},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.53400000000000003,"15min":0.51600000000000001},"min":{"1min":0.42999999999999999,"5min":0.39000000000000001,"15min":0.33400000000000002},"max":{"1min":0.73999999999999999,"5min":0.82599999999999996,"15min":0.88800000000000001},"last":0.60899999999999999}]},{"osd":3,"last update":"Wed Feb 10 11:40:41 2021","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.51700000000000002,"15min":0.51800000000000002},"min":{"1min":0.40899999999999997,"5min":0.36399999999999999,"15min":0.35399999999999998},"max":{"1min":0.66800000000000004,"5min":0.71599999999999997,"15min":0.78100000000000003},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.59699999999999998,"15min":0.60199999999999998},"min":{"1min":0.40100000000000002,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.76900000000000002,"5min":0.88600000000000001,"15min":1.0289999999999999},"last":0.46800000000000003}]},{"osd":4,"last update":"Wed Feb 10 11:40:41 2021","interfaces":[{"interface":"back","average":{"1min":1.0840000000000001,"5min":1.117,"15min":1.0840000000000001},"min":{"1min":0.81399999999999995,"5min":0.57299999999999995,"15min":0.52300000000000002},"max":{"1min":1.409,"5min":1.4430000000000001,"15min":1.4690000000000001},"last":1.248},{"interface":"front","average":{"1min":1.1120000000000001,"5min":1.1319999999999999,"15min":1.0980000000000001},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.373},"max":{"1min":1.3799999999999999,"5min":1.419,"15min":1.77},"last":1.1910000000000001}]},{"osd":5,"last update":"Wed Feb 10 11:40:41 2021","interfaces":[{"interface":"back","average":{"1min":1.004,"5min":1.0109999999999999,"15min":0.98899999999999999},"min":{"1min":0.75600000000000001,"5min":0.58899999999999997,"15min":0.48999999999999999},"max":{"1min":1.47,"5min":1.47,"15min":1.47},"last":1.296},{"interface":"front","average":{"1min":0.92100000000000004,"5min":1.0029999999999999,"15min":0.98999999999999999},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":1.2529999999999999,"5min":1.4419999999999999,"15min":1.613},"last":1.2210000000000001}]},{"osd":6,"last update":"Wed Feb 10 11:40:41 2021","interfaces":[{"interface":"back","average":{"1min":0.97699999999999998,"5min":1.012,"15min":1},"min":{"1min":0.629,"5min":0.61699999999999999,"15min":0.53300000000000003},"max":{"1min":1.4550000000000001,"5min":1.569,"15min":1.569},"last":1.0509999999999999},{"interface":"front","average":{"1min":0.95099999999999996,"5min":1.0289999999999999,"15min":0.998},"min":{"1min":0.73499999999999999,"5min":0.60899999999999999,"15min":0.55000000000000004},"max":{"1min":1.2450000000000001,"5min":1.3999999999999999,"15min":1.504},"last":1.1379999999999999}]},{"osd":7,"last update":"Wed Feb 10 11:40:41 2021","interfaces":[{"interface":"back","average":{"1min":0.97699999999999998,"5min":1.0169999999999999,"15min":1.008},"min":{"1min":0.66100000000000003,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":1.395,"5min":1.4810000000000001,"15min":1.4810000000000001},"last":1.0629999999999999},{"interface":"front","average":{"1min":0.95299999999999996,"5min":1.048,"15min":1.026},"min":{"1min":0.64200000000000002,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":1.379,"5min":1.4750000000000001,"15min":1.4950000000000001},"last":1.1619999999999999}]}]},{"osd":3,"up_from":13,"seq":55834575309,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1079744,"kb_used_data":31160,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93292096,"statfs":{"total":96636764160,"available":95531106304,"internally_reserved":1073741824,"allocated":31907840,"data_stored":30454869,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 11:40:51 2021","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.47899999999999998,"15min":0.48699999999999999},"min":{"1min":0.32700000000000001,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.63100000000000001,"5min":0.96099999999999997,"15min":1.1479999999999999},"last":0.497},{"interface":"front","average":{"1min":0.46600000000000003,"5min":0.45500000000000002,"15min":0.46999999999999997},"min":{"1min":0.38700000000000001,"5min":0.34200000000000003,"15min":0.30199999999999999},"max":{"1min":0.64500000000000002,"5min":0.66800000000000004,"15min":0.79600000000000004},"last":0.505}]},{"osd":1,"last update":"Wed Feb 10 11:40:51 2021","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.48699999999999999,"15min":0.49099999999999999},"min":{"1min":0.36699999999999999,"5min":0.32100000000000001,"15min":0.311},"max":{"1min":0.70299999999999996,"5min":0.83399999999999996,"15min":1.2709999999999999},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.56000000000000005,"15min":0.56499999999999995},"min":{"1min":0.375,"5min":0.373,"15min":0.316},"max":{"1min":0.77200000000000002,"5min":0.86399999999999999,"15min":0.997},"last":0.56699999999999995}]},{"osd":2,"last update":"Wed Feb 10 11:40:51 2021","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.56100000000000005,"15min":0.55900000000000005},"min":{"1min":0.38400000000000001,"5min":0.374,"15min":0.33000000000000002},"max":{"1min":0.995,"5min":0.995,"15min":0.995},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.626,"15min":0.63300000000000001},"min":{"1min":0.48699999999999999,"5min":0.44600000000000001,"15min":0.34899999999999998},"max":{"1min":0.83999999999999997,"5min":0.91800000000000004,"15min":1.04},"last":0.63}]},{"osd":4,"last update":"Wed Feb 10 11:40:51 2021","interfaces":[{"interface":"back","average":{"1min":0.996,"5min":0.98199999999999998,"15min":0.95699999999999996},"min":{"1min":0.63,"5min":0.52200000000000002,"15min":0.39100000000000001},"max":{"1min":1.3779999999999999,"5min":1.417,"15min":1.5760000000000001},"last":1.099},{"interface":"front","average":{"1min":0.97799999999999998,"5min":1.0089999999999999,"15min":0.98899999999999999},"min":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52000000000000002},"max":{"1min":1.327,"5min":1.494,"15min":1.6659999999999999},"last":1.131}]},{"osd":5,"last update":"Wed Feb 10 11:40:51 2021","interfaces":[{"interface":"back","average":{"1min":0.99099999999999999,"5min":1,"15min":0.995},"min":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.55800000000000005},"max":{"1min":1.274,"5min":1.448,"15min":1.556},"last":0.96999999999999997},{"interface":"front","average":{"1min":0.997,"5min":1.002,"15min":0.97399999999999998},"min":{"1min":0.60199999999999998,"5min":0.51100000000000001,"15min":0.47999999999999998},"max":{"1min":1.3959999999999999,"5min":1.3959999999999999,"15min":1.405},"last":1.1200000000000001}]},{"osd":6,"last update":"Wed Feb 10 11:40:51 2021","interfaces":[{"interface":"back","average":{"1min":1.0329999999999999,"5min":1.014,"15min":1.002},"min":{"1min":0.69099999999999995,"5min":0.54100000000000004,"15min":0.51100000000000001},"max":{"1min":1.3500000000000001,"5min":1.5409999999999999,"15min":1.5409999999999999},"last":1.0329999999999999},{"interface":"front","average":{"1min":1.0189999999999999,"5min":1.022,"15min":0.998},"min":{"1min":0.64400000000000002,"5min":0.58599999999999997,"15min":0.57599999999999996},"max":{"1min":1.458,"5min":1.458,"15min":1.458},"last":1.077}]},{"osd":7,"last update":"Wed Feb 10 11:40:51 2021","interfaces":[{"interface":"back","average":{"1min":1.0249999999999999,"5min":1.0169999999999999,"15min":0.99299999999999999},"min":{"1min":0.74199999999999999,"5min":0.55700000000000005,"15min":0.50600000000000001},"max":{"1min":1.264,"5min":1.3819999999999999,"15min":1.389},"last":1.0640000000000001},{"interface":"front","average":{"1min":1.0429999999999999,"5min":1,"15min":0.997},"min":{"1min":0.77900000000000003,"5min":0.67000000000000004,"15min":0.56999999999999995},"max":{"1min":1.488,"5min":1.488,"15min":1.4970000000000001},"last":1.1479999999999999}]}]}]}} 2021-02-10T11:41:47.326 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:41:47.792 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:47.773 7eff81efe700 1 Processor -- start 2021-02-10T11:41:47.792 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:47.773 7eff81efe700 1 -- start start 2021-02-10T11:41:47.792 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:47.774 7eff81efe700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7eff7c0d1870 0x7eff7c1b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:47.792 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:47.774 7eff81efe700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7eff7c1b0a10 0x7eff7c0cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:47.792 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:47.774 7eff81efe700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff7c0f0b10 con 0x7eff7c0d1c70 2021-02-10T11:41:47.792 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:47.774 7eff81efe700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff7c0fe230 con 0x7eff7c1b0a10 2021-02-10T11:41:47.793 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:47.774 7eff81efe700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff7c0e5ed0 con 0x7eff7c0d1870 2021-02-10T11:41:47.793 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:47.774 7eff7affd700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7eff7c1b0a10 0x7eff7c0cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:47.793 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:47.774 7eff7affd700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7eff7c1b0a10 0x7eff7c0cf330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:41700/0 (socket says 172.21.2.102:41700) 2021-02-10T11:41:47.793 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:47.774 7eff7affd700 1 -- 172.21.2.102:0/2234161829 learned_addr learned my addr 172.21.2.102:0/2234161829 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:41:47.793 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:47.774 7eff7affd700 1 -- 172.21.2.102:0/2234161829 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7eff7c0d1870 msgr2=0x7eff7c1b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:41:47.793 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:47.774 7eff7affd700 1 --2- 172.21.2.102:0/2234161829 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7eff7c0d1870 0x7eff7c1b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:47.793 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:47.774 7eff7affd700 1 -- 172.21.2.102:0/2234161829 >> v1:172.21.2.113:6789/0 conn(0x7eff7c0d1c70 legacy=0x7eff7c0cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:41:47.793 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:47.774 7eff7affd700 1 -- 172.21.2.102:0/2234161829 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff7c0f0650 con 0x7eff7c1b0a10 2021-02-10T11:41:47.794 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:47.774 7eff7affd700 1 --2- 172.21.2.102:0/2234161829 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7eff7c1b0a10 0x7eff7c0cf330 secure :-1 s=READY pgs=3448 cs=0 l=1 rev1=1 rx=0x7eff64006700 tx=0x7eff64009b80).ready entity=mon.0 client_cookie=6b91215ddc0875c1 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:47.794 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:47.774 7eff7a7fc700 1 -- 172.21.2.102:0/2234161829 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7eff6400c0c0 con 0x7eff7c1b0a10 2021-02-10T11:41:47.794 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:47.774 7eff7a7fc700 1 -- 172.21.2.102:0/2234161829 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff6400c0c0 con 0x7eff7c1b0a10 2021-02-10T11:41:47.794 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:47.774 7eff7a7fc700 1 -- 172.21.2.102:0/2234161829 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7eff6400c440 con 0x7eff7c1b0a10 2021-02-10T11:41:47.794 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:47.774 7eff81efe700 1 -- 172.21.2.102:0/2234161829 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7eff7c1b0a10 msgr2=0x7eff7c0cf330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:47.794 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:47.774 7eff81efe700 1 --2- 172.21.2.102:0/2234161829 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7eff7c1b0a10 0x7eff7c0cf330 secure :-1 s=READY pgs=3448 cs=0 l=1 rev1=1 rx=0x7eff64006700 tx=0x7eff64009b80).stop 2021-02-10T11:41:47.794 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:47.775 7eff81efe700 1 -- 172.21.2.102:0/2234161829 shutdown_connections 2021-02-10T11:41:47.794 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:47.775 7eff81efe700 1 --2- 172.21.2.102:0/2234161829 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7eff7c1b0a10 0x7eff7c0cf330 unknown :-1 s=CLOSED pgs=3448 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:47.795 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:47.775 7eff81efe700 1 --2- 172.21.2.102:0/2234161829 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7eff7c0d1870 0x7eff7c1b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:47.795 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:47.775 7eff81efe700 1 -- 172.21.2.102:0/2234161829 >> 172.21.2.102:0/2234161829 conn(0x7eff7c0ce450 msgr2=0x7eff7c0d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:47.795 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:47.775 7eff81efe700 1 -- 172.21.2.102:0/2234161829 shutdown_connections 2021-02-10T11:41:47.795 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:47.775 7eff81efe700 1 -- 172.21.2.102:0/2234161829 wait complete. 2021-02-10T11:41:47.795 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:47.789 7eff81efe700 1 Processor -- start 2021-02-10T11:41:47.795 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:47.789 7eff81efe700 1 -- start start 2021-02-10T11:41:47.797 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:47.790 7eff81efe700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7eff7c0b1180 0x7eff7c0b1580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:47.798 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:47.790 7eff81efe700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7eff7c0b1ba0 0x7eff7c1a98a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:47.798 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:47.790 7eff81efe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7eff7c1a9ec0 0x7eff7c1aac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:47.798 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:47.790 7eff81efe700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff7c0b1fa0 con 0x7eff7c1a9ec0 2021-02-10T11:41:47.798 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:47.790 7eff81efe700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff7c1af830 con 0x7eff7c0b1180 2021-02-10T11:41:47.798 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:47.790 7eff81efe700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff7c1af3c0 con 0x7eff7c0b1ba0 2021-02-10T11:41:47.798 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:47.790 7eff7bfff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7eff7c1a9ec0 0x7eff7c1aac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:47.798 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:47.790 7eff7bfff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7eff7c1a9ec0 0x7eff7c1aac10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:41704/0 (socket says 172.21.2.102:41704) 2021-02-10T11:41:47.799 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:47.790 7eff7bfff700 1 -- 172.21.2.102:0/747651747 learned_addr learned my addr 172.21.2.102:0/747651747 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:41:47.799 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:47.790 7eff7bfff700 1 -- 172.21.2.102:0/747651747 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7eff7c0b1ba0 msgr2=0x7eff7c1a98a0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:41:47.799 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:47.790 7eff7bfff700 1 --2- 172.21.2.102:0/747651747 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7eff7c0b1ba0 0x7eff7c1a98a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:47.800 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:47.790 7eff7bfff700 1 -- 172.21.2.102:0/747651747 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7eff7c0b1180 msgr2=0x7eff7c0b1580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:41:47.800 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:47.790 7eff7bfff700 1 --2- 172.21.2.102:0/747651747 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7eff7c0b1180 0x7eff7c0b1580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:47.800 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:47.790 7eff7bfff700 1 -- 172.21.2.102:0/747651747 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff7c1aa2c0 con 0x7eff7c1a9ec0 2021-02-10T11:41:47.800 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:47.790 7eff7bfff700 1 --2- 172.21.2.102:0/747651747 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7eff7c1a9ec0 0x7eff7c1aac10 secure :-1 s=READY pgs=3449 cs=0 l=1 rev1=1 rx=0x7eff74009940 tx=0x7eff7400bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:47.800 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:47.791 7eff73fff700 1 -- 172.21.2.102:0/747651747 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7eff74000f90 con 0x7eff7c1a9ec0 2021-02-10T11:41:47.800 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:47.791 7eff73fff700 1 -- 172.21.2.102:0/747651747 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff7400c2f0 con 0x7eff7c1a9ec0 2021-02-10T11:41:47.801 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:47.791 7eff73fff700 1 -- 172.21.2.102:0/747651747 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7eff7400c750 con 0x7eff7c1a9ec0 2021-02-10T11:41:47.801 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:47.791 7eff81efe700 1 -- 172.21.2.102:0/747651747 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff7c1aa2c0 con 0x7eff7c1a9ec0 2021-02-10T11:41:47.801 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:47.791 7eff81efe700 1 -- 172.21.2.102:0/747651747 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff7c1ab520 con 0x7eff7c1a9ec0 2021-02-10T11:41:47.801 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:47.791 7eff73fff700 1 -- 172.21.2.102:0/747651747 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7eff74001220 con 0x7eff7c1a9ec0 2021-02-10T11:41:47.801 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:47.791 7eff73fff700 1 --2- 172.21.2.102:0/747651747 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7eff600186c0 0x7eff6001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:47.801 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:47.791 7eff73fff700 1 -- 172.21.2.102:0/747651747 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 5573+0+0 (secure 0 0 0) 0x7eff740329a0 con 0x7eff7c1a9ec0 2021-02-10T11:41:47.801 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:47.791 7eff7b7fe700 1 --2- 172.21.2.102:0/747651747 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7eff600186c0 0x7eff6001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:47.801 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:47.792 7eff7b7fe700 1 --2- 172.21.2.102:0/747651747 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7eff600186c0 0x7eff6001abc0 crc :-1 s=READY pgs=1501 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:47.826 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:47.824 7eff81efe700 1 -- 172.21.2.102:0/747651747 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff7c0ce450 con 0x7eff7c1a9ec0 2021-02-10T11:41:47.829 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:47.826 7eff73fff700 1 -- 172.21.2.102:0/747651747 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7eff74000f40 con 0x7eff7c1a9ec0 2021-02-10T11:41:47.952 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:47.950 7eff81efe700 1 -- 172.21.2.102:0/747651747 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7eff7c0ce450 con 0x7eff7c1a9ec0 2021-02-10T11:41:47.952 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:47.950 7eff73fff700 1 -- 172.21.2.102:0/747651747 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v197) v1 ==== 74+0+14007 (secure 0 0 0) 0x7eff7401f860 con 0x7eff7c1a9ec0 2021-02-10T11:41:47.953 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:41:47.953 INFO:teuthology.orchestra.run.gibba002.stdout:{"epoch":197,"fsid":"68db30ea-dbce-4aef-a6df-f01c6fea0513","created":"2021-02-10 11:03:06.331557","modified":"2021-02-10 11:40:25.367346","last_up_change":"2021-02-10 11:03:32.690562","last_in_change":"2021-02-10 11:03:32.690562","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":43,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:03:37.322519","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":42,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:40:22.330567","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"196","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":43,"pool_name":"cephfs_data","create_time":"2021-02-10 11:40:23.357509","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"197","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"be691b4d-a039-408a-af80-9d53f02b6f1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":185,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6818","nonce":36088},{"type":"v1","addr":"172.21.2.102:6820","nonce":36088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6822","nonce":36088},{"type":"v1","addr":"172.21.2.102:6824","nonce":36088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6830","nonce":36088},{"type":"v1","addr":"172.21.2.102:6832","nonce":36088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6826","nonce":36088},{"type":"v1","addr":"172.21.2.102:6828","nonce":36088}]},"public_addr":"172.21.2.102:6820/36088","cluster_addr":"172.21.2.102:6824/36088","heartbeat_back_addr":"172.21.2.102:6832/36088","heartbeat_front_addr":"172.21.2.102:6828/36088","state":["exists","up"]},{"osd":1,"uuid":"64e3ccc8-9c69-4d7c-865e-af8e83822635","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6819","nonce":36087},{"type":"v1","addr":"172.21.2.102:6821","nonce":36087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6823","nonce":36087},{"type":"v1","addr":"172.21.2.102:6825","nonce":36087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6831","nonce":36087},{"type":"v1","addr":"172.21.2.102:6833","nonce":36087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6827","nonce":36087},{"type":"v1","addr":"172.21.2.102:6829","nonce":36087}]},"public_addr":"172.21.2.102:6821/36087","cluster_addr":"172.21.2.102:6825/36087","heartbeat_back_addr":"172.21.2.102:6833/36087","heartbeat_front_addr":"172.21.2.102:6829/36087","state":["exists","up"]},{"osd":2,"uuid":"153700af-747f-481f-bdf4-d6072d3a842f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6810","nonce":36089},{"type":"v1","addr":"172.21.2.102:6811","nonce":36089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6812","nonce":36089},{"type":"v1","addr":"172.21.2.102:6813","nonce":36089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6816","nonce":36089},{"type":"v1","addr":"172.21.2.102:6817","nonce":36089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6814","nonce":36089},{"type":"v1","addr":"172.21.2.102:6815","nonce":36089}]},"public_addr":"172.21.2.102:6811/36089","cluster_addr":"172.21.2.102:6813/36089","heartbeat_back_addr":"172.21.2.102:6817/36089","heartbeat_front_addr":"172.21.2.102:6815/36089","state":["exists","up"]},{"osd":3,"uuid":"cbc678f4-83a9-45e6-8830-2f7c232b1ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6802","nonce":36090},{"type":"v1","addr":"172.21.2.102:6803","nonce":36090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6804","nonce":36090},{"type":"v1","addr":"172.21.2.102:6805","nonce":36090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6808","nonce":36090},{"type":"v1","addr":"172.21.2.102:6809","nonce":36090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.102:6806","nonce":36090},{"type":"v1","addr":"172.21.2.102:6807","nonce":36090}]},"public_addr":"172.21.2.102:6803/36090","cluster_addr":"172.21.2.102:6805/36090","heartbeat_back_addr":"172.21.2.102:6809/36090","heartbeat_front_addr":"172.21.2.102:6807/36090","state":["exists","up"]},{"osd":4,"uuid":"c01e4008-b258-4f10-afd2-9e7f37214351","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6808","nonce":36368},{"type":"v1","addr":"172.21.2.113:6811","nonce":36368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6814","nonce":36368},{"type":"v1","addr":"172.21.2.113:6818","nonce":36368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6826","nonce":36368},{"type":"v1","addr":"172.21.2.113:6828","nonce":36368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6820","nonce":36368},{"type":"v1","addr":"172.21.2.113:6824","nonce":36368}]},"public_addr":"172.21.2.113:6811/36368","cluster_addr":"172.21.2.113:6818/36368","heartbeat_back_addr":"172.21.2.113:6828/36368","heartbeat_front_addr":"172.21.2.113:6824/36368","state":["exists","up"]},{"osd":5,"uuid":"551f6007-daf8-4a98-8ee0-c2b16bcccb5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6802","nonce":36366},{"type":"v1","addr":"172.21.2.113:6805","nonce":36366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6807","nonce":36366},{"type":"v1","addr":"172.21.2.113:6810","nonce":36366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6819","nonce":36366},{"type":"v1","addr":"172.21.2.113:6822","nonce":36366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6812","nonce":36366},{"type":"v1","addr":"172.21.2.113:6815","nonce":36366}]},"public_addr":"172.21.2.113:6805/36366","cluster_addr":"172.21.2.113:6810/36366","heartbeat_back_addr":"172.21.2.113:6822/36366","heartbeat_front_addr":"172.21.2.113:6815/36366","state":["exists","up"]},{"osd":6,"uuid":"bf67c040-0e50-48be-b102-c3943bc9ebef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6800","nonce":36367},{"type":"v1","addr":"172.21.2.113:6801","nonce":36367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6803","nonce":36367},{"type":"v1","addr":"172.21.2.113:6804","nonce":36367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6813","nonce":36367},{"type":"v1","addr":"172.21.2.113:6816","nonce":36367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6806","nonce":36367},{"type":"v1","addr":"172.21.2.113:6809","nonce":36367}]},"public_addr":"172.21.2.113:6801/36367","cluster_addr":"172.21.2.113:6804/36367","heartbeat_back_addr":"172.21.2.113:6816/36367","heartbeat_front_addr":"172.21.2.113:6809/36367","state":["exists","up"]},{"osd":7,"uuid":"f5461f80-018d-428f-be89-2eb2bcbe7bb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6817","nonce":36365},{"type":"v1","addr":"172.21.2.113:6821","nonce":36365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6823","nonce":36365},{"type":"v1","addr":"172.21.2.113:6825","nonce":36365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6830","nonce":36365},{"type":"v1","addr":"172.21.2.113:6831","nonce":36365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.113:6827","nonce":36365},{"type":"v1","addr":"172.21.2.113:6829","nonce":36365}]},"public_addr":"172.21.2.113:6821/36365","cluster_addr":"172.21.2.113:6825/36365","heartbeat_back_addr":"172.21.2.113:6831/36365","heartbeat_front_addr":"172.21.2.113:6829/36365","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:41:47.954 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:47.952 7eff81efe700 1 -- 172.21.2.102:0/747651747 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7eff600186c0 msgr2=0x7eff6001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:47.954 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:47.952 7eff81efe700 1 --2- 172.21.2.102:0/747651747 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7eff600186c0 0x7eff6001abc0 crc :-1 s=READY pgs=1501 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:47.954 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:47.952 7eff81efe700 1 -- 172.21.2.102:0/747651747 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7eff7c1a9ec0 msgr2=0x7eff7c1aac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:47.955 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:47.952 7eff81efe700 1 --2- 172.21.2.102:0/747651747 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7eff7c1a9ec0 0x7eff7c1aac10 secure :-1 s=READY pgs=3449 cs=0 l=1 rev1=1 rx=0x7eff74009940 tx=0x7eff7400bcf0).stop 2021-02-10T11:41:47.955 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:47.952 7eff81efe700 1 -- 172.21.2.102:0/747651747 shutdown_connections 2021-02-10T11:41:47.955 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:47.952 7eff81efe700 1 --2- 172.21.2.102:0/747651747 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7eff7c1a9ec0 0x7eff7c1aac10 unknown :-1 s=CLOSED pgs=3449 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:47.956 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:47.952 7eff81efe700 1 --2- 172.21.2.102:0/747651747 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7eff7c0b1ba0 0x7eff7c1a98a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:47.956 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:47.952 7eff81efe700 1 --2- 172.21.2.102:0/747651747 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7eff600186c0 0x7eff6001abc0 unknown :-1 s=CLOSED pgs=1501 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:47.956 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:47.952 7eff81efe700 1 --2- 172.21.2.102:0/747651747 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7eff7c0b1180 0x7eff7c0b1580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:47.956 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:47.952 7eff81efe700 1 -- 172.21.2.102:0/747651747 >> 172.21.2.102:0/747651747 conn(0x7eff7c19e6c0 msgr2=0x7eff7c1b3610 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:47.956 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:47.953 7eff81efe700 1 -- 172.21.2.102:0/747651747 shutdown_connections 2021-02-10T11:41:47.956 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:47.953 7eff81efe700 1 -- 172.21.2.102:0/747651747 wait complete. 2021-02-10T11:41:48.963 INFO:tasks.ceph:Scrubbing osd.0 2021-02-10T11:41:48.964 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2021-02-10T11:41:49.396 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:49.378 7f53ba4b9700 1 Processor -- start 2021-02-10T11:41:49.397 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:49.378 7f53ba4b9700 1 -- start start 2021-02-10T11:41:49.397 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:49.378 7f53ba4b9700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f53b40d1880 0x7f53b41b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:49.397 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:49.378 7f53ba4b9700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f53b41b0af0 0x7f53b4108950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:49.397 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:49.378 7f53ba4b9700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53b40fe1d0 con 0x7f53b40d1c80 2021-02-10T11:41:49.397 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:49.378 7f53ba4b9700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53b4109a70 con 0x7f53b40d1880 2021-02-10T11:41:49.397 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:49.378 7f53ba4b9700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53b4109760 con 0x7f53b41b0af0 2021-02-10T11:41:49.397 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:49.378 7f53b3fff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f53b40d1880 0x7f53b41b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:49.398 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:49.378 7f53b3fff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f53b40d1880 0x7f53b41b04d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:41712/0 (socket says 172.21.2.102:41712) 2021-02-10T11:41:49.398 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:49.378 7f53b3fff700 1 -- 172.21.2.102:0/1792540762 learned_addr learned my addr 172.21.2.102:0/1792540762 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:41:49.398 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:49.378 7f53b8ad6700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f53b40d1c80 0x7f53b4108f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:33936/0 (socket says 172.21.2.102:33936) 2021-02-10T11:41:49.398 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:49.379 7f53b37fe700 1 --2- 172.21.2.102:0/1792540762 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f53b41b0af0 0x7f53b4108950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:49.398 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:49.379 7f53b3fff700 1 -- 172.21.2.102:0/1792540762 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f53b41b0af0 msgr2=0x7f53b4108950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:41:49.398 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:49.379 7f53b3fff700 1 --2- 172.21.2.102:0/1792540762 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f53b41b0af0 0x7f53b4108950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:49.399 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:49.379 7f53b3fff700 1 -- 172.21.2.102:0/1792540762 >> v1:172.21.2.102:6789/0 conn(0x7f53b40d1c80 legacy=0x7f53b4108f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:49.399 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:49.379 7f53b2ffd700 1 -- 172.21.2.102:0/1792540762 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1437683960 0 0) 0x7f53ac001480 con 0x7f53b40d1c80 2021-02-10T11:41:49.399 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:49.379 7f53b3fff700 1 -- 172.21.2.102:0/1792540762 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53b4088070 con 0x7f53b40d1880 2021-02-10T11:41:49.399 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:49.379 7f53b3fff700 1 --2- 172.21.2.102:0/1792540762 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f53b40d1880 0x7f53b41b04d0 secure :-1 s=READY pgs=3451 cs=0 l=1 rev1=1 rx=0x7f53a4006700 tx=0x7f53a4009b00).ready entity=mon.0 client_cookie=328f55c5f65457c server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:49.399 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:49.379 7f53b2ffd700 1 -- 172.21.2.102:0/1792540762 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f53a400c0c0 con 0x7f53b40d1880 2021-02-10T11:41:49.399 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:49.379 7f53b2ffd700 1 -- 172.21.2.102:0/1792540762 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53a400c340 con 0x7f53b40d1880 2021-02-10T11:41:49.399 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:49.379 7f53b2ffd700 1 -- 172.21.2.102:0/1792540762 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f53a400c820 con 0x7f53b40d1880 2021-02-10T11:41:49.399 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:49.379 7f53ba4b9700 1 -- 172.21.2.102:0/1792540762 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f53b40d1880 msgr2=0x7f53b41b04d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:49.399 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:49.379 7f53ba4b9700 1 --2- 172.21.2.102:0/1792540762 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f53b40d1880 0x7f53b41b04d0 secure :-1 s=READY pgs=3451 cs=0 l=1 rev1=1 rx=0x7f53a4006700 tx=0x7f53a4009b00).stop 2021-02-10T11:41:49.400 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:49.380 7f53ba4b9700 1 -- 172.21.2.102:0/1792540762 shutdown_connections 2021-02-10T11:41:49.400 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:49.380 7f53ba4b9700 1 --2- 172.21.2.102:0/1792540762 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f53b41b0af0 0x7f53b4108950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:49.400 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:49.380 7f53ba4b9700 1 --2- 172.21.2.102:0/1792540762 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f53b40d1880 0x7f53b41b04d0 unknown :-1 s=CLOSED pgs=3451 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:49.400 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:49.380 7f53ba4b9700 1 -- 172.21.2.102:0/1792540762 >> 172.21.2.102:0/1792540762 conn(0x7f53b40ce430 msgr2=0x7f53b40d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:49.400 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:49.380 7f53ba4b9700 1 -- 172.21.2.102:0/1792540762 shutdown_connections 2021-02-10T11:41:49.401 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:49.380 7f53ba4b9700 1 -- 172.21.2.102:0/1792540762 wait complete. 2021-02-10T11:41:49.403 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:49.394 7f53ba4b9700 1 Processor -- start 2021-02-10T11:41:49.403 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:49.395 7f53ba4b9700 1 -- start start 2021-02-10T11:41:49.403 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:49.395 7f53ba4b9700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f53b40aef00 0x7f53b4099c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:49.403 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:49.395 7f53ba4b9700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f53b409a260 0x7f53b409a660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:49.403 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:49.395 7f53ba4b9700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f53b409ac80 0x7f53b419a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:49.404 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:49.395 7f53ba4b9700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53b419bdd0 con 0x7f53b40aef00 2021-02-10T11:41:49.404 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:49.395 7f53ba4b9700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53b419de70 con 0x7f53b409a260 2021-02-10T11:41:49.404 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:49.395 7f53ba4b9700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53b419a2d0 con 0x7f53b409ac80 2021-02-10T11:41:49.405 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:49.395 7f53b3fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f53b40aef00 0x7f53b4099c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:49.405 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:49.395 7f53b3fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f53b40aef00 0x7f53b4099c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:41720/0 (socket says 172.21.2.102:41720) 2021-02-10T11:41:49.405 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:49.395 7f53b3fff700 1 -- 172.21.2.102:0/1518446677 learned_addr learned my addr 172.21.2.102:0/1518446677 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:41:49.405 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:49.395 7f53b3fff700 1 -- 172.21.2.102:0/1518446677 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f53b409ac80 msgr2=0x7f53b419a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:49.406 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:49.395 7f53b3fff700 1 --2- 172.21.2.102:0/1518446677 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f53b409ac80 0x7f53b419a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:49.406 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:49.395 7f53b3fff700 1 -- 172.21.2.102:0/1518446677 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f53b409a260 msgr2=0x7f53b409a660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:49.406 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:49.395 7f53b3fff700 1 --2- 172.21.2.102:0/1518446677 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f53b409a260 0x7f53b409a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:49.406 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:49.396 7f53b3fff700 1 -- 172.21.2.102:0/1518446677 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53b4199f50 con 0x7f53b40aef00 2021-02-10T11:41:49.407 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:49.396 7f53b3fff700 1 --2- 172.21.2.102:0/1518446677 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f53b40aef00 0x7f53b4099c40 secure :-1 s=READY pgs=3452 cs=0 l=1 rev1=1 rx=0x7f53a4006f20 tx=0x7f53a400b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:49.407 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:49.396 7f53b0ff9700 1 -- 172.21.2.102:0/1518446677 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f53a400bd80 con 0x7f53b40aef00 2021-02-10T11:41:49.407 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:49.396 7f53b0ff9700 1 -- 172.21.2.102:0/1518446677 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53a4000c90 con 0x7f53b40aef00 2021-02-10T11:41:49.407 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:49.396 7f53b0ff9700 1 -- 172.21.2.102:0/1518446677 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f53a40022b0 con 0x7f53b40aef00 2021-02-10T11:41:49.408 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:49.396 7f53ba4b9700 1 -- 172.21.2.102:0/1518446677 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53b4199f50 con 0x7f53b40aef00 2021-02-10T11:41:49.408 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:49.396 7f53ba4b9700 1 -- 172.21.2.102:0/1518446677 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53b419b160 con 0x7f53b40aef00 2021-02-10T11:41:49.408 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:49.397 7f53b0ff9700 1 -- 172.21.2.102:0/1518446677 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f53a400a000 con 0x7f53b40aef00 2021-02-10T11:41:49.408 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:49.397 7f53b0ff9700 1 --2- 172.21.2.102:0/1518446677 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f53940186c0 0x7f539401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:49.409 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:49.397 7f53b0ff9700 1 -- 172.21.2.102:0/1518446677 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f53a4032760 con 0x7f53b40aef00 2021-02-10T11:41:49.409 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:49.397 7f53b37fe700 1 --2- 172.21.2.102:0/1518446677 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f53940186c0 0x7f539401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:49.409 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:49.397 7f53b37fe700 1 --2- 172.21.2.102:0/1518446677 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f53940186c0 0x7f539401abc0 crc :-1 s=READY pgs=1502 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:49.430 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:49.428 7f53ba4b9700 1 --2- 172.21.2.102:0/1518446677 >> [v2:172.21.2.102:6818/36088,v1:172.21.2.102:6820/36088] conn(0x7f53b4087120 0x7f53b40627c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:49.431 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:49.428 7f53ba4b9700 1 -- 172.21.2.102:0/1518446677 --> [v2:172.21.2.102:6818/36088,v1:172.21.2.102:6820/36088] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f53b419b160 con 0x7f53b4087120 2021-02-10T11:41:49.431 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:49.428 7f53b8ad6700 1 --2- 172.21.2.102:0/1518446677 >> [v2:172.21.2.102:6818/36088,v1:172.21.2.102:6820/36088] conn(0x7f53b4087120 0x7f53b40627c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:49.431 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:49.428 7f53b8ad6700 1 --2- 172.21.2.102:0/1518446677 >> [v2:172.21.2.102:6818/36088,v1:172.21.2.102:6820/36088] conn(0x7f53b4087120 0x7f53b40627c0 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:49.432 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:49.429 7f53b0ff9700 1 -- 172.21.2.102:0/1518446677 <== osd.0 v2:172.21.2.102:6818/36088 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+5151 (crc 0 0 0) 0x7f53ac007ae0 con 0x7f53b4087120 2021-02-10T11:41:49.437 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:49.434 7f53ba4b9700 1 -- 172.21.2.102:0/1518446677 --> [v2:172.21.2.102:6818/36088,v1:172.21.2.102:6820/36088] -- command(tid 2: {"prefix": "config set", "key": "osd_debug_deep_scrub_sleep", "value": "0"}) v1 -- 0x7f53b4082a70 con 0x7f53b4087120 2021-02-10T11:41:49.437 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:49.435 7f53b0ff9700 1 -- 172.21.2.102:0/1518446677 <== osd.0 v2:172.21.2.102:6818/36088 2 ==== command_reply(tid 2: 0 Set osd_debug_deep_scrub_sleep to 0) v1 ==== 43+0+0 (crc 0 0 0) 0x7f53ac007d00 con 0x7f53b4087120 2021-02-10T11:41:49.439 INFO:teuthology.orchestra.run.gibba002.stderr:Set osd_debug_deep_scrub_sleep to 0 2021-02-10T11:41:49.439 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:49.435 7f53ba4b9700 1 -- 172.21.2.102:0/1518446677 >> [v2:172.21.2.102:6818/36088,v1:172.21.2.102:6820/36088] conn(0x7f53b4087120 msgr2=0x7f53b40627c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:49.439 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:49.435 7f53ba4b9700 1 --2- 172.21.2.102:0/1518446677 >> [v2:172.21.2.102:6818/36088,v1:172.21.2.102:6820/36088] conn(0x7f53b4087120 0x7f53b40627c0 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:49.439 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:49.436 7f53ba4b9700 1 -- 172.21.2.102:0/1518446677 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f53940186c0 msgr2=0x7f539401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:49.439 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:49.436 7f53ba4b9700 1 --2- 172.21.2.102:0/1518446677 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f53940186c0 0x7f539401abc0 crc :-1 s=READY pgs=1502 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:49.440 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:49.436 7f53ba4b9700 1 -- 172.21.2.102:0/1518446677 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f53b40aef00 msgr2=0x7f53b4099c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:49.440 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:49.436 7f53ba4b9700 1 --2- 172.21.2.102:0/1518446677 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f53b40aef00 0x7f53b4099c40 secure :-1 s=READY pgs=3452 cs=0 l=1 rev1=1 rx=0x7f53a4006f20 tx=0x7f53a400b990).stop 2021-02-10T11:41:49.440 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:49.436 7f53b8ad6700 1 -- 172.21.2.102:0/1518446677 reap_dead start 2021-02-10T11:41:49.441 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:49.436 7f53ba4b9700 1 -- 172.21.2.102:0/1518446677 shutdown_connections 2021-02-10T11:41:49.441 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:49.436 7f53ba4b9700 1 -- 172.21.2.102:0/1518446677 >> 172.21.2.102:0/1518446677 conn(0x7f53b418e810 msgr2=0x7f53b418c290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:49.441 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:49.436 7f53ba4b9700 1 -- 172.21.2.102:0/1518446677 shutdown_connections 2021-02-10T11:41:49.442 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:49.436 7f53ba4b9700 1 -- 172.21.2.102:0/1518446677 wait complete. 2021-02-10T11:41:49.447 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd deep-scrub 0 2021-02-10T11:41:49.877 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:49.857 7f5d5ab10700 1 Processor -- start 2021-02-10T11:41:49.878 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:49.857 7f5d5ab10700 1 -- start start 2021-02-10T11:41:49.878 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:49.858 7f5d5ab10700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f5d540d1870 0x7f5d541b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:49.878 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:49.858 7f5d5ab10700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f5d540f0650 0x7f5d540cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:49.878 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:49.858 7f5d5ab10700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d540f0b10 con 0x7f5d541b0a10 2021-02-10T11:41:49.878 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:49.858 7f5d5ab10700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d540e5ed0 con 0x7f5d540d1870 2021-02-10T11:41:49.878 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:49.858 7f5d5ab10700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d540d1c70 con 0x7f5d540f0650 2021-02-10T11:41:49.879 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:49.858 7f5d5892c700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f5d540d1870 0x7f5d541b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:49.879 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:49.858 7f5d5892c700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f5d540d1870 0x7f5d541b03f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:41726/0 (socket says 172.21.2.102:41726) 2021-02-10T11:41:49.879 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:49.858 7f5d5892c700 1 -- 172.21.2.102:0/2399217690 learned_addr learned my addr 172.21.2.102:0/2399217690 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:41:49.879 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:49.858 7f5d5912d700 1 --2- 172.21.2.102:0/2399217690 >> v2:172.21.2.113:3300/0 conn(0x7f5d540f0650 0x7f5d540cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:49.879 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:49.859 7f5d5892c700 1 -- 172.21.2.102:0/2399217690 >> v2:172.21.2.113:3300/0 conn(0x7f5d540f0650 msgr2=0x7f5d540cfb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:41:49.879 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:49.859 7f5d5892c700 1 --2- 172.21.2.102:0/2399217690 >> v2:172.21.2.113:3300/0 conn(0x7f5d540f0650 0x7f5d540cfb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:49.879 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:49.859 7f5d5892c700 1 -- 172.21.2.102:0/2399217690 >> v1:172.21.2.113:6789/0 conn(0x7f5d541b0a10 legacy=0x7f5d540cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:49.879 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:49.859 7f5d5892c700 1 -- 172.21.2.102:0/2399217690 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d540d1e90 con 0x7f5d540d1870 2021-02-10T11:41:49.880 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:49.859 7f5d5912d700 1 --2- 172.21.2.102:0/2399217690 >> v2:172.21.2.113:3300/0 conn(0x7f5d540f0650 0x7f5d540cfb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:41:49.880 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:49.859 7f5d5892c700 1 --2- 172.21.2.102:0/2399217690 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5d540d1870 0x7f5d541b03f0 secure :-1 s=READY pgs=3453 cs=0 l=1 rev1=1 rx=0x7f5d3c006700 tx=0x7f5d3c009b80).ready entity=mon.0 client_cookie=27437901606d28dc server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:49.880 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:49.859 7f5d537fe700 1 -- 172.21.2.102:0/2399217690 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5d3c00c0c0 con 0x7f5d540d1870 2021-02-10T11:41:49.880 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:49.859 7f5d537fe700 1 -- 172.21.2.102:0/2399217690 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d3c00c3c0 con 0x7f5d540d1870 2021-02-10T11:41:49.880 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:49.859 7f5d537fe700 1 -- 172.21.2.102:0/2399217690 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5d3c00c8e0 con 0x7f5d540d1870 2021-02-10T11:41:49.880 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:49.860 7f5d5ab10700 1 -- 172.21.2.102:0/2399217690 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5d540d1870 msgr2=0x7f5d541b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:49.880 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:49.860 7f5d5ab10700 1 --2- 172.21.2.102:0/2399217690 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5d540d1870 0x7f5d541b03f0 secure :-1 s=READY pgs=3453 cs=0 l=1 rev1=1 rx=0x7f5d3c006700 tx=0x7f5d3c009b80).stop 2021-02-10T11:41:49.880 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:49.860 7f5d5ab10700 1 -- 172.21.2.102:0/2399217690 shutdown_connections 2021-02-10T11:41:49.881 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:49.860 7f5d5ab10700 1 --2- 172.21.2.102:0/2399217690 >> v2:172.21.2.113:3300/0 conn(0x7f5d540f0650 0x7f5d540cfb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:49.881 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:49.860 7f5d5ab10700 1 --2- 172.21.2.102:0/2399217690 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5d540d1870 0x7f5d541b03f0 unknown :-1 s=CLOSED pgs=3453 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:49.881 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:49.860 7f5d5ab10700 1 -- 172.21.2.102:0/2399217690 >> 172.21.2.102:0/2399217690 conn(0x7f5d540ce450 msgr2=0x7f5d540d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:49.881 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:49.860 7f5d5ab10700 1 -- 172.21.2.102:0/2399217690 shutdown_connections 2021-02-10T11:41:49.881 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:49.860 7f5d5ab10700 1 -- 172.21.2.102:0/2399217690 wait complete. 2021-02-10T11:41:49.881 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:49.875 7f5d5ab10700 1 Processor -- start 2021-02-10T11:41:49.881 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:49.875 7f5d5ab10700 1 -- start start 2021-02-10T11:41:49.883 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:49.875 7f5d5ab10700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5d54189d00 0x7f5d5418a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:49.883 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:49.875 7f5d5ab10700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5d5418a720 0x7f5d5418ab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:49.883 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:49.875 7f5d5ab10700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5d54187e50 0x7f5d54188250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:49.883 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:49.875 7f5d5ab10700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d540c37e0 con 0x7f5d54189d00 2021-02-10T11:41:49.883 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:49.875 7f5d5ab10700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d540c39d0 con 0x7f5d5418a720 2021-02-10T11:41:49.884 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:49.875 7f5d5ab10700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d54188870 con 0x7f5d54187e50 2021-02-10T11:41:49.884 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:49.876 7f5d5892c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5d54189d00 0x7f5d5418a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:49.884 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:49.876 7f5d5892c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5d54189d00 0x7f5d5418a100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:41732/0 (socket says 172.21.2.102:41732) 2021-02-10T11:41:49.884 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:49.876 7f5d5892c700 1 -- 172.21.2.102:0/2485202745 learned_addr learned my addr 172.21.2.102:0/2485202745 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:41:49.884 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:49.876 7f5d5912d700 1 --2- 172.21.2.102:0/2485202745 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5d54187e50 0x7f5d54188250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:49.885 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:49.876 7f5d53fff700 1 --2- 172.21.2.102:0/2485202745 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5d5418a720 0x7f5d5418ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:49.885 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:49.876 7f5d5892c700 1 -- 172.21.2.102:0/2485202745 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5d54187e50 msgr2=0x7f5d54188250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:49.885 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:49.876 7f5d5892c700 1 --2- 172.21.2.102:0/2485202745 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5d54187e50 0x7f5d54188250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:49.885 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:49.876 7f5d5892c700 1 -- 172.21.2.102:0/2485202745 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5d5418a720 msgr2=0x7f5d5418ab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:49.885 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:49.876 7f5d5892c700 1 --2- 172.21.2.102:0/2485202745 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5d5418a720 0x7f5d5418ab20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:49.885 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:49.876 7f5d5892c700 1 -- 172.21.2.102:0/2485202745 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d54188a90 con 0x7f5d54189d00 2021-02-10T11:41:49.885 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:49.876 7f5d53fff700 1 --2- 172.21.2.102:0/2485202745 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5d5418a720 0x7f5d5418ab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:41:49.885 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:49.876 7f5d5892c700 1 --2- 172.21.2.102:0/2485202745 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5d54189d00 0x7f5d5418a100 secure :-1 s=READY pgs=3454 cs=0 l=1 rev1=1 rx=0x7f5d3c009720 tx=0x7f5d3c00b8e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:49.886 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:49.876 7f5d517fa700 1 -- 172.21.2.102:0/2485202745 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5d3c00bcd0 con 0x7f5d54189d00 2021-02-10T11:41:49.886 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:49.876 7f5d517fa700 1 -- 172.21.2.102:0/2485202745 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d3c000940 con 0x7f5d54189d00 2021-02-10T11:41:49.886 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:49.876 7f5d5ab10700 1 -- 172.21.2.102:0/2485202745 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d54188a90 con 0x7f5d54189d00 2021-02-10T11:41:49.886 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:49.876 7f5d517fa700 1 -- 172.21.2.102:0/2485202745 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5d3c002270 con 0x7f5d54189d00 2021-02-10T11:41:49.886 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:49.876 7f5d5ab10700 1 -- 172.21.2.102:0/2485202745 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d54188a90 con 0x7f5d54189d00 2021-02-10T11:41:49.886 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:49.877 7f5d517fa700 1 -- 172.21.2.102:0/2485202745 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f5d3c000c90 con 0x7f5d54189d00 2021-02-10T11:41:49.886 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:49.877 7f5d517fa700 1 --2- 172.21.2.102:0/2485202745 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5d380186d0 0x7f5d3801abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:49.886 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:49.877 7f5d517fa700 1 -- 172.21.2.102:0/2485202745 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f5d3c032670 con 0x7f5d54189d00 2021-02-10T11:41:49.887 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:49.877 7f5d53fff700 1 --2- 172.21.2.102:0/2485202745 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5d380186d0 0x7f5d3801abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:49.887 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:49.877 7f5d53fff700 1 --2- 172.21.2.102:0/2485202745 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5d380186d0 0x7f5d3801abd0 crc :-1 s=READY pgs=1503 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:49.910 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:49.908 7f5d5ab10700 1 -- 172.21.2.102:0/2485202745 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d540ebc50 con 0x7f5d54189d00 2021-02-10T11:41:49.912 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:49.910 7f5d517fa700 1 -- 172.21.2.102:0/2485202745 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f5d3c0052e0 con 0x7f5d54189d00 2021-02-10T11:41:50.043 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:50.041 7f5d5ab10700 1 -- 172.21.2.102:0/2485202745 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mgr", ""]}) v1 -- 0x7f5d540f0b10 con 0x7f5d380186d0 2021-02-10T11:41:50.044 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:50.041 7f5d517fa700 1 -- 172.21.2.102:0/2485202745 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) v1 ==== 41+0+0 (crc 0 0 0) 0x7f5d44005830 con 0x7f5d380186d0 2021-02-10T11:41:50.044 INFO:teuthology.orchestra.run.gibba002.stderr:instructed osd(s) 0 to deep-scrub 2021-02-10T11:41:50.045 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:50.043 7f5d5ab10700 1 -- 172.21.2.102:0/2485202745 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5d380186d0 msgr2=0x7f5d3801abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:50.046 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:50.043 7f5d5ab10700 1 --2- 172.21.2.102:0/2485202745 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5d380186d0 0x7f5d3801abd0 crc :-1 s=READY pgs=1503 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:50.046 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:50.043 7f5d5ab10700 1 -- 172.21.2.102:0/2485202745 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5d54189d00 msgr2=0x7f5d5418a100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:50.046 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:50.043 7f5d5ab10700 1 --2- 172.21.2.102:0/2485202745 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5d54189d00 0x7f5d5418a100 secure :-1 s=READY pgs=3454 cs=0 l=1 rev1=1 rx=0x7f5d3c009720 tx=0x7f5d3c00b8e0).stop 2021-02-10T11:41:50.046 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:50.044 7f5d5ab10700 1 -- 172.21.2.102:0/2485202745 shutdown_connections 2021-02-10T11:41:50.047 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:50.044 7f5d5ab10700 1 --2- 172.21.2.102:0/2485202745 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5d54187e50 0x7f5d54188250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:50.047 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:50.044 7f5d5ab10700 1 --2- 172.21.2.102:0/2485202745 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f5d380186d0 0x7f5d3801abd0 unknown :-1 s=CLOSED pgs=1503 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:50.047 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:50.044 7f5d5ab10700 1 --2- 172.21.2.102:0/2485202745 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5d5418a720 0x7f5d5418ab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:50.047 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:50.044 7f5d5ab10700 1 --2- 172.21.2.102:0/2485202745 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5d54189d00 0x7f5d5418a100 unknown :-1 s=CLOSED pgs=3454 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:50.047 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:50.044 7f5d5ab10700 1 -- 172.21.2.102:0/2485202745 >> 172.21.2.102:0/2485202745 conn(0x7f5d540aed50 msgr2=0x7f5d5409c9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:50.047 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:50.044 7f5d5ab10700 1 -- 172.21.2.102:0/2485202745 shutdown_connections 2021-02-10T11:41:50.047 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:50.044 7f5d5ab10700 1 -- 172.21.2.102:0/2485202745 wait complete. 2021-02-10T11:41:50.055 INFO:tasks.ceph:Scrubbing osd.1 2021-02-10T11:41:50.055 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2021-02-10T11:41:50.489 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:50.469 7f8283a5a700 1 Processor -- start 2021-02-10T11:41:50.489 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:50.470 7f8283a5a700 1 -- start start 2021-02-10T11:41:50.489 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:50.470 7f8283a5a700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f827c0d1c80 0x7f827c0cef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:50.489 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:50.470 7f8283a5a700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f827c0cf590 0x7f827c0cf990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:50.489 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:50.470 7f8283a5a700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f827c0fe1d0 con 0x7f827c0d1880 2021-02-10T11:41:50.489 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:50.470 7f8283a5a700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f827c1b0cc0 con 0x7f827c0d1c80 2021-02-10T11:41:50.490 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:50.470 7f8283a5a700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f827c109a70 con 0x7f827c0cf590 2021-02-10T11:41:50.490 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:50.470 7f8281075700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f827c0d1c80 0x7f827c0cef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:50.490 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:50.470 7f8281075700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f827c0d1c80 0x7f827c0cef70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:41740/0 (socket says 172.21.2.102:41740) 2021-02-10T11:41:50.490 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:50.470 7f8281075700 1 -- 172.21.2.102:0/1322772673 learned_addr learned my addr 172.21.2.102:0/1322772673 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:41:50.490 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:50.471 7f8281075700 1 -- 172.21.2.102:0/1322772673 >> v2:172.21.2.113:3300/0 conn(0x7f827c0cf590 msgr2=0x7f827c0cf990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:41:50.490 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:50.471 7f8281075700 1 --2- 172.21.2.102:0/1322772673 >> v2:172.21.2.113:3300/0 conn(0x7f827c0cf590 0x7f827c0cf990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:50.490 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:50.471 7f8281075700 1 -- 172.21.2.102:0/1322772673 >> v1:172.21.2.113:6789/0 conn(0x7f827c0d1880 legacy=0x7f827c1b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:41:50.490 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:50.471 7f8281075700 1 -- 172.21.2.102:0/1322772673 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f827c088070 con 0x7f827c0d1c80 2021-02-10T11:41:50.491 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:50.471 7f8281075700 1 --2- 172.21.2.102:0/1322772673 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f827c0d1c80 0x7f827c0cef70 secure :-1 s=READY pgs=3455 cs=0 l=1 rev1=1 rx=0x7f826c006700 tx=0x7f826c009b80).ready entity=mon.0 client_cookie=cf3480fa4330dcf1 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:50.491 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:50.471 7f8280874700 1 -- 172.21.2.102:0/1322772673 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f826c00c0c0 con 0x7f827c0d1c80 2021-02-10T11:41:50.491 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:50.471 7f8280874700 1 -- 172.21.2.102:0/1322772673 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f826c00c340 con 0x7f827c0d1c80 2021-02-10T11:41:50.491 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:50.471 7f8280874700 1 -- 172.21.2.102:0/1322772673 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f826c00c860 con 0x7f827c0d1c80 2021-02-10T11:41:50.491 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:50.471 7f8283a5a700 1 -- 172.21.2.102:0/1322772673 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f827c0d1c80 msgr2=0x7f827c0cef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:50.491 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:50.471 7f8283a5a700 1 --2- 172.21.2.102:0/1322772673 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f827c0d1c80 0x7f827c0cef70 secure :-1 s=READY pgs=3455 cs=0 l=1 rev1=1 rx=0x7f826c006700 tx=0x7f826c009b80).stop 2021-02-10T11:41:50.491 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:50.472 7f8283a5a700 1 -- 172.21.2.102:0/1322772673 shutdown_connections 2021-02-10T11:41:50.491 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:50.472 7f8283a5a700 1 --2- 172.21.2.102:0/1322772673 >> v2:172.21.2.113:3300/0 conn(0x7f827c0cf590 0x7f827c0cf990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:50.492 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:50.472 7f8283a5a700 1 --2- 172.21.2.102:0/1322772673 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f827c0d1c80 0x7f827c0cef70 unknown :-1 s=CLOSED pgs=3455 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:50.492 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:50.472 7f8283a5a700 1 -- 172.21.2.102:0/1322772673 >> 172.21.2.102:0/1322772673 conn(0x7f827c0ce430 msgr2=0x7f827c0d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:50.492 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:50.472 7f8283a5a700 1 -- 172.21.2.102:0/1322772673 shutdown_connections 2021-02-10T11:41:50.492 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:50.472 7f8283a5a700 1 -- 172.21.2.102:0/1322772673 wait complete. 2021-02-10T11:41:50.495 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:50.487 7f8283a5a700 1 Processor -- start 2021-02-10T11:41:50.495 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:50.487 7f8283a5a700 1 -- start start 2021-02-10T11:41:50.495 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:50.487 7f8283a5a700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f827c09ef20 0x7f827c18d430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:50.495 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:50.487 7f8283a5a700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f827c08eae0 0x7f827c08eee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:50.495 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:50.487 7f8283a5a700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f827c08b0e0 0x7f827c08b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:50.495 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:50.487 7f8283a5a700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f827c08f500 con 0x7f827c09ef20 2021-02-10T11:41:50.495 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:50.487 7f8283a5a700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f827c09f320 con 0x7f827c08eae0 2021-02-10T11:41:50.496 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:50.487 7f8283a5a700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f827c09f540 con 0x7f827c08b0e0 2021-02-10T11:41:50.496 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:50.487 7f8281876700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f827c09ef20 0x7f827c18d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:50.496 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:50.487 7f8281876700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f827c09ef20 0x7f827c18d430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:41746/0 (socket says 172.21.2.102:41746) 2021-02-10T11:41:50.496 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:50.487 7f8281876700 1 -- 172.21.2.102:0/942467227 learned_addr learned my addr 172.21.2.102:0/942467227 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:41:50.496 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:50.488 7f8281876700 1 -- 172.21.2.102:0/942467227 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f827c08b0e0 msgr2=0x7f827c08b4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:41:50.496 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:50.488 7f8281876700 1 --2- 172.21.2.102:0/942467227 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f827c08b0e0 0x7f827c08b4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:50.496 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:50.488 7f8281876700 1 -- 172.21.2.102:0/942467227 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f827c08eae0 msgr2=0x7f827c08eee0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:41:50.496 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:50.488 7f8281876700 1 --2- 172.21.2.102:0/942467227 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f827c08eae0 0x7f827c08eee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:50.497 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:50.488 7f8281876700 1 -- 172.21.2.102:0/942467227 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f827c08bb00 con 0x7f827c09ef20 2021-02-10T11:41:50.497 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:50.488 7f8281876700 1 --2- 172.21.2.102:0/942467227 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f827c09ef20 0x7f827c18d430 secure :-1 s=READY pgs=3456 cs=0 l=1 rev1=1 rx=0x7f827800b5b0 tx=0x7f827800b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:50.497 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:50.488 7f82727fc700 1 -- 172.21.2.102:0/942467227 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f827800bd40 con 0x7f827c09ef20 2021-02-10T11:41:50.497 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:50.488 7f82727fc700 1 -- 172.21.2.102:0/942467227 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8278000e20 con 0x7f827c09ef20 2021-02-10T11:41:50.497 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:50.488 7f82727fc700 1 -- 172.21.2.102:0/942467227 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8278006210 con 0x7f827c09ef20 2021-02-10T11:41:50.497 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:50.489 7f8283a5a700 1 -- 172.21.2.102:0/942467227 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f827c08bb00 con 0x7f827c09ef20 2021-02-10T11:41:50.497 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:50.489 7f8283a5a700 1 -- 172.21.2.102:0/942467227 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f827c08bb00 con 0x7f827c09ef20 2021-02-10T11:41:50.497 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:50.489 7f82727fc700 1 -- 172.21.2.102:0/942467227 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f8278006020 con 0x7f827c09ef20 2021-02-10T11:41:50.498 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:50.489 7f82727fc700 1 --2- 172.21.2.102:0/942467227 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f82640186c0 0x7f826401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:50.498 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:50.489 7f82727fc700 1 -- 172.21.2.102:0/942467227 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f8278032580 con 0x7f827c09ef20 2021-02-10T11:41:50.498 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:50.489 7f8281075700 1 --2- 172.21.2.102:0/942467227 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f82640186c0 0x7f826401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:50.498 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:50.490 7f8281075700 1 --2- 172.21.2.102:0/942467227 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f82640186c0 0x7f826401abc0 crc :-1 s=READY pgs=1504 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:50.522 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:50.520 7f8283a5a700 1 --2- 172.21.2.102:0/942467227 >> [v2:172.21.2.102:6819/36087,v1:172.21.2.102:6821/36087] conn(0x7f827c0627c0 0x7f827c062bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:50.522 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:50.520 7f8283a5a700 1 -- 172.21.2.102:0/942467227 --> [v2:172.21.2.102:6819/36087,v1:172.21.2.102:6821/36087] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f827c082ad0 con 0x7f827c0627c0 2021-02-10T11:41:50.523 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:50.520 7f8282077700 1 --2- 172.21.2.102:0/942467227 >> [v2:172.21.2.102:6819/36087,v1:172.21.2.102:6821/36087] conn(0x7f827c0627c0 0x7f827c062bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:50.523 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:50.520 7f8282077700 1 --2- 172.21.2.102:0/942467227 >> [v2:172.21.2.102:6819/36087,v1:172.21.2.102:6821/36087] conn(0x7f827c0627c0 0x7f827c062bc0 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:50.523 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:50.521 7f82727fc700 1 -- 172.21.2.102:0/942467227 <== osd.1 v2:172.21.2.102:6819/36087 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+5151 (crc 0 0 0) 0x7f8274006d80 con 0x7f827c0627c0 2021-02-10T11:41:50.528 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:50.526 7f8283a5a700 1 -- 172.21.2.102:0/942467227 --> [v2:172.21.2.102:6819/36087,v1:172.21.2.102:6821/36087] -- command(tid 2: {"prefix": "config set", "key": "osd_debug_deep_scrub_sleep", "value": "0"}) v1 -- 0x7f827c08bb00 con 0x7f827c0627c0 2021-02-10T11:41:50.529 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:50.527 7f82727fc700 1 -- 172.21.2.102:0/942467227 <== osd.1 v2:172.21.2.102:6819/36087 2 ==== command_reply(tid 2: 0 Set osd_debug_deep_scrub_sleep to 0) v1 ==== 43+0+0 (crc 0 0 0) 0x7f8274001030 con 0x7f827c0627c0 2021-02-10T11:41:50.529 INFO:teuthology.orchestra.run.gibba002.stderr:Set osd_debug_deep_scrub_sleep to 0 2021-02-10T11:41:50.530 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:50.528 7f8283a5a700 1 -- 172.21.2.102:0/942467227 >> [v2:172.21.2.102:6819/36087,v1:172.21.2.102:6821/36087] conn(0x7f827c0627c0 msgr2=0x7f827c062bc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:50.530 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:50.528 7f8283a5a700 1 --2- 172.21.2.102:0/942467227 >> [v2:172.21.2.102:6819/36087,v1:172.21.2.102:6821/36087] conn(0x7f827c0627c0 0x7f827c062bc0 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:50.531 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:50.528 7f8283a5a700 1 -- 172.21.2.102:0/942467227 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f82640186c0 msgr2=0x7f826401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:50.531 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:50.528 7f8283a5a700 1 --2- 172.21.2.102:0/942467227 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f82640186c0 0x7f826401abc0 crc :-1 s=READY pgs=1504 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:50.531 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:50.528 7f8283a5a700 1 -- 172.21.2.102:0/942467227 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f827c09ef20 msgr2=0x7f827c18d430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:50.531 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:50.528 7f8283a5a700 1 --2- 172.21.2.102:0/942467227 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f827c09ef20 0x7f827c18d430 secure :-1 s=READY pgs=3456 cs=0 l=1 rev1=1 rx=0x7f827800b5b0 tx=0x7f827800b950).stop 2021-02-10T11:41:50.531 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:50.528 7f8282077700 1 -- 172.21.2.102:0/942467227 reap_dead start 2021-02-10T11:41:50.532 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:50.528 7f8283a5a700 1 -- 172.21.2.102:0/942467227 shutdown_connections 2021-02-10T11:41:50.532 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:50.528 7f8283a5a700 1 -- 172.21.2.102:0/942467227 >> 172.21.2.102:0/942467227 conn(0x7f827c19f970 msgr2=0x7f827c1a61d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:50.532 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:50.528 7f8283a5a700 1 -- 172.21.2.102:0/942467227 shutdown_connections 2021-02-10T11:41:50.532 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:50.528 7f8283a5a700 1 -- 172.21.2.102:0/942467227 wait complete. 2021-02-10T11:41:50.538 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd deep-scrub 1 2021-02-10T11:41:50.968 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:50.949 7fcd693cd700 1 Processor -- start 2021-02-10T11:41:50.968 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:50.949 7fcd693cd700 1 -- start start 2021-02-10T11:41:50.968 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:50.950 7fcd693cd700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7fcd640d1870 0x7fcd641b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:50.968 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:50.950 7fcd693cd700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fcd641b0a10 0x7fcd640cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:50.969 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:50.950 7fcd693cd700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcd640d1c70 0x7fcd640cf950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:50.969 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:50.950 7fcd693cd700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd640e5ed0 con 0x7fcd641b0a10 2021-02-10T11:41:50.969 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:50.950 7fcd693cd700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd640f0b10 con 0x7fcd640d1870 2021-02-10T11:41:50.969 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:50.950 7fcd693cd700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd640cff70 con 0x7fcd640d1c70 2021-02-10T11:41:50.969 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:50.950 7fcd6287c700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fcd641b0a10 0x7fcd640cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:50.969 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:50.950 7fcd6287c700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fcd641b0a10 0x7fcd640cf330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:41756/0 (socket says 172.21.2.102:41756) 2021-02-10T11:41:50.969 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:50.950 7fcd6287c700 1 -- 172.21.2.102:0/1768705907 learned_addr learned my addr 172.21.2.102:0/1768705907 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:41:50.969 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:50.950 7fcd6287c700 1 -- 172.21.2.102:0/1768705907 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcd640d1c70 msgr2=0x7fcd640cf950 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:41:50.970 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:50.950 7fcd6287c700 1 --2- 172.21.2.102:0/1768705907 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcd640d1c70 0x7fcd640cf950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:50.970 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:50.950 7fcd6287c700 1 -- 172.21.2.102:0/1768705907 >> v2:172.21.2.113:3300/0 conn(0x7fcd640d1870 msgr2=0x7fcd641b03f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:41:50.970 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:50.950 7fcd6287c700 1 --2- 172.21.2.102:0/1768705907 >> v2:172.21.2.113:3300/0 conn(0x7fcd640d1870 0x7fcd641b03f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:50.970 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:50.950 7fcd6287c700 1 -- 172.21.2.102:0/1768705907 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd640d0150 con 0x7fcd641b0a10 2021-02-10T11:41:50.970 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:50.950 7fcd6287c700 1 --2- 172.21.2.102:0/1768705907 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcd641b0a10 0x7fcd640cf330 secure :-1 s=READY pgs=3457 cs=0 l=1 rev1=1 rx=0x7fcd58006700 tx=0x7fcd58009b80).ready entity=mon.0 client_cookie=e4d1ccc30070b637 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:50.970 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:50.951 7fcd6207b700 1 -- 172.21.2.102:0/1768705907 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcd5800c0c0 con 0x7fcd641b0a10 2021-02-10T11:41:50.970 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:50.951 7fcd6207b700 1 -- 172.21.2.102:0/1768705907 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd5800c440 con 0x7fcd641b0a10 2021-02-10T11:41:50.970 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:50.951 7fcd6207b700 1 -- 172.21.2.102:0/1768705907 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcd5800c920 con 0x7fcd641b0a10 2021-02-10T11:41:50.971 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:50.951 7fcd693cd700 1 -- 172.21.2.102:0/1768705907 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcd641b0a10 msgr2=0x7fcd640cf330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:50.971 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:50.951 7fcd693cd700 1 --2- 172.21.2.102:0/1768705907 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcd641b0a10 0x7fcd640cf330 secure :-1 s=READY pgs=3457 cs=0 l=1 rev1=1 rx=0x7fcd58006700 tx=0x7fcd58009b80).stop 2021-02-10T11:41:50.971 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:50.951 7fcd693cd700 1 -- 172.21.2.102:0/1768705907 shutdown_connections 2021-02-10T11:41:50.971 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:50.951 7fcd693cd700 1 --2- 172.21.2.102:0/1768705907 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcd640d1c70 0x7fcd640cf950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:50.971 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:50.951 7fcd693cd700 1 --2- 172.21.2.102:0/1768705907 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcd641b0a10 0x7fcd640cf330 unknown :-1 s=CLOSED pgs=3457 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:50.971 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:50.951 7fcd693cd700 1 --2- 172.21.2.102:0/1768705907 >> v2:172.21.2.113:3300/0 conn(0x7fcd640d1870 0x7fcd641b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:50.971 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:50.951 7fcd693cd700 1 -- 172.21.2.102:0/1768705907 >> 172.21.2.102:0/1768705907 conn(0x7fcd640ce450 msgr2=0x7fcd640d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:50.971 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:50.951 7fcd693cd700 1 -- 172.21.2.102:0/1768705907 shutdown_connections 2021-02-10T11:41:50.971 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:50.951 7fcd693cd700 1 -- 172.21.2.102:0/1768705907 wait complete. 2021-02-10T11:41:50.973 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:50.966 7fcd693cd700 1 Processor -- start 2021-02-10T11:41:50.973 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:50.966 7fcd693cd700 1 -- start start 2021-02-10T11:41:50.973 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:50.967 7fcd693cd700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fcd640a2fb0 0x7fcd640a33b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:50.973 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:50.967 7fcd693cd700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcd640a39d0 0x7fcd640a4860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:50.973 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:50.967 7fcd693cd700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcd640a3dd0 0x7fcd640a4e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:50.973 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:50.967 7fcd693cd700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd640a2620 con 0x7fcd640a3dd0 2021-02-10T11:41:50.974 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:50.967 7fcd693cd700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd640a2840 con 0x7fcd640a2fb0 2021-02-10T11:41:50.974 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:50.967 7fcd693cd700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd640a2b80 con 0x7fcd640a39d0 2021-02-10T11:41:50.974 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:50.967 7fcd6387e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcd640a3dd0 0x7fcd640a4e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:50.974 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:50.967 7fcd6387e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcd640a3dd0 0x7fcd640a4e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:41760/0 (socket says 172.21.2.102:41760) 2021-02-10T11:41:50.974 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:50.967 7fcd6387e700 1 -- 172.21.2.102:0/2466425696 learned_addr learned my addr 172.21.2.102:0/2466425696 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:41:50.974 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:50.967 7fcd6387e700 1 -- 172.21.2.102:0/2466425696 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcd640a39d0 msgr2=0x7fcd640a4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:50.974 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:50.967 7fcd6387e700 1 --2- 172.21.2.102:0/2466425696 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcd640a39d0 0x7fcd640a4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:50.974 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:50.967 7fcd6387e700 1 -- 172.21.2.102:0/2466425696 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fcd640a2fb0 msgr2=0x7fcd640a33b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:50.974 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:50.967 7fcd6387e700 1 --2- 172.21.2.102:0/2466425696 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fcd640a2fb0 0x7fcd640a33b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:50.975 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:50.967 7fcd6387e700 1 -- 172.21.2.102:0/2466425696 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd640a54a0 con 0x7fcd640a3dd0 2021-02-10T11:41:50.975 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:50.967 7fcd6387e700 1 --2- 172.21.2.102:0/2466425696 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcd640a3dd0 0x7fcd640a4e80 secure :-1 s=READY pgs=3458 cs=0 l=1 rev1=1 rx=0x7fcd5c009960 tx=0x7fcd5c00bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:50.975 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:50.968 7fcd4bfff700 1 -- 172.21.2.102:0/2466425696 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcd5c003030 con 0x7fcd640a3dd0 2021-02-10T11:41:50.975 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:50.968 7fcd4bfff700 1 -- 172.21.2.102:0/2466425696 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd5c001310 con 0x7fcd640a3dd0 2021-02-10T11:41:50.975 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:50.968 7fcd4bfff700 1 -- 172.21.2.102:0/2466425696 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcd5c001830 con 0x7fcd640a3dd0 2021-02-10T11:41:50.975 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:50.968 7fcd693cd700 1 -- 172.21.2.102:0/2466425696 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd640a54a0 con 0x7fcd640a3dd0 2021-02-10T11:41:50.975 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:50.968 7fcd693cd700 1 -- 172.21.2.102:0/2466425696 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd640a5a30 con 0x7fcd640a3dd0 2021-02-10T11:41:50.975 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:50.969 7fcd4bfff700 1 -- 172.21.2.102:0/2466425696 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fcd5c0014f0 con 0x7fcd640a3dd0 2021-02-10T11:41:50.976 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:50.969 7fcd4bfff700 1 --2- 172.21.2.102:0/2466425696 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcd440186c0 0x7fcd4401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:50.976 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:50.969 7fcd4bfff700 1 -- 172.21.2.102:0/2466425696 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fcd5c032970 con 0x7fcd640a3dd0 2021-02-10T11:41:50.976 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:50.969 7fcd6307d700 1 --2- 172.21.2.102:0/2466425696 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcd440186c0 0x7fcd4401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:50.976 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:50.969 7fcd6307d700 1 --2- 172.21.2.102:0/2466425696 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcd440186c0 0x7fcd4401abc0 crc :-1 s=READY pgs=1505 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:51.002 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:51.000 7fcd693cd700 1 -- 172.21.2.102:0/2466425696 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd640a4330 con 0x7fcd640a3dd0 2021-02-10T11:41:51.005 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:51.003 7fcd4bfff700 1 -- 172.21.2.102:0/2466425696 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fcd5c0016e0 con 0x7fcd640a3dd0 2021-02-10T11:41:51.134 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:51.131 7fcd693cd700 1 -- 172.21.2.102:0/2466425696 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mgr", ""]}) v1 -- 0x7fcd640a5a30 con 0x7fcd440186c0 2021-02-10T11:41:51.134 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:51.132 7fcd4bfff700 1 -- 172.21.2.102:0/2466425696 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) v1 ==== 41+0+0 (crc 0 0 0) 0x7fcd540009b0 con 0x7fcd440186c0 2021-02-10T11:41:51.135 INFO:teuthology.orchestra.run.gibba002.stderr:instructed osd(s) 1 to deep-scrub 2021-02-10T11:41:51.136 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:51.134 7fcd693cd700 1 -- 172.21.2.102:0/2466425696 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcd440186c0 msgr2=0x7fcd4401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:51.136 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:51.134 7fcd693cd700 1 --2- 172.21.2.102:0/2466425696 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcd440186c0 0x7fcd4401abc0 crc :-1 s=READY pgs=1505 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:51.136 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:51.134 7fcd693cd700 1 -- 172.21.2.102:0/2466425696 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcd640a3dd0 msgr2=0x7fcd640a4e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:51.136 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:51.134 7fcd693cd700 1 --2- 172.21.2.102:0/2466425696 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcd640a3dd0 0x7fcd640a4e80 secure :-1 s=READY pgs=3458 cs=0 l=1 rev1=1 rx=0x7fcd5c009960 tx=0x7fcd5c00bcf0).stop 2021-02-10T11:41:51.137 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:51.134 7fcd693cd700 1 -- 172.21.2.102:0/2466425696 shutdown_connections 2021-02-10T11:41:51.137 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:51.134 7fcd693cd700 1 --2- 172.21.2.102:0/2466425696 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcd640a3dd0 0x7fcd640a4e80 unknown :-1 s=CLOSED pgs=3458 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:51.137 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:51.134 7fcd693cd700 1 --2- 172.21.2.102:0/2466425696 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcd640a39d0 0x7fcd640a4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:51.137 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:51.134 7fcd693cd700 1 --2- 172.21.2.102:0/2466425696 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fcd440186c0 0x7fcd4401abc0 unknown :-1 s=CLOSED pgs=1505 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:51.138 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:51.134 7fcd693cd700 1 --2- 172.21.2.102:0/2466425696 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fcd640a2fb0 0x7fcd640a33b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:51.138 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:51.134 7fcd693cd700 1 -- 172.21.2.102:0/2466425696 >> 172.21.2.102:0/2466425696 conn(0x7fcd6408f030 msgr2=0x7fcd640b5e80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:51.138 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:51.134 7fcd693cd700 1 -- 172.21.2.102:0/2466425696 shutdown_connections 2021-02-10T11:41:51.138 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:51.134 7fcd693cd700 1 -- 172.21.2.102:0/2466425696 wait complete. 2021-02-10T11:41:51.145 INFO:tasks.ceph:Scrubbing osd.2 2021-02-10T11:41:51.145 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2021-02-10T11:41:51.581 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:51.562 7f41f97a6700 1 Processor -- start 2021-02-10T11:41:51.582 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:51.562 7f41f97a6700 1 -- start start 2021-02-10T11:41:51.582 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:51.562 7f41f97a6700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f41f40d1880 0x7f41f41b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:51.582 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:51.562 7f41f97a6700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41f40fe1d0 con 0x7f41f40cff80 2021-02-10T11:41:51.583 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:51.562 7f41f97a6700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41f4109a70 con 0x7f41f41b0af0 2021-02-10T11:41:51.583 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:51.562 7f41f97a6700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41f4088070 con 0x7f41f40d1880 2021-02-10T11:41:51.583 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:51.563 7f41f3d75700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f41f40cff80 0x7f41f4109140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:33992/0 (socket says 172.21.2.102:33992) 2021-02-10T11:41:51.583 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:51.563 7f41f3d75700 1 -- 172.21.2.102:0/368937007 learned_addr learned my addr 172.21.2.102:0/368937007 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:41:51.584 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:51.563 7f41f3574700 1 --2- 172.21.2.102:0/368937007 >> v2:172.21.2.113:3300/0 conn(0x7f41f40d1880 0x7f41f41b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:51.584 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:51.563 7f41f2572700 1 -- 172.21.2.102:0/368937007 <== mon.1 v1:172.21.2.113:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1718036425 0 0) 0x7f41e0000f30 con 0x7f41f41b0af0 2021-02-10T11:41:51.584 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:51.563 7f41f2572700 1 -- 172.21.2.102:0/368937007 --> v1:172.21.2.113:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41d8002fe0 con 0x7f41f41b0af0 2021-02-10T11:41:51.584 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:51.563 7f41f2572700 1 -- 172.21.2.102:0/368937007 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4233470339 0 0) 0x7f41ec001500 con 0x7f41f40cff80 2021-02-10T11:41:51.585 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:51.563 7f41f3574700 1 -- 172.21.2.102:0/368937007 >> v1:172.21.2.113:6789/0 conn(0x7f41f41b0af0 legacy=0x7f41f4108950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:51.585 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:51.563 7f41f3574700 1 -- 172.21.2.102:0/368937007 >> v1:172.21.2.102:6789/0 conn(0x7f41f40cff80 legacy=0x7f41f4109140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:51.585 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:51.563 7f41f3574700 1 -- 172.21.2.102:0/368937007 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41f4088260 con 0x7f41f40d1880 2021-02-10T11:41:51.585 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:51.563 7f41f3574700 1 --2- 172.21.2.102:0/368937007 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f41f40d1880 0x7f41f41b04d0 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 rx=0x7f41e4006650 tx=0x7f41e4009a80).ready entity=mon.1 client_cookie=189e373113fa6211 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:51.586 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:51.564 7f41f2572700 1 -- 172.21.2.102:0/368937007 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f41e400c0c0 con 0x7f41f40d1880 2021-02-10T11:41:51.586 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:51.564 7f41f2572700 1 -- 172.21.2.102:0/368937007 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41e400c340 con 0x7f41f40d1880 2021-02-10T11:41:51.586 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:51.564 7f41f2572700 1 -- 172.21.2.102:0/368937007 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f41e400c860 con 0x7f41f40d1880 2021-02-10T11:41:51.587 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:51.564 7f41f97a6700 1 -- 172.21.2.102:0/368937007 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f41f40d1880 msgr2=0x7f41f41b04d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:51.587 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:51.564 7f41f97a6700 1 --2- 172.21.2.102:0/368937007 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f41f40d1880 0x7f41f41b04d0 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 rx=0x7f41e4006650 tx=0x7f41e4009a80).stop 2021-02-10T11:41:51.587 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:51.564 7f41f97a6700 1 -- 172.21.2.102:0/368937007 shutdown_connections 2021-02-10T11:41:51.587 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:51.564 7f41f97a6700 1 --2- 172.21.2.102:0/368937007 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f41f40d1880 0x7f41f41b04d0 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:51.588 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:51.564 7f41f97a6700 1 -- 172.21.2.102:0/368937007 >> 172.21.2.102:0/368937007 conn(0x7f41f40ce430 msgr2=0x7f41f40d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:51.588 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:51.564 7f41f97a6700 1 -- 172.21.2.102:0/368937007 shutdown_connections 2021-02-10T11:41:51.588 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:51.564 7f41f97a6700 1 -- 172.21.2.102:0/368937007 wait complete. 2021-02-10T11:41:51.588 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:51.579 7f41f97a6700 1 Processor -- start 2021-02-10T11:41:51.589 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:51.579 7f41f97a6700 1 -- start start 2021-02-10T11:41:51.591 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:51.579 7f41f97a6700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f41f41b8e50 0x7f41f4091bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:51.591 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:51.579 7f41f97a6700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f41f41b9250 0x7f41f40aa460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:51.592 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:51.579 7f41f97a6700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f41f40aaa80 0x7f41f41acaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:51.592 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:51.579 7f41f97a6700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41f41b9770 con 0x7f41f40aaa80 2021-02-10T11:41:51.592 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:51.579 7f41f97a6700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41f41ac570 con 0x7f41f41b8e50 2021-02-10T11:41:51.592 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:51.579 7f41f97a6700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41f41ac160 con 0x7f41f41b9250 2021-02-10T11:41:51.593 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:51.580 7f41f3d75700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f41f40aaa80 0x7f41f41acaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:51.593 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:51.580 7f41f3d75700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f41f40aaa80 0x7f41f41acaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:41772/0 (socket says 172.21.2.102:41772) 2021-02-10T11:41:51.593 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:51.580 7f41f3d75700 1 -- 172.21.2.102:0/999507011 learned_addr learned my addr 172.21.2.102:0/999507011 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:41:51.593 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:51.580 7f41f3574700 1 --2- 172.21.2.102:0/999507011 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f41f41b8e50 0x7f41f4091bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:51.594 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:51.580 7f41f2d73700 1 --2- 172.21.2.102:0/999507011 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f41f41b9250 0x7f41f40aa460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:51.594 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:51.580 7f41f3d75700 1 -- 172.21.2.102:0/999507011 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f41f41b9250 msgr2=0x7f41f40aa460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:51.594 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:51.580 7f41f3d75700 1 --2- 172.21.2.102:0/999507011 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f41f41b9250 0x7f41f40aa460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:51.594 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:51.580 7f41f3d75700 1 -- 172.21.2.102:0/999507011 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f41f41b8e50 msgr2=0x7f41f4091bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:51.595 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:51.580 7f41f3d75700 1 --2- 172.21.2.102:0/999507011 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f41f41b8e50 0x7f41f4091bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:51.595 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:51.580 7f41f3d75700 1 -- 172.21.2.102:0/999507011 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41f41ad110 con 0x7f41f40aaa80 2021-02-10T11:41:51.595 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:51.580 7f41f3d75700 1 --2- 172.21.2.102:0/999507011 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f41f40aaa80 0x7f41f41acaf0 secure :-1 s=READY pgs=3460 cs=0 l=1 rev1=1 rx=0x7f41ec001c10 tx=0x7f41ec00ac60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:51.596 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:51.581 7f41ebfff700 1 -- 172.21.2.102:0/999507011 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f41ec000f90 con 0x7f41f40aaa80 2021-02-10T11:41:51.596 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:51.581 7f41ebfff700 1 -- 172.21.2.102:0/999507011 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41ec007020 con 0x7f41f40aaa80 2021-02-10T11:41:51.596 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:51.581 7f41ebfff700 1 -- 172.21.2.102:0/999507011 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f41ec0074b0 con 0x7f41f40aaa80 2021-02-10T11:41:51.596 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:51.581 7f41f97a6700 1 -- 172.21.2.102:0/999507011 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41f41ad110 con 0x7f41f40aaa80 2021-02-10T11:41:51.597 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:51.581 7f41f97a6700 1 -- 172.21.2.102:0/999507011 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41f41ad110 con 0x7f41f40aaa80 2021-02-10T11:41:51.597 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:51.581 7f41ebfff700 1 -- 172.21.2.102:0/999507011 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f41ec007200 con 0x7f41f40aaa80 2021-02-10T11:41:51.597 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:51.581 7f41ebfff700 1 --2- 172.21.2.102:0/999507011 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f41d00186c0 0x7f41d001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:51.597 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:51.581 7f41ebfff700 1 -- 172.21.2.102:0/999507011 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f41ec0328b0 con 0x7f41f40aaa80 2021-02-10T11:41:51.598 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:51.582 7f41f3574700 1 --2- 172.21.2.102:0/999507011 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f41d00186c0 0x7f41d001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:51.598 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:51.582 7f41f3574700 1 --2- 172.21.2.102:0/999507011 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f41d00186c0 0x7f41d001abc0 crc :-1 s=READY pgs=1506 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:51.615 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:51.612 7f41f97a6700 1 --2- 172.21.2.102:0/999507011 >> [v2:172.21.2.102:6810/36089,v1:172.21.2.102:6811/36089] conn(0x7f41f41adc80 0x7f41f40627c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:51.615 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:51.612 7f41f97a6700 1 -- 172.21.2.102:0/999507011 --> [v2:172.21.2.102:6810/36089,v1:172.21.2.102:6811/36089] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f41f4082ad0 con 0x7f41f41adc80 2021-02-10T11:41:51.616 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:51.613 7f41f2d73700 1 --2- 172.21.2.102:0/999507011 >> [v2:172.21.2.102:6810/36089,v1:172.21.2.102:6811/36089] conn(0x7f41f41adc80 0x7f41f40627c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:51.616 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:51.613 7f41f2d73700 1 --2- 172.21.2.102:0/999507011 >> [v2:172.21.2.102:6810/36089,v1:172.21.2.102:6811/36089] conn(0x7f41f41adc80 0x7f41f40627c0 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:51.617 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:51.613 7f41ebfff700 1 -- 172.21.2.102:0/999507011 <== osd.2 v2:172.21.2.102:6810/36089 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+5151 (crc 0 0 0) 0x7f41e0006a50 con 0x7f41f41adc80 2021-02-10T11:41:51.623 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:51.621 7f41f97a6700 1 -- 172.21.2.102:0/999507011 --> [v2:172.21.2.102:6810/36089,v1:172.21.2.102:6811/36089] -- command(tid 2: {"prefix": "config set", "key": "osd_debug_deep_scrub_sleep", "value": "0"}) v1 -- 0x7f41f4082ad0 con 0x7f41f41adc80 2021-02-10T11:41:51.624 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:51.621 7f41ebfff700 1 -- 172.21.2.102:0/999507011 <== osd.2 v2:172.21.2.102:6810/36089 2 ==== command_reply(tid 2: 0 Set osd_debug_deep_scrub_sleep to 0) v1 ==== 43+0+0 (crc 0 0 0) 0x7f41e0000cd0 con 0x7f41f41adc80 2021-02-10T11:41:51.626 INFO:teuthology.orchestra.run.gibba002.stderr:Set osd_debug_deep_scrub_sleep to 0 2021-02-10T11:41:51.627 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:51.622 7f41f97a6700 1 -- 172.21.2.102:0/999507011 >> [v2:172.21.2.102:6810/36089,v1:172.21.2.102:6811/36089] conn(0x7f41f41adc80 msgr2=0x7f41f40627c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:51.627 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:51.622 7f41f97a6700 1 --2- 172.21.2.102:0/999507011 >> [v2:172.21.2.102:6810/36089,v1:172.21.2.102:6811/36089] conn(0x7f41f41adc80 0x7f41f40627c0 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:51.627 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:51.623 7f41f97a6700 1 -- 172.21.2.102:0/999507011 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f41d00186c0 msgr2=0x7f41d001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:51.627 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:51.623 7f41f97a6700 1 --2- 172.21.2.102:0/999507011 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f41d00186c0 0x7f41d001abc0 crc :-1 s=READY pgs=1506 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:51.628 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:51.623 7f41f97a6700 1 -- 172.21.2.102:0/999507011 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f41f40aaa80 msgr2=0x7f41f41acaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:51.628 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:51.623 7f41f97a6700 1 --2- 172.21.2.102:0/999507011 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f41f40aaa80 0x7f41f41acaf0 secure :-1 s=READY pgs=3460 cs=0 l=1 rev1=1 rx=0x7f41ec001c10 tx=0x7f41ec00ac60).stop 2021-02-10T11:41:51.628 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:51.623 7f41f3d75700 1 -- 172.21.2.102:0/999507011 reap_dead start 2021-02-10T11:41:51.628 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:51.623 7f41f97a6700 1 -- 172.21.2.102:0/999507011 shutdown_connections 2021-02-10T11:41:51.629 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:51.623 7f41f97a6700 1 -- 172.21.2.102:0/999507011 >> 172.21.2.102:0/999507011 conn(0x7f41f40846e0 msgr2=0x7f41f41ab3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:51.629 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:51.623 7f41f97a6700 1 -- 172.21.2.102:0/999507011 shutdown_connections 2021-02-10T11:41:51.629 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:51.623 7f41f97a6700 1 -- 172.21.2.102:0/999507011 wait complete. 2021-02-10T11:41:51.634 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd deep-scrub 2 2021-02-10T11:41:52.062 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:52.043 7fcaba003700 1 Processor -- start 2021-02-10T11:41:52.063 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:52.043 7fcaba003700 1 -- start start 2021-02-10T11:41:52.063 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:52.044 7fcaba003700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fcab40d1870 0x7fcab41b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:52.063 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:52.044 7fcaba003700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcab40f0650 0x7fcab40cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:52.063 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:52.044 7fcaba003700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcab40f0b10 con 0x7fcab41b0a10 2021-02-10T11:41:52.063 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:52.044 7fcaba003700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcab40e5ed0 con 0x7fcab40d1870 2021-02-10T11:41:52.063 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:52.044 7fcaba003700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcab40d1c70 con 0x7fcab40f0650 2021-02-10T11:41:52.064 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:52.044 7fcab37fe700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fcab40d1870 0x7fcab41b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:52.064 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:52.044 7fcab37fe700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fcab40d1870 0x7fcab41b03f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:41782/0 (socket says 172.21.2.102:41782) 2021-02-10T11:41:52.064 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:52.044 7fcab2ffd700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fcab41b0a10 0x7fcab40cf330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:34006/0 (socket says 172.21.2.102:34006) 2021-02-10T11:41:52.064 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:52.044 7fcab37fe700 1 -- 172.21.2.102:0/1803480472 learned_addr learned my addr 172.21.2.102:0/1803480472 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:41:52.064 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:52.044 7fcab3fff700 1 --2- 172.21.2.102:0/1803480472 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcab40f0650 0x7fcab40cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:52.064 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:52.044 7fcab37fe700 1 -- 172.21.2.102:0/1803480472 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcab40f0650 msgr2=0x7fcab40cfb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:41:52.064 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:52.044 7fcab37fe700 1 --2- 172.21.2.102:0/1803480472 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcab40f0650 0x7fcab40cfb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:52.065 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:52.044 7fcab37fe700 1 -- 172.21.2.102:0/1803480472 >> v1:172.21.2.102:6789/0 conn(0x7fcab41b0a10 legacy=0x7fcab40cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:52.065 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:52.044 7fcab27fc700 1 -- 172.21.2.102:0/1803480472 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2553001707 0 0) 0x7fcaa40010b0 con 0x7fcab41b0a10 2021-02-10T11:41:52.065 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:52.045 7fcab37fe700 1 -- 172.21.2.102:0/1803480472 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcab40d1e90 con 0x7fcab40d1870 2021-02-10T11:41:52.065 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:52.045 7fcab3fff700 1 --2- 172.21.2.102:0/1803480472 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcab40f0650 0x7fcab40cfb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:41:52.065 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:52.045 7fcab37fe700 1 --2- 172.21.2.102:0/1803480472 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcab40d1870 0x7fcab41b03f0 secure :-1 s=READY pgs=3462 cs=0 l=1 rev1=1 rx=0x7fca9c006700 tx=0x7fca9c009b80).ready entity=mon.0 client_cookie=4a8eda2626edd40f server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:52.065 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:52.045 7fcab27fc700 1 -- 172.21.2.102:0/1803480472 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fca9c00c0c0 con 0x7fcab40d1870 2021-02-10T11:41:52.065 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:52.045 7fcab27fc700 1 -- 172.21.2.102:0/1803480472 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca9c00c0c0 con 0x7fcab40d1870 2021-02-10T11:41:52.065 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:52.045 7fcab27fc700 1 -- 172.21.2.102:0/1803480472 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fca9c00c340 con 0x7fcab40d1870 2021-02-10T11:41:52.066 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:52.045 7fcaba003700 1 -- 172.21.2.102:0/1803480472 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcab40d1870 msgr2=0x7fcab41b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:52.066 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:52.045 7fcaba003700 1 --2- 172.21.2.102:0/1803480472 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcab40d1870 0x7fcab41b03f0 secure :-1 s=READY pgs=3462 cs=0 l=1 rev1=1 rx=0x7fca9c006700 tx=0x7fca9c009b80).stop 2021-02-10T11:41:52.066 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:52.046 7fcaba003700 1 -- 172.21.2.102:0/1803480472 shutdown_connections 2021-02-10T11:41:52.066 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:52.046 7fcaba003700 1 --2- 172.21.2.102:0/1803480472 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcab40f0650 0x7fcab40cfb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:52.066 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:52.046 7fcaba003700 1 --2- 172.21.2.102:0/1803480472 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcab40d1870 0x7fcab41b03f0 unknown :-1 s=CLOSED pgs=3462 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:52.066 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:52.046 7fcaba003700 1 -- 172.21.2.102:0/1803480472 >> 172.21.2.102:0/1803480472 conn(0x7fcab40ce450 msgr2=0x7fcab40d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:52.067 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:52.046 7fcaba003700 1 -- 172.21.2.102:0/1803480472 shutdown_connections 2021-02-10T11:41:52.067 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:52.046 7fcaba003700 1 -- 172.21.2.102:0/1803480472 wait complete. 2021-02-10T11:41:52.069 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:52.061 7fcaba003700 1 Processor -- start 2021-02-10T11:41:52.069 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:52.061 7fcaba003700 1 -- start start 2021-02-10T11:41:52.069 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:52.061 7fcaba003700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcab40b7510 0x7fcab40b7910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:52.069 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:52.061 7fcaba003700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fcab409d530 0x7fcab409d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:52.069 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:52.061 7fcaba003700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcab409df50 0x7fcab409eca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:52.070 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:52.061 7fcaba003700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcab40b8140 con 0x7fcab409df50 2021-02-10T11:41:52.070 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:52.061 7fcaba003700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcab409e750 con 0x7fcab409d530 2021-02-10T11:41:52.070 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:52.061 7fcaba003700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcab409e350 con 0x7fcab40b7510 2021-02-10T11:41:52.070 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:52.061 7fcab3fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcab409df50 0x7fcab409eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:52.070 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:52.061 7fcab3fff700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcab409df50 0x7fcab409eca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:41784/0 (socket says 172.21.2.102:41784) 2021-02-10T11:41:52.070 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:52.061 7fcab3fff700 1 -- 172.21.2.102:0/1303092989 learned_addr learned my addr 172.21.2.102:0/1303092989 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:41:52.070 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:52.062 7fcab37fe700 1 --2- 172.21.2.102:0/1303092989 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcab40b7510 0x7fcab40b7910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:52.070 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:52.062 7fcab3fff700 1 -- 172.21.2.102:0/1303092989 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcab40b7510 msgr2=0x7fcab40b7910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:52.071 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:52.062 7fcab3fff700 1 --2- 172.21.2.102:0/1303092989 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcab40b7510 0x7fcab40b7910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:52.071 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:52.062 7fcab3fff700 1 -- 172.21.2.102:0/1303092989 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fcab409d530 msgr2=0x7fcab409d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:52.071 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:52.062 7fcab3fff700 1 --2- 172.21.2.102:0/1303092989 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fcab409d530 0x7fcab409d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:52.071 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:52.062 7fcab3fff700 1 -- 172.21.2.102:0/1303092989 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcab409f2c0 con 0x7fcab409df50 2021-02-10T11:41:52.071 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:52.062 7fcab3fff700 1 --2- 172.21.2.102:0/1303092989 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcab409df50 0x7fcab409eca0 secure :-1 s=READY pgs=3463 cs=0 l=1 rev1=1 rx=0x7fcaac009980 tx=0x7fcaac00bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:52.071 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:52.062 7fcaabfff700 1 -- 172.21.2.102:0/1303092989 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcaac00c2f0 con 0x7fcab409df50 2021-02-10T11:41:52.071 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:52.062 7fcaabfff700 1 -- 172.21.2.102:0/1303092989 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcaac006790 con 0x7fcab409df50 2021-02-10T11:41:52.071 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:52.062 7fcaabfff700 1 -- 172.21.2.102:0/1303092989 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcaac00c870 con 0x7fcab409df50 2021-02-10T11:41:52.072 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:52.062 7fcaba003700 1 -- 172.21.2.102:0/1303092989 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcab409f2c0 con 0x7fcab409df50 2021-02-10T11:41:52.072 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:52.062 7fcaba003700 1 -- 172.21.2.102:0/1303092989 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcab409f2c0 con 0x7fcab409df50 2021-02-10T11:41:52.072 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:52.063 7fcaabfff700 1 -- 172.21.2.102:0/1303092989 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fcaac0011f0 con 0x7fcab409df50 2021-02-10T11:41:52.072 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:52.063 7fcaabfff700 1 --2- 172.21.2.102:0/1303092989 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fca980186c0 0x7fca9801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:52.072 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:52.063 7fcaabfff700 1 -- 172.21.2.102:0/1303092989 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fcaac032980 con 0x7fcab409df50 2021-02-10T11:41:52.072 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:52.063 7fcab37fe700 1 --2- 172.21.2.102:0/1303092989 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fca980186c0 0x7fca9801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:52.072 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:52.064 7fcab37fe700 1 --2- 172.21.2.102:0/1303092989 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fca980186c0 0x7fca9801abc0 crc :-1 s=READY pgs=1507 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:52.096 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:52.093 7fcaba003700 1 -- 172.21.2.102:0/1303092989 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcab40ebc50 con 0x7fcab409df50 2021-02-10T11:41:52.098 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:52.096 7fcaabfff700 1 -- 172.21.2.102:0/1303092989 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fcaac00be10 con 0x7fcab409df50 2021-02-10T11:41:52.227 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:52.225 7fcaba003700 1 -- 172.21.2.102:0/1303092989 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mgr", ""]}) v1 -- 0x7fcab40f0b10 con 0x7fca980186c0 2021-02-10T11:41:52.228 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:52.226 7fcaabfff700 1 -- 172.21.2.102:0/1303092989 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) v1 ==== 41+0+0 (crc 0 0 0) 0x7fca9c0009b0 con 0x7fca980186c0 2021-02-10T11:41:52.228 INFO:teuthology.orchestra.run.gibba002.stderr:instructed osd(s) 2 to deep-scrub 2021-02-10T11:41:52.230 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:52.228 7fcaba003700 1 -- 172.21.2.102:0/1303092989 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fca980186c0 msgr2=0x7fca9801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:52.230 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:52.228 7fcaba003700 1 --2- 172.21.2.102:0/1303092989 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fca980186c0 0x7fca9801abc0 crc :-1 s=READY pgs=1507 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:52.230 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:52.228 7fcaba003700 1 -- 172.21.2.102:0/1303092989 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcab409df50 msgr2=0x7fcab409eca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:52.231 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:52.228 7fcaba003700 1 --2- 172.21.2.102:0/1303092989 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcab409df50 0x7fcab409eca0 secure :-1 s=READY pgs=3463 cs=0 l=1 rev1=1 rx=0x7fcaac009980 tx=0x7fcaac00bcd0).stop 2021-02-10T11:41:52.231 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:52.228 7fcaba003700 1 -- 172.21.2.102:0/1303092989 shutdown_connections 2021-02-10T11:41:52.231 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:52.228 7fcaba003700 1 --2- 172.21.2.102:0/1303092989 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fcab409df50 0x7fcab409eca0 unknown :-1 s=CLOSED pgs=3463 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:52.231 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:52.228 7fcaba003700 1 --2- 172.21.2.102:0/1303092989 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fca980186c0 0x7fca9801abc0 unknown :-1 s=CLOSED pgs=1507 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:52.231 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:52.228 7fcaba003700 1 --2- 172.21.2.102:0/1303092989 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fcab409d530 0x7fcab409d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:52.231 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:52.228 7fcaba003700 1 --2- 172.21.2.102:0/1303092989 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fcab40b7510 0x7fcab40b7910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:52.231 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:52.228 7fcaba003700 1 -- 172.21.2.102:0/1303092989 >> 172.21.2.102:0/1303092989 conn(0x7fcab40b83b0 msgr2=0x7fcab40a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:52.231 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:52.228 7fcaba003700 1 -- 172.21.2.102:0/1303092989 shutdown_connections 2021-02-10T11:41:52.231 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:52.228 7fcaba003700 1 -- 172.21.2.102:0/1303092989 wait complete. 2021-02-10T11:41:52.239 INFO:tasks.ceph:Scrubbing osd.3 2021-02-10T11:41:52.239 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2021-02-10T11:41:52.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:52.654 7f4a7c491700 1 Processor -- start 2021-02-10T11:41:52.673 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:52.654 7f4a7c491700 1 -- start start 2021-02-10T11:41:52.674 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:52.654 7f4a7c491700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f4a740d1880 0x7f4a741b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:52.674 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:52.654 7f4a7c491700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f4a741b0af0 0x7f4a74108950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:52.674 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:52.654 7f4a7c491700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a740fe1d0 con 0x7f4a740d1c80 2021-02-10T11:41:52.674 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:52.654 7f4a7c491700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a74109a70 con 0x7f4a740d1880 2021-02-10T11:41:52.674 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:52.654 7f4a7c491700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a74109760 con 0x7f4a741b0af0 2021-02-10T11:41:52.674 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:52.655 7f4a7a2ad700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f4a740d1880 0x7f4a741b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:52.674 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:52.655 7f4a7aaae700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f4a740d1c80 0x7f4a74108f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:34016/0 (socket says 172.21.2.102:34016) 2021-02-10T11:41:52.674 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:52.655 7f4a7aaae700 1 -- 172.21.2.102:0/1201894160 learned_addr learned my addr 172.21.2.102:0/1201894160 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:41:52.675 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:52.655 7f4a7a2ad700 1 -- 172.21.2.102:0/1201894160 >> v2:172.21.2.113:3300/0 conn(0x7f4a741b0af0 msgr2=0x7f4a74108950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:41:52.675 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:52.655 7f4a7a2ad700 1 --2- 172.21.2.102:0/1201894160 >> v2:172.21.2.113:3300/0 conn(0x7f4a741b0af0 0x7f4a74108950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:52.675 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:52.655 7f4a7a2ad700 1 -- 172.21.2.102:0/1201894160 >> v1:172.21.2.102:6789/0 conn(0x7f4a740d1c80 legacy=0x7f4a74108f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:52.675 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:52.655 7f4a7a2ad700 1 -- 172.21.2.102:0/1201894160 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a74088070 con 0x7f4a740d1880 2021-02-10T11:41:52.675 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:52.655 7f4a7a2ad700 1 --2- 172.21.2.102:0/1201894160 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4a740d1880 0x7f4a741b04d0 secure :-1 s=READY pgs=3465 cs=0 l=1 rev1=1 rx=0x7f4a70006650 tx=0x7f4a70009a80).ready entity=mon.0 client_cookie=a2186c0eda5d5b26 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:52.675 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:52.655 7f4a792ab700 1 -- 172.21.2.102:0/1201894160 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4a700009b0 con 0x7f4a740d1880 2021-02-10T11:41:52.675 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:52.656 7f4a792ab700 1 -- 172.21.2.102:0/1201894160 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a7000c0c0 con 0x7f4a740d1880 2021-02-10T11:41:52.675 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:52.656 7f4a792ab700 1 -- 172.21.2.102:0/1201894160 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4a7000c510 con 0x7f4a740d1880 2021-02-10T11:41:52.676 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:52.656 7f4a7c491700 1 -- 172.21.2.102:0/1201894160 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4a740d1880 msgr2=0x7f4a741b04d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:52.676 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:52.656 7f4a7c491700 1 --2- 172.21.2.102:0/1201894160 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4a740d1880 0x7f4a741b04d0 secure :-1 s=READY pgs=3465 cs=0 l=1 rev1=1 rx=0x7f4a70006650 tx=0x7f4a70009a80).stop 2021-02-10T11:41:52.676 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:52.656 7f4a7c491700 1 -- 172.21.2.102:0/1201894160 shutdown_connections 2021-02-10T11:41:52.676 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:52.656 7f4a7c491700 1 --2- 172.21.2.102:0/1201894160 >> v2:172.21.2.113:3300/0 conn(0x7f4a741b0af0 0x7f4a74108950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:52.676 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:52.656 7f4a7c491700 1 --2- 172.21.2.102:0/1201894160 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4a740d1880 0x7f4a741b04d0 unknown :-1 s=CLOSED pgs=3465 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:52.676 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:52.656 7f4a7c491700 1 -- 172.21.2.102:0/1201894160 >> 172.21.2.102:0/1201894160 conn(0x7f4a740ce430 msgr2=0x7f4a740d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:52.676 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:52.656 7f4a7c491700 1 -- 172.21.2.102:0/1201894160 shutdown_connections 2021-02-10T11:41:52.676 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:52.656 7f4a7c491700 1 -- 172.21.2.102:0/1201894160 wait complete. 2021-02-10T11:41:52.677 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:52.671 7f4a7c491700 1 Processor -- start 2021-02-10T11:41:52.677 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:52.671 7f4a7c491700 1 -- start start 2021-02-10T11:41:52.678 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:52.671 7f4a7c491700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4a7408fb40 0x7f4a74195720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:52.678 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:52.671 7f4a7c491700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4a7409b4f0 0x7f4a7409b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:52.678 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:52.671 7f4a7c491700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4a7409bf10 0x7f4a741940a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:52.679 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:52.671 7f4a7c491700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a741951a0 con 0x7f4a7408fb40 2021-02-10T11:41:52.679 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:52.671 7f4a7c491700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a74094990 con 0x7f4a7409b4f0 2021-02-10T11:41:52.679 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:52.671 7f4a7c491700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a741946c0 con 0x7f4a7409bf10 2021-02-10T11:41:52.679 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:52.672 7f4a7a2ad700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4a7408fb40 0x7f4a74195720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:52.679 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:52.672 7f4a7a2ad700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4a7408fb40 0x7f4a74195720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:41796/0 (socket says 172.21.2.102:41796) 2021-02-10T11:41:52.679 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:52.672 7f4a7a2ad700 1 -- 172.21.2.102:0/3601435279 learned_addr learned my addr 172.21.2.102:0/3601435279 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:41:52.679 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:52.672 7f4a79aac700 1 --2- 172.21.2.102:0/3601435279 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4a7409b4f0 0x7f4a7409b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:52.679 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:52.672 7f4a7aaae700 1 --2- 172.21.2.102:0/3601435279 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4a7409bf10 0x7f4a741940a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:52.680 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:52.672 7f4a7a2ad700 1 -- 172.21.2.102:0/3601435279 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4a7409bf10 msgr2=0x7f4a741940a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:52.680 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:52.672 7f4a7a2ad700 1 --2- 172.21.2.102:0/3601435279 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f4a7409bf10 0x7f4a741940a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:52.680 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:52.672 7f4a7a2ad700 1 -- 172.21.2.102:0/3601435279 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4a7409b4f0 msgr2=0x7f4a7409b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:52.680 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:52.672 7f4a7a2ad700 1 --2- 172.21.2.102:0/3601435279 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f4a7409b4f0 0x7f4a7409b8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:52.680 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:52.672 7f4a7a2ad700 1 -- 172.21.2.102:0/3601435279 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a741948e0 con 0x7f4a7408fb40 2021-02-10T11:41:52.680 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:52.672 7f4a7a2ad700 1 --2- 172.21.2.102:0/3601435279 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4a7408fb40 0x7f4a74195720 secure :-1 s=READY pgs=3466 cs=0 l=1 rev1=1 rx=0x7f4a70009630 tx=0x7f4a7000b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:52.680 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:52.672 7f4a6affd700 1 -- 172.21.2.102:0/3601435279 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4a7000bd70 con 0x7f4a7408fb40 2021-02-10T11:41:52.680 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:52.672 7f4a6affd700 1 -- 172.21.2.102:0/3601435279 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a70000960 con 0x7f4a7408fb40 2021-02-10T11:41:52.680 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:52.672 7f4a6affd700 1 -- 172.21.2.102:0/3601435279 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4a70005560 con 0x7f4a7408fb40 2021-02-10T11:41:52.681 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:52.672 7f4a7c491700 1 -- 172.21.2.102:0/3601435279 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a741948e0 con 0x7f4a7408fb40 2021-02-10T11:41:52.681 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:52.672 7f4a7c491700 1 -- 172.21.2.102:0/3601435279 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a74194e70 con 0x7f4a7408fb40 2021-02-10T11:41:52.681 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:52.673 7f4a6affd700 1 -- 172.21.2.102:0/3601435279 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4a7000a000 con 0x7f4a7408fb40 2021-02-10T11:41:52.681 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:52.673 7f4a6affd700 1 --2- 172.21.2.102:0/3601435279 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4a5c0186c0 0x7f4a5c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:52.681 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:52.673 7f4a6affd700 1 -- 172.21.2.102:0/3601435279 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f4a700323a0 con 0x7f4a7408fb40 2021-02-10T11:41:52.681 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:52.673 7f4a79aac700 1 --2- 172.21.2.102:0/3601435279 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4a5c0186c0 0x7f4a5c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:52.681 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:52.673 7f4a79aac700 1 --2- 172.21.2.102:0/3601435279 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4a5c0186c0 0x7f4a5c01abc0 crc :-1 s=READY pgs=1508 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:52.706 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:52.704 7f4a7c491700 1 --2- 172.21.2.102:0/3601435279 >> [v2:172.21.2.102:6802/36090,v1:172.21.2.102:6803/36090] conn(0x7f4a740627c0 0x7f4a74062bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:52.706 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:52.704 7f4a7c491700 1 -- 172.21.2.102:0/3601435279 --> [v2:172.21.2.102:6802/36090,v1:172.21.2.102:6803/36090] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4a74094990 con 0x7f4a740627c0 2021-02-10T11:41:52.706 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:52.704 7f4a7aaae700 1 --2- 172.21.2.102:0/3601435279 >> [v2:172.21.2.102:6802/36090,v1:172.21.2.102:6803/36090] conn(0x7f4a740627c0 0x7f4a74062bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:52.708 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:52.704 7f4a7aaae700 1 --2- 172.21.2.102:0/3601435279 >> [v2:172.21.2.102:6802/36090,v1:172.21.2.102:6803/36090] conn(0x7f4a740627c0 0x7f4a74062bc0 crc :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:52.708 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:52.705 7f4a6affd700 1 -- 172.21.2.102:0/3601435279 <== osd.3 v2:172.21.2.102:6802/36090 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+5151 (crc 0 0 0) 0x7f4a6c001000 con 0x7f4a740627c0 2021-02-10T11:41:52.713 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:52.710 7f4a7c491700 1 -- 172.21.2.102:0/3601435279 --> [v2:172.21.2.102:6802/36090,v1:172.21.2.102:6803/36090] -- command(tid 2: {"prefix": "config set", "key": "osd_debug_deep_scrub_sleep", "value": "0"}) v1 -- 0x7f4a74094990 con 0x7f4a740627c0 2021-02-10T11:41:52.713 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:52.711 7f4a6affd700 1 -- 172.21.2.102:0/3601435279 <== osd.3 v2:172.21.2.102:6802/36090 2 ==== command_reply(tid 2: 0 Set osd_debug_deep_scrub_sleep to 0) v1 ==== 43+0+0 (crc 0 0 0) 0x7f4a6c000d00 con 0x7f4a740627c0 2021-02-10T11:41:52.713 INFO:teuthology.orchestra.run.gibba002.stderr:Set osd_debug_deep_scrub_sleep to 0 2021-02-10T11:41:52.714 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:52.712 7f4a7c491700 1 -- 172.21.2.102:0/3601435279 >> [v2:172.21.2.102:6802/36090,v1:172.21.2.102:6803/36090] conn(0x7f4a740627c0 msgr2=0x7f4a74062bc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:52.714 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:52.712 7f4a7c491700 1 --2- 172.21.2.102:0/3601435279 >> [v2:172.21.2.102:6802/36090,v1:172.21.2.102:6803/36090] conn(0x7f4a740627c0 0x7f4a74062bc0 crc :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:52.715 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:52.712 7f4a7c491700 1 -- 172.21.2.102:0/3601435279 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4a5c0186c0 msgr2=0x7f4a5c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:52.715 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:52.712 7f4a7c491700 1 --2- 172.21.2.102:0/3601435279 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f4a5c0186c0 0x7f4a5c01abc0 crc :-1 s=READY pgs=1508 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:52.715 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:52.712 7f4a7c491700 1 -- 172.21.2.102:0/3601435279 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4a7408fb40 msgr2=0x7f4a74195720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:52.715 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:52.712 7f4a7c491700 1 --2- 172.21.2.102:0/3601435279 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f4a7408fb40 0x7f4a74195720 secure :-1 s=READY pgs=3466 cs=0 l=1 rev1=1 rx=0x7f4a70009630 tx=0x7f4a7000b980).stop 2021-02-10T11:41:52.715 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:52.712 7f4a7aaae700 1 -- 172.21.2.102:0/3601435279 reap_dead start 2021-02-10T11:41:52.715 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:52.712 7f4a7c491700 1 -- 172.21.2.102:0/3601435279 shutdown_connections 2021-02-10T11:41:52.715 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:52.712 7f4a7c491700 1 -- 172.21.2.102:0/3601435279 >> 172.21.2.102:0/3601435279 conn(0x7f4a7409afc0 msgr2=0x7f4a740953e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:52.715 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:52.712 7f4a7c491700 1 -- 172.21.2.102:0/3601435279 shutdown_connections 2021-02-10T11:41:52.715 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:52.712 7f4a7c491700 1 -- 172.21.2.102:0/3601435279 wait complete. 2021-02-10T11:41:52.722 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd deep-scrub 3 2021-02-10T11:41:53.153 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:53.134 7f5a136b7700 1 Processor -- start 2021-02-10T11:41:53.153 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:53.134 7f5a136b7700 1 -- start start 2021-02-10T11:41:53.153 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:53.135 7f5a136b7700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f5a0c0d0330 0x7f5a0c0ef630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:53.153 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:53.135 7f5a136b7700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a0c0f0b00 con 0x7f5a0c0d1860 2021-02-10T11:41:53.153 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:53.135 7f5a136b7700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a0c1b0bd0 con 0x7f5a0c0d1c60 2021-02-10T11:41:53.154 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:53.135 7f5a136b7700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a0c0e5ec0 con 0x7f5a0c0d0330 2021-02-10T11:41:53.154 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:53.135 7f5a10cd2700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f5a0c0d0330 0x7f5a0c0ef630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:53.154 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:53.135 7f5a114d3700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f5a0c0d1860 0x7f5a0c1b03e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:34030/0 (socket says 172.21.2.102:34030) 2021-02-10T11:41:53.154 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:53.135 7f5a114d3700 1 -- 172.21.2.102:0/1585315325 learned_addr learned my addr 172.21.2.102:0/1585315325 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:41:53.154 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:53.135 7f5a10cd2700 1 -- 172.21.2.102:0/1585315325 >> v1:172.21.2.113:6789/0 conn(0x7f5a0c0d1c60 legacy=0x7f5a0c0efc50 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:41:53.154 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:53.135 7f5a10cd2700 1 -- 172.21.2.102:0/1585315325 >> v1:172.21.2.102:6789/0 conn(0x7f5a0c0d1860 legacy=0x7f5a0c1b03e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:53.154 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:53.135 7f5a10cd2700 1 -- 172.21.2.102:0/1585315325 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a0c0f0440 con 0x7f5a0c0d0330 2021-02-10T11:41:53.155 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:53.136 7f5a10cd2700 1 --2- 172.21.2.102:0/1585315325 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5a0c0d0330 0x7f5a0c0ef630 secure :-1 s=READY pgs=3468 cs=0 l=1 rev1=1 rx=0x7f5a00006650 tx=0x7f5a00009a80).ready entity=mon.0 client_cookie=18df3c1e6c4b577b server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:53.155 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:53.136 7f59fffff700 1 -- 172.21.2.102:0/1585315325 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5a000009b0 con 0x7f5a0c0d0330 2021-02-10T11:41:53.155 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:53.136 7f59fffff700 1 -- 172.21.2.102:0/1585315325 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a0000c0c0 con 0x7f5a0c0d0330 2021-02-10T11:41:53.155 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:53.136 7f59fffff700 1 -- 172.21.2.102:0/1585315325 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5a0000c5e0 con 0x7f5a0c0d0330 2021-02-10T11:41:53.155 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:53.136 7f5a136b7700 1 -- 172.21.2.102:0/1585315325 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5a0c0d0330 msgr2=0x7f5a0c0ef630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:53.155 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:53.136 7f5a136b7700 1 --2- 172.21.2.102:0/1585315325 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5a0c0d0330 0x7f5a0c0ef630 secure :-1 s=READY pgs=3468 cs=0 l=1 rev1=1 rx=0x7f5a00006650 tx=0x7f5a00009a80).stop 2021-02-10T11:41:53.155 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:53.136 7f5a136b7700 1 -- 172.21.2.102:0/1585315325 shutdown_connections 2021-02-10T11:41:53.155 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:53.136 7f5a136b7700 1 --2- 172.21.2.102:0/1585315325 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5a0c0d0330 0x7f5a0c0ef630 unknown :-1 s=CLOSED pgs=3468 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:53.155 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:53.136 7f5a136b7700 1 -- 172.21.2.102:0/1585315325 >> 172.21.2.102:0/1585315325 conn(0x7f5a0c0ce440 msgr2=0x7f5a0c0d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:53.156 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:53.137 7f5a136b7700 1 -- 172.21.2.102:0/1585315325 shutdown_connections 2021-02-10T11:41:53.156 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:53.137 7f5a136b7700 1 -- 172.21.2.102:0/1585315325 wait complete. 2021-02-10T11:41:53.158 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:53.151 7f5a136b7700 1 Processor -- start 2021-02-10T11:41:53.158 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:53.151 7f5a136b7700 1 -- start start 2021-02-10T11:41:53.158 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:53.151 7f5a136b7700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5a0c0ac050 0x7f5a0c1c9370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:53.158 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:53.151 7f5a136b7700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5a0c0ac450 0x7f5a0c1c9990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:53.159 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:53.151 7f5a136b7700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5a0c09e520 0x7f5a0c09e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:53.159 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:53.151 7f5a136b7700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a0c0abd90 con 0x7f5a0c0ac450 2021-02-10T11:41:53.159 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:53.151 7f5a136b7700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a0c0ab8f0 con 0x7f5a0c0ac050 2021-02-10T11:41:53.159 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:53.152 7f5a136b7700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a0c1c9fb0 con 0x7f5a0c09e520 2021-02-10T11:41:53.159 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:53.152 7f5a10cd2700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5a0c0ac450 0x7f5a0c1c9990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:53.159 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:53.152 7f5a10cd2700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5a0c0ac450 0x7f5a0c1c9990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:41810/0 (socket says 172.21.2.102:41810) 2021-02-10T11:41:53.159 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:53.152 7f5a10cd2700 1 -- 172.21.2.102:0/1274035445 learned_addr learned my addr 172.21.2.102:0/1274035445 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:41:53.159 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:53.152 7f5a10cd2700 1 -- 172.21.2.102:0/1274035445 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5a0c09e520 msgr2=0x7f5a0c09e920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:53.160 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:53.152 7f5a10cd2700 1 --2- 172.21.2.102:0/1274035445 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5a0c09e520 0x7f5a0c09e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:53.160 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:53.152 7f5a10cd2700 1 -- 172.21.2.102:0/1274035445 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5a0c0ac050 msgr2=0x7f5a0c1c9370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:53.160 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:53.152 7f5a114d3700 1 --2- 172.21.2.102:0/1274035445 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5a0c0ac050 0x7f5a0c1c9370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:53.160 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:53.152 7f5a10cd2700 1 --2- 172.21.2.102:0/1274035445 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5a0c0ac050 0x7f5a0c1c9370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:53.160 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:53.152 7f5a10cd2700 1 -- 172.21.2.102:0/1274035445 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a0c09ef40 con 0x7f5a0c0ac450 2021-02-10T11:41:53.160 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:53.152 7f5a114d3700 1 --2- 172.21.2.102:0/1274035445 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5a0c0ac050 0x7f5a0c1c9370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:41:53.160 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:53.152 7f5a10cd2700 1 --2- 172.21.2.102:0/1274035445 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5a0c0ac450 0x7f5a0c1c9990 secure :-1 s=READY pgs=3469 cs=0 l=1 rev1=1 rx=0x7f5a00009630 tx=0x7f5a0000b900).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:53.160 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:53.153 7f59fdffb700 1 -- 172.21.2.102:0/1274035445 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5a000093a0 con 0x7f5a0c0ac450 2021-02-10T11:41:53.160 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:53.153 7f59fdffb700 1 -- 172.21.2.102:0/1274035445 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a0000bcf0 con 0x7f5a0c0ac450 2021-02-10T11:41:53.161 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:53.153 7f5a136b7700 1 -- 172.21.2.102:0/1274035445 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a0c09ef40 con 0x7f5a0c0ac450 2021-02-10T11:41:53.161 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:53.153 7f59fdffb700 1 -- 172.21.2.102:0/1274035445 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5a000054d0 con 0x7f5a0c0ac450 2021-02-10T11:41:53.161 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:53.153 7f5a136b7700 1 -- 172.21.2.102:0/1274035445 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a0c09ef40 con 0x7f5a0c0ac450 2021-02-10T11:41:53.161 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:53.153 7f59fdffb700 1 -- 172.21.2.102:0/1274035445 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f5a000054f0 con 0x7f5a0c0ac450 2021-02-10T11:41:53.161 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:53.153 7f59fdffb700 1 --2- 172.21.2.102:0/1274035445 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f59ec018730 0x7f59ec01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:53.161 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:53.153 7f59fdffb700 1 -- 172.21.2.102:0/1274035445 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f5a00032630 con 0x7f5a0c0ac450 2021-02-10T11:41:53.161 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:53.153 7f5a114d3700 1 --2- 172.21.2.102:0/1274035445 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f59ec018730 0x7f59ec01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:53.161 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:53.154 7f5a114d3700 1 --2- 172.21.2.102:0/1274035445 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f59ec018730 0x7f59ec01abc0 crc :-1 s=READY pgs=1509 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:53.187 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:53.185 7f5a136b7700 1 -- 172.21.2.102:0/1274035445 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a0c0ff1b0 con 0x7f5a0c0ac450 2021-02-10T11:41:53.189 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:53.187 7f59fdffb700 1 -- 172.21.2.102:0/1274035445 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f5a000052e0 con 0x7f5a0c0ac450 2021-02-10T11:41:53.318 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:53.315 7f5a136b7700 1 -- 172.21.2.102:0/1274035445 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mgr", ""]}) v1 -- 0x7f5a0c0f06e0 con 0x7f59ec018730 2021-02-10T11:41:53.318 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:53.315 7f59fdffb700 1 -- 172.21.2.102:0/1274035445 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) v1 ==== 41+0+0 (crc 0 0 0) 0x7f5a080015e0 con 0x7f59ec018730 2021-02-10T11:41:53.319 INFO:teuthology.orchestra.run.gibba002.stderr:instructed osd(s) 3 to deep-scrub 2021-02-10T11:41:53.320 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:53.318 7f5a136b7700 1 -- 172.21.2.102:0/1274035445 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f59ec018730 msgr2=0x7f59ec01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:53.320 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:53.318 7f5a136b7700 1 --2- 172.21.2.102:0/1274035445 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f59ec018730 0x7f59ec01abc0 crc :-1 s=READY pgs=1509 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:53.321 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:53.318 7f5a136b7700 1 -- 172.21.2.102:0/1274035445 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5a0c0ac450 msgr2=0x7f5a0c1c9990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:53.321 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:53.318 7f5a136b7700 1 --2- 172.21.2.102:0/1274035445 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5a0c0ac450 0x7f5a0c1c9990 secure :-1 s=READY pgs=3469 cs=0 l=1 rev1=1 rx=0x7f5a00009630 tx=0x7f5a0000b900).stop 2021-02-10T11:41:53.321 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:53.318 7f5a136b7700 1 -- 172.21.2.102:0/1274035445 shutdown_connections 2021-02-10T11:41:53.321 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:53.318 7f5a136b7700 1 --2- 172.21.2.102:0/1274035445 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5a0c09e520 0x7f5a0c09e920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:53.322 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:53.318 7f5a136b7700 1 --2- 172.21.2.102:0/1274035445 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f5a0c0ac450 0x7f5a0c1c9990 unknown :-1 s=CLOSED pgs=3469 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:53.322 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:53.318 7f5a136b7700 1 --2- 172.21.2.102:0/1274035445 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f59ec018730 0x7f59ec01abc0 unknown :-1 s=CLOSED pgs=1509 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:53.322 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:53.318 7f5a136b7700 1 --2- 172.21.2.102:0/1274035445 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5a0c0ac050 0x7f5a0c1c9370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:53.323 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:53.318 7f5a136b7700 1 -- 172.21.2.102:0/1274035445 >> 172.21.2.102:0/1274035445 conn(0x7f5a0c0b8980 msgr2=0x7f5a0c0b34a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:53.323 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:53.318 7f5a136b7700 1 -- 172.21.2.102:0/1274035445 shutdown_connections 2021-02-10T11:41:53.323 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:53.318 7f5a136b7700 1 -- 172.21.2.102:0/1274035445 wait complete. 2021-02-10T11:41:53.329 INFO:tasks.ceph:Scrubbing osd.4 2021-02-10T11:41:53.330 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2021-02-10T11:41:53.765 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:53.746 7f2e2d851700 1 Processor -- start 2021-02-10T11:41:53.765 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:53.746 7f2e2d851700 1 -- start start 2021-02-10T11:41:53.765 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:53.747 7f2e2d851700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f2e280d1c80 0x7f2e280cef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:53.765 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:53.747 7f2e2d851700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f2e280cf590 0x7f2e280cf990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:53.766 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:53.747 7f2e2d851700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e280fe1d0 con 0x7f2e280d1880 2021-02-10T11:41:53.766 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:53.747 7f2e2d851700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e281b0cc0 con 0x7f2e280cf590 2021-02-10T11:41:53.766 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:53.747 7f2e2d851700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e28109a70 con 0x7f2e280d1c80 2021-02-10T11:41:53.766 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:53.747 7f2e27574700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f2e280d1880 0x7f2e281b04d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:34040/0 (socket says 172.21.2.102:34040) 2021-02-10T11:41:53.766 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:53.747 7f2e27574700 1 -- 172.21.2.102:0/2170011409 learned_addr learned my addr 172.21.2.102:0/2170011409 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:41:53.766 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:53.747 7f2e27d75700 1 --2- 172.21.2.102:0/2170011409 >> v2:172.21.2.102:3300/0 conn(0x7f2e280cf590 0x7f2e280cf990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:53.767 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:53.747 7f2e27d75700 1 -- 172.21.2.102:0/2170011409 >> v2:172.21.2.113:3300/0 conn(0x7f2e280d1c80 msgr2=0x7f2e280cef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:41:53.767 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:53.747 7f2e27d75700 1 --2- 172.21.2.102:0/2170011409 >> v2:172.21.2.113:3300/0 conn(0x7f2e280d1c80 0x7f2e280cef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:53.767 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:53.747 7f2e27d75700 1 -- 172.21.2.102:0/2170011409 >> v1:172.21.2.102:6789/0 conn(0x7f2e280d1880 legacy=0x7f2e281b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:53.767 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:53.747 7f2e27d75700 1 -- 172.21.2.102:0/2170011409 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e28088070 con 0x7f2e280cf590 2021-02-10T11:41:53.768 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:53.747 7f2e26572700 1 -- 172.21.2.102:0/2170011409 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3312698784 0 0) 0x7f2e20001500 con 0x7f2e280d1880 2021-02-10T11:41:53.768 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:53.748 7f2e27d75700 1 --2- 172.21.2.102:0/2170011409 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2e280cf590 0x7f2e280cf990 secure :-1 s=READY pgs=3471 cs=0 l=1 rev1=1 rx=0x7f2e18006ab0 tx=0x7f2e18009b40).ready entity=mon.0 client_cookie=5712e9a9590aa769 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:53.768 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:53.748 7f2e26572700 1 -- 172.21.2.102:0/2170011409 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2e180022e0 con 0x7f2e280cf590 2021-02-10T11:41:53.768 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:53.748 7f2e26572700 1 -- 172.21.2.102:0/2170011409 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e18002660 con 0x7f2e280cf590 2021-02-10T11:41:53.769 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:53.748 7f2e26572700 1 -- 172.21.2.102:0/2170011409 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2e18002a70 con 0x7f2e280cf590 2021-02-10T11:41:53.769 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:53.748 7f2e2d851700 1 -- 172.21.2.102:0/2170011409 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2e280cf590 msgr2=0x7f2e280cf990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:53.769 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:53.748 7f2e2d851700 1 --2- 172.21.2.102:0/2170011409 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2e280cf590 0x7f2e280cf990 secure :-1 s=READY pgs=3471 cs=0 l=1 rev1=1 rx=0x7f2e18006ab0 tx=0x7f2e18009b40).stop 2021-02-10T11:41:53.769 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:53.748 7f2e2d851700 1 -- 172.21.2.102:0/2170011409 shutdown_connections 2021-02-10T11:41:53.769 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:53.748 7f2e2d851700 1 --2- 172.21.2.102:0/2170011409 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2e280cf590 0x7f2e280cf990 unknown :-1 s=CLOSED pgs=3471 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:53.769 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:53.748 7f2e2d851700 1 --2- 172.21.2.102:0/2170011409 >> v2:172.21.2.113:3300/0 conn(0x7f2e280d1c80 0x7f2e280cef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:53.769 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:53.748 7f2e2d851700 1 -- 172.21.2.102:0/2170011409 >> 172.21.2.102:0/2170011409 conn(0x7f2e280ce430 msgr2=0x7f2e280d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:53.769 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:53.749 7f2e2d851700 1 -- 172.21.2.102:0/2170011409 shutdown_connections 2021-02-10T11:41:53.770 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:53.749 7f2e2d851700 1 -- 172.21.2.102:0/2170011409 wait complete. 2021-02-10T11:41:53.771 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:53.763 7f2e2d851700 1 Processor -- start 2021-02-10T11:41:53.772 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:53.763 7f2e2d851700 1 -- start start 2021-02-10T11:41:53.772 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:53.763 7f2e2d851700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2e281b58d0 0x7f2e280aa4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:53.772 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:53.763 7f2e2d851700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2e280aaaf0 0x7f2e281b80a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:53.772 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:53.763 7f2e2d851700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2e281b86c0 0x7f2e2819de70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:53.772 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:53.763 7f2e2d851700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e281b8ac0 con 0x7f2e280aaaf0 2021-02-10T11:41:53.772 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:53.763 7f2e2d851700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e281b8e20 con 0x7f2e281b58d0 2021-02-10T11:41:53.772 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:53.763 7f2e2d851700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e281b5cd0 con 0x7f2e281b86c0 2021-02-10T11:41:53.773 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:53.764 7f2e26d73700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2e280aaaf0 0x7f2e281b80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:53.773 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:53.764 7f2e26d73700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2e280aaaf0 0x7f2e281b80a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:41826/0 (socket says 172.21.2.102:41826) 2021-02-10T11:41:53.773 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:53.764 7f2e26d73700 1 -- 172.21.2.102:0/3430663247 learned_addr learned my addr 172.21.2.102:0/3430663247 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:41:53.773 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:53.764 7f2e27574700 1 --2- 172.21.2.102:0/3430663247 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2e281b58d0 0x7f2e280aa4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:53.773 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:53.764 7f2e26d73700 1 -- 172.21.2.102:0/3430663247 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2e281b86c0 msgr2=0x7f2e2819de70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:53.773 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:53.764 7f2e26d73700 1 --2- 172.21.2.102:0/3430663247 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f2e281b86c0 0x7f2e2819de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:53.773 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:53.764 7f2e26d73700 1 -- 172.21.2.102:0/3430663247 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2e281b58d0 msgr2=0x7f2e280aa4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:53.773 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:53.764 7f2e26d73700 1 --2- 172.21.2.102:0/3430663247 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2e281b58d0 0x7f2e280aa4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:53.774 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:53.764 7f2e26d73700 1 -- 172.21.2.102:0/3430663247 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e28095710 con 0x7f2e280aaaf0 2021-02-10T11:41:53.774 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:53.764 7f2e27574700 1 --2- 172.21.2.102:0/3430663247 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f2e281b58d0 0x7f2e280aa4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:41:53.774 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:53.764 7f2e26d73700 1 --2- 172.21.2.102:0/3430663247 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2e280aaaf0 0x7f2e281b80a0 secure :-1 s=READY pgs=3472 cs=0 l=1 rev1=1 rx=0x7f2e1c006f00 tx=0x7f2e1c00b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:53.774 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:53.765 7f2e0ffff700 1 -- 172.21.2.102:0/3430663247 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2e1c00bd30 con 0x7f2e280aaaf0 2021-02-10T11:41:53.774 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:53.765 7f2e0ffff700 1 -- 172.21.2.102:0/3430663247 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e1c000c90 con 0x7f2e280aaaf0 2021-02-10T11:41:53.774 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:53.765 7f2e2d851700 1 -- 172.21.2.102:0/3430663247 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e28095710 con 0x7f2e280aaaf0 2021-02-10T11:41:53.774 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:53.765 7f2e0ffff700 1 -- 172.21.2.102:0/3430663247 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2e1c0022a0 con 0x7f2e280aaaf0 2021-02-10T11:41:53.774 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:53.765 7f2e2d851700 1 -- 172.21.2.102:0/3430663247 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e2819e780 con 0x7f2e280aaaf0 2021-02-10T11:41:53.775 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:53.765 7f2e0ffff700 1 -- 172.21.2.102:0/3430663247 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f2e1c005310 con 0x7f2e280aaaf0 2021-02-10T11:41:53.775 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:53.765 7f2e0ffff700 1 --2- 172.21.2.102:0/3430663247 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2e080186c0 0x7f2e0801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:53.775 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:53.765 7f2e0ffff700 1 -- 172.21.2.102:0/3430663247 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f2e1c032580 con 0x7f2e280aaaf0 2021-02-10T11:41:53.775 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:53.766 7f2e27574700 1 --2- 172.21.2.102:0/3430663247 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2e080186c0 0x7f2e0801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:53.775 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:53.766 7f2e27574700 1 --2- 172.21.2.102:0/3430663247 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2e080186c0 0x7f2e0801abc0 crc :-1 s=READY pgs=1510 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:53.797 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:53.795 7f2e2d851700 1 --2- 172.21.2.102:0/3430663247 >> [v2:172.21.2.113:6808/36368,v1:172.21.2.113:6811/36368] conn(0x7f2e281b8ce0 0x7f2e280627c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:53.798 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:53.795 7f2e2d851700 1 -- 172.21.2.102:0/3430663247 --> [v2:172.21.2.113:6808/36368,v1:172.21.2.113:6811/36368] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2e28082aa0 con 0x7f2e281b8ce0 2021-02-10T11:41:53.798 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:53.796 7f2e27d75700 1 --2- 172.21.2.102:0/3430663247 >> [v2:172.21.2.113:6808/36368,v1:172.21.2.113:6811/36368] conn(0x7f2e281b8ce0 0x7f2e280627c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:53.799 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:53.797 7f2e27d75700 1 --2- 172.21.2.102:0/3430663247 >> [v2:172.21.2.113:6808/36368,v1:172.21.2.113:6811/36368] conn(0x7f2e281b8ce0 0x7f2e280627c0 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:53.800 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:53.798 7f2e0ffff700 1 -- 172.21.2.102:0/3430663247 <== osd.4 v2:172.21.2.113:6808/36368 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+5151 (crc 0 0 0) 0x7f2e18006dd0 con 0x7f2e281b8ce0 2021-02-10T11:41:53.807 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:53.804 7f2e2d851700 1 -- 172.21.2.102:0/3430663247 --> [v2:172.21.2.113:6808/36368,v1:172.21.2.113:6811/36368] -- command(tid 2: {"prefix": "config set", "key": "osd_debug_deep_scrub_sleep", "value": "0"}) v1 -- 0x7f2e28082aa0 con 0x7f2e281b8ce0 2021-02-10T11:41:53.807 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:53.804 7f2e0ffff700 1 -- 172.21.2.102:0/3430663247 <== osd.4 v2:172.21.2.113:6808/36368 2 ==== command_reply(tid 2: 0 Set osd_debug_deep_scrub_sleep to 0) v1 ==== 43+0+0 (crc 0 0 0) 0x7f2e18002ba0 con 0x7f2e281b8ce0 2021-02-10T11:41:53.808 INFO:teuthology.orchestra.run.gibba002.stderr:Set osd_debug_deep_scrub_sleep to 0 2021-02-10T11:41:53.808 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:53.805 7f2e2d851700 1 -- 172.21.2.102:0/3430663247 >> [v2:172.21.2.113:6808/36368,v1:172.21.2.113:6811/36368] conn(0x7f2e281b8ce0 msgr2=0x7f2e280627c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:53.808 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:53.805 7f2e2d851700 1 --2- 172.21.2.102:0/3430663247 >> [v2:172.21.2.113:6808/36368,v1:172.21.2.113:6811/36368] conn(0x7f2e281b8ce0 0x7f2e280627c0 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:53.808 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:53.805 7f2e2d851700 1 -- 172.21.2.102:0/3430663247 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2e080186c0 msgr2=0x7f2e0801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:53.809 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:53.805 7f2e2d851700 1 --2- 172.21.2.102:0/3430663247 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f2e080186c0 0x7f2e0801abc0 crc :-1 s=READY pgs=1510 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:53.809 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:53.805 7f2e2d851700 1 -- 172.21.2.102:0/3430663247 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2e280aaaf0 msgr2=0x7f2e281b80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:53.809 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:53.805 7f2e2d851700 1 --2- 172.21.2.102:0/3430663247 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2e280aaaf0 0x7f2e281b80a0 secure :-1 s=READY pgs=3472 cs=0 l=1 rev1=1 rx=0x7f2e1c006f00 tx=0x7f2e1c00b940).stop 2021-02-10T11:41:53.809 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:53.806 7f2e27d75700 1 -- 172.21.2.102:0/3430663247 reap_dead start 2021-02-10T11:41:53.809 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:53.806 7f2e2d851700 1 -- 172.21.2.102:0/3430663247 shutdown_connections 2021-02-10T11:41:53.809 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:53.806 7f2e2d851700 1 -- 172.21.2.102:0/3430663247 >> 172.21.2.102:0/3430663247 conn(0x7f2e2819f940 msgr2=0x7f2e2818c290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:53.809 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:53.806 7f2e2d851700 1 -- 172.21.2.102:0/3430663247 shutdown_connections 2021-02-10T11:41:53.809 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:53.806 7f2e2d851700 1 -- 172.21.2.102:0/3430663247 wait complete. 2021-02-10T11:41:53.816 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd deep-scrub 4 2021-02-10T11:41:54.247 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:54.228 7f1e09d8c700 1 Processor -- start 2021-02-10T11:41:54.247 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:54.228 7f1e09d8c700 1 -- start start 2021-02-10T11:41:54.247 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:54.229 7f1e09d8c700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f1e040d0340 0x7f1e040ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:54.247 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:54.229 7f1e09d8c700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e040f0b10 con 0x7f1e040d1c70 2021-02-10T11:41:54.247 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:54.229 7f1e09d8c700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e041b0be0 con 0x7f1e040d1870 2021-02-10T11:41:54.248 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:54.229 7f1e09d8c700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e040e5ed0 con 0x7f1e040d0340 2021-02-10T11:41:54.248 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:54.229 7f1e03fff700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f1e040d1c70 0x7f1e040efc60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:34058/0 (socket says 172.21.2.102:34058) 2021-02-10T11:41:54.248 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:54.229 7f1e03fff700 1 -- 172.21.2.102:0/2274778168 learned_addr learned my addr 172.21.2.102:0/2274778168 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:41:54.248 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:54.229 7f1e02ffd700 1 --2- 172.21.2.102:0/2274778168 >> v2:172.21.2.113:3300/0 conn(0x7f1e040d0340 0x7f1e040ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:54.248 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:54.229 7f1e027fc700 1 -- 172.21.2.102:0/2274778168 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1583787019 0 0) 0x7f1dfc001500 con 0x7f1e040d1c70 2021-02-10T11:41:54.248 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:54.229 7f1e027fc700 1 -- 172.21.2.102:0/2274778168 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1df0002fe0 con 0x7f1e040d1c70 2021-02-10T11:41:54.248 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:54.229 7f1e02ffd700 1 -- 172.21.2.102:0/2274778168 >> v1:172.21.2.113:6789/0 conn(0x7f1e040d1870 legacy=0x7f1e041b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:54.249 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:54.229 7f1e027fc700 1 -- 172.21.2.102:0/2274778168 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3751367807 0 0) 0x7f1dfc001460 con 0x7f1e040d1c70 2021-02-10T11:41:54.249 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:54.229 7f1e02ffd700 1 -- 172.21.2.102:0/2274778168 >> v1:172.21.2.102:6789/0 conn(0x7f1e040d1c70 legacy=0x7f1e040efc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:54.249 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:54.229 7f1e02ffd700 1 -- 172.21.2.102:0/2274778168 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e040f0450 con 0x7f1e040d0340 2021-02-10T11:41:54.249 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:54.229 7f1e02ffd700 1 --2- 172.21.2.102:0/2274778168 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1e040d0340 0x7f1e040ef640 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 rx=0x7f1dec006650 tx=0x7f1dec009a80).ready entity=mon.1 client_cookie=a86f9efe4ea2a962 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:54.249 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:54.230 7f1e027fc700 1 -- 172.21.2.102:0/2274778168 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1dec00c0c0 con 0x7f1e040d0340 2021-02-10T11:41:54.249 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:54.230 7f1e027fc700 1 -- 172.21.2.102:0/2274778168 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1dec00c340 con 0x7f1e040d0340 2021-02-10T11:41:54.249 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:54.230 7f1e027fc700 1 -- 172.21.2.102:0/2274778168 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1dec00c860 con 0x7f1e040d0340 2021-02-10T11:41:54.249 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:54.230 7f1e09d8c700 1 -- 172.21.2.102:0/2274778168 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1e040d0340 msgr2=0x7f1e040ef640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:54.249 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:54.230 7f1e09d8c700 1 --2- 172.21.2.102:0/2274778168 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1e040d0340 0x7f1e040ef640 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 rx=0x7f1dec006650 tx=0x7f1dec009a80).stop 2021-02-10T11:41:54.250 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:54.230 7f1e09d8c700 1 -- 172.21.2.102:0/2274778168 shutdown_connections 2021-02-10T11:41:54.250 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:54.230 7f1e09d8c700 1 --2- 172.21.2.102:0/2274778168 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1e040d0340 0x7f1e040ef640 unknown :-1 s=CLOSED pgs=919 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:54.250 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:54.230 7f1e09d8c700 1 -- 172.21.2.102:0/2274778168 >> 172.21.2.102:0/2274778168 conn(0x7f1e040ce450 msgr2=0x7f1e040d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:54.250 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:54.230 7f1e09d8c700 1 -- 172.21.2.102:0/2274778168 shutdown_connections 2021-02-10T11:41:54.250 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:54.230 7f1e09d8c700 1 -- 172.21.2.102:0/2274778168 wait complete. 2021-02-10T11:41:54.252 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:54.245 7f1e09d8c700 1 Processor -- start 2021-02-10T11:41:54.252 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:54.245 7f1e09d8c700 1 -- start start 2021-02-10T11:41:54.252 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:54.245 7f1e09d8c700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1e040a2fb0 0x7f1e040a33b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:54.252 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:54.245 7f1e09d8c700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1e040a39d0 0x7f1e040a4860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:54.253 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:54.245 7f1e09d8c700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1e040a3dd0 0x7f1e040a4e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:54.253 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:54.245 7f1e09d8c700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e040a2620 con 0x7f1e040a39d0 2021-02-10T11:41:54.253 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:54.245 7f1e09d8c700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e040a2840 con 0x7f1e040a3dd0 2021-02-10T11:41:54.253 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:54.245 7f1e09d8c700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e040a2b80 con 0x7f1e040a2fb0 2021-02-10T11:41:54.253 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:54.246 7f1e02ffd700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1e040a39d0 0x7f1e040a4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:54.253 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:54.246 7f1e02ffd700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1e040a39d0 0x7f1e040a4860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:41838/0 (socket says 172.21.2.102:41838) 2021-02-10T11:41:54.253 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:54.246 7f1e02ffd700 1 -- 172.21.2.102:0/2715772619 learned_addr learned my addr 172.21.2.102:0/2715772619 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:41:54.254 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:54.246 7f1e02ffd700 1 -- 172.21.2.102:0/2715772619 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1e040a2fb0 msgr2=0x7f1e040a33b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:41:54.254 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:54.246 7f1e02ffd700 1 --2- 172.21.2.102:0/2715772619 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1e040a2fb0 0x7f1e040a33b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:54.254 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:54.246 7f1e02ffd700 1 -- 172.21.2.102:0/2715772619 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1e040a3dd0 msgr2=0x7f1e040a4e80 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:41:54.254 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:54.246 7f1e02ffd700 1 --2- 172.21.2.102:0/2715772619 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1e040a3dd0 0x7f1e040a4e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:54.254 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:54.246 7f1e02ffd700 1 -- 172.21.2.102:0/2715772619 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e040a54a0 con 0x7f1e040a39d0 2021-02-10T11:41:54.254 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:54.246 7f1e02ffd700 1 --2- 172.21.2.102:0/2715772619 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1e040a39d0 0x7f1e040a4860 secure :-1 s=READY pgs=3474 cs=0 l=1 rev1=1 rx=0x7f1dec009650 tx=0x7f1dec00b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:54.255 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:54.246 7f1dfbfff700 1 -- 172.21.2.102:0/2715772619 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1dec00bd50 con 0x7f1e040a39d0 2021-02-10T11:41:54.255 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:54.246 7f1dfbfff700 1 -- 172.21.2.102:0/2715772619 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1dec006030 con 0x7f1e040a39d0 2021-02-10T11:41:54.255 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:54.246 7f1dfbfff700 1 -- 172.21.2.102:0/2715772619 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1dec005560 con 0x7f1e040a39d0 2021-02-10T11:41:54.255 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:54.246 7f1e09d8c700 1 -- 172.21.2.102:0/2715772619 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e040a54a0 con 0x7f1e040a39d0 2021-02-10T11:41:54.255 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:54.246 7f1e09d8c700 1 -- 172.21.2.102:0/2715772619 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e040a5a30 con 0x7f1e040a39d0 2021-02-10T11:41:54.256 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:54.247 7f1dfbfff700 1 -- 172.21.2.102:0/2715772619 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1dec0052e0 con 0x7f1e040a39d0 2021-02-10T11:41:54.256 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:54.247 7f1dfbfff700 1 --2- 172.21.2.102:0/2715772619 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1de8018730 0x7f1de801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:54.256 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:54.247 7f1dfbfff700 1 -- 172.21.2.102:0/2715772619 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f1dec032640 con 0x7f1e040a39d0 2021-02-10T11:41:54.256 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:54.247 7f1e037fe700 1 --2- 172.21.2.102:0/2715772619 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1de8018730 0x7f1de801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:54.256 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:54.247 7f1e037fe700 1 --2- 172.21.2.102:0/2715772619 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1de8018730 0x7f1de801abc0 crc :-1 s=READY pgs=1511 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:54.281 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:54.278 7f1e09d8c700 1 -- 172.21.2.102:0/2715772619 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e040ebc50 con 0x7f1e040a39d0 2021-02-10T11:41:54.283 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:54.281 7f1dfbfff700 1 -- 172.21.2.102:0/2715772619 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f1dec009360 con 0x7f1e040a39d0 2021-02-10T11:41:54.413 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:54.410 7f1e09d8c700 1 -- 172.21.2.102:0/2715772619 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mgr", ""]}) v1 -- 0x7f1e040a5a30 con 0x7f1de8018730 2021-02-10T11:41:54.413 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:54.410 7f1dfbfff700 1 -- 172.21.2.102:0/2715772619 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) v1 ==== 41+0+0 (crc 0 0 0) 0x7f1df4000960 con 0x7f1de8018730 2021-02-10T11:41:54.414 INFO:teuthology.orchestra.run.gibba002.stderr:instructed osd(s) 4 to deep-scrub 2021-02-10T11:41:54.415 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:54.412 7f1e09d8c700 1 -- 172.21.2.102:0/2715772619 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1de8018730 msgr2=0x7f1de801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:54.415 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:54.412 7f1e09d8c700 1 --2- 172.21.2.102:0/2715772619 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1de8018730 0x7f1de801abc0 crc :-1 s=READY pgs=1511 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:54.415 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:54.412 7f1e09d8c700 1 -- 172.21.2.102:0/2715772619 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1e040a39d0 msgr2=0x7f1e040a4860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:54.416 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:54.412 7f1e09d8c700 1 --2- 172.21.2.102:0/2715772619 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1e040a39d0 0x7f1e040a4860 secure :-1 s=READY pgs=3474 cs=0 l=1 rev1=1 rx=0x7f1dec009650 tx=0x7f1dec00b960).stop 2021-02-10T11:41:54.416 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:54.412 7f1e09d8c700 1 -- 172.21.2.102:0/2715772619 shutdown_connections 2021-02-10T11:41:54.416 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:54.412 7f1e09d8c700 1 --2- 172.21.2.102:0/2715772619 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f1de8018730 0x7f1de801abc0 unknown :-1 s=CLOSED pgs=1511 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:54.416 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:54.412 7f1e09d8c700 1 --2- 172.21.2.102:0/2715772619 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f1e040a3dd0 0x7f1e040a4e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:54.416 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:54.412 7f1e09d8c700 1 --2- 172.21.2.102:0/2715772619 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f1e040a39d0 0x7f1e040a4860 unknown :-1 s=CLOSED pgs=3474 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:54.416 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:54.412 7f1e09d8c700 1 --2- 172.21.2.102:0/2715772619 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f1e040a2fb0 0x7f1e040a33b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:54.417 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:54.412 7f1e09d8c700 1 -- 172.21.2.102:0/2715772619 >> 172.21.2.102:0/2715772619 conn(0x7f1e0408f030 msgr2=0x7f1e040b5e80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:54.417 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:54.412 7f1e09d8c700 1 -- 172.21.2.102:0/2715772619 shutdown_connections 2021-02-10T11:41:54.417 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:54.412 7f1e09d8c700 1 -- 172.21.2.102:0/2715772619 wait complete. 2021-02-10T11:41:54.423 INFO:tasks.ceph:Scrubbing osd.5 2021-02-10T11:41:54.424 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2021-02-10T11:41:54.859 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:54.839 7f6b2633f700 1 Processor -- start 2021-02-10T11:41:54.859 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:54.839 7f6b2633f700 1 -- start start 2021-02-10T11:41:54.860 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:54.840 7f6b2633f700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f6b200d1880 0x7f6b201b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:54.860 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:54.840 7f6b2633f700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6b200cff80 0x7f6b20109140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:54.860 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:54.840 7f6b2633f700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b20109a70 con 0x7f6b201b0af0 2021-02-10T11:41:54.860 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:54.840 7f6b2633f700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b20109760 con 0x7f6b200d1880 2021-02-10T11:41:54.860 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:54.840 7f6b2633f700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b20088070 con 0x7f6b200cff80 2021-02-10T11:41:54.860 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:54.840 7f6b1ffff700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f6b200d1880 0x7f6b201b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:54.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:54.840 7f6b1ffff700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f6b200d1880 0x7f6b201b04d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.113:3300/0 says I am v2:172.21.2.102:45274/0 (socket says 172.21.2.102:45274) 2021-02-10T11:41:54.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:54.840 7f6b1ffff700 1 -- 172.21.2.102:0/383801884 learned_addr learned my addr 172.21.2.102:0/383801884 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:41:54.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:54.840 7f6b1f7fe700 1 --1- 172.21.2.102:0/383801884 >> v1:172.21.2.102:6789/0 conn(0x7f6b201b0af0 0x7f6b20108950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:34072/0 (socket says 172.21.2.102:34072) 2021-02-10T11:41:54.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:54.840 7f6b2495c700 1 --2- 172.21.2.102:0/383801884 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6b200cff80 0x7f6b20109140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:54.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:54.840 7f6b1ffff700 1 -- 172.21.2.102:0/383801884 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6b200cff80 msgr2=0x7f6b20109140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:41:54.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:54.840 7f6b1ffff700 1 --2- 172.21.2.102:0/383801884 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6b200cff80 0x7f6b20109140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:54.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:54.840 7f6b1ffff700 1 -- 172.21.2.102:0/383801884 >> v1:172.21.2.102:6789/0 conn(0x7f6b201b0af0 legacy=0x7f6b20108950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:54.861 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:54.840 7f6b1ffff700 1 -- 172.21.2.102:0/383801884 --> v2:172.21.2.113:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b200d1c80 con 0x7f6b200d1880 2021-02-10T11:41:54.862 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:54.840 7f6b1ffff700 1 --2- 172.21.2.102:0/383801884 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6b200d1880 0x7f6b201b04d0 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 rx=0x7f6b10006700 tx=0x7f6b10009b80).ready entity=mon.1 client_cookie=3a6019828cd5e666 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:54.862 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:54.841 7f6b1effd700 1 -- 172.21.2.102:0/383801884 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6b1000c0c0 con 0x7f6b200d1880 2021-02-10T11:41:54.862 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:54.841 7f6b1effd700 1 -- 172.21.2.102:0/383801884 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b1000c300 con 0x7f6b200d1880 2021-02-10T11:41:54.862 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:54.841 7f6b1effd700 1 -- 172.21.2.102:0/383801884 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6b1000c840 con 0x7f6b200d1880 2021-02-10T11:41:54.862 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:54.841 7f6b2633f700 1 -- 172.21.2.102:0/383801884 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6b200d1880 msgr2=0x7f6b201b04d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:54.862 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:54.841 7f6b2633f700 1 --2- 172.21.2.102:0/383801884 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6b200d1880 0x7f6b201b04d0 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 rx=0x7f6b10006700 tx=0x7f6b10009b80).stop 2021-02-10T11:41:54.862 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:54.841 7f6b2633f700 1 -- 172.21.2.102:0/383801884 shutdown_connections 2021-02-10T11:41:54.862 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:54.841 7f6b2633f700 1 --2- 172.21.2.102:0/383801884 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6b200cff80 0x7f6b20109140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:54.862 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:54.841 7f6b2633f700 1 --2- 172.21.2.102:0/383801884 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6b200d1880 0x7f6b201b04d0 unknown :-1 s=CLOSED pgs=920 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:54.863 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:54.841 7f6b2633f700 1 -- 172.21.2.102:0/383801884 >> 172.21.2.102:0/383801884 conn(0x7f6b200ce430 msgr2=0x7f6b200d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:54.863 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:54.842 7f6b2633f700 1 -- 172.21.2.102:0/383801884 shutdown_connections 2021-02-10T11:41:54.863 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:54.842 7f6b2633f700 1 -- 172.21.2.102:0/383801884 wait complete. 2021-02-10T11:41:54.863 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:54.856 7f6b2633f700 1 Processor -- start 2021-02-10T11:41:54.863 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:54.856 7f6b2633f700 1 -- start start 2021-02-10T11:41:54.864 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:54.857 7f6b2633f700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6b201a3570 0x7f6b201a3970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:54.864 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:54.857 7f6b2633f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6b2009c9f0 0x7f6b2009cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:54.865 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:54.857 7f6b2633f700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6b2009d410 0x7f6b20098170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:54.865 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:54.857 7f6b2633f700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b2009c470 con 0x7f6b2009c9f0 2021-02-10T11:41:54.865 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:54.857 7f6b2633f700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b200b2840 con 0x7f6b201a3570 2021-02-10T11:41:54.865 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:54.857 7f6b2633f700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b200b2a20 con 0x7f6b2009d410 2021-02-10T11:41:54.865 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:54.857 7f6b1f7fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6b2009c9f0 0x7f6b2009cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:54.865 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:54.857 7f6b1f7fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6b2009c9f0 0x7f6b2009cdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:41850/0 (socket says 172.21.2.102:41850) 2021-02-10T11:41:54.865 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:54.857 7f6b1f7fe700 1 -- 172.21.2.102:0/994876450 learned_addr learned my addr 172.21.2.102:0/994876450 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:41:54.865 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:54.857 7f6b2495c700 1 --2- 172.21.2.102:0/994876450 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6b2009d410 0x7f6b20098170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:54.866 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:54.857 7f6b1ffff700 1 --2- 172.21.2.102:0/994876450 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6b201a3570 0x7f6b201a3970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:54.866 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:54.857 7f6b1f7fe700 1 -- 172.21.2.102:0/994876450 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6b2009d410 msgr2=0x7f6b20098170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:54.866 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:54.857 7f6b1f7fe700 1 --2- 172.21.2.102:0/994876450 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f6b2009d410 0x7f6b20098170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:54.866 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:54.857 7f6b1f7fe700 1 -- 172.21.2.102:0/994876450 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6b201a3570 msgr2=0x7f6b201a3970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:54.866 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:54.857 7f6b1f7fe700 1 --2- 172.21.2.102:0/994876450 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f6b201a3570 0x7f6b201a3970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:54.866 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:54.857 7f6b1f7fe700 1 -- 172.21.2.102:0/994876450 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b2009d880 con 0x7f6b2009c9f0 2021-02-10T11:41:54.866 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:54.857 7f6b1f7fe700 1 --2- 172.21.2.102:0/994876450 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6b2009c9f0 0x7f6b2009cdf0 secure :-1 s=READY pgs=3476 cs=0 l=1 rev1=1 rx=0x7f6b1400a4c0 tx=0x7f6b1400a860).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:54.866 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:54.858 7f6b1cff9700 1 -- 172.21.2.102:0/994876450 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6b1400ac50 con 0x7f6b2009c9f0 2021-02-10T11:41:54.867 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:54.858 7f6b1cff9700 1 -- 172.21.2.102:0/994876450 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b14004140 con 0x7f6b2009c9f0 2021-02-10T11:41:54.867 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:54.858 7f6b2633f700 1 -- 172.21.2.102:0/994876450 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b2009d880 con 0x7f6b2009c9f0 2021-02-10T11:41:54.867 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:54.858 7f6b2633f700 1 -- 172.21.2.102:0/994876450 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b20098a80 con 0x7f6b2009c9f0 2021-02-10T11:41:54.867 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:54.858 7f6b1cff9700 1 -- 172.21.2.102:0/994876450 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6b14004660 con 0x7f6b2009c9f0 2021-02-10T11:41:54.867 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:54.858 7f6b1cff9700 1 -- 172.21.2.102:0/994876450 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f6b14004320 con 0x7f6b2009c9f0 2021-02-10T11:41:54.867 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:54.858 7f6b1cff9700 1 --2- 172.21.2.102:0/994876450 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6b000186c0 0x7f6b0001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:54.867 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:54.858 7f6b1cff9700 1 -- 172.21.2.102:0/994876450 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f6b14032520 con 0x7f6b2009c9f0 2021-02-10T11:41:54.868 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:54.859 7f6b1ffff700 1 --2- 172.21.2.102:0/994876450 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6b000186c0 0x7f6b0001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:54.868 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:54.859 7f6b1ffff700 1 --2- 172.21.2.102:0/994876450 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6b000186c0 0x7f6b0001abc0 crc :-1 s=READY pgs=1512 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:54.891 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:54.889 7f6b2633f700 1 --2- 172.21.2.102:0/994876450 >> [v2:172.21.2.113:6802/36366,v1:172.21.2.113:6805/36366] conn(0x7f6b200c4620 0x7f6b20099310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:54.892 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:54.889 7f6b2633f700 1 -- 172.21.2.102:0/994876450 --> [v2:172.21.2.113:6802/36366,v1:172.21.2.113:6805/36366] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6b200b2840 con 0x7f6b200c4620 2021-02-10T11:41:54.892 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:54.889 7f6b2495c700 1 --2- 172.21.2.102:0/994876450 >> [v2:172.21.2.113:6802/36366,v1:172.21.2.113:6805/36366] conn(0x7f6b200c4620 0x7f6b20099310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:54.893 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:54.891 7f6b2495c700 1 --2- 172.21.2.102:0/994876450 >> [v2:172.21.2.113:6802/36366,v1:172.21.2.113:6805/36366] conn(0x7f6b200c4620 0x7f6b20099310 crc :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:54.894 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:54.892 7f6b1cff9700 1 -- 172.21.2.102:0/994876450 <== osd.5 v2:172.21.2.113:6802/36366 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+5151 (crc 0 0 0) 0x7f6b18006a10 con 0x7f6b200c4620 2021-02-10T11:41:54.900 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:54.898 7f6b2633f700 1 -- 172.21.2.102:0/994876450 --> [v2:172.21.2.113:6802/36366,v1:172.21.2.113:6805/36366] -- command(tid 2: {"prefix": "config set", "key": "osd_debug_deep_scrub_sleep", "value": "0"}) v1 -- 0x7f6b20098a80 con 0x7f6b200c4620 2021-02-10T11:41:54.901 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:54.898 7f6b1cff9700 1 -- 172.21.2.102:0/994876450 <== osd.5 v2:172.21.2.113:6802/36366 2 ==== command_reply(tid 2: 0 Set osd_debug_deep_scrub_sleep to 0) v1 ==== 43+0+0 (crc 0 0 0) 0x7f6b18006a10 con 0x7f6b200c4620 2021-02-10T11:41:54.902 INFO:teuthology.orchestra.run.gibba002.stderr:Set osd_debug_deep_scrub_sleep to 0 2021-02-10T11:41:54.902 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:54.899 7f6b2633f700 1 -- 172.21.2.102:0/994876450 >> [v2:172.21.2.113:6802/36366,v1:172.21.2.113:6805/36366] conn(0x7f6b200c4620 msgr2=0x7f6b20099310 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:54.902 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:54.899 7f6b2633f700 1 --2- 172.21.2.102:0/994876450 >> [v2:172.21.2.113:6802/36366,v1:172.21.2.113:6805/36366] conn(0x7f6b200c4620 0x7f6b20099310 crc :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:54.903 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:54.899 7f6b2633f700 1 -- 172.21.2.102:0/994876450 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6b000186c0 msgr2=0x7f6b0001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:54.903 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:54.899 7f6b2633f700 1 --2- 172.21.2.102:0/994876450 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f6b000186c0 0x7f6b0001abc0 crc :-1 s=READY pgs=1512 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:54.903 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:54.899 7f6b2633f700 1 -- 172.21.2.102:0/994876450 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6b2009c9f0 msgr2=0x7f6b2009cdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:54.903 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:54.899 7f6b2633f700 1 --2- 172.21.2.102:0/994876450 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f6b2009c9f0 0x7f6b2009cdf0 secure :-1 s=READY pgs=3476 cs=0 l=1 rev1=1 rx=0x7f6b1400a4c0 tx=0x7f6b1400a860).stop 2021-02-10T11:41:54.904 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:54.899 7f6b2495c700 1 -- 172.21.2.102:0/994876450 reap_dead start 2021-02-10T11:41:54.904 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:54.899 7f6b2633f700 1 -- 172.21.2.102:0/994876450 shutdown_connections 2021-02-10T11:41:54.904 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:54.899 7f6b2633f700 1 -- 172.21.2.102:0/994876450 >> 172.21.2.102:0/994876450 conn(0x7f6b200c2cd0 msgr2=0x7f6b200b5950 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:54.904 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:54.900 7f6b2633f700 1 -- 172.21.2.102:0/994876450 shutdown_connections 2021-02-10T11:41:54.904 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:54.900 7f6b2633f700 1 -- 172.21.2.102:0/994876450 wait complete. 2021-02-10T11:41:54.910 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd deep-scrub 5 2021-02-10T11:41:55.340 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:55.321 7fb0e393d700 1 Processor -- start 2021-02-10T11:41:55.340 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:55.322 7fb0e393d700 1 -- start start 2021-02-10T11:41:55.340 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:55.322 7fb0e393d700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb0dc0d0330 0x7fb0dc0ef630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:55.340 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:55.322 7fb0e393d700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fb0dc0d1c60 0x7fb0dc0efc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:55.340 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:55.322 7fb0e393d700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0dc0f0b00 con 0x7fb0dc0d1860 2021-02-10T11:41:55.341 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:55.322 7fb0e393d700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0dc0e5ec0 con 0x7fb0dc0d1c60 2021-02-10T11:41:55.341 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:55.322 7fb0e393d700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0dc1b0bd0 con 0x7fb0dc0d0330 2021-02-10T11:41:55.341 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:55.322 7fb0e1f5a700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fb0dc0d1c60 0x7fb0dc0efc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:55.341 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:55.322 7fb0e1f5a700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7fb0dc0d1c60 0x7fb0dc0efc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:41864/0 (socket says 172.21.2.102:41864) 2021-02-10T11:41:55.341 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:55.322 7fb0e1f5a700 1 -- 172.21.2.102:0/4001968279 learned_addr learned my addr 172.21.2.102:0/4001968279 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:41:55.341 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:55.322 7fb0e1f5a700 1 -- 172.21.2.102:0/4001968279 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb0dc0d0330 msgr2=0x7fb0dc0ef630 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:41:55.341 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:55.322 7fb0e1f5a700 1 --2- 172.21.2.102:0/4001968279 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb0dc0d0330 0x7fb0dc0ef630 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:55.342 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:55.322 7fb0e1f5a700 1 -- 172.21.2.102:0/4001968279 >> v1:172.21.2.113:6789/0 conn(0x7fb0dc0d1860 legacy=0x7fb0dc1b03e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:41:55.342 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:55.322 7fb0e1f5a700 1 -- 172.21.2.102:0/4001968279 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0dc0f2090 con 0x7fb0dc0d1c60 2021-02-10T11:41:55.342 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:55.323 7fb0e1f5a700 1 --2- 172.21.2.102:0/4001968279 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb0dc0d1c60 0x7fb0dc0efc50 secure :-1 s=READY pgs=3477 cs=0 l=1 rev1=1 rx=0x7fb0d4006f00 tx=0x7fb0d4009f10).ready entity=mon.0 client_cookie=78e77cff67e2a49e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:55.342 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:55.323 7fb0cffff700 1 -- 172.21.2.102:0/4001968279 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb0d400b3b0 con 0x7fb0dc0d1c60 2021-02-10T11:41:55.342 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:55.323 7fb0cffff700 1 -- 172.21.2.102:0/4001968279 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0d400b630 con 0x7fb0dc0d1c60 2021-02-10T11:41:55.342 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:55.323 7fb0cffff700 1 -- 172.21.2.102:0/4001968279 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb0d400bb50 con 0x7fb0dc0d1c60 2021-02-10T11:41:55.342 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:55.323 7fb0e393d700 1 -- 172.21.2.102:0/4001968279 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb0dc0d1c60 msgr2=0x7fb0dc0efc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:55.342 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:55.323 7fb0e393d700 1 --2- 172.21.2.102:0/4001968279 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb0dc0d1c60 0x7fb0dc0efc50 secure :-1 s=READY pgs=3477 cs=0 l=1 rev1=1 rx=0x7fb0d4006f00 tx=0x7fb0d4009f10).stop 2021-02-10T11:41:55.343 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:55.323 7fb0e393d700 1 -- 172.21.2.102:0/4001968279 shutdown_connections 2021-02-10T11:41:55.343 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:55.323 7fb0e393d700 1 --2- 172.21.2.102:0/4001968279 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb0dc0d0330 0x7fb0dc0ef630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:55.343 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:55.323 7fb0e393d700 1 --2- 172.21.2.102:0/4001968279 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb0dc0d1c60 0x7fb0dc0efc50 unknown :-1 s=CLOSED pgs=3477 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:55.343 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:55.323 7fb0e393d700 1 -- 172.21.2.102:0/4001968279 >> 172.21.2.102:0/4001968279 conn(0x7fb0dc0ce440 msgr2=0x7fb0dc0d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:55.343 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:55.324 7fb0e393d700 1 -- 172.21.2.102:0/4001968279 shutdown_connections 2021-02-10T11:41:55.343 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:55.324 7fb0e393d700 1 -- 172.21.2.102:0/4001968279 wait complete. 2021-02-10T11:41:55.344 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:55.338 7fb0e393d700 1 Processor -- start 2021-02-10T11:41:55.345 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:55.338 7fb0e393d700 1 -- start start 2021-02-10T11:41:55.345 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:55.338 7fb0e393d700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb0dc1a30e0 0x7fb0dc1a34e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:55.345 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:55.338 7fb0e393d700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb0dc1a3b00 0x7fb0dc0c16c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:55.345 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:55.338 7fb0e393d700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb0dc0c1ce0 0x7fb0dc0c20e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:55.345 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:55.338 7fb0e393d700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0dc097160 con 0x7fb0dc1a3b00 2021-02-10T11:41:55.345 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:55.338 7fb0e393d700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0dc0c2700 con 0x7fb0dc1a30e0 2021-02-10T11:41:55.345 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:55.338 7fb0e393d700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0dc0c2920 con 0x7fb0dc0c1ce0 2021-02-10T11:41:55.345 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:55.339 7fb0e0f58700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb0dc1a3b00 0x7fb0dc0c16c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:55.346 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:55.339 7fb0e0f58700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb0dc1a3b00 0x7fb0dc0c16c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:41868/0 (socket says 172.21.2.102:41868) 2021-02-10T11:41:55.346 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:55.339 7fb0e0f58700 1 -- 172.21.2.102:0/1816076112 learned_addr learned my addr 172.21.2.102:0/1816076112 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:41:55.346 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:55.339 7fb0e0f58700 1 -- 172.21.2.102:0/1816076112 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb0dc0c1ce0 msgr2=0x7fb0dc0c20e0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:41:55.346 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:55.339 7fb0e0f58700 1 --2- 172.21.2.102:0/1816076112 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb0dc0c1ce0 0x7fb0dc0c20e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:55.346 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:55.339 7fb0e0f58700 1 -- 172.21.2.102:0/1816076112 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb0dc1a30e0 msgr2=0x7fb0dc1a34e0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:41:55.346 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:55.339 7fb0e0f58700 1 --2- 172.21.2.102:0/1816076112 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb0dc1a30e0 0x7fb0dc1a34e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:55.346 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:55.339 7fb0e0f58700 1 -- 172.21.2.102:0/1816076112 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0dc0bfa10 con 0x7fb0dc1a3b00 2021-02-10T11:41:55.347 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:55.339 7fb0e0f58700 1 --2- 172.21.2.102:0/1816076112 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb0dc1a3b00 0x7fb0dc0c16c0 secure :-1 s=READY pgs=3478 cs=0 l=1 rev1=1 rx=0x7fb0d000b5c0 tx=0x7fb0d000b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:55.347 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:55.339 7fb0cdffb700 1 -- 172.21.2.102:0/1816076112 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb0d000bd50 con 0x7fb0dc1a3b00 2021-02-10T11:41:55.347 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:55.339 7fb0cdffb700 1 -- 172.21.2.102:0/1816076112 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0d0006030 con 0x7fb0dc1a3b00 2021-02-10T11:41:55.347 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:55.339 7fb0e393d700 1 -- 172.21.2.102:0/1816076112 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0dc0bfa10 con 0x7fb0dc1a3b00 2021-02-10T11:41:55.347 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:55.339 7fb0e393d700 1 -- 172.21.2.102:0/1816076112 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0dc0bfa10 con 0x7fb0dc1a3b00 2021-02-10T11:41:55.347 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:55.339 7fb0cdffb700 1 -- 172.21.2.102:0/1816076112 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb0d00054d0 con 0x7fb0dc1a3b00 2021-02-10T11:41:55.347 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:55.340 7fb0cdffb700 1 -- 172.21.2.102:0/1816076112 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fb0d0006210 con 0x7fb0dc1a3b00 2021-02-10T11:41:55.347 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:55.340 7fb0cdffb700 1 --2- 172.21.2.102:0/1816076112 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb0bc0186d0 0x7fb0bc01abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:55.348 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:55.340 7fb0cdffb700 1 -- 172.21.2.102:0/1816076112 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fb0d00327d0 con 0x7fb0dc1a3b00 2021-02-10T11:41:55.348 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:55.340 7fb0e1759700 1 --2- 172.21.2.102:0/1816076112 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb0bc0186d0 0x7fb0bc01abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:55.348 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:55.340 7fb0e1759700 1 --2- 172.21.2.102:0/1816076112 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb0bc0186d0 0x7fb0bc01abd0 crc :-1 s=READY pgs=1513 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:55.374 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:55.371 7fb0e393d700 1 -- 172.21.2.102:0/1816076112 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0dc0c1190 con 0x7fb0dc1a3b00 2021-02-10T11:41:55.376 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:55.374 7fb0cdffb700 1 -- 172.21.2.102:0/1816076112 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fb0d0006b80 con 0x7fb0dc1a3b00 2021-02-10T11:41:55.504 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:55.502 7fb0e393d700 1 -- 172.21.2.102:0/1816076112 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mgr", ""]}) v1 -- 0x7fb0dc0c1190 con 0x7fb0bc0186d0 2021-02-10T11:41:55.505 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:55.502 7fb0cdffb700 1 -- 172.21.2.102:0/1816076112 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) v1 ==== 41+0+0 (crc 0 0 0) 0x7fb0d8000c60 con 0x7fb0bc0186d0 2021-02-10T11:41:55.505 INFO:teuthology.orchestra.run.gibba002.stderr:instructed osd(s) 5 to deep-scrub 2021-02-10T11:41:55.506 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:55.504 7fb0e393d700 1 -- 172.21.2.102:0/1816076112 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb0bc0186d0 msgr2=0x7fb0bc01abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:55.507 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:55.504 7fb0e393d700 1 --2- 172.21.2.102:0/1816076112 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb0bc0186d0 0x7fb0bc01abd0 crc :-1 s=READY pgs=1513 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:55.507 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:55.504 7fb0e393d700 1 -- 172.21.2.102:0/1816076112 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb0dc1a3b00 msgr2=0x7fb0dc0c16c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:55.507 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:55.504 7fb0e393d700 1 --2- 172.21.2.102:0/1816076112 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb0dc1a3b00 0x7fb0dc0c16c0 secure :-1 s=READY pgs=3478 cs=0 l=1 rev1=1 rx=0x7fb0d000b5c0 tx=0x7fb0d000b960).stop 2021-02-10T11:41:55.507 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:55.505 7fb0e393d700 1 -- 172.21.2.102:0/1816076112 shutdown_connections 2021-02-10T11:41:55.508 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:55.505 7fb0e393d700 1 --2- 172.21.2.102:0/1816076112 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fb0dc0c1ce0 0x7fb0dc0c20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:55.508 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:55.505 7fb0e393d700 1 --2- 172.21.2.102:0/1816076112 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fb0dc1a3b00 0x7fb0dc0c16c0 unknown :-1 s=CLOSED pgs=3478 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:55.508 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:55.505 7fb0e393d700 1 --2- 172.21.2.102:0/1816076112 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fb0bc0186d0 0x7fb0bc01abd0 unknown :-1 s=CLOSED pgs=1513 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:55.508 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:55.505 7fb0e393d700 1 --2- 172.21.2.102:0/1816076112 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fb0dc1a30e0 0x7fb0dc1a34e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:55.508 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:55.505 7fb0e393d700 1 -- 172.21.2.102:0/1816076112 >> 172.21.2.102:0/1816076112 conn(0x7fb0dc09fd30 msgr2=0x7fb0dc09a3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:55.508 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:55.505 7fb0e393d700 1 -- 172.21.2.102:0/1816076112 shutdown_connections 2021-02-10T11:41:55.508 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:55.505 7fb0e393d700 1 -- 172.21.2.102:0/1816076112 wait complete. 2021-02-10T11:41:55.516 INFO:tasks.ceph:Scrubbing osd.6 2021-02-10T11:41:55.516 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2021-02-10T11:41:55.947 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:55.929 7f7cda172700 1 Processor -- start 2021-02-10T11:41:55.948 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:55.929 7f7cda172700 1 -- start start 2021-02-10T11:41:55.948 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:55.930 7f7cda172700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f7cd40d1880 0x7f7cd41b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:55.948 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:55.930 7f7cda172700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7cd41b0af0 0x7f7cd4108950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:55.948 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:55.930 7f7cda172700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f7cd40d1c80 0x7f7cd4108f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:55.948 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:55.930 7f7cda172700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7cd40fe1d0 con 0x7f7cd40d1880 2021-02-10T11:41:55.948 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:55.930 7f7cda172700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7cd4109a70 con 0x7f7cd40d1c80 2021-02-10T11:41:55.948 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:55.930 7f7cda172700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7cd4088070 con 0x7f7cd41b0af0 2021-02-10T11:41:55.949 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:55.930 7f7cd37fe700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f7cd40d1880 0x7f7cd41b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:55.949 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:55.930 7f7cd37fe700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f7cd40d1880 0x7f7cd41b04d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:41874/0 (socket says 172.21.2.102:41874) 2021-02-10T11:41:55.949 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:55.930 7f7cd37fe700 1 -- 172.21.2.102:0/3794956660 learned_addr learned my addr 172.21.2.102:0/3794956660 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:41:55.949 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:55.930 7f7cd37fe700 1 -- 172.21.2.102:0/3794956660 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7cd41b0af0 msgr2=0x7f7cd4108950 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:41:55.949 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:55.930 7f7cd37fe700 1 --2- 172.21.2.102:0/3794956660 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7cd41b0af0 0x7f7cd4108950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:55.949 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:55.930 7f7cd37fe700 1 -- 172.21.2.102:0/3794956660 >> v2:172.21.2.113:3300/0 conn(0x7f7cd40d1c80 msgr2=0x7f7cd4108f70 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:41:55.949 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:55.930 7f7cd37fe700 1 --2- 172.21.2.102:0/3794956660 >> v2:172.21.2.113:3300/0 conn(0x7f7cd40d1c80 0x7f7cd4108f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:55.949 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:55.930 7f7cd37fe700 1 -- 172.21.2.102:0/3794956660 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cd4109590 con 0x7f7cd40d1880 2021-02-10T11:41:55.949 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:55.930 7f7cd37fe700 1 --2- 172.21.2.102:0/3794956660 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7cd40d1880 0x7f7cd41b04d0 secure :-1 s=READY pgs=3479 cs=0 l=1 rev1=1 rx=0x7f7cc4006700 tx=0x7f7cc4009b00).ready entity=mon.0 client_cookie=1734cb57b95128e5 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:55.950 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:55.931 7f7cd27fc700 1 -- 172.21.2.102:0/3794956660 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7cc400c0c0 con 0x7f7cd40d1880 2021-02-10T11:41:55.950 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:55.931 7f7cd27fc700 1 -- 172.21.2.102:0/3794956660 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cc400c340 con 0x7f7cd40d1880 2021-02-10T11:41:55.950 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:55.931 7f7cd27fc700 1 -- 172.21.2.102:0/3794956660 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7cc400c820 con 0x7f7cd40d1880 2021-02-10T11:41:55.950 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:55.931 7f7cda172700 1 -- 172.21.2.102:0/3794956660 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7cd40d1880 msgr2=0x7f7cd41b04d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:55.950 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:55.931 7f7cda172700 1 --2- 172.21.2.102:0/3794956660 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7cd40d1880 0x7f7cd41b04d0 secure :-1 s=READY pgs=3479 cs=0 l=1 rev1=1 rx=0x7f7cc4006700 tx=0x7f7cc4009b00).stop 2021-02-10T11:41:55.950 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:55.931 7f7cda172700 1 -- 172.21.2.102:0/3794956660 shutdown_connections 2021-02-10T11:41:55.950 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:55.931 7f7cda172700 1 --2- 172.21.2.102:0/3794956660 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7cd41b0af0 0x7f7cd4108950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:55.950 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:55.931 7f7cda172700 1 --2- 172.21.2.102:0/3794956660 >> v2:172.21.2.113:3300/0 conn(0x7f7cd40d1c80 0x7f7cd4108f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:55.951 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:55.931 7f7cda172700 1 --2- 172.21.2.102:0/3794956660 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7cd40d1880 0x7f7cd41b04d0 unknown :-1 s=CLOSED pgs=3479 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:55.951 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:55.931 7f7cda172700 1 -- 172.21.2.102:0/3794956660 >> 172.21.2.102:0/3794956660 conn(0x7f7cd40ce430 msgr2=0x7f7cd40d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:55.951 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:55.931 7f7cda172700 1 -- 172.21.2.102:0/3794956660 shutdown_connections 2021-02-10T11:41:55.951 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:55.931 7f7cda172700 1 -- 172.21.2.102:0/3794956660 wait complete. 2021-02-10T11:41:55.952 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:55.946 7f7cda172700 1 Processor -- start 2021-02-10T11:41:55.952 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:55.946 7f7cda172700 1 -- start start 2021-02-10T11:41:55.953 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:55.946 7f7cda172700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7cd4190df0 0x7f7cd41b7ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:55.953 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:55.946 7f7cda172700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7cd419ba20 0x7f7cd41b87c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:55.953 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:55.946 7f7cda172700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7cd419be20 0x7f7cd418f830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:55.953 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:55.946 7f7cda172700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cd41b8240 con 0x7f7cd4190df0 2021-02-10T11:41:55.953 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:55.946 7f7cda172700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cd41911f0 con 0x7f7cd419be20 2021-02-10T11:41:55.953 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:55.946 7f7cda172700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7cd41908a0 con 0x7f7cd419ba20 2021-02-10T11:41:55.953 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:55.946 7f7cd37fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7cd4190df0 0x7f7cd41b7ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:55.953 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:55.946 7f7cd37fe700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7cd4190df0 0x7f7cd41b7ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:41880/0 (socket says 172.21.2.102:41880) 2021-02-10T11:41:55.954 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:55.946 7f7cd37fe700 1 -- 172.21.2.102:0/3234647064 learned_addr learned my addr 172.21.2.102:0/3234647064 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:41:55.954 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:55.947 7f7cd37fe700 1 -- 172.21.2.102:0/3234647064 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7cd419ba20 msgr2=0x7f7cd41b87c0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:41:55.954 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:55.947 7f7cd37fe700 1 --2- 172.21.2.102:0/3234647064 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f7cd419ba20 0x7f7cd41b87c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:55.954 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:55.947 7f7cd37fe700 1 -- 172.21.2.102:0/3234647064 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7cd419be20 msgr2=0x7f7cd418f830 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:41:55.954 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:55.947 7f7cd37fe700 1 --2- 172.21.2.102:0/3234647064 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f7cd419be20 0x7f7cd418f830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:55.954 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:55.947 7f7cd37fe700 1 -- 172.21.2.102:0/3234647064 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cd41b8de0 con 0x7f7cd4190df0 2021-02-10T11:41:55.954 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:55.947 7f7cd37fe700 1 --2- 172.21.2.102:0/3234647064 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7cd4190df0 0x7f7cd41b7ac0 secure :-1 s=READY pgs=3480 cs=0 l=1 rev1=1 rx=0x7f7cc4006f20 tx=0x7f7cc400b9b0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:55.954 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:55.947 7f7ccbfff700 1 -- 172.21.2.102:0/3234647064 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7cc4009100 con 0x7f7cd4190df0 2021-02-10T11:41:55.954 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:55.947 7f7ccbfff700 1 -- 172.21.2.102:0/3234647064 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cc400bda0 con 0x7f7cd4190df0 2021-02-10T11:41:55.955 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:55.947 7f7cda172700 1 -- 172.21.2.102:0/3234647064 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7cd41b8de0 con 0x7f7cd4190df0 2021-02-10T11:41:55.955 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:55.947 7f7cda172700 1 -- 172.21.2.102:0/3234647064 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7cd418fe50 con 0x7f7cd4190df0 2021-02-10T11:41:55.955 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:55.947 7f7ccbfff700 1 -- 172.21.2.102:0/3234647064 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7cc40052e0 con 0x7f7cd4190df0 2021-02-10T11:41:55.955 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:55.948 7f7ccbfff700 1 -- 172.21.2.102:0/3234647064 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f7cc4009340 con 0x7f7cd4190df0 2021-02-10T11:41:55.955 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:55.948 7f7ccbfff700 1 --2- 172.21.2.102:0/3234647064 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7cb80186c0 0x7f7cb801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:55.955 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:55.948 7f7ccbfff700 1 -- 172.21.2.102:0/3234647064 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f7cc4032530 con 0x7f7cd4190df0 2021-02-10T11:41:55.955 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:55.948 7f7cd2ffd700 1 --2- 172.21.2.102:0/3234647064 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7cb80186c0 0x7f7cb801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:55.955 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:55.948 7f7cd2ffd700 1 --2- 172.21.2.102:0/3234647064 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7cb80186c0 0x7f7cb801abc0 crc :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:55.982 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:55.979 7f7cda172700 1 --2- 172.21.2.102:0/3234647064 >> [v2:172.21.2.113:6800/36367,v1:172.21.2.113:6801/36367] conn(0x7f7cd40627c0 0x7f7cd4062bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:55.982 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:55.979 7f7cda172700 1 -- 172.21.2.102:0/3234647064 --> [v2:172.21.2.113:6800/36367,v1:172.21.2.113:6801/36367] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7cd418fe50 con 0x7f7cd40627c0 2021-02-10T11:41:55.982 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:55.980 7f7cd3fff700 1 --2- 172.21.2.102:0/3234647064 >> [v2:172.21.2.113:6800/36367,v1:172.21.2.113:6801/36367] conn(0x7f7cd40627c0 0x7f7cd4062bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:55.983 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:55.981 7f7cd3fff700 1 --2- 172.21.2.102:0/3234647064 >> [v2:172.21.2.113:6800/36367,v1:172.21.2.113:6801/36367] conn(0x7f7cd40627c0 0x7f7cd4062bc0 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:55.984 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:55.982 7f7ccbfff700 1 -- 172.21.2.102:0/3234647064 <== osd.6 v2:172.21.2.113:6800/36367 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+5151 (crc 0 0 0) 0x7f7ccc006d10 con 0x7f7cd40627c0 2021-02-10T11:41:55.990 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:55.988 7f7cda172700 1 -- 172.21.2.102:0/3234647064 --> [v2:172.21.2.113:6800/36367,v1:172.21.2.113:6801/36367] -- command(tid 2: {"prefix": "config set", "key": "osd_debug_deep_scrub_sleep", "value": "0"}) v1 -- 0x7f7cd418fe50 con 0x7f7cd40627c0 2021-02-10T11:41:55.991 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:55.989 7f7ccbfff700 1 -- 172.21.2.102:0/3234647064 <== osd.6 v2:172.21.2.113:6800/36367 2 ==== command_reply(tid 2: 0 Set osd_debug_deep_scrub_sleep to 0) v1 ==== 43+0+0 (crc 0 0 0) 0x7f7ccc006db0 con 0x7f7cd40627c0 2021-02-10T11:41:55.993 INFO:teuthology.orchestra.run.gibba002.stderr:Set osd_debug_deep_scrub_sleep to 0 2021-02-10T11:41:55.993 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:55.990 7f7cda172700 1 -- 172.21.2.102:0/3234647064 >> [v2:172.21.2.113:6800/36367,v1:172.21.2.113:6801/36367] conn(0x7f7cd40627c0 msgr2=0x7f7cd4062bc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:55.993 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:55.990 7f7cda172700 1 --2- 172.21.2.102:0/3234647064 >> [v2:172.21.2.113:6800/36367,v1:172.21.2.113:6801/36367] conn(0x7f7cd40627c0 0x7f7cd4062bc0 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:55.994 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:55.990 7f7cda172700 1 -- 172.21.2.102:0/3234647064 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7cb80186c0 msgr2=0x7f7cb801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:55.994 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:55.990 7f7cda172700 1 --2- 172.21.2.102:0/3234647064 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f7cb80186c0 0x7f7cb801abc0 crc :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:55.995 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:55.990 7f7cda172700 1 -- 172.21.2.102:0/3234647064 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7cd4190df0 msgr2=0x7f7cd41b7ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:55.995 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:55.990 7f7cda172700 1 --2- 172.21.2.102:0/3234647064 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f7cd4190df0 0x7f7cd41b7ac0 secure :-1 s=READY pgs=3480 cs=0 l=1 rev1=1 rx=0x7f7cc4006f20 tx=0x7f7cc400b9b0).stop 2021-02-10T11:41:55.995 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:55.990 7f7cd3fff700 1 -- 172.21.2.102:0/3234647064 reap_dead start 2021-02-10T11:41:55.995 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:55.990 7f7cda172700 1 -- 172.21.2.102:0/3234647064 shutdown_connections 2021-02-10T11:41:55.996 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:55.990 7f7cda172700 1 -- 172.21.2.102:0/3234647064 >> 172.21.2.102:0/3234647064 conn(0x7f7cd419f940 msgr2=0x7f7cd418c290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:55.996 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:55.990 7f7cda172700 1 -- 172.21.2.102:0/3234647064 shutdown_connections 2021-02-10T11:41:55.996 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:55.990 7f7cda172700 1 -- 172.21.2.102:0/3234647064 wait complete. 2021-02-10T11:41:56.000 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd deep-scrub 6 2021-02-10T11:41:56.429 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:56.409 7f563cc35700 1 Processor -- start 2021-02-10T11:41:56.430 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:56.409 7f563cc35700 1 -- start start 2021-02-10T11:41:56.430 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:56.410 7f563cc35700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f56380d1870 0x7f56381b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:56.430 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:56.410 7f563cc35700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56380f0b10 con 0x7f56381b0a10 2021-02-10T11:41:56.430 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:56.410 7f563cc35700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56380d0310 con 0x7f56380f0650 2021-02-10T11:41:56.430 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:56.410 7f563cc35700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56380d0530 con 0x7f56380d1870 2021-02-10T11:41:56.430 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:56.410 7f563605d700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f56381b0a10 0x7f56380cf330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:34114/0 (socket says 172.21.2.102:34114) 2021-02-10T11:41:56.431 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:56.410 7f563605d700 1 -- 172.21.2.102:0/1877365186 learned_addr learned my addr 172.21.2.102:0/1877365186 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:41:56.431 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:56.410 7f563585c700 1 -- 172.21.2.102:0/1877365186 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1939913735 0 0) 0x7f56200010b0 con 0x7f56381b0a10 2021-02-10T11:41:56.431 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:56.410 7f563585c700 1 -- 172.21.2.102:0/1877365186 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5624002fe0 con 0x7f56381b0a10 2021-02-10T11:41:56.431 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:56.411 7f563685e700 1 --2- 172.21.2.102:0/1877365186 >> v2:172.21.2.113:3300/0 conn(0x7f56380d1870 0x7f56381b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:56.431 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:56.411 7f563585c700 1 -- 172.21.2.102:0/1877365186 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4181310 0 0) 0x7f56200010b0 con 0x7f56381b0a10 2021-02-10T11:41:56.431 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:56.411 7f563585c700 1 -- 172.21.2.102:0/1877365186 >> v2:172.21.2.113:3300/0 conn(0x7f56380d1870 msgr2=0x7f56381b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:41:56.431 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:56.411 7f563585c700 1 --2- 172.21.2.102:0/1877365186 >> v2:172.21.2.113:3300/0 conn(0x7f56380d1870 0x7f56381b03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:56.431 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:56.411 7f563585c700 1 -- 172.21.2.102:0/1877365186 >> v1:172.21.2.113:6789/0 conn(0x7f56380f0650 legacy=0x7f56380cfb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:56.432 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:56.411 7f563585c700 1 -- 172.21.2.102:0/1877365186 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56380d1c70 con 0x7f56381b0a10 2021-02-10T11:41:56.432 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:56.411 7f563585c700 1 -- 172.21.2.102:0/1877365186 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7f56200015b0 con 0x7f56381b0a10 2021-02-10T11:41:56.432 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:56.411 7f563585c700 1 -- 172.21.2.102:0/1877365186 >> v1:172.21.2.102:6789/0 conn(0x7f56381b0a10 legacy=0x7f56380cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:56.432 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:56.411 7f563585c700 1 --2- 172.21.2.102:0/1877365186 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5624003a60 0x7f5624005f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:56.432 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:56.411 7f563585c700 1 --2- 172.21.2.102:0/1877365186 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5624006780 0x7f5624008ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:56.432 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:56.411 7f563585c700 1 --2- 172.21.2.102:0/1877365186 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f56240095a0 0x7f562400baa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:56.432 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:56.411 7f563585c700 1 -- 172.21.2.102:0/1877365186 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f562400c2f0 con 0x7f56240095a0 2021-02-10T11:41:56.432 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:56.411 7f563585c700 1 -- 172.21.2.102:0/1877365186 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f562400c510 con 0x7f5624003a60 2021-02-10T11:41:56.433 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:56.411 7f563585c700 1 -- 172.21.2.102:0/1877365186 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f562400c730 con 0x7f5624006780 2021-02-10T11:41:56.433 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:56.411 7f563705f700 1 --2- 172.21.2.102:0/1877365186 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f56240095a0 0x7f562400baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:56.433 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:56.411 7f563685e700 1 --2- 172.21.2.102:0/1877365186 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5624003a60 0x7f5624005f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:56.433 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:56.411 7f563605d700 1 --2- 172.21.2.102:0/1877365186 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5624006780 0x7f5624008ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:56.433 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:56.411 7f563705f700 1 -- 172.21.2.102:0/1877365186 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5624006780 msgr2=0x7f5624008ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:41:56.433 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:56.411 7f563705f700 1 --2- 172.21.2.102:0/1877365186 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f5624006780 0x7f5624008ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:56.433 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:56.411 7f563705f700 1 -- 172.21.2.102:0/1877365186 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5624003a60 msgr2=0x7f5624005f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:41:56.433 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:56.411 7f563705f700 1 --2- 172.21.2.102:0/1877365186 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f5624003a60 0x7f5624005f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:56.434 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:56.411 7f563705f700 1 -- 172.21.2.102:0/1877365186 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f562400c9d0 con 0x7f56240095a0 2021-02-10T11:41:56.434 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:56.411 7f563705f700 1 -- 172.21.2.102:0/1877365186 reap_dead start 2021-02-10T11:41:56.434 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:56.412 7f563705f700 1 --2- 172.21.2.102:0/1877365186 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f56240095a0 0x7f562400baa0 secure :-1 s=READY pgs=3482 cs=0 l=1 rev1=1 rx=0x7f563004da40 tx=0x7f563004dde0).ready entity=mon.0 client_cookie=c00c91ff725b0631 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:56.434 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:56.412 7f563585c700 1 -- 172.21.2.102:0/1877365186 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5630006a60 con 0x7f56240095a0 2021-02-10T11:41:56.434 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:56.412 7f563585c700 1 -- 172.21.2.102:0/1877365186 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f563004f300 con 0x7f56240095a0 2021-02-10T11:41:56.434 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:56.412 7f563585c700 1 -- 172.21.2.102:0/1877365186 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f563004f750 con 0x7f56240095a0 2021-02-10T11:41:56.434 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:56.412 7f563cc35700 1 -- 172.21.2.102:0/1877365186 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f56240095a0 msgr2=0x7f562400baa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:56.435 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:56.412 7f563cc35700 1 --2- 172.21.2.102:0/1877365186 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f56240095a0 0x7f562400baa0 secure :-1 s=READY pgs=3482 cs=0 l=1 rev1=1 rx=0x7f563004da40 tx=0x7f563004dde0).stop 2021-02-10T11:41:56.435 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:56.412 7f563cc35700 1 -- 172.21.2.102:0/1877365186 shutdown_connections 2021-02-10T11:41:56.435 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:56.412 7f563cc35700 1 --2- 172.21.2.102:0/1877365186 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f56240095a0 0x7f562400baa0 unknown :-1 s=CLOSED pgs=3482 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:56.435 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:56.412 7f563cc35700 1 -- 172.21.2.102:0/1877365186 >> 172.21.2.102:0/1877365186 conn(0x7f56380ce450 msgr2=0x7f56380d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:56.435 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:56.413 7f563cc35700 1 -- 172.21.2.102:0/1877365186 shutdown_connections 2021-02-10T11:41:56.435 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:56.413 7f563cc35700 1 -- 172.21.2.102:0/1877365186 wait complete. 2021-02-10T11:41:56.435 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:56.427 7f563cc35700 1 Processor -- start 2021-02-10T11:41:56.435 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:56.427 7f563cc35700 1 -- start start 2021-02-10T11:41:56.437 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:56.428 7f563cc35700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f563808e8f0 0x7f563808ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:56.437 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:56.428 7f563cc35700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f56380ebe80 0x7f56380ec280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:56.437 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:56.428 7f563cc35700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f56380ec8a0 0x7f56380ed650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:56.437 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:56.428 7f563cc35700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56380a0a50 con 0x7f563808e8f0 2021-02-10T11:41:56.437 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:56.428 7f563cc35700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f563808f310 con 0x7f56380ebe80 2021-02-10T11:41:56.437 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:56.428 7f563cc35700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f563808f530 con 0x7f56380ec8a0 2021-02-10T11:41:56.438 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:56.428 7f563685e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f563808e8f0 0x7f563808ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:56.438 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:56.428 7f563685e700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f563808e8f0 0x7f563808ecf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:41902/0 (socket says 172.21.2.102:41902) 2021-02-10T11:41:56.438 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:56.428 7f563685e700 1 -- 172.21.2.102:0/4077134334 learned_addr learned my addr 172.21.2.102:0/4077134334 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:41:56.438 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:56.428 7f563685e700 1 -- 172.21.2.102:0/4077134334 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f56380ec8a0 msgr2=0x7f56380ed650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:56.438 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:56.428 7f563685e700 1 --2- 172.21.2.102:0/4077134334 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f56380ec8a0 0x7f56380ed650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:56.438 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:56.428 7f563685e700 1 -- 172.21.2.102:0/4077134334 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f56380ebe80 msgr2=0x7f56380ec280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:56.438 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:56.428 7f563685e700 1 --2- 172.21.2.102:0/4077134334 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f56380ebe80 0x7f56380ec280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:56.439 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:56.428 7f563685e700 1 -- 172.21.2.102:0/4077134334 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56380ecca0 con 0x7f563808e8f0 2021-02-10T11:41:56.439 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:56.428 7f563685e700 1 --2- 172.21.2.102:0/4077134334 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f563808e8f0 0x7f563808ecf0 secure :-1 s=READY pgs=3483 cs=0 l=1 rev1=1 rx=0x7f5628006950 tx=0x7f562800b8e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:56.439 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:56.429 7f562f7fe700 1 -- 172.21.2.102:0/4077134334 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f562800bcd0 con 0x7f563808e8f0 2021-02-10T11:41:56.439 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:56.429 7f562f7fe700 1 -- 172.21.2.102:0/4077134334 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5628006030 con 0x7f563808e8f0 2021-02-10T11:41:56.440 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:56.429 7f562f7fe700 1 -- 172.21.2.102:0/4077134334 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5628005560 con 0x7f563808e8f0 2021-02-10T11:41:56.440 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:56.429 7f563cc35700 1 -- 172.21.2.102:0/4077134334 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56380ecca0 con 0x7f563808e8f0 2021-02-10T11:41:56.440 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:56.429 7f563cc35700 1 -- 172.21.2.102:0/4077134334 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56380edf60 con 0x7f563808e8f0 2021-02-10T11:41:56.440 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:56.429 7f562f7fe700 1 -- 172.21.2.102:0/4077134334 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f56280052e0 con 0x7f563808e8f0 2021-02-10T11:41:56.440 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:56.429 7f562f7fe700 1 --2- 172.21.2.102:0/4077134334 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f561c0186c0 0x7f561c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:56.441 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:56.429 7f562f7fe700 1 -- 172.21.2.102:0/4077134334 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f5628032590 con 0x7f563808e8f0 2021-02-10T11:41:56.441 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:56.430 7f563605d700 1 --2- 172.21.2.102:0/4077134334 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f561c0186c0 0x7f561c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:56.441 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:56.430 7f563605d700 1 --2- 172.21.2.102:0/4077134334 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f561c0186c0 0x7f561c01abc0 crc :-1 s=READY pgs=1515 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:56.462 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:56.460 7f563cc35700 1 -- 172.21.2.102:0/4077134334 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56380a0a50 con 0x7f563808e8f0 2021-02-10T11:41:56.465 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:56.463 7f562f7fe700 1 -- 172.21.2.102:0/4077134334 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f5628000990 con 0x7f563808e8f0 2021-02-10T11:41:56.594 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:56.591 7f563cc35700 1 -- 172.21.2.102:0/4077134334 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mgr", ""]}) v1 -- 0x7f56380edf60 con 0x7f561c0186c0 2021-02-10T11:41:56.594 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:56.592 7f562f7fe700 1 -- 172.21.2.102:0/4077134334 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) v1 ==== 41+0+0 (crc 0 0 0) 0x7f5620001580 con 0x7f561c0186c0 2021-02-10T11:41:56.595 INFO:teuthology.orchestra.run.gibba002.stderr:instructed osd(s) 6 to deep-scrub 2021-02-10T11:41:56.596 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:56.594 7f563cc35700 1 -- 172.21.2.102:0/4077134334 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f561c0186c0 msgr2=0x7f561c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:56.597 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:56.594 7f563cc35700 1 --2- 172.21.2.102:0/4077134334 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f561c0186c0 0x7f561c01abc0 crc :-1 s=READY pgs=1515 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:56.597 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:56.594 7f563cc35700 1 -- 172.21.2.102:0/4077134334 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f563808e8f0 msgr2=0x7f563808ecf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:56.597 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:56.594 7f563cc35700 1 --2- 172.21.2.102:0/4077134334 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f563808e8f0 0x7f563808ecf0 secure :-1 s=READY pgs=3483 cs=0 l=1 rev1=1 rx=0x7f5628006950 tx=0x7f562800b8e0).stop 2021-02-10T11:41:56.598 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:56.594 7f563cc35700 1 -- 172.21.2.102:0/4077134334 shutdown_connections 2021-02-10T11:41:56.598 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:56.594 7f563cc35700 1 --2- 172.21.2.102:0/4077134334 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f56380ec8a0 0x7f56380ed650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:56.598 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:56.594 7f563cc35700 1 --2- 172.21.2.102:0/4077134334 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f561c0186c0 0x7f561c01abc0 unknown :-1 s=CLOSED pgs=1515 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:56.598 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:56.594 7f563cc35700 1 --2- 172.21.2.102:0/4077134334 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f56380ebe80 0x7f56380ec280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:56.599 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:56.594 7f563cc35700 1 --2- 172.21.2.102:0/4077134334 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f563808e8f0 0x7f563808ecf0 unknown :-1 s=CLOSED pgs=3483 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:56.599 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:56.594 7f563cc35700 1 -- 172.21.2.102:0/4077134334 >> 172.21.2.102:0/4077134334 conn(0x7f5638094d90 msgr2=0x7f563808a420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:56.599 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:56.594 7f563cc35700 1 -- 172.21.2.102:0/4077134334 shutdown_connections 2021-02-10T11:41:56.600 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:56.594 7f563cc35700 1 -- 172.21.2.102:0/4077134334 wait complete. 2021-02-10T11:41:56.605 INFO:tasks.ceph:Scrubbing osd.7 2021-02-10T11:41:56.606 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2021-02-10T11:41:57.041 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.020 7fe5b370f700 1 Processor -- start 2021-02-10T11:41:57.041 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.020 7fe5b370f700 1 -- start start 2021-02-10T11:41:57.041 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.020 7fe5b370f700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe5ac0d1c80 0x7fe5ac0cef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:57.041 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.020 7fe5b370f700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5ac0d0180 con 0x7fe5ac0d1880 2021-02-10T11:41:57.041 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.020 7fe5b370f700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5ac0fe1d0 con 0x7fe5ac0cf590 2021-02-10T11:41:57.041 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.020 7fe5b370f700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5ac1b0cc0 con 0x7fe5ac0d1c80 2021-02-10T11:41:57.042 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.021 7fe5b152b700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7fe5ac0d1880 0x7fe5ac1b04d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:34132/0 (socket says 172.21.2.102:34132) 2021-02-10T11:41:57.042 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.021 7fe5b152b700 1 -- 172.21.2.102:0/4229920376 learned_addr learned my addr 172.21.2.102:0/4229920376 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:41:57.042 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.021 7fe59ffff700 1 -- 172.21.2.102:0/4229920376 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 40937666 0 0) 0x7fe5a80009b0 con 0x7fe5ac0d1880 2021-02-10T11:41:57.042 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.021 7fe59ffff700 1 -- 172.21.2.102:0/4229920376 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe598002fe0 con 0x7fe5ac0d1880 2021-02-10T11:41:57.042 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.021 7fe5b0d2a700 1 --2- 172.21.2.102:0/4229920376 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe5ac0d1c80 0x7fe5ac0cef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:57.042 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.021 7fe59ffff700 1 -- 172.21.2.102:0/4229920376 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1137822553 0 0) 0x7fe5a8001010 con 0x7fe5ac0d1880 2021-02-10T11:41:57.042 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.021 7fe59ffff700 1 -- 172.21.2.102:0/4229920376 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe5ac0d1c80 msgr2=0x7fe5ac0cef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:41:57.042 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.021 7fe59ffff700 1 --2- 172.21.2.102:0/4229920376 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe5ac0d1c80 0x7fe5ac0cef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:57.043 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.022 7fe59ffff700 1 -- 172.21.2.102:0/4229920376 >> v1:172.21.2.113:6789/0 conn(0x7fe5ac0cf590 legacy=0x7fe5ac0cf990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:57.043 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.022 7fe59ffff700 1 -- 172.21.2.102:0/4229920376 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5ac109a70 con 0x7fe5ac0d1880 2021-02-10T11:41:57.043 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.022 7fe59ffff700 1 -- 172.21.2.102:0/4229920376 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7fe5a8001590 con 0x7fe5ac0d1880 2021-02-10T11:41:57.043 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.022 7fe59ffff700 1 -- 172.21.2.102:0/4229920376 >> v1:172.21.2.102:6789/0 conn(0x7fe5ac0d1880 legacy=0x7fe5ac1b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:57.043 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.022 7fe59ffff700 1 --2- 172.21.2.102:0/4229920376 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe598003a60 0x7fe598005ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:57.043 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.022 7fe59ffff700 1 --2- 172.21.2.102:0/4229920376 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe598006790 0x7fe598008c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:57.043 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.022 7fe59ffff700 1 --2- 172.21.2.102:0/4229920376 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe598009590 0x7fe59800ba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:57.043 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.022 7fe59ffff700 1 -- 172.21.2.102:0/4229920376 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe59800c2d0 con 0x7fe598003a60 2021-02-10T11:41:57.044 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.022 7fe59ffff700 1 -- 172.21.2.102:0/4229920376 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe59800c4f0 con 0x7fe598006790 2021-02-10T11:41:57.044 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.022 7fe59ffff700 1 -- 172.21.2.102:0/4229920376 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe59800c710 con 0x7fe598009590 2021-02-10T11:41:57.044 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.022 7fe5b152b700 1 --2- 172.21.2.102:0/4229920376 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe598003a60 0x7fe598005ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:57.044 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.022 7fe5b0d2a700 1 --2- 172.21.2.102:0/4229920376 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe598006790 0x7fe598008c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:57.044 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.022 7fe5b1d2c700 1 --2- 172.21.2.102:0/4229920376 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe598009590 0x7fe59800ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:57.044 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.022 7fe5b0d2a700 1 -- 172.21.2.102:0/4229920376 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe598009590 msgr2=0x7fe59800ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:41:57.044 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.022 7fe5b0d2a700 1 --2- 172.21.2.102:0/4229920376 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe598009590 0x7fe59800ba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:57.044 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.022 7fe5b0d2a700 1 -- 172.21.2.102:0/4229920376 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe598003a60 msgr2=0x7fe598005ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:41:57.045 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.022 7fe5b0d2a700 1 --2- 172.21.2.102:0/4229920376 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe598003a60 0x7fe598005ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:57.045 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.022 7fe5b0d2a700 1 -- 172.21.2.102:0/4229920376 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe598050980 con 0x7fe598006790 2021-02-10T11:41:57.045 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.022 7fe5b152b700 1 --2- 172.21.2.102:0/4229920376 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe598003a60 0x7fe598005ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:41:57.045 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.022 7fe5b0d2a700 1 --2- 172.21.2.102:0/4229920376 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe598006790 0x7fe598008c90 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 rx=0x7fe5a0009310 tx=0x7fe5a0009a90).ready entity=mon.1 client_cookie=2b641590c1f7dd13 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:57.045 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.023 7fe59ffff700 1 -- 172.21.2.102:0/4229920376 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe5a000c0c0 con 0x7fe598006790 2021-02-10T11:41:57.045 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.023 7fe59ffff700 1 -- 172.21.2.102:0/4229920376 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5a0006400 con 0x7fe598006790 2021-02-10T11:41:57.045 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.023 7fe59ffff700 1 -- 172.21.2.102:0/4229920376 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe5a000c400 con 0x7fe598006790 2021-02-10T11:41:57.045 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.023 7fe5b370f700 1 -- 172.21.2.102:0/4229920376 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe598006790 msgr2=0x7fe598008c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:57.046 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.023 7fe5b370f700 1 --2- 172.21.2.102:0/4229920376 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe598006790 0x7fe598008c90 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 rx=0x7fe5a0009310 tx=0x7fe5a0009a90).stop 2021-02-10T11:41:57.046 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.023 7fe5b1d2c700 1 -- 172.21.2.102:0/4229920376 reap_dead start 2021-02-10T11:41:57.046 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.024 7fe5b370f700 1 -- 172.21.2.102:0/4229920376 shutdown_connections 2021-02-10T11:41:57.046 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.024 7fe5b370f700 1 -- 172.21.2.102:0/4229920376 >> 172.21.2.102:0/4229920376 conn(0x7fe5ac0ce430 msgr2=0x7fe5ac0d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:57.046 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.024 7fe5b370f700 1 -- 172.21.2.102:0/4229920376 shutdown_connections 2021-02-10T11:41:57.046 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.024 7fe5b370f700 1 -- 172.21.2.102:0/4229920376 wait complete. 2021-02-10T11:41:57.046 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.038 7fe5b370f700 1 Processor -- start 2021-02-10T11:41:57.046 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.038 7fe5b370f700 1 -- start start 2021-02-10T11:41:57.048 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.039 7fe5b370f700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe5ac1a7a70 0x7fe5ac1a7e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:57.048 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.039 7fe5b370f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe5ac0bf730 0x7fe5ac0bfb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:57.048 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.039 7fe5b370f700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe5ac0c0150 0x7fe5ac0909c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:57.048 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.039 7fe5b370f700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5ac090fe0 con 0x7fe5ac0bf730 2021-02-10T11:41:57.048 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.039 7fe5b370f700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5ac091200 con 0x7fe5ac1a7a70 2021-02-10T11:41:57.049 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.039 7fe5b370f700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5ac091420 con 0x7fe5ac0c0150 2021-02-10T11:41:57.049 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.039 7fe5b0d2a700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe5ac0bf730 0x7fe5ac0bfb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:57.049 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.039 7fe5b0d2a700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe5ac0bf730 0x7fe5ac0bfb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:41920/0 (socket says 172.21.2.102:41920) 2021-02-10T11:41:57.049 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.039 7fe5b0d2a700 1 -- 172.21.2.102:0/644245372 learned_addr learned my addr 172.21.2.102:0/644245372 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:41:57.049 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.040 7fe5b1d2c700 1 --2- 172.21.2.102:0/644245372 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe5ac0c0150 0x7fe5ac0909c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:57.049 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.040 7fe5b0d2a700 1 -- 172.21.2.102:0/644245372 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe5ac0c0150 msgr2=0x7fe5ac0909c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:57.049 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.040 7fe5b0d2a700 1 --2- 172.21.2.102:0/644245372 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe5ac0c0150 0x7fe5ac0909c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:57.049 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.040 7fe5b0d2a700 1 -- 172.21.2.102:0/644245372 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe5ac1a7a70 msgr2=0x7fe5ac1a7e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:57.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.040 7fe5b0d2a700 1 --2- 172.21.2.102:0/644245372 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7fe5ac1a7a70 0x7fe5ac1a7e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:57.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.040 7fe5b0d2a700 1 -- 172.21.2.102:0/644245372 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5ac091640 con 0x7fe5ac0bf730 2021-02-10T11:41:57.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.040 7fe5b1d2c700 1 --2- 172.21.2.102:0/644245372 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7fe5ac0c0150 0x7fe5ac0909c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:41:57.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.040 7fe5b0d2a700 1 --2- 172.21.2.102:0/644245372 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe5ac0bf730 0x7fe5ac0bfb30 secure :-1 s=READY pgs=3485 cs=0 l=1 rev1=1 rx=0x7fe5a000b610 tx=0x7fe5a000b9b0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:57.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.040 7fe59dffb700 1 -- 172.21.2.102:0/644245372 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe5a00093e0 con 0x7fe5ac0bf730 2021-02-10T11:41:57.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.040 7fe59dffb700 1 -- 172.21.2.102:0/644245372 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5a0000940 con 0x7fe5ac0bf730 2021-02-10T11:41:57.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.040 7fe59dffb700 1 -- 172.21.2.102:0/644245372 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe5a0002220 con 0x7fe5ac0bf730 2021-02-10T11:41:57.050 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.040 7fe5b370f700 1 -- 172.21.2.102:0/644245372 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5ac091640 con 0x7fe5ac0bf730 2021-02-10T11:41:57.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.040 7fe5b370f700 1 -- 172.21.2.102:0/644245372 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5ac091640 con 0x7fe5ac0bf730 2021-02-10T11:41:57.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.041 7fe59dffb700 1 -- 172.21.2.102:0/644245372 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fe5a0000c90 con 0x7fe5ac0bf730 2021-02-10T11:41:57.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.041 7fe59dffb700 1 --2- 172.21.2.102:0/644245372 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe58c0186c0 0x7fe58c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:57.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.041 7fe59dffb700 1 -- 172.21.2.102:0/644245372 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 5573+0+0 (secure 0 0 0) 0x7fe5a00327a0 con 0x7fe5ac0bf730 2021-02-10T11:41:57.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.041 7fe5b152b700 1 --2- 172.21.2.102:0/644245372 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe58c0186c0 0x7fe58c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:57.051 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.041 7fe5b152b700 1 --2- 172.21.2.102:0/644245372 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe58c0186c0 0x7fe58c01abc0 crc :-1 s=READY pgs=1516 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:57.072 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.070 7fe5b370f700 1 --2- 172.21.2.102:0/644245372 >> [v2:172.21.2.113:6817/36365,v1:172.21.2.113:6821/36365] conn(0x7fe5ac090fe0 0x7fe5ac0c1450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:57.073 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.070 7fe5b370f700 1 -- 172.21.2.102:0/644245372 --> [v2:172.21.2.113:6817/36365,v1:172.21.2.113:6821/36365] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe5ac091420 con 0x7fe5ac090fe0 2021-02-10T11:41:57.073 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.071 7fe5b1d2c700 1 --2- 172.21.2.102:0/644245372 >> [v2:172.21.2.113:6817/36365,v1:172.21.2.113:6821/36365] conn(0x7fe5ac090fe0 0x7fe5ac0c1450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:57.074 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.072 7fe5b1d2c700 1 --2- 172.21.2.102:0/644245372 >> [v2:172.21.2.113:6817/36365,v1:172.21.2.113:6821/36365] conn(0x7fe5ac090fe0 0x7fe5ac0c1450 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:57.075 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.073 7fe59dffb700 1 -- 172.21.2.102:0/644245372 <== osd.7 v2:172.21.2.113:6817/36365 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+5151 (crc 0 0 0) 0x7fe5a4005820 con 0x7fe5ac090fe0 2021-02-10T11:41:57.081 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.079 7fe5b370f700 1 -- 172.21.2.102:0/644245372 --> [v2:172.21.2.113:6817/36365,v1:172.21.2.113:6821/36365] -- command(tid 2: {"prefix": "config set", "key": "osd_debug_deep_scrub_sleep", "value": "0"}) v1 -- 0x7fe5ac0870f0 con 0x7fe5ac090fe0 2021-02-10T11:41:57.082 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.080 7fe59dffb700 1 -- 172.21.2.102:0/644245372 <== osd.7 v2:172.21.2.113:6817/36365 2 ==== command_reply(tid 2: 0 Set osd_debug_deep_scrub_sleep to 0) v1 ==== 43+0+0 (crc 0 0 0) 0x7fe5a4007d80 con 0x7fe5ac090fe0 2021-02-10T11:41:57.082 INFO:teuthology.orchestra.run.gibba002.stderr:Set osd_debug_deep_scrub_sleep to 0 2021-02-10T11:41:57.083 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.081 7fe5b370f700 1 -- 172.21.2.102:0/644245372 >> [v2:172.21.2.113:6817/36365,v1:172.21.2.113:6821/36365] conn(0x7fe5ac090fe0 msgr2=0x7fe5ac0c1450 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:57.084 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.081 7fe5b370f700 1 --2- 172.21.2.102:0/644245372 >> [v2:172.21.2.113:6817/36365,v1:172.21.2.113:6821/36365] conn(0x7fe5ac090fe0 0x7fe5ac0c1450 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:57.084 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.081 7fe5b370f700 1 -- 172.21.2.102:0/644245372 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe58c0186c0 msgr2=0x7fe58c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:57.084 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.081 7fe5b370f700 1 --2- 172.21.2.102:0/644245372 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7fe58c0186c0 0x7fe58c01abc0 crc :-1 s=READY pgs=1516 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:57.084 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.081 7fe5b370f700 1 -- 172.21.2.102:0/644245372 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe5ac0bf730 msgr2=0x7fe5ac0bfb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:57.084 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.081 7fe5b370f700 1 --2- 172.21.2.102:0/644245372 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7fe5ac0bf730 0x7fe5ac0bfb30 secure :-1 s=READY pgs=3485 cs=0 l=1 rev1=1 rx=0x7fe5a000b610 tx=0x7fe5a000b9b0).stop 2021-02-10T11:41:57.084 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.081 7fe5b1d2c700 1 -- 172.21.2.102:0/644245372 reap_dead start 2021-02-10T11:41:57.084 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.081 7fe5b370f700 1 -- 172.21.2.102:0/644245372 shutdown_connections 2021-02-10T11:41:57.084 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.081 7fe5b370f700 1 -- 172.21.2.102:0/644245372 >> 172.21.2.102:0/644245372 conn(0x7fe5ac0a69d0 msgr2=0x7fe5ac0b5950 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:57.085 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.081 7fe5b370f700 1 -- 172.21.2.102:0/644245372 shutdown_connections 2021-02-10T11:41:57.085 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.081 7fe5b370f700 1 -- 172.21.2.102:0/644245372 wait complete. 2021-02-10T11:41:57.091 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd deep-scrub 7 2021-02-10T11:41:57.522 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.502 7f84fb42f700 1 Processor -- start 2021-02-10T11:41:57.522 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.502 7f84fb42f700 1 -- start start 2021-02-10T11:41:57.523 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.502 7f84fb42f700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f84f40d0340 0x7f84f40ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:57.523 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.502 7f84fb42f700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f84f40d1c70 0x7f84f40efc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:57.523 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.502 7f84fb42f700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84f40f0b10 con 0x7f84f40d1870 2021-02-10T11:41:57.523 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.502 7f84fb42f700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84f40e5ed0 con 0x7f84f40d0340 2021-02-10T11:41:57.523 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.502 7f84fb42f700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84f41b0be0 con 0x7f84f40d1c70 2021-02-10T11:41:57.523 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.503 7f84f924b700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f84f40d1870 0x7f84f41b03f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:34152/0 (socket says 172.21.2.102:34152) 2021-02-10T11:41:57.523 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.503 7f84f924b700 1 -- 172.21.2.102:0/3312332441 learned_addr learned my addr 172.21.2.102:0/3312332441 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:41:57.524 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.503 7f84e7fff700 1 -- 172.21.2.102:0/3312332441 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1854946086 0 0) 0x7f84e80009b0 con 0x7f84f40d1870 2021-02-10T11:41:57.524 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.503 7f84f9a4c700 1 --2- 172.21.2.102:0/3312332441 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f84f40d1c70 0x7f84f40efc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:57.524 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.503 7f84f8a4a700 1 --2- 172.21.2.102:0/3312332441 >> v2:172.21.2.113:3300/0 conn(0x7f84f40d0340 0x7f84f40ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:57.524 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.503 7f84e7fff700 1 -- 172.21.2.102:0/3312332441 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f84e0002fe0 con 0x7f84f40d1870 2021-02-10T11:41:57.524 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.503 7f84e7fff700 1 -- 172.21.2.102:0/3312332441 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1753181226 0 0) 0x7f84e8001010 con 0x7f84f40d1870 2021-02-10T11:41:57.524 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.503 7f84e7fff700 1 -- 172.21.2.102:0/3312332441 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f84f40d1c70 msgr2=0x7f84f40efc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:41:57.524 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.503 7f84e7fff700 1 --2- 172.21.2.102:0/3312332441 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f84f40d1c70 0x7f84f40efc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:57.524 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.503 7f84e7fff700 1 -- 172.21.2.102:0/3312332441 >> v2:172.21.2.113:3300/0 conn(0x7f84f40d0340 msgr2=0x7f84f40ef640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:41:57.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.503 7f84e7fff700 1 --2- 172.21.2.102:0/3312332441 >> v2:172.21.2.113:3300/0 conn(0x7f84f40d0340 0x7f84f40ef640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:57.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.503 7f84e7fff700 1 -- 172.21.2.102:0/3312332441 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84f40f20a0 con 0x7f84f40d1870 2021-02-10T11:41:57.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.503 7f84e7fff700 1 -- 172.21.2.102:0/3312332441 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7f84e8001530 con 0x7f84f40d1870 2021-02-10T11:41:57.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.503 7f84f9a4c700 1 --2- 172.21.2.102:0/3312332441 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f84f40d1c70 0x7f84f40efc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:41:57.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.503 7f84e7fff700 1 -- 172.21.2.102:0/3312332441 >> v1:172.21.2.102:6789/0 conn(0x7f84f40d1870 legacy=0x7f84f41b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:57.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.503 7f84e7fff700 1 --2- 172.21.2.102:0/3312332441 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f84e0003af0 0x7f84e0005f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:57.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.503 7f84e7fff700 1 --2- 172.21.2.102:0/3312332441 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f84e004a790 0x7f84e004cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:57.525 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.504 7f84e7fff700 1 --2- 172.21.2.102:0/3312332441 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f84e004d590 0x7f84e004fa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:57.526 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.504 7f84e7fff700 1 -- 172.21.2.102:0/3312332441 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84e00502d0 con 0x7f84e0003af0 2021-02-10T11:41:57.526 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.504 7f84e7fff700 1 -- 172.21.2.102:0/3312332441 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84e00504f0 con 0x7f84e004a790 2021-02-10T11:41:57.526 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.504 7f84e7fff700 1 -- 172.21.2.102:0/3312332441 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84e0050710 con 0x7f84e004d590 2021-02-10T11:41:57.526 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.504 7f84f924b700 1 --2- 172.21.2.102:0/3312332441 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f84e0003af0 0x7f84e0005f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:57.526 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.504 7f84f8a4a700 1 --2- 172.21.2.102:0/3312332441 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f84e004a790 0x7f84e004cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:57.526 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.504 7f84f924b700 1 -- 172.21.2.102:0/3312332441 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f84e004d590 msgr2=0x7f84e004fa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:41:57.526 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.504 7f84f9a4c700 1 --2- 172.21.2.102:0/3312332441 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f84e004d590 0x7f84e004fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:57.527 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.504 7f84f924b700 1 --2- 172.21.2.102:0/3312332441 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f84e004d590 0x7f84e004fa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:57.527 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.504 7f84f924b700 1 -- 172.21.2.102:0/3312332441 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f84e004a790 msgr2=0x7f84e004cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:41:57.527 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.504 7f84f924b700 1 --2- 172.21.2.102:0/3312332441 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f84e004a790 0x7f84e004cc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:57.527 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.504 7f84f924b700 1 -- 172.21.2.102:0/3312332441 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84e00509b0 con 0x7f84e0003af0 2021-02-10T11:41:57.527 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.504 7f84f9a4c700 1 --2- 172.21.2.102:0/3312332441 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f84e004d590 0x7f84e004fa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:41:57.527 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.504 7f84f924b700 1 --2- 172.21.2.102:0/3312332441 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f84e0003af0 0x7f84e0005f60 secure :-1 s=READY pgs=3487 cs=0 l=1 rev1=1 rx=0x7f84e80017e0 tx=0x7f84e8007d40).ready entity=mon.0 client_cookie=ab10a1a1f5875b7d server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:57.527 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.504 7f84e7fff700 1 -- 172.21.2.102:0/3312332441 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f84e80011b0 con 0x7f84e0003af0 2021-02-10T11:41:57.527 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.504 7f84e7fff700 1 -- 172.21.2.102:0/3312332441 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84e800a520 con 0x7f84e0003af0 2021-02-10T11:41:57.527 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.504 7f84e7fff700 1 -- 172.21.2.102:0/3312332441 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f84e800a970 con 0x7f84e0003af0 2021-02-10T11:41:57.528 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.504 7f84fb42f700 1 -- 172.21.2.102:0/3312332441 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f84e0003af0 msgr2=0x7f84e0005f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:57.528 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.504 7f84fb42f700 1 --2- 172.21.2.102:0/3312332441 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f84e0003af0 0x7f84e0005f60 secure :-1 s=READY pgs=3487 cs=0 l=1 rev1=1 rx=0x7f84e80017e0 tx=0x7f84e8007d40).stop 2021-02-10T11:41:57.528 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.504 7f84f9a4c700 1 -- 172.21.2.102:0/3312332441 reap_dead start 2021-02-10T11:41:57.528 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.505 7f84fb42f700 1 -- 172.21.2.102:0/3312332441 shutdown_connections 2021-02-10T11:41:57.528 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.505 7f84fb42f700 1 -- 172.21.2.102:0/3312332441 >> 172.21.2.102:0/3312332441 conn(0x7f84f40ce450 msgr2=0x7f84f40d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:57.528 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.505 7f84fb42f700 1 -- 172.21.2.102:0/3312332441 shutdown_connections 2021-02-10T11:41:57.528 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.505 7f84fb42f700 1 -- 172.21.2.102:0/3312332441 wait complete. 2021-02-10T11:41:57.529 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.520 7f84fb42f700 1 Processor -- start 2021-02-10T11:41:57.529 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.520 7f84fb42f700 1 -- start start 2021-02-10T11:41:57.530 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.520 7f84fb42f700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f84f419ffa0 0x7f84f40eda10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:57.530 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.520 7f84fb42f700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f84f41a03a0 0x7f84f40ee030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:57.530 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.520 7f84fb42f700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f84f40ee650 0x7f84f40ebe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:57.530 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.520 7f84fb42f700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84f40ed490 con 0x7f84f41a03a0 2021-02-10T11:41:57.531 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.520 7f84fb42f700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84f40eea50 con 0x7f84f40ee650 2021-02-10T11:41:57.531 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.520 7f84fb42f700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84f40eec30 con 0x7f84f419ffa0 2021-02-10T11:41:57.531 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.521 7f84f8a4a700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f84f41a03a0 0x7f84f40ee030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:57.531 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.521 7f84f8a4a700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f84f41a03a0 0x7f84f40ee030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:41940/0 (socket says 172.21.2.102:41940) 2021-02-10T11:41:57.531 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.521 7f84f8a4a700 1 -- 172.21.2.102:0/2409127453 learned_addr learned my addr 172.21.2.102:0/2409127453 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:41:57.531 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.521 7f84f8a4a700 1 -- 172.21.2.102:0/2409127453 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f84f419ffa0 msgr2=0x7f84f40eda10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:57.531 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.521 7f84f8a4a700 1 --2- 172.21.2.102:0/2409127453 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f84f419ffa0 0x7f84f40eda10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:57.531 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.521 7f84f8a4a700 1 -- 172.21.2.102:0/2409127453 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f84f40ee650 msgr2=0x7f84f40ebe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:57.532 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.521 7f84f8a4a700 1 --2- 172.21.2.102:0/2409127453 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f84f40ee650 0x7f84f40ebe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:57.532 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.521 7f84f8a4a700 1 -- 172.21.2.102:0/2409127453 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84f40ec4a0 con 0x7f84f41a03a0 2021-02-10T11:41:57.532 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.521 7f84f8a4a700 1 --2- 172.21.2.102:0/2409127453 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f84f41a03a0 0x7f84f40ee030 secure :-1 s=READY pgs=3488 cs=0 l=1 rev1=1 rx=0x7f84f000b560 tx=0x7f84f000b900).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:57.532 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.521 7f84e5ffb700 1 -- 172.21.2.102:0/2409127453 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f84f000bcf0 con 0x7f84f41a03a0 2021-02-10T11:41:57.532 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.521 7f84e5ffb700 1 -- 172.21.2.102:0/2409127453 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84f0000c90 con 0x7f84f41a03a0 2021-02-10T11:41:57.532 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.521 7f84fb42f700 1 -- 172.21.2.102:0/2409127453 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84f40ec4a0 con 0x7f84f41a03a0 2021-02-10T11:41:57.532 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.521 7f84e5ffb700 1 -- 172.21.2.102:0/2409127453 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f84f00022b0 con 0x7f84f41a03a0 2021-02-10T11:41:57.532 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.521 7f84fb42f700 1 -- 172.21.2.102:0/2409127453 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84f40eca30 con 0x7f84f41a03a0 2021-02-10T11:41:57.533 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.522 7f84e5ffb700 1 -- 172.21.2.102:0/2409127453 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f84f0006b00 con 0x7f84f41a03a0 2021-02-10T11:41:57.533 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.522 7f84e5ffb700 1 --2- 172.21.2.102:0/2409127453 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f84d40186c0 0x7f84d401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:57.533 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.522 7f84e5ffb700 1 -- 172.21.2.102:0/2409127453 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f84f0032430 con 0x7f84f41a03a0 2021-02-10T11:41:57.533 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.522 7f84f924b700 1 --2- 172.21.2.102:0/2409127453 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f84d40186c0 0x7f84d401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:57.533 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.522 7f84f924b700 1 --2- 172.21.2.102:0/2409127453 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f84d40186c0 0x7f84d401abc0 crc :-1 s=READY pgs=1517 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:57.554 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.552 7f84fb42f700 1 -- 172.21.2.102:0/2409127453 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84f40eee50 con 0x7f84f41a03a0 2021-02-10T11:41:57.557 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.555 7f84e5ffb700 1 -- 172.21.2.102:0/2409127453 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f84f0000940 con 0x7f84f41a03a0 2021-02-10T11:41:57.686 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.683 7f84fb42f700 1 -- 172.21.2.102:0/2409127453 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mgr", ""]}) v1 -- 0x7f84f40eca30 con 0x7f84d40186c0 2021-02-10T11:41:57.686 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.683 7f84e5ffb700 1 -- 172.21.2.102:0/2409127453 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) v1 ==== 41+0+0 (crc 0 0 0) 0x7f84e80019f0 con 0x7f84d40186c0 2021-02-10T11:41:57.686 INFO:teuthology.orchestra.run.gibba002.stderr:instructed osd(s) 7 to deep-scrub 2021-02-10T11:41:57.688 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.685 7f84fb42f700 1 -- 172.21.2.102:0/2409127453 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f84d40186c0 msgr2=0x7f84d401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:57.688 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.685 7f84fb42f700 1 --2- 172.21.2.102:0/2409127453 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f84d40186c0 0x7f84d401abc0 crc :-1 s=READY pgs=1517 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:57.688 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.685 7f84fb42f700 1 -- 172.21.2.102:0/2409127453 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f84f41a03a0 msgr2=0x7f84f40ee030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:57.688 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.685 7f84fb42f700 1 --2- 172.21.2.102:0/2409127453 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f84f41a03a0 0x7f84f40ee030 secure :-1 s=READY pgs=3488 cs=0 l=1 rev1=1 rx=0x7f84f000b560 tx=0x7f84f000b900).stop 2021-02-10T11:41:57.689 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.686 7f84fb42f700 1 -- 172.21.2.102:0/2409127453 shutdown_connections 2021-02-10T11:41:57.689 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.686 7f84fb42f700 1 --2- 172.21.2.102:0/2409127453 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f84d40186c0 0x7f84d401abc0 unknown :-1 s=CLOSED pgs=1517 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:57.689 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.686 7f84fb42f700 1 --2- 172.21.2.102:0/2409127453 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f84f40ee650 0x7f84f40ebe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:57.689 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.686 7f84fb42f700 1 --2- 172.21.2.102:0/2409127453 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f84f41a03a0 0x7f84f40ee030 unknown :-1 s=CLOSED pgs=3488 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:57.689 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.686 7f84fb42f700 1 --2- 172.21.2.102:0/2409127453 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f84f419ffa0 0x7f84f40eda10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:57.689 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.686 7f84fb42f700 1 -- 172.21.2.102:0/2409127453 >> 172.21.2.102:0/2409127453 conn(0x7f84f4094d90 msgr2=0x7f84f408a420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:57.689 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.686 7f84fb42f700 1 -- 172.21.2.102:0/2409127453 shutdown_connections 2021-02-10T11:41:57.690 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:57.686 7f84fb42f700 1 -- 172.21.2.102:0/2409127453 wait complete. 2021-02-10T11:41:57.697 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph pg dump --format=json 2021-02-10T11:41:58.126 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:58.108 7f21cdbe3700 1 Processor -- start 2021-02-10T11:41:58.127 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:58.108 7f21cdbe3700 1 -- start start 2021-02-10T11:41:58.127 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:58.108 7f21cdbe3700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f21c81b0470 0x7f21c81b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:58.127 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:58.108 7f21cdbe3700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f21c80ef610 0x7f21c80efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:58.127 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:58.109 7f21cdbe3700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f21c80f0030 0x7f21c80f0430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:58.127 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:58.109 7f21cdbe3700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21c80e5ea0 con 0x7f21c80f0030 2021-02-10T11:41:58.127 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:58.109 7f21cdbe3700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21c80f0b50 con 0x7f21c81b0470 2021-02-10T11:41:58.128 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:58.109 7f21cdbe3700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21c80e52e0 con 0x7f21c80ef610 2021-02-10T11:41:58.128 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:58.109 7f21c7fff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f21c80f0030 0x7f21c80f0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:58.128 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:58.109 7f21c7fff700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7f21c80f0030 0x7f21c80f0430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:41948/0 (socket says 172.21.2.102:41948) 2021-02-10T11:41:58.128 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:58.109 7f21c7fff700 1 -- 172.21.2.102:0/4289111812 learned_addr learned my addr 172.21.2.102:0/4289111812 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:41:58.128 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:58.109 7f21c7fff700 1 -- 172.21.2.102:0/4289111812 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f21c80ef610 msgr2=0x7f21c80efa10 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:41:58.128 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:58.109 7f21c7fff700 1 --2- 172.21.2.102:0/4289111812 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f21c80ef610 0x7f21c80efa10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:58.128 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:58.109 7f21c7fff700 1 -- 172.21.2.102:0/4289111812 >> v2:172.21.2.113:3300/0 conn(0x7f21c81b0470 msgr2=0x7f21c81b0870 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:41:58.129 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:58.109 7f21c7fff700 1 --2- 172.21.2.102:0/4289111812 >> v2:172.21.2.113:3300/0 conn(0x7f21c81b0470 0x7f21c81b0870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:58.129 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:58.109 7f21c7fff700 1 -- 172.21.2.102:0/4289111812 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21c80f10d0 con 0x7f21c80f0030 2021-02-10T11:41:58.129 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:58.109 7f21c7fff700 1 --2- 172.21.2.102:0/4289111812 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f21c80f0030 0x7f21c80f0430 secure :-1 s=READY pgs=3489 cs=0 l=1 rev1=1 rx=0x7f21c0006f00 tx=0x7f21c0009f10).ready entity=mon.0 client_cookie=6002746582474091 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:58.129 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:58.109 7f21c67fc700 1 -- 172.21.2.102:0/4289111812 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f21c000b3b0 con 0x7f21c80f0030 2021-02-10T11:41:58.129 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:58.109 7f21c67fc700 1 -- 172.21.2.102:0/4289111812 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21c000b630 con 0x7f21c80f0030 2021-02-10T11:41:58.129 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:58.109 7f21c67fc700 1 -- 172.21.2.102:0/4289111812 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f21c000bb50 con 0x7f21c80f0030 2021-02-10T11:41:58.129 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:58.109 7f21cdbe3700 1 -- 172.21.2.102:0/4289111812 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f21c80f0030 msgr2=0x7f21c80f0430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:58.129 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:58.109 7f21cdbe3700 1 --2- 172.21.2.102:0/4289111812 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f21c80f0030 0x7f21c80f0430 secure :-1 s=READY pgs=3489 cs=0 l=1 rev1=1 rx=0x7f21c0006f00 tx=0x7f21c0009f10).stop 2021-02-10T11:41:58.130 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:58.110 7f21cdbe3700 1 -- 172.21.2.102:0/4289111812 shutdown_connections 2021-02-10T11:41:58.130 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:58.110 7f21cdbe3700 1 --2- 172.21.2.102:0/4289111812 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f21c80ef610 0x7f21c80efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:58.130 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:58.110 7f21cdbe3700 1 --2- 172.21.2.102:0/4289111812 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f21c80f0030 0x7f21c80f0430 unknown :-1 s=CLOSED pgs=3489 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:58.130 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:58.110 7f21cdbe3700 1 --2- 172.21.2.102:0/4289111812 >> v2:172.21.2.113:3300/0 conn(0x7f21c81b0470 0x7f21c81b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:58.130 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:58.110 7f21cdbe3700 1 -- 172.21.2.102:0/4289111812 >> 172.21.2.102:0/4289111812 conn(0x7f21c80ce3e0 msgr2=0x7f21c80e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:58.130 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:58.110 7f21cdbe3700 1 -- 172.21.2.102:0/4289111812 shutdown_connections 2021-02-10T11:41:58.130 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:58.110 7f21cdbe3700 1 -- 172.21.2.102:0/4289111812 wait complete. 2021-02-10T11:41:58.132 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:58.124 7f21cdbe3700 1 Processor -- start 2021-02-10T11:41:58.132 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:58.125 7f21cdbe3700 1 -- start start 2021-02-10T11:41:58.132 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:58.125 7f21cdbe3700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f21c80aa8d0 0x7f21c819a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:58.132 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:58.125 7f21cdbe3700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f21c819e9d0 0x7f21c819a720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:58.133 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:58.125 7f21cdbe3700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f21c819ad40 0x7f21c819b770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:58.133 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:58.125 7f21cdbe3700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21c819b1f0 con 0x7f21c819e9d0 2021-02-10T11:41:58.133 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:58.125 7f21cdbe3700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21c8109b30 con 0x7f21c819ad40 2021-02-10T11:41:58.133 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:58.125 7f21cdbe3700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21c8099620 con 0x7f21c80aa8d0 2021-02-10T11:41:58.133 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:58.125 7f21c6ffd700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f21c819e9d0 0x7f21c819a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:58.133 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:58.125 7f21c6ffd700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f21c819e9d0 0x7f21c819a720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:41954/0 (socket says 172.21.2.102:41954) 2021-02-10T11:41:58.133 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:58.125 7f21c6ffd700 1 -- 172.21.2.102:0/698198987 learned_addr learned my addr 172.21.2.102:0/698198987 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:41:58.133 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:58.125 7f21c6ffd700 1 -- 172.21.2.102:0/698198987 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f21c80aa8d0 msgr2=0x7f21c819a100 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:41:58.134 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:58.125 7f21c6ffd700 1 --2- 172.21.2.102:0/698198987 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f21c80aa8d0 0x7f21c819a100 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:58.134 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:58.125 7f21c6ffd700 1 -- 172.21.2.102:0/698198987 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f21c819ad40 msgr2=0x7f21c819b770 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:41:58.134 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:58.125 7f21c6ffd700 1 --2- 172.21.2.102:0/698198987 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f21c819ad40 0x7f21c819b770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:58.134 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:58.125 7f21c6ffd700 1 -- 172.21.2.102:0/698198987 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21c8109640 con 0x7f21c819e9d0 2021-02-10T11:41:58.134 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:58.126 7f21c6ffd700 1 --2- 172.21.2.102:0/698198987 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f21c819e9d0 0x7f21c819a720 secure :-1 s=READY pgs=3490 cs=0 l=1 rev1=1 rx=0x7f21b800b5c0 tx=0x7f21b800b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:58.134 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:58.126 7f21bffff700 1 -- 172.21.2.102:0/698198987 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f21b800bd50 con 0x7f21c819e9d0 2021-02-10T11:41:58.134 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:58.126 7f21bffff700 1 -- 172.21.2.102:0/698198987 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21b8006030 con 0x7f21c819e9d0 2021-02-10T11:41:58.135 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:58.126 7f21bffff700 1 -- 172.21.2.102:0/698198987 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f21b80054d0 con 0x7f21c819e9d0 2021-02-10T11:41:58.135 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:58.126 7f21cdbe3700 1 -- 172.21.2.102:0/698198987 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21c8109640 con 0x7f21c819e9d0 2021-02-10T11:41:58.135 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:58.126 7f21cdbe3700 1 -- 172.21.2.102:0/698198987 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21c8109640 con 0x7f21c819e9d0 2021-02-10T11:41:58.135 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:58.126 7f21bffff700 1 -- 172.21.2.102:0/698198987 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f21b8006210 con 0x7f21c819e9d0 2021-02-10T11:41:58.135 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:58.126 7f21bffff700 1 --2- 172.21.2.102:0/698198987 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f21ac0186c0 0x7f21ac01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:58.135 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:58.126 7f21bffff700 1 -- 172.21.2.102:0/698198987 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f21b80327d0 con 0x7f21c819e9d0 2021-02-10T11:41:58.135 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:58.127 7f21c77fe700 1 --2- 172.21.2.102:0/698198987 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f21ac0186c0 0x7f21ac01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:58.136 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:58.127 7f21c77fe700 1 --2- 172.21.2.102:0/698198987 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f21ac0186c0 0x7f21ac01abc0 crc :-1 s=READY pgs=1518 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:58.160 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:58.158 7f21cdbe3700 1 -- 172.21.2.102:0/698198987 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21c80f0b50 con 0x7f21c819e9d0 2021-02-10T11:41:58.163 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:58.161 7f21bffff700 1 -- 172.21.2.102:0/698198987 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f21b8006b80 con 0x7f21c819e9d0 2021-02-10T11:41:58.299 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:58.296 7f21cdbe3700 1 -- 172.21.2.102:0/698198987 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "pg dump", "target": ["mgr", ""], "format": "json"}) v1 -- 0x7f21c8109ae0 con 0x7f21ac0186c0 2021-02-10T11:41:58.301 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:58.299 7f21bffff700 1 -- 172.21.2.102:0/698198987 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 dumped all) v1 ==== 18+0+101100 (crc 0 0 0) 0x7f21b0006970 con 0x7f21ac0186c0 2021-02-10T11:41:58.304 INFO:teuthology.orchestra.run.gibba002.stderr:dumped all 2021-02-10T11:41:58.304 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:58.301 7f21cdbe3700 1 -- 172.21.2.102:0/698198987 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f21ac0186c0 msgr2=0x7f21ac01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:58.304 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:58.301 7f21cdbe3700 1 --2- 172.21.2.102:0/698198987 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f21ac0186c0 0x7f21ac01abc0 crc :-1 s=READY pgs=1518 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:58.304 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:58.301 7f21cdbe3700 1 -- 172.21.2.102:0/698198987 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f21c819e9d0 msgr2=0x7f21c819a720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:58.304 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:58.301 7f21cdbe3700 1 --2- 172.21.2.102:0/698198987 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f21c819e9d0 0x7f21c819a720 secure :-1 s=READY pgs=3490 cs=0 l=1 rev1=1 rx=0x7f21b800b5c0 tx=0x7f21b800b960).stop 2021-02-10T11:41:58.305 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:58.301 7f21cdbe3700 1 -- 172.21.2.102:0/698198987 shutdown_connections 2021-02-10T11:41:58.305 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:58.301 7f21cdbe3700 1 --2- 172.21.2.102:0/698198987 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f21ac0186c0 0x7f21ac01abc0 unknown :-1 s=CLOSED pgs=1518 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:58.305 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:58.301 7f21cdbe3700 1 --2- 172.21.2.102:0/698198987 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f21c819ad40 0x7f21c819b770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:58.305 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:58.301 7f21cdbe3700 1 --2- 172.21.2.102:0/698198987 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f21c819e9d0 0x7f21c819a720 unknown :-1 s=CLOSED pgs=3490 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:58.305 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:58.301 7f21cdbe3700 1 --2- 172.21.2.102:0/698198987 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f21c80aa8d0 0x7f21c819a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:58.305 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:58.301 7f21cdbe3700 1 -- 172.21.2.102:0/698198987 >> 172.21.2.102:0/698198987 conn(0x7f21c819f7b0 msgr2=0x7f21c818f380 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:58.305 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:58.302 7f21cdbe3700 1 -- 172.21.2.102:0/698198987 shutdown_connections 2021-02-10T11:41:58.305 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:41:58.302 7f21cdbe3700 1 -- 172.21.2.102:0/698198987 wait complete. 2021-02-10T11:41:58.306 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:41:58.313 INFO:teuthology.orchestra.run.gibba002.stdout:{"pg_ready":true,"pg_map":{"version":1353,"stamp":"2021-02-10 11:41:58.170599","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":135146961,"num_objects":154,"num_object_clones":1,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":154,"num_whiteouts":0,"num_read":65,"num_read_kb":65537,"num_write":431,"num_write_kb":132080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":416,"ondisk_log_size":416,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":8,"num_per_pool_osds":8,"kb":754974720,"kb_used":8672596,"kb_used_data":283920,"kb_used_omap":0,"kb_used_meta":8388608,"kb_avail":746302124,"statfs":{"total":773094113280,"available":764213374976,"internally_reserved":8589934592,"allocated":290734080,"data_stored":279050754,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8589934592},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":13665,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002574"},"pg_stats":[{"pgid":"43.6","version":"197'22","reported_seq":"40","reported_epoch":"197","state":"active+clean","last_fresh":"2021-02-10 11:41:37.311138","last_change":"2021-02-10 11:40:24.375639","last_active":"2021-02-10 11:41:37.311138","last_peered":"2021-02-10 11:41:37.311138","last_clean":"2021-02-10 11:41:37.311138","last_became_active":"2021-02-10 11:40:24.375377","last_became_peered":"2021-02-10 11:40:24.375377","last_unstale":"2021-02-10 11:41:37.311138","last_undegraded":"2021-02-10 11:41:37.311138","last_fullsized":"2021-02-10 11:41:37.311138","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:40:23.720923","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:40:23.720923","last_clean_scrub_stamp":"2021-02-10 11:40:23.720923","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":23068672,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":9,"num_read_kb":9216,"num_write":22,"num_write_kb":22528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"42.7","version":"197'8","reported_seq":"18","reported_epoch":"197","state":"active+clean","last_fresh":"2021-02-10 11:40:25.375917","last_change":"2021-02-10 11:40:23.729705","last_active":"2021-02-10 11:40:25.375917","last_peered":"2021-02-10 11:40:25.375917","last_clean":"2021-02-10 11:40:25.375917","last_became_active":"2021-02-10 11:40:23.729462","last_became_peered":"2021-02-10 11:40:23.729462","last_unstale":"2021-02-10 11:40:25.375917","last_undegraded":"2021-02-10 11:40:25.375917","last_fullsized":"2021-02-10 11:40:25.375917","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:40:22.718405","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:40:22.718405","last_clean_scrub_stamp":"2021-02-10 11:40:22.718405","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":566,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"43.7","version":"197'14","reported_seq":"30","reported_epoch":"197","state":"active+clean","last_fresh":"2021-02-10 11:41:37.314870","last_change":"2021-02-10 11:40:24.371366","last_active":"2021-02-10 11:41:37.314870","last_peered":"2021-02-10 11:41:37.314870","last_clean":"2021-02-10 11:41:37.314870","last_became_active":"2021-02-10 11:40:24.371106","last_became_peered":"2021-02-10 11:40:24.371106","last_unstale":"2021-02-10 11:41:37.314870","last_undegraded":"2021-02-10 11:41:37.314870","last_fullsized":"2021-02-10 11:41:37.314870","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:40:23.720923","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:40:23.720923","last_clean_scrub_stamp":"2021-02-10 11:40:23.720923","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":14680064,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":7,"num_read_kb":7168,"num_write":14,"num_write_kb":14336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"42.6","version":"197'16","reported_seq":"26","reported_epoch":"197","state":"active+clean","last_fresh":"2021-02-10 11:41:38.724076","last_change":"2021-02-10 11:40:23.729347","last_active":"2021-02-10 11:41:38.724076","last_peered":"2021-02-10 11:41:38.724076","last_clean":"2021-02-10 11:41:38.724076","last_became_active":"2021-02-10 11:40:23.729136","last_became_peered":"2021-02-10 11:40:23.729136","last_unstale":"2021-02-10 11:41:38.724076","last_undegraded":"2021-02-10 11:41:38.724076","last_fullsized":"2021-02-10 11:41:38.724076","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:40:22.718405","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:40:22.718405","last_clean_scrub_stamp":"2021-02-10 11:40:22.718405","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":35,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"43.5","version":"197'23","reported_seq":"41","reported_epoch":"197","state":"active+clean","last_fresh":"2021-02-10 11:41:37.352951","last_change":"2021-02-10 11:40:24.372300","last_active":"2021-02-10 11:41:37.352951","last_peered":"2021-02-10 11:41:37.352951","last_clean":"2021-02-10 11:41:37.352951","last_became_active":"2021-02-10 11:40:24.372091","last_became_peered":"2021-02-10 11:40:24.372091","last_unstale":"2021-02-10 11:41:37.352951","last_undegraded":"2021-02-10 11:41:37.352951","last_fullsized":"2021-02-10 11:41:37.352951","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:40:23.720923","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:40:23.720923","last_clean_scrub_stamp":"2021-02-10 11:40:23.720923","log_size":23,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16777506,"num_objects":18,"num_object_clones":1,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":10,"num_read_kb":10240,"num_write":20,"num_write_kb":16387,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"42.4","version":"197'222","reported_seq":"232","reported_epoch":"197","state":"active+clean","last_fresh":"2021-02-10 11:41:48.724580","last_change":"2021-02-10 11:40:23.728534","last_active":"2021-02-10 11:41:48.724580","last_peered":"2021-02-10 11:41:48.724580","last_clean":"2021-02-10 11:41:48.724580","last_became_active":"2021-02-10 11:40:23.728257","last_became_peered":"2021-02-10 11:40:23.728257","last_unstale":"2021-02-10 11:41:48.724580","last_undegraded":"2021-02-10 11:41:48.724580","last_fullsized":"2021-02-10 11:41:48.724580","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:40:22.718405","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:40:22.718405","last_clean_scrub_stamp":"2021-02-10 11:40:22.718405","log_size":222,"ondisk_log_size":222,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":927945,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":222,"num_write_kb":974,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"43.2","version":"197'20","reported_seq":"37","reported_epoch":"197","state":"active+clean","last_fresh":"2021-02-10 11:41:37.294458","last_change":"2021-02-10 11:40:24.372473","last_active":"2021-02-10 11:41:37.294458","last_peered":"2021-02-10 11:41:37.294458","last_clean":"2021-02-10 11:41:37.294458","last_became_active":"2021-02-10 11:40:24.372214","last_became_peered":"2021-02-10 11:40:24.372214","last_unstale":"2021-02-10 11:41:37.294458","last_undegraded":"2021-02-10 11:41:37.294458","last_fullsized":"2021-02-10 11:41:37.294458","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:40:23.720923","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:40:23.720923","last_clean_scrub_stamp":"2021-02-10 11:40:23.720923","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20971520,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":8,"num_read_kb":8192,"num_write":20,"num_write_kb":20480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"42.3","version":"197'6","reported_seq":"16","reported_epoch":"197","state":"active+clean","last_fresh":"2021-02-10 11:40:25.502736","last_change":"2021-02-10 11:40:23.728906","last_active":"2021-02-10 11:40:25.502736","last_peered":"2021-02-10 11:40:25.502736","last_clean":"2021-02-10 11:40:25.502736","last_became_active":"2021-02-10 11:40:23.728701","last_became_peered":"2021-02-10 11:40:23.728701","last_unstale":"2021-02-10 11:40:25.502736","last_undegraded":"2021-02-10 11:40:25.502736","last_fullsized":"2021-02-10 11:40:25.502736","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:40:22.718405","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:40:22.718405","last_clean_scrub_stamp":"2021-02-10 11:40:22.718405","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"43.3","version":"197'25","reported_seq":"42","reported_epoch":"197","state":"active+clean","last_fresh":"2021-02-10 11:41:37.358591","last_change":"2021-02-10 11:40:24.371035","last_active":"2021-02-10 11:41:37.358591","last_peered":"2021-02-10 11:41:37.358591","last_clean":"2021-02-10 11:41:37.358591","last_became_active":"2021-02-10 11:40:24.370863","last_became_peered":"2021-02-10 11:40:24.370863","last_unstale":"2021-02-10 11:41:37.358591","last_undegraded":"2021-02-10 11:41:37.358591","last_fullsized":"2021-02-10 11:41:37.358591","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:40:23.720923","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:40:23.720923","last_clean_scrub_stamp":"2021-02-10 11:40:23.720923","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":18874505,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":8,"num_read_kb":8192,"num_write":25,"num_write_kb":18436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"42.2","version":"196'1","reported_seq":"11","reported_epoch":"196","state":"active+clean","last_fresh":"2021-02-10 11:40:24.372892","last_change":"2021-02-10 11:40:23.728597","last_active":"2021-02-10 11:40:24.372892","last_peered":"2021-02-10 11:40:24.372892","last_clean":"2021-02-10 11:40:24.372892","last_became_active":"2021-02-10 11:40:23.728297","last_became_peered":"2021-02-10 11:40:23.728297","last_unstale":"2021-02-10 11:40:24.372892","last_undegraded":"2021-02-10 11:40:24.372892","last_fullsized":"2021-02-10 11:40:24.372892","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:40:22.718405","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:40:22.718405","last_clean_scrub_stamp":"2021-02-10 11:40:22.718405","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"43.0","version":"197'19","reported_seq":"38","reported_epoch":"197","state":"active+clean","last_fresh":"2021-02-10 11:41:37.271520","last_change":"2021-02-10 11:40:24.372861","last_active":"2021-02-10 11:41:37.271520","last_peered":"2021-02-10 11:41:37.271520","last_clean":"2021-02-10 11:41:37.271520","last_became_active":"2021-02-10 11:40:24.372579","last_became_peered":"2021-02-10 11:40:24.372579","last_unstale":"2021-02-10 11:41:37.271520","last_undegraded":"2021-02-10 11:41:37.271520","last_fullsized":"2021-02-10 11:41:37.271520","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:40:23.720923","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:40:23.720923","last_clean_scrub_stamp":"2021-02-10 11:40:23.720923","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":18874368,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":10,"num_read_kb":10240,"num_write":18,"num_write_kb":18432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"42.1","version":"197'2","reported_seq":"13","reported_epoch":"197","state":"active+clean","last_fresh":"2021-02-10 11:41:38.383325","last_change":"2021-02-10 11:40:23.728372","last_active":"2021-02-10 11:41:38.383325","last_peered":"2021-02-10 11:41:38.383325","last_clean":"2021-02-10 11:41:38.383325","last_became_active":"2021-02-10 11:40:23.728061","last_became_peered":"2021-02-10 11:40:23.728061","last_unstale":"2021-02-10 11:41:38.383325","last_undegraded":"2021-02-10 11:41:38.383325","last_fullsized":"2021-02-10 11:41:38.383325","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:40:22.718405","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:40:22.718405","last_clean_scrub_stamp":"2021-02-10 11:40:22.718405","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":101,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"43.1","version":"197'11","reported_seq":"26","reported_epoch":"197","state":"active+clean","last_fresh":"2021-02-10 11:41:37.334933","last_change":"2021-02-10 11:40:24.373836","last_active":"2021-02-10 11:41:37.334933","last_peered":"2021-02-10 11:41:37.334933","last_clean":"2021-02-10 11:41:37.334933","last_became_active":"2021-02-10 11:40:24.373558","last_became_peered":"2021-02-10 11:40:24.373558","last_unstale":"2021-02-10 11:41:37.334933","last_undegraded":"2021-02-10 11:41:37.334933","last_fullsized":"2021-02-10 11:41:37.334933","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:40:23.720923","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:40:23.720923","last_clean_scrub_stamp":"2021-02-10 11:40:23.720923","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":10485760,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":6,"num_read_kb":6144,"num_write":10,"num_write_kb":10240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"42.0","version":"196'8","reported_seq":"18","reported_epoch":"196","state":"active+clean","last_fresh":"2021-02-10 11:40:24.377584","last_change":"2021-02-10 11:40:23.729211","last_active":"2021-02-10 11:40:24.377584","last_peered":"2021-02-10 11:40:24.377584","last_clean":"2021-02-10 11:40:24.377584","last_became_active":"2021-02-10 11:40:23.728990","last_became_peered":"2021-02-10 11:40:23.728990","last_unstale":"2021-02-10 11:40:24.377584","last_undegraded":"2021-02-10 11:40:24.377584","last_fullsized":"2021-02-10 11:40:24.377584","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:40:22.718405","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:40:22.718405","last_clean_scrub_stamp":"2021-02-10 11:40:22.718405","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":"190","reported_epoch":"196","state":"active+clean","last_fresh":"2021-02-10 11:40:24.371416","last_change":"2021-02-10 11:03:38.737712","last_active":"2021-02-10 11:40:24.371416","last_peered":"2021-02-10 11:40:24.371416","last_clean":"2021-02-10 11:40:24.371416","last_became_active":"2021-02-10 11:03:38.737435","last_became_peered":"2021-02-10 11:03:38.737435","last_unstale":"2021-02-10 11:40:24.371416","last_undegraded":"2021-02-10 11:40:24.371416","last_fullsized":"2021-02-10 11:40:24.371416","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:03:37.729316","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:03:37.729316","last_clean_scrub_stamp":"2021-02-10 11:03:37.729316","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":"190","reported_epoch":"196","state":"active+clean","last_fresh":"2021-02-10 11:40:24.371275","last_change":"2021-02-10 11:03:38.738386","last_active":"2021-02-10 11:40:24.371275","last_peered":"2021-02-10 11:40:24.371275","last_clean":"2021-02-10 11:40:24.371275","last_became_active":"2021-02-10 11:03:38.738144","last_became_peered":"2021-02-10 11:03:38.738144","last_unstale":"2021-02-10 11:40:24.371275","last_undegraded":"2021-02-10 11:40:24.371275","last_fullsized":"2021-02-10 11:40:24.371275","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:03:37.729316","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:03:37.729316","last_clean_scrub_stamp":"2021-02-10 11:03:37.729316","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":"188","reported_epoch":"196","state":"active+clean","last_fresh":"2021-02-10 11:40:24.373770","last_change":"2021-02-10 11:03:38.738326","last_active":"2021-02-10 11:40:24.373770","last_peered":"2021-02-10 11:40:24.373770","last_clean":"2021-02-10 11:40:24.373770","last_became_active":"2021-02-10 11:03:38.738088","last_became_peered":"2021-02-10 11:03:38.738088","last_unstale":"2021-02-10 11:40:24.373770","last_undegraded":"2021-02-10 11:40:24.373770","last_fullsized":"2021-02-10 11:40:24.373770","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:03:37.729316","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:03:37.729316","last_clean_scrub_stamp":"2021-02-10 11:03:37.729316","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":"180","reported_epoch":"196","state":"active+clean","last_fresh":"2021-02-10 11:40:24.369187","last_change":"2021-02-10 11:03:38.740643","last_active":"2021-02-10 11:40:24.369187","last_peered":"2021-02-10 11:40:24.369187","last_clean":"2021-02-10 11:40:24.369187","last_became_active":"2021-02-10 11:03:38.740334","last_became_peered":"2021-02-10 11:03:38.740334","last_unstale":"2021-02-10 11:40:24.369187","last_undegraded":"2021-02-10 11:40:24.369187","last_fullsized":"2021-02-10 11:40:24.369187","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:03:37.729316","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:03:37.729316","last_clean_scrub_stamp":"2021-02-10 11:03:37.729316","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"180","reported_epoch":"196","state":"active+clean","last_fresh":"2021-02-10 11:40:24.372572","last_change":"2021-02-10 11:03:38.739319","last_active":"2021-02-10 11:40:24.372572","last_peered":"2021-02-10 11:40:24.372572","last_clean":"2021-02-10 11:40:24.372572","last_became_active":"2021-02-10 11:03:38.739150","last_became_peered":"2021-02-10 11:03:38.739150","last_unstale":"2021-02-10 11:40:24.372572","last_undegraded":"2021-02-10 11:40:24.372572","last_fullsized":"2021-02-10 11:40:24.372572","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:03:37.729316","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:03:37.729316","last_clean_scrub_stamp":"2021-02-10 11:03:37.729316","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"43.4","version":"197'10","reported_seq":"25","reported_epoch":"197","state":"active+clean","last_fresh":"2021-02-10 11:41:37.279158","last_change":"2021-02-10 11:40:24.371458","last_active":"2021-02-10 11:41:37.279158","last_peered":"2021-02-10 11:41:37.279158","last_clean":"2021-02-10 11:41:37.279158","last_became_active":"2021-02-10 11:40:24.371177","last_became_peered":"2021-02-10 11:40:24.371177","last_unstale":"2021-02-10 11:41:37.279158","last_undegraded":"2021-02-10 11:41:37.279158","last_fullsized":"2021-02-10 11:41:37.279158","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:40:23.720923","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:40:23.720923","last_clean_scrub_stamp":"2021-02-10 11:40:23.720923","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":10485760,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":6,"num_read_kb":6144,"num_write":10,"num_write_kb":10240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"42.5","version":"197'9","reported_seq":"19","reported_epoch":"197","state":"active+clean","last_fresh":"2021-02-10 11:41:38.383320","last_change":"2021-02-10 11:40:23.729552","last_active":"2021-02-10 11:41:38.383320","last_peered":"2021-02-10 11:41:38.383320","last_clean":"2021-02-10 11:41:38.383320","last_became_active":"2021-02-10 11:40:23.729375","last_became_peered":"2021-02-10 11:40:23.729375","last_unstale":"2021-02-10 11:41:38.383320","last_undegraded":"2021-02-10 11:41:38.383320","last_fullsized":"2021-02-10 11:41:38.383320","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:40:22.718405","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:40:22.718405","last_clean_scrub_stamp":"2021-02-10 11:40:22.718405","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":160,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":"175","reported_epoch":"196","state":"active+clean","last_fresh":"2021-02-10 11:40:24.371231","last_change":"2021-02-10 11:03:38.738931","last_active":"2021-02-10 11:40:24.371231","last_peered":"2021-02-10 11:40:24.371231","last_clean":"2021-02-10 11:40:24.371231","last_became_active":"2021-02-10 11:03:38.738675","last_became_peered":"2021-02-10 11:03:38.738675","last_unstale":"2021-02-10 11:40:24.371231","last_undegraded":"2021-02-10 11:40:24.371231","last_fullsized":"2021-02-10 11:40:24.371231","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:03:37.729316","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:03:37.729316","last_clean_scrub_stamp":"2021-02-10 11:03:37.729316","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":"180","reported_epoch":"196","state":"active+clean","last_fresh":"2021-02-10 11:40:24.372508","last_change":"2021-02-10 11:03:38.739203","last_active":"2021-02-10 11:40:24.372508","last_peered":"2021-02-10 11:40:24.372508","last_clean":"2021-02-10 11:40:24.372508","last_became_active":"2021-02-10 11:03:38.739032","last_became_peered":"2021-02-10 11:03:38.739032","last_unstale":"2021-02-10 11:40:24.372508","last_undegraded":"2021-02-10 11:40:24.372508","last_fullsized":"2021-02-10 11:40:24.372508","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:03:37.729316","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:03:37.729316","last_clean_scrub_stamp":"2021-02-10 11:03:37.729316","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":"178","reported_epoch":"196","state":"active+clean","last_fresh":"2021-02-10 11:40:24.370029","last_change":"2021-02-10 11:03:38.740597","last_active":"2021-02-10 11:40:24.370029","last_peered":"2021-02-10 11:40:24.370029","last_clean":"2021-02-10 11:40:24.370029","last_became_active":"2021-02-10 11:03:38.740308","last_became_peered":"2021-02-10 11:03:38.740308","last_unstale":"2021-02-10 11:40:24.370029","last_undegraded":"2021-02-10 11:40:24.370029","last_fullsized":"2021-02-10 11:40:24.370029","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:03:37.729316","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:03:37.729316","last_clean_scrub_stamp":"2021-02-10 11:03:37.729316","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":43,"num_pg":8,"stat_sum":{"num_bytes":134218155,"num_objects":131,"num_object_clones":1,"num_object_copies":262,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":131,"num_whiteouts":0,"num_read":64,"num_read_kb":65536,"num_write":139,"num_write_kb":131079,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":268460032,"data_stored":268436310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":144,"ondisk_log_size":144,"up":16,"acting":16,"num_store_stats":8},{"poolid":42,"num_pg":8,"stat_sum":{"num_bytes":928806,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":292,"num_write_kb":1001,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1925120,"data_stored":1857612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":272,"ondisk_log_size":272,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575312,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1087952,"kb_used_data":39368,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93283888,"statfs":{"total":96636764160,"available":95522701312,"internally_reserved":1073741824,"allocated":40312832,"data_stored":38844297,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 11:41:47 2021","interfaces":[{"interface":"back","average":{"1min":0.91900000000000004,"5min":0.92600000000000005,"15min":0.94999999999999996},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.2070000000000001,"5min":1.4630000000000001,"15min":1.4630000000000001},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.92500000000000004,"5min":0.91600000000000004,"15min":0.92700000000000005},"min":{"1min":0.502,"5min":0.38800000000000001,"15min":0.36799999999999999},"max":{"1min":1.141,"5min":1.367,"15min":1.367},"last":0.47399999999999998}]},{"osd":1,"last update":"Wed Feb 10 11:41:47 2021","interfaces":[{"interface":"back","average":{"1min":0.81699999999999995,"5min":0.79700000000000004,"15min":0.81899999999999995},"min":{"1min":0.434,"5min":0.38,"15min":0.38},"max":{"1min":1.2689999999999999,"5min":1.361,"15min":1.361},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.83699999999999997,"5min":0.81299999999999994,"15min":0.82699999999999996},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1.0649999999999999,"5min":1.288,"15min":1.508},"last":0.57199999999999995}]},{"osd":2,"last update":"Wed Feb 10 11:41:47 2021","interfaces":[{"interface":"back","average":{"1min":0.81100000000000005,"5min":0.81999999999999995,"15min":0.82299999999999995},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":1.1870000000000001,"5min":1.323,"15min":1.3859999999999999},"last":0.5},{"interface":"front","average":{"1min":0.85699999999999998,"5min":0.85299999999999998,"15min":0.86699999999999999},"min":{"1min":0.49199999999999999,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1.1040000000000001,"5min":1.2370000000000001,"15min":1.329},"last":0.80100000000000005}]},{"osd":3,"last update":"Wed Feb 10 11:41:47 2021","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":0.83099999999999996,"15min":0.84299999999999997},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.40500000000000003},"max":{"1min":1.089,"5min":1.272,"15min":1.4139999999999999},"last":0.63},{"interface":"front","average":{"1min":0.86899999999999999,"5min":0.89000000000000001,"15min":0.89400000000000002},"min":{"1min":0.60999999999999999,"5min":0.47799999999999998,"15min":0.45500000000000002},"max":{"1min":1.3220000000000001,"5min":1.3220000000000001,"15min":1.389},"last":0.745}]},{"osd":4,"last update":"Wed Feb 10 11:41:47 2021","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.79400000000000004,"15min":0.80600000000000005},"min":{"1min":0.47799999999999998,"5min":0.47099999999999997,"15min":0.46800000000000003},"max":{"1min":1.1279999999999999,"5min":1.1279999999999999,"15min":1.4330000000000001},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.82899999999999996,"5min":0.81899999999999995,"15min":0.83099999999999996},"min":{"1min":0.48899999999999999,"5min":0.438,"15min":0.438},"max":{"1min":1.1060000000000001,"5min":1.1970000000000001,"15min":1.5489999999999999},"last":0.72599999999999998}]},{"osd":5,"last update":"Wed Feb 10 11:41:47 2021","interfaces":[{"interface":"back","average":{"1min":0.76800000000000002,"5min":0.76300000000000001,"15min":0.753},"min":{"1min":0.54800000000000004,"5min":0.49299999999999999,"15min":0.46899999999999997},"max":{"1min":1.1579999999999999,"5min":1.3360000000000001,"15min":1.3360000000000001},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.79000000000000004,"15min":0.77100000000000002},"min":{"1min":0.52800000000000002,"5min":0.47699999999999998,"15min":0.44700000000000001},"max":{"1min":1.1319999999999999,"5min":1.232,"15min":1.232},"last":0.81299999999999994}]},{"osd":6,"last update":"Wed Feb 10 11:41:47 2021","interfaces":[{"interface":"back","average":{"1min":0.876,"5min":0.85399999999999998,"15min":0.84099999999999997},"min":{"1min":0.67100000000000004,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.21,"5min":1.329,"15min":1.329},"last":0.76300000000000001},{"interface":"front","average":{"1min":0.82999999999999996,"5min":0.82299999999999995,"15min":0.80300000000000005},"min":{"1min":0.55000000000000004,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.0780000000000001,"5min":1.429,"15min":1.429},"last":0.77700000000000002}]}]},{"osd":6,"up_from":13,"seq":55834575312,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1070420,"kb_used_data":21836,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93301420,"statfs":{"total":96636764160,"available":95540654080,"internally_reserved":1073741824,"allocated":22360064,"data_stored":20896951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 11:40:54 2021","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.78100000000000003,"15min":0.79700000000000004},"min":{"1min":0.42199999999999999,"5min":0.36599999999999999,"15min":0.35599999999999998},"max":{"1min":1.1879999999999999,"5min":1.254,"15min":1.4099999999999999},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.79200000000000004,"5min":0.76100000000000001,"15min":0.79700000000000004},"min":{"1min":0.48299999999999998,"5min":0.34999999999999998,"15min":0.34799999999999998},"max":{"1min":1.1539999999999999,"5min":1.2969999999999999,"15min":1.3140000000000001},"last":0.57199999999999995}]},{"osd":1,"last update":"Wed Feb 10 11:40:54 2021","interfaces":[{"interface":"back","average":{"1min":0.86899999999999999,"5min":0.82199999999999995,"15min":0.85099999999999998},"min":{"1min":0.58499999999999996,"5min":0.41299999999999998,"15min":0.37},"max":{"1min":1.2,"5min":1.224,"15min":1.427},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.82099999999999995,"5min":0.80700000000000005,"15min":0.82799999999999996},"min":{"1min":0.47499999999999998,"5min":0.371,"15min":0.371},"max":{"1min":1.1830000000000001,"5min":1.2230000000000001,"15min":1.2809999999999999},"last":0.503}]},{"osd":2,"last update":"Wed Feb 10 11:40:54 2021","interfaces":[{"interface":"back","average":{"1min":0.77700000000000002,"5min":0.76600000000000001,"15min":0.78800000000000003},"min":{"1min":0.44400000000000001,"5min":0.434,"15min":0.39000000000000001},"max":{"1min":1.1559999999999999,"5min":1.2569999999999999,"15min":1.2569999999999999},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.82699999999999996,"5min":0.81699999999999995,"15min":0.85599999999999998},"min":{"1min":0.45900000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":1.1499999999999999,"5min":1.3029999999999999,"15min":1.431},"last":0.77400000000000002}]},{"osd":3,"last update":"Wed Feb 10 11:40:54 2021","interfaces":[{"interface":"back","average":{"1min":0.89000000000000001,"5min":0.91100000000000003,"15min":0.94199999999999995},"min":{"1min":0.433,"5min":0.433,"15min":0.42099999999999999},"max":{"1min":1.2589999999999999,"5min":1.4319999999999999,"15min":1.4319999999999999},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.88400000000000001,"5min":0.91800000000000004,"15min":0.93799999999999994},"min":{"1min":0.51700000000000002,"5min":0.499,"15min":0.49099999999999999},"max":{"1min":1.2230000000000001,"5min":1.3440000000000001,"15min":1.3999999999999999},"last":0.747}]},{"osd":4,"last update":"Wed Feb 10 11:40:54 2021","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.70699999999999996,"15min":0.71299999999999997},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.35699999999999998},"max":{"1min":0.96099999999999997,"5min":1.099,"15min":1.163},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.73899999999999999,"5min":0.72999999999999998,"15min":0.73099999999999998},"min":{"1min":0.52700000000000002,"5min":0.373,"15min":0.29699999999999999},"max":{"1min":1.119,"5min":1.119,"15min":1.2769999999999999},"last":0.64900000000000002}]},{"osd":5,"last update":"Wed Feb 10 11:40:54 2021","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.36399999999999999},"max":{"1min":1.125,"5min":1.2649999999999999,"15min":1.2649999999999999},"last":0.81200000000000006},{"interface":"front","average":{"1min":0.87,"5min":0.86499999999999999,"15min":0.876},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.53000000000000003},"max":{"1min":1.198,"5min":1.198,"15min":1.468},"last":0.83699999999999997}]},{"osd":7,"last update":"Wed Feb 10 11:40:54 2021","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":0.79300000000000004,"15min":0.80400000000000005},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":1.1859999999999999,"5min":1.1859999999999999,"15min":1.452},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.76500000000000001,"15min":0.78000000000000003},"min":{"1min":0.48499999999999999,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.137,"5min":1.1499999999999999,"15min":1.3220000000000001},"last":0.68000000000000005}]}]},{"osd":5,"up_from":13,"seq":55834575312,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1083844,"kb_used_data":35260,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93287996,"statfs":{"total":96636764160,"available":95526907904,"internally_reserved":1073741824,"allocated":36106240,"data_stored":34649196,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 11:41:42 2021","interfaces":[{"interface":"back","average":{"1min":0.93600000000000005,"5min":0.85799999999999998,"15min":0.88200000000000001},"min":{"1min":0.49099999999999999,"5min":0.40000000000000002,"15min":0.34300000000000003},"max":{"1min":1.113,"5min":1.222,"15min":1.2829999999999999},"last":1.0940000000000001},{"interface":"front","average":{"1min":0.995,"5min":0.90900000000000003,"15min":0.92100000000000004},"min":{"1min":0.73699999999999999,"5min":0.38400000000000001,"15min":0.33100000000000002},"max":{"1min":1.2589999999999999,"5min":1.27,"15min":1.444},"last":1.1080000000000001}]},{"osd":1,"last update":"Wed Feb 10 11:41:42 2021","interfaces":[{"interface":"back","average":{"1min":0.81899999999999995,"5min":0.75900000000000001,"15min":0.78100000000000003},"min":{"1min":0.44400000000000001,"5min":0.432,"15min":0.36499999999999999},"max":{"1min":1.081,"5min":1.2030000000000001,"15min":1.3169999999999999},"last":0.90300000000000002},{"interface":"front","average":{"1min":0.82699999999999996,"5min":0.77200000000000002,"15min":0.80200000000000005},"min":{"1min":0.42899999999999999,"5min":0.41499999999999998,"15min":0.40400000000000003},"max":{"1min":1.0049999999999999,"5min":1.0980000000000001,"15min":1.2849999999999999},"last":0.77500000000000002}]},{"osd":2,"last update":"Wed Feb 10 11:41:42 2021","interfaces":[{"interface":"back","average":{"1min":0.89000000000000001,"5min":0.84899999999999998,"15min":0.89400000000000002},"min":{"1min":0.51200000000000001,"5min":0.47299999999999998,"15min":0.38400000000000001},"max":{"1min":1.202,"5min":1.258,"15min":1.534},"last":0.91400000000000003},{"interface":"front","average":{"1min":0.95099999999999996,"5min":0.91300000000000003,"15min":0.92500000000000004},"min":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.42199999999999999},"max":{"1min":1.232,"5min":1.3100000000000001,"15min":1.357},"last":1.0780000000000001}]},{"osd":3,"last update":"Wed Feb 10 11:41:42 2021","interfaces":[{"interface":"back","average":{"1min":0.93999999999999995,"5min":0.88800000000000001,"15min":0.93000000000000005},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1.2729999999999999,"5min":1.367,"15min":1.6399999999999999},"last":0.83399999999999996},{"interface":"front","average":{"1min":1,"5min":0.93000000000000005,"15min":0.95099999999999996},"min":{"1min":0.54600000000000004,"5min":0.51300000000000001,"15min":0.48399999999999999},"max":{"1min":1.252,"5min":1.252,"15min":1.3640000000000001},"last":1.0249999999999999}]},{"osd":4,"last update":"Wed Feb 10 11:41:42 2021","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.68000000000000005,"15min":0.69399999999999995},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.95099999999999996,"5min":1.002,"15min":1.127},"last":1.034},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.69499999999999995,"15min":0.72499999999999998},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.97199999999999998,"5min":1.0620000000000001,"15min":1.3009999999999999},"last":0.58799999999999997}]},{"osd":6,"last update":"Wed Feb 10 11:41:42 2021","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.71299999999999997,"15min":0.71399999999999997},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.86399999999999999,"5min":1.125,"15min":1.139},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.75900000000000001,"15min":0.76800000000000002},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.48099999999999998},"max":{"1min":1.036,"5min":1.2,"15min":1.2170000000000001},"last":0.64800000000000002}]},{"osd":7,"last update":"Wed Feb 10 11:41:42 2021","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.78800000000000003,"15min":0.80300000000000005},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.0029999999999999,"5min":1.262,"15min":1.347},"last":0.745},{"interface":"front","average":{"1min":0.77700000000000002,"5min":0.81399999999999995,"15min":0.81899999999999995},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1.0369999999999999,"5min":1.117,"15min":1.2669999999999999},"last":0.79900000000000004}]}]},{"osd":4,"up_from":13,"seq":55834575312,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1074512,"kb_used_data":25928,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93297328,"statfs":{"total":96636764160,"available":95536463872,"internally_reserved":1073741824,"allocated":26550272,"data_stored":25091221,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 11:41:48 2021","interfaces":[{"interface":"back","average":{"1min":0.82699999999999996,"5min":0.80100000000000005,"15min":0.80700000000000005},"min":{"1min":0.51700000000000002,"5min":0.41099999999999998,"15min":0.32200000000000001},"max":{"1min":1.1379999999999999,"5min":1.2430000000000001,"15min":1.3979999999999999},"last":0.81999999999999995},{"interface":"front","average":{"1min":0.79200000000000004,"5min":0.79900000000000004,"15min":0.81299999999999994},"min":{"1min":0.442,"5min":0.379,"15min":0.379},"max":{"1min":1.099,"5min":1.3320000000000001,"15min":1.3320000000000001},"last":0.89900000000000002}]},{"osd":1,"last update":"Wed Feb 10 11:41:48 2021","interfaces":[{"interface":"back","average":{"1min":0.81799999999999995,"5min":0.81299999999999994,"15min":0.82799999999999996},"min":{"1min":0.53500000000000003,"5min":0.46899999999999997,"15min":0.307},"max":{"1min":1.23,"5min":1.23,"15min":1.417},"last":0.84799999999999998},{"interface":"front","average":{"1min":0.84099999999999997,"5min":0.79700000000000004,"15min":0.82099999999999995},"min":{"1min":0.53100000000000003,"5min":0.434,"15min":0.38300000000000001},"max":{"1min":1.0880000000000001,"5min":1.2929999999999999,"15min":1.387},"last":0.91200000000000003}]},{"osd":2,"last update":"Wed Feb 10 11:41:48 2021","interfaces":[{"interface":"back","average":{"1min":0.95899999999999996,"5min":0.90900000000000003,"15min":0.92800000000000005},"min":{"1min":0.57999999999999996,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1.1879999999999999,"5min":1.2949999999999999,"15min":1.4239999999999999},"last":1.0409999999999999},{"interface":"front","average":{"1min":0.95599999999999996,"5min":0.97599999999999998,"15min":0.98499999999999999},"min":{"1min":0.63700000000000001,"5min":0.57699999999999996,"15min":0.499},"max":{"1min":1.369,"5min":1.3979999999999999,"15min":1.571},"last":0.97999999999999998}]},{"osd":3,"last update":"Wed Feb 10 11:41:48 2021","interfaces":[{"interface":"back","average":{"1min":0.82299999999999995,"5min":0.80300000000000005,"15min":0.82799999999999996},"min":{"1min":0.54400000000000004,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":1.196,"5min":1.196,"15min":1.4550000000000001},"last":0.86399999999999999},{"interface":"front","average":{"1min":0.85099999999999998,"5min":0.84399999999999997,"15min":0.86799999999999999},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.39000000000000001},"max":{"1min":1.054,"5min":1.2190000000000001,"15min":1.325},"last":0.88400000000000001}]},{"osd":5,"last update":"Wed Feb 10 11:41:48 2021","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.71999999999999997,"15min":0.70699999999999996},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.39500000000000002},"max":{"1min":0.98299999999999998,"5min":1.0669999999999999,"15min":1.109},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.68899999999999995,"15min":0.69999999999999996},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.36599999999999999},"max":{"1min":1.004,"5min":1.0920000000000001,"15min":1.272},"last":0.48399999999999999}]},{"osd":6,"last update":"Wed Feb 10 11:41:48 2021","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.73699999999999999,"15min":0.74099999999999999},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":1.1080000000000001,"5min":1.1080000000000001,"15min":1.254},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.78500000000000003,"15min":0.79300000000000004},"min":{"1min":0.504,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.0309999999999999,"5min":1.2350000000000001,"15min":1.262},"last":0.624}]},{"osd":7,"last update":"Wed Feb 10 11:41:48 2021","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.77700000000000002,"15min":0.79500000000000004},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.0620000000000001,"5min":1.1180000000000001,"15min":1.3240000000000001},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.78400000000000003,"5min":0.84799999999999998,"15min":0.84299999999999997},"min":{"1min":0.57899999999999996,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.069,"5min":1.2330000000000001,"15min":1.3129999999999999},"last":0.65400000000000003}]}]},{"osd":0,"up_from":13,"seq":55834575311,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1061312,"kb_used_data":12728,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93310528,"statfs":{"total":96636764160,"available":95549980672,"internally_reserved":1073741824,"allocated":13033472,"data_stored":11580398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Feb 10 11:40:55 2021","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.503,"15min":0.49199999999999999},"min":{"1min":0.32900000000000001,"5min":0.32400000000000001,"15min":0.30299999999999999},"max":{"1min":0.72499999999999998,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.44},{"interface":"front","average":{"1min":0.501,"5min":0.499,"15min":0.49199999999999999},"min":{"1min":0.36499999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.79000000000000004,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.501}]},{"osd":2,"last update":"Wed Feb 10 11:40:55 2021","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.54700000000000004,"15min":0.54200000000000004},"min":{"1min":0.379,"5min":0.318,"15min":0.318},"max":{"1min":0.83799999999999997,"5min":0.94799999999999995,"15min":1.0249999999999999},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54300000000000004,"15min":0.55100000000000005},"min":{"1min":0.41999999999999998,"5min":0.40600000000000003,"15min":0.35599999999999998},"max":{"1min":0.747,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.55500000000000005}]},{"osd":3,"last update":"Wed Feb 10 11:40:55 2021","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.52000000000000002,"15min":0.51200000000000001},"min":{"1min":0.371,"5min":0.34799999999999998,"15min":0.33900000000000002},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.84499999999999997},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61099999999999999},"min":{"1min":0.499,"5min":0.45100000000000001,"15min":0.36299999999999999},"max":{"1min":0.83299999999999996,"5min":0.878,"15min":0.97599999999999998},"last":0.56499999999999995}]},{"osd":4,"last update":"Wed Feb 10 11:40:55 2021","interfaces":[{"interface":"back","average":{"1min":0.871,"5min":1.0029999999999999,"15min":0.98199999999999998},"min":{"1min":0.499,"5min":0.48799999999999999,"15min":0.41199999999999998},"max":{"1min":1.1859999999999999,"5min":1.5189999999999999,"15min":1.53},"last":1.2070000000000001},{"interface":"front","average":{"1min":0.88800000000000001,"5min":0.99299999999999999,"15min":0.98399999999999999},"min":{"1min":0.56100000000000005,"5min":0.498,"15min":0.498},"max":{"1min":1.224,"5min":1.4390000000000001,"15min":1.4710000000000001},"last":1.0900000000000001}]},{"osd":5,"last update":"Wed Feb 10 11:40:55 2021","interfaces":[{"interface":"back","average":{"1min":0.93400000000000005,"5min":1.0209999999999999,"15min":1.008},"min":{"1min":0.626,"5min":0.59599999999999997,"15min":0.48099999999999998},"max":{"1min":1.353,"5min":1.353,"15min":1.484},"last":1.04},{"interface":"front","average":{"1min":0.91900000000000004,"5min":1.0209999999999999,"15min":1.002},"min":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.51700000000000002},"max":{"1min":1.3049999999999999,"5min":1.4059999999999999,"15min":1.4359999999999999},"last":1.1220000000000001}]},{"osd":6,"last update":"Wed Feb 10 11:40:55 2021","interfaces":[{"interface":"back","average":{"1min":0.95999999999999996,"5min":1.0409999999999999,"15min":1.016},"min":{"1min":0.66500000000000004,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.3169999999999999,"5min":1.417,"15min":1.417},"last":1.1910000000000001},{"interface":"front","average":{"1min":0.95399999999999996,"5min":1.0449999999999999,"15min":1.024},"min":{"1min":0.629,"5min":0.629,"15min":0.55100000000000005},"max":{"1min":1.2350000000000001,"5min":1.4870000000000001,"15min":1.4870000000000001},"last":1.24}]},{"osd":7,"last update":"Wed Feb 10 11:40:55 2021","interfaces":[{"interface":"back","average":{"1min":0.97899999999999998,"5min":1.0589999999999999,"15min":1.038},"min":{"1min":0.71099999999999997,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":1.286,"5min":1.726,"15min":1.726},"last":1.157},{"interface":"front","average":{"1min":0.96599999999999997,"5min":1.026,"15min":1.0069999999999999},"min":{"1min":0.66600000000000004,"5min":0.62,"15min":0.60499999999999998},"max":{"1min":1.2010000000000001,"5min":1.6080000000000001,"15min":1.6080000000000001},"last":1.03}]}]},{"osd":1,"up_from":13,"seq":55834575312,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1092044,"kb_used_data":43456,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93279796,"statfs":{"total":96636764160,"available":95518511104,"internally_reserved":1073741824,"allocated":44498944,"data_stored":43038035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 11:40:58 2021","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.60199999999999998,"15min":0.60299999999999998},"min":{"1min":0.33700000000000002,"5min":0.32600000000000001,"15min":0.32200000000000001},"max":{"1min":1.0169999999999999,"5min":1.056,"15min":1.119},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.60399999999999998,"15min":0.62},"min":{"1min":0.41299999999999998,"5min":0.40500000000000003,"15min":0.316},"max":{"1min":0.88500000000000001,"5min":1.038,"15min":1.038},"last":0.621}]},{"osd":2,"last update":"Wed Feb 10 11:40:58 2021","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.48599999999999999,"15min":0.48799999999999999},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.30499999999999999},"max":{"1min":0.64000000000000001,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.55500000000000005,"15min":0.55400000000000005},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.77100000000000002,"5min":1.1399999999999999,"15min":1.1399999999999999},"last":0.44700000000000001}]},{"osd":3,"last update":"Wed Feb 10 11:40:58 2021","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.60799999999999998,"15min":0.60899999999999999},"min":{"1min":0.38400000000000001,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.94199999999999995,"5min":1.244,"15min":1.244},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.35099999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.71699999999999997,"5min":1.155,"15min":1.155},"last":0.52300000000000002}]},{"osd":4,"last update":"Wed Feb 10 11:40:58 2021","interfaces":[{"interface":"back","average":{"1min":0.89100000000000001,"5min":0.95399999999999996,"15min":0.96699999999999997},"min":{"1min":0.47799999999999998,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.3220000000000001,"5min":1.5229999999999999,"15min":1.5229999999999999},"last":0.92700000000000005},{"interface":"front","average":{"1min":0.92800000000000005,"5min":0.99099999999999999,"15min":0.997},"min":{"1min":0.47599999999999998,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.2709999999999999,"5min":1.508,"15min":1.508},"last":0.95899999999999996}]},{"osd":5,"last update":"Wed Feb 10 11:40:58 2021","interfaces":[{"interface":"back","average":{"1min":0.91600000000000004,"5min":0.997,"15min":0.98899999999999999},"min":{"1min":0.502,"5min":0.496,"15min":0.496},"max":{"1min":1.2430000000000001,"5min":1.5329999999999999,"15min":1.6279999999999999},"last":0.91700000000000004},{"interface":"front","average":{"1min":0.93300000000000005,"5min":0.98399999999999999,"15min":0.998},"min":{"1min":0.57099999999999995,"5min":0.56999999999999995,"15min":0.46899999999999997},"max":{"1min":1.276,"5min":1.413,"15min":1.5760000000000001},"last":1.089}]},{"osd":6,"last update":"Wed Feb 10 11:40:58 2021","interfaces":[{"interface":"back","average":{"1min":0.93200000000000005,"5min":1.0149999999999999,"15min":1.0049999999999999},"min":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"max":{"1min":1.2410000000000001,"5min":1.3740000000000001,"15min":1.4059999999999999},"last":0.90700000000000003},{"interface":"front","average":{"1min":0.94799999999999995,"5min":1.0289999999999999,"15min":1.0269999999999999},"min":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":1.2989999999999999,"5min":1.429,"15min":1.456},"last":0.97799999999999998}]},{"osd":7,"last update":"Wed Feb 10 11:40:58 2021","interfaces":[{"interface":"back","average":{"1min":0.96199999999999997,"5min":1.036,"15min":1.038},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":1.298,"5min":1.472,"15min":1.786},"last":1.042},{"interface":"front","average":{"1min":0.93700000000000006,"5min":0.99199999999999999,"15min":1.0109999999999999},"min":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.56599999999999995},"max":{"1min":1.284,"5min":1.415,"15min":1.494},"last":1.1539999999999999}]}]},{"osd":2,"up_from":13,"seq":55834575311,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1122768,"kb_used_data":74184,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93249072,"statfs":{"total":96636764160,"available":95487049728,"internally_reserved":1073741824,"allocated":75964416,"data_stored":74495787,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 11:41:44 2021","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.63700000000000001,"15min":0.628},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":1.0960000000000001,"5min":1.0960000000000001,"15min":1.103},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.63300000000000001,"15min":0.63},"min":{"1min":0.41599999999999998,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.99199999999999999,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.66300000000000003}]},{"osd":1,"last update":"Wed Feb 10 11:41:44 2021","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53300000000000003,"15min":0.52900000000000003},"min":{"1min":0.379,"5min":0.34200000000000003,"15min":0.32000000000000001},"max":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":1.0509999999999999},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.53100000000000003,"15min":0.51900000000000002},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.33400000000000002},"max":{"1min":0.68400000000000005,"5min":0.82599999999999996,"15min":0.88800000000000001},"last":0.499}]},{"osd":3,"last update":"Wed Feb 10 11:41:44 2021","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51200000000000001,"15min":0.52000000000000002},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.63600000000000001,"5min":0.69099999999999995,"15min":0.78100000000000003},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.58499999999999996,"15min":0.60299999999999998},"min":{"1min":0.438,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.79200000000000004,"5min":0.88600000000000001,"15min":1.0289999999999999},"last":0.57599999999999996}]},{"osd":4,"last update":"Wed Feb 10 11:41:44 2021","interfaces":[{"interface":"back","average":{"1min":0.97499999999999998,"5min":1.091,"15min":1.081},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.335,"5min":1.4430000000000001,"15min":1.4690000000000001},"last":0.90200000000000002},{"interface":"front","average":{"1min":0.96699999999999997,"5min":1.101,"15min":1.095},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.373},"max":{"1min":1.359,"5min":1.419,"15min":1.77},"last":0.95799999999999996}]},{"osd":5,"last update":"Wed Feb 10 11:41:44 2021","interfaces":[{"interface":"back","average":{"1min":0.89400000000000002,"5min":0.995,"15min":0.98599999999999999},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.296,"5min":1.47,"15min":1.47},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.86099999999999999,"5min":0.97099999999999997,"15min":0.98899999999999999},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":1.3220000000000001,"5min":1.4199999999999999,"15min":1.613},"last":0.85999999999999999}]},{"osd":6,"last update":"Wed Feb 10 11:41:44 2021","interfaces":[{"interface":"back","average":{"1min":0.89400000000000002,"5min":0.98399999999999999,"15min":0.996},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.2509999999999999,"5min":1.569,"15min":1.569},"last":0.91900000000000004},{"interface":"front","average":{"1min":0.92200000000000004,"5min":1.002,"15min":0.998},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.1519999999999999,"5min":1.3999999999999999,"15min":1.504},"last":0.85199999999999998}]},{"osd":7,"last update":"Wed Feb 10 11:41:44 2021","interfaces":[{"interface":"back","average":{"1min":0.89600000000000002,"5min":0.996,"15min":1.0049999999999999},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.2290000000000001,"5min":1.4810000000000001,"15min":1.4810000000000001},"last":0.81200000000000006},{"interface":"front","average":{"1min":0.92700000000000005,"5min":1.0209999999999999,"15min":1.0229999999999999},"min":{"1min":0.61199999999999999,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":1.284,"5min":1.4750000000000001,"15min":1.4950000000000001},"last":0.79200000000000004}]}]},{"osd":3,"up_from":13,"seq":55834575311,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1079744,"kb_used_data":31160,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93292096,"statfs":{"total":96636764160,"available":95531106304,"internally_reserved":1073741824,"allocated":31907840,"data_stored":30454869,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 11:41:51 2021","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.49099999999999999,"15min":0.49199999999999999},"min":{"1min":0.376,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.75,"5min":0.96099999999999997,"15min":1.1479999999999999},"last":0.37},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.46999999999999997,"15min":0.47199999999999998},"min":{"1min":0.34799999999999998,"5min":0.34399999999999997,"15min":0.30199999999999999},"max":{"1min":0.748,"5min":0.748,"15min":0.79600000000000004},"last":0.40100000000000002}]},{"osd":1,"last update":"Wed Feb 10 11:41:51 2021","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.49299999999999999,"15min":0.495},"min":{"1min":0.36899999999999999,"5min":0.32100000000000001,"15min":0.311},"max":{"1min":0.66600000000000004,"5min":0.83399999999999996,"15min":1.2709999999999999},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.55800000000000005,"15min":0.56299999999999994},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.316},"max":{"1min":0.70599999999999996,"5min":0.86399999999999999,"15min":0.997},"last":0.45200000000000001}]},{"osd":2,"last update":"Wed Feb 10 11:41:51 2021","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.55700000000000005,"15min":0.56100000000000005},"min":{"1min":0.46000000000000002,"5min":0.374,"15min":0.35299999999999998},"max":{"1min":0.76600000000000001,"5min":0.995,"15min":0.995},"last":0.442},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.61399999999999999,"15min":0.63200000000000001},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.35099999999999998},"max":{"1min":0.74299999999999999,"5min":0.91800000000000004,"15min":1.04},"last":0.504}]},{"osd":4,"last update":"Wed Feb 10 11:41:51 2021","interfaces":[{"interface":"back","average":{"1min":0.82699999999999996,"5min":0.94599999999999995,"15min":0.95199999999999996},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.39100000000000001},"max":{"1min":1.1180000000000001,"5min":1.3779999999999999,"15min":1.5760000000000001},"last":1.2130000000000001},{"interface":"front","average":{"1min":0.90200000000000002,"5min":0.98699999999999999,"15min":0.98299999999999998},"min":{"1min":0.55200000000000005,"5min":0.52600000000000002,"15min":0.52000000000000002},"max":{"1min":1.1950000000000001,"5min":1.494,"15min":1.6659999999999999},"last":1.1990000000000001}]},{"osd":5,"last update":"Wed Feb 10 11:41:51 2021","interfaces":[{"interface":"back","average":{"1min":0.95199999999999996,"5min":0.99199999999999999,"15min":0.99299999999999999},"min":{"1min":0.63100000000000001,"5min":0.58199999999999996,"15min":0.55800000000000005},"max":{"1min":1.27,"5min":1.448,"15min":1.556},"last":1.0289999999999999},{"interface":"front","average":{"1min":0.91700000000000004,"5min":0.98399999999999999,"15min":0.97299999999999998},"min":{"1min":0.64000000000000001,"5min":0.51100000000000001,"15min":0.47999999999999998},"max":{"1min":1.2130000000000001,"5min":1.3959999999999999,"15min":1.405},"last":1.1000000000000001}]},{"osd":6,"last update":"Wed Feb 10 11:41:51 2021","interfaces":[{"interface":"back","average":{"1min":0.89900000000000002,"5min":0.98599999999999999,"15min":0.996},"min":{"1min":0.59299999999999997,"5min":0.54100000000000004,"15min":0.51100000000000001},"max":{"1min":1.216,"5min":1.3500000000000001,"15min":1.5409999999999999},"last":1.1299999999999999},{"interface":"front","average":{"1min":0.96599999999999997,"5min":1.0029999999999999,"15min":0.995},"min":{"1min":0.68500000000000005,"5min":0.58599999999999997,"15min":0.57599999999999996},"max":{"1min":1.22,"5min":1.458,"15min":1.458},"last":1.1419999999999999}]},{"osd":7,"last update":"Wed Feb 10 11:41:51 2021","interfaces":[{"interface":"back","average":{"1min":0.93300000000000005,"5min":0.99399999999999999,"15min":0.99299999999999999},"min":{"1min":0.67900000000000005,"5min":0.55700000000000005,"15min":0.50600000000000001},"max":{"1min":1.2310000000000001,"5min":1.264,"15min":1.389},"last":1.179},{"interface":"front","average":{"1min":0.93000000000000005,"5min":0.98699999999999999,"15min":0.99299999999999999},"min":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.56999999999999995},"max":{"1min":1.242,"5min":1.488,"15min":1.4970000000000001},"last":0.98999999999999999}]}]}]}} 2021-02-10T11:41:58.322 INFO:tasks.ceph:pgid 43.6 last_scrub_stamp 2021-02-10 11:40:23.720923 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=10, tm_hour=11, tm_min=40, tm_sec=23, tm_wday=2, tm_yday=41, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=10, tm_hour=11, tm_min=41, tm_sec=47, tm_wday=2, tm_yday=41, tm_isdst=0) 2021-02-10T11:41:58.322 INFO:tasks.ceph:pgid 42.7 last_scrub_stamp 2021-02-10 11:40:22.718405 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=10, tm_hour=11, tm_min=40, tm_sec=22, tm_wday=2, tm_yday=41, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=10, tm_hour=11, tm_min=41, tm_sec=47, tm_wday=2, tm_yday=41, tm_isdst=0) 2021-02-10T11:41:58.322 INFO:tasks.ceph:pgid 43.7 last_scrub_stamp 2021-02-10 11:40:23.720923 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=10, tm_hour=11, tm_min=40, tm_sec=23, tm_wday=2, tm_yday=41, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=10, tm_hour=11, tm_min=41, tm_sec=47, tm_wday=2, tm_yday=41, tm_isdst=0) 2021-02-10T11:41:58.323 INFO:tasks.ceph:pgid 42.6 last_scrub_stamp 2021-02-10 11:40:22.718405 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=10, tm_hour=11, tm_min=40, tm_sec=22, tm_wday=2, tm_yday=41, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=10, tm_hour=11, tm_min=41, tm_sec=47, tm_wday=2, tm_yday=41, tm_isdst=0) 2021-02-10T11:41:58.323 INFO:tasks.ceph:pgid 43.5 last_scrub_stamp 2021-02-10 11:40:23.720923 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=10, tm_hour=11, tm_min=40, tm_sec=23, tm_wday=2, tm_yday=41, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=10, tm_hour=11, tm_min=41, tm_sec=47, tm_wday=2, tm_yday=41, tm_isdst=0) 2021-02-10T11:41:58.323 INFO:tasks.ceph:pgid 42.4 last_scrub_stamp 2021-02-10 11:40:22.718405 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=10, tm_hour=11, tm_min=40, tm_sec=22, tm_wday=2, tm_yday=41, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=10, tm_hour=11, tm_min=41, tm_sec=47, tm_wday=2, tm_yday=41, tm_isdst=0) 2021-02-10T11:41:58.324 INFO:tasks.ceph:pgid 43.2 last_scrub_stamp 2021-02-10 11:40:23.720923 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=10, tm_hour=11, tm_min=40, tm_sec=23, tm_wday=2, tm_yday=41, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=10, tm_hour=11, tm_min=41, tm_sec=47, tm_wday=2, tm_yday=41, tm_isdst=0) 2021-02-10T11:41:58.325 INFO:tasks.ceph:pgid 42.3 last_scrub_stamp 2021-02-10 11:40:22.718405 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=10, tm_hour=11, tm_min=40, tm_sec=22, tm_wday=2, tm_yday=41, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=10, tm_hour=11, tm_min=41, tm_sec=47, tm_wday=2, tm_yday=41, tm_isdst=0) 2021-02-10T11:41:58.325 INFO:tasks.ceph:pgid 43.3 last_scrub_stamp 2021-02-10 11:40:23.720923 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=10, tm_hour=11, tm_min=40, tm_sec=23, tm_wday=2, tm_yday=41, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=10, tm_hour=11, tm_min=41, tm_sec=47, tm_wday=2, tm_yday=41, tm_isdst=0) 2021-02-10T11:41:58.326 INFO:tasks.ceph:pgid 42.2 last_scrub_stamp 2021-02-10 11:40:22.718405 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=10, tm_hour=11, tm_min=40, tm_sec=22, tm_wday=2, tm_yday=41, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=10, tm_hour=11, tm_min=41, tm_sec=47, tm_wday=2, tm_yday=41, tm_isdst=0) 2021-02-10T11:41:58.326 INFO:tasks.ceph:pgid 43.0 last_scrub_stamp 2021-02-10 11:40:23.720923 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=10, tm_hour=11, tm_min=40, tm_sec=23, tm_wday=2, tm_yday=41, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=10, tm_hour=11, tm_min=41, tm_sec=47, tm_wday=2, tm_yday=41, tm_isdst=0) 2021-02-10T11:41:58.326 INFO:tasks.ceph:pgid 42.1 last_scrub_stamp 2021-02-10 11:40:22.718405 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=10, tm_hour=11, tm_min=40, tm_sec=22, tm_wday=2, tm_yday=41, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=10, tm_hour=11, tm_min=41, tm_sec=47, tm_wday=2, tm_yday=41, tm_isdst=0) 2021-02-10T11:41:58.327 INFO:tasks.ceph:pgid 43.1 last_scrub_stamp 2021-02-10 11:40:23.720923 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=10, tm_hour=11, tm_min=40, tm_sec=23, tm_wday=2, tm_yday=41, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=10, tm_hour=11, tm_min=41, tm_sec=47, tm_wday=2, tm_yday=41, tm_isdst=0) 2021-02-10T11:41:58.327 INFO:tasks.ceph:pgid 42.0 last_scrub_stamp 2021-02-10 11:40:22.718405 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=10, tm_hour=11, tm_min=40, tm_sec=22, tm_wday=2, tm_yday=41, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=10, tm_hour=11, tm_min=41, tm_sec=47, tm_wday=2, tm_yday=41, tm_isdst=0) 2021-02-10T11:41:58.327 INFO:tasks.ceph:pgid 1.7 last_scrub_stamp 2021-02-10 11:03:37.729316 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=10, tm_hour=11, tm_min=3, tm_sec=37, tm_wday=2, tm_yday=41, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=10, tm_hour=11, tm_min=41, tm_sec=47, tm_wday=2, tm_yday=41, tm_isdst=0) 2021-02-10T11:41:58.328 INFO:tasks.ceph:pgid 1.6 last_scrub_stamp 2021-02-10 11:03:37.729316 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=10, tm_hour=11, tm_min=3, tm_sec=37, tm_wday=2, tm_yday=41, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=10, tm_hour=11, tm_min=41, tm_sec=47, tm_wday=2, tm_yday=41, tm_isdst=0) 2021-02-10T11:41:58.328 INFO:tasks.ceph:pgid 1.5 last_scrub_stamp 2021-02-10 11:03:37.729316 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=10, tm_hour=11, tm_min=3, tm_sec=37, tm_wday=2, tm_yday=41, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=10, tm_hour=11, tm_min=41, tm_sec=47, tm_wday=2, tm_yday=41, tm_isdst=0) 2021-02-10T11:41:58.328 INFO:tasks.ceph:pgid 1.4 last_scrub_stamp 2021-02-10 11:03:37.729316 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=10, tm_hour=11, tm_min=3, tm_sec=37, tm_wday=2, tm_yday=41, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=10, tm_hour=11, tm_min=41, tm_sec=47, tm_wday=2, tm_yday=41, tm_isdst=0) 2021-02-10T11:41:58.329 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2021-02-10 11:03:37.729316 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=10, tm_hour=11, tm_min=3, tm_sec=37, tm_wday=2, tm_yday=41, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=10, tm_hour=11, tm_min=41, tm_sec=47, tm_wday=2, tm_yday=41, tm_isdst=0) 2021-02-10T11:41:58.330 INFO:tasks.ceph:pgid 43.4 last_scrub_stamp 2021-02-10 11:40:23.720923 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=10, tm_hour=11, tm_min=40, tm_sec=23, tm_wday=2, tm_yday=41, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=10, tm_hour=11, tm_min=41, tm_sec=47, tm_wday=2, tm_yday=41, tm_isdst=0) 2021-02-10T11:41:58.330 INFO:tasks.ceph:pgid 42.5 last_scrub_stamp 2021-02-10 11:40:22.718405 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=10, tm_hour=11, tm_min=40, tm_sec=22, tm_wday=2, tm_yday=41, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=10, tm_hour=11, tm_min=41, tm_sec=47, tm_wday=2, tm_yday=41, tm_isdst=0) 2021-02-10T11:41:58.330 INFO:tasks.ceph:pgid 1.1 last_scrub_stamp 2021-02-10 11:03:37.729316 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=10, tm_hour=11, tm_min=3, tm_sec=37, tm_wday=2, tm_yday=41, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=10, tm_hour=11, tm_min=41, tm_sec=47, tm_wday=2, tm_yday=41, tm_isdst=0) 2021-02-10T11:41:58.330 INFO:tasks.ceph:pgid 1.2 last_scrub_stamp 2021-02-10 11:03:37.729316 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=10, tm_hour=11, tm_min=3, tm_sec=37, tm_wday=2, tm_yday=41, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=10, tm_hour=11, tm_min=41, tm_sec=47, tm_wday=2, tm_yday=41, tm_isdst=0) 2021-02-10T11:41:58.331 INFO:tasks.ceph:pgid 1.3 last_scrub_stamp 2021-02-10 11:03:37.729316 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=10, tm_hour=11, tm_min=3, tm_sec=37, tm_wday=2, tm_yday=41, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=10, tm_hour=11, tm_min=41, tm_sec=47, tm_wday=2, tm_yday=41, tm_isdst=0) 2021-02-10T11:41:58.331 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2021-02-10T11:42:17.467 DEBUG:teuthology.orchestra.run.gibba002:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:42:17.471 DEBUG:teuthology.orchestra.run.gibba013:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:42:18.331 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph pg dump --format=json 2021-02-10T11:42:18.761 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:18.740 7f17e58af700 1 Processor -- start 2021-02-10T11:42:18.761 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:18.741 7f17e58af700 1 -- start start 2021-02-10T11:42:18.761 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:18.741 7f17e58af700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f17e00d1c70 0x7f17e00f20a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:18.761 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:18.741 7f17e58af700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17e00f0b10 con 0x7f17e00d0340 2021-02-10T11:42:18.761 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:18.741 7f17e58af700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17e01b0be0 con 0x7f17e00d1870 2021-02-10T11:42:18.762 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:18.741 7f17e58af700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17e00e5ed0 con 0x7f17e00d1c70 2021-02-10T11:42:18.762 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:18.741 7f17ded73700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f17e00d0340 0x7f17e00ef640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:34186/0 (socket says 172.21.2.102:34186) 2021-02-10T11:42:18.762 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:18.741 7f17ded73700 1 -- 172.21.2.102:0/2156744074 learned_addr learned my addr 172.21.2.102:0/2156744074 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:42:18.762 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:18.742 7f17de572700 1 -- 172.21.2.102:0/2156744074 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2009489645 0 0) 0x7f17c8001500 con 0x7f17e00d0340 2021-02-10T11:42:18.762 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:18.742 7f17dfd75700 1 --2- 172.21.2.102:0/2156744074 >> v2:172.21.2.113:3300/0 conn(0x7f17e00d1c70 0x7f17e00f20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:18.762 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:18.742 7f17de572700 1 -- 172.21.2.102:0/2156744074 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f17cc002fe0 con 0x7f17e00d0340 2021-02-10T11:42:18.762 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:18.742 7f17de572700 1 -- 172.21.2.102:0/2156744074 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3725678071 0 0) 0x7f17c8001500 con 0x7f17e00d0340 2021-02-10T11:42:18.763 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:18.742 7f17de572700 1 -- 172.21.2.102:0/2156744074 >> v2:172.21.2.113:3300/0 conn(0x7f17e00d1c70 msgr2=0x7f17e00f20a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:42:18.763 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:18.742 7f17de572700 1 --2- 172.21.2.102:0/2156744074 >> v2:172.21.2.113:3300/0 conn(0x7f17e00d1c70 0x7f17e00f20a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:18.763 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:18.742 7f17de572700 1 -- 172.21.2.102:0/2156744074 >> v1:172.21.2.113:6789/0 conn(0x7f17e00d1870 legacy=0x7f17e01b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:18.763 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:18.742 7f17de572700 1 -- 172.21.2.102:0/2156744074 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17e00efe30 con 0x7f17e00d0340 2021-02-10T11:42:18.763 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:18.742 7f17de572700 1 -- 172.21.2.102:0/2156744074 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7f17c8001a00 con 0x7f17e00d0340 2021-02-10T11:42:18.763 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:18.742 7f17de572700 1 -- 172.21.2.102:0/2156744074 >> v1:172.21.2.102:6789/0 conn(0x7f17e00d0340 legacy=0x7f17e00ef640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:18.763 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:18.742 7f17de572700 1 --2- 172.21.2.102:0/2156744074 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f17cc003ae0 0x7f17cc005f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:18.763 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:18.742 7f17de572700 1 --2- 172.21.2.102:0/2156744074 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f17cc006830 0x7f17cc008d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:18.763 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:18.742 7f17de572700 1 --2- 172.21.2.102:0/2156744074 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f17cc009630 0x7f17cc00bb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:18.764 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:18.742 7f17de572700 1 -- 172.21.2.102:0/2156744074 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17cc00c3a0 con 0x7f17cc003ae0 2021-02-10T11:42:18.764 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:18.742 7f17de572700 1 -- 172.21.2.102:0/2156744074 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17cc00c5c0 con 0x7f17cc009630 2021-02-10T11:42:18.764 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:18.742 7f17de572700 1 -- 172.21.2.102:0/2156744074 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17cc0507f0 con 0x7f17cc006830 2021-02-10T11:42:18.764 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:18.742 7f17dfd75700 1 --2- 172.21.2.102:0/2156744074 >> v2:172.21.2.113:3300/0 conn(0x7f17e00d1c70 0x7f17e00f20a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:42:18.764 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:18.742 7f17df574700 1 --2- 172.21.2.102:0/2156744074 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f17cc003ae0 0x7f17cc005f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:18.764 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:18.743 7f17ded73700 1 --2- 172.21.2.102:0/2156744074 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f17cc006830 0x7f17cc008d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:18.764 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:18.743 7f17dfd75700 1 --2- 172.21.2.102:0/2156744074 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f17cc009630 0x7f17cc00bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:18.764 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:18.743 7f17df574700 1 -- 172.21.2.102:0/2156744074 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f17cc006830 msgr2=0x7f17cc008d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:42:18.765 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:18.743 7f17df574700 1 --2- 172.21.2.102:0/2156744074 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f17cc006830 0x7f17cc008d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:18.765 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:18.743 7f17df574700 1 -- 172.21.2.102:0/2156744074 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f17cc009630 msgr2=0x7f17cc00bb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:42:18.765 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:18.743 7f17df574700 1 --2- 172.21.2.102:0/2156744074 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f17cc009630 0x7f17cc00bb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:18.765 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:18.743 7f17df574700 1 -- 172.21.2.102:0/2156744074 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17cc050a50 con 0x7f17cc003ae0 2021-02-10T11:42:18.765 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:18.743 7f17dfd75700 1 --2- 172.21.2.102:0/2156744074 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f17cc009630 0x7f17cc00bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:42:18.765 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:18.743 7f17dfd75700 1 -- 172.21.2.102:0/2156744074 reap_dead start 2021-02-10T11:42:18.765 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:18.743 7f17df574700 1 --2- 172.21.2.102:0/2156744074 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f17cc003ae0 0x7f17cc005f90 secure :-1 s=READY pgs=3492 cs=0 l=1 rev1=1 rx=0x7f17d0006ab0 tx=0x7f17d0009ac0).ready entity=mon.0 client_cookie=ba84bdbb57789736 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:18.765 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:18.743 7f17de572700 1 -- 172.21.2.102:0/2156744074 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f17d00066a0 con 0x7f17cc003ae0 2021-02-10T11:42:18.766 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:18.743 7f17de572700 1 -- 172.21.2.102:0/2156744074 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17d00022e0 con 0x7f17cc003ae0 2021-02-10T11:42:18.766 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:18.743 7f17de572700 1 -- 172.21.2.102:0/2156744074 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f17d00026b0 con 0x7f17cc003ae0 2021-02-10T11:42:18.766 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:18.743 7f17e58af700 1 -- 172.21.2.102:0/2156744074 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f17cc003ae0 msgr2=0x7f17cc005f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:18.766 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:18.743 7f17e58af700 1 --2- 172.21.2.102:0/2156744074 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f17cc003ae0 0x7f17cc005f90 secure :-1 s=READY pgs=3492 cs=0 l=1 rev1=1 rx=0x7f17d0006ab0 tx=0x7f17d0009ac0).stop 2021-02-10T11:42:18.766 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:18.744 7f17e58af700 1 -- 172.21.2.102:0/2156744074 shutdown_connections 2021-02-10T11:42:18.766 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:18.744 7f17e58af700 1 --2- 172.21.2.102:0/2156744074 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f17cc003ae0 0x7f17cc005f90 unknown :-1 s=CLOSED pgs=3492 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:18.766 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:18.744 7f17e58af700 1 -- 172.21.2.102:0/2156744074 >> 172.21.2.102:0/2156744074 conn(0x7f17e00ce450 msgr2=0x7f17e00d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:18.767 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:18.744 7f17e58af700 1 -- 172.21.2.102:0/2156744074 shutdown_connections 2021-02-10T11:42:18.767 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:18.744 7f17e58af700 1 -- 172.21.2.102:0/2156744074 wait complete. 2021-02-10T11:42:18.768 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:18.759 7f17e58af700 1 Processor -- start 2021-02-10T11:42:18.769 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:18.759 7f17e58af700 1 -- start start 2021-02-10T11:42:18.769 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:18.759 7f17e58af700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f17e00f0280 0x7f17e00a2310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:18.769 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:18.759 7f17e58af700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f17e00f0680 0x7f17e00a2930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:18.769 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:18.759 7f17e58af700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f17e00ec180 0x7f17e00ec580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:18.769 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:18.759 7f17e58af700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17e00efd00 con 0x7f17e00ec180 2021-02-10T11:42:18.769 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:18.759 7f17e58af700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17e00ecba0 con 0x7f17e00f0280 2021-02-10T11:42:18.769 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:18.759 7f17e58af700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17e00ecd80 con 0x7f17e00f0680 2021-02-10T11:42:18.769 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:18.760 7f17df574700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f17e00f0280 0x7f17e00a2310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:18.770 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:18.760 7f17df574700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f17e00f0280 0x7f17e00a2310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.113:3300/0 says I am v2:172.21.2.102:45402/0 (socket says 172.21.2.102:45402) 2021-02-10T11:42:18.770 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:18.760 7f17df574700 1 -- 172.21.2.102:0/748449764 learned_addr learned my addr 172.21.2.102:0/748449764 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:42:18.770 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:18.760 7f17dfd75700 1 --2- 172.21.2.102:0/748449764 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f17e00ec180 0x7f17e00ec580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:18.770 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:18.760 7f17ded73700 1 --2- 172.21.2.102:0/748449764 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f17e00f0680 0x7f17e00a2930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:18.770 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:18.760 7f17dfd75700 1 -- 172.21.2.102:0/748449764 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f17e00f0680 msgr2=0x7f17e00a2930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:18.770 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:18.760 7f17dfd75700 1 --2- 172.21.2.102:0/748449764 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f17e00f0680 0x7f17e00a2930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:18.770 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:18.760 7f17dfd75700 1 -- 172.21.2.102:0/748449764 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f17e00f0280 msgr2=0x7f17e00a2310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:18.770 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:18.760 7f17dfd75700 1 --2- 172.21.2.102:0/748449764 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f17e00f0280 0x7f17e00a2310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:18.771 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:18.760 7f17dfd75700 1 -- 172.21.2.102:0/748449764 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17e00ecfa0 con 0x7f17e00ec180 2021-02-10T11:42:18.771 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:18.760 7f17df574700 1 --2- 172.21.2.102:0/748449764 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f17e00f0280 0x7f17e00a2310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:42:18.771 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:18.760 7f17dfd75700 1 --2- 172.21.2.102:0/748449764 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f17e00ec180 0x7f17e00ec580 secure :-1 s=READY pgs=3493 cs=0 l=1 rev1=1 rx=0x7f17d800b4b0 tx=0x7f17d800b850).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:18.771 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:18.761 7f17d7fff700 1 -- 172.21.2.102:0/748449764 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f17d800bc40 con 0x7f17e00ec180 2021-02-10T11:42:18.771 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:18.761 7f17d7fff700 1 -- 172.21.2.102:0/748449764 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17d8002030 con 0x7f17e00ec180 2021-02-10T11:42:18.771 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:18.761 7f17d7fff700 1 -- 172.21.2.102:0/748449764 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f17d8002490 con 0x7f17e00ec180 2021-02-10T11:42:18.771 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:18.761 7f17e58af700 1 -- 172.21.2.102:0/748449764 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17e00ecfa0 con 0x7f17e00ec180 2021-02-10T11:42:18.771 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:18.761 7f17e58af700 1 -- 172.21.2.102:0/748449764 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17e00ed530 con 0x7f17e00ec180 2021-02-10T11:42:18.772 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:18.761 7f17d7fff700 1 -- 172.21.2.102:0/748449764 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f17d8002210 con 0x7f17e00ec180 2021-02-10T11:42:18.772 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:18.761 7f17d7fff700 1 --2- 172.21.2.102:0/748449764 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f17c40186c0 0x7f17c401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:18.772 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:18.761 7f17d7fff700 1 -- 172.21.2.102:0/748449764 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f17d8032460 con 0x7f17e00ec180 2021-02-10T11:42:18.772 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:18.762 7f17df574700 1 --2- 172.21.2.102:0/748449764 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f17c40186c0 0x7f17c401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:18.772 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:18.762 7f17df574700 1 --2- 172.21.2.102:0/748449764 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f17c40186c0 0x7f17c401abc0 crc :-1 s=READY pgs=1519 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:18.793 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:18.790 7f17e58af700 1 -- 172.21.2.102:0/748449764 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17e00ebc60 con 0x7f17e00ec180 2021-02-10T11:42:18.795 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:18.793 7f17d7fff700 1 -- 172.21.2.102:0/748449764 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f17d8006ae0 con 0x7f17e00ec180 2021-02-10T11:42:18.928 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:18.925 7f17e58af700 1 -- 172.21.2.102:0/748449764 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "pg dump", "target": ["mgr", ""], "format": "json"}) v1 -- 0x7f17e00ed530 con 0x7f17c40186c0 2021-02-10T11:42:18.930 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:18.928 7f17d7fff700 1 -- 172.21.2.102:0/748449764 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 dumped all) v1 ==== 18+0+101414 (crc 0 0 0) 0x7f17d0000c60 con 0x7f17c40186c0 2021-02-10T11:42:18.932 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:42:18.932 INFO:teuthology.orchestra.run.gibba002.stderr:dumped all 2021-02-10T11:42:18.934 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:18.930 7f17e58af700 1 -- 172.21.2.102:0/748449764 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f17c40186c0 msgr2=0x7f17c401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:18.934 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:18.930 7f17e58af700 1 --2- 172.21.2.102:0/748449764 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f17c40186c0 0x7f17c401abc0 crc :-1 s=READY pgs=1519 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:18.934 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:18.930 7f17e58af700 1 -- 172.21.2.102:0/748449764 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f17e00ec180 msgr2=0x7f17e00ec580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:18.934 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:18.930 7f17e58af700 1 --2- 172.21.2.102:0/748449764 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f17e00ec180 0x7f17e00ec580 secure :-1 s=READY pgs=3493 cs=0 l=1 rev1=1 rx=0x7f17d800b4b0 tx=0x7f17d800b850).stop 2021-02-10T11:42:18.934 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:18.930 7f17e58af700 1 -- 172.21.2.102:0/748449764 shutdown_connections 2021-02-10T11:42:18.934 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:18.930 7f17e58af700 1 --2- 172.21.2.102:0/748449764 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f17e00ec180 0x7f17e00ec580 unknown :-1 s=CLOSED pgs=3493 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:18.934 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:18.930 7f17e58af700 1 --2- 172.21.2.102:0/748449764 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f17e00f0680 0x7f17e00a2930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:18.935 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:18.930 7f17e58af700 1 --2- 172.21.2.102:0/748449764 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f17c40186c0 0x7f17c401abc0 unknown :-1 s=CLOSED pgs=1519 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:18.935 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:18.930 7f17e58af700 1 --2- 172.21.2.102:0/748449764 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f17e00f0280 0x7f17e00a2310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:18.935 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:18.930 7f17e58af700 1 -- 172.21.2.102:0/748449764 >> 172.21.2.102:0/748449764 conn(0x7f17e00c2130 msgr2=0x7f17e009e3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:18.935 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:18.931 7f17e58af700 1 -- 172.21.2.102:0/748449764 shutdown_connections 2021-02-10T11:42:18.935 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:18.931 7f17e58af700 1 -- 172.21.2.102:0/748449764 wait complete. 2021-02-10T11:42:18.941 INFO:teuthology.orchestra.run.gibba002.stdout:{"pg_ready":true,"pg_map":{"version":1363,"stamp":"2021-02-10 11:42:18.175997","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":135146961,"num_objects":154,"num_object_clones":1,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":154,"num_whiteouts":0,"num_read":65,"num_read_kb":65537,"num_write":431,"num_write_kb":132080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5006,"num_omap_keys":18,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":486,"ondisk_log_size":486,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":8,"num_per_pool_osds":8,"kb":754974720,"kb_used":8672596,"kb_used_data":283920,"kb_used_omap":0,"kb_used_meta":8388608,"kb_avail":746302124,"statfs":{"total":773094113280,"available":764213374976,"internally_reserved":8589934592,"allocated":290734080,"data_stored":279050754,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8589934592},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":386,"num_omap_keys":8,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003544"},"pg_stats":[{"pgid":"43.6","version":"197'22","reported_seq":"40","reported_epoch":"197","state":"active+clean","last_fresh":"2021-02-10 11:41:37.311138","last_change":"2021-02-10 11:40:24.375639","last_active":"2021-02-10 11:41:37.311138","last_peered":"2021-02-10 11:41:37.311138","last_clean":"2021-02-10 11:41:37.311138","last_became_active":"2021-02-10 11:40:24.375377","last_became_peered":"2021-02-10 11:40:24.375377","last_unstale":"2021-02-10 11:41:37.311138","last_undegraded":"2021-02-10 11:41:37.311138","last_fullsized":"2021-02-10 11:41:37.311138","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:40:23.720923","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:40:23.720923","last_clean_scrub_stamp":"2021-02-10 11:40:23.720923","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":23068672,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":9,"num_read_kb":9216,"num_write":22,"num_write_kb":22528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"42.7","version":"197'12","reported_seq":"25","reported_epoch":"197","state":"active+clean","last_fresh":"2021-02-10 11:42:08.863940","last_change":"2021-02-10 11:42:08.863964","last_active":"2021-02-10 11:42:08.863940","last_peered":"2021-02-10 11:42:08.863940","last_clean":"2021-02-10 11:42:08.863940","last_became_active":"2021-02-10 11:40:23.729462","last_became_peered":"2021-02-10 11:40:23.729462","last_unstale":"2021-02-10 11:42:08.863940","last_undegraded":"2021-02-10 11:42:08.863940","last_fullsized":"2021-02-10 11:42:08.863940","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"197'12","last_scrub_stamp":"2021-02-10 11:42:08.863937","last_deep_scrub":"197'12","last_deep_scrub_stamp":"2021-02-10 11:42:08.863937","last_clean_scrub_stamp":"2021-02-10 11:42:08.863937","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":566,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"43.7","version":"197'21","reported_seq":"40","reported_epoch":"197","state":"active+clean","last_fresh":"2021-02-10 11:42:01.639909","last_change":"2021-02-10 11:42:01.639936","last_active":"2021-02-10 11:42:01.639909","last_peered":"2021-02-10 11:42:01.639909","last_clean":"2021-02-10 11:42:01.639909","last_became_active":"2021-02-10 11:40:24.371106","last_became_peered":"2021-02-10 11:40:24.371106","last_unstale":"2021-02-10 11:42:01.639909","last_undegraded":"2021-02-10 11:42:01.639909","last_fullsized":"2021-02-10 11:42:01.639909","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"197'21","last_scrub_stamp":"2021-02-10 11:42:01.639905","last_deep_scrub":"197'21","last_deep_scrub_stamp":"2021-02-10 11:42:01.639905","last_clean_scrub_stamp":"2021-02-10 11:42:01.639905","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":14680064,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":7,"num_read_kb":7168,"num_write":14,"num_write_kb":14336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"42.6","version":"197'17","reported_seq":"30","reported_epoch":"197","state":"active+clean","last_fresh":"2021-02-10 11:42:04.144020","last_change":"2021-02-10 11:42:04.144045","last_active":"2021-02-10 11:42:04.144020","last_peered":"2021-02-10 11:42:04.144020","last_clean":"2021-02-10 11:42:04.144020","last_became_active":"2021-02-10 11:40:23.729136","last_became_peered":"2021-02-10 11:40:23.729136","last_unstale":"2021-02-10 11:42:04.144020","last_undegraded":"2021-02-10 11:42:04.144020","last_fullsized":"2021-02-10 11:42:04.144020","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"197'17","last_scrub_stamp":"2021-02-10 11:42:04.144017","last_deep_scrub":"197'17","last_deep_scrub_stamp":"2021-02-10 11:42:04.144017","last_clean_scrub_stamp":"2021-02-10 11:42:04.144017","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":35,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":386,"num_omap_keys":8,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"43.5","version":"197'33","reported_seq":"54","reported_epoch":"197","state":"active+clean","last_fresh":"2021-02-10 11:41:57.805463","last_change":"2021-02-10 11:41:57.805484","last_active":"2021-02-10 11:41:57.805463","last_peered":"2021-02-10 11:41:57.805463","last_clean":"2021-02-10 11:41:57.805463","last_became_active":"2021-02-10 11:40:24.372091","last_became_peered":"2021-02-10 11:40:24.372091","last_unstale":"2021-02-10 11:41:57.805463","last_undegraded":"2021-02-10 11:41:57.805463","last_fullsized":"2021-02-10 11:41:57.805463","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"197'33","last_scrub_stamp":"2021-02-10 11:41:57.805460","last_deep_scrub":"197'33","last_deep_scrub_stamp":"2021-02-10 11:41:57.805460","last_clean_scrub_stamp":"2021-02-10 11:41:57.805460","log_size":33,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16777506,"num_objects":18,"num_object_clones":1,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":10,"num_read_kb":10240,"num_write":20,"num_write_kb":16387,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"42.4","version":"197'225","reported_seq":"238","reported_epoch":"197","state":"active+clean","last_fresh":"2021-02-10 11:41:59.676970","last_change":"2021-02-10 11:41:59.676995","last_active":"2021-02-10 11:41:59.676970","last_peered":"2021-02-10 11:41:59.676970","last_clean":"2021-02-10 11:41:59.676970","last_became_active":"2021-02-10 11:40:23.728257","last_became_peered":"2021-02-10 11:40:23.728257","last_unstale":"2021-02-10 11:41:59.676970","last_undegraded":"2021-02-10 11:41:59.676970","last_fullsized":"2021-02-10 11:41:59.676970","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"197'225","last_scrub_stamp":"2021-02-10 11:41:59.676968","last_deep_scrub":"197'225","last_deep_scrub_stamp":"2021-02-10 11:41:59.676968","last_clean_scrub_stamp":"2021-02-10 11:41:59.676968","log_size":225,"ondisk_log_size":225,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":927945,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":222,"num_write_kb":974,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"43.2","version":"197'27","reported_seq":"47","reported_epoch":"197","state":"active+clean","last_fresh":"2021-02-10 11:41:54.733382","last_change":"2021-02-10 11:41:54.733418","last_active":"2021-02-10 11:41:54.733382","last_peered":"2021-02-10 11:41:54.733382","last_clean":"2021-02-10 11:41:54.733382","last_became_active":"2021-02-10 11:40:24.372214","last_became_peered":"2021-02-10 11:40:24.372214","last_unstale":"2021-02-10 11:41:54.733382","last_undegraded":"2021-02-10 11:41:54.733382","last_fullsized":"2021-02-10 11:41:54.733382","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"197'27","last_scrub_stamp":"2021-02-10 11:41:54.733378","last_deep_scrub":"197'27","last_deep_scrub_stamp":"2021-02-10 11:41:54.733378","last_clean_scrub_stamp":"2021-02-10 11:41:54.733378","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20971520,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":8,"num_read_kb":8192,"num_write":20,"num_write_kb":20480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"42.3","version":"197'9","reported_seq":"22","reported_epoch":"197","state":"active+clean","last_fresh":"2021-02-10 11:42:01.381819","last_change":"2021-02-10 11:42:01.381847","last_active":"2021-02-10 11:42:01.381819","last_peered":"2021-02-10 11:42:01.381819","last_clean":"2021-02-10 11:42:01.381819","last_became_active":"2021-02-10 11:40:23.728701","last_became_peered":"2021-02-10 11:40:23.728701","last_unstale":"2021-02-10 11:42:01.381819","last_undegraded":"2021-02-10 11:42:01.381819","last_fullsized":"2021-02-10 11:42:01.381819","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"197'9","last_scrub_stamp":"2021-02-10 11:42:01.381815","last_deep_scrub":"197'9","last_deep_scrub_stamp":"2021-02-10 11:42:01.381815","last_clean_scrub_stamp":"2021-02-10 11:42:01.381815","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4620,"num_omap_keys":10,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"43.3","version":"197'30","reported_seq":"50","reported_epoch":"197","state":"active+clean","last_fresh":"2021-02-10 11:41:58.657003","last_change":"2021-02-10 11:41:58.657036","last_active":"2021-02-10 11:41:58.657003","last_peered":"2021-02-10 11:41:58.657003","last_clean":"2021-02-10 11:41:58.657003","last_became_active":"2021-02-10 11:40:24.370863","last_became_peered":"2021-02-10 11:40:24.370863","last_unstale":"2021-02-10 11:41:58.657003","last_undegraded":"2021-02-10 11:41:58.657003","last_fullsized":"2021-02-10 11:41:58.657003","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"197'30","last_scrub_stamp":"2021-02-10 11:41:58.656999","last_deep_scrub":"197'30","last_deep_scrub_stamp":"2021-02-10 11:41:58.656999","last_clean_scrub_stamp":"2021-02-10 11:41:58.656999","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":18874505,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":8,"num_read_kb":8192,"num_write":25,"num_write_kb":18436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"42.2","version":"196'1","reported_seq":"14","reported_epoch":"197","state":"active+clean","last_fresh":"2021-02-10 11:41:53.740556","last_change":"2021-02-10 11:41:53.740590","last_active":"2021-02-10 11:41:53.740556","last_peered":"2021-02-10 11:41:53.740556","last_clean":"2021-02-10 11:41:53.740556","last_became_active":"2021-02-10 11:40:23.728297","last_became_peered":"2021-02-10 11:40:23.728297","last_unstale":"2021-02-10 11:41:53.740556","last_undegraded":"2021-02-10 11:41:53.740556","last_fullsized":"2021-02-10 11:41:53.740556","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"196'1","last_scrub_stamp":"2021-02-10 11:41:53.740554","last_deep_scrub":"196'1","last_deep_scrub_stamp":"2021-02-10 11:41:53.740554","last_clean_scrub_stamp":"2021-02-10 11:41:53.740554","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"43.0","version":"197'26","reported_seq":"48","reported_epoch":"197","state":"active+clean","last_fresh":"2021-02-10 11:42:02.399504","last_change":"2021-02-10 11:42:02.399531","last_active":"2021-02-10 11:42:02.399504","last_peered":"2021-02-10 11:42:02.399504","last_clean":"2021-02-10 11:42:02.399504","last_became_active":"2021-02-10 11:40:24.372579","last_became_peered":"2021-02-10 11:40:24.372579","last_unstale":"2021-02-10 11:42:02.399504","last_undegraded":"2021-02-10 11:42:02.399504","last_fullsized":"2021-02-10 11:42:02.399504","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"197'26","last_scrub_stamp":"2021-02-10 11:42:02.399500","last_deep_scrub":"197'26","last_deep_scrub_stamp":"2021-02-10 11:42:02.399500","last_clean_scrub_stamp":"2021-02-10 11:42:02.399500","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":18874368,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":10,"num_read_kb":10240,"num_write":18,"num_write_kb":18432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"42.1","version":"197'2","reported_seq":"16","reported_epoch":"197","state":"active+clean","last_fresh":"2021-02-10 11:41:56.794220","last_change":"2021-02-10 11:41:56.794284","last_active":"2021-02-10 11:41:56.794220","last_peered":"2021-02-10 11:41:56.794220","last_clean":"2021-02-10 11:41:56.794220","last_became_active":"2021-02-10 11:40:23.728061","last_became_peered":"2021-02-10 11:40:23.728061","last_unstale":"2021-02-10 11:41:56.794220","last_undegraded":"2021-02-10 11:41:56.794220","last_fullsized":"2021-02-10 11:41:56.794220","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"197'2","last_scrub_stamp":"2021-02-10 11:41:56.794217","last_deep_scrub":"197'2","last_deep_scrub_stamp":"2021-02-10 11:41:56.794217","last_clean_scrub_stamp":"2021-02-10 11:41:56.794217","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":101,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"43.1","version":"197'21","reported_seq":"39","reported_epoch":"197","state":"active+clean","last_fresh":"2021-02-10 11:42:10.123767","last_change":"2021-02-10 11:42:10.123794","last_active":"2021-02-10 11:42:10.123767","last_peered":"2021-02-10 11:42:10.123767","last_clean":"2021-02-10 11:42:10.123767","last_became_active":"2021-02-10 11:40:24.373558","last_became_peered":"2021-02-10 11:40:24.373558","last_unstale":"2021-02-10 11:42:10.123767","last_undegraded":"2021-02-10 11:42:10.123767","last_fullsized":"2021-02-10 11:42:10.123767","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"197'21","last_scrub_stamp":"2021-02-10 11:42:10.123763","last_deep_scrub":"197'21","last_deep_scrub_stamp":"2021-02-10 11:42:10.123763","last_clean_scrub_stamp":"2021-02-10 11:42:10.123763","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":10485760,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":6,"num_read_kb":6144,"num_write":10,"num_write_kb":10240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"42.0","version":"197'12","reported_seq":"25","reported_epoch":"197","state":"active+clean","last_fresh":"2021-02-10 11:42:00.114037","last_change":"2021-02-10 11:42:00.114075","last_active":"2021-02-10 11:42:00.114037","last_peered":"2021-02-10 11:42:00.114037","last_clean":"2021-02-10 11:42:00.114037","last_became_active":"2021-02-10 11:40:23.728990","last_became_peered":"2021-02-10 11:40:23.728990","last_unstale":"2021-02-10 11:42:00.114037","last_undegraded":"2021-02-10 11:42:00.114037","last_fullsized":"2021-02-10 11:42:00.114037","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"197'12","last_scrub_stamp":"2021-02-10 11:42:00.114034","last_deep_scrub":"197'12","last_deep_scrub_stamp":"2021-02-10 11:42:00.114034","last_clean_scrub_stamp":"2021-02-10 11:42:00.114034","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":"193","reported_epoch":"197","state":"active+clean","last_fresh":"2021-02-10 11:41:59.375156","last_change":"2021-02-10 11:41:59.375207","last_active":"2021-02-10 11:41:59.375156","last_peered":"2021-02-10 11:41:59.375156","last_clean":"2021-02-10 11:41:59.375156","last_became_active":"2021-02-10 11:03:38.737435","last_became_peered":"2021-02-10 11:03:38.737435","last_unstale":"2021-02-10 11:41:59.375156","last_undegraded":"2021-02-10 11:41:59.375156","last_fullsized":"2021-02-10 11:41:59.375156","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:41:59.375153","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:41:59.375153","last_clean_scrub_stamp":"2021-02-10 11:41:59.375153","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":"193","reported_epoch":"197","state":"active+clean","last_fresh":"2021-02-10 11:41:57.367313","last_change":"2021-02-10 11:41:57.367344","last_active":"2021-02-10 11:41:57.367313","last_peered":"2021-02-10 11:41:57.367313","last_clean":"2021-02-10 11:41:57.367313","last_became_active":"2021-02-10 11:03:38.738144","last_became_peered":"2021-02-10 11:03:38.738144","last_unstale":"2021-02-10 11:41:57.367313","last_undegraded":"2021-02-10 11:41:57.367313","last_fullsized":"2021-02-10 11:41:57.367313","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:41:57.367310","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:41:57.367310","last_clean_scrub_stamp":"2021-02-10 11:41:57.367310","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":"191","reported_epoch":"197","state":"active+clean","last_fresh":"2021-02-10 11:41:57.677931","last_change":"2021-02-10 11:41:57.677961","last_active":"2021-02-10 11:41:57.677931","last_peered":"2021-02-10 11:41:57.677931","last_clean":"2021-02-10 11:41:57.677931","last_became_active":"2021-02-10 11:03:38.738088","last_became_peered":"2021-02-10 11:03:38.738088","last_unstale":"2021-02-10 11:41:57.677931","last_undegraded":"2021-02-10 11:41:57.677931","last_fullsized":"2021-02-10 11:41:57.677931","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:41:57.677928","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:41:57.677928","last_clean_scrub_stamp":"2021-02-10 11:41:57.677928","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":"183","reported_epoch":"197","state":"active+clean","last_fresh":"2021-02-10 11:41:56.542642","last_change":"2021-02-10 11:41:56.542672","last_active":"2021-02-10 11:41:56.542642","last_peered":"2021-02-10 11:41:56.542642","last_clean":"2021-02-10 11:41:56.542642","last_became_active":"2021-02-10 11:03:38.740334","last_became_peered":"2021-02-10 11:03:38.740334","last_unstale":"2021-02-10 11:41:56.542642","last_undegraded":"2021-02-10 11:41:56.542642","last_fullsized":"2021-02-10 11:41:56.542642","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:41:56.542639","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:41:56.542639","last_clean_scrub_stamp":"2021-02-10 11:41:56.542639","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"183","reported_epoch":"197","state":"active+clean","last_fresh":"2021-02-10 11:42:01.752631","last_change":"2021-02-10 11:42:01.752675","last_active":"2021-02-10 11:42:01.752631","last_peered":"2021-02-10 11:42:01.752631","last_clean":"2021-02-10 11:42:01.752631","last_became_active":"2021-02-10 11:03:38.739150","last_became_peered":"2021-02-10 11:03:38.739150","last_unstale":"2021-02-10 11:42:01.752631","last_undegraded":"2021-02-10 11:42:01.752631","last_fullsized":"2021-02-10 11:42:01.752631","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:42:01.752626","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:42:01.752626","last_clean_scrub_stamp":"2021-02-10 11:42:01.752626","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"43.4","version":"197'16","reported_seq":"34","reported_epoch":"197","state":"active+clean","last_fresh":"2021-02-10 11:42:01.543378","last_change":"2021-02-10 11:42:01.543407","last_active":"2021-02-10 11:42:01.543378","last_peered":"2021-02-10 11:42:01.543378","last_clean":"2021-02-10 11:42:01.543378","last_became_active":"2021-02-10 11:40:24.371177","last_became_peered":"2021-02-10 11:40:24.371177","last_unstale":"2021-02-10 11:42:01.543378","last_undegraded":"2021-02-10 11:42:01.543378","last_fullsized":"2021-02-10 11:42:01.543378","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"197'16","last_scrub_stamp":"2021-02-10 11:42:01.543375","last_deep_scrub":"197'16","last_deep_scrub_stamp":"2021-02-10 11:42:01.543375","last_clean_scrub_stamp":"2021-02-10 11:42:01.543375","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":10485760,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":6,"num_read_kb":6144,"num_write":10,"num_write_kb":10240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"42.5","version":"197'12","reported_seq":"25","reported_epoch":"197","state":"active+clean","last_fresh":"2021-02-10 11:42:01.135794","last_change":"2021-02-10 11:42:01.135858","last_active":"2021-02-10 11:42:01.135794","last_peered":"2021-02-10 11:42:01.135794","last_clean":"2021-02-10 11:42:01.135794","last_became_active":"2021-02-10 11:40:23.729375","last_became_peered":"2021-02-10 11:40:23.729375","last_unstale":"2021-02-10 11:42:01.135794","last_undegraded":"2021-02-10 11:42:01.135794","last_fullsized":"2021-02-10 11:42:01.135794","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"197'12","last_scrub_stamp":"2021-02-10 11:42:01.135791","last_deep_scrub":"197'12","last_deep_scrub_stamp":"2021-02-10 11:42:01.135791","last_clean_scrub_stamp":"2021-02-10 11:42:01.135791","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":160,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":"178","reported_epoch":"197","state":"active+clean","last_fresh":"2021-02-10 11:41:58.190812","last_change":"2021-02-10 11:41:58.190854","last_active":"2021-02-10 11:41:58.190812","last_peered":"2021-02-10 11:41:58.190812","last_clean":"2021-02-10 11:41:58.190812","last_became_active":"2021-02-10 11:03:38.738675","last_became_peered":"2021-02-10 11:03:38.738675","last_unstale":"2021-02-10 11:41:58.190812","last_undegraded":"2021-02-10 11:41:58.190812","last_fullsized":"2021-02-10 11:41:58.190812","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:41:58.190809","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:41:58.190809","last_clean_scrub_stamp":"2021-02-10 11:41:58.190809","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":"183","reported_epoch":"197","state":"active+clean","last_fresh":"2021-02-10 11:42:06.780050","last_change":"2021-02-10 11:42:06.780089","last_active":"2021-02-10 11:42:06.780050","last_peered":"2021-02-10 11:42:06.780050","last_clean":"2021-02-10 11:42:06.780050","last_became_active":"2021-02-10 11:03:38.739032","last_became_peered":"2021-02-10 11:03:38.739032","last_unstale":"2021-02-10 11:42:06.780050","last_undegraded":"2021-02-10 11:42:06.780050","last_fullsized":"2021-02-10 11:42:06.780050","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:42:06.780047","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:42:06.780047","last_clean_scrub_stamp":"2021-02-10 11:42:06.780047","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":"181","reported_epoch":"197","state":"active+clean","last_fresh":"2021-02-10 11:41:54.789223","last_change":"2021-02-10 11:41:54.789259","last_active":"2021-02-10 11:41:54.789223","last_peered":"2021-02-10 11:41:54.789223","last_clean":"2021-02-10 11:41:54.789223","last_became_active":"2021-02-10 11:03:38.740308","last_became_peered":"2021-02-10 11:03:38.740308","last_unstale":"2021-02-10 11:41:54.789223","last_undegraded":"2021-02-10 11:41:54.789223","last_fullsized":"2021-02-10 11:41:54.789223","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:41:54.789220","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:41:54.789220","last_clean_scrub_stamp":"2021-02-10 11:41:54.789220","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":43,"num_pg":8,"stat_sum":{"num_bytes":134218155,"num_objects":131,"num_object_clones":1,"num_object_copies":262,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":131,"num_whiteouts":0,"num_read":64,"num_read_kb":65536,"num_write":139,"num_write_kb":131079,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":268460032,"data_stored":268436310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":196,"ondisk_log_size":196,"up":16,"acting":16,"num_store_stats":8},{"poolid":42,"num_pg":8,"stat_sum":{"num_bytes":928806,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":292,"num_write_kb":1001,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5006,"num_omap_keys":18,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1925120,"data_stored":1857612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":290,"ondisk_log_size":290,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575316,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1087952,"kb_used_data":39368,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93283888,"statfs":{"total":96636764160,"available":95522701312,"internally_reserved":1073741824,"allocated":40312832,"data_stored":38844297,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 11:41:47 2021","interfaces":[{"interface":"back","average":{"1min":0.91900000000000004,"5min":0.92600000000000005,"15min":0.94999999999999996},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.2070000000000001,"5min":1.4630000000000001,"15min":1.4630000000000001},"last":1.141},{"interface":"front","average":{"1min":0.92500000000000004,"5min":0.91600000000000004,"15min":0.92700000000000005},"min":{"1min":0.502,"5min":0.38800000000000001,"15min":0.36799999999999999},"max":{"1min":1.141,"5min":1.367,"15min":1.367},"last":0.99099999999999999}]},{"osd":1,"last update":"Wed Feb 10 11:41:47 2021","interfaces":[{"interface":"back","average":{"1min":0.81699999999999995,"5min":0.79700000000000004,"15min":0.81899999999999995},"min":{"1min":0.434,"5min":0.38,"15min":0.38},"max":{"1min":1.2689999999999999,"5min":1.361,"15min":1.361},"last":1.0600000000000001},{"interface":"front","average":{"1min":0.83699999999999997,"5min":0.81299999999999994,"15min":0.82699999999999996},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1.0649999999999999,"5min":1.288,"15min":1.508},"last":0.878}]},{"osd":2,"last update":"Wed Feb 10 11:41:47 2021","interfaces":[{"interface":"back","average":{"1min":0.81100000000000005,"5min":0.81999999999999995,"15min":0.82299999999999995},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":1.1870000000000001,"5min":1.323,"15min":1.3859999999999999},"last":0.97699999999999998},{"interface":"front","average":{"1min":0.85699999999999998,"5min":0.85299999999999998,"15min":0.86699999999999999},"min":{"1min":0.49199999999999999,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1.1040000000000001,"5min":1.2370000000000001,"15min":1.329},"last":1.234}]},{"osd":3,"last update":"Wed Feb 10 11:41:47 2021","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":0.83099999999999996,"15min":0.84299999999999997},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.40500000000000003},"max":{"1min":1.089,"5min":1.272,"15min":1.4139999999999999},"last":0.86499999999999999},{"interface":"front","average":{"1min":0.86899999999999999,"5min":0.89000000000000001,"15min":0.89400000000000002},"min":{"1min":0.60999999999999999,"5min":0.47799999999999998,"15min":0.45500000000000002},"max":{"1min":1.3220000000000001,"5min":1.3220000000000001,"15min":1.389},"last":0.93899999999999995}]},{"osd":4,"last update":"Wed Feb 10 11:41:47 2021","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.79400000000000004,"15min":0.80600000000000005},"min":{"1min":0.47799999999999998,"5min":0.47099999999999997,"15min":0.46800000000000003},"max":{"1min":1.1279999999999999,"5min":1.1279999999999999,"15min":1.4330000000000001},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.82899999999999996,"5min":0.81899999999999995,"15min":0.83099999999999996},"min":{"1min":0.48899999999999999,"5min":0.438,"15min":0.438},"max":{"1min":1.1060000000000001,"5min":1.1970000000000001,"15min":1.5489999999999999},"last":0.80400000000000005}]},{"osd":5,"last update":"Wed Feb 10 11:41:47 2021","interfaces":[{"interface":"back","average":{"1min":0.76800000000000002,"5min":0.76300000000000001,"15min":0.753},"min":{"1min":0.54800000000000004,"5min":0.49299999999999999,"15min":0.46899999999999997},"max":{"1min":1.1579999999999999,"5min":1.3360000000000001,"15min":1.3360000000000001},"last":0.82799999999999996},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.79000000000000004,"15min":0.77100000000000002},"min":{"1min":0.52800000000000002,"5min":0.47699999999999998,"15min":0.44700000000000001},"max":{"1min":1.1319999999999999,"5min":1.232,"15min":1.232},"last":0.81799999999999995}]},{"osd":6,"last update":"Wed Feb 10 11:41:47 2021","interfaces":[{"interface":"back","average":{"1min":0.876,"5min":0.85399999999999998,"15min":0.84099999999999997},"min":{"1min":0.67100000000000004,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.21,"5min":1.329,"15min":1.329},"last":0.92600000000000005},{"interface":"front","average":{"1min":0.82999999999999996,"5min":0.82299999999999995,"15min":0.80300000000000005},"min":{"1min":0.55000000000000004,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.0780000000000001,"5min":1.429,"15min":1.429},"last":0.91400000000000003}]}]},{"osd":6,"up_from":13,"seq":55834575316,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1070420,"kb_used_data":21836,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93301420,"statfs":{"total":96636764160,"available":95540654080,"internally_reserved":1073741824,"allocated":22360064,"data_stored":20896951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 11:41:56 2021","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.79000000000000004,"15min":0.79100000000000004},"min":{"1min":0.45600000000000002,"5min":0.36599999999999999,"15min":0.35599999999999998},"max":{"1min":0.97999999999999998,"5min":1.216,"15min":1.4099999999999999},"last":0.92900000000000005},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.78500000000000003,"15min":0.79500000000000004},"min":{"1min":0.53900000000000003,"5min":0.378,"15min":0.34799999999999998},"max":{"1min":1.2170000000000001,"5min":1.2969999999999999,"15min":1.3140000000000001},"last":0.92100000000000004}]},{"osd":1,"last update":"Wed Feb 10 11:41:56 2021","interfaces":[{"interface":"back","average":{"1min":0.81999999999999995,"5min":0.83099999999999996,"15min":0.84899999999999998},"min":{"1min":0.45000000000000001,"5min":0.41299999999999998,"15min":0.39300000000000002},"max":{"1min":1.1359999999999999,"5min":1.224,"15min":1.427},"last":1.046},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.81299999999999994,"15min":0.82099999999999995},"min":{"1min":0.38,"5min":0.371,"15min":0.371},"max":{"1min":0.98599999999999999,"5min":1.2230000000000001,"15min":1.2809999999999999},"last":0.97199999999999998}]},{"osd":2,"last update":"Wed Feb 10 11:41:56 2021","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.78400000000000003,"15min":0.78600000000000003},"min":{"1min":0.50800000000000001,"5min":0.44400000000000001,"15min":0.40000000000000002},"max":{"1min":1.0800000000000001,"5min":1.2569999999999999,"15min":1.2569999999999999},"last":0.96599999999999997},{"interface":"front","average":{"1min":0.81699999999999995,"5min":0.82999999999999996,"15min":0.85199999999999998},"min":{"1min":0.51200000000000001,"5min":0.45900000000000002,"15min":0.33000000000000002},"max":{"1min":1.2090000000000001,"5min":1.3029999999999999,"15min":1.431},"last":0.98099999999999998}]},{"osd":3,"last update":"Wed Feb 10 11:41:56 2021","interfaces":[{"interface":"back","average":{"1min":0.91700000000000004,"5min":0.91800000000000004,"15min":0.93600000000000005},"min":{"1min":0.626,"5min":0.433,"15min":0.42099999999999999},"max":{"1min":1.1539999999999999,"5min":1.4319999999999999,"15min":1.4319999999999999},"last":1.077},{"interface":"front","average":{"1min":0.90100000000000002,"5min":0.92300000000000004,"15min":0.93500000000000005},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.167,"5min":1.3440000000000001,"15min":1.3999999999999999},"last":1.0920000000000001}]},{"osd":4,"last update":"Wed Feb 10 11:41:56 2021","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.69399999999999995,"15min":0.70799999999999996},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35699999999999998},"max":{"1min":1.028,"5min":1.099,"15min":1.163},"last":0.80300000000000005},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.71299999999999997,"15min":0.72699999999999998},"min":{"1min":0.44400000000000001,"5min":0.373,"15min":0.29699999999999999},"max":{"1min":1.046,"5min":1.119,"15min":1.2769999999999999},"last":0.78500000000000003}]},{"osd":5,"last update":"Wed Feb 10 11:41:56 2021","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":0.748,"15min":0.752},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":1.1659999999999999,"5min":1.2649999999999999,"15min":1.2649999999999999},"last":0.90800000000000003},{"interface":"front","average":{"1min":0.82399999999999995,"5min":0.85699999999999998,"15min":0.871},"min":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":1.216,"5min":1.216,"15min":1.468},"last":0.875}]},{"osd":7,"last update":"Wed Feb 10 11:41:56 2021","interfaces":[{"interface":"back","average":{"1min":0.77800000000000002,"5min":0.77600000000000002,"15min":0.80300000000000005},"min":{"1min":0.505,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":1.2589999999999999,"5min":1.2589999999999999,"15min":1.452},"last":0.86099999999999999},{"interface":"front","average":{"1min":0.752,"5min":0.76600000000000001,"15min":0.77700000000000002},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.103,"5min":1.1499999999999999,"15min":1.3220000000000001},"last":0.84499999999999997}]}]},{"osd":5,"up_from":13,"seq":55834575316,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1083844,"kb_used_data":35260,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93287996,"statfs":{"total":96636764160,"available":95526907904,"internally_reserved":1073741824,"allocated":36106240,"data_stored":34649196,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 11:41:42 2021","interfaces":[{"interface":"back","average":{"1min":0.93600000000000005,"5min":0.85799999999999998,"15min":0.88200000000000001},"min":{"1min":0.49099999999999999,"5min":0.40000000000000002,"15min":0.34300000000000003},"max":{"1min":1.113,"5min":1.222,"15min":1.2829999999999999},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.995,"5min":0.90900000000000003,"15min":0.92100000000000004},"min":{"1min":0.73699999999999999,"5min":0.38400000000000001,"15min":0.33100000000000002},"max":{"1min":1.2589999999999999,"5min":1.27,"15min":1.444},"last":0.83999999999999997}]},{"osd":1,"last update":"Wed Feb 10 11:41:42 2021","interfaces":[{"interface":"back","average":{"1min":0.81899999999999995,"5min":0.75900000000000001,"15min":0.78100000000000003},"min":{"1min":0.44400000000000001,"5min":0.432,"15min":0.36499999999999999},"max":{"1min":1.081,"5min":1.2030000000000001,"15min":1.3169999999999999},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.82699999999999996,"5min":0.77200000000000002,"15min":0.80200000000000005},"min":{"1min":0.42899999999999999,"5min":0.41499999999999998,"15min":0.40400000000000003},"max":{"1min":1.0049999999999999,"5min":1.0980000000000001,"15min":1.2849999999999999},"last":0.61799999999999999}]},{"osd":2,"last update":"Wed Feb 10 11:41:42 2021","interfaces":[{"interface":"back","average":{"1min":0.89000000000000001,"5min":0.84899999999999998,"15min":0.89400000000000002},"min":{"1min":0.51200000000000001,"5min":0.47299999999999998,"15min":0.38400000000000001},"max":{"1min":1.202,"5min":1.258,"15min":1.534},"last":0.79500000000000004},{"interface":"front","average":{"1min":0.95099999999999996,"5min":0.91300000000000003,"15min":0.92500000000000004},"min":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.42199999999999999},"max":{"1min":1.232,"5min":1.3100000000000001,"15min":1.357},"last":0.94099999999999995}]},{"osd":3,"last update":"Wed Feb 10 11:41:42 2021","interfaces":[{"interface":"back","average":{"1min":0.93999999999999995,"5min":0.88800000000000001,"15min":0.93000000000000005},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1.2729999999999999,"5min":1.367,"15min":1.6399999999999999},"last":0.69299999999999995},{"interface":"front","average":{"1min":1,"5min":0.93000000000000005,"15min":0.95099999999999996},"min":{"1min":0.54600000000000004,"5min":0.51300000000000001,"15min":0.48399999999999999},"max":{"1min":1.252,"5min":1.252,"15min":1.3640000000000001},"last":0.55000000000000004}]},{"osd":4,"last update":"Wed Feb 10 11:41:42 2021","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.68000000000000005,"15min":0.69399999999999995},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.95099999999999996,"5min":1.002,"15min":1.127},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.69499999999999995,"15min":0.72499999999999998},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.97199999999999998,"5min":1.0620000000000001,"15min":1.3009999999999999},"last":0.60599999999999998}]},{"osd":6,"last update":"Wed Feb 10 11:41:42 2021","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.71299999999999997,"15min":0.71399999999999997},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.86399999999999999,"5min":1.125,"15min":1.139},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.75900000000000001,"15min":0.76800000000000002},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.48099999999999998},"max":{"1min":1.036,"5min":1.2,"15min":1.2170000000000001},"last":0.66800000000000004}]},{"osd":7,"last update":"Wed Feb 10 11:41:42 2021","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.78800000000000003,"15min":0.80300000000000005},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.0029999999999999,"5min":1.262,"15min":1.347},"last":0.748},{"interface":"front","average":{"1min":0.77700000000000002,"5min":0.81399999999999995,"15min":0.81899999999999995},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1.0369999999999999,"5min":1.117,"15min":1.2669999999999999},"last":0.85699999999999998}]}]},{"osd":4,"up_from":13,"seq":55834575316,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1074512,"kb_used_data":25928,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93297328,"statfs":{"total":96636764160,"available":95536463872,"internally_reserved":1073741824,"allocated":26550272,"data_stored":25091221,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 11:41:48 2021","interfaces":[{"interface":"back","average":{"1min":0.82699999999999996,"5min":0.80100000000000005,"15min":0.80700000000000005},"min":{"1min":0.51700000000000002,"5min":0.41099999999999998,"15min":0.32200000000000001},"max":{"1min":1.1379999999999999,"5min":1.2430000000000001,"15min":1.3979999999999999},"last":0.93100000000000005},{"interface":"front","average":{"1min":0.79200000000000004,"5min":0.79900000000000004,"15min":0.81299999999999994},"min":{"1min":0.442,"5min":0.379,"15min":0.379},"max":{"1min":1.099,"5min":1.3320000000000001,"15min":1.3320000000000001},"last":0.92100000000000004}]},{"osd":1,"last update":"Wed Feb 10 11:41:48 2021","interfaces":[{"interface":"back","average":{"1min":0.81799999999999995,"5min":0.81299999999999994,"15min":0.82799999999999996},"min":{"1min":0.53500000000000003,"5min":0.46899999999999997,"15min":0.307},"max":{"1min":1.23,"5min":1.23,"15min":1.417},"last":0.82199999999999995},{"interface":"front","average":{"1min":0.84099999999999997,"5min":0.79700000000000004,"15min":0.82099999999999995},"min":{"1min":0.53100000000000003,"5min":0.434,"15min":0.38300000000000001},"max":{"1min":1.0880000000000001,"5min":1.2929999999999999,"15min":1.387},"last":0.751}]},{"osd":2,"last update":"Wed Feb 10 11:41:48 2021","interfaces":[{"interface":"back","average":{"1min":0.95899999999999996,"5min":0.90900000000000003,"15min":0.92800000000000005},"min":{"1min":0.57999999999999996,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1.1879999999999999,"5min":1.2949999999999999,"15min":1.4239999999999999},"last":1.038},{"interface":"front","average":{"1min":0.95599999999999996,"5min":0.97599999999999998,"15min":0.98499999999999999},"min":{"1min":0.63700000000000001,"5min":0.57699999999999996,"15min":0.499},"max":{"1min":1.369,"5min":1.3979999999999999,"15min":1.571},"last":0.96699999999999997}]},{"osd":3,"last update":"Wed Feb 10 11:41:48 2021","interfaces":[{"interface":"back","average":{"1min":0.82299999999999995,"5min":0.80300000000000005,"15min":0.82799999999999996},"min":{"1min":0.54400000000000004,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":1.196,"5min":1.196,"15min":1.4550000000000001},"last":0.89500000000000002},{"interface":"front","average":{"1min":0.85099999999999998,"5min":0.84399999999999997,"15min":0.86799999999999999},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.39000000000000001},"max":{"1min":1.054,"5min":1.2190000000000001,"15min":1.325},"last":0.98699999999999999}]},{"osd":5,"last update":"Wed Feb 10 11:41:48 2021","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.71999999999999997,"15min":0.70699999999999996},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.39500000000000002},"max":{"1min":0.98299999999999998,"5min":1.0669999999999999,"15min":1.109},"last":0.499},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.68899999999999995,"15min":0.69999999999999996},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.36599999999999999},"max":{"1min":1.004,"5min":1.0920000000000001,"15min":1.272},"last":0.48599999999999999}]},{"osd":6,"last update":"Wed Feb 10 11:41:48 2021","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.73699999999999999,"15min":0.74099999999999999},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":1.1080000000000001,"5min":1.1080000000000001,"15min":1.254},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.78500000000000003,"15min":0.79300000000000004},"min":{"1min":0.504,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.0309999999999999,"5min":1.2350000000000001,"15min":1.262},"last":0.60099999999999998}]},{"osd":7,"last update":"Wed Feb 10 11:41:48 2021","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.77700000000000002,"15min":0.79500000000000004},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.0620000000000001,"5min":1.1180000000000001,"15min":1.3240000000000001},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.78400000000000003,"5min":0.84799999999999998,"15min":0.84299999999999997},"min":{"1min":0.57899999999999996,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.069,"5min":1.2330000000000001,"15min":1.3129999999999999},"last":0.61199999999999999}]}]},{"osd":0,"up_from":13,"seq":55834575315,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1061312,"kb_used_data":12728,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93310528,"statfs":{"total":96636764160,"available":95549980672,"internally_reserved":1073741824,"allocated":13033472,"data_stored":11580398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Feb 10 11:41:55 2021","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.51100000000000001,"15min":0.49099999999999999},"min":{"1min":0.36099999999999999,"5min":0.32900000000000001,"15min":0.30299999999999999},"max":{"1min":0.73599999999999999,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.50600000000000001,"15min":0.49299999999999999},"min":{"1min":0.32800000000000001,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.746,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.52600000000000002}]},{"osd":2,"last update":"Wed Feb 10 11:41:55 2021","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.54800000000000004,"15min":0.54200000000000004},"min":{"1min":0.36399999999999999,"5min":0.318,"15min":0.318},"max":{"1min":0.81499999999999995,"5min":0.94799999999999995,"15min":1.0249999999999999},"last":0.91800000000000004},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.54800000000000004,"15min":0.55200000000000005},"min":{"1min":0.42299999999999999,"5min":0.40600000000000003,"15min":0.35599999999999998},"max":{"1min":0.64300000000000002,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.58499999999999996}]},{"osd":3,"last update":"Wed Feb 10 11:41:55 2021","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.52600000000000002,"15min":0.51300000000000001},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.33900000000000002},"max":{"1min":0.93700000000000006,"5min":0.93700000000000006,"15min":0.93700000000000006},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.61799999999999999,"15min":0.61099999999999999},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.36299999999999999},"max":{"1min":0.86399999999999999,"5min":0.878,"15min":0.97599999999999998},"last":0.80400000000000005}]},{"osd":4,"last update":"Wed Feb 10 11:41:55 2021","interfaces":[{"interface":"back","average":{"1min":0.92600000000000005,"5min":0.98399999999999999,"15min":0.98399999999999999},"min":{"1min":0.53100000000000003,"5min":0.499,"15min":0.41199999999999998},"max":{"1min":1.2270000000000001,"5min":1.5189999999999999,"15min":1.53},"last":1.2729999999999999},{"interface":"front","average":{"1min":0.95299999999999996,"5min":0.98699999999999999,"15min":0.98299999999999998},"min":{"1min":0.67300000000000004,"5min":0.498,"15min":0.498},"max":{"1min":1.1950000000000001,"5min":1.4390000000000001,"15min":1.4710000000000001},"last":1.008}]},{"osd":5,"last update":"Wed Feb 10 11:41:55 2021","interfaces":[{"interface":"back","average":{"1min":0.90500000000000003,"5min":0.997,"15min":1.008},"min":{"1min":0.60399999999999998,"5min":0.59599999999999997,"15min":0.48099999999999998},"max":{"1min":1.2,"5min":1.353,"15min":1.484},"last":1.2909999999999999},{"interface":"front","average":{"1min":1.006,"5min":1.0289999999999999,"15min":1.0029999999999999},"min":{"1min":0.68600000000000005,"5min":0.57299999999999995,"15min":0.51700000000000002},"max":{"1min":1.2649999999999999,"5min":1.4059999999999999,"15min":1.4359999999999999},"last":1.099}]},{"osd":6,"last update":"Wed Feb 10 11:41:55 2021","interfaces":[{"interface":"back","average":{"1min":0.96599999999999997,"5min":1.03,"15min":1.018},"min":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.57299999999999995},"max":{"1min":1.2629999999999999,"5min":1.417,"15min":1.417},"last":1.1639999999999999},{"interface":"front","average":{"1min":0.97799999999999998,"5min":1.0429999999999999,"15min":1.024},"min":{"1min":0.69599999999999995,"5min":0.629,"15min":0.55100000000000005},"max":{"1min":1.331,"5min":1.4870000000000001,"15min":1.4870000000000001},"last":1.1120000000000001}]},{"osd":7,"last update":"Wed Feb 10 11:41:55 2021","interfaces":[{"interface":"back","average":{"1min":1.0169999999999999,"5min":1.0640000000000001,"15min":1.04},"min":{"1min":0.77100000000000002,"5min":0.71099999999999997,"15min":0.57599999999999996},"max":{"1min":1.417,"5min":1.726,"15min":1.726},"last":1.4039999999999999},{"interface":"front","average":{"1min":0.98199999999999998,"5min":1.022,"15min":1.0069999999999999},"min":{"1min":0.73599999999999999,"5min":0.62,"15min":0.60499999999999998},"max":{"1min":1.3460000000000001,"5min":1.6080000000000001,"15min":1.6080000000000001},"last":1.3779999999999999}]}]},{"osd":1,"up_from":13,"seq":55834575316,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1092044,"kb_used_data":43456,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93279796,"statfs":{"total":96636764160,"available":95518511104,"internally_reserved":1073741824,"allocated":44498944,"data_stored":43038035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 11:42:03 2021","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.59099999999999997,"15min":0.60299999999999998},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.82599999999999996,"5min":1.056,"15min":1.119},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.60399999999999998,"15min":0.61699999999999999},"min":{"1min":0.46600000000000003,"5min":0.40500000000000003,"15min":0.316},"max":{"1min":0.84599999999999997,"5min":0.98799999999999999,"15min":1.038},"last":0.51500000000000001}]},{"osd":2,"last update":"Wed Feb 10 11:42:03 2021","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.47999999999999998,"15min":0.48499999999999999},"min":{"1min":0.35899999999999999,"5min":0.33100000000000002,"15min":0.30499999999999999},"max":{"1min":0.66000000000000003,"5min":0.75900000000000001,"15min":1.0209999999999999},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.504,"5min":0.53800000000000003,"15min":0.55100000000000005},"min":{"1min":0.41499999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.63300000000000001,"5min":0.94599999999999995,"15min":1.1399999999999999},"last":0.60499999999999998}]},{"osd":3,"last update":"Wed Feb 10 11:42:03 2021","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.58999999999999997,"15min":0.60399999999999998},"min":{"1min":0.41099999999999998,"5min":0.38100000000000001,"15min":0.35999999999999999},"max":{"1min":0.76800000000000002,"5min":0.94199999999999995,"15min":1.244},"last":0.74099999999999999},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.53600000000000003,"15min":0.54800000000000004},"min":{"1min":0.36799999999999999,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.72499999999999998,"5min":0.83799999999999997,"15min":1.155},"last":0.64600000000000002}]},{"osd":4,"last update":"Wed Feb 10 11:42:03 2021","interfaces":[{"interface":"back","average":{"1min":0.93799999999999994,"5min":0.95599999999999996,"15min":0.96999999999999997},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.45300000000000001},"max":{"1min":1.2529999999999999,"5min":1.351,"15min":1.5229999999999999},"last":0.85599999999999998},{"interface":"front","average":{"1min":0.92700000000000005,"5min":0.98099999999999998,"15min":0.99399999999999999},"min":{"1min":0.66700000000000004,"5min":0.47599999999999998,"15min":0.46800000000000003},"max":{"1min":1.194,"5min":1.3540000000000001,"15min":1.508},"last":0.98199999999999998}]},{"osd":5,"last update":"Wed Feb 10 11:42:03 2021","interfaces":[{"interface":"back","average":{"1min":0.89700000000000002,"5min":0.97899999999999998,"15min":0.98699999999999999},"min":{"1min":0.55200000000000005,"5min":0.496,"15min":0.496},"max":{"1min":1.2689999999999999,"5min":1.3440000000000001,"15min":1.6279999999999999},"last":1.0229999999999999},{"interface":"front","average":{"1min":0.95399999999999996,"5min":0.98199999999999998,"15min":0.998},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.46899999999999997},"max":{"1min":1.3149999999999999,"5min":1.413,"15min":1.5760000000000001},"last":0.79100000000000004}]},{"osd":6,"last update":"Wed Feb 10 11:42:03 2021","interfaces":[{"interface":"back","average":{"1min":0.96199999999999997,"5min":1.006,"15min":1.0049999999999999},"min":{"1min":0.72599999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"max":{"1min":1.3080000000000001,"5min":1.3740000000000001,"15min":1.4059999999999999},"last":0.877},{"interface":"front","average":{"1min":0.94199999999999995,"5min":1.0169999999999999,"15min":1.0249999999999999},"min":{"1min":0.628,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":1.3200000000000001,"5min":1.387,"15min":1.456},"last":0.80100000000000005}]},{"osd":7,"last update":"Wed Feb 10 11:42:03 2021","interfaces":[{"interface":"back","average":{"1min":0.95399999999999996,"5min":1.0169999999999999,"15min":1.036},"min":{"1min":0.67700000000000005,"5min":0.495,"15min":0.495},"max":{"1min":1.2589999999999999,"5min":1.472,"15min":1.786},"last":0.92100000000000004},{"interface":"front","average":{"1min":0.97299999999999998,"5min":0.995,"15min":1.008},"min":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.56599999999999995},"max":{"1min":1.361,"5min":1.3720000000000001,"15min":1.494},"last":0.90300000000000002}]}]},{"osd":2,"up_from":13,"seq":55834575315,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1122768,"kb_used_data":74184,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93249072,"statfs":{"total":96636764160,"available":95487049728,"internally_reserved":1073741824,"allocated":75964416,"data_stored":74495787,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 11:41:44 2021","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.63700000000000001,"15min":0.628},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":1.0960000000000001,"5min":1.0960000000000001,"15min":1.103},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.63300000000000001,"15min":0.63},"min":{"1min":0.41599999999999998,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.99199999999999999,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.81699999999999995}]},{"osd":1,"last update":"Wed Feb 10 11:41:44 2021","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53300000000000003,"15min":0.52900000000000003},"min":{"1min":0.379,"5min":0.34200000000000003,"15min":0.32000000000000001},"max":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":1.0509999999999999},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.53100000000000003,"15min":0.51900000000000002},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.33400000000000002},"max":{"1min":0.68400000000000005,"5min":0.82599999999999996,"15min":0.88800000000000001},"last":0.628}]},{"osd":3,"last update":"Wed Feb 10 11:41:44 2021","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51200000000000001,"15min":0.52000000000000002},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.63600000000000001,"5min":0.69099999999999995,"15min":0.78100000000000003},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.58499999999999996,"15min":0.60299999999999998},"min":{"1min":0.438,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.79200000000000004,"5min":0.88600000000000001,"15min":1.0289999999999999},"last":0.88700000000000001}]},{"osd":4,"last update":"Wed Feb 10 11:41:44 2021","interfaces":[{"interface":"back","average":{"1min":0.97499999999999998,"5min":1.091,"15min":1.081},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.335,"5min":1.4430000000000001,"15min":1.4690000000000001},"last":1.119},{"interface":"front","average":{"1min":0.96699999999999997,"5min":1.101,"15min":1.095},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.373},"max":{"1min":1.359,"5min":1.419,"15min":1.77},"last":1.135}]},{"osd":5,"last update":"Wed Feb 10 11:41:44 2021","interfaces":[{"interface":"back","average":{"1min":0.89400000000000002,"5min":0.995,"15min":0.98599999999999999},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.296,"5min":1.47,"15min":1.47},"last":1.087},{"interface":"front","average":{"1min":0.86099999999999999,"5min":0.97099999999999997,"15min":0.98899999999999999},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":1.3220000000000001,"5min":1.4199999999999999,"15min":1.613},"last":1.1819999999999999}]},{"osd":6,"last update":"Wed Feb 10 11:41:44 2021","interfaces":[{"interface":"back","average":{"1min":0.89400000000000002,"5min":0.98399999999999999,"15min":0.996},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.2509999999999999,"5min":1.569,"15min":1.569},"last":1.1000000000000001},{"interface":"front","average":{"1min":0.92200000000000004,"5min":1.002,"15min":0.998},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.1519999999999999,"5min":1.3999999999999999,"15min":1.504},"last":1.0269999999999999}]},{"osd":7,"last update":"Wed Feb 10 11:41:44 2021","interfaces":[{"interface":"back","average":{"1min":0.89600000000000002,"5min":0.996,"15min":1.0049999999999999},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.2290000000000001,"5min":1.4810000000000001,"15min":1.4810000000000001},"last":0.95599999999999996},{"interface":"front","average":{"1min":0.92700000000000005,"5min":1.0209999999999999,"15min":1.0229999999999999},"min":{"1min":0.61199999999999999,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":1.284,"5min":1.4750000000000001,"15min":1.4950000000000001},"last":1.2330000000000001}]}]},{"osd":3,"up_from":13,"seq":55834575315,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1079744,"kb_used_data":31160,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93292096,"statfs":{"total":96636764160,"available":95531106304,"internally_reserved":1073741824,"allocated":31907840,"data_stored":30454869,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 11:41:51 2021","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.49099999999999999,"15min":0.49199999999999999},"min":{"1min":0.376,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.75,"5min":0.96099999999999997,"15min":1.1479999999999999},"last":0.5},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.46999999999999997,"15min":0.47199999999999998},"min":{"1min":0.34799999999999998,"5min":0.34399999999999997,"15min":0.30199999999999999},"max":{"1min":0.748,"5min":0.748,"15min":0.79600000000000004},"last":0.45400000000000001}]},{"osd":1,"last update":"Wed Feb 10 11:41:51 2021","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.49299999999999999,"15min":0.495},"min":{"1min":0.36899999999999999,"5min":0.32100000000000001,"15min":0.311},"max":{"1min":0.66600000000000004,"5min":0.83399999999999996,"15min":1.2709999999999999},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.55800000000000005,"15min":0.56299999999999994},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.316},"max":{"1min":0.70599999999999996,"5min":0.86399999999999999,"15min":0.997},"last":0.627}]},{"osd":2,"last update":"Wed Feb 10 11:41:51 2021","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.55700000000000005,"15min":0.56100000000000005},"min":{"1min":0.46000000000000002,"5min":0.374,"15min":0.35299999999999998},"max":{"1min":0.76600000000000001,"5min":0.995,"15min":0.995},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.61399999999999999,"15min":0.63200000000000001},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.35099999999999998},"max":{"1min":0.74299999999999999,"5min":0.91800000000000004,"15min":1.04},"last":0.77600000000000002}]},{"osd":4,"last update":"Wed Feb 10 11:41:51 2021","interfaces":[{"interface":"back","average":{"1min":0.82699999999999996,"5min":0.94599999999999995,"15min":0.95199999999999996},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.39100000000000001},"max":{"1min":1.1180000000000001,"5min":1.3779999999999999,"15min":1.5760000000000001},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.90200000000000002,"5min":0.98699999999999999,"15min":0.98299999999999998},"min":{"1min":0.55200000000000005,"5min":0.52600000000000002,"15min":0.52000000000000002},"max":{"1min":1.1950000000000001,"5min":1.494,"15min":1.6659999999999999},"last":0.95699999999999996}]},{"osd":5,"last update":"Wed Feb 10 11:41:51 2021","interfaces":[{"interface":"back","average":{"1min":0.95199999999999996,"5min":0.99199999999999999,"15min":0.99299999999999999},"min":{"1min":0.63100000000000001,"5min":0.58199999999999996,"15min":0.55800000000000005},"max":{"1min":1.27,"5min":1.448,"15min":1.556},"last":1.0369999999999999},{"interface":"front","average":{"1min":0.91700000000000004,"5min":0.98399999999999999,"15min":0.97299999999999998},"min":{"1min":0.64000000000000001,"5min":0.51100000000000001,"15min":0.47999999999999998},"max":{"1min":1.2130000000000001,"5min":1.3959999999999999,"15min":1.405},"last":0.88500000000000001}]},{"osd":6,"last update":"Wed Feb 10 11:41:51 2021","interfaces":[{"interface":"back","average":{"1min":0.89900000000000002,"5min":0.98599999999999999,"15min":0.996},"min":{"1min":0.59299999999999997,"5min":0.54100000000000004,"15min":0.51100000000000001},"max":{"1min":1.216,"5min":1.3500000000000001,"15min":1.5409999999999999},"last":0.81100000000000005},{"interface":"front","average":{"1min":0.96599999999999997,"5min":1.0029999999999999,"15min":0.995},"min":{"1min":0.68500000000000005,"5min":0.58599999999999997,"15min":0.57599999999999996},"max":{"1min":1.22,"5min":1.458,"15min":1.458},"last":0.85199999999999998}]},{"osd":7,"last update":"Wed Feb 10 11:41:51 2021","interfaces":[{"interface":"back","average":{"1min":0.93300000000000005,"5min":0.99399999999999999,"15min":0.99299999999999999},"min":{"1min":0.67900000000000005,"5min":0.55700000000000005,"15min":0.50600000000000001},"max":{"1min":1.2310000000000001,"5min":1.264,"15min":1.389},"last":0.82699999999999996},{"interface":"front","average":{"1min":0.93000000000000005,"5min":0.98699999999999999,"15min":0.99299999999999999},"min":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.56999999999999995},"max":{"1min":1.242,"5min":1.488,"15min":1.4970000000000001},"last":0.873}]}]}]}} 2021-02-10T11:42:18.944 INFO:tasks.ceph:pgid 43.6 last_scrub_stamp 2021-02-10 11:40:23.720923 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=10, tm_hour=11, tm_min=40, tm_sec=23, tm_wday=2, tm_yday=41, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=10, tm_hour=11, tm_min=41, tm_sec=47, tm_wday=2, tm_yday=41, tm_isdst=0) 2021-02-10T11:42:18.944 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2021-02-10T11:42:38.946 DEBUG:teuthology.orchestra.run.gibba002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph pg dump --format=json 2021-02-10T11:42:39.377 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:39.358 7ff163acc700 1 Processor -- start 2021-02-10T11:42:39.377 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:39.358 7ff163acc700 1 -- start start 2021-02-10T11:42:39.377 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:39.358 7ff163acc700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7ff15c1b0870 0x7ff15c0f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:39.377 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:39.358 7ff163acc700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7ff15c0efe00 0x7ff15c0f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:39.378 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:39.358 7ff163acc700 1 -- --> v1:172.21.2.113:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff15c1b0c70 con 0x7ff15c1b0470 2021-02-10T11:42:39.378 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:39.358 7ff163acc700 1 -- --> v2:172.21.2.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff15c0e5ea0 con 0x7ff15c0efe00 2021-02-10T11:42:39.378 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:39.358 7ff163acc700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff15c0f0b50 con 0x7ff15c1b0870 2021-02-10T11:42:39.378 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:39.358 7ff1620e9700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7ff15c0efe00 0x7ff15c0f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:39.378 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:39.359 7ff1620e9700 1 --2- >> v2:172.21.2.102:3300/0 conn(0x7ff15c0efe00 0x7ff15c0f0200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:41984/0 (socket says 172.21.2.102:41984) 2021-02-10T11:42:39.378 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:39.359 7ff1620e9700 1 -- 172.21.2.102:0/1663297114 learned_addr learned my addr 172.21.2.102:0/1663297114 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:42:39.379 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:39.359 7ff1620e9700 1 -- 172.21.2.102:0/1663297114 >> v2:172.21.2.113:3300/0 conn(0x7ff15c1b0870 msgr2=0x7ff15c0f20e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:42:39.379 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:39.359 7ff1620e9700 1 --2- 172.21.2.102:0/1663297114 >> v2:172.21.2.113:3300/0 conn(0x7ff15c1b0870 0x7ff15c0f20e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:39.379 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:39.359 7ff1620e9700 1 -- 172.21.2.102:0/1663297114 >> v1:172.21.2.113:6789/0 conn(0x7ff15c1b0470 legacy=0x7ff15c0ef610 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:42:39.379 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:39.359 7ff1620e9700 1 -- 172.21.2.102:0/1663297114 --> v2:172.21.2.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff15c0f0820 con 0x7ff15c0efe00 2021-02-10T11:42:39.379 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:39.359 7ff1620e9700 1 --2- 172.21.2.102:0/1663297114 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff15c0efe00 0x7ff15c0f0200 secure :-1 s=READY pgs=3494 cs=0 l=1 rev1=1 rx=0x7ff154006f00 tx=0x7ff154009f10).ready entity=mon.0 client_cookie=184eaebe4a2a5927 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:39.379 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:39.359 7ff1608e6700 1 -- 172.21.2.102:0/1663297114 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff15400b3b0 con 0x7ff15c0efe00 2021-02-10T11:42:39.379 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:39.359 7ff1608e6700 1 -- 172.21.2.102:0/1663297114 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff15400b630 con 0x7ff15c0efe00 2021-02-10T11:42:39.379 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:39.359 7ff1608e6700 1 -- 172.21.2.102:0/1663297114 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff15400bb50 con 0x7ff15c0efe00 2021-02-10T11:42:39.380 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:39.359 7ff163acc700 1 -- 172.21.2.102:0/1663297114 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff15c0efe00 msgr2=0x7ff15c0f0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:39.380 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:39.359 7ff163acc700 1 --2- 172.21.2.102:0/1663297114 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff15c0efe00 0x7ff15c0f0200 secure :-1 s=READY pgs=3494 cs=0 l=1 rev1=1 rx=0x7ff154006f00 tx=0x7ff154009f10).stop 2021-02-10T11:42:39.380 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:39.360 7ff163acc700 1 -- 172.21.2.102:0/1663297114 shutdown_connections 2021-02-10T11:42:39.380 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:39.360 7ff163acc700 1 --2- 172.21.2.102:0/1663297114 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff15c0efe00 0x7ff15c0f0200 unknown :-1 s=CLOSED pgs=3494 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:39.380 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:39.360 7ff163acc700 1 --2- 172.21.2.102:0/1663297114 >> v2:172.21.2.113:3300/0 conn(0x7ff15c1b0870 0x7ff15c0f20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:39.380 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:39.360 7ff163acc700 1 -- 172.21.2.102:0/1663297114 >> 172.21.2.102:0/1663297114 conn(0x7ff15c0ce3e0 msgr2=0x7ff15c0e4510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:39.380 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:39.360 7ff163acc700 1 -- 172.21.2.102:0/1663297114 shutdown_connections 2021-02-10T11:42:39.380 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:39.360 7ff163acc700 1 -- 172.21.2.102:0/1663297114 wait complete. 2021-02-10T11:42:39.382 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:39.375 7ff163acc700 1 Processor -- start 2021-02-10T11:42:39.382 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:39.375 7ff163acc700 1 -- start start 2021-02-10T11:42:39.382 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:39.375 7ff163acc700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff15c0b0970 0x7ff15c092ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:39.382 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:39.375 7ff163acc700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff15c0934f0 0x7ff15c094dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:39.382 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:39.375 7ff163acc700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff15c0938f0 0x7ff15c0953e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:39.382 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:39.375 7ff163acc700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff15c195ad0 con 0x7ff15c0b0970 2021-02-10T11:42:39.382 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:39.375 7ff163acc700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff15c095f40 con 0x7ff15c0938f0 2021-02-10T11:42:39.383 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:39.375 7ff163acc700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff15c095a30 con 0x7ff15c0934f0 2021-02-10T11:42:39.383 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:39.375 7ff1618e8700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff15c0b0970 0x7ff15c092ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:39.383 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:39.375 7ff1618e8700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff15c0b0970 0x7ff15c092ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:41990/0 (socket says 172.21.2.102:41990) 2021-02-10T11:42:39.383 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:39.375 7ff1618e8700 1 -- 172.21.2.102:0/4214807258 learned_addr learned my addr 172.21.2.102:0/4214807258 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:42:39.383 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:39.376 7ff1618e8700 1 -- 172.21.2.102:0/4214807258 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff15c0934f0 msgr2=0x7ff15c094dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:39.383 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:39.376 7ff1618e8700 1 --2- 172.21.2.102:0/4214807258 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff15c0934f0 0x7ff15c094dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:39.383 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:39.376 7ff1618e8700 1 -- 172.21.2.102:0/4214807258 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff15c0938f0 msgr2=0x7ff15c0953e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:39.383 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:39.376 7ff1618e8700 1 --2- 172.21.2.102:0/4214807258 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff15c0938f0 0x7ff15c0953e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:39.384 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:39.376 7ff1618e8700 1 -- 172.21.2.102:0/4214807258 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff15c095c50 con 0x7ff15c0b0970 2021-02-10T11:42:39.384 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:39.376 7ff1618e8700 1 --2- 172.21.2.102:0/4214807258 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff15c0b0970 0x7ff15c092ed0 secure :-1 s=READY pgs=3495 cs=0 l=1 rev1=1 rx=0x7ff15800b5b0 tx=0x7ff15800b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:39.384 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:39.376 7ff14e7fc700 1 -- 172.21.2.102:0/4214807258 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff15800bd40 con 0x7ff15c0b0970 2021-02-10T11:42:39.384 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:39.376 7ff14e7fc700 1 -- 172.21.2.102:0/4214807258 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1580009b0 con 0x7ff15c0b0970 2021-02-10T11:42:39.384 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:39.376 7ff163acc700 1 -- 172.21.2.102:0/4214807258 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff15c095c50 con 0x7ff15c0b0970 2021-02-10T11:42:39.384 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:39.376 7ff14e7fc700 1 -- 172.21.2.102:0/4214807258 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff1580062a0 con 0x7ff15c0b0970 2021-02-10T11:42:39.384 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:39.376 7ff163acc700 1 -- 172.21.2.102:0/4214807258 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff15c095c50 con 0x7ff15c0b0970 2021-02-10T11:42:39.384 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:39.377 7ff14e7fc700 1 -- 172.21.2.102:0/4214807258 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ff15800a000 con 0x7ff15c0b0970 2021-02-10T11:42:39.384 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:39.377 7ff14e7fc700 1 --2- 172.21.2.102:0/4214807258 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff144018730 0x7ff14401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:39.385 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:39.377 7ff14e7fc700 1 -- 172.21.2.102:0/4214807258 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 5573+0+0 (secure 0 0 0) 0x7ff1580325f0 con 0x7ff15c0b0970 2021-02-10T11:42:39.385 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:39.377 7ff1610e7700 1 --2- 172.21.2.102:0/4214807258 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff144018730 0x7ff14401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:39.385 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:39.377 7ff1610e7700 1 --2- 172.21.2.102:0/4214807258 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff144018730 0x7ff14401ac10 crc :-1 s=READY pgs=1520 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:39.410 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:39.408 7ff163acc700 1 -- 172.21.2.102:0/4214807258 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff15c0e52e0 con 0x7ff15c0b0970 2021-02-10T11:42:39.413 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:39.411 7ff14e7fc700 1 -- 172.21.2.102:0/4214807258 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ff15801f4b0 con 0x7ff15c0b0970 2021-02-10T11:42:39.547 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:39.544 7ff163acc700 1 -- 172.21.2.102:0/4214807258 --> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] -- command(tid 0: {"prefix": "pg dump", "target": ["mgr", ""], "format": "json"}) v1 -- 0x7ff15c0e52e0 con 0x7ff144018730 2021-02-10T11:42:39.549 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:39.547 7ff14e7fc700 1 -- 172.21.2.102:0/4214807258 <== mgr.4101 v2:172.21.2.102:6800/35161 1 ==== command_reply(tid 0: 0 dumped all) v1 ==== 18+0+101436 (crc 0 0 0) 0x7ff150006e40 con 0x7ff144018730 2021-02-10T11:42:39.551 INFO:teuthology.orchestra.run.gibba002.stderr:dumped all 2021-02-10T11:42:39.551 INFO:teuthology.orchestra.run.gibba002.stdout: 2021-02-10T11:42:39.552 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:39.549 7ff163acc700 1 -- 172.21.2.102:0/4214807258 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff144018730 msgr2=0x7ff14401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:39.552 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:39.549 7ff163acc700 1 --2- 172.21.2.102:0/4214807258 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff144018730 0x7ff14401ac10 crc :-1 s=READY pgs=1520 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:39.552 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:39.549 7ff163acc700 1 -- 172.21.2.102:0/4214807258 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff15c0b0970 msgr2=0x7ff15c092ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:39.553 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:39.549 7ff163acc700 1 --2- 172.21.2.102:0/4214807258 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff15c0b0970 0x7ff15c092ed0 secure :-1 s=READY pgs=3495 cs=0 l=1 rev1=1 rx=0x7ff15800b5b0 tx=0x7ff15800b950).stop 2021-02-10T11:42:39.553 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:39.549 7ff163acc700 1 -- 172.21.2.102:0/4214807258 shutdown_connections 2021-02-10T11:42:39.553 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:39.549 7ff163acc700 1 --2- 172.21.2.102:0/4214807258 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7ff144018730 0x7ff14401ac10 unknown :-1 s=CLOSED pgs=1520 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:39.553 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:39.549 7ff163acc700 1 --2- 172.21.2.102:0/4214807258 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7ff15c0938f0 0x7ff15c0953e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:39.553 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:39.549 7ff163acc700 1 --2- 172.21.2.102:0/4214807258 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7ff15c0934f0 0x7ff15c094dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:39.553 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:39.549 7ff163acc700 1 --2- 172.21.2.102:0/4214807258 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7ff15c0b0970 0x7ff15c092ed0 unknown :-1 s=CLOSED pgs=3495 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:39.553 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:39.549 7ff163acc700 1 -- 172.21.2.102:0/4214807258 >> 172.21.2.102:0/4214807258 conn(0x7ff15c19f7a0 msgr2=0x7ff15c0940e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:39.553 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:39.549 7ff163acc700 1 -- 172.21.2.102:0/4214807258 shutdown_connections 2021-02-10T11:42:39.554 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:39.549 7ff163acc700 1 -- 172.21.2.102:0/4214807258 wait complete. 2021-02-10T11:42:39.560 INFO:teuthology.orchestra.run.gibba002.stdout:{"pg_ready":true,"pg_map":{"version":1373,"stamp":"2021-02-10 11:42:38.181373","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":135146961,"num_objects":154,"num_object_clones":1,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":154,"num_whiteouts":0,"num_read":65,"num_read_kb":65537,"num_write":431,"num_write_kb":132080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5006,"num_omap_keys":18,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":508,"ondisk_log_size":508,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":8,"num_per_pool_osds":8,"kb":754974720,"kb_used":8672596,"kb_used_data":283920,"kb_used_omap":0,"kb_used_meta":8388608,"kb_avail":746302124,"statfs":{"total":773094113280,"available":764213374976,"internally_reserved":8589934592,"allocated":290734080,"data_stored":279050754,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8589934592},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003051"},"pg_stats":[{"pgid":"43.6","version":"197'44","reported_seq":"65","reported_epoch":"197","state":"active+clean","last_fresh":"2021-02-10 11:42:15.112357","last_change":"2021-02-10 11:42:15.112385","last_active":"2021-02-10 11:42:15.112357","last_peered":"2021-02-10 11:42:15.112357","last_clean":"2021-02-10 11:42:15.112357","last_became_active":"2021-02-10 11:40:24.375377","last_became_peered":"2021-02-10 11:40:24.375377","last_unstale":"2021-02-10 11:42:15.112357","last_undegraded":"2021-02-10 11:42:15.112357","last_fullsized":"2021-02-10 11:42:15.112357","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"197'44","last_scrub_stamp":"2021-02-10 11:42:15.112353","last_deep_scrub":"197'44","last_deep_scrub_stamp":"2021-02-10 11:42:15.112353","last_clean_scrub_stamp":"2021-02-10 11:42:15.112353","log_size":44,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":23068672,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":9,"num_read_kb":9216,"num_write":22,"num_write_kb":22528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"42.7","version":"197'12","reported_seq":"25","reported_epoch":"197","state":"active+clean","last_fresh":"2021-02-10 11:42:08.863940","last_change":"2021-02-10 11:42:08.863964","last_active":"2021-02-10 11:42:08.863940","last_peered":"2021-02-10 11:42:08.863940","last_clean":"2021-02-10 11:42:08.863940","last_became_active":"2021-02-10 11:40:23.729462","last_became_peered":"2021-02-10 11:40:23.729462","last_unstale":"2021-02-10 11:42:08.863940","last_undegraded":"2021-02-10 11:42:08.863940","last_fullsized":"2021-02-10 11:42:08.863940","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"197'12","last_scrub_stamp":"2021-02-10 11:42:08.863937","last_deep_scrub":"197'12","last_deep_scrub_stamp":"2021-02-10 11:42:08.863937","last_clean_scrub_stamp":"2021-02-10 11:42:08.863937","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":566,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"43.7","version":"197'21","reported_seq":"40","reported_epoch":"197","state":"active+clean","last_fresh":"2021-02-10 11:42:01.639909","last_change":"2021-02-10 11:42:01.639936","last_active":"2021-02-10 11:42:01.639909","last_peered":"2021-02-10 11:42:01.639909","last_clean":"2021-02-10 11:42:01.639909","last_became_active":"2021-02-10 11:40:24.371106","last_became_peered":"2021-02-10 11:40:24.371106","last_unstale":"2021-02-10 11:42:01.639909","last_undegraded":"2021-02-10 11:42:01.639909","last_fullsized":"2021-02-10 11:42:01.639909","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"197'21","last_scrub_stamp":"2021-02-10 11:42:01.639905","last_deep_scrub":"197'21","last_deep_scrub_stamp":"2021-02-10 11:42:01.639905","last_clean_scrub_stamp":"2021-02-10 11:42:01.639905","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":14680064,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":7,"num_read_kb":7168,"num_write":14,"num_write_kb":14336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"42.6","version":"197'17","reported_seq":"30","reported_epoch":"197","state":"active+clean","last_fresh":"2021-02-10 11:42:04.144020","last_change":"2021-02-10 11:42:04.144045","last_active":"2021-02-10 11:42:04.144020","last_peered":"2021-02-10 11:42:04.144020","last_clean":"2021-02-10 11:42:04.144020","last_became_active":"2021-02-10 11:40:23.729136","last_became_peered":"2021-02-10 11:40:23.729136","last_unstale":"2021-02-10 11:42:04.144020","last_undegraded":"2021-02-10 11:42:04.144020","last_fullsized":"2021-02-10 11:42:04.144020","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"197'17","last_scrub_stamp":"2021-02-10 11:42:04.144017","last_deep_scrub":"197'17","last_deep_scrub_stamp":"2021-02-10 11:42:04.144017","last_clean_scrub_stamp":"2021-02-10 11:42:04.144017","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":35,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":386,"num_omap_keys":8,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"43.5","version":"197'33","reported_seq":"54","reported_epoch":"197","state":"active+clean","last_fresh":"2021-02-10 11:41:57.805463","last_change":"2021-02-10 11:41:57.805484","last_active":"2021-02-10 11:41:57.805463","last_peered":"2021-02-10 11:41:57.805463","last_clean":"2021-02-10 11:41:57.805463","last_became_active":"2021-02-10 11:40:24.372091","last_became_peered":"2021-02-10 11:40:24.372091","last_unstale":"2021-02-10 11:41:57.805463","last_undegraded":"2021-02-10 11:41:57.805463","last_fullsized":"2021-02-10 11:41:57.805463","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"197'33","last_scrub_stamp":"2021-02-10 11:41:57.805460","last_deep_scrub":"197'33","last_deep_scrub_stamp":"2021-02-10 11:41:57.805460","last_clean_scrub_stamp":"2021-02-10 11:41:57.805460","log_size":33,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16777506,"num_objects":18,"num_object_clones":1,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":10,"num_read_kb":10240,"num_write":20,"num_write_kb":16387,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"42.4","version":"197'225","reported_seq":"238","reported_epoch":"197","state":"active+clean","last_fresh":"2021-02-10 11:41:59.676970","last_change":"2021-02-10 11:41:59.676995","last_active":"2021-02-10 11:41:59.676970","last_peered":"2021-02-10 11:41:59.676970","last_clean":"2021-02-10 11:41:59.676970","last_became_active":"2021-02-10 11:40:23.728257","last_became_peered":"2021-02-10 11:40:23.728257","last_unstale":"2021-02-10 11:41:59.676970","last_undegraded":"2021-02-10 11:41:59.676970","last_fullsized":"2021-02-10 11:41:59.676970","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"197'225","last_scrub_stamp":"2021-02-10 11:41:59.676968","last_deep_scrub":"197'225","last_deep_scrub_stamp":"2021-02-10 11:41:59.676968","last_clean_scrub_stamp":"2021-02-10 11:41:59.676968","log_size":225,"ondisk_log_size":225,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":927945,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":222,"num_write_kb":974,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"43.2","version":"197'27","reported_seq":"47","reported_epoch":"197","state":"active+clean","last_fresh":"2021-02-10 11:41:54.733382","last_change":"2021-02-10 11:41:54.733418","last_active":"2021-02-10 11:41:54.733382","last_peered":"2021-02-10 11:41:54.733382","last_clean":"2021-02-10 11:41:54.733382","last_became_active":"2021-02-10 11:40:24.372214","last_became_peered":"2021-02-10 11:40:24.372214","last_unstale":"2021-02-10 11:41:54.733382","last_undegraded":"2021-02-10 11:41:54.733382","last_fullsized":"2021-02-10 11:41:54.733382","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"197'27","last_scrub_stamp":"2021-02-10 11:41:54.733378","last_deep_scrub":"197'27","last_deep_scrub_stamp":"2021-02-10 11:41:54.733378","last_clean_scrub_stamp":"2021-02-10 11:41:54.733378","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20971520,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":8,"num_read_kb":8192,"num_write":20,"num_write_kb":20480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"42.3","version":"197'9","reported_seq":"22","reported_epoch":"197","state":"active+clean","last_fresh":"2021-02-10 11:42:01.381819","last_change":"2021-02-10 11:42:01.381847","last_active":"2021-02-10 11:42:01.381819","last_peered":"2021-02-10 11:42:01.381819","last_clean":"2021-02-10 11:42:01.381819","last_became_active":"2021-02-10 11:40:23.728701","last_became_peered":"2021-02-10 11:40:23.728701","last_unstale":"2021-02-10 11:42:01.381819","last_undegraded":"2021-02-10 11:42:01.381819","last_fullsized":"2021-02-10 11:42:01.381819","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"197'9","last_scrub_stamp":"2021-02-10 11:42:01.381815","last_deep_scrub":"197'9","last_deep_scrub_stamp":"2021-02-10 11:42:01.381815","last_clean_scrub_stamp":"2021-02-10 11:42:01.381815","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4620,"num_omap_keys":10,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"43.3","version":"197'30","reported_seq":"50","reported_epoch":"197","state":"active+clean","last_fresh":"2021-02-10 11:41:58.657003","last_change":"2021-02-10 11:41:58.657036","last_active":"2021-02-10 11:41:58.657003","last_peered":"2021-02-10 11:41:58.657003","last_clean":"2021-02-10 11:41:58.657003","last_became_active":"2021-02-10 11:40:24.370863","last_became_peered":"2021-02-10 11:40:24.370863","last_unstale":"2021-02-10 11:41:58.657003","last_undegraded":"2021-02-10 11:41:58.657003","last_fullsized":"2021-02-10 11:41:58.657003","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"197'30","last_scrub_stamp":"2021-02-10 11:41:58.656999","last_deep_scrub":"197'30","last_deep_scrub_stamp":"2021-02-10 11:41:58.656999","last_clean_scrub_stamp":"2021-02-10 11:41:58.656999","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":18874505,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":8,"num_read_kb":8192,"num_write":25,"num_write_kb":18436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"42.2","version":"196'1","reported_seq":"14","reported_epoch":"197","state":"active+clean","last_fresh":"2021-02-10 11:41:53.740556","last_change":"2021-02-10 11:41:53.740590","last_active":"2021-02-10 11:41:53.740556","last_peered":"2021-02-10 11:41:53.740556","last_clean":"2021-02-10 11:41:53.740556","last_became_active":"2021-02-10 11:40:23.728297","last_became_peered":"2021-02-10 11:40:23.728297","last_unstale":"2021-02-10 11:41:53.740556","last_undegraded":"2021-02-10 11:41:53.740556","last_fullsized":"2021-02-10 11:41:53.740556","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"196'1","last_scrub_stamp":"2021-02-10 11:41:53.740554","last_deep_scrub":"196'1","last_deep_scrub_stamp":"2021-02-10 11:41:53.740554","last_clean_scrub_stamp":"2021-02-10 11:41:53.740554","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"43.0","version":"197'26","reported_seq":"48","reported_epoch":"197","state":"active+clean","last_fresh":"2021-02-10 11:42:02.399504","last_change":"2021-02-10 11:42:02.399531","last_active":"2021-02-10 11:42:02.399504","last_peered":"2021-02-10 11:42:02.399504","last_clean":"2021-02-10 11:42:02.399504","last_became_active":"2021-02-10 11:40:24.372579","last_became_peered":"2021-02-10 11:40:24.372579","last_unstale":"2021-02-10 11:42:02.399504","last_undegraded":"2021-02-10 11:42:02.399504","last_fullsized":"2021-02-10 11:42:02.399504","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"197'26","last_scrub_stamp":"2021-02-10 11:42:02.399500","last_deep_scrub":"197'26","last_deep_scrub_stamp":"2021-02-10 11:42:02.399500","last_clean_scrub_stamp":"2021-02-10 11:42:02.399500","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":18874368,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":10,"num_read_kb":10240,"num_write":18,"num_write_kb":18432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"42.1","version":"197'2","reported_seq":"16","reported_epoch":"197","state":"active+clean","last_fresh":"2021-02-10 11:41:56.794220","last_change":"2021-02-10 11:41:56.794284","last_active":"2021-02-10 11:41:56.794220","last_peered":"2021-02-10 11:41:56.794220","last_clean":"2021-02-10 11:41:56.794220","last_became_active":"2021-02-10 11:40:23.728061","last_became_peered":"2021-02-10 11:40:23.728061","last_unstale":"2021-02-10 11:41:56.794220","last_undegraded":"2021-02-10 11:41:56.794220","last_fullsized":"2021-02-10 11:41:56.794220","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"197'2","last_scrub_stamp":"2021-02-10 11:41:56.794217","last_deep_scrub":"197'2","last_deep_scrub_stamp":"2021-02-10 11:41:56.794217","last_clean_scrub_stamp":"2021-02-10 11:41:56.794217","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":101,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"43.1","version":"197'21","reported_seq":"39","reported_epoch":"197","state":"active+clean","last_fresh":"2021-02-10 11:42:10.123767","last_change":"2021-02-10 11:42:10.123794","last_active":"2021-02-10 11:42:10.123767","last_peered":"2021-02-10 11:42:10.123767","last_clean":"2021-02-10 11:42:10.123767","last_became_active":"2021-02-10 11:40:24.373558","last_became_peered":"2021-02-10 11:40:24.373558","last_unstale":"2021-02-10 11:42:10.123767","last_undegraded":"2021-02-10 11:42:10.123767","last_fullsized":"2021-02-10 11:42:10.123767","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"197'21","last_scrub_stamp":"2021-02-10 11:42:10.123763","last_deep_scrub":"197'21","last_deep_scrub_stamp":"2021-02-10 11:42:10.123763","last_clean_scrub_stamp":"2021-02-10 11:42:10.123763","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":10485760,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":6,"num_read_kb":6144,"num_write":10,"num_write_kb":10240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"42.0","version":"197'12","reported_seq":"25","reported_epoch":"197","state":"active+clean","last_fresh":"2021-02-10 11:42:00.114037","last_change":"2021-02-10 11:42:00.114075","last_active":"2021-02-10 11:42:00.114037","last_peered":"2021-02-10 11:42:00.114037","last_clean":"2021-02-10 11:42:00.114037","last_became_active":"2021-02-10 11:40:23.728990","last_became_peered":"2021-02-10 11:40:23.728990","last_unstale":"2021-02-10 11:42:00.114037","last_undegraded":"2021-02-10 11:42:00.114037","last_fullsized":"2021-02-10 11:42:00.114037","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"197'12","last_scrub_stamp":"2021-02-10 11:42:00.114034","last_deep_scrub":"197'12","last_deep_scrub_stamp":"2021-02-10 11:42:00.114034","last_clean_scrub_stamp":"2021-02-10 11:42:00.114034","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":"193","reported_epoch":"197","state":"active+clean","last_fresh":"2021-02-10 11:41:59.375156","last_change":"2021-02-10 11:41:59.375207","last_active":"2021-02-10 11:41:59.375156","last_peered":"2021-02-10 11:41:59.375156","last_clean":"2021-02-10 11:41:59.375156","last_became_active":"2021-02-10 11:03:38.737435","last_became_peered":"2021-02-10 11:03:38.737435","last_unstale":"2021-02-10 11:41:59.375156","last_undegraded":"2021-02-10 11:41:59.375156","last_fullsized":"2021-02-10 11:41:59.375156","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:41:59.375153","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:41:59.375153","last_clean_scrub_stamp":"2021-02-10 11:41:59.375153","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":"193","reported_epoch":"197","state":"active+clean","last_fresh":"2021-02-10 11:41:57.367313","last_change":"2021-02-10 11:41:57.367344","last_active":"2021-02-10 11:41:57.367313","last_peered":"2021-02-10 11:41:57.367313","last_clean":"2021-02-10 11:41:57.367313","last_became_active":"2021-02-10 11:03:38.738144","last_became_peered":"2021-02-10 11:03:38.738144","last_unstale":"2021-02-10 11:41:57.367313","last_undegraded":"2021-02-10 11:41:57.367313","last_fullsized":"2021-02-10 11:41:57.367313","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:41:57.367310","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:41:57.367310","last_clean_scrub_stamp":"2021-02-10 11:41:57.367310","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":"191","reported_epoch":"197","state":"active+clean","last_fresh":"2021-02-10 11:41:57.677931","last_change":"2021-02-10 11:41:57.677961","last_active":"2021-02-10 11:41:57.677931","last_peered":"2021-02-10 11:41:57.677931","last_clean":"2021-02-10 11:41:57.677931","last_became_active":"2021-02-10 11:03:38.738088","last_became_peered":"2021-02-10 11:03:38.738088","last_unstale":"2021-02-10 11:41:57.677931","last_undegraded":"2021-02-10 11:41:57.677931","last_fullsized":"2021-02-10 11:41:57.677931","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:41:57.677928","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:41:57.677928","last_clean_scrub_stamp":"2021-02-10 11:41:57.677928","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":"183","reported_epoch":"197","state":"active+clean","last_fresh":"2021-02-10 11:41:56.542642","last_change":"2021-02-10 11:41:56.542672","last_active":"2021-02-10 11:41:56.542642","last_peered":"2021-02-10 11:41:56.542642","last_clean":"2021-02-10 11:41:56.542642","last_became_active":"2021-02-10 11:03:38.740334","last_became_peered":"2021-02-10 11:03:38.740334","last_unstale":"2021-02-10 11:41:56.542642","last_undegraded":"2021-02-10 11:41:56.542642","last_fullsized":"2021-02-10 11:41:56.542642","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:41:56.542639","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:41:56.542639","last_clean_scrub_stamp":"2021-02-10 11:41:56.542639","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"183","reported_epoch":"197","state":"active+clean","last_fresh":"2021-02-10 11:42:01.752631","last_change":"2021-02-10 11:42:01.752675","last_active":"2021-02-10 11:42:01.752631","last_peered":"2021-02-10 11:42:01.752631","last_clean":"2021-02-10 11:42:01.752631","last_became_active":"2021-02-10 11:03:38.739150","last_became_peered":"2021-02-10 11:03:38.739150","last_unstale":"2021-02-10 11:42:01.752631","last_undegraded":"2021-02-10 11:42:01.752631","last_fullsized":"2021-02-10 11:42:01.752631","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:42:01.752626","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:42:01.752626","last_clean_scrub_stamp":"2021-02-10 11:42:01.752626","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"43.4","version":"197'16","reported_seq":"34","reported_epoch":"197","state":"active+clean","last_fresh":"2021-02-10 11:42:01.543378","last_change":"2021-02-10 11:42:01.543407","last_active":"2021-02-10 11:42:01.543378","last_peered":"2021-02-10 11:42:01.543378","last_clean":"2021-02-10 11:42:01.543378","last_became_active":"2021-02-10 11:40:24.371177","last_became_peered":"2021-02-10 11:40:24.371177","last_unstale":"2021-02-10 11:42:01.543378","last_undegraded":"2021-02-10 11:42:01.543378","last_fullsized":"2021-02-10 11:42:01.543378","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"197'16","last_scrub_stamp":"2021-02-10 11:42:01.543375","last_deep_scrub":"197'16","last_deep_scrub_stamp":"2021-02-10 11:42:01.543375","last_clean_scrub_stamp":"2021-02-10 11:42:01.543375","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":10485760,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":6,"num_read_kb":6144,"num_write":10,"num_write_kb":10240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"42.5","version":"197'12","reported_seq":"25","reported_epoch":"197","state":"active+clean","last_fresh":"2021-02-10 11:42:01.135794","last_change":"2021-02-10 11:42:01.135858","last_active":"2021-02-10 11:42:01.135794","last_peered":"2021-02-10 11:42:01.135794","last_clean":"2021-02-10 11:42:01.135794","last_became_active":"2021-02-10 11:40:23.729375","last_became_peered":"2021-02-10 11:40:23.729375","last_unstale":"2021-02-10 11:42:01.135794","last_undegraded":"2021-02-10 11:42:01.135794","last_fullsized":"2021-02-10 11:42:01.135794","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"197'12","last_scrub_stamp":"2021-02-10 11:42:01.135791","last_deep_scrub":"197'12","last_deep_scrub_stamp":"2021-02-10 11:42:01.135791","last_clean_scrub_stamp":"2021-02-10 11:42:01.135791","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":160,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":"178","reported_epoch":"197","state":"active+clean","last_fresh":"2021-02-10 11:41:58.190812","last_change":"2021-02-10 11:41:58.190854","last_active":"2021-02-10 11:41:58.190812","last_peered":"2021-02-10 11:41:58.190812","last_clean":"2021-02-10 11:41:58.190812","last_became_active":"2021-02-10 11:03:38.738675","last_became_peered":"2021-02-10 11:03:38.738675","last_unstale":"2021-02-10 11:41:58.190812","last_undegraded":"2021-02-10 11:41:58.190812","last_fullsized":"2021-02-10 11:41:58.190812","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:41:58.190809","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:41:58.190809","last_clean_scrub_stamp":"2021-02-10 11:41:58.190809","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":"183","reported_epoch":"197","state":"active+clean","last_fresh":"2021-02-10 11:42:06.780050","last_change":"2021-02-10 11:42:06.780089","last_active":"2021-02-10 11:42:06.780050","last_peered":"2021-02-10 11:42:06.780050","last_clean":"2021-02-10 11:42:06.780050","last_became_active":"2021-02-10 11:03:38.739032","last_became_peered":"2021-02-10 11:03:38.739032","last_unstale":"2021-02-10 11:42:06.780050","last_undegraded":"2021-02-10 11:42:06.780050","last_fullsized":"2021-02-10 11:42:06.780050","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:42:06.780047","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:42:06.780047","last_clean_scrub_stamp":"2021-02-10 11:42:06.780047","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":"181","reported_epoch":"197","state":"active+clean","last_fresh":"2021-02-10 11:41:54.789223","last_change":"2021-02-10 11:41:54.789259","last_active":"2021-02-10 11:41:54.789223","last_peered":"2021-02-10 11:41:54.789223","last_clean":"2021-02-10 11:41:54.789223","last_became_active":"2021-02-10 11:03:38.740308","last_became_peered":"2021-02-10 11:03:38.740308","last_unstale":"2021-02-10 11:41:54.789223","last_undegraded":"2021-02-10 11:41:54.789223","last_fullsized":"2021-02-10 11:41:54.789223","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:41:54.789220","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:41:54.789220","last_clean_scrub_stamp":"2021-02-10 11:41:54.789220","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":43,"num_pg":8,"stat_sum":{"num_bytes":134218155,"num_objects":131,"num_object_clones":1,"num_object_copies":262,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":131,"num_whiteouts":0,"num_read":64,"num_read_kb":65536,"num_write":139,"num_write_kb":131079,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":268460032,"data_stored":268436310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":218,"ondisk_log_size":218,"up":16,"acting":16,"num_store_stats":8},{"poolid":42,"num_pg":8,"stat_sum":{"num_bytes":928806,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":292,"num_write_kb":1001,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5006,"num_omap_keys":18,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1925120,"data_stored":1857612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":290,"ondisk_log_size":290,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575320,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1087952,"kb_used_data":39368,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93283888,"statfs":{"total":96636764160,"available":95522701312,"internally_reserved":1073741824,"allocated":40312832,"data_stored":38844297,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 11:41:47 2021","interfaces":[{"interface":"back","average":{"1min":0.91900000000000004,"5min":0.92600000000000005,"15min":0.94999999999999996},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.2070000000000001,"5min":1.4630000000000001,"15min":1.4630000000000001},"last":1.347},{"interface":"front","average":{"1min":0.92500000000000004,"5min":0.91600000000000004,"15min":0.92700000000000005},"min":{"1min":0.502,"5min":0.38800000000000001,"15min":0.36799999999999999},"max":{"1min":1.141,"5min":1.367,"15min":1.367},"last":1.2010000000000001}]},{"osd":1,"last update":"Wed Feb 10 11:41:47 2021","interfaces":[{"interface":"back","average":{"1min":0.81699999999999995,"5min":0.79700000000000004,"15min":0.81899999999999995},"min":{"1min":0.434,"5min":0.38,"15min":0.38},"max":{"1min":1.2689999999999999,"5min":1.361,"15min":1.361},"last":1.2569999999999999},{"interface":"front","average":{"1min":0.83699999999999997,"5min":0.81299999999999994,"15min":0.82699999999999996},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1.0649999999999999,"5min":1.288,"15min":1.508},"last":1.0960000000000001}]},{"osd":2,"last update":"Wed Feb 10 11:41:47 2021","interfaces":[{"interface":"back","average":{"1min":0.81100000000000005,"5min":0.81999999999999995,"15min":0.82299999999999995},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":1.1870000000000001,"5min":1.323,"15min":1.3859999999999999},"last":1.0720000000000001},{"interface":"front","average":{"1min":0.85699999999999998,"5min":0.85299999999999998,"15min":0.86699999999999999},"min":{"1min":0.49199999999999999,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1.1040000000000001,"5min":1.2370000000000001,"15min":1.329},"last":1.0509999999999999}]},{"osd":3,"last update":"Wed Feb 10 11:41:47 2021","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":0.83099999999999996,"15min":0.84299999999999997},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.40500000000000003},"max":{"1min":1.089,"5min":1.272,"15min":1.4139999999999999},"last":0.98599999999999999},{"interface":"front","average":{"1min":0.86899999999999999,"5min":0.89000000000000001,"15min":0.89400000000000002},"min":{"1min":0.60999999999999999,"5min":0.47799999999999998,"15min":0.45500000000000002},"max":{"1min":1.3220000000000001,"5min":1.3220000000000001,"15min":1.389},"last":1.1539999999999999}]},{"osd":4,"last update":"Wed Feb 10 11:41:47 2021","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.79400000000000004,"15min":0.80600000000000005},"min":{"1min":0.47799999999999998,"5min":0.47099999999999997,"15min":0.46800000000000003},"max":{"1min":1.1279999999999999,"5min":1.1279999999999999,"15min":1.4330000000000001},"last":0.77200000000000002},{"interface":"front","average":{"1min":0.82899999999999996,"5min":0.81899999999999995,"15min":0.83099999999999996},"min":{"1min":0.48899999999999999,"5min":0.438,"15min":0.438},"max":{"1min":1.1060000000000001,"5min":1.1970000000000001,"15min":1.5489999999999999},"last":0.78100000000000003}]},{"osd":5,"last update":"Wed Feb 10 11:41:47 2021","interfaces":[{"interface":"back","average":{"1min":0.76800000000000002,"5min":0.76300000000000001,"15min":0.753},"min":{"1min":0.54800000000000004,"5min":0.49299999999999999,"15min":0.46899999999999997},"max":{"1min":1.1579999999999999,"5min":1.3360000000000001,"15min":1.3360000000000001},"last":0.81599999999999995},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.79000000000000004,"15min":0.77100000000000002},"min":{"1min":0.52800000000000002,"5min":0.47699999999999998,"15min":0.44700000000000001},"max":{"1min":1.1319999999999999,"5min":1.232,"15min":1.232},"last":0.88300000000000001}]},{"osd":6,"last update":"Wed Feb 10 11:41:47 2021","interfaces":[{"interface":"back","average":{"1min":0.876,"5min":0.85399999999999998,"15min":0.84099999999999997},"min":{"1min":0.67100000000000004,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.21,"5min":1.329,"15min":1.329},"last":0.96099999999999997},{"interface":"front","average":{"1min":0.82999999999999996,"5min":0.82299999999999995,"15min":0.80300000000000005},"min":{"1min":0.55000000000000004,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.0780000000000001,"5min":1.429,"15min":1.429},"last":0.90000000000000002}]}]},{"osd":6,"up_from":13,"seq":55834575320,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1070420,"kb_used_data":21836,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93301420,"statfs":{"total":96636764160,"available":95540654080,"internally_reserved":1073741824,"allocated":22360064,"data_stored":20896951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 11:41:56 2021","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.79000000000000004,"15min":0.79100000000000004},"min":{"1min":0.45600000000000002,"5min":0.36599999999999999,"15min":0.35599999999999998},"max":{"1min":0.97999999999999998,"5min":1.216,"15min":1.4099999999999999},"last":1.1519999999999999},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.78500000000000003,"15min":0.79500000000000004},"min":{"1min":0.53900000000000003,"5min":0.378,"15min":0.34799999999999998},"max":{"1min":1.2170000000000001,"5min":1.2969999999999999,"15min":1.3140000000000001},"last":1.008}]},{"osd":1,"last update":"Wed Feb 10 11:41:56 2021","interfaces":[{"interface":"back","average":{"1min":0.81999999999999995,"5min":0.83099999999999996,"15min":0.84899999999999998},"min":{"1min":0.45000000000000001,"5min":0.41299999999999998,"15min":0.39300000000000002},"max":{"1min":1.1359999999999999,"5min":1.224,"15min":1.427},"last":0.97699999999999998},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.81299999999999994,"15min":0.82099999999999995},"min":{"1min":0.38,"5min":0.371,"15min":0.371},"max":{"1min":0.98599999999999999,"5min":1.2230000000000001,"15min":1.2809999999999999},"last":1.0740000000000001}]},{"osd":2,"last update":"Wed Feb 10 11:41:56 2021","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.78400000000000003,"15min":0.78600000000000003},"min":{"1min":0.50800000000000001,"5min":0.44400000000000001,"15min":0.40000000000000002},"max":{"1min":1.0800000000000001,"5min":1.2569999999999999,"15min":1.2569999999999999},"last":1.046},{"interface":"front","average":{"1min":0.81699999999999995,"5min":0.82999999999999996,"15min":0.85199999999999998},"min":{"1min":0.51200000000000001,"5min":0.45900000000000002,"15min":0.33000000000000002},"max":{"1min":1.2090000000000001,"5min":1.3029999999999999,"15min":1.431},"last":1.0600000000000001}]},{"osd":3,"last update":"Wed Feb 10 11:41:56 2021","interfaces":[{"interface":"back","average":{"1min":0.91700000000000004,"5min":0.91800000000000004,"15min":0.93600000000000005},"min":{"1min":0.626,"5min":0.433,"15min":0.42099999999999999},"max":{"1min":1.1539999999999999,"5min":1.4319999999999999,"15min":1.4319999999999999},"last":1.2490000000000001},{"interface":"front","average":{"1min":0.90100000000000002,"5min":0.92300000000000004,"15min":0.93500000000000005},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.167,"5min":1.3440000000000001,"15min":1.3999999999999999},"last":1.302}]},{"osd":4,"last update":"Wed Feb 10 11:41:56 2021","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.69399999999999995,"15min":0.70799999999999996},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35699999999999998},"max":{"1min":1.028,"5min":1.099,"15min":1.163},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.71299999999999997,"15min":0.72699999999999998},"min":{"1min":0.44400000000000001,"5min":0.373,"15min":0.29699999999999999},"max":{"1min":1.046,"5min":1.119,"15min":1.2769999999999999},"last":0.73399999999999999}]},{"osd":5,"last update":"Wed Feb 10 11:41:56 2021","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":0.748,"15min":0.752},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":1.1659999999999999,"5min":1.2649999999999999,"15min":1.2649999999999999},"last":0.80100000000000005},{"interface":"front","average":{"1min":0.82399999999999995,"5min":0.85699999999999998,"15min":0.871},"min":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":1.216,"5min":1.216,"15min":1.468},"last":0.81599999999999995}]},{"osd":7,"last update":"Wed Feb 10 11:41:56 2021","interfaces":[{"interface":"back","average":{"1min":0.77800000000000002,"5min":0.77600000000000002,"15min":0.80300000000000005},"min":{"1min":0.505,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":1.2589999999999999,"5min":1.2589999999999999,"15min":1.452},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.752,"5min":0.76600000000000001,"15min":0.77700000000000002},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.103,"5min":1.1499999999999999,"15min":1.3220000000000001},"last":0.65400000000000003}]}]},{"osd":5,"up_from":13,"seq":55834575320,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1083844,"kb_used_data":35260,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93287996,"statfs":{"total":96636764160,"available":95526907904,"internally_reserved":1073741824,"allocated":36106240,"data_stored":34649196,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 11:41:42 2021","interfaces":[{"interface":"back","average":{"1min":0.93600000000000005,"5min":0.85799999999999998,"15min":0.88200000000000001},"min":{"1min":0.49099999999999999,"5min":0.40000000000000002,"15min":0.34300000000000003},"max":{"1min":1.113,"5min":1.222,"15min":1.2829999999999999},"last":1.2989999999999999},{"interface":"front","average":{"1min":0.995,"5min":0.90900000000000003,"15min":0.92100000000000004},"min":{"1min":0.73699999999999999,"5min":0.38400000000000001,"15min":0.33100000000000002},"max":{"1min":1.2589999999999999,"5min":1.27,"15min":1.444},"last":1.3220000000000001}]},{"osd":1,"last update":"Wed Feb 10 11:41:42 2021","interfaces":[{"interface":"back","average":{"1min":0.81899999999999995,"5min":0.75900000000000001,"15min":0.78100000000000003},"min":{"1min":0.44400000000000001,"5min":0.432,"15min":0.36499999999999999},"max":{"1min":1.081,"5min":1.2030000000000001,"15min":1.3169999999999999},"last":1.2290000000000001},{"interface":"front","average":{"1min":0.82699999999999996,"5min":0.77200000000000002,"15min":0.80200000000000005},"min":{"1min":0.42899999999999999,"5min":0.41499999999999998,"15min":0.40400000000000003},"max":{"1min":1.0049999999999999,"5min":1.0980000000000001,"15min":1.2849999999999999},"last":1.109}]},{"osd":2,"last update":"Wed Feb 10 11:41:42 2021","interfaces":[{"interface":"back","average":{"1min":0.89000000000000001,"5min":0.84899999999999998,"15min":0.89400000000000002},"min":{"1min":0.51200000000000001,"5min":0.47299999999999998,"15min":0.38400000000000001},"max":{"1min":1.202,"5min":1.258,"15min":1.534},"last":1.2450000000000001},{"interface":"front","average":{"1min":0.95099999999999996,"5min":0.91300000000000003,"15min":0.92500000000000004},"min":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.42199999999999999},"max":{"1min":1.232,"5min":1.3100000000000001,"15min":1.357},"last":1.345}]},{"osd":3,"last update":"Wed Feb 10 11:41:42 2021","interfaces":[{"interface":"back","average":{"1min":0.93999999999999995,"5min":0.88800000000000001,"15min":0.93000000000000005},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1.2729999999999999,"5min":1.367,"15min":1.6399999999999999},"last":1.157},{"interface":"front","average":{"1min":1,"5min":0.93000000000000005,"15min":0.95099999999999996},"min":{"1min":0.54600000000000004,"5min":0.51300000000000001,"15min":0.48399999999999999},"max":{"1min":1.252,"5min":1.252,"15min":1.3640000000000001},"last":1.125}]},{"osd":4,"last update":"Wed Feb 10 11:41:42 2021","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.68000000000000005,"15min":0.69399999999999995},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.95099999999999996,"5min":1.002,"15min":1.127},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.69499999999999995,"15min":0.72499999999999998},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.97199999999999998,"5min":1.0620000000000001,"15min":1.3009999999999999},"last":0.65200000000000002}]},{"osd":6,"last update":"Wed Feb 10 11:41:42 2021","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.71299999999999997,"15min":0.71399999999999997},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.86399999999999999,"5min":1.125,"15min":1.139},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.75900000000000001,"15min":0.76800000000000002},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.48099999999999998},"max":{"1min":1.036,"5min":1.2,"15min":1.2170000000000001},"last":0.70699999999999996}]},{"osd":7,"last update":"Wed Feb 10 11:41:42 2021","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.78800000000000003,"15min":0.80300000000000005},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.0029999999999999,"5min":1.262,"15min":1.347},"last":0.82799999999999996},{"interface":"front","average":{"1min":0.77700000000000002,"5min":0.81399999999999995,"15min":0.81899999999999995},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1.0369999999999999,"5min":1.117,"15min":1.2669999999999999},"last":0.80500000000000005}]}]},{"osd":4,"up_from":13,"seq":55834575320,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1074512,"kb_used_data":25928,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93297328,"statfs":{"total":96636764160,"available":95536463872,"internally_reserved":1073741824,"allocated":26550272,"data_stored":25091221,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 11:41:48 2021","interfaces":[{"interface":"back","average":{"1min":0.82699999999999996,"5min":0.80100000000000005,"15min":0.80700000000000005},"min":{"1min":0.51700000000000002,"5min":0.41099999999999998,"15min":0.32200000000000001},"max":{"1min":1.1379999999999999,"5min":1.2430000000000001,"15min":1.3979999999999999},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.79200000000000004,"5min":0.79900000000000004,"15min":0.81299999999999994},"min":{"1min":0.442,"5min":0.379,"15min":0.379},"max":{"1min":1.099,"5min":1.3320000000000001,"15min":1.3320000000000001},"last":0.70999999999999996}]},{"osd":1,"last update":"Wed Feb 10 11:41:48 2021","interfaces":[{"interface":"back","average":{"1min":0.81799999999999995,"5min":0.81299999999999994,"15min":0.82799999999999996},"min":{"1min":0.53500000000000003,"5min":0.46899999999999997,"15min":0.307},"max":{"1min":1.23,"5min":1.23,"15min":1.417},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.84099999999999997,"5min":0.79700000000000004,"15min":0.82099999999999995},"min":{"1min":0.53100000000000003,"5min":0.434,"15min":0.38300000000000001},"max":{"1min":1.0880000000000001,"5min":1.2929999999999999,"15min":1.387},"last":0.46999999999999997}]},{"osd":2,"last update":"Wed Feb 10 11:41:48 2021","interfaces":[{"interface":"back","average":{"1min":0.95899999999999996,"5min":0.90900000000000003,"15min":0.92800000000000005},"min":{"1min":0.57999999999999996,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1.1879999999999999,"5min":1.2949999999999999,"15min":1.4239999999999999},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.95599999999999996,"5min":0.97599999999999998,"15min":0.98499999999999999},"min":{"1min":0.63700000000000001,"5min":0.57699999999999996,"15min":0.499},"max":{"1min":1.369,"5min":1.3979999999999999,"15min":1.571},"last":0.78300000000000003}]},{"osd":3,"last update":"Wed Feb 10 11:41:48 2021","interfaces":[{"interface":"back","average":{"1min":0.82299999999999995,"5min":0.80300000000000005,"15min":0.82799999999999996},"min":{"1min":0.54400000000000004,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":1.196,"5min":1.196,"15min":1.4550000000000001},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.85099999999999998,"5min":0.84399999999999997,"15min":0.86799999999999999},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.39000000000000001},"max":{"1min":1.054,"5min":1.2190000000000001,"15min":1.325},"last":0.66300000000000003}]},{"osd":5,"last update":"Wed Feb 10 11:41:48 2021","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.71999999999999997,"15min":0.70699999999999996},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.39500000000000002},"max":{"1min":0.98299999999999998,"5min":1.0669999999999999,"15min":1.109},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.68899999999999995,"15min":0.69999999999999996},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.36599999999999999},"max":{"1min":1.004,"5min":1.0920000000000001,"15min":1.272},"last":0.73899999999999999}]},{"osd":6,"last update":"Wed Feb 10 11:41:48 2021","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.73699999999999999,"15min":0.74099999999999999},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":1.1080000000000001,"5min":1.1080000000000001,"15min":1.254},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.78500000000000003,"15min":0.79300000000000004},"min":{"1min":0.504,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.0309999999999999,"5min":1.2350000000000001,"15min":1.262},"last":0.59399999999999997}]},{"osd":7,"last update":"Wed Feb 10 11:41:48 2021","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.77700000000000002,"15min":0.79500000000000004},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.0620000000000001,"5min":1.1180000000000001,"15min":1.3240000000000001},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.78400000000000003,"5min":0.84799999999999998,"15min":0.84299999999999997},"min":{"1min":0.57899999999999996,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.069,"5min":1.2330000000000001,"15min":1.3129999999999999},"last":0.55000000000000004}]}]},{"osd":0,"up_from":13,"seq":55834575319,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1061312,"kb_used_data":12728,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93310528,"statfs":{"total":96636764160,"available":95549980672,"internally_reserved":1073741824,"allocated":13033472,"data_stored":11580398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Feb 10 11:41:55 2021","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.51100000000000001,"15min":0.49099999999999999},"min":{"1min":0.36099999999999999,"5min":0.32900000000000001,"15min":0.30299999999999999},"max":{"1min":0.73599999999999999,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.50600000000000001,"15min":0.49299999999999999},"min":{"1min":0.32800000000000001,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.746,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.39500000000000002}]},{"osd":2,"last update":"Wed Feb 10 11:41:55 2021","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.54800000000000004,"15min":0.54200000000000004},"min":{"1min":0.36399999999999999,"5min":0.318,"15min":0.318},"max":{"1min":0.81499999999999995,"5min":0.94799999999999995,"15min":1.0249999999999999},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.54800000000000004,"15min":0.55200000000000005},"min":{"1min":0.42299999999999999,"5min":0.40600000000000003,"15min":0.35599999999999998},"max":{"1min":0.64300000000000002,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.48399999999999999}]},{"osd":3,"last update":"Wed Feb 10 11:41:55 2021","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.52600000000000002,"15min":0.51300000000000001},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.33900000000000002},"max":{"1min":0.93700000000000006,"5min":0.93700000000000006,"15min":0.93700000000000006},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.61799999999999999,"15min":0.61099999999999999},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.36299999999999999},"max":{"1min":0.86399999999999999,"5min":0.878,"15min":0.97599999999999998},"last":0.61499999999999999}]},{"osd":4,"last update":"Wed Feb 10 11:41:55 2021","interfaces":[{"interface":"back","average":{"1min":0.92600000000000005,"5min":0.98399999999999999,"15min":0.98399999999999999},"min":{"1min":0.53100000000000003,"5min":0.499,"15min":0.41199999999999998},"max":{"1min":1.2270000000000001,"5min":1.5189999999999999,"15min":1.53},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.95299999999999996,"5min":0.98699999999999999,"15min":0.98299999999999998},"min":{"1min":0.67300000000000004,"5min":0.498,"15min":0.498},"max":{"1min":1.1950000000000001,"5min":1.4390000000000001,"15min":1.4710000000000001},"last":0.70899999999999996}]},{"osd":5,"last update":"Wed Feb 10 11:41:55 2021","interfaces":[{"interface":"back","average":{"1min":0.90500000000000003,"5min":0.997,"15min":1.008},"min":{"1min":0.60399999999999998,"5min":0.59599999999999997,"15min":0.48099999999999998},"max":{"1min":1.2,"5min":1.353,"15min":1.484},"last":0.67400000000000004},{"interface":"front","average":{"1min":1.006,"5min":1.0289999999999999,"15min":1.0029999999999999},"min":{"1min":0.68600000000000005,"5min":0.57299999999999995,"15min":0.51700000000000002},"max":{"1min":1.2649999999999999,"5min":1.4059999999999999,"15min":1.4359999999999999},"last":0.628}]},{"osd":6,"last update":"Wed Feb 10 11:41:55 2021","interfaces":[{"interface":"back","average":{"1min":0.96599999999999997,"5min":1.03,"15min":1.018},"min":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.57299999999999995},"max":{"1min":1.2629999999999999,"5min":1.417,"15min":1.417},"last":0.76700000000000002},{"interface":"front","average":{"1min":0.97799999999999998,"5min":1.0429999999999999,"15min":1.024},"min":{"1min":0.69599999999999995,"5min":0.629,"15min":0.55100000000000005},"max":{"1min":1.331,"5min":1.4870000000000001,"15min":1.4870000000000001},"last":0.74399999999999999}]},{"osd":7,"last update":"Wed Feb 10 11:41:55 2021","interfaces":[{"interface":"back","average":{"1min":1.0169999999999999,"5min":1.0640000000000001,"15min":1.04},"min":{"1min":0.77100000000000002,"5min":0.71099999999999997,"15min":0.57599999999999996},"max":{"1min":1.417,"5min":1.726,"15min":1.726},"last":0.79000000000000004},{"interface":"front","average":{"1min":0.98199999999999998,"5min":1.022,"15min":1.0069999999999999},"min":{"1min":0.73599999999999999,"5min":0.62,"15min":0.60499999999999998},"max":{"1min":1.3460000000000001,"5min":1.6080000000000001,"15min":1.6080000000000001},"last":0.752}]}]},{"osd":1,"up_from":13,"seq":55834575320,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1092044,"kb_used_data":43456,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93279796,"statfs":{"total":96636764160,"available":95518511104,"internally_reserved":1073741824,"allocated":44498944,"data_stored":43038035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 11:42:03 2021","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.59099999999999997,"15min":0.60299999999999998},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.82599999999999996,"5min":1.056,"15min":1.119},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.60399999999999998,"15min":0.61699999999999999},"min":{"1min":0.46600000000000003,"5min":0.40500000000000003,"15min":0.316},"max":{"1min":0.84599999999999997,"5min":0.98799999999999999,"15min":1.038},"last":0.58399999999999996}]},{"osd":2,"last update":"Wed Feb 10 11:42:03 2021","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.47999999999999998,"15min":0.48499999999999999},"min":{"1min":0.35899999999999999,"5min":0.33100000000000002,"15min":0.30499999999999999},"max":{"1min":0.66000000000000003,"5min":0.75900000000000001,"15min":1.0209999999999999},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.504,"5min":0.53800000000000003,"15min":0.55100000000000005},"min":{"1min":0.41499999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.63300000000000001,"5min":0.94599999999999995,"15min":1.1399999999999999},"last":0.46000000000000002}]},{"osd":3,"last update":"Wed Feb 10 11:42:03 2021","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.58999999999999997,"15min":0.60399999999999998},"min":{"1min":0.41099999999999998,"5min":0.38100000000000001,"15min":0.35999999999999999},"max":{"1min":0.76800000000000002,"5min":0.94199999999999995,"15min":1.244},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.53600000000000003,"15min":0.54800000000000004},"min":{"1min":0.36799999999999999,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.72499999999999998,"5min":0.83799999999999997,"15min":1.155},"last":0.51600000000000001}]},{"osd":4,"last update":"Wed Feb 10 11:42:03 2021","interfaces":[{"interface":"back","average":{"1min":0.93799999999999994,"5min":0.95599999999999996,"15min":0.96999999999999997},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.45300000000000001},"max":{"1min":1.2529999999999999,"5min":1.351,"15min":1.5229999999999999},"last":1.2949999999999999},{"interface":"front","average":{"1min":0.92700000000000005,"5min":0.98099999999999998,"15min":0.99399999999999999},"min":{"1min":0.66700000000000004,"5min":0.47599999999999998,"15min":0.46800000000000003},"max":{"1min":1.194,"5min":1.3540000000000001,"15min":1.508},"last":1.2669999999999999}]},{"osd":5,"last update":"Wed Feb 10 11:42:03 2021","interfaces":[{"interface":"back","average":{"1min":0.89700000000000002,"5min":0.97899999999999998,"15min":0.98699999999999999},"min":{"1min":0.55200000000000005,"5min":0.496,"15min":0.496},"max":{"1min":1.2689999999999999,"5min":1.3440000000000001,"15min":1.6279999999999999},"last":1.071},{"interface":"front","average":{"1min":0.95399999999999996,"5min":0.98199999999999998,"15min":0.998},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.46899999999999997},"max":{"1min":1.3149999999999999,"5min":1.413,"15min":1.5760000000000001},"last":1.244}]},{"osd":6,"last update":"Wed Feb 10 11:42:03 2021","interfaces":[{"interface":"back","average":{"1min":0.96199999999999997,"5min":1.006,"15min":1.0049999999999999},"min":{"1min":0.72599999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"max":{"1min":1.3080000000000001,"5min":1.3740000000000001,"15min":1.4059999999999999},"last":1.321},{"interface":"front","average":{"1min":0.94199999999999995,"5min":1.0169999999999999,"15min":1.0249999999999999},"min":{"1min":0.628,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":1.3200000000000001,"5min":1.387,"15min":1.456},"last":1.2829999999999999}]},{"osd":7,"last update":"Wed Feb 10 11:42:03 2021","interfaces":[{"interface":"back","average":{"1min":0.95399999999999996,"5min":1.0169999999999999,"15min":1.036},"min":{"1min":0.67700000000000005,"5min":0.495,"15min":0.495},"max":{"1min":1.2589999999999999,"5min":1.472,"15min":1.786},"last":1.2310000000000001},{"interface":"front","average":{"1min":0.97299999999999998,"5min":0.995,"15min":1.008},"min":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.56599999999999995},"max":{"1min":1.361,"5min":1.3720000000000001,"15min":1.494},"last":1.1000000000000001}]}]},{"osd":2,"up_from":13,"seq":55834575319,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1122768,"kb_used_data":74184,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93249072,"statfs":{"total":96636764160,"available":95487049728,"internally_reserved":1073741824,"allocated":75964416,"data_stored":74495787,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 11:41:44 2021","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.63700000000000001,"15min":0.628},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":1.0960000000000001,"5min":1.0960000000000001,"15min":1.103},"last":0.83299999999999996},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.63300000000000001,"15min":0.63},"min":{"1min":0.41599999999999998,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.99199999999999999,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.76100000000000001}]},{"osd":1,"last update":"Wed Feb 10 11:41:44 2021","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53300000000000003,"15min":0.52900000000000003},"min":{"1min":0.379,"5min":0.34200000000000003,"15min":0.32000000000000001},"max":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":1.0509999999999999},"last":0.72099999999999997},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.53100000000000003,"15min":0.51900000000000002},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.33400000000000002},"max":{"1min":0.68400000000000005,"5min":0.82599999999999996,"15min":0.88800000000000001},"last":0.59299999999999997}]},{"osd":3,"last update":"Wed Feb 10 11:41:44 2021","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51200000000000001,"15min":0.52000000000000002},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.63600000000000001,"5min":0.69099999999999995,"15min":0.78100000000000003},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.58499999999999996,"15min":0.60299999999999998},"min":{"1min":0.438,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.79200000000000004,"5min":0.88600000000000001,"15min":1.0289999999999999},"last":0.55100000000000005}]},{"osd":4,"last update":"Wed Feb 10 11:41:44 2021","interfaces":[{"interface":"back","average":{"1min":0.97499999999999998,"5min":1.091,"15min":1.081},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.335,"5min":1.4430000000000001,"15min":1.4690000000000001},"last":1.0449999999999999},{"interface":"front","average":{"1min":0.96699999999999997,"5min":1.101,"15min":1.095},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.373},"max":{"1min":1.359,"5min":1.419,"15min":1.77},"last":1.0669999999999999}]},{"osd":5,"last update":"Wed Feb 10 11:41:44 2021","interfaces":[{"interface":"back","average":{"1min":0.89400000000000002,"5min":0.995,"15min":0.98599999999999999},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.296,"5min":1.47,"15min":1.47},"last":0.95599999999999996},{"interface":"front","average":{"1min":0.86099999999999999,"5min":0.97099999999999997,"15min":0.98899999999999999},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":1.3220000000000001,"5min":1.4199999999999999,"15min":1.613},"last":0.752}]},{"osd":6,"last update":"Wed Feb 10 11:41:44 2021","interfaces":[{"interface":"back","average":{"1min":0.89400000000000002,"5min":0.98399999999999999,"15min":0.996},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.2509999999999999,"5min":1.569,"15min":1.569},"last":0.80400000000000005},{"interface":"front","average":{"1min":0.92200000000000004,"5min":1.002,"15min":0.998},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.1519999999999999,"5min":1.3999999999999999,"15min":1.504},"last":0.93799999999999994}]},{"osd":7,"last update":"Wed Feb 10 11:41:44 2021","interfaces":[{"interface":"back","average":{"1min":0.89600000000000002,"5min":0.996,"15min":1.0049999999999999},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.2290000000000001,"5min":1.4810000000000001,"15min":1.4810000000000001},"last":0.85499999999999998},{"interface":"front","average":{"1min":0.92700000000000005,"5min":1.0209999999999999,"15min":1.0229999999999999},"min":{"1min":0.61199999999999999,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":1.284,"5min":1.4750000000000001,"15min":1.4950000000000001},"last":0.66700000000000004}]}]},{"osd":3,"up_from":13,"seq":55834575319,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1079744,"kb_used_data":31160,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93292096,"statfs":{"total":96636764160,"available":95531106304,"internally_reserved":1073741824,"allocated":31907840,"data_stored":30454869,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 11:41:51 2021","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.49099999999999999,"15min":0.49199999999999999},"min":{"1min":0.376,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.75,"5min":0.96099999999999997,"15min":1.1479999999999999},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.46999999999999997,"15min":0.47199999999999998},"min":{"1min":0.34799999999999998,"5min":0.34399999999999997,"15min":0.30199999999999999},"max":{"1min":0.748,"5min":0.748,"15min":0.79600000000000004},"last":0.56699999999999995}]},{"osd":1,"last update":"Wed Feb 10 11:41:51 2021","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.49299999999999999,"15min":0.495},"min":{"1min":0.36899999999999999,"5min":0.32100000000000001,"15min":0.311},"max":{"1min":0.66600000000000004,"5min":0.83399999999999996,"15min":1.2709999999999999},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.55800000000000005,"15min":0.56299999999999994},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.316},"max":{"1min":0.70599999999999996,"5min":0.86399999999999999,"15min":0.997},"last":0.69899999999999995}]},{"osd":2,"last update":"Wed Feb 10 11:41:51 2021","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.55700000000000005,"15min":0.56100000000000005},"min":{"1min":0.46000000000000002,"5min":0.374,"15min":0.35299999999999998},"max":{"1min":0.76600000000000001,"5min":0.995,"15min":0.995},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.61399999999999999,"15min":0.63200000000000001},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.35099999999999998},"max":{"1min":0.74299999999999999,"5min":0.91800000000000004,"15min":1.04},"last":0.85699999999999998}]},{"osd":4,"last update":"Wed Feb 10 11:41:51 2021","interfaces":[{"interface":"back","average":{"1min":0.82699999999999996,"5min":0.94599999999999995,"15min":0.95199999999999996},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.39100000000000001},"max":{"1min":1.1180000000000001,"5min":1.3779999999999999,"15min":1.5760000000000001},"last":1.2370000000000001},{"interface":"front","average":{"1min":0.90200000000000002,"5min":0.98699999999999999,"15min":0.98299999999999998},"min":{"1min":0.55200000000000005,"5min":0.52600000000000002,"15min":0.52000000000000002},"max":{"1min":1.1950000000000001,"5min":1.494,"15min":1.6659999999999999},"last":1.1419999999999999}]},{"osd":5,"last update":"Wed Feb 10 11:41:51 2021","interfaces":[{"interface":"back","average":{"1min":0.95199999999999996,"5min":0.99199999999999999,"15min":0.99299999999999999},"min":{"1min":0.63100000000000001,"5min":0.58199999999999996,"15min":0.55800000000000005},"max":{"1min":1.27,"5min":1.448,"15min":1.556},"last":1.21},{"interface":"front","average":{"1min":0.91700000000000004,"5min":0.98399999999999999,"15min":0.97299999999999998},"min":{"1min":0.64000000000000001,"5min":0.51100000000000001,"15min":0.47999999999999998},"max":{"1min":1.2130000000000001,"5min":1.3959999999999999,"15min":1.405},"last":1.339}]},{"osd":6,"last update":"Wed Feb 10 11:41:51 2021","interfaces":[{"interface":"back","average":{"1min":0.89900000000000002,"5min":0.98599999999999999,"15min":0.996},"min":{"1min":0.59299999999999997,"5min":0.54100000000000004,"15min":0.51100000000000001},"max":{"1min":1.216,"5min":1.3500000000000001,"15min":1.5409999999999999},"last":1.1910000000000001},{"interface":"front","average":{"1min":0.96599999999999997,"5min":1.0029999999999999,"15min":0.995},"min":{"1min":0.68500000000000005,"5min":0.58599999999999997,"15min":0.57599999999999996},"max":{"1min":1.22,"5min":1.458,"15min":1.458},"last":1.258}]},{"osd":7,"last update":"Wed Feb 10 11:41:51 2021","interfaces":[{"interface":"back","average":{"1min":0.93300000000000005,"5min":0.99399999999999999,"15min":0.99299999999999999},"min":{"1min":0.67900000000000005,"5min":0.55700000000000005,"15min":0.50600000000000001},"max":{"1min":1.2310000000000001,"5min":1.264,"15min":1.389},"last":1.409},{"interface":"front","average":{"1min":0.93000000000000005,"5min":0.98699999999999999,"15min":0.99299999999999999},"min":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.56999999999999995},"max":{"1min":1.242,"5min":1.488,"15min":1.4970000000000001},"last":1.1160000000000001}]}]}]}} 2021-02-10T11:42:39.563 DEBUG:teuthology.orchestra.run.gibba002:> sudo ceph --cluster ceph --mon-client-directed-command-retry 5 tell mon.* injectargs -- --no-mon-health-to-clog 2021-02-10T11:42:39.991 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:39.971 7f25209c9700 1 Processor -- start 2021-02-10T11:42:39.991 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:39.971 7f25209c9700 1 -- start start 2021-02-10T11:42:39.991 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:39.971 7f25209c9700 1 --2- >> v2:172.21.2.113:3300/0 conn(0x7f251c10a040 0x7f251c10a440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:39.992 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:39.971 7f25209c9700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f251c10aa60 0x7f251c10ae60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:39.992 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:39.971 7f25209c9700 1 -- --> v1:172.21.2.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f251c1b0d40 con 0x7f251c0e53a0 2021-02-10T11:42:39.992 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:39.971 7f25209c9700 1 -- --> v2:172.21.2.113:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f251c0e5f80 con 0x7f251c10a040 2021-02-10T11:42:39.992 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:39.971 7f25209c9700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f251c109ac0 con 0x7f251c10aa60 2021-02-10T11:42:39.992 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:39.972 7f251a61c700 1 --1- >> v1:172.21.2.102:6789/0 conn(0x7f251c0e53a0 0x7f251c1b0550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.102:6789/0 says I am v1:172.21.2.102:34218/0 (socket says 172.21.2.102:34218) 2021-02-10T11:42:39.992 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:39.972 7f251a61c700 1 -- 172.21.2.102:0/2117186836 learned_addr learned my addr 172.21.2.102:0/2117186836 (peer_addr_for_me v1:172.21.2.102:0/0) 2021-02-10T11:42:39.992 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:39.972 7f251961a700 1 -- 172.21.2.102:0/2117186836 <== mon.0 v1:172.21.2.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2376375792 0 0) 0x7f25040010b0 con 0x7f251c0e53a0 2021-02-10T11:42:39.993 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:39.972 7f251961a700 1 -- 172.21.2.102:0/2117186836 --> v1:172.21.2.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2508002fe0 con 0x7f251c0e53a0 2021-02-10T11:42:39.993 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:39.972 7f2519e1b700 1 --2- 172.21.2.102:0/2117186836 >> v2:172.21.2.113:3300/0 conn(0x7f251c10a040 0x7f251c10a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:39.993 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:39.972 7f251ae1d700 1 --2- 172.21.2.102:0/2117186836 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f251c10aa60 0x7f251c10ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:39.993 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:39.972 7f251961a700 1 -- 172.21.2.102:0/2117186836 <== mon.0 v1:172.21.2.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1121750445 0 0) 0x7f25040010b0 con 0x7f251c0e53a0 2021-02-10T11:42:39.993 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:39.972 7f251961a700 1 -- 172.21.2.102:0/2117186836 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f251c10aa60 msgr2=0x7f251c10ae60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:42:39.993 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:39.972 7f251961a700 1 --2- 172.21.2.102:0/2117186836 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f251c10aa60 0x7f251c10ae60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:39.993 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:39.972 7f251961a700 1 -- 172.21.2.102:0/2117186836 >> v2:172.21.2.113:3300/0 conn(0x7f251c10a040 msgr2=0x7f251c10a440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:42:39.994 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:39.972 7f251961a700 1 --2- 172.21.2.102:0/2117186836 >> v2:172.21.2.113:3300/0 conn(0x7f251c10a040 0x7f251c10a440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:39.994 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:39.972 7f251961a700 1 -- 172.21.2.102:0/2117186836 --> v1:172.21.2.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f251c0e57a0 con 0x7f251c0e53a0 2021-02-10T11:42:39.994 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:39.972 7f251961a700 1 -- 172.21.2.102:0/2117186836 <== mon.0 v1:172.21.2.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 3317211690 0 0) 0x7f25040015b0 con 0x7f251c0e53a0 2021-02-10T11:42:39.994 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:39.972 7f251961a700 1 -- 172.21.2.102:0/2117186836 >> v1:172.21.2.102:6789/0 conn(0x7f251c0e53a0 legacy=0x7f251c1b0550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:39.994 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:39.972 7f251961a700 1 --2- 172.21.2.102:0/2117186836 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2508003ac0 0x7f2508005ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:39.994 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:39.973 7f251961a700 1 --2- 172.21.2.102:0/2117186836 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f250804a7a0 0x7f250804cca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:39.994 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:39.973 7f251961a700 1 --2- 172.21.2.102:0/2117186836 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f250804d5a0 0x7f250804fa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:39.995 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:39.973 7f251961a700 1 -- 172.21.2.102:0/2117186836 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25080502e0 con 0x7f2508003ac0 2021-02-10T11:42:39.995 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:39.973 7f251961a700 1 -- 172.21.2.102:0/2117186836 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2508050500 con 0x7f250804a7a0 2021-02-10T11:42:39.995 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:39.973 7f251961a700 1 -- 172.21.2.102:0/2117186836 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2508050720 con 0x7f250804d5a0 2021-02-10T11:42:39.995 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:39.973 7f251a61c700 1 --2- 172.21.2.102:0/2117186836 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2508003ac0 0x7f2508005ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:39.995 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:39.973 7f2519e1b700 1 --2- 172.21.2.102:0/2117186836 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f250804a7a0 0x7f250804cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:39.995 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:39.973 7f251a61c700 1 -- 172.21.2.102:0/2117186836 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f250804d5a0 msgr2=0x7f250804fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:42:39.995 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:39.973 7f251a61c700 1 --2- 172.21.2.102:0/2117186836 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f250804d5a0 0x7f250804fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:39.996 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:39.973 7f251a61c700 1 -- 172.21.2.102:0/2117186836 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f250804a7a0 msgr2=0x7f250804cca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:42:39.996 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:39.973 7f251a61c700 1 --2- 172.21.2.102:0/2117186836 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f250804a7a0 0x7f250804cca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:39.996 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:39.973 7f251a61c700 1 -- 172.21.2.102:0/2117186836 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25080509c0 con 0x7f2508003ac0 2021-02-10T11:42:39.996 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:39.973 7f251a61c700 1 --2- 172.21.2.102:0/2117186836 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2508003ac0 0x7f2508005ef0 secure :-1 s=READY pgs=3497 cs=0 l=1 rev1=1 rx=0x7f2504007990 tx=0x7f2504007d30).ready entity=mon.0 client_cookie=ef0cfa50bddc5ac server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:39.996 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:39.973 7f251961a700 1 -- 172.21.2.102:0/2117186836 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f25040009b0 con 0x7f2508003ac0 2021-02-10T11:42:39.996 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:39.973 7f251961a700 1 -- 172.21.2.102:0/2117186836 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f250400a310 con 0x7f2508003ac0 2021-02-10T11:42:39.996 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:39.973 7f251961a700 1 -- 172.21.2.102:0/2117186836 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f250400a830 con 0x7f2508003ac0 2021-02-10T11:42:39.996 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:39.973 7f25209c9700 1 -- 172.21.2.102:0/2117186836 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2508003ac0 msgr2=0x7f2508005ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:39.996 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:39.973 7f25209c9700 1 --2- 172.21.2.102:0/2117186836 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f2508003ac0 0x7f2508005ef0 secure :-1 s=READY pgs=3497 cs=0 l=1 rev1=1 rx=0x7f2504007990 tx=0x7f2504007d30).stop 2021-02-10T11:42:39.997 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:39.973 7f251ae1d700 1 -- 172.21.2.102:0/2117186836 reap_dead start 2021-02-10T11:42:39.997 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:39.974 7f25209c9700 1 -- 172.21.2.102:0/2117186836 shutdown_connections 2021-02-10T11:42:39.997 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:39.974 7f25209c9700 1 -- 172.21.2.102:0/2117186836 >> 172.21.2.102:0/2117186836 conn(0x7f251c0ce4b0 msgr2=0x7f251c0e45d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:39.997 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:39.974 7f25209c9700 1 -- 172.21.2.102:0/2117186836 shutdown_connections 2021-02-10T11:42:39.997 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:39.974 7f25209c9700 1 -- 172.21.2.102:0/2117186836 wait complete. 2021-02-10T11:42:39.997 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:39.989 7f25209c9700 1 Processor -- start 2021-02-10T11:42:39.997 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:39.989 7f25209c9700 1 -- start start 2021-02-10T11:42:39.998 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:39.989 7f25209c9700 1 --2- >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f251c09feb0 0x7f251c0a07d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:39.999 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:39.989 7f25209c9700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f251c1a35e0 0x7f251c1a39e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:39.999 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:39.989 7f25209c9700 1 --2- >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f251c0bee50 0x7f251c0bf250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:39.999 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:39.989 7f25209c9700 1 -- --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f251c0a0450 con 0x7f251c1a35e0 2021-02-10T11:42:39.999 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:39.989 7f25209c9700 1 -- --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f251c0a0df0 con 0x7f251c09feb0 2021-02-10T11:42:39.999 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:39.989 7f25209c9700 1 -- --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f251c0bd230 con 0x7f251c0bee50 2021-02-10T11:42:39.999 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:39.989 7f2519e1b700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f251c1a35e0 0x7f251c1a39e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:39.999 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:39.989 7f2519e1b700 1 --2- >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f251c1a35e0 0x7f251c1a39e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.102:3300/0 says I am v2:172.21.2.102:42008/0 (socket says 172.21.2.102:42008) 2021-02-10T11:42:40.000 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:39.989 7f2519e1b700 1 -- 172.21.2.102:0/109960675 learned_addr learned my addr 172.21.2.102:0/109960675 (peer_addr_for_me v2:172.21.2.102:0/0) 2021-02-10T11:42:40.000 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:39.990 7f2519e1b700 1 -- 172.21.2.102:0/109960675 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f251c0bee50 msgr2=0x7f251c0bf250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:40.000 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:39.990 7f2519e1b700 1 --2- 172.21.2.102:0/109960675 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f251c0bee50 0x7f251c0bf250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:40.000 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:39.990 7f2519e1b700 1 -- 172.21.2.102:0/109960675 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f251c09feb0 msgr2=0x7f251c0a07d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:40.000 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:39.990 7f2519e1b700 1 --2- 172.21.2.102:0/109960675 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f251c09feb0 0x7f251c0a07d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:40.000 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:39.990 7f2519e1b700 1 -- 172.21.2.102:0/109960675 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f251c0bd450 con 0x7f251c1a35e0 2021-02-10T11:42:40.000 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:39.990 7f2519e1b700 1 --2- 172.21.2.102:0/109960675 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f251c1a35e0 0x7f251c1a39e0 secure :-1 s=READY pgs=3498 cs=0 l=1 rev1=1 rx=0x7f250c006f60 tx=0x7f250c00b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:40.000 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:39.990 7f2512ffd700 1 -- 172.21.2.102:0/109960675 <== mon.0 v2:172.21.2.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f250c00bd60 con 0x7f251c1a35e0 2021-02-10T11:42:40.000 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:39.990 7f2512ffd700 1 -- 172.21.2.102:0/109960675 <== mon.0 v2:172.21.2.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f250c000c90 con 0x7f251c1a35e0 2021-02-10T11:42:40.001 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:39.990 7f2512ffd700 1 -- 172.21.2.102:0/109960675 <== mon.0 v2:172.21.2.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f250c002220 con 0x7f251c1a35e0 2021-02-10T11:42:40.001 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:39.990 7f25209c9700 1 -- 172.21.2.102:0/109960675 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f251c0bd450 con 0x7f251c1a35e0 2021-02-10T11:42:40.001 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:39.990 7f25209c9700 1 -- 172.21.2.102:0/109960675 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f251c0bd9e0 con 0x7f251c1a35e0 2021-02-10T11:42:40.001 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:39.991 7f2512ffd700 1 -- 172.21.2.102:0/109960675 <== mon.0 v2:172.21.2.102:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f250c00a000 con 0x7f251c1a35e0 2021-02-10T11:42:40.001 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:39.991 7f2512ffd700 1 --2- 172.21.2.102:0/109960675 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f25000186c0 0x7f250001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:40.001 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:39.991 7f2512ffd700 1 -- 172.21.2.102:0/109960675 <== mon.0 v2:172.21.2.102:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 5573+0+0 (secure 0 0 0) 0x7f250c032570 con 0x7f251c1a35e0 2021-02-10T11:42:40.001 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:39.991 7f251a61c700 1 --2- 172.21.2.102:0/109960675 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f25000186c0 0x7f250001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:40.001 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:39.991 7f251a61c700 1 --2- 172.21.2.102:0/109960675 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f25000186c0 0x7f250001abc0 crc :-1 s=READY pgs=1521 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:40.023 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:40.021 7f25209c9700 1 -- 172.21.2.102:0/109960675 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f251c1026e0 con 0x7f251c1a35e0 2021-02-10T11:42:40.024 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:40.022 7f2512ffd700 1 -- 172.21.2.102:0/109960675 <== mon.0 v2:172.21.2.102:3300/0 6 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+968 (secure 0 0 0) 0x7f250c0009b0 con 0x7f251c1a35e0 2021-02-10T11:42:40.025 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:40.023 7f25209c9700 1 -- 172.21.2.102:0/109960675 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f251c109ac0 con 0x7f251c1a35e0 2021-02-10T11:42:40.027 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:40.025 7f2512ffd700 1 -- 172.21.2.102:0/109960675 <== mon.0 v2:172.21.2.102:3300/0 7 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f250c030d20 con 0x7f251c1a35e0 2021-02-10T11:42:40.157 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:40.155 7f25209c9700 1 -- 172.21.2.102:0/109960675 --> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] -- mon_command({"injected_args": ["--no-mon-health-to-clog"], "prefix": "injectargs"} v 0) v1 -- 0x7f251c1026e0 con 0x7f251c1a35e0 2021-02-10T11:42:40.158 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:40.156 7f2512ffd700 1 -- 172.21.2.102:0/109960675 <== mon.0 v2:172.21.2.102:3300/0 8 ==== mon_command_ack([{"injected_args": ["--no-mon-health-to-clog"], "prefix": "injectargs"}]=0 injectargs:mon_health_to_clog = 'false' v0) v1 ==== 144+0+0 (secure 0 0 0) 0x7f250c0009b0 con 0x7f251c1a35e0 2021-02-10T11:42:40.161 INFO:teuthology.orchestra.run.gibba002.stderr:mon.a: injectargs:mon_health_to_clog = 'false' 2021-02-10T11:42:40.161 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:40.159 7f25209c9700 1 -- 172.21.2.102:0/109960675 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f251c1a35e0 msgr2=0x7f251c1a39e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:40.161 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:40.159 7f25209c9700 1 --2- 172.21.2.102:0/109960675 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f251c1a35e0 0x7f251c1a39e0 secure :-1 s=READY pgs=3498 cs=0 l=1 rev1=1 rx=0x7f250c006f60 tx=0x7f250c00b970).stop 2021-02-10T11:42:40.161 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:40.159 7f25209c9700 1 --2- 172.21.2.102:0/109960675 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f251c09feb0 0x7f251c0be270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:40.161 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:40.159 7f25209c9700 1 -- 172.21.2.102:0/109960675 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f251c0a1010 con 0x7f251c09feb0 2021-02-10T11:42:40.162 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:40.159 7f2519e1b700 1 --2- 172.21.2.102:0/109960675 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f251c09feb0 0x7f251c0be270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:40.162 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:40.160 7f2519e1b700 1 -- 172.21.2.102:0/109960675 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f251c0bd9e0 con 0x7f251c09feb0 2021-02-10T11:42:40.162 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:40.160 7f2519e1b700 1 -- 172.21.2.102:0/109960675 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f250c00a5f0 con 0x7f251c09feb0 2021-02-10T11:42:40.162 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:40.160 7f2519e1b700 1 --2- 172.21.2.102:0/109960675 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f251c09feb0 0x7f251c0be270 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 rx=0x7f250c021c70 tx=0x7f250c021eb0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:40.163 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:40.161 7f2512ffd700 1 -- 172.21.2.102:0/109960675 <== mon.1 v2:172.21.2.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f250c0009b0 con 0x7f251c09feb0 2021-02-10T11:42:40.163 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:40.161 7f2512ffd700 1 -- 172.21.2.102:0/109960675 <== mon.1 v2:172.21.2.113:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f250c00ec00 con 0x7f251c09feb0 2021-02-10T11:42:40.163 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:40.161 7f2512ffd700 1 -- 172.21.2.102:0/109960675 <== mon.1 v2:172.21.2.113:3300/0 3 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f250c012080 con 0x7f251c09feb0 2021-02-10T11:42:40.165 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:40.164 7f2512ffd700 1 -- 172.21.2.102:0/109960675 <== mon.1 v2:172.21.2.113:3300/0 4 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f250c02e310 con 0x7f251c09feb0 2021-02-10T11:42:40.296 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:40.294 7f25209c9700 1 -- 172.21.2.102:0/109960675 --> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] -- mon_command({"injected_args": ["--no-mon-health-to-clog"], "prefix": "injectargs"} v 0) v1 -- 0x7f251c109ac0 con 0x7f251c09feb0 2021-02-10T11:42:40.298 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:40.296 7f2512ffd700 1 -- 172.21.2.102:0/109960675 <== mon.1 v2:172.21.2.113:3300/0 5 ==== mon_command_ack([{"injected_args": ["--no-mon-health-to-clog"], "prefix": "injectargs"}]=0 injectargs:mon_health_to_clog = 'false' v0) v1 ==== 144+0+0 (secure 0 0 0) 0x7f250c014760 con 0x7f251c09feb0 2021-02-10T11:42:40.299 INFO:teuthology.orchestra.run.gibba002.stderr:mon.b: injectargs:mon_health_to_clog = 'false' 2021-02-10T11:42:40.299 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:40.298 7f25209c9700 1 -- 172.21.2.102:0/109960675 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f251c09feb0 msgr2=0x7f251c0be270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:40.300 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:40.298 7f25209c9700 1 --2- 172.21.2.102:0/109960675 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f251c09feb0 0x7f251c0be270 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 rx=0x7f250c021c70 tx=0x7f250c021eb0).stop 2021-02-10T11:42:40.300 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:40.298 7f25209c9700 1 --2- 172.21.2.102:0/109960675 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f251c0c4660 0x7f251c0c3de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:40.300 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:40.298 7f25209c9700 1 -- 172.21.2.102:0/109960675 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f251c0be980 con 0x7f251c0c4660 2021-02-10T11:42:40.300 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:40.299 7f2519e1b700 1 --2- 172.21.2.102:0/109960675 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f251c0c4660 0x7f251c0c3de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:40.301 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:40.299 7f2519e1b700 1 -- 172.21.2.102:0/109960675 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f251c109ac0 con 0x7f251c0c4660 2021-02-10T11:42:40.301 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:40.299 7f2519e1b700 1 -- 172.21.2.102:0/109960675 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f250c02e310 con 0x7f251c0c4660 2021-02-10T11:42:40.302 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:40.299 7f2519e1b700 1 --2- 172.21.2.102:0/109960675 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f251c0c4660 0x7f251c0c3de0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 rx=0x7f250c013b20 tx=0x7f250c013d50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:40.302 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:40.300 7f2512ffd700 1 -- 172.21.2.102:0/109960675 <== mon.2 v2:172.21.2.113:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f250c014760 con 0x7f251c0c4660 2021-02-10T11:42:40.302 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:40.300 7f2512ffd700 1 -- 172.21.2.102:0/109960675 <== mon.2 v2:172.21.2.113:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f250c02eff0 con 0x7f251c0c4660 2021-02-10T11:42:40.303 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:40.300 7f2512ffd700 1 -- 172.21.2.102:0/109960675 <== mon.2 v2:172.21.2.113:3301/0 3 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f250c02bd50 con 0x7f251c0c4660 2021-02-10T11:42:40.305 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:40.303 7f2512ffd700 1 -- 172.21.2.102:0/109960675 <== mon.2 v2:172.21.2.113:3301/0 4 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f250c023570 con 0x7f251c0c4660 2021-02-10T11:42:40.434 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:40.432 7f25209c9700 1 -- 172.21.2.102:0/109960675 --> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] -- mon_command({"injected_args": ["--no-mon-health-to-clog"], "prefix": "injectargs"} v 0) v1 -- 0x7f251c1026e0 con 0x7f251c0c4660 2021-02-10T11:42:40.436 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:40.435 7f2512ffd700 1 -- 172.21.2.102:0/109960675 <== mon.2 v2:172.21.2.113:3301/0 5 ==== mon_command_ack([{"injected_args": ["--no-mon-health-to-clog"], "prefix": "injectargs"}]=0 injectargs:mon_health_to_clog = 'false' v0) v1 ==== 144+0+0 (secure 0 0 0) 0x7f250c01f140 con 0x7f251c0c4660 2021-02-10T11:42:40.437 INFO:teuthology.orchestra.run.gibba002.stderr:mon.c: injectargs:mon_health_to_clog = 'false' 2021-02-10T11:42:40.439 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:40.437 7f25209c9700 1 -- 172.21.2.102:0/109960675 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f25000186c0 msgr2=0x7f250001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:40.439 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:40.437 7f25209c9700 1 --2- 172.21.2.102:0/109960675 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f25000186c0 0x7f250001abc0 crc :-1 s=READY pgs=1521 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:40.439 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:40.437 7f25209c9700 1 -- 172.21.2.102:0/109960675 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f251c0c4660 msgr2=0x7f251c0c3de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:40.439 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:40.437 7f25209c9700 1 --2- 172.21.2.102:0/109960675 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f251c0c4660 0x7f251c0c3de0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 rx=0x7f250c013b20 tx=0x7f250c013d50).stop 2021-02-10T11:42:40.440 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:40.437 7f25209c9700 1 -- 172.21.2.102:0/109960675 shutdown_connections 2021-02-10T11:42:40.440 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:40.437 7f25209c9700 1 --2- 172.21.2.102:0/109960675 >> [v2:172.21.2.113:3301/0,v1:172.21.2.113:6790/0] conn(0x7f251c0c4660 0x7f251c0c3de0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:40.440 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:40.437 7f25209c9700 1 --2- 172.21.2.102:0/109960675 >> [v2:172.21.2.102:3300/0,v1:172.21.2.102:6789/0] conn(0x7f251c1a35e0 0x7f251c1a39e0 unknown :-1 s=CLOSED pgs=3498 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:40.440 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:40.437 7f25209c9700 1 --2- 172.21.2.102:0/109960675 >> [v2:172.21.2.113:3300/0,v1:172.21.2.113:6789/0] conn(0x7f251c09feb0 0x7f251c0be270 unknown :-1 s=CLOSED pgs=925 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:40.441 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:40.437 7f25209c9700 1 --2- 172.21.2.102:0/109960675 >> [v2:172.21.2.102:6800/35161,v1:172.21.2.102:6801/35161] conn(0x7f25000186c0 0x7f250001abc0 unknown :-1 s=CLOSED pgs=1521 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:40.441 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:40.437 7f25209c9700 1 -- 172.21.2.102:0/109960675 >> 172.21.2.102:0/109960675 conn(0x7f251c0b1db0 msgr2=0x7f251c0c25e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:40.441 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:40.437 7f25209c9700 1 -- 172.21.2.102:0/109960675 shutdown_connections 2021-02-10T11:42:40.441 INFO:teuthology.orchestra.run.gibba002.stderr:2021-02-10 11:42:40.437 7f25209c9700 1 -- 172.21.2.102:0/109960675 wait complete. 2021-02-10T11:42:40.447 INFO:teuthology.misc:Shutting down mds daemons... 2021-02-10T11:42:40.448 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2021-02-10T11:42:40.448 INFO:teuthology.orchestra.run:waiting for 300 2021-02-10T11:42:40.454 INFO:tasks.ceph.mds.a:Stopped 2021-02-10T11:42:40.454 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2021-02-10T11:42:40.454 INFO:teuthology.orchestra.run:waiting for 300 2021-02-10T11:42:40.462 INFO:tasks.ceph.mds.b:Stopped 2021-02-10T11:42:40.462 INFO:teuthology.misc:Shutting down osd daemons... 2021-02-10T11:42:40.463 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2021-02-10T11:42:40.463 INFO:teuthology.orchestra.run:waiting for 300 2021-02-10T11:42:40.521 INFO:tasks.ceph.osd.0:Stopped 2021-02-10T11:42:40.522 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2021-02-10T11:42:40.522 INFO:teuthology.orchestra.run:waiting for 300 2021-02-10T11:42:40.595 INFO:tasks.ceph.osd.1:Stopped 2021-02-10T11:42:40.596 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2021-02-10T11:42:40.596 INFO:teuthology.orchestra.run:waiting for 300 2021-02-10T11:42:40.661 INFO:tasks.ceph.osd.2:Stopped 2021-02-10T11:42:40.661 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2021-02-10T11:42:40.661 INFO:teuthology.orchestra.run:waiting for 300 2021-02-10T11:42:40.731 INFO:tasks.ceph.osd.3:Stopped 2021-02-10T11:42:40.731 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2021-02-10T11:42:40.731 INFO:teuthology.orchestra.run:waiting for 300 2021-02-10T11:42:40.758 INFO:tasks.ceph.osd.4:Stopped 2021-02-10T11:42:40.758 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2021-02-10T11:42:40.758 INFO:teuthology.orchestra.run:waiting for 300 2021-02-10T11:42:40.826 INFO:tasks.ceph.osd.5:Stopped 2021-02-10T11:42:40.826 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2021-02-10T11:42:40.826 INFO:teuthology.orchestra.run:waiting for 300 2021-02-10T11:42:40.887 INFO:tasks.ceph.osd.6:Stopped 2021-02-10T11:42:40.887 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2021-02-10T11:42:40.887 INFO:teuthology.orchestra.run:waiting for 300 2021-02-10T11:42:40.955 INFO:tasks.ceph.osd.7:Stopped 2021-02-10T11:42:40.955 INFO:teuthology.misc:Shutting down mgr daemons... 2021-02-10T11:42:40.956 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2021-02-10T11:42:40.956 INFO:teuthology.orchestra.run:waiting for 300 2021-02-10T11:42:40.976 INFO:tasks.ceph.mgr.y:Stopped 2021-02-10T11:42:40.976 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2021-02-10T11:42:40.976 INFO:teuthology.orchestra.run:waiting for 300 2021-02-10T11:42:41.008 INFO:tasks.ceph.mgr.x:Stopped 2021-02-10T11:42:41.008 INFO:teuthology.misc:Shutting down mon daemons... 2021-02-10T11:42:41.008 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2021-02-10T11:42:41.008 INFO:teuthology.orchestra.run:waiting for 300 2021-02-10T11:42:41.025 INFO:tasks.ceph.mon.a:Stopped 2021-02-10T11:42:41.025 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2021-02-10T11:42:41.026 INFO:teuthology.orchestra.run:waiting for 300 2021-02-10T11:42:41.056 INFO:tasks.ceph.mon.b:Stopped 2021-02-10T11:42:41.056 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2021-02-10T11:42:41.056 INFO:teuthology.orchestra.run:waiting for 300 2021-02-10T11:42:41.104 INFO:tasks.ceph.mon.c:Stopped 2021-02-10T11:42:41.104 INFO:tasks.ceph:Checking cluster log for badness... 2021-02-10T11:42:41.105 DEBUG:teuthology.orchestra.run.gibba002:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v 'OSD full dropping all updates' | egrep -v 'OSD near full' | egrep -v 'pausewr flag' | egrep -v 'failsafe engaged, dropping updates' | egrep -v 'failsafe disengaged, no longer dropping' | egrep -v 'is full \(reached quota' | egrep -v POOL_FULL | egrep -v POOL_BACKFILLFULL | head -n 1 2021-02-10T11:42:41.454 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@gibba002.front.sepia.ceph.com 2021-02-10T11:42:41.455 DEBUG:teuthology.orchestra.run.gibba002:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2021-02-10T11:42:41.904 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@gibba002.front.sepia.ceph.com 2021-02-10T11:42:41.904 DEBUG:teuthology.orchestra.run.gibba002:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2021-02-10T11:42:42.293 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@gibba002.front.sepia.ceph.com 2021-02-10T11:42:42.293 DEBUG:teuthology.orchestra.run.gibba002:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2021-02-10T11:42:42.706 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@gibba002.front.sepia.ceph.com 2021-02-10T11:42:42.706 DEBUG:teuthology.orchestra.run.gibba002:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2021-02-10T11:42:43.094 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@gibba013.front.sepia.ceph.com 2021-02-10T11:42:43.094 DEBUG:teuthology.orchestra.run.gibba013:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2021-02-10T11:42:43.552 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@gibba013.front.sepia.ceph.com 2021-02-10T11:42:43.553 DEBUG:teuthology.orchestra.run.gibba013:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2021-02-10T11:42:43.958 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@gibba013.front.sepia.ceph.com 2021-02-10T11:42:43.959 DEBUG:teuthology.orchestra.run.gibba013:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2021-02-10T11:42:44.366 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@gibba013.front.sepia.ceph.com 2021-02-10T11:42:44.366 DEBUG:teuthology.orchestra.run.gibba013:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2021-02-10T11:42:44.762 INFO:tasks.ceph:Archiving mon data... 2021-02-10T11:42:44.763 DEBUG:teuthology.misc:Transferring archived files from gibba002:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/khiremat-2021-02-10_10:13:45-fs-wip-khiremat-mgr-authid-bp-7-nautilus-distro-basic-gibba/5872718/data/mon.a.tgz 2021-02-10T11:42:44.763 DEBUG:teuthology.orchestra.run.gibba002:> mktemp 2021-02-10T11:42:45.100 INFO:teuthology.orchestra.run.gibba002.stdout:/tmp/tmp.Vom2wuMLoi 2021-02-10T11:42:45.100 DEBUG:teuthology.orchestra.run.gibba002:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.Vom2wuMLoi 2021-02-10T11:42:45.660 DEBUG:teuthology.orchestra.run.gibba002:> sudo chmod 0666 /tmp/tmp.Vom2wuMLoi 2021-02-10T11:42:46.342 DEBUG:teuthology.orchestra.remote:gibba002:/tmp/tmp.Vom2wuMLoi is 1MB 2021-02-10T11:42:46.740 DEBUG:teuthology.orchestra.run.gibba002:> rm -fr /tmp/tmp.Vom2wuMLoi 2021-02-10T11:42:47.075 DEBUG:teuthology.misc:Transferring archived files from gibba013:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/khiremat-2021-02-10_10:13:45-fs-wip-khiremat-mgr-authid-bp-7-nautilus-distro-basic-gibba/5872718/data/mon.b.tgz 2021-02-10T11:42:47.076 DEBUG:teuthology.orchestra.run.gibba013:> mktemp 2021-02-10T11:42:47.410 INFO:teuthology.orchestra.run.gibba013.stdout:/tmp/tmp.J6funzOSnW 2021-02-10T11:42:47.411 DEBUG:teuthology.orchestra.run.gibba013:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.J6funzOSnW 2021-02-10T11:42:47.878 DEBUG:teuthology.orchestra.run.gibba002:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:42:47.883 DEBUG:teuthology.orchestra.run.gibba013:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:42:47.989 DEBUG:teuthology.orchestra.run.gibba013:> sudo chmod 0666 /tmp/tmp.J6funzOSnW 2021-02-10T11:42:48.677 DEBUG:teuthology.orchestra.remote:gibba013:/tmp/tmp.J6funzOSnW is 1MB 2021-02-10T11:42:49.037 DEBUG:teuthology.orchestra.run.gibba013:> rm -fr /tmp/tmp.J6funzOSnW 2021-02-10T11:42:49.370 DEBUG:teuthology.misc:Transferring archived files from gibba013:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/khiremat-2021-02-10_10:13:45-fs-wip-khiremat-mgr-authid-bp-7-nautilus-distro-basic-gibba/5872718/data/mon.c.tgz 2021-02-10T11:42:49.370 DEBUG:teuthology.orchestra.run.gibba013:> mktemp 2021-02-10T11:42:49.705 INFO:teuthology.orchestra.run.gibba013.stdout:/tmp/tmp.8SYriMJ0xf 2021-02-10T11:42:49.705 DEBUG:teuthology.orchestra.run.gibba013:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.8SYriMJ0xf 2021-02-10T11:42:50.270 DEBUG:teuthology.orchestra.run.gibba013:> sudo chmod 0666 /tmp/tmp.8SYriMJ0xf 2021-02-10T11:42:50.949 DEBUG:teuthology.orchestra.remote:gibba013:/tmp/tmp.8SYriMJ0xf is 1MB 2021-02-10T11:42:51.346 DEBUG:teuthology.orchestra.run.gibba013:> rm -fr /tmp/tmp.8SYriMJ0xf 2021-02-10T11:42:51.679 INFO:tasks.ceph:Cleaning ceph cluster... 2021-02-10T11:42:51.679 DEBUG:teuthology.orchestra.run.gibba002:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2021-02-10T11:42:51.683 DEBUG:teuthology.orchestra.run.gibba013:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2021-02-10T11:42:52.026 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2021-02-10T11:42:52.026 DEBUG:teuthology.orchestra.run.gibba002:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2021-02-10T11:42:52.029 DEBUG:teuthology.orchestra.run.gibba013:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2021-02-10T11:42:52.372 INFO:teuthology.orchestra.run.gibba002.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2021-02-10T11:42:52.373 INFO:teuthology.orchestra.run.gibba013.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2021-02-10T11:42:52.378 INFO:tasks.ceph:Archiving crash dumps... 2021-02-10T11:42:52.380 DEBUG:teuthology.misc:Transferring archived files from gibba002:/var/lib/ceph/crash to /home/teuthworker/archive/khiremat-2021-02-10_10:13:45-fs-wip-khiremat-mgr-authid-bp-7-nautilus-distro-basic-gibba/5872718/remote/gibba002/crash 2021-02-10T11:42:52.381 DEBUG:teuthology.orchestra.run.gibba002:> sudo tar cz -f - -C /var/lib/ceph/crash -- . 2021-02-10T11:42:52.725 DEBUG:teuthology.misc:Transferring archived files from gibba013:/var/lib/ceph/crash to /home/teuthworker/archive/khiremat-2021-02-10_10:13:45-fs-wip-khiremat-mgr-authid-bp-7-nautilus-distro-basic-gibba/5872718/remote/gibba013/crash 2021-02-10T11:42:52.726 DEBUG:teuthology.orchestra.run.gibba013:> sudo tar cz -f - -C /var/lib/ceph/crash -- . 2021-02-10T11:42:53.070 INFO:tasks.ceph:Shutting down logrotate 2021-02-10T11:42:53.071 DEBUG:teuthology.orchestra.run.gibba002:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:42:53.075 DEBUG:teuthology.orchestra.run.gibba013:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:42:53.436 DEBUG:teuthology.orchestra.run.gibba002:> sudo rm /etc/logrotate.d/ceph-test.conf 2021-02-10T11:42:53.780 DEBUG:teuthology.orchestra.run.gibba013:> sudo rm /etc/logrotate.d/ceph-test.conf 2021-02-10T11:42:54.125 INFO:tasks.ceph:Compressing logs... 2021-02-10T11:42:54.125 DEBUG:teuthology.orchestra.run.gibba002:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2021-02-10T11:42:54.129 DEBUG:teuthology.orchestra.run.gibba013:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2021-02-10T11:43:13.465 INFO:tasks.ceph:Archiving logs... 2021-02-10T11:43:13.465 DEBUG:teuthology.misc:Transferring archived files from gibba002:/var/log/ceph to /home/teuthworker/archive/khiremat-2021-02-10_10:13:45-fs-wip-khiremat-mgr-authid-bp-7-nautilus-distro-basic-gibba/5872718/remote/gibba002/log 2021-02-10T11:43:13.466 DEBUG:teuthology.orchestra.run.gibba002:> sudo tar cz -f - -C /var/log/ceph -- . 2021-02-10T11:43:18.741 DEBUG:teuthology.misc:Transferring archived files from gibba013:/var/log/ceph to /home/teuthworker/archive/khiremat-2021-02-10_10:13:45-fs-wip-khiremat-mgr-authid-bp-7-nautilus-distro-basic-gibba/5872718/remote/gibba013/log 2021-02-10T11:43:18.742 DEBUG:teuthology.orchestra.run.gibba013:> sudo tar cz -f - -C /var/log/ceph -- . 2021-02-10T11:43:22.442 DEBUG:teuthology.run_tasks:Unwinding manager install 2021-02-10T11:43:22.454 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits... 2021-02-10T11:43:22.455 DEBUG:teuthology.orchestra.run.gibba002:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits 2021-02-10T11:43:22.460 DEBUG:teuthology.orchestra.run.gibba013:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits 2021-02-10T11:43:22.861 DEBUG:teuthology.run_tasks:Unwinding manager clock 2021-02-10T11:43:22.872 INFO:teuthology.task.clock:Checking final clock skew... 2021-02-10T11:43:22.873 DEBUG:teuthology.orchestra.run.gibba002:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2021-02-10T11:43:23.218 INFO:teuthology.orchestra.run.gibba002.stdout: remote refid st t when poll reach delay offset jitter 2021-02-10T11:43:23.219 INFO:teuthology.orchestra.run.gibba002.stdout:============================================================================== 2021-02-10T11:43:23.219 INFO:teuthology.orchestra.run.gibba002.stdout:+hv01.front.sepi 216.240.36.24 3 u 83 128 377 0.088 -1.218 1.326 2021-02-10T11:43:23.219 INFO:teuthology.orchestra.run.gibba002.stdout:*hv02.front.sepi 163.237.218.18 2 u 77 128 377 0.066 -0.697 1.317 2021-02-10T11:43:23.219 INFO:teuthology.orchestra.run.gibba002.stdout:-hv03.front.sepi 74.6.168.72 3 u 72 128 377 0.046 0.894 1.102 2021-02-10T11:43:23.219 INFO:teuthology.orchestra.run.gibba002.stdout:+hv04.front.sepi 144.172.126.201 3 u 74 128 377 0.095 -0.625 1.390 2021-02-10T11:43:23.220 DEBUG:teuthology.orchestra.run.gibba013:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2021-02-10T11:43:23.564 INFO:teuthology.orchestra.run.gibba013.stdout: remote refid st t when poll reach delay offset jitter 2021-02-10T11:43:23.564 INFO:teuthology.orchestra.run.gibba013.stdout:============================================================================== 2021-02-10T11:43:23.565 INFO:teuthology.orchestra.run.gibba013.stdout:+hv01.front.sepi 216.240.36.24 3 u 69 128 377 0.081 -0.458 0.166 2021-02-10T11:43:23.565 INFO:teuthology.orchestra.run.gibba013.stdout:*hv02.front.sepi 163.237.218.18 2 u 85 128 377 0.127 0.094 0.214 2021-02-10T11:43:23.565 INFO:teuthology.orchestra.run.gibba013.stdout:-hv03.front.sepi 74.6.168.72 3 u 75 128 377 0.077 1.225 0.057 2021-02-10T11:43:23.565 INFO:teuthology.orchestra.run.gibba013.stdout:+hv04.front.sepi 144.172.126.201 3 u 23 128 377 0.065 0.157 0.036 2021-02-10T11:43:23.565 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2021-02-10T11:43:23.577 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2021-02-10T11:43:23.577 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2021-02-10T11:43:23.588 DEBUG:teuthology.orchestra.run.gibba002:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2021-02-10T11:43:23.957 DEBUG:teuthology.orchestra.run.gibba013:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2021-02-10T11:43:24.333 DEBUG:teuthology.orchestra.run.gibba002:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\)' 2021-02-10T11:43:24.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:43:24.679 DEBUG:teuthology.orchestra.run.gibba013:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\)' 2021-02-10T11:43:25.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:43:25.237 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2021-02-10T11:43:25.251 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2021-02-10T11:43:25.268 INFO:teuthology.task.internal:Duration was 3074.461168 seconds 2021-02-10T11:43:25.270 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2021-02-10T11:43:25.286 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2021-02-10T11:43:25.286 DEBUG:teuthology.orchestra.run.gibba002:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2021-02-10T11:43:25.290 DEBUG:teuthology.orchestra.run.gibba013:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2021-02-10T11:43:25.648 INFO:teuthology.orchestra.run.gibba002.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2021-02-10T11:43:25.650 INFO:teuthology.orchestra.run.gibba013.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2021-02-10T11:43:25.667 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2021-02-10T11:43:25.667 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@gibba002.front.sepia.ceph.com 2021-02-10T11:43:25.667 DEBUG:teuthology.orchestra.run.gibba002:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/*.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2021-02-10T11:43:26.012 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@gibba013.front.sepia.ceph.com 2021-02-10T11:43:26.013 DEBUG:teuthology.orchestra.run.gibba013:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/*.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2021-02-10T11:43:26.354 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2021-02-10T11:43:26.354 DEBUG:teuthology.orchestra.run.gibba002:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2021-02-10T11:43:26.358 DEBUG:teuthology.orchestra.run.gibba013:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2021-02-10T11:43:26.720 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2021-02-10T11:43:26.734 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2021-02-10T11:43:26.734 DEBUG:teuthology.orchestra.run.gibba002:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2021-02-10T11:43:27.080 DEBUG:teuthology.orchestra.run.gibba013:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2021-02-10T11:43:27.424 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2021-02-10T11:43:27.435 DEBUG:teuthology.orchestra.run.gibba002:> sudo sysctl -w kernel.core_pattern=core && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2021-02-10T11:43:27.439 DEBUG:teuthology.orchestra.run.gibba013:> sudo sysctl -w kernel.core_pattern=core && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2021-02-10T11:43:27.781 INFO:teuthology.orchestra.run.gibba002.stdout:kernel.core_pattern = core 2021-02-10T11:43:27.783 INFO:teuthology.orchestra.run.gibba013.stdout:kernel.core_pattern = core 2021-02-10T11:43:27.785 DEBUG:teuthology.orchestra.run.gibba002:> test -e /home/ubuntu/cephtest/archive/coredump 2021-02-10T11:43:28.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:43:28.120 DEBUG:teuthology.orchestra.run.gibba013:> test -e /home/ubuntu/cephtest/archive/coredump 2021-02-10T11:43:28.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:43:28.456 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2021-02-10T11:43:28.469 INFO:teuthology.task.internal:Transferring archived files... 2021-02-10T11:43:28.470 DEBUG:teuthology.misc:Transferring archived files from gibba002:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/khiremat-2021-02-10_10:13:45-fs-wip-khiremat-mgr-authid-bp-7-nautilus-distro-basic-gibba/5872718/remote/gibba002 2021-02-10T11:43:28.470 DEBUG:teuthology.orchestra.run.gibba002:> sudo tar cz -f - -C /home/ubuntu/cephtest/archive -- . 2021-02-10T11:43:28.822 DEBUG:teuthology.misc:Transferring archived files from gibba013:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/khiremat-2021-02-10_10:13:45-fs-wip-khiremat-mgr-authid-bp-7-nautilus-distro-basic-gibba/5872718/remote/gibba013 2021-02-10T11:43:28.823 DEBUG:teuthology.orchestra.run.gibba013:> sudo tar cz -f - -C /home/ubuntu/cephtest/archive -- . 2021-02-10T11:43:29.175 INFO:teuthology.task.internal:Removing archive directory... 2021-02-10T11:43:29.176 DEBUG:teuthology.orchestra.run.gibba002:> rm -rf -- /home/ubuntu/cephtest/archive 2021-02-10T11:43:29.179 DEBUG:teuthology.orchestra.run.gibba013:> rm -rf -- /home/ubuntu/cephtest/archive 2021-02-10T11:43:29.513 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2021-02-10T11:43:29.532 INFO:teuthology.task.internal:Not uploading archives. 2021-02-10T11:43:29.532 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2021-02-10T11:43:29.637 INFO:teuthology.task.internal:Tidying up after the test... 2021-02-10T11:43:29.637 DEBUG:teuthology.orchestra.run.gibba002:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2021-02-10T11:43:29.641 DEBUG:teuthology.orchestra.run.gibba013:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2021-02-10T11:43:29.971 INFO:teuthology.orchestra.run.gibba002.stdout:393221 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 10 11:43 /home/ubuntu/cephtest 2021-02-10T11:43:29.975 INFO:teuthology.orchestra.run.gibba013.stdout:393217 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 10 11:43 /home/ubuntu/cephtest 2021-02-10T11:43:29.976 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2021-02-10T11:43:30.002 INFO:teuthology.nuke:Checking targets against current locks 2021-02-10T11:43:30.090 CRITICAL:teuthology:Uncaught exception Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/virtualenv/bin/teuthology", line 33, in sys.exit(load_entry_point('teuthology', 'console_scripts', 'teuthology')()) File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/scripts/run.py", line 38, in main teuthology.run.main(args) File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/teuthology/run.py", line 412, in main report_outcome(config, archive, fake_ctx.summary, fake_ctx) File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/teuthology/run.py", line 269, in report_outcome nuke(fake_ctx, fake_ctx.lock) File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/teuthology/nuke/__init__.py", line 266, in nuke for unnuked in p: File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/teuthology/nuke/__init__.py", line 280, in nuke_one keep_logs = ctx.keep_logs File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/teuthology/config.py", line 255, in __getattr__ raise AttributeError(name) AttributeError: keep_logs